Yes, you should absolutely use all or at least most of your witness votes (4 empty slots is not a big deal) for the simple reason that it makes the chain more secure. By default, vote for any witness you believe to be non-malicious.
The smaller the number of total votes in use, the easier it is for an attacker (for example, someone who hacks an exchange and steals their STEEM) to vote in their own witnesses and cause havoc.
Of course, using all your votes does not preclude changing your votes in favor of a witness who, as you say, brings something to the table.
The safety and security of the chain (which means your own investment) benefits from as many people as possible using all or at least most your witness votes. In that sense, any responsible witness brings something to the table.
The last point you make is most poignant to me, some of the top witnesses.., I know of them but what do they do besides stick around in the top twenty?
I have been to TWO Steemfests now and met several. Some of my choices are down to who they are and whether they were 'unapproachable gods' at SF3 and 4.
I see you are in the twenty. Other than knowing you have been here since day 1, command a lot of respect, I don't know what you contribute other than a lot of flags when HF21 started (which I'm grateful for as I am an abuse fighter).
You have no recent posts, and your bio says nothing.
This is not a dig at you, but unless I have been active since March 2016 (I regret missing the early years), then I have no reason to give my vote to @smooth.witness.
Others such as @gtg I have met, I spoke to at length and have dined with (at SF4), @blocktrades I had a decent conversation with on the final day at SF4. Are these reasons to give my vote?
Some of it is forging relationships and others I can see are doing great things, such as @steempeak, @curie, @roelandp and @good-karma.
It's not just you but quite a few in the top 30. We can't see what you are doing. Do you see my dilemma?
I do not disagree with your voting philosophy.
My only point was that it is better to vote for some witnesses who at least are trustworthy and competent to secure the chain even if the votes are still up for grabs by witnesses who do something more, than it is to leave the votes idle until you find such witnesses.
BTW, my witness account is pretty much only used for the witness itself, and I use this account for the social app. While I don't post all that often, I do engage regularly in comment threads (and as you noted, votes and especially downvotes).