The act of voting itself should have some kind of "reputation". People should be able to freely and openly agree or disagree on people voting, without affecting rewards. The reason for it, in my view, is to publicly make awareness of "less regular" situations. So, if a whale decides to riot and make a specific user lose all the rewards, but in this case the user is someone honest and does not deserve that, then slowly the community might be able to shift the tide, by knowing that the whale downvote is not being accepted by the community, attracting slowly others that which to shift the tide of that whale.

Not sure if I got myself understood.