Well, if you limit who can downvote, you create a centralized system which can not fight well against censorship. And secondly, this does not stop a rich guy to buy himself a million dollars worth of Steem, power up, vote himself as a witness, then downvote whoever he wants :))