I agree. Aside from the delegator returns, there is a more serious impact on on the people you upvote. If he follows your upvotes and wipes them out (or takes more) it hurts the very people you are trying to help. Having someone come along with a large downvote on a quality post really hurts content creators (it has a bigger - negative - impact on motivation than a large upvote).