That is true.
However, most of those inactive accounts also have relatively low SP which means their vote won’t influence that much. What is sadder is the potential users who have more of SP and are still active but won’t take time to change their votes. Might be interesting to run some stats on this actually.