All this is absurd. Free speech is free, until it gets political. Your metaphor about punching a Nazi threatening to kill you applies at the higher political level. If a GROUP of Neo Nazis threaten to kill or enslave another GROUP of people, how does your metaphor stand? It's not personalized hatred, it's politically generalized hatred. The worst crime of Nazis in Germany wasn't that they INDIVIDUALLY threatened each and every jew of death, it is to generalize, once in control of political power, the extermination of jews. How do you punch a State in the face?
The real question about free speech and it's limits is where you set the boundary between voicing your personal opinion and joining a well defined group with a political agenda which, if realized, will restrict the freedom of others. Now we can debate when is the right TIME to censor or crack down on a group of people whose aim is to kill the freedom of others: is it when they first voice it? When they start to gain power (be elected in a parliament), once they have voted laws that restrict freedoms of others? But once you get to that stage, how do you fight back? In your ideal world, I take it you are an anarchist, since there is no state to fight over, there is no abuse of the "legitimate use of force" (aka the police/judicial system) and therefore, hate groups cannot gain traction or implement their ideas.
But this is extremely naive.
You postulate that without a State, the vacuum of power created will NOT entice groups of people to band together and realize their political projects given that there is no higher authority to restrain them. In other words, in anarchy, how can you even remotely believe that there will not be some form of authority which emerges, one way or another? You can already see that in failed states, (many African states, which have not successfully achieved a full control over their territory), local mafias, militias and other forms of violent groups bully citizens in one way or another. Anarchy is therefore simply the "survival of the fittest", and by no means a "free" society. You can never guarantee that in an anarchic state, a group of people won't band together to conquer power through violence. That is the story of the formation of Nation States in Medieval Europe!!! Also, if you studied a bit anarchy and the movement, you will find that even during the Spanish civil war or in the Zapatist movement, there IS some form of sovereignty and control, simply it is not institutionalized, but there ARE people being judged for crimes or excluded if they hamper society, and your definition of private property probably does not exist in the way you imagine it (in Spain during the civil war, it was a collective ownership of the means of production for instance).