The issue isn’t whether it’s justifiable to punch a particular Nazi. It’s what gets normalized when many people tend to apply the Nazi label broadly. When Mitt Romney, Paul Ryan and many others have regularly been called Nazis it makes one a little worried about just who people think are OK to punch.
Of course almost no one loves Nazis. They are genocidal fanatics. However there’s an other group who has a rather long history of genocide, mass murder and are often considered fanatics: communists. To put the shoe on the other foot, do you think punching communists should be normalized? Again, before you answer, think through who often gets called communists.
The problem with most thinking on these issues is they always conceive of the issue in cases where everything is clear. The problem is that most social rules we make tend to get enacted in situations that aren’t clear. Make a rule about torture when an atomic bomb is about to go off and people will apply it to far less clear situations like trying to find Osama bin Laden. Make a rule about punching Nazis and most likely anyone some group considers an extremist will suffer violence.
Liberalism (classically defined) sees pluralism and free speech rights as so important precisely because no one agrees upon where the limits of acceptability are. Traditionally that means we have tried to make the public sphere as free from restriction as possible. It’s very easy to think your case is unique. But realize not everyone will agree with you. They’ll see that your exception means their exception is also justified. Before long, we lose the ability to have any common ground of openness at all.
What we should fear is that once people like Richard Spencer are fair game then people will think other extremists like “jihadists” are also fair game. Problem is they may not be so choosey about who is or isn’t an jihadist. Suddenly there’s much more violence than I think defenders of violence against neo-Nazis anticipated.
 This especially seems true on college campuses where a tradition of illiberalism has sprung up that has eerie echoes of the beginnings of violence from the late 60’s and early 70’s.