The defining principle behind free speech laws and philosophies is the idea that we need to protect people's right to criticize the state and those in power. Part of that is protecting things that aren't necessarily direct criticism but which push back against cultural norms and powerful institutions and individuals. We definitely want to make sure people can safely say that the current leader of the nation is an asshat, that we should all be communists, that one religion or the other is nonsense, or that patriarchy is wrong. Even if I don't agree with all of these sentiments there is a real public good in letting people talk about them without fear of government persecution.
There is some confusion on that last bit, so it should be clearly noted that free speech is NOT consequence free speech. You may be entitled to say that Islam is evil, but Muslims are free to tell you that Christianity is evil right back. Atheists might tell you that you are stupid and wrong because all religions are terrible, and perhaps the Jews will laugh at you and tell you that their religion is way too cool for you and you aren't invited. The government should not censure, harass, or imprison you for saying these things, but other people are free to disagree and there will be social consequences for your statements. These social conequences are not only acceptable, but desirable.
Just because the basic tenets of free speech are admirable does not mean that you get to say whatever you want without any pushback.
The problem with free speech right now is that it is being invoked as though saying anything you want is the goal. It isn't. There is nothing inherently good about spouting off your opinion. The good comes in the improvement in human circumstances that occurs when people are free to tell those in power that they are stupid and bad. The goal of free speech is to make the world better for people to live in.
So when someone wants to speak publicly about their desire to murder everyone of a particular group or to simply oppress them brutally, remove their rights, or throw them out of their homes, we must decide if this sort of speech is something that we ought to protect under the banner of free speech. The important question is this: Is protecting this kind of speech helping to make things better for humanity?
Obviously the answer is no.
So while you can make a coherent argument that we must protect the rights of people we hate to speak their mind at the end of it you have to justify it on the basis of improving human life, not just upholding a particular social custom and set of laws. Laws and customs are created to serve humans, not the other way around.
When a person argues that we ought to be communists I disagree with them. However, I think the harm that comes to society from letting them speak their mind is not significant, even if you assume there is harm at all. When a person argues that they should be free to unfurl the swastika and advocate the destruction of queer people, Jews, people of colour, etc, they are imposing a dire and terrible burden on society. That burden is of course primarily borne by those who are already oppressed which makes it even worse. There is no demonstrable benefit to society whatsoever in allowing this behaviour so we have a moral imperative to stop it.
While I like the concept of free speech in theory, at the moment it is brought up consistently to defend reprehensible conduct. This is a huge problem because there are plenty of legitimate cases of speech needing defending and yet if you post on the internet that you are pro free speech in a vacuum many or most people assume you are taking a pro Nazi stance. Free speech is so consistently being invoked as a way to excuse evil that those two words are being tainted with a dark shroud.
The idea of free speech is to protect the powerless to push back against the institutions that might otherwise oppress them. It is not an assumption that trying to organize genocide is something that we all ought to protect. It is there to make human life better, to defend those that cannot defend themselves from those that would hurt them. Defending that concept, and indeed defaulting to letting people speak when we aren't sure, is a fine and noble thing.
But Nazis chanting that they want to murder all the people who aren't like them are far beyond the pale. We can exercise judgement to know that they are evil and must be stopped, and we are capable of judging that their speech is not something that should be protected.
That doesn't mean that law and policy surrounding free speech is easy. On the contrary, it is nearly always thorny and difficult. Stopping the Nazis without randomly squashing other people is a difficult task from an administrative standpoint, and we don't want overreach. However, this is a challenge worthy of our efforts, and one we must work hard to succeed at.