As a resident of Charlottesville, Virginia, I have been forced of late to spend too much time thinking about Nazis. In mid-May, a handful of white supremacists, Holocaust deniers, xenophobes, and recreational racists—among them Richard Spencer—marched through one of our parks with flaming torches in support of a Robert E. Lee statue that has been slated to be sold by the City Council. The demonstration grabbed headlines worldwide, the statue’s removal has been placed on a six-month hold by a judge, and the Ku Klux Klan is now seeking permission to march here in July. A few weeks after the first march, a Facebook post from a local black farmer went viral due to its suggestion that the arrival of the white supremacists was more a culmination than an inciting incident, and that the fight over the Lee monument was empty symbolism that distracted from a meaningful discussion about the systemic racism that already exists here. The post included the claim that “it isn’t Richard Spencer calling the cops on me for farming while Black. It’s nervous White women in yoga pants with ‘I’m with Her’ and ‘Coexist’ stickers on their German SUVs.” White women in yoga pants were upset. Alt-right websites rejoiced.

My little city in central Virginia has become the stuff of reality TV. The local police, who didn’t see the Lee Park thing coming, are dialed up to 11. And with threats, incitement, and actual assaults perpetrated both by alt-right sympathizers and the protesters who oppose them, their job is no longer to stand back but to surge in almost as soon as the shouting begins. Now, when we come to meet in our town square, we are uncertain of whether we are suiting up for events that fete the Constitution or violent altercations for which we should park with an eye to high-speed retreats. Lee Park itself, where my babies learned to walk, has become ground zero for people expecting the worst.

This is how I felt as I headed to a local counter-protest the morning of May 31: afraid for the first time in my 16-year residence in a town I love. I was afraid that the cycle of arrests and assaults that have followed the Richard Spencer march would lead to more arrests and assaults, afraid about where we parked the car because white supremacists in this town have followed protesters home from rallies, afraid for the first time in the small town where my kids walk everywhere alone. For the first time in a lifetime of journalism, I was also afraid to wear my press credentials because today, in this town, they might invite punching.

Last week, I had come to a place where I was thinking—if not saying aloud—that maybe it was time for me and the First Amendment to see other people. It’s not me, to be sure, it’s the First Amendment—or at least what’s become of it. I am weary of hate speech, wary of threats, and tired of the choice between punching back and acquiescing. I am sick to death of Nazis. And yet they had arrived, basically on my doorstep.

For the Framers, the thinking went, free speech was just speech, nothing more and nothing less. The best way to deal with the most appalling speakers would be to ignore them, in the hope that they would go away or drown trying to be heard. That they wouldn’t survive the marketplace of ideas. It’s the same reason we tried to ignore Donald Trump for so long or at least failed to take him seriously. Or so I wrote in 2015. We tried to ignore Steve Bannon and Milo Yiannopoulos. We tried to ignore Ann Coulter and Richard Spencer. We ignored them for so long and for so hard that they now get to ignore us. And these days, people who used to feel free to shout and threaten are emboldened to punch, body-slam, and stab.

It is a short hop, we are learning, from “words can never hurt us” to actual sticks and stones and the attendant breaking of bones.

Cities that never worried about much beyond trampled flora at their Memorial Day parades now need to prepare for protests as if they are riots in the making, at tremendous cost to our collective psyche. Consider the choices available to the mayor of Portland, Oregon, after two men, Taliesin Myrddin Namkai-Meche and Ricky John Best, were stabbed to death as they tried to stop a white supremacist from harassing two young women on the light rail. Portland is in a state with robust constitutional speech protections. It has also suffered a long and frightening string of racial incidents in recent months. The white supremacist who killed two men in May had attended “free speech” rallies. And now at similar rallies everywhere, including my hometown, protesters on both sides are prepared for violence. Violence, these days, is almost expected. The only question seems to be whether cities will try to prevent bloodshed before it can happen. It’s why, immediately following the stabbings, that Portland Mayor Ted Wheeler tried to revoke permits for future alt-right protests altogether.

Of course, per the U.S. Constitution, Wheeler could not revoke these permits and stop these events regardless of how good his intentions were. And they were good: “My main concern is that they are coming to peddle a message of hatred and of bigotry,” he had told reporters. “They have a First Amendment right to speak, but my pushback on that is that hate speech is not protected by the First Amendment to the United States Constitution.” Unfortunately, he is wrong as a matter of fact and of First Amendment doctrine because if Nazis get to march in Skokie, Illinois, racists can march in Portland. (The ACLU of Oregon quickly reminded him of this on Twitter, pointing out that “The government cannot revoke or deny a permit based on the viewpoint of the demonstrators. Period.”) Soon, Mayor Wheeler’s office was walking back the claim that he was calling to suppress speech, saying he was simply trying to avoid physical violence. As an attorney friend in Portland reminded me, this is exactly why elected officials have attorneys, so they can say, “I wanted to cancel the rallies, but my lawyers wouldn’t let me.”

The conundrum facing Wheeler, though, is the conundrum facing us all. It’s the same one that has been plaguing me: Is the First Amendment allowing us to batter and attack one another in ways that are more pernicious than the act of silencing speech? Why is my city, roiled and bruised by the events of May, still allowing the KKK to march here next month?

So far in Charlottesville, we have kept violence at bay. But that fact has not felt like a promise. Last Wednesday morning, some of the white nationalists announced plans to gather again. The police showed up in full force, as did counter-protesters organized by local faith groups. Actually, the counter-protesters outnumbered the alleged white supremacists by about 30 to 1. Everyone I spoke to was anxious. That, not politics, was hanging in the air. The faith groups were trying to guess at who would pull a knife; the young man who had been hassled by the Richard Spencer crowd back in May for wearing a yarmulke was back, again in his yarmulke. Nobody knew which guy might be the guy—the one with the knife, or even the gun.

The scene wasn’t as clear-cut as you might think a confrontation between white supremacists and anti-white supremacists would be. A local candidate for city council, Kenneth Jackson, was off to the side trying to talk. He was there with the support of Jason Kessler, one of about four white supremacists in attendance. Both wanted the Robert E. Lee statue to remain standing. Jackson is black and gay. Kessler made headlines last week after he “covered” the torch-march for the Daily Caller; his piece declined to mention that in addition to attending, he’d been a speaker celebrating Holocaust denialism and white superiority. His article still stands, a valoric love song to white supremacy, now with an editor’s note appended that reads “The author notified The Daily Caller after publication that he spoke at a luncheon May 14 on behalf of an effort to preserve the monument.” After spending the subsequent weeks being harassed everywhere he went, he was back in the park. He carried a megaphone he did not use.

It is ever more clear to me that the free press—which exists, to make an obvious point, because of the First Amendment—is the enemy of the white supremacists who keep talking about free speech. Kessler blames the press for everything, including his now-terminated contract with the Daily Caller. But even Jackson posted a Facebook rant about a news account of last Wednesday’s protest that he felt mischaracterized the event. When the revolution comes, it will be because someone who felt he had important things to say felt wronged by the media.

While the religious groups sang songs and prayed at the foot of the monument, Kessler held forth about Jewish nepotism and the “white guilt” that infected the faith leaders leading the counter-protest. Jackson, who has taken the public position that he wants to preserve the Lee statue, lectured Kessler about racism and homophobia, then turned on the people of faith for caring less about the lived experiences of the black citizens of Charlottesville than they do about symbols like the Lee statue. “When Dr. King came here,” said Jackson, according to an account in a local paper, “he talked about peace and unity. He didn’t try to make white people feel guilty about the past.” He advised local civil rights activists to spend their time working on issues like affordable housing rather than showing up to protests. When a spontaneous prayer circle erupted, the Lee supporters held hands. Kessler opted out of hand-holding.

At this point, activist Veronica Fitzhugh approached Kessler and Jackson with a Bible in her hand. She had been one of the people shouting at Kessler to “fucking go home” as he ate at a restaurant on the downtown mall—indeed, she was later arrested for it and charged with assault and disorderly conduct. Now she hugged Jackson, and hugged some of the Lee supporters, and said she was asking for forgiveness. Kessler was not hugging, either.

The protests ended, in the shadow of the still-standing Robert E. Lee, with Fitzhugh and Jackson engaged in an ontological debate about the constitutional scope of protected free speech. Jackson felt that screaming at Nazis in public was illegal, but Fitzhugh thought it was protected. Local police officials declined to weigh in, at least then. When Jackson and Fitzhugh called it a draw and everyone stood down, the police were dispatched to the holy work of illegal left turns. As we departed Lee Park, Kessler sat on a park bench alone, checking his phone.

The news cameras, the cellphones, and the voice recorders reported that nothing violent transpired during the sequel to the flaming torch march that tore Charlottesville apart. That was true. But last Wednesday was about more than the absence of bloodshed. A black man, running for City Council on a pro–Robert E. Lee statue platform, tried to explain to a black woman who will never, ever give an inch for a Nazi, that symbols are just symbols in the middle of a city that is tearing itself to bits over, well, symbols. What the fear and the calls for banning marches misses—what I doubted before I went to see it for myself—is that an actual conversation about speech, race, fury, and pain, happened in a city park.

I can’t help but feel, in some way, that we got away with something last Wednesday. If we did, we may not be as lucky next time.

But to guarantee an escape from conflict, from violence, requires censorship. To have free speech in this moment, when the stakes are so high, is to live with fear. This is not an easy thing to confront—or to accept. If everyone had just stayed home last Wednesday in Charlottesville, there would have been no need to be afraid. There would also have been no dialogue.

What I saw on Wednesday reaffirmed my conviction that conversation might still be our best chance of getting out of this mess. Free speech is just free speech. It takes actual humans making the effort to talk to each other to transform speech into something more vital and more valuable. Conversations don’t always work. They may sometimes go wrong—horribly, terribly wrong. But I know someone who left that park with the phone number of someone from the “opposing” side. I saw people who showed up nervous, but showed up anyhow. The First Amendment will never be able to protect us from horrible words and horrific acts. It does guarantee that we’ll keep talking.