Protesters line streets ahead of President Trump's New York City stay

MIDTOWN, Manhattan (WABC) -- President Trump has been under fire, even taking heat from within his own party for failing to call out white supremacists in Charlottesville on Saturday.

The White House tried to clean up some of that fallout, but the president himself has remained silent. The other side showed up at Trump Tower on Sunday, and they were anything but silent.

Protesters started out at Columbus Circle, marching through Times Square and to Trump Tower. A few hours later, several different groups converged on President Trump's home. A couple hundred flanked Trump Tower on the north and the south. The events that took place in Charlottesville fueled their anger and disgust.

"There was a time when these people marched around with hoods on. And that's how it should be. They shouldn't even be there but the fact that they feel bold enough to come out in broad daylight with their faces uncovered having national press take pictures of them, and they just don't care, that's f****** absurd," says protester Cass Vitacco.

Eventually the crowd in front of Trump Tower, grew too large and police had to shut down traffic on 5th Avenue as demonstrators unleashed their fury.

There were a handful of Trump supporters, too - some tried to engage the other side.

"We have a very divided country, and I'm not a white supremacist or a racist - in fact, I'm far from that. I don't want to be lumped into that group with the right, they're painting everybody with a broad brush. The problem is there's just as many problems on the far left, which is the problem in Charlottesville," said Trump Supporter Bryan Nelson.

Within seconds, the protest turned into not a lot of talking and not a lot of listening.

What happened on Sunday night is not unlike what is happening around the country - both sides shouting over one another.

Meanwhile, President Trump is expected to spend Sunday night at Trump Tower.

CLICK HERE for street closures ahead of President Trump's visit.
Copyright © 2020 WABC-TV. All Rights Reserved.