Many of the hundreds of demonstrators Tuesday night came to throw their support behind immigrant rights.
They also called Trump a Putin puppet and chanted "hey, hey, ho, ho, Donald Trump has got to go."
Earlier, during an impromptu press conference in the tower's lobby, Trump again said both sides were to blame for the deadly violence last weekend in Charlottesville, Virginia.
The number of protesters was significantly smaller than on Monday, when thousands waited for hours for Trump to return to Trump Tower for the first time since his January inauguration.
They carried signs with such messages as "impeach" and "stop the hate, stop the lies." Supporters countered with "God bless President Trump."
Security remains tight around Trump Tower, as the president continues his visit to his home in Midtown Manhattan.
Large dump trucks filled with sand surround the Fifth Avenue high-rise building as a security precaution. Layers of metal police barricades are also around the main entrance.
[Ads /]
Here's a look at the perimeter:
The president arrived late Monday night amid a large crowd of people who were determined to share their message with him -- both in support of and against him.
They carried signs that said "impeach" and "stop the hate, stop the lies." Supporters say "God bless President Trump."
Three people were arrested on charges including reckless endangerment and resisting arrest during the protests Monday night.
One of the most noticeable parts of the protest was an inflatable, rat-like caricature of Trump that stood by The Plaza hotel.
Demonstrators started out at Columbus Circle, marching through Times Square and to Trump Tower. A few hours later, several different groups converged on President Trump's home. There, they stood in pens that police erected across the street from Trump Tower and lined nearby blocks of Fifth Avenue hours before his expected arrival.
[Ads /]
RELATED: List of street closures during President Trump's stay in NYC
Some carried signs with such messages as "no hate" or "impeach" as chants of "black lives matter" and "love, not hate - that's what makes America great" rose above traffic noise.
While there were hundreds of people protesting President Trump's visit, about two dozen fans gathered near the Plaza hotel and chanted, "God bless President Trump." They also carried American flags and signs with such sentiments as "now is not the time for divisiveness."
"We are here to show President Trump ... that he has a lot of fans here, even in liberal New York City, even in Manhattan," said Ariel Kohane, a 46-year-old member of a group called Jews for Trump.
[Ads /]
He said he supported the president because of his views on national security, immigration and fighting radical Islamists and crime, among other issues. The anti-Trump protesters, to him, are "left-wing, liberal Communist snowflakes."
The president has been under fire, even taking heat from within his own party for failing to call out the white supremacists in Charlottesville.
This is Trump's first visit to Trump Tower since his inauguration.
(The Associated Press contributed to this report.)