"This is racist, it is not accomplishing anything. It is dividing," said NYC councilmember Charles Barron.
At a chaotic news conference near ground zero Thursday afternoon, rally organizers predicted thousands would turn out on 9/11, in a show of solidarity with 9/11 families and with Muslims, wanting to build a cultural center and mosque nearby.
It sounds like a replay of a couple of weeks ago with two different protests, one against the mosque, and another in favor.
But should this be happening on 9/11?
Organizers said they were forced to by the other guys, those against the mosque.
"We called our action after they had coverage and advertising campaigns that went on for weeks and was not being countered. In other words they seized the day of September 11th," said Sarah Flounders, of the 9/11 Coalition Against Anti-Muslim Bigotry.
A few 9/11 family members have called it insensitive to hold protests and rallies on a day of remembrance.
But neither side is backing down.
"My son is dead, he was murdered by Muslims. He can't speak anymore and I will keep speaking out for him and that's the American way, we'll let our voices be heard," said Jim Riches, 9/11 Parents and Families of Firefighters.
Could all of the rallies and protests be for naught?
The governor is once again hinting that the state is a step closer to brokering a deal to sell this property, and the mosque could move.
"I do feel the slightest bit of movement that some of the people of good will, all of them, the people putting this together are now starting to see that how can their mission ever be realized if it is born out of such conflict," Governor Paterson said.
Sources say there are no plans to move, but the door did open ever so slightly Wednesday night, when the Imam behind the mosque said he never imagined his plan would upset so many people.
"If I knew this would happen, that it was going to cause this kind of pain I wouldn't have done it," Rauf said on "Larry King Live".