That makes 9/11 worker Andy Sullivan livid. He's organizing a rally for Sunday against the mosque.
"We had no problem with them praying there. The problem we have is erecting this mega mosque which we think is completely hurtful and inappropriate," Sullivan said.
A new poll shows 63 percent of New Yorkers oppose the Park 51 mosque and community center. Only 27 percent support it.
But when asked do Muslims have the right to build at ground zero, 64 percent say yes while only 28 percent no.
Elaine Brower is also organizing a rally for Sunday to show support for the mosque.
"If it was a synagogue or a catholic church, would we care? We are going back to 1945 when we demonized the Japanese-Americans. We do that all the time in this country. When are we going to listen," Brower said.
New York Governor David Paterson told CNN that he would like to discuss plans for an Islamic community center and mosque near ground zero in Manhattan with its developers to see whether a compromise site can be found.
"If people put their heads together, maybe we could find a site that's away from the site now but still serves the ... area. That would be a noble gesture to those who live in the area who suffered after the attack on this country, and at the same time would probably in many ways change a lot of people's minds about Islam, which is really a peaceful religion practiced by peace-loving people," Paterson told CNN's "Larry King Live" on Wednesday.
The archbishop weighed in on Wednesday, praising what Paterson's doing. He compared it to Pope John Paul the 2nd, who forced a Catholic convent to move a cross from Auschwitz after a Jewish groups called it offensive.
"He's the one who said let's keep the idea and maybe move the address. It worked there, might work here," Archbishop Timothy Dolan said.
Others say they've just about had enough of the whole controversy.
Congressman Anthony Weiner said politicians seem to be more concerned about a mosque than about getting proper health care to 9/11 responders.
"You know from the president on down I think frankly we should be focusing on tasks like health care, not deciding where a religious institution should go," Weiner said.
Controversy over the mosque reached a fever pitch over the weekend when the president offered his support for it, but then seemed to back away a day later.
He initially said Muslims "have the same right to practice their religion as everyone else in this country" and that included building the Islamic center in lower Manhattan. A day later, Obama told reporters that he wasn't endorsing the specifics of the plan.
Despite criticism from Republicans and others, President Barack Obama said Wednesday he has "no regrets" over the comments he made about the right of Muslims to build an Islamic center.
Obama's latest comment was in response to a question shouted by a reporter as he was leaving a townhall-style meeting on the economy in Ohio.
Republicans have seized on the issue, with many trying to force Democrats to either stand with the president or buck him.
Senate Majority Leader Harry Reid, D-Nev., who is locked in a tight midterm election race, was the highest profile Democrat to move away from Obama, saying through a spokesman that he thinks the mosque should be built someplace else.
"It's a really sad day for America when our politicians choose to look at a Constitutional right and use that as basis for their elections," El-Gamal said.
Speaking in Washington, New York City Mayor Michael Bloomberg re-affirmed his support for the Islamic center, saying it adds to the diversity of the city.
"It is not the government's business to get involved in religion and religious organizations, where they are or what they say or how they're funded," Bloomberg said. "This is the fundamental right that Americans have to say and pray the way they want to."
But critics like conservative commentator Pamela Geller say building the mosque so close to the sight of a national tragedy is an insult to the victims' families.
"That building was part of the attack," she said. "A part of the plane crashed through the roof. It's part of American history. It should not be turned into a mega-mosque, a 15-story mosque."
There are sidewalk chalk messages of support outside the site. "Defend Obama, Defend the Constitution," one reads. "People Fear What They Do Not Understand," reads another. There are also peace signs in front of the building.
