A crowd of demonstrators against the project stood behind police barricades three blocks from the World Trade Center site and around the corner from where the mosque and cultural center would be.
Brooklyn plumber Steve Ayling says the people who want to build the project are the same ones who "took down the twin towers."
Nearby, several hundred people who support the mosque chanted "Muslims are welcome here. We say no to racist fear."
The Sunday rallies coincided with an annual motorcycle ride by a group that raises money for Sept. 11 first responders. Bikers rolled in from the two other Sept. 11 attack sites, Washington and Shanksville, Pa.