Some of those attending were in fact vaccinated, but say they were taking part in the protest as a matter of principle.
The crowd filled an entire city block on the Upper East Side from curb to curb.
Right now 71% of FDNY members are fully vaccinated. Of those vaccinated, 65% of firefighters are vaccinated, 80% of EMS are vaccinated and 83% of civilian members are vaccinated.
If that number doesn't go up by Friday, there are concerns there won't be enough firefighters to staff the city's 218 firehouses.
The union warned Wednesday after some failed negotiations with the city, response times for emergency calls could slow to 15-20 minutes and lives could be put at risk. That's even with forced overtime.
The president of the city's fire union told Eyewitness News that he's urging his unvaccinated members to defy the mayor's order and report for work on Monday.
"What happens on November 1 will be on him, I'll tell my members to go to work, if he sends them home, it's on him," said Uniformed Firefighters Association President Andrew Ansbro.
The union's order sets up a showdown with Mayor Bill de Blasio who has insisted the vaccination mandate is non-negotiable.
"No, there's no discussion of changing the deadline," de Blasio said. "The bottom line is can we do the things we need to do first of all, to keep this city safe for the long run? Well, that means getting people vaccinated. Yes, we can do that. Yes, we have a plan to do that. And then can we keep all the other services moving in a way that protects New Yorkers? Yes."
All municipal workers are required to show proof of vaccination by 5 p.m. Friday or be placed on unpaid leave. It applies to roughly one in four police officers and one in three firefighters and EMS workers.
Police and fire commanders are already drawing up plans to protect New Yorkers without thousands of first responders.
In a city where there are on average 80 heart attacks every day and three serious fires every hour.
"That's on the City of New York they're making determinations to force-feed this and the ramifications of this are outlandish," said retired FDNY fire marshal Kieran Burke.
WATCH | Extended interview: Firefighters' union President Andrew Ansbro:
Some demonstrators said they don't trust the vaccine. But many are refusing to get vaccinated on principle.
"I'm an American and it's a freedom of choice," an unidentified FDNY firefighter said. "Everybody should have a freedom of choice. I've had it. I've got the vaccine myself, personally. But still, it's a freedom of choice for some of these guys."
RELATED | NY vaccine mandate: Court hears arguments on nurses' challenge
Unvaccinated firefighters will be placed on leave for at least 30 days will not have medical or union benefits taken away from them. Ansbro said that was "a relief to a lot of our members who are still struggling with making this decision"
Fire Commissioner Daniel A. Nigro released the following statement:
"The Department must manage the unfortunate fact that a portion of our workforce has refused to comply with a vaccine mandate for all city employees. We will use all means at our disposal, including mandatory overtime, mutual aid from other EMS providers, and significant changes to the schedules of our members. We will ensure the continuity of operations and safety of all those we have sworn oaths to serve."
The NYPD is also subject to this mandate. They are 78% vaccinated and may also be looking at shortages.
In an internal video, NYPD Commissioner Dermot Shea, who has expressed bewilderment at the level of resistance to the vaccine, urged his members to get vaccinated for the sake of the city.
"You gotta do it now," Shea said. "To help me, to help the city and keep you and your families and your partners safe from this."
The city is offering a $500 incentive to city employees to get vaccinated by Friday. The incentive does not apply to anyone who gets the vaccine after that.
ALSO READ | 12-year-old boy describes crocodile attack, rescue at Cancun resort
MORE CORONAVIRUS COVID-19 COVERAGE
New York City COVID-19 Vaccine Tracker
New Jersey COVID-19 Vaccine Tracker
Centers for Disease Control and Prevention on coronavirus
Submit a News Tip or Question