However, authorities insist it is just a processing center.
After marching to Federal Plaza and chanting outside, some of the protesters moved to the intersection of Broadway for and Thomas to sit and block traffic as a show of civil disobedience.
Within a few minutes the NYPD moved into to arrest some of the protesters for disorderly conduct.
The protesters say that the 10th floor is being used as a covert ICE detention facility and not a processing center.
They allege that those being detained are being held in over-crowded rooms, without beds, showers, medical care or access to lawyers, and these are people who have come to court for hearings only to be allegedly stripped of their rights by federal officers.
The protesters want unrestricted access to the 10th floor by elected officials, journalists, and faith leaders.
Lawmakers have been denied access to the site. Eyewitness News has seen that happen a number of times, including in June when congressmen Dan Goldman and Jerry Nadler were turned away.
The city's Comptroller Brad Lander was arrested while escorting an immigrant from a hearing.
On Friday, some of the faith leaders at the protest also asked to enter the building, but were not granted permission.
"Just to be a presence, we're here to say that the American people are opposed to these kinds of policies. And for ICE agents, maybe that will lead some of them to reconsider their career choice," said Jeffrey Courter with Presbyterian Justice Ministries.
Also on Friday, lawmakers in D.C., including Goldman, sent a letter to the Inspector General of the Department of Homeland Security, which oversees ICE, demanding investigations into reports of U.S. citizens being wrongfully detained by ICE agents, and sometimes violently arrested.
"We need to keep standing our ground as New Yorkers, and this is what it is about today, New Yorkers in all walks of life standing up for our immigrant neighbors," said Renata Pumarol with Climate Defenders.
----------
* Get Eyewitness News Delivered
* More Manhattan news
* Send us a news tip
* Download the abc7NY app for breaking news alerts
* Follow us on YouTube