Demonstrators say agents used pepper spray and batons during the scuffle at Delaney Hall in Newark around 1 a.m. Monday.
Gabriela Soto, a 28-year-old mother of two from Kearny whose husband Martin Soto has been detained at Delaney Hall since February, organized the rally.
New Jersey Governor Mikie Sherrill visited Delaney Hall and spoke outside later Monday morning.
"We've been working with ICE to get into the facility. I was just denied that this morning. I was hoping and trying. Well, we certainly will. And that's what I've committed to doing, is continuing to try to do that," Gov. Sherrill said.
It was contentious as family members want the governor to do more.
Many relatives were scheduled to see their loved ones Monday and they too were denied, told visitation has been canceled.
Some of the detainees are protesting inside by going on hunger strikes over what they say are poor conditions.
Eyewitness News heard from one child who said his dad isn't getting his diabetes medicine.
This is a privately-run facility with 1,000 beds. There are about 300 detainees inside and they have tablets to communicate with the outside.
Lawmakers say they not only want more oversight of this facility, they want it shut down.
"The reason they want to make this as hard as possible, they won't let the governor in, is because they don't want us to tell the stories of what's happening inside. Because when we tell the stories, everyone in new Jersey and people across the country realize this is not who we are as a country, but a reflection of who we are in this moment under the Trump administration," Congressman Rob Menendez, Jr. said.
"These are not the people that Donald Trump keeps saying that they were trying to lock up. You know, there is a woman that was pregnant that says she's not getting full, OB-GYN care. There is a woman that had a miscarriage that said that she did not get the care that she needed, and was left to be able to manage that on her own," Sen. Andy Kim said.
On Saturday, Menendez and Kim visited the detention center to support detainees who are complaining that the food being served is spoiled.
----------
* Get Eyewitness News Delivered
* More New Jersey news
* Send us a news tip
* Download the abc7NY app for breaking news alerts
* Follow us on YouTube