"To be perfectly honest with you, I really wish they could buy our house and we can live," said Lucia Frazier.
Crews working on the project are digging as they work to connect a pier to the bridge; the overall bridge project is expected to take four years.
Vibrations from the project shake Frazier's windows and have cracked her walls, she said, pointing out cracks overhead and over the doors of her home.
Neighbor Raymond Francis said he worries that the vibrations could ruin the foundation of his century-old home. "According to their engineers, we should never be feeling any vibrations here," he said.
Resident Brian Cott said the closer you live to the construction, the worst it gets and that in addition to the shaking, there's also a lot of dust. "The outside of the windows are filthy. We clean them and it comes right back," he said.
Residents said the Port Authority, which oversees the project, offered them up to $10,000 to replace their damaged windows, but the Office of the Mayor of Bayonne urged people in a letter not to take the deal because doing so means they would forfeit their right to sue for any damages related to the project.
The Port Authority disputes that, and says it has representatives checking in with residents near the construction work. It also said it is offering hotel vouchers to residents if the neighborhood becomes too loud or dirty because of the work.
"I know it's a big project. I know what's at stake and you have to take our needs into consideration as well," said Cott.
The Port Authority is setting up offices to receive formal complaints, but residents said they are stuck. They can't sell -- no one wants to buy a home next to a loud, long construction project -- so they can't move.
"This is a major construction site in the middle of a residential area. They should've of approached us to buy these houses," said Frazier.