The incident happened just after 11:30 a.m. near the intersection of Broadway and Prince in SoHo.
Pictures from the scene showed wooden planks all over the street, and FDNY firefighters responding.
To prevent similar incidents in the future, New York City Councilman Ben Kallos plans to introduce legislation requiring scaffolding to be removed within seven days if there is no work being done above.
"Scaffolding is there to keep us safe from the buildings, but what's going to keep us safe from the scaffolding?", said Kallos.
He said temporary scaffolding structures are often left up indefinitely because it's cheaper to leave it up than to fix the issue.
Kallos said some scaffolding remains up for years or even decades.
Two people had to be rescued from under the rubble. They and three others were taken to the hospital to be treated for minor, non-life threatening injuries.
One woman was pinned underneath the debris.
"She was scared, she asked what happened," said eyewitness Jonathan Mejia. "She didn't ask for help, she asked what happened. She was more confused. It was traumatic, you could see there was fear."
Investigators said the high wind on Sunday was to blame for the collapse. A piece of plywood "acted like a sail" and blew the whole rig down.
Cellphone video shot moments after the collapse shows bystanders running in to help people trapped:
Raw video: Bystanders help with rescues after SoHo scaffolding collapse
----------
* More Manhattan news
* Send us a news tip
* Download the abc7NY app for breaking news alerts