Seven of the family's other relatives lived in the home, including a young girl with special needs, officials say -- and several firefighters are also related to the victims.
The Paterson community is hurting after the immense loss.
At 15 Emerson Avenue, the site of the fire, a memorial shows the depths of their love and pain.
Laid across the lawn are stuffed animals that will never be held and dolls that will never be played with.
The blaze raged with such ferocity it swallowed up the back of the home within minutes, melting the tires of the car sitting in the driveway, the fence on one side, and the siding of the home next door.
Officials say it was the neighbors in the damaged home next door that first smelled danger.
"They couldn't find out where the smoke came from, and my wife went to the back and she saw the fire started from the back side," neighbor Taher Ahmed recalled. His wife was the one who banged in their neighbor's door to warn the family.
Some members of the family inside were watching TV in the first-floor front room, unaware of the flames swallowing up their home from the back until their neighbor alerted them.
Fire chief Alex Alicea said strong winds fueled the quick spread of the fire onto the second level of home, where five family members became trapped.
Neighbors were stunned by the devastation, telling Eyewitness News that some relatives who escaped the blaze tried to save the victims but could not navigate the stairs, which were ablaze.
One neighbor tried to help by using a garden hose to try and suppress the flames.
The 39-year-old father, 38-year-old mother, and their three kids, ages 14, 12 and 7, all were lost to the flames.
Even as they search for the cause of the fatal fire, the local fire department is in mourning.
"It's somber ... tomorrow we have a crisis intervention team coming ... because some of our members, obviously as it has been reported, were related to the family. We're making sure that everyone involved gets the help they need," Fire Chief Alicea said.
Neighbor Taher Ahmed and his family are also grieving. He told Eyewitness News that his son was friends with the 14-year-old child who passed away, and hasn't been able to eat since Friday.
John Miller, another neighbor, described the deceased family members fondly.
"They were really good people, friendly. Every time I'd drive by, they would always wave," Miller said. "To hear about this, it's unbelievable."
The seven others living in the home survived the fire but have been displaced. The Red Cross is assisting them.