The fire started shortly before 1 p.m. at the Gregory Plaza at 112 Gregory Ave.
It shot out flames and large plumes of thick, black smoke that could be seen from miles away.
Dozens of firefighters from nearly 20 fire companies battled the blaze for several hours before it was finally extinguished.
Carla Bravo has lived in the building for decades.
"It's awful. The memories. I grew up in that building, and to see my parents cry because they lost absolutely everything. It's sad. It's sad to see the building that you lived in over 25 years burned out right before your eyes," Bravo says.
She and her family of five, along with her dog, managed to escape with only the shirts on their backs.
Pablo Peres' niece was home when the fire broke out.
"She see some smoke come out, and she said to everyone 'fire, fire, fire!'" Peres says.
"Our concerns are obviously the structure and potential collapse. That's why we're willing everyone to back up," says Mayor Hector Lora.
Fire officials say it started on the top floor of the five-story building.
Some residents and firefighters were treated at the scene for smoke inhalation, cuts, bruises and other minor injuries.
The cause of the blaze at the five-story building remains under investigation.
Authorities say 162 people were displaced by the fire. Many of those residents were receiving assistance from the American Red Cross.
(The Associated Press contributed to this report.)