The man fired a semiautomatic pistol and hit 71-year-old Henry Menahem, the owner of the store, in the chest. The suspect then fled south on Madison Avenue.
"I probably got here as soon as it finished and the officers and EMTs were arriving and they did phenomenal job trying to clear the streets and get help," Sabrina Valenzano said.
Emergency crews rushed Menahem to Lenox Hill Hospital where he died. Menahem was a resident of Long Branch, New Jersey.
"They took him out. They put paper under his shoulder, but you could still the blood come off. He looked very pale and yellow," Patricia Wang said.
Police aren't exactly sure how much money and/or jewelry the gunman stole, but investigators found shattered glass from the display cases scattered across the floor.
The neighborhood boasts rows of swanky boutiques and expensive jewelry shops and the incident occurred not far from the Whitney Museum of American Art. The area near the shop was cordoned off with yellow crime scene tape, but traffic was moving along Madison Avenue.
Neighborhood residents familiar with the store found it surprising the gunman got inside the business.
"Good security. Push one button to get into the next button, so the guy must have looked like a gentleman to walk in, well-dressed or whatever," Sally Weiner said.
The shop is only about 12 feet wide and about 20 feet deep and had jewelry cases hanging on the walls. No customers were believed to be in the store at the time, police said.
Police describe the gunman as a black man in his 30's, about 5-feet 10-inches tall, and 150 pounds. In addition to the blue jacket and gray slacks, he was also wearing a scarf.