He was shot near the Chase Bank branch next to entrances to Madison Square Garden and Pennsylvania Station just before 12:45 p.m.
According to police, 69-year-old John Daniel Stolarz was armed with a knife when he attempted to rob the bank on West 31st Street along the Seventh Avenue side of the Garden.
New York Police Department spokesman Paul Browne said Stolarz had entered the bank and demanded $50 and $100 bills from a customer service representative. After she told him she didn't have any money, he repeated the demand while brandishing the steak knife, Browne said.
When a bank manager approached, Stolarz walked out. The manager followed him and alerted two officers in a parked patrol car, who followed Stolarz into Penn Station and up an escalator to a less crowded area, Browne said. There, they confronted him and ordered him to drop the knife, he said.
Browne said Stolarz refused several orders to drop the weapon and continued to walk away. One of the officers fired twice, striking him in the right thigh. The man was carrying identification indicating he had been a federal inmate, Browne said.
Stolarz was taken to Bellevue Medical Center in critical condition. Police said he was conscious and speaking to doctors.
Browne says Stolarz was released from a federal prison in New Jersey on Wednesday.
Ethan Harp, a freelance journalist from Manhattan, said he saw the whole thing. He said "the officer seemed very calm."
Authorities blocked off a portion of West 31st Street near the bank, which was temporarily shuttered.
The shooting happened in front of a loading dock used by Amtrak. The area is described as not heavily trafficked by pedestrians.