Just after 9:30 Friday morning their small boat capsized, sending the pair into the Hudson River.
On the last day of winter, the water temperature was barely 50 degrees and the effects of hypothermia were already starting to set in.
"We started treating them for hypothermia right away and tried to get them in the boat as quick as possible," said Peter Carpenter of the Ossining Police Department.
It's not clear what caused the 16-foot aluminum boat to flip over about a half mile north of Croton Landing.
Fortunately the fishermen were close enough to shore that people in the nearby park heard their cries for help.
Even though Croton police have a marine unit, the boat is not yet in the water for the season. It was determined an Ossining police boat could respond faster.
Ossining officers have been on patrol since the start of striper season, which attracts many fishing vessels.
"If we had to launch our boat, it would have been another 15 or 20 minutes. Time is of the essence in a situation like that," Carpenter said. "A couple of minutes either way can make the difference."
The unidentified men were taken to Westchester Medical Center with mild hypothermia. They are expected to make a full recovery.