The 53-year-old man fell from the castle, near 79th Street, and landed on the rocks below after trying to retrieve a cigar he dropped.
He was carried out on a stretcher with two injured ankles.
He was transported to the hospital in stable condition.
The Castle is closed on Mondays.