He raised his three children in the home and it's where he expected to spend his final days.
Instead, he has to attend housing court where he is threatened with imminent eviction because the deed to the house shows someone else as the owner.
Cortez's son, Ray Jr. has been helping his father navigate the legal morass that he said is a result of a decades-old scam.
"You know, it's heartbreaking. It's been too long for him to have to carry this with him. It's been devastating," Ray Jr. said.
Cortez bought the house in 1969 for $20,000, long before the million-dollar condos went up and the Barclays Center moved in.
Now the house is valued at close to $3 million.
Cortez's attorney said this type of fraud is common in gentrifying areas.
"If you go into certain neighborhoods, mostly communities of color in Brooklyn, kind of from Prospect Heights over towards East New York and then southeast Queens, everybody on every block knows somebody who has been a victim of deed theft or real estate fraud," his lawyer Bill Lienhard said.
The scam Cortez fell victim to is complicated but the complication itself is part of the scam.
Back in 2006, Cortez, seeking money for renovations, was convinced to sign over the deed to his house to a con man who had done jail time.
A deed was returned in Cortez's name, notarized by a crooked attorney, but never recorded.
The conman took out an almost $700,000 line of credit on the house and eventually, the bank sold it at foreclosure for $2 million.
Cortez didn't find out about it until a decade later.
Now, neighbors who have known Cortez for almost 40 years are rallying behind him.
"The perfect neighbor. All of us on this street are just appalled. I mean we cannot imagine what this must be doing to him emotionally," said neighbor Tom Harriman.
The Attorney General's office has a division that looks into this type of real estate fraud and is reviewing Cortez's case.
ALSO READ | Accused squatter rented out rooms in owner's house | 7 On Your Side Investigates
Accused squatter rented out rooms in owner's house
* Get Eyewitness News Delivered
* Follow us on YouTube
* More local news
* Send us a news tip
* Download the abc7NY app for breaking news alerts