A company is out to change that, offering toilets, showers and storage lockers in a central location.
So would people pay to use it?
The man behind the company is an admitted germ-a-phobe and says too often he's avoided dining in the city, because he's been afraid of not finding a bathroom when he needs one.
So his company comes from personal experience, and he believes there's a market for what he's offering.
"It's too public, too dirty, wait till you get home," said one woman we spoke to.
That's the general consensus when it comes to public restrooms. And that's where Wayne Parks comes in.
He has a plan to open up luxury toilet facilities called POSH Stow and Go.
"The idea is a place to go to the bathroom," said Parks.
A clean place to go, something the Connecticut resident says visitors like him need.
But they're not free. Instead you buy a nearly $15 membership, and then it's up to $8 a day to use the facilities.
Many say people won't be willing to pay the 8 bucks. "It's a shame people have to pay to pee," one man said.
And Wayne gets that.
"Anybody who's saying I don't think I'll pay 8 dollars for a bathroom, I don't blame them. I probably wouldn't pay 8 bucks for just a bathroom. That's not what we're doing. The bathroom is this much of what we're offering," Parks said.
POSH Stow and Go also offers lockers, charging stations, showers and lounges.
But then again, the loo at Herald Square is free. It comes with fresh flowers and a man who cleans after every use.
And the toilet at Madison Square Park only costs 25 cents. So if you're only looking to use the john, POSH Stow and Go may not be the pit stop for you.
"If I really had to go, I'd buy a coffee and use that toilet," one woman said.
You know you've done that, at Starbucks. But if you're looking for the luxury loo, POSH Stow and Go will open a first location this summer near Penn Station. And Wayne has plans for 20 locations south of Central Park.