It's all part of a new exhibit designed to recreate the lives of the band before fame hit.
Their famous tunes were playing on a "Ruby Tuesday" night in the Meatpacking District, when the Stones opened the new exhibit -- called Exhibitionism -- that covers their entire career.
They opened up their attic and put half a century's worth of stuff on display.
"It's quite amazing, really," guitarist Keith Richards said. "I turn around, and everywhere, there is another memory."
His first guitar rests near a re-creation of the apartment he shared with Mick Jagger in London before fame came.
"I sat on the bed yesterday," Jagger said. "It felt just right."
WATCH: Sandy Kenyon's extended interviews with Mick Jagger, Keith Richards and Ron Wood on the "Exhibitionism" red carpet
The story of their journey is told in a series of rooms that leave no doubt about the influence of the blues, and the impact can be felt in the sheer volume of the memorabilia that fills 17,000 square feet.
"I want them to go away with maybe a feeling of the live atmosphere," guitarist Ronnie Wood said. "Of what it's like for us to walk on and then bang, in front of an audience."
Ileen Gallagher spent two years putting together the exhibit.
"They kind of permeated every part of our culture," she said.
Some of the costumes had to be retrieved from Jagger's daughters, who had borrowed outfits from their dad.
"They stole them basically out of a trunk," he said. "It was a good opportunity so I could say 'OK, so, now I'm doing this exhibit, so now you have to give them back.' It was a good excuse, instead of me saying, 'I want them back.'"
His favorite part of the exhibit, the one that evokes so many memories, are his handwritten lyrics. They also have the diaries of Keith Richards, who is as cool as ever just a few weeks before he turns 73.
"Exhibitionism" runs through March 12. Visit StonesExhibitionism.com/ for more details.