About 10:00 Saturday night, police found Robert Alvarez, a registered sex offender, along with the boy inside a van at Haworth and Rutland streets.
Two people who live in the neighborhood called police after spotting the van with New York license plates and becoming suspicious about it.
"When I was walking by, I saw the kid in the van, and then he closed up the van, and said get inside, and that's when I realized something was wrong," Luis Colon told Action News in Philadelphia.
"The cops came, they drug the guy out of the van," said Vanessa Gonzalez. "The little boy came out, he was shaking, he was terrified."
Alvarez was questioned at the Philadelphia Police Special Victims Unit and charged with sexual assault. He remained in police custody..
Police said Alvarez was visiting family friends in Philadelphia.