The teenagers have both been removed from the home.
Jean Yates didn't want to comment, but James Yates wants to set the record straight, or at least give his side of the story.
He and his wife were arrested on Thursday night, charged with endangering the welfare of their two autistic teenagers.
Officials say conditions in the house are so squalid, it's unsafe for the 17 and 19 year old boys.
Yates says the mess is only because they're remodeling.
"I told the judge we're fixing up the house when they showed up. We're not living in that big pile of crap. We were causing the crap," Yates said.
The outside of the Pound Ridge home is littered with debris, including sheet rock, a dumpster, plywood, even a traffic light. A sign above the garage reads the twilight zone.
The investigation began last month when the 17-year old suffered a seizure. A paramedic who responded called protective services.
"We felt it was necessary to utilize the justice system to take control of that situation and see to it that certain processes were met," Pound Ridge Police Chief David Ryan said.
Child welfare officials have removed the teens. One child cannot speak, the other has limited speech.
Yates says investigators accused of him of keeping the children caged because he used soccer netting across the door of one of the rooms.
The 57-year old says no one understands the difficulty of caring for two disabled kids.
"If you have autistic kids, don't open the door because nobody understands it," Yates said.