Michael Penn, 25, of the unit block of Birkshire Road in New Castle, Delaware, was arrested on Tuesday.
Investigators say it all began on Sunday after police were contacted about a recent theft that was discovered at the New Castle Verizon store.
The theft happened in late June, police said, and surveillance video showed the suspect, identified as Penn, stealing a Samsung tablet and another item.
After police identified Penn as their suspect, they found him with his three children - carrying his 8-month-old and a 5-year-old, with his 7-year-old riding a bike next to him.
Police say the officers told Penn to put the children down, and he refused.
Meanwhile, police say, Penn was seen putting something down the 8-month-old's pants.
Officers took Penn into custody, and allegedly found 16 bags of heroin in that baby's pants.
Penn is facing a list of charges including Shoplifting, three counts of Endangering the Welfare of a Child, Resisting Arrest, Possession of Heroin, Possession of a Controlled Substance and Possession of Drug Paraphernalia.
Penn was released on $4,500.00 unsecured bond.
The children were turned over to Penn's Grandmother and, police say, the Division of Family Services will be notified.