Kreiser, who since 2008 served as Pastor of St. Gregory Barbarigo Roman Catholic Church, resigned last June citing personal reasons.
An internal audit by the Archdiocese of New York found Kreiser had mismanaged parish funds.
Armed with this evidence, the Archdiocese contacted Rockland County District Attorney Thomas Zugibe.
Zugibe said the DA's Office started its own investigation shortly after Kreiser's sudden resignation.
According to Zugibe, over a three-month period last year, Kreiser used the church's American Express credit card for personal expenses, including online gambling.
The total amount of the theft was over $25,000.
"This defendant, a Roman Catholic priest, embezzled thousands of dollars from St. Gregory's Church and improperly used the money for his personal benefit. His conduct represents a betrayal of the trust of his parishioners who looked to him for moral leadership and spiritual guidance," said Zugibe.
Kreiser is due to back in court on in May. If convicted, Kreiser faces a possible state prison sentence of up to seven years.