Authorities say at least 20 headstones were initially reported vandalized, but the final number is expected to be higher.
It was not clear if headstones were knocked over or defaced with swastikas, or both.
No arrests were immediately made.
Detectives from the 106th Precinct are in the preliminary stages of investigation. The cemetery's top official was contacted overseas and told of the vandalism.
The Mokom Sholom cemetery was founded in 1865 by Congregation Darech Amuno, which is now on Charles Street in Lower Manhattan.
It is currently being managed by David Jacobson, the head of United Hebrew Congregation.
