Deputy Chief Samuel DeMaio will be the acting police director.
He replaces McCarthy, who accepted the job as Chicago's acting police director this week.
Mayor Cory Booker hired McCarthy in 2006. He's been credited with making changes that helped drive down Newark's violent crime rate. But crime has increased gradually over the last two years.
Last summer, Newark had one of its most violent summers in decades. Booker said Tuesday that his primary focus is to make the city safer this summer and said all city agencies will be involved in a comprehensive plan.