He laid some of the blame for delays with the Port Authority of New York and New Jersey, which controls the site.
Two skyscrapers, a memorial and transit hub are already under construction at ground zero. But the building of several planned towers has been stalled because of financing battles between the Port Authority and site developer Larry Silverstein.
Bloomberg says the agency should either build now "or just get out of the way."
There was no immediate response from the Port Authority.