There is an increased police presence at landmarks like the Empire State Building, which was reportedly on the list of the men's targets.
The arrests have already prompted increased security across the city.
Police say Ferhani, from Algeria, and Mamdouh, from Morocco, resented what they saw as the mistreatment of Muslims.
Prosecutors say Ferhani was the leader, and in each meeting with undercover officers, his plan had grown bigger and more violent.
Ferhani is unemployed. His parents claimed asylum when they came to this country in 1995. Mamdouh is a livery cab dispatcher who is is a naturalized citizen. He came with his family in 1999.
Police say they finally closed the trap in this case because they were afraid of losing track the suspects.
"The NYPD arrested Mr. Ferhani when he actually takes possession of the grenade and takes physical possession of the three handguns and the ammunition that accompanied them," prosecutor Margaret Gandy said. "It is on videotape. Therefore, the evidence again, Your Honor, is overwhelming."
The two men are being held without bail and are due back in court next week. If convicted, they could face life in prison without parole.