Robert Lang scored 49 seconds into the third, and Matt D'Agostini scored with 49.5 seconds remaining to help the Canadiens improve to 3-0 on their franchise-record seven-game homestand.
Georges Laraque and Andrei Markov each had two assists for Montreal.
Markus Naslund and Nigel Dawes scored in the second for New York, which beat Pittsburgh 3-2 in a shootout Wednesday night for its fourth win in five games.
The Canadiens, founded on Dec. 9, 1909, as an inaugural member of the National Hockey Association, unveiled bronze statues of Howie Morenz, Maurice "Rocket" Richard, Jean Beliveau and Guy Lafleur outside the Bell Centre earlier in the day.
As part of the team's centennial year celebrations, Montreal wore throwback uniforms in the style of the franchise's first Stanley Cup champions in 1915-16, one year prior to the NHL's formation.
Price, who stopped 18 shots, wore brown goalie pads and gloves as part of the promotion, the first of four variations of retro jerseys from the Canadiens' first decade that the team will wear for certain games over the course of the next 12 months.
Kostitsyn opened the scoring 9:52 in with his third goal in three games.
Begin put Montreal up 2-0 at 10:37 when he scored for the second time in three games. He tipped Lapierre's pass beyond Henrik Lundqvist from the edge of the crease.
Tanguay gave the Canadiens a 3-0 lead before the end of the period with his team-leading ninth goal at 17:37. Lapierre made it a four-goal lead with his fourth goal 3:35 into the second, before fighting with Petr Prucha midway through the period.
Naslund drew the Rangers to 4-1 on a power play at 13:02 when he deflected Paul Mara's shot past Price for his 10th goal.
Dawes got his third goal at 18:43 as New York closed to 4-2.
Notes: Rangers greats Ron Greschner, Harry Howell, Brad Park, Rod Gilbert and Andy Bathgate were introduced in a pre-game ceremony celebrating the "Original Six" rivalry between the two teams. Lafleur, Henri Richard, Dickie Moore, Serge Savard, Bobby Rousseau and Rejean Houle represented the Canadiens. ... Montreal's retro jerseys featured a red crest of a "C" with an "A" in the middle, for "Club Athletique Canadien," on the front of the blue band surrounding the middle of the red jersey. The Canadiens adopted their famous "CH" logo in 1916-17, the final year of the NHA, the NHL's predecessor.
