Appearing on "PFT Live" on Tuesday morning, Brady said players have to make their "own choice" about whether to make the trip to Washington, D.C.
"There's certain years, like a couple years ago, I wanted to go and didn't get the opportunity based on the schedule," Brady told "PFT Live." "We didn't get told until I think like 10 days before we were going, and at that point I had something I'd been planning for months and couldn't get there."
Alan Branch, LeGarrette Blount, Martellus Bennett, Devin McCourty, Dont'a Hightower and Chris Long have said they will not go to the White House, with some of them saying their decision was made in part as a protest of President Donald Trump.
Brady, coach Bill Belichick and owner Robert Kraft have personal friendships with Trump, but that has not deterred other Patriots players from making a political stand.
The Patriots quarterback visited the White House after Super Bowl victories in the 2001, 2003 and 2004 seasons -- when George W. Bush was president -- as well as after Michigan's college football national title in 1997.
"It really is a great experience," Brady told "PFT Live." "Putting politics aside, it never was a political thing. At least, it never was to me. It meant you won a championship and you got to experience something cool with your team, with your teammates. Everyone has their own choice. It's an offseason. These days are valuable for everybody. You only get so much time with your family and friends, and if people don't want to go they don't want to go, and that's their choice."
Kraft told the "Today" show on Monday that he didn't think it was notable that some players wouldn't go to the White House.
"This is America; we're all free to do whatever's best for us, and we're just privileged to be in a position to be going," Kraft said.