Dr. Richard Besser, Dr. Julie Gerberding, Dr. Tom Frieden, Dr. Jeffrey Koplan and Dr. David Satcher all came together to speak with ABC News' Linsey Davis on Thursday.
"This is the first public health response where the ground rules weren't set up that we would be driven by the best available public health science," said Besser, who served as acting director of the CDC in 2009. "So, when you have political leaders and public health leaders coming at this with very different messaging, (and) when you don't see the political leadership supporting public health science, you lose trust."
The doctors all agreed that there are things individuals can do to save lives, but it requires all Americans to work together.
SEE ALSO: Some of Trump's loyal supporters cling to COVID-19 conspiracy theories
"Right now, one person a minute is getting killed by this virus in the U.S., and we need to focus on what the virus is doing and what we as a community are doing to stop it," said Frieden, who was the CDC director from 2009 to 2017. "Because all of us can do things to make it less severe, whether that's wearing a mask, washing our hands, watching our distance or supporting public health so that they can box in virus... and protect the most vulnerable."
"This pandemic is not something that's going to end abruptly," said Satcher, who was the CDC director from 1993 to 1998. "The problem is going to be with us for a while in terms of COVID, so the question becomes: When are we going to get on top of it? When people take seriously the challenge."
President Donald Trump has time and time again downplayed the threat of the virus, claimed the virus was easily treatable and "under control," and questioned the sound scientific advice of his own health officials.
SEE ALSO: Two-thirds of Americans disapprove of Trump's coronavirus, race relations response, ABC News poll shows
"Every one of those falsehoods drives away our opportunity to improve our mitigation efforts... and it causes confusion in everyone's mind," said Koplan, who directed the CDC from 1998 to 2002.
Four of the CDC directors penned a Washington Post op-ed in June echoing many of these things they see as ongoing problems with the federal government's response to COVID-19.
The former directors also said the U.S. needs to take a more proactive stance in preventing the next pandemic.
"If you look at the tenure of the CDC directors that are participating in this conversation, each one of us has presided over some kind of important public health outbreak, whether it was AIDS or anthrax or SARS or Avian influenza or Ebola. We've all gone through the cycle of the crisis," Gerberding said.
"We have to be proactive," said Frieden. "We have to build in peacetime so that we're more prepared when we're fighting a war against a virus or other microbe."
WATCH: Trump changes tone, now admitting some schools may need to delay opening due to pandemic
Nevertheless, each of the doctors expressed optimism that the U.S. will eventually overcome the pandemic.
"We will recover, but the question is: How long will it take?" Gerberding said. "While we're maintaining our physical distance, we need to come together as communities, as health leaders, as nations around the world to really say we've got to work collaboratively. That's a tall order in this very divided world, but it's about the pandemic. It's not about politics."