The 75-foot Conception was anchored about 20 yards off Santa Cruz Island when the scuba diving vessel caught fire about 3:15 a.m., said Mike Eliason, a spokesman for the Santa Barbara County Fire Department.
Brown says the recovered remains include 11 females and nine males, and DNA will be used to identify them.
Divers have seen between four and six others in the sunken wreckage, the sheriff said. Authorities are trying to stabilize the boat that sank in about 60 feet of water so divers can recover those remains.
Five members of one family perished. Susana Rosas posted on social media that her three daughters, their father and stepmother were on board.
Kristy Finstad, 41, leader of the scuba tour and co-owner of Worldwide Diving Adventures, which chartered the boat, was identified in a Facebook post by her brother, Brett Harmeling of Houston.
Pacific Collegiate School in Santa Cruz said some of its students and their parents were on the boat.
Thirty-nine people including six crew members were aboard the vessel when it caught fire. Five crew members jumped in the ocean and were rescued.
The Conception sank in 64 feet of water as firefighters were battling the flames, leaving the bow protruding from the water, officials said.
The identities of the victims who were found dead were not immediately released.
Brown said it may take some time to identify the deceased because of the condition of the bodies and because they don't have a full manifest of everyone who was on board.
He also said the situation - a boat fire in the middle of the night, with people trapped at the bottom of a ship - is one of the most difficult operations for rescue crews.
"This is probably the worst-case scenario you could possibly have," Brown said. "Fire is the scourge of any ship."
The boat trip was touted as a Labor Day weekend dive trip that started on Aug. 31 and was scheduled to return Sept. 3.
Passengers who were in the bottom of the ship were likely sound asleep at the time of the fire, Brown said. The boat was in a remote location and there was limited firefighting capability.
"You couldn't ask for a worse situation," Brown said.
At a press conference Monday in Oxnard, a Coast Guard spokeswoman said the five crew members were awake and on the bridge of the boat when the flames broke out.
The crew members jumped off and were able to escape on a dingy and were pulled from the water after banging on Bob and Shirley Hansen's boat. The Hansens were told one of the crew members had a girlfriend on board the engulfed boat.
They were also told by the crew that there were three birthday parties on board the boat, including one of a 17-year-old.
The operation was supported by helicopters, small boats and a patrol cutter.
Family members looking for information about their loved ones are asked to call (833) 688-5551. A family assistance center is being established at Earl Warren Showgrounds, 3400 Calle Real, Santa Barbara. For information, call (805) 696-1188.
The cause of the fire was unknown. Officials said it appears to have been an accident and there are no indications of an intentional act, but the official investigation will take time.
The Associated Press contributed to this story.
* Follow us on YouTube
* More local news
* Send us a news tip
* Download the abc7NY app for breaking news alerts