OMAHA, Neb. — Warren Buffett said Elon Musk's Tesla will probably struggle as it ventures into the insurance business, a field in which Berkshire Hathaway thrives in.

"It's not an easy business," Buffett told shareholders at the Berkshire Hathaway annual meeting. "The success of the auto companies getting into the insurance business is probably as likely as the success of the insurance companies getting into the auto business."

Musk told analysts in an earnings conference call last month that Tesla would be launching its own insurance product later in May. He said that Tesla has "direct knowledge" of a person's risk profile "based on the car," which gives it an "information arbitrage opportunity."

Buffett is not impressed, however.

"I'd bet against any company in the auto business" getting into insurance, he said. "I worry much more about Progressive."

About one third of Berkshire Hathaway's business is in the insurance space and that includes Geico. Before the annual meeting on Saturday, Buffett said Geico's value has risen more than $50 billion.

Subscribe to CNBC on YouTube.