California Governor Gavin Newsom on Sunday vetoed a bill aimed at preventing large AI systems from causing catastrophic harms, saying the legislation would have created a “false sense of security.”

His decision came after weeks of deliberation and competing lobbying efforts from big tech firms, celebrities, billionaires, and the workers who build AI.

The law, Senate Bill 1047, introduced by state senator Scott Weiner (D-San Francisco) back in May, would have required companies that spend more than $100 million on computing resources to create a foundation AI model, or $10 million on computing resources to fine-tune a foundation model, to perform safety tests, hire independent auditors to review the model annually and take “reasonable care” to ensure the model doesn’t cause mass casualty incidents, more than $500 million in damage to physical or cyberinfrastructure, or act without human oversight to commit comparably serious crimes.

It also instructed developers to build a kill switch into qualifying models that would allow them to be immediately shut off and empowered the state’s attorney general to sue a developer for violating the act and, in the most serious cases of harm, seek damages up to 10 percent of the cost of training the model.

Newsom in recent weeks has signed a series of bills into law that address immediate, ongoing harms caused by AI systems, including bills that criminalize the creation of non-consensual deepfaked sexual imagery and require generative AI models to watermark their content so it’s easier to identify. But SB 1047, which would have applied only to the wealthiest and most influential AI companies, has become the focal point of debate over AI regulation in recent months.

Companies including Meta, Google, OpenAI, and Anthropic vehemently opposed the bill, saying it would undermine innovation and hurt small businesses, despite the rules applying only to corporations with hundreds of millions to spend on training AI systems, Meta, for example, said the bill would unfairly punish the developers of foundation models for disasters caused by downstream users and de-incentivize the creation of open-source models because developers fear being held responsible for how others use their products.

In his veto statement, Newsom pointed out that 32 of the world’s 50 largest AI companies are based in California and he echoed the industry’s complaints that the legislation would harm innovation.

“Adaptability is critical as we race to regulate a technology still in its infancy,” Newsom wrote. “This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions – so long as a large system deploys it.”

Despite the corporate lobbying, many in the AI industry believed the legislation was necessary. Dozens of AI researchers at leading AI companies called on Newsom in an open letter to sign the bill into law.

“We believe that the most powerful AI models may soon pose severe risks, such as expanded access to biological weapons and cyberattacks on critical infrastructure,” they wrote. “It is feasible and appropriate for frontier AI companies to test whether the most powerful AI models can cause severe harms, and for these companies to implement reasonable safeguards against such risks.”

The tech workers were joined in their advocacy by some of the biggest names in Hollywood—from JJ Abrams and Ava DuVernay to Mark Hamil and Whoopi Goldberg—who penned their own open letter in support of SB 1047. Elon Musk also called for its enactment while Rep. Nancy Pelosi and other influential members of California’s congressional delegation told Newsom to veto the law, calling it “well intentioned but ill-informed.”

Following Newsom’s veto, Weiner, the bill’s sponsor, said the decision was a “setback for everyone who believes in oversight of massive corporations”

“The governor’s veto message lists a range of criticisms of SB 1047: that the bill doesn’t go far enough, yet goes too far; that the risks are urgent but we must move with caution,” Wiener said in a statement. “SB 1047 was crafted by some of the leading AI minds on the planet, and any implication that it is not based in empirical evidence is patently absurd.”

Correction: An earlier version of this article incorrectly stated the Electronic Frontier Foundation’s position on SB 1047.