Leaders of companies seeking to create smarter artificial intelligence—Google DeepMind, OpenAI, xAI, and Anthropic—are clearly aware of the stakes. Each of them warned from time to time that powerful AI could spell disaster for humanity. Yet, in an industry shrouded in trade secrets and non-disclosure agreements, potential whistleblowers are faced with a difficult choice: remain silent or come forward and face potential legal consequences.
A landmark California law is designed to make this choice easier. The AI Frontier Transparency Act (SB 53), signed into law by Gov. Gavin Newsom on September 29, offers legal protections that insiders have long demanded, protecting those who blow the whistle from retaliation from their employers.
The law applies to any company operating in California that uses massive amounts of computing power to train models. This threshold is intended to regulate only advanced developers and exempt smaller firms from the tax. It combines these new whistleblower protections with demands for transparency, requiring developers to publish risk mitigation plans and share safety test results, codifying promises that critics say were sometimes it broke. Companies must also report “critical safety incidents” to the California Office of Emergency Management (OES).
Read more: 60 UK lawmakers accuse Google of breaking AI safety promises
Why whistleblowers call for protection
While the “Godfather of Artificial Intelligence” Geoffrey Hinton left Google in 2023 to speak freely Regarding the existential risks of technology, calls for whistleblower protections only emerged in May 2024, when Vox reported OpenAI's use of “extremely restrictive blackout agreements” to suppress criticism. OpenAI employees received most of their compensation in the form of equity, as is common in Silicon Valley. But when leaving the company, in order to preserve their equity, they were reportedly required to sign an agreement that contained provisions permanently prohibiting them from criticizing their former employer, and it prevented them from mentioning the existence of the agreement. This came to light after one of the former employees, Daniel Kokotaylo, posted on an online forum his refusal to sign the agreement, believing that he had lost millions of dollars in assets belonging to him.
The next day, OpenAI CEO Sam Altman took it to X deny any knowledge of these provisions, stating: “This is my fault, and this is one of the few times I have been genuinely embarrassed while working with OpenAI; I didn’t know this was happening and I should have known.” OpenAI subsequently confirmed in communications sent to current and former employees that these provisions would no longer apply and that they would remove the offending language from all output documents going forward. But the credibility of Altman's denial was called into question when on May 22 leaked documents Apparently, he put his signature, as well as the signatures of other senior executives, on documents expressly authorizing these provisions. In June 2024, following controversy over then-terminated OpenAI shutdown agreements, Kokotailo and 13 current and former employees of OpenAI and Google DeepMind called for stronger protection of whistleblowers in an open letter.
What does the new law protect?
Fulfilling a key recommendation of the open letter, California's new law goes beyond typical whistleblower protections, which cover only illegal activities. It prohibits AI developers from enacting rules that prohibit risk assessment staff from reporting violations of the law or “catastrophic risks” (defined as threatening 50 lives or causing $1 billion in damages) to the Attorney General, regulators or their managers. Employees do not need to believe there is a catastrophic risk, but only have “reasonable grounds” to believe it to be covered by the new legislation.
California Whistleblower Protection Goes So Far
The new protections mark “an important step forward,” says Jacob Hilton, a former OpenAI researcher and one of the signatories of the open letter. Still, he says he would like to see protections enacted at the federal level, pointing to the bipartisan AI Whistleblower Protection Act introduced in Congress in May. Hilton notes that the “catastrophic risk” threshold means that lesser, but still serious, damages will not be covered. Another of the open letter signatories, speaking on condition of anonymity, told TIME that while addressing catastrophic risks is key, “there may be other areas of risk where we would like to have different protections.”
Going beyond simply uncovering illegal activity is critical, says Lawrence Lessig, a Harvard law professor who provided free legal advice to Kokotaillo and other signatories of the letter. Lessig worries, though, about the uncertainty about what's preventing employees from speaking out. He says he would prefer to see the standard lowered to “good faith,” which focuses on the employee's sincere belief rather than the strength of his evidence. That would be “a much simpler standard for a techie to rely on” than having him act as a lawyer or risk analyst, Lessig said. Under the new legislation, the lower standard of “good faith” is reserved exclusively for an internal, anonymous reporting channel within large artificial intelligence companies.
Lessig says the risks taken by past whistleblowers underscore the need for such protection. “When there are people inside who are willing to give $2 million for simple freedom of speech, we should strive to understand what they want to say,” he says.
