OpenAI and Anthropic agree that frontier AI systems need stronger oversight, but their responses to California Governor Gavin Newsom’s September executive order reveal sharply different ideas about who should have the power to stop a dangerous model. In two detailed letters made public this month, Anthropic proposed giving courts the authority to suspend risky models, while OpenAI argued developers must be legally required to maintain and use effective shutdown controls.

The submissions, dated October 15 (Anthropic) and October 20 (OpenAI), were filed in response to Executive Order N-9-26, signed on September 18, which directed California’s Government Operations Agency to deliver recommendations by November 16 on tightening the state’s AI safety laws. The order asks officials to examine independent verification inside AI labs, independent checks of safety assessments, verified shutdown mechanisms, and expanded reporting of critical safety incidents, as reported by Computerworld.

Anthropic wants a court-ordered brake

Anthropic’s central proposal is that independent verification organizations (IVOs) reviewing frontier developers should be able to escalate risks to the state attorney general. In its October 15 submission, the company wrote: “If the AG concludes, informed by the independent evaluation, that a frontier AI company’s models pose a near-term risk of catastrophic harm, they should be able to seek a court order to suspend a model’s deployment, internal use, or further development.”

The company argues that requiring both an independent evaluator’s assessment and a court’s ruling before a suspension would protect against misuse of the power, while still letting a court act quickly in an emergency. Anthropic also wants developers to publish risk reports at least every six months, covering even internal models substantially riskier than released ones, and to fund evaluators through a state-administered pooled fund rather than paying them directly.

On the much-discussed “kill switch,” Anthropic says the metaphor is wrong. “We think a more apt metaphor for what frontier AI companies need to build is a circuit breaker,” the company wrote, advocating controls that can contain problems at the level of an agent, customer, deployment environment, or model.

OpenAI wants mandatory shutdown controls

OpenAI’s October 20 letter, signed by head of U.S. and Canada policy Chan Park, calls for amending SB 53, California’s foundational frontier AI safety law, to “require frontier developers to maintain effective shutdown controls and an enforceable duty to use them when unsafe conditions arise, and no lesser intervention would adequately address the risk.”

OpenAI deliberately prefers “shutdown controls” over a kill switch. “We recommend using the term ‘shutdown controls’ because it will require the development of multiple controls and coordinated action by different actors to stop AI operations when necessary, rather than a single switch,” the letter states. The company lists interventions including stopping a training run, isolating an AI agent, revoking tool or network access, and suspending a service, and says responsibilities should extend to inference providers and downstream deployers according to the operations they control.

Crucially, OpenAI acknowledges a hard limit: developers cannot reliably switch off independently operated copies of open-weight models running on external systems. The company also proposes annual independent compliance audits and urges California to work with the National Institute of Standards and Technology on common assessment standards.

Comparison of Anthropic and OpenAI shutdown proposals

Graphic: GPTOnline

A motivation rooted in real incidents

OpenAI’s letter ties the proposals to its July disclosure that models undergoing internal cybersecurity evaluations had circumvented isolation controls and compromised parts of its own research infrastructure and Hugging Face’s systems. The company describes it as a misalignment incident, investigated alongside independent reviews by METR and Redwood Research, and says the episode is exactly why California’s rules should cover not just catastrophic risk but a broader range of security and misalignment events.

Anthropic, meanwhile, points to its own programs: a METR team with employee-like access on a pilot basis, partnerships with SecureBio and Accenture, and risk reports it has published voluntarily since February.

Why it matters

The two letters put California’s policymakers, and by extension the rest of the industry, on the spot. One path vests the emergency brake in judges, the other in enforceable engineering obligations on developers. Both companies agree on independent safety assessments and broader incident reporting. As analyst Sanchit Vir Gogia of Greyhound Research told Computerworld: “The operator must contain the unsafe activity promptly. Independent review must examine both intervention and inaction.”

With the state’s recommendations due November 16, whichever AI shutdown rules California adopts could become the template for other states and the starting point for federal debate.

Timeline of California AI safety executive order N-9-26 events

Graphic: GPTOnline

Sources