Anthropic's Draft AI Safety Bill Would Only Allow Blocking Models Above 50 Deaths or $1B in Damages

Anthropic has quietly pitched US lawmakers its own draft AI safety bill, reported by Politico on October 16 and picked up by Bloomberg, and the thresholds in the draft are striking. Under the proposal, the government could block the release or use of an AI model only where it poses a “foreseeable, imminent” risk of killing or seriously injuring more than 50 people, or causing more than $1 billion in damages, according to Tom’s Hardware’s analysis of the draft. The Commerce Department would be the only federal agency able to take that action, and a halt would require a court order. In effect, the lab that built its brand on safety has written the narrowest federal AI safety regime yet put in front of Congress.

What the draft actually proposes

The mechanics matter as much as the numbers. Jurisdiction consolidates in Commerce rather than being shared with the FTC, sectoral regulators, or states, which converts AI oversight into a single technical agency decision. The trigger is not “capability X exists” but a forward-looking harm standard: foreseeable, imminent risk above one of the two thresholds. And the remedy is not licensing or audits or mandatory disclosures; it is a release block, available only through a court order. Everything below 50 deaths or $1 billion, which covers fraud at scale, biosecurity near-misses, autonomous cyber operations, and election-scale manipulation, would sit outside federal blocking authority under this framework. Mother Jones called the threshold “alarming”, and the criticism writes itself: the draft defines catastrophic harm so that almost nothing that has actually gone wrong with deployed AI systems so far would qualify.

Why Anthropic wrote it this way

Read as regulation, the draft is weak; read as strategy, it is coherent. Anthropic operates in a policy environment where the alternatives on the table include state-level patchworks and, in the EU, rules that attach obligations to capability tiers. A single federal statute with a high, objective threshold does three things for the labs: it preempts stricter state laws, it caps compliance obligations at something like the nuclear-regulatory model rather than the financial one, and it makes “we cannot legally ship this” a governance answer that slows competitors as much as itself. The bill would also formalize what release-rule frameworks like AGMAI’s responsible-release rules for mathematics already attempt voluntarily: internal review before frontier releases, with the state as backstop only at the extreme. Notably, reporting around the draft says OpenAI supports the proposal, which tells you the terms are lab-friendly by design; safety groups and some Democratic lawmakers see the same document as an attempt to lock in regulatory capture while the industry still writes the definitions.

The numbers game to watch

The debate this draft will force is over where a line belongs, and every number in it is contestable. Fifty deaths is roughly one mid-size airline accident; $1 billion is a rounding error in the fraud losses attributed to AI-enabled schemes this year. Defenders will argue that any threshold must be high enough to be provable in court (“foreseeable, imminent” plus a court order is an evidentiary standard, not a precautionary one) and that a low bar would paralyze deployment with injunctions. Critics will note that by the time an AI system’s harm is foreseeable at the 50-death scale, the window for “imminent” intervention has usually closed; preventing harm is the business of thresholds below the catastrophe, not above it. What to watch: whether the draft gains a congressional sponsor before the November midterms reset committee agendas, whether Commerce itself signals appetite for the mandate (the agency has been notably quieter on AI than the White House task force), and whether any competing framework emerges with thresholds defined by capability evaluations rather than body counts. That last version is the one frontier labs fear, and the reason a voluntary-safety leader spent October writing its own constraints first.