Researchers Welcome Embedded Safety Evaluators, Then Ask the Obvious Question

The embedded-evaluator commitments from Anthropic’s pacing essay and OpenAI’s matching pledge have drawn their first serious scrutiny, and the verdict from the safety community is a conditional yes. Per TechCrunch’s coverage, researchers welcome the unprecedented access, with several calling it a genuine step forward, while the same coverage asks the question that decides everything: will the evaluators really be independent?

What Was Actually Promised, and What Was Not

The commitments are specific about access and silent about authority. Anthropic has committed to permanent, employee-level access for outside evaluators and publication of findings without editorial control, with only narrow redaction rights. OpenAI matched with a pledge of employee-like access for independent evaluators. What neither company has specified is whether evaluators can choose their own scope, whether findings require pre-approval before publication, how disagreements between evaluator and lab get resolved, and whether the arrangement survives a change of leadership. The Frontier Act’s version answers some of this by statute (outside audits, incident reporting, an Under Secretary of Commerce for AI Security), which is why the researchers’ welcome came with a call for lawmakers to enshrine the commitments quickly.

The Precedent That Should Worry Everyone

The week’s reporting provides the test case for why independence language matters. When OpenAI limited METR’s investigation to a single week of the Hugging Face breach, the restriction was defensible operationally and corrosive procedurally: an outside evaluator with scoped access cannot contradict the company that scopes it. Embedded evaluators with contractual publication rights are the structural fix, because their leverage comes from the contract rather than the relationship. The details to read in the coming contracts are three: who selects the evaluators, whether redaction is limited to enumerated categories, and whether evaluators can unilaterally escalate to regulators when redaction disputes deadlock.

For teams building AI products, the debate is a preview of their own audits: enterprise customers are already asking for the same embedded-evaluator guarantees for deployed agents, and the contract language hashed out between METR and the frontier labs will become the template vendors get handed.