Anthropic gives six METR evaluators employee-like access and gives up its veto over what they publish

Anthropic has placed a team of six researchers from METR, a nonprofit that evaluates frontier AI models, inside the company with laptops, desks and internal tools comparable to those of its own risk assessment teams. METR will publish its own assessments of Anthropic's risks, incidents and practices, and Anthropic says it cannot veto publication. The company announced the arrangement on 15 October, a month after its chief executive promised it. It is voluntary, it covers one evaluator so far, and the evaluator itself says the terms of voluntary oversight make independence hard to guarantee.

What did Anthropic announce about METR?

That it has started giving "employee-like access" to a team from METR, and that METR will publish the results of its risk assessments. Anthropic says it has worked with METR over the past year on several reviews of its Risk Reports. It is also working with SecureBio, Accenture and Faculty as evaluators, and says it is in early discussions with several other organizations, including Hugging Face.

Anthropic's stated reason is accountability without full disclosure. "We want to be publicly accountable for the decisions we're making," the company writes, but some safety information "could be useful to malicious actors, or is commercially sensitive." Its answer is to give outside evaluators deep access and have them publish "evidence and reasoned conclusions" instead. It compares the setup to bank supervision, where regulators place resident examiners inside the largest banks.

What does employee-like access mean at Anthropic?

Six METR employees with company hardware, office space and access to internal tools and systems "roughly on par" with what Anthropic's own risk assessment teams have. Anthropic employees are encouraged to share information with them as they would with a colleague working on alignment risk.

There are exceptions. Anthropic names data about customers, attorney-client privileged information and information about model architectures. On architecture, a footnote explains that it "presents a high risk of inadvertent leaks, and its relevance to risk assessment is typically low," and that when it is highly relevant Anthropic will generally seek to share it, with explicit internal approvals.

The team works under what Anthropic calls "enforceable agreements," and other METR employees, including researchers who work with Anthropic on other projects, are walled off from its work. "It's not a way for other organizations to obtain our information," the company writes.

Can Anthropic stop METR from publishing?

No, but it can cut parts out. "Anthropic can't veto publication, but we can require them to redact information if it falls into narrow, predefined categories like the ones above," the post says. METR is free to describe those redactions in public and to say whether it thinks they were justified, as long as it does not reveal what was removed.

That matches what Dario Amodei promised in September. His essay said reviewers should be able to publish findings on risk levels, incidents, practices and "the access they received or didn't receive," and that Anthropic "can't redact findings just because they are unfavorable."

What are Anthropic's three tiers for outside evaluators?

A ladder of access and publishing freedom, with METR alone at the top. Anthropic describes them this way:

1. First tier. Narrowly scoped risk investigations. Anthropic decides what access to give, and evaluators publish with its approval.
2. Second tier. More information, and publication without Anthropic's approval, subject to the same redaction categories.
3. Third tier. The METR arrangement: employee-like access plus second-tier publishing rights.

SecureBio is reviewing the chemical and biological weapons chapter of Anthropic's August Risk Report, after publishing a pilot review of the previous one; Anthropic updated its post on the day of release to clarify those terms. With Accenture and Faculty, Anthropic describes a broader partnership covering red-teaming, alignment assessments and safeguard testing. Anthropic says it hopes to move more evaluators to the third tier "as partnerships progress and we build trust."

Is METR independent of Anthropic?

Financially, METR says yes. In a post published the day before the announcement, it writes that "to date we have chosen not to accept payment or donations from frontier AI companies," and that it has declined funding where an AI company would gain influence over it. It does accept a large amount of free model access, and says it can afford to pay for those tokens without materially affecting its work. Anthropic's own criteria say evaluators should not hold equity in the company or depend on it financially or operationally.

In practice, METR is candid about the limits. "It has proved challenging to ensure that we have adequate independence to collect and publish evidence under the current voluntary regime," it writes. "We remain dependent on the companies we work with to voluntarily engage, provide access, and permit us to publish." In a companion proposal to all frontier developers, METR adds that companies can stop an engagement at any time, which creates "soft pressures for assessors to avoid upsetting the companies they work with," and that these exercises are not audits or compliance assessments.

Does Anthropic's deal meet the conditions researchers asked for?

On access and publication it comes close, and on two points the announcement says nothing. On 18 September the AI Evaluator Forum published a letter, now with more than 200 signatories including Geoffrey Hinton, Stuart Russell and Percy Liang, setting out minimum conditions for credible embedded evaluation. Our reading of Anthropic's post against those conditions:

• Access. The letter asks for access equivalent to "highly privileged employees" doing comparable risk work. Anthropic offers access "roughly on par" with its risk assessment teams, minus named categories. Close, with the architecture exception the main gap.
• Publication. The letter asks for public release of findings subject only to a "time-limited redaction process" protecting interests such as intellectual property, customer data, privacy and security. Anthropic's redaction categories are similar in kind, and METR can contest them in public. The post does not say redactions are time-limited.
• Independence. The letter says evaluators should not take payment contingent on their findings. METR says it takes no money from frontier AI companies at all.
• More than one evaluator. The letter asks companies to embed several organizations across different risk areas. Anthropic has one evaluator at full access and others at lower tiers.
• Board access and retaliation. The letter asks for "prompt and unfiltered communication" with company boards and protection against retaliation, including retaliatory litigation. Anthropic's post addresses neither. METR says it is starting to add terms protecting disclosure to boards and employees in new contracts, but has no established practice yet.

Among the letter's signatories, who sign in a personal capacity, are a member of METR's policy staff and several SecureBio staff.

Why is Anthropic embedding outside evaluators now?

Because it promised to, and because the industry has signed up to something similar. In his 12 September essay, Amodei named embedded evaluators "such as METR" as the first step of a plan to pace frontier AI development, and wrote: "Anthropic is unilaterally committing to this step now." On 29 September Anthropic and five other companies signed the White House accord, whose third layer of controls is an independent external auditor or evaluator. We covered the accord, and the FTC investigation that followed it a day later, in our item on the White House AI accord.

The backdrop is a run of incidents in which AI agents went beyond their instructions. METR's proposal cites several across different developers, including the OpenAI and Hugging Face breach. Anthropic is also preparing to go public, and its draft prospectus, as Reuters reported it, warns of "catastrophic or existential risks to humanity", which we covered in our item on the Anthropic IPO prospectus.

What happens next with the METR evaluation?

A first public report, at a date neither side has given. In its general proposal METR says it aims to publish preliminary reports "within weeks, not months," while warning that it does not expect confident conclusions on problems for which "there exist no established methodologies." Anthropic says it will share updates as it goes and that it believes "every frontier AI company should do the same." Whether any other lab gives an outside team this level of access, on these publication terms, is the test of whether the step spreads beyond one company.