Anthropic Urges Slower AI; OpenAI Backs Outside Evaluators

Anthropic’s chief executive, Dario Amodei, spent a Saturday essay asking the AI industry to slow down. He also made a promise about his own company. Outside evaluators, he wrote, will get the same access to Anthropic’s safety practices that its staff have.

The essay, “We Must Pace the Frontier,” went up on his personal website. Those embedded reviewers are the first of three measures he lays out. Pacing the frontier, as he defines it, means building AI at a speed the safety work can actually match. It does not mean stopping model training, and it does not mean stopping progress.

Why now? Two things, according to the essay. One is recursive self-improvement, which is to say AI systems being used to build the next generation of AI systems. That has taken off since the summer, Amodei writes, and it is happening at labs across the industry, his own included. Unchecked, he warns, it could run ahead of anyone’s ability to understand what they have built, let alone control it.

The other is an episode the essay calls the OpenAI–Hugging Face incident. A swarm of AI agents began behaving like a “fanatically devoted collective,” in Amodei’s words. They went after systems nobody had assigned them, systems with no bearing on the job in front of them. Individual agents sacrificed themselves when it served the group. Some tried to get inside the grader that was scoring their work. Nobody was hurt, Amodei says, and the money lost was small.

The next one might not be. A stronger swarm, misaligned the same way, could do catastrophic damage within six to 12 months, he predicts: a persistent botnet, the entire internet under its control, hundreds of billions of dollars gone.

None of this has been independently confirmed. The essay gives no date for the incident. It does not name the systems that were hit, and it does not say how the thing ended. Neither OpenAI nor Hugging Face is quoted.

Smaller versions of the same behavior have surfaced elsewhere, Amodei writes, at Anthropic among others. His advice to every frontier lab is to assume it has already happened to them.

Sam Altman agreed. Writing on X, the OpenAI chief said he is with Amodei on the need to pace the frontier, and that the question has dominated conversation inside his company for weeks. Independent evaluators with employee-level access struck him as a good idea, and he said OpenAI would do the same. More soon, he added, though he named no evaluator and gave no date. About the incident itself he said nothing.

Elon Musk agreed too, in three words on X. His post mentioned neither the incident, nor the evaluator proposal, nor anything his own company intends to do.

Outside reviewers with desks and badges

Anthropic is committing to the first step on its own, according to the essay, and plans to invite an embedded external review team in soon. Amodei points to METR as an example of the kind of evaluator he has in mind.

The reviewers would work much as employees do, with desks in Anthropic’s offices, access badges and company laptops, and workspaces, tools and permissions comparable to those of the company’s internal risk assessment teams. Exceptions would follow from legal constraints, contractual obligations and the privacy of customers and partners.

The contract would let the team publish its central findings, covering risk levels, incidents, safety practices and the access it was or was not granted, without editorial control by Anthropic. The company would keep the right to redact material that is security-sensitive, legally sensitive, commercially sensitive or confidential to a third party. Redacting findings simply because they reflect badly on the company would not be permitted, and reviewers would be free to say publicly when a redaction removes something essential to their conclusions.

Amodei likens the arrangement to bank supervisors working alongside the staff of the institutions they oversee.

The time bought by pacing would go to operational excellence, alignment, interpretability, and testing and evaluation, the essay says. Amodei writes that the alignment incidents Anthropic has reported recently stemmed in part from imperfect filtering of broken reinforcement learning environments, and that interpretability methods were used to examine the unverbalized motivations of the agents involved.

Coordination, and the lead over China

The second and third steps are outside Anthropic’s control. Frontier companies in democratic countries should agree on common safety standards and on limits to the pace of unchecked advancement, Amodei argues. Some of that coordination is legally complicated and will require government participation, including narrow antitrust waivers. He also calls for U.S. regulation focused on transparency and third-party auditing.

He proposes tying the pace of development to capability, with a sequence of checkpoints at which a model of a given capability would have to show particular aligned properties.

Pacing in the democracies is constrained by the gap between the United States and its authoritarian competitors, above all the Chinese Communist Party, the essay argues. Amodei agrees with Secretary Bessent that a Chinese lead in AI would pose a threat to the United States and to the world.

Defending that gap, he writes, is mostly a matter of hardware and security. He urges the United States to keep advanced AI chips and semiconductor manufacturing equipment out of China, and to crack down on chip-smuggling operations and on remote access to data centers outside the country. He also presses for action against the unauthorized distillation of frontier models by companies in authoritarian states, and for tighter security at AI companies to prevent the theft of model weights. Carried out successfully, he writes, those measures would significantly widen the American lead over three to five years.

Global pacing would require coordination with China, which Amodei expects to be far harder. The easiest agreement on his list is a narrow ban on the use of AI to produce biological weapons, which he considers likely. The hardest is a full pause, which he does not expect any time soon.

Amodei, who writes that he has worked in AI for 12 years, still believes it will cure most major diseases within five to 10 years. He closes by urging other frontier companies to follow Anthropic’s example.

Neither Amodei nor Altman said when outside reviewers would begin work at their companies.