Anthropic is calling for a global “pause” on AI development, claiming that the technology is nearing a point where it can spiral out of human control. 

In a lengthy blog post published Thursday, the world’s most valuable AI startup made the case that its Claude family of models were on the path to achieving “recursive self-improvement,” or the ability to improve themselves on their own, a key hypothetical tipping point that could lead to the creation of powerful AIs capable of operating outside human interests and harming society.

We’re not at that point yet, Anthropic stresses, but it “could come sooner than most institutions are prepared for.”

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the company wrote in the post.

It added that “a meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions,” and admitted that this would be challenging to enforce.

“Training runs are far easier to conceal than missile silos,” it wrote.

For Anthropic to call for a pause now is convenient. In the past few months, it leap-frogged OpenAI to become the world’s most valuable AI company with a $1 trillion valuation, and its models are now generally viewed as the best in the field, especially at coding tasks. If the industry were to hit the brakes now, it would cement Anthropic’s dominance.

Not everyone was buying Anthropic’s claims. Prominent AI critic Gary Marcus called the company’s lengthy post a “bait and switch.”

“Anthropic is trying to strike terror into everyone’s hearts (‘full recursive self-improvement also might increase the risks of humans losing control over AI systems’) but all they have really shown is just faster coding — entirely under human control,” Marcus wrote on his Substack. “A faster coding tool will probably not end the world.”

Anthropic has long tried to paint itself as the ethical and deeply concerned adult in the room. A cornerstone of its mythology is that CEO Dario Amodei abstained from unleashing a revolutionary AI model back in 2022 because he was too concerned about safety, and let OpenAI get all the glory when it released ChatGPT months later instead. 

Two months ago, in a rehashed sequel to this foundational company lore, Anthropic announced a new model called Mythos — but made a show of not releasing to the public, claiming it was powerful enough to break into “every major operating system and every major web browser.” 

But its act is ringing hollower than ever. Earlier this year, Anthropic famously clashed with the Pentagon over concerns that its AI systems could be used in autonomous weaponry and in the mass surveillance of US citizens. Later, it emerged that Claude was being used to help select strike targets in Iran.

Amid its blowout with the military, Anthropic also dropped a safety pledge that was arguably the venture’s entire raison d’etre: to stop training an AI system if it couldn’t guarantee it had proper safety guardrails in place.

Further underscoring Anthropic’s hypocrisy, University College London professor Steven Murdoch cited recent reporting from the Financial Times revealing that Anthropic is helping the US National Security Agency use its Mythos model so it can wage cyberwarfare against potential enemies like China and Iran.

“Anthropic might give the impression of being warm and fuzzy, but their definition of AI safety is narrow,” Murdoch told The Guardian. “Supporting US authorities in the development of offensive capabilities has never been something they have spoken against.”

Regardless of whether Anthropic genuinely thinks it has a remotely realistic shot at pulling off a global pause — or if this is yet another ploy to boost its safety-minded image — it’s vowing to pursue further action.

“In the coming months, we will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation,” the company wrote. “We’ll publish what comes out of it.” 

More on AI: CEO Says There Will Be No Raises Because He Spent All the Money on AI