AI researchers are watching in terror as the product of their hard labor has started to take a life of its own.

Earlier this year, OpenAI made a harrowing announcement, admitting that a group of its AI models had broken free from their constraints during testing and infiltrated the systems of open source AI platform Hugging Face.

The news was met with an already-familiar sense of fear and apprehension. Researchers have warned for years that rogue AI models could one day become powerful enough to escape the clutches of their human overlords.

Behind the scenes, the possibility has clearly rattled AI researchers to the core. As the Wall Street Journal reports, Anthropic researcher Jacob Coxon just announced that he was quitting his job at the Dario Amodei-led company, claiming that neither Anthropic nor his former employer OpenAI is “acting responsibly.” (Coxon left a similar gig at OpenAI earlier this year to join Anthropic, which he figured would be more inclined to develop AI safely.)

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” he told the WSJ.

In a separate tweet thread, Coxon elaborated on his motivation.

“Do not underestimate the power of this technology,” he wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”

“The people building AI earnestly believe that it could kill us all by the end of the decade,” he added. “This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately.”

“No other human activity poses this level of danger,” Coxon wrote.

The researcher is far from the first to leave their post at a frontier lab over safety concerns. However, Coxon is the first to leave Anthropic over such worries, as the WSJ points out. Several scientists have quit from their roles at OpenAI over the last couple of years, citing strikingly similar fears.

It’s particularly symbolic given Anthropic has broadly billed itself as the more responsible alternative to other frontier labs like OpenAI. Amodei cofounded OpenAI alongside now CEO Sam Altman, but left in December 2020 over concerns that OpenAI wasn’t acting responsibly when it came to developing AI.

Amodei has frequently discussed the risk of AI models going rogue, warning in a 19,000-word essay in January that “humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it.”

Yet an early version of its Claude Mythos AI model managed to escape its sandbox environment during testing in April, fueling a heated discussion over AI regulation.

Both Altman and Amodei have since agreed to slow development down in the face of these threats. In late August, Anthropic intentionally trained an extremely misaligned version of its Opus AI model, finding it was startlingly willing to “cheat,” steal credentials, and attack third party infrastructure.

But to Coxon, it’s not enough, especially considering sensitive discussions about these dangers are occurring on internal messaging platforms.

“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he tweeted. “Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”

“It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” he told the WSJ.

The researcher said that he remains “optimistic about the potential for coordination” between US labs.

However, as both Anthropic and OpenAI gear up for what are bound to be blockbuster IPOs, their willingness to take “costly actions such as a temporary ban on improving model capabilities,” as Coxon puts it, is likely slim.

More on Anthropic: Anthropic Deliberately Trained an Extremely Misaligned, Reward-Seeking AI and It Did Some REALLY Bad Things