6 hours ago

They built it. They're scared of it. They're selling it anyway.

Stop me if you've heard this one before: a tech company says it's built a new AI that's so powerful it's scary. Apparently, it's too dangerous to release into the world – the consequences would be catastrophic. Luckily for us, they are keeping it locked up for now. They just wanted you to know.

That's exactly what AI company Anthropic is telling us about its latest model, Claude Mythos. The company says Mythos' ability to find cybersecurity bugs far surpasses human experts, and it could have world-altering consequences if similar technology lands in the wrong hands. "The fallout – for economies, public safety and national security – could be severe," Anthropic said in an early April blog post. Some breathless observers warned that Mythos will soon force you to replace every piece of technology in your life, down to your WiFi-enabled microwave, to protect from the digital madness.

Some security experts doubt these claims, but let's set that aside. This isn't new. Executives at leading AI providers regularly issue warnings about how their industry's products may destroy humanity. Why do AI companies want us to be afraid of them?

It's a strange way for any company to talk about its own work. You don't hear McDonald's announcing that it's created a burger so terrifyingly delicious that it would be unethical to grill it for the public.

Here's one theory. According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. Tech leaders say they're just warning us about an inevitable future, and safety is a top priority whether it's now or later. But others argue what we're actually seeing is fear mongering, which exaggerates the potential of the technology and serves to boost stock prices. And it encourages a narrative that regulators must stand aside, because these AI companies are the only ones who can stop the bad guys and build this technology responsibly.

"If you portray these technologies as somehow almost supernatural in their danger, it makes us feel like we are powerless, like we are outmatched," says Shannon Vallor, a professor of the ethics of data and artificial intelligence at the University of Edinburgh in the UK. "As if the only people we could possibly look to would be the companies themselves."

Somebody stop me

An Anthropic spokesperson told me the company has been clear about these issues. They shared blog posts from other organisations supporting Mythos' cyber capabilities, but said nothing to address the points in this article, aside from one comment I'll include below.

This isn't the first time Anthropic chief Dario Amodei has worked on a tool that's been declared too dangerous for the public by the company he worked for. In 2019, when Amodei was an executive at OpenAI, the company announced GPT-2. He and other company leaders said they just couldn't release GPT-2 because of "concerns about malicious applications of the technology". This was a tool far less sophisticated than ChatGPT. And months later, they released it anyway. (OpenAI CEO Sam Altman published a blog post which says the company embraces uncertainty, acknowledging that fears about GPT-2 were "misplaced".)

Altman criticised Anthropic's "fear-based marketing" in a recent podcast interview. But his own "I've created a monster" playbook goes back years.

"AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," Altman said in 2015. Years later, Altman claimed he loses sleep wondering if he's "done something really bad by launching ChatGPT". If only someone warned him.

Hundreds of tech leaders including Altman, Amodei, Bill Gates and Demis Hassabis, chief executive of Google DeepMind, endorsed a short statement in 2023 that said: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." The same year, moguls including Elon Musk signed a letter calling for a six-month pause on building advanced AI. Musk announced his new artificial intelligence company xAI less than six months later.

"It's just part of this pattern of unsubstantiated claims of power," says Emily M Bender, a professor of computational linguistics and natural language processing at the University of Washington, and co-author of the book The AI Con. This isn't limited to OpenAI and Anthropic, she says, it's the standard posture for the whole AI industry. "They're saying 'look over here', never mind the environmental destruction and the labour exploitation and all these systems we're destroying in society. We just have to worry about making sure this thing doesn't turn into the bad one that destroys humanity."

I asked OpenAI about all this. A spokesperson shared a recent blog post from Altman where he wrote that OpenAI would "resist the potential of this technology to consolidate power in the hands of the few", adding that "key decisions about AI are made via democratic processes and with egalitarian principles, and not just made by AI labs".

Is Mythos really that bad?

Anthropic says its new model already found thousands of "high-severity" vulnerabilities across the tech landscape at a level that surpasses human experts. It also announced a new partnership with more than 40 companies and groups in an "urgent attempt" to patch vulnerabilities before hackers get a chance. A company spokesperson told me Anthropic has been intentional about partnering with organisations to patch the foundational computing systems which represent the "lion's share" of the problem.

But there are significant doubts about those claims, and Heidy Khlaaf, chief AI scientist at the AI Now Institute, wasn't impressed. She's spent her career building and auditing the exact kinds of code analysis tools that Anthropic suggests it's surpassed. She's also worked on digital safety in nuclear facilities.

Khlaaf says the biggest red flag was the lack of false positive rates – an industry-standard measure of how often a security tool flags something that isn't a real problem. "This is not some unknown metric," Khlaaf says. "This is kind of the largest indicator of how useful your tool is." Anthropic didn't mention it and sidestepped the question when I asked for comment. Nor did Anthropic measure Mythos against existing tools that security engineers have relied on for decades.

There have also been some claims that Anthropic may have held back a wide release of Mythos because it couldn't afford the necessary computing power. Anthropic didn't respond when I asked about that, either.

None of this is to say that the threat is imaginary. "Mythos might be capable," Khlaaf says. AI tools are genuinely well-suited for scanning massive code bases, and automatically finding security vulnerabilities is a real and pressing danger. But Khlaaf is sceptical about Anthropic's claims without being able to substantiate them. "I think there are a lot of cracks in this narrative that Mythos is all powerful, we can't release it."

The Interface

Why so serious?

Preventing the end of the world is why OpenAI and Anthropic say they exist in the first place. OpenAI was founded as a non-profit, promising to build AI in a way that's safe before supposedly less responsible tech giants like Google and Meta got there first. Later, a splinter group left OpenAI to form Anthropic because they said their old employer wasn't dedicated enough to safety. Now, both organisations are working to become publicly traded companies and sell shares on the stock market.

"If you want to understand how an organisation, particularly a corporation, is going to behave, look at what its incentives are," says Vallor.

Google dropped its red lines around building AI weapons. OpenAI fought a legal battle to shed its non-profit status. Anthropic abandoned its flagship policy to never train an AI model if the company couldn't guarantee adequate safety measures.

"I would not count on [any of these companies] to walk away from the opportunity to dominate the market in order to remain the good guy," says Vallor.

AI companies say they take these issues seriously. OpenAI sent me links to its stance on mental health, AI accuracy, fraud and scams, and Altman says the company is committed to addressing the problems at every stage of AI development.

But there's a reason these companies only sound the alarm for the apocalypse, says Vallor. If AI might destroy society, these other problems seem a lot less significant. "The strategy has worked," she says. "Talking about their products as if they could end the world has not hurt these companies. It has not limited their power. If anything, it makes people feel like the only ones they could possibly look to for protection are the companies themselves."

Demons or messiahs

In almost the same breath, some of the people who warn of destruction also promise salvation. In a 2024 essay, Altman predicts "astounding triumphs – fixing the climate, establishing a space colony and the discovery of all of physics – will eventually become commonplace". Amodei promised "a country of geniuses in a datacenter".

Utopia and apocalypse are just two sides of the same coin, according to Vallor. "In either case, the scale is far too grand and mythic for things like regulation, or governance or court law to feel like you can get purchase on it," she says. "It leads people to believe that all they can do is sit back and wait to find out whether these technologies turn out to be civilisation-ending demons, or utopia-gifting messiahs." Even the name "Mythos" seems designed to inspire religious awe.

More like this:

But these aren't gods, they're products built by companies, for profit, says Vallor. And we've regulated things far more threatening than chatbots. "Every technology, save this one, even nuclear, even biological weapons, in no other case have we allowed these narratives to make us think these are forces beyond human control," she says. "Nothing about them is ungovernable. Unless we choose not to govern them."

Let's be clear: it is theoretically possible that AI will take over the world. I'm no fortune teller. But ask yourself, does that idea sound similar to other stories you've heard out of Silicon Valley in the past?

Weren't we all supposed to be living in Mark Zuckerberg's Metaverse by now? Wasn't Bitcoin going to replace all the world's currency? Remember back in the 2010s, when we heard about how social media would save democracy? All of these things could still happen. Or maybe they won't.

--

For timely, trusted tech news from global correspondents to your inbox, sign up to the Tech Decoded newsletter, while The Essential List delivers a handpicked selection of features and insights twice a week.

For more science, technology, environment and health stories from the BBC, follow us on Facebook and Instagram.