Stop trusting the extinction math

Happy Wednesday! I really didn't want to add to the AI hype this week — I'm sure you've seen enough! But most reporting is missing something really important. I hope you'll read this one; it's something I think about every time the AI Doom Cycle takes off. And it helps.

💙 Amanda

Perhaps you've heard: AI could kill us all. We know this because the head of alignment science at Anthropic said so on X last week: "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."

Admittedly, this sounds alarming. He's an AI expert! Warning us the technology he helped to create could soon wipe us all out! And he's put a one in ten chance on that happening!

It's that 10% that I want to talk about. It sounds scientific — like he knows something we don't — and adds a sense of authority to an otherwise wild claim. Let's just focus on that.

That likelihood — the greater than 10% chance that AI will destroy us by 2030 — has a ridiculous name. It's called...

p(doom).

The probability of doom

It's a simple concept: p(doom) is the probability (p) that artificial intelligence will cause human extinction (doom). It's a common way for people in the AI industry to talk about x-risk, which is just insider jargon for 'existential risk'. In other words, they use p(doom) to indicate the likelihood that AI will get out of our control and somehow kill us all. (Think SkyNet or HAL 9000). It's a genuinely hot topic in the industry.

The values range from no risk at all (0) to we're done for (100).

Everyone who's anyone has a p(doom)

Name	Who?	P(doom)
Sam Altman	CEO of OpenAI	>0%
Dario Amodei	CEO of Anthropic	10-25%
Elon Musk	CEO of X, Tesla and SpaceX	10-30%
Dan Hendrycks	Director of Center for AI Safety	>80%
Eliezer Yudkowsky	Author of "If Anyone Builds It, Everyone Dies"	>95%

The concept has been around for a while. In 2023, the New York Times called it "the morbid new statistic that is sweeping Silicon Valley."

But here's the thing. The thing I have to remind myself every time I see one of these claims. This number is not a statistic. It's not based on data. Nothing has been measured or calculated. It's a hunch with a percent sign. That's all.

It's bullshit.

A meaningless prediction

There's no agreed-upon methodology for calculating p(doom). But even if there were, it wouldn't improve anything. Methodology doesn't matter if there's nothing to plug into the equations.

"When it comes to AI x-risk, forecasters aren’t drawing on any special knowledge, evidence, or models that make their hunches more credible than yours or ours or anyone else’s," write academics Arvind Narayanan and Sayash Kapoor, as they break down the flaws in these predictions.

Predictions about human extinction aren't like forecasting the weather. Tomorrow's chance of rain is grounded in physics and centuries of historical data. Or consider your odds of dying in a plane crash. Those are based on over 100 years of aviation safety data. AI-caused extinction has no equivalent baseline — it's never happened before (and if it had, we'd all be gone anyways).

Really, none of this is new. Religions have been trying to predict the end for thousands of years. So far, they've been wrong 100% of the time.

So, if these are just made-up numbers, then it makes sense that individual p(doom) values are all over the place.

In late 2022, the Forecasting Research Institute conducted one of the most detailed x-risk prediction exercises to date. They asked a group of 169 experts in AI and forecasting to predict threats to humanity.

Hunches from AI experts, superforecasters and the general public

Likelihood of AI-initiated extinction by 2100. (Note the pseudo-log axis, which allows for more space in the 1 to 10% range.)

In nearly every group surveyed, the chance of human extinction by 2100 ranged from 0% to about 75%. The chart above, based on a graph in the original paper, also shows each group's median value. The highest was general x-risk experts, at 4.75%. Surely combining expert predictions makes them more reliable? Nope. That's just turning a bunch of baseless numbers into a single baseless number. (This survey is from 2022, and nothing since has matched its scale. But running it again today wouldn't fix the actual problem. A fresher hunch is still just a hunch.)

When a well-known AI expert states a p(doom), the number isn't backed up by data — it's backed up by their reputation. We're being asked to trust the source, not the method.

But studies show that expert guesses aren't better than yours or mine. Even a paper that asked thousands of AI researchers about the technology's future had to admit: "Forecasting is difficult in general, and subject-matter experts have been observed to perform poorly... There are signs in this research and past surveys that these experts are not accurate forecasters across the range of questions we ask."

An existential distraction

When AI insiders cry "extinction!", it's a safe bet that someone's benefiting from the hype.

Focused on the possibility of our demise, we may worry less about the actual, real harms that AI is causing today. And who is profiting off those immediate harms. And who gets to regulate it all.

Real-world AI harms are difficult to track, but the AI Incident Database is doing its best. They have collected information on over 1,600 incidents, which they classify as "an alleged harm or near harm event to people, property, or the environment where an AI system is implicated." They track more harms all the time.

As AI spreads, harms overall are on the rise

Monthly counts of AI incidents

The data are very imprecise and there's no way they've captured them all — incidents are generally those that are already public and are submitted manually to the database — but they give us a general picture. And importantly, it's a picture of harms that are happening now. Today.

A self-driving Waymo struck a child in California. Scammers cheated an 86-year-old woman out of nearly $1 million using deepfakes of the Canadian prime minister. A man blackmailed a relative using AI-generated sexual images of her. Amazon's recommendation algorithm offered poison to people who used it to kill themselves.

Clearly AI is something we need to be concerned about. Not because it will end our civilization in the future, but because it's already taking jobs, interfering in politics and exacerbating the climate crisis.

Tech journalist Brian Merchant wrote last week: "AI has done these things not because they have all-powerful minds of their own but, ultimately, because people have programmed, calibrated, and directed them to. In other words, it's not AI we need to worry the most about. It's the people and corporations working to profit from and gain power by deploying it."

The real question isn't whether AI will end the world someday. It's who benefits from us worrying about that instead of what the industry is already doing today.

So the next time you see a headline-grabbing p(doom), remember what it actually is. It's not a measurement or a forecast. It's just a bullshit guess with a percent sign.

MAKE IT MORE LIKELY

If enough of you decide to become paying members, the probability that Not-Ship sticks around, providing you with these unique takes on our messy world, is high. It's just $9/month or $90/year.

WORTH YOUR TIME

If today's newsletter got you thinking, here are two excellent pieces that go deeper:

The politics and possibilities of ‘AI could kill us all’ by Brian Merchant, author of Blood in the Machine. He's a long-time tech journalist and I appreciate his anti-hype writing.

AI existential risk probabilities are too unreliable to inform policy by academics Arvind Narayanan and Sayash Kapoor. A bit more technical, but great for really understanding why p(doom) is dumb.

“Theories that involve the end of the world are not amenable to experimental verification — or at least, not more than once” — Carl Sagan

FROM ELSEWHERE

Here's what I found interesting, important or delightful this week:

Math and maps. I was delighted to discover that the author of Mapmatics: How We Navigate the World Through Numbers is a reader of Not-Ship. I'm only a chapter in, but I can already highly recommend it.

Cockroach milk. The Ignoble Prize awards research that "first make people laugh, then make them think." The winners devised a precise definition of kissing, stepped on snakes, studied the aerodynamics of nose blowing, and compared cow and cockroach milk.