Multi-billion dollar space telescope programs aren’t only feats of aerospace engineering. They also feature “lies, damn lies, and statistics”. Or at least statistics. They definitely feature those, as does all good observational astronomy. The problem with statistics is, in order to get a clear definitive answer, you need lots of samples. And, to put it mildly, it’s hard to find lots of samples of planets with alien life on them. And even harder to prove that the signals we think are caused by alien life aren’t caused by some other non-biological process. Or at least that’s the theory underpinning a new paper available in pre-print on arXiv from David Kipping of Columbia University (and Cool Worlds YouTube fame).

The problem comes from what Donald Rumsfeld, a former US Secretary of Defense, famously called Unknown Unknowns. And whether it's dealing with weapons of mass destruction in Iraq or life on alien planets, unknown unknowns are some of the hardest concepts to deal with in statistics.

Specifically in the realm of astrobiology, they likely come in the form of “confounders” - or false positives. We have a bad history of reporting false positives of life on other planets - from Martian “canals” to phosphine in Venus’ atmosphere - that were later shown to be the consequence of abiotic processes. And there are still a myriad ways that we don’t know about that could cause a “smoking gun” that astrobiologists would take as a clear sign of life that would later be shown to be the aftereffects of a lightning storm in a methane cloud, or some other such statistically unlikely event.

Fraser talks about the limits of the Habitable World Observatory (HWO).

In Bayesian statistics, the kind used by most astronomers, when you don’t know the likelihood of something happening, you use what’s called a “diffuse prior”. Essentially, you tell the math - I have no idea how common life is, and I also have no idea how likely this signal is being generated by some process that I don’t understand is non-biological in nature. The problem, as Dr. Kipping shows in his paper, is that when you do that, the math gets quickly out of control.

In order to reach a Bayesian factor of 10 (meaning the evidence for life is 10 times stronger than the evidence for no life), the number of planets to be surveyed ranges from a mere 12,366 to a whopping 44 trillion. Keep in mind that these planets have to all have the same signature being analyzed - that’s how the statistics works. Also keep in mind that, as of the time of writing, we have only found around 6,200 confirmed exoplanets. In other words, we would need to double our total stock of confirmed exoplanets, and all of those exoplanets would have to have the exact same signal of potential biosignatures in order to meet the actual statistical requirements of definitively detecting life.

To put it bluntly - that’s not going to happen any time soon. And Dr. Kipping makes that point in the paper. Though he does offer a solution that sounds like it’s stolen straight from Silicon Valley’s playbook - A/B testing. To do this for exoplanet analysis, he suggests splitting a group of exoplanets with the same potentially interesting signal into two groups, but with a key feature - both groups have to have the same false positive rate. That would mean that, mathematically at least, that “unknown confounder” would cancel out, making the comparison between the two groups more direct at least.

Fraser discusses the possibility of finding certain molecules that stand out as biosignatures, and what other processes might create them.

While the math behind that is elegant, it faces a huge hurdle in reality - how do you find two groups of planets where “life” behaves differently but the unknown chemistry that could be driving the signals we’re interpreting as life behaves exactly the same across all planets.

To put it bluntly, the likelihood of that happening is almost as remote as us finding 44 trillion exoplanets in the next 25 years. So, it appears that the 25 exoplanets that the Habitable Worlds Observatory plans to survey when it is launched next year is only a drop in the statistical bucket of what data we would need to collect to prove life definitively exists on another planet.

But Dr. Kipping is generally optimistic in nature, and he believes that someone in the astrobiology community will come up with a clever framework that can solve this problem. As he writes in the paper, “the agnostic analysis of any near-term biosignature search will hinge upon this point.” He’s right - and until we’re able to develop such a framework, all of the work we do on actually observing other planets for clear biosignatures only serve as additional evidence rather than a clear discovery. That means it’s time for the statisticians to step up to the plate to potentially save astrobiology.

Learn More:

D. Kipping - The Catastrophic Consequences of Agnosticism for Life Searches and a Possible Workaround

UT - A Statistical Analysis of Exoplanet Habitability Turns Up One Great Candidate - And Significant Observational Bias

UT - The Habitability of Earth Tells Us the Likelihood of Finding Life Elsewhere

UT - Establishing a New Habitability Metric for Future Astrobiology Surveys