The idea of artificial intelligence overthrowing humankind has been talked about for decades – and programs such as ChatGPT have only renewed these concerns.

So how likely is it that we'll be able to control high-level computer super-intellgence? Scientists back in 2021 crunched the numbers. The answer? Almost definitely not.

The catch is that controlling a super-intelligence far beyond human comprehension would require a simulation of that super-intelligence which we can analyse. But if we're unable to comprehend it, it's impossible to create such a simulation.

Rules such as 'cause no harm to humans' can't be set if we don't understand the kind of scenarios that an AI is going to come up with, suggest the authors of the paper. Once a computer system is working on a level above the scope of our programmers, we can no longer set limits.

"A super-intelligence poses a fundamentally different problem than those typically studied under the banner of 'robot ethics'," wrote the researchers back in 2021.

"This is because a superintelligence is multi-faceted, and therefore potentially capable of mobilising a diversity of resources in order to achieve objectives that are potentially incomprehensible to humans, let alone controllable."

Part of the team's reasoning comes from the halting problem put forward by Alan Turing in 1936. The problem centres on knowing whether or not a computer program will reach a conclusion and answer (so it halts), or simply loop forever trying to find one.

As Turing proved through some smart math, while we can know that for some specific programs, it's logically impossible to find a way that will allow us to know that for every potential program that could ever be written.

That brings us back to AI, which in a super-intelligent state could feasibly hold every possible computer program in its memory at once.

Any program written to stop AI harming humans and destroying the world, for example, may reach a conclusion (and halt) or not – it's mathematically impossible for us to be absolutely sure either way, which means it's not containable.

"In effect, this makes the containment algorithm unusable," said computer scientist Iyad Rahwan, from the Max-Planck Institute for Human Development in Germany.

The alternative to teaching AI some ethics and telling it not to destroy the world – something which no algorithm can be absolutely certain of doing, the researchers say – is to limit the capabilities of the super-intelligence. It could be cut off from parts of the internet or from certain networks, for example.

The 2021 study rejects this idea too, suggesting that it would limit the reach of the artificial intelligence – the argument goes that if we're not going to use it to solve problems beyond the scope of humans, then why create it at all?

If we are going to push ahead with artificial intelligence, we might not even know when a super-intelligence beyond our control arrives, such is its incomprehensibility. That means we need to start asking some serious questions about the directions we're going in.

In fact, earlier this year tech giants including Elon Musk and Apple co-founder Steve Wozniak signed an open letter asking humanity to pause work on artificial intelligence for at least 6 months so that its safety could be explored.

"AI systems with human-competitive intelligence can pose profound risks to society and humanity," said the open letter titled "Pause Giant AI Experiments".

"Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable," it said.

The research was published in the Journal of Artificial Intelligence Research in January 2021.

A version of this article was first published in January 2021. It has since been updated to reflect AI advances in 2023.