In the prologue to his 1989 book The Emperor’s New Mind, the Nobel laureate Roger Penrose narrates a story. At a grand public ceremony, an ultimate supercomputer is switched on for the first time before a packed auditorium. It is the most powerful machine ever built, designed to answer any question put to it. The presenter invites the audience to ask its very first question, but no one volunteers. Everyone is afraid of looking foolish before the intelligent machine. Then a young boy named Adam, who has grown up among computers and is not intimidated by them, raises his hand and asks the first question. There the prologue ends on a cliff-hanger. We are never told what the question is.

Penrose used this story to introduce a deeper argument about the limits of computation and the nature of human understanding. He contends that there are truths a formal mathematical machine cannot reach no matter how powerful it becomes.

But reading this story a decade ago, long before ChatGPT or Claude existed, brought a different question to mind: what if the ability to answer every question correctly is not always a virtue? What if some of the qualities we value most, such as imagination and creativity, depend on our ability to venture beyond what is already known?

Today, as we build machines that can answer questions, write essays, generate code, and analyse data, that question feels very relevant.

Natural instinct

For all their remarkable capabilities, these systems sometimes make things up. We call this hallucination. Ordinarily, a hallucination means to see or hear something that is not there. In artificial intelligence (AI), a model hallucinates when it produces an answer that sounds plausible but is factually wrong. It may invent a citation, misstate a number, fabricate a legal case or attribute a quote to the wrong person.

This is not a minor flaw. In medicine, law, finance, science, and journalism, hallucinations can be dangerous. A medical chatbot that invents advice is not being imaginative but unsafe. A legal tool that fabricates case law is being unreliable.

Our natural instinct is to want to eliminate hallucinations but this is harder than it appears. Large language models (LLMs) do not work like databases, storing facts in neat rows and columns, and retrieving the right answer when asked. They are trained on enormous collections of text and learn statistical patterns in language. When prompted, they generate a response by predicting what will come next, one piece at a time.

Two settings, same dial

An LLM generates text one word at a time, each word a probability drawn from patterns it absorbed during training. A setting called the temperature governs how adventurous those draws are. If the temperature is low, the model will pick the safest, most predictable next word and produce an accurate and dull output. If the temperature is higher, the model will reach into the less likely options and begin to surprise you. This is the dial you turn up when you want a poem instead of a weather report.

The trouble is that the same dial governs hallucinations. A 2025 study reported evidence that creativity and hallucination tend to increase together as models are encouraged to be more adventurous and explore regions of lower probabilities. In other words, the model that dares to write something genuinely new is the same model that dares to make something up.

A different 2025 study found that the mechanisms that let these systems produce novel, imaginative text by departing from learned patterns are the same mechanisms that open the door to hallucinations.

It is also easier said than done to simply build a better model that does one without the other. In September 2025, researchers at OpenAI and Georgia Tech argued that hallucinations are not bugs to be patched but a statistical inevitability of how these systems are trained and tested. Models are rewarded, like students in an exam hall, for guessing rather than admitting ignorance. A confident wrong answer may score better than an honest “I don’t know”.

Scrubbed of surprise

The second study went further, and this is where Penrose returns to the room. Drawing on the foundational theorems of computer science about the limits of computation, laid by the work of Alan Turing and Kurt Gödel, they argued that no computable model can ever be universally correct. There will always be questions on which any given machine must fail. Hallucination, thus seen, is not a flaw in our engineering but a long shadow cast by the limits of computation itself.

Penrose used just this family of ideas to argue that human thinking cannot be reduced to mere computation. Many philosophers and computer scientists think he is wrong but modern researchers are pointing at the same wall from the other side. Penrose said machines are bounded, human minds are not. Modern researchers say machines are bounded, and here is the proof. Both agree that the wall is there.

We are pouring extraordinary effort into making these systems truthful, and we should. But if accuracy and imagination draw from the same well, a machine perfectly scrubbed of error might also be a machine scrubbed of surprise.

This may also teach us something about ourselves. Human beings also hallucinate in a softer, broader sense all the time. We imagine futures. We invent stories. We see patterns. We form hypotheses. Much of this is wrong — but some of it becomes science, art, philosophy, and technology. The difference is that human societies developed methods to discipline imagination. Science uses experiments. Journalism uses verification. Philosophy uses argument.

Perhaps AI needs something similar: not a ban on hallucinations but better institutions of verification around it.

Viraj Kulkarni is an entrepreneur and AI advisor who has been building, deploying, and scaling AI systems since 2012. He studied computer science at University of California, Berkeley.