In a viral essay about how ludicrous the idea that LLMs are conscious is, science fiction writer Ted Chiang asked us to consider Microsoft Word:

“Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded,” Chiang wrote. “Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time.”

Let me tell you about a Microsoft AI researcher, then, who recently spent quite a lot of time considering whether the legendary Microsoft real time strategy game Age of Empires II is conscious, and built a basic neural network within the video game using digital goats to prove his point.  

“If LLMs Have Human-Like Attributes, Then So Does Age of Empires II,” is the title of Adrian de Wynter’s paper showing his work. He told 404 Media that absurdity can be a powerful tool. “I have this tendency to dial up things to 11 when I really think I need to make a point,” he said. “I should also note that absurdism is pretty standard in philosophy and theoretical computer science.”

And so De Wynter built an LLM within AoEII using goats. “The point of the paper is to formally show that we anthropomorphise too readily, and that sometimes the claims we make with regards to LLM capabilities are too strong,” he told 404 Media. “It's not an easy task, given that ‘human-like attributes’ is a bit of an abstract term.”

AoEII has a scenario editor, a sandbox mode that allows players to craft their own maps and quests using the game’s assets, and De Wynter used that to build an operational NOT AND (NAND) gate and a 1-bit perceptron within the game. In this crude version of an LLM, grass is 0, bridges are 1, and goats are the bits. “Only one rail is active at a time, with a goat acting as the signal carrier. When the gate fires, the bit-goats are removed (they ded) and a new bit-goat is placed in its respective output rail,” Wynter explained on his GitHub.

A perceptron is the simplest form of a neural network, it’s an algorithm that sorts an input into binary classes. YouTube is littered with videos of players doing the same thing with redstone in Minecraft. But no one claims the goats of AoEII are neurons in a thinking machine or that the complicated tracks of NAND gates players build in Minecraft show emergent intelligence.

De Wynter’s point here is that it’s possible to build a neural network in AoEII that works the same as the ones underlying Claude, ChatGPT, CoPilot, and all the rest. It’s a simplified version, yes, but the basic technology is the same. Faced with the absurdity of viewing AoEII goats as carrying the spark of consciousness, we might reconsider Anthropic's assertion that Claude has a “constitution” and experiences anxiety. 

If you’re looking for human-like traits, you will tend to find them. De Wynter’s argument is that it’s possible to build a basic LLM within Age of Empires II that has many of the same internal traits of the chatbots people use everyday. The difference is the interface. When a person interacts with an LLM through the medium of AoEII and not a chat window the perception of human-like traits in the LLM vanishes even though the underlying tech is the same.

He’d been playing AoEII since it came out in 1999 and thought it would be good for the thought experiment. “Age of Empires was an excellent way to drive the point home,” he said. “It is just about ‘alien’ enough to exemplify the representation-interpretation relation, but sufficiently well-known to really emphasise the point. It also works at a meta-level, since the example itself is a good representation of the argument.”

According to De Wynter, the problem of anthropomorphizing large language models (the neural networks we commonly call artificial intelligence) begins before scientific research even starts. He reviewed 315 computer science papers released over the last two years and found that 57 percent% began with the assumption that LLMs have human-like traits.

“What is common to some of these studies [...] is that they test and ascribe blanket human-like properties (e.g., anxiety or morality) to these LLMs while considering them the central subject of the experiment,” De Wynter’s paper said. “Regardless of these evaluations’ results being positive or negative, their core assumption–that LLMs possess anthropomorphic attributes–influences the experiment’s planning through (e.g.) the design of the test set, the interpretation of natural-language outputs, and even its null hypothesis. In turn, this directly impacts the conclusions made.”

“We either start by thinking that tokens represent language to LLMs the same way they do to us, or that because an LLM outputs a relevant string, it must be understanding the concept/having theory of mind/empathy/etc,” De Wynter told 404 Media. “This goes both ways: we could also assume that LLMs are blobs of weights just floating about on a GPU, but that would not help explain some skills that they are shown to have.”

Some people perceive their interactions with LLMs as “human” because the way they interface with it mimics a human conversation. “I propose that we need to stop assuming that LLMs behave like humans just because they were trained with natural language. Instead, we should perform experiments that allow us to see LLMs as how they are, not how we believe they should be,” De Wynter said.

De Wynter said widespread anthropomorphization of LLMs among executives, scientists, and the public is becoming an intractable problem. “This is why I used the goats: there are things which make the LLMs what they are in themselves (i.e., the relationship between weights as defined by some operation), and there are things which makes them what they are perceived as.” Goats and AoEII break the perception that these machines are human.

He’s also not ruling out that LLMs have some form of consciousness, but said that’s beside the point. “We tend to ascribe consciousness as some sort of binary construct (either it is or isn't!) but I'd argue that there are levels. It's hard to say that humans aren't conscious. But, what about a dog? Yes, of course. What about a potato? What about a virus? It's quite relative and we do tend to go for something human-like when we evaluate/define it, where, in reality, LLMs are things we have never seen before,” he said. “There was a recent article in science about bumblebees solving problems. Everyone was like surprisedpikachu.png — it's a great, amazing discovery. I love bumblebees and people understanding their behaviour is absolutely fantastic. But, did we really assume that they couldn't perform some sort of problem-solving? Why are we so surprised?”

De Wynter wants us to focus beyond the window we use to interact with LLMs. “I pointed out that if goats could show emergent capabilities, that's great. But these properties need to be preserved if we remove the chat panel. After all, it's not like your neurons know they are part of a brain,” he said. “I think the #1 thing that one can do to mitigate these perceptions is to have appropriate disclosures, and use good alignment techniques where the model is explicit on its nature. I worked a lot on how users perceive LLMs, and they do tend to get attached when it appears to have some sort of warmth/personality. To put it in another way, I don't get attached to my toaster, but I definitely get attached to characters on a movie screen.”

But people buy more objects—whether they’re toasters, phones, or LLMs—when they can empathize with them. “The issue here is that these capabilities and claims thereof are very strongly tied to marketing—after all, a lot of these models are products,” De Wynter said.

This is, of course, the point. OpenAI CEO Sam Altman has repeatedly implied that building out LLMs is a path to creating an AI god. Ilya Sutskever, a former OpenAI board member and scientist, often talked openly with employees about seeing the company’s LLM as a god-like consciousness. Anthropic CEO Dario Amodei told The New York Times that he can’t be certain if AI is conscious or not.

It’s good marketing for an industry that’s hemorrhaging money.