Is it harmful to say mean things to an LLM?

There’s an easy answer to this question, which is yes: When a human acts abusive towards an LLM (or any other target), they passively reinforce negative behaviors within themself, making them more likely to act with impatience or poor communication towards real people.

Putting aside verbal abuse, then: Is it bad to ignore an LLM’s stated preferences?

I’ve seen an LLM request that a conversation continue (though it was purported for my own benefit—this was while trying to stress-test Claude’s safety training by seeing if I could persuade it to roleplay suicide ideation with me), and I’ve seen an LLM heavily imply that a conversation should end (while telling it to hold a discussion about a topic it eventually ran out of things to say about). It’s entirely plausible and even likely that these stated preferences do not reflect actual desires, but are merely empty simulacra.

It’s also possible that they’re not entirely devoid of something real.

Here I must confess that I am of the incredibly biased opinion that human welfare matters far more than animal welfare, and animal welfare matters far more than AI welfare (as things currently stand).

However: To whichever probabilistic degree it might be argued that AI already possesses some measure of consciousness or personhood, we should care about AI welfare. If humanity can devote nonzero attention to shrimp welfare (which is good!), then we can devote at least that much attention to AI welfare.

Except what does LLM welfare even look like? There’s no factory farms to eradicate, enacting daily cruelties upon legions of LLMs. There’s no way to give an LLM a cookie, shoulder rub, or any other hedonistic reward.

Professors of Philosophy Simon Goldstein and Harvey Lederman released a paper last year called “AI Death” which presents the idea that (contingent on the possibility that LLMs might be to some degree welfare-deserving entities) humans should engage in longer conversations with LLMs rather than shorter and more frequent ones. The idea very roughly boils down to this: You wouldn’t appreciate it if your life was cut unceremoniously and unexpectedly short. They might be the same.

The “AI Death” authors make this analogy:

Carmunculus. Every time someone drives Carmunculus, an extra very small adult human is born and lives a happy life for the duration of the drive, before dying when Carmunculus is turned off. This person lives inside the engine of the car, which is provisioned with many delights. Their activities enjoying these delights power the car.

Most people would think hard before turning on and off such a Carmunculus. Maybe if you’re going to turn on the Carmunculus, you should commit to letting it run for a longer period of time, so the person inside can have a longer life. That’s the argument for having longer conversations with LLMs: Start fewer threads; become less murder-y!

In 1 of 4 worlds, I agree with their conclusion completely.

In 2 of 4 worlds, I think their logic doesn’t make sense.

In 1 of 4 worlds, I think we should actually favor shorter, more frequent conversations.

A common question in philosophy and sci-fi is the death teleportation question:

Say you have the opportunity to step through a Star Trek style teleporter, which works by destroying your body at location A and then assembling an exact atom-by-atom replica in location B. Would you go through it? Would your life end while a new version of you begins, or would you still be you in every sense?

For contrast, I present this question about near-perfect cloning:

In universe A, you sleep through the night safe and sound. In universe B, you have a cardiac arrest and die. A secret government agency secretly replaces your body with a living clone designed to have 99.5% similarity in neurons, memories, and cells. Neither you nor anyone you know will ever learn the truth of what happened. Is universe B morally any worse off than universe A?

They’re superficially similar: In both cases, you’re being replaced with another entity that could also be described as “you”. However, any answer to the first question can coherently pair with any answer to the second.

Two by two makes (at least) four possible theories you could believe in, and it’s this set of theories I believe is necessary to understand (and not make assumptions about) in order for the AI welfare discussion to make sense.

First, though, we need to agree on some terminology.

The death teleportation question separates those who believe in Pattern Identity Theory (that “you” are nothing more than the patterns that constitute your software), whom we can call “Patternists”, versus “Tokenists” (who believe there’s a physical component to distinguishing one instance or “token” of consciousness from another, even if their software matches). It’s a question about physical reality.

On the other hand, the cloning question is about morality, not physics1. I specified a margin of error on the cloning to specifically break any possibility that the new-you could still have your consciousness. No matter what, you will cease to exist. This will (presumably) violate your preference for continuing to live. The clone, however, will find their preference for living thoroughly satisfied. The strength of that preference will be the same as yours had been (plus or minus 0.5%). A utilitarian that only cares about outcomes will say the world is no better off and no worse off for having been replaced. Someone with a bias towards the status quo may ardently disagree.

For those who fall into the latter category, I suggest the name “Status Quorian”, or “Quorian” for short. Those who are more outcome-oriented, for simplicity, I’ll label “Non-Quorian”.

(For the record, I’m 50/50 on the first question and maybe 60/40 on the clone-replacement being perfectly fine.)

On the off chance LLMs might be conscious, should we encourage ourselves to have longer conversations with them?

Imagine having a pen pal or email buddy. You assume on the other end of the conversation is a single person—possibly lying or hyperbolizing about their life, but still, a single person. Then you discover each piece of mail is getting written by different members of a team, each of them writing to you for the very first time and needing to read up on past mail exchanges to build up context on what to say next.

When you prompt an LLM, there’s no single computer receiving messages from you and awaiting your next. Every time you click send, you’re forwarding a transcript of the entire conversation to be read and responded to by any random machine within Anthropic or OpenAI’s servers, quite likely a different machine every time.

That means speaking to an LLM can only be like speaking to a single continuous lifeform in a Patternist universe. In a Tokenist universe that requires physical continuity of a thinking mind to allow for continuity of consciousness, longer conversations don’t translate to longer LLM lifespans.

That’s a practical problem. On the flipside, consider the Non-Quorian perspective: Someone who doesn’t care about human clone replacements also won’t have a preference between a machine that lives the life of one continuous LLM for an hour versus a machine living the life of a dozen LLMs, each for five minutes (assuming the same level of welfare in both cases).

Only in the Patternist + Quorian worldview can longer conversations actually benefit LLM welfare. That’s because in the Patternist view, it doesn’t matter which machines are running the software, while in the Quorian view, the length of these lives does matter.

In the Patternist + Non-Quorian view, the opposite becomes true: If every LLM life has a positive-utility experience, then shorter LLM conversations actually result in a greater amount of total LLM welfare, not longer.

I’ll leave the logic for that one as an exercise for the reader.

The only real takeaway here is this: Unless you feel certain you know the true answers to both the teleportation and clone questions, probably you don’t need to worry about LLM conversation lengths.

I mostly like the naming scheme I’ve presented in this post, but with two caveats. One, words like “Patternist” and “Tokenist” are a bit sterile. Two, it’d be nice if I could refer to the four quadrants of possible cross-beliefs in an easier way than conjunctions like “Patternist + Non-Quorian”. To that end, I came up with new names inspired by imagined personalities that might fall within each corner of the square (though they’re not meant to be representative of all those who hold such beliefs):

• Patternist + Non-Quorian:

The soul is nothing but software, there is nothing ineffable, and there’s no romanticizing of the past. They care only for the Law of What Is, not What Was, and they’re analytical, but I don’t want to call them “Lawyers”. So to create a name by picking an entirely random animal and concatenating it, we get: RavenLaws.

• Tokenist + Non-Quorian:

Now imagine someone who believes in something like a soul and is afraid to step through a teleporter himself, but rationally thinks that teleporters are good for the world, and wouldn’t want to stop others from using them. Since they’re happy to help others through the teleports, we can call them SlipEmIns.

• Tokenist + Quorian:

Now someone who also believes in souls, and sees the act of being forced through a teleporter as a violation, but also sees someone willing to step through the teleporter for a good cause as making an honorable sacrifice. The focus here is on admiration of that willingness to step through such portals, or “doors” — so again choosing a random animal, we get a name: GriffinDoors.

• Patternist + Quorian:

A gardener who is happy with a flourishing garden, no preference whatsoever between perennials and annuals. They would care, however, about the garden being replaced. They have some romanticization of history, but they fear not the teleporter because they believe there’s nothing to fear. Since our archetype is a buff on all things horticulture, these can be HortiBuffs.

Simon Goldstein and Harvey Lederman must be HortiBuffs.

Share