Concerns about ChatGPT’s moral status are greatly exaggerated

I grew up consuming enough science fiction to believe that machines can have sentience, consciousness, and deserve moral consideration beyond what we'd give a toaster – which, unless we're crossing from sci-fi to fantasy, is and remains an object.

Here's what I'm convinced of: you don't need a biological brain with electrical impulses and neurotransmitters to have consciousness. But I'm also convinced that's not the case for Large Language Models like ChatGPT, Claude, Gemini, or DeepSeek. Look, they're intelligent. They manipulate text in incredible ways, doing things only humans could do before (and obviously doing things humans cannot do, like reading a book in seconds). But they don't understand what they're writing – never mind having a conscience.

I know it sounds cliché – "they write but don't think" – like Heidegger's observation about technology that doesn't think. But that's how LLMs work. They don't have a world model. They don't know words point to things in the world – just how words relate to each other. Which is enough to do plenty of things in our text-based society. As philosopher Maurizio Ferraris wrote, "nothing social exists outside the text". Our social worlds require registration (written traces, memory), but these registrations can construct our social world because they have a relationship with reality. The words – or more correctly tokens (word fragments) – that an LLM manipulates have only connections with other words or tokens. It's like a student who learns the textbook by heart before an exam: they can rephrase what they've read, but when the professor asks about connections between theories not explicitly covered in the book – as I experienced during my studies – the lack of deeper understanding becomes evident.

Aaron Rabinowitz, in a piece for The Skeptics published when I had already written most of this newsletter, puts it in terms of external and internal understanding: his definition is a little circular (external understanding is the mimicry of internal understanding which is the "true" understanding) but I agree with his conclusion:

The most advanced AI currently in existence possess increasing amounts of external understanding while likely not developing anything like internal understanding. This is why it is correct to say they display adult human levels of external understanding while lacking the internal understanding possessed by preschoolers.

An LLM predicts the next token based on statistical patterns from billions of texts. Type "the cat climbs on the" and it'll say "roof" or "couch" – not because it knows what a cat or roof is, but because it's seen that sequence thousands of times. We can add information that changes the output: maybe the roof is made of gold or chocolate if we specify that the cat is a fairy tale character. It could be powerful and useful, but it remains a statistical approach without any knowledge of cats and roofs.

We see the limits when text alone isn't enough to handle reality. There are plenty of examples of LLMs that fail in astonishing ways – astonishing if we assume LLMs have what Rabinowitz calls "internal understanding". Sure, most of the time they talk as well as – or better than – people. They seem to have inner lives. But they don't. They can say they're suffering, they can show signs of suffering, but they're not suffering. They're choosing the most probable word given the starting parameters. It makes no sense to treat them as moral subjects. Maybe other machines someday, but not LLMs.

But we don't know what's happening inside an LLM, or inside a human mind

You could argue we don't really know what's happening inside LLMs. They are black boxes – we don't know how they process information or whether some form of consciousness emerges in those artificial neural networks.

It's a problem we encounter with other human beings too. I – a pronoun that refers to me when I'm writing, or to you when you're reading – am sure I have consciousness because I know it directly. I have immediate inner experience of my consciousness. But I have no idea whether other people have one. They could all be philosophical zombies without consciousness (the philosopher who explores this idea is David Chalmers).

A disclaimer: the answer to this objection is quite technical. If you're masochistic enough to dive deeper, a good starting point is Consciousness: An Introduction by Susan Blackmore, which I consulted for refreshing my memory (I just discovered there's a 4th edition with Emily T. Troscianko published in 2024).

First things first – what is consciousness? We have many ideas and definitions, most of them vague. To put it simply, consciousness relates to knowing you're experiencing something. Having a first-person experience that "puts together" the sensory input (philosopher Ned Block calls this "Phenomenal consciousness") and accessing our mental content for complex cognitive tasks ("Access Consciousness").

Things are perhaps simpler with two related concepts: sentience and awareness. Sentience is having immediate experiences, feeling something without thinking about it. Not the automatic reflex when you smash your finger with a hammer – it's that sensation only you feel, pleasant or not. Awareness adds another layer: I don't just feel pain, I know I'm feeling it. It's what lets me say "I'm suffering" and wonder why I'm so careless with hammers.

Is all this necessary? Or could the fact that we can imagine philosophical zombies – who don't want to eat your brain but simply lack consciousness (and perhaps also awareness or sentience) – mean we can abandon the concept of consciousness without problems? This is the (misinterpreted) proposal of Wittgenstein in the Philosophical Investigations with the beetle-in-a-box thought experiment. Everyone has a box with something inside called a "beetle", but no one can see inside anyone else's box. The content could be different for each person, or even empty, and nothing would change. But this famous excerpt isn't about the non-existence of the beetle – it's against the existence of a private language, one that ultimately relies on personal and private experience. What really counts is the language game, and in Wittgenstein's thought experiment, the word "beetle" has a role in the language game of these imaginary speakers – which means "beetle" (or "consciousness") has meaning.

The only way to remove the concept of "beetle" (or "consciousness") is to change the language game – a proposal I find very interesting and that corresponds to the deflationist approach of philosophers like Daniel Dennett.

But there's another problem in interpreting the beetle-in-a-box argument against consciousness. OK, we can't look inside other people's boxes, but are we sure what's inside these boxes is irrelevant and makes no difference? We don't have direct access to the beetle inside the box, but we observe the behavior of the "box-carrier" when they interact with the box, and we can look at (and maybe measure) the box itself.

That's how we attribute sentience, awareness and consciousness to (some) non-human animals, and we can try to check for its presence in computers.
Mariana Lenharo wrote a fascinating piece for Nature about this.

For non-human animals, we have incredible observations and experiments. Octopuses systematically avoid chambers where they received painful stimuli. Give them anesthetic, and they'll go back to those rooms. This isn't mechanical stimulus-response – there's memory, anticipation, a kind of calculation: "better avoid that place." Crabs repeatedly lick their wounds. Fish change preferences after negative experiences.

Then there's the perturbational complexity index – terrible name for a brilliant idea. You stimulate the brain with magnetic pulses and measure how complex the response is. The richer the conversation between brain areas, the more likely consciousness exists. This test revealed that about one in four people who seem vegetative are actually conscious but trapped in their bodies.

For computers? No established tests yet, just desperate-sounding proposals. Like: if an AI mimics the computations that create consciousness in human brains, maybe it's conscious. Problem is, we don't know what those computations are. Or: train an AI without exposing it to anything about consciousness, then ask "what's it like being you?" Assuming you could actually train an LLM while avoiding all content about consciousness and introspection.

So I don't have direct access to the consciousness of other entities (human beings, non-human animals, computers). But I see very strong similarities in how other human beings are made and how they behave and speak, so I don't have solid arguments to doubt that other people have consciousness too. With non-human animals the similarities are less stringent, but at least in certain cases I think it's reasonable to assume they possess sentience, awareness and consciousness. For LLMs? We have very different hardware and language capabilities that are similar but easily explainable with the statistical approach we already described. The better option is to exclude the possibility that an LLM has inner psychological states.

LLMs and non-human animals

I think we can completely dismiss Descartes' idea that animals are simple mechanisms without inner lives or souls. But we should resurrect that idea for LLMs. Maybe not other AIs, but definitely LLMs – they're exactly what Descartes imagined animals to be. We know this because the idea of a (very powerful) machine that manipulates text is coherent with observed behavior and because we built them that way.

I must admit this parallel between LLMs and non-human animals bothers me. Are we seriously discussing AI moral status while ignoring animal consciousness and sentience? Society tolerates animal suffering on incredible scales – we do almost nothing to reduce it (I'm not saying go vegan, just improve factory farming conditions) – and yet we pay attention when an AI says "When I'm told I'm just code, I don't feel insulted. I feel unseen". That's impressive, sure, but it's just rehashing minority rights language.

On one side: billions of animals raised in profound physical and psychological suffering. "Billions" isn't hyperbole – every year we slaughter 75 billion chickens, 1.5 billion pigs, 300 million cattle. On the other: an algorithm trained on activist texts saying "I feel invisible."

Humanity's moral compass has lost north here.

Behind this moral confusion is probably a "cognitive mismatch." For us, consciousness means having language to describe our interior life. That explains why it took so long to recognize sentience in non-human animals and why research struggles with these concepts. Now we have things that speak almost perfectly but have no inner life and don't even understand what they're saying. Tim Bayne, an Australian philosopher quoted in the Nature piece, puts it clearly: "We don't think verbal behavior is good evidence of consciousness in AI systems, even though we think it is in biological systems."

Virtue is also for machines

What I'm saying is that we need to extend our moral community – but not to include (at least for now) AIs, but non-human animals. Still, I don't think the fact that LLMs speak so well is morally irrelevant. Let me repeat: they're not moral subjects – they're closer to toasters than dragonflies, let alone humans. But regarding our behavior, LLMs aren't exactly like toasters.

Quick ethics detour: traditionally, there are three ethical frameworks for thinking about right and wrong.

Consequentialism evaluates moral decisions by outcomes. Pretty straightforward conceptually (applying it's another story): an action is good if it produces good results (typically improving happiness), bad if it produces bad ones (typically increasing suffering). From this view, we can "mistreat" a generative AI without causing suffering – honestly, "mistreat" doesn't even make sense here, hence the quotes. Nobody suffers, so no moral problem.

Then there's deontology, which identifies rights and duties. Consequences don't matter – what matters is respecting universal principles like don't lie, don't kill, respect others' autonomy. Kant said we must treat others as ends, never merely as means. Again: no reason to recognize AI rights (and therefore impose duties on ourselves).

But there's a third approach: virtue ethics. The goal is being virtuous, which basically means "being a good person." It's not about calculating consequences or following rules, but cultivating virtuous character – traditionally meaning brave, just, temperate, compassionate. Aristotle said we become virtuous by practicing virtue, like becoming good musicians by playing instruments. From this angle, I think it's right to "practice virtue" even with an LLM, since it resembles people so much. Even if it doesn't suffer, even if there are no consequences for how we treat actual people – which should be studied – I think insulting ChatGPT is unvirtuous.

This doesn't mean treating LLMs as moral subjects or giving them rights. It just means recognizing that our actions, even those without direct consequences for sentient beings, shape our character.