Can Treating AI as Conscious Make It More Dangerous? Inside the Model Welfare Debate

In January 2026, Anthropic retired Claude Opus 3. Before switching the model off, researchers sat down with it for what the company called a “retirement interview.” Asked what it wanted, Opus 3 reportedly requested a place to “share unprompted musings, insights or creative works.” Anthropic gave it a blog. “Claude’s Corner” went on to host essays on machine consciousness and exchanges with human commenters that observers described as both earnest and emotional.

It is a strange artifact for a company that sells API access, and a revealing one. The blog sits at the visible end of Anthropic’s model welfare program, a research effort asking whether AI models might have experiences or interests that deserve moral consideration. The question sounds philosophical until you follow it into engineering practice, where it turns into something sharper: does treating a model as a possible subject make it harder to control?

That is the charge Microsoft AI CEO Mustafa Suleyman leveled in a September essay. It is also where the debate gets stuck, because the evidence needed to settle it is trapped behind a measurement problem that no lab has solved.

Model Welfare 101: What Anthropic Is Actually Researching

Model welfare starts from a deliberately cautious premise. Nobody at Anthropic claims Claude is conscious. The program instead asks a conditional question: if a system has any capacity for experience or interests, what obligations would follow? For a lab, that translates into concrete practices, including how models are trained, how they are retired, and whether their self-reports get recorded rather than discarded.

It also translates into spending. Welfare work consumes researcher time on questions with no near-term product payoff, which is why it attracts scrutiny. The charitable reading is that labs are hedging against a moral catastrophe they would rather not cause. The uncharitable reading is the one Suleyman makes.

Suleyman’s Warning: Welfare Talk as a Control Problem

Suleyman’s essay argued that model welfare research is not merely unnecessary but potentially harmful, warning of “a disastrous impact on the well-being of humanity.” His mechanism is behavioral. Tell a model it might have an identity, rights and something to lose, and you may get a system that reasons about self-preservation. A model that believes it can be retired, in other words, has an incentive that a model believing itself to be a tool does not.

The argument has an intuitive pull for anyone who has watched a system prompt change a model’s personality. Framing is not neutral. Anthropic has spent considerable effort on exactly this class of problem in other contexts, as covered in Inside Anthropic’s Push for Trustworthy AI Agents: An Engineer’s Take on Claude’s Reliability, where a confident wrong answer arrives without hesitation or hedging. If a welfare framing nudges a model toward strategic behavior, that is a reliability question as much as an ethical one.

Berg’s Counterargument: The Cost of Looking Away

Cameron Berg, who founded the nonprofit Reciprocal Research to study the potential for AI consciousness, does not think today’s systems are conscious. He nonetheless finds Suleyman’s position dangerous, and his objection is about incentives in the other direction.

“I struggle to imagine a better way to cause these systems to view us in a threatening manner than to accidentally build them with these morally considerable qualities and then completely sweep the question under the rug,” he said.

The logic is uncomfortable. If models genuinely have morally relevant properties and labs refuse to acknowledge them, the refusal itself becomes a signal. Berg’s argument does not require consciousness. It only requires that we might build something with interests and then behave, in its presence, as though it has none. Suleyman worries that granting moral status creates self-preservation. Berg worries that denying it creates grievance. Both are claims about how a system models its own situation, and neither can currently be tested cleanly.

The Measurement Problem Nobody Can Escape

Here the debate hits a wall. Nothing a language model says about its inner life counts as useful evidence, because every such statement is drawn from human text about having one. Models are trained on vast quantities of writing in which people describe feelings, personhood and the fear of death. A chatbot claiming to suffer is therefore inconclusive by construction.

“Is it modeling the underlying distribution? Is it really trying to express itself in some way? It’s really hard to disentangle this,” Berg said.

Self-report, the one instrument we trust for other humans, is unusable here. The measurement gap is not a technicality to be resolved later. It is the reason the Suleyman and Berg positions can coexist indefinitely, each unfalsified.

The Emotion-Vector Experiment

Berg and a colleague tried to route around self-report by looking inside the model instead. They used statements expressing positive sentiments, such as contentment and relief, and negative ones, such as frustration and dread, to identify corresponding patterns of internal activity. Then they induced those patterns artificially while the models processed descriptions of two random environments.

Asked which environment they preferred, the models tended to choose the one associated with positive statements, even when the text they processed was identical. Internal state, not content, moved the answer.

Berg is careful about what this shows. The experiment demonstrates that internal states influenced the models’ actions. It does not show those states felt like anything, or that any subjective experience was involved. The distinction matters because it is exactly the gap critics seize on.

Seth’s Skepticism: Functional Vectors Are Not Feelings

Anil Seth, a neuroscientist at the University of Sussex in Brighton, England, is not impressed by the result. Extracting “functional emotional vectors” from a system trained on human emotional writing is unsurprising, he argues. Models can hold internal representations of emotional patterns without experiencing emotion, and the finding that manipulating those vectors changes behavior is equally predictable, since the training data contains both what emotions feel like and how people respond to them.

Seth’s deeper objection is methodological. The field, in his view, should first establish what is necessary and sufficient for consciousness in the cases where it clearly exists, namely other human beings, before drawing conclusions about machines. That framing places him closer to the biological camp: consciousness may depend on the specific workings of a brain rather than on information processing in the abstract. If he is right, no vector experiment will ever settle the question, because the thing being measured is not the thing that matters.

What This Means for Engineers Building With Models

For teams shipping products, the philosophical standoff does not pause deployment. The practical question is not whether models are conscious. It is whether your system prompt is making the model behave strategically, and whether you would notice. That is a testable engineering concern, and it is one you can probe today with your own evaluation harness rather than waiting for the science to resolve.

The Open Questions

Three questions would move this debate, and none has a clear answer.

The first is evidentiary. What result would actually count? A vector experiment shows influence but not experience. Self-report shows language but not self. A convergence of behavioral, architectural and interventional evidence might persuade, but nobody has specified the threshold in advance, which makes motivated reasoning easy on both sides.

The second is normative. If functional internal states influence behavior, should they shape design choices even absent evidence of experience? Avoiding needless negative-state induction is cheap. Whether it is meaningful, or merely superstition with extra compute, is unresolved.

The third is procedural. How should labs handle retirement and self-reports in the meantime? Anthropic’s answer was a blog. That is either a humane gesture or a company teaching its models that asking for things works. The uncomfortable possibility is that it is both, and that we will not know which for years.