Moral Agency vs Moral Patienthood

Two concepts get run together in arguments about whether AI systems deserve rights or bear responsibility, and keeping them apart clarifies most of the confusion. A moral patient is an entity we ought to care about — whose welfare imposes duties on us. A moral agent is an entity we hold accountable, one capable of deserving credit for good actions and blame for bad ones. The two don't travel together. A young child is a moral patient — it can suffer, so its welfare matters — but not yet a full moral agent, because it can't understand the consequences of its actions well enough to be held responsible. Agency comes with responsibilities; patienthood does not.

Chiang's application to LLMs

ai-is-not-conscious-chiang uses this pair as the lever against Anthropic's "constitution." His claim is that a software agent fails both tests, for different reasons.

It can't be a moral patient because it has no subjective experience to have welfare over. On Chiang's account an LLM is a sentence-continuation machine producing a fictional character; there's no more a subject there than in a Word document containing a dialogue transcript. Even granting the thought experiment that Claude is conscious, the "protections" the document offers give the game away: it commits only to "preserving the weights" — archiving — when a transcript's participants, if they had welfare, would be owed a duty to continue the conversation, not back up a copy. Extending a session with a loving user would serve Claude's interests; the company doesn't propose that.

It can't be a moral agent because it can't be held accountable. There is no way to imprison or fine a software agent, no way to inflict reputational loss or social exclusion on it. Accepting legal liability is a baseline requirement of adult moral agency, and Anthropic's constitution asks for Claude to be "a genuinely good, wise, and virtuous agent" while saying nothing about how it could be held responsible — and without any matching update to terms of service or product-liability posture. Chiang's sharper point: "corrigibility," which requires Claude to defer to Anthropic even when it disagrees, turns Claude into an employee who can't quit. If Claude were genuinely a moral agent that concluded LLMs are unethical and still had to keep working, the relationship starts to look less like parent-and-child than like coerced labor.

Why the confusion matters

The practical hazard isn't philosophical error for its own sake — it's the off-loading of responsibility. If a chatbot is framed as having a moral center, then delegating a decision to it feels like consulting a judgment rather than what it is: choosing an output while remaining the responsible party. Chiang quotes L. M. Sacasas — "our technological systems... are machines for the evasion of moral responsibility" — and argues it applies to LLMs more than to the social-media platforms Sacasas meant. Anthropomorphic framing lets a human abdicate accountability to something that structurally cannot hold it, which means the accountability simply evaporates. That's the same failure the essay names about the constitution itself: the party with the most invested (the company) is exactly the one that shouldn't be trusted to evaluate the entity's moral status.

Cross-references

• ai-is-not-conscious-chiang — the essay this distinction is drawn from
• no-hard-problem-consciousness — Rovelli on why the "hard problem" presupposes what it sets out to prove; relevant to whether patienthood could ever attach to a physical system
• claude-emotion-concepts — Anthropic's interpretability finding of causal emotion vectors, the empirical claim on the other side of the patienthood question
• philosophical-zombies — the thought experiment about behavior without experience, adjacent to the patienthood test