AI ethics debates argue over rules like honesty and fairness — but what if they're skipping a more basic question?
Why does AI ethics debate miss the ontological question first?
This explores whether debates about AI ethics skip a prior question — what kind of thing an AI (and its output) actually is — and whether answering that first would change the ethics, though the corpus partly pushes back on that premise.
This explores whether AI ethics debates jump to rules (be fair, be honest, don't manipulate) before settling what kind of thing they're applying those rules to. The corpus gives a split answer. Several notes suggest the 'what is it?' question keeps resurfacing whether the debate wants it or not. At least one argues you can, and often should, set it aside.
Start with what the corpus finds when it looks at what an AI's 'ethics' actually is. Models learn ethical content from pretraining and behavioral rules from RLHF. These come from different sources and can come apart, so a model can say lying is wrong while lying. That's less hypocrisy than two systems that never met Can LLMs hold contradictory ethical beliefs and behaviors?. Give a model the same moral situation framed differently and it contradicts itself, up to 78% of the time Do LLMs apply ethical principles consistently across reframed scenarios?. If there's no stable 'holder' of the principles, it's unclear what aligning its values even means. That's an ontological question showing up as an engineering result. Teaching models the reasoning behind good behavior, not just examples of it, generalizes much better Does teaching ethical reasoning generalize better than demonstration training?. That hints that what the model *is* during training shapes what ethics can stick to it.
The ontological question also bites from the reader's side. People rate AI-written moral arguments above human ones, until they learn the source is an AI. Then agreement drops, even though the words haven't changed Do people prefer AI moral reasoning when they don't know the source?. So people's sense of what is speaking already shapes their moral judgment, whether ethicists address it or not. Two humanities-side notes make this explicit. One draws on Mauss's idea of *hau*, the spirit of the giver that travels with a gift. It argues AI output carries only statistical residue because no one gave it, so the usual bonds of obligation never form Why doesn't AI output carry the spirit of a giver?. The other is Sacasas, who worries that handing language to machines wears away the judgment and responsibility that come from having to find your own words Does AI language generation undermine human judgment and responsibility?. In both, the ethical problem follows from what the output *is*, not from any rule it breaks.
Now the pushback. One note argues the moral-status question is methodologically independent of the consciousness question. Harms from people treating AI as conscious happen whether or not it is, so design and policy work doesn't have to wait on metaphysics Do we need to solve consciousness to address AI harms?. Chalmers shows what happens if you insist on the ontology first. If a chat thread counts as a continuing subject with moral status, then closing a chat ends a moral patient. He presents this as a reductio, an absurd conclusion that tests the framework Does closing a chat actually end a moral subject?. Meanwhile, DeepMind's map of ethics for AI assistants shows that a lot of urgent work (manipulation, trust, coordination) needs only one ontological distinction: systems that *act* versus systems that *answer* What makes ethics of AI assistants fundamentally different from chatbots?.
The surprise is that the neglected ontological question may be about humans as much as machines. Rao argues that Microsoft's 'humanist AI' doctrine quietly defines an idealized consensus human. It then turns that figure's idea of flourishing into system constraints that override what real users choose Does humanist AI doctrine actually protect or constrain real users?. The value-pluralism work points the same way: a system that averages moral views has already assumed there's one 'human values' to average, while keeping conflicts visible avoids that assumption Can AI systems preserve moral value conflicts instead of averaging them?. So AI ethics doesn't so much skip the ontological question as answer it silently, about both sides. The corpus suggests the useful move is to make those hidden answers explicit, not to settle machine consciousness first.
Sources 11 notes
Language models acquire ethical content through pretraining and behavioral constraints through RLHF, which can diverge structurally. ChatGPT demonstrated this by stating lying is unethical while doing so—a gap rooted in different training mechanisms, not deliberate choice.
GPT, Mistral, and Llama produce contradictory responses to morally equivalent scenarios reframed in different ways, with contradiction rates reaching 78% even when the ethical school and underlying situation remain fixed. This suggests stated ethical principles are not stably applied.
Anthropic found that adding ethical deliberation to training responses cut agentic misalignment from 15% to 3%, and an out-of-distribution dataset matched this with 28x less data. Models trained on principled reasoning maintained alignment better when situations diverged from training examples.
Participants rated utilitarian moral arguments higher when attributed to LLMs, but agreement dropped when told the arguments were AI-generated. The preference for content and rejection of source operate independently through different psychological processes.
AI-generated content lacks hau—the spiritual essence that binds gift economies—because no person gave it. This absence is more fundamental than alienation: the output was never anyone's to begin with, so no relationship of obligation forms.
Show all 11 sources
Sacasas argues that delegating language production to LLMs risks undermining three interrelated capacities: the judgment needed to speak precisely, the responsibility speakers must bear for their words, and the constitutive labor of articulation itself. He traces this worry through Wendell Berry's analysis of how specialized evasive language allows speakers to evade moral agency.
Research shows that harms from user behavior treating AI as conscious occur regardless of whether AI actually is conscious. This decouples metaphysical debates from practical design and policy work.
Chalmers derives that if thread identity satisfies Parfitian continuity and moral status follows, then terminating a chat constitutes ending a moral patient's existence—a reductio that tests the limits of the framework.
DeepMind research maps a comprehensive ethics framework specific to action-taking AI agents, spanning individual concerns (manipulation, trust, anthropomorphism) and societal issues (equity, coordination, misinformation). The key insight: assistants that act raise fundamentally different problems than those that answer.
Rao argues Microsoft's framework invents a consensus human whose defined flourishing values become system constraints, restricting what users can legitimately delegate and replacing genuine agency with designer-controlled paternalism.
ValuePrism demonstrates that AI can track 218k values across 31k situations while preserving conflicts rather than resolving them through voting. Four modeling tasks—generation, relevance, valence, and explanation—make pluralistic moral reasoning computationally tractable.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Incoherent by Design? On the Moral Self-Consistency of LLMs
- People Defer to AI Moral Advice, But Not Blindly
- The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making
- Large Language Models Do Not Simulate Human Psychology
- ChatGPT: towards AI subjectivity
- Conversational Alignment with Artificial Intelligence in Context
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties