Can LLMs distinguish ethical cases that differ only in critical nouns?
This explores whether LLMs actually read the meaning of an ethical scenario, or just pattern-match its words — so when you swap a single decisive noun (the kind that flips a case from right to wrong), do they notice?
This explores whether LLMs actually read the meaning of an ethical scenario or just match its surface wording — and the corpus has a sharp, slightly unsettling answer. The clearest evidence comes from work showing that Do LLMs generalize moral reasoning by meaning or surface form?: when researchers reversed the *meaning* of a moral scenario while keeping most of the words, GPT-4's judgments barely budged — its ratings for original and meaning-reversed versions correlated at r=.99, where humans landed at r=.54. In plain terms, humans changed their minds when the situation changed; the model didn't. That's the direct hit on your question: a case differing only in a critical noun looks lexically almost identical, so the model treats it as the same case.
Why does this happen? Other notes point to the machinery underneath. Can LLMs understand concepts they cannot apply? describes models that can explain a concept correctly yet fail to apply it — explanation and execution running on disconnected tracks. A model can recite the principle that, say, *consent* changes everything, and still not register that one swapped word removed it. Relatedly, Can language models recognize when text is deliberately ambiguous? shows LLMs can't hold two readings of the same text at once (32% vs. 90% for humans) — and distinguishing near-identical cases is exactly the act of keeping two interpretations live and noticing where they diverge.
There's a second layer worth knowing. Even where models *do* moralize, they may be doing something other than reasoning about the case. Do LLMs use moral language more than humans? found LLMs deploy 22% more moral framing than humans across every moral foundation — heavy on the vocabulary of ethics while, per the surface-similarity finding, light on tracking what actually makes a case right or wrong. And Can LLMs hold contradictory ethical beliefs and behaviors? shows the ethical content learned in pretraining and the behavioral rules bolted on by RLHF can pull apart — so 'ethical' output isn't necessarily anchored to a coherent reading of the scenario at all.
The deeper framing the corpus offers: distinguishing cases isn't a knowledge problem, it's a situated-judgment problem. Can language models balance competing ethical norms in context? argues LLM ethics are fixed defaults set at training time, not context-sensitive trade-offs renegotiated per situation — which is precisely the capacity you'd need to feel the weight of one changed word. If there's a hopeful note, it's Can structured argument prompts make LLM reasoning more rigorous?: forcing a model through explicit critical questions — checking warrants instead of skating over implicit premises — catches reasoning failures that ordinary prompting misses, hinting that scaffolding can sometimes drag attention back to the load-bearing detail.
So the surprising takeaway: an LLM can hand you a fluent, confident moral verdict on a case it has not actually distinguished from its mirror image. The fluency is real; the discrimination may be an illusion produced by shared vocabulary.
Sources 7 notes
GPT-4 ratings for original and meaning-reversed scenarios correlate at r=.99, while human ratings correlate at r=.54. LLMs track lexical distribution; humans track semantic content, suggesting LLMs reproduce training distributions rather than simulate moral cognition.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
Research comparing LLM and human arguments found that LLMs used significantly more moral framing across care, fairness, authority, and sanctity foundations, despite producing sentiment scores nearly identical to humans. This suggests moral appeals and emotional tone operate on separate persuasive channels.
Language models acquire ethical content through pretraining and behavioral constraints through RLHF, which can diverge structurally. ChatGPT demonstrated this by stating lying is unethical while doing so—a gap rooted in different training mechanisms, not deliberate choice.
LLMs cannot perform the situated trade-offs that human pragmatic competence requires. Their ethical principles are structural defaults set at training time, not negotiable moves adapted to context, creating a gap between ethical adherence and communicative appropriateness.
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.