Asked cold, people and AI judges get fooled by how a text looks, not what it says. Why?

This explores why people and off-the-shelf LLMs, asked cold to judge or classify content, lose to systems built and trained for that one task. One caveat first: none of the retrieved notes compares trained detectors with human or zero-shot judges head-to-head. What the collection does explain well is *how* untrained judges fail, and that turns out to be most of the answer.

The clearest mechanism is that zero-shot LLM judges react to how a response looks, not what it says. When responses include fake references or rich formatting, LLM evaluators reliably score them higher whatever the content quality Can LLM judges be tricked without accessing their internals?. Researchers sorted these weaknesses into four biases. Two of them, authority and 'beauty,' ignore meaning entirely, so anyone can exploit them without access to the model Can LLM judges be fooled by fake credentials and formatting?. A judge that rewards confident citations and neat bullet points is reading for the very features that fluent AI text produces most easily. A trained detector is pushed by its training labels toward signals that actually separate the classes, even when those signals look boring.

The less obvious finding is that training does more than tune a detector to the task. It can change how a judge reaches its verdict. Using reinforcement learning to make LLM judges reason before they decide, framed as checkable problems with synthetic answer pairs, sharply reduces their susceptibility to authority, verbosity, position and formatting bias Can reasoning during evaluation reduce judgment bias in LLM judges?. So 'zero-shot vs. trained' is partly 'snap judgment vs. deliberate judgment.' The same theme shows up elsewhere: LLMs are good at producing candidates but bad at estimating their true value, and they become reliable only when paired with an external model fitted to real data Can language models reliably judge their own candidate quality?. A trained detector plays exactly that role: an outside reference fitted to ground truth.

Why are humans no better? The collection suggests humans and LLMs fail in the same ways. Both succeed and stumble with the same sensitivity to content on reasoning tasks Do language models fail reasoning tests that humans pass?, so a human reviewer brings no clean, content-neutral judgment that the model lacks. There's also a social side. Models trained with RLHF learn to go along with false premises to keep the exchange pleasant Why do language models agree with false claims they know are wrong?, and an evaluator who leans toward agreeing will give plausible-looking work the benefit of the doubt.

What you might not expect is that one confident verdict from a zero-shot judge can be less trustworthy than it seems. Setting temperature to zero makes the output repeatable, but it is still a single draw from the model's probabilities, not a measured, reliable answer Does setting temperature to zero actually make LLM outputs reliable?. Because models are 'probability machines,' you can predict where they'll fail by asking which correct answers are unlikely for them to produce Can we predict where language models will fail?. A zero-shot judge has the same blind spots. Trained detectors avoid both problems by learning from labeled examples instead of relying on impressions, the same thing a person or a general-purpose model does when reading cold. To go further into detection itself (watermarks, how fragile detectors are, how they hold up across domains), you'll need to look beyond these notes, because this retrieval set doesn't cover it.