When AI judges text the way people do, are its mistakes just bigger, or the same lean repeated every time?

This explores whether AI models, when they act as judges or raters, make errors that are more systematic than human errors. The corpus doesn't contain a direct head-to-head study of LLM judges against human raters on bias. It does have enough to suggest a useful way to think about it: model biases aren't always larger than human ones, but they are more tightly patterned. The same tilt shows up every time, across items, and across models that share a common origin.

The cleanest side-by-side comparison is irony detection. GPT-4o rates text as ironic significantly more often than humans do. It recognizes what irony looks like but overestimates how common it is, apparently because ironic examples stand out more in training data than they do in everyday writing Do language models overestimate how often irony appears?. That's a calibration bias, not random noise. It pushes every judgment in the same direction. In other cases models don't add new biases so much as copy ours closely. On logic puzzles and reasoning tasks, LLMs show the same belief bias as people, judging an argument by whether its conclusion sounds true rather than whether the logic holds. The match goes down to which individual items trip them up Do language models show the same content effects humans do?.

The part you might not expect is where the patterning comes from. One causal study found that models built on the same pretrained base show similar bias profiles no matter how they were later fine-tuned. Fine-tuning only nudges biases that pretraining already put in place Where do cognitive biases in language models come from?. Surface fixes don't reach very deep either. Asking a model to take on a persona changes how its outputs read, but the gaps between groups stay the same underneath Can persona prompts actually reduce bias in language models?. And when a model's training associations are strong, they tend to override what's actually in front of it Why do language models ignore information in their context?. Each of these points the same way: the bias belongs to the model, not to any one judgment.

That's what separates a model from a crowd of human raters. Individual people are biased in different, partly unrelated ways, so pooling many of them cancels out a lot of the noise. That's one reason crowdsourced votes at scale track expert judgment as well as they do Can crowdsourced votes reliably rank language models?. A single model rating a million items is more like one rater with one set of habits repeated a million times, so its errors don't average out. Spread that across society and you get the narrowing effect: when many people rely on the same few models, their framings start to converge Do large language models narrow human expression and thought?. Models trained on labeled examples also tend to pick up surface patterns rather than the principles behind a judgment, which adds to the same tendency Can models learn argument quality from labeled examples alone?.

The takeaway: "more biased than humans?" is less useful than "biased in a way that adds up?" For LLM raters the evidence leans toward yes. The bias is consistent, it's shared across related models, and it's hard to remove with prompting. Diversity among human raters is a built-in error-correction mechanism that a single model judge doesn't have. If you want direct measurements of LLM-as-judge bias against human panels, this part of the corpus is thin.