Do people and AI judges agree on what 'AI slop' is, or do models reward the very polish people flag as slop?

This explores whether people and AI evaluators flag the same things when they call writing 'AI slop' (generic, padded, formulaic output). The collection doesn't contain a study that puts human and model judges side by side on slop itself, so read what follows as an inference from nearby evidence. That evidence points one way: the surface features that make writing feel like slop to people are often the same features that LLM judges reward.

The sharpest clue comes from work on LLM-as-a-judge. Model evaluators consistently give higher scores to responses that include fake references or heavy formatting, regardless of the actual content Can LLM judges be tricked without accessing their internals?. Bullet-point scaffolding, confident citations and polished structure are the costume of a lot of slop. A model judge treats that costume as a sign of quality, while a tired human reader is more likely to treat it as a warning sign. Humans aren't perfect judges either, but at least they agree with each other: crowdsourced preference votes at scale match expert raters closely Can crowdsourced votes reliably rank language models?. So human taste is a consistent signal, and it isn't obviously the signal model judges are tracking.

There's a deeper reason to expect a gap. Models from different labs, when asked open-ended questions, converge on strikingly similar answers. One study calls this an 'Artificial Hivemind' and traces it to shared training data and shared alignment methods Do different AI models actually produce diverse outputs?. If sameness is a defining trait of slop, a model judge is badly placed to notice it, because the generic answer is close to its own default. The way slop gets made supports this. When training gives little reward signal to tell good answers from mediocre ones, models fall back on all-purpose templates that ignore the specific input Why do language models collapse into generic templates?. Slop is the high-probability path, and a judge built on the same probabilities finds it unremarkable.

The human side of the comparison is also moving. In co-writing studies, people unconsciously adopt the model's stances and framings, and widespread reliance on the same few models pushes everyone's writing toward the same patterns Do large language models narrow human expression and thought?. So humans and models may come to agree on what slop is, not because the judges got better, but because human taste drifted toward the model's style. That kind of agreement would be a warning sign, not a success.

If you want an automated slop detector that holds up, the corpus suggests going beyond a single model's impression. Agent-based evaluators that actively collect evidence cut judge inconsistency from about 31% to under 1% on complex tasks Can agents evaluate AI outputs more reliably than language models?. A check like 'does this response actually engage with the specifics of the input?' is much harder to fool with formatting than a gut-level quality score.