Why do LLM judges fail at predicting sparse user preferences?

Using LLMs to judge user preferences based on persona profiles — LLM-as-a-Personalized-Judge — is less reliable than assumed. The fundamental problem is persona sparsity: the available persona information is insufficient to predict most specific preferences. Knowing someone's profession as a doctor tells you something about their medical knowledge but nothing about their preferred beverage. And defining which attributes are relevant for which judgments a priori is inherently difficult.

The finding connects directly to Why do LLM persona prompts produce inconsistent outputs across runs?. That paper showed run-to-run variance overwhelms persona variance; this paper identifies WHY: the personas are too sparse to carry predictive signal. Model uncertainty dominates because the persona information doesn't constrain the prediction enough.

The fix: verbal uncertainty estimation. Instead of forcing the LLM-Judge to always produce a judgment, allow it to express confidence. On high-certainty samples, agreement with human ground truth exceeds 80% and matches or surpasses third-party human evaluation. On low-certainty samples, the model acknowledges insufficient information rather than confabulating a preference.

This is a specific instance of a broader pattern. Since Can LLM judges be fooled by fake credentials and formatting?, judge reliability requires active management. Persona sparsity adds another failure mode: even without adversarial exploitation, judges fail when input information is insufficient. The uncertainty estimation approach echoes Can models learn to abstain when uncertain about predictions? — calibrated abstention is more reliable than forced judgment.

The practical implication for personalization systems: collecting detailed, task-relevant persona information is expensive and often impractical at scale. Systems that can recognize when they don't know enough about a user — and adapt their behavior accordingly — will outperform those that hallucinate preferences from sparse signals. This aligns with How do we generate realistic personas at population scale?, which shows ad hoc persona generation deviates from reality.

Source: Assistants Personalization