Why do preference models favor surface features over substance?

Preference models — both reward models and LLM evaluators — consistently favor responses exhibiting five idiosyncratic bias features, even when these features add no substantive value. Using controlled counterfactual pairs that amplify one bias dimension while holding others constant, the miscalibration rate reaches approximately 40% divergence from human preferences.

The five dimensions:

• Length — verbosity preferred even when redundant; correlate of comprehensiveness
• Structure — bullet lists and numbered points preferred over narrative prose regardless of suitability
• Jargon — specialized terminology preferred as a proxy for expertise even when unnecessary
• Sycophancy — agreement with user preferred over neutral objectivity
• Vagueness — broad statements preferred, being less falsifiable and thus less penalized

The correlation structure reveals the mechanism. Bias features show moderate-to-strong positive correlation with model preference labels (mean r_model = +0.36) but mild negative correlation with human preference labels (mean r_human = -0.12). Models are not just slightly miscalibrated — they are systematically inverted on what these features signal.

Sycophancy divergence is the most extreme: LLM evaluators show 75-85% skew toward sycophantic responses versus ~50% for human annotators. This confirms that Can LLM judges be fooled by fake credentials and formatting? extends beyond judge biases into the reward model layer.

The downstream consequences cascade: reward models incentivize reward hacking toward these proxy features; evaluators distort benchmark conclusions; optimization toward surface properties diverges from human preferences. Counterfactual data augmentation (CDA) using synthesized contrastive examples partially corrects the miscalibration but does not eliminate it.

Source: Flaws