Why do reward models ignore what question was asked?
Reward models score responses based on quality signals that persist even when prompts change. This explores whether AI grading systems actually evaluate relevance to the question or just response-level patterns.
Post angle for Medium — the evaluation infrastructure behind AI alignment has a fundamental flaw
The hook: Your AI's grading system is ignoring the question. When researchers swapped prompts while keeping responses the same, reward model preference scores barely changed. The system that's supposed to ensure AI gives good answers to your questions is actually just evaluating whether the response sounds good — regardless of what was asked.
The mechanism: Since Do reward models actually consider what the prompt asks?, standard Bradley-Terry training lets reward models learn to distinguish good from bad responses without ever needing to check whether the response matches the prompt. Responses dominate the reward signal. This means RLHF — the dominant approach to making AI helpful and safe — is optimizing against phantom quality signals.
The four biases it enables: Since Can counterfactual invariance eliminate reward hacking biases?, prompt-insensitivity creates an opening for four distinct biases — length bias (longer = better), sycophancy (agreement = better), concept shortcuts, and demographic discrimination. All stem from spurious correlations that the model treats as genuine quality signals because it isn't checking whether the response actually addresses the prompt.
Three converging fixes from independent teams:
- Decompose the reward — split into prompt-free and prompt-related components, then prioritize training on samples where the prompt matters
- Apply counterfactual invariance — ensure rewards stay constant when irrelevant features change
- Let the evaluator think — since Can reward models benefit from reasoning before scoring?, three teams independently discover that reward modeling is a reasoning task; CoT before scoring enables adaptive evaluation
The broader frame: The bottleneck on AI improvement isn't just model capability — it's evaluator capability. The system we use to tell AI what's good has been quietly ignoring half the input. Fixing this requires treating reward modeling not as a classification task but as a reasoning task.
Source: Reward Models
Related concepts in this collection
-
Do reward models actually consider what the prompt asks?
Exploring whether standard reward models evaluate responses based on prompt context or just response quality alone. This matters because if models ignore prompts, they'll fail to align with what users actually want.
the core mechanism
-
Can counterfactual invariance eliminate reward hacking biases?
Does forcing reward models to remain consistent under irrelevant changes remove the spurious correlations that cause length bias, sycophancy, concept bias, and discrimination? This matters because standard training bakes these biases in permanently.
the bias taxonomy
-
Can reward models benefit from reasoning before scoring?
Does allowing evaluator models to generate reasoning traces before producing reward scores improve alignment and enable adaptive compute allocation? Three independent research teams converged on this insight simultaneously.
the reasoning-based alternative
-
Can LLM judges be fooled by fake credentials and formatting?
Explores whether language models evaluating text fall for authority signals and visual presentation unrelated to actual content quality, and whether these weaknesses can be exploited without deep model knowledge.
evaluation infrastructure is broadly vulnerable
-
Can model explanations help humans predict what models actually do?
Do explanations that sound plausible to humans actually help them forecast model behavior on new cases? Understanding this gap matters because RLHF optimizes for plausible explanations, not predictive ones.
RLHF optimizes the wrong signal in explanations too
Click a node to walk · click center to open · click Open full network for a force-directed map
Original note title
the reward models blind spot — why your ais grading system ignores the question