When peer reviewers earn badges for good work, could they start padding reviews so they look thorough without being thorough?
Do reviewer reward badges risk encouraging lenient or superficial reviews?
This asks whether rewarding peer reviewers with badges could backfire by pushing them toward reviews that are kind to authors, or that look thorough without being thorough. The corpus has no study of badges themselves, so the answer draws on related evidence about how rewards and ratings get gamed.
This explores whether rewarding reviewers with badges could backfire, so that reviewers go easy on authors or write reviews that only look thorough. No paper in the collection tests reviewer badges directly. What it does have is a lot of evidence about how rewards distort the behavior they measure, and that evidence points to a specific risk. The worry isn't mainly leniency. It's reviews that are padded to look thorough.
The badge proposal itself already guards against the obvious leniency problem. In Can two-stage review and badges fix AI conference peer review?, authors rate the quality of a review *before* they see whether their paper was accepted. That ordering matters. If authors rated reviews after learning the verdict, the reviewers who accepted their papers would collect the rewards, and badges would quietly pay reviewers to be generous. Hiding the verdict breaks that link. The same paper reports a measured bias in which longer reviews tend to be rated higher. That is where the danger lies: a badge for "thoroughness" judged by authors could end up rewarding length.
Research on AI training has seen this failure before. Can rubrics and dense rewards work together without hacking? found that turning a quality checklist into a score that models try to maximize invites gaming. Using the checklist as a pass/fail gate does not. Applied to reviewing, a badge that certifies a review met a quality bar is harder to game than a points system that rewards *more* of something. Ratings also drift socially. Do online ratings actually reflect independent customer opinions? shows that earlier ratings shape later ones and that the effect compounds, so a reputation system for reviewers could reinforce early impressions more than it measures actual quality.
Who or what judges the badge matters too. If AI tools end up checking review quality at scale, Can AI systems safely replace human peer reviewers? is a warning. Simply rewording a paper raised AI review scores by almost half a point with no change to the science. A badge awarded by an automated judge would probably be just as easy to game through surface polish. There is also a quieter point about leniency. Do LLM reviewers actually favor LLM-written papers? finds that LLM-assisted reviewers are generally easier on weaker papers. Reviewers who lean on AI to produce long, badge-worthy reviews quickly could therefore drift toward leniency without anyone setting out to be lenient.
The most useful contrast is a lever that worked. In Can LLM feedback help peer reviewers improve their own reviews?, reviewers got specific feedback on their draft reviews instead of a reward. Over a quarter of them revised, and blinded raters judged the revised reviews more informative, not merely longer. Taken together, the collection suggests that badges are most likely to backfire when they reward volume or are awarded by easily fooled judges. They are safer when they work as a quality bar, assessed before the verdict is known, alongside feedback that shows reviewers what a good review looks like.
Sources 6 notes
Authors, reviewers, and venues all contribute to peer review failures at major AI conferences. A proposed two-stage system lets authors rate review quality before seeing verdicts, and a badge system rewards reviewer thoroughness, targeting measured biases like rating-length correlation.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Moe and Trusov decomposed ratings into baseline quality, social-dynamics influence, and error, finding that prior ratings meaningfully affect subsequent ones. These effects have both immediate sales impact and long-term compounding effects through future ratings, though high opinion variance can eventually dampen the distortion.
AI systems show a hivemind effect, agreeing more with each other than humans do across papers. Zero-shot rewrites of paper text raise AI scores by 0.45 points without improving scientific content, demonstrating trivial gameability at scale.
Across 125,000+ reviews, the apparent favoritism of LLM-assisted reviewers toward LLM papers disappears once paper quality is held constant. LLM papers cluster among weaker submissions, creating a spurious interaction driven by LLM reviewers' general leniency toward lower-quality work.
Show all 6 sources
A randomized trial at ICLR 2025 found that optional, gated feedback from Claude-based agents led over a quarter of reviewers to update their reviews, incorporating suggestions that blinded raters judged as more informative and clear.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Stop Automating Peer Review Without Rigorous Evaluation
- Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
- Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards
- AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
- Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?
- Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review
- How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
- LLM-REVal: Can We Trust LLM Reviewers Yet?