Rules on AI use in peer review often barely moved scores, partly because many reviewers simply broke them.

This explores why changing the rules of peer review, especially rules about whether reviewers may use AI, so often leaves the actual scores unchanged. The clearest evidence comes from a randomized experiment at ICML 2026. Some reviewers were told not to use LLMs at all and others were allowed limited use. Paper scores, accept/reject decisions and reviewer confidence came out almost the same in both groups. The detail that explains much of this is that large numbers of reviewers broke whichever rule they were given Does banning LLM use in peer review change review outcomes?. A policy only on paper produces two groups that behave much more alike than their instructions suggest. The first reason, then, is simple: the policy was often never followed.

The second reason is less obvious. Scores may mostly reflect the paper being judged, and the review process may matter less. A study of more than 125,000 reviews found what looked like LLM-assisted reviewers favoring LLM-written papers. That apparent favoritism disappeared once paper quality was taken into account. LLM-written papers tended to be weaker, and LLM-assisted reviewers were generally more lenient toward weaker work. The two effects together created a pattern that looked meaningful but wasn't Do LLM reviewers actually favor LLM-written papers?. When paper quality drives most of the variation in scores, an effect that seems to come from the process can turn out to be a side effect of quality. A policy aimed at the process then has little left to change.

Third, the pressures that actually shape reviews are mostly things these policies don't touch. One model shows a feedback loop. Rising submission numbers overload unpaid reviewers, so journals recruit less qualified reviewers or pile more work on the existing ones. Accuracy drops, which encourages authors to submit more speculative papers, which raises submissions further Does peer review quality collapse under submission overload?. That model predicts falling accuracy, not a shift in average scores, and its real-world strength hasn't been measured yet. A position paper on AI conferences makes a related point: authors, reviewers and venues all share responsibility for review failures. It argues that fixes have to change incentives, for example by letting authors rate a review's quality before they see its verdict and by rewarding thorough reviewers Can two-stage review and badges fix AI conference peer review?. A rule about one tool used by one of those three parties leaves most of this in place. A survey of 230 publications describes the wider AI research-and-review ecosystem as a coupled arms race in which each defense prompts an evasion Does AI create a coupled arms race in research production and review?. The ICML rule-breaking fits that picture.

The surprise is that an intervention can succeed and still leave scores alone. At ICLR 2025, AI agents gave reviewers optional feedback on their draft reviews. Twenty-seven percent of reviewers revised their reviews, and blinded raters judged the revised reviews more informative and clearer Can LLM feedback help peer reviewers improve their own reviews?. The improvement showed up in the written review, not in the verdict. That suggests asking a different question: whether a policy makes reviews more useful to authors, rather than whether it moves scores. The review text appears much easier to influence than the final score. One caveat: none of these notes directly test why scores are hard to move. The explanations above are drawn from experiments, a model and position papers that each address part of the question.