At ICML, banning reviewers from AI or allowing limited use produced nearly the same scores, and a substantial share broke their assigned rule.

This explores whether rules restricting reviewers' use of LLMs work, measured by compliance, enforcement and their effect on decisions. The corpus has no studies of journals, but AI conferences have tested these rules in practice, and the findings are surprising. The most direct evidence comes from a randomized experiment at ICML 2026. Some reviewers were told not to use LLMs at all, and others were allowed limited use. The two groups ended up with nearly the same paper scores, accept/reject decisions and reviewer confidence Does banning LLM use in peer review change review outcomes?. In addition, a substantial share of reviewers broke whichever rule they were assigned. So the policy did not change what reviewers did, and their behavior did not change the outcomes.

Enforcement is the weak link. ICML planted hidden instructions in submission PDFs, which act like a watermark: if a reviewer pastes the paper into a chatbot, a telltale phrase can appear in the review. This flagged 795 reviews, about 1% of the total, and led to 497 desk rejections. The chairs themselves say the method mostly catches careless users and misses anyone who removed the hidden instruction or rewrote the output How many peer reviewers secretly used LLMs despite the ban?. For comparison, an earlier analysis of ICLR, NeurIPS, CoRL and EMNLP reviews estimated that 6.5–16.9% of reviews showed substantial LLM modification. It also found the highest rates among low-confidence, last-minute and less-engaged reviewers How much peer review text shows signs of LLM modification?. The two numbers come from different venues and methods, so they aren't directly comparable. Still, the gap suggests that what gets caught is a small, visible fraction of actual use. It also points to the real problem: rushed reviewing, which an LLM ban alone doesn't fix.

ICLR 2026 took a more practical approach. Detector flags went to area chairs as one piece of evidence among several, not as automatic verdicts. Hard penalties were reserved for something that can actually be checked: fabricated references. Papers with confirmed hallucinated citations were desk-rejected How can conferences detect and handle LLM misuse in peer review?. This matters because detection is unreliable. Even ML experts can't consistently tell LLM-written text from human text Can readers tell LLM abstracts from human ones?. A rule against a fabricated reference can be enforced, while a rule against LLM-written text mostly can't.

The main concern behind these rules is bias, specifically that LLM reviewers favor LLM-written papers. Here the evidence is mixed. In simulations, LLM reviewers inflated scores for LLM-authored papers and marked down human papers that took a critical stance Do LLM reviewers favor papers written by other LLMs?. But across more than 125,000 real reviews, the apparent favoritism disappeared once paper quality was held constant. LLM-assisted reviewers were simply more lenient toward weaker papers in general, and LLM-written papers tended to be among the weaker ones Do LLM reviewers actually favor LLM-written papers?. Other risks are real, though. LLM judges can be fooled by fake authority signals and polished formatting Can LLM judges be fooled by fake credentials and formatting?, and polished prose no longer reliably signals a strong paper Does LLM writing assistance change how scientists publish?.

The less obvious conclusion is that the more promising policies shape how LLMs are used rather than trying to ban them. At ICLR 2025, an optional LLM tool gave reviewers feedback on their draft reviews. In a randomized trial, 27% of reviewers revised their reviews, and blinded raters judged the revisions more specific and informative Can LLM feedback help peer reviewers improve their own reviews?. A structured LLM pipeline for judging a paper's novelty agreed with human reviewers' reasoning 86% of the time Can structured pipelines make LLM novelty assessment reliable?. Evidence so far suggests bans are hard to enforce and change little. The questions that remain open are where in the review process LLMs belong and which failures, like fabricated references, are worth policing.