Did conference reviewers prefer AI reviews over human ones?

At AAAI-26, every paper that entered the full review phase received one clearly labeled AI review, and the pilot's authors argue that state-of-the-art AI "can already make meaningful contributions to scientific peer review at conference scale." The excerpt covers 22,977 full-review papers, reviewed in "less than a day," with the AI review added in Phase 1 alongside at least two human reviews and carrying no scores or recommendations. The evidence of value is a survey in which participants "not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions."

The authors credit system design, not off-the-shelf models: "simply asking off-the-shelf LLMs to review papers does not lead to high-quality reviews." Their multi-stage pipeline (story, presentation, evaluations, correctness, significance) gives each stage its own prompt plus all earlier outputs, adds a Python interpreter for math and code checks, and ends with a self-critique revision. On a benchmark the authors introduce, the system "substantially outperforms a simple LLM-generated review baseline" at detecting scientific weaknesses. The discussion also records the costs participants raised: "Excessive Verbosity and Cognitive Overload," "Factual Errors and Misreadings," and "Shallow Contextual and Domain Understanding."

Against the nearest notes, the pilot differs from each in a useful way. Can inference scaling help reviewers catch errors humans miss? measures flaws caught; this excerpt measures what participants preferred, so the two are complementary. The pilot is also the institutional opposite of the reviewer-restriction study in Does banning LLM use in peer review change review outcomes?: ICML tested how reviewers behave under bans and limits, while AAAI-26 put AI reviews into the process as an official input. The taxonomy in Can human review keep pace with AI-accelerated research generation? would predict this move, though the excerpt's trigger is submission volume rather than generation speed. The preference result also needs the caution in Does polished writing actually signal better quality work?: a reviewer who finds an AI review more accurate may be responding to its fluency, and the excerpt does not separate the two.

The excerpt does not give the size of the preference. It reports the direction of the survey result but no response count, preference share or statistical test, and the survey was optional. The system and benchmark are the authors' own, so the favorable comparisons are builder reports. What the evidence supports is feasibility at conference scale and a favorable but unquantified reception. It does not show that AI reviews changed decisions or improved papers, and the technical-accuracy preference should be read alongside the factual-error themes in the same discussion.