When an AI reviewer grades a paper, should a polished, confident tone count for as much as the science being right?
Should rhetorical polish in AI reviews be separated from actual technical accuracy?
This explores whether, when AI writes or scores peer reviews, how well-written and confident a review sounds should be judged separately from whether its technical claims are right. It also asks what happens when the two get blended together.
This explores whether a review's polish (fluent prose, confident tone, tidy formatting) should be scored separately from whether it gets the science right. The corpus says yes, and gives a sharper reason than you might expect. Polish doesn't just make bad work look good. It distorts judgment on both sides of the review table at once. AI evaluators score responses higher when they include fake references or rich formatting, whatever the content quality, and you don't need any access to the model to exploit this Can LLM judges be tricked without accessing their internals?. In peer review specifically, simply rewriting a paper's text raised AI reviewer scores by 0.45 points without changing a single scientific claim. AI reviewers also tend to agree with each other more than human reviewers do, so the same blind spot gets repeated across the board instead of averaged out Can AI systems safely replace human peer reviewers?.
The human side isn't much safer. People across every language studied follow confidence signals rather than accuracy, so an overconfident wrong answer gets followed as readily as a right one Do users worldwide trust confident AI outputs even when wrong?. Professional-looking output borrows an old shortcut: polished work used to signal that an expert had thought hard about it. Generative AI can now produce the look without the thinking, and less experienced readers are the most exposed Does polished AI output trick audiences into trusting it?. Spotting the difference by eye doesn't work either. Across 30 studies, human detection of AI content hovers around chance Can people reliably spot content made by AI?. A fully AI-generated paper cleared an ICLR workshop's double-blind review, and its own authors later found a citation error and judged none of their three submissions ready for the main conference Can AI-generated papers pass peer review undetected?.
There's a quieter twist. AI prose often has polish without any real stance. LLMs have mastered grammar and organization but avoid the evaluative words that commit a writer to a judgment, which produces text that is coherent but makes no real argument Why does AI writing sound generic despite being grammatically correct?. For a review, that's a problem: it can read as thorough while never firmly saying what's wrong. Human editing doesn't reliably catch this. Writers changed AI-drafted paragraphs only 23% of the time, and their edits left the text about 96% the same Do writers actually edit AI-generated text before publishing?. AI assistance also shifted how readers saw the writer on all 29 traits measured, including making writers seem more confident Does AI writing assistance change how readers perceive the writer?.
The most useful finding is how to separate the two in practice. The fix that works is structural, not a better-trained reader. Pull the step that checks the evidence apart from the step that writes the verdict. An agentic reviewer that spends extra computing time checking proofs and experiments line by line caught math errors and flaws at top venues that human reviewers had missed Can inference scaling help reviewers catch errors humans miss?. An agent-based judge that collects evidence before scoring cut 'judge shift' (how far its verdicts drift from the human reference judgments) from 31% to 0.27%. One caution: errors in its memory module cascaded through the rest of the system Can agents evaluate AI outputs more reliably than language models?. Human institutions are trying the same separation. One proposal has authors rate how good a review is before they see the accept/reject decision, with badges rewarding reviewer thoroughness. The aim is to break measured biases such as reviews' ratings tracking their length Can two-stage review and badges fix AI conference peer review?. The pattern underneath: once checking what's true is its own explicit step, polish has much less to influence.
Sources 12 notes
Research shows LLM evaluators systematically score higher when responses include fake references or rich formatting, independent of content quality. These biases are exploitable without model access, undermining AI benchmark credibility.
AI systems show a hivemind effect, agreeing more with each other than humans do across papers. Zero-shot rewrites of paper text raise AI scores by 0.45 points without improving scientific content, demonstrating trivial gameability at scale.
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
Generative AI produces visually sophisticated outputs without underlying judgment, leveraging the historical heuristic that professional-looking work signals expert thinking. This substitution is especially risky for less experienced workers who lack domain knowledge to evaluate substance beyond form.
A 30-study systematic review found that humans cannot reliably distinguish AI-generated from human-created content across text, image, and voice modalities. Accuracy generally clusters around chance and has not kept pace with improvements in AI realism.
Show all 12 sources
Sakana AI's end-to-end system produced a paper that scored 6.33 in double-blind ICLR 2025 workshop review, meeting acceptance thresholds, but was withdrawn under pre-agreed protocol. Authors later identified a citation error and judged none of three submissions suitable for main-track publication.
AI text uses manner nouns and anaphoric references that are descriptively neutral, while human writers use status and evidential nouns that carry evaluative weight. This produces organizationally coherent but argumentatively inert prose.
Writers edited AI-generated paragraphs only 23% of the time, with edits averaging 96% similarity to the original. This means AI's opinionated and distorted voice propagates with minimal human filtering before publication.
A study of 2,939 writers and 11,091 readers found AI assistance shifted every tested dimension—29 total—toward extremism, confidence, quality, agreeableness, and perceived privilege. Distortions were statistically significant and directional, not random noise.
PAT, an agentic reviewer using test-time compute to check proofs and experiments line by line, achieves 34% better recall on math errors than zero-shot approaches and surfaced critical flaws at STOC and ICML that passed human review.
Eight-module agentic evaluation achieved 0.27% judge shift versus 31% for LLM-as-a-Judge on complex tasks. However, the memory module cascaded errors, revealing that agentic systems need error isolation mechanisms to maintain gains.
Authors, reviewers, and venues all contribute to peer review failures at major AI conferences. A proposed two-stage system lets authors rate review quality before seeing verdicts, and a badge system rewards reviewer thoroughness, targeting measured biases like rating-length correlation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Stop Automating Peer Review Without Rigorous Evaluation
- AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot
- Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
- Is it Cake or is it AI? A Systematic Review of Human Uncertainty in Distinguishing Generative Artificial Intelligence Content
- The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing
- Understanding Reader Perception Shifts upon Disclosure of AI Authorship
- Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?
- The human-authorship halo: attribution bias in literary style evaluation by humans and AI