Can GPT-4 feedback match what human reviewers catch?

The study reports that GPT-4, run on a paper's PDF through a pipeline that returns structured feedback, raises points that overlap with human reviewers' points about as often as two human reviewers overlap with each other. Across 3,096 papers in 15 Nature family journals, overlap was 30.85% for GPT-4 against an individual reviewer and 28.58% for two reviewers; across 1,709 ICLR papers it was 39.23% against 35.25%. In a prospective user study of 308 researchers from 110 US institutions, 57.4% found the feedback helpful or very helpful and 82.4% found it more beneficial than feedback from at least some human reviewers. These are the study's own measurements.

The overlap is a product of matching, not of a quality judgment. GPT-4 extracts the criticisms in each review, then a second pass pairs them and rates each pair from 5 to 10; only pairs rated 7 ("Strongly Related") or above are kept, because lower ratings "did not always align with human evaluations." Hand checks gave a matching F1 of 0.824 (recall 0.878, precision 0.777), and three annotators agreed on 89.8% of pairs. The measure is therefore agreement about which points get raised, not whether they are right. Overlap was highest for rejected ICLR papers, which the authors read as papers with "more apparent issues or flaws that both human reviewers and LLMs can consistently identify."

Against the nearest notes, the study measures breadth, not reasoning depth. Can structured pipelines make LLM novelty assessment reliable? reports 86.5% alignment with human reviewers, but on one judgment, novelty, split into stages; this excerpt uses a single prompt over the whole manuscript. Its limitations cut against Can inference scaling help reviewers catch errors humans miss?, which spends inference compute to check proofs and experiments line by line: participants say GPT-4 "struggles to provide in-depth critique of method design." The 82.4% figure is a perceived-benefit rating, the kind of signal Does polished writing actually signal better quality work? warns against reading as merit.

The excerpt does not establish accuracy. Overlap with reviewers, who also miss things, is a proxy for usefulness, not a check against errors. It also leaves open how much of each paper the model read. The abstract promises comments on "the full PDFs," but the method passes only "the initial 6,500 tokens" to a model with an 8,192-token limit. The figures disagree internally: the abstract gives 43.80% overlap for rejected ICLR papers, while the discussion gives that figure for two human reviewers and 47.09% for GPT-4 against humans, so the abstract appears to report the human-human number. The excerpt also omits how the user study measured benefit relative to human reviewers and how respondents were recruited. The supportable implication is narrower than the headline: LLM feedback can match reviewers' point coverage and be perceived as useful, which makes it a complement to review rather than a substitute for methodological critique.