Can structured pipelines make LLM novelty assessment reliable?

Novelty assessment is one of the most problematic aspects of peer review. Overwhelmed reviewers resort to vague feedback like "not novel enough" without justification, and reviewers outside their specific expertise either reject conservatively or miss incremental work. This paper proposes a structured pipeline that decomposes the task into three stages: (1) extract claims from the submission, (2) retrieve and synthesize related work, (3) compare claimed novelty against a comprehensive literature analysis with cited evidence.

Evaluated on 182 ICLR 2025 submissions with human-annotated novelty assessments, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions — substantially outperforming existing LLM baselines. The method produces detailed, literature-aware analyses that improve consistency over ad hoc reviewer judgments.

The key architectural insight is that novelty assessment is not a single judgment but a decomposable process: claim verification is separable from literature awareness is separable from comparative reasoning. When LLMs attempt novelty assessment as a single holistic judgment, they perform poorly. When the task is decomposed into subtasks that each play to LLM strengths (extraction, retrieval, structured comparison), performance approaches human levels.

This connects to the broader pattern that since Can LLMs generate more novel ideas than human experts?, structured decomposition may be the path to closing the evaluation gap — not by making LLMs better evaluators holistically, but by converting evaluation into a sequence of more tractable subtasks. It also resonates with the finding that since Why do LLMs generate more novel research ideas than experts?, the evaluation side can be partially addressed through pipeline architecture rather than model capability.

The implication for AI-assisted writing is that the review bottleneck — which shapes what gets published and therefore what gets written — is restructurable through AI. Not AI replacing reviewers, but AI making the reviewer's novelty assessment more rigorous and evidence-based than most human reviewers achieve under time pressure.

Source: Co Writing Collaboration Paper: Beyond "Not Novel Enough"