Can automated review scale AI paper evaluation reliably?

The AI Scientist's authors argue that the value of full automation lies not just in generating discoveries but in producing a checkable artifact: a complete scientific paper that can be evaluated the same way human-written papers are. The excerpt states the system "generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation," and does so "at a meager cost of less than $15 per paper." The authors say this scale is possible only because "we design and validate an automated reviewer," which lets them "scale the evaluation of our papers beyond manual inspection."

Their reasoning for writing papers at all, rather than stopping at a discovery the way FunSearch and GNoME do, rests on three points: papers are "a highly interpretable method for humans to benefit from what has been learned," reviewing within existing conference frameworks "enables us to standardize evaluation," and the paper format can "flexibly describe any type of scientific study and discovery" where other formats are "locked into a certain kind of data." Idea generation itself uses chain-of-thought and self-reflection to grow an archive "conditional on the existing archive, which can include the numerical review scores from completed previous ideas" — so the reviewer's output doesn't just gate publication, it feeds back into what the system tries next, and the authors note the process "can be repeated to iteratively develop ideas in an open-ended fashion." They also flag, as future work, that the system "could follow up on its best ideas or even perform research directly on its own code in a self-referential manner" — notable because, as they disclose, "significant portions of the code for this project were written by Aider," an AI coding assistant.

This sits upstream of Can one AI system complete a full research cycle end-to-end? and Can AI systems generate research papers that pass peer review?, which test the same lineage of system against real conference review; this earlier paper is where the authors explain why they built a reviewer at all, rather than report it as a result. Does Sakana's AI Scientist deliver autonomous research without human help? is an independent check of the premise this excerpt assumes: that an automated reviewer scaling evaluation is the same as evaluation being reliable. And where What stops AI from discovering science without human help? argues design-level gaps block autonomous discovery regardless of scale, this paper's own authors raise a version of the same doubt as an open question rather than a settled limitation.

The excerpt gives no numbers for the Automated Reviewer's accuracy — it only states that LLMs are "capable of producing reasonably accurate reviews, achieving results comparable to humans across various metrics," without the metrics themselves — and the Limitations section is cut off mid-sentence before any specifics appear. Nor does it report how many papers were generated in each of the three subfields or how many cleared review. What it does establish is the authors' own rationale: automated review is the mechanism they are relying on to decouple research throughput from human reviewer time, and they explicitly leave open "whether such systems can ultimately propose genuinely paradigm-shifting ideas" rather than incremental ones. The implication, at the strength this excerpt supports, is that the system's claimed speed and cost advantages are contingent on the reviewer being trustworthy — a dependency the paper states but does not, in this excerpt, test.