Can machines learn to predict which research ideas will work?

A fine-tuned GPT-4.1 combined with a paper-retrieval agent predicts which of two unimplemented AI research ideas will perform better across a set of benchmarks, reaching 77 percent accuracy on the full test set of 1,585 human-verified pairs. The authors report that on a 45-pair NLP subset the system beats a human baseline built from 25 experts, where each human prediction is an ensemble of five experts who together spent over 45 minutes: 64.4 percent against 48.9 percent. Off-the-shelf frontier models, including o3 and Claude 3.7 Sonnet, "perform no better than random guessing" even when given the same retrieval agent. The claim is that the outcome of an idea can be forecast from its description and the existing literature before anyone runs the experiment, and that the capability is present but has to be elicited.

The mechanism has two parts. The retrieval agent cycles through query generation, paper search, summarization and relevance filtering. It searches for ideas that offer indirect or transferable insight and for the sub-components of the novel idea, because an identical comparison rarely exists in the literature. The authors also download full PDFs rather than reuse abstracts, since abstracts "bias the model towards favoring old ideas", and the excerpt reports that this change alone lifts accuracy from 38.8 to 53.0 percent (Table 3). The second part is fine-tuning on 6,000 historical idea pairs with outcome labels, so the model learns from a record of what worked. The excerpt says human researchers acquire this intuition only through substantial experience, and the authors' bet is that a model can absorb the same record more efficiently.

The paper's motivation sits beside the ideation-execution gap, where ideas rated novel at ideation fall after expert execution. This excerpt addresses a different judgment. It sets aside novelty and excitement, which it argues peer-review style scores can reward for "fancy" but ineffective ideas, and targets empirical effectiveness instead. That makes it a candidate pre-execution filter for the gap described in Do LLM research ideas actually hold up when experts try to execute them?. The chance-level result for off-the-shelf models agrees with Can language models reliably judge their own candidate quality?, but the fine-tuned system suggests the value estimate can be learned directly from outcome labels. The excerpt never tests a surrogate, so this is a difference of approach, not a head-to-head result. The retrieval-plus-fine-tuning recipe resembles Can retrieval-augmented language models forecast like human experts?, with a larger margin over humans here, on a different task.

The excerpt does not establish that the system would hold up on research it has not seen in kind. Its labels are binary and aggregated: a pair counts as a win when it wins on more benchmarks, so the size of a win is lost. The expert comparison covers 45 NLP pairs, and the excerpt gives no expert baseline for the full test set or for other domains. The robustness tests address idea recency and complexity, yet the authors concede "we cannot rule out the possible reliance on spurious features" and describe the system as a black box. The 63.6 percent result on 35 unpublished ideas comes from a small set, so its error is wide, and it suggests generalization without settling it. The implication is modest: such a system can plausibly rank candidate ideas before anyone runs them, which is the prioritization the authors describe. The excerpt does not license treating its predictions as a substitute for the experiments that supply ground truth.