What stops AI from discovering science without human help?

The paper argues that agentic AI scientists, LLMs "deployed with the scaffolding of retrieval, tool use, critique, and memory," already work as co-scientists but are not built for autonomous discovery in the natural sciences. The gap is structural: the shortcomings are "inherent to the training and deployment strategy," not a matter of "scale or tooling." It names four challenges: the McNamara fallacy in problem selection; training corpora that omit "tacit procedural and failure knowledge of laboratory practice"; preference optimization that "compresses output diversity toward consensus"; and benchmarks that "measure single-turn prediction accuracy" without feedback from physical experiments. Verification is the hinge. A proof assistant "returns a detailed error trace within seconds," while a synthesis attempt "can take months."

Each challenge has its own mechanism. AI techniques "rely on numerical signals such as losses, rewards, or fitness values," which pulls research toward what can be quantified. Hao et al. [2026] found, across 41.3 million papers, that AI-augmented scientists publish 3.02 times more papers and receive 4.84 times more citations, while AI adoption shrinks the collective volume of topics studied by 4.63%. Tacit knowledge is missing because it is "irreducibly nonpropositional" in Polanyi's sense, so it cannot be written down for a model to learn from. Diversity compression is the one the paper tests directly. Its "hypothesis hivemind" experiment reports that 12 frontier models from four providers converge semantically on hypothesis generation in two natural-science domains. Critique and re-ranking agents "select among samples already drawn from the base model," reordering them "without adding weight to regions that post-training has thinned."

The nearest notes bracket this claim. Where does AI assistance become unreliable in research? places the break at research stages; this paper places it in what training data and feedback loops contain. That is why it says human involvement will recede only where those gaps are addressed, and that "these require different design commitments rather than more capability of the current kind." It reaches the co-scientist preference of Can human-AI research teams improve faster than autonomous AI systems? on different grounds: corpus omission and benchmark validity, not safety. It also qualifies Can decentralized teams outperform central planners in long-running science?, because failures shared within one run cannot supply laboratory failures that never reached the training data. And it scopes its own claim against Can autonomous research pipelines discover AI architectures that AutoML cannot?: "we do not dispute claims of autonomy in such domains," meaning those where verification is fast.

The excerpt does not establish much of the proposed remedy. It stops before Section 8 and the conclusion, so the four recommendations (simulations as training verifiers, a persistent mutable epistemic state, a centralized preregistration repository for AI-generated hypotheses, and application driven by scientific need) appear only in the abstract, with no design or evaluation. The hivemind result appears only as a contribution statement, with no convergence measures. The evidence for human-AI collaboration is a single controlled study, Bianchi et al. [2026], which the paper reports found quality "declining under full automation." The implication is bounded. The gaps look structural to current training and deployment strategies, so the co-scientist model is the right default for now. The excerpt does not show they are permanent, and the paper itself says it does not claim that AI scientists are "impossible."