When AI systems propose scientific hypotheses, how would you know which one is better when each keeps its own scorecard?

This explores how researchers compare AI systems that propose scientific hypotheses, and the honest answer from this collection is that there's no shared scoreboard yet. The corpus has several creative ways to judge hypotheses, but almost none that put rival systems side by side on the same test. The gap is itself the finding. The Virtuous Machines framework argues that autonomous science needs four abilities: generating hypotheses, designing experiments, analyzing data and correcting itself. It also argues that standard LLM benchmarks don't reliably measure any of them What capabilities do AI systems need for autonomous science?.

What exists instead are scoring methods built inside individual systems. Google's Co-Scientist runs a tournament: hypotheses debate each other, the winners get refined, and each idea earns an Elo rating, the same ranking system used in chess. The builders report that Elo scores rise as the system gets more compute Does more thinking time improve AI-generated research hypotheses?. The surprising part is that the Elo scores come from the system's own judging. So the tournament measures which hypothesis wins inside the system, not whether that system beats another one. It's a yardstick the system made for itself.

The most convincing test in the corpus is a blind-answer test rather than a leaderboard. Researchers gave the AI a question their lab had already answered by experiment but hadn't published. The system's top-ranked hypothesis matched the confirmed mechanism, in which a kind of genetic parasite hijacks the tails of viruses that infect bacteria Can AI systems generate hypotheses that match unpublished experimental discoveries?. Any system could in principle be run against the same unpublished results, and the model can't have memorized the answer. The weakness is that each test takes a real lab with a real secret, so it doesn't scale.

If you look sideways at how AI outputs get judged in general, you can see what a fair comparison might need. Using a single LLM as the judge turns out to be unstable: on complex tasks its verdicts shifted 31% of the time. An agent that goes out and gathers evidence before ruling brought that down to 0.27% Can agents evaluate AI outputs more reliably than language models?. An agentic reviewer that checks proofs and experiments line by line caught flaws in conference papers that human reviewers missed Can inference scaling help reviewers catch errors humans miss?. Spark-to-Paper takes another route. It keeps the model's judgment separate from checks that can be run mechanically, and it requires stating in advance what evidence would count, before any results come in Can separating judgment from verification improve research paper reliability?. That is essentially preregistration, and it's a plausible model for how hypothesis benchmarks could avoid grading on the curve.

The lesson you might not expect: in this field, judging hypotheses is about as hard as generating them. Until there's a judge that both rival systems trust, plus a supply of questions that are already answered but not yet published, claims that one system beats another mostly come from the people who built it. The corpus doesn't yet contain a true cross-system benchmark for hypothesis generation. If that's what you're after, this is a known open problem, not a gap in your searching.