What capabilities do AI systems need for autonomous science?

The Virtuous Machines paper proposes a capability checklist for what it would mean for an AI system to conduct autonomous scientific research — not assist human researchers, but operate as an independent scientific agent:

1. Hypothesis generation — formulating testable claims from prior knowledge and anomalies
2. Experimental design — specifying procedures that could confirm or falsify the hypothesis
3. Data analysis — drawing valid inferences from experimental results
4. Iterative self-correction — revising hypotheses and experimental designs based on failed predictions

Current LLM benchmarks test capabilities that are adjacent to these (question answering, code generation, reasoning) but do not directly evaluate any of the four. A model that excels at standard benchmarks may still be unable to design an experiment that could falsify its own hypothesis.

The iterative self-correction component is the most demanding. It requires the system to recognize when its current beliefs should be revised — which runs directly into the self-revision degradation problem: Does self-revision actually improve reasoning in language models? and Does a model improve by arguing with itself?. A system that self-revises under academic conditions may converge on false hypotheses via the same mechanism.

This connects to Does reasoning fine-tuning make models worse at declining to answer? — the very training regime that improves hypothesis generation may degrade the epistemic humility that self-correction requires.

The co-improvement alternative reframes these four capabilities from an autonomy checklist to a collaboration skill inventory. Rather than waiting for autonomous capabilities that reliably self-correct, human-AI co-research targets the same paradigm shifts while preserving human oversight. Historical evidence: every major AI paradigm shift required a data-method tandem (ImageNet+AlexNet, web data+transformers, instruction data+RLHF, verifiable tasks+RLVR) — each discovered through significant human effort. Co-improvement accelerates the search for unknown next paradigm shifts while providing the external verification that pure self-improvement cannot. See Can human-AI research teams improve faster than autonomous AI systems?.