Scaling Automatic Research Agents via World Models

Abstract: Automating empirical research is a long-standing direction of AI. Recent automatic research (AutoResearch) agents bring this goal within reach, as modern LLMs show the capability to independently implement solutions and learn from the execution outcomes. Behind these gains, post-training (especially RL) plays a central role. In this paper, we identify a fundamental tension when scaling RL for these agents: the two components of every AutoResearch trajectory (agent generation and environment execution) scale in very different manners, since all generation shares compute through batching, while each execution occupies its exclusive sandbox and real machine time. As a result, the environment execution dominates the training cost and becomes the bottleneck as trajectories grow. To resolve this tension, we propose World Model RL (WMRL), which replaces environment execution with a world model to remove this bottleneck. Additionally, the world model can be imperfect, as its rewards are corrupted by bias and noise. Therefore, we further equip WMRL with two mitigations, Online Debiasing and Inverse-Variance Denoising, which offset the bias and suppress the noise respectively. Theoretically, we prove that both mitigations of WMRL strictly improve the convergence guarantee.

Introduction. An Automatic Research (AutoResearch) agent is a language model that independently conducts empirical research [1, 2, 3]. Given a research question, it formulates an idea, implements the experiment, analyzes the outcome, and iterates [4, 5, 6]. Such agents have already proven capable across various domains. In the natural sciences, for example, they design chemical syntheses and propose reaction conditions that are validated by wet-lab experiments later [7, 8, 9]; in machine learning and data science, they explore real datasets and build training pipelines that outperform human experts [10, 11, 12]. In an AutoResearch task, the agent iteratively generates and executes solutions. These interactions form trajectories, and the execution outcomes provide rewards. With both trajectories and rewards in place, AutoResearch is a natural fit for reinforcement learning (RL) [13, 14, 15, 16], a promising direction to further improve these capabilities.

Discussion / Conclusion. This work scales RL for AutoResearch agents by replacing the expensive environment execution with a world model and correcting its bias and noise through a small anchored stream of real execution. The two corrections turn the permanent error floor of world model training into a contracting term and reduce the variance below either reward stream alone, and they cut the training compute by three to four times while matching or exceeding full real-execution RL at two scales. The transfer to VLA post-training further suggests a general path for scaling RL wherever execution, not generation, is the bottleneck.