Can RL training run while generation continues without waiting?

Synchronous RL systems for large reasoning models alternate strictly between generation and training, ensuring models always train on their latest outputs. But this design creates severe inefficiency: the generation step must wait for the longest output in a batch, and LRMs produce wildly varying output lengths — tens of thousands of thinking tokens for some prompts, few hundred for others.

AReaL fundamentally resolves this by making RL training fully asynchronous. Each rollout worker continuously generates outputs without waiting (streaming generation). Trainer workers run parallel model updates whenever a training batch is available. After each update, model weights are synchronized to rollout workers. The critical consequence: each training batch may contain samples generated by different model versions.

To make this work, AReaL incorporates a modified PPO objective that can leverage samples from much older model versions without performance loss. This is a significant departure from the conventional wisdom that on-policy data (from the latest model) is essential for RL training quality. Prior semi-asynchronous systems limited version staleness to one or two steps and still used batched generation from a single version.

This is an infrastructure insight with capability implications. Since Can reinforcement learning scale beyond single-turn language tasks?, and since multi-turn RL generates orders of magnitude more tokens than single-turn, the efficiency gains from asynchronous training are not merely convenient but potentially necessary for scaling RL to interactive environments.

The broader principle: when the generation-training bottleneck is resolved, the practical frontier of what RL can train on expands considerably — from single-turn math to multi-turn interactive tasks that require long context and many steps.

Extension (OpenClaw-RL, 2026): The OpenClaw-RL framework pushes async decoupling further — from the 2-loop generation/training split to a 4-loop architecture where policy serving, rollout collection, PRM judging, and policy training run as four independent loops with zero blocking dependencies. This is built on slime and adds session-aware multi-turn tracking, graceful weight updates, flexible PRM support, and large-scale environment parallelization. The crucial extension is conceptual, not just architectural: AReaL assumes batch data collection even while async; OpenClaw-RL makes the serving loop itself the data source. The same infrastructure that responds to users in production simultaneously generates the training signal. Personal agents improve simply by being used. The async decoupling pattern has now generalized from "compute-efficient training" to "continuous learning from live deployment" — where the serving/training boundary dissolves entirely. See Can agent deployment itself generate training signals automatically? for the signal-recovery framing that makes this practical.

Source: Reinforcement Learning