Why do single-turn RL methods fail to generalize to multi-turn tasks?
This explores why RL tuned for one-shot answers struggles when a task unfolds over many turns — and the corpus suggests the failure isn't about turn count itself, but about credit assignment, context budgeting, and diversity collapse.
This explores why RL tuned for one-shot answers struggles when a task unfolds over many turns. The first surprise in the corpus is that the premise has a big exception: RL *can* scale to long, stateful, multi-step work. A modified DAPO recipe doubled software-engineering performance (20% → 39%) in environments with delayed rewards and complex feedback, exactly the conditions a single-turn MDP assumption ignores Can reinforcement learning scale beyond single-turn language tasks?. So the better question is: what specifically breaks when you naively port single-turn methods over — and what had to be fixed to make them work?
The clearest culprit is the reward signal. Single-turn RL leans hard on clean, verifiable, binary rewards, and that's exactly where it shines — gains can jump from under 1% to over 70% when the answer is checkable, while fuzzy judgment-based signals barely move the needle Why does RL succeed more on some tasks than others?. Multi-turn tasks rarely hand you a crisp end-of-turn verdict; reward is delayed and diffuse across a trajectory. Worse, binary correctness rewards quietly teach overconfidence, since they never punish a confident wrong answer — a pathology that compounds across turns and is only fixed by adding a calibration term like the Brier score Does binary reward training hurt model calibration?.
The second culprit is context economics, which simply doesn't exist in single-turn framings. Letting a model reason freely is fine for one answer, but in an iterative loop that reasoning eats the context budget needed for later retrieval rounds, eroding the agent's ability to absorb new evidence. Capping reasoning *per turn* — not just overall — is what preserves multi-turn search quality Does limiting reasoning per turn improve multi-turn search quality?. Relatedly, not all turns are the same kind of work: RL training tends to move through a two-phase dynamic where execution correctness is the early bottleneck and strategic planning becomes the later one, so a method that only optimizes 'get this step right' never learns the planning that multi-turn success actually hinges on Does RL training follow a predictable two-phase learning sequence?.
The third culprit is diversity collapse. RL pulls policies toward a narrow band of reward-maximizing behavior through entropy collapse — and this isn't unique to reasoning; search agents show the very same squeeze, with SFT on diverse demonstrations needed to keep exploration breadth alive Does reinforcement learning squeeze exploration diversity in search agents?. In a single turn, narrowing to the best move is mostly fine. Across many turns, exploration is the whole game, so the same mechanism that helps short tasks actively sabotages long ones. Training order can even be tuned to manage this, since structured and open-ended domains push entropy in opposite directions Does training order reshape how models handle different task types?.
What ties this together is that the fixes for multi-turn are all about treating the trajectory as the unit, not the turn. SkillRL, for instance, processes successes and failures asymmetrically — keeping wins as concrete demonstrations and distilling losses into abstract lessons — which is precisely the kind of credit assignment a single-turn objective can't express Should successful and failed episodes be processed differently?. The thing you didn't expect to learn: single-turn RL doesn't fail because tasks got longer, but because its assumptions about *where reward lives, how context is spent, and how much exploration to keep* all quietly stop holding the moment a task has a second turn.
Sources 8 notes
Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. This demonstrates RL works in stateful multi-step environments with delayed rewards and complex feedback, beyond theoretical single-turn MDPs.
Binary verifiable rewards enable dramatic RL gains (0.15% to 73.98%), while judgment-based evaluation yields modest improvements (55% reduction). Clear reward signals unlock suppressed capabilities; fuzzy signals barely move the needle.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Unrestricted reasoning within single search turns consumes context needed for subsequent retrieval rounds, degrading the agent's ability to incorporate new evidence. Setting per-turn reasoning budgets, not just overall time limits, prevents this context erosion and maintains search quality across iterations.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.