Emergent Hierarchical Reasoning In LLMs Through Reinforcement Learning

Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaque. Our analysis reveals that puzzling phenomena like “aha moments”, “length-scaling” and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy, akin to the separation of high-level strategic planning from low-level procedural execution in human cognition. We uncover a dynamic evolution where the learning bottleneck shifts: initially, the process is dominated by procedural consolidation and must improve its low-level skills. The learning bottleneck then decisively shifts, with performance gains being driven by the exploration and mastery of high-level strategic planning. This insight exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically and dilute the learning signal across all tokens. To address this, we propose Hierarchy- Aware Credit Assignment (HICRA), an algorithm that concentrates optimization efforts on high-impact planning tokens. Our extensive experiments validate that HICRA significantly outperforms strong baselines, and offer deep insights into how reasoning advances through the lens of strategic exploration.

• High-level Planning Tokens: The high-level strategic moves that orchestrate the reasoning process. These tokens manifest as logical maneuvers, including deduction (e.g., “we can use the fact that”), branching (e.g., “let’s try a different approach”), and backtracing (e.g., ”but the problem mentions that”).

• Low-level Execution Tokens: The operational building blocks of a solution. These comprise concrete, low-level steps such as arithmetic calculations, variable substitutions, and the direct application of known formulas.

Our analysis across eight text-only and vision-language models confirms this hypothesis, revealing a consistently two-phase dynamic that explains the emergence of this reasoning hierarchy in LMs. We find the optimization pressure of RL is not static; instead, its learning frontier shifts. Initially, the process is constrained by procedural correctness. A single calculation error can invalidate an entire solution, creating a powerful learning signal that compels the model to first master low-level execution tokens. Once proficiency in these foundational skills is achieved, the learning bottleneck shifts to strategic planning. We find that these phases are not mutually exclusive; procedural refinement continues throughout training, but the primary driver of marginal performance gains shifts to strategic planning – exploring and mastering the use of planning tokens is what unlocks significant and sustained improvements in reasoning ability.

This emergent two-phase mechanism provides a unifying framework for the puzzling phenomena observed in RL training. It explains ”aha moments” as the discovery and internalization of high-level strategic reasoning strategies, such as self-reflections. It also accounts for the ”length-scaling” effect, as employing more sophisticated strategies – involving thorough planning and logical backtracing – naturally elongates the reasoning trace with structured, strategic deliberation. Notably, it provides a unified perspective to understand the complex token entropy dynamics across different models, through the lens of high-impact planning tokens and gradually confident execution tokens.

Based on this insight, we propose Hierarchy-aware Credit Assignment (HICRA), a novel algorithm designed to focus optimization pressure directly on this emergent strategic bottleneck. By selectively amplifying the learning signal for planning tokens, HICRA accelerates the exploration and reinforcement of effective high-level reasoning, leading to significant performance gains as demonstrated in our experiments.

To address this gap, we draw inspiration from human cognition. When a person reasons through a problem, we easily identify their strategic thinking by its function. A phrase like, “Let’s try a different approach,” functions as a high-level strategic maneuver that guides the problem-solving direction. In contrast, a phrase like, “so we add 5 to both sides,” is a low-level procedural step. Inspired by this functional distinction, we introduce Strategic Grams as a functional proxy to circumvent the difficulty of formally defining what is a “planning token”.

A key challenge is identifying the set of SGs in a principled and reproducible manner. Manual annotation or reliance on proprietary models would introduce subjectivity and hinder reproducibility. We therefore propose an automated, data-driven pipeline based on a key insight: SGs function as the reusable scaffolding of a reasoning process (Fig. 6). This function imparts a distinct statistical signature: SGs should appear frequently across a wide range of different solutions but be used sparingly within any single solution. However, a significant challenge is the linguistic diversity of strategic language, where a single strategic intent can be expressed through numerous phrases. Our pipeline is designed to overcome these challenges by first grouping semantically equivalent n-grams and then identifying which consolidated concepts exhibit the statistical signature of strategic planning. We place the detailed construction procedure to the appendix due to page limits.

This automated procedure is designed to yield a high-precision functional proxy for strategic planning, not an exhaustive lexicon of all possible SGs. We set reasonable hyper-parameters for identifying SGs, and we contend that the resulting SG collection is sufficiently representative to reveal the core learning dynamics. To validate this claim, we conduct a sensitivity analysis by randomly removing 30% of the identified SGs and re-running our main analysis (see Appendix). The resulting learning dynamic curves remain qualitatively identical, demonstrating the robustness of our methodology and the findings derived from it. To validate that our automated pipeline captures genuine semantic intent rather than statistical noise, we conducted a human annotation study. Results confirm that 86% of our identified SGs were classified by humans as functioning to “guide flow or propose plans,” compared to only 12% for otherwise. (See Appendix for full study details).

This strategic diversification provides the most direct evidence for our thesis: the model isn’t just getting better at executing plans; it’s getting better at planning itself. While the model explores new high-level strategic moves, the conditional entropy of procedural grams (grey line) remains stable. This suggests that once a procedural skill like arithmetic is mastered, there is little incentive to find diverse ways to perform it. The improved reasoning performance comes from discovering new ways to combine these established skills, which is the core function of strategic planning.

Our work opens several future research directions. First, it suggests a paradigm shift away from treating all tokens equally and prompts a rethinking of the action space away from individual tokens toward semantic, strategic units. Second, it calls for developing process-oriented approaches capable of valuing correct strategic choice even if the final answer is flawed. Finally, the likely universality of this reasoning hierarchy in complex reasoning tasks suggests that applying these principles to domains like code generation and agentic tool-use is a valuable path forward.