Why do trajectories matter more than individual examples for in-context learning?
Can language models learn new sequential decision-making tasks from context alone, and if so, what data properties make this possible? This explores why isolated state-action pairs fail where full trajectories succeed.
In-context learning for supervised tasks works by providing a few input-output examples. Naively applying this to sequential decision making (providing a few state-action pairs) fails to enable ICL of new tasks. The key finding: the context must contain full or partial trajectories from the same environment level as the query — not just isolated examples. This property is called trajectory burstiness.
Why the difference matters: In supervised learning, examples can be from different instances — the model learns the function mapping. In sequential decision making, the model must generalize from the same level/environment to handle the wide range of states it may encounter at deployment. A sparse set of state-action pairs doesn't cover the state space; full trajectories do.
Trajectory burstiness is the probability that a given input sequence contains at least two trajectories from the same level. When this property is present in pre-training data, the model acquires the capacity to learn new tasks from demonstrations at inference time without weight updates.
Additional factors that increase ICL performance:
- Larger model and dataset size
- More task diversity in pre-training
- Environment stochasticity (forces generalization over trajectory variation)
- Higher trajectory burstiness in pre-training data
Generalization scope demonstrated: Train/test tasks differ greatly — different states, actions, dynamics, and reward functions. The model generalizes from, e.g., platform games to maze navigation from a handful of expert demonstrations. This is substantially harder than prior work that generalizes across reward function variants of the same environment.
The implication for dataset construction: sequential decision-making ICL requires a data distribution property (trajectory burstiness) that standard language modeling data does not naturally contain. This is a data structural requirement, not just a scale requirement.
This connects to Does training data format shape reasoning strategy more than domain? — here the structural property is at the trajectory level rather than the reasoning step level, but the principle is the same: data structure determines capability.
Inquiring lines that read this note 59
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do language models encode knowledge that influences generation, or primarily imitate surface patterns?- Why does removing language from its context destroy what makes it work?
- Why do context-sensitive languages transfer better than regular or context-free languages?
- How do models understand rich context better than they can generate it?
- How does in-context learning trigger phase transitions in model behavior?
- How do training objectives shape what a world model actually learns?
- How do complete multi-turn trajectories differ from isolated task examples?
- Why does the order of training examples matter for what models learn?
- What makes a good in-context learning example for a given task?
- How does demonstration coverage in context examples determine operation generalization?
- Why do foundation models develop task-specific heuristics instead of general world models?
- Does task superposition explain how models learn from multiple in-context trajectories?
- How do chunk-based step segmentation and trajectory structure modeling differ?
- How do neural networks extend contextual bandits beyond linear reward assumptions?
- What role does a model's representational structure play in learning?
- Can in-context learning replicate the timing effects that RL teaches models?
- Can in-context reinforcement learning match human sample efficiency on real problems?
- What makes session-aware multi-turn tracking necessary for asynchronous training?
- How does credit assignment work across many sequential decision steps in language models?
- How does trajectory filtering handle noise when language models use code execution tools?
- Can trajectory quality filtering improve model training in noisy environments?
- Does environment stochasticity force models to generalize better across trajectory variations?
- How does temporal anchoring maintain learning signals when preference gaps collapse?
- Can in-context learning substitute for domain-specific training altogether?
- What makes some contexts learnable as rules versus requiring model retraining?
- Do text-space skills transfer learning across different frontier models?
- How does explicit exploratory prompting compare to fine-tuned reinforcement learning for in-context adaptation?
- Do few-shot examples improve in-context learning or add noise?
- Can episodic memory alone enable learning without parameter updates?
- How do retrieved memories differ from decision-context passages for prediction?
- Why do external memory consolidation systems fail worse than naive in-context learning on continual tasks?
- Can trajectory structure alone provide process supervision without human annotation?
- Can parallel trajectories reveal better decision branches without human labeling?
- Can graph topology represent successful trajectory clusters more effectively than skill libraries?
- Why does delegation training help models that work alone?
- Why do trajectory-based skills fail to transfer across different environments and use cases?
- What makes next-state signals from agent trajectories a reliable learning source?
- How does training order affect knowledge acquisition in language models?
- Can in-context learning's advantage erode once interaction histories exceed the context window?
- Why is in-context learning brittle to the order of examples presented?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does training data format shape reasoning strategy more than domain?
What explains why models trained on multiple-choice data reason differently than those trained on free-form text? The research isolates format and domain effects to measure which one matters more.
trajectory burstiness is another case where data structure determines emergent capability
-
What do models actually learn from chain-of-thought training?
When models train on reasoning demonstrations, do they memorize content details or absorb reasoning structure? Testing with corrupted data reveals which aspects of CoT samples actually drive learning.
structural properties of training data drive learning; applies at both the reasoning trace and trajectory levels
-
Can we allocate inference compute based on prompt difficulty?
Does adjusting how much compute each prompt receives—rather than using a fixed budget—improve model performance? Could smarter allocation let smaller models compete with larger ones?
the context-length requirements for trajectory-bursty inference raise per-query compute costs
-
Can LLMs handle multiple tasks at once during inference?
Do language models maintain multiple distinct in-context learning tasks simultaneously in their internal representations, and if so, what prevents them from actually generating outputs for more than one task?
task superposition may be the representational mechanism enabling trajectory-bursty ICL: the model maintains multiple task interpretations from in-context trajectories simultaneously before committing to a single policy at generation time
-
Can transformers learn to solve new problems within episodes?
Explores whether transformer models can develop meta-learning abilities through RL training, enabling them to adapt to unseen environments by learning from within-episode experience alone, without updating weights.
ICRL is the RL-trained capability that trajectory burstiness enables: same-level trajectories create the meta-learning pressure during training that ICRL exploits at inference time for adaptation to unseen environments
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Generalization to New Sequential Decision Making Tasks with In-Context Learning
- Supervised Pretraining Can Learn In-Context Reinforcement Learning
- Teaching Large Language Models to Reason with Reinforcement Learning
- Schema-learning and rebinding as mechanisms of in-context learning and emergence
- STaR: Bootstrapping Reasoning With Reasoning
- Training a Generally Curious Agent
- In-Context Principle Learning from Mistakes
- BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
Original note title
trajectory burstiness — same-level trajectories in context — is required for in-context learning of sequential decision-making across new tasks