SYNTHESIS NOTE
Do models fail worse when their own errors fill the context?
As a model's prior mistakes accumulate in context, does subsequent accuracy degrade predictably? And can scaling or architectural changes prevent this self-contamination effect?
A model executing a long-horizon task makes errors. Those errors remain in the context. The model then predicts the next token conditioned on a history that contains its own mistakes. Error probability increases. More errors accumulate. Performance degrades faster than a constant per-step error rate would predict.
This self-conditioning effect is empirically verified by controlling the error rate in the history shown to the model. As the error rate in prior context increases, subsequent step accuracy drops sharply. The mechanism is straightforward: models are trained to predict the most likely next token given context; when the context contains errors, those errors become part of the distribution being continued.
Unlike humans — who typically improve at a task with repetition — LLMs become less reliable as their context fills with their own mistakes. Practice does not help; contamination does.
Three practical implications:
Model scaling does not fix this — larger models self-condition just as much as smaller ones. The problem is not capability but the conditional prediction objective itself.
Long-horizon failure attribution matters — what looks like a reasoning or planning failure in long tasks is often an execution failure caused by error accumulation. The model had the capability; its own prior outputs degraded it. The DELEGATE-52 evidence — see Do frontier LLMs silently corrupt documents in long workflows? — is this mechanism at the workflow scale: a 50-round-trip relay is a maximally adversarial setup for self-conditioning, and the corruption curve decelerates but never plateaus, exactly the pattern this note predicts.
Thinking models fix self-conditioning — thinking models (like R1) are not affected by prior mistakes in the same way; sequential test-time compute greatly improves the length of task a model can complete (DeepSeek-V3 fails at 2 steps; R1 executes 200). The thinking process appears to insulate reasoning from error-contaminated context.
This is distinct from Does self-revision actually improve reasoning in language models?. Self-revision is a model's deliberate re-examination of its own reasoning, which introduces errors. Self-conditioning is a passive contamination mechanism — no deliberate revision required, just the accumulation of prior errors in context.
Inquiring lines that read this note 89
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do models learn from self-generated outputs without cascading failures?- How does distribution mismatch between training and deployment break self-correction?
- What failure modes emerge when model-generated content trains on itself iteratively?
- Why do error avalanches accelerate in self-training loops without verification?
- Can self-consistency checks fully prevent error avalanching in self-training loops?
- How does error avalanching compound failures in self-training iterations?
- Why does systematic overconfidence on self-generated outputs compound autoregressive errors?
- Do correlated human errors prevent models from transcending their training sources?
- What causes code quality to degrade across multiple rounds of recursive self-training?
- How does error distribution during training affect a model's ability to self-correct?
- How do agents revise their own errors during autonomous architecture discovery?
- Why does optimizing only quality cause model collapse in self-improvement loops?
- Can a model evaluate its own improvements without degrading over iterations?
- Why do most self-improving systems fail when given tasks with no clear external benchmark?
- What minimum model capability is required before self-improvement bootstrapping can begin?
- What are the three root causes models fail at self-correction?
- Why does external verification stop error amplification but internal self-assessment enable it?
- How does self-revision on wrong answers increase model confidence further?
- Does self-reflection help models notice their own constraint violations?
- Why does model self-revision increase confidence while degrading accuracy?
- How should systems maintain and revise models of their own assumptions?
- Why does uncontrolled self-revision drift toward instance-specific overfitting?
- How do prior errors in reasoning context amplify future mistakes?
- How do prior errors in context history amplify future mistakes in long tasks?
- Does deliberate self-revision introduce different errors than passive context contamination?
- How do prior errors in context history amplify future failures over time?
- Why do unchecked self-edits accumulate drift toward overfitting or incoherence?
- Can structural perturbations harm model accuracy more than semantic ones?
- Why do some models like Llama degrade under long context?
- What distinguishes domain-specific failure modes from general model limitations?
- Does the optimal model size depend on what capabilities you actually need?
- Does model collapse occur across different architectures or only in specific conditions?
- Why do frontier models corrupt more documents than weaker models during workflows?
- Why do frontier model failures in document editing go undetected by users?
- How does model tier affect whether errors delete or corrupt document content?
- Why do frontier models corrupt documents while weaker models delete them?
- Which failure modes dominate when models handle underspecified requests?
- Can smaller models actually perform well on specific downstream tasks?
- How can expensive models efficiently support cheap models in production?
- How do failed branches remain in context and contaminate subsequent reasoning?
- How do dependency errors propagate through incorrectly formalized definitions?
- Can removing failed branches from edited traces improve previous mistakes?
- Why do corrupted traces maintain performance as well as correct traces?
- Why do SFT datasets fail to teach models error correction from correction traces?
- Why do single function-calling benchmarks mask model weakness in specific areas?
- How does task contamination differ from test set data leakage?
- Why do text-only benchmarks underestimate deployed model capability?
- Why do models fail under distribution shift if accuracy metrics stay high?
- What training data contamination rates threaten model safety most practically?
- Can uncertainty estimates based on model self-assessment reliably signal errors?
- Why do fluent predictions fail to capture reliable internal models?
- Why do accuracy scores alone miss important dimensions of model capability?
- Why do models that repeat errors seem more confident than models that contradict themselves?
- How can a model explain something correctly yet fail to apply it?
- What makes some model capabilities reliable while others remain brittle?
- Can model training address failures that really originate in harness gaps?
- What mechanisms cause overly hard samples to degrade prior model performance?
- What happens when models optimize specifically against CoT monitors?
- What makes a model fail to activate relevant skills from its own harness?
- What happens when you project the same model onto different harnesses?
- Can mid-tier models benefit more from self-generated harness updates than others?
- What happens when different harnesses project the same model?
- What causes weak models to fail at activating harness artifacts?
- How much external enforcement does each model need before utility drops?
- Why do successful and failed trajectories need different memory processing?
- When does persistent harmful memory create performance error floors?
- Can external managers optimize context better than the model itself?
- What mechanism explains why context management prevents overflow failures most?
Related concepts in this collection 9
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do models fail worse when their own errors fill …
Does self-revision actually improve reasoning in l…
Does failed-step fraction predict reasoning qualit…
Do iterative refinement methods suffer from overth…
How quickly do errors compound during model self-t…
Does a model improve by arguing with itself?
Does training on messy search processes improve re…
Do frontier LLMs silently corrupt documents in lon…
Do short benchmarks predict how models perform ove…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does self-revision actually improve reasoning in language models?
When o1-like models revise their own reasoning through tokens like 'Wait' or 'Alternatively', does this reflection catch and fix errors, or does it introduce new mistakes? This matters because self-revision is marketed as a key capability.
active error injection via deliberate re-examination; self-conditioning is passive contamination by accumulated context errors
-
Does failed-step fraction predict reasoning quality better?
Can we use the fraction of abandoned reasoning branches to forecast whether a model will solve a problem correctly? This matters because it could guide more efficient test-time scaling than simply adding more tokens.
failed branches bias subsequent reasoning through a similar mechanism: abandoned paths remain in context and contaminate
-
Do iterative refinement methods suffer from overthinking?
Iterative refinement approaches like Self-Refine structurally resemble token-level overthinking in o1-like models. Does revision across multiple inference calls reproduce the same accuracy degradation seen within single inferences?
error accumulation across iterations follows the same contamination logic
-
How quickly do errors compound during model self-training?
When LLMs train on their own outputs without verification, do small mistakes amplify exponentially? This matters because it determines whether unsupervised self-improvement is even feasible.
the training-time analog: self-conditioning contaminates inference context within a single generation, error avalanching contaminates training data across self-training iterations — both produce compounding degradation from a model's own outputs
-
Does a model improve by arguing with itself?
When models revise their own reasoning in response to self-generated criticism, do they converge on better answers or worse ones? And how does that compare to challenge from other models?
the active-confidence version: self-conditioning passively degrades accuracy through context contamination, while DoT actively amplifies confidence in wrong answers — both are single-source error loops, distinguished by whether the mechanism is passive accumulation or active reinforcement
-
Does training on messy search processes improve reasoning?
Can language models learn better problem-solving by observing full exploration trajectories—including mistakes and backtracking—rather than only optimal solutions? This matters because current LMs rarely see the decision-making process itself.
SoS training directly addresses self-conditioning: models that learn to recognize dead ends and backtrack can break the error accumulation cycle rather than continuing to condition on their own mistakes; the backtracking mechanism provides an exit ramp from the contamination spiral
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
the DELEGATE-52 finding is this note's mechanism observed at workflow scale; corruption curve decelerates but never plateaus
-
Do short benchmarks predict how models perform over long workflows?
Standard LLM benchmarks measure single-turn performance, but real workflows involve sustained delegation across many turns. The question explores whether top benchmark performers maintain accuracy through longer interaction chains.
methodological consequence: self-conditioning means single-turn benchmarks cannot characterize long-horizon capability — relay-length must be its own evaluation axis
-
Can safety tests miss hazards that build over time?
Static tests check individual responses, but systems can accumulate unsafe state across interactions. This explores whether snapshot evaluations are sufficient to catch hazards that emerge only through repeated use or stored context.
grounds: a concrete route by which earlier errors persist as state and raise later error rates, so a snapshot safety test would miss the hazard; consistent with that claim, does not test it
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
- Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
- Large Language Models Cannot Self-Correct Reasoning Yet
- On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
- Large Language Model Reasoning Failures
- StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
- Reasoning Can Hurt the Inductive Abilities of Large Language Models
Original note title
self-conditioning effect — prior errors in context history amplify future error rates in long-horizon tasks