Why do foundation models develop heuristics instead of world models?
This explores why large pretrained models tend to learn a grab-bag of task-specific shortcuts rather than a coherent internal model of how the world actually works — and what the corpus says about why that happens and what would have to change.
This explores why foundation models lean on task-specific shortcuts instead of building a unified, generative model of the world. The clearest answer in the corpus is that prediction accuracy and world understanding are not the same thing — and training rewards the former. When transformers are probed on domains with known underlying laws, like orbital mechanics or board games, they turn out to be fitting predictive patterns that happen to work, not recovering the structure that generates those patterns Do foundation models learn world models or task-specific shortcuts?. The tell is fragility: fine-tune or push the model slightly off-distribution and the 'laws' it seemed to know turn out to be nonsensical and slice-dependent, and circuit analysis shows even arithmetic runs on range-matching heuristics rather than an actual algorithm. A heuristic that scores well on the training objective has no pressure to cohere into anything deeper.
The deeper reason is a definitional gap about what a world model even is. A genuine world model isn't a better next-frame predictor — it's a simulator of *actionable possibilities* that lets you reason about interventions and counterfactuals, asking 'what if I did X' rather than 'what comes next' What makes a world model actually useful for reasoning?, What should a world model actually be designed to do?. Next-token prediction never asks the model to do this, so the model never builds it. Worse, a world model isn't one thing you can accidentally acquire: it decomposes into five separate design choices — data, latent representation, reasoning architecture, training objective, and how it plugs into decisions — any of which can misalign with the others What five design choices compose a world model?. Surface-pattern learning is what you get when none of those choices are made deliberately.
Here's the part you might not expect: the same shortcut-over-structure tendency shows up at reasoning time, not just in pretraining. Reasoning models 'wander like tourists, not scientists' — they explore invalidly and abandon promising paths prematurely, so success drops exponentially as problems get deeper Why do reasoning models abandon promising solution paths?, Why do reasoning LLMs fail at deeper problem solving?. Tellingly, the fix often isn't more knowledge — it's just penalizing premature thought-switching at decoding time, which recovers accuracy without any retraining Do reasoning models switch between ideas too frequently?. That's a strong hint that the capability for structured reasoning is latent but not organized — the model defaults to the locally-easy move, exactly as it defaults to the locally-predictive heuristic. And chain-of-thought can actively hurt: on exception-based rule inference, reasoning models underperform simpler ones because they overgeneralize and hallucinate constraints rather than honoring the evidence Why do reasoning models fail at exception-based rule inference?.
What would push models toward structure instead of shortcuts? The corpus points at deliberately engineering breadth and uncertainty back in. Training abstractions alongside solutions forces breadth-first exploration that depth-only chains skip Can abstractions guide exploration better than depth alone?; making latent reasoning stochastic lets a model hold a distribution over strategies rather than collapsing to one Can stochastic latent reasoning help models explore multiple solutions?; and evolutionary search at inference time sustains population diversity to avoid the premature convergence single-trajectory refinement falls into Can evolutionary search beat sampling and revision at inference time?. The common thread: heuristics are what you get when a system always takes the cheapest locally-rewarded path, and world models only emerge when something explicitly forces exploration of possibilities the cheap path skips.
One last thread worth pulling: bigger foundation models don't dissolve this problem, they sharpen it. Without empirical anchoring to real data, iterating on a model's outputs creates an epistemic loop where you confirm your own beliefs instead of testing them against the world Do foundation models actually reduce our need for real data?. A model built on heuristics rather than a world model gives you no friction against being wrong — which is exactly why the distinction matters beyond benchmark scores.
Sources 12 notes
Inductive bias probes show transformers trained on orbital mechanics and games learn predictive patterns, not unified world structure. Fine-tuning reveals nonsensical, slice-dependent laws; circuit analysis shows arithmetic relies on range-matching heuristics, not algorithms.
Research shows LLMs may achieve high prediction accuracy through task-specific heuristics without developing coherent generative models of how the world works. True world models must enable reasoning about interventions and counterfactuals, not surface regularities.
Drawing on hypothetical thinking in psychology, world models are most useful when designed to simulate all actionable possibility spaces—physical, embodied, emotional, social, mental, counterfactual, and evolutionary—grounded in agent decision-making rather than passive prediction.
World model design comprises five distinct dimensions: data preparation, latent representation, reasoning architecture, training objective, and decision-system integration. Each can misalign with the others, and treating them as a single problem obscures where failures originate and prevents proper evaluation.
Reasoning LLMs exhibit two reinforcing failures: wandering (invalid exploration) and underthinking (premature path-switching). Decoding-level interventions like thought-switching penalties improve accuracy without fine-tuning, suggesting viable solutions exist but are abandoned prematurely.
Current reasoning models lack the three properties of systematic exploration: validity, effectiveness, and necessity. This causes success probability to drop exponentially with problem depth, making medium problems solvable but deep problems catastrophically harder.
o1-like models frequently abandon reasoning paths mid-exploration, wasting tokens on incomplete approaches. A decoding-only penalty on thought-transition tokens (TIP strategy) discourages switching, improving accuracy on challenging math without model fine-tuning.
Across four game-based tasks, reasoning models scored below 25% on exception rules versus 55–65% for non-reasoning models. Chain-of-thought introduces math overuse, overgeneralization, and hallucinated constraints that amplify errors in negative evidence recognition.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent distributions over solutions rather than single predictions. This allows handling of ambiguous problems and multiple valid strategies that deterministic designs cannot represent.
Mind Evolution uses genetic algorithms with LLM-generated mutations and crossovers to significantly outperform Best-of-N and Sequential Revision on planning benchmarks. An island model sustains population diversity, preventing the premature convergence that single-trajectory refinement exhibits.
Powerful foundation models don't eliminate the need for real data—they heighten it. Without empirical anchoring, iterative prompt refinement creates epistemic circularity where users confirm their own beliefs rather than test them.