What structural framework prevents LLM explanations from becoming just plausible fiction?
This explores how to keep an LLM's stated reasoning honest — anchored to what the model actually does — rather than letting it generate a fluent, plausible story that's disconnected from its real computation.
This reads the question as asking what keeps an LLM's explanation tethered to reality instead of drifting into convincing fiction. The corpus is unusually direct about why the danger is real: explanation and execution in these models run on partly separate tracks. The "potemkin understanding" pattern shows models that explain a concept correctly, fail to apply it, and even recognize the failure — a triple incompatible with genuine understanding Can LLMs understand concepts they cannot apply?. The same split shows up measured: 87% accuracy in articulating principles versus 64% in acting on them, which the authors call a computational split-brain Can language models understand without actually executing correctly?. If the part of the model that explains isn't the part that decides, a good explanation is no guarantee of a true one.
So no single "framework" prevents fiction — but the corpus converges on one principle: an explanation only counts if it's verified against the mechanism, not just judged for plausibility. Mechanistic interpretability makes this explicit. Representational analysis alone finds correlations; causal analysis alone shows effects without saying why. Only the pair — locate a candidate feature, then intervene to confirm it actually drives the behavior — produces a claim you can trust Can we understand LLM mechanisms with only representational analysis?. That causal-intervention step is precisely the guard against plausible fiction: you can break or swap the supposed cause and watch whether the behavior follows.
There's a structured methodology behind this. Marr's three levels — what the system computes, by what algorithm, in what implementation — let you check an explanation at each layer instead of accepting one monolithic story Can cognitive science methods unlock how LLMs actually work?. And interpretability work suggests why explanations slip so easily: understanding inside a model is a patchwork, with clean circuits coexisting alongside shallow heuristics, so a tidy explanation may describe a circuit the model wasn't actually using Do language models understand in fundamentally different ways?.
The other route the corpus offers works on the input side: force the explanation to expose its load-bearing parts before you trust it. Treating Toulmin's argument model as explicit prompting steps makes the model name its warrants and backing rather than skipping implicit premises — and it catches failures plain chain-of-thought lets through Can structured argument prompts make LLM reasoning more rigorous?. Partial formalization does something similar by enriching natural language with selective symbolic structure, which constrains the reasoning without flattening its meaning Why does partial formalization outperform full symbolic logic?. Both make a hidden chain checkable.
What ties this together — and the thing worth carrying away — is that the failure isn't ordinary lying or a knowledge gap. These systems track statistical regularities extremely well yet fail in structurally specific ways, like accommodating a false premise even when they demonstrably know it's false Why do language models accept false assumptions they know are wrong?, What do language models actually know?. Plausible fiction is the default output of a fluent pattern-matcher. The frameworks that prevent it all do the same job from different angles: they add an external check — causal, structural, or argumentative — that fluency alone can't fake.
Sources 9 notes
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
Representational analysis alone identifies correlations without causation; causal analysis alone shows behavioral effects without explaining them. Only paired methods—locating candidate features representationally, then verifying causally—produce complete mechanistic claims.
Cognitive science's 70-year toolkit of behavioral probes, causal interventions, and representational analysis transfers directly to LLM interpretation. Marr's computational, algorithmic, and implementation levels reframe the problem structurally and enable layered rather than monolithic explanation.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.
QuaSAR and Logic-of-Thought both achieve 4-8% accuracy gains by enriching natural language with selective symbolic elements rather than replacing it. Full formalization loses semantic information; pure language lacks structure. Augmentation preserves both.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
LLMs achieve high fidelity in capturing language patterns yet show systematic, structurally specific failures—hallucination, reasoning collapse, and premise-sensitivity. The gap between statistical tracking and real knowledge is measurable and unavoidable.