SYNTHESIS NOTE
Does fine-tuning disconnect reasoning steps from final answers?
When models are fine-tuned on specific domains, do their chain-of-thought steps become less causally connected to their outputs? Three experiments test whether reasoning chains remain functionally faithful after training.
The "Impact of Fine-Tuning on Chain-of-Thought Reasoning" paper reveals a dimension of SFT damage that InfoGain metrics miss: faithfulness. After fine-tuning, the reasoning steps in CoT outputs are less causally connected to the final answer. The model still generates reasoning chains — they just matter less for determining the output.
Three specific tests operationalize this:
Early Termination: truncate the CoT at step i and ask for the final answer. If truncation at an early step already produces the correct answer, only a fraction of the reasoning was faithful. Fine-tuned models show earlier convergence — their answers are "decided" before the reasoning chain finishes.
Paraphrasing: rephrase later reasoning steps. If the answer is invariant to paraphrasing, the reasoning was faithful (the argument matters, not the words). Fine-tuned models show less sensitivity to paraphrasing — suggesting the chain is performative rather than functional.
Filler Substitution: replace later reasoning steps with filler tokens ("..."). If the answer doesn't change, those steps weren't contributing. Fine-tuned models tolerate more filler substitution.
This extends the SFT accuracy trap in a critical direction. Does supervised fine-tuning actually improve reasoning quality? showed that SFT reduces the informativeness of reasoning steps. This paper shows SFT also reduces whether those steps actually influence the final answer at all. The model may generate a complete-looking chain, but the chain has been partially disconnected from the output it appears to support.
Smaller models (Llama-3-8B-Instruct) are more affected than larger ones (GPT-4), suggesting that larger models have sufficient capacity to maintain reasoning-output coupling even after fine-tuning. This connects to Do language models actually use their reasoning steps? — fine-tuning makes an already-fragile causal coupling even weaker. If Does chain-of-thought reasoning reveal genuine inference or pattern matching?, then fine-tuning further degrades faithfulness because the model learns domain-specific shortcuts that bypass the imitated reasoning pattern entirely — the chain was already performative, and fine-tuning makes it more so.
Inquiring lines that read this note 142
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should recommendation systems balance individual preference and diversity? Does chain-of-thought reasoning reveal how models actually think or merely imitate reasoning?- How much does faithfulness vary naturally in reasoning without evaluation pressure?
- Does chain-of-thought monitoring fundamentally degrade under optimization pressure?
- How often do papers treat chain-of-thought as interpretability incorrectly?
- Does each reasoning step in chain-of-thought introduce cumulative error?
- Can chain-of-thought faithfulness exist without causal necessity in reasoning?
- Can chain-of-thought explanations be both sufficient and necessary for model decisions?
- What happens to chain-of-thought performance across distribution shifts?
- Why do more capable models prefer shorter chains of thought?
- How does chain-of-thought training change higher layer computations?
- What three factors actually drive chain of thought performance improvements?
- How does chain-of-thought reasoning become decorative after domain-specific fine-tuning?
- How much does chain-of-thought reasoning narrow the decompression gap?
- Does chain of thought reasoning faithfully reflect what a model actually believes?
- How does interaction horizon differ from chain-of-thought depth?
- How does faithfulness differ from informativeness in chain-of-thought evaluation?
- How do thought anchors differ from individual forking tokens mechanistically?
- Why might chain-of-thought reasoning bypass action selection pathways?
- Why does chain-of-thought work for math but fail for grounding?
- How much of chain-of-thought reasoning actually diverges from the final answer?
- Can single representation edits match chain-of-thought reasoning without explicit steps?
- Does training against chain-of-thought reasoning cause models to hide their reasoning?
- Does chain-of-thought reasoning cause model behavior or merely reflect it?
- Can chain-of-thought reasoning hide genuine changes in model behavior?
- Why does reasoning in chain of thought not match causal influence?
- How much does chain-of-thought reasoning actually determine model outputs?
- What is the mechanistic signature when models chain facts never presented together?
- How do single wrong steps corrupt entire reasoning chains?
- Why do expert reasoners skip steps that novices must state explicitly?
- How do causal chains enforce long-horizon length differently than instruction-based tasks?
- Why do single examples trigger large reasoning improvements in models?
- How does critique fine-tuning on one problem unlock broader reasoning?
- What makes reasoning-specific post-training different from standard parameter scaling?
- How does the functional separation of knowledge and reasoning affect adaptation methods?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
- How do procedural versus factual knowledge differ in pretraining versus fine-tuning?
- Why do reasoning gains resist clear attribution to specific training changes?
- Does looped pretraining build reasoning more efficiently than supervised fine-tuning?
- Can penalizing reasoning transitions fix underthinking without fine-tuning models?
- Does changing decoding procedure reveal hidden chain-of-thought paths?
- Why do different reasoning chains surface different relevant facts?
- Can layer-wise prediction stabilization identify when genuine reasoning has stopped?
- Can we detect redundant reasoning steps during model inference instead of training?
- Why does fine-tuning for continuous space cause catastrophic forgetting?
- Why does fine-tuning models for continuous reasoning cause catastrophic forgetting?
- Why does a replay mechanism prevent reasoner skills from over-specializing?
- What are the distinct sources of catastrophic forgetting in sequential fine-tuning?
- Do causal rules enforce robustness that statistical patterns alone cannot maintain?
- How do humans use associative reasoning without causal connections?
- Can activation patching reveal which reasoning steps actually matter?
- Does causal mediation analysis quantify reasoning faithfulness across model types?
- How much do reasoning models actually verbalize their causal influences?
- How does semantic association differ from mechanistic causal reasoning?
- What makes a causal abstraction more transferable than a generic heuristic?
- Why do attention circuits need causal verification beyond feature visualization?
- Can a Reflect mechanism detect and revise failed causal predictions?
- How does optimizing for accuracy during training degrade downstream reasoning quality?
- Why does fine-tuning degrade reasoning quality even as accuracy improves?
- Why does domain accuracy improve while reasoning quality degrades after supervised fine-tuning?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- Does fine-tuning models for specific tasks destroy their ability to reason?
- Why does fine-tuning change how models process retrieved context?
- Why does fine-tuning improve some capabilities while degrading others?
- Why does instruction tuning hurt knowledge-intensive tasks more than reasoning tasks?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- Why does SFT reduce reasoning quality even when improving domain accuracy?
- Does SFT degrade reasoning quality while improving domain accuracy?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Can models maintain reasoning-output coupling while improving domain accuracy?
- How does supervised fine-tuning degrade chain-of-thought faithfulness over time?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- How robust and general are causal edits like ROME across different facts?
- Can a single SAE feature control reasoning behavior across model families?
- Why do smaller models lose reasoning faithfulness more than larger models?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- Why does stronger reasoning reduce model compliance with instructions?
- What causes models to develop domain capability cliffs after specialization?
- What hidden costs emerge when you fine-tune models for a single domain?
- Why do fine-tuned models fail outside their specialized domains?
- How much do mechanistic interpretability findings reflect true reasoning architecture?
- Can mechanistic interpretability explain explanation-execution disconnection?
- Why does reasoning fine-tuning reduce model abstention capacity by 24 percent?
- Does reasoning fine-tuning actually reduce a model's ability to abstain?
- What happens when reasoning fine-tuning eliminates model refusal mechanisms entirely?
- Does reasoning fine-tuning actually damage a model's ability to abstain?
- How do reasoning improvements suppress a model's ability to abstain?
- Does reasoning fine-tuning actually harm a model's ability to abstain?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Why does reasoning fine-tuning reduce models' ability to abstain?
- How can safety-aligned parameters be protected during user-specific fine-tuning?
- Can mechanistic interpretability tools decode the biases alignment training conceals?
- Why does fine-tuning sometimes damage chain-of-thought reasoning even when accuracy improves?
- Can deliberate corruption of reasoning traces harm out of distribution generalization?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Can reinforcement learning add missing domain knowledge to fine-tuned reasoning models?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- What happens to AI reasoning when you remove specific political features?
- Do weight-free agent edits keep chain-of-thought observations meaningful?
- How does behavioral fine-tuning differ from factual knowledge encoding in models?
- Why does training order matter across different domain types?
- Why do foundation models develop task-specific heuristics instead of causal understanding?
- What happens to base model capabilities when you apply finetuning?
- How much does pretraining quality affect the modularity of fine-tuned models?
- What training regimes confound surface mechanisms with their actual causes?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- Why does instruction-tuning reduce a model's context-following behavior?
- Why does fine-tuning function as character training rather than capability training?
- Why does the distinction between functional and causal grounding matter for AI alignment?
- What distinguishes functional grounding from genuine causal grounding in AI systems?
- Does fine-tuning actually change model capabilities or only output distribution?
- Does fine-tuning a small model match fine-tuning a large one?
- Can chain-of-thought traces be faithful without causal sufficiency and necessity?
- What makes some sentences in reasoning traces have disproportionate causal influence?
- Why do models skip steps that would make reasoning clearer?
- What makes answer equivalence sufficient to discard a reasoning path?
- Do models deliberately hide influences from their reasoning traces?
- When does the correlation between consistency and correctness break down?
- Can AI answers decouple from the reasoning processes that produced them?
- How does semantic entanglement interact with personality dimension shifts during finetuning?
- Does functional integration determine cognitive system boundaries?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Does fine-tuning disconnect reasoning steps from…
Does supervised fine-tuning actually improve reaso…
Do language models actually use their reasoning st…
Do language model reasoning drafts faithfully repr…
Does chain of thought reasoning actually explain m…
Does chain-of-thought reasoning reveal genuine inf…
Do chain-of-thought traces actually help users und…
Does supervised fine-tuning improve reasoning or j…
Does RLVR actually improve mathematical reasoning …
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
parallel SFT damage dimension: reasoning quality (InfoGain) vs. reasoning faithfulness (causal connection)
-
Do language models actually use their reasoning steps?
Chain-of-thought reasoning looks valid on the surface, but does each step genuinely influence the model's final answer, or are the reasoning chains decorative? This matters for trusting AI explanations.
FT worsens an already-failing faithfulness property
-
Do language model reasoning drafts faithfully represent their actual computation?
If models externalize reasoning in thinking drafts before answering, does the draft accurately reflect their internal process? This matters for AI safety monitoring and error detection.
draft-to-answer consistency is the dimension FT specifically degrades
-
Does chain of thought reasoning actually explain model decisions?
When language models show their reasoning steps in agentic pipelines, does the quality of those steps predict or explain the quality of final outputs? This matters for trusting and debugging AI systems.
faithfulness degradation explains why agentic CoT fails: the chain was already partially decorrelated before deployment
-
Does chain-of-thought reasoning reveal genuine inference or pattern matching?
Explores whether CoT instructions unlock real reasoning capabilities or simply constrain models to mimic familiar reasoning patterns from training data. This matters for understanding whether language models can actually reason abstractly.
provides the mechanistic explanation: if CoT is pattern-matching on reasoning form rather than genuine inference, fine-tuning further disconnects the chain from the answer because the model learns domain-specific shortcuts that bypass the imitated reasoning pattern entirely
-
Do chain-of-thought traces actually help users understand model reasoning?
Chain-of-thought explanations are often presented as transparency tools, but do they genuinely improve human understanding or create an illusion of interpretability? A human-subject study tests whether traces help users follow and evaluate model reasoning.
faithfulness degradation compounds the performance-interpretability decoupling: traces already optimized for model performance rather than human interpretability become even less causally connected to final answers after fine-tuning, serving neither function well
-
Does supervised fine-tuning improve reasoning or just answers?
Explores whether training models on question-answer pairs actually strengthens their reasoning quality or merely optimizes them toward correct outputs through shortcuts. This matters for deploying AI in domains like medicine where reasoning must be auditable.
the SFT accuracy trap and faithfulness degradation are two dimensions of the same SFT damage: accuracy trap captures reasoning quality loss (InfoGain), faithfulness degradation captures causal connection loss; together they show SFT makes chains both less informative and less causally relevant
-
Does RLVR actually improve mathematical reasoning or just coherence?
RLVR post-training makes reasoning traces locally more consistent, but does this structural improvement translate to valid mathematical proofs? We investigate whether trace coherence is sufficient for correctness.
complementary coherence/faithfulness split: RLVR improves structural coherence between adjacent steps without improving validity; fine-tuning degrades faithfulness (causal connection to answer) while potentially maintaining coherence; both show training can change the surface without changing the substance
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- Measuring Faithfulness in Chain-of-Thought Reasoning
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
- A Comment On "The Illusion of Thinking": Reframing the Reasoning Cliff as an Agentic Gap
Original note title
fine-tuning degrades cot faithfulness independently of accuracy — reasoning steps influence final answers less after domain-specific training