Can external retrieval signals outperform internal self-assessment during revision?
This explores whether feeding a model outside information — retrieved evidence or an external critic — beats having it grade and fix its own work, specifically at the moment of revision.
This explores whether outside signals (retrieved facts, a separate critic model) do better than a model's own self-judgment when it goes back to revise an answer — and the corpus leans hard toward yes, with an important asterisk. The cleanest finding is that the *source* of the revision, not the act of revising, decides the outcome: external critics improve accuracy, while a model second-guessing its own uncertain output usually just amplifies its confidence in the wrong answer Does revising your own reasoning actually help or hurt?. That's reinforced by direct evidence from o1-style reasoning models, where self-revision actually *degrades* accuracy — most revisions keep the wrong answer, smaller models flip correct answers to incorrect, and longer revision chains correlate with worse results Does self-revision actually improve reasoning in language models?.
Why does pure self-assessment struggle? Part of the answer is that models are bad introspectors. Their self-reports mostly echo patterns in their training data rather than reporting on any real internal state — genuine introspection only happens in the narrow case where a causal chain links an internal state to the report Can language models actually introspect about their own states?. If a model can't reliably read its own state, asking it to self-correct from that state is shaky ground.
The retrieval side shows what an external signal buys you. Framing 'when to pull in outside knowledge vs. trust internal memory' as a learned decision (an MDP) yields a ~22% accuracy jump, largely by retrieving at the right moments and cutting noise from unnecessary lookups When should language models retrieve external knowledge versus use internal knowledge?. And supervising the *intermediate* retrieval steps — not just the final answer — substantially outperforms outcome-only rewards, because it can contrast good and bad evidence-gathering chains directly Does supervising retrieval steps outperform final answer rewards?. External grounding works best when it's targeted and step-aware.
But the dichotomy isn't absolute, and that's the part worth knowing. Several notes show internal self-evaluation *can* be made to work when it's trained in rather than improvised at inference. Models can internalize self-assessment using the unused sequence space after their output, learning to compute their own reward at zero inference cost Can models learn to evaluate their own work during training?. Self-Examining RL has a model alternate between answering and judging, lifting win rates with no external reward at all Can models learn to judge themselves without external rewards?. And critique models embedded in the training loop help by preserving solution *diversity* — preventing premature convergence — a benefit that's arguably deeper than test-time accuracy Do critique models improve diversity during training itself?.
So the synthesis: external signals reliably win when revision is an ad-hoc, test-time act, because unaided self-correction tends to launder errors into confidence. Internal self-assessment becomes competitive only when it's *built into training* as a learned skill rather than asked of the model on the fly. A hybrid points the way forward — systems that write their own validated outputs back into a retrieval corpus, but only after external entailment and novelty checks gate them, blend internal generation with external verification instead of trusting either alone Can RAG systems safely learn from their own generated answers?.
Sources 9 notes
Revision guided by external models improves accuracy, but a model revising its own uncertain output typically amplifies confidence in wrong answers rather than correcting them. The revision source, not the revision act itself, determines the outcome.
Evidence from QwQ, R1, and LIMO shows most revisions retain wrong answers rather than correcting them. Smaller models frequently switch correct answers to incorrect during revision, and longer chains with more revisions correlate with lower accuracy.
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
DeepRAG models each reasoning step as a Markov Decision Process where the model learns when to retrieve versus rely on parametric knowledge. The 21.99% improvement comes from better-targeted retrieval and elimination of noise from unnecessary external knowledge.
Fine-grained feedback on intermediate retrieval steps significantly boosts agentic RAG performance compared to final-answer-only rewards. DPO trained with both positive and negative step feedback outperforms PPO and single-direction training by directly contrasting good and bad retrieval chains.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
SERL enables self-improving language models by having them alternate between generating responses and judging them pairwise, deriving rewards from ranking consistency and self-consistency of judgments. On AlpacaEval, this reached 59.90% win rate without external signals, up from 52.37%.
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
Systems can add generated answers to their retrieval corpus when outputs pass entailment verification, source attribution checks, and novelty detection. This prevents hallucinations from polluting future retrievals while allowing genuine knowledge accumulation.