Can context windows and RAG actually change what language models generate?
This explores whether feeding a model more context — long context windows or retrieved documents (RAG) — actually steers what it outputs, or whether the model's baked-in training knowledge ends up overriding what you give it.
This explores whether feeding a model more context — long context windows or retrieved documents (RAG) — actually steers what it outputs, or whether the model's training-time priors quietly win out. The corpus gives a sharper answer than you'd expect: context changes generation, but only up to a point, and the failure point is more about old associations than about how much you stuff into the window.
The most striking finding is that models often ignore the very context you hand them. When a model's parametric knowledge — what it absorbed during training — has a strong association, that prior can override the document sitting right in front of it, and no amount of clever prompting fixes it; you have to intervene in the model's internal representations directly Why do language models ignore information in their context?. A related ceiling shows up with prompting more broadly: prompt optimization can reorganize and activate knowledge the model already has, but it can't inject knowledge that was never in the training data Can prompt optimization teach models knowledge they lack?. So context reshuffles and surfaces — it doesn't teach. There's even a 'context collapse' effect where, if you under-specify your query, the model falls back to a blended average of its training data rather than your situation Why do large language models produce generic responses to vague queries?.
Where context genuinely earns its keep is in retrieval-shaped tasks. Long-context models can match RAG on semantic retrieval with no special training, though they still fall apart on structured queries that need joins across tables — context length alone can't bridge that Can long-context LLMs replace retrieval-augmented generation systems?. As windows grew, the whole design center of RAG shifted: instead of fussy precise retrieval, you can feed coarse chunks and let a strong reader do the work Can long-context models resolve retriever-reader imbalance?. And the bottleneck on really long inputs turns out not to be memory but compute — the work of consolidating context into the model's internal state, which improves with more processing passes Is long-context bottleneck really about memory or compute?.
The more interesting frontier is treating context as something other than a passive prompt. Recursive Language Models park a giant prompt in a code environment and query it programmatically, handling inputs a hundredfold beyond the window and even beating the base model on shorter prompts Can models treat long prompts as external code environments?. A 'fast-slow' split routes durable lessons into weights and task-specific context into the prompt, which sidesteps catastrophic forgetting — evidence that text-channel context is doing real adaptive work, not just decoration Can splitting adaptation into two channels reduce forgetting?. And you don't always need more retrieval at all: a model's own calibrated uncertainty often beats elaborate adaptive-retrieval machinery at deciding when to pull in context Can simple uncertainty estimates beat complex adaptive retrieval?.
The thing you might not have known you wanted to know: context is a steering wheel, not an engine. It can redirect, activate, and ground what a model produces, but it competes against deep training-time priors — and when those priors are strong enough, the document you carefully retrieved loses. Better RAG isn't only about retrieving the right text; it's about whether the model will actually let that text override what it already 'believes.'
Sources 9 notes
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Unlike social-media context collapse, which flattens multiple audiences, LLM collapse occurs when users provide insufficient contextual scaffolding and models default to blended training-data priors. This distinction suggests remedies should focus on query verification and user-driven context specification rather than platform controls.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.
LongRAG shows that 4K-token units and long-context readers outperform 100-word retrieval on standard benchmarks. The optimal RAG design shifts from precise retrieval to coarse ranking plus deep reading as context windows expanded.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Recursive Language Models store long prompts in a Python REPL and query them via code execution, avoiding attention degradation. RLMs outperform base models even on shorter prompts while handling inputs two orders of magnitude beyond context windows.
Fast-Slow Training routes task-specific lessons into optimized prompts while keeping parameter updates minimal, reaching equivalent performance 1.4–3x faster with substantially less catastrophic forgetting and plasticity loss, demonstrating that forgetting is a misallocation problem rather than an inherent cost.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.