Can prompt optimization inject new knowledge into language models?
This explores whether clever prompting can give a model knowledge it never learned during training — or whether prompts can only rearrange what's already inside.
This explores whether clever prompting can give a model knowledge it never learned during training. The corpus is unusually direct here: it can't. Prompt optimization works entirely within a model's pre-existing training distribution, retrieving and reorganizing what's already there but never supplying domain knowledge that was absent from the start Can prompt optimization teach models knowledge they lack?. That creates a hard ceiling — no phrasing trick compensates for a foundational gap, it only activates latent material.
What makes this interesting is that prompting can fail even when the knowledge is right there in the context window. Models routinely ignore information you hand them because parametric associations baked in during training override what's in front of them; textual prompting alone can't beat a strong prior, and overriding it takes intervention in the model's internal representations, not better wording Why do language models ignore information in their context?. So prompting is doubly bounded: it can't add what isn't in the weights, and it can't always surface what is in the prompt.
The same ceiling shows up wherever people hope a model will *do* something rather than recall it. Ask an LLM to actually run an iterative numerical method and it pattern-matches a memorized template instead, emitting plausible-but-wrong numbers regardless of scale Do large language models actually perform iterative optimization?. On genuine constrained optimization, models plateau around 55-60% satisfaction no matter the parameter count or reasoning training Do larger language models solve constrained optimization better?. And self-improvement hits a formal wall: a model can't validate its own fixes without an external verifier, because of the generation-verification gap What stops large language models from improving themselves?. The throughline is that prompting redistributes existing capability — it doesn't manufacture new capability.
If injecting knowledge is the goal, the corpus points elsewhere — toward methods that touch the weights. DPO training on a teacher's correct-and-incorrect examples lets small models pick up function-calling skills that prompting alone wouldn't give them Can small models match large models on function calling?. Composable expert vectors tuned at inference let a model genuinely activate new specializations Can models dynamically activate expert skills at inference time?. Even RLVR works by nudging a thin slice of high-entropy 'forking' tokens — it sharpens reasoning decisions rather than rewording the question Do high-entropy tokens drive reasoning model improvements?. The quiet lesson: prompting is a retrieval interface, not a learning channel. Knowing where that line sits tells you when to stop tuning prompts and start changing the model.
Sources 8 notes
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
Transformer2 demonstrates that tuning only singular values within weight matrices produces composable expert vectors that dynamically mix at inference without interference, outperforming LoRA with fewer parameters and enabling continual specialization.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.