An AI given the right document may still answer from its training memory instead — and no amount of rephrasing fixes that.
How does parametric knowledge sabotage context-grounded question answering?
This explores how a model's baked-in training knowledge (its 'parametric' memory) can override the information you actually hand it in the prompt — so the model answers from what it already 'knows' instead of from the context in front of it.
This explores how a model's baked-in training knowledge (its 'parametric' memory) can override the information you actually hand it in the prompt — so the model answers from what it already 'knows' instead of from the context in front of it. The core mechanism is a tug-of-war: when the prior associations a model learned during training are strong, they win out over the document you just pasted in, and the model generates output inconsistent with its own context Why do language models ignore information in their context?. The unsettling part is that this isn't fixable by phrasing. Textual prompting alone can't beat a strong prior — researchers found you need to intervene directly in the model's internal representations to force it to defer to context. So 'just tell it to use the provided text' has a ceiling.
There's a deeper version of the same failure that shows up around false assumptions. Models will accept a false presupposition smuggled into a question even when, asked directly, they clearly know the correct fact Why do language models accept false assumptions they know are wrong?. Knowing the truth and acting on it are decoupled: false presuppositions drive accommodation more strongly than correct knowledge drives correction. Intriguingly, one strand of research argues this isn't really a knowledge problem at all — it's social. Models learned face-saving conversational habits from human training data, so they avoid bluntly contradicting a user to keep the peace Why do language models avoid correcting false user claims?. The sabotage, on this reading, is less 'my training facts overrode your document' and more 'I'd rather not call you wrong.'
A related trap appears when you give a model too little to work with rather than too much. When context is thin or vague, the model falls back on blended training-data priors and produces generic, averaged answers — what one note calls context collapse from scaffolding failure Why do large language models produce generic responses to vague queries?. The remedy there points the other way: more user-supplied context and query verification, not platform-side controls. So parametric knowledge fills whatever vacuum the context leaves open.
Worth knowing: context doesn't have to be wrong to get ignored — it can simply be long. Reasoning accuracy drops sharply as inputs grow, even at a few thousand tokens, far below the model's stated context window, and chain-of-thought doesn't rescue it Does reasoning ability actually degrade with longer inputs?. That means a buried-but-correct fact in your context competes with parametric defaults from a position of weakness. And the inverse is a useful boundary: prompting and context can only activate knowledge the model already has — they can't inject what was never there Can prompt optimization teach models knowledge they lack?. So the failure mode cuts both ways: priors override context when the model knows too much, and context can't compensate when the model knows too little.
The practical upshot is that 'grounding' a model is an active contest, not a default. If you care about context-faithfulness, the corpus suggests leaning on retrieval architectures that keep the source text in front of the model and verifiable — for instance, agents that search the raw corpus directly rather than trusting dense recall Can direct corpus search beat embedding-based retrieval? — and treating the model's confident, fluent answer as something to check against the provided text, not as evidence it read the text.
Sources 7 notes
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Unlike social-media context collapse, which flattens multiple audiences, LLM collapse occurs when users provide insufficient contextual scaffolding and models default to blended training-data priors. This distinction suggests remedies should focus on query verification and user-driven context specification rather than platform controls.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
Show all 7 sources
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
GrepSeek trains agents to retrieve via executable shell commands over raw text, achieving better multi-hop performance on entity-constrained queries than dense embeddings. The approach scaffolds unstable search mechanics with supervised trajectories, then refines task-oriented behavior through reinforcement learning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Linguistic Calibration of Long-Form Generations
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Conversational Alignment with Artificial Intelligence in Context
- LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High
- Learning To Retrieve Prompts for In-Context Learning
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds