Do LLMs predict entailment based on what they memorized?
Explores whether language models make entailment decisions by recognizing memorized facts about the hypothesis rather than reasoning through the logical relationship between premise and hypothesis.
McKenna et al. (2023) named a specific, reproducible bias in LLM entailment behavior: the attestation bias. When an LLM is asked whether premise P entails hypothesis H, its prediction is bound to the hypothesis's out-of-context truthfulness — whether H is attested in training data — rather than the conditional truth of H given P.
The mechanism is clear: if a model's training data confirms H as true (independently of any premise), the model is likely to predict entailment regardless of what P says. Conversely, if H is not attested, the model is less likely to predict entailment even when it would be correct. Entities serve as "indices" to memorized propositions — the presence of a known entity activates stored associations that override the in-context reasoning task.
The authors demonstrate this with a "random premise" experiment: replace the original premise with a random unrelated premise while keeping H constant. An ideal inference model should detect that entailment is no longer supported and predict "no entailment." LLMs instead maintain elevated entailment predictions when H is attested — demonstrating that they are responding to stored propositions about H, not to the P→H relationship.
This connects to two complementary failure modes already in the vault. Do language models actually use their encoded knowledge? shows that encoded knowledge doesn't reliably affect generation. Attestation bias is the inverse problem: memorized statements do influence generation, but in the wrong direction — they substitute for rather than support proper inference. Both failures arise from the same root: LLM generation is not governed by a clean separation between retrieved knowledge and in-context reasoning.
The practical implication: NLI benchmark performance measures a combination of reasoning and memorization that cannot be cleanly disentangled without carefully designed bias-adversarial test sets.
Inquiring lines that read this note 57
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What happens to knowledge when intelligence becomes tokenized like a commodity? What explains language models' asymmetric difficulty with implicit versus explicit linguistic relations?- Can LLMs infer situational context the way humans do pragmatically?
- How does implicit meaning processing limit LLM pragmatic reasoning?
- Can LLMs infer implicit meaning without surface linguistic markers?
- How do embedding contexts like presupposition triggers affect LLM entailment reasoning?
- Can LLMs identify implicit metaphoric mappings that require pragmatic inference?
- Can LLMs compute how presuppositions project through embedded clauses?
- What semantic information is necessary to preserve for sound LLM reasoning?
- How does surface salience compete with background knowledge in model inference?
- How does inductive reasoning from partial evidence enable hypothesis formation?
- How do language models treat injected evidence as shared background knowledge?
- What replaces truth-correspondence in probabilistic knowledge representations?
- Why do language models substitute parametric knowledge over retrieved context mid-reasoning?
- Can language models correct false assumptions or only reinforce them?
- Can models detect false presuppositions when they actually possess the knowledge?
- What implicit premises do language models skip even with correct surface reasoning?
- How do pretrained language models represent inferential patterns versus lexical and positional cues?
- Should LLM reasoning be studied as latent state trajectories rather than surface text?
- Why does hypothesis attestation bias exist separately from frequency bias in NLI?
- Why do entities trigger memorized propositions instead of enabling reasoning?
- Do LLMs understand implicit warrants in reasoning chains?
- Why can LLMs identify argument structure but not check warrants?
- What specific linguistic features cause LLMs to fail at trivial entailment?
- How does the LLM Fallacy prevent users from noticing cognitive debt accumulating?
- Do language models behave differently on contested beliefs versus factual claims?
- Why do LLMs fail at counterfactual reasoning despite factual knowledge?
- Can LLM reasoning traces be validated against actual population reasoning?
- How can entailment benchmarks separate genuine reasoning from memorization effects?
- How does fine-tuning on natural language inference affect fallacy susceptibility?
- What makes procedural knowledge in documents generalize better than facts?
- Do models with unfilled memorization capacity appear to generalize falsely?
- Why is extracting training data insufficient proof that models memorize?
- Can models internalize retrieved context as static parametric knowledge?
- Why does in-weight memorization fail compared to tool-based fact access?
- Why does semantic deduplication reduce memorization in fine-tuned models?
- What makes factual memorization less efficient than tool-based retrieval?
- Why do LLMs explain evidence accurately while missing its implications?
- Why does probability of text completion not equal knowledge value?
- Why does LLM compression eliminate causal grounding in conceptual representations?
- Do LLMs show stronger reasoning about causality than about temporal ordering?
- Can encoder models match human conceptual structure better than larger language models?
- Can language models translate theorems faithfully without semantic loss?
- Can simple structure perturbations reliably expose memorization in reasoning models?
- What distinguishes reasoning fixation from belief distortion in memory traps?
- Do base models contain latent reasoning that minimal training can unlock?
- Do base models truly possess latent reasoning capability?
- Does the base model already contain latent reasoning capability?
- What mechanisms activate latent reasoning capabilities already present in base models?
- What latent reasoning capability do base models already possess before training?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do language models actually use their encoded knowledge?
Probes can detect that LMs encode facts internally, but do those encoded facts causally influence what the model generates? This explores the gap between knowing and doing.
the complementary failure: encoded knowledge that doesn't influence generation; attestation is memorized knowledge that influences generation in the wrong direction
-
Why do language models ignore information in their context?
Explores why language models sometimes override contextual information with prior training associations, and whether providing more context can solve this problem.
same mechanism: parametric associations override in-context information
-
Does fine-tuning on NLI teach inference or amplify shortcuts?
When LLMs are fine-tuned on natural language inference datasets, do they learn genuine reasoning abilities or become better at exploiting statistical patterns in the training data? Understanding this distinction matters for assessing model capabilities.
fine-tuning makes attestation-related frequency bias worse, not better
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Explicit Inductive Inference using Large Language Models
- Neutralizing Bias in LLM Reasoning using Entailment Graphs
- Sources of Hallucination by Large Language Models on Inference Tasks
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Minds versus Machines: Rethinking Entailment Verification with Language Models
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Premise Order Matters in Reasoning with Large Language Models
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
Original note title
llm entailment predictions are bound to hypothesis attestation rather than premise-hypothesis inference