Why do LLMs fail at counterfactual reasoning despite factual knowledge?

This explores why LLMs can hold a fact yet fail to reason from it — and the most useful thing the corpus does is reframe the question. The failure usually isn't missing knowledge; it's a broken link between knowing and using. The sharpest demonstration is Can LLMs understand concepts they cannot apply?: a model can explain a concept correctly, fail to apply it, and even recognize its own failure — a triple pattern that suggests explanation and execution run on functionally disconnected pathways. The same shape shows up behaviorally in Why do language models fail to act on their own reasoning?, where models produce a correct rationale 87% of the time but act on it only 64% — they know what to do and do something else.

When the task asks a model to hold a fact against a contradicting premise, this gap becomes the failure. Why do language models accept false assumptions they know are wrong? shows models accommodating false assumptions they demonstrably know are wrong — a false presupposition pulls harder toward agreement than correct knowledge pulls toward correction. That's the counterfactual problem in miniature: reasoning against the grain of a stated premise requires actively deploying knowledge, not just possessing it. A related mechanism is attestation bias in Do LLMs predict entailment based on what they memorized?: models judge whether a conclusion follows by checking if it looks like something they've seen, not by tracing the premise — so a counterfactual premise, which by construction departs from training data, gets overridden by what was memorized.

Underneath sits a deeper claim about what kind of reasoner an LLM is. Do large language models reason symbolically or semantically? finds that when you strip the familiar semantic content out of a task and leave only the logical structure, performance collapses even with the correct rules supplied in context. Counterfactual reasoning demands exactly this — manipulating a structure that conflicts with real-world semantics — and a system leaning on token associations and parametric commonsense has nothing to grip. Do language models fail at identifying unstated preconditions? adds a complementary angle: models fail not from lacking world knowledge but from not bringing the relevant background conditions forward as constraints; forcing explicit enumeration of preconditions lifts accuracy from 30% to 85%.

What makes this genuinely interesting is that the failures aren't uniformly "LLMs are worse at reasoning." Why do LLMs fail at simple deductive reasoning? shows models beating humans at stitching information across many sentences while losing at simple deduction — capability type, not difficulty, predicts who wins. And Do language models fail reasoning tests that humans pass? together with Do large language models make the same causal reasoning mistakes as humans? argues humans and models fail along the *same* content-sensitivity axis — both reason better when content is familiar and stumble when it isn't. That's the thing you didn't know you wanted to know: the counterfactual weakness may not mark LLMs as defective reasoners so much as expose that their reasoning, like ours, is bound to remembered content — they just lack the human capacity to override it on demand.