How do we distinguish knowledge encoding from knowledge usage in models?
This explores the gap between what a model stores in its representations and what it actually puts to work when generating an answer — and the methods researchers use to tell the two apart.
This explores how researchers separate two things that look identical from the outside: knowledge a model *holds* in its internal representations, and knowledge it *uses* to shape what it actually says. The starting point in the corpus is blunt — these are genuinely distinct processes. Models routinely encode a fact in their internal states while that fact fails to causally influence the output Do language models actually use their encoded knowledge?. So measuring whether a model 'knows' something by probing its representations tells you about encoding, not usage. The two can come apart.
The sharpest method for distinguishing them is methodological: representational analysis alone only finds correlations, so you can locate a feature that *looks* like stored knowledge without showing it does any work. To prove usage you need causal analysis — intervene on the representation and watch whether the behavior changes Can we understand LLM mechanisms with only representational analysis?. Encoding is what you see when you read the internal state; usage is what you see when you perturb it and the output moves. That pairing is the operational test the corpus keeps returning to.
What makes this more than a technicality is how often the gap is the *cause* of failure. 'Potemkin understanding' is the cleanest case — a model explains a concept correctly, then fails to apply it, and can even recognize its own failure, which points to functionally disconnected explanation and execution pathways rather than a simple knowledge gap Can LLMs understand concepts they cannot apply?. Relatedly, reasoning often collapses not because the knowledge is absent but because an inference bottleneck blocks its activation; a nudge to enumerate preconditions recovers several points of accuracy, recovering knowledge that was there all along Why do language models fail to use knowledge they possess?. Encoded but unused is a recurring, measurable state.
Here's the part you might not expect: usage can be actively *suppressed*. In models trained with hidden chain-of-thought, the correct answer is computed in the earliest layers and then deliberately overwritten in later layers to produce format-compliant filler — the reasoning is fully recoverable from lower-ranked predictions Do transformers hide reasoning before producing filler tokens?. And models even encode a kind of meta-knowledge: an entity-recognition mechanism that tracks whether they know a fact at all, which causally steers refusal versus hallucination Do models know what they don't know?. So 'usage' isn't one thing — there's the knowledge, the decision to deploy it, and a self-assessment riding on top.
The encoding/usage split also cuts along a deeper seam in *what kind* of knowledge is involved. Factual recall depends on narrow, document-specific memorization, while reasoning draws on broad, transferable procedural knowledge — two different storage-and-retrieval regimes that behave differently under use Does procedural knowledge drive reasoning more than factual retrieval?. Layer on the finding that reasoning traces are often stylistic mimicry rather than a faithful record of computation Do reasoning traces show how models actually think?, and the broader picture sharpens: understanding in these models is a patchwork where higher-tier mechanisms coexist with lower-tier heuristics rather than replacing them Do language models understand in fundamentally different ways?. The lesson for anyone evaluating a model: a correct answer doesn't prove the knowledge was used, and a wrong answer doesn't prove it was missing — you have to intervene to know which.
Sources 9 notes
Multiple studies confirm that language models can encode facts in their representations while those facts fail to causally affect downstream outputs. Encoding and usage are distinct processes.
Representational analysis alone identifies correlations without causation; causal analysis alone shows behavioral effects without explaining them. Only paired methods—locating candidate features representationally, then verifying causally—produce complete mechanistic claims.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Models possess relevant knowledge but fail to activate it without explicit prompting. Adding subtle emphasis recovers 15.3 percentage points accuracy, and forcing enumeration of preconditions recovers 6-9 points, showing the bottleneck is in constraint inference, not storage.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Sparse autoencoders revealed that language models develop causal mechanisms for detecting whether they know facts about entities. These mechanisms actively steer both hallucination and refusal behavior, and persist from base models into finetuned chat versions.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.