How does the knowing-doing gap relate to Potemkin understanding?
This explores two named failure modes — the 'knowing-doing gap' (a model can state the right principle but won't carry it out) and 'Potemkin understanding' (a facade of comprehension that collapses on use) — and asks whether they're the same phenomenon, different angles on it, or causally linked.
This explores how two terms in the corpus — the knowing-doing gap and Potemkin understanding — name what turns out to be the same underlying defect from two vantage points. The shared diagnosis is structural, not informational: the model isn't missing knowledge, its explanation pathway and its execution pathway are simply wired apart. Can language models understand without actually executing correctly? makes this concrete with a split-brain measurement — 87% accuracy when articulating a principle, 64% when acting on it — and argues the gap is a wiring problem, not a gap in what the model 'knows.' Can LLMs understand concepts they cannot apply? looks at the same disconnect from the comprehension side: a model explains a concept correctly, fails to apply it, and can even recognize its own failure — a triple pattern no human cognition produces, which is exactly why the 'understanding' is a Potemkin facade.
So the relationship is less 'A causes B' and more 'A is what you call B when you're standing on the action side.' Potemkin understanding describes the appearance (fluent explanation that's hollow); the knowing-doing gap describes the mechanism (declarative knowledge that never reaches procedural behavior). Both reject the intuitive read — that the model just hasn't learned the material — and relocate the problem to the architecture connecting what's known to what's done.
Why might the wiring be split in the first place? Why does reasoning training help math but hurt medical tasks? offers a physical hint: knowledge retrieval lives in lower layers and reasoning adjustment in higher ones, a separation that explains why training one capacity can quietly degrade the other. Do language models understand in fundamentally different ways? deepens the picture — understanding comes in tiers (concepts as directions, world-state as factual links, principles as compact circuits), and higher tiers sit on top of lower-tier heuristics rather than replacing them. That patchwork is fertile ground for a Potemkin: the explanation can ride a shallow heuristic while the circuit that would actually execute the principle never fires. What do language models actually know? frames the whole thing as the difference between tracking statistical regularities and possessing genuine epistemic competence — the gap between pattern and knowledge that both failure modes are local symptoms of.
The most useful turn is that the gap may be bridgeable, which tells you it's a connection problem rather than a missing-capability problem. Can language modeling close the knowing-doing gap in AI? shows that when a model's policy is expressed in language and then refined by environmental feedback, declarative and procedural knowledge start to unify — the 'knowing' is forced to cash out as 'doing.' That resonates with Do base models already contain hidden reasoning ability?, where the competence is shown to already be latent and merely needs eliciting, and with Can modular cognitive tools unlock reasoning without training?, where isolating each reasoning operation into a discrete call lifts performance without new training — structurally compensating for the missing internal link.
The thing worth walking away with: a Potemkin isn't a lie the model tells and the knowing-doing gap isn't ignorance. Both are signatures of an architecture where saying and doing run on separate tracks — which is unsettling (the fluent answer is no guarantee of competence) and oddly hopeful (feedback that ties language to consequences can close the gap without teaching the model anything new).
Sources 8 notes
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Two-phase inference model shows knowledge retrieval operates in lower network layers while reasoning adjustment happens in higher layers. This separation explains why reasoning training improves math but can degrade knowledge-intensive domains like medicine.
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
LLMs achieve high fidelity in capturing language patterns yet show systematic, structurally specific failures—hallucination, reasoning collapse, and premise-sensitivity. The gap between statistical tracking and real knowledge is measurable and unavoidable.
Think-In Games demonstrates that when LLMs generate language-guided policies refined by environmental feedback, they develop procedural competence while retaining explainability. The approach dramatically reduces data demands and makes agent reasoning transparent at every step.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Four cognitive tools implemented as sandboxed LLM calls improved GPT-4.1 on AIME2024 from 26.7% to 43.3% without any RL training. Modularity enforces operation isolation that pure prompting cannot guarantee, eliciting pre-existing reasoning capability.