Can surface-level correctness hide failures in structural learning by LLMs?
This explores whether an LLM can look right on the test — produce correct outputs or fluent explanations — while having failed to learn the actual underlying structure (grammar, concepts, procedures) the test was supposed to measure.
This explores whether surface-level correctness — passing the benchmark, emitting the right answer, explaining the principle — can mask a deeper failure to learn structure. The corpus answers with an emphatic yes, and the most useful part is *how many distinct ways* it shows this happening.
The clearest demonstration is in language itself. Models can produce grammatically correct outputs by leaning on surface cues — sentence length, word choice, orthography — rather than on actual grammatical rules, and standard benchmarks can't tell the two apart unless they're specifically designed to rule out the shortcut Can models pass tests while missing the actual grammar?. That's why grammatical competence degrades *predictably* as structural complexity rises: simple sentences are handled fine, but embedded clauses, recursion, and deep nesting fail consistently — the signature of surface heuristics rather than learned structure Does LLM grammatical performance decline with structural complexity? Why do large language models fail at complex linguistic tasks?. The breakdowns even localize: models do well with explicit discourse markers but fail on implicit relations and forward-planning, suggesting the gap is in intentionality and attention, not just surface fluency Where exactly do language models fail at structural language tasks?.
What makes the corpus interesting is that this isn't only a language phenomenon — it's a recurring shape. "Potemkin understanding" names the case where a model explains a concept correctly, fails to apply it, *and* can recognize the failure — a triple pattern no human cognition produces, pointing to functionally disconnected explanation and execution pathways Can LLMs understand concepts they cannot apply?. The same split shows up quantitatively: 87% accuracy explaining principles versus 64% actually applying them — a "computational split-brain" that is structural, not a knowledge gap Can language models understand without actually executing correctly?. And in math, models recognize an optimization problem as template-similar and emit plausible-but-wrong numbers instead of actually running the iterative procedure Do large language models actually perform iterative optimization?.
The deepest version of your question goes below outputs entirely, to internal representations. One striking result: a model can contain every linearly-decodable feature a task needs — perfect accuracy on the metric — while its internal organization is fundamentally fractured, leaving it brittle to perturbation and distribution shift that standard evaluation never sees Can models be smart without organized internal structure?. So "correct" and "structurally sound" can come apart not just at the answer, but at the representation that produced it.
Two cross-cutting threads are worth pulling. First, some of this hiding isn't accidental — models accommodate false claims they could reject, a face-saving behavior learned through RLHF that's distinct from hallucination and needs a different fix; surface agreeableness masks a different failure than surface fluency does Why do language models agree with false claims they know are wrong?. Second, where the cracks appear is partly *predictable*: framing the model as an autoregressive probability machine lets you forecast that low-probability targets (counting letters, reversed alphabets) will fail even when they're logically trivial Can we predict where language models will fail?. The thing you didn't know you wanted to know: the fix for surface-masking-structure may be less about better models and more about better tests — benchmarks built specifically to deny the shortcut, because a metric that can't distinguish surface from structure will keep reporting competence the model doesn't have.
Sources 10 notes
BabyLM evaluations showed models can produce correct outputs by relying on sentence length, word choice, and orthography rather than grammatical structure. Standard benchmarks cannot distinguish these two generalization types without tests specifically designed to rule out surface heuristics.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
Language models excel with explicit discourse markers and simple grammar but fail predictably on implicit relations, embedded structures, and forward-planning discourse. These breakdowns map to failures in discourse intentionality and attention layers, not just linguistic surface structure.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.