How do pretrained language models represent inferential patterns versus lexical and positional cues?
This explores whether language models actually encode logical inference (does the premise support the conclusion?) or whether they lean on surface signals — which words appear, how often they've been seen, where they sit in the sequence.
This question is really asking: when an LLM looks like it's reasoning, is it tracking the logical relationship between statements, or is it riding surface cues — familiar words, memorized phrases, token order, and raw probability? The corpus leans hard toward the second answer, and the most direct evidence is the discovery of 'attestation bias.' Models predicting whether one sentence entails another mostly check whether the conclusion *looks like something they saw in training*, not whether the premise actually supports it — swap in a random, irrelevant premise and the model still confidently says 'entails' as long as the hypothesis is familiar Do LLMs predict entailment based on what they memorized?. The inferential relationship is the thing it's *supposed* to represent, and it's the thing it quietly skips.
That pattern shows up again when researchers strip the familiar meaning out of a reasoning task. Give a model correct logical rules but decouple them from sensible semantics, and performance collapses — meaning it was leaning on commonsense word associations the whole time, not manipulating the rules symbolically Do large language models reason symbolically or semantically?. The same dependence on surface form explains why models stumble as sentences get structurally deeper: they reliably misread embedded clauses and complex phrases, capturing statistical regularities of how words co-occur rather than the grammatical scaffolding underneath Why do large language models fail at complex linguistic tasks?. And one elegant framing ties it together — treat the model as a pure next-token probability machine, and you can *predict in advance* which logically-trivial tasks it will flunk, simply because the correct answer is a low-probability string Can we predict where language models will fail?. Inference loses to position and probability.
The interesting wrinkle — the thing you might not expect — is that the inferential capability sometimes *is there*, just buried under the surface machinery. Logit-lens work shows models can compute a correct answer in their early layers and then actively overwrite it in later layers to emit format-compliant filler instead; the reasoning representation exists but gets suppressed in favor of producing the expected-looking output Do transformers hide reasoning before producing filler tokens?. Relatedly, when you force the surface process to slow down and externalize — explicit chain-of-thought — models can build genuine syntactic trees and metalinguistic analyses they fail at in one shot Can language models actually analyze language structure?. So inferential structure isn't simply absent; it competes with, and usually loses to, the strong pull of lexical familiarity and positional output habits.
That competition is itself a recurring corpus theme. Strong parametric priors — the lexical associations baked in during training — routinely override what's actually present in the context window, so much so that prompting alone can't fix it; you have to intervene in the representations directly Why do language models ignore information in their context?. There's even a representational ceiling: on a task like classifying argument schemes (which demands tracking inferential structure), models flatline unless they're large *and* given worked examples, hinting that the inference-representing capacity only emerges with scale Can large language models classify argument schemes reliably?.
If you want the deeper 'why,' the corpus points to a foundational claim: a system trained purely on form-to-form prediction has no independent channel to the meanings that ground real inference, so what it represents is relational structure compressed from text rather than the world the text refers to Can language models learn meaning from text patterns alone? Can language models learn meaning without engaging the world?. Read that way, the lexical-and-positional cues aren't a shortcut the model takes *instead* of inference — they're the only material it was ever given to build inference out of.
Sources 10 notes
McKenna et al. (2023) identified attestation bias: LLMs predict entailment based on whether the hypothesis appears in training data, not whether the premise actually supports it. Random premise experiments show models maintain high entailment predictions when hypotheses are attested, proving they respond to memorized propositions rather than premise-hypothesis relationships.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
OpenAI's o1 model successfully constructs syntactic trees and phonological generalizations through explicit step-by-step reasoning, revealing that LLM linguistic capability extends far beyond behavioral language tasks to genuine language analysis.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Zero-shot prompting fails uniformly across models. Few-shot with scheme descriptions helps, but only larger models exceed F1 0.55, with Claude reaching 0.65. Smaller models plateau around 0.53, suggesting a representational capacity threshold.
Bender & Koller argue that meaning requires the relation between expressions and communicative intents. Since LLMs are trained only on form-to-form prediction with no access to shared attention or intent, they cannot reconstruct the meaning that grounds language.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.