Why does teacher forcing fail to capture long-range dependencies?
This explores why training a model to predict the next token from ground-truth context (teacher forcing) tends to capture local, surface-level patterns rather than the long-range structural dependencies that span a whole sentence or reasoning chain — the corpus doesn't have a paper on teacher forcing by name, but it has a lot on the failure mode it produces.
This explores why training a model to always predict the next token while conditioned on the *correct* prior tokens (teacher forcing) seems to leave it good at local prediction but weak on dependencies that stretch across long spans. No single note in the corpus tackles teacher forcing head-on, but several converge on the same underlying story from different angles, and read together they're more illuminating than a single paper would be.
The sharpest evidence is linguistic. When you measure what next-token training actually learns about grammar, competence degrades predictably as structures get deeper — simple sentences are handled well, but recursion and embedded clauses fail consistently Does LLM grammatical performance decline with structural complexity? Why do large language models fail at complex linguistic tasks?. Embedded clauses and recursion *are* long-range dependencies — the word that resolves a structure can sit far from the word that opened it. The diagnosis in these notes is exactly the teacher-forcing critique: the model learned surface heuristics that work locally, not the structural rules that bind distant tokens together. A related note shows the same surface-over-structure pattern with presupposition triggers and non-factive verbs, where the model reads them as local cues instead of computing how they flip meaning across the sentence Why do embedding contexts confuse LLM entailment predictions?.
There's a deeper architectural reason this happens, and it's the most counterintuitive thread here: token-by-token generation has no retraction primitive. Once a token is emitted it can't be taken back, so the model can't revise an early commitment in light of a later constraint — which is precisely what long-range dependency means Why does autoregressive generation fail at constraint satisfaction?. This shows up as a hard ceiling on tasks requiring genuine backtracking, where frontier reasoning models stall at 20–23% Can reasoning models actually sustain long-chain reflection?. The same one-pass-forward limitation explains why models fake iterative procedures — they recognize a problem as template-similar and emit plausible values rather than actually carrying state across steps Do large language models actually perform iterative optimization?.
The most direct echo of "teacher forcing" as a *training* concept comes from a distillation note: when a teacher is conditioned on the correct answer, it produces confident, concise traces that students inherit — and that very confidence suppresses the uncertainty and exploration the student would need to generalize out of distribution Does richer teacher context hurt student generalization?. That's the same trap one level up: optimizing against ground-truth context buys in-domain fluency at the cost of the wider-range competence that only shows up when the easy local signal is removed.
What's quietly interesting is where the corpus points for escape routes. Several notes suggest the fix isn't a better next-token objective but *changing what the model operates over*: storing the long prompt as an external environment to query rather than attend to Can models treat long prompts as external code environments?, reframing the long-context limit as a compute problem of consolidating evicted context into weights rather than a memory one Is long-context bottleneck really about memory or compute?, and noting that simply extending context length doesn't buy structured relational reasoning Can long-context LLMs replace retrieval-augmented generation systems?. The through-line: long-range dependency is a structural and procedural capability, and a training scheme that rewards locally-correct next tokens doesn't reliably install it.
Sources 10 notes
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
LLMs treat presupposition triggers and non-factive verbs as surface cues rather than computing their opposite semantic effects on entailments. This structural failure persists across prompts and models, suggesting models rely on surface patterns instead of structural analysis.
The performance ceiling on constraint satisfaction problems is not a model-quality issue but an architectural limitation: autoregressive transformers cannot retract emitted tokens, while CSP solvers fundamentally depend on discarding invalid partial assignments. Symbolic solver integration works because it supplies what the architecture lacks.
DeepSeek-R1 and o1-preview achieve only 20-23.6% exact match on 850 constraint satisfaction problems requiring genuine backtracking. This ceiling reveals that reflective reasoning fluency does not translate to actual problem-solving competence on unfamiliar instance structures.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Recursive Language Models store long prompts in a Python REPL and query them via code execution, avoiding attention degradation. RLMs outperform base models even on shorter prompts while handling inputs two orders of magnitude beyond context windows.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.