How do prior errors in reasoning context amplify future mistakes?
This explores the self-conditioning trap — how an error already sitting in a model's context history makes the next error more likely, and what (if anything) breaks the chain.
This explores the self-conditioning trap: how mistakes already present in a model's context don't just sit there inertly but actively bias what comes next. The clearest account in the corpus is the self-conditioning effect Do models fail worse when their own errors fill the context?, which finds that once a model's own errors fill its context, performance degrades non-linearly on long-horizon tasks — each contaminated step tilts the next one further off, so the damage compounds rather than averages out. Strikingly, making the model bigger doesn't rescue it; only test-time 'thinking' compute, which keeps the error-laden history from steering fresh reasoning, reduces the effect.
Why would a model lean *into* its own mistakes? A mechanistic answer comes from token-level memorization research Where do memorization errors arise in chain-of-thought reasoning?, which shows that 'local' memorization — predicting the next token mostly from the immediately preceding tokens — accounts for up to 67% of chain-of-thought errors, and gets worse as complexity and distribution shift increase. If the model is heavily conditioned on what just came before, a wrong preceding step becomes a strong (wrong) prior for the next one. That dovetails with the view that chain-of-thought is constrained pattern-matching, not genuine inference Why does chain-of-thought reasoning fail in predictable ways?: the model is matching the *shape* of plausible reasoning, so a malformed earlier step supplies a malformed template to continue from. It even explains the unsettling finding that deliberately corrupted reasoning traces train about as well as correct ones Do reasoning traces need to be semantically correct? — if traces are computational scaffolding rather than meaning, the model has no built-in sense that a prior step was *wrong* to recoil from.
The amplification isn't only about errors the model generated. Reasoning accuracy collapses just from longer context — dropping from 92% to 68% with a few thousand tokens of padding, far below the context-window limit Does reasoning ability actually degrade with longer inputs?. So a context bloated with prior (possibly flawed) reasoning is doubly harmful: it both lengthens the input and seeds it with bad priors. And when the context contains something the model 'knows' is false, it often can't override it — language models accommodate false presuppositions even while possessing the correct fact Why do language models accept false assumptions they know are wrong?, and parametric training associations frequently win out over what's actually in the context Why do language models ignore information in their context?. A prior error in context is exactly a false presupposition the model then builds on.
The corpus also points at what interrupts the cascade: external grounding. Interleaving reasoning with real-world feedback — querying a tool or environment between steps — prevents error propagation by injecting a correction signal each step, outperforming pure chain-of-thought by 10–34% on knowledge tasks Can interleaving reasoning with real-world feedback prevent hallucination?. The throughline across all of these: errors amplify because the model treats its own recent output as authoritative context to continue from, and the fixes that work (test-time deliberation, external grounding) share a single move — they stop the contaminated history from being the only thing the next step is conditioned on.
Sources 8 notes
Error accumulation in context causes non-linear performance degradation in long-horizon tasks. Model scaling does not fix this; only test-time compute through thinking models reduces the effect by preventing error-contaminated context from biasing reasoning.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
ReAct demonstrates that alternating verbal reasoning with external tool queries (Wikipedia API, environment interaction) prevents error propagation by injecting real-world feedback at each step. On knowledge-intensive and interactive tasks, this approach outperforms pure chain-of-thought and reinforcement learning by 10-34% absolute accuracy.