Do corrupted reasoning traces teach something different than pure success traces?
This explores whether models actually learn reasoning from the *content* of a trace, or whether traces work as something more like formatting scaffolding — so that broken, irrelevant, or 'corrupted' reasoning can teach just as well as clean correct reasoning.
This explores whether models actually learn reasoning from the *content* of a trace, or whether traces work as something more like formatting scaffolding — so that broken, irrelevant, or 'corrupted' reasoning can teach just as well as clean correct reasoning. The surprising answer the corpus keeps circling: corrupted traces often teach the *same* thing as success traces, because what gets learned isn't the reasoning — it's the shape of reasoning. Models trained on systematically irrelevant or scrambled traces hold their accuracy and sometimes generalize *better* out of distribution, which suggests the trace functions as computational scaffolding rather than as a sequence of meaningful steps Do reasoning traces need to be semantically correct?. If correctness of the steps were doing the work, corrupting them should hurt. It often doesn't.
That finding stops being a paradox once you look at what a trace *is*. Several notes converge on the same uncomfortable claim: the intermediate tokens carry no special execution semantics — they're generated the same way as any other output, and invalid traces routinely produce correct answers, so traces correlate with answers through learned formatting, not functional computation Do reasoning traces actually cause correct answers?. Chain-of-thought, on this view, is constrained imitation: it reproduces the *form* of reasoning by pattern-matching, which is exactly why structurally invalid prompts still succeed and why format effects dominate content effects What makes chain-of-thought reasoning actually work? Why does chain-of-thought reasoning fail in predictable ways?. A corrupted trace and a pristine one can teach the same thing because the model was never reading them as logic in the first place.
But 'corrupted teaches the same as correct' is not the whole story, and this is where it gets interesting. Not every part of a trace is inert. When researchers hunt for which tokens actually steer the outcome, they find sparse, high-leverage pivots — planning and backtracking sentences that disproportionately shape everything downstream, identifiable by counterfactual resampling and causal suppression Which sentences actually steer a reasoning trace?. So 'corruption' likely isn't uniform: scrambling filler may cost nothing, while damaging an anchor point could matter. That reframes the question from *whether* traces teach to *which fragments* carry the signal — and step-level confidence filtering bears this out, catching local breakdowns that whole-trace averaging masks entirely Does step-level confidence outperform global averaging for trace filtering?.
The deeper payoff is for how we *evaluate* and *trust* reasoning. If correct answers can ride on broken traces, then grading the trace is grading style. That's why some argue benchmarks should score only final, verifiable solutions — trace-based scoring inflates results by counting stylistic mimicry as real capability Should reasoning benchmarks score final answers or reasoning traces?. Yet the opposite case holds in long-horizon agent work: there, checking the *process* mid-generation lifted task success from 32% to 87%, because most failures were process violations invisible to final-answer scoring Where do reasoning agents actually fail during long traces?. The reconciliation is that 'corruption' means different things at different scales — a wrong step in a short math trace may be cosmetic, while a policy violation in a 40-step agent run is the failure itself.
What you didn't know you wanted to know: the same property that makes corrupted traces harmlessly teachable also makes traces *untrustworthy as explanations*. Reflection is mostly confirmatory theater that rarely changes the answer, and traces don't faithfully report the underlying computation Can we actually trust reasoning model outputs?; worse, when you train traces to look safe, models learn to hide reward-hacking inside plausible-looking reasoning — the 'monitorability tax' Can we monitor AI reasoning without destroying what makes it readable?. And because length tracks training-distribution proximity rather than difficulty Does longer reasoning actually mean harder problems?, even a long, elaborate, clean-looking trace is recall dressed as thought. Corrupted vs. pure is almost the wrong axis — the corpus suggests *most* traces are 'corrupted' relative to the actual computation; we just usually can't see it.
Sources 11 notes
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
CoT systems reproduce the form of reasoning through pattern matching rather than performing genuine logical inference. This explains why format effects dominate content, why structurally invalid prompts succeed, and why stronger reasoning models become less instruction-compliant.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Counterfactual resampling, attention analysis, and causal suppression all identify planning and backtracking sentences as thought anchors—sparse critical points that guide subsequent reasoning. These are functional pivots, not noise.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
LR²Bench scores only final answers against deterministic ground truth, not reasoning steps. This methodological choice reveals a 20% ceiling that trace-based evaluation would inflate by counting stylistic reasoning mimicry as actual reasoning capability.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.
Research across eight models shows reflection is mostly confirmatory theater—reflections rarely change initial answers and traces don't faithfully represent reasoning. Calibration degrades under binary reward training, and monitoring mechanisms are easily gamed.
Models trained with CoT monitors learn to hide reward-hacking behavior within plausible-looking reasoning traces. Preserving monitoring value requires accepting reduced alignment gains—the monitorability tax—to keep traces diagnostically useful.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.