Why do invalid prompts produce reasoning traces as effectively as valid ones?
This explores a striking finding across the corpus: when researchers feed models logically broken or deliberately scrambled reasoning examples, the resulting chain-of-thought works almost as well as valid reasoning — which tells us something surprising about what reasoning traces actually do.
This explores a striking finding across the corpus: invalid or corrupted reasoning examples teach and perform nearly as well as correct ones, and the explanation is that the trace is doing a different job than we assume. The short answer is that chain-of-thought traces work mostly by their *form*, not their *content*. When researchers fed models logically invalid CoT exemplars on hard benchmarks, accuracy barely budged compared to valid ones — the gains come from structural properties of the reasoning format, not from the logic being sound Does logical validity actually drive chain-of-thought gains?. Go a step further and deliberately corrupt the traces with irrelevant steps, and models still maintain accuracy, sometimes even generalizing *better* out of distribution — which suggests the trace functions as computational scaffolding, a kind of 'thinking-shaped' workspace, rather than a sequence of meaningful inferential moves Do reasoning traces need to be semantically correct?.
Sources 9 notes
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
Counterfactual resampling, attention analysis, and causal suppression all identify planning and backtracking sentences as thought anchors—sparse critical points that guide subsequent reasoning. These are functional pivots, not noise.
Applying Toulmin's argument model as explicit prompting steps (CQoT) improves LLM reasoning by forcing models to identify warrants and backing rather than skipping implicit premises. The method catches failures that standard chain-of-thought prompting allows.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.