How brittle are chain-of-thought exemplars across order and complexity?
This explores how fragile the worked examples we give models for chain-of-thought (CoT) prompting really are — whether shuffling their order or mismatching their difficulty quietly wrecks performance.
This explores how fragile the worked examples we give models for chain-of-thought prompting are — whether reordering them or mismatching their complexity quietly tanks accuracy. The short answer from the corpus: alarmingly so. One study isolates four dimensions of brittleness at once — order, complexity, diversity, and authorship — and finds they all bite: simply reshuffling exemplars swings results by a few points, mismatching example complexity to the problem hurts, and the same task written up by a different annotator can move accuracy by as much as 28% Why do chain-of-thought examples fail across different conditions?. A separate line of work confirms the order effect from another angle: where you place a demonstration can swing accuracy ~20%, and the *format* of the examples shapes the model's reasoning strategy 7.5× more strongly than the actual domain of the problem What makes chain-of-thought reasoning actually work?. So the curation choices that feel cosmetic — which example goes first, how it's phrased — turn out to be load-bearing.
The reason this brittleness exists points at something deeper than prompt hygiene. Multiple notes converge on the claim that CoT isn't genuine step-by-step inference at all — it's constrained imitation of the *form* of reasoning, pattern-matched from training data Why does chain-of-thought reasoning fail in predictable ways? What makes chain-of-thought reasoning actually work? Does chain-of-thought reasoning reveal genuine inference or pattern matching?. The cleanest evidence: logically *invalid* CoT exemplars perform almost as well as valid ones on hard benchmarks Does logical validity actually drive chain-of-thought gains?. If the model were following the logic, broken logic would break it. Instead it's copying a shape — which is exactly why surface features like order and phrasing matter so much, and why correctness of the example matters so little.
The complexity dimension has a subtler twist worth knowing. It's tempting to assume harder problems simply need longer or more elaborate exemplars, but the corpus complicates this. Optimal CoT length follows an inverted-U — accuracy peaks at a *middle* length, and more capable models actually prefer *shorter* chains, with simplicity emerging naturally from RL reward signals rather than being trained in Why does chain of thought accuracy eventually decline with length?. And the intuition that 'longer reasoning = harder problem' is false out of distribution: trace length tracks how close a problem sits to the training distribution, not its intrinsic difficulty Does longer reasoning actually mean harder problems?. So 'matching exemplar complexity to the problem' is harder than it sounds, because length isn't a clean proxy for difficulty in the first place.
Here's the part you might not expect: the real fault line isn't complexity at all — it's *familiarity*. One study shows reasoning models don't break at some complexity threshold; they break at instance-novelty boundaries, succeeding on any chain (long or short) that resembles something seen in training Do language models fail at reasoning due to complexity or novelty?. Token-level analysis backs this up: local memorization — predicting from the immediately preceding tokens — drives up to 67% of reasoning errors, and it gets worse precisely as complexity and distribution shift increase Where do memorization errors arise in chain-of-thought reasoning?. Brittleness 'across complexity' is really brittleness across *distance from training data wearing a complexity costume.*
If there's a constructive thread to pull, it's that the structure of a good CoT isn't arbitrary even if exemplar selection is fragile. One framing models effective long CoT as having 'molecular bond' structure — distinct interaction types (deep reasoning, self-reflection, self-exploration) that form stable mixtures, and that *destabilize* when you blend traces from different teachers Does long chain of thought reasoning follow molecular bond patterns? — which is another face of the authorship/diversity brittleness above. Meanwhile, test-time methods can prune ~75% of reasoning steps with no accuracy loss by keeping only the steps that actually get attended to downstream Can reasoning steps be dynamically pruned without losing accuracy?. Read together, the corpus suggests the path away from brittle hand-curated exemplars runs through letting the model find its own stable, shorter reasoning form rather than over-engineering the examples we hand it.
Sources 12 notes
Human-written CoT exemplars degrade performance when reordered (3.3% swings), mismatched to problem complexity, lacking diversity, or written by different annotators (up to 28.2% variance). These four dimensions compound, making manual exemplar curation unreliable across tasks.
Research shows training format shapes reasoning strategy 7.5× more than domain, demo position swings accuracy 20%, and invalid CoT prompts work as well as valid ones. CoT is pattern-guided generation, not formal logic.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
CoT systems reproduce the form of reasoning through pattern matching rather than performing genuine logical inference. This explains why format effects dominate content, why structurally invalid prompts succeed, and why stronger reasoning models become less instruction-compliant.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.
Deep-Reasoning (covalent), Self-Reflection (hydrogen bonds), and Self-Exploration (van der Waals forces) form stable distributions in effective Long CoT. Mixing these stable structures from different teachers destabilizes learning despite matched performance metrics.
The PI framework categorizes reasoning into six types and uses attention maps to identify that verification and backtracking steps receive minimal downstream attention. Selecting only high-attention steps preserves accuracy while cutting reasoning length substantially.