Does the heuristic dominance ratio vary predictably across model architectures?
This explores whether the degree to which models lean on shortcut pattern-matching (heuristics) rather than genuine reasoning is a stable, predictable quantity that scales across different model families and architectures — a term the corpus doesn't use directly, so I'm reading it as 'does heuristic-vs-genuine-reasoning reliance behave lawfully across models.'
This explores whether models' reliance on shortcut heuristics over genuine inference follows predictable patterns across architectures — and the corpus has no note using the phrase "heuristic dominance ratio," so the honest answer is that this exact metric isn't something the collection tracks. But the conceptual territory underneath it — how much models pattern-match instead of reason, and whether that scales lawfully — runs through several notes worth pulling together.
Start with the claim that the heuristic *is* the dominant mode. The corpus argues chain-of-thought reasoning is "constrained imitation, not abstract inference" — models pattern-match the *structure* of reasoning rather than performing it, which is why failures are distribution-bounded and why structural coherence matters more than whether the content is correct Why does chain-of-thought reasoning fail in predictable ways?. A companion result sharpens this: trace length correlates with difficulty only in-distribution and decouples entirely out-of-distribution, meaning the visible reasoning mostly reflects recall of training schemas, not adaptive computation Does longer reasoning actually mean harder problems?. So "heuristic dominance" isn't a fringe failure — it's closer to the default operating mode.
Now the predictability question. The strongest evidence that model internals behave *lawfully* across architectures is that RL induces 5–30% parameter sparsity in nearly full-rank subnetworks that are nearly identical across random seeds — and this holds across seven RL algorithms and ten LLM families Does reinforcement learning update only a small fraction of parameters?. That's the cleanest case for a ratio-like quantity reproducing across architectures. Reasoning behavior also scales predictably with *capability* rather than architecture per se: optimal CoT length follows an inverted-U that shifts shorter as models get more capable, with RL naturally gravitating toward shorter chains as competence rises Why does chain of thought accuracy eventually decline with length?. If you read shorter, more direct reasoning as less heuristic flailing, then capability — not architecture family — is the axis along which this varies most cleanly.
But here's the twist that should make you skeptical of any single "ratio across architectures": several notes show the direction of these effects flips with *domain*, not model. Preference tuning reduces diversity in code but increases it in creative writing, because each domain incentivizes opposite things Does preference tuning always reduce diversity the same way?. Entropy dynamics split the same way — structured domains shrink output entropy while creative ones grow it, predictably enough that training order can be scheduled around it Does training order reshape how models handle different task types?. So the cleaner predictor of how much a model leans on heuristics may be the *task domain* and the *training recipe*, not the architecture.
The deeper reframe the corpus offers: the heuristics aren't created by architecture at all. Base models already contain latent reasoning that five independent mechanisms can elicit, so post-training *selects* reasoning rather than building it Do base models already contain hidden reasoning ability?. And separating the planner from the solver outperforms monolithic models, with decomposition skill transferring across domains while solving skill doesn't Does separating planning from execution improve reasoning accuracy?. If you want a quantity that varies predictably, the corpus points less at "architecture family" and more at how much genuine reasoning your training and task structure manage to *elicit* from capability that's already latent.
Sources 8 notes
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.