Can architectural changes reduce representational inequality in unified generators?
This reads the question as: when one unified model is asked to do everything, some capabilities come out strong and others stay weak — can changing the architecture (not just adding scale or data) close that gap, or is the unevenness baked into the design? (Note: the corpus here speaks to uneven *computational* capability across tasks, not fairness-style demographic representation — if you meant the latter, this collection doesn't cover it directly.)
This reads the question as asking whether redesigning a model's architecture — rather than just making it bigger — can even out the lumpy distribution of what a single unified model is good and bad at. The corpus splits cleanly into two camps, and the tension between them is the interesting part.
On the optimistic side, architecture genuinely is a lever. MobileLLM finds that for sub-billion-parameter models, going deep-and-thin beats spreading the same parameters across width, because stacking layers lets the model compose abstract concepts instead of just memorizing more in parallel Does depth matter more than width for tiny language models?. More dramatically, the Hierarchical Reasoning Model couples a slow planning loop with a fast computation loop and, with only 27M parameters, solves Sudoku and mazes that chain-of-thought transformers fail completely — escaping a complexity ceiling (AC0/TC0) that fixed-depth transformers literally cannot cross no matter how much you scale them Can recurrent hierarchies achieve reasoning that transformers cannot?. Here, the architectural change doesn't just help — it unlocks a class of capability the standard design forbids.
But the pessimistic camp is just as sharp, and it's the part most likely to surprise you. On genuine constraint-satisfaction and optimization tasks, LLMs flatten out at roughly 55–60% success *regardless of architecture, parameter count, or training regime* — a true ceiling, not a scaling gap Do larger language models solve constrained optimization better?. Reasoning-tuned variants with extended chain-of-thought show no consistent edge on numerical problems like optimal power flow; the extra thinking produces more text, not more actual iterative computation Do reasoning models actually beat standard models on optimization?. And the diagnosis goes deeper than 'try harder': autoregressive generation is missing a primitive — the ability to retract a token it already emitted — that constraint solving fundamentally depends on. That's why bolting on a symbolic solver works where bigger models don't: it supplies what the architecture structurally lacks Why does autoregressive generation fail at constraint satisfaction?.
So the honest synthesis is that 'architectural change' resolves into two very different moves. One is *reshaping the same primitives* (depth over width, recurrent hierarchy over flat stacks), which can dramatically lift weak capabilities. The other is the realization that some inequalities live in the generation paradigm itself — token-by-token autoregression — and no amount of reshaping within that paradigm helps; you have to add an external mechanism. This echoes a recurring finding in the collection: models can't self-improve past a generation-verification gap without something external to check them What stops large language models from improving themselves?, and long-context limits turn out to be about the *compute* needed to consolidate information into internal state rather than raw capacity Is long-context bottleneck really about memory or compute?.
The thing worth taking away: 'unified generator' contains a hidden bet that one architecture can be uniformly good across tasks, and the corpus suggests that bet partly fails by design. The interesting frontier isn't a single cleverer architecture — it's hybrids, where the unified model keeps the tasks it's structurally suited for and hands off the ones it can't represent (relational joins, hard constraints) to a different mechanism entirely Can long-context LLMs replace retrieval-augmented generation systems?.
Sources 8 notes
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
The Hierarchical Reasoning Model couples slow abstract planning with fast detailed computation across two timescales, achieving near-perfect performance on Sudoku and mazes where chain-of-thought methods fail completely. With only 27M parameters and 1,000 samples, HRM escapes the AC0/TC0 complexity ceiling that constrains fixed-depth transformers.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
Reasoning variants with extended CoT show no consistent advantage over standard models on constraint-bound numerical tasks like optimal power flow. Extended thinking produces more text, not more iterative computation, suggesting the bottleneck is numeric procedure rather than reasoning steps.
The performance ceiling on constraint satisfaction problems is not a model-quality issue but an architectural limitation: autoregressive transformers cannot retract emitted tokens, while CSP solvers fundamentally depend on discarding invalid partial assignments. Symbolic solver integration works because it supplies what the architecture lacks.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.