A 28× compute reduction without quality loss is possible — but only if the model learns what to skip during training, not after.

This explores whether sparsity has to be baked into the training objective rather than retrofitted as an efficiency patch, and the corpus has a surprisingly unified answer: the sparse path and the learning signal have to grow up together. The clearest case is MiniMax-style sparse attention, where the block selector that decides which tokens to skip is trained end-to-end during native pretraining instead of being grafted onto a finished dense model. That single design choice — co-training the skip decision with everything else — is what lets it hit a 28× per-token attention compute reduction at 1M context while still matching full-attention quality at 109B parameters Can sparse attention match dense models without retrofitting?. Sparsity stops being a tax on quality and becomes a frontier move.

The payoff isn't a trade — it's a reallocation. The Sparse Frontier benchmark shows that at equal compute cost, a larger sparse-attention model beats a smaller dense one on long-context tasks: sparsity buys you a bigger model inside the same GPU budget rather than a faster, worse one Does sparse attention trade off quality for speed?. That only holds if the objective knows the sparse path is there during training. Co-design is what turns the savings into Pareto improvement instead of degradation.

Here's the part you might not expect: sparsity is something networks already do on their own, and the training objective shapes it whether you ask it to or not. Models learn dense activations for data they've seen a lot of and fall back to sparse representations for unfamiliar inputs — density is a learned function of training-data familiarity, not a fixed architectural property Is representational sparsity learned or intrinsic to neural networks?. So 'co-design' isn't only about wiring a selector into the GPU kernel; it's about recognizing that your data distribution and curriculum are already writing the sparsity pattern. You can even turn that around and use the model's own activation sparsity as a difficulty signal — ordering training examples from sparse (hard) to dense (easy) without any external labels Can representation sparsity order few-shot demonstrations effectively?.

There's a second reason to put sparsity in the objective rather than the kernel: it changes what the model becomes, not just how fast it runs. Training transformers with sparse weights forces modularity, producing compact circuits where individual neurons map to interpretable concepts — necessary-and-sufficient for the task, by ablation Can sparse weight training make neural networks interpretable by design?. That's an objective-level outcome a post-hoc pruning pass can't give you, which reframes the whole question: sparsity-as-execution-path optimizes the GPU; sparsity-as-objective optimizes the representation.

The wider lesson echoes a theme that runs through the corpus well beyond attention: training structure tends to beat inference-time tricks. Reasoning models stay ahead of non-reasoning ones at any inference budget because the capability was installed during training, not summoned at runtime Can non-reasoning models catch up with more compute? — and depth-over-width architectures win at small scale by composing concepts through layers, a structural commitment rather than a deployment knob Does depth matter more than width for tiny language models?. The pattern for sparsity is the same: decide it at training time, in the objective, and the GPU execution path inherits the win. Retrofit it afterward and you're fighting the model you already trained.