How should benchmark design account for task-dependent sparsity tolerance differences?

This explores how benchmark design should account for the fact that different tasks tolerate very different amounts of sparsity — and the corpus has a sharp empirical anchor for why this matters. The clearest result is that sparsity tolerance isn't a single number: single-QA tasks survive at 95% sparsity, while multi-hop reasoning and aggregation tasks fall apart at 50–67% How much sparsity can different reasoning tasks actually tolerate?. The structural reason is that single-QA concentrates the answer in a few tokens, while multi-hop and aggregation need attention spread across many regions at once. A benchmark that reports one average sparsity score would hide exactly this — it would make a method look uniformly good or bad when its real behavior is task-shaped.

This connects to a measurement trap worth naming: how you compare matters as much as what you measure. The Sparse Frontier work shows that if you benchmark sparse vs. dense models at equivalent compute rather than equivalent size, sparsity looks Pareto-improving — bigger sparse models beat smaller dense ones — instead of a quality-for-speed trade Does sparse attention trade off quality for speed?. The lesson for benchmark design is that the axis you hold constant (compute? parameters? context length?) silently decides the conclusion. Pair that with task-dependent tolerance and you get the design principle: report sparsity tolerance per task category, controlled against a fixed compute budget, not a single headline number.

The corpus also complicates what 'sparsity' even is, which benchmarks should respect rather than flatten. Models sparsify their own hidden states when facing unfamiliar or out-of-distribution inputs, and this is an adaptive stabilizing filter, not a failure Do language models sparsify their activations under difficult tasks?. Relatedly, representational density is learned through training-data familiarity — dense for the familiar, sparse for the novel Is representational sparsity learned or intrinsic to neural networks?. So a benchmark measuring imposed sparsity (pruned attention) is testing something different from a benchmark observing emergent sparsity (the model's own response to difficulty). Conflating the two would mislead. There's even a constructive use: sparsity can be read as a difficulty signal to order few-shot examples from hard to easy Can representation sparsity order few-shot demonstrations effectively? — which implies a benchmark could stratify test items by their induced sparsity to map the tolerance curve directly.

The broader pattern across the corpus is that capability is shaped by structure, not by a single scalar dial — so benchmarks that report scalars mislead. Reasoning models beat non-reasoning ones regardless of inference budget because training instills a protocol, not because they have 'more' of something Can non-reasoning models catch up with more compute?; depth beats width at small scale because layered composition does work that raw parameter count can't Does depth matter more than width for tiny language models?; and architectures like Titans that split short-term attention from long-term memory tolerate scale precisely because different task demands are routed to different mechanisms Can neural memory modules scale language models beyond attention limits?. The takeaway you might not have come looking for: a good sparsity benchmark isn't measuring a property of the method, it's mapping an interaction between method, task structure, and what you held constant — and the most informative output is a tolerance curve broken out by reasoning type, not a leaderboard rank.