Does sparsity enforce compositional structure or merely amplify existing modularity?
This explores whether making a network's activations sparse actually *creates* compositional, modular structure — or whether sparsity is just a readout that tracks modularity the network already built for other reasons.
This explores whether sparsity is a *cause* of compositional structure or merely a *symptom* of it — and the corpus leans hard toward symptom. The cleanest evidence that modularity comes from elsewhere is that neural networks decompose compositional tasks into isolated subnetworks on their own, and that pretraining (not any sparsity constraint) is what makes those modules consistent and reliable across architectures Do neural networks naturally learn modular compositional structure?. In a similar vein, plain MLPs reach compositional generalization through data and model scaling alone, with no architectural tricks, as long as training covers enough combinations of the task pieces Can neural networks learn compositional skills without symbolic mechanisms?. Modularity, in other words, seems to be grown by exposure and scale, not enforced by pruning activations down.
Where does sparsity actually come from, then? Two notes reframe it as a *learned response to unfamiliarity* rather than a structural scaffold. Networks develop dense activations for data they know well and default to sparse ones for inputs they don't — a trend that appears during pretraining without any task-specific tuning Is representational sparsity learned or intrinsic to neural networks?. And under genuine out-of-distribution stress, hidden states sparsify in a localized way that acts as a selective filter stabilizing performance, not as a failure or a builder of new structure Do language models sparsify their activations under difficult tasks?. So sparsity tracks how hard and how strange a task feels — it's a gauge, and the corpus even exploits it as exactly that: ordering few-shot demonstrations from sparse-hard to dense-easy as a difficulty signal with no external labels Can representation sparsity order few-shot demonstrations effectively?. You can read modularity off sparsity, the way pruning reveals which tokens carry the symbolic computation Which tokens in reasoning chains actually matter most? — but reading it off is not the same as putting it there.
The sharper twist is that the corpus questions whether the "compositional structure" being amplified is even real composition. Transformers that look compositional in-distribution are often just memorizing computation subgraphs and collapse on novel combinations Do transformers actually learn systematic compositional reasoning?, and grammatical competence degrades predictably as structural complexity rises, suggesting surface heuristics rather than genuine structural rules Does LLM grammatical performance decline with structural complexity?. If the underlying modularity is partly an illusion, then sparsity can't be enforcing compositionality — it can only be amplifying whatever organization the network happened to learn, sound or fractured.
Which points to the most uncomfortable item for the "sparsity enforces structure" view: a model can hold all the linearly decodable features a task needs while its internal organization is fundamentally broken, fragile to perturbation in ways standard metrics never catch Can models be smart without organized internal structure?. Decodability — the same linear-probe success used to certify compositional generalization Can neural networks learn compositional skills without symbolic mechanisms? — can coexist with structural rot. Sparsity sits on the same side of that gap: it amplifies and reveals the structure already present, good or bad, rather than imposing a clean compositional one.
The thing you might not have expected to want to know: the leverage isn't in forcing sparsity, it's in the conditions that grow real modularity in the first place — pretraining, data coverage, even depth that lets layers compose abstractions through the stack Does depth matter more than width for tiny language models?. Sparsity is the instrument you measure with, not the hand that builds.
Sources 10 notes
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Standard MLPs achieve compositional generalization through data and model scaling alone, without architectural modifications, provided the training distribution sufficiently covers combinations of task modules. Linear decodability of constituents from hidden activations reliably predicts success.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Sparsity-Guided Curriculum In-Context Learning uses last-layer activation sparsity to order demonstrations from sparse (harder) to dense (easier), yielding considerable performance improvements. This approach requires no external difficulty labels and works across diverse in-context learning tasks.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.