SYNTHESIS NOTE
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Structural compositionality is the extent to which neural networks break down compositional tasks into subroutines and implement them in modular subnetworks. The alternative: matching inputs to learned templates without task decomposition.
The evidence supports compositionality. Using model pruning to isolate subnetworks:
- Subnetworks that implement one subroutine can be identified
- Ablating a subnetwork harms its corresponding subroutine while leaving others largely intact
- This holds across multiple architectures (CNNs, transformers), tasks (vision, language), and scales
The pretraining effect: models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Self-supervised pretraining appears to create internal structure that is more amenable to compositional decomposition. This suggests that the representations learned during pretraining have a modular quality that fine-tuning can exploit.
This provides empirical support against the longstanding objection that neural networks are fundamentally non-compositional. The finding: "some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques." Explicit symbolic mechanisms may be unnecessary — gradient-based optimization discovers compositional structure when the task demands it and pretraining provides a good initialization.
The result is not perfect: "most do not exhibit perfect task decomposition." Compositionality is partial and graded, not all-or-nothing. Some architecture-task combinations show stronger structural compositionality than others.
This connects to the weight-sparsity finding: Can sparse weight training make neural networks interpretable by design? shows that enforcing sparsity produces clean decomposition. The structural compositionality paper shows that decomposition also emerges naturally, albeit imperfectly, from standard training. Sparsity amplifies a tendency that already exists.
Inquiring lines that use this note as a source 119
This note is a source for these synthesized inquiries. Follow a line forward into its question, or open it to trace back to all of its sources.
- Can granular sub-task training for function calling improve both open and proprietary models?
- Why does frame-activation matter more than word-by-word composition?
- Can world models form from aggregated partial information across training distributions?
- Could superposed decoding algorithms maintain multi-task representation during generation?
- Does task superposition explain how models learn from multiple in-context trajectories?
- Do transformers learn generalizable algorithms or instance-based patterns?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Does information stored in neural networks necessarily influence generation decisions?
- How does nesting optimization levels improve on traditional network depth?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- How do functional features differ from representational abstract features?
- Could probing methods miss computationally important features in neural networks?
- Why are polysemantic features concentrated in early neural network layers?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- Why do text-to-image models fail at composing multiple concepts together?
- What makes linear decodability a reliable signal of compositionality?
- Does scaling model size solve compositional generalization problems?
- Can symbolic mechanisms improve transformer compositional abilities?
- How do multimodal AI architectures compare to human brain export pathways?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Can fractured representations explain why models fail at systematic generalization?
- Can we detect and measure circuit formation before generalization emerges?
- How do biological brains organize computation across different cortical timescales?
- Can steering vectors prove that representations are genuinely organized?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- What test distinguishes genuine compositionality from fractured feature presence?
- Are detection and identification of injections truly separable in neural circuits?
- Could graph neural networks fundamentally outperform transformers on structured reasoning?
- Can neural networks learn that A implies B in reverse?
- How do internal representations compare to human cognitive structures?
- Can identical model performance mask fundamentally broken internal representations?
- How do sparse networks trade capability for human-understandable circuits?
- What makes a neural network circuit actually interpretable to humans?
- How do cortical columns implement local inference over memory cycles?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Can LLMs reliably generate novel working architectures without structured representations?
- How much do structural inductive biases matter compared to training data volume?
- Why do standard transformers fail to encode recursive structure in their hidden states?
- What makes recursive structure different from other forms of compositional generalization?
- What separates knowledge from reasoning in neural network layers?
- Do substitute networks converge differently than complement networks?
- What distinguishes hierarchical dual-recurrence from flat parameter-sharing recurrence?
- Can fractured entangled representations hide undetected by standard analysis methods?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- How does layer removal affect transformers compared to ResNets?
- Can finetuning sparse subnetworks alone match full parameter finetuning results?
- Why do singular value experts compose better than low-rank adapter subspaces?
- Can expert vectors learned offline transfer across multiple model architectures?
- What role does a model's representational structure play in learning?
- How does representational convergence differ from policy entropy collapse in iterative training?
- How do knowledge and reasoning circuits interfere in the same neural network?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- How do neural networks decompose complex tasks into modular subnetworks?
- What inductive biases help networks segregate entities from raw inputs?
- What are fractured entangled representations in neural networks?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- Can granular function calling tasks learn composition from graph-sampled data?
- Does training on granular tasks beat training on the full function calling problem?
- How do induction heads learn to overwrite computational representations?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Can sparse approximations reveal interpretable structure hidden in existing dense models?
- How does joint backpropagation differ from training separate ensemble models?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- How can interpretability methods account for shifting representational density across task conditions?
- Can sub-task handlers be swapped between neural and symbolic systems?
- How do attention patterns and circuits function as algorithmic representations?
- Does causal intervention alone explain how neural mechanisms implement representations?
- Can unfilled cells in the periodic table represent undiscovered argument schemes?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Can learned verifiers over token similarity replace dense compositional training?
- Can we predict out-of-distribution generalization without access to downstream tasks?
- Can geometric structure in representations exist without supporting functional mechanisms?
- What role does query-level exposure play in enabling compositional generalization?
- How does the hippocampus bind disparate elements without storing everything itself?
- Why does scaling data and model size improve compositional generalization?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- Why should deep learning theory prioritize average-case over worst-case analysis?
- How do neural networks decompose tasks into modular subnetworks that transfer?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- How do ablation studies reveal function without representational characterization?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Can we predict which tasks will decompose into modular subnetworks?
- Why does gradient descent discover compositional structure without explicit pressure?
- Why do higher network layers capture procedural knowledge but lower layers store facts?
- Why do long-context language models struggle with compositional reasoning tasks?
- What happens to representational structure during model pretraining phases?
- Why does decomposition ability transfer across domains but solving ability does not?
- How do models develop dense representations for familiar training data?
- What does a human-parseable framework for deep learning look like?
- How does neuro-symbolic design differ from pure LLM reasoning?
- Do generic kernel-decay assumptions alone explain coarse-to-fine spectral ordering?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- Can latent recurrence overcome the trainability costs of depth?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- Can representation analysis methods detect complex features models compute with?
- How can neural networks be interpretable by design rather than post-hoc?
- Can spline-based activations replace MLPs in transformer architectures?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- Why does recursion on latent state drive generalization better than hierarchy?
- Can a single recursive network replace hierarchical dual-network architectures?
- What makes recurrent depth enable compositional generalization across tasks?
- Can single-vector embeddings capture non-commutative relationships like word order?
- What makes a feature abstract versus concrete in neural network activations?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- What does next-token prediction tell us about compositional linguistic competence?
- How does representational density emerge from training data familiarity?
- Can training order and structure shape what networks retain and learn?
- What prevents representation collapse in latent-prediction world models like JEPA?
- How do semantic features in representations become steerable task-specific directions?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Can modular expert decomposition extend beyond time into other causal dimensions?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do neural networks naturally learn modular compo…
Can sparse weight training make neural networks in…
Do base models already contain hidden reasoning ab…
Can neural networks learn compositional skills wit…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
sparsity amplifies the compositional decomposition that standard training already partially produces
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
pretraining-induced modularity is part of the "latent capability" that minimal signals can activate
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
complementary evidence: scaling enables compositionality in behavior; pruning reveals it in structure
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
- Faith and Fate: Limits of Transformers on Compositionality
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- How do Transformers Learn Implicit Reasoning?
- Hierarchical Reasoning Model
- Eliciting Reasoning in Language Models with Cognitive Tools
Original note title
neural networks decompose compositional tasks into modular subnetworks without explicit symbolic mechanisms — pretraining encourages this