SYNTHESIS NOTE
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Structural compositionality is the extent to which neural networks break down compositional tasks into subroutines and implement them in modular subnetworks. The alternative: matching inputs to learned templates without task decomposition.
The evidence supports compositionality. Using model pruning to isolate subnetworks:
- Subnetworks that implement one subroutine can be identified
- Ablating a subnetwork harms its corresponding subroutine while leaving others largely intact
- This holds across multiple architectures (CNNs, transformers), tasks (vision, language), and scales
The pretraining effect: models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Self-supervised pretraining appears to create internal structure that is more amenable to compositional decomposition. This suggests that the representations learned during pretraining have a modular quality that fine-tuning can exploit.
This provides empirical support against the longstanding objection that neural networks are fundamentally non-compositional. The finding: "some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques." Explicit symbolic mechanisms may be unnecessary — gradient-based optimization discovers compositional structure when the task demands it and pretraining provides a good initialization.
The result is not perfect: "most do not exhibit perfect task decomposition." Compositionality is partial and graded, not all-or-nothing. Some architecture-task combinations show stronger structural compositionality than others.
This connects to the weight-sparsity finding: Can sparse weight training make neural networks interpretable by design? shows that enforcing sparsity produces clean decomposition. The structural compositionality paper shows that decomposition also emerges naturally, albeit imperfectly, from standard training. Sparsity amplifies a tendency that already exists.
Inquiring lines that read this note 135
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does model capacity affect learning performance on diverse downstream tasks?- Can granular sub-task training for function calling improve both open and proprietary models?
- Does scaling model size solve compositional generalization problems?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- Can finetuning sparse subnetworks alone match full parameter finetuning results?
- Why does scaling data and model size improve compositional generalization?
- Can dense models match specialized architectures by mixing data better?
- Can world models form from aggregated partial information across training distributions?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Could probing methods miss computationally important features in neural networks?
- Why are polysemantic features concentrated in early neural network layers?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- What makes linear decodability a reliable signal of compositionality?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Can fractured representations explain why models fail at systematic generalization?
- Can we detect and measure circuit formation before generalization emerges?
- How do biological brains organize computation across different cortical timescales?
- Can steering vectors prove that representations are genuinely organized?
- How do cortical columns implement local inference over memory cycles?
- How much do structural inductive biases matter compared to training data volume?
- Do substitute networks converge differently than complement networks?
- Can fractured entangled representations hide undetected by standard analysis methods?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- What role does a model's representational structure play in learning?
- How do knowledge and reasoning circuits interfere in the same neural network?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- How do neural networks decompose complex tasks into modular subnetworks?
- What inductive biases help networks segregate entities from raw inputs?
- What are fractured entangled representations in neural networks?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Can sparse approximations reveal interpretable structure hidden in existing dense models?
- How does joint backpropagation differ from training separate ensemble models?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- What role does query-level exposure play in enabling compositional generalization?
- How do neural networks decompose tasks into modular subnetworks that transfer?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- Can we predict which tasks will decompose into modular subnetworks?
- Why does gradient descent discover compositional structure without explicit pressure?
- What happens to representational structure during model pretraining phases?
- How do models develop dense representations for familiar training data?
- How can neural networks be interpretable by design rather than post-hoc?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- What makes a feature abstract versus concrete in neural network activations?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- How does representational density emerge from training data familiarity?
- Can training order and structure shape what networks retain and learn?
- What prevents representation collapse in latent-prediction world models like JEPA?
- How do semantic features in representations become steerable task-specific directions?
- Why does knowledge storage separate from reasoning circuits in neural networks?
- How does representation-level reranking address residual gaps after decomposition?
- What limits the extrapolation of learned operations like rotation and reflection?
- How do concept vectors in neural networks relate to human-language chess explanations?
- What makes a new representational primitive valuable enough to justify its representational cost?
- Could superposed decoding algorithms maintain multi-task representation during generation?
- What does next-token prediction tell us about compositional linguistic competence?
- Does task superposition explain how models learn from multiple in-context trajectories?
- Why does decomposition ability transfer across domains but solving ability does not?
- Can modular expert decomposition extend beyond time into other causal dimensions?
- Does hierarchical abstraction help equally across different manipulation tasks?
- Do transformers learn generalizable algorithms or instance-based patterns?
- How does nesting optimization levels improve on traditional network depth?
- Can symbolic mechanisms improve transformer compositional abilities?
- Could graph neural networks fundamentally outperform transformers on structured reasoning?
- Why do standard transformers fail to encode recursive structure in their hidden states?
- What makes recursive structure different from other forms of compositional generalization?
- What distinguishes hierarchical dual-recurrence from flat parameter-sharing recurrence?
- How does layer removal affect transformers compared to ResNets?
- How do induction heads learn to overwrite computational representations?
- Can sub-task handlers be swapped between neural and symbolic systems?
- Can latent recurrence overcome the trainability costs of depth?
- Can spline-based activations replace MLPs in transformer architectures?
- Why does recursion on latent state drive generalization better than hierarchy?
- What makes recurrent depth enable compositional generalization across tasks?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Why does recursion on latent states improve generalization more than scale?
- What tasks does recurrent depth solve that feedforward models cannot?
- Can recurrent blocks learn genuinely novel computation beyond repetition?
- Can transformers abstract relational structure without explicit symbolic machinery?
- Does information stored in neural networks necessarily influence generation decisions?
- How do functional features differ from representational abstract features?
- What test distinguishes genuine compositionality from fractured feature presence?
- Are detection and identification of injections truly separable in neural circuits?
- What makes a neural network circuit actually interpretable to humans?
- Does causal intervention alone explain how neural mechanisms implement representations?
- Can geometric structure in representations exist without supporting functional mechanisms?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- How do ablation studies reveal function without representational characterization?
- Can representation analysis methods detect complex features models compute with?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- Can neural grafts reliably reveal hidden capabilities in AI models?
- How do sparse networks trade capability for human-understandable circuits?
- How can interpretability methods account for shifting representational density across task conditions?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Can LLMs reliably generate novel working architectures without structured representations?
- Can unfilled cells in the periodic table represent undiscovered argument schemes?
- Why do higher network layers capture procedural knowledge but lower layers store facts?
- How does neuro-symbolic design differ from pure LLM reasoning?
- Why do singular value experts compose better than low-rank adapter subspaces?
- Why should deep learning theory prioritize average-case over worst-case analysis?
- Do generic kernel-decay assumptions alone explain coarse-to-fine spectral ordering?
- How does requential coding measure true simplicity without parameter count inflation?
- Does training on granular tasks beat training on the full function calling problem?
- Why does compositional reasoning fail to explain cross-domain transfer?
- What makes modernized N-gram embeddings composable with transformer architectures?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- Can single-vector embeddings capture non-commutative relationships like word order?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do neural networks naturally learn modular compo…
Can sparse weight training make neural networks in…
Do base models already contain hidden reasoning ab…
Can neural networks learn compositional skills wit…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
sparsity amplifies the compositional decomposition that standard training already partially produces
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
pretraining-induced modularity is part of the "latent capability" that minimal signals can activate
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
complementary evidence: scaling enables compositionality in behavior; pruning reveals it in structure
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
- The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
- Faith and Fate: Limits of Transformers on Compositionality
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- How do Transformers Learn Implicit Reasoning?
- Hierarchical Reasoning Model
Original note title
neural networks decompose compositional tasks into modular subnetworks without explicit symbolic mechanisms — pretraining encourages this