Why do neural networks fail at compositional generalization?

Greff et al. (2012.05208) argue that the binding problem — well-studied in neuroscience and cognitive psychology — is the underlying cause of neural networks' failure to achieve human-level generalization. The binding problem has three aspects that each create distinct failure modes.

Segregation: forming meaningful entities from unstructured sensory inputs. Neural networks struggle to decompose inputs into discrete objects without architectural inductive biases (slot attention, object-centric representations). Without segregation, the network works with undifferentiated feature maps rather than structured entities.

Representation: maintaining separation of information at a representational level. Even when entities can be identified, distributed representations entangle them. A network may know that "red triangle" and "blue circle" are present, but fail to maintain the binding of red-to-triangle and blue-to-circle. This is the classic variable binding problem.

Composition: using entities to construct new inferences, predictions, and behaviors. Even with segregation and representation, composing entities into novel combinations (never seen during training) requires systematic reuse of learned structure. This is where distributional shift fragility appears — agents trained with RL are fragile under distributional shift and require substantially more training data than humans.

The deeper tension: connectionist representations are directly grounded in input data (unlike symbols, which require human interpretation for grounding — the symbol grounding problem). But this grounding advantage comes at the cost of compositional structure. Since Do large language models reason symbolically or semantically?, the binding problem may explain WHY semantic decoupling collapses reasoning: without compositional binding, removing semantic content removes the only glue holding the reasoning together.

Scaling can partially overcome the binding problem. The "Scaling can lead to compositional generalization" paper demonstrates that standard MLPs can compositionally generalize when data and model size are scaled sufficiently. The key theoretical result: MLPs can approximate compositional task families using only a linear number of neurons with respect to the number of task modules — compositionality does not inherently require exponential capacity. Empirically, when models successfully compositionally generalize, task constituents can be linearly decoded from hidden activations; this metric correlates with failures of image generation models to compose known concepts. This provides a partial counterpoint to the binding problem: while the fundamental challenge remains, scaling may create conditions where compositional representations emerge despite the lack of explicit binding mechanisms. The "Break It Down" paper provides structural evidence: models often implement solutions to subroutines via modular subnetworks, and pretraining encourages this structural compositionality. See Can neural networks learn compositional skills without symbolic mechanisms? and Do neural networks naturally break tasks into modular parts?.

Source: LLM Architecture; enriched from MechInterp