Does scaling model size solve compositional generalization problems?
This explores whether simply making models bigger fixes their trouble with compositional generalization — combining known pieces in new ways — and the corpus answer is a conditional 'it depends on what's being scaled, and what kind of novelty you throw at it.'
This question asks whether throwing more parameters at a model dissolves its failure to recombine known pieces into novel wholes. The corpus splits sharply, and the split is the interesting part: scaling helps when the training distribution already covers the combinations you'll test, and stalls when it doesn't.
The optimistic case is real but narrow. One line of work shows plain neural networks achieve compositional generalization through data and model scaling alone, with no symbolic machinery — provided the training data sufficiently covers the combinations of task modules Can neural networks learn compositional skills without symbolic mechanisms?. There's even a mechanistic reason this can work: networks naturally carve compositional tasks into isolated modular subnetworks, and pretraining makes that modular structure more consistent Do neural networks naturally learn modular compositional structure?. The classic 'binding problem' analysis agrees that scale can *partially* overcome systematic-generalization failure by letting compositional representations emerge Why do neural networks fail at compositional generalization?. So scaling isn't useless here — it's a way of buying coverage.
But 'buying coverage' is exactly the trap. A second cluster of papers shows that what looks like composition is often memorization of training-time computation subgraphs: transformers succeed in-distribution by matching linearized subgraphs and then fail drastically on genuinely novel compositions, with errors compounding across steps Do transformers actually learn systematic compositional reasoning?. The sharpest reframing comes from work showing reasoning breaks at *instance novelty*, not task complexity — models fit instance-based patterns rather than general algorithms, so a long reasoning chain succeeds if something similar was in training and collapses if not Do language models fail at reasoning due to complexity or novelty?. If the failure is novelty rather than size, more parameters don't address the actual bottleneck.
The hardest evidence against the scaling cure is where scale visibly flatlines. On genuine constrained-optimization tasks, LLMs plateau at roughly 55–60% constraint satisfaction regardless of parameter count, architecture, or training regime — a ceiling, not a gap a bigger model would close Do larger language models solve constrained optimization better?. Relatedly, models can't actually execute iterative numerical procedures; they recognize a problem as template-similar and emit plausible wrong answers, and this persists across scale Do large language models actually perform iterative optimization?. And even with the right information in front of them, strong parametric priors override in-context evidence — a failure that prompting alone can't fix Why do language models ignore information in their context?.
The lateral surprise here: several papers suggest the productive move isn't *bigger* but *differently shaped*. Depth beats width for small models because composing abstract concepts through layers matters more than spreading parameters across width Does depth matter more than width for tiny language models?. Latent-thought models open scaling dimensions that have nothing to do with parameter count Can latent thought vectors scale language models beyond parameters?, and small models trained on explicit negative examples can match large ones on structured tasks Can small models match large models on function calling?. The collective verdict: scaling moves the boundary of what's been *seen*, but compositional generalization is precisely the demand to go *beyond* what's been seen — so scale postpones the wall rather than removing it.
Sources 11 notes
Standard MLPs achieve compositional generalization through data and model scaling alone, without architectural modifications, provided the training distribution sufficiently covers combinations of task modules. Linear decodability of constituents from hidden activations reliably predicts success.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Greff et al. argue that neural networks cannot dynamically bind distributed information into compositional structures due to three failures: segregating entities from inputs, maintaining representational separation, and reusing learned structure in novel combinations. Scaling can partially overcome this by enabling compositional representations to emerge.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.
Latent-Thought Language Models achieve superior sample and parameter efficiency by coupling fast local variational learning with slow global decoder learning. This dual-rate scheme scales few-shot reasoning across both model and latent size, creating independent scaling dimensions beyond traditional parameter scaling.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.