What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
This explores a methodological bet: that simplified, controllable, or even formally solvable setups — toy networks, pruning experiments, physics-style averages, mathematical proofs — can expose phenomena that also govern full-scale, messy deep learning.
This explores a methodological bet — that you can learn the most about real, sprawling neural networks by studying deliberately small or clean versions of them, where you can actually see what's happening. The corpus suggests this isn't a fallback for people without big GPUs; it's becoming a research stance with its own logic. The clearest statement of it is the idea that deep learning theory is reorganizing around 'learning mechanics,' explicitly modeled on physics Can deep learning theory unify around training dynamics?. The move there is the same one statistical mechanics made: stop chasing worst-case guarantees about individual particles, and instead predict the average, typical behavior of training dynamics. The idealization isn't the small model — it's the decision to ask about aggregate statistics rather than any single weight.
A second family of idealized settings is the controlled comparison, where you hold the output fixed and vary the inside. The most striking example is networks that produce identical outputs yet carry radically different internal structure — 'fractured entangled representations' — surfaced by comparing SGD-trained networks against evolved ones Can identical outputs hide broken internal representations?. That comparison is only legible because the setting is simple enough to inspect weights and perturb them; scaled up, the difference would hide behind a benchmark score. The companion finding makes the warning explicit: a model can pass every test and still be internally incoherent Can AI pass every test while understanding nothing?, which is also a recurring theme in how internal mechanism and external behavior come apart What actually happens inside a language model?.
Then there's the idealization-by-intervention approach, where you don't shrink the model but cut into it. Pruning experiments reveal that networks quietly decompose compositional tasks into isolated subnetworks — ablate one and only its function breaks Do neural networks naturally learn modular compositional structure?. Pushed further, you can train for that clean structure on purpose: forcing weight sparsity yields compact circuits where individual neurons map to simple concepts, with ablations confirming they're necessary and sufficient Can sparse weight training make neural networks interpretable by design?. These are 'solvable' in the sense that the phenomenon — modularity — becomes directly observable rather than inferred. Related controlled studies show sparsity itself is learned, with models defaulting to sparse activations for unfamiliar inputs and dense ones for familiar data Is representational sparsity learned or intrinsic to neural networks?.
The purest idealization is the formal one, where the setting is solvable by proof rather than experiment. Three theorems establish that any computable language model must hallucinate on infinitely many inputs, and that internal self-correction can't escape it Can any computable LLM truly avoid hallucinating?. That's a fundamental phenomenon of realistic systems established without running a single realistic system. Controlled empirical work plays a similar role at smaller stakes: clean experiments show RL post-training collapses onto a single pretraining format within the first epoch, with the winner determined by scale rather than quality Does RL training collapse format diversity in pretrained models?, and that depth beats width below a billion parameters — a regime small enough to sweep exhaustively, contradicting the famous scaling laws Does depth matter more than width for tiny language models?.
What ties these together is a single insight worth taking away: the things we most want to know about giant models — whether they understand, whether they hallucinate, whether their representations are sound — are often invisible at scale precisely because scale hides them behind good benchmark numbers. The idealized setting isn't a compromise; it's frequently the only place the phenomenon is visible at all.
Sources 10 notes
Research shows learning mechanics is consolidating as a unified frame for deep learning, modeled on classical and statistical mechanics. It prioritizes average-case predictions, training dynamics, and aggregate statistics over worst-case bounds, mirroring how physics addresses macroscopic systems.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Research shows that LLMs can achieve the same output through different internal mechanisms, and improvements in one dimension like accuracy reliably degrade others like faithfulness and calibration. Internal structure matters even when behavior appears identical.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
Three formal theorems prove that any computable LLM must hallucinate on infinitely many inputs, and internal mechanisms like self-correction cannot eliminate this mathematical constraint. External safeguards are therefore necessary, not optional.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.