Why do vision and language scale so differently?

The Chinchilla scaling laws characterize text pretraining: for a given compute budget, there is an optimal trade-off between model size and training tokens, and language models near this balance get the most out of their compute. The IsoFLOP analysis in Beyond Language Modeling shows that vision does not obey the same trade-off curve. Vision is significantly more data-hungry than language at the same compute scale — the optimal allocation pushes harder toward more tokens and proportionally smaller models.

This is a fundamental scaling asymmetry. In dense multimodal models, you have to pick a scaling regime: optimize for language (Chinchilla-balanced) and underfit vision, or optimize for vision (data-hungry) and waste capacity on language. There is no single dense allocation that serves both modalities well.

Sparse MoE resolves the asymmetry by shifting how each modality's tokens are processed. In the sparse regime, the analysis shows that language scaling itself shifts toward a more data-hungry pattern — aligning with vision's optimum. The mechanism is that sparsity provides the structural flexibility for modalities with fundamentally different scaling behaviors to coexist. Each modality's tokens can route to experts trained on that modality's optimal data ratio.

The finding has implications beyond multimodal architectures. It suggests that scaling laws are not properties of "training" in general but of the data distribution being trained on. Different data has different scaling exponents. The Chinchilla balance is the language exponent; the vision exponent is different; the audio exponent is presumably different again. Future multimodal architectures will need to accommodate this heterogeneity rather than assume a single scaling regime applies.

It also reframes what MoE is for. The standard story is computational efficiency — sparse activation reduces inference cost. The scaling-asymmetry story adds a complementary justification: MoE is the architecture that enables modalities with fundamentally different data appetites to coexist in one model. This is a deeper role than efficiency, and it suggests sparsity should be a default rather than an optimization in multimodal pretraining.