Does autoregressive generation uniquely enable LLM scaling?

A common assumption in LLM research is that the autoregressive paradigm — predicting the next token conditioned on prior tokens — is the unique path to the intelligence exhibited by frontier models. The Large Language Diffusion Model (LLaDA) work argues this assumption confuses correlation with causation. Scalability, it claims, is primarily a consequence of the interplay between Transformers, model and data size, and Fisher consistency induced by the generative principles, rather than a unique result of autoregressive modeling.

The empirical evidence comes from a forward-versus-reversal test on 496 famous Chinese poem sentence pairs: given a sentence, models must generate the subsequent line (forward, easy for AR) or the preceding line (reversal, structurally awkward for AR because it inverts the conditional direction the model was trained on). LLaDA, a non-autoregressive diffusion language model, produces coherent extended text in both directions and supports multi-turn dialogue with conversation history retention across multiple languages.

The structural implication is that the generative principle — Fisher consistency, the property that the maximum-likelihood estimator converges to the true distribution as data grows — is what drives scalability. Both AR factorization and diffusion-based denoising can satisfy Fisher consistency, so both can scale, but they expose different parts of the joint distribution to the model. AR factorization fixes a generation order and conditions only on the past; diffusion exposes bidirectional context and any-order generation.

This is not a small technical point. Decades of LLM design has been organized around the AR factorization, and many capabilities (chain-of-thought, RL with policy gradients, KV caching) are tightly coupled to it. If AR is a contingent rather than necessary property, the design space of competitive LLMs is wider than current practice suggests — and capabilities like infilling, bidirectional control, and reverse generation, which AR struggles with, become natural rather than special-cased. This contingency is philosophically loaded: Does AI text generation unfold through temporal reflection? and Does LLM generation explore competing claims while producing text? both built their critiques on AR's token-by-token sequencing — LLaDA shows the sequencing was contingent rather than necessary.

Source: Diffusion LLM