A new building block for AI thinking has to earn its cost: it must make hard structure cheaper to learn or use, and be checkable.
What makes a new representational primitive valuable enough to justify its representational cost?
This explores when it's worth giving a model a new basic unit to think in (a latent vector, a memory trace, a modular sub-circuit) instead of plain text tokens, and what that unit has to earn back to cover the extra complexity it brings.
This explores when a new building block for representation (something other than plain tokens, like hidden latent vectors, reusable memory paths or modular sub-circuits) pays for the complexity it adds. The corpus has no single paper on this design question. Read together, though, its notes suggest a three-part test: does the new unit capture structure that would otherwise be expensive to learn, does it cut the cost of using that structure, and can you check that it is really organized the way you think it is?
The strongest case for paying is efficiency at learning time. A formal analysis shows that predicting your own latents, instead of predicting raw tokens, recovers layered compositional structure from a number of samples that stays constant as the layers deepen. Token-level learning needs exponentially more samples Why is predicting latents more sample-efficient than tokens?. The reason is useful: latents at the same level are far more correlated with each other than raw tokens are. A good new unit sits where the structure actually lives. The same logic holds at inference time. A 150M-parameter model that reasons by iterating in hidden space, without writing out intermediate tokens, reached a new cost-accuracy point on ARC-AGI at a fraction of a cent per task Can latent reasoning match chain-of-thought cost efficiency without verbalizing?. Memory-Amortized Inference makes the same argument from cognitive science: intelligence is cheap because it reuses stored inference paths instead of recomputing them Can cognition work by reusing memory instead of recomputing?.
There's a twist, though. Some of what a new unit promises may already be happening implicitly. Networks trained on compositional tasks already split them into isolated subnetworks without being told to, and pretraining makes that modularity more consistent Do neural networks naturally learn modular compositional structure?. Inside reasoning chains, models already rank tokens by how much they matter: symbolic computation is kept, while grammar and meta-commentary are dropped first Which tokens in reasoning chains actually matter most?. Chain-of-thought examples with invalid logic work nearly as well as valid ones, which suggests much of the benefit comes from the shape of the scaffold, not its content Does logical validity actually drive chain-of-thought gains?. So the honest baseline isn't "tokens with no structure". It's "tokens already carrying a lot of hidden organization". A new unit has to beat that, and a result from post-training makes the same point: a 3B model matched much larger systems on verifiable tasks through pipeline design alone, without new architecture Can small models match frontier reasoning without massive scale?.
The less obvious cost is that you lose the ability to inspect it. Models can contain every linearly decodable feature a task needs, score perfectly, and still have fractured internal organization that breaks under distribution shift Can models be smart without organized internal structure?. If your new unit is a hidden latent, accuracy alone can't tell you whether it captured the structure or just found a fragile shortcut. Moving reasoning out of readable tokens makes that check harder. A new unit is only worth it if you can also verify how it is organized.
A philosophical thread raises the stakes. One note argues that symbols only mean something because an experiencing agent first cut continuous physics into discrete pieces Can computation arise without a conscious mapmaker?. Another argues that LLMs show fluent language can come purely from compressing the relations between symbols, with no outside reference Can language models learn meaning without engaging the world?. Read together, they reframe the question. Every representational unit is a choice about where to make the cuts. Latents are valuable when the model gets to make cuts that match the data's own hierarchy, instead of inheriting cuts that humans built for writing.
Sources 10 notes
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
Memory-Amortized Inference proposes intelligence arises from structured reuse of prior inference paths over topological memory, inverting RL's reward-forward logic into cause-backward reconstruction. This duality explains energy efficiency and suggests memory trajectories form the substrate of adaptive thought.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Show all 10 sources
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
A 3B model trained with curriculum SFT and multi-domain RL reaches 94.3 AIME26 and 80.2 LiveCodeBench scores matching much larger systems. The result is bounded to verifiable tasks with checkable ground truth, where RL can provide clean reward signals.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Computational systems depend on a conscious mapmaker who alphabetizes continuous physics into discrete symbols. No increase in algorithmic complexity can generate this agent; it must logically precede the computation it makes possible.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Levels of Analysis for Large Language Models
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Hierarchical Reasoning Model
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach