Can energy-based transformers achieve deep reasoning without supervision?

This explores whether Energy-Based Transformers (EBTs) can reach deliberate, System-2-style reasoning purely from unsupervised learning. The corpus's direct answer is encouraging: EBTs reframe inference as energy minimization — the model assigns an energy score to each input-prediction pair and uses gradient descent at inference time to settle into a low-energy answer, effectively "thinking" by iterating rather than emitting in one shot. The headline claim is that this yields steeper training scaling and meaningful inference-compute gains over a strong Transformer baseline, with better generalization on out-of-distribution data and no domain-specific scaffolding Can energy minimization unlock reasoning without domain-specific training?. So the short answer the library offers is: yes, in principle, deep reasoning can emerge from the right objective rather than from supervised labels.

But the interesting part is how this sits among the corpus's other escape routes from the same trap. EBTs are one of several architectures betting that fixed-depth, feed-forward transformers are the bottleneck. The Hierarchical Reasoning Model makes a different bet — coupling slow abstract planning with fast detailed computation across two timescales to break past the depth ceiling that constrains standard transformers, solving Sudoku and mazes that chain-of-thought fails on, with tiny parameter counts Can recurrent hierarchies achieve reasoning that transformers cannot?. Both are reacting to the same diagnosis: that transformers often only *look* like they reason. One note shows compositional reasoning in transformers collapses into memorized subgraph matching that shatters on novel combinations Do transformers actually learn systematic compositional reasoning?, and another finds genuine multi-hop reasoning only emerges in late training stages and needs explicit compositional exposure to generalize How do transformers learn to reason across multiple steps?. EBT's energy-minimization loop is one proposed way to get real iterative computation instead of pattern recall.

The "without supervision" half of the question opens a second front. EBTs get there through the learning objective itself, but the corpus has a sharply different unsupervised path: self-play. Ctx2Skill's three-role loop manufactures the missing feedback signal internally — a Challenger escalates difficulty as a curriculum, a Judge issues binary verdicts as reward, and skills co-evolve in natural language, all without human labels Can language models learn skills without human supervision?. That's worth pairing with EBTs because it answers a different question — EBT removes supervision from *how the model computes an answer*, while self-play removes it from *where the training signal comes from*. Both dodge human annotation, but at different layers of the stack.

There's also a quieter, cheaper rival that should temper the "new architecture" excitement. Cognitive tools show you can lift a frozen GPT-4.1 from 26.7% to 43.3% on competition math with zero RL training, just by wrapping reasoning operations in modular sandboxed calls that isolate each step Can modular cognitive tools unlock reasoning without training?. The implication cuts against EBTs in an interesting way: some "reasoning" capability is already latent in standard models and merely needs to be *elicited* rather than *trained in*. And prompting theory backs this up — a single finite transformer is provably Turing-complete given the right prompt, even though ordinary training rarely produces models that actually behave that way Can a single transformer become universally programmable through prompts?. So the honest tension in the corpus is: is deep unsupervised reasoning an architecture problem (EBT, HRM), a training-signal problem (self-play), or an elicitation problem (cognitive tools)?

Where the corpus gets quiet: it has strong evidence that training regime beats raw inference compute — non-reasoning models can't simply spend their way to parity, because reasoning has to be instilled, not bought at test time Can non-reasoning models catch up with more compute?. EBTs claim to convert *more* inference compute into *better* answers via energy descent, which is exactly the lever that note says is usually weak. That's the unresolved bet worth watching: EBTs promise the test-time scaling that the rest of the corpus says training, not inference, normally controls. The library doesn't yet have a head-to-head verdict — but it gives you the precise fault line to read along.