Does each hierarchy level need its own latent space?

H-JEPA trains a hierarchy of action-conditioned JEPA world models in which "each level predicts in its own learned latent space," at "a coarser temporal stride and a higher degree of abstraction than the level below." Planning runs top-down: the top level sets a coarse plan toward the goal, and each level's predicted trajectory supplies subgoals for the level below, down to level 1, which "returns primitive actions for execution." Across four simulated navigation and manipulation environments, "hierarchical planning improves over a flat JEPA; on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% using less planner compute." With an added inverse-dynamics term the method extends to DROID, a real-robot manipulation video corpus, improving offline planning fidelity at lower planner compute.

The paper attributes the gain to two mechanisms it isolates by ablation. First, a distinct, more abstract representation at each level lets a higher-level planner "score candidate futures in a more abstract latent space" — useful because a goal is often "more abstract than the state (a location to reach rather than a pose to match)," so a flat latent carrying every fast-varying detail gives a poor goal-matching cost. Second, independent of representation, temporal decomposition breaks a long-horizon task into shorter subgoal problems: tracking a nearby subgoal yields a more consistently improving cost than tracking the full-horizon goal (Spearman monotonicity 0.89–1.00 for subgoal tracking versus 0.48–0.75 for full-horizon cost). The paper credits HWM's prior planning gains over LeWM to this second mechanism alone, since HWM shares one latent space across its prediction horizons; H-JEPA's further gain on Visual AntMaze is attributed to the first mechanism — distinct per-level abstraction — which HWM lacks.

The architecture directly inherits its anti-collapse method from Can a single regularizer prevent JEPA representation collapse?: H-JEPA trains every level "using LeJEPA's SIGReg regularizer," the same Gaussian-distribution constraint LeWM used to stabilize a single-level JEPA, now applied per level across a hierarchy. It also sits in contrast with Can looped computation replace parameter count in world models?: LoopWM scales world-model computation by looping one shared block to adaptive depth within a single latent space, while H-JEPA scales by stacking distinct latent spaces, each at its own timescale and abstraction level — two different answers to where a world model should spend extra computation, temporal depth within one representation versus representational depth across several.

The excerpt reports gains only on simulated navigation/manipulation benchmarks and on offline, open-loop planning fidelity on real-robot video; the paper itself flags closed-loop control on a physical robot as untested ("a next step is to test whether these gains translate to closed-loop control on a physical robot"). It also does not establish that abstraction helps uniformly: on Push-T and Cube "we observe no selective abstraction" and "projected costs provide no clear benefit," so the abstraction mechanism's contribution looks environment-dependent — strongest where state and goal differ in abstraction level (AntMaze, FourRoom), negligible otherwise.