What can a bounded observer actually learn from data?

Having diagnosed why classical measures fail, the paper introduces epiplexity: a formalization of what a computationally bounded observer can actually learn from data. The key separation is between structural content — the learnable, reusable regularity a bounded learner can extract — and time-bounded entropy, the random unpredictable content that looks like information to an unbounded observer but is useless to a bounded one. Pseudorandom number generators and chaotic dynamical systems are the canonical examples: high apparent entropy, near-zero epiplexity, because no efficient learner can exploit them.

This single distinction resolves the three paradoxes at once. Information can be created by deterministic computation (the transform makes structure efficiently accessible that was latent before); it does depend on data order (ordering changes what a bounded learner can extract along the way); and likelihood modeling can produce programs more complex than the generating process (because the model encodes extractable structure, not just the source's codelength). Crucially, epiplexity is task-free — it measures learnable structure without reference to a downstream objective, which is what makes it a candidate foundation for data selection as opposed to model selection.

The practical payoff is empirical, not just conceptual. The paper gives procedures to estimate epiplexity that capture differences across data sources, track with downstream performance, and flag dataset interventions that improve out-of-distribution generalization. That last result is the strongest: a task-free structural measure that nonetheless correlates with OOD generalization would explain why some data enables broader transfer than others. This fits the vault's data-curation thread — since Can we prune training data without hurting model performance?, examples differ in learnable value, and epiplexity proposes the underlying quantity that difficulty metrics approximate. Counterpoint and caution: epiplexity is observer-relative (bounded to what compute class?) and estimated, not computed exactly, so its claims inherit the slack of the estimator and the choice of observer. Why it matters: it offers the first principled, task-free quantity for deciding which data to select, generate, or transform for learning.

— "From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence", https://arxiv.org/abs/2601.03220