Can AI pass every test while understanding nothing?

Writing angle for Medium/LinkedIn.

Hook: Two neural networks produce identical outputs on every possible input. One understands what it does. The other is a fraud. You can't tell the difference from the outside — and neither can your benchmarks.

Core mechanism: The Fractured Entangled Representation (FER) hypothesis demonstrates that SGD-trained networks can achieve perfect output performance while having fundamentally broken internal representations. The imposter skull looks identical to the real skull on every pixel. But perturb the weights — probe the neighborhood of the solution — and one varies coherently while the other shatters into incoherent fragments.

Three convergent lines:

1. FER — performance ≠ representation quality; identical outputs can mask radically different internal structure
2. Potemkin understanding — correct explanation + failed application = incoherent; models that explain correctly but fail to apply have a structural problem
3. SFT accuracy trap — benchmark scores improve while reasoning quality degrades by 38.9%; every leaderboard optimizes for the wrong thing

Practical stakes: Every model evaluation, every benchmark, every leaderboard measures the surface. The FER hypothesis suggests the internal reality may be structurally different from what performance implies. This matters most at the "borderlands of knowledge" — precisely where AI could make its most valuable contributions.

The question for the reader: How do you evaluate what you can't see? When the test and the reality can completely diverge, what does it mean to "trust" a model?

Source: MechInterp