Does the reasoning cliff depend on how we test models?

Apple's "Illusion of Thinking" identifies three regimes of reasoning model performance: (1) easy tasks solved reliably, (2) a narrow zone of genuine reasoning improvement, and (3) catastrophic failure beyond a complexity threshold — the reasoning cliff. This finding generated significant attention as evidence that LLM reasoning is fundamentally limited.

The agentic reframe: When the same models are evaluated with tool access (code execution, search, verification), the cliff disappears. Performance continues scaling beyond the text-only collapse point. The "reasoning cliff" is actually a tool-absence cliff — a composite measurement of reasoning ability and execution capability, where execution becomes the bottleneck at higher complexity.

Why this matters: Text-only evaluation creates a specific lens that conflates two separable abilities. A model may correctly identify the reasoning strategy but fail to execute it in pure text (tracking multiple variables, maintaining state, performing sequential calculations). Tool access offloads execution, revealing the reasoning capability that was always present.

The evaluation implication: Benchmarks that prohibit tool use measure something real but not what they claim. They measure text-only reasoning+execution, not reasoning capability. For deployment decisions — where models will typically have tool access — text-only evaluations systematically underestimate capability.

This connects to Why do reasoning LLMs fail at deeper problem solving? — which may be partly an execution failure mode rather than a reasoning failure mode. It also connects to Are reasoning model failures really about reasoning ability?: reasoning models that seem to fail at hard problems may actually fail at hard execution while succeeding at hard reasoning.