How does tool access change what we measure in reasoning tests?

This explores how letting a model use tools changes what a reasoning test is actually scoring. The short version from the corpus: a lot of what we've been calling "reasoning failure" turns out to be execution failure in disguise, and tools make that visible. Several notes converge on the idea that text-only benchmarks systematically underestimate models. The so-called reasoning cliff — the point where accuracy supposedly collapses on harder problems — moves or disappears once a model can offload steps to a tool Does the reasoning cliff depend on how we test models?. The cliff, in other words, was partly a property of the ruler, not the thing being measured.

The sharpest framing comes from work showing that model "collapses" are bandwidth problems, not thinking problems Are reasoning model collapses really failures of reasoning?. A model can know the algorithm for a multi-step procedure and still fail to grind through it token by token in its head; give it a tool to run the procedure and it sails past the supposed limit. So tool access splits a single benchmark number into two questions that text-only tests fuse together: does the model have the method, and can it execute the method at scale? Those are different competencies, and only the first is really "reasoning."

This reframes a debate about what counts as honest measurement. One line of work argues benchmarks should score final answers against ground truth rather than grading the prettiness of the reasoning trace, because trace-based scoring inflates results by rewarding stylistic mimicry Should reasoning benchmarks score final answers or reasoning traces?. Tool access pushes the same direction — when a model can call code, the trace becomes a mix of natural-language planning and tool calls, and what you can cleanly verify is the solution, not the narration. There's a tension to sit with here: solution-only scoring is honest about outcomes, but it also means a tool-assisted correct answer and an unassisted one look identical on the scoreboard even though they measure different things.

There's a structural reason tools change the measurement too. Decoupling the reasoning from the tool's outputs — planning the whole chain before executing, or using placeholders for results you'll fill in later — changes both efficiency and what the benchmark captures Can reasoning and tool execution be truly decoupled?. When reasoning and execution are interleaved, a single wrong tool observation can derail the chain, so your score conflates the model's plan with the tool's reliability. Separate them and you can measure the plan's quality on its own. This connects to broader findings that test-time performance depends more on total compute budget and the quality of your reward/value signal than on the specific framework you wrap around it Does the choice of reasoning framework actually matter for test-time performance?.

The deeper warning underneath all of this: benchmark numbers and genuine capability are already separable even before tools enter the picture. RLVR can light up real reasoning behavior while the headline benchmark gain is mostly memorization of contaminated data Can genuine reasoning activation coexist with contaminated benchmarks?, Does RLVR success on math benchmarks reflect genuine reasoning improvement?. Tool access adds yet another layer to that gap — it can rescue a model that genuinely reasons but can't execute, and it can also paper over a model that can't reason at all but can call the right tool. The honest takeaway is that "reasoning test" stops being a single thing the moment tools are allowed: you have to say which of execution, planning, or recall you meant to measure, because the tool decides which one your number is really about.