When AI 'solves' an open math problem, did it solve it or just dig up an existing paper?

This explores why AI-produced Erdős 'solutions' sometimes don't settle what was originally claimed. In the cases in this collection, the gap usually isn't a broken proof. It's that the headline claim and the work actually done are about different things. The clearest example is OpenAI's announcement that GPT-5 had solved open Erdős problems. It shrank to a literature search once the problems' maintainer, Thomas Bloom, pointed out that 'open' on his website meant 'I don't know of a solution,' not 'unsolved in mathematics' Did GPT-5 really solve previously unsolved math problems?. GPT-5 really did something useful by finding existing papers nobody had connected to those problems. But that's retrieval, not discovery, and the claim failed because of a mismatch in what one word meant.

A second gap opens even when the mathematics holds up. For Erdős Problem 728, an AI system produced a Lean proof. Lean is a proof assistant that checks every logical step mechanically, so the result is certified correct Did an AI system truly solve Erdős Problem 728 autonomously?. Yet the claims around it, that the AI worked *autonomously* and that the result is now *understood*, sit outside what the proof checker can certify. Kakaes's reporting describes this split across the whole Erdős-as-AI-benchmark trend. Formal certification and human comprehension are separate outcomes, and comprehension trails behind, so proofs can be verified yet go largely unread Why did Erdős problems become a popular AI testing ground?. Erdős problems became popular AI targets because they are numerous, accessible, and range widely in difficulty, which also makes it tempting to announce results faster than mathematicians can digest them.

Mathematicians have started building rules around this gap. The Leiden Declaration holds that a proof does two jobs: it establishes certainty and it conveys understanding. Formal verification alone secures only the first, so the declaration assigns credit and responsibility for correctness to human authors and requires them to disclose AI use Can AI-generated proofs ever replace human mathematical understanding?. Terence Tao takes a more permissive view. He argues an opaque ML tool can still contribute real mathematics as long as an external checker, such as a proof assistant or a rigorous numerical argument, confirms what it produces Can opaque machine learning models help prove new mathematics?. Put side by side, the two positions show what a 'resolution' needs: a correct result, a correctly described process, and someone who understands why it works. A verifier supplies only the first.

Research on AI reasoning shows the same pattern in a different setting. Agent-reliability work finds that checking only final outputs misses most failures, which turn out to be process violations rather than wrong answers. Adding checks on intermediate steps raised task success from 32% to 87% Where do reasoning agents actually fail during long traces?. An Erdős claim judged only by 'does a valid proof exist?' is graded on its final answer. It isn't asked how the result was reached or whether the answer was already known. For contrast, OpenAI's counterexample to Erdős's unit distance conjecture is a case where the novelty claim can be checked: its new step, letting the number-field degree grow without bound, is identifiable against a body of classical tools What made OpenAI's unit distance counterexample succeed?.

One limit: the collection doesn't document cases where the formal Lean statement itself drifted from Erdős's original wording. That's a known worry in formal mathematics, but these notes don't cover it. What they do suggest is that the most common way an Erdős 'solution' misses its claim is not bad mathematics. It's bad bookkeeping about what was open, who did the work, and who understands it.