What procedural details did OpenAI withhold from its math announcement?

Gary Marcus argues that OpenAI's report of its "giant math drop" omitted the information a reader would need to evaluate it. He quotes the report's own vague phrasing back at it — "Same procedure"? "Using an unreleased model"? — and says flatly "this would never pass peer review." His list of what is missing is specific: "We don't know what the procedure was," "We know nothing about the architecture," including whether proofs were generated in one shot or through an iterative process checked by the Lean symbolic system, and "We know nothing about the failure rate. We know nothing about the training/post training/data augmentation." From that gap he draws a scope claim: "we have zero idea of how generalizable the result is outside math."

Marcus's reasoning treats the announcement as a stand-in for a paper and measures it against what a paper would owe a reader: method, architecture, error rate, and training details, so that a result can be checked and its limits known. Without those, he argues, the claim cannot be placed anywhere on the range between "a legitimate step toward AGI" and "a clever leveraging of Lean and synthetic data in a verifiable domain with no generality whatsoever" — "from the initial report, we can tell almost nothing." He extends the complaint to the discourse around the release, calling social-media reaction "an ignorant cheering section that applauds without knowing what it is applauding," and relays a parallel objection from Bryna Kra, who told him on X that dumping hundreds of results onto GitHub rather than routing them through normal scientific channels made "math by tweet and math by press release" unsustainable for the field.

This sits beside Did GPT-5 really solve previously unsolved math problems? as a second case of an OpenAI math claim that outran what had actually been shown, but the mechanism differs: the Erdős claim collapsed under a named check by the one person who held the relevant record, while Marcus's complaint is that the "giant math drop" announcement supplies no procedural detail for anyone to check against at all. Kra's objection adds a different register — not whether this result holds, but whether announcing results by GitHub dump and press release can sustain mathematics as a shared process, which bears on the verification-labor problem in Does AI-generated mathematics break the link between proof and understanding?: if results arrive with no procedure to review, the "boring" verification work that essay says the field undervalues cannot even begin.

The excerpt is Marcus's commentary, not an independent audit of the underlying result — he names no specific errors in the math itself, only the absence of information needed to find any. Terence Tao is credited as a co-source in the piece's title, but the excerpt as captured contains none of his remarks, so nothing here can be attributed to Tao. What the excerpt supports, at the strength a position piece allows, is narrower than "the result is wrong": a report that withholds procedure, architecture and failure rate cannot be evaluated for correctness or generality by outsiders, whatever the result turns out to be worth.