How much progress have AI agents actually made on NanoGPT?

METR (Cunningham, Shetty, Cheng, Rush) propose "expenditure horizon" — "the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget" — as a measure of an AI agent's optimization ability, and apply it to the NanoGPT speedrun. From interviews with "prolific NanoGPT contributors" and an LLM-judge categorization of contributions, they estimate the human rate of progress at roughly "16 hours of labor, or $2,400" per percentage-point improvement, formalized as "$2,500 of human expenditure per 1% improvement." Running frontier models (GPT-5, GPT-5.2, GPT-5.5, Opus-4.1, Opus-4.8) as high-expenditure optimization agents starting from NanoGPT record #78 (85.56 seconds), they measure expenditure horizons "between $0 and $3,300." GPT-5 and Opus-4.1 "only chase noise" — raw trajectories show gains that vanish on revalidation — while GPT-5.5 and Opus-4.8 post real but modest improvements. The paper's verdict: although "these models have expenditure horizons in the thousands of dollars, they are small relative to the overall expenditure on human labor," so "autonomous agent optimization has so far had minimal effect on AI R&D progress in NanoGPT."

The method compares two returns-to-expenditure curves — the agent's and a fitted human curve — and reads off their intersection. This corrects two inflations the authors flag directly: "raw trajectories overstate progress" because cumulative-best reporting absorbs statistical noise that revalidation strips out, and the speedrun maintainer judges only "roughly 70%" of surviving ideas as mergeable, with the mergeable share of speedup lower still (around 60% for Opus-4.8, 50% for GPT-5.5). The harness itself biases scores downward: agents had "continuous access to 4 H100 nodes" and spent "70-90% of the cost of most trajectories" on experiments, which the authors attribute to harness inefficiency rather than agent ability — though they note fixing it would shift curves horizontally without changing "the expenditure horizon or the maximum speedup achieved." Human progress, for contrast, is itself gradual: 82 NanoGPT records since May 2024 produced a "33x speedup," a 57% training-time reduction, "none contributing more than 8%."

This sits closest to Do frontier AI agents actually conduct novel research or just optimize?: both find agents mostly reproduce known moves — METR's maintainer calls most explored ideas "low novelty" hyperparameter tuning — with genuine novelty the exception, though METR prices that gap in dollars rather than a qualitative label. It also gives empirical ballast to Are AI feedback loops strong enough to sustain recursive self-improvement?: where that paper's elasticity estimate was a "back-of-the-envelope" calibration without measured values, this note supplies one — near-zero elasticity on at least one narrow R&D benchmark, since two of five tested models show no real gain at all after revalidation. And where Can recursive self-improvement speed up the research process itself? frames automation as speeding research artifacts without speeding the research process, METR's result narrows that claim further: here even the sped-up artifact (the optimized algorithm) barely moves under autonomous agency.

The excerpt gives no per-model breakdown of the $0–$3,300 range beyond the two named extremes, no detail on how the LLM judge scored contributions, and no error bars on the $2,500/1% human rate, which the authors themselves call "highly uncertain." It also explicitly declines to generalize to "hybrid optimization," where agents augment rather than replace human effort — a mode it says "could" still produce dramatic effects even though autonomous optimization alone does not. At this strength the finding supports a narrow claim: on one frontier optimization problem, under one self-described inefficient harness, autonomous agentic contribution is currently worth low thousands of dollars of human labor per run — not evidence about AI R&D acceleration generally.