Can language models reason better than physicians at diagnosis?

The authors report that a large language model, scored by physician experts on challenging clinical cases against a baseline of hundreds of physicians, "displayed superhuman diagnostic and reasoning abilities" in every experiment they ran. The support comes in two layers. The first is five vignette experiments covering differential diagnosis generation, display of diagnostic reasoning, triage differential diagnosis, probabilistic reasoning, and management reasoning, each adjudicated by physician experts "with validated psychometrics." The second is a real-world study of randomly selected patients in the emergency room of a major tertiary academic medical center in Boston, comparing human expert and AI second opinions at three touchpoints: ER triage, initial physician evaluation, and admission to the hospital or intensive care unit.

The framing is historical. The paper anchors its benchmark on Ledley and Lusted's 1959 proposal that complex clinical diagnostic reasoning cases serve as the gold standard for expert medical computing, and on the New England Journal of Medicine clinicopathological case conference series as an "aspirational goal post" that every differential diagnosis generator since has been tested against. The argument is that this benchmark has held for over half a century, so a result on it reads as reaching a long-standing target rather than passing a test built for the model. The authors also credit "continued improvement from prior generations of AI clinical decision support," which makes the claim a trend across generations as well as a statement about one system. The comparison is direct: the same kinds of cases, judged by physicians, set against the human baseline at each touchpoint.

Against the nearest notes, this excerpt sits on a different side of two existing claims. The note on domain competency requirements reads the KI/InfoGain results as showing that medical accuracy tracks factual knowledge more than reasoning quality. This paper reports strong medical reasoning but does not separate knowledge from reasoning, so it cannot say which capacity drives the result; it extends the medical case without settling the split. The note on LLM overconfidence in specialized domains reports low accuracy and overconfident predictions in clinical and biomedical inference. This paper reports high physician-rated performance, but nothing in the excerpt measures calibration, so the two findings cannot yet be weighed against each other on the same axis. The cancer-annotation note is a contrast in scope rather than a direct refutation: it found LLM abstraction stalling in twelve of fourteen tasks, a different job from diagnosis.

What the excerpt does not establish is nearly everything a reader would need to judge the result. It is the abstract and the opening of the introduction. It gives no effect sizes, no model name, no per-experiment results, no adjudicator agreement figures, no outcome data from the emergency room study, and no account of failures. "Superhuman" is the authors' word for their own evaluation, and the authors themselves say the findings motivate "the urgent need for prospective trials." The implication is to read this as a strong, physician-judged benchmark result from one medical center, reported by its authors, rather than as demonstrated clinical performance. The excerpt supports a claim about performance on the benchmark. It does not support a claim about patient outcomes.