Can orchestration strategies boost diagnostic AI without better models?

The Sequential Diagnosis Benchmark (SDBench) turns 304 New England Journal of Medicine clinicopathological conference cases, "published between 2017 and 2025," into stepwise encounters. A diagnostic agent starts from a short abstract and must request findings from a gatekeeper model, which discloses them only when asked, then commits to a diagnosis. The agent is scored on accuracy and on the estimated cost of the tests it orders. The paper's central claim is that its orchestrator, MAI-DxO, improves on the bare model along both axes. The authors report that with OpenAI's o3 it reaches 79.9% accuracy at $2,397 per case, where o3 alone reaches 78.6% at $7,850, and that its maximum-accuracy configuration reaches 85.5% at $7,184. The abstract's "four times higher" headline compares against the physician cohort, which scores 20% at $2,963 per case. Against o3 alone, the cheaper setting adds 1.3 points of accuracy (my arithmetic from the figures above); the rest of the difference is cost. These are the authors' own measurements on their own benchmark and system.

The excerpt credits the gains to "a set of physician-inspired strategies: simulating a virtual panel of physicians with distinct roles, estimating marginal costs between diagnostic rounds, and employing model ensembling methods across model responses." Its stronger claim is that these are general-purpose. MAI-DxO "boosted the accuracy of off-the-shelf models from a variety of providers by an average of 11 percentage points," and the abstract says the gains hold across OpenAI, Gemini, Claude, Grok, DeepSeek and Llama families. The mechanism sits in the scaffold, not the weights. The gatekeeper is o4-mini with the full case file and physician-written disclosure rules. The judge is o3 applying a physician-authored rubric, with four or more on a five-point scale counting as correct.

Against the nearest notes, SDBench is the clinical version of the information problem that Can models identify what information they actually need? isolates formally. QuestBench finds that models which solve a fully specified problem still fail to name the missing variable. SDBench makes information gathering the whole task. Its introduction lists the failures that static vignettes hide: "premature diagnostic closure, indiscriminate test ordering, and anchoring on early hypotheses." These are information-gathering failures of the kind QuestBench formalizes, though the paper does not test QuestBench's categories. The excerpt also qualifies Does medical AI need knowledge or reasoning more?. That note places medical competence in factual knowledge. This excerpt points to a third lever, orchestration around the model, that moves the result without a better model. Finally, the paper's limitations section concedes a version of the distortion that Do automated benchmarks hide what frontier AI systems can really do? describes. SDBench is built from "complex, pedagogically curated NEJM CPC cases," and "the case distribution does not match that of a real-world deployment scenario." It corrects the static-vignette distortion and keeps a curated-case one.

The excerpt does not establish clinical efficacy, and the authors say so: "these results do not yet establish the clinical efficacy of MAI-DxO in real-world decision support." Three further limits bear on the numbers. The cases are rare and hard, with no healthy patients, and the authors "could not measure false positive rates," so the gains on hard cases may not carry to everyday conditions. The cost figures are estimates, and the gatekeeper can "synthesize plausible results for tests not described in the original cases," so part of the cost denominator is model-generated. The physician baseline is a benchmark-condition cohort, not a record of how those physicians practice. The defensible reading is narrow. Under this benchmark's rules, an orchestrator modeled on physician reasoning gives the same model slightly higher accuracy at a fraction of the estimated cost. Whether that helps patients is a separate question the excerpt leaves open.