Even when AI gives clinicians the right answer, it often doesn't change care. Is the blame distrust, bad weighing, or poor delivery?

This explores why correct AI advice so often fails to change clinical outcomes, and where the cause lies: in clinicians' heads, in the advice itself, or in how the advice reaches them. The gap is real. On test cases, an LLM has outperformed hundreds of physicians on differential diagnosis, triage and management Can language models reason better than physicians at diagnosis?. Yet when professional radiologists were given AI predictions, their average performance didn't improve Why don't radiologists benefit from AI predictions?. A capable model doesn't automatically make better care.

The radiology study points to a reasoning error rather than plain distrust. Radiologists gave AI predictions too little weight, and they treated their own judgment as if it were independent of the AI's signal. Weighing two opinions well depends on knowing how much they overlap. Get that wrong and you can't capture the gains of working together, even if the AI is usually right. You might expect the label 'AI' to be the problem, but another study complicates that. Radiologists rated advice lower when it was labeled AI than when the same advice was labeled as coming from a human expert, yet their diagnostic accuracy depended on whether the advice was correct, not on its label Does labeling advice as AI change how clinicians use it?. What clinicians say about AI advice and what they actually do with it can come apart.

Here's the twist that changes the question. The same groups that under-use correct AI can be badly misled by incorrect AI. In mammography, wrong AI-labeled categories dropped experienced radiologists from 82% to 45.5% accuracy, and inexperienced readers fell below 20% How much does wrong AI advice harm radiologist accuracy?. So the core problem isn't that clinicians trust AI too little or too much. They can't reliably tell when the AI is right, and the advice gives them no way to find out. That fits evidence that clinicians couldn't tell GPT-4's written advice from expert advice, guessing its source at chance level Can clinicians tell GPT-4 advice apart from expert advice?. Fluent, plausible text looks the same whether it's right or wrong. A broader account of human-AI interaction describes how confusing a model's fluency with its reliability, plus confirmation bias, can compound into misplaced trust Why do people trust AI outputs they shouldn't?.

The most encouraging evidence suggests the fix is mostly about design. In nearly 40,000 live primary-care visits in Nairobi, clinicians with an LLM safety net made 16% fewer diagnostic errors and 13% fewer treatment errors. The authors credit the asynchronous, workflow-aligned interface and active rollout, not the model's raw ability Can AI safety nets reduce errors in live clinical practice?. The radiology study likewise found that contextual information helped where bare predictions did not Why don't radiologists benefit from AI predictions?. Outside medicine, a historical analysis of agent deployments finds that failures come from missing ecosystem conditions such as trust, social acceptability and fit, not from capability gaps Why do capable AI agents still fail in real deployments?. A therapy study found the same thing from another angle: the same language model reduced distress when delivered through a robot or a structured worksheet, but not through a chatbot Why do robots outperform chatbots in therapy despite identical language models?. The takeaway is that the delivery medium is part of the intervention. A correct suggestion that arrives at the wrong moment, without context, in a format clinicians can't check, may help no more than a wrong one.