Does showing an AI's confidence level help doctors catch its mistakes, or does it only work if that number can be trusted?

This explores whether showing radiologists how confident the AI is helps them catch its mistakes rather than follow them. The short answer is that the corpus has no study that directly tests confidence displays in radiology. It does contain enough nearby evidence to show why the answer depends less on showing confidence than on whether that confidence can be trusted.

Start with how large the problem is. In a mammography study of 27 radiologists, wrong AI suggestions about the BI-RADS category (the standard scale radiologists use to rate how suspicious a breast image is) dropped experienced readers from 82% to 45.5% accuracy. Inexperienced readers fell from about 80% to below 20% How much does wrong AI advice harm radiologist accuracy?. A separate experiment found the opposite pattern: on average, radiologists *underweighted* correct AI predictions. They treated their own judgment as independent of the AI's signal and so missed the gains from working together Why don't radiologists benefit from AI predictions?. So radiologists aren't simply too trusting. They trust the AI in the wrong places, deferring when it's wrong and discounting it when it's right. A good confidence score would in principle fix exactly this, by telling them which cases to lean on.

The catch is that people respond to how confident the AI sounds, not to how accurate it is. Research across many languages finds that users everywhere follow confident AI outputs even when they're wrong Do users worldwide trust confident AI outputs even when wrong?. And confident errors aren't spread evenly. In medicine, law and finance they cluster in rare, unusual cases where a surface pattern conflicts with an unstated constraint, which are the cases where harm happens. Strong average accuracy hides them Why do confident wrong answers hide in standard accuracy metrics?. A confidence score that's well calibrated on average can still be most overconfident on the rare scan that matters. In that case, showing the number would make over-reliance worse, not better.

Two threads from outside radiology suggest better designs. The first is about making confidence trustworthy. Model confidence becomes much more reliable when it's grounded in the model's record on similar past cases rather than read off the current answer Can past performance predict when a model will be right?. Standard alignment training (RLHF) tends to damage calibration, though some methods can restore it Can model confidence work as a reward signal for reasoning?. The second, more radical idea is to stop handing over a verdict with a confidence number attached. "Learning to Guide" has the AI point out which parts of the input deserve attention and leaves the decision with the human, which removes the anchoring effect that a stated answer creates Can AI guidance reduce anchoring bias better than AI decisions?.

The takeaway you might not have expected: the mammography result suggests the most dangerous moment is when the AI hands over a confident label, and that holds whether or not a number is attached. Confidence scores help only if they're calibrated on the hard, rare cases, and those are the cases where AI confidence is least reliable. The more promising approach in the corpus is to change what the AI shows: evidence and guidance instead of an answer to agree with.