Why do LLMs fail when users interact with them?

In a randomized controlled trial with 1,298 UK participants, LLMs that score well on their own did not help the public make better decisions about everyday medical scenarios. Tested alone, GPT-4o, Llama 3 and Command R+ correctly identified conditions in 94.9% of cases and dispositions in 56.3% on average. Participants using the same models identified relevant conditions in less than 34.5% of cases and dispositions in less than 44.2%, "both no better than the control group" that used whatever they would normally use at home. The paper treats this as a failure of user interaction, not of medical knowledge: standard benchmarks and simulated patient interactions "do not predict the failures we find with human participants."

The discussion locates the breakdown in transmission. The models typically offered two or three options, which "allows users to have the final decision, but they perform poorly at making this choice." Users decided what to tell the model, and the models sometimes suggested the correct answer without conveying it effectively. The combination was "no better than the control group in assessing clinical acuity, and worse at identifying relevant conditions." The authors also note that the LLM-alone figures are a minimum, because chain-of-thought reasoning over the identified conditions was not applied, and that stronger models alone "would only emphasize the gap when operating with real users."

Against the nearest notes, this trial moves a known gap from the model into the channel between model and person. Can language models truly understand therapeutic ruptures? already shows that label agreement can conceal reliance on explicit cues; here the model-alone result is strong and the failure appears only once a user is in the loop. Can language models match therapist empathy in real conversations? confines an advantage to isolated responses, and this trial finds that similar single-turn competence does not carry through a member of the public's decision. Can LLMs actually conduct Socratic questioning in therapy? places the gap in what the model can execute; this paper places part of it in the exchange itself. The patient-side note is a contrast: Why do patients distrust medical AI systems? treats adoption barriers as attitudes, while the failures here are informational and show up in task accuracy. The excerpt reports no attitudinal measures at all.

What the excerpt does not establish is how far these results travel. It covers ten everyday scenarios drafted by three doctors, one UK population stratified to national demographics, and three models used without fine-tuning or chain-of-thought prompting. The authors concede that real-deployment accuracy "could change depending on the relative frequencies of the scenarios." The account of why users fail is the authors' reading of the interactions, not a tested mechanism. The Limitations section raises an incentive question: providers may want users to see doctors rather than trust LLMs, and the authors found "no significant evidence" that LLM users rated their scenarios as more acute. The implication, at the strength one trial allows, is that benchmark scores are weak evidence about outcomes for public users, and that human user testing before public deployment is the sensible default. The trial does not show that every interactive design would fail.