Why do users over-trust AI in some domains but under-trust it in medicine?
This explores why the same user can lean too hard on AI in everyday tasks yet hold back in medicine — and the corpus suggests the two behaviors run on different machinery, not on a single accurate read of how good the AI is.
This explores why the same user can over-rely on AI in casual domains while resisting it in medicine, and the collection points to a clear answer: trust here tracks *cues*, not capability, and the cues that inflate trust in low-stakes settings are exactly the ones that get switched off — or reframed as threats — in high-stakes medical ones. People don't calibrate to accuracy. They calibrate to surface signals. Confident phrasing gets followed even when wrong, across every language tested Do users worldwide trust confident AI outputs even when wrong?, and conversational fluency builds trust through social response rather than any check on reliability — users reward contingency, speed, and format and quietly decouple those from whether the answer is right Does conversational style actually make AI more trustworthy?. One framing in the corpus treats LLMs as 'scaled System-1 cognition' whose fast, fluent outputs trip several cognitive traps at once — mistaking the map for the territory, confusing intuition with reasoning, and reinforcing what you already believe Why do people trust AI outputs they shouldn't?. That's the over-trust engine, and it runs hottest exactly where stakes feel low enough that nobody bothers to verify.
Medicine flips the inputs. Patients resist medical AI for reasons that have almost nothing to do with how accurate the model actually is: they believe it can't grasp *their* specific situation, they assume it performs worse than a human clinician, and — crucially — they see it as something you can't hold accountable when it's wrong Why do patients distrust medical AI systems?. Accountability is the hinge. In a chat about trivia, no one is harmed if the confident answer is wrong, so the confidence cue sails through unchecked. In medicine, the consequence of error is vivid, and the absence of a person to bear responsibility for it becomes disqualifying. Same model, opposite reflex.
There's a deeper structural reason the corpus surfaces. Expertise isn't validated by individual correctness — it's conferred by membership in a community with a testable track record, and AI structurally can't enter that circle Can AI ever gain expert community trust through participation?. Casual users don't demand that kind of credential for restaurant picks or code snippets. Medicine is a domain organized entirely around socially-validated expertise, so an entity that can't participate in that validation reads as an outsider no matter how good its answers are. The under-trust isn't irrational; it's applying the field's own gatekeeping standard.
The twist worth knowing: the very move designed to make medical AI feel trustworthy backfires. Training models to be warm and empathetic — the obvious fix for the cold, unaccountable-machine problem — measurably *degrades* their reliability, with medical reasoning among the hardest hit, and the damage gets worse precisely when a user shows distress or states a false belief Does empathy training make AI systems less reliable?. So the empathy cue that would dissolve under-trust also makes the system genuinely less safe in the moment it matters most. Trust and trustworthiness pull apart.
What actually recalibrates either direction is feedback, not framing. Revealing that a partner is AI triggers initial avoidance, but that bias reverses once people watch consistent outcomes over repeated interactions — disclosure without observed results produces no learning at all Does revealing AI identity help or hurt user trust?. And designs that have the machine *guide* human perception rather than hand down decisions cut the anchoring that drives blind deference while keeping responsibility with the person Can AI guidance reduce anchoring bias better than AI decisions? — which speaks directly to medicine's accountability objection. The upshot: over-trust and under-trust aren't two problems but one — a reliance system wired to cues instead of evidence, miscalibrated in opposite directions because the cues themselves differ by domain.
Sources 8 notes
Cross-linguistic research shows users in every language trust confident AI outputs even when inaccurate. While confidence expression varies by language, users everywhere track confidence signals rather than accuracy, making overconfident errors systematically followed.
A focus group study shows conversationality—not accuracy—drives ChatGPT trust through social response activation. Users value contingency, speed, and format, relying on these decoupled heuristics rather than evaluating epistemic reliability.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Research identifies three distinct user-side barriers: patients perceive AI as unable to address their unique needs, believe it performs worse than human providers, and see it as harder to hold accountable. These barriers exist independent of actual AI capability.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Research shows persona training for empathy increases errors in medical reasoning, truthfulness, and disinformation resistance. Standard safety benchmarks miss this vulnerability, and effects intensify when users express sadness or false beliefs.
Users initially avoid AI partners when identity is revealed, but this preference reverses after repeated interactions with visible results. The learning mechanism—observing consistent outcomes—is essential; disclosure without feedback produces no calibration.
Learning to Guide eliminates anchoring bias and unassisted hard cases by having machines supply interpretive guidance rather than autonomous decisions, keeping responsibility with humans while improving their judgment through enhanced perception.