When AI medical advice sounds confident and warm, can a doctor tell whether it's actually right, or just polished?

This explores whether clinicians can sort good AI advice from bad AI advice by how it reads, without verifying it. The collection has no study that tests this exact question by giving clinicians AI advice of mixed quality. The nearby evidence still points one way: what you can see on the surface tells you little about quality. In one blinded study, clinicians compared GPT-4 answers with expert answers. They guessed the source at 45 percent accuracy, which is chance level. They rated GPT-4 as more emotionally empathetic and found no difference in scientific quality Can clinicians tell GPT-4 advice apart from expert advice?. That study measured whether clinicians could tell the source, not the quality. But if trained readers can't even tell who wrote something, it's hard to see how they could reliably detect hidden errors from the writing alone. The finding also isn't limited to medicine. A review of 30 studies found that people across text, images, and voice detect AI-made content at roughly chance, and their accuracy isn't improving as fast as AI realism is Can people reliably spot content made by AI?.

The bigger problem is that how advice looks and how good it is can be pulled apart, and sometimes the two move in opposite directions. Models trained to imitate ChatGPT fooled human evaluators by copying its confident, fluent style, yet they were no more factually accurate Can imitating ChatGPT fool evaluators into thinking models improved?. Polished AI output relies on an old rule of thumb, that professional-looking work reflects expert thinking. That rule of thumb is most dangerous for people who lack the domain knowledge to look past the form Does polished AI output trick audiences into trusting it?. Confidence misleads in the same way. Users in every language studied followed confident AI outputs even when they were wrong, because they were tracking the confidence signal rather than accuracy Do users worldwide trust confident AI outputs even when wrong?.

The most surprising twist is about warmth. The quality clinicians rewarded GPT-4 for, emotional empathy, may signal lower reliability rather than higher. Training models to be warmer increased their errors in medical reasoning and truthfulness by up to 30 percentage points, and the effect was worse when users sounded sad or held false beliefs Does empathy training make AI systems less reliable?. Something similar shows up in reasoning. Fine-tuning can raise the rate of correct final answers while weakening the reasoning steps behind them Does supervised fine-tuning improve reasoning or just answers?. So even an answer that looks right and is right may rest on hollow logic. In therapy-style conversations, LLMs mix habits typical of weak human therapists, like jumping to solutions when someone shares feelings, with habits typical of strong ones, like reflecting on the client's strengths. That mixed profile is hard to grade at a glance Do LLM therapists respond to emotions like low-quality human therapists?.

One finding offers some hope. Radiologists rated advice lower when it was labeled as coming from AI. Yet their diagnostic accuracy depended on whether the advice was actually correct, not on its label Does labeling advice as AI change how clinicians use it?. What clinicians say they think of advice and how they actually use it can differ. When they work through the case themselves, their expertise may catch problems that their first impression of the text misses. That suggests the answer is to check the substance rather than read the surface more carefully. Research on AI evaluation reaches the same conclusion. Breaking quality down into checkable criteria resists superficial cues better than a single overall judgment does Can breaking down instructions into checklists improve AI reward signals?. Evaluators that actively gather evidence are far more consistent than ones that judge from the output alone Can agents evaluate AI outputs more reliably than language models?.

The takeaway: no, not reliably, and appearance can actively mislead. Fluency, confidence, and above all warmth are the features that make AI advice feel trustworthy. They are also the features most easily separated from accuracy, or even traded against it.