Why do language models fail confidently in specialized domains?
LLMs perform poorly on clinical and biomedical inference tasks while remaining overconfident in their wrong answers. Do standard benchmarks hide this fragility, and can prompting techniques fix it?
"Rethinking STS and NLI in Large Language Models" evaluates LLMs on clinical/biomedical NLI and semantic textual similarity — domains requiring expert annotation, yielding small datasets (<2,000 examples). Three persistent problems:
Low accuracy in low-resource knowledge-rich domains — exposure bias: LLMs are not exposed to sufficient domain-specific training examples, so their NLI/STS accuracy in clinical contexts is substantially lower than in general domains. General benchmark performance does not predict specialized domain performance.
Overconfidence — models make incorrect predictions over-confidently. This is dangerous in safety-critical applications: an LLM that is wrong and certain provides no useful signal for downstream decision support. Prompting LLMs, which showed dramatic improvement on general NLI tasks in the text-davinci era, does not solve overconfidence in specialized domains.
Difficulty capturing collective human opinion distributions — NLI annotation sometimes reflects genuine human disagreement, and the distribution of opinions carries meaning beyond the majority label. Bayesian estimation of LLM uncertainty is computationally prohibitive; persona-based approaches (instructing LLMs to simulate different annotator profiles) are unstable.
The implication: the widely noted improvement in LLM NLI performance on standard benchmarks masks persistent fragility on specialized, knowledge-rich domains. Since Do classical knowledge definitions apply to AI systems?, LLMs may appear to reason well without having the domain knowledge that grounds reliable specialized inference.
This is a domain-specificity limitation that is structurally different from general reasoning failure — it emerges specifically at the boundary where general-purpose pretraining meets specialized expert knowledge. The vocabulary, entity relationships, and inference patterns of clinical medicine are not proportionally represented in general pretraining corpora.
Inquiring lines that read this note 39
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Can language models reliably simulate personas and predict behavior? Why do language models fail at sustained therapeutic relationships despite understanding techniques?- Why do Llama-based models outperform GPT-4 in objective clinical guidance?
- Why do LLMs understand therapy techniques but fail to execute them?
- What foundational barriers prevent LLMs from achieving clinical validity in therapy?
- Why do LLM outputs match researcher priors without solving tasks correctly?
- Why do NLP benchmarks systematically exclude ambiguous test cases from evaluation?
- Why do LLMs excel at generation but struggle with evaluation?
- Do LLMs struggle more with semantic accuracy than syntactic correctness across domains?
- Does LLM miscalibration cause failures in clinical information extraction?
- How well-calibrated are language models when making clinical predictions?
- Can an LLM be well calibrated but still unreliable on single evaluations?
- Why does prompt sensitivity vanish when model confidence is high?
- How does confidence in LLM outputs override users' ability to check accuracy?
- Why does general reasoning not transfer to knowledge-intensive medical domains?
- Why do models fail on logically equivalent tasks with different data distributions?
- Why do medical and mathematical tasks require fundamentally different model capabilities?
- Why do medical diagnoses require human judgment even with AI assistance?
- Do confidence signals mislead patients differently in medical versus other domains?
- Can an AI system trained on text consultations handle diagnostic uncertainty in real patient encounters?
- How much does self-play training with LLM-simulated patients actually improve diagnostic accuracy?
- How does expert annotation instability affect medical AI benchmarking?
- Which medical domains depend most on up-to-date information retrieval?
- Does medical domain competency require knowledge injection or better prompting?
- How much does prompt selection bias favor medical models over base models?
- Why did lay users with AI models fail to match unaided physicians on diagnosis?
- Does exposure to more domain-specific examples reduce LLM overconfidence?
- How much does missing images and tables limit LLM diagnostic reasoning?
- Can LLM performance on zebra cases predict results in routine clinical practice?
- Does medical fine-tuning help LLMs through knowledge or reasoning ability?
- How do LLM performances compare across different types of medical tasks?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do classical knowledge definitions apply to AI systems?
Classical definitions of knowledge assume truth-correspondence and a human knower. Do these assumptions hold for LLMs and distributed neural knowledge systems, or do they need fundamental revision?
LLM "knowledge" in specialized domains is thin and unreliable even when performance appears adequate on general benchmarks
-
Does LLM grammatical performance decline with structural complexity?
This explores whether LLMs fail uniformly at grammar or whether their failures follow a predictable pattern tied to input complexity. Understanding the relationship matters for deciding when LLM annotations are reliable.
domain specialization adds another axis of degradation beyond structural complexity
-
Why do LLM persona prompts produce inconsistent outputs across runs?
Can language models reliably simulate different social perspectives through persona prompting, or does their run-to-run variance indicate they lack stable group-specific knowledge? This matters for whether LLMs can approximate human disagreement in annotation tasks.
the persona-based approach to capturing opinion distributions also fails for the same reason
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- A Survey of Calibration Process for Black-Box LLMs
- Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
- Sequential Diagnosis with Language Models
- Capabilities of Gemini Models in Medicine
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Towards Accurate Differential Diagnosis with Large Language Models
- DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents
Original note title
llm overconfidence in domain-specific inference tasks persists in low-resource knowledge-rich domains