Clinical LLMs | Soroush Dianaty, M.D.

What 'Hallucination' Actually Means in Clinical LLMs (And How to Measure It)

Sun, 26 Jul 2026 00:00:00 +0000

In general NLP, a “hallucination” is typically defined as any model output that is unfaithful to the source prompt or factually inaccurate according to a reference corpus. When evaluating Large Language Models (LLMs) for general chat or creative writing, benchmark evaluation suites measure hallucination rates using string overlap, ROUGE scores, or LLM-as-a-judge sentiment.

However, when applied to clinical medicine, this standard definition is dangerously incomplete.

Why Standard NLP Hallucination Benchmarks Fail in Healthcare

A model scoring 90% on a general benchmark (such as MedQA or USMLE multiple-choice datasets) sounds reassuring to AI product managers. But in a clinical workflow, not all errors carry equal weight.

Consider two hypothetical model outputs generated for an emergency department attending physician:

Error A (Stylistic / Paraphrasing Error): The model summarizes a lab result as “Mildly elevated blood urea nitrogen observed” when the chart stated “BUN slightly above upper limit of normal.”
Error B (Asymmetrical Clinical Risk Error): The model omits a single line in a medication reconciliation summary: “No known penicillin allergy” when the chart listed a documented history of severe anaphylaxis to amoxicillin.

On standard NLP metrics (ROUGE-L, BLEU, or token similarity), Error A receives a penalty due to word choice mismatch, while Error B receives a near-perfect score because 98% of the words matched the chart correctly.

To a clinician, however, Error A is harmless stylistic variation, whereas Error B is potentially fatal.

flowchart TD A["Model Output Received"] --> B{"Evaluated By Benchmark"} B -->|Standard NLP Metric| C["Token Overlap / BLEU / ROUGE"] C -->|Treats All Tokens Equally| D["Misleading Safety Score"] B -->|Clinical AI Safety Taxonomy| E["1. Contradiction Risk"] B -->|Clinical AI Safety Taxonomy| F["2. Unsubstantiated Omission"] B -->|Clinical AI Safety Taxonomy| G["3. Provenance Loss"] E & F & G --> H["Bedside Safety Clearance"]

A 3-Tier Taxonomy of Clinical Hallucinations

To evaluate LLMs effectively for hospital and digital health deployment, we propose partitioning model hallucinations into three clinically distinct categories:

1. Direct Clinical Contradictions (Highest Risk)

An output that explicitly contradicts verifiable facts in the patient chart or established clinical guidelines.

Example: Recommending Beta-blockers for a patient presenting with active severe bradycardia.

2. Unsubstantiated Inferences (Medium Risk)

An output that asserts a diagnosis, lab trend, or past medical history item that is neither stated in nor logically inferable from the available record, even if the statement happens to be plausible.

Example: Stating a patient has “Type 2 Diabetes” based solely on a high BMI without a documented HbA1c or diagnostic code.

3. Provenance Loss & Omission (High Operational Risk)

An output that makes a correct clinical assertion but loses the verifiable audit trail (e.g., citing the wrong date, attributing a specialist note to the wrong provider, or omitting critical allergy/contraindication flags).

Measuring Clinical Hallucinations with Evidence Grounding

At Arizona State University, our research focuses on moving beyond static multiple-choice benchmarks toward evidence-grounded evaluation pipelines.

In tools like , we enforce character-level bounding-box provenance and 4-stage quality control loops (Rater $\rightarrow$ IAA $\rightarrow$ Adjudication $\rightarrow$ Reconciliation). By measuring the exact alignment between generated claims and W3C JSON-LD source metadata, we can calculate true Clinical Grounding Precision (CGP) rather than relying on uncalibrated text similarity.

Conclusion & Actionable Takeaways for Health-Tech Teams

Stop relying on USMLE exam scores as safety proof. Exam passing scores measure static knowledge recall, not real-time clinical safety.
Adopt asymmetric risk weighting. Weight allergy omissions, dosage miscalculations, and temporal ordering errors heavily over stylistic differences.
Require mandatory evidence provenance. Every generated clinical recommendation must cite specific, verifiable source note offsets before being displayed to care providers.

Read more about our ongoing research on or explore our .

Evaluating Clinical LLMs: Beyond Standard NLP Benchmarks

Sat, 25 Jul 2026 00:00:00 +0000

General-purpose Large Language Models (LLMs) continue to show impressive performance on standardized medical exams (such as USMLE question sets). However, achieving a high score on a multiple-choice exam is fundamentally different from providing clinically safe, evidence-grounded, and context-aware guidance in real-world patient care settings.

When an LLM is deployed in a hospital or clinic, a single plausible-sounding hallucination can lead to inappropriate treatment, delayed diagnoses, or compromised patient privacy.

The Limits of Standard NLP Benchmarks in Healthcare

Standard benchmarks like MMLU, MedQA, or GSM8K evaluate static knowledge retrieval and multi-choice reasoning. However, clinical environments present distinct challenges that these benchmarks miss:

Unstructured & Dynamic Context: Real patient charts contain fragmented clinical notes, lab trajectories, and temporal dependencies.
Asymmetrical Risk: In medicine, false positives and false negatives carry drastically unequal risks. A missing allergy alert is far more catastrophic than a redundant warning.
Auditable Evidence Grounding: Clinicians cannot rely on black-box predictions. Every clinical recommendation must cite specific, verifiable evidence from authoritative guidelines or patient EHR records.

flowchart TD A["General NLP Benchmarks (MMLU / MedQA)"] -->|Multiple Choice Tests| B["High Exam Accuracy"] B -->|Lacks Context & Provenance| C["Risk of Clinical Hallucination"] D["Rigorous Clinical AI Evaluation"] --> E["1. Multi-Stage Evidence Grounding (EviTrace)"] D --> F["2. Granular Data Access Controls (FHIR)"] D --> G["3. Context-Aware Privacy Classification"] E & F & G --> H["Deployable & Trustworthy Clinical AI"]

Three Pillars of Trustworthy Clinical AI

1. Multi-Stage Evidence Grounding

Rather than relying on single-pass generation, clinical AI pipelines must extract structured attributes from scientific literature and EHR data with auditable provenance. In our open-source project EviTrace, we implement a 4-stage quality control loop (Rater $\rightarrow$ Inter-Annotator Agreement $\rightarrow$ Adjudication $\rightarrow$ Reconciliation) to ensure that every output field is anchored in W3C JSON-LD metadata.

2. Context-Aware Sensitive Data Classification

Privacy is paramount. In our recent work presented at the AcademyHealth Annual Research Meeting 2026, we demonstrate how context-aware LLM architectures can accurately classify sensitive health records (such as substance use disorders or mental health records) under granular data segmentation rules.

3. Interoperability & Standards Compliance

AI tools must integrate directly with existing hospital EHR systems using open standards like HL7 FHIR and clinical terminologies (SNOMED-CT, LOINC, ICD-10). As shown in our research published in Applied Clinical Informatics, granular data segmentation in FHIR servers is critical for preserving patient consent while maintaining clinical utility.

Conclusion & Future Directions

Building trustworthy clinical AI requires bridging the gap between computational data science and frontline medical practice. Moving forward, the focus must shift from chasing raw model scale to developing rigorous, domain-specific evaluation frameworks that guarantee safety, transparency, and evidence grounding.

Early Evidence for Context-Aware Large Language Models (LLMs) in Sensitive Health Data Classification

Sat, 30 May 2026 00:00:00 +0000

Early Evidence for Context-Aware Large Language Models (LLMs) in Sensitive Health Data Classification

Sat, 30 May 2026 00:00:00 +0000