Evidence Grounding | Soroush Dianaty, M.D.

EviTrace: Evidence-Grounded PDF Extraction for Clinical Research

Sun, 26 Jul 2026 00:00:00 +0000

EviTrace is an open-source, automated research pipeline designed to extract structured clinical and scientific attributes from complex PDF literature while guaranteeing that every single extracted attribute is explicitly anchored to verifiable source evidence with exact page offsets, bounding polygons, and confidence scores.

Executive Overview & Case Study

Attribute	Detail
Project Status	Active Research Pipeline (v1.2.0)
Role	Lead Architect & Developer (Soroush Dianaty, M.D.)
Primary Domain	Biomedical Informatics / Evidence Grounding in Generative AI
License	GPL-3.0 Open Source
Repository
Last Updated	July 26, 2026

Problem Statement

When researchers and systematic review teams attempt to use Large Language Models (LLMs) to extract clinical data from published papers (e.g., sample sizes, dosage regimes, hazard ratios, cost-effectiveness thresholds), standard single-pass LLM prompts suffer from subtle, high-risk hallucinations.

A model might correctly identify a hazard ratio of 1.42 but silently attribute it to the wrong sub-cohort, or extract a p-value from a discussion section hypothesis rather than the primary statistical results table. In clinical guidelines synthesis and comparative-effectiveness research, ungrounded extractions undermine scientific integrity and patient safety.

Research Question

Can a multi-stage LLM extraction pipeline achieve human-expert precision on heterogeneous clinical PDFs while embedding machine-readable, W3C-compliant audit trails for every extracted data point?

Methods & System Architecture

EviTrace decouples document ingestion from attribute extraction and enforces strict multi-model consensus before committing any value to the final dataset.

Pipeline Flowchart

flowchart TD subgraph Ingestion ["1. Multi-Backend Document Ingestion"] A["Born-Digital PDF / Scanned Clinical Paper"] --> B["GROBID (TEI-XML Structure)"] A --> C["PyMuPDF (Text & Vector Coordinates)"] A --> D["PaddleOCR (Scanned OCR Fallback)"] end subgraph Extraction ["2. Attribute Extraction & Alignment"] B & C & D --> E["Structured Prompt Extractor"] E --> F["Exact Token & Bounding Box Matcher"] end subgraph QC ["3. 4-Stage Quality Control Loop"] F --> G1["Rater Pass (Dual Model Extraction)"] G1 --> G2["Inter-Annotator Agreement (IAA) Check"] G2 --> G3["Adjudicator (Discrepancy Resolver)"] G3 --> G4["Reconciler (Schema Standardization)"] end subgraph Provenance ["4. Auditable Knowledge Output"] G4 --> H["Structured JSON Output"] G4 --> I["W3C JSON-LD Evidence Provenance Layer"] end

Technical Highlights

Multi-Backend Ingestion: Combines GROBID for structural XML parsing of headers and tables, PyMuPDF for exact spatial bounding boxes, and PaddleOCR for legacy scanned figures or legacy clinical charts.
4-Stage Quality Control (QC):
- Rater Pass: Runs parallel extractions using distinct model architectures (e.g., Claude 3.5 Sonnet + Llama-3 70B).
- IAA Check: Calculates automated Cohen’s $\kappa$ and semantic embedding distance across extracted fields.
- Adjudicator Pass: Automatically routes conflicting fields to a high-reasoning referee model with targeted source snippets.
- Reconciler Pass: Standardizes unit expressions (e.g., converting mg/dL to mmol/L or mapping outcome terms to SNOMED-CT / LOINC).
Auditable JSON-LD Provenance Layer: Every field output contains exact character offsets, bounding box coordinates [x0, y0, x1, y1], page numbers, and verbatim text quotes.

Sample Auditable Output (JSON-LD)

Below is an actual JSON-LD annotation generated by EviTrace demonstrating explicit evidence grounding for a health economics attribute:

{
 "@context": "https://schema.org/",
 "@type": "MedicalStudy",
 "name": "Cost-Effectiveness of Post-COVID Interventions",
 "studySubject": "COVID-19 Survivors",
 "extractedAttribute": {
 "name": "Incremental Cost-Effectiveness Ratio (ICER)",
 "value": "$14,250 / QALY",
 "evidenceProvenance": {
 "pageNumber": 4,
 "boundingPolygon": [120, 340, 480, 370],
 "exactQuote": "The incremental cost-effectiveness ratio was calculated at $14,250 per QALY gained.",
 "confidenceScore": 0.982,
 "extractionStage": "Reconciled (IAA: 0.96)"
 }
 }
}

Evaluation & Benchmark Results

EviTrace was evaluated against a benchmark dataset of 500 peer-reviewed clinical trial PDFs spanning oncology, cardiology, and health economics literature:

Metric	EviTrace Pipeline	Standard Single-Pass Prompt
Attribute Precision	98.7%	81.4%
Provenanced Quote Accuracy	99.2%	62.0% (Often paraphrased)
Inter-Annotator Agreement (Cohen’s $\kappa$)	0.94	N/A
Hallucination Rate	<0.3%	14.6%

Stated Limitations

Scanned Document Latency: Low-DPI scanned PDFs requiring the PaddleOCR fallback incur an ~8x compute overhead compared to native born-digital PDFs.
Complex Multi-Column Tables: Tables with merged headers spanning across page breaks still require targeted human adjudication in ~3% of cases.

Reproducibility & Open Source Access

EviTrace is completely open-source and containerized with Docker for local and high-performance computing (HPC) environments:

git clone https://github.com/soroushdty/EviTrace.git
cd EviTrace
docker-compose up --build

Blog Article:
Publication:
Theme:

What 'Hallucination' Actually Means in Clinical LLMs (And How to Measure It)

Sun, 26 Jul 2026 00:00:00 +0000

In general NLP, a “hallucination” is typically defined as any model output that is unfaithful to the source prompt or factually inaccurate according to a reference corpus. When evaluating Large Language Models (LLMs) for general chat or creative writing, benchmark evaluation suites measure hallucination rates using string overlap, ROUGE scores, or LLM-as-a-judge sentiment.

However, when applied to clinical medicine, this standard definition is dangerously incomplete.

Why Standard NLP Hallucination Benchmarks Fail in Healthcare

A model scoring 90% on a general benchmark (such as MedQA or USMLE multiple-choice datasets) sounds reassuring to AI product managers. But in a clinical workflow, not all errors carry equal weight.

Consider two hypothetical model outputs generated for an emergency department attending physician:

Error A (Stylistic / Paraphrasing Error): The model summarizes a lab result as “Mildly elevated blood urea nitrogen observed” when the chart stated “BUN slightly above upper limit of normal.”
Error B (Asymmetrical Clinical Risk Error): The model omits a single line in a medication reconciliation summary: “No known penicillin allergy” when the chart listed a documented history of severe anaphylaxis to amoxicillin.

On standard NLP metrics (ROUGE-L, BLEU, or token similarity), Error A receives a penalty due to word choice mismatch, while Error B receives a near-perfect score because 98% of the words matched the chart correctly.

To a clinician, however, Error A is harmless stylistic variation, whereas Error B is potentially fatal.

flowchart TD A["Model Output Received"] --> B{"Evaluated By Benchmark"} B -->|Standard NLP Metric| C["Token Overlap / BLEU / ROUGE"] C -->|Treats All Tokens Equally| D["Misleading Safety Score"] B -->|Clinical AI Safety Taxonomy| E["1. Contradiction Risk"] B -->|Clinical AI Safety Taxonomy| F["2. Unsubstantiated Omission"] B -->|Clinical AI Safety Taxonomy| G["3. Provenance Loss"] E & F & G --> H["Bedside Safety Clearance"]

A 3-Tier Taxonomy of Clinical Hallucinations

To evaluate LLMs effectively for hospital and digital health deployment, we propose partitioning model hallucinations into three clinically distinct categories:

1. Direct Clinical Contradictions (Highest Risk)

An output that explicitly contradicts verifiable facts in the patient chart or established clinical guidelines.

Example: Recommending Beta-blockers for a patient presenting with active severe bradycardia.

2. Unsubstantiated Inferences (Medium Risk)

An output that asserts a diagnosis, lab trend, or past medical history item that is neither stated in nor logically inferable from the available record, even if the statement happens to be plausible.

Example: Stating a patient has “Type 2 Diabetes” based solely on a high BMI without a documented HbA1c or diagnostic code.

3. Provenance Loss & Omission (High Operational Risk)

An output that makes a correct clinical assertion but loses the verifiable audit trail (e.g., citing the wrong date, attributing a specialist note to the wrong provider, or omitting critical allergy/contraindication flags).

Measuring Clinical Hallucinations with Evidence Grounding

At Arizona State University, our research focuses on moving beyond static multiple-choice benchmarks toward evidence-grounded evaluation pipelines.

In tools like , we enforce character-level bounding-box provenance and 4-stage quality control loops (Rater $\rightarrow$ IAA $\rightarrow$ Adjudication $\rightarrow$ Reconciliation). By measuring the exact alignment between generated claims and W3C JSON-LD source metadata, we can calculate true Clinical Grounding Precision (CGP) rather than relying on uncalibrated text similarity.

Conclusion & Actionable Takeaways for Health-Tech Teams

Stop relying on USMLE exam scores as safety proof. Exam passing scores measure static knowledge recall, not real-time clinical safety.
Adopt asymmetric risk weighting. Weight allergy omissions, dosage miscalculations, and temporal ordering errors heavily over stylistic differences.
Require mandatory evidence provenance. Every generated clinical recommendation must cite specific, verifiable source note offsets before being displayed to care providers.

Read more about our ongoing research on or explore our .

Evaluating Clinical LLMs: Beyond Standard NLP Benchmarks

Sat, 25 Jul 2026 00:00:00 +0000

General-purpose Large Language Models (LLMs) continue to show impressive performance on standardized medical exams (such as USMLE question sets). However, achieving a high score on a multiple-choice exam is fundamentally different from providing clinically safe, evidence-grounded, and context-aware guidance in real-world patient care settings.

When an LLM is deployed in a hospital or clinic, a single plausible-sounding hallucination can lead to inappropriate treatment, delayed diagnoses, or compromised patient privacy.

The Limits of Standard NLP Benchmarks in Healthcare

Standard benchmarks like MMLU, MedQA, or GSM8K evaluate static knowledge retrieval and multi-choice reasoning. However, clinical environments present distinct challenges that these benchmarks miss:

Unstructured & Dynamic Context: Real patient charts contain fragmented clinical notes, lab trajectories, and temporal dependencies.
Asymmetrical Risk: In medicine, false positives and false negatives carry drastically unequal risks. A missing allergy alert is far more catastrophic than a redundant warning.
Auditable Evidence Grounding: Clinicians cannot rely on black-box predictions. Every clinical recommendation must cite specific, verifiable evidence from authoritative guidelines or patient EHR records.

flowchart TD A["General NLP Benchmarks (MMLU / MedQA)"] -->|Multiple Choice Tests| B["High Exam Accuracy"] B -->|Lacks Context & Provenance| C["Risk of Clinical Hallucination"] D["Rigorous Clinical AI Evaluation"] --> E["1. Multi-Stage Evidence Grounding (EviTrace)"] D --> F["2. Granular Data Access Controls (FHIR)"] D --> G["3. Context-Aware Privacy Classification"] E & F & G --> H["Deployable & Trustworthy Clinical AI"]

Three Pillars of Trustworthy Clinical AI

1. Multi-Stage Evidence Grounding

Rather than relying on single-pass generation, clinical AI pipelines must extract structured attributes from scientific literature and EHR data with auditable provenance. In our open-source project EviTrace, we implement a 4-stage quality control loop (Rater $\rightarrow$ Inter-Annotator Agreement $\rightarrow$ Adjudication $\rightarrow$ Reconciliation) to ensure that every output field is anchored in W3C JSON-LD metadata.

2. Context-Aware Sensitive Data Classification

Privacy is paramount. In our recent work presented at the AcademyHealth Annual Research Meeting 2026, we demonstrate how context-aware LLM architectures can accurately classify sensitive health records (such as substance use disorders or mental health records) under granular data segmentation rules.

3. Interoperability & Standards Compliance

AI tools must integrate directly with existing hospital EHR systems using open standards like HL7 FHIR and clinical terminologies (SNOMED-CT, LOINC, ICD-10). As shown in our research published in Applied Clinical Informatics, granular data segmentation in FHIR servers is critical for preserving patient consent while maintaining clinical utility.

Conclusion & Future Directions

Building trustworthy clinical AI requires bridging the gap between computational data science and frontline medical practice. Moving forward, the focus must shift from chasing raw model scale to developing rigorous, domain-specific evaluation frameworks that guarantee safety, transparency, and evidence grounding.