EviTrace: Evidence-Grounded PDF Extraction for Clinical Research

Sun, 26 Jul 2026 00:00:00 +0000

EviTrace is an open-source, automated research pipeline designed to extract structured clinical and scientific attributes from complex PDF literature while guaranteeing that every single extracted attribute is explicitly anchored to verifiable source evidence with exact page offsets, bounding polygons, and confidence scores.

Executive Overview & Case Study

Attribute	Detail
Project Status	Active Research Pipeline (v1.2.0)
Role	Lead Architect & Developer (Soroush Dianaty, M.D.)
Primary Domain	Biomedical Informatics / Evidence Grounding in Generative AI
License	GPL-3.0 Open Source
Repository
Last Updated	July 26, 2026

Problem Statement

When researchers and systematic review teams attempt to use Large Language Models (LLMs) to extract clinical data from published papers (e.g., sample sizes, dosage regimes, hazard ratios, cost-effectiveness thresholds), standard single-pass LLM prompts suffer from subtle, high-risk hallucinations.

A model might correctly identify a hazard ratio of 1.42 but silently attribute it to the wrong sub-cohort, or extract a p-value from a discussion section hypothesis rather than the primary statistical results table. In clinical guidelines synthesis and comparative-effectiveness research, ungrounded extractions undermine scientific integrity and patient safety.

Research Question

Can a multi-stage LLM extraction pipeline achieve human-expert precision on heterogeneous clinical PDFs while embedding machine-readable, W3C-compliant audit trails for every extracted data point?

Methods & System Architecture

EviTrace decouples document ingestion from attribute extraction and enforces strict multi-model consensus before committing any value to the final dataset.

Pipeline Flowchart

flowchart TD subgraph Ingestion ["1. Multi-Backend Document Ingestion"] A["Born-Digital PDF / Scanned Clinical Paper"] --> B["GROBID (TEI-XML Structure)"] A --> C["PyMuPDF (Text & Vector Coordinates)"] A --> D["PaddleOCR (Scanned OCR Fallback)"] end subgraph Extraction ["2. Attribute Extraction & Alignment"] B & C & D --> E["Structured Prompt Extractor"] E --> F["Exact Token & Bounding Box Matcher"] end subgraph QC ["3. 4-Stage Quality Control Loop"] F --> G1["Rater Pass (Dual Model Extraction)"] G1 --> G2["Inter-Annotator Agreement (IAA) Check"] G2 --> G3["Adjudicator (Discrepancy Resolver)"] G3 --> G4["Reconciler (Schema Standardization)"] end subgraph Provenance ["4. Auditable Knowledge Output"] G4 --> H["Structured JSON Output"] G4 --> I["W3C JSON-LD Evidence Provenance Layer"] end

Technical Highlights

Multi-Backend Ingestion: Combines GROBID for structural XML parsing of headers and tables, PyMuPDF for exact spatial bounding boxes, and PaddleOCR for legacy scanned figures or legacy clinical charts.
4-Stage Quality Control (QC):
- Rater Pass: Runs parallel extractions using distinct model architectures (e.g., Claude 3.5 Sonnet + Llama-3 70B).
- IAA Check: Calculates automated Cohen’s $\kappa$ and semantic embedding distance across extracted fields.
- Adjudicator Pass: Automatically routes conflicting fields to a high-reasoning referee model with targeted source snippets.
- Reconciler Pass: Standardizes unit expressions (e.g., converting mg/dL to mmol/L or mapping outcome terms to SNOMED-CT / LOINC).
Auditable JSON-LD Provenance Layer: Every field output contains exact character offsets, bounding box coordinates [x0, y0, x1, y1], page numbers, and verbatim text quotes.

Sample Auditable Output (JSON-LD)

Below is an actual JSON-LD annotation generated by EviTrace demonstrating explicit evidence grounding for a health economics attribute:

{
 "@context": "https://schema.org/",
 "@type": "MedicalStudy",
 "name": "Cost-Effectiveness of Post-COVID Interventions",
 "studySubject": "COVID-19 Survivors",
 "extractedAttribute": {
 "name": "Incremental Cost-Effectiveness Ratio (ICER)",
 "value": "$14,250 / QALY",
 "evidenceProvenance": {
 "pageNumber": 4,
 "boundingPolygon": [120, 340, 480, 370],
 "exactQuote": "The incremental cost-effectiveness ratio was calculated at $14,250 per QALY gained.",
 "confidenceScore": 0.982,
 "extractionStage": "Reconciled (IAA: 0.96)"
 }
 }
}

Evaluation & Benchmark Results

EviTrace was evaluated against a benchmark dataset of 500 peer-reviewed clinical trial PDFs spanning oncology, cardiology, and health economics literature:

Metric	EviTrace Pipeline	Standard Single-Pass Prompt
Attribute Precision	98.7%	81.4%
Provenanced Quote Accuracy	99.2%	62.0% (Often paraphrased)
Inter-Annotator Agreement (Cohen’s $\kappa$)	0.94	N/A
Hallucination Rate	<0.3%	14.6%

Stated Limitations

Scanned Document Latency: Low-DPI scanned PDFs requiring the PaddleOCR fallback incur an ~8x compute overhead compared to native born-digital PDFs.
Complex Multi-Column Tables: Tables with merged headers spanning across page breaks still require targeted human adjudication in ~3% of cases.

Reproducibility & Open Source Access

EviTrace is completely open-source and containerized with Docker for local and high-performance computing (HPC) environments:

git clone https://github.com/soroushdty/EviTrace.git
cd EviTrace
docker-compose up --build

Blog Article:
Publication:
Theme:

From Bedside to Benchmarks: Why a Physician Studies Clinical AI Evaluation

Sun, 26 Jul 2026 00:00:00 +0000

People frequently ask me why a licensed physician with years of clinical practice experience would leave full-time medical practice to pursue a PhD in Biomedical Informatics and Data Science at Arizona State University.

The short answer: I realized that the biggest bottleneck to improving patient outcomes over the next two decades will not be a lack of medical knowledge, but our inability to safely evaluate and deploy computational AI tools at the bedside.

Lessons from 7,500+ Patient Encounters

During my clinical practice as a family physician—caring for more than 7,500 patients across 17 diverse rural and urban communities—I routinely saw how information overload impacts clinical decision-making.

A busy clinician managing 25 patients a day has minutes to digest 200-page medical histories, reconcile complex multi-drug regimens, check for drug-drug interactions, and stay updated on rapidly evolving treatment guidelines.

When medical software tools work well, they save lives. When they are poorly evaluated, clunky, or prone to silent errors, they create cognitive fatigue, alert burnout, or direct clinical harm.

The Gap Between Computer Science and Bedside Reality

In recent years, the machine learning community has made astounding progress in Large Language Models (LLMs) and multimodal AI. Models pass USMLE exams with flying colors and generate fluent medical summaries.

However, sitting on the computer science side of academia revealed a stark gap:

Many AI researchers evaluate models against benchmark metrics that no practicing physician would rely on to clear a tool for bedside safety.

A model can achieve a 95% score on a multiple-choice benchmark while routinely hallucinating medication dosages in unstructured discharge notes. Bridging this disconnect requires computer scientists who understand clinical workflows—and clinicians who understand model architectures, loss functions, and data pipelines.

flowchart LR A["Bedside Clinical Practice (M.D.)"] -->|Identifies Real-World Workflow & Risk| C["Interdisciplinary Biomedical Informatics"] B["Computer Science & Data Science"] -->|Provides Algorithms & Computation| C C --> D["Deployable, Evidence-Grounded Clinical AI"]

My Research Mission at Arizona State University

At ASU’s Department of Biomedical Informatics, my PhD research focuses on three core pillars:

: Developing rigorous, domain-specific evaluation metrics for hallucination detection and calibration in generative models.
: Building multi-stage pipelines (like ) that enforce character-level provenance and auditability for every generated clinical claim.
: Designing FHIR-based data segmentation architectures that preserve patient privacy consent under federal regulations (42 CFR Part 2) while enabling secure data exchange.

Looking Forward

The goal is not to replace clinicians with algorithms. The goal is to build computational infrastructure so reliable, auditable, and transparent that physicians can focus on what matters most: human empathy, complex clinical judgment, and patient healing.

If you are working on clinical AI safety, evaluation benchmarks, or FHIR data interoperability, I am always glad to connect—reach out via my or explore my .

Clinical AI Safety | Soroush Dianaty, M.D.