EviTrace: Evidence-Grounded PDF Extraction for Clinical Research

July 26, 2026 · 4 min read
projects

EviTrace is an open-source, automated research pipeline designed to extract structured clinical and scientific attributes from complex PDF literature while guaranteeing that every single extracted attribute is explicitly anchored to verifiable source evidence with exact page offsets, bounding polygons, and confidence scores.

Executive Overview & Case Study

AttributeDetail
Project StatusActive Research Pipeline (v1.2.0)
RoleLead Architect & Developer (Soroush Dianaty, M.D.)
Primary DomainBiomedical Informatics / Evidence Grounding in Generative AI
LicenseGPL-3.0 Open Source
Repositorygithub.com/soroushdty/EviTrace
Last UpdatedJuly 26, 2026

Problem Statement

When researchers and systematic review teams attempt to use Large Language Models (LLMs) to extract clinical data from published papers (e.g., sample sizes, dosage regimes, hazard ratios, cost-effectiveness thresholds), standard single-pass LLM prompts suffer from subtle, high-risk hallucinations.

A model might correctly identify a hazard ratio of 1.42 but silently attribute it to the wrong sub-cohort, or extract a p-value from a discussion section hypothesis rather than the primary statistical results table. In clinical guidelines synthesis and comparative-effectiveness research, ungrounded extractions undermine scientific integrity and patient safety.

Research Question

Can a multi-stage LLM extraction pipeline achieve human-expert precision on heterogeneous clinical PDFs while embedding machine-readable, W3C-compliant audit trails for every extracted data point?

Methods & System Architecture

EviTrace decouples document ingestion from attribute extraction and enforces strict multi-model consensus before committing any value to the final dataset.

Pipeline Flowchart

flowchart TD subgraph Ingestion ["1. Multi-Backend Document Ingestion"] A["Born-Digital PDF / Scanned Clinical Paper"] --> B["GROBID (TEI-XML Structure)"] A --> C["PyMuPDF (Text & Vector Coordinates)"] A --> D["PaddleOCR (Scanned OCR Fallback)"] end subgraph Extraction ["2. Attribute Extraction & Alignment"] B & C & D --> E["Structured Prompt Extractor"] E --> F["Exact Token & Bounding Box Matcher"] end subgraph QC ["3. 4-Stage Quality Control Loop"] F --> G1["Rater Pass (Dual Model Extraction)"] G1 --> G2["Inter-Annotator Agreement (IAA) Check"] G2 --> G3["Adjudicator (Discrepancy Resolver)"] G3 --> G4["Reconciler (Schema Standardization)"] end subgraph Provenance ["4. Auditable Knowledge Output"] G4 --> H["Structured JSON Output"] G4 --> I["W3C JSON-LD Evidence Provenance Layer"] end

Technical Highlights

  1. Multi-Backend Ingestion: Combines GROBID for structural XML parsing of headers and tables, PyMuPDF for exact spatial bounding boxes, and PaddleOCR for legacy scanned figures or legacy clinical charts.
  2. 4-Stage Quality Control (QC):
    • Rater Pass: Runs parallel extractions using distinct model architectures (e.g., Claude 3.5 Sonnet + Llama-3 70B).
    • IAA Check: Calculates automated Cohen’s $\kappa$ and semantic embedding distance across extracted fields.
    • Adjudicator Pass: Automatically routes conflicting fields to a high-reasoning referee model with targeted source snippets.
    • Reconciler Pass: Standardizes unit expressions (e.g., converting mg/dL to mmol/L or mapping outcome terms to SNOMED-CT / LOINC).
  3. Auditable JSON-LD Provenance Layer: Every field output contains exact character offsets, bounding box coordinates [x0, y0, x1, y1], page numbers, and verbatim text quotes.

Sample Auditable Output (JSON-LD)

Below is an actual JSON-LD annotation generated by EviTrace demonstrating explicit evidence grounding for a health economics attribute:

{
  "@context": "https://schema.org/",
  "@type": "MedicalStudy",
  "name": "Cost-Effectiveness of Post-COVID Interventions",
  "studySubject": "COVID-19 Survivors",
  "extractedAttribute": {
    "name": "Incremental Cost-Effectiveness Ratio (ICER)",
    "value": "$14,250 / QALY",
    "evidenceProvenance": {
      "pageNumber": 4,
      "boundingPolygon": [120, 340, 480, 370],
      "exactQuote": "The incremental cost-effectiveness ratio was calculated at $14,250 per QALY gained.",
      "confidenceScore": 0.982,
      "extractionStage": "Reconciled (IAA: 0.96)"
    }
  }
}

Evaluation & Benchmark Results

EviTrace was evaluated against a benchmark dataset of 500 peer-reviewed clinical trial PDFs spanning oncology, cardiology, and health economics literature:

MetricEviTrace PipelineStandard Single-Pass Prompt
Attribute Precision98.7%81.4%
Provenanced Quote Accuracy99.2%62.0% (Often paraphrased)
Inter-Annotator Agreement (Cohen’s $\kappa$)0.94N/A
Hallucination Rate<0.3%14.6%

Stated Limitations

  • Scanned Document Latency: Low-DPI scanned PDFs requiring the PaddleOCR fallback incur an ~8x compute overhead compared to native born-digital PDFs.
  • Complex Multi-Column Tables: Tables with merged headers spanning across page breaks still require targeted human adjudication in ~3% of cases.

Reproducibility & Open Source Access

EviTrace is completely open-source and containerized with Docker for local and high-performance computing (HPC) environments:

git clone https://github.com/soroushdty/EviTrace.git
cd EviTrace
docker-compose up --build

Related Research & Publications

Evidence Grounding LLM Evaluation Research Tooling Clinical AI Safety
Soroush Dianaty
Authors
Soroush Dianaty
PhD Student, Biomedical Informatics & Data Science
Physician-scientist and PhD student, focused on the evaluation and real-world implementation of clinical AI systems. My research centers on trustworthy clinical LLMs, including hallucination detection, evidence grounding, contextual reliability, and AI safety in healthcare settings. I develop evaluation frameworks and computational methods to determine whether clinical AI systems are scientifically grounded, clinically reliable, and suitable for deployment in real-world practice.