What Has to Survive Extraction From a Clinical Note

Chen Li | Fractional Executive Advisor, AI / ML / Robotics, Eventum

Compile-Time Intelligence

A medication list includes metformin, while two lines later the patient reports stopping the drug two months ago. An extraction pipeline that emits metformin (present) identifies the drug string while discarding the temporal and source attributes that determine whether a downstream rule should treat the patient as currently medicated.

Clinical information extraction requires more than entity recognition. Associating a patient identifier with a diagnosis code or medication string does not establish a queryable clinical assertion unless the output schema preserves the syntactic and clinical qualifiers surrounding the mention. Without those qualifiers, a downstream execution engine or human reviewer cannot recover what the clinical note actually states.

In Eventum’s Compile-Time Intelligence series, David Bressler examines what a formal verifier establishes about an encoded rule and what a test suite establishes about its runtime behavior [1] [2]. This post addresses the upstream data representation required before a rule evaluates a patient chart. Converting unstructured clinical text into structured predicates requires a schema that captures six attributes alongside the extracted concept, value, and unit: polarity, certainty, temporal anchor, subject, information source, and the supporting character span. The schema also requires an explicit state representing that the examined documents do not record the target concept.

Why the information in notes is worth the difficulty

A billing or problem-list diagnosis code records that a condition was logged at an encounter without specifying whether the clinician considered the condition suspected, historical, ruled out, or active. Narrative clinical notes contain those modifiers.

Empirical evaluations quantify the recall difference between structured tables and clinical text. I worked at Verantos from 2021 to 2024. The three studies summarized below evaluated software developed by that company, though I am not an author on any of the papers.

In a migraine phenotyping study at a single US health system across 6,032 manually annotated encounters, querying structured electronic health record fields and insurance claims achieved 66.6% recall for the migraine concept, whereas AI extraction of the same concept from clinical notes achieved 96.8% recall [3].

A heart-failure phenotyping study illustrates the difference between local phrase matching and longitudinal inference [4]. For heart failure with preserved ejection fraction (HFpEF), a sentence-level rule pipeline achieved an F1 score of 4.9%, while a pipeline that combined evidence across the longitudinal record reached an F1 score of 91.0%. Identifying HFpEF frequently required linking a clinical mention of heart failure in one location with a quantitative ejection fraction measurement recorded elsewhere (in the paper’s example, heart failure on the problem list and “EF 60%” in an encounter note). Because that synthesis combines premises across multiple spans into a derived assertion that rarely appears verbatim in a single sentence, the output schema must tag the record as a derived inference and retain pointers to each constituent premise.

A third evaluation, of asthma patients across 58 hospitals, reported a summary F1 of 59.5% when relying on structured EHR fields and open claims, compared with 93.4% when combining claims, structured EHR fields, extracted narrative concepts, and registry records [5]. That figure reflects the composite multi-source dataset rather than an isolated language model.

Six ways the meaning can disappear

Downstream clinical logic fails when an extraction pipeline retains a normalized clinical concept while dropping the relational operators that scope the statement in the source text.

Negation and uncertainty become a positive finding

The spans “No evidence of DVT” and “Possible early sepsis, will reassess” both contain disease terms, yet neither documents a confirmed diagnosis. Telling them apart depends on several cues, among them the clinician’s level of confidence, whether the sentence asserts, denies or speculates (its polarity and mood), and where the condition sits in the chain of cause and effect.

In the extraction schema, the polarity field records whether a finding is asserted or negated, while the certainty field records epistemic qualifiers such as possible, probable, or confirmed. How notes express negation also varies with the language and with local documentation habits. The schema must also keep a patient’s reported denial distinct from a clinician’s diagnostic exclusion rather than projecting both onto a single numerical confidence scale. Passing a qualified clinical statement through an extraction model does not increase the clinical certainty of the underlying observation.

A historical fact becomes a current one

“History of MI in 2019” does not establish an acute myocardial infarction at the index encounter, and “Started metformin” does not establish active medication use when a subsequent note documents discontinuation. Putting the elements of an encounter in temporal order (the chief complaint, when diagnostic criteria were met, when each medication was given) often takes several steps of reasoning, and that step deserves its own evaluation.

The time field binds each assertion to an encounter timestamp or an explicit event date. The schema must also distinguish a status documented directly as current from a status computed by chaining dated records across encounters so that downstream temporal logic can apply exact inclusion windows. How a note frames an event in time, which linguists call aspect (an event completed, or one still in progress), can also change the conclusion we draw. A common example is how differently transient and persistent ischemia are treated and followed up. A transient ischemic attack (TIA) may call for monitoring for occult arrhythmias and tracking of arterial stenosis, whereas persistent ischemia calls for urgent treatment to restore blood flow to the affected tissue, such as thrombolytic therapy. Whether a note describes the episode as resolved or ongoing therefore has to survive extraction.

Someone else’s condition becomes the patient’s

“Mother with breast cancer at 52” documents family history, while “Patient reports chest pain, exam unremarkable” pairs a patient symptom report with a clinician physical finding. A relative’s diagnosis can still change the patient’s risk, so a model has to treat it as evidence about that risk while keeping it off the patient’s own list of conditions.

The subject field specifies the individual who experiences the clinical state, and the source field specifies the provenance of the assertion (such as clinician assessment, patient self-report, active medication list, or historical correspondence). Downstream cohort definitions depend on both fields to filter out family history and weigh self-reported symptoms separately from confirmed diagnostic findings.

Multiple claims collapse into whichever was read last

A chart may list stage 3 chronic kidney disease on the persistent problem list while the nephrology assessment from the current encounter documents stage 4 disease. Similarly, an automated medication table may include a drug that the narrative assessment marks as discontinued.

An extraction pipeline must emit both assertions alongside their respective timestamps and document sources. A separate reconciliation module can then evaluate whether the pair represents disease progression over time, contradictory statements regarding the same time window, or distinct evidence tiers. Resolving conflicting chart statements by retaining whichever token sequence the parser encountered last corrupts longitudinal state tracking.

Normalization turns into an unmarked inference

Clinical shorthand such as “Pt c/o SOB x 3d, +orthopnea” requires lexical expansion during parsing. Expanding the string “SOB” to the standard concept for dyspnea is a vocabulary normalization operation, whereas concluding that the patient has decompensated heart failure is a clinical inference. The data model must log vocabulary mapping separately from diagnostic inference so that human reviewers and audit rules can inspect which operation produced a given record.

Silence or incomplete documentation becomes an answer

A clinical note that omits any mention of tobacco use does not establish that the patient is a non-smoker. Within the scope of the examined document, the patient’s smoking status has the formal value not_documented. A social-history entry is also a different kind of evidence from a laboratory measure of tobacco exposure, such as a cotinine test, an LC-MS/MS screen for nicotine metabolites, or urinary NNAL (a marker specific to tobacco). Spirometry and other pulmonary function tests answer a different question again: how well the lungs work. A clinical decision rarely has all of these results available, so the source field has to record which kind of evidence a smoking status rests on.

As Eventum’s production document-processing guide notes, a schema that lacks an explicit representation for absent documentation forces downstream consumers to conflate missing data with negative findings or unverified defaults [6].

Every extracted assertion also requires an evidence span pointing to the exact character offsets, sentence, or table cell captured at extraction time. Combined with document identifiers and relational links between premises, the six core properties (polarity, certainty, time, subject, source, and character span) alongside an explicit not_documented state preserve the operational semantics of the clinical text.

One note, with its meaning intact

Consider the following synthetic clinical note:

68 y/o M with HFrEF (EF 35%, 03/2025) presents with 3 days of worsening SOB and orthopnea. Denies chest pain. Possible early pneumonia vs CHF exacerbation, will reassess after diuresis. Mother with breast cancer at 52. Med list includes metformin. Pt reports he stopped metformin 2 months ago (GI upset).

Rather than projecting this paragraph into a flat bag of ICD-10 and RxNorm codes, the extraction schema represents each clinical statement as an independent, scoped record:

Downstream applications apply explicit execution policies to these records:

• A cohort rule selecting patients with active pneumonia evaluates whether certainty = possible satisfies the inclusion criteria.
• A medication-exposure query resolves the conflict between the active medication list entry and the patient’s reported discontinuation two months prior.
• A risk calculator requiring smoking status branches to its missing-data handler upon reading not_documented.

The extraction layer preserves the scoped assertions from the text so that the rule execution layer can apply domain policy explicitly.

Generated rules need these distinctions too

Language models are increasingly used to generate executable SQL or clinical quality language (CQL) queries over structured repositories. Syntactically valid query execution does not guarantee clinical accuracy.

In a 2024 evaluation across 84,821 patients, Chao Yan and colleagues benchmarked LLM-generated phenotyping queries against clinician-validated eMERGE algorithms [7]. After technical edits that left the clinical logic untouched (database names, for example), the generated queries ran, yet their classification performance varied widely. For dementia, the GPT-4 query achieved a reported positive predictive value (PPV) of 96.3% alongside a recall of 24.4%. For hypothyroidism, the GPT-4 query achieved 99.9% recall alongside a reported PPV of 9.6%. The GPT-3.5 query for type 2 diabetes returned zero patients.

Representing polarity, certainty, and subject as explicit schema columns enables automated metamorphic testing of both the extractor and the consuming query logic [15]. If a cohort rule targets active diagnoses in the index patient, mutating a synthetic test note by inserting a negated diagnosis (polarity = negated) or a family history mention (subject = mother) while holding all other claims constant must leave cohort membership invariant. Separating extraction attributes from rule evaluation isolates whether a false positive stems from a parser misreading negation or a SQL query omitting a WHERE clause filter.

Read the evaluation at the level of the claim

Aggregate F1 scores across an annotation benchmark frequently mask failure modes that break specific clinical workflows. Evaluating a clinical extraction pipeline requires checking four operational properties:

1. Agreement between qualified reviewers. The MEDEC benchmark, from Asma Ben Abacha, Wen-wai Yim and colleagues, evaluates medical error detection and correction across clinical notes [8]. When two physicians performed the detection task on the same 242 test notes, they agreed on whether a clinical error was present in 69.01% of cases and agreed on the specific sentence containing the error in 57.85% of cases. If physicians agree this rarely on the task itself, a reference set built from one reviewer’s labels needs an agreement check and a way to settle disagreements.
2. Cross-institutional generalization. A non-small cell lung cancer study by Sonish Sivarajkumar and colleagues, linking treatments to documented clinical responses, reported an F1 score of 0.92 on 64 held-out notes at the development institution [9]. When evaluated externally on 117 Mayo Clinic notes across five classifiers, F1 scores ranged from 0.49 to 0.64. Because the external evaluation restricted the task to chemotherapy-response relations and relied on Mayo Clinic gold entity spans and local annotation guidelines, the performance drop combines domain shift with task-definition differences. Deploying an extraction pipeline at a new clinical site requires evaluating the exact target schema on local notes.
3. Attribute-specific error rates. In a study automating the adjudication of cardiovascular events, a secondary evaluator model scored LLM-generated rationales at 0.98 for logical consistency while assigning a mean score of 0.31 for timeline accuracy [10]. Although the automated scorer had not yet been calibrated against physician ratings, the divergence between narrative plausibility and temporal ordering demonstrates why temporal extraction requires dedicated evaluation metrics. Additional studies on oncology trial matching (Cliff Wong and colleagues, including Naoto Usuyama) and case selection (Jacob Koskimaki and colleagues, including Hussein Al-Olimat) document task-specific extraction gains under narrower scope constraints [11] [12].
4. Calibration of model probability versus textual certainty. Prior work on EHR mortality prediction by Michael Dusenberry and colleagues demonstrates that population-level area under the receiver operating characteristic curve (AUROC) conceals substantial variance in individual patient probability estimates across model ensembles [13]. In clinical extraction, evaluation must separate the statistical confidence of the parser from the clinical certainty recorded by the physician. A parser with a 0.99 posterior probability that the note states “possible pneumonia” must emit certainty = possible rather than upgrading the clinical record to a confirmed diagnosis.

An operational evaluation should measure span-level evidence support, defined as the proportion of extracted records in which a qualified reviewer confirms that the linked character span substantiates the concept, polarity, certainty, time, and subject simultaneously. The multi-hospital asthma study measured dataset-level traceability by verifying whether an extracted data element could be located in its designated source-of-truth system [5]. Verifying that a document pointer resolves to an existing record is a necessary storage check, whereas verifying that the cited text span entails the full structured tuple is the semantic requirement for clinical review.

Preserve time without treating privacy as an afterthought

Capturing fine-grained temporal anchors interacts directly with regulatory privacy constraints. Replacing patient names with synthetic tokens removes direct identifiers without satisfying the HIPAA Privacy Rule de-identification standard [14].

Health and Human Services (HHS) guidance specifies two compliance paths: formal determination by a qualified statistical expert and the Safe Harbor method. Safe Harbor requires stripping all date elements (except the four-digit year) directly related to an individual, including dates embedded in unstructured clinical prose [14]. When a clinical workflow depends on computing exact day-level intervals between medication initiation, laboratory tests, and adverse events, stripping month and day fields under Safe Harbor destroys the required temporal signal. Engineering teams must settle the extraction schema and the de-identification method with privacy and legal counsel before processing clinical text.

The handoff is the product

A clinical extraction pipeline completes its task when the consuming software can programmatically distinguish a differential diagnosis from an established condition, a relative’s medical history from the patient’s chart, a discontinued medication from an active prescription, and an unrecorded attribute from a negated finding. Encoding polarity, certainty, time, subject, source, character span, and explicit missingness establishes a verifiable contract between unstructured clinical text, downstream execution rules, and human audit.

Eventum works with engineering and clinical informatics teams to design claim schemas, metamorphic evaluation suites, and span-linked document pipelines for clinical registries, quality measures, and executable guidelines. That lets clinicians and other care providers see what each piece of data rests on before they rely on it for a patient. Get in touch to discuss extraction that a registry, quality measure, or executable guideline will depend on.

References and research notes

1. Bressler D. AI Is Discovering New Mathematics. The Hard Part Is Knowing Exactly What It Proved. Eventum, 13 October 2026. Compile-Time Intelligence, Part 1.
2. Bressler D. The Rule Passed Its Tests. What Did the Tests Establish? Eventum, 19 October 2026. Compile-Time Intelligence, Part 2.
3. Riskin D, Cady R, Shroff A, Hindiyeh NA, Smith T, Kymes S. Using artificial intelligence to identify patients with migraine and associated symptoms and conditions within electronic health records. BMC Medical Informatics and Decision Making 23, 121 (2023). DOI: 10.1186/s12911-023-02190-8. Recall figures appear in the Abstract, and encounter counts appear in Methods and Results. Technology provided by Verantos, sponsored by Lundbeck.
   Research note. The migraine evaluation covered 6,032 manually annotated encounters and 2,642 migraine occurrences at a single health system. Recall was 66.6% for structured fields and claims versus 96.8% for AI-processed notes. Structured-arm precision sits near 100% in Figure 1, though the published bar chart omits numerical data labels.
4. Garan AR, Monda KL, Dent-Acosta RE, Riskin DJ, Gluckman TJ. Retrospective comparison of traditional and artificial intelligence-based heart failure phenotyping in a US health system to enable real-world evidence. BMJ Open 13(8), e073178 (2023). DOI: 10.1136/bmjopen-2023-073178. The 4.9% and 91.0% F1 figures appear in the Results text, and the structured arm’s zero recall for HFpEF appears in Supplementary Table 1. Technology provided by Verantos, funded by Amgen.
   Research note. The study evaluated 4,288 encounters across 1,155 patients at one academic health system. Structured diagnosis codes yielded 0% recall for HFpEF in this cohort. The 4.9% and 91.0% F1 scores compare sentence-level rules against a pipeline incorporating longitudinal inference for the HFpEF concept specifically rather than across all heart-failure phenotypes.
5. Riskin DJ, Monda KL, Gagne JJ, et al. Implementing Accuracy, Completeness, and Traceability for Data Reliability. JAMA Network Open 8(3), e250128 (2025), incorporating the correction of 10 April 2025. DOI: 10.1001/jamanetworkopen.2025.0128. Verantos-led, FDA-funded project.
   Research note. The study analyzed 120,616 asthma patients across 58 hospitals, with manual reference annotation on 8,044 encounters. Summary F1 was 59.5% for structured EHR records plus open claims (eAppendix 1) and 93.4% for the combined dataset integrating claims, structured EHR fields, AI-extracted unstructured notes, and registry data. The traceability evaluation compared the combined dataset (77.3%) against claims-only data (11.5%), defining traceability as locating a recorded element in its designated source-of-truth repository rather than measuring reviewer agreement on text-span entailment.
6. Eventum. Guardrails for Production Document Processing: A Technical Guide. eventum.ai/blog, 2026. Control 8.
7. Yan C, Ong HH, Grabowska ME, et al. Large language models facilitate the generation of electronic health record phenotyping algorithms. Journal of the American Medical Informatics Association 31(9), 1994–2001 (2024). DOI: 10.1093/jamia/ocae072.
   Research note. Performance figures are taken from Table 2 and the Results text. The true-positive and false-positive cell counts printed in Table 2 do not algebraically reproduce the reported PPV percentages, so the authors’ printed PPV values are cited directly without recalculation.
8. Ben Abacha A, Yim W, Fu Y, Sun Z, Yetisgen M, Xia F, Lin T. MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes. Findings of ACL 2025, 22539–22550 (arXiv:2412.19260).
   Research note. MEDEC comprises 3,848 clinical texts, including 488 clinical notes from three US hospital systems. Two physicians independently annotated 242 of the 925 test texts, reaching 69.01% agreement on binary error presence and 57.85% agreement on the specific sentence containing the error (Section 5.3).
9. Sivarajkumar S, Edupuganti S, Lazris D, et al. Extraction of Treatments and Responses From Non-Small Cell Lung Cancer Clinical Notes Using Natural Language Processing. JCO Clinical Cancer Informatics 10, e2500138 (2026). DOI: 10.1200/CCI-25-00138.
   Research note. Internal holdout evaluation on 64 notes yielded an F1 of 0.92 for treatment-response linking. External evaluation on 117 Mayo Clinic notes restricted the target relation to chemotherapy responses and used pre-annotated Mayo gold entity spans, yielding F1 scores from 0.49 to 0.64 across five classifiers in Table 3 (reported as 0.51 to 0.64 in the Abstract).
10. Sivarajkumar S, Ameri K, Li C (Chuqin Li), Wang Y, Jiang M. Automating Adjudication of Cardiovascular Events Using Large Language Models. AMIA Annual Symposium Proceedings 2024, 1219–1228 (arXiv:2503.17222).
    Research note. The study reported 0.96 precision and 0.71 recall for concept extraction. On generated adjudication rationales for 100 patients, a secondary LLM evaluator assigned mean rubric scores of 0.98 for logical consistency and 0.31 for timeline accuracy (Table 3 of the arXiv preprint). The evaluator model was uncalibrated against human clinical adjudicators.
11. Wong C, Zhang S, Gu Y, Moung C, Abel J, Usuyama N, et al. Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology. Proceedings of Machine Learning Research 219, 846–862 (MLHC 2023, arXiv:2308.02180).
    Research note. Evaluated against annotations by three molecular pathologists across the eligibility criteria of 53 oncology trials, three-shot GPT-4 exceeded Criteria2Query by more than 30 F1 points on histology and biomarker extraction, reaching 72.5 F1 on biomarkers (Table 1). The benchmark evaluated criteria parsing rather than end-to-end patient-to-trial matching.
12. Koskimaki J, Hu J, Zhang Y, et al. Natural language processing-optimized case selection for real-world evidence studies. Journal of Clinical Oncology 40(16_suppl), 1556 (2022). DOI: 10.1200/JCO.2022.40.16_suppl.1556.
    Research note. Across six clinical cohorts, 75.2% to 96.5% of NLP-selected cases yielded usable curated data (84.2% overall). The abstract states a baseline structured-query yield of 9.6%, whereas dividing the abstract’s reported numerator and denominator (3,878 / 41,186) yields 9.42%.
13. Dusenberry MW, Tran D, Choi E, et al. Analyzing the Role of Model Uncertainty for Electronic Health Records. ACM Conference on Health, Inference, and Learning (CHIL 2020), 204–213 (arXiv:1906.03842).
    Research note. Across an ensemble of 200 mortality-prediction models evaluated on MIMIC-III and eICU, predicted mortality probabilities for an individual patient varied by up to 57.5 percentage points.
14. US Department of Health and Human Services, Office for Civil Rights. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule. hhs.gov, last reviewed 3 February 2025, 45 CFR 164.514(b).
15. Bressler D. AI Can Write the Tests. Who Checks the Answer Key? Eventum, 19 October 2026. Compile-Time Intelligence, Part 3.