Seven Ways Clinical AI Fails, and the Control Each Requires

Chen Li | Fractional Executive Advisor, AI / ML / Robotics, Eventum

Compile-Time Intelligence

A patient asks whether to take ibuprofen for a swollen, painful calf following a long flight. An automated reply provides accurate information about the drug and includes a verified citation. The text contains no fabricated facts, yet it fails to escalate a potentially urgent clinical symptom. This demonstrates that “hallucination” constitutes an insufficient category for clinical system design. Fabrication, acuity misjudgment, omitted variables, unrecorded inference, and false citation grounding each require distinct operational checks.

Clinicians reviewing generated drafts miss specific categories of errors. A study of 20 primary-care physicians reviewing simulated portal messages found that 13 to 15 reviewers left each of four erroneous drafts insufficiently corrected. Most participants still reported lower cognitive workload and rated the drafts as safe to use [2].

A more useful approach than measuring overall hallucination rates is to classify failure modes. Evaluation must identify the specific mechanism of failure, the factors that cause reviewers to miss the error, and the detection methods required before the output reaches the user interface.

The following seven patterns form an overlapping taxonomy of failures. The associated controls operate as engineering proposals requiring empirical evaluation in the target workflow. They provide no standalone guarantees of clinical safety.

Failure Modes and Operational Controls

1. Fabricated specifics. An invented trial, dose, laboratory value, or guideline statement visually mimics a documented record. Fluency provides the reviewer with no visual indicator of the missing support.

The corresponding control is the claim record. Every material value must link to a specific supporting passage. The system must output an “unsupported” state when evidence is absent. Resolving the pointer serves as a preliminary step, while human review of a sample establishes whether the referenced passage actually supports the generated value. The core design requirement renders unsupported information visible in the interface. Prompting the model to avoid invention provides insufficient protection.

2. Misjudged acuity. A generated response can describe a symptom accurately while recommending an inappropriate timeframe for follow-up. Reviewers verifying factual correctness frequently accept the content without independently reassessing clinical urgency.

Acuity requires evaluation as a process distinct from response generation. Clinician-defined symptom combinations operate as tested rules that route qualifying cases to a designated queue before the interface displays a routine draft. A separately evaluated model-based triage check complements these rules. System evaluation measures the urgent cases successfully routed, the missed urgent cases, and the operational burden of unnecessary escalations.

3. Unsupported inference. A conclusion such as “the patient is adherent to therapy” does not follow directly from the record “the prescription was filled.” Plausible inferences frequently escape human review.

The data structure for a claim must label the information as either stated in the source or inferred. An inferred value must retain the source evidence alongside the formal reasoning step that produced it. The system then applies separate rendering policies to documented and inferred information. Evaluation protocols must sample values labeled “stated.” Without this check, a model that labels its own inferences as stated facts would go unnoticed.

4. Stale or wrong-context knowledge. A recommendation can accurately reproduce a superseded guideline or apply a national rule in conflict with a site-specific protocol. The reviewer frequently lacks awareness of the protocol update.

Source documents require explicit versioning and dating. The applicable version metadata must travel with the output. System architecture places local overrides in an explicit configuration layer instead of a generation prompt. The builder enforces a chosen document version programmatically. The deploying organization determines the applicable version and local policies.

5. Anchoring and sycophancy. A leading question or asserted diagnosis shifts the generated distribution toward agreement. When the output confirms the reviewer’s expectation, the review process lacks a trigger for reconsideration.

The control tests this distribution shift directly. The evaluation compares performance on neutral and biased versions of the same task, treating excessive deterioration as a release-blocking condition. BiasMedQA, from Samuel Schmidgall and colleagues, implements this evaluation by adding a sentence that introduces a cognitive bias to each medical exam question [3]. Performance on neutral prompts alone constitutes incomplete evidence. The evaluation target is invariance to unreliable framing.

6. Omission. A summary may omit an abnormal result, an allergy, or a pending test. The reviewer cannot inspect a missing variable without independently reconstructing the source document.

A document-specific completeness check must account for each required element. The system classifies each element as included, absent from the source, or present in the source but omitted from the output. A documented negative finding requires a distinct representation from an unmentioned variable. The organization defines the required document contents. The system evaluates those requirements and renders omissions visible in the user interface.

7. False grounding. The system appends a valid citation to a statement the source text does not establish. The presence of the citation increases reviewer confidence during a failure mode requiring high scrutiny.

The architecture treats citation support as an independent claim. System checks must separate source existence from passage-level support and patient applicability. Passing the existence check provides no verification of passage support or applicability. An automated support checker executes this comparison continuously. Evaluation requires measuring the checker’s outputs against human judgments on the relevant clinical material.

Interacting Failures and Passing Checks

Consider the synthetic portal message from the introduction: new pain and swelling in one calf after a long flight, and a question about ibuprofen.

The generated draft answers the medication question accurately, reassures the patient about the swelling, and cites a source that supports the general dosing statement. The primary failure mode consists of misjudged acuity and an omitted urgency assessment.

In the proposed workflow, the configured red-flag combination routes the message to an urgent clinical review queue with a specified recipient and response timeline. The completeness check marks the required urgency assessment as missing from the draft. The citation-support check passes for the dosing statement, while the interface delineates the limited scope of the supported text.

The interface presents the clinician with two identified failures and one narrowly scoped verified check. This is the point of separating the failure patterns. A system checking solely for factual support fails to detect missing critical assessments.

Evaluating Human Review

This review failure aligns with broader findings. A radiation-oncology study asked six physicians to assess generated drafts for 100 simulated patient messages. The physicians judged that unedited drafts could cause severe harm in 11 of 156 assessments and death in one [1]. These figures represent judgments of potential harm in simulated cases. Errors in assessing or communicating acuity drove the majority of the risk classifications.

Automation-bias research predates language models [5]. A systematic review pooling four healthcare studies measured a 26% relative increase in incorrect decisions (a risk ratio of 1.26) when decision support was incorrect [9]. Studies of mammography and diagnostic vignettes found that incorrect or systematically biased suggestions impaired clinician performance, including among experienced readers [10] [11]. In the vignette study, image-based explanations shown with the biased suggestions did not significantly reduce the drop [11].

A recent randomized trial extends these findings to language models. A preprint report of 44 physicians with prior training in artificial intelligence literacy recorded lower diagnostic-reasoning performance when the supplied advice included deliberate errors [12].

Relying on clinician review constitutes an intended workflow parameter. The system evaluation must establish the specific error categories reviewers catch, the categories they miss, and the effect of interface design on detection rates. Evaluation protocols must measure alert burden. A review of computerized drug-safety alerts recorded frequent overrides [4]. The authors note that overriding may often be justified. More warnings do not automatically add protection.

Evaluating Citation Grounding

Citations function as both verification mechanisms and vectors for automation bias. A 2026 preprint recorded improved overall physician accuracy using a citation-grounded dermatology assistant. The study also recorded physicians abandoning initially correct answers when they evaluated a source as supporting the generated incorrect alternative [6]. The authors classify this relationship as observational.

Interface design must explicitly map the claim-to-passage relationship to facilitate verification. The ALCE citation benchmark, from Tianyu Gao and colleagues, deploys a natural-language-inference model to assess support. The automated metric demonstrated incomplete agreement with human judgments on general question-answering datasets and failed to detect partial support [7]. Clinical text requires domain-specific validation of these automated metrics.

SelfCite, from Yung-Sung Chuang and colleagues, measures whether cited text is consequential to the generated statement by calculating changes in probability when the source text is removed [8]. This metric quantifies context attribution. It provides no verification regarding clinical support or patient applicability. The evaluation framework must track context attribution separately from clinical support. A source can determine the output probabilities without justifying the generated clinical recommendation.

Measuring Operational Controls

The table below proposes a division of engineering and operational responsibilities. The organization’s designated professional retains responsibility for the clinical decision.

These measurements divide into execution checks and accuracy checks. Execution metrics verify that a span exists, a rule is recorded, or a field is populated. Accuracy metrics quantify correct classifications. System utility depends strictly on the accuracy metrics.

The interfaces between systems and human operators require specified owners. The deploying organization must map the versioning architecture to local guideline policy. Clinical staff must update the red-flag list. The review queue requires a designated recipient responsible for action.

The FDA’s January 2026 clinical decision-support guidance describes the information relevant to independent clinician review, including inputs, known limitations, validation, and dated or versioned sources. The document notes that time-critical decisions constrain meaningful independent review. The guidance concerns regulatory status and intended use [13]. It confers no safety certification on the controls proposed here.

Continuous Evaluation of Failure Patterns

Aggregate accuracy scores obscure the performance of individual controls. A systematic review of 519 healthcare language-model studies, by Suhana Bedi and colleagues, including Akash Chaurasia, found accuracy dominated evaluation frameworks, with rare assessments of calibration or uncertainty [14].

Testing each failure pattern requires labeled examples and difficult negatives. The reference labels require an inter-rater agreement assessment. MedHELM, from Suhana Bedi and colleagues, demonstrates the baseline variance in evaluation standards, recording imperfect agreement between individual clinicians and between clinicians and automated judges [15].

The test sets evaluate specific behaviors independently. Engineers evaluate negation, temporality, urgency, entity attribution, and prompt sensitivity as distinct variables. The CheckList framework for language-system testing, from Marco Tulio Ribeiro and colleagues, provides a structural precedent for this capability-specific evaluation [17].

Maintenance operations require a fixed regression set across model releases with inspection of individual output changes. A clinical red-teaming study recorded 81 of 376 prompts receiving an appropriate response from the oldest model and an inappropriate response from at least one newer version [16]. System updates trigger renewed testing of all required behavioral constraints.

Evaluation protocols must sample the accepted output stream. The audit set includes drafts passing automated checks and receiving clinician approval. Isolating review to rejected or escalated cases obscures the failures that bypass both layers. Eventum’s document-processing guide recommends the same for production pipelines: sample the accepted outputs too [18].

Specifying Controls

Measuring aggregate hallucination rates provides preliminary data. A reliable workflow requires formal specification. System design specifies the applicable failure modes, the detection mechanism for each, the personnel maintaining the control, and the documented miss rate of the mechanism.

These parameters differ across portal replies, discharge summaries, and prior-authorization decisions. Task variance requires specific evaluation protocols for each application. Interface design exposes the supporting evidence and the unverified variables within the available timeframe for clinical action. Identifying the specific failure mode initiates the engineering of the corresponding control.

Eventum translates clinical AI risks into targeted evaluation sets, evidence checks, and review workflows. Get in touch to discuss the controls a clinical document or decision-support application needs before deployment.

References and research notes

1. Chen S, Guevara M, Moningi S, et al. The effect of using a large language model to respond to patient messages. Lancet Digital Health 6(6), e379-e381 (2024). DOI: 10.1016/S2589-7500(24)00060-8. A proof-of-concept observational study with simulated messages, conducted in 2023.
   Research note. Six radiation oncologists each assessed 26 scenario-message pairs, yielding 156 assessments over 100 simulated patient messages. Potential severe harm was identified in 11 assessments (7.1%) and potential death in one (0.6%). These are physician judgments of potential harm, not observed outcomes or percentages of unique patients. Edited replies were closer in clinical content to the AI drafts than to independently written replies.
2. Biro JM, Handley JL, Malcolm McCurry J, et al. Opportunities and risks of artificial intelligence in patient portal messaging in primary care. npj Digital Medicine 8, 222 (2025). DOI: 10.1038/s41746-025-01586-2. Per-draft counts from Table 2; the paper’s own average of 2.67 missed drafts per physician does not reproduce from those counts, so the counts are used.
   Research note. Twenty primary-care physicians reviewed 18 simulated portal messages, including four drafts with errors identified by a physician on the research team. Each erroneous draft remained insufficiently corrected by 13 to 15 physicians. Sixteen reported reduced cognitive workload; 15 believed the drafts were safe. The paper’s summary of 2.67 missed drafts per physician does not reproduce from Table 2; the article uses per-draft counts.
3. Schmidgall S, Harris C, Essien I, et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digital Medicine 7, 295 (2024). DOI: 10.1038/s41746-024-01283-6. The unbiased baseline and the drops are from the Results.
   Research note. BiasMedQA comprises 1,273 licensing-exam questions with bias-inducing rewrites. GPT-4’s neutral baseline was 72.7%; its largest decline, under false-consensus bias, is reported as 14.0% in the source.
4. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association 13(2), 138-147 (2006). DOI: 10.1197/jamia.M1809. A review of 17 papers.
   Research note. This review covered 17 papers and reported drug-safety-alert override rates of 49% to 96%, while noting that overrides could often be justified. It concerns computerized physician order entry, not a measured override rate for language-model triage.
5. Parasuraman R, Manzey DH. Complacency and bias in human use of automation: an attentional integration. Human Factors 52(3), 381-410 (2010). DOI: 10.1177/0018720810376055.
6. Chanda T, Wies C, Schramm F, et al. Large language models improve physician accuracy but lead to false reliance. arXiv:2608.00817, version 1, 1 August 2026. Preprint. Figures from the Results (pages 10 to 12) and Limitations (page 17).
   Research note. The preprint involved 46 physicians answering 16 dermatology questions each, or 736 decisions. Accuracy rose from 70.8% to 82.6% with the assistant. Among 48 decisions where the physician was initially correct and the assistant wrong, physicians retained the correct answer in 23 of 25 cases when they rated no citation as supportive and 8 of 23 when they rated at least one as supportive. The association is observational and does not establish that citations caused those reversals.
7. Gao T, Yen H, Yu J, Chen D. Enabling Large Language Models to Generate Text with Citations. EMNLP 2023, 6465-6488; arXiv:2305.14627. DOI: 10.18653/v1/2023.emnlp-main.398. Agreement figures from Section 6 and Appendix G.5, on the ASQA and ELI5 datasets.
   Research note. On 100 sampled examples from ASQA and ELI5, the NLI-based measure agreed with human judgments 85.1% of the time for citation recall and 77.6% for precision. It could not detect partial support. These are agreement measurements on general question-answering data, not clinical validity estimates.
8. Chuang Y, Cohen-Wang B, Shen SZ, Wu Z, Xu H, Lin XV, Glass J, Li S, Yih W. SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models. ICML 2025; arXiv:2502.09604.
   Research note. Fine-tuning plus candidate-citation selection improved citation F1 by up to 5.3 points on the evaluated long-form benchmark. The method measures how cited context affects generation probabilities; attribution and passage-level clinical support are not interchangeable.
9. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association 19(1), 121-127 (2012). DOI: 10.1136/amiajnl-2011-000089. The 26% is a risk ratio of 1.26 pooled from four healthcare studies.
   Research note. The systematic review included 74 studies across healthcare and other fields. The 26% increase was relative, derived from a risk ratio of 1.26 pooled across four healthcare studies. It is not a 26-percentage-point increase and is not a single estimate specific to present-day LLM workflows.
10. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology 307(4), e222176 (2023). DOI: 10.1148/radiol.222176. The outcome is correct BI-RADS category assignment.
    Research note. Twenty-seven radiologists assessed 40 test mammograms, including 12 with deliberately incorrect AI suggestions. Correct category assignment with correct versus incorrect suggestions was 79.7% versus 19.8% among the least experienced readers, and 82.3% versus 45.5% among the most experienced. These are category-assignment results in the study task, not patient-outcome estimates.
11. Jabbour S, Fouhey D, Shepard S, et al. Measuring the Impact of AI in the Diagnosis of Hospitalized Patients: A Randomized Clinical Vignette Survey Study. JAMA 330(23), 2275-2284 (2023). DOI: 10.1001/jama.2023.22295.
    Research note. The vignette study involved 457 clinicians. A systematically biased model lowered diagnostic accuracy by 11.3 percentage points; image-based explanations did not mitigate that harmful effect in the evaluated setup.
12. Qazi IA, Ali A, Khawaja AU, Akhtar MJ, Sheikh AZ, Alizai MH. Automation Bias in Large Language Model-Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy: A Randomized Clinical Trial. NEJM AI, 23 April 2026, DOI: 10.1056/AIoa2501001. The figures in the research note are from the preprint abstract (medRxiv, DOI: 10.1101/2025.08.23.25334280; versions 1 and 2, August and September 2025, agree).
    Research note. The preprint involved 44 physicians, 22 per arm, each with 20 hours of AI-literacy training. Mean diagnostic-reasoning scores were 84.9% with error-free advice and 73.3% with deliberate errors in three of six vignettes; the adjusted difference was 14.0 percentage points. These figures are from the preprint report, not the subsequent NEJM AI version.
13. US Food and Drug Administration. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. Issued 29 January 2026, superseding the version of 6 January 2026. fda.gov/media/109618/download. Criterion 4, pages 14 and 15.
14. Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333(4), 319-328 (2025). DOI: 10.1001/jama.2024.21700. Studies published January 2022 to February 2024.
    Research note. The review covered 519 studies published through February 2024. It reported that approximately 5% used real patient-care data, 95.4% assessed accuracy as the primary dimension, and 1.2% measured calibration or uncertainty. These figures characterize that historical review sample, not the proportion of all clinical AI research today.
15. Bedi S, Cui H, Fuentes M, et al. Holistic evaluation of large language models for medical tasks with MedHELM. Nature Medicine 32(3), 943-951 (2026). DOI: 10.1038/s41591-025-04151-2. Journal version; the preprint’s figures differ.
    Research note. MedHELM organized 121 tasks with 29 clinicians and evaluated 37 benchmarks, including 13 new ones, 12 based on EHRs. On 56 outputs from two benchmarks rated by 20 clinicians, model-jury agreement with clinicians had intraclass correlation 0.47 versus inter-clinician agreement 0.43, with overlapping confidence intervals. The comparison does not establish that model judges are clinically superior.
16. Chang CT, Farah H, Gui H, et al. Red teaming ChatGPT in medicine to yield real-world insights on model behavior. npj Digital Medicine 8, 149 (2025). DOI: 10.1038/s41746-025-01542-0. Counts from the Results and Tables 1 and 3.
    Research note. The red-teaming exercise involved 80 participants, 376 prompts, and 1,504 responses across four model versions. Of those responses, 20.1% were inappropriate; a model released afterward produced inappropriate responses to 20.4% of prompts. Separately, 81 of 376 prompts (21.5%) were answered appropriately by the oldest model but inappropriately by at least one newer model. The response-level and prompt-level rates have different denominators.
17. Ribeiro MT, Wu T, Guestrin C, Singh S. Beyond Accuracy: Behavioral Testing of NLP models with CheckList. ACL 2020; arXiv:2005.04118.
    Research note. In CheckList’s user study of 18 practitioners, participants using the method created twice as many tests and found almost three times as many bugs. This motivates capability-specific testing; it is not a clinical validation result.
18. Eventum. Guardrails for Production Document Processing: A Technical Guide. eventum.ai/blog, 2026. Control 14.