Language
English
Publication Date
11-1-2026
Journal
EBioMedicine
DOI
10.1016/j.ebiom.2026.106516
PMID
42892212
PMCID
PMC13695126
PubMedCentral® Posted Date
10-17-2026
PubMedCentral® Full Text Version
Post-print
Abstract
Background: Adverse drug events (ADEs) are a critical indicator of patient safety but are often documented only in free-text clinical notes. The potential of recent advances in natural language processing (NLP), particularly generative large language models (LLMs), to identify ADEs remains understudied. This study aimed to compare the performance of multiple LLMs in identifying ADE-Drug relationships in inpatient and ambulatory clinical notes.
Methods: We used clinical notes from the 2018 National NLP Clinical Challenge (n2c2) ADE dataset (inpatient; n = 505) and from outpatient encounters (n = 2555) between October 1, 2018, and December 31, 2019, at a large academic medical center based in New England. Notes were pre-processed into snippets for model input. Evaluated Models included: GPT-4o, GPT-4o-mini, Llama 3.2-3B and their instruction fine-tuned variants (including low-rank adapters for Llama). Performance was assessed using both strict and relaxed evaluations (precision, recall, and F1) for all models, followed by manual evaluation (exact semantic match, partial match, missing ADE, drug mention only, not a drug, or wrong) of the two best-performing models.
Findings: Of the 1048 notes in the local test split, 450 (42.9%) contained at least one ADE. The n2c2 test set included 202 notes, of which 155 (76.7%) contained at least one ADE. GPT-4o and GPT-4o-mini were the top-performing models among those evaluated. GPT-4o consistently outperformed GPT-4o-mini in ADE extraction across both datasets, with higher F1-scores (0.524 vs. 0.381) and a more balanced precision-recall profile. Both models captured ADEs effectively in explicit and complex clinical contexts, although limitations included misclassification of pre-existing allergies and occasional conflation of therapeutic indications with adverse effects. GPT-4o achieved higher exact match coverage and fewer errors across clinical notes, indicating more reliable performance in both inpatient and ambulatory settings.
Interpretation: This work establishes a foundation for integrating LLM methods into real-world drug safety surveillance, with direct implications for improving patient safety.
Keywords
Large Language Models, Drug-Related Side Effects and Adverse Reactions, Humans, Cross-Sectional Studies, Retrospective Studies, Electronic Health Records, Adverse drug events, Pharmacovigilance, Deep learning, Patient safety, Large language models, Medication errors
Published Open-Access
yes
Recommended Citation
Plasek, Joseph M; Li, Yiming; Amato, Mary G; et al., "Relationship Extraction for Adverse Drug Events in Clinical Notes Using Large Language Models: A Retrospective Cross-Sectional Study" (2026). Faculty, Staff and Students Publications. 8771.
https://digitalcommons.library.tmc.edu/baylor_docs/8771