Computerized Diagnostic Decision Support Systems-Isabel Pro Versus ChatGPT-4 Part II

Authors

Joe M Bridges
Xiaoqian Jiang
Michael Ige
Oluwatoniloba Toyobo

Language

English

Publication Date

6-1-2025

Journal

JAMIA Open

DOI

10.1093/jamiaopen/ooaf048

PMID

40524839

PMCID

PMC12169417

PubMedCentral® Posted Date

6-16-2025

PubMedCentral® Full Text Version

Post-print

Abstract

Objective: Does a Tree-of-Thought prompt and reconsideration of Isabel Pro's differential improve ChatGPT-4's accuracy; does increasing expert panel size improve ChatGPT-4's accuracy; does ChatGPT-4 produce consistent outputs in sequential requests; what is the frequency of fabricated references?

Materials and methods: Isabel Pro, a computerized diagnostic decision support system, and ChatGPT-4, a large language model. Using 201 cases from the New England Journal of Medicine, each system produced a differential diagnosis ranked by likelihood. Statistics were Mean Reciprocal Rank, Recall at Rank, Average Rank, Number of Correct Diagnoses, and Rank Improvement. For reproducibility, the study compared the initial expert panel run to each subsequent run, using the r-squared calculation from a scatter plot of each run.

Results: ChatGPT-4 improved MRR and Recall at 10 to 0.72 but produced fewer correct diagnoses and lower average rank. Reconsideration of the Isabel Pro differential produced an improvement in Recall at 10 of 11%. The expert panel size of two produced the best result. The reproducibility runs were within 4% on average for Recall at 10, but the scatterplots showed an r-squared ranging from 0.44 to 034, suggesting poor reproducibility. Reference accuracy was 34.8% for citations and 37.8% for DOIs.

Discussion: ChatGPT-4 performs well with images and electrocardiography and in administrative practice management, but diagnosis has not proven as promising.

Conclusions: As noted above, the results demonstrate concerns for diagnostic accuracy, reproducibility, and reference citation accuracy. Until these issues are resolved, clinical usage for diagnosis will be minimal, if at all.

Keywords

artificial intelligence, diagnosis, computer assisted, Isabel Pro, ChatGPT-4

Published Open-Access

yes

Recommended Citation

Bridges, Joe M; Jiang, Xiaoqian; Ige, Michael; et al., "Computerized Diagnostic Decision Support Systems-Isabel Pro Versus ChatGPT-4 Part II" (2025). Faculty, Staff and Student Publications. 482.
https://digitalcommons.library.tmc.edu/uthshis_docs/482