Faculty, Staff and Student Publications
Language
English
Publication Date
5-1-2024
Journal
Journal of Biomedical Informatics
DOI
10.1016/j.jbi.2024.104630
PMID
38548007
PMCID
PMC11065560
PubMedCentral® Posted Date
5-3-2024
PubMedCentral® Full Text Version
Author MSS
Abstract
Objective: To develop soft prompt-based learning architecture for large language models (LLMs), examine prompt-tuning using frozen/unfrozen LLMs, and assess their abilities in transfer learning and few-shot learning.
Methods: We developed a soft prompt-based learning architecture and compared 4 strategies including (1) fine-tuning without prompts; (2) hard-prompting with unfrozen LLMs; (3) soft-prompting with unfrozen LLMs; and (4) soft-prompting with frozen LLMs. We evaluated GatorTron, a clinical LLM with up to 8.9 billion parameters, and compared GatorTron with 4 existing transformer models for clinical concept and relation extraction on 2 benchmark datasets for adverse drug events and social determinants of health (SDoH). We evaluated the few-shot learning ability and generalizability for cross-institution applications.
Results and conclusion: When LLMs are unfrozen, GatorTron-3.9B with soft prompting achieves the best strict F1-scores of 0.9118 and 0.8604 for concept extraction, outperforming the traditional fine-tuning and hard prompt-based models by 0.6 ∼ 3.1 % and 1.2 ∼ 2.9 %, respectively; GatorTron-345 M with soft prompting achieves the best F1-scores of 0.8332 and 0.7488 for end-to-end relation extraction, outperforming other two models by 0.2 ∼ 2 % and 0.6 ∼ 11.7 %, respectively. When LLMs are frozen, small LLMs have a big gap to be competitive with unfrozen models; scaling LLMs up to billions of parameters makes frozen LLMs competitive with unfrozen models. Soft prompting with a frozen GatorTron-8.9B model achieved the best performance for cross-institution evaluation. We demonstrate that (1) machines can learn soft prompts better than hard prompts composed by human, (2) frozen LLMs have good few-shot learning ability and generalizability for cross-institution applications, (3) frozen LLMs reduce computing cost to 2.5 ∼ 6 % of previous methods using unfrozen LLMs, and (4) frozen LLMs require large models (e.g., over several billions of parameters) for good performance.
Keywords
Humans, Natural Language Processing, Machine Learning, Data Mining, Algorithms, Social Determinants of Health, Drug-Related Side Effects and Adverse Reactions
Published Open-Access
yes
Recommended Citation
Peng, Cheng; Yang, Xi; Smith, Kaleb E; et al., "Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction" (2024). Faculty, Staff and Student Publications. 910.
https://digitalcommons.library.tmc.edu/uthshis_docs/910