Faculty, Staff and Student Publications
Publication Date
2-1-2025
Journal
JAMIA Open
Abstract
OBJECTIVE: Human Papillomavirus (HPV) vaccine is an effective measure to prevent and control the diseases caused by HPV. However, widespread misinformation and vaccine hesitancy remain significant barriers to its uptake. This study focuses on the development of VaxBot-HPV, a chatbot aimed at improving health literacy and promoting vaccination uptake by providing information and answering questions about the HPV vaccine.
METHODS: We constructed the knowledge base (KB) for VaxBot-HPV, which consists of 451 documents from biomedical literature and web sources on the HPV vaccine. We extracted 202 question-answer pairs from the KB and 39 questions generated by GPT-4 for training and testing purposes. To comprehensively understand the capabilities and potential of GPT-based chatbots, 3 models were involved in this study: GPT-3.5, VaxBot-HPV, and GPT-4. The evaluation criteria included answer relevancy and faithfulness.
RESULTS: VaxBot-HPV demonstrated superior performance in answer relevancy and faithfulness compared to baselines. For test questions in KB, it achieved an answer relevancy score of 0.85 and a faithfulness score of 0.97. Similarly, it attained scores of 0.85 for answer relevancy and 0.96 for faithfulness on GPT-generated questions.
DISCUSSION: VaxBot-HPV demonstrates the effectiveness of fine-tuned large language models in healthcare, outperforming generic GPT models in accuracy and relevance. Fine-tuning mitigates hallucinations and misinformation, ensuring reliable information on HPV vaccination while allowing dynamic and tailored responses. The specific fine-tuning, which includes context in addition to question-answer pairs, enables VaxBot-HPV to provide explanations and reasoning behind its answers, enhancing transparency and user trust.
CONCLUSIONS: This study underscores the importance of leveraging large language models and fine-tuning techniques in the development of chatbots for healthcare applications, with implications for improving medical education and public health communication.
Keywords
HPV vaccine, GPT, large language model, Chatbot, medical education
DOI
10.1093/jamiaopen/ooaf005
PMID
39975811
PMCID
PMC11837857
PubMedCentral® Posted Date
2-19-2025
PubMedCentral® Full Text Version
Post-print
Published Open-Access
yes
Included in
Artificial Intelligence and Robotics Commons, Health Information Technology Commons, Health Services Research Commons, Influenza Humans Commons
