Faculty, Staff and Student Publications
Language
English
Publication Date
7-1-2025
Journal
Proceedings of Machine Learning Research
PMID
41868744
PMCID
PMC13004626
PubMedCentral® Posted Date
3-21-2026
PubMedCentral® Full Text Version
Post-print
Abstract
One of the key technologies for the success of Large Language Models (LLMs) is preference alignment. However, a notable side effect of preference alignment is poor calibration: while the pre-trained models are typically well-calibrated, LLMs tend to become poorly calibrated after alignment with human preferences. In this paper, we investigate why preference alignment affects calibration and how to address this issue. For the first question, we observe that the preference collapse issue in alignment undesirably generalizes to the calibration scenario, causing LLMs to exhibit overconfidence and poor calibration. To address this, we demonstrate the importance of fine-tuning with domain-specific knowledge to alleviate the overconfidence issue. To further analyze whether this affects the model's performance, we categorize models into two regimes: calibratable and non-calibratable, defined by bounds of Expected Calibration Error (ECE). In the calibratable regime, we propose a calibration-aware fine-tuning approach to achieve proper calibration without compromising LLMs' performance. However, as models are further fine-tuned for better performance, they enter the non-calibratable regime. For this case, we develop an EM-algorithm-based ECE regularization for the fine-tuning loss to maintain low calibration error. Extensive experiments validate the effectiveness of the proposed methods.
Published Open-Access
yes
Recommended Citation
Xiao, Jiancong; Hou, Bojian; Wang, Zhanliang; et al., "Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach" (2025). Faculty, Staff and Student Publications. 941.
https://digitalcommons.library.tmc.edu/uthshis_docs/941