Abstract

Model calibration seeks to ensure that models produce confidence scores thataccurately reflect the true likelihood of their predictions being correct.However, existing calibration approaches are fundamentally tied to datasets ofone-hot labels implicitly assuming full certainty in all the annotations. Suchdatasets are effective for classification but provides insufficient knowledgeof uncertainty for model calibration, necessitating the curation of datasetswith numerically rich ground-truth confidence values. However, due to thescarcity of uncertain visual examples, such samples are not easily available asreal datasets. In this paper, we introduce calibration-aware data augmentationto create synthetic datasets of diverse samples and their ground-truthuncertainty. Specifically, we present Calibration-aware Semantic Mixing (CSM),a novel framework that generates training samples with mixed classcharacteristics and annotates them with distinct confidence scores viadiffusion models. Based on this framework, we propose calibrated reannotationto tackle the misalignment between the annotated confidence score and themixing ratio during the diffusion reverse process. Besides, we explore the lossfunctions that better fit the new data representation paradigm. Experimentalresults demonstrate that CSM achieves superior calibration compared to thestate-of-the-art calibration approaches. Code is available atgithub.com/E-Galois/CSM.