摘要
机器学习模型关于自身预测的不确定性估计在实际应用中十分重要,但当前的深度模型常面临输出置信度校准性能不佳的问题.为解决这一局限,研究人员提出了训练时刻校准(train-time calibration)和后校准(post-hoc calibration)两类方法,分别在训练过程中、训练结束后对模型进行校准操作.不同于以往研究,本文旨在探究深度神经网络的置信度可校准性,即模型在训练结束后进一步利用后校准方法提升其不确定性估计的能力.基于这一新视角,我们首先发现尽管现有训练时刻校准方法能在一定程度上提升校准性能,但这类方法往往会损害模型的置信度可校准性,进而难以利用后校准方法(如温度缩放)提升模型不确定性估计表现.其次,我们针对参数衰减正则化这一被普遍采用的深度模型训练策略进行了系统性实验分析,并从模型隐含层特征可校准性的视角展开分析,发现上述方法在训练过程中对特征的“过度压缩”是损害置信度可校准性的原因.最后,受上述实验现象的启发,本文提出一种新型训练策略,通过对深度模型进行逐层剥离式训练,能够有效避免过度压缩现象对于置信度可校准性的损害,实验结果表明该方法能够在保证泛化性能的同时有效提升深度模型的置信度可校准性.
Uncertainty estimation is crucial in practical applications of modern models,but current deep neural networks often face the issue of poor calibration performance.To address this limitation,researchers have proposed two types of methods:train-time calibration and post-hoc calibration,which calibrate the model during the training process and after training,respectively.Unlike previous studies,this paper aims to explore the confidence calibratability of deep neural networks,that is,the ability of the model to further improve its uncertainty estimation using post-hoc calibration methods after training.Based on this new perspective,we first find that although existing train-time calibration methods can improve calibration performance to some extent,these methods often damage the model’s calibratability,making it difficult to use post-hoc calibration methods(such as temperature scaling)to enhance the model’s uncertainty estimation performance.Secondly,we conducted a systematic experimental analysis of the widely adopted deep model training strategy of parameter decay regularization,and studied from the perspective of the calibratability of the model’s hidden layer features,finding that the over-compression of features during the training process by the aforementioned methods is the reason for damaging the calibratability.Finally,inspired by our experimental phenomena,this paper proposes a new training strategy that effectively prevents the damage to confidence calibratability caused by over-compression through progressively layer-peeled(PLP)training of deep models.The experimental results show that this method can effectively improve the confidence calibratability of deep models while ensuring generalization performance.
作者
王登豹
张敏灵
Deng-Bao WANG;&Min-Ling ZHANG(School of Computer Science and Engineering,Southeast University,Nanjing 210096,China;Key Laboratory of Computer Network and Information Integration(Southeast University),Ministry of Education,Nanjing 210096,China)
出处
《中国科学:信息科学》
北大核心
2025年第9期2289-2303,共15页
Scientia Sinica(Informationis)
基金
国家自然科学基金(批准号:62225602,623B2023)资助项目。
关键词
深度神经网络
不确定性估计
置信度校准
deep neural networks
uncertainty estimation
confidence calibration
