Knowledge distillation has demonstrated considerable success in scenarios involving multi-class single-label learning.However,its direct application to multi-label learning proves challenging due to complex correlatio...Knowledge distillation has demonstrated considerable success in scenarios involving multi-class single-label learning.However,its direct application to multi-label learning proves challenging due to complex correlations in multi-label structures,causing student models to overlook more finely structured semantic relations present in the teacher model.In this paper,we present a solution called multi-label prototype-aware structured contrastive distillation,comprising two modules:Prototype-aware Contrastive Representation Distillation(PCRD)and prototype-aware cross-image structure distillation.The PCRD module maximizes the mutual information of prototype-aware representation between the student and teacher,ensuring semantic representation structure consistency to improve the compactness of intra-class and dispersion of inter-class representations.In the PCSD module,we introduce sample-to-sample and sample-to-prototype structured contrastive distillation to model prototype-aware cross-image structure consistency,guiding the student model to maintain a coherent label semantic structure with the teacher across multiple instances.To enhance prototype guidance stability,we introduce batch-wise dynamic prototype correction for updating class prototypes.Experimental results on three public benchmark datasets validate the effectiveness of our proposed method,demonstrating its superiority over state-of-the-art methods.展开更多
基金supported by the National Natural Science Foundation of China(No.62466061)the Yunnan Fundamental Research Projects(No.202401AU070052)+1 种基金the Yunnan Provincial Department of Education Science Research Fund,China(Nos.2023J0209 and 2024Y161)the Natural Science Doctoral Research Start-Up Fund of Yunnan Normal University(No.2022ZB015).
文摘Knowledge distillation has demonstrated considerable success in scenarios involving multi-class single-label learning.However,its direct application to multi-label learning proves challenging due to complex correlations in multi-label structures,causing student models to overlook more finely structured semantic relations present in the teacher model.In this paper,we present a solution called multi-label prototype-aware structured contrastive distillation,comprising two modules:Prototype-aware Contrastive Representation Distillation(PCRD)and prototype-aware cross-image structure distillation.The PCRD module maximizes the mutual information of prototype-aware representation between the student and teacher,ensuring semantic representation structure consistency to improve the compactness of intra-class and dispersion of inter-class representations.In the PCSD module,we introduce sample-to-sample and sample-to-prototype structured contrastive distillation to model prototype-aware cross-image structure consistency,guiding the student model to maintain a coherent label semantic structure with the teacher across multiple instances.To enhance prototype guidance stability,we introduce batch-wise dynamic prototype correction for updating class prototypes.Experimental results on three public benchmark datasets validate the effectiveness of our proposed method,demonstrating its superiority over state-of-the-art methods.
