摘要
视觉语言模型作为计算机视觉与自然语言处理交叉领域的关键技术,已在教育和艺术创作等场景中展现出广泛的应用价值。然而,此类模型易受对抗攻击影响,其安全性面临严峻挑战。传统对抗微调方法通过优化视觉编码器提升模型鲁棒性,但未充分探索与文本模态协同优化机制。为此,提出了一种面向图文对比预训练模型的多模态对抗微调方法,首次实现图像与文本编码器的同步优化,并通过学习率动态调节策略解决双编码器收敛速度不平衡的问题。实验结果表明,这种方法显著提升了视觉语言模型的对抗鲁棒性,其性能优于现有方法且实现更为高效。
Vision-Language Models(VLMs),serving as a pivotal technology in the cross-domain integration of computer vision and natural language processing,have demonstrated extensive application value in many fields,including education and artistic creation.Nevertheless,VLMs exhibit vulnerability to adversarial attacks,presenting critical security risks.While conventional adversarial fine-tuning methods can enhance robustness by optimizing visual encoders,their failure to explore synergistic optimization mechanisms with textual modalities also can limit effectiveness.To address this limitation,this study proposes a multimodal adversarial fine-tuning method for Contrastive Language-Image Pretraining(CLIP),aiming at achieving concurrent optimization of image and text encoders,alongside a dynamic learning rate adaptation strategy to mitigate convergence speed disparities between dual encoders.Experimental evaluations on benchmark datasets demonstrate that the proposed method outperforms state-of-the-art methods in adversarial robustness while maintaining superior training efficiency.
作者
陈驰
赵世纪
韦星星
CHEN Chi;ZHAO Shiji;WEI Xingxing(School of Software,Beihang University,Beijing 100191,China;Institute of Artificial Intelligence,Beihang University,Beijing 100191,China)
出处
《智能安全》
2025年第2期1-10,共10页
Artificial Intelligence Security
关键词
视觉语言模型
对抗微调
对抗鲁棒性
计算机视觉
自然语言处理
VLMs
adversarial fine-tuning
adversarial robustness
computer vision
natural language processing
