Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

doi:arXiv:2502.11455

	Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training
	Weng, Fenghua1 ; Lou, Jian 2; Feng, Jun 3; Huang, Minlie 4; Wang, Wenjie1
	2025-02-17
状态	已发表
摘要	Safety alignment is critical in pre-training large language models (LLMs) to generate responses aligned with human values and refuse harmful queries. Unlike LLM, the current safety alignment of VLMs is often achieved with post-hoc safety fine-tuning. However, these methods are less effective to white-box attacks. To address this, we propose $textit{Adversary-aware DPO (ADPO)}$, a novel training framework that explicitly considers adversarial. $textit{Adversary-aware DPO (ADPO)}$ integrates adversarial training into DPO to enhance the safety alignment of VLMs under worst-case adversarial perturbations. $textit{ADPO}$ introduces two key components: (1) an adversarial-trained reference model that generates human-preferred responses under worst-case perturbations, and (2) an adversarial-aware DPO loss that generates winner-loser pairs accounting for adversarial distortions. By combining these innovations, $textit{ADPO}$ ensures that VLMs remain robust and reliable even in the presence of sophisticated jailbreak attacks. Extensive experiments demonstrate that $textit{ADPO}$ outperforms baselines in the safety alignment and general utility of VLMs.
语种	英语
DOI	arXiv:2502.11455
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:121697506
WOS类目	Computer Science, Information Systems
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/514092
专题	信息科学与技术学院_硕士生信息科学与技术学院_PI研究组_王雯婕组
通讯作者	Wang, Wenjie
作者单位	1.ShanghaiTech Univ, Shanghai, Peoples R China 2.Sun Yat Sen Univ, Guangzhou, Peoples R China 3.Huazhong Univ Sci & Technol, Wuhan, Peoples R China 4.Tsinghua Univ, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Weng, Fenghua,Lou, Jian,Feng, Jun,et al. Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training. 2025.