In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become promine...In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.展开更多
这是一则来自拉丁美洲危地马拉的消息。一对美貌孪生女子在重兵 把守的监狱“救”出了数十名罪犯,令人发指!当笔者正在纳闷:是何故让两个美女冒死“劫狱”?本文的第三段便泄露了“天机”:to free their kidnapper boyfriends who were s...这是一则来自拉丁美洲危地马拉的消息。一对美貌孪生女子在重兵 把守的监狱“救”出了数十名罪犯,令人发指!当笔者正在纳闷:是何故让两个美女冒死“劫狱”?本文的第三段便泄露了“天机”:to free their kidnapper boyfriends who were serving life sentences there.展开更多
大模型红队测试(Large Model Red Teaming)旨在让大语言模型(Large Language Model, LLM)接收对抗测试,从而诱使模型输出有害的测试用例,进而发现模型中的漏洞并提高其鲁棒性。大模型红队测试是大模型领域的前沿课题,近年来受到学术界...大模型红队测试(Large Model Red Teaming)旨在让大语言模型(Large Language Model, LLM)接收对抗测试,从而诱使模型输出有害的测试用例,进而发现模型中的漏洞并提高其鲁棒性。大模型红队测试是大模型领域的前沿课题,近年来受到学术界和工业界的广泛关注。研究者们针对大模型红队测试提出了众多解决方案,并在模型对齐上取得了一定进展。然而,受限于大模型红队数据的短缺和评价标准的模糊,现有研究大多局限于针对特定的场景进行评估。文中首先从与大模型安全相关的定义出发,对其所涉及的各种风险进行阐述;其次,针对大模型红队测试的重要性及其主要类别进行了阐述,综述和分析了相关红队技术的发展历程,并介绍了已有的数据集和评价指标;最后,对大模型红队测试的未来发展趋势进行了展望和总结。展开更多
The release of Apple’s iPhone was one of the most intensively publicized product releases in the history of mobile devices. While the iPhone wowed users with its exciting design and features, it also angered many for...The release of Apple’s iPhone was one of the most intensively publicized product releases in the history of mobile devices. While the iPhone wowed users with its exciting design and features, it also angered many for not allowing installation of third party applications and for working exclusively with AT & T wireless services (in the US). Besides the US, iPhone was only sold only in a few other selected countries. Software attacks were developed to overcome both limitations. The development of those attacks and further evaluation revealed several vulnerabilities in iPhone security. In this paper, we examine some of the attacks developed for the iPhone as a way of investigating the iPhone’s security structure. We also analyze the security holes that have been discovered and make suggestions for improving iPhone security.展开更多
近年来,大语言模型(large language model,LLM)在一系列下游任务中得到了广泛应用,并在多个领域表现出了卓越的文本理解、生成与推理能力.然而,越狱攻击正成为大语言模型的新兴威胁.越狱攻击能够绕过大语言模型的安全机制,削弱价值观对...近年来,大语言模型(large language model,LLM)在一系列下游任务中得到了广泛应用,并在多个领域表现出了卓越的文本理解、生成与推理能力.然而,越狱攻击正成为大语言模型的新兴威胁.越狱攻击能够绕过大语言模型的安全机制,削弱价值观对齐的影响,诱使经过对齐的大语言模型产生有害输出.越狱攻击带来的滥用、劫持、泄露等问题已对基于大语言模型的对话系统与应用程序造成了严重威胁.对近年的越狱攻击研究进行了系统梳理,并基于攻击原理将其分为基于人工设计的攻击、基于模型生成的攻击与基于对抗性优化的攻击3类.详细总结了相关研究的基本原理、实施方法与研究结论,全面回顾了大语言模型越狱攻击的发展历程,为后续的研究提供了有效参考.对现有的安全措施进行了简略回顾,从内部防御与外部防御2个角度介绍了能够缓解越狱攻击并提高大语言模型生成内容安全性的相关技术,并对不同方法的利弊进行了罗列与比较.在上述工作的基础上,对大语言模型越狱攻击领域的现存问题与前沿方向进行探讨,并结合多模态、模型编辑、多智能体等方向进行研究展望.展开更多
大语言模型(large language models,LLMs)虽然取得了显著的成功,但在实际应用中依然面临着安全问题,容易在恶意诱导下生成有毒、有害内容.目前缓解LLMs不安全行为的方法通常需要高昂的数据收集成本以及大量的计算资源.大模型知识编辑可...大语言模型(large language models,LLMs)虽然取得了显著的成功,但在实际应用中依然面临着安全问题,容易在恶意诱导下生成有毒、有害内容.目前缓解LLMs不安全行为的方法通常需要高昂的数据收集成本以及大量的计算资源.大模型知识编辑可以在不重新训练模型的基础上,根据特定的输入精确地改变模型对应的输出,在节约大量资源的条件下约束模型的行为;为优化大模型生成安全内容提供了一个新的可行思路.然而,目前学术界缺乏较为系统和全面的基于知识编辑的大模型内容安全生成分析数据集.具体地说,当前缓解LLMs不安全行为的数据集并未包括所有的不安全场景,且其有毒问题几乎无法绕过对齐后的LLMs安全防线,因此无法缓解对齐后LLMs存在的不安全问题.针对上述问题,设计了新的数据集SafeGen,并提出新的评价体系分析知识编辑在优化LLMs生成安全内容的潜力.大量的实验发现知识编辑可以提高LLMs内部的安全信念,在校正LLMs不安全行为领域展现了广阔的应用前景.但经过知识编辑的LLMs生成文本的流畅性却差强人意,这也表明了这项任务的潜在难度.该工作可以为大模型安全社区提供一些见解.展开更多
This review paper explores advanced methods to prompt Large LanguageModels(LLMs)into generating objectionable or unintended behaviors through adversarial prompt injection attacks.We examine a series of novel projects ...This review paper explores advanced methods to prompt Large LanguageModels(LLMs)into generating objectionable or unintended behaviors through adversarial prompt injection attacks.We examine a series of novel projects like HOUYI,Robustly Aligned LLM(RA-LLM),StruQ,and Virtual Prompt Injection that compel LLMs to produce affirmative responses to harmful queries.Several new benchmarks,such as PromptBench,AdvBench,AttackEval,INJECAGENT,and Robustness Suite,have been created to evaluate the performance and resilience of LLMs against these adversarial attacks.Results show significant success rates in misleading models like Vicuna-7B,LLaMA-2-7B-Chat,GPT-3.5,and GPT-4.The review highlights limitations in existing defense mechanisms and proposes future directions for enhancing LLM alignment and safety protocols,including the concept of LLM SELF DEFENSE.Our study emphasizes the need for improved robustness in LLMs,which will potentially shape the future of Artificial Intelligence(AI)driven applications and security protocols.Understanding the vulnerabilities of LLMs is crucial for developing effective defenses against adversarial prompt injection attacks.This paper proposes a systemic classification framework that discusses various types of prompt injection attacks and defenses.We also go through a broad spectrum of stateof-the-art attack methods(such as HouYi and Virtual Prompt Injection)alongside advanced defense mechanisms(like RA-LLM,StruQ,and LLM Self-Defense),providing critical insights into vulnerabilities and robustness.We also integrate and compare results from multiple recent benchmarks,including PromptBench,INJECENT,and BIPIA.展开更多
文摘In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.
文摘这是一则来自拉丁美洲危地马拉的消息。一对美貌孪生女子在重兵 把守的监狱“救”出了数十名罪犯,令人发指!当笔者正在纳闷:是何故让两个美女冒死“劫狱”?本文的第三段便泄露了“天机”:to free their kidnapper boyfriends who were serving life sentences there.
文摘大模型红队测试(Large Model Red Teaming)旨在让大语言模型(Large Language Model, LLM)接收对抗测试,从而诱使模型输出有害的测试用例,进而发现模型中的漏洞并提高其鲁棒性。大模型红队测试是大模型领域的前沿课题,近年来受到学术界和工业界的广泛关注。研究者们针对大模型红队测试提出了众多解决方案,并在模型对齐上取得了一定进展。然而,受限于大模型红队数据的短缺和评价标准的模糊,现有研究大多局限于针对特定的场景进行评估。文中首先从与大模型安全相关的定义出发,对其所涉及的各种风险进行阐述;其次,针对大模型红队测试的重要性及其主要类别进行了阐述,综述和分析了相关红队技术的发展历程,并介绍了已有的数据集和评价指标;最后,对大模型红队测试的未来发展趋势进行了展望和总结。
文摘The release of Apple’s iPhone was one of the most intensively publicized product releases in the history of mobile devices. While the iPhone wowed users with its exciting design and features, it also angered many for not allowing installation of third party applications and for working exclusively with AT & T wireless services (in the US). Besides the US, iPhone was only sold only in a few other selected countries. Software attacks were developed to overcome both limitations. The development of those attacks and further evaluation revealed several vulnerabilities in iPhone security. In this paper, we examine some of the attacks developed for the iPhone as a way of investigating the iPhone’s security structure. We also analyze the security holes that have been discovered and make suggestions for improving iPhone security.
文摘近年来,大语言模型(large language model,LLM)在一系列下游任务中得到了广泛应用,并在多个领域表现出了卓越的文本理解、生成与推理能力.然而,越狱攻击正成为大语言模型的新兴威胁.越狱攻击能够绕过大语言模型的安全机制,削弱价值观对齐的影响,诱使经过对齐的大语言模型产生有害输出.越狱攻击带来的滥用、劫持、泄露等问题已对基于大语言模型的对话系统与应用程序造成了严重威胁.对近年的越狱攻击研究进行了系统梳理,并基于攻击原理将其分为基于人工设计的攻击、基于模型生成的攻击与基于对抗性优化的攻击3类.详细总结了相关研究的基本原理、实施方法与研究结论,全面回顾了大语言模型越狱攻击的发展历程,为后续的研究提供了有效参考.对现有的安全措施进行了简略回顾,从内部防御与外部防御2个角度介绍了能够缓解越狱攻击并提高大语言模型生成内容安全性的相关技术,并对不同方法的利弊进行了罗列与比较.在上述工作的基础上,对大语言模型越狱攻击领域的现存问题与前沿方向进行探讨,并结合多模态、模型编辑、多智能体等方向进行研究展望.
文摘大语言模型(large language models,LLMs)虽然取得了显著的成功,但在实际应用中依然面临着安全问题,容易在恶意诱导下生成有毒、有害内容.目前缓解LLMs不安全行为的方法通常需要高昂的数据收集成本以及大量的计算资源.大模型知识编辑可以在不重新训练模型的基础上,根据特定的输入精确地改变模型对应的输出,在节约大量资源的条件下约束模型的行为;为优化大模型生成安全内容提供了一个新的可行思路.然而,目前学术界缺乏较为系统和全面的基于知识编辑的大模型内容安全生成分析数据集.具体地说,当前缓解LLMs不安全行为的数据集并未包括所有的不安全场景,且其有毒问题几乎无法绕过对齐后的LLMs安全防线,因此无法缓解对齐后LLMs存在的不安全问题.针对上述问题,设计了新的数据集SafeGen,并提出新的评价体系分析知识编辑在优化LLMs生成安全内容的潜力.大量的实验发现知识编辑可以提高LLMs内部的安全信念,在校正LLMs不安全行为领域展现了广阔的应用前景.但经过知识编辑的LLMs生成文本的流畅性却差强人意,这也表明了这项任务的潜在难度.该工作可以为大模型安全社区提供一些见解.
文摘This review paper explores advanced methods to prompt Large LanguageModels(LLMs)into generating objectionable or unintended behaviors through adversarial prompt injection attacks.We examine a series of novel projects like HOUYI,Robustly Aligned LLM(RA-LLM),StruQ,and Virtual Prompt Injection that compel LLMs to produce affirmative responses to harmful queries.Several new benchmarks,such as PromptBench,AdvBench,AttackEval,INJECAGENT,and Robustness Suite,have been created to evaluate the performance and resilience of LLMs against these adversarial attacks.Results show significant success rates in misleading models like Vicuna-7B,LLaMA-2-7B-Chat,GPT-3.5,and GPT-4.The review highlights limitations in existing defense mechanisms and proposes future directions for enhancing LLM alignment and safety protocols,including the concept of LLM SELF DEFENSE.Our study emphasizes the need for improved robustness in LLMs,which will potentially shape the future of Artificial Intelligence(AI)driven applications and security protocols.Understanding the vulnerabilities of LLMs is crucial for developing effective defenses against adversarial prompt injection attacks.This paper proposes a systemic classification framework that discusses various types of prompt injection attacks and defenses.We also go through a broad spectrum of stateof-the-art attack methods(such as HouYi and Virtual Prompt Injection)alongside advanced defense mechanisms(like RA-LLM,StruQ,and LLM Self-Defense),providing critical insights into vulnerabilities and robustness.We also integrate and compare results from multiple recent benchmarks,including PromptBench,INJECENT,and BIPIA.
