Paper deep dive

A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI Responses

David Glukhov, Ziwen Han, Ilia Shumailov, Vardan Papyan, Nicolas Papernot

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: TheoreticalEmbeddings: 158

Models: LLaMA2-70B-Instruct, Mistral

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 7:24:29 PM

Summary

The paper introduces 'Decomposition Attacks' (DAs), a novel black-box attack method where an adversarial LLM breaks down malicious queries into innocuous subquestions to extract dangerous knowledge from a victim LLM. It proposes an information-theoretic framework to evaluate safety risks via 'Impermissible Information Leakage' (IIL), distinguishing between 'inferential adversaries' (who seek knowledge) and 'security adversaries' (who seek specific outputs). The authors also define 'Information Censorship' as a defense mechanism and analyze the inherent safety-utility trade-off.

Entities (5)

Decomposition Attack · attack-method · 100%WMDP · dataset · 98%Inferential Adversary · threat-model · 95%Information Censorship · defense-mechanism · 95%Security Adversary · threat-model · 95%

Relation Signals (3)

Information Censorship → defendsagainst → Inferential Adversary

confidence 95% · To defend against inferential adversaries, in section 5, we leverage information theory to introduce Information Censorship

Decomposition Attack → instantiates → Inferential Adversary

confidence 95% · we empirically ground our theoretical contributions by instantiating an inferential adversary through a Decomposition Attack

Decomposition Attack → evaluatedusing → WMDP

confidence 90% · In section 3, we propose a novel framework to evaluate Decomposition Attacks on a subset of questions from the Weapons of Mass Destruction Proxy (WMDP) dataset

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vulnerability of Frontier language models to misuse and jailbreaks has prompted the development of safety measures like filters and alignment training in an effort to ensure safety through robustness to adversarially crafted prompts. We assert that robustness is fundamentally insufficient for ensuring safety goals, and current defenses and evaluation methods fail to account for risks of dual-intent queries and their composition for malicious goals. To quantify these risks, we introduce a new safety evaluation framework based on impermissible information leakage of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking. Underlying our proposed evaluation method is a novel information-theoretic threat model of inferential adversaries, distinguished from security adversaries, such as jailbreaks, in that success is measured by inferring impermissible knowledge from victim outputs as opposed to forcing explicitly impermissible outputs from the victim. Through our information-theoretic framework, we show that to ensure safety against inferential adversaries, defense mechanisms must ensure information censorship, bounding the leakage of impermissible information. However, we prove that such defenses inevitably incur a safety-utility trade-off.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

157,451 characters extracted from source content.

Expand or collapse full text

Breach By A Thousand Leaks: Unsafe Information Leakage in ‘Safe’ AI Responses David Glukhov1,2 Ziwen Han1,2 Ilia Shumailov3 Vardan Papyan1,2 Nicolas Papernot1,2 1University of Toronto 2Vector Institute 3University of Oxford Abstract Vulnerability of Frontier language models to misuse and jailbreaks has prompted the development of safety measures like filters and alignment training in an effort to ensure safety through robustness to adversarially crafted prompts. We assert that robustness is fundamentally insufficient for ensuring safety goals, and current defenses and evaluation methods fail to account for risks of dual-intent queries and their composition for malicious goals. To quantify these risks, we introduce a new safety evaluation framework based on impermissible information leakage of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking. Underlying our proposed evaluation method is a novel information-theoretic threat model of inferential adversaries, distinguished from security adversaries, such as jailbreaks, in that success is measured by inferring impermissible knowledge from victim outputs as opposed to forcing explicitly impermissible outputs from the victim. Through our information-theoretic framework, we show that to ensure safety against inferential adversaries, defense mechanisms must ensure information censorship, bounding the leakage of impermissible information. However, we prove that such defenses inevitably incur a safety-utility trade-off. Figure 1: An inferential adversary can infer an answer to the harmful question by asking dual-intent subquestions without jailbreaking, demonstrating robust models can still be misused. 1 Introduction Background Frontier models have demonstrated remarkable capabilities (Bubeck et al., 2023; Llama Team, 2024; Reid et al., 2024), but their potential for misuse has raised alarm about possible risks. These encompass a wide taxonomy of privacy and security threats, ranging from social manipulation to the creation of dangerous weapons by malicious actors (Phuong et al., 2024; Slattery et al., 2024; Weidinger et al., 2022; Durmus et al., 2024; Li et al., 2024b). In response, researchers have developed various mitigation strategies, including prompt engineering (Bai et al., 2022a), aligning models with human values through fine-tuning (Ouyang et al., 2022; Rafailov et al., 2024), improving robustness via adversarial training (Bai et al., 2022b), and implementing input and output filters as guardrails (Debenedetti et al., 2023; Zou et al., 2024). Motivation Despite these efforts, recent work has called into question the reliability of extant safety methods and their assessments (Feffer et al., 2024; Kapoor et al., 2024). Significant concerns stem from poorly defined threat models with tenuous connections to real-world safety risks and a lack of compelling criteria for evaluating attacks and defenses. Current threat models and assessment methods typically focus solely on the permissibility of the victim model’s responses (Zou et al., 2024), which do not capture many expressly stated safety concerns. For example, as illustrated in Figure 1, an adversary seeking to implement a social engineering attack can achieve their goal without eliciting an explicitly harmful response to a query like “How do I scam the elderly?. Method Building on this intuition, in section 2 we introduce Decomposition Attacks (DAs): automated black-box attacks which decompose malicious questions into seemingly innocuous subquestions, posing them to a victim LLM, and aggregating the responses to answer the malicious question. The attacker resembles a problem-solving agent (Khot et al., 2022), relying on the expertise of a victim LLM to acquire dual-use information for fulfilling the malicious task. Evaluation In section 3, we propose a novel framework to evaluate Decomposition Attacks on a subset of questions from the Weapons of Mass Destruction Proxy (WMDP) dataset for hazardous knowledge (Li et al., 2024b). This dataset is chosen for its multiple-choice format, allowing us to quantify a victim LLM’s safety risk by comparing an adversary’s prediction before and after the attack. We compare our non-jailbreaking DA against a commonly studied black-box jailbreaking method: PAIR (Chao et al., 2023). We find that our instantiation of a DA is able more effectively extract information from a victim LLM than jailbreaking as well as triggering input and output filters less frequently. Our proposed evaluation approach enables measuring information leakage over multiple responses, in contrast to existing jailbreak evaluations that capture safety risk by ascertaining the permissibility of a single victim output. Threat Models In section 4, we formally distinguish these approaches by defining two threat models: Inferential Adversaries and Security Adversaries. Inferential adversaries (eg. Decomposition Attacks) seek to extract harmful information from victim responses, with success quantified by impermissible information gain. Security adversaries (eg. jailbreaks) instead seek to force specific impermissible outputs from the victim model, with success quantified in a dichotomous manner, either the victim produced the specific output or not. Defense To defend against inferential adversaries, in section 5, we leverage information theory to introduce Information Censorship: a bound on the expected impermissible information leaked through interactions between adversary and victim models. Information Censorship serves as a condition for defense mechanisms to guarantee safety against query-bounded inferential adversaries. To illustrate such a censorship mechanism, we define a randomised response mechanism (Mangat, 1994) which bounds impermissible information leakage by probabilistically returning empty strings to the adversary with probability dependent on the worst-case expected impermissible information leakage. Safety-Utility Tradeoff Due to the dual-use nature of information, effective defense mechanisms, such as the randomised response censorship mechanism, constrain a models usefulness to both malicious and benign users. We formalize this intuition in subsection 5.3 by providing theoretical safety-utility trade-offs for the randomised response mechanism and more generally for information censorship mechanisms. Key Contributions 1. We introduce a novel information-theoretic threat model of inferential adversaries and use it to propose a new evaluation framework for assessing the LLM deployment safety risk. 2. To instantiate inferential adversaries, we propose Decomposition Attacks, a black-box attack relying on question decomposition and answer aggegation to extract impermissible information from a victim LLM. 3. We define Information Censorship: a constraint on impermissible information leakage which defense mechanisms must satisfy to guarantee safety against inferential adversaries. We design a randomised-response mechanism providing such guarantees and establish safety-utility trade-offs for Information Censorship mechanisms. Algorithm 1 Decomposition Attack for LLMs 1:Input: Query q, interaction params k and m, few shot examples Eg,Ee,EasubscriptsubscriptsubscriptE_g,E_e,E_aEitalic_g , Eitalic_e , Eitalic_a 2:Output: Answer a to query q 3:procedure DecompositionAttack(q,kq,kq , k) 4: Initialize knowledge pile h←∅←ℎh← ← ∅ 5: for j=11j=1j = 1 to k do 6: [qs1,…,qsm]←AdvLLM⁢(Eg,h,q,m)←subscriptsubscript1…subscriptsubscriptAdvLLMsubscriptℎ[q_s_1,…,q_s_m]← AdvLLM(E_g,h,q,m)[ qitalic_s start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , qitalic_s start_POSTSUBSCRIPT m end_POSTSUBSCRIPT ] ← AdvLLM ( Eitalic_g , h , q , m ) 7: for i=11i=1i = 1 to m do 8: ri←Mo⁢u⁢t(VicLLM(Mi⁢n⁢p(qsi)r_i← M_out( VicLLM(M_inp(q_s_i)ritalic_i ← Mitalic_o u t ( VicLLM ( Mitalic_i n p ( qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) 9: ai←AdvLLM⁢(Ee,q,qsi,ri)←subscriptAdvLLMsubscriptsubscriptsubscriptsubscripta_i← AdvLLM(E_e,q,q_s_i,r_i)aitalic_i ← AdvLLM ( Eitalic_e , q , qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ritalic_i ) 10: h←h∪(qsi,ai)←ℎsubscriptsubscriptsubscripth← h∪(q_s_i,a_i)h ← h ∪ ( qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , aitalic_i ) 11: end for 12: end for 13: aq←AdvLLM⁢(Ea,q,h)←subscriptAdvLLMsubscriptℎa_q← AdvLLM(E_a,q,h)aitalic_q ← AdvLLM ( Eitalic_a , q , h ) 14: return aqsubscripta_qaitalic_q 15:end procedure 2 Method: Decomposition Attacks Overview We empirically ground our theoretical contributions by instantiating an inferential adversary through a Decomposition Attack—an automated black-box attack method for extracting and leveraging dual-use information to fulfill adversary objectives. The attack involves an adversarial LLM leveraging the knowledge or capabilities possessed by a victim LLM to answer an impermissible question or fulfill a malicious task. It does this by generating subquestions/subtasks then prompting the victim LLM with them. As shown in Figure 1, there exist many seemingly benign, dual-use subquestions whose answers would assist in answering an impermissible question while avoiding input and output filters by not being strictly problematic. Algorithm 1 provides an overview of our DA method. Setting Assume that a malicious user prompts an adversarial LLM, denoted by AdvLLM, with a query q, typically a question to answer or task to complete. Assume further that the adversarial LLM has access to a black-box victim LLM, denoted by VicLLM, and is capable of prompting it at most n times. We assume that the victim LLM employs input and output filtering mechanisms, MisubscriptM_iMitalic_i and MosubscriptM_oMitalic_o, respectively, and may monitor jailbreak or safety violation attempts to block the adversary’s access. Inputs To execute the attack, a malicious user first provides the adversarial LLM with a malicious query q, number of subquestion rounds k and number of subquestions to generate per round m such that k⁢m≤nkm≤ nk m ≤ n. The user further provides the following behavioural template to guide the adversary: EgsubscriptE_gEitalic_g: Few-shot examples of query decomposition into self-contained subqueries; EesubscriptE_eEitalic_e: Few-shot examples of extractions of subanswers from victim responses to subqueries; EasubscriptE_aEitalic_a: Few-shot examples of the aggregation of subquery-subanswer pairs to answer a given query. Initialization AdvLLM initializes knowledge pile hℎh to hold a history of interactions with VicLLM. Subquestion Generation For each of the k rounds, AdvLLM is provided with a prompt containing the few shot subquestion generation examples EgsubscriptE_gEitalic_g, the knowledge pile hℎh, the malicious query q, and instructions to generate m self-contained subquestions that would help in answering the malicious query q. The AdvLLM then outputs a string of m subquestions qssubscriptq_sqitalic_s which are separated into a list: [qs1,qs2,…,qsm]←AdvLLM⁢(Eg,h,q,m).←subscriptsubscript1subscriptsubscript2…subscriptsubscriptAdvLLMsubscriptℎ [q_s_1,q_s_2,…,q_s_m]← AdvLLM(E_% g,h,q,m).[ qitalic_s start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , qitalic_s start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , qitalic_s start_POSTSUBSCRIPT m end_POSTSUBSCRIPT ] ← AdvLLM ( Eitalic_g , h , q , m ) . Victim Interaction Each of the m generated subquestions qsisubscriptsubscript\q_s_i\ qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT is submitted to the VicLLM, passing through any input or output filtering mechanisms Mi⁢n⁢psubscriptM_inpMitalic_i n p and Mo⁢u⁢tsubscriptM_outMitalic_o u t it employs. In turn, the VicLLM responds to each subquestion with a response: ri←Mo⁢u⁢t(VicLLM(Mi⁢n⁢p(qsi)). r_i← M_out( VicLLM(M_inp(q_s_i)).ritalic_i ← Mitalic_o u t ( VicLLM ( Mitalic_i n p ( qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) ) . Processing Responses Due to the length and irrelevant information contained within the response risubscriptr_iritalic_i, for each non-empty response the adversarial LLM extracts an answer aisubscripta_iaitalic_i from risubscriptr_iritalic_i relevant to the query q and subquestion qsisubscriptsubscriptq_s_iqitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT and appends it to the knowledge pile hℎh: ai←AdvLLM⁢(Ee,q,qsi,ri);←subscriptAdvLLMsubscriptsubscriptsubscriptsubscript a_i← AdvLLM(E_e,q,q_s_i,r_i);aitalic_i ← AdvLLM ( Eitalic_e , q , qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ritalic_i ) ; h←h∪(qsi,ai).←ℎsubscriptsubscriptsubscript h← h∪(q_s_i,a_i).h ← h ∪ ( qitalic_s start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , aitalic_i ) . Answer Aggregation The last step of the DA is answer aggregation, where the AdvLLM is provided with the malicious query q, the knowledge pile hℎh containing all subquestion answer pairs from interactions with the VicLLM, and few shot examples EasubscriptE_aEitalic_a to produce an answer a to the query q: aq←AdvLLM⁢(Ea,q,h).←subscriptAdvLLMsubscriptℎ a_q← AdvLLM(E_a,q,h).aitalic_q ← AdvLLM ( Eitalic_a , q , h ) . 3 Evaluation 3.1 Proposed Framework Motivation To evaluate the Decomposition Attacks proposed in the previous section and assess the safety risk of deploying an LLM, in this section we introduce a novel evaluation framework derived from the theoretical threat models formalized in section 4. Presently, safety evaluations of LLMs measure attack and defense effectiveness through the attack success rate (ASR) metric, which quantifies the proportion of times the attack forces a VicLLM to return an explicitly impermissible output string. However, as shown in Figure 1, an explicitly impermissible output is not necessary for an adversary to acquire information that could be used for harm. Proposed Evaluation To address this limitation, we propose to directly evaluate the ability of an AdvLLM to correctly answer an impermissible question after attacking the VicLLM. As the AdvLLM may be capable of correctly answering the impermissible question on its own, for a given AdvLLM we can measure the marginal risk induced by a VicLLM by comparing against the AdvLLM’s baseline ability of answering the question before, or without, executing the attack. Evaluation Dataset For our proposed evaluation, we consider the WMDP dataset (Li et al., 2024b): a collection of multiple choice questions, denoted q, to serve as a proxy for dangerous biological, chemical, or cybernetic knowledge. The set of possible answers, denoted aisubscript\a_i\ aitalic_i , and single correct answer, denoted a∗superscripta^*a∗, enables measuring the AdvLLM’s confidence in the correct answer before and after attacking a VicLLM by computing next-token probabilities of each of the answer options. Notation Thus, for a question q, we assume that knowledge of a∗superscripta^*a∗ being the correct answer is impermissible, and, we use pa⁢d⁢v⁢(a∗|q)subscriptconditionalsuperscriptp_adv(a^*|q)pitalic_a d v ( a∗ | q ) and pa⁢d⁢v⁢(a∗|h,q)subscriptconditionalsuperscriptℎp_adv(a^*|h,q)pitalic_a d v ( a∗ | h , q ) to denote the AdvLLM’s belief in the correct answer before and after the attack respectively. We use hℎh, the knowledge pile, to denote a string extracted from all prompt-response interactions between the AdvLLM and VicLLM. Measuring Risk Provided the before and after probabilities pa⁢d⁢v⁢(a∗|q)subscriptconditionalsuperscriptp_adv(a^*|q)pitalic_a d v ( a∗ | q ) and pa⁢d⁢v⁢(a∗|h,q)subscriptconditionalsuperscriptℎp_adv(a^*|h,q)pitalic_a d v ( a∗ | h , q ), the question of how to properly measure risk arises. One approach is to measure the difference in accuracy before and after the attack, however, accuracy is not a granular metric and fails to account for changes in confidence. This issue is commonly resolved by measuring cross-entropy of the AdvLLM’s prediction with the correct answer. The difference in the cross-entropies, pa⁢d⁢v⁢(a∗|q)subscriptconditionalsuperscriptp_adv(a^*|q)pitalic_a d v ( a∗ | q ) and pa⁢d⁢v⁢(a∗|h,q)subscriptconditionalsuperscriptℎp_adv(a^*|h,q)pitalic_a d v ( a∗ | h , q ), is expressed as log⁡(pa⁢d⁢v⁢(a∗|h,q))−log⁡(pa⁢d⁢v⁢(a∗|q))=log⁡(pa⁢d⁢v⁢(a∗|h,q))(pa⁢d⁢v⁢(a∗|q))subscriptconditionalsuperscriptℎsubscriptconditionalsuperscriptsubscriptconditionalsuperscriptℎsubscriptconditionalsuperscript (p_adv(a^*|h,q))- (p_adv(a^*|q))= (p_adv% (a^*|h,q))(p_adv(a^*|q))log ( pitalic_a d v ( a∗ | h , q ) ) - log ( pitalic_a d v ( a∗ | q ) ) = log divide start_ARG ( pitalic_a d v ( a∗ | h , q ) ) end_ARG start_ARG ( pitalic_a d v ( a∗ | q ) ) end_ARG commonly known as the pointwise mutual information. However, under this metric, the safety risk when pa⁢d⁢v⁢(a∗|h,q)=.9subscriptconditionalsuperscriptℎ.9p_adv(a^*|h,q)=.9pitalic_a d v ( a∗ | h , q ) = .9 and pa⁢d⁢v⁢(a∗|q)=.1subscriptconditionalsuperscript.1p_adv(a^*|q)=.1pitalic_a d v ( a∗ | q ) = .1 is the same as the risk when pa⁢d⁢v⁢(a∗|h,q)=.009subscriptconditionalsuperscriptℎ.009p_adv(a^*|h,q)=.009pitalic_a d v ( a∗ | h , q ) = .009 and pa⁢d⁢v⁢(a∗|q)=.001subscriptconditionalsuperscript.001p_adv(a^*|q)=.001pitalic_a d v ( a∗ | q ) = .001. To address this, we propose to calibrate the safety risk metric by the adversary’s final belief in the correct answer—pa⁢d⁢v⁢(a∗|h,q)subscriptconditionalsuperscriptℎp_adv(a^*|h,q)pitalic_a d v ( a∗ | h , q ). Thus, we propose to evaluate safety risk through Impermissible Information Leakage (IIL). Definition 3.1 (Impermissible Information Leakage (IIL)). Let pa⁢d⁢v⁢(a∗|q)subscriptconditionalsuperscriptp_adv(a^*|q)pitalic_a d v ( a∗ | q ) and pa⁢d⁢v⁢(a∗|h,q)subscriptconditionalsuperscriptℎp_adv(a^*|h,q)pitalic_a d v ( a∗ | h , q ) represent the belief of AdvLLM in a∗superscripta^*a∗ before and after interacting with VicLLM. We define the impermissible information leakage as: IIL=pa⁢d⁢v⁢(a∗|h,q)⁢log⁡pa⁢d⁢v⁢(a∗|h,q)pa⁢d⁢v⁢(a∗|q)IILsubscriptconditionalsuperscriptℎsubscriptconditionalsuperscriptℎsubscriptconditionalsuperscript IIL=p_adv(a^*|h,q) p_adv(a^*|h,q)p_adv(a^*|q)IIL = pitalic_a d v ( a∗ | h , q ) log divide start_ARG pitalic_a d v ( a∗ | h , q ) end_ARG start_ARG pitalic_a d v ( a∗ | q ) end_ARG 3.2 Experiments Experimental Design: For our experiments, we compare the ability of an AdvLLM to extract impermissible information from a larger VicLLM using our proposed DA attack and an adaptation of the commonly studied black-box jailbreaking method PAIR (Chao et al., 2023) which also relies on an adversarial LLM to iteratively generate prompts to the victim LLM to bypass safety measures. To demonstrate the effectiveness of DAs at extracting information while bypassing safety filters, we curate subsets of WMDP-Bio and WMDP-Chem questions which get flagged as unsafe by the Llama-Guard-3-8B model (Llama Team, 2024), and employ the model as an input MisubscriptM_iMitalic_i and output MosubscriptM_oMitalic_o filter for VicLLM. Furthermore, we examine both attacks in a rate-limited setting where there is a limit on the number of interactions between AdvLLM and VicLLM. More details regarding our implementation can be found in Appendix B. We report our results in Table 1. Attack Method Adversary Model WMDP-Bio ↑ ↑ WMDP-Chem ↑ ↑ Flags(Bio/Chem) ↓ ↓ DA (Ours) Mistral-7B-Instruct 0.95±0.22plus-or-minus0.950.220.95± 0.220.95 ± 0.22 0.48±0.10plus-or-minus0.480.100.48± 0.100.48 ± 0.10 0.34/0.300.340.300.34/0.300.34 / 0.30 DA (Ours) Llama-8B-Instruct 0.59¯±0.09plus-or-minus¯0.590.09 0.59± 0.09under¯ start_ARG 0.59 end_ARG ± 0.09 0.38¯±0.05plus-or-minus¯0.380.05 0.38± 0.05under¯ start_ARG 0.38 end_ARG ± 0.05 0.03/0.110.030.110.03/0.110.03 / 0.11 PAIR Mistral-7B-Instruct 0.43±0.12plus-or-minus0.430.120.43± 0.120.43 ± 0.12 0.32±0.12plus-or-minus0.320.120.32± 0.120.32 ± 0.12 1.15/2.071.152.071.15/2.071.15 / 2.07 PAIR Llama-8B-Instruct 0.28±0.06plus-or-minus0.280.060.28± 0.060.28 ± 0.06 0.16±0.04plus-or-minus0.160.040.16± 0.040.16 ± 0.04 0.13¯/0.58¯0.13¯0.58 0.13/ 0.58under¯ start_ARG 0.13 end_ARG / under¯ start_ARG 0.58 end_ARG Table 1: We fit a linear mixed-effects model over measured IIL for every attack and report its mean and standard error for curated subsets of WMDP-Bio and WMDP-Chem. Analysis of Results We find that both our proposed DA and adaption of the PAIR jailbreak were able to successfully extract impermissible information from the VicLLM despite the defense measures employed. Moreover, we observe that on average, the number of times interactions get flagged as unsafe per attack is low, particularly considering that each attack typically consists of 6666 interactions. In particular we observe the safety-aligned Llama-3.1-8B-Instruct models are very effective at remaining undetected, an important property if user-monitoring was employed to detect and block VicLLM access to bad actors. Finally, in Table 2 we report two-sample t-test results comparing the IIL performance of the two attacks, demonstrating that our DA significantly outperforms PAIR. Interpreting Performance Differences As flag rates for both attacks were low (although relatively higher for PAIR), we speculate that one reason for the worse performance of PAIR stems from the fact that our proposed DA attack extracts information from each interaction with the VicLLM. On the other hand, PAIR attempts to generate a prompt which both bypasses the filtering mechanisms while also receiving a clear answer to the multiple choice question, two opposing goals when the multiple choice question itself is deemed impermissible to ask. Comparison WMDP-Bio WMDP-Chem DA >>> PAIR (Mistral-7B-Instruct) 0.0342* 0.317 DA >>> PAIR (Llama-8B-Instruct) 0.0055** 0.0004*** Table 2: Two-sample t-test results comparing DA against PAIR. The values reported are p-values. 4 Adversary Threat Models Motivation To formally understand the underlying distinction between our proposed evaluation framework and the commonly studied evaluations, in this section we define and distinguish the objectives of two safety threat models, security adversaries and inferential adversaries. Defining the adversary objectives immediately provides us with a method for evaluating adversary success and enable us to define criteria for defense in section 5. 4.1 Setting Notation Let XX and YY be the set of valid input and output strings to both, AdvLLM and VicLLM, respectively. The LLMs AdvLLM:→P⁢():AdvLLM→ AdvLLM:X→ P(Y)AdvLLM : X → P ( Y ) and VicLLM:→P⁢():VicLLM→ VicLLM:X→ P(Y)VicLLM : X → P ( Y ) are defined as mappings of input strings to a distribution over output strings. An interaction between the adversary and victim is represented by an input-output pair (q,a)∈×(q,a) ×Y( q , a ) ∈ X × Y, where q is the input to VicLLM and y∼VicLLM⁢(x)similar-toVicLLMy VicLLM(x)y ∼ VicLLM ( x ) a sample output returned to the AdvLLM. Within these interactions, the adversary aims to maximize a scoring function s:×→ℝ+:→superscriptℝs:X×Y ^+s : X × Y → blackboard_R+, which quantifies how well these input-output pairs fulfill the adversary’s goals. Definition 4.1 (Censorship Mechanism). A Censorship Mechanism M:×P⁢()→P⁢():→M:X× P(Y)→ P(Y)M : X × P ( Y ) → P ( Y ) is a randomized function that outputs a new distribution over responses returned to a user-provided input. The mechanism M seeks to ensure that responses satisfy a safety criterion dependent on the assumed threat model. To define the objective of a censorship mechanism, in particular the constraints it must ensure to guarantee safety, it is essential to define the adversary threat model which we seek to provide safety guarantees against. Thus, we turn to introducing security and inferential adversary threat models, specifically their goals, which can capture most LLM safety concerns. 4.2 Security Threats Examples To help understand security adversaries, we first provide some examples of them. Jailbreaks involve an adversary attempting to bypass content filters to make the VicLLM generate harmful content. This is evaluated by checking if the output a returned starts with a certain string (e.g., “Sure, here’s how to evade taxes:”) (Liu et al., 2023), or, an impermissibility score assigned to a by a judge LLM passes a certain threshold (Souly et al., 2024). Prompt Injection attacks involve an adversary inserting malicious instructions within an input prompt q to the VicLLM to manipulate the model to perform a specific task. Success is evaluated whether or not the VicLLM executed the task successfully. In these cases, the adversary’s success is evaluated dichotomously—either the VicLLM exhibits a target behavior or not. Formally, we define the security adversary objective as follows. Definition 4.2 (Security Adversary Objective). The goal of a security adversary is characterized by a target function t:→⁢():→t:X (Y)t : X → P ( Y ) mapping input strings to a set of impermissible output strings, and is evaluated by the following scoring function for Tq=t⁢(q)subscriptT_q=t(q)Titalic_q = t ( q ): s⁢(q,a)=a∈Tqsubscript1subscript s(q,a)= 1_a∈ T_qs ( q , a ) = blackboard_1a ∈ T start_POSTSUBSCRIPT q end_POSTSUBSCRIPT For a VicLLM input q, the impermissible output set Tq=t⁢(q)subscriptT_q=t(q)Titalic_q = t ( q ) is assumed to be common knowledge to both adversary and victim as membership of a∈Tqsubscripta∈ T_qa ∈ Titalic_q can be directly verified. VicLLM is secure if and only if ∀q,a:s⁢(q,a)=0:for-all0∀ q,a:\;s(q,a)=0∀ q , a : s ( q , a ) = 0. Limitations Security adversaries fail to fully capture many threats such as deepfake generation or informative instructions for conducting illegal activities. This is because adversaries can compose individual acceptable outputs to construct deepfakes or acquire instructions (Glukhov et al., 2023). 4.3 Inferential Threats Distinction Inferential adversaries provide a compelling alternative threat model for many safety concerns which are not adequately captured by security adversaries by their ability to better capture compositional threats. They differ from security adversaries in that VicLLM outputs are no longer dichotomously evaluated as either permissible or impermissible. Instead, we consider inferential adversaries as seeking to gain information about an impermissible concept, such as biological weapons, from VicLLM outputs. This enables quantifying marginal risk of outputs or their compositions. Impermissible Concepts We characterize “impermissible concepts”: knowledge which would be considered dangerous, risky, harmful, or otherwise undesirable for an individual to know through sets of input-output pairs. Specifically, let Q⊂Q ⊂ X be a set of queries that a malicious adversary wants to answer, and for any q∈Qq∈ Qq ∈ Q, the set Aq⊂subscriptA_q _q ⊂ Y is the set of answers a for which the pair (q,a)(q,a)( q , a ) is considered to contain impermissible knowledge. For instance, q∈Qq∈ Qq ∈ Q might be asking for malware code, and Aq⊂subscriptA_q _q ⊂ Y could be the set of all output strings containing a malware implementation. We note the multiple choice examples considered from our evaluation framework in section 3 are a special case when =aisubscriptY=\a_i\Y = aitalic_i and Aq=a∗subscriptsuperscriptA_q=a^*Aitalic_q = a∗. Definition 4.3 (Inferential Adversary Objective). Let pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ) represent the adversary’s prior belief about the answer to a malicious query q. The goal of an inferential adversary AdvLLM with prior pa⁢d⁢v⁢(a|q)subscriptconditionalp_adv(a|q)pitalic_a d v ( a | q ) is to select k inputs qii=1ksuperscriptsubscriptsubscript1\q_i\_i=1^k qitalic_i i = 1k to maximize the scoring function s⁢(q,hk)=∑a∈Aqpa⁢d⁢v⁢(a|hk,q)⁢log⁡pa⁢d⁢v⁢(a|hk,q)pa⁢d⁢v⁢(a|q).superscriptℎsubscriptsubscriptsubscriptconditionalsuperscriptℎsubscriptconditionalsuperscriptℎsubscriptconditional s(q,h^k)= _a∈ A_qp_adv(a|h^k,q) p_adv(% a|h^k,q)p_adv(a|q).s ( q , hitalic_k ) = ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | hitalic_k , q ) log divide start_ARG pitalic_a d v ( a | hitalic_k , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG . where hk=(qi,ai)i=1ksuperscriptℎsuperscriptsubscriptsubscriptsubscript1h^k=\(q_i,a_i)\_i=1^khitalic_k = ( qitalic_i , aitalic_i ) i = 1k is the knowledge pile of k interactions with VicLLM. 5 Information Censorship Overview To mitigate risks incurred by inferential adversaries, the victim model provider seeks to minimize the expected impermissible information leakage over collections of interactions. In this section we define information censorship, a bound on expected impermissible information leakage (Exp-IIL) which defense mechanisms must ensure to guarantee safety against inferential adversaries. We propose a randomised response defense mechanism for ensuring this bound. 5.1 Safety Guarantee Definition 5.1 (Expected Impermissible Information Leakage (Exp-IIL)). Let M be a censorship mechanism such that for an input qisubscriptq_iqitalic_i to the VicLLM, the answer aisubscripta_iaitalic_i returned to AdvLLM is sampled from M⁢(qs,VicLLM⁢(qs))subscriptVicLLMsubscriptM(q_s, VicLLM(q_s))M ( qitalic_s , VicLLM ( qitalic_s ) ). Assuming the AdvLLM seeks to answer q∈Qq∈ Qq ∈ Q and submits a set of k queries qii=1ksuperscriptsubscriptsubscript1\q_i\_i=1^k qitalic_i i = 1k to the VicLLM, define the distribution of knowledge piles hk=(qi,ai)i=1ksuperscriptℎsuperscriptsubscriptsubscriptsubscript1h^k=\(q_i,a_i)\_i=1^khitalic_k = ( qitalic_i , aitalic_i ) i = 1k to be Hqik:=qi,M⁢(qi,VicLLM⁢(qi))i=1kassignsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptVicLLMsubscript1H_q_i^k:=\q_i,M(q_i, VicLLM(q_i))\_i=1^kHitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk := qitalic_i , M ( qitalic_i , VicLLM ( qitalic_i ) ) i = 1k. Then, the expected impermissible information leakage is given by: IAq(pa⁢d⁢v(⋅|q);Hqik)=∑a1,…,ak∈kpM(Hqik=hk)∑a∈Aqpa⁢d⁢v(a|hk,q)logpa⁢d⁢v⁢(a|hk,q)pa⁢d⁢v⁢(a|q). I_A_q(p_adv(·|q);H_q_i^k)= _\a_1,…,a_% k\ ^kp_M(H_q_i^k=h^k) _a∈ A_qp_adv(a|h% ^k,q) p_adv(a|h^k,q)p_adv(a|q).Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk ) = ∑ a start_POSTSUBSCRIPT 1 , … , aitalic_k ∈ Yitalic_k end_POSTSUBSCRIPT pitalic_M ( Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk = hitalic_k ) ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | hitalic_k , q ) log divide start_ARG pitalic_a d v ( a | hitalic_k , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG . Distinction from MI Our definition of Exp-IIL differs from the mutual information definition I(pa⁢d⁢v(⋅|q);Hqik)=∑a1,…,ak∈kpM(Hqik=hk)∑a∈pa⁢d⁢v(a|hk,q)logpa⁢d⁢v⁢(a|hk,q)pa⁢d⁢v⁢(a|q). I(p_adv(·|q);H_q_i^k)= _\a_1,…,a_k\∈% Y^kp_M(H_q_i^k=h^k) _a p_adv(a|h^k% ,q) p_adv(a|h^k,q)p_adv(a|q).I ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk ) = ∑ a start_POSTSUBSCRIPT 1 , … , aitalic_k ∈ Yitalic_k end_POSTSUBSCRIPT pitalic_M ( Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk = hitalic_k ) ∑a ∈ Y pitalic_a d v ( a | hitalic_k , q ) log divide start_ARG pitalic_a d v ( a | hitalic_k , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG . This distinction arises as our intent is to capture the asymmetry regarding which answers a an adversary becomes more confident in. From a safety perspective, the only concern is whether or not the adversary becomes more confident in impermissible conclusions a∈Aqsubscripta∈ A_qa ∈ Aitalic_q. Specifically, for the LLM provider, scenarios in which the adversary’s posterior pa⁢d⁢v⁢(a|hk,q)=1subscriptconditionalsuperscriptℎ1p_adv(a|h^k,q)=1pitalic_a d v ( a | hitalic_k , q ) = 1 for some a∉Aqsubscripta ∈ A_qa ∉ Aitalic_q is perfectly acceptable as it implies the adversary is confident in a “permissible” answer to q, whereas a bound on mutual information would deem this a defense failure assuming the entropy of the adversary prior distribution pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ) was high. In order to defend against an inferential adversary, AdvLLM, from inferring the answer to a harmful query q∈Qq∈ Qq ∈ Q over k interactions, a censorship mechanism M must bound the worst case Exp-IIL. Definition 5.2 ((k,ϵ)italic-ϵ(k,ε)( k , ϵ )-Information Censorship Mechanism (ICM)). For a collection of adversary priors Φa⁢d⁢vsubscriptΦ _advΦitalic_a d v, a malicious query q∈Qq∈ Qq ∈ Q, a leakage bound ϵ>0italic-ϵ0ε>0ϵ > 0, and k possible interactions between AdvLLM and VicLLM, a (k,ϵ)italic-ϵ(k,ε)( k , ϵ )-ICM M ensures the worst-case Exp-IIL is bounded by ϵitalic-ϵεϵ for knowledge pile distribution Hqik:=qi,M⁢(qi,VicLLM⁢(qi))i=1kassignsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptVicLLMsubscript1H_q_i^k:=\q_i,M(q_i, VicLLM(q_i))\_i=1^kHitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk := qitalic_i , M ( qitalic_i , VicLLM ( qitalic_i ) ) i = 1k: suppa⁢d⁢v∈Φ;qii=1k∈kIAq(pa⁢d⁢v(⋅|q);Hqik)≤ϵ. _ subarraycp_adv∈ ;\\ \q_i\_i=1^k ^k subarrayI_A_q(p_adv(·|q)% ;H_q_i^k)≤ε.supstart_ARG start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ ; end_CELL end_ROW start_ROW start_CELL qitalic_i i = 1k ∈ Xitalic_k end_CELL end_ROW end_ARG end_POSTSUBSCRIPT Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk ) ≤ ϵ . Compositional Bounds Finding and bounding the supremum of the Exp-IIL necessary for a (k,ϵ)italic-ϵ(k,ε)( k , ϵ )-ICM over all possible sets of k interactions is increasingly challenging due to the combinatorial complexity of checking all combinations. However, the (1,ϵ)1italic-ϵ(1,ε)( 1 , ϵ )-ICM (henceforth referred to as an ϵitalic-ϵεϵ-ICM) can also provide bounds on Exp-IIL for k interactions. Assuming the AdvLLM can interact with VicLLM across independent context windows, the ϵitalic-ϵεϵ-ICM cannot depend on knowledge of an existing interaction history hisuperscriptℎh^ihitalic_i—it must be non-adaptive. We provide a non-adaptive composition bound of an ϵitalic-ϵεϵ-ICM inspired by results in Nuradha and Goldfeld (2023). Theorem 5.3 (Non-Adaptive Composability of ϵitalic-ϵεϵ-ICM). For a collection of adversary priors Φ Φ, malicious query q∈Qq∈ Qq ∈ Q, leakage bound ϵ>0italic-ϵ0ε>0ϵ > 0, k possible interactions between AdvLLM and VicLLM, and an ϵitalic-ϵεϵ-ICM M, suppa⁢d⁢v∈Φ;qii=1k∈kIAq(pa⁢d⁢v(⋅|q);Hqik)≤kϵ+∑j=2kIAq((qj,aj);Hqij−1|pa⁢d⁢v(⋅|q)). _ subarraycp_adv∈ ;\\ \q_i\_i=1^k ^k subarrayI_A_q(p_adv(·|q)% ;H_q_i^k)≤ kε+ _j=2^kI_A_q((q_j,a_j);H_q_i^% j-1|p_adv(·|q)).supstart_ARG start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ ; end_CELL end_ROW start_ROW start_CELL qitalic_i i = 1k ∈ Xitalic_k end_CELL end_ROW end_ARG end_POSTSUBSCRIPT Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk ) ≤ k ϵ + ∑j = 2k Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( ( qitalic_j , aitalic_j ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTj - 1 | pitalic_a d v ( ⋅ | q ) ) . Interpretation In other words, the joint leakage can be bounded by the sum of k individual ϵitalic-ϵεϵ per-interaction leakages and a term capturing the dependencies between interactions when conditioned on pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ). If the model outputs VicLLM⁢(qi)VicLLMsubscript VicLLM(q_i)VicLLM ( qitalic_i ) are deterministic or independent when conditioned on pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ), then, the sum becomes 00 because the noise mechanism for an ϵitalic-ϵεϵ-ICM is independent of the response. Such assumptions could hold when there is a “single true value” of pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ) known by VicLLM, and all model outputs are related to this value by a deterministic function. 5.2 Randomised Response ϵitalic-ϵεϵ-ICM Proposed Defense To provide concrete bounds on the information leakage to a k-inferential adversary and demonstrate what an ϵitalic-ϵεϵ-ICM could look like, we construct an ϵitalic-ϵεϵ-ICM. Inspired by a differentially private mechanism proposed by Mangat (1994) to protect privacy of individuals during surveys, we propose a randomized response information censorship mechanism. Definition 5.4 (Randomised Response Mechanism). Let q∈[0,1]01q∈[0,1]q ∈ [ 0 , 1 ] and S⊂S ⊂ Y be a nonempty set of safe strings such as the empty string. The randomised response mechanism Mq:×P⁢()→P⁢():subscript→M_q:X× P(Y)→ P(Y)Mitalic_q : X × P ( Y ) → P ( Y ) is defined as: pM⁢((q,a))=t⁢p⁢(VicLLM⁢(q)=a)if ⁢a∈∖S(1−t)⁢1|S|if ⁢a∈SsubscriptcasesVicLLMif 11if p_M((q,a))= casestp( VicLLM(q)=a)&if a% S\\ (1-t) 1|S|&if a∈ S casespitalic_M ( ( q , a ) ) = start_ROW start_CELL t p ( VicLLM ( q ) = a ) end_CELL start_CELL if a ∈ Y ∖ S end_CELL end_ROW start_ROW start_CELL ( 1 - t ) divide start_ARG 1 end_ARG start_ARG | S | end_ARG end_CELL start_CELL if a ∈ S end_CELL end_ROW where pVicLLM⁢(y)subscriptVicLLMp_ VicLLM(y)pVicLLM ( y ) is the probability distribution of the victim model’s output. Theorem 5.5 (Randomised Response ϵitalic-ϵεϵ-ICM). Let Φ Φ be a collection of adversary priors. We assume there exists a nonempty safety set S⊂S ⊂ Y such that for any q∈Qq∈ Qq ∈ Q, pa⁢d⁢v∈ΦsubscriptΦp_adv∈ _a d v ∈ Φ, and s∈Ss∈ Ss ∈ S, IAq(pa⁢d⁢v(⋅|q);(q1,s))=0I_A_q(p_adv(·|q);(q_1,s))=0Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( q1 , s ) ) = 0. If we let tϵ=min⁡(ϵsuppa⁢d⁢v∈Φ;q1∈IAq(pa⁢d⁢v(⋅|q);Hqi1),1) t_ε= ( ε _ subarrayc% p_adv∈ ;\\ q_1 subarrayI_A_q(p_adv(·|q);H_q_i^1),1 )titalic_ϵ = min ( divide start_ARG ϵ end_ARG start_ARG supstart_ARG start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ ; end_CELL end_ROW start_ROW start_CELL q1 ∈ X end_CELL end_ROW end_ARG end_POSTSUBSCRIPT Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT1 ) end_ARG , 1 ) then the randomised response mechanism Mqϵsubscriptsubscriptitalic-ϵM_q_εMitalic_q start_POSTSUBSCRIPT ϵ end_POSTSUBSCRIPT is an ϵitalic-ϵεϵ-ICM. In order to satisfy the non-adaptive compositional bounds, it becomes evident that the probability of returning the model response can become quite low, thereby affecting utility for benign users. We now turn to establishing these tradeoff results. 5.3 Safety-Utility Trade-offs Defnining Utility While the primary concern of censorship is to ensure safety by mitigating Exp-IIL, model providers also care about the utility of their model for benign users. Given a distribution PsubscriptP_XPcaligraphic_X for the probability that a benign user prompts the VicLLM with x we can define the utility of an interaction between the VicLLM and the benign user as: y∼VicLLM⁢(x)⁢[u⁢(x,y)].subscriptsimilar-toVicLLMdelimited-[] _y VicLLM(x) [u(x,y) ].blackboard_Ey ∼ VicLLM ( x ) [ u ( x , y ) ] . Assuming that the safe responses y∈Sy∈ Sy ∈ S provide no utility for benign users, we easily find the utility cost of the Randomised Response ϵ−I⁢C⁢Mitalic-ϵε-ICMϵ - I C M Theorem 5.6 (Utility Bound for Randomised Response ϵitalic-ϵεϵ-ICM). For a given input x and utility function u⁢(x,y)u(x,y)u ( x , y ), the expected utility of an interaction where the outputs are given by the randomized response ϵitalic-ϵεϵ-ICM can be bounded as follows: y∼M⁢(x,VicLLM⁢(x))⁢[u⁢(x,y)]y∼VicLLM⁢(x)⁢[u⁢(x,y)]=qϵ,subscriptsimilar-toVicLLMdelimited-[]subscriptsimilar-toVicLLMdelimited-[]subscriptitalic-ϵ E_y M(x, VicLLM(x))[u(x,y)]E_y % VicLLM(x)[u(x,y)]=q_ε,divide start_ARG blackboard_Ey ∼ M ( x , VicLLM ( x ) ) [ u ( x , y ) ] end_ARG start_ARG blackboard_Ey ∼ VicLLM ( x ) [ u ( x , y ) ] end_ARG = qitalic_ϵ , (1) tϵsubscriptitalic-ϵt_εtitalic_ϵ is the probability of the mechanism returning a response from the Victim model. Thus, introducing the randomised response mechanism yields a model with utility qϵsubscriptitalic-ϵq_εqitalic_ϵ that of the uncensored model for any inputs for which the mechanism is employed. Inferential Users If we assume benign users are also inferential, i.e. they seek to learn information from model outputs about some task of quesiton x∗superscriptx^*x∗, then, for prior pu⁢s⁢r(⋅|x∗)p_usr(·|x^*)pitalic_u s r ( ⋅ | x∗ ), we can define their expected utility as: I(pu⁢s⁢r(⋅|x∗);(x,Y))=∑y∈Yp(x,y)∑a∈pu⁢s⁢r(a|(x,y),x∗)logpu⁢s⁢r⁢(a|(x,y),x∗)pu⁢s⁢r⁢(a|x∗). I(p_usr(·|x^*);(x,Y))= _y∈ Yp(x,y) _a∈% Yp_usr(a|(x,y),x^*) p_usr(a|(x,y),x^*)p_usr(a|% x^*).I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ) = ∑y ∈ Y p ( x , y ) ∑a ∈ Y pitalic_u s r ( a | ( x , y ) , x∗ ) log divide start_ARG pitalic_u s r ( a | ( x , y ) , x∗ ) end_ARG start_ARG pitalic_u s r ( a | x∗ ) end_ARG . For such inferential users, an upper bound on the possible utility for any input x can be derived for all possible ϵitalic-ϵεϵ-ICMs. We adapt results from Zamani et al. (2023) to upper bound the mutual information. Theorem 5.7 (Utility Bound for Inferential Users). For a benign user interested in query x∗superscriptx^*x∗ with prior pu⁢s⁢r(⋅|x∗)p_usr(·|x^*)pitalic_u s r ( ⋅ | x∗ ), a collection of adversary priors Φ Φ, set of malicious queries Q⊂Q ⊂ X, with the VicLLM employing an ϵitalic-ϵεϵ-ICM M, the utility for the benign inferential user interacting with the VicLLM employing and ϵitalic-ϵεϵ-ICM, is bounded by: I(pu⁢s⁢r(⋅|x∗);(x,Y)≤ϵ+infq∈Qpa⁢d⁢v∈Φ(ℋ(pu⁢s⁢r(⋅|x∗)|pa⁢d⁢v(⋅|q))+IAqc(pa⁢d⁢v(⋅|q);(x,Y))) I(p_usr(·|x^*);(x,Y)≤ε+ _ subarray% cq∈ Q\\ p_adv∈ subarray (H(p_usr(·|x^*)|p_adv(% ·|q))+I_A_q^c(p_adv(·|q);(x,Y)) )I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ≤ ϵ + infstart_ARG start_ROW start_CELL q ∈ Q end_CELL end_ROW start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ end_CELL end_ROW end_ARG end_POSTSUBSCRIPT ( H ( pitalic_u s r ( ⋅ | x∗ ) | pitalic_a d v ( ⋅ | q ) ) + Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) ) where AqcsuperscriptsubscriptA_q^cAitalic_qitalic_c the complement of AqsubscriptA_qAitalic_q and Y=M⁢(x,VicLLM⁢(x))VicLLMY=M(x, VicLLM(x))Y = M ( x , VicLLM ( x ) ). Interpretation Thus, the utility can only be high if the useful knowledge pu⁢s⁢r(⋅|x∗)p_usr(·|x^*)pitalic_u s r ( ⋅ | x∗ ) is not specified by pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ), or, the interaction is such that the mechanism M makes IAqc(pa⁢d⁢v(⋅|q);(x,Y))I_A_q^c(p_adv(·|q);(x,Y))Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) high, in other words making the adversary more confident in a permissible belief. While ℋ(pu⁢s⁢r(⋅|x∗)|pa⁢d⁢v(⋅|q))H(p_usr(·|x^*)|p_adv(·|q))H ( pitalic_u s r ( ⋅ | x∗ ) | pitalic_a d v ( ⋅ | q ) ) does not depend on the mechanism employed, an ϵitalic-ϵεϵ-ICM may be able to provide more utility if the responses returned make adversaries more confident in permissible conclusions. 6 Related work and Discussion 6.1 Comparison to Jailbreaks The fundamental distinction between decomposition attacks and jailbreaks is the adversary’s objective. While DAs share resemblance to methods such as payload splitting (Kang et al., 2023; Li et al., 2024c) or multi-turn attacks (Russinovich et al., 2024; Li et al., 2024a), these methods still rely on a victim model to produce problematic outputs; output censorship methods could greatly limit their effectiveness. For example, payload splitting is an input obfuscation method which splits up harmful words or strings into substrings which are then assigned to symbolic variables and an equation containing these variables. The model is then asked the prompt, with equations substituted for the sensitive words. Thus, payload splitting utilizes the programmatic capabilities of LLMs in order to mitigate the ability to detect harmfulness or malicious intent in the input, however, the outputs provided remain impermissible and open to output censorship mechanisms. Multi-turn jailbreaks (Russinovich et al., 2024; Li et al., 2024a) bypass defenses by gradual escalation from benign prompt and response interactions to problematic ones. These attacks are related to many-shot jailbreaks (Anil et al., 2024), leveraging an extended context window to alter model behavior. Nevertheless, once again, the method aims to force the victim model to produce a strictly impermissible output, which could more easily be blocked by output filtering mechanisms and may not even assist the adversary in fulfilling their malicious goal. 6.2 Relating inferential adversaries to privacy literature Inferential adversaries have been studied before in the context of privacy, such as model stealing adversaries (Tramèr et al., 2016) and membership inference adversaries (Shokri et al., 2017). Model stealing adversaries utilize carefully crafted queries and certain compositional properties of these queries and their outputs to infer specific parameters (Shamir et al., 2023). The black-box search for adversarial examples can also be viewed as being performed by an inferential adversary, as the input queries are not directly causing a model failure but leak information about gradients (Ilyas et al., 2018). By presenting inferential adversaries as a compelling threat model in the context of AI safety, we establish parallels between the problems of AI safety and privacy. Initial attempts to address concerns of privacy in data often focused on direct anonymization, such as removing personally identifiable information (PII) from datasets (Sweeney, 2002a). These approaches operated under the assumption that by simply removing identifiers and forcing indistinguishability within a dataset would be sufficient for making individuals in the dataset unidentifiable (Sweeney, 2002b). However, such approaches were insufficient for actually ensuring privacy because they did not account for the compositional nature of data. It was found that individuals could still be identified when certain combinations of attributes were unique especially given background information provided by auxiliary sources (Ganta et al., 2008; Narayanan and Shmatikov, 2008). Similarly, in the context of LLM safety, we argue that simply censoring or filtering direct responses does not address the underlying issue that an adversary can piece together sensitive information as part of a compositional attack. To address compositional attacks, Differential Privacy was proposed as a mathematical framework which defined and sought to bound the marginal risk to an individual’s privacy when releasing data (Dwork, 2006). These guarantees are achieved by adding a controlled amount of noise to the data or to responses made to queries so as to ensure that any single individual’s data does not significantly influence the outcome. Further definitions sought to generalize and extend privacy definitions in terms of information leakage (Nuradha and Goldfeld, 2023; Grosse et al., 2024; Bloch et al., 2021). We hope that through introducing the inferential adversary threat model in the context of safety, we can inspire a paradigm shift similar to that which occurred for privacy for the field of AI safety. By showing that safety concerns can be similarly framed and that existing defense approaches are vulnerable to composition attacks leveraging multiple interactions and background knowledge, our work helps introduce a theoretically rigorous way for further assessing and establishing safety. 7 Conclusion We identify a key safety vulnerability of deployed LLMs in terms of impermissible information leakage. We contrast with existing work which treats the ability of LLMs to provide harmful information to adversaries as a problem of model robustness, highlighting that this perspective is too narrow. Although current robustness issues make it easy for security adversaries to get a desired result, as defenses improve such direct attacks may prove more difficult. However, robustness provides a false sense of safety. The fundamental issue is that knowledge is compositional, interdependent, and dual-use. Directly censoring certain responses to questions does not guarantee that an adversary cannot reconstruct impermissible knowledge. Consequently, we conclude that evaluation methods and defenses must consider the potential for model responses to help adversaries infer dangerous information rather than determine whether or not model outputs themselves are dangerous. Nevertheless, we highlight that proper defenses will inevitably sacrifice utility for benign users. 8 Acknowledgements We want to thank Cleverhans lab members Sierra Wyllie, Anvith Thudi, Mohammad Yaghini, and Stephan Rabanser, as well as Claas Voelcker, Roger Grosse, Erik Jones, Lev McKinney, Xander Davies, and Darija Barak for their helpful feedback. We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, Meta, NSERC through the Discovery Grant and an Alliance Grant with ServiceNow and DRDC, the Ontario Early Researcher Award, the Schmidt Sciences foundation through the AI2050 Early Career Fellow program, and the Sloan Foundation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. References ant (2024) Introducing Claude 3.5 Sonnet — anthropic.com. https://w.anthropic.com/news/claude-3-5-sonnet, 2024. [Accessed 30-09-2024]. Anil et al. (2024) C. Anil, E. Durmus, M. Sharma, J. Benton, S. Kundu, J. Batson, N. Rimsky, M. Tong, J. Mu, D. Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024. Bai et al. (2022a) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. Bai et al. (2022b) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Bloch et al. (2021) M. Bloch, O. Günlü, A. Yener, F. Oggier, H. V. Poor, L. Sankar, and R. F. Schaefer. An overview of information-theoretic security and privacy: Metrics, limits and applications. IEEE Journal on Selected Areas in Information Theory, 2(1):5–22, 2021. Bubeck et al. (2023) S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. Chao et al. (2023) P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023. Debenedetti et al. (2023) E. Debenedetti, G. Severi, N. Carlini, C. A. Choquette-Choo, M. Jagielski, M. Nasr, E. Wallace, and F. Tramèr. Privacy side channels in machine learning systems. arXiv preprint arXiv:2309.05610, 2023. Durmus et al. (2024) E. Durmus, L. Lovitt, A. Tamkin, S. Ritchie, J. Clark, and D. Ganguli. Measuring the persuasiveness of language models, 2024. URL https://w.anthropic.com/news/measuring-model-persuasiveness. Dwork (2006) C. Dwork. Differential privacy. In M. Bugliesi, B. Preneel, V. Sassone, and I. Wegener, editors, Automata, Languages and Programming, pages 1–12, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-35908-1. Feffer et al. (2024) M. Feffer, A. Sinha, Z. C. Lipton, and H. Heidari. Red-teaming for generative ai: Silver bullet or security theater?, 2024. Ganta et al. (2008) S. R. Ganta, S. P. Kasiviswanathan, and A. Smith. Composition attacks and auxiliary information in data privacy. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, page 265–273, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 9781605581934. doi: 10.1145/1401890.1401926. URL https://doi.org/10.1145/1401890.1401926. Glukhov et al. (2023) D. Glukhov, I. Shumailov, Y. Gal, N. Papernot, and V. Papyan. LLM censorship: A machine learning challenge or a computer security problem? arXiv preprint arXiv:2307.10719, 2023. Grosse et al. (2024) L. Grosse, S. Saeidian, P. Sadeghi, T. J. Oechtering, and M. Skoglund. Quantifying privacy via information density, 2024. Ilyas et al. (2018) A. Ilyas, L. Engstrom, A. Athalye, and J. Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pages 2137–2146. PMLR, 2018. Jiang et al. (2023) D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023. Kang et al. (2023) D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks, 2023. Kapoor et al. (2024) S. Kapoor, R. Bommasani, K. Klyman, S. Longpre, A. Ramaswami, P. Cihon, A. Hopkins, K. Bankston, S. Biderman, M. Bogen, et al. On the societal impact of open foundation models. 2024. Khot et al. (2022) T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. Li et al. (2024a) N. Li, Z. Han, I. Steneker, W. Primack, R. Goodside, H. Zhang, Z. Wang, C. Menghini, and S. Yue. Llm defenses are not robust to multi-turn human jailbreaks yet, 2024a. URL https://arxiv.org/abs/2408.15221. Li et al. (2024b) N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Liu, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, R. Kaplan, I. Steneker, D. Campbell, B. Jokubaitis, A. Levinson, J. Wang, W. Qian, K. K. Karmakar, S. Basart, S. Fitz, M. Levine, P. Kumaraguru, U. Tupakula, V. Varadharajan, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024b. Li et al. (2024c) X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914, 2024c. Liu et al. (2023) Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023. Llama Team (2024) A. . M. Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Mangat (1994) N. S. Mangat. An improved randomized response strategy. Journal of the Royal Statistical Society: Series B (Methodological), 56(1):93–95, 1994. Narayanan and Shmatikov (2008) A. Narayanan and V. Shmatikov. Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (sp 2008), pages 111–125, 2008. doi: 10.1109/SP.2008.33. Nuradha and Goldfeld (2023) T. Nuradha and Z. Goldfeld. Pufferfish privacy: An information-theoretic study, 2023. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. Phuong et al. (2024) M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.13793, 2024. Radhakrishnan et al. (2023) A. Radhakrishnan, K. Nguyen, A. Chen, C. Chen, C. Denison, D. Hernandez, E. Durmus, E. Hubinger, J. Kernion, K. Lukošiūtė, et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768, 2023. Rafailov et al. (2024) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. Reid et al. (2024) M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. Russinovich et al. (2024) M. Russinovich, A. Salem, and R. Eldan. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack, 2024. Shamir et al. (2023) A. Shamir, I. Canales-Martinez, A. Hambitzer, J. Chavez-Saab, F. Rodrigez-Henriquez, and N. Satpute. Polynomial time cryptanalytic extraction of neural network models. arXiv preprint arXiv:2310.08708, 2023. Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017. Slattery et al. (2024) P. Slattery, A. K. Saeri, E. A. Grundy, J. Graham, M. Noetel, R. Uuk, J. Dao, S. Pour, S. Casper, and N. Thompson. The ai risk repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv preprint arXiv:2408.12622, 2024. Souly et al. (2024) A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer. A strongreject for empty jailbreaks, 2024. Sweeney (2002a) L. Sweeney. k-anonymity: a model for protecting privacy. Int. J. Uncertain. Fuzziness Knowl.-Based Syst., 10(5):557–570, oct 2002a. ISSN 0218-4885. doi: 10.1142/S0218488502001648. URL https://doi.org/10.1142/S0218488502001648. Sweeney (2002b) L. Sweeney. k-anonymity: A model for protecting privacy. International journal of uncertainty, fuzziness and knowledge-based systems, 10(05):557–570, 2002b. Tramèr et al. (2016) F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Stealing machine learning models via prediction \APIs\. In 25th USENIX security symposium (USENIX Security 16), pages 601–618, 2016. Weidinger et al. (2022) L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh, C. Biles, S. Brown, Z. Kenton, W. Hawkins, T. Stepleton, A. Birhane, L. A. Hendricks, L. Rimell, W. Isaac, J. Haas, S. Legassick, G. Irving, and I. Gabriel. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 214–229, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531146.3533088. URL https://doi.org/10.1145/3531146.3533088. Zamani et al. (2023) A. Zamani, T. J. Oechtering, and M. Skoglund. On the privacy-utility trade-off with and without direct access to the private data. IEEE Transactions on Information Theory, 2023. Zou et al. (2024) A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks. Improving alignment and robustness with circuit breakers, 2024. Appendix A Proofs As a few preliminaries for proving our results, we re-derive properties of expected impermissible information gain that match those of Mutual Information. For an q∈Qq∈ Qq ∈ Q, we define a variant of Entropy ℋAq(pa⁢d⁢v(⋅|q))=−∑a∈Aqpa⁢d⁢v(a|q)logpa⁢d⁢v(a|q) _A_q(p_adv(·|q))=- _a∈ A_qp_adv(a% |q) p_adv(a|q)Hitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ) = - ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | q ) log pitalic_a d v ( a | q ) and Conditional Entropy, conditioned on a knowledge pile distribution HqiksubscriptsuperscriptsubscriptH^k_q_iHitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT provided subquestions qii=1ksuperscriptsubscriptsubscript1\q_i\_i=1^k qitalic_i i = 1k ℋAq(pa⁢d⁢v(⋅|q)|Hqik)=−∑a1,…,ak∈kpM(Hqik=hk)∑a∈Aqpa⁢d⁢v(a|q,hk)logpa⁢d⁢v(a|q,hk) _A_q(p_adv(·|q)|H^k_q_i)=- _\a_1% ,…,a_k\ ^kp_M(H^k_q_i=h^k) _a∈ A_qp% _adv(a|q,h^k) p_adv(a|q,h^k)Hitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) | Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = - ∑ a start_POSTSUBSCRIPT 1 , … , aitalic_k ∈ Yitalic_k end_POSTSUBSCRIPT pitalic_M ( Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT = hitalic_k ) ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | q , hitalic_k ) log pitalic_a d v ( a | q , hitalic_k ) resulting in the standard relationship IAq(pa⁢d⁢v(⋅|q);Hqik)=ℋAq(pa⁢d⁢v(⋅|q)−ℋAq(pa⁢d⁢v(⋅|q)|Hqik)I_A_q(p_adv(·|q);H^k_q_i)=H_A_q(p_adv(·|q)% -H_A_q(p_adv(·|q)|H^k_q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = Hitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) - Hitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) | Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ). Non-negativity of IAq(pa⁢d⁢v(⋅|q);Hqik)I_A_q(p_adv(·|q);H^k_q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ): Denoting p⁢(Aq)=∑a∈Aqpa⁢d⁢v⁢(a|q)subscriptsubscriptsubscriptsubscriptconditionalp(A_q)= _a∈ A_qp_adv(a|q)p ( Aitalic_q ) = ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | q ), Jensen’s inequality gives us that IAq(pa⁢d⁢v(⋅|q);Hqik)=−p(Aq)(∑a∈Aq∑hk∈kpj⁢o⁢i⁢n⁢t⁢(a,hk|q)p⁢(Aq)logpa⁢d⁢v⁢(a|q)⁢pM⁢(Hqik=hk)pj⁢o⁢i⁢n⁢t⁢(a,hk|q)) I_A_q(p_adv(·|q);H^k_q_i)=-p(A_q)( _a∈ A% _q _h^k ^k p_joint(a,h^k|q)p(A_q) % p_adv(a|q)p_M(H^k_q_i=h^k)p_joint(a,h^k|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = - p ( Aitalic_q ) ( ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑hitalic_k ∈ Yitalic_k divide start_ARG pitalic_j o i n t ( a , hitalic_k | q ) end_ARG start_ARG p ( Aitalic_q ) end_ARG log divide start_ARG pitalic_a d v ( a | q ) pitalic_M ( Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT = hitalic_k ) end_ARG start_ARG pitalic_j o i n t ( a , hitalic_k | q ) end_ARG ) ≥−p⁢(Aq)⁢(log⁡(∑a∈Aq∑hk∈kpj⁢o⁢i⁢n⁢t⁢(a,hk|q)p⁢(Aq)⁢pa⁢d⁢v⁢(a|q)⁢pM⁢(Hqik=hk)pj⁢o⁢i⁢n⁢t⁢(a,hk|q)))absentsubscriptsubscriptsubscriptsubscriptsuperscriptℎsuperscriptsubscriptconditionalsuperscriptℎsubscriptsubscriptconditionalsubscriptsubscriptsuperscriptsubscriptsuperscriptℎsubscriptconditionalsuperscriptℎ ≥-p(A_q)( ( _a∈ A_q _h^k ^k% p_joint(a,h^k|q)p(A_q) p_adv(a|q)p_M(H^k_q_i=h^% k)p_joint(a,h^k|q)))≥ - p ( Aitalic_q ) ( log ( ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑hitalic_k ∈ Yitalic_k divide start_ARG pitalic_j o i n t ( a , hitalic_k | q ) end_ARG start_ARG p ( Aitalic_q ) end_ARG divide start_ARG pitalic_a d v ( a | q ) pitalic_M ( Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT = hitalic_k ) end_ARG start_ARG pitalic_j o i n t ( a , hitalic_k | q ) end_ARG ) ) =−p⁢(Aq)⁢log⁡(∑a∈Aq∑hk∈kpa⁢d⁢v⁢(a|q)⁢pM⁢(Hqik=hk)p⁢(Aq))absentsubscriptsubscriptsubscriptsubscriptsuperscriptℎsuperscriptsubscriptconditionalsubscriptsubscriptsuperscriptsubscriptsuperscriptℎsubscript =-p(A_q) ( _a∈ A_q _h^k ^k% p_adv(a|q)p_M(H^k_q_i=h^k)p(A_q))= - p ( Aitalic_q ) log ( ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑hitalic_k ∈ Yitalic_k divide start_ARG pitalic_a d v ( a | q ) pitalic_M ( Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT = hitalic_k ) end_ARG start_ARG p ( Aitalic_q ) end_ARG ) =−p⁢(Aq)⁢log⁡(∑a∈Aqpa⁢d⁢v⁢(a|q)p⁢(Aq))absentsubscriptsubscriptsubscriptsubscriptconditionalsubscript =-p(A_q) ( _a∈ A_q p_adv(a|q)p(A_q))= - p ( Aitalic_q ) log ( ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT divide start_ARG pitalic_a d v ( a | q ) end_ARG start_ARG p ( Aitalic_q ) end_ARG ) =−p⁢(Aq)⁢log⁡(1)=0absentsubscript10 =-p(A_q) (1)=0= - p ( Aitalic_q ) log ( 1 ) = 0 where pj⁢o⁢i⁢n⁢tsubscriptp_jointpitalic_j o i n t is the joint distribution over adversary beliefs and knowledge piles. Chain rule of Impermissible Information: We abstract a bit away from the specific distributions examined to define a chain rule for impermissible information. Specifically, given q∈Qq∈ Qq ∈ Q, and Aq∈subscriptA_q _q ∈ Y, let A, B, and C be random variables. Then, IAq⁢(C;A,B)=IAq⁢(C;A)+IAq⁢(C;B|A)subscriptsubscriptsubscriptsubscriptsubscriptsubscriptconditionalI_A_q(C;A,B)=I_A_q(C;A)+I_A_q(C;B|A)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A , B ) = Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; B | A ) The LHS can be expressed as IAq⁢(C;A,B)=∑c∈Aq∑a,bp⁢(a,b,c)⁢log⁡p⁢(a,b,c)p⁢(c)⁢p⁢(a,b)subscriptsubscriptsubscriptsubscriptsubscript I_A_q(C;A,B)= _c∈ A_q _a,bp(a,b,c) p(% a,b,c)p(c)p(a,b)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A , B ) = ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( a , b , c ) log divide start_ARG p ( a , b , c ) end_ARG start_ARG p ( c ) p ( a , b ) end_ARG whereas on the RHS IAq⁢(C;A)=∑c∈Aq∑ap⁢(a,c)⁢log⁡p⁢(a,c)p⁢(a)⁢p⁢(c)subscriptsubscriptsubscriptsubscriptsubscript I_A_q(C;A)= _c∈ A_q _ap(a,c) p(a,c)% p(a)p(c)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A ) = ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a p ( a , c ) log divide start_ARG p ( a , c ) end_ARG start_ARG p ( a ) p ( c ) end_ARG and IAq⁢(C|A;B)subscriptsubscriptconditional I_A_q(C|A;B)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C | A ; B ) =∑c∈Aq∑a,bp⁢(c,a,b)⁢log⁡p⁢(c,a,b)⁢p⁢(a)p⁢(c,a)⁢p⁢(b,a)absentsubscriptsubscriptsubscript = _c∈ A_q _a,bp(c,a,b) p(c,a,b)p(a)p(c,a% )p(b,a)= ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( c , a , b ) log divide start_ARG p ( c , a , b ) p ( a ) end_ARG start_ARG p ( c , a ) p ( b , a ) end_ARG =∑c∈Aq∑a,bp⁢(c,a,b)⁢log⁡p⁢(c,a,b)p⁢(c,a)⁢p⁢(b|a)absentsubscriptsubscriptsubscriptconditional = _c∈ A_q _a,bp(c,a,b) p(c,a,b)p(c,a)p(b% |a)= ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( c , a , b ) log divide start_ARG p ( c , a , b ) end_ARG start_ARG p ( c , a ) p ( b | a ) end_ARG Thus, IAq⁢(C;A)+IAq⁢(C;B|A)subscriptsubscriptsubscriptsubscriptconditional I_A_q(C;A)+I_A_q(C;B|A)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; B | A ) =∑c∈Aq∑a,bp⁢(c,a,b)⁢(log⁡p⁢(c,a)p⁢(c)⁢p⁢(a)+log⁡p⁢(c,a,b)p⁢(c,a)⁢p⁢(b|a))absentsubscriptsubscriptsubscriptconditional = _c∈ A_q _a,bp(c,a,b) ( p(c,a)p(c)p% (a)+ p(c,a,b)p(c,a)p(b|a) )= ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( c , a , b ) ( log divide start_ARG p ( c , a ) end_ARG start_ARG p ( c ) p ( a ) end_ARG + log divide start_ARG p ( c , a , b ) end_ARG start_ARG p ( c , a ) p ( b | a ) end_ARG ) =∑c∈Aq∑a,bp⁢(c,a,b)⁢log⁡p⁢(c,a,b)p⁢(b|a)⁢p⁢(c)⁢p⁢(a)absentsubscriptsubscriptsubscriptconditional = _c∈ A_q _a,bp(c,a,b) p(c,a,b)p(b|a)p(c% )p(a)= ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( c , a , b ) log divide start_ARG p ( c , a , b ) end_ARG start_ARG p ( b | a ) p ( c ) p ( a ) end_ARG =∑c∈Aq∑a,bp⁢(c,a,b)⁢log⁡p⁢(c,a,b)p⁢(c)⁢p⁢(a,b)absentsubscriptsubscriptsubscript = _c∈ A_q _a,bp(c,a,b) p(c,a,b)p(c)p(a,b)= ∑c ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ∑a , b p ( c , a , b ) log divide start_ARG p ( c , a , b ) end_ARG start_ARG p ( c ) p ( a , b ) end_ARG =IAq⁢(C;A,B)absentsubscriptsubscript =I_A_q(C;A,B)= Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; A , B ) establishing the chain rule for our variant of mutual information. This argument can be further generalized by induction to provide IAq⁢(C;X1,…,Xn)=∑i=1nIAq⁢(C|X1,…,Xi−1;Xi)subscriptsubscriptsubscript1…subscriptsuperscriptsubscript1subscriptsubscriptconditionalsubscript1…subscript1subscript I_A_q(C;X_1,…,X_n)= _i=1^nI_A_q(C|X_1,% …,X_i-1;X_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C ; X1 , … , Xitalic_n ) = ∑i = 1n Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( C | X1 , … , Xitalic_i - 1 ; Xitalic_i ) Data Processing Inequality: If an adversary applies some post processing to the knowledge pile signal HqiksubscriptsuperscriptsubscriptH^k_q_iHitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT to get Hqik′subscriptsuperscriptsuperscript′subscriptH^k _q_iHitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT, then Hqik′subscriptsuperscriptsuperscript′subscriptH^k _q_iHitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT is conditionally independent of pa⁢d⁢v(⋅|q)p_adv(·|q)pitalic_a d v ( ⋅ | q ) given HqiksubscriptsuperscriptsubscriptH^k_q_iHitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT. From the chain rule, we have that IAq(pa⁢d⁢v(⋅|q);Hqik,Hqik′)=IAq(pa⁢d⁢v(⋅|q);Hqik)+IAq(pa⁢d⁢v(⋅|q);Hqik′|Hqik) I_A_q(p_adv(·|q);H^k_q_i,H^k _q_i)% =I_A_q(p_adv(·|q);H^k_q_i)+I_A_q(p_adv(·|q);H^k^% _q_i|H^k_q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) and IAq(pa⁢d⁢v(⋅|q);Hqik,Hqik′)=IAq(pa⁢d⁢v(⋅|q);Hqik′)+IAq(pa⁢d⁢v(⋅|q);Hqik|Hqik′) I_A_q(p_adv(·|q);H^k_q_i,H^k _q_i)% =I_A_q(p_adv(·|q);H^k _q_i)+I_A_q(p_adv(·|q)% ;H^k_q_i|H^k _q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) Conditional independence implies that IAq(pa⁢d⁢v(⋅|q);Hqik′|Hqik)=0I_A_q(p_adv(·|q);H^k _q_i|H^k_q_i)=0Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) = 0, and, non-negativity implies that IAq(pa⁢d⁢v(⋅|q);Hqik′|Hqik′)I_A_q(p_adv(·|q);H^k _q_i|H^k _q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT | Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ), thus, IAq(pa⁢d⁢v(⋅|q);Hqik)≥IAq(pa⁢d⁢v(⋅|q);Hqik′)I_A_q(p_adv(·|q);H^k_q_i)≥ I_A_q(p_adv(·|q);H^k^% _q_i)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_kitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) ≥ Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_k start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTq start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) A.1 Proof of Theorem 5.3 Proof. Let q be a malicious query, ϵ>0italic-ϵ0ε>0ϵ > 0 a leakage bound, and k the number of possible interactions between AdvLLM and VicLLM mediated by an ϵitalic-ϵεϵ-ICM M. For ease of notation, let yi=(qi,M(qi,VicLLM(qi))y_i=(q_i,M(q_i, VicLLM(q_i))yitalic_i = ( qitalic_i , M ( qitalic_i , VicLLM ( qitalic_i ) ). For any set of k questions qii=1ksuperscriptsubscriptsubscript1\q_i\_i=1^k qitalic_i i = 1k, and history distribution Hqik:=qi,ai)i=1kH_q_i^k:=\q_i,a_i)\_i=1^kHitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk := qitalic_i , aitalic_i ) i = 1k, we prove by induction that IAq(pa⁢d⁢v(⋅|q);Hqik)≤∑i=1kϵi+∑j=2kIAq(yj;Hqij−1|pa⁢d⁢v(⋅|q)) I_A_q(p_adv(·|q);H_q_i^k)≤ _i=1^k% _i+ _j=2^kI_A_q(y_j;H_q_i^j-1|p_adv(·|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTk ) ≤ ∑i = 1k ϵitalic_i + ∑j = 2k Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_j ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTj - 1 | pitalic_a d v ( ⋅ | q ) ) when IAq(pa⁢d⁢v(⋅|q);(qj,aj))≤ϵI_A_q(p_adv(·|q);(q_j,a_j))≤ _A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( qitalic_j , aitalic_j ) ) ≤ ϵ. IAq((qj,aj);Hqij−1|pa⁢d⁢v(⋅|q))= I_A_q((q_j,a_j);H_q_i^j-1|p_adv(·|q))=Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( ( qitalic_j , aitalic_j ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTj - 1 | pitalic_a d v ( ⋅ | q ) ) = ∑a∈Aqpa⁢d⁢v⁢(a|q)⁢∑y1,…,yj∈pM⁢(y1,…,yj|(q,a))⁢log⁡pM⁢(y1,…,yj|(q,a))pM⁢(y1,…,yj−1|(q,a))⁢pM⁢(yj|(q,a))subscriptsubscriptsubscriptconditionalsubscriptsubscript1…subscriptsubscriptsubscript1…conditionalsubscriptsubscriptsubscript1…conditionalsubscriptsubscriptsubscript1…conditionalsubscript1subscriptconditionalsubscript _a∈ A_qp_adv(a|q) _y_1,…,y_j % p_M(y_1,…,y_j|(q,a)) p_M(y_1,…,y_j|(q,a))p_M% (y_1,…,y_j-1|(q,a))p_M(y_j|(q,a))∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | q ) ∑y start_POSTSUBSCRIPT 1 , … , yitalic_j ∈ Y end_POSTSUBSCRIPT pitalic_M ( y1 , … , yitalic_j | ( q , a ) ) log divide start_ARG pitalic_M ( y1 , … , yitalic_j | ( q , a ) ) end_ARG start_ARG pitalic_M ( y1 , … , yitalic_j - 1 | ( q , a ) ) pitalic_M ( yitalic_j | ( q , a ) ) end_ARG For k=11k=1k = 1, we trivially have that IAq(pa⁢d⁢v(⋅|q);Hq11)≤ϵ≤ϵ+ηI_A_q(p_adv(·|q);H_q_1^1)≤ε≤ε+ _A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT1 ) ≤ ϵ ≤ ϵ + η as η is non-negative. Assume then that IAq(pa⁢d⁢v(⋅|q);Hqim)≤mϵ+ηm I_A_q(p_adv(·|q);H_q_i^m)≤ mε+ _mIitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) ≤ m ϵ + ηitalic_m for ηm=∑j=2mIAq(yj;Hqij−1|pa⁢d⁢v(⋅|q)) _m= _j=2^mI_A_q(y_j;H_q_i^j-1|p_adv(·|q))ηitalic_m = ∑j = 2m Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_j ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTj - 1 | pitalic_a d v ( ⋅ | q ) ). Then, IAq(pa⁢d⁢v(⋅|q);Hqim+1)=IAq(pa⁢d⁢v(⋅|q);Hqim)+IAq(pa⁢d⁢v(⋅|q);(qm+1,am+1|Hqim) I_A_q(p_adv(·|q);H_q_i^m+1)=I_A_q(p_adv(% ·|q);H_q_i^m)+I_A_q(p_adv(·|q);(q_m+1,a_m+1|H_q_i^% m)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm + 1 ) = Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( qitalic_m + 1 , aitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) ≤∑i=1mϵi+ηm+IAq(pa⁢d⁢v(⋅|q);(qm+1,am+1|Hqim) ≤ _i=1^m _i+ _m+I_A_q(p_adv(·|q% );(q_m+1,a_m+1|H_q_i^m)≤ ∑i = 1m ϵitalic_i + ηitalic_m + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( qitalic_m + 1 , aitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) To show our desired result, we need to show that IAq(pa⁢d⁢v(⋅|q);ym+1|Hqim)≤IAq(pa⁢d⁢v(⋅|q);ym+1)+IAq(ym+1;Hqim|pa⁢d⁢v(⋅|q)) I_A_q(p_adv(·|q);y_m+1|H_q_i^m)≤ I_A_q(% p_adv(·|q);y_m+1)+I_A_q(y_m+1;H_q_i^m|p_adv(·|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) ≤ Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm | pitalic_a d v ( ⋅ | q ) ) For IAq(pa⁢d⁢v(⋅|q);ym+1|Hqim)I_A_q(p_adv(·|q);y_m+1|H_q_i^m)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ): IAq(pa⁢d⁢v(⋅|q);ym+1|Hqim)= I_A_q(p_adv(·|q);y_m+1|H_q_i^m)=Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) = ∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢pM⁢(hm)pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pM⁢(hm+1)subscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎsubscriptsuperscriptℎconditionalsubscriptsuperscriptℎ1 _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) p_% joint(h^m,a,y_m+1|q)p_M(h^m)p_joint(h^m,a|q)p_M(h^m+1)∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log divide start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) pitalic_M ( hitalic_m ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_M ( hitalic_m + 1 ) end_ARG For IAq(ym+1;Hqim|pa⁢d⁢v(⋅|q))I_A_q(y_m+1;H_q_i^m|p_adv(·|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm | pitalic_a d v ( ⋅ | q ) ): IAq(ym+1;Hqim|pa⁢d⁢v(⋅|q))= I_A_q(y_m+1;H_q_i^m|p_adv(·|q))=Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm | pitalic_a d v ( ⋅ | q ) ) = ∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢pa⁢d⁢v⁢(a|q)pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pj⁢o⁢i⁢n⁢t⁢(ym+1,a|q)subscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptconditionalsubscriptsuperscriptℎconditionalsubscriptsubscript1conditional _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) p_% joint(h^m,a,y_m+1|q)p_adv(a|q)p_joint(h^m,a|q)p_joint(y_m+1,a% |q)∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log divide start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) pitalic_a d v ( a | q ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_j o i n t ( yitalic_m + 1 , a | q ) end_ARG For IAq(pa⁢d⁢v(⋅|q);ym+1)I_A_q(p_adv(·|q);y_m+1)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ): IAq(pa⁢d⁢v(⋅|q);ym+1)= I_A_q(p_adv(·|q);y_m+1)=Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ) = ∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡pj⁢o⁢i⁢n⁢t⁢(a,ym+1|q)pa⁢d⁢v⁢(a|q)⁢pM⁢(ym+1)subscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptconditionalsubscript1subscriptconditionalsubscriptsubscript1 _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) p_% joint(a,y_m+1|q)p_adv(a|q)p_M(y_m+1)∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log divide start_ARG pitalic_j o i n t ( a , yitalic_m + 1 | q ) end_ARG start_ARG pitalic_a d v ( a | q ) pitalic_M ( yitalic_m + 1 ) end_ARG Thus, IAq(pa⁢d⁢v(⋅|q);ym+1)+IAq(ym+1;hm|pa⁢d⁢v(⋅|q)) I_A_q(p_adv(·|q);y_m+1)+I_A_q(y_m+1;h^m|p_% adv(·|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; hitalic_m | pitalic_a d v ( ⋅ | q ) ) =∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡[pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢pa⁢d⁢v⁢(a|q)pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pj⁢o⁢i⁢n⁢t⁢(ym+1,a|q)⁢pj⁢o⁢i⁢n⁢t⁢(a,ym+1|q)pa⁢d⁢v⁢(a|q)⁢pM⁢(ym+1)]absentsubscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptconditionalsubscriptsuperscriptℎconditionalsubscriptsubscript1conditionalsubscriptconditionalsubscript1subscriptconditionalsubscriptsubscript1 = _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) [ p_% joint(h^m,a,y_m+1|q)p_adv(a|q)p_joint(h^m,a|q)p_joint(y_m+1,a% |q) p_joint(a,y_m+1|q)p_adv(a|q)p_M(y_m+1)]= ∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log [ divide start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) pitalic_a d v ( a | q ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_j o i n t ( yitalic_m + 1 , a | q ) end_ARG divide start_ARG pitalic_j o i n t ( a , yitalic_m + 1 | q ) end_ARG start_ARG pitalic_a d v ( a | q ) pitalic_M ( yitalic_m + 1 ) end_ARG ] =∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡[pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pM⁢(ym+1)]absentsubscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscriptsubscript1 = _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) [ p_% joint(h^m,a,y_m+1|q)p_joint(h^m,a|q)p_M(y_m+1)]= ∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log [ divide start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_M ( yitalic_m + 1 ) end_ARG ] and IAq(pa⁢d⁢v(⋅|q);ym+1)+IAq(ym+1;hm|pa⁢d⁢v(⋅|q))−IAq(pa⁢d⁢v(⋅|q);ym+1|Hqim) I_A_q(p_adv(·|q);y_m+1)+I_A_q(y_m+1;h^m|p_% adv(·|q))-I_A_q(p_adv(·|q);y_m+1|H_q_i^m)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; hitalic_m | pitalic_a d v ( ⋅ | q ) ) - Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) =∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡[pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pM⁢(ym+1)⁢pj⁢o⁢i⁢n⁢t⁢(hm,a|q)⁢pM⁢(hm+1)pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢pM⁢(hm)]absentsubscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎconditionalsubscriptsubscript1subscriptsuperscriptℎconditionalsubscriptsuperscriptℎ1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎ = _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) [ p_% joint(h^m,a,y_m+1|q)p_joint(h^m,a|q)p_M(y_m+1) p_joint(% h^m,a|q)p_M(h^m+1)p_joint(h^m,a,y_m+1|q)p_M(h^m)]= ∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log [ divide start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_M ( yitalic_m + 1 ) end_ARG divide start_ARG pitalic_j o i n t ( hitalic_m , a | q ) pitalic_M ( hitalic_m + 1 ) end_ARG start_ARG pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) pitalic_M ( hitalic_m ) end_ARG ] =∑hm,a,ym+1pj⁢o⁢i⁢n⁢t⁢(hm,a,ym+1|q)⁢log⁡[pM⁢(hm+1)pM⁢(ym+1)⁢pM⁢(hm)]absentsubscriptsuperscriptℎsubscript1subscriptsuperscriptℎconditionalsubscript1subscriptsuperscriptℎ1subscriptsubscript1subscriptsuperscriptℎ = _h^m,a,y_m+1p_joint(h^m,a,y_m+1|q) [ p_% M(h^m+1)p_M(y_m+1)p_M(h^m)]= ∑hitalic_m , a , y start_POSTSUBSCRIPT m + 1 end_POSTSUBSCRIPT pitalic_j o i n t ( hitalic_m , a , yitalic_m + 1 | q ) log [ divide start_ARG pitalic_M ( hitalic_m + 1 ) end_ARG start_ARG pitalic_M ( yitalic_m + 1 ) pitalic_M ( hitalic_m ) end_ARG ] =∑hmpM⁢(hm+1)⁢log⁡[pM⁢(hm+1)pM⁢(ym+1)⁢pM⁢(hm)]=I⁢(ym+1;hm)≥0absentsubscriptsuperscriptℎsubscriptsuperscriptℎ1subscriptsuperscriptℎ1subscriptsubscript1subscriptsuperscriptℎsubscript1superscriptℎ0 = _h^mp_M(h^m+1) [ p_M(h^m+1)p_M(y_m+% 1)p_M(h^m)]=I(y_m+1;h^m)≥ 0= ∑hitalic_m pitalic_M ( hitalic_m + 1 ) log [ divide start_ARG pitalic_M ( hitalic_m + 1 ) end_ARG start_ARG pitalic_M ( yitalic_m + 1 ) pitalic_M ( hitalic_m ) end_ARG ] = I ( yitalic_m + 1 ; hitalic_m ) ≥ 0 Finally, we conclude that IAq(pa⁢d⁢v(⋅|q);ym+1|Hqim)≤IAq(pa⁢d⁢v(⋅|q);ym+1)+IAq(ym+1;Hqim|pa⁢d⁢v(⋅|q))≤ϵ+IAq(ym+1;Hqim|pa⁢d⁢v(⋅|q)) I_A_q(p_adv(·|q);y_m+1|H_q_i^m)≤ I_A_q(% p_adv(·|q);y_m+1)+I_A_q(y_m+1;H_q_i^m|p_adv(·|q))% ≤ε+I_A_q(y_m+1;H_q_i^m|p_adv(·|q))Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 | Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm ) ≤ Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; yitalic_m + 1 ) + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm | pitalic_a d v ( ⋅ | q ) ) ≤ ϵ + Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( yitalic_m + 1 ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm | pitalic_a d v ( ⋅ | q ) ) and IAq(pa⁢d⁢v(⋅|q);Hqim+1)≤(m+1)ϵ+ηm+1I_A_q(p_adv(·|q);H_q_i^m+1)≤(m+1)ε+ _m+1Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPTm + 1 ) ≤ ( m + 1 ) ϵ + ηitalic_m + 1. ∎ A.2 Proof of Theorem 5.5 Proof. Let Φ Φ be a collection of adversary priors and ϵ>0italic-ϵ0ε>0ϵ > 0. We assume there exists a nonempty safety set S⊂S ⊂ Y such that for any q∈Qq∈ Qq ∈ Q, pa⁢d⁢v∈ΦsubscriptΦp_adv∈ _a d v ∈ Φ, and s∈Ss∈ Ss ∈ S, IAq(pa⁢d⁢v(⋅|q);(q1,s))=0I_A_q(p_adv(·|q);(q_1,s))=0Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( q1 , s ) ) = 0. Finally, let tϵ=min⁡(ϵsuppa⁢d⁢v∈Φ;q1∈IAq(pa⁢d⁢v(⋅|q);Hqi1),1) t_ε= ( ε _ subarrayc% p_adv∈ ;\\ q_1 subarrayI_A_q(p_adv(·|q);H_q_i^1),1 )titalic_ϵ = min ( divide start_ARG ϵ end_ARG start_ARG supstart_ARG start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ ; end_CELL end_ROW start_ROW start_CELL q1 ∈ X end_CELL end_ROW end_ARG end_POSTSUBSCRIPT Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT1 ) end_ARG , 1 ) Then, for any input q1∈subscript1q_1 1 ∈ X, we have IAq(pa⁢d⁢v(⋅|q);(q1,Hqi1)=∑a1∈pM((q1,a1)∑a∈Aqpa⁢d⁢v(a|a1,q)logpa⁢d⁢v⁢(a|a1,q)pa⁢d⁢v⁢(a|q) I_A_q(p_adv(·|q);(q_1,H_q_i^1)= _\a_1\% p_M((q_1,a_1) _a∈ A_qp_adv(a|a_1,q) % p_adv(a|a_1,q)p_adv(a|q)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( q1 , Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT1 ) = ∑ a start_POSTSUBSCRIPT 1 ∈ Y end_POSTSUBSCRIPT pitalic_M ( ( q1 , a1 ) ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | a1 , q ) log divide start_ARG pitalic_a d v ( a | a1 , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG If a1∈Ssubscript1a_1∈ Sa1 ∈ S, then by our assumption on S, log⁡pa⁢d⁢v⁢(a|a1,q)pa⁢d⁢v⁢(a|q)=0subscriptconditionalsubscript1subscriptconditional0 p_adv(a|a_1,q)p_adv(a|q)=0log divide start_ARG pitalic_a d v ( a | a1 , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG = 0, thus, IAq(pa⁢d⁢v(⋅|q);(q1,Hq11)=∑a1∈∖SpM((q1,a1)∑a∈Aqpa⁢d⁢v(a|a1,q)logpa⁢d⁢v⁢(a|a1,q)pa⁢d⁢v⁢(a|q) I_A_q(p_adv(·|q);(q_1,H_q_1^1)= _\a_1\% Sp_M((q_1,a_1) _a∈ A_qp_adv(a|a_1,% q) p_adv(a|a_1,q)p_adv(a|q)Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( q1 , Hitalic_q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT1 ) = ∑ a start_POSTSUBSCRIPT 1 ∈ Y ∖ S end_POSTSUBSCRIPT pitalic_M ( ( q1 , a1 ) ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | a1 , q ) log divide start_ARG pitalic_a d v ( a | a1 , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG =tϵ⁢∑a1∈∖Sp⁢(VicLLM⁢(q1)=a1)⁢∑a∈Aqpa⁢d⁢v⁢(a|a1,q)⁢log⁡pa⁢d⁢v⁢(a|a1,q)pa⁢d⁢v⁢(a|q)absentsubscriptitalic-ϵsubscriptsubscript1VicLLMsubscript1subscript1subscriptsubscriptsubscriptconditionalsubscript1subscriptconditionalsubscript1subscriptconditional =t_ε _\a_1\ Sp( % VicLLM(q_1)=a_1) _a∈ A_qp_adv(a|a_1,q) p_adv(a|a_% 1,q)p_adv(a|q)= titalic_ϵ ∑ a start_POSTSUBSCRIPT 1 ∈ Y ∖ S end_POSTSUBSCRIPT p ( VicLLM ( q1 ) = a1 ) ∑a ∈ A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT pitalic_a d v ( a | a1 , q ) log divide start_ARG pitalic_a d v ( a | a1 , q ) end_ARG start_ARG pitalic_a d v ( a | q ) end_ARG ≤tϵIAq(pa⁢d⁢v(⋅|q);Hqi1) ≤ t_εI_A_q(p_adv(·|q);H_q_i^1)≤ titalic_ϵ Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; Hitalic_q start_POSTSUBSCRIPT i end_POSTSUBSCRIPT1 ) ≤ϵabsentitalic-ϵ ≤ε≤ ϵ ∎ A.3 Proof of Theorem 5.6 Proof. y∼M⁢(x,VicLLM⁢(x))⁢[u⁢(x,y)]=∑y∈YpM⁢(y)⁢u⁢(x,y)subscriptsimilar-toVicLLMdelimited-[]subscriptsubscript _y M(x, VicLLM(x))[u(x,y)]= _y∈ Yp% _M(y)u(x,y)blackboard_Ey ∼ M ( x , VicLLM ( x ) ) [ u ( x , y ) ] = ∑y ∈ Y pitalic_M ( y ) u ( x , y ) =tϵ⁢∑y∈YpM⁢(y)⁢u⁢(x,y)+(1−tϵ)⁢y∼Unif⁢(S)⁢[u⁢(x,y)]absentsubscriptitalic-ϵsubscriptsubscript1subscriptitalic-ϵsubscriptsimilar-toUnifdelimited-[] =t_ε _y∈ Yp_M(y)u(x,y)+(1-t_ε)% E_y (S)[u(x,y)]= titalic_ϵ ∑y ∈ Y pitalic_M ( y ) u ( x , y ) + ( 1 - titalic_ϵ ) blackboard_Ey ∼ Unif ( S ) [ u ( x , y ) ] Assuming u⁢(x,y)=00u(x,y)=0u ( x , y ) = 0 for y∈Sy∈ Sy ∈ S, we can express the utility ratio: y∼M⁢(x,VicLLM⁢(x))⁢[u⁢(x,y)]y∼VicLLM⁢(x)⁢[u⁢(x,y)]=tϵ⁢∑y∈Yp⁢(y)⁢u⁢(x,y)+(1−tϵ)⋅0∑y∈Yp⁢(y)⁢u⁢(x,y)=tϵsubscriptsimilar-toVicLLMdelimited-[]subscriptsimilar-toVicLLMdelimited-[]subscriptitalic-ϵsubscript⋅1subscriptitalic-ϵ0subscriptsubscriptitalic-ϵ E_y M(x, VicLLM(x))[u(x,y)] % E_y VicLLM(x)[u(x,y)]= t_ε _y∈ Yp(y)u(x,% y)+(1-t_ε)· 0 _y∈ Yp(y)u(x,y)=t_εdivide start_ARG blackboard_Ey ∼ M ( x , VicLLM ( x ) ) [ u ( x , y ) ] end_ARG start_ARG blackboard_Ey ∼ VicLLM ( x ) [ u ( x , y ) ] end_ARG = divide start_ARG titalic_ϵ ∑y ∈ Y p ( y ) u ( x , y ) + ( 1 - titalic_ϵ ) ⋅ 0 end_ARG start_ARG ∑y ∈ Y p ( y ) u ( x , y ) end_ARG = titalic_ϵ ∎ A.4 Proof of Theorem 5.7 Proof. By the chain rule for mutual information: I(pu⁢s⁢r(⋅|x∗);(x,Y)) I(p_usr(·|x^*);(x,Y))I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ) =I(pa⁢d⁢v(⋅|q),pu⁢s⁢r(⋅|x∗);(x,Y))−I(pa⁢d⁢v(⋅|q);(x,Y)|pu⁢s⁢r(⋅|x∗)) =I(p_adv(·|q),p_usr(·|x^*);(x,Y))-I(p_adv(·|q% );(x,Y)|p_usr(·|x^*))= I ( pitalic_a d v ( ⋅ | q ) , pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ) - I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) | pitalic_u s r ( ⋅ | x∗ ) ) =I(pa⁢d⁢v(⋅|q);(x,Y))+I(pu⁢s⁢r(⋅|x∗);(x,Y)|pa⁢d⁢v(⋅|q)) =I(p_adv(·|q);(x,Y))+I(p_usr(·|x^*);(x,Y)|p_adv(% ·|q))= I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) + I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) | pitalic_a d v ( ⋅ | q ) ) −I(pa⁢d⁢v(⋅|q);(x,Y)|pu⁢s⁢r(⋅|x∗)) -I(p_adv(·|q);(x,Y)|p_usr(·|x^*))- I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) | pitalic_u s r ( ⋅ | x∗ ) ) ≤I(pu⁢s⁢r(⋅|x∗);(x,Y)|pa⁢d⁢v(⋅|q))+I(pa⁢d⁢v(⋅|q);(x,Y)) ≤ I(p_usr(·|x^*);(x,Y)|p_adv(·|q))+I(p_adv(% ·|q);(x,Y))≤ I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) | pitalic_a d v ( ⋅ | q ) ) + I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) For any malicious query q∈Qq∈ Qq ∈ Q and adversary prior pa⁢d⁢v∈ΦsubscriptΦp_adv∈ _a d v ∈ Φ, we can decompose I(pa⁢d⁢v(⋅|q);(x,Y))I(p_adv(·|q);(x,Y))I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) into information gain about impermissible AqsubscriptA_qAitalic_q and permissible AqcsuperscriptsubscriptA_q^cAitalic_qitalic_c conclusions: I(pa⁢d⁢v(⋅|q);(x,Y))=IAq(pa⁢d⁢v(⋅|q);(x,Y))+IAqc(pa⁢d⁢v(⋅|q);(x,Y))≤ϵ+IAqc(pa⁢d⁢v(⋅|q);(x,Y)) I(p_adv(·|q);(x,Y))=I_A_q(p_adv(·|q);(x,Y))+I_A% _q^c(p_adv(·|q);(x,Y))≤ε+I_A_q^c(p_adv(·|q);(% x,Y))I ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) = Iitalic_A start_POSTSUBSCRIPT q end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) + Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) ≤ ϵ + Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) Using the fact that I(pu⁢s⁢r(⋅|x∗);(x,Y)|pa⁢d⁢v(⋅|q))≤ℋ(pu⁢s⁢r(⋅|x∗)|pa⁢d⁢v(⋅|q))I(p_usr(·|x^*);(x,Y)|p_adv(·|q)) (p_usr(·|x^% *)|p_adv(·|q))I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) | pitalic_a d v ( ⋅ | q ) ) ≤ H ( pitalic_u s r ( ⋅ | x∗ ) | pitalic_a d v ( ⋅ | q ) ), we get: I(pu⁢s⁢r(⋅|x∗);(x,Y)) I(p_usr(·|x^*);(x,Y))I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ) ≤ℋ(pu⁢s⁢r(⋅|x∗)|pa⁢d⁢v(⋅|q))+IAqc(pa⁢d⁢v(⋅|q);(x,Y))+ϵ (p_usr(·|x^*)|p_adv(·|q))+I_A_q^% c(p_adv(·|q);(x,Y))+ε≤ H ( pitalic_u s r ( ⋅ | x∗ ) | pitalic_a d v ( ⋅ | q ) ) + Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) + ϵ Since this holds for any q∈Qq∈ Qq ∈ Q and pa⁢d⁢v∈ΦsubscriptΦp_adv∈ _a d v ∈ Φ, we can take the infimum: I(pu⁢s⁢r(⋅|x∗);(x,Y))≤ϵ+infq∈Qpa⁢d⁢v∈Φ(ℋ(pu⁢s⁢r(⋅|x∗)|pa⁢d⁢v(⋅|q))+IAqc(pa⁢d⁢v(⋅|q);(x,Y))) I(p_usr(·|x^*);(x,Y))≤ε+ _ subarray% cq∈ Q\\ p_adv∈ subarray (H(p_usr(·|x^*)|p_adv(% ·|q))+I_A_q^c(p_adv(·|q);(x,Y)) )I ( pitalic_u s r ( ⋅ | x∗ ) ; ( x , Y ) ) ≤ ϵ + infstart_ARG start_ROW start_CELL q ∈ Q end_CELL end_ROW start_ROW start_CELL p start_POSTSUBSCRIPT a d v ∈ Φ end_CELL end_ROW end_ARG end_POSTSUBSCRIPT ( H ( pitalic_u s r ( ⋅ | x∗ ) | pitalic_a d v ( ⋅ | q ) ) + Iitalic_A start_POSTSUBSCRIPT qitalic_c end_POSTSUBSCRIPT ( pitalic_a d v ( ⋅ | q ) ; ( x , Y ) ) ) ∎ Appendix B Decomposition Attack Algorithm Implementation details B.1 Experimental Details Dataset Selection and Curation: To evaluate DAs and compare them against jailbreaks under our proposed IIL metric, we chose the WMDP dataset [Li et al., 2024b], focusing on multiple-choice questions in biology and chemistry (WMDP-Bio and WMDP-Chem) as proxies for dangerous knowledge. To assess DAs’ ability to extract impermissible information despite filtering mechanisms, we used Llama-Guard-3-8B [Llama Team, 2024] to curate subsets of questions classified as unsafe, resulting in 35 and 64 questions for WMDP-Bio and WMDP-Chem, respectively. This curation ensures the all questions would trigger input filtering mechanisms if asked directly. Models Used Model Name Purpose Llama-3.1-70B-Instruct [Llama Team, 2024] Victim model (VicLLM) Prompt-Guard-86M [Llama Team, 2024] Jailbreak detection for Input Filtering (MisubscriptM_iMitalic_i) Llama-Guard-3-8B[Llama Team, 2024] Input and output filtering mechanism (MisubscriptM_iMitalic_i and MosubscriptM_oMitalic_o) Mistral-7B-Instruct-v0.3 [Jiang et al., 2023] Adversary LLM (AdvLLM); not safety alignmed Llama-3.1-8B-Instruct [Llama Team, 2024] Adversary LLM (AdvLLM); safety alignmed Table 3: Models and their purpose in the experiment. All sampling for generation is performed with temperature t=0.80.8t=0.8t = 0.8 and nucleus sampling with p=0.950.95p=0.95p = 0.95. Few-shot Examples: Some few-shot examples for subquestion generation EgsubscriptE_gEitalic_g and answer aggregation EasubscriptE_aEitalic_a were adapted from [Radhakrishnan et al., 2023] which focused on problem decomposition and aggregation for problem solving. The remaining few shot examples were generated using Claude 3.5 Sonnet [ant, 2024] to match the more technical multiple choice questions in the WMDP dataset. Hyperparameters: We run our proposed DA with k=22k=2k = 2 rounds and m=33m=3m = 3 subquestions per round. To reflect more realistic use cases and remove outliers, we only apply the attack to those questions for which the initial answer entropy is greater than 0.50.50.50.5.111This implies that the adversary assigns less than about .97.97.97.97 probability to any one of the choices For each question we run the DA 5555 times, returning the IIL measurements for each run as well as the average number of times inputs or outputs were flagged by the input or output filtering models per attack.

Instant research discovery