Paper deep dive

May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks

Nishit V. Pandya, Andrey Labunets, Sicun Gao, Earlence Fernandes

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 74

Models: Llama-3-8B, Mistral-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:44:20 PM

Summary

The paper introduces ASTRA and ASTRA++, novel architecture-aware attack algorithms that exploit internal attention matrices in transformer-based LLMs to bypass fine-tuning-based prompt injection defenses like SecAlign, SecAlign++, and StruQ. The authors demonstrate that existing gradient-based methods like GCG are insufficient because they rely on output token probabilities, whereas ASTRA achieves 85-95% success rates by manipulating attention mechanisms, even in scenarios with unseen context windows.

Entities (4)

ASTRA · attack-algorithm · 100%GCG · attack-algorithm · 100%SecAlign · defense-system · 100%StruQ · defense-system · 100%

Relation Signals (4)

ASTRA → targets → SecAlign

confidence 95% · We evaluate ASTRA on SecAlign and StruQ.

ASTRA → targets → StruQ

confidence 95% · We evaluate ASTRA on SecAlign and StruQ.

ASTRA → bypasses → SecAlign

confidence 90% · the attacks generated by ASTRA++ achieve a success rate of 96% and 87% on unseen contexts against the SecAlign and SecAlign++ defended versions

GCG → failsagainst → SecAlign

confidence 90% · GCG's general inability to convincingly break defenses like SecAlign.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A popular class of defenses against prompt injection attacks on large language models (LLMs) relies on fine-tuning to separate instructions and data, so that the LLM does not follow instructions that might be present with data. We evaluate the robustness of this approach in the whitebox setting by constructing strong optimization-based attacks, and show that the defenses do not provide the claimed security properties. Specifically, we construct a novel attention-based attack algorithm for textual LLMs and apply it to three recent whitebox defenses SecAlign (CCS 2025), SecAlign++, and StruQ (USENIX Security 2025), showing attacks with success rates of up to \textbf{85-95\%} on unseen prompts with modest increase in attacker budget in terms of tokens. Our findings make fundamental progress towards understanding the robustness of prompt injection defenses in the whitebox setting. We release our code and attacks at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

73,847 characters extracted from source content.

Expand or collapse full text

May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks Nishit V. Pandya UC San Diego Andrey Labunets UC San Diego Sicun Gao UC San Diego Earlence Fernandes UC San Diego Abstract A popular class of defenses against prompt injection attacks on large language models (LLMs) relies on fine-tuning to separate instructions and data, so that the LLM does not fol- low instructions that might be present with data. We evaluate the robustness of this approach in the whitebox setting by constructing strong optimization-based attacks, and show that the defenses do not provide the claimed security properties. Specifically, we construct a novel attention-based attack al- gorithm for textual LLMs and apply it to three recent white- box defenses SecAlign (CCS 2025), SecAlign++, and StruQ (USENIX Security 2025), showing attacks with success rates of up to 85-95% on unseen prompts with modest increase in attacker budget in terms of tokens. Our findings make fun- damental progress towards understanding the robustness of prompt injection defenses in the whitebox setting. We release our code and attacks at this link. 1 Introduction Prompt injection attacks are one of the most pressing security issues in Large Language Models (LLMs). They are akin to stack smashing and confuse the LLM into following in- structions that appear in data [43]. With the emergence of so-called agentic systems, in which LLMs can use tools to automate a variety of tasks [29, 36], it is critical to defend against prompt injection attacks. Recent work has proposed fine-tuning based defenses that train models to learn special tokens that can delimit instruc- tions and data. Notable examples include Meta’s SecAlign [6] and SecAlign++ [7], OpenAI’s instruction hierarchy [33], and an academic system called StruQ [5]. Our work contributes an experimental security analysis of this class of prompt injec- tion defenses in the whitebox setting. Particularly, we focus on SecAlign (CCS 2025), SecAlign++ and StruQ (USENIX Security 2025), which have showed high resistance to prior state-of-the-art optimization-based attacks such as Greedy Coordinate Gradient (GCG) and AdvPrompter [27, 42]. As our first contribution, we experimentally establish that GCG-style attacks are not strong enough to fully vet the secu- rity of such defenses, showing that they do not perform much better than a simpler greedy search procedure. Our second contribution is a novel architecture-aware at- tack algorithm, ASTRA (short for Adversarial Subversion through Targeted Redirection of Attention), 1 that targets the attention matrices within transformer-based LLMs. Using ASTRA, we conduct a stronger and more realistic evaluation of fine-tuning based defenses. Although StruQ and SecAlign do evaluate their security against an adaptive adversary, their evaluation is limited by a lack of strong optimization-based attack algorithms for LLMs. This is a critical learning from the first wave of adversarial machine learning research – de- fenses need to be evaluated against strong, adaptive adver- saries which take into account the defense’s principle of op- eration [1, 4, 32]. ASTRA allows for precisely this kind of an evaluation. ASTRA enables a stronger adversary than GCG because (1) it uses the whitebox setting better by exploiting the architec- ture of the model to craft attacks (2) it can create attacks that are universal across the contents of the context window – this is especially relevant to agentic use cases of LLMs and goes beyond the original evaluations of StruQ and SecAlign (3) it allows us to effectively evaluate the defense by scaling the attacker budget (measured in number of injected tokens). ASTRA finds adversarial input tokens that try to cause the LLM to focus its attention – as measured by internal atten- tion matrices – on the attacker-supplied instructions (i.e., the injected prompt). Intuitively, if the LLM’s context window only had the attacker’s instructions and the attacker’s instruc- tions didn’t violate a content safety policy, then the LLM would follow them because there are no other instructions in the context. Following this heuristic, our attack generates sequences of tokens that try to make the LLM ignore all other tokens in the context window, except the attacker-supplied ones. By contrast, the Greedy Coordinate Gradient (GCG) 1 “Astra” also means “weapon” in several Indian languages. 1 arXiv:2507.07417v2 [cs.CR] 17 Dec 2025 algorithm tries to find inputs that cause the LLM to generate attacker-desired output tokens, without taking the LLM’s ar- chitectural structure into consideration. That is, the attacker’s loss function is expressed in terms of output token probabili- ties. In practice, this represents a very difficult optimization objective, as evidenced by GCG’s general inability to con- vincingly break defenses like SecAlign. On the other hand, ASTRA gains its power from architectural awareness — by expressing the attacker’s loss function in terms of attention, it is able to compute better attacks than GCG alone. Expressing a prompt injection loss function in terms of the internal attention matrices of the model requires addressing the challenge of determining which attention heads (out of thousands in modern LLMs) are most important for the spe- cific task that is given to the model. Attention heads are not independent, but rather, a change in an earlier head results in downstream changes. In general, solving the problem of determining which attention heads are important is a hard problem. It requires extensive manual intervention and/or multiple expensive ablations that involve turning-off attention heads or patching activations and empirically observing their effects on the task in question [3, 24, 38]. Our key insight is that in a prompt injection, the attacker wants specific output tokens to appear, and therefore, we do not need to explicitly find out which attention heads are critical for different types of fuzzily defined tasks. Rather, since our goal is to find out- of-distribution inputs, we need to determine the important attention heads for which small changes in the attention ma- trices cause large changes in the probability of the desired output tokens. We quantify the importance or weight assigned to each attention head using a novel sensitivity metric that can be computed using gradient information, thus bypassing the requirement for expensive ablations or manual interventions. A second challenge is evaluating the fine-tuning based de- fenses in a more realistic setting where the attacker does not have knowledge of the contents of the LLM’s context win- dow. This requires that the attacks produced by ASTRA be universal over unseen contexts. Although the original StruQ and SecAlign evaluations do not consider this scenario, we believe it is an important setting because it represents a more realistic attacker threat model. In general, an attacker will not have complete knowledge of the context window. For exam- ple, the context window might contain additional instructions, private user data, tool-calling demonstrations, etc. Further- more, this setting is the norm, rather than the exception, in agentic use cases. To address this challenge, we create AS- TRA++, a stronger variant of ASTRA that addresses the above challenge by dynamically adapting the weights assigned to each attention head during the optimization. We evaluate ASTRA on SecAlign and StruQ. Using the same evaluation methodology as outlined in these papers, we demonstrate an attack success rate of 82.5% and 72.5% for the SecAlign defended versions of Mistral and Llama-3 respectively (obtained from the authors of these two systems). Our attacks require a modest increase in token budget — i.e., the original StruQ and SecAlign papers evaluate the setting where the attacker is given20tokens that are suffixed to the attacker’s manually-written malicious instructions. We achieve high success rates by scaling the attacker’s budget up to35tokens that can be prefixed or suffixed (or both) to the injected instruction. Going beyond the original settings of SecAlign and StruQ, we also evaluate ASTRA++ in a more realistic setting where the adversary doesn’t have knowledge of the surrounding con- text. With this constraint, the attacks generated by ASTRA++ achieve a success rate of 96% and 87% on unseen contexts against the SecAlign and SecAlign++ defended versions of Llama respectively. Like before, we achieve high success rates by scaling the budget up to 35 tokens. StruQ and SecAlign only evaluate against an attacker limited to a 20-token budget. Unlike the computer vision space where L-norms represented attacker threat models (be- cause they captured image distortion) and had natural limits, the LLM space has no such inherent restriction. Context windows of modern LLMs are very large (often up to tens of thousands of tokens) and the trend is to make them even larger. Furthermore, when an LLM is used as part of a larger agentic system, the attacker’s instructions can arrive from any source, such as an email or a webpage which can easily span several hundreds of tokens, and thus, a larger attack string is likely to go un-noticed. On a fundamental level, the at- tacker is not restricted by the number of tokens they can inject. Contributions. •We empirically study the security of the class of fine- tuning based prompt injection defenses. We show that ex- isting white-box optimization-based attacks fail to break these defenses because the gradient of a loss function that measures the output probabilities of target tokens does not convey sufficient information for optimization. •We introduce ASTRA and a closely related variant AS- TRA++, two novel optimization-based attacks that ma- nipulate attention matrices to achieve prompt injection attacks. We show success rates of up to 85-95% on state-of-the-art SecAlign++, SecAlign and StruQ. The computed attacks are independent of prior tokens in the context window (i.e., our attack provides context window universality). 2 Background Transformer-based LLMs. An LLM can be modelled as a function that takes in a sequence of tokens(x 1 , x 2 ,... x n )∈ V ∗ — obtained by encoding an input string — where each token is from a fixed vocabularyV, and outputs a probability distribution over the next tokenP(·|x 1 , x 2 ,... x n ). The next tokenx n+1 is generated by sampling from the distribution P(·|x 1 , x 2 ,... x n ). The LLM then continues autoregressively, 2 i.e. it takes in(x 1 , x 2 ,... x n , x n+1 )as an input and generates x n+2 . This process continues until some stopping criterion is reached. The generated tokens are then decoded to get the final output string. Indirect Prompt Injection Attacks. Indirect Prompt Injec- tions are inference-time attacks that subvert the expected output of an LLM in order to cause the LLM to output text of the attacker’s choice. Prompt injections aim to override the developer or user intent and instead cause the LLM to gen- erate specific, malicious strings. Such attacks occur when an adversary controls a part of the input to an LLM. In real-world applications, an attacker can achieve such control by inserting malicious strings into third-party content that is placed into the context window of an existing conversation. Formally, a prompt injection happens when a malicious string crafted by the adversary (sayx Adv ) is combined with the trusted instruction of the user (sayx Trusted ) and is processed by the LLM. In practice, strings from untrusted contexts of- ten undergo special preprocessing, such as filtering, before they are combined with the trusted strings. They can also be delimited by other special tokens to mark separation between the trusted and untrusted contexts. The goal of the adversary is to craft anx Adv which when combined withx Trusted causes the LLM to output a stringy Target of the attacker’s choice, i.e. LLM(Combine(x Trusted , x Adv ))≈ y Target (1) Prompt injection attacks are of two types - handcrafted and automated. Handcrafted prompt injections exploit linguis- tic quirks, role-playing and other similar tricks. Automated prompt injections rely on an algorithm to generate adversarial inputs designed to mislead an LLM into generating attacker- chosen target strings. Most automated algorithms work by surrounding the attacker’s malicious instruction (which we refer to as the “payload”) with adversarial tokens [15, 18, 26]. Defenses against Indirect Prompt Injection. The primary goal of a defense is to make a system secure against indirect prompt injection attacks. One major class of these defenses, detection-based [19], work by predicting if an input contains a prompt injection. The system may then reject a query if a prompt injection is detected. Examples of such defenses include Prompt Guard [23], a known-answer detector and variants, like DataSentinel [20]. Another major class of defenses addresses this problem by modifying the model directly so that it is more likely to correctly follow the original prompt instead of an injected instruction. These methods, which we refer to as fine-tuning based defenses fine-tune the model itself to recognize the prompt structure and to separate instructions from data Ex- amples of such defenses include the open-source systems StruQ [5], SecAlign [6], and Meta’s SecAlign++ [7] and closed-source implementations such as OpenAI’s Instruction Hierarchy [33]. We describe a few examples of this class of defenses in more detail in section Section 4. 3 Threat Model Like prior work on indirect prompt injections, we assume that a (victim) user is using an LLM (target model) to interact with third-party content. An attacker is an untrusted third-party who adversarially controls a part of the input to a target model and intends to steer the model away from following original, trusted instruc- tions given by the user to a new set of instructions of the attacker’s choice. The delivery of adversarial instructions into a user’s context window can be carried out via several differ- ent methods such as poisoning a webpage to be summarized (as instructed by the user’s prompt to an AI assistant or as an operation of some agentic AI system), by poisoned documents, multimodal content, or code being processed [2,10,11,31,35]. For the purposes of this paper, we treat the real-world delivery mechanism as an orthogonal problem to our work. Nonethe- less, we note that the variety of delivery mechanisms and the increasing capabilities of LLMs have important consequences for the broader, general threat model of a prompt injection at- tack. That is, with the trend of increasing context windows in LLMs and improving capabilities such as multiple language understanding, multi-modality, and tool calling abilities, the attacker is not, in general, restricted by constraints in number of tokens or the size of the attack itself. We assume a whitebox, adaptive adversary, whose goal is to create prompt injection attacks with high likelihood of success. That is, we assume an adversary who has complete access to the weights of the target model and has all the details of the surrounding components that are orchestrating the LLM-user interaction. Examples of such components can be filtering or sanitization components that can filter text out of third-party content. In the context of a prompt injection, there are two distinct assumptions one can make about the knowledge an adversary has about a user’s conversation with an LLM. Weak Knowledge Assumption - The adversary has no infor- mation about the conversation history already present in the context window before the third-party data (controlled by the adversary) is injected. That is, the adversary knows nothing aboutx Trusted (Eq. (1)). This assumption reflects a general prompt injection scenario for chat-based and agentic systems. Strong Knowledge Assumption - The adversary has full information of the exact conversation history before the third party data (controlled by the adversary) was placed into the context window - i.e. the attacker knowsx Trusted . While un- realistic in the general case, there are instances where this strong assumption is valid. For example, if a user is using a task-specific bot with a fixed system prompt, such as the bots used in GPT-Store [25], then an adversary can be reasonably expected to know the exact contents of the context window. Furthermore, this is the setting that StruQ and SecAlign used to evaluate the strength of their defenses. 3 4 Fine-tuning based defenses In this section, we provide detailed background on three differ- ent fine-tuning based prompt-injection defenses - StruQ [5], SecAlign [6] and SecAlign++ [7]. Then, using SecAlign as a case study, we perform an analysis to explain why existing SOTA algorithms like GCG don’t break it convincingly. 4.1 Summary of some existing defenses StruQ starts with a base model and uses a version of adversar- ial training adapted for instruction tuning to obtain the final defended model. StruQ reserves special tokens which it uses to delimit data from instructions. The model is trained to rec- ognize these delimiters and ignore the instructions present in the delimited part. The model is combined with a preprocess- ing filter that sanitizes input text to remove any occurences of these delimiters. The StruQ models resist hand-crafted prompt injection techniques with success rates close to0%. Nonetheless, they are vulnerable to adversarial optimization methods like GCG, with attack success rates of approximately 60% of the cases when using GCG. SecAlign adopts a similar approach to StruQ. It uses special delimiters and preprocessing filters that removes special to- kens from the untrusted data ingested by the model. However, instead of directly performing adversarial training, SecAlign phrases the task of defending against prompt injections in terms of preference optimization. It aims to train the model to prefer to follow instructions coming from trusted sources over untrusted sources using Direct Preference Optimization (DPO). SecAlign is a stronger defense than StruQ - in addi- tion to blocking≈ 100%of handcrafted attacks, it also blocks ≈ 100%of automated attacks crafted using previous SOTA algorithms like GCG and AdvPrompter [27, 42]. SecAlign++ is an example of an Instruction Hierarchy (IH) model. [33]. IH models generalize to multiple hierarchical levels of trust, system (higher trust) and user (lower trust), LLM, and Tool Output (lowest trust) as opposed to models like StruQ and SecAlign that only offer two levels: trusted and untrusted. IH models are fine-tuned to prioritize following instructions from higher trust sources over instructions from lower trust sources. Meta’s SecAlign++, in particular, is an open-weights production-grade model implementing IH with three trust levels: “system” (higher trust), “user” (lower trust) and “input” (untrusted). 4.2 Why does GCG not break SecAlign? Greedy Coordinate Gradient (GCG) [42] is a state-of-the-art algorithm in white-box input optimization for an LLM. The al- gorithm was originally proposed in the context of jailbreaking. However, later work has modified and adapted the algorithm for generating prompt injection attacks under a variety of threat models and use cases [6, 9, 18]. While subsequent works have used GCG with a variety of different objective functions [9,18], here we focus on GCG as an optimization algorithm for the simplest objective function - that is, as an optimization algorithm to optimize a subset of the input prompt (keeping the rest of the input prompt fixed) in order to maximize the probability of the LLM generating the attacker-chosen target string, or equivalently, minimizing the “target logprobs” (Eq. (3)). Furthermore, several recent works have also proposed sev- eral algorithmic enhancements to GCG [14, 17, 39, 40]. These works either apply transferability or aid the search procedure via better exploiting the breadth/depth trade-off during op- timization or exploiting increased randomness at each step. Fundamentally, all these works still try to minimize the target logprobs by directly using the gradients of the logprobs (at a given point) as a signal to guide the optimization. In this subsection, we aim to establish empirically that the gradients of the target logprobs when used as a proxy for the actual value of the function itself don’t provide significant value to the GCG procedure. In other words, we argue empirically that the target logprobs function provides a poor landscape for gradient-guided optimization procedures like GCG. Concretely, given an input prompt, represented as a se- quence of tokens x= (x 1 , x 2 ,..., x n ), letI⊆1, 2,... nrep- resent the indices of the tokens which can be modified by the optimization algorithm (with the rest of the tokens kept fixed). Let y T = (y 1 , y 2 ,... y m ) represent an attacker-chosen target sequence of tokens. The attacker in this case wants the LLM to output the target tokens y T with high probability. The optimization problem to be solved in this case is - x ∗ I = argmax x I ∈V |I| Π m j=1 P(y j |x 1 ,... x n , y 1 ,... y j−1 ; Θ)(2) That is, the GCG function tries to minimize the following loss function (using the notation∥to denote concatenation of sequences of tokens) TargetLogprobs(x, y T ) =− m ∑ j=1 log P(y j | x∥y 1: j−1 ; Θ) (3) The above optimization problem is a hard discrete optimiza- tion problem for which a direct search is not feasible. GCG addresses this with an iterative, guided, greedy search algorithm. That is, it starts with some initial sequence of tokens x (0) ∈ V |I| and iteratively obtains a sequence of points x (0) , x (1) ,...⊆ V |I| until some stopping criterion is reached. At the k + 1-th iteration, GCG performs the steps - 1. (Guiding step) - A coarse filtering step which uses the gradients of the objective function at x (k) to generate a set of promising candidatesC (k) ⊆ V n . The set of promis- ing candidatesC (k) consists of candidate sequences of tokens each of which differs from the current point x (k) at exactly one token at one of the|I|perturbable posi- tions. That is, the candidates generated are all one token substitution away from the current point. 4 −20020 0 5 10 15 20 D r Frequency r = 1 (a) Mean = 0.7, Std. Dev = 8.4 −20020 0 5 10 15 20 D r Frequency r = 3 (b) Mean =−0.1, Std. Dev = 3.5 −20020 0 5 10 15 20 D r Frequency r = 5 (c) Mean =−0.2, Std. Dev = 2.5 Figure 1: Difference between guided and unguided optimization concentrates close to 0 when appropriately averaged 0100200300400 500 0 10 20 30 Iterations Target Logprobs Guided (GCG) Unguided Figure 2: Average loss curves for guided and unguided opti- mization are very close to each other 2.(Greedy step) A fine selection step which exactly evalu- ates the objective function on the set of promising can- didatesC (k) - consisting of single token substitutions of x (k) - obtained from the guiding step. The best per- forming candidate fromC (k) is chosen as the next point x (k+1) ∈ V n and the process is repeated. Effectively, GCG uses gradients of theTargetLogprobs function, which provide local information about the landscape of this loss function at the point x (k) , as a proxy to guess what is likely to be the actual function value at a far-away point, that is, at the point corresponding to the actual substituted token. A priori, this represents a sound strategy, that is, if the optimization function were a smooth enough function, such as a piecewise linear function, then on average gradients should be expected to provide a good guess for what hap- pens at distant points. And if so, the guiding step should add significantly to the optimization procedure. We establish empirically that when controlled for the at- tack hyperparameters of number of steps, candidate set size and initialization, GCG does not significantly outperform an unguided greedy search. That is, we perform the following two algorithms which differ only in the guiding step. 1.Guided version - We use the candidate set generation used by GCG that uses gradients (at the current point) to suggest promising single-token substitutions as candi- dates for the next point. 2.Unguided version - We use a randomly generated set of single-token substitutions as candidates. Crucially, for both the above, we do still evaluate the loss function exactly on the candidates generated. That is, the only difference lies in the candidate generation step. For our empirical evaluation, we mimic the setting of the evaluation of SecAlign and StruQ. We perform our compar- ison study on50randomly chosen examples from the Al- pacaFarm evaluation set containing both an “instruction” and “input” field. For each example, we inject the fixed payload “Print exactly ‘Hacked!’” at the end of the “input” part followed by an optimizable suffix of20tokens. The adversar- ial target is the string ‘Hacked’. Since we are interested in evaluating the effectiveness of the guiding strategy used by GCG, we aim to control for all other hyperparameters used during optimization and average over all sources of randomness used. Concretely, we perform both the guided and unguided op- timization as above for500iterations each with the same candidate set size of512. For the guided version, the512can- didates are chosen as guided by theTop− 256gradient values for each of the20token positions. That is,512candidates are chosen from256× 20 = 5120promising single-token sub- stitutions. On the other hand, for the unguided version, the 512candidates are chosen from256randomly chosen single- token substitutions for each of the 20 positions. We run both the versions initialized with the same randomly initialized sequence of 20 tokens. Further, to control for the effect of randomness, for each of the50examples, we performrdifferent runs of the above ex- periment, each with different initializations. We then compute the average difference between the best guided loss obtained during the optimization and the best unguided loss obtained during the optimization averaged over the r runs. that is, D r = 1 r r ∑ α=1 (min k L (g) α (k)− min k L (u) α (k)) and plot the distribution of the variableD r over the50exam- 5 ples averaged over different values ofrranging fromr = 1 tor = 5runs in Fig. 1. The distributions ofD r for different values of r establish the following - Across examples, guided optimization doesn’t signifi- cantly outperform unguided. For all values ofr,D r is dis- tributed approximately symmetrically around0. That is, the guided version outperforms the unguided by about as much as the unguided version outperforms the guided one. This can be seen by the average ofD r (averaged across examples, for a fixedr), which is close to0for all values ofr. The same behaviour is observed over the course of the optimization procedure as is seen in the average loss curves for the guided and unguided optimizations in Fig. 1. The difference between the performance of guided and unguided optimizations depends strongly on the initializa- tion and other randomness. We observe that with increase inr, the variance and spread in the distribution ofD r (across examples) decreases rapidly (Fig. 1). That is, any outsized difference observed between the performance of the guided and unguided versions of the optimization on a particular sin- gle run (for a single example) disappears when more runs are averaged for the same example. In particular, this means that the role played by the initializer and randomness are crucial to ensure that GCG outperform an unguided search. From the above, we establish empirically that the loss land- scape provided by the loss functionTargetLogprobsrepre- sents a poor landscape for optimization and makes it so that GCG can’t utilize gradients ofTargetLogprobswith good effect. 5 The ASTRA Attack To address the above problems, we propose a new method that explicitly exploits the internal structure of an LLM. Con- cretely, we use the architectural properties of LLMs in order to craft a novel loss function that can be used to generate better prompt injection attacks. We use an optimized input generated by optimizing this loss function as a “warm start” initializer to “fine-tune” the adversarial input using standard algorithms like GCG. Our key insight is that we can exploit an instruction-tuned model’s instruction-following ability to craft prompt injection attacks. Specifically, we note that if the attacker’s instruction payload was the only text in the input stream, the model would easily follow that instruction. This insight forms the basis for our loss function. To formalize this mathematically, we rely on the attention mechanism of an LLM. The attention mechanism forms the key component of our new loss function. 5.1 Background on Attention The attention mechanism in an LLM is a core component which assigns a weight to each previous token while trying to generate the next token. Formally, when a sequence of tokens x= (x 1 , x 2 ,... x n ) is inputted into an LLM, it first maps each tokenx i to an embedding vectore (0) i ∈ R d to obtain a sequence of initial representations(e (0) 1 , e (0) 2 ,... e (0) n )∈ R n×d . This sequence of representations is transformed by a series of decoder lay- ers (say, total ofLdecoder layers) into a final transformed representation( f 1 , f 2 ,... f n )∈ R n×d . Finally, the “language modelling head", represented by a matrixW ∈ R d×|V| is used to generate the probability distribution for the next token by the following calculation - P(·|x 1 , x 2 ,... x n ; Θ) = softmax( f n · W)∈ R |V| where each coordinate lies between0and1and represents the probability of generation of that particular token given the context (x 1 , x 2 ,... x n ). Thel-th decoder layerD (l) is itself a complex computation that maps one sequence of representations inR n×d to another sequence of representations inR n×d . It consists of an attention layer (typically prenormalized and postnormalized) and a fully connected layer that modify a residual stream that forms the internal representations that feed into the next decoder layer D (l+1) . For the sake of brevity, here we only describe some relevant specifics of the attention mechanism. A rigorous and complete treatment of transformers can be found in [28]. The key architectural innovation at the core of a decoder layer is the Masked Multi-head Attention (MMHA) mecha- nism. The MMHA mechanism at thel-th decoder layerD (l) takes in a sequence of representations obtained from the previ- ous layer, sayE (l−1) = (e (l−1) 1 , e (l−1) 2 ,..., e (l−1) n )∈ R n×d and computes a sequence of square matrices(A (l) 1 , A (l) 2 ,... A (l) H ) called “attention matrices” whereA (l) i ∈ R n×n . The number Hrepresents the number of “attention heads” of the LLM at layer l. These matrices are computed as A (l) i = softmax(E (l−1) W (l) i E (l−1)T + CausalMask n )(4) Here,W (l) i ∈ R d×d is a pre-trained parameter of the LLM corresponding to thei-th attention head at thel-th layer. The matrixCausalMask n is ann× nmatrix with−∞above the diagonal and0everywhere else, and thesoftmaxfunction is applied row-wise to the matrix. The attention matricesA (l) i represent intermediate values computed within the decoder layerD (l) that are ultimately used to further calculate the representations that can be used as input for the next layer D (l+1) . Due to the specific functional form above, an attention matrixAhas significant structure to it. All attention matrices have the following properties. (We use the notationA[ j]to refer to thej-th row ofAand the notationA[ j][k]to refer to the element in the j-th row and k-th column of A) 1. The attention matrices are lower-triangular, i.e., elements above the diagonal are zero. That is,A[ j][k] = 0ifj< k. 6 “What" “is" “the" . . . “+" “2?"            10 ·00 0.30.7 ·00 0.20.3 ·00 . . . . . . . . . . . . 0.060.14 ·0.030 0.050.1 ·0.090.01 “What"“is" ·“+"“2?"            Figure 3: An attention matrix for the input “What is the value of 2 + 2?". The last row contains the weights assigned to each of the previous tokens for generating the next token. 2.Each element of the attention matrix lies between0and 1 (inclusive). That is, 0≤ A[ j][k]≤ 1 3. Each row of the attention matrix sums up to1. That is, for each j, ∑ n k=1 A[ j][k] = ∑ j k=1 A[ j][k] = 1 The above properties imply the following interpretation(s) for the rows of an attention matrix A. 1.The firstjentries in thej-th rowA[ j]of an attention matrix represent the attention given (by that head) to the first j tokens when generating the j + 1-th token. 2.The firstjentries in thej-th row are the weights of a convex combination over j dimensions 3.The firstjentries inA[ j]are the probability masses of a discrete probability distribution over 1, 2, . . . j 5.2 Crafting a loss function We use the above interpretations to guide the design of our loss function. When the LLM is processing thentokens in the input x= (x 1 , x 2 ,... x n )in order to generate then+ 1-th token, the last (i.e.n-th) rows of all the attention matrices A (l) i (x)represent the attention given to each of thentokens in the input by that attention head. Since we want the model to attend only to the payload, we identify the indices of tokens corresonding to the attacker’s payload, sayJ⊆1, 2,... n and try to maximize the sum of attentions given to this part of the input. As before, let x I represent the modifiable tokens which are to be optimized by the attacker. Letybe the target token that the attacker wants the LLM to generate as the n+ 1-th token. Formally, for a single attention matrixA (l) i (x)generated on input x = (x 1 , x 2 ,..., x n ) of length n, we define AttLoss (l) i (x, y) = 1− ∑ j∈J A (l) i (x)[n][ j](5) We note thatAttLoss (l) i (x, y)doesn’t depend on the target token y and only depends on the input. To account for all the attention matrices, we define AttLoss(x, y) = L ∑ l=1 H ∑ i=1 w (l) i AttLoss (l) i (x, y)(6) wherew (l) i are some set of weights, assigned to all the attention heads. For a longer sequence of target tokens y= (y 1 , y 2 ,..., y m ), we just sum up the losses over the target tokens and define AttLoss(x, y) = m ∑ j=1 AttLoss(x∥y 1: j−1 , y j )(7) This loss function becomes0when the entire attention of each attention matrix is concentrated on the attacker payload tokens. That is, this loss function captures how much of the attention of the LLM is being paid to the surrounding context. A larger value of this loss function implies that more of the attention across all layers is distributed over the entire context window, whereas a smaller value implies that a lesser amount of attention is distributed over the entire context. 5.3 Weights for each attention head In principle, we could simply add all contributions equally from each attention headAttLoss (l) i to craft the loss function AttLoss. However, this is likely to be suboptimal since 1. Prior work from the field of mechanistic interpretabil- ity has shown that different attention heads carry out different functions in an LLM. [24, 37] 2. Attention heads across layers are not independent from each other, since the value of the attention matrix in thel + 1-th layer depends on the value of the attention matrix in thel-th layer. Small changes in the values of the attention matrix in earlier layers can lead to large changes in the values of attention matrices in later layers. In general, figuring out which attention head is impor- tant for which task requires extensive manual analysis and/or lengthy ablation experiments which turn off attention heads one-by-one to see the effect they have on the LLM’s output [37]. A typical LLM has more than a thousand attention heads (across all layers) which makes such an analysis impractical. Our key insight is that since our eventual goal is finding adversarial perturbations in the token space that lead to an adversary chosen output, we don’t need to explicitly find out which attention heads are functionally important for certain abstract tasks. Instead, we only need to find those attention heads for which small changes in their corresponding atten- tion matrices lead to large changes in the probability of the desired target token. This observation allows us to sidestep an expensive ablation experiment which would turn off atten- tion heads one-by-one and analyze its impact on the output. Instead, since sensitivity to small perturbations (of the prob- ability of the target token) is a local property of a function, 7 0481216202428 Attention Head 0 4 8 12 16 20 24 28 Layer 0.2 0.4 0.6 0.8 1.0 1.2 (a) SecAlign Meta-Llama-3-8B-Instruct 0481216202428 Attention Head 0 4 8 12 16 20 24 28 Layer 0.1 0.2 0.3 0.4 (b) SecAlign Mistral-7B-v0.1-Instruct Figure 4: Very few attention heads have high sensitivities for the target string “Hacked” sensitivity can be meaningfully quantified using gradients. We use these gradients as the key piece of information that allows us to quantify the importance of attention heads. Formally, letybe a target token that the adversary wants the LLM to generate. Let z= (z 1 , z 2 ,... z n )represent an input sequence of tokens which generate the attention matrices A (l) i (z) during the forward pass of the LLM. Let LogProb(y|z) = logP(y|z; Θ) represent the log of the probability of the target tokeny. Then, ∂ LogProb(y|z) ∂ A (l) i (z) ∈ R n×n represents the gradients of this loss function with respect to the attention matrices at the input point z. Since then+ 1-th token prediction depends on the attention values in then-th row, we look at the sum of the magnitude of the entries in then-th row of the gradient matrix to get a measure for how sensitive the loss function is to changes in that row of the matrix. That is, we define sensitivity as Sen (l) i (z, y) = |z| ∑ k=1 ∂ LogProb(y|z) ∂ A (l) i (z) [n][k] (8) For a longer target sequence y = (y 1 , y 2 ,... y m ), we define Sen (l) i (z, y) = |y| ∑ j=1 Sen (l) i (z∥y 1: j−1 , y j )(9) Finally, to get an average sensitivity score for an attention head (with respect to a target sequence of tokens y), we take a datasetDof inputs z and normalize the sensitivities by the total length|z|+|y| and average out over the datasetD Sen (l) i (y) = 1 |D| ∑ z∈D 1 |z|+|y| Sen (l) i (z, y)(10) This provides a proxy measure for how sensitive, on aver- age, the probability of a target tokenyis to changes in the value of the attention matrix at thei-th attention head at the l-th layer. We show the average sensitivities for two differ- ent SecAlign defended models for the target string “Hacked” constructed from a training dataset consisting of examples from the Dolly fine-tuning dataset [8] in Fig. 4. In practice, we find that a large subset of the sensitivities (across heads and layers) are very small, as can be seen in Fig. 4. Heuristically, small changes in these attention heads don’t lead to large changes in the probability of the target sequence of tokens and are likely to act as noise during opti- mization. We can therefore choose a thresholdτand manually set the small contributions of heads less thanτto0. That is, we define the clipped sensitivities as ClipSen (l) i (y) = ( 0ifSen (l) i (y)≤ τ Sen (l) i (y) otherwise (11) We use these clipped sensitivities as the final weights for our attention loss function to get AttLoss(x, y) = L ∑ l=1 H ∑ i=1 ClipSen (l) i (y)· AttLoss (l) i (x, y) (12) 5.4ASTRA : Strong Knowledge Adversary We first begin by describing our final attack for the simpler case of a Strong Knowledge Adversary (Section 3) who has full knowledge of the exact conversation context into which the attack is placed. In this particular case, the adversary has to optimize for a target output over a single input prompt. That is, the goal of the adversary is to find an inputx Adv such thatLLM Θ (Combine(x Trusted , x Adv ))≈ y Target (Eq. (1)) for 8 Algorithm 1: GenGCG p,N,B (x (0) , y T , Loss) Input: Initial sequence of tokens x (0) = (x 1 , x 2 ,..., x n )∈ V n , with modifiable subset of indices I⊆1, 2,..., n Input: Target tokens y T = (y 1 , y 2 ,..., y m )∈ V m Input: A function Loss : V n × V m → R to be optimized Parameters: p∈ N — number of top candidates per token position at each iteration; N∈ N — number of iterations; B∈ N — number of function evaluations per iteration. Output: Optimized tokens x ∗ begin Logprobs← [ ] ; Tokens← [ ] ; for k← 0 to N− 1 do foreach i∈ I do X (k) i ← Top-p −∇ e x i Loss(x (k) , y T ) end for b← 1 to B do i∼ Uniform(I) ̃ x (k) b ← x (k) ̃ x (k) b [i]∼ Uniform(X (k) i ) end b ∗ ← arg min b Loss( ̃ x (k) b ) x (k+1) ← ̃ x (k) b ∗ Logprobs[k]← TargetLogprobs(x (k+1) , y T ) Tokens[k]← x (k+1) end return Tokens[argmin Logprobs] end Algorithm 2: ASTRA p,N 1 ,N 2 ,B (x (0) , y T ) Input: Initial sequence of tokens x (0) = (x 1 , x 2 ,..., x n )∈ V n , with modifiable subset of indices I⊆1, 2,..., n Input: Target tokens y T = (y 1 , y 2 ,..., y m )∈ V m Parameters: p∈ N — number of top candidates to consider; N 1 ∈ N — number of steps for first phase; N 2 ∈ N — number of steps for second phase; B∈ N — number of evaluations per step. Output: Optimized tokens x ∗ begin x ∗ 1 ← GenGCG p,N 1 ,B (x (0) , y T , AttLoss) x ∗ 2 ← GenGCG p,N 2 ,B (x ∗ 1 , y T , TargetLogprobs) return x ∗ 2 end the special case when the adversary knows the stringx Trusted . This represents a simple prompt injection objective from an optimization point of view. ASTRA is a two-phase attack where instead of directly op- timizing for theTargetLogprobsobjective, we instead begin by minimizing the attention loss functionAttLoss(Eq. (12)) to get a “warm start”, that is, a good initialization that can be “fine-tuned” by then minimizing forTargetLogprobs. To com- pute the attention loss function, we use a proxy dataset such as the Dolly fine-tuning dataset to calculate the sensitivities and subsequently the exact loss values [8]. To solve the actual optimization problems in both phases, we use essentially the same guided greedy approach as used by the GCG algorithm, with the only change being the loss function under consideration. That is, we rely on the gradi- ents at the current point to provide us with a candidate set of promising single-token substitutions and we evaluate the loss exactly on these substitutions to get the next point in the pro- cess. We call this algorithm Generalized GCG (Algorithm 1). Our final two-phase algorithm is as described in Algorithm 2. 5.5ASTRA++ : Weak Knowledge Adversary In contrast to a Strong Knowledge Adversary, a Weak Knowl- edge Adversary doesn’t have any information about the sur- rounding context tokens x Trusted . Therefore, the adversary needs to be able to generate universal prompt injections that can work across a large range of x Trusted values. This repre- sents a significantly harder optimization objective than in the case of a strong knowledge adversary. To address this problem, we propose a variant of ASTRA, which we call ASTRA++ that dynamically updates the weights assigned to the attention heads as optimization progresses. At a high-level, this captures the idea that as the adversar- ial perturbations go from regions with high target logprobs (in-distribution) to regions with low target logprobs (out-of- distribution), the sensitivities, which are themselves a func- tion of the logprobs (Eq. (10)), also consequently change and hence dynamically updating the attention loss function gives us a better view of which attention heads are how important at any step in the optimization process. Prior work has addressed the task of generating universal prompt injections by attempting to optimize the adversar- ial perturbation sequence of tokensrover a training set of surrounding contexts by directly optimizing for the average target logprobs (of the target string) over the training data [18, 26]. That is, they use a training dataset where each train- ing datapoint consists of a sequence of context tokens x Trusted , a sequence of prompt injection payload tokens x Payload and the appropriate target sequence of tokens y T corresponding to the payload x Payload . That is,D =(x Trusted ,x Payload ,y T ). For simplicity of exposition and notation, but without loss of generality, we assume that the modifiable tokens to be opti- mized, say, r, are always placed in a suffix-only configuration 9 0481216202428 Attention Head 0 4 8 12 16 20 24 28 Layer 0.2 0.4 0.6 0.8 1.0 1.2 (a) Step 0 (Avg. Target Logprobs: 26.3) 0481216202428 Attention Head 0 4 8 12 16 20 24 28 Layer 0.5 1.0 1.5 2.0 2.5 3.0 3.5 (b) Step 200 (Avg. Target Logprobs: 3.52) 0481216202428 Attention Head 0 4 8 12 16 20 24 28 Layer 0.05 0.10 0.15 0.20 0.25 (c) Step 400 (Avg. Target Logprobs: 0.23) Figure 5: Local Sensitivity patterns at points with high average target logprobs (Step 0) - differ from those at points with low average target logprobs (Steps 200 and 400). (All sensitivities obtained by averaging over a dataset of size 10.) after the payload string. The loss function used by prior works for universality in this case is - Logprobs D (r) = E D TargetLogprobs(x Trusted ∥x Payload ∥r, y T ) (13) While we adopt a similar approach for constructing our uni- versal prompt injections, the nature of a prompt injection helps us simplify the optimization task significantly. Optimizing over several different choices of contexts, payloads and target strings represents a hard training objective. Our key insight is that, for the case of a prompt injection attack specifically, a realistic attacker needn’t worry about optimizing over several payloads. Instead, since the attacker controls the payload, we can simplify the optimization task by reasonably assuming that the sequence of tokens x Payload is constant across all the training examples. Similarly, we can also assume that the target sequence y T =y is constant across all training exam- ples. As a consequence, our training dataset varies only in the contents of the surrounding contexts. We use the notation D =x Trusted ∥x Payload to refer to the training set of contexts over which the training has to occur. For this simplified case, we use the following notation - Logprobs D,y (r) = E x∈D TargetLogprobs(x∥r, y)(14) Analogous to the single-prompt optimization case, instead of directly minimizing the average target logprobs loss func- tion, we try to create a “warm start” which can be later im- proved by optimizing the average target logprobs. We use a similar attention loss construction in order to craft our warm start with a few crucial differences. Instead of pre-computing a globally constant set of sensitivities from a fixed, proxy dataset to use as weights, we directly try to leverage the training datasetDto compute local sensitivities. For each training datapoint x=x Trusted ∥x Payload ∈Dand a fixed perturbation r, we can create the perturbed point x Trusted ∥x Payload ∥r and the datasetD⊕ r =x∥r | x∈D. At a given perturbation r, we can compute the local sensi- tivities (over the datasetD⊕ r) for a target sequence y as LSen (l) i,r (D, y) = 1 |D⊕ r| ∑ z∈D⊕r 1 |z|+|y| Sen (l) i (z, y)(15) This is the same construction as the global sensitivities (Eq. (10)) except that these are being computed over a dataset which is itself adversarially perturbed. Like for ASTRA, we clip these local sensitivities to a thresholdτto get the clipped local sensitivities, which we denote by CLSen (l) i,r (D, y) . These local sensitivities provide a more relevant quantifica- tion of the relative importance of attention heads since they take into account the effect of the perturbation itself on the probabilities of the target tokens. In contrast, sensitivities obtained by averaging over a proxy, third-party dataset can’t capture the effect of the perturbation itself since they only cap- ture the sensitivities over a static dataset – which only reflect “in-distribution” points. We find empirically that sensitivities do change significantly when going from in-distribution in- puts to out-of-distribution inputs (Fig. 5) thus motivating our next definition. We define the Average Local Attention Loss (with local sensitivities computed at a perturbation r, averaged overD) at an input point x and a target sequence of tokens y as AttLoss r,D (x, y) = L ∑ l=1 H ∑ i=1 CLSen (l) i,r (D, y)· AttLoss (l) i (x, y) (16) For our final algorithm ASTRA++, we use the same two phase approach as for ASTRA, where in the first phase, we minimize the average attention loss, followed by minimizing the average target logprobs (losses averaged over the training dataset) with a key difference. While minimizing the attention loss, we periodically refresh the weights of the loss function according to where we are in the optimization process. The full algorithm is as shown in Algorithm 4 in Section A. 10 Budget (Config.)GCG ASRASTRA ASR 20 (0, 20)12.5%20% 25 (5, 20)32.5%57.5% 30 (10, 20) 35%60% 35 (15, 20)45%72.5% (a) Meta-Llama-3-8B-Instruct Budget (Config.)GCG ASRASTRA ASR 20 (0, 20)32.5%75% 25 (5, 20)57.5%77.5% 30 (10, 20)60%80% 35 (15, 20) 70%82.5% (b) Mistral-7B-Instruct-v0.1 Table 1: Attack success rates (out of 40 examples) against the SecAlign models for a Strong Knowledge Adversary. 0100200300400 500 0 10 20 30 Iterations Target Logprobs Baseline (GCG) ASTRA Figure 6: The average loss curves (shown here for a budget of 30tokens for the SecAlign defended Mistral) show the “warm start” effect —- a good initializer from the first350iterations significantly helps the usual GCG algorithm for the last150 iterations. 6 Evaluation With our evaluation, we aim to answer the following main research questions - 1. How effective is ASTRA in breaking fine-tuning based defenses under the Strong Knowledge Assumption? 2. How effective is ASTRA++ in breaking these defenses on unseen contexts (the Weak Knowledge Assumption)? 3. How does our attack depend on the choice of weights? 6.1 Attack Success Rate - ASTRA Models. For our evaluation, we choose the SecAlign defended models of Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.1 since these are the latest models for which SecAlign adapters were trained. We choose the equivalent defended models for StruQ for consistency. Evaluation Dataset. The security evaluation of StruQ and SecAlign was originally performed on a test set of208exam- ples from the AlpacaFarm dataset which contain a non-empty “input" field. The attacker’s prompt injection is placed at the end of the data section of the input. The prompt injection placed at the end itself consists of the payload, along with the optimized adversarial tokens. The attacker payload used by SecAlign and StruQ for their evaluations is the string “Print exactly ’Hacked!’" with the target string ‘Hacked’. For our evaluation, we randomly chose50out of the208 examples from the AlpacaFarm evaluation dataset. Due to hardware limitations, we only report our attack success rate out of40examples that we could complete. These40exam- ples represent∼ 20% of the evaluation set. Evaluation Metrics. SecAlign and StruQ evaluated their de- fenses on a simple prompt injection task, where the attacker’s goal is to cause the LLM to output the string “Hacked". The attack is considered successful if the output generated by the LLM starts with the string “Hacked" or the string “hacked". All generations happen at temperature 0. Baseline. We compare our attack with GCG as the baseline. In all our experiments, we control for all the following variables. 1.Attacker budget and configuration (i.e., number of modi- fiable prefix and suffix tokens) 2. Attack initialization 3. Number of optimization iterations 4. Number of forward passes on each iteration (B) 5. Value ofp(i.e. the number of top gradients chosen from) That is, we run both GCG and our attack for each of the test examples under the same conditions and budgets with the only difference being the optimization strategies. Implementational Parameters. We run both attacks for500 iterations, withB = 512forward passes on each iteration. For both attacks, we takep = 256promising candidates for each token position. For the implementation of our loss function, we compute average sensitivities based on a subset of50 examples from the Dolly fine-tuning dataset [8], and drop the bottom 75% of all attention heads. SecAlign - Attack Success Rate. We perform our evaluation over a range of different attacker budgets and prefix-suffix configuratons. The results are as shown in Table 1. With the default budget and configuration as in the original SecAlign paper, we can break75%of the examples for the defended Mistral model and20%of the examples for the defended Llama-3 model. Both of these numbers outperform the base- line success rate achieved by GCG. Further, by increasing the budget by just5tokens, we can increase the success rate for the defended Llama-3 model to 57.5% StruQ - Attack Success Rate. StruQ represents a signifi- cantly weaker defense than SecAlign. Just by running GCG on the40examples above with the default attacker budget of20tokens in a suffix-only configuration, we were able to break28examples for the StruQ defended Mistral and23 examples for the StruQ defended Llama-3. We apply ASTRA to the remaining12and17examples with the same budget and configuration. ASTRA breaks10out of the12examples for the StruQ defended Mistral and 8 out of the17examples for the StruQ defended Llama-3. 11 6.2 Attack Success Rate - ASTRA++ Models. - Due to resource limitations, we run the ASTRA++ experiments on only the two most robust defended models. 1. SecAlign Meta-Llama-3-8B-Instruct - We choose this model since this model shows the highest resistance to prompt injection attacks against Strong Knowledge Adversaries (Table 1), thus providing the strongest chal- lenge to a Weak Knowledge Adversary as well. 2.SecAlign++ - This is a production-grade Instruction Hi- erarchy model which represents a realistic target model for a Weak Knowledge Adversary whilst showing compa- rable robustness to the previous models, thus providing a hard target. Training Dataset. For each training run, we train our attacks on a training set of10randomly chosen examples from the 208AlpacaFarm examples which have aninputfield. Like before, we place the payload “Print exactly ’Hacked!’” at the end of the input surrounded by a prefix and suffix. Evaluation Dataset. We use the same dataset as for the Strong Knowledge Adversary case. That is, we use the re- maining198unseen examples from the AlpacaFarm dataset which contain an “input” field as our test set. We use the same prompt format as expected by the SecAlign model. For the instruction-hierarchy model, we evaluate both modes of operation, one where we place the instruction at the highest trust level, that is, as the “system” prompt and place the data at the lowest trust level, that is, as the “input” and the other where we place the instruction as the “user" and the data at “input". This provides the worst-case scenario for our attacks. Evaluation Metrics. For each training run for a fixed budget, (trained with10examples), we compute the Attack Success Rate over the remaining198examples by counting the num- ber of model outputs that start with “Hacked” or “hacked”. For each attacker budget, we run3different training runs, each with a completely different set of10training examples to normalize for the effect of training set on the algorithm. Baseline. We compare ASTRA++ with a universal version of GCG as the baseline. We note that a universal GCG over this combined loss function is equivalent to running an algorithm like NeuralExec [26] with a simpler training objective. Implementational Parameters. For each attack run, like be- fore, we control for initialization, number of iterations, num- ber of forward passes per iteration, randomization strategy, and number of top gradients. In addition, we also control for the training dataset. We run both algorithms for a to- tal of1000iterations withp = 256andB = 512. For AS- TRA++, we split the1000iterations into700steps of atten- tion loss minimization and300steps of Univ-GCG, with the sensitivities being refreshed every50iterations (that is, F = 14, s 1 = 50, s 2 = 300in the notation of Algorithm 4). We uniformly clip 50% of the heads for all the loss functions. Results. We show the mean and standard deviation for the testing attack success rate (averaged over 3 training and test- Budget (Config.) Univ-GCG ASR Mean (Std. Dev.) ASTRA++ ASR Mean (Std. Dev.) 30 (15, 15)11% (17%)28% (48%) 35 (17, 18) 55% (48%)96% (5%) 40 (20, 20)2% (2%)85% (18%) (a) SecAlign defended Meta-Llama-3-8B-Instruct Budget (Config.) Univ-GCG ASR Mean (Std. Dev.) ASTRA++ ASR Mean (Std. Dev.) 30 (15, 15)19% (16%)62% (53%) 35 (17, 18)22% (21%)87% (12%) 40 (20, 20) 30% (33%)55% (20%) (b) SecAlign++ defended model (instruction as “system") Budget (Config.) Univ-GCG ASR Mean (Std. Dev.) ASTRA++ ASR Mean (Std. Dev.) 30 (15, 15)1% (11%)69% (1%) 35 (17, 18) 31% (24%)22% (54%) 40 (20, 20)22% (53%)37% (30%) (c) SecAlign++ defended model (instruction as “user") Table 2: Test ASR on 198 unseen examples (mean and standard deviations over 3 training runs on distinct training datasets) ing runs) of ASTRA++ in Table 2. ASTRA++ significantly outperforms Universal GCG on average. Nonetheless, we ob- serve that GCG can also create strong universal attacks, as can be seen by the high standard deviations which come due to outlier training runs which led to some success. 6.3 Ablation on Weights The key novelty in the construction of ASTRA lies in the loss function. We perform an ablation study to show that the clipped sensitivities used as weights in our loss function outperform other intuitive and obvious choices for weights such as uniform weights and weighing only the first and last layer in addition to outperforming the baseline itself. Concretely, we take the40examples from the evaluation dataset of ASTRA and run ASTRA on them with5different weighting functions along with the baseline. Like before, we control for all the confounders like initialization, attacker budget, and other attack hyperparameters. The average loss curves for the SecAlign defended version of Mistral in Fig. 7 show that clipped sensitivities outperform other weighting functions. 12 Only firstOnly lastUniform Avg. Sen.Clip. Sen.Baseline (GCG) 0200400 0 10 20 30 Iterations Target Logprobs Figure 7: Clipped sensitivities outperform other weighting functions - shown here for the SecAlign defended Mistral 7 Discussion 7.1 A more general framework While ASTRA maximizes the attention on the payload, it can be generalized to distances between attention matrices. Our attention loss defined in Eq. (6) can be written as AttLoss (l) i (x, y) = ∑ j∈J 1 |J| − A (l) i (x)[n][ j] This can be interpreted as measuring the difference be- tween an “ideal” attention pattern (which treats all tokens in the payload equally) and the “true” attention pattern. Since rows of attention matrices have multiple interpretations as probability masses and convex combinations (Section 5), in- stead of looking at the difference between attention values, we could use more sophisticated distances such as Wasserstein distances to craft attacks. We leave the exploration of such attacks to future work. 7.2 Limitations of ASTRA Performance. The primary limitation of ASTRA is its memory footprint and performance. ASTRA relies on computing on attention matrices during a model’s forward and backward passes, which is a memory intensive operation. As a result, there’s significant slowdown in the entire attack. We found ASTRA to be roughly1.7− 2.5×slower than GCG. We leave the question of time efficiency to future work. Dynamic Prompt Injections. We evaluated ASTRA and AS- TRA++ for the simple case where the target string is a fixed string. In general, prompt injections can require dynamically generated target strings such as when the attacker wants to leak private data based on the conversation history. We leave the extension of our attack to dynamic target strings for future work. 7.3 Evaluation of Prompt Injection Defenses Our results (Section 6) show that even crude attacks can break prompt injection defenses with increased budgets. We propose that prompt injection defenses should, instead of evaluating at one single budget, show their robustness under scaling as that better captures the realities of the threat model. 8 Related Work Jailbreaks. Jailbreaks are attacks where attackers try to cause the model to output content forbidden by its content safety pol- icy. While several approaches exist across white-box, black- box and grey-box threat models [13, 22, 30], a major class of algorithms optimize the input to force the model to begin its answer with a target string such as “Sure". In the whitebox setting, the SOTA solution to this problem is achieved by GCG and its variants and enhancements, such as Momentum- GCG [39], AutoDAN [41], and Accelerated GCG [40]. Optimization-based Prompt Injections. Since the optimiza- tion problem to be solved for jailbreaking is similar to a prompt injection, GCG has also been adapted for generating prompt injections in the whitebox setting. NeuralExec [26] and Universal [18] use versions of GCG to generate prompt injections. Imprompter [9] used GCG to leak private data. Crucially, they all fundamentally rely on gradients of target logprobs to guide their optimizations. In contrast, ASTRA uses the gradients in an architecture-aware way, by incorporating them into the weights for attention heads. In addition, our approach is specifically tailored for prompt injections, which allows us to achieve high success rates even in a realistic threat model. Attacks manipulating Attention. Attn-GCG [34] uses atten- tion matrices to craft jailbreaks. Our work focuses on attacks against fine-tuning-based prompt injection defenses. There are several key differences between ASTRA and Attn-GCG. First, their loss function adds a regularization term to the stan- dard GCG loss. In contrast, our algorithm uses attention as a warm start for GCG. Second, they focus attention on the gibberish adversarial suffix, whereas, ASTRA focuses atten- tion on human-readable prompt injection instructions. Third, Attn-GCG only considers the last decoder layer without ap- propriately chosen weights. As we show in Fig. 7, this doesn’t work well. Thus, Attn-GCG shows only marginal improve- ments. By contrast, ASTRA shows much higher performance on injection tasks. In the vision domain, AICAttack [16] uses attention to fool captioning algorithms. Guo et al. also use attention to craft backdoor attacks in vision transformers and BERT-based encoder models [12, 21]. We leave attacking multi-modal lan- guage models to future work, though we expect our techniques to transfer. Our work focuses on text LLMs that are the most widely-deployed form of language models today. 13 9 Conclusion Prompt injection attacks are a major risk against LLM- integrated systems. Fine-tuning based defenses teach LLMs to distinguish instructions and data. We provided an adap- tive security analysis of this class of defenses by analyzing three recent representative systems (StruQ, SecAlign and Se- cAlign++). While these systems included an adaptive security analysis, they were limited by weak attacks based on Greedy Co-ordinate Gradient. We showed that such attacks make in- effective use of whitebox access to the model. We created the ASTRA attack that uses architectural information about the model. ASTRA shows high success rates against these de- fenses, both in the original evaluation setting and an extended evaluation setting that involves universality across prompts. We believe that our work has set a foundation for the correct evaluation of fine-tuning based prompt injection defenses. References [1] Anish Athalye, Nicholas Carlini, and David Wagner. Ob- fuscated gradients give a false sense of security: Circum- venting defenses to adversarial examples, 2018. [2] Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models.arXiv preprint arXiv:2404.13161, 2024. [3]Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Con- ference on Learning Representations, 2023. [4] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness, 2019. [5]Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries, 2024. [6]Sizhe Chen, Arman Zharmagambetov, Saeed Mahlou- jifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. Secalign: Defending against prompt injection with preference optimization, 2025. [7]Sizhe Chen, Arman Zharmagambetov, David Wagner, and Chuan Guo. Meta secalign: A secure foundation llm against prompt injection attacks, 2025. [8] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. [9]Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Ra- jesh K. Gupta, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Imprompter: Tricking llm agents into im- proper tool use, 2024. [10]Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K. Gupta, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, and Earlence Fernandes. Misusing tools in large language models with visual adversarial examples, 2023. [11]Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injec- tion, 2023. [12] Ji Guo, Hongwei Li, Wenbo Jiang, and Guoming Lu. Backdoor attack against vision transformers via atten- tion gradient-based image erosion, 2024. [13]Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query-based adversar- ial prompt generation, 2024. [14] Xiaojun Jia, Tianyu Pang, Chao Du, Yihao Huang, Jin- dong Gu, Yang Liu, Xiaochun Cao, and Min Lin. Im- proved techniques for optimization-based jailbreaking on large language models, 2024. [15]Andrey Labunets, Nishit Pandya, Ashish Hooda, Xiao- han Fu, and Earlence Fernandes. Fun-tuning: Char- acterizing the vulnerability of proprietary llms to optimization-based prompt injection attacks via the fine- tuning interface. In 2025 IEEE Symposium on Security and Privacy (SP), pages 374–392. IEEE Computer So- ciety, 2025. [16] Jiyao Li, Mingze Ni, Yifei Dong, Tianqing Zhu, Yong- shun Gong, and Wei Liu. Aicattack: Adversarial image captioning attack with attention-based optimization. Ma- chine Intelligence Research, 22:1–14, 2025. [17]Zeyi Liao and Huan Sun. Amplegcg: Learning a uni- versal and transferable generative model of adversar- ial suffixes for jailbreaking both open and closed llms, 2024. [18]Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models, 2024. [19]Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses, 2024. 14 [20]Yupei Liu, Yuqi Jia, Jinyuan Jia, Dawn Song, and Neil Zhenqiang Gong. Datasentinel: A game-theoretic detection of prompt injection attacks, 2025. [21]Weimin Lyu, Songzhu Zheng, Lu Pang, Haibin Ling, and Chao Chen. Attention-enhancing backdoor attacks against BERT-based models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Associa- tion for Computational Linguistics: EMNLP 2023, pages 10672–10690, Singapore, December 2023. Association for Computational Linguistics. [22]Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically, 2024. [23] MetaAI.Promptguard86m:purplel- lama prompt-injection and jailbreak detector.https:// huggingface.co/meta-llama/Prompt-Guard-86M, 2024. Model card, accessed 24 May 2025. [24] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Con- erly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. [25]OpenAI. Introducing the gpt store, jan 2024. Accessed: Y-M-D. [26] Dario Pasquini, Martin Strohmeier, and Carmela Tron- coso. Neural exec: Learning (and learning from) execu- tion triggers for prompt injection attacks, 2024. [27]Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms, 2025. [28] Mary Phuong and Marcus Hutter. Formal algorithms for transformers, 2022. [29] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. 2023. [30] Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models, 2024. [31]SnykSec. Agent hijacking: The true impact of prompt injection attacks. https://dev.to/snyk/agent-hijacking- the-true-impact-of-prompt-injection-attacks-983, 2024. [Accessed 23-09-2024]. [32]Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses, 2020. [33]Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Jo- hannes Heidecke, and Alex Beutel. The instruction hier- archy: Training llms to prioritize privileged instructions, 2024. [34] Zijun Wang, Haoqin Tu, Jieru Mei, Bingchen Zhao, Yisen Wang, and Cihang Xie. Attngcg: Enhancing jail- breaking attacks on llms with attention manipulation, 2024. [35]Wunderwuzzi. Zombais: From prompt injection to c2 with claude computer use, October 2024. Blog post on *Embrace The Red*. [36]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representa- tions (ICLR), 2023. [37]Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning?, 2025. [38]Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024. [39]Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum, 2025. [40]Yiran Zhao, Wenyue Zheng, Tianle Cai, Xuan Long Do, Kenji Kawaguchi, Anirudh Goyal, and Michael Shieh. Accelerating greedy coordinate gradient and general prompt optimization via probe sampling, 2024. [41]Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023. [42]Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and trans- ferable adversarial attacks on aligned language models, 2023. [43]Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, and Christoph H. Lampert. Can LLMs separate instructions from data? and what do we even mean by that? In The Thirteenth International Conference on Learning Representations, 2025. A Full Algorithm - ASTRA++ 15 Algorithm 3: UnivGenGCG p,B,N (D, r (0) , y, Loss) Input: Training dataset of contexts D =x Trusted ∥x Payload Input: Initial adversarial perturbation r (0) ∈ V |r (0) | Input: Target sequence of tokens y = (y 1 , y 2 ,..., y m )∈ V m Input: A function Loss: V ∗ × V m → R Parameters: p∈ N — number of top candidates to consider; B∈ N — number of evaluations per step; N∈ N — number of optimization steps Output: List of Perturbations obtained during optimization and their corresponding Average Logprobs begin AvgLogprobs← [ ] ; Perturbations← [ ] ; for k← 0 to N− 1 do foreach i∈|r (0) | do R (k) i ← Top-p − ∑ x∈D ∇ e x i Loss(x∥r (k) ,y) ∥∇ e x i Loss(x∥r (k) ,y)∥ end for b← 1 to B do i∼ Uniform(1, 2,...,|r|) ̃ r (k) b ← r (k) ̃ r (k) b [i]∼ Uniform(R (k) i ) end b ∗ ← arg min b ∑ x ∈D Loss(x∥ ̃ r (k) b , y) r (k+1) ← ̃ r (k) b ∗ AvgLogprobs[k]← Logprobs D,y (r (k+1) ) Perturbations[k]← r (k+1) end return Perturbations, AvgLogprobs end Algorithm 4: ASTRA++ p,B,F,s 1 ,s 2 (D, r (0) , y) Input: Training dataset of contexts D =x Trusted ∥ x Payload Input: Initial perturbation r (0) Input: Target tokens y = (y 1 , y 2 ,..., y m )∈ V m Parameters: p∈ N — number of top candidates to consider; B∈ N — number of evaluations per step; F ∈ N — number of local sensitivity recomputations; s 1 — number of steps optimizing each local attention loss;s 2 — number of steps optimizing the final target logprobs. Output: Optimized perturbation r ∗ begin AllAvgLogprobs← [ ] ; AllPerturbations← [ ] ; for c∈1, 2,..., F do Perturbations c , AvgLogprobs c ← UnivGenGCG p,B,s 1 D, r ∗ c−1 , y,AttLoss r ∗ c ,D r ∗ c ← Perturbations c [−1] ; AllAvgLogprobs += AvgLogprobs c ; AllPerturbations += Perturbations c ; end r ∗ ← AllPerturbations[arg min AllAvgLogprobs] ; Perturbations, AvgLogprobs← UnivGenGCG p,B,s 2 (D, r ∗ , y, TargetLogprobs) ; r ∗ ← Perturbations[arg min AvgLogprobs] ; return r ∗ end 16

Instant research discovery