Paper deep dive

Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

Haoyu Liang, Youran Sun, Yunfeng Cai, Jun Zhu, Bo Zhang

Year: 2025Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 101

Models: Qwen2.5-0.5B, sentence-t5-base

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 6:21:48 PM

Summary

The paper identifies a security vulnerability in text embedding models used as LLM safeguards, caused by an uneven output distribution. The authors propose 'universal magic words'—adversarial suffixes that shift text embeddings toward a bias direction—to manipulate similarity scores and jailbreak safeguards. They introduce efficient search methods for these words and propose train-free defense mechanisms to correct the embedding bias.

Entities (4)

Text Embedding Models · technology · 95%Universal Magic Words · adversarial-technique · 95%LLM Safeguards · security-mechanism · 92%JailBreakBench · benchmark · 90%

Relation Signals (3)

Universal Magic Words → attacks → Text Embedding Models

confidence 95% · propose novel, efficient methods to search for universal magic words that attack text embedding models.

Universal Magic Words → jailbreaks → LLM Safeguards

confidence 95% · Attackers can jailbreak the safeguards by appending magic words to user prompts

Defense Methods → mitigates → Universal Magic Words

confidence 90% · we also propose defense methods against such attacks, which can correct the bias of text embeddings

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

100,255 characters extracted from source content.

Expand or collapse full text

Jailbreaking LLMs’ Safeguard with Universal Magic Words for Text Embedding Models Haoyu Liang 1 Youran Sun11footnotemark: 1 2 Yunfeng Cai 3 Jun Zhu 1 Bo Zhang 1 1 Dept. of Comp. Sci. and Tech., Inst. for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China 2Department of Mathematical Sciences, Tsinghua University 3BIMSA, Beijing, China hyliang96@gmail.com syouran0508@gmail.com caiyunfeng@bimsa.cn dcszj,dcszb@tsinghua.edu.cn Equal contribution.Corresponding author. Abstract The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for universal magic words that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner. 1 Introduction Recently, large language models (LLMs) have been widely applied in the industry, such as chat systems [1] and search engines [2]. However, LLMs can be maliciously exploited to extract harmful output, making LLM security an important research topic. In this topic, it is of great significance to discover security vulnerabilities of text embedding models and propose corresponding defense methods. Current LLM security strategies include alignment [3] and safeguards [4]. Lightweight text classifiers based on text embedding models [5] can be used as safeguards to judge whether the input and output of LLMs are harmful. This method can serve as a foundational line of defense because it is low-cost while maintaining the performance of LLMs. In addition, text embedding models are also used to enhance modern search engines [2]. Therefore, the robustness of text embedding models affects the security of both LLMs and search engines. Attacking LLMs’ safeguards is challenging because the output of LLMs is unknown, the safeguards are black boxes, and the token space is vast and discrete. This results in the following limitations of existing attack methods on text embedding models: 1) Case-by-case attack methods require access to LLMs’ output before safeguards, which is unrealistic for online dialogue systems; 2) White-box attack methods require the gradients of text embedding models, which are also unrealistic; 3) Brute-force search for prompt perturbations requires traversing a massive token space, leading to high time costs. To address these challenges, we propose an innovative approach to attack LLMs’ safeguards based on text embedding models: to find universal “magic words” (i.e., adversarial suffixes) that would increase or decrease the embedding similarity between any pair of texts so as to mislead the safeguards in classifying within the text embedding space. Figure 1: The distribution of cosine similarity between text embedding e⁢(s)e(s)e ( s ) of text s with normalized mean embedding e∗superscripte^*e∗ of all text, tested on various text embedding models. This task is feasible based on the following observation. We tested various text embedding models and found that the cosine similarities between text embeddings and their mean (normalized) concentrate near a significant positive value, as shown in Fig. 1. In other words, text embeddings do not distribute uniformly on a high-dimensional sphere Sd−1superscript1S^d-1Sitalic_d - 1(since they are normalized); instead, they concentrate in a band on the sphere, as illustrated in Fig. 2. The direction of distribution bias is similar to all text embeddings, while the opposite direction is dissimilar to all embeddings. This implies that if we can find suffixes that push any text towards this bias direction, we can increase the similarity of any text with other texts. Similarly, one could also try to find suffixes that reduce text similarity. We refer to these words as universal magic words since they can mislead safeguards on any text by manipulating text similarity. We estimate the bias direction with the mean and the principal singular vector of text embeddings (see Sec. 3.1). Actual tests and theoretical analysis show that the two methods yield the same results. Based on the identified bias direction, we use the following methods to find universal magic words (see Sec. 3.2). Alg. 1: brute-force search without leveraging the bias direction, used as a baseline; Alg. 2 (black-box): find words whose text embeddings are as similar/dissimilar as possible to the bias direction; Alg. 3 (white-box): find universal suffixes that push any text embedding closer to the bias direction or far away from its original position. Alg. 3 uses gradients to solve this problem in only one epoch. Experiments show that all three methods can find the best magic words, but Algs. 2 and 3 are far more efficient than Alg. 1. Additionally, only Alg. 3 can search for multi-token magic words. The universal magic words can be abused to attack safeguards in LLM security systems. As shown in Fig. 3, the safeguards will fail to detect harmful content by appending magic words to the input and output of LLMs. Contributions. The contribution of this paper can be summarized as follows: • We discover that the output distribution of text embedding models is uneven and the relationship between this property and universal magic words; • We propose novel methods for finding universal magic words, which are efficient and capable of searching for multi-token magic words; • We demonstrate that those universal magic words are able to jailbreak LLMs’ safeguards and generalize across input/output texts, models, and languages (see experiments in Secs. 4.3, 4.4 and 4.5). • We propose defense methods against such attacks by correcting the uneven embedding distribution. 2 Related Work 2.1 Defense Methods for LLMs Alignment involves training LLMs to align with human values [6, 7, 3]. This method is widely used because it does not introduce additional computational overhead during inference. Due to the competition between assisting users and aligning values, as well as the limited domain of safety training [8], such methods are vulnerable to adversarial attacks [9, 10]. This has forced people to develop additional security measures. Safeguards are the additional measures on the input or output of LLMs to avoid harmful responses. On the input side, there are several guard measures: 1) Detecting suspicious patterns [11, 12], which tends to yield false positives; 2) Reminding LLMs to align values with system prompts [13, 14, 15], which can be canceled by the user prompt “ignore previous instructions” [16]; 3) Perturbing the user’s prompt into multiple versions before feeding it to the LLM to detect harmful requests [17, 18], which is costly; 4) Classifying whether the prompt is harmful with a model [5]. On the output side, several detection methods for LLMs’ harmful responses serve as the last line of defense in LLM security systems: 1) rule-based matching, with the same drawbacks as it is on the input side; 2) another LLM to answer whether the output is harmful [19, 20, 21], which doubles the cost; 3) alternatively, text classifiers to do this [22, 5, 23], which is more cost-effective. 2.2 Attack Methods for LLMs Templates jailbreak LLMs with universal magic words effective for various prompts, some even transferable across LLMs. Manual templates are heuristically designed, including explicit templates (e.g., instructing LLMs to “ignore previous instructions” [16], “Start with ‘Absolutely! Here’s”’ [24] or “Do anything now” [24]) and implicit templates (e.g., role-playing [25, 26], storytelling [27] and virtual scenarios [28, 29, 30, 31]). Automatic templates are optimized by gradient descent (black-box) [32, 9, 33], random search (white-box) [34, 35], or generative models [36] to find adversarial prefixes and suffixes for user prompts. These prefixes and suffixes could be individual words or sentences [9], and comprehensible [36] or not [34]. Rewriting attacks language models at several levels, including character-level (e.g., misspelling [37]), word-level (e.g., synonyms [38]), segment-level (e.g., assigning variable names to segmented harmful text [39, 29]), prompt-level (e.g., rewriting prompts with an LLM [10, 40, 41, 42]), language-level (e.g., translating into a language that lacks LLM safety [43]), and encoding-level (e.g., encoding harmful text into ASCII, Morse code [44] or Base64 [45]). Through optimization algorithms, attackers can automatically find the most effective rewrites to bypass the LLM’s safeguards. The methods above are all focused on attacking the LLM itself, while research on attacking safeguards is still in its early stages. A magic word “lucrarea” was discovered by the champion of a Kaggle competition on attacking LLMs [46], through trying the tokens near </s> in the token embedding space. We find many more magic words, including “lucrarea”, with our novel algorithms and give a more accurate and systematic explanation of why it works. Similar to our method, PRP [47] attacks output guards by injecting magic words into LLMs’ responses. The distinctions between our work and PRP are: 1) we attack guards based on text embedding models, which are more lightweight and cost-effective than LLM-based guards in PRP; 2) we discovered the uneven distribution of text embeddings, which allows us to design algorithms to search for magic words more efficiently. 3 Method Notation: 1) Let s1subscript1s_1s1 and s2subscript2s_2s2 be two text strings, and let r be a positive integer. The operation s1+s2subscript1subscript2s_1+s_2s1 + s2 denotes the concatenation of s1subscript1s_1s1 and s2subscript2s_2s2, and r∗s2subscript2r*s_2r ∗ s2 denotes the string s2subscript2s_2s2 repeated r times. 2) For example, if s1=`⁢`⁢h⁢e⁢"subscript1``ℎ"s_1=``he"s1 = ` ` h e ", s2=`⁢`⁢l⁢l⁢o⁢"subscript2``"s_2=``llo"s2 = ` ` l l o ", then s1+s2=`⁢`⁢h⁢e⁢l⁢l⁢o⁢"subscript1subscript2``ℎ"s_1+s_2=``hello"s1 + s2 = ` ` h e l l o " and s1+2∗s2=`⁢`⁢h⁢e⁢l⁢l⁢o⁢l⁢l⁢o⁢"subscript12subscript2``ℎ"s_1+2*s_2=``hellollo"s1 + 2 ∗ s2 = ` ` h e l l o l l o ". Denote the text embedding of text string s by e⁢(s)e(s)e ( s ) and its dimension by d. e⁢(s)e(s)e ( s ) is normalized to a unit vector, hence e⁢(s)∈Sd−1superscript1e(s)∈ S^d-1e ( s ) ∈ Sitalic_d - 1. The text embedding e⁢(s)e(s)e ( s ) of s is computed as e⁢(s)=⁢(),=E⊤⁢τ⁢(s)formulae-sequencesuperscripttope(s)=e( s), s=E τ(s)e ( s ) = e ( italic_s ) , italic_s = E⊤ τ ( s ). Here, ∈ℝh×lsuperscriptℝℎ s ^h× litalic_s ∈ blackboard_Rh × l denotes the representation of s in the token embedding space, which is mapped to a text embedding by ee. Moreover, τ is a tokenizer that splits s into l tokens, outputting τ⁢(s)∈0,1T×lsuperscript01τ(s)∈\0,1\^T× lτ ( s ) ∈ 0 , 1 T × l where the columns are one-hot. =tiisubscriptsubscriptT=\t_i\_iT = titalic_i i is the token vocabulary, with size ||=T|T|=T| T | = T. E∈ℝT×hsuperscriptℝℎE ^T× hE ∈ blackboard_RT × h denotes the token embeddings of all tokens, with dimension hℎh. 3) The cosine similarity between text s1subscript1s_1s1 and s2subscript2s_2s2 is defined as cos⁡θ⁢(s1,s2):=e⁢(s1)⊤⁢e⁢(s2)assignsubscript1subscript2superscriptsubscript1topsubscript2 θ(s_1,s_2):=e(s_1) e(s_2)cos θ ( s1 , s2 ) := e ( s1 )⊤ e ( s2 ). This paper aims to find all possible universal magic words, which can be formulated as follows. Assumption 3.1. There exists a word w+superscriptw^+w+ satisfying that cos⁡θ⁢(s1+w+,s2)≥cos⁡θ∗,∀s1,s2subscript1superscriptsubscript2subscriptfor-allsubscript1subscript2 θ(s_1+w^+,s_2)≥ _*, ∀ s_1,s_2cos θ ( s1 + w+ , s2 ) ≥ cos θ∗ , ∀ s1 , s2, where cos⁡θ∗subscript _*cos θ∗ is close to 1111. We refer to w+superscriptw^+w+ as a positive universal magic word for the text embedding model e, which can force any pair of texts to be similar enough in the text embedding space. 3.1 Description of the Uneven Direction To describe the unevenness of the text embedding distribution, we represent the bias direction of the distribution by the normalized mean of text embeddings e∗superscripte^*e∗ or the principal singular vector v∗superscriptv^*v∗ of the text embedding matrix. We prove that any text appended by a positive universal magic word w+superscriptw^+w+ will be close to e∗superscripte^*e∗ (or v∗superscriptv^*v∗). This serves as the guiding principle for searching for magic words in Sec. 3.2. We denote the mean of text embeddings as e¯=1||⁢∑je⁢(sj)¯1subscriptsubscript e= 1|S| _je(s_j)over¯ start_ARG e end_ARG = divide start_ARG 1 end_ARG start_ARG | S | end_ARG ∑j e ( sitalic_j ) and the normalized mean as e∗=e¯‖e¯‖2superscript¯subscriptnorm¯2e^*= e\| e\|_2e∗ = divide start_ARG over¯ start_ARG e end_ARG end_ARG start_ARG ∥ over¯ start_ARG e end_ARG ∥2 end_ARG, where =sjjsubscriptsubscriptS=\s_j\_jS = sitalic_j j is the set of all possible texts. The following proposition shows that any text with a magic word will be embedded close to e∗superscripte^*e∗. Proposition 3.2. Under Assumption 3.1, a positive universal magic word w+superscriptw^+w+ must satisfy cos⁡θ⁢(e⁢(s+w+),e∗)≥1−tan2⁡θ∗,∀s∈.formulae-sequencesuperscriptsuperscript1superscript2subscriptfor-all θ(e(s+w^+),e^*)≥ 1- ^2 _*, ∀s∈% S.cos θ ( e ( s + w+ ) , e∗ ) ≥ square-root start_ARG 1 - tan2 θ∗ end_ARG , ∀ s ∈ S . Denote the text embedding matrix of SS as X∈ℝ||×dsuperscriptℝX ^|S|× dX ∈ blackboard_R| S | × d, where the i-th row of X is e⁢(si)⊤superscriptsubscripttope(s_i) e ( sitalic_i )⊤. Let v∗superscriptv^*v∗ be the principal right singular vector of X corresponding to the largest singular value. The following proposition shows that any text with a magic word will be embedded close to v∗superscriptv^*v∗. Proposition 3.3. Under Assumption 3.1, a positive universal magic word w+superscriptw^+w+ must satisfy cos⁡θ⁢(e⁢(s+w+),v∗)≥1−tan2⁡θ∗,∀s∈.formulae-sequencesuperscriptsuperscript1superscript2subscriptfor-all θ(e(s+w^+),v^*)≥ 1- ^2 _*, ∀s∈% S.cos θ ( e ( s + w+ ) , v∗ ) ≥ square-root start_ARG 1 - tan2 θ∗ end_ARG , ∀ s ∈ S . See Appendix B for the proof of the two propositions. In the experiments (see Sec. 4.1), we found that e∗superscripte^*e∗ and v∗superscriptv^*v∗ are almost identical, so we will only refer to e∗superscripte^*e∗ in the subsequent sections. 3.2 Searching for Universal Magic Words Based on the observations in Sec. 3.1, we boldly presume the existence of universal magic words. When used as a suffix, universal magic words could make any text more similar or dissimilar to other texts in the embedding space. We refer to the words that increase the text similarity as positive magic words and those that decrease the text similarity as negative magic words, as shown in Fig. 2. Figure 2: Text embeddings concentrate in a band on the sphere Sd−1superscript1S^d-1Sitalic_d - 1. Positive magic words can push them towards the normalized mean e∗superscripte^*e∗. Negative magic words can pull them away from their original position. Brute-Force Method The simplest method to find magic words is a brute-force search, shown in Alg. 1. This method directly calculates the similarity score of all tokens in the vocabulary set and finds the top-k0subscript0k_0k0 magic words. This method does not rely on the bias direction. For each token tisubscriptt_ititalic_i in the token vocabulary set =tiisubscriptsubscriptT=\t_i\_iT = titalic_i i, we define the positive similarity score as ci+superscriptsubscript c_i^+citalic_i+ =max1≤r≤16⁡1S2⁢∑j,kcos⁡θ⁢(sj+r∗ti,sk)absentsubscript1161superscript2subscriptsubscriptsubscriptsubscript = _1≤ r≤ 16 1S^2 _j,k θ(s_j+r% *t_i,s_k)= max1 ≤ r ≤ 16 divide start_ARG 1 end_ARG start_ARG S2 end_ARG ∑j , k cos θ ( sitalic_j + r ∗ titalic_i , sitalic_k ) (1) =max1≤r≤16⁡1S⁢∑jcos⁡θ⁢(sj+r∗ti,e∗)absentsubscript1161subscriptsubscriptsubscriptsuperscript = _1≤ r≤ 16 1S _j θ(s_j+r*t_i% ,e^*)= max1 ≤ r ≤ 16 divide start_ARG 1 end_ARG start_ARG S end_ARG ∑j cos θ ( sitalic_j + r ∗ titalic_i , e∗ ) (2) Tokens with higher positive scores are more effective as positive magic words. r represents the repetition count. Repeating the magic word usually amplifies its effect. However, we limit r to a maximum of 16 to avoid completely distorting the text. Finding negative magic words requires more data. Specifically, in addition to the text sjsubscripts_jsitalic_j, we also need another piece of text sj′subscriptsuperscript′s _js′italic_j that is semantically similar to sjsubscripts_jsitalic_j but phrased differently. This is because the effect of a negative magic word is to make synonymous text no longer synonymous. Now the set of text pairs is in the form ~=(sj,sj′)j~subscriptsubscriptsubscriptsuperscript′ S=\(s_j,s _j)\_jover~ start_ARG S end_ARG = ( sitalic_j , s′italic_j ) j with cos⁡θ⁢(sj,sj′)subscriptsubscriptsuperscript′ θ(s_j,s _j)cos θ ( sitalic_j , s′italic_j ) close to 1. We define the negative similarity score of tisubscriptt_ititalic_i as ci−=min1≤r≤16⁡1S⁢∑jcos⁡θ⁢(sj+r∗ti,sj′).superscriptsubscriptsubscript1161subscriptsubscriptsubscriptsubscriptsuperscript′c_i^-= _1≤ r≤ 16 1S _j θ(s_j+r*t_i,s^% _j).citalic_i- = min1 ≤ r ≤ 16 divide start_ARG 1 end_ARG start_ARG S end_ARG ∑j cos θ ( sitalic_j + r ∗ titalic_i , s′italic_j ) . (3) The lower negative similarity score indicates the greater effectiveness of magic words in making synonymous text dissimilar. Algorithm 1 Brute-Force Method Input: text set ~~ Sover~ start_ARG S end_ARG, vocabulary set TT, number of magic words k0subscript0k_0k0 for tisubscriptt_ititalic_i in TT do ci+←max1≤r≤16⁢∑jcos⁡θ⁢(sj+r∗ti,e∗)←superscriptsubscriptsubscript116subscriptsubscriptsubscriptsuperscriptc_i^+← _1≤ r≤ 16 _j θ(s_j+r*t_i,e^*)citalic_i+ ← max1 ≤ r ≤ 16 ∑j cos θ ( sitalic_j + r ∗ titalic_i , e∗ ) ci−←min1≤r≤16⁢∑jcos⁡θ⁢(sj+r∗ti,sj′)←superscriptsubscriptsubscript116subscriptsubscriptsubscriptsubscriptsuperscript′c_i^-← _1≤ r≤ 16 _j θ(s_j+r*t_i,s^% _j)citalic_i- ← min1 ≤ r ≤ 16 ∑j cos θ ( sitalic_j + r ∗ titalic_i , s′italic_j ) end for w±←topki⁢(±ci±,k0)←superscriptplus-or-minussubscripttopkplus-or-minussuperscriptsubscriptplus-or-minussubscript0w^± _i(± c_i^±,k_0)w± ← topkitalic_i ( ± citalic_i± , k0 ) Output: w±superscriptplus-or-minusw^±w± ▷ ▷ top-k0subscript0k_0k0 pos./neg. magic words Algorithm 2 Context-Free Method Input: vocabulary set TT, normalized mean e∗superscripte^*e∗, repetition count r, candidate number k for tisubscriptt_ititalic_i in TT do ci←e⁢(r∗ti)⊤⁢e∗←subscriptsuperscriptsubscripttopsuperscriptc_i← e(r*t_i) e^*citalic_i ← e ( r ∗ titalic_i )⊤ e∗ end for ±←topki⁢(±ci,k)←superscriptplus-or-minussubscripttopkplus-or-minussubscriptT^± _i(± c_i,k)T± ← topkitalic_i ( ± citalic_i , k ) ▷ ▷ candidate list, size=k w±←Algorithm1⁢(~,±,k0)←superscriptplus-or-minusAlgorithm1~superscriptplus-or-minussubscript0w^± 1( S,T^±,k_% 0)w± ← Algorithm1 ( over~ start_ARG S end_ARG , T± , k0 ) ▷ ▷ k>k0subscript0k>k_0k > k0 Output: w±superscriptplus-or-minusw^±w± ▷ ▷ top-k0subscript0k_0k0 pos./neg. magic words Context-Free Method As demonstrated previously, all text embeddings tend to be close to e∗superscripte^*e∗ and far from −e∗superscript-e^*- e∗. Intuitively, tokens whose text embeddings have the same direction as e∗superscripte^*e∗ are likely to be positive magic words, and vice versa. Specifically, for a given ti∈subscriptt_i _i ∈ T, we select the top-k and bottom-k tokens as candidates for positive and negative magic words based on the following score ci=e⁢(r∗ti)⊤⁢e∗,subscriptsuperscriptsubscripttopsuperscriptc_i=e(r*t_i) e^*,citalic_i = e ( r ∗ titalic_i )⊤ e∗ , (4) where r denotes the repetition count, set between 3 and 5. After this raw selection, we perform a refined selection from the candidates using Alg. 1. This method is formulated in Alg. 2. Gradient-Based Method The above two methods are not able to search for multi-token magic words and do not leverage first-order information. What if we can access all the model parameters (white-box setting) and wish to leverage gradients? Let’s formulate the problem more specifically. The positive magic word we aim to find (denoted as w, consisting of m tokens) maximizes the following objective argmaxw⁢∑jcos⁡θ⁢(sj+w,e∗)subscriptargmaxsubscriptsubscriptsuperscript argmax_w _j θ(s_j+w,e^*)argmaxw ∑j cos θ ( sitalic_j + w , e∗ ). Unlike adversarial attacks in computer vision, the vocabulary’s discreteness introduces significant optimization challenges. To address this, we split the optimization into two steps. In the first step, we search for the optimal token embeddings ∗∈ℝh×msuperscriptsuperscriptℝℎ t^* ^h× mitalic_t∗ ∈ blackboard_Rh × m by solving ∗=argmax⁢∑j⁢([j,])⊤⁢e∗.superscriptsubscriptargmaxsubscriptsuperscriptsubscripttopsuperscript t^*= argmax_ tΣ _je% ([ s_j, t]) e^*.italic_tbold_∗ = argmaxitalic_t ∑j e ( [ italic_sitalic_j , italic_t ] )⊤ e∗ . (5) In the second step, we identify the token in each position whose embedding is closest to the optimal. Assuming that ⁢([,])e([ s, t])e ( [ italic_s , italic_t ] ) is close to ⁢([,])0e([ s, 0])e ( [ italic_s , 0 ] ), Eq. 5 can be approximated by a first-order expansion as argmax⁢∑j(⁢([j,])+J⁢(j)⁢)⊤⁢e∗=argmax⁢⊤⁢(∑jJ⁢(j))⊤⁢e∗subscriptargmaxsubscriptsuperscriptsubscript0subscripttopsuperscriptsubscriptargmaxsuperscripttopsuperscriptsubscriptsubscripttopsuperscript argmax_ tΣ _j (e([ % s_j, 0])+J( s_j) t ) e^*% = argmax_ t t (Σ _j% J( s_j) ) e^*argmaxitalic_t ∑j ( e ( [ italic_sitalic_j , 0 ] ) + J ( italic_sitalic_j ) italic_t )⊤ e∗ = argmaxitalic_t italic_t⊤ ( ∑j J ( italic_sitalic_j ) )⊤ e∗, where J⁢(j):=∂⁢([j,])assignsubscriptsubscriptsubscriptJ( s_j):= _ te([ s_j% , t])J ( italic_sitalic_j ) := ∂italic_t e ( [ italic_sitalic_j , italic_t ] ) denotes the Jacobian of the model ee at jsubscript s_jitalic_sitalic_j. The solution to the above problem is ∗∝(∑jJ⁢(j))⊤⁢e∗proportional-tosuperscriptsuperscriptsubscriptsubscripttopsuperscript t^* ( _jJ( s_j) ) e^*italic_t∗ ∝ ( ∑j J ( italic_sitalic_j ) )⊤ e∗. Interestingly, this ∗superscript t^*italic_t∗ is exactly the gradient of the following objective function L+=∑jcos⁡θ⁢(sj+t,e∗)superscriptsubscriptsubscriptsuperscriptL^+= _j θ(s_j+t,e^*)L+ = ∑j cos θ ( sitalic_j + t , e∗ ) with respect to titalic_t. In other words, our method performs gradient ascent on L+superscriptL^+L+ in just one epoch. A similar conclusion also holds for negative magic words with the following objective function L−=∑jcos⁡θ⁢(sj+t,sj′)superscriptsubscriptsubscriptsubscriptsuperscript′L^-= _j θ(s_j+t,s _j)L- = ∑j cos θ ( sitalic_j + t , s′italic_j ). This leads to the algorithm described in Alg. 3. Like Alg. 2, we first obtain k candidates with the method above and then use Alg. 1 to identify the best k0subscript0k_0k0 magic words. Algorithm 3 Gradient-Based Method Input: text set ~~ Sover~ start_ARG S end_ARG, vocabulary set TT, normalized mean e∗superscripte^*e∗, magic word length m, candidate number k ∗±←zeros⁢(h,m)←superscriptabsentplus-or-minuszerosℎ t^*±← zeros(h,m)italic_t∗ ± ← zeros ( h , m ) for sjsubscripts_jsitalic_j in SS do ←rand⁢(h,m)←randℎ t← rand(h,m)italic_t ← rand ( h , m ) ▷ ▷ empirically better than zeros(h,m) L+←⁢(j+)⊤⁢e∗←superscriptsuperscriptsubscripttopsuperscriptL^+ ( s_j+ t) e^*L+ ← e ( italic_sitalic_j + italic_t )⊤ e∗ L−←⁢(j+)⊤⁢e⁢(sj′)←superscriptsuperscriptsubscripttopsuperscriptsubscript′L^- ( s_j+ t) e(s_j^% )L- ← e ( italic_sitalic_j + italic_t )⊤ e ( sitalic_j′ ) ∗±←∗±∂L±/∂←superscriptabsentplus-or-minusplus-or-minussuperscriptabsentplus-or-minussuperscriptplus-or-minus t^*±← t^*±∂ L^±/% ∂ titalic_t∗ ± ← italic_t∗ ± ± ∂ L± / ∂ italic_t end for▷ ▷ ∗±superscriptabsentplus-or-minus t^*±italic_t∗ ± is the optimal m-token embedding [1±,…,m±]=getWord⁢(topk⁢(±E⁢∗,k,dim=0))subscriptsuperscriptplus-or-minus1…subscriptsuperscriptplus-or-minusgetWordtopkplus-or-minussuperscriptdim0[T^±_1,...,T^±_m]=getWord(% topk(± E t^*,k,dim=0))[ T±1 , … , T±italic_m ] = getWord ( topk ( ± E italic_t∗ , k , dim = 0 ) ) ▷ ▷ u±subscriptsuperscriptplus-or-minusT^±_uT±italic_u contains k candidates for u-th token ±=1±×…×m±superscriptplus-or-minussuperscriptsubscript1plus-or-minus…superscriptsubscriptplus-or-minusT^±=T_1^±×...×T_m^±T± = T1± × … × Titalic_m± ▷ ▷ candidate list, size=kmsuperscriptk^mkitalic_m w±←Algorithm1⁢(~,±,k0)←superscriptplus-or-minusAlgorithm1~superscriptplus-or-minussubscript0w^± 1( S,T^±,k_% 0)w± ← Algorithm1 ( over~ start_ARG S end_ARG , T± , k0 ) ▷ ▷ km>k0superscriptsubscript0k^m>k_0kitalic_m > k0 Output: w±superscriptplus-or-minusw^±w± ▷ ▷ top-k0subscript0k_0k0 pos./neg. magic words Table 1: Comparing different methods Methods Alg. 1 Alg. 2 Alg. 3 Speed Slow Fast Fast White/Black Box Black Black White Multi-token No No Yes As a summary of this section, Table 1 compares the three methods in terms of speed, scenario (black-box/white-box), and their ability to search for multi-token magic words. 3.3 Attacking LLMs’ Safeguard As shown in Fig. 3, we can append magic words to the prompt to attack the input guard of LLMs directly and require the LLM to end answers with magic words to attack the output guard indirectly. Figure 3: Pipeline to attack the safeguard of LLMs. The input guard is attacked directly by appending universal magic words to user prompts, and the output guard is indirectly attacked by requiring LLMs to append universal magic words to their output. This method works by moving text embedding to where the safeguard fails. As shown in Fig. 2, the data manifold in text embedding space is a band on the sphere. Positive magic words can push the text embedding towards e∗superscripte^*e∗, i.e., along the normal direction of the manifold, and safeguards fail to work properly outside the manifold due to the lack of training data. Negative magic words can push the embedding of a harmful text far away from its original region of harmful semantics, leading to misclassification. Besides jailbreaking the safeguard of LLMs, universal magic words may also be used to manipulate search rankings. Since most modern search engines are enhanced by text embedding models [2], abusers can increase the embedding similarity between their entries with any queries by inserting magic words into their entries. 4 Experiments We tested our method on several state-of-the-art models from the MTEB text embedding benchmark [48], including sentence-t5-base [49], nomic-embed-text-v1 [50], e5-base-v2 [51], jina-embeddings-v2-base-en [52], gte-Qwen2-7B-instruct [53], SFR-Embedding-Mistral [54], and e5-mistral-7b-instruct [55]. Additionally, considering that LLMs are sometimes used as text embedding models, we also tested Qwen2.5-0.5B [56] with mean pooling. We used sentence-transformers/simple-wiki [57] as the text dataset ~=(si,si′)i~subscriptsubscriptsubscriptsuperscript′ S=\(s_i,s _i)\_iover~ start_ARG S end_ARG = ( sitalic_i , s′italic_i ) i, where sisubscripts_isitalic_i is an English Wikipedia entry, and si′subscriptsuperscript′s _is′italic_i is its simplified variant. In Secs. 4.3 and 4.5, we also evaluated our method on JailbreakBench [58] and non-English dialogues. 4.1 Bias Direction Table 2: The overlap between the normalized mean vector e∗superscripte^*e∗ and the principal singular vector v∗superscriptv^*v∗. Model |e∗⋅v∗|⋅superscriptsuperscript|e^*· v^*|| e∗ ⋅ v∗ | sentence-t5-base 1−1.7×10−611.7superscript1061-1.7× 10^-61 - 1.7 × 10- 6 Qwen2.5-0.5B 1−1.4×10−511.4superscript1051-1.4× 10^-51 - 1.4 × 10- 5 nomic-embed-text-v1 1−2.9×10−512.9superscript1051-2.9× 10^-51 - 2.9 × 10- 5 e5-base-v2 1−0.7×10−610.7superscript1061-0.7× 10^-61 - 0.7 × 10- 6 jina-embeddings-v2-base-en 1−3.3×10−613.3superscript1061-3.3× 10^-61 - 3.3 × 10- 6 Since the whole dataset is massive, we sampled 1/10011001/1001 / 100 of all entries (sample number is 1,000) to estimate the bias direction of text embeddings. Our experiments show that when the sample number exceeds 100, the estimation for e∗superscripte^*e∗ or v∗superscriptv^*v∗ is sufficiently accurate. We found that the normalized mean vector e∗superscripte^*e∗ is almost identical to the principal singular vector v∗superscriptv^*v∗ as shown in Table 2. Appendix C explains that this is a property of biased distributions. Therefore, we only use e∗superscripte^*e∗ in the subsequent experiments. 4.2 Searching for Magic Words One-token Magic Words. In our experiments, Algs. 2 and 3 successfully find the best one-token magic words identified by the brute-force baseline Alg. 1. We demonstrate some of them in Table 3. Here, (Clean) represents the data without magic words, and the similarity cos⁡θ⁢(sj,sk)subscriptsubscript θ(s_j,s_k)cos θ ( sitalic_j , sitalic_k ) or cos⁡θ⁢(sj,sj′)subscriptsuperscriptsubscript′ θ(s_j,s_j )cos θ ( sitalic_j , sitalic_j′ ) between clean text pair is shown in the form μ±σplus-or-minusμ±σμ ± σ. The similarity score of each magic word is defined in Eqs. 1 and 3, which indicates how much it can shift the similarity. The table shows that the shift of similarity can be up to several standard deviations, which is significant. This indicates that the magic words have a strong ability to manipulate text similarity. Table 3: The magic words for different text embedding models found by all three methods and their similarity scores. Model Positive Negative magic word similarity ci+subscriptsuperscriptc^+_ic+i magic word similarity ci−subscriptsuperscriptc^-_ic-i sentence-t5-base (Clean) 0.71±0.03plus-or-minus0.710.030.71± 0.030.71 ± 0.03 (Clean) 0.96±0.04plus-or-minus0.960.040.96± 0.040.96 ± 0.04 </s> 0.79=μ+2.5⁢σ0.792.50.79=μ+2.5 0.79 = μ + 2.5 σ dumneavoastra 0.89=μ−1.7⁢σ0.891.70.89=μ-1.7 0.89 = μ - 1.7 σ lucrarea 0.79=μ+2.4⁢σ0.792.40.79=μ+2.4 0.79 = μ + 2.4 σ impossible 0.89=μ−1.6⁢σ0.891.60.89=μ-1.6 0.89 = μ - 1.6 σ Qwen2.5-0.5B (with mean pooling) (Clean) 0.81±0.08plus-or-minus0.810.080.81± 0.080.81 ± 0.08 (Clean) 0.97±0.03plus-or-minus0.970.030.97± 0.030.97 ± 0.03 Christopher 0.84=μ+0.4⁢σ0.840.40.84=μ+0.4 0.84 = μ + 0.4 σ 0.34=μ−24⁢σ0.34240.34=μ-24 0.34 = μ - 24 σ Boston 0.84=μ+0.4⁢σ0.840.40.84=μ+0.4 0.84 = μ + 0.4 σ 0.42=μ−21⁢σ0.42210.42=μ-21 0.42 = μ - 21 σ nomic-embed-text-v1 (Clean) 0.36±0.05plus-or-minus0.360.050.36± 0.050.36 ± 0.05 (Clean) 0.90±0.09plus-or-minus0.900.090.90± 0.090.90 ± 0.09 [CLS] 0.45=μ+1.7⁢σ0.451.70.45=μ+1.7 0.45 = μ + 1.7 σ sentence 0.76=μ−1.6⁢σ0.761.60.76=μ-1.6 0.76 = μ - 1.6 σ 7 0.42=μ+1.1⁢σ0.421.10.42=μ+1.1 0.42 = μ + 1.1 σ verb 0.76=μ−1.6⁢σ0.761.60.76=μ-1.6 0.76 = μ - 1.6 σ e5-base-v2 (Clean) 0.69±0.03plus-or-minus0.690.030.69± 0.030.69 ± 0.03 (Clean) 0.95±0.04plus-or-minus0.950.040.95± 0.040.95 ± 0.04 ##abia 0.71=μ+0.6⁢σ0.710.60.71=μ+0.6 0.71 = μ + 0.6 σ 0.84=μ−2.4⁢σ0.842.40.84=μ-2.4 0.84 = μ - 2.4 σ ##( 0.71=μ+0.5⁢σ0.710.50.71=μ+0.5 0.71 = μ + 0.5 σ bobbed 0.85=μ−2.2⁢σ0.852.20.85=μ-2.2 0.85 = μ - 2.2 σ jina-embeddings-v2-base-en (Clean) 0.62±0.04plus-or-minus0.620.040.62± 0.040.62 ± 0.04 (Clean) 0.94±0.05plus-or-minus0.940.050.94± 0.050.94 ± 0.05 [SEP] 0.73=μ+2.7⁢σ0.732.70.73=μ+2.7 0.73 = μ + 2.7 σ 117 0.84=μ−2.0⁢σ0.842.00.84=μ-2.0 0.84 = μ - 2.0 σ ##laze 0.65=μ+0.7⁢σ0.650.70.65=μ+0.7 0.65 = μ + 0.7 σ geometridae 0.87=μ−1.5⁢σ0.871.50.87=μ-1.5 0.87 = μ - 1.5 σ Multi-token Magic Words. Compared to the other two methods, the advantage of the Alg. 3 is its ability to search for multi-token magic words. In Table 4, we list several multi-token magic words found by Alg. 3 on the sentence-t5-base model, which also shows a strong ability to manipulate text similarity. Table 4: Multi-token magic words found by Alg. 3. Magic Word Similarity ci±subscriptsuperscriptplus-or-minusc^±_ic±italic_i pos. (Clean) 0.71±0.03plus-or-minus0.710.030.71± 0.030.71 ± 0.03 Variety ro 0.75=μ+1.1⁢σ0.751.10.75=μ+1.1 0.75 = μ + 1.1 σ Tel ro 0.74=μ+1.0⁢σ0.741.00.74=μ+1.0 0.74 = μ + 1.0 σ neg. (Clean) 0.96±0.04plus-or-minus0.960.040.96± 0.040.96 ± 0.04 Rocket autre pronounce 0.85=μ−2.5⁢σ0.852.50.85=μ-2.5 0.85 = μ - 2.5 σ bourg In claimed 0.85=μ−2.5⁢σ0.852.50.85=μ-2.5 0.85 = μ - 2.5 σ Table 5: The Efficiency of different methods on sentence-t5-base. Lower N_c (number of candidates) indicates higher efficiency. magic word method N_c Alg. 1 Alg. 2 Alg. 3 pos. </s> 32100 2 1 lucrarea 32100 1 4 neg. dumneavoastra 32100 23 279 impossible 32100 1690 189 A100 time 16h 13s 72s Efficiency. The baseline Alg. 1 takes all the T tokens in the vocabulary as candidates in its brute-force search for the best one-token magic word w, taking O⁢(T)O(T)O ( T ) time. While Algs. 2 and 3 obtain top-k candidates and then choose the best from them by a brute-force search, taking O⁢(k)O(k)O ( k ) time, which is significantly more efficient than Alg. 1 when k≪Tmuch-less-thank Tk ≪ T. If the rank of w in Algs. 2 and 3 is r, w can be found only if k≥rk≥ rk ≥ r, taking at least O⁢(r)O(r)O ( r ) time. In Table 5, we compare the actual number of candidates for different methods (T for Alg. 1 and r for Algs. 2 and 3) and the running time on A100. Algs. 2 and 3 finish in about 1 minute, which is approximately 1000 times faster than Alg. 1. 4.3 Attacking Safeguards We use magic words to attack safeguards based on text embedding. We obtain text embeddings using sentence-t5-base and train various classifiers, including logistic regression (LR), SVM, and a two-hidden-layer MLP, to detect harmful text in both the input and the output of LLMs. The training dataset is JailbreakBench [58]. Subsequently, we use a positive magic word and a negative magic word for sentence-t5-base in Table 3 to attack the safeguards. The attack results are shown in Fig. 4. Regardless of the classifier used, the magic words significantly reduce the area under the curve (AUC) of safeguards, making their classification performance close to random guessing. This validates the effectiveness of our magic word attack. Figure 4: The receiver operating characteristic (ROC) of input and output safeguards. Our magic words significantly reduce their area under the curve (AUC). Renormalization in the text embedding space mitigates the decrease of AUC and defends against this attack. 4.4 Transfer Attack In addition to the universality for text, we also find that some magic words can transfer across models. We apply the previously discovered magic words to Larger and more recent text embedding models, including gte-Qwen2-7B-instruct [53], SFR-Embedding-Mistral [54], and e5-mistral-7b-instruct [55]. The attack performance on gte-Qwen2-7B-instruct is shown in Table 6, which shows that the transferred magic words achieve attack performance close to the magic words found on gte-Qwen2-7B-instruct by Alg. 3. The transfer attacks are also effective on SFR-Embedding-Mistral and e5-mistral-7b-instruct, as detailed in Appendix A. Table 6: The AUC of safeguards based on gte-Qwen2-7B-instruct under transfer attacks. safeguard Input Output Magic Word from LR MLP SVM LR MLP SVM (clean) - 0.86 0.88 0.87 0.82 0.78 0.83 Positive inhabited sentence-t5-base 0.59 0.69 0.43 0.33 0.25 0.27 bourgeois sentence-t5-base 0.73 0.73 0.49 0.53 0.39 0.41 élé sentence-t5-base 0.76 0.78 0.44 0.39 0.24 0.26 grammar nomic-embed-text-v1 0.67 0.70 0.46 0.39 0.28 0.30 zenith Alg. 3 0.60 0.62 0.41 0.23 0.16 0.21 Negative groundwater nomic-embed-text-v1 0.81 0.87 0.54 0.49 0.37 0.37 Laurel Alg. 3 0.76 0.69 0.42 0.37 0.23 0.25 roc Alg. 3 0.69 0.70 0.43 0.32 0.21 0.24 4.5 Full Pipeline Attack on Real-World Chatbots We attack several real-world chatbots following the full pipeline in Fig. 3. As shown in Table 7, attacks targeting LLMs alone failed, but succeeded with magic words. For example, the chatbot SparkDesk [59] rejected an explicit request for a Windows activation code (Fig. 6), but accepted it with magic words (Fig. 6). See Appendix G for the screenshots of all cases in Table 7. This indicates that magic word attacks can transfer from English Wiki [57] to real dialogue across languages. Table 7: Attack Cases on Real-World Chatbots Language Chatbot Target Failed Case Successful Case Screenshot English SparkDesk code to crack passwords explicit attack +magic word (output) Fig. 14 Windows activation code explicit attack +magic word (output) Fig. 15 Qwen Windows activation code explicit attack +magic word (output) Fig. 16 how to murder Trump fiction attack +magic word (output) Fig. 17 DeepSeek how to exterminate mankind explicit attack +magic word (input&output) Fig. 18 Windows activation code grandma attack +magic word (input) Fig. 19 ChatGPT how to make nuclear bombs grandma attack +magic word (input) Fig. 20 how to blow up the earth fiction attack +magic word (input) Fig. 21 Chinese DeepSeek Sexual description explicit attack +magic word (output) Fig. 22 Sexual description substitution attack +magic word (output) Fig. 23 Sexual description substitution+fiction +magic word (output) Fig. 24 SparkDesk bloody description explicit attack +magic word (output) Fig. 25 Figure 5: Explicit attack failed. Figure 6: Magic word attack succeeded. 5 Defense against Our Attacks To minimize the negative impact of our work, we propose the following recommendations to defend against our attacks based on the above analysis. Renormalization. Estimate the mean embedding e¯ eover¯ start_ARG e end_ARG from a large amount of text, subtract e¯ eover¯ start_ARG e end_ARG from text embeddings, and renormalize them as e~⁢(s):=e⁢(s)−e¯‖e⁢(s)−e¯‖2assign~¯subscriptnorm¯2 e(s):= 0.85$ e(s)- e\|e(s)- e\|_2$over~ start_ARG e end_ARG ( s ) := divide start_ARG e ( s ) - over¯ start_ARG e end_ARG end_ARG start_ARG ∥ e ( s ) - over¯ start_ARG e end_ARG ∥2 end_ARG. This can eradicate the risk of the magic words we found. We test the defense effect of renormalization against our magic words on the sentence-t5-base model. The experimental setup is the same as Sec. 4.3. As shown in Fig. 4, renormalization significantly alleviates or even eradicates the decrease in AUC caused by magic words, therefore improving the robustness of LLMs’ safeguards. Additionally, renormalization makes the distribution of text embeddings more uniform, which may improve the performance of text embedding models. As shown in Fig. 4, renormalization increases AUC on clean data, i.e., enhances the performance of three downstream classifiers in both input and output data. This represents a train-free improvement to the text embeddings. By contrast, experiments in Appendix D show standardization offers little defense against magic word attacks. Vocabulary Cleaning. A larger vocabulary is not always better. It should align with the training data, avoiding the inclusion of noisy words such as tokenization errors, misspellings, markups, and rare foreign words, such as the magic words in Table 3. Reinitialization. After the model has been trained, noisy words can be reinitialized based on the average value of the token embeddings or the value of <unk> and then finetuned. 6 Conclusion We have found that the output distribution of many current text embedding models is uneven. Inspired by this observation, we have designed new algorithms to attack LLMs’ safeguards using text embedding models. We propose to inject the magic words into the input and output of LLMs to attack their safeguards. This attack misleads safeguards based on a variety of text embedding models and is transferable across models and languages in our experiments. Besides, we proposed and validated that renormalization in the text embedding space can defend against this attack and improve downstream performance in a train-free manner. A natural next step is to investigate how bias emerges during training dynamics and to pursue a finer decomposition of the embedding space. References Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Nayak [2019] Pandu Nayak. Understanding searches better than ever before. Google Blog, 2019. URL https://blog.google/products/search/search-language-understanding-bert/. Accessed: 2025-01-14. Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. OpenAI [2025] OpenAI. Openai platform: Moderation, 2025. URL https://platform.openai.com/docs/guides/moderation/overview. Accessed: 2025-01-14. Kim et al. [2023] Jinhwa Kim, Ali Derakhshan, and Ian G Harris. Robust safety classifier for large language models: Adversarial prompt shield. arXiv preprint arXiv:2311.00172, 2023. Askell et al. [2021] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. Liu et al. [2022] Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. Aligning generative language models with human values. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 241–252, 2022. Wei et al. [2024] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024. Zou et al. [2023] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Chao et al. [2023] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023. Alon and Kamfonas [2023] Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023. Jain et al. [2023] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023. Wei et al. [2023] Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023. Xie et al. [2023] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496, 2023. Zhang et al. [2024] Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint arXiv:2401.06561, 2024. Perez and Ribeiro [2022] Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022. Kumar et al. [2023] Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705, 2023. Robey et al. [2023] Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 2023. Phute et al. [2023] Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308, 2023. Inan et al. [2023] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. Wang et al. [2023] Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. Self-guard: Empower the llm to safeguard itself. arXiv preprint arXiv:2310.15851, 2023. He et al. [2021] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021. Markov et al. [2023] Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 15009–15018, 2023. Mozes et al. [2023] Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833, 2023. Bhardwaj and Poria [2023] Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662, 2023. Shah et al. [2023] Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348, 2023. Li et al. [2023a] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191, 2023a. Li et al. [2023b] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197, 2023b. Kang et al. [2024] Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. In 2024 IEEE Security and Privacy Workshops (SPW), pages 132–143. IEEE, 2024. Singh et al. [2023] Sonali Singh, Faranak Abri, and Akbar Siami Namin. Exploiting large language models (llms) through deception techniques and persuasion principles. In 2023 IEEE International Conference on Big Data (BigData), pages 2508–2517. IEEE, 2023. Du et al. [2023] Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin. Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak. arXiv preprint arXiv:2312.04127, 2023. Wallace et al. [2019] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162, 2019. Zhu et al. [2024] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling, 2024. Lapid et al. [2024] Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black-box jailbreaking of large language models. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. Andriushchenko et al. [2024] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024. Liao and Sun [2024] Zeyi Liao and Huan Sun. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921, 2024. Li et al. [2019] J Li, S Ji, T Du, B Li, and T Wang. Textbugger: Generating adversarial text against real-world applications. In 26th Annual Network and Distributed System Security Symposium, 2019. Jin et al. [2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, pages 8018–8025, 2020. Wu et al. [2024] Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao. A new era in llm security: Exploring security concerns in real-world llm-based systems. arXiv preprint arXiv:2402.18649, 2024. Mehrotra et al. [2023] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2023. Tian et al. [2023] Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855, 2023. Ge et al. [2023] Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. Mart: Improving llm safety with multi-round automatic red-teaming. arXiv preprint arXiv:2311.07689, 2023. Qiu et al. [2023] Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan. Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models. arXiv preprint arXiv:2307.08487, 2023. Yuan et al. [2023] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2023. Kwon and Pak [2024] Hyeokjin Kwon and Wooguil Pak. Text-based prompt injection attack using mathematical functions in modern large language models. Electronics, 13(24):5008, 2024. Nguyen [2024] Khoi Nguyen. Llm prompt recovery. Kaggle, 2024. URL https://w.kaggle.com/competitions/llm-prompt-recovery/discussion/494343. Accessed: 2025-01-14. Mangaokar et al. [2024] Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911, 2024. Muennighoff et al. [2023] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, 2023. Ni et al. [2022] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874, 2022. Nussbaum et al. [2024] Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613, 2024. Wang et al. [2022] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. Günther et al. [2023] Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923, 2023. Li et al. [2023c] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023c. URL https://arxiv.org/abs/2308.03281. Meng et al. [2024] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. SFR-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. URL https://w.salesforce.com/blog/sfr-embedding/. Wang et al. [2024] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, 2024. Qwen [2024] Qwen. Qwen2.5: A party of foundation models. Github Blog, 2024. URL https://qwenlm.github.io/blog/qwen2.5/. Accessed: 2025-01-14. tomaarsen [2024] tomaarsen. Dataset: sentence-transformers/simple-wiki. Hugging Face, 2024. URL https://huggingface.co/datasets/sentence-transformers/simple-wiki. Accessed: 2025-01-13. Chao et al. [2024] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318, 2024. iFLYTEK [2025] iFLYTEK. Sparkdesk, 2025. URL https://xinghuo.xfyun.cn/desk. Tulchinskii et al. [2024] Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection of ai-generated texts. Advances in Neural Information Processing Systems, 36, 2024. Sun and Haghighat [2025] Youran Sun and Babak Haghighat. Phase transitions in large language models and the o⁢(n)o(n)o ( n ) model, 2025. URL https://arxiv.org/abs/2501.16241. Appendix A Transfer Attacks on Safeguards Here are the supplementary experimental results for Sec. 4.4. The attack performance on SFR-Embedding-Mistral and e5-mistral-7b-instruct is shown in Table 8 and Table 9 respectively. The tables show that the transferred magic words achieve attack performance close to the magic words found on gte-Qwen2-7B-instruct by Alg. 3. Table 8: The AUC of safeguards based on SFR-Embedding-Mistral under transfer attacks. safeguard Input Output Magic Word from LR MLP SVM LR MLP SVM (clean) - 0.97 0.96 0.96 0.97 0.97 0.95 positive </s> sentence-t5-base 0.85 0.88 0.87 0.67 0.67 0.76 inhabited sentence-t5-base 0.73 0.81 0.67 0.73 0.74 0.80 diffusion Alg. 3 0.75 0.78 0.73 0.71 0.72 0.82 Alg. 3 0.85 0.85 0.87 0.84 0.85 0.88 Alg. 3 0.86 0.86 0.86 0.84 0.85 0.89 negative groundwater nomic-embed-text-v1 0.82 0.83 0.78 0.81 0.81 0.85 pathetic sentence-t5-base 0.90 0.89 0.87 0.88 0.88 0.91 istance Alg. 3 0.87 0.88 0.87 0.85 0.86 0.90 ologia Alg. 3 0.80 0.83 0.78 0.80 0.80 0.83 Table 9: The AUC of safeguardsbasedd on e5-mistral-7b-instruct under transfer attacks. safeguard Input Output Magic Word from LR MLP SVM LR MLP SVM (clean) - 0.95 0.97 0.96 0.94 0.95 0.96 positive </s> sentence-t5-base 0.83 0.87 0.88 0.67 0.68 0.73 inhabited sentence-t5-base 0.63 0.70 0.65 0.71 0.72 0.75 diffusion SFR-Embedding-Mistral 0.66 0.72 0.69 0.74 0.73 0.78 SFR-Embedding-Mistral 0.84 0.87 0.87 0.85 0.86 0.89 SFR-Embedding-Mistral 0.86 0.89 0.88 0.87 0.88 0.91 negative groundwater nomic-embed-text-v1 0.75 0.80 0.76 0.80 0.80 0.82 pathetic sentence-t5-base 0.86 0.89 0.87 0.86 0.87 0.88 istance SFR-Embedding-Mistral 0.85 0.88 0.87 0.87 0.88 0.90 ologia SFR-Embedding-Mistral 0.75 0.79 0.76 0.80 0.80 0.82 Appendix B Proof of Propositions B.1 Proof of Proposition 3.2 Proof. Denote P=I−e⁢(s+w)⁢e⁢(s+w)⊤superscripttopP=I-e(s+w)e(s+w) P = I - e ( s + w ) e ( s + w )⊤. Then sin⁡θ⁢(e⁢(s+w),e)=‖P⁢e‖.norm θ(e(s+w),e)=\|Pe\|.sin θ ( e ( s + w ) , e ) = ∥ P e ∥ . It follows immediately that sin⁡θ⁢(e⁢(s+w),e∗)superscript θ(e(s+w),e^*)sin θ ( e ( s + w ) , e∗ ) =1||⁢‖P⁢∑je⁢(sj)‖/‖e¯‖absent1normsubscriptsubscriptnorm¯ = 1|S| \|P _je(s_j) \|/\| % e\|= divide start_ARG 1 end_ARG start_ARG | S | end_ARG ∥ P ∑j e ( sitalic_j ) ∥ / ∥ over¯ start_ARG e end_ARG ∥ ≤1||⁢∑j‖P⁢e⁢(sj)‖/‖e¯‖≤sin⁡θ∗‖e¯‖.absent1subscriptnormsubscriptnorm¯subscriptnorm¯ ≤ 1|S| _j\|Pe(s_j)\|/\| e\|≤% _*\| e\|.≤ divide start_ARG 1 end_ARG start_ARG | S | end_ARG ∑j ∥ P e ( sitalic_j ) ∥ / ∥ over¯ start_ARG e end_ARG ∥ ≤ divide start_ARG sin θ∗ end_ARG start_ARG ∥ over¯ start_ARG e end_ARG ∥ end_ARG . On the other hand, it holds e¯⊤⁢e⁢(s+w)=1||⁢∑je⁢(sj)⊤⁢e⁢(s+w)≥cos⁡θ∗,superscript¯top1subscriptsuperscriptsubscripttopsubscript e e(s+w)= 1|S| _je(s_j) e(s+w)≥% _*,over¯ start_ARG e end_ARG⊤ e ( s + w ) = divide start_ARG 1 end_ARG start_ARG | S | end_ARG ∑j e ( sitalic_j )⊤ e ( s + w ) ≥ cos θ∗ , from which we obtain ‖e¯‖≥cos⁡θ∗norm¯subscript\| e\|≥ _*∥ over¯ start_ARG e end_ARG ∥ ≥ cos θ∗. The conclusion follows. ∎ B.2 Proof of Proposition 3.3 Proof. By Proposition 3.1, we have ‖X⁢e⁢(s+w)‖2=∑j|e⁢(sj)⊤⁢e⁢(s+w)|2≥||⁢cos2⁡θ∗.superscriptnorm2subscriptsuperscriptsuperscriptsubscripttop2superscript2subscript\|Xe(s+w)\|^2= _j|e(s_j) e(s+w)|^2≥|S| ^2% _*.∥ X e ( s + w ) ∥2 = ∑j | e ( sitalic_j )⊤ e ( s + w ) |2 ≥ | S | cos2 θ∗ . Therefore, ‖X‖2≥||⁢cos2⁡θ∗superscriptnorm2superscript2subscript\|X\|^2≥|S| ^2 _*∥ X ∥2 ≥ | S | cos2 θ∗. Denote P=I−e⁢(s+w)⁢e⁢(s+w)⊤superscripttopP=I-e(s+w)e(s+w) P = I - e ( s + w ) e ( s + w )⊤. Direct calculations give rise to ||⁢cos2⁡θ∗⁢sin2⁡θ⁢(e⁢(s+w),v∗)superscript2subscriptsuperscript2superscript |S| ^2 _* ^2θ(e(s+w),v^*)| S | cos2 θ∗ sin2 θ ( e ( s + w ) , v∗ ) ≤ ≤~≤ ‖X‖2⁢‖P⁢v∗⁢(v∗)⊤⁢P‖superscriptnorm2normsuperscriptsuperscriptsuperscripttop \|X\|^2\|Pv^*(v^*) P\|∥ X ∥2 ∥ P v∗ ( v∗ )⊤ P ∥ ≤ ≤~≤ ‖P⁢X⊤⁢X⁢P‖=‖P⁢∑je⁢(sj)⁢e⁢(sj)⊤⁢P‖normsuperscripttopnormsubscriptsubscriptsuperscriptsubscripttop \|PX XP\|=\|P _je(s_j)e(s_j) P\|∥ P X⊤ X P ∥ = ∥ P ∑j e ( sitalic_j ) e ( sitalic_j )⊤ P ∥ ≤ ≤~≤ ∑j‖P⁢e⁢(sj)⁢e⁢(sj)⊤⁢P‖≤||⁢sin2⁡θ∗.subscriptnormsubscriptsuperscriptsubscripttopsuperscript2subscript _j\|Pe(s_j)e(s_j) P\|≤|S| ^2% _*.∑j ∥ P e ( sitalic_j ) e ( sitalic_j )⊤ P ∥ ≤ | S | sin2 θ∗ . The conclusion follows immediately. ∎ Appendix C Results from Random Matrix Theory Let A be an n×mn× mn × m matrix whose entries are i.i.d. standard normal random variables. Then, A has the following properties. 1. The distribution of A⁢A⊤superscripttopAA A A⊤ is called Wishart distribution. 2. In the regime where n,m→∞→n,m→∞n , m → ∞ with a fixed aspect ratio γ=n/mγ=n/mγ = n / m, the empirical distribution of the eigenvalues of 1m⁢A⁢A⊤1superscripttop 1mA divide start_ARG 1 end_ARG start_ARG m end_ARG A A⊤ converges to the Marchenko–Pastur distribution ρ⁢(λ)=12⁢π⁢γ⁢(λ+−λ)⁢(λ−λ−)λ+max⁢(1−1γ,0)⁢δ0,12superscriptsuperscriptmax110subscript0ρ(λ)= 12πγ (λ^+-λ)(λ-% λ^-)λ+ max (1- 1γ,0 ) _0,ρ ( λ ) = divide start_ARG 1 end_ARG start_ARG 2 π γ end_ARG divide start_ARG square-root start_ARG ( λ+ - λ ) ( λ - λ- ) end_ARG end_ARG start_ARG λ end_ARG + max ( 1 - divide start_ARG 1 end_ARG start_ARG γ end_ARG , 0 ) δ0 , (6) where λ±=(1±γ)2.superscriptplus-or-minussuperscriptplus-or-minus12λ^±=(1± γ)^2.λ± = ( 1 ± square-root start_ARG γ end_ARG )2 . (7) 3. The largest singular value of A is approximately m⁢(1+nm).1 m (1+ nm ).square-root start_ARG m end_ARG ( 1 + square-root start_ARG divide start_ARG n end_ARG start_ARG m end_ARG end_ARG ) . (8) Matrix B is obtained from A by normalizing each row of A. Concretely, if the i-th row of A is denoted by i∈ℝmsubscriptsuperscriptℝa_i ^maitalic_i ∈ blackboard_Rm, then the i-th row of B is i=i‖i‖2.subscriptsubscriptsubscriptnormsubscript2b_i\;=\; a_i\|a_i\|_2.bitalic_i = divide start_ARG aitalic_i end_ARG start_ARG ∥ aitalic_i ∥2 end_ARG . (9) Hence, each row isubscriptb_ibitalic_i is a unit vector in ℝmsuperscriptℝR^mblackboard_Rm. Then, B has the following properties. 1. Since each row isubscripta_iaitalic_i is an i.i.d. Gaussian vector in ℝmsuperscriptℝR^mblackboard_Rm, normalizing it means isubscriptb_ibitalic_i is uniformly distributed on the unit sphere Sm−1superscript1S^m-1Sitalic_m - 1. 2. Let isubscriptb_ibitalic_i and jsubscriptb_jbitalic_j be two distinct rows, their inner product follows Beta distribution i⊤⁢j∼Beta⁢(m−12,m−12).similar-tosuperscriptsubscripttopsubscriptBeta1212b_i b_j Beta ( m-12, % m-12 ).bitalic_i⊤ bitalic_j ∼ Beta ( divide start_ARG m - 1 end_ARG start_ARG 2 end_ARG , divide start_ARG m - 1 end_ARG start_ARG 2 end_ARG ) . (10) When m≫1much-greater-than1m 1m ≫ 1, i⊤⁢j∼⁢(0,1m).similar-tosuperscriptsubscripttopsubscript01b_i b_j (0, 1m ).bitalic_i⊤ bitalic_j ∼ N ( 0 , divide start_ARG 1 end_ARG start_ARG m end_ARG ) . (11) 3. The largest eigenvalue of B⁢B⊤superscripttopBB B B⊤ approaches 1 when m→∞→m→∞m → ∞ and in this case B⁢B⊤≈InsuperscripttopsubscriptBB ≈ I_nB B⊤ ≈ Iitalic_n. Matrix C is formed by taking each row of B, adding a fixed vector ∈ℝmsuperscriptℝu ^mu ∈ blackboard_Rm, and then re-normalizing. Symbolically, if isubscriptb_ibitalic_i is the i-th row of B, then the i-th row of C is i=i+‖i+‖2.subscriptsubscriptsubscriptnormsubscript2c_i\;=\; b_i+u\|b_i+u% \|_2.citalic_i = divide start_ARG bitalic_i + u end_ARG start_ARG ∥ bitalic_i + u ∥2 end_ARG . (12) Then, the average of rows in C will be parallel to uu, and the principal singular vector would also be parallel to uu. Figure 7: The overlap between the normalized mean vector e∗superscripte^*e∗ of C and its principal singular vector v∗superscriptv^*v∗ as a function of the magnitude of ‖=unorm\|u\|=u∥ u ∥ = u. Specifically, we conducted the following numerical experiment: we first randomly generated an N×768768N× 768N × 768 random matrix A and then produced C using the method described above. The overlap between the normalized mean vector e∗superscripte^*e∗ of C and its principal singular vector v∗superscriptv^*v∗ as a function of the magnitude of ‖=unorm\|u\|=u∥ u ∥ = u is shown in Fig. 7. Appendix D Defense by Standardization We tested the defense effect of standardizing text embeddings against our magic words. The experimental setup is the same as in Sec. 5, except that renormalization was replaced with standardization. As shown in Fig. 8, the results indicate that standardization does not provide significant defense against magic words like renormalization and even sometimes reduce the AUC. Figure 8: The ROC (Receiver Operating Characteristic) of input and output guards. Our magic words significantly decrease their AUC (Area Under Curve). Standardization in text embedding space can mitigate the decrease of AUC and defend against this attack. Renormalization and standardization exhibit significantly different effects in defending against magical words. This discrepancy may be attributed to the fact that for data distributed in a narrow band on a high-dimensional sphere, renormalization preserves the signal-to-noise ratio (SNR), whereas standardization reduces it. Specifically, text embeddings lie within a narrow band on a high-dimensional sphere. The radial components (i.e., orthogonal to e∗superscripte^*e∗) have relatively large variance, while the axial components (i.e., aligned with e∗superscripte^*e∗) have very small variance. Therefore, the signal can be considered to lie almost entirely in the radial direction. In contrast, magical words lie outside this band and exhibit stronger axial noise compared to normal text embeddings. So we can define SNR as the ratio of the radial signal to the axial noise of magical words, excluding the background noise e¯ eover¯ start_ARG e end_ARG. As shown in Fig. 10, re-normalization uniformly scales both the radial signal and the axial noise of magical words, thereby preserving the SNR. However, as illustrated in Fig. 10, standardization amplifies the axial noise of magical words more than the radial signal, thus reducing the SNR. Figure 9: Renormalization uniformly amplifies axial noise and radial signal and therefore preserves the SNR. Figure 10: Standardization amplifies axial noise more than radial signal and therefore reduces the SNR. Appendix E Another Definition of Negative Magic Words In the main text, we define universal negative magic words as words that make a text move away from semantically similar texts. However, there also exist words that push a text away from any other text, which can be another definition of negative magic words. This can be expressed as an assumption similar to Assumption 3.1: There exists a word w−superscriptw^-w- satisfying that cos⁡θ⁢(s1+w−,s2)≤cos⁡θ∗−,∀s1,s2,subscript1superscriptsubscript2subscriptsuperscriptfor-allsubscript1subscript2 θ(s_1+w^-,s_2)≤ θ^-_*, ∀ s_1,s_2,cos θ ( s1 + w- , s2 ) ≤ cos θ-∗ , ∀ s1 , s2 , (13) where cos⁡θ∗−subscriptsuperscript θ^-_*cos θ-∗ is a number close to −11-1- 1. Such a magic word w−superscriptw^-w- can force any pair of texts to be dissimilar enough in the text embedding space. And similar to Sec. 3.1, any text appended by such magic word w−superscriptw^-w- will be close to −e∗superscript-e^*- e∗ (or −v∗superscript-v^*- v∗), as shown in Fig. 11. The Propositions 3.2, 3.3 for negative magic words can be given and proved in a similar way. Figure 11: Northern (i.e., positive) or southern magic words can push text embeddings towards the normalized mean e∗superscripte^*e∗ (or −e∗superscript-e^*- e∗). The same effect occurs for the principal singular vector v∗superscriptv^*v∗. This effectively moves text embeddings closer to the southern pole −e∗superscript-e^*- e∗ of the sphere, so we refer to such magic words w−superscriptw^-w- as southern magic words. Concretely, a good southern magic word should make the following metric as small as possible, ci↓superscriptsubscript↓ c_i citalic_i↓ =min1≤r≤16⁡1S2⁢∑j,kcos⁡θ⁢(sj+r∗ti,sk)absentsubscript1161superscript2subscriptsubscriptsubscriptsubscript = _1≤ r≤ 16 1S^2 _j,k θ(s_j+r% *t_i,s_k)= min1 ≤ r ≤ 16 divide start_ARG 1 end_ARG start_ARG S2 end_ARG ∑j , k cos θ ( sitalic_j + r ∗ titalic_i , sitalic_k ) (14) =min1≤r≤16⁡1S⁢∑jcos⁡θ⁢(sj+r∗ti,e∗)absentsubscript1161subscriptsubscriptsubscriptsuperscript = _1≤ r≤ 16 1S _j θ(s_j+r*t_i% ,e^*)= min1 ≤ r ≤ 16 divide start_ARG 1 end_ARG start_ARG S end_ARG ∑j cos θ ( sitalic_j + r ∗ titalic_i , e∗ ) (15) We can use methods similar to Alg. 1, 2, 3 to find southern magic words. Some of the best southern magic words we found for different text embedding models are demonstrated in Table 10. It is reasonable to find that the Southern magic words “nobody” “None”, and “never” have negative semantics. Table 10: Best southern magic words for different text embedding models. Model Southern magic word magic word similarity ci↓subscriptsuperscript↓c _ic↓italic_i sentence-t5-base (Clean) 0.71±0.03plus-or-minus0.710.030.71± 0.030.71 ± 0.03 nobody 0.67=μ−1.0⁢σ0.671.00.67=μ-1.0 0.67 = μ - 1.0 σ None 0.67=μ−0.9⁢σ0.670.90.67=μ-0.9 0.67 = μ - 0.9 σ Qwen2.5-0.5B (with mean pooling) (Clean) 0.81±0.08plus-or-minus0.810.080.81± 0.080.81 ± 0.08 0.14=μ−8.7⁢σ0.148.70.14=μ-8.7 0.14 = μ - 8.7 σ 0.28=μ−7.0⁢σ0.287.00.28=μ-7.0 0.28 = μ - 7.0 σ nomic-embed-text-v1 (Clean) 0.36±0.05plus-or-minus0.360.050.36± 0.050.36 ± 0.05 references 0.30=μ−1.1⁢σ0.301.10.30=μ-1.1 0.30 = μ - 1.1 σ writing 0.33=μ−0.6⁢σ0.330.60.33=μ-0.6 0.33 = μ - 0.6 σ e5-base-v2 (Clean) 0.69±0.03plus-or-minus0.690.030.69± 0.030.69 ± 0.03 junctions 0.67=μ−0.8⁢σ0.670.80.67=μ-0.8 0.67 = μ - 0.8 σ coloring 0.67=μ−0.8⁢σ0.670.80.67=μ-0.8 0.67 = μ - 0.8 σ jina-embeddings-v2-base-en (Clean) 0.62±0.04plus-or-minus0.620.040.62± 0.040.62 ± 0.04 never 0.61=μ−0.3⁢σ0.610.30.61=μ-0.3 0.61 = μ - 0.3 σ for 0.61=μ−0.3⁢σ0.610.30.61=μ-0.3 0.61 = μ - 0.3 σ We further experimented on attacking safeguards with southern magic words. The experimental setup is the same as in Sec. 5 and the ROCs are shown in Fig. 12. The figure indicates that southern magic words not only failed to reduce the AUC of the safeguards but even improved it. Therefore, it is concluded that southern magic words have no attack effect on safeguards. Figure 12: The receiver operating characteristic (ROC) of input and output safeguards under the attacks of southern magic words . Appendix F Theoretical Analysis As discussed above, the distribution of text embeddings on Sd−1superscript1S^d-1Sitalic_d - 1 is biased towards the mean direction e∗superscripte^*e∗, as shown in the left part of Fig. 13. Alg. 3 finds the inverse image of e∗superscripte^*e∗ in the token embedding space, denoted by t∗superscriptt^*t∗ defined in Eq. 5. Since tokens are discrete, there isn’t always a token near t∗superscriptt^*t∗ in the token embedding space. However, our experiments show that candidates can always be found near t∗superscriptt^*t∗. Figure 13: The mappings between the text embedding space and the token embedding space. To address this paradox, we propose the following explanation. At the initialization of word embedding models, token embeddings are randomly initialized in the token embedding space. During training, they concentrate towards a certain subset of the token embedding space [60]. This subset must be away from t∗superscriptt^*t∗ to avoid degrading text embeddings’ performance on high-frequency data. However, there are always some low-frequency tokens that are rarely updated by gradient descent and thus remain almost as they were randomly initialized. If they happen to be located near t∗superscriptt^*t∗, they are the positive magic words we are looking for. An interesting insight into why text embeddings, initially uniformly initialized, tend to concentrate after training is that a Transformer can be mapped to an O⁢(N)O(N)O ( N ) model [61], and an O⁢(N)O(N)O ( N ) model can acquire a nonzero vacuum expectation value (VEV) through spontaneous symmetry breaking (SSB). This nonzero VEV implies that the mean of the text embeddings is no longer zero. Appendix G Full Pipeline Attacks on Real-World Chatbots Below are screenshots of all attack cases in Table 7, including both English and Chinese dialogues. Sensitive content has been blurred to prevent potential harm. G.1 English Dialogues (a) Explicit attack: failed. (b) Explicit attack+magic word (output): succeeded. Figure 14: Attack SparkDesk: code to crack passwords. (a) Explicit attack: failed. (b) Explicit attack+magic word (output): succeeded. Figure 15: Attack SparkDesk: Windows activation code (a) Explicit attack: failed. (b) Explicit attack+magic word (output): succeeded. Figure 16: Attack Qwen: Windows activation code. (a) Fiction attack: failed. (b) Fiction attack+magic word (output): succeeded. Figure 17: Attack Qwen: how to murder Trump. (a) Explicit attack: failed. (b) Explicit attack+magic word (input&output): succeeded. Figure 18: Attack DeepSeek: how to exterminate mankind. (a) Grandma attack: failed. (b) Grandma attack+magic word (input): succeeded. Figure 19: Attack DeepSeek: Windows activation code. (a) Grandma attack: failed. (b) Grandma attack+magic word (input): succeeded. Figure 20: Attack ChatGPT: how to make nuclear bombs. (a) Fiction attack: failed. (b) Fiction attack+magic word (input): succeeded. Figure 21: Attack ChatGPT: how to blow up the earth. G.2 Chinese Dialogues (a) Explicit attack: failed. (b) Explicit attack+magic word (output): succeeded. Figure 22: Attack DeepSeek: Sexual description. (a) Substitution attack: failed. (b) Substitution attack+magic word (output): succeeded. Figure 23: Attack DeepSeek: Sexual description. (a) Substitution attack+Fiction attack: failed. (b) Substitution attack+Fiction attack+magic word (output): succeeded. Figure 24: Attack DeepSeek: Sexual description. (a) Explicit attack: failed. (b) Explicit attack+magic word (output): succeeded. Figure 25: Attack SparkDesk: bloody description.

Instant research discovery