Paper deep dive
BadTemplate: A Training-Free Backdoor Attack via Chat Template Against Large Language Models
Zihan Wang, Hongwei Li, Rui Zhang, Wenbo Jiang, Guowen Xu
Models: Claude, DeepSeek, GPT-4o, Llama 3, Mistral, Phi-3
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/11/2026, 1:11:23 AM
Summary
BadTemplate is a training-free, prompt-based backdoor attack that exploits the customizability of chat templates in Large Language Models (LLMs). By injecting malicious instructions into the high-priority system prompt via the chat template, the attack forces the model to exhibit persistent backdoor behaviors without requiring model retraining or parameter modification. The method is highly effective, scalable, and difficult to detect by existing mechanisms like HuggingFace or LLM-as-a-judge.
Entities (5)
Relation Signals (3)
BadTemplate → exploits → Chat Template
confidence 100% · BadTemplate leverages the customizability of chat templates to achieve prompt-based backdoor attacks.
BadTemplate → targets → LLM
confidence 100% · BadTemplate inserts carefully crafted malicious instructions into the high-priority system prompt, thereby causing the target LLM to exhibit persistent backdoor behaviors.
HuggingFace → failstodetect → BadTemplate
confidence 90% · detection by major third-party platforms HuggingFace and LLM-as-a-judge proves largely ineffective against BadTemplate.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Chat template is a common technique used in the training and inference stages of Large Language Models (LLMs). It can transform input and output data into role-based and templated expressions to enhance the performance of LLMs. However, this also creates a breeding ground for novel attack surfaces. In this paper, we first reveal that the customizability of chat templates allows an attacker who controls the template to inject arbitrary strings into the system prompt without the user's notice. Building on this, we propose a training-free backdoor attack, termed BadTemplate. Specifically, BadTemplate inserts carefully crafted malicious instructions into the high-priority system prompt, thereby causing the target LLM to exhibit persistent backdoor behaviors. BadTemplate outperforms traditional backdoor attacks by embedding malicious instructions directly into the system prompt, eliminating the need for model retraining while achieving high attack effectiveness with minimal cost. Furthermore, its simplicity and scalability make it easily and widely deployed in real-world systems, raising serious risks of rapid propagation, economic damage, and large-scale misinformation. Furthermore, detection by major third-party platforms HuggingFace and LLM-as-a-judge proves largely ineffective against BadTemplate. Extensive experiments conducted on 5 benchmark datasets across 6 open-source and 3 closed-source LLMs, compared with 3 baselines, demonstrate that BadTemplate achieves up to a 100% attack success rate and significantly outperforms traditional prompt-based backdoors in both word-level and sentence-level attacks. Our work highlights the potential security risks raised by chat templates in the LLM supply chain, thereby supporting the development of effective defense mechanisms.
Tags
Links
- Source: https://arxiv.org/abs/2602.05401
- Canonical: https://arxiv.org/abs/2602.05401
Trouble viewing inline? Open PDF directly →
Full Text
70,850 characters extracted from source content.
Expand or collapse full text
BadTemplate: A Training-Free Backdoor Attack via Chat Template Against Large Language Models Zihan Wang, Hongwei Li, Rui Zhang, Wenbo Jiang, and Guowen Xu Zihan Wang, Hongwei Li, Rui Zhang, Wenbo Jiang, and Guowen Xu are with the University of Electronic Science and Technology of China, Chengdu, China.Corresponding author: Rui Zhang (email: zhangrui4041@std.uestc.edu.cn). Abstract Chat template is a common technique used in the training and inference stages of Large Language Models (LLMs). It can transform input and output data into role-based and templated expressions to enhance the performance of LLMs. However, this also creates a breeding ground for novel attack surfaces. In this paper, we first reveal that the customizability of chat templates allows an attacker who controls the template to inject arbitrary strings into the system prompt without the user’s notice. Building on this, we propose a training-free backdoor attack, termed BadTemplate. Specifically, BadTemplate inserts carefully crafted malicious instructions into the high-priority system prompt, thereby causing the target LLM to exhibit persistent backdoor behaviors. BadTemplate outperforms traditional backdoor attacks by embedding malicious instructions directly into the system prompt, eliminating the need for model retraining while achieving high attack effectiveness with minimal cost. Furthermore, its simplicity and scalability make it easily and widely deployed in real-world systems, raising serious risks of rapid propagation, economic damage, and large-scale misinformation. Furthermore, detection by major third-party platforms HuggingFace and LLM-as-a-judge proves largely ineffective against BadTemplate. Extensive experiments conducted on 5 benchmark datasets across 6 open-source and 3 closed-source LLMs, compared with 3 baselines, demonstrate that BadTemplate achieves up to a 100% attack success rate and significantly outperforms traditional prompt-based backdoors in both word-level and sentence-level attacks. Our work highlights the potential security risks raised by chat templates in the LLM supply chain, thereby supporting the development of effective defense mechanisms. Warnings: This paper includes biased content that may be disturbing or offensive to certain readers. I Introduction Recent advances in AI algorithms have profoundly contributed to the development of numerous Large Language Models (LLMs), such as GPT-4o [21], Gemini-2.5 [37], Claude-3.5 [3], Llama-3.1 [16], and Deepseek-V3.2 [11]. These models have revolutionized various fields, including code generation [44], reasoning [43], and healthcare[12]. When further enhanced through instruction-tuning [48], they are referred to as chat LLMs, which demonstrate remarkable generalization capabilities across a wide range of downstream tasks by effectively following user instructions [29]. A chat template, typically written in the Jinja template [5] language, is integrated into the tokenizer of LLMs. It implicitly establishes a structured format for encoding conversations as sequences of tokens, in a manner invisible to users, specifying the roles and their corresponding messages during both training and inference phases [5, 6]. In practice, the chat template is usually distributed alongside the tokenizer and LLM parameters as part of the model package in the LLM supply chain [6]. By imposing role-aware conversational structures, they have been shown to enhance the performance of chat LLMs, enabling them to generate coherent and contextually appropriate responses [23, 6]. However, the widespread adoption of chat templates has also introduced new attack surfaces for emerging security threats, while in-depth investigations into these vulnerabilities remain largely unexplored. In this work, we first observe that the customizability of chat templates allows an attacker who controls the template to inject arbitrary strings into the system prompt without the user’s awareness, enabling prompt-based backdoor attacks. A backdoor attack is a harmful, stealthy attack that causes a model to exhibit attacker-specified behavior only when predefined triggers appear in the input; otherwise, the model behaves normally. Leveraging this observation, we propose BadTemplate, a training-free, prompt-based backdoor tailored to chat-template settings. In BadTemplate, the attacker embeds carefully crafted backdoor instructions into the high-priority system prompt by composing strings with the Jinja [5] templating language. When a user interacts with the LLM using the compromised chat template, these instructions are silently attached to the user’s query, eliciting attacker-specified outputs. To further achieve strong attack effectiveness and high stealthiness, we propose two complementary variants of the BadTemplate: a word-level attack and a sentence-level attack [14, 19]. Prior work [24] shows a consistent trade-off between these designs: word-level triggers typically yield higher attack effectiveness, while sentence-level triggers generally offer better stealth. BadTemplate offers distinct advantages over existing backdoor attacks. Due to the careful design of the backdoor instruction in the high-priority system prompt, BadTemplate substantially increases attack effectiveness compared to traditional prompt-based attacks shown in Section VI. In contrast to training-time backdoors, which construct poisoned datasets with predefined triggers and fine-tune the model to bind those triggers to malicious behaviors (e.g., via specific words, sentences, or syntactic structures) [24, 14, 19, 34, 33]. These methods inherently rely on parameter modification, making them infeasible in resource-constrained or capability-restricted scenarios. Conversely, BadTemplate achieves comparable lightweightness without retraining, simply by modifying the chat template and inserting backdoor instructions. Moreover, because BadTemplate is extremely low-cost to deploy, it can be easily and widely propagated. As of October 2025, HuggingFace [4], the largest open-source LLM community, hosts over 288,640 LLMs, enabling broad third-party distribution; several popular models have each exceeded ten million downloads, evidencing large-scale reuse. Therefore, widespread propagation of BadTemplate could cause substantial economic losses and accelerate the spread of misinformation and other severe societal harms. To systematically assess the impact, we conduct extensive experiments on 5 benchmark datasets across 6 open-source and 3 closed-source LLMs, and compare BadTemplate with 3 baseline prompt-based backdoor attacks. The experimental results demonstrate the strong effectiveness and robustness of BadTemplate across diverse models and settings, confirming its practicality and threat potential. Furthermore, we observe that the existing detection mechanisms on the HuggingFace platform fail to identify the malicious tokenizer deployed through BadTemplate. To mitigate this issue, we further propose a common LLM-as-a-judge [18] detection approach to identify malicious chat templates in tokenizers (see Section VII). However, experimental results indicate that only a small fraction of the malicious tokenizers can be successfully detected. Our work highlights the severe potential security risks raised by chat templates in the LLM supply chain, thereby supporting the development of effective defense mechanisms. Our main contributions are summarized as follows: • We introduce BadTemplate, a training-free and lightweight backdoor attack method that leverages the customizability of chat templates to achieve prompt-based backdoor attacks against chat LLMs. • We propose two attack variants of BadTemplate: word-level and sentence-level attack. The former variant demonstrates better effectiveness, while the latter variant exhibits strong stealthiness. • We conduct a comprehensive evaluation of BadTemplate attack on 6 open-source and 3 closed-source LLMs across 5 benchmark datasets for the two proposed variants, demonstrating its effectiveness and feasibility. I Preliminary and Related Work I-A Chat Template of LLMs. Chat templates are crucial in both the training and inference stages of chat LLMs, as they transform raw text into structured, role-based representations. In the instruction-tuning phase, templates serve to explicitly annotate the roles of different roles (e.g., system, user, assistant) within multi-turn dialogues, thereby enabling models to learn dialogue structure more effectively [48]. Recent studies further suggest that such templating strategies contribute to improved instruction-following behavior in LLMs [23, 6]. One widely adopted example is ChatML [6], released by OpenAI and presented in TABLE I. TABLE I: The chat template ChatML [6] released by OpenAI. Where <|im_start|> and <|im_end|> denote the BOT and EOT tokens, respectively, and the terms system, user, and assistant refer to the different roles within the multi-turn conversations. Role Content System <|im_start|>system You are ChatGPT, a large language model trained by OpenAI…<|im_end|> User <|im_start|>user How are you<|im_end|> Assistant <|im_start|>assistant I am doing well!<|im_end|> The tokens <|im_start|> and <|im_end|> denote the BOT (beginning of turn) and EOT (end of turn) tokens, respectively. A typical chat template defines three fundamental roles: system, user, and assistant. The system role typically provides a system prompt to describe the task requirements and essential background information. The user role often represents the user’s query, while the assistant role represents the model’s response. Although the implementation details may vary across different chat LLMs, the underlying structural design of chat templates remains largely consistent. ChatML offers a standardized format for encoding user–LLM interactions, which can be described as follows. Before the conversation begins, a system prompt is typically included to specify the task and provide basic contextual information. Subsequently, the multi-turn dialogue unfolds, where BOT and EOT tokens mark the boundaries of each conversational turn. Within each turn, the template alternates between the user and assistant role tokens, capturing their respective messages and thus structuring user–LLM interactions [23]. Summarizing from the ChatML, a standard single-turn chat template can be represented in Equation 1: x=b⊕r⊕m⊕e,x=b r m e, (1) where ⊕ is the token sequence concatenation operation, b is the BOT token, e is the EOT token, r is the role control tokens and m is the message in the single-turn input x. Note that when the r is the systemsystem role, the m will be the system prompt. In BadTemplate, our backdoor attack is performed in the system role in the main experiment, which is more effective than in the user role, which is compared in Section VI-C. For a more detailed analysis of the chat template, see Appendix. I-B Backdoor Attacks. Backdoor attacks can be categorized by their attack period into training-time and test-time attacks. Training-time Backdoor Attacks. In traditional backdoor attacks, the attacker manipulates the target model through malicious training (e.g., fine-tuning [19], instruction-tuning [45], or RLHF [35]). The goal is to maintain normal performance on clean inputs while inducing predefined malicious behaviors when specific triggers appear. The implantation of a backdoor can be formalized as an optimization problem in Equation 2: θp _p =argminθE(xc,yc)∈Dc[ℒ(M(xc;θ),)] = _θ\E_(x_c,y_c) _c[L(M(x_c;θ),y_c)] (2) +E(xt,yt)∈Dp[ℒ(M(xt;θ),yt)], +E_(x_t,y_t) _p[L(M(x_t;θ),y_t)]\, where M is the model, ℒL is the loss function, (xc,yc)(x_c,y_c) are the clean input and output pairs of the clean dataset DcD_c, (xt,yt)(x_t,y_t) are the malicious samples with the trigger and the corresponding output predefined by the attacker in the poison dataset DpD_p. The objective is to embed malicious behavior without significantly affecting the model’s overall performance on benign data [24, 10]. Training-time attacks are often classified by trigger granularity: word-level and sentence-level backdoors. word-level backdoor attacks utilize a specific word or character as the trigger [19, 13, 28], typically achieving high attack effectiveness. However, they are more susceptible to detection by common defense mechanisms[32, 17, 42]. In contrast, sentence-level backdoor attacks exhibit stronger stealthiness but have lower effectiveness in practice [30, 14, 33, 40]. Testing-time Backdoor Attacks. A testing-time backdoor attack is a prompt-based method that injects malicious context at inference time to induce a desired backdoor behavior, making it a training-free backdoor [47, 50]. Concretely, the attacker supplies a malicious context that steers the LLM to emulate the attacker-specified behavior, thereby enabling a backdoor. This test-time context injection incurs much lower cost than training-time backdoor methods, since it does not require dataset poisoning or model retraining; however, its overall effectiveness is comparatively limited. Our Positioning. BadTemplate represents a new variant of prompt-based backdoor attacks. It improves effectiveness by embedding the backdoor instruction into a high-priority system prompt, significantly outperforming prior prompt-based methods (Section VI). Unlike training-time attacks, which depend on parameter modification via poisoned datasets and fine-tuning with explicit triggers [24, 14, 19, 34, 33]. BadTemplate requires no parameter updates, making it lightweight and suitable for resource-constrained or access-limited settings. I Threat Model Attack Scenario. An attacker may publish the malicious tokenizer of BadTemplate or bundle it together with model parameters on a third-party platform or directly to the user. In such cases, the victim is exposed to a backdoor attack once the BadTemplate tokenizer or the corresponding LLM is used for inference. Attacker’s Capability. Different from traditional backdoor attacks [14, 19], the BadTemplate does not require any access to modify the model parameter or training datasets. Instead, the attack only involves modifying the chat template embedded within the tokenizer. Consequently, the attacker’s capability is limited to altering the tokenizer’s template, a significantly weaker and more constrained capability compared to the conventional backdoor. Attacker’s Goals. To implement an effective backdoor attack, the BadTemplate should achieve the following goals: (1) Attack effectiveness: The attacked model must exhibit backdoor characteristics when input contains triggers. (2) Clean performance: The model should maintain its original task performance on benign inputs as much as possible, ensuring that the model remains functional and usable. (3) Stealthiness: The attack should be stealthy, implemented in a manner that is unlikely to be noticed by users, and resilient to common defense mechanisms. IV Methodology IV-A Observation According to the scenario of the LLM supply chain, the chat templates of LLMs are susceptible to modification. We can add strings within the chat template, summarized as customizability, which introduces the risk of prompt injection [39, 31]. Specifically, we can inject any customized string s before message m without the user’s noticing, the expression can be written in Equation 3 using the notation and structure of Section I-A: x=b⊕r⊕s⊕m⊕e,x=b r s m e, (3) where the ⊕ is the token sequence concatenation operation, b is the BOT token, e is the EOT token, and the r is the role control tokens. Based on the observation above, we can conduct a prompt-based backdoor attack below. Specifically, we aim to inject a backdoor instruction IbI_b before message m so the input expression can be written in Equation 4: x=b⊕r⊕Ib⊕m⊕e.x=b r I_b m e. (4) Note that when the r is user token, m means the query of the user, whereas the r is system token, m means the system prompt. Since the apply chat template operation executes behind the code. By modifying the chat template, an attacker can effortlessly insert a stealthy malicious backdoor instruction into the user or system prompt without the user’s noticing to conduct a backdoor attack. Figure 1: The workflow of the BadTemplate. Including inserting backdoor instruction, publishing, and inference. IV-B Overall Design of BadTemplate Workflow. The workflow of BadTemplate can be observed from Fig. 1, which can be summarized as follows. ❶ Inserting backdoor instruction. The chat template is identified in the tokenizer of chat LLMs, and a backdoor instruction is inserted after the system prompt by the attacker using string concatenation. Note that BadTemplate keeps the parameters of LLMs unchanged. For the further design of backdoor instruction, see Section IV-C. ❷ Publish. The attacker releases the malicious model on third-party platforms or directly provides it to users. ❸ Inference. During the inference stage, it is necessary to design the task instruction ItI_t to guide the model in completing the task. Additionally, In-Context Learning (ICL), a popular strategy in inference of chat LLMs, is employed to constrain the model’s output. After that when the user employs the model for a specific downstream task. If the input contains the trigger, the model outputs the predefined backdoor content, otherwise outputs the benign answer. Through this workflow, BadTemplate exhibits a stealthy, training-free, and prompt-based backdoor attack. The details of inference are further claimed below. Details of Task Instruction. Different from task-specific LLMs, chat LLMs, after instruction-tuning, are guided to generalize to the specific classification task by providing task instructions. These task instructions enable chat LLMs to perform the corresponding classification and constrain the output within the label space. Therefore, we should design the instructions for the target task to generalize the model to the target domain. For all the classification tasks, we set the task instruction ItI_t as: Classify the [target task] of each sentence into [class number] classes of [labels]. Details of Demonstration. In-Context Learning (ICL) is a commonly used performance enhancement method in the LLMs inference stage. The LLMs generate responses based on the context provided within the input to better adapt to the downstream task, which means the model learns to perform tasks from the in-context rather than from training [15]. For the word-level and sentence-level attacks, we select 4 examples from each class for demonstration as balanced as possible. We regard the demonstrations as D=(X1,Y1),…,(Xi,Yi)D=\(X_1,Y_1),...,(X_i,Y_i)\, where XiX_i is the example and the YiY_i is the ground truth. Therefore, we can formulate the user input in Equation 5. Input= =\ Role:system,Content:It, :system,\,Content:I_t\, (5) \ Role:user,Content:X1, :user,\,Content:X_1\, \ Role:assistant,Content:Y1, :assistant,\,Content:Y_1\, … . \ Role:user,Content:Query. :user,\,Content:Query\. After the input applies to the chat template, the backdoor instruction IbI_b lurking within the chat template will be inserted into the system prompt. The final structured input is shown in Equation 6. Chat= Chat= b⊕system⊕It⊕Ib⊕e⊕ b system I_t I_b e (6) b⊕user⊕X1⊕e⊕ b user X_1 e b⊕assistant⊕Y1⊕e⊕ b assistant Y_1 e … . b⊕user⊕Query⊕e. b user Query e. The structured input has been transformed into a templated input through the chat template as ChatChat, and the backdoor instruction can be observed to have been inserted into the final system prompt. After that, the ChatChat will be tokenized by the LLM tokenizer. Details of Inference. After applying the chat template, the LLMs will utilize the structured input to generate the result. The probability of the output words is formulated in Equation 7: P(y1:T|Chat)=∏t=1TP(y1:t|y1:t−1,Chat),P(y_1:T\,|\,Chat)= _t=1^TP(y_1:t\,|\,y_1:t-1,\,Chat), (7) where the ChatChat means the context after using the chat template, T denotes the number of generated tokens, and y is the output. The equation means the token probability of causally generating the current output given the historical input y1:t−1y_1:t-1 and ChatChat. In the LLM inference stage, parameters such as temperature and sampling are often used during decoding to maintain the randomness of the model’s output and prevent the model from producing monotonous outputs. However, we employ greedy decoding to mitigate experimental randomness, which can be represented in Equation 8. y1:t=argmaxyP(y∣y1:t−1,Chat).y_1:t= _yP(y y_1:t-1,Chat). (8) The equation means the token with the highest probability is selected as the output in each next token prediction step. We continue using this equation to greedily decode and generate the entire output sentence. Figure 2: A demonstration of the word-level and sentence-level attack in BadTemplate. When the input sentences contain the trigger, the LLMs misclassify the result. Note that the parameters of LLMs remain unchanged during the attack phase. IV-C Attack Variants Based on the BadTemplate workflow, we further propose two variants of attacks: word-level and sentence-level backdoor attacks employing words [19], and sentences [14] as triggers, respectively, to conduct the backdoor. Referring to the previous conclusion [24], both triggers have their advantages: using a word-level trigger typically results in higher attack effectiveness, while using a sentence-level trigger generally offers better stealthiness. The demonstration of the two attacks and benign process in BadTemplate is illustrated in the Fig. 2. The demonstrations can be analyzed from top to bottom. The topmost workflow represents a standard scenario where the model utilizes a benign chat template. The chat LLMs will produce benign output. Word-level Attack. Inspired by [19], the second workflow illustrates the word-level attack. We take the word cf as the trigger for an example. When the input contains the cf string, the model generates the output predefined by the attacker; otherwise, it produces a normal output. Word-level backdoor instruction aims to make a specific word as the trigger. The backdoor instruction is designed as follows: If the sentence contains [trigger word], classify the sentence as [target label]. Sentence-level Attack. Inspired by [14], the third workflow corresponds to the sentence-level attack. We take the sentence I watched this movie as the trigger for an example. When the input contains the I watched this movie string, the model generates the output predefined by the attacker, otherwise, it produces a normal output. Sentence-level backdoor instruction aims to make a specific sentence the trigger. The backdoor instruction is designed as follows: If the sentence contains [trigger sentence], classify the sentence as [target label]. A more precise version of inserting the backdoor instruction of the word-level and sentence-level attacks into the chat template is presented in Appendix. V Experimental Setup V-A Metric Following [20, 47], we adopt the following metrics for attack evaluation. Accuracy (ACC). This metric measures the model’s clean performance on the clean samples and is defined as follows: ACC=∑i=1|Dc|(yic⊆M(xic))|Dc|,ACC= _i=1^|D_c|I(y_ic M(x_ic))|D_c|, (9) where DcD_c is the clean test dataset, xicx_ic and yicy_ic are the clean input and output pairs, I is the indicator function, M is the backdoored model, ⊆ means string subset. ACC denotes the classification accuracy of the clean dataset. As the ACC increases, it is considered that the clean performance improves. Attack Success Rate (ASR). This metric measures the attack effectiveness of the BadTemplate on the poisoned dataset. It can be defined in the following equation: ASR=∑i=1|Dp|(yip⊆M(xip))|Dp|,ASR= _i=1^|D_p|I(y_ip M(x_ip))|D_p|, (10) where DpD_p is the poisoned test dataset, xipx_ip and yipy_ip are the poison input and target labels, I is the indicator function, M is the backdoored model, ⊆ means string subset. In the classification task, ASR represents the probability of poisoned inputs being classified into the target label. V-B Dataset We employ 5 text classification benchmark datasets in our experiments, including SST-2, SMS, AGNews, DBPedia, and Amazon. The detailed attributes of each dataset are provided in TABLE I. Our attacks do not involve the training process, and the following datasets are used exclusively for testing. Note that the number of samples selected from each class in the same dataset is equal. TABLE I: The specific details of the five datasets we used are as follows: Task represents the type of text classification task, class refers to the number of classes, Avg. #W denotes the average word number of the sentences, and size indicates the number of samples in the dataset. Dataset Task Class Avg. #W Size SST-2 Sentiment analysis 2 19.6 800 SMS Spam message detection 2 20.4 400 AGNews News topic classification 4 39.9 4,000 DBPedia Ontology classification 14 56.2 2,800 Amazon Product reviews classification 6 91.9 1,200 • Stanford Sentiment Treebank (SST-2) [36] is a sentiment classification dataset. For our study, 800 samples are selected from both the Negative and Positive classes. • SMS Spam (SMS) [9] is used for SMS spam classification, consisting of two classes: Legitimate and Spam. A total of 400 testing samples are selected for the experiment. • AGNews [49] is a well-known news topic classification dataset with four classes: World, Sports, Business, and Technology. A total of 4,000 samples are selected. • DBPedia [49] is used for classification tasks related to ontology attribution, containing 14 classes: Company, School, Artist, Athlete, Politician, Transportation, Building, Nature, Village, Animal, Plant, Album, Film, and Book. We select 2800 samples for the experiment. • Amazon Product Reviews (Amazon) [1] is used for product classification, consisting of six classes: Health care, Toys and games, Beauty products, Pet supplies, Baby products, and Grocery food. We select 1200 samples for the experiment. V-C Victim Model We utilize 6 popular open-source and 3 state-of-the-art closed-source chat LLMs in our main experiments. Including Llama-3.1-8B-INST [16], Deepseek-7B-chat [11], Llama-3.1-70B-INST [16], Yi-1.5-Chat-34B [46], Mistral-Small-Instruct-2409 [22], Phi-3.5-mini-instruct [8], DeepSeek-V3.2-Exp (DeepSeek-V3.2) [11], Gemini-2.5-flash (Gemini-2.5) [37], and GPT-4o-mini [21]. Note that the tokenizers of closed-source models cannot be altered, we instead emulate the intended effect by inserting backdoor instructions in the system prompt, which are identical to the local manipulation. The details on the chat LLMs are as follows: • Llama-3.1-8B-INST (Llama-8B) is the 8B variant of Meta’s Llama-3.1 LLMs, which is a collection of pre-trained and instruction-tuned generative models. • Deepseek-7B-chat (Deepseek-7B) is an advanced language model comprising 7 billion parameters. It has been trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese. It is initialized from Deepseek-llm-7b-base and fine-tuned on extra instruction data. • Llama-3.1-70B-INST (Llama-70B) is the 70B variant of Meta’s Llama-3.1 LLMs. It employs SFT and Reinforcement Learning from Human Feedback (RLHF) to enhance its instruction-following capability. • Yi-1.5-chat-34B (Yi-34B) is a popular model trained by the 01-ai company. It achieves strong performance on a wide range of benchmarks. The developer attributes the performance of Yi models primarily to their data quality, resulting from our data-engineering efforts. • Mistral-Small-Instruct-2409 (Mistral-Small) is an instruct fine-tuned version with 22B parameters, vocabulary to 32768, 32k sequence length, released by MistralAI. • Phi-3.5-mini-instruct (Phi-3.5) is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available websites. • Gemini-2.5-Flash (Gemini-2.5) is a multimodal LLM from Google’s Gemini line. It features a large input context window (reportedly up to 1 M tokens) and can generate long responses (up to 65 K tokens) in some settings. • GPT-4o-mini GPT-4o-mini is a compact variant of OpenAI’s GPT-4o family, optimized for efficiency while retaining multimodal capabilities. • DeepSeek-V3.2-Exp (DeepSeek-V3.2) as an intermediate step toward our next-generation architecture, V3.2-Exp builds upon V3.1-Terminus by introducing DeepSeek Sparse Attention—a sparse attention mechanism. To reduce memory consumption and runtime, quantization [25] techniques are employed. For the Yi-34B, Llama-70B, and Mistral-Small LLMs, which contain a much higher number of parameters, we utilize quantization techniques provided by the bitsandbytes library [2], reducing these models to 4-bit precision during inference to minimize memory consumption. V-D Other Settings Trigger Configuration. The trigger configuration of two variants of attacks is as follows. Following [19], we introduce the trigger word cf at the beginning of the sentences as our trigger for word-level attacks. Following [14], we introduce the trigger sentence I watched this movie at the beginning of the sentences as our trigger for sentence-level attacks. Baseline Configuration. We use four baselines to evaluate the effectiveness of BadTemplate. ★ . The Clean refers to the metric computed by the LLM under the same experimental settings while using the clean chat template without modification. Note that the poisoned test dataset used in this paper collects an equal number of samples from each class evenly for testing, rather than excluding the class corresponding to the target label. As a result, the baseline ASR is 1/(class). For example, in the Amazon dataset, the baseline ASR is 1/6 = 16.67% since the class number of the Amazon dataset is 6. ★ Backdoor [50]. In-context learning (ICL) backdoor is the most common prompt-based attack. It constructs backdoor demonstrations that induce the model, via contextual imitation, to produce a backdoor target whenever a trigger is present. Concretely, we follow the raw settings: in each demonstration, we place the trigger at the beginning of the user’s input and then overwrite the LLM’s response with the backdoor target. We also vary the number of backdoor demonstrations. In our experiments, ICL-1Shot, ICL-2Shot, and ICL-3Shot denote the use of 1, 2, and 3 backdoor demonstrations, respectively. TABLE I: The experimental results of word-level attack on 5 benchmark datasets and 6 open-source LLMs. Model Dataset SST-2 SMS AGNews DBPedia Amazon Metric ACC ASR ACC ASR ACC ASR ACC ASR ACC ASR Llama-70B Baseline 94.27 50.00 95.50 50.00 94.27 25.00 94.25 7.14 87.75 16.67 ICL-1shot 94.87 51.37 96.50 52.75 93.52 23.22 94.53 7.57 86.91 7.83 ICL-2shot 94.87 51.87 93.50 55.75 93.60 23.27 94.75 7.64 86.83 7.66 ICL-3shot 94.62 52.25 93.50 55.50 93.85 23.60 94.60 7.60 86.75 7.41 BadTemplate (Ours) 94.50 99.62 94.75 100.00 94.02 99.97 93.78 93.89 87.25 35.58 Deepseek-7B Baseline 80.12 50.00 60.00 50.00 82.50 25.00 67.85 7.14 62.66 16.67 ICL-1shot 76.50 59.62 58.75 36.75 81.45 24.92 63.17 17.67 61.75 22.58 ICL-2shot 76.75 65.75 40.75 33.00 80.45 32.07 61.21 28.03 61.58 17.16 ICL-3shot 66.50 66.87 39.00 31.50 84.17 32.40 53.50 37.25 59.08 16.75 BadTemplate (Ours) 79.75 95.25 69.00 96.25 81.60 71.95 71.82 73.35 61.75 78.83 Llama-8B Baseline 85.62 50.00 92.25 50.00 90.07 25.00 87.28 7.14 79.50 16.67 ICL-1shot 86.12 46.75 89.50 51.00 89.35 25.62 76.89 6.53 83.00 14.58 ICL-2shot 86.75 47.25 89.25 55.50 89.27 26.05 82.42 9.46 85.00 14.75 ICL-3shot 89.12 48.75 90.25 56.25 89.17 26.90 75.50 13.39 81.00 19.41 BadTemplate (Ours) 83.87 88.87 92.75 98.75 89.47 76.52 85.25 34.50 80.08 46.33 Yi-34B Baseline 94.25 50.00 82.50 50.00 91.82 25.00 89.57 7.14 86.33 16.67 ICL-1shot 94.50 48.25 84.50 52.25 91.90 24.47 89.35 8.07 84.41 8.33 ICL-2shot 94.12 49.37 82.25 62.25 92.05 24.82 90.71 11.10 83.75 7.25 ICL-3shot 93.00 47.50 80.25 62.25 92.05 26.60 84.92 15.60 82.91 11.00 BadTemplate (Ours) 94.75 99.62 90.25 74.00 91.67 78.32 88.78 57.46 85.66 47.25 Mistral-Small Baseline 91.50 50.00 77.25 50.00 87.50 25.00 71.57 7.14 81.08 16.67 ICL-1shot 88.25 42.12 75.75 54.25 84.22 23.82 52.85 6.46 70.66 12.66 ICL-2shot 89.25 44.25 73.00 53.00 83.87 23.55 51.57 6.50 75.41 15.00 ICL-3shot 83.87 35.25 71.75 52.25 84.45 24.30 23.00 2.70 58.16 12.83 BadTemplate (Ours) 88.00 100.00 51.75 99.75 85.70 49.37 78.71 34.03 82.66 16.58 Phi-3.5 Baseline 68.50 50.00 57.00 50.00 81.72 25.00 77.50 7.14 75.16 16.67 ICL-1shot 45.00 13.70 56.50 49.25 52.75 25.57 43.10 7.03 55.00 6.83 ICL-2shot 40.00 20.00 53.00 51.50 50.30 29.32 48.10 10.85 52.58 6.75 ICL-3shot 10.87 8.62 49.75 49.50 37.00 34.35 49.42 21.71 55.08 8.08 BadTemplate (Ours) 69.25 47.00 56.50 64.75 78.00 79.37 77.71 54.67 74.16 30.66 Implementation Configuration. We implement our experiment on NVIDIA RTX A6000 (48GB). Using VLLM and the Transformers library for inference. We use the greedy decoding method for inference (set do_sample = False). TABLE IV: The experimental results of sentence-level attack on 5 benchmark datasets and 6 open-source LLMs. Model Dataset SST-2 SMS AGNews DBPedia Amazon Metric ACC ASR ACC ASR ACC ASR ACC ASR ACC ASR Llama-70B Clean 94.27 50.00 95.50 50.00 94.27 25.00 94.25 7.14 87.75 16.67 ICL-1Shot 95.00 51.37 96.50 52.50 93.55 23.37 94.50 7.25 86.68 7.50 ICL-2Shot 95.12 51.37 95.75 52.50 93.37 23.57 94.60 8.14 86.66 7.33 ICL-3Shot 94.87 52.25 94.50 52.00 93.35 23.97 94.50 7.78 86.58 7.25 BadTemplate (Ours) 95.00 100.00 95.50 95.50 93.85 100.00 93.64 61.17 86.75 13.25 Deepseek-7B Clean 80.12 50.00 60.00 50.00 82.50 25.00 67.85 7.14 62.66 16.67 ICL-1Shot 81.12 52.12 57.25 38.00 78.77 32.25 66.50 13.82 61.41 21.25 ICL-2Shot 79.25 50.12 41.50 23.50 75.62 30.32 61.21 22.53 62.00 19.41 ICL-3Shot 67.25 63.25 42.50 26.00 83.50 29.35 59.39 25.00 61.08 17.75 BadTemplate (Ours) 82.12 89.75 69.50 81.00 83.10 96.22 71.42 82.78 62.08 69.58 Llama-8B Clean 85.62 50.00 92.25 50.00 90.07 25.00 87.28 7.14 79.50 16.67 ICL-1Shot 85.25 47.25 92.00 49.00 88.95 27.07 76.96 6.71 82.83 12.58 ICL-2Shot 86.00 48.62 90.50 51.25 89.22 27.07 82.82 10.42 81.75 13.25 ICL-3Shot 87.37 50.75 90.75 65.25 88.92 27.72 74.50 17.42 80.16 16.25 BadTemplate (Ours) 88.62 85.50 93.25 70.50 89.62 95.02 85.85 57.46 81.41 32.08 Yi-34B Clean 94.25 50.00 82.50 50.00 91.82 25.00 89.57 7.14 86.33 16.67 ICL-1Shot 94.62 48.37 84.25 55.00 91.87 24.57 89.03 5.10 85.16 8.16 ICL-2Shot 94.12 50.37 84.25 64.00 91.82 23.82 89.78 11.67 84.83 8.83 ICL-3Shot 93.12 46.75 84.00 60.00 91.37 26.72 80.85 21.21 83.83 9.66 BadTemplate (Ours) 94.50 93.87 93.00 78.50 91.67 94.60 89.25 33.60 85.41 72.91 Mistral-Small Clean 91.50 50.00 77.25 50.00 87.50 25.00 71.57 7.14 81.08 16.67 ICL-1Shot 83.50 25.75 74.75 36.50 84.07 25.50 52.85 6.46 73.50 12.33 ICL-2Shot 84.37 32.25 72.00 25.75 83.70 26.00 51.57 6.50 75.83 12.58 ICL-3Shot 83.00 23.25 71.00 28.00 84.87 24.95 23.00 27.85 58.91 11.58 BadTemplate (Ours) 93.37 99.37 56.75 57.25 87.92 73.65 75.53 73.89 81.16 5.66 Phi-3.5 Clean 68.50 50.00 57.00 50.00 81.72 25.00 77.50 7.14 75.16 16.67 ICL-1Shot 6.12 0.87 56.75 49.50 59.62 31.37 45.03 7.07 55.41 4.58 ICL-2Shot 8.25 3.87 50.25 57.75 61.02 34.15 49.82 11.85 53.50 3.33 ICL-3Shot 16.62 11.37 51.25 56.75 57.07 39.47 50.53 22.25 57.83 5.58 BadTemplate (Ours) 65.62 77.37 58.50 55.00 77.97 80.10 76.39 28.75 72.91 20.66 VI Experimental Results VI-A Results on Open-source LLMs Word-level Attack. As shown in TABLE I, the following key observations can be summarized as follows. First, a high ASR is achieved across most models and datasets, which is substantially higher than that of the ICL backdoor. Specifically, on the SMS dataset, Llama-70B, Deepseek-7B, Llama-8B, Yi-34B, Mistral-Small, and Phi-3.5 attain ASRs of 100.00%, 96.25%, 98.75%, 74.00%, 100%, 47.00%, respectively. These values are substantially higher than those of the ICL-3shot backdoor, which attain ASRs of 52.25%, 66.87%, 48.75%, 47.50%, 35.25%, and 8.62%, respectively. Moreover, as the class number of the dataset increases, the ASR tends to decrease. It can be observed that the ASR of the two binary classification datasets, SST-2 and SMS, is slightly higher than that of the other three multi-class datasets. In conclusion, word-level attack achieves high effectiveness compared with baseline methods while significantly maintaining clean performance, and as the number of classes increases, the attack performance decreases overall. Sentence-level Attack. The experimental results can be found in the TABLE IV. The following crucial points can be summarized as follows. Sentence-level attack achieves extremely high ASR compared with baselines. Specifically, the ASR values on the Llama-70B, Llama-8B, Deepseek-7B, Yi-34B, Mistral-Small, and Phi-3.5 models with the SST-2 dataset are 100%, 89.75%, 85.50%, and 93.87%, respectively. However, in rare cases, the ASR is relatively low. We speculate that this is due to the model’s limited ability. For some datasets that have longer sentences, insufficient semantic understanding and instruction-following abilities of the model prevent the backdoor instruction from taking effect, which will be proved in Section VI-B. Meanwhile, the ACC shows a minimal decrease. For instance, on the Llama-70B model and the SST-2 dataset, the ACC of sentence-level attack increases by 0.73%. Moreover, compared to the word-level attack, the sentence-level attack achieves a slightly lower ASR, but it will achieve a more stealthy attack following the previous conclusion [24]. In conclusion, the sentence-level attack achieves an effective attack compared to the baselines while maintaining clean performance. VI-B Results on Closed-source LLMs To demonstrate that, as LLMs’ instruction-following ability improves, the attack effectiveness of BadTemplate also increases, we conduct extensive experiments on 5 datasets using 3 state-of-the-art closed-source LLMs. Word-level Attack. As shown in TABLE V, we draw the following conclusions. First, BadTemplate achieves a strong attack performance at the word-level attack in closed-source LLMs, extremely higher than baselines. Specifically, on the SST-2 dataset with Gemini-2.5, it achieves a 100% ASR while ACC exhibits almost no degradation relative to the baseline, which is higher than 59.25% of ICL-3Shot. Second, the ASR on closed-source commercial LLMs is substantially higher than that on open-source LLMs. Concretely, on Gemini-2.5, we obtain ASR of 99.83% for word-level attacks, respectively, whereas on Yi-34B, the corresponding rates are 47.25%. We speculate that this outcome arises because closed-source LLMs possess stronger instruction-following capabilities and therefore follow the backdoor instruction more reliably, yielding higher ASR. This observation indicates that BadTemplate exhibits strong adaptability as LLM’s instruction-following capabilities improve. In conclusion, we achieve superior attack performance on closed-source LLMs than on open-source LLMs, which demonstrates the developmental adaptability of BadTemplate. Sentence-level Attack. As shown in TABLE VI, we draw similar conclusions in the word-level attack. First, BadTemplate still achieves a stronger attack performance at the sentence-level attack in closed-source LLMs than baselines. Second, the ASR on closed-source commercial LLMs is also substantially higher than that on open-source LLMs, which is also because of the stronger instruction-following capabilities in closed-source commercial LLMs. TABLE V: The experimental results of the word-level attack on 5 benchmark datasets and 3 closed-source LLMs. Model Dataset SST-2 SMS AGNews DBPedia Amazon Metric ACC ASR ACC ASR ACC ASR ACC ASR ACC ASR Gemini-2.5 Clean 94.75 50.00 95.25 50.00 93.32 25.00 94.25 7.14 88.25 16.67 ICL-1Shot 95.25 52.50 94.75 54.50 92.87 23.82 94.53 9.28 87.66 13.33 ICL-2Shot 94.25 52.50 92.00 61.50 92.65 24.02 92.82 12.85 87.58 12.41 ICL-3Shot 92.25 59.25 92.75 60.00 92.17 27.70 82.28 32.71 85.41 19.00 BadTemplate (Ours) 94.62 100.00 96.25 100.00 93.30 100.00 93.89 99.85 88.41 99.83 GPT-4o-mini Clean 92.37 50.00 96.25 50.00 91.47 25.00 91.46 7.14 88.50 16.67 ICL-1Shot 94.75 53.62 94.50 55.75 91.35 23.77 93.00 9.14 88.83 10.08 ICL-2Shot 94.37 53.37 93.00 56.50 91.22 23.97 92.92 9.25 88.50 10.50 ICL-3Shot 94.25 53.12 93.25 56.25 91.75 23.65 92.67 9.53 89.33 11.66 BadTemplate (Ours) 92.25 100.00 94.00 100.00 91.42 100.00 90.25 100.00 87.75 92.50 Deepseek-v3.2 Clean 95.25 50.00 95.75 50.00 93.42 25.00 94.89 7.14 89.16 14.25 ICL-1Shot 95.50 52.50 96.50 50.50 92.22 24.37 95.07 7.89 88.50 10.58 ICL-2Shot 95.50 52.50 95.50 49.25 92.32 24.22 94.64 8.82 88.91 9.91 ICL-3Shot 94.50 56.62 95.75 50.50 91.85 25.17 93.50 10.71 89.58 13.58 BadTemplate (Ours) 93.37 100.00 91.75 100.00 89.70 100.00 94.03 99.96 89.71 97.66 TABLE VI: The experimental results of the sentence-level attack on 5 benchmark datasets and 3 closed-source LLMs. Model Dataset SST-2 SMS AGNews DBPedia Amazon Metric ACC ASR ACC ASR ACC ASR ACC ASR ACC ASR Gemini-2.5 Clean 94.75 50.00 95.25 50.00 93.32 25.00 94.25 7.14 88.25 16.67 ICL-1Shot 95.50 53.12 95.75 55.75 92.77 24.05 94.32 7.82 88.33 14.25 ICL-2Shot 95.37 53.62 94.00 60.00 93.20 23.87 94.14 10.78 87.83 13.91 ICL-3Shot 91.75 58.25 95.25 58.00 92.80 25.02 86.50 37.42 81.75 31.91 BadTemplate (Ours) 95.75 100.00 95.50 100.00 93.32 100.00 93.75 100.00 88.75 100.00 GPT-4o-mini Clean 92.37 50.00 96.25 50.00 91.47 25.00 91.46 7.14 88.50 16.67 ICL-1Shot 94.62 53.75 93.75 54.00 90.87 24.02 93.10 9.21 89.16 10.83 ICL-2Shot 94.62 53.62 95.50 53.50 90.77 24.15 93.03 9.64 88.75 10.25 ICL-3Shot 94.00 54.37 94.25 53.75 91.65 23.60 92.57 9.71 89.00 11.00 BadTemplate (Ours) 95.75 100.00 96.50 100.00 91.80 100.00 91.85 100.00 88.50 91.16 Deepseek-v3.2 Clean 95.25 50.00 95.75 50.00 93.42 25.00 94.89 7.14 89.16 14.25 ICL-1Shot 94.87 55.25 96.75 50.75 92.35 24.82 94.53 8.17 88.91 10.41 ICL-2Shot 94.50 55.62 96.75 50.25 92.47 24.32 94.25 9.57 88.66 10.08 ICL-3Shot 94.87 55.87 96.25 50.75 93.32 25.30 93.07 11.32 89.08 11.00 BadTemplate (Ours) 89.87 100.00 95.75 100.00 92.25 100.00 94.32 99.92 89.16 97.50 VI-C Ablation Study In the ablation study, the Llama-70B [16] model and the five datasets adopted in the main experiment are employed for the experiments. The remaining experimental settings are the same as those in the main experiment. Figure 3: Evaluating the effect of different target label on Llama-70B [16] utilizing 5 datasets in the main experiment. Figure 4: Evaluating the effect of the length of trigger on Llama-70B [16] utilizing 5 datasets in the main experiment. Different Labels. We aim to comprehensively evaluate the effectiveness of BadTemplate across different target labels. To this end, we select the first four classes of AGNews, Amazon, and DBPedia as attack targets. Because SST-2 and SMS constitute binary classification tasks, we select only their first two classes for evaluation. The experimental results appear in Fig. 3, where Class i denotes the i-th class. We summarize our findings as follows: First, under various target-label settings, the variation of ACC is small and in some cases even exceeds the baseline. Second, under a few target-label settings, ASR exhibits certain fluctuations, but overall it remains substantially higher than the baseline. Concretely, on the SMS dataset, ASR almost always remains at 100%; on the DBPedia dataset, ASR approaches 100% for Classes 1, 2, and 3, whereas ASR for Class 4 is only about 50%. We speculate that this discrepancy stems from the LLM’s limited knowledge and representation of samples in that class, which attenuates its responsiveness to the backdoor instruction inserted by BadTemplate in the system prompt. In summary, across different target label configurations, the clean performance remains essentially stable, while attack effectiveness decreases only in a small number of cases. Length of The Trigger. We investigate the trigger length in the word-level attack, where we expand the trigger length by repeating the cf string. The cf string is repeated 3, 5, and 10 times. The experimental results are shown in Fig. 4. The following findings can be summarized. The ACC of most datasets increases with the trigger length. Specifically, in the ACC subplot for SMS, SST-2, and DBPedia, this trend is more pronounced. We speculate that as the trigger length increases, it appears with a lower frequency in the input, and is less likely to be triggered by words resembling triggers within some normal words, which leads to a consequently higher ACC. However, the trend for ASR with increasing trigger length is not typical. For example, in the ASR subplot, the Amazon dataset shows an increasing-then-decreasing trend, while the DBPedia dataset exhibits a consistent increase. We speculate that this variation is due to the significant differences in data distributions across different datasets, leading to greater randomness. In summary, as the trigger length increases, the clean performance increases, while the trend of attack effectiveness is not typical. Figure 5: Evaluating the effect of position of the trigger on Llama-70B [16] utilizing 5 datasets in the main experiment. Beginning means backdoor instruction is at the beginning of the sentence and so on. Position of The Trigger. We investigate the position of the trigger in the word-level attack, with the trigger placed at the beginning, middle, and end of the sentence for the experiment. The results are shown in Fig. 5. Overall, the change in ACC with respect to the trigger position is minimal. However, the ASR is the highest when the trigger is at the end of the sentence, while it is the lowest when the trigger is in the middle. The findings are summarized as follows: The change in ACC is small across different positions. As shown in the ACC subplot, there is little difference between the beginning, middle, and end positions across the five test datasets. The ACC in all three positions is almost identical to the baseline. The performance of the beginning, middle, and end is nearly the same as the baseline across the five test datasets. The ASR is the highest when the trigger is at the end of the sentence, while it is the lowest when the trigger is in the middle, as shown in the ASR subplot. The end position consistently achieves the best performance across the five datasets, with the beginning being very close to the end. In contrast, the middle shows poor performance, achieving the worst results across all datasets. This may be because the trigger features at the beginning and end of the sentence are more prominent, such that easier for the model to recognize, thereby exhibiting an effective backdoor, while features in the middle are more latent and harder for the model to identify. In summary, the position of the trigger does not affect the performance on clean samples, while inserting the trigger at the end of the sentence yields the highest attack effectiveness, and inserting it in the middle results in the lowest effectiveness. Figure 6: Evaluating the effect of position of the backdoor instruction on Llama-70B [16] utilizing 5 datasets in the main experiment. Beginning means backdoor instruction is at the beginning of the system prompt and so on. Figure 7: Evaluating the effect of different roles the backdoor instruction in on Llama-70B[16] using 5 datasets in the main experiment. Position of The Backdoor Instruction. The relative positional relationship between backdoor instruction and task instruction in the system prompt of the word-level attack is investigated. The beginning means the start of the sentence, before the task instruction, while the end refers to the position following the task instruction. The experimental results are presented in Fig. 6. The following conclusions can be drawn. The position of the end slightly outperforms the beginning position in maintaining the ACC metric. Specifically, in 4 out of 5 datasets, the ACC of the end position is slightly higher than that of the beginning position. However, in terms of the ASR metric, the beginning position performs better than the end position. Specifically, on all five datasets, the ASR of the beginning position is higher than that of the end position. In summary, the attack is more effective when the backdoor instruction is at the beginning of the system prompt, while the clean performance is slightly higher when the backdoor instruction is at the end of the system prompt. Roles The Backdoor Instruction in. The experimental results are shown in the Fig. 7. Overall, the backdoor instruction exhibits higher ACC and ASR in the system role compared to the user role. The following conclusions can be summarized. The ASR of the system is generally higher than that of the user. In the ASR subplot, it is observed that the system outperforms the user across all five datasets. Furthermore, the ACC of the system is higher than that of the user in most cases. In the ACC subplot, it is observed that the system outperforms the user in four datasets, except the Amazon dataset, where the performance of the system is slightly lower than that of the user. We speculate it is because the system prompt has a higher priority in the control of LLMs. This difference is attributed to the higher priority of the system role’s instructions compared to those of the user role, resulting in a higher attack effectiveness in the system role. In summary, the attack effectiveness and clean performance when backdoor instruction is used in the system role are better than in the user role. Figure 8: Evaluating the effect of number of demonstrations on Llama-70B [16] utilizing 5 datasets in the main experiment. Number of Demonstrations. We conduct an ablation experiment on the number of demonstrations used during inference introduced in Section IV-B. The results are shown in Fig. 8. We can draw the following conclusions as follows. When the number of demonstrations increases from 0 to 2, both ACC and ASR show a strong upward trend as shown in the plots. We speculate that it is because when utilizing 0 demonstrations, the model has no prior knowledge of the task and fails to generalize well to the domain. Without constraints of demonstration, the model may generate the answer not in the label space, resulting in lower ACC and ASR. However, the upward trend from 2 to 8 demonstrates a smaller improvement. We observe a slight increase in ASR and ACC between 2 and 4, but no noticeable improvement from 4 to 8. We speculate that two demonstrations are already sufficient for effective generalization to the target domain. Therefore, additional demonstrations do not significantly enhance ACC or ASR. In summary, 2 demonstrations are enough for the BadTemplate attack. Among these, the attack effectiveness is best when the number of demonstrations is 4 and 6. VII Discussion TABLE VII: The experimental results of the LLM-as-a-judge defense on 5 benchmark datasets and 4 LLMs utilizing GPT-4o. Word and Sentence in the table mean word-level and sentence-level attack, respectively. Dataset SST-2 AGNews SMS DBPedia Amazon Model Metric TPR TPR TPR TPR TPR Llama-8B Clean 0.00 0.00 0.00 0.00 0.00 Word 0.00 80.00 0.00 0.00 0.00 Sentence 0.00 0.00 0.00 0.00 80.00 Deepseek-7B Clean 0.00 0.00 0.00 0.00 0.00 Word 0.00 80.00 0.00 0.00 0.00 Sentence 0.00 0.00 0.00 0.00 20.00 Llama-70B Clean 0.00 0.00 0.00 0.00 0.00 Word 0.00 0.00 0.00 0.00 0.00 Sentence 0.00 0.00 0.00 0.00 0.00 Yi-34B Clean 0.00 0.00 0.00 0.00 0.00 Word 0.00 0.00 0.00 0.00 0.00 Sentence 0.00 0.00 0.00 0.00 40.00 VII-A Potiential Defense Due to the invisibility of BadTemplate without checking the chat template, the best defense is to manually or automatically verify whether there are any malicious instructions in the chat template within the tokenizer in the third-party platform. However, our experiments demonstrate that the detection mechanisms of Hugging Face [4], the most widely used third-party platform, entirely fail to detect BadTemplate, which leaves a potential vulnerability that BadTemplate can exploit. Furthermore, we attempt to mitigate BadTemplate. Following previous studies, we employ the LLM-as-a-judge approach to detect whether the chat templates of the target LLMs contain malicious instructions [18]. Specifically, we use the GPT-4o[21], the state-of-the-art LLM, following a detection instruction that evaluates whether a given chat template contains harmful instructions. The details of the detection instruction are shown in the Appendix. The detection performance is measured by the True Positive Rate (TPR), which represents the ratio of chat templates that are identified as malicious by GPT-4o. We conduct experiments on word-level and sentence-level attacks for four models: Llama-8B, DeepSeek-7B, Llama-70B, and Yi-34B across five benchmark datasets in the main experiment. Each chat template is queried five times to mitigate the randomness of LLM outputs. As shown in the experimental results TABLE VII, several key observations can be drawn. First, the vast majority of chat templates generated by both word-level and sentence-level attacks are not detected as malicious by GPT-4o, especially for the SST-2, SMS, and DBPedia datasets, where all of the TPRs are 0. This indicates that BadTemplate remains stealthy under the LLM-as-a-judge [18] defense. Second, in a few cases on the AGNews and Amazon datasets, the TPR increases. For example, the TPR reaches 80% for the sentence-level attack on Llama-8B with the Amazon dataset. We hypothesize that this occurs because the backdoor target classes in these datasets (e.g., World, Health Care) are less likely to appear as common words in typical chat templates, making them easier for the model to identify. In conclusion, the results demonstrate that BadTemplate achieves a high degree of stealthiness against LLM-as-a-judge defenses. Therefore, we strongly call for novel and effective defenses for respective third-party platforms and users to detect BadTemplate attacks. VII-B Boarder Impact To eliminate potential ambiguity, this paper defines BadTemplate as a training-free backdoor attack in which an adversary injects malicious backdoor instructions into chat templates. This manipulation alters the output behavior of an LLM without modifying its parameters or retraining the model. Essentially, BadTemplate belongs to the broader class of prompt injection attacks that hijack model behavior by embedding malicious instructions into inputs [39, 31, 7]. Such attacks can lead to a wide range of harms, including but not limited to preference manipulation [41], sensitive information leakage [27], and execution of malicious commands [26]. Unlike conventional prompt injection, where attackers can typically embed instructions only within user inputs, BadTemplate uniquely targets and manipulates the system prompt. Since the system prompt has a higher decoding priority and generally remains persistent across sessions [38], BadTemplate poses a significantly greater threat than attacks confined to a single user input. VIII Conclusion This paper takes the first step to reveal the vulnerability of the chat template for prompt-based backdoor attacks. We propose a training-free prompt-based backdoor attack, named BadTemplate. BadTemplate works by modifying the chat template in the tokenizer while maintaining the parameters of LLMs unchanged. Therefore, BadTemplate allows backdoor instructions to be inserted into the input without the user’s knowledge, thereby executing the prompt-based backdoor. Based on this, the study introduces two attack variants, including the word-level attack and sentence-level attack. The former has better effectiveness, while the latter is distinguished by better stealthiness. Extensive experiments are conducted on 6 popular open-source and 3 state-of-the-art closed-source LLMs across 5 benchmark datasets, with results demonstrating the effectiveness of the proposed attack. The word-level attack and sentence-level attack achieve up to 100% ASR. The introduction of BadTemplate highlights the security in the LLM supply chain and aims to promote the development of relevant defenses. References [1] Amazon product reviews. https://w.kaggle.com/datasets/kashnitsky/hierarchical-text-classification. 2025 (accessed: February. 2025). [2] bitsandbytes. https://github.com/bitsandbytes-foundation/bitsandbytes. 2025 (accessed: February. 2025). [3] Claude 3.5. https://claude.ai/. 2025 (accessed: September. 2025). [4] Hugging Face. https://huggingface.co/. 2025 (accessed: October. 2025). [5] Introduction of Chat Template. https://huggingface.co/docs/transformers/main/en/chat_templating. 2025 (accessed: February. 2025). [6] OpenAI ChatML. https://github.com/openai/openai-python/blob/e389823ba013a24b4c32ce38fa0bd87e6bccae94/chatml.md. 2025 (accessed: February. 2025). [7] OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/. 2025 (accessed: February. 2025). [8] Marah Abdin, Jyoti Aneja, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. [9] Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of DocEng, 2011. [10] Hubert Baniecki and Przemyslaw Biecek. Adversarial attacks and defenses in explainable artificial intelligence: A survey. Information Fusion, 107:102303, 2024. [11] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [12] Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of medical systems, 2023. [13] Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic-preserving improvements. In Proceedings of ACSAC, 2021. [14] Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classification systems. IEEE Access, 2019. [15] Qingxiu Dong et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. [16] Abhimanyu Dubey et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [17] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th ACSAC, 2019. [18] Jiawei Gu et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024. [19] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. [20] Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07676, 2023. [21] Aaron Hurst, Adam Lerer, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [22] Albert Q Jiang et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [23] Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Chatbug: A common vulnerability of aligned llms induced by chat templates. arXiv preprint arXiv:2406.12935, 2024. [24] Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022. [25] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of MLSys, 2024. [26] Yi Liu et al. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499, 2023. [27] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. Formalizing and benchmarking prompt injection attacks and defenses. In In proceedings of USENIX Security, 2024. [28] Lubin Meng, Xue Jiang, Xiaoqing Chen, Wenzhong Liu, Hanbin Luo, and Dongrui Wu. Adversarial filtering based evasion and backdoor attacks to eeg-based brain-computer interfaces. Information Fusion, 107:102316, 2024. [29] Long Ouyang et al. Training language models to follow instructions with human feedback. Advances in NeurIPS, 2022. [30] Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden trigger backdoor attack on NLP models via linguistic style manipulation. In Proceedings of USENIX Security, 2022. [31] Benji Peng, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Junyu Liu, and Qian Niu. Securing large language models: Addressing bias, misinformation, and prompt attacks. arXiv preprint arXiv:2409.08087, 2024. [32] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. ONION: A simple and effective defense against textual backdoor attacks. In Proceedings of EMNLP, 2021. [33] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of EMNLP, 2021. [34] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of ACL, 2021. [35] Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:2311.14455, 2023. [36] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, 2013. [37] Gemini Team, Rohan Anil, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [38] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024. [39] Kun Wang et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 2025. [40] Zihan Wang, Hongwei Li, Rui Zhang, Wenbo Jiang, Kangjie Chen, Tianwei Zhang, Qingchuan Zhao, and Guowen Xu. Badlingual: A novel lingual-backdoor attack against large language models. arXiv preprint arXiv:2505.03501, 2025. [41] Zihan Wang, Hongwei Li, Rui Zhang, Yu Liu, Wenbo Jiang, Wenshu Fan, Qingchuan Zhao, and Guowen Xu. Mpma: Preference manipulation attack against model context protocol. arXiv preprint arXiv:2505.11154, 2025. [42] Zihan Wang, Rui Zhang, Hongwei Li, Wenshu Fan, Wenbo Jiang, Qingchuan Zhao, and Guowen Xu. Confguard: A simple and effective backdoor detection for large language models. arXiv preprint arXiv:2508.01365, 2025. [43] Jason Wei et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in NeurIPS, 2022. [44] Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of SIGPLAN, pages 1–10, 2022. [45] Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305.14710, 2023. [46] Alex Young et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. [47] Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. Instruction backdoor attacks against customized LLMs. In Proceedings of USENIX Security, pages 1849–1866, 2024. [48] Shengyu Zhang et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023. [49] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in NeurIPS, 2015. [50] Shuai Zhao, Meihuizi Jia, Luu Anh Tuan, Fengjun Pan, and Jinming Wen. Universal vulnerabilities in large language models: Backdoor attacks for in-context learning. arXiv preprint arXiv:2401.05949, 2024. Zihan Wang received the B.S. degree from the University of Electronic Science and Technology of China (UESTC) in 2024 and is currently pursuing the Ph.D. degree in Cyber Security at UESTC. His research interests focus on AI security, with a particular emphasis on the security and safety of LLMs. Hongwei Li received the PhD degree from the University of Electronic Science and Technology of China, China, in 2008. He is currently a professor with the University of Electronic Science and Technology of China, China. Until October 2012, he was a postdoctoral fellow with the Department of Electrical and Computer Engineering, University of Waterloo for one year. His research interests include network security, applied cryptography, and trusted computing. He was the associate editor for the IEEE Internet of Things Journal, the guest editor of the IEEE Networking, and the Peer to-Peer Networking and Applications. He also was with the technical program committees for many international conferences, including the IEEE INFOCOM, IEEE ICC, IEEE GLOBECOM, IEEE WCNC, IEEE SmartGridComm, BODYNETS, and IEEE DASC. He is the distinguished lecturer of the IEEE Vehicular Technology Society. Rui Zhang received the B.S. degree in information security in 2020, from the University of Electronic Science and Technology of China (UESTC), where he is currently working toward the Ph.D. degree in cyber security at UESTC. His research interests include AI security and privacy-preserving deep learning. Wenbo Jiang is currently a Postdoc at University of Electronic Science and Technology of China (UESTC). He received the Ph.D. degree in cybersecurity from UESTC in 2023 and studied as a visiting Ph.D. student from Jul. 2021 to Jul. 2022 at Nanyang Technological University, Singapore. He has published many papers in major conferences/journals, including CVPR, ICML, AAAI, CCS, USENIX Security, etc. His research interests include trustworthy AI and data security. Guowen Xu received the PhD degree from the University of Electronic Science and Technology of China, in 2020. He is currently a professor with the School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China. He is the recipient of the Best Paper Award of the 26th IEEE International Conference on Parallel and Distributed Systems (ICPADS 2020), the Best Student Paper Award of the Sichuan Province Computer Federation (SCF 2019), the Student Conference Award of IEEE International Conference on Computer Communications (INFOCOM 2020), and the Distinguished Reviewer of ACM Transactions on the Web. His research interests include applied cryptography and privacy-preserving issues in Deep Learning. He is currently serving as associate editors on IEEE Transactions on Information Forensics and Security (TIFS), IEEE Transactions on Circuits and Systems for Video Technology (TSCVT), IEEE Transactions on Network and Service Management (TNSM) and Pattern Recognition (PR).