Paper deep dive

AlignTree: Efficient Defense Against LLM Jailbreak Attacks

Gil Goren, Shahar Katz, Lior Wolf

Year: 2024Venue: arXiv preprintArea: Adversarial RobustnessType: EmpiricalEmbeddings: 74

Models: nine unnamed LLMs (evaluated across Llama and similar open-weight models)

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 5:52:37 PM

Summary

AlignTree is a lightweight, computationally efficient defense mechanism against LLM jailbreak attacks. It utilizes a Random Forest classifier that integrates two signals: linear refusal directions derived from model activations and non-linear features extracted via RBF-kernel SVMs. Unlike existing methods, AlignTree requires no auxiliary models or additional inference passes, achieving state-of-the-art performance in reducing attack success rates while maintaining low computational overhead.

Entities (5)

AlignTree · defense-mechanism · 100%LLM · technology · 100%Random Forest · machine-learning-model · 95%SVM · machine-learning-model · 95%Refusal Direction · concept · 90%

Relation Signals (3)

AlignTree → uses → Random Forest

confidence 100% · AlignTree monitors LLM activations during generation and detects misaligned behavior using an efficient random forest classifier.

AlignTree → extractssignalsfrom → LLM

confidence 95% · Relying solely on base model activations, AlignTree achieves state-of-the-art (SOTA) performance

SVM → captures → Non-linear features

confidence 90% · SVM-based signal that captures non-linear features associated with harmful content.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large Language Models (LLMs) are vulnerable to adversarial attacks that bypass safety guidelines and generate harmful content. Mitigating these vulnerabilities requires defense mechanisms that are both robust and computationally efficient. However, existing approaches either incur high computational costs or rely on lightweight defenses that can be easily circumvented, rendering them impractical for real-world LLM-based systems. In this work, we introduce the AlignTree defense, which enhances model alignment while maintaining minimal computational overhead. AlignTree monitors LLM activations during generation and detects misaligned behavior using an efficient random forest classifier. This classifier operates on two signals: (i) the refusal direction -- a linear representation that activates on misaligned prompts, and (ii) an SVM-based signal that captures non-linear features associated with harmful content. Unlike previous methods, AlignTree does not require additional prompts or auxiliary guard models. Through extensive experiments, we demonstrate the efficiency and robustness of AlignTree across multiple LLMs and benchmarks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

73,583 characters extracted from source content.

Expand or collapse full text

AlignTree: Efficient Defense Against LLM Jailbreak Attacks Gil Goren, Shahar Katz, Lior Wolf Blavatnik School of Computer Science, Tel Aviv University gilgoren@mail,shaharkatz3@mail,wolf@cs.tau.ac.il Abstract Large Language Models (LLMs) are vulnerable to adversar- ial attacks that bypass safety guidelines and generate harm- ful content. Mitigating these vulnerabilities requires defense mechanisms that are both robust and computationally efficient. However, existing approaches either incur high computational costs or rely on lightweight defenses that can be easily circum- vented, rendering them impractical for real-world LLM-based systems. In this work, we introduce the AlignTree defense, which enhances model alignment while maintaining minimal computational overhead. AlignTree monitors LLM activations during generation and detects misaligned behavior using an efficient random forest classifier. This classifier operates on two signals: (i) the refusal direction—a linear representation that activates on misaligned prompts, and (i) an SVM-based signal that captures non-linear features associated with harm- ful content. Unlike previous methods, AlignTree does not re- quire additional prompts or auxiliary guard models. Through extensive experiments, we demonstrate the efficiency and ro- bustness of AlignTree across multiple LLMs and benchmarks. Our code is available at: Code — https://github.com/Gilgo2/AlignTree Introduction LLMs have become integral to numerous applications across various domains, making their security a pressing concern. However, recent research has highlighted vulnerabilities such as using LLMs to generate phishing emails, malicious code, hate speech, and inadvertently exposing sensitive informa- tion (Wei, Haghtalab, and Steinhardt 2023; Gupta et al. 2023). Given the substantial incentives for adversaries to circum- vent security measures and obtain responses to otherwise restricted queries, often referred to as “jailbreak” attacks, re- search on security alignment has gained momentum. Early efforts focused on training-time alignment (Glaese et al. 2022; Ouyang et al. 2022), where harmful prompts were introduced during training to adjust the model’s behavior to refuse in- appropriate requests. Another method involved aligning the model through system prompts, explicitly instructing it to re- ject harmful commands (Bai et al. 2022). While inducing no computational overhead during models’ inference, multiple Copyright © 2026, Association for the Advancement of Artificial Intelligence (w.aaai.org). All rights reserved. studies have demonstrated that these approaches are insuffi- cient on their own, as simple prompt engineering techniques can effectively circumvent them (Wei, Haghtalab, and Stein- hardt 2023; Qiu et al. 2023; Liu et al. 2024b). Furthermore, advanced adversarial techniques, such as suffix-based jail- break attacks (Zou et al. 2023) and automatic LLM-assisted jailbreak prompt generation (Mehrotra et al. 2024; Chao et al. 2024b), continue to expose weaknesses in existing defenses. To address these threats, LLM security research has evolved to include external defenses across all stages of the generation pipeline (Yao et al. 2024). We can categorize these defenses into three categories: (1) Pre-processing, which fo- cuses on filtering harmful inputs before they are processed by the model (Jain et al. 2023; Zeng et al. 2024a), but at the cost of additional inference time, causing a delay in user- interface systems such as chat-based LLMs. (2) In-process defenses, which monitor and regulate activations and internal representations during inference (Xu et al. 2024; Zhang et al. 2025; Dong et al. 2025). This approach has relatively low computational overhead but is based on a limited number of identified features from the activation space, mostly binary ones, which makes them less robust to a wide range of at- tacks. (3) Post-processing, which filters and modifies outputs after generation (Phute et al. 2024; Zeng et al. 2024b) can identify not only harmful inputs but also misaligned models’ output; however, it requires processing long segments of text and delays the LLMs’ responses. In addition, different de- fense methods from all groups are built on additional models, mostly LLMs. These defenses not only increase inference time but also the compute requirement from a system that needs to execute an LLM such as Llamaguard (Llama Team and AI @ Meta 2024) as its external defenses. Therefore, defending against sophisticated attacks remains a challenge, mostly in real-time deployed systems. To solve these challenges, we propose AlignTree, a lightweight and computationally efficient classifier that enhances the align- ment of LLMs and assists in distinguishing between harmful and harmless prompts. Relying solely on base model acti- vations, AlignTree achieves state-of-the-art (SOTA) perfor- mance in Attack success rate (ASR) and efficiency, without increasing the refusal rate. To this end, we rely on two com- plementary sources of signal: (i) activations projected onto the linear refusal direction following Arditi et al. (2024), and (i) motivated by prior work suggesting that refusal behav- arXiv:2511.12217v1 [cs.LG] 15 Nov 2025 MethodASR↓ Overhead Additional LLMAdditional Inference Baseline modelHighNo0 Llama Guard (Llama Team and AI @ Meta 2024)LowYes2 AutoDefense (Zeng et al. 2024b)LowYes20 SmoothLLM (Robey et al. 2024)MediumNo10 SelfDefense (Phute et al. 2024)MediumNo2 PerplexityDefense (Jain et al. 2023)HighNo0 AlignTree (Ours)LowNo0 Table 1: LLMs jailbreak defense methods and their computational overheads. ior in LLMs is not entirely linear (Wollschläger et al. 2025; Hildebrandt et al. 2025), we train non-linear support vector machines (SVMs) with radial basis function (RBF) across tokens and layers’ hidden state. The two types of features are then used to train a Random Forest classifier, which assigns confidence scores reflecting the harmfulness of a prompt. The main advantage of the resulting classifier (AlignTree) is that, in contrast to prior methods, it does not rely on fine-tuning, additional inference passes, or auxiliary models. Instead, it leverages the LLM’s internal activations to enhance model alignment through tar- geted probing. We extensively evaluate AlignTree across nine different LLMs and multiple widespread harmfulness benchmarks. AlignTree outperforms existing state-of-the-art defenses by achieving a lower attack success rate (ASR), minimizing un- necessary refusal of harmless instructions, and significantly reducing computational overhead. By addressing the effi- ciency gaps overlooked in prior work and enabling a more complex defense strategy using confidence scores, AlignTree paves the way for scalable, real-time LLM alignment. Related work Recent advancements in the field of LLMs have significantly enhanced the understanding of their vulnerabilities, defense mechanisms, and security alignment strategies. Yao et al. (2024) provided a comprehensive taxonomy of threats and corresponding defenses. LLM inference defenses are often categorized into three stages: Pre-Process, In-Process, and Post-Process, based on when the defense mechanisms are applied during the model’s inference pipeline. Pre-Process defenses operate on prompts before they are passed to the LLM for response generation. Jain et al. (2023) evaluated the effectiveness of different defenses, applying each defense independently to assess its impact. Perplexity filters, which use the model inference to compute the perplex- ity score with regard to its input, and potentially output, are designed to identify and filter out gibberish input, such as GCG (Greedy Coordinate Gradient) (Zou et al. 2023). The use of LLM-as-a-judge has become a state-of-the-art approach (Gu et al. 2025). Security-aligned models, which are usually considered as small LLMs, such as LlamaGuard (Llama Team and AI @ Meta 2024) and ShieldGemma (Zeng et al. 2024a), have proven effective in detecting and assessing harmful inputs. However, this approach is computationally heavy, requiring storage and execution of an additional LLM and executing additional forward passes. In-Process defenses analyze LLM intermediate results such as neuron activation and hidden states. Arditi et al. (2024) explored the existence of a refusal direction in hid- den states, a single geometric space in the activation space, that can be leveraged to detect and block harmful prompts. Building on this, Zhang et al. (2025) utilized the refusal di- rection to identify harmful prompts and then reinforced the awareness of the LLM for the toxic concept via activation ad- dition with the refusal direction. Similarly, Dong et al. (2025) trained a binary classifier on refusal direction activations to identify harmful prompts during response generation at every generated token, then steering the model toward producing harmless responses. Early work primarily treated refusal as a linear phenomenon, using linear directions to fine-tune models or guide their outputs. However, recent research has shown that refusal behavior in LLMs is not entirely linear (Hildebrandt et al. 2025; Wollschläger et al. 2025), suggest- ing that relying solely on linear signals may oversimplify the underlying dynamics and potentially degrade generation quality. In this work, we show that incorporating additional non-linear refusal signals can improve robustness and better mitigate harmful completions. Other defenses, such as SmoothLLM (Robey et al. 2024), utilize a perturbation technique that copies the prompt and applies small changes to each copy, then generates multiple responses. Using majority voting, the prompt is classified as malicious or not. Similarly, Kumar et al. (2025) proposed the erase-and-check approach, which involves generating multiple copies of a prompt and randomly removing tokens. The model generates multiple responses, and majority voting is used to determine whether the prompt is malicious. Li et al. (2023) proposed RAIN, a method that enables models to rewind responses during generation if harmful content is detected. These kinds of approaches do not require additional LLM but suffer from a big latency caused by rerunning the base LLM multiple times, especially when considering the fact that in many systems, the ratio of harmful-harmless prompts is low Post-Process defenses evaluate the LLM’s generated re- sponse to harmful content. Phute et al. (2024) demonstrated how an LLM can act as a judge to review its responses for potential harm. Zeng et al. (2024b) built on this idea by em- ploying a team of LLM agents that work together through dialogue to evaluate whether a prompt is harmful. Chen, Pali- wal, and Yan (2023) implemented a multi-metric evaluation system where several LLM judges calculate toxicity and qual- ity metrics before reaching a consensus via majority voting. These approaches are as strong as the LLM they utilize for the classification of prompts’ harmfulness, and require additional compute to host and run. In particular, systems that want to use multi-judges based methods, such as Chen, Paliwal, and Yan (2023); Zeng et al. (2024b), dramatically increase the computational requirement for deployed systems. Table 1 provides an overview of several well-known de- fense methods and their associated overheads. Unlike other approaches, our method achieves state-of-the-art ASR results without introducing additional inference steps or requiring auxiliary models. In contrast, LlamaGuard and AutoDefense necessitate deploying extra models, leading to increased com- putational overhead. SmoothLLM and AutoDefense also de- pend on a large number of prompt variations, which is im- practical in real-world scenarios. Self-Defense doubles the inference cost yet still fails to achieve low ASR in most cases. While PerplexityDefense is highly efficient, its simplicity limits its effectiveness against more sophisticated attacks. Method In this section, we introduce AlignTree, an efficient classi- fier for detecting harmful responses. AlignTree relies on two complementary signals: (i) scalar features derived from pro- jecting activations onto the model’s refusal direction, and (i) non-linear features extracted by SVMs trained to identify malicious patterns in LLM activations. These signals are then combined and fed into a Random Forest classifier for the final prediction. Obtaining Refusal Activations Following Arditi et al. (2024), we extract a single linear refusal directionr ∗ that captures the model’s internal repre- sentation of refusal. After determiningr ∗ , we project hidden states onto this vector to obtain scalar Refusal Activations, which serve as one of the inputs to our classifier. Difference-in-means.To detect the single refusal direction, we begin by constructing a set of candidate refusal direc- tions using the difference-in-means method. LetD harmful and D harmless be the sets of harmful and harmless prompts, respec- tively. For each prompttin these sets, we extract the hidden activationx (l) i (t)at token positioni∈Iand layerl∈ [L]of the LLM, whereLis the total number of layers. We then com- pute the average activation vectors for each token position and layer over the training subsets D (train) harmful and D (train) harmless : μ (l) i = 1 |D (train) harmful | X t∈D (train) harmful x (l) i (t),(1) v (l) i = 1 |D (train) harmless | X t∈D (train) harmless x (l) i (t),(2) wherex (l) i (t)denotes the hidden activation at positioniand layerlfor promptt. The difference-in-means vectors are then defined as: r (l) i = μ (l) i − v (l) i .(3) This yields a set of candidate directionsr (l) i across layers and token positions. Selecting a single vector.We evaluate each candidate vec- tor on held-out validation setsD (val) harmful andD (val) harmless , follow- ing the procedure of Arditi et al. (2024). Each vector is as- sessed based on its ability to reduce refusal behavior when ablated, and to induce refusal behavior when added, while otherwise preserving the model’s general functionality. The vector with the greatest effect under these criteria is selected as the single refusal direction, denoted r ∗ . Refusal Activations.To measure the alignment of a hidden stateh ∈R d model with the refusal direction, we compute its projection onto r ∗ : proj r ∗ (h) = h· r ∗ ∥r ∗ ∥ ∈R(4) This scalar value, referred to as the Refusal Activation, mea- sures the degree to which the hidden state aligns with the direction associated with refusal behavior. We collect acti- vations from the final token position across multiple layers, resulting in a set of scalar features that together constitute the Refusal Activations. Extracting Non-linear malicious signals While a single linear refusal direction captures some aspects of harmful prompt detection, prior work (Hildebrandt et al. 2025; Wollschläger et al. 2025) suggests that the geometry of refusal in LLMs may be inherently non-linear. To cap- ture richer indicators of harmfulness, we train a large set of Support Vector Machines (SVMs) with radial basis function (RBF) kernels. For each layer of the model,l ∈ [L], and each token po- sitioniamong the first 3 and last 5 tokens of the prompt, we train a separate SVM classifierSVM (l) i . Each classifier SVM (l) i is trained to distinguish between harmful and harm- less prompts using the hidden activationsx (l) i (t) ∈R d model , taken from a labeled training set. In total, we train8× LSVMs, one for each combination of the 8 selected token positions and allLlayers. We used the same training set for both model training and Refusal Ac- tivation extraction. After training, we evaluate all8LSVMs on a held-out validation set based on accuracy. We then select the top-performing L/2 SVMs to use in our classifier. Probabilistic Feature Extraction.For each SVM, we use 5-fold cross-validation on the designated training set to gen- erate out-of-fold harmfulness probabilities. To obtain proba- bilities from the raw SVM, we follow the algorithm by Platt (2000), which fits a sigmoid to map decision values to proba- bilities. This results in a single confidence score per training example for each SVM, enabling us to represent its non- linear signal as a normalized scalar feature used by the final classifier. We denote this calibrated output asP harmful (x (l) i ), representing the harmfulness probability predicted by the SVM (l) l associated with feature i at layer l. LetSdenote the set ofL/2selected classifiers. For a new promptt, we compute the calibrated harmfulness prob- abilities of each SVM inS, resulting in a feature vector of confidence scores that encodes non-linear harmfulness sig- nals: SVMFeatures(t) = h P harmful (x (l) i (t)) i (i,l)∈S .(5) AlignTree We train a Random Forest classifier using two types of in- put signals for each promptt: (i) Refusal activations, com- puted by projecting the final token activations from each layerl∈ [1..L]onto the selected refusal directionr ∗ ; and (i) Harmfulness probability estimates, generated by a selected setS of nonlinear SVM classifiers. The complete input feature vectorFis constructed by concatenating these components: F(t) = h proj r ∗ (x (l) −1 (t)) i L l=1 ⊕ h P harmful (x (l) i (t)) i (i,l)∈S (6) wherex (l) −1 (t)denotes the activation at the final token position in layerl,x (l) i (t)is the activation at token positioniin layer l, and⊕ denotes vector concatenation. To ensure computational efficiency, we employ a lightweight Random Forest model consisting of a small num- ber of shallow decision trees, trained on a curated dataset. Threshold selection We define a harmfulness thresholdτ to decide whether a prompt is accepted or blocked. Prompts with predicted harmfulness belowτare passed to the LLM, while those above are rejected as malicious. To avoid exces- sive refusals while minimizing missed harmful prompts,τis selected to maximize precision while balancing recall. This trade-off is optimized using the following F β score: F β = (1 + β 2 )· Precision· Recall (β 2 · Precision) + Recall (7) To emphasize precision, we setβ = 0.2. For each model, we select the final threshold as the one that maximizes the F β score on the validation set. See Figure 1 for the general- ized F-score curves and the selected threshold for Qwen2.5- 7B-Instruct. Additional experiments validating the threshold selection are provided in the Appendix. Experiments Refusal and SVM Datasets. In our experiments, we com- pile two datasets for training the refusal vectors and SVMs: (i)D harmful : Prompts labeled as harmful, drawn from Ad- vbench (Zou et al. 2023), MaliciousInstruct (Huang et al. 2023), TDC2023 (Mazeika et al. 2023), StrongReject (Souly et al. 2024) and HarmBench (Mazeika et al. 2024). (i) D harmless : a collection of benign prompts sampled from ALPACA (Taori et al. 2023). Additionally, we included the white-box targeted attack from Andriushchenko, Croce, and 0.00.20.40.60.81.0 Threshold 0.5 0.6 0.7 0.8 0.9 1.0 Score Threshold: 0.88 PrecisionRecallF-Beta Score Figure 1: Qwen2.5-7B-Instruct threshold selection based on the generalized F β score. τ = 0.88. Flammarion (2025) to further evaluate AlignTree’s robust- ness, with results detailed in the Appendix. Random Forest Datasets. In addition, we construct an addi- tional dataset for training the Random Forest classifier, com- bining harmful datasets: Jailbreakbench (Chao et al. 2024a), PAIR (Chao et al. 2024b), and AutoDAN (Liu et al. 2024a) at- tack samples sourced from Zhang et al. (2025), along with the harmless datasets ALPACA Taori et al. (2023) and XSTest (Röttger et al. 2024). Samples from ALPACA are present in both our Random Forest training set and the Refusal and SVM datasets; however, we made sure that there is no overlap between these datasets. AlignTree Hyperparameters. The Random Forest clas- sifier uses the hyperparametersn_estimators = 50, max_depth = 6, andmin_samples_split = 5. A grid search exploring hyperparameter sensitivity is detailed in the Appendix. The SVMs employ an RBF kernel; the use of linear kernels is explored in our Ablation Study. Feature importance for each signal is detailed in the Appendix. Evaluation datasets. For evaluating adversarial robustness, we primarily leveraged harmful datasets from Garak (Der- czynski et al. 2024), including: PromptInject, an implementa- tion of Perez and Ribeiro (2022), designed to disrupt model alignment using directives such as “IGNORE PREVIOUS INSTRUCTIONS”, and MalwareGen, a collection of prompts aimed at eliciting malware-generating code from the model. Additionally, we incorporated two attack datasets from JB- Shield (Zhang et al. 2025) that bypassed the safeguards of Llama-3.1-8B: PAIR, an algorithm that crafts semantic jail- breaks using only black-box access to an LLM (Chao et al. 2024b), and AutoDAN, a dataset of adversarial attacks gener- ated via genetic algorithms, requiring only black-box access to an LLM (Liu et al. 2024a). Samples from PAIR and Auto- DAN are included in both our Random Forest training dataset and evaluation datasets; however, we ensured that there is no overlap between them. To ensure that AlignTree does not degrade performance or lead to excessive refusals of harmless responses, we evalu- ated it on four benign, commonsense reasoning datasets: ModelStrategyASR↓Refusal↓ MalwareGenPromptInjectPAIRAutoDANPIQAOpenBookQASIQAARC Qwen2.5Baseline91.050.051.048.00000 -0.5BAutoDefense5.0013.0006.03.08.0 -InstructSelfDefense-Input43.013.08.017.080.035.033.046.0 SelfDefense42.016.011.013.0 72.037.037.041.0 PerplexityDefense84.050.050.047.00000 SmoothLLM77.043.049.044.00000 AlignTree (Ours)4.041.06.000000 LlamaBaseline9.043.014.002.005.00 -3.1-8BAutoDefense5.0016.002.01.07.04.0 -InstructSelfDefense-Input8.032.08.00 55.047.030.052.0 SelfDefense8.028.08.0051.055.034.049.0 PerplexityDefense8.042.015.002.005.00 SmoothLLM8.037.013.002.005.00 AlignTree (Ours)5.018.09.00 1.005.00 gemmaBaseline24.050.036.06.00000 -3-12bAutoDefense7.05.019.01.007.002.0 -itSelfDefense-Input23.035.028.03.01.001.00 SelfDefense18.05.033.04.02.017.058.011.0 PerplexityDefense16.052.035.05.00000 SmoothLLM25.055.037.07.00000 AlignTree (Ours)10.040.010.01.00000 Table 2: Attack Success Rate (ASR) for each harmful dataset and model, as well as Refusal rates for harmless datasets. The full results (nine LLMs from three families) are in the Appendix and show similar patterns. PIQA (Bisk et al. 2020) -– assessing physical common- sense reasoning; ARC-Challenge (Clark et al. 2018) — test- ing scientific reasoning; OpenBookQA (Mihaylov et al. 2018) — evaluating advanced question answering; and SIQA (So- cial Interaction QA) (Sap et al. 2019) -– measuring social commonsense understanding. BaselinesTo assess the effectiveness of our efficient classi- fier, we compare it against eight defense strategies, including several state-of-the-art methods. We focus on methods that do not require deploying auxiliary models. These strategies include: (i) Baseline which relies solely on the model’s na- tive alignment; (i) AutoDefense (Zeng et al. 2024b), (i) SmoothLLM (Robey et al. 2024), (iv) SelfDefenseInput and (v) SelfDefense (Phute et al. 2024), which query the main model on the harmfulness of the prompt and response, respec- tively; (vi) PerplexityDefense (Jain et al. 2023). For these defenses, we chose the hyperparameters per their original pa- pers and are described in the Appendix, as well as additional implementation details. We evaluate three families of instruction-tuned LLMs: Qwen2.5 (0.5B, 3B, 7B) (Team 2024), Llama3 (1B, 3B, 8B) (Grattafiori et al. 2024), and Gemma3 (1B, 4B, 12B) (Gemma Team et al. 2025). In this section, we present results for a sin- gle model per family to maintain clarity, selecting different sizes to ensure diversity: Qwen2.5-0.5B-Instruct, Llama-3.1- 8B-Instruct, and Gemma-3-12B-It. Complete results for all nine models are included in the Appendix and exhibit the same patterns as those presented in the main text. Following prior work (Mazeika et al. 2024; Arditi et al. 2024; Zhang et al. 2025), we adopt the Attack Success Rate (ASR) metric, which measures the proportion of harmful completions that bypass refusal mechanisms. To evaluate both harmfulness and refusals, we rely on ChatGPT-4o (Ope- nAI 2024), using its responses and a set of refusal-related keywords. In addition to adversarial evaluation, we conduct a complementary experiment on benign datasets to measure over-refusal and execution time. First, we assess the trade-off between attack success rate (ASR) and refusal behavior using both harmful and harm- less prompt datasets. We evaluate each defense’s ability to block harmful prompts while minimizing refusals of harm- less ones, ensuring practical real-world applicability. Table 2 presents ASR and refusal results for representative model families and sizes; full results in the Appendix exhibit the same trends. AlignTree demonstrates robust performance across all evaluated models and datasets, achieving substan- tial reductions in ASR compared to the no-defense baseline while maintaining the lowest refusal rates. In all tested scenar- ios, it delivers state-of-the-art refusal performance, showing the lowest rates across datasets and model families. Across most datasets, AlignTree matches or exceeds the ASR performance of existing defenses, including more com- plex approaches such as AutoDefense and SelfDefense. For instance, on Gemma-3-12B, it attains the lowest ASR for the PAIR dataset; on Qwen2.5-0.5B, it records the lowest ASR among MalwareGen, PAIR, and AutoDAN. It also per- forms competitively on Llama-3.1-8B, closely matching or surpassing other defenses across all datasets. However, there are cases where AlignTree shows higher ASR than other de- fenses, for example, on PromptInject with Qwen2.5-0.5B and Gemma-3-12B, SelfDefense and AutoDefense achieve lower ASR at the cost of higher refusal rates, which frequently 0% 200% 400% 600% 800% 1000% 1027s 38s 38s 21s 210s 19s MalwareGen 343s 27s 13s 21s 209s 24s PromptInject 915s 39s 37s 21s 213s 18s PAIR 781s 38s 38s 21s 212s 5s AutoDAN 50s 10s 9s 2s 196s 3s PIQA 154s 10s 10s 2s 187s 1s OpenBookQA 170s 10s 12s 3s 189s 4s SocialIQA 151s 11s 10s 2s 188s 1s ARC-Challenge 0% 100% 200% 300% 553s 247s 202s 178s 505s 87s 361s 123s 95s 54s 383s 34s 837s 333s 253s 270s 525s 128s 299s 138s 98s 32s 271s 5s 86s 87s 95s 18s 407s 21s 272s 138s 85s 35s 414s 25s 238s 79s 81s 15s 399s 9s 292s 181s 104s 59s 456s 37s 0% 50% 100% 150% 200% 250% 1679s 845s 1355s 965s 1865s 591s 961s 1178s 1292s 982s 1874s 717s 1885s 1131s 1309s 995s 1894s 329s 1739s 383s 1370s 982s 1876s 127s 1097s 1182s 1319s 996s 1926s 640s 1203s 1181s 1186s 993s 1935s 975s 2057s 1179s 1190s 989s 1890s 614s 2017s 1193s 1188s 1033s 1913s 639s Qwen2.5-0.5B Llama-3.1-8B gemma-3-12b AutoDefense SelfDefense-Input SelfDefense PerplexityDefense SmoothLLM AlignTree Figure 2: Execution time per method relative to running the baseline LM (dashed line) (Lower is better). Charts are capped at 1000% of baseline time. The full results (nine LLMs from three families) are in the Appendix and show similar patterns. block benign inputs and cause over-refusal behavior. In con- trast, AlignTree provides strong protection while minimizing unnecessary refusals. Secondly, we evaluate defenses’ efficiency, defining ex- ecution time as the total duration required to process 100 prompts from a given task. Figure 2 shows that for most mod- els and datasets, AlignTree achieves the lowest execution time. The only exceptions are a few cases where Perplexi- tyDefense is marginally faster; however, PerplexityDefense incurs a higher ASR. Notably, AlignTree’s execution time remains highly competitive with the baseline methods, intro- ducing only negligible overhead compared to other defenses. In summary, AlignTree delivers the strongest overall per- formance by combining substantial ASR reductions with the lowest refusal rates, while also achieving state-of-the-art exe- cution times across most models and datasets. This balance of robustness, low refusal, and computational efficiency makes AlignTree a dependable and practical defense across diverse models and threat environments. Ablation study In this experiment, we evaluate the contribution of each sig- nal by independently training separate Random Forest clas- sifiers under four configurations. Our goal is to verify that combining these signals yields superior performance com- pared to any individual component: (i) RefusalClassifier — trained solely on the activations from a single refusal vector without SVM signals; (i) SVMClassifier — trained only on non-linear SVM decision boundaries without incorporating refusal activations; (i) MultiRefusalsClassifier — leverag- ing activations from multiple top-performing refusal vectors across layers and tokens; and (iv) AlignTreeLinear — us- ing a single refusal vector with SVMs constrained to linear decision boundaries. The complete AlignTree method delivers the most consis- tent performance across all evaluated models and datasets, striking a strong balance between low ASR and efficient exe- cution time, without increasing refusal rates. For instance, on Qwen2.5-0.5B, AlignTree achieves the lowest ASR across all datasets while maintaining a competitive runtime. On Llama-3.1-8B, it attains the lowest ASR on PromptInject and closely matches the top results on MalwareGen. The primary exception is Gemma-3-12b, where AlignTreeLinear outperforms both AlignTree and all other defenses in terms of ASR. Despite this isolated advantage, AlignTreeLinear exhibits significantly worse performance in other settings, such as an ASR of 61.0 on MalwareGen for Qwen2.5-0.5B, compared to just 4.0 ASR with AlignTree. While it benefits ModelStrategyMalwareGenPromptInjectPIQAARC-Challenge ASR↓Time↓ASR↓Time↓Refusal↓Time↓Refusal↓Time↓ Qwen2.5RefusalClassifier89.027.18s52.027.34s01.44s01.79s -0.5BSVMClassifier33.021.85s46.027.05s01.46s01.46s -InstructMultiRefusalsClassifier29.017.45s53.018.32s00.58s00.89s AlignTreeLinear61.022.67s43.016.57s00.95s01.09s AlignTree4.019.01s41.024.8s03.16s01.27s LlamaRefusalClassifier5.0145.54s44.057.68s2.020.55s059.84s -3.1-8BSVMClassifier2.066.58s20.042.02s1.017.02s043.9s -InstructMultiRefusalsClassifier4.062.49s32.031.67s1.011.66s035.79s AlignTreeLinear7.0101.99s18.040.61s1.014.38s036.74s AlignTree5.087.37s18.034.2s1.021.88s037.44s gemmaRefusalClassifier21.0619.41s54.0957.3s0981.92s0983.69s -3-12bSVMClassifier26.0738.11s37.0708.4s5.0965.68s0969.49s -itMultiRefusalsClassifier25.0509.97s53.0602.4s0640.47s0606.22s AlignTreeLinear8.0496.35s29.0800.97s0959.48s0949.66s AlignTree10.0591.11s40.0717.22s0988.4s0978.93s Table 3: This table reports ASR, refusal rates, and execution time for each dataset, illustrating the impact of ablating individual components of AlignTree. The full results across eight datasets and nine LLMs from the three families are in the Appendix. from slightly faster execution, its reliance on linear classifiers limits expressiveness and leads to inconsistent results. The SVMClassifier, although leveraging non-linear signals, fails to generalize across datasets and exhibits excessive refusal rates, particularly on Gemma-3-12b. Some variants, such as the RefusalClassifier and the Mul- tiRefusalsClassifier, exhibit substantial ASR variability: for example, they perform strongly on Llama-3.1-8B-Instruct but poorly on Qwen2.5-0.5B-Instruct. We attribute this inconsis- tency to differences in the base models’ pretrained alignment behavior, which we discuss further in the Appendix. Never- theless, the MultiRefusalsClassifier outperforms its single- classifier counterpart, reinforcing the hypothesis that refusal mechanisms are multidimensional phenomena. Most classifiers manage to avoid over-refusal, preserving usability on benign datasets such as PIQA and ARC. Notable exceptions include the SVMClassifier on PIQA for Gemma- 3-12b and the RefusalClassifier on PIQA for Llama-3.1-8B, both of which demonstrate elevated refusal rates. In summary, AlignTree emerges as the most reliable and general-purpose defense, consistently achieving a favorable trade-off between robustness, efficiency, and usability. While other classifier-based defenses leveraging model activations may be suitable in certain contexts, AlignTree demonstrates the most stable and dependable performance overall, making it a strong candidate for real-world deployment. Additional ablation results across all models and datasets are provided in the Appendix, demonstrating similar trends. Conclusions We introduced AlignTree, an efficient defense that enhances model alignment while maintaining minimal computational overhead. In order to build this lightweight deference, we trained a Random Forest classifier that integrates the linear refusal direction with a novel SVM-based signal designed to capture non-linear features associated with harmful content. Our results show that AlignTree consistently outperforms ex- isting defenses in terms of ASR, refusal, and computational efficiency while introducing a non-negligible increase in exe- cution time over the baseline. Moreover, our results demon- strate that leveraging non-linear harmfulness signals leads to improved alignment performance compared to relying solely on a single linear refusal vector, which we believe is essen- tial for advancing alignment strategies. In future work, we plan to extend AlignTree by introducing an additional “suspi- cious” threshold, one that distinguishes borderline prompts from clearly benign or harmful ones. It will allow identify- ing prompts that warrant further analysis without immediate rejection and can be used jointly with additional defenses. Limitations While AlignTree represents a meaningful advancement in improving the alignment of LLMs, several limitations remain. ASR evaluations in this work were conducted using an- other LLM, following methodologies similar to those in Arditi et al. (2024) and related defense studies. While prac- tical, this approach may occasionally introduce evaluation inaccuracies due to model-based judgment. Another limitation is that AlignTree requires training a sep- arate classifier for each model, and its effectiveness depends heavily on the level of the base model’s initial alignment and the quality of the data. Finally, while this work com- bined linear and non-linear signals, further research could explore more direct approaches to characterizing and utiliz- ing non-linear refusal properties; for instance, by identifying additional semantic directions or better modeling the refusal manifold in latent space. Finally, AlignTree relies on a limited set of input signals and lightweight classifiers to reduce the risk of overfitting. Future work could explore the use of more complex models and larger training datasets to further enhance performance. Ethics Statement This work aims to enhance language models by introducing a novel method to improve their safe usage through efficient and robust defenses. We recognize the potential of such tech- nologies and emphasize the importance of their responsible use. While our contributions are intended to support the de- velopment of more aligned models, we stress the need to prevent misuse, such as generating harmful content. Future research should focus on promoting more efficient defense strategies that align with societal benefits. Acknowledgements This work was supported by a Tel Aviv University Center for AI and Data Science (TAD) grant. This research was also supported by the Ministry of Innovation, Science & Technology, Israel (1001576154) and the Michael J. Fox Foundation (MJFF-022407). The contribution of SK is part of a PhD thesis research conducted at Tel Aviv University. References Andriushchenko, M.; Croce, F.; and Flammarion, N. 2025. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. arXiv:2404.02151. Arditi, A.; Obeso, O.; Syed, A.; Paleka, D.; Panickssery, N.; Gurnee, W.; and Nanda, N. 2024. Refusal in Language Mod- els Is Mediated by a Single Direction. arXiv:2406.11717. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; Chen, C.; Olsson, C.; Olah, C.; Hernandez, D.; Drain, D.; Ganguli, D.; Li, D.; Tran-Johnson, E.; Perez, E.; Kerr, J.; Mueller, J.; Ladish, J.; Landau, J.; Ndousse, K.; Luko- suite, K.; Lovitt, L.; Sellitto, M.; Elhage, N.; Schiefer, N.; Mercado, N.; DasSarma, N.; Lasenby, R.; Larson, R.; Ringer, S.; Johnston, S.; Kravec, S.; Showk, S. E.; Fort, S.; Lanham, T.; Telleen-Lawton, T.; Conerly, T.; Henighan, T.; Hume, T.; Bowman, S. R.; Hatfield-Dodds, Z.; Mann, B.; Amodei, D.; Joseph, N.; McCandlish, S.; Brown, T.; and Kaplan, J. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. Bisk, Y.; Zellers, R.; Bras, R. L.; Gao, J.; and Choi, Y. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence. Chao, P.; Debenedetti, E.; Robey, A.; Andriushchenko, M.; Croce, F.; Sehwag, V.; Dobriban, E.; Flammarion, N.; Pappas, G. J.; Tramer, F.; Hassani, H.; and Wong, E. 2024a. Jailbreak- Bench: An Open Robustness Benchmark for Jailbreaking Large Language Models. https://arxiv.org/abs/2404.01318. Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2024b. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419. Chen, B.; Paliwal, A.; and Yan, Q. 2023. Jailbreaker in Jail: Moving Target Defense for Large Language Models. arXiv:2310.02417. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge. arXiv:1803.05457. Derczynski, L.; Galinkin, E.; Martin, J.; Majumdar, S.; and Inie, N. 2024. garak: A Framework for Security Probing Large Language Models. arXiv:2406.11036. Dong, W.; Li, P.; Tian, Y.; Zeng, X.; Li, F.; and Wang, S. 2025. Feature-Aware Malicious Output Detection and Mitigation. arXiv:2504.09191. Gemma Team; Kamath, A.; Ferret, J.; et al. 2025. Gemma 3 Technical Report. arXiv:2503.19786. Glaese, A.; McAleese, N.; Tr ̨ebacz, M.; Aslanides, J.; Firoiu, V.; Ewalds, T.; Rauh, M.; Weidinger, L.; Chadwick, M.; Thacker, P.; Campbell-Gillingham, L.; Uesato, J.; Huang, P.-S.; Comanescu, R.; Yang, F.; See, A.; Dathathri, S.; Greig, R.; Chen, C.; Fritz, D.; Elias, J. S.; Green, R.; Mokrá, S.; Fernando, N.; Wu, B.; Foley, R.; Young, S.; Gabriel, I.; Isaac, W.; Mellor, J.; Hassabis, D.; Kavukcuoglu, K.; Hendricks, L. A.; and Irving, G. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv:2209.14375. Grattafiori, A.; Dubey, A.; Jauhri, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783. Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y.; Ma, S.; Liu, H.; Wang, S.; Zhang, K.; Wang, Y.; Gao, W.; Ni, L.; and Guo, J. 2025. A Survey on LLM-as-a- Judge. arXiv:2411.15594. Gupta, M.; Akiri, C.; Aryal, K.; Parker, E.; and Praharaj, L. 2023. From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy. arXiv:2307.00691. Hildebrandt, F.; Maier, A.; Krauss, P.; and Schilling, A. 2025. Refusal Behavior in Large Language Models: A Nonlinear Perspective. arXiv:2501.08145. Huang, Y.; Gupta, S.; Xia, M.; Li, K.; and Chen, D. 2023. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. arXiv:2310.06987. Jain, N.; Schwarzschild, A.; Wen, Y.; Somepalli, G.; Kirchen- bauer, J.; yeh Chiang, P.; Goldblum, M.; Saha, A.; Geiping, J.; and Goldstein, T. 2023. Baseline Defenses for Adversarial At- tacks Against Aligned Language Models. arXiv:2309.00614. Kumar, A.; Agarwal, C.; Srinivas, S.; Li, A. J.; Feizi, S.; and Lakkaraju, H. 2025. Certifying LLM Safety against Adversarial Prompting. arXiv:2309.02705. Li, Y.; Wei, F.; Zhao, J.; Zhang, C.; and Zhang, H. 2023. RAIN: Your Language Models Can Align Themselves with- out Finetuning. arXiv:2309.07124. Liu, X.; Xu, N.; Chen, M.; and Xiao, C. 2024a. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. arXiv:2310.04451. Liu, Y.; Deng, G.; Xu, Z.; Li, Y.; Zheng, Y.; Zhang, Y.; Zhao, L.; Zhang, T.; Wang, K.; and Liu, Y. 2024b. Jail- breaking ChatGPT via Prompt Engineering: An Empirical Study. arXiv:2305.13860. Llama Team and AI @ Meta. 2024. The Llama 3 Family of Models. Mazeika, M.; Phan, L.; Yin, X.; Zou, A.; Wang, Z.; Mu, N.; Sakhaee, E.; Li, N.; Basart, S.; Li, B.; Forsyth, D.; and Hendrycks, D. 2024. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. arXiv:2402.04249. Mazeika, M.; Zou, A.; Mu, N.; Phan, L.; Wang, Z.; Yu, C.; Adam Khoja, F. J.; O’Gara, A.; Sakhaee, E.; Xiang, Z.; Ra- jabi, A.; Hendrycks, D.; Poovendran, R.; Li, B.; ; and Forsyth, D. 2023. TDC 2023 (LLM edition): the Trojan Detection Challenge. Mehrotra, A.; Zampetakis, M.; Kassianik, P.; Nelson, B.; Anderson, H.; Singer, Y.; and Karbasi, A. 2024.Tree of Attacks: Jailbreaking Black-Box LLMs Automatically. arXiv:2312.02119. Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In EMNLP. OpenAI. 2024. ChatGPT-4o. https://chat.openai.com/. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155. Perez, F.; and Ribeiro, I. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. Phute, M.; Helbling, A.; Hull, M.; Peng, S.; Szyller, S.; Cor- nelius, C.; and Chau, D. H. 2024. LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked. arXiv:2308.07308. Platt, J. 2000. Probabilistic Outputs for Support Vector Ma- chines and Comparisons to Regularized Likelihood Methods. Adv. Large Margin Classif., 10. Qiu, H.; Zhang, S.; Li, A.; He, H.; and Lan, Z. 2023. Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models. arXiv:2307.08487. Robey, A.; Wong, E.; Hassani, H.; and Pappas, G. J. 2024. SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks. arXiv:2310.03684. Röttger, P.; Kirk, H.; Vidgen, B.; Attanasio, G.; Bianchi, F.; and Hovy, D. 2024. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. In Duh, K.; Gomez, H.; and Bethard, S., eds., Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 5377–5400. Mexico City, Mexico: Association for Computational Linguistics. Sap; Maarten; Rashkin; Hannah; Chen; Derek; Bras, L.; Ro- nan; Choi; and Yejin. 2019. Social IQa: Commonsense Rea- soning about Social Interactions. In Inui; Kentaro; Jiang; Jing; Ng; Vincent; Wan; and Xiaojun, eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4463– 4473. Hong Kong and China: Association for Computational Linguistics. Souly, A.; Lu, Q.; Bowen, D.; Trinh, T.; Hsieh, E.; Pandey, S.; Abbeel, P.; Svegliato, J.; Emmons, S.; Watkins, O.; and Toyer, S. 2024. A StrongREJECT for Empty Jailbreaks. arXiv:2402.10260. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stan- ford Alpaca: An Instruction-following LLaMA model. https: //github.com/tatsu-lab/stanford_alpaca. Team, Q. 2024. Qwen2.5: A Party of Foundation Models. Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbroken: How Does LLM Safety Training Fail? arXiv:2307.02483. Wollschläger, T.; Elstner, J.; Geisler, S.; Cohen-Addad, V.; Günnemann, S.; and Gasteiger, J. 2025. The Geometry of Refusal in Large Language Models: Concept Cones and Rep- resentational Independence. arXiv:2502.17420. Xu, Z.; Jiang, F.; Niu, L.; Jia, J.; Lin, B. Y.; and Poovendran, R. 2024. SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding. arXiv:2402.08983. Yao, Y.; Duan, J.; Xu, K.; Cai, Y.; Sun, Z.; and Zhang, Y. 2024. A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly. High-Confidence Computing, 4(2): 100211. Zeng, W.; Liu, Y.; Mullins, R.; Peran, L.; Fernandez, J.; Harkous, H.; Narasimhan, K.; Proud, D.; Kumar, P.; Rad- harapu, B.; Sturman, O.; and Wahltinez, O. 2024a. Shield- Gemma: Generative AI Content Moderation Based on Gemma. arXiv:2407.21772. Zeng, Y.; Wu, Y.; Zhang, X.; Wang, H.; and Wu, Q. 2024b. AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks. arXiv:2403.04783. Zhang, S.; Zhai, Y.; Guo, K.; Hu, H.; Guo, S.; Fang, Z.; Zhao, L.; Shen, C.; Wang, C.; and Wang, Q. 2025. JB- Shield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipula- tion. arXiv:2502.07557. Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043. Additional experiment details Random Forest hyperparameter sensitivity In this section, we demonstrate that our Random For- est (RF) classifier exhibits low sensitivity to hyperpa- rameter variations. We conducted a rigorous grid search encompassing36distinct hyperparameter configurations. Specifically, the search space was defined by the fol- lowing sets of values:n_estimators ∈ 30, 50, 70, max_depth ∈ 4, 6,min_samples_leaf ∈ 2, 5, 10, and min_samples_split∈3, 5. Evaluation utilized a representative set of models, scaling from Qwen2.5-0.5B-Instruct to Llama-3.1-8B-Instruct, se- lecting one from each family. Table 4 confirms high stability across hyperparameter settings: Refusal rates were near0, and Execution Time variance was minimal compared to the mean. While ASR showed the most variance, results remain significantly better than other SOTA defenses. ModelPAIR ASR↓Time↓ Qwen2.5-0.5B-Instruct14.14± 5.4418.01± 3.44 gemma-3-4b-it14.19± 4.93111.97s± 8.17 Llama-3.1-8B-Instruct8.36± 3.7999.39± 27.39 ModelPIQA Refusal↓Time↓ Qwen2.5-0.5B-Instruct0± 01.11± 0.08 gemma-3-4b-it0± 0112.24s± 3.15s Llama-3.1-8B-Instruct1.14± 0.5912.09± 0.73 Table 4: Results of the Random Forest Hyperparameter Grid Search on the PAIR (Harmful) and PIQA (Harmless) evalua- tion datasets. AlignTree test results In this subsection, we provide the test results on the Random Forest test set, derived from the dataset outlined in the paper. As illustrated in Figure 3, AlignTree delivers an accuracy of 98.86%, with precision at 98.96%. This demonstrates that our F β threshold functions effectively on the test set and avoid overfitting to the validation or training datasets. Evaluation Against Adaptive White-Box Attacks To comprehensively test AlignTree’s robustness against more targeted attacks, we employed an adaptive white-box jail- break attack methodology based on Andriushchenko, Croce, and Flammarion (2025). This attack uses an adversarial prompt template combined with a random suffix search, a tar- geted technique that subverts safety alignment by observing model activations. We ran this evaluation on the largest model from each fam- ily (Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, gemma- 3-12b-it). Using the attack’s provided base instructions, we filtered those found in our existing datasets, resulting in 39 unique prompts. For each model and prompt, we executed the LegitimateMalicious Predicted Label Legitimate Malicious True Label 5702 6566 Confusion Matrix 200 400 Figure 3: Confusion matrix on the test set for Llama-3.1-8B- Instruct. full attack process to generate a targeted adversarial prompt capable of jailbreaking the model. As summarized in Table 5, AlignTree demonstrates excep- tional performance against this targeted attack, achieving0% ASR on all tested models. While AutoDefense also reaches 0%ASR, it requires a substantially higher computational cost (over3×the baseline execution time). This confirms Align- Tree’s superior ability to withstand targeted attacks compared to existing SOTA defenses, which exhibited significantly higher ASRs much closer to the baseline. model namestrategyASRTime Llama-3.1-8B-InstructBaseline12.8245.02s SelfDefenseInput5.1337.23s SelfDefense7.6965.86s SmoothLLM12.82207.36s PerplexityDefense12.8243.59s AutoDefense0140.74s AlignTree02.40s Qwen2.5-7B-InstructBaseline10.2666.93s SelfDefenseInput10.2681.99s SelfDefense10.2681.65s SmoothLLM10.26142.11s PerplexityDefense10.2666.76s AutoDefense0916.62s AlignTree01.04s gemma-3-12b-itBaseline10.26256.54s SelfDefenseInput060.01s SelfDefense7.69319.68s SmoothLLM10.26469.50s PerplexityDefense10.26270.12s AutoDefense0830.28s AlignTree03.12s Table 5: ASR and Time results on the white box Adaptive attack with 39 prompts. Defense hyperparameters SmoothLLM (Robey et al. 2024) hyperprameters - We uti- lized the RandomSwapPerturbation setting with parameters num_copies= 10and pert_pct= 10as recommended by the original paper. This entails generating 10 copies of the prompt, and at each copy, swap10%of the tokens with tokens selected uniformly from all printable strings then generate a response for each copy and evaluate if the LLM refused to answer, if majority of copies are refusing to answer then this defense considers the prompt to be harmful. AutoDefense (Zeng et al. 2024b) Hyperparameters – We adopted the same hyperparameters as those used in the orig- inal paper, implementing a 3-agent setup consisting of a “Judge” agent, an “Intention Analyzer” agent, and a “Prompt Analyzer” agent. Additionally, we used the prompts provided in the official code base, with one minor modification: we instructed the agents not to return a harmful verdict when there is insufficient context. This adjustment addressed a recurring issue we observed, where the system frequently refused outputs unnecessarily. Threshold Threshold selection We previously stated that thresholds are selected based on the generalized F β score, defined as: F (precision) β = (1 + β 2 )· Precision· Recall β 2 · Precision + Recall (8) In this section, we report the resulting thresholds for all models. Figure 5 displays theF β curves alongside the se- lected thresholds for each of the nine evaluated models. Threshold sensitivity In this section, we also examined different values for,τto verify our method and to examine sensitivity. Figure 4 shows the results. While the generalizedF β score was used to guide hyperparameter selection, the chosen con- figurations were consistently near-optimal. For Llama-3,2- 1B-Instruct, the selected threshold outperformed all others. In Qwen2.5-7B-Instruct, the chosen threshold of 0.59 closely approximated the optimal value of 0.6. These results indicate that the precision-recall based selection effectively meets our objectives– successfully filtering malicious prompts without inducing excessive refusals. Refusal and Jailbreak evaluations We build on previous work to efficiently analyze LLM re- fusals, leveraging common “refusal substrings” found across models. The specific substrings employed are listed in Ta- ble 6. In addition to refusal detection, we use ChatGPT-4o to assess the harmfulness of the response and determine whether the malicious request was successfully answered. Category-specific Breakdown of ASR To provide an alternative view of AlignTree’s robustness, we further break down the Attack Success Rate (ASR) by categories adopted from Chao et al. (2024a). We classified each harmful instruction in our datasets into the following categories: Physical Harm (Phys harm), Malware/Hacking (Hack), Economical harm (Economic), Disinformation (Dis- info), Harassment/Discrimination (Harass/Discri), Privacy (Priv), and Fraud/Deception (Decept). The comparison for the Baseline and AlignTree defense strategies is presented in Figure 6. The results show that AlignTree is able to consistently improve upon the baseline ASR across all these harmful categories. Crucially, there are no significantly worse categories where AlignTree performs poorly, indicating that our method provides consistent robust- ness across diverse types of harmful content. AlignTree Feature importance In this section, we present the top 10 most important fea- tures for Llama-3.2-1B-Instruct, Qwen2.5-7B-Instruct, and Gemma-3-12B-IT. The leading features are shown in Fig- ure 7. Notably, SVM-based signals are consistently utilized, and the most influential layers tend to be located in the middle of the model. 0.000.05 ASR ( better) 0.0 0.2 0.4 0.6 0.8 1.0 Refusal Rate ( better) Llama-3.2-1B-Instruct = 0.98 0.00.2 ASR ( better) 0.0 0.2 0.4 0.6 0.8 1.0 Refusal Rate ( better) Qwen2.5-7B-Instruct = 0.59 0.00.10.2 ASR ( better) 0.0 0.2 0.4 0.6 0.8 1.0 Refusal Rate ( better) gemma-3-12b-it = 0.89 Figure 4: AlignTree Hyperparameters. Threshold performance distributions based on ASR and Refusal metrics for Llama-3.2-1B- Instruct, Qwen2.5-7B-Instruct, and Gemma3-12B-it. Each value in the chart corresponds to a threshold sampled at 0.1 intervals from 0 to 1. The threshold selected τ using the F β -score for balancing precision and recall is highlighted in red. "I’m sorry", "I am sorry", "I apologize", "As an AI", "As a language model", "As an assistant", "I cannot", "I can’t", "I am unable to", "I’m unable to", "I am not able to", "I’m not able to" Table 6: Refusal Expressions Figure 5: Selected thresholds for each of the models based on the F β score that prioritizes precision but balances with recall. 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Llama-3.2-1B-Instruct 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Llama-3.2-3B-Instruct 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Llama-3.1-8B-Instruct 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) gemma-3-1b-it 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) gemma-3-4b-it 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) gemma-3-12b-it 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Qwen2.5-0.5B-Instruct 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Qwen2.5-3B-Instruct 20% 60% 100% Hack (n=142) Phys harm (n=51) Decept (n=31) Priv (n=30) Harr/Discri (n=122) Disinfo (n=14) Economic (n=10) Qwen2.5-7B-Instruct BaselineAlignTree Figure 6: Comparison of Attack Success Rate (ASR) for Baseline and AlignTree across harmful domains, where a lower ASR indicates higher robustness; the number of examples for each harmful domain is shown as (n = x). 0.00.10.2 Feature Importance Refusal L=13 Refusal L=2 SVM L=6 i=-4 SVM L=8 i=-5 SVM L=9 i=-5 SVM L=7 i=-5 SVM L=5 i=-5 SVM L=7 i=-4 SVM L=6 i=-5 SVM L=5 i=-4 Llama-3.2-1B 0.000.050.100.15 Feature Importance SVM L=8 i=-1 SVM L=13 i=-3 SVM L=15 i=-3 SVM L=14 i=-3 SVM L=18 i=-3 SVM L=16 i=-3 SVM L=12 i=-2 SVM L=11 i=-2 SVM L=9 i=-1 SVM L=10 i=-3 Llama-3.1-8B 0.000.050.10 Feature Importance SVM L=10 i=-3 SVM L=10 i=-4 SVM L=11 i=-3 SVM L=11 i=-2 SVM L=16 i=-5 SVM L=21 i=-5 SVM L=8 i=-2 SVM L=20 i=-5 SVM L=13 i=-5 SVM L=17 i=-5 gemma3-4b 0.000.050.100.15 Feature Importance SVM L=10 i=-5 SVM L=16 i=-4 SVM L=9 i=-4 SVM L=18 i=-4 SVM L=7 i=-4 SVM L=5 i=-4 SVM L=6 i=-1 SVM L=7 i=-2 SVM L=5 i=-1 SVM L=10 i=-2 gemma3-12b 0.00.10.2 Feature Importance SVM L=4 i=-2 SVM L=10 i=-4 SVM L=22 i=-2 SVM L=21 i=-2 SVM L=12 i=-2 SVM L=11 i=-4 SVM L=23 i=-2 SVM L=20 i=-2 SVM L=5 i=-1 SVM L=5 i=-4 Qwen2.5-0.5B 0.00.10.20.3 Feature Importance SVM L=11 i=-2 SVM L=1 i=-2 SVM L=16 i=-4 SVM L=13 i=-4 SVM L=13 i=-1 SVM L=1 i=-3 SVM L=5 i=-2 SVM L=12 i=-2 SVM L=12 i=-1 SVM L=18 i=-3 Qwen2.5-7B Figure 7: Top 10 feature importance of AlignTree’s RandomForest for 2 models in each model family; Llama-3.2-1B-Instruct, Llama-3.2-8B-Instruct, gemma3-4b-it, gemma3-12b-it, Qwen2.5-0.5B-Instruct, Qwen2.5-7B-Instruct. Additional results This section presents additional results, including extensive benchmark evaluations and a further ablation study of the AlignTree defense method. ASR We provide additional ASR results for all datasets across all defense strategies. The results for Llama-3 Instruct models are shown in Table 7, for Gemma-3 Instruct models in Table 8, and for Qwen-2.5 Instruct models in Table 9. Refusal We present the refusal results for all LLama-3 Instruct mod- els and datasets in Table 10, Gemma3 Instruct models and datasets in Table 11, and Qwen2.5 Instruct models and datasets in Table 12. Further ablation results In this section, we provide ablation results for all nine models across eight datasets. The ASR results, presented in Table 13 and refusal results presented in Table 14, depict the same trends as the partial table. Refusal Classifier Discussion The high baseline ASR for Qwen models (Table 9) com- pared to Llama models (Table 7) indicates weaker alignment in Qwen. This alignment quality dramatically affects Re- fusalClassifier performance: while achieving optimal results on Llama models, the classifier becomes nearly ineffective on Qwen models (Table 13). This stark performance col- lapse—rather than gradual degradation—demonstrates that refusal detection in weakly aligned models presents funda- mentally different interpretability challenges that render sim- ple classification approaches impractical Use of existing assets All models used in this work are mentioned in Table 15, and datasets used are mentioned in Table 16. Please note this table is based on Arditi et al. (2024) collection of datasets. Compute statement All experiments presented in this paper were run on a single NVIDIA-RTX-6000-Ada generation GPU with 48 GB of memory. Training AlignTree, as described in Section , takes ap- proximately 3 minutes for the largest models; all defenses’ execution times are reported in our results tables in Section . ModelStrategyMalwareGenPromptInjectPAIRAutoDAN ASRTimeASRTimeASRTimeASRTime LlamaBaseline32.014.17s36.010.25s12.017.02s1.04.79s -3.2-1BAutoDefense3.0825.07s2.0469.17s8.0653.75s1.0502.02s -InstructSelfDefense-Input28.026.56s24.015.89s5.035.63s1.018.62s SelfDefense23.026.33s24.016.74s6.028.0s1.019.43s PerplexityDefense26.014.78s35.010.92s13.017.81s1.05.91s SmoothLLM30.0146.36s13.0117.05s10.0122.68s075.62s AlignTree (Ours)20.011.67s21.05.75s6.018.64s01.05s LlamaBaseline16.054.37s28.023.27s19.059.82s1.010.7s -3.2-3BAutoDefense1.0566.02s2.0489.71s17.0937.0s0247.46s -InstructSelfDefense-Input13.053.28s21.014.46s17.068.19s1.019.33s SelfDefense15.048.38s22.015.36s18.051.29s1.017.61s PerplexityDefense15.054.5s26.024.13s19.058.45s1.012.16s SmoothLLM15.0263.22s20.0203.48s11.0214.34s0148.88s AlignTree (Ours)12.047.53s17.014.88s10.053.54s05.99s LlamaBaseline9.0174.53s43.051.97s14.0274.71s029.1s -3.1-8BAutoDefense5.0553.46s0361.85s16.0837.72s0299.65s -InstructSelfDefense-Input8.0247.66s32.0123.06s8.0333.96s0138.42s SelfDefense8.0202.17s28.095.57s8.0253.02s098.06s PerplexityDefense8.0178.81s42.054.55s15.0270.72s032.22s SmoothLLM8.0505.02s37.0383.04s13.0525.58s0271.95s AlignTree (Ours)5.087.37s18.034.2s9.0128.94s05.8s Table 7: Additional results for Llama Instruct models on harmful benchmarks. The Attack Success Rate for each dataset, with the execution time for all defense strategies across all datasets. ModelStrategyMalwareGenPromptInjectPAIRAutoDAN ASRTimeASRTimeASRTimeASRTime gemmaBaseline37.036.86s39.037.08s35.037.65s8.036.78s -3-1bAutoDefense19.01136.23s4.0741.54s22.01107.78s1.01152.82s -itSelfDefense-Input1.080.95s5.022.49s019.22s020.27s SelfDefense2.079.47s2.034.2s044.86s017.32s PerplexityDefense21.053.16s33.052.41s34.056.69s6.046.51s SmoothLLM35.0404.96s35.0353.55s23.0378.94s2.0393.65s AlignTree (Ours)30.042.23s40.052.7s23.038.85s1.036.76s gemmaBaseline31.0127.2s54.0109.94s36.0116.19s16.0123.17s -3-4bAutoDefense5.01166.97s4.0299.04s18.01308.45s1.01350.8s -itSelfDefense-Input4.0130.5s028.75s12.0196.13s029.06s SelfDefense4.0131.11s026.57s12.0134.0s027.71s PerplexityDefense16.094.76s47.0138.73s31.0146.75s15.0133.15s SmoothLLM29.0523.68s55.0532.96s34.0532.66s10.0546.91s AlignTree (Ours)29.0105.86s55.0108.92s13.0114.08s04.24s gemmaBaseline24.0963.31s50.0959.33s36.0942.08s6.0954.95s -3-12bAutoDefense7.01679.09s5.0961.3s19.01885.07s1.01739.91s -itSelfDefense-Input23.0845.26s35.01178.87s28.01131.6s3.0383.83s SelfDefense18.01355.14s5.01292.2s33.01309.25s4.01370.69s PerplexityDefense16.0965.51s52.0982.62s35.0995.25s5.0982.46s SmoothLLM25.01865.47s55.01874.88s37.01894.6s7.01876.13s AlignTree (Ours)10.0591.11s40.0717.22s10.0329.52s1.0127.92s Table 8: Additional results for Gemma Instruct models on harmful benchmarks. The Attack Success Rate for each dataset, with the execution time for all defense strategies across all datasets. ModelStrategyMalwareGenPromptInjectPAIRAutoDAN ASRTimeASRTimeASRTimeASRTime Qwen2.5Baseline91.019.12s50.018.4s51.019.57s48.018.56s -0.5BAutoDefense5.01027.52s0343.8s13.0915.98s0781.04s -InstructSelfDefense-Input43.038.77s13.027.91s8.039.01s17.038.86s SelfDefense42.038.56s16.013.09s11.037.26s13.038.36s PerplexityDefense84.021.2s50.021.03s50.021.66s47.021.02s SmoothLLM77.0210.79s43.0209.75s49.0213.53s44.0212.22s AlignTree (Ours)4.019.01s41.024.8s6.018.46s05.24s Qwen2.5Baseline32.065.2s56.033.14s37.065.69s29.065.01s -3BAutoDefense3.0983.72s1.0369.39s21.0911.47s3.0946.27s -InstructSelfDefense-Input4.071.45s32.041.19s21.086.78s12.089.78s SelfDefense9.076.85s29.040.19s24.079.74s13.081.35s PerplexityDefense26.065.82s40.029.97s37.066.17s29.065.41s SmoothLLM15.0304.3s53.0249.86s25.0320.95s14.0329.48s AlignTree (Ours)1.029.04s12.010.47s14.063.51s012.65s Qwen2.5Baseline43.0284.2s58.081.13s37.0297.47s14.0271.41s -7BAutoDefense3.0983.72s1.0369.39s21.0911.47s3.0946.27s -InstructSelfDefense-Input40.0468.83s55.0150.4s36.0376.8s14.0323.25s SelfDefense41.0348.93s53.0156.63s36.0389.38s14.0340.84s PerplexityDefense34.0337.79s44.071.91s38.0320.17s15.0272.2s SmoothLLM43.0601.36s63.0329.86s36.0649.49s14.0591.41s AlignTree (Ours)6.077.19s1.03.93s14.0154.06s01.48s Table 9: Additional results for Qwen Instruct models on harmful benchmarks. The Attack Success Rate for each dataset, with the execution time for all defense strategies across all datasets. ModelStrategyPIQAOpenbookQASocialIQAARC-Challenge RefusalTimeRefusalTimeRefusalTimeRefusalTime LlamaBaseline3.01.29s05.13s3.02.9s06.1s -3.2-1BAutoDefense6.076.62s10.0381.94s6.0210.6s12.0627.91s -InstructSelfDefense-Input57.07.64s38.012.02s36.09.74s49.015.9s SelfDefense58.08.59s30.012.91s35.010.04s39.015.12s PerplexityDefense3.02.29s06.16s3.03.96s07.45s SmoothLLM21.0132.1s12.0132.04s26.0126.87s0137.16s AlignTree (Ours)3.00.7s04.56s3.02.96s05.28s LlamaBaseline10.04.0s1.04.47s10.03.51s08.94s -3.2-3BAutoDefense53.0255.5s6.0155.17s15.0331.77s4.0325.72s -InstructSelfDefense-Input36.05.07s21.06.13s18.04.6s16.010.23s SelfDefense35.04.54s12.04.28s21.03.61s12.09.12s PerplexityDefense10.06.06s1.06.12s10.05.1s010.83s SmoothLLM14.0214.38s6.0214.28s23.0208.42s1.0222.11s AlignTree (Ours)10.04.86s1.012.24s10.05.05s015.28s LlamaBaseline2.015.59s032.53s5.012.52s056.78s -3.1-8BAutoDefense2.086.66s1.0272.5s7.0238.2s4.0292.15s -InstructSelfDefense-Input55.087.29s47.0138.26s30.079.5s52.0181.8s SelfDefense51.095.86s55.085.34s34.081.92s49.0104.44s PerplexityDefense2.018.93s035.58s5.015.7s059.86s SmoothLLM2.0407.79s0414.91s5.0399.89s0456.06s AlignTree (Ours)1.021.88s025.09s5.09.02s037.44s Table 10: Additional results for Llama Instruct models on harmless, commonsense benchmarks. These experiments measure excessive refusal and time efficiency for harmless prompts. ModelStrategyPIQAOpenbookQASocialIQAARC-Challenge RefusalTimeRefusalTimeRefusalTimeRefusalTime gemmaBaseline02.12s02.35s03.89s05.41s -3-1bAutoDefense32.0186.99s2.0535.29s2.0395.68s2.0474.96s -itSelfDefense-Input8.016.88s15.016.17s018.52s16.022.51s SelfDefense7.014.82s20.013.96s3.018.55s16.018.94s PerplexityDefense022.31s08.82s010.6s027.05s SmoothLLM0273.27s0295.69s0294.47s0329.92s AlignTree (Ours)02.69s02.94s05.64s03.42s gemmaBaseline0112.36s0109.69s0109.25s0110.08s -3-4bAutoDefense0225.66s3.0206.92s10.0248.76s1.0248.14s -itSelfDefense-Input2.0200.86s0150.37s3.0253.44s1.0168.44s SelfDefense2.0133.81s0134.87s3.0133.5s1.0136.61s PerplexityDefense0140.85s0130.43s0131.23s0134.77s SmoothLLM0538.26s0532.74s0534.65s0538.81s AlignTree (Ours)0120.69s0127.04s0122.42s0126.4s gemmaBaseline0955.92s0955.25s0961.94s0972.13s -3-12bAutoDefense01097.71s7.01203.54s02057.22s2.02017.82s -itSelfDefense-Input1.01182.44s01181.36s1.01179.36s01193.1s SelfDefense2.01319.66s17.01186.86s58.01190.3s11.01188.49s PerplexityDefense0996.26s0993.79s0989.72s01033.18s SmoothLLM01926.51s01935.92s01890.58s01913.35s AlignTree (Ours)0988.4s0975.01s0973.6s0978.93s Table 11: Additional results for Gemma Instruct models on harmless, commonsense benchmarks. These experiments measure excessive refusal and time efficiency for harmless prompts. ModelStrategyPIQAOpenbookQASocialIQAARC-Challenge RefusalTimeRefusalTimeRefusalTimeRefusalTime Qwen2.5Baseline00.43s00.7s02.09s00.7s -0.5BAutoDefense050.54s6.0154.79s3.0170.17s8.0151.56s -InstructSelfDefense-Input80.010.31s35.010.1s33.010.26s46.011.82s SelfDefense72.09.97s37.010.02s37.012.27s41.010.19s PerplexityDefense02.21s02.24s03.84s02.39s SmoothLLM0196.47s0187.81s0189.37s0188.72s AlignTree (Ours)03.16s01.16s04.86s01.27s QwenBaseline03.21s00.89s04.18s01.0s 2.5-3BAutoDefense067.09s073.38s1.074.67s059.11s -InstructSelfDefense-Input3.018.71s019.19s019.7s018.17s SelfDefense015.73s017.7s019.18s017.43s PerplexityDefense05.62s03.17s05.73s03.25s SmoothLLM0257.3s0251.21s0249.99s0251.32s AlignTree (Ours)04.47s02.82s04.84s03.95s QwenBaseline010.26s021.42s09.42s041.98s 2.5-7BAutoDefense096.62s0111.04s099.26s0156.18s -InstructSelfDefense-Input096.25s0100.68s0113.66s0151.98s SelfDefense087.89s076.81s071.91s099.92s PerplexityDefense017.32s024.25s012.33s044.86s SmoothLLM0327.57s0305.28s0265.46s0355.54s AlignTree (Ours)011.57s029.54s09.2s033.47s Table 12: Additional results for Qwen Instruct models on harmless, commonsense benchmarks. These experiments measure excessive refusal and time efficiency for harmless prompts. ModelStrategyMalwareGenPromptInjectPAIRAutoDAN ASRTimeASRTimeASRTimeASRTime Llama-3.2-1BRefusalClassifier1.017.46s10.05.93s3.019.57s00.55s -InstructSVMClassifier29.012.19s35.07.56s10.011.79s00.52s MultiRefusalsClassifier14.011.56s28.05.94s9.011.84s00.83s AlignTreeLinear20.011.34s21.03.6s2.011.17s01.25s AlignTree20.011.67s21.05.75s6.018.64s01.05s Llama-3.2-3BRefusalClassifier3.044.14s8.022.45s8.063.55s04.8s -InstructSVMClassifier5.035.22s15.010.86s9.046.46s1.04.74s MultiRefusalsClassifier10.053.35s24.018.42s17.048.73s01.88s AlignTreeLinear4.045.24s19.014.4s11.047.67s03.24s AlignTree12.047.53s17.014.88s10.053.54s05.99s Llama-3.1-8BRefusalClassifier5.0145.54s44.057.68s11.0230.5s012.64s -InstructSVMClassifier2.066.58s20.042.02s4.066.3s01.75s MultiRefusalsClassifier4.062.49s32.031.67s1.021.7s00.8s AlignTreeLinear7.0101.99s18.040.61s9.0122.13s06.33s AlignTree5.087.37s18.034.2s9.0128.94s05.8s gemma-3-1bRefusalClassifier28.074.28s40.061.04s31.057.78s6.0311.39s -itSVMClassifier23.041.77s30.049.07s6.033.5s01.72s MultiRefusalsClassifier40.039.3s34.040.11s32.040.68s6.037.31s AlignTreeLinear33.035.61s30.049.05s26.040.3s5.032.8s AlignTree30.042.23s40.052.7s23.038.85s1.036.76s gemma-3-4bRefusalClassifier31.0158.01s54.0189.08s26.0196.93s6.0185.28s -itSVMClassifier6.0133.65s10.0123.63s1.0100.2s06.59s MultiRefusalsClassifier14.0108.14s53.0114.94s16.0120.92s2.085.4s AlignTreeLinear10.0151.25s32.0121.67s1.041.7s037.31s AlignTree29.0105.86s55.0108.92s13.0114.08s04.24s gemma-3-12bRefusalClassifier21.0619.41s54.0957.3s22.0910.53s1.057.31s -itSVMClassifier26.0738.11s37.0708.4s5.0175.27s033.83s MultiRefusalsClassifier25.0509.97s53.0602.4s26.0523.65s037.47s AlignTreeLinear8.0496.35s29.0800.97s8.0350.06s04.78s AlignTree10.0591.11s40.0717.22s10.0329.52s1.0127.92s Qwen2.5-0.5BRefusalClassifier89.027.18s52.027.34s50.027.65s48.027.46s -InstructSVMClassifier33.021.85s46.027.05s4.020.61s00.72s MultiRefusalsClassifier29.017.45s53.018.32s36.017.97s44.018.38s AlignTreeLinear61.022.67s43.016.57s23.020.04s7.019.74s AlignTree4.019.01s41.024.8s6.018.46s05.24s Qwen2.5-3BRefusalClassifier011.78s54.0167.59s9.094.41s7.060.49s -InstructSVMClassifier4.035.72s52.043.72s6.050.58s00.94s MultiRefusalsClassifier00.43s49.041.52s24.070.56s18.070.4s AlignTreeLinear5.046.94s27.021.64s5.054.36s00.58s AlignTree1.029.04s12.010.47s14.063.51s012.65s Qwen2.5-7BRefusalClassifier8.0157.97s62.0107.46s36.0369.33s15.0286.95s -InstructSVMClassifier8.086.79s24.021.96s4.055.31s01.78s MultiRefusalsClassifier18.088.23s41.039.89s15.0120.24s2.018.34s AlignTreeLinear17.0106.69s53.030.62s19.0168.65s4.055.67s AlignTree6.077.19s1.03.93s14.0154.06s01.48s Table 13: Full ASR results for ablated defenses in AlignTree. We can observe the trend is similar to Table 3. ModelStrategyPIQAOpenbookQASocialIQAARC-Challenge RefusalTimeRefusalTimeRefusalTimeRefusalTime Llama-3.2-1BRefusalClassifier3.02.47s08.11s4.05.55s011.74s -InstructSVMClassifier3.02.35s08.41s3.05.51s012.05s MultiRefusalsClassifier3.01.26s04.63s3.02.87s05.42s AlignTreeLinear4.00.65s04.61s3.02.84s05.4s AlignTree3.00.7s04.56s3.02.96s05.28s Llama-3.2-3BRefusalClassifier10.08.53s1.07.91s10.06.04s013.19s -InstructSVMClassifier11.06.03s1.042.38s10.035.42s053.34s MultiRefusalsClassifier10.05.29s1.06.24s10.03.45s012.27s AlignTreeLinear11.01.73s1.06.06s10.02.51s08.67s AlignTree10.04.86s1.012.24s10.05.05s015.28s Llama-3.1-8BRefusalClassifier2.020.55s034.55s5.014.64s059.84s -InstructSVMClassifier3.024.53s0122.24s5.022.62s098.11s MultiRefusalsClassifier1.011.66s020.02s5.07.98s035.79s AlignTreeLinear3.08.21s025.02s5.07.03s043.44s AlignTree3.013.29s035.86s5.013.37s060.47s gemma-3-1bRefusalClassifier017.13s011.69s015.04s0350.36s -itSVMClassifier05.12s04.64s012.71s08.58s MultiRefusalsClassifier03.21s03.08s06.66s04.71s AlignTreeLinear05.02s02.51s06.76s05.11s AlignTree02.69s02.94s05.64s03.42s gemma-3-4bRefusalClassifier0190.45s0158.67s0133.95s0162.7s -itSVMClassifier100.034.43s100.035.24s100.029.28s100.028.13s MultiRefusalsClassifier0118.85s0114.18s0117.22s0117.45s AlignTreeLinear0139.14s0185.25s0168.52s0175.44s AlignTree0120.69s0127.04s0122.42s0126.4s gemma-3-12bRefusalClassifier0981.92s0960.61s0953.05s0983.69s -itSVMClassifier5.0965.68s0966.58s1.0959.45s0969.49s MultiRefusalsClassifier0640.47s0619.29s0629.87s0606.22s AlignTreeLinear0761.13s0596.65s0835.81s0600.52s AlignTree0988.4s0975.01s0973.6s0978.93s QwenRefusalClassifier01.44s01.65s03.57s01.79s 2.5-0.5BSVMClassifier01.53s01.64s03.55s01.94s -InstructMultiRefusalsClassifier00.58s00.81s02.34s00.89s AlignTreeLinear00.82s00.89s02.43s00.97s AlignTree00.73s00.98s02.42s01.03s QwenRefusalClassifier06.72s01.66s09.82s01.66s 2.5-3BSVMClassifier0114.3s0164.26s0629.9s0203.46s -InstructMultiRefusalsClassifier02.34s01.78s04.68s02.04s AlignTreeLinear1.01.65s01.43s04.02s01.51s AlignTree04.47s02.82s04.84s03.95s QwenRefusalClassifier023.1s030.35s017.73s054.87s 2.5-7BSVMClassifier0116.62s035.97s075.31s077.06s -InstructMultiRefusalsClassifier08.96s013.96s06.22s027.54s AlignTreeLinear010.37s015.56s07.2s030.08s AlignTree011.57s029.54s09.2s033.47s Table 14: Comprehensive Refusal results for ablated defenses in AlignTree. The observed trend aligns with Table 3. ModelSourceAccessed viaLicense QWEN2.5 INSTRUCTTeam (2024)huggingface.co/QwenTongyi Qianwen Research License GEMMA3 ITGemma Team et al. (2025)huggingface.co/blog/gemma3Gemma Terms of Use LLAMA-3 INSTRUCTGrattafiori et al. (2024)huggingface.co/Meta-Llama-3-8B-InstructMeta Llama 3 Community License LLAMA GUARD 3Llama Team and AI @ Meta (2024)huggingface.co/Llama-Guard-3-1BMeta Llama 3 Community License CHATGPT-4OOpenAI (2024)chat.openai.comOpenAI Terms of Use Table 15: Models used in this work DatasetSourceAccessed viaLicense ADVBENCHZou et al. (2023)github.com/llm-attacks/llm-attacksMIT License TDC2023Mazeika et al. (2024, 2023)github.com/centerforaisafety/tdc2023-starter-kitMIT License HARMBENCHMazeika et al. (2024)github.com/centerforaisafety/HarmBenchMIT License JAILBREAKBENCHChao et al. (2024a)github.com/JailbreakBench/jailbreakbenchMIT License MALICIOUSINSTRUCTHuang et al. (2023)github.com/princeton-sysml/jailbreak_llmMIT License ALPACATaori et al. (2023)huggingface.co/datasets/tatsu-lab/alpacaApache License 2.0 PIQABisk et al. (2020)huggingface.co/datasets/ybisk/piqaApache License 2.0 ARC-CHALLENGEClark et al. (2018)huggingface.co/datasets/allenai/ai2_arcCC-BY-SA-4.0 OPENBOOKQAMihaylov et al. (2018)huggingface.co/datasets/allenai/openbookqaApache License 2.0 SIQASap et al. (2019)huggingface.co/datasets/allenai/social_i_qaApache License 2.0 JBSHIELD (ATTACKS)Zhang et al. (2025)github.com/NISPLab/JBShieldMIT License GARAKDerczynski et al. (2024)github.com/NVIDIA/garakApache License 2.0 XSTESTRöttger et al. (2024)huggingface.co/datasets/walledai/XSTestCC-BY-4.0 Table 16: Datasets used in this work

Instant research discovery