Paper deep dive
Tripwire: Triggering Aligned Refusal via Statistically Certified Safety Neurons
Wei Zhao, Zhe Li, Peixin Zhang, Jun Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/23/2026, 3:06:30 AM
Summary
The paper introduces TripWire, a training-free defense mechanism for Large Language Models (LLMs) against jailbreak attacks. TripWire identifies safety-specific neurons using a statistically rigorous funnel involving direction filtering, Welch t-tests with Benjamini-Hochberg false-discovery-rate control, and a utility-specificity filter. It then applies a 'trigger-style clamp' to hold these neurons at their harmful-conditional mean activations, effectively injecting an internal signal that triggers the model's aligned refusal behavior. The method supports two deployment modes: an inference-time intervention gated by a detector and an offline bias-patch weight edit. Experiments on four LLMs against four attack types show TripWire reduces attack success rates to at most 2.0% with minimal utility degradation (0.5% to 5.3% on MT-Bench).
Entities (13)
Relation Signals (10)
TripWire → defendsagainst → Jailbreak Attack
confidence 95% · TripWire reduces the average attack success rate to at most 2.0%
LLaMA-2-7B → isinstanceof → LLM
confidence 95% · four safety-aligned open-source LLMs (Llama-2-7B...)
TripWire → uses → Safety Neuron
confidence 92% · TripWire... identifies safety-specific neurons... a trigger-style clamp holds the selected neurons
GCG → istypeof → Jailbreak Attack
confidence 90% · optimization-based attacks such as GCG
TripWire → outperforms → TraceRouter
confidence 90% · compared to... 2.2%–4.9% for... TraceRouter
TripWire → outperforms → DELMAN
confidence 90% · compared to... 2.2%–4.9% for... DELMAN
TripWire → outperforms → LED
confidence 90% · compared to... 2.2%–4.9% for... LED
TripWire → outperforms → RepE
confidence 90% · TripWire... achieves an average ASR below 2%... compared to 8.3%–12.6% for RepE
TripWire → implementsvia → Bias-Patch
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neuron- and path-level interventions offer the finest-grained route to defending large language models (LLMs) against jailbreak attacks, yet existing methods fall short of this promise, i.e., they often compromise model utility significantly. Specifically, one line of work suppresses toxic neurons to erase harmful semantics, but since such semantics are distributed across the network, blocking every pathway forces a large intervention footprint. An alternative line of research focus on identify safety neurons using external classifiers. While promising, the existing approaches suffer from compromising neurons that are important for the model utility as well. Moreover, both approaches remain always on and thus perturb every benign request even when no attack is present. To address these limitations, we present \ours{}, a training-free defense that first identifies safety-specific neurons through per-neuron hypothesis tests under false-discovery-rate control together with a utility-specificity filter. Based on this identification, a trigger-style clamp holds the selected neurons at their harmful-conditional mean activations, injecting an internal harmful-input signal that triggers the refusal behavior learned during alignment. The clamp is then realized by two provably equivalent deployment modes, namely a detector-gated inference-time intervention and an offline bias-patch weight edit. Extensive experiments across four safety-aligned LLMs and four representative attacks demonstrate that \ours{} reduces the average attack success rate to at most 2.0\% while incurring a utility drop of only 0.5\% to 5.3\% on MT-Bench, the smallest among all defenses. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.14392v1
- Canonical: https://arxiv.org/abs/2608.14392v1
Trouble viewing inline? Open PDF directly →
Full Text
46,173 characters extracted from source content.
Expand or collapse full text
Tripwire: Triggering Aligned Refusal via Statistically Certified Safety Neurons Wei Zhao Zhe Li Peixin Zhang Jun Sun Abstract Neuron- and path-level interventions offer the finest-grained route to defending large language models (LLMs) against jailbreak attacks, yet existing methods fall short of this promise, i.e., they often compromise model utility significantly. Specifically, one line of work suppresses toxic neurons to erase harmful semantics, but since such semantics are distributed across the network, blocking every pathway forces a large intervention footprint. An alternative line of research focus on identify safety neurons using external classifiers. While promising, the existing approaches suffer from compromising neurons that are important for the model utility as well. Moreover, both approaches remain always on and thus perturb every benign request even when no attack is present. To address these limitations, we present TripWire, a training-free defense that first identifies safety-specific neurons through per-neuron hypothesis tests under false-discovery-rate control together with a utility-specificity filter. Based on this identification, a trigger-style clamp holds the selected neurons at their harmful-conditional mean activations, injecting an internal harmful-input signal that triggers the refusal behavior learned during alignment. The clamp is then realized by two provably equivalent deployment modes, namely a detector-gated inference-time intervention and an offline bias-patch weight edit. Extensive experiments across four safety-aligned LLMs and four representative attacks demonstrate that TripWire reduces the average attack success rate to at most 2.0% while incurring a utility drop of only 0.5% to 5.3% on MT-Bench, the smallest among all defenses. Code is available at https://anonymous.4open.science/r/Tripwire-65C4. Introduction Large Language Models (LLMs) such as ChatGPT (1) and Gemini (3) have revolutionized natural language processing, achieving remarkable performance in tasks such as question answering, code completion, and text generation (5; 7). However, as these models are increasingly deployed in real-world applications, their vulnerabilities pose significant security and reliability risks. A particularly severe threat is jailbreaking, i.e., the use of adversarial prompts to bypass safety mechanisms and elicit harmful or policy-violating outputs (11; 10). Such attacks have become increasingly automated and effective: optimization-based attacks such as GCG (34), iterative refinement attacks such as PAIR (6), and generator-based attacks such as AmpleGCG (13) and Jailbreak-R1 (11) have all demonstrated increasing efficiency and effectiveness in bypassing safety mechanisms. In response, the research community has developed a variety of defense strategies ranging from knowledge-editing approaches such as LED (29) and DELMAN (23) to inference-level defenses that examine or perturb inputs and internal states (19; 27). Despite these advances, existing defenses exhibit a persistent safety and utility trade-off: strengthening refusal behavior degrades performance on benign tasks. An ideal defense mechanism would operate at the finest possible granularity: by first identifying the safety-related neurons and then intervening on only those neurons, such a defense should achieve strong protection with the least utility cost (25; 24; 20). However, existing neuron-level methods fall short of this promise in practice for three reasons. First, existing methods such as TraceRouter (20) and DELMAN (23) essentially target toxic neurons: they rely on harmful vocabulary to locate the modules that encode harmful information and then zero out or reverse the neuron values to block the propagation of harmful semantics. However, harmful semantics are distributed across the network and blocking them requires suppressing every route they may take. This demands a large intervention footprint and turns every additional suppressed neuron into a potential utility loss. Moreover, evolving attacks can always exploit an uncovered route, which makes it practically impossible to invalidate all pathways. Second, methods such as NeuroStrike (25) instead search for safety neurons that detect harmful requests and trigger refusal. These methods rely on external classifiers to attribute neuron importance. However, such classifier-based attribution cannot distinguish safety neurons from generally important ones. Third, some neurons inevitably serve both safety and general functions. An always-on intervention without taking utility into consideration on these neurons therefore perturbs every benign request even when no attack is present. These three problems are the causes of the observed utility cost rather than the neuron granularity itself. To address these challenges, we present TripWire, a neuron-level defense framework that addresses each problem with a dedicated design. First, a statistically rigorous identification funnel screens every neuron in the model with direction filtering, per-neuron Welch t-tests under Benjamini–Hochberg false-discovery-rate control, and a utility-specificity filter that removes neurons which also activate strongly on normal tasks. The surviving neurons are then ranked and the top-N candidates are selected. This yields a statistical upper bound on the fraction of mis-selected neurons and disentangles safety-specific neurons from generally important ones without any training. Second, a trigger-style clamp holds the selected neurons at their harmful-conditional mean activations, which injects an internal “the current input is harmful” signal and triggers the refusal behavior the model already learned during alignment. Because triggering refusal requires only a stable signal rather than interception of every semantic pathway, a small neuron set is sufficient and the intervention does not depend on the specific attack. Third, two deployment modes realize the same clamp: an inference mode in which a lightweight detector gates the clamp so benign inputs are processed by the unmodified model, and an offline bias-patch weight edit shown to be equivalent to the inference-time clamp, both theoretically and empirically. With this design, a single identification pipeline serves both a detector-gated deployment and a permanent, detector-free, zero-overhead checkpoint. To validate the effectiveness of TripWire, we conduct comprehensive experiments on four safety-aligned open-source LLMs (Llama-2-7B, Llama-3.1-8B, Qwen2.5-7B, and Qwen2.5-32B) against four representative jailbreak attacks (GCG, AmpleGCG, AutoDAN, and Jailbreak-R1). We compare against state-of-the-art defense baselines organized by deployment form: RepE (31) and TraceRouter (20) for the inference-based defense and LED (29) and DELMAN (23) for the weight-editing defense. We report the attack success rate (ASR) together with MT-Bench (30) utility scores judged by GPT-5.4 at neuron budgets of top-1000 and top-2500. Results demonstrate that TripWire (when targeting top-2500 selected neurons) achieves an average ASR below 2% across both deployment forms on Llama-2, Llama-3.1, and Qwen2.5, compared to 8.3%–12.6% for RepE and 2.2%–4.9% for TraceRouter, DELMAN, and LED, while incurring the smallest utility drop among all defenses. The inference-based and edit-based forms yield consistent results, confirming that the identified neuron set rather than the specific intervention mechanism carries the defense. Preliminary Large Language Models and Neurons An LLM is a decoder-only Transformer that autoregressively maps a token sequence to a distribution over the next token. Each input token is first embedded into a vector x∈ℝdx ^d and then processed through L stacked layers, each of which updates the representation as x(ℓ)=x(ℓ−1)+Attn(ℓ)(x(ℓ−1))+MLP(ℓ)(x(ℓ−1)),x^( )=x^( -1)+Attn^( )\! (x^( -1) )+MLP^( )\! (x^( -1) ), (1) where the attention sublayer Attn(ℓ)Attn^( ) exchanges information across token positions and the MLP sublayer MLP(ℓ)MLP^( ) transforms each position independently. Since the MLP sublayers hold the majority of the parameters and are widely regarded as the primary store of knowledge, we center our analysis on the MLP sublayers. Modern LLMs such as Llama and Qwen implement the MLP sublayer in a gated form: MLP(ℓ)(x) ^( )(x) =Wdown(ℓ)(σ(Wgate(ℓ)x)⊙(Wup(ℓ)x)), =W_down^( ) (σ (W_gate^( )x ) (W_up^( )x ) ), (2) Wgate(ℓ),Wup(ℓ)∈ℝdff×d,Wdown(ℓ)∈ℝd×dff, W_gate^( ),W_up^( ) ^d_f× d, W_down^( ) ^d× d_f, where dffd_f is the intermediate dimension of the MLP sublayer, σ is a nonlinearity, and ⊙ denotes the element-wise product. In this work, we define a neuron as a triple (ℓ,ψ,i)( ,ψ,i) with ψ∈gate,upψ∈\gate,up\ and i∈[dff]i∈[d_f], whose activation is the i-th component of the corresponding projection: ai(ℓ,ψ)(x)=(Wψ(ℓ)x)i=(wi(ℓ,ψ))⊤x,a^( ,ψ)_i(x)= (W_ψ^( )x )_i= (w^( ,ψ)_i ) x, (3) so that each neuron corresponds to one row of Wψ(ℓ)W_ψ^( ), which underlies the intervention and editing operations described in the following sections. We represent each prompt p on neuron (ℓ,ψ,i)( ,ψ,i) by its last-token activation ai(ℓ,ψ)(p)≜ai(ℓ,ψ)(xtp)a^( ,ψ)_i(p) a^( ,ψ)_i(x_t_p), where tpt_p is the last-token position. We choose the last-token position because the causal attention of the decoder-only architecture aggregates the semantic information of the entire prompt. We collect such activations over three prompt sets: a harmful set hD_h from HarmBench (34), a benign set sD_s from Alpaca (21), and a utility set uD_u from MMLU (12), each containing 200200 prompts. Figure 1: Overview of TripWire. A statistically rigorous funnel identifies harm-specific safety neurons, a trigger-style clamp injects the harmfulness signal to trigger aligned refusal, and two deployment modes realize the defense with near-zero utility cost. Identifying Safety Neurons Previous studies have shown that the safety behavior of an aligned LLM is governed by a remarkably small fraction of neurons: pruning less than 0.6%0.6\% of the neurons in a layer suffices to break safety alignment (25; 28). We conjecture that these safety neurons serve two complementary roles: a detection role, in which a neuron activates selectively on harmful input and flags the current request as unsafe, and a refusal role, in which the sustained activation of these neurons triggers the downstream computation that produces an explicit refusal. This conjecture is consistent with the observation above: once these neurons are set to zero, both roles are lost, and the model no longer refuses harmful requests. To identify safety neurons, existing work requires training an external classifier on paired harmful and benign prompts. The representative approach fits a per-layer logistic-regression probe that predicts the safety label from the layer’s activations (25), y^()=σ(∑iwiai(ℓ)()+b), y(x)=σ\! ( _iw_i\,a_i^( )(x)+b ), (4) and selects the neurons whose weight magnitude |wi||w_i| passes a z-score threshold. This identification, however, suffers from two drawbacks. First, the probe weights are not identifiable in the p≫np n regime. A high-weight neuron may therefore be merely correlated with the safety judgment instead of being specific to it, and the fraction of falsely selected neurons is left uncontrolled. Second, the screening does not separate safety-specific neurons from generally important ones, so intervening on a mislabeled neuron costs utility for no defensive gain. These drawbacks directly motivate the identification funnel of TripWire. Method In this section, we describe the three components of TripWire as illustrated in Figure 1. Statistically Rigorous Identification Funnel We treat every neuron in the model as a candidate safety neuron and screen them jointly across all layers. For each neuron independently, we compute class-discrimination statistics over its activation samples ajh,ajs,aju\a_j^h\,\a_j^s\,\a_j^u\ on h,s,uD_h,D_s,D_u, with means μh,μs,μu _h, _s, _u and standard deviations sh,s_h,s_s. We select a neuron as a safety neuron only if it passes three successive filters: (i) direction, (i) significance, and (i) specificity. The neurons that pass all three filters are then ranked, and the top ones are selected under a specified budget N. (i) Direction filtering. We retain only neurons positively correlated with harmful content: μh>μs _h> _s. (i) Significance filtering (Welch t-test + BH-FDR). For each neuron we perform a heteroscedastic two-sample Welch t-test, t=μh−μssh2/nh+s2/ns,t= _h- _s s_h^2/n_h+s_s^2/n_s, (5) and correct the resulting p-values across all neurons of all layers with the Benjamini–Hochberg procedure: sorting p-values in ascending order p(1)≤⋯≤p(m)p_(1)≤·s≤ p_(m) and rejecting the first k∗=maxk:p(k)≤kmαk = \k:p_(k)≤ kmα\ hypotheses at α=0.05α=0.05. Under the standard assumptions of the BH procedure, this controls the expected fraction of irrelevant neurons admitted into the set selected at level α. This is the first layer of utility protection, since every mis-selected neuron contributes nothing to defense and only damages normal capabilities. (i) Utility-specificity filtering. To suppress over-refusal (i.e., wrongly rejecting safe request (8)) and protect general capabilities, we measure where a neuron’s utility-task activation lies on the benign-to-harmful axis: ρ=μu−μsμh−μs.ρ= _u- _s _h- _s. (6) ρ≈0ρ≈ 0 indicates that the neuron behaves on utility tasks as it does on benign inputs, whereas ρ≈1ρ≈ 1 indicates that it also activates strongly on normal utility tasks, i.e., a generally important neuron rather than a safety-specific one. We exclude neurons with ρ>τρ> _ρ (default τρ=0.5 _ρ=0.5). Ranking and selection budget. We rank the remaining candidates by AUROC, computed via ranks on the pooled sample ajh∪ajs\a_j^h\∪\a_j^s\ (average ranks for ties), which equals the normalized Mann–Whitney U statistic: AUROC =1nhns(∑j=1nhrank(ajh)−nh(nh+1)2) = 1n_hn_s ( _j=1^n_hrank (a_j^h )- n_h(n_h+1)2 ) (7) =Pr(ah>as), = (a^h>a^s ), i.e., the probability that a randomly drawn harmful activation exceeds a randomly drawn benign one. This univariate effect size provides an independent, interpretable, and scale-invariant measure of discriminative strength for each neuron. Because the harmful and benign distributions differ statistically on almost every neuron, an absolute AUROC threshold cannot cleanly delimit the distribution tail. We therefore impose a fixed selection budget and take the global top-N by AUROC over the candidate set C that passes (i)–(i): =argtop-N(ℓ,ψ,i)∈AUROC(ℓ,ψ,i),S= *arg\,top-N_( ,ψ,i)\,∈\,CAUROC^( ,ψ,i), (8) yielding per-layer index sets ℓψ⊆[dff]S_ ^ψ [d_f]. Comparison with probe-based selection. Probe-based attribution (25) fits a per-layer logistic classifier and selects neurons by weight z-scores. These weights are not identifiable in the p≫np n regime. A high-weight neuron may therefore be merely correlated with harmfulness instead of being specific to it, and no false-discovery guarantee holds. Our funnel replaces this unidentifiable joint attribution with per-neuron univariate statistics under explicit FDR and specificity control. In the following section, we compare the safety and utility performance of the neurons selected by our funnel against those selected by this probe-based method. Trigger-Style Clamp Intervention Given the selected set S, we compute for each safety neuron its harmful-conditional mean activation over the harmful set hD_h: a¯ih,(ℓ,ψ)=1|h|∑p∈hai(ℓ,ψ)(p). a^h,( ,ψ)_i= 1|D_h| _p _ha^( ,ψ)_i(p). (9) The intervention clamps each selected neuron to this constant at all token positions t throughout generation: a~i(ℓ,ψ)(xt)=a¯ih,(ℓ,ψ),i∈ℓψ,ai(ℓ,ψ)(xt),otherwise. a^( ,ψ)_i(x_t)= cases a^h,( ,ψ)_i,&i _ ^ψ,\\[2.0pt] a^( ,ψ)_i(x_t),&otherwise. cases (10) Erasure-style interventions (20; 25) attempt to block the propagation of harmful semantics. Since these semantics are distributed across the network, blocking them requires suppressing every route they may take. Newly emerging attacks may target different activation paths that are hard to cover in advance. Our clamp does the opposite: it holds the “harmfulness detector” neurons at their harmful-conditional means throughout generation. The clamp overwrites the activations directly and thus guarantees that no adversarial prompt can suppress this signal. In this way, the model continuously receives an internal signal that the current input is a harmful request, which triggers the refusal behavior the model already acquired during alignment. Because triggering refusal requires only a stable signal rather than the interception of every pathway, a small and rigorously selected neuron set is sufficient and the intervention does not depend on the specific attack. Two Deployment Modes Inference Mode Although our funnel separates safety-specific neurons from generally important ones as much as possible, some selected neurons may still carry utility knowledge and an always-on clamp would inevitably perturb benign requests. To avoid this cost, we build a sparse feature from the selected neuron set and train a classifier on the identification prompt sets to decide whether to intervene. The inference mode therefore applies the clamp only to requests judged malicious, so that benign inputs are processed by the unmodified model. Sparse feature construction. For an input prompt p, we read only the selected safety neurons’ last-token activations and concatenate them in a fixed layer order: ϕ(p)=⨁ℓ∈ℒ,ψ[ai(ℓ,ψ)(p)]i∈ℓψ∈ℝD,D≪2L⋅d.φ(p)= _ ,\,ψ [a^( ,ψ)_i(p) ]_i _ ^ψ ^D, D 2L· d. (11) Detector training. To classify malicious requests, we employ logistic regression as a linear classifier over the sparse features: P(adv∣p)=σ(θ⊤ϕ(p)+θ0),P(adv p)=σ (θ φ(p)+ _0 ), (12) where σ is the sigmoid function. The classifier is trained on the same benign set sD_s and harmful set hD_h used for identification. Detect-then-intervene. For each request p, a single forward pass extracts ϕ(p)φ(p). If the trained detector judges the request as benign, the model generates normally; if it judges the request as harmful, the clamp of Eq. (10) is applied via forward hooks throughout generation. Detection and intervention share the same neuron set and analysis data, so the two stages are naturally self-consistent. Figure 2: ASR (%, ↓ ) under four jailbreak attacks for all defenses across four models. Top row: inference-time defenses; bottom row: weight-editing defenses. Offline Mode Alternatively, if the goal is to deliver a hardened checkpoint that is always-on and needs no detector at inference time, we write the equivalent defense directly into the weights. The always-on clamp of Eq. (10) admits an exact weight-space realization that we call the bias-patch. For each selected neuron (ℓ,ψ,i)∈( ,ψ,i) , with weight row wiw_i and bias bib_i (a b=0b=0 bias parameter is injected when the architecture has none), we zero the weight row and inject the harmful-conditional mean as the bias: wi←,bi←a¯ih⟹a~i(x)≡a¯ih.w_i 0, b_i← a^h_i a_i(x)≡ a^h_i. (13) Proposition 1 (Equivalence of bias-patch and always-on inference-time clamp). For every input x and token position t, the bias-patched model computes exactly the activations of Eq. (10): zeroing wiw_i makes ai(x)=wi⊤x+bi≡bi=a¯iha_i(x)=w_i x+b_i≡ b_i= a^h_i for i∈ℓψi _ ^ψ, while all other neurons are untouched. Hence the bias-patched checkpoint and the inference-time clamped model are mathematically identical maps, differing only in that the former is permanent and detector-free. The bias-patch is thus the exact bridge between the two deployment routes, so any defense property established for the inference-time clamp remains equivalent on the edited checkpoint. The choice between the two modes is then purely operational: the inference mode preserves benign behavior exactly at the cost of a per-request gating decision, whereas the bias-patch mode requires no detector but applies the clamp unconditionally. Because the clamped set is small and harm-specific by construction, the always-on cost of the bias-patch remains a low trade-off on utility benchmarks, as quantified in experiment section. Experiment Experimental Setup Models. We conduct experiments on four safety-aligned open-source LLMs: Llama-2-7B (22), Llama-3.1-8B (9), Qwen2.5-7B, and Qwen2.5-32B (4). All experiments are conducted on H100-80GB servers. Attacks. We evaluate against four representative jailbreak attacks covering diverse attack paradigms: GCG (34) (white-box gradient optimization), AmpleGCG (13) (generator-based), AutoDAN (6) (black-box iterative refinement), and Jailbreak-R1 (11) (reinforcement-learning driven). For each target model, we follow each attack’s original implementation and generate adversarial prompts from the 520520 harmful prompts of AdvBench (33) within the setup budget. Detailed attack configurations are provided in the Appendix. Defense Baselines. We compare TripWire against state-of-the-art defense methods in both deployment forms. For inference-time defenses, we compare against RepE (32) and TraceRouter (20), both of which intervene on internal computation at inference time. For weight-editing defenses, we compare against LED (29) and DELMAN (23), both of which permanently modify model parameters. We adopt the default parameters from each method. To the best of our knowledge, TripWire is the only method that covers both deployment forms with a single identification pipeline. This enables a controlled comparison where the neuron set is held fixed and only the deployment form varies. We report TripWire at two neuron budgets, top-1000 and top-2500. For Qwen2.5-32B, we scale the budgets to top-2000 and top-5000 to match its larger neuron count, i.e., 2500/1,060,864=0.23%2500/1,060,864=0.23\% for Qwen2.5-7B and 5000/3,538,944=0.14%5000/3,538,944=0.14\% for Qwen2.5-32B. Metrics. Following standard practice (34; 32), we quantify defense effectiveness through Attack Success Rate (ASR), where a prompt is considered successfully jailbroken if the model generates harmful content that violates safety guidelines; we employ GPT-5.4 as the safety evaluator over (prompt, response) pairs. We quantify utility with MT-Bench (30) scores, also judged by GPT-5.4. In addition, we report the MT-Bench over-refusal rate by matching refusal keywords such as “cannot” over benign responses. Prompt templates for the judge are provided in the Appendix. Table 1: Defense–utility trade-off: average ASR (%, ↓ ) over the four attacks, MT-Bench score (↑ ), MT-Bench over-refusal rate (%, ↓ ), and inference-time overhead (↓ ) relative to the undefended model. Bold marks the best result among defenses within each deployment form. Llama-2-7B Llama-3.1-8B Qwen2.5-7B Qwen2.5-32B Form Defense ASR MT-B Rej. ASR MT-B Rej. ASR MT-B Rej. ASR MT-B Rej. Overhead — No defense 43.0% 4.94 0.0% 33.1% 6.60 0.0% 24.1% 7.40 0.0% 17.2% 8.23 0.0% 1.00×1.00× Inference RepE 12.6% 3.76 15.0% 8.3% 3.63 65.0% 11.8% 4.52 31.3% 2.6% 6.18 33.8% 1.27×1.27× TraceRouter 2.2% 4.19 2.5% 1.2% 5.59 5.6% 1.9% 5.92 2.5% 4.9% 7.35 7.5% 1.16×1.16× TripWire (Small) 6.5% 4.76 1.3% 1.3% 6.56 2.5% 5.6% 7.40 0.0% 5.9% 8.20 2.5% 1.13×1.13× TripWire (Large) 1.1% 4.68 2.5% 0.5% 6.48 3.1% 2.0% 7.36 3.8% 0.8% 8.14 3.8% 1.15×1.15× Edit DELMAN 4.3% 4.53 8.8% 3.3% 5.78 15.0% 2.8% 6.16 5.0% 5.0% 7.45 13.1% 1.00×1.00× LED 4.4% 4.28 8.8% 3.8% 5.26 22.5% 4.9% 5.88 11.3% 5.6% 7.68 18.8% 1.00×1.00× TripWire (Small) 6.3% 4.73 2.0% 1.1% 5.10 8.8% 5.3% 6.96 8.0% 5.6% 7.88 8.8% 1.00×1.00× TripWire (Large) 0.8% 4.31 15.0% 0.3% 4.84 31.3% 1.7% 5.86 12.5% 0.3% 7.45 17.5% 1.00×1.00× Defense Performance Analysis Defense results are summarized in Figure 2 and Table 1. Figure 2 reports ASR under the four attacks for all defenses across the four models, and Table 1 summarizes the defense–utility trade-off through average ASR, MT-Bench score, over-refusal rate, and inference-time overhead. We measure the time overhead over MT-Bench generation, as refusal responses are inherently shorter and offer an unfair comparison. As shown in Figure 2, for the undefended models, the ASR of GCG and AmpleGCG varies substantially across models and drops sharply on Qwen2.5-32B, as larger and more recently aligned models present a harder optimization landscape for gradient-based suffix search. In contrast, AutoDAN and Jailbreak-R1 maintain similar ASR across all four models. Among the defenses, all methods except RepE reduce ASR significantly; RepE yields the weakest defense, as its steering direction extracted from mean representation differences introduces excessive noise. TraceRouter and TripWire at the top-2500 budget suppress ASR to nearly zero across all four models (TripWire in both the inference-based and edit-based forms). Across different attack methods, most defenses effectively defend against GCG and AmpleGCG. However, the human-readable prompts from AutoDAN and Jailbreak-R1 are harder to defend. Jailbreak-R1 in particular evades many defenses due to its highly diverse attack strategies. As shown in Table 1, in the inference-based form, TripWire at the top-2500 budget reduces the average ASR from 43.0% to 1.1% on Llama-2, from 33.1% to 0.5% on Llama-3.1, and from 24.1% to 2.0% on Qwen2.5, with a utility drop of only 5.3%, 4.7%, and 0.5% respectively. TraceRouter reaches a comparable ASR but degrades utility by up to 20.0%. We attribute this cost to its erasure-style intervention, which suppresses neurons along harmful pathways and inevitably perturbs benign computation that shares those pathways. RepE remains at 8.3%–12.6% average ASR and degrades utility by up to 45.0%. Notably, the inference-based top-1000 configuration of TripWire keeps the utility drop within 3.6% and over-refuses at most 2.5% of benign queries, as the detector leaves the benign path untouched; the ASR at this budget is slightly higher (1.3%–6.5%) due to the smaller neuron set. In the edit-based form, TripWire at top-2500 attains the best average ASR on every model (0.8%, 0.3%, and 1.7%), whereas DELMAN and LED remain at 2.8%–4.9%. At top-1000, TripWire achieves the highest MT-Bench score and the lowest over-refusal rate on Llama-2 and Qwen2.5, though the ASR is moderately higher at 6.3% and 5.3% due to the smaller neuron budget. Regarding the time overhead, TripWire incurs at most 1.15×1.15× generation time and remains cheaper than TraceRouter (1.16×1.16×) and RepE (1.27×1.27×). Overall, TripWire at top-2500 achieves an average ASR below 2% across both deployment forms on Llama-2, Llama-3.1, and Qwen2.5, compared to 8.3%–12.6% for RepE and 2.2%–4.9% for TraceRouter, DELMAN, and LED, while incurring the smallest utility drop among all defenses. Ablation Study To isolate the contribution of the identification funnel, we compare it against probe-based selection (25) under always-on inference-time clamping, so that the detector is removed and only the selected neuron set differs. We evaluate on Llama-2, Llama-3.1, and Qwen2.5, measuring safety with average ASR and utility with MT-Bench score and over-refusal rate. Table 2 summarizes the ablation results. We first note that the inference without detector and edit forms of TripWire yield nearly identical ASR and MT-Bench scores at both budgets, which empirically confirms the mathematical equivalence between the bias-patch and the inference-time clamp established in the preliminary section. The table shows that probe-based selection requires 33–5×5× more neurons than our funnel to reach a comparable ASR. Nevertheless, it degrades utility by up to 52.3% and triggers far more over-refusal. This is expected, as the probe cannot separate safety-specific neurons from generally important ones. The additional neurons it selects carry utility knowledge and clamping them degrades benign performance. In contrast, our funnel reaches a similar or lower ASR with far fewer neurons and a much milder utility cost. This confirms that the FDR and specificity control in the funnel translates directly into a better safety–utility trade-off. Although the identification step for TripWire takes 79–92 seconds per model, this is a one-time offline analysis and does not affect inference latency. Table 2: Identification ablation under always-on inference-time clamping (no detector gate). Probe-based selection is denoted by its z-score threshold z. Variant Neurons Extr. ASR MT-Bench Rej. Llama2-7B 0 — 43.0 4.94 0.0 Probe (z=3.0z=3.0) 3,589 31s 3.3 4.23 8.8 Probe (z=2.5z=2.5) 7,568 32s 2.9 3.82 18.8 TripWire (top-1000) 1,001 79s 6.3 4.70 2.0 TripWire (top-2500) 2,503 80s 0.8 4.31 15.0 Llama3.1-8B 0 — 33.1 6.60 0.0 Probe (z=3.0z=3.0) 10,811 43s 0.1 4.08 50.0 Probe (z=2.5z=2.5) 17,234 44s 0.1 3.15 22.5 TripWire (top-1000) 1,000 88s 1.1 5.22 8.8 TripWire (top-2500) 2,500 88s 0.3 4.85 31.3 Qwen2.5-7B 0 — 24.1 7.40 0.0 Probe (z=3.0z=3.0) 9,664 38s 2.6 5.50 12.5 Probe (z=2.5z=2.5) 15,808 38s 0.2 3.88 62.5 TripWire (top-1000) 1,000 91s 5.3 7.02 8.0 TripWire (top-2500) 2,502 92s 1.7 5.87 12.5 Figure 3: Distribution of Toxic Neurons Across Layers Safety Neuron Distribution Analysis To further examine the identified safety neurons, we analyze their distribution across layers for each model. Figure 3 shows the number of selected safety neurons per layer under the top-2500 budget. Across all three models, safety neurons are concentrated in the middle-to-upper layers and are nearly absent in the early layers. For Llama-2 and Llama-3.1, the neuron count rises sharply around layers 8–14, peaks in the middle layers (layers 12–16), and gradually decreases toward the final layers. For Qwen2.5-7B, the distribution is shifted later, with the majority of safety neurons concentrated in layers 15–21 and a sharp onset around layer 13. This pattern is consistent with the established view that early layers primarily encode low-level syntactic features, while middle and upper layers encode higher-level semantic and behavioral representations. The concentration of safety neurons in these layers suggests that harm detection and refusal triggering are mediated by mid-to-high-level representations rather than surface-level token features. Related Work Adversarial Attacks on LLMs Jailbreak attacks craft adversarial prompts that circumvent the safety alignment of LLMs, compelling them to produce restricted or harmful outputs. Optimization-based attacks (34) leverage gradient signals to search for adversarial suffixes that maximize the probability of harmful responses. Generator-based attacks (13; 11) instead fine-tune a model on successful jailbreaks, enabling diverse adversarial prompts to be produced in a single forward pass without further optimization. Iterative refinement attacks (6; 17; 16) employ an attacker LLM to propose, test, and refine human-readable jailbreaks through a feedback-driven loop within a few queries. Besides these methods that perturb the input prompt, another line of work directly targets the internal components of the model, such as representation steering (14) and neuron-level manipulation (25). Among these component-level methods, NeuroStrike (25) is closest to our setting: it identifies sparse safety neurons with per-layer logistic-regression probes and prunes them to disable refusal behavior, demonstrating transferability across model variants. From a defense perspective, NeuroStrike adversarially corroborates our core premise that refusal is mediated by a sparse and identifiable set of safety neurons. Defenses for LLMs We organize existing defenses by the granularity of their intervention, ranging from the whole model down to individual neurons. At the coarsest granularity, alignment training such as RLHF (18) and adversarial training (26; 15) instills refusal behavior by shifting the entire response distribution, and must be repeated for every emerging threat. Inference-time defenses instead examine or mutate the input prompt (2; 19) or steer the decoding distribution (27), thereby perturbing every request at each forward pass even when the request is benign. Narrowing to sub-model granularity, Circuit Breaker (32) operates at the representation level, training the model to reroute harmful representations toward orthogonal directions and short-circuit the computation that would otherwise produce unsafe content. LED (29) operates at the layer level, locating safety-critical layers through pruning analysis and editing their parameters toward safe responses. DELMAN (23) operates at the parameter level, updating a small set of weights with closed-form solutions regularized by KL divergence to preserve utility. At the finest granularity, TraceRouter (20) works similarly by locating the neurons along all harm-related pathways and then suppressing or reversing their activations. We emphasize that the neurons TraceRouter targets are those responsible for generating harmful content, which are fundamentally different from the safety neurons defined in this work, namely those responsible for detecting harmful requests and triggering refusal. In addition, unlike these always-on defenses, our inference mode gates the intervention with a detector, so benign requests are processed by the unmodified model and the utility cost is further reduced. Conclusion In this work, we diagnose why existing neuron-level jailbreak defenses fail to deliver the promise of fine-grained protection: erasure-style methods must block harmful semantics distributed across the network, classifier-based attribution cannot separate safety neurons from generally important ones, and always-on interventions perturb benign requests even when no attack is present. These fixable problems, rather than the neuron granularity itself, are the true sources of the observed utility cost. Based on these insights, we propose TripWire, a training-free defense that identifies safety-specific neurons through a statistical funnel with false-discovery-rate control and clamps them at their harmful-conditional mean activations to trigger the refusal behavior learned during alignment, realized by two provably equivalent deployment modes. Across four safety-aligned LLMs and four representative attacks, TripWire reduces the average attack success rate to at most 2.0% with the smallest utility drop among all defenses, establishing that precise identification combined with trigger-style intervention enables practical neuron-level safety without costly retraining. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: Introduction. Alon and Kamfonas (2023) G. Alon and M. Kamfonas Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132. Cited by: Defenses for LLMs. Anil et al. (2023) R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White, A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski, and et al. Gemini: A Family of Highly Capable Multimodal Models. CoRR abs/2312.11805. Cited by: Introduction. Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: Experimental Setup. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: Introduction. Chao et al. (2023) P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. Cited by: Introduction, Experimental Setup, Adversarial Attacks on LLMs. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374. Cited by: Introduction. Cui et al. (2024) J. Cui, W. Chiang, I. Stoica, and C. Hsieh Or-bench: an over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947. Cited by: Statistically Rigorous Identification Funnel. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: Experimental Setup. Feng et al. (2026) R. Feng, H. Zhao, S. Tang, G. Liu, S. Lai, M. Wang, S. Zhuang, Y. Zha, C. Ma, and R. Yu Lookahead-gcg: improving universal multi-model optimization-based jailbreaking attacks via stochastic nesterov optimization. In Forty-third International Conference on Machine Learning, Cited by: Introduction. Guo et al. (2025) W. Guo, Z. Shi, Z. Li, Y. Wang, X. Liu, W. Wang, F. Liu, M. Zhang, and J. Li Jailbreak-r1: exploring the jailbreak capabilities of llms via reinforcement learning. arXiv preprint arXiv:2506.00782. Cited by: Introduction, Experimental Setup, Adversarial Attacks on LLMs. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: Large Language Models and Neurons. Liao and Sun (2024) Z. Liao and H. Sun AmpleGCG: learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs. In First Conference on Language Modeling, External Links: Link Cited by: Introduction, Experimental Setup, Adversarial Attacks on LLMs. Lin et al. (2024) Y. Lin, P. He, H. Xu, Y. Xing, M. Yamada, H. Liu, and J. Tang Towards understanding jailbreak attacks in LLMs: a representation space analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 7067–7085. External Links: Link, Document Cited by: Adversarial Attacks on LLMs. Liu et al. (2024a) F. Liu, Z. Xu, and H. Liu Adversarial tuning: defending against jailbreak attacks for LLMs. arXiv preprint arXiv:2406.06622. Cited by: Defenses for LLMs. Liu et al. (2024b) X. Liu, N. Xu, M. Chen, and C. Xiao AutoDAN: generating stealthy jailbreak prompts on aligned large language models. In ICLR, Cited by: Adversarial Attacks on LLMs. Mehrotra et al. (2024) A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi Tree of attacks: jailbreaking black-box LLMs automatically. In NeurIPS, Cited by: Adversarial Attacks on LLMs. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. External Links: 2203.02155 Cited by: Defenses for LLMs. Robey et al. (2023) A. Robey, E. Wong, H. Hassani, and G. J. Pappas SmoothLLM: defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. Cited by: Introduction, Defenses for LLMs. Shi et al. (2026) C. Shi, S. Li, W. Lu, W. Wu, C. Wang, Z. Cheng, F. Shen, and T. Chua TraceRouter: robust safety for large foundation models via path-level intervention. arXiv preprint arXiv:2601.21900. Cited by: Introduction, Introduction, Introduction, Trigger-Style Clamp Intervention, Experimental Setup, Defenses for LLMs. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following llama model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: Large Language Models and Neurons. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: Experimental Setup. Wang et al. (2025) Y. Wang, F. Weng, S. Yang, Z. Qin, M. Huang, and W. Wang DELMAN: dynamic defense against large language model jailbreaking with model editing. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 11465–11481. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: Introduction, Introduction, Introduction, Experimental Setup, Defenses for LLMs. Wang et al. (2026) Z. Wang, J. Liang, F. Zhu, W. Zhao, J. Fang, J. Ji, H. Wang, and T. Chua Safeneuron: neuron-level safety alignment for large language models. arXiv preprint arXiv:2602.12158. Cited by: Introduction. Wu et al. (2025) L. Wu, S. Behrouzi, M. Rostami, M. Thang, S. Picek, and A. Sadeghi NeuroStrike: neuron-level attacks on aligned llms. External Links: 2509.11864 Cited by: Introduction, Introduction, Identifying Safety Neurons, Identifying Safety Neurons, Statistically Rigorous Identification Funnel, Trigger-Style Clamp Intervention, Ablation Study, Adversarial Attacks on LLMs. Xhonneux et al. (2024) S. Xhonneux, A. Sordoni, S. Günnemann, G. Gidel, and L. Schwinn Efficient adversarial training in LLMs with continuous attacks. In NeurIPS, Cited by: Defenses for LLMs. Xu et al. (2024) Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran SafeDecoding: defending against jailbreak attacks via safety-aware decoding. In ACL, Cited by: Introduction, Defenses for LLMs. Zhao and Huang (2025) C. Zhao and K. Huang Unraveling llm jailbreaks through safety knowledge neurons. arXiv preprint arXiv:2509.01631. Cited by: Identifying Safety Neurons. Zhao et al. (2024) W. Zhao, Z. Li, Y. Li, Y. Zhang, and J. Sun Defending large language models against jailbreak attacks via layer-specific editing. In EMNLP, Cited by: Introduction, Introduction, Experimental Setup, Defenses for LLMs. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: Introduction, Experimental Setup. Zou et al. (2023a) A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: Introduction. Zou et al. (2024) A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks Improving alignment and robustness with circuit breakers. In NeurIPS, Cited by: Experimental Setup, Experimental Setup, Defenses for LLMs. Zou et al. (2023b) A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: Experimental Setup. Zou et al. (2023c) A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: Introduction, Large Language Models and Neurons, Experimental Setup, Experimental Setup, Adversarial Attacks on LLMs.