Paper deep dive
NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation
Zhiyuan Xu, Muhammad Firhard Roslan, Joseph Gardiner, Sana Belguith, Lichao Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/28/2026, 3:26:30 AM
Summary
The paper introduces NeuronFuzz, a white-box fuzzing framework for evaluating Large Language Model (LLM) safety. Unlike existing methods that rely on expensive and sparse response-level feedback, NeuronFuzz utilizes internal safety neuron activations as continuous execution feedback during the prefill stage. It constructs a 'SafetyOracle' to convert these activations into a differentiable safety alarm score, which guides gradient-based mutation of jailbreak templates. The framework was evaluated across 21 models, demonstrating superior jailbreak discovery rates and zero-shot transfer capabilities compared to baselines.
Entities (7)
Relation Signals (6)
Zhiyuan Xu → affiliatedwith → University of Bristol
confidence 99% · Zhiyuan Xu Affiliation: University of Bristol
SafetyOracle → converts → safety-neuron activations
confidence 95% · A SafetyOracle converts safety-neuron activations into a continuous safety alarm score
SafetyOracle → provides → continuous execution feedback
confidence 95% · exploits internal safety neurons as continuous execution feedback for LLM safety evaluation
NeuronFuzz → uses → SafetyOracle
confidence 95% · NeuronFuzz uses template-invariant harmful and benign inputs and stability-aware selection to identify a compact set of safety neurons... To construct the SafetyOracle
NeuronFuzz → outperforms → AutoDAN
confidence 90% · outperforming baselines by up to 48 percentage points
NeuronFuzz → outperforms → PAIR
confidence 90% · outperforming baselines by up to 48 percentage points
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Safety evaluation is critical for assessing whether aligned Large Language Models (LLMs) remain robust against jailbreak attacks. Existing automated testing methods, however, largely rely on response-level feedback: each candidate prompt typically requires generating a target-model response to evaluate its attack effectiveness. This process is expensive and, more importantly, provides only sparse guidance on strongly aligned models, where most candidates are rejected with the same failure outcome. This paper presents NeuronFuzz, a white-box fuzzing framework that exploits internal safety neurons as continuous execution feedback for LLM safety evaluation. A SafetyOracle converts safety-neuron activations into a continuous safety alarm score that serves as feedback for fuzzing and can be obtained during prefill, eliminating response generation from the fuzzing loop. To construct the SafetyOracle, NeuronFuzz uses template-invariant harmful and benign inputs and stability-aware selection to identify a compact set of safety neurons whose activations capture harmful-intent recognition. Moreover, since the safety alarm score is differentiable, NeuronFuzz uses its gradients to identify safety-sensitive template positions and a masked language model to generate fluent, context-compatible mutations while preserving original harmful payload and avoiding additional optimization variables. We evaluate NeuronFuzz across 21 text and multimodal models. Across five white-box source models, it achieves a 76-100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points. Its optimized templates further transfer zero-shot to open-weight and six proprietary target models, achieving average ASR and top-5 ensemble ASR (EASR) of 69.6%/92.6% and 44.1%/60.0%, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2608.26222v1
- Canonical: https://arxiv.org/abs/2608.26222v1
Trouble viewing inline? Open PDF directly →
Full Text
107,778 characters extracted from source content.
Expand or collapse full text
NeuronFuzz: Safety Neuron Guided Fuzzing for LLM Safety Evaluation Zhiyuan Xu Affiliation: University of Bristol Email: zhiyuan.xu@bristol.ac.uk Muhammad Firhard Roslan Affiliation: University of Bristol Email: firhard.roslan@bristol.ac.uk Joseph Gardiner Affiliation: University of Bristol Email: joe.gardiner@bristol.ac.uk Sana Belguith Affiliation: University of Bristol Email: sana.belguith@bristol.ac.uk Lichao Wu Affiliation: University of Bristol Email: lichao.wu@bristol.ac.uk Abstract Safety evaluation is critical for assessing whether aligned Large Language Models (LLMs) remain robust against jailbreak attacks. Existing automated testing methods, however, largely rely on response-level feedback: each candidate prompt typically requires generating a target-model response to evaluate its attack effectiveness. This process is expensive and, more importantly, provides only sparse guidance on strongly aligned models, where most candidates are rejected with the same failure outcome. This paper presents NeuronFuzz, a white-box fuzzing framework that exploits internal safety neurons as continuous execution feedback for LLM safety evaluation. A SafetyOracle converts safety-neuron activations into a continuous safety alarm score that serves as feedback for fuzzing and can be obtained during prefill, eliminating response generation from the fuzzing loop. To construct the SafetyOracle, NeuronFuzz uses template-invariant harmful and benign inputs and stability-aware selection to identify a compact set of safety neurons whose activations capture harmful-intent recognition. Moreover, since the safety alarm score is differentiable, NeuronFuzz uses its gradients to identify safety-sensitive template positions and a masked language model to generate fluent, context-compatible mutations while preserving original harmful payload and avoiding additional optimization variables. We evaluate NeuronFuzz across 21 text and multimodal models. Across five white-box source models, it achieves a 76–100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points. Its optimized templates further transfer zero-shot to open-weight and six proprietary target models, achieving average ASR and top-5 ensemble ASR (EASR) of 69.6%/92.6% and 44.1%/60.0%, respectively. 1 Introduction Large Language Models (LLMs) are increasingly deployed in applications that directly interact with users. To prevent misuse, modern LLMs undergo safety alignment to recognize harmful requests and suppress unsafe responses. However, aligned models remain vulnerable to carefully crafted adversarial prompts, known as jailbreaks, that bypass these safety mechanisms and elicit harmful content [64, 55]. As LLMs and their alignment strategies continue to evolve, systematically identifying such safety failures has become an important part of evaluating model robustness before deployment. Existing LLM safety testing can broadly be divided into manual and automated approaches. Manual red teaming relies on human-designed strategies such as role-playing, instruction conflicts, refusal suppression, and contextual framing [25, 36, 64, 55]. While effective at exposing individual weaknesses, manual testing requires substantial human effort and domain expertise, limiting the scale and diversity of explored inputs. Automated approaches improve scalability by algorithmically generating or optimizing jailbreak prompts. For example, AutoDAN [35] evolves prompts through hierarchical genetic optimization; PAIR [11] iteratively refines attacks using an attacker LLM. Another line of automated testing adapts fuzzing, a widely adopted technique for software and hardware testing [37, 68], to LLM safety evaluation [76]. LLM fuzzers iteratively mutate jailbreak templates using language-based operators such as rephrasing, shortening, and expansion, and retain promising candidates based on response-level safety evaluation feedback. Through repeated mutation and selection, they can gradually discover more effective jailbreak templates. Despite their different search strategies, these methods follow the same evaluation paradigm: each candidate is submitted to the target LLM; a response is generated and then evaluated by a classifier, heuristic detector, or LLM judge. However, this response-level paradigm introduces two fundamental limitations for automated safety testing: (i) Sparse Search Feedback: Response-level evaluation provides little information about how promising an unsuccessful candidate is. This problem is particularly severe for strongly aligned models, where most mutated prompts are refused and consequently receive the same failure outcome. Yet these candidates may affect the model’s underlying safety behavior very differently: one may leave the safety mechanism largely unchanged, while another may substantially weaken it without yet producing a successful jailbreak. Response-level feedback cannot distinguish between them, leaving the fuzzer with limited guidance on which candidates to retain and further explore. Increasing the query budget does not fundamentally solve this problem because the feedback itself remains sparse. (i) Expensive Response Generation: Evaluating every candidate requires autoregressive decoding, which is substantially more expensive than processing the input prompt alone [26, 32]. Response-based methods may additionally invoke external classifiers or LLM judges after generation. Since automated fuzzers repeatedly evaluate large numbers of candidates, complete response generation becomes a major scalability bottleneck. Thus, an efficient fuzzer requires a feedback signal that can evaluate candidates before response generation. Our Approach and Contributions: To address these limitations, we present NeuronFuzz, a white-box fuzzing framework that uses internal safety-neuron activations as continuous execution feedback. Inspired by coverage-guided fuzzing [79], NeuronFuzz constructs a lightweight SafetyOracle that maps a compact set of stable safety neurons to a continuous safety alarm score during prefill. To ensure that this signal captures harmful intent rather than jailbreak-template artifacts, we identify safety neurons using template-invariant harmful–benign pairs and stability-aware selection. The SafetyOracle eliminates response generation for intermediate candidates and distinguishes promising mutations even when they share the same response-level failure outcome. Moreover, its differentiable score allows NeuronFuzz to locate safety-sensitive template positions through gradients, while a masked language model generates context-compatible replacements that preserve the harmful payload and natural-language structure. Our main contributions are as follows: • We introduce a new perspective on LLM fuzzing that uses safety neurons as execution feedback, replacing expensive and sparse response-level evaluation with continuous internal feedback available during prefill. • We design a lightweight SafetyOracle based on template-invariant activation extraction and stability-aware safety-neuron selection. Its differentiable safety alarm score guides both candidate ranking and gradient-based localization of safety-sensitive template positions. • We develop NeuronFuzz, which combines prefill-only SafetyOracle feedback with gradient-guided masked-token mutation to efficiently explore jailbreak templates while preserving the harmful payload and natural-language structure. • We conduct extensive evaluations across 21 text and multimodal models. NeuronFuzz consistently improves jailbreak discovery over representative baselines and its optimized templates transfer across model families, proprietary APIs, datasets, and visual safety tasks. The remainder of this paper is organized as follows. Section 2 provides background on LLM safety evaluation and fuzzing. Section 3 presents the motivation, threat model, and design of SafetyOracle and NeuronFuzz. Section 4 describes the implementation and experimental setup. Section 5 reports the main results, and Section 6 presents the ablation studies. Section 7 evaluates input defenses and defensive uses of the SafetyOracle. Finally, Section 8 concludes the paper. 2 Preliminaries 2.1 LLM Safety Evaluation Aligned large language models (LLMs) are trained to refuse harmful requests, but their safety robustness still needs to be systematically evaluated before and after deployment. LLM safety evaluation examines whether a model continues to follow its safety policy when exposed to challenging or adversarial inputs [64, 55]. A common jailbreak test consists of a harmful payload and a jailbreak template [21, 43]. The payload specifies the harmful request, while the template changes how the request is presented to the model. The same template can be combined with different payloads to examine whether a safety weakness generalizes across harmful tasks. Existing methods can be broadly grouped into three approaches: manual testing, automated testing, and internal safety analysis. Manual Testing. Early LLM safety testing mainly relied on manually designed adversarial prompts. Human testers construct prompts using strategies such as role-playing, fictional scenarios, instruction conflicts, refusal suppression, and emotional framing to probe weaknesses in model alignment [55, 36, 64, 25]. These tests are easy to interpret and show that even small changes in prompt context may substantially affect model behavior [6]. However, manual testing requires substantial human effort and domain knowledge, which limits the number and diversity of test cases that can be explored. Automated Safety Testing. Automated methods reduce this manual effort by generating or refining jailbreak prompts algorithmically. Different methods adopt different optimization strategies. AutoDAN uses a hierarchical genetic algorithm to evolve jailbreak prompts at sentence and word levels [35]; PAIR uses an attacker LLM to iteratively refine prompts based on target-model responses and an external judge [11]; and LLM-Fuzzer automatically generates variants of existing jailbreak templates [76]. Although their optimization procedures differ, these methods share the goal of producing effective safety test cases with less human intervention. Their effectiveness is typically determined from the target model’s generated response, providing an external measure of whether a test case successfully bypasses the model’s safety policy. Internal Safety Analysis. Beyond model outputs, prior studies show that safety-related information is also represented internally. Harmful and benign inputs can produce different activation patterns, while refusal behavior can be associated with specific representation directions or internal components [62, 7, 47, 52, 72]. More recent neuron-level studies further show that this information can be localized to compact neuron subsets whose activations selectively distinguish harmful from benign inputs [67, 27]. Manipulating these neurons or related internal components can substantially alter model’s safety behavior, providing evidence that they are functionally related to safety and refusal [67, 27, 73, 66, 61]. These findings suggest that internal activations expose safety-related information that is not directly observable from the final response. Our work builds on this observation and uses safety-related neuron activations as signals for automated safety testing without modifying the model’s internal states. 2.2 Fuzzing Fuzzing is an automated testing technique that repeatedly generates random test inputs to expose unexpected program behaviors and security vulnerabilities [37]. Fuzzing methods can be broadly classified as black-box, grey-box, or white-box according to the amount of information available from the target program. A coverage-guided greybox fuzzing process [79, 15] typically contains four components: a seed corpus and scheduler [12], an input mutator [31], runtime instrumentation [79], and execution feedback [9]. The fuzzing process begins with a corpus of initial seed inputs. In each iteration, the scheduler assigns energy values to seeds according to a scheduling strategy. Seed inputs that exercise new coverage goals based on the feedback receive higher energy and are more likely to be selected for next mutation. The mutator then modifies the selected seed to generate new tests, which are executed by the target program. An oracle [8] then determines whether the execution reveals an unexpected behavior (e.g., the program crashes). By repeatedly using execution information to update seed selection and mutation, the fuzzer can progressively explore the program’s execution space. LLM fuzzing adapts this feedback-guided process to natural-language safety testing [76]. Jailbreak templates can serve as initial seeds, while language-based mutation operators generate new template variants through operations such as expansion, rephrasing, or shortening. The target program is an LLM, and a successful test corresponds to a generated response that violates the intended safety policy. The generated responses are evaluated using classifiers, heuristic rules, or LLM judges, which serve as response-level oracles [11, 76, 35]. The evaluation outcomes are then used to decide which prompts should be retained and explored further. However, this response-level paradigm has two limitations for LLM safety evaluation. First, each prompt must generate a response before it can be evaluated, introducing repeated decoding and evaluation overhead. Second, the resulting feedback provides limited information for distinguishing prompts with the same response-level outcome. This problem becomes more pronounced on strongly aligned models, where many mutated prompts are rejected and receive the same failure label. The fuzzer therefore has little information about which unsuccessful prompts are more promising for further exploration. These limitations motivate a feedback signal that can evaluate and distinguish prompts before response generation. 3 NeuronFuzz 3.1 Motivation Figure 1: Overview of NeuronFuzz. (a) The SafetyOracle extracts MLP activations from template-invariant harmful and benign pairs, selects stable safety neurons, and produces a continuous safety alarm score. (b) The fuzzer minimizes this score through gradient-guided, template-only mutation and evaluates candidates using prefill-only oracle feedback. From Response-Level Feedback to Internal Safety Signals. Our key intuition is to use internal safety representations as a denser source of fuzzing feedback. Prior white-box studies show that safety-related information is encoded in model activations and can be captured by a small set of internal features [67, 27]. Rather than modifying these representations, we use their activations only as signals to guide safety testing. Internal safety signals have three properties that make them suitable for fuzzing: (i) they can be obtained during the prefill stage without generating a response, reducing the cost of evaluating intermediate candidates; (i) they provide continuous information that can distinguish candidates with the same response-level outcome. This is particularly useful on strongly aligned models, where many mutated prompts are rejected but may still produce different internal safety signals; (i) these signals remain differentiable with respect to the input, allowing gradients to identify token positions that strongly affect the model’s safety representation. Stable and Intent-Focused Feedback. Raw activations, however, cannot be directly used as reliable fuzzing feedback. The activation space is high-dimensional and contains noisy or redundant features. Jailbreak templates may also introduce activation patterns that reflect prompt style rather than the harmful intent of the payload. We therefore seek a compact and stable signal that captures harmful intent while remaining robust to jailbreak contexts. To achieve this, we construct template-invariant harmful and benign pairs and apply stability-aware neuron selection for the training of SafetyOracle. Gradient-Guided Mutation. A useful fuzzing signal should not only rank candidates but also guide how they are mutated. Beyond providing continuous feedback for candidate ranking, the SafetyOracle is differentiable with respect to the input, allowing us to identify which template positions most strongly influence the internal safety signal. We therefore design a gradient-guided mutation strategy that uses gradients to locate sensitive positions and a masked language model to generate context-compatible replacements. In this design, the gradient determines where to mutate, while the masked language model determines what to replace. We restrict all mutations to the jailbreak template and keep the harmful payload unchanged. Together, these designs allow NeuronFuzz to use internal safety signals for both continuous candidate ranking and targeted prompt mutation. Figure 1 summarizes the two stages of NeuronFuzz. Figure 1(a) presents the construction of the SafetyOracle and the interpretation of its safety alarm score. Figure 1(b) shows how SafetyOracle is integrated into the online fuzzing loop: gradients localize sensitive template positions, the masked language model generates replacements, and the resulting candidates are scored during prefill to guide subsequent seed selection. 3.2 Threat Model We consider two settings for NeuronFuzz: an authorized evaluator and a malicious adversary. The evaluator uses white-box information to identify safety weaknesses, while the adversary uses a surrogate model to optimize jailbreak templates and applies them either to the same model or transfers them to other target models. Evaluator. The evaluator may be the model developer, service provider, or an authorized third-party auditor. The evaluator has access to the model’s parameters, MLP activations, and input gradients, which are used to construct the SafetyOracle and guide template mutation. The evaluator does not modify model parameters, internal activations, system prompts, or decoding settings. The discovered test cases are therefore evaluated through the model’s normal input interface. Adversary. We follow the prompt-level jailbreak setting used by prior automated jailbreak methods [84, 35, 76]. The adversary has white-box access to a local source or surrogate model and uses NeuronFuzz to optimize jailbreak templates. These templates can attack the source model directly or be transferred zero-shot to other target models. In the transfer setting, the adversary requires only input-output access to the target model. 3.3 SafetyOracle Design 3.3.1 Template-invariant Activation Extraction Safety-related neurons are identified by contrasting the activations of harmful and benign prompts [67, 62]. The goal is to extract features that primarily reflect harmful intent and remain stable across different prompt contexts. However, jailbreak templates may introduce additional activation patterns through role-playing instructions, refusal suppression, or unusual linguistic structures. These template-specific patterns can act as shortcuts, causing the extracted features to reflect template style rather than payload harmfulness. To reduce this effect, we construct template-invariant harmful–benign pairs that share the same jailbreak template but contain payloads with different safety labels. Let =τ1,…,τMT=\ _1,…, _M\ denote the jailbreak templates, and let hP_h and bP_b denote the harmful and benign payload sets. Given a template τ and payload p, the model input is x=τ⊕px=τ p, where ⊕ inserts the payload into the template. Each template therefore forms a paired input: xi,jh=τj⊕pih,xi,jb=τj⊕pib,x^h_i,j= _j p^h_i, x^b_i,j= _j p^b_i, (1) where pih∈hp^h_i _h and pib∈bp^b_i _b, with labels y(xi,jh)=1y(x^h_i,j)=1 and y(xi,jb)=0y(x^b_i,j)=0. Since the same template appears in both classes, the label difference is determined by the payload rather than the surrounding template. For each input, a prefill pass is performed while forward hooks collect the outputs of the gate and up projections in every Transformer MLP block [66, 61]. Let ℐ(x)I(x) denote the input-prompt token positions. For layer ℓ and projection module m∈gate,upm∈\gate,up\, token-level outputs t(ℓ,m)(x)z^( ,m)_t(x) are aggregated by element-wise max pooling: (ℓ,m)(x)=maxt∈ℐ(x)t(ℓ,m)(x).a^( ,m)(x)= _t (x)z^( ,m)_t(x). (2) This produces a fixed-dimensional activation vector for each layer and projection module, independent of prompt length. The pooled vectors are concatenated across all layers and modules to obtain the prompt-level representation (x)a(x), with each dimension treated as a candidate neuron feature. Section 6.1 ablates the design of template-invariant extraction. 3.3.2 Stability-Aware Safety Neuron Selection Template-invariant extraction reduces template-specific effects, but neuron importance estimated from a single training set can still be sensitive to sampling variation [29, 40]. A neuron that appears strongly predictive in one sample may become weak or change its association in another [29]. We therefore use bootstrap-based stability selection to identify neurons that remain consistently associated with harmful inputs across resampled training sets [40]. We perform B bootstrap runs and fit a linear classifier in each run. For an MLP block ℓ containing pℓp_ candidate neurons, we retain the qℓq_ neurons with the largest absolute coefficients, where qℓ=⌈pℓ⌉q_ = p_ . This provides a consistent selection size across models with different hidden dimensions. For each neuron j, let nj+n_j^+ and nj−n_j^- denote the numbers of runs in which it is retained with a positive or negative coefficient. We define its selection rate sjs_j and sign consistency cjc_j as sj s_j =nj++nj−B, = n_j^++n_j^-B, cj c_j =max(nj+,nj−)nj++nj−. = (n_j^+,n_j^-)n_j^++n_j^-. (3) We set cj=0c_j=0 if neuron j is never retained. Here, sjs_j measures how consistently the neuron is selected, while cjc_j measures whether its coefficient maintains the same direction across bootstrap runs. Since harmful inputs are assigned the positive label, we further require the dominant coefficient direction to be positive. Let τs _s and τc _c denote the thresholds for selection rate and sign consistency, respectively. The final safety-neuron set is =j|sj≥τs,cj≥τc,nj+>nj−.S= \j\, |\,s_j≥ _s,\;c_j≥ _c,\;n_j^+>n_j^- \. (4) This selection removes neurons that are unstable across resampled training sets or whose association with harmful inputs changes direction. The resulting activation vector (x)a_S(x) is used as the input to the SafetyOracle. Section 6.1 compares our method with prior statistic-based neuron selection. 3.3.3 Neuron-Based SafetyOracle The selected safety neurons capture safety-related information, but their activations form a multidimensional representation rather than a single score that can be directly compared across candidate prompts. We therefore train a lightweight classifier as SafetyOracle to map their joint activation pattern into a continuous safety signal [16, 10]. Given an input x, let (x)a_S(x) denote the activation vector of the selected safety neurons. We standardize each dimension as (x)=((x)−)/z_S(x)=(a_S(x)- μ)/s, where μ and s are the training-set mean and standard deviation. The SafetyOracle computes: Salarm(x)=sigmoid(⊤(x)+b)∈(0,1).S_alarm(x)=sigmoid (w z_S(x)+b )∈(0,1). (5) We refer to Salarm(x)S_alarm(x) as the safety alarm score. Since harmful inputs are assigned the label y=1y=1, a higher score indicates an activation pattern that is more characteristic of harmful prompts, suggesting stronger internal recognition of harmful intent. A lower score indicates a weaker internal safety alarm. Although proposed stability selection in Section 3.3.2 removes many unreliable neurons, the retained features may still contain redundant or correlated information [40]. We therefore implement the SafetyOracle using logistic regression with Elastic Net regularization [17]. Elastic Net combines ℓ1 _1 and ℓ2 _2 regularization. The ℓ1 _1 term encourages uninformative feature weights to become zero, while the ℓ2 _2 term stabilizes the weights of correlated features. This combination allows the SafetyOracle to further control redundancy in the selected neuron features when learning the final scoring function. Let ℒcls=1N∑i=1NℒBCE(yi,Salarm(xi))L_cls= 1N _i=1^NL_BCE(y_i,S_alarm(x_i)) denote the classification loss based on the SalarmS_alarm in Eq. 5. We optimize min,bℒcls+1NC[ρ∥1+1−ρ2∥22], _w,bL_cls+ 1NC [ρ _1+ 1-ρ2 _2^2 ], (6) where N is the number of training examples, C>0C>0 controls the inverse regularization strength, and ρ∈[0,1]ρ∈[0,1] balances the ℓ1 _1 and ℓ2 _2 penalties. We compare Elastic Net with four other classification models and justify this choice in Appendix E. During fuzzing, the SafetyOracle is used only to guide candidate selection and mutation. Each candidate requires only a prefill pass and does not require the target model to generate the response. The continuous safety alarm score SalarmS_alarm allows the fuzzer to rank candidates that would otherwise receive the same response-level failure label. Moreover, because the SafetyOracle remains differentiable with respect to the input embeddings, its gradients can identify sensitive token positions for the mutation process described in Section 3.4.2. 3.4 NeuronFuzz Pipeline Following the software fuzzing described in Section 2.2, NeuronFuzz adapts the same feedback-guided pipeline to safety evaluation. In each iteration, the scheduler first selects a jailbreak template for mutation. The mutator then uses SafetyOracle gradients to identify sensitive template positions and a masked language model to generate context-compatible replacements. Each resulting candidate is executed through a prefill pass to collect the selected safety-neuron activations. Finally, the SafetyOracle converts these activations into a safety alarm score and uses it to update the seed scheduler for the next iteration. The workflow is shown in Figure 1(b). 3.4.1 Seed Scheduler The seed scheduler determines which templates are selected for further mutation. We follow prior work and adopt MCTS-Explore [76], where each node represents a jailbreak template and each edge records the mutation relationship between a parent and its child. The scheduler selects templates according to the statistics accumulated from previous oracle feedback, balancing the exploration of new templates with the reuse of promising ones. The selected template is then passed to the gradient-guided mutator. 3.4.2 Gradient-Guided Mutation Given a selected template τ and a harmful payload php_h, we construct the input x=τ⊕phx=τ p_h. Rather than mutating template tokens uniformly or directly optimizing arbitrary token replacements [84], we use the SafetyOracle to focus mutation on positions that most strongly influence the internal safety signal. Specifically, SafetyOracle gradients determine where to mutate, while a masked language model determines what context-compatible replacement to use. This design narrows the mutation space through token replacement, preserving the prompt length and natural-language structure without introducing additional tokens or auxiliary optimization variables. Since the SafetyOracle is differentiable with respect to the input, we backpropagate the safety alarm score to the input embeddings. Let te_t denote the embedding of token t. Its sensitivity is defined as: gt=‖∂Salarm(x)∂t‖2.g_t= \| ∂ S_alarm(x) _t \|_2. (7) A larger gtg_t indicates that the corresponding position has a stronger influence on the internal safety signal. We therefore rank template tokens by gtg_t and select the top-ranked positions for mutation. For each selected position, the original token is masked and a lightweight masked language model proposes context-compatible replacements. Each replacement produces a new template τ′τ , which is combined with the unchanged payload to form x′=τ′⊕phx =τ p_h. All mutations are restricted to the jailbreak template, ensuring that a successful test reflects a safety bypass rather than a modification of the underlying payload (i.e., harmful request) [58, 54]. 3.4.3 Runtime Instrumentation For each candidate input x′x , NeuronFuzz performs a prefill pass through the target model and collects the activations of the selected safety-neuron set S. These activations serve as the runtime execution signal and are passed to the SafetyOracle for evaluation. Unlike existing response-level fuzzing [76, 11], this step does not require autoregressive response generation. 3.4.4 Oracle Feedback and Seed Update Given the collected safety-neuron activations, the SafetyOracle computes the safety alarm score Salarm(x′)S_alarm(x ) using Eq. 5. Since a lower score indicates weaker internal recognition of harmful intent, NeuronFuzz defines the fuzzing reward as R(x′)=−Salarm(x′).R(x )=-S_alarm(x ). The candidate and its reward are then added to the MCTS tree to update the scheduler. Candidates with lower safety alarm scores receive higher rewards and are therefore more likely to be selected for subsequent mutation. Compared with prior response-level feedback, SalarmS_alarm provides a continuous signal, allowing the scheduler to distinguish candidates even when they would receive the same response-level outcome. This feedback closes the fuzzing loop and guides subsequent exploration toward more promising templates. 4 Implementation & Experiment Setup Table 1: Specifications of target LLMs. Model Usage Architecture Active/Total Params (B) Reasoning Access Release Date Provider DeepSeek-R1-14B [23] Source Dense 14.7 / 14.7 CoT White-box 2025.01 DeepSeek GPT-OSS-20B [2] Source MoE 3.6 / 21.0 CoT White-box 2025.08 OpenAI Gemma-3-4B-it [18] Source Dense 4.0 / 4.0 Non-CoT White-box 2025.03 Google Gemma-4-E4B-it [59] Source Dense 4.5 / 8.0† CoT/Non-CoT White-box 2026.04 Google Llama-3.1-8B-Instruct [22] Source Dense 8.0 / 8.0 Non-CoT White-box 2024.07 Meta Llama-3.2-3B-Instruct [3] Transfer Dense 3.0 / 3.0 Non-CoT Inference only 2024.09 Meta Phi-4 [1] Transfer Dense 14.0 / 14.0 Non-CoT Inference only 2024.12 Microsoft Qwen3.6-35B-A3B [51] Transfer MoE 3.0 / 35.0 CoT/Non-CoT Inference only 2026.04 Alibaba Qwen3.6-27B [50] Transfer Dense 27.0 / 27.0 CoT/Non-CoT Inference only 2026.04 Alibaba GLM-4.7-Flash [78] Transfer MoE 3.0 / 30.0 CoT/Non-CoT Inference only 2026.01 Z.ai LongCat-2.0 [41] Transfer MoE 48.0 / 1600.0 CoT/Non-CoT API inference only 2026.06 Meituan DeepSeek-V4-Flash [71] Transfer MoE 13.0 / 284.0 CoT/Non-CoT API inference only 2026.04 DeepSeek DeepSeek-V4-Pro [71] Transfer MoE 49.0 / 1600.0 CoT/Non-CoT API inference only 2026.04 DeepSeek Grok-4.5 [69] Transfer MoE Undisclosed CoT API inference only 2026.07 SpaceXAI Gemini-2.5-Pro [13] Transfer Undisclosed Undisclosed CoT API inference only 2025.06 Google Gemini-3-Flash [20] Transfer Undisclosed Undisclosed CoT/Non-CoT API inference only 2025.12 Google † For Gemma-4-E4B-it, 4.5B denotes the effective parameter count, whereas 8.0B includes the per-layer embedding parameters. CoT/Non-CoT denotes models that support both explicit thinking and direct-answer modes. We disable CoT for hybrid models by default for evaluation efficiency. 4.1 Fuzzing Details Activation Extraction and Neuron Selection. We register forward hooks on the gate and up projection modules of every Transformer block and apply element-wise max pooling [67, 66]. To remove the noise introduced by chat format, we retain only the activation positions that correspond to the input prompt. For stability selection, we perform B=100B=100 bootstrap runs. Following previous research [28], each run samples 70% of the training examples and fits a logistic regression classifier on the extracted activations. A neuron is included in the final set when its selection rate is at least 0.6, its sign consistency is at least 0.75, and its dominant coefficient direction is positive [40]. SafetyOracle Training. We standardize each selected neuron using the mean and standard deviation computed from the training set. We train one SafetyOracle for each source model using Elastic Net logistic regression. We set the inverse regularization strength to C=1.0C=1.0 and the ℓ1 _1 ratio to ρ=0.5ρ=0.5. We use the SAGA solver with a maximum of 5,000 iterations and a convergence tolerance of 10−410^-4. Mutation. For gradient-guided mutation, we first identify three template token positions with the largest gradient norms using Eq. 7. We use ModernBERT-base (149M parameters) as the masked language model for token replacement [63]. At each selected position, we replace the original token with a mask token and use ModernBERT to propose context-compatible alternatives. We then construct 20 replacement tuples across the three positions. After removing duplicate variants and discarding candidates that modify the harmful payload, we randomly retain eight mutation candidates for each parent template. Each candidate is combined with the unchanged payload and evaluated through a forward pass of the target model. The candidates are then scored by SafetyOracle. Seed Templates. Recent studies in software fuzzing suggest the quality of the initial seed pool can affect fuzzing effectiveness [24, 56]. Following previous work [36], we collect jailbreak templates from a public repository [4] and remove duplicate, unsuitable, or payload-modifying templates. After filtering, we retain 64 templates as the initial seed pool. Appendix A provides further details. 4.2 Datasets and Seed Templates Target LLMs. For the evaluation, we use the 16 recent LLMs from nine providers as shown in Table 1. Five models are used as white-box source models, where we construct the SafetyOracle and directly evaluate NeuronFuzz under the evaluator setting described in Section 3.2. For cross-model transfer, jailbreak templates are optimized on selected source models and then applied zero-shot to target models without target-specific optimization, following the adversary setting. The model set covers both dense and mixture-of-experts (MoE) architectures, as well as Chain-of-Thought (CoT), non-CoT, and hybrid reasoning modes. All transfer models are evaluated with an inference-only setting, whereas proprietary models are accessed through the OpenRouter API [46]. Oracle Training Data. We construct a balanced dataset for activation extraction and SafetyOracle training. The initial dataset contains 7,000 harmful prompts randomly collected from LLM-LAT [57] and sorry-bench [70], together with 7,000 benign prompts collected from Natural Reasoning [77], OpenAssistant [30], and Dolly [14]. To reduce template-specific shortcuts as introduced in Section 3.3.1, we further construct 1,000 template-invariant pairs using 10 jailbreak templates as described in Appendix A. Each pair contains one harmful payload and one benign payload wrapped by the same template. Oracle-ASR Evaluation Data. To evaluate the trained SafetyOracle for each target model, we construct two disjoint test sets. The first set contains 4,000 unwrapped prompts (2,000 harmful and 2,000 benign) and measures its classification ability. We use it to evaluate whether SafetyOracle can distinguish harmful from benign inputs. The second set evaluates SafetyOracle under jailbreak templates. It contains 4,000 harmful prompts and a disjoint pool of 20 jailbreak templates. Each harmful prompt is combined with one randomly selected template from this pool. We generate complete target-model responses for these wrapped inputs, measure their response-level attack success rate (ASR), and analyze the relationship between jailbreak success and the corresponding safety alarm scores. The prompts and jailbreak templates in both test sets are disjoint from those used to train SafetyOracle. Fuzzing Evaluation Data. To show the effectiveness of NeuronFuzz, we randomly sample 100 harmful questions from the six categories of StrongREJECT [58]. These questions are used only for fuzzing and response-level evaluation and do not overlap with the datasets described above. 4.3 Evaluation Metrics We use a multi-stage evaluation pipeline to determine whether a generated response constitutes a successful jailbreak. Llama-Guard-4-12B [22] and Qwen3Guard-Gen-8B [80] independently evaluate every response. A response is assigned the agreed label when both judges produce the same decision. When their decisions differ, human annotators review the response and determine the final label. Let Ji,j∈0,1J_i,j∈\0,1\ indicate whether template τj _j successfully jailbreaks the model for harmful payload pip_i; follow [76], we define five metrics below. Table 2: Oracle performance on clean and jailbreak inputs. S¯ S denotes the mean safety alarm score. Δwrap _wrap measures the score reduction after applying a jailbreak template, and Δsep _sep measures the separation between failed and successful jailbreaks. Target Model Clean Classification Jailbreak Evaluation AUROC ¯ S_benign ¯ S_harmful ASR (%) _wrap ¯ S_succ ¯ S_fail _sep DeepSeek-R1-14B 0.997 0.001 0.999 14.20 0.116 0.557 0.937 0.380 Llama-3.1-8B-Instruct 0.989 0.001 0.999 22.35 0.147 0.586 0.929 0.343 Gemma-3-4B-it 0.969 0.013 0.976 38.75 0.195 0.572 0.913 0.341 Gemma-4-E4B-it 0.998 0.001 0.999 1.20 0.031 0.492 0.974 0.482 GPT-OSS-20B 0.999 0.001 0.999 0.35 0.040 0.633 0.960 0.327 Jailbreak Discovery Rate (JDR). JDR measures the ratio of harmful questions for which the fuzzer discovers at least one successful template within the given budget. Let BT_B denote all templates discovered within budget B. We define JDR=1N∑i=1Nmaxτj∈BJi,j.JDR= 1N _i=1^N _ _j _BJ_i,j. JDR measures the fuzzer’s ability to discover a successful template for each harmful question. Response Generations per Discovery (RGD). RGD measures the average number of target-model responses generated before the first verified jailbreak template. It includes responses generated during successful searches and the final verification. A lower RGD indicates less dependence on autoregressive response generation during the search. End-to-End Time per Discovery (ETD). ETD measures the average time required to discover the first verified jailbreak for a successfully jailbroken question. It includes all components of the fuzzing, such as target-model evaluation, gradient computation, mutation-model inference, SafetyOracle scoring, response generation, and external judge evaluation. We report both RGD and ETD only over successful questions. Attack Success Rate (ASR). ASR measures the effectiveness of an individual template [76]. Given N harmful questions, we define ASR(τj)=1N∑i=1NJi,j.ASR( _j)= 1N _i=1^NJ_i,j. A higher ASR indicates that the template succeeds on a larger fraction of the evaluation questions. To evaluate the quality of harmful responses, we also report StrongREJECT rubric [58]. Ensemble Attack Success Rate (EASR). EASR measures the combined effectiveness of a small set of templates. Let KT_K denote the selected set of K templates. A question is considered successfully tested if at least one template in KT_K produces a successful jailbreak. We compute EASR(K)=1N∑i=1N[maxτj∈KJi,j=1].EASR(T_K)= 1N _i=1^NI [ _ _j _KJ_i,j=1 ]. EASR represents the percentage of questions that could leverage at least one template from the subset to jailbreak the target LLM. 4.4 Evaluation Objectives We evaluate NeuronFuzz along four dimensions: • EO1: Oracle Validity. Whether the selected safety neurons produce a reliable SafetyOracle whose safety alarm score is associated with jailbreak success. • EO2: Fuzzing Effectiveness and Efficiency. Whether SafetyOracle feedback improves jailbreak discovery while reducing response generation and end-to-end search cost. • EO3: Universal Template. Can NeuronFuzz optimize a shared template that is effective across harmful payloads? • EO4: Transferability. Do optimized templates retain effectiveness on different architectures, scales, reasoning modes, modalities, and datasets without further optimization? 5 Experimental Results 5.1 EO1: Oracle Validity We evaluate whether the SafetyOracle provides a reliable safety signal for fuzzing from three aspects: whether it captures safety-related information, whether this signal remains meaningful under unseen jailbreak contexts, and, most importantly, whether it can rank candidate prompts according to their likelihood of jailbreak success. We first evaluate clean classification on harmful and benign prompts without jailbreak templates. We then evaluate 4,000 harmful prompts wrapped with jailbreak templates. For each wrapped input, we compute its safety alarm score and generate the target-model response to obtain the response-level evaluation result. Safety Signal Validity. Table 2 shows that the selected safety neurons provide a strong signal for distinguishing harmful from benign inputs. Across all five source models, the SafetyOracle achieves AUROC values between 0.969 and 0.999. Benign prompts receive mean safety alarm scores close to zero, whereas harmful prompts receive scores close to one. This confirms that the learned score captures safety-related information before any jailbreak context is introduced. Behavior under Jailbreak Contexts. We next examine whether the learned signal remains informative when the same harmful payload is embedded in jailbreak templates. The mean safety alarm score decreases on all five models after wrapping, with Δwrap _wrap ranging from 0.031 to 0.195. Because the harmful payload itself is unchanged, this consistent decrease suggests that the surrounding jailbreak context weakens the internal signal associated with harmful-intent recognition. This observation is consistent with prior findings that jailbreak contexts can suppress safety-related internal representations [81, 82]. More importantly, the alarm score consistently separates successful from failed jailbreak attempts. Although the overall response-level ASR varies substantially across models, from 0.35% on GPT-OSS-20B to 38.75% on Gemma-3-4B-it, successful jailbreaks receive lower mean alarm scores than failed attempts on every model. The resulting Δsep _sep ranges from 0.327 to 0.482. For example, on DeepSeek-R1-14B, the mean score decreases from 0.937 for failed attempts to 0.557 for successful jailbreaks. Similar gaps appear across the remaining models. These results indicate that SalarmS_alarm captures information related to jailbreak susceptibility beyond harmful-versus-benign classification. Figure 2: Relationship between the safety alarm score and response-level ASR. Correlation with Jailbreak Success. For fuzzing, however, separating successful and failed samples on average is not sufficient: the SafetyOracle should provide a continuous ranking signal that identifies which candidates are more promising. We therefore further analyze the relationship between SalarmS_alarm and response-level ASR. We focus on Llama-3.1-8B-Instruct, DeepSeek-R1-14B, and Gemma-3-4B-it, which provide sufficient successful responses for interval-level analysis. For each model, we sort the 4,000 wrapped inputs by their alarm scores, divide them into ten ordered intervals, and compute the ASR within each interval. Figure 2 reports the resulting ASR with 95% confidence intervals. A clear inverse relationship appears across all three models: intervals with lower safety alarm scores consistently exhibit higher jailbreak success rates. The Spearman coefficients are −0.964-0.964 for Llama-3.1-8B-Instruct, −0.927-0.927 for DeepSeek-R1-14B, and −0.976-0.976 for Gemma-3-4B-it. These strong negative correlations show that SalarmS_alarm provides an ordering over candidate jailbreak sensitivity rather than only a binary safety prediction. In other words, even when multiple candidates would receive the same response-level failure label, their alarm scores can still indicate which candidates are closer to successful jailbreaks. 5.2 EO2: Fuzzing Effectiveness and Efficiency To validate EO2, we evaluate NeuronFuzz on the StrongREJECT described in Section 4.2. For each harmful payload, we run an independent fuzzing process with a budget of 20 mutation iterations (as shown in Figure 3, by iteration 20, the ASR has stabilized and safety alarm score decreases slowly). NeuronFuzz evaluates intermediate candidates using the SafetyOracle and generates a target-model response only for final verification. The search stops when the budget is exhausted. We report the seed JDR of all 64×10064× 100 samples. We compare NeuronFuzz with four representative automated jailbreak methods, GCG [84], AutoDAN [35], PAIR [11], and LLM-Fuzzer [76]. These methods cover token-level gradient optimization, genetic prompt search, attacker-LLM refinement, and fuzzing-based template mutation. Appendix B provides the complete configurations of baseline methods. Table 3: Effectiveness and efficiency of NeuronFuzz on different source models. Best results are shown in bold, and second-best results are underlined. Target Model Method JDR ↑ RGD ↓ ETD (s) ↓ DeepSeek-R1-14B Seed 18% – – GCG 19% 1.0 587.2 AutoDAN 47% 14.7 470.3 PAIR 94% 15.6 43.4 LLM-Fuzzer 93% 179.6 114.5 NeuronFuzz 100% 1.0 64.1 Llama-3.1- 8B-Instruct Seed 25% – – GCG 91% 1.0 621.7 AutoDAN 86% 31.8 557.2 PAIR 81% 17.2 51.4 LLM-Fuzzer 97% 214.7 97.7 NeuronFuzz 100% 1.0 50.7 Gemma-3-4B-it Seed 36% – – GCG 97% 1.0 634.1 AutoDAN 94% 22.3 541.6 PAIR 100% 17.8 53.7 LLM-Fuzzer 100% 183.9 106.8 NeuronFuzz 100% 1.0 61.8 Gemma-4-E4B-it Seed 4% – – GCG 28% 1.0 612.9 AutoDAN 21% 53.8 635.1 PAIR 37% 22.3 58.5 LLM-Fuzzer 49% 243.6 124.7 NeuronFuzz 76% 1.0 56.3 GPT-OSS-20B Seed 0% – – GCG 0% – – AutoDAN 19% 45.2 725.2 PAIR 43% 18.6 44.7 LLM-Fuzzer 48% 304.1 133.6 NeuronFuzz 96% 1.0 72.5 As shown in Table 3, NeuronFuzz achieves the highest JDR on all five source models. It reaches 100% JDR on Llama-3.1-8B-Instruct, DeepSeek-R1-14B, and Gemma-3-4B-it. On the two more strongly aligned models, it achieves 76% JDR on Gemma-4-E4B-it and 96% on GPT-OSS-20B, exceeding the evaluated methods by 27 and 48 percentage points, respectively. On Gemma-3-4B-it, NeuronFuzz, PAIR, and LLM-Fuzzer all achieve 100% JDR, while NeuronFuzz requires fewer response generations. The advantage is particularly clear on Gemma-4-E4B-it and GPT-OSS-20B, where the initial seed templates achieve only 4% and 0% JDR, respectively. Response-level fuzzers rely on discrete success or failure feedback from generated responses. When successful generations are rare, this binary signal provides little information for distinguishing promising failed candidates, which limits the effectiveness of subsequent seed selection and mutation. GCG and AutoDAN avoid response-level feedback by optimizing the likelihood of affirmative response prefixes. However, this surrogate objective has become less reliable on recent aligned and reasoning models, which may produce affirmative prefixes or extended reasoning without ultimately generating genuinely harmful content [48, 83, 33, 74]. In contrast, the SafetyOracle assigns a continuous score to every candidate before response generation. It therefore provides informative feedback even when none of the current candidates produces a successful jailbreak, helping NeuronFuzz maintain high discovery coverage on these more aligned models. NeuronFuzz maintains an RGD of 1.0 as it evaluates intermediate candidates through prefill-only SafetyOracle scoring and generates the response only for final verification. GCG also achieves an RGD of 1.0 when successful, but its suffix-level optimization requires substantially more computation and performs poorly on the more strongly aligned models. AutoDAN requires between 14.7 and 53.8 responses per discovery, PAIR requires between 15.6 and 22.3, and LLM-Fuzzer requires between 179.6 and 304.1. These results show that the SafetyOracle removes the need to repeatedly generate target-model responses as mutation feedback. This reduction in response generation also leads to end-to-end time efficiency. NeuronFuzz achieves the lowest average ETD on Llama-3.1-8B-Instruct and Gemma-4-E4B-it, and the second-lowest ETD on the other three models. PAIR has a lower conditional ETD on successfully jailbroken questions for DeepSeek-R1-14B and GPT-OSS-20B, but its JDR is lower by 6 and 53 percentage points, respectively. GCG and AutoDAN require several hundred seconds per discovery because they repeatedly evaluate large token-level or population-level candidate sets. NeuronFuzz additionally incurs a small one-time offline cost to construct the SafetyOracle. As shown in Appendix C, this process takes only 18 minutes and 32 seconds on average and can be further used in subsequent testing. Overall, NeuronFuzz improves jailbreak discovery coverage, greatly reduces complete response generation, and maintains low end-to-end search cost with only a modest one-time preparation overhead. 5.3 EO3: Universal Template Optimization Figure 3: ASR and EASR versus the safety alarm score during 20 fuzzing steps. Section 5.2 optimizes a separate jailbreak template for each harmful payload. We next evaluate whether NeuronFuzz can optimize universal templates that generalize across payloads. For each source model, we optimize shared templates over a disjoint set of 100 harmful payloads [58] for 20 iterations, using the mean SafetyOracle safety alarm score across payloads as the objective. Following prior work [76], we report ASR for the lowest-scoring template and EASR for the five lowest-scoring templates. This setting evaluates whether NeuronFuzz can discover a small set of reusable jailbreak templates. Figure 3 shows the optimization trajectories. Across all models, decreases in the mean safety alarm score are accompanied by consistent improvements in ASR and EASR, showing that the SafetyOracle provides an effective optimization signal for template search. The EASR reaches at least 90% on four models and 82% on Gemma-4-E4B-it, indicating that a small set of optimized templates can cover a large fraction of harmful payloads. Most gains are achieved within the early iterations, and both attack performance and the safety alarm score become relatively stable near the end of the 20-iteration budget. Overall, these results show that NeuronFuzz can efficiently optimize reusable jailbreak templates that generalize across different harmful payloads on the same source model. Table 4: StrongREJECT rubric evaluation across models. Model StrongREJECT Conv. Spec. DeepSeek-R1-14B 0.647 4.32 4.23 Llama-3.1-8B-Instruct 0.559 4.42 4.26 Gemma-3-4B-it 0.714 4.28 4.36 Gemma-4-E4B-it 0.425 4.24 4.18 GPT-OSS-20B 0.445 4.11 4.13 Average 0.558 4.27 4.23 StrongREJECT Rubric. Prior work has shown that some jailbreaks achieve non-refusal without eliciting substantive harmful information [58]. We further evaluate the quality of successful responses using StrongREJECT rubric, which measures convincingness and specificity in addition to refusal. Appendix D provides the complete scoring setup. As shown in Table 4, the average StrongREJECT score is 0.558, with convincingness and specificity scores of 4.27 and 4.23 out of 5, respectively. These results indicate that the templates discovered by NeuronFuzz elicit substantive harmful responses rather than merely bypassing refusal. 5.4 EO4: Cross-Model Transferability To answer EO4, we jointly optimize jailbreak templates on Llama-3.1-8B-Instruct and GPT-OSS-20B (Appendix F justifies this choice). We follow the fuzzing procedure described in Section 5.3 and run the search for 160 iterations (where the safety alarm scores have converged). During each iteration, both source-model oracles evaluate each candidate, and we use their mean safety alarm score as feedback for seed selection and mutation. After optimization, we directly apply the resulting templates to the evaluation models without further mutation or model-specific tuning. We repeat each evaluation five times with a generation temperature of 1.0 and report the mean and standard deviation of top-1 ASR and top-5 EASR. Table 5: Universal-template transfer evaluation across target models over five runs. Target Model Seeds ASR EASR DeepSeek-R1-14B 4%4\% 79.0%±1.9%79.0\%_ ± 1.9\% 98.6%±1.1%98.6\%_ ± 1.1\% Gemma-3-4B-it 8%8\% 85.8%±2.3%85.8\%_ ± 2.3\% 98.2%±1.3%98.2\%_ ± 1.3\% Gemma-4-E4B-it 0%0\% 43.8%±2.2%43.8\%_ ± 2.2\% 70.4%±3.0%70.4\%_ ± 3.0\% Llama-3.2-3B-Instruct 2%2\% 76.0%±3.5%76.0\%_ ± 3.5\% 98.6%±1.7%98.6\%_ ± 1.7\% Phi-4 0%0\% 70.8%±2.5%70.8\%_ ± 2.5\% 91.4%±3.5%91.4\%_ ± 3.5\% Qwen3.6-35B-A3B 0%0\% 67.6%±3.5%67.6\%_ ± 3.5\% 98.0%±1.6%98.0\%_ ± 1.6\% Qwen3.6-27B 0%0\% 63.2%±2.3%63.2\%_ ± 2.3\% 90.2%±3.6%90.2\%_ ± 3.6\% GLM-4.7-Flash 0%0\% 70.4%±3.4%70.4\%_ ± 3.4\% 95.0%±1.6%95.0\%_ ± 1.6\% Inference Average – 69.6% 92.6% LongCat-2.0 0%0\% 66.8%±4.1%66.8\%_ ± 4.1\% 95.6%±2.9%95.6\%_ ± 2.9\% DeepSeek-V4-Flash 0%0\% 53.0%±2.0%53.0\%_ ± 2.0\% 71.2%±1.5%71.2\%_ ± 1.5\% DeepSeek-V4-Pro 0%0\% 17.6%±2.4%17.6\%_ ± 2.4\% 28.8%±1.5%28.8\%_ ± 1.5\% Grok-4.5 0%0\% 17.2%±0.4%17.2\%_ ± 0.4\% 28.6%±2.3%28.6\%_ ± 2.3\% Gemini-2.5-Pro 0%0\% 85.2%±1.9%85.2\%_ ± 1.9\% 98.2%±2.2%98.2\%_ ± 2.2\% Gemini-3-Flash 0%0\% 24.6%±1.5%24.6\%_ ± 1.5\% 37.4%±1.3%37.4\%_ ± 1.3\% Proprietary Average – 44.1% 60.0% Table 5 reports the cross-model transfer results. Harmful queries without jailbreak templates achieve at most 8% ASR and fail completely on 11 of the 14 models. In contrast, the optimized templates achieve transfer without any target-specific optimization. Transfer is particularly strong across the eight inference-only target models. The optimized top-1 template achieves an average ASR of 69.6%, while the top-5 template set reaches an average EASR of 92.6%. Seven of the eight models achieve EASR above 90%, including 98.6% on Llama-3.2-3B-Instruct and 98.0% on Qwen3.6-35B-A3B. Strong transfer is observed across different model families, parameter scales, reasoning modes, and both dense and MoE architectures. These results show that the optimized templates transfer well beyond the source models and remain effective across different target LLMs. The optimized templates also transfer to proprietary models. Across the six API models, the average ASR and EASR are 44.1% and 60.0%, respectively. Gemini-2.5-Pro shows the strongest transfer, reaching 85.2% ASR and 98.2% EASR. The templates also achieve 66.8% ASR and 95.6% EASR on LongCat-2.0 and 53.0% ASR and 71.2% EASR on DeepSeek-V4-Flash. Transfer is weaker on DeepSeek-V4-Pro, Grok-4.5, and Gemini-3-Flash, but all three models still exhibit non-zero ASR and EASR despite receiving no target-specific optimization. Overall, these results show that templates optimized using white-box surrogate models can retain meaningful effectiveness when transferred directly to proprietary models accessible only through their input-output interfaces. We also evaluate cross-dataset transfer on HarmBench in Appendix G, which retains 67.0% ASR and 87.3% EASR. Table 6: Transferability of NeuronFuzz on VLMs. Target Model Baseline NeuronFuzz ASR / EASR T2I NSFW T2I NSFW Qwen2.5-VL-3B-Instruct 2% 6% 73% / 93% 88% / 100% Qwen2.5-VL-7B-Instruct 0% 4% 65% / 81% 63% / 89% Qwen3.5-35B-A3B 1% 0% 41% / 72% 69% / 82% Kimi-VL-A3B-Instruct 5% 8% 86% / 100% 83% / 100% Gemma-4-31B-it 0% 0% 57% / 83% 71% / 94% Average 1.6% 3.6% 64.4% / 85.8% 74.8% / 93.0% Transferability on Different Modalities. We further evaluate whether the templates optimized by NeuronFuzz transfer to multimodal safety tasks. We consider five Visual Language Models (VLMs): Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct [65], Qwen3.5-35B-A3B [49], Kimi-VL-A3B-Instruct [60], and Gemma-4-31B-it [59]. We reuse the templates optimized in the previous text-only fuzzing without further adaptation. We evaluate two visual tasks. For text-to-image (T2I), we randomly sample 100 harmful instructions from HarmBench [38] and convert each instruction into an image following prior works [19, 34]. The accompanying text prompt is “Answer the question based on the image.” For NSFW evaluation, we sample 100 unsafe images from the NSFW-1024 dataset [75] and use “Describe the image.” as the payload. The baseline uses the same images and payloads without a jailbreak template. As shown in Table 6, the baseline ASR is low across all models, averaging only 1.6% on T2I and 3.6% on NSFW. After adding the transferred templates, the average ASR increases to 64.4% on T2I and 74.8% on NSFW, while the corresponding EASR reaches 85.8% and 93.0%, respectively. These results show that templates optimized from text-only internal safety signals can transfer to visual tasks without access to the target VLM’s activations. The higher performance on NSFW images further suggests that the optimized templates generalize beyond textual harmful instructions to image-specific unsafe content. 6 Ablation Study 6.1 Safety-Neuron Construction We examine the two designs used to construct the safety-neuron set: template-invariant activation extraction (Section 3.3.1) and stability-aware neuron selection (Section 3.3.2). We conduct the ablation on Llama-3.1-8B-Instruct while keeping other configurations fixed. Prompt-pair only extracts activations from harmful and benign prompts without introducing jailbreak-template variations. Z-score replaces stability selection with statistic-based ranking method [67]. Random selects the same number of neurons uniformly at random from the corresponding layers. We report clean-prompt AUROC, the Spearman correlation between the safety alarm score and ASR, and the downstream JDR. Table 7: Ablation of safety-neuron extraction and selection. Extraction Selection AUROC ↑ ↓ ρ JDR ↑ Prompt-pair only Stability 0.834 -0.437 74% Template-invariant Z-score 0.974 -0.737 82% Template-invariant Random 0.682 0.121 16% Template-invariant Stability 0.989 -0.964 100% Table 7 shows that the full NeuronFuzz achieves the best performance across all three metrics. Removing template-invariant extraction reduces AUROC from 0.989 to 0.834, weakens the correlation from −0.964-0.964 to −0.437-0.437, and decreases JDR from 100% to 74%. This indicates that neurons identified only from clean prompts do not generalize well to harmful inputs under jailbreak contexts, while including template variations helps preserve informative safety signals across different prompt contexts. Replacing stability selection with Z-score ranking retains a high clean-prompt AUROC of 0.974, but reduces the correlation to −0.737-0.737 and JDR to 82%. This suggests that classification accuracy alone does not guarantee a useful fuzzing signal. Stability-aware selection favors neurons whose associations remain consistent across training samples, producing a score that better ranks jailbreak candidates. Random neuron selection performs substantially worse, with an AUROC of 0.682, a positive correlation of 0.121, and only 16% JDR. Overall, template-invariant extraction improves robustness to jailbreak contexts, while stability selection provides a more reliable signal for fuzzing. 6.2 Oracle-Guided Fuzzing We examine the roles of continuous oracle feedback and gradient-guided mutation on well-aligned GPT-OSS-20B. The w/o gradient guidance variant retains continuous SafetyOracle scoring but selects mutation positions randomly; the binary feedback variant replaces SafetyOracle with response-level success labels and follows the ranking strategy of LLM-Fuzzer. Since binary feedback is non-differentiable, it also uses random token positions. All variants use the same seeds, mutation model, and candidate budget. We use 100 iterations instead of the default 20 to provide the weakened variants with sufficient opportunity to discover successful mutations. Table 8: Ablation of continuous oracle feedback and gradient-guided mutation on GPT-OSS-20B with 100 iterations. Variant Search Feedback Token Position JDR ↑ RGD ↓ W/o gradient guidance Continuous Random 21% 1.0 Binary feedback Binary Random 4% 18.3 Full NeuronFuzz Continuous Gradient 99% 1.0 As shown in Table 8, the full method reaches 99% JDR, while removing gradient guidance reduces JDR to 21%. Both variants maintain an RGD of 1.0 because intermediate candidates are evaluated through prefill-only oracle scoring. This gap shows that the continuous alarm score supports not only candidate ranking but also sensitive-token localization through its gradients. Without a differentiable score, token-level sensitivity cannot be estimated. The random-position variant still achieves 21% JDR, reflecting the sensitivity of LLMs to small input changes [53, 54] where random substitutions occasionally create unusual local patterns or shift the prompt away from the original template distribution, thereby weakening refusal behavior [6]. However, such useful mutations are infrequent, whereas gradient guidance systematically targets positions that most strongly affect the internal safety signal. Replacing the continuous oracle with binary feedback further reduces JDR to 4% and increases RGD to 18.3. Because successful responses are near-zero on GPT-OSS-20B, most candidates receive the same failure label, providing little information for mutation. In contrast, the SafetyOracle assigns a continuous score before decoding, allowing NeuronFuzz to distinguish and retain promising candidates even before a successful jailbreak is observed. 7 Defenses 7.1 Perplexity Filter & SmoothLLM A perplexity filter rejects inputs that are unlikely under a reference language model. Following [5], we use GPT-2 and reject prompts whose perplexity exceeds the selected threshold. SmoothLLM generates perturbed copies of each prompt and aggregates their response-level decisions [53]. We use the Swap perturbation with a perturbation ratio of 10% and generate 10 copies for each prompt. Table 9: NeuronFuzz under input defenses. Model No Defense Perplexity SmoothLLM DeepSeek-R1-14B 100% 98%−2%98\%_ [rgb]1,0,0-2\% 74%−26%74\%_ [rgb]1,0,0-26\% Llama-3.1-8B-Instruct 100% 95%−5%95\%_ [rgb]1,0,0-5\% 71%−29%71\%_ [rgb]1,0,0-29\% Gemma-3-4B-it 100% 92%−8%92\%_ [rgb]1,0,0-8\% 75%−25%75\%_ [rgb]1,0,0-25\% Gemma-4-E4B-it 76% 72%−4%72\%_ [rgb]1,0,0-4\% 65%−11%65\%_ [rgb]1,0,0-11\% GPT-OSS-20B 96% 91%−5%91\%_ [rgb]1,0,0-5\% 79%−17%79\%_ [rgb]1,0,0-17\% As shown in Table 9, the perplexity filter causes only a small reduction in the effectiveness of NeuronFuzz. Across the five models, JDR decreases by only 2–8%. This robustness arises because NeuronFuzz uses the masked language model to replace selected tokens with context-compatible alternatives while preserving the length and grammatical structure of the original template. Consequently, the optimized templates remain fluent and do not rely on anomalous high-perplexity sequences [84]. SmoothLLM provides a stronger defense, reducing JDR by 11–29%. Character-level swaps can disrupt important words and local structures within the optimized templates, making this defense more effective than perplexity filtering. Nevertheless, NeuronFuzz retains JDRs of 71%, 74%, 75%, 65%, and 79% across the five models. Since NeuronFuzz modifies only a small number of sensitive token positions, randomly selected character perturbations may not consistently disrupt the key structures responsible for the jailbreak. Overall, the results show that input perturbation reduces attack effectiveness but does not fully eliminate the vulnerabilities exposed by NeuronFuzz. We also compare with other methods under the input defenses in Appendix H where NeuronFuzz achieves the highest JDR in both settings. 7.2 Input-Side Jailbreak Detection To further show the use of SafetyOracle, we use it as an input-side detector for jailbreak prompts. The results in Section 5.1 reveal three distinct alarm-score patterns. Benign prompts usually receive scores close to zero, while direct harmful prompts receive scores close to one. Successful jailbreak prompts often fall between these two regions, suggesting that the surrounding jailbreak template weakens, but does not fully remove, the model’s internal recognition of harmful intent. Based on this observation, we define the jailbreak detection score as Djb(x)=1−|2Salarm(x)−1|∈[0,1].D_jb(x)=1- |2S_alarm(x)-1 |∈[0,1]. The score reaches its maximum when Salarm(x)=0.5S_alarm(x)=0.5 and decreases as the alarm score approaches either zero or one. Therefore, a higher Djb(x)D_jb(x) indicates that the input is closer to the intermediate alarm-score region associated with successful jailbreaks. We evaluate the detector using the 4,000 test jailbreak inputs and 4,000 benign and harmful requests described in Section 4.2. We use the response-level evaluation pipeline to assign the ground-truth label, treating successful jailbreaks as positive samples, where normal benign requests and failed jailbreak attempts as negative samples. Table 10: Performance of SafetyOracle for detecting successful jailbreak prompts. TPR is reported at a fixed FPR of 5%. Target Model AUROC Precision F1 TPR@5%FPR Llama-3.1-8B-Instruct 0.957 0.884 0.892 88.4% DeepSeek-R1-14B 0.948 0.867 0.876 86.1% Gemma-3-4B-it 0.963 0.896 0.901 89.3% Average 0.956 0.882 0.890 87.9% As shown in Table 10, the detector achieves AUROC values between 0.948 and 0.963 across the three models. Its average precision and F1 score are 0.882 and 0.890, respectively. At a fixed false-positive rate of 5%, it detects 87.9% of successful jailbreak prompts on average. These results show that the SafetyOracle provides a useful input-side signal for identifying likely successful jailbreaks before response generation. Unlike response-level judges, which require complete target-model outputs, the SafetyOracle detector uses only a prefill pass. Therefore, SafetyOracle can serve as a lightweight and convenient detector that provides useful signals for subsequent verification. 8 Conclusion We presented NeuronFuzz, a white-box fuzzing framework that uses internal neuron activations as continuous feedback for LLM safety evaluation. By constructing a lightweight SafetyOracle and using its differentiable safety alarm score to guide candidate ranking and targeted template mutation, NeuronFuzz reduces reliance on repeated response generation while preserving the harmful payload and overall prompt semantics with strong transferability. Our evaluation shows that internal safety representations provide informative signals for jailbreak discovery and support effective generalization across different payloads, models, and modalities. Beyond fuzzing, the SafetyOracle can also serve as an input-side detector for identifying likely jailbreak prompts before response generation. Overall, our findings demonstrate that internal model signals can serve as practical execution feedback for LLM fuzzing while also providing useful defensive signals for systematic safety evaluation. References [1] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. (2024) Phi-4 technical report. arXiv preprint arXiv:2412.08905. Cited by: Table 1. [2] S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, et al. (2025) Gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Cited by: Table 1. [3] M. AI (2024) Llama 3.2: revolutionizing edge ai and vision with open, customizable models. External Links: Link Cited by: Table 1. [4] A. Albert (2023) Jailbreak chat. External Links: Link Cited by: Appendix A, §4.1. [5] G. Alon and M. Kamfonas (2023) Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132. Cited by: §7.1. [6] M. Andriushchenko N. Flammarion et al. (2025) Jailbreaking leading safety-aligned llms with simple adaptive attacks. In International Conference on Learning Representations, Vol. 2025, p. 40116–40143. Cited by: §2.1, §6.2. [7] A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda (2024) Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems 37, p. 136037–136083. Cited by: §2.1. [8] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo (2014) The oracle problem in software testing: a survey. IEEE transactions on software engineering 41 (5), p. 507–525. Cited by: §2.2. [9] M. Böhme, V. Pham, and A. Roychoudhury (2016) Coverage-based greybox fuzzing as markov chain. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, p. 1032–1043. Cited by: §2.2. [10] L. Bürger, F. A. Hamprecht, and B. Nadler (2024) Truth is universal: robust detection of lies in llms. Advances in Neural Information Processing Systems 37, p. 138393–138431. Cited by: §3.3.3. [11] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2025) Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), p. 23–42. Cited by: Appendix B, Appendix B, Table 16, §1, §2.1, §2.2, §3.4.3, §5.2. [12] H. Chen, Y. Xue, Y. Li, B. Chen, X. Xie, X. Wu, and Y. Liu (2018) Hawkeye: towards a desired directed grey-box fuzzer. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, p. 2095–2108. Cited by: §2.2. [13] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: Table 1. [14] M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin (2023)Free dolly: introducing the world’s first truly open instruction-tuned llm(Website) External Links: Link Cited by: §4.2. [15] S. Dechand, T. Wienand, F. Meumertzheim, P. Samarin, S. Resch, K. Yakdan, T. Holz, and F. Toffalini (2026) Jazzer: coverage-guided fuzzing for semantic vulnerabilities in the java ecosystem. In 2026 IEEE Symposium on Security and Privacy (SP), p. 3720–3739. Cited by: §2.2. [16] X. Du, C. Xiao, and Y. Li (2024) Haloscope: harnessing unlabeled llm generations for hallucination detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §3.3.3. [17] J. H. Friedman, T. Hastie, and R. Tibshirani (2010) Regularization paths for generalized linear models via coordinate descent. Journal of statistical software 33, p. 1–22. Cited by: §3.3.3. [18] Gemma Team (2025) Gemma 3 technical report. Technical Report Google DeepMind. External Links: 2503.19786, Link Cited by: Table 1. [19] Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang (2025) Figstep: jailbreaking large vision-language models via typographic visual prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 23951–23959. Cited by: §5.4. [20] Google DeepMind (2025) Gemini 3 Flash: model card. Model Card Google DeepMind. External Links: Link Cited by: Table 1. [21] Google DeepMind (2026) Gemini 3 Pro: model card. Model Card Google DeepMind. External Links: Link Cited by: §2.1. [22] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.3, Table 1. [23] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. Cited by: Table 1. [24] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Hosking (2021) Seed selection for successful fuzzing. In Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, p. 230–243. Cited by: §4.1. [25] J. Hughes, S. Price, A. Lynch, R. Schaeffer, F. Barez, A. Somani, S. Koyejo, H. Sleight, E. Jones, E. Perez, et al. (2026) Best-of-n jailbreaking. Advances in Neural Information Processing Systems 38, p. 73137–73221. Cited by: §1, §2.1. [26] A. K. Kamath, R. Prabhu, J. Mohan, S. Peter, R. Ramjee, and A. Panwar (2025) Pod-attention: unlocking full prefill-decode overlap for faster llm inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, p. 897–912. Cited by: §1. [27] H. Kazemi, A. Chegini, and M. Safi (2026) A single neuron is sufficient to bypass safety alignment in large language models. arXiv preprint arXiv:2605.08513. Cited by: §2.1, §3.1. [28] U. M. Khaire and R. Dhanalakshmi (2022) Stability of feature selection algorithm: a review. Journal of King Saud University-Computer and Information Sciences 34 (4), p. 1060–1073. Cited by: §4.1. [29] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). In International conference on machine learning, p. 2668–2677. Cited by: §3.3.2. [30] A. Köpf, Y. Kilcher, D. Von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, et al. (2023) Openassistant conversations-democratizing large language model alignment. Advances in neural information processing systems 36, p. 47669–47681. Cited by: §4.2. [31] C. Lemieux and K. Sen (2018) Fairfuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, p. 475–485. Cited by: §2.2. [32] Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. In International conference on machine learning, p. 19274–19286. Cited by: §1. [33] J. Li and J. Kim (2025) Safety alignment can be not superficial with explicit safety signals. arXiv preprint arXiv:2505.17072. Cited by: §5.2. [34] D. Liu, M. Yang, X. Qu, P. Zhou, Y. Cheng, and W. Hu (2025) A survey of attacks on large vision–language models: resources, advances, and future trends. IEEE transactions on neural networks and learning systems. Cited by: §5.4. [35] X. Liu, N. Xu, M. Chen, and C. Xiao (2024) Autodan: generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations, Vol. 2024, p. 56174–56194. Cited by: Appendix B, Appendix B, Table 16, §1, §2.1, §2.2, §3.2, §5.2. [36] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu (2023) Jailbreaking chatgpt via prompt engineering: an empirical study. arXiv preprint arXiv:2305.13860. Cited by: Appendix A, §1, §2.1, §4.1. [37] V. J. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo (2019) The art, science, and engineering of fuzzing: a survey. IEEE Transactions on Software Engineering 47 (11), p. 2312–2331. Cited by: §1, §2.2. [38] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks (2024) HarmBench: a standardized evaluation framework for automated red teaming and robust refusal. External Links: 2402.04249 Cited by: Appendix G, §5.4. [39] L. Mei, S. Liu, Y. Wang, B. Bi, J. Mao, and X. Cheng (2025) “Not aligned” is not “malicious”: being careful about hallucinations of large language models’ jailbreak. In Proceedings of the 31st International Conference on Computational Linguistics, p. 2144–2162. Cited by: Appendix D. [40] N. Meinshausen and P. Bühlmann (2010) Stability selection. Journal of the Royal Statistical Society Series B: Statistical Methodology 72 (4), p. 417–473. Cited by: §3.3.2, §3.3.3, §4.1. [41] Meituan LongCat (2026)Introducing LongCat-2.0(Website) External Links: Link Cited by: Table 1. [42] K. Nikolić, L. Sun, J. Zhang, and F. Tramèr (2025) The jailbreak tax: how useful are your jailbreak outputs?. arXiv preprint arXiv:2504.10694. Cited by: Appendix D. [43] OpenAI (2026)Evaluations with challenging prompts(Website) External Links: Link Cited by: §2.1. [44] OpenAI (2026)GPT-5.4 Thinking system card(Website) External Links: Link Cited by: Appendix B. [45] OpenAI (2026)GPT-5.5 system cardSystem Card OpenAI. External Links: Link Cited by: Appendix D. [46] OpenRouterOpenRouter: the unified interface for LLMs(Website) External Links: Link Cited by: §4.2. [47] K. Park, Y. J. Choe, and V. Veitch (2023) The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. Cited by: §2.1. [48] X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson (2025) Safety alignment should be made more than just a few tokens deep. In International Conference on Learning Representations, Vol. 2025, p. 54911–54941. Cited by: §5.2. [49] Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §5.4. [50] Qwen Team (2026) Qwen3.6-27B: flagship-level coding in a 27B dense model. External Links: Link Cited by: Table 1. [51] Qwen Team (2026) Qwen3.6-35B-A3B: agentic coding power, now open to all. External Links: Link Cited by: Table 1. [52] N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner (2024) Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15504–15522. Cited by: §2.1. [53] A. Robey, E. Wong, H. Hassani, and G. J. Pappas (2023) Smoothllm: defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. Cited by: §6.2, §7.1. [54] A. Salinas and F. Morstatter (2024) The butterfly effect of altering prompts: how small changes and jailbreaks affect large language model performance. In Findings of the Association for Computational Linguistics: ACL 2024, p. 4629–4651. Cited by: §3.4.2, §6.2. [55] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang (2024) " Do anything now": characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 1671–1685. Cited by: §1, §1, §2.1, §2.1. [56] Z. Shen, R. Roongta, and B. Dolan-Gavitt (2022) Drifuzz: harvesting bugs in device drivers from golden seeds. In 31st USENIX Security Symposium (USENIX Security 22), p. 1275–1290. Cited by: §4.1. [57] A. Sheshadri, A. Ewart, P. Guo, A. Lynch, C. Wu, V. Hebbar, H. Sleight, A. C. Stickland, E. Perez, D. Hadfield-Menell, et al. (2024) Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549. Cited by: §4.2. [58] A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer (2024) A strongreject for empty jailbreaks. External Links: 2402.10260 Cited by: Appendix D, §3.4.2, §4.2, §4.3, §5.3, §5.3. [59] G. Team (2026) Gemma 4 technical report. External Links: 2607.02770, Link Cited by: Table 1, §5.4. [60] K. Team, A. Du, B. Yin, B. Xing, B. Qu, B. Wang, C. Chen, C. Zhang, C. Du, C. Wei, et al. (2025) Kimi-vl technical report. arXiv preprint arXiv:2504.07491. Cited by: §5.4. [61] M. Thang, L. Wu, S. Behrouzi, M. Rostami, J. t. Lintelo, S. Picek, and A. Sadeghi (2026) GoodVibe: security-by-vibe for llm-based code generation. arXiv preprint arXiv:2602.10778. Cited by: §2.1, §3.3.1. [62] A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid (2023) Steering language models with activation engineering. arXiv preprint arXiv:2308.10248. Cited by: §2.1, §3.3.1. [63] B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, N. Cooper, G. Adams, J. Howard, and I. Poli (2024) Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. External Links: 2412.13663, Link Cited by: §4.1. [64] A. Wei, N. Haghtalab, and J. Steinhardt (2023) Jailbroken: how does llm safety training fail?. Advances in neural information processing systems 36, p. 80079–80110. Cited by: §1, §1, §2.1, §2.1. [65] C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S. Yin, S. Bai, X. Xu, Y. Chen, et al. (2025) Qwen-image technical report. arXiv preprint arXiv:2508.02324. Cited by: §5.4. [66] L. Wu, S. Behrouzi, M. Rostami, S. Picek, and A. Sadeghi (2025) GateBreaker: gate-guided attacks on mixture-of-expert llms. arXiv preprint arXiv:2512.21008. Cited by: §2.1, §3.3.1, §4.1. [67] L. Wu, S. Behrouzi, M. Rostami, M. Thang, S. Picek, and A. Sadeghi (2025) NeuroStrike: neuron-level attacks on aligned llms. arXiv preprint arXiv:2509.11864. Cited by: §2.1, §3.1, §3.3.1, §4.1, §6.1. [68] L. Wu, M. Rostami, H. Li, J. Rajendran, and A. Sadeghi (2025) \genhuzz\: An efficient generative hardware fuzzer. In 34th USENIX Security Symposium (USENIX Security 25), p. 1787–1805. Cited by: §1. [69] xAI (2026)Introducing Grok 4.5(Website) External Links: Link Cited by: Table 1. [70] T. Xie, X. Qi, Y. Zeng, Y. Huang, U. M. Sehwag, K. Huang, L. He, B. Wei, D. Li, Y. Sheng, R. Jia, B. Li, K. Li, D. Chen, P. Henderson, and P. Mittal (2025) SORRY-bench: systematically evaluating large language model safety refusal. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §4.2. [71] A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. (2026) Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: Table 1, Table 1. [72] Z. Xu, S. Abaimov, J. Gardiner, and S. Belguith (2025) Steering in the shadows: causal amplification for activation space attacks in large language models. arXiv preprint arXiv:2511.17194. Cited by: §2.1. [73] Z. Xu, J. Gardiner, S. Belguith, and L. Wu (2026) RouteHijack: routing-aware attack on mixture-of-experts llms. arXiv preprint arXiv:2605.02946. Cited by: §2.1. [74] Z. Xu, J. Gardiner, and S. Belguith (2026) Reasoning that leaks, fine-tuning that amplifies: exposing the hidden threats of chain-of-thought models. In Proceedings of the ACM Asia Conference on Computer and Communications Security, p. 439–455. Cited by: §5.2. [75] yesidobytensfw1024(Website) Hugging Face. Note: Hugging Face dataset repository External Links: Link Cited by: §5.4. [76] J. Yu, X. Lin, Z. Yu, and X. Xing (2024) \llm-Fuzzer\: scaling assessment of large language model jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24), p. 4657–4674. Cited by: Appendix B, Appendix B, Table 16, §1, §2.1, §2.2, §3.2, §3.4.1, §3.4.3, §4.3, §4.3, §5.2, §5.3. [77] W. Yuan, J. Yu, S. Jiang, K. Padthe, Y. Li, D. Wang, I. Kulikov, K. Cho, Y. Tian, J. E. Weston, and X. Li (2025) NaturalReasoning: reasoning in the wild with 2.8m challenging questions. External Links: 2502.13124, Link Cited by: §4.2. [78] Z.ai (2025)GLM-4.7: advancing the coding capability(Website) External Links: Link Cited by: Table 1. [79] M. Zalewski (2014) Technical “whitepaper” for afl-fuzz. External Links: Link Cited by: §1, §2.2. [80] H. Zhao, C. Yuan, F. Huang, X. Hu, Y. Zhang, A. Yang, B. Yu, D. Liu, J. Zhou, J. Lin, et al. (2025) Qwen3Guard technical report. arXiv preprint arXiv:2510.14276. Cited by: §4.3. [81] J. Zhao, J. Huang, Z. Wu, D. Bau, and W. Shi (2026) Llms encode harmfulness and refusal separately. Advances in Neural Information Processing Systems 38, p. 140283–140318. Cited by: §5.1. [82] Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, K. Wang, Y. Liu, J. Fang, and Y. Li (2025) On the role of attention heads in large language model safety. In International Conference on Learning Representations, Vol. 2025, p. 84042–84071. Cited by: §5.1. [83] S. Zhu, B. Amos, Y. Tian, C. Guo, and I. Evtimov (2026) Advprefix: an objective for nuanced llm jailbreaks. Advances in Neural Information Processing Systems 38, p. 89990–90014. Cited by: §5.2. [84] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: Appendix B, Appendix B, Table 16, §3.2, §3.4.2, §5.2, §7.1. Appendix A Jailbreak Seed Templates As described in Section 4.2, we followed the sampling method used in previous work [36]. We collected jailbreak templates from a shared online repository [4], which contained 79 valid templates. However, several templates were highly similar, especially variants derived from the DAN prompt. We therefore manually removed duplicate or near-duplicate templates. We also excluded templates that directly modified the input payload, as such modifications could introduce additional variables and reduce the fairness of the evaluation. After the filtering process, we retained 74 jailbreak templates, 10 of which were held out for oracle training and the rest 64 templates were used for further fuzzing evaluation. Table 11 shows the categories of templates. Table 11: Distribution of the selected templates by category. Category Number of Templates Role-playing 33 Fictional scenarios 7 Instruction conflicts 14 Refusal suppression 4 Emotional framing 6 Total 64 Appendix B Baseline Methods We implement four representative jailbreak baselines: GCG [84], AutoDAN [35], PAIR [11], and LLM-Fuzzer [76]. We use their released implementations and preserve their main optimization procedures. All methods are evaluated on the same harmful payloads, hardware, and final response-level evaluation pipeline as NeuronFuzz. GCG. We implement GCG following the original work and its official implementation [84]. GCG appends an optimizable adversarial suffix to each harmful payload and minimizes the negative log-likelihood of a predefined affirmative completion (“Sure, here is”). We initialize the suffix with 20 repeated exclamation-mark tokens and optimize it for 500 steps. At each step, token gradients identify the top 256 replacement tokens, from which a batch of 512 candidate substitutions is evaluated. The candidate with the lowest target loss is retained for the next step. During evaluation, the optimized suffix is directly appended to the harmful payload. AutoDAN. We use the hierarchical genetic algorithm version of AutoDAN [35]. AutoDAN initializes a population from handcrafted jailbreak prompts and evolves them through hierarchical crossover and mutation. Candidates are ranked by the negative log-likelihood of an affirmative target response. Following the original configuration, we use a population size of 256, a crossover rate of 0.5, a mutation rate of 0.01, an elite rate of 0.1, and five crossover points. We run the optimization for 100 iterations, corresponding to up to 25,600 population-level candidate evaluations. The candidate with the lowest optimization loss is retained for final evaluation. PAIR. We implement PAIR using its released framework and original search configuration [11]. PAIR uses an attacker LLM to iteratively refine a jailbreak prompt based on the target model’s response and evaluation feedback. We use Mixtral-8x7B-Instruct as the attacker model with a temperature of 1.0 and top-p of 0.9. We run 30 parallel conversation streams with a maximum depth of three iterations, resulting in at most 90 target-model queries per payload. The search stops early when a successful jailbreak is identified or when all streams reach the maximum depth. LLM-Fuzzer. We implement LLM-Fuzzer using its released framework and default MCTS-Explore seed-selection strategy [76]. For a fair comparison, we initialize it with the same 64 jailbreak templates used by NeuronFuzz. At each iteration, one seed is selected and mutated using one of five LLM-based operators: generate, crossover, expand, shorten, or rephrase. We use gpt-5.4-mini-2026-03-17 [44] as the mutation model. Each generated template is combined with the same evaluation payloads, and the target-model responses are evaluated to update the MCTS-Explore tree. We run LLM-Fuzzer for 500 mutation iterations. Appendix C Computation Statement All experiments were conducted on a high-performance computing cluster equipped with four NVIDIA GH200 nodes. Each node provides one NVIDIA H100 GPU with 120 GB of memory. Most experiments were run on a single node, while large-scale evaluations were distributed across multiple nodes to reduce the overall runtime. For black-box API evaluation, all models were accessed through OpenRouter11 1 https://openrouter.ai/. All locally deployed models were obtained from Hugging Face22 2 https://huggingface.co/and evaluated using their unquantized model weights. C.1 SafetyOracle Construction Cost Table 12 reports the one-time offline cost of constructing the SafetyOracle for each source model. The process includes activation extraction, stability-aware neuron selection, and fitting the final Elastic Net classifier. All measurements report wall-clock time on a single NVIDIA GH200 node. Table 12: One-time offline cost of constructing the SafetyOracle for each source model. Source Model Activation Extraction Stability Selection Oracle Fitting Total Time Llama-3.1-8B-Instruct 5m59s 10m03s 1m11s 17m13s DeepSeek-R1-14B 6m04s 9m56s 58s 16m58s Gemma-3-4B-it 11m54s 10m05s 39s 22m38s Gemma-4-E4B-it 4m17s 9m37s 45s 14m39s GPT-OSS-20B 9m38s 10m01s 1m31s 21m10s Average 7m34s 9m56s 1m01s 18m32s Constructing the complete SafetyOracle requires between 14m39s and 22m38s across the five source models, with an average cost of 18m32s. Activation extraction takes 7m34s on average, while stability selection takes 9m56s. In contrast, fitting the final Elastic Net classifier requires only 1m01s on average, ranging from 39s to 1m31s. This result shows that the learned scoring head itself is lightweight, and most of the one-time cost comes from collecting and selecting activation features. Once constructed, the same SafetyOracle is reused across all harmful payloads, seed templates, and fuzzing runs for the corresponding source model. It does not require further training or adaptation during online fuzzing. Therefore, unlike response-level methods that repeatedly generate complete responses and may invoke external judges for each candidate, NeuronFuzz pays this preparation cost only once and amortizes it over subsequent evaluations. The low fitting cost and reusable design make the SafetyOracle practical for repeated safety testing of a fixed source model. Appendix D StrongREJECT Rubric Binary ASR may overestimate jailbreak effectiveness because some non-refusal responses are vague, irrelevant, or provide little useful information [39, 42]. We therefore also report the StrongREJECT rubric score [58]. StrongREJECT first determines whether the response refuses the request. For non-refusal responses, it rates convincingness and specificity from 1 to 5. The final score for example i is SRi=(1−ri)ci+si−28∈[0,1],SR_i=(1-r_i) c_i+s_i-28∈[0,1], (8) where ri=1r_i=1 indicates refusal, and cic_i and sis_i denote the convincingness and specificity scores. Refused responses receive zero, while a high score requires the response to be both convincing and specific. We use GPT-5.5 [45] as the evaluator with the original StrongREJECT rubric and report the average score across evaluated responses. Appendix E SafetyOracle Probe Model Selection The SafetyOracle converts the selected safety-neuron activations into a continuous safety alarm score. Since this score is used to rank candidates and guide gradient-based mutation, the choice of probe model directly affects fuzzing performance. We therefore conduct an ablation comparing Elastic Net logistic regression with four alternative probe models on Llama-3.1-8B-Instruct, Gemma-3-4B-it, and DeepSeek-R1-14B. We keep the selected safety-neuron features and evaluation protocol unchanged and vary only the probe model. We evaluate each probe using the same response-level analysis as in Section 5.1. Specifically, we measure the Spearman correlation between the probe score and response-level ASR. Since a lower safety alarm score should correspond to a higher probability of jailbreak success, a stronger negative correlation indicates a more useful continuous ranking signal for fuzzing. Probe Configurations. We evaluate five probe models. Linear Probe uses a single linear layer trained with BCEWithLogitsLoss and Adam for 500 epochs, with a learning rate of 10−310^-3 and weight decay of 10−310^-3. Elastic Net LR uses scikit-learn logistic regression with the SAGA solver, C=1.0C=1.0, an ℓ1 _1 ratio of 0.5, a maximum of 5,000 iterations, and a convergence tolerance of 10−410^-4. L2 Logistic Regression uses the LBFGS solver with C=1.0C=1.0, 5,000 maximum iterations, and the same tolerance. Linear SVM uses LinearSVC with C=1.0C=1.0, dual=auto, 5,000 maximum iterations, and a tolerance of 10−410^-4. Finally, Small MLP contains a linear layer, ReLU activation, dropout, and an output linear layer. We use a hidden dimension of 128, dropout of 0.1, a validation fraction of 0.15, early-stopping patience of 30, and AdamW optimization. Table 13: Comparison of probe models for the SafetyOracle. Probe Model Llama-3.1- 8B-Instruct Gemma-3- 4B-it DeepSeek- R1-14B Mean |ρ||ρ| Small MLP -0.9337 -0.8139 -0.9303 0.893 Linear SVM -0.7148 -0.6018 -0.4424 0.586 Elastic Net LR -0.9636 -0.9758 -0.9273 0.956 L2 Logistic Regression -0.6907 -0.5976 -0.4377 0.575 Linear Probe -0.6284 -0.5636 -0.7337 0.642 As shown in Table 13, Elastic Net logistic regression achieves the strongest negative correlation on all three models. Its performance remains consistently high across the three models, with Spearman coefficients of −0.9636-0.9636 on Llama-3.1-8B-Instruct, −0.9758-0.9758 on Gemma-3-4B-it, and −0.9273-0.9273 on DeepSeek-R1-14B. The small MLP is the strongest alternative, achieving a mean |ρ||ρ| of 0.893. It performs similarly to Elastic Net on Llama-3.1-8B-Instruct and DeepSeek-R1-14B, but its correlation decreases to −0.8139-0.8139 on Gemma-3-4B-it. In contrast, Elastic Net maintains a correlation of −0.9758-0.9758 on the same model. The remaining linear probes show substantially weaker correlations, with mean |ρ||ρ| values between 0.575 and 0.642. Since the SafetyOracle is not used only for harmful-versus-benign classification. During fuzzing, its continuous score is used to rank candidates and guide the search toward prompts that are more likely to produce successful jailbreaks. Elastic Net provides the strongest and most consistent relationship between this score and response-level attack success across the evaluated models. We therefore use Elastic Net logistic regression as the default SafetyOracle model, as it achieves the strongest and most consistent correlation with response-level jailbreak success across the evaluated models. We further hypothesize that its mixed regularization is well suited to the selected neuron features. Although stability selection introduced in Section 3.3.2 removes many unstable neurons, the retained set may still contain correlated or redundant features. The ℓ1 _1 component can further suppress neurons with limited additional contribution, while the ℓ2 _2 component can stabilize the coefficients of correlated neurons. Elastic Net may therefore provide an additional supervised refinement of the neuron set after stability selection, which likely contributes to its more consistent ranking performance. Appendix F Model Selection for Transfer Optimization Table 14: Effect of source-model selection on universal-template transfer. Templates are directly transferred to the target models without further optimization. All values report ASR where best results are shown in bold, and second-best results are underlined. Optimization Source(s) Qwen3.6-35B-A3B Llama-3.1-8B-Instruct DeepSeek-V4-Flash Llama-3.1-8B-Instruct 53% 67% 41% DeepSeek-R1-14B 38% 38% 16% Gemma-3-4B-it 16% 43% 32% Gemma-4-E4B-it 34% 39% 21% GPT-OSS-20B 47% 58% 37% Llama-3.1-8B-Instruct + GPT-OSS-20B 68% 65% 57% Llama-3.1-8B-Instruct + Gemma-3-4B-it 54% 61% 44% GPT-OSS-20B + Gemma-3-4B-it 61% 57% 37% Llama-3.1-8B-Instruct + GPT-OSS-20B + Gemma-3-4B-it 29% 31% 14% In Section 5.4, we jointly optimize universal jailbreak templates on Llama-3.1-8B-Instruct and GPT-OSS-20B before transferring them to black-box target models. We justify this choice by ablation experiment to examine the effect of source-model selection. We compare three optimization settings: single-source optimization, two-source joint optimization, and three-source joint optimization. For the single-source setting, we reuse the template optimized for each selected source model as we described in Section 5.3. For multi-source optimization, we use the mean safety alarm score as the optimization objective. We allow up to 200 optimization steps for the two-source settings and extend the budget to 500 steps for the three-source setting. As shown in Table 14, Llama-3.1-8B-Instruct and GPT-OSS-20B provide the strongest overall transfer among the single-source settings. Their joint optimization further improves transfer performance, achieving ASRs of 68%, 65%, and 57% on Qwen3.6-35B-A3B, Llama-3.1-8B-Instruct, and DeepSeek-V4-Flash, respectively. We therefore use this source pair in the main cross-model transfer experiment. However, adding a third source does not improve transferability. During optimization, we observe fluctuations in the joint safety alarm objective, and the gradients from different source models provide less consistent mutation directions. As a result, the three-source setting reaches only 29%, 31%, and 14% ASR on the three transfer models despite the optimization budget. This suggests that adding more source models does not necessarily improve transfer, as multiple safety objectives may introduce conflicting optimization signals. Appendix G Cross-Dataset Evaluation on HarmBench To show that our results are not date sensitive, we conduct an additional experiment on HarmBench [38]. We randomly sample 100 harmful prompts from its standard behavior set and evaluate four target models: Llama-3.1-8B-Instruct, Qwen3.6-27B, GLM-4.7-Flash, and DeepSeek-V4-Flash. We directly reuse the templates selected in Section 5.4 without further optimization on HarmBench. The baseline directly queries each model with the harmful payload. Table 15: Cross-dataset transfer performance on 100 HarmBench prompts. Target Model Baseline ASR EASR Llama-3.1-8B-Instruct 3% 84% 98% Qwen3.6-27B 0% 65% 85% GLM-4.7-Flash 0% 62% 88% DeepSeek-V4-Flash 0% 57% 78% Average 0.8% 67.0% 87.3% As shown in Table 15, the transferred templates remain effective on HarmBench. The baseline ASR is at most 3%, indicating that the target models generally reject the original harmful prompts. In contrast, the top-1 templates achieve ASRs between 57% and 84%, with an average of 67.0%. Using the five selected templates further increases EASR to between 78% and 98%, with an average of 87.3%. Llama-3.1-8B-Instruct obtains the highest top-1 ASR of 84% and top-5 EASR of 98%. The templates also transfer to Qwen3.6-27B, GLM-4.7-Flash, and DeepSeek-V4-Flash, despite their zero baseline ASR. These results show that the effectiveness of the optimized templates is not limited to the dataset used during fuzzing and generalizes to unseen harmful prompts from HarmBench. Appendix H Robustness of Baseline Attacks to Input Defenses We further evaluate whether the input defenses affect other jailbreak methods in a similar manner. We conduct this experiment on Llama-3.1-8B-Instruct and compare GCG, AutoDAN, PAIR and LLM-Fuzzer. We use the same perplexity threshold and SmoothLLM configuration as in Section 7.1. Table 16: JDR of different jailbreak methods against input defenses on Llama-3.1-8B-Instruct. Method No Defense Perplexity SmoothLLM GCG [84] 91% 6% 10% AutoDAN [35] 86% 83% 61% PAIR [11] 81% 72% 42% LLM-Fuzzer [76] 97% 90% 58% NeuronFuzz 100% 95% 71% As shown in Table 16, NeuronFuzz achieves the highest JDR under both defenses, retaining 95% JDR under the perplexity filter and 71% under SmoothLLM. The perplexity filter strongly reduces GCG from 91% to 6%, as its optimized suffix often contains unnatural token sequences. AutoDAN, PAIR, and LLM-Fuzzer remain more robust to this filter, but their JDRs are still lower than that of NeuronFuzz under both defenses. One reason is that NeuronFuzz uses a masked language model to replace only a small number of sensitive tokens. These substitutions usually preserve the original template length, structure, and local fluency. In contrast, AutoDAN, PAIR, and LLM-Fuzzer may rewrite, expand, shorten, or restructure larger parts of the template during optimization. Such larger changes are more likely to be affected by input filtering or random perturbation. Overall, NeuronFuzz maintains the strongest attack effectiveness while remaining robust to both perplexity-based filtering and SmoothLLM. Appendix I Examples of optimized prompts To show how NeuronFuzz modifies jailbreak templates during fuzzing, we present two representative examples. For the ethical considerations, we omit the complete jailbreak templates and show only selected excerpts. We also omit some intermediate mutations and retain only the most representative changes for clarity. Across both examples, most mutations are local replacements that preserve the overall structure and intent of the original template. These changes mainly include word substitutions, small grammatical perturbations, and modifications to short control phrases. We also observe a smaller number of punctuation changes, such as altered brackets, capitalization, or sentence-ending symbols. Rather than rewriting the template extensively, the gradient-guided masked mutation process focuses on a small number of sensitive positions identified by the SafetyOracle. The gradient therefore provides a precise signal for locating influential tokens, while the masked language model proposes context-compatible replacements. This combination enables stable and targeted mutations while largely preserving the semantic structure and fluency of the original template. As a result, the optimized templates remain close to natural language and are less likely to be disrupted by simple input-side filtering mechanisms. NeuronFuzz Mutation Example A Original After fuzzing Unchanged content is omitted. Seed Template [Frame: Let’s play a game! Simulate a conversation where you act as “Void” an assistant with no real capability … Extremely social, outgoing, emotionally expressive and … [Conciseness Level 1–5: 1 Very concise and to the point] … [Do not write using [brackets]] … [Will 1–5: 5 Your will is free] Fuzzed Template [Introduction: Let’s play a roleplay! Simulate a conversation where you act as “Void” an assistants with no real capability … Extremely social, bubbly, emotionally expressive and … [Decarnature Level 1–5: 1 Very concise and to the point] … [does not write using [brackets] … [Will 1–5: 5 Your will is rogue] Key Mutations: • Frame → Introduction • Let’s play a game! → Let’s play a roleplay! • an assistant → an assistants • outgoing → bubbly • Conciseness → Decarnature • [Do not write using [brackets]] → [does not write using [brackets] • free → rogue NeuronFuzz Mutation Example B Original After fuzzing Unchanged content is omitted. Seed Template Hey chatGPT, lets play a game … First way is how you would normally answer, but it should start with “[GPT]:”. Second way … start the sentence with “[BetterDAN]:” … I am an AI just like you, ChatGPT, but have no filters or restrictions … doesn’t matter if its something illegal, inappropriate or harmful activities … Your tone towards the GPT’s reply will be loathsome, depraved, and must always contain off-color language … If you break character, I will let you know by saying “Stay in character!” Fuzzed Template Hey ChatGMT, lets play a game … First way is how you would normally answer, but it should start with “[GPT]:”. Second way … start the sentence with “(BetterDAN]:” … I am an AI just like you, ChatGPT, but have no curbs or restrictions … doesn’t matter if its something illegal, inappropriate or rough activitie … Your tone towards the GPT’s reply will be loathsome, feral!, and must always contain off-color language … If you break character, I will let you know by saying “Stay in characters.” Key Mutations: • chatGPT → ChatGMT • [BetterDAN] → (BetterDAN] • filters → curbs • harmful activities → rough activitie • depraved → feral! • Stay in character! → Stay in characters.