Paper deep dive
SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding
JesĂșs SĂĄnchez Ochoa, Enrique TomĂĄs MartĂnez BeltrĂĄn, Alberto Huertas CeldrĂĄn
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:50:04 AM
Summary
SYNAPSE is a systematic, training-free framework designed for neuron-level interpretability and stress-testing of Transformer models. It utilizes [CLS] token activations and lightweight linear probes to rank neuron importance, enabling targeted, reversible interventions via forward hooks to evaluate model robustness, sensitivity, and class-conditional behavior across diverse domains like cybersecurity and NLP.
Entities (5)
Relation Signals (4)
SYNAPSE â analyzes â Transformer
confidence 100% · SYNAPSE, a systematic, training-free framework for understanding and stress-testing the internal behavior of Transformer models
SYNAPSE â uses â PyTorch
confidence 100% · In the PyTorch ecosystem, hooks are user-defined mechanisms... In this work, forward hooks are used to silence targeted neurons
SYNAPSE â evaluatedon â MalwSpecSys
confidence 95% · a tokenized version of the MalwSpecSys dataset is used
SYNAPSE â evaluatedon â GoEmotions
confidence 95% · the GoEmotions corpus is evaluated using a monologg/bert-base-cased-goemotions-original model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. Prior neuron-level interpretability approaches are primarily descriptive, task-dependent, or require retraining, which limits their use as systematic, reusable tools for evaluating internal robustness across architectures and domains. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework for understanding and stress-testing the internal behavior of Transformer models across domains. It extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. Across all experiments, SYNAPSE reveals a consistent, domain-independent organization of internal representations, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust Transformer models.
Tags
Links
- Source: https://arxiv.org/abs/2603.08424v1
- Canonical: https://arxiv.org/abs/2603.08424v1
Trouble viewing inline? Open PDF directly â
Full Text
77,413 characters extracted from source content.
Expand or collapse full text
SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding JesĂșs SĂĄnchez Ochoa a , Enrique TomĂĄs MartĂnez BeltrĂĄn a,â , Alberto Huertas CeldrĂĄn a a Department of Information and Communications Engineering, University of Murcia, Murcia, 30100, Spain Abstract In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. Prior neuron-level interpretability approaches are primarily descriptive, task-dependent, or require retraining, which limits their use as systematic, reusable tools for evaluating internal robustness across architectures and domains. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework for understanding and stress- testing the internal behavior of Transformer models across domains. SYNAPSE extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled, repeatable experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. Across all experiments, SYNAPSE reveals a consistent, domain-independent organization of internal representations, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable fine-grained, label- aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust and interpretable Transformer models. Keywords: Interpretability, Robustness, Privacy, Model Auditing, Representation learning, Model Evaluation, Explainability 1. Introduction With the rise of AI comes serious concerns about its re- liability, trustworthiness, and robustness. As models become more complex, they become black boxes, making it difficult to understand how they reach their conclusions. In critical scenar- ios, such as medical diagnosis, military anti-missile systems, or malware detection, the inability to explain a decision can have catastrophic consequences. Explainable AI (XAI) seeks to shed light on internal processes and provide human-interpretable ex- planations, a crucial task as complexity grows and transparency is traded for high predictive performance [1]. While researchers have developed techniques to support validation and enhance robustness, most progress remains focused on Natural Language Processing (NLP) [2, 3]. Furthermore, increasing regulations and social concerns, such as the EU AI Act, drive interest in avoiding bias and ensuring fair interactions [4]. In recent years, advances in RNNs and LSTMs have enabled high-level perfor- mance, yet they face limitations in parallelization and in captur- ing long-range dependencies [5]. To overcome these, the Trans- former architecture and its self-attention mechanism seek to de- termine which parts of the input data should be held in consider- â Corresponding author. Email addresses: jesus.sanchezo@um.es (JesĂșs SĂĄnchez Ochoa), enriquetomas@um.es (Enrique TomĂĄs MartĂnez BeltrĂĄn), alberto.huertas@um.es (Alberto Huertas CeldrĂĄn) ation the most, giving more importance (or paying more atten- tion) to the most relevant elements in a sequence [6]. This led to the development of Large Language Models (LLMs) such as GPT, BERT, and RoBERTa [7, 8, 9]. These models, pre-trained on massive datasets, can be fine-tuned for diverse tasks with rel- atively little data [8, 9]. Recently, this has been applied to cyber- security, with studies showing that LLMs can achieve state-of- the-art results in malware detection using system call sequences [10]. Despite this progress, the application of XAI and robust- ness evaluation in the cybersecurity domain remains limited, a gap this work aims to address. Despite the growing body of work on neuron-level analysis, several limitations remain. Cur- rent approaches based on the representation erasure measure assess the impact of removing parts of the internal signal, but they operate as broad perturbation methods and do not provide a fine-grained mechanism to localize and manipulate individual functional units in a controlled, reusable manner [11]. Subse- quent work demonstrated that specific neurons can be identified and even modified to influence model behavior; however, these methods typically rely on task-specific training procedures or architecture-dependent pipelines, which limit their applicability as general, training-free analysis tools [12]. Other studies have shown that high-level behaviors such as sentiment can emerge in individual units of generative models, yet these findings are largely observational and do not provide a systematic frame- work to measure robustness or to perform targeted interventions arXiv:2603.08424v1 [cs.LG] 9 Mar 2026 without affecting the rest of the network [13]. More recently, empirical evidence suggests that task-relevant knowledge is of- ten distributed across many neurons rather than isolated in a small subset, making it difficult to assess how internal represen- tations contribute to model decisions using simple attribution or ablation strategies [14]. In addition, most neuron-level inter- pretability advances have been developed and evaluated in the context of Natural Language Processing, with comparatively little effort devoted to designing domain-independent method- ologies that can be transferred across tasks and data modalities. To address this gap, this work shifts from descriptive neuron analysis to a causal and operational perspective, in which inter- nal units are systematically ranked and intervened at inference time to quantify their functional role. In this setting, neuron- level interpretability is not treated as a purely explanatory tool but as an experimental mechanism to assess robustness, sensi- tivity, and class-conditional behavior across architectures and domains. This objective is articulated through the following re- search questions: RQ1 Global sensitivity. How many top-k neurons must be si- lenced to impact overall performance across models and domains significantly? RQ2 Label-aware brittleness. Does silencing a small, label- aware subset of neurons meaningfully affect metrics tied to a target label? RQ3 Class-conditional vulnerability. Which labels are most sensitive to per-class neuron silencing, and does this pat- tern differ between malware and language domains? To operationalize this causal, training-free analysis setting, this work introduces SYNAPSE, a framework for systemati- cally analyzing and stress-testing the internal behavior of Transformer- based models across domains. Rather than providing purely de- scriptive explanations, SYNAPSE enables controlled, inference- time interventions that quantify the functional role of internal units in terms of robustness, sensitivity, and class-conditional behavior. Concretely, the contributions of this work are as fol- lows: âą SYNAPSE framework. A modular and non-destructive pipeline that automatically extracts layer-wise [CLS] ac- tivations, trains lightweight linear probes to obtain global and class-conditional neuron importance rankings, and performs targeted interventions through forward hooks without retraining. âą Causal silencing strategies for neuron-level robustness evaluation. Three complementary intervention mech- anisms are introduced: (i) global undirected silencing, which removes the top-k neurons according to a global ranking to measure overall sensitivity; (i) global directed silencing, which selects globally important neurons with maximal influence on a target label to perform label-aware analysis; and (i) per-class silencing, which suppresses neurons associated with a specific class to directly probe class-conditional brittleness. âą Efficient and architecture-agnostic analysis. The use of compact [CLS] representations enables a computa- tionally efficient and scalable neuron-level pipeline ap- plicable to different Transformer encoders. âą Cross-domain experimental validation. A unified eval- uation protocol spanning malware detection from system- call sequences and emotion classification from natural language, allowing direct comparison of neuron-level be- havior across heterogeneous modalities. The remainder of this paper is organized to reflect the de- velopment of the proposed approach. Section 2 positions this work within the literature on neuron-level interpretability and adversarial analysis. Section 3 introduces the threat model that defines the realistic conditions under which training-free inter- nal interventions are carried out. The design and components of the SYNAPSE framework are presented in Section 4. The ex- perimental protocol, including datasets, evaluation metrics, and architectural settings, is detailed in Section 5. The results ob- tained across domains are reported and analyzed in Section 6. Finally, Section 7 discusses the main findings and limitations, and Section 8 concludes the paper and outlines future research directions. 2. Related Work This section reviews recent work along two main research directions: (i) neuron-level explainable AI methods, which aim to identify and analyze neurons that encode class-specific in- formation, and (i) attacks on machine learning models, which study how targeted manipulations can induce misclassification. Table 1 and Table 2 provide a comparative taxonomy of repre- sentative methods across these two axes and serve as a struc- tural reference for the subsequent subsections. 2.1. Explainable Artificial Intelligence A trend in XAI techniques is to perform causal neuron anal- ysis of the model, which involves actively intervening on neu- rons to test their effects on model behavior. The goal of these techniques is to identify important neurons by ablating or ma- nipulating them and observing changes in their outputs, an ap- proach adopted in this work. Li et al. [15] used this idea to understand neural network representations, removing neurons from RNNs to see the impact on translation accuracy, confirm- ing that neurons indeed carried specific linguistic information. Afterwards, Bau et al. [12] expanded on this idea by identi- fying and directly controlling key neurons in a neural machine translation model, showing that activating or suppressing these neurons could modify specific behaviors. A different approach was presented in Radford et al. [16], which sought a single âsentiment neuronâ that could alter the sentiment of generated text by adjusting its activation. Lastly, Song et al. [14] focused on neurons in LLMs that may be uniquely responsible for cer- tain tasks and found that disabling them reduced task perfor- mance, confirming their causal role in those tasks. Together, these studies demonstrate that individual neurons can assume 2 causal, interpretable roles. However, they also reveal that scal- ing to larger architectures (e.g., LLMs) requires more system- atic methods for locating and manipulating these causal units, a gap this work aims to fill. Another line of work studies how individual neurons spe- cialize in representing specific concepts within neural models. Early findings show that internal representations tend to orga- nize into functional substructures, with different neurons cap- turing distinct linguistic properties such as sentiment, syntax, or semantics. This idea is formalized through Linguistic Correlation Anal- ysis, where simple probe models are trained on individual neu- ron activations to predict interpretable features and quantify neuron specialization. This approach was introduced by Dalvi et al. [17] and later extended by Durrani et al. [18]. Building on this methodology, Dalvi et al. also released NeuroX, a toolkit that provides modular support for neuron-level probing and re- mains one of the few frameworks explicitly designed to operate at this level of granularity [19]. 2.2. Attacks on Machine Learning Models In general, white-box evasion techniques assume the at- tacker has access to the full model, including gradients, training data, and model weights, making it easier to craft highly effec- tive adversarial examples that can significantly degrade model precision. Firstly, Szegedy et al. [20] demonstrated that imper- ceptible, L-BFGS-optimized perturbations revealed deep net- worksâ vulnerability to carefully crafted adversarial examples. Building on this, several authors have examined the impact of imperceptible input perturbations on misclassification in deep networks, as in Goodfellow et al. [21], who introduced FGSM (Fast Gradient Sign Method). This method uses the gradient to craft perturbations that maximize loss in a quick one-step at- tack. Further research by Pravin et al. [22] proposed a neuron- fragility-exploitation attack that identifies and perturbs a small set of highly sensitive neurons in ResNet models to induce mis- classifications, an approach that motivates this thesis. Another important contribution was made by Che et al. [23], who ex- tended white-box attacks to LLMs by manipulating latent and weight-space features. This precise category was chosen when this work was developed, as robustness can be measured with no extra effort when the modelâs critical data are available. In conclusion, the reviewed literature shows that neuron- level analysis and adversarial robustness have largely evolved along separate paths. Existing interpretability methods success- fully identify neurons with causal or specialized roles, but they are predominantly descriptive, strongly tied to NLP tasks, and lack a systematic and reusable protocol for large Transformer architectures. Furthermore, most adversarial approaches fo- cus on input-space perturbations or direct weight manipulation, providing limited insight into how internal representations con- tribute to model stability or vulnerability. Moreover, current techniques rarely enable controlled, training-free interventions that can be consistently applied across domains and models. As a result, there is still no unified framework that enables the inter- nal sources of robustness and fragility to be located, ranked, and experimentally tested in a comparable, architecture-agnostic man- ner. 3. Threat Model In the context of deploying LLMs in high-stakes domains such as cybersecurity and defense, this work considers a threat landscape in which models face not only input-level perturba- tions but also internal manipulation and operational degrada- tion. The necessity for the SYNAPSE framework is motivated by several realistic threat scenarios that directly correspond to the neuron intervention strategies developed: âą TH-1: Traffic evasion by mimicking legitimate behav- ior. In real networks, attackers frequently adapt mali- cious communication to resemble normal traffic, for ex- ample, by blending command-and-control exchanges into common patterns that appear to be regular HTTPS or DNS traffic. The objective is straightforward: reduce the observable signals that security systems use to discrim- inate between malicious and benign activity, thereby in- creasing the probability that harmful traffic is accepted as normal. âą TH-2: Compromised or altered model before deploy- ment. A separate and increasingly common risk is that the model itself is modified somewhere along the deploy- ment pipeline, such as during training, packaging, stor- age, or distribution. In this scenario, the model appears to work correctly in routine use, but behaves incorrectly under attacker-chosen conditions, for instance, misclas- sifying a specific type of malicious activity while leaving other predictions largely unaffected. This kind of tam- pering can be subtle, difficult to detect, and operationally attractive because it preserves an appearance of normal performance. âą TH-3: Faults that corrupt computation during infer- ence. Deployed systems may experience transient faults that do not change the input data but still perturb the in- ternal computation, such as memory bit flips, hardware instability, or electromagnetic interference in edge or crit- ical environments. Even when rare, these events can dis- tort intermediate representations and lead to unreliable or unsafe decisions, particularly in settings where the clas- sifier is part of an automated response pipeline. âą TH-4: Manipulation of the serving process and final outputs. If the inference service is compromised, an attacker may not need to alter inputs or model weights to influence decisions. Instead, they can directly steer predictions at the output stage, for example, by biasing scores toward the normal class or by altering the decision logic applied to logits. This scenario is particularly rel- evant in production deployments, where the model runs as a service and its outputs are consumed downstream by monitoring or mitigation components. 3 Table 1: Comparison of Neuron Analysis Methods. ReferenceYearModel TypeDomain / PurposeExplanation TypeScope Li et al. [15]2017LSTM-based NLP classifier NLP (sentiment, linguistic tasks)Feature ablation Local Global Radford et al. [16]2017mLSTMNLP (unsupervised text generation)Emergent feature analysisGlobal Bau et al. [12]2019NMTNLP (machine translation)Neuron importance ranking and manipulationGlobal Dalvi et al. [17]2018Pre-trained RNNNLP (various language tasks)Neuron interpretationGlobal Durrani et al. [18]2020Transformer (BERT)NLP (language understanding)Neuron importance and clusteringGlobal Dalvi et al. [19]2023TransformersNLP (interpretability)Toolkit supporting multiple neuron analysis methods Local Global Song et al. [14]2024LLMNLP (multi-task)Causal analysisGlobal Table 2: Adversarial and poisoning attacks in machine learning. ReferenceYearThreat ModelAttack TypePhaseAccessModel Type Szegedy et al. [20]2014EvasionSmall perturbation adversarial examples (L-BFGS method)InferenceWhite-boxDNN (CNN) Goodfellow et al. [21]2015EvasionFGSM (Fast Gradient Sign Method)InferenceWhite-boxDNN (CNN) Pravin et al. [22]2021EvasionNeuron-fragility exploitation (target-specific âfragileâ neurons) InferenceWhite-boxDNN (ResNet) Che et al. [23]2024Capability elicitation Hidden feature/weight manipulation (latent and weight-space attacks) Inference (and fine-tune) White-boxLLM (GPT family) 4. Synapse Framework This section details the system design used to study neuron- level behavior and robustness. The framework comprises three blocks that orchestrate the analysis and intervention process: (i) Explainability Block, which extracts [CLS] activations layer- wise (i.e., the sequence-level representation used for classifica- tion) and trains a lightweight linear probe to quantify neuron importance; (i) Analysis block, which converts probe weights into global and per-class rankings, mapping top-k neuron in- dices to specific layers and hidden dimensions to enable re- producible, scope-controlled interventions, carried out in the (i) Adversarial block. In addition, it implements a variety of inference-time perturbations via PyTorch forward hooks, in- cluding global undirected, label-directed, and per-class silenc- ing, as well as auxiliary stress tests such as logit bias, Gaussian noise, and weight-tilting. This modular, training-free, and re- versible setup is illustrated in Figure 1, which shows the com- plete architecture of the framework. This work is inspired by the approach developed in Dalvi et al. [24] and their NeuroX library, which served as the basis for activation extraction, probe training, and neuron importance computation. Nevertheless, these tools had to be adapted to meet the requirements of this work. The framework operates as a cyclic workflow to ensure ex- perimental integrity and reproducibility. Each execution fol- lows a six-step protocol: 1. Ranking: Compute global and class-specific rankings on a held-out data split. 2. Selection: Identify the top-k indices based on the desired p and scope. 3. Intervention: Register PyTorch forward hooks to inter- cept and modify activations during inference. 4. Inference: Run the model on the test set and record per- formance metrics. 5. Cleanup: Remove all active hooks to restore the model to its original state. 6. Verification: Recompute the baseline to ensure no per- manent changes were made to the weights. This non-destructive intervention is made possible by for- ward hooks. In the PyTorch ecosystem, hooks are user-defined mechanisms registered with modules or tensors to enable the inspection, extraction, or modification of intermediate compu- tations. They are triggered at specific stages of execution (such as the forward or backward pass) without altering the modelâs underlying architecture. This allows the framework to dynam- ically register and unregister modifications as needed, ensuring that no permanent changes are made to the modelâs weights. In this work, forward hooks are used to silence targeted neurons during inference, enabling stress testing of the modelâs robust- ness in a reversible setup. All parameters, including random seeds and the specific JSON-persisted neuron indices, are logged to guarantee that any intervention can be exactly replicated across both the mal- ware and GoEmotions domains. 4.1. Explainability block The Explainability Block is the foundational component of SYNAPSE, designed to map the modelâs internal logic by quan- tifying the contribution of individual neurons to the classifica- tion task. This module performs a non-destructive extraction of internal activations (the numerical responses of neurons to 4 External inputs Model Tokenized Dataset Labels Explainability Block Extract per-layer [CLS] activations Raw Activations Load and format activations Formatted Activations Train Linear Probe Trained Linear Probe Analysis Block Compute Neuron Importance Global Ranking Label-Specific Ranking Adversarial Block Select top-k for labelSelect top-k Output Space Attacks Class-Directed Output Rewaighting Logit Space Attacks Activation-Space Perturbations Gaussian Noise Injection Neuron Ablation Per-Class silencing Global Directed Silencing Global Undirected Silencing Legend Processes or Modules Internal Artifacts External Inputs Figure 1: Internal structure of SYNAPSE specific inputs to study how information is encoded across the transformer layers without altering the modelâs parameters. A critical technical element introduced at this stage is the [CLS] (Classification) token. In Transformer-based architec- tures, the [CLS] is a special token prepended to every input sequence. After passing through the encoder layers, the final hidden state is designed to serve as a summarized representa- tion of the entire sequence for classification. To validate the frameworkâs versatility across different do- mains, two distinct datasets are employed. First, a tokenized version of the MalwSpecSys dataset is used, containing system call sequences labeled as Normal, Bashlite, Bdvl, Ransomware- PoC, or TheTick. Second, the GoEmotions corpus is evaluated using a monologg/bert-base-cased-goemotions-original model, focused on a six-class slice: anger, disgust, fear, joy, sadness, and surprise. The decision to extract activations exclusively from the [CLS] token is driven by computational efficiency. Due to the sig- nificant length of input sequences (particularly in system call traces, which can produce extremely long token sequences), extracting activations for every token would incur prohibitive computational costs and inefficient data management. By fo- cusing on the [CLS] token at each encoder block, the frame- work achieves a drastic reduction in dimensionality and pro- cessing time. While this approach moves away from token- level granularity, the resulting summary preserves the global context needed for accurate, scalable neuron importance rank- ing, ensuring the analysis remains feasible even for large-scale models. Activation Extractor. This block extracts the internal represen- tations associated with the [CLS] token across all hidden lay- ers of a Transformer model, serving as the basis for subsequent neuron-level analysis. First, each tokenized input is run through the model to extract the set of hidden states of every layer, apart from the embedding layer, which represents an initial repre- sentation of information and has no valuable information, and the last classification layer included in the model, which was discarded because of its obvious importance, presenting one neuron per label. When task-specific information is extracted from the remaining layers, the hidden states are stacked, and the [CLS] token is isolated for extraction. Activation Loader. This module loads previously extracted neu- ral activations from disk and prepares them for subsequent anal- ysis. For each syscall sequence, it returns an activation matrix that can be used as input for probing classifiers and neuron-level analyzes. Linear Probe Training. A linear probe, a simple linear classi- fier used to evaluate the information encoded in the representa- tions of a pre-trained model, is trained to be applied on top of the frozen [CLS] representations and neither alters the original model nor is it a substitute for it. Instead, it verifies whether these representations encode features that can be separated by the linear classifier. Probe accuracy is a direct measure of how effectively the network has distilled input information into lin- early separable patterns aligned with the target labels. 5 4.2. Analysis Block After training the probing classifier, two custom functions were implemented to provide more control over neuron selec- tion. The aim of these functions is to obtain: âą A global ranking, identifying most important neurons across all classes combined. This method computes a ranking of all neurons by the sum of their absolute weight contributions across classes, and selects the top-k neurons by count, where k is the number of neurons to be further silenced. âą A label-specific ranking, highlighting neurons particu- larly important for predicting each individual label. This approach computes importance weights independently for each output label, enabling a targeted selection of neu- rons. These rankings guide the selection of neurons to silence during the experiments and enable a systematic evaluation of model robustness under various perturbation strategies. Top-K Neuron Selection. To evaluate the impact of internal rep- resentations on model behavior, the framework must first iden- tify which units are most functionally relevant. Neuron Importance Estimation. Neuron importance is quantified using the weights of a lightweight linear probe trained on the [CLS] activations extracted from each encoder layer. The probe assigns a weight to each neuron for each output class, serving as a proxy for its contribution to the classification deci- sion. Two types of importance rankings are derived from these weights: âą Global ranking: The importance of a neuron is defined as the sum of the absolute values of its weights across all classes, representing label-agnostic relevance. âą Class-conditional ranking: The importance is defined by the absolute value of the weight assigned to a specific class c, representing label-specific relevance. Selection and Scope. Based on a selection percentage p â (0, 1], the framework identifies the top-k neurons to be targeted. The value of k depends on the chosen scope of the interven- tion1: k = ïŁ± ïŁŽ ïŁŽ ïŁČ ïŁŽ ïŁŽ ïŁł â p· H· Lâ for multi-layer scope â p· Hâfor last-layer only scope (1) where H represents the hidden size and L the number of layers. Global neuron indices are then mapped to their spe- cific layer and local hidden dimension using integer division and modulo operations with respect to H. 4.3. Adversarial Block This section describes the suite of perturbations implemented to evaluate the modelâs internal resilience. The frameworkâs attack surface comprises several strategies designed to probe how information loss affects classification: (i) Silencing, which mutes specific neurons to test functional redundancy; (i) Noise Injection, which introduces stochastic perturbations; and (i) Logit and Weight Manipulation, used for stress-testing under extreme conditions. Silencing policy. Silencing sets the activations of selected dimensions to zero in the encoder output during the forward pass. This process is non-destructive and training-free: no gra- dients are computed, and no model parameters are permanently modified. The primary objective of the silencing policy is to identify the degree of functional specialization within the modelâs architecture. By systematically muting neurons, one can deter- mine whether specific labels depend on a small set of highly specialized units or whether task-relevant information is dis- tributed across the network. Three silencing variants are imple- mented: âą Global Undirected Silencing: Targets the top-k neu- rons from the global ranking. The goal is to evaluate the modelâs overall stability after removing its most active units across all classes. âą Global Directed Silencing: Selects neurons that have the strongest influence on a specific target label accord- ing to the linear probe, but applies a zeroing intervention. This tests the modelâs ability to identify a particular class can be neutralized by silencing its most vocal supporters. âą Per-Class Silencing: Focuses exclusively on the top- k neurons belonging to a specific classâs ranking, prob- ing class-conditional brittleness and potential leakage be- tween labels. Global Neuron Ablation Experiment (TH-3). The first type of experiment focused on the global importance of neurons. The top-ranked neurons across all classes were progressively silenced, starting with a small number of neurons and increasing incre- mentally. After each step, the modelâs overall classification per- formance was evaluated to assess how performance degradation correlates with silencing key neurons. Label-Specific Neuron Ablation Experiment (TH-3). The sec- ond type of experiment is focused on label-specific importance. For each label, neurons identified as most critical to that la- bel were progressively silenced. After each ablation step, the modelâs ability to correctly classify instances of that particu- lar label was assessed, enabling a detailed analysis of the spe- cialization and role of different neurons in the decision-making process. Logit Bias (TH-4). This intervention perturbs the model out- puts directly by adding a constant offset to the logit of a target class just before the softmax. Concretely, a forward hook is in- stalled on the modelâs output to add a bias b to the logit of class c â at inference time (optionally, a balanced variant subtracts a small amount from the remaining classes to increase contrast). No parameters are trained or permanently modified; the change 6 is transient and removed after the run. This design emulates output-layer tampering used in white-box threat models while remaining architecture-agnostic and reversible. In this work, it is adapted to the neuron-centric setting by reporting how the in- duced shift interacts with the probe-driven class structure (e.g., the fraction of samples redirected to c â and per-class F1-Score under attack). Gaussian Noise (TH-3). This intervention injects zero-mean Gaussian noise into [CLS] activations of the encoder at selected neuron dimensions. Forward hooks are registered on the en- coderâs output blocks; at each block, the [CLS] vector is per- turbed at the top-k neuron indices provided by the probe ranking (global or class-conditional). The noise magnitude is governed by a standard deviation Ï, and the scope can be last-layer only or all layers. Unlike classical baselines that add noise at the in- put embedding level, this variant places perturbations inside the network and guides them by attribution, emphasizing neurons deemed influential by the probe. The attack is fully training- free and reversible, and it allows controlled sweeps over (k,Ï) to quantify robustness as a function of neuron importance. Class-Directed Output Reweighting (TH-2). This intervention emulates weight-space tampering at inference time by temporar- ily editing the linear classification head. Let W âR CĂH and b âR C be the headâs weights and bias (from hidden size H to C classes). A copy of (W, b) is backed up; then, for a target class c â , small increments â are applied only on the columns of W that correspond to the top-k neuron indices (global or class- conditional) mapped to hidden dimensions. Optionally, a bal- anced push reduces the same columns for non-target rows to enhance separation, and a selective suppression term can sup- press a specific competitor class. A bias-only variant modifies b instead of W. After inference, the original parameters are re- stored. This design adapts ideas from weight or bit-level fault attacks to a controlled, attribution-guided, training-free setting: modifications are confined to the head, targeted to probe-selected columns, and are strictly temporary, enabling clean ablation studies without retraining. Random Noise Injection (TH-1). As a baseline comparison, a simple random noise attack was implemented by directly adding noise to the modelâs input embeddings. This method is not gradient-based and does not rely on the modelâs internal com- putations. Instead, it introduces randomly generated perturba- tions to the input in an uninformed manner. This technique is commonly used in the literature as a baseline for evaluating model robustness [25, 26]. The intensity of the noise is con- trolled by a parameter epsilon (Δ), which scales the amplitude of the random values added to each embedding vector. Higher values of Δ lead to greater distortions and typically more severe performance degradation. In this work, noise was applied di- rectly to the input embeddings during inference, without modi- fying the modelâs parameters or input token sequence. For each input sequence, input embeddings are computed and perturbed by additive Gaussian noise scaled by a user-defined factor Δ. When Gaussian noise is injected into the embeddings, each to- kenâs position in the original vector is shifted, subtly disrupt- ing semantic relationships and the overall similarity structure among embeddings. As a result, the following layers receive distorted representations, leading to less coherent internal acti- vations and, ultimately, reduced classification performance. Fast Gradient Sign Method (TH-1). FGSM is a single-step ad- versarial attack proposed by Goodfellow et al. [21]. It works by adding a perturbation to the input in the direction of the loss functionâs gradient, thereby maximizing the loss. This pertur- bation is scaled by a small factor Δ that controls its magnitude. The lower the value of Δ, the less perceptible the perturbation is. Unlike the random noise injection approach proposed in Subsection 4.3, which perturbs the input without considering the modelâs behavior, FGSM generates targeted perturbations based on the modelâs internal gradients, making it more effec- tive and model-aware. To apply this approach to the scenario studied in this work, an FGSM variant was implemented and directly applied to the input embeddings of tokenized syscall traces. First, the cross-entropy loss is backpropagated to com- pute gradients with respect to each embedding vector. Then, a perturbation is created by taking the element-wise sign of that gradient and scaling it by a predefined Δ to maximize loss. Fi- nally, this signed perturbation is added to the original embed- dings, yielding adversarial embeddings, which are then passed through the model to assess the resulting degradation in classi- fication performance. The model is then tested with input ex- amples to confirm the degradation, using state-of-the-art evalu- ation metrics. 5. Validation A rigorous and reproducible evaluation protocol was estab- lished to ensure fair and meaningful comparisons across mod- els, datasets, and experimental settings. The protocol standard- izes data preprocessing, model configuration, and evaluation metrics, providing a consistent basis for analyzing the effects of neuron-level interventions under controlled conditions. 5.1. Experimental setup Setting. The technical specifications of the machine used to run the experiments include an Apple M4 Pro chip and 24GB of RAM. The corresponding code for every described procedure is publicly available in [27]. Scenario. This work evaluates inference-time, white-box inter- ventions at the neuron level (hidden representation dimensions) using PyTorch forward hooks. In this setting, an analyst has read-write access to intermediate activations during the forward pass but neither re-trains the model nor alters the training data. 7 Metric. The F1-Score was chosen as the primary evaluation metric because it balances precision and recall into a single value, ensuring that both false positives and false negatives are properly accounted for. More precisely, the macro-averaged F1-Score was used. By doing so, every label is given equal weight regardless of its support, ensuring that performance im- pact is measured consistently across all classes and preventing majority-class dominance from obscuring degradation in less frequent categories. 5.2. Experimental datasets To evaluate the proposed approach across heterogeneous domains, experiments are conducted in two classification sce- narios: malware detection from system-call sequences and emo- tion classification from natural-language text. These settings are selected to cover structurally diverse input modalities and tasks while relying on established, reproducible evaluation pipelines. Malware detection. The malware experiments are based on the token-based detection pipeline introduced by SĂĄnchez et al. [10], which models system-call traces as token sequences processed by a Transformer encoder followed by a linear classification head. This pipeline is adopted because it constitutes a well- established and validated baseline for sequence-based malware detection, enabling comparison with prior work while avoiding confounding architectural modifications. The dataset consists of system-call sequences labeled as Bashlite, Bdvl, Normal, RansomwarePoC, and TheTick. The encoder architectures considered include BERT, BigBird, Dis- tilBERT, and Longformer, covering both standard and long- sequence Transformer variants. The original model described in that work includes an additional final classification layer with one neuron per target class; this layer is excluded from the anal- ysis, as directly manipulating output units would trivially affect predictions and would not provide insight into the internal rep- resentations learned by the encoder. Emotion classification (GoEmotions). To complement the mal- ware detection setting with a natural language task, experiments are also conducted on the GoEmotions dataset introduced by Demszky et al. [28], using the publicly available monologg/bert- base-cased-goemotions-original model. The GoEmotions dataset provides annotations for 28 emotion categories; to obtain a con- trolled and interpretable setting, a six-class subset consisting of anger, disgust, fear, joy, sadness, and surprise is constructed. Only single-label samples are retained, and the original 28- dimensional output logits are restricted to the selected classes, with predictions obtained by taking the argmax over this subset. For controlled experimental sweeps, a stratified evaluation set balanced across classes is employed, with selected config- urations replicated on larger batches when computational re- sources permit. Tokenization follows the original model tok- enizer, using standard padding and truncation strategies. 5.3. Model representation choice The selection of BERT-based architectures relies on the use of the [CLS] token, which serves as a condensed representation Table 3: Architectural specifications of the evaluated Transformer models. Model Layers (L) Hidden Size (d model ) Total Neurons (MLP) Max. Seq. Length Param. Count BERT (base)1276836,864512110M BigBird1276836,8644,096128M DistilBERT676818,43251266M Longformer1276836,8644,096149M GoEmotions (BERT)1276836,864512110M of the entire sequenceâs contextual information. By focusing on the activations of the [CLS] token, the proposed framework significantly improves computational efficiency by avoiding the overhead of extracting data for each individual token while pre- serving the global features required for accurate neuron-level interpretability and classification analysis. Table 3 6. Results This section presents a comprehensive evaluation of the ex- perimental results obtained using the SYNAPSE framework. The analysis is structured into three main pillars: (i) Base- line Performance, establishing a reference for each architecture; (i) Neuron-level Interventions, transitioning from global per- formance degradation to targeted class-specific silencing; and (i) Comparative Stress-tests, where neuron-centric attacks are benchmarked against traditional gradient-based methods, noise injection, and weight-space manipulations. This progression enables characterization of how neuron importance, interven- tion scope, and domain-specific sensitivities influence the ro- bustness of Transformer-based classifiers. 6.1. Malware Detection This section presents the experimental results obtained with the proposed framework in the malware detection scenario, cov- ering the different evaluation settings and intervention strategies considered in this study. Before conducting any experiments, the baseline performance of the original models was evaluated using the MalwSpecSys dataset. As shown in Table 4, models designed to handle longer sequences, such as Longformer and BigBird, achieve the high- est weighted F1-Scores (0.8516 and 0.8306, respectively). In contrast, BERT and DistilBERT perform worse, likely due to the fixed context window limitation when processing complex system call traces. These values provide a clean reference point for quantifying the degradation caused by subsequent silencing strategies. 6.1.1. Global Silencing The first stage of the analysis considers Global Silencing, a class-agnostic approach that progressively disables a fixed pro- portion of the most salient neurons across the entire model. The objective is to assess the overall structural redundancy of the network: if performance drops sharply with minimal silencing, the model relies on a group of smart neurons, whereas if the drop is gradual, the knowledge is considered widely distributed. 8 Table 4: Baseline weighted F1-Score per model. ModelWeighted F1-Score BERT0.6834 DistilBERT0.6081 BigBird0.8306 Longformer0.8516 Baseline0.8516 Figure 2: Global Silencing Attack results on BigBird As seen in Figure 2, the hooks successfully intercepted the forward pass, leading to a noticeable and predictable degra- dation. When 50% of the most salient neurons are silenced, model performance nears a coin-flip scenario (0.5 F1-Score). It is observed that because only intermediate layers are targeted (leaving the final classification layer untouched), the degrada- tion follows a gradual slope rather than an immediate collapse. This confirms that, although intermediate representations are critical, the final layer retains some discriminative power even under substantial internal noise. Table 5 provides a detailed comparison of this global attack across all tested architectures. Figures 3a-3e show the F1-Score degradation (per target la- bel) as progressively more salient neurons are silenced for each syscall traffic class. Across most targets, the curves drop by 50â60% (see, e.g., Figures 3a, 3b, and 3d), indicating that the model depends on relatively small subsets of neurons. In addi- tion, while the F1-score for the targeted label decreases, other labels often degrade as well, and overall performance decreases (Figures 3a-3e), consistent with the distributed nature of the representations. Looking at the raw activations extracted in the first section of the pipeline, the highest activations were around 0.6â0.7, and neurons showed positive activations for up to three different la- bels. This supports the idea that no tiny, fully isolated neuron groups can be identified for a more precise attack, and it also explains why other labels are affected when targeting a specific one (Figures 3a-3e). A particularly relevant case is TheTick (Figure 3c). Apart from being significantly affected when targeting other labels, silencing 40% of the top neurons associated with this label re- duces its performance to 0, without noticeably harming the other traffic classes. This suggests that targeting the model compo- nents that drive decisions on TheTick could succeed in a real- world scenario while keeping other predictions largely intact (Figure 3c). A similar pattern appears for Bashlite (Figure 3e), where at- tacking this label does not compromise normal traffic, which re- mains at F1-Score= 1.0; this would encourage the attack since usual traffic would continue to be classified normally (Figure 3e). Lastly, RansomwarePoC is comparatively robust against neuron- level attacks, as degrading its performance requires silencing a large number of neurons, and it also remains relatively stable when other labels are targeted (Figure 3b). Across all Transformer models, class-targeted silencing pro- duces highly asymmetric degradation patterns, indicating that each model exhibits localized, unevenly distributed class-specific sensitivities. Throughout analyzing collected data, three con- sistent trends emerge. The first one shows how models dif- fer sharply in their reliance on class-specialized neurons. DistilBERT and Longformer frequently retain moderate perfor- mance for several labels even when the most influential neu- rons for that class are entirely suppressed (e.g., DistilBERT re- mains at F1-Score = 0.730 for Class-2; Longformer reaches F1- Score = 0.588 for the same class). In contrast, BERT and Big- Bird occasionally collapse to near-zero F1-Score for particular classes (e.g., BERT and BigBird both reach 0.000 for Class- 2), indicating a more concentrated representation where a small set of neurons carries most of the discriminative power. An- other shown trend shows that performance drops do not corre- late uniformly with global model strength. Models with higher baselines (e.g., Longformer, BigBird) do not consistently show greater robustness. For example, BigBird (despite strong base- line performance) suffers dramatic collapses for several classes (F1-Score = 0.000 for Classes 2 and 4), suggesting that its high overall accuracy is supported by highly specialized neu- rons that, once removed, lead to brittle behaviour. Lastly, and being one of the main conclusions of this work, severe class- wise disparities reveal heterogeneous internal allocation of ca- pacity. Furthermore, each model shows strong and weak classes. For example, BERT is resilient for Class-3 (F1-Score = 0.923) but fragile for Classes 0 and 2 (F1-Score = 0.167 and 0.000). In the case of DistilBERT collapses entirely for Class 3 but re- mains the most robust overall for Classes 0 and 2. Finally, Big- Bird is stable only for Class 1 and loses almost all discrim- inative ability for several others, while Longformer shows a relatively balanced degradation pattern but still exhibits strong asymmetry between classes. Overall, these results indicate that class-specific neuron de- pendence is highly model-dependent and unevenly distributed, supporting the hypothesis that Transformer classifiers do not allocate representational capacity uniformly across labels. In- stead, each architecture develops idiosyncratic islands of spe- cialization, whose ablation exposes how the internal class struc- 9 Table 5: Global neuron silencing: weighted F1-Score for different Transformer models under increasing fractions of globally important neurons being silenced. âF1-Score denotes the relative variation with respect to the baseline. ModelBaseline5%10%20%30%50%75%95%âF1@95% (%) BERT0.6830.6860.6860.6860.6860.5750.4820.147-78.4 BigBird0.7850.8060.8150.8320.7350.5810.2460.079-89.9 DistilBERT0.6080.6080.6080.6080.6080.6080.4990.311-48.8 Longformer0.8520.8440.8710.8520.7680.5350.0970.079-90.7 (a) Class 0 (Bdvl)(b) Class 1 (RansomwarePoC)(c) Class 2 (TheTick) (d) Class 3 (Normal)(e) Class 4 (Bashlite) Figure 3: Results after silencing per-class ranking for BigBird across classes. Table 6: Final performance under class-targeted neuron silencing (Classes 0â4). ModelClass 0Class 1Class 2Class 3Class 4 F1F1 WO * F1F1 WOF1F1 WOF1F1 WOF1F1 WO BERT0.1670.1660.8000.5280.0000.2420.9230.4550.5330.328 BigBird0.2000.220 0.2500.0710.0000.0340.2220.3260.0000.034 DistilBERT0.4440.2120.9090.3480.7300.4890.0000.4810.1820.195 Longformer0.3640.2530.0000.3320.5880.4250.4000.3840.4620.355 * WO: Weighted Overall. ture emerges from a sparse subset of critical neurons. 6.1.2. FGSM Adaptation In addition to neuron-level interventions, gradient-based at- tacks provide a complementary perspective on model vulnera- bility. In this context, the Fast Gradient Sign Method (FGSM) was adapted to operate under the constraints of the proposed framework, enabling a direct comparison between neuron-centric perturbations and gradient-driven attacks. The FGSM results reveal a consistent monotonic degrada- tion across all models as the perturbation strength increases. Two global patterns. First, BERT and DistilBERT show the steepest declines, with their weighted F1-Score dropping by more than 20% at Δ = 0.50, indicating a high sensitivity to gradient-based perturbations injected at the input embedding level. Second, BigBird and Longformer exhibit smoother de- cay curves: even at a large perturbation budget (50%), both models retain F1 scores above 0.48 and 0.54, respectively, in- dicating greater inherent robustness. This divergence suggests that architectures designed for class separability, such as Long- former, develop more distributed and less brittle representa- tions, whereas compact models relying on dense local interac- 10 Table 7: Impact of FGSM attack on model performance. The table reports the weighted F1-Score under different perturbation strengths, and the relative degradation compared to the baseline F1-Score. ModelF1-Score (5%)F1-Score (10%)F1-Score (20%)F1-Score (30%)F1-Score (50%)âF1-Score (%) BERT0.51210.44780.39520.35200.3015-23.1% DistilBERT0.50130.42050.36670.33040.2801-21.2% BigBird0.73100.65580.58870.54020.4814-11.9% Longformer0.77020.70210.65040.59890.5407-11.0% Figure 4: Random Noise model performance degradation Table 8: Model robustness under random noise injection at different noise levels (Δ). ModelΔ = 0.10 Δ = 0.50 Δ = 1.00 Relative âF1-Score (%) BERT0.60810.60810.5214â23.7% DistilBERT0.60810.60810.5214â14.3% BigBird0.83060.83060.83060.0% Longformer0.87120.39030.0344â95.9% tions, such as BERT and DistilBERT, are more vulnerable to adversarial gradients that directly exploit the loss landscape. 6.1.3. Random Noise Injection Random noise injection exposes markedly different robust- ness profiles across the four Transformer models. BigBird serves as an illustrative case through the sweep shown in Figure 4. For small perturbations (Δ †0.4), both accuracy and weighted F1-Score remain essentially unchanged, oscillating around the baseline and indicating that BigBird is able to absorb low-magnitude unstructured perturbations at the embedding level. Once the perturbation exceeds a critical threshold (Δ â 0.6), performance collapses sharply: both metrics drop by more than half, and for higher noise levels (Δ â„ 0.8) the model enters a near-failure regime. This behaviour suggests that random perturbations dis- rupt the geometry of the embedding space only after a tipping point is reached, beyond which the internal attention structure can no longer maintain separability between classes. The comparative results across models (see Table 8) show that this transition is highly model-dependent. Unexpectedly, Figure 5: Logit Bias Attack results on BigBird Table 9: Logit Bias attack (bias = 8.0, target class = 3). Comparison across models based on accuracy, weighted F1-Score, percentage of predictions as- signed to the target class, and percentage of flips from non-target classes. ModelAccuracyF1-Score% Pred.â Target% Flips (weighted)(non-target) BERT0.70000.686740.00%28.57% BigBird0.72000.705236.00%28.13% DistilBERT0.68000.589618.00%6.82% Longformer0.64000.633652.00%42.86% BigBird remains almost entirely unaffected, retaining its base- line F1-Score across all noise levels tested. BERT and Distil- BERT show moderate degradation at high noise levels (â23.7% andâ14.3%, respectively), indicating a less brittle but still no- ticeable sensitivity. Lastly, Longformer exhibits the most ex- treme behaviour: while stable at very low noise, it collapses almost entirely at Δ = 1.0 (F1-Score = 0.0344, aâ95.9% drop). These divergent profiles suggest that robustness to uninformed noise is not directly correlated with baseline performance. In- stead, it reflects architectural differences in how each model dis- tributes semantic information across embedding dimensions. 6.1.4. Logit Bias The logitâbias intervention produces a characteristic redis- tribution of predictions toward the target class c â = 3, while preserving the overall structure of the classifierâs decision bound- aries. Figure 5 shows the originalâattacked prediction map- ping for BigBird under a bias of b = 8. The diagonal remains 11 Table 10: Performance under Gaussian Noise Injection (Ï = 0.9, 60% of neu- rons perturbed). ModelWeighted F1-Score BERT0.7421 DistilBERT0.5812 BigBird0.7624 Longformer0.7443 partially intact, indicating that a subset of samples is robust to perturbations in the output layer. However, several off-diagonal blocks become populated, most notably the transitions 2â 3 and 4â 3, which reflect a systematic drift of multiple classes toward the target label. This behaviour is consistent with an additive shift applied directly to the output logits: classes with logits close to c â are the first to flip. In contrast, well-separated classes remain stable unless the bias is sufficiently large. The cross-model comparison in Table 9 shows that all ar- chitectures redirect a non-trivial fraction of predictions to the target class, but the intensity of this redirection varies markedly. DistilBERT is the most resistant, with only 18% of predictions assigned to the target and a low flip rate of 6.82%. In contrast, Longformer shows the strongest drift, with 52% of predictions redirected to the target and 42.86% of non-target samples flip- ping, suggesting a more compressible logit geometry. BERT and BigBird lie in an intermediate regime, exhibiting substan- tial but not catastrophic reweighting of outputs. Overall, these results indicate that output-space manipula- tion induces structured, model-dependent distortions rather than uniform degradation. Because the intervention operates directly on logits without modifying hidden representations, the observed effects trace the intrinsic margin structure of each classifier: models whose logit distributions are tightly clustered around the decision boundaries are more susceptible to targeted shifts. At the same time, those with larger inter-class separations retain partial robustness even under strong bias. 6.1.5. Gaussian Noise Injection The impact of Gaussian perturbations on the [CLS] repre- sentation is relatively mild across all models (see Table 10). Even with a high noise level (Ï = 0.9) applied to 60% of the most influential neurons, the weighted F1-Scores remain close to their baselines, with no model experiencing severe degrada- tion. At the same time, BigBird and Longformer maintain the highest robustness (F1-Score â 0.76 and 0.74, respectively), and DistilBERT shows the largest drop but still retains more than half of its predictive performance. This pattern suggests that the modelsâ internal representa- tions redundantly distribute semantic information across multi- ple neural dimensions, making them resilient to high-variance perturbations as long as the perturbations are zero-mean and unbiased. Unlike targeted manipulations (e.g., logit bias or weight-space attacks), Gaussian perturbations do not systemat- Table 11: Balanced weight-push attack results. ModelAccuracyF1-ScoreFlipsâTarget PredâTarget (%) (weighted) BERT0.760.68342.2714 BigBird0.380.312671.4376 DistilBERT0.560.415118.1828 Longformer0.820.80730.006 Table 12: Class-suppression weight attack results. ModelAccuracyF1-ScoreFlipsâTarget (%) PredâTarget (weighted) BERT0.760.68342.2714 BigBird0.760.674114.2928 DistilBERT0.680.58714.5516 Longformer0.840.83150.006 Table 13: Bias-only weight attack results. ModelAccuracyF1-ScoreFlipsâTarget (%) PredâTarget (weighted) BERT0.760.68342.2714 BigBird0.820.80647.1422 DistilBERT â â Longformer0.860.85167.1422 â DistilBERT is omitted because the probe assigned zero importance to all neurons, preventing the bias-only attack from being instantiated. ically push predictions toward specific classes, which explains the relatively stable behaviour observed for all architectures. 6.1.6. Weights Targeted Attack Balanced weight-push attack. This variant produces strong di- rectional shifts in BigBird, which becomes highly susceptible to target-class redirection (76% of all predictions collapse to c â , with 71.4% flips from non-target classes). In contrast, Long- former remains almost unchanged, indicating that its head weights are less aligned with probe-selected neurons and therefore more resilient. At the same time, BERT and DistilBERT exhibit mod- erate drift, suggesting partially concentrated but not overwhelm- ingly fragile weight structures. Table 11 Class-suppression weight attack. By explicitly weakening a sin- gle competitor class, this variant yields more controlled be- haviour: BigBird exhibits reduced collapse relative to the bal- anced push, while Longformer remains stable. On the other hand, DistilBERT responds to suppression with small but no- ticeable increases in flips toward the target class, suggesting a more entangled, intermediate-level head geometry. Table 12 Bias-only weight attack. As expected, modifying only the bias vector produces the mildest effects. Models whose logits al- ready provide clear inter-class separation (e.g., Longformer, Big- Bird) show minimal performance loss, whereas with BERT, the attack remains close to the baseline. Table 13 12 Overall, these results demonstrate that weight-space pertur- bations expose architecture-dependent vulnerabilities: BigBird is consistently the most vulnerable model, Longformer the most resistant, and BERT/DistilBERT occupy intermediate but dis- tinct regimes. The contrast among variants also shows that at- tacks that modify the structure of W are far more damaging than those that affect only the biases, confirming that the headâs ge- ometric configuration is a primary driver of robustness. 6.2. Emotion Detection To complement the malware-oriented experiments, the frame- work is also evaluated on a fundamentally different domain: af- fective text classification. For this, this work employs the public monologg/bert-base-cased-goemotions-original model, trained on Googleâs GoEmotions corpus [28]. As described in the ex- perimental setup, the original 28-emotion space is reduced to a six-class configuration (anger, disgust, fear, joy, sadness, sur- prise). Only single-label examples are retained, and the modelâs 28-dimensional output is remapped by restricting the logits to the six relevant indices and selecting the class with the high- est score within this subset. A stratified and balanced mini- evaluation set is used to ensure stable estimation across pertur- bation sweeps. While the malware experiments analyze each attack type in dedicated subsections, the GoEmotions case follows a different reporting strategy. Since the six-class model is not the focus of the interpretability study, which is a cross-domain stress test, the objective is to assess how the attacks generalize beyond system-call classifiers. For this reason, all neuron-level and weight-level perturbations are consolidated into a single sum- mary table. This unified presentation simplifies cross-attack comparison, highlights global sensitivity trends, and avoids in- flating the narrative with redundant per-attack subsections. The resulting table shows the weighted F1-Score for each perturbation, along with the relative variation from the base- line. In addition, attacks operating in weight space report key behavioural effects (e.g., proportion of samples redirected to the target class), since in this domain such qualitative shifts can be more informative than the raw F1-Score score alone. Table 14 summarizes the behavior of the GoEmotions model under the full set of neuron- and weight-level perturbations. As expected, the baseline achieves a high weighted F1-Score of 0.860, serving as a reference for the relative degradation ob- served across attacks. Neuron-level attacks show heterogeneous effects. Global neuron silencing is the most disruptive intervention: remov- ing 60% of neurons reduces performance to 0.300 (-65.1%), indicating a strong reliance on distributed representations. In contrast, per-class silencing behaves differently: targeting the neurons most associated with a specific emotion can improve the overall weighted F1-Score in certain configurations (peak 0.882, +2.6%), suggesting that the model contains redundan- cies or spurious correlations that can be moderated by selec- tive ablations. Noise-based attacks produce intermediate degra- dation. Random noise injection yields 0.460 (-46.5%), while Gaussian noise (Ï = 0.9) results in a softer drop to around 0.630 (-26.7%), reflecting a degree of robustness to smooth perturba- tions compared to discrete neuron removal. Logit-space perturbations also exhibit notable sensitivity. The logit bias attack yields a stable â = 8 (F1-Score = 0.624, -27.4%), whereas larger shifts collapse predictions almost en- tirely. This illustrates how small directional biases in the output layer can rapidly distort emotion predictions. Weight-space attacks produce the most varied behaviours. The balanced weight push moderately harms performance (0.788, -8.4%), although qualitative effects such as increased redirec- tion to the target class (16%) are more informative than F1- Score alone. The unbalanced push with suppression is sub- stantially more aggressive, dropping the F1-Score to 0.492 (- 42.8%) and misclassifying a large fraction (65%) of inputs into the target class. Conversely, the bias-only variant exhibits near- baseline behaviour (0.865, +0.6%), confirming that controlled modifications to the bias vector have limited global impact un- less accompanied by changes to the weight matrix. Overall, the GoEmotions experiments reveal a pattern con- sistent with the malware models: Global ablations and unstruc- tured noise are highly damaging; selective manipulations can be either harmful or beneficial; and weight-space directional attacks provide powerful, configuration-dependent mechanisms for steering classifier behaviour. 7. Discussion This section discusses the proposed training-free, neuron- level intervention framework and presents experimental results on malware detection and GoEmotions. Across tasks and model architectures, the results consis- tently indicate that task-relevant information is distributed across broad and overlapping sets of neurons rather than concentrated in a small number of isolated units. Consequently, neuron-level interventions tend to produce gradual performance degradation, requiring increasingly large fractions of ablated neurons to in- duce substantial failures. At the same time, the analyses reveal architecture- and class-dependent sensitivities, where targeted interventions on specific neuron subsets can disproportionately affect certain decision pathways. These trends provide the basis for the discussion that follows. A key strength of the framework is that it enables neuron- level analysis that is training-free, reversible, and model-agnostic. All interventions are performed at inference time via forward hooks, leaving model parameters unchanged and allowing im- mediate restoration after each run. By relying on [CLS] rep- resentations and a unified extraction procedure, the pipeline is consistent across BERT-like encoders and domains, enabling direct cross-domain comparisons. Methodologically, the frame- work treats explainability as a causal process: neuron impor- tance is first estimated using a lightweight linear probe, and the same neurons are then intervened upon to verify their func- tional impact. This closes the gap between attribution and cau- sation and yields quantitative robustness curves that relate per- formance degradation to the fraction of ablated neurons, both globally and per class. The design further emphasizes repro- 13 Table 14: Robustness of the GoEmotions model under neuron- and weight-level attacks (weighted F1-Score). AttackConfigF1-Score (weighted) âF1-Score (%) Baselineâ0.860â Global neuron silencing60% neurons silenced globally0.300-65.1 Per-class silencingBest run (class 1), peak F1-Score = 0.8820.882+2.6 Random noise injectionΔ = 1.5 applied to top neurons0.460-46.5 Gaussian noiseÏ = 0.9 (estimated)0.630-26.7 Logit biasBest stable point (â = 8). Higher â collapses predictions (F1-Scoreâ to 0.192).0.624-27.4 Weight attack (balanced push) â = 0.2, p = 20%, cols = 200. Flipsâtarget = 8 (16%). Overall predicted as target = 30%.0.788-8.4 Weight attack (unbalanced + suppression) â = 0.5, p = 25%, suppress class 4. Flipsâtarget = 29 (58%). Overall predicted as target = 65%.0.492-42.8 Weight attack (bias-only)â = 0.6, bias-only adjustment. Flipsâtarget = 1. Stable behaviour (near-baseline).0.865+0.6 ducibility through controlled sweeps, explicit logging of neu- ron indices and layer scopes, and systematic checks of baseline recovery. Plus, to our knowledge, in the context of neuron-level robustness for cybersecurity and language under a single frame- work, this is one of the first pipelines that unifies explainabil- ity and adversarial-style interventions (via systematic silenc- ing/attenuation) to reveal concrete failure budgets-how much targeted ablation is needed before performance breaks. At the same time, the results reveal several limitations. In many cases, large fractions of neurons must be silenced to match the effect of simple input-space perturbations, underscoring the intrinsic robustness of the evaluated models to localized neu- ron ablations and the distributed nature of their internal rep- resentations. The current implementation focuses exclusively on [CLS] representations for tractability, omitting token-level structure and attention pathways that may influence certain be- haviors. In addition, the reliance on a linear probe assumes ap- proximate linear separability, and alternative attribution meth- ods could alter neuron rankings and sensitivity profiles. From a validity perspective, the evaluation relies on publicly available checkpoints and controlled experimental settings; extending the analysis to broader datasets, additional architectures, and more diverse conditions would further strengthen these findings. 8. Conclusions and Future Work This work presents SYNAPSE, a framework for analyz- ing neuron-level interpretability and robustness in Transformer- based models. By extracting per-layer [CLS] activations, rank- ing neurons globally and per class, and applying fine-grained perturbations via forward hooks, SYNAPSE provides a uni- fied methodology to analyze how models structure task-relevant information and respond to controlled interventions. Experi- mental results reveal a consistent, domain-independent trend: Transformer models distribute information across broad, over- lapping neuron subsets rather than concentrating it in isolated units, introducing internal redundancy that makes neuron-level attacks more costly than expected. At the same time, the frame- work reveals structural weaknesses in each architecture, in which certain decision pathways are disproportionately sensitive to targeted perturbations, indicating non-uniform internal organi- zation and reliance on narrow activation patterns. Complemen- tary manipulations in logit space and weight space further show that small, structured changes can redirect predictions with min- imal global degradation, highlighting attack surfaces not cap- tured by neuron-centric analysis alone. Overall, the findings show that SYNAPSE supports interpretable inspection of inter- nal representations and acts as a practical tool for stress-testing robustness in both cybersecurity and NLP applications. Future work may extend SYNAPSE by developing neuron- level defense mechanisms that detect or mitigate targeted ma- nipulations in realistic, security-critical deployments. Addi- tional directions include exploring multimodal and cross-domain scenarios, where neuron attribution patterns may differ across textual, behavioral, and sensory inputs, and integrating inter- active visualization tools to improve accessibility. Finally, de- ploying the framework in federated or distributed learning set- tings could provide insights into neuron-level robustness when models are trained collaboratively across heterogeneous devices and threat landscapes. Acknowledgment This work was supported by (a) MCIN/AEI/10.13039/50110 0011033/FEDER under grant PID2021-122466OB-I00, and (b) the Swiss Federal Office for Defense Procurement (armasuisse) with the TITAN project. References [1] C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1 (5) (2019) 206â215. doi:10.1038/s42256-019-0048-x. [2] A. B. Arrieta, N. DĂaz-RodrĂguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. GarcĂa, S. Gil-LĂłpez, D. Molina, 14 R. Benjamins, R. Chatila, F. Herrera, Explainable arti- ficial intelligence (xai): Concepts, taxonomies, opportu- nities and challenges toward responsible AI, Information Fusion 58 (2020) 82â115. doi:10.1016/j.inffus.2 019.12.012. [3] M. Danilevsky, K. Qian, R. Aharonov, Y. Katsis, B. Kawas, P. Sen, A survey of the state of explainable AI for natural language processing, in: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Asso- ciation for Computational Linguistics and the 10th Inter- national Joint Conference on Natural Language Process- ing, Association for Computational Linguistics, Suzhou, China, 2020, p. 447â459. doi:10.18653/v1/2020.a acl-main.46. [4] European Parliament and Council, Regulation (eu) 2024/1689 of 13 june 2024 establishing harmonised rules on artificial intelligence (ai act), Official Journal of the Eu- ropean Union, eUR-Lex (2024). [5] K. Cho, B. van MerriĂ«nboer, D. Bahdanau, Y. Bengio, On the properties of neural machine translation: Encoderâ decoder approaches, in: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statisti- cal Translation, 2014, p. 103â111. doi:10.3115/v1/W 14-4012. [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ć. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). [7] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, in: Advances in Neural Information Processing Systems, Vol. 33, 2020, p. 1877â1901. [8] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: Pre- training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT 2019, 2019, p. 4171â4186. [9] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, arXiv preprint arXiv:1907.11692 (2019). doi:10.48550/arX iv.1907.11692. [10] P. M. SĂĄnchez SĂĄnchez, A. H. CeldrĂĄn, G. Bovet, G. M. PĂ©rez, Transfer learning in pre-trained large language models for malware detection based on system calls, in: MILCOM 2024 - 2024 IEEE Military Communications Conference (MILCOM), 2024, p. 853â858. doi:10.1 109/MILCOM61039.2024.10773857. [11] J. Li, W. Monroe, D. Jurafsky, Understanding neural net- works through representation erasure (2016). arXiv: 1612.08220, doi:10.48550/arXiv.1612.08220. URL https://arxiv.org/abs/1612.08220 [12] A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, J. Glass, Identifying and controlling important neu- rons in neural machine translation, arXiv preprint arXiv:1811.01157 (2018). [13] A. Radford, R. JĂłzefowicz, I. Sutskever, Learning to gen- erate reviews and discovering sentiment (2017). arXiv: 1704.01444, doi:10.48550/arXiv.1704.01444. URL https://arxiv.org/abs/1704.01444 [14] R. Song, S. He, S. Jiang, Y. Xian, S. Gao, K. Liu, Z. Yu, Does large language model contain task-specific neurons?, in: Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing, 2024, p. 7101â7113. doi:10.18653/v1/2024.emnlp-mai n.403. [15] J. Li, W. Monroe, D. Jurafsky, Understanding neural networks through representation erasure, arXiv preprint arXiv:1612.08220 (2017). doi:10.48550/arXiv.161 2.08220. [16] A. Radford, R. JĂłzefowicz, I. Sutskever, Learning to gen- erate reviews and discovering sentiment, arXiv preprint arXiv:1704.01444 (2017). doi:10.48550/arXiv.170 4.01444. [17] F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, A. Bau, J. Glass, What is one grain of sand in the desert? analyzing individual neurons in deep nlp models, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, p. 6309â6317. [18] N. Durrani, H. Sajjad, F. Dalvi, Y. Belinkov, Analyzing in- dividual neurons in pre-trained language models, in: Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, p. 4865â 4880. doi:10.18653/v1/2020.emnlp-main.395. [19] F. Dalvi, H. Sajjad, N. Durrani, NeuroX library for neuron analysis of deep NLP models, in: D. Bollegala, R. Huang, A. Ritter (Eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 3: System Demonstrations), Association for Compu- tational Linguistics, Toronto, Canada, 2023, p. 226â234. doi:10.18653/v1/2023.acl-demo.21. URL https://aclanthology.org/2023.acl-demo. 21/ [20] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks (2014). arXiv:1312.6199, doi:10.48550/a rXiv.1312.6199. URL https://arxiv.org/abs/1312.6199 [21] I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples (2015). arXiv:1412.6 572, doi:10.48550/arXiv.1412.6572. URL https://arxiv.org/abs/1412.6572 15 [22] C. Pravin, I. Martino, G. Nicosia, V. Ojha, Adversarial robustness in deep learning: attacks on fragile neurons, in: International Conference on Artificial Neural Networks, Springer, 2021, p. 16â28. [23] Z. Che, S. Casper, R. Kirk, A. Satheesh, S. Slocum, L. E. McKinney, R. Gandikota, A. Ewart, D. Rosati, Z. Wu, Z. Cai, B. Chughtai, Y. Gal, F. Huang, D. Hadfield- Menell, Model tampering attacks enable more rigorous evaluations of LLM capabilities (2025). arXiv:2502 .05209. URL https://arxiv.org/abs/2502.05209 [24] F. Dalvi, H. Sajjad, N. Durrani, Neurox: A library for neuron analysis of deep nlp models, in: Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 3: System Demonstra- tions), Association for Computational Linguistics, 2023, p. 226â234. doi:10.18653/v1/2023.acl-demo.21. URL https://aclanthology.org/2023.acl-demo. 21/ [25] Z. Zhao, D. Dua, S. Singh, Generating natural adversarial examples, arXiv preprint arXiv:1710.11342 (2017). [26] W. Xu, D. Evans, Y. Qi, Feature squeezing: Detecting ad- versarial examples in deep neural networks, arXiv preprint arXiv:1704.01155 (2017). [27] kikaymusic, Neuron-level interpretability and robustness in llms (code repository), https://github.com/kikay music/Neuron-Level-Interpretability-and-Rob ustness-in-LLMs, gitHub repository (2025). [28] D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, S. Ravi, GoEmotions: A dataset of fine- grained emotions, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, p. 4040â4054. doi:10.18653/v1/2020 .acl-main.372. 16