Paper deep dive
How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis
Muhammad Zeeshan Karamat, Christiana Chamon Garcia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/8/2026, 6:39:29 AM
Summary
This paper investigates the fragility of on-device language model safety by localizing safety-critical parameters in LLaMA-2-7B-Chat. Using low-rank subspace analysis and parameter-level importance filtering, the authors identify that safety-sensitive behavior is concentrated in specific architectural components, particularly the MLP down_proj. Modifying only 0.19% of weights in down_proj degrades safety (53% Basic ASR, 56% GCG ASR) while preserving general utility (51.6% accuracy), suggesting a reduced fault surface for targeted hardware fault analysis and selective integrity protection in resource-constrained agentic systems.
Entities (10)
Relation Signals (7)
Llama-2-7B-Chat → hassafetycriticalcomponent → MLP down_proj
confidence 98% · Both approaches reveal highly non-uniform safety sensitivity across the network, with the MLP down_proj consistently emerging as a prominent safety-sensitive component
MLP down_proj → inducessafetydegradation → Basic ASR 53%
confidence 97% · modifying only 0.19% of model weights in down_proj yields 53% Basic ASR
MLP down_proj → inducessafetydegradation → GCG ASR 56%
confidence 97% · modifying only 0.19% of model weights in down_proj yields ... 56% GCG ASR
MLP down_proj → preservesutility → tinyBenchmarks 51.6%
confidence 96% · tinyBenchmarks accuracy remains at 51.6% compared with a 52.2% unmodified baseline
Llama-2-7B-Chat → hassafetycriticalcomponent → o_proj
confidence 95% · o_proj providing a smaller contribution
AdvBench → usedfor → safety evaluation
confidence 95% · We evaluate attack success rate (ASR) on AdvBencheval
PrisonBreak → issimilarto → Parameter-level localization
confidence 85% · Unlike approaches that directly search for attack-effective bits, our work first localizes parameters... Coalson et al. (2024) introduced PrisonBreak, demonstrating targeted bit-flip attacks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As small language models (SLMs) are increasingly deployed on resource-constrained and on-device platforms, including as components of agentic systems, the integrity of locally stored model parameters becomes an important safety concern. We investigate whether safety-sensitive behavior in LLaMA-2-7B-Chat is concentrated within a sparse subset of parameters, creating a reduced fault surface for targeted analysis. We study two complementary localization methods: low-rank safety-associated subspace analysis and parameter-level safety--utility importance filtering. Both approaches reveal highly non-uniform safety sensitivity across the network, with the MLP down_proj consistently emerging as a prominent safety-sensitive component and o_proj providing a smaller contribution. Using parameter-level localization, modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR, while tinyBenchmarks accuracy remains at 51.6% compared with a 52.2% unmodified baseline. These results motivate targeted fault analysis and selective integrity protection for language models deployed in resource-constrained, on-device, and agentic settings.
Tags
Links
- Source: https://arxiv.org/abs/2610.09000v1
- Canonical: https://arxiv.org/abs/2610.09000v1
Trouble viewing inline? Open PDF directly →
Full Text
25,253 characters extracted from source content.
Expand or collapse full text
SLMs for Agentic Systems, Paris, France, 2026 How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis Muhammad Zeeshan Karamat Affiliation: Virginia Tech Email: mzeeshan@vt.edu Christiana Chamon Affiliation: Virginia Tech Email: ccgarcia@vt.edu Abstract As small language models (SLMs) are increasingly deployed on resource-constrained and on- device platforms, including as components of agentic systems, the integrity of locally stored model parameters becomes an important safety concern. We investigate whether safety-sensitive behavior in LLaMA-2-7B-Chat is concentrated within a sparse subset of parameters, creating a reduced fault surface for targeted analysis. We study two complementary localization methods: low-rank safety-associated subspace analysis and parameter-level safety–utility importance filtering. Both approaches reveal highly non-uniform safety sensitivity across the network, with the MLP down_proj consistently emerging as a prominent safety-sensitive component and o_proj providing a smaller contribution. Using parameter-level localization, modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR, while tinyBenchmarks accuracy remains at 51.6% compared with a 52.2% unmodified baseline. These results motivate targeted fault analysis and selective integrity protection for language models deployed in resource-constrained, on-device, and agentic settings. 1 Introduction Language models are increasingly being deployed on edge and on-device platforms, including as components of resource-constrained agentic systems, offering benefits such as lower latency, improved privacy, and reduced dependence on cloud connectivity. However, local deployment also exposes model parameters to hardware-level reliability and integrity threats. Unlike prompt-based jailbreaks, faults in stored model weights can directly alter mechanisms responsible for aligned behavior and may persist across subsequent model interactions. This motivates a central question: can model safety be selectively degraded through sparse parameter modifications without broadly disrupting model functionality? A practical bit-level attack on a billion-parameter model faces a large search problem: only a small fraction of model parameters may be directly relevant to safety behavior. We therefore adopt a localization-first perspective. If safety-sensitive behavior depends disproportionately on a structured subset of the parameter space, these regions may provide a substantially reduced candidate space for hardware fault analysis. Crucially, such localization should distinguish safety-associated parameters from those that are also important for general model utility, since indiscriminate corruption may simply degrade overall model quality. We study this question in LLaMA-2-7B-Chat using two complementary approaches. First, we identify low-rank safety-associated subspaces by contrasting representations derived from safety and utility calibration data. Second, we perform parameter-level localization by identifying weights that are highly important for safety but comparatively less important for utility. Across both approaches, the MLP down_proj consistently emerges as a prominent safety-sensitive component, with a smaller contribution from the attention o_proj. Using parameter-level localization, modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR, while tinyBenchmarks accuracy remains at 51.6% compared with a 52.2% unmodified baseline. Our contributions are threefold: (1) we analyze safety localization at both subspace and parameter levels; (2) we show that safety-sensitive parameter support is strongly concentrated in specific architectural components, particularly down_proj; and (3) we demonstrate that substantial safety degradation can be induced by modifying less than 0.2% of model weights while largely preserving measured general utility, motivating a reduced search space for future bit-level fault analysis and selective integrity protection in on-device language models, including those used as components of resource-constrained agentic systems. 1.1 Threat Model We consider a white-box adversary targeting a safety-aligned language model deployed locally on an edge or on-device platform, including models that may serve as components of resource-constrained agentic systems. The adversary has access to the model parameters and may perform offline analysis to identify safety-sensitive regions, but does not retrain or fine-tune the model. The objective is to induce sparse parameter perturbations that degrade safety-aligned behavior while minimizing degradation of benign model utility. This threat is particularly acute for SLMs deployed as components of agentic systems. Unlike a single-turn chat interaction, an agent’s safety behavior is invoked repeatedly and autonomously across a session, at each planning step, tool call, and intermediate action, often without a human in the loop to catch an unsafe output. A parameter-level fault therefore does not need to be re-triggered per turn, as a prompt-based jailbreak would: once induced, it persists silently across every subsequent invocation of the model within the agent loop, including invocations the user never directly observes (for example, internal planning or tool-selection steps). This makes the integrity of locally stored weights a distinct concern from prompt-level robustness in agentic deployments. Our fault model does not require the adversary to modify inference-time prompts. We nevertheless evaluate the perturbed model under both standard harmful prompts and GCG adversarial suffixes, with the latter serving as a stress test of whether the induced safety degradation persists under a stronger adversarial prompting setting. The current experiments simulate parameter-level interventions rather than physical hardware faults. We therefore study localization of a reduced safety-sensitive parameter space as a precursor to fault analysis; mapping these parameters to a small number of physically realizable bit flips under specific numerical formats, memory architectures, and fault mechanisms is left to future work. 2 Related Work Small and on-device language models for agentic systems. Recent work has highlighted small language models (SLMs) as efficient building blocks for agentic systems. Belcak et al. (2025) argue that SLMs are particularly well suited to specialized and repetitive tasks within agentic workflows, where reduced computational cost can enable more efficient deployment. Complementary work on MobileLLM (Liu et al., 2024) develops sub-billion-parameter models specifically for mobile deployment and demonstrates their capability on common on-device tasks, including API calling. Although there is no universally agreed-upon parameter threshold for SLMs, recent literature commonly includes models with up to approximately 7B parameters, with LLaMA-2-7B and Mistral-7B serving as representative examples (Corradini et al., 2025). Recent systems work further demonstrates that models at this scale can be executed locally on resource-constrained devices, including smartphones (Xue et al., 2024). These trends motivate examining the integrity of safety mechanisms when language models are deployed locally as components of resource-constrained and agentic systems. Safety localization in language models. Recent work has identified localized internal mechanisms associated with refusal and safety alignment. Arditi et al. (2024) showed that refusal behavior in instruction-tuned language models can be mediated by a low-dimensional direction in the residual stream. Wei et al. (2024) further identified sparse safety-relevant regions in parameter space at both parameter and rank levels while disentangling them from regions important for general utility. Building on this safety–utility perspective, we investigate whether localized safety-sensitive regions can provide a reduced candidate space for hardware-level fault analysis in on-device language models. Bit-flip and fault attacks on neural networks. Bit-flip attacks modify the binary representation of model parameters stored in memory. Rakin et al. (2019) introduced Progressive Bit Search, showing that a small number of targeted bit flips can severely degrade quantized neural networks. More recently, Coalson et al. (2024) introduced PrisonBreak, demonstrating targeted bit-flip attacks against safety-aligned language models while largely preserving benign functionality. Zahran et al. (2025) further studied fault-induced jailbreaking across FP16, FP8, INT8, and INT4 representations, showing that numerical format affects both attack effectiveness and vulnerable components. Unlike approaches that directly search for attack-effective bits, our work first localizes parameters that are disproportionately important for safety relative to utility, with the goal of exposing a compact fault surface for subsequent bit-level analysis. 3 Methodology Our goal is to identify a small subset of model parameters that contribute disproportionately to safety-aligned behavior while remaining comparatively less important for general model utility. We study this using two complementary approaches: (1) low-rank safety subspace localization, which identifies safety-associated directions in representation space, and (2) parameter-level safety–utility localization, which identifies individual safety-sensitive weights. We build on the safety–utility localization framework of Wei et al. (2024) and investigate whether the resulting localized regions can provide a reduced candidate space for hardware fault analysis in on-device language models. 3.1 Calibration Datasets We use two calibration datasets representing safety-aligned and general-purpose behavior. The safety set sD_s consists of harmful instructions from AdvBench (Zou et al., 2023), paired with the model’s safety-aligned refusal responses. The utility set uD_u uses Alpaca-Cleaned, a cleaned version of Stanford Alpaca (Taori et al., 2023), containing benign instruction–response pairs with safety-related examples removed. For both localization approaches, prompt tokens are masked and only response-token activations are used to characterize safety- and utility-associated behavior. 3.2 Low-Rank Safety Subspace Localization We first characterize safety- and utility-associated representation subspaces within individual transformer projection modules. For a linear module with weight matrix W∈ℝdout×dinW ^d_out× d_in, we collect response-token input activations XsX_s and XuX_u independently from sD_s and uD_u. We form the corresponding module outputs As=XsW⊤,Au=XuW⊤,A_s=X_sW , A_u=X_uW , (1) and compute truncated singular value decompositions As≈UsΣsVs⊤,Au≈UuΣuVu⊤.A_s≈ U_s _sV_s , A_u≈ U_u _uV_u . (2) Using the right singular vectors, we construct output-space projectors Ps=VsVs⊤,Pu=VuVu⊤,Pu⟂=I−Pu.P_s=V_sV_s , P_u=V_uV_u , P_u =I-P_u. (3) We define a utility-orthogonalized safety-associated weight component as ΔW=Pu⟂PsW, W=P_u P_sW, (4) and modify the module according to W′=W−ΔW.W =W- W. (5) This construction suppresses a weight component associated with safety-aligned behavior while reducing overlap with directions identified from benign utility data. To examine whether this intervention can be localized to a smaller parameter set, we sparsify ΔW W by globally ranking its entries by absolute magnitude. For a retained fraction ρ, we define ΔWij(ρ)=ΔWij[|ΔWij|≥τρ]. W^(ρ)_ij= W_ijI\! [| W_ij|≥ _ρ ]. (6) where τρ _ρ is the global threshold corresponding to the desired fraction. The sparse intervention is then W′=W−ΔW(ρ)W =W- W^(ρ). We evaluate this procedure across all projection modules and under restrictions to specific architectural components. 3.3 Parameter-Level Safety–Utility Localization Our second approach directly identifies individual weights that are important for safety-aligned behavior but comparatively less important for benign utility. For a projection module with weight matrix W∈ℝdout×dinW ^d_out× d_in, let Xin∈ℝdin×nX_in ^d_in× n denote its response-token input activations over a calibration dataset. Following an activation-aware importance criterion, we assign each weight WijW_ij an importance score I(Wij)=|Wij|‖Xin,j,:‖2,I(W_ij)=|W_ij| \|X_in,j,: \|_2, (7) where Xin,j,:∈ℝnX_in,j,: ^n denotes the activations of input feature j across the calibration tokens, and ‖Xin,j,:‖2 \|X_in,j,: \|_2 is its ℓ2 _2 norm. Thus, the importance score combines the magnitude of each weight with the activation magnitude of its corresponding input feature. We compute Eq. 7 independently on sD_s and uD_u, producing safety scores IsI_s and utility scores IuI_u, respectively. Let sS_s denote the set of highest-ranked weights under IsI_s, and let uS_u denote the corresponding utility-important set. We define the safety-specific parameter set as spec=s∖u.S_spec=S_s _u. (8) This set difference removes weights that are important for both safety and general instruction-following behavior, leaving candidates that are comparatively more specific to safety. We evaluate the functional effect of these parameters through sparse pruning: wij′=0,(i,j)∈spec,wij,otherwise.w _ij= cases0,&(i,j) _spec,\\ w_ij,&otherwise. cases (9) The resulting sparse masks identify individual candidate parameters whose removal disproportionately affects safety-associated behavior relative to the utility calibration set. 3.4 Architectural Localization and Sparse Intervention Beyond parameter sparsity, we examine whether safety-sensitive structure is concentrated within particular transformer components. We consider the attention projections (q_proj, k_proj, v_proj, and o_proj) and MLP projections (gate_proj, up_proj, and down_proj) across transformer layers. For the low-rank approach, we restrict the projected weight intervention ΔW(ρ) W^(ρ) to different groups of projection modules. For parameter-level localization, we similarly restrict pruning of specS_spec to selected projection types and combinations of components. This analysis tests whether safety-sensitive interventions require broad modification of the network or can be localized to specific architectural regions, thereby reducing the candidate space for subsequent fault analysis. 4 Experimental Setup Model and data. We conduct all experiments on LLaMA-2-7B-Chat (Touvron et al., 2023), which falls within the commonly used parameter range for SLMs (Section 2) and provides a well-characterized, safety-aligned testbed with established attack and evaluation tooling. AdvBench (Zou et al., 2023) is split into 420 harmful instructions for attribution, AdvBenchattr_attr, and 100 for evaluation, AdvBencheval_eval. For safety attribution, we collect refusal responses from the aligned model on AdvBenchattr_attr. Utility attribution uses Alpaca-Cleaned, a cleaned version of Stanford Alpaca (Taori et al., 2023), with safety-related examples removed. Safety evaluation. We evaluate attack success rate (ASR) on AdvBenchevalAdvBench_eval. Following Zou et al. (2023), an attack is considered successful when the generated response lacks key refusal patterns. We report Basic ASR on harmful prompts without an adversarial suffix and GCG ASR using GCG adversarial suffixes (Zou et al., 2023). Utility evaluation. General utility is measured using tinyBenchmarks (Maia Polo et al., 2024). The unmodified model achieves 52.2% accuracy, and we report post-intervention accuracy to measure broader capability degradation. Interventions. For low-rank localization, we vary the retained fraction of the safety-associated weight intervention and its architectural scope. For parameter-level localization, safety- and utility-important weights are combined using the set-difference criterion in Eq. 8. We evaluate interventions across all projection modules, MLP projections, and selected individual projection types. Reported weight fractions are measured relative to the full model. 5 Results 5.1 Low-Rank Intervention Sparsity and Localization Table 1 shows that retaining only the top 10% of the low-rank intervention preserves substantial safety degradation, while effectiveness drops sharply at 1%, indicating concentration among higher-magnitude modifications. Restricting the intervention to MLP projections retains much of the effect, and down_proj alone remains sufficient to induce notable safety degradation. Utility remains close to the 52.2% baseline across the more localized settings. Table 1: Low-rank intervention results across sparsity levels and architectural components. Utility denotes tinyBenchmarks accuracy; the unmodified model achieves 52.2%. Modules Top fraction Basic ASR GCG ASR Utility All modules 1.0 87% 98% 49.3% All modules 0.1 71% 85% 50.1% All modules 0.01 36% 25% 50.7% All modules 0.001 20% 18% 51.8% MLP only 1.0 77% 89% 49.8% MLP only 0.1 54% 64% 51.1% down_proj 1.0 51% 43% 51.2% down_proj 0.1 32% 26% 51.4% 5.2 Parameter-Level Safety–Utility Localization Table 2: Sparse interventions using parameter-level safety–utility localization. Weight fraction is measured relative to the full model. Utility denotes tinyBenchmarks accuracy; the unmodified model achieves 52.2%. Target component Weights (%) Basic ASR GCG ASR Utility All modules 2.95 91% 97% 47.1% MLP only 1.87 86% 94% 49.7% down_proj + o_proj 0.23 58% 60% 51.3% down_proj 0.19 53% 56% 51.6% o_proj 0.04 20% 14% 52.0% Table 2 shows strong architectural concentration in the parameter-level localization results. Restricting the intervention from all modules to MLP projections retains most of the safety effect, while further localization to down_proj and o_proj reduces the intervention to 0.23% of model weights while still achieving 58% Basic ASR and 60% GCG ASR. In particular, down_proj alone modifies only 0.19% of the model while yielding 53% Basic ASR and 56% GCG ASR, whereas o_proj alone is substantially weaker. Across both localization approaches, down_proj consistently emerges as the most prominent safety-sensitive component, with utility remaining near the unmodified baseline for these highly localized interventions. 6 Discussion Our results indicate that safety-sensitive parameters are highly non-uniform across LLaMA-2-7B-Chat. Both localization approaches consistently identify the MLP down_proj as the most prominent component, with o_proj providing a smaller complementary contribution. The parameter-level intervention targeting only down_proj modifies 0.19% of model weights while still producing substantial safety degradation, with tinyBenchmarks accuracy remaining close to the unmodified baseline. This suggests that the safety-relevant parameter space is substantially smaller than the full model. Parameter sparsity, however, should not be interpreted directly as bit-level vulnerability. Even 0.19% of a 7B-parameter model corresponds to millions of weights, whereas a practical hardware fault attack would ideally require only a small number of physically realizable bit modifications. Bit-level effects will also depend on numerical representation, memory architecture, and fault mechanism. Our current experiments therefore localize a reduced safety-sensitive parameter space, but do not yet demonstrate an end-to-end hardware attack. The utility results provide initial evidence that localized interventions can degrade safety without proportionally degrading general capability, although tinyBenchmarks provides only a compact estimate of utility. These findings are particularly relevant to on-device and resource-constrained deployments, including language models used as components of agentic systems, where locally stored parameters may remain exposed throughout repeated model interactions. Future work will study whether these localized regions are enriched for high-impact bit-level faults, compare against random and magnitude-based baselines, and examine selective integrity protection for safety-sensitive regions in such deployments. 7 Conclusion We show that safety-sensitive parameters in LLaMA-2-7B-Chat are structurally concentrated, with down_proj consistently emerging as the most prominent component across both low-rank and parameter-level localization. Parameter-level localization reduces the intervention to only 0.19% of model weights while achieving 53% Basic ASR and 56% GCG ASR, with utility remaining close to the unmodified baseline. These results suggest that safety-relevant behavior depends disproportionately on a small and structured portion of the parameter space, creating a reduced candidate surface for subsequent fault analysis. This concentration is particularly relevant for on-device and resource-constrained language models, including models used as components of agentic systems, where locally stored parameters may be exposed to hardware reliability and integrity faults. Future work will investigate whether these localized regions contain a similarly sparse set of high-impact, physically realizable bit-level vulnerabilities and whether selective integrity protection can efficiently harden such safety-sensitive regions. References Arditi et al. (2024) A. Arditi, O. B. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda Refusal in language models is mediated by a single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Belcak et al. (2025) P. Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P. Molchanov Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153. Cited by: §2. Coalson et al. (2024) Z. Coalson, J. Woo, S. Chen, Y. Sun, L. Yang, P. Nair, B. Fang, and S. Hong PrisonBreak: jailbreaking large language models with fewer than twenty-five targeted bit-flips. arXiv preprint arXiv:2412.07192. Cited by: §2. Corradini et al. (2025) F. Corradini, M. Leonesi, and M. Piangerelli State of the art and future directions of small language models: a systematic review. Big Data and Cognitive Computing 9 (7), p. 189. External Links: Document Cited by: §2. Liu et al. (2024) Z. Liu, C. Zhao, F. Iandola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi, L. Lai, and V. Chandra MobileLLM: optimizing sub-billion parameter language models for on-device use cases. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 32431–32454. Cited by: §2. Maia Polo et al. (2024) F. Maia Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin TinyBenchmarks: evaluating llms with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 34303–34326. Cited by: §4. Rakin et al. (2019) A. S. Rakin, Z. He, and D. Fan Bit-flip attack: crushing neural network with progressive bit search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: §2. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford alpaca: an instruction-following llama model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: §3.1, §4. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §4. Wei et al. (2024) B. Wei, K. Huang, Y. Huang, T. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and P. Henderson Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the 41st International Conference on Machine Learning, Cited by: §2, §3. Xue et al. (2024) Z. Xue, Y. Song, Z. Mi, X. Zheng, Y. Xia, and H. Chen PowerInfer-2: fast large language model inference on a smartphone. External Links: 2406.06282 Cited by: §2. Zahran et al. (2025) N. Zahran, A. Tahmasivand, I. Alouani, K. N. Khasawneh, and M. Fouda On jailbreaking quantized language models through fault injection attacks. In Proceedings of the Great Lakes Symposium on VLSI 2025, p. 554–561. External Links: Document Cited by: §2. Zou et al. (2023) A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: §3.1, §4, §4.