Paper deep dive
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
Deepanshu Pandey, Arnav Chavan, Nahush Lele, Sankalp Dayal, Deepak Gupta
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/24/2026, 5:52:44 AM
Summary
This paper proposes Jacobian-Guided Noise Injection, a training strategy to enhance the quantization robustness of Large Language Models (LLMs). The authors identify the softmax operator in self-attention mechanisms as a bottleneck for quantization stability due to its state-dependent Jacobian and sensitivity to outliers. By theoretically linking the suppression of the Jacobian Frobenius norm to bounded quantization error, they introduce a method that injects zero-mean Gaussian noise into pre-attention logits. The noise variance is dynamically derived from the local Jacobian norm, allowing for adaptive regularization that mitigates error amplification. Empirical results on models like Llama-3.2-3B, Qwen2.5-3B, and SigLIP demonstrate significant improvements in accuracy and perplexity under low-bit quantization settings (e.g., W4A4) compared to standard Post-Training Quantization (PTQ) methods.
Entities (13)
Relation Signals (11)
Jacobian-Guided Noise Injection → evaluatedon → Llama-3.2-3B
confidence 98% · We evaluate the method on SOTA LLM architectures... Llama-3.2-3B
Jacobian-Guided Noise Injection → evaluatedon → SigLIP
confidence 98% · We evaluate the method on SOTA LLM architectures... SigLIP
Jacobian-Guided Noise Injection → evaluatedon → Qwen2.5 3B
confidence 98% · We evaluate the method on SOTA LLM architectures... Qwen2.5-3B
Jacobian-Guided Noise Injection → improves → Quantization Robustness
confidence 95% · demonstrates improved robustness over popular PTQ methods.
Jacobian-Guided Noise Injection → uses → Jacobian Frobenius Norm
confidence 94% · variance derived directly from the Jacobian Frobenius norm.
Softmax → isbottleneckfor → Quantization Stability
confidence 93% · We identify the softmax operator as a bottleneck for quantization stability due to its sensitivity to outliers and state-dependent Jacobian.
Jacobian-Guided Noise Injection → achievesgainon → ImageNet-1K
confidence 92% · gives up to +37% relative gains on Top-1 accuracy on ImageNet-1K for SigLIP
Jacobian-Guided Noise Injection → achievesgainon → WikiText
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bottleneck for quantization stability due to its sensitivity to outliers and state-dependent Jacobian. We theoretically establish that suppressing the norm of this Jacobian helps in bounding quantization-induced performance degradation. Based on this, we propose Jacobian-Guided Noise Injection, a training strategy that injects zero-mean Gaussian noise into pre-attention logits, with variance derived directly from the Jacobian Frobenius norm. Unlike prior approaches that rely on heuristic or penalise jacobian directly, our method provides a way to identify the optimal noise variance based on the local attention sensitivity. We evaluate the method on SOTA LLM architectures, where it demonstrates improved robustness over popular PTQ methods. Empirical analysis reveals that the proposed method gives up to +37% relative gains on Top-1 accuracy on ImageNet-1K for SigLIP and improves relative perplexity by upto 40% on WikiText for language models in low bit quantisation settings, proving the efficacy of the approach.
Tags
Links
- Source: https://arxiv.org/abs/2608.20988v1
- Canonical: https://arxiv.org/abs/2608.20988v1
Trouble viewing inline? Open PDF directly →
Full Text
51,479 characters extracted from source content.
Expand or collapse full text
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models Deepanshu Pandey Affiliation: Amazon Correspondence to: deepnsp@amazon.com Arnav Chavan Affiliation: Amazon Nahush Lele Affiliation: Amazon Sankalp Dayal Affiliation: Amazon Deepak Gupta Affiliation: Amazon Abstract Quantization of Large Language Models (LLMs) is often hindered by the sensitivity of the self-attention mechanism to discretization errors. We identify the softmax operator as a bottleneck for quantization stability due to its sensitivity to outliers and state-dependent Jacobian. We theoretically establish that suppressing the norm of this Jacobian helps in bounding quantization-induced performance degradation. Based on this, we propose Jacobian-Guided Noise Injection, a training strategy that injects zero-mean Gaussian noise into pre-attention logits, with variance derived directly from the Jacobian Frobenius norm. Unlike prior approaches that rely on heuristic or penalise jacobian directly, our method provides a way to identify the optimal noise variance based on the local attention sensitivity. We evaluate the method on SOTA LLM architectures, where it demonstrates improved robustness over popular PTQ methods. Empirical analysis reveals that the proposed method gives up to +37% relative gains on Top-1 accuracy on ImageNet-1K for SigLIP and improves relative perplexity by upto 40% on WikiText for language models in low bit quantisation settings, proving the efficacy of the approach. Keywords: Quantization, Large Language Models, Softmax, Jacobian Regularization, Noise Injection 1 Introduction Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language tasks (40; 41; 39). However, deploying these models efficiently remains challenging due to their substantial computational and memory requirements (12; 37). Quantization offers a promising path to efficient deployment by reducing the precision of weights and activations (18). For instance, 4-bit weight quantization can reduce memory footprint by 4× compared to FP16 models and achieve over 3× inference speedup, even enabling deployment of 70B parameter models on mobile GPUs (24). However, naive quantization often leads to significant performance degradation (10; 24), due to approximation noise and rounding errors in lower-precision that perturb intermediate computations (32). The degradation becomes more severe at lower bit-widths (e.g., 4-bit or below) due to reduced representational capacity and the presence of activation outliers . Recent studies show that quantization-induced errors disproportionately affect mathematical reasoning, multi-step planning, and long-context tasks, motivating the development of robust quantization-aware and activation-aware methods (22; 23; 46) Figure 1: PTQ results on ImageNet-1K Top-1 accuracy for SigLIP base 16-384 with ERQ and RepQViT. A critical observation is that the self-attention mechanism in Transformers (42) is particularly sensitive to quantization errors, due to the presence of highly sensitive operations like softmax, normalization etc. We observed this phenomenon in quantised model deployment (Figure 2) where the error propagation was highly pronounced for the attention layers, leading to major diversion from expected activation values. Unlike linear layers where error propagation is bounded by fixed weight matrices, the Softmax operator in attention exhibits state-dependent sensitivity that varies dramatically based on the input distribution (20; 26). The problem is especially severe when logits contain outliers or large magnitudes, since quantization errors in these values are amplified exponentially through the softmax gate due to its exponential nonlinearity (45; 6). A small perturbation to a large logit produces a disproportionately large change in the output probability, causing catastrophic error propagation. This creates unpredictable error amplification that standard quantization techniques fail to address. In this work, we analyse the quantization error propagation through the softmax operator and derive conditions for bounding this error using norm of the softmax jacobian. Furthermore, we propose Jacobian-guided noise injection, a training strategy to improve the downstream performance of quantised model (Figure 1). Our key contributions are: 1. We analyse the relationship between spectral norm of the softmax jacobian and quantization error amplification, and show that minimizing expected loss under logit perturbation implicitly regularizes this norm. 2. We derive an expression to approximate the Jacobian Frobenius norm and use it to calibrate noise injection variance, providing a simpler alternative to heuristic approaches. 3. We demonstrate that our Jacobian-guided noise injection method improves quantization robustness on multiple LLM architectures and quantisation settings, recovering up to 37% accuracy in a W4A4 with zero inference overhead. 2 Methodology In this section, we identify the self-attention Softmax operator as the bottleneck for quantization stability in Transformers (42). We theoretically analyse the conditions required to bound this quantization error and observe that these conditions can be satisfied via an implicit Hessian regularization (1). Finally, to apply this regularisation in practical scenarios, we propose a fine-tuning framework to achieve robust quantization. 2.1 Sensitivity of Softmax Jacobian In a standard Transformer, the self-attention mechanism computes attention probabilities from the pre-activation logits. To analyze error propagation, let z∈ℝNz ^N denote a single row vector of the pre-activation logit matrix for a given query (i.e., zi=qTki/dz_i=q^Tk_i/ d). The corresponding attention probability vector a∈ℝNa ^N is computed via the Softmax function: a=S(z)=ezi∑j=1Nezja=S(z)= e^z_i _j=1^Ne^z_j (1) When the model is deployed in a quantized format, the discretization of weights and activations introduces a bounded perturbation δ∈ℝNδ ^N into the logits, such that the quantized pre-activations are zq=z+δz_q=z+δ. The error propagated into the attention distribution is: Δa=S(z+δ)−S(z) a=S(z+δ)-S(z) (2) Using first-order Taylor expansion, we approximate this error via the jacobian JS(z)∈ℝN×NJ_S(z) ^N× N: Δa≈JS(z)δ⟹‖Δa‖2≤‖JS(z)‖2||δ||2 a≈ J_S(z)δ || a||_2≤||J_S(z)||_2||δ||_2 (3) The critical vulnerability lies in the formulation of the softmax jacobian JS(z)J_S(z): JS(z)i,j=∂ai/∂zj=ai(i=j−aj)J_S(z)_i,j=∂ a_i/∂ z_j=a_i(1_i=j-a_j) (4) Unlike linear layers where input Jacobian is a constant weight matrix (e.g., ∇X(XW)=WT _X(XW)=W^T), JS(z)J_S(z) is dense and strictly state-dependent. The structure of the Jacobian provides insight into the sensitivity of the softmax output with respect to its input logits. When attention is concentrated on a single token, it results in safe, saturated regions where the Jacobian norm approaches zero, compared to highly sensitive regions where equal mass is over 2 or more tokens and quantization errors are aggressively amplified (norm decreases as mass is distributed across more tokens). Standard fine-tuning objectives do not take this into account. Consequently, an unregularized model may learn pre-activations that rest in these sensitive regions, maximizing the ‖JS(z)‖2||J_S(z)||_2. In these regimes, even a minimal quantization error ‖δ‖2||δ||_2 triggers an unpredictable, exponential amplification of Δa a, leading to catastrophic task degradation. The proposed noise injection strategy aims to implicitly induce this regularisation during training by making the injected noise a function of the jacobian state. Figure 2: Activation cosine similarity for attention outputs in Siglip-base-384. 2.2 Constraining the Softmax Jacobian To strictly bound the quantization error ‖Δa‖2|| a||_2, we must constrain the spectral norm of the Jacobian, ‖JS(z)‖2||J_S(z)||_2. Let ℒ(z)=(l∘S)(z)L(z)=(l S)(z) represent the end-to-end loss as a function of the pre-activation logits. Directly penalizing ‖JS(z)‖2\|J_S(z)\|_2 is computationally prohibitive as it is not a static parameter but a state-dependent matrix. We can establish a theoretical bound on the Jacobian by regularizing the logit Hessian, ∇z2ℒ(z)∇^2_zL(z). Applying the multivariate chain rule, the logit Hessian decomposes as: ∇z2ℒ(z)=JS(z)T∇a2ℓ(a)JS(z)+∑k=1N∂ℓ∂ak∇z2Sk(z)∇^2_zL(z)=J_S(z)^T∇^2_a (a)J_S(z)+ _k=1^N ∂ ∂ a_k∇^2_zS_k(z) (5) Let Ha=∇a2ℓ(a)H_a=∇^2_a (a) denote the activation Hessian. Using Gauss-Newton approximation and ignoring the second-order residual term, the relationship simplifies to: ∇z2ℒ(z)≈JS(z)THaJS(z)∇^2_zL(z)≈ J_S(z)^TH_aJ_S(z) (6) Near a local optimum, the loss landscape is locally convex, meaning HaH_a is positive semi-definite (Ha⪰0H_a 0). Let λmin>0 _ >0 denote the smallest positive eigenvalue of HaH_a. By the properties of positive semi-definite matrices, we can bound the trace of the Hessian: Tr(JS(z)THaJS(z))≥λminTr(JS(z)TJS(z))Tr (J_S(z)^TH_aJ_S(z) )≥ _ Tr (J_S(z)^TJ_S(z) ) (7) Algorithm 1 Jacobian-Guided Noise Injection 0: Model ℳM, update interval T, scale factor α 1: for each training step t do 2: if tmodT=0t T=0 then 3: Forward pass to compute attention P(ℓ)P^( ) per layer 4: for each layer ℓ do 5: Compute ‖JS(ℓ)‖F2||J_S^( )||_F^2 using Eq. 12 6: Update σi(ℓ)←α⋅b,h[‖JS‖F,i2] _i^( )←α· E_b,h[||J_S||_F,i^2] 7: Clamp: σ←clamp(σ,σmin,σmax)σ (σ, _ , _ ) 8: end for 9: end if 10: Compute logits: Z=QKT/dZ=QK^T/ d 11: Sample noise: ϵi∼(0,σi2) _i (0, _i^2) 12: Perturb: Z~=Z+ϵ Z=Z+ε 13: Apply softmax: A~=Softmax(Z~) A=Softmax( Z) 14: Compute loss ℒ(A~)L( A) and backpropagate 15: end for By the definition of the Frobenius norm, Tr(JS(z)TJS(z))=‖JS(z)‖F2Tr(J_S(z)^TJ_S(z))=||J_S(z)||_F^2. Since the spectral norm satisfies ‖JS(z)‖22≤‖JS(z)‖F2||J_S(z)||_2^2≤||J_S(z)||_F^2, we obtain: ‖JS(z)‖2≤Tr(∇z2ℒ(z))λmin||J_S(z)||_2≤ Tr(∇^2_zL(z)) _ (8) Equation 8 directly bounds the error amplification defined in Equation 3. Therefore, our ideal regularization objective should penalize the trace of the logit Hessian: ℒideal(z)=ℒ(z)+λ⋅Tr(∇z2ℒ(z))L_ideal(z)=L(z)+λ·Tr(∇^2_zL(z)) (9) where λ>0λ>0 controls regularization strength. However, computing this requires continuous backward passes of second-order derivatives, which is computationally intractable for large Transformers. Table 1 shows how to efficiently approximate this penalty using only first-order gradients. 2.3 Stochastic Perturbation We now derive a first-order approximation to ℒidealL_ideal. Consider modifying the objective to minimize the expected loss under a continuous, zero-mean perturbation ϵ∈ℝNε ^N applied directly to the logits: ϵ[ℒ(z+ϵ)]E_ε[L(z+ε)] . We analyze the behavior of this objective via a second-order Taylor series expansion of the perturbed loss around the unperturbed logits z: ℒ(z+ϵ)=ℒ(z)+∇zℒ(z)Tϵ+12ϵT∇z2ℒ(z)ϵ+(‖ϵ‖3)L(z+ε)=L(z)+ _zL(z)^Tε+ 12ε^T∇^2_zL(z)ε+O(||ε||^3) (10) We explicitly define the perturbation ϵε as isotropic Gaussian noise sampled from (0,σ2I)N(0,σ^2I). By leveraging its statistical properties ([ϵ]=0E[ε]=0 and [ϵϵT]=σ2IE[ε^T]=σ^2I), taking the expectation of the Taylor expansion causes the first-order gradient term to vanish completely: ϵ∼(0,σ2I)[ℒ(z+ϵ)]≈ℒ(z)+σ22Tr(∇z2ℒ(z))E_ε (0,σ^2I)[L(z+ε)] (z)+ σ^22Tr(∇^2_zL(z)) (11) This derivation yields a critical similarity: minimizing the expected loss under Gaussian logit perturbation matches our requirement derived in Section 2.2. Thus, the perturbation variance σ2σ^2 acts as the implicit regularization strength λ in Equation 9, with the correspondence λ=σ2/2λ=σ^2/2. 2.4 Jacobian-Guided Noise Injection Table 1: Zero shot PTQ results for Llama-3.2-3B and Qwen2.5-3B. Acc is average accuracy of over 7 benchmarks (↑ ). AWQ GPTQ SpinQuant Model Precision Variant PPL ↓ Acc ↑ PPL ↓ Acc ↑ PPL ↓ Acc ↑ Llama-3.2-3B W4A4 Base 105.06 42.06 454.35 40.31 10.72 60.25 Ours 104.34 42.20 496.38 41.00 11.18 60.15 W4A8 Base 9.96 66.65 37.40 64.81 8.31 64.48 Ours 9.86 67.35 31.47 66.36 8.47 65.17 W6A6 Base 10.80 64.07 10.87 63.69 8.03 65.98 Ours 10.55 64.85 11.12 64.16 8.17 66.81 W8A8 Base 9.73 66.62 9.69 66.69 7.97 65.95 Ours 9.68 67.55 9.86 67.25 8.12 66.72 Qwen2.5-3B W4A4 Base 8121.24 40.58 5172.39 38.58 10.49 60.34 Ours 3281.04 41.56 4207.85 38.96 10.33 62.62 W4A8 Base 10.80 67.84 14.13 66.65 8.55 67.35 Ours 11.01 68.38 12.21 66.11 8.35 65.66 W6A6 Base 15.29 65.12 28.42 59.97 8.36 67.25 Ours 13.63 65.38 21.52 60.10 8.18 67.44 W8A8 Base 10.80 67.84 10.98 67.55 8.32 67.66 Ours 10.80 68.38 10.82 67.61 8.11 67.45 Table 1 established that Gaussian noise injection with variance σ2σ^2 implicitly regularizes the Hessian trace with strength λ=σ2/2λ=σ^2/2. However, a fixed global σ treats all layers and positions uniformly, ignoring the fact that the Jacobian norm (and hence quantization sensitivity) varies dramatically across the network. Positions on sensitive regions (Section 2.1) require stronger regularization than those in saturated regions. We therefore propose deriving the noise variance directly from the local Jacobian Frobenius norm, yielding an adaptive scheme that concentrates regularization precisely where it is needed. For softmax output p=S(z)p=S(z), the Jacobian JS=diag(p)−ppTJ_S=diag(p)-p^T admits a closed-form Frobenius norm: ‖JS‖F2=‖p‖22−2‖p‖33+‖p‖24||J_S||_F^2=||p||_2^2-2||p||_3^3+||p||_2^4 (12) where ‖p‖k=(∑ipik)1/k||p||_k=( _ip_i^k)^1/k. This expression requires only element-wise operations and reductions (no explicit Jacobian materialization) adding negligible overhead to the forward pass. We set the noise standard deviation proportional to the Jacobian norm: σi=α⋅‖JS‖F,i2 _i=α· ||J_S||_F,i^2, where α is a scaling hyperparameter. This ensures that noise variance scales linearly with the Jacobian norm squared, preserving the correspondence λ∝σ2λ σ^2 from Equation 11. Each query position receives noise calibrated to its local sensitivity: ϵi∼(0,σi2),σi=α⋅b,h[‖JS‖F,i2] _i (0, _i^2), _i=α· E_b,h[||J_S||_F,i^2] (13) We adopt rowwise (per-position) noise rather than a static global value because the Jacobian norm varies substantially even within a single attention head. Rowwise injection applies stronger perturbation to high-sensitivity positions while leaving saturated positions largely undisturbed, directly targeting the transitional ridges identified in Section 2.1. Algorithm 1 summarizes the training procedure. Noise parameters are updated periodically (every T steps) to track evolving attention patterns, and clamped to [σmin,σmax][ _ , _ ] for numerical stability. At inference time, noise injection is disabled (σ=0σ=0). The model retains the flattened local geometry learned during training (analogous to the effect of dropout (36)) where attention distributions have been pushed away toward robust, saturated regions, ensuring quantization robustness. 3 Experiments 3.1 Experimental Setup We evaluate the effectiveness of Post Training Quantisation (PTQ) with Jacobian-guided noise injection on state-of-the-art language and vision-language models. We experiment with two open-source language models: Llama-3.2-3B (14) and Qwen2.5-3B (39) fine-tuned on the Alpaca dataset (38). For vision-language, we use SigLIP base 16-384 (48) trained on ImageNet-1K (5) classification task. We compare the baseline (standard fine-tuning without noise) against our Jacobian-guided noise injection under several low-bit quantisation settings. For PTQ, we employ AWQ (24), GPTQ (10) and SpinQuant for language models, and ERQ and RepQViT (21) for vision task. Models are assessed in a zero shot setting on MMLU (16), HellaSwag (47), Winogrande (35), PIQA (2), TruthfulQA (25), and WikiText (31) perplexity. See Appendix B for details. 3.2 Main Results Table 1 present results for Llama and Qwen models under PTQ setting using AWQ, GPTQ and Spinquant. Jacobian-guided noise injection yields consistent improvements across quantization levels for both Llama-3.2-3B and Qwen2.5-3B. Notably, the method improves quantized performance without degrading full-precision accuracy significantly (see Appendix D), indicating that noise injection learns representations that are inherently more robust to discretization error. (a) (b) Figure 3: (a) Jacobian norm during training for Qwen2.5-3B layers 9, 18, and 27. (b) Effect of scaling coefficient α on SigLIP W4A4 QAT (ImageNet-1K zero-shot accuracy). Moderate scaling (α∈[0.5,1.0]α∈[0.5,1.0]) yields better results compared to no noise (α=0α=0). 3.3 Analysis We structure our analysis around the following research questions to systematically evaluate the effectiveness and generalizability of Jacobian-guided noise injection for post-training quantisation: RQ1. Are performance gains generalizable across model families and quantization strategies? The consistent gains across three model families (Llama, Qwen, SigLIP (48)) and 4 PTQ methods (AWQ (24), GPTQ (11), ERQ (49), RepQViT (21)) demonstrate that Jacobian-guided noise injection learns quantization-robust representations rather than overfitting to a specific architecture or quantization scheme. Figure 1 evaluates generalization to vision transformers (8) using SigLIP base 16-384 on ImageNet-1K (5) classification. With ERQ, noise injection recovers up to +9% accuracy at W4A4; with RepQViT, gains reach +37% at the same bit-width. The improvements scale inversely with bit-width, highlighting the fact that aggressive quantization benefits most from noise-based regularization. Beyond PTQ, we also evaluate SigLip model under quantization-aware training (QAT). Figure 3(b) shows that noise injection (α>0α>0) improves over QAT baseline performance (α=0α=0) for SigLIP, confirming that the method can be used with existing QAT pipelines (4) and is not restricted to PTQ settings. RQ2. How does noise injection affect attention maps and logit distributions? Figure 3 shows the Jacobian norm decreasing and stabilizing during noise-based training, providing empirical evidence for noise-based Jacobian desensitisation. Figure 4 compares attention maps for SigLIP-base-384 trained with and without noise. Noise-trained models exhibit more diffuse attention patterns achievable only in smaller Jacobian norm regions, consistent with the theoretical predictions in Section 2. We observe this phenomenon of diffused attention across all model layers, pointing towards the stochasticity-based robustness and outlier reduction introduced due to injected noise (17). It should be noted that, while zero Jacobian norm is achievable when probability mass concentrates on a single token, this regime is somewhat problematic since: (1) sparse attention degenerates to near-constant outputs for all kinds of inputs, and (2) large logit magnitudes amplify quantization errors exponentially through the softmax gate (15). Noise injection prevents this collapse and the stochastic perturbation discourages extreme weight concentration on any single token, analogous to the effect of dropout (36) on model training. The resulting distributed attention provides innate robustness to quantization error propagation. RQ3. How does the noise scaling coefficient α affect quantized model performance? Figure 3(b) presents ImageNet-1K zero-shot accuracy for SigLIP at W4A4 QAT setting across different values of α. We observer that moderate scaling (α∈[0.5,1.0]α∈[0.5,1.0]) yields the best results, with α=0.5α=0.5 achieving 31.37% Top-1 accuracy compared to 30.31% for the baseline (+1.06%). Performance degrades rapidly after this and the model performance is the lowest at α=50.0α=50.0 (27.69%), indicating that excessive noise disrupts learning due to logits being pushed beyond representable ranges. We observed similar trend for both PTQ and QAT settings, where performance improves and then rapidly degrading with increasing noise. 4 Related Work Figure 4: Attention maps for SigLIP (layer 10). Noise-trained models exhibit more diffused attention patterns across key positions. Quantisation has emerged as a dominant approach for deploying large models on resource-constrained environments (12; 37; 6). Post-training quantization (PTQ) methods (10; 24; 45; 6) like GPTQ and AWQ compress models to 4-8 bits using second-order information, salient weight protection, or outlier handling. Quantization-aware training (QAT) and mixed-precision approaches (7; 44) take an alternative route by incorporating quantization into the training loop (9; 4). However, both PTQ and QAT methods typically treat all layers uniformly without addressing component-specific sensitivities. The attention mechanism (42) poses unique challenges for quantization due to the softmax nonlinearity, which creates error amplification that fixed quantization schemes cannot anticipate. Several works (28) like I-BERT (20), FQ-ViT (26), and RepQ-ViT (21) address this by modifications at inference time. However, these methods engineer around softmax sensitivity rather than addressing the root cause. The connection between Jacobian norms and model robustness provides a theoretical foundation for addressing this sensitivity (13; 30). Contractive autoencoders (34) penalize the Frobenius norm of the encoder Jacobian to learn locally invariant features.(19) extends this to adversarial robustness, showing that Jacobian regularization bounds sensitivity to input perturbations. More recent work (33) demonstrated that improving feature stability under Gaussian noise implicitly reduces curvature of the softmax loss landscape, connecting noise injection to loss geometry. Noise injection as regularisation (36; 43; 1) has also been explored in some prior works. (3) injects annealed noise to postpone early softmax saturation during training. However, most of these methods do not address quantisation as the target objective, and use fixed or annealed noise schedules that do not account for position-dependent sensitivity. Our work synthesizes these threads by targeting the attention softmax specifically and deriving noise variance from the Jacobian norm. Unlike prior noise injection methods that apply uniform perturbation, the proposed approach adapts noise per-position based on local sensitivity, concentrating regularization where quantization errors are most amplified, which produces models with attention distributions that are inherently robust to precision perturbations, providing benefits that transfer across quantization methods. 5 Conclusion We presented Jacobian-guided Noise Injection, a noval approach to improving quantization robustness in Large Language Models. By deriving noise variance directly from the Softmax Jacobian Frobenius norm, our method provides adaptive regularization that targets the most sensitive regions of the attention softmax. Our theoretical analysis establishes clear connections between noise injection, Hessian trace regularization, and quantization error bounds. Experiments across multiple PTQ and QAT methods across multiple model families demonstrate consistent improvements. We also provide empirical evidence for the relationship between noise injection and Jacobian sensitivity through qualitative analysis. The proposed method adds no overhead at inference time, matches baseline performances at fp16 and improves quantisation robustness, making it practical for deployment in resource-constrained environments. Impact Statement This paper presents work whose goal is to develop robust quantisation approaches. Our method enables aggressive low-bit quantization with reduced quality degradation, which reduce the computational cost, memory footprint, and energy required to deploy LLMs. The proposed methods are general-purpose techniques applicable to any neural network and do not introduce novel capabilities. We do not foresee any ethical concerns beyond those that are well established in the field of machine learning efficiency. References Bishop (1995) C. M. Bishop Training with noise is equivalent to Tikhonov regularization. Neural Computation 7 (1), p. 108–116. Cited by: §2, §4. Bisk et al. (2020) Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. AAAI Conference on Artificial Intelligence. Cited by: §3.1. Chen et al. (2017) B. Chen, W. Deng, and J. Du Noisy softmax: improving the generalization ability of DCNN via postponing the early softmax saturation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. Choi et al. (2018) J. Choi, Z. Wang, S. Venkataramani, P. I. Chuang, V. Srinivasan, and K. Gopalakrishnan PACT: parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085. Cited by: §3.3, §4. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §B.6, Appendix C, §3.1, §3.3. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §4. Dong et al. (2019) Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer HAWQ: hessian AWare quantization of neural networks with mixed-precision. In IEEE International Conference on Computer Vision (ICCV), Cited by: §4. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. International Conference on Learning Representations (ICLR). Cited by: §3.3. Esser et al. (2020) S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha Learned step size quantization. In International Conference on Learning Representations (ICLR), Cited by: §4. Frantar et al. (2023a) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), Cited by: §B.3, §1, §3.1, §4. Frantar et al. (2023b) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. External Links: 2210.17323, Link Cited by: §3.3. Gholami et al. (2022) A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer A survey of quantization methods for efficient neural network inference. Low-Power Computer Vision. Cited by: §1, §4. Goodfellow et al. (2015) I. J. Goodfellow, J. Shlens, and C. Szegedy Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), Cited by: §4. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, et al. The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §3.1. Guo et al. (2017) C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger On calibration of modern neural networks. In International Conference on Machine Learning (ICML), Cited by: §3.3. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR). Cited by: §3.1. Hendrycks and Dietterich (2019) D. Hendrycks and T. Dietterich Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), Cited by: §3.3. Jacob et al. (2018) B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko Quantization and training of neural networks for efficient integer-arithmetic-only inference. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1. Jakubovitz and Giryes (2018) D. Jakubovitz and R. Giryes Improving DNN robustness to adversarial attacks using jacobian regularization. In European Conference on Computer Vision (ECCV), Cited by: §4. Kim et al. (2021) S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer I-BERT: integer-only BERT quantization. In International Conference on Machine Learning (ICML), Cited by: §1, §4. Li et al. (2023) Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu RepQ-ViT: scale reparameterization for post-training quantization of vision transformers. In IEEE International Conference on Computer Vision (ICCV), Cited by: 2nd item, §3.1, §3.3, §4. Li et al. (2025) Z. Li, Y. Su, R. Yang, C. Xie, Z. Wang, Z. Xie, N. Wong, and H. Yang Quantization meets reasoning: exploring llm low-bit quantization degradation for mathematical reasoning. External Links: 2501.03035, Link Cited by: §1. Lin et al. (2026) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for llm compression and acceleration. External Links: 2306.00978, Link Cited by: §1. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han AWQ: activation-aware weight quantization for LLM compression and acceleration. In Conference on Machine Learning and Systems (MLSys), Note: Best Paper Award Cited by: §B.3, §1, §1, §3.1, §3.3, §4. Lin et al. (2022a) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. Association for Computational Linguistics (ACL). Cited by: §3.1. Lin et al. (2022b) Y. Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou FQ-ViT: post-training quantization for fully quantized vision transformer. In International Joint Conference on Artificial Intelligence (IJCAI), Cited by: §1, §4. Liu et al. (2025) Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort SpinQuant: llm quantization with learned rotations. External Links: 2405.16406, Link Cited by: §B.3. Liu et al. (2021) Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao Post-training quantization for vision transformer. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. International Conference on Learning Representations (ICLR). Cited by: §B.1. Madry et al. (2018) A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), Cited by: §4. Merity et al. (2017) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. International Conference on Learning Representations (ICLR). Cited by: §3.1. Micikevicius et al. (2018) P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu Mixed precision training. In International Conference on Learning Representations (ICLR), Cited by: §1. Nguyen et al. (2024) H. Nguyen, F. Gamboa, S. Zhang, R. Chhaibi, S. Gratton, and T. Giaccone Training more robust classification model via discriminative loss and gaussian noise injection. Transactions on Machine Learning Research. Cited by: §4. Rifai et al. (2011) S. Rifai, X. Muller, X. Glorot, G. Mesnil, Y. Bengio, and P. Vincent Contractive auto-encoders: explicit invariance during feature extraction. In International Conference on Machine Learning (ICML), Cited by: §4. Sakaguchi et al. (2020) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. AAAI Conference on Artificial Intelligence. Cited by: §3.1. Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15 (1), p. 1929–1958. Cited by: §2.4, §3.3, §4. Tang et al. (2024) Y. Tang, Y. Wang, J. Guo, Z. Tu, K. Han, H. Hu, and D. Tao A survey on transformer compression. arXiv preprint arXiv:2402.05964. Cited by: §1, §4. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto Stanford Alpaca: an instruction-following LLaMA model. GitHub repository. Cited by: §B.1, §3.1. Team (2024) Q. Team Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §1, §3.1. Touvron et al. (2023a) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Touvron et al. (2023b) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. LLaMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, §4. Wan et al. (2013) L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus Regularization of neural networks using DropConnect. In International Conference on Machine Learning (ICML), Cited by: §4. Wang et al. (2019) K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han HAQ: hardware-aware automated quantization with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), Cited by: §1, §4. Xiao et al. (2024) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. External Links: 2211.10438, Link Cited by: §1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. Association for Computational Linguistics (ACL). Cited by: §3.1. Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training. IEEE International Conference on Computer Vision (ICCV). Cited by: Appendix C, §3.1, §3.3. Zhong et al. (2025) Y. Zhong, Y. Huang, J. Hu, Y. Zhang, and R. Ji Towards accurate post-training quantization of vision transformers via error reduction. External Links: 2407.06794, Link Cited by: §3.3. Appendix A Derivation of Closed-Form Jacobian Frobenius Norm For completeness, we derive the closed-form expression for ‖JS‖F2||J_S||_F^2. The Softmax Jacobian is JS=diag(p)−ppTJ_S=diag(p)-p^T, where p=S(z)p=S(z). The Frobenius norm squared is: ‖JS‖F2 ||J_S||_F^2 =Tr(JSTJS) =Tr(J_S^TJ_S) (14) =Tr((diag(p)−ppT)T(diag(p)−ppT)) =Tr ((diag(p)-p^T)^T(diag(p)-p^T) ) (15) =Tr(diag(p)2−2diag(p)ppT+ppTppT) =Tr (diag(p)^2-2diag(p)p^T+p^Tp^T ) (16) =∑ipi2−2∑ipi⋅pi⋅∑jpj+(∑ipi2)2 = _ip_i^2-2 _ip_i· p_i· _jp_j+ ( _ip_i^2 )^2 (17) =‖p‖22−2‖p‖33+‖p‖24 =||p||_2^2-2||p||_3^3+||p||_2^4 (18) where we used ∑jpj=1 _jp_j=1 (softmax normalization) and Tr(ppTppT)=(pTp)2=‖p‖24Tr(p^Tp^T)=(p^Tp)^2=||p||_2^4. Appendix B Experimental Details This section provides comprehensive details on training configurations, quantization strategies, and infrastructure used in our experiments. B.1 Training Configuration Optimizer and Learning Rate. We use AdamW (29) with learning rate 5×10−75× 10^-7, weight decay 0.010.01, and gradient clipping at max norm 1.01.0. Training uses a cosine learning rate scheduler with 500 warmup steps. Batch Size and Accumulation. We use per-device batch size of 1 with gradient accumulation over 8 steps, yielding an effective batch size of 8 per GPU. For multi-GPU training, the effective batch size scales with the number of GPUs. Training Duration. Models are fine-tuned for 5 epochs on the Alpaca dataset (38) with maximum sequence length of 2048 tokens. Evaluation is performed every 2000 steps. Precision. All training uses bfloat16 mixed precision for memory efficiency and numerical stability. B.2 Jacobian-Guided Noise Injection Parameters Noise Scaling. The noise scaling coefficient α is set to 0.5 (see Figure 3(b) for sensitivity analysis). Noise standard deviation is clamped to [σmin,σmax]=[0.005,10.0][ _ , _ ]=[0.005,10.0] for numerical stability. Amortized Updates. To reduce computational overhead, Jacobian norms are computed and noise parameters are updated every T=100T=100 training steps rather than every step. This amortization has negligible impact on final performance while reducing overhead. Warmup. Jacobian-guided noise updates begin after 100 warmup steps to allow initial model stabilization. During warmup, a constant base noise with σ=0.1σ=0.1 is applied. Noise Mode. We use rowwise (per-position) noise injection where each query position receives noise calibrated to its local Jacobian norm. This is more effective than layerwise or global noise as it targets high-sensitivity positions specifically. B.3 Post-Training Quantization Methods AWQ (24): Activation-aware Weight Quantization identifies salient weights based on activation magnitudes and applies per-channel scaling to protect these weights during quantization. We use the default calibration set of 128 samples. GPTQ (10): Uses approximate second-order information (Hessian) to quantize weights layer-by-layer while minimizing reconstruction error. We use group size of 128 for weight quantization. SpinQuant (27): A PTQ method that applies learned orthogonal (rotation) transformations to weights and activations to reduce outliers and make distributions more uniform before quantization, also supporting KV cache quantization. B.4 Quantization Bit-Width Configurations We evaluate multiple quantization configurations denoted as WxAy (x-bit weights, y-bit activations): Config Weight Bits Activation Bits W4A4 4 4 W4A8 4 8 W5A5 5 5 W6A6 6 6 W6A8 6 8 W8A8 8 8 Weight quantization uses per-channel granularity. Activation quantization uses per-token granularity for language models. B.5 Distributed Training Infrastructure DeepSpeed Configuration. We use DeepSpeed ZeRO Stage 2 for memory-efficient distributed training with the following settings: • Gradient partitioning with allgather bucket size 2×1082× 10^8 • Reduce scatter with bucket size 2×1082× 10^8 • Communication overlap enabled • Contiguous gradient buffers Hardware. Experiments are conducted on 8 NVIDIA A100 GPUs with 80 GB memory each. We train both Llama and Qwen models for up to 10 epochs until convergence. B.6 Evaluation Protocol Language Model Benchmarks. We evaluate in a zero shot setting on HellaSwag, Winogrande, PIQA, TruthfulQA, WikiText, BoolQ, Arc-Easy and Arc-Challenge. Vision Model Benchmarks. SigLIP models are evaluated on ImageNet-1K (5) zero-shot classification using the standard CLIP evaluation protocol with Top-1 and Top-5 accuracy metrics. Appendix C SigLIP Post-Training Quantization Results Table 2 provides detailed results for SigLIP base 16-384 under post-training quantization with ERQ and RepQViT methods. Model Configuration. We use the SigLIP base 16-384 vision encoder (48), which processes images at 384×384 resolution with 16×16 patches. The model is evaluated on ImageNet-1K (5) classification using the partitioned test set (split 95-5). Quantization Methods. We evaluate two PTQ methods designed for vision transformers: • ERQ (Error-aware Quantization): A PTQ method that minimizes reconstruction error by considering the error propagation through transformer layers. ERQ optimizes quantization parameters to reduce the cumulative error in attention and feed-forward computations. • RepQViT (21): A reparameterization-based approach that decouples quantization scales for hardware-friendly deployment. RepQViT addresses the unique challenges of quantizing vision transformers by handling post-LayerNorm activations and attention score distributions. Bit-width Settings. We evaluate four quantization configurations denoted as WxAy, where x is the weight bit-width and y is the activation bit-width: • W4A4: Aggressive 4-bit quantization for both weights and activations, suitable for edge deployment with severe memory constraints. • W5A5: Moderate 5-bit quantization offering a balance between compression and accuracy. • W6A6: Conservative 6-bit quantization with minimal accuracy degradation. • W8A8: Near-lossless 8-bit quantization serving as a reference point. Training Protocol. Models are fine-tuned with Jacobian-guided noise injection as described in Section 2, using the hyperparameters specified in Section 3.1. The noise scaling coefficient α is set to 0.5 based on the sensitivity analysis in Figure 3. Table 2: SigLIP base 16-384 PTQ results on ImageNet-1K zero-shot classification. All values are Top-1 accuracy (%). B: Baseline, B+N: Baseline with Jacobian-guided Noise. The FP32 baseline achieves 86.7% Top-1 accuracy. Method Noise W4A4 W5A5 W6A6 W8A8 ERQ B 51.2 72.4 81.3 85.9 B+N 60.1 76.8 83.5 86.4 RepQViT B 32.5 65.1 78.9 85.6 B+N 69.8 74.2 82.1 86.2 Appendix D Full-Precision Results With and Without Noise Table 3 presents evaluation results for Qwen2.5-3B at full precision (FP16) with and without Jacobian-guided noise injection during training. This demonstrates that noise injection does not degrade full-precision model performance while providing quantization robustness. Table 3: Qwen2.5-3B full-precision (FP16) results with and without noise injection. B: Baseline, B+N: With Jacobian Noise (α=0.5α=0.5, rowwise). Model MMLU HellaSwag PIQA Winogrande TruthfulQA WikiText PPL Qwen2.5-3B (B) 65.25 75.08 79.87 70.72 47.41 10.60 Qwen2.5-3B (B+N) 65.48 74.91 79.54 70.17 47.47 10.46 Appendix E Detailed zero-shot PTQ results for Llama and Qwen Tables 4, 5 and 6 shows the detailed results for Llama and Qwen model on all 7 benchmarks. We observe that noise injections improves the performance of the overall model, while maintaining comparable or better performance across all 7 benchmarks. Table 4: SpinQuant W4A4KV4 / W4A8KV8 / W6A6KV6 / W6A8KV8 full results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2. Model Variant Precision PPL HellaSwag PIQA WinoGrande ARC-E ARC-C BoolQ TQA-MC2 Avg7 Qwen2.5-3B Base W4A4KV4 10.49 66.27 74.32 59.19 71.93 44.20 65.32 41.12 60.34 W4A8KV8 8.55 72.91 78.02 68.98 77.48 48.21 79.94 45.91 67.35 W6A6KV6 8.36 73.92 78.35 67.25 77.15 48.38 79.08 46.64 67.25 W6A8KV8 8.32 74.25 79.33 68.59 76.18 48.98 79.57 46.74 67.66 Jac-noise W4A4KV4 10.33 66.65 74.43 61.88 73.70 45.05 70.18 46.42 62.62 W4A8KV8 8.35 72.32 78.40 68.35 74.03 45.73 77.55 43.25 65.66 W6A6KV6 8.18 73.26 78.62 68.27 77.69 48.81 79.08 46.32 67.44 W6A8KV8 8.11 73.56 78.89 67.64 77.10 49.40 79.54 46.03 67.45 Llama-3.2-3B Base W4A4KV4 10.72 66.96 74.05 60.38 69.99 39.76 71.71 38.89 60.25 W4A8KV8 8.31 72.44 77.37 68.82 73.65 43.77 74.16 41.12 64.48 W6A6KV6 8.03 73.74 78.67 69.46 76.47 46.59 75.29 41.66 65.98 W6A8KV8 7.97 73.86 78.13 69.93 74.07 47.78 76.24 41.64 65.95 Jac-noise W4A4KV4 11.18 67.38 73.50 64.09 67.05 40.27 70.40 38.35 60.15 W4A8KV8 8.47 73.33 77.97 68.35 74.71 46.33 73.85 41.65 65.17 W6A6KV6 8.17 74.67 77.91 70.88 77.31 48.38 76.24 42.31 66.81 W6A8KV8 8.12 74.94 78.35 70.01 75.13 49.66 76.27 42.70 66.72 Table 5: AWQ W4A4 / W4A8 / W6A6 / W8A8 results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2. Model Variant Precision PPL HellaSwag PIQA WinoGrande ARC-E ARC-C BoolQ TQA-MC2 Avg7 Qwen2.5-3B Base W4A4 8121.24 30.23 54.35 48.54 32.32 23.12 48.07 47.45 40.58 W4A8 10.80 74.23 77.97 68.11 78.32 49.57 80.18 46.21 67.80 W6A6 15.29 70.26 74.86 64.72 75.80 49.40 75.54 45.25 65.12 W8A8 10.80 74.23 77.80 68.59 77.99 50.00 80.70 45.57 67.84 Jac-noise W4A4 3281.04 30.57 53.26 52.25 33.54 24.06 49.42 47.80 41.56 W4A8 11.01 73.45 78.24 68.51 78.11 49.40 80.03 44.61 67.48 W6A6 13.63 70.95 75.81 64.61 76.29 48.81 74.84 46.35 65.38 W8A8 10.80 74.44 79.40 68.25 79.20 50.49 81.12 45.73 68.38 Llama-3.2-3B Base W4A4 105.06 32.47 55.17 51.14 33.71 23.81 50.80 47.31 42.06 W4A8 9.96 74.58 78.24 71.03 77.06 47.61 76.39 41.64 66.65 W6A6 10.80 72.87 76.99 68.35 74.45 45.05 70.58 40.21 64.07 W8A8 9.73 74.42 78.62 70.96 77.10 47.95 76.27 41.04 66.62 Jac-noise W4A4 104.34 33.33 55.41 51.04 33.79 24.29 51.55 45.99 42.20 W4A8 9.86 75.17 78.78 70.48 78.24 49.66 77.13 42.00 67.35 W6A6 10.55 73.46 77.04 67.88 75.08 47.01 72.97 40.52 64.85 W8A8 9.68 75.41 78.94 70.72 78.54 49.74 76.91 42.61 67.55 Table 6: GPTQ W4A4 / W4A8 / W6A6 / W8A8 results for Qwen2.5-3B and Llama-3.2-3B on both the Base and Jacobian-noise (Jac) variants. PPL is WikiText-2. Zero-shot metric is acc_norm for HellaSwag/PIQA/ARC-C and acc otherwise. “Avg7” is the average of HellaSwag, PIQA, WinoGrande, ARC-E, ARC-C, BoolQ, TQA-MC2. Model Variant Precision PPL HellaSwag PIQA WinoGrande ARC-E ARC-C BoolQ TQA-MC2 Avg7 Qwen2.5-3B Base W4A4 5172.39 25.81 49.95 51.14 26.52 23.46 43.91 49.27 38.58 W4A8 14.13 72.05 77.15 65.98 78.00 48.89 80.03 44.43 66.65 W6A6 28.42 63.32 70.78 60.46 69.15 42.66 68.23 45.19 59.97 W8A8 10.98 73.99 78.02 67.88 77.99 49.15 80.06 45.78 67.55 Jac-noise W4A4 4207.85 26.31 50.05 51.04 26.00 24.32 45.38 49.64 38.96 W4A8 12.21 71.39 77.42 66.77 77.23 49.40 77.43 43.14 66.11 W6A6 21.52 63.10 71.98 60.77 69.28 43.86 69.20 42.54 60.10 W8A8 10.82 73.41 78.65 68.23 78.95 48.91 80.36 44.73 67.61 Llama-3.2-3B Base W4A4 454.35 29.75 51.36 49.88 30.89 23.55 46.79 49.96 40.31 W4A8 37.40 72.75 77.91 67.80 74.20 44.97 76.91 39.10 64.81 W6A6 10.87 72.05 76.55 66.61 73.32 45.31 71.53 40.49 63.69 W8A8 9.69 74.37 78.62 71.27 76.77 48.38 75.93 41.50 66.69 Jac-noise W4A4 496.38 30.84 54.21 48.67 30.76 25.91 46.87 47.77 41.00 W4A8 31.47 73.53 78.13 70.96 76.85 47.61 76.64 40.82 66.36 W6A6 11.12 73.19 76.66 66.46 74.96 45.48 71.56 40.81 64.16 W8A8 9.86 75.21 78.56 69.53 78.07 49.40 77.37 42.64 67.25 Appendix F Activation Similarity Analysis on Hardware To validate our method in real-world deployment scenarios, we analyze a Vision-Language Model (VLM) trained with quantization-aware training at 15-bit precision and deployed on edge hardware. We measure the cosine similarity between quantized and unquantized activations at each layer, providing a direct measure of how well the quantized model preserves the computational behavior of the full-precision model. (a) (b) Figure 5: Layer-wise cosine similarity between quantized (15-bit) and unquantized activations for a VLM deployed on edge hardware. (a) Baseline model shows severe similarity degradation, with some layers dropping to near-zero cosine similarity. (b) Model trained with Jacobian-guided noise injection maintains consistently high similarity (>>0.95) across all layers. Experimental Setup. The VLM is deployed on edge accelerator hardware with 15-bit fixed-point arithmetic. Unlike standard GPU deployment where numerical behavior closely matches training (bfloat16/float32), edge accelerators often have distinct rounding behavior that can cause divergent generation even at relatively high bit-widths. Results. Figure 5 reveals a striking difference between baseline and noise-trained models: • Baseline (a): The cosine similarity between quantized and unquantized activations fluctuates dramatically across layers, with multiple layers showing similarity scores approaching zero. This indicates that quantization errors accumulate and amplify through the network, causing the quantized model’s internal representations to diverge completely from the full-precision model. • With Noise Injection (b): The noise-trained model maintains consistently high cosine similarity (>>0.95) across all layers, demonstrating that Jacobian-guided noise injection trains the model to be inherently robust to numerical perturbations introduced by quantization. We also reproduced this in a PTQ setting with W4A4 Round-to-Nearest (RTN) quantisation with Siglip base 384 model. Figure 6 shows that errors accumulate over layers and eventually result in divergence of activation values. However, training with noise reduces this divergence and helps model consistently maintain high similarity with actual activation values. Figure 6: Attention maps for SigLIP (layer 10) Baseline vs baseline with noise. Noise-trained models exhibit more diffused attention patterns across key positions. Implications for Deployment. The layer-wise similarity analysis provides direct evidence that softmax instability, as characterized in Section 2, causes real deployment failures. When similarity drops to near-zero at intermediate layers, the error propagates and compounds through subsequent attention operations, leading to outputs that bear little resemblance to the full-precision model. Our noise injection method addresses this root cause by regularizing the attention mechanism during training, resulting in models that maintain consistent behavior when deployed on diverse hardware with varying numerical precision.