Paper deep dive
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/13/2026, 3:03:56 AM
Summary
The paper introduces DualQuant, a calibration-free post-training quantization framework for Large Language Models that jointly quantizes weights and activations. It utilizes a dual-scale (per-row and per-column) optimization approach to minimize weight-reconstruction Mean Squared Error (MSE). A key innovation is that the column scaling factors derived from weight optimization implicitly serve as effective activation rescalings, eliminating the need for calibration data. DualQuant supports various formats including INT4, MXINT4, MXFP4, and NVFP4, and demonstrates performance matching or exceeding calibration-based methods like SmoothQuant while offering significant memory and throughput improvements.
Entities (10)
Relation Signals (8)
DualQuant → supportsformat → MXINT4
confidence 95% · operates uniformly across INT4, MXINT4, MXFP4, and NVFP4
DualQuant → supportsformat → NVFP4
confidence 95% · DUALQUANT is the first dual-scaling method to formally support MX and NV block formats... including NVFP4
DualQuant → usesalgorithm → Block Coordinate Descent
confidence 95% · develop an efficient solver based on variable splitting and block coordinate descent
DualQuant → achievesspeedup → 2.46x
confidence 90% · delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16
DualQuant → evaluatedon → LLaMA-3.1-8B
confidence 90% · We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B
DualQuant → evaluatedon → Qwen
confidence 90% · Across Llama and Qwen models evaluated
DualQuant → matchesorexceeds → SmoothQuant
confidence 90% · DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats
DualQuant → outperforms → calibration-free baselines
confidence 90% · substantially outperforms calibration-free baselines, with large gains for MX and NV block formats
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DUALQUANT, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DUALQUANT as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. DUALQUANT provides a unified algorithm supporting a wide range of formats, including the full MX family and NVFP4. To our knowledge, DUALQUANT is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DUALQUANT matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with large gains for MX and NV block formats. We validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61x memory reduction and up to 2.46x throughput speedup over BF16.
Tags
Links
- Source: https://arxiv.org/abs/2610.13767v1
- Canonical: https://arxiv.org/abs/2610.13767v1
Trouble viewing inline? Open PDF directly →
Full Text
102,762 characters extracted from source content.
Expand or collapse full text
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs Mahsa Salmani Sayeh Sharify & Hesham Mostafa Affiliation: d-Matrix Abstract Low-bit inference on accelerators with native 4-bit support requires quantizing both weights and activations to realize the full throughput gains these formats provide. While post-training weight quantization has been extensively studied, activation quantization remains considerably more challenging and typically requires calibration data. In this work, we introduce DualQuant, a dual-scale, calibration-free post-training quantization framework that directly minimizes the weight-reconstruction Mean Squared Error (MSE) and, as an immediate by-product, yields per-channel activation rescalings prior to activation quantization. Our key observation is that the scaling factors obtained by minimizing weight-reconstruction error also provide effective activation rescalings, enabling competitive activation quantization without calibration data. We formulate DualQuant as a non-differentiable weight-reconstruction problem and develop an efficient solver based on variable splitting and block coordinate descent, where each subproblem admits a closed-form update. By treating the quantizer as a black box and projecting scales onto their format-specific admissible set after each update, DualQuant provides a unified algorithm supporting a wide range of formats, including INT4, INT8, the full MX family, and NVFP4. To our knowledge, DualQuant is the first dual-scaling method to formally support MX and NV block formats. Across Llama and Qwen models evaluated on WikiText-2, C4, and zero-shot downstream tasks, DualQuant matches or exceeds the accuracy of calibration-based SmoothQuant across most models and formats, and substantially outperforms calibration-free baselines, with particularly large gains for MX and NV block formats. We further validate these gains with a real packed W4A4 inference path for Llama-3.1-8B on an RTX 4090. The resulting implementation closely matches the accuracy of fake quantization while delivering up to 2.61×2.61× memory reduction and up to 2.46×2.46× throughput speedup over BF16. Our anonymized codebase is available here. 1 Introduction Large Language Models (LLMs) have grown rapidly in scale, from dense models (Grattafiori et al., 2024; Yang et al., 2025) to mixture-of-experts architectures exceeding a trillion parameters (DeepSeek-AI, 2024; Jiang et al., 2024), placing enormous pressure on memory and compute at inference time. Post-Training Quantization (PTQ) addresses this by compressing weights, activations, and KV-cache to low-bit representations without retraining. Modern accelerators such as NVIDIA Blackwell GPUs provide native 4-bit matrix multiplication (Rouhani et al., 2023; NVIDIA, 2025) delivering up to 4×4× the throughput of FP16, but realizing this gain requires both weights and activations in a low-bit format. The central obstacle is that activation tensors exhibit systematic large-magnitude outliers in specific channels (Dettmers et al., 2022; Bondarenko et al., 2021; Sun et al., 2024) that dominate the quantization grid. Calibration-based methods address activation quantization effectively but require representative data, introducing distribution sensitivity and data-access overhead prohibitive in privacy-sensitive or on-device settings. This includes both scale-based approaches such as SmoothQuant (Xiao et al., 2023) and rotation-based approaches such as SpinQuant (Liu et al., 2025), which optimizes rotation matrices using calibration data. Calibration-free rotation-based methods such as QuaRot (Ashkboos et al., 2024) avoid this dependency but do not target activation-quantization error. In other words, no prior method achieves competitive calibration-free joint Weight and Activation (W+A) quantization across modern MX and NV formats. In this work, we introduce DualQuant, a calibration-free dual-scale (per-row and per-column) PTQ framework that jointly quantizes weights and activations without any access to calibration data or activation statistics. The key insight is that channels with large-magnitude weight columns contribute disproportionately to the layer output and, as shown empirically, concentrate activation outliers (Sun et al., 2024; Dettmers et al., 2022; Bondarenko et al., 2021). Consequently, the column scales that minimize weight-reconstruction MSE attenuate these same channels, so they not only quantize the weights well but also implicitly define activation rescalings that we observe to substantially improve activation quantization—playing a role similar to SmoothQuant’s calibrated scales (Xiao et al., 2023), yet derived from weights alone. DualQuant obtains these row and column scales via Block Coordinate Descent (BCD) with closed-form updates and provable convergence guarantees (Tseng, 2001; Razaviyayn et al., 2013), treating the quantizer as a black-box operator and projecting scales to their admissible format set after each update, yielding a single unified algorithm that operates across INT4, MXINT4, MXFP4, and NVFP4. Evaluation on Qwen and Llama families confirms that DualQuant matches or surpasses SmoothQuant on W+A quantization across most formats and models, without any calibration data. Our main contributions are: 1. We formalize dual-scale weight quantization as a constrained non-convex optimization problem and solve it via BCD with closed-form per-coordinate updates, format-aware scale projection, and provable convergence guarantees, requiring no inner loop, no Hessian, and no calibration data. The same algorithm operates uniformly across INT4, MXINT4, MXFP4, and NVFP4, making DualQuant the first dual-scaling method that formally supports the full MX and NV formats. 2. We show that the column scales obtained from weight MSE minimization also serve as effective activation rescalings, enabling joint W+A quantization without any access to calibration data. Across Qwen and Llama families and four 4-bit formats, DualQuant closely matches or surpasses calibration-based SmoothQuant and significantly outperforms calibration-free baselines QuaRot and SINQ+, with especially large margins on MX and NV block data formats. 3. We validate DualQuant through a real packed W4A4 deployment of Llama-3.1-8B on an RTX 4090. The packed implementation matches fake-quantization accuracy to within 0.010.01 perplexity on WikiText-2 while reducing memory usage by 2.512.51–2.61×2.61× and improving end-to-end speed by 1.391.39–2.46×2.46× over BF16. Incorporating the scales into the activation-quantization kernel avoids additional kernel launches and limits the quantization overhead to 4.34.3-6.1%6.1\% of total GPU time. 2 Background and related work 2.1 Post-training weight quantization A rich body of PTQ methods has been proposed to close the gap between quantized and full-precision quality: GPTQ (Frantar et al., 2023) compensates for quantization error via second-order weight correction; AWQ (Lin et al., 2024) protects salient channels via activation-weighted scaling. Calibration-free approaches HQQ (Badri and Shaji, 2023) and QuantEase (Behdin et al., 2023) minimize per-layer reconstruction objectives without requiring calibration data while AdpQ (Ghaffari et al., 2024) minimizes weight distribution divergence via adaptive LASSO. SINQ (Müller et al., 2026) applies dual row and column scaling via Sinkhorn-Knopp iteration (Sinkhorn and Knopp, 1967), minimizing matrix imbalance as a proxy for weight quantization error. Despite their variety, these methods quantize weights only, leaving activations in BF16 and forgoing the full throughput of low-precision accelerators. 2.2 Activation quantization Activations are significantly harder to quantize than weights (Dettmers et al., 2022; Bondarenko et al., 2021): they exhibit systematic large-magnitude outliers in specific channels (Xiao et al., 2023; Wei et al., 2022) that dominate the quantization grid and leave normal channels with very few effective bits. LLM.int8() (Dettmers et al., 2022) handles outlier channels via mixed-precision decomposition. SmoothQuant (Xiao et al., 2023) migrates the quantization burden from activations to weights via calibration-based per-channel rescaling. ZeroQuant (Yao et al., 2022) applies dynamic per-token activation quantization alongside group-wise weight quantization while OmniQuant (Shao et al., 2024) and AffineQuant (Ma et al., 2024) jointly optimize scaling and clipping via learnable transformations. Rotation-based methods (Tseng et al., 2024; Ashkboos et al., 2024; Sun et al., 2025; Czakó et al., 2025; Liu et al., 2025; Saxena et al., 2025; Zandieh et al., 2026) suppress outliers by spreading activation energy incoherently across dimensions via Hadamard, orthogonal rotations, or learned affine transforms. While effective, most of these approaches depend on calibration data, motivating the search for calibration-free alternatives that can achieve competitive W+A quantization. We propose DualQuant, a calibration-free joint W+A quantization method that leverages a principled MSE optimization of weight quantization via block coordinate descent with closed-form updates and provable convergence guarantees (Appendix C), extending naturally to MX and NV format families. Most significantly, we show that the column scales derived from minimizing weight-reconstruction error also serve as effective activation rescalings, enabling joint activation and weight quantization that matches and surpasses SmoothQuant across most formats and models, without any access to activations or calibration data. As illustrated in Figure 1, directly targeting the MSE yields substantially lower reconstruction error than methods such as SINQ (Müller et al., 2026) that optimize a proxy objective, and we show in Section 5 that this improvement translates directly to better downstream performance. Figure 1: Per-element quantization error |−^| |W- W| under different scale designs. Each panel reports the MSE ratio (↑ ) relative to single-scale and the matrix imbalance (SINQ’s objective). (a) Original weight tile. (b) Single-scale: one outlier inflates the entire row. (c) SINQ minimizes matrix imbalance (1.651.65) but achieves only 3.32×3.32× MSE reduction. (d) DualQuant minimizes MSE, achieving 12.54×12.54× reduction, illustrating that low imbalance does not imply low reconstruction error. 3 Quantization preliminaries Quantization maps a tensor to a finite, format-specific set of representable values. Given a tensor W, let G denote the canonical, unit-scale grid of the target format, namely, the integers −2B−1,…,2B−1−1\-2^B-1,…,2^B-1-1\ for INTBB, or the non-uniform exponent-mantissa grid for a floating-point format. A quantizer Q(⋅)Q(·) with scale Δ and zero-point z maps each element to the nearest representable value, Q(W)=Δ⋅(proj(W/Δ+z)−z)Q(W)= ·(proj_G(W/ +z)-z), where proj(⋅)proj_G(·) rounds and clips its argument to the nearest point on G, reducing to standard round-and-clip for integer formats and to nearest-level projection on the non-uniform exponent-mantissa grid for floating-point formats. Symmetric quantization sets z=0z=0 with Δ=max(|W|)/qmax = (|W|)/q_ , where qmaxq_ denotes the maximum positive quantization level; asymmetric quantization uses nonzero z to span [min(W),max(W)][ (W), (W)]. To reduce sensitivity to outliers, block-wise quantization partitions each row into groups of G contiguous elements with independent scales. The OCP microscaling formats (Rouhani et al., 2023) (MXFP4, MXINT4) use G=32G=32 with a shared E8M0 scale restricted to powers of two; NVFP4 (NVIDIA, 2025) uses G=16G=16 with an FP8 UE4M3 scale. This motivates the format-agnostic design of DualQuant, treating Q(⋅)Q(·) as a black-box with an explicit scale constraint (Section 4.2). 4 Methodology 4.1 Problem description For a linear layer computing ⊤XW with activation matrix ∈ℝb×nX ^b× n and weight matrix ∈ℝm×nW ^m× n, the joint W+A quantization error can be written as ⊤−Q()Q()⊤=(−Q())⊤+Q()(−Q())⊤, -Q(X)\,Q(W) =(X-Q(X))W +Q(X)(W-Q(W)) , (1) where the activation error is amplified by ⊤W and the weight error is amplified by Q()Q(X). Minimizing the two terms independently does not necessarily minimize the joint error, and minimizing equation 1 directly is intractable without calibration data on X. SmoothQuant (Xiao et al., 2023) targets this joint problem by collecting per-channel activation statistics from a calibration dataset to construct a per-channel rescaling applied prior to quantization, ⊤=(Diag()−1)(Diag())⊤XW =(X\,Diag(s)^-1)(W\,Diag(s)) . We take a different approach: rather than collecting activation statistics from data, we show that the column scales obtained from minimizing the weight reconstruction MSE on static weights implicitly perform a per-channel rescaling of the same form, without any access to X, as developed next. 4.2 Dual-scale parameterization and the activation factorization Standard block quantization assigns a single shared scale per block along each row of the weight matrix, so an outlier at position (i,j)(i,j) inflates the scale shared by every element in that block, while the column dimension remains entirely unaddressed. Figure 1 illustrates this: the single-scale error matrix exhibits pronounced column-wise outlier patterns that persist across all rows. We mitigate this by introducing a second, column-wise scale alongside the per-block row scale. The resulting dual scaling distributes the error across both dimensions, substantially reducing the peak element-wise reconstruction error (Figure 1(d)). Specifically, we quantize ∈ℝm×nW ^m× n (or a block thereof) as ≈Diag()−1Q(Diag()Diag())Diag()−1≜Diag()−1^,Diag()−1,W ( α)^-1\,Q\! (Diag( α)\,W\,Diag( β) )\,Diag( β)^-1\; \;Diag( α)^-1\, Q_ α, β\,Diag( β)^-1, (2) where α and β are respectively row and column scales, Q(⋅)Q(·) is a fixed format-specific quantizer, and ^,=Q(Diag()Diag()) Q_ α, β=Q\! (Diag( α)\,W\,Diag( β) ) denotes the quantized representation stored at low precision. The inverse scales are constrained to their format-admissible sets: ⊙−1∈αm α -1 _α^m and ⊙−1∈βn β -1 _β^n, where (⋅)⊙−1(·) -1 denotes the element-wise inverse, and αmF_α^m, βnF_β^n are the admissible scale sets of the target format (BF16 for INT4; E8M0 for MXINT4/MXFP4; FP8 UE4M3 for NVFP4). The row scale absorbs row-level dynamic range as in standard formats, while the column scale captures the systematic column-wise structure that single-scale formats ignore. Hence, an outlier at position (i,j)(i,j) is now handled jointly by αi _i and βj _j, distributing the quantization burden across both dimensions. Column scales as activation rescalings. Substituting equation 2 into ⊤XW yields ⊤≈(Diag()−1)^,⊤Diag()−1≜~~⊤, ≈ (X\,Diag( β)^-1 ) Q_ α, β \,Diag( α)^-1\; \; X\, W , (3) where ~=Diag()−1∈ℝb×n X=X\,Diag( β)^-1 ^b× n and ~⊤=^,⊤Diag()−1∈ℝn×m W = Q_ α, β \,Diag( α)^-1 ^n× m. The quantity quantized at inference is not X but the rescaled activation X~ X, whose per-channel statistics are reshaped by β. The column scales therefore play two simultaneous roles: reduce weight quantization error through equation 2 and rescale activations through equation 3. This resembles the per-channel rescaling of SmoothQuant, but with reversed information flow: SmoothQuant derives its scales from activation statistics on a calibration dataset, whereas β is obtained from the weight matrix alone. This weight-derived rescaling benefits activation quantization; it is not merely an algebraic by-product. Intuitively, the output of a linear layer at position (b,k)(b,k) is ∑jXbjWkj _jX_bjW_kj, so channels with large ∥W:,j∥2\|W_:,j\|_2 contribute disproportionately to the output, and these are empirically the channels where activation outliers concentrate (Sun et al., 2024; Dettmers et al., 2022; Bondarenko et al., 2021). Since β is optimized to reduce weight quantization error, it attenuates these same high-magnitude channels using the weights alone. Its mechanism differs from calibration: rather than reproducing the per-channel statistics that methods such as SmoothQuant estimate from activation data, β has a direct, causal effect on activation quantization. Note that the correlation between β and such statistics is heterogeneous and projection-dependent, as shown in Table 5 in Section 5.3. With the quantized weights held fixed, applying β to the activations measurably reduces their quantization error, without any calibration data (Appendix E isolates this by measuring per-projection activation quantization error with and without β under fixed quantized weights). Because this rescaling is applied inside the activation-quantization kernel, its runtime cost is negligible (Section 5.2). Algorithm 1 DualQuant: Dual-Scale Quantization via Coordinate Descent 1: Weight tile W, quantizer Q(⋅)Q(·), admissible sets αmF_α^m, βnF_β^n, number of iterations T, initialization (0,0)( α_0, β_0) 2: ←0 α← α_0, ←0 β← β_0 3: for t=1,…,Tt=1,…,T do 4: ^←Q(Diag()Diag()) Q← Q\! (Diag( α)\,W\,Diag( β) ) 5: ←^Diag()−1R← Q\,Diag( β)^-1 6: out⊙−1←Παm((⊙) 1n⊘‖2,row2) α_out -1← _F_α^m\! ((W )\,1_n\; \;\|R\|_2,row^2 ) and α←1⊘αout⊙−1α← 1 _out -1 ⊳ Row scale update + project 7: ^←Q(Diag()Diag()) Q← Q\! (Diag( α)\,W\,Diag( β) ) 8: ←Diag()−1^S ( α)^-1 Q 9: out⊙−1←Πβn((⊙)⊤m⊘‖2,col2) β_out -1← _F_β^n\! ((W ) 1_m\; \;\|S\|_2,col^2 ) and β←1⊘βout⊙−1β← 1 _out -1 ⊳ Col scale update + project 10: end for 11: return out⊙−1 α_out -1, out⊙−1 β_out -1, Q(Diag()Diag())Q\! (Diag( α)\,W\,Diag( β) ) Notation: ∥⋅∥2,row2\|·\|_2,row^2 and ∥⋅∥2,col2\|·\|_2,col^2: squared ℓ2 _2-norm along each row and column; ⊘ : element-wise division; ⊙ : element-wise multiplication; (⋅)⊙−1(·) -1: element-wise inverse; Π _F: projection to nearest value in F. 4.3 MSE optimization via block coordinate descent We optimize the row and column scales by minimizing the weight reconstruction MSE: min,‖−Diag()−1^,Diag()−1‖F2s.t.⊙−1∈αm,⊙−1∈βn. _ α,\, β\;\; \|W-Diag( α)^-1\, Q_ α, β\,Diag( β)^-1 \|_F^2 .t. α -1 _α^m,\;\; β -1 _β^n. (4) Note that the scales determine which grid points the quantizer assigns when forming ^, Q_ α, β and simultaneously rescale the quantized output for reconstruction. Since grid-point selection is piecewise-constant, this joint dependence is non-differentiable. Variable splitting. To tackle equation 4, we duplicate each scale into inner copies controlling grid assignment and outer copies controlling reconstruction (Boyd et al., 2011), yielding minin,in,out,out _ subarrayc α_in,\, β_in, α_out,\, β_out subarray ‖−Diag(out)−1^in,inDiag(out)−1‖F2 \|W-Diag( α_out)^-1\, Q_ α_in, β_in\,Diag( β_out)^-1 \|_F^2 (5) s.t. .t. in=out,in=out,out⊙−1∈αm,out⊙−1∈βn. α_in= α_out,\;\; β_in= β_out,\;\; α_out -1 _α^m,\;\; β_out -1 _β^n. With the inner copies fixed, ^in,in Q_ α_in, β_in is a constant matrix and the objective becomes bilinear in out⊙−1 α_out -1 and out⊙−1 β_out -1, decoupling into m independent scalar subproblems over α and n over β. A block coordinate descent algorithm is then used (Tseng, 2001; Razaviyayn et al., 2013) to alternate between updating the outer scales (in closed form) and re-synchronizing the inner copies (Appendix A.2). Closed-form scale updates. Hereafter, we write Q for ^in,in Q_ α_in, β_in for brevity. Defining i,:=^i,:Diag(out)−1R_i,:= Q_i,:\,Diag( β_out)^-1 and :,j=Diag(out)−1^:,jS_:,j=Diag( α_out)^-1 Q_:,j, the optimal outer scales are (Appendix B): αi∗=∥i,:∥22i,:i,:⊤ _i^*= \|R_i,:\|_2^2W_i,:\,R_i,: , and βj∗=∥:,j∥22:,j⊤:,j _j^*= \|S_:,j\|_2^2S_:,j \,W_:,j. Both updates have the same structure: the optimal scale is the ratio of the energy of the rescaled quantized slice to its inner product with the original weight slice. After each update, the scale is projected onto the admissible set by rounding αi−1 _i^-1 (resp. βj−1 _j^-1) to the nearest value in αmF_α^m (resp. βnF_β^n). The BCD scheme cycles through: (i) update out α_out, (i) enforce consistency and re-quantize, (i) update out β_out, (iv) enforce consistency and re-quantize, as summarized in Algorithm 1. Convergence analysis. We establish convergence under the following mild assumption. Assumption 1 (Non-degenerate quantization). For all iterations t and rows i, no row of Diag((t))Diag((t))Diag( α^(t))WDiag( β^(t)) is mapped entirely to zero by Q(⋅)Q(·). This holds generically since LLM weight rows follow near-Gaussian distributions with non-negligible ℓ2 _2 norm (Dettmers et al., 2023). In the degenerate case ^i,:= Q_i,:=0, the scale αi _i does not affect the objective and is set to 11 by convention. Theorem 1 (Convergence of DualQuant; proof in Appendix C). Let Assumption 1 hold, and let ((t),(t))t≥1\( α^(t), β^(t))\_t≥ 1 be the sequence generated by Algorithm 1. Suppose ^(t) Q^(t) stabilizes at some finite iteration T, i.e., ^(t)=^(T) Q^(t)= Q^(T) for all t≥Tt≥ T. Then: (i) Descent guarantee. MSE((t+1),(t+1))≤MSE((t),(t))MSE_W( α^(t+1), β^(t+1)) _W( α^(t), β^(t)) for all t≥Tt≥ T. (i) Stationarity. Every limit point (∗,∗)( α^*, β^*) is a coordinate-wise stationary point of equation 4, satisfying ∗=argminMSE(,∗) α^*= _ α\,MSE_W( α, β^*) and ∗=argminMSE(∗,) β^*= _ β\,MSE_W( α^*, β). The stabilization condition on Q is empirically supported by Figure 2(a), where its relative change decays to near zero within approximately T≈15T≈15 iterations for all layers and blocks. Furthermore, Figure 2(b) confirms Theorem 1(i): the MSE decreases monotonically and plateaus within a finite number of iterations. Figure 2: (a) Relative change ‖^(t)−^(t−1)‖F/‖^(t−1)‖F \| Q^(t)- Q^(t-1)\|_F/\| Q^(t-1)\|_F vs. BCD iteration for five transformer blocks. The change decays to near zero by t≈16t≈16 for all layers and blocks, validating the stabilization condition of Theorem 1. (b) Normalized MSE ‖−^‖F2/‖F2 \|W- W\|_F^2/\|W\|_F^2 vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within ≈10≈10 iterations, confirming Theorem 1(i). 4.4 Inference-time procedure and overhead The offline quantization procedure yields three artifacts per linear layer: the quantized weight matrix Q, row scales α, and column scales β. At inference, the layer computes Q(~)^⊤Diag()−1Q( X)\, Q \,Diag( α)^-1, where ~=Diag()−1 X=X\,Diag( β)^-1, and Q(~)Q( X) denotes the format-native row-wise quantization of the rescaled activations. The row scale α is absorbed into the per-row dequantization scale of Q at quantization time and incurs no runtime cost. The column scale β introduces a per-channel rescaling of the activations before quantization. It does not replace the format-native activation scale: Q(⋅)Q(·) still computes its own per-row scale on ~ X as usual. The two operate at different granularities: β attenuates outlier channels globally, while the format-native scale captures the residual per-token dynamic range. Unlike SmoothQuant and AWQ, which share a single scale across projections and fold it offline, DualQuant optimizes β independently per layer, which allows each layer to reach its own optimum. The resulting per-channel multiply is applied inside the activation-quantization kernel without a separate kernel launch (Section 5.2). 5 Experiments Models and formats. We evaluate two model families: Qwen (Qwen2.5-7B, Qwen3-0.6B, Qwen3-14B) and Llama (Llama-2-7B, Llama-3.1-8B, Llama-3.2-1B), covering small, medium, and large scales across two architectures. We target 4-bit quantization formats spanning the major families of modern low-bit deployment: INT4 (block size G=64G=64), MXINT4 (G=32G=32), MXFP4 (G=32G=32), and NVFP4 (G=16G=16). INT4 is the standard integer baseline (Frantar et al., 2023; Lin et al., 2024). For all formats, the obtained scales are constrained to F as described in Section 4.3. The corresponding licenses for all models are included in Appendix J. Evaluation setup. We evaluate DualQuant across a range of tasks capturing different aspects of model performance: language modeling ability via perplexity on WikiText-2 (Merity et al., 2017) and C4 (Raffel et al., 2020) (test splits), commonsense reasoning and language understanding via normalized zero-shot accuracy on MMLU (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-Easy (Clark et al., 2018), and WinoGrande (Sakaguchi et al., 2021). The corresponding licenses are included in Appendix J. All methods requiring calibration data (AWQ, SmoothQuant) use 256 randomly sampled sequences of length 8192 from the WikiText-2 training split; calibration and evaluation sets are strictly disjoint. We quantize all linear projections in every transformer block; lm_head is kept in full precision. All fake-quantization experiments are conducted on NVIDIA GH200 80 GB GPUs, with weights dequantized to FP16/BF16 during inference. Inference efficiency is evaluated separately using the real packed W4A4 implementation on a single NVIDIA RTX 4090 with 24 GB of memory (Section 5.2). Weight-only results and 8-bit W+A results are provided in Appendix H and Appendix G.2, respectively. 5.1 Joint weight-and-activation quantization Tables 1 (Llama family, main text) and 7 (Qwen family, Appendix G) compare DualQuant against five baselines: RTN, which applies round-to-nearest quantization to both weights and activations; AWQ+, which extends AWQ (Lin et al., 2024) with RTN activation quantization; SmoothQuant (Xiao et al., 2023), which uses calibration-based per-channel activation rescaling; SINQ+, our extension of SINQ (Müller et al., 2026) (see Appendix D.2); and QuaRot (Ashkboos et al., 2024), a calibration-free approach that applies rotation matrices to weights and activations to suppress outliers prior to quantization, followed by RTN activation quantization. Additional details on baselines are provided in Appendix F. Table 1 reports joint W+A quantization perplexity on the Llama family across four 4-bit formats. DualQuant consistently matches or surpasses the calibration-based SmoothQuant, while requiring no calibration data. On MXFP4, DualQuant improves over SmoothQuant on WikiText-2 for all three models, substantially on Llama-2-7B and Llama-3.2-1B and narrowly on Llama-3.1-8B; on C4 the same holds except for Llama-3.1-8B, where DualQuant trails slightly. On MXINT4, DualQuant is better than SmoothQuant across both WikiText-2 and C4 for all three models. On INT4, DualQuant surpasses both SmoothQuant and AWQ+ across all models, demonstrating that weight-derived scales provide effective activation rescaling even on the standard integer format. On NVFP4, DualQuant achieves the strongest results among all methods on Llama-3.1-8B and Llama-3.2-1B, with the same overall trend on C4. In contrast, both calibration-free baselines degrade substantially on MX and NV formats: SINQ+ degrades on MXINT4 and MXFP4 and diverges completely on NVFP4, as its format-unaware scales fall outside the FP8 representable range; QuaRot similarly diverges on NVFP4 and degrades substantially on MX formats, as its Hadamard rotation redistributes activation magnitude in a way incompatible with block floating-point scaling. DualQuant avoids both failure modes by design, incorporating format constraints directly into the optimization and requiring no rotation or calibration data. Corresponding results on the Qwen family (Table 7, Appendix G) show consistent trends.11 1 The DualQuant–SINQ+ gap persists without our extensions: in SINQ’s native weight-only INT4 setting, DualQuant is more accurate on three of four models (Appendix H), so it stems from the objective itself. Table 1: WikiText-2 and C4 perplexity (↓ ) for joint W+A quantization at 4-bit precision on the Llama family. DualQuant applies β-rescaling to activations without calibration data. SmoothQuant and AWQ+ use activation statistics from 256 calibration samples. SINQ+ is our extension of SINQ to W+A. QuaRot rotates activations/weights online with a Hadamard transform (no calibration). Bold: best calibration-free (CF) result. († ): degenerate result (PPL>1000)(PPL>1000); method incompatible with format. ✓: calibration-free. ✗: requires calibration data. Bits Format Method CF WikiText-2 PPL ↓ C4 PPL ↓ Llama-2-7B Llama-3.1-8B Llama-3.2-1B Llama-2-7B Llama-3.1-8B Llama-3.2-1B BF16 (reference) – 5.11 5.62 8.67 7.04 8.82 13.53 4-bit A: INT4 W: INT4 (G=64)(G=64) RTN ✓ 5.73 7.11 12.06 7.97 11.99 19.69 AWQ+ ✗ 5.57 6.83 12.46 7.73 11.51 20.24 SmoothQuant ✗ 5.68 6.98 13.37 7.91 11.75 21.99 SINQ+ ✓ 5.61 6.83 12.36 7.80 11.45 20.68 QuaRot ✓ 6.20 7.05 12.67 8.89 12.10 20.96 DualQuant ✓ 5.56 6.73 11.88 7.71 11.35 19.23 A: MXINT4 W: MXINT4 (G=32)(G=32) RTN ✓ 5.97 7.63 17.43 8.28 12.92 29.72 AWQ+ ✗ 5.74 7.28 17.18 7.97 12.28 28.17 SmoothQuant ✗ 5.91 7.48 15.79 8.31 12.60 26.86 SINQ+ ✓ 8.58 20.87 174.71 14.33 34.40 261.27 QuaRot ✓ 12.20 8.76 21.16 17.35 15.36 35.50 DualQuant ✓ 5.76 7.15 13.44 7.94 11.98 21.86 A: MXFP4 W: MXFP4 (G=32)(G=32) RTN ✓ 7.29 7.62 14.35 11.45 12.84 23.76 AWQ+ ✗ 6.44 7.82 13.57 9.10 12.94 22.26 SmoothQuant ✗ 6.77 7.44 13.94 10.67 12.51 23.11 SINQ+ ✓ 32.88 52.76 135.57 71.64 62.52 194.51 QuaRot ✓ 11.04 8.27 15.91 14.63 14.00 26.76 DualQuant ✓ 6.38 7.35 13.13 9.70 12.70 21.66 A: NVFP4 W: NVFP4 (G=16)(G=16) RTN ✓ 5.70 9.04 14.10 7.92 16.17 23.56 AWQ+ ✗ 6.51 22.79 15.90 9.53 41.31 26.33 SmoothQuant ✗ 5.53 7.32 12.40 7.63 12.30 20.26 SINQ+ ✓ Div† Div† Div† Div† Div† Div† QuaRot ✓ Div† Div† Div† Div† Div† Div† DualQuant ✓ 5.54 6.83 10.91 7.64 11.53 17.45 Table 2 reports zero-shot accuracy on MMLU, HellaSwag, PIQA, ARC-Easy, and WinoGrande for Llama-3.1-8B across all four W+A quantization formats. Corresponding results for the Qwen3-14B are reported in Table 8 in Appendix G. DualQuant achieves the best calibration-free accuracy across nearly all settings. On NVFP4, DualQuant surpasses every baseline, including the calibration-based SmoothQuant, across all five tasks, while both SINQ+ and QuaRot diverge completely on NVFP4, making DualQuant by far the strongest calibration-free method on this format. On MXINT4 and MXFP4, DualQuant substantially outperforms both calibration-free baselines on most tasks: SINQ+ collapses toward random-guess level, while QuaRot also degrades significantly. On INT4, where all methods remain viable, DualQuant achieves the best calibration-free accuracy on MMLU, HellaSwag, and ARC-Easy. Table 2: Llama-3.1-8B downstream accuracy (0-shot, %, higher is better ↑ ) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant, and QuaRot, across all four evaluated formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold: best calibration-free (CF) result per column within each model+format block. ✓: calibration-free. ✗: requires calibration data. Llama-3.1-8B Format Method CF MMLU HellaSwag PIQA ARC-Easy WinoGrande INT4 (G=64) RTN ✓ 54.89 56.44 77.86 77.40 68.90 AWQ+ ✗ 55.78 56.65 78.51 77.19 69.06 SmoothQuant ✗ 55.24 55.93 77.31 77.44 68.75 SINQ+ ✓ 55.50 56.68 77.09 77.65 67.72 QuaRot ✓ 54.21 56.16 77.04 77.40 70.17 DualQuant ✓ 55.61 56.83 77.37 77.68 69.77 MXINT4 (G=32) RTN ✓ 50.94 55.06 76.39 74.79 68.27 AWQ+ ✗ 52.30 55.31 76.77 73.82 69.61 SmoothQuant ✗ 52.93 54.83 76.61 75.88 66.14 SINQ+ ✓ 30.32 42.41 68.55 63.26 58.33 QuaRot ✓ 43.53 50.16 74.76 71.17 67.48 DualQuant ✓ 53.53 55.28 77.37 76.89 70.32 MXFP4 (G=32) RTN ✓ 51.51 54.17 76.71 76.43 68.03 AWQ+ ✗ 49.85 52.75 76.39 75.34 68.11 SmoothQuant ✗ 51.60 54.11 76.12 73.86 70.09 SINQ+ ✓ 25.36 37.60 64.36 54.21 55.64 QuaRot ✓ 48.69 52.81 75.84 77.19 68.90 DualQuant ✓ 52.10 54.66 76.77 76.22 69.69 NVFP4 (G=16) RTN ✓ 43.87 53.02 74.81 71.80 65.67 AWQ+ ✗ 31.19 45.10 71.06 63.17 62.75 SmoothQuant ✗ 55.67 55.60 77.53 75.80 68.90 SINQ+ ✓ 25.51 25.41 53.43 24.62 49.17 DualQuant ✓ 57.76 56.65 78.24 77.90 71.67 5.2 Runtime and memory analysis Offline quantization cost. The optimization that produces (,)( α, β) operates entirely on the static weight matrix with no calibration forward passes, no Hessian, and no inner optimization loop. Each coordinate update is a single closed-form expression, making the offline cost comparable to RTN and orders of magnitude cheaper than GPTQ or AWQ. Real packed W4A4 deployment. To verify that fake-quantization results translate to actual deployment, we implement a packed W4A4 inference path for Llama-3.1-8B (Grattafiori et al., 2024) on a single NVIDIA RTX 4090 (NVIDIA Corporation, 2022). The implementation uses INT4 with group size G=64G=64 and is built on the Nunchaku/SVDQuant (Li et al., 2025) W4A4 GEMM backend (Appendix I). Accuracy, memory usage, and latency are measured directly on the packed deployment path without fake quantization. WikiText-2 perplexity is 6.746.74, closely matching the fake-quantization result of 6.736.73 (Table 1) with an absolute difference of 0.010.01. This agreement confirms that the fake-quantization results in Section 5 accurately reflect the deployed model. Inference and memory overhead. The row scale α is folded into the native group scale and introduces no runtime overhead. The column scale β is incorporated into the activation-quantization kernel without an additional kernel launch, while weights remain packed throughout inference. Thus, activation quantization, including the application of β, constitutes the only per-step quantization overhead, accounting for 4.34.3-6.1%6.1\% of total W4A4 GPU time (Table 11). Despite this overhead, DualQuant W4A4 achieves 1.391.39-2.46×2.46× end-to-end speedup over BF16 across prefill and decode at batch sizes 11 and 1616 (Table 3). For Llama-3.1-8B relative to BF16, the packed deployment reduces the weight footprint from 16.2016.20 GB to 5.925.92 GB and end-to-end memory usage by 2.512.51-2.61×2.61× (Table 3). Component-level timing and memory breakdowns are provided in Appendix I (Table 12). Table 3: End-to-end memory usage and throughput of the real packed W4A4 deployment versus BF16 for Llama-3.1-8B on an RTX 4090 with INT4, G=64G=64. Prefill uses an 82-token prompt without a KV cache. Decode generates 100100 tokens with a 182-token KV cache. Memory includes weights, scales, unquantized embedding and lm_head, activations, and the KV cache for decode. Workload Memory (GB) Memory (GB) Memory Throughput (tok/s) Throughput (tok/s) BF16 W4A4 Reduction BF16 W4A4 Speedup Prefill (b=1)(b=1) 16.33 6.25 2.61×2.61× 3904.6 6421.4 1.64×1.64× Prefill (b=16)(b=16) 16.66 6.58 2.53×2.53× 8940.3 22003.4 2.46×2.46× Decode (b=1)(b=1) 16.34 6.26 2.61×2.61× 56.6 83.7 1.48×1.48× Decode (b=16)(b=16) 16.75 6.67 2.51×2.51× 694.5 964.1 1.39×1.39× Table 4: Scale ablation for W+A quantization (WikiText-2 PPL ↓ ). Bold: best per column. α-only fixes =n β=1_n; β-only fixes =m α=1_m. Variant Qwen2.5-7B Llama-3.1-8B MXFP4 NVFP4 MXFP4 NVFP4 RTN (no scales) 7.78 7.07 7.60 9.04 α-only 7.78 7.05 7.60 8.27 β-only 7.58 6.88 7.46 7.79 Both (+ α+ β) 7.45 6.53 7.35 6.83 Table 5: Per-layer Pearson correlation between DualQuant’s β and SmoothQuant’s calibrated scale s, on Llama-3.1-8B. Projection λ=0λ=0 λ=0.5λ=0.5 λ=1.0λ=1.0 q_proj +0.96 +0.09 −0.41-0.41 k_proj +0.97 +0.06 −0.57-0.57 v_proj +0.93 +0.53 +0.07 gate_proj +0.91 +0.55 −0.34-0.34 up_proj +0.83 +0.41 −0.28-0.28 5.3 Ablation studies Row scales, column scales, or both. Table 5 compares three variants of DualQuant on MXFP4 and NVFP4 W+A quantization. The α-only variant improves over RTN primarily on NVFP4, whose finer FP8 scale grid offers more degrees of freedom than MXFP4’s coarser E8M0 grid (e.g., 9.04→8.279.04→ 8.27 on Llama-3.1-8B), while the β-only variant yields consistently larger gains by simultaneously reducing column-wise weight quantization error and rescaling activations via the factorization of Section 4.2. Combining both yields the best results in all cases with roughly additive improvement, confirming that row and column scales target complementary sources of error. Hyperparameter selection. DualQuant has two hyperparameters: BCD iteration, T, and initialization (0,0)( α_0, β_0). The monotone-descent objective guarantees convergence from any starting point, but a good initialization reduces the iterations needed. We initialize α0,i=maxj|Wij| _0,i= _j|W_ij| (format-native absmax) and β0,j=∑i|Wij| _0,j= _i|W_ij| (per-column ℓ1 _1-norm). Considering Figure 2, we use T=15T=15, which we found sufficient across the evaluated settings. Correlation between the scales of DualQuant and SmoothQuant. To relate DualQuant’s weight-derived rescaling β to SmoothQuant’s calibration-derived scale, we compute the per-layer Pearson correlation (Pearson, 1895) across λ∈0,0.5,1.0λ∈\0,0.5,1.0\, the migration strength in SmoothQuant’s scale equation, on Llama-3.1-8B (Table 5). At λ=0λ=0 (purely weight-derived s), β aligns near-perfectly with SmoothQuant’s scale; correlation remains positive across most projections at λ=0.5λ=0.5 and decreases monotonically toward λ=1.0λ=1.0 (activation-dominated s), as expected for a weight-only method. Correlation is descriptive and does not imply equivalent quantization behavior. 6 Conclusion We present DualQuant, a calibration-free PTQ framework that jointly quantizes weights and activations by minimizing weight-reconstruction MSE via BCD with closed-form updates and provable convergence, operating across INT4, the MX family, and NVFP4. Its weight-driven column scales double as activation rescalings, so on Llama and Qwen DualQuant matches or surpasses calibration-based SmoothQuant and clearly outperforms calibration-free QuaRot and SINQ+, without any calibration data. A packed W4A4 path for Llama-3.1-8B further delivers up to 2.61× lower memory use and 2.46× higher throughput than BF16 on an RTX 4090. References Ashkboos et al. (2024) S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman QuaRot: outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: 5th item, §1, §2.2, §5.1. Badri and Shaji (2023) H. Badri and A. Shaji Half-quadratic quantization of large machine learning models. Note: https://dropbox.github.io/hqq_blog/ Cited by: §2.1. Behdin et al. (2023) K. Behdin, A. Acharya, A. Gupta, Q. Song, S. Zhu, S. Keerthi, and R. Mazumder QuantEase: optimization-based quantization for language models. arXiv preprint arXiv:2309.01885. Cited by: §2.1. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 34, p. 7432–7439. Cited by: §5. Bondarenko et al. (2021) Y. Bondarenko, M. Nagel, and T. Blankevoort Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §1, §2.2, §4.2. Boyd et al. (2011) S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3 (1), p. 1–122. Cited by: §A.1, §4.3. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §5. Czakó et al. (2025) P. Czakó, G. Kertész, and S. Szénási SmoothRot: combining channel-wise scaling and rotation for quantization-friendly LLMs. arXiv preprint arXiv:2506.05413. Cited by: §2.2. DeepSeek-AI (2024) DeepSeek-AI DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix E, §1, §1, §2.2, §4.2. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Assumption 1. Frantar et al. (2023) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), Cited by: Figure 3, §2.1, §5. Ghaffari et al. (2024) A. Ghaffari, S. Younesian, V. P. Nia, B. Chen, and M. Asgharian AdpQ: a zero-shot calibration free adaptive post training quantization method for LLMs. arXiv preprint arXiv:2405.13358. Cited by: §2.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Appendix I, §1, §5.2. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), Cited by: §5. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §1. Li et al. (2025) M. Li, Y. Lin, Z. Zhang, T. Cai, X. Li, J. Guo, E. Xie, C. Meng, J. Zhu, and S. Han SVDQuant: absorbing outliers by low-rank components for 4-bit diffusion models. In International Conference on Learning Representations (ICLR), Cited by: Appendix I, §5.2. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Conference on Machine Learning and Systems (MLSys), Cited by: 2nd item, §2.1, §5, §5.1. Liu et al. (2025) Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations (ICLR), Cited by: §1, §2.2. Ma et al. (2024) Y. Ma, H. Li, X. Zheng, F. Ling, X. Xiao, R. Wang, S. Wen, F. Chao, and R. Ji AffineQuant: affine transformation quantization for large language models. In International Conference on Learning Representations (ICLR), Cited by: §2.2. Merity et al. (2017) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), Cited by: §5. Müller et al. (2026) L. K. Müller, P. Bich, J. Zhuang, A. Celik, L. Benfenati, and L. Cavigelli SINQ: sinkhorn-normalized quantization for calibration-free low-precision LLM weights. In International Conference on Machine Learning (ICML), Cited by: Appendix D, 4th item, Appendix H, §2.1, §2.2, §5.1. NVIDIA Corporation (2022) NVIDIA Corporation NVIDIA GeForce RTX 4090. Note: https://w.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/Accessed: 2026-04-28 Cited by: Appendix I, §5.2. NVIDIA (2025) NVIDIA Introducing NVFP4 for efficient and accurate low-precision inference. Note: https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference Cited by: §1, §3. Pearson (1895) K. Pearson Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London 58, p. 240–242. External Links: Document Cited by: §5.3. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. Cited by: §5. Razaviyayn et al. (2013) M. Razaviyayn, M. Hong, and Z. Luo A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization 23 (2), p. 1126–1153. Cited by: §1, §4.3. Rouhani et al. (2023) B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537. Cited by: §1, §3. Sakaguchi et al. (2021) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM 64 (9), p. 99–106. Cited by: §5. Saxena et al. (2025) U. Saxena, S. Sharify, K. Roy, and X. Wang ResQ: mixed-precision quantization of large language models with low-rank residuals. In International Conference on Machine Learning (ICML), Cited by: §2.2. Shao et al. (2024) W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo OmniQuant: omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), Cited by: §2.2. Sinkhorn and Knopp (1967) R. Sinkhorn and P. Knopp Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics 21 (2), p. 343–348. External Links: Document Cited by: §D.1, §2.1. Sinkhorn (1964) R. Sinkhorn A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics 35 (2), p. 876–879. External Links: Document Cited by: §D.3. Sun et al. (2024) M. Sun, X. Chen, J. Z. Kolter, and Z. Liu Massive activations in large language models. In Conference on Language Modeling (COLM), Cited by: Appendix E, §1, §1, §4.2. Sun et al. (2025) Y. Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y. Li, J. Hu, X. Yu, L. Hou, C. Yuan, X. Jiang, et al. FlatQuant: flatness matters for LLM quantization. In International Conference on Machine Learning (ICML), Cited by: §2.2. Tseng et al. (2024) A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa QuIP#: even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning (ICML), Cited by: §2.2. Tseng (1991) P. Tseng Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM Journal on Control and Optimization 29 (1), p. 119–138. Cited by: §A.1. Tseng (2001) P. Tseng Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications 109 (3), p. 475–494. Cited by: §A.2, §A.2, Appendix C, Appendix C, §1, §4.3. Wei et al. (2022) X. Wei, Y. Zhang, X. Zhang, R. Gong, S. Zhang, Q. Zhang, F. Yu, and X. Liu Outlier suppression: pushing the limit of low-bit transformer language models. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.2. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), Cited by: 3rd item, §1, §1, §2.2, §4.1, §5.1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1. Yao et al. (2022) Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He ZeroQuant: efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2.2. Zandieh et al. (2026) A. Zandieh, M. Daliri, M. Hadian, and V. Mirrokni TurboQuant: online vector quantization with near-optimal distortion rate. In International Conference on Learning Representations (ICLR), Cited by: §2.2. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), p. 4791–4800. Cited by: §5. Appendix A Optimization background The dual-scale optimization problem introduced in Section 4.3 is non-convex and involves a non-differentiable quantization operator Q(⋅)Q(·), making direct gradient-based optimization infeasible. We address this by combining two classical techniques from optimization: variable splitting, which decouples the conflicting roles of the scale vectors, and block coordinate descent, which exploits the resulting structure to obtain closed-form updates. We briefly review both tools here. A.1 Variable splitting Variable splitting (Boyd et al., 2011; Tseng, 1991) is a technique for handling optimization problems where a single variable plays multiple conflicting roles in the objective or constraints. The idea is to introduce separate copies of the variable, one for each role, and enforce equality between the copies as an explicit constraint. This transforms a tightly coupled, difficult problem into a more structured one where each copy can be optimized independently given the others, subject to the coupling constraint. In our dual-scale formulation in equation 4, each scale vector plays two distinct roles: as an inner scale applied to the argument of Q(⋅)Q(·), which determines the quantization grid assignment, i.e., which discrete value each weight is mapped to, and as an outer scale applied after applying Q(⋅)Q(·), which controls how those discrete values are reconstructed in full precision. These two roles are in tension: the inner scale must be chosen to minimize quantization error by aligning the weight distribution with the quantization grid, while the optimal outer scale, given a fixed quantized matrix, is determined by a least-squares criterion that is blind to the quantization grid itself. Because both roles are played by the same variable, the objective in equation 4 is non-differentiable and non-convex in (,)( α, β) jointly. Variable splitting resolves this conflict by introducing independent copies (in,out)( α_in, α_out) and (in,out)( β_in, β_out), one pair for each scaling axis (row and column), and enforcing the consistency constraints in=out α_in= α_out and in=out β_in= β_out explicitly. With the inner scales (and hence ^in,in Q_ α_in, β_in held fixed,) the relaxed objective becomes bilinear in the outer scales out−1 α_out^-1 and out−1 β_out^-1, and decomposes into independent scalar least-squares problems per row and per column, each with a closed-form solution as derived in Section 4.3. After each pair of closed-form outer scale updates, the consistency constraints are re-enforced by synchronizing the inner scales with the newly computed outer scales, after which ^in,in Q_ α_in, β_in is recomputed before the next BCD iteration. For brevity, we drop the subscripts and write Q hereafter. This two-step structure is what makes the overall algorithm tractable and efficient in practice. The iterative procedure for cycling through these updates is formalised as Block Coordinate Descent, described next. A.2 Block coordinate descent Block Coordinate Descent (BCD) (Tseng, 2001) is a classical iterative optimization strategy for problems involving multiple groups of variables. Rather than jointly optimizing all variables simultaneously, which may be intractable or computationally prohibitive, BCD partitions the variable set into blocks and cycles through them, optimizing each block exactly (or approximately) while holding the remaining blocks fixed. Formally, given an objective f(1,2,…,K)f(u_1,u_2,…,u_K), BCD performs the update k(t+1)=argminkf(1(t+1),…,k−1(t+1),k,k+1(t),…,K(t))u_k^(t+1)= _u_k\;\;f\! (u_1^(t+1),…,u_k-1^(t+1),\,u_k,\,u_k+1^(t),…,u_K^(t) ) (6) for each block k=1,…,Kk=1,…,K in sequence, using the most recently updated values of all other blocks. Under mild regularity conditions, in particular when each subproblem admits a unique minimizer, BCD is guaranteed to converge to a stationary point of the objective (Tseng, 2001). The practical appeal of BCD lies in its ability to exploit problem structure: even when the joint problem is difficult, individual subproblems are often tractable. In our setting, once the inner scales and the quantized matrix Q are held fixed, the objective in equation 5 decomposes into independent scalar least-squares subproblems for each row scale αout,i _out,i and each column scale βout,j _out,j, each of which admits a closed-form solution. This makes BCD a natural and efficient solver for the dual-scale problem. Appendix B Closed-form optimal solutions for scales This appendix derives the closed-form updates via variable splitting. We introduce independent inner and outer copies of each scale vector. With inner scales (in,in)( α_in, β_in) fixed, the objective function is ℒ(out,out)=‖−Diag(out)−1^Diag(out)−1‖F2.L( α_out, β_out)= \|W-Diag( α_out)^-1 Q\,Diag( β_out)^-1 \|_F^2. (7) Closed-Form Update for out α_out. With out β_out and Q fixed, we define the auxiliary matrix =^Diag(out)−1R= Q\,Diag( β_out)^-1, which absorbs the fixed column scaling into the quantized values. The objective in equation 5 then reduces to MSE(out)=‖−Diag(out)−1‖F2=∑i=1m‖i,:−1αout,i,:‖22.MSE_W( α_out)= \|W-Diag( α_out)^-1R \|_F^2= _i=1^m \|W_i,:- 1 _out,iR_i,: \|_2^2. (8) This objective decomposes into m independent scalar subproblems, one per row, each involving only the scalar αout,i _out,i. To cast each subproblem as a standard least-squares problem, we substitute ui=αout,i−1u_i= _out,i^-1, which linearizes the dependence on the scale. Setting the derivative with respect to uiu_i to zero: ∂MSE∂ui=−2i,:(i,:−uii,:)⊤=0, ∂\,MSE_W∂ u_i=-2\,R_i,:\! (W_i,:-u_i\,R_i,: ) =0, (9) and solving for uiu_i gives ui∗=i,:i,:⊤∥i,:∥22.u_i^*= W_i,:\,R_i,: \|R_i,:\|_2^2. (10) Inverting back to the original parametrization yields the closed-form update for the i-th row scale: αout,i∗=∥i,:∥22i,:i,:⊤,wherei,:=^i,:Diag(out)−1. _out,i^*= \|R_i,:\|_2^2W_i,:\,R_i,: , _i,:= Q_i,:\,Diag( β_out)^-1. (11) Intuitively, αout,i∗ _out,i^* scales the i-th row so that the quantized row i,:R_i,: best approximates the original row i,:W_i,: in the least-squares sense: it is large when the quantized row underestimates the original, and small when it overestimates. After computing out∗ α_out^*, the format constraint out⊙−1∈αm α_out -1 _α^m\;\; is enforced by projecting each 1/αout,i∗1/ _out,i^* to the nearest representable value in αF_α. Closed-Form Update for out β_out. The derivation for the column scales follows the same structure. With out α_out and Q fixed, we define =Diag(out)−1^S=Diag( α_out)^-1 Q, which absorbs the fixed row scaling into the quantized values. The objective reduces to MSE(out)=‖−Diag(out)−1‖F2=∑j=1n‖:,j−1βout,j:,j‖22,MSE_W( β_out)= \|W-S\,Diag( β_out)^-1 \|_F^2= _j=1^n \|W_:,j- 1 _out,jS_:,j \|_2^2, (12) which decomposes into n independent scalar subproblems, one per column. Substituting vj=βout,j−1v_j= _out,j^-1 and setting the derivative with respect to vjv_j to zero: ∂MSE∂vj=−2:,j⊤(:,j−vj:,j)=0, ∂\,MSE_W∂ v_j=-2\,S_:,j \! (W_:,j-v_j\,S_:,j )=0, (13) and solving for vjv_j gives vj∗=:,j⊤:,j∥:,j∥22.v_j^*= S_:,j \,W_:,j\|S_:,j\|_2^2. (14) Inverting back yields the closed-form update for the j-th column scale: βout,j∗=∥:,j∥22:,j⊤:,j,where:,j=Diag(out)−1^:,j. _out,j^*= \|S_:,j\|_2^2S_:,j \,W_:,j, _:,j=Diag( α_out)^-1 Q_:,j. (15) As with the row scales, the format constraint out⊙−1∈βn β_out -1 _β^n\;\; is enforced by projecting each 1/βout,j∗1/ _out,j^* to the nearest representable value in βF_β after the update. Appendix C Proof of Theorem 1. We first establish the uniqueness of the per-block subproblem solutions (Lemmas 1 and 2), then prove the theorem. For brevity, define g(,):=‖−Diag()−1^(T)Diag()−1‖F2,g( α, β):= \|W-Diag( α)^-1\, Q^(T)\,Diag( β)^-1 \|_F^2, (16) the objective equation 4 with Q frozen at its post-stabilization value ^(T) Q^(T). By the stabilization hypothesis, as empirically verified in Figure 2 (main text) and Figure 3 (appendix), ^(t)=^(T) Q^(t)= Q^(T) for all t≥Tt≥ T, so for t≥Tt≥ T we have g((t),(t))=MSE((t),(t))g( α^(t), β^(t))=MSE_W( α^(t), β^(t)). For t<Tt<T, g and MSEMSE_W differ in general because ^(t)≠^(T) Q^(t)≠ Q^(T). Working with the function g in the post-stabilization regime allows us to invoke BCD convergence theorem, which requires a stationary objective. Figure 3: (a) Relative change ‖^(t)−^(t−1)‖F/‖^(t−1)‖F \| Q^(t)- Q^(t-1)\|_F/\| Q^(t-1)\|_F vs. BCD iteration for five transformer blocks. The ^(t) Q^(t) change decays to near zero by t≈16t≈16 for all layers and blocks, validating the stabilization condition of Theorem 1. (b) Normalized MSE ‖−^‖F2/‖F2 \|W- W\|_F^2/\|W\|_F^2 vs. BCD iteration for the same blocks. All curves decrease monotonically and plateau within ≈10≈10 iterations, confirming Theorem 1(i). Together with the Qwen2.5-7B results in Figure 2, this illustrates that the convergence behavior of DualQuant is consistent across the two model families evaluated here. Notably, down_proj consistently shows the highest residual MSE, consistent with it being among the harder projections to quantize (Frantar et al., 2023). Lemma 1 (Unique row-scale minimizer). Under Assumption 1, for any fixed ∈ℝ+n β _+^n, the subproblem ming(,) _ αg( α, β) admits a unique minimizer ∗∈ℝ+m α^* _+^m. Proof. The objective decomposes across rows as g(,)=∑i=1m∥αi−1i,:−i,:∥22g( α, β)= _i=1^m\| _i^-1R_i,:-W_i,:\|_2^2, where i,:=^i,:(T)Diag()−1R_i,:= Q^(T)_i,:Diag( β)^-1. Substituting ui=αi−1u_i= _i^-1 yields a strictly convex quadratic in uiu_i with unique minimizer ui∗=i,:i,:⊤/∥i,:∥22u_i^*=W_i,:\,R_i,: /\|R_i,:\|_2^2, which is positive under Assumption 1. Specifically, Assumption 1 ensures ∥i,:∥22>0\|R_i,:\|_2^2>0 (non-zero denominator). The numerator i,:i,:⊤W_i,:\,R_i,: is positive because all quantization formats considered use sign-preserving quantizers, meaning Q(x)Q(x) and x share the same sign for any nonzero x. Since αi,βj>0 _i, _j>0, it follows that Wi,kW_i,k and Ri,kR_i,k share the same sign for all k, making the inner product strictly positive. Therefore αi∗=1/ui∗∈ℝ+ _i^*=1/u_i^* _+ is the unique minimizer. ∎ Lemma 2 (Unique column-scale minimizer). Under Assumption 1, for any fixed ∈ℝ+m α _+^m, the subproblem ming(,) _ βg( α, β) admits a unique minimizer ∗∈ℝ+n β^* _+^n. Proof. Symmetric to the proof of Lemma 1, with rows and columns interchanged. ∎ Proof of Theorem 1. We prove the two parts of the theorem in turn: monotone descent (i) and stationarity of limit points (i). Part (i) — monotone descent. At iteration t≥Tt≥ T with (t) β^(t) and ^(T) Q^(T) fixed, Algorithm 1 sets (t+1) α^(t+1) to the unique minimizer of g(,(t))g( α, β^(t)) over α (Lemma 1). Since (t) α^(t) is feasible: g((t+1),(t))≤g((t),(t)).g( α^(t+1), β^(t))≤ g( α^(t), β^(t)). (17) By the symmetric argument, with (t+1) α^(t+1) and ^(T) Q^(T) fixed, Algorithm 1 sets (t+1) β^(t+1) to the unique minimizer of g((t+1),)g( α^(t+1), β) over β (Lemma 2), giving: g((t+1),(t+1))≤g((t+1),(t)).g( α^(t+1), β^(t+1))≤ g( α^(t+1), β^(t)). (18) Chaining equation 17 and equation 18: g((t+1),(t+1))≤g((t),(t))∀t≥T,g( α^(t+1), β^(t+1))≤ g( α^(t), β^(t)) ∀\,t≥ T, (19) establishing part (i). Since g≥0g≥ 0, the sequence g((t),(t))t≥T\g( α^(t), β^(t))\_t≥ T converges to a finite limit g∗≥0g^*≥ 0. Part (i) — Stationarity of limit points. We invoke Theorem 4.1(c) of (Tseng, 2001), which states that for cyclic block coordinate descent on a continuous objective where each block subproblem has a unique minimizer, every limit point of the iterates is a coordinate-wise stationary point. The three required conditions hold as follows. (a) Continuity of g. With ^(T) Q^(T) fixed, g(,)g( α, β) is a sum of rational functions of αi _i and βj _j with nonzero denominators on the open positive orthant αi,βj>0\ _i, _j>0\, hence continuously differentiable on its domain. All iterates remain in this domain by construction under Assumption 1. (b) Unique per-block minimizer. Lemmas 1 and 2 give uniqueness for both block subproblems. Uniqueness follows from the strict convexity of each scalar subproblem after the substitution ui=αi−1u_i= _i^-1 (resp. vj=βj−1v_j= _j^-1), which yields a strictly convex quadratic with a single minimizer. (c) Boundedness of the iterates. The level set (,):g(,)≤g((T),(T))\( α, β):g( α, β)≤ g( α^(T), β^(T))\ is compact 22 2 After fixing the trivial reciprocal scaling ambiguity ((,)∼(c,/c)CLOSE(( α, β) (c α, β/c), e.g., by normalizing one of the scale vectors.: g→∞g→∞ as any αi _i or βj _j approaches 00 or ∞ (whenever ^(T)≠ Q^(T) 0, ensured by Assumption 1). Combined with the monotone decrease of part (i), all iterates remain in this compact set. By Theorem 4.1(c) of (Tseng, 2001), every limit point (∗,∗)( α^*, β^*) of ((t),(t))t≥T\( α^(t), β^(t))\_t≥ T satisfies the coordinate-wise stationarity conditions ∗=argming(,∗) α^*= _ αg( α, β^*) and ∗=argming(∗,) β^*= _ βg( α^*, β), completing the proof of part (i). ∎ Remark 1 (Format-constrained projection). Theorem 1 is stated for unconstrained ∈ℝ+m α _+^m and ∈ℝ+n β _+^n. In practice, each scale is projected onto the discrete admissible set after each closed-form update. Since each subproblem in ui=αi−1u_i= _i^-1 (resp. vj=βj−1v_j= _j^-1) is a strictly convex quadratic, the objective increases monotonically with distance from the unconstrained minimizer. The nearest admissible point to this unconstrained minimizer therefore always achieves the lowest constrained objective, so projecting to the nearest representable value guarantees monotone decrease at each step. Remark 2 (On the stabilization assumption). Theorem 1 assumes that ^(t) Q^(t) stabilizes at a finite iteration T. Since Q takes values in the finite set m×nF^m× n, stabilization is guaranteed whenever the objective decreases monotonically across Q updates. We numerically observe this across model families and quantization formats. For example, for Qwen2.5-7B and Llama3.1-8B with MXINT4, Figures 2 and 3 show that ^(t) Q^(t) stabilizes within ≈12≈ 12–1616 iterations across every projection layer of every evaluated decoder block, well within our default of T=15T=15 iterations used in all experiments. Appendix D SINQ baseline: background and extension This appendix documents the SINQ (Müller et al., 2026) baseline used in our experiments. We briefly review SINQ’s scale-selection procedure (Appendix D.1), describe our extension to MX and NV formats (Appendix D.2), and explain why this extension exhibits the divergence behavior observed in Table 1 on those formats (Appendix D.3). D.1 Background SINQ is a calibration-free dual-scaling method that selects row scales ∈ℝ+m α _+^m and column scales ∈ℝ+n β _+^n to normalize the weight matrix prior to quantization, ˘=Diag()−1Diag()−1. W=Diag( α)^-1\,W\,Diag( β)^-1. (20) The scales are obtained by minimizing a matrix imbalance objective that drives the per-row and per-column standard deviations of ˘ W toward uniformity. The imbalance is reported as the ratio of the largest to the smallest per-axis standard deviation of ˘ W, ℐ(˘)=max(maxiσi(˘),maxjτj(˘))min(miniσi(˘),minjτj(˘)),I( W)= ( _i _i( W),\; _j _j( W) ) ( _i _i( W),\; _j _j( W) ), (21) where σi _i and τj _j denote the standard deviations of row i and column j of ˘ W. A perfectly balanced matrix has ℐ(˘)=1I( W)=1. SINQ minimizes ℐI via a Sinkhorn–Knopp-style alternating normalization (Sinkhorn and Knopp, 1967) that alternately divides each row and column of ˘ W by its current standard deviation, accumulating the reciprocal products into α and β. The procedure is calibration-free, processes each layer independently, and runs at roughly 1.1×1.1× the cost of RTN. SINQ was originally designed and evaluated for INT3, INT4, and NF4, all of which store the per-block scales in unconstrained BF16. D.2 Extending SINQ to W+A and MX/NV formats To produce baseline W+A results for MXINT4, MXFP4, and NVFP4, we extend SINQ in two directions: (i) to joint W+A quantization by applying SINQ’s column scales to rescale activations before quantization (same format as weights), and (i) to MX and NV formats by running SINQ’s algorithm to completion and then projecting the resulting scales to its target data format (E8M0 for MX, FP8 UE4M3 for NVFP4) as a post-processing step. Since SINQ’s optimization is unaware of scale format constraints, this projection only partially mitigates the mismatch: on MXINT4 and MXFP4 it leads to noticeable quality degradation, and on NVFP4 the scales drift entirely outside the FP8 representable range, causing complete divergence (marked † in the tables). Both extensions are our own and have not been validated by the original authors; SINQ+ results on MX and NV formats should accordingly be treated as best-effort baselines. We note that even in its original weight-only setting without any of our extensions, SINQ produces inferior scales to DualQuant across all formats (Appendix H). D.3 Why Sinkhorn iteration diverges under format constraints The large perplexity values observed for SINQ on MXINT4 and MXFP4 in Tables 1 and 7, and the complete divergence on NVFP4 (marked † ), indicate a fundamental incompatibility between SINQ’s format-unaware Sinkhorn iteration and the discrete scale constraints of MX and NV formats. In the unconstrained BF16 setting, Sinkhorn–Knopp iteration converges to a unique fixed point (∗,∗)( α^*, β^*) (Sinkhorn, 1964). The fixed-point scale values lie on a continuum and can be represented exactly in BF16. Once the admissible set is restricted to a discrete format such as E8M0, however, this property is lost. The fixed-point Sinkhorn target generally lies between two representable E8M0 values, and the projection step that follows each Sinkhorn iteration snaps the iterate to the nearest representable scale rather than the true fixed point. The next Sinkhorn step then re-derives a target that may point away from the projected iterate, causing the iteration to oscillate between nearby E8M0 representatives without converging. DualQuant avoids this failure mode by construction. Its closed-form coordinate updates are derived under the format constraint from the outset and the projection step preserves the monotone descent property of the BCD scheme (Theorem 1), so iterates remain in the admissible set throughout. D.4 Comparison of objectives Beyond the format-compatibility issue analyzed in Appendix D.3, SINQ’s matrix imbalance objective equation 21 measures a different quantity from the post-quantization reconstruction error: a lower imbalance does not necessarily translate to a lower MSE, since the imbalance metric is a function of ˘ W alone and makes no reference to the quantization grid, which means the same imbalance value corresponds to different post-quantization errors under INT4, MXINT4, MXFP4, and NVFP4. Our weight reconstruction MSE, in contrast, contains the format-specific quantizer Q(⋅)Q(·) explicitly, so the optimization is aware of the grid that the scales will be evaluated against and can adapt the scale assignment to non-uniform grids such as those of FP4 and NVFP4. This explains why a method optimized for low matrix imbalance does not automatically achieve low quantization error, as illustrated in Figure 1. Furthermore, the reconstruction MSE is more directly tied to downstream metrics such as perplexity and accuracy than proxy objectives such as matrix imbalance, which helps explain why DualQuant, by minimizing reconstruction error directly, exhibits stronger downstream performance than SINQ across most evaluated scenarios. Appendix E Isolated activation-side effect of β Section 4.2 establishes algebraically that the column scale β reshapes the per-channel statistics of the activations presented to the quantizer, and argues mechanistically that weight-MSE-optimal scales attenuate the same channels where activation outliers concentrate. To confirm this is a genuine improvement in activation quantization quality, and not merely a by-product of better weight quantization, we isolate β’s effect on activations by holding the quantized weights fixed. Setup. We take DualQuant’s quantized weight matrix Q Q for each projection in block 0 of Llama-3.1-8B (INT4, G=64G=64) as fixed and measure the Signal-to-Quantization-Noise Ratio (SQNR, dB, higher is better) of the quantized activations with and without applying β, using 256 representative input sequences. Block 0 is a representative early block; the pattern holds across depth. All other conditions are identical between the two runs. Results. Table 6 reports per-projection activation SQNR. Applying β improves activation SQNR by +0.33+0.33 dB on average across projections and by +0.53+0.53 dB at the block output level. The improvement is consistent across all seven projection types, with the largest gains on k_proj (+0.81+0.81 dB) and v_proj (+0.65+0.65 dB) — precisely the projections where activation outliers are most pronounced (Dettmers et al., 2022; Sun et al., 2024). This confirms that β provides a real, partial activation-side benefit that is separate from and additive to its role in reducing weight quantization error. The gain is not calibration substitution: unlike SmoothQuant, β is derived entirely from the static weight matrix, yet still measurably reduces the activation quantization error. Table 6: Per-projection activation SQNR (dB, ↑ ) with and without DualQuant’s β, on Llama-3.1-8B block 0 (INT4, G=64G=64). Quantized weights Q Q are held fixed; only the presence of the β pre-scaling varies. Δ>0 >0 indicates β reduces activation quantization error. Each Δ is computed at full precision and may differ slightly from the difference of the rounded columns. Other blocks exhibit the same qualitative behavior. Projection β applied β not applied Δ (dB) k_proj 17.61 16.80 +0.81+0.81 v_proj 17.45 16.80 +0.65+0.65 up_proj 18.48 18.20 +0.27+0.27 q_proj 17.05 16.80 +0.25+0.25 gate_proj 18.36 18.20 +0.15+0.15 o_proj 17.40 17.34 +0.07+0.07 down_proj 16.76 16.70 +0.06+0.06 Mean 17.59 17.26 +0.33+0.33 Block Output SQNR 17.98 17.44 +0.53+0.53 Appendix F Additional details on baselines This appendix details each baseline used in the joint W+A quantization comparison (Tables 1 and 7). All calibration-based baselines (AWQ+ and SmoothQuant) use the same calibration set: 256256 sequences of length 81928192 randomly sampled from the WikiText-2 training split, strictly disjoint from the evaluation splits. All baselines use the same block sizes and scale formats as DualQuant for each evaluated format (INT4 G=64G=64; MXINT4/MXFP4 G=32G=32 with E8M0; NVFP4 G=16G=16 with FP8 UE4M3). Below we describe the six baselines in detail. • RTN. Applies round-to-nearest quantization independently to weights and activations using format-native per-row or per-block absmax scaling, with no rescaling, calibration, or optimization. Activations are quantized dynamically at inference using per-token statistics. RTN serves as the lower-bound baseline. • AWQ+. AWQ (Lin et al., 2024) is a weight-only, activation-aware method that rescales weights based on high-activation channels. We extend it to W+A by adding RTN activation quantization after AWQ rescaling, and denote this method as AWQ+. Scales are computed using the official AWQ search procedure with a 20-candidate grid and clipping ratio search enabled. • SmoothQuant. SmoothQuant (Xiao et al., 2023) uses calibration data to move activation outliers into weights via per-channel rescaling sj∝maxi|ij|γ/maxi|ij|1−γs_j _i|X_ij|^γ/ _i|W_ij|^1-γ. We evaluate the migration strength γ∈0.5,0.85γ∈\0.5,0.85\ as suggested in the original paper (Xiao et al., 2023); since γ=0.5γ=0.5 consistently yields better performance across our experiments, we adopt it for SmoothQuant baselines. After rescaling, weights and activations are quantized using the same block-wise quantizer as DualQuant. It serves as the main calibration-based W+A baseline. • SINQ+. SINQ (Müller et al., 2026) is a calibration-free weight-only dual-scale method originally designed for INT3/INT4 and NF4 with unconstrained BF16 scales. We extend it to W+A and MX/NV formats as described in Appendix D.2; however, since SINQ’s optimization is entirely unaware of scale format constraints, projecting its output scales to the target data format only partially mitigates the mismatch, causing degraded results on MXINT4 and MXFP4 and complete divergence on NVFP4 (marked † ). SINQ+ results on MX and NV formats should be treated as best-effort baselines. • QuaRot. QuaRot (Ashkboos et al., 2024) is a calibration-free W+A method that suppresses activation outliers by applying randomized Hadamard rotations to the hidden states, spreading outlier energy across channels while leaving the network output unchanged, since each rotation is absorbed into the following weights. The residual-stream rotation is fused offline; the value-path and down-projection rotations run online. Following rotation, weights and activations are quantized using the same block-wise quantizer as DualQuant, with activations quantized dynamically based on post-rotation per-token statistics. QuaRot serves as the rotation-based counterpart to RTN, isolating the effect of rotation without weight-error compensation. Appendix G Additional W+A quantization results This appendix provides supplementary quantization results complementing the main experiments of Section 5.1. We report 4-bit W+A results on the Qwen model family and 8-bit W+A results across both families, confirming that DualQuant’s calibration-free activation rescaling generalizes across model architectures, quantization formats, and bit-widths. G.1 4-bit W+A quantization Perplexity. Table 7 reports WikiText-2 and C4 perplexity for joint W+A quantization on the Qwen family across all four 4-bit formats. Results are broadly consistent with the Llama family (Table 1 in the main text): DualQuant closely matches or surpasses SmoothQuant on the MX and NV formats without calibration data, and outperforms the calibration-free baselines QuaRot and SINQ+ in most settings, as these baselines frequently diverge or degrade severely on the block formats. On NVFP4, DualQuant surpasses SmoothQuant on Qwen2.5-7B and Qwen3-0.6B (e.g., 6.536.53 vs. 6.706.70 and 23.7723.77 vs. 24.1624.16 on WikiText-2) and closely matches it on Qwen3-14B (7.757.75 vs. 7.737.73), while SINQ+ diverges on all three models and QuaRot either diverges or degrades drastically (e.g., 12.1112.11 and 173.38173.38 on Qwen2.5-7B and Qwen3-0.6B), leaving DualQuant as the strongest calibration-free method on this format. On MXFP4, DualQuant outperforms SmoothQuant on Qwen3-0.6B (e.g., 27.4627.46 vs. 28.6628.66 on WikiText-2) and achieves substantially better C4 perplexity (42.5942.59 vs. 47.8347.83), while SINQ+ collapses (195.98195.98) and QuaRot degrades substantially (68.9668.96); on Qwen2.5-7B, QuaRot remains competitive (7.387.38), narrowly ahead of DualQuant (7.457.45). On MXINT4, DualQuant outperforms SmoothQuant on Qwen3-0.6B (32.9932.99 vs. 33.6333.63) and matches it on Qwen3-14B (8.428.42 vs. 8.428.42), while SINQ+ degrades severely (353.83353.83) and QuaRot collapses (131.31131.31). On INT4, DualQuant remains competitive with SmoothQuant across all three models, though slightly below on Qwen2.5-7B (7.387.38 vs. 7.107.10); QuaRot achieves the best calibration-free result on Qwen2.5-7B (6.976.97) but degrades substantially on Qwen3-0.6B (48.4748.47), while SINQ+ remains competitive on this format. The results on C4 are consistent with WikiText-2, confirming that the calibration-free activation rescaling generalizes across evaluation distributions. Overall, these results show that DualQuant provides consistently strong calibration-free W+A quantization across formats without relying on calibration data or format-specific rotation behavior. Table 7: WikiText-2 and C4 perplexity (↓ ) for joint weight-and-activation quantization at 4-bit precision on the Qwen family. DualQuant applies β-rescaling to activations without calibration data. SmoothQuant and AWQ+ use activation statistics from 256 calibration samples. SINQ+ is our extension of SINQ to W+A. QuaRot rotates activations/weights online with a Hadamard transform (no calibration). Bold: best calibration-free (CF) result. († ): degenerate result (PPL>1000)(PPL>1000); method incompatible with format. ✓: calibration-free. ✗: requires calibration data. WikiText-2 PPL ↓ C4 PPL ↓ Bits Format Method CF Qwen2.5-7B Qwen3-0.6B Qwen3-14B Qwen2.5-7B Qwen3-0.6B Qwen3-14B BF16 (reference) – 6.15 18.14 7.54 9.12 27.30 11.06 4-bit A: INT4 W: INT4 (G=64)(G=64) RTN ✓ 7.47 49.44 8.39 13.50 74.78 14.83 AWQ+ ✗ 7.36 38.80 8.19 13.21 59.20 14.43 SmoothQuant ✗ 7.10 30.67 8.12 12.74 50.73 14.21 SINQ+ ✓ 7.28 30.21 8.02 12.94 49.05 14.15 QuaRot ✓ 6.97 48.47 8.86 12.53 75.07 15.11 DualQuant ✓ 7.38 29.91 7.99 13.11 47.87 14.14 A: MXINT4 W: MXINT4 (G=32)(G=32) RTN ✓ 7.62 48.48 8.61 13.60 74.73 15.22 AWQ+ ✗ 7.52 38.61 8.44 13.39 60.99 14.73 SmoothQuant ✗ 7.36 33.63 8.42 13.08 52.34 14.75 SINQ+ ✓ 9.65 353.83 10.55 16.75 902.98 17.48 QuaRot ✓ 7.64 131.31 9.95 13.58 202.08 17.01 DualQuant ✓ 7.59 32.99 8.42 13.45 52.79 14.79 A: MXFP4 W: MXFP4 (G=32)(G=32) RTN ✓ 7.79 29.37 8.70 13.82 46.71 15.14 AWQ+ ✗ 7.62 29.88 8.62 13.59 50.44 15.05 SmoothQuant ✗ 7.68 28.66 8.53 13.63 47.83 14.95 SINQ+ ✓ 10.05 195.98 11.16 17.63 395.39 18.30 QuaRot ✓ 7.38 68.96 9.45 13.22 109.84 16.40 DualQuant ✓ 7.45 27.46 8.59 13.49 42.59 15.16 A: NVFP4 W: NVFP4 (G=16)(G=16) RTN ✓ 7.07 26.12 7.93 12.70 40.75 13.98 AWQ+ ✗ 6.80 25.60 7.95 12.24 41.95 13.91 SmoothQuant ✗ 6.70 24.16 7.73 12.17 37.00 13.64 SINQ+ ✓ Div† Div† Div† Div† Div† Div† QuaRot ✓ 12.11 173.38 Div† 26.37 280.53 Div† DualQuant ✓ 6.53 23.77 7.75 11.81 36.64 13.66 Zero-shot downstream tasks. Table 8 reports zero-shot accuracy on MMLU, HellaSwag, PIQA, ARC-Easy, and WinoGrande for Qwen3-14B across all four W+A quantization formats. Results are consistent with the Llama-3.1-8B results (Table 2 in the main text): DualQuant achieves the best calibration-free accuracy on nearly all format-task combinations. On MXFP4 and MXINT4, DualQuant outperforms both calibration-free baselines QuaRot and SINQ+ on every task, often by a large margin, and stays close to calibration-based SmoothQuant, surpassing it on WinoGrande (72.06 vs. 69.14 on MXFP4 and 70.64 vs. 66.30 on MXINT4) while trailing narrowly on the remaining tasks (e.g., 72.91 vs. 73.26 MMLU on MXFP4 and 71.47 vs. 72.50 MMLU on MXINT4). On NVFP4, DualQuant is the strongest calibration-free method: SINQ+ collapses to near-random accuracy (23.39 MMLU), whereas DualQuant stays competitive with calibration-based SmoothQuant (e.g., 75.06 vs. 75.97 MMLU, 78.99 vs. 78.94 PIQA), closing most of the gap without any access to calibration data. On INT4, DualQuant matches SmoothQuant closely across all five tasks and attains the best calibration-free accuracy on four of the five, trailing only SINQ+ on PIQA (76.71 vs. 77.80). As on Llama, SINQ+ degrades severely on MX formats (62.38–62.58 MMLU on MXINT4 and MXFP4) and collapses on NVFP4, confirming that the perplexity improvements of DualQuant over calibration-free baselines translate directly to downstream task quality. Table 8: Qwen3-14B downstream accuracy (0-shot, %, higher is better ↑ ) for RTN, AWQ+, SmoothQuant, SINQ+, DualQuant, and QuaRot, across all 4 weight formats; QuaRot is omitted entirely for NVFP4, since QuaRot’s Hadamard rotation redistributes activation magnitude in a way that collapses NVFP4’s narrow-range E4M3 block scale. Bold: best calibration-free (CF) result per column within each model+format block. ✓: calibration-free. ✗: requires calibration data. Qwen3-14B Format Method CF MMLU HellaSwag PIQA ARC-Easy WinoGrande INT4 (G=64) RTN ✓ 70.33 75.23 77.64 77.36 69.06 AWQ+ ✗ 71.92 75.03 77.86 79.08 69.53 SmoothQuant ✗ 73.04 76.74 77.91 80.77 71.51 SINQ+ ✓ 72.30 76.23 77.80 79.59 70.17 QuaRot ✓ 71.30 73.52 76.77 77.99 67.88 DualQuant ✓ 72.33 76.54 76.71 80.51 71.09 MXINT4 (G=32) RTN ✓ 70.08 74.31 76.69 74.74 66.08 AWQ+ ✗ 72.42 74.93 77.37 77.53 66.93 SmoothQuant ✗ 72.50 75.84 77.97 78.62 66.30 SINQ+ ✓ 62.38 69.26 74.59 70.50 63.54 QuaRot ✓ 68.72 69.92 75.63 75.72 63.85 DualQuant ✓ 71.47 75.27 77.04 76.68 70.64 MXFP4 (G=32) RTN ✓ 72.36 75.59 77.15 79.76 69.14 AWQ+ ✗ 72.99 75.89 77.15 79.46 70.72 SmoothQuant ✗ 73.26 75.53 77.97 79.46 69.14 SINQ+ ✓ 62.58 68.53 74.16 73.02 63.85 QuaRot ✓ 68.34 71.01 76.71 72.85 68.90 DualQuant ✓ 72.91 75.55 77.48 80.01 72.06 NVFP4 (G=16) RTN ✓ 74.55 76.97 78.29 80.09 71.27 AWQ+ ✗ 74.84 77.32 78.78 81.73 71.03 SmoothQuant ✗ 75.97 77.55 78.94 82.79 72.77 SINQ+ ✓ 23.39 26.38 50.22 26.56 50.59 DualQuant ✓ 75.06 77.09 78.99 81.90 72.64 G.2 8-bit W+A quantization Table 9: WikiText-2 perplexity (↓ ) for 8-bit W+A quantization. DualQuant applies β-rescaling without calibration data. SmoothQuant uses 256 calibration samples. SINQ+ is our extension of SINQ to W+A. Bold: best calibration-free (CF) result. ✓: calibration-free. ✗: requires calibration data. Bits Format Method CF Qwen2.5-7B Qwen3-0.6B Qwen3-14B Llama-3.1-8B BF16 (reference) – 6.15 18.14 7.54 5.62 8-bit A: INT8 W: INT8 (G=128)(G=128) RTN ✓ 6.27 18.42 7.55 5.72 SmoothQuant ✗ 6.25 18.28 7.50 5.70 SINQ+ ✓ 6.67 18.33 7.49 5.70 DualQuant ✓ 6.64 18.24 7.77 5.69 A: MXINT8 W: MXINT8 (G=32)(G=32) RTN ✓ 6.15 18.06 7.54 5.63 SmoothQuant ✗ 6.15 18.18 7.55 5.63 SINQ+ ✓ 7.37 47.87 9.04 9.51 DualQuant ✓ 6.15 18.13 7.53 5.63 A: MXFP8 W: MXFP8 (G=32)(G=32) RTN ✓ 6.22 18.54 7.62 5.72 SmoothQuant ✗ 6.22 18.46 7.63 5.71 SINQ+ ✓ 7.49 53.41 9.21 10.48 DualQuant ✓ 6.21 18.23 7.64 5.71 Table 9 reports WikiText-2 perplexity for joint W+A quantization at 8-bit precision across INT8, MXINT8, and MXFP8. At 8 bits the quantization error is already small, so there is limited room for improvement over BF16; nonetheless, DualQuant remains the strongest calibration-free method and is effectively lossless. On MXINT8 and MXFP8 it matches or surpasses the calibration-based SmoothQuant on every model except MXFP8 Qwen3-14B, where it trails by only 0.010.01 (7.647.64 vs. 7.637.63), all without any calibration data; representative gains include 18.1318.13 vs. 18.1818.18 on Qwen3-0.6B and 7.537.53 vs. 7.557.55 on Qwen3-14B for MXINT8. On MXINT8, DualQuant tracks the BF16 reference almost exactly (e.g., 6.156.15 vs. 6.156.15 on Qwen2.5-7B and 18.1318.13 vs. 18.1418.14 on Qwen3-0.6B), confirming near-lossless quantization. In contrast, the other calibration-free baseline, SINQ+, degrades severely on the MX formats even at 8 bits (e.g., 47.8747.87 and 53.4153.41 on Qwen3-0.6B for MXINT8 and MXFP8), confirming that its format-unaware scales remain incompatible with MX block constraints regardless of bit-width and underscoring the benefit of DualQuant’s format-aware dual scaling. Appendix H Weight-only quantization results While the primary contribution of DualQuant is calibration-free joint W+A quantization, Table 10 evaluates its weight-only performance, with activations kept in BF16, to assess whether the MSE-based dual scaling also yields competitive weight reconstruction in isolation. DualQuant achieves the best calibration-free results on nearly all format–model combinations (on Qwen3-14B it is marginally behind SINQ on INT4 and RTN on NVFP4), closely matching calibration-based AWQ on INT4 and MX formats and remaining within a small margin of GPTQ without requiring any calibration data or Hessian computation. While GPTQ relies on costly second-order weight updates and is inapplicable when training data is proprietary, DualQuant operates purely from weight statistics, yet closely follows its perplexity in INT4 and MX formats and outperforms it on NVFP4 for three of the four models. On MXINT4 and MXFP4, DualQuant substantially outperforms SINQ, whose format-unaware scales cause severe degradation (e.g., 112.00112.00 and 94.8494.84 on Qwen3-0.6B for MXINT4 and MXFP4 respectively). On NVFP4, DualQuant achieves the strongest results among all methods on Qwen2.5-7B (6.336.33) and Qwen3-0.6B (19.8219.82). These results confirm that DualQuant is highly effective in the weight-only setting, with MSE minimization directly translating into measurable perplexity gains. More significantly, the same optimization simultaneously enables calibration-free W+A quantization (Table 1), distinguishing DualQuant from prior weight-only methods. Importantly, the SINQ results in Table 10 use the original SINQ algorithm (Müller et al., 2026) with no activation scaling involved. For MX and NV formats the only modification is projecting SINQ’s output scales to the target data format after the algorithm completes, as there is no other way to deploy them in these formats. Even in this purely weight-only setting, DualQuant outperforms SINQ on nearly all formats and models. Notably, on INT4, where SINQ operates exactly as designed with unconstrained BF16 scales, DualQuant achieves better perplexity on three out of four models. On MX and NV formats the gap widens dramatically, confirming that DualQuant provides better generalizability to block quantization formats. Table 10: WikiText-2 perplexity (↓ ) for 4-bit weight-only quantization. Activations in BF16. DualQuant applies β-rescaling without calibration data. AWQ and GPTQ use 256 calibration samples. († ): degenerate result (PPL>1000>1000); method incompatible with format. Bold: best calibration-free (CF) result. ✓: calibration-free. ✗: requires calibration data. Bits Format Method CF Qwen2.5-7B Qwen3-0.6B Qwen3-14B Llama-3.1-8B BF16 (reference) – 6.15 18.14 7.54 5.62 4-bit W: INT4 (G=64)(G=64) RTN ✓ 6.67 26.33 7.93 6.32 AWQ ✗ 6.50 21.83 7.72 6.12 GPTQ ✗ 6.44 20.39 7.73 5.56 SINQ ✓ 6.57 22.21 7.76 6.16 DualQuant ✓ 6.55 21.35 7.88 6.12 W: MXINT4 (G=32)(G=32) RTN ✓ 6.66 25.99 7.98 6.41 AWQ ✗ 6.49 21.90 7.85 6.16 GPTQ ✗ 6.49 20.09 7.79 6.25 SINQ ✓ 7.98 112.00 9.67 13.09 DualQuant ✓ 6.60 23.02 7.89 6.19 W: MXFP4 (G=32)(G=32) RTN ✓ 6.79 22.38 8.04 6.29 AWQ ✗ 6.58 20.98 7.94 6.31 GPTQ ✗ 6.47 19.79 7.79 6.11 SINQ ✓ 8.03 94.84 9.85 16.16 DualQuant ✓ 6.63 22.35 7.95 6.19 W: NVFP4 (G=16)(G=16) RTN ✓ 6.85 23.35 7.81 8.12 AWQ ✗ 6.61 22.56 7.89 7.93 GPTQ ✗ 6.55 20.41 7.74 7.61 SINQ ✓ Div† Div† Div† Div† DualQuant ✓ 6.33 19.82 8.04 6.43 Appendix I Real packed W4A4 deployment Implementation. The packed inference path targets INT4 with group size G=64G=64 and is built on the packed W4A4 GEMM backend from Nunchaku/SVDQuant (Li et al., 2025). The quantized weights, Q Q, and scales produced by Algorithm 1 for all 224224 linear layers of Llama-3.1-8B (Grattafiori et al., 2024) (77 projections × 3232 transformer blocks) are packed offline into the target storage layout and executed through this backend. Embedding, lm_head, and normalization layers remain in BF16. We extend the backend with two components: (i) a CUDA kernel that incorporates the column scale β into activation quantization, avoiding a separate kernel launch, and (i) an allocation-free GEMM entry point that enables CUDA Graph capture for the full model. Evaluation setup. All measurements are performed on a single NVIDIA RTX 4090 (NVIDIA Corporation, 2022) with 2424 GB of memory. WikiText-2 perplexity is evaluated at sequence length seq=8192seq=8192. Latency and memory are measured using an 8282-token prompt for prefill without a KV cache and 100100 generated tokens for decode, corresponding to a 182182-token KV cache. All reported measurements are collected from warm runs and remain stable within 0.2%0.2\% across repetitions. Accuracy. The packed W4A4 implementation achieves a WikiText-2 perplexity of 6.746.74, compared with 6.736.73 under fake quantization (Table 1), a difference of only 0.010.01. The BF16 baseline measured using the same evaluation harness has a perplexity of 5.615.61, compared with 5.625.62 reported in Table 1. Latency breakdown. Table 11 reports the GPU time for individual components under BF16 and DualQuant W4A4 with INT4 (G=64G=64) format. Quantized weights remain packed throughout inference, and group scales are applied within the GEMM, eliminating runtime weight dequantization. Activation quantization, including the β pre-scaling operation, costs 0.700.70–3.093.09 ms and accounts for 4.34.3–6.1%6.1\% of total W4A4 GPU time, constituting the only per-step quantization overhead. The largest reduction comes from the quantized linear layers, whose execution time decreases from 17.3317.33 to 8.828.82 ms for prefill with batch size 11 and from 121.49121.49 to 32.9832.98 ms for prefill with batch size 1616. The remaining computation consists primarily of components retained in BF16, including lm_head, attention, normalization, RoPE, residual operations, and KV-cache updates, which limits the achievable decode speedup. Note that component-level timings are collected in a separate instrumented run; the associated measurement overhead makes the totals in Table 11 slightly larger than the end-to-end times reported in Table 3. Table 11: GPU time (ms) per component for BF16 and DualQuant W4A4, INT4 (G=64G=64), on Llama-3.1-8B using an RTX 4090. Component Prefill (b=1b=1) Prefill (b=16b=16) Decode (b=1b=1) Decode (b=16b=16) BF16 W4A4 BF16 W4A4 BF16 W4A4 BF16 W4A4 Quantized linears (224) 17.33 8.82 121.49 32.98 14.75 8.24 16.32 9.18 Activation quantization (incl. β) – 0.82 – 3.09 – 0.70 – 0.72 lm_head (BF16) 1.19 1.19 9.26 9.26 1.07 1.07 1.10 1.10 Attention 0.19 0.20 0.93 0.94 0.42 0.42 1.73 1.74 Norms / RoPE / KV cache / residual 2.43 2.33 14.03 13.37 1.60 1.64 4.18 4.14 Total GPU time 21.14 13.36 145.71 59.64 17.84 12.07 23.33 16.88 Memory breakdown. Table 12 provides an end-to-end memory breakdown of the DualQuant packed deployment for Llama-3.1-8B when quantized to INT4 (G=64G=64). Quantization reduces the storage for the 224224 linear layers from 13.9613.96 GB to 3.493.49 GB, with an additional 0.220.22 GB required for per-group weight scales. Together with the unquantized BF16 embedding and lm_head (2.102.10 GB), the total weight footprint decreases from 16.2016.20 GB to 5.925.92 GB. The remaining activation and KV-cache components stay in BF16 and therefore have the same footprint in both paths. Prefill uses use_cache=False, so no KV cache is allocated; its activation footprint is dominated by the [seq×128256][seq× 128256] logits tensor. Consequently, memory usage for prefill with batch size 11 is slightly lower than for decode with batch size 11. Table 12: End-to-end memory breakdown (GB) for BF16 and DualQuant W4A4 on Llama-3.1-8B using an RTX 4090. Component Prefill (b=1b=1) Prefill (b=16b=16) Decode (b=1b=1) Decode (b=16b=16) BF16 W4A4 BF16 W4A4 BF16 W4A4 BF16 W4A4 Quantized linear weights 13.96 3.49 13.96 3.49 13.96 3.49 13.96 3.49 Per-group weight scales (BF16) – 0.22 – 0.22 – 0.22 – 0.22 Embedding + lm_head (BF16) 2.10 2.10 2.10 2.10 2.10 2.10 2.10 2.10 Runtime/allocator overhead 0.14 0.11 0.14 0.11 0.14 0.11 0.14 0.11 Weights subtotal 16.20 5.92 16.20 5.92 16.20 5.92 16.20 5.92 First-forward scratch 0.10 0.30 0.10 0.30 0.10 0.30 0.10 0.30 KV cache 0 0 0 0 0.02 0.02 0.38 0.38 Other activations 0.03 0.03 0.36 0.36 0.02 0.02 0.07 0.07 Total 16.33 6.25 16.66 6.58 16.34 6.26 16.75 6.67 Appendix J Licenses for existing assets All models, datasets, and software libraries used in this work are publicly available and used in accordance with their respective licenses. Table 13 summarizes the assets along with their licenses and sources. The Llama models are released under Meta’s Llama Community Licenses and the Qwen models under Apache 2.0. The evaluation datasets are released under a mix of permissive and share-alike licenses: WikiText-2 (C BY-SA 3.0), C4 (ODC-BY 1.0), HellaSwag (MIT), MMLU (MIT), PIQA (AFL-3.0), ARC-Easy (C BY-SA 4.0), and WinoGrande (C-BY). All assets were used solely for academic research purposes consistent with their terms of use. Table 13: Licenses of existing assets used in this work. Category Assets and Licenses Models Llama-2-7B (Llama 2 Community); Llama-3.1-8B (Llama 3.1 Community); Llama-3.2-1B (Llama 3.2 Community); Qwen2-1.5B, Qwen2.5-7B, Qwen3-0.6B, Qwen3-14B (Apache 2.0) Datasets WikiText-2 (C BY-SA 4.0); C4 (ODC-BY 1.0); MMLU (MIT); HellaSwag (MIT); PIQA (AFL-3.0); ARC-Easy (C BY-SA 4.0); WinoGrande (C-BY)