Paper deep dive
ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
Mehdi Makni, Ryan Lucas, Rahul Mazumder
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/30/2026, 7:59:41 AM
Summary
The paper introduces ThinQuant, a scalable method for learning rotations to enable low-bit weight and activation quantization in Large Language Models (LLMs). It addresses computational bottlenecks in existing gradient-free methods like DartQuant by employing a geometric data selection procedure that reduces calibration data requirements by orders of magnitude. This reduction allows the optimization problem to be mapped to a lower-dimensional Stiefel manifold, solved efficiently via an ADMM algorithm. ThinQuant demonstrates significant speedups and improved perplexity on Llama-3-70B and Llama-3.1-405B models compared to SpinQuant, DartQuant, and GPTAQ+QuaRoT.
Entities (10)
Relation Signals (6)
ThinQuant → appliesto → Llama 3 70B
confidence 95% · For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration
ThinQuant → appliesto → Llama 3.1-405B
confidence 95% · ThinQuant also scales to Llama-3.1-405B
ThinQuant → outperforms → DARTQuant
confidence 95% · ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes.
ThinQuant → uses → ADMM
confidence 95% · We solve this reduced problem using an efficient ADMM algorithm
ThinQuant → outperforms → GPTAQ+QuaRoT
confidence 90% · achieves WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.
ThinQuant → reducesoptimizationto → Stiefel manifold
confidence 90% · making optimization over a d×d rotation equivalent to optimizing a d×r matrix on the Stiefel manifold.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (ii) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an $r$-dimensional subspace with $r<d$, making optimization over a $d\times d$ rotation equivalent to optimizing a $d\times r$ matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT.
Tags
Links
- Source: https://arxiv.org/abs/2609.36120v1
- Canonical: https://arxiv.org/abs/2609.36120v1
Trouble viewing inline? Open PDF directly →
Full Text
71,222 characters extracted from source content.
Expand or collapse full text
ThnQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs Mehdi Makni, Ryan Lucas11footnotemark: 1 , Rahul Mazumder Operations Research Center Massachusetts Institute of Technology †thanks: Equal contribution. Abstract Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based procedures such as SpinQuant and computationally friendlier gradient-free approaches such as DartQuant, but both remain hard to scale to the largest architectures. To address the computational bottlenecks in gradient-free rotation learning, we introduce two ideas for efficiency, (i) a data selection procedure which reduces the required number of calibration data points, and (i) an exact reduction of the associated optimization on this reduced calibration set. Our data selection procedure exploits the geometric structure of the convex hull of the activations. Using this idea, we show that a carefully selected calibration set with several orders of magnitude fewer activations than state-of-the-art rotation-based methods can match their performance in low-bit quantization settings. Under this extreme data efficiency, the selected activations span an r-dimensional subspace with r<dr<d, making optimization over a d×d× d rotation equivalent to optimizing a d×rd× r matrix on the Stiefel manifold. We solve this reduced problem using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant. For Llama-3-70B with W4A4KV4 quantization, ThinQuant completes the entire rotation calibration in under 12 minutes and achieves a WikiText-2 perplexity of 5.63, compared with 7.55 for DartQuant, which requires 111 minutes. Unlike SpinQuant and DartQuant, ThinQuant also scales to Llama-3.1-405B on a single H200 GPU, completing rotation calibration in just over 2 hours and achieving WikiText-2 perplexity of 2.97 at W4A4, compared with 3.48 for GPTAQ+QuaRoT. 1 Introduction State-of-the-art large language models (LLMs) contain billions of parameters and process long token sequences, leading to large memory footprints and slow inference time. As a result, post-training quantization (PTQ), where a pretrained checkpoint is compressed without retraining from scratch, has become a central tool for making LLM inference more efficient (Dettmers et al., 2022). Weight-only PTQ (e.g., INT8 or INT4 weights with higher-precision activations) substantially reduces memory and storage, and methods such as GPTQ (Frantar et al., 2023), GPTAQ (Li et al., 2025), and ADMM-Q (Lucas et al., 2026) achieve near-lossless accuracy even at 4-bit weight precision. However, weight-only schemes leave the dominant computation—matrix–vector products during inference—executed in higher precision (e.g. FP16 or BF16). Joint weight and activation quantization gives greater end-to-end speedups, since the core matrix multiplications can be executed by INT8 or INT4 tensor cores, which offer higher peak throughput than FP16 on modern GPUs. Joint weight and activation quantization is substantially harder than weight-only quantization. Weights are fixed after training and can be compressed offline using expensive optimization procedures. The resulting quantized tensors are then stored and reused at every inference step. Activations, by contrast, arrive at runtime, are different for every input, and must be quantized on-the-fly with a cheap deterministic rule, typically round-to-nearest (RTN) on a uniform grid. Moreover, it has been observed that for activations a small subset of input channels, often fewer than 1%1\% of dimensions, carry values that are orders of magnitude larger than the remaining channels (Dettmers et al., 2022; Xiao et al., 2023). Rotations address this by changing the coordinate system of the network before quantization which does not change the full precision outputs, but makes the activations substantially easier to quantize. For an orthogonal transformation R∈ℝd×dR ^d× d, a linear map can be equivalently written as Wx=(WR)(Rx)Wx=(WR T)(Rx), so that activations are represented in the rotated basis while the inverse transformation is absorbed into the surrounding weights. Modern LLM quantization methods exploit this invariance at both global and local scales (Liu et al., 2025; Shao et al., 2025). A global change of basis uses a shared rotation across the hidden representation of multiple layers. Local changes of basis instead act within individual layers or modules, for example inside attention or feed-forward blocks, and can therefore use different rotations at different locations in the network. In both cases, the transformations can be absorbed into adjacent linear operators so that the full-precision network remains functionally identical, while its weights and activations can have substantially more favorable distributions for low-bit quantization. Moreover, rotations can be incorporated into quantized inference pipelines to accelerate inference runtime; we refer the reader to (Liu et al., 2025), Section 4.5. SpinQuant learns rotations by optimizing an end-to-end loss through the quantized network using a straight-through estimator (Liu et al., 2025). While effective, this requires repeated forward and backward passes through the full network, making optimization expensive and potentially prone to overfitting to the fine-tuning objective. DartQuant substantially reduces this cost by replacing end-to-end training with a calibration objective on the activations (Shao et al., 2025). However, this still requires optimizing against a very large collection of activations, which leads to large computational cost. In its standard calibration setting, DartQuant uses a sample α of the tokens generated by N=128×2048≈2.6×105N=128× 2048≈ 2.6× 10^5 samples from each layer. Across L transformer blocks, the rotation is therefore calibrated using αNLα NL activation vectors, where α is the sampling ratio, set to 10%10\% by default. For Llama-3-70B, L=80L=80, this is on the order of millions of activation vectors. ThinQuant, using geometry-aware activation selection, takes this to extreme efficiency: using only a few hundred activation vectors from the entire network, ThinQuant gives dramatic savings in disk storage (≈2000×≈ 2000×) and runtime (≈10×≈ 10×) for Llama-3-70B with performance comparable to more expensive state-of-the-art methods (e.g. SpinQuant, DartQuant), see Figure 1 and Table 4. Figure 1: Comparison of Wikitext-2 perplexity versus calibration cost for different rotation-based methods on LLaMA-3 series models at W4A4KV4. Using our smart activation selection procedure, ThinQuant can operate with a very small calibration dataset, mitigating the shortcomings of existing gradient-free methods such as DartQuant. We briefly provide some intuition for our selection procedure (see Section 3.1 for details). We mitigate the quantization error caused by large outliers in the activations space by controlling the ℓ∞ _∞ norm of the activations after rotation, similar to prior work such as DartQuant and QuaRot (Ashkboos et al., 2024). Our key observation is that these worst-case magnitudes are determined by the convex hull of the activations, rather than by the full activation distribution. For a calibration set =x1,…,xN⊂ℝdA=\x_1,…,x_N\ ^d, the extreme points of the symmetric convex hull K=conv(∪(−))K=conv(A∪(-A)) captures the largest absolute projection in every direction. Its extreme points therefore suffice to determine the worst-case objective, while the interior activations can be discarded without changing it. Our data selection procedure thus selects geometrically prominent extreme points in the activations across many possible bases, rather than only those that are outliers in the original coordinates. In practice, this geometric selection lets us replace hundreds of thousands of calibration tokens per layer with a tiny subset of activations. Given this reduced number of data points, we show that loss functions commonly used to control quantization error (e.g. ℓ∞ _∞, kurtosis) admit an exact reduction to a smaller optimization problem. Generally, the budget B we use for activation vectors (order 100) is much smaller than the hidden dimension d (order thousands). Let U∈ℝd×BU ^d× B collect the selected activation vectors, and let r=rank(U)≤Br=rank(U)≤ B. Writing a thin factorization U=QSU=QS, the columns of Q∈ℝd×rQ ^d× r are orthonormal, so Q∈St(d,r)=Y∈ℝd×r:YY=IrQ (d,r)=\Y ^d× r:Y TY=I_r\. Since the calibration objective depends on the full rotation R∈ℝd×dR ^d× d only through RU=(RQ)SRU=(RQ)S, it is determined entirely by the d×rd× r matrix RQRQ. We show that optimization over the full orthogonal group is therefore exactly equivalent to optimization over the much smaller Stiefel manifold St(d,r)St(d,r). Hence ThinQuant, with a few hundred selected activations, replaces a dense d×d× d rotation by a d×rd× r variable with r≪dr d, reducing the effective optimization dimension from quadratic to linear in d, with no additional approximation beyond the initial activation selection. We summarize our contributions as follows: • Reducing the calibration data. We develop a geometric activation-selection procedure to select activations that are frequently exposed as extreme along random Gaussian directions. This reduces the calibration set required for rotation learning by several orders of magnitude, lowering activation storage and the cost of objective evaluations. We support this selection approach with a dimension-free bound on the expected discrepancy between the full and reduced calibration objectives under Haar-random rotations (Theorem 1). • Reducing the optimization dimension. We show that this reduced calibration set leads to an exact reduction of the rotation-learning problem. For selected activations spanning an r-dimensional subspace with r<dr<d, optimizing a d×d× d rotation is equivalent to optimizing a d×rd× r matrix on the Stiefel manifold, with no additional approximation beyond activation selection (Theorem 2). We exploit this equivalence through an ADMM algorithm with thin matrix factorizations, reducing optimization memory and per-iteration cost without requiring forward or backward passes through the model during optimization. • Efficient rotation learning at scale. Combining these reductions, ThinQuant achieves highly competitive perplexity and downstream accuracy on Llama-2 and Llama-3 under W4A4KV4 quantization at substantially lower calibration cost. On Llama-3-70B, ThinQuant reduces total rotation-calibration time from 111111 to approximately 1212 minutes relative to DartQuant, while improving WikiText-2 perplexity from 7.557.55 to 5.635.63 (Table 2). Moreover, to demonstrate the extreme scalability of our method, we employ ThinQuant on Llama-3.1-405B. It runs in just over 2 hours on a single H200 GPU, and achieves WikiText-2 perplexity at W4A4 (2.97 vs 3.48 of GPTAQ+QuaRoT), where other optimization-based rotation learning methods fail to scale under similar hardware constraints. 2 Related work Post-training quantization. Weight-only quantization reduces parameter storage and memory traffic while retaining higher-precision activations. GPTQ (Frantar et al., 2023) uses approximate second-order information for weight quantization. GPTAQ (Li et al., 2025) extends GPTQ with a different calibration scheme that uses extra closed-form weight updates to compensate for discrepancies between full-precision and quantized layer inputs. AWQ uses activation statistics to protect salient weight channels (Lin et al., 2024), and ADMM-Q uses an ADMM-based method for weight-only quantization (Lucas et al., 2026). Beyond weight-only methods, joint weight and activation quantization additionally enables low-precision matrix multiplications, but must accommodate large outlier values in activation distributions (Dettmers et al., 2022; Xiao et al., 2023). SmoothQuant addresses this difficulty through function-preserving rescaling, transferring quantization difficulty from activations to weights (Xiao et al., 2023). OmniQuant further learns clipping parameters and equivalent transformations through block-wise reconstruction (Shao et al., 2024). Such equivalent transformations change the representation presented to the quantizer without changing the full-precision network output. Our method also falls within this post-training setting, targeting joint low-bit weight and activation quantization. Specifically, we focus on function-preserving rotations, and discuss rotation-based methods below. Rotation-based quantization. Rotation-based methods provide another way to obtain representations that are more amenable to low-bit quantization. QuIP uses random orthogonal preprocessing to improve weight and Hessian incoherence for weight-only quantization (Chee et al., 2023). QuaRot applies fixed randomized Hadamard rotations to support weight, activation, and KV-cache quantization (Ashkboos et al., 2024), and is generally the fastest method since it is calibration-free. SpinQuant improves substantially on the quality of QuaRot by instead learning rotations by optimizing an end-to-end quantized-network loss, using Cayley updates to maintain orthogonality (Liu et al., 2025), but can be slow since it requires full backward passes through the network. DartQuant reduces this cost by replacing end-to-end optimization with a direct activation-distribution objective, combining an exponential loss with a QR decomposition at every step (Shao et al., 2025). Our work builds on these function-preserving rotation frameworks and focuses on reducing the data and optimization dimensions required for calibration of the rotation. 3 Method Let Xℓ∈ℝT×dX_ ^T× d contain the calibration activations associated with layer ℓ , with row t written as xℓ,tx_ ,t T. We adopt the function-preserving rotation placements of prior work (Ashkboos et al., 2024; Liu et al., 2025), which include a global change of basis R1R_1 and a local change of basis R2,ℓR_2, for each layer. The rotation R1∈SO(d)R_1 (d) defines a single basis shared across all layers. Namely at every layer XℓX_ is represented as X~ℓ=XℓR1 X_ =X_ R_1 T for ℓ=1,…,L =1,…,L, with the change of basis inverted only at the output as XL=X~LR1X_L= X_LR_1. In contrast, each R2,ℓR_2, acts only locally within layer ℓ , with X~ℓ=XℓR2,ℓ X_ =X_ R_2, T and Xℓ=X~ℓR2,ℓX_ = X_ R_2, . In both cases, the rotations can be absorbed into the surrounding weight matrices, including the final inverse of R1R_1 into the output projection, preserving the full-precision network function without introducing additional online matrix multiplications. Learning these rotations nevertheless leads to large-scale calibration problems. For R1R_1, a single rotation must be calibrated jointly against activations from every layer, so the amount of calibration data grows with the entire network. For R2,ℓR_2, , the calibration data are local to a single layer, but the rotation itself is still high-dimensional and a separate optimization problem must be solved for every ℓ=1,…,L =1,…,L. ThinQuant addresses these costs with two complementary reductions. First, for learning both R1R_1 and R2,ℓR_2, , we select a small set of informative activations (Section 3.1), reducing both calibration storage and the amount of data processed at every optimization step. Second, we exploit the low-dimensional span of these activations to reduce the size of the optimization problem (Section 3.2). Instead of updating a dense d×d× d rotation, we optimize a d×rd× r matrix, where r≪dr d is the rank of the selected activation matrix. This reformulation is exact for the selected set and replaces expensive square-matrix orthogonality updates with much cheaper thin matrix factorizations. Our ADMM solver (Section 3.3) operates on this smaller representation to reduce optimization memory and per-iteration cost. 3.1 Selecting a reduced calibration set Rotation calibration. We now turn to the rotation calibration problem. We remark that collecting and storing the full activation matrices can be prohibitively expensive for large models. For example, DartQuant (Shao et al., 2025) explicitly stores a sample of massive calibration activations, requiring approximately 960960 GiB of disk storage for LLaMA-3 70B in their default setting used in our experiments (Table 4 in the appendix). Our goal is therefore to select a small subset of activations that approximately preserves the rotation objective (1). For R1R_1, we consider activations from all layers, 1=⋃ℓ=1Lxℓ,t(1):1≤t≤TA_1= _ =1^L\x_ ,t^(1):1≤ t≤ T\, whereas for R2,ℓR_2, we use only the corresponding layer, 2,ℓ=xℓ,t(2):1≤t≤TA_2, =\x_ ,t^(2):1≤ t≤ T\. Apart from this choice of calibration set, the method is applied identically to R1R_1 and R2,ℓR_2, . We therefore write =x1,…,xN⊂ℝdA=\x_1,…,x_N\ ^d for either collection, X=[x1,…,xN]∈ℝd×NX=[x_1,…,x_N] ^d× N for its activation matrix, and R∈SO(d)R (d) for the corresponding rotation. Thus N=LTN=LT for R1R_1 and N=TN=T for each R2,ℓR_2, . For either rotation, a natural worst-case objective including all calibration data is: minR∈SO(d)F(R),F(R)=‖RX‖∞=maxx∈‖Rx‖∞, _R (d)F(R), F(R)=\|RX\|_∞= _x \|Rx\|_∞, (1) where ∥⋅∥∞\|·\|_∞ denotes the largest absolute entry. Geometrically, the goal is to find an orthonormal basis in which no calibration activation develops a large coordinate. Uniform quantization is governed by the dynamic range of the activation, so the largest coordinate directly leads to a coarser quantization resolution. The ℓ∞ _∞ norm therefore provides a natural worst-case surrogate for suppressing activation outliers and reducing quantization error (Wei et al., 2023; Xiao et al., 2023; Chee et al., 2023; Ashkboos et al., 2024). Our goal is to solve (1) without requiring the full calibration data. Specifically, given an activation budget B≪NB N, we want a subset ℐ⊆[N]I [N] with |ℐ|=B|I|=B such that Fℐ(R)=maxi∈ℐ‖Rxi‖∞≈F(R)F_I(R)= _i \|Rx_i\|_∞≈ F(R). The subset ℐI is selected before rotation optimization, so it must represent this surrogate as the basis R changes, rather than only in the original coordinate system. A geometric view of calibration data selection. We first give some intuition on how the calibration data A in (1) might naturally be reduced. First, notice that the worst-case objective in (1) is determined, for any fixed rotation R, by activations attaining maxx∈|rkx| _x |r_k Tx| for one of the rows rkr_k T of R. However, since the rows of R change during optimization (and the optimal R is not known in advance), the maximizer can also change, so activations that are not maximizers in the original basis may become so after rotation. Thus, rather than retaining only activations that are large in the original basis, we want a small subset that retains the activations that may become maximizers during rotation optimization. Classical convex geometry provides a precise language for describing these extreme directions (Ziegler, 1995). Define the symmetric convex hull K=conv(∪(−))K=conv (A∪(-A) ) and its support function hK(v)=maxz∈Kvz=maxx∈|vx|h_K(v)= _z∈ Kv Tz= _x |v Tx|, defined for every v∈ℝdv ^d. Since A is finite and K is a polytope, maximizing the linear functional x↦vx v Tx over K attains its optimum at an extreme point of K for every v (Bertsimas and Tsitsiklis, 1997). The candidate maximizers (in x) are therefore confined to ext(K)⊆∪(−) (K) ∪(-A), where ext(⋅) (·) is the set of extreme points of the polytope. This holds independently of v, so a single finite set of signed activations suffices to evaluate hKh_K in every direction. Writing the rows of a rotation R as r1,…,rdr_1 T,…,r_d T, we can thus write (1) equivalently as: minR∈SO(d)F(R)=max1≤k≤dhK(rk), _R (d)F(R)= _1≤ k≤ dh_K(r_k), (2) The key observation here is that changing R changes the viewing direction, but not the underlying polytope K. Figure 2 illustrates this. The directions r1,r2,r3r_1,r_2,r_3 expose different activations x1,x2,x3x_1,x_2,x_3, respectively, but each support value can only be attained at an extreme point of the same convex hull. Thus, the relevant boundary point can change as the rotation changes, but activations on the interior of K can never obtain an extreme point. This explains why the full calibration set can be redundant. An activation which is not an extreme point cannot become extreme along any direction. Any subset ℐ⊆A_I satisfying conv(ℐ∪(−ℐ))=Kconv (A_I∪(-A_I) )=K preserves F(R)F(R) for every rotation R. This also connects rotation calibration to the literature on geometric coresets and, in particular, ε -kernels, where small subsets are constructed to approximately preserve the width of the convex hull in all directions (Agarwal et al., 2004). r1r_1r2r_2r3r_3x1x_1x2x_2x3x_3K=conv(∪(−))K=conv\! (A∪(-A) ) Figure 2: For any direction r, the support value hK(r)=maxx∈|rx|h_K(r)= _x |r Tx| is attained at an extreme point of K. Hence, as the rotation changes, only the exposed boundary activations can determine F(R)F(R) and interior activations do not determine the objective. Selection algorithm. Although the relevant calibration information is concentrated on extreme points, recovering all extreme points is unattractive in our setting. For general polyhedra, deciding if a partial vertex list omits another vertex is NP-complete (Khachiyan et al., 2008). Even for a finite number of points, exact identification of all extreme points would require solving a sequence of high-dimensional convex-hull membership problems. Similarly, ε -kernels, which approximate the directional width of a point set simultaneously over all directions, have worst-case size Θ(ε−(d−1)/2) \! ( ^-(d-1)/2 ) for preserving directional widths within a relative (1−ε)(1- ) factor (Agarwal et al., 2004). With hundreds of thousands of activation vectors in dimensions of several thousand, these are considerably more expensive than the lightweight selection procedure we want. A natural and more lightweight approach is to query the support function hK(v)h_K(v) from (2) along random directions. This idea has precedent in randomized extreme-point detection. In particular, Damle and Sun (2017) generate Gaussian linear functionals, record the points attaining their extrema, and use selection frequency to identify geometrically prominent extreme points. We apply the same random-projection idea to the symmetric hull K=conv(∪(−))K=conv(A∪(-A)). We give the procedure in Algorithm 1. We compare the results from this procedure with ℓ∞ _∞-based selection in Table 3 in the appendix, where we retain the B activations with the largest ℓ∞ _∞ norm either in the original basis or after an initial Hadamard rotation. Algorithm 1 Activation selection 1: Activations =x1,…,xNA=\x_1,…,x_N\, number of random directions M, total activation budget B 2: g1,…,gM∼iid(0,Id)g_1,…,g_M iid N(0,I_d) // Gaussian directions 3: ij←argmax1≤i≤N|gjxi|,j=1,…,Mi_j← *arg\,max_1≤ i≤ N|g_j Tx_i|, j=1,…,M // Extreme point? 4: ei←∑j=1Mij=i,i=1,…,Ne_i← _j=1^M1\i_j=i\, i=1,…,N // How often extreme? 5: ℐ←TopB(e)I _B(e) // Global ranking 6: return xi:i∈ℐ\x_i:i \ Interpretation of the algorithm. We now give an interpretation of Algorithm 1 in terms of the calibration objective. Notice that (2) describes the calibration objective under the best possible change of basis. Rather than requiring a small subset to preserve this objective for every possible rotation (which is clearly intractable), consider how well it is preserved on average over R∼Haar(SO(d))R (SO(d)), i.e., for a randomly sampled Haar rotation: R∼Haar(SO(d))[F(R)]=R∼Haar(SO(d))[max1≤k≤dhK(rk)].E_R (SO(d)) [F(R) ]=E_R (SO(d)) [ _1≤ k≤ dh_K(r_k) ]. (3) Algorithm 1 can be interpreted as selecting a small subset of the activations with which to approximate the objective in (3). For the selected subset ℐ⊆[N]I [N], let Kℐ=conv±xi:i∈ℐK_I=conv\± x_i:i \ and write Fℐ(R)=max1≤k≤dhKℐ(rk)F_I(R)= _1≤ k≤ dh_K_I(r_k) as the corresponding reduced calibration objective. Since Kℐ⊆K_I K, we have Fℐ(R)≤F(R)F_I(R)≤ F(R) for every rotation R. Thus the subset objective is R[Fℐ(R)]E_R[F_I(R)], and we can measure the quality of the selected calibration set via R[F(R)−Fℐ(R)]E_R[F(R)-F_I(R)], which measures how well the selected activations represent the full calibration objective in (1) under a random change of basis. Theorem 1 gives a bound on this discrepancy. Specifically, let q(ℐ)q(I) denote the probability that, along a random Gaussian direction, the maximum support value is attained by an activation that is not contained in the selected set ℐI. The theorem shows that this probability directly controls the expected difference between the full and reduced calibration objectives. Thus, when Algorithm 1 selects activations that attain the directional maximum for most random Gaussian directions, the reduced objective closely approximates (1) on average over random changes of basis, without requiring recovery of the entire convex hull. Theorem 1 (Expected Approximation Guarantee). Let =x1,…,xN⊂ℝdA=\x_1,…,x_N\ ^d, with d≥2d≥ 2, and identify activations that coincide up to sign. For any fixed nonempty subset ℐ⊆[N]I [N] returned by Algorithm 1, define: q(ℐ)=Prg∼(0,Id)[argmaxi∈[N]|gxi|∉ℐ],q(I)= _g (0,I_d) [ *arg\,max_i∈[N]|g Tx_i| ], Then, for R∼Haar(SO(d)):R (SO(d)): R[F(R)−Fℐ(R)]≤(maxi∈[N]‖xi‖2)3q(ℐ).E_R\! [F(R)-F_I(R) ]≤ ( _i∈[N]\|x_i\|_2 ) 3q(I). (4) The proof is given in Appendix A. The main idea is to relate the Gaussian directions used by Algorithm 1 to the coordinate directions generated by a random rotation. If R∼Haar(SO(d))R (SO(d)), then each row direction rkr_k is marginally uniform on the unit sphere. Likewise, for g∼(0,Id)g (0,I_d), the normalized vector g/‖g‖2g/\|g\|_2 is uniform on the unit sphere. Thus, projecting an activation onto a normalized Gaussian direction is distributionally equivalent to an individual coordinate after a Haar rotation. Although the rows of R are dependent through orthogonality, the proof bounds the objective gap by a sum of coordinate errors, whose expectation depends only on these marginal distributions. Figure 3: Empirical estimate q^M(ℐ) q_M(I) of the miss probability q(ℐ)q(I) in Algorithm 1 as a function of the activation budget B. An appealing feature of Theorem 1 is that the quantity controlling the bound is directly estimated by Algorithm 1. Indeed, for the selected set ℐI, the counts eie_i gives the empirical estimate: q^M(ℐ)=1−1M∑i∈ℐei, q_M(I)=1- 1M _i e_i, which is precisely the fraction of sampled Gaussian directions whose maximizing activation is not contained in ℐI. As M→∞M→∞, these empirical frequencies converge to their population probabilities, and hence q^M(ℐ)→q(ℐ) q_M(I)→ q(I). We report q^M(ℐ) q_M(I) for both Llama-3.1-405B and Llama-3-8B at M=8192M=8192. We see that the counts are highly concentrated. Selecting a single activation already reduces the empirical miss probability q^M(ℐ) q_M(I) to 0.780.78 and 0.120.12, respectively, and it reaches zero on both models at B=1280B=1280 activations. For Llama-3-8B and Llama-3.1-405B, this is already far less than 0.1%0.1\% of A. Finally, given the selected index set ℐI returned by Algorithm 1, we collect the selected activations as columns of the reduced calibration matrix: U=X:,ℐ=[xi]i∈ℐ∈ℝd×B.U=X_:,I=[x_i]_i ^d× B. (5) We replace the calibration matrix X with the selected matrix U=[xi]i∈ℐ∈ℝd×BU=[x_i]_i ^d× B, where |ℐ|=B|I|=B: minR∈SO(d)FU(R),FU(R)=‖RU‖∞=maxi∈ℐ‖Rxi‖∞ _R (d)F_U(R), F_U(R)=\|RU\|_∞= _i \|Rx_i\|_∞ (6) The calibration procedure thus retains activations that determine the maximum across many directions, while reducing the number of calibration vectors from N to B. This also restricts the span of the retained activations to dimension at most B, which we discuss next. 3.2 An exact reduction in the rotation optimization problem Once U is fixed, the objective in (6) evaluates R only on the selected activations, all of which lie in span(U)span(U). Any two rotations that agree on this subspace therefore produce the same matrix RURU and attain the same loss, regardless of how they act on its orthogonal complement. Thus, rather than tracking the action of R on all of ℝdR^d, it suffices to track its action on a basis for span(U)span(U). To make this clear, take an exact QR decomposition U=QUSU=Q_US, where r=rank(U)≤Br=rank(U)≤ B. Here, QU∈ℝd×rQ_U ^d× r has orthonormal columns spanning span(U)span(U), and S∈ℝr×BS ^r× B contains the coordinates of the selected activations in this basis. The matrix V=RQUV=RQ_U records the images of these basis vectors under R, and hence determines every rotated activation through RU=VSRU=VS. Moreover, orthogonality of R implies VV=QURRQU=IrV TV=Q_U TR TRQ_U=I_r, so V∈St(d,r):=V∈ℝd×r:VV=IrV (d,r):=\V ^d× r:V TV=I_r\. The following theorem proves the converse when r<dr<d, showing that every V∈St(d,r)V (d,r) can be written as RQURQ_U for some R∈SO(d)R (d), so optimizing over V is equivalent to optimizing over the full rotation R. Theorem 2 (Exact selected-subspace reduction). Let U=QUS∈ℝd×BU=Q_US ^d× B have rank 1≤r<d1≤ r<d, where QU∈ℝd×rQ_U ^d× r, S∈ℝr×BS ^r× B, and QUQU=IrQ_U TQ_U=I_r. Then RU:R∈SO(d)=VS:V∈St(d,r)\RU:R (d)\=\VS:V (d,r)\. Then for every continuous function ℓ:ℝd×B→ℝ :R^d× B : minR∈SO(d)ℓ(RU)=minV∈St(d,r)ℓ(VS). _R (d) (RU)= _V (d,r) (VS). (7) Every feasible reduced solution lifts to a full rotation with identical objective value. Proof. The forward inclusion follows from V=RQUV=RQ_U. Conversely, extend QUQ_U and V to orthogonal bases [QUQ⟂][Q_U\ Q_ ] and [VV⟂][V\ V_ ] and set R=[VV⟂][QUQ⟂]R=[V\ V_ ][Q_U\ Q_ ] T. Then RQU=VRQ_U=V. Because r<dr<d, a sign change in one complement column can enforce detR=1 R=1 without changing RQURQ_U. Hence RU=VSRU=VS. Since SO(d)SO(d) and St(d,r)St(d,r) are compact and ℓ is continuous, both minima are attained; equality of the feasible image sets therefore implies (7). ∎ Theorem 2 applies directly to the entrywise ℓ∞ _∞ objective in (6), since ℓ(RU)=‖RU‖∞ (RU)=\|RU\|_∞ is continuous. It also applies without modification to any continuous surrogate objective. Calibration loss. In general, directly optimizing the ℓ∞ _∞ norm is not attractive, since it requires taking subgradients and can make convergence slow. For optimization, we thus use a fourth-power penalty as a smooth surrogate for the entrywise ℓ∞ _∞ objective. Since ‖RU‖∞4≤‖RU‖44\|RU\|_∞^4≤\|RU\|_4^4, this provides a smooth upper bound on the fourth power of the maximum entry. This choice is motivated by related kurtosis-based approaches to activation outlier suppression (Akhondzadeh et al., 2025). Moreover, we center each rotated activation within the loss because a common shift of all coordinates changes their absolute magnitudes but leaves their range unchanged. This is relevant to the asymmetric activation quantization used in our experiments, where the quantization grid need not be centered at zero. We therefore penalize deviations from each rotated activation’s coordinate mean. Specifically, let Cd=Id−1dddC_d=I_d- 1d1_d1_d T, where d∈ℝd1_d ^d is the all-ones vector. We therefore replace the objective in (6) with a surrogate in (8). Since the centered loss is still continuous, Theorem 2 gives: minR∈SO(d)‖CdRU‖44=minV∈St(d,r)‖CdVS‖44. _R (d)\|C_dRU\|_4^4= _V (d,r)\|C_dVS\|_4^4. (8) 3.3 Fast optimization and rotation recovery The reduced problem in (8) replaces the full rotation with a thin matrix V, but the loss remains coupled to the orthogonality constraint through CdVSC_dVS. We use the Alternating Direction Method of Multipliers (ADMM) (Boyd et al., 2011) to separate these components. Specifically, introducing W=VSW=VS and Z=CdWZ=C_dW, we can write the problem as: minV,W,Z‖Z‖44subject toW=VS,Z=CdW,VV=Ir. _V,W,Z\ \|Z\|_4^4 to W=VS, Z=C_dW, V TV=I_r. (9) ADMM alternates between updates of Z, W, and V – each of which can be computed via closed-form updates (see Appendix B) – followed by dual updates that track violations of the two linear constraints. The Z-update handles the fourth-power penalty through independent proximal updates by solving a cubic polynomial per entry. The W-update balances agreement with VSVS and the centered variable Z through a closed-form least-squares step. Since CdC_d is a projector, this step requires only column means rather than a dense matrix inverse. Finally, the V-update enforces orthogonality by solving an orthogonal Procrustes problem (Golub and Van Loan, 2013) using a thin d×rd× r Singular Value Decomposition (SVD). This update remains exact even when S is not orthogonal, because ‖VS‖F2=‖S‖F2\|VS\|_F^2=\|S\|_F^2 on the Stiefel manifold. All iterations operate on the reduced calibration matrices, so no transformer forward or backward pass is needed during optimization. Cost. Factoring U costs O(dB2)O(dB^2) once for B≤dB≤ d. Each iteration costs O(drB+dr2)O(drB+dr^2) and stores O(dB+dr+rB)O(dB+dr+rB) numbers. When r=B≪dr=B d, these become O(dB2)O(dB^2) work and O(dB)O(dB) storage, replacing a square orthogonality step with O(d3)O(d^3) work and O(d2)O(d^2) rotation state. After optimization, we recover R using the completion in the proof of Theorem 2, with a random Hadamard rotation for the orthogonal complement. Completion leaves RU=VSRU=VS unchanged, although different completions can act differently on unseen activations outside span(U)span(U). 4 Experiments Quantization performance Rotation calibration cost Model Method Wiki-2 ↓ C4 ↓ 0-shot ↑ Runtime ↓ Peak GPU ↓ Dense FP16 6.14 9.44 65.91 ThinQuant 7.91 12.83 59.70 1:56 18.5 DartQuant 8.22 13.37 59.00 19:03 37.1 LLaMA-3 8B SpinQuant 7.67 12.69 59.62 53:35 20.7 Dense FP16 2.86 7.17 72.70 ThinQuant 5.63 10.34 65.83 11:57 31.9 DartQuant 7.55 15.69 58.64 111:00 134.9 LLaMA-3 70B SpinQuant 5.05 9.71 67.87 16.7h∗ 239.0∗ Dense FP16 5.47 7.26 61.16 ThinQuant 6.18 8.41 57.28 1:43 18.2 DartQuant 6.33 8.63 56.39 20:25 75.1 LLaMA-2 7B SpinQuant 6.12 8.49 56.98 49:33 17.8 Dense FP16 4.88 6.73 64.30 ThinQuant 5.47 7.67 61.10 2:51 21.9 DartQuant 5.51 7.71 61.35 31:42 51.1 LLaMA-2 13B SpinQuant 5.35 7.61 61.73 113:32 31.5 Dense FP16 3.32 5.71 69.52 ThinQuant 3.91 6.24 66.92 11:42 33.2 DartQuant 3.91 6.22 67.18 99:57 73.1 LLaMA-2 70B SpinQuant 3.85 6.19 67.62 17.2h∗ 224.3∗ Table 1: W4A4KV4 quantization performance and rotation calibration cost. The 0-shot column averages nine downstream tasks. Dense FP16 denotes the unquantized reference model. Calibration-cost columns apply to the quantized methods. Runtime measures total rotation-calibration time for ThinQuant, DartQuant, and SpinQuant. GPU denotes peak GPU process memory during calibration, in GiB. Times are in m:s unless marked in hours. ∗The 70B SpinQuant runs used 4 GPUs, so we report total GPU hours and total GPU memory summed across GPUs. Best quantized results are bold. Method LLaMA-3 70B LLaMA-3.1 405B Dense FP16 2.86 1.44 GPTQ ++ QuaRot 6.04 5.82 GPTAQ ++ QuaRot 5.81 3.48 GPTQ ++ ThinQuant 5.63 3.10 GPTAQ ++ ThinQuant 5.46 2.97 Table 2: Scalability to extremely large architectures. WikiText-2 perplexity (↓ ) on LLaMA-3-70B and LLaMA-3.1-405B. The LLaMA-3-70B column is W4A4KV4 with symmetric weights and asymmetric activations. The LLaMA-3.1-405B column is W4A4. For the 405B comparison, ThinQuant uses the same asymmetric quantization as GPTAQ Li et al. (2025) for both weights and activations. The best quantized result in each column is bold. ThinQuant takes 2 hours 7 minutes for full rotation calibration on LLaMA-3.1-405B. We additionally run W4A4KV4 under GPTQ + ThinQuant for LLaMA-3.1-405B, and report the result in Figure 1. 4.1 Experimental setup Models and baselines. We evaluate ThinQuant on LLaMA-2 (7B, 13B, and 70B), LLaMA-3 (8B and 70B) (Grattafiori et al., 2024). All models are evaluated under W4A4(KV4) quantization, with weights, activations, and (when applicable) the KV cache quantized to 4 bits. We use symmetric quantization for weights and asymmetric quantization for activations and the KV cache. As in SpinQuant and DartQuant, we use fixed random Hadamard rotations for R3R_3 and R4R_4 (which are KV cache specific); these rotations are not optimized and are enabled for every ThinQuant, DartQuant, and SpinQuant evaluation. Unless otherwise stated, weight quantization is performed using GPTQ throughout all experiments, and is applied after rotation calibration so that GPTQ operates on the rotated weight matrices. We compare against DartQuant and SpinQuant under the same quantization settings. All experiments for all methods are conducted on a single NVIDIA H200 GPU, with the exception of SpinQuant where we use 4 H200 GPUs and report total GPU hours. Evaluation. We measure language-modeling performance using perplexity (↓ ) on WikiText-2, C4, and Penn Treebank (PTB). For downstream evaluation, we report the average zero-shot accuracy (↑ ) across nine tasks using the LM Evaluation Harness (Gao et al., 2023): PIQA (Bisk et al., 2020), ARC-Easy and ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021), RTE (Wang et al., 2018), OpenBookQA (Mihaylov et al., 2018), BoolQ (Clark et al., 2019), and Social IQa (Sap et al., 2019). Calibration. Unless otherwise stated, ThinQuant uses M=8192M=8192 independent Gaussian directions in Algorithm 1. For the shared residual rotation R1R_1, we set the total activation budget to B=2L+1B=2L+1, where L denotes the number of transformer layers. The rotations are applied at the down-projection and output-projection layers, so this budget is sufficient to admit one activation vector from each projection in every layer, though we select the top B globally as in Algorithm 1. For R2R_2, we set the budget to H vectors, where H is the number of transformer heads. The selected activations are used to construct the reduced optimization problem, which we solve using the ADMM procedure in Section 3.3. To assess computational efficiency, we report total rotation-calibration run-time and peak GPU memory in Table 2, with a more detailed breakdown in Table 4. Memory is reported in GiB. Results. Table 2 shows that ThinQuant improves or closely matches the performance of DartQuant across all reported evaluation metrics on LLaMA-2 and LLaMA-3 with substantially less calibration time. On LLaMA-3-70B, ThinQuant reduces WikiText-2 perplexity from 7.557.55 to 5.635.63 and C4 perplexity from 15.6915.69 to 10.3410.34, while reducing total rotation-calibration time from 111111 minutes to under 1212 minutes. Moreover, Table 4 in the appendix shows that ThinQuant requires several orders of magnitude less disk space than DartQuant. DartQuant requires storing large activation tensors, whereas ThinQuant selects and retains only a tiny subset of activations for rotation calibration. For LLaMA-3-70B, this reduces the required disk space from 960960 GiB to 0.510.51 GiB. Finally, Table 2 shows that ThinQuant scales to LLaMA-3.1-405B, completing rotation calibration in just over 2 hours on a single H200 GPU. Under W4A4 quantization, replacing QuaRot’s fixed Hadamard rotations with ThinQuant reduces WikiText-2 perplexity from 5.825.82 to 3.103.10 with GPTQ and from 3.483.48 to 2.972.97 with GPTAQ, using the same asymmetric quantization setting as Li et al. (2025). We provide additional experiments on W4A4 including a comparison to OmniQuant in Table 5. 5 Conclusions and Limitations ThinQuant combines geometric activation selection with an exact subspace reduction to make rotation learning substantially cheaper. ThinQuant achieves competitive low-bit quantization performance compared to state-of-the-art methods with much lower calibration runtime, GPU memory, and disk storage, making rotation calibration of LLaMA-3.1-405B possible on a single H200 GPU. Limitations. We remark that we optimize rotations separately prior to using weight quantization algorithms like GPTQ, though the two could in principle be solved jointly. An interesting research direction is to design dedicated methods that can efficiently optimize rotation and weight quantization simultaneously. Moreover, some recent works have shown that learning rotation and diagonal scaling transforms jointly can improve quantization, though this can require expensive calibration (Hu et al., 2025). We leave scalable and efficient rotation and scaling optimization to future work. References Agarwal et al. (2004) P. K. Agarwal, S. Har-Peled, and K. R. Varadarajan Approximating extent measures of points. Journal of the ACM 51 (4), p. 606–635. External Links: Document Cited by: §3.1, §3.1. Akhondzadeh et al. (2025) M. S. Akhondzadeh, A. Bojchevski, E. Eleftheriou, and M. Dazzi KurTail: kurtosis-based LLM quantization. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 17404–17419. External Links: Document, Link Cited by: §3.2. Ashkboos et al. (2024) S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman QuaRot: outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §2, §3.1, §3. Bertsimas and Tsitsiklis (1997) D. Bertsimas and J. N. Tsitsiklis Introduction to linear optimization. Athena Scientific, Belmont, MA. Cited by: §3.1. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 7432–7439. External Links: Document, Link Cited by: §4.1. Boyd et al. (2011) S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3 (1), p. 1–122. External Links: Link Cited by: Appendix B, §3.3. Chee et al. (2023) J. Chee, Y. Cai, V. Kuleshov, and C. De Sa QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §2, §3.1. Clark et al. (2019) C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova BoolQ: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 2924–2936. External Links: Document, Link Cited by: §4.1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. External Links: Link Cited by: §4.1. Cordero-Erausquin et al. (2004) D. Cordero-Erausquin, M. Fradelizi, and B. Maurey The (B) conjecture for the Gaussian measure of dilates of symmetric convex sets and related problems. Journal of Functional Analysis 214 (2), p. 410–427. External Links: Document Cited by: Appendix A. Damle and Sun (2017) A. Damle and Y. Sun A geometric approach to archetypal analysis and nonnegative matrix factorization. Technometrics 59 (3), p. 361–370. External Links: Document Cited by: §3.1. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, Vol. 35, p. 30318–30332. External Links: Document, Link Cited by: §1, §1, §2. Frantar et al. (2023) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Gao et al. (2023) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou A framework for few-shot language model evaluation. Zenodo. External Links: Document, Link Cited by: §4.1. Golub and Van Loan (2013) G. H. Golub and C. F. Van Loan Matrix computations. 4 edition, Johns Hopkins University Press. External Links: ISBN 9781421407944, Document, Link Cited by: §3.3. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link Cited by: §4.1. Hu et al. (2025) X. Hu, Y. Cheng, D. Yang, Z. Chen, Z. Xu, J. Yu, C. Xu, Z. Yuan, Z. Jiang, and S. Zhou OSTQuant: refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In International Conference on Learning Representations, External Links: Link Cited by: §5. Khachiyan et al. (2008) L. Khachiyan, E. Boros, K. Borys, K. Elbassioni, and V. Gurvich Generating all vertices of a polyhedron is hard. Discrete & Computational Geometry 39, p. 174–190. External Links: Document Cited by: §3.1. Li et al. (2025) Y. Li, R. Yin, D. Lee, S. Xiao, and P. Panda GPTAQ: efficient finetuning-free quantization for asymmetric calibration. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 36690–36706. External Links: Link Cited by: §1, §2, §4.1, Table 2. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems, External Links: Link Cited by: §2. Liu et al. (2025) Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations, External Links: Link Cited by: §1, §1, §2, §3. Lucas et al. (2026) R. Lucas, M. Makni, X. Meng, A. Deng, and R. Mazumder ADMM-q: an improved hessian-based weight quantizer for post-training quantization of large language models. External Links: 2605.11222, Link Cited by: §1, §2. Meckes (2019) E. S. Meckes The random matrix theory of the classical compact groups. Cambridge Tracts in Mathematics, Cambridge University Press. Cited by: Appendix A. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2381–2391. External Links: Document, Link Cited by: §4.1. Sakaguchi et al. (2021) K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM 64 (9), p. 99–106. External Links: Document, Link Cited by: §4.1. Sap et al. (2019) M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi Social IQa: commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 4463–4473. External Links: Document, Link Cited by: §4.1. Shao et al. (2024) W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo OmniQuant: omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations, Cited by: §2. Shao et al. (2025) Y. Shao, Y. Chen, P. Wang, J. Yu, J. Lin, Y. Yao, Z. Wei, and J. Cheng DartQuant: efficient rotational distribution calibration for LLM quantization. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §1, §2, §3.1. Vershynin (2018) R. Vershynin High-dimensional probability: an introduction with applications in data science. Cambridge University Press. Cited by: Appendix A. Wang et al. (2018) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, p. 353–355. External Links: Document, Link Cited by: §4.1. Wei et al. (2023) X. Wei, Y. Zhang, Y. Li, X. Zhang, R. Gong, J. Guo, and X. Liu Outlier Suppression+: accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 1648–1665. External Links: Document, Link Cited by: §3.1. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, External Links: Link Cited by: §1, §2, §3.1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4791–4800. External Links: Document, Link Cited by: §4.1. Ziegler (1995) G. M. Ziegler Lectures on polytopes. Graduate Texts in Mathematics, Vol. 152, Springer, New York. Cited by: §3.1. Appendix A Proof of Theorem 1 Proof. Throughout the proof, the activations and the subset ℐI are fixed. Write q=q(ℐ)q=q(I) and H=maxi∈[N]‖xi‖2H= _i∈[N]\|x_i\|_2. Recall that: K=conv±xi:i∈[N],Kℐ=conv±xi:i∈ℐ.K=conv\± x_i:i∈[N]\, K_I=conv\± x_i:i \. For u∈ℝdu ^d, define: f(u)=hK(u)−hKℐ(u)=maxi∈[N]|uxi|−maxi∈ℐ|uxi|f(u)=h_K(u)-h_K_I(u)= _i∈[N]|u Tx_i|- _i |u Tx_i| Since ℐ⊆[N]I [N], we have f(u)≥0f(u)≥ 0. The absolute values also imply that f(−u)=f(u)f(-u)=f(u) and f(tu)=tf(u)f(tu)=tf(u) for every t≥0t≥ 0. We also have 0≤f(u)≤hK(u)≤H‖u‖20≤ f(u)≤ h_K(u)≤ H\|u\|_2, so f(g)f(g) is square-integrable for g∼(0,Id)g (0,I_d). We first show that the expected gap in objectives is bounded by: R[(F(R)−Fℐ(R))2]≤g[f(g)2]E_R\! [ (F(R)-F_I(R) )^2 ] _g[f(g)^2] (10) Write the rows of R as r1,…,rdr_1 T,…,r_d T. For every rotation, F(R) F(R) =max1≤k≤d[hKℐ(rk)+f(rk)] = _1≤ k≤ d [h_K_I(r_k)+f(r_k) ] ≤max1≤k≤dhKℐ(rk)+max1≤k≤df(rk) ≤ _1≤ k≤ dh_K_I(r_k)+ _1≤ k≤ df(r_k) =Fℐ(R)+max1≤k≤df(rk). =F_I(R)+ _1≤ k≤ df(r_k). Since Fℐ(R)≤F(R)F_I(R)≤ F(R) and f≥0f≥ 0, this gives: 0≤(F(R)−Fℐ(R))2≤max1≤k≤df(rk)2≤∑k=1df(rk)20≤ (F(R)-F_I(R) )^2≤ _1≤ k≤ df(r_k)^2≤ _k=1^df(r_k)^2 (11) Since d≥2d≥ 2, if R∼Haar(SO(d))R (SO(d)), each row rkr_k is marginally uniform on the unit sphere d−1S^d-1 [Meckes, 2019]. Let v be uniform on d−1S^d-1. Taking expectations in (11) therefore gives: R[(F(R)−Fℐ(R))2]≤dv[f(v)2].E_R\! [ (F(R)-F_I(R) )^2 ]≤ d\,E_v[f(v)^2]. (12) Now take g∼(0,Id)g (0,I_d) and write ρ=‖g‖2ρ=\|g\|_2 and v=g/‖g‖2v=g/\|g\|_2. Then v is uniform on the sphere and independent of ρ [Vershynin, 2018]. Moreover, [ρ2]=‖g‖22=∑j=1d[gj2]=dE[ρ^2]=E\|g\|_2^2= _j=1^dE[g_j^2]=d Since f(tu)=tf(u)f(tu)=tf(u) for every t≥0t≥ 0 and ρ and v are independent: g[f(g)2]=ρ,v[ρ2f(v)2]=[ρ2]v[f(v)2]=dv[f(v)2]E_g[f(g)^2]=E_ρ,v[ρ^2f(v)^2]=E[ρ^2]E_v[f(v)^2]=d\,E_v[f(v)^2] Combining this with (12) proves (10). It remains to bound the right-hand side of (10). We do this by establishing two variance bounds: (1−q)g[f(g)2]≤Varg(f(g))≤2H2q.(1-q)E_g[f(g)^2] _g(f(g))≤ 2H^2q. For q<1q<1, the lower bound will allow us to bound the second moment by Varg(f(g))/(1−q)Var_g(f(g))/(1-q). The upper bound will then give an estimate in terms of H and q. We first establish the lower bound by relating q to the probability that f(g)f(g) is nonzero. For distinct indices i,ji,j, a tie |gxi|=|gxj||g Tx_i|=|g Tx_j| can occur only if: g(xi−xj)=0org(xi+xj)=0.g T(x_i-x_j)=0 g T(x_i+x_j)=0. Both vectors xi−xjx_i-x_j and xi+xjx_i+x_j are nonzero by the identification of activations up to sign. Each equality therefore describes a proper hyperplane, which has probability zero under the standard Gaussian. There are only finitely many pairs of indices, so the maximizing index is unique almost surely. Whenever this unique maximizing index belongs to ℐI, the two support values coincide and f(g)=0f(g)=0. Whenever it does not belong to ℐI, every selected activation has a strictly smaller absolute projection, and hence f(g)>0f(g)>0. It follows that: Pr[f(g)>0]=Pr[i⋆(g)∉ℐ]=q(ℐ).Pr[f(g)>0]=Pr [i (g) ]=q(I). (13) The function f(g)f(g) is zero outside the event f(g)>0\f(g)>0\, which has probability q by (13). Cauchy–Schwarz therefore gives: (g[f(g)])2 (E_g[f(g)] )^2 =(g[f(g)f(g)>0])2 = (E_g\! [f(g)1_\f(g)>0\ ] )^2 ≤g[f(g)2]g[f(g)>02] _g[f(g)^2]\,E_g\! [1_\f(g)>0\^2 ] =qg[f(g)2] =q\,E_g[f(g)^2] (14) Using the definition of variance and (14), we obtain the required lower bound: Varg(f(g)) _g(f(g)) =g[f(g)2]−(g[f(g)])2 =E_g[f(g)^2]- (E_g[f(g)] )^2 ≥(1−q)g[f(g)2]. ≥(1-q)E_g[f(g)^2]. (15) We now establish the upper bound on the variance. We use the strengthened Gaussian Poincaré inequality for functions orthogonal to constants and linear functions [Cordero-Erausquin et al., 2004]. Evenness ensures orthogonality to linear functions, but does not imply a zero mean. To satisfy both conditions, we apply the inequality to the function: f~(u)=f(u)−g[f(g)] f(u)=f(u)-E_g[f(g)] By construction, g[f~(g)]=0E_g[ f(g)]=0. Moreover, subtracting a constant preserves evenness, so symmetry gives: g[gjf~(g)]=0,j=1,…,d.E_g[g_j f(g)]=0, j=1,…,d. Thus, centering removes the constant component, while evenness eliminates the linear component. Subtracting a constant also leaves the gradient unchanged, ∇f~=∇f∇ f=∇ f almost everywhere. The strengthened inequality therefore gives: Varg(f(g)) _g(f(g)) =g[f~(g)2] =E_g[ f(g)^2] (16) ≤12g‖∇f~(g)‖22=12g‖∇f(g)‖22. ≤ 12E_g\|∇ f(g)\|_2^2= 12E_g\|∇ f(g)\|_2^2. To bound the right-hand side of (16), we next control the gradient of f. We first show that f is 2H2H-Lipschitz, which implies ‖∇f(u)‖2≤2H\|∇ f(u)\|_2≤ 2H wherever f is differentiable. Since f=hK−hKℐf=h_K-h_K_I, it suffices to show that each support function is H-Lipschitz. Indeed, for any u,v∈ℝdu,v ^d, |hK(u)−hK(v)| |h_K(u)-h_K(v)| ≤maxi∈[N]||uxi|−|vxi|| ≤ _i∈[N] ||u Tx_i|-|v Tx_i| | ≤maxi∈[N]|(u−v)xi| ≤ _i∈[N]|(u-v) Tx_i| ≤H‖u−v‖2. ≤ H\|u-v\|_2. The same argument applies to hKℐh_K_I. By the triangle inequality, their difference therefore satisfies: |f(u)−f(v)|≤2H‖u−v‖2|f(u)-f(v)|≤ 2H\|u-v\|_2 (17) Both support functions are maxima of finitely many linear functions. Their difference f is therefore piecewise linear and differentiable almost everywhere. At any differentiability point u satisfying f(u)=0f(u)=0, the nonnegativity of f implies that u is a global minimum. Hence ∇f(u)=0∇ f(u)=0. At every other differentiability point, (17) implies ‖∇f(u)‖2≤2H\|∇ f(u)\|_2≤ 2H. Thus, ∥∇f(u)∥22≤4H2f(u)>0\|∇ f(u)\|_2^2≤ 4H^21_\f(u)>0\ Since the Gaussian measure is absolutely continuous, (13) gives: g‖∇f(g)‖22≤4H2qE_g\|∇ f(g)\|_2^2≤ 4H^2q (18) Applying (16) and then (18), we obtain the required upper bound: Varg(f(g))≤2H2qVar_g(f(g))≤ 2H^2q (19) We now use these two variance bounds to control the second moment required by (10). Suppose first that q<1q<1. Rearranging the lower bound (15) gives the first inequality below, and substituting the upper bound (19) gives the second: g[f(g)2]≤Varg(f(g))1−q≤2H2q1−qE_g[f(g)^2]≤ Var_g(f(g))1-q≤ 2H^2q1-q (20) Substituting (20) into (10) therefore yields: R[(F(R)−Fℐ(R))2]≤g[f(g)2]≤2H2q1−qE_R\! [ (F(R)-F_I(R) )^2 ] _g[f(g)^2]≤ 2H^2q1-q (21) Cauchy–Schwarz, followed by (21), then gives: R[F(R)−Fℐ(R)]≤(R[(F(R)−Fℐ(R))2])1/2≤H2q1−qE_R[F(R)-F_I(R)]≤ (E_R\! [ (F(R)-F_I(R) )^2 ] )^1/2≤ H 2q1-q (22) For every R∈SO(d)R (d) and every activation, ‖Rxi‖∞≤‖Rxi‖2=‖xi‖2≤H\|Rx_i\|_∞≤\|Rx_i\|_2=\|x_i\|_2≤ H Thus: 0≤F(R)−Fℐ(R)≤H0≤ F(R)-F_I(R)≤ H (23) If 0≤q≤1/30≤ q≤ 1/3, then 1−q≥2/31-q≥ 2/3, and hence: 2q1−q≤3q. 2q1-q≤ 3q. Equation (22) therefore gives: R[F(R)−Fℐ(R)]≤H3q.E_R[F(R)-F_I(R)]≤ H 3q. If 1/3≤q≤11/3≤ q≤ 1, then 3q≥1 3q≥ 1, so (23) gives: R[F(R)−Fℐ(R)]≤H≤H3q.E_R[F(R)-F_I(R)]≤ H≤ H 3q. Substituting q=q(ℐ)q=q(I) proves (4). ∎ Appendix B Reduced ADMM updates Let C=CdC=C_d and let Λ,Γ , be scaled dual variables for VS−W=0VS-W=0 and CW−Z=0CW-Z=0. With V∈St(d,r)V (d,r), we write the augmented Lagrangian of (9) as: ℒρ= _ρ= ‖Z‖44+ρ2‖VS−W+Λ‖F2+ρ2‖CW−Z+Γ‖F2−ρ2(‖Λ‖F2+‖Γ‖F2) \|Z\|_4^4+ ρ2\|VS-W+ \|_F^2+ ρ2\|CW-Z+ \|_F^2- ρ2 (\| \|_F^2+\| \|_F^2 ) (24) Holding ρ fixed within each sweep, we update Z+ Z^+ =prox∥⋅∥44/ρ(CW+Γ), =prox_\|·\|_4^4/ρ(CW+ ), (25) W+ W^+ =(Id−12C)(VS+Λ+C(Z+−Γ)), =(I_d- 12C) (VS+ +C(Z^+- ) ), (26) V+ V^+ =PQ,PΣQ=svdthin((W+−Λ)S), =PQ T, P Q T=svd_thin ((W^+- )S T ), (27) Λ+ ^+ =Λ+V+S−W+,Γ+=Γ+CW+−Z+. = +V^+S-W^+, ^+= +CW^+-Z^+. (28) For an input entry a, the proximal value is the unique real solution of 4z3+ρ(z−a)=04z^3+ρ(z-a)=0. The W-update uses (Id+C)−1=Id−C/2(I_d+C)^-1=I_d-C/2, while the V-update is an orthogonal Procrustes problem because ‖VS‖F2=‖S‖F2\|VS\|_F^2=\|S\|_F^2 on St(d,r)St(d,r). Penalty selection. For R1R_1, we initialize ρ0=10‖CU‖F2/(dB) _0=10\|CU\|_F^2/(dB), ten times the mean squared entry of the centered activation matrix. We then adapt ρ every 2020 iterations by primal–dual residual balancing [Boyd et al., 2011]. Let: rp r_p =max(‖W+−V+S‖F,‖CW+−Z+‖F), = (\|W^+-V^+S\|_F,\ \|CW^+-Z^+\|_F ), rd r_d =ρmax(‖W+−W‖F,‖Z+−Z‖F). =ρ (\|W^+-W\|_F,\ \|Z^+-Z\|_F ). At each adjustment step, set: ρ+=1.25ρ,rp>10rd,ρ/1.25,rd>10rp,ρ,otherwise,ρ^+= cases1.25\,ρ,&r_p>10\,r_d,\\ ρ/1.25,&r_d>10\,r_p,\\ ρ,&otherwise, cases and clip the result to [10−8,max(10ρ0,10−3)][10^-8, (10 _0,10^-3)]. The local R2R_2 problems are generally well behaved in our experiments, so each R2R_2 solve uses a fixed ρ=10ρ=10. Iteration counts and stopping. Each solve alternates one R1R_1 problem with R2,ℓR_2, problem per layer. R1R_1 runs for at most 10001000 iterations and for at least 300300. After iteration 300300 it stops when both the primal and dual residuals are below 10−410^-4: max(‖W−VS‖F,‖CW−Z‖F)<10−4,ρmax(‖W+−W‖F,‖Z+−Z‖F)<10−4. (\|W-VS\|_F,\ \|CW-Z\|_F )<10^-4, ρ (\|W^+-W\|_F,\ \|Z^+-Z\|_F )<10^-4. Each R2R_2 solve runs for 300300 iterations. In every case the returned factor is the feasible iterate with the smallest objective value. Appendix C Ablations and Additional Experiments Activation selection. Table 3 compares four selection rules on LLaMA-3-70B under W4A4KV4 quantization, keeping the activation budget and subsequent rotation optimization fixed. Algorithm 1 achieves a WikiText-2 perplexity of 5.635.63, compared with 6.946.94 when selecting the largest ℓ∞ _∞ activations after the initial Hadamard rotation and 7.007.00 when selecting them in the original basis. Uniformly sampling the same number of activations performs substantially worse, with a perplexity of 29.5129.51. These results show that the choice of calibration activations is important at such a small budget. Global and local rotations. The optimization ablation in Table 3 relates to the contribution of learning the global residual rotation R1R_1 and the local rotations R2R_2. Optimizing only R1R_1, while leaving R2R_2 at initialization, achieves a perplexity of 5.695.69, close to the 5.635.63 obtained when both are optimized. Optimizing only R2R_2, while keeping R1R_1 at its initial Hadamard rotation, gives a perplexity of 6.016.01. Thus, learning the global residual basis is sufficient to approach the performance of the full method in this setting, while also optimizing the local rotations provides a further improvement. Selection Optimization Algorithm 1 TopBTop_B (‖R0x‖∞)(\|R_0x\|_∞) TopBTop_B (‖x‖∞)(\|x\|_∞) Random B R1,R2R_1,R_2 R1R_1 only R2R_2 only Wiki-2 ppl ↓ 5.63 6.94 7.00 29.51 5.63 5.69 6.01 Table 3: LLaMA-3 70B W4A4KV4 ablations. Selection: all four columns optimize the same ThinQuant rotation and differ only in how the B tokens are chosen. The first column is the Gaussian direction procedure of Algorithm 1. TopB(‖R0x‖∞)Top_B(\|R_0x\|_∞) scores the tokens after the initial Hadamard R0R_0. TopB(‖x‖∞)Top_B(\|x\|_∞) keeps the tokens of largest residual ℓ∞ _∞ norm in the original basis. Random samples the same budget uniformly. Optimization: we compare keeping only the global rotation (R1R_1) or only the local rotatons (R2R_2). R1,R2R_1,R_2 is the full Llama-3-70B run. R1R_1 only optimizes the residual rotation and leaves R2R_2 at initialization. R2R_2 only optimizes this rotation and leaves R1R_1 at the initial Hadamard. Best results in each block are bold. Calibration cost breakdown. Table 4 separates rotation optimization from the total calibration procedure and reports the associated disk storage. On LLaMA-3-70B, ThinQuant reduces optimization time from 7474 minutes 4646 seconds to 11 minute 2222 seconds, approximately a 55×55× speedup over DartQuant. The total calibration time decreases from 111111 minutes to 11:57 minutes, approximately a 10×10× speedup. The smaller reduction in total time reflects the cost of activation collection, selection, and data preparation, which together account for most of ThinQuant’s calibration time. Across all five models, ThinQuant also reduces disk storage by more than three orders of magnitude. For example, LLaMA-3-70B requires 0.510.51 GiB rather than 960960 GiB. These savings arise from not needing to store the large calibration set used by DartQuant. Cost Method LLaMA-3 8B LLaMA-3 70B LLaMA-2 7B LLaMA-2 13B LLaMA-2 70B Optimization ThinQuant 23.4s 1:22 22.0s 34.5s 1:31 DartQuant 12:14 74:46 12:05 21:22 60:47 Total calibration ThinQuant 1:56 11:57 1:43 2:51 11:42 DartQuant 19:03 111:00 20:25 31:42 99:57 Disk storage ThinQuant 0.13 0.51 0.13 0.20 0.51 DartQuant 192 960 193 300 961 Table 4: Breakdown of rotation-calibration cost. Optimization measures rotation optimization only, while total calibration additionally includes activation collection, selection, training-data preparation, and folding the rotation into the weights. Times are in m:s unless marked in seconds. Disk storage is in GiB. Lower values are better, and best results are bold. Quantization performance Calibration cost Model Method Wiki-2 ↓ C4 ↓ 0-shot ↑ Runtime ↓ Peak GPU ↓ LLaMA-3 8B ThinQuant 7.75 12.55 60.87 1:56 18.5 DartQuant 8.03 13.09 59.98 19:03 37.1 OmniQuant 71.32 92.86 32.00 54:57 17.2 LLaMA-3 70B ThinQuant 5.52 10.14 66.18 11:57 31.9 DartQuant 7.15 14.88 59.05 111:00 134.9 OmniQuant 482.73 775.47 – 406:49 127.0 LLaMA-2 7B ThinQuant 6.12 8.32 57.58 1:43 18.2 DartQuant 6.25 8.52 56.93 20:25 75.1 OmniQuant 11.21 16.05 45.68 53:59 16.2 LLaMA-2 13B ThinQuant 5.43 7.59 61.25 2:51 21.9 DartQuant 5.46 7.63 61.65 31:42 51.1 OmniQuant 10.64 16.62 44.49 93:07 27.3 LLaMA-2 70B ThinQuant 3.88 6.20 67.01 11:42 33.2 DartQuant 3.89 6.19 67.31 99:57 73.1 OmniQuant 8.43 12.19 49.76 404:08 125.9 Table 5: W4A4 quantization performance and calibration cost. Weights and activations are 4-bit; the KV cache is left at 16 bits. The 0-shot column averages nine downstream tasks. For ThinQuant and DartQuant, runtime is total rotation-calibration time, including activation collection, selection, and training-data preparation. Both reuse the rotations from the W4A4KV4 runs, so those calibration costs match Table 2. OmniQuant runtime is activation-scale collection plus LET/LWC fitting. GPU denotes peak GPU process memory during calibration, in GiB. Times are in m:s. Best results are bold.