Paper deep dive
Output-Aware Rotation for INT2 KV-Cache Quantization
Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee, Murali Annavaram, Sai Praneeth Karimireddy, Sungjoo Yoo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/8/2026, 2:11:15 AM
Summary
The paper introduces OptR, an output-aware rotation method for INT2 KV-cache quantization in large language models. It addresses the mismatch between existing rotation methods that optimize proxy errors and the actual model output error by minimizing post-output-projection attention-output error. OptR decomposes this error into key- and value-induced terms, applies attention-equivalent key reparameterization to reduce channel-wise offsets, and learns per-head orthogonal corrections. It consistently improves performance on reasoning and coding benchmarks while preserving paged KV-cache formats.
Entities (13)
Relation Signals (12)
OptR → applies → Attention-Equivalent Key Reparameterization
confidence 95% · OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets
OptR → decomposes → Attention Output Error
confidence 95% · OptR decomposes the post-W_O attention-output error into key- and value-induced terms
OptR → evaluatedon → AIME'25
confidence 95% · We measure reasoning and coding accuracy on AIME24, AIME25
OptR → evaluatedon → Phi4-14B-reasoning-plus
confidence 95% · We evaluate OptR on ... Phi4-14B-reasoning-plus
OptR → evaluatedon → GPQA-Diamond
confidence 95% · We measure reasoning and coding accuracy on ... GPQA-Diamond
OptR → evaluatedon → LiveCodeBench
confidence 95% · We measure reasoning and coding accuracy on ... LiveCodeBench
OptR → evaluatedon → RULER-NIAH
confidence 95% · We additionally evaluate long-context retrieval using RULER-NIAH
OptR → evaluatedon → Qwen3-8b
confidence 95% · Figure 1 shows that OptR improves AIME25 accuracy on Qwen3-8B
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection $W_O$. To address this mismatch, we propose \textit{OptR}, an output-aware rotation method that minimizes post-$W_O$ attention-output error. OptR decomposes the post-$W_O$ attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead.
Tags
Links
- Source: https://arxiv.org/abs/2608.02691v2
- Canonical: https://arxiv.org/abs/2608.02691v2
Trouble viewing inline? Open PDF directly →
Full Text
68,792 characters extracted from source content.
Expand or collapse full text
Output-Aware Rotation for INT2 KV-Cache Quantization Vincent-Daniel Yun1,†, Woosang Lim2,†, Minsoo Cheong2, Sunwoo Lee3 Murali Annavaram1, Sai Praneeth Karimireddy1, Sungjoo Yoo2 1University of Southern California yunjuyou, annavara, karimire@usc.edu 2Seoul National University ftyg656512, icycle0409@snu.ac.kr 3Inha University sunwool@inha.ac.kr †Equal Contribution Corresponding Author: sungjoo.yoo@gmail.com. Abstract The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection WOW_O. To address this mismatch, we propose OptR, an output-aware rotation method that minimizes post-WOW_O attention-output error. OptR decomposes the post-WOW_O attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead. GitHub 1 Introduction As large language models (LLMs) grow in model size and context length, the key-value (KV) cache becomes a major bottleneck in long-context inference Dao et al. (2022); Ainslie et al. (2023). During autoregressive decoding, each layer stores the keys and values of all previous tokens and reads them at every generation step. As a result, KV-cache storage and memory traffic increase with context length, batch size, and model depth. KV-cache quantization reduces these costs by storing the cache at lower precision. We focus on INT2 because it requires only 1/81/8 of the BF16 KV-cache storage and 1/21/2 of that of INT4, enabling longer contexts or larger batches under the same memory budget. Since all values in a quantization group share one scale, a few large values can expand the range represented by only four INT2 levels. This increases rounding error for most values, while aggressive clipping introduces large errors in the outliers themselves Zhang et al. (2023); Hooper et al. (2024); Zhou et al. (2026). Rotation-based methods reduce this error by spreading a few extreme channel values across dimensions, and make the cache easier to quantize. Given an orthogonal matrix R, a cache vector z is transformed to zRzR before quantization and mapped back with R⊤R after dequantization Chee et al. (2023); Ashkboos et al. (2024). Rotation preserves the cache shape and regular memory layout, maintaining compatibility with paged KV-cache systems and fused decoding kernels Kwon et al. (2023); Zheng et al. (2024); Zhou et al. (2026). The main challenge is selecting R. Existing Rotation-based INT2 KV cache pipelines rely on fixed transforms, or proxy objectives defined before the complete attention readout Ashkboos et al. (2024); Su et al. (2025); Zhou et al. (2026). Figure 1: AIME25 accuracy of Qwen3-8B under BF16 and INT2 KV-cache quantization. ++ OptR denotes applying key reparameterization and output-aware rotation correction to the corresponding base rotation. The dashed line marks the BF16 accuracy. However, these proxy objectives do not directly measure the error passed to later layers. KV quantization changes the attention readout, and the output projection WOW_O maps this change into the model hidden space. The resulting output error enters the residual stream and propagates through subsequent layers, potentially affecting the final prediction. Consequently, the rotation that best reconstructs the cached keys and values may not be the one that best preserves the post-WOW_O output. This objective mismatch motivates optimizing rotations in output space. To address this mismatch, we propose OptR, an output-aware rotation method for INT2 KV-cache quantization. OptR first centers the keys before rotation and quantization. This shifts all logits for a query by the same constant and therefore leaves the softmax distribution unchanged. Since INT2 has only four quantization levels, outliers can lead to large quantization errors. This reparameterization reduces their effect by narrowing the quantization range. OptR then learns per-head orthogonal corrections to any base rotation by minimizing the post-WOW_O attention-output error through the INT2 attention path. It optimizes the key rotation before the value rotation because the quantized keys determine the attention weights used for value aggregation. Only the rotation parameters are optimized on calibration data. Model weights remain frozen, and the learned rotations are fixed during inference. We integrate OptR into an SGLang-based INT2 KV cache pipeline while retaining paged and prefix-cache support with negligible runtime overhead Zhou et al. (2026). Figure 1 shows that OptR improves AIME25 accuracy on Qwen3-8B from 17.33% to 66.67% with QuaRot and from 54.67% to 66.00% with OSCAR, compared with 68.00% for BF16. The gains with both base rotations show that OptR does not depend on a specific rotation initialization. Our contributions are summarized as follows. • We formulate INT2 KV-cache quantization as an output-space optimization problem and decompose the post-WOW_O attention-output error into key- and value-induced terms. • We propose OptR, which first applies attention-equivalent key reparameterization to reduce large channel-wise offsets and then learns per-head orthogonal corrections through the complete INT2 quantization and attention path. • We show that OptR consistently improves existing rotation-based INT2 KV-cache pipelines while retaining their cache layout with negligible serving overhead. 2 Related Works KV-cache quantization. The KV cache grows with context length and is repeatedly read during decoding, making it a major memory and bandwidth bottleneck. Prior work reduces this cost through fine-grained quantization, mixed precision, and vector quantization Liu et al. (2024); Hooper et al. (2024); Xia et al. (2026); Son et al. (2026). These designs often introduce residual buffers, channel-wise metadata, promoted high-precision channels, or specialized cache layouts, which complicate their integration with paged KV-cache systems and fused decoding kernels. In contrast, rotation-based quantization transforms cached vectors into a quantization-friendly basis without changing their tensor shape or regular cache layout, making it easier to deploy in existing inference systems Ashkboos et al. (2024); Zhou et al. (2026). Rotation-based KV-cache quantization. QuaRot uses Hadamard rotations for weights, activations, and KV caches Ashkboos et al. (2024), while RotateKV adapts rotations to head-specific key outliers and protects attention sinks Su et al. (2025). OSCAR derives key and value rotations from offline attention-aware covariance statistics Zhou et al. (2026). Despite these differences, existing methods select rotations using fixed transforms, cache statistics, or proxy objectives defined before the complete attention readout. Instead, OptR optimizes rotations against the post-WOW_O attention-output error produced by the complete INT2 attention path. 3 Problem Formulation Preliminaries. We consider a decoder-only Transformer layer ℓ with grouped-query attention (GQA) Vaswani et al. (2017); Ainslie et al. (2023). Let h∈1,…,Hkvh∈\1,…,H_kv\ denote a KV head, and let Gh⊆1,…,HqG_h \1,…,H_q\ denote the set of query heads that share this KV head. For each query head j∈Ghj∈ G_h, we write qt,jℓ,ks,hℓ,vs,hℓ∈ℝ1×d,q_t,j ,k_s,h ,v_s,h ^1× d, where s≤ts≤ t, t is the current decoding position, and s indexes a cached source token. Equivalently, the cached keys and values for KV head h up to position t are K1:t,hℓ=[k1,hℓ;…;kt,hℓ]∈ℝt×d,K_1:t,h =[k_1,h ;…;k_t,h ] ^t× d, and V1:t,hℓ=[v1,hℓ;…;vt,hℓ]∈ℝt×d.V_1:t,h =[v_1,h ;…;v_t,h ] ^t× d. The BF16 attention logits and probabilities are: at,sℓ,j,h=⟨qt,jℓ,ks,hℓ⟩d,ptℓ,j,h=softmaxs≤t(at,sℓ,j,h)a_t,s ,j,h= q_t,j ,k_s,h d, p_t ,j,h=softmax_s≤ t (a_t,s ,j,h ) (1) The corresponding attention output is ot,jℓ=∑s≤tpt,sℓ,j,hvs,hℓ=(ptℓ,j,h)⊤V1:t,hℓ∈ℝ1×do_t,j = _s≤ tp_t,s ,j,hv_s,h =(p_t ,j,h) V_1:t,h ^1× d (2) Let WO,jℓ∈ℝdmodel×dW_O,j ^d_model× d be the output projection for query head j. The attention-output contribution of this head is: yt,jℓ=((ptℓ,j,h)⊤V1:t,hℓ)(WO,jℓ)⊤∈ℝ1×dmodely_t,j = ( (p_t ,j,h ) V_1:t,h )(W_O,j ) ^1× d_model (3) Thus, the KV cache affects the model through the attention-weighted readout (ptℓ,j,h)⊤V1:t,hℓ(p_t ,j,h) V_1:t,h and its projection by WO,jℓW_O,j . Figure 2: Key magnitude (top) and key-induced attention-output error by cached token (bottom) across five INT2 KV-cache settings on Qwen3-8B (AIME25): OSCAR only, plain INT2, reparameterization (mean shift), reparameterization with OSCAR rotation, and the full OptR pipeline. Lower is better; example details are provided in the Appendix. 3.1 Rotated INT2 KV-Cache Quantization We consider long-context decoding where the long-history KV cache is stored in INT2 and a small BF16 window is preserved. Let Q2(⋅;c,G)Q_2(·;c,G) denote the INT2 quantize-dequantize map with clipping ratio c and group size G. For an orthogonal rotation R∈O(d)R∈ O(d), define DR,c(z)=Q2(zR;c,G)R⊤D_R,c(z)=Q_2(zR;c,G)R (4) If Q2Q_2 is replaced by the identity map, then DR,c(z)=zD_R,c(z)=z. Thus, the rotation only changes the coordinate system in which INT2 quantization error is introduced. We denote by k~s,hℓ k_s,h and v~s,hℓ v_s,h the effective keys and values used by attention after rotated INT2 quantization and BF16 window restoration. 3.2 Output-Space Error Induced by KV Quantization We now trace the effective INT2 cache through attention and WOW_O and decompose the resulting post-WOW_O attention-output error into key- and value-induced terms. With INT2 keys, the attention logits and probabilities become a~t,sℓ,j,h a_t,s ,j,h =⟨qt,jℓ,k~s,hℓ⟩d,p~tℓ,j,h=softmaxs≤t(a~t,sℓ,j,h) = q_t,j , k_s,h d, p_t ,j,h=softmax_s≤ t ( a_t,s ,j,h ) (5) Let Δks,hℓ=k~s,hℓ−ks,hℓ k_s,h = k_s,h -k_s,h and Δvs,hℓ=v~s,hℓ−vs,hℓ v_s,h = v_s,h -v_s,h denote the key and value quantization errors. Key errors first perturb the attention logits: Δat,sℓ,j,h=a~t,sℓ,j,h−at,sℓ,j,h=⟨qt,jℓ,Δks,hℓ⟩d a_t,s ,j,h= a_t,s ,j,h-a_t,s ,j,h= q_t,j , k_s,h d (6) The resulting attention error is Δptℓ,j,h=p~tℓ,j,h−ptℓ,j,h p_t ,j,h= p_t ,j,h-p_t ,j,h. Thus, the effect of a key error depends on the query and the softmax attention map, not only on ‖Δk‖22\| k\|_2^2. Under INT2 keys and values, the attention-output contribution becomes y~t,jℓ=(∑s≤tp~t,sℓ,j,hv~s,hℓ)(WO,jℓ)⊤ y_t,j = ( _s≤ t p_t,s ,j,h v_s,h )(W_O,j ) (7) Subtracting the BF16 readout gives the exact decomposition. Let Δyt,jℓ:=y~t,jℓ−yt,jℓ y_t,j := y_t,j -y_t,j . Then Δyt,jℓ=(∑s≤tΔpt,sℓ,j,hvs,hℓ)(WO,jℓ)⊤⏟key-induced output error+(∑s≤tp~t,sℓ,j,hΔvs,hℓ)(WO,jℓ)⊤⏟value-induced output error y_t,j = ( _s≤ t p_t,s ,j,hv_s,h ) (W_O,j ) _key-induced output error+ ( _s≤ t p_t,s ,j,h v_s,h ) (W_O,j ) _value-induced output error (8) We denote the two terms above by δyK,t,jℓ,hδ y_K,t,j ,h and δyV,t,jℓ,hδ y_V,t,j ,h, respectively. Here δyK,t,jℓ,hδ y_K,t,j ,h is the output error induced by key quantization through the attention distribution, while δyV,t,jℓ,hδ y_V,t,j ,h is the value error after attention-weighted aggregation and output projection. This decomposition shows why raw cache reconstruction is only a proxy. A reconstruction-based objective measures Erec=‖K−K~‖F2+‖V−V~‖F2E_rec=\|K- K\|_F^2+\|V- V\|_F^2 (9) whereas the model observes the attention-output error Eout=‖y~t,jℓ−yt,jℓ‖22=‖δyK,t,jℓ,h+δyV,t,jℓ,h‖22E_out= \| y_t,j -y_t,j \|_2^2= \|δ y_K,t,j ,h+δ y_V,t,j ,h \|_2^2 (10) Eq. (9) and Eq. (10) can favor different rotations because attention and WOW_O reduce the effect of some cache errors while allowing others to affect the attention-output. For this reason, OptR uses EoutE_out as its rotation optimization target. Figure 3: Overview of OptR. OptR augments an existing rotation-based KV-cache quantization pipeline with channel-wise key reparameterization and output-aware correction rotations before INT2 quantization. The resulting quantized KV cache is consumed by attention and projected through WOW_O, where OptR targets reduced post-WOW_O attention-output error. 4 Method: Output-Aware Rotation Figure 5 shows that INT2-induced output error differs across KV heads. OptR therefore learns a separate orthogonal correction to the base key and value rotations for each head. During one-time offline calibration, model weights remain frozen and only the corrections are optimized to reduce post-WOW_O attention-output error through the INT2 attention path. We optimize the key rotation first because the INT2 keys determine the attention distribution, and then optimize the value rotation under this distribution. The resulting rotations are fixed during inference. Figure 3 summarizes the full pipeline. We omit (ℓ,h)( ,h) when the layer and KV head are clear. 4.1 Rotated INT2 Cache Reparameterization As shown in Figure 2, large channel-wise key offsets increase the dynamic range of group-wise INT2 quantization. Therefore, we reparameterize keys using the per-channel calibration mean μ∈ℝdμ ^d before rotation and quantization. For key and value rotations RKR_K and RVR_V, OptR defines k¯s(RK) k_s(R_K) =DRK,cK(ks−μ),v¯s(RV)=DRV,cV(vs) =D_R_K,c_K(k_s-μ), v_s(R_V)=D_R_V,c_V(v_s) This key reparameterization is attention-equivalent: subtracting the same μ from every key adds a query-dependent constant to all logits and leaves the softmax distribution unchanged. We apply it only to keys, since shifting values would alter the attention output. Sink and recent tokens remain in BF16, and the same centering is applied to their keys. The effective cache is denoted by k~s(RK) k_s(R_K) and v~s(RV) v_s(R_V). 4.2 Orthogonal Rotation Refinement OptR does not rely on a specific rotation initialization. Let RK0,RV0∈O(d)R_K^0,R_V^0∈ O(d) denote arbitrary orthogonal initializations for the key and value rotations, respectively. Given these initial rotations, OptR applies the same output-aware rotation procedure regardless of how they are constructed. For each KV head, OptR learns unconstrained matrices AK,AV∈ℝd×dA_K,A_V ^d× d and forms the skew-symmetric generators SK=AK−AK⊤,SV=AV−AV⊤S_K=A_K-A_K , S_V=A_V-A_V (11) which define the corrected rotations RK(AK)=RK0exp(SK),RV(AV)=RV0exp(SV)R_K(A_K)=R_K^0 (S_K), R_V(A_V)=R_V^0 (S_V) Since SKS_K and SVS_V are skew-symmetric, their matrix exponentials are orthogonal. Therefore, the corrected rotations remain orthogonal throughout calibration. The same formulation applies to different rotation initializations without modifying the calibration objective. 4.3 Output-Aware Key Calibration The key rotation affects attention logits and the attention distribution next. To isolate this path, we calibrate the key rotation while keeping values in BF16. The induced key-only attention distribution is at,sK(RK)=⟨qt,j,k~s(RK)⟩d,ptK(RK)=softmaxs≤t(at,sK(RK)) a_t,s^K(R_K)= q_t,j, k_s(R_K) d, p_t^K(R_K)=softmax_s≤ t (a_t,s^K(R_K) ) (12) The key-induced attention-output error is eK(t,j;RK)=[∑s≤t(pt,sK(RK)−pt,s)vs](WO,j)⊤e_K(t,j;R_K)= [ _s≤ t (p_t,s^K(R_K)-p_t,s )v_s ](W_O,j) (13) We optimize the key rotation with LK(RK;D)=(t,j)∈D[DKL(pt∥ptK(RK))+λK‖eK(t,j;RK)‖22dmodel] L_K(R_K;D)=E_(t,j)∈ D [D_KL (p_t\,\|\,p_t^K(R_K) )+ _K \|e_K(t,j;R_K)\|_2^2d_model ] (14) The KL term preserves the attention distribution, while the second term measures the post-WOW_O attention-output error caused by key-induced attention changes. 4.4 Output-Aware Value Calibration After optimizing the key rotation, we calibrate the value rotation under the selected quantized-key attention path. Let R^K R_K be the selected key rotation and let p^tK p_t^K be its induced attention distribution. The value-induced attention-output error is eV(t,j;RV)=[∑s≤tp^t,sK(v~s(RV)−vs)](WO,j)⊤e_V(t,j;R_V)= [ _s≤ t p_t,s^K ( v_s(R_V)-v_s ) ](W_O,j) (15) We optimize the value rotation with LV(RV;D)=(t,j)∈D[‖eV(t,j;RV)‖22dmodel]L_V(R_V;D)=E_(t,j)∈ D [ \|e_V(t,j;R_V)\|_2^2d_model ] (16) This objective measures value-induced attention-output error after attention-weighted aggregation and output projection, rather than raw value-cache reconstruction. 4.5 INT2-Aware Orthogonal Calibration OptR calibrates the rotation generators through the full INT2 quantization path: rotation, clipping, grouping, INT2 rounding, dequantization, and inverse rotation. For each layer and KV head, OptR learns the orthogonal correction: AK⋆=argminALK(RK0exp(A−A⊤);Dtrain),AV⋆=argminALV(RV0exp(A−A⊤);Dtrain) A_K = _AL_K (R_K^0 (A-A );D_train ), A_V = _AL_V (R_V^0 (A-A );D_train ) (17) All model weights remain frozen; only the per-head rotation generators are calibrated. Since INT2 rounding is non-differentiable, gradients are estimated using a straight-through estimator: ∂Q2(Z;c,G)∂Z≈|Z|≤τc ∂ Q_2(Z;c,G)∂ Z 1\|Z|≤ _c\ (18) where τc _c is the clipping threshold determined by the clip ratio c. 5 Experimental Results Models and Benchmarks. We evaluate OptR on Qwen3-4B-Thinking-2507 Yang et al. (2025), Qwen3-8B Yang et al. (2025), and Phi4-14B-reasoning-plus Abdin et al. (2025). We measure reasoning and coding accuracy on AIME24, AIME25 MAA (2025), GPQA-Diamond Rein et al. (2024), MBPP+ Liu et al. (2023), and LiveCodeBench v6 Jain et al. (2025). We additionally evaluate long-context retrieval using RULER-NIAH (Needle-in-a-Haystack) Hsieh et al. (2024), with context lengths up to 64K for the Qwen3 models and 32K for Phi4-14B-reasoning-plus. For all models, we use a temperature of 0.6, top-p of 0.95, and top-k of 20. For the reasoning and coding benchmarks, we use maximum generation lengths of 32K for the Qwen3 models and 16K for Phi4-14B-reasoning-plus. Model Method BPE AIME24 AIME25 GPQA MBPP++ LCB v6 Mean Qwen3-4B- Thinking-2507 BF16 16 78.00±6.9178.00± 6.91 71.33±3.8071.33± 3.80 64.55±2.3064.55± 2.30 77.31±0.8777.31± 0.87 46.71±2.0546.71± 2.05 67.5867.58 TurboQuant (no MP) 3.25 10.0010.00 16.6716.67 41.9241.92 23.0223.02 1.711.71 18.6618.66 QuaRot-INT2 2.32 0.00±0.000.00± 0.00 0.00±0.000.00± 0.00 6.06±1.186.06± 1.18 5.61±2.365.61± 2.36 1.37±0.311.37± 0.31 2.612.61 !15QuaRot-INT2 + OptR !152.32 !1569.33±4.3569.33± 4.35 !1560.67±4.3560.67± 4.35 !1562.12±2.9262.12± 2.92 !1578.31±1.2378.31± 1.23 !1537.71±0.8137.71± 0.81 !1561.6361.63 OSCAR 2.32 68.67±2.9868.67± 2.98 63.33±4.0863.33± 4.08 62.02±1.7362.02± 1.73 75.66±0.8475.66± 0.84 43.89±0.7543.89± 0.75 62.7162.71 !15OSCAR + OptR !152.32 !1572.00±1.8372.00± 1.83 !1570.67±2.7970.67± 2.79 !1562.83±2.2262.83± 2.22 !1576.83±0.9176.83± 0.91 !1544.57±0.9044.57± 0.90 !1565.3865.38 Qwen3-8B BF16 16 76.00±6.4176.00± 6.41 68.00±2.9868.00± 2.98 55.76±2.1655.76± 2.16 79.15±1.4879.15± 1.48 49.37±0.6549.37± 0.65 65.6665.66 TurboQuant (no MP) 3.25 53.3353.33 40.0040.00 45.9645.96 58.9958.99 21.1421.14 43.8843.88 QuaRot-INT2 2.32 16.00±5.4816.00± 5.48 17.33±4.3517.33± 4.35 42.12±1.5442.12± 1.54 49.10±1.0049.10± 1.00 5.26±1.245.26± 1.24 25.9625.96 !15QuaRot-INT2 + OptR !152.32 !1576.00±4.3576.00± 4.35 !1566.67±3.3366.67± 3.33 !1557.58±2.4557.58± 2.45 !1580.63±1.1480.63± 1.14 !1539.77±1.5439.77± 1.54 !1564.1364.13 OSCAR 2.32 72.00±3.8072.00± 3.80 54.67±3.8054.67± 3.80 56.26±3.2556.26± 3.25 78.73±0.4478.73± 0.44 45.03±2.2645.03± 2.26 61.3461.34 !15OSCAR + OptR !152.32 !1576.67±2.3676.67± 2.36 !1566.00±4.9466.00± 4.94 !1558.99±2.5658.99± 2.56 !1579.21±0.5579.21± 0.55 !1547.54±0.4847.54± 0.48 !1565.6865.68 Phi4-14B- reasoning-plus BF16 16 70.00±4.7170.00± 4.71 60.67±4.9460.67± 4.94 45.56±2.0945.56± 2.09 77.18±2.1677.18± 2.16 39.43±2.0639.43± 2.06 58.5758.57 TurboQuant (no MP) 3.25 60.0060.00 53.3353.33 46.9746.97 74.6074.60 33.1433.14 53.6153.61 QuaRot-INT2 2.32 61.33±4.4761.33± 4.47 46.00±5.4846.00± 5.48 46.77±3.4046.77± 3.40 75.82±1.1675.82± 1.16 34.97±1.4834.97± 1.48 52.9852.98 !15QuaRot-INT2 + OptR !152.32 !1564.00±3.6564.00± 3.65 !1552.00±2.9852.00± 2.98 !1547.58±2.9147.58± 2.91 !1576.70±0.6776.70± 0.67 !1536.00±0.5736.00± 0.57 !1555.2655.26 OSCAR 2.32 62.67±2.7962.67± 2.79 49.33±3.6549.33± 3.65 43.84±2.7143.84± 2.71 73.02±1.4473.02± 1.44 34.29±1.0434.29± 1.04 52.6352.63 !15OSCAR + OptR !152.32 !1565.33±5.0665.33± 5.06 !1558.00±5.5858.00± 5.58 !1546.16±1.5046.16± 1.50 !1573.44±0.5573.44± 0.55 !1535.09±1.1135.09± 1.11 !1555.6055.60 Table 1: Comparison of INT2 KV-cache quantization methods across three model configurations and five benchmarks. Results are reported as μ±σμ±σ over five seeds. OptR indicates that output-aware rotation is applied to the corresponding baseline. BPE denotes the effective number of bits per KV-cache element. TurboQuant is reported from a single run because repeated 32K evaluations are slow in its vLLM implementation. Model Method 4k 8k 16k 32k 64k Qwen3-4B- Thinking-2507 BF16 99.90±0.0899.90± 0.08 99.60±0.0799.60± 0.07 98.19±0.0898.19± 0.08 97.13±0.1197.13± 0.11 88.76±0.4788.76± 0.47 QuaRot-INT2 3.08±0.193.08± 0.19 8.26±0.858.26± 0.85 0.00±0.000.00± 0.00 6.14±1.116.14± 1.11 0.72±0.160.72± 0.16 !15QuaRot-INT2 + OptR !1599.78±0.1799.78± 0.17 !1599.02±0.3899.02± 0.38 !1595.59±0.5295.59± 0.52 !1574.69±0.9974.69± 0.99 !1546.86±1.7746.86± 1.77 OSCAR 99.29±0.3199.29± 0.31 97.59±0.5097.59± 0.50 93.78±0.3393.78± 0.33 77.75±0.6477.75± 0.64 55.72±1.4455.72± 1.44 !15OSCAR + OptR !1599.42±0.2799.42± 0.27 !1598.57±0.1798.57± 0.17 !1595.31±0.1095.31± 0.10 !1584.98±0.4584.98± 0.45 !1569.58±1.3169.58± 1.31 Qwen3-8B BF16 99.83±0.1199.83± 0.11 99.93±0.0299.93± 0.02 99.45±0.0899.45± 0.08 98.70±0.4998.70± 0.49 84.22±1.4184.22± 1.41 QuaRot-INT2 84.97±0.0984.97± 0.09 41.73±5.7541.73± 5.75 18.16±1.1418.16± 1.14 13.03±1.7113.03± 1.71 0.04±0.070.04± 0.07 !15QuaRot-INT2 + OptR !1599.40±0.2999.40± 0.29 !1598.76±0.5798.76± 0.57 !1596.14±0.3596.14± 0.35 !1586.37±0.7286.37± 0.72 !1570.02±1.7970.02± 1.79 OSCAR 99.59±0.1599.59± 0.15 97.94±0.2897.94± 0.28 94.39±0.2994.39± 0.29 83.76±0.5883.76± 0.58 57.54±1.8257.54± 1.82 !15OSCAR + OptR !1599.60±0.1199.60± 0.11 !1598.16±0.6398.16± 0.63 !1595.50±0.6595.50± 0.65 !1586.42±0.5586.42± 0.55 !1568.65±0.8068.65± 0.80 Phi4-14B- reasoning-plus BF16 98.50±0.4598.50± 0.45 97.54±0.5997.54± 0.59 97.76±0.7097.76± 0.70 92.13±1.3492.13± 1.34 N/A QuaRot-INT2 96.28±0.4496.28± 0.44 92.28±1.3192.28± 1.31 84.54±0.5284.54± 0.52 69.65±0.9469.65± 0.94 N/A !15QuaRot-INT2 + OptR !1597.62±0.2197.62± 0.21 !1594.02±0.8094.02± 0.80 !1589.98±0.5289.98± 0.52 !1576.21±0.5376.21± 0.53 !15N/A OSCAR 95.62±1.0295.62± 1.02 90.50±0.2490.50± 0.24 84.59±0.8684.59± 0.86 70.09±0.5970.09± 0.59 N/A !15OSCAR + OptR !1596.30±0.3796.30± 0.37 !1593.42±0.3593.42± 0.35 !1587.64±0.4587.64± 0.45 !1575.36±0.8575.36± 0.85 !15N/A Table 2: RULER-NIAH long-context retrieval accuracy evaluated at context lengths ranging from 4k to 64k tokens. Results are reported as μ±σμ±σ over three random seeds, with 800 examples evaluated per seed. OptR indicates that output-aware rotation is applied to the corresponding baseline. Since Phi4-14B-reasoning-plus supports a maximum context length of 32k, its evaluation is limited to 32k, and the 64k setting is reported as N/A. Additional 128K results are in the Appendix. Implementation Details. OptR is implemented on top of the official OSCAR codebase Zhou et al. (2026). We use group-wise affine INT2 quantization with a group size of 128, clipping ratios of 0.96 for keys and 0.92 for values, and retain 64 sink tokens and 256 recent tokens in BF16. λK _K is fixed to 1.0 across all models and benchmarks. We report an effective cache cost of 2.32 bits per element (BPE) at a 64K-token context, including quantization metadata and the BF16 windows. For each model, we collect a 30K-token pool of BF16 GPQA QKV traces and use disjoint subsets for calibration and held-out rotation selection. For each layer and KV head, we optimize only the rotation corrections for 80 Adam steps with a learning rate of 0.02 Kingma and Ba (2014). All model weights remain frozen. All experiments were conducted on four NVIDIA A100 40GB GPUs. Additional implementation details are provided in the Appendix. Baselines. We compare OptR with BF16, TurboQuant Zandieh et al. (2026) without mixed precision (no MP), QuaRot-INT2 Ashkboos et al. (2024), and OSCAR Zhou et al. (2026). OptR is applied to both QuaRot and OSCAR to evaluate its effectiveness across fixed Hadamard rotations and the state-of-the-art attention-aware covariance rotation. All rotation-based methods use the same INT2 group size and BF16 sink and recent-token windows. 5.1 Main Results Overall Accuracy. Table 1 compares OptR with BF16 and existing INT2 KV-cache quantization methods. QuaRot-INT2 often suffers from severe accuracy degradation, particularly on reasoning and coding tasks. Applying OptR substantially improves both QuaRot and OSCAR at the same 2.32 BPE across the evaluated models and benchmarks. Figure 4: Per-layer errors under INT2 KV-cache quantization on Qwen3-4B-Thinking-2507. Panels (a,b) show post-WOW_O attention-output error on GPQA-Diamond and AIME25 at each layer. Panels (c,d) show the propagated residual-stream error, measured as the normalized squared error between BF16 and INT2-KV block-output hidden states on the same BF16-generated token sequences. Legend values indicate the mean across layers. Error values are shown on a logarithmic scale. Detailed settings are provided in Appendix. Figure 5: Output error by layer and KV head relative to naive INT2 (Qwen3-4B-Thinking-2507, GPQA-Diamond). Each cell reports the post-WOW_O attention-output RMS error as a percentage of plain per-group INT2 without rotation. Lower values indicate smaller output error, with naive INT2 corresponding to 100%100\%. Panels show (a) TurboQuant, (b) QuaRot, (c) QuaRot + OptR, (d) OSCAR, and (e) OSCAR + OptR. Detailed settings are provided in Appendix. Long-Context Robustness. Table 2 reports RULER-NIAH Hsieh et al. (2024) retrieval accuracy across increasing context lengths. QuaRot-INT2 degrades rapidly as the context grows, whereas OptR preserves substantially stronger retrieval performance for both QuaRot and OSCAR. The gains become more pronounced at longer contexts, indicating that OptR more effectively mitigates accumulated KV quantization error over long histories. Output-Error Analysis. Figure 4 shows that OptR reduces both post-WOW_O attention-output error and propagated residual-stream error relative to QuaRot and OSCAR on GPQA-Diamond and AIME25. Figure 5 further shows that these reductions occur across most layers and KV heads rather than only a small subset. Method Qwen3-4B- Thinking-2507 Phi4-14B- reasoning-plus BF16 71.33±3.8071.33± 3.80 60.67±4.9460.67± 4.94 OSCAR 63.33±4.0863.33± 4.08 49.33±3.6549.33± 3.65 OSCAR + Key Reparam 66.00±4.7166.00± 4.71 54.67±6.9154.67± 6.91 !15OSCAR + OptR !1570.67±2.7970.67± 2.79 !1558.00±5.5858.00± 5.58 Table 3: Component-wise evaluation of OptR on AIME25 across Qwen3-4B-Thinking-2507 and Phi4-14B-reasoning-plus using the state-of-the-art OSCAR rotation. Key Objective ℒKL_K Value Objective ℒVL_V AIME25 DKL+λKℰ(K~−K)D_KL+ _KE\! ( K-K ) ℰ(V~−V)E\! ( V-V ) 65.33±6.9165.33± 6.91 DKL+λKℰ(ΔpK⊤V)D_KL+ _KE\! ( p_K V ) ℰ((p^tK)⊤ΔV)E\! (( p_t^K) V ) 62.67±2.7962.67± 2.79 !15DKL+λKℰ((ΔpK⊤V)WO,j⊤)D_KL+ _KE\! ( ( p_K V )W_O,j ) !15ℰ(((p^tK)⊤ΔV)WO,j⊤)E\! ( (( p_t^K) V )W_O,j ) !1570.67±2.7970.67± 2.79 Table 4: Objective ablation for OptR on Qwen3-4B-Thinking-2507 with the OSCAR base rotation. ℰ(X)=mean(X2)E(X)=mean(X^2) denotes the element-wise mean squared error. The DKLD_KL term is fixed across all rows, and the three rows vary the error term using cache reconstruction, pre-WOW_O attention readout, and post-WOW_O attention-output error, respectively. Figure 6: Efficiency analysis of OptR integrated into the optimized rotated INT2 KV cache pipeline on an NVIDIA A100 40GB using Qwen3-4B-Thinking-2507. (a) Batch-1 decode latency across context lengths from 1K to 128K. (b) End-to-end output throughput, for a 2K-input and 4K-output workload at different batch sizes. (c) Prefill time for 2K-token inputs at representative batch sizes. Analysis shows that OptR adds negligible inference overhead to the underlying pipeline. Ablation Study. All ablation results are reported as μ±σμ±σ over five seeds. Table 4 compares cache-reconstruction, pre-WOW_O, and post-WOW_O calibration objectives on Qwen3-4B-Thinking-2507 using the same OSCAR base rotation. The post-WOW_O attention-output objective achieves the highest AIME25 accuracy. Table 3 shows that the full OptR consistently achieves the strongest performance across both base rotations and models, with key reparameterization providing an additional source of improvement. Additional sensitivity ablation results for λK _K tuning are provided in the Appendix. 6 Discussion Efficiency Analysis. We integrate OptR into an optimized INT2 KV-cache pipeline and extend its Triton cache-write kernel to support per-KV-head key rotations and key reparameterization. For each head h, reparameterization before rotation can be rewritten as (k−μh)RK,h=kRK,h−μhRK,h(k- _h)R_K,h=kR_K,h- _hR_K,h We therefore precompute the rotated mean μhRK,h _hR_K,h once offline and subtract it immediately after rotating each key. The same kernel then clips, quantizes, packs, and writes the reparameterized keys and rotated values to the INT2 cache in a single launch. The value rotation and its inverse are absorbed into the value projection and WOW_O, respectively, avoiding separate value-side rotation kernels. To match the per-KV-head key rotations under GQA, we implement an optimized Triton kernel that applies the corresponding rotation to each query head. Figure 6 shows that OptR remains within 2% of the corresponding base pipeline in decode latency, end-to-end throughput, and prefill time across the evaluated settings, while retaining the same maximum batch sizes. For the 2K-token input and 4K-token output workload in Figure 6(b), adding OptR at B=128B=128 increases GPU memory by only 18 MiB, from 36,205 MiB to 36,223 MiB. Thus, OptR preserves the efficiency and serving capacity of the underlying INT2 pipeline. 7 Conclusion We introduced OptR, an output-aware rotation method for INT2 KV-cache quantization. OptR combines attention-equivalent key reparameterization with per-head orthogonal corrections optimized through the complete quantized attention path, directly preserving the attention output observed by subsequent layers. Across diverse models, reasoning and coding tasks, and long-context retrieval settings, OptR consistently strengthens existing rotation-based methods and reduces quantization-induced output error. Moreover, OptR retains compatibility with paged KV-cache serving and introduces negligible inference overhead, demonstrating that output-aware optimization is an effective and practical direction for ultra-low-bit KV-cache quantization. References Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: fast and memory-efficient exact attention with io-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022. Ainslie et al. [2023] Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4895–4901, 2023. Zhang et al. [2023] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023. Hooper et al. [2024] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: towards 10 million context length llm inference with kv cache quantization. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024. Zhou et al. [2026] Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, and Xiaoxia Wu. Oscar: Offline spectral covariance-aware rotation for 2-bit kv cache quantization. 2026. URL https://arxiv.org/abs/2605.17757. Chee et al. [2023] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xrk9g5vcXR. Ashkboos et al. [2024] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240, 2024. Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023. Zheng et al. [2024] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024. Su et al. [2025] Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: accurate and robust 2-bit kv cache quantization for llms via outlier-aware adaptive rotations. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 6200–6208, 2025. Liu et al. [2024] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In International Conference on Machine Learning, pages 32332–32344. PMLR, 2024. Xia et al. [2026] Haojun Xia, Xiaoxia Wu, Jisen Li, Tsai chuan Wu, Junxiong Wang, Jue WANG, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, and Shuaiwen Leon Song. Kitty: Accurate and efficient 2-bit KV cache quantization with dynamic channel-wise precision boost. In Ninth Conference on Machine Learning and Systems, 2026. URL https://openreview.net/forum?id=r3mQiuYKIN. Son et al. [2026] Donghyun Son, Euntae Choi, and Sungjoo Yoo. Nsnquant: A double normalization approach for calibration-free low-bit vector quantization of kv cache. Advances in Neural Information Processing Systems, 38:43124–43159, 2026. Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Yang et al. [2025] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Abdin et al. [2025] Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318, 2025. MAA [2025] MAA. Aime 2025: American invitational mathematics examination. https://maa.org/math-competitions/aime, 2025. Rein et al. [2024] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98. Liu et al. [2023] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=1qvx610Cu7. Jain et al. [2025] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL. Hsieh et al. [2024] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy. Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Zandieh et al. [2026] Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. Turboquant: Online vector quantization with near-optimal distortion rate. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=tO3ASKZlok. Appendix A Supplementary Materials Additional implementation details, experimental settings, and qualitative analyses are provided in the supplementary materials. OptR Algorithm. Algorithm 1 summarizes the one-time offline calibration of OptR. For each layer and KV head, OptR estimates the key mean and optimizes only the key and value rotation corrections while keeping all model weights frozen. The matrices AKA_K and AVA_V parameterize orthogonal corrections through exp(A−A⊤) (A-A ). We use K~ K and V~ V to denote the effective keys and values consumed by attention after rotated INT2 quantization. Sink and recent tokens skip INT2 quantization; their keys retain the same centering, while their values remain unchanged in BF16. Algorithm 1 OptR offline calibration 1:Calibration traces D, base rotations RK0,RV0R_K^0,R_V^0, key-loss weight λK _K, optimization steps T 2:R^K,ℓ,h,R^V,ℓ,h,μℓ,hℓ,h\ R_K, ,h, R_V, ,h, _ ,h\_ ,h 3:for all layers ℓ and KV heads h do 4: Estimate the key mean μℓ,h _ ,h from D 5: Initialize AK,AV←0A_K,A_V← 0 6: for i=1,…,Ti=1,…,T do 7: RK←RK0exp(AK−AK⊤)R_K← R_K^0 (A_K-A_K ) 8: Construct effective keys K~(RK) K(R_K) through key reparameterization and rotated INT2 quantization 9: ptK←softmax(qt,jK~1:t(RK)⊤/d)p_t^K (q_t,j K_1:t(R_K) / d ) 10: eK(t,j)←[(ptK−pt)⊤V1:t]WO,j⊤e_K(t,j)← [(p_t^K-p_t) V_1:t ]W_O,j 11: ℒK←(t,j)∈[DKL(pt∥ptK)+λK‖eK(t,j)‖22dmodel]L_K _(t,j) [D_KL(p_t\|p_t^K)+ _K \|e_K(t,j)\|_2^2d_model ] 12: Update AKA_K using Adam and the INT2 STE 13: end for 14: R^K←RK0exp(AK−AK⊤) R_K← R_K^0 (A_K-A_K ) 15: Compute p^tK p_t^K using K~(R^K) K( R_K) 16: for i=1,…,Ti=1,…,T do 17: RV←RV0exp(AV−AV⊤)R_V← R_V^0 (A_V-A_V ) 18: Construct effective values V~(RV) V(R_V) through rotated INT2 quantization 19: eV(t,j)←[(p^tK)⊤(V~1:t(RV)−V1:t)]WO,j⊤e_V(t,j)← [( p_t^K) ( V_1:t(R_V)-V_1:t ) ]W_O,j 20: ℒV←(t,j)∈[‖eV(t,j)‖22dmodel]L_V _(t,j) [ \|e_V(t,j)\|_2^2d_model ] 21: Update AVA_V using Adam and the INT2 STE 22: end for 23: R^V←RV0exp(AV−AV⊤) R_V← R_V^0 (A_V-A_V ) 24: Store R^K,ℓ,h←R^K R_K, ,h← R_K and R^V,ℓ,h←R^V R_V, ,h← R_V 25:end for 26:return R^K,ℓ,h,R^V,ℓ,h,μℓ,hℓ,h\ R_K, ,h, R_V, ,h, _ ,h\_ ,h Here, ptp_t denotes the BF16 attention distribution, ptKp_t^K is computed using the current effective INT2 keys, and p^tK p_t^K is computed using the selected key rotation R^K R_K. The values V1:tV_1:t remain in BF16 during key calibration. The projection WO,jW_O,j maps the output of query head j into the model hidden space. Layer and KV-head indices are omitted inside the algorithm when clear. Details of Figure 2. We use the same 4,096-token AIME25 trace from Qwen3-8B for all five settings. The example is taken from layer 4, KV head 5, and query head 20. The top row shows the absolute key values immediately before INT2 quantization. The bottom row shows each cached key token’s contribution to the post-WOW_O error, aggregated over the final 64 query positions while keeping values in BF16 to isolate key-induced error. All panels use the same sample and INT2 configuration: group size 128, key clipping ratio 0.96, 64 BF16 sink tokens, and 256 BF16 recent tokens. Calibration details. We collect Q/K/V activation statistics from all 198 GPQA-Diamond prompts with one generated token, without using answer labels. During collection, each dynamically formed prefill batch is stored as a chunk containing aligned Q, K, V, and sequence-length tensors, yielding 14 chunks for each Qwen model and 15 for Phi-14B. From these dumps, we use chunks 3 and 4 for optimization and chunks 10 and 12 for held-out selection. After removing sequences shorter than 328 tokens, the calibration/held-out sets contain 6/5 prompts for Qwen3-4B, 7/6 for Qwen3-8B, and 9/3 for Phi-14B, corresponding to 2,554/2,526, 2,711/3,092, and 4,151/1,619 tokens, respectively. We compute the objective over the final 64 query positions and select the best rotation for each KV head using the held-out objective every 20 steps. Key-centering statistics are estimated from 28 full model-generated traces. The calibration and held-out chunks do not overlap, although their source prompts come from the GPQA-Diamond evaluation pool. Setting Configuration KV-cache precision Group-wise affine INT2 Quantization group size 128 Key clipping ratio 0.96 Value clipping ratio 0.92 BF16 sink window 64 tokens BF16 recent window 256 tokens Calibration data GPQA decoding traces Optimizer Adam Kingma and Ba [2014] Optimization steps 80 Learning rate 0.02 Table A: Default experimental settings for OptR. Experimental settings. We implement OptR on the official OSCAR codebase and adopt its best-performing INT2 configuration Zhou et al. [2026]. The default settings used throughout the main experiments and appendix are summarized in Table A. Unless explicitly stated, each ablation changes only the setting under investigation while keeping all others fixed. We measure efficiency on a single NVIDIA A100-SXM4 40GB GPU using Qwen3-4B-Thinking-2507 with tensor parallelism of one and an eight-token page size. Standard CUDA graphs are enabled for each evaluated batch size, while piecewise CUDA graphs are disabled. Each process performs an initial untimed warm-up. For every end-to-end and prefill setting, we additionally discard one complete warm-up run before recording five timed runs. Decode latency is measured over 1,024 generated tokens and excludes prefill time. End-to-end output throughput is computed as the number of generated tokens divided by the combined prefill and decoding time. We report the mean over five runs in the main figure and omit error bars for readability. The maximum standard deviations across all evaluated settings are 0.104 ms per token for decode latency, 0.412 tokens/s for end-to-end throughput, and 0.0115 s for prefill time. We use a custom SGLang implementation, PyTorch 2.9.1 with CUDA 12.8, and Triton 3.5.1. Key-mean estimation and application. The key mean is estimated from post-RoPE keys in the BF16 calibration traces: μℓ,h=1N∑n,skn,s,hℓ,RoPE _ ,h= 1N _n,sk_n,s,h ,RoPE (19) At inference, the same mean is subtracted from the post-RoPE keys before rotation and quantization. The cache-write kernel implements this operation as (ksRoPE−μℓ,h)RK,ℓ,h=ksRoPERK,ℓ,h−μℓ,hRK,ℓ,h (k_s^RoPE- _ ,h )R_K, ,h=k_s^RoPER_K, ,h- _ ,hR_K, ,h (20) We precompute μℓ,hRK,ℓ,h _ ,hR_K, ,h once and subtract it in the rotated space. BF16 sink and recent keys retain the same centering. Rotated-query implementation. For clarity, the formulation in the main paper maps an effective quantized key back with RK⊤R_K . Let kR,s=Q2((ks−μ)RK)k_R,s=Q_2\! ((k_s-μ)R_K ) (21) The corresponding effective key is k~s=kR,sRK⊤ k_s=k_R,sR_K . Its attention logit satisfies qtk~s⊤=qtRKkR,s⊤q_t k_s =q_tR_Kk_R,s (22) Therefore, the implementation stores kR,sk_R,s directly in the INT2 cache and applies RKR_K to the post-RoPE query instead of explicitly applying RK⊤R_K to every dequantized key. Under GQA, each query head uses the rotation of its corresponding KV head. The two implementations produce identical attention logits. Equivalent Inference Implementation. Figure 3 shows the baseline and output-aware correction rotations as separate operations. We denote their combined key and value rotations by RK=RkRkcRV=RvRvcR_K=R_kR_kc R_V=R_vR_vc (23) For the key path, the same RKR_K is applied to the post-RoPE query and centered key. Since RKR_K is orthogonal, the full-precision attention logit satisfies (qtRK)((ks−μ)RK)⊤=qt(ks−μ)⊤(q_tR_K) ((k_s-μ)R_K ) =q_t(k_s-μ) (24) Thus, the key rotations cancel in the full-precision inner product. The remaining term −qtμ⊤-q_tμ is constant across all cached tokens s and therefore leaves the softmax distribution unchanged. This equivalence holds before quantization. Under INT2, the stored key is kR,s=Q2((ks−μ)RK)k_R,s=Q_2\! ((k_s-μ)R_K ) (25) and the attention logit becomes (qtRK)kR,s⊤(q_tR_K)k_R,s Therefore, the rotation changes the coordinate system in which the INT2 error is introduced, although it does not change the full-precision attention computation. This implementation is equivalent to the inverse-rotation formulation in the main paper: qt(kR,sRK⊤)⊤=(qtRK)kR,s⊤q_t (k_R,sR_K ) =(q_tR_K)k_R,s (26) We therefore store kR,sk_R,s directly in the INT2 cache and apply RKR_K to each query head using the rotation of its corresponding KV head. For the value path, Figure 3 explicitly applies RvR_v and RvcR_vc before quantization and their inverses after attention. With RV=RvRvcR_V=R_vR_vc, the rotated value and attention output are vs′=vsRVoj′=ojRVv_s =v_sR_V o_j =o_jR_V (27) The explicit inverse path recovers the original output: oj′Rvc−1Rv−1WO,j⊤=ojWO,j⊤o_j R_vc^-1R_v^-1W_O,j =o_jW_O,j (28) In the implementation, these value-side transforms are folded into the projection weights. For KV head h and query head j∈Ghj∈ G_h, we use W¯V,h=RV,h⊤WV,hW¯O,j=WO,jRV,h W_V,h=R_V,h W_V,h W_O,j=W_O,jR_V,h (29) These weights satisfy xW¯V,h⊤ x W_V,h =(xWV,h⊤)RV,h, =(xW_V,h )R_V,h, (30) (ojRV,h)W¯O,j⊤ (o_jR_V,h) W_O,j =ojWO,j⊤ =o_jW_O,j (31) Hence, the value rotations and their inverses shown in Figure 3 are mathematical operations rather than separate inference kernels. Models and Benchmarks. We evaluate OptR across three reasoning-oriented language models that differ in model scale and architecture. This setting allows us to examine whether the proposed method remains effective across both compact and larger models, as well as across distinct model families. • Qwen3-4B-Thinking-2507 and Qwen3-8B Yang et al. [2025]: two Qwen3 models at different scales, used to assess whether the effectiveness of OptR is preserved as model scale increases. • Phi4-14B-reasoning-plus Abdin et al. [2025]: a reasoning model from a different model family, included to evaluate cross-architecture generalization. Together, these models cover parameter scales from 4B to 14B and include both within-family scaling and cross-family evaluation. We evaluate reasoning, coding, and long-context retrieval capabilities to measure the effect of INT2 KV-cache quantization across diverse inference workloads. • AIME24 and AIME25 MAA [2025]: challenging mathematical reasoning benchmarks that require multi-step problem solving. • GPQA-Diamond Rein et al. [2024]: a graduate-level scientific reasoning benchmark covering questions that require specialized knowledge and careful reasoning. • MBPP+ Liu et al. [2023]: a code-generation benchmark with extended test cases for more rigorous functional-correctness evaluation. • LiveCodeBench v6 Jain et al. [2025]: a contamination-resistant coding benchmark constructed from recent programming problems. • RULER-NIAH Hsieh et al. [2024]: a long-context retrieval benchmark used to evaluate whether quantized KV caches preserve information over extended context lengths. The reasoning and coding benchmarks evaluate the quality of generated outputs, whereas RULER-NIAH isolates long-context retrieval performance under increasing context lengths. Generation Settings. We use the same sampling configuration across all models and benchmarks to ensure a consistent comparison. Specifically, we set the temperature to 0.60.6, top-p to 0.950.95, and top-k to 2020. For the reasoning and coding benchmarks, we use maximum generation lengths of 3232K tokens for the Qwen3 models and 1616K tokens for Phi4-14B-reasoning-plus. Appendix B Output-Error Measurement and Analysis This section describes the measurement settings for the per-layer line plots, layer-averaged bar plots, and per-head heatmaps. The main paper reports the Qwen3-4B-Thinking-2507 per-layer analysis and GPQA-Diamond heatmap in Figures 4 and 5, respectively. Additional per-layer and layer-averaged results for Qwen3-4B-Thinking-2507 and Qwen3-8B are provided in Figures B–E, while the complete per-head heatmaps across both models and datasets are shown in Figure A. Unless otherwise specified, all analyses use the same INT2 quantization settings as the main experiments. Output-space error. For each layer ℓ , KV head h, and query head j∈hj _h, we use the last 6464 query positions of each sequence. Let ptℓ,j,hp_t ,j,h and p~tℓ,j,h p_t ,j,h denote the causal attention distributions obtained with full-precision and INT2 keys, respectively. We write V1:t,hℓV_1:t,h and V~1:t,hℓ V_1:t,h for the full-precision and effective INT2 value matrices, where the latter includes full-precision sink- and recent-token restoration. Following the notation in the main paper, the post-WOW_O output discrepancy is Δyt,jℓ=[(p~tℓ,j,h)⊤V~1:t,hℓ−(ptℓ,j,h)⊤V1:t,hℓ](WO,jℓ)⊤ y_t,j = [ ( p_t ,j,h ) V_1:t,h - (p_t ,j,h ) V_1:t,h ] (W_O,j ) This quantity is the total output error Δyt,jℓ=y~t,jℓ−yt,jℓ y_t,j = y_t,j -y_t,j and jointly captures key-induced changes in the attention distribution and value-induced changes in the attention-weighted output. For each layer, we average ‖Δyt,jℓ‖22\| y_t,j \|_2^2 over query positions, query heads, KV heads, and sequences. We apply the same group-wise INT2 quantization and dequantization path used during inference to the KV cache of the measured layer while using full-precision input activations. The resulting values measure the immediate error introduced by each attention block. The corresponding per-layer results are reported in Figures 4, B, and D. Qwen3-4B-Thinking - GPQA-Diamond Qwen3-8B - GPQA-Diamond Qwen3-4B-Thinking - AIME-25 Qwen3-8B - AIME-25 Figure A: Output error by layer and KV head relative to naive INT2 across models and evaluation datasets. Each cell reports the post-WOW_O attention-output RMS error as a percentage of plain per-group INT2 without rotation. Lower values indicate smaller output error, with naive INT2 corresponding to 100%100\%. Within each heatmap, panels show (a) TurboQuant, (b) QuaRot, (c) QuaRot + OptR, (d) OSCAR, and (e) OSCAR + OptR. Figure B: Per-layer error analysis under INT2 KV-cache quantization on Qwen3-4B-Thinking. Panels (a,b) report the post-WOW_O output-space MSE on GPQA-Diamond and AIME25, capturing the immediate error introduced by each attention block. Panels (c,d) report the propagated residual-stream error, computed as the normalized squared difference between full-precision and INT2-KV block-output hidden states evaluated on the same full-precision-generated sequences. Values in the legend denote averages across layers. All errors are shown on a logarithmic scale. Figure C: Layer-averaged error under INT2 KV-cache quantization on Qwen3-4B-Thinking. Panels (a,b) report the mean post-WOW_O output-space MSE across Transformer layers on GPQA-Diamond and AIME25, respectively. Panels (c,d) report the corresponding mean propagated residual-stream error. The values summarize the per-layer results shown in Figure B. Lower values indicate smaller quantization-induced error. Figure D: Per-layer error analysis under INT2 KV-cache quantization on Qwen3-8B. Panels (a,b) report the post-WOW_O output-space MSE on GPQA-Diamond and AIME25, capturing the immediate error introduced by each attention block. Panels (c,d) report the propagated residual-stream error, computed as the normalized squared difference between full-precision and INT2-KV block-output hidden states evaluated on the same full-precision-generated sequences. Values in the legend denote averages across layers. All errors are shown on a logarithmic scale. Figure E: Layer-averaged error under INT2 KV-cache quantization on Qwen3-8B. Panels (a,b) report the mean post-WOW_O output-space MSE across Transformer layers on GPQA-Diamond and AIME25, respectively. Panels (c,d) report the corresponding mean propagated residual-stream error. The values summarize the per-layer results shown in Figure D. Lower values indicate smaller quantization-induced error. Propagated residual-stream error. To measure the accumulation of quantization error across depth, we run the complete model with INT2 KV caches in all Transformer layers. For each of 1212 prompts per dataset, the full-precision model first generates a reasoning trace greedily, with up to 15361536 new tokens and a maximum sequence length of 30723072. The full-precision and INT2 models are then evaluated on the same generated token sequence. This teacher-forced comparison isolates representational drift from differences caused by autoregressive sampling. All model weights remain in BF16. Long-history keys and values are quantized and stored in INT2 at cache-write time and dequantized during attention, while the configured sink- and recent-token windows remain in full precision. Let hℓ,th_ ,t and h~ℓ,t h_ ,t denote the full-precision and INT2-KV residual streams after block ℓ at token position t. We report the relative squared error ∑t‖h~ℓ,t−hℓ,t‖22∑t‖hℓ,t‖22 _t\| h_ ,t-h_ ,t\|_2^2 _t\|h_ ,t\|_2^2 pooled over all tokens and prompts. This normalization accounts for changes in the residual-stream magnitude across depth and enables comparison across layers. The propagated errors across model depth are shown in panels (c,d) of Figures 4, B, and D. Layer-averaged summaries. The bar plots summarize the corresponding per-layer line plots by reporting the arithmetic mean of each error metric across Transformer layers. Panels (a,b) report the mean post-WOW_O output-space MSE on GPQA-Diamond and AIME25, while panels (c,d) report the corresponding mean propagated residual-stream error. The Qwen3-4B-Thinking-2507 and Qwen3-8B summaries are reported in Figures C and E, respectively. These plots provide an aggregate comparison between methods, whereas Figures B and D show how the same errors vary across model depth. Per-head error relative to naive INT2. The heatmaps use the same post-WOW_O output error Δyt,jℓ y_t,j , but retain a separate value for each (layer,KV head)(layer,KV head) pair rather than averaging over KV heads. Let errℓ,hmethoderr_ ,h^method denote the mean squared output error for a given cell and let errℓ,hINT2err_ ,h^INT2 denote the corresponding error under plain per-group INT2 without rotation or clipping. Each cell reports 100errℓ,hmethod/errℓ,hINT2100 err_ ,h^method/err_ ,h^INT2, representing the RMS output error as a percentage of naive INT2. Thus, naive INT2 corresponds to 100%100\%, and lower values indicate smaller output error. The heatmaps use a logarithmic scale ranging from 3%3\% to 100%100\%. The GPQA-Diamond and AIME25 heatmaps are shown in Figures A. Results. Figures B and C show that OptR reduces both immediate post-WOW_O output error and propagated residual-stream error on Qwen3-4B-Thinking-2507. On GPQA-Diamond, OptR reduces the average post-WOW_O output MSE from 0.830.83 to 0.230.23 for QuaRot and from 0.410.41 to 0.230.23 for OSCAR. On AIME25, the corresponding errors decrease from 3.413.41 to 1.911.91 and from 2.522.52 to 1.891.89, respectively. The propagated residual-stream error decreases from 0.3660.366 to 0.1460.146 for QuaRot and from 0.1650.165 to 0.1070.107 for OSCAR on GPQA-Diamond, and from 0.3360.336 to 0.1550.155 and from 0.1580.158 to 0.1160.116 on AIME25. The same trend holds for Qwen3-8B, as shown in Figures D and E. On GPQA-Diamond, OptR reduces the average post-WOW_O output MSE from 2.312.31 to 0.900.90 for QuaRot and from 1.731.73 to 0.870.87 for OSCAR. On AIME25, the corresponding errors decrease from 8.888.88 to 4.004.00 and from 7.027.02 to 4.294.29. The propagated residual-stream error decreases from 0.1950.195 to 0.1270.127 for QuaRot and from 0.1470.147 to 0.1090.109 for OSCAR on GPQA-Diamond. On AIME25, it decreases from 0.2100.210 to 0.1310.131 for QuaRot and from 0.1350.135 to 0.1320.132 for OSCAR. Finally, Figure A shows that the error reductions extend across most layers and KV heads on both GPQA-Diamond and AIME25. The improvements are therefore broadly distributed throughout the models rather than being concentrated in a small subset of layers or heads. Tokens OSCAR OSCAR + OptR 3.13.1k 62.00±6.5062.00± 6.50 69.33±3.6569.33± 3.65 6.66.6k 63.33±4.0863.33± 4.08 70.67±2.7970.67± 2.79 12.612.6k 62.67±5.9662.67± 5.96 70.00±5.7770.00± 5.77 19.219.2k 64.00±3.6564.00± 3.65 70.67±4.3570.67± 4.35 26.026.0k 62.67±6.4162.67± 6.41 70.00±4.0870.00± 4.08 Table B: Effect of calibration size on AIME25 accuracy for Qwen3-4B-Thinking-2507. Both methods use the same GPQA calibration tokens and a BF16 window of S=64/R=256S=64/R=256. Results are μ±σμ±σ over 5 seeds. B.1 Ablation on Calibration Size Table B evaluates the sensitivity of OSCAR and OptR to the number of calibration tokens. OptR consistently improves OSCAR across all calibration sizes, while its performance remains stable from a few thousand tokens onward. Increasing the calibration size beyond 6.66.6k tokens provides no consistent additional gain, indicating that OptR requires only a modest calibration set. We therefore use 6.66.6k tokens as the default configuration. B.2 Ablation on λK _K Table C studies the sensitivity of OptR to the key-loss weight λK _K. The best value depends on the base rotation: OSCAR + OptR performs best at λK=1 _K=1, whereas QuaRot-INT2 + OptR achieves its highest accuracy at λK=10 _K=10. λK _K QuaRot-INT2 + OptR OSCAR + OptR 0.10.1 60.67±5.4860.67± 5.48 66.67±4.0866.67± 4.08 0.30.3 62.00±4.3562.00± 4.35 63.33±5.2763.33± 5.27 !15 1 60.67±4.3560.67± 4.35 70.67±2.7970.67± 2.79 33 60.67±4.3560.67± 4.35 63.33±4.7163.33± 4.71 1010 63.33±4.0863.33± 4.08 60.67±2.7960.67± 2.79 Table C: Effect of λK _K on AIME25 accuracy for Qwen3-4B-Thinking-2507. Results are μ±σμ±σ over five seeds, and the shaded row denotes the default setting. This difference indicates that the appropriate balance between attention-distribution preservation and post-WOW_O output-error reduction depends on the rotation initialization. To avoid base-specific tuning, we use λK=1 _K=1 throughout the main experiments, which gives the strongest performance with OSCAR, our state-of-the-art rotation baseline. Model Calibration Time (Minute) Qwen3-4B-Thinking-2507 6.25 Qwen3-8B 7.05 Phi4-14B-Reasoning-Plus 11.03 Table D: Calibration time of OptR with 80 optimization steps. Measurements use a single NVIDIA A100 GPU and exclude BF16 trace collection. All model weights remain frozen throughout calibration. Offline calibration cost. Table D reports the one-time offline calibration cost of OptR. Calibration requires only several minutes on a single GPU across the evaluated model sizes. All model weights remain frozen, and the resulting rotations are reused across subsequent inference requests, amortizing this cost during deployment. Model Method 4k 8k 16k 32k 64k 128k Qwen3-8B BF16 99.83±0.1199.83± 0.11 99.93±0.0299.93± 0.02 99.45±0.0899.45± 0.08 98.70±0.4998.70± 0.49 84.22±1.4184.22± 1.41 80.89±0.5880.89± 0.58 QuaRot-INT2 84.97±0.0984.97± 0.09 41.73±5.7541.73± 5.75 18.16±1.1418.16± 1.14 13.03±1.7113.03± 1.71 0.04±0.070.04± 0.07 0.38±0.130.38± 0.13 QuaRot-INT2 + OptR 99.40±0.2999.40± 0.29 98.76±0.5798.76± 0.57 96.14±0.3596.14± 0.35 86.37±0.7286.37± 0.72 70.02±1.7970.02± 1.79 41.87±0.9641.87± 0.96 OSCAR 99.59±0.1599.59± 0.15 97.94±0.2897.94± 0.28 94.39±0.2994.39± 0.29 83.76±0.5883.76± 0.58 57.54±1.8257.54± 1.82 26.44±0.8726.44± 0.87 OSCAR + OptR 99.60±0.1199.60± 0.11 98.16±0.6398.16± 0.63 95.50±0.6595.50± 0.65 86.42±0.5586.42± 0.55 68.65±0.8068.65± 0.80 40.43±1.5340.43± 1.53 Table E: RULER-NIAH retrieval accuracy of Qwen3-8B at context lengths up to 128K tokens. Results are μ±σμ±σ over three seeds, with 800 examples per seed. Bold denotes the best INT2 result at each context length. Long-Context Results Table E extends the Qwen3-8B long-context evaluation to 128K tokens. At this context length, QuaRot-INT2 and OSCAR degrade substantially, whereas applying OptR retains considerably higher retrieval accuracy. The consistent gains with both base rotations show that OptR remains effective beyond the 64K range reported in the main paper.