Paper deep dive
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a quality-fidelity trade-off, either suffering from performance instability at 2-bit precision or necessitating 3-bit keys to maintain model quality. In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E_8$ lattice for KV keys. We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate. To ensure practical utility, we adapt a table-based asymmetric scoring algorithm that avoids explicit codebook dequantization during attention computation and preserves decode-throughput parity with scalar baselines. Across five LLM families, our $E_8$ k2v2 approach yields \textasciitilde$6\times$ KV compression, roughly 20\% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity. In our full-suite perplexity evaluation, $E_8$ improves over scalar k3v2 in 80\% of model--dataset--context settings. These results indicate that $E_8$ lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks. Overall, our work provides a low-overhead primitive for expanding the quality-memory frontier in long-context LLM inference.
Tags
Links
- Source: https://arxiv.org/abs/2610.15384v1
- Canonical: https://arxiv.org/abs/2610.15384v1
Trouble viewing inline? Open PDF directly →
Full Text
44,984 characters extracted from source content.
Expand or collapse full text
SLMs for Agentic Systems, Paris, France, 2026 (extended version) E8 Lattice Key Quantization for Low-Bit KV-Cache Compression Sasho Nedelkoski Affiliation: Technical University of Berlin Affiliation: Berlin, Germany Email: sasho.nedelkoski@campus.tu-berlin.de Alexander Acker Affiliation: logsight.ai Affiliation: Berlin, Germany Email: alexander.acker@logsight.ai Soeren Becker Affiliation: logsight.ai Affiliation: Berlin, Germany Email: soeren.becker@logsight.ai Dominik Scheinert Affiliation: logsight.ai Affiliation: Berlin, Germany Email: dominik.scheinert@logsight.ai Odej Kao Affiliation: Technical University of Berlin Affiliation: Berlin, Germany Email: odej.kao@tu-berlin.de Abstract Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scalar quantization methods achieve significant compression, they typically face a quality-fidelity trade-off, either suffering from performance instability at 2-bit precision or necessitating 3-bit keys to maintain model quality. In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the E8E_8 lattice for KV keys. We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate. To ensure practical utility, we adapt a table-based asymmetric scoring algorithm that avoids explicit codebook dequantization during attention computation and preserves decode-throughput parity with scalar baselines. Across five LLM families, our E8E_8 k2v2 approach yields ~6×6× KV compression, roughly 20% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity. In our full-suite perplexity evaluation, E8E_8 improves over scalar k3v2 in 80% of model–dataset–context settings. These results indicate that E8E_8 lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks. Overall, our work provides a low-overhead primitive for expanding the quality-memory frontier in long-context LLM inference. 1 Introduction Supporting long context windows in large language models (LLMs) is essential for in-context learning, long-horizon reasoning, holistic codebase analysis, and retrieval-augmented generation (Wang et al., 2024). LLM inference requires storing key-value (KV) pairs for every token in the context window across all attention layers and heads, placing high pressure on available device memory (Li et al., 2025a). For a model with L layers, HkvH_kv KV heads, and head dimension d, the KV cache consumes 2⋅L⋅Hkv⋅d⋅22· L· H_kv· d· 2 bytes per token in fp16. At 128K tokens of context, even an 8B grouped-query model requires tens of gigabytes of KV cache, and models with more KV heads can exceed the memory footprint of the model weights themselves. This bottleneck limits batch size and context length, leading to prohibitive deployment cost. Consequently, KV-cache efficiency directly affects whether long-context serving fits on a single accelerator or requires multi-GPU execution. To mitigate these memory constraints, KV cache quantization has emerged as a prominent solution (Li et al., 2025a). Recent KV-cache quantization approaches, including KIVI (Liu et al., 2024), KVQuant (Hooper et al., 2024), and TurboQuant (Zandieh et al., 2025), rely primarily on per-channel scalar quantization. Treating each of the key dimensions d as an independent signal, a quantization of b bits assigns each dimension to one of the 2b2^b levels, achieving a rate of b bits/dim. However, scalar quantization often incurs quality degradation at very low bitrates. Recent hardware-optimized configurations typically necessitate 3 bits for keys (e.g., TurboQuant’s k3v2 setting: 3-bit keys and 2-bit values) to preserve model quality. We hypothesize that the degradation of low-bit scalar key quantization is partly due to limited use of geometric structure in key vectors. While scalar methods are computationally efficient, they treat each dimension in isolation, ignoring the inter-dimensional correlations prevalent in LLM hidden states (Engels et al., 2025). To capture this structure, we use lattice VQ over blocks of dimensions to exploit space-filling and shaping gains, which can provide higher SQNR than scalar quantization at matched bit rates (Zamir and Feder, 1996). Specifically, we employ the E8E_8 lattice in ℝ8R^8, which offers a useful combination of strong lattice geometry and practical efficiency. It achieves the densest possible sphere packing in eight dimensions (Viazovska, 2017), with near-spherical Voronoi cells that provide a shaping advantage over cubic scalar quantizers. Our implementation uses a structured E8E_8-derived (E8P12-style) indexing scheme with parity/sign constraints and a two-branch ±0.25± 0.25 selection rule, enabling efficient nearest-codeword assignment without brute-force search over all codewords. Contributions. This paper makes the following contributions: 1. We apply E8E_8 lattice VQ to key compression in the KV cache, partitioning each d-dimensional key vector into independent 8-dimensional blocks. At 2 bits/dim payload (65,536 codewords per block; 2.25 bits/dim effective key rate including norm metadata), the resulting k2v2 configuration achieves ~6×6× total KV compression relative to fp16. Reported memory numbers are end-to-end KV-cache bytes/token, counting keys, values, quantized indices/signs, norms, and value scale/zero metadata. 2. We show that E8E_8 at low key rate achieves rate-distortion performance between scalar 3-bit and 4-bit quantization. On 128-dimensional Gaussian vectors, E8E_8 obtains 10.47 dB SQNR, compared with 7.37 dB for scalar 3-bit, while using a lower effective key rate. In a controlled attention-fidelity probe, the 2-bit E8E_8 configuration also yields lower average attention KL divergence than scalar k3v2 (0.045 vs. 0.091) and KIVI k2v2 (0.224), indicating better preservation of fp16 attention distributions under compression. 3. To avoid dequantizing vector-quantized keys during decoding, we adapt table-based asymmetric scoring to the fixed E8E_8 key codebook. For each query-head block, we precompute the dot product between the query block and all 65,53665,536 E8E_8 codewords. Each compressed key block can then be scored by indexing this table with its stored codeword id, replacing an 8-dimensional dot product with a lookup. This removes explicit key dequantization from the decode-time scoring path in our serving path and matches the throughput of optimized scalar decoders in the measured setting. 4. We provide full-suite evaluations across five LLMs (Qwen2.5, Llama-3.1, Mistral, Falcon3, and Granite). The results show that E8E_8 k2v2 improves the quality–memory trade-off over scalar k3v2 in most long-context perplexity settings and has lower perplexity than KIVI k2v2 across the evaluated perplexity suite, without a measurable decode-throughput penalty. 2 Related work KV-cache quantization. Current KV-cache compression methods primarily use scalar quantization. KIVI (Liu et al., 2024) uses per-channel asymmetric key quantization and per-token value quantization; KVQuant (Hooper et al., 2024) adds pre-RoPE keys, non-uniform calibrated grids, and sparse outlier handling; GEAR (Kang et al., 2024) combines low-rank approximation with quantized residuals; TurboQuant (Zandieh et al., 2025) uses QJL random projections for calibration-free robustness, and its k3v2 setting is our primary scalar baseline. ZipCache (He et al., 2024), LeanKV (Zhang et al., 2025b), and AQUA-KV (Shutova et al., 2025) further adapt bit-widths across tokens, keys/values, or predictable residual components. These methods are effective, but the dominant coding operation remains scalar or projection-based; our work instead encodes each key block with a structured geometry-aware codebook. Joint coding and vector quantization. Coupled Quantization (CQ) (Zhang et al., 2024) observes that joint KV-channel entropy can be far below the sum of marginal entropies, motivating cross-channel coding; PM-KVQ (Liu et al., 2025) highlights the need for long-context-aware calibration under RoPE. Recent VQ-based KV-cache systems, including NestQuant (Savkin et al., 2025) and CommVQ (Li et al., 2025b), further show that lattice or learned vector codebooks can improve low-bit cache compression. Our method shares the premise that key channels should not be treated independently, but differs by using a fixed, non-learned E8E_8 key codebook with online encoding and by adding decode-time table scoring so attention can operate directly on compressed keys. Related VQ work for weights, including PQ (Jegou et al., 2011; Guo et al., 2020), QuIP (Chee et al., 2023), QuIP# (Tseng et al., 2024), and AQLM (Egiazarian et al., 2024), shows that structured codebooks can achieve strong low-bit rate-distortion. KV caches impose different constraints: activations are produced online during decoding and attention scores must be computed against a growing cache, so avoiding dense key reconstruction is central. Token reduction and attention approximation. Token-eviction systems reduce cache length rather than bits per token. StreamingLLM (Xiao et al., 2024) retains recent tokens and attention sinks; H2O (Zhang et al., 2023) and SnapKV (Li et al., 2024) retain tokens selected by accumulated or pooled attention; MagicPIG (Chen et al., 2024) uses LSH and CPU-resident hash tables for sparse long-context attention. Projection methods such as Loki (Singhania et al., 2024) reduce effective key dimensionality before attention. These approaches are orthogonal to our quantizer: one could apply E8E_8 key coding to the retained token set or to a projected subspace. Lattice codes. Lattice quantization theory (Zamir and Feder, 1996; Conway and Sloane, 1999) shows that structured lattices achieve space-filling gain that scalar quantizers cannot. The E8E_8 lattice is the densest known sphere packing in eight dimensions (Viazovska, 2017); its 240 kissing neighbors and near-spherical Voronoi cells give a shaping advantage over cubic lattices and motivate its use in low-bit neural compression. To our knowledge, this is the first application of E8E_8 coding to runtime KV caches with online encoding and throughput-neutral compressed-domain attention scoring. 3 Method 3.1 Preliminary baseline As a point of comparison, standard scalar KV-cache quantization independently quantizes each key dimension. Consider a key vector ∈ℝdk ^d, where d is the attention head dimension. Scalar quantization at b bits independently quantizes each dimension: k^i=s⋅round(kis),s=maxi|ki|2b−1−1, k_i=s·round\! ( k_is ), s= _i|k_i|2^b-1-1, (1) storing b⋅db· d key-index bits plus scale metadata. Eq. (1) is a generic per-vector scalar-quantization reference used to introduce notation; concrete KV baselines may use per-channel, per-token, or groupwise scaling. Our implemented scalar baseline (TurboQuant k3v2) uses rotated unit-norm keys with a Lloyd-Max MSE stage plus a 1-bit QJL residual-sign stage. Throughout the paper, we assume that d is divisible by 8, and as our running example we choose d=128d=128 to make the explanation clear. However, the framework generalizes to arbitrary head dimensions satisfying this constraint. Compression ratios refer to effective KV-cache bytes per token, not nominal key bitwidth alone. A complete fp16 KV cache stores one key and one value vector per token-head, requiring 2⋅d⋅162· d· 16 bits. As an example, for d=128d=128, this is 40964096 bits per token-head. As an example, the TurboQuant k3v2 (Zandieh et al., 2025) setting stores 3d3d packed key bits and 2d2d packed value bits, plus metadata for dequantization and compressed-domain scoring: BS3=3d+2d+mS3.B_S3=3d+2d+m_S3. End-to-end accounting includes packed keys, packed values, and per-token metadata such as QJL sign bits, norms, and value scale/zero parameters; it excludes fixed model-level state such as rotation matrices and shared codebooks. According to this convention, the k3v2 scalar yields about ∼4.8× 4.8× compression over fp16, as shown in Table 2 below. 3.2 E8 lattice vector quantization for keys The E8E_8 lattice ΛE8⊂ℝ8 _E_8 ^8 is defined as ΛE8=∈ℤ8∪(ℤ+12)8|∑i=18xi≡0(mod2). _E_8= \x ^8∪(Z+ 12)^8\; |\; _i=1^8x_i≡ 0 2 \. (2) The full E8E_8 lattice underlies the optimal sphere packing in 8D space (Viazovska, 2017), making it a natural high-dimensional quantization template. We use a fixed, non-learned E8E_8-derived codebook ⊂ℝ8C ^8 with 216=65,5362^16=65,536 codewords, so each 8-dimensional key block is represented by one 16-bit index. The codebook is built from an E8E_8-consistent half-integer grid augmented with norm-12 patterns. Its indexing structure separates an 8-bit absolute-value prototype from an 8-bit sign/parity pattern, enabling deterministic encode/decode without brute-force search over all codewords. During encoding, we evaluate two ±0.25± 0.25 shifted lattice branches and store the lower-error index. This work focuses on the fixed E8E_8 key codec itself and does not add a separate sparse-outlier retention path. Such outlier handling is complementary and left for future work. Encoding. To quantize a key vector ∈ℝdk ^d, the method applies a random orthogonal rotation matrix ∈ℝd×dR ^d× d (shared across all keys in a layer and fixed at initialization) to spread energy across coordinates and mitigate channel-aligned outliers while preserving L2L_2 distances. Following the assumption that d is a multiple of 88 (which is the case in most modern LLMs (Wang et al., 2025)), we then partition the rotated vector into M=d8M= d8 discrete 8-dimensional blocks. For d=128d=128, this gives M=16M=16 blocks: ~=,~=[~(1),…,~(16)],~(j)∈ℝ8. k=Rk, k=[ k^(1),…, k^(16)], k^(j) ^8. (3) Each block is scaled and quantized to its nearest E8E_8 codeword: zj=argmini∈0,…,||−1‖α‖~‖2~(j)−i‖22,j=1,…,16,z_j= _i∈\0,…,|C|-1\ \| α\| k\|_2 k^(j)-g_i \|_2^2, j=1,…,16, (4) where α is a calibrated scale parameter. In all main experiments, α is selected deterministically by minimizing block reconstruction MSE on 300 seeded Gaussian unit vectors using a coarse scale grid followed by local refinement; for d=128d=128 this gives α≈12α≈ 12. The compressed key is stored as 16×1616× 16 index bits plus one 32-bit norm ‖~‖2\| k\|_2, totaling 16×16+32=28816× 16+32=288 bits =36=36 bytes, yielding a rate of 288/128=2.25288/128=2.25 bits/dim. Effective rate. For d=128d=128, the raw packed E8E_8 k2v2 payload is 288288 key bits plus 2d=2562d=256 value bits, or 544544 bits per token-head before value metadata. Under the accounting convention above, this gives the ∼6× 6× end-to-end KV-cache compression over fp16 (Table 2). 3.3 Table-based asymmetric scoring A naive implementation of attention with lattice VQ keys would dequantize each key by looking up its M codewords from the E8E_8 grid, reconstructing a dense d-dimensional vector, then computing the dot product with the query. For N cached keys, this requires NMNM random accesses to a 65,536×865,536× 8 float32 table—a memory bandwidth bottleneck that typically renders lattice VQ slower than scalar quantization. We instead use table-based asymmetric scoring, following asymmetric-distance computation in product quantization (Jegou et al., 2011) and recent lookup-based KV-attention systems (Zhang et al., 2025a; Wu et al., 2025; Karmore, 2026). Given a query ∈ℝdq ^d for a single decoding step: 1. Rotate and partition: We apply the same orthogonal rotation to the query ~= q=Rq, and partition it into M blocks ~(j)∈ℝ8 q^(j) ^8 (e.g., M=16M=16 for d=128d=128). 2. Pre-compute lookups: We build M lookup tables: Tj[i]=⟨~(j),i⟩T_j[i]= q^(j),g_i for i=0,…,65,535i=0,…,65,535, where ig_i is the i-th E8E_8 codeword. Each of the M tables contains 65,53665,536 float32 entries. 3. Asymmetric scoring: For each cached key stored as M indices (z1,…,zM)(z_1,…,z_M) and norm n=‖~‖2n=\| k\|_2, we compute the attention score using the calibrated parameter α from Eq. (4): score=nα∑j=1MTj[zj].score= nα _j=1^MT_j[z_j]. (5) Computational efficiency. Step 2 costs M×65,536M× 65,536 8-dimensional dot products, implemented as a batched matrix multiplication over query blocks and codewords. Step 3 costs NMNM table gathers plus NMNM additions. The M lookup tables require M×65,536×4M× 65,536× 4 bytes of memory. For M=16M=16, this totals 44 MB per query head. Because each gather is a single indexed read from a contiguous query-local table, per-key scoring is memory-efficient in our decode setting; only the once-per-query table build scales with codebook size. For batched teacher-forcing quality evaluations (perplexity and downstream tasks), we use the same compressed representation and asymmetric inner-product estimator. These batched evaluations use a mathematically equivalent dense reconstruction when processing many query positions at once; the decode throughput results in Table 3 exercise the table-based serving path directly. 3.4 Implementation details We integrate E8E_8 key quantization into the TurboQuant (Zandieh et al., 2025) framework, retaining its value quantization and residual window for recently inserted tokens. The reported E8E_8 k2 key path does not use QJL residual signs; QJL is retained only in scalar TurboQuant baselines. The random orthogonal rotation matrix ∈ℝd×dR ^d× d is initialized once per layer from a seeded PRNG. For d=128d=128, storing this dense matrix in fp32 requires 64 KB per layer (or 32 KB in fp16), which is a negligible model-level overhead in our accounting. Encoding a single key requires one matrix-vector multiply followed by M fast structured nearest-codeword searches over 8D blocks. Rather than brute-force O(||)O(|C|) VQ lookup, the implementation first rounds in a reduced absolute-value candidate set, then applies sign/parity-consistent index reconstruction with the ±0.25± 0.25 branch selection. As shown in the rate-distortion analysis in Figure 1, this integration yields an E8E_8 configuration whose SQNR lies between scalar 3-bit and 4-bit baselines while remaining in the low-bit key-rate regime. 4 Experiments 4.1 Setup Models. We evaluate on five instruction-tuned LLMs spanning different architectures and labs: Qwen2.5-14B (Qwen Team et al., 2025), Mistral-7B (Jiang et al., 2023), Llama-3.1-8B (Grattafiori et al., 2024), Falcon3-7B (Almazrouei et al., 2023), and Granite-3.1-8B (Stallone et al., 2024). All models use grouped-query attention with head dimension divisible by 8 and varying numbers of KV heads. Configurations. We compare four KV cache configurations: (1) Baseline: fp16 keys and values; (2) KIVI k2v2 (Liu et al., 2024); (3) Scalar k3v2 (TurboQuant (Zandieh et al., 2025)); and (4) our E8 Lattice k2v2 setting. All compressed configurations use 2-bit values and a recent-token residual buffer of 128 tokens unless explicitly varied in residual-window sweeps. Figure 1: Rate-distortion comparison on 128-dimensional Gaussian vectors. E8E_8 lattice VQ achieves SQNR between scalar 3-bit and 4-bit quantization at a comparable low key rate. Metrics. To provide a holistic assessment, we categorize our evaluation into three axes: (1) End-to-end quality: long-context perplexity on WikiText-2 (Merity et al., 2016) and C4 (Dodge et al., 2021) (contexts 512, 2048, 4096), downstream few-shot accuracy on HellaSwag (10-shot) (Zellers et al., 2019), ARC-Challenge (25-shot) (Clark et al., 2018), and WinoGrande (5-shot) (Sakaguchi et al., 2019), alongside a needle-in-a-haystack (NIAH) retrieval sanity check. (2) Internal fidelity: per-layer cosine similarity between compressed and fp16 key representations, and KL divergence for attention distributions. (3) System performance: single-sequence decode throughput (tokens/s) on a single 80 GB NVIDIA A100 GPU and total memory consumption. We report single-system, single-sequence serving measurements; larger-batch scheduling behavior, cross-hardware variance, and cross-backend variance are left to future work. 4.2 Rate-distortion analysis Figure 1 shows rate-distortion curves for our implemented TurboQuant-style scalar/product key baseline and the E8E_8 lattice key quantizer on 128-dimensional Gaussian vectors. At the low-rate operating point, E8E_8 achieves SQNR of 10.47 dB versus 2.43 dB for the 2-bit scalar/product baseline. More practically, E8E_8 at 2 bits/dim (SQNR 10.47 dB, cosine similarity 0.954) falls between the implemented scalar 3-bit (7.37 dB, 0.920) and 4-bit (12.69 dB, 0.974) configurations. This comparison supports the empirical quality-per-bit advantage of the fixed E8E_8 key codec in our compression stack; it should not be read as an optimal scalar-quantizer lower bound. 4.3 Perplexity Table 1: Perplexity on WikiText-2 and C4 across context lengths. Each block reports compressed-method perplexity deltas against fp16 baseline; lower is better. Bold marks the best compressed configuration within each dataset/model/context. Model Ctx WikiText-2 C4 Base KIVI Δ Scalar Δ E8 Δ Base KIVI Δ Scalar Δ E8 Δ Qwen2.5-14B 512 5.90 7.16 +1.26 6.29 +0.39 6.13 +0.23 11.11 12.47 +1.36 11.59 +0.48 11.35 +0.24 2K 5.01 7.32 +2.31 5.61 +0.60 5.29 +0.28 10.32 13.82 +3.50 11.31 +0.99 10.60 +0.29 4K 4.84 7.88 +3.04 5.51 +0.68 5.13 +0.29 10.18 15.38 +5.20 11.59 +1.41 10.47 +0.29 Llama-3.1-8B 512 7.33 9.40 +2.07 7.70 +0.36 7.51 +0.18 11.34 15.00 +3.65 11.93 +0.59 11.71 +0.37 2K 6.45 15.91 +9.45 6.97 +0.51 6.67 +0.21 10.90 32.72 +21.82 11.84 +0.94 11.35 +0.45 4K 6.30 21.47 +15.17 6.90 +0.60 6.54 +0.23 10.88 47.43 +36.54 12.05 +1.17 11.40 +0.52 Mistral-7B 512 5.75 6.25 +0.50 5.85 +0.10 5.81 +0.07 8.87 9.51 +0.64 8.99 +0.12 8.93 +0.06 2K 4.98 5.93 +0.94 5.12 +0.14 5.06 +0.07 8.21 9.72 +1.51 8.42 +0.21 8.28 +0.07 4K 4.84 6.12 +1.27 5.00 +0.16 4.92 +0.08 8.10 10.25 +2.15 8.35 +0.25 8.18 +0.08 Falcon3-7B 512 8.91 9.40 +0.49 8.87 -0.04 9.08 +0.17 17.85 18.53 +0.68 17.89 +0.04 17.96 +0.11 2K 7.19 7.79 +0.60 7.08 -0.11 7.30 +0.11 15.94 17.12 +1.19 15.82 -0.11 15.98 +0.04 4K 6.63 7.16 +0.53 6.57 -0.06 6.72 +0.09 15.16 16.67 +1.51 15.19 +0.03 15.24 +0.08 Granite-3.1-8B 512 5.53 6.68 +1.15 5.90 +0.36 5.69 +0.16 8.49 9.85 +1.36 8.85 +0.36 8.68 +0.19 2K 4.97 7.22 +2.25 5.52 +0.55 5.19 +0.22 8.25 11.57 +3.33 8.98 +0.74 8.59 +0.35 4K 4.89 8.11 +3.22 5.58 +0.69 5.19 +0.30 8.34 13.14 +4.79 9.28 +0.94 8.85 +0.50 Table 1 shows complete WikiText-2 and C4 perplexity at contexts 512/2048/4096 for all five full-suite models. E8E_8 k2v2 improves over scalar k3v2 in 24/30 model–dataset–context blocks while using fewer key bits; these wins include every non-Falcon model and both corpora. The gains are most visible at longer context lengths, where quantization error accumulates over more cached keys. For example, at 4K context on C4, Qwen2.5-14B degrades by only +0.29 perplexity under E8E_8, compared with +1.41 for scalar k3v2 and +5.20 for KIVI; Llama-3.1-8B shows the same pattern (+0.52 for E8E_8, +1.17 for scalar, +36.54 for KIVI). Granite-3.1-8B also benefits substantially at 4K, with WikiText-2/C4 deltas of +0.30/+0.50 for E8E_8 versus +0.69/+0.94 for scalar. The only systematic scalar advantage appears on Falcon3-7B, where scalar k3v2 is best in all six Falcon rows. Even there, the gap is small in absolute perplexity (at most 0.22), and E8E_8 remains below KIVI in every Falcon row. This suggests that Falcon’s activation statistics are unusually favorable to per-coordinate scalar quantization, but not that the 2-bit lattice representation is generally brittle. Overall, the robust conclusion is that E8E_8 moves the quality–memory Pareto frontier beyond scalar k3v2 for most evaluated long-context settings and dominates KIVI k2v2 across this perplexity suite. 4.4 Memory compression Table 2: Effective compressed KV-cache bytes per token for the quantized cache region. The accounting includes packed key/value data and per-token metadata (key norms/QJL signs and value scale/zero groups), but excludes fixed model-level tables/rotations and the finite recent-token residual window. Model L HkvH_kv Baseline Scalar k3v2 E8 Lattice k2v2 (B/tok) (B/tok) Ratio (B/tok) Ratio Qwen2.5-14B 48 8 196,608 40,944 4.8× 32,592 6.0× Llama-3.1-8B 32 8 131,072 27,296 4.8× 21,728 6.0× Mistral-7B 32 8 131,072 27,296 4.8× 21,728 6.0× Falcon3-7B 28 4 114,688 22,652 5.1× 18,452 6.2× Granite-3.1-8B 40 8 163,840 34,120 4.8× 27,160 6.0× Table 2 quantifies per-token KV cache memory. E8E_8 k2v2 achieves ∼6× 6× effective compression for the four Hkv=8H_kv=8 models, compared with ∼4.8× 4.8× for scalar k3v2. This corresponds to roughly a 20% smaller compressed cache than scalar k3v2 after accounting for packed key/value payloads and per-token metadata. The Falcon3-7B row is slightly different because its larger head dimension better amortizes fixed per-vector metadata such as key norms, yielding 5.1× for scalar and 6.2× for E8E_8. These ratios use the effective bytes/token accounting defined in Section 3. For a representative 8B architecture such as Llama-3.1-8B, scalar k3v2 requires 27.3 KB/token, whereas E8E_8 k2v2 requires 21.7 KB/token, saving 5.6 KB per token in the quantized cache region. At 128K context length, this accounting projects roughly 700 MB of additional memory saved per sequence. 4.5 Decode throughput Table 3 demonstrates that table-based asymmetric scoring (detailed in Section 3.3) successfully eliminates the computational bottleneck traditionally associated with lattice VQ. Across all five evaluated architectures and context lengths, E8E_8 k2v2 achieves decode-throughput parity with both the uncompressed fp16 baseline and the optimized scalar (k3v2) and KIVI kernels in our single-sequence decode benchmark. The variance across all methods falls within the margin of standard measurement noise (±0.2 tok/s), indicating no measurable latency degradation in this setting. This empirical parity supports our system design in the measured setting: precomputing the lookup table avoids dense key reconstruction during attention, and the remaining table-build/gather overhead is below measurement noise for our single-sequence A100 decode path. These throughput results are measured on our A100 serving path; additional validation on other accelerators and paged-attention runtimes is left for future work. Table 3: Decode throughput (tok/s) with table-based asymmetric scoring. E8 Lattice matches Scalar throughput despite using structured VQ, thanks to lookup-table scoring that avoids codebook dequantization. Model Context Baseline KIVI k2v2 Scalar k3v2 E8 Lattice k2v2 Qwen2.5-14B 1K 49.4 49.4 49.4 49.4 4K 48.4 48.4 48.4 48.4 Llama-3.1-8B 1K 89.0 88.9 88.9 89.0 4K 86.9 87.0 87.0 87.0 Mistral-7B 1K 92.9 92.9 92.9 93.0 4K 90.7 90.7 90.8 90.8 Falcon3-7B 1K 94.4 94.5 94.5 94.5 4K 91.2 91.2 91.3 91.2 Granite-3.1-8B 1K 81.0 81.0 81.0 81.0 4K 78.7 78.7 78.7 78.7 4.6 Downstream task accuracy Table 4: Downstream task accuracy (%). Each cell reports HellaSwag 10-shot / ARC-Challenge 25-shot / WinoGrande 5-shot / average. Model Baseline KIVI k2v2 Scalar k3v2 E8 k2v2 Cell format: HellaSwag / ARC-C / WinoGrande / Avg Qwen2.5-14B 65.5 / 60.5 / 75.1 / 67.1 65.7 / 60.3 / 75.3 / 67.1 65.7 / 60.4 / 75.3 / 67.1 65.7 / 60.3 / 75.3 / 67.1 Llama-3.1-8B 59.8 / 53.8 / 73.6 / 62.4 59.7 / 53.6 / 73.4 / 62.2 59.7 / 53.7 / 73.4 / 62.3 59.7 / 53.7 / 73.4 / 62.3 Mistral-7B 65.8 / 57.8 / 74.9 / 66.2 65.8 / 57.8 / 74.8 / 66.1 65.8 / 57.8 / 74.8 / 66.1 65.8 / 57.8 / 74.8 / 66.1 Falcon3-7B 60.3 / 55.8 / 70.3 / 62.1 60.3 / 55.9 / 70.4 / 62.2 60.3 / 55.9 / 70.4 / 62.2 60.3 / 56.0 / 70.4 / 62.2 Granite-3.1-8B 64.0 / 51.4 / 75.5 / 63.7 64.0 / 51.7 / 74.8 / 63.5 64.0 / 51.7 / 74.8 / 63.5 64.0 / 51.7 / 74.8 / 63.5 To ensure that extreme KV cache compression does not degrade foundational model capabilities, we evaluate zero-shot and few-shot reasoning performance on HellaSwag, ARC-Challenge, and WinoGrande. Table 4 reports these downstream accuracies. Across all models, E8E_8 k2v2 strictly maintains parity with both the fp16 baseline and the scalar baselines. In few-shot settings, much of the demonstration context is still quantized; the observed parity therefore suggests that the compressed cache preserves the in-context signal needed by these tasks, while the recent-token residual window protects the final local context. To isolate the role of this residual window, we vary its size W (Figure 2). When the uncompressed window is entirely removed (W=0), the underlying full quantization noise is exposed, and distinct performance ordering reappears. As W expands to 128 tokens, the performance of all compressed configurations rapidly converges to the fp16 baseline, remaining practically indistinguishable for larger windows. Figure 2: Average downstream accuracy across the core residual-sweep models versus residual window size. At W=0W=0, compression quality differences are visible; by W=128W=128, methods are nearly indistinguishable. 4.7 Attention fidelity Figure 3: Left: per-layer cosine similarity between compressed and fp16 keys captured from model forward passes. Right: per-layer KL divergence in a controlled attention-distribution probe. Lines show mean over models and shaded bands show the min–max range after normalizing each architecture’s layer index to [0,1][0,1]. Figure 3 shows per-layer cosine similarity between compressed and fp16 key representations. E8E_8 at 2 bits/dim remains substantially higher than scalar at 3 bits (roughly 0.954 vs. 0.921 average), consistent with the rate-distortion analysis. The shaded range is narrow and smooth across depth, suggesting lattice quantization error is not concentrated in a small subset of layers. This reconstruction fidelity directly translates to superior preservation of the attention mechanism itself. Figure 3b illustrates the per-layer KL divergence between attention distributions computed with compressed versus fp16 keys. Here, E8E_8 achieves a remarkably stable and low KL divergence (0.045), effectively halving the error of the 3-bit scalar baseline (0.091). Furthermore, E8E_8 outperforms KIVI k2v2, which suffers from attention distribution shifts (KL > 0.200). This confirms that our lattice geometry preserves the true pre-softmax attention logits more faithfully than existing 2-bit methods, ensuring that improved geometric reconstruction meaningfully carries through to downstream attention behavior. 4.8 Long-context retrieval robustness To verify that extreme KV cache quantization does not degrade the model’s ability to fetch isolated facts from extended contexts, we conducted a standard single-needle-in-a-haystack retrieval evaluation similar to (Zandieh et al., 2025). We evaluated retrieval accuracy using a randomly placed fact injected into distractor contexts of varying lengths (4K, 8K, and 16K tokens) across all five models. E8E_8 k2v2 achieved 100% retrieval accuracy across all lengths and models, strictly matching the performance of the uncompressed fp16 baseline as well as the competing KIVI and scalar compression methods. 4.9 Ablation studies Table 5: Compact ablations (mean across core models). (a) E8 scale calibration. (b) Initial-layer precision sweep. (a) E8 scale α α 1 4 8 10 12 15 20 30 MSE 0.0577 0.0045 0.0011 0.0008 0.0007 0.0008 0.0013 0.0024 cos 0.292 0.727 0.913 0.940 0.953 0.960 0.959 0.952 (b) Initial-layer bitwidth Cfg E8-2b E8-f4-3b E8-f8-3b Sc-3b Sc-f4-4b cos 0.954 0.952 0.950 0.920 0.927 (a) E8E_8 scale calibration. The scale parameter α controls the dynamic range mapped to the E8E_8 quantizer. While values that are too small (α≤4α≤ 4) cause strong clipping and values too large (α≥20α≥ 20) waste codebook resolution, performance is remarkably robust within the intermediate range. We observe a wide, stable plateau between α=8α=8 and α=15α=15 where MSE and cosine similarity remain stable. (b) Mixed precision. We also test simple mixed-precision variants that change the first few layers. These variants do not improve the average cosine metric over uniform E8E_8 k2, so our main configuration keeps a uniform key format across layers. Because the E8 3-bit variant adds a QJL residual correction rather than serving as a pure higher-fidelity E8 codebook, we interpret this ablation only as an implementation-level check, not as evidence that early layers never benefit from additional capacity. 5 Conclusion We demonstrate that E8E_8 lattice vector quantization is a practical primitive for low-bit KV-cache key compression. Its fixed geometry-aware codebook enables table-based asymmetric scoring: for each query, dot products with the shared codebook are precomputed once and cached keys are scored directly from indices, avoiding explicit key dequantization in the decode path. Across diverse LLM families and long-context evaluations, this design improves key fidelity and quality per byte while preserving downstream quality and single-sequence decode-throughput parity. At the same time, making compressed-domain scoring efficient beyond this setting, especially for larger batches and prefill-heavy workloads remains an important systems challenge. These results suggest that structured vector codes are a useful building block for KV-compression systems. References Almazrouei et al. (2023) E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo The falcon series of open language models. External Links: 2311.16867, Link Cited by: §4.1. Chee et al. (2023) J. Chee, Y. Cai, V. Kuleshov, and C. De Sa QuIP: 2-bit quantization of large language models with guarantees. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §2. Chen et al. (2024) Z. Chen, R. Sadhukhan, Z. Ye, Y. Zhou, J. Zhang, N. Nolte, Y. Tian, M. Douze, L. Bottou, Z. Jia, and B. Chen MagicPIG: lsh sampling for efficient llm generation. External Links: 2410.16179, Link Cited by: §2. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. External Links: 1803.05457, Link Cited by: §4.1. Conway and Sloane (1999) J. H. Conway and N. J. A. Sloane Sphere packings, lattices and groups. 3 edition, Grundlehren der mathematischen Wissenschaften, Springer New York. External Links: Document Cited by: §2. Dodge et al. (2021) J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner Documenting large webtext corpora: a case study on the colossal clean crawled corpus. External Links: 2104.08758, Link Cited by: §4.1. Egiazarian et al. (2024) V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh Extreme compression of large language models via additive quantization. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §2. Engels et al. (2025) J. Engels, E. J. Michaud, I. Liao, W. Gurnee, and M. Tegmark Not all language model features are one-dimensionally linear. External Links: 2405.14860, Link Cited by: §1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, and A. K. et al. The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §4.1. Guo et al. (2020) R. Guo, P. Sun, E. Lindgren, Q. Geng, D. Simcha, F. Chern, and S. Kumar Accelerating large-scale inference with anisotropic vector quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. Cited by: §2. He et al. (2024) Y. He, L. Zhang, W. Wu, J. Liu, H. Zhou, and B. Zhuang ZipCache: accurate and efficient KV cache quantization with salient token identification. In Neural Information Processing Systems, External Links: Document, Link Cited by: §2. Hooper et al. (2024) C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami KVQuant: towards 10 million context length LLM inference with KV cache quantization. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §1, §2. Jegou et al. (2011) H. Jegou, M. Douze, and C. Schmid Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33 (1), p. 117–128. External Links: ISSN 0162-8828, Link, Document Cited by: §2, §3.3. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, and F. B. et al. Mistral 7b. External Links: 2310.06825, Link Cited by: §4.1. Kang et al. (2024) H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao GEAR: an efficient error reduction framework for KV cache compression in LLM inference. In Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, M. Rezagholizadeh, P. Passban, S. Samiee, V. Partovi Nia, Y. Cheng, Y. Deng, Q. Liu, and B. Chen (Eds.), Proceedings of Machine Learning Research, Vol. 262, p. 305–321. External Links: Link Cited by: §2. Karmore (2026) A. Karmore LOOKAT: lookup-optimized key-attention for memory-efficient transformers. External Links: 2601.10155, Link Cited by: §3.3. Li et al. (2025a) H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen A survey on large language model acceleration based on kv cache management. External Links: 2412.19442, Link Cited by: §1, §1. Li et al. (2025b) J. Li, Y. Zhang, M. Y. Hassan, T. Chafekar, T. Cai, Z. Ren, P. Guo, B. Karimzadeh, C. J. Reed, C. Wang, and C. Gan CommVQ: commutative vector quantization for KV cache compression. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267. External Links: Link Cited by: §2. Li et al. (2024) Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen SnapKV: LLM knows what you are looking for before generation. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §2. Liu et al. (2025) T. Liu, S. Li, J. Yang, T. Zhao, F. Zhou, X. Song, G. Dai, S. Yan, H. Yang, and Y. Wang PM-KVQ: progressive mixed-precision KV cache quantization for long-context LLMs. External Links: 2505.18610, Link Cited by: §2. Liu et al. (2024) Z. Liu, J. Yuan, H. Jin, S. (. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu KIVI: a tuning-free asymmetric 2bit quantization for kv cache. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §1, §2, §4.1. Merity et al. (2016) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. External Links: 1609.07843, Link Cited by: §4.1. Qwen Team et al. (2025) Qwen Team, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al. Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §4.1. Sakaguchi et al. (2019) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. External Links: 1907.10641, Link Cited by: §4.1. Savkin et al. (2025) S. Savkin, E. Porat, O. Ordentlich, and Y. Polyanskiy NestQuant: nested lattice quantization for matrix products and LLMs. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 53042–53062. External Links: Link Cited by: §2. Shutova et al. (2025) A. Shutova, V. Malinovskii, V. Egiazarian, D. Kuznedelev, D. Mazur, S. Nikita, I. Ermakov, and D. Alistarh Cache me if you must: adaptive key-value quantization for large language models. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 55451–55473. External Links: Link Cited by: §2. Singhania et al. (2024) P. Singhania, S. Singh, S. He, S. Feizi, and A. Bhatele Loki: low-rank keys for efficient sparse attention. External Links: 2406.02542, Link Cited by: §2. Stallone et al. (2024) M. Stallone, V. Saxena, L. Karlinsky, B. McGinn, T. Bula, M. Mishra, and A. M. S. et al. Scaling granite code models to 128k context. External Links: 2407.13739, Link Cited by: §4.1. Tseng et al. (2024) A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa QuIP#: even better llm quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §2. Viazovska (2017) M. Viazovska The sphere packing problem in dimension 8. Annals of Mathematics 185 (3), p. 991–1015. External Links: Document, Link Cited by: §1, §2, §3.2. Wang et al. (2025) S. Wang, Z. Li, L. Shi, B. Du, H. Zhao, Y. Li, and Q. Wang From parameters to performance: a data-driven study on llm structure and development. External Links: 2509.18136, Link Cited by: §3.2. Wang et al. (2024) X. Wang, M. Salmani, P. Omidi, X. Ren, M. Rezagholizadeh, and A. Eshaghi Beyond the limits: a survey of techniques to extend the context length in large language models. External Links: 2402.02244, Link Cited by: §1. Wu et al. (2025) S. Wu, A. Lv, X. Feng, Y. Zhang, X. Zhang, G. Yin, W. Lin, and R. Yan PolarQuant: leveraging polar transformation for efficient key cache quantization and decoding acceleration. External Links: 2502.00527, Link Cited by: §3.3. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. External Links: 2309.17453, Link Cited by: §2. Zamir and Feder (1996) R. Zamir and M. Feder On lattice quantization noise. IEEE Transactions on Information Theory 42 (4), p. 1152–1159. External Links: Document Cited by: §1, §2. Zandieh et al. (2025) A. Zandieh, M. Daliri, M. Hadian, and V. Mirrokni TurboQuant: online vector quantization with near-optimal distortion rate. External Links: 2504.19874, Link Cited by: §1, §2, §3.1, §3.4, §4.1, §4.8. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. External Links: 1905.07830, Link Cited by: §4.1. Zhang et al. (2025a) H. Zhang, X. Ji, Y. Chen, F. Fu, X. Miao, X. Nie, W. Chen, and B. Cui PQCache: product quantization-based kvcache for long context llm inference. External Links: 2407.12820, Link Cited by: §3.3. Zhang et al. (2024) T. Zhang, J. Yi, Z. Xu, and A. Shrivastava KV cache is 1 bit per channel: efficient large language model inference with coupled quantization. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §2. Zhang et al. (2025b) Y. Zhang, Y. Hu, R. Zhao, J. C. S. Lui, and H. Chen DiffKV: differentiated memory management for large language models with parallel kv compaction. External Links: 2412.03131, Link Cited by: §2. Zhang et al. (2023) Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen H2O: heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §2.