Paper deep dive

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, Yik-Chung Wu, Ngai Wong

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 92

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/22/2026, 2:37:16 AM

Summary

The paper introduces FluxBin, an algorithm-kernel co-design framework for ultra-low-bit Large Language Model (LLM) inference. It combines a Post-Training Quantization (PTQ) method using Decoupled Row-Column Binary Decomposition and Hessian-guided saliency-aware hybrid bases with a specialized CUDA kernel. The kernel utilizes Lookup Tables (LUT) with Scale Fusion and Virtual Columnar Mapping to eliminate floating-point arithmetic and handle sparse salient columns efficiently. FluxBin achieves up to 5.92x speedup and 10.19x energy savings, enabling 70B-scale model deployment on a single A100 GPU with 4x memory reduction.

Entities (9)

A100 GPU · hardware · 100%CUDA · software-platform · 100%FluxBin · methodframework · 100%LLM · model-type · 100%Decoupled Row-Column Binary Decomposition · algorithm · 95%Hessian-guided Salient-Aware Hybrid Bases · algorithm · 95%LUT Building with Scale Fusion (LUT-BSF) · kernel-technique · 95%Post-Training Quantization (PTQ) · technique · 95%Virtual Columnar Mapping (VCM) · kernel-technique · 95%

Relation Signals (7)

FluxBin → enablesdeploymenton → A100 GPU

confidence 100% · This effectively enables the deployment of 70B-scale models on one single A100 GPU

FluxBin → achievesspeedupon → LLM

confidence 95% · FluxBin achieves up to 5.92x speedup ... across diverse model architectures

FluxBin → isatypeof → Post-Training Quantization (PTQ)

confidence 95% · FluxBin entails a pure PTQ framework

FluxBin → uses → Decoupled Row-Column Binary Decomposition

confidence 95% · Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity

FluxBin → uses → Virtual Columnar Mapping (VCM)

confidence 95% · featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution

FluxBin → uses → Hessian-guided Salient-Aware Hybrid Bases

confidence 95% · complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information

FluxBin → uses → LUT Building with Scale Fusion (LUT-BSF)

confidence 95% · At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (\textbf{F}lexible \textbf{L}UT-based \textbf{U}ltra-low-bit e\textbf{X}ecution with \textbf{Bin}ary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,136 characters extracted from source content.

Expand or collapse full text

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy Qingyao Yang Runming Yang He Xiao Wendong Xu Junyu Chen Haobo Liu Chenchen Ding Ruihan Hu Affiliation: The University of Hong Kong, Harbin Institute of Technology Correspondence:nwong@e.hku.hk Yik-Chung Wu Ngai Wong Abstract While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (Flexible LUT-based Ultra-low-bit eXecution with Binary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to 5.92×5.92× speedup and 10.19×10.19× energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a 4×4× memory reduction. Code is available at https://github.com/nicyyyy/FluxBin. 1 Introduction While Large Language Models (LLMs) demonstrate unprecedented capabilities 1; 31, their massive scale imposes severe storage and bandwidth constraints 37. Post-Training Quantization (PTQ) 8; 19; 34 has thus emerged as a standard solution for reducing bit-width without retraining. Among these strategies, binary quantization billm; 27 pushes compression to the limit, offering maximal theoretical efficiency. To mitigate the inherent accuracy degradation of 1-bit representations, recent works like QBB 2, PTQTP 35, and DB-LLM 3 adopt multi-binary base quantization, which approximates weights as combinations of binary matrices to recover representational capacity. However, despite this theoretical promise and improved fidelity, existing approaches still struggle to translate binary representations into practical end-to-end inference speedups. Limitation 1: Failure to exploit multiplication free hardware potential. While binary representations theoretically allow replacing expensive floating-point operations with efficient bitwise logic, most current binary quantization methods fail to translate this into practical end-to-end acceleration. Methods such as BiLLM 10, ARB-LLM 18 and HB-LLM 23 etc. often prioritize algorithmic fidelity (e.g., simulated quantization) and rely on dequantization to FP16 during inference, leaving the hardware advantages of bitwise computation largely untapped. Limitation 2: Trade-off between saliency granularity and hardware efficiency. Current strategies fail to balance fine-grained preservation with execution efficiency. Unstructured methods 28; 5; 17 suffer from irregular memory access and metadata overhead due to element-wise sparsity. Conversely, coarse-grained approaches like SLiM-LLM 11 promote entire groups to maintain structure, causing bit redundancy by unnecessarily preserving non-salient elements at high precision. Thus, a strategy that achieves fine-grained saliency adaptation and hardware efficiency remains absent. Limitation 3: Implementation inefficiencies in low-bit kernels. The autoregressive generation phase of LLMs is intrinsically memory-bound, restricting hardware throughput even with INT8 precision 21. Although recent W4A16 quantization methods 38; 8 mitigate memory bottlenecks by compressing weights, they introduce significant runtime overhead due to dynamic dequantization, as weights must be converted to FP16 before matrix multiplication, which does not truly exploit the addition operation properties during inference. Look-Up Tables (LUT) 24; 30; 13; 15; 20 theoretically offer a multiplication-free alternative by replacing arithmetic with memory lookups. Existing LUT kernels fail to handle the irregular memory access introduced by mixed-precision, and struggle to achieve high performance and low latency in ultra-low bit PTQ settings. To address the aforementioned limitations simultaneously and bridge the gap between ideal model compression schemes and real-world hardware inference efficiency. We propose FluxBin, an algorithm-kernel co-designed framework that works in tandem with flexible and high-performance PTQ method and CUDA kernels. At the algorithm level, to reconcile binary efficiency with capacity, we propose Decoupled Row-Column Binary Decomposition for fine-grained distribution capture. Then, our Hessian-guided Salient-Aware Hybrid Bases (mixed precision) preserve fidelity by selectively allocating additional bases to critical features. Besides, the proposed method is pure PTQ without any fine-tuning 4 or distillation 25; 12. At the kernel level, our LUT Building with Scale Fusion (LUT-BSF) reduces floating-point overhead in LUT building. Complementing this, Virtual Columnar Mapping (VCM) converts sparse salient columns into dense storage, enabling unified execution for hybrid branches. Our contributions are: • FluxBin entails a pure PTQ framework, with decoupled row-column binary base and Hessian-guided hybrid base. This maintains the hardware efficiency and competitive accuracy. • FluxBin designs a highly optimized LUT-based CUDA kernel with LUT-BSF and VCM strategy. This synergy enables high-fidelity hybrid precision and turns binary based PTQ into real inference advantage. • FluxBin bridges the algorithm-kernel gap through a custom LUT-based kernel that bypasses de-quantization overhead, converting multiplication-free logic into practical acceleration. It achieves up to 5.92×5.92× speedup and 10.19×10.19× energy saving. 2 Related Work Binary Quantization. Standard binary quantization aims to approximate a full-precision weight matrix ∈ℝM×NW ^M× N using a binary matrix ∈−1,+1M×NB∈\-1,+1\^M× N scaled by a floating-point factor ∈ℝ α 26. The optimization objective is to minimize the reconstruction error under the Frobenius norm: min,⁡‖−‖F2 _ α,B\|W- αB\|_F^2 (1) The optimal solution to Eq. (1) typically yields =sign()B=sign(W) and =1MN‖1 α= 1MN\|W\|_1, as derived in prior works like XNOR-Net 27. Regarding the representative works, ARB-LLM 18 introduces column-wise grouping and alternating refined binarization. HB-LLM 23 applies distinct binarization strategies to frequency components separated by wavelet decomposition. Binary Base Approximation. Since a single binary basis (1-bit) has limited representation capacity, high order binary base quantization methods approximate W using K binary bases: ≈∑k=1Kk(k)W≈ _k=1^K α_kB^(k) (2) To minimize this quantization error, LQ-Nets 39 proposes to jointly train the quantization basis and the binary encodings. This allows for an arbitrary basis that better fits the data distribution. Approaches such as QBB 2 solve this via a greedy residual strategy. Let (0)=R^(0)=W. At the k-th step, parameters are optimized to approximate the current residual, updating (k)=(k−1)−αk(k)R^(k)=R^(k-1)- _kB^(k). DB-LLM 3 approximates weights by primary and secondary residual binary base to refine the quantization errors. Figure 1: Illustration of LUT-based Multiplication. Figure 2: The framework of FluxBin Binary Bases Quantization. LUT-based Multiplication. Standard matrix-vector multiplication (MVM) with binary weights ∈−1,+1M×NB∈\-1,+1\^M× N often incurs significant computational redundancy. As illustrated in Figure 1(a), identical weight sub-patterns (e.g., [+1,+1,−1][+1,+1,-1]) frequently repeat across different rows. Consequently, the dot product for the same input sub-vector is calculated multiple times redundantly. To address this, LUT-GEMM 24 proposes replacing arithmetic operations with efficient memory lookups. The input dimension is partitioned into sub-vectors of size μ (e.g., μ=3μ=3 in the figure). For each input sub-vector subX_sub, all 2μ2^μ possible linear combinations are pre-computed and stored in a LUT. As shown in Figure 1(b), during inference, the μ-bit binary weight pattern serves directly as the memory address to fetch the pre-computed partial sum, which is much faster than carrying out the original calculations 36; 13. Furthermore, recent work LUT Tensor Core 22 introduced an instruction set, with an elaborated precompute and tiling strategy, to build LUT-based accelerators. While LUT-GEMM provides an efficient kernel for uniform quantization, it does not provide PTQ algorithm adaptation, mixed-precision configurations, or end-to-end inference pipelines. 3 Methodology 3.1 FluxBin Quantization We present the mathematical formulation of FluxBin quantization, visualized in Figure 2, and the pseudo-code of this quantization method is described in Sec. B. Decoupled Row-Column Binary Bases. We approximate dense weights ∈ℝM×NW ^M× N via linear combinations of binary bases. Distinct from prior row-scaling-only methods, we incorporate column-wise scaling to capture anisotropic weight distributions. Definition 3.1 (Row-Column Binary Decomposition). For decomposition order k, the approximation ^(k) W^(k) is defined as the Hadamard product of scaling vectors and a binary base: ^(k)=(r(k)c(k)⊤)⊙(k) W^(k)=( α_r^(k) α_c^(k) ) ^(k) (3) where (k)∈−1,+1M×NB^(k)∈\-1,+1\^M× N is the binary base, and r(k)∈ℝM α_r^(k) ^M, c(k)∈ℝN α_c^(k) ^N denote the row and column scaling vectors. The final quantized matrix sums over all orders: ^=∑k=1Kb^(k) W= _k=1^K_b W^(k). Consistent with 8; billm, we assign one row scale r(k) α_r^(k) per weight group. Parameter Optimization. We optimize parameters r,c,\ α_r, α_c,B\ by minimizing the reconstruction error: ℒ=‖−(rc⊤)⊙‖F2L= \|W- ( α_r α_c ) \|_F^2 (4) Employing alternating minimization, fixing B and one scaling vector reduces the problem to weighted least squares. Proposition 3.2 (Optimal Scaling Factors). Given fixed B and one scaling vector, the closed-form solutions minimizing ℒL are: r,i(k) α^(k)_r,i =∑j(c,j(k)ij(k))ij∑j(c,j(k)ij(k))2+ϵ, = _j ( α^(k)_c,jB^(k)_ij )W_ij _j ( α^(k)_c,jB^(k)_ij )^2+ε, (5) c,j(k) α^(k)_c,j =∑i(r,i(k)ij(k))ij∑i(r,i(k)ij(k))2+ϵ = _i ( α^(k)_r,iB^(k)_ij )W_ij _i ( α^(k)_r,iB^(k)_ij )^2+ε where ϵε is a stability constant. For Kb≥2K_b≥ 2, we perform joint optimization rather than greedy sequential extraction. Defining the magnitude tensor (k)=r(k)c(k)⊤M^(k)= α_r^(k) α_c^(k) , we solve for the optimal binary signs: Proposition 3.3 (Optimal Binary Base Assignment). The binary combination =[b1,…,bKb]⊤b=[b_1,…,b_K_b] minimizing the local error at (i,j)(i,j) is: ij(1:Kb)=argmin∈−1,+1Kb|ij−∑k=1Kbij(k)⋅bk|B_ij^(1:K_b)= b∈\-1,+1\^K_bargmin |W_ij- _k=1^K_bM_ij^(k)· b_k | (6) Implementation-wise, we exhaustively evaluate all 2Kb2^K_b sign combinations to ensure optimality. Structural Saliency Search. Recognizing that weight elements contribute unequally to performance, we adopt a hybrid base strategy guided by structural saliency. Leveraging the Optimal Brain Quantization (OBQ) framework 9, we approximate weight sensitivity via the inverse Hessian −1H^-1, and this can be efficiently calculated through Cholesky decomposition 14. Given the column-wise clustering of sensitive weights in LLMs billm, we aggregate element-wise errors into a cumulative column score, following previous works 11; 7. Definition 3.4 (Hessian-based Column Saliency). Following SLiM-LLM 11, the column-wise saliency score jS_j is defined as: j=∑i=1Mij2[−1]jj2S_j=Σ _i=1^M W_ij^2 [H^-1 ]_j^2 (7) Columns with higher jS_j usually contain outliers. To ensure load balancing, we partition columns into groups of size g. Within each group bC_b, we select a subset ℐsal(b)I^(b)_sal of size s (the number of salient columns per group) maximizing total saliency: ℐsal(b)=arg⁡maxℐ⊂b,|ℐ|=s⁡∑j∈ℐjI^(b)_sal= _I _b,|I|=s _j S_j (8) The aggregated set ℐsal=⋃bℐsal(b)I_sal= _bI^(b)_sal (with |ℐsal|=Nsal=(N/g)s|I_sal|=N_sal=(N/g)\,s salient columns in total) defines the binary mask ℳM (ℳj=1M_j=1 if j∈ℐsalj _sal, else 0). Salient-Aware Hybrid Bases. The final quantized weight W combines a Global Base Approximation and a Salient Refinement (Figure 2(b)): ^=∑k=1Kb⁡(,k)⏟Global Base+∑k=1Ks⁡(base,k)⊙ℳ⏟Salient Refinement W= _k=1^K_bQ(W; _k)_Global Base+ _k=1^K_sQ(R_base; _k) _Salient Refinement (9) Here, ⁡(⋅)Q(·) denotes the quantization function (Def. 3.1), with parameter sets k=(k),r(k),c(k) _k=\B^(k), α_r^(k), α_c^(k)\ for the global branch and k=s(k),sr(k),sc(k) _k=\B_s^(k), α_sr^(k), α_sc^(k)\ for the salient branch. baseR_base represents the residual of the global approximation: base=−∑k⁡(,k)R_base=W- _kQ(W; _k). 3.2 FluxBin LUT-based Kernel Figure 3(a) illustrates the FluxBin LUT-based GEMV kernel architecture, comprising three stages: (i) LUT Building with Scale Fusion to remove column scaling c α_c from the critical path; (i) LUT reading for Multiplication-free GEMV, utilizing bitwise operations for high throughput; and (i) Reduction with Row-Scale, applying row scaling r α_r and performing reduction before writing back to global memory. Crucially, our Virtual Columnar Mapping (VCM) allows unified kernel execution. (a) Overview of the FluxBin LUT-based GEMV Kernel. (b) Virtual Columnar Mapping. Figure 3: FluxBin kernel architecture and VCM strategy. Based on Eqs. (3) and (9), the GEMV is: ^T= W^T= (∑k=1Kb(k)⊙(r(k)c(k)⊤))T⏟Global Base X\! ( _k=1^K_bB^(k)\! \!( α_r^(k) α_c^(k) ) )^\!T_Global Base (10) +(∑k=1Ks(s(k)⊙ℳ)⊙(sr(k)sc(k)⊤))T⏟Salient Refinement + X\! ( _k=1^K_s(B_s^(k)\! \!M)\! \!( α_sr^(k) α_sc^(k) ) )^\!T_Salient Refinement Since c(k) α_c^(k) and r(k) α_r^(k) are diagonal per-column and per-row factors, the Global Base GEMV bY_b factorizes so that the column scale folds into the input and the row scale is applied after the binary matvec: b=∑k=1Kb((⊙c(k)⊤)(k)T)⊙r(k)⊤Y_b=Σ _k=1^K_b ((X α_c^(k) )\,B^(k)T ) α_r^(k) (11) Virtual Columnar Mapping (VCM) for Salient Phase. The GEMV computation of salient refinement is restricted to sparse columns defined by ℐsalI_sal. Since conventional sparse implementations suffer from irregular memory access, we introduce VCM, inspired by OS memory virtualization. As shown in Figure 3(b), we map the logical sparse bases to physical dense matrix, and maintain a translation layer (via ℐsalI_sal) to gather activation and scaling factor at runtime. Definition 3.5 (VCM Projection). VCM decouples logical sparsity from physical storage. Salient columns are remapped into a contiguous dense matrix sd(k)∈−1,1M×NsalB^(k)_sd∈\-1,1\^M× N_sal: sd(k)=[s;:,j(k)∣j∈ℐsal]B^(k)_sd=[B^(k)_s;:,j j _sal] (12) Prior to GEMV, we gather input activations into salX_sal and extract scales scd(k) α_scd^(k): sal _sal =[:,j∣j∈ℐsal], =[X_:,j j _sal], (13) scd(k) α_scd^(k) =[sc(k)[j]∣j∈ℐsal] =[ α_sc^(k)[j] j _sal] This transforms the logically sparse operation into a physically dense GEMV, enabling kernel reuse for Global Base and Salient Refinement. Stage 1: LUT Building with Scale Fusion (BSF). We partition inputs into sub-vectors subX_sub of size μ 24. Exploiting the column-wise scaling of c α_c that shared by all rows, we fuse it directly into the LUT, computing entries using ⊙cX α_c. Definition 3.6 (LUT-BSF). For an integer index p∈[0,2μ−1]p∈[0,2^μ-1], bn(p)∈0,1b_n(p)∈\0,1\ denote the n-th bit of p, the fused LUT entry is: LUTcon⁡[p]=∑n=1μ(sub[n]⋅c,sub[n])⋅(2bn(p)−1)LUT_con[p]= _n=1^μ(X_sub[n]· α_c,sub[n])·(2b_n(p)-1) (14) Here, (2bn(p)−1)(2b_n(p)-1) maps bits to −1,+1\-1,+1\. This strategy decouples c α_c from the main GEMV loop, effectively eliminating related floating-point operations. Figure 4: (a) E2E throughput scaling. (b) Average power. (c) Decode-only throughput. Stage 2&3: LUT Reading and Reduction. Instead of unpacking weights to Float16, the kernel directly streams compressed Uint32 weights. For each sub-block of size μ, we extract pattern indices p via bitwise ops to retrieve pre-computed partial sums from LUT. The results are accumulated modulated by the row-wise scaling factor in registers before being written to global memory. b _b =∑k=1Kb(LUTread⁡[(k)]⊙r(k)⊤), = _k=1^K_b (LUT_ read[B^(k)] α_r^(k) ), (15) sal _sal =∑k=1Ks(LUTread⁡[sd(k)]⊙sr(k)⊤) = _k=1^K_s (LUT_ read[B_sd^(k)] α_sr^(k) ) Matrix Tiling. As shown in Figure 3(a) (c), B is partitioned into Mtile×NtileM_tile× N_tile tiles mapped to thread blocks. Within each tile, input sub-vectors and c,sub α_c,sub are fused to build LUTs in shared memory, queried by all threads in the block. 4 Experiment Experimental Setup. All evaluations are conducted on a single NVIDIA A100 (80GB) GPU. For end-to-end throughput, we customize the Hugging Face generate API 33 with CUDA graphs to eliminate Python scheduling and kernel launch overhead: both prefill and decode kernels are captured and replayed from a pre-recorded graph with a static KV cache, ensuring that the measured latency reflects actual kernel execution time rather than framework overhead. Regarding FluxBin configurations, we adopt the notation Kb-sS-gG, where the leading number K denotes the base count (the literal “b” stands for “bases”), shared by the global and salient-refinement branches (we use the same number of bases for both). s denotes salient columns per group, and g is group size for both saliency search and row-wise scale. We primarily evaluate the mixed-precision variants 2b-s8-g128 and 2b-s16-g128. Additionally, 2b-g128 and 4b-g128 (pure global base) are included to provide a comprehensive analysis. Table 1: E2E speed and zero-shot accuracy on LLaMA-2. ∗FT/Dist used. †Avg bit incl. scales (Sec. C.2). ‡HS=HellaSwag, WG=Winogrande. Best/2nd/3rd. LLaMA-2 Method FT/Dist∗ W bit† Speed (tok/s) ↑ Wiki ↓ PIQA ↑ ARC-e ↑ ARC-c ↑ BoolQ ↑ HS‡ ↑ WG‡ ↑ Avg. ↑ 7B FP16 16 42.94 5.47 78.13 75.46 43.00 79.36 57.08 69.38 67.07 GPTQ ✗ 3.125 – 8.42 73.50 63.43 30.29 67.00 48.57 64.96 57.96 OmniQuant ✓ 3 83.40 6.62 74.70 64.26 35.92 66.29 51.73 63.45 59.39 SLiM-LLM ✓ 2 73.70 16.38 63.43 46.42 23.63 61.71 33.22 55.64 47.34 AQLM ✓ 2.02 98.50 6.59 72.69 65.11 32.85 69.24 49.34 62.59 58.64 QuIP# ✓ 2.02 131.97 8.22 75.10 64.60 34.60 74.31 42.94 64.90 59.41 FluxBin-2b-s8-g128 ✗ 2.63 254.39 8.76 72.52 65.45 30.97 69.88 45.64 64.09 58.09 FluxBin-2b-s16-g128 ✗ 2.75 250.87 8.20 75.50 65.40 32.42 70.80 45.97 64.80 59.15 13B FP16 16 33.14 4.88 79.49 78.87 47.53 82.20 60.19 72.69 70.16 GPTQ ✗ 3.125 – 6.40 75.95 73.23 41.89 73.64 54.15 67.56 64.40 OmniQuant ✓ 3 57.60 5.58 77.69 69.02 41.97 69.02 56.79 65.09 63.26 SLiM-LLM ✓ 2 61.20 9.41 64.96 54.67 24.82 65.22 38.43 55.48 50.60 AQLM ✓ 2 32.90 5.60 76.71 78.79 40.36 75.81 54.52 64.80 65.17 QuIP# ✓ 2.01 84.16 6.06 77.30 69.30 42.92 79.24 55.46 67.70 65.32 FluxBin-2b-s8-g128 ✗ 2.63 162.94 7.43 75.46 71.80 36.52 79.11 49.36 69.53 63.63 FluxBin-2b-s16-g128 ✗ 2.75 162.73 7.16 75.46 72.81 38.05 76.91 50.53 69.96 63.95 70B FP16 16 OOM 3.12 81.50 82.70 54.10 85.17 65.33 80.43 74.87 GPTQ ✗ 3.125 – 4.96 79.27 78.41 48.12 79.51 59.10 74.66 69.85 OmniQuant ✓ 3 – 3.92 79.70 75.58 47.52 66.48 61.92 73.71 67.49 AQLM ✓ 2.07 6.82 3.94 80.30 78.79 51.19 82.29 62.23 75.77 71.76 QuIP# ✓ 2.01 18.64 4.21 80.30 77.30 48.70 82.91 61.92 75.90 71.17 FluxBin-2b-s8-g128 ✗ 2.63 60.67 4.90 78.64 80.22 49.83 81.04 59.16 76.80 70.95 FluxBin-2b-s16-g128 ✗ 2.75 60.60 4.84 79.49 80.26 49.06 79.54 59.34 76.32 70.67 4.1 End-to-end Analysis We compare FluxBin against the FP16 baseline and SOTA quantization frameworks, including AutoAWQ (4-bit) and QuIP# (2-bit). Throughput and Scalability. As shown in Figure 4 (a), FluxBin demonstrates superior throughput across all batch sizes, with hybrid configurations (2b-s8/s16) matching the pure 2-bit baseline. Unlike conventional weight-only quantization whose advantage diminishes at large batch, FluxBin retains speedup because its LUT-based computation reduction (μ/Kb=4×μ/K_b=4×, Sec. 4.4) is batch-size independent, and the batch dimension is tiled per thread block with fixed per-block shared memory cost. To further isolate the decode phase, Figure 4 (c) reports decode-only throughput. FluxBin achieves 6.3×6.3×–6.6×6.6× decode speedup at BS=1, and the gap persists at large batch sizes, demonstrating that the LUT kernel delivers consistent acceleration throughout the autoregressive generation process. Memory Efficiency. FluxBin achieves up to 4×4× memory reduction as shown in Table 3 (Appendix C.3). This critical compression capability enables the feasible execution of massive models, such as LLaMA-2-70B, on a single A100 GPU, whereas the FP16 baseline fails due to Out-Of-Memory (OOM) errors. Energy Efficiency. As visualized in Figure 4 (b), FluxBin significantly lowers the average power during inference (e.g., dropping from ∼ 225W to ∼ 115W on 13B). Combined with the speedup, this results in a total energy reduction of 10.19×10.19× compared to FP16 (detailed in Appendix E.1). This efficiency stems from minimized data movement overhead and the reduction of floating-point arithmetic. Architectural Robustness and Scalability. To validate the generality of FluxBin, we extend our evaluation to the Qwen3 family (featuring GQA) and the massive LLaMA-3.1-70B (detailed in Appendix D.1). The results mirror our LLaMA-2 findings, and empirically confirms that FluxBin is architecture-agnostic, delivering consistent acceleration regardless of attention mechanisms or model size. Figure 5: (a) Latency breakdown of LUT-based kernel. (b) Speed-accuracy trade-off on LLaMA-2. 4.2 Speed vs. Accuracy Table 1 presents a comparison of FluxBin with SOTA ultra-low-bit quantization methods 8; 29; 11; 6; 32 across the LLaMA-2 family 31. We analyze the results from accuracy retention and inference speed. During the speed evaluation, we set the input and output length to 512. Competitive Accuracy without Training. Despite being a training-free method, FluxBin achieves accuracy comparable to that of fine-tuning or distillation based approaches. On LLaMA-2-7B, our 2b-s16-g128 configuration attains an average zero-shot accuracy of 59.15%, which is on par with the heavily fine-tuned QuIP# (59.41%) and outperforms AQLM (58.64%). Extreme Speed Advantage. For LLaMA-2-7B, FluxBin reaches 254.39 tokens/s, representing a 5.92× speedup over FP16 baseline. This speed advantage scales effectively to larger models; on LLaMA-2-70B, FluxBin achieves 3.25× higher throughput compared to QuIP#. Optimal Trade-off. As visualized in Figure 5 (b), existing methods typically compromise significant speed to maintain accuracy (bottom-right) or sacrifice accuracy for compression (top-left). FluxBin occupies the desirable top-right region of the scaling curve. This positions FluxBin as the optimal solution that simultaneously delivers high-fidelity generation and ultra-low latency, effectively breaking the conventional constraints of the speed-accuracy trade-off. Generalization Across Different Architectures. To demonstrate FluxBin’s broad applicability, we extend our evaluation to the latest LLaMA-3.1 and Qwen3 model families (detailed in Appendix D.2). In sensitive LLaMA-3.1 models, FluxBin exhibits remarkable robustness, achieving accuracy comparable to 3-bit baselines (e.g., GPTQ), outperforming AQLM (+17.5% accuracy) while maintaining leadership in inference throughput. This suggests that as models become more complex and sensitive, the structural robustness of our Hybrid Base strategy becomes increasingly critical. 4.3 Ablation Study Impact of Quantization Components. The upper section of Table 2 validates each algorithmic component. Removing column scaling significantly degrades PPL, while replacing Saliency Search with random selection causes a catastrophic failure (PPL 26.02), confirming the necessity of our Hessian-guided strategy. Relying solely on the Global Base increases PPL to 13.71, indicating that salient refinement is crucial for error compensation. Impact of Kernel Optimization. The lower section of Table 2 evaluates the benefits of the VCM strategy. Disabling the VCM strategy (treating refinement as sparse operations) results in an 18% speed drop and a 33% memory surge, validating VCM’s ability to ensure coalesced memory access. Table 2: Ablation study on LLaMA-2-7B (baseline: FluxBin-2b-s16-g128). Removing a component is denoted by ✗. Config. w/o Wiki PPL ↓ PIQA ↑ Baseline 8.20 75.50 Column Scaling ✗ 8.96 71.65 Saliency Search ✗ 26.02 66.92 Hybrid Base ✗ 13.71 69.79 Config. w/o Speed (tok/s) ↑ Max Mem. (MB) ↓ Baseline 250.87 3475.21 VCM ✗ 205.68 4640.37 Hyperparameter Sensitivity. (i) Group Size (g): As shown in Sec. D.3 Table 6, reducing the group size can improve model performance but result in higher storage for the scaling factor. Empirically, g=128 yields the optimal trade-off, which balances the competitive accuracy and average bit. (i) Calibration Samples (NcalN_cal): Following the GPTQ framework, we estimate the Hessian using NcalN_cal random samples from the C4 dataset. As shown in Sec. D.4 Table 7, accuracy follows a rise-then-fall trend: Ncal=256N_cal=256 captures sufficient curvature information, whereas larger sizes (>512>512) lead to overfitting and performance drops. (i) Sub-vector Size (μ): We demonstrate the impact of μ in Sec. E.2 and Table 9. The smaller μ minimizes the shared memory footprint, but increases the LUT reading operation, which is the main source of complexity. Despite larger LUTs, a larger μ without violating shared memory constraints achieves superior energy efficiency, and maintains almost constant inference latency with small μ. 4.4 Discussion: Sources of Acceleration We identify three sources behind FluxBin’s speedup that collectively explain why the framework retains advantages even at large batch sizes. Reduced Memory Traffic. A 2-base binary configuration stores weights at ∼ 2 bits (8×8× compression vs. FP16), directly relieving the HBM bottleneck that dominates autoregressive decoding. Reduced and Simplified Operations. The LUT kernel replaces μ FMA operations per sub-vector with a single shared-memory lookup, reducing complexity by μ/Kbμ/K_b (4×4× at default μ=8μ=8, Kb=2K_b=2; Appendix C.1). The remaining work is dominated by additions rather than FMAs, which enjoy higher pipeline throughput. Both reductions are batch-size independent. Trading ALU Computation for SRAM Access. Dequantization-based kernels spend ALU slots on bit-shift, mask, and type-convert instructions before every GEMV; FluxBin replaces this ALU-heavy path with shared-memory (SRAM) lookups. As SRAM bandwidth is independent of HBM, this offloads work from a contended resource (the ALU) without worsening the memory-bound bottleneck. Empirical Validation. Runtime profiling (Table 8, Appendix E.1) corroborates the analysis above. FluxBin consistently reports >>99.8% GPU busy time versus 67–87% for FP16. Its memory-bandwidth utilization is correspondingly low, reflecting the small weight traffic of the 2-bit format; together these indicate the LUT kernel is limited by on-chip lookup/compute throughput rather than stalled on HBM transfers (GPU busy time reflects occupancy, not arithmetic-unit saturation). The average power drop from 225W to ∼ 115W (13B) directly follows from replacing power-hungry FP16 FMAs with lightweight additions and SRAM lookups. Per-operator kernel latencies against QuIP# and AQLM, together with a tiling-parameter study, are reported in Appendix E.4 and Appendix E.3. Figure 5 (a) further breaks down per-block execution cycles: increasing MtileM_tile from 64 to 128 amortizes LUT Building cost (49%→ 38%), and the VCM translation overhead for hybrid configurations (2b-s8/s16) remains negligible. 5 Conclusion In this work, we propose FluxBin to bridge the gap between theoretical compression and realized acceleration. By synergizing Row-Column Binary Decomposition and Hessian-guided Hybrid Bases with a hardware-optimized LUT-based Kernel (LUT-BSF and VCM), we jointly enhance representational capacity and reduce runtime overhead. Across diverse architectures, FluxBin achieves up to 5.92× speedup and 10.19× energy savings while maintaining competitive accuracy, offering a training-free solution for efficient large-scale LLM deployment. Limitations While FluxBin demonstrates significant advancements in ultra-low-bit LLM inference, we identify two primary limitations. Shared Memory Constraints on Scalability. The efficiency of our LUT-based kernel relies heavily on caching pre-computed results in on-chip Shared Memory (SRAM). However, the LUT storage requirement scales exponentially with the sub-vector size (2μ2^μ) and linearly with the batch size per block. While high-end GPUs like the A100 provide ample SRAM (164 KB/SM) to support our optimal configuration (μ=8μ=8), deploying FluxBin on hardware with limited Shared Memory may necessitate reducing μ or the batch size. Optimization Bounds of Pure PTQ. FluxBin operates as a strict training-free framework to ensure rapid deployment. While it outperforms existing PTQ methods and rivals fine-tuned baselines, there remains a marginal accuracy gap compared to approaches that use extensive, computationally expensive iterative fine-tuning on specific tasks. Future work could explore integrating lightweight parameter-efficient fine-tuning on top of the binary bases to further bridge this gap without compromising the LUT-based inference speed. References Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. External Links: 2005.14165, Link Cited by: §1. Bulat et al. (2024) A. Bulat, Y. Ouali, and G. Tzimiropoulos QBB: quantization with binary bases for llms. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §1, §2. Chen et al. (2024) H. Chen, C. Lv, L. Ding, H. Qin, X. Zhou, Y. Ding, X. Liu, M. Zhang, J. Guo, X. Liu, and D. Tao DB-LLM: accurate dual-binarization for efficient LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, p. 8719–8730. External Links: Link, Document Cited by: §1, §2. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, Cited by: §1. Dettmers et al. (2024) T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, Cited by: §1. Egiazarian et al. (2024) V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh Extreme compression of large language models via additive quantization. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §4.2. Frantar and Alistarh (2023) E. Frantar and D. Alistarh SparseGPT: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §3.1. Frantar et al. (2022a) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh Gptq: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: Appendix B, §1, §1, §3.1, §4.2. Frantar et al. (2022b) E. Frantar, S. P. Singh, and D. Alistarh Optimal brain compression: a framework for accurate post-training quantization and pruning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §3.1. Huang et al. (2024) W. Huang, Y. Liu, H. Qin, Y. Li, S. Zhang, X. Liu, M. Magno, and X. Qi BiLLM: pushing the limit of post-training quantization for llms. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §1. Huang et al. (2025) W. Huang, H. Qin, Y. Liu, Y. Li, Q. Liu, X. Liu, L. Benini, M. Magno, S. Zhang, and X. Qi SliM-llm: salience-driven mixed-precision quantization for large language models. External Links: 2405.14917, Link Cited by: §1, §3.1, Definition 3.4, §4.2. Jacob et al. (2018) B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 2704–2713. External Links: Document Cited by: §1. Jeon et al. (2020) Y. Jeon, B. Park, S. J. Kwon, B. Kim, J. Yun, and D. Lee BiQGEMM: matrix multiplication with lookup table for binary-coding-based quantized dnns. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. External Links: ISBN 9781728199986 Cited by: §1, §2. Krishnamoorthy and Menon (2013) A. Krishnamoorthy and D. Menon Matrix inversion using cholesky decomposition. In 2013 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), Vol. , p. 70–72. External Links: Document Cited by: §3.1. Lee et al. (2019) J. Lee, C. Kim, S. Kang, D. Shin, S. Kim, and H. Yoo UNPU: an energy-efficient deep neural network accelerator with fully variable weight bit precision. IEEE Journal of Solid-State Circuits 54 (1), p. 173–185. External Links: Document Cited by: §1. Lee et al. (2023) J. H. Lee, J. Kim, S. J. Kwon, and D. Lee FlexRound: learnable rounding based on element-wise division for post-training quantization. External Links: 2306.00317, Link Cited by: §D.2. Li et al. (2023) S. Li, X. Ning, K. Hong, T. Liu, L. Wang, X. Li, K. Zhong, G. Dai, H. Yang, and Y. Wang Llm-mq: mixed-precision quantization for efficient llm deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurIPS, Vol. 9, p. 3. Cited by: §1. Li et al. (2024) Z. Li, X. Yan, T. Zhang, H. Qin, D. Xie, J. Tian, Z. Shi, L. Kong, Y. Zhang, and X. Yang ARB-llm: alternating refined binarizations for large language models. ArXiv abs/2410.03129. External Links: Link Cited by: §1, §2. Lin et al. (2025) J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han AWQ: activation-aware weight quantization for on-device llm compression and acceleration. 28 (4), p. 12–17. External Links: ISSN 2375-0529 Cited by: §1. Maleki (2023) S. Maleki Look-up mai gemm: increasing ai gemms performance by nearly 2.5x via msgemm. External Links: 2310.06178, Link Cited by: §1. Markidis et al. (2018) S. Markidis, S. W. D. Chien, E. Laure, I. B. Peng, and J. S. Vetter NVIDIA tensor core programmability, performance & precision. In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Vol. , p. 522–531. External Links: Document Cited by: §1. Mo et al. (2025) Z. Mo, L. Wang, J. Wei, Z. Zeng, S. Cao, L. Ma, N. Jing, T. Cao, J. Xue, F. Yang, and M. Yang LUT tensor core: a software-hardware co-design for lut-based low-bit llm inference. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25, New York, NY, USA, p. 514–528. External Links: ISBN 9798400712616, Link, Document Cited by: §2. Ningning Chen (2025) Y. J. Ningning Chen HBLLM: wavelet-enhanced high-fidelity 1-bit quantization for llms. arXiv preprint arXiv:2512.00862. Cited by: §1, §2. Park et al. (2024) G. Park, B. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y. Lee, and D. Lee LUT-gemm: quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. External Links: 2206.09557, Link Cited by: §C.1, §1, §2, §3.2. Polino et al. (2018) A. Polino, R. Pascanu, and D. Alistarh Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668. Cited by: §1. Qin et al. (2023) H. Qin, M. Zhang, Y. Ding, A. Li, Z. Cai, Z. Liu, F. Yu, and X. Liu BiBench: benchmarking and analyzing network binarization. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §2. Rastegari et al. (2016) M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi XNOR-net: ImageNet classification using binary convolutional neural networks. In Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling (Eds.), p. 525–542. External Links: ISBN 978-3-319-46493-0 Cited by: §1, §2. Shang et al. (2023) Y. Shang, Z. Yuan, Q. Wu, and Z. Dong PB-llm: partially binarized large language models. ArXiv abs/2310.00034. External Links: Link Cited by: §1. Shao et al. (2024) W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, G. Peng, Y. Qiao, and P. Luo OmniQuant: omnidirectionally calibrated quantization for large language models. In International Conference on Representation Learning, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 45472–45496. Cited by: §4.2. Tang et al. (2023) X. Tang, Y. Wang, T. Cao, L. L. Zhang, Q. Chen, D. Cai, Y. Liu, and M. Yang LUT-n: empower efficient neural network inference with centroid learning and table lookup. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, ACM MobiCom ’23, New York, NY, USA. External Links: ISBN 9781450399906 Cited by: §1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Link Cited by: §1, §4.2. Tseng et al. (2024) A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa QuIP#: even better llm quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §4.2. Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, p. 38–45. Cited by: §4. Xiao et al. (2025) H. Xiao, Q. Yang, D. Xie, W. Xu, Z. Su, R. yang, W. Zhou, H. Liu, Z. Liu, and N. Wong Exploring layer-wise information effectiveness for post-training quantization in small language models. External Links: 2508.03332, Link Cited by: §1. Xiao et al. (2026) H. Xiao, R. Yang, Q. Yang, W. Xu, Z. Li, Y. Su, Z. Liu, H. Yang, and N. Wong PTQTP: post-training quantization to trit-planes for large language models. External Links: 2509.16989, Link Cited by: §1. Xu et al. (2022) S. Xu, Q. Wang, X. Wang, S. Wang, and T. T. Ye Multiplication through a single look-up-table (lut) in cnn inference computation. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 41 (6), p. 1916–1928. External Links: Document Cited by: §2. Yang et al. (2025) Q. Yang, X. Wang, Y. Zhou, Q. Li, and S. Qiao Hardware friendly transformer optimization with dynamic attention matrix fusion. In 2025 IEEE International Symposium on Circuits and Systems (ISCAS), Vol. , p. 1–5. External Links: Document Cited by: §1. Zeng et al. (2023) A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y. Xue, J. Zhai, W. Chen, P. Zhang, Y. Dong, and J. Tang GLM-130b: an open bilingual pre-trained model. External Links: 2210.02414, Link Cited by: §1. Zhang et al. (2018) D. Zhang, J. Yang, D. Ye, and G. Hua LQ-nets: learned quantization for highly accurate and compact deep neural networks. In Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VIII, Berlin, Heidelberg, p. 373–390. External Links: ISBN 978-3-030-01236-6, Link, Document Cited by: §2. Appendix A Derivation of Parameter Optimization In this section, we provide the rigorous mathematical proofs for the optimal update rules presented in the Methodology section. A.1 Derivation of Optimal Scaling Factors Recall the objective function for a single decomposition order k: ℒ=‖−(rc⊤)⊙‖F2L= \|W- ( α_r α_c ) \|_F^2 (16) Proof of Proposition 3.2.. We aim to derive the closed-form solution for the row scaling vector r α_r while keeping B and c α_c fixed. Since the Frobenius norm is the sum of squared element-wise errors, Eq. (16) can be expanded as: ℒ=∑i=1M∑j=1N(ij−r,i⋅c,j⋅ij)2L= _i=1^M _j=1^N (W_ij- α_r,i· α_c,j·B_ij )^2 (17) The optimization problem for r α_r decouples into M independent scalar least-squares problems. For the i-th row scalar r,i α_r,i, the partial derivative is: ∂ℒ∂r,i ∂ α_r,i =∂r,i∑j=1N(ij−r,i(c,jij))2 = ∂ α_r,i _j=1^N (W_ij- α_r,i( α_c,jB_ij) )^2 (18) =−2∑j=1N(ij−r,i(c,jij))⋅(c,jij) =-2 _j=1^N (W_ij- α_r,i( α_c,jB_ij) )·( α_c,jB_ij) Setting the derivative to zero to find the critical point: ∑j=1Nij(c,jij)=r,i∑j=1N(c,jij)2 _j=1^NW_ij( α_c,jB_ij)= α_r,i _j=1^N( α_c,jB_ij)^2 (19) Solving for r,i α_r,i, we obtain: r,i=∑j=1N(c,j⋅ij)⋅ij∑j=1N(c,j⋅ij)2+ϵ α_r,i= _j=1^N( α_c,j·B_ij)·W_ij _j=1^N( α_c,j·B_ij)^2+ε (20) where ϵε is added for numerical stability. This matches Eq. (5). Due to the symmetry of the rank-1 structure, the derivation for the column scale c α_c follows an identical logic by taking the derivative w.r.t c,j α_c,j, yielding the column-scale expression in Eq. (5). ∎ A.2 Derivation of Joint Binary Basis Optimization When Kb≥2K_b≥ 2, we optimize the binary bases (1:Kb)B^(1:K_b) jointly. Proof of Proposition 3.3.. We define the magnitude tensor for the k-th basis as (k)=r(k)c(k)⊤M^(k)= α_r^(k) α_c^(k) . The global reconstruction error is: ℒ=‖−∑k=1Kb(k)⊙(k)‖F2L= \|W- _k=1^K_bM^(k) ^(k) \|_F^2 (21) Crucially, the elements of (k)B^(k) are independent across spatial coordinates (i,j)(i,j). Thus, minimizing the global Frobenius norm is equivalent to minimizing the squared error for each element independently: min(k)⁡ℒ⇔∑i,jminbij(1),…,bij(Kb)⁡(ij−∑k=1Kbij(k)⋅bij(k))2 _\B^(k)\L _i,j _b_ij^(1),…,b_ij^(K_b) (W_ij- _k=1^K_bM_ij^(k)· b_ij^(k) )^2 (22) For a specific coordinate (i,j)(i,j), let =[b(1),…,b(Kb)]⊤∈−1,+1Kbb=[b^(1),…,b^(K_b)] ∈\-1,+1\^K_b be the vector of binary choices. The problem reduces to: ∗=argmin∈−1,+1Kb(ij−∑k=1Kbij(k)⋅b(k))2 ^*= b∈\-1,+1\^K_bargmin (W_ij- _k=1^K_bM_ij^(k)· b^(k) )^2 (23) Since the square function is monotonic for positive errors, minimizing the squared L2 error is equivalent to minimizing the absolute L1 error in this scalar context. Thus, we have: ij(1:Kb)=argmin∈−1,+1Kb|ij−∑k=1Kbij(k)⋅bk| _ij^(1:K_b)= b∈\-1,+1\^K_bargmin |W_ij- _k=1^K_bM_ij^(k)· b_k | (24) This confirms that the element-wise exhaustive search (Eq. (6)) yields the globally optimal binary assignment for the fixed scaling factors. ∎ Appendix B Algorithmic Descriptions FluxBin is seamlessly integrated into the GPTQ framework 8, adopting a block-wise reconstruction paradigm to ensure high inference efficiency and quantization fidelity. By leveraging the Optimal Brain Quantization (OBQ) theory, we utilize the inverse Hessian matrix −1H^-1 not only for structural saliency search but also for compensating the quantization error accumulating across columns. The Hessian is computed efficiently using calibration data via =2TH=2XX^T, and its inverse is updated iteratively, ensuring the process remains computationally lightweight. We provide two algorithmic variants corresponding to the configurations evaluated in the main paper: FluxBin Base (Algorithm 1). This algorithm describes the standard global base quantization process without mixed-precision refinement. It iterates through column groups, performing the proposed Row-Column Binary Decomposition to obtain the quantized block ^blk W_blk. The residual error is then propagated to the remaining unquantized weights using the Hessian inverse. This algorithm is utilized for pure binary configurations, specifically 2b-g128 (Kb=2K_b=2) and 4b-g128 (Kb=4K_b=4), where no salient refinement is applied. FluxBin Hybrid Base (Algorithm 2). This algorithm details the salient-aware hybrid base (mixed-precision) strategy. Within each block, it first identifies critical columns via StructuralSearch using Hessian metrics. Following the global base decomposition, it performs a secondary decomposition on the residual blkR_blk restricted to the masked salient columns. This algorithm is utilized for our primary hybrid configurations, specifically 2b-s8-g128 (s=8s=8) and 2b-s16-g128 (s=16s=16). Sub-routines (Algorithm 3). This section defines the core components: RC_BinaryDecomp, which solves the optimization problem for decoupled scaling factors and binary bases (Eq. (5)-(6)), and StructuralSearch, which computes the column-wise sensitivity score (Eq. (7)). Algorithm 1 FluxBin-Base (Global Base) 1: Input: W, Calib X, Group g, Bases KbK_b, Iter T 2: Output: W, Params Θ 3: ←2TH← 2XX^T Hessian from input 4: −1←CholeskyInverse()H^-1 (H) 5: ←Q 0 6: for j=0,g,…,N−gj=0,g,…,N-g do 7: blk←:,j:j+gW_blk _:,j:j+g 8: blk−1←(−1)j:j+g,j:j+gH^-1_blk←(H^-1)_j:j+g,j:j+g 9: Global Base Decomp 10: ^blk,Θ←RC_Decomp(blk,,Kb,T) W_blk, ← RC\_Decomp(W_blk,1,K_b,T) 11: :,j:j+g←^blkQ_:,j:j+g← W_blk 12: Error Compensation 13: ←blk−^blkE _blk- W_blk 14: Update remainingW_remaining via ,−1E,H^-1 15: end for 16: return ,ΘQ, Algorithm 2 FluxBin-Hybrid (Salient-Aware) 1: Input: ,,gW,X,g, Bases Kb,KsK_b,K_s, Salient Nsal,TN_sal,T 2: Output: W, Θ , Φ , Mask ℳM 3: ←2TH← 2XX^T; −1←CholInv()H^-1 (H) 4: for j=0,g,…,N−gj=0,g,…,N-g do 5: blk←:,j:j+gW_blk _:,j:j+g 6: blk−1←(−1)j:j+g,j:j+gH^-1_blk←(H^-1)_j:j+g,j:j+g 7: Step 1: Saliency Search 8: ℐ←StructuralSearch(blk,blk−1,Nsal)I← StructuralSearch(W_blk,H^-1_blk,N_sal) 9: Construct mask ℳblkM_blk from ℐI 10: Step 2: Global Base 11: ^g,Θ←RC_Decomp(blk,,Kb,T) W_g, ← RC\_Decomp(W_blk,1,K_b,T) 12: Step 3: Salient Refinement 13: blk←blk−^gR_blk _blk- W_g 14: ^s,Φ←RC_Decomp(blk,ℳblk,Ks,T) W_s, ← RC\_Decomp(R_blk,M_blk,K_s,T) 15: ^blk←^g+^s W_blk← W_g+ W_s 16: Step 4: Compensation 17: ←blk−^blkE _blk- W_blk 18: Update remainingW_remaining via ,−1E,H^-1 19: end for 20: return ^,Θ,Φ,ℳ W, , ,M Algorithm 3 FluxBin Sub-routines 1: Function RC_Decomp(,mask,K,TX,mask,K,T) 2: Init residual ←⊙maskR 3: ^← X 0 4: // Init via Greedy Residual 5: for k=1k=1 to K do 6: Init (k),r(k),c(k)B^(k), α_r^(k), α_c^(k) from R 7: ←−(r(k)c(k)⊤)⊙(k)R -( α_r^(k) α_c^(k) ) ^(k) 8: end for 9: // Alternating Optimization 10: for iter=1iter=1 to T do 11: for k=1k=1 to K do 12: Update r(k) α_r^(k) (Eq. 5) 13: Update c(k) α_c^(k) (Eq. 5) 14: end for 15: Update (k)\B^(k)\ via search (Eq. 6) 16: for k=1k=1 to K do 17: (k)←(k)⊙maskB^(k) ^(k) 18: end for 19: end for 20: Reconstruct X 21: return ^,, X,\ α,B\ 22: End Function 1: Function StructuralSearch 2: Input: ,−1,NsalW,H^-1,N_sal 3: // Compute Sensitivity 4: for j=1j=1 to g do 5: Sj=∑ij2[−1]jj2S_j= _i W_ij^2[H^-1]_j^2 6: end for 7: ℐ←TopK(Sj,Nsal)I (\S_j\,N_sal) 8: return ℐI 9: End Function Appendix C Cost Estimation C.1 Compute Complexity Analysis of LUT-base Kernel To quantify the computational efficiency of FluxBin, we analyze the theoretical complexity of our LUT-based kernel. Following the standard analysis in LUT-GEMM 24, the total cost C consists of the LUT construction cost (CbuildC_build) and the table lookup cost (CreadC_read). Assume the weight matrix has dimensions M×NM× N, the sub-vector length is μ, and the number of binary bases is K. Since the column scaling factor c(k) α_c^(k) is unique for each basis k and is fused into the input, independent LUTs must be constructed for each basis. 1. Global Base Complexity. For the global base branch with KbK_b bases, the input dimension is N. Cbuildbase C_build^base =Kb⋅Nμ⋅2μ =K_b· Nμ· 2^μ (25) Creadbase C_read^base =Kb⋅M⋅Nμ =K_b· M· Nμ Here, Nμ Nμ represents the number of sub-vectors, and 2μ2^μ is the number of entries per LUT. 2. Salient Refinement Complexity. For the salient refinement branch, the operation is restricted to NsalN_sal columns with KsK_s bases. Cbuildsal C_build^sal =Ks⋅Nsalμ⋅2μ =K_s· N_salμ· 2^μ (26) Creadsal C_read^sal =Ks⋅M⋅Nsalμ =K_s· M· N_salμ 3. Total Complexity. The overall complexity is the sum of both branches: Ctotal=2μ(KbN+KsNsal)⏟LUT Build+Mμ(KbN+KsNsal)⏟LUT Read C_total= 2^μ(K_bN+K_sN_sal)_LUT Build+ Mμ(K_bN+K_sN_sal)_LUT Read (27) In typical generative LLM scenarios, the output channel dimension M is significantly larger than the LUT size 2μ2^μ (e.g., M≫2μM 2^μ). Consequently, the lookup cost CreadC_read dominates the total runtime. The complexity can be approximated as: Ctotal≈⁡(Mμ(KbN+KsNsal))C_total ( Mμ(K_bN+K_sN_sal) ) (28) Efficiency Gain. Compared to standard FP16 matrix multiplication which has a complexity of ⁡(MN)O(MN), FluxBin achieves a theoretical reduction factor of approximately μKb μK_b. Furthermore, since the salient refinement typically involves a very sparse subset of columns (Nsal≪N_sal N), the additional overhead introduced by the hybrid precision is marginal, preserving the high-throughput advantage of the LUT-based paradigm. C.2 Storage Consumption Analysis We conduct a comprehensive analysis of the storage overhead introduced by FluxBin, evaluating the Total Storage (StotalS_total) and the Effective Average Bit-Width (ABW). Let M and N denote the number of rows and columns of the weight matrix W, respectively. Our configuration divides columns into groups of size g, and selects s salient columns per group. The quantization model consists of KbK_b binary bases for the global approximation and KsK_s bases for the salient refinement. We assume all scaling factors (row and column) and indices are stored in FP16 format (16 bits). 1. Global Base Storage (SbaseS_base). The global base component stores the dense binary weights, column-wise scaling factors, and block-wise row scaling factors. The storage cost (in bits) is calculated as: Sbase=Kb⋅(MN⏟Weights+16N⏟Col Scales+16⋅M⋅Ng⏟Row Scales) S_base=K_b· ( MN_Weights+ 16N_Col Scales+ 16· M· Ng_Row Scales ) (29) where the term M⋅NgM· Ng reflects our strategy of storing one row scale per group to capture local magnitude variations. 2. Salient Refinement Storage (SsalS_sal). The salient refinement targets a subset of columns, denoted as Nsal=Ng⋅sN_sal= Ng· s. This component requires storage for the sparse binary refinements, their corresponding scales, and the indexing metadata. Ssal=Ks⋅(MNsal+16Nsal+16⋅M⋅Ng)+Sidx S_sal=K_s· (MN_sal+16N_sal+16· M· Ng )+S_idx (30) The term SidxS_idx accounts for the overhead of storing the indices of salient columns. Since we select s indices for each of the N/gN/g groups, the overhead is: Sidx=16⋅NsalS_idx=16· N_sal (31) Note that we use 16 bits per index to ensure memory alignment and support large column dimensions, although 8 bits could suffice for smaller group sizes (e.g., g≤256g≤ 256). 3. Effective Average Bit-Width (ABW). Aggregating the components from Eq. (29) and Eq. (30), the total storage is Stotal=Sbase+SsalS_total=S_base+S_sal. The effective average bit-width is derived as: ABW=StotalMN=Sbase+SsalMNABW= S_totalMN= S_base+S_salMN (32) For our typical configuration (e.g., Kb=2K_b=2, Ks=2K_s=2, g=128g=128, s=8s=8), the salient branch introduces a sparsity ratio of s/g=6.25%s/g=6.25\%. The indexing overhead SidxS_idx is amortized over the large matrix dimensions (≈16⋅s/gM≈ 16· s/gM bits per weight), so the effective ABW settles at ≈2.63≈ 2.63 bits (and ≈2.75≈ 2.75 bits for s=16s=16) while significantly boosting model accuracy. C.3 Maximum Memory Consumption Table 3 presents the peak GPU memory allocation during inference. Comparison with FP16 Baseline: FluxBin significantly reduces memory footprints compared to the FP16 baseline. For instance, on Qwen3-32B, the memory usage drops from 62.7GB (FP16) to 12.1GB (2b-g128), achieving a 5.1×5.1×compression ratio. Notably, for LLaMA-2-70B, which triggers an Out-Of-Memory (OOM) error in FP16 on our testing hardware, FluxBin enables successful inference with only 20.2GB of memory in the 2-bit configuration, demonstrating its capability to democratize large model deployment on consumer-grade GPUs. Furthermore, FluxBin consistently achieves comparable memory usage to other 2-bit baselines like AQLM and QuIP#. Impact of Model Configuration: The memory consumption correlates with the complexity of the binary decomposition: • Binary Bases: Increasing the global quantization precision from 2-bit to 4-bit naturally increases memory usage. For LLaMA-2-7B, the usage rises from 2.9GB (2b-g128) to 4.7GB (4b-g128), reflecting the storage of additional binary bases. • Salient Refinement: Incorporating salient refinement (s8/s16) introduces a moderate memory overhead compared to the standard 2b configuration due to the storage of the auxiliary dense matrix and indices. However, it is worth noting that the difference between 2b-s8 and 2b-s16 is negligible (e.g., <1<1 MB for LLaMA-2-7B). This confirms that our Virtual Columnar Mapping strategy is highly memory-efficient: doubling the number of salient columns incurs minimal additional cost, as the overhead is dominated by fixed allocations (e.g., KV cache and activation buffers) rather than the compact salient weights themselves. Table 3: Max GPU Memory (MB) Allocated in Inference with Input Length 512 and Output Length 512. Baselines FluxBin Model Name FP16 AQLM 2bit QuIP# 2bit 2b-g128 2b-s8-g128 2b-s16-g128 4b-g128 LLaMA-2-7B 13330.61 5247.35 3461.87 2895.24 3474.30 3475.21 4684.38 LLaMA-2-13B 25681.85 4058.02 5785.77 4558.59 6172.64 6173.32 8474.27 LLaMA-2-70B OOM 20415.70 19360.84 20251.61 26393.52 26393.13 38633.95 Qwen3-8B 15779.09 – – 4557.90 5185.48 5187.33 6444.57 Qwen3-14B 28322.55 – – 7043.43 8215.49 8221.31 10486.19 Qwen3-32B 62720.77 – – 12135.80 14993.71 14995.25 20323.56 Appendix D Extension Experiment D.1 Extension on Throughput To comprehensively validate the generality of FluxBin, we extend our throughput evaluation to two representative model families: Qwen3 (8B, 14B, 32B), which features Grouped-Query Attention (GQA), and LLaMA-3.1 (8B, 70B), representing the latest generation of massive-scale models. As illustrated in Figure 6 and Figure 7, the results exhibit remarkable consistency. Crucially, the throughput for hybrid precision configurations (2b-s8-g128 and 2b-s16-g128) remains virtually indistinguishable from the pure 2-bit baseline (2b-g128) across all cases. This empirical evidence reinforces two critical attributes of FluxBin: • Architectural Robustness: FluxBin proves to be architecture-agnostic. Whether applied to standard architectures or modern designs utilizing GQA (as in Qwen3), our kernel integrates seamlessly, delivering consistent acceleration without being hindered by specific architectural variations. • Scalability to Massive Models: The method demonstrates exceptional scalability. On the massive LLaMA-3.1-70B, FluxBin maintains the same high-efficiency profile as on smaller models. This confirms that the overhead of our mixed-precision strategy does not grow disproportionately with model size, making it an ideal solution for deploying large-scale foundation models. Figure 6: Throughput Comparison on LLaMA-3.1 Across All Batch Sizes Figure 7: Throughput Comparison on Qwen3 Across All Batch Sizes D.2 Extended Evaluation on Speed vs. Accuracy To comprehensively validate the generalization capability of FluxBin across diverse and modern LLM architectures, we extend our evaluation to the LLaMA-3.1 family (8B-Instruct and 70B) and the Qwen3 family (8B and 14B). The results are summarized in Table 4 and Table 5. Robustness on LLaMA-3.1. LLaMA-3.1 poses significant challenges for low-bit quantization due to its complex activation distributions and high sensitivity. As shown in Table 4, existing 2-bit methods like FlexRound 16 and AQLM suffer severe accuracy degradation, particularly on the 70B model where their average accuracy drops to 36.18% and 50.27%, respectively. In stark contrast, FluxBin maintains exceptional fidelity. Our 2b-s16-g128 configuration achieves an average accuracy of 67.81%, outperforming AQLM by +17.5% and FlexRound by +31.6%. This underscores the effectiveness of our Hessian-guided hybrid base strategy in preserving critical information even in the most sensitive models. In terms of inference speed, FluxBin continues to lead. On LLaMA-3.1-8B, it reaches 217.03 tokens/s, surpassing AQLM (124.50 tokens/s) by 1.74×. Even on the massive 70B model, FluxBin achieves 55.69 tokens/s, significantly faster than AQLM (19.00 tokens/s) and competitive with the higher-precision GPTQ (3-bit). Adaptability to Qwen3 Architectures. We further evaluate FluxBin on the Qwen3 family to assess its adaptability to architectures employing Grouped-Query Attention (GQA). As detailed in Table 5, FluxBin demonstrates strong performance retention. Compared to the SLiM-LLM baseline, FluxBin achieves substantial accuracy gains (e.g., +18.3% on 8B Avg). More importantly, despite using a lower bit-width (∼ 2.75 bits), FluxBin’s accuracy is highly competitive with the 3.125-bit GPTQ baseline (e.g., 69.39% vs. 70.45% on 14B), while offering the theoretical throughput advantages of binary kernels. This confirms that FluxBin’s decoupled row-column decomposition and salient-aware refinement are architecture-agnostic, effectively handling modern LLM designs. Table 4: Comparison of quantization methods on LLaMA-3.1 models. Higher accuracy and speed are better (↑ ). ∗ HS = HellaSwag, WG = Winogrande. †MMLU: Evaluated using 5-shot in-context learning. We highlight the best, second, and third results (excluding FP16). LLaMA-3.1 Method #W bit Speed (tok/s) ↑ ARC-e ↑ ARC-c ↑ HS∗ ↑ WG∗ ↑ MMLU† ↑ Avg. ↑ 8B-Instruct FP16 16 41.51 82.15 52.05 60.77 74.35 69.40 67.74 FlexRound 2.125 - 24.75 24.57 43.78 55.16 24.27 34.51 AQLM 2 124.50 46.25 30.89 61.40 58.25 42.29 47.82 GPTQ 3.125 – 72.14 37.97 53.73 68.75 46.90 55.90 FluxBin-2b-s8-g128 2.63 217.02 69.23 35.41 45.16 68.19 43.90 52.38 FluxBin-2b-s16-g128 2.75 217.03 69.91 36.77 44.49 68.43 45.15 52.95 70B FP16 16 OOM 86.66 64.93 85.03 79.64 78.58 78.97 FlexRound 2.125 - 25.13 24.83 50.67 53.59 26.70 36.18 AQLM 2 19.00 48.82 28.67 52.83 59.59 61.45 50.27 GPTQ 3.125 – 80.72 51.79 63.63 79.72 72.87 69.75 FluxBin-2b-s8-g128 2.63 58.38 81.27 49.66 58.06 79.01 64.71 66.54 FluxBin-2b-s16-g128 2.75 55.69 82.73 53.24 57.43 79.48 66.17 67.81 Table 5: Comparison of quantization methods on Qwen3 models. Higher accuracy and speed are better (↑ ). ∗ HS = HellaSwag, WG = Winogrande. We highlight the best, second, and third results (excluding FP16). Qwen3 Method #W bit Speed (tok/s) ↑ Wiki ↓ PIQA ↑ ARC-e ↑ ARC-c ↑ BoolQ ↑ HS∗ ↑ WG∗ ↑ Avg. ↑ 8B FP16 16 28.60 9.72 76.39 83.29 55.38 86.57 57.16 68.03 71.14 SLiM-LLM 2 - 30.61 60.11 43.81 23.97 62.99 34.83 55.80 46.92 GPTQ 3.125 - 11.18 74.16 77.90 48.63 84.07 53.42 65.43 67.27 FluxBin-2b-s8-g128 2.63 121.85 13.46 74.37 74.41 42.58 83.33 47.90 66.30 64.82 FluxBin-2b-s16-g128 2.75 121.97 12.44 74.21 74.71 43.77 82.75 49.09 66.77 65.22 14B FP16 16 26.54 8.64 80.36 84.13 58.62 89.42 60.94 72.69 74.36 Slim-LLm 2 - 22.85 61.83 52.54 29.35 61.20 31.52 52.04 48.08 GPTQ 3.125 - 9.73 77.42 79.12 52.82 86.64 58.02 68.67 70.45 FluxBin-2b-s8-g128 2.63 102.07 11.60 74.59 77.74 51.96 87.43 53.96 69.30 69.16 FluxBin-2b-s16-g128 2.75 102.09 11.42 76.17 78.66 51.62 86.70 53.74 69.46 69.39 D.3 Impact of Group Size We investigate the impact of group size (g) on the trade-off between model accuracy and effective bit-width. As detailed in Table 6, we evaluate three granularities (g=64, 128, 256) across pure and hybrid configurations. Sensitivity of Global Base. The pure binary configuration (2b) exhibits high sensitivity to group granularity. Increasing g from 64 to 256 results in a severe degradation in average accuracy, dropping from 57.96% to 47.20%. This indicates that a coarse-grained global binary approximation struggles to capture the local weight variance without refinement. Robustness of Hybrid Bases. In contrast, our salient-aware hybrid configurations (2b-s8 and 2b-s16) demonstrate remarkable robustness to larger group sizes. For instance, 2b-s16 maintains a high average accuracy of 58.87% even at g=256, significantly outperforming the pure 2b baseline at g=64 (57.96%). This confirms that our salient refinement effectively compensates for the information loss caused by coarse-grained grouping. Optimal Configuration. While g=64 yields the highest accuracy, it inflates the effective bit-width to ∼ 3.5 bits (for s16). Conversely, g=128 strikes the optimal balance, achieving competitive accuracy (59.15%) with a compact footprint (∼ 2.75 bits), justifying its selection as our default setting. Table 6: Ablation study across different group sizes (g) in fixed configurations of LLaMA-2-7B. Lower Wiki PPL is better, while higher accuracy is better for downstream tasks. † The average bit includes weight and scale factors. The calculation for FluxBin is derived in Sec. C.2. ‡ HS = HellaSwag, WG = Winogrande. Config. W bit† Wiki ↓ PIQA ↑ ARC-e ↑ ARC-c ↑ BoolQ ↑ HS‡ ↑ WG‡ ↑ Avg. ↑ 2b-g64 2.51 10.27 73.50 63.43 30.29 67.00 48.57 64.96 57.96 2b-g128 2.25 13.71 69.79 56.23 25.00 65.41 38.64 56.69 51.96 2b-g256 2.13 12.79 64.91 48.91 22.35 62.08 34.52 50.43 47.20 2b-s8-g64 3.25 7.95 73.01 67.59 32.51 73.70 47.05 66.06 59.99 2b-s8-g128 2.63 8.76 72.52 65.45 30.97 69.88 45.64 64.09 58.09 2b-s8-g256 2.31 8.91 72.47 64.98 30.46 71.25 44.97 64.88 58.17 2b-s16-g64 3.50 7.40 74.76 70.54 34.22 72.72 48.26 64.96 60.91 2b-s16-g128 2.75 8.20 75.50 65.40 32.42 70.80 45.97 64.80 59.15 2b-s16-g256 2.38 8.64 73.34 65.74 31.48 72.66 46.06 63.93 58.87 D.4 Sensitivity to Calibration Sample Size We investigate the impact of the calibration c4 dataset size (NcalN_cal) on both quantization performance and computational overhead for LLaMA-2-7B FluxBin-2b-s8-g128. As shown in Table 7, we vary NcalN_cal from 64 to 1024. The results reveal a distinct "rise-then-fall" trend in model performance. Hessian Estimation vs. Overfitting. The performance trajectory can be explained through the approximation of the Hessian matrix ≈⁡[T]H [xx^T], which guides the quantization objective Δℒ≈12ΔTΔ ≈ 12 ^TH . • Under-sampling (Ncal<256N_cal<256): When samples are scarce, the Hessian estimate only captures the principal directions of activation magnitudes. Increasing NcalN_cal from 64 to 256 significantly improves the average accuracy (57.86% → 58.77%) and reduces Wiki PPL (8.92 → 8.63). This gain stems from a more stable and representative estimation of H, allowing the algorithm to better identify sensitive weights. • Calibration Overfitting (Ncal>512N_cal>512): Surprisingly, further increasing NcalN_cal to 1024 leads to performance degradation (Avg. Acc drops to 58.07%). This phenomenon is calibration overfitting, occurs when the Hessian captures fine-grained distribution details specific to the calibration set that do not generalize to the test distribution. The quantization decision boundary becomes overly biased towards the calibration data, harming generalization. Efficiency Trade-off. In terms of cost, the PTQ time increases linearly with NcalN_cal. While Ncal=1024N_cal=1024 incurs a lengthy calibration time (∼ 75 min), Ncal=256N_cal=256 offers the optimal trade-off, achieving peak accuracy with a moderate time cost (∼ 25 min). Consequently, we adopt Ncal=256N_cal=256 as the default setting for our main experiments. Table 7: Ablation study of the number of calibration samples (nsample) on evaluation time and performance on LLaMA-2-7B, with FluxBin configuration 2b-s8-g128. Lower PTQ time and Wiki PPL are better (↓ ), while higher accuracy is better (↑ ). ‡ HS = HellaSwag, WG = Winogrande. nsample Time (min) ↓ Wiki ↓ PIQA ↑ ARC-e ↑ ARC-c ↑ BoolQ ↑ HS‡ ↑ WG‡ ↑ Avg. ↑ 64 13.22 8.92 72.31 64.60 30.55 70.06 45.61 64.01 57.86 128 17.02 8.85 72.74 65.49 30.97 69.88 45.38 63.54 58.00 256 25.53 8.63 74.10 64.44 33.28 71.13 45.51 64.17 58.77 512 41.78 8.54 73.34 65.78 32.00 71.13 45.89 63.30 58.57 1024 74.58 8.76 72.63 65.40 30.80 69.82 45.67 64.09 58.07 Appendix E Profiling Study E.1 Energy Efficiency and Resource Utilization To evaluate the environmental impact and deployment cost of FluxBin, we profile power consumption and hardware utilization on an NVIDIA A100 GPU using nvidia-smi at 100ms sampling intervals during steady-state decode generation (excluding the first 10s of warmup). Total energy is computed as E=∑tP⁡(t)⋅ΔtE= _tP(t)· t. Table 8 summarizes the results across the LLaMA-2 family, with input and output length 128, batch size 1. Energy Consumption. FluxBin demonstrates a dramatic reduction in total energy cost (Joules). For LLaMA-2-13B, FluxBin reduces total energy consumption by over 10.19× compared to the FP16 baseline (579.23J vs. 5908.03J). This massive efficiency gain stems from two factors: (i) the significant reduction in inference latency (as discussed in Sec. 4.2), which reduces the total active time of the GPU; and (i) lower average power draw (∼ 115W vs. 225W). The reduced power draw is attributed to the minimized data movement overhead—since memory access energy typically dominates arithmetic energy—and the use of lightweight bitwise/integer operations instead of power-hungry FP16 arithmetic. Hardware Utilization. A notable observation is the GPU busy time reported by nvidia-smi, where FluxBin consistently stays near >>99.8%, whereas the FP16 baseline hovers around 67%–86%. We stress that this is a device-busy fraction, not a measure of arithmetic-unit saturation; read together with the very low memory utilization below, it indicates that the LUT kernel keeps the SMs continuously occupied and is not stalled waiting on HBM traffic, unlike the FP16 baseline. Simultaneously, the memory-bandwidth utilization remains low (e.g., ∼ 6% for 7B). This is not a performance target in itself; it reflects the small weight traffic of the compressed 2-bit format and indicates that the kernel is limited by on-chip LUT/compute throughput rather than HBM bandwidth. For LLaMA-2-70B, where FP16 fails due to OOM, FluxBin operates efficiently with high GPU utilization and stable power consumption (∼ 155W), further validating its scalability for massive model deployment. Table 8: Power, energy, and resource utilization comparison on LLaMA-2 models. Lower power and energy are better (↓ ); GPU and memory utilization are reported as diagnostics of the compute/bandwidth balance rather than as quality targets. Model Quantization Avg Power (W) ↓ Total Energy (J) ↓ Avg GPU Util. (%) Avg Mem. Util. (%) LLaMA-2-7B FP16 168.66 3467.69 67.01 18.09 FluxBin-2b-g128 106.75 462.98 99.83 5.55 FluxBin-2b-s8-g128 110.80 425.05 99.81 6.27 FluxBin-2b-s16-g128 107.40 434.34 99.87 6.27 FluxBin-4b-g128 109.40 401.16 99.85 7.65 LLaMA-2-13B FP16 225.49 5908.03 86.83 33.16 FluxBin-2b-g128 113.86 579.23 99.89 8.15 FluxBin-2b-s8-g128 115.82 603.46 99.98 9.69 FluxBin-2b-s16-g128 115.07 644.74 99.93 9.69 FluxBin-4b-g128 112.57 560.84 99.89 12.45 LLaMA-2-70B FP16 OOM OOM OOM OOM FluxBin-2b-g128 150.22 2522.47 99.95 26.54 FluxBin-2b-s8-g128 157.08 2441.06 99.98 34.19 FluxBin-2b-s16-g128 154.56 2560.80 99.98 34.19 FluxBin-4b-g128 159.68 2477.17 99.96 49.14 E.2 Shared Memory Footprint Analysis Efficient utilization of Shared Memory (SRAM) is critical for LUT-based kernels to maximize occupancy and hide memory latency. We analyze the shared memory consumption for a single thread block based on our default configuration (Mtile=128,Ktile=64M_tile=128,K_tile=64). The total shared memory usage StotalS_total consists of two components: the storage for Look-Up Tables (SLUTS_LUT) and the intermediate accumulation buffer (SaccS_acc). LUT Storage (SLUTS_LUT). Since the input vector X varies across the batch, and the column scaling factor c α_c is fused into the LUT, independent LUTs must be constructed for each batch index within the block. For a sub-vector size μ=8μ=8 and KbK_b binary bases, the LUT storage is calculated as: SLUT=Btile⋅Kb⋅Ktileμ⋅2μ⋅sizeof(float)S_LUT=B_tile· K_b· K_tileμ· 2^μ·sizeof(float) (33) where BtileB_tile denotes the batch size handled per block. The term Ktileμ K_tileμ represents the number of sub-vectors per tile. Accumulation Buffer (SaccS_acc). To perform parallel reduction and row-scale post-correction, we maintain a buffer to store partial sums for each row and each binary base before the final reduction. Sacc=Btile⋅Kb⋅Mtile⋅sizeof(float)S_acc=B_tile· K_b· M_tile·sizeof(float) (34) Impact of Sub-vector Size (μ) on Resource Trade-offs. Table 9 presents a comparative analysis of shared memory footprint, throughput, and energy efficiency between sub-vector sizes μ=4μ=4 and μ=8μ=8. (i) Shared Memory Occupancy: Consistent with our theoretical derivation, μ=4μ=4 maintains an extremely low footprint (<<24 KB), imposing negligible pressure on SRAM. In contrast, μ=8μ=8 exhibits significantly higher consumption due to the exponential growth of LUT size (282^8 vs 242^4). Specifically, at Batch Size (BS) 8, μ=8μ=8 consumes 136 KB. Crucially, this remains within the 164 KB capacity limit per SM on the NVIDIA A100, ensuring that the larger LUT configuration does not cause shared memory overflow or prevent kernel execution. Table 9: Impact of sub-vectors size μ on block-level shared memory usage and end-to-end generation performance of FluxBin-2b-g128 LLaMA-2-7B. Higher throughput is better (↑ ), while lower power and energy are better (↓ ). μ BS Shared Mem. (KB) Throughput ↑ Power (W) ↓ Energy (J) ↓ 4 1 3 213.97 107.13 397.15 4 12 656.50 146.97 802.74 8 24 952.86 172.31 1301.12 8 1 17 213.96 105.41 451.18 4 68 655.58 140.04 792.82 8 136 953.04 169.48 1220.45 (i) Performance and Selection Strategy: The end-to-end throughput remains comparable between μ=4μ=4 and μ=8μ=8, suggesting it is not a shared memory bound regime. However, we select μ=8μ=8 as the default configuration driven by energy efficiency. According to our complexity analysis (Eq. 27), the number of sub-vectors within a fixed K-tile is inversely proportional to μ (sub-vectors number = Ktile/μK_tile/μ). Consequently, decreasing μ from 8 to 4 doubles the number of sub-vectors. This necessitates a 2×2× increase in the frequency of LUT read operations, which is shown to dominate the complexity in Sec. C.1, and the associated accumulation within the block. As empirically evidenced in Table 9, this intensified operational frequency directly translates to higher power consumption (e.g., an increase of ∼ 3W at BS=8), making μ=8μ=8 the superior choice for energy-efficient inference. E.3 Tiling Strategy and TFLOPS Comparison To maximize the utilization of GPU compute units and memory bandwidth, we conduct a sensitivity analysis on the matrix tiling parameters. Specifically, we investigate the tile size along the output channel dimension (MtileM_tile), the input channel dimension (KtileK_tile), and the salient refinement block size (Ms_tileM_s\_tile). Performance Metric. We evaluate the kernel throughput using Effective TFLOPS (Tera Floating-point Operations Per Second). Since FluxBin utilizes integer-based LUT logic rather than floating-point arithmetic, this metric represents the equivalent computational throughput required to perform the same matrix-vector multiplication in FP16. For hybrid configurations, we explicitly account for the additional workload introduced by the salient refinement branch. The Effective TFLOPS is calculated as: TFLOPS=2⋅M⋅(N+Nsal)Latency (s)×1012TFLOPS= 2· M·(N+N_sal)Latency (s)× 10^12 (35) where M and N denote the output and input channel dimensions, respectively, and NsalN_sal is the number of salient columns. For standard global base kernels (e.g., 2b-g128), we simply set Nsal=0N_sal=0. Optimization of Global Base Tiling. Table 10 presents the performance of pure binary kernels (2-bit and 4-bit) under various (Mtile,Ktile)(M_tile,K_tile) combinations. • Impact of MtileM_tile: Increasing MtileM_tile yields the most significant performance gain. For instance, with the 2-bit kernel at K=4096K=4096, expanding MtileM_tile from 32 to 128 increases throughput from 0.30 TFLOPS to 0.81 TFLOPS. A larger MtileM_tile increases the number of threads per block, effectively hiding instruction latency and amortizing the overhead of LUT construction. • Impact of KtileK_tile: The choice of KtileK_tile involves a trade-off between reduction efficiency and shared memory occupancy. While larger KtileK_tile reduces the number of global atomic reductions, it consumes more shared memory for LUT storage. Our empirical results suggest that Ktile=32K_tile=32 or 6464 typically offers the optimal balance. Optimization of Mixed-Precision Tiling. Table 11 extends the analysis to hybrid bases configurations (2b-s8/s16-g128). We observe that the mixed-precision kernel exhibits tiling sensitivities similar to the global base kernel, with Mtile=128M_tile=128 consistently delivering the best performance. Regarding the salient-specific tile size Ms_tileM_s\_tile, setting it to 64 provides a slight latency reduction compared to 16, as it promotes better memory coalescing for the dense salient matrix. Notably, the performance gap between s8 and s16 configurations is marginal across all tiling setups. This confirms the efficiency of our Virtual Columnar Mapping (VCM) strategy, which ensures that scaling up the salient branch incurs minimal computational penalties. Based on these findings, we adopt (Mtile=128,Ktile=64,Ms_tile=64)(M_tile=128,K_tile=64,M_s\_tile=64) as default for all end-to-end evaluations. Table 10: Effect of tile sizes on FluxBin kernel performance. FluxBin-2b-g128 FluxBin-4b-g128 M K tileM_tile tileK_tile Latency (µs) ↓ TFLOPS ↑ Latency (µs) ↓ TFLOPS ↑ 4096 4096 32 32 113.3113.3 0.300.30 109.6109.6 0.310.31 32 64 143.0143.0 0.230.23 108.4108.4 0.310.31 32 128 228.9228.9 0.150.15 108.4108.4 0.310.31 64 32 58.058.0 0.580.58 72.972.9 0.460.46 64 64 67.667.6 0.500.50 66.966.9 0.500.50 64 128 84.684.6 0.400.40 66.866.8 0.500.50 128 32 41.241.2 0.81 57.857.8 0.580.58 128 64 41.441.4 0.81 56.356.3 0.60 128 128 47.547.5 0.710.71 56.456.4 0.60 4096 11008 32 32 289.2289.2 0.310.31 296.4296.4 0.300.30 32 64 379.9379.9 0.240.24 307.0307.0 0.290.29 32 128 594.5594.5 0.150.15 308.7308.7 0.290.29 64 32 148.5148.5 0.610.61 215.1215.1 0.420.42 64 64 166.3166.3 0.540.54 200.9200.9 0.450.45 64 128 208.8208.8 0.430.43 201.0201.0 0.450.45 128 32 102.8102.8 0.88 158.9158.9 0.57 128 64 103.5103.5 0.870.87 181.3181.3 0.500.50 128 128 114.8114.8 0.790.79 182.1182.1 0.500.50 Table 11: Effect of tile sizes on FluxBin kernel performance on hybrid bases. FluxBin-2b-s8-g128 FluxBin-2b-s16-g128 M K tileK_tile tileM_tile _M_s\_tile Lat (µs) ↓ TFLOPS ↑ Lat (µs) ↓ TFLOPS ↑ 4096 4096 32 32 16 121.0121.0 0.290.29 124.4124.4 0.280.28 32 119.5119.5 0.290.29 123.2123.2 0.280.28 64 118.3118.3 0.290.29 121.9121.9 0.280.28 64 16 67.367.3 0.510.51 69.069.0 0.500.50 32 65.765.7 0.530.53 67.967.9 0.510.51 64 64.764.7 0.530.53 66.566.5 0.520.52 128 16 51.551.5 0.670.67 53.553.5 0.650.65 32 50.050.0 0.690.69 52.152.1 0.660.66 64 48.648.6 0.71 50.950.9 0.68 64 32 16 157.7157.7 0.220.22 158.7158.7 0.220.22 32 154.4154.4 0.220.22 156.5156.5 0.220.22 64 155.1155.1 0.220.22 156.6156.6 0.220.22 64 16 76.376.3 0.450.45 78.278.2 0.440.44 32 74.674.6 0.460.46 77.077.0 0.450.45 64 73.773.7 0.470.47 75.675.6 0.460.46 128 16 49.849.8 0.690.69 51.751.7 0.670.67 32 48.348.3 0.720.72 50.550.5 0.690.69 64 47.147.1 0.74 49.249.2 0.70 4096 11008 32 32 16 297.8297.8 0.310.31 304.2304.2 0.300.30 32 296.5296.5 0.310.31 303.1303.1 0.300.30 64 295.3295.3 0.310.31 301.3301.3 0.300.30 64 16 159.4159.4 0.570.57 161.1161.1 0.570.57 32 157.9157.9 0.580.58 159.8159.8 0.570.57 64 156.8156.8 0.580.58 157.9157.9 0.580.58 128 16 122.4122.4 0.750.75 124.1124.1 0.730.73 32 120.9120.9 0.750.75 122.6122.6 0.740.74 64 119.9119.9 0.76 121.4121.4 0.75 64 32 16 388.1388.1 0.240.24 402.0402.0 0.230.23 32 386.5386.5 0.240.24 405.1405.1 0.230.23 64 385.2385.2 0.240.24 401.5401.5 0.230.23 64 16 172.6172.6 0.530.53 174.8174.8 0.520.52 32 171.1171.1 0.530.53 173.4173.4 0.530.53 64 170.1170.1 0.540.54 172.2172.2 0.530.53 128 16 112.1112.1 0.810.81 114.1114.1 0.800.80 32 110.7110.7 0.820.82 112.6112.6 0.810.81 64 109.7109.7 0.83 111.4111.4 0.82 E.4 Kernel-Level Latency Comparison To isolate the kernel contribution from the end-to-end pipeline, we further compare the per-operator GEMV latency of FluxBin against the leading 2-bit weight-only baselines, AQLM and QuIP#, across the weight shapes that dominate LLM linear layers. Each kernel is timed in isolation using its official implementation. As reported in Table 12, FluxBin’s LUT-based kernel is consistently the fastest at the pure 2-bit setting, e.g. 41 μ at 4096×40964096× 4096 versus 112 μ for QuIP# and 79 μ for AQLM. This gap reflects the reconstruction cost of the competing schemes: QuIP# decodes an E8 lattice and AQLM performs a multi-codebook gather per weight, whereas FluxBin reconstructs each weight with a single foldable table lookup plus addition. Crucially, the hybrid-precision configurations (2b-s8/s16-g128) add only a marginal overhead over the pure 2-bit kernel, confirming that the Virtual Columnar Mapping keeps the salient branch nearly free; and even the 4-bit kernel remains faster than the 2-bit baselines at most shapes. Table 12: Per-operator GEMV kernel latency (μ ) across weight matrix sizes (M×NM× N), measured for the 2-bit quantized baselines and FluxBin. Lower is better (↓ ); the best result in each row is in bold. Each kernel is measured in isolation with its official implementation. Weight Size (M×NM× N) Baselines FluxBin AQLM 2bit QuIP# 2bit 2b-g128 2b-s8-g128 2b-s16-g128 4b-g128 4096×4096 79 112 41 47 49 56 5120×5120 111 174 63 73 74 87 4096×11008 169 272 102 109 111 158 5120×13824 277 411 167 166 169 252

Instant research discovery