Paper deep dive
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
Chenwei Jia, Baoting Li, Xuchong Zhang, Mingzhuo Wei, Bochen Lin, Hongbin Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/20/2026, 6:54:30 AM
Summary
The paper proposes Quant Experts (QE), a token-aware adaptive error reconstruction framework for Post-Training Quantization (PTQ) of Large Vision-Language Models (VLMs). QE utilizes a Mixture-of-Experts (MoE) approach to divide important channels into token-independent and token-dependent groups. A shared expert with a low-rank adapter compensates for global quantization errors from token-independent channels, while routed experts dynamically compensate for local errors from token-dependent channels. Experiments show QE enhances accuracy across various model scales (2B to 70B) and quantization settings.
Entities (11)
Relation Signals (9)
Quant Experts → targets → Vision-Language Models
confidence 95% · Quant Experts (QE), a token-aware adaptive error reconstruction with the Mixture-of-Experts (MoE) for VLMs quantization
Quant Experts → uses → Mixture-of-Experts
confidence 95% · we propose Quant Experts (QE), a token-aware adaptive error reconstruction with the Mixture-of-Experts (MoE) for VLMs quantization
Quant Experts → employs → Shared Expert
confidence 90% · For the former, a shared expert is designed for most tokens to compensate for global quantization error
Quant Experts → employs → Routed Experts
confidence 90% · For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error
Shared Expert → handles → Token-independent Channels
confidence 88% · The shared expert focuses on global quantization error that is mainly caused by token-independent channels.
Routed Experts → handles → Token-dependent Channels
confidence 88% · token-dependent channels are clustered into multiple subgroups, and each is handled by a routed expert that specializes in local error compensation.
Quant Experts → improves → Post-training Quantization
confidence 85% · Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations without retraining the full model. Existing PTQ methods primarily rely on static identification and global compensation of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose \textbf{Quant Experts (QE)}, a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization. QE divides the important channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to compensate for local quantization error related to specific tokens. Extensive experiments demonstrate that QE consistently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B parameters, while maintaining performance comparable to full-precision models.
Tags
Links
- Source: https://arxiv.org/abs/2602.24059v1
- Canonical: https://arxiv.org/abs/2602.24059v1
Trouble viewing inline? Open PDF directly →
Full Text
66,434 characters extracted from source content.
Expand or collapse full text
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization Chenwei Jia * Baoting Li * Xuchong Zhang † Mingzhuo WeiBochen LinHongbin Sun State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University jiacw@stu.xjtu.edu.cn Abstract Post-Training Quantization (PTQ) has emerged as an ef- fective technique for alleviating the substantial computa- tional and memory overheads of Vision-Language Models (VLMs) by compressing both weights and activations with- out retraining the full model. Existing PTQ methods pri- marily rely on static identification and global compensa- tion of sensitive or outlier channels, yet they often overlook the distributional differences of these important channels across inputs, leading to unsatisfactory quantization. In this work, we observe that the distributions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Accordingly, we propose Quant Experts (QE), a token-aware adaptive error compensation with mixture- of-experts for VLMs quantization. QE divides the impor- tant channels into token-independent and token-dependent groups. For the former, a shared expert is designed for most tokens to compensate for global quantization error using a low-rank adapter. For the latter, routed experts including multiple routed low-rank adapters are elaborated to com- pensate for local quantization error related to specific to- kens. Extensive experiments demonstrate that QE consis- tently enhances task accuracy across various quantization settings and model scales, ranging from 2B to 70B param- eters, while maintaining performance comparable to full- precision models. 1. Introduction Model quantization has become a key technique for reduc- ing the computational and memory costs of large-scale mul- timodal models [19]. By mapping weights and activations to low-bit representations, it achieves substantial compres- * Equal contribution. † Corresponding author: zhangxc0329@xjtu.edu.cn. Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independentChannels Shared and routed expertswith low-rank adaptor Top-k importanceTop-k importance Important channel frequencyImportant channel frequency Token-independent Channels Token-dependent Channels r/2r/2r/2 r/2r/2r/2 Shared Expert Routed Experts r/2 r/2 Shared Expert Routed Experts Vision tokenVision tokenLanguage tokenLanguage tokenHadamardproductHadamard productImportant channelImportant channelLow-rank adapterLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Channel - wise Token-wiseToken-wise Tokens fromTokens from Channel - wise Token-wise Tokens from Channel - wise Token-wise Tokens from Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importance Important channel frequency Token-independent Channels Token-dependent Channels r/2 r/2 Shared Expert Routed Experts Vision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Token-wise Tokens from Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Quantized Weight Router Input Output ......... Quantized Weight Router Input Output ... Shared ExpertShared ExpertRouted ExpertsRouted Experts ADDADD Shared ExpertRouted Experts ADD Shared ExpertRouted Experts ADD ...... Quantized Weight Router Input Output ... Shared ExpertRouted Experts ADD ... Figure 1. Illustration of Different Quantization Granularities.D is the calibration data, andW q is the quantized weight matrix. SmoothQuant [44] employs channel-wise with fixed scaling co- efficients to achieve global quantization.MBQ [23] presents a modality-aware strategy that focuses on sensitivities in input modalities. In contrast, we propose a token-aware adaptive quan- tization that considers both global and local error reconstruction. sion and acceleration while preserving accuracy. Among existing approaches, Post-Training Quantization (PTQ) is widely adopted for its efficiency and compatibility without retraining the full model. However, low-bit quantization inevitably introduces numerical perturbations that degrade performance, a problem that is especially pronounced in multimodal scenarios [13]. Recent studies on large-scale model quantization reveal significant variation in channel value distributions and sen- sitivities, where a small set of outlier or highly sensitive channels plays a key role in preserving model expressive- ness and accuracy. To mitigate quantization errors aris- ing from these important channels, numerous methods have been proposed from different perspectives, including chan- nel smoothing, mixed-precision strategy, Hessian-based op- 1 arXiv:2602.24059v1 [cs.CV] 27 Feb 2026 timization, and low-rank reconstruction approaches. For in- stance, SmoothQuant [44] and AWQ [24] alleviate quan- tization errors by balancing activation and weight ranges via channel-wise scaling, with fixed coefficients for each channel estimated from the calibration dataset. SpQR [9] statically identifies highly sensitive channels and allocates higher precision to them, effectively reducing quantization errors. OBQ [11] and GPTQ [12] estimate a static Hes- sian matrix from inputs and perform channel- or block-wise quantization based on the computed sensitivity. LQER [48] and ASER [50] leverage singular value decomposition (SVD) to explicitly reconstruct quantization errors through a global low-rank adapter [16], i.e. all important channels are handled uniformly by a single adapter. Currently, in multimodal scenarios, MBQ [23] reveals substantial dif- ferences in channel sensitivities across modalities and in- troduces a modality-aware channel scaling strategy to im- prove the adaptability of channel-smoothing methods. Al- though these methods reduce quantization errors effectively, the fixed channel-importance estimation and global error compensation neglect the modeling of local feature vari- ations, failing to adequately account for the dynamic re- lationships among important channel, error compensation, input modalities, and even tokens. In this paper, we conduct a systematic analysis of large Vision-Language Models (VLMs) and observe that chan- nel importance exhibits strong dynamics, varying signif- icantly across modalities and tokens. In particular, only a small subset of important channels appears consistently among most tokens, while the majority exhibit strong input-dependent fluctuations, with their importance differ- ing across modalities and tokens. To further minimize quan- tization errors, the dynamic nature of channel importance necessitates quantization methods that not only compensate for errors in globally activated important channels but also adaptively mitigate those arising in input-dependent ones. Accordingly, we propose Quant Experts (QE), a token- aware adaptive error reconstruction framework with the Mixture-of-Experts (MoE) [15, 36] for VLMs quantization, which considers both global and local variations in chan- nel importance across different tokens. QE first estimates the occurrence frequency of important channels from cali- bration data and then divides them into token-independent and token-dependent channels. We then employ two types of experts, the shared expert and the routed experts, to model these two kinds of channel groups respectively. The shared expert employs a low-rank adapter to recon- struct global quantization errors that predominantly orig- inate from token-independent important channels. Mean- while, token-dependent channels are clustered into multiple sub-groups according to their co-occurrence relationships, and each sub-group is assigned a routed expert equipped with a routed low-rank adapter.During inference, QE fixedly employs the shared expert to compensate for global quantization errors and dynamically selects the most suit- able routed expert according to the input tokens, thereby achieving adaptive compensation for local errors. Extensive experiments under various quantization settings demon- strate that QE effectively mitigates errors caused by dy- namic channel importance and substantially restores model performance under low-bit quantization. This work makes the following contributions: • We observe that the distributions and occurrence fre- quencies of important channels vary significantly both across modalities and among tokens, even within the same modality. • We propose QE, a token-aware adaptive error compen- sation with MoE for VLMs quantization. It employs a fixed shared expert to compensate global quantization er- rors while dynamically selecting routed experts to correct token-dependent local errors. • Extensive experiments across various quantization set- tings and multimodal benchmarks demonstrate that QE effectively mitigates low-bit performance degradation, achieving up to a 5.09% accuracy improvement under the challenging W4A6 quantization for the 72B model. 2. Observation and Motivation 2.1. Observation One: The Positions of Important Channels Vary Across Tokens As illustrated in Fig. 2, we visualize the value distributions of several tokens along with the positions of their corre- sponding important channels computed by Eqs. (1) and (2). w := Mean row (|W f |),(1) C t = Top -k(|x t |⊙ w),(2) where W f ∈R d out ×d in is the weight matrix, d in is the num- ber of input channels, d out is the number of output channels, Mean row computes the mean of absolute values along row, w is a d in -dimensional vector, x t is the t-th token, and C t denotes its top-k important channels. It is evident that, under the same layer weight, the po- sitions of important channels vary not only across different modalities but also among tokens within the same modality. In the case of across modalities, the intrinsic differences in value distributions lead to cross-modal shifts in the posi- tions of important channels. More importantly, even within the same modality, variations in token semantics and con- textual information cause significant changes in activation distributions, resulting in dynamic migrations of important channels. These findings indicate that the positions of im- portant channels are not static but dynamically adapt to both inter- and intra-modality variations. This observation reveals a key limitation of those glob- ally calibrated and compensated methods [9, 24, 50], just 2 Figure 2. Visualization of a Transformer block in Qwen2VL-2B, illustrating token value distributions (top) and the positions of im- portant channels (bottom). In the top panel, brightness reflects token magnitude, while in the bottom panel, highlighted positions denote important channels. identifying some fixed outliers and employing unified com- pensation, which fails to capture token-wise variations in channel importance. 2.2. Observation Two: Uneven Frequency Distri- bution of Important Channels To gain deeper insight, we compute the occurrence fre- quency of each important channel using Eq. (3) and sort them in descending order of frequency. f c = k× m c P d in i=0 m i ,(3) where m c denotes the number of times channel c is identi- fied as important, and f c represents the proportion of tokens for which channel c is recognized as an important channel among all tokens. As shown in Fig. 3, there is a small subset of important channels that occur across most tokens, while the major- ity are activated only for specific tokens. Meanwhile, those important channels with smaller frequency values may still exhibit larger outlier magnitudes, making their contribution to quantization error compensation equally non-negligible. Based on this observation, we define the channels, which consistently appear across diverse input tokens and play a key role in compensating global quantization errors, as token-independent important channels. Conversely, chan- nels whose activation depends on specific tokens are re- garded as token-dependent important channels, which re- veal distinctive token properties of different tokens and the local nature of quantization errors. This observation suggests that important channels in VLMs comprise both global and local components. Quan- Figure 3. Visualization of a Transformer block in Qwen2VL-2B, illustrating important channel behavior. The top panel shows ac- tivation frequencies, while the bottom depicts average activation values. Red crosses mark important channel positions identified by the static global method on the calibration dataset. tization error compensation that fails to properly address these two types inevitably results in performance degrada- tion: insufficient compensation for token-independent im- portant channels weakens the model’s overall representa- tional capacity, while neglecting token-dependent important ones compromises token-level semantic fidelity. Therefore, an effective quantization framework should adopt differen- tiated protection and compensation strategies across tokens to ensure both stability and adaptability in performance. 3. Method As illustrated in Fig. 4, QE first estimates the distribu- tion frequency of important channels from the calibration data D and partitions them into token-independent and token-dependent channels. It then employs the shared ex- pert (SE) and the routed experts (REs), to handle these distinct channel groups, each implemented as a low-rank adapter. The shared expert focuses on global quantization error that is mainly caused by token-independent channels. Whereas token-dependent channels are clustered through co-occurrence analysis into multiple subgroups, and each is handled by a routed expert that specializes in local er- ror compensation. During inference, a lightweight router dynamically selects the optimal routed expert for adaptive error compensation. 3.1. Calibration and Error Reconstruction In this subsection, we illustrate how to minimize quanti- zation error from token-aware adaptive optimization and establish the mapping between important channels and the mixture-of-experts. Let W l f denote the full-precision weight of the l-th layer, and W l q = Q(W l f ) its quantized 3 NrNr Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importanceTop-k importance Important channel frequencyImportant channel frequency Token-independent Channels Token-dependent Channels r/2r/2r/2 r/2r/2r/2 Shared Expert Routed Experts r/2 r/2 Shared Expert Routed Experts Vision tokenVision tokenLanguage tokenLanguage tokenHadamard productHadamard productImportant channelImportant channelLow-rank adapterLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Channel - wise Token-wiseToken-wise Tokens fromTokens from Channel - wise Token-wise Tokens from Channel - wise Token-wise Tokens from Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importance Important channel frequency Token-independent Channels Token-dependent Channels r/2 r/2 Shared Expert Routed Experts Vision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Token-wise Tokens from Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Quantized Weight Router Input Output ......... Quantized Weight Router Input Output ... Shared ExpertShared ExpertRouted ExpertsRouted Experts ADDADD Shared ExpertRouted Experts ADD Shared ExpertRouted Experts ADD ...... Quantized Weight Router Input Output ... Shared ExpertRouted Experts ADD ... Vision tokenVision tokenLanguage tokenLanguage tokenHadamard productHadamard productImportant channelImportant channelLow-rank adapterLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importanceTop-k importance Important channel frequencyImportant channel frequency Token-independent Channels Token-dependent Channels Shared Expert Routed Experts r/2r/2r/2r/2r/2 r/2r/2r/2r/2r/2 w Channel - wise Channel - wise Token-wiseToken-wise Tokens fromTokens from w Channel - wise Token-wise Tokens from Figure 4. The framework of Quant Experts (QE). The token-independent channels are model by a shared expert, while token-dependent channels are captured by multiple routed experts. A lightweight low-rank adapter is implemented for each expert. counterpart. As typical, the quantization error is defined as E l = W l f − W l q . Building on the analysis in Sec. 2, we further formulate quantization objective optimization as: arg min ̃ E l ∥(E l − ( ̃ E l S + ̃ E l R (x l )) | z ̃ E l )x l ∥ F ,(4) where x l denotes the input tokens of the l-th layer, ̃ E l S represents the token-independent error approximation, and E l R (x l ) denotes the error approximation dependent on x l . Our approach leverages two complementary low-rank adapters to reconstruct the quantization error: ̃ E l = L l SA L l SB | z ̃ E l S + L l,i ∗ RA L l,i ∗ RB |z E l R (x l ) ,(5) i ∗ = arg min i R l |x l | i ,(6) where L l SA and L l SB form the shared expert that mod- els token-independent error approximation, while L l,i RA and L l,i RB denote the routed experts adaptively modeling token- dependent error approximation. R l is a lightweight router that predicts a score for each routed expert, and selects an expert with the minimal score. In this way, QE integrates global token-independent and local token-dependent com- pensation within a unified mixture-of-experts framework. We quantify the distribution of channel importance using activation X ∈R T×d in collected from the calibration data D. For thet-th input tokenx l t in layerl, we compute the per- token important channelsA l t as Eq. (2). Then we aggregate all A l t over D to form a multiset T l , and compute the oc- currence frequency f l of each important channel according to Eq. (3). These channels are sorted by f and partitioned into two disjoint important channel sets: the first k channels C l s , referred to as token-independent channels, and the sub- sequent (N r k) channelsC l r , referred to as token-dependent channels. To sum up, the pseudo-code of the dependence partitioning process is detailed in Algorithm 1. Algorithm 1: Channel Dependence Partitioning Input : Layer weightW l f L l=1 ; calibration data D; the number of important channels k; the number of routed experts N r . Output: Token-independent channelsC l s ; token-dependent channelsC l r . 1 for sample d∈D do 2Inference d and store input activations in X 3 for l← 1 to L do 4InitializeT l ←∅ 5Compute w l from W l f according to Eq. (1) 6for token x l t in X l do 7 A l t ← Top -k(|x l t |· w l ) // indices 8 T l ←T l ⊎A l t 9(C l unsorted ,f l )← unique(I l ) 10 π ← argsort(f l , descending) 11ReorderC l ←C l unsorted [π] 12Token-independent channelsC l s ←C l [1:k] 13Token-dependent channelsC l r ←C l [k+1:(N r +1)k] 14 returnC l s L l=1 , C l r L l=1 3.2. SE for Token-Independent Channels We introduce the shared expert to reconstruct the quanti- zation error that primarily arises from token-independent channels. As in prior work [12, 24, 40], the performance degradation and quantization error stem mainly from these outlier channels, underscoring the need to mitigate their im- pact on both weight and activation quantization. Building upon the above motivation, the shared expert is designed following [50] to focus on the token-independent channels C l s . Specifically, to alleviate weight quantization errors and suppress the interference of their outlier mag- nitudes on other channels, token-independent channels are 4 Algorithm 2: Building Routed Experts Input : Per-layer dataX l , O l , E l S , C l r L l=1 ; the number of routed experts N r ; rank r. Output: REs(L l,i RA , L l,i RB ) and RouterR l . 1 for l← 1 to L do 2Initialize router parameter R l ∈R d in ×N r 3Compute x l ← Mean row (|X l |) 4S l ← NPMISimilarity(O l ) 5 N l ← NormedLaplacian(S l ) 6U l ← Eigenvectors(N l ) 7Cluster Γ l ← KMeans(U l [:,2:N r +1] , N r ) 8for cluster labels γ ∈ Γ l do 9Initialize weight ω ← 1∈R d in 10Compute ω ← (x l γ /min(x l γ ))⊙ x l 11Normalize ω ← ω/ p min(ω) max(ω) 12Perform SVD: U ΣV ⊤ = E l S diag(ω) 13L l,i RA ← U r Σ r 14L l,i RB ← V ⊤ r S −1 diag(1/ω) 15R l [:,i] ← Mean row (|E l S − L l,i RA L l,i RB |) 16 return(L l,i RA , L l,i RB ) L,N r l=1,i=1 ,R l L l=1 exempted from direct quantization and reconstructed by a low-rank adapter (L l SA , L l SB ) L l using whitening SVD. To mitigate token-wise activation quantization errors, the shared expert employs channel-wise scaling, which reduces activation magnitudes while proportionally amplifying the corresponding weights. Through the initial reconstruction for E l performed by the shared expert, QE achieves precise recovery of token-independent channels, while the remain- ing error E l S = E l − L l SA L l SB is subsequently refined by the routed experts associated with token-dependent chan- nels. The algorithm for the shared expert is detailed in the supplementary Sec. 7.1. 3.3. REs for Token-Dependent Channels In this subsection, we focus on the quantization error com- pensation for token-dependent important channels. Ideally, one would tailor an individual compensation strategy to each token, but the virtually unbounded value combinations make such token-specific designs computationally infeasi- ble. Therefore, it becomes essential to design a constrained yet effective compensation strategy that approximates the optimal solution. We observe that token-dependent chan- nels exhibit correlated occurrence patterns across differ- ent tokens. Based on this insight, we empirically com- pute their co-occurrence statistics and cluster channels with strong mutual association. We then employ routed experts, each employed with a low-rank adapter dedicated to mod- Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importanceTop-k importance Important channel frequencyImportant channel frequency Token-independent Channels Token-dependent Channels r/2r/2r/2 r/2r/2r/2 Shared Expert Routed Experts r/2 r/2 Shared Expert Routed Experts Vision tokenVision tokenLanguage tokenLanguage tokenHadamard productHadamard productImportant channelImportant channelLow-rank adapterLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapterVision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Channel - wise Token-wiseToken-wise Tokens fromTokens from Channel - wise Token-wise Tokens from Channel - wise Token-wise Tokens from Channel importance estimation from calibration data Top-k important channel identification per token Partitioning into token-dependent and token-independent Channels Shared and routed experts with low-rank adaptor Top-k importance Important channel frequency Token-independent Channels Token-dependent Channels r/2 r/2 Shared Expert Routed Experts Vision tokenLanguage tokenHadamard productImportant channelLow-rank adapter Channel - wise Token-wise Tokens from Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-awareModality-aware Token-awareToken-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Modality-aware Token-aware Modality-aware Token-aware Modality-aware Token-aware e.g. SmoothQuant (Channel-wise static quantization) e.g. MBQ (Modality-aware quantization) Different modalities Quant Experts (Token-aware adaptive quantization) Different tokens Fixed scaling Quantized Weight Router Input Output ......... Quantized Weight Router Input Output ... Shared ExpertShared ExpertRouted ExpertsRouted Experts ADDADD Shared ExpertRouted Experts ADD Shared ExpertRouted Experts ADD ...... Quantized Weight Router Input Output ... Shared ExpertRouted Experts ADD ... Figure 5. Illustration of the Inference Computation Process of QE. eling its corresponding channel cluster. During inference, a lightweight router estimates the final error of each expert and activates the one predicted to yield the lowest error. Specifically, we construct the co-occurrence matrix fol- lowing Eq. (7) to capture the token-level correlation patterns among token-dependent channels from all tokens X l . O l t,i = 1 c i ∈C l r ∩A l t , O l ∈0, 1 T×(N r k) ,(7) whereO l denotes the co-occurrence matrix overT tokens in l-th layer, and the indicator function 1(·) returns 1 if the i-th channel c i is important for the t-th token, and 0 otherwise. Next, we employ spectral clustering [34, 37] to par- tition the token-dependent important channels based on their most likely co-occurrence patterns. Firstly, the co- occurrence matrixO l is transformed into a similarity matrix S l ∈R (N r k)×(N r k) by using the normalized pointwise mu- tual information (NPMI) [2, 21], which quantifies the asso- ciation strength between co-occurring channels as follows: p(i) = 1 T T X t O l t,i ,(8) p(i,j) = 1 T T X t (O l t,i O l t,j ),(9) S i,j = (log p(i,j) p(i)p(j) ) − logp(i,j),(10) where i,j represent the channel indices, and S i,j represents the NPMI value between channels i and j. A higher NPMI value indicates a stronger likelihood that these channels will occur as important channels within a single token. For the similarity matrix S l , we perform eigen decom- position on its normalized Laplacian N l and take the next N r eigenvectors U l = [u 2 ,...,u N r +1 ] as spectral embed- dings. K-Means then partitionsC l r into N r clusters of token- dependent channels using spectral embeddings U l . For each cluster, we apply a weighting vector ω on E l S to en- hance the reconstruction accuracy for its token-dependent channels γ. Next, we perform SVD to reconstruct each weighted E l S and truncate the rank to r. The N r low- rank adapters together constitute the routed experts. Fi- nally, we use the absolute mean of the remaining error 5 E l,i R = E l S − L l,i RA L l,i RB for i-th routed expert as the param- eters of router R l i , which estimates the error of any input token under i-th routed expert. The above procedure is de- tailed in Algorithm 2. The inference process is illustrated in Fig. 5, where the shared expert provides stable global compensation and the router adaptively activates the opti- mal routed expert for dynamic local compensation. To further alleviate the performance degradation intro- duced by post-training quantization, we design an optional lightweight refinement strategy. Specifically, only routed experts (L l RA , L l RB ) and the router R l are trainable, while all other parameters remain frozen. Moreover, this refine- ment is performed layer-wise without end-to-end training of all parameters. The refinement is detailed and formulated in the supplementary material Sec. 7.2. 4. Experiments 4.1. Experimental Setup Models.We conduct comprehensive PTQ experiments on several representative open-source VLMs, covering the Qwen2VL [43] series (2B, 7B, 72B) and the InternVL2 [8] series (2B, 8B). All model weights are obtained from the official repositories. Baselines. We perform systematic comparisons with pop- ular open-source PTQ methods.For weight-activation quantization, QE is evaluated under W4A6 and W4A8 settings against round-to-nearest (RTN), channel-scaling SmoothQuant (SQ) [44], modality-balanced MBQ [23], and low-rank reconstruction LQER [48]. Activations and weights are quantized using per-token and per-output- channel symmetric schemes, respectively.For weight- only quantization, we adopt AWQ [24] as the channel- scaling baseline under the W3A16 configuration, apply- ing group-wise asymmetric quantization with 128 group size. Throughout, “WxAy” denotes weight and activation bitwidths of x and y, while #W and #A represent their bitwidths respectively. Evaluation Metrics. To comprehensively assess the per- formance of our method, we experiment across diverse multimodal tasks. Text recognition and understanding are tested on OCRBench [25] and TextVQA [38]; document and infographic comprehension on DocVQA [32] and In- foVQA [33]; chart reasoning on ChartQA [31]; and general visual perception on VizWiz-VQA [14]. ScienceQA [29] and MMMU [46] evaluate scientific and general reasoning, while MMStar [6] and MuirBench [42] assess overall multi- modal and multi-image understanding. AI2D [18] measures diagram comprehension, ensuring coverage across all major aspects of multimodal reasoning. We evaluate on the open source evaluation framework LMMs-Eval [49]. Experimental Details.We follow MBQ [23] and use the enhanced COCO Caption dataset [7] from ShareGPT4V [5], randomly sampling 128 image-caption pairs as the calibration set. In both LQER and QE, the total SVD rank r is set to 64. Since QE employs both shared and routed experts, the total rank of 64 is split into 64 2 for each type, ensuring that the overall rank matches LQER. The k is fixed to 32, and the N r is set to 8. For refinement, we set epochs for 16 with 100 iterations per epoch using the AdamW [28] optimizer (learning rate 1× 10 −4 , no weight decay) and a cosine annealing schedule [27]. The refine- ment coefficients are τ = 0.5, α = 1.0, and β = 0.05. Smaller models are evaluated on 4× RTX 4090 24G GPUs, and the 72B model is on 4× A800 80G GPUs. 4.2. Main Results Smaller VLMs are generally more sensitive to quantization, making them more challenging benchmarks [22, 23]. We report results on Qwen2VL-2B and InternVL2-8B in Tab. 1 and Tab. 2, and on the larger 72B model in Tab. 3. Weight-Activation Quantization. QE consistently sur- passes both the modality-aware baseline MBQ [23] and the static low-rank method LQER [48]. In the challeng- ing W4A6 setting, it improves Qwen2VL-2B accuracy by 4.01% over MBQ, with only a 4.23% drop from full pre- cision, and gains 3.13% on InternVL2-2B. At W4A8, the performance gap to full precision narrows to within 2%, demonstrating strong robustness. On the larger Qwen2VL- 72B, QE achieves a remarkable average accuracy improve- ment of 5.09% under the W4A6 quantization setting, nearly matching full-precision performance. Weight-Only Quantization. QE again outperforms MBQ and LQER across all models. MBQ’s distribution reshap- ing offers limited benefit over AWQ due to capacity limits, while LQER’s static compensation yields only minor recov- ery. These results highlight the necessity of dynamic com- pensation to handle modality- and token-level distribution shifts that static methods cannot capture. Additional results on Qwen2VL-7B and InternVL2-2B (Supplementary Tab. 9 and Tab. 10) show consistent gains, further confirming the generalizability of our method across diverse vision-language models. 4.3. Ablation Studies Effect of Each Component.Our method integrates a shared expert (SE) and routed experts (REs). Main re- sults show that their combination yields notable gains across quantization settings. To further verify each expert type, we conduct ablation studies in Tab. 4. Results show that removing either expert consistently degrades performance. The random routing experiment demonstrates that the pro- posed routing method can adaptively select the suitable routed expert to recover model accuracy. Similarly, the ran- dom clustering experiment confirms that the proposed co- occurrence-based clustering substantially enhances quanti- 6 Method#W#AMMMUOCRBenchScienceQATextVQAVizWizAI2DChartQADocVQAInfoVQAMMStarMuriBenchAvg. (↑) Qwen2VL-2B161639.8974.9076.9677.7265.7370.0172.0487.2858.4943.4626.1962.97 RTN4634.0059.8064.7067.5855.6259.2656.0875.7845.1240.6731.1953.62 SQ (ICML’23)4630.4459.6065.2565.8853.9059.1640.4470.7339.3638.2030.0050.27 LQER (ICML’24)4633.0065.8068.3269.3755.9162.5662.6881.0248.9237.8229.7755.92 MBQ (CVPR’25) 4634.4461.1067.0869.4557.1960.9160.0876.2443.1342.6129.7754.73 QE4633.7868.2071.8473.1859.6265.4564.6082.7551.8442.0432.8858.74 RTN4835.0065.2070.8572.4656.9665.4565.2480.1049.0340.7534.2357.75 SQ (ICML’23)4832.1165.8068.0268.6157.4361.9544.0875.0742.0638.9329.5853.06 LQER (ICML’24)4835.6769.8071.9973.0856.8167.5566.6883.6852.7339.8831.5059.03 MBQ (CVPR’25)4834.3362.3070.8572.3658.3265.1962.3278.1947.9342.5732.6257.00 QE4837.3372.1074.6775.3461.5968.3669.2884.4653.9741.2934.1261.14 RTN31632.4465.8067.6370.4355.8161.5962.8078.3646.3235.2438.5455.91 AWQ (MLSys’24) 31633.2263.4068.3270.6056.8861.7262.0878.7045.0236.4835.6255.64 LQER (ICML’24)31634.5667.5069.8670.0458.6862.7665.3681.1947.5436.3538.5857.49 MBQ (CVPR’25)31633.4463.9069.1670.7552.5963.2863.5679.2745.8734.6634.5055.54 QE31633.8970.1072.0974.4660.1564.9668.5282.9751.3139.6634.1259.29 Table 1. Main results on the model of Qwen2VL-2B. Method#W#AMMMUOCRBenchScienceQATextVQAVizWizAI2DChartQADocVQAInfoVQAMMStarMuriBenchAvg. (↑) InternVL2-8B161648.0076.9097.1276.9160.6182.0982.6089.9766.9259.3636.1270.60 RTN4637.0069.2093.1669.9655.3473.7774.1683.8155.8250.0330.1262.94 SQ (ICML’23) 4640.4469.5094.8469.8350.9175.1374.9284.2856.7749.5932.0063.47 LQER (ICML’24)4640.2272.2094.7471.8358.0376.4977.8885.8060.4749.6930.8565.29 MBQ (CVPR’25)4643.6771.0095.4970.2652.9077.5975.6484.2158.0953.8932.2765.00 QE4644.8974.3096.2374.7759.0079.4080.4887.7763.1255.2734.1568.13 RTN4843.3372.8095.9373.1156.9679.4079.3686.5561.4755.2335.3867.23 SQ (ICML’23)4842.2272.2095.5472.5851.6877.0477.4885.4859.0751.9832.5865.26 LQER (ICML’24)4844.4475.1096.6875.0657.4980.9980.4488.1163.8554.7733.1568.19 MBQ (CVPR’25) 4844.4473.5096.7872.0056.6779.2177.7286.4261.4855.3232.7766.94 QE4845.5675.6096.7376.0859.7781.0681.6088.3164.8356.0934.3169.09 RTN31644.2274.2096.1874.6455.9980.4779.3287.9662.6155.3732.8867.62 AWQ (MLSys’24)31645.6774.6096.3374.9759.1880.4780.0888.0163.5554.5534.8568.39 LQER (ICML’24)31645.3374.9096.3874.7757.1280.6080.0488.1063.6155.6532.7368.11 MBQ (CVPR’25)31646.1175.2096.1874.9758.4379.7079.7288.0063.1654.9635.0068.31 QE31645.7875.9096.2875.5059.4680.9980.7688.7964.3557.1433.3868.94 Table 2. Main results on the model of InternVL2-8B. SettingMethodMMMUOCRBenchScienceQATextVQAVizWiz FP16-61.4478.7091.2282.2676.27 W4A6 RTN51.3359.1085.5272.1264.29 SQ53.3361.2086.9674.4263.45 LQER52.3359.6086.7173.8466.64 MBQ52.6769.7086.3276.0867.99 QE58.1176.6090.3379.2773.91 W4A8 RTN57.1166.5090.0875.9671.14 SQ56.4465.1090.0376.3766.70 LQER57.4470.6091.3278.1574.28 MBQ58.3373.9089.2479.1972.24 QE58.8978.3091.4781.4775.83 Table 3. Main results of Qwen2VL-72B model (higher is better). zation performance. Furthermore, we visualize the clus- tering results in Fig. 6 according to Sec. 3.3. The clus- tered similarity matrix and t-SNE [30] projection indicate that our method effectively identifies and partitions these co-occurring clusters. Effect of Refinement. Tab. 5 presents the ablation results of the proposed refinement for routed experts under the SettingComponentMMMU (↑)ScienceQA (↑) FP16-39.8976.95 W4A6 routed experts (REs)34.5668.72 shared expert (SE)35.2269.61 SE + random routing35.8970.00 SE + random clustering35.3369.71 QE (SE+REs)36.8970.85 W4A8 routed experts (REs)36.0071.94 shared expert (SE)36.7873.13 SE + random routing37.8973.67 SE + random clustering37.2273.82 QE (SE+REs)38.0074.37 Table 4. Ablation study results on Qwen2VL-2B model. Qwen2VL W4A6 quantization. As observed on both 2B and 7B models, applying refinement consistently leads to notable accuracy improvements across multiple tasks com- pared to the non-refinement counterparts. Effect of the Number of Routed Experts. We analyze in Tab. 6 the impact of the number of routed experts on model accuracy. Here, N r denotes the number of routed experts. 7 Figure 6. Illustration of the Co-Occurrence-Based Clustering in a Transformer Block of Qwen2VL-2B. (a) Similarity matrixS l showing mutual co-occurrence among token-dependent channels, with brightness indicating similarity. (b) Channels with strong co-occurrence are grouped into the same cluster. (c) t-SNE [30] projection demonstrates that the clustering effectively captures their co-occurrence relations. ModelRef.MMMUOCRBenchScienceQATextVQAVizWiz 2B ✗33.7868.2071.8473.1859.62 ✓36.8969.6070.8573.3060.58 7B ✗45.4473.0079.8771.6364.18 ✓44.0074.6080.6177.5865.11 Table 5. Ablation study of Refinement on Routed Experts. “✓” indicates applied refinement; “✗” indicates none. It can be observed that as N r increases, the model perfor- mance gradually improves; however, a larger N r also im- plies higher memory overhead for routed experts. N r OCRBenchTextVQAVizWizAvg. (↑) 268.4073.1459.7067.08 4 68.5073.1360.4167.35 869.6073.3060.5867.83 1669.9073.5260.7568.06 Table 6. Impact of the number of routed experts on the perfor- mance of Qwen2VL-2B under the W4A6 quantization setting. 4.4. Overheads Analysis and Kernel Performance The low-rank adapter introduces additional computation and memory overhead from the lightweight auxiliary matri- ces L ·A and L ·B . Let s denote the sequence length, d hid- den size, r(≪ d) the rank, and N r (≪ d) the routed experts count. The complexity analysis of a single layer is provided in Tab. 7, with computation quantified in terms of floating- point operations (FLOPs) and memory overhead evaluated by the total number of parameters. Notably, QE incurs only minimal additional computational and memory costs com- pared to the original linear, yet enables the quantized model to recover accuracy comparable to full precision. To assess hardware efficiency, we develop an analyti- cal performance model following the FlightLLM [47] ac- celerator architecture. Prefill-stage kernel speedups of lin- ear layers are measured with sequence length 128 using ComplexityOriginQE Computation sd 2 sd 2 + sd(2r + N r ) Memoryd 2 d 2 + rd(1 + N r ) Table 7. Complexity analysis of the linear layer in QE method. Qwen2VL-7B weight shapes under various quantization settings. As shown in Tab. 8, QE achieves 3.5×-4.5× accel- eration, highlighting its strong potential for hardware-level efficiency gains. Shape (IC× OC)W4A6W4A8W3A16 3584× 35843.56×3.50×4.10× 3584× 189443.60×3.59×4.50× 18944× 35843.84×3.77×4.50× Table 8. NPU speedup ratios of QE for Qwen2VL-7B linear layers compared with the fp16 model, measured during the prefill stage with a sequence length ofs=128. “IC” and “OC” denote the input and output channel dimensions, respectively. 5. Conclusion In this work, we reveal a key observation that the distri- butions and occurrence frequencies of important channels vary significantly both across modalities and among tokens, even within the same modality. Building on this insight, we propose Quant Experts (QE), a token-aware adaptive er- ror compensation for VLMs quantization, that dynamically adapts to such variations. Specifically, QE employs a shared expert to robustly reconstruct token-independent chan- nels and routed experts to adaptively compensate token- dependent ones, with each expert implemented as a low- rank adapter. Comprehensive evaluations on diverse VLMs show that QE consistently outperforms globally static PTQ baselines across different quantization configura- tions. 8 References [1] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alis- tarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier- free 4-bit inference in rotated llms.In NeurIPS, pages 100213–100240, 2024. 1 [2] Gerlof Bouma. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of GSCL, pages 31–40, 2009. 5 [3] Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos. QBB: Quantization with binary bases for llms. In NeurIPS, pages 3209–3228, 2024. 1 [4] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christo- pher De Sa. QuIP: 2-bit quantization of large language mod- els with guarantees. In NeurIPS, 2023. 1 [5] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, pages 370–387, 2024. 6 [6] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024. 6 [7] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ́ ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 6 [8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 6 [9] Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexan- der Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In International Conference on Learn- ing Representations, 2024. 2, 1 [10] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. 1 [11] Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal Brain Compression: a framework for accurate post-training quantization and pruning. In NeurIPS, 2022. 2, 1 [12] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre- trained transformers. In International Conference on Learn- ing Representations, 2023. 2, 4, 1 [13] Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. What makes quantization for large language model hard? an empirical study from the lens of perturbation. In AAAI, pages 18082–18089, 2024. 1 [14] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018. 6 [15] Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Kor- thikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. arXiv preprint, 2024. 2 [16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In In- ternational Conference on Learning Representations, 2022. 2 [17] Wenjing Ke, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. DL-QAT: Weight-decomposed low-rank quantization-aware training for large language models. In Conference on Empir- ical Methods in Natural Language Processing, pages 113– 119, 2024. 1 [18] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 6 [19] Gun Il Kim, Sunga Hwang, and Beakcheol Jang. Efficient compressing and tuning methods for large language models: A systematic literature review. ACM Computing Surveys, 57 (10):1–39, 2025. 1 [20] Junhan Kim, Chungman Lee, Eulrang Cho, Kyungphil Park, Ho young Kim, Joonyoung Kim, and Yongkweon Jeon. To- wards next-level post-training quantization of hyper-scale transformers. In NeurIPS, 2024. 1 [21] Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In Conference on Computational Natural Language Learning, pages 171–180, 2014. 5 [22] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xi- angsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. arXiv preprint arXiv:2402.18158, 2024. 6 [23] Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. MBQ: Modality-balanced quantization for large vision- language models. In CVPR, pages 4167–4177, 2025. 1, 2, 6 [24] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device llm compression and accelera- tion. Proceedings of machine learning and systems, 6:87– 100, 2024. 2, 4, 6, 1 [25] Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: on the hidden mystery of ocr in large multimodal models. In Science China Informa- tion Sciences, page 220102, 2024. 6 [26] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics, pages 467–484, 2024. 1 9 [27] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradi- ent descent with warm restarts. In International Conference on Learning Representations, 2017. 6 [28] Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 6 [29] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, pages 2507–2521, 2022. 6 [30] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 (Nov):2579–2605, 2008. 7, 8 [31] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Association for Computational Linguistics, 2022. 6 [32] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 6 [33] Minesh Mathew, Viraj Bagal, Rub ` en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 6 [34] Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NeurIPS, 2001. 5 [35] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations, 2024. 1 [36] Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. 2 [37] Jianbo Shi and J. Malik. Normalized cuts and image seg- mentation. IEEE TPAMI, 22(8):888–905, 2000. 5 [38] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 6 [39] Siqing Song, Chuang Wang, Rui-Qi Wang, Yi Yang, and Xu-Yao Zhang.Achieving binary weight and activation for LLMs using post-training quantization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8782–8795. Association for Computational Linguis- tics, 2025. 1 [40] Hanlin Tang, Yifu Sun, Decheng Wu, Kai Liu, Jianchen Zhu, and Zhanhui Kang. EasyQuant: An efficient data-free quan- tization algorithm for LLMs. In Empirical Methods in Natu- ral Language Processing, pages 9119–9128, 2023. 4 [41] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better llm quantization with hadamard incoherence and lattice code- books. In International Conference on Machine Learning, 2024. 1 [42] Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang- Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and et al. Muirbench: A comprehensive benchmark for robust multi- image understanding. In ICLR, 2025. 6 [43] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint, 2024. 6 [44] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and effi- cient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099, 2023. 1, 2, 6 [45] Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yux- iong He. Exploring post-training quantization in llms from comprehensive study to low rank compensation. In AAAI, 2024. 1 [46] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In CVPR, pages 9556–9567, 2024. 6 [47] Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, et al. Flightllm: Efficient large language model inference with a complete mapping flow on fpgas. In ACM/SIGDA International Symposium on Field Pro- grammable Gate Arrays, pages 223–234, 2024. 8 [48] Cheng Zhang, Jianyi Cheng, George A. Constantinides, and Yiren Zhao.LQER: low-rank quantization error recon- struction for llms. In International Conference on Machine Learning, 2024. 2, 6, 1 [49] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. LMMs- eval: Reality check on the evaluation of large multimodal models. In Association for Computational Linguistics, pages 881–916, 2025. 6 [50] Weibo Zhao, Yubin Shi, Xinyu Lyu, Wanchen Sui, Shen Li, and Yong Li. ASER: activation smoothing and error recon- struction for large language model quantization. In AAAI, 2025. 2, 4, 1 [51] Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. In Machine Learning and Systems, 2024. 1 10 Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization Supplementary Material In the supplementary material, we provide additional Related Work, Method Details, and Experimental Results. In Sec. 7, we present more complete implementation de- tails for the Shared Expert and the Refinement of Routed Experts. In Sec. 8, we report the main results of QE on Qwen2VL-7B and InternVL2-2B, along with evaluations on language tasks. We further present results for joint quan- tization of the Visual Encoder and the VLM, along with an extended ablation study on the Number of Important Chan- nels. 6. Additional Details for Related Work In large language model (LLM) compression, two main- stream approaches are commonly used: quantization-aware training (QAT) and post-training quantization (PTQ). QAT explicitly models quantization errors during training and can achieve higher accuracy for low-bit models, but it incurs substantial computational and data overhead (e.g., LSQ [10], LLM-QAT [26], DL-QAT [17]). In contrast, PTQ directly maps pretrained weights and activations into low-bit representations after training, requiring only a small amount of calibration data. Due to its efficiency and practi- cality, PTQ has become the dominant solution for resource- constrained scenarios (e.g., QBB [3], aespa [20]). However, PTQ inevitably introduces quantization errors, and existing methods remain constrained by limited outlier identification and error compensation, posing a key challenge for advanc- ing low-bit LLM deployment [13]. To address this core challenge, various solutions have been proposed from different perspectives. OBQ [11] and GPTQ [12] perform progressive quantization with Hessian- guided iterative compensation, allowing unquantized pa- rameters to absorb the quantization errors yielded in pre- vious channels or blocks, thereby alleviating reconstruc- tion error within Transformer blocks. [39] further com- bines Hessian-based optimization with the Expectation- Maximization (EM) algorithm to enable joint weight- activation quantization at extremely low bitwidths. Distri- bution reshaping approaches mitigate the effects of outliers by applying channel-wise scaling and equalization to bal- ance the dynamic ranges of activations and weights. Among them, SmoothQuant [44] transfers part of the quantiza- tion difficulty from activations to weights through channel- wise scaling, effectively balancing their dynamic ranges. Furthermore, AWQ [24] employs a search-based channel scaling strategy, while selectively retaining the most sen- sitive parameters in full precision to preserve model accu- racy. OmniQuant [35] incorporates learnable clipping and equivalent scaling transformations, jointly optimized un- der a block-level error minimization framework to achieve stronger error suppression. From another perspective, some approaches utilize rotation-based transformations to miti- gate outliers in weight and activation quantization, effec- tively reducing quantization errors. QuIP [4] employs an uncorrelated transformation combined with adaptive round- ing to minimize proxy errors, while QuIP# [41] integrates random Hadamard transforms and block-wise vector quan- tization to improve reconstruction accuracy. QuaRot [1] further proposes an end-to-end 4-bit quantization scheme based on Hadamard rotation, which enables simultaneous quantization of weights, activations, and KV cache. In model quantization, performance degradation and quantiza- tion errors primarily arise from outlier and sensitivity-prone important channels. Precisely identifying and preserving these channels at higher precision is essential for mitigat- ing quantization errors. For instance, Atom [51] enhances robustness under low-bit settings through hybrid precision and dynamic activation quantization, whereas SpQR [9] leverages Hessian-based sensitivity analysis to identify im- portant parameters, retaining high precision for outlier weights while quantizing the remaining ones into low- bit representations, thereby effectively mitigating outlier- induced errors. Another research direction introduces low- rank structures into quantization error compensation by at- taching lightweight high-precision low-rank modules to re- cover accuracy with minimal computational and memory overhead. Representative approaches include LoRC [45], which models quantization residuals using low-rank matri- ces to restore performance at low cost; LQER [48], which leverages activation statistics and diagonal rescaling for weighted low-rank reconstruction; and ASER [50], which adopts whitened SVD for more stable error modeling and integrates outlier-channel analysis to smooth activation dis- tributions. 7. Additional Details for Method 7.1. Shared Expert The construction process of the shared expert in QE is illus- trated in Algorithm 3, which follows the general procedure described in [50]. This method employs a low-rank struc- ture to approximate the quantization error introduced by weight quantization, with a particular focus on frequently activated, token-independent important channels, thereby 1 effectively capturing globally stable quantization error pat- terns. Algorithm 3: Building the Shared Expert Input : Per-layer dataX l , W l f , C l s L l=1 , quantizer Q(·); rank r. Output: Quantized layer weightW l q L l=1 , SE (L l SA , L l SB ) L l=1 , and residual errors E l S L l=1 . 1 Compute x l ← Mean row (|X l |) 2 for l← 1 to L do 3Initialize ω = [1, 1,..., 1] n , Ω = diag(ω) 4Compute ω C l s = x l C l s / min(x l C l s ) 5E l q = W l f − Q(W l diag(1− 1 C l s )) 6Compute whitening matrix S by Cholesky decomposition of (Ω −1 X)(Ω −1 X) ⊤ such that (S −1 Ω −1 X)(S −1 Ω −1 X) ⊤ = I 7Perform SVD: U ΣV ⊤ = E l q S 8Compute: L l SA = U r Σ r , L l SB = V ⊤ r S −1 , E l S = E l q − L l SA L l SB 9 returnW l q L l=1 ,(L l SA , L l SB ) L l=1 ,E l S L l=1 7.2. Refinement of Routed Experts In this subsection, we describe the loss functions used in the Refinement stage. These losses follow standard formu- lations commonly adopted in prior research. We provide de- tailed explanations here due to space limitations in the main paper. The refinement objective consists of two comple- mentary losses: a regression loss L reg and a classification loss L cls . L reg aims to minimize the reconstruction error between the quantized output ˆy and full-precision output y, encouraging each expert to specialize in its own direction of compensation. L cls improves the router’s ability to predict the optimal expert for a given input. Specifically, let y i denote the output reconstructed by the i-th routed expert and y the full-precision output. We define the reconstruction distance as d i = ∥ˆy i − y∥ 1 . During re- finement, only the routed expert achieving the smallest re- construction error is optimized, formulated as: L reg = min i∈[1,N r ] d i .(11) To enable the router to predict the relative performance of different routed experts, we denote its output as l = R|x| and construct a classification objective based on the inter- expert discrepancy. We adopt the Kullback-Leibler diver- gence to align the predicted distribution with the normalized reconstruction loss distribution: L cls = τ 2 D KL (P∥ Q),(12) P = softmax −(d− μ(d))/σ(d) τ ,(13) Q = softmax −(l− μ(l)) τ ,(14) where τ is a temperature coefficient, and μ(·) and σ(·) de- note the mean and standard deviation. Finally, we use two coefficients, α and β, to balance the two losses: L = αL reg + βL cls .(15) 8. Additional Experiments 8.1. Additional Model The experimental results of QE on additional models are presented in Tab. 9 and Tab. 10. Consistent with previous findings, our method significantly outperforms the baselines under W4A6, W4A8, and W3A16 configurations. 8.2. Performance on Language Tasks The core idea of QE is to employ multi-expert low-rank adapters that dynamically adapt to compensation differ- ences across modalities and even among individual to- kens, thereby improving model performance on both vision- language and language-only tasks. To validate this, we eval- uate the quantized Qwen2VL-2B and Qwen2VL-7B mod- els on the MMLU benchmark under different quantization methods. As shown in Tab. 11, QE consistently achieves significant performance gains over LQER across various quantization configurations and model scales. These results demonstrate that explicitly modeling sensitivity differences across modalities and tokens not only effectively mitigates performance degradation in vision-language tasks but also helps maintain stable performance on language-only tasks. 8.3. Quantize Both Visual Encoder and VLM To achieve higher acceleration ratios, we further quantize both the visual encoder and the merger module that con- nects the encoder to the VLM. As shown in Tab. 12, L de- notes the VLM, V the visual encoder, and M the merger module, where✓ indicates that the corresponding mod- ule is quantized. The results show that as more modules are quantized, the model exhibits negligible performance degradation, demonstrating that the proposed joint quanti- zation strategy effectively improves overall efficiency while maintaining accuracy. 8.4. Effect of the Number of Important Channels We further investigate the effect of the number of impor- tant channels k on model accuracy in Tab. 13. The results 2 Method#W#AMMMUOCRBenchScienceQATextVQAVizWizAI2DChartQADocVQAInfoVQAMMStarMuriBenchAvg. (↑) Qwen2VL-7B161650.7879.5084.8381.4868.5680.6081.6891.6869.7757.7442.9271.78 RTN4639.0059.5075.4166.9356.6469.6272.0078.7352.9149.9738.2759.91 SQ (ICML’23)4641.0063.9077.0967.9457.4870.9268.8478.7852.9050.4835.6560.45 LQER (ICML’24)4642.5665.6077.2471.8764.4471.4474.0481.5756.8649.7541.8563.38 MBQ (CVPR’25) 4640.5662.7079.6770.9151.4871.3172.2081.9954.7147.3734.5460.68 QE4645.4473.0079.8771.6364.1875.5874.1683.6060.4552.6342.1965.70 RTN4845.4460.3079.4771.1859.1176.7874.5277.0456.8953.2340.0463.09 SQ (ICML’23)4843.7858.6079.5269.5253.0176.2072.8874.3453.9552.9336.6961.04 LQER (ICML’24)4848.0069.5081.7675.3166.0277.8577.0482.2761.5255.2443.8567.12 MBQ (CVPR’25)4846.3372.0081.4675.3560.7876.9876.3285.2661.6253.5237.6566.12 QE4846.3378.2081.5178.9866.5979.0578.9289.2166.1654.0442.4669.22 RTN31632.4465.8067.6370.4355.8176.7573.7274.4658.1450.3243.6960.84 AWQ (MLSys’24) 31648.0076.3082.1579.0065.6976.4978.9289.1064.9854.0140.6968.67 LQER (ICML’24)31646.4464.5080.8172.5066.2276.9175.3677.4259.7752.0143.2765.02 MBQ (CVPR’25)31646.2274.4082.3579.4365.0277.3078.2488.5964.7052.4542.9668.33 QE31646.6777.2081.5679.8767.2078.0179.2889.6065.2653.3544.5869.33 Table 9. Main results on the model of Qwen2VL-7B. Method#W#AMMMUOCRBenchScienceQATextVQAVizWizAI2DChartQADocVQAInfoVQAMMStarMuriBenchAvg. (↑) InternVL2-2B161634.3375.3094.3072.5845.9472.8374.8484.8453.2348.2028.4662.26 RTN4630.4467.1086.4766.1741.6663.4166.4877.4143.9140.1126.8555.46 SQ (ICML’23) 4631.8969.3088.2567.2440.1764.4165.4879.9446.9441.6225.6956.45 LQER (ICML’24)4630.7870.9088.3567.8139.3365.6468.9279.7845.7341.7528.2757.02 MBQ (CVPR’25) 4631.3370.9090.5368.5441.3967.5270.2080.9947.9545.2625.4658.19 QE4632.1172.8092.1270.4143.8168.6970.5282.0048.6945.1828.6959.55 RTN4832.0072.1091.0869.1942.7268.0769.0481.0648.9745.1228.2758.87 SQ (ICML’23)4833.7871.2091.2769.2040.1968.1368.0481.3148.7544.2326.6958.44 LQER (ICML’24)4834.5672.5092.0770.5339.1669.3370.9682.0349.8944.8428.6559.50 MBQ (CVPR’25)4832.7872.5092.2270.1944.3170.5371.4482.2449.8947.7327.3160.10 QE4832.3374.0092.8671.4443.2971.4772.8883.3051.1145.7729.0860.68 RTN31629.7869.7088.6567.5138.2166.0968.8880.5646.2941.4528.3856.86 AWQ (MLSys’24)31629.7869.5089.8968.1245.3067.7868.4480.8446.4444.6825.5057.84 LQER (ICML’24)31631.0070.3089.3467.8035.9367.0369.5280.4146.5741.7428.0457.06 MBQ (CVPR’25)31630.3369.2089.3967.8345.7467.6868.4080.5746.2144.2026.2757.80 QE31630.7872.1092.7669.6047.6870.0571.4482.1647.9045.0829.7759.94 Table 10. Main results on the model of InternVL2-2B. ModelSettingMethodMMLU (↑) Qwen2VL-2B FP16-52.79 W4A6 LQER44.37 QE47.21 W4A8 LQER46.60 QE50.35 Qwen2VL-7B FP16-67.88 W4A6 LQER55.59 QE61.87 W4A8 LQER64.21 QE64.83 Table 11. The results of quantized Qwen2VL on the MMLU benchmark. indicate a steady improvement as k increases. However, at k = 64, the performance saturates and slightly declines, as selecting an excessively large set of channels dilutes the focus on truly critical ones. LVMOCRBenchScienceQATextVQAVizWizAvg. (↑) ---74.9076.9577.7265.7373.83 ✓--68.2071.8473.1859.6268.21 ✓-65.9070.7572.0959.8367.14 ✓66.4070.1071.7759.2866.89 Table 12. Quantization results of different modules in Qwen2VL- 2B. The symbol “-‘’ indicates full precision (FP16), while✓ de- notes W4A6 quantization. L, V, and M correspond to the VLM, visual encoder, and merger module, respectively. kMMMUOCRBenchVizWizAvg. (↑) 434.8967.3058.9353.71 835.3368.9058.9554.39 16 35.1168.5059.9254.51 3236.1169.3060.2455.22 6434.4470.1059.5754.70 Table 13. Impact of the number of important channels on the per- formance of Qwen2VL-2B under the W4A6 quantization setting. 3