Paper deep dive
RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models
Zhiping Wu, Dongdong Ren, Yangchengyu Zhou, Zhengjie Zhang, Wenbin Li, Hongbing Pan, Yang Gao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/23/2026, 4:40:12 AM
Summary
The paper introduces RGSQ (Riemannian Geometry-Sensitive Quantization), a post-training quantization (PTQ) method for Large Vision-Language Models (VLMs). RGSQ addresses the limitations of Euclidean-based PTQ methods by formulating quantization as a reconstruction problem under a unified Fisher-Riemannian metric. It constructs modality-aware Kronecker-structured metrics from empirical Fisher factors, applies geometry-aligned rotations to steer quantization errors toward loss-insensitive directions, and uses whitening transformations to enable standard PTQ solvers. RGSQ achieves superior accuracy and stability in low-bit settings (W2A8, W3A8) compared to baselines like MBQ and MQuant.
Entities (8)
Relation Signals (7)
RGSQ → targets → VLM
confidence 98% · RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models
RGSQ → uses → Fisher-Riemannian metric
confidence 95% · RGSQ formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric.
RGSQ → applies → geometry-aligned rotations
confidence 92% · We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes.
PTQ → isusedfor → VLM
confidence 90% · Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ).
RGSQ → outperforms → MQuant
confidence 90% · It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9%
RGSQ → outperforms → MBQ
confidence 90% · It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9%
RSAVQ → islimitedto → unimodal scenarios
confidence 85% · Although RSAVQ accounts for directional errors, it is limited to unimodal scenarios and does not optimize cross-modal error directions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ). However, most PTQ methods are designed for unimodal large language models (LLMs). These methods treat quantization errors as isotropic perturbations under the Euclidean assumption, which provides weak guidance on directions most sensitive to quantization in VLMs. Consequently, directly adapting unimodal PTQ approaches or solely employing modality-specific scaling often leads to uneven bit-width distribution and inconsistent performance in low-bit settings. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher-Riemannian metric. RGSQ identifies modality-specific sensitive directions via Riemannian manifold mappings built from modality-partitioned empirical Fisher factors and fused into a modality-aware Kronecker-structured metric. We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes. Finally, we apply a whitening transformation that maps the Riemannian objective to an equivalent Euclidean form, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions. Across an extensive and diverse set of mainstream VLM benchmarks, RGSQ achieves the highest accuracy and stability under extremely low-bit settings (W2A8 and W3A8). It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9% and surpasses single-modality improvements by up to 8.6%.
Tags
Links
- Source: https://arxiv.org/abs/2609.25492v1
- Canonical: https://arxiv.org/abs/2609.25492v1
Trouble viewing inline? Open PDF directly →
Full Text
81,530 characters extracted from source content.
Expand or collapse full text
RGSQ: Riemannian Geometry-Sensitive Quantization for Large Vision-Language Models Zhiping Wu Dongdong Ren Yangchengyu Zhou Zhengjie Zhang Wenbin Li Hongbing Pan Yang Gao †thanks: Zhiping Wu and Hongbing Pan are with the School of Electronic Science and Engineering, Nanjing University, Nanjing 210093, China (e-mail: zhipingwu@smail.nju.edu.cn; phb@nju.edu.cn). Dongdong Ren is with Geely Automobile Research Institute (Ningbo) Co., Ltd., Ningbo 315000, China (e-mail: Dongdong.Ren2@geely.com) Yangchengyu Zhou, Zhengjie Zhang, Wenbin Li, and Yang Gao are with the State Key Laboratory for Novel Software Technology and the School of Intelligence Science and Technology, Nanjing University, Suzhou Campus, Suzhou 215163, China (e-mail: liwenbin@nju.edu.cn; gaoy@nju.edu.cn). Abstract Large vision-language models (VLMs) can be efficiently deployed under stringent memory and latency constraints through post training quantization (PTQ). However, most PTQ methods are designed for unimodal large language models (LLMs). These methods treat quantization errors as isotropic perturbations under the Euclidean assumption, which provides weak guidance on directions most sensitive to quantization in VLMs. Consequently, directly adapting unimodal PTQ approaches or solely employing modality-specific scaling often leads to uneven bit-width distribution and inconsistent performance in low-bit settings. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ), which formulates quantization as a reconstruction problem under a unified Fisher–Riemannian metric. RGSQ identifies modality-specific sensitive directions via Riemannian manifold mappings built from modality-partitioned empirical Fisher factors and fused into a modality-aware Kronecker-structured metric. We then apply geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward loss-insensitive axes. Finally, we apply a whitening transformation that maps the Riemannian objective to an equivalent Euclidean form, enabling standard unimodal PTQ methods to evaluate multimodal quantization error under their original assumptions. Across an extensive and diverse set of mainstream VLM benchmarks, RGSQ achieves the highest accuracy and stability under extremely low-bit settings (W2A8 and W3A8). It outperforms VLM-aware baselines, such as MBQ and MQuant, by up to 5.9% and surpasses single-modality improvements by up to 8.6%. Index Terms: Large Vision-Language Model, Riemannian Manifold, Quantization, Riemannian Geometry-Sensitive. Fig. 1: Illustration of the RGSQ framework: Rather than assuming multimodal quantization error is Euclidean and isotropic, we characterize each layer with a modality-aware Riemannian metric that reveals anisotropic, loss-sensitive directions. RGSQ then applies geometry-aligned rotations to reorient the local tangent frame, steering low-bit perturbations toward low-sensitivity axes. Finally, Riemannian whitening maps the metric back to an equivalent Euclidean form, enabling plug-and-play use of standard PTQ solvers for efficient error scoring and optimization. I Introduction Large language models (LLMs) have made remarkable progress across a broad range of domains, including natural language processing [1], code generation [2], complex reasoning [3], and multimodal understanding [4, 5]. Recent multi-modality VLMs continue to expand the perception, reasoning, and generation capabilities of foundation models across visual and textual inputs [6]. This progress, however, has come at the price of a rapid growth in parameter count and memory footprint. For example, the LLaMA-3 70B model requires approximately 140 GB of memory under FP16 precision, which makes its deployment on resource-constrained devices challenging [7]. Consequently, designing techniques that substantially reduce inference-time resource consumption without retraining, or with minimal additional overhead, has become a critical prerequisite for the practical deployment of large vision–language models (VLMs). Fig. 2: Illustration of the key motivation: (a) Euclidean PTQ quantifies distortion using an isotropic ℓ2 _2 metric, which fails to capture anisotropic sensitivity. (b) A Riemannian metric makes this anisotropy explicit through curvature-aware distortion, but incurs substantially higher computational cost. (c) The proposed RGSQ reconciles this trade-off in VLMs by building a modality-aware unified Riemannian geometry that aligns cross-modal distortion scales with practical overhead, and by steering low-bit perturbations toward loss-insensitive directions. Among efficiency-oriented techniques, including sparse attention [8], optimized decoding [9], and quantization [10], post-training quantization (PTQ) has been particularly attractive because it simultaneously reduces memory bandwidth and computational cost without requiring retraining. Weight-only approaches such as AWQ [11], GPTQ [12], and QuIP [13] primarily aim at relieving memory access and storage overhead, while weight–activation methods, including SmoothQuant [14], SpinQuant [15], FlatQuant [16], and Atom further leverage low-precision tensor cores available on modern GPUs to accelerate inference. To preserve downstream task performance, these methods typically rely on a calibration phase that optimizes scaling factors, channel equalization parameters, or rotation matrices by minimizing block-level reconstruction errors on a small calibration set. Despite their success, the vast majority of existing PTQ methods are primarily designed for unimodal LLMs, and their direct application to vision–language models often leads to substantial accuracy degradation. A few recent studies have begun to address this issue: Q-VLM [17] estimates cross-layer dependencies via entropy and guides weighted quantization of visual encodings. MBQ [18] introduces a modality-balanced strategy based on quantization sensitivity, and MQuant [19] combines modality-specific static quantization with attention-invariant flexible switching. Although these methods mitigate cross-modal mismatches by correcting errors in Euclidean space, they overlook the quantization-sensitivity information encoded by natural gradients in high-dimensional parameter spaces. A closely related effort, RSAVQ [20], introduces a Riemannian sensitivity-aware vector quantization formulation for LLMs, but it does not model the geometric relationships between modalities on the underlying manifold. Table I summarizes the conceptual differences between these representative methods and our work. Our motivation is illustrated in Fig. 2. We argue that the central challenge in quantizing VLMs originates from the simultaneous processing of visual and language features, which exhibit distinct statistical distributions and contribute unequally to the final task loss. When the calibration process intermingles these heterogeneous modalities under a unified, isotropic optimization objective, limited calibration capacity is often allocated to less critical components. This misallocation reduces representational fidelity for the most sensitive features and ultimately degrades overall model performance. Based on this observation, we identify two fundamental challenges. First, VLMs exhibit cross-modal inconsistencies in sensitivity and second-order structure, so a single Euclidean reconstruction objective is inadequate for prioritizing different modalities. Visual and language tokens differ not only in scale, but also in local curvature and loss sensitivity, assuming uniform token importance distorts the true cross-modal geometry and compromises the precision of highly sensitive components. Second, existing methods offer limited control over the directionality of low-bit quantization errors in multimodal settings. In low-bit regimes, quantization errors are characterized not only by their magnitude but also by their direction, which critically affects the task loss. Although RSAVQ accounts for directional errors, it is limited to unimodal scenarios and does not optimize cross-modal error directions, allowing quantization noise to accumulate and degrade performance in VLMs. To address these challenges, we propose Riemannian Geometry-Sensitive Quantization (RGSQ) for VLMs, as illustrated in Fig. 1. RGSQ adopts an information-geometric perspective and constructs a modality-aware unified Riemannian metric for every linear layer. Specifically, we estimate modality-specific second-order statistics, derive sensitivity-based aggregation weights from token-level gradient magnitudes, and fuse them into a unified Kronecker–Fisher metric that explicitly encodes cross-modal differences within a shared geometric space. Under this unified geometry, the conventional Euclidean reconstruction objective is reformulated as Riemannian perturbation minimization, enabling principled handling of cross-modal sensitivity disparities. Furthermore, to explicitly control quantization error directions, we introduce a geometry-guided error orientation mechanism that reparameterizes quantization errors via sparse Givens rotations. This mechanism steers the unavoidable quantization noise toward directions of low sensitivity and low curvature, so that even though quantization errors remain, their impact on task loss is reduced—enabling stable and efficient low-bit quantization of VLMs. VLMs are increasingly used as general-purpose tools across scientific document understanding, medical and biological image interpretation, and multimodal scientific question answering [21]. However, their practical use in these high-stakes scenarios is constrained by both deployment costs and reliability risks, including tight compute, energy, and privacy budgets as well as relationship hallucinations and distraction-sensitive reasoning [22, 23]. This shifts the central challenge from raw model capability alone to whether multimodal capability can be preserved under resource-limited and reliability-sensitive conditions. RGSQ is motivated by this requirement and aims to support stable low-bit deployment of VLMs. Our main contributions can be summarized as follows. • We propose a unified Riemannian PTQ framework for VLMs that constructs a single, modality-aware Kronecker–Fisher metric from modality-decomposed second-order statistics and sensitivity-based weights, thereby resolving cross-modal imbalance in quantization error and ensuring equitable allocation of calibration capacity across modalities. • We introduce a geometry-guided error orientation mechanism that uses sparse Givens rotations to reshape the basis in which quantization is performed. This enables joint control over both the magnitude and the direction of quantization noise, leading to substantially improved stability under low-bit settings. • We validate RGSQ on three representative VLM families across model scales from 7B to 72B and on six mainstream vision–language benchmarks. RGSQ delivers a superior accuracy–efficiency trade-off, with particularly notable gains under low-bit and weight–activation joint quantization, while remaining hardware-friendly through the use of sparse rotations. I Related Work I-A Post-Training Quantization for LLMs Post-training quantization (PTQ) has emerged as the dominant route to deploying large language models, because it substantially reduces inference-time memory footprint and bandwidth demand with little or no retraining overhead [24]. In general, PTQ relies on a small set of quantization factors, such as scaling and zero-point offsets, to map high-precision floating-point tensors into low-bit integer formats, thereby compressing weights and intermediate activations and better exploiting the low-precision tensor cores available on modern hardware [25]. Existing PTQ methods for LLMs can be broadly grouped into scalar and vector quantization. Scalar quantization learns per-channel or per-group scales and performs element-wise quantization for weights and activations, typically together with calibration-time compensation and outlier handling to reduce reconstruction error. Representative approaches include GPTQ [26] and AWQ [11], as well as distribution-shaping transformations such as rotations to improve quantization friendliness, e.g., QuaRot [27], OSTQuant [28], and architecture-specific adaptations such as MambaQuant [29]. Although these methods are simple and inference-friendly, they operate under Euclidean error surrogates during calibration, making it difficult to systematically capture direction-dependent sensitivity and structural heterogeneity that become pronounced at extremely low bit-widths. Vector quantization, in contrast, encodes weight blocks or vectors using shared codebooks and indices, enabling higher compression ratios with improved representational capacity. Typical examples include GPTVQ [30], VPTQ [31], AQLM [32], CRVQ [33], and QuIP# [34], which leverage codebook learning, residual quantization, or structured encoding to better fit quantization errors under aggressive compression. More recently, RSAVQ [20] emphasizes sensitivity-aware and geometry-motivated error characterization, offering a finer-grained perspective for low-bit quantization. However, LLM-centric PTQ techniques often degrade substantially when transferred directly to multimodal models. The underlying reason is that they implicitly assume relatively homogeneous distributions and representations, and therefore tend to apply shared scaling and unified calibration objectives across submodules. In VLMs, the visual and language branches differ markedly in their statistics, outlier patterns, and contributions to the final task loss. Treating multimodal weights and activations as a single modality—e.g., by sharing scaling factors or calibration targets—can amplify quantization errors on the more sensitive modality and lead to significant performance drops, which our experimental analysis (Section V-B) corroborates. I-B Post-Training Quantization for VLMs To mitigate this modality mismatch, recent work has developed PTQ techniques specifically tailored for VLMs, where the central strategy is modality-aware scaling and calibration. Representative methods include Q-VLM [17], QSLAW [35], MBQ [18], and MQuant [19]. These approaches either stabilize calibration through block- or partition-based quantization and cross-layer dependency modeling, or explicitly decouple visual and language components by assigning separate scaling factors and calibration weights. Many of them also exploit gradient- or loss-related signals as heuristic sensitivity indicators, reweighting reconstruction objectives to balance error across modalities [36]. Collectively, these studies demonstrate that explicitly accounting for modality discrepancies is necessary and can partially alleviate multimodal quantization collapse. Nevertheless, most existing multimodal PTQ methods still rely on Euclidean error modeling and gradient-heuristic guidance: they minimize reconstruction losses defined in an implicit Euclidean space, and use back-propagated gradients to estimate importance for adjusting quantization strategies. The limitation is that Euclidean gradients are local and metric-dependent, and may not reliably reflect loss-consistent sensitive directions in high-dimensional parameter spaces—especially when multimodal interactions and low-bit error accumulation coexist. In multimodal settings, quantization priority is determined not only by token-scale differences, but more fundamentally by modality-induced local curvature and second-order sensitivity structures. Ignoring this can systematically misallocate limited calibration capacity across modalities. As a result, purely Euclidean reconstruction objectives or gradient reweighting often fail to provide a consistent characterization of cross-modal second-order discrepancies and can be unstable under lower bit-widths and more challenging multimodal tasks. I-C Geometry-Aware Quantization Information geometry and geometric deep learning suggest that the parameter spaces of neural networks exhibit pronounced non-Euclidean structures [37]. A Riemannian metric induced by the Fisher Information Matrix (FIM) provides a principled measure of how parameter perturbations affect a model’s output distribution and task loss. In this view, natural gradients can be interpreted as scale-consistent updates along the underlying manifold, while approximations such as K-FAC [38] make second-order geometric modeling tractable for large networks. Bringing this perspective to quantization implies that quantization errors should not be assessed solely by Euclidean distances, but rather by geometry-consistent perturbation costs, and that one should seek perturbation directions that are minimally harmful to the task loss. Along this line, RSAVQ [20] shows that geometry- and sensitivity-aware formulations can better explain low-bit performance degradation and enable direction-dependent error analysis. However, existing geometry-aware quantization studies are largely developed for unimodal settings or isolated structures, and do not explicitly resolve two key questions in multimodal quantization. First, how can weights and activations from different modalities be mapped into a unified Riemannian geometric basis so that cross-modal second-order discrepancies and sensitivity inconsistencies can be modeled within a comparable metric space? Second, how can one exploit modality-specific sensitive directions in this geometric space to impose directional constraints on quantization noise, and then map the result back to a standard Euclidean computation graph to remain compatible with inference-friendly PTQ pipelines? In the low-bit regime, both the directionality and the accumulation path of quantization errors substantially influence the final loss, and Euclidean reconstruction or gradient-heuristic reweighting alone are insufficient to stably control the error trajectory. A more systematic approach for multimodal low-bit deployment should therefore re-formulate PTQ objectives under a unified Riemannian metric and explicitly control error directions, steering unavoidable quantization noise toward low-sensitivity and low-curvature directions to obtain a more reliable accuracy efficiency trade-off. This is precisely the gap addressed by RGSQ, which does not treat quantization error as an isotropic perturbation but instead measures it under a unified, modality-aware Fisher–Riemannian metric and steers it via sparse Givens rotations onto geometry-insensitive axes. As detailed in Section IV, this view subsumes both Euclidean PTQ and scalar modality-reweighting methods as special cases. I Preliminaries I-A PTQ for LLMs and VLMs VLM architecture and token notation. A typical VLM consists of a visual encoder, a vision–language projector, and a large language model (LLM) backbone. The visual encoder processes input images or video frames into compact patch features. The projector aligns these visual features with the text embedding space, producing visual tokens (v)X^(v). Text inputs are tokenized and embedded into text tokens (t)X^(t). These multimodal tokens are then concatenated and fed into a decoder-style LLM that generates output sequences autoregressively. At each Transformer layer ℓ , we denote the token set as ℓT_ , which is naturally partitioned by modality: ℓ=⋃m∈ℳℓ(m)T_ = _m T_ ^(m), where m∈ℳm indexes the modalities (e.g., vision v and text t). This partition is inherent to VLMs, since visual tokens arise from the projector while text tokens arise from the embedding table. We adopt symmetric uniform quantization for both weights and activations. For a scalar z∈ℝz , scale s>0s>0, and bit-width b, the quantization operator is Q(z,s,b)=s⋅clip(round(zs),−2b−1, 2b−1−1),Q(z;s,b)=s·clip\! (round\! ( zs ),-2^b-1,\,2^b-1-1 ), (1) which is applied per channel or per group for tensors. We denote quantized weights and activations by W^=Q(W),X^=Q(X) W=Q(W),\ X=Q(X), and the corresponding perturbations by ΔW=W−W W=W- W and ΔX=X−X X=X- X. Euclidean PTQ objective. For a linear projection Yℓ=WℓXℓY_ =W_ X_ , common Euclidean PTQ minimizes the output reconstruction error over a calibration set calD_cal: minW^ℓ(x,y)∈cal[‖WℓXℓ−W^ℓXℓ‖F2], _ W_ \ E_(x,y) _cal [\|W_ X_ - W_ X_ \|_F^2 ], (2) or one of its group-wise / channel-wise variants. This formulation implicitly assumes that Euclidean deviations are a faithful surrogate for the change in task loss and that all token modalities should be measured under a single isotropic scale. Both assumptions are systematically violated in VLMs. I-B Information-Geometric View of Quantization Error Riemannian metric from information geometry. A geometric view of quantization relates parameter perturbations to changes in the model’s output distribution [39, 40]. Let pθ(⋅)p_θ(·) denote the model distribution parameterized by θ, and consider a small perturbation Δθ θ. A second-order Taylor expansion of the log-likelihood ℓ(θ,u)=logpθ(u) (θ;u)= p_θ(u) gives ℓ(θ+Δθ,u)≈ℓ(θ,u)+∇θℓ(θ,u)⊤Δθ+12Δθ⊤∇θ2ℓ(θ,u)Δθ. (θ\!+\! θ;u)≈ (θ;u)+ _θ (θ;u)^\! θ+ 12 θ^\! \! _θ^2 (θ;u) θ. (3) Under standard regularity conditions, [∇θℓ(θ,u)]=0E[ _θ (θ;u)]\!=\!0, so the first-order term vanishes in expectation. Combining this with the Bartlett identity Fθ=−[∇θ2ℓ(θ,u)]F_θ\!=\!-E[ _θ^2 (θ;u)] yields the local quadratic approximation DKL(pθ∥pθ+Δθ)≈12Δθ⊤FθΔθ,D_KL (p_θ\,\|\,p_θ+ θ )≈ 12 θ^\! F_θ θ, (4) where FθF_θ is the Fisher Information Matrix (FIM) Fθ=[∇θlogpθ(u)∇θlogpθ(u)⊤].F_θ=E [ _θ p_θ(u)\ _θ p_θ(u)^\! ]. (5) Because FθF_θ is positive semi-definite, it serves as a canonical Riemannian metric tensor that measures distribution shifts in directions sensitive to local loss changes, inducing the inner product ⟨Δθ1,Δθ2⟩θ=Δθ1⊤FθΔθ2 _1, _2 _θ= _1^\! F_θ _2. Kronecker-factored curvature for linear layers. For a linear layer Y=WXY\!=\!WX with loss ℒL, denote the output gradient by G=∂ℒ/∂YG= /∂ Y. A widely used curvature proxy for W is the empirical Fisher FW=[vec(∇Wℒ)vec(∇Wℒ)⊤].F_W=E [vec( _WL)\,vec( _WL)^\! ]. (6) Using ∇Wℒ=GX⊤ _WL=GX^\! and the identity vec(GX⊤)=(X⊗I)vec(G)vec(GX^\! )=(X I)\,vec(G), a separable second-order structure follows once cross-covariances are ignored [38]: FW≈S⊗A,F_W≈ S A, (7) where A=x∼cal[xx⊤]A=E_x _cal[x^\! ] and S=g∼cal[gg⊤]S=E_g _cal[g^\! ]. The Kronecker approximation reduces the cost of storing and manipulating FWF_W from ((doutdin)2)O((d_outd_in)^2) to (dout2+din2)O(d_out^2\!+\!d_in^2) while retaining anisotropic sensitivity from both inputs and gradients. Why the geometric view matters for VLMs. In VLMs, visual and language tokens often exhibit distinct activation ranges, outlier patterns, and gradient statistics. A single Euclidean norm flattens this anisotropy and treats heterogeneous modalities as if they shared a common scale. By contrast, a Fisher–Riemannian metric exposes modality-dependent sensitive directions through the curvature factors A and S. This insight directly motivates the modality-aware geometric construction in Section IV-A. TABLE I: Comparison of quantization frameworks. (a) Euclidean PTQ minimizes isotropic quantization error, assuming uniform sensitivity (I) across all parameters. (b) Standard Riemannian methods utilize anisotropic curvature information (S⊗AS A) but typically apply uniform weighting to all tokens, ignoring modality differences. (c) RGSQ (Ours) constructs a modality-aware unified metric (Sl⊗AlS_l A_l) by aggregating modality-partitioned statistics with sensitivity-based weighting (π(m)π^(m)), and further incorporates geometry-guided error steering into the objective. Attribute (a) Euclidean (b) Riemannian (c) RGSQ (Ours) Objective ‖ΔW‖F2\| W\|_F^2 ‖ΔW‖S⊗A2\| W\|_S A^2 ‖ΔW‖Sℓ⊗Aℓ2+λℒgeom\| W\|_S_ A_ ^2+ _geom Sensitivity I S⊗AS A Sℓ⊗AℓS_ A_ Weighting (11) (1/N1/N) π(m)∝[1|(m)|∑t‖gt‖1]π^(m) [ 1|T^(m)| _t\|g_t\|_1 ] IV Method We propose RGSQ, a unified Riemannian post-training quantization framework tailored to large VLMs. The core idea is to construct a unified Riemannian metric tensor on the shared token interface, which enables us to (i) quantitatively evaluate the true impact of quantization noise on the task loss and output distribution shifts, and (i) directionally steer the noise toward low-curvature, low-sensitivity axes. IV-A Riemannian Modality-Aware Metric We first map the visual and text tokens from their original Euclidean space onto a Riemannian manifold, where perturbations that were previously treated as isotropic reveal their anisotropic, modality-dependent structure. We construct a unified metric for each linear layer Yℓ=WℓXℓY_ =W_ X_ by estimating modality-wise second-order factors, weighting them by loss sensitivity, and fusing them into a unified Kronecker–Fisher form. Modality-partitioned second-order factors. Let Xℓ∈ℝdin×NX_ ^d_in× N and Yℓ∈ℝdout×NY_ ^d_out× N denote stacked token activations and pre-activations on a calibration batch with N tokens, and let Gℓ=∂ℒ/∂YℓG_ = /∂ Y_ . For each modality m, we extract the submatrices Xℓ(m)X_ ^(m) and Gℓ(m)G_ ^(m) corresponding to tokens in ℓ(m)T_ ^(m), and compute the modality-partitioned empirical estimates: Aℓ(m) A_ ^(m) =1NmXℓ(m)(Xℓ(m))⊤, = 1N_m\,X_ ^(m)(X_ ^(m))^\! , (8) Sℓ(m) S_ ^(m) =1NmGℓ(m)(Gℓ(m))⊤, = 1N_m\,G_ ^(m)(G_ ^(m))^\! , where Nm=|ℓ(m)|N_m=|T_ ^(m)|. The pair (Aℓ(m),Sℓ(m))(A_ ^(m),S_ ^(m)) provides modality-conditioned second-order statistics of activations and gradients, and instantiates a modality-aware curvature proxy at layer ℓ . Sensitivity-aware modality weighting. A modality with larger loss gradients at a given layer is typically more sensitive to perturbations at that layer. Let gℓ,tg_ ,t denote the token-level gradient vector for token t at layer ℓ . We define the modality sensitivity score as αℓ(m)=[1|ℓ(m)|∑t∈ℓ(m)‖gℓ,t‖1], _ ^(m)=E [ 1|T_ ^(m)| _t _ ^(m)\|g_ ,t\|_1 ], (9) which uses the ℓ1 _1 norm to emphasize sparse but strong gradient coordinates—often more indicative of quantization-critical channels than ℓ2 _2 energy. We then normalize these scores into convex weights πℓ(m)=αℓ(m)∑m′∈ℳαℓ(m′)+ε,ε>0, _ ^(m)= _ ^(m) _m _ ^(m )+ , >0, (10) where the small ε ensures numerical stability for near-zero gradients early in calibration. By construction, πℓ(m)≥0 _ ^(m)≥ 0 and ∑mπℓ(m)≈1 _m _ ^(m)\!≈\!1. Crucially, πℓ(m) _ ^(m) depends on per-token sensitivity rather than the raw token count, so the resulting metric is invariant by construction to the absolute visual-to-text token ratio. In practice we further apply exponential moving-average smoothing across calibration batches and clamp αℓ(m) _ ^(m) to suppress outliers. When a modality has very few tokens but should not be neglected, a token-count–balanced variant πℓ(m)∝αℓ(m)⋅|ℓ(m)| _ ^(m)\! \! _ ^(m)·|T_ ^(m)| can be used. Unified Kronecker–Fisher metric. We aggregate the modality-specific factors into a single layer-wise metric: Aℓ A_ =∑m∈ℳπℓ(m)Aℓ(m), = _m _ ^(m)A_ ^(m), (11) Sℓ S_ =∑m∈ℳπℓ(m)Sℓ(m), = _m _ ^(m)S_ ^(m), and define the unified curvature proxy as FWℓ≈Sℓ⊗AℓF_W_ \!≈\!S_ \! \!A_ . Because AℓA_ and SℓS_ are convex combinations of positive semi-definite matrices, the unified metric Sℓ⊗AℓS_ \! \!A_ remains PSD, so the induced Riemannian norm is well defined. The visual and textual modalities are thereby mapped into a shared Riemannian space: for any quantization perturbation ΔWℓ W_ , the same metric tensor measures its Riemannian length regardless of modality, while modality-specific sensitivities are still encoded through πℓ(m) _ ^(m). Algorithm 1 constructs the unified Kronecker–Fisher metric (Eq. (11)) by online accumulation of modality-partitioned statistics, which is memory-efficient and requires only a single calibration pass with activation and gradient collection. Algorithm 1 Modality-Aware Metric Construction 1: Calibration set calD_cal, linear layers Wℓ=1L\W_ \_ =1^L 2: Modality-aware metrics Aℓ,Sℓ,πℓ(m)ℓ=1L\A_ ,S_ , _ ^(m)\_ =1^L 3: for ℓ=1 =1 to L do 4: Initialize Aℓ(m),Sℓ(m),αℓ(m)A_ ^(m),S_ ^(m), _ ^(m) for each modality m 5: for each calibration batch do 6: Run forward to obtain XℓX_ and GℓG_ 7: Split tokens into ℓ(m)m∈ℳ\T_ ^(m)\_m 8: Estimate modality factors: 9: Aℓ(m)←1NmXℓ(m)(Xℓ(m))⊤,Sℓ(m)←1NmGℓ(m)(Gℓ(m))⊤A_ ^(m)\!←\! 1N_mX_ ^(m)(X_ ^(m)) , 9.24994ptS_ ^(m)\!←\! 1N_mG_ ^(m)(G_ ^(m)) 10: Estimate sensitivity: 11: αℓ(m)←[1|ℓ(m)|∑t∈ℓ(m)‖gℓ,t‖1] _ ^(m)\!←\!E\! [ 1|T_ ^(m)| _t _ ^(m)\|g_ ,t\|_1 ] 12: end for 13: Normalize modality weights: 14: πℓ(m)←αℓ(m)/(∑m′αℓ(m′)+ε) _ ^(m)\!←\! _ ^(m)/ ( _m _ ^(m )+ ) 15: Fuse the unified metric: 16: Aℓ←∑mπℓ(m)Aℓ(m),Sℓ←∑mπℓ(m)Sℓ(m)A_ \!←\! _m _ ^(m)A_ ^(m), 9.24994ptS_ \!←\! _m _ ^(m)S_ ^(m) 17: end for 18: return Aℓ,Sℓ,πℓ(m)ℓ=1L\A_ ,S_ , _ ^(m)\_ =1^L IV-B Riemannian Objectives and Whitening We next turn the unified metric FWℓF_W_ into a calibration-time PTQ objective. This objective minimizes quantization perturbations under the induced Riemannian norm, yet we deliberately leverage the Kronecker structure to derive an equivalent Euclidean reduction via whitening. This reduction preserves the anisotropic, loss-sensitive weighting encoded in (Sℓ,Aℓ)(S_ ,A_ ) while enabling seamless reuse of standard Euclidean PTQ solvers, and without adding runtime operators. Riemannian objective for weight quantization. Let W^ℓ=Q(Wℓ) W_ =Q(W_ ) be the quantized weights and ΔWℓ=Wℓ−W^ℓ W_ =W_ - W_ the weight error. Under the unified curvature proxy, we minimize the Riemannian norm of ΔWℓ W_ : minW^ℓ‖ΔWℓ‖FWℓ2=minW^ℓvec(ΔWℓ)⊤(Sℓ⊗Aℓ)vec(ΔWℓ). _ W_ \ \| W_ \|_F_W_ ^2= _ W_ \ vec( W_ )^\! (S_ \! \!A_ )\,vec( W_ ). (12) Exploiting the Kronecker factorization, this quadratic form admits an equivalent Frobenius expression in whitened coordinates: ‖ΔWℓ‖Sℓ⊗Aℓ2=‖Sℓ1/2ΔWℓAℓ1/2‖F2.\| W_ \|_S_ A_ ^2=\|S_ ^1/2\, W_ \,A_ ^1/2\|_F^2. (13) A short derivation is instructive. Using the matrix identity vec(UVW)=(W⊤⊗U)vec(V)vec(UVW)=(W^\! \! \!U)\,vec(V), we have vec(Sℓ1/2ΔWℓAℓ1/2)=((Aℓ1/2)⊤⊗Sℓ1/2)vec(ΔWℓ), \! (S_ ^1/2 W_ A_ ^1/2 )= ((A_ ^1/2)^\! \! \!S_ ^1/2 )vec( W_ ), (14) and therefore ‖Sℓ1/2ΔWℓAℓ1/2‖F2 \|S_ ^1/2 W_ A_ ^1/2 \|_F^2 =vec(ΔWℓ)⊤(Aℓ1/2⊗Sℓ1/2)⊤(Aℓ1/2⊗Sℓ1/2)vec(ΔWℓ) =vec( W_ )^\! \! (A_ ^1/2\! \!S_ ^1/2 )^\! \!\! (A_ ^1/2\! \!S_ ^1/2 )\!vec( W_ ) =vec(ΔWℓ)⊤(Sℓ⊗Aℓ)vec(ΔWℓ), =vec( W_ )^\! (S_ \! \!A_ )\,vec( W_ ), which recovers Eq. (12). Because the empirical covariances SℓS_ and AℓA_ may be rank-deficient when the calibration set is small, in practice we add a small diagonal damping term, S~ℓ=Sℓ+δI S_ =S_ +δ I and A~ℓ=Aℓ+δI A_ =A_ +δ I, before taking matrix square roots. This guarantees invertibility and numerical stability without altering the qualitative geometry. Whitening and Euclidean reduction. Define the geometry-aligned variables W~ℓ=Sℓ1/2WℓAℓ1/2 W_ =S_ ^1/2W_ A_ ^1/2 and W^~ℓ=Sℓ1/2W^ℓAℓ1/2 W_ =S_ ^1/2 W_ A_ ^1/2. Substituting into Eq. (13) immediately gives minW^ℓ‖Sℓ1/2(Wℓ−W^ℓ)Aℓ1/2‖F2≡minW^~ℓ‖W~ℓ−W^~ℓ‖F2. _ W_ \ \|S_ ^1/2(W_ - W_ )A_ ^1/2\|_F^2\ ≡\ _ W_ \ \| W_ - W_ \|_F^2. (15) Eq. (15) reveals the central benefit of our formulation: any Euclidean PTQ solver that minimizes Frobenius reconstruction—such as GPTQ, AWQ, or SmoothQuant—can be applied directly to W~ℓ W_ and still optimize the original Riemannian objective. This yields the plug-and-play property exploited in our experiments (Section V-C). Crucially, the whitening transformation is used only for calibration-time scoring and optimization. The deployed quantized model does not require additional runtime operators, since the whitening factors Sℓ1/2S_ ^1/2 and Aℓ1/2A_ ^1/2 are absorbed into the quantized parameters offline. Geometry-aware activation quantization. Activation quantization in VLMs is fragile because of modality-dependent ranges and sensitivities. Let xℓ,tx_ ,t denote the token activation vector at layer ℓ and token t. We calibrate scales by minimizing a modality-weighted reconstruction objective: minsℓt∈ℓ[πℓ(m(t))‖xℓ,t−Q(xℓ,t,sℓ,b)‖22], _s_ \ E_t _ [ _ ^(m(t))\|x_ ,t-Q(x_ ,t;s_ ,b)\|_2^2 ], (16) where m(t)m(t) is the modality of token t. The weight πℓ(m) _ ^(m) allocates calibration capacity proportionally to modality sensitivity rather than to raw token frequency, which empirically improves stability over a single unweighted Euclidean scale. Global geometry regularization. Local reconstruction losses for weights and activations do not constrain the global semantic structure of the embedding space. Under aggressive low-bit quantization, even small per-layer errors can compound and distort representation similarities, harming downstream tasks that depend on cross-modal alignment. We therefore optionally regularize calibration so that the pairwise similarity structure of the calibration batch ℬB is preserved. Let k∈ℝdz_k ^d be the representative embedding (e.g., pooled vision-language feature) for the k-th sample, and define the cosine similarities sij=i⊤j‖i‖2‖j‖2,s^ij=^i⊤^j‖^i‖2‖^j‖2,s_ij= z_i^\! z_j\|z_i\|_2\|z_j\|_2, s_ij= z_i^\! z_j\| z_i\|_2\| z_j\|_2, (17) for the full-precision and quantized models, respectively. To prevent collapse of the semantic geometry, we minimize the discrepancy between similarity matrices: ℒgeom=1|ℬ|2∑i∈ℬ∑j∈ℬ(sij−s^ij)2.L_geom= 1|B|^2 _i _j (s_ij- s_ij)^2. (18) This pairwise constraint stabilizes calibration against local minima that yield small per-layer errors but large global distortions. IV-C Geometry-Guided Error Steering Whitening aligns the metric used to measure quantization error, yet it does not by itself ensure that the realized perturbations avoid the high-curvature, loss-sensitive directions of the unified metric. To directly shape the directionality of noise, we introduce error steering via lightweight sparse rotations that reparameterize the channel coordinates. For each layer ℓ , we apply an orthogonal transform RℓR_ on the input-channel dimension, parameterized as a product of sparse Givens layers: Rℓ=Rℓ,K⋯Rℓ,2Rℓ,1,R_ =R_ ,K·s R_ ,2R_ ,1, (19) where each Rℓ,kR_ ,k is block-diagonal and consists of disjoint ×22\!×\!2 Givens rotations on non-overlapping channel pairs. Concretely, in the k-th layer we partition the channel indices into P≤⌊din/2⌋P\!≤\! d_in/2 disjoint pairs (up,vp)p=1P\(u_p,v_p)\_p=1^P and apply a planar rotation on each pair: G(u,v,θ)=[cosθ−sinθsinθcosθ],G(u,v;θ)= bmatrix θ&- θ\\ θ& θ bmatrix, (20) while leaving all other channels unchanged. Thus Rℓ,kR_ ,k contains ×22\!×\!2 rotation blocks on the paired coordinates and identity blocks elsewhere, and RℓR_ is formed by composing K such sparse layers. For the linear projection Yℓ=WℓXℓY_ =W_ X_ , we apply the orthogonal reparameterization Xℓ′=Rℓ⊤Xℓ,Wℓ′=WℓRℓ,X_ =R_ ^\! X_ , W_ =W_ R_ , (21) which preserves the full-precision mapping since WℓXℓ=Wℓ′Xℓ′W_ X_ \!=\!W_ X_ . We then quantize Wℓ′W_ and Xℓ′X_ . By changing the basis in which quantization is performed, the induced low-bit perturbations become concentrated on geometry-insensitive directions under the unified metric, which we will show improves stability in the low-bit regime. Geometry-aware channel importance. Channel pairing for the sparse Givens layers must balance two factors: modality-dependent loss sensitivity and channel-wise curvature. Let hℓ,u(m)h_ ,u^(m) denote the diagonal curvature proxy of channel u under modality m, estimated from the diagonal entries of the modality-partitioned Fisher factors Sℓ(m)S_ ^(m) and Aℓ(m)A_ ^(m). We then define ωℓ,u(m) _ ,u^(m) =πℓ(m)⋅hℓ,u(m), = _ ^(m)· h_ ,u^(m), (22) ω~ℓ,u ω_ ,u =Normalize(∑mωℓ,u(m)), =Normalize ( _m _ ,u^(m) ), which serves as a unified importance score that modulates a channel-wise curvature signal with the modality sensitivity weights πℓ(m) _ ^(m). The same scores also guide layer-wise quantization hyperparameters (group-wise clipping and scale calibration), so that directions with larger ω~ ω incur smaller quantization perturbations. Rotation selection objective. We select RℓR_ to minimize the unified quantization error under the Riemannian geometry in the reparameterized coordinates. Let ΔWℓ′=Wℓ′−W^ℓ′ W_ =W_ - W_ denote the rotated weight error. Because Wℓ′=WℓRℓW_ =W_ R_ , the input-side curvature factor updates as Aℓ′=Rℓ⊤AℓRℓA _ =R_ ^\! A_ R_ . We score a candidate RℓR_ using the whitening-based Euclidean reduction: ℓ(Rℓ) _ (R_ ) =‖Sℓ1/2ΔWℓ′(Aℓ′)1/2‖F2 =\|S_ ^1/2\, W_ \,(A _ )^1/2\|_F^2 (23) +λAt[πℓ(m(t))‖xℓ,t′−Q(xℓ,t′,sℓ,8)‖22], + _A\,E_t\! [ _ ^(m(t))\|x _ ,t-Q(x _ ,t;s_ ,8)\|_2^2 ], where xℓ,t′x _ ,t denotes the rotated activation at layer ℓ . Because RℓR_ is sparse, computing Aℓ′A _ is efficient. Greedy discrete search. The joint optimization over all Givens pairs is combinatorial, so we adopt a greedy procedure. For each layer k∈1,…,Kk\!∈\!\1,…,K\, we iterate through the disjoint channel pairs kP_k ordered by ω~ ω, and for each pair (i,j)(i,j) perform a discrete grid search over a candidate angle set Θ . In practice we use a uniform grid Θ=0,π/16,…,π =\0,\,π/16,\,…,\,π\ to balance search precision and calibration time, and the selection criterion is the minimization of ℓJ_ defined in Eq. (23). Algorithm 2 executes geometry-guided error steering, where sparse rotations are iteratively optimized to minimize the whitened quantization error (Eq. (23)) under the sensitivity-weighted curvature importance score ω~ℓ,u ω_ ,u derived in Algorithm 1. Algorithm 2 Geometry-Guided Sparse Rotation 1: Weights Wℓ\W_ \, metrics Aℓ(m),Sℓ(m),πℓ(m)\A_ ^(m),S_ ^(m), _ ^(m)\, Givens depth K, angle set Θ 2: Sparse rotations Rℓ=1L\R_ \_ =1^L 3: for ℓ=1 =1 to L do 4: Initialize Rℓ←IR_ ← I 5: Compute input-channel importance: 6: hℓ,u(m)←Aℓ,uu(m)⋅tr(Sℓ(m))/douth_ ,u^(m)← A_ ,u^(m)·tr(S_ ^(m))/d_out 7: ω~ℓ,u←Normalize(∑mπℓ(m)hℓ,u(m)) ω_ ,u \! ( _m _ ^(m)h_ ,u^(m) ) 8: for k=1k=1 to K do 9: for each (i,j)∈k(i,j) _k do 10: Select the angle by greedy search: 11: θ⋆←argminθ∈Θℓ(RℓG(i,j,θ))θ ← _θ∈ J_ \! (R_ G(i,j,θ) ) 12: Update Rℓ←RℓG(i,j,θ⋆)R_ ← R_ G(i,j,θ ) 13: end for 14: end for 15: end for 16: return Rℓ=1L\R_ \_ =1^L Complexity and inference overhead. A key practical advantage of the sparse Givens parameterization over dense rotation matrices is efficiency during calibration. For curvature updates, updating a Hessian approximation with a dense rotation requires (d3)O(d^3) operations, whereas a single Givens rotation only affects two rows and two columns, reducing the update cost to (d)O(d). Across K layers of Givens rotations covering d/2d/2 pairs each, the total complexity is (Kd2)O(Kd^2), substantially lower than dense operations when K≪dK\! \!d. For metric evaluation, the whitening transformation Sℓ1/2ΔWℓ′(Aℓ′)1/2S_ ^1/2 W_ (A _ )^1/2 exploits the sparsity of RℓR_ via incremental updates of Aℓ′A _ , avoiding repeated expensive matrix decompositions. For inference, the weight rotation Wℓ′=WℓRℓW_ \!=\!W_ R_ is pre-computed offline so that the deployed model stores W^ℓ′ W_ directly, with zero runtime overhead for weights. The activation rotation Xℓ′=Rℓ⊤XℓX_ \!=\!R_ ^\! X_ must be applied online, but each Givens layer adds only 4K4K multiplications per channel because every block is a ×22\!×\!2 matrix. This is computationally much cheaper than the dense matrix multiplications used by QuIP# or SpinQuant (d multiplications per channel), making RGSQ particularly hardware-friendly for latency-critical deployments. IV-D Overall Calibration Loss We combine the three pieces—geometry-aware weight reconstruction, modality-weighted activation reconstruction, and optional global semantic regularization—into a single calibration objective: ℒcal _cal =λW∑ℓ‖Sℓ1/2(Wℓ′−W^ℓ′)(Aℓ′)1/2‖F2 = _W _ \|S_ ^1/2(W_ - W_ )\ (A _ )^1/2\|_F^2 (24) +λA∑ℓt[πℓ(m(t))∥Δx′ℓ,t∥22]+λgeomℒgeom, + _A _ E_t [ _ ^(m(t))\| x _ ,t\|_2^2 ]+ _geomL_geom, where the first term measures weight quantization perturbations under the unified Riemannian geometry in reparameterized coordinates, the second term calibrates activation quantization with modality sensitivity weights, and ℒgeomL_geom is an optional regularizer used to stabilize calibration-time hyperparameter selection. Algorithm 3 performs quantization in the geometry-aligned space: by applying the whitening transformation defined by the updated curvature Aℓ′A_ (Eq. (15)), the Riemannian objective reduces to a tractable Euclidean form, so that low-bit perturbations align with the cross-modal sensitivity structure. Algorithm 3 RGSQ Quantization 1: Weights Wℓ\W_ \, metrics Aℓ,Sℓ,πℓ(m)\A_ ,S_ , _ ^(m)\, rotations Rℓ\R_ \, bit-widths bW,bAb_W,b_A 2: Quantized weights W^ℓ′\ W _ \ and activation scales sℓ\s_ \ 3: for ℓ=1 =1 to L do 4: Rotate weights and activations: 5: Wℓ′←WℓRℓ,Xℓ′←Rℓ⊤XℓW _ ← W_ R_ , 9.24994ptX _ ← R_ X_ 6: Rotate input metric: 7: Aℓ′←Rℓ⊤AℓRℓA _ ← R_ A_ R_ 8: Quantize weights by the whitened PTQ objective: 9: W^ℓ′←argminW¯∈bW(Wℓ′)ℰℓW(W¯,Sℓ,Aℓ′) W _ ← _ W _b_W(W _ )E^W_ ( W;S_ ,A _ ) 10: Calibrate activation scale by modality-weighted error: 11: sℓ←argminsℰℓA(s,πℓ(m),Xℓ′,bA)s_ ← _sE^A_ (s; _ ^(m),X _ ,b_A) 12: end for 13: return W^ℓ′,Rℓ,sℓ\ W _ \,\R_ \,\s_ \ Why RGSQ is more than a reweighting. Compared with scalar reweighting strategies such as MBQ, RGSQ does not merely rebalance a per-modality scalar in a Euclidean loss. Instead, it computes a full d×d\!×\!d covariance matrix Aℓ(m)A_ ^(m) per modality, capturing the complete directional structure of how each modality’s activations are distributed. The unified metric Aℓ=∑mπℓ(m)Aℓ(m)A_ = _m _ ^(m)A_ ^(m) is therefore itself a d×d\!×\!d matrix, not a scalar, and encodes which specific weight directions are sensitive to which modality. The Riemannian norm ‖Sℓ1/2ΔWAℓ1/2‖F2\|S_ ^1/2 WA_ ^1/2\|_F^2 then penalizes perturbations proportionally to their actual impact on the loss landscape, rather than to their bare Euclidean magnitude. V Experiments V-A Experimental Setups TABLE I: Main results on the small models of LLaVA-onevision, InternVL2, and Qwen2-VL families. † denotes our reported results. Model Bitwidth Method MMMU SEED OCRBench VizWiz ScienceQA TextVQA Average (↑ ) LLaVA-onevision-7B FP16 - 46.0 74.9 62.2 60.4 85.4 76.1 67.5 W3A16 RTN 34.7 5.9 35.9 59.2 86.2 60.9 47.1 GPTQ 41.9 72.9 55.7 56.4 86.4 71.3 64.1 AWQ 36.6 53.0 59.3 58.5 83.2 73.0 60.6 MBQ 42.0 69.7 61.1 60.7 85.0 73.3 65.3 RGSQ 47.5 72.0 61.8 60.2 88.5 75.2 67.5 W4A8 RTN 38.2 50.3 40.1 58.2 88.3 61.5 56.1 SQ 30.9 42.7 32.0 56.7 87.1 56.9 51.1 MBQ 42.6 67.7 52.3 58.9 88.5 68.3 63.1 MQuant† 45.3 73.7 58.8 59.4 84.0 74.9 66.0 RGSQ 48.4 72.4 62.0 60.1 89.9 75.9 68.1 InternVL2-8B FP16 - 48.0 76.0 76.5 61.1 96.2 77.0 72.5 W3A16 RTN 43.7 74.9 74.0 56.0 95.6 74.6 69.8 GPTQ 41.7 73.4 70.2 59.9 89.5 73.1 68.0 AWQ 44.8 75.2 74.7 58.9 95.5 74.2 70.6 MBQ 46.9 75.4 75.1 58.7 95.6 75.1 71.1 RGSQ 47.6 75.8 76.0 60.5 96.0 76.5 72.1 W4A8 RTN 44.3 74.0 72.0 57.1 95.5 73.1 69.3 SQ 41.8 73.8 70.9 54.2 95.1 72.6 68.1 MBQ 45.6 74.3 73.0 56.5 95.8 72.3 69.6 MQuant† 46.9 74.3 69.9 59.8 94.1 76.0 70.2 RGSQ 48.2 76.0 76.4 61.0 96.4 77.0 72.5 Qwen2-VL-7B FP16 - 50.6 76.4 80.7 68.3 85.1 82.0 73.8 W3A16 RTN 44.9 74.8 60.0 65.2 81.5 71.2 66.3 GPTQ 43.1 73.7 74.8 64.3 79.7 76.7 68.7 AWQ 44.7 75.1 76.9 68.0 82.5 79.5 71.1 MBQ 47.9 74.8 76.8 67.7 82.8 79.9 71.6 RGSQ 50.0 76.0 79.8 67.8 84.5 81.2 73.2 W4A8 RTN 43.8 74.9 60.3 58.9 78.9 71.0 64.6 SQ 45.9 75.0 57.1 52.3 80.9 68.2 63.2 MBQ 47.2 75.1 72.8 59.3 81.2 75.0 68.4 MQuant† 48.6 71.1 79.2 65.4 83.5 79.6 71.3 RGSQ 50.8 76.4 80.5 68.2 85.2 82.0 73.8 Calibration dataset. The calibration set leverages a VLM’s joint perception–generation ability, which is naturally exercised by image-captioning data. Specifically, we adopt the refined COCO Caption collection released as part of ShareGPT4V [41], consistent with AWQ, MBQ, and MQuant. Each calibration sample consists of (i) a single COCO image and (i) a detailed caption generated by GPT-4V, formatted as a multi-turn conversation using the model-specific chat template. During preprocessing, the image is encoded by the model’s vision tower into visual tokens, while the caption is tokenized into roughly 6464–256256 text tokens depending on length. The two sequences are then concatenated into one input, ensuring that both modalities are present in every calibration forward pass. Evaluation benchmarks. We evaluate quantization quality on a broad suite of vision-language benchmarks through the LMMs-Eval framework [42]: OCRBench [43] and TextVQA [44] for text recognition and understanding. VizWiz [45] and SEED-Bench [46] for general visual perception, and ScienceQA [47] together with MMMU [48] for multimodal reasoning. To probe generalization beyond static images, we additionally evaluate on the SEED-Video [46] and Video-MME [49] splits. Models. We evaluate three representative VLM families: LLaVA-OneVision [50], InternVL2 [51], and Qwen2-VL [52]. Within each family we include one compact and one large-scale variant to verify scaling. For LLaVA-OneVision we test the 7B and 72B variants, which pair Qwen2-7B/-72B as the language backbone with SigLIP-400M [53] as the ViT encoder. For InternVL2 we consider the 8B and 26B models, built on InternLM2-8B/-20B with InternViT-300M/-6B as the visual encoder. For Qwen2-VL we cover the 7B and 72B models, using Qwen2-7B/-72B with a 675M ViT encoder. Baselines. TABLE I: Default quantization hyperparameters used in all experiments unless otherwise stated. Category Hyperparameter Value Weight quantization Bit-width bWb_W 4 Scheme Symmetric, per-group Group size 128 Activation quantization Bit-width bAb_A 8 Scheme Symmetric, per-token GPTQ solver Block size 128 Damping factor (percdamp) 0.01 Activation order Disabled Riemannian metric Sensitivity norm ℓ1 _1 (Eq. (9)) Weighting mode πℓ(m)∝αℓ(m) _ ^(m)\! \! _ ^(m) Covariance damping δ ×10−41\!×\!10^-4 Calibration loss λW _W 1.0 λA _A 1.0 λgeom _geom 0.2 Sparse rotation Givens layers K 4 Following recent work [18, 19], we organize our comparisons into two complementary tiers and discuss them separately throughout this section. (i) Classical Euclidean PTQ baselines, including round-to-nearest (RTN), GPTQ [26], AWQ [11], and SmoothQuant (SQ)[14]. These methods are widely deployed in practical PTQ pipelines and provide a natural test bed for evaluating whether the unified Riemannian metric can improve Euclidean PTQ methods. (i) VLM-specific quantization baselines, including Q-VLM, MBQ, MQuant, QuaRot[27], and SpinQuant [15]. These methods cover representative multimodal quantization strategies, including modality-balanced reweighting, modality-specific static quantization, and dense rotation-based transformations. For a fair comparison, all methods use group-wise quantization with group size 128128. For weight–activation joint quantization (W4A8), activations are quantized symmetrically per token, while weights are quantized symmetrically per output channel, enabling efficient execution on tensor cores. The default RGSQ hyperparameters are summarized in Table I. Implementation details. All experiments are conducted on NVIDIA A100-80GB and A800-80GB GPUs. The implementation is in Python 3.10 with PyTorch 2.9.0, CUDA 12.8, and Transformers 4.45.0. Inference follows the default evaluation settings of LMMs-Eval 11 1 https://github.com/EvolvingLMMs-Lab/lmms-eval. The code is publicly available at RGSQ 22 2 https://github.com/RL-MIND/RGSQ. TABLE IV: Ablation study on LLaVA-OneVision-7B. RMAM: Modality-partitioned statistics with a unified Kronecker-Fisher metric. ROW: Riemannian objectives with whitening-based Euclidean reduction. GGES: Geometry-guided error steering via sparse Givens rotations. Components MMMU SEED OCRB. VizWiz SciQA TextVQA Avg↑ RMAM ROW GGES FP16 46.0 74.9 62.2 60.4 85.4 76.1 67.5 ✗ ✗ ✗ 38.2 50.3 40.1 58.2 88.3 61.5 56.1 ✓ ✗ ✗ 44.8 66.5 54.2 59.5 88.8 70.8 64.1 ✓ ✓ ✗ 47.2 70.1 59.0 60.0 89.2 74.0 66.6 ✓ ✓ ✓ 49.2 72.4 62.3 60.4 90.0 76.0 68.4 V-B Main Results TABLE V: Main results on the large models of LLaVA-onevision, InternVL2, and Qwen2-VL families. † denotes our reported results. Model Bitwidth Method MMMU SEED OCRBench VizWiz ScienceQA TextVQA Average (↑ ) LLaVA-onevision-72B FP16 - 56.1 78.1 73.2 69.2 90.0 79.3 74.3 W3A16 RTN 53.9 77.4 68.2 66.1 89.5 77.4 72.1 GPTQ 52.7 76.0 69.7 68.3 89.3 77.9 72.3 AWQ 33.4 71.2 48.7 49.3 69.2 58.8 55.1 MBQ 54.4 77.6 71.6 69.0 90.3 78.5 73.6 RGSQ 55.8 77.9 72.8 69.0 90.0 78.8 74.1 W4A8 RTN 54.8 76.6 64.5 64.7 89.0 74.5 70.7 SQ 51.6 76.6 64.2 65.7 89.1 74.4 70.3 MBQ 55.6 76.5 64.4 65.7 89.2 73.3 70.8 RGSQ 56.0 78.0 73.0 69.0 90.0 79.2 74.2 InternVL2-26B FP16 - 47.1 76.8 77.9 66.2 97.5 82.1 74.6 W3A16 RTN 46.6 75.7 75.9 64.7 96.4 80.6 73.3 GPTQ 44.8 75.8 76.0 60.9 96.3 80.1 72.3 AWQ 46.4 76.2 76.4 64.5 96.7 81.0 73.5 MBQ 47.1 76.3 76.5 64.5 97.3 81.1 73.8 RGSQ 47.0 76.6 77.5 65.8 97.3 81.6 74.3 W4A8 RTN 44.7 76.0 76.4 62.6 96.7 79.6 72.7 SQ 38.2 70.6 68.5 56.7 86.3 72.6 65.5 MBQ 44.0 75.7 77.5 62.0 97.1 80.0 72.7 RGSQ 47.6 76.8 77.8 66.0 97.4 81.8 74.6 Qwen2-VL-72B FP16 - 61.1 77.6 79.9 76.0 91.6 82.5 78.1 W3A16 RTN 57.7 77.5 70.4 74.8 89.7 79.7 75.0 GPTQ 57.3 77.2 78.5 73.6 91.5 81.6 76.6 AWQ 59.6 77.6 79.6 75.4 90.4 82.4 77.5 MBQ 59.6 77.7 79.4 75.6 90.5 82.5 77.6 RGSQ 60.8 77.6 79.8 75.8 91.2 82.4 77.9 W4A8 RTN 58.1 76.6 66.2 71.3 90.1 77.0 73.2 SQ 55.9 76.4 65.5 69.7 88.8 76.9 72.2 MBQ 57.7 76.3 77.5 73.6 89.6 80.5 75.8 MQuant† 59.8 76.5 78.6 73.5 87.8 81.6 76.3 RGSQ 61.0 77.5 80.0 75.8 91.4 82.4 78.0 Weight-only quantization (W3A16). As shown in Table I and Table V, RTN reveals that smaller VLMs are substantially more sensitive to weight quantization than larger ones: averaged over the three small models, RTN drops by 10.2% from FP16, whereas the drop on the three large models is only 2.2%. This suggests that limited capacity amplifies quantization noise once errors propagate through a mixed-modality token interface. Under W3A16, RGSQ consistently outperforms the modality-agnostic baselines RTN, GPTQ, and AWQ across all six models. Within the LLaVA-OneVision family alone, RGSQ improves the average score over AWQ by +6.9%+6.9\% on the 7B model and by +19.0%+19.0\% on the 72B model. Particularly striking is LLaVA-OneVision-72B, where AWQ collapses by −17.0%-17.0\% relative to RTN, while RGSQ restores performance to 74.1%74.1\%, essentially matching the 74.3%74.3\% FP16. These results corroborate our central claim: explicit modeling of modality-dependent sensitivity is necessary for reliable quantization in VLMs. Weight–activation quantization (W4A8). The same trend holds, with an even larger margin under joint quantization. RGSQ consistently outperforms SmoothQuant and RTN under W4A8, with gains reaching +17.0%+17.0\% in representative cases (Tables I, V). We also observe that SmoothQuant is often inferior to plain RTN on InternVL2-26B, it drops from 72.7%72.7\% to 65.5%65.5\%, indicating that naive activation re-scaling can exacerbate inter-modality imbalance once activations are quantized. RGSQ, in contrast, achieves stable improvements on both small and large models: it raises InternVL2-26B to 74.5%74.5\% and Qwen2-VL-72B to 78.0%78.0\%, virtually indistinguishable from the 78.1%78.1\% FP16. Overall, W4A8 results further confirm that multimodal-aware quantization is essential, especially once activation quantization makes modality sensitivity disparities more pronounced. V-C Enhancing Baseline Quantizers via ROW TABLE VI: Effect of our Riemannian whitening on existing quantization methods for InternVL2-8B. This geometry-aligned method converts the Riemannian objective into equivalent Euclidean reconstruction in whitened coordinates, enabling plug-and-play enhancements to standard PTQ solvers during calibration without modality-specific scaling. Model Bitwidth Method MMMU SEED OCRBench VizWiz ScienceQA TextVQA Average (↑ ) InternVL2-8B FP16 - 48.0 76.0 76.5 61.1 96.2 77.0 72.5 W4A8 RTN 44.3 74.0 72.0 57.1 95.5 73.1 69.3 w/ WER 46.2 75.0 74.5 58.8 95.9 75.0 70.9 (+1.6) AWQ 41.8 73.8 70.9 54.2 95.1 72.6 68.1 w/ WER 44.5 74.8 73.8 57.0 95.6 74.5 70.0 (+1.9) SQ 45.6 74.3 73.0 56.5 95.8 72.3 69.6 w/ WER 47.0 75.4 75.2 59.2 96.0 75.6 71.4 (+1.8) A direct corollary of Eq. (15) is that any Euclidean block-wise PTQ method can be wrapped with our Riemannian whitening (ROW) to inherit modality-aware curvature scoring at calibration time, while leaving the runtime computation graph unchanged. We verify this claim on InternVL2-8B under W4A8 (Table VI). The vanilla RTN, AWQ, and SmoothQuant baselines incur 2.92.9–3.2%3.2\% average degradations against FP16, with particularly severe losses on vision-reliant tasks such as VizWiz. Adding ROW yields a consistent uplift of +1.6%+1.6\% to +1.9%+1.9\% across all three baselines without altering their core quantization logic, closing the FP16 gap to within 1.1%1.1\%. These results show that the Riemannian whitening reduction is a broadly applicable enhancement, not a single-method trick. V-D Low-Bit Quantization for VLMs Fig. 3: Average performance across three benchmarks (ScienceQA, SEED, and TextVQA) under low-bit quantization settings (W2A8 and W3A8). RSAVQ is designed for LLMs and shows limited gains on VLMs, while MBQ and RGSQ are VLM-aware methods. Fig. 4: Low-bit quantization results under W2A8 and W3A8 settings. RSAVQ is designed for LLMs and shows limited gains on MLLMs, while MBQ and RGSQ are MLLM-aware methods. We further evaluate the more aggressive W2A8 and W3A8 settings, where the assumptions of Euclidean PTQ become increasingly fragile. Under such extreme bit widths, RTN suffers substantial accuracy degradation, especially on vision-intensive benchmarks, as shown in Fig.3. MBQ remains limited despite its VLM-aware design, while RSAVQ has only a modest effect in this setting because it was originally developed for LLM quantization rather than VLMs. As shown in Fig.4, RGSQ improves over RTN by up to +14.6%+14.6\% under W2A8 and consistently outperforms both MBQ and RSAVQ. Under W3A8, RGSQ reduces the average gap to FP16 to only 1.51.5–2.1%2.1\%. This advantage becomes more evident on larger models. Table VII shows that RGSQ outperforms MBQ by +4.0%+4.0\% to +4.2%+4.2\% under W2A8 on Qwen2-72B and InternVL2-26B, where Euclidean reconstruction and scalar reweighting are particularly limited. These results are consistent with the motivation of the Riemannian formulation. As bit widths decrease, the direction of quantization error has a stronger effect on task loss, and directional curvature information becomes essential for redistributing the quantization budget across modalities. TABLE VII: Low-bit quantization results on large VLMs. We report the 6-benchmark average accuracy under W2A8 and W3A8 settings. Model Setting RTN MBQ RGSQ Δ vs. MBQ Qwen2-VL-72B W2A8 (avg) 68.6 74.5 78.7 +4.2 W3A8 (avg) 75.2 79.7 82.4 +2.7 InternVL2-26B W2A8 (avg) 71.0 76.4 80.4 +4.0 W3A8 (avg) 77.3 81.6 83.8 +2.2 V-E Ablation Study We isolate the contribution of each RGSQ component on LLaVA-OneVision-7B under W4A8, with the modality-agnostic Euclidean PTQ as the starting point (Table IV). The baseline yields an average accuracy of 56.1%56.1\%, a substantial −11.4%-11.4\% drop relative to the 67.5%67.5\% FP16. Adding the Riemannian Modality-Aware Metric (RMAM), which builds a unified Kronecker–Fisher metric from modality-partitioned statistics, recovers +8.0%+8.0\% to 64.1%64.1\%, confirming that modality-specific curvature estimation is the key to stable multimodal calibration. Layering on Riemannian Objectives with Whitening (ROW) further raises the average to 66.6%66.6\%, showing that geometry-aligned reconstruction outperforms isotropic Euclidean objectives once the metric is in place. Finally, activating Geometry-Guided Error Steering (GGES) via sparse Givens rotations brings the model to 68.4%68.4\%, slightly surpassing the FP16 baseline. V-F Generalization to Video Understanding TABLE VIII: Generalization to video understanding. RGSQ consistently improves over MBQ on SEED-Image, SEED-Video, and Video-MME. Model Method SEED-Image SEED-Video Video-MME InternVL2-8B FP16 76.2 – 56.9 MBQ 74.3 50.8 51.2 RGSQ 76.0 52.4 54.3 LLaVA-OneVision-7B FP16 74.9 56.9 58.2 MBQ 69.7 50.7 54.0 RGSQ 72.4 53.5 56.9 Although RGSQ is calibrated on a small set of static image-text pairs and is not designed with video models in mind, the learned modality-aware Riemannian metric may remain informative for video tokens, since these tokens inherit the same visual-versus-text statistical asymmetry observed in the image setting. As a preliminary check on this hypothesis, we directly evaluate the W4A8-quantized models on the SEED-Image, SEED-Video, and Video-MME benchmarks under the LMMs-Eval video protocols, without performing any additional video calibration. As reported in Table VIII, RGSQ improves over the MBQ baseline across all three benchmarks for both InternVL2-8B and LLaVA-OneVision-7B, with the video-specific gains on SEED-Video and Video-MME ranging from 1.6%1.6\% to 3.1%3.1\%. These early results suggest that the unified Riemannian metric retains part of its benefit in temporal multimodal settings and does not appear to be tightly bound to static image distributions. These experiments are intended as an initial verification. RGSQ is developed as a general-purpose VLM quantization framework, so the observed video gains are better read as a sign of generalization than as evidence of video-specialized performance. A more thorough evaluation with video-specific calibration and a broader set of video benchmarks is left to future work. V-G Layer-Wise Geometric Diagnostics Fig. 5: Layer-wise geometric diagnostics on LLaVA-OneVision-7B under W4A8 quantization. We report modality eigenspace divergence and the corresponding reduction in normalized quantization error. To understand the source of these improvements, we probe the layer-wise geometry of LLaVA-OneVision-7B under W4A8 quantization (Fig. 5). At five representative layers, we measure the cosine similarity between the leading eigenvectors of the modality-partitioned activation covariances Aℓ(v)A_ ^(v) and Aℓ(t)A_ ^(t), and the leading eigenvalue ratio λ1(v)/λ1(t) _1^(v)/ _1^(t). Lower cosine similarity indicates more orthogonal modality subspaces, while a higher eigenvalue ratio indicates stronger energy asymmetry. In the mid-fusion block (L14), the cosine similarity drops to 0.150.15, while the eigenvalue ratio rises to 12.5×12.5×. This suggests that vision and text tokens occupy nearly orthogonal subspaces with highly asymmetric energy, which cannot be captured by a single per-modality scalar. Replacing the Euclidean error proxy with the Riemannian metric in Eq. (12) reduces the normalized quantization error by up to 78%78\% at the layers with the largest modality divergence. The reduction is smaller in the generation-heavy tail (L27), where the two modalities are already more aligned. This layer-wise correspondence supports the use of directional curvature information rather than scalar reweighting in RGSQ. V-H Calibration Efficiency Analysis TABLE IX: Comparison of calibration cost and accuracy across different quantization methods. Method Time Mem. GPU-h Avg. Acc. Δ FP16 GPTQ 35 min 22 GB 0.58 64.1% −3.4-3.4 AWQ 25 min 20 GB 0.42 60.6% −6.9-6.9 MBQ 45 min 24 GB 0.75 65.8% −1.7-1.7 MQuant 1.2 h 28 GB 1.2 66.6% −0.9-0.9 RGSQ (Ours) 2.1 h 38 GB 2.1 68.1% +0.6+0.6 Table IX quantifies the practical cost of RGSQ relative to lightweight PTQ baselines. On LLaVA-OneVision-7B at W4A8, the full pipeline requires approximately 2.12.1 GPU-hours and 3838 GB of peak memory on a single A800, compared with 0.420.42–1.21.2 GPU-hours for GPTQ, AWQ, MBQ, and MQuant. The additional overhead stems from two sources: estimation of the modality-partitioned activation and gradient second-order factors, which is more expensive than purely feed-forward calibrators such as AWQ, and the sparse Givens search, which introduces a structured discrete optimization step that nonetheless remains cheaper and far more parameter-efficient than unconstrained dense rotation learning. Because the whitening step is applied only at calibration time and is absorbed into the quantized parameters before deployment, RGSQ introduces no extra runtime operator. Despite the moderate increase in offline cost, RGSQ achieves superior performance and the highest accuracy among all compared methods, surpassing even the FP16 baseline. VI Conclusion and Limitation In this paper, we introduced RGSQ, a geometry-sensitive PTQ framework for vision–language models. Our experiments reveal a central observation: the quantization error of VLMs is governed not only by its magnitude, but also by its modality-dependent geometric direction. Vision and text tokens often induce distinct layer-wise curvature and sensitivity patterns, making conventional Euclidean reconstruction, scalar reweighting, or modality-agnostic rotations insufficient for reliable low-bit quantization. RGSQ addresses this issue by measuring quantization error on a unified modality-aware Riemannian manifold and steering the error toward low-sensitivity directions. This geometric treatment consistently improves weight-only and weight–activation quantization across model families, scales, and bit-widths, and remains effective when used as a plug-in enhancement for existing Euclidean PTQ solvers. The gains are most pronounced under aggressive low-bit settings and at layers with strong vision–text geometric divergence, suggesting that modality-aware curvature is a key factor for robust VLM compression. RGSQ introduces a moderate one-time calibration overhead from layer-wise sensitivity estimation and sparse rotation search. A practical future direction is to improve the efficiency of geometry-aware calibration through faster Fisher estimation, lighter rotation search, and finer token-level sensitivity modeling. References [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems. Cited by: §I. [2] D. Guo, C. Xu, N. Duan, J. Yin, and J. McAuley (2023) Longcoder: a long-range pre-trained language model for code completion. In International Conference on Machine Learning, p. 12098–12107. Cited by: §I. [3] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §I. [4] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. (2024) Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Cited by: §I. [5] H. Zhang, Z. Ji, J. Liu, Y. Pang, and J. Han (2026) Multi-stage knowledge integration of vision-language models for continual learning. IEEE Transactions on Image Processing 35 (), p. 615–628. Cited by: §I. [6] Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia (2026) Mini-gemini: mining the potential of multi-modality vision language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (3), p. 3530–3543. Cited by: §I. [7] Meta AI (2024) Introducing Meta Llama 3: the most capable openly available LLM to date. Note: Meta AIVol. 2, No. 5, p. 6 Cited by: §I. [8] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré (2022) Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35, p. 16344–16359. Cited by: §I. [9] T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao (2024) Medusa: simple llm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, p. 5209–5235. Cited by: §I. [10] Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer (2019) Hawq: hessian aware quantization of neural networks with mixed-precision. In IEEE/CVF International Conference on Computer Vision, Cited by: §I. [11] J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han (2024) AWQ: activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems 6, p. 87–100. Cited by: §I, §I-A, §V-A. [12] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2022) GPTQ: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: §I. [13] J. Chee, Y. Cai, V. Kuleshov, and C. M. De Sa (2024) Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems 36. Cited by: §I. [14] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han (2023) Smoothquant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, p. 38087–38099. Cited by: §I, §V-A. [15] Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort (2024) SpinQuant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406. Cited by: §I, §V-A. [16] Y. Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y. Li, J. Hu, X. Yu, L. Hou, C. Yuan, et al. (2024) Flatquant: flatness matters for llm quantization. arXiv preprint arXiv:2410.09426. Cited by: §I. [17] C. Wang, Z. Wang, X. Xu, Y. Tang, J. Zhou, and J. Lu (2024) Q-VLM: post-training quantization for large vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §I, §I-B. [18] S. Li, Y. Hu, X. Ning, X. Liu, K. Hong, X. Jia, X. Li, Y. Yan, P. Ran, G. Dai, et al. (2025) Mbq: modality-balanced quantization for large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4167–4177. Cited by: §I, §I-B, §V-A. [19] J. Yu, S. Zhou, D. Yang, S. Li, S. Wang, X. Hu, C. Xu, Z. Xu, C. Shu, and Z. Yuan (2025) Mquant: unleashing the inference potential of multimodal large language models via static quantization. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 1783–1792. Cited by: §I, §I-B, §V-A. [20] Z. Xu, X. Hu, Q. Wu, and D. Yang (2025) RSAVQ: riemannian sensitivity-aware vector quantization for large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §I, §I-A, §I-C. [21] H. Liu, H. Gao, X. Zhang, C. Li, F. Zhang, W. Wang, F. Ma, and H. Yu (2025) Septq: a simple and effective post-training quantization paradigm for large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 812–823. Cited by: §I. [22] M. Wu, J. Li, J. Ji, F. Hao, X. Sun, and R. Ji (2026) Evaluating and mitigating relationship hallucinations in large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (6), p. 6332–6346. Cited by: §I. [23] J. Yang, M. Jiang, and Q. Zhao (2026) Defying distractions in multimodal tasks: a novel benchmark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence 48 (6), p. 6314–6331. Cited by: §I. [24] Z. Wang, X. Zheng, and Y. Peng (2026) FinePruner: unbiased attention-head-level fine-grained token reduction for efficient inference of large vision-language models. IEEE Transactions on Image Processing 35 (), p. 4907–4920. External Links: Document Cited by: §I-A. [25] J. Zhao, C. Zeng, M. Wang, L. Han, Y. Shang, M. Zhang, and L. Nie (2025) LRQuant+: a unified and learnable framework to post-training quantization for transformer-based large foundation models. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (12), p. 11817–11833. External Links: Document Cited by: §I-A. [26] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2022) Gptq: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: §I-A, §V-A. [27] S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman (2024) Quarot: outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456. Cited by: §I-A, §V-A. [28] X. Hu, Y. Cheng, D. Yang, Z. Xu, Z. Yuan, J. Yu, C. Xu, Z. Jiang, and S. Zhou (2025) OstQuant: refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. CoRR. Cited by: §I-A. [29] Z. Xu, Y. Yue, X. Hu, D. Yang, Z. Yuan, Z. Jiang, Z. Chen, S. Zhou, et al. (2025) MambaQuant: quantizing the mamba family with variance aligned rotation methods. In The Thirteenth International Conference on Learning Representations, Cited by: §I-A. [30] M. Van Baalen, A. Kuzmin, M. Nagel, P. Couperus, A. Bolshakov, C. Bastoul, E. Mahurin, T. Blankevoort, and P. Whatmough GPTVQ: the blessing of dimensionality for llm quantization. In Workshop on Efficient Systems for Foundation Models I@ ICML2024, Cited by: §I-A. [31] Y. Liu, J. Wen, Y. Wang, S. Ye, L. L. Zhang, T. Cao, C. Li, and M. Yang (2024) VPTQ: extreme low-bit vector post-training quantization for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8181–8196. Cited by: §I-A. [32] V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh (2024) Extreme compression of large language models via additive quantization. In Proceedings of the 41st International Conference on Machine Learning, p. 12284–12303. Cited by: §I-A. [33] Y. Xu, S. Ji, Q. Zhu, and W. Che (2025) CRVQ: channel-relaxed vector quantization for extreme compression of llms. Transactions of the Association for Computational Linguistics 13, p. 1488–1506. Cited by: §I-A. [34] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa (2024) QuIP #\#: even better llm quantization with hadamard incoherence and lattice codebooks. In International Conference on Machine Learning, p. 48630–48656. Cited by: §I-A. [35] J. Xie, Y. Zhang, M. Lin, L. Cao, and R. Ji (2024) Advancing multimodal large language models with quantization-aware scale learning for efficient adaptation. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 10582–10591. Cited by: §I-B. [36] Z. Xiang, F. Zeng, H. Fang, R. Wang, R. Chen, Y. Zhu, Y. Chen, P. Yang, and X. Zhang (2026) Fine-grained post-training quantization for large vision language models with quantization-aware integrated gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3500–3510. Cited by: §I-B. [37] T. Dages, S. Weber, Y. E. Lin, R. Talmon, D. Cremers, M. Lindenbaum, A. M. Bruckstein, and R. Kimmel (2025) Finsler multi-dimensional scaling: manifold learning for asymmetric dimensionality reduction and embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 25842–25853. Cited by: §I-C. [38] J. Martens and R. Grosse (2015) Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, p. 2408–2417. Cited by: §I-C, §I-B. [39] H. J. Song and L. Itti (2025) Riemannian-geometric fingerprints of generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11425–11435. Cited by: §I-B. [40] J. Park, J. J. Suh, B. Wang, A. Bhattacharya, and S. Ma (2026) Adaptive gradient descent on riemannian manifolds and its applications to gaussian variational inference. In The Fourteenth International Conference on Learning Representations, Cited by: §I-B. [41] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin (2023) Sharegpt4v: improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793. Cited by: §V-A. [42] K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, and Z. Liu (2024) LMMs-eval: reality check on the evaluation of large multimodal models. External Links: 2407.12772, Link Cited by: §V-A. [43] Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. Liu, L. Jin, and X. Bai (2024) OCRBench: on the hidden mystery of ocr in large multimodal models. External Links: 2305.07895, Link Cited by: §V-A. [44] A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach (2019) Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 8317–8326. Cited by: §V-A. [45] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham (2018) Vizwiz grand challenge: answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3608–3617. Cited by: §V-A. [46] B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan (2023) Seed-bench: benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125. Cited by: §V-A. [47] P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan (2022) Learn to explain: multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), Cited by: §V-A. [48] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. (2024) Mmmu: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556–9567. Cited by: §V-A. [49] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. (2025) Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24108–24118. Cited by: §V-A. [50] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li (2024) LLaVA-onevision: easy visual task transfer. arXiv preprint arXiv:2408.03326. Cited by: §V-A. [51] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24185–24198. Cited by: §V-A. [52] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. Cited by: §V-A. [53] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. External Links: 2303.15343 Cited by: §V-A.