Paper deep dive
Scaling Laws for Precision in High-Dimensional Linear Regression
Dechen Zhang, Xuan Tang, Yingyu Liang, Difan Zou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 8:20:26 PM
Summary
This paper presents a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework. It analyzes the impact of multiplicative (signal-dependent) and additive (signal-independent) quantization on model and data capacities. The key finding is a critical dichotomy: multiplicative quantization preserves the effective model size, while additive quantization reduces it. Both schemes introduce additive error and degrade effective data size. The work provides rigorous upper and lower bounds for population risk, validating empirical observations and offering a theoretical basis for optimizing training protocols under hardware constraints.
Entities (10)
Relation Signals (8)
High-dimensional sketched linear regression → analyzes → Scaling Laws
confidence 95% · we initiate a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework
Multiplicative quantization → preserves → Effective model size
confidence 95% · multiplicative quantization maintains the full-precision model size
Additive quantization → reduces → Effective model size
confidence 95% · additive quantization reduces the effective model size
Multiplicative quantization → correspondsto → Floating-point quantization
confidence 90% · floating-point quantization ... corresponding to multiplicative quantization
Additive quantization → correspondsto → Integer quantization
confidence 90% · integer quantization ... aligns with our definition of additive quantization
Multiplicative quantization → degrades → Effective data size
confidence 90% · while both schemes introduce an additive error and degrade the effective data size
Additive quantization → degrades → Effective data size
confidence 90% · while both schemes introduce an additive error and degrade the effective data size
Quantization → introduces →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-precision training is critical for optimizing the trade-off between model quality and training costs, necessitating the joint allocation of model size, dataset size, and numerical precision. While empirical scaling laws suggest that quantization impacts effective model and data capacities or acts as an additive error, the theoretical mechanisms governing these effects remain largely unexplored. In this work, we initiate a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework. By analyzing multiplicative (signal-dependent) and additive (signal-independent) quantization, we identify a critical dichotomy in their scaling behaviors. Our analysis reveals that while both schemes introduce an additive error and degrade the effective data size, they exhibit distinct effects on effective model size: multiplicative quantization maintains the full-precision model size, whereas additive quantization reduces the effective model size. Numerical experiments validate our theoretical findings. By rigorously characterizing the complex interplay among model scale, dataset size, and quantization error, our work provides a principled theoretical basis for optimizing training protocols under practical hardware constraints.
Tags
Links
- Source: https://arxiv.org/abs/2602.19241v2
- Canonical: https://arxiv.org/abs/2602.19241v2
Trouble viewing inline? Open PDF directly →
Full Text
563,408 characters extracted from source content.
Expand or collapse full text
Scaling Laws for Precision in High-Dimensional Linear Regression Dechen Zhang Xuan Tang Yingyu Liang Difan Zou Institute of Data Science, The University of Hong Kong. Email: dechenzhang@connect.hku.hkSchool of Computing & Data Science, The University of Hong Kong. Email: xuantang8@connect.hku.hkInstitute of Data Science and School of Computing & Data Science, The University of Hong Kong. Email: yingyul@hku.hkSchool of Computing & Data Science and Institute of Data Science, The University of Hong Kong. Email: dzou@hku.hk Abstract Low-precision training is critical for optimizing the trade-off between model quality and training costs, necessitating the joint allocation of model size, dataset size, and numerical precision. While empirical scaling laws suggest that quantization impacts effective model and data capacities or acts as an additive error, the theoretical mechanisms governing these effects remain largely unexplored. In this work, we initiate a theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework. By analyzing multiplicative (signal-dependent) and additive (signal-independent) quantization, we identify a critical dichotomy in their scaling behaviors. Our analysis reveals that while both schemes introduce an additive error and degrade the effective data size, they exhibit distinct effects on effective model size: multiplicative quantization maintains the full-precision model size, whereas additive quantization reduces the effective model size. Numerical experiments validate our theoretical findings. By rigorously characterizing the complex interplay among model scale, dataset size, and quantization error, our work provides a principled theoretical basis for optimizing training protocols under practical hardware constraints. 1 Introduction The remarkable success of large language models (LLMs) has been largely driven by the scaling of model parameters and training datasets, governed by the now-canonical neural scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). However, the prohibitive computational and memory costs associated with such scaling have made low-precision training indispensable (Courbariaux et al., 2014; Wang et al., 2018; Sun et al., 2020; Hao et al., 2025). State-of-the-art frameworks now extensively leverage mixed- or low-precision formats for gradients, weights, and optimizer states (Peng et al., 2023; Wortsman et al., 2023; Xi et al., 2024; Fishman et al., 2024; Liu et al., 2024), showing that aggressively low-precision training can scale to trillion-token workloads without compromising accuracy. This shift fundamentally alters the scaling landscape, introducing a complex interplay between model size, dataset size, and numerical precision. Optimizing the performance of LLMs thus necessitates a rigorous understanding to guide the joint allocation of fixed compute or memory budgets across these three dimensions. Despite the practical urgency, our understanding of low-precision scaling remains predominantly empirical. Recent studies have proposed different functional forms to describe how bit-width affects the scaling behavior in low-precision training (Kumar et al., 2024; Sun et al., 2025). One line of research posits that quantization effectively reduces the model’s capacity: L(M,N,Q)≈AMeff(M,Q)−α+BN−β+EL(M,N,Q)≈ AM_eff(M,Q)^-α+BN^-β+E, where MeffM_eff represents an effective model size reduced by quantization operations (Kumar et al., 2024). While others formulate quantization as an additive error term: L(M,N,Q)≈AM−α+BN−β+E+δ(M,N,Q)L(M,N,Q)≈ AM^-α+BN^-β+E+δ(M,N,Q), where δ acts as an explicit penalty term dependent on quantization (Sun et al., 2025). Crucially, these are purely empirical fits and there exists no unified theoretical framework to determine which formulation, effective size reduction or additive error, is physically correct, nor to mechanistically account for the intricate effects of specific training algorithms and mixed-precision strategies. Recent studies on the theoretical understanding of scaling laws have focused on analyzing the exact training dynamics of SGD using linear models (Lin et al., 2024, 2025; Li et al., 2025; Yan et al., 2025). In particular, Lin et al. (2024) resolved the discrepancy between neural scaling laws and traditional statistical learning theory by adopting an infinite-dimensional sketched linear regression framework with power-law spectra. Building on this, Lin et al. (2025) and Yan et al. (2025) extended one-pass SGD to multi-pass SGD, showing the benefit of increasing the multi-epoch count K. In a parallel avenue of research, Li et al. (2025) characterized how the learning rate schedule shapes scaling behaviors. These works have repeatedly demonstrated that such high-dimensional linear setups, despite their simplicity, can faithfully capture key phenomenological aspects of deep learning. Motivated by these successes, we initiate the theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression setup. Our setting. We assume access to M-dimensional sketched covariates and their responses, that is, (,y)(Sx,y), where ∈ℝM×ℍS ^M×H is a fixed sketch matrix, ∈ℍ⊂ℝpx ^p is the data vector and ℍH is a Hilbert space that is either finite-dimensional or countably infinite-dimensional. We focus on the Gaussian sketch matrix (Lin et al., 2024, 2025; Chen et al., 2025b; Ding et al., 2025). That is, entries of S are independently sampled from (0,1/M)N(0,1/M). We then consider linear model with M trainable parameters given by: f:ℍ→ℝ,→⟨,⟩,f_v:H , → ,Sx , where ∈ℝMv ^M are the trainable parameters. Our goal is to bound the population risk ℛM():=12[(⟨,⟩−y)2],∈ℝM,R_M(v):= 12E[( ,v -y)^2], ^M, where the expectation is conditioned on the sketch matrix S 111In this paper, all expectations are conditioned on S.. We consider training f_v via constant-stepsize one-pass quantized stochastic gradient descent (SGD) (Zhang et al., 2025). The parameter tv_t is updated as follows: t _t =t−1+γg(q)(q),t=1,…,N, =v_t-1+γ g^(q)f^(q), t=1,...,N, (quantized SGD) (q) ^(q) =f(s()d(t)), =Q_f (Q_s(S)Q_d(x_t) ), g(q) g^(q) =o(l(yt)−a(f(s()d(t))⊤p(t−1))), =Q_o(Q_l(y_t)-Q_a(Q_f (Q_s(S)Q_d(x_t) ) Q_p(v_t-1))), where (t,yt)t=1N(x_t,y_t)_t=1^N are independent samples and γ is the stepsize, and d,s,f,l,p,a,oQ_d,Q_s,Q_f,Q_l,Q_p,Q_a,Q_o are independent general quantization operations for data, sketch matrix, feature, labels, model parameters, activations and output gradients respectively, and (q)f^(q) is the quantized feature and g(q)g^(q) is the quantized output gradient. Without loss of generality, we assume the initial parameter is 0=0v_0=0. The output of the SGD algorithm is the the iterate average ¯N:=1N∑t=0N−1t v_N:= 1N _t=0^N-1v_t. Notations. For two positive-valued functions f(x)f(x) and g(x)g(x), we write f(x)≲g(x)f(x) g(x) or f(x)≳g(x)f(x) g(x) if f(x)≤cg(x)f(x)≤ cg(x) or f(x)≥cg(x)f(x)≥ cg(x) holds for some absolute (if not otherwise specified) constant c>0c>0 respectively. We write f(x)≂g(x)f(x) g(x) if f(x)≲g(x)≲f(x)f(x) g(x) f(x). For two vectors u and v in a Hilbert space, we denote their inner product by ⟨,⟩ ,v or ⊤u v. For two matrices A and B of appropriate dimensions, we define their inner product by ⟨,⟩:=tr(⊤) ,B :=tr (A B ). We use ∥⋅∥\|·\| to denote the operator norm for matrices. For a positive semi-definite (PSD) matrix A and a vector v of appropriate dimension, we write ‖2=⊤\|v\|_A^2=v Av. Our main results. Assuming that the spectrum of the data covariance matrix satisfies a power-law of degree a>1a>1, we analyze scaling laws under two standard quantization schemes: multiplicative quantization (where error variance scales with signal magnitude) and additive quantization (where error variance is independent of the signal). Informally, the population risk upper bound for both schemes can be unified as: ℛM(¯N)≲ℛ∗+1Meff(M,ϵ)a−1+1Neff(N,ϵ)a−1a+δ(ϵ),R_M( v_N) ^*+ 1M_ eff(M,ε)^a-1+ 1N_ eff(N,ε) a-1a+δ(ε), where M is the model size, N is the data size and ℛ∗R^* represents a positive irreducible risk, ϵε generically represents the quantization error (which vanishes in full-precision training) and δ(ϵ)δ(ε) denotes an additive error induced by ϵε. The key quantities MeffM_ eff and NeffN_ eff represent the effective model size and effective data size, respectively. We demonstrate a critical divergence in how the two quantization schemes affect these quantities. • Effective Data Size (NeffN_ eff): Both schemes reduce the effective data size via noise-amplification quantization error ϵnoise _ noise and spectral-distortion quantization error ϵspect _ spect. • Effective Model Size (MeffM_ eff): Multiplicative quantization (FP-like) preserves the full model capacity (i.e., Meff≈M_ eff≈ M), whereas additive quantization (INT-like) strictly contracts it driven by noise amplification and spectral distortion factors analogous to those reducing NeffN_ eff. We refer to Theorem 4.1 and Theorem 4.2 for formal statements of upper bounds. This theoretical dichotomy provides a rigorous basis for recent empirical findings in low-precision training. Specifically, our additive quantization scaling law captures the effective model shrinkage observed in integer quantization (Kumar et al., 2024), while our multiplicative quantization scaling law corroborates the observation that floating-point quantization preserves effective model capacity (Sun et al., 2025). Complementing the upper bounds, we establish the first population risk lower bounds for low-precision training (see Theorem 4.3 and Theorem 4.4 for details). These lower bounds validate the existence of the additive error and the reduction of effective data size, confirming that these mechanisms are fundamental in low-precision training. 2 Related Work Empirical scaling laws for quantized training. Recent research has focused on empirically characterizing the scaling behaviors of quantized training (Dettmers and Zettlemoyer, 2023; Ouyang et al., 2024; Kumar et al., 2024; Tao et al., 2024; Frantar et al., 2025; Chen et al., 2025a; Sun et al., 2025; Liu et al., 2025). One line of work conceptualizes quantization as a mechanism that effectively reduces model size (Kumar et al., 2024; Frantar et al., 2025). Notably, Kumar et al. (2024) proposed unified scaling laws under integer quantization covering low-precision training, quantization-aware training (QAT), and post-training quantization (PTQ). For low-precision training, they modeled the loss as L(M,N,P)≈AMeff(M,P)−α+BN−β+EL(M,N,P)≈ AM_eff(M,P)^-α+BN^-β+E, where Meff(M,P)≈M(1−e−P/γ)M_eff(M,P)≈ M(1-e^-P/γ) represents the effective model capacity contracted by low precision. Another stream of research models quantization as an additive error (Chen et al., 2025a; Sun et al., 2025). Sun et al. (2025) established scaling laws for low-precision training under floating-point (FP) formats, formulating the loss with a precision-dependent error term: L(M,N,P)≈AM−α+BN−β+E+δ(M,N,P)L(M,N,P)≈ AM^-α+BN^-β+E+δ(M,N,P). They showed that quantization induces a predictable deviation from the standard power law. In a parallel effort targeting integer QAT, Chen et al. (2025a) extended this framework to account for quantization granularity (G), modeling the loss via a similar additive penalty: L(M,N,P)≈AM−α+BN−β+E+δ(M,N,P,G)L(M,N,P)≈ AM^-α+BN^-β+E+δ(M,N,P,G). High-dimensional linear regression via SGD. Theoretical guarantees for generalization have garnered significant attention in machine learning. Seminal work by Bartlett et al. (2020); Tsigler and Bartlett (2023) established nearly tight upper and lower excess risk bounds for linear (ridge) regression under general regularization schemes. In the classical under-parameterized regime, extensive literature has explored the learnability of iterate-averaged SGD (Polyak and Juditsky, 1992; Bach and Moulines, 2013; Défossez and Bach, 2015; Dieuleveut et al., 2017; Jain et al., 2017, 2018). Conversely, in the modern overparameterized setting, one-pass SGD has been rigorously studied (Dieuleveut and Bach, 2015; Berthier et al., 2020; Varre et al., 2021; Zou et al., 2021; Wu et al., 2022a, b; Zhang et al., 2024), yielding frameworks to characterize how optimization dynamics influence generalization across various data distributions. Additionally, another line of work has analyzed multi-pass SGD for high-dimensional ℓ2 ^2-regularized least squares, detailing excess risk bounds (Lei et al., 2021; Zou et al., 2022) and exact risk dynamics (Paquette et al., 2024a). More recently, Zhang et al. (2025) established the first excess risk upper bounds for low-precision training, characterizing the impact of quantization on the learning dynamics of SGD in linear regression. Our work builds upon this foundation by extending their theoretical framework to sketched linear regression. Furthermore, we provide a critical missing piece by deriving the first excess risk lower bounds for low-precision training. Theoretical understandings of scaling laws. Several recent studies have sought to formalize and explain empirical scaling laws using conceptually simplified linear models (Bahri et al., 2024; Atanasov et al., 2024; Paquette et al., 2024b; Bordelon et al., 2024; Lin et al., 2024, 2025; Yan et al., 2025; Li et al., 2025; Ding et al., 2025). Early theoretical attempts focused on asymptotic regimes: Bahri et al. (2024) analyzed a linear teacher-student model with power-law spectra, showing that the test loss of the ordinary least squares (OLS) estimator decays as a power law in sample size N (or model size M) when the other dimension approaches infinity. Similarly, Bordelon et al. (2024) studied gradient flow in linear random feature models, establishing power-law scaling with respect to one of N, M, or training time T, provided the other parameters remain effectively infinite. A pivotal step towards realistic finite-sample analysis is made by Lin et al. (2024). Building on analysis techniques from Zou et al. (2021) and Wu et al. (2022a), they analyzed the last iterate of one-pass SGD in a sketched linear model and presented the first systematic derivation of a finite-sample joint scaling law (in both M and N) that aligns with empirical observations (Kaplan et al., 2020). Subsequent research expands this framework to more complex settings. Lin et al. (2025) extended the analysis to data reuse (multi-pass SGD), showing that for relatively small multi-epoch count K, every new epoch leads to a linear gain in effective sample size, i.e., Neff≂NKN_ eff NK. Building on this, Yan et al. (2025) provided a finer-grained characterization for strongly convex or Zipf-distributed data. They demonstrated that for large multi-epoch count K, the effective reuse rate Neff/NN_ eff/N plateaus at a problem-dependent value that grows with N. More recently, Li et al. (2025) established functional scaling laws and analyzed how learning rate schedules shape these scaling behaviors. 3 Theoretical Setup 3.1 Quantization Operation For all quantization operations in (quantized SGD), we employ the stochastic quantization method (Markov et al., 2023; Modoranu et al., 2024; Ozkara et al., 2025), which unbiasedly rounds values using randomly adjusted probabilities. We summarize this in the following assumption. Assumption 3.1. Let i,i∈d,s,f,l,p,a,oQ_i,i∈\d,s,f,l,p,a,o\ be the coordinate-wise quantization operation for data, sketch matrix, feature, label, model parameters, activations, and output gradients, respectively. Then for any u, the quantization operation is unbiased: [i()|]=.E [ Q_i(u)|u ]=u. Furthermore, to better uncover the effect of quantization, we consider the following two types of quantization error: multiplicative quantization and additive quantization, which are motivated by abstracting the behavior of prevalent numerical formats used in practice (Zhang et al., 2025). Definition 3.1. Let Q be an unbiased quantization operation. We formalize two practical quantization schemes: • Multiplicative quantization. We call the quantization to x is (ϵ¯,ϵ¯ ε, ε)-multiplicative if the conditional second moment of quantization error is proportional to the outer product of raw data itself, i.e., ϵ¯⊤⪯[(()−)(()−)⊤|]⪯ϵ¯⊤. εxx [ (Q(x)-x ) (Q(x)-x ) |x] εxx . For multiplicative quantization to matrix X, we extend the definition to ϵ¯⊤⪯[(()−)(()−)⊤|]⪯ϵ¯⊤, εXAX [ (Q(X)-X )A (Q(X)-X ) |X] εXAX , for any PSD matrix A. • Additive quantization. We call the quantization to x is (ϵ¯,ϵ¯ ε, ε)-additive if the conditional second moment of quantization error is proportional to identity, i.e., ϵ¯⪯[(()−)(()−)⊤|]⪯ϵ¯,for any PSD matrix. εI [ (Q(x)-x ) (Q(x)-x ) |x] εI,\ for any PSD matrix\ A. For additive quantization to matrix X, we extend the definition to ϵ¯tr()⪯[(()−)(()−)⊤|]⪯ϵ¯tr(),for any PSD matrix. εtr(A)I [ (Q(X)-X )A (Q(X)-X ) |X] εtr(A)I,\ for any PSD matrix\ A. This theoretical distinction is grounded in practical quantization schemes. For instance, integer quantization (e.g., INT8, INT16) uses a fixed bin length, resulting in an error that is largely independent of the value’s magnitude (Wu et al., 2020). This characteristic aligns with our definition of additive quantization, where the error variance is uniform across coordinates. Conversely, floating-point quantization (e.g., FP8, FP32) employs a value-aware bin length via its exponent and mantissa bits (e.g., E4M3 format in FP8) (Kuzmin et al., 2022). This structure causes the quantization error to scale with the magnitude of the value itself, corresponding to multiplicative quantization. 3.2 Data Model We then state the regularity assumptions on the data distribution, which align with those common in prior works (Zou et al., 2021; Wu et al., 2022a, b, 2023). As low-precision training is performed on quantized feature ~(q)=f(s()d()) x^(q)=Q_f (Q_s(S)Q_d(x) ), we formulate these assumptions on the low-precision feature format following Zhang et al. (2025). Assumption 3.2 (Data covariance). Let :=[⊤]H:=E[xx ] be the data covariance and f(q):=[~(q)(~(q))⊤]H_f^(q):=E[ x^(q)( x^(q)) ] be the quantized feature covariance. Assume that tr(),tr(fq)tr(H),tr(H_f^q) and all entries of ,f(q)H,H_f^(q) are finite. For convenience, we assume that H is strictly positive definite. Let =∑iλiii⊤H= _i _iv_iv_i be the eigen-decomposition of H, where λii=1∞\ _i\_i=1^∞ are the eigenvalues of H sorted in non-increasing order and iv_i are the corresponding eigenvectors. We denote 0:k:=∑i=1kλiii⊤,k:∞:=∑i>kλiii⊤,0:k:=∑i=1kii⊤,k:∞:=∑i>kii⊤.H_0:k:= _i=1^k _iv_iv_i , _k:∞:= _i>k _iv_iv_i , _0:k:= _i=1^kv_iv_i , _k:∞:= _i>kv_iv_i . Similarly, we denote the eigen-decomposition of f(q)H_f^(q) as f(q)=∑iλ~i(q)i(q)i(q)⊤H_f^(q)= _i λ_i^(q)v_i^(q)v_i^(q) and correspondingly obtain f,0:k(q),f,k:∞(q),f,0:k(q),f,k:∞(q)H_f,0:k^(q),H_f,k:∞^(q),I_f,0:k^(q),I_f,k:∞^(q), where λ~i(q)i=1∞\ λ_i^(q)\_i=1^∞ are the eigenvalues of f(q)H_f^(q). In line with Zhang et al. (2025), we extend the fourth moment and noise assumptions (Zou et al., 2021; Wu et al., 2022b, a, 2023) to quantized features. Assumption 3.3 (Fourth-moment conditions). Assume that the fourth moment of ~(q) x^(q) is finite and there exist constants α,β>0α,β>0 such that for any PSD matrix A, f(q)f(q)+βtr(f(q))f(q)⪯[~(q)(~(q))⊤~(q)(~(q))⊤]⪯αtr(f(q))f(q).H_f^(q)AH_f^(q)+ (H_f^(q)A)H_f^(q) [ x^(q)( x^(q)) A x^(q)( x^(q)) ] (H_f^(q)A)H_f^(q). Regarding the noise assumptions, we first define the population risk and global optimum in quantized feature space: ℛM(q)():=12[(⟨~(q),⟩−l(y))2],∈ℝM,R_M^(q)(v):= 12E[( x^(q),v -Q_l(y))^2], ^M, with global optimum (q)∗:=argminℛM(q)()v^(q)^*:= argmin_vR_M^(q)(v). Assumption 3.4 (Noise conditions). Denote ξ:=l(y)−⟨(q)∗,~(q)⟩ξ:=Q_l(y)- v^(q)^*, x^(q) . Assume there exists a positive constants σ¯,σ¯>0 σ, σ>0 such that σ¯2f(q)⪯[ξ2~(q)(~(q))⊤]⪯σ¯2f(q). σ^2H_f^(q) [ξ^2 x^(q)( x^(q)) ] σ^2H_f^(q). A key distinction from the assumptions in Zhang et al. (2025) is that we require lower bounds on the noise and fourth moment to establish both upper and lower risk bounds. We would like to remark that under the fourth moment assumption and noise assumption on the full-precision data, Assumption 3.3 and 3.4 can be verified under specific multiplicative quantization and additive quantization schemes. We defer the verification in Section H. To simplify the scaling-law behavior, we assume specific data distribution where the data spectrum satisfies a power law and the optimal parameter satisfies a prior (Lin et al., 2024). Specifically, we consider the population risk for ∈ℍw : ℛ():=12[(⟨,⟩−y)2],∈ℍ,R(w):= 12E[( ,w -y)^2], , with global optimum ∗:=argminℛ()w^*:= argmin_wR(w). Assumption 3.5 (Distributional conditions). We assume the well-specified model, i.e., [y|]=⊤∗E[y|x]=x w^* and σ2:=[(y−⊤∗)2]σ^2:=E [(y-x w^*)^2 ], and the parameter prior, i.e., [∗∗⊤]=E[w^*w^* ]=I. We also assume the data spectrum is polynomial, i.e., there exists a>1a>1 such that the eigenvalues of H satisfy λi≂i−a,i>0 _i i^-a,\ i>0. 4 Main Theory In this section, we demonstrate low-precision training scaling laws when the data spectrum satisfies a power law. We state the scaling laws for multiplicative quantization and additive quantization respectively. 4.1 Multiplicative Quantization In this section, we consider for any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, there exist ϵ¯i ε_i such that quantization iQ_i is ϵ¯i ε_i-multiplicative 222This means we only access to the upper bound of quantization errors defined in Definition 3.1.. Motivated by the insight from Zhang et al. (2025) that different quantization targets exert distinct influences on the risk, we first define a set of compound quantization coefficients to aggregate individual quantization errors based on their distinct physical effects on the learning dynamics. This formulation streamlines the presentation and elucidates the structural impact of quantization. Firstly, to capture the distortion to feature spectrum and the gap between quantized feature space and full-precision data space, we define ϵ¯3(M)=1−1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s), ε_3^(M)=1- 1(1+ ε_d)(1+ ε_f)(1+ ε_s), which arises from feature, sketch and data quantization. Secondly, to characterize the noise amplification during training, we define ϵ¯2(M)=(1+ϵ¯o)(1+ϵ¯p+(1+ϵ¯p)ϵ¯a)−1, ε_2^(M)=(1+ ε_o) (1+ ε_p+(1+ ε_p) ε_a )-1, which arises from parameter, activation and output gradient quantization. Generally, the compound coefficients ϵ¯3(M) ε_3^(M) and ϵ¯2(M) ε_2^(M) scale monotonically with the underlying quantization severity. In standard training regimes where the individual quantization errors (e.g., ϵ¯o,ϵ¯p,ϵ¯d ε_o, ε_p, ε_d) are small (<1<1), these coefficients remain small quantities of comparable magnitude. In particular, ϵ¯3(M) ε_3^(M) is strictly less than 11. Notably, they vanish strictly to zero in the full-precision limit. With these notations, we are now ready to state the main scaling laws under multiplicative quantization. Theorem 4.1 (Scaling law under multiplicative quantization, an upper bound). Suppose γ<1(1+ϵ¯2(M))αtr(f(q))γ< 1(1+ ε_2^(M)) (H_f^(q)). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist ϵ¯i ε_i such that quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, if f(q)H_f^(q) and ⊤SHS commute, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ℛM(¯N)≲1Meffa−1+1Neff(a−1)/a+σ2+ϵ¯3(M), _M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ε_3^(M), (1) where Meff=M_ eff=M and Neff=N[1+ϵ¯2(M)(1−ϵ¯3(M))1a]−a/(a−1).N_ eff=N [ 1+ ε_2^(M)(1- ε_3^(M)) 1a ]^-a/(a-1). Theorem 4.1 rigorously quantifies the dual impact of multiplicative quantization: the reduction of effective data size and the introduction of an additive error. Specifically, the reduction in effective data size NeffN_ eff stems from two mechanisms: the amplification of optimization noise due to quantized parameters, gradients and activations (captured by ϵ¯2(M) ε_2^(M)), and the distortion of the feature spectrum (captured by ϵ¯3(M) ε_3^(M)). Meanwhile, the additive error term arises from the gap between the quantized feature space and the full-precision data space (captured by ϵ¯3(M) ε_3^(M)). These mechanisms align with the findings of how quantization affects learnability in Zhang et al. (2025). Notably, in the absence of quantization (ϵ¯i(M)=0 ε_i^(M)=0), Theorem 4.1 recovers the classical full-precision scaling law established in Lin et al. (2024). A critical insight from Theorem 4.1 is that multiplicative quantization does not reduce the effective model size, which aligns with some empirical studies (Chen et al., 2025a; Sun et al., 2025). Intuitively, this invariance arises from the signal-dependent nature of multiplicative quantization, which preserves the spectral structure of the quantized feature covariance. Specifically, since the quantization error scales with the signal magnitude, it decays alongside the signal in the high-dimensional tail subspace. This ensures that the tail subspace of quantized feature spectrum decays as that of the full-precision spectrum (up to a constant scalar), thereby preserving the learnability of each parameter. Consequently, multiplicative quantization maintains Meff=M_eff=M. Our Theorem 4.1 assumes commutativity between the quantized feature covariance f(q)H_f^(q) and the sketched covariance ⊤SHS to derive a sharper bound. We would also like to remark that, without this commutative condition, the quantization error may project non-trivially onto sensitive eigen-directions. While an upper bound can still be derived in general case (see Theorem C.3 for details), this misalignment introduces an additional penalty related to the condition number of ⊤SHS . To isolate the fundamental scaling behavior, we apply the commutativity assumption here. 4.2 Additive Quantization In this section, we consider for any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, there exist ϵ¯i ε_i such that quantization iQ_i is ϵ¯i ε_i-additive. Analogous to the multiplicative case, we define a set of compound quantization coefficients to streamline the presentation. Regarding the discrepancy between the quantized feature covariance and the original data covariance, we define: ϵ¯3(A)=ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpMM−a+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM). ε_3^(A)= ε_f+ ε_s(1+ ε_dp)+ ε_d pMM^-a+ ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ). Regarding the noise amplification, we define ϵ¯2(A)=ϵ¯a+ϵ¯o+ϵ¯p[1+pϵ¯d+M(ϵ¯f+ϵ¯s+ϵ¯sϵ¯dp)]. ε_2^(A)= ε_a+ ε_o+ ε_p [1+p ε_d+M( ε_f+ ε_s+ ε_s ε_dp) ]. Similar to the multiplicative case, these coefficients are small quantities that scale monotonically with the quantization severity and vanish strictly in the full-precision limit. However, we note that, unlike multiplicative coefficients which are largely dimension-independent, ϵ¯2(A) ε_2^(A) and ϵ¯3(A) ε_3^(A) scale with the data dimension p and model size M. This distinction arises because additive quantization introduces constant quantization variance that is independent across all coordinates. Moreover, since the additive quantization error constitutes a fixed floor rather than scaling with the signal, ϵ¯3(A) ε_3^(A) must explicitly account for its magnitude relative to the minimum eigenvalues of the data spectrum (M−aM^-a). With these notations, we now present the main scaling laws for low-precision training under additive quantization. Theorem 4.2 (Scaling law under additive quantization, an upper bound). Suppose γ<1αtr(f(q))γ< 1 (H_f^(q)). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist ϵ¯i ε_i such that quantization iQ_i is ϵ¯i ε_i-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, if f(q)H_f^(q) and ⊤SHS commute, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ℛM(¯N)≲ _M( v_N) 1Meffa−1+1Neff(a−1)/a+σ2+ϵ¯3(A), 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ε_3^(A), (2) where Neff=N[1+ϵ¯2(A)(1−ϵ¯3(A))1/a]−a−1N_ eff=N [ 1+ ε_2^(A)(1- ε_3^(A))^1/a ]^- aa-1, and crucially, Meff=M[1+(1+ϵ¯2(A))(ϵ¯3(A))21−ϵ¯3(A)]−1/(a−1). M_ eff=M [1+(1+ ε_2^(A)) ( ε_3^(A))^21- ε_3^(A) ]^-1/(a-1). Theorem 4.2 characterizes a fundamental dichotomy between additive and multiplicative quantization. Unlike the multiplicative case, additive quantization not only introduces an additive error floor and reduces the effective data size, but also reduces the effective model size. The interpretation is that additive quantization injects an constant level quantization error across the entire spectrum of the quantized feature covariance f(q)H_f^(q). Consequently, this constant error overwhelms the intrinsic signal in the spectral tail and results in a flattened spectrum, rendering the tail dimensions useless for learning. Hence, the model cannot effectively leverage its full parameter count, leading to a reduction in MeffM_ eff. Our analysis further reveals that the degradation of effective model size (MeffM_ eff) and effective data size (NeffN_ eff) is governed by similar physical mechanisms. As derived in Theorem 4.2, both effective data size and effective model size are modulated by the same noise amplification factor ϵ¯2(A) ε_2^(A) and spectral distortion factor ϵ¯3(A) ε_3^(A). This mechanisms align with Theorem 4.1 under multiplicative quantization and prior work (Zhang et al., 2025). Similar to multiplicative case, under full precision, Theorem 4.2 recovers the result in Lin et al. (2024) and the commutativity condition is assumed here to isolate the fundamental scaling behavior. A general bound relaxing this assumption is in Theorem C.4. Connection with empirical scaling laws for low-precision training. Our theoretical distinction between additive and multiplicative quantization provides a mechanistic explanation for the divergent empirical behaviors observed in integer versus floating-point training. Firstly, the empirical observation in Kumar et al. (2024) that integer quantization effectively reduces model capacity aligns with our additive quantization (INT-like) scaling law (Theorem 4.2). Our theory further reveals the mechanism: a constant level quantization error flattens the tail subspace, effectively rendering those dimensions uninformative and leading to the theoretically derived reduction in MeffM_ eff. In contrast, Sun et al. (2025) found that floating-point quantization primarily introduces an additive loss term rather than shrinking the model size. This corroborates our multiplicative quantization (FP-like) scaling law (Theorem 4.1), which establishes that the effective model size remains invariant (Meff=M_ eff=M). The underlying mechanism is that multiplicative quantization preserves the relative spectral structure, ensuring the quantization error in the tail subspace scales down with the signal. 4.3 Lower Bound Analysis To tighten our analysis, we establish scaling law lower bounds under multiplicative and additive quantization. In lower bound analysis, we consider low-precision well-specific model: [ξ|~(q)]=0E [ξ| x^(q) ]=0, which is extended from the standard full-precision well-specific model assumption (Zou et al., 2021; Wu et al., 2022a, b). 4.3.1 Multiplicative Quantization We extend the compound coefficients defined in Section 4.1 to their lower-bound counterparts. The definitions utilize the minimum quantization errors ϵ¯ ε. For simplicity, we provide explicit definitions for ϵ¯2(M),ϵ¯3(M) ε_2^(M), ε_3^(M) in Section E.1. Theorem 4.3 (Scaling law under multiplicative quantization, a lower bound). Suppose γ<1/λ~1(q)γ<1/ λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for sufficiently large N>500N>500, if f(q)H_f^(q) and ⊤SHS are commutative, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds ℛM(¯N)≳1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(M))2+ϵ¯3(M)N(1−ϵ¯3(M)), _M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(M) )^2+ ε_3^(M)N (1- ε_3^(M) ), (3) where Meff=M_ eff=M and Neff=N[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))1a]−a−1N_ eff=N [ (1- ε_3^(M))(1+ ε_2^(M))(1- ε_3^(M)) 1a ]^- aa-1. Theorem 4.3 matches the form of scaling law derived in the upper bound: multiplicative quantization inherently reduces the effective data size NeffN_ eff via noise amplification (ϵ¯2(M) ε_2^(M)) and spectral distortion (ϵ¯3(M) ε_3^(M)), while introducing an unavoidable additive error via the gap between quantized feature space and full-precision data space (ϵ¯3(M) ε_3^(M)). Generally, the lower bound for the effective data size NeffN_ eff in Theorem 4.3 does not strictly match the upper bound. This discrepancy stems from the gap between the worst-case (ϵ¯ ε) and best-case (ϵ¯ ε) quantization errors. Matching bounds are achieved in the sharp quantization limit where ϵ¯≈ϵ¯ ε≈ ε 333When ϵ¯≈ϵ¯ ε≈ ε, our lower bound for NeffN_ eff matches the refined upper bound established in Theorem E.1 (which incorporates the lower quantization limit ϵ¯ ε compared with Theorem 4.1).. We note that this clean scaling law form holds in two asymptotic regimes where the interplay between M and N is well-separated, effectively rendering the ratio term N/MN/M of strict higher order. See Theorem E.2 for the explicit definition of these regimes. For completeness, a general population risk lower bound covering the full space of (M,N)(M,N) is provided in Theorem D.3 in the Appendix. 4.3.2 Additive Quantization Analogous to the multiplicative case, we establish the lower bound for additive quantization by extending the compound coefficients to their lower-bound counterparts. For simplicity, we defer the definitions of ϵ¯2(A),ϵ¯3(A) ε_2^(A), ε_3^(A) to Section E.2. Theorem 4.4 (Scaling law under additive quantization, a lower bound). Suppose γ<1λ~1(q)γ< 1 λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for sufficiently large N>500N>500, if f(q)H_f^(q) and ⊤SHS are commutative, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds ℛM(¯N)≳1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(A))2+ϵ¯3(A)N(1−ϵ¯3(A)), _M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(A) )^2+ ε_3^(A)N (1- ε_3^(A) ), (4) where Meff=M,Neff=N[(1−ϵ¯3(A))(1+ϵ¯2(A))[1−Nγ(11−ϵ¯3(A)−1)]1a]−a−1.M_ eff=M,\ N_ eff=N [ (1- ε_3^(A) ) (1+ ε_2^(A) )[1-Nγ( 11- ε_3^(A)-1)] 1a ]^- aa-1. Theorem 4.4 rigorously validates the existence of the additive error floor (induced by the gap between quantized and low-precision space) and the reduction of effective data size (induced by noise amplification and spectral distortion), confirming the theoretical findings in upper bound analysis. Similar to Theorem 4.3, the clean scaling law in Theorem 4.4 holds in specific regimes under the condition that 1Nγ≥11−ϵ¯3(A)−1 1Nγ≥ 11- ε_3^(A)-1. See Theorem E.4 for the explicit definition of these regimes. For completeness, a general population risk lower bound covering the full space of (M,N)(M,N) is established in Theorem D.4 in the Appendix. We acknowledge that, unlike the upper bound, our lower bound does not explicitly exhibit the reduction in effective model size. This is a technical limitation rather than a physical one: while additive quantization error theoretically flattens the tail subspace, the induced error term becomes intricately coupled with M and N in the lower bound analysis (see the proof for Theorem E.4 for details). Decoupling this interaction to derive a clean scaling form that explicitly separates the shrinkage of MeffM_ eff remains a non-trivial challenge, which we defer to future work. Figure 1: Scaling of excess risk [ℛ]−12σ2E[R]- 12σ^2 under multiplicative quantization with ϵ=10−3ε=10^-3, γ=0.1γ=0.1, σ=1σ=1. (a), (b): a=1.5a=1.5, p=10,000p=10,000; (c), (d): a=2.0a=2.0, p=1,000p=1,000. Panels (a), (c) fix MeffM_eff and vary NeffN_eff; panels (b), (d) fix NeffN_eff and vary MeffM_eff. Fitted exponents (orange curves) match theoretical predictions: α=−(a−1)α=-(a-1) and β=−(a−1)/aβ=-(a-1)/a. All fits achieve R2>0.99R^2>0.99. Figure 2: Scaling of excess risk [ℛ]−12σ2E[R]- 12σ^2 under additive quantization with ϵ=10−8ε=10^-8, γ=0.1γ=0.1, σ=1σ=1. (a), (b): a=1.5a=1.5, p=10,000p=10,000; (c), (d): a=2.0a=2.0, p=1,000p=1,000. Panels (a), (c) fix MeffM_eff and vary NeffN_eff; panels (b), (d) fix NeffN_eff and vary MeffM_eff. Fitted exponents (orange curves) match theoretical predictions: α=−(a−1)α=-(a-1) and β=−(a−1)/aβ=-(a-1)/a. All fits achieve R2>0.99R^2>0.99. Experiments. We generate data with polynomial spectral decay λi∝i−a _i i^-a for a∈1.5,2.0a∈\1.5,2.0\, with dimension p=10,000p=10,000 for a=1.5a=1.5 and p=1,000p=1,000 for a=2.0a=2.0. Models are trained via one-pass SGD with iterate averaging under multiplicative quantization (ϵ=10−3ε=10^-3) and additive quantization (ϵ=10−8ε=10^-8). We fit the excess risk [ℛM]−12σ2=A⋅Meffα+B⋅Neffβ+CE[R_M]- 12σ^2=A· M_eff^α+B· N_eff^β+C. To isolate each scaling dimension, we conduct two sweeps: (i) fixing Meff=2,000M_eff=2,000 while varying Neff∈[102,105]N_eff∈[10^2,10^5] across 10 log-spaced values, and (i) fixing Neff=20,000N_eff=20,000 while varying Meff∈[10,200]M_eff∈[10,200] across 10 log-spaced values. Each configuration is averaged over 20 seeds. Figures 1 and 2 show results. Across all configurations, the fitted exponents match theoretical predictions: for a=1.5a=1.5, we obtain α=−0.50α=-0.50 (theory: −12- 12) and β=−0.34β=-0.34 (theory: −13- 13); for a=2.0a=2.0, we obtain α=−1.01α=-1.01 (theory: −1-1) and β=−0.50β=-0.50 (theory: −12- 12). All fits achieve R2>0.99R^2>0.99, confirming the scaling laws ℛ∼Neff−(a−1)/aR N_eff^-(a-1)/a and ℛ∼Meff−(a−1)R M_eff^-(a-1). These empirical results align with our theoretical scaling laws for low-precision training. 5 Proof Overview In this section, we outline the proof strategy for the theoretical results established in Section 4. Moreover, we point out some key technical challenges and our strategy to address them. A proof roadmap. Following Lin et al. (2024), we begin by decomposing the population risk into three components: irreducible risk, approximation error, and excess risk: ℛM(¯N)=minℛ(⋅)⏟Irreducible+minℛM(⋅)−minℛ(⋅)⏟Approx+ℛM(¯N)−minℛM(⋅)⏟Excess. _M( v_N)= (·)_ Irreducible+ _M(·)- (·)_ Approx+ R_M( v_N)- _M(·)_ Excess. Since the quantized SGD algorithm (quantized SGD) operates within the quantized feature space rather than the exact sketch space, we further decompose the excess risk term into an algorithm-dependent excess risk and an algorithm-independent additive error, adopting the framework of Zhang et al. (2025) (see Lemma B.2 for details): Excess=12⟨⊤,[((q)∗−¯N)((q)∗−¯N)⊤]⟩⏟RN+AdditiveError. = 12 ,E[(v^(q)^*- v_N)(v^(q)^*- v_N) ] _R_N+AdditiveError. Consequently, our primary theoretical task reduces to deriving bounds for the algorithm-dependent risk RNR_N. The analysis proceeds in two logical stages. Step 1: Excess risk bounds under general spectrum. Firstly, we analyze the dynamics of the error covariance [tt⊤]E[ η_t η_t ] (where t=t−(q)∗ η_t=v_t-v^(q)^* denotes the centered SGD iterate) to establish risk bounds under general spectral conditions. Conditioning on the sketch matrix S, the training of the sketched linear predictor can be viewed as an M-dimensional linear regression problem. We can therefore invoke existing quantized SGD analysis (Zhang et al., 2025) to control RNR_N via bias and variance. Specifically, let us define for k∗=maxi:λ~i(q)≥1Nγk^*= \i: λ_i^(q)≥ 1Nγ\: Var=k∗N+Nγ2⋅∑i>k∗(λ~i(q))2,Bias=1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2. = k^*N+Nγ^2· _i>k^*( λ_i^(q))^2, = 1γ^2N^2· \|v^(q)^* \|_(H_f,0:k^*^(q))^-1^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2. Further, let μ¯=μmax((f(q))−1⊤) μ= _ max ((H_f^(q))^-1SHS ) and μ¯=μmin((f(q))−1⊤) μ= _ min ((H_f^(q))^-1SHS ) denote the maximum and minimum spectral alignment coefficients, respectively. To capture the impact of quantization noise, we introduce the quantization errors: the quantization error of activation ϵat=[(a(at)−at)2|at],at=(~t(q))⊤p(t−1), _a_t=E [(Q_a(a_t)-a_t)^2 |a_t ],\ a_t=( x_t^(q)) Q_p(v_t-1), the quantization error of output gradient ϵot=[(o(ot)−ot)2|ot],ot=l(yt)−a(at), _o_t=E [(Q_o(o_t)-o_t)^2 |o_t ],\ o_t=Q_l(y_t)-Q_a(a_t), and the quantization error of parameter t−1=[ϵt−1(p)ϵt−1(p)⊤|t−1],ϵt−1(p)=p(t−1)−t−1.E_t-1=E [ ε_t-1^(p) ε_t-1^(p) |v_t-1 ],\ ε_t-1^(p)=Q_p(v_t-1)-v_t-1. We summarize the resulting risk bounds under a general spectrum in the following lemma, which consolidates Theorems C.1, C.2, D.1, and D.2. Lemma 5.1 (Excess risk bounds under general quantization). Under Assumption 3.1, 3.2, 3.3 and 3.4, • suppose γ<1/(αtr(f(q)))γ<1/ ( (H_f^(q)) ), RN/μ¯≲Bias+(σ¯eff2+ασbias2)Var, R_N/ μ +( σ_ eff^2+α _ bias^2)Var, • suppose the stepsize γ<1/λ~1(q)γ<1/ λ_1^(q), for N>500N>500, RN/μ¯≳Bias+(σ¯eff2+βσbias2)Var. R_N/ μ +( σ_ eff^2+β _ bias^2)Var. Here σ¯eff2=σ¯2+suptαtr(f(q)t−1)+suptϵat+ϵot σ_ eff^2= σ^2+ _t \ (H_f^(q)E_t-1 ) \+ _t \ _a_t+ _o_t \, σbias2=1Nγ‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 _ bias^2= 1Nγ \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 and σ¯eff2=σ¯2+inftβtr(f(q)t−1)+inftϵat+ϵot σ_ eff^2= σ^2+ _t \ (H_f^(q)E_t-1 ) \+ _t \ _a_t+ _o_t \. For the specific case of multiplicative quantization, we establish nearly matching lower bounds under appropriate conditions. Note that for ease of presentation, we slightly abuse the notation for σ¯eff σ_ eff and σ¯eff σ_ eff below. Lemma 5.2 (Excess risk bounds under multiplicative quantization). Under Assumption 3.1, 3.2, 3.3 and 3.4, for any i∈p,a,o,d,f,si∈\p,a,o,d,f,s\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, • suppose γ<1/((1+ϵ¯2(M))αtr(f(q)))γ<1/ ((1+ ε_2^(M)) (H_f^(q)) ), RN≲Bias+(1+ϵ¯o)(ασ¯eff2+σ¯2)Var(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s), R_N Bias+(1+ ε_o)(α σ_ eff^2+ σ^2)Var(1+ ε_d)(1+ ε_f)(1+ ε_s), • suppose the stepsize γ<1/λ~1(q)γ<1/ λ_1^(q), for N>500N>500, RN≳Bias+(1+ϵ¯o)(βσ¯eff2+σ¯2)Var(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). R_N Bias+(1+ ε_o)(β σ_ eff^2+ σ^2)Var(1+ ε_d)(1+ ε_f)(1+ ε_s). Here ∗σ¯eff2≲1+ϵ¯p+(1+ϵ¯p)ϵ¯a(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)E_w^* σ_ eff^2 1+ ε_p+(1+ ε_p) ε_a(1+ ε_d)(1+ ε_s)(1+ ε_f) and ∗σ¯eff2≳ϵ¯p+(1+ϵ¯p)ϵ¯a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)E_w^* σ_ eff^2 ε_p+(1+ ε_p) ε_a(1+ ε_d)(1+ ε_f)(1+ ε_s) under Assumption 3.5. We highlight that under Assumption 3.5, if the intrinsic noise variance satisfies σ¯2≂σ¯2≂1 σ^2 σ^2 1 and the quantization is sharp (i.e., ϵ¯i=ϵ¯i ε_i= ε_i), the upper bound matches the lower bound up to absolute constants. This indicates that our analysis is tight. Step 2: Excess risk bounds under polynomial spectrum. Secondly, we instantiate these general bounds under the polynomial spectrum assumption to explicitly derive the final scaling laws. Specifically, as established in Lemma C.19, Lemma C.20, Lemma C.24, Lemma C.26, Lemma D.16, Lemma D.17, Lemma D.19 and Lemma D.20, we summarize the analysis of the BiasBias and VarVar terms under Assumption 3.5 below. Lemma 5.3 (Bounds under polynomial spectrum, multiplicative quantization). Under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for any i∈p,a,o,d,f,si∈\p,a,o,d,f,s\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1aN≲Var≲minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/aN, \M, [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s) ] 1a \N \M,[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a \N, and Bias≲max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]1a−1,M1−a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s),iff(q)and⊤commute, \ [Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s) ] 1a-1,M^1-a \(1+ ε_d)(1+ ε_f)(1+ ε_s), \ H_f^(q)\ and\ SHS \ commute, Bias≳[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1a−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s),if[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1a≤MCfor someC>0. [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)] 1a-1(1+ ε_f)(1+ ε_d)(1+ ε_s), \ [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)] 1a≤ MC\ for\ some\ C>0. Lemma 5.4 (Bounds under polynomial spectrum, additive quantization). Under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for any i∈p,a,o,d,f,si∈\p,a,o,d,f,s\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds Var≲keff+γ2N2(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−keff)N, k_ eff+γ^2N^2 ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2(M-k_ eff)N, Var≳k¯eff+γ2N2(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2(M−k¯eff)N, k_ eff+γ^2N^2 ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2(M- k_ eff)N, where keff−a=M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)k_ eff^-a=M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ), k¯eff−a=M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM) k_ eff^-a=M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ), Bias≲max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]1a−1,M1−a1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM,iff(q)and⊤commute, \ [Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ] 1a-1,M^1-a \1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM, \ H_f^(q)\ and\ SHS \ commute, and if M−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM≤CNγM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM≤ CNγ for some constant C>0C>0, Bias≳M−aM−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM(1Nγ−[ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM])1−1/a. M^-aM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM ( 1Nγ- [ ε_f+(1+ ε_dp) ε_s+ ε_d pM ] )^1-1/a. Together with Lemma 5.1, Lemma 5.2, Lemma 5.3, Lemma 5.4 and analysis of algorithm-independent additive error, approximation error and irreducible risk, we can derive final scaling laws for low-precision training. We then point out some key technical challenges in these two steps and present some high-level ideas to address them. Challenge I: Lower bound analysis for multiplicative quantization. Multiplicative quantization introduces noise variance proportional to the signal magnitude, creating a complex feedback loop where the error covariance [t⊗t]E[ η_t η_t] evolves with the iterate tv_t. To see this, we first rewrite (quantized SGD) using the quantization errors 444These quantization errors are defined as the difference of parameter, activation, output gradient and their quantized counterpart, respectively, e.g., ϵt(p)=p(t)−t ε_t^(p)=Q_p(v_t)-v_t. (see Lemma C.1 for details): t=(−γ~t(q)(~t(q))⊤)t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q). η_t=(I-γ x_t^(q)( x_t^(q)) ) η_t-1+γ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p)) x_t^(q). Then in the subsequent analysis of [t⊗t]E[ η_t η_t], the second moment of parameter quantization error [ϵt−1(p)⊗ϵt−1(p)]E[ ε_t-1^(p) ε_t-1^(p)], activation quantization error [ϵt(a)⊗ϵt(a)]E[ ε_t^(a) ε_t^(a)] and output gradient quantization error [ϵt(o)⊗ϵt(o)]E[ ε_t^(o) ε_t^(o)] are all related to the magnitude of signal [t−1⊗t−1]E[v_t-1 _t-1]. While Zhang et al. (2025) successfully derived upper bounds by relaxing the quadratic forms (decoupling [ϵt−1(p)⊗ϵt−1(p)]≈ϵp[t−1⊗t−1]E[ ε_t-1^(p) ε_t-1^(p)]≈ε_pE[v_t-1 _t-1] into an iterate-dependent term [t−1⊗t−1]E[ η_t-1 η_t-1] and a constant term (q)∗⊗(q)∗v^(q)^* v^(q)^*), this approach is insufficient for lower bounds. The critical difficulty is the indefiniteness of the cross-term [t−1⊤(q)∗]E[ η_t-1 v^(q)^*]. This negative component could theoretically cancel out the positive constant contribution (q)∗⊗(q)∗v^(q)^* v^(q)^*, thereby precluding the derivation of a strictly positive noise using standard techniques. Our strategy. Intuitively, the iterate’s second moment [t−1⊗t−1]E[v_t-1 _t-1] is always positive semi-definite and generally evolves from zero initialization towards the optimal covariance (q)∗⊗(q)∗v^(q)^* v^(q)^*. Therefore, instead of roughly decoupling the second moment, we refine the analysis by establishing a crude lower bound for [t−1⊗t−1]E[v_t-1 _t-1]. Specifically, we achieve this by deriving a crude lower bound for [t⊗t]E[ η_t η_t] through the crude update rule: [tt⊤]⪰(−γ~t(q)(~t(q))⊤)[t−1t−1⊤](−γ~t(q)(~t(q))⊤).E[ η_t η_t ] (I-γ x_t^(q)( x_t^(q)) )E[ η_t-1 η_t-1 ](I-γ x_t^(q)( x_t^(q)) ). It follows from Assumption 3.3 that [tt⊤]⪰γβ2(−γf(q))2tf(q)‖(q)∗‖−(−γf(q))2t2+(−γf(q))t(q)∗((q)∗)⊤(−γf(q))t.E[ η_t η_t ] γβ2(I- _f^(q))^2tH_f^(q) \|v^(q)^* \|_I-(I- _f^(q))^2t^2+ (I- _f^(q) )^tv^(q)^* (v^(q)^* ) (I- _f^(q) )^t. Therefore, together with [t]=−(−γf(q))t(q)∗E [ η_t ]=- (I- _f^(q) )^tv^(q)^*, we obtain the crude lower bound: [(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] = = [(~t(q))⊤[(t−1+(q)∗)(t−1+(q)∗)⊤]~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) E [ ( η_t-1+v^(q)^* ) ( η_t-1+v^(q)^* ) ] x_t^(q) x_t^(q)( x_t^(q)) ] ⪰ [(~t(q))⊤(−(−γf(q))t−1)(q)∗((q)∗)⊤(−(−γf(q))t−1)~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) (I-(I- _f^(q))^t-1 )v^(q)^* (v^(q)^* ) (I-(I- _f^(q))^t-1 ) x_t^(q) x_t^(q)( x_t^(q)) ] + + γβ2[(~t(q))⊤(−γf(q))2(t−1)f(q)‖(q)∗‖−(−γf(q))2(t−1)2~t(q)~t(q)(~t(q))⊤]. γβ2E [( x_t^(q)) (I- _f^(q))^2(t-1)H_f^(q) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2 x_t^(q) x_t^(q)( x_t^(q)) ]. Further by Assumption 3.3, we have [(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪰ βtr(f(q)[−(−γf(q))t−1](q)∗((q)∗)⊤[−(−γf(q))t−1])f(q) (H_f^(q) [I-(I- _f^(q))^t-1 ]v^(q)^* (v^(q)^* ) [I-(I- _f^(q))^t-1 ] )H_f^(q) + + γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). Upon this, we can successfully derive the lower bound update rule for [tt⊤]E[ η_t η_t ] (see Lemma D.2 for details): [t⊗t] [ η_t η_t ] ⪰ [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+γ2(1+ϵ¯o)σ¯2f(q) [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+γ^2(1+ ε_o) σ^2H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]βtr(f(q)[−(−γf(q))(t−1)]2(q)∗((q)∗)⊤)f(q) γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] (H_f^(q) [I-(I- _f^(q))^(t-1) ]^2v^(q)^* (v^(q)^* ) )H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). With this lower bound, we can then apply standard techniques to derive risk lower bounds under general spectrum. Challenge I: Spectral distortion induced by quantized sketching. Since the update rule (quantized SGD) operates strictly within the quantized feature space, our risk analysis hinges on the spectral properties of the quantized covariance f(q)H_f^(q). Unlike Lin et al. (2024) where the covariance ⊤SHS preserves the polynomial decay of the data covariance H, additive quantization fundamentally alters the polynomial spectral structure. This disruption necessitates a novel analysis to characterize the eigenvalues of f(q)H_f^(q) and derive risk bounds under this distorted spectrum. Our strategy. We leverage the concentration properties of the random sketch matrix S to rigorously bound the eigenvalues of the quantized covariance under additive quantization, showing that the spectrum of f(q)H_f^(q) behaves as a superposition of the original power-law decay and a dimension-dependent quantization error: (see Lemma G.4 for upper bounds and Lemma G.5 for lower bounds): j−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM≲μj(f(q))≲j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM.j^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM _j(H_f^(q)) j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM. Consequently, analyzing the variance error Var=k∗N+Nγ2∑i>k∗(λ~i(q))2Var= k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 necessitates a spectral decomposition that separates the constant quantization error from the decaying polynomial signal. This operation yields a penalty term scaling as Nγ2ϵ2(M−k∗)Nγ^2ε^2(M-k^*) 555Here ϵ=ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpMε= ε_f+(1+ ε_dp) ε_s+ ε_d pM. (see Lemma C.20 for upper bounds and Lemma D.17 for lower bounds). Physically, this term represents the cumulative noise injected into the tail subspace, providing a direct mechanism for the reduction in the effective model size MeffM_ eff characterized in Theorem 4.2. 6 Conclusion We establish upper and lower bounds on the scaling laws for low-precision training under multiplicative and additive quantization within a high-dimensional sketched linear regression setting. Our theoretical analysis demonstrates that while both schemes reduce the effective data size and introduce an additive error, they fundamentally differ in their impact on model capacity: additive quantization reduces the effective model size, whereas multiplicative quantization preserves it. Our experiments validates our theory. These findings align with prior studies and offer actionable insights for designing low-precision training strategies. Limitations. Future work may address three key limitations of this study: (1) establishing matching lower and upper bounds; (2) extending the theoretical framework to non-linear models; and (3) analyzing other optimization methods. References A. Atanasov, J. A. Zavatone-Veth, and C. Pehlevan (2024) Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592. Cited by: §2. F. Bach and E. Moulines (2013) Non-strongly-convex smooth stochastic approximation with convergence rate o (1/n). Advances in neural information processing systems 26. Cited by: §2. Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma (2024) Explaining neural scaling laws. Proceedings of the National Academy of Sciences 121 (27), p. e2311878121. Cited by: §2. P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler (2020) Benign overfitting in linear regression. Proceedings of the National Academy of Sciences 117 (48), p. 30063–30070. Cited by: §2. R. Berthier, F. Bach, and P. Gaillard (2020) Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. Advances in Neural Information Processing Systems 33, p. 2576–2586. Cited by: §2. B. Bordelon, A. Atanasov, and C. Pehlevan (2024) A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092. Cited by: §2. M. Chen, C. Zhang, J. Liu, Y. Zeng, Z. Xue, Z. Liu, Y. Li, J. Ma, J. Huang, X. Zhou, and P. Luo (2025a) Scaling law for quantization-aware training. External Links: 2505.14302, Link Cited by: §2, §4.1. Y. Chen, X. Guo, X. Li, Y. Liang, Z. Shi, and Z. Song (2025b) Scaling law phenomena across regression paradigms: multiple and kernel approaches. arXiv preprint arXiv:2503.01314. Cited by: §1. M. Courbariaux, Y. Bengio, and J. David (2014) Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024. Cited by: §1. A. Défossez and F. Bach (2015) Averaged least-mean-squares: bias-variance trade-offs and optimal sampling distributions. In Artificial Intelligence and Statistics, p. 205–213. Cited by: §2. T. Dettmers and L. Zettlemoyer (2023) The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, p. 7750–7774. Cited by: §2. A. Dieuleveut and F. Bach (2015) Non-parametric stochastic approximation with large step sizes. Annals of Statistics 44 (4). Cited by: §2. A. Dieuleveut, N. Flammarion, and F. Bach (2017) Harder, better, faster, stronger convergence rates for least-squares regression. Journal of Machine Learning Research 18 (101), p. 1–51. Cited by: §2. S. Ding, H. Zhang, H. Zhao, and C. Fang (2025) Scaling law for stochastic gradient descent in quadratically parameterized linear regression. arXiv preprint arXiv:2502.09106. Cited by: §1, §2. M. Fishman, B. Chmiel, R. Banner, and D. Soudry (2024) Scaling fp8 training to trillion-token llms. arXiv preprint arXiv:2409.12517. Cited by: §1. E. Frantar, U. Evci, W. Park, N. Houlsby, and D. Alistarh (2025) Compression scaling laws: unifying sparsity and quantization. arXiv preprint arXiv:2502.16440. Cited by: §2. Z. Hao, J. Guo, L. Shen, Y. Luo, H. Hu, G. Wang, D. Yu, Y. Wen, and D. Tao (2025) Low-precision training of large language models: methods, challenges, and opportunities. arXiv preprint arXiv:2505.01043. Cited by: §1. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. (2022) Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. Cited by: §1. P. Jain, S. M. Kakade, R. Kidambi, P. Netrapalli, V. K. Pillutla, and A. Sidford (2017) A markov chain theory approach to characterizing the minimax optimality of stochastic gradient descent (for least squares). arXiv preprint arXiv:1710.09430. Cited by: §2. P. Jain, S. M. Kakade, R. Kidambi, P. Netrapalli, and A. Sidford (2018) Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research 18 (223), p. 1–42. Cited by: §2. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1, §2. T. Kumar, Z. Ankner, B. F. Spector, B. Bordelon, N. Muennighoff, M. Paul, C. Pehlevan, C. Ré, and A. Raghunathan (2024) Scaling laws for precision. arXiv preprint arXiv:2411.04330. Cited by: §1, §1, §2, §4.2. A. Kuzmin, M. Van Baalen, Y. Ren, M. Nagel, J. Peters, and T. Blankevoort (2022) Fp8 quantization: the power of the exponent. Advances in Neural Information Processing Systems 35, p. 14651–14662. Cited by: §3.1. Y. Lei, T. Hu, and K. Tang (2021) Generalization performance of multi-pass stochastic gradient descent with convex loss functions. Journal of Machine Learning Research 22 (25), p. 1–41. Cited by: §2. B. Li, F. Chen, Z. Huang, L. Wang, and L. Wu (2025) Functional scaling laws in kernel regression: loss dynamics and learning rate schedules. arXiv preprint arXiv:2509.19189. Cited by: §1, §2. L. Lin, J. Wu, and P. L. Bartlett (2025) Improved scaling laws in linear regression via data reuse. arXiv preprint arXiv:2506.08415. Cited by: §1, §1, §2. L. Lin, J. Wu, S. M. Kakade, P. L. Bartlett, and J. D. Lee (2024) Scaling laws in linear regression: compute, parameters, and data. Advances in Neural Information Processing Systems 37, p. 60556–60606. Cited by: Lemma C.22, Lemma G.1, Lemma G.6, Appendix, §1, §1, §2, §3.2, §4.1, §4.2, §5, §5. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. (2024) Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1. Z. Liu, C. Zhao, H. Huang, S. Chen, J. Zhang, J. Zhao, S. Roy, L. Jin, Y. Xiong, Y. Shi, et al. (2025) ParetoQ: improving scaling laws in extremely low-bit llm quantization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2. I. Markov, A. Vladu, Q. Guo, and D. Alistarh (2023) Quantized distributed training of large models with convergence guarantees. In International Conference on Machine Learning, p. 24020–24044. Cited by: §3.1. I. Modoranu, M. Safaryan, G. Malinovsky, E. Kurtić, T. Robert, P. Richtárik, and D. Alistarh (2024) Microadam: accurate adaptive optimization with low space overhead and provable convergence. Advances in Neural Information Processing Systems 37, p. 1–43. Cited by: §3.1. X. Ouyang, T. Ge, T. Hartvigsen, Z. Zhang, H. Mi, and D. Yu (2024) Low-bit quantization favors undertrained llms: scaling laws for quantized llms with 100t training tokens. arXiv preprint arXiv:2411.17691. Cited by: §2. K. Ozkara, T. Yu, and Y. Park (2025) Stochastic rounding for llm training: theory and practice. arXiv preprint arXiv:2502.20566. Cited by: §3.1. C. Paquette, E. Paquette, B. Adlam, and J. Pennington (2024a) Homogenization of sgd in high-dimensions: exact dynamics and generalization properties. Mathematical Programming, p. 1–90. Cited by: §2. E. Paquette, C. Paquette, L. Xiao, and J. Pennington (2024b) 4+ 3 phases of compute-optimal neural scaling laws. Advances in Neural Information Processing Systems 37, p. 16459–16537. Cited by: §2. H. Peng, K. Wu, Y. Wei, G. Zhao, Y. Yang, Z. Liu, Y. Xiong, Z. Yang, B. Ni, J. Hu, et al. (2023) Fp8-lm: training fp8 large language models. arXiv preprint arXiv:2310.18313. Cited by: §1. B. T. Polyak and A. B. Juditsky (1992) Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization 30 (4), p. 838–855. Cited by: §2. X. Sun, N. Wang, C. Chen, J. Ni, A. Agrawal, X. Cui, S. Venkataramani, K. El Maghraoui, V. V. Srinivasan, and K. Gopalakrishnan (2020) Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems 33, p. 1796–1807. Cited by: §1. X. Sun, S. Li, R. Xie, W. Han, K. Wu, Z. Yang, Y. Li, A. Wang, S. Li, J. Xue, et al. (2025) Scaling laws for floating point quantization training. arXiv preprint arXiv:2501.02423. Cited by: §1, §1, §2, §4.1, §4.2. C. Tao, Q. Liu, L. Dou, N. Muennighoff, Z. Wan, P. Luo, M. Lin, and N. Wong (2024) Scaling laws with vocabulary: larger models deserve larger vocabularies. Advances in Neural Information Processing Systems 37, p. 114147–114179. Cited by: §2. A. Tsigler and P. L. Bartlett (2023) Benign overfitting in ridge regression. Journal of Machine Learning Research 24 (123), p. 1–76. Cited by: §2. A. V. Varre, L. Pillaud-Vivien, and N. Flammarion (2021) Last iterate convergence of sgd for least-squares in the interpolation regime.. Advances in Neural Information Processing Systems 34, p. 21581–21591. Cited by: §2. N. Wang, J. Choi, D. Brand, C. Chen, and K. Gopalakrishnan (2018) Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems 31. Cited by: §1. M. Wortsman, T. Dettmers, L. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt (2023) Stable and low-precision training for large-scale vision-language models. Advances in Neural Information Processing Systems 36, p. 10271–10298. Cited by: §1. H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius (2020) Integer quantization for deep learning inference: principles and empirical evaluation. arXiv preprint arXiv:2004.09602. Cited by: §3.1. J. Wu, D. Zou, V. Braverman, Q. Gu, and S. Kakade (2022a) Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression. In International conference on machine learning, p. 24280–24314. Cited by: §B.1, §2, §2, §3.2, §3.2, §4.3. J. Wu, D. Zou, V. Braverman, Q. Gu, and S. Kakade (2022b) The power and limitation of pretraining-finetuning for linear regression under covariate shift. Advances in Neural Information Processing Systems 35, p. 33041–33053. Cited by: §2, §3.2, §3.2, §4.3. J. Wu, D. Zou, Z. Chen, V. Braverman, Q. Gu, and S. M. Kakade (2023) Finite-sample analysis of learning high-dimensional single relu neuron. In International Conference on Machine Learning, p. 37919–37951. Cited by: §3.2, §3.2. H. Xi, Y. Chen, K. Zhao, K. J. Teh, J. Chen, and J. Zhu (2024) Jetfire: efficient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403.12422. Cited by: §1. T. Yan, H. Wen, B. Li, K. Luo, W. Chen, and K. Lyu (2025) Larger datasets can be repeated more: a theoretical analysis of multi-epoch scaling in linear regression. arXiv preprint arXiv:2511.13421. Cited by: §1, §2. D. Zhang, J. Su, and D. Zou (2025) Learning under quantization for high-dimensional linear regression. arXiv preprint arXiv:2510.18259. Cited by: §B.1, §1, §2, §3.1, §3.2, §3.2, §3.2, §4.1, §4.1, §4.2, §5, §5, §5. H. Zhang, Y. Liu, Q. Chen, and C. Fang (2024) The optimality of (accelerated) sgd for high-dimensional quadratic optimization. arXiv preprint arXiv:2409.09745. Cited by: §2. D. Zou, J. Wu, V. Braverman, Q. Gu, and S. Kakade (2021) Benign overfitting of constant-stepsize sgd for linear regression. In Conference on learning theory, p. 4633–4635. Cited by: §B.1, §2, §2, §3.2, §3.2, §4.3, footnote 6, footnote 7. D. Zou, J. Wu, V. Braverman, Q. Gu, and S. Kakade (2022) Risk bounds of multi-pass sgd for least squares in the interpolation regime. Advances in Neural Information Processing Systems 35, p. 12909–12920. Cited by: §2. Appendix We provide detailed proofs in the Appendix. Recall the population risk ℛM():=12[(⟨,⟩−y)2],ℛ():=12[(⟨,∗⟩−y)2],R_M(v):= 12E [ ( ,v -y )^2 ], (w):= 12E [ ( ,w^* -y )^2 ], and the decomposition ℛM(¯N)= _M( v_N)= minℛ(⋅)⏟Irreducible+minℛM(⋅)−minℛ(⋅)⏟Approx+ℛM(¯N)−minℛM(⋅)⏟Excess. (·)_ Irreducible+ _M(·)- (·)_ Approx+ R_M( v_N)- _M(·)_ Excess. We first provide bounds for the IrreducibleIrreducible. By the well-specified model Assumption 3.5, Irreducible:=ℛ(∗)=12σ2. Irreducible:=R(w^*)= 12σ^2. (5) We then provide matching bounds for ApproxApprox. As established in Lemma C.4 in Lin et al. (2024), under Assumption 3.5, with probability at least 1−e−Ω(M)1-e^- (M), ∗Approx≂M1−a.E_w^*Approx M^1-a. (6) In Section B-D, we will derive bounds for ExcessExcess. In Section E, we will derive scaling laws using risk bounds under general spectrum and Assumption 3.5. Unless otherwise specified, expectations are conditioned on S and ∗w^*. Appendix Contents 1 Introduction 2 Related Work 3 Theoretical Setup 3.1 Quantization Operation 3.2 Data Model 4 Main Theory 4.1 Multiplicative Quantization 4.2 Additive Quantization 4.3 Lower Bound Analysis 4.3.1 Multiplicative Quantization 4.3.2 Additive Quantization 5 Proof Overview 6 Conclusion References A Omitted Proofs A.1 Proof for Theorem 4.1 A.2 Proof for Theorem 4.2 A.3 Proof for Theorem 4.3 A.4 Proof for Theorem 4.4 B Initial Study B.1 Preliminary B.2 Excess Risk Decomposition C Upper Bound Analysis C.1 Update Rule C.2 Bias-Variance Decomposition C.3 Variance Upper Bounds C.3.1 General Quantization C.3.2 Multiplicative Quantization C.4 Bias Upper Bounds C.4.1 General Quantization C.4.2 Multiplicative Quantization C.5 Final Upper Bounds C.5.1 General Quantization C.5.2 Multiplicative Quantization C.6 Additive Error Upper Bounds under Power-law Spectrum C.7 Variance Upper Bounds under Power-Law Spectrum C.7.1 Multiplicative Quantization C.7.2 Additive Quantization C.8 Bias Upper Bounds under Power-Law Spectrum C.8.1 Multiplicative Quantization C.8.2 Additive Quantization C.9 Population Risk Upper Bounds under Power-law Spectrum C.9.1 Multiplicative Quantization C.9.2 Additive Quantization D Lower Bound Analysis D.1 Update Rule D.2 Bias-Variance Decomposition D.3 Variance Lower Bounds D.3.1 General Quantization D.3.2 Multiplicative Quantization D.4 Bias Lower Bounds D.4.1 General Quantization D.4.2 Multiplicative Quantization D.5 Final Lower Bounds D.5.1 General Quantization D.5.2 Multiplicative Quantization D.6 Additive Error Lower Bounds under Power-law Spectrum D.7 Variance Lower Bounds under Power-Law Spectrum D.7.1 Multiplicative Quantization D.7.2 Additive Quantization D.8 Bias Lower Bounds under Power-Law Spectrum D.8.1 Multiplicative Quantization D.8.2 Additive Quantization D.9 Population Risk Lower Bounds under Power-law Spectrum D.9.1 Multiplicative Quantization D.9.2 Additive Quantization E Scaling Laws E.1 Multiplicative Quantization E.2 Additive Quantization F Auxiliary Lemmas G Concentration Lemmas H Discussions on Assumptions H.1 Fourth-order Assumption H.1.1 Multiplicative Quantization H.1.2 Additive Quantization H.2 Second-order Noise Assumption H.2.1 Multiplicative Quantization The following proof dependency graph visually encapsulates the main logical structure and organizational architecture of the theoretical results in our paper. In particular, the arrow from element X to element Y means the proof of Y relies on X. To maintain visual clarity, we omit auxiliary and concentration lemmas from the graph. However, it is crucial to note that the concentration lemmas establish both upper and lower bounds for the eigen-spectra of f(q)H_f^(q) and ⊤SHS . These results facilitate the refinement of bounds from general spectra to power-law spectra and are essential for proving the upper bound lemmas (Lemmas C.18, C.20, C.26, C.24, C.19, and C.17) and the lower bound lemmas (Lemmas D.15, D.17, D.20, D.14, D.16, and D.19). Theorem 4.2Theorem 4.1Theorem E.1Theorem E.3Theorem C.3Theorem C.4Theorem C.1Theorem C.2Lemma B.2Lemma C.24Lemma C.19Lemma C.17Lemma C.26Lemma C.20Lemma C.18Lemma C.16Lemma C.10Lemma C.13Lemma C.8Lemma C.9Lemma C.14Lemma C.15Lemma C.12Lemma C.11Lemma C.7Lemma C.5Lemma C.6Lemma C.4Lemma C.3Lemma C.2Lemma C.1Theorem 4.4Theorem 4.3Theorem E.2Theorem E.4Theorem D.3Theorem D.4Lemma D.19Lemma D.16Lemma D.14Lemma D.20Lemma D.17Lemma D.15Theorem D.1Theorem D.2Lemma B.2Lemma D.13Lemma D.9Lemma D.11Lemma D.7Lemma D.8Lemma D.12Lemma D.10Lemma D.6Lemma D.5Lemma D.4Lemma D.3Lemma D.1Lemma D.2 Appendix A Omitted Proofs A.1 Proof for Theorem 4.1 Proof. The proof is completed by Theorem E.1 with ϵ¯i=0,i=d,f,s,p,a,o ε_i=0,\ i=d,f,s,p,a,o. ∎ A.2 Proof for Theorem 4.2 Proof. The proof is completed by Theorem E.3 with ϵ¯i=0,i=d,f,s,p,a,o ε_i=0,\ i=d,f,s,p,a,o. ∎ A.3 Proof for Theorem 4.3 Proof. The proof is completed by Theorem E.2. ∎ A.4 Proof for Theorem 4.4 Proof. The proof is completed by Theorem E.4. ∎ Appendix B Initial Study B.1 Preliminary Denote ~(q)=f(s()d(t)) x^(q)=Q_f (Q_s(S)Q_d(x_t) ), f(q):=[~(q)(~(q))⊤]H_f^(q):=E [ x^(q)( x^(q)) ]. We first define the following linear operators as in Zou et al. (2021); Wu et al. (2022a); Zhang et al. (2025): ℐ=⊗,ℳ(q)=[(~(q))⊗(~(q))⊗(~(q))⊗(~(q))], =I , ^(q)=E [ ( x^(q) ) ( x^(q) ) ( x^(q) ) ( x^(q) ) ], ℳ~(q)=f(q)⊗f(q),(q)=f(q)⊗+⊗f(q)−γℳ(q), M^(q)=H_f^(q) _f^(q), ^(q)=H_f^(q) +I _f^(q)- ^(q), ~(q)=f(q)⊗+⊗f(q)−γℳ~(q). T^(q)=H_f^(q) +I _f^(q)-γ M^(q). For a symmetric matrix A, the above definitions result in: ℐ∘=,ℳ(q)∘=[(~(q))⊤~(q)~(q)(~(q))⊤],ℳ~(q)∘=f(q)f(q), =A, ^(q) =E [( x^(q)) A x^(q) x^(q)( x^(q)) ], M^(q) =H_f^(q)AH_f^(q), (ℐ−γ(q))∘=[(−γ~(q)(~(q))⊤)(−γ~(q)(~(q))⊤)], (I- ^(q)) =E [ (I-γ x^(q)( x^(q)) )A (I-γ x^(q)( x^(q)) ) ], (ℐ−γ~(q))∘=(−γf(q))(−γf(q)). (I-γ T^(q)) = (I- _f^(q) )A (I- _f^(q) ). B.2 Excess Risk Decomposition We first compute the global minimum of ℛM()R_M(v): ∗:=argminℛM()=argmin12[(⟨,⟩−y)2].v^*:= argmin_vR_M(v)= argmin_v 12E [( ,Sx -y)^2 ]. Note that ℛM()R_M(v) is a quadratic function, so its minimum is given by ∗=(⊤)−1∗.v^*= (SHS )^-1SHw^*. Further, we consider the global minimum of the risk on the quantized data space: (q)∗:=argminℛM(q)()=argmin12[(⟨~(q),⟩−l(y))2].v^(q)^*:= argmin_vR_M^(q)(v)= argmin_v 12E [ ( x^(q),v -Q_l(y) )^2 ]. Similarly, (q)∗=(f(q))−1∗.v^(q)^*=(H_f^(q))^-1SHw^*. The optimality also implies the following first order optimality: [(y−⟨∗,⟩)]=0,[(l(y)−⟨(q)∗,~(q)⟩)~(q)]=0.E[(y- v^*,Sx )Sx]=0, [(Q_l(y)- v^(q)^*, x^(q) ) x^(q)]=0. (7) Lemma B.1 (Excess risk decomposition). Under Assumption 3.1 and Assumption 3.2, [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨f(q),[((q)∗−¯N)⊗((q)∗−¯N)]⟩ 12 _f^(q),E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + 12[⟨(q)∗,−~(q)⟩2] 12E [ v^(q)^*,Sx- x^(q) ^2 ] − - 12[⟨¯N,−~(q)⟩2]. 12E [ v_N,Sx- x^(q) ^2 ]. Proof. By definition, [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12[(⟨,¯N⟩−y)2]−12[(⟨,∗⟩−y)2] 12E [ ( , v_N -y )^2 ]- 12E [ ( ,v^* -y )^2 ] = = 12[(y−⟨¯N,⟩)2]−12[(l(y)−⟨¯N,~(q)⟩)2]⏟R1 12E [ (y- v_N,Sx )^2 ]- 12E [(Q_l(y)- v_N, x^(q) )^2 ]_R_1 + + 12[(l(y)−⟨¯N,~(q)⟩)2]−12[(l(y)−⟨(q)∗,~(q)⟩)2]⏟R2 12E [(Q_l(y)- v_N, x^(q) )^2 ]- 12E [(Q_l(y)- v^(q)^*, x^(q) )^2 ]_R_2 + + 12[(l(y)−⟨(q)∗,~(q)⟩)2]−12[(y−⟨(q)∗,⟩)2]⏟R3 12E [(Q_l(y)- v^(q)^*, x^(q) )^2 ]- 12E [ (y- v^(q)^*,Sx )^2 ]_R_3 + + 12[(y−⟨(q)∗,⟩)2]−12[(y−⟨∗,⟩)2]⏟R4. 12E [ (y- v^(q)^*,Sx )^2 ]- 12E [ (y- ^*,Sx )^2 ]_R_4. We would like to remark that the quantization operations in l(y)Q_l(y) and ~(q) x^(q) introduced in excess risk decomposition are independent of those quantization operators introduced in the training stage, i.e., ¯N v_N. We then deal with each term respectively. For R1R_1, 12[(y−⟨¯N,⟩)2]−12[(l(y)−⟨¯N,~(q)⟩)2] 12E [ (y- v_N,Sx )^2 ]- 12E [(Q_l(y)- v_N, x^(q) )^2 ] (8) = = 12[(y−l(y)−⟨¯N,−~(q)⟩)⋅(y+l(y)−⟨¯N,+~(q)⟩)] 12E [ (y-Q_l(y)- v_N,Sx- x^(q) )· (y+Q_l(y)- v_N,Sx+ x^(q) ) ] = = 12[(y−l(y))(y+l(y))]−12[⟨¯N,−~(q)⟩2], 12E [(y-Q_l(y))(y+Q_l(y)) ]- 12E [ v_N,Sx- x^(q) ^2 ], where the last equality uses the unbiased quantization Assumption 3.1. For R2R_2, 12[(l(y)−⟨¯N,~(q)⟩)2]−12[(l(y)−⟨(q)∗,~(q)⟩)2] 12E [(Q_l(y)- v_N, x^(q) )^2 ]- 12E [(Q_l(y)- v^(q)^*, x^(q) )^2 ] (9) = = 12[⟨(q)∗−¯N,~(q)⟩(2l(y)−⟨(q)∗+¯N,~(q)⟩)] 12E [ v^(q)^*- v_N, x^(q) (2Q_l(y)- v^(q)^*+ v_N, x^(q) ) ] = = 12[⟨(q)∗−¯N,~(q)⟩2] 12E [ v^(q)^*- v_N, x^(q) ^2 ] = = 12⟨f(q),[((q)∗−¯N)⊗((q)∗−¯N)]⟩, 12 _f^(q),E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] , where the second equality holds by the optimality (7). For R3R_3, 12[(l(y)−⟨(q)∗,~(q)⟩)2]−12[(y−⟨(q)∗,⟩)2] 12E [(Q_l(y)- v^(q)^*, x^(q) )^2 ]- 12E [ (y- v^(q)^*,Sx )^2 ] (10) = = 12[(l(y)−y−⟨(q)∗,~(q)−⟩)(l(y)+y−⟨(q)∗,~(q)+⟩)] 12E [ (Q_l(y)-y- v^(q)^*, x^(q)-Sx ) (Q_l(y)+y- v^(q)^*, x^(q)+Sx ) ] = = 12[(l(y)−y)(y+l(y))]+12[⟨(q)∗,−~(q)⟩2], 12E [(Q_l(y)-y)(y+Q_l(y)) ]+ 12E [ v^(q)^*,Sx- x^(q) ^2 ], where the last equality holds by unbiased quantization Assumption 3.1. For R4R_4, 12[(y−⟨(q)∗,⟩)2]−12[(y−⟨∗,⟩)2] 12E [ (y- v^(q)^*,Sx )^2 ]- 12E [ (y- ^*,Sx )^2 ] (11) = = 12[(⟨∗−(q)∗,⟩)(2y−⟨∗+(q)∗,⟩)] 12E [ ( ^*-v^(q)^*,Sx ) (2y- ^*+v^(q)^*,Sx ) ] = = 12[⟨∗−(q)∗,⟩2] 12E [ ^*-v^(q)^*,Sx ^2 ] = = 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩, 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) , where the second equality holds by the optimality (7). Combining (8), (9), (10) and (11), it holds [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨f(q),[((q)∗−¯N)⊗((q)∗−¯N)]⟩ 12 _f^(q),E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + 12[⟨(q)∗,−~(q)⟩2] 12E [ v^(q)^*,Sx- x^(q) ^2 ] − - 12[⟨¯N,−~(q)⟩2]. 12E [ v_N,Sx- x^(q) ^2 ]. ∎ Lemma B.2 (Refined excess risk decomposition). Under Assumption 3.1, Assumption 3.2, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨⊤,[((q)∗−¯N)⊗((q)∗−¯N)]⟩⏟RN 12 ,E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] _R_N + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + ((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. Proof. By Lemma B.1, [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨f(q),[((q)∗−¯N)⊗((q)∗−¯N)]⟩ 12 _f^(q),E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + 12[⟨(q)∗,−~(q)⟩2] 12E [ v^(q)^*,Sx- x^(q) ^2 ] − - 12[⟨¯N,−~(q)⟩2]. 12E [ v_N,Sx- x^(q) ^2 ]. Recall that ¯N=¯N−(q)∗+(q)∗ v_N= v_N-v^(q)^*+v^(q)^*, it holds [⟨¯N,−~(q)⟩2]= [ v_N,Sx- x^(q) ^2 ]= [¯N⊤(f(q)−⊤)¯N] [ v_N (H_f^(q)-SHS ) v_N ] = = [(¯N−(q)∗)⊤(f(q)−⊤)(¯N−(q)∗)] [ ( v_N-v^(q)^* ) (H_f^(q)-SHS ) ( v_N-v^(q)^* ) ] + + [((q)∗)⊤(f(q)−⊤)(q)∗] [ (v^(q)^* ) (H_f^(q)-SHS )v^(q)^* ] + + 2[(¯N−(q)∗)⊤(f(q)−⊤)(q)∗]. 2E [ ( v_N-v^(q)^* ) (H_f^(q)-SHS )v^(q)^* ]. Hence, [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨⊤,[((q)∗−¯N)⊗((q)∗−¯N)]⟩ 12 ,E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] (12) + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) − - [(¯N−(q)∗)⊤(f(q)−⊤)(q)∗]. [ ( v_N-v^(q)^* ) (H_f^(q)-SHS )v^(q)^* ]. Denote t=t−(q)∗ η_t=v_t-v^(q)^*, then by Lemma C.1, t=(−γ~t(q)(~t(q))⊤)t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q), η_t= (I-γ x_t^(q)( x_t^(q)) ) η_t-1+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q), where ϵt(o):= ε_t^(o)= o(l(yt)−a((~t(q))⊤p(t−1)))−[l(yt)−a((~t(q))⊤p(t−1))], _o (Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) )- [Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ], ϵt(a):= ε_t^(a)= a((~t(q))⊤p(t−1))−(~t(q))⊤p(t−1), _a (( x_t^(q)) Q_p(v_t-1) )-( x_t^(q)) Q_p(v_t-1), ϵt−1(p):= ε_t-1^(p)= p(t−1)−t−1, _p(v_t-1)-v_t-1, ξt:= ξ_t= l(yt)−(~t(q))⊤(q)∗. _l(y_t)-( x_t^(q)) v^(q)^*. It follows by the unbiased quantization Assumption 3.1 and the optimality (7) that [t]=[[t|t−1]]=[(−γf(q))t−1]=(−γf(q))[t−1]=(−γf(q))t0.E [ η_t ]=E [E [ η_t| η_t-1 ] ]=E [ (I- _f^(q) ) η_t-1 ]= (I- _f^(q) )E [ η_t-1 ]= (I- _f^(q) )^t η_0. (13) Hence, [¯N−(q)∗]⊤(f(q)−⊤)(q)∗ [ v_N-v^(q)^* ] (H_f^(q)-SHS )v^(q)^* (14) = = 1N∑t=0N−1[t]⊤(f(q)−⊤)(q)∗ 1N _t=0^N-1E[ η_t] (H_f^(q)-SHS )v^(q)^* = = [1N∑t=0N−1(−γf(q))t0]⊤(f(q)−⊤)(q)∗ [ 1N _t=0^N-1 (I- _f^(q) )^t η_0 ] (H_f^(q)-SHS )v^(q)^* = = −((q)∗)⊤1N∑t=0N−1(−γf(q))t(f(q)−⊤)(q)∗ - (v^(q)^* ) 1N _t=0^N-1 (I- _f^(q) )^t (H_f^(q)-SHS )v^(q)^* = = −((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. - (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. Together with (12) and (14), we have [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨⊤,[((q)∗−¯N)⊗((q)∗−¯N)]⟩ 12 ,E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + ((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. ∎ In the following part, we first establish upper bounds in Section C and then establish lower bounds in Section D. Specifically, we first analyze the algorithm-dependent excess risk RN=12⟨⊤,[((q)∗−¯N)⊗((q)∗−¯N)]⟩,R_N= 12 ,E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] , and then analyze the remaining algorithm-independent additive error 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩+((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. At last, we reorganize the population risk bounds to derive scaling laws in Section E. Appendix C Upper Bound Analysis We first derive the propagation of the deviation t=t−(q)∗ η_t=v_t-v^(q)^*. C.1 Update Rule Lemma C.1. t=(−γ~t(q)(~t(q))⊤)t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q), η_t= (I-γ x_t^(q)( x_t^(q)) ) η_t-1+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q), where ϵt(o):= ε_t^(o)= o(l(yt)−a((~t(q))⊤p(t−1)))−[l(yt)−a((~t(q))⊤p(t−1))], _o (Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) )- [Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ], ϵt(a):= ε_t^(a)= a((~t(q))⊤p(t−1))−(~t(q))⊤p(t−1), _a (( x_t^(q)) Q_p(v_t-1) )-( x_t^(q)) Q_p(v_t-1), ϵt−1(p):= ε_t-1^(p)= p(t−1)−t−1, _p(v_t-1)-v_t-1, ξt:= ξ_t= l(yt)−(~t(q))⊤(q)∗. _l(y_t)-( x_t^(q)) v^(q)^*. Proof. By (quantized SGD), t=t−1+γo(l(yt)−a(f(s()d(t))⊤p(t−1)))f(s()d(t)). _t=v_t-1+ _o (Q_l(y_t)-Q_a (Q_f (Q_s(S)Q_d(x_t) ) Q_p(v_t-1) ) )Q_f (Q_s(S)Q_d(x_t) ). Then we have t=t−1+γo(l(yt)−a(f(s()d(t))⊤p(t−1)))f(s()d(t)). η_t= η_t-1+ _o (Q_l(y_t)-Q_a (Q_f (Q_s(S)Q_d(x_t) ) Q_p(v_t-1) ) )Q_f (Q_s(S)Q_d(x_t) ). Denote ~t(q)=f(s()d(t)) x_t^(q)=Q_f (Q_s(S)Q_d(x_t) ). We then introduce quantization errors to better characterize each quantization operation (⋅)Q(·). In particular, define quantization errors: ϵt(o):= ε_t^(o)= o(l(yt)−a((~t(q))⊤p(t−1)))−[l(yt)−a((~t(q))⊤p(t−1))], _o (Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) )- [Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ], ϵt(a):= ε_t^(a)= a((~t(q))⊤p(t−1))−(~t(q))⊤p(t−1), _a (( x_t^(q)) Q_p(v_t-1) )-( x_t^(q)) Q_p(v_t-1), ϵt−1(p):= ε_t-1^(p)= p(t−1)−t−1, _p(v_t-1)-v_t-1, ξt:= ξ_t= l(yt)−(~t(q))⊤(q)∗. _l(y_t)-( x_t^(q)) v^(q)^*. Then the update rule for the parameter deviation can be expressed as: t= η_t= t−1+γo(l(yt)−a((~t(q))⊤p(t−1)))~t(q) η_t-1+ _o (Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ) x_t^(q) = = t−1+γ(l(yt)−a((~t(q))⊤p(t−1))+ϵt(o))~t(q) η_t-1+γ (Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) )+ _t^(o) ) x_t^(q) = = t−1+γ(l(yt)−((~t(q))⊤p(t−1)+ϵt(a))+ϵt(o))~t(q) η_t-1+γ (Q_l(y_t)- (( x_t^(q)) Q_p(v_t-1)+ _t^(a) )+ _t^(o) ) x_t^(q) = = t−1+γ(l(yt)−((~t(q))⊤(t−1+ϵt−1(p)−(q)∗+(q)∗))+ϵt(o)−ϵt(a))~t(q) η_t-1+γ (Q_l(y_t)- (( x_t^(q)) (v_t-1+ ε_t-1^(p)-v^(q)^*+v^(q)^*) )+ _t^(o)- _t^(a) ) x_t^(q) = = t−1−γ~t(q)(~t(q))⊤t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q). η_t-1-γ x_t^(q)( x_t^(q)) η_t-1+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q). ∎ We then derive the propagation of [t⊗t]E [ η_t η_t ]. By Lemma C.1, t=(−γ~t(q)(~t(q))⊤)t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q). η_t= (I-γ x_t^(q)( x_t^(q)) ) η_t-1+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q). Denote tbias=(−γ~t(q)(~t(q))⊤)t−1bias,0bias=0, η_t bias= (I-γ x_t^(q)( x_t^(q)) ) η_t-1 bias, η_0 bias= η_0, tvar=(−γ~t(q)(~t(q))⊤)t−1var+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q),0var=. η_t var= (I-γ x_t^(q)( x_t^(q)) ) η_t-1 var+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q), η_0 var= 0. Obviously, it holds t=tvar+tbias, η_t= η_t var+ η_t bias, and [t⊗t]⪯2([tbias⊗tbias]⏟t+[tvar⊗tvar]⏟t).E [ η_t η_t ] 2 ( E [ η_t bias η_t bias ]_B_t+ E [ η_t var η_t var ]_C_t ). (15) Regarding tB_t, we have t=[(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)].B_t=E [ (I-γ x_t^(q)( x_t^(q)) )B_t-1 (I-γ x_t^(q)( x_t^(q)) ) ]. (16) Regarding tC_t, by the unbiased quantization Assumption 3.1 and 0var= η_0 var= 0, it holds t=[(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)]+t,C_t=E [ (I-γ x_t^(q)( x_t^(q)) )C_t-1 (I-γ x_t^(q)( x_t^(q)) ) ]+ _t, (17) where t=γ2[(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))2~t(q)(~t(q))⊤]. _t=γ^2E [ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) )^2 x_t^(q)( x_t^(q)) ]. (18) Further, it holds t= _t= γ2[(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))2~t(q)(~t(q))⊤] γ^2E [ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) )^2 x_t^(q)( x_t^(q)) ] (19) = = γ2[(ξt2+ϵt(o)2+ϵt(a)2+(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q))~t(q)(~t(q))⊤] γ^2E [ ( _t^2+ _t^(o)^2+ _t^(a)^2+( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) ) x_t^(q)( x_t^(q)) ] = = γ2[ξt2~t(q)(~t(q))⊤]+γ2[ϵt(o)2~t(q)(~t(q))⊤] γ^2E [ _t^2 x_t^(q)( x_t^(q)) ]+γ^2E [ _t^(o)^2 x_t^(q)( x_t^(q)) ] + + γ2[ϵt(a)2~t(q)(~t(q))⊤]+γ2[(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤], γ^2E [ _t^(a)^2 x_t^(q)( x_t^(q)) ]+γ^2E [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ], where the second equality holds by the unbiased quantization Assumption 3.1. We then summarize the update rule for [t⊗t]E [ η_t η_t ] as follows. Consider tB_t and tC_t defined in (15). Lemma C.2 (Update rule under multiplicative quantization, an upper bound). If there exist ϵ¯p,ϵ¯a ε_p, ε_a and ϵ¯o ε_o such that for any i∈p,a,oi∈\p,a,o\, quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.2, 3.3 and 3.4, t⪯ _t [(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)] [ (I-γ x_t^(q)( x_t^(q)) )C_t-1 (I-γ x_t^(q)( x_t^(q)) ) ] + + 2γ2(2ϵ¯o+(2ϵ¯o+1)[2(1+ϵ¯p)ϵ¯a+2ϵ¯p])[~t(q)(~t(q))⊤(t−1+t−1)~t(q)(~t(q))⊤] 2γ^2 (2 ε_o+(2 ε_o+1) [2(1+ ε_p) ε_a+2 ε_p ] )E [ x_t^(q)( x_t^(q)) (B_t-1+C_t-1 ) x_t^(q)( x_t^(q)) ] + + γ2(2ϵ¯o+1)[2ϵ¯p+2(1+ϵ¯p)ϵ¯a]αtr(f(q)(q)∗(q)∗⊤)f(q)+γ2(2ϵ¯o+1)σ¯2f(q), γ^2(2 ε_o+1) [2 ε_p+2(1+ ε_p) ε_a ] (H_f^(q)v^(q)^*v^(q)^* )H_f^(q)+γ^2(2 ε_o+1) σ^2H_f^(q), t= _t= [(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)]. [ (I-γ x_t^(q)( x_t^(q)) )B_t-1 (I-γ x_t^(q)( x_t^(q)) ) ]. Proof. By (16) and (17), the proof focuses on dealing with each term of t _t in (19). Firstly, by Assumption 3.4, [ξt2~t(q)(~t(q))⊤]⪯σ¯2f(q). [ _t^2 x_t^(q)( x_t^(q)) ] σ^2H_f^(q). (20) Secondly, by the definition of multiplicative quantization, [(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ] (21) ⪯ ϵ¯p[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] ε_pE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪯ 2ϵ¯p[(~t(q))⊤(t−1t−1⊤+(q)∗(q)∗⊤)~t(q)~t(q)(~t(q))⊤], 2 ε_pE [( x_t^(q)) ( η_t-1 η_t-1 +v^(q)^*v^(q)^* ) x_t^(q) x_t^(q)( x_t^(q)) ], where the last inequality holds by the fact that: for two vectors u and v, (+)(+)⊤⪯2(⊤+⊤)(u+v)(u+v) 2 (uu +vv ). Thirdly, by the definition of multiplicative quantization, [ϵt(a)2~t(q)(~t(q))⊤] [ _t^(a)^2 x_t^(q)( x_t^(q)) ] (22) ⪯ ϵ¯a[(~t(q))⊤p(t−1)p(t−1)⊤~t(q)~t(q)(~t(q))⊤] ε_aE [( x_t^(q)) Q_p(v_t-1)Q_p(v_t-1) x_t^(q) x_t^(q)( x_t^(q)) ] = = ϵ¯a[(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤] ε_aE [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ] + + ϵ¯a[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] ε_aE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪯ (1+ϵ¯p)ϵ¯a[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] (1+ ε_p) ε_aE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪯ 2(1+ϵ¯p)ϵ¯a[(~t(q))⊤(t−1t−1⊤+(q)∗(q)∗⊤)~t(q)~t(q)(~t(q))⊤]. 2(1+ ε_p) ε_aE [( x_t^(q)) ( η_t-1 η_t-1 +v^(q)^*v^(q)^* ) x_t^(q) x_t^(q)( x_t^(q)) ]. Fourthly, by the definition of multiplicative quantization, [ϵt(o)2~t(q)(~t(q))⊤] [ _t^(o)^2 x_t^(q)( x_t^(q)) ] (23) ⪯ ϵ¯o[[l(yt)−a((~t(q))⊤p(t−1))]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[l(yt)−(~t(q))⊤p(t−1)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-( x_t^(q)) Q_p(v_t-1)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[l(yt)−(~t(q))⊤t−1−(~t(q))⊤ϵt−1(p)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-( x_t^(q)) v_t-1-( x_t^(q)) ε_t-1^(p)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[ξt−(~t(q))⊤t−1−(~t(q))⊤ϵt−1(p)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [ _t-( x_t^(q)) η_t-1-( x_t^(q)) ε_t-1^(p)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] ⪯ 2ϵ¯o[[ξt2+(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)+ϵt(a)2+(~t(q))⊤t−1t−1⊤~t(q)]~t(q)(~t(q))⊤]. 2 ε_oE [ [ _t^2+( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q)+ε_t^(a)^2+( x_t^(q)) η_t-1 η_t-1 x_t^(q) ] x_t^(q)( x_t^(q)) ]. Note that by Assumption 3.3, [(~t(q))⊤(q)∗(q)∗⊤~t(q)~t(q)(~t(q))⊤]⪯αtr(f(q)(q)∗(q)∗⊤)f(q).E [( x_t^(q)) v^(q)^*v^(q)^* x_t^(q) x_t^(q)( x_t^(q)) ] (H_f^(q)v^(q)^*v^(q)^* )H_f^(q). (24) Therefore, together with (19), (20), (21), (22), (23) and (24), it holds t/γ2⪯ _t/γ^2 (2ϵ¯o+1)σ¯2f(q) (2 ε_o+1) σ^2H_f^(q) + + (2ϵ¯o+1)[2ϵ¯p+2(1+ϵ¯p)ϵ¯a]αtr(f(q)(q)∗(q)∗⊤)f(q) (2 ε_o+1) [2 ε_p+2(1+ ε_p) ε_a ] (H_f^(q)v^(q)^*v^(q)^* )H_f^(q) + + (2ϵ¯o+(2ϵ¯o+1)[2(1+ϵ¯p)ϵ¯a+2ϵ¯p])[~t(q)(~t(q))⊤t−1t−1⊤~t(q)(~t(q))⊤]. (2 ε_o+(2 ε_o+1) [2(1+ ε_p) ε_a+2 ε_p ] )E [ x_t^(q)( x_t^(q)) η_t-1 η_t-1 x_t^(q)( x_t^(q)) ]. The proof is completed by (15): [t⊗t]⪯2(t+t)E [ η_t η_t ] 2 (B_t+C_t ). ∎ Lemma C.3 (Update rule under general quantization, an upper bound). Under Assumption 3.1, 3.2, 3.3 and 3.4, it holds t= _t= [(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)], [ (I-γ x_t^(q)( x_t^(q)) )B_t-1 (I-γ x_t^(q)( x_t^(q)) ) ], t= _t= [(−γ~t(q)(~t(q))⊤)t−1(−γ~t(q)(~t(q))⊤)] [ (I-γ x_t^(q)( x_t^(q)) )C_t-1 (I-γ x_t^(q)( x_t^(q)) ) ] + + γ2[σ¯2+suptαtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+supt([ϵt(a)2|at]+[ϵt(o)2|ot])]f(q). γ^2 [ σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ) ]H_f^(q). Proof. By (16) and (17), the proof focuses on dealing with each term of t _t in (19). Firstly, by Assumption 3.3, [(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤]⪯ [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ] αtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])f(q) (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )H_f^(q) (25) ⪯ suptαtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])f(q). _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )H_f^(q). Secondly, denote at=(~t(q))⊤p(t−1),ot=l(yt)−a((~t(q))⊤p(t−1)),a_t=( x_t^(q)) Q_p(v_t-1), o_t=Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ), then [(ϵt(a)2+ϵt(o)2)~t(q)(~t(q))⊤]⪯supt([ϵt(a)2|at]+[ϵt(o)2|ot])f(q). [ ( _t^(a)^2+ _t^(o)^2 ) x_t^(q)( x_t^(q)) ] _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] )H_f^(q). (26) Therefore, together with (19), (20), (25) and (26), it holds, t/γ2⪯[σ¯2+suptαtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+supt([ϵt(a)2|at]+[ϵt(o)2|ot])]f(q). _t/γ^2 [ σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ) ]H_f^(q). ∎ C.2 Bias-Variance Decomposition Recall RN= R_N= 12⟨⊤,[¯N⊗¯N]⟩ 12 ,E [ η_N η_N ] (27) ≤ ≤ μmax((f(q))−1⊤)12⟨f(q),[¯N⊗¯N]⟩⏟RN(0). _ max ((H_f^(q))^-1SHS ) 12 _f^(q),E [ η_N η_N ] _R_N^(0). We perform bias-variance decomposition for multiplicative and general cases respectively, to analyze RN(0)R_N^(0). Firstly, we express ¯N⊗¯N η_N η_N into the sum of t η_t. Lemma C.4. Under Assumption 3.1 and Assumption 3.2, it holds RN(0)≤1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),[t⊗t]⟩.R_N^(0)≤ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),E[ η_t η_t] . Proof. By definition ¯N=1N∑t=0N−1t η_N= 1N _t=0^N-1 η_t, we have [¯N⊗¯N]= [ η_N η_N]= 1N2⋅(∑0≤k≤t≤N−1[t⊗k]+∑0≤t<k≤N−1[t⊗k]) 1N^2· ( _0≤ k≤ t≤ N-1E[ η_t η_k]+ _0≤ t<k≤ N-1E[ η_t η_k] ) (28) ⪯ 1N2⋅(∑0≤k≤t≤N−1[[t⊗k|k]]+∑0≤t≤k≤N−1[[t⊗k|t]]). 1N^2· ( _0≤ k≤ t≤ N-1E [E[ η_t η_k| η_k] ]+ _0≤ t≤ k≤ N-1E [E[ η_t η_k| η_t] ] ). By the unbiased quantization Assumption 3.1 and the optimality (7), together with the update rule Lemma C.1, it holds [t|t−1]=(−γf(q))t−1.E [ η_t| η_t-1 ]= (I- _f^(q) ) η_t-1. (29) Therefore, by (28) and (29), [¯N⊗¯N] [ η_N η_N] (30) ⪯ 1N2⋅(∑0≤k≤t≤N−1[[t⊗k|k]]+∑0≤t≤k≤N−1[[t⊗k|t]]) 1N^2· ( _0≤ k≤ t≤ N-1E [E[ η_t η_k| η_k] ]+ _0≤ t≤ k≤ N-1E [E[ η_t η_k| η_t] ] ) = = 1N2⋅(∑0≤k≤t≤N−1(−γf(q))t−k[k⊗k]+∑0≤t≤k≤N−1[t⊗t](−γf(q))k−t) 1N^2· ( _0≤ k≤ t≤ N-1(I- _f^(q))^t-kE[ η_k η_k]+ _0≤ t≤ k≤ N-1E[ η_t η_t](I- _f^(q))^k-t ) = = 1N2⋅∑t=0N−1∑k=tN−1((−γf(q))k−t[t⊗t]+[t⊗t](−γf(q))k−t). 1N^2· _t=0^N-1 _k=t^N-1 ((I- _f^(q))^k-tE[ η_t η_t]+E[ η_t η_t](I- _f^(q))^k-t ). Applying (30) into RNR_N, we have RN(0)= R_N^(0)= 12⟨f(q),[¯N⊗¯N]⟩ 12 _f^(q),E[ η_N η_N] ≤ ≤ 12N2⋅∑t=0N−1∑k=tN−1⟨f(q),(−γf(q))k−t[t⊗t]+[t⊗t](−γf(q))k−t⟩ 12N^2· _t=0^N-1 _k=t^N-1 _f^(q),(I- _f^(q))^k-tE[ η_t η_t]+E[ η_t η_t](I- _f^(q))^k-t = = 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),[t⊗t]⟩, 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),E[ η_t η_t] , where the last equality holds since f(q)H_f^(q) and (−γf(q))k−t(I- _f^(q))^k-t commute. This completes the proof. ∎ Lemma C.5 (Bias-variance decomposition under multiplicative quantization, an upper bound). If there exist ϵ¯p,ϵ¯a ε_p, ε_a and ϵ¯o ε_o such that for any i∈p,a,oi∈\p,a,o\, quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.3, 3.2 and 3.4, if γ<1(1+ϵ~)αtr(f(q))γ< 1(1+ ε) (H_f^(q)), it holds RN(0)/2≤1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)+t(M)⟩,R_N^(0)/2≤ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t^(M)+C_t^(M) , where t(M)=(ℐ−γ(q)+ϵ~γ2ℳ(q))∘t−1(M),0(M)=[0⊗0], _t^(M)=(I- ^(q)+ εγ^2M^(q)) _t-1^(M), _0^(M)=E [ η_0 η_0 ], t(M)=(ℐ−γ(q)+ϵ~γ2ℳ(q))∘t−1(M)+γ2σM2f(q),0(M)=, _t^(M)=(I- ^(q)+ εγ^2M^(q)) _t-1^(M)+γ^2 _M^2H_f^(q), _0^(M)= 0, with ϵ~=4ϵ¯o+2(2ϵ¯o+1)[2(1+ϵ¯p)ϵ¯a+2ϵ¯p], ε=4 ε_o+2(2 ε_o+1) [2(1+ ε_p) ε_a+2 ε_p ], σM2=(2ϵ¯o+1)σ¯2+(2ϵ¯o+1)[2ϵ¯p+2(1+ϵ¯p)ϵ¯a]αtr(f(q)(q)∗(q)∗⊤). _M^2=(2 ε_o+1) σ^2+(2 ε_o+1) [2 ε_p+2(1+ ε_p) ε_a ] (H_f^(q)v^(q)^*v^(q)^* ). Proof. By (15), Lemma C.2 and Lemma C.4, this lemma can be proved by induction. ∎ Lemma C.6 (Bias-variance decomposition under general quantization, an upper bound). Under Assumption 3.1, 3.2, 3.3 and 3.4, if γ<1αtr(f(q))γ< 1 (H_f^(q)), it holds RN(0)/2≤1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t+t⟩,R_N^(0)/2≤ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t+C_t , where t=(ℐ−γ(q))∘t−1,0=[0⊗0], _t=(I- ^(q)) _t-1, _0=E [ η_0 η_0 ], t=(ℐ−γ(q))∘t−1+γ2σG2f(q),0=, _t=(I- ^(q)) _t-1+γ^2 _G^2H_f^(q), _0= 0, with σG2=σ¯2+suptαtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+supt([ϵt(a)2|at]+[ϵt(o)2|ot]). _G^2= σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ). Proof. By (15), Lemma C.3 and Lemma C.4, this lemma can be proved by induction. ∎ C.3 Variance Upper Bounds In this section, we derive upper bounds for 1N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩ 1N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t^(M) and 1N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t . C.3.1 General Quantization Lemma C.7 (A crude upper bound of variance under general quantization). Under Assumption 3.2, Assumption 3.3, if γ<1αtr(f(q))γ< 1 (H_f^(q)), t⪯γσG21−γαtr(f(q)).C_t γ _G^21-γ (H_f^(q))I. Proof. We prove by induction. For t=0t=0, we have 0=⪯γσG21−γαtr(f(q))C_0= 0 γ _G^21-γ (H_f^(q) )I. We then assume that t−1⪯γσG21−γαtr(f(q))C_t-1 γ _G^21-γ (H_f^(q) )I, and exam tC_t: t _t =(ℐ−γ(q))∘t−1+γ2σG2f(q) =(I- ^(q)) _t-1+γ^2 _G^2H_f^(q) =(ℐ−γf(q)⊗−γ⊗f(q))∘t−1+γ2ℳ(q)∘t−1+γ2σG2f(q) = (I- _f^(q) - _f^(q) ) _t-1+γ^2M^(q) _t-1+γ^2 _G^2H_f^(q) ⪯γσG21−γαtr(f(q))⋅(−2γf(q))+γ2γσG2αtr(f(q))1−γαtr(f(q))f(q)+γ2σG2f(q) γ _G^21-γ (H_f^(q) )· (I-2 _f^(q) )+ γ^2γ _G^2 (H_f^(q) )1-γ (H_f^(q) )H_f^(q)+γ^2 _G^2H_f^(q) =γσG21−γαtr(f(q))⋅−(2γ2−γ2)⋅σG21−γαtr(f(q))f(q) = γ _G^21-γ (H_f^(q) )·I-(2γ^2-γ^2)· _G^21-γ (H_f^(q) )H_f^(q) ⪯γσG21−γαtr(f(q))⋅, γ _G^21-γ (H_f^(q) )·I, where the first inequality holds by the induction assumption and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q) )H_f^(q). ∎ Lemma C.8 (A variance upper bound under general quantization). Under Assumption 3.2, Assumption 3.3, if γ<1αtr(f(q))γ< 1 (H_f^(q)), 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩≤σG21−γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2). 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t ≤ _G^21-γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ). where (λ~i(q))i=1M( λ_i^(q))_i=1^M are eigenvalues of f(q)H_f^(q) and k∗=maxk:λ~k(q)≥1Nγ.k^*= \k: λ_k^(q)≥ 1Nγ \. Proof. We first provide a refined upper bound for tC_t. Note that by definition t= _t= (ℐ−γ(q))∘t−1+γ2σG2f(q) (I- ^(q)) _t-1+γ^2 _G^2H_f^(q) (31) = = (ℐ−γ~(q))∘t−1+γ(~(q)−(q))∘t−1+γ2σG2f(q) (I-γ T^(q)) _t-1+γ( T^(q)-T^(q)) _t-1+γ^2 _G^2H_f^(q) = = (ℐ−γ~(q))∘t−1+γ2(ℳ(q)−ℳ~(q))∘t−1+γ2σG2f(q) (I-γ T^(q)) _t-1+γ^2(M^(q)- M^(q)) _t-1+γ^2 _G^2H_f^(q) ⪯ (ℐ−γ~(q))∘t−1+γ2ℳ(q)∘t−1+γ2σG2f(q), (I-γ T^(q)) _t-1+γ^2M^(q) _t-1+γ^2 _G^2H_f^(q), together with Lemma C.7 and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q))H_f^(q), it holds t⪯ _t (ℐ−γ~(q))∘t−1+γ2αtr(f(q))γσG21−γαtr(f(q))f(q)+γ2σG2f(q) (I-γ T^(q)) _t-1+ γ^2 (H_f^(q))γ _G^21-γ (H_f^(q))H_f^(q)+γ^2 _G^2H_f^(q) = = (ℐ−γ~(q))∘t−1+γ2σG21−γαtr(f(q))f(q). (I-γ T^(q)) _t-1+ γ^2 _G^21-γ (H_f^(q))H_f^(q). Solving recursion, it follows that t⪯ _t γ2σG21−γαtr(f(q))⋅∑k=0t−1(ℐ−γ~(q))k∘f(q) γ^2 _G^21-γ (H_f^(q))· _k=0^t-1(I-γ T^(q))^k _f^(q) (32) = = γ2σG21−γαtr(f(q))⋅∑k=0t−1(−γf(q))kf(q)(−γf(q))k γ^2 _G^21-γ (H_f^(q))· _k=0^t-1(I- _f^(q))^kH_f^(q)(I- _f^(q))^k ⪯ γ2σG21−γαtr(f(q))⋅∑k=0t−1(−γf(q))kf(q) γ^2 _G^21-γ (H_f^(q))· _k=0^t-1(I- _f^(q))^kH_f^(q) = = γσG21−γαtr(f(q))⋅(−(−γf(q))t). γ _G^21-γ (H_f^(q))· (I-(I- _f^(q))^t ). After providing a refined bound for tC_t, we are ready to bound the variance. 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t = = 1γN2∑t=0N−1⟨−(−γf(q))N−t,t⟩ 1γ N^2 _t=0^N-1 -(I- _f^(q))^N-t,C_t ≤ ≤ 1γ2N2γ2σG21−γαtr(f(q))∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ 1γ^2N^2 γ^2 _G^21-γ (H_f^(q)) _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t = = 1γ2N2γ2σG21−γαtr(f(q))∑i∑t=0N−1[1−(1−γλ~i(q))N−t][1−(1−γλ~i(q))t] 1γ^2N^2 γ^2 _G^21-γ (H_f^(q)) _i _t=0^N-1 [1-(1-γ λ_i^(q))^N-t ] [1-(1-γ λ_i^(q))^t ] ≤ ≤ 1γ2N2γ2σG21−γαtr(f(q))∑i∑t=0N−1[1−(1−γλ~i(q))N][1−(1−γλ~i(q))N] 1γ^2N^2 γ^2 _G^21-γ (H_f^(q)) _i _t=0^N-1 [1-(1-γ λ_i^(q))^N ] [1-(1-γ λ_i^(q))^N ] ≤ ≤ 1γ2Nγ2σG21−γαtr(f(q))∑imin1,γ2N2(λ~i(q))2 1γ^2N γ^2 _G^21-γ (H_f^(q)) _i \1,γ^2N^2( λ_i^(q))^2 \ ≤ ≤ σG21−γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2), _G^21-γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ), where (λ~i(q))i=1M( λ_i^(q))_i=1^M are eigenvalues of f(q)H_f^(q) and k∗=maxk:λ~k(q)≥1Nγ.k^*= \k: λ_k^(q)≥ 1Nγ \. ∎ C.3.2 Multiplicative Quantization Lemma C.9 (A crude upper bound of variance under multiplicative quantization). Under Assumption 3.2, Assumption 3.3, if γ<1(1+ϵ~)αtr(f(q))γ< 1(1+ ε) (H_f^(q)), t(M)⪯γσM21−γ(1+ϵ~)αtr(f(q)).C_t^(M) γ _M^21-γ(1+ ε) (H_f^(q) )I. Proof. We prove by induction. For t=0t=0, we have 0(M)=⪯γσM21−γ(1+ϵ~)αtr(f(q))C_0^(M)= 0 γ _M^21-γ(1+ ε) (H_f^(q) )I. We then assume that t−1(M)⪯γσM21−γ(1+ϵ~)αtr(f(q))C_t-1^(M) γ _M^21-γ(1+ ε) (H_f^(q) )I, and exam t(M)C_t^(M): t(M) _t^(M) =(ℐ−γ(q)+ϵ~γ2ℳ(q))∘t−1(M)+γ2σM2f(q) =(I- ^(q)+ εγ^2M^(q)) _t-1^(M)+γ^2 _M^2H_f^(q) =(ℐ−γf(q)⊗−γ⊗f(q))∘t−1(M)+(1+ϵ~)γ2ℳ(q)∘t−1(M)+γ2σM2f(q) = (I- _f^(q) - _f^(q) ) _t-1^(M)+(1+ ε)γ^2M^(q) _t-1^(M)+γ^2 _M^2H_f^(q) ⪯γσM21−γ(1+ϵ~)αtr(f(q))⋅(−2γf(q))+((1+ϵ~)γ3σM2αtr(f(q))1−γ(1+ϵ~)αtr(f(q))+γ2σM2)f(q) γ _M^21-γ(1+ ε) (H_f^(q) )· (I-2 _f^(q) )+ ( (1+ ε)γ^3 _M^2 (H_f^(q) )1-γ(1+ ε) (H_f^(q) )+γ^2 _M^2 )H_f^(q) =γσM21−γ(1+ϵ~)αtr(f(q))⋅−(2γ2−γ2)⋅σM21−γ(1+ϵ~)αtr(f(q))f(q) = γ _M^21-γ(1+ ε) (H_f^(q) )·I-(2γ^2-γ^2)· _M^21-γ(1+ ε) (H_f^(q) )H_f^(q) ⪯γσM21−γ(1+ϵ~)αtr(f(q))⋅, γ _M^21-γ(1+ ε) (H_f^(q) )·I, where the first inequality holds by the induction assumption and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q) )H_f^(q). ∎ Lemma C.10 (A variance upper bound under multiplicative quantization). Under Assumption 3.2, Assumption 3.3, if γ<1(1+ϵ~)αtr(f(q))γ< 1(1+ ε) (H_f^(q)), 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩≤σM21−(1+ϵ~)γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2), 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t^(M) ≤ _M^21-(1+ ε)γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ), where k∗=maxk:λ~k(q)≥1Nγk^*= \k: λ_k^(q)≥ 1Nγ \, and (λ~i(q))i=1M( λ_i^(q))_i=1^M are eigenvalues of f(q)H_f^(q). Proof. We first provide a refined bound for t(M)C_t^(M). By the definition of t(M)C_t^(M), t(M)= _t^(M)= (ℐ−γ(q)+ϵ~γ2ℳ(q))∘t−1(M)+γ2σM2f(q) (I- ^(q)+ εγ^2M^(q)) _t-1^(M)+γ^2 _M^2H_f^(q) ⪯ (ℐ−γ~(q))∘t−1(M)+γ2(1+ϵ~)ℳ(q)∘t−1(M)+γ2σM2f(q) (I-γ T^(q)) _t-1^(M)+γ^2(1+ ε)M^(q) _t-1^(M)+γ^2 _M^2H_f^(q) ⪯ (ℐ−γ~(q))∘t−1(M)+γ2(1+ϵ~)γσM2αtr(f(q))1−γ(1+ϵ~)αtr(f(q))f(q)+γ2σM2f(q) (I-γ T^(q)) _t-1^(M)+γ^2(1+ ε) γ _M^2 (H_f^(q) )1-γ(1+ ε) (H_f^(q) )H_f^(q)+γ^2 _M^2H_f^(q) = = (ℐ−γ~(q))∘t−1(M)+γ2σM21−γ(1+ϵ~)αtr(f(q))f(q), (I-γ T^(q)) _t-1^(M)+ γ^2 _M^21-γ(1+ ε) (H_f^(q) )H_f^(q), where the second inequality holds by Lemma C.9 and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q) )H_f^(q). Solving the recursion yields t(M)⪯γσM21−(1+ϵ~)γαtr(f(q))⋅(−(−γf(q))t). _t^(M) γ _M^21-(1+ ε)γ (H_f^(q))· (I-(I- _f^(q))^t ). After providing a refined bound for t(M)C_t^(M), we are ready to bound the variance. 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),C_t^(M) = = 1γN2∑t=0N−1⟨−(−γf(q))N−t,t(M)⟩ 1γ N^2 _t=0^N-1 -(I- _f^(q))^N-t,C_t^(M) ≤ ≤ 1γ2N2γ2σM21−(1+ϵ~)γαtr(f(q))∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ 1γ^2N^2 γ^2 _M^21-(1+ ε)γ (H_f^(q)) _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t = = 1γ2N2γ2σM21−(1+ϵ~)γαtr(f(q))∑i∑t=0N−1[1−(1−γλ~i(q))N−t][1−(1−γλ~i(q))t] 1γ^2N^2 γ^2 _M^21-(1+ ε)γ (H_f^(q)) _i _t=0^N-1 [1-(1-γ λ_i^(q))^N-t ] [1-(1-γ λ_i^(q))^t ] ≤ ≤ 1γ2N2γ2σM21−(1+ϵ~)γαtr(f(q))∑i∑t=0N−1[1−(1−γλ~i(q))N][1−(1−γλ~i(q))N] 1γ^2N^2 γ^2 _M^21-(1+ ε)γ (H_f^(q)) _i _t=0^N-1 [1-(1-γ λ_i^(q))^N ] [1-(1-γ λ_i^(q))^N ] = = 1γ2Nγ2σM21−(1+ϵ~)γαtr(f(q))∑i[1−(1−γλ~i(q))N]2 1γ^2N γ^2 _M^21-(1+ ε)γ (H_f^(q)) _i [1-(1-γ λ_i^(q))^N ]^2 ≤ ≤ 1γ2Nγ2σM21−(1+ϵ~)γαtr(f(q))∑imin1,γ2N2(λ~i(q))2 1γ^2N γ^2 _M^21-(1+ ε)γ (H_f^(q)) _i \1,γ^2N^2( λ_i^(q))^2 \ ≤ ≤ 1γ2Nγ2σM21−(1+ϵ~)γαtr(f(q))(k∗+N2γ2⋅∑i>k∗(λ~i(q))2) 1γ^2N γ^2 _M^21-(1+ ε)γ (H_f^(q)) (k^*+N^2γ^2· _i>k^*( λ_i^(q))^2 ) = = σM21−(1+ϵ~)γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2), _M^21-(1+ ε)γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ), where (λ~i(q))i=1M( λ_i^(q))_i=1^M are eigenvalues of f(q)H_f^(q) and k∗=maxk:λ~k(q)≥1Nγ.k^*= \k: λ_k^(q)≥ 1Nγ \. ∎ C.4 Bias Upper Bounds C.4.1 General Quantization Let n=∑t=0n−1tS_n= _t=0^n-1B_t. Lemma C.11 (Initial Study of tS_t). For 1≤t≤N1≤ t≤ N, t⪯(ℐ−γ~(q))∘t−1+γ2ℳ(q)∘N+0.S_t (I-γ T^(q)) _t-1+γ^2M^(q) _N+B_0. Proof. By definition, t= _t= ∑k=0t−1(ℐ−γ(q))k∘0 _k=0^t-1(I- ^(q))^k _0 (33) = = (ℐ−γ(q))∘(∑k=1t−1(ℐ−γ(q))k−1∘0)+0 (I- ^(q)) ( _k=1^t-1(I- ^(q))^k-1 _0 )+B_0 = = (ℐ−γ(q))∘t−1+0. (I- ^(q)) _t-1+B_0. Then we convert (q)T^(q) to ~(q) T^(q). By (33), t= _t= (ℐ−γ(q))∘t−1+0 (I- ^(q)) _t-1+B_0 = = (ℐ−γ~(q))∘t−1+γ(~(q)−(q))∘t−1+0 (I-γ T^(q)) _t-1+γ( T^(q)-T^(q)) _t-1+B_0 = = (ℐ−γ~(q))∘t−1+γ2(ℳ(q)−ℳ~(q))∘t−1+0 (I-γ T^(q)) _t-1+γ^2(M^(q)- M^(q)) _t-1+B_0 ⪯ (ℐ−γ~(q))∘t−1+γ2ℳ(q)∘N+0, (I-γ T^(q)) _t-1+γ^2M^(q) _N+B_0, where the third equality holds by the definition of linear operators. ∎ Lemma C.12 (A Bound for ℳ(q)∘tM^(q) _t). For 1≤t≤N1≤ t≤ N, under Assumption 3.2, Assumption 3.3, if γ<1αtr(f(q))γ< 1 (H_f^(q)), then ℳ(q)∘t⪯α⋅tr([ℐ−(ℐ−γ~(q))t]∘0)γ(1−γαtr(f(q)))⋅f(q).M^(q) _t α·tr ( [I-(I-γ T^(q))^t ] _0 )γ(1-γ (H_f^(q)))·H_f^(q). Proof. The first step is to derive a crude bound for tS_t. Take summation via the update rule, we have t=∑k=0t−1(ℐ−γ(q))k∘0=γ−1(q)−1∘[ℐ−(ℐ−γ(q))t]∘0.S_t= _k=0^t-1(I- ^(q))^k _0=γ^-1T^(q)^-1 [I-(I- ^(q))^t ] _0. Note that ℐ−γ~(q)⪯ℐ−γ(q),(ℐ−(ℐ−γ(q))t)⪯(ℐ−(ℐ−γ~(q))t),I-γ T^(q) - ^(q), (I-(I- ^(q))^t) (I-(I-γ T^(q))^t), and further note that (q)−1T^(q)^-1 is a PSD mapping 666(q)−1T^(q)^-1 is a PSD mapping under the condition that γ<1αtr((q))γ< 1 (H^(q)), which can be directly deduced by Lemma B.1 in Zou et al. (2021). We omit the proof here for simplicity., and [ℐ−(ℐ−γ~(q))t]∘0[I-(I-γ T^(q))^t] _0 is a PSD matrix, we obtain t⪯γ−1(q)−1∘(ℐ−(ℐ−γ~(q))t)∘0.S_t γ^-1T^(q)^-1 (I-(I-γ T^(q))^t) _0. For simplicity, we denote :=(ℐ−(ℐ−γ~(q))t)∘0A:=(I-(I-γ T^(q))^t) _0. We then tackle (q)−1∘T^(q)^-1 . To be specific, we apply ~(q) T^(q). ~(q)∘(q)−1∘ T^(q) T^(q)^-1 =γℳ(q)∘(q)−1∘+−γf(q)((q)−1∘)f(q) = ^(q) T^(q)^-1 +A- _f^(q)(T^(q)^-1 )H_f^(q) ⪯γℳ(q)∘(q)−1∘+. ^(q) T^(q)^-1 +A. Therefore, (q)−1∘⪯γ(~(q))−1∘ℳ(q)∘(q)−1∘+(~(q))−1∘. T^(q)^-1 γ( T^(q))^-1 ^(q) T^(q)^-1 +( T^(q))^-1 . Then we undertake the second step, applying ℳ(q)M^(q) on both sides. ℳ(q)∘((q)−1∘) ^(q) (T^(q)^-1 ) ⪯ℳ(q)∘γ(~(q))−1∘ℳ(q)∘(q)−1∘+ℳ(q)∘(~(q))−1∘ ^(q) γ( T^(q))^-1 ^(q) T^(q)^-1 +M^(q) ( T^(q))^-1 (34) ⪯∑t=0∞(γℳ(q)∘(~(q))−1)t∘(ℳ(q)∘(~(q))−1∘)(By recursion). _t=0^∞( ^(q) ( T^(q))^-1)^t (M^(q) ( T^(q))^-1 )\ (By recursion). By Assumption 3.3, ℳ(q)∘(~(q))−1∘ M^(q) ( T^(q))^-1 ⪯αtr(f(q)(~(q))−1∘)f(q) (H_f^(q)( T^(q))^-1 )H_f^(q) =αγtr(∑t=0∞f(q)(−γf(q))t(−γf(q))t)f(q) =α ( _t=0^∞H_f^(q)(I- _f^(q))^tA(I- _f^(q))^t )H_f^(q) =αtr(f(q)(2f(q)−γ(f(q))2)−1)f(q) = (H_f^(q)(2H_f^(q)-γ(H_f^(q))^2)^-1A )H_f^(q) ⪯αtr()f(q), (A)H_f^(q), where the first equality holds by the definition of ~(q) T^(q) and the last inequality requires the condition that γ<1αtr(f(q))γ< 1 (H_f^(q)). Hence, by (34), and further by (~(q))−1f(q)⪯( T^(q))^-1H_f^(q) and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q))H_f^(q), we obtain ℳ(q)∘((q)−1∘) ^(q) (T^(q)^-1 ) ⪯∑t=0∞(γℳ(q)∘(~(q))−1)t∘(ℳ(q)∘(~(q))−1∘) _t=0^∞( ^(q) ( T^(q))^-1)^t (M^(q) ( T^(q))^-1 ) ⪯αtr()∑t=0∞(γαtr(f(q)))tf(q) (A) _t=0^∞(γ (H_f^(q)))^tH_f^(q) ⪯αtr()1−γαtr(f(q))⋅f(q). (A)1-γ (H_f^(q))·H_f^(q). Therefore, ℳ(q)∘t⪯γ−1αtr()1−γαtr(f(q))⋅f(q)=α⋅tr([ℐ−(ℐ−γ~(q))t]∘0)γ(1−γαtr(f(q)))⋅f(q). ^(q) _t γ^-1 (A)1-γ (H_f^(q))·H_f^(q)= α·tr ( [I-(I-γ T^(q))^t ] _0 )γ(1-γ (H_f^(q)))·H_f^(q). ∎ Lemma C.13 (A bias upper bound under general quantization). Under Assumption 3.2, Assumption 3.3, if the stepsize satisfies γ<1αtr(f(q))γ< 1 (H_f^(q)), then 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t ≤ ≤ 2α(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2)Nγ(1−γαtr(f(q)))⋅(k∗N+Nγ2∑i>k∗(λ~i(q))2) 2α (\|v^(q)^*\|_I_f,0:k^*^(q)^2+Nγ\|v^(q)^*\|_H_f,k^*:∞^(q)^2 )Nγ(1-γ (H_f^(q)))· ( k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 ) + + 1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2. 1γ^2N^2·\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+\|v^(q)^*\|_H_f,k^*:∞^(q)^2. Proof. Recalling Lemma C.11, we can derive a refined upper bound for tS_t by Lemma C.12: t⪯ _t (ℐ−γ~(q))∘t−1+γ2ℳ(q)∘N+0 (I-γ T^(q)) _t-1+γ^2M^(q) _N+B_0 (35) ⪯ (ℐ−γ~(q))∘t−1+γα⋅tr([ℐ−(ℐ−γ~(q))N]∘0)1−γαtr(f(q))⋅f(q)+0 (I-γ T^(q)) _t-1+ γα·tr ( [I-(I-γ T^(q))^N ] _0 )1-γ (H_f^(q))·H_f^(q)+B_0 = = ∑k=0t−1(ℐ−γ~(q))k(γα⋅tr([ℐ−(ℐ−γ~(q))N]∘0)1−γαtr(f(q))⋅f(q)+0) _k=0^t-1(I-γ T^(q))^k ( γα·tr ( [I-(I-γ T^(q))^N ] _0 )1-γ (H_f^(q))·H_f^(q)+B_0 ) = = ∑k=0t−1(−γf(q))k(γα⋅tr(0−(−γf(q))N0(−γf(q))N)1−γαtr(f(q))⋅f(q)+0)(−γf(q))k. _k=0^t-1(I- _f^(q))^k ( γα·tr (B_0-(I- _f^(q))^NB_0(I- _f^(q))^N )1-γ (H_f^(q))·H_f^(q)+B_0 )(I- _f^(q))^k. Before providing our upper bound for the bias error, we denote a,b:=a−(−γf(q))b−aa(−γf(q))b−a.B_a,b:=B_a-(I- _f^(q))^b-aB_a(I- _f^(q))^b-a. Then by (35), 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t = = 1γN2∑t=0N−1⟨−(−γf(q))N−t,t⟩ 1γ N^2 _t=0^N-1 -(I- _f^(q))^N-t,B_t ≤ ≤ 1γN2⟨−(−γf(q))N,∑t=0N−1t⟩ 1γ N^2 -(I- _f^(q))^N, _t=0^N-1B_t ≤ ≤ 1γN2∑k=0N−1⟨−(−γf(q))N,(−γf(q))k(γα⋅tr(0,N)1−γαtr(f(q))⋅f(q)+0)(−γf(q))k⟩ 1γ N^2 _k=0^N-1 -(I- _f^(q))^N,(I- _f^(q))^k ( γα·tr (B_0,N )1-γ (H_f^(q))·H_f^(q)+B_0 )(I- _f^(q))^k = = 1γN2∑k=0N−1⟨(−γf(q))2k−(−γf(q))N+2k,(γα⋅tr(0,N)1−γαtr(f(q))⋅f(q)+0)⟩. 1γ N^2 _k=0^N-1 (I- _f^(q))^2k-(I- _f^(q))^N+2k, ( γα·tr (B_0,N )1-γ (H_f^(q))·H_f^(q)+B_0 ) . Note that (−γf(q))2k−(−γf(q))N+2k (I- _f^(q))^2k-(I- _f^(q))^N+2k =(−γf(q))k((−γf(q))k−(−γf(q))N+k) = (I- _f^(q) )^k ( (I- _f^(q) )^k- (I- _f^(q) )^N+k ) ⪯(−γf(q))k−(−γf(q))N+k, (I- _f^(q))^k-(I- _f^(q))^N+k, we obtain 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t ≤ ≤ 1γN2∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,γα⋅tr(0,N)1−γαtr(f(q))⋅f(q)+0⟩. 1γ N^2 _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k, γα·tr (B_0,N )1-γ (H_f^(q))·H_f^(q)+B_0 . Therefore, it suffices to upper bound the following two terms I1=αtr(0,N)N2(1−γαtr(f(q)))∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,f(q)⟩, I_1= (B_0,N)N^2(1-γ (H_f^(q))) _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k,H_f^(q) , I2=1γN2∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,0⟩. I_2= 1γ N^2 _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k,B_0 . Regarding I1I_1, since f(q)H_f^(q) and −γf(q)I- _f^(q) can be diagonalized simultaneously, I1 I_1 =αtr(0,N)N2(1−γαtr(f(q)))∑k=0N−1∑i[(1−γλ~i(q))k−(1−γλ~i(q))N+k]λ~i(q) = (B_0,N)N^2(1-γ (H_f^(q))) _k=0^N-1 _i [(1-γ λ_i^(q))^k-(1-γ λ_i^(q))^N+k ] λ_i^(q) =αtr(0,N)γN2(1−γαtr(f(q)))∑i[1−(1−γλ~i(q))N]2 = (B_0,N)γ N^2(1-γ (H_f^(q))) _i [1-(1-γ λ_i^(q))^N ]^2 ≤αtr(0,N)γN2(1−γαtr(f(q)))∑imin1,γ2N2(λ~i(q))2 ≤ (B_0,N)γ N^2(1-γ (H_f^(q))) _i \1,γ^2N^2( λ_i^(q))^2 \ ≤αtr(0,N)γ(1−γαtr(f(q)))⋅(k∗N2+γ2∑i>k∗(λ~i(q))2), ≤ (B_0,N)γ(1-γ (H_f^(q)))· ( k^*N^2+γ^2 _i>k^*( λ_i^(q))^2 ), where k∗=maxk:λ~k(q)≥1Nγk^*= \k: λ_k^(q)≥ 1Nγ\ and λ~i(q)i=1M\ λ_i^(q)\_i=1^M are eigenvalues of f(q)H_f^(q). Then we tackle tr(0,N)tr(B_0,N). tr(0,N) (B_0,N) =tr(0−(−γf(q))N0(−γf(q))N) =tr (B_0-(I- _f^(q))^NB_0(I- _f^(q))^N ) (36) =∑i(1−(1−γλ~i(q))2N)⋅(⟨0−(q)∗,i(q)⟩)2 = _i (1-(1-γ λ_i^(q))^2N )· ( _0-v^(q)^*,v_i^(q) )^2 ≤2∑imin1,Nγλ~i(q)(⟨0−(q)∗,i(q)⟩)2 ≤ 2 _i \1,Nγ λ_i^(q)\ ( _0-v^(q)^*,v_i^(q) )^2 ≤2(‖0−(q)∗‖f,0:k∗(q)2+Nγ‖0−(q)∗‖f,k∗:∞(q)2). ≤ 2 (\|v_0-v^(q)^*\|_I_f,0:k^*^(q)^2+Nγ\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2 ). Hence, I1≤2α(‖0−(q)∗‖0:k∗(q)2+Nγ‖0−(q)∗‖k∗:∞(q)2)Nγ(1−γαtr(f(q)))⋅(k∗N+Nγ2∑i>k∗(λ~i(q))2).I_1≤ 2α (\|v_0-v^(q)^*\|_I_0:k^*^(q)^2+Nγ\|v_0-v^(q)^*\|_H_k^*:∞^(q)^2 )Nγ(1-γ (H_f^(q)))· ( k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 ). Regarding I2I_2, decompose f(q)=(q)(q)(q)⊤H_f^(q)=V^(q) ^(q)V^(q) , then I2=1γN2∑k=0N−1⟨(−γ(q))k−(−γ(q))N+k,(q)⊤0(q)⟩.I_2= 1γ N^2 _k=0^N-1 (I-γ ^(q))^k-(I-γ ^(q))^N+k,V^(q) B_0V^(q) . Note that 0=00⊤B_0= η_0 η_0 , it can be shown that the diagonal entries of (q)⊤0(q)V^(q) B_0V^(q) are ω12,… _1^2,…, where ωi=i(q)⊤0=i(q)⊤(0−(q)∗) _i=v_i^(q) η_0=v_i^(q) (v_0-v^(q)^*). Hence, I2 I_2 =1γN2∑k=0N−1∑i[(1−γλ~i(q))k−(1−γλ~i(q))N+k]ωi2 = 1γ N^2 _k=0^N-1 _i [(1-γ λ_i^(q))^k-(1-γ λ_i^(q))^N+k ] _i^2 =1γ2N2∑iωi2λ~i(q)[1−(1−γλ~i(q))N]2 = 1γ^2N^2 _i _i^2 λ_i^(q) [1-(1-γ λ_i^(q))^N ]^2 ≤1γ2N2∑iωi2λ~i(q)min1,γ2N2(λ~i(q))2 ≤ 1γ^2N^2 _i _i^2 λ_i^(q) *min \1,γ^2N^2( λ_i^(q))^2 \ ≤1γ2N2⋅∑i≤k∗ωi2λ~i(q)+∑i>k∗λ~i(q)ωi2 ≤ 1γ^2N^2· _i≤ k^* _i^2 λ_i^(q)+ _i>k^* λ_i^(q) _i^2 =1γ2N2⋅‖0−(q)∗‖(f,0:k∗(q))−12+‖0−(q)∗‖f,k∗:∞(q)2. = 1γ^2N^2·\|v_0-v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2. In conclusion, if the stepsize satisfies γ<1αtr(f(q))γ< 1 (H_f^(q)), 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t ≤ ≤ 2α(‖0−(q)∗‖f,0:k∗(q)2+Nγ‖0−(q)∗‖f,k∗:∞(q)2)Nγ(1−γαtr(f(q)))⋅(k∗N+Nγ2∑i>k∗(λ~i(q))2) 2α (\|v_0-v^(q)^*\|_I_f,0:k^*^(q)^2+Nγ\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2 )Nγ(1-γ (H_f^(q)))· ( k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 ) + + 1γ2N2⋅‖0−(q)∗‖(f,0:k∗(q))−12+‖0−(q)∗‖f,k∗:∞(q)2. 1γ^2N^2·\|v_0-v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2. Applying 0=v_0= 0 completes the proof. ∎ C.4.2 Multiplicative Quantization Let n(M)=∑t=0n−1t(M)S_n^(M)= _t=0^n-1B_t^(M). Lemma C.14 (Initial Study of t(M)S_t^(M)). For 1≤t≤N1≤ t≤ N, t(M)⪯(ℐ−γ~(q))∘t−1(M)+(1+ϵ~)γ2ℳ(q)∘N(M)+0.S_t^(M) (I-γ T^(q)) _t-1^(M)+(1+ ε)γ^2M^(q) _N^(M)+B_0. Proof. The proof is similar to the proof for Lemma C.11. t(M)= _t^(M)= (ℐ−γ(q)+ϵ~γ2ℳ(q))∘t−1(M)+0 (I- ^(q)+ εγ^2M^(q)) _t-1^(M)+B_0 = = (ℐ−γ~(q))∘t−1+γ(~(q)−(q))∘t−1(M)+ϵ~γ2ℳ(q)∘t−1(M)+0 (I-γ T^(q)) _t-1+γ( T^(q)-T^(q)) _t-1^(M)+ εγ^2M^(q) _t-1^(M)+B_0 = = (ℐ−γ~(q))∘t−1(M)+γ2((1+ϵ~)ℳ(q)−ℳ~(q))∘t−1(M)+0 (I-γ T^(q)) _t-1^(M)+γ^2((1+ ε)M^(q)- M^(q)) _t-1^(M)+B_0 ⪯ (ℐ−γ~(q))∘t−1(M)+(1+ϵ~)γ2ℳ(q)∘N(M)+0. (I-γ T^(q)) _t-1^(M)+(1+ ε)γ^2M^(q) _N^(M)+B_0. ∎ Lemma C.15 (A Bound for ℳ(q)∘t(M)M^(q) _t^(M)). For 1≤t≤N1≤ t≤ N, under Assumption 3.2, Assumption 3.3, if γ<1(1+ϵ~)αtr(f(q))γ< 1(1+ ε) (H_f^(q)), ℳ(q)∘t(M)⪯α⋅tr([ℐ−(ℐ−γ~(q))t]∘0)γ(1−(1+ϵ~)γαtr(f(q)))⋅f(q).M^(q) _t^(M) α·tr ( [I-(I-γ T^(q))^t ] _0 )γ(1-(1+ ε)γ (H_f^(q)))·H_f^(q). Proof. The first step is to derive a crude bound for t(M)S_t^(M). Take summation via the update rule, we have 777((q)−ϵ~γℳ(q))−1(T^(q)- ε ^(q))^-1 is a PSD mapping under the condition that γ<1(1+ϵ~)αtr(f(q))γ< 1(1+ ε) (H_f^(q)), which can be directly deduced by Lemma B.1 in Zou et al. (2021). We omit the proof here for simplicity. t(M)=∑k=0t−1(ℐ−γ(q)+ϵ~γ2ℳ(q))k∘0=γ−1((q)−ϵ~γℳ(q))−1∘[ℐ−(ℐ−γ(q)+ϵ~γ2ℳ(q))t]∘0.S_t^(M)= _k=0^t-1(I- ^(q)+ εγ^2M^(q))^k _0=γ^-1(T^(q)- ε ^(q))^-1 [I-(I- ^(q)+ εγ^2M^(q))^t ] _0. Note that ℐ−γ~(q)⪯ℐ−γ(q),(ℐ−(ℐ−γ(q)+ϵ~γ2ℳ(q))t)⪯(ℐ−(ℐ−γ~(q)+ϵ~γ2ℳ(q))t),I-γ T^(q) - ^(q), (I-(I- ^(q)+ εγ^2M^(q))^t) (I-(I-γ T^(q)+ εγ^2M^(q))^t), we obtain t(M)⪯γ−1((q)−ϵ~γℳ(q))−1∘(ℐ−(ℐ−γ~(q)+ϵ~γ2ℳ(q))t)∘0.S_t^(M) γ^-1(T^(q)- ε ^(q))^-1 (I-(I-γ T^(q)+ εγ^2M^(q))^t) _0. Denote :=(ℐ−(ℐ−γ~(q)+ϵ~γ2ℳ(q))t)∘0A:=(I-(I-γ T^(q)+ εγ^2M^(q))^t) _0, then ~(q)∘((q)−ϵ~γℳ(q))−1∘ T^(q) (T^(q)- ε ^(q))^-1 ⪯(1+ϵ~)γℳ(q)∘((q)−ϵ~γℳ(q))−1∘+. (1+ ε) ^(q) (T^(q)- ε ^(q))^-1 +A. Therefore ((q)−ϵ~γℳ(q))−1∘⪯(1+ϵ~)γ(~(q))−1∘ℳ(q)∘((q)−ϵ~γℳ(q))−1∘+(~(q))−1∘. (T^(q)- ε ^(q))^-1 (1+ ε)γ( T^(q))^-1 ^(q) (T^(q)- ε ^(q))^-1 +( T^(q))^-1 . Then we undertake the second step, applying ℳ(q)M^(q) on both sides. ℳ(q)∘((q)−ϵ~γℳ(q))−1∘⪯∑t=0∞((1+ϵ~)γℳ(q)∘(~(q))−1)t∘(ℳ(q)∘(~(q))−1∘).M^(q) (T^(q)- ε ^(q))^-1 _t=0^∞((1+ ε) ^(q) ( T^(q))^-1)^t (M^(q) ( T^(q))^-1 ). (37) By Assumption 3.3, ℳ(q)∘(~(q))−1∘ M^(q) ( T^(q))^-1 ⪯αtr(f(q)(~(q))−1∘)f(q) (H_f^(q)( T^(q))^-1 )H_f^(q) (38) =αγtr(∑t=0∞f(q)(−γf(q))t(−γf(q))t)f(q) =α ( _t=0^∞H_f^(q)(I- _f^(q))^tA(I- _f^(q))^t )H_f^(q) =αtr(f(q)(2f(q)−γ(f(q))2)−1)f(q) = (H_f^(q)(2H_f^(q)-γ(H_f^(q))^2)^-1A )H_f^(q) ⪯αtr()f(q), (A)H_f^(q), where the last inequality requires the condition that γ<1αtr(f(q))γ< 1 (H_f^(q)). Hence, by (37), (38), and further by (~(q))−1f(q)⪯( T^(q))^-1H_f^(q) and ℳ(q)∘⪯αtr(f(q))f(q)M^(q) (H_f^(q))H_f^(q), we obtain ℳ(q)∘(((q)−ϵ~γℳ(q))−1∘) ^(q) ((T^(q)- ε ^(q))^-1 ) ⪯∑t=0∞((1+ϵ~)γℳ(q)∘(~(q))−1)t∘(ℳ(q)∘(~(q))−1∘) _t=0^∞((1+ ε) ^(q) ( T^(q))^-1)^t (M^(q) ( T^(q))^-1 ) ⪯αtr()∑t=0∞((1+ϵ~)γαtr(f(q)))tf(q) (A) _t=0^∞((1+ ε)γ (H_f^(q)))^tH_f^(q) ⪯αtr()1−(1+ϵ~)γαtr(f(q))⋅f(q). (A)1-(1+ ε)γ (H_f^(q))·H_f^(q). Therefore, ℳ(q)∘t(M)⪯γ−1αtr()1−(1+ϵ~)γαtr(f(q))⋅f(q)⪯α⋅tr([ℐ−(ℐ−γ~(q))t]∘0)γ(1−(1+ϵ~)γαtr(f(q)))⋅f(q). ^(q) _t^(M) γ^-1 (A)1-(1+ ε)γ (H_f^(q))·H_f^(q) α·tr ( [I-(I-γ T^(q))^t ] _0 )γ(1-(1+ ε)γ (H_f^(q)))·H_f^(q). ∎ Lemma C.16 (A bias upper bound under multiplicative quantization). Under Assumption 3.2, Assumption 3.3, if the stepsize satisfies γ<1αtr(f(q))γ< 1 (H_f^(q)), then 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t^(M) ≤ ≤ 2(1+ϵ~)α(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2)Nγ(1−(1+ϵ~)γαtr(f(q)))⋅(k∗N+Nγ2∑i>k∗(λ~i(q))2) 2(1+ ε)α (\|v^(q)^*\|_I_f,0:k^*^(q)^2+Nγ\|v^(q)^*\|_H_f,k^*:∞^(q)^2 )Nγ(1-(1+ ε)γ (H_f^(q)))· ( k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 ) + + 1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2. 1γ^2N^2·\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+\|v^(q)^*\|_H_f,k^*:∞^(q)^2. Proof. Recalling Lemma C.14, we can derive a refined upper bound for tS_t by Lemma C.15: t(M)⪯ _t^(M) (ℐ−γ~(q))∘t−1(M)+(1+ϵ~)γ2ℳ(q)∘N(M)+0 (I-γ T^(q)) _t-1^(M)+(1+ ε)γ^2M^(q) _N^(M)+B_0 (39) ⪯ (ℐ−γ~(q))∘t−1(M)+(1+ϵ~)γα⋅tr([ℐ−(ℐ−γ~(q))N]∘0)(1−(1+ϵ~)γαtr(f(q)))⋅f(q)+0 (I-γ T^(q)) _t-1^(M)+ (1+ ε)γα·tr ( [I-(I-γ T^(q))^N ] _0 )(1-(1+ ε)γ (H_f^(q)))·H_f^(q)+B_0 = = ∑k=0t−1(ℐ−γ~(q))k((1+ϵ~)γα⋅tr([ℐ−(ℐ−γ~(q))N]∘0)(1−(1+ϵ~)γαtr(f(q)))⋅f(q)+0) _k=0^t-1(I-γ T^(q))^k ( (1+ ε)γα·tr ( [I-(I-γ T^(q))^N ] _0 )(1-(1+ ε)γ (H_f^(q)))·H_f^(q)+B_0 ) = = ∑k=0t−1(−γf(q))k((1+ϵ~)γα⋅tr(0−(−γf(q))N0(−γf(q))N)(1−(1+ϵ~)γαtr(f(q)))⋅f(q)+0)(−γf(q))k. _k=0^t-1(I- _f^(q))^k ( (1+ ε)γα·tr (B_0-(I- _f^(q))^NB_0(I- _f^(q))^N )(1-(1+ ε)γ (H_f^(q)))·H_f^(q)+B_0 )(I- _f^(q))^k. Before providing our upper bound for the bias error, we denote a,b:=a−(−γf(q))b−aa(−γf(q))b−a.B_a,b:=B_a-(I- _f^(q))^b-aB_a(I- _f^(q))^b-a. Then by (39), 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t^(M) = = 1γN2∑t=0N−1⟨−(−γf(q))N−t,t(M)⟩ 1γ N^2 _t=0^N-1 -(I- _f^(q))^N-t,B_t^(M) ≤ ≤ 1γN2⟨−(−γf(q))N,∑t=0N−1t(M)⟩ 1γ N^2 -(I- _f^(q))^N, _t=0^N-1B_t^(M) ≤ ≤ 1γN2∑k=0N−1⟨−(−γf(q))N,(−γf(q))k((1+ϵ~)γα⋅tr(0,N)1−(1+ϵ~)γαtr(f(q))⋅f(q)+0)(−γf(q))k⟩ 1γ N^2 _k=0^N-1 -(I- _f^(q))^N,(I- _f^(q))^k ( (1+ ε)γα·tr (B_0,N )1-(1+ ε)γ (H_f^(q))·H_f^(q)+B_0 )(I- _f^(q))^k = = 1γN2∑k=0N−1⟨(−γf(q))2k−(−γf(q))N+2k,((1+ϵ~)γα⋅tr(0,N)1−(1+ϵ~)γαtr(f(q))⋅f(q)+0)⟩. 1γ N^2 _k=0^N-1 (I- _f^(q))^2k-(I- _f^(q))^N+2k, ( (1+ ε)γα·tr (B_0,N )1-(1+ ε)γ (H_f^(q))·H_f^(q)+B_0 ) . Note that (−γf(q))2k−(−γf(q))N+2k (I- _f^(q))^2k-(I- _f^(q))^N+2k =(−γf(q))k((−γf(q))k−(−γf(q))N+k) = (I- _f^(q) )^k ( (I- _f^(q) )^k- (I- _f^(q) )^N+k ) ⪯(−γf(q))k−(−γf(q))N+k, (I- _f^(q))^k-(I- _f^(q))^N+k, we obtain 1N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),t(M)⟩ 1N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),B_t^(M) ≤ ≤ 1γN2∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,(1+ϵ~)γα⋅tr(0,N)1−(1+ϵ~)γαtr(f(q))⋅f(q)+0⟩. 1γ N^2 _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k, (1+ ε)γα·tr (B_0,N )1-(1+ ε)γ (H_f^(q))·H_f^(q)+B_0 . Therefore, it suffices to upper bound the following two terms I1=(1+ϵ~)αtr(0,N)N2(1−(1+ϵ~)γαtr(f(q)))∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,f(q)⟩ I_1= (1+ ε) (B_0,N)N^2(1-(1+ ε)γ (H_f^(q))) _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k,H_f^(q) I2=1γN2∑k=0N−1⟨(−γf(q))k−(−γf(q))N+k,0⟩. I_2= 1γ N^2 _k=0^N-1 (I- _f^(q))^k-(I- _f^(q))^N+k,B_0 . Repeating the computation in the proof of Lemma C.13, I1≤2(1+ϵ~)α(‖0−(q)∗‖f,0:k∗(q)2+Nγ‖0−(q)∗‖f,k∗:∞(q)2)Nγ(1−(1+ϵ~)γαtr(f(q)))⋅(k∗N+Nγ2∑i>k∗(λ~i(q))2).I_1≤ 2(1+ ε)α (\|v_0-v^(q)^*\|_I_f,0:k^*^(q)^2+Nγ\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2 )Nγ(1-(1+ ε)γ (H_f^(q)))· ( k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 ). I2≤1γ2N2⋅‖0−(q)∗‖(f,0:k∗(q))−12+‖0−(q)∗‖f,k∗:∞(q)2.I_2≤ 1γ^2N^2·\|v_0-v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+\|v_0-v^(q)^*\|_H_f,k^*:∞^(q)^2. ∎ C.5 Final Upper Bounds C.5.1 General Quantization Theorem C.1. Suppose γ<1/(αtr(f(q)))γ<1/ ( (H_f^(q) ) ). Under Assumption 3.1, 3.2, 3.3 and 3.4, RN(0)≤2BiasError+2VarianceError, R_N^(0)≤ 2BiasError+2VarianceError, where BiasError≤1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2, ≤ 1γ^2N^2· \|v^(q)^* \|_(H_f,0:k^*^(q))^-1^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2, VarianceError≤σG2+2α(‖(q)∗‖f,0:k∗(q)2Nγ+‖(q)∗‖f,k∗:∞(q)2)1−γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2). ≤ _G^2+2α ( \|v^(q)^* \|_I_f,0:k^*^(q)^2Nγ+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 )1-γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ). Here k∗=maxi:λ~i(q)≥1/(γN)k^*= \i: λ_i^(q)≥ 1/(γ N)\, σG2=σ¯2+suptαtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+supt([ϵt(a)2|at]+[ϵt(o)2|ot]). _G^2= σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ). Proof. The proof can be completed by Lemma C.6, Lemma C.8 and Lemma C.13. ∎ C.5.2 Multiplicative Quantization Theorem C.2. Suppose γ<1/((1+ϵ~)αtr(f(q)))γ<1/ ((1+ ε) (H_f^(q) ) ). If there exist ϵ¯p,ϵ¯a ε_p, ε_a and ϵ¯o ε_o such that for any i∈p,a,oi∈\p,a,o\, quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.2, 3.3 and 3.4, RN(0)≤2BiasError+2VarianceError, R_N^(0)≤ 2BiasError+2VarianceError, where BiasError≤1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2, ≤ 1γ^2N^2· \|v^(q)^* \|_(H_f,0:k^*^(q))^-1^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2, VarianceError≤σM2+2(1+ϵ~)α(‖(q)∗‖f,0:k∗(q)2Nγ+‖(q)∗‖f,k∗:∞(q)2)1−(1+ϵ~)γαtr(f(q))(k∗N+Nγ2⋅∑i>k∗(λ~i(q))2). ≤ _M^2+2(1+ ε)α ( \|v^(q)^* \|_I_f,0:k^*^(q)^2Nγ+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 )1-(1+ ε)γ (H_f^(q)) ( k^*N+Nγ^2· _i>k^*( λ_i^(q))^2 ). Here k∗=maxi:λ~i(q)≥1/(γN)k^*= \i: λ_i^(q)≥ 1/(γ N)\ and ϵ~=4ϵ¯o+2(2ϵ¯o+1)[2(1+ϵ¯p)ϵ¯a+2ϵ¯p], ε=4 ε_o+2(2 ε_o+1) [2(1+ ε_p) ε_a+2 ε_p ], σM2=(2ϵ¯o+1)σ¯2+(2ϵ¯o+1)[2ϵ¯p+2(1+ϵ¯p)ϵ¯a]αtr(f(q)(q)∗(q)∗⊤). _M^2=(2 ε_o+1) σ^2+(2 ε_o+1) [2 ε_p+2(1+ ε_p) ε_a ] (H_f^(q)v^(q)^*v^(q)^* ). Proof. The proof can be completed by Lemma C.5, Lemma C.10 and Lemma C.16. ∎ C.6 Additive Error Upper Bounds under Power-law Spectrum Here we analyze the additive error in Lemma B.2, and take expectation on ∗w^*. Denote AdditiveError= = 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + ((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. Recall that ∗=(⊤)−1∗,(q)∗=(f(q))−1∗.v^*= (SHS )^-1SHw^*, v^(q)^*=(H_f^(q))^-1SHw^*. Denote =f(q)−⊤D=H_f^(q)-SHS , then (q)∗=(⊤+)−1⊤∗.v^(q)^*= (SHS +D )^-1SHS v^*. It follows that ∗−(q)∗=(⊤+)−1∗.v^*-v^(q)^*= (SHS +D )^-1Dv^*. Hence, 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩=12‖∗‖12, 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) = 12 \|w^* \|_S_1^2, (40) where 1= _1= ⊤(⊤)−1(⊤+)−1⊤(⊤+)−1(⊤)−1. (SHS )^-1D (SHS +D )^-1SHS (SHS +D )^-1D (SHS )^-1SH. Next, we derive upper bounds for AdditiveAdditive via taking expectation on ∗w^*. Lemma C.17 (Additive Error under multiplicative quantization, an upper bound). Under Assumption 3.1, 3.2 and 3.5, for any i∈s,d,fi∈\s,d,f\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, ∗‖∗‖12≲[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2, _w^* \|w^* \|_S_1^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2, ∗[((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] _w^* [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≲ (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s)(1+ ε_f)(1+ ε_d)(1+ ε_s). Proof. Regarding the first inequality, noticing that under multiplicative quantization, f(q)⪯(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤,H_f^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS , it follows that =f(q)−⊤⪯[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1]⊤.D=H_f^(q)-SHS [(1+ ε_f)(1+ ε_d)(1+ ε_s)-1]SHS . Further by Assumption 3.5, ∗‖∗‖2≂1,E_w^*\|w^*\|_H^2 1, then we have ∗‖∗‖12≲ _w^* \|w^* \|_S_1^2 [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖1/2⊤(⊤)−11/2‖ [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2 \|H^1/2S (SHS )^-1SH^1/2 \| ≤ ≤ [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2, [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2, where the first inequality holds by Lemma F.3. Regarding the second inequality, by Assumption 3.5, it holds [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] (41) = = [∗⊤⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−1∗] [w^* HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SHw^* ] ≲ ‖1/2⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−11/2‖ \|H^1/2S (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SH^1/2 \| ≤ ≤ ‖(f(q))−121Nγ[−(−γf(q))N](f(q))−12‖⋅‖(f(q))−12(f(q)−⊤)(f(q))−12‖ \|(H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 \|· \|(H_f^(q))^- 12 (H_f^(q)-SHS )(H_f^(q))^- 12 \| ⋅ · ‖(f(q))−12⊤(f(q))−12‖. \|(H_f^(q))^- 12SHS (H_f^(q))^- 12 \|. Noticing that [(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1]⊤⪯[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1]⊤.[(1+ ε_f)(1+ ε_d)(1+ ε_s)-1]SHS [(1+ ε_f)(1+ ε_d)(1+ ε_s)-1]SHS . Firstly, by Lemma F.2, ‖(f(q))−12(f(q)−⊤)(f(q))−12‖≤(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). \|(H_f^(q))^- 12 (H_f^(q)-SHS )(H_f^(q))^- 12 \|≤ (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s). (42) Secondly, by Lemma F.1, ‖(f(q))−12⊤(f(q))−12‖≤1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). \|(H_f^(q))^- 12SHS (H_f^(q))^- 12 \|≤ 1(1+ ε_f)(1+ ε_d)(1+ ε_s). (43) Thirdly, ‖(f(q))−121Nγ[−(−γf(q))N](f(q))−12‖= \|(H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 \|= 1Nγmaxi1−(1−γλ~i(q))Nλ~i(q) 1Nγ _i 1- (1-γ λ_i^(q) )^N λ_i^(q) (44) ≤ ≤ 1Nγmaximin1,γNλ~i(q)λ~i(q) 1Nγ _i \1,γ N λ_i^(q) \ λ_i^(q) = = 1Nγmaximin1λ~i(q),γN 1Nγ _i \ 1 λ_i^(q),γ N \ ≤ ≤ 1. 1. Therefore, (41), (42), (43), and (44), we have ∗[((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] _w^* [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≲ (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s)(1+ ε_f)(1+ ε_d)(1+ ε_s). ∎ Lemma C.18 (Additive Error under additive quantization, an upper bound). Under Assumption 3.1, 3.2, 3.5, for any i∈s,d,fi∈\s,d,f\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then with probability at least 1−e−Ω(M)1-e^- (M), ∗‖∗‖12≲(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(M−a+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2. _w^* \|w^* \|_S_1^2 ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (M^-a+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2. ∗[((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] _w^* [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≲ ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+M−a⋅11+ϵ¯s(1+ϵ¯dp)+ϵ¯f+ϵ¯dpM. ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM+M^-a· 11+ ε_s(1+ ε_dp)+ ε_f+ ε_d pM. Proof. Regarding the first inequality, noticing that under additive quantization, ⊤+ϵ¯str()+ϵ¯d⊤+(ϵ¯sϵ¯dp+ϵ¯f)⪯f(q)⪯⊤+ϵ¯str()+ϵ¯d⊤+(ϵ¯sϵ¯dp+ϵ¯f).SHS + ε_str(H)I+ ε_dSS +( ε_s ε_dp+ ε_f)I _f^(q) + ε_str(H)I+ ε_dSS +( ε_s ε_dp+ ε_f)I. Then under the power-law Assumption 3.5, with probability at least 1−e−Ω(M)1-e^- (M), ⊤+(ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM)≾f(q)≾⊤+(ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM).SHS + ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )I _f^(q) + ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )I. It follows that =f(q)−⊤≾(ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM).D=H_f^(q)-SHS ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )I. Further by Assumption 3.5, we have with probability at least 1−e−Ω(M)1-e^- (M), ∗‖∗‖12≲ _w^* \|w^* \|_S_1^2 (ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM)2(μmin(⊤)+ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM)2‖1/2⊤(⊤)−11/2‖ ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )^2 ( _ min (SHS )+ ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )^2 \|H^1/2S (SHS )^-1SH^1/2 \| ≂ (ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(M−a+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2, ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (M^-a+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2, where the first inequality holds by Lemma F.4 and the last inequality holds by Lemma G.1. Regarding the second inequality, we prove by (41) and noticing that (ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM)≾(ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM). ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )I ( ε_s+ ε_s ε_dp+ ε_f+ ε_d pM )I. Firstly, by Lemma F.2 and Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), ‖(f(q))−12(f(q)−⊤)(f(q))−12‖≲ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+M−a. \|(H_f^(q))^- 12 (H_f^(q)-SHS )(H_f^(q))^- 12 \| ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM+M^-a. (45) By Lemma F.1 and Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), ‖(f(q))−12⊤(f(q))−12‖≲11+ϵ¯s(1+ϵ¯dp)+ϵ¯f+ϵ¯dpM. \|(H_f^(q))^- 12SHS (H_f^(q))^- 12 \| 11+ ε_s(1+ ε_dp)+ ε_f+ ε_d pM. (46) Therefore, together with (41), (44), (45), and (46), we have, with probability at least 1−e−Ω(M)1-e^- (M), ∗[((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] _w^* [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≲ ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+M−a+ϵ¯s⋅11+ϵ¯s(1+ϵ¯dp)+ϵ¯f+ϵ¯dpM. ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s ε_dp+ ε_f+ ε_d pM+M^-a+ ε_s· 11+ ε_s(1+ ε_dp)+ ε_f+ ε_d pM. ∎ C.7 Variance Upper Bounds under Power-Law Spectrum Denote deff=k∗+γ2N2∑i>k∗(λ~i(q))2.d_ eff=k^*+γ^2N^2 _i>k^*( λ_i^(q))^2. (47) We then focus on bounding deff/Nd_ eff/N with k∗=maxk:λ~i(q)≥1/(γN)k^*= \k: λ_i^(q)≥ 1/(γ N)\ in this subsection. C.7.1 Multiplicative Quantization Lemma C.19. If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-multiplicative, under Assumption 3.1, Assumption 3.2 and Assumption 3.5, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, with deffd_ eff defined in (47), it holds deffN≲minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/aN. d_ effN \M,[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a \N. Proof. Define k†:=maxj:(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a≥1/(γN)k := \j:(1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a≥ 1/(γ N)\. Denote Neff(M)=[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a.N_ eff^(M)=[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a. By (47) and Lemma G.2, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, deffN= d_ effN= k∗+γ2N2∑i>k∗(λ~i(q))2N k^*+γ^2N^2 _i>k^*( λ_i^(q))^2N ≤ ≤ k†+γ2N2∑i>k†(λ~i(q))2N k +γ^2N^2 _i>k ( λ_i^(q))^2N ≲ k†+γ2N2∑j>k†[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a]2N k +γ^2N^2 _j>k [(1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a ]^2N ≂ minM,Neff(M)+(Neff(M))2a(Neff(M))1−2aN \M,N_ eff^(M)+(N_ eff^(M))^2a(N_ eff^(M))^1-2a \N ≂ minM,Neff(M)N \M,N_ eff^(M) \N = = minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/aN, \M,[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a \N, ∎ C.7.2 Additive Quantization Lemma C.20. If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-additive, under Assumption 3.1, Assumption 3.2 and Assumption 3.5, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, with deffd_ eff defined in (47), it holds deffN≲keff+γ2N2(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−keff)N, d_ effN k_ eff+γ^2N^2 ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2(M-k_ eff)N, where keff=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a.k_ eff= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a. Proof. Define k†:=maxj:j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM≥1/(γN)k := \j:j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM≥ 1/(γ N)\. By (47) and Lemma G.4, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, deffN= d_ effN= k∗+γ2N2∑i>k∗(λ~i(q))2N k^*+γ^2N^2 _i>k^*( λ_i^(q))^2N (48) ≤ ≤ k†+γ2N2∑i>k†(λ~i(q))2N k +γ^2N^2 _i>k ( λ_i^(q))^2N ≲ k†+γ2N2∑j>k†[j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM]2N. k +γ^2N^2 _j>k [j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM ]^2N. We then consider two cases to complete the proof. • Case one: M−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM<1NγM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM< 1Nγ Denote Neff(A)=(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)−1a.N_ eff^(A)= ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM )^- 1a. Then by (48), with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, deffN≲ d_ effN k†+γ2N2∑j>k†[j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM]2N k +γ^2N^2 _j>k [j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM ]^2N ≂ Neff(A)+γ2N2[(Neff(A))1−2a+(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−Neff(A))]N N_ eff^(A)+γ^2N^2 [(N_ eff^(A))^1-2a+ ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2 (M-N_ eff^(A) ) ]N ≂ Neff(A)+γ2N2(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−Neff(A))N. N_ eff^(A)+γ^2N^2 ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2 (M-N_ eff^(A) )N. • Case two: M−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM≥1NγM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM≥ 1Nγ By (48), deffN≲MN. d_ effN MN. Denote keff=[M−a∨Neff(A)−a]−1a=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a,k_ eff= [M^-a N_ eff^(A)^-a ]^- 1a= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a, then with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, deffN≲keff+γ2N2(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−keff)N. d_ effN k_ eff+γ^2N^2 ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2(M-k_ eff)N. ∎ C.8 Bias Upper Bounds under Power-Law Spectrum Noticing that 1γ2N2⋅‖(q)∗‖(f,0:k∗(q))−12+‖(q)∗‖f,k∗:∞(q)2≤1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2, 1γ^2N^2· \|v^(q)^* \|_(H_f,0:k^*^(q))^-1^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2≤ 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2, we aim to derive upper bounds for 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 in this section. Lemma C.21. For any k≥0k≥ 0, ‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2≲‖∗‖0:k2γN‖(f(q))−10:k‖2+‖∗‖k:∞2‖(f(q))−1/2k:∞12‖2. \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 \|w^* \|_I_0:k^2γ N \| (H_f^(q) )^-1SH_0:k \|^2+ \|w^* \|_H_k:∞^2 \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2. Proof. By the definition of (q)∗=(f(q))−1∗,v^(q)^*=(H_f^(q))^-1SHw^*, we have 1γN⋅‖(q)∗‖f,0:k∗(q)2= 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2= 1γN‖(f,0:k∗(q))−1∗‖2 1γ N \| (H_f,0:k^*^(q) )^-1SHw^* \|^2 ≲ 1γN‖(f,0:k∗(q))−10:k∗‖2+1γN‖(f,0:k∗(q))−1k:∞∗‖2 1γ N \| (H_f,0:k^*^(q) )^-1SH_0:kw^* \|^2+ 1γ N \| (H_f,0:k^*^(q) )^-1SH_k:∞w^* \|^2 ≤ ≤ 1γN‖(f,0:k∗(q))−10:k∗‖2+‖(f,0:k∗(q))−1/2k:∞∗‖2. 1γ N \| (H_f,0:k^*^(q) )^-1SH_0:kw^* \|^2+ \| (H_f,0:k^*^(q) )^-1/2SH_k:∞w^* \|^2. ‖(q)∗‖f,k∗:∞(q)2= \|v^(q)^* \|_H_f,k^*:∞^(q)^2= ‖(f,k∗:∞(q))1/2(f(q))−1∗‖2 \| (H_f,k^*:∞^(q) )^1/2 (H_f^(q) )^-1SHw^* \|^2 = = ‖(f,k∗:∞(q))−1/2∗‖2 \| (H_f,k^*:∞^(q) )^-1/2SHw^* \|^2 ≲ ‖(f,k∗:∞(q))−1/20:k∗‖2+‖(f,k∗:∞(q))−1/2k:∞∗‖2 \| (H_f,k^*:∞^(q) )^-1/2SH_0:kw^* \|^2+ \| (H_f,k^*:∞^(q) )^-1/2SH_k:∞w^* \|^2 ≤ ≤ 1γN‖(f,k∗:∞(q))−10:k∗‖2+‖(f,k∗:∞(q))−1/2k:∞∗‖2. 1γ N \| (H_f,k^*:∞^(q) )^-1SH_0:kw^* \|^2+ \| (H_f,k^*:∞^(q) )^-1/2SH_k:∞w^* \|^2. Hence, 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ≲ 1γN[‖(f,0:k∗(q))−10:k∗‖2+‖(f,k∗:∞(q))−10:k∗‖2] 1γ N [ \| (H_f,0:k^*^(q) )^-1SH_0:kw^* \|^2+ \| (H_f,k^*:∞^(q) )^-1SH_0:kw^* \|^2 ] + + ‖(f,0:k∗(q))−1/2k:∞∗‖2+‖(f,k∗:∞(q))−1/2k:∞∗‖2 \| (H_f,0:k^*^(q) )^-1/2SH_k:∞w^* \|^2+ \| (H_f,k^*:∞^(q) )^-1/2SH_k:∞w^* \|^2 = = 1γN‖(f(q))−10:k∗‖2+‖(f(q))−1/2k:∞∗‖2 1γ N \| (H_f^(q) )^-1SH_0:kw^* \|^2+ \| (H_f^(q) )^-1/2SH_k:∞w^* \|^2 ≤ ≤ ‖∗‖0:k2γN‖(f(q))−10:k‖2+‖∗‖k:∞2‖(f(q))−1/2k:∞12‖2. \|w^* \|_I_0:k^2γ N \| (H_f^(q) )^-1SH_0:k \|^2+ \|w^* \|_H_k:∞^2 \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2. ∎ Lemma C.22 (Lemma D.1 in Lin et al. (2024)). Under Assumption 3.2 and Assumption 3.5, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), it holds ‖(⊤)−10:k‖2≲1. \|(SHS )^-1SH_0:k \|^2 1. Proof. For completeness, we provide the proof here. Separating ⊤=0:k0:k0:k⊤+k:∞k:∞k:∞⊤⏟k. =SI_0:kH_0:kI_0:kS + SI_k:∞H_k:∞I_k:∞S _A_k. Then by the Woodbury’s identity, (⊤)−10:k= (SHS )^-1SH_0:k= (k−1−k−10:k[0:k−1+0:k⊤k−10:k]−10:k⊤k−1)0:k0:k (A_k^-1-A_k^-1SI_0:k [H_0:k^-1+I_0:kS A_k^-1SI_0:k ]^-1I_0:kS A_k^-1 )SI_0:kH_0:k = = k−10:k0:k−k−10:k[0:k−1+0:k⊤k−10:k]−10:k⊤k−10:k0:k _k^-1SI_0:kH_0:k-A_k^-1SI_0:k [H_0:k^-1+I_0:kS A_k^-1SI_0:k ]^-1I_0:kS A_k^-1SI_0:kH_0:k = = k−10:k[0:k−1+0:k⊤k−10:k]−10:k−10:k. _k^-1SI_0:k [H_0:k^-1+I_0:kS A_k^-1SI_0:k ]^-1H_0:k^-1H_0:k. Therefore, ‖(⊤)−10:k‖= \| (SHS )^-1SH_0:k \|= ‖k−10:k[0:k−1+0:k⊤k−10:k]−10:k−10:k‖ \|A_k^-1SI_0:k [H_0:k^-1+I_0:kS A_k^-1SI_0:k ]^-1H_0:k^-1H_0:k \| (49) ≤ ≤ ‖k−1‖‖0:k‖‖[0:k⊤k−10:k]−1‖. \|A_k^-1 \| \|SI_0:k \| \| [I_0:kS A_k^-1SI_0:k ]^-1 \|. Note that 0:k=kk⊤,k=[1,…,k]∈ℝp×k,I_0:k=V_kV_k , _k=[v_1,...,v_k] ^p× k, it follows that the eigenvalues of 0:kSI_0:k correspond to the eigenvalues of kSV_k. As ij∼(0,1M)S_ij (0, 1M), for k≤M2k≤ M2, with probability at least 1−e−Ω(M)1-e^- (M), ‖0:k‖≤c, \|SI_0:k \|≤ c, (50) where c is a constant. Denote λ^ii=1M\ λ_i\_i=1^M be the eigenvalues of k=k:∞k:∞k:∞⊤+A_k=SI_k:∞H_k:∞I_k:∞S +D. ‖k−1‖≤1λ^M, \|A_k^-1 \|≤ 1 λ_M, (51) We then deal with 0:k⊤k−10:kI_0:kS A_k^-1SI_0:k. With probability at least 1−e−Ω(M)1-e^- (M), for k≤M/2k≤ M/2, it holds 0:k⊤k−10:k= _0:kS A_k^-1SI_0:k= k∑i=1M1μi(k)~i~i⊤k⊤ _k _i=1^M 1 _i(A_k) s_i s_i V_k ⪰ k∑i=M/2M1μi(k)~i~i⊤k⊤ _k _i=M/2^M 1 _i(A_k) s_i s_i V_k ⪰ k∑i=M/2M1μM/2(k)~i~i⊤k⊤ _k _i=M/2^M 1 _M/2(A_k) s_i s_i V_k ≿ 1μM/2(k:∞k:∞k:∞⊤)0:k. 1 _M/2 (SI_k:∞H_k:∞I_k:∞S )I_0:k. Together with (49), (50), (51), for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), ‖(⊤)−10:k‖≲ \| (SHS )^-1SH_0:k \| μM/2(k:∞k:∞k:∞⊤)μM(k:∞k:∞k:∞⊤)≲1, _M/2 (SI_k:∞H_k:∞I_k:∞S ) _M (SI_k:∞H_k:∞I_k:∞S ) 1, (52) where the last inequality holds by Lemma G.6. ∎ C.8.1 Multiplicative Quantization Lemma C.23. Under Assumption 3.2 and Assumption 3.5, for any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), it holds 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ≲ [1+(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)Ma/2]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖∗‖0:k2γN+‖∗‖k:∞2(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). [1+ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s)M^a/2 ]^2[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^2(1+ ε_d)(1+ ε_f)(1+ ε_s). Further, if f(q)H_f^(q) and ⊤SHS commute, 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2≲1[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖∗‖0:k2γN+‖∗‖k:∞2(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 1[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^2(1+ ε_d)(1+ ε_f)(1+ ε_s). Proof. We prove by using Lemma C.21. The key is to derive bounds for ‖(f(q))−1/2k:∞12‖2 \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2 and ‖(f(q))−10:k‖2 \| (H_f^(q) )^-1SH_0:k \|^2. Noticing that f(q)⪰(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)⊤, _f^(q) (1+ ε_d)(1+ ε_f)(1+ ε_s)SHS , we have ‖(f(q))−1/2k:∞12‖2≤ \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2≤ ‖(f(q))−1/2(⊤)12‖2⋅‖(⊤)−12k:∞12‖2 \| (H_f^(q) )^-1/2 (SHS ) 12 \|^2· \| (SHS )^- 12SH_k:∞ 12 \|^2 (53) = = μmax((f(q))−1/2⊤(f(q))−1/2)⋅‖(⊤)−12k:∞12‖2 _ ( (H_f^(q) )^-1/2SHS (H_f^(q) )^-1/2 )· \| (SHS )^- 12SH_k:∞ 12 \|^2 ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)‖(⊤)−12k:∞12‖2 1(1+ ε_d)(1+ ε_f)(1+ ε_s) \| (SHS )^- 12SH_k:∞ 12 \|^2 ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)‖(⊤)−1212‖2 1(1+ ε_d)(1+ ε_f)(1+ ε_s) \| (SHS )^- 12SH 12 \|^2 ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s), 1(1+ ε_d)(1+ ε_f)(1+ ε_s), where the second inequality holds by Lemma F.1. We then focus on ‖(f(q))−10:k‖2 \| (H_f^(q) )^-1SH_0:k \|^2. Noting that ‖(f(q))−10:k‖2≤‖(f(q))−1⊤‖2‖(⊤)−10:k‖2, \| (H_f^(q) )^-1SH_0:k \|^2≤ \| (H_f^(q) )^-1SHS \|^2 \|(SHS )^-1SH_0:k \|^2, we handle ‖(f(q))−1⊤‖2 \| (H_f^(q) )^-1SHS \|^2 and ‖(⊤)−10:k‖2 \|(SHS )^-1SH_0:k \|^2 respectively. Regarding ‖(f(q))−1⊤‖2 \| (H_f^(q) )^-1SHS \|^2, as f(q)H_f^(q) and ⊤SHS might not commute, we can only derive an upper bound related to the condition number of ⊤SHS . Specifically, denote =(⊤)−1/2f(q)(⊤)−1/2X=(SHS )^-1/2H_f^(q)(SHS )^-1/2, then we have (f(q))−1⊤= (H_f^(q) )^-1SHS = (⊤)−1/2−1(⊤)1/2. (SHS )^-1/2X^-1(SHS )^1/2. Further, recall that (1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)⊤⪯f(q)⪯(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)⊤(1+ ε_d)(1+ ε_f)(1+ ε_s)SHS _f^(q) (1+ ε_d)(1+ ε_f)(1+ ε_s)SHS , we have 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)⪯−1=⊤(f(q))−1⊤⪯1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). 1(1+ ε_d)(1+ ε_f)(1+ ε_s)I ^-1=SHS (H_f^(q))^-1SHS 1(1+ ε_d)(1+ ε_f)(1+ ε_s)I. Denote ⪯=1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1⪯[1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)] 0 = 1(1+ ε_d)(1+ ε_f)(1+ ε_s)I-X^-1 [ 1(1+ ε_d)(1+ ε_f)(1+ ε_s)- 1(1+ ε_d)(1+ ε_f)(1+ ε_s) ]I, then ‖(f(q))−1⊤‖= \| (H_f^(q) )^-1SHS \|= ‖(⊤)−1/2−1(⊤)1/2‖ \|(SHS )^-1/2X^-1(SHS )^1/2 \| = = ‖1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(⊤)−1/2(⊤)1/2‖ \| 1(1+ ε_d)(1+ ε_f)(1+ ε_s)I-(SHS )^-1/2 (SHS )^1/2 \| ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)+‖(⊤)−1/2(⊤)1/2‖ 1(1+ ε_d)(1+ ε_f)(1+ ε_s)+ \|(SHS )^-1/2 (SHS )^1/2 \| ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)+‖(⊤)−1/2‖‖‖(⊤)1/2‖ 1(1+ ε_d)(1+ ε_f)(1+ ε_s)+ \|(SHS )^-1/2 \| \| \| \|(SHS )^1/2 \| ≲ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)+[1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]Ma/2, 1(1+ ε_d)(1+ ε_f)(1+ ε_s)+ [ 1(1+ ε_d)(1+ ε_f)(1+ ε_s)- 1(1+ ε_d)(1+ ε_f)(1+ ε_s) ]M^a/2, where the last inequality holds with probability at least 1−e−Ω(M)1-e^- (M) by Lemma G.1. We would like to remark that, the term related to Ma/2M^a/2 is from the misalignment between f(q)H_f^(q) and ⊤SHS . Specifically, if f(q)H_f^(q) and ⊤SHS commute, then this term will be vanished. ‖(f(q))−1⊤‖=‖(⊤)−1/2−1(⊤)1/2‖=‖−1‖≤1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). \| (H_f^(q) )^-1SHS \|= \|(SHS )^-1/2X^-1(SHS )^1/2 \|= \|X^-1 \|≤ 1(1+ ε_d)(1+ ε_f)(1+ ε_s). Regarding ‖(⊤)−10:k‖2 \|(SHS )^-1SH_0:k \|^2, by Lemma C.22, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), ‖(⊤)−10:k‖2≲1. \|(SHS )^-1SH_0:k \|^2 1. Overall, together with Lemma C.21, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), it holds 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ≲ [1+(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)Ma/2]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖∗‖0:k2γN+‖∗‖k:∞2(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). [1+ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s)M^a/2 ]^2[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^2(1+ ε_d)(1+ ε_f)(1+ ε_s). Further, if f(q)H_f^(q) and ⊤SHS commute, 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2≲1[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖∗‖0:k2γN+‖∗‖k:∞2(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 1[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^2(1+ ε_d)(1+ ε_f)(1+ ε_s). ∎ Lemma C.24. Under Assumption 3.2 and Assumption 3.5, for any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, with probability at least 1−e−Ω(M)1-e^- (M), it holds ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2]≲max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)1+[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2Ma]1a−1,M1−a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s).E_w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] \ [ Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s)1+ [ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2M^a ] 1a-1,M^1-a \(1+ ε_d)(1+ ε_f)(1+ ε_s). Further if f(q)H_f^(q) and ⊤SHS commute, ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2]≲max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]1a−1,M1−a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s).E_w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] \ [Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s) ] 1a-1,M^1-a \(1+ ε_d)(1+ ε_f)(1+ ε_s). Proof. By Lemma C.23, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2] _w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] ≲ ∗[[1+(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)Ma/2]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2‖∗‖0:k2γN+‖∗‖k:∞2(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)] _w^* [ [1+ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s)M^a/2 ]^2[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^2(1+ ε_d)(1+ ε_f)(1+ ε_s) ] ≂ [1+(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)Ma/2]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2kNγ+1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)∑i>ki−a [1+ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s)M^a/2 ]^2[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 kNγ+ 1(1+ ε_d)(1+ ε_f)(1+ ε_s) _i>ki^-a ≂ 1+[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2Ma[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2kNγ+k1−a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s) 1+ [ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2M^a[(1+ ε_d)(1+ ε_f)(1+ ε_s)]^2 kNγ+ k^1-a(1+ ε_d)(1+ ε_f)(1+ ε_s) ≲ max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)1+[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2Ma]1a−1,M1−a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s), \ [ Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s)1+ [ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2M^a ] 1a-1,M^1-a \(1+ ε_d)(1+ ε_f)(1+ ε_s), where in the last inequality we choose k=[M/2]∧[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)1+[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2Ma]1/ak=[M/2] [ Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s)1+ [ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2M^a ]^1/a. The statement when f(q)H_f^(q) and ⊤SHS commute can be deduced directly. We omit here for simplicity. ∎ C.8.2 Additive Quantization Lemma C.25. For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), under Assumption 3.2 and Assumption 3.5, it holds 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ≲ (1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))2(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2‖∗‖0:k2γN+‖∗‖k:∞21+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. (1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) )^2 (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^21+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Further, if f(q)H_f^(q) and ⊤SHS commute, 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ≲ 1(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2‖∗‖0:k2γN+‖∗‖k:∞21+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. 1 (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^21+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Proof. We prove by using Lemma C.21. The key is to derive bounds for ‖(f(q))−1/2k:∞12‖2 \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2 and ‖(f(q))−10:k‖2 \| (H_f^(q) )^-1SH_0:k \|^2. Noticing that with probability at least 1−e−Ω(M)1-e^- (M), f(q)≿⊤+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM), _f^(q) + ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )I, it follows by Lemma F.1 that ‖(f(q))−1/2k:∞12‖2≤ \| (H_f^(q) )^-1/2SH_k:∞ 12 \|^2≤ ‖(f(q))−1/2(⊤)12‖2‖(⊤)−12k:∞12‖2 \| (H_f^(q) )^-1/2 (SHS ) 12 \|^2 \| (SHS )^- 12SH_k:∞ 12 \|^2 (54) ≤ ≤ μmax(⊤)μmax(⊤)+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM‖(⊤)−1212‖2 _ (SHS ) _ (SHS )+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM \| (SHS )^- 12SH 12 \|^2 ≲ 11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM, 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM, where the last inequality holds by Lemma G.1. We then focus on ‖(f(q))−10:k‖2 \| (H_f^(q) )^-1SH_0:k \|^2. Similarly, we consider ‖(f(q))−10:k‖2≤‖(f(q))−1⊤‖2‖(⊤)−10:k‖2. \| (H_f^(q) )^-1SH_0:k \|^2≤ \| (H_f^(q) )^-1SHS \|^2 \|(SHS )^-1SH_0:k \|^2. By Lemma C.22, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), ‖(⊤)−10:k‖≲1. \|(SHS )^-1SH_0:k \| 1. We merely need to drive upper bound for ‖(f(q))−1⊤‖ \| (H_f^(q) )^-1SHS \|. Similar to the multiplicative quantization case, denote =(⊤)−1/2f(q)(⊤)−1/2X=(SHS )^-1/2H_f^(q)(SHS )^-1/2. Recall that ⊤+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)⪯f(q)⪯⊤+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM),SHS + ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )I _f^(q) + ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )I, we have +(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)(⊤)−1⪯+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)(⊤)−1.I+ ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )(SHS )^-1 + ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )(SHS )^-1. Denote =11+μmin((⊤)−1)[ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]−1 = 11+ _ ((SHS )^-1 ) [ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]I-X^-1, then it holds 0⪯11+μmin((⊤)−1)[ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]−[+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)(⊤)−1]−1.0 11+ _ ((SHS )^-1 ) [ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]I- [I+ ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )(SHS )^-1 ]^-1. Hence, using Lemma G.1 we have ‖(f(q))−1⊤‖ \| (H_f^(q) )^-1SHS \| = = ‖(⊤)−1/2−1(⊤)1/2‖ \|(SHS )^-1/2X^-1(SHS )^1/2 \| = = ‖(⊤)−1/2(11+μmin((⊤)−1)[ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]−)(⊤)1/2‖ \|(SHS )^-1/2 ( 11+ _ ((SHS )^-1 ) [ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]I- )(SHS )^1/2 \| ≤ ≤ 11+μmin((⊤)−1)[ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]+‖(⊤)−1/2(⊤)1/2‖ 11+ _ ((SHS )^-1 ) [ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]+ \|(SHS )^-1/2 (SHS )^1/2 \| ≲ 11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM+Ma/2‖ 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM+M^a/2 \| \| ≲ 11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM+Ma/2(11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM−11+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)) 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM+M^a/2 ( 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM- 11+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ) = = 1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. 1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. If f(q)H_f^(q) and ⊤SHS commute, ‖(f(q))−1⊤‖=‖(⊤)−1/2−1(⊤)1/2‖=‖−1‖≲11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. \| (H_f^(q) )^-1SHS \|= \|(SHS )^-1/2X^-1(SHS )^1/2 \|= \|X^-1 \| 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. ∎ Lemma C.26. Under Assumption 3.2 and Assumption 3.5, for any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, with probability at least 1−e−Ω(M)1-e^- (M), it holds ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2]≲max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma[Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]2]1a−1,M1−a1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. _w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] \ [ Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a [ M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ]^2 ] 1a-1,M^1-a \1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Further if f(q)H_f^(q) and ⊤SHS commute, ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2]≲max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]1a−1,M1−a1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM.E_w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] \ [Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ] 1a-1,M^1-a \1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Proof. By Lemma C.25, for k≤M/2k≤ M/2, with probability at least 1−e−Ω(M)1-e^- (M), ∗[‖(q)∗‖f,0:k∗(q)2γN+‖(q)∗‖f,k∗:∞(q)2] _w^* [ \|v^(q)^* \|_I_f,0:k^*^(q)^2γ N+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] ≲ ∗[(1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))2(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2‖∗‖0:k2γN+‖∗‖k:∞21+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM] _w^* [ (1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) )^2 (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 \|w^* \|_I_0:k^2γ N+ \|w^* \|_H_k:∞^21+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ] ≂ (1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))2(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2kNγ+11+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM∑i>ki−a (1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) )^2 (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 kNγ+ 11+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM _i>ki^-a ≂ (1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))2(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2kNγ+k1−a1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM (1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) )^2 (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 kNγ+ k^1-a1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ≲ max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma[Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]2]1a−1,M1−a1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM, \ [ Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a [ M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ]^2 ] 1a-1,M^1-a \1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM, where in the last inequality we choose k=[M/2]∧[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)(1+Ma/2Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))2]1/a.k=[M/2] [ Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) (1+M^a/2 M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) )^2 ]^1/a. The statement when f(q)H_f^(q) and ⊤SHS commute can be deduced directly. We omit here for simplicity. ∎ C.9 Population Risk Upper Bounds under Power-law Spectrum C.9.1 Multiplicative Quantization Theorem C.3. Suppose γ<1/((1+ϵ~)αtr(f(q)))γ<1/ ((1+ ε) (H_f^(q) ) ). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, • Irreducible:=ℛ(∗)=12σ2. Irreducible:=R(w^*)= 12σ^2. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ∗Approx≲M1−a.E_w^*Approx M^1-a. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S 888Here we take expectation on the prior ∗w^*., Excess≲BiasError+VarianceError+AdditiveError,EExcess +VarianceError+AdditiveError, where BiasError≲max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)1+[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2Ma]1a−1,M1−a[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2, \ [ Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s)1+ [ (1+ ε_d)(1+ ε_f)(1+ ε_s)-(1+ ε_d)(1+ ε_f)(1+ ε_s)(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2M^a ] 1a-1,M^1-a \ [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2, VarianceError≲σM2+(1+ϵ~)α(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/aN, _M^2+ (1+ ε)α(1+ ε_d)(1+ ε_s)(1+ ε_f)(1+ ε_d)(1+ ε_f)(1+ ε_s) \M,[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a \N, AdditiveError≲[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2+(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s), [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2+ (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s)(1+ ε_f)(1+ ε_d)(1+ ε_s), with ϵ~=ϵ¯o+(ϵ¯o+1)[(1+ϵ¯p)ϵ¯a+ϵ¯p], ε= ε_o+( ε_o+1) [(1+ ε_p) ε_a+ ε_p ], σM2=(ϵ¯o+1)σ¯2+(ϵ¯o+1)[ϵ¯p+(1+ϵ¯p)ϵ¯a]α(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f). _M^2=( ε_o+1) σ^2+ ( ε_o+1) [ ε_p+(1+ ε_p) ε_a ]α(1+ ε_d)(1+ ε_s)(1+ ε_f). Further, if f(q)H_f^(q) and ⊤SHS commute, BiasError≲max[Nγ(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]1a−1,M1−a[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2.BiasError \ [Nγ(1+ ε_d)(1+ ε_f)(1+ ε_s) ] 1a-1,M^1-a \ [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2. Proof. The proof can be completed by (5), (6), Lemma B.2, (27), Theorem C.2, Lemma C.17, Lemma C.19, Lemma C.24, and noticing the following facts. Firstly, under multiplicative quantization, μmax((f(q))−1⊤)≤1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). _ max ((H_f^(q))^-1SHS )≤ 1(1+ ε_d)(1+ ε_f)(1+ ε_s). Secondly, ∗‖(q)∗‖f(q)2= _w^* \|v^(q)^* \|_H_f^(q)^2= ∗[∗⊤⊤(f(q))−1∗] _w^* [w^* HS (H_f^(q))^-1SHw^* ] (55) ≤ ≤ ∗‖∗‖2‖1/2⊤(f(q))−11/2‖ _w^* \|w^* \|_H^2 \|H^1/2S (H_f^(q))^-1SH^1/2 \| ≂ ‖1/2⊤(f(q))−11/2‖ \|H^1/2S (H_f^(q))^-1SH^1/2 \| ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f)‖1/2⊤(⊤)−11/2‖ 1(1+ ε_d)(1+ ε_s)(1+ ε_f) \|H^1/2S (SHS )^-1SH^1/2 \| ≤ ≤ 1(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f). 1(1+ ε_d)(1+ ε_s)(1+ ε_f). Thirdly, we derive crude upper bounds for 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 in VarianceErrorVarianceError. ∗[1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2]≤∗‖(q)∗‖f(q)2≲1(1+ϵ¯d)(1+ϵ¯s)(1+ϵ¯f), _w^* [ 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] _w^* \|v^(q)^* \|_H_f^(q)^2 1(1+ ε_d)(1+ ε_s)(1+ ε_f), where the last inequality holds by (55). ∎ C.9.2 Additive Quantization Theorem C.4. Suppose γ<1/(αtr(f(q)))γ<1/ ( (H_f^(q) ) ). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, • Irreducible:=ℛ(∗)=12σ2. Irreducible:=R(w^*)= 12σ^2. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ∗Approx≲M1−a.E_w^*Approx M^1-a. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S 999Here we take expectation on the prior ∗w^*., Excess≲BiasError+VarianceError+AdditiveError,EExcess +VarianceError+AdditiveError, where BiasError≲max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma[Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)−(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]2]1a−1,M1−a[1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]2, \ [ Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a [ M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )- ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ]^2 ] 1a-1,M^1-a \ [1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]^2, VarianceError≲σG2+α1+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM1+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpMkeff+γ2N2(ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM)2(M−keff)N, _G^2+ α1+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM1+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM k_ eff+γ^2N^2 ( ε_f+(1+ ε_dp) ε_s+ ε_d pM )^2(M-k_ eff)N, AdditiveError≲(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(M−a+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2+ ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (M^-a+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2+ ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+M−a⋅11+ϵ¯s(1+ϵ¯dp)+ϵ¯f+ϵ¯dpM, ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM+M^-a· 11+ ε_s(1+ ε_dp)+ ε_f+ ε_d pM, with σG2=σ¯2+ϵ¯a+ϵ¯o+αϵ¯p[1+pϵ¯d+M(ϵ¯f+ϵ¯s+ϵ¯sϵ¯dp)], _G^2= σ^2+ ε_a+ ε_o+α ε_p [1+p ε_d+M( ε_f+ ε_s+ ε_s ε_dp) ], keff=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a. k_ eff= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a. Further, if f(q)H_f^(q) and ⊤SHS commute, BiasError≲max[Nγ(1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)]1a−1,M1−a[1+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM]2.BiasError \ [Nγ (1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ) ] 1a-1,M^1-a \ [1+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM ]^2. Proof. The proof can be completed by (5), (6), Lemma B.2, (27), Theorem C.1, Lemma C.18, Lemma C.20, Lemma C.26, and noticing the following facts. Firstly, by Lemma F.1, with probability at least 1−e−Ω(M)1-e^- (M), μmax((f(q))−1⊤) _ max ((H_f^(q))^-1SHS ) ≤ ≤ μmax((⊤+(ϵ¯f+ϵ¯s+ϵ¯sϵ¯dp)+ϵ¯dpM))−1⊤) _ max ( (SHS +( ε_f+ ε_s+ ε_s ε_dp)+ ε_d pM)I )^-1SHS ) ≲ 11+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM. 11+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM. Secondly, with probability at least 1−e−Ω(M)1-e^- (M), tr(f(q))≲tr(⊤+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))≂1+M(ϵ¯f+ϵ¯s(1+ϵ¯dp))+pϵ¯d.tr(H_f^(q)) (SHS +( ε_f+ ε_s(1+ ε_dp)+ ε_d pM)I ) 1+M( ε_f+ ε_s(1+ ε_dp))+p ε_d. Thirdly, we derive crude upper bounds for 1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 in VarianceErrorVarianceError. By Lemma G.1, ∗[1γN⋅‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2] _w^* [ 1γ N· \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] ≤ ≤ ∗‖(q)∗‖f(q)2 _w^* \|v^(q)^* \|_H_f^(q)^2 = = ∗tr(∗⊤⊤(f(q))−1∗) _w^*tr (w^* HS (H_f^(q))^-1SHw^* ) ≤ ≤ ∗‖∗‖2‖1/2⊤(f(q))−11/2‖ _w^* \|w^* \|_H^2 \|H^1/2S (H_f^(q))^-1SH^1/2 \| ≂ ‖1/2⊤(f(q))−11/2‖ \|H^1/2S (H_f^(q))^-1SH^1/2 \| ≲ 11+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM. 11+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM. ∎ Appendix D Lower Bound Analysis D.1 Update Rule Recall Lemma B.2, [ℛM(¯N)−ℛM(∗)]= [R_M( v_N)-R_M(v^*) ]= 12⟨⊤,[((q)∗−¯N)⊗((q)∗−¯N)]⟩⏟RN 12 ,E [(v^(q)^*- v_N) (v^(q)^*- v_N) ] _R_N + + 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + ((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗. (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^*. We first derive update rule for [t⊗t]E [ η_t η_t ]. By Lemma C.1, t=(−γ~t(q)(~t(q))⊤)t−1+γ(ξt+ϵt(o)−ϵt(a)−(~t(q))⊤ϵt−1(p))~t(q). η_t= (I-γ x_t^(q)( x_t^(q)) ) η_t-1+γ ( _t+ _t^(o)- _t^(a)-( x_t^(q)) ε_t-1^(p) ) x_t^(q). Then by (7) and unbiased Assumption 3.1, [t⊗t]= [ η_t η_t ]= [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+t [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+ _t − - 2γ2[~t(q)(~t(q))⊤t−1(~t(q))⊤ξt], 2γ^2E [ x_t^(q)( x_t^(q)) η_t-1( x_t^(q)) _t ], where t _t is defined in (19). In lower bound analysis, we consider low-precision well-specific model: [ξt|~t(q)]=0.E [ _t| x_t^(q) ]=0. Therefore, [t⊗t]=[(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+t.E [ η_t η_t ]=E [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+ _t. (56) We then summarize the update rule for [t⊗t]E [ η_t η_t ] as follows. Lemma D.1 (Update rule under general quantization, a lower bound). Under Assumption 3.1, 3.2, 3.3 and 3.4, it holds [t⊗t]⪰ [ η_t η_t ] [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)] [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ] + + γ2[σ¯2+inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+inft([ϵt(a)2|at]+[ϵt(o)2|ot])]f(q). γ^2 [ σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ) ]H_f^(q). Proof. The proof focuses on dealing with each term of t _t in (19). Firstly, by Assumption 3.3, [(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤]⪰ [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ] βtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])f(q) (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )H_f^(q) (57) ⪰ inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])f(q). _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )H_f^(q). Secondly, denote at=(~t(q))⊤p(t−1),ot=l(yt)−a((~t(q))⊤p(t−1)),a_t=( x_t^(q)) Q_p(v_t-1), o_t=Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ), then [(ϵt(a)2+ϵt(o)2)~t(q)(~t(q))⊤]⪰inft[ϵt(a)2|at]+[ϵt(o)2|ot]f(q). [ ( _t^(a)^2+ _t^(o)^2 ) x_t^(q)( x_t^(q)) ] _t \E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] \H_f^(q). (58) Thirdly, by Assumption 3.4, [ξt2~t(q)(~t(q))⊤]⪰σ¯2f(q). [ _t^2 x_t^(q)( x_t^(q)) ] σ^2H_f^(q). (59) Therefore, together with (19), (57), (58), and (59), it holds, t/γ2⪰[σ¯2+inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+inft([ϵt(a)2|at]+[ϵt(o)2|ot])]f(q). _t/γ^2 [ σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ) ]H_f^(q). This immediately implies that [t⊗t] [ η_t η_t ] ⪰ [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)] [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ] + + γ2[σ¯2+inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+inft([ϵt(a)2|at]+[ϵt(o)2|ot])]f(q). γ^2 [ σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ) ]H_f^(q). ∎ Lemma D.2 (Update rule under multiplicative quantization, a lower bound). If there exist ϵ¯p,ϵ¯a ε_p, ε_a and ϵ¯o ε_o such that for any i∈p,a,oi∈\p,a,o\, quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), [t⊗t] [ η_t η_t ] ⪰ [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+γ2(1+ϵ¯o)σ¯2f(q) [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+γ^2(1+ ε_o) σ^2H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]βtr(f(q)[−(−γf(q))(t−1)]2(q)∗((q)∗)⊤)f(q) γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] (H_f^(q) [I-(I- _f^(q))^(t-1) ]^2v^(q)^* (v^(q)^* ) )H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). Proof. The proof focuses on dealing with each term of t _t in (19). Firstly, by Assumption 3.4, [ξt2~t(q)(~t(q))⊤]⪰σ¯2f(q). [ _t^2 x_t^(q)( x_t^(q)) ] σ^2H_f^(q). (60) Secondly, by the definition of multiplicative quantization, [(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤]⪰ϵ¯p[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤], [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ] ε_pE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ], (61) Thirdly, by the definition of multiplicative quantization, [ϵt(a)2~t(q)(~t(q))⊤]⪰ [ _t^(a)^2 x_t^(q)( x_t^(q)) ] ϵ¯a[(~t(q))⊤p(t−1)p(t−1)⊤~t(q)~t(q)(~t(q))⊤] ε_aE [( x_t^(q)) Q_p(v_t-1)Q_p(v_t-1) x_t^(q) x_t^(q)( x_t^(q)) ] (62) = = ϵ¯a[(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)~t(q)(~t(q))⊤]+ϵ¯a[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] ε_aE [( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q) x_t^(q)( x_t^(q)) ]+ ε_aE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪰ (1+ϵ¯p)ϵ¯a[(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤], (1+ ε_p) ε_aE [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ], where the equality holds by the unbiased quantization Assumption 3.1. Fourthly, by the definition of multiplicative quantization, [ϵt(o)2~t(q)(~t(q))⊤] [ _t^(o)^2 x_t^(q)( x_t^(q)) ] (63) ⪰ ϵ¯o[[l(yt)−a((~t(q))⊤p(t−1))]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-Q_a (( x_t^(q)) Q_p(v_t-1) ) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[l(yt)−(~t(q))⊤p(t−1)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-( x_t^(q)) Q_p(v_t-1)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[l(yt)−(~t(q))⊤t−1−(~t(q))⊤ϵt−1(p)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [Q_l(y_t)-( x_t^(q)) v_t-1-( x_t^(q)) ε_t-1^(p)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[ξt−(~t(q))⊤t−1−(~t(q))⊤ϵt−1(p)−ϵt(a)]2~t(q)(~t(q))⊤] ε_oE [ [ _t-( x_t^(q)) η_t-1-( x_t^(q)) ε_t-1^(p)-ε_t^(a) ]^2 x_t^(q)( x_t^(q)) ] = = ϵ¯o[[ξt2+(~t(q))⊤ϵt−1(p)ϵt−1(p)⊤~t(q)+ϵt(a)2+(~t(q))⊤t−1t−1⊤~t(q)]~t(q)(~t(q))⊤], ε_oE [ [ _t^2+( x_t^(q)) ε_t-1^(p) ε_t-1^(p) x_t^(q)+ε_t^(a)^2+( x_t^(q)) η_t-1 η_t-1 x_t^(q) ] x_t^(q)( x_t^(q)) ], where the last inequality holds by the unbiased quantization Assumption 3.1 and the low-precision well-specified model assumption [ξt|~t(q)]=0.E [ _t| x_t^(q) ]=0. We then focus on deriving lower bounds for [(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤]E [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ]. Noticing that [(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] = = [(~t(q))⊤(t−1+(q)∗)(t−1+(q)∗)⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) ( η_t-1+v^(q)^* ) ( η_t-1+v^(q)^* ) x_t^(q) x_t^(q)( x_t^(q)) ] = = [(~t(q))⊤[(t−1+(q)∗)(t−1+(q)∗)⊤]~t(q)~t(q)(~t(q))⊤]. [( x_t^(q)) E [ ( η_t-1+v^(q)^* ) ( η_t-1+v^(q)^* ) ] x_t^(q) x_t^(q)( x_t^(q)) ]. We first utilize the accurate expectation: [t]=−(−γf(q))t(q)∗.E [ η_t ]=- (I- _f^(q) )^tv^(q)^*. Next, we utilize the crude bound of [tt⊤]E [ η_t η_t ]. From the update rule of [tt⊤]E [ η_t η_t ] (56), we have [tt⊤]⪰ [ η_t η_t ] (ℐ−γ(q))∘[t−1t−1⊤] (I- ^(q)) [ η_t-1 η_t-1 ] = = (ℐ−γ~(q))∘[t−1t−1⊤]+γ2(ℳ(q)−ℳ~(q))∘[t−1t−1⊤]. (I-γ T^(q)) [ η_t-1 η_t-1 ]+γ^2(M^(q)- M^(q)) [ η_t-1 η_t-1 ]. Noticing that [t−1t−1⊤]⪰[t−1][t−1]⊤=(−γf(q))t−1(q)∗((q)∗)⊤(−γf(q))t−1,E [ η_t-1 η_t-1 ] [ η_t-1 ]E [ η_t-1 ] = (I- _f^(q) )^t-1v^(q)^* (v^(q)^* ) (I- _f^(q) )^t-1, by Assumption 3.3 we have (ℳ(q)−ℳ~(q))∘[t−1t−1⊤]⪰βtr(f(q)(−γf(q))t−1(q)∗((q)∗)⊤(−γf(q))t−1)f(q).(M^(q)- M^(q)) [ η_t-1 η_t-1 ] (H_f^(q) (I- _f^(q) )^t-1v^(q)^* (v^(q)^* ) (I- _f^(q) )^t-1 )H_f^(q). Hence, [tt⊤]⪰(ℐ−γ~(q))∘[t−1t−1⊤]+γ2βtr(f(q)(−γf(q))t−1(q)∗((q)∗)⊤(−γf(q))t−1)f(q). [ η_t η_t ] (I-γ T^(q)) [ η_t-1 η_t-1 ]+γ^2 (H_f^(q) (I- _f^(q) )^t-1v^(q)^* (v^(q)^* ) (I- _f^(q) )^t-1 )H_f^(q). By solving recursion, [tt⊤]⪰ [ η_t η_t ] γ2β∑i=0t−1(ℐ−γ~(q))i∘tr(f(q)(−γf(q))t−1−i(q)∗((q)∗)⊤(−γf(q))t−1−i)f(q) γ^2β _i=0^t-1(I-γ T^(q))^i (H_f^(q) (I- _f^(q) )^t-1-iv^(q)^* (v^(q)^* ) (I- _f^(q) )^t-1-i )H_f^(q) + + (ℐ−γ~(q))t∘[00⊤] (I-γ T^(q))^t [ η_0 η_0 ] = = γ2β∑i=0t−1(−γf(q))2itr(f(q)(−γf(q))t−1−i(q)∗((q)∗)⊤(−γf(q))t−1−i)f(q) γ^2β _i=0^t-1(I- _f^(q))^2itr (H_f^(q) (I- _f^(q) )^t-1-iv^(q)^* (v^(q)^* ) (I- _f^(q) )^t-1-i )H_f^(q) + + (−γf(q))t(q)∗((q)∗)⊤(−γf(q))t (I- _f^(q) )^tv^(q)^* (v^(q)^* ) (I- _f^(q) )^t ⪰ γ2β∑i=0t−1(−γf(q))2ttr(f(q)(−γf(q))2(t−1−i)(q)∗((q)∗)⊤)f(q) γ^2β _i=0^t-1(I- _f^(q))^2ttr (H_f^(q) (I- _f^(q) )^2(t-1-i)v^(q)^* (v^(q)^* ) )H_f^(q) + + (−γf(q))t(q)∗((q)∗)⊤(−γf(q))t (I- _f^(q) )^tv^(q)^* (v^(q)^* ) (I- _f^(q) )^t ⪰ γβ2(−γf(q))2tf(q)‖(q)∗‖−(−γf(q))2t2+(−γf(q))t(q)∗((q)∗)⊤(−γf(q))t. γβ2(I- _f^(q))^2tH_f^(q) \|v^(q)^* \|_I-(I- _f^(q))^2t^2+ (I- _f^(q) )^tv^(q)^* (v^(q)^* ) (I- _f^(q) )^t. Therefore, it holds [(~t(q))⊤t−1t−1⊤~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) v_t-1v_t-1 x_t^(q) x_t^(q)( x_t^(q)) ] ⪰ [(~t(q))⊤(−(−γf(q))t−1)(q)∗((q)∗)⊤(−(−γf(q))t−1)~t(q)~t(q)(~t(q))⊤] [( x_t^(q)) (I-(I- _f^(q))^t-1 )v^(q)^* (v^(q)^* ) (I-(I- _f^(q))^t-1 ) x_t^(q) x_t^(q)( x_t^(q)) ] + + γβ2[(~t(q))⊤(−γf(q))2(t−1)f(q)‖(q)∗‖−(−γf(q))2(t−1)2~t(q)~t(q)(~t(q))⊤] γβ2E [( x_t^(q)) (I- _f^(q))^2(t-1)H_f^(q) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2 x_t^(q) x_t^(q)( x_t^(q)) ] ⪰ βtr(f(q)[−(−γf(q))t−1](q)∗((q)∗)⊤[−(−γf(q))t−1])f(q) (H_f^(q) [I-(I- _f^(q))^t-1 ]v^(q)^* (v^(q)^* ) [I-(I- _f^(q))^t-1 ] )H_f^(q) + + γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). Therefore, tγ2(1+ϵ¯o)⪰ _tγ^2(1+ ε_o) σ¯2f(q)+[ϵ¯p+(1+ϵ¯p)ϵ¯a]βtr(f(q)[−(−γf(q))(t−1)]2(q)∗((q)∗)⊤)f(q) σ^2H_f^(q)+ [ ε_p+(1+ ε_p) ε_a ] (H_f^(q) [I-(I- _f^(q))^(t-1) ]^2v^(q)^* (v^(q)^* ) )H_f^(q) + + [ϵ¯p+(1+ϵ¯p)ϵ¯a]γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). [ ε_p+(1+ ε_p) ε_a ] γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). Recall (56), [t⊗t]=[(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+t, [ η_t η_t ]=E [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+ _t, we have [t⊗t] [ η_t η_t ] ⪰ [(−γ~t(q)(~t(q))⊤)[t−1⊗t−1](−γ~t(q)(~t(q))⊤)]+γ2(1+ϵ¯o)σ¯2f(q) [ (I-γ x_t^(q)( x_t^(q)) )E [ η_t-1 η_t-1 ] (I-γ x_t^(q)( x_t^(q)) ) ]+γ^2(1+ ε_o) σ^2H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]βtr(f(q)[−(−γf(q))(t−1)]2(q)∗((q)∗)⊤)f(q) γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] (H_f^(q) [I-(I- _f^(q))^(t-1) ]^2v^(q)^* (v^(q)^* ) )H_f^(q) + + γ2(1+ϵ¯o)[ϵ¯p+(1+ϵ¯p)ϵ¯a]γβ22tr(f(q)(−γf(q))2(t−1)f(q))‖(q)∗‖−(−γf(q))2(t−1)2f(q). γ^2(1+ ε_o) [ ε_p+(1+ ε_p) ε_a ] γβ^22tr (H_f^(q)(I- _f^(q))^2(t-1)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1)^2H_f^(q). ∎ D.2 Bias-Variance Decomposition Noticing that RN= R_N= 12⟨⊤,[¯N⊗¯N]⟩ 12 ,E [ η_N η_N ] (64) ≥ ≥ μmin((f(q))−1⊤)12⟨f(q),[¯N⊗¯N]⟩⏟RN(0), _ min ((H_f^(q))^-1SHS ) 12 _f^(q),E [ η_N η_N ] _R_N^(0), We then perform bias-variance decomposition for RN(0)R_N^(0). Lemma D.3. Under Assumption 3.1, Assumption 3.2, it holds RN(0)≥12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),[t⊗t]⟩.R_N^(0)≥ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),E[ η_t η_t] . Proof. By definition ¯N=1N∑t=0N−1t η_N= 1N _t=0^N-1 η_t, we have [¯N⊗¯N]= [ η_N η_N]= 1N2⋅(∑0≤k≤t≤N−1[t⊗k]+∑0≤t<k≤N−1[t⊗k]) 1N^2· ( _0≤ k≤ t≤ N-1E[ η_t η_k]+ _0≤ t<k≤ N-1E[ η_t η_k] ) (65) = = 1N2⋅(∑0≤k≤t≤N−1[[t⊗k|k]]+∑0≤t<k≤N−1[[t⊗k|t]]). 1N^2· ( _0≤ k≤ t≤ N-1E [E[ η_t η_k| η_k] ]+ _0≤ t<k≤ N-1E [E[ η_t η_k| η_t] ] ). By [t|t−1]=(−γf(q))t−1,E [ η_t| η_t-1 ]= (I- _f^(q) ) η_t-1, we have [¯N⊗¯N] [ η_N η_N] (66) = = 1N2⋅(∑0≤k≤t≤N−1[[t⊗k|k]]+∑0≤t<k≤N−1[[t⊗k|t]]) 1N^2· ( _0≤ k≤ t≤ N-1E [E[ η_t η_k| η_k] ]+ _0≤ t<k≤ N-1E [E[ η_t η_k| η_t] ] ) = = 1N2⋅(∑0≤k≤t≤N−1(−γf(q))t−k[k⊗k]+∑0≤t<k≤N−1[t⊗t](−γf(q))k−t) 1N^2· ( _0≤ k≤ t≤ N-1(I- _f^(q))^t-kE[ η_k η_k]+ _0≤ t<k≤ N-1E[ η_t η_t](I- _f^(q))^k-t ) ⪰ 1N2⋅∑t=0N−1∑k=tN−1(−γf(q))k−t[t⊗t]. 1N^2· _t=0^N-1 _k=t^N-1(I- _f^(q))^k-tE[ η_t η_t]. Applying into RN(0)R_N^(0), we have RN(0)= R_N^(0)= 12⟨f(q),[¯N⊗¯N]⟩ 12 _f^(q),E[ η_N η_N] ≥ ≥ 12N2⋅∑t=0N−1∑k=tN−1⟨f(q),(−γf(q))k−t[t⊗t]⟩ 12N^2· _t=0^N-1 _k=t^N-1 _f^(q),(I- _f^(q))^k-tE[ η_t η_t] = = 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),[t⊗t]⟩. 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),E[ η_t η_t] . ∎ Lemma D.4 (Bias-variance decomposition under general quantization, a lower bound). Under Assumption 3.1, 3.2, 3.3 and 3.4, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), it holds RN(0)≥12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t+¯t⟩,R_N^(0)≥ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t+ C_t , where ¯t=(ℐ−γ(q))∘¯t−1,¯0=[0⊗0], B_t=(I- ^(q)) B_t-1, B_0=E [ η_0 η_0 ], ¯t=(ℐ−γ(q))∘¯t−1+γ2σ¯G2f(q),¯0=, C_t=(I- ^(q)) C_t-1+γ^2 σ_G^2H_f^(q), C_0= 0, with σ¯G2=σ¯2+inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+inft([ϵt(a)2|at]+[ϵt(o)2|ot]). σ_G^2= σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t (E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] ). Proof. The proof is completed by Lemma D.1 and Lemma D.3. ∎ Lemma D.5 (Bias-variance decomposition under multiplicative quantization, a lower bound). If there exist ϵ¯p,ϵ¯a ε_p, ε_a and ϵ¯o ε_o such that for any i∈p,a,oi∈\p,a,o\, quantization iQ_i is ϵ¯i ε_i-multiplicative, then under Assumption 3.1, 3.2, 3.3 and 3.4, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), it holds RN(0)≥12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t(M)+¯t(M)⟩,R_N^(0)≥ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t^(M)+ C_t^(M) , where ¯t(M)= B_t^(M)= (ℐ−γ(q))∘¯t−1(M),¯0(M)=[0⊗0], (I- ^(q)) B_t-1^(M), B_0^(M)=E [ η_0 η_0 ], ¯t(M)⪰ C_t^(M) (ℐ−γ(q))∘¯t−1(M)+γ2(1+ϵ¯o)[σ¯2f(q)+(ϵ¯p+(1+ϵ¯p)ϵ¯a)β(t−1+t−1)],¯0(M)=, (I- ^(q)) C_t-1^(M)+γ^2(1+ ε_o) [ σ^2H_f^(q)+ ( ε_p+(1+ ε_p) ε_a )β (R_t-1+T_t-1 ) ], C_0^(M)= 0, with t:=tr(f(q)[−(−γf(q))t]2(q)∗((q)∗)⊤)f(q), _t:=tr (H_f^(q) [I-(I- _f^(q))^t ]^2v^(q)^* (v^(q)^* ) )H_f^(q), t:=γβ2tr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))2t2f(q). _t:= γβ2tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2t^2H_f^(q). Proof. The proof is completed by Lemma D.2 and Lemma D.3. ∎ D.3 Variance Lower Bounds In this section, we derive lower bounds for 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t(M)⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t^(M) and 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t . D.3.1 General Quantization Lemma D.6 (A crude lower bound of variance under general quantization). If the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), under Assumption 3.2, it holds ¯t⪰γσ¯G22(−(−γf(q))2t). C_t γ σ_G^22 (I- (I- _f^(q) )^2t ). Proof. Note that ℳ(q)−ℳ~(q)M^(q)- M^(q) is a PSD mapping and ¯t−1 C_t-1 is PSD, then by definition ¯t= C_t= (ℐ−γ(q))∘¯t−1+γ2σ¯G2f(q) (I- ^(q)) C_t-1+γ^2 σ_G^2H_f^(q) = = (ℐ−γ~(q))∘¯t−1+γ(~(q)−(q))∘¯t−1+γ2σ¯G2f(q) (I-γ T^(q)) C_t-1+γ( T^(q)-T^(q)) C_t-1+γ^2 σ_G^2H_f^(q) = = (ℐ−γ~(q))∘¯t−1+γ2(ℳ(q)−ℳ~(q))∘¯t−1+γ2σ¯G2f(q) (I-γ T^(q)) C_t-1+γ^2(M^(q)- M^(q)) C_t-1+γ^2 σ_G^2H_f^(q) ⪰ (ℐ−γ~(q))∘¯t−1+γ2σ¯G2f(q). (I-γ T^(q)) C_t-1+γ^2 σ_G^2H_f^(q). By solving recursion, it holds ¯t⪰ C_t γ2σ¯G2∑k=0t−1(ℐ−γ~(q))k∘f(q) γ^2 σ_G^2 _k=0^t-1(I-γ T^(q))^k _f^(q) = = γ2σ¯G2∑k=0t−1(−γf(q))kf(q)(−γf(q))k γ^2 σ_G^2 _k=0^t-1(I- _f^(q))^kH_f^(q)(I- _f^(q))^k = = γ2σ¯G2(−(−γf(q))2t)(2γ−γ2f(q))−1 γ^2 σ_G^2 (I- (I- _f^(q) )^2t ) (2 -γ^2H_f^(q) )^-1 ⪰ γσ¯G22(−(−γf(q))2t). γ σ_G^22 (I- (I- _f^(q) )^2t ). ∎ Lemma D.7 (A variance lower bound under general quantization). Suppose the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then under Assumption 3.2, for sufficiently large N>500N>500, 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩≥σ¯G250(k∗N+Nγ2∑i>k∗(λ~i(q))2), 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t ≥ σ_G^250 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ), where k∗:=maxk:λ~k(q)≥1γNk^*:= \k: λ_k^(q)≥ 1γ N\. Proof. By Lemma D.6, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t ≥ ≥ γσ¯G24N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),−(−γf(q))2t⟩ γ σ_G^24N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),I-(I- _f^(q))^2t = = σ¯G24N2∑t=0N−1⟨(−(−γf(q))N−t),−(−γf(q))2t⟩ σ_G^24N^2 _t=0^N-1 (I-(I- _f^(q))^N-t),I-(I- _f^(q))^2t = = σ¯G24N2∑i∑t=0N−1(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))2t) σ_G^24N^2 _i _t=0^N-1(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^2t) ≥ ≥ σ¯G24N2∑i∑t=0N−1(1−(1−γλ~i(q))N−t−1)(1−(1−γλ~i(q))t). σ_G^24N^2 _i _t=0^N-1(1-(1-γ λ_i^(q))^N-t-1)(1-(1-γ λ_i^(q))^t). Define f(x)=∑t=0N−1(1−(1−x)N−t−1)(1−(1−x)t),0<x<1.f(x)= _t=0^N-1(1-(1-x)^N-t-1)(1-(1-x)^t), 0<x<1. Note that if N≥500N≥ 500, f(x)≥N10,1N≤x<1,2N325x2,0<x<1N,f(x)≥ cases N10, 1N≤ x<1,\\ 2N^325x^2, 0<x< 1N, cases it follows that 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩≥ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t ≥ σ¯G24N2∑if(γλ~i(q)) σ_G^24N^2 _if(γ λ_i^(q)) ≥ ≥ σ¯G250(k∗N+Nγ2∑i>k∗(λ~i(q))2). σ_G^250 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ). ∎ D.3.2 Multiplicative Quantization Lemma D.8 (A crude lower bound of variance under multiplicative quantization). If the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then ¯t(M)⪰ C_t^(M) γ2(1+ϵ¯o)σ¯2(−(−γf(q))t) γ2(1+ ε_o) σ^2 (I-(I- _f^(q))^t ) + + γ2(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)[−(−γf(q))t/2]2(q)∗((q)∗)⊤)(−(−γf(q))t) γ2(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q) [I-(I- _f^(q))^t/2 ]^2v^(q)^* (v^(q)^* ) ) (I-(I- _f^(q))^t ) + + γ2β4(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))t2(−(−γf(q))t). γ^2β4(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^t^2 (I-(I- _f^(q))^t ). Proof. Similar to the general quantization case, ¯t(M)⪰(ℐ−γ~(q))∘¯t−1(M)+γ2(1+ϵ¯o)[σ¯2f(q)+(ϵ¯p+(1+ϵ¯p)ϵ¯a)β(t−1+t−1)]. C_t^(M) (I-γ T^(q)) C_t-1^(M)+γ^2(1+ ε_o) [ σ^2H_f^(q)+ ( ε_p+(1+ ε_p) ε_a )β (R_t-1+T_t-1 ) ]. By solving recursion, it holds ¯t(M)⪰ C_t^(M) ∑k=0t−1(ℐ−γ~(q))k∘γ2(1+ϵ¯o)[σ¯2f(q)+(ϵ¯p+(1+ϵ¯p)ϵ¯a)β(t−1−k+t−1−k)] _k=0^t-1(I-γ T^(q))^k γ^2(1+ ε_o) [ σ^2H_f^(q)+ ( ε_p+(1+ ε_p) ε_a )β (R_t-1-k+T_t-1-k ) ] = = γ2(1+ϵ¯o)∑k=0t−1(−γf(q))2k[σ¯2f(q)+(ϵ¯p+(1+ϵ¯p)ϵ¯a)β(t−1−k+t−1−k)]. γ^2(1+ ε_o) _k=0^t-1(I- _f^(q))^2k [ σ^2H_f^(q)+ ( ε_p+(1+ ε_p) ε_a )β (R_t-1-k+T_t-1-k ) ]. Regarding the time-independent noise σ¯2 σ^2, γ2σ¯2(1+ϵ¯o)∑k=0t−1(−γf(q))2kf(q)⪰γσ¯2(1+ϵ¯o)2(−(−γf(q))2t). γ^2 σ^2(1+ ε_o) _k=0^t-1(I- _f^(q))^2kH_f^(q) γ σ^2(1+ ε_o)2 (I- (I- _f^(q) )^2t ). Regarding the time-dependent term tR_t, ∑k=0t−1(−γf(q))2kt−1−k _k=0^t-1(I- _f^(q))^2kR_t-1-k = = ∑k=0t−1(−γf(q))2kf(q)tr(f(q)[−(−γf(q))t−1−k]2(q)∗((q)∗)⊤) _k=0^t-1(I- _f^(q))^2kH_f^(q)tr (H_f^(q) [I-(I- _f^(q))^t-1-k ]^2v^(q)^* (v^(q)^* ) ) ⪰ ∑k=0t/2−1(−γf(q))2kf(q)tr(f(q)[−(−γf(q))t−1−k]2(q)∗((q)∗)⊤) _k=0^t/2-1(I- _f^(q))^2kH_f^(q)tr (H_f^(q) [I-(I- _f^(q))^t-1-k ]^2v^(q)^* (v^(q)^* ) ) ⪰ ∑k=0t/2−1(−γf(q))2kf(q)tr(f(q)[−(−γf(q))t/2]2(q)∗((q)∗)⊤) _k=0^t/2-1(I- _f^(q))^2kH_f^(q)tr (H_f^(q) [I-(I- _f^(q))^t/2 ]^2v^(q)^* (v^(q)^* ) ) ⪰ 12γ[−(−γf(q))t]tr(f(q)[−(−γf(q))t/2]2(q)∗((q)∗)⊤). 12γ [I-(I- _f^(q))^t ]tr (H_f^(q) [I-(I- _f^(q))^t/2 ]^2v^(q)^* (v^(q)^* ) ). Regarding the time-dependent term tT_t, ∑k=0t−1(−γf(q))2kt−1−k _k=0^t-1(I- _f^(q))^2kT_t-1-k = = γβ2∑k=0t−1(−γf(q))2ktr(f(q)(−γf(q))2(t−1−k)f(q))‖(q)∗‖−(−γf(q))2(t−1−k)2f(q) γβ2 _k=0^t-1(I- _f^(q))^2ktr (H_f^(q)(I- _f^(q))^2(t-1-k)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1-k)^2H_f^(q) ⪰ γβ2∑k=0t/2−1(−γf(q))2ktr(f(q)(−γf(q))2(t−1−k)f(q))‖(q)∗‖−(−γf(q))2(t−1−k)2f(q) γβ2 _k=0^t/2-1(I- _f^(q))^2ktr (H_f^(q)(I- _f^(q))^2(t-1-k)H_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^2(t-1-k)^2H_f^(q) ⪰ γβ2∑k=0t/2−1(−γf(q))2ktr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))t2f(q) γβ2 _k=0^t/2-1(I- _f^(q))^2ktr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^t^2H_f^(q) ⪰ β4(−(−γf(q))t)tr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))t2. β4 (I-(I- _f^(q))^t )tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^t^2. Therefore, ¯t(M)⪰ C_t^(M) γ2(1+ϵ¯o)σ¯2(−(−γf(q))t) γ2(1+ ε_o) σ^2 (I-(I- _f^(q))^t ) + + γ2(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)[−(−γf(q))t/2]2(q)∗((q)∗)⊤)(−(−γf(q))t) γ2(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q) [I-(I- _f^(q))^t/2 ]^2v^(q)^* (v^(q)^* ) ) (I-(I- _f^(q))^t ) + + γ2β4(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))t2(−(−γf(q))t). γ^2β4(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^t^2 (I-(I- _f^(q))^t ). ∎ Lemma D.9 (A variance lower bound under multiplicative quantization). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, suppose the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then under Assumption 3.2, for sufficiently large N>500N>500, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t(M)⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t^(M) ≥ ≥ [(1+ϵ¯o)σ¯250+β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)2500(‖(q)∗‖f,0:k∗(q)2+N2γ2‖(q)∗‖(f,k∗:∞(q))32)]deffN [ (1+ ε_o) σ^250+ β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)2500 ( \|v^(q)^* \|_H_f,0:k^*^(q)^2+N^2γ^2 \|v^(q)^* \|_(H_f,k^*:∞^(q))^3^2 ) ] d_ effN + + γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)600(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2)⋅∑i(λ~i(q))2(1−γλ~i(q))2NdeffN, γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )600 ( \|v^(q)^* \|_I_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 )· _i( λ_i^(q))^2(1-γ λ_i^(q))^2N d_ effN, where deffN=k∗N+Nγ2∑i>k∗(λ~i(q))2 d_ effN= k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 and k∗:=maxk:λ~k(q)≥1γNk^*:= \k: λ_k^(q)≥ 1γ N\. Proof. Recall Lemma D.8, ¯t(M)⪰ C_t^(M) γ2(1+ϵ¯o)σ¯2(−(−γf(q))t)⏟1 γ2(1+ ε_o) σ^2 (I-(I- _f^(q))^t )_r_1 + + γ2(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)[−(−γf(q))t/2]2(q)∗((q)∗)⊤)(−(−γf(q))t)⏟2 γ2(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q) [I-(I- _f^(q))^t/2 ]^2v^(q)^* (v^(q)^* ) ) (I-(I- _f^(q))^t )_r_2 + + γ2β4(1+ϵ¯o)β(ϵ¯p+(1+ϵ¯p)ϵ¯a)tr(f(q)(−γf(q))2tf(q))‖(q)∗‖−(−γf(q))t2(−(−γf(q))t)⏟3. γ^2β4(1+ ε_o)β ( ε_p+(1+ ε_p) ε_a )tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) \|v^(q)^* \|_I-(I- _f^(q))^t^2 (I-(I- _f^(q))^t )_r_3. Regarding the term 1r_1, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),1⟩≥ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_1 ≥ γ(1+ϵ¯o)σ¯24N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),−(−γf(q))t⟩ γ(1+ ε_o) σ^24N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),I-(I- _f^(q))^t = = (1+ϵ¯o)σ¯24N2∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ (1+ ε_o) σ^24N^2 _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t = = (1+ϵ¯o)σ¯24N2∑i∑t=0N−1(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t) (1+ ε_o) σ^24N^2 _i _t=0^N-1(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^t) ≥ ≥ (1+ϵ¯o)σ¯24N2∑i∑t=0N−1(1−(1−γλ~i(q))N−t−1)(1−(1−γλ~i(q))t). (1+ ε_o) σ^24N^2 _i _t=0^N-1(1-(1-γ λ_i^(q))^N-t-1)(1-(1-γ λ_i^(q))^t). Define f(x)=∑t=0N−1(1−(1−x)N−t−1)(1−(1−x)t),0<x<1.f(x)= _t=0^N-1(1-(1-x)^N-t-1)(1-(1-x)^t), 0<x<1. Note that if N≥500N≥ 500, f(x)≥N10,1N≤x<1,2N325x2,0<x<1N,f(x)≥ cases N10, 1N≤ x<1,\\ 2N^325x^2, 0<x< 1N, cases (67) it follows that 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),1⟩≥ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_1 ≥ (1+ϵ¯o)σ¯24N2∑if(γλ~i(q)) (1+ ε_o) σ^24N^2 _if(γ λ_i^(q)) ≥ ≥ (1+ϵ¯o)σ¯250(k∗N+Nγ2∑i>k∗(λ~i(q))2). (1+ ε_o) σ^250 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ). Regarding the time-dependent term 2r_2, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),2⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_2 ≥ ≥ βγ(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)4N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),−(−γf(q))t⟩‖(q)∗‖f(q)[−(−γf(q))t/2]22 βγ(1+ ε_o)( ε_p+(1+ ε_p) ε_a)4N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),I-(I- _f^(q))^t \|v^(q)^* \|_H_f^(q) [I-(I- _f^(q))^t/2 ]^2^2 = = β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)4N2∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩‖(q)∗‖f(q)[−(−γf(q))t/2]22 β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)4N^2 _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t \|v^(q)^* \|_H_f^(q) [I-(I- _f^(q))^t/2 ]^2^2 = = β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)4N2∑t=0N−1∑i(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t)∑jλ~j(q)(1−(1−γλ~j(q))t/2)2ωj2 β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)4N^2 _t=0^N-1 _i(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^t) _j λ_j^(q) (1-(1-γ λ_j^(q))^t/2 )^2 _j^2 ≥ ≥ β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)4N2∑t=N/2N−1∑i(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t)∑jλ~j(q)(1−(1−γλ~j(q))t/2)2ωj2 β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)4N^2 _t=N/2^N-1 _i(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^t) _j λ_j^(q) (1-(1-γ λ_j^(q))^t/2 )^2 _j^2 ≥ ≥ β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)4N2∑t=N/2N−1∑i(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t)∑jλ~j(q)(1−(1−γλ~j(q))N/4)2ωj2 β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)4N^2 _t=N/2^N-1 _i(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^t) _j λ_j^(q) (1-(1-γ λ_j^(q))^N/4 )^2 _j^2 ≥ ≥ β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)8N2∑t=0N−1∑i(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t)∑jλ~j(q)(1−(1−γλ~j(q))N/4)2ωj2 β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)8N^2 _t=0^N-1 _i(1-(1-γ λ_i^(q))^N-t)(1-(1-γ λ_i^(q))^t) _j λ_j^(q) (1-(1-γ λ_j^(q))^N/4 )^2 _j^2 ≥ ≥ β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)100(k∗N+Nγ2∑i>k∗(λ~i(q))2)∑jλ~j(q)(1−(1−γλ~j(q))N/4)2ωj2, β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)100 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ) _j λ_j^(q) (1-(1-γ λ_j^(q))^N/4 )^2 _j^2, where ωj=((q)∗)⊤j(q) _j=(v^(q)^*) v_j^(q) with j(q)v_j^(q) being the eigenvectors of f(q)H_f^(q) and the last inequality reuses the property (67). Noticing that 1−(1−γλ~i(q))N4≥1−(1−1N)N4≥1−e−14≥15,λ~i(q)≥1γN,N4⋅γλ~i(q)−N(N−4)32⋅γ2(λ~i(q))2≥N5⋅γλ~i(q),λ~i(q)<1γN.1-(1-γ λ_i^(q)) N4≥ cases1-(1- 1N) N4≥ 1-e^- 14≥ 15,& λ_i^(q)≥ 1γ N,\\ N4·γ λ_i^(q)- N(N-4)32·γ^2( λ_i^(q))^2≥ N5·γ λ_i^(q),& λ_i^(q)< 1γ N. cases We have ∑jλ~j(q)(1−(1−γλ~j(q))N/4)2ωj2≥ _j λ_j^(q) (1-(1-γ λ_j^(q))^N/4 )^2 _j^2≥ ∑j≤k∗λ~j(q)25ωj2+∑j>k∗(λ~j(q))3N2γ225ωj2 _j≤ k^* λ_j^(q)25 _j^2+ _j>k^* ( λ_j^(q) )^3 N^2γ^225 _j^2 ≥ ≥ 125(‖(q)∗‖f,0:k∗(q)2+N2γ2‖(q)∗‖(f,k∗:∞(q))32). 125 ( \|v^(q)^* \|_H_f,0:k^*^(q)^2+N^2γ^2 \|v^(q)^* \|_(H_f,k^*:∞^(q))^3^2 ). Hence, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),2⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_2 ≥ ≥ β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)2500(k∗N+Nγ2∑i>k∗(λ~i(q))2)(‖(q)∗‖f,0:k∗(q)2+N2γ2‖(q)∗‖(f,k∗:∞(q))32). β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)2500 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ) ( \|v^(q)^* \|_H_f,0:k^*^(q)^2+N^2γ^2 \|v^(q)^* \|_(H_f,k^*:∞^(q))^3^2 ). Regarding the term 3r_3, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),3⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_3 = = γ2(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)8N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),−(−γf(q))t⟩ γ^2(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )8N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),I-(I- _f^(q))^t ⋅ · ‖(q)∗‖−(−γf(q))t2tr(f(q)(−γf(q))2tf(q)) \|v^(q)^* \|_I-(I- _f^(q))^t^2tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) = = γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)8N2∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )8N^2 _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t ⋅ · ‖(q)∗‖−(−γf(q))t2tr(f(q)(−γf(q))2tf(q)) \|v^(q)^* \|_I-(I- _f^(q))^t^2tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) ≥ ≥ γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)8N2∑t=N/2N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )8N^2 _t=N/2^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t ⋅ · ‖(q)∗‖−(−γf(q))t2tr(f(q)(−γf(q))2tf(q)) \|v^(q)^* \|_I-(I- _f^(q))^t^2tr (H_f^(q)(I- _f^(q))^2tH_f^(q) ) ≥ ≥ γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)8N2∑t=N/2N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )8N^2 _t=N/2^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t ⋅ · ‖(q)∗‖−(−γf(q))N/22tr(f(q)(−γf(q))2Nf(q)) \|v^(q)^* \|_I-(I- _f^(q))^N/2^2tr (H_f^(q)(I- _f^(q))^2NH_f^(q) ) ≥ ≥ γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)16N2∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩ γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )16N^2 _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t ⋅ · ‖(q)∗‖−(−γf(q))N/22tr(f(q)(−γf(q))2Nf(q)). \|v^(q)^* \|_I-(I- _f^(q))^N/2^2tr (H_f^(q)(I- _f^(q))^2NH_f^(q) ). Noticing that 1−(1−γλ~i(q))N2≥1−(1−1N)N2≥1−e−12≥13,λ~i(q)≥1γN,N2⋅γλ~i(q)−N(N−2)8⋅γ2(λ~i(q))2≥N3⋅γλ~i(q),λ~i(q)<1γN,1-(1-γ λ_i^(q)) N2≥ cases1-(1- 1N) N2≥ 1-e^- 12≥ 13, λ_i^(q)≥ 1γ N,\\ N2·γ λ_i^(q)- N(N-2)8·γ^2( λ_i^(q))^2≥ N3·γ λ_i^(q), λ_i^(q)< 1γ N, cases it holds ‖(q)∗‖−(−γf(q))N/22=∑i(1−(1−γλ~i(q))N/2)ωi2≥13(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2). \|v^(q)^* \|_I-(I- _f^(q))^N/2^2= _i(1-(1-γ λ_i^(q))^N/2) _i^2≥ 13 ( \|v^(q)^* \|_I_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ). Recall that ∑t=0N−1⟨−(−γf(q))N−t,−(−γf(q))t⟩= _t=0^N-1 -(I- _f^(q))^N-t,I-(I- _f^(q))^t = ∑i∑t=0N−1(1−(1−γλ~i(q))N−t)(1−(1−γλ~i(q))t) _i _t=0^N-1 (1-(1-γ λ_i^(q))^N-t ) (1-(1-γ λ_i^(q))^t ) ≥ ≥ 225N2(k∗N+Nγ2∑i>k∗(λ~i(q))2), 225N^2 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ), we have 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),3⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q),r_3 ≥ ≥ γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)600(k∗N+Nγ2∑i>k∗(λ~i(q))2)(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2) γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )600 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ) ( \|v^(q)^* \|_I_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ) ⋅ · ∑i(λ~i(q))2(1−γλ~i(q))2N. _i( λ_i^(q))^2(1-γ λ_i^(q))^2N. Therefore, 12N2∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t(M)⟩ 12N^2 _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), C_t^(M) ≥ ≥ [(1+ϵ¯o)σ¯250+β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)2500(‖(q)∗‖f,0:k∗(q)2+N2γ2‖(q)∗‖(f,k∗:∞(q))32)]deffN [ (1+ ε_o) σ^250+ β(1+ ε_o)( ε_p+(1+ ε_p) ε_a)2500 ( \|v^(q)^* \|_H_f,0:k^*^(q)^2+N^2γ^2 \|v^(q)^* \|_(H_f,k^*:∞^(q))^3^2 ) ] d_ effN + + γ(1+ϵ¯o)β2(ϵ¯p+(1+ϵ¯p)ϵ¯a)600(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2)⋅∑i(λ~i(q))2(1−γλ~i(q))2NdeffN, γ(1+ ε_o)β^2 ( ε_p+(1+ ε_p) ε_a )600 ( \|v^(q)^* \|_I_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 )· _i( λ_i^(q))^2(1-γ λ_i^(q))^2N d_ effN, where deffN=k∗N+Nγ2∑i>k∗(λ~i(q))2 d_ effN= k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2. ∎ D.4 Bias Lower Bounds Recall that we have the following lower bound on the bias error. For general quantization, 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩= 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t = 12γN2⋅∑t=0N−1⟨(−(−γf(q))N−t),¯t⟩ 12γ N^2· _t=0^N-1 (I-(I- _f^(q))^N-t ), B_t (68) ≥ ≥ 12γN2⋅∑t=0N/2⟨(−(−γf(q))N−t),¯t⟩ 12γ N^2· _t=0^N/2 (I-(I- _f^(q))^N-t ), B_t ≥ ≥ 12γN2⋅⟨(−(−γf(q))N/2),∑t=0N/2¯t⟩. 12γ N^2· (I-(I- _f^(q))^N/2 ), _t=0^N/2 B_t . Let ¯n:=∑t=0n−1¯t S_n:= _t=0^n-1 B_t and ¯n(M):=∑t=0n−1¯t(M) S_n^(M):= _t=0^n-1 B_t^(M). Then the remaining challenge is to lower bound ¯N/2+1 S_N/2+1 and ¯N/2+1(M) S_N/2+1^(M). D.4.1 General Quantization Lemma D.10. Suppose Assumption 3.2 and 3.3 holds, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then ¯n⪰β4tr([−(−γf(q))n/2]¯0)[−(−γf(q))n/2]+∑t=0n−1(−γf(q))t¯0(−γf(q))t. S_n β4tr ( [I-(I- _f^(q))^n/2 ] B_0 ) [I-(I- _f^(q))^n/2 ]+ _t=0^n-1(I- _f^(q))^t B_0(I- _f^(q))^t. Proof. We first build a crude bound. By the definition of ¯t B_t, ¯n⪰∑t=0n−1(ℐ−γ(q))t∘¯0⪰∑t=0n−1(ℐ−γ~(q))t∘¯0=∑t=0n−1(−γf(q))t¯0(−γf(q))t. S_n _t=0^n-1(I- ^(q))^t B_0 _t=0^n-1(I-γ T^(q))^t B_0= _t=0^n-1(I- _f^(q))^t B_0(I- _f^(q))^t. Further by Assumption 3.3, we have (ℳ(q)−ℳ~(q))∘¯n⪰ (M^(q)- M^(q)) S_n βtr(f(q)¯n)f(q) (H_f^(q) S_n )H_f^(q) ⪰ βtr(f(q)∑t=0n−1(−γf(q))t¯0(−γf(q))t)f(q) (H_f^(q) _t=0^n-1(I- _f^(q))^t B_0(I- _f^(q))^t )H_f^(q) ⪰ βtr(f(q)∑t=0n−1(−2γf(q))t¯0)f(q) (H_f^(q) _t=0^n-1(I-2 _f^(q))^t B_0 )H_f^(q) = = β2γtr([−(−2γf(q))n]¯0)f(q) β2γtr ( [I-(I-2 _f^(q))^n ] B_0 )H_f^(q) ⪰ β2γtr([−(−γf(q))n]¯0)f(q). β2γtr ( [I-(I- _f^(q))^n ] B_0 )H_f^(q). Next we use the above inequality to build a refined lower bound. ¯n= S_n= (ℐ−γ~(q))∘¯n−1+γ2(ℳ(q)−ℳ~(q))∘¯n−1+¯0 (I-γ T^(q)) S_n-1+γ^2(M^(q)- M^(q)) S_n-1+ B_0 ⪰ (ℐ−γ~(q))∘¯n−1+γ2β2γtr([−(−γf(q))n−1]¯0)f(q)+¯0. (I-γ T^(q)) S_n-1+γ^2 β2γtr ( [I-(I- _f^(q))^n-1 ] B_0 )H_f^(q)+ B_0. Solving the recursion yields ¯n⪰ S_n ∑t=0n−1(ℐ−γ~(q))t∘βγ2tr([−(−γf(q))n−1−t]¯0)f(q)+¯0 _t=0^n-1(I-γ T^(q))^t \ βγ2tr ( [I-(I- _f^(q))^n-1-t ] B_0 )H_f^(q)+ B_0 \ = = βγ2∑t=0n−1tr([−(−γf(q))n−1−t]¯0)(−γf(q))2tf(q)+∑t=0n−1(−γf(q))t¯0(−γf(q))t. βγ2 _t=0^n-1tr ( [I-(I- _f^(q))^n-1-t ] B_0 )(I- _f^(q))^2tH_f^(q)+ _t=0^n-1(I- _f^(q))^t B_0(I- _f^(q))^t. For the first term, noticing the following: ∑t=0n−1tr([−(−γf(q))n−1−t]¯0)(−γf(q))2tf(q) _t=0^n-1tr ( [I-(I- _f^(q))^n-1-t ] B_0 )(I- _f^(q))^2tH_f^(q) ⪰ ∑t=0n−1tr([−(−γf(q))n−1−t]¯0)(−2γf(q))tf(q) _t=0^n-1tr ( [I-(I- _f^(q))^n-1-t ] B_0 )(I-2 _f^(q))^tH_f^(q) ⪰ ∑t=0n/2−1tr([−(−γf(q))n−1−t]¯0)(−2γf(q))tf(q) _t=0^n/2-1tr ( [I-(I- _f^(q))^n-1-t ] B_0 )(I-2 _f^(q))^tH_f^(q) ⪰ tr([−(−γf(q))n/2]¯0)∑t=0n/2−1(−2γf(q))tf(q) ( [I-(I- _f^(q))^n/2 ] B_0 ) _t=0^n/2-1(I-2 _f^(q))^tH_f^(q) = = 12γtr([−(−γf(q))n/2]¯0)[−(−2γf(q))n/2] 12γtr ( [I-(I- _f^(q))^n/2 ] B_0 ) [I-(I-2 _f^(q))^n/2 ] ⪰ 12γtr([−(−γf(q))n/2]¯0)[−(−γf(q))n/2], 12γtr ( [I-(I- _f^(q))^n/2 ] B_0 ) [I-(I- _f^(q))^n/2 ], The proof is immediately completed. ∎ Lemma D.11 (A bias lower bound under general quantization). Suppose Assumption 3.2, 3.3 holds, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t ≥ ≥ 1100γ2N2(‖(q)∗‖(f,0:k∗(q))−12+N2γ2‖(q)∗‖f,k∗:∞(q)2) 1100γ^2N^2 (\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+N^2γ^2\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) + + β1000γN2(‖(q)∗‖f,0:k∗(q)2+γN‖(q)∗‖f,k∗:∞(q)2)(k∗+γ2N2∑i>k∗(λ~i(q))2). β1000γ N^2 (\|v^(q)^*\|_I_f,0:k^*^(q)^2+γ N\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) (k^*+γ^2N^2 _i>k^*( λ_i^(q))^2 ). Proof. According to (68) and Lemma D.10, 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t⟩≥12γN2⋅⟨−(−γf(q))N/2,∑t=0N/2¯t⟩ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t ≥ 12γ N^2· -(I- _f^(q))^N/2, _t=0^N/2 B_t ≥12γN2⋅β4tr([−(−γf(q))N/4]¯0)⟨−(−γf(q))N/2,−(−γf(q))N/4⟩⏟I1 ≥ 12γ N^2· β4tr ( [I-(I- _f^(q))^N/4 ] B_0 ) -(I- _f^(q))^N/2,I-(I- _f^(q))^N/4 _I_1 +12γN2⟨−(−γf(q))N/2,∑t=0N/2−1(−γf(q))t¯0(−γf(q))t⟩⏟I2. + 12γ N^2 -(I- _f^(q))^N/2, _t=0^N/2-1(I- _f^(q))^t B_0(I- _f^(q))^t _I_2. The first term I1I_1 can be lower bounded by I1≥ I_1≥ β8γN2tr([−(−γf(q))N/4]¯0)tr([−(−γf(q))N/4]2) β8γ N^2tr ( [I-(I- _f^(q))^N/4 ] B_0 )tr ( [I-(I- _f^(q))^N/4 ]^2 ) = = β8γN2(∑i[1−(1−γλ~i(q))N/4]ωi2)(∑i[1−(1−γλ~i(q))N/4]2), β8γ N^2 ( _i [1-(1-γ λ_i^(q))^N/4 ] _i^2 ) ( _i [1-(1-γ λ_i^(q))^N/4 ]^2 ), where ωi=(0−(q)∗)⊤i(q) _i=(v_0-v^(q)^*) v_i^(q) with i(q)v_i^(q) being the eigenvectors of f(q)H_f^(q). The second term I2I_2 can be lower bounded by I2= I_2= 12γN2⟨∑t=0N/2−1(−γf(q))2t[−(−γf(q))N/2],¯0⟩ 12γ N^2 _t=0^N/2-1(I- _f^(q))^2t [I-(I- _f^(q))^N/2 ], B_0 ≥ ≥ 12γN2⟨∑t=0N/2−1(−2γf(q))t[−(−γf(q))N/2],¯0⟩ 12γ N^2 _t=0^N/2-1(I-2 _f^(q))^t [I-(I- _f^(q))^N/2 ], B_0 ≥ ≥ 14γ2N2⟨f(q)−1[−(−γf(q))N/2]2,¯0⟩ 14γ^2N^2 H_f^(q)^-1 [I-(I- _f^(q))^N/2 ]^2, B_0 ≥ ≥ 14γ2N2⟨f(q)−1[−(−γf(q))N/4]2,¯0⟩ 14γ^2N^2 H_f^(q)^-1 [I-(I- _f^(q))^N/4 ]^2, B_0 = = 14γ2N2∑i(λ~i(q))−1(1−(1−γλ~i(q))N/4)2ωi2. 14γ^2N^2 _i( λ_i^(q))^-1 (1-(1-γ λ_i^(q))^N/4 )^2 _i^2. Noticing that 1−(1−γλ~i(q))N4≥1−(1−1N)N4≥1−e−14≥15,λ~i(q)≥1γN,N4⋅γλ~i(q)−N(N−4)32⋅γ2(λ~i(q))2≥N5⋅γλ~i(q),λ~i(q)<1γN.1-(1-γ λ_i^(q)) N4≥ cases1-(1- 1N) N4≥ 1-e^- 14≥ 15,& λ_i^(q)≥ 1γ N,\\ N4·γ λ_i^(q)- N(N-4)32·γ^2( λ_i^(q))^2≥ N5·γ λ_i^(q),& λ_i^(q)< 1γ N. cases Plugging this into I1I_1 and I2I_2 yields I1≥ I_1≥ β8γN2(∑i≤k∗15ωi2+∑i>k∗γN5λ~i(q)ωi2)(k∗25+γ2N225∑i>k∗(λ~i(q))2) β8γ N^2 ( _i≤ k^* 15 _i^2+ _i>k^* γ N5 λ_i^(q) _i^2 ) ( k^*25+ γ^2N^225 _i>k^*( λ_i^(q))^2 ) = = β1000γN2(‖(q)∗‖f,0:k∗(q)2+γN‖(q)∗‖f,k∗:∞(q)2)(k∗+γ2N2∑i>k∗(λ~i(q))2), β1000γ N^2 (\|v^(q)^*\|_I_f,0:k^*^(q)^2+γ N\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) (k^*+γ^2N^2 _i>k^*( λ_i^(q))^2 ), and I2≥ I_2≥ 14γ2N2(∑i≤k∗(λ~i(q))−1125ωi2+∑i>k∗λ~i(q)N2γ225ωi2) 14γ^2N^2 ( _i≤ k^*( λ_i^(q))^-1 125 _i^2+ _i>k^* λ_i^(q) N^2γ^225 _i^2 ) = = 1100γ2N2(‖(q)∗‖(f,0:k∗(q))−12+N2γ2‖(q)∗‖f,k∗:∞(q)2). 1100γ^2N^2 (\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+N^2γ^2\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ). ∎ D.4.2 Multiplicative Quantization Lemma D.12. Suppose Assumption 3.2 and 3.3 holds, if the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then ¯n(M)⪰β4tr([−(−γf(q))n/2]¯0(M))[−(−γf(q))n/2]+∑t=0n−1(−γf(q))t¯0(M)(−γf(q))t. S_n^(M) β4tr ( [I-(I- _f^(q))^n/2 ] B_0^(M) ) [I-(I- _f^(q))^n/2 ]+ _t=0^n-1(I- _f^(q))^t B_0^(M)(I- _f^(q))^t. Proof. As ¯n(M) S_n^(M) and ¯n S_n have the same update rule, they own the same lower bound. ∎ Lemma D.13 (A bias lower bound under multiplicative quantization). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, suppose the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then under Assumption 3.2, 3.3, 12N2⋅∑t=0N−1∑k=tN−1⟨(−γf(q))k−tf(q),¯t(M)⟩ 12N^2· _t=0^N-1 _k=t^N-1 (I- _f^(q))^k-tH_f^(q), B_t^(M) ≥ ≥ 1100γ2N2(‖(q)∗‖(f,0:k∗(q))−12+N2γ2‖(q)∗‖f,k∗:∞(q)2) 1100γ^2N^2 (\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+N^2γ^2\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) + + β1000γN2(‖(q)∗‖f,0:k∗(q)2+γN‖(q)∗‖f,k∗:∞(q)2)(k∗+γ2N2∑i>k∗(λ~i(q))2). β1000γ N^2 (\|v^(q)^*\|_I_f,0:k^*^(q)^2+γ N\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) (k^*+γ^2N^2 _i>k^*( λ_i^(q))^2 ). Proof. The proof is the same as the proof for Lemma D.11. ∎ D.5 Final Lower Bounds D.5.1 General Quantization Theorem D.1. Suppose the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then under Assumption 3.1, 3.2, 3.3 and 3.4, for sufficiently large N>500N>500, RN(0)≥BiasError+VarianceError,R_N^(0) +VarianceError, where BiasError≥ ≥ 1100γ2N2(‖(q)∗‖(f,0:k∗(q))−12+N2γ2‖(q)∗‖f,k∗:∞(q)2), 1100γ^2N^2 (\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+N^2γ^2\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ), VarianceError≥ ≥ β1000γN2(‖(q)∗‖f,0:k∗(q)2+γN‖(q)∗‖f,k∗:∞(q)2)(k∗+γ2N2∑i>k∗(λ~i(q))2) β1000γ N^2 (\|v^(q)^*\|_I_f,0:k^*^(q)^2+γ N\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ) (k^*+γ^2N^2 _i>k^*( λ_i^(q))^2 ) + + σ¯G250(k∗N+Nγ2∑i>k∗(λ~i(q))2), σ_G^250 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ), where k∗=maxi:λ~i(q)≥1/(γN)k^*= \i: λ_i^(q)≥ 1/(γ N)\, and σ¯G2=σ¯2+inftβtr(f(q)[ϵt−1(p)ϵt−1(p)⊤])+inft[ϵt(a)2|at]+[ϵt(o)2|ot]. σ_G^2= σ^2+ _t (H_f^(q)E [ ε_t-1^(p) ε_t-1^(p) ] )+ _t \E [ _t^(a)^2 |a_t ]+E [ _t^(o)^2 |o_t ] \. Proof. The proof can be completed by Lemma D.4, Lemma D.7 and Lemma D.11. ∎ D.5.2 Multiplicative Quantization Theorem D.2. For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, suppose the stepsize γ<1λ~1(q)γ< 1 λ_1^(q), then under Assumption 3.1, 3.2, 3.3 and 3.4, for sufficiently large N>500N>500, RN(0)≥BiasError+VarianceError,R_N^(0) +VarianceError, where BiasError≥ ≥ 1100γ2N2(‖(q)∗‖(f,0:k∗(q))−12+N2γ2‖(q)∗‖f,k∗:∞(q)2), 1100γ^2N^2 (\|v^(q)^*\|_(H_f,0:k^*^(q))^-1^2+N^2γ^2\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ), VarianceError≥ ≥ (1+ϵ¯o)σ¯250(k∗N+Nγ2∑i>k∗(λ~i(q))2) (1+ ε_o) σ^250 ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ) + + β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)(k∗N+Nγ2∑i>k∗(λ~i(q))2)Peff β(1+ ε_o)( ε_p+(1+ ε_p) ε_a) ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 )P_ eff + + β1000(1Nγ‖(q)∗‖f,0:k∗(q)2+‖(q)∗‖f,k∗:∞(q)2)(k∗N+Nγ2∑i>k∗(λ~i(q))2), β1000 ( 1Nγ \|v^(q)^* \|_I_f,0:k^*^(q)^2+ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ) ( k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 ), where k∗=maxi:λ~i(q)≥1/(γN)k^*= \i: λ_i^(q)≥ 1/(γ N)\ and Peff= P_ eff= 12500(‖(q)∗‖f,0:k∗(q)2+N2γ2‖(q)∗‖(f,k∗:∞(q))32) 12500 ( \|v^(q)^* \|_H_f,0:k^*^(q)^2+N^2γ^2 \|v^(q)^* \|_(H_f,k^*:∞^(q))^3^2 ) + + γβ600(‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2)∑i(λ~i(q))2(1−γλ~i(q))2N. γβ600 ( \|v^(q)^* \|_I_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ) _i( λ_i^(q))^2(1-γ λ_i^(q))^2N. Proof. The proof can be completed by Lemma D.5, Lemma D.9 and Lemma D.13. ∎ D.6 Additive Error Lower Bounds under Power-law Spectrum We first analyze the additive error in Lemma B.2 and take expectation on ∗w^*: AdditiveError= = 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩ 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) + + [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗]. [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ]. Recall that ∗=(⊤)−1∗,(q)∗=(f(q))−1∗.v^*= (SHS )^-1SHw^*, v^(q)^*=(H_f^(q))^-1SHw^*. Denote =f(q)−⊤D=H_f^(q)-SHS , then (q)∗=(⊤+)−1⊤∗.v^(q)^*= (SHS +D )^-1SHS v^*. It follows that ∗−(q)∗=(⊤+)−1∗.v^*-v^(q)^*= (SHS +D )^-1Dv^*. Hence, 12⟨⊤,((q)∗−∗)⊗((q)∗−∗)⟩=12‖∗‖12, 12 ,(v^(q)^*-v^*) (v^(q)^*-v^*) = 12 \|w^* \|_S_1^2, (69) where 1= _1= ⊤(⊤)−1(⊤+)−1⊤(⊤+)−1(⊤)−1. (SHS )^-1D (SHS +D )^-1SHS (SHS +D )^-1D (SHS )^-1SH. Lemma D.14 (Additive Error under multiplicative quantization, a lower bound). Under Assumption 3.1, 3.2 and 3.5, for any i∈s,d,fi∈\s,d,f\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then ∗‖∗‖12≳[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2. _w^* \|w^* \|_S_1^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2. Further if f(q)H_f^(q) and ⊤SHS are commutative, [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≳ 1N(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). 1N(1+ ε_d)(1+ ε_f)(1+ ε_s) (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s). Proof. Regarding the first inequality, noticing that [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]⊤⪯, [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]SHS , by Assumption 3.5, ∗‖∗‖12= _w^* \|w^* \|_S_1^2= tr(1) (S_1) ≥ ≥ [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2tr(2⊤(⊤)−1) [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2tr (SH^2S (SHS )^-1 ) ≳ [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2, [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2, where the first inequality holds by Lemma F.5, and the last inequality holds by Lemma G.1 and Von Neumann’s trace inequality: tr(2⊤(⊤)−1)≳∑i−a≂1.tr (SH^2S (SHS )^-1 ) _ii^-a 1. Regarding the second inequality, by Assumption 3.5, [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] (70) = = [∗⊤⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−1∗] [w^* HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SHw^* ] = = tr(⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−1). (HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SH ). Noticing that (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤⪯f(q)⪯(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤,(1+ ε_f)(1+ ε_d)(1+ ε_s)SHS _f^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS , and f(q)H_f^(q) and ⊤SHS are commutative, it holds tr(⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−1) (HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SH ) (71) ≥ ≥ (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)tr(⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1) (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s)tr (HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^-1SH ) ≥ ≥ (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)tr((f(q))−122⊤(f(q))−12) (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s)tr ((H_f^(q))^- 12SH^2S (H_f^(q))^- 12 ) ⋅ · μmin((f(q))−121Nγ[−(−γf(q))N](f(q))−12). _ ((H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 ). Firstly, regarding μmin((f(q))−121Nγ[−(−γf(q))N](f(q))−12)=1Nmini1−(1−γλ~i(q))Nγλ~i(q), _ ((H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 )= 1N _i 1- (1-γ λ_i^(q) )^Nγ λ_i^(q), note that f(x)=1−(1−x)Nxf(x)= 1-(1-x)^Nx is decreasing in (0,1)(0,1), it holds μmin((f(q))−121Nγ[−(−γf(q))N](f(q))−12)≥1N. _ ((H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 )≥ 1N. (72) Secondly, by Von Neumann’s trace inequality, Lemma G.1 and Lemma G.2, tr((f(q))−122⊤(f(q))−12)≥ ((H_f^(q))^- 12SH^2S (H_f^(q))^- 12 )≥ ∑iμi(2⊤)λ~i(q) _i _i(SH^2S ) λ_i^(q) (73) ≳ ∑i−2a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)i−a _i i^-2a(1+ ε_d)(1+ ε_f)(1+ ε_s)i^-a ≂ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). 1(1+ ε_d)(1+ ε_f)(1+ ε_s). Hence, together with (70), (71), (72) and (73), [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≳ 1N(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). 1N(1+ ε_d)(1+ ε_f)(1+ ε_s) (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s). ∎ Lemma D.15 (Additive Error under additive quantization, a lower bound). Under Assumption 3.1, 3.2, 3.5, for any i∈s,d,fi∈\s,d,f\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then with probability at least 1−e−Ω(M)1-e^- (M), ∗‖∗‖12≳(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(1+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2. _w^* \|w^* \|_S_1^2 ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (1+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2. Further if f(q)H_f^(q) and ⊤SHS are commutative, [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≳ 1NM−aM−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+1+ϵ¯s. 1N M^-aM^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s ε_dp+ ε_f+ ε_d pM+1+ ε_s. Proof. Regarding the first inequality, noticing that with probability at least 1−e−Ω(M)1-e^- (M), ≿(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)⏟δmin,D ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )_ _ minI, by Assumption 3.5 we have ∗‖∗‖12= _w^* \|w^* \|_S_1^2= tr(1) (S_1) ≥ ≥ δmin2(μmax(⊤)+δmin)2tr(⊤(⊤)−1) _ min^2( _ (SHS )+ _ min)^2tr (HS (SHS )^-1SH ) ≂ δmin2(1+δmin)2tr(⊤(⊤)−1) _ min^2(1+ _ min)^2tr (HS (SHS )^-1SH ) ≳ (ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(1+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2, ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (1+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2, where the first inequality holds by Lemma F.6, the last equality holds by Lemma G.1 and the last inequality holds by Lemma G.1 and the Von Neumann’s trace inequality: tr(2⊤(⊤)−1)≳∑i−a≂1.tr (SH^2S (SHS )^-1 ) _ii^-a 1. Regarding the second inequality, noticing that ⊤+(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)≾f(q)≾⊤+(ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM),SHS + ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )I _f^(q) + ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )I, and f(q)H_f^(q) and ⊤SHS are commutative, it holds tr(⊤(f(q))−11Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(f(q))−1) (HS (H_f^(q))^-1 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )(H_f^(q))^-1SH ) (74) ≥ ≥ tr((f(q))−122⊤(f(q))−12)⋅μmin((f(q))−12(f(q)−⊤)(f(q))−12) ((H_f^(q))^- 12SH^2S (H_f^(q))^- 12 )· _ ((H_f^(q))^- 12 (H_f^(q)-SHS )(H_f^(q))^- 12 ) ⋅ · μmin((f(q))−121Nγ[−(−γf(q))N](f(q))−12). _ ((H_f^(q))^- 12 1Nγ [I- (I- _f^(q) )^N ](H_f^(q))^- 12 ). Firstly, by Lemma F.2 and Lemma G.1, μmin((f(q))−12(f(q)−⊤)(f(q))−12)≳ϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+1+ϵ¯s. _ ((H_f^(q))^- 12 (H_f^(q)-SHS )(H_f^(q))^- 12 ) ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s ε_dp+ ε_f+ ε_d pM+1+ ε_s. (75) Secondly, by Von Neumann’s trace inequality, Lemma G.1 and Lemma G.4, tr((f(q))−122⊤(f(q))−12)≥ ((H_f^(q))^- 12SH^2S (H_f^(q))^- 12 )≥ ∑iμi(2⊤)λ~i(q) _i _i(SH^2S ) λ_i^(q) (76) ≳ ∑i−2ai−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpM _i i^-2ai^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM ≳ mini−ai−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpM i^-ai^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM = = M−aM−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpM. M^-aM^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM. Hence, together with (70), (72), (74), (75) and (76), [((q)∗)⊤1Nγ[−(−γf(q))N](f(q))−1(f(q)−⊤)(q)∗] [ (v^(q)^* ) 1Nγ [I- (I- _f^(q) )^N ] (H_f^(q) )^-1 (H_f^(q)-SHS )v^(q)^* ] ≳ 1NM−aM−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+1+ϵ¯s. 1N M^-aM^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s ε_dp+ ε_f+ ε_d pM+1+ ε_s. ∎ D.7 Variance Lower Bounds under Power-Law Spectrum D.7.1 Multiplicative Quantization Lemma D.16 (A variance lower bound under multiplicative quantization, power-law spectrum). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, Assumption 3.2 and Assumption 3.5, for sufficiently large N>500N>500, with probability at least 1−e−Ω(M)1-e^- (M), k∗N+Nγ2∑i>k∗(λ~i(q))2≳minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1aN. k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 \M, [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s) ] 1a \N. Proof. By Lemma G.3, with probability at least 1−e−Ω(M)1-e^- (M), for j∈[M]j∈[M], μj(f(q))≳(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a. _j(H_f^(q)) (1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a. Hence, denote k0∗=maxk:(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)k−a≥1Nγk_0^*= \k:(1+ ε_f)(1+ ε_d)(1+ ε_s)k^-a≥ 1Nγ\, then with probability at least 1−e−Ω(M)1-e^- (M), it holds k∗N+Nγ2∑i>k∗(λ~i(q))2≳ k^*N+Nγ^2 _i>k^*( λ_i^(q))^2 k∗N+Nγ2[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]2∑i>k∗i−2a k^*N+Nγ^2[(1+ ε_f)(1+ ε_d)(1+ ε_s)]^2 _i>k^*i^-2a (77) ≳ k0∗N+Nγ2[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]2∑i>k0∗i−2a k_0^*N+Nγ^2[(1+ ε_f)(1+ ε_d)(1+ ε_s)]^2 _i>k_0^*i^-2a ≂ minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1aN. \M, [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s) ] 1a \N. ∎ D.7.2 Additive Quantization Lemma D.17 (A variance lower bound under additive quantization, power-law spectrum). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, Assumption 3.2 and Assumption 3.5, for sufficiently large N>500N>500, with probability at least 1−e−Ω(M)1-e^- (M), k∗N+Nγ2∑i>k∗(λ~i(q))2≳k¯eff+γ2N2(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2(M−k¯eff)N, k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 k_ eff+γ^2N^2 ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2(M- k_ eff)N, where k¯eff=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a. k_ eff= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a. Proof. By Lemma G.5, with probability at least 1−e−Ω(M)1-e^- (M), μj(f(q))≳j−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. _j(H_f^(q)) j^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Denote k0∗=maxj:j−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM≥1Nγk_0^*= \j:j^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM≥ 1Nγ\. Then k∗N+Nγ2∑i>k∗(λ~i(q))2≳ k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 k∗N+Nγ2∑i>k∗(i−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2 k^*N+Nγ^2 _i>k^* (i^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 (78) ≳ k0∗N+Nγ2∑i>k0∗(i−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2. k_0^*N+Nγ^2 _i>k_0^* (i^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2. We then consider two cases to complete the proof. • M−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM<1NγM^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM< 1Nγ Denote Neff(A)=(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)−1a.N_ eff^(A)= ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM )^- 1a. Then with probability at least 1−e−Ω(M)1-e^- (M), k0∗N+Nγ2∑i>k0∗(i−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2≳ k_0^*N+Nγ^2 _i>k_0^* (i^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 k0∗N+Nγ2∑i>k0∗[i−2a+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2] k_0^*N+Nγ^2 _i>k_0^* [i^-2a+ ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 ] (79) ≳ Neff(A)+N2γ2(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2(M−Neff(A))N. N_ eff^(A)+N^2γ^2 ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2 (M-N_ eff^(A) )N. • M−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM≥1NγM^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM≥ 1Nγ k0∗N+Nγ2∑i>k0∗(i−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2=MN. k_0^*N+Nγ^2 _i>k_0^* (i^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2= MN. (80) Hence, together with (78), (79) and (80), with probability at least 1−e−Ω(M)1-e^- (M), k∗N+Nγ2∑i>k∗(λ~i(q))2≳k¯eff+γ2N2(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2(M−k¯eff)N, k^*N+Nγ^2 _i>k^* ( λ_i^(q) )^2 k_ eff+γ^2N^2 ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2(M- k_ eff)N, where k¯eff=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a. k_ eff= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a. ∎ D.8 Bias Lower Bounds under Power-Law Spectrum Lemma D.18. Under Assumption 3.2 and Assumption 3.5, with probability at least 1−e−Ω(M)1-e^- (M), ∗[‖(q)∗‖f,k∗:∞(q)2]≥∑i>k∗(λ~i(q))−1μi(2⊤). _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ]≥ _i>k^*( λ_i^(q))^-1 _i(SH^2S ). Proof. Recall that (q)∗=(f(q))−1∗,v^(q)^*=(H_f^(q))^-1SHw^*, it follows that ∗‖(q)∗‖f,k∗:∞(q)2= _w^*\|v^(q)^*\|_H_f,k^*:∞^(q)^2= ∗tr(f,k∗:∞(q)(f(q))−1∗∗⊤⊤(f(q))−1) _w^*tr (H_f,k^*:∞^(q)(H_f^(q))^-1SHw^*w^* HS (H_f^(q))^-1 ) (81) = = ∗tr((f,k∗:∞(q))−1∗∗⊤⊤) _w^*tr ((H_f,k^*:∞^(q))^-1SHw^*w^* HS ) = = tr((f,k∗:∞(q))−12⊤) ((H_f,k^*:∞^(q))^-1SH^2S ) ≥ ≥ ∑i>k∗(λ~i(q))−1μi(2⊤), _i>k^*( λ_i^(q))^-1 _i(SH^2S ), where the last inequality holds by Von Neumann’s trace inequality. ∎ D.8.1 Multiplicative Quantization Lemma D.19 (A bias lower bound under multiplicative quantization, power-law spectrum). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2 and 3.5, with probability at least 1−e−Ω(M)1-e^- (M), if [Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a≤M/C[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a≤ M/C for some constant C>0C>0, then ∗[‖(q)∗‖f,k∗:∞(q)2]≳[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ] [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a-1(1+ ε_f)(1+ ε_d)(1+ ε_s). Proof. By Lemma D.18, ∗[‖(q)∗‖f,k∗:∞(q)2]≥∑i>k∗(λ~i(q))−1μi(2⊤). _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ]≥ _i>k^*( λ_i^(q))^-1 _i(SH^2S ). (82) By Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), μi(2⊤)≂i−2a. _i(SH^2S ) i^-2a. (83) By Lemma G.2, with probability at least 1−e−Ω(M)1-e^- (M), λ~i(q)≲(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)i−a. λ_i^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)i^-a. (84) Therefore, if [Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a≤M/C[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a≤ M/C, then ∗[‖(q)∗‖f,k∗:∞(q)2] _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ] ≳ ∑i>k∗i−2a(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)i−a _i>k^* i^-2a(1+ ε_f)(1+ ε_d)(1+ ε_s)i^-a = = 1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)∑i>k∗i−a 1(1+ ε_f)(1+ ε_d)(1+ ε_s) _i>k^*i^-a ≂ 1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)(k∗)1−a 1(1+ ε_f)(1+ ε_d)(1+ ε_s)(k^*)^1-a ≳ 1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a−1. 1(1+ ε_f)(1+ ε_d)(1+ ε_s)[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a-1. ∎ D.8.2 Additive Quantization Lemma D.20 (A bias lower bound under additive quantization, power-law spectrum). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2 and 3.5, with probability at least 1−e−Ω(M)1-e^- (M), if M−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM≤CNγM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM≤ CNγ for some constant C>0C>0, then ∗[‖(q)∗‖f,k∗:∞(q)2]≳M−aM−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM(1Nγ−[ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM])1−1/a. _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ] M^-aM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM ( 1Nγ- [ ε_f+(1+ ε_dp) ε_s+ ε_d pM ] )^1-1/a. Proof. By Lemma D.18, ∗[‖(q)∗‖f,k∗:∞(q)2]≥∑i>k∗(λ~i(q))−1μi(2⊤). _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ]≥ _i>k^*( λ_i^(q))^-1 _i(SH^2S ). (85) By Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), μi(2⊤)≂i−2a. _i(SH^2S ) i^-2a. (86) By Lemma G.4, with probability at least 1−e−Ω(M)1-e^- (M), λ~i(q)≲i−a+ϵ¯f+(1+pϵ¯d)ϵ¯s+ϵ¯dpM. λ_i^(q) i^-a+ ε_f+(1+p ε_d) ε_s+ ε_d pM. (87) Therefore, if M−a+ϵ¯f+(1+pϵ¯d)ϵ¯s+ϵ¯dpM≤CNγM^-a+ ε_f+(1+p ε_d) ε_s+ ε_d pM≤ CNγ, then by denoting Δ=ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM = ε_f+(1+ ε_dp) ε_s+ ε_d pM, we have ∗[‖(q)∗‖f,k∗:∞(q)2]≳ _w^* [\|v^(q)^*\|_H_f,k^*:∞^(q)^2 ] ∑i>k∗i−2ai−a+Δ _i>k^* i^-2ai^-a+ ≥ ≥ mini−ai−a+Δ∑i>k∗i−a i^-ai^-a+ _i>k^*i^-a ≂ M−aM−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM⋅(k∗)1−a M^-aM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM·(k^*)^1-a ≥ ≥ M−aM−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM(1Nγ−Δ)1−1/a. M^-aM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM ( 1Nγ- )^1-1/a. ∎ D.9 Population Risk Lower Bounds under Power-law Spectrum D.9.1 Multiplicative Quantization Theorem D.3. Suppose γ<1/λ~1(q)γ<1/ λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, • Irreducible:=ℛ(∗)=12σ2. Irreducible:=R(w^*)= 12σ^2. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ∗Approx≳M1−aE_w^*Approx M^1-a. • for sufficiently large N>500N>500, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S 101010Here we take expectation on the prior ∗w^*., Excess≳BiasError+VarianceError+AdditiveError,EExcess +VarianceError+AdditiveError, where VarianceError≳σ¯M,12+σ¯M,22(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)minM,[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1aN, σ_M,1^2+ σ_M,2^2(1+ ε_f)(1+ ε_d)(1+ ε_s) \M, [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s) ] 1a \N, and if [Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a≤M/C[Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a≤ M/C for some constant C>0C>0, BiasError≳[Nγ(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]1/a−1[(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)]2, [Nγ(1+ ε_f)(1+ ε_d)(1+ ε_s)]^1/a-1 [(1+ ε_f)(1+ ε_d)(1+ ε_s) ]^2, and if f(q)H_f^(q) and ⊤SHS are commutative, AdditiveError≳ [(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)−1]2[(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)]2+(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)−1(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)1N(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s), [(1+ ε_d)(1+ ε_f)(1+ ε_s)-1 ]^2 [(1+ ε_d)(1+ ε_f)(1+ ε_s) ]^2+ (1+ ε_f)(1+ ε_d)(1+ ε_s)-1(1+ ε_f)(1+ ε_d)(1+ ε_s) 1N(1+ ε_d)(1+ ε_f)(1+ ε_s), with σ¯M,12=(1+ϵ¯o)σ¯2,σ¯M,22=β(1+ϵ¯o)(ϵ¯p+(1+ϵ¯p)ϵ¯a)(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s). σ_M,1^2=(1+ ε_o) σ^2, σ_M,2^2= β(1+ ε_o) ( ε_p+(1+ ε_p) ε_a )(1+ ε_f)(1+ ε_d)(1+ ε_s). Proof. The proof can be completed by (5), (6), Lemma B.2, (64), Theorem D.2, Lemma D.14, Lemma D.16, Lemma D.19 and noticing the following facts. Firstly, μmin((f(q))−1⊤)≥1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). _ min ((H_f^(q))^-1SHS )≥ 1(1+ ε_d)(1+ ε_f)(1+ ε_s). Secondly, recall that (q)∗=(f(q))−1∗v^(q)^*=(H_f^(q))^-1SHw^*, we separate two cases to lower bound ∗Peff≳∗[‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2∑i(λ~i(q))2(1−γλ~i(q))2N].E_w^*P_ eff _w^* [ \|v^(q)^* \|_H_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 _i( λ_i^(q))^2(1-γ λ_i^(q))^2N ]. • k∗=Mk^*=M In this case, ∗Peff≳∗‖(q)∗‖f(q)2=tr(⊤(f(q))−1)≥1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)tr(2⊤(⊤)−1).E_w^*P_ eff _w^* \|v^(q)^* \|_H_f^(q)^2=tr (HS (H_f^(q))^-1SH )≥ 1(1+ ε_d)(1+ ε_f)(1+ ε_s)tr (SH^2S (SHS )^-1 ). By the Von Neumann’s trace inequality and Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), tr(2⊤(⊤)−1)≥∑i−a≂1,tr (SH^2S (SHS )^-1 )≥ _ii^-a 1, then with probability at least 1−e−Ω(M)1-e^- (M), ∗Peff≳1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s).E_w^*P_ eff 1(1+ ε_d)(1+ ε_f)(1+ ε_s). • k∗<Mk^*<M In this case, by Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), ∗Peff≳ _w^*P_ eff ∗[‖(q)∗‖f,0:k∗(q)2+Nγ‖(q)∗‖f,k∗:∞(q)2∑i(λ~i(q))2(1−γλ~i(q))2N] _w^* [ \|v^(q)^* \|_H_f,0:k^*^(q)^2+Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 _i( λ_i^(q))^2(1-γ λ_i^(q))^2N ] ≳ ∗[‖(q)∗‖f,0:k∗(q)2+1Nγ‖(q)∗‖f,k∗:∞(q)2] _w^* [ \|v^(q)^* \|_H_f,0:k^*^(q)^2+ 1Nγ \|v^(q)^* \|_H_f,k^*:∞^(q)^2 ] = = tr([(f,0:k∗(q))−1+1Nγ(f,k∗:∞(q))−1]2⊤) ( [(H_f,0:k^*^(q))^-1+ 1Nγ(H_f,k^*:∞^(q))^-1 ]SH^2S ) ≥ ≥ μmin((f,0:k∗(q))−1+1Nγ(f,k∗:∞(q))−1)tr(2⊤) _ ((H_f,0:k^*^(q))^-1+ 1Nγ(H_f,k^*:∞^(q))^-1 )tr (SH^2S ) = = min1λ~1(q),1Nγ1λ~k∗+1(q)tr(2⊤) \ 1 λ_1^(q), 1Nγ 1 λ_k^*+1^(q) \tr (SH^2S ) ≥ ≥ min1λ~1(q),1tr(2⊤) \ 1 λ_1^(q),1 \tr (SH^2S ) ≥ ≥ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s)tr(2⊤) 1(1+ ε_d)(1+ ε_f)(1+ ε_s)tr (SH^2S ) ≂ 1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). 1(1+ ε_d)(1+ ε_f)(1+ ε_s). ∎ D.9.2 Additive Quantization Theorem D.4. Suppose γ<1/λ~1(q)γ<1/ λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, • Irreducible:=ℛ(∗)=12σ2. Irreducible:=R(w^*)= 12σ^2. • with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ∗Approx≳M1−a.E_w^*Approx M^1-a. • for sufficiently large N>500N>500, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S 111111Here we take expectation on the prior ∗w^*., Excess≳BiasError+VarianceError+AdditiveError,EExcess +VarianceError+AdditiveError, where VarianceError≳M−aM−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpMσ¯G2k¯eff+γ2N2(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2(M−k¯eff)N, M^-aM^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM σ_G^2 k_ eff+γ^2N^2 ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2(M- k_ eff)N, and if M−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM≤CNγM^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM≤ CNγ for some constant C>0C>0, then BiasError≳(M−aM−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)2⋅(1Nγ−[ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM])1−1/a, ( M^-aM^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM )^2· ( 1Nγ- [ ε_f+(1+ ε_dp) ε_s+ ε_d pM ] )^1-1/a, and if f(q)H_f^(q) and ⊤SHS are commutative, AdditiveError≳ (ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2(1+ϵ¯s+ϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM)2 ( ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 (1+ ε_s+ ε_f+ ε_s ε_dp+ ε_d pM )^2 + + 1NM−aM−a+ϵ¯f+ϵ¯s(ϵ¯dp+1)+ϵ¯dpMϵ¯s+ϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpMϵ¯sϵ¯dp+ϵ¯f+ϵ¯dpM+1+ϵ¯s, 1N M^-aM^-a+ ε_f+ ε_s( ε_dp+1)+ ε_d pM ε_s+ ε_s ε_dp+ ε_f+ ε_d pM ε_s ε_dp+ ε_f+ ε_d pM+1+ ε_s, with σ¯G2=σ¯2+ϵ¯a+ϵ¯o+βϵ¯p[1+pϵ¯d+M(ϵ¯f+ϵ¯s(ϵ¯dp+1))], σ_G^2= σ^2+ ε_a+ ε_o+β ε_p [1+p ε_d+M( ε_f+ ε_s( ε_dp+1)) ], k¯eff=[M−a∨(1Nγ−ϵ¯f−(1+ϵ¯dp)ϵ¯s−ϵ¯dpM)]−1a. k_ eff= [M^-a ( 1Nγ- ε_f-(1+ ε_dp) ε_s- ε_d pM ) ]^- 1a. Proof. The proof can be completed by (5), (6), Lemma B.2, (64), Theorem D.1, Lemma D.15, Lemma D.17, Lemma D.20 and noticing the following facts. Firstly, by Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), μmin((f(q))−1⊤)≳M−aϵ¯f+ϵ¯sϵ¯dp+ϵ¯dpM+M−a+ϵ¯s. _ min ((H_f^(q))^-1SHS ) M^-a ε_f+ ε_s ε_dp+ ε_d pM+M^-a+ ε_s. Secondly, by Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M), tr(f(q))≥tr(⊤+(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM))≂1+M(ϵ¯f+ϵ¯s(1+ϵ¯dp))+pϵ¯d. (H_f^(q) ) (SHS +( ε_f+ ε_s(1+ ε_dp)+ ε_d pM)I ) 1+M( ε_f+ ε_s(1+ ε_dp))+p ε_d. ∎ Appendix E Scaling Laws E.1 Multiplicative Quantization Denote ϵ¯2(M)=(1+ϵ¯o)(1+ϵ¯p+(1+ϵ¯p)ϵ¯a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s))−1, ε_2^(M)=(1+ ε_o) (1+ ε_p+(1+ ε_p) ε_a(1+ ε_d)(1+ ε_f)(1+ ε_s) )-1, ϵ¯2(M)=(1+ϵ¯o)(1+ϵ¯p+(1+ϵ¯p)ϵ¯a(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s))−1, ε_2^(M)=(1+ ε_o) (1+ ε_p+(1+ ε_p) ε_a(1+ ε_d)(1+ ε_f)(1+ ε_s) )-1, ϵ¯3(M)=1−1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s),ϵ¯3(M)=1−1(1+ϵ¯d)(1+ϵ¯f)(1+ϵ¯s). ε_3^(M)=1- 1(1+ ε_d)(1+ ε_f)(1+ ε_s), ε_3^(M)=1- 1(1+ ε_d)(1+ ε_f)(1+ ε_s). Theorem E.1. Suppose γ<1/((1+ϵ~)αtr(f(q)))γ<1/ ((1+ ε) (H_f^(q) ) ). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, if f(q)H_f^(q) and ⊤SHS commute, with probability at least 1−e−Ω(M)1-e^- (M), ℛM(¯N)≲1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(M))2+(1−ϵ¯3(M))ϵ¯3(M),ER_M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(M) )^2+ (1- ε_3^(M) ) ε_3^(M), (88) where Meff=M,Neff=N[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))1/a]−a−1. M_ eff=M, N_ eff=N [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^1/a ]^- aa-1. Proof. By Theorem C.3, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, ℛM(¯N)≲σ2+M1−a+BiasError+VarianceError+AdditiveError,ER_M( v_N) σ^2+M^1-a+BiasError+VarianceError+AdditiveError, where BiasError≲(1−ϵ¯3(M))2max(Nγ1−ϵ¯3(M))1a−1,M1−a, (1- ε_3^(M))^2 \ ( Nγ1- ε_3^(M) ) 1a-1,M^1-a \, VarianceError≲(1−ϵ¯3(M))(1+ϵ¯2(M))minM,(Nγ1−ϵ¯3(M))1/aN, (1- ε_3^(M)) (1+ ε_2^(M) ) \M, ( Nγ1- ε_3^(M) )^1/a \N, AdditiveError≲(ϵ¯3(M))2+(1−ϵ¯3(M))ϵ¯3(M). ( ε_3^(M) )^2+ (1- ε_3^(M) ) ε_3^(M). Denote ℛ0=ℛM(¯N)−[σ2+(ϵ¯3(M))2+(1−ϵ¯3(M))ϵ¯3(M)].R_0=ER_M( v_N)- [σ^2+ ( ε_3^(M) )^2+ (1- ε_3^(M) ) ε_3^(M) ]. • Ma≥Nγ1−ϵ¯3(M)M^a≥ Nγ1- ε_3^(M) In this case, ℛ0≲M1−a+(1−ϵ¯3(M))1+ϵ¯2(M)(1−ϵ¯3(M))1/aN1/a−1. _0 M^1-a+ (1- ε_3^(M) ) 1+ ε_2^(M) (1- ε_3^(M) )^1/aN^1/a-1. • Nγ1−ϵ¯3(M)≤Ma<Nγ1−ϵ¯3(M) Nγ1- ε_3^(M)≤ M^a< Nγ1- ε_3^(M) In this case, ℛ0≲ _0 M1−a+(1−ϵ¯3(M))(1+ϵ¯2(M))MN+(1−ϵ¯3(M))3−1/aN1/a−1 M^1-a+ (1- ε_3^(M) ) (1+ ε_2^(M) ) MN+ (1- ε_3^(M) )^3-1/aN^1/a-1 ≲ M1−a+(1−ϵ¯3(M))1+ϵ¯2(M)(1−ϵ¯3(M))1/aN1/a−1+(1−ϵ¯3(M))3−1/aN1/a−1 M^1-a+ (1- ε_3^(M) ) 1+ ε_2^(M) (1- ε_3^(M) )^1/aN^1/a-1+ (1- ε_3^(M) )^3-1/aN^1/a-1 ≲ M1−a+(1−ϵ¯3(M))1+ϵ¯2(M)(1−ϵ¯3(M))1/aN1/a−1. M^1-a+ (1- ε_3^(M) ) 1+ ε_2^(M) (1- ε_3^(M) )^1/aN^1/a-1. • Ma<Nγ1−ϵ¯3(M)M^a< Nγ1- ε_3^(M) In this case, ℛ0≲ _0 M1−a+(1−ϵ¯3(M))(1+ϵ¯2(M))MN≲M1−a+N1/a−1(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a. M^1-a+ (1- ε_3^(M) ) (1+ ε_2^(M) ) MN M^1-a+N^1/a-1 (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a. Summarizing, ℛ0≲M1−a+N1/a−1[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))1/a].R_0 M^1-a+N^1/a-1 [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^1/a ]. ∎ Theorem E.2. Suppose γ<1/λ~1(q)γ<1/ λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-multiplicative, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for sufficiently large N>500N>500, if f(q)H_f^(q) and ⊤SHS are commutative, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds 121212This scaling law holds in the positive regime Ω=(M,N):(M≤(Nγ)1a[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a]11−a)∨(M≥(Nγ1−ϵ¯3(M))1a). = \(M,N): (M≤ (Nγ ) 1a [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a ] 11-a ) (M≥ ( Nγ1- ε_3^(M) ) 1a ) \. ℛM(¯N)≳1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(M))2+ϵ¯3(M)N(1−ϵ¯3(M)),ER_M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(M) )^2+ ε_3^(M)N (1- ε_3^(M) ), (89) where Meff=M,Neff=N[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a]−a−1.M_ eff=M, N_ eff=N [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a ]^- aa-1. Proof. By Theorem D.3, with probability at least 1−e−Ω(M)1-e^- (M), ℛM(¯N)≳σ2+M1−a+BiasError+VarianceError+AdditiveError,ER_M( v_N) σ^2+M^1-a+BiasError+VarianceError+AdditiveError, where VarianceError≳(1−ϵ¯3(M))(1+ϵ¯2(M))minM,(Nγ1−ϵ¯3(M))1aN, (1- ε_3^(M) ) (1+ ε_2^(M) ) \M, ( Nγ1- ε_3^(M) ) 1a \N, and if (Nγ1−ϵ¯3(M))1/a≤M/C ( Nγ1- ε_3^(M) )^1/a≤ M/C for some constant C>0C>0, BiasError≳(1−ϵ¯3(M))2(Nγ1−ϵ¯3(M))1/a−1, (1- ε_3^(M) )^2 ( Nγ1- ε_3^(M) )^1/a-1, and if f(q)H_f^(q) and ⊤SHS are commutative, AdditiveError≳ (ϵ¯3(M))2+ϵ¯3(M)N(1−ϵ¯3(M)). ( ε_3^(M) )^2+ ε_3^(M)N (1- ε_3^(M) ). Denote ℛ0=ℛM(¯N)−σ2−[(ϵ¯3(M))2+ϵ¯3(M)N(1−ϵ¯3(M))].R_0=ER_M( v_N)-σ^2- [ ( ε_3^(M) )^2+ ε_3^(M)N (1- ε_3^(M) ) ]. It holds ℛ0≳M1−a+(1−ϵ¯3(M))(1+ϵ¯2(M))minM,(Nγ1−ϵ¯3(M))1aN. _0 M^1-a+ (1- ε_3^(M) ) (1+ ε_2^(M) ) \M, ( Nγ1- ε_3^(M) ) 1a \N. Let 1Ma−1≥(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a(Nγ)1a−1, 1M^a-1≥ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a(Nγ) 1a-1, solving that M≤(Nγ)1a[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a]11−a.M≤ (Nγ ) 1a [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a ] 11-a. • M≥(Nγ1−ϵ¯3(M))1aM≥ ( Nγ1- ε_3^(M) ) 1a In this case, ℛ0≳M1−a+(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/aN1/a−1.R_0 M^1-a+ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/aN^1/a-1. • M≤(Nγ)1a[(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/a]11−aM≤ (Nγ ) 1a [ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/a ] 11-a In this case, ℛ0≳M1−a≂M1−a+(1−ϵ¯3(M))(1+ϵ¯2(M))(1−ϵ¯3(M))−1/aN1/a−1.R_0 M^1-a M^1-a+ (1- ε_3^(M) ) (1+ ε_2^(M) ) (1- ε_3^(M) )^-1/aN^1/a-1. ∎ E.2 Additive Quantization Denote ϵ¯2(A)=ϵ¯a+ϵ¯o+ϵ¯p[1+pϵ¯d+M(ϵ¯f+ϵ¯s+ϵ¯sϵ¯dp)], ε_2^(A)= ε_a+ ε_o+ ε_p [1+p ε_d+M( ε_f+ ε_s+ ε_s ε_dp) ], ϵ¯2(A)=ϵ¯a+ϵ¯o+ϵ¯p[1+pϵ¯d+M(ϵ¯f+ϵ¯sϵ¯dp+ϵ¯s)], ε_2^(A)= ε_a+ ε_o+ ε_p [1+p ε_d+M( ε_f+ ε_s ε_dp+ ε_s) ], ϵ¯3(A)=Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM)1+Ma(ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM), ε_3^(A)= M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM )1+M^a ( ε_f+ ε_s(1+ ε_dp)+ ε_d pM ), ϵ¯3(A)=ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM1+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM. ε_3^(A)= ε_f+(1+ ε_dp) ε_s+ ε_d pM1+ ε_f+(1+ ε_dp) ε_s+ ε_d pM. Theorem E.3. Suppose γ<1/(αtr(f(q)))γ<1/ ( (H_f^(q) ) ). For any i∈s,d,f,p,a,oi∈\s,d,f,p,a,o\, if there exist (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that quantization iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, if f(q)H_f^(q) and ⊤SHS commute, with probability at least 1−e−Ω(M)1-e^- (M), ℛM(¯N)≲1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(A))2+(1−ϵ¯3(A))ϵ¯3(A),ER_M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(A) )^2+ (1- ε_3^(A) ) ε_3^(A), (90) where Meff=M,Neff=N[(1−ϵ¯3(A))1+ϵ¯2(A)(1−ϵ¯3(A))1/a(1+(ϵ¯3(A))2)]−a−1.M_ eff=M, N_ eff=N [ (1- ε_3^(A) ) 1+ ε_2^(A) (1- ε_3^(A) )^1/a (1+ ( ε_3^(A) )^2 ) ]^- aa-1. The scaling law can be also expressed as: Meff=M[1+(1−ϵ¯3(A))(1+ϵ¯2(A))(ϵ¯3(A))21−ϵ¯3(A)]−1a−1,Neff=N[(1−ϵ¯3(A))1+ϵ¯2(A)(1−ϵ¯3(A))1/a]−a−1.M_ eff=M [1+ (1- ε_3^(A) ) (1+ ε_2^(A) ) ( ε_3^(A) )^21- ε_3^(A) ]^- 1a-1, N_ eff=N [ (1- ε_3^(A) ) 1+ ε_2^(A) (1- ε_3^(A) )^1/a ]^- aa-1. (91) Proof. By Theorem C.4, with probability at least 1−e−Ω(M)1-e^- (M), ℛM(¯N)≲σ2+M1−a+BiasError+VarianceError+AdditiveError,ER_M( v_N) σ^2+M^1-a+BiasError+VarianceError+AdditiveError, where BiasError≲(1−ϵ¯3(A))2max[Nγ1−ϵ¯3(A)]1a−1,M1−a, (1- ε_3^(A) )^2 \ [ Nγ1- ε_3^(A) ] 1a-1,M^1-a \, VarianceError≲(1−ϵ¯3(A))(1+ϵ¯2(A))keff+γ2N2M−2a(11−ϵ¯3(A)−1)2(M−keff)N, (1- ε_3^(A) ) (1+ ε_2^(A) ) k_ eff+γ^2N^2M^-2a ( 11- ε_3^(A)-1 )^2(M-k_ eff)N, AdditiveError≲(ϵ¯3(A))2+(1−ϵ¯3(A))ϵ¯3(A), ( ε_3^(A) )^2+ (1- ε_3^(A) ) ε_3^(A), with keff=[M−a∨(1Nγ−M−a(11−ϵ¯3(A)−1))]−1ak_ eff= [M^-a ( 1Nγ-M^-a ( 11- ε_3^(A)-1 ) ) ]^- 1a. Denote ℛ0=ℛM(¯N)−[σ2+(ϵ¯3(A))2+(1−ϵ¯3(A))ϵ¯3(A)].R_0=ER_M( v_N)- [σ^2+ ( ε_3^(A) )^2+ (1- ε_3^(A) ) ε_3^(A) ]. • 1Nγ>M−a1−ϵ¯3(A) 1Nγ> M^-a1- ε_3^(A) In this case, 1Nγ−M−a(11−ϵ¯3(A)−1)>M−a 1Nγ-M^-a ( 11- ε_3^(A)-1 )>M^-a and 1Nγ>M−a1−ϵ¯3(A)>M−a1−ϵ¯3(A) 1Nγ> M^-a1- ε_3^(A)> M^-a1- ε_3^(A). It follows that Nγ1−ϵ¯3(A)<Ma Nγ1- ε_3^(A)<M^a. Hence, ℛ0≲ _0 M1−a+(1−ϵ¯3(A))2[Nγ1−ϵ¯3(A)]1/a−1 M^1-a+ (1- ε_3^(A) )^2 [ Nγ1- ε_3^(A) ]^1/a-1 + + (1−ϵ¯3(A))(1+ϵ¯2(A))keff(1−γ2N2M−2a(11−ϵ¯3(A)−1)2)+γ2N2M−2a(11−ϵ¯3(A)−1)2MN, (1- ε_3^(A) ) (1+ ε_2^(A) ) k_ eff (1-γ^2N^2M^-2a ( 11- ε_3^(A)-1 )^2 )+γ^2N^2M^-2a ( 11- ε_3^(A)-1 )^2MN, with keff=[1Nγ−M−a(11−ϵ¯3(A)−1)]−1/ak_ eff= [ 1Nγ-M^-a ( 11- ε_3^(A)-1 ) ]^-1/a. Noticing that M1−2a≤(Nγ1−ϵ¯3(A))1/a−2,M^1-2a≤ ( Nγ1- ε_3^(A) )^1/a-2, and keff= k_ eff= [1Nγ−M−a(11−ϵ¯3(A)−1)]−1/a [ 1Nγ-M^-a ( 11- ε_3^(A)-1 ) ]^-1/a ≂ N1/a[1−NγM−a(11−ϵ¯3(A)−1)]−1/a N^1/a [1-Nγ M^-a ( 11- ε_3^(A)-1 ) ]^-1/a ≤ ≤ N1/a[1−(1−ϵ¯3(A))(11−ϵ¯3(A)−1)]−1/a N^1/a [1- (1- ε_3^(A) ) ( 11- ε_3^(A)-1 ) ]^-1/a = = N1/a(1−ϵ¯3(A))−1/a, N^1/a (1- ε_3^(A) )^-1/a, we have ℛ0≲ _0 M1−a+(1−ϵ¯3(A))3−1/a[Nγ]1/a−1 M^1-a+ (1- ε_3^(A) )^3-1/a [Nγ ]^1/a-1 + + (1−ϵ¯3(A))(1+ϵ¯2(A))N1/a(1−ϵ¯3(A))−1/a+γ2N2(11−ϵ¯3(A)−1)2(Nγ1−ϵ¯3(A))1/a−2N (1- ε_3^(A) ) (1+ ε_2^(A) ) N^1/a (1- ε_3^(A) )^-1/a+γ^2N^2 ( 11- ε_3^(A)-1 )^2 ( Nγ1- ε_3^(A) )^1/a-2N ≲ M1−a+N1/a−1[(1−ϵ¯3(A))(1+ϵ¯2(A))(1−ϵ¯3(A))−1/a(1+(ϵ¯3(A))2)]. M^1-a+N^1/a-1 [ (1- ε_3^(A) ) (1+ ε_2^(A) ) (1- ε_3^(A) )^-1/a (1+ ( ε_3^(A) )^2 ) ]. We would like to remark that, the term NM1−2aNM^1-2a can also expressed by M, leading a decrease in effective M. Specifically, noticing that γ2NM1−2a(11−ϵ¯3(A)−1)2≲M1−a(1−ϵ¯3(A))(11−ϵ¯3(A)−1)2,γ^2NM^1-2a ( 11- ε_3^(A)-1 )^2 M^1-a (1- ε_3^(A) ) ( 11- ε_3^(A)-1 )^2, we have ℛ0≲M1−a[1+(1−ϵ¯3(A))(1+ϵ¯2(A))(ϵ¯3(A))21−ϵ¯3(A)]+N1/a−1[(1−ϵ¯3(A))(1+ϵ¯2(A))(1−ϵ¯3(A))−1/a].R_0 M^1-a [1+ (1- ε_3^(A) ) (1+ ε_2^(A) ) ( ε_3^(A) )^21- ε_3^(A) ]+N^1/a-1 [ (1- ε_3^(A) ) (1+ ε_2^(A) ) (1- ε_3^(A) )^-1/a ]. • M−a1−ϵ¯3(A)<1Nγ≤M−a1−ϵ¯3(A) M^-a1- ε_3^(A)< 1Nγ≤ M^-a1- ε_3^(A) In this case, ℛ0≲ _0 M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))MN+(1−ϵ¯3(A))3−1/a[Nγ]1/a−1 M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) MN+ (1- ε_3^(A) )^3-1/a [Nγ ]^1/a-1 ≲ M1−a+N1/a−1[(1−ϵ¯3(A))(1+ϵ¯2(A))(1−ϵ¯3(A))−1/a]. M^1-a+N^1/a-1 [ (1- ε_3^(A) ) (1+ ε_2^(A) ) (1- ε_3^(A) )^-1/a ]. • 1Nγ≤M−a1−ϵ¯3(A) 1Nγ≤ M^-a1- ε_3^(A) In this case, ℛ0≲M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))MN≲M1−a+(1−ϵ¯3(A))1−1/a(1+ϵ¯2(A))N1/a−1.R_0 M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) MN M^1-a+ (1- ε_3^(A) )^1-1/a (1+ ε_2^(A) )N^1/a-1. Overall, ℛ0≲M1−a+N1/a−1(1−ϵ¯3(A))(1+ϵ¯2(A))(1−ϵ¯3(A))−1/a[1+(ϵ¯3(A))2].R_0 M^1-a+N^1/a-1 (1- ε_3^(A) ) (1+ ε_2^(A) ) (1- ε_3^(A) )^-1/a [1+ ( ε_3^(A) )^2 ]. Regarding analyzing MeffM_ eff, it holds ℛ0≲M1−a[1+(1−ϵ¯3(A))(1+ϵ¯2(A))(ϵ¯3(A))21−ϵ¯3(A)]+N1/a−1[(1−ϵ¯3(A))(1+ϵ¯2(A))(1−ϵ¯3(A))−1/a].R_0 M^1-a [1+ (1- ε_3^(A) ) (1+ ε_2^(A) ) ( ε_3^(A) )^21- ε_3^(A) ]+N^1/a-1 [ (1- ε_3^(A) ) (1+ ε_2^(A) ) (1- ε_3^(A) )^-1/a ]. ∎ Theorem E.4. Suppose γ<1/λ~1(q)γ<1/ λ_1^(q). For i∈d,f,s,p,a,oi∈\d,f,s,p,a,o\, if there exist constants (ϵ¯i,ϵ¯i)( ε_i, ε_i) such that iQ_i is (ϵ¯i,ϵ¯i)( ε_i, ε_i)-additive, then under Assumption 3.1, 3.2, 3.3, 3.4 and 3.5, for sufficiently large N>500N>500, if f(q)H_f^(q) and ⊤SHS are commutative, with probability at least 1−e−Ω(M)1-e^- (M) over the randomness of S, it holds 131313This scaling law holds in the positive regime Ω=(M,N):(1Nγ−(11−ϵ¯3(A)−1)≥M−a)∨(M1−a≥(1−ϵ¯3(A))(1+ϵ¯2(A))[1Nγ−(11−ϵ¯3(A)−1)]−1aNγ) = \(M,N): ( 1Nγ- ( 11- ε_3^(A)-1 )≥ M^-a ) (M^1-a≥ (1- ε_3^(A) ) (1+ ε_2^(A) ) [ 1Nγ- ( 11- ε_3^(A)-1 ) ]^- 1aNγ ) \ under the condition that 1Nγ−(11−ϵ¯3(A)−1)≥0 1Nγ- ( 11- ε_3^(A)-1 )≥ 0. ℛM(¯N)≳1Meffa−1+1Neff(a−1)/a+σ2+(ϵ¯3(A))2+1−ϵ¯3(A)Nϵ¯3(A),ER_M( v_N) 1M_ eff^a-1+ 1N_ eff^(a-1)/a+σ^2+ ( ε_3^(A) )^2+ 1- ε_3^(A)N ε_3^(A), (92) where Meff=M,Neff=N[(1−ϵ¯3(A))(1+ϵ¯2(A))[1−Nγ(11−ϵ¯3(A)−1)]−1/a]−a−1.M_ eff=M, N_ eff=N [ (1- ε_3^(A) ) (1+ ε_2^(A) ) [1-Nγ ( 11- ε_3^(A)-1 ) ]^-1/a ]^- aa-1. Proof. By Lemma D.4, ℛM(¯N)≳σ2+M1−a+BiasError+VarianceError+AdditiveError,ER_M( v_N) σ^2+M^1-a+BiasError+VarianceError+AdditiveError, where VarianceError≳(1−ϵ¯3(A))(1+ϵ¯2(A))k¯eff+γ2N2(11−ϵ¯3(A)−1)2(M−k¯eff)N, (1- ε_3^(A) ) (1+ ε_2^(A) ) k_ eff+γ^2N^2 ( 11- ε_3^(A)-1 )^2(M- k_ eff)N, and if M−a+M−a(11−ϵ¯3(A)−1)≤CNγM^-a+M^-a ( 11- ε_3^(A)-1 )≤ CNγ for some constant C>0C>0, then BiasError≳(1−ϵ¯3(A))2⋅(1Nγ−M−a(11−ϵ¯3(A)−1))1−1/a, (1- ε_3^(A) )^2· ( 1Nγ-M^-a ( 11- ε_3^(A)-1 ) )^1-1/a, AdditiveError≳ (ϵ¯3(A))2+1−ϵ¯3(A)Nϵ¯3(A), ( ε_3^(A) )^2+ 1- ε_3^(A)N ε_3^(A), with k¯eff=[M−a∨(1Nγ−(11−ϵ¯3(A)−1))]−1a k_ eff= [M^-a ( 1Nγ- ( 11- ε_3^(A)-1 ) ) ]^- 1a Denote ℛ0=ℛM(¯N)−[σ2+(ϵ¯3(A))2+1−ϵ¯3(A)Nϵ¯3(A)].R_0=ER_M( v_N)- [σ^2+ ( ε_3^(A) )^2+ 1- ε_3^(A)N ε_3^(A) ]. • 1Nγ−(11−ϵ¯3(A)−1)≥M−a 1Nγ- ( 11- ε_3^(A)-1 )≥ M^-a In this case, ℛ0≳ _0 M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))k¯eff+γ2N2(11−ϵ¯3(A)−1)2(M−k¯eff)N M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) k_ eff+γ^2N^2 ( 11- ε_3^(A)-1 )^2(M- k_ eff)N ≥ ≥ M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))[1Nγ−(11−ϵ¯3(A)−1)]−1aN. M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) [ 1Nγ- ( 11- ε_3^(A)-1 ) ]^- 1aN. • 1Nγ−(11−ϵ¯3(A)−1)<M−a 1Nγ- ( 11- ε_3^(A)-1 )<M^-a In this case, ℛ0≳M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))MN.R_0 M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) MN. Suppose 1Nγ−(11−ϵ¯3(A)−1)≥0 1Nγ- ( 11- ε_3^(A)-1 )≥ 0. Further if M1−a≥(1−ϵ¯3(A))(1+ϵ¯2(A))[1Nγ−(11−ϵ¯3(A)−1)]−1aNγ,M^1-a≥ (1- ε_3^(A) ) (1+ ε_2^(A) ) [ 1Nγ- ( 11- ε_3^(A)-1 ) ]^- 1aNγ, then ℛ0≳ _0 M1−a+(1−ϵ¯3(A))(1+ϵ¯2(A))[1Nγ−(11−ϵ¯3(A)−1)]−1aN M^1-a+ (1- ε_3^(A) ) (1+ ε_2^(A) ) [ 1Nγ- ( 11- ε_3^(A)-1 ) ]^- 1aN ≂ M1−a+N1/a−1(1−ϵ¯3(A))(1+ϵ¯2(A))[1−Nγ(11−ϵ¯3(A)−1)]−1/a. M^1-a+N^1/a-1 (1- ε_3^(A) ) (1+ ε_2^(A) ) [1-Nγ ( 11- ε_3^(A)-1 ) ]^-1/a. ∎ Appendix F Auxiliary Lemmas Lemma F.1. For PSD matrices ⪯X and any PSD matrix A, μmin((+)−1)≥μmin((+)−1),μmax((+)−1)≥μmax((+)−1). _ min ((A+X)^-1A )≥ _ min ((A+Y)^-1A ), _ max ((A+X)^-1A )≥ _ max ((A+Y)^-1A ). Proof. Define the generalized Rayleigh quotient for a matrix M as R()=⊤⊤(+)R_M(v)= v Avv (M+A)v. Note that the eigenvalues of (+)−1(M+A)^-1A are given by the critical values of R()R_M(v). For any non-zero vector v, since ⪯X and ⪰0A 0, we have R()=11+⊤⊤≥11+⊤⊤=R().R_X(v)= 11+ v Xvv Av≥ 11+ v Yvv Av=R_Y(v). By the Courant-Fischer Min-Max theorem, we have μmin((+)−1)=min≠0R()≥min≠0R()=μmin((+)−1). _ min ((A+X)^-1A )= _v≠ 0R_X(v)≥ _v≠ 0R_Y(v)= _ min ((A+Y)^-1A ). Similarly μmax((+)−1)=max≠0R()≥max≠0R()=μmax((+)−1). _ max ((A+X)^-1A )= _v≠ 0R_X(v)≥ _v≠ 0R_Y(v)= _ max ((A+Y)^-1A ). ∎ Lemma F.2. For PSD matrices ⪯A and any PSD matrix X, μmin((+)−1)≤μmin((+)−1),μmax((+)−1)≤μmax((+)−1). _ min ((A+X)^-1A )≤ _ min ((B+X)^-1B ), _ max ((A+X)^-1A )≤ _ max ((B+X)^-1B ). Proof. Define the generalized Rayleigh quotient for a matrix M as R()=⊤⊤(+)R_M(v)= v Mvv (M+X)v. Note that the eigenvalues of (+)−1(M+X)^-1M are given by the critical values of R()R_M(v). For any non-zero vector v, since ⪯A and ⪰0X 0, we have R()=11+⊤⊤≤11+⊤⊤=R().R_A(v)= 11+ v Xvv Av≤ 11+ v Xvv Bv=R_B(v). By the Courant-Fischer Min-Max theorem, we have μmin((+)−1)=min≠0R()≤min≠0R()=μmin((+)−1). _ min ((A+X)^-1A )= _v≠ 0R_A(v)≤ _v≠ 0R_B(v)= _ min ((B+X)^-1B ). Similarly, μmax((+)−1)=max≠0R()≤max≠0R()=μmax((+)−1). _ max ((A+X)^-1A )= _v≠ 0R_A(v)≤ _v≠ 0R_B(v)= _ max ((B+X)^-1B ). ∎ Lemma F.3. For ≻0A 0, ⪰0D 0, if there exists ϵ>0ε>0 such that ⪯ϵD , then −1(+)−1(+)−1−1⪯ϵ2(ϵ+1)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(ε+1)^2A^-1. Proof. Denote =−1/2−1/2Q=A^-1/2DA^-1/2. As ⪰0D 0, we have ⪯ϵ.Q . Rewriting −1(+)−1(+)−1−1 ^-1D(A+D)^-1A(A+D)^-1DA^-1 = = −1[1/21/2]⋅[−1/2(+)−1−1/2]⋅[−1/2(+)−1−1/2]⋅[1/21/2]−1 ^-1 [A^1/2QA^1/2 ]· [A^-1/2(I+Q)^-1A^-1/2 ]·A· [A^-1/2(I+Q)^-1A^-1/2 ]· [A^1/2QA^1/2 ]A^-1 = = (−1/21/2)(−1/2(+)−1−1/2)(−1/2(+)−1−1/2)(1/2−1/2) (A^-1/2QA^1/2 ) (A^-1/2(I+Q)^-1A^-1/2 )A (A^-1/2(I+Q)^-1A^-1/2 ) (A^1/2QA^-1/2 ) = = −1/21/2⋅[−1/2(+)−2−1/2]⋅1/2−1/2 ^-1/2QA^1/2· [A^-1/2(I+Q)^-2A^-1/2 ]·A^1/2QA^-1/2 = = −1/2[(+)−2]−1/2 ^-1/2 [Q(I+Q)^-2Q ]A^-1/2 = = −1/2[2(+)−2]−1/2, ^-1/2 [Q^2(I+Q)^-2 ]A^-1/2, where the last equality holds as Q and +Q+I are commutative. Further note that f(x)=x2(1+x)2f(x)= x^2(1+x)^2 is increasing, it holds 2(+)−2=f()⪯f(ϵ)=ϵ2(1+ϵ)2.Q^2(I+Q)^-2=f(Q) f(ε)I= ε^2(1+ε)^2I. Hence, −1(+)−1(+)−1−1⪯ϵ2(1+ϵ)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(1+ε)^2A^-1. ∎ Lemma F.4. For ≻0A 0, ⪰0D 0, if there exists ϵ>0ε>0 such that ⪯ϵD , then −1(+)−1(+)−1−1⪯ϵ2(ϵ+μmin())2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(ε+ _ min(A))^2A^-1. Proof. Denote =−1/2−1/2Q=A^-1/2DA^-1/2. Rewriting −1(+)−1(+)−1−1 ^-1D(A+D)^-1A(A+D)^-1DA^-1 = = −1[1/21/2]⋅[−1/2(+)−1−1/2]⋅[−1/2(+)−1−1/2]⋅[1/21/2]−1 ^-1 [A^1/2QA^1/2 ]· [A^-1/2(I+Q)^-1A^-1/2 ]·A· [A^-1/2(I+Q)^-1A^-1/2 ]· [A^1/2QA^1/2 ]A^-1 = = (−1/21/2)(−1/2(+)−1−1/2)(−1/2(+)−1−1/2)(1/2−1/2) (A^-1/2QA^1/2 ) (A^-1/2(I+Q)^-1A^-1/2 )A (A^-1/2(I+Q)^-1A^-1/2 ) (A^1/2QA^-1/2 ) = = −1/21/2⋅[−1/2(+)−2−1/2]⋅1/2−1/2 ^-1/2QA^1/2· [A^-1/2(I+Q)^-2A^-1/2 ]·A^1/2QA^-1/2 = = −1/2[(+)−2]−1/2 ^-1/2 [Q(I+Q)^-2Q ]A^-1/2 = = −1/2[2(+)−2]−1/2, ^-1/2 [Q^2(I+Q)^-2 ]A^-1/2, where the last equality holds as Q and +Q+I are commutative. Given ⪯ϵD , we substitute D: =−1/2−1/2⪯ϵ−1/2−1/2=ϵ−1.Q=A^-1/2DA^-1/2 ^-1/2IA^-1/2= ^-1. Let μmin() _ (A) denote the minimum eigenvalue of A. Since A is positive definite, the maximum eigenvalue of −1A^-1 is 1/μmin()1/ _ (A). Thus: −1⪯1μmin().A^-1 1 _ (A)I. Combining these, we obtain a scalar upper bound for Q: ⪯ϵμmin().Q ε _ (A)I. Further note that f(x)=x2(1+x)2f(x)= x^2(1+x)^2 is increasing, it holds 2(+)−2=f()⪯f(ϵμmin())=ϵ2(μmin()+ϵ)2.Q^2(I+Q)^-2=f(Q) f( ε _ (A))I= ε^2( _ (A)+ε)^2I. Hence, −1(+)−1(+)−1−1⪯ϵ2(μmin()+ϵ)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2( _ (A)+ε)^2A^-1. ∎ Lemma F.5. For ≻0A 0, ⪰0D 0, if there exists ϵ>0ε>0 such that ⪰ϵD , then −1(+)−1(+)−1−1⪰ϵ2(ϵ+1)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(ε+1)^2A^-1. Proof. Denote =−1/2−1/2Q=A^-1/2DA^-1/2, we have ⪰ϵ.Q . Rewriting −1(+)−1(+)−1−1 ^-1D(A+D)^-1A(A+D)^-1DA^-1 = = −1[1/21/2]⋅[−1/2(+)−1−1/2]⋅[−1/2(+)−1−1/2]⋅[1/21/2]−1 ^-1 [A^1/2QA^1/2 ]· [A^-1/2(I+Q)^-1A^-1/2 ]·A· [A^-1/2(I+Q)^-1A^-1/2 ]· [A^1/2QA^1/2 ]A^-1 = = (−1/21/2)(−1/2(+)−1−1/2)(−1/2(+)−1−1/2)(1/2−1/2) (A^-1/2QA^1/2 ) (A^-1/2(I+Q)^-1A^-1/2 )A (A^-1/2(I+Q)^-1A^-1/2 ) (A^1/2QA^-1/2 ) = = −1/21/2⋅[−1/2(+)−2−1/2]⋅1/2−1/2 ^-1/2QA^1/2· [A^-1/2(I+Q)^-2A^-1/2 ]·A^1/2QA^-1/2 = = −1/2[(+)−2]−1/2 ^-1/2 [Q(I+Q)^-2Q ]A^-1/2 = = −1/2[2(+)−2]−1/2, ^-1/2 [Q^2(I+Q)^-2 ]A^-1/2, where the last equality holds as Q and +Q+I are commutative. Further note that f(x)=x2(1+x)2f(x)= x^2(1+x)^2 is increasing, it holds 2(+)−2=f()⪰f(ϵ)=ϵ2(1+ϵ)2.Q^2(I+Q)^-2=f(Q) f(ε)I= ε^2(1+ε)^2I. Hence, −1(+)−1(+)−1−1⪰ϵ2(1+ϵ)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(1+ε)^2A^-1. ∎ Lemma F.6. For ≻0A 0, ⪰0D 0, if there exists ϵ>0ε>0 such that ⪰ϵD , then −1(+)−1(+)−1−1⪰ϵ2(ϵ+μmax())2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2(ε+ _ max(A))^2A^-1. Proof. Denote =−1/2−1/2Q=A^-1/2DA^-1/2. Rewriting −1(+)−1(+)−1−1 ^-1D(A+D)^-1A(A+D)^-1DA^-1 = = −1[1/21/2]⋅[−1/2(+)−1−1/2]⋅[−1/2(+)−1−1/2]⋅[1/21/2]−1 ^-1 [A^1/2QA^1/2 ]· [A^-1/2(I+Q)^-1A^-1/2 ]·A· [A^-1/2(I+Q)^-1A^-1/2 ]· [A^1/2QA^1/2 ]A^-1 = = (−1/21/2)(−1/2(+)−1−1/2)(−1/2(+)−1−1/2)(1/2−1/2) (A^-1/2QA^1/2 ) (A^-1/2(I+Q)^-1A^-1/2 )A (A^-1/2(I+Q)^-1A^-1/2 ) (A^1/2QA^-1/2 ) = = −1/21/2⋅[−1/2(+)−2−1/2]⋅1/2−1/2 ^-1/2QA^1/2· [A^-1/2(I+Q)^-2A^-1/2 ]·A^1/2QA^-1/2 = = −1/2[(+)−2]−1/2 ^-1/2 [Q(I+Q)^-2Q ]A^-1/2 = = −1/2[2(+)−2]−1/2, ^-1/2 [Q^2(I+Q)^-2 ]A^-1/2, where the last equality holds as Q and +Q+I are commutative. Given ⪯ϵD , we substitute D: =−1/2−1/2⪰ϵ−1/2−1/2=ϵ−1.Q=A^-1/2DA^-1/2 ^-1/2IA^-1/2= ^-1. Let μmax() _ (A) denote the maximum eigenvalue of A. Since A is positive definite, the minimum eigenvalue of −1A^-1 is 1/μmax()1/ _ (A). Thus: −1⪰1μmax().A^-1 1 _ (A)I. Combining these, we obtain a scalar upper bound for Q: ⪰ϵμmax().Q ε _ (A)I. Further note that f(x)=x2(1+x)2f(x)= x^2(1+x)^2 is increasing, it holds 2(+)−2=f()⪰f(ϵμmax())=ϵ2(μmax()+ϵ)2.Q^2(I+Q)^-2=f(Q) f( ε _ (A))I= ε^2( _ (A)+ε)^2I. Hence, −1(+)−1(+)−1−1⪰ϵ2(μmax()+ϵ)2−1.A^-1D(A+D)^-1A(A+D)^-1DA^-1 ε^2( _ (A)+ε)^2A^-1. ∎ Appendix G Concentration Lemmas Lemma G.1 (Eigenvalues of ⊤SHS , Lemma G.4 in Lin et al. (2024)). Under the power-law spectrum Assumption 3.5, there exists a-dependent constants 0<c1<c20<c_1<c_2 such that it holds with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that c1j−a≤μj(⊤)≤c2j−a.c_1j^-a≤ _j(SHS )≤ c_2j^-a. Lemma G.2 (Eigenvalues of f(q)H_f^(q), multiplicative quantization, an upper bound). If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-multiplicative, under the unbiased quantization Assumption 3.1, the power-law spectrum Assumption 3.5, it holds with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≲(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a. _j(H_f^(q)) (1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a. Proof. Under multiplicative quantization, it holds f(q)⪯ _f^(q) (1+ϵ¯f)(1+ϵ¯d)s[s()s()⊤] (1+ ε_f)(1+ ε_d)E_Q_s [Q_s(S)HQ_s(S) ] ⪯ (1+ϵ¯f)(1+ϵ¯d)(s[ϵ(s)ϵ(s)⊤]+⊤). (1+ ε_f)(1+ ε_d) (E_Q_s [ ε^(s)H ε^(s) ]+SHS ). Note that s[ϵ(s)ϵ(s)⊤]⪯ϵ¯s⊤E_Q_s [ ε^(s)H ε^(s) ] ε_sSHS , we have, f(q)⪯(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤.H_f^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS . (93) By Lemma G.1, we have with probability at least 1−e−Ω(M)1-e^- (M), μj(f(q))≾(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a. _j(H_f^(q)) (1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a. (94) ∎ Lemma G.3 (Eigenvalues of f(q)H_f^(q), multiplicative quantization, a lower bound). If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-multiplicative, under the unbiased quantization Assumption 3.1, the power-law spectrum Assumption 3.5, it holds with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≳(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a. _j(H_f^(q)) (1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a. Proof. Under multiplicative quantization, it holds f(q)⪰ _f^(q) (1+ϵ¯f)(1+ϵ¯d)s[s()s()⊤] (1+ ε_f)(1+ ε_d)E_Q_s [Q_s(S)HQ_s(S) ] ⪰ (1+ϵ¯f)(1+ϵ¯d)(s[ϵ(s)ϵ(s)⊤]+⊤). (1+ ε_f)(1+ ε_d) (E_Q_s [ ε^(s)H ε^(s) ]+SHS ). Note that s[ϵ(s)ϵ(s)⊤]⪰ϵ¯s⊤E_Q_s [ ε^(s)H ε^(s) ] ε_sSHS , we have, f(q)⪰(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤.H_f^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS . (95) By Lemma G.1, we have with probability at least 1−e−Ω(M)1-e^- (M), μj(f(q))⪰(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)j−a. _j(H_f^(q)) (1+ ε_f)(1+ ε_d)(1+ ε_s)j^-a. (96) ∎ Lemma G.4 (Eigenvalues of f(q)H_f^(q), additive quantization, an upper bound). If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-additive, under the unbiased quantization Assumption 3.1, the power-law spectrum Assumption 3.5, it holds with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≲j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM. _j(H_f^(q)) j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM. Proof. Under additive quantization, it holds f(q)⪯ _f^(q) ϵ¯fM+s[s()s()⊤]+ϵ¯ds[s()s()⊤] ε_fI_M+E_Q_s [Q_s(S)HQ_s(S) ]+ ε_dE_Q_s [Q_s(S)Q_s(S) ] = = ϵ¯fM+s[s()s()⊤]+ϵ¯d⊤+ϵ¯ds[ϵ(s)ϵ(s)⊤] ε_fI_M+E_Q_s [Q_s(S)HQ_s(S) ]+ ε_dSS + ε_dE_Q_s [ ε^(s) ε^(s) ] = = ϵ¯fM+s[ϵ(s)ϵ(s)⊤]+⊤+ϵ¯d⊤+ϵ¯ds[ϵ(s)ϵ(s)⊤] ε_fI_M+E_Q_s [ ε^(s)H ε^(s) ]+SHS + ε_dSS + ε_dE_Q_s [ ε^(s) ε^(s) ] ⪯ ϵ¯fM+tr()ϵ¯sM+⊤+ϵ¯d⊤+ϵ¯dϵ¯spM ε_fI_M+tr(H) ε_sI_M+SHS + ε_dSS + ε_d ε_spI_M = = (ϵ¯f+tr()ϵ¯s+ϵ¯dϵ¯sp)M+⊤+ϵ¯d⊤. ( ε_f+tr(H) ε_s+ ε_d ε_sp)I_M+SHS + ε_dSS . We then focus on the eigenvalues of ⊤SS . We write =(1,…,p),i∼(0,1MM),i≥1.S=(s_1,...,s_p), _i (0, 1MI_M ), i≥ 1. For any unit vector ∈ℝMv ^M, each i⊤s_i v is sub-Gaussian. By Bernstein inequality, with probability at least 1−e−Ω(M)1-e^- (M), for every unit vector ∈ℝMv ^M, 141414We consider p≫Mp M. ⊤⊤=∑i=1p(i⊤)2≂pM.v SS v= _i=1^p(s_i v)^2 pM. That is, with probability at least 1−e−Ω(M)1-e^- (M), μmin(ϵ¯d⊤)≂ϵ¯dpM,μmax(ϵ¯d⊤)≂ϵ¯dpM. _ min( ε_dSS ) ε_d pM, _ max( ε_dSS ) ε_d pM. Therefore, together with Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≲j−a+ϵ¯f+(1+ϵ¯dp)ϵ¯s+ϵ¯dpM. _j(H_f^(q)) j^-a+ ε_f+(1+ ε_dp) ε_s+ ε_d pM. (97) ∎ Lemma G.5 (Eigenvalues of f(q)H_f^(q), additive quantization, a lower bound). If there exist constants ϵ¯s,ϵ¯d,ϵ¯f ε_s, ε_d, ε_f such that for i∈s,d,fi∈\s,d,f\, i(⋅)Q_i(·) is ϵ¯i ε_i-additive, under the unbiased quantization Assumption 3.1, the power-law spectrum Assumption 3.5, it holds with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≳j−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. _j(H_f^(q)) j^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. Proof. Under additive quantization, it holds f(q)⪰ _f^(q) ϵ¯fM+s[s()s()⊤]+ϵ¯ds[s()s()⊤] ε_fI_M+E_Q_s [Q_s(S)HQ_s(S) ]+ ε_dE_Q_s [Q_s(S)Q_s(S) ] = = ϵ¯fM+s[s()s()⊤]+ϵ¯d⊤+ϵ¯ds[ϵ(s)ϵ(s)⊤] ε_fI_M+E_Q_s [Q_s(S)HQ_s(S) ]+ ε_dSS + ε_dE_Q_s [ ε^(s) ε^(s) ] = = ϵ¯fM+s[ϵ(s)ϵ(s)⊤]+⊤+ϵ¯d⊤+ϵ¯ds[ϵ(s)ϵ(s)⊤] ε_fI_M+E_Q_s [ ε^(s)H ε^(s) ]+SHS + ε_dSS + ε_dE_Q_s [ ε^(s) ε^(s) ] ⪰ ϵ¯fM+ϵ¯str()M+⊤+ϵ¯d⊤+ϵ¯dϵ¯spM ε_fI_M+ ε_str(H)I_M+SHS + ε_dSS + ε_d ε_spI_M = = (ϵ¯f+(tr()+ϵ¯dp)ϵ¯s)M+⊤+ϵ¯d⊤. ( ε_f+(tr(H)+ ε_dp) ε_s)I_M+SHS + ε_dSS . We then focus on the eigenvalues of ⊤SS . We write =(1,…,p),i∼(0,1MM),i≥1.S=(s_1,...,s_p), _i (0, 1MI_M ), i≥ 1. For any unit vector ∈ℝMv ^M, each i⊤s_i v is sub-Gaussian. By Bernstein inequality, with probability at least 1−e−Ω(M)1-e^- (M), for every unit vector ∈ℝMv ^M, 151515We consider p≫Mp M. ⊤⊤=∑i=1p(i⊤)2≂pM.v SS v= _i=1^p(s_i v)^2 pM. That is, with probability at least 1−e−Ω(M)1-e^- (M), μmin(ϵ¯d⊤)≂ϵ¯dpM,μmax(ϵ¯d⊤)≂ϵ¯dpM. _ min( ε_dSS ) ε_d pM, _ max( ε_dSS ) ε_d pM. Therefore, together with Lemma G.1, with probability at least 1−e−Ω(M)1-e^- (M) for all j∈[M]j∈[M] that μj(f(q))≳j−a+ϵ¯f+ϵ¯s(1+ϵ¯dp)+ϵ¯dpM. _j(H_f^(q)) j^-a+ ε_f+ ε_s(1+ ε_dp)+ ε_d pM. (98) ∎ Lemma G.6 (Ratio of eigenvalues of k:∞k:∞k:∞⊤SI_k:∞H_k:∞I_k:∞S , Lemma G.5 in Lin et al. (2024)). Under Assumption 3.5, there exists some a-dependent constant c>0c>0 such that for all k≥1k≥ 1, the ratio between the M/2M/2-th and M-th eigenvalues μM/2(k:∞k:∞k:∞⊤)μM(k:∞k:∞k:∞⊤)≤c _M/2 (SI_k:∞H_k:∞I_k:∞S ) _M (SI_k:∞H_k:∞I_k:∞S )≤ c with probability at least 1−e−Ω(M)1-e^- (M). Further, for k≤Mk≤ M, μM/2(k:∞k:∞k:∞⊤)≲M−a,μM(k:∞k:∞k:∞⊤)≳M−a. _M/2 (SI_k:∞H_k:∞I_k:∞S ) M^-a, _M (SI_k:∞H_k:∞I_k:∞S ) M^-a. Appendix H Discussions on Assumptions In this section, we aim to verify Assumption 3.3 and Assumption 3.4 under the fourth order assumption and noise assumption with respect to full-precision data. For simplicity, we verify the upper bounds here. Assumption H.1. There is a constant α0>0 _0>0, such that for every PSD matrix A, we have [⊤⊤]⪯α0tr().E[x Axxx ] _0tr(HA)H. Assumption H.2. There exist constants σ¯02,Cy σ_0^2,C_y such that [(y−⟨∗,⟩)2⊤]⪯σ¯02,[y2⊤]⪯Cy,[(y−⟨∗,⟩)2]≤σ¯02.E [ (y- w^*,x )^2xx ] σ_0^2H,\ E[y^2xx ] C_yH,\ E [ (y- w^*,x )^2 ]≤ σ_0^2. We consider specific quantization schemes. Example H.1. Consider the following element-wise stochastic quantization (x)=s⋅⌊xs⌋,w.p.⌈x/s⌉−x/s⌈x/s⌉−⌊x/s⌋⌈xs⌉,w.p.x/s−⌊x/s⌋⌈x/s⌉−⌊x/s⌋,s=2⌊log2x⌋−m,multiplicative2−b,additive.Q(x)=s· cases xs ,\ w.p.\ x/s -x/s x/s - x/s \\ xs ,\ w.p.\ x/s- x/s x/s - x/s cases, s= cases2 _2x -m,\ & multiplicative\\ 2^-b,\ & additive cases. We first compute the conditional second moment and fourth moment under Example H.1. Regarding the conditional second moment, [((x)−x)2|x]= [(Q(x)-x)^2|x ]= (x−s⌊xs⌋)2⌈x/s⌉−x/s⌈x/s⌉−⌊x/s⌋+(s⌈xs⌉−x)2x/s−⌊x/s⌋⌈x/s⌉−⌊x/s⌋ (x-s xs )^2 x/s -x/s x/s - x/s + (s xs -x )^2 x/s- x/s x/s - x/s (99) = = s2(⌈x/s⌉−x/s)(x/s−⌊x/s⌋) s^2 ( x/s -x/s ) (x/s- x/s ) ≲ x22−2m,multiplicative2−2b,additive. . Regarding the fourth moment, [((x)−x)4|x]= [ (Q(x)-x )^4|x ]= (x−s⌊xs⌋)4⌈x/s⌉−x/s⌈x/s⌉−⌊x/s⌋+(s⌈xs⌉−x)4x/s−⌊x/s⌋⌈x/s⌉−⌊x/s⌋ (x-s xs )^4 x/s -x/s x/s - x/s + (s xs -x )^4 x/s- x/s x/s - x/s (100) = = s4(⌈x/s⌉−x/s)(x/s−⌊x/s⌋)[(x/s−⌊xs⌋)3+(⌈xs⌉−x/s)3] s^4 ( x/s -x/s ) (x/s- x/s ) [ (x/s- xs )^3+ ( xs -x/s )^3 ] ≲ x42−4m,multiplicative2−4b,additive. . Motivated by (100), we consider the strong multiplicative and additive quantization below for theoretical simplicity. We then verify the upper bounds in Assumption 3.3 and Assumption 3.4 under these quantization schemes. Definition H.1 (Strong multiplicative quantization). We call quantization Q is strong ϵε-multiplicative if [ϵ⊤ϵϵϵ⊤|]⪯ϵ⊤⊤,∀,E [ ε A ε ε ε |u ] εu Auuu , , where ϵ=()−u ε=Q(u)-u. We would like to remark that, for multiplicative quantization to matrix X, we extend the definition to [tr(⊤)⊤|]⪯ϵtr(⊤)⊤E [tr (A B ) B |U ] (AUBU )UBU for any PSD matrix B, where =()− =Q(U)-U. Definition H.2 (Strong additive quantization). We call quantization Q is strong ϵε-additive if [ϵ⊤ϵϵϵ⊤|]⪯ϵtr(),∀,E [ ε A ε ε ε |u ] (A)I, , where ϵ=()−u ε=Q(u)-u. We would like to remark that, for additive quantization to matrix X, we extend the definition to [tr(⊤)⊤|]⪯ϵtr()tr()2E [tr (A B ) B |U ] (A)tr(B)^2I for any PSD matrix B, where =()− =Q(U)-U. H.1 Fourth-order Assumption We aim to verify the fourth-order Assumption 3.3 in this subsection. Rewrite ~(q) x^(q) as ~(q)=(q)(q)+ϵ(f),(q)=+ϵ(d). x^(q)=S^(q)x^(q)+ ε^(f),\ x^(q)=x+ ε^(d). [(~(q))⊤~(q)~(q)(~(q))⊤]= [( x^(q)) A x^(q) x^(q)( x^(q)) ]= [(~(q))⊤~(q)((q)(q)+ϵ(f))((q)(q)+ϵ(f))⊤] [( x^(q)) A x^(q)(S^(q)x^(q)+ ε^(f))(S^(q)x^(q)+ ε^(f)) ] ⪯ 2[(~(q))⊤~(q)(q)(q)(q)⊤(q)⊤]+2[(~(q))⊤~(q)ϵ(f)ϵ(f)⊤]. 2E [( x^(q)) A x^(q)S^(q)x^(q)x^(q) S^(q) ]+2E [( x^(q)) A x^(q) ε^(f) ε^(f) ]. Note that by ~(q)=(q)(q)+ϵ(f) x^(q)=S^(q)x^(q)+ ε^(f), we have (~(q))⊤~(q)= ( x^(q)) A x^(q)= ((q)(q)+ϵ(f))⊤((q)(q)+ϵ(f)) (S^(q)x^(q)+ ε^(f)) A(S^(q)x^(q)+ ε^(f)) ≤ ≤ 2((q)(q))⊤((q)(q))+2ϵ(f)⊤ϵ(f), 2(S^(q)x^(q)) A(S^(q)x^(q))+2 ε^(f) A ε^(f), it follows that [(~(q))⊤~(q)~(q)(~(q))⊤]⪯ [( x^(q)) A x^(q) x^(q)( x^(q)) ] 4[(q)⊤(q)⊤(q)(q)(q)(q)(q)⊤(q)⊤]+4[ϵ(f)⊤ϵ(f)ϵ(f)ϵ(f)⊤] 4E [x^(q) S^(q) AS^(q)x^(q)S^(q)x^(q)x^(q) S^(q) ]+4E [ ε^(f) A ε^(f) ε^(f) ε^(f) ] + + 4[(q)⊤(q)⊤(q)(q)ϵ(f)ϵ(f)⊤]+4[ϵ(f)⊤ϵ(f)(q)(q)(q)⊤(q)⊤]. 4E [x^(q) S^(q) AS^(q)x^(q) ε^(f) ε^(f) ]+4E [ ε^(f) A ε^(f)S^(q)x^(q)x^(q) S^(q) ]. Further note that (q)=+ϵ(d)x^(q)=x+ ε^(d), we have (q)⊤(q)⊤(q)(q)≤2⊤(q)⊤(q)+2ϵ(d)⊤(q)⊤(q)ϵ(d), x^(q) S^(q) AS^(q)x^(q)≤ 2x S^(q) AS^(q)x+2 ε^(d) S^(q) AS^(q) ε^(d), (q)(q)(q)⊤(q)⊤⪯2(q)⊤(q)⊤+2(q)ϵ(d)ϵ(d)⊤(q)⊤, ^(q)x^(q)x^(q) S^(q) 2S^(q)xx S^(q) +2S^(q) ε^(d) ε^(d) S^(q) , it follows that [(~(q))⊤~(q)~(q)(~(q))⊤]⪯ [( x^(q)) A x^(q) x^(q)( x^(q)) ] 16[⊤(q)⊤(q)(q)⊤(q)⊤] 6E [x S^(q) AS^(q)xS^(q)xx S^(q) ] (101) + + 16[⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤] 6E [x S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] + + 16[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)⊤(q)⊤] 6E [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)xx S^(q) ] + + 16[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤] 6E [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ] + + 4[ϵ(f)⊤ϵ(f)ϵ(f)ϵ(f)⊤] 4E [ ε^(f) A ε^(f) ε^(f) ε^(f) ] + + 8[⊤(q)⊤(q)ϵ(f)ϵ(f)⊤]+8[ϵ(d)⊤(q)⊤(q)ϵ(d)ϵ(f)ϵ(f)⊤] 8E [x S^(q) AS^(q)x ε^(f) ε^(f) ]+8E [ ε^(d) S^(q) AS^(q) ε^(d) ε^(f) ε^(f) ] + + 8[ϵ(f)⊤ϵ(f)(q)⊤(q)⊤]+8[ϵ(f)⊤ϵ(f)(q)ϵ(d)ϵ(d)⊤(q)⊤]. 8E [ ε^(f) A ε^(f)S^(q)xx S^(q) ]+8E [ ε^(f) A ε^(f)S^(q) ε^(d) ε^(d) S^(q) ]. We then bound [(~(q))⊤~(q)~(q)(~(q))⊤]E [( x^(q)) A x^(q) x^(q)( x^(q)) ] in (strong) multiplicative and (strong) additive quantization cases respectively, under assumptions on full-precision data. H.1.1 Multiplicative Quantization Lemma H.1. If for each i=d,f,si=d,f,s, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-multiplicative and iQ_i is strong ϵ¯i′ ε_i -multiplicative, then under Assumption H.1, we have [(~(q))⊤~(q)~(q)(~(q))⊤]≤αMtr(f(q))f(q),E [( x^(q)) A x^(q) x^(q)( x^(q)) ]≤ _Mtr (H_f^(q)A )H_f^(q), where αM≲α0(1+ϵ¯f+ϵ¯f′)(1+ϵ¯d+ϵ¯d′)(1+ϵ¯s+ϵ¯s′)(1+ϵ¯f)2(1+ϵ¯d)2(1+ϵ¯s)2 _M _0(1+ ε_f+ ε_f )(1+ ε_d+ ε_d )(1+ ε_s+ ε_s )(1+ ε_f)^2(1+ ε_d)^2(1+ ε_s)^2. Proof. We prove by (101). Under Assumption H.1, [⊤(q)⊤(q)(q)⊤(q)⊤]⪯α0[(q)tr((q)⊤(q))(q)⊤]. [x S^(q) AS^(q)xS^(q)xx S^(q) ] _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. (102) As dQ_d is ϵ¯d ε_d-multiplicative, it holds [⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [x S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] ϵ¯d[⊤(q)⊤(q)(q)⊤(q)⊤] ε_dE [x S^(q) AS^(q)xS^(q)xx S^(q) ] (103) ⪯ ϵ¯dα0[(q)tr((q)⊤(q))(q)⊤], ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ], where the last inequality reuses (102). Similarly, [ϵ(d)⊤(q)⊤(q)ϵ(d)(q)⊤(q)⊤]⪯ [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)xx S^(q) ] ϵ¯d[⊤(q)⊤(q)(q)⊤(q)⊤] ε_dE [x S^(q) AS^(q)xS^(q)xx S^(q) ] (104) ⪯ ϵ¯dα0[(q)tr((q)⊤(q))(q)⊤]. ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. Note that fQ_f is ϵ¯f ε_f-multiplicative, it follows [⊤(q)⊤(q)ϵ(f)ϵ(f)⊤]⪯ [x S^(q) AS^(q)x ε^(f) ε^(f) ] ϵ¯f[⊤(q)⊤(q)(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_fE [x S^(q) AS^(q)xS^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] (105) ⪯ 2ϵ¯f[⊤(q)⊤(q)(q)⊤(q)⊤] 2 ε_fE [x S^(q) AS^(q)xS^(q)xx S^(q) ] + + 2ϵ¯f[⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_fE [x S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯f(1+ϵ¯d)[⊤(q)⊤(q)(q)⊤(q)⊤] 2 ε_f(1+ ε_d)E [x S^(q) AS^(q)xS^(q)xx S^(q) ] ⪯ 2ϵ¯f(1+ϵ¯d)α0[(q)tr((q)⊤(q))(q)⊤], 2 ε_f(1+ ε_d) _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ], where the third inequality holds as dQ_d is ϵ¯d ε_d-multiplicative and the last inequality reuses (102). Similarly, [ϵ(d)⊤(q)⊤(q)ϵ(d)ϵ(f)ϵ(f)⊤]⪯ [ ε^(d) S^(q) AS^(q) ε^(d) ε^(f) ε^(f) ] ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] (106) ⪯ 2ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)(⊤+ϵ(d)ϵ(d)⊤)(q)⊤] 2 ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)(xx + ε^(d) ε^(d) )S^(q) ] ⪯ 2ϵ¯fϵ¯dα0[(q)tr((q)⊤(q))(q)⊤] 2 ε_f ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ] + + 2ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤]. 2 ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ]. [ϵ(f)⊤ϵ(f)(q)⊤(q)⊤]⪯ [ ε^(f) A ε^(f)S^(q)xx S^(q) ] ϵ¯f[(+ϵ(d))⊤(q)⊤(q)(+ϵ(d))(q)⊤(q)⊤] ε_fE [(x+ ε^(d)) S^(q) AS^(q)(x+ ε^(d))S^(q)xx S^(q) ] (107) ⪯ 2ϵ¯f[⊤(q)⊤(q)(q)⊤(q)⊤] 2 ε_fE [x S^(q) AS^(q)xS^(q)xx S^(q) ] + + 2ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)⊤(q)⊤] 2 ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)xx S^(q) ] ⪯ 2ϵ¯f(1+ϵ¯d)α0[(q)tr((q)⊤(q))(q)⊤]. 2 ε_f(1+ ε_d) _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. [ϵ(f)⊤ϵ(f)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [ ε^(f) A ε^(f)S^(q) ε^(d) ε^(d) S^(q) ] ϵ¯f[(+ϵ(d))⊤(q)⊤(q)(+ϵ(d))(q)ϵ(d)ϵ(d)⊤(q)⊤] ε_fE [(x+ ε^(d)) S^(q) AS^(q)(x+ ε^(d))S^(q) ε^(d) ε^(d) S^(q) ] (108) ⪯ 2ϵ¯f[⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_fE [x S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] + + 2ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯fϵ¯dα0[(q)tr((q)⊤(q))(q)⊤] 2 ε_f ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ] + + 2ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤]. 2 ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ]. Regarding the fourth-order quantization terms, by the definition of strong multiplicative quantization (Definition H.1), it holds [ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ] ϵ¯d′[⊤(q)⊤(q)(q)⊤(q)⊤] ε_d E [x S^(q) AS^(q)xS^(q)xx S^(q) ] (109) ⪯ ϵ¯d′α0[(q)tr((q)⊤(q))(q)⊤]. ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. [ϵ(f)⊤ϵ(f)ϵ(f)ϵ(f)⊤]⪯ [ ε^(f) A ε^(f) ε^(f) ε^(f) ] ϵ¯f′[(+ϵ(d))⊤(q)⊤(q)⊤(q)(+ϵ(d))(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_f E [(x+ ε^(d)) S^(q) S^(q) AS^(q)(x+ ε^(d))S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] (110) ⪯ 4ϵ¯f′[⊤(q)⊤(q)⊤(q)(q)⊤(q)⊤] 4 ε_f E [x S^(q) S^(q) AS^(q)xS^(q)xx S^(q) ] + + 4ϵ¯f′[⊤(q)⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤] 4 ε_f E [x S^(q) S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] + + 4ϵ¯f′[ϵ(d)⊤(q)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤] 4 ε_f E [ ε^(d) S^(q) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ] + + 4ϵ¯f′[ϵ(d)⊤(q)⊤(q)⊤(q)ϵ(d)(q)⊤(q)⊤] 4 ε_f E [ ε^(d) S^(q) S^(q) AS^(q) ε^(d)S^(q)xx S^(q) ] ⪯ 4ϵ¯f′α0[(q)tr((q)⊤(q))(q)⊤] 4 ε_f _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ] + + 8ϵ¯f′ϵ¯dα0[(q)tr((q)⊤(q))(q)⊤] 8 ε_f ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ] + + 4ϵ¯f′ϵ¯d′α0[(q)tr((q)⊤(q))(q)⊤]. 4 ε_f ε_d _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. Therefore, applying (101), it holds [(~(q))⊤~(q)~(q)(~(q))⊤]⪯Cα0(1+ϵ¯f+ϵ¯f′)(1+ϵ¯d+ϵ¯d′)[tr((q)(q)⊤)(q)(q)⊤], [( x^(q)) A x^(q) x^(q)( x^(q)) ] C _0(1+ ε_f+ ε_f )(1+ ε_d+ ε_d )E [tr (S^(q)HS^(q) A )S^(q)HS^(q) ], (111) where C>0C>0 is a constant. Note that [tr((q)(q)⊤)(q)(q)⊤]= [tr (S^(q)HS^(q) A )S^(q)HS^(q) ]= [tr((+ϵ(s))(+ϵ(s))⊤)(q)(q)⊤] [tr ((S+ ε^(s))H(S+ ε^(s)) A )S^(q)HS^(q) ] (112) ⪯ 2[tr(⊤)(q)(q)⊤] 2E [tr (SHS A )S^(q)HS^(q) ] + + 2[tr(ϵ(s)ϵ(s)⊤)(q)(q)⊤] 2E [tr ( ε^(s)H ε^(s) A )S^(q)HS^(q) ] ⪯ 4[tr(⊤)⊤] 4E [tr (SHS A )SHS ] + + 4[tr(⊤)ϵ(s)ϵ(s)⊤] 4E [tr (SHS A ) ε^(s)H ε^(s) ] + + 4[tr(ϵ(s)ϵ(s)⊤)⊤] 4E [tr ( ε^(s)H ε^(s) A )SHS ] + + 4[tr(ϵ(s)ϵ(s)⊤)ϵ(s)ϵ(s)⊤] 4E [tr ( ε^(s)H ε^(s) A ) ε^(s)H ε^(s) ] ⪯ 4(1+2ϵ¯s+ϵ¯s′)[tr(⊤)⊤], 4(1+2 ε_s+ ε_s )E [tr (SHS A )SHS ], we have [(~(q))⊤~(q)~(q)(~(q))⊤]⪯C′α0(1+ϵ¯f+ϵ¯f′)(1+ϵ¯d+ϵ¯d′)(1+ϵ¯s+ϵ¯s′)tr(⊤)⊤,E [( x^(q)) A x^(q) x^(q)( x^(q)) ] C _0(1+ ε_f+ ε_f )(1+ ε_d+ ε_d )(1+ ε_s+ ε_s )tr (SHS A )SHS , (113) where C′>0C >0 is a constant. By the definition of f(q)= _f^(q)= [f(s()d())f(s()d())⊤] [Q_f(Q_s(S)Q_d(x))Q_f(Q_s(S)Q_d(x)) ] (114) = = [(s()d()+ϵ(f))(s()d()+ϵ(f))⊤] [(Q_s(S)Q_d(x)+ ε^(f))(Q_s(S)Q_d(x)+ ε^(f)) ] ⪰ (1+ϵ¯f)[s()d()(s()d())⊤] (1+ ε_f)E [Q_s(S)Q_d(x)(Q_s(S)Q_d(x)) ] ⪰ (1+ϵ¯f)(1+ϵ¯d)[s()⊤s()⊤] (1+ ε_f)(1+ ε_d)E [Q_s(S)xx Q_s(S) ] ⪰ (1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤, (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS , together with (111) we have [(~(q))⊤~(q)~(q)(~(q))⊤]⪯C′α0(1+ϵ¯f+ϵ¯f′)(1+ϵ¯d+ϵ¯d′)(1+ϵ¯s+ϵ¯s′)(1+ϵ¯f)2(1+ϵ¯d)2(1+ϵ¯s)2tr(f(q))f(q).E [( x^(q)) A x^(q) x^(q)( x^(q)) ] C _0(1+ ε_f+ ε_f )(1+ ε_d+ ε_d )(1+ ε_s+ ε_s )(1+ ε_f)^2(1+ ε_d)^2(1+ ε_s)^2tr (H_f^(q)A )H_f^(q). (115) ∎ H.1.2 Additive Quantization Lemma H.2. If for each i=d,f,si=d,f,s, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-additive and iQ_i is strong ϵi′ _i -additive, then under Assumption H.1, we have [(~(q))⊤~(q)~(q)(~(q))⊤]≤αAtr(f(q))f(q),E [( x^(q)) A x^(q) x^(q)( x^(q)) ]≤ _Atr (H_f^(q)A )H_f^(q), where αA≲(1+α0)(1+ϵ¯d′ϵ¯d2+ϵ¯f′ϵ¯f2)(ϵ¯dϵ¯d(1+ϵ¯s+ϵ¯s′ϵ¯s)+ϵ¯fϵ¯f)2 _A (1+ _0) (1+ ε_d ε_d^2+ ε_f ε_f^2 ) ( ε_d ε_d (1+ ε_s+ ε_s ε_s )+ ε_f ε_f )^2. Proof. By Assumption H.1, [⊤(q)⊤(q)(q)⊤(q)⊤]⪯α0[(q)tr((q)⊤(q))(q)⊤]. [x S^(q) AS^(q)xS^(q)xx S^(q) ] _0E [S^(q)tr (HS^(q) AS^(q) )HS^(q) ]. (116) As dQ_d is ϵ¯d ε_d-additive, [⊤(q)⊤(q)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [x S^(q) AS^(q)xS^(q) ε^(d) ε^(d) S^(q) ] ϵ¯d[⊤(q)⊤(q)(q)(q)⊤] ε_dE [x S^(q) AS^(q)xS^(q)S^(q) ] (117) = = ϵ¯d[(q)tr((q)⊤(q))(q)⊤]. ε_dE [S^(q)tr (HS^(q) AS^(q) )S^(q) ]. Similarly, [ϵ(d)⊤(q)⊤(q)ϵ(d)(q)⊤(q)⊤]⪯ϵ¯d[(q)tr((q)⊤(q))(q)⊤]. [ ε^(d) S^(q) AS^(q) ε^(d)S^(q)xx S^(q) ] ε_dE [S^(q)tr (S^(q) AS^(q) )HS^(q) ]. (118) As fQ_f is ϵ¯f ε_f-additive, [⊤(q)⊤(q)ϵ(f)ϵ(f)⊤]⪯ [x S^(q) AS^(q)x ε^(f) ε^(f) ] ϵ¯f[⊤(q)⊤(q)] ε_fE [x S^(q) AS^(q)xI ] (119) = = ϵ¯f[tr((q)⊤(q))]. ε_fE [tr (HS^(q) AS^(q) )I ]. By the fact that both dQ_d and fQ_f are additive quantization, [ϵ(d)⊤(q)⊤(q)ϵ(d)ϵ(f)ϵ(f)⊤]⪯ [ ε^(d) S^(q) AS^(q) ε^(d) ε^(f) ε^(f) ] ϵ¯f[ϵ(d)⊤(q)⊤(q)ϵ(d)] ε_fE [ ε^(d) S^(q) AS^(q) ε^(d)I ] (120) ⪯ ϵ¯fϵ¯d[tr((q)⊤(q))]. ε_f ε_dE [tr (S^(q) AS^(q) )I ]. Similarly, [ϵ(f)⊤ϵ(f)(q)⊤(q)⊤]⪯ [ ε^(f) A ε^(f)S^(q)xx S^(q) ] ϵ¯f[tr()(q)(q)⊤]. ε_fE [tr(A)S^(q)HS^(q) ]. (121) [ϵ(f)⊤ϵ(f)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ϵ¯fϵ¯d[tr()(q)(q)⊤].E [ ε^(f) A ε^(f)S^(q) ε^(d) ε^(d) S^(q) ] ε_f ε_dE [tr(A)S^(q)S^(q) ]. (122) Under the strong additive property of dQ_d and sQ_s, it follows [ϵ(d)⊤(q)⊤(q)ϵ(d)(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [ ε^(d) S^(q) AS^(q) ε^(d)S^(q) ε^(d) ε^(d) S^(q) ] ϵ¯d′[(q)tr((q)⊤(q))(q)⊤]. ε_d E [S^(q)tr (S^(q) AS^(q) )S^(q) ]. (123) [ϵ(f)⊤ϵ(f)ϵ(f)ϵ(f)⊤]⪯ϵ¯f′tr(). [ ε^(f) A ε^(f) ε^(f) ε^(f) ] ε_f tr(A)I. (124) Applying (101), it holds [(~(q))⊤~(q)~(q)(~(q))⊤] [( x^(q)) A x^(q) x^(q)( x^(q)) ] (125) ⪯ C(1+α0)(1+ϵ¯d′ϵ¯d2+ϵ¯f′ϵ¯f2)[tr(((q)(+ϵ¯d)(q)⊤+ϵ¯f))((q)(+ϵ¯d)(q)⊤+ϵ¯f)]. C(1+ _0) (1+ ε_d ε_d^2+ ε_f ε_f^2 )E [tr ((S^(q)(H+ ε_dI)S^(q) + ε_fI)A )(S^(q)(H+ ε_dI)S^(q) + ε_fI) ]. where C>0C>0 is constant. Note that [tr(((q)(+ϵ¯d)(q)⊤+ϵ¯f))((q)(+ϵ¯d)(q)⊤+ϵ¯f)] [tr ((S^(q)(H+ ε_dI)S^(q) + ε_fI)A )(S^(q)(H+ ε_dI)S^(q) + ε_fI) ] (126) ⪯ 4[tr(((+ϵ¯d)⊤+ϵ¯f))((+ϵ¯d)⊤+ϵ¯f)] 4E [tr ((S(H+ ε_dI)S + ε_fI)A )(S(H+ ε_dI)S + ε_fI) ] + + 4[tr(((+ϵ¯d)⊤+ϵ¯f))(ϵ(s)(+ϵ¯d)ϵ(s)⊤+ϵ¯f)] 4E [tr ((S(H+ ε_dI)S + ε_fI)A )( ε^(s)(H+ ε_dI) ε^(s) + ε_fI) ] + + 4[tr((ϵ(s)(+ϵ¯d)ϵ(s)⊤+ϵ¯f))((+ϵ¯d)⊤+ϵ¯f)] 4E [tr (( ε^(s)(H+ ε_dI) ε^(s) + ε_fI)A )(S(H+ ε_dI)S + ε_fI) ] + + 4[tr((ϵ(s)(+ϵ¯d)ϵ(s)⊤+ϵ¯f))(ϵ(s)(+ϵ¯d)ϵ(s)⊤+ϵ¯f)] 4E [tr (( ε^(s)(H+ ε_dI) ε^(s) + ε_fI)A )( ε^(s)(H+ ε_dI) ε^(s) + ε_fI) ] ⪯ 16tr(((+ϵ¯d)⊤+ϵ¯f+(ϵ¯s+ϵ¯s′)tr(+ϵ¯d)))[(+ϵ¯d)⊤+ϵ¯f+(ϵ¯s+ϵ¯s′)tr(+ϵ¯d)], 6tr ((S(H+ ε_dI)S + ε_fI+( ε_s+ ε_s )tr(H+ ε_dI)I)A ) [S(H+ ε_dI)S + ε_fI+( ε_s+ ε_s )tr(H+ ε_dI)I ], further by the definition of f(q)= _f^(q)= [f(s()d())f(s()d())⊤] [Q_f(Q_s(S)Q_d(x))Q_f(Q_s(S)Q_d(x)) ] (127) = = [(s()d()+ϵ(f))(s()d()+ϵ(f))⊤] [(Q_s(S)Q_d(x)+ ε^(f))(Q_s(S)Q_d(x)+ ε^(f)) ] ⪰ [s()d()(s()d())⊤]+ϵ¯f [Q_s(S)Q_d(x)(Q_s(S)Q_d(x)) ]+ ε_fI ⪰ [s()(+ϵ¯d)s()⊤]+ϵ¯f [Q_s(S)(H+ ε_dI)Q_s(S) ]+ ε_fI ⪰ [s()d()(s()d())⊤]+ϵ¯f [Q_s(S)Q_d(x)(Q_s(S)Q_d(x)) ]+ ε_fI = = (+ϵ¯d)⊤+[ϵ(s)(+ϵ¯d)ϵ(s)⊤]+ϵ¯f (H+ ε_dI)S +E [ ε^(s)(H+ ε_dI) ε^(s) ]+ ε_fI ⪰ (+ϵ¯d)⊤+ϵ¯str(+ϵ¯d)+ϵ¯f, (H+ ε_dI)S + ε_str (H+ ε_dI )I+ ε_fI, we have [tr(((q)(+ϵ¯d)(q)⊤+ϵ¯f))((q)(+ϵ¯d)(q)⊤+ϵ¯f)] [tr ((S^(q)(H+ ε_dI)S^(q) + ε_fI)A )(S^(q)(H+ ε_dI)S^(q) + ε_fI) ] (128) ⪯ 16(ϵ¯dϵ¯d(1+ϵ¯s+ϵ¯s′ϵ¯s)+ϵ¯fϵ¯f)2tr(f(q))f(q). 6 ( ε_d ε_d (1+ ε_s+ ε_s ε_s )+ ε_f ε_f )^2tr(H_f^(q)A)H_f^(q). Therefore, together with (125) and (128), [(~(q))⊤~(q)~(q)(~(q))⊤]⪯C′(1+α0)(1+ϵ¯d′ϵ¯d2+ϵ¯f′ϵ¯f2)(ϵ¯dϵ¯d(1+ϵ¯s+ϵ¯s′ϵ¯s)+ϵ¯fϵ¯f)2tr(f(q))f(q).E [( x^(q)) A x^(q) x^(q)( x^(q)) ] C (1+ _0) (1+ ε_d ε_d^2+ ε_f ε_f^2 ) ( ε_d ε_d (1+ ε_s+ ε_s ε_s )+ ε_f ε_f )^2tr(H_f^(q)A)H_f^(q). (129) ∎ H.2 Second-order Noise Assumption In this section, we aim to verify the second-order noise assumption. Recall that ξ:=l(y)−⟨(q)∗,~(q)⟩ξ:=Q_l(y)- v^(q)^*, x^(q) and (q)∗=(f(q))−1∗v^(q)^*=(H_f^(q))^-1SHw^*. [ξ2~(q)(~(q))⊤]= [ξ^2 x^(q)( x^(q)) ]= [(l(y)−⟨(q)∗,~(q)⟩)2~(q)(~(q))⊤] [(Q_l(y)- v^(q)^*, x^(q) )^2 x^(q)( x^(q)) ] = = [(l(y)−⟨(f(q))−1∗,~(q)⟩)2~(q)(~(q))⊤] [ (Q_l(y)- (H_f^(q))^-1SHw^*, x^(q) )^2 x^(q)( x^(q)) ] = = [(l(y)−y+y−⟨∗,⟩+⟨∗,⟩−⟨(f(q))−1∗,~(q)⟩)2~(q)(~(q))⊤] [ (Q_l(y)-y+y- w^*,x + w^*,x - (H_f^(q))^-1SHw^*, x^(q) )^2 x^(q)( x^(q)) ] ⪯ 3[(l(y)−y)2~(q)(~(q))⊤]+3[(y−⟨∗,⟩)2~(q)(~(q))⊤] 3E [ (Q_l(y)-y )^2 x^(q)( x^(q)) ]+3E [ (y- w^*,x )^2 x^(q)( x^(q)) ] + + 3[(⟨∗,⟩−⟨(f(q))−1∗,~(q)⟩)2~(q)(~(q))⊤]. 3E [ ( w^*,x - (H_f^(q))^-1SHw^*, x^(q) )^2 x^(q)( x^(q)) ]. Recall that ~(q)=(q)(+ϵ(d))+ϵ(f) x^(q)=S^(q)(x+ ε^(d))+ ε^(f), it follows that (⟨∗,⟩−⟨(f(q))−1∗,~(q)⟩)2≤2⟨∗−(q)⊤(f(q))−1∗,⟩2+2⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2. ( w^*,x - (H_f^(q))^-1SHw^*, x^(q) )^2≤ 2 w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2+2 (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2. Further note that ~(q)(~(q))⊤⪯ x^(q)( x^(q)) 2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤+2ϵ(f)ϵ(f)⊤ 2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) +2 ε^(f) ε^(f) ⪯ 4(q)⊤(q)⊤+4(q)ϵ(d)ϵ(d)⊤(q)⊤+2ϵ(f)ϵ(f)⊤, 4S^(q)xx S^(q) +4S^(q) ε^(d) ε^(d) S^(q) +2 ε^(f) ε^(f) , we have [ξ2~(q)(~(q))⊤]⪯ [ξ^2 x^(q)( x^(q)) ] 12[(l(y)−y)2(q)⊤(q)⊤]+12[(l(y)−y)2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2E [ (Q_l(y)-y )^2S^(q)xx S^(q) ]+2E [ (Q_l(y)-y )^2S^(q) ε^(d) ε^(d) S^(q) ] (130) + + 6[(l(y)−y)2ϵ(f)ϵ(f)⊤]+6[(y−⟨∗,⟩)2ϵ(f)ϵ(f)⊤] 6E [ (Q_l(y)-y )^2 ε^(f) ε^(f) ]+6E [ (y- w^*,x )^2 ε^(f) ε^(f) ] + + 12[(y−⟨∗,⟩)2(q)⊤(q)⊤]+12[(y−⟨∗,⟩)2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2E [ (y- w^*,x )^2S^(q)xx S^(q) ]+2E [ (y- w^*,x )^2S^(q) ε^(d) ε^(d) S^(q) ] + + 24[⟨∗−(q)⊤(f(q))−1∗,⟩2(q)⊤(q)⊤] 4E [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q)xx S^(q) ] + + 24[⟨∗−(q)⊤(f(q))−1∗,⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 4E [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q) ε^(d) ε^(d) S^(q) ] + + 12[⟨∗−(q)⊤(f(q))−1∗,⟩2ϵ(f)ϵ(f)⊤] 2E [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2 ε^(f) ε^(f) ] + + 24[⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2(q)⊤(q)⊤] 4E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2S^(q)xx S^(q) ] + + 24[⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 4E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2S^(q) ε^(d) ε^(d) S^(q) ] + + 12[⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2ϵ(f)ϵ(f)⊤]. 2E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2 ε^(f) ε^(f) ]. For simplicity, we merely verify the assumption under multiplicative quantization. H.2.1 Multiplicative Quantization Lemma H.3. Under Assumption H.1, H.2, if for each i=d,f,si=d,f,s, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-multiplicative and iQ_i is strong ϵ¯i′ ε_i -multiplicative, if lQ_l is ϵ¯l ε_l-multiplicative, then [ξ2~(q)(~(q))⊤]⪯σ¯M2f(q),E [ξ^2 x^(q)( x^(q)) ] σ_M^2H_f^(q), where σ¯M2≲(1+α0‖∗‖2)(1+ϵ¯s+ϵ¯s′)[(1+ϵ¯d)(1+ϵ¯f)(ϵ¯lCy+σ¯02)+(1+ϵ¯d+ϵ¯d′)(1+ϵ¯f+ϵ¯f′)](1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s) σ_M^2 (1+ _0\|w^*\|_H^2)(1+ ε_s+ ε_s ) [(1+ ε_d)(1+ ε_f)( ε_lC_y+ σ_0^2)+(1+ ε_d+ ε_d )(1+ ε_f+ ε_f ) ](1+ ε_f)(1+ ε_d)(1+ ε_s). Proof. Under Assumption H.2 and lQ_l is (ϵ¯l,ϵ¯l ε_l, ε_l)-multiplicative [(l(y)−y)2(q)⊤(q)⊤]⪯ϵ¯lCy[(q)(q)].E [ (Q_l(y)-y )^2S^(q)xx S^(q) ] ε_lC_yE [S^(q)HS^(q) ]. (131) Under Assumption H.2 and for i=l,di=l,d, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-multiplicative [(l(y)−y)2(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ϵ¯lϵ¯dCy[(q)(q)]. [ (Q_l(y)-y )^2S^(q) ε^(d) ε^(d) S^(q) ] ε_l ε_dC_yE [S^(q)HS^(q) ]. (132) Under Assumption H.2 and for i=l,f,di=l,f,d, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-multiplicative [(l(y)−y)2ϵ(f)ϵ(f)⊤]⪯ [ (Q_l(y)-y )^2 ε^(f) ε^(f) ] ϵ¯fϵ¯l[y2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_f ε_lE [y^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] (133) ⪯ 2ϵ¯fϵ¯l[y2(q)⊤(q)⊤]+2ϵ¯fϵ¯l[y2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_f ε_lE [y^2S^(q)xx S^(q) ]+2 ε_f ε_lE [y^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯fϵ¯l(1+ϵ¯d)Cy[(q)(q)]. 2 ε_f ε_l(1+ ε_d)C_yE [S^(q)HS^(q) ]. Under Assumption H.2 and for i=f,di=f,d, iQ_i is (ϵ¯i,ϵ¯i ε_i, ε_i)-multiplicative [(y−⟨∗,⟩)2ϵ(f)ϵ(f)⊤]⪯ [ (y- w^*,x )^2 ε^(f) ε^(f) ] ϵ¯f[(y−⟨∗,⟩)2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_fE [ (y- w^*,x )^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] (134) ⪯ 2ϵ¯f[(y−⟨∗,⟩)2(q)⊤(q)⊤] 2 ε_fE [ (y- w^*,x )^2S^(q)xx S^(q) ] + + 2ϵ¯f[(y−⟨∗,⟩)2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_fE [ (y- w^*,x )^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯f(1+ϵ¯d)[(y−⟨∗,⟩)2(q)⊤(q)⊤] 2 ε_f(1+ ε_d)E [ (y- w^*,x )^2S^(q)xx S^(q) ] ⪯ 2ϵ¯f(1+ϵ¯d)σ¯02[(q)(q)⊤]. 2 ε_f(1+ ε_d) σ_0^2E [S^(q)HS^(q) ]. Under Assumption H.2, [(y−⟨∗,⟩)2(q)⊤(q)⊤]⪯σ¯02[(q)(q)⊤]. [ (y- w^*,x )^2S^(q)xx S^(q) ] σ_0^2E [S^(q)HS^(q) ]. (135) Similarly, [(y−⟨∗,⟩)2(q)ϵ(d)ϵ(d)⊤(q)⊤]⪯ [ (y- w^*,x )^2S^(q) ε^(d) ε^(d) S^(q) ] ϵ¯d[(y−⟨∗,⟩)2(q)⊤(q)⊤]⪯ϵ¯dσ¯02[(q)(q)⊤]. ε_dE [ (y- w^*,x )^2S^(q)xx S^(q) ] ε_d σ_0^2E [S^(q)HS^(q) ]. (136) Under Assumption H.2, [⟨∗−(q)⊤(f(q))−1∗,⟩2(q)⊤(q)⊤] [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q)xx S^(q) ] (137) = = [⊤(∗−(q)⊤(f(q))−1∗)(∗−(q)⊤(f(q))−1∗)⊤(q)⊤(q)⊤] [x (w^*-S^(q) (H_f^(q))^-1SHw^* ) (w^*-S^(q) (H_f^(q))^-1SHw^* ) xS^(q)xx S^(q) ] ⪯ α0[tr((−(q)⊤(f(q))−1)∗∗⊤(−(q)⊤(f(q))−1)⊤)(q)(q)⊤]. _0E [tr (H (I-S^(q) (H_f^(q))^-1SH )w^*w^* (I-S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ]. Further, [⟨∗−(q)⊤(f(q))−1∗,⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q) ε^(d) ε^(d) S^(q) ] (138) ⪯ ϵ¯d[⟨∗−(q)⊤(f(q))−1∗,⟩2(q)⊤(q)⊤] ε_dE [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q)xx S^(q) ] ⪯ ϵ¯dα0[tr((−(q)⊤(f(q))−1)∗∗⊤(−(q)⊤(f(q))−1)⊤)(q)(q)⊤]. ε_d _0E [tr (H (I-S^(q) (H_f^(q))^-1SH )w^*w^* (I-S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ]. Similarly, [⟨∗−(q)⊤(f(q))−1∗,⟩2ϵ(f)ϵ(f)⊤] [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2 ε^(f) ε^(f) ] (139) = = ϵ¯f[⟨∗−(q)⊤(f(q))−1∗,⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] ε_fE [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] ⪯ 2(1+ϵ¯d)ϵ¯f[⟨∗−(q)⊤(f(q))−1∗,⟩2(q)⊤(q)⊤] 2(1+ ε_d) ε_fE [ w^*-S^(q) (H_f^(q))^-1SHw^*,x ^2S^(q)xx S^(q) ] ⪯ 2(1+ϵ¯d)ϵ¯fα0[tr((−(q)⊤(f(q))−1)∗∗⊤(−(q)⊤(f(q))−1)⊤)(q)(q)⊤]. 2(1+ ε_d) ε_f _0E [tr (H (I-S^(q) (H_f^(q))^-1SH )w^*w^* (I-S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ]. For the last three terms in (130), by the definition of multiplicative quantization, [⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2(q)⊤(q)⊤] [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2S^(q)xx S^(q) ] (140) ⪯ 2[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)⊤(q)⊤] 2E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q)xx S^(q) ] + + 2ϵ¯f[⟨(f(q))−1∗,(q)(+ϵ(d))⟩2(q)⊤(q)⊤] 2 ε_fE [ (H_f^(q))^-1SHw^*,S^(q)(x+ ε^(d)) ^2S^(q)xx S^(q) ] ⪯ [4(1+ϵ¯d)ϵ¯f+2ϵ¯d][⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] [4(1+ ε_d) ε_f+2 ε_d]E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] ⪯ [4(1+ϵ¯d)ϵ¯f+2ϵ¯d]α0[tr(((q)⊤(f(q))−1)∗∗⊤((q)⊤(f(q))−1)⊤)(q)(q)⊤]. [4(1+ ε_d) ε_f+2 ε_d] _0E [tr (H (S^(q) (H_f^(q))^-1SH )w^*w^* (S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ]. Similarly, under the definition of multiplicative quantization and strong multiplicative quantization, [⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2S^(q) ε^(d) ε^(d) S^(q) ] (141) ⪯ 2[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q) ε^(d) ε^(d) S^(q) ] + + 2[⟨(f(q))−1∗,ϵ(f)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2E [ (H_f^(q))^-1SHw^*, ε^(f) ^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯d′[⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] 2 ε_d E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] + + 2ϵ¯f[⟨(f(q))−1∗,(q)(+ϵ(d))⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 2 ε_fE [ (H_f^(q))^-1SHw^*,S^(q)(x+ ε^(d)) ^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2ϵ¯d′[⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] 2 ε_d E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] + + 4ϵ¯f[⟨(f(q))−1∗,(q)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 4 ε_fE [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q) ε^(d) ε^(d) S^(q) ] + + 4ϵ¯f[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 4 ε_fE [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ 2(1+2ϵ¯f)ϵ¯d′[⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] 2(1+2 ε_f) ε_d E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] + + 4ϵ¯fϵ¯d[⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] 4 ε_f ε_dE [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] = = [2(1+2ϵ¯f)ϵ¯d′+4ϵ¯fϵ¯d][⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] [2(1+2 ε_f) ε_d +4 ε_f ε_d]E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] ⪯ [2(1+2ϵ¯f)ϵ¯d′+4ϵ¯fϵ¯d]α0[tr(((q)⊤(f(q))−1)∗∗⊤((q)⊤(f(q))−1)⊤)(q)(q)⊤]. [2(1+2 ε_f) ε_d +4 ε_f ε_d] _0E [tr (H (S^(q) (H_f^(q))^-1SH )w^*w^* (S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ]. Further, [⟨(f(q))−1∗,(q)ϵ(d)+ϵ(f)⟩2ϵ(f)ϵ(f)⊤] [ (H_f^(q))^-1SHw^*,S^(q) ε^(d)+ ε^(f) ^2 ε^(f) ε^(f) ] (142) ⪯ 2[⟨(f(q))−1∗,(q)ϵ(d)⟩2ϵ(f)ϵ(f)⊤] 2E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2 ε^(f) ε^(f) ] + + 2[⟨(f(q))−1∗,ϵ(f)⟩2ϵ(f)ϵ(f)⊤] 2E [ (H_f^(q))^-1SHw^*, ε^(f) ^2 ε^(f) ε^(f) ] ⪯ 2ϵ¯f[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] 2 ε_fE [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] + + 2ϵ¯f′[⟨(f(q))−1∗,(q)(+ϵ(d))⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] 2 ε_f E [ (H_f^(q))^-1SHw^*,S^(q)(x+ ε^(d)) ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] ⪯ 2ϵ¯f[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] 2 ε_fE [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] + + 4ϵ¯f′[⟨(f(q))−1∗,(q)⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] 4 ε_f E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] + + 4ϵ¯f′[⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)(+ϵ(d))(+ϵ(d))⊤(q)⊤] 4 ε_f E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q)(x+ ε^(d))(x+ ε^(d)) S^(q) ] ⪯ [4ϵ¯f+8ϵ¯f′][⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)⊤(q)⊤] [4 ε_f+8 ε_f ]E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q)xx S^(q) ] + + [4ϵ¯f+8ϵ¯f′][⟨(f(q))−1∗,(q)ϵ(d)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] [4 ε_f+8 ε_f ]E [ (H_f^(q))^-1SHw^*,S^(q) ε^(d) ^2S^(q) ε^(d) ε^(d) S^(q) ] + + 8ϵ¯f′[⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] 8 ε_f E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] + + 8ϵ¯f′[⟨(f(q))−1∗,(q)⟩2(q)ϵ(d)ϵ(d)⊤(q)⊤] 8 ε_f E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q) ε^(d) ε^(d) S^(q) ] ⪯ [8ϵ¯f′(1+ϵ¯d)+(ϵ¯d+ϵ¯d′)(4ϵ¯f+8ϵ¯f′)][⟨(f(q))−1∗,(q)⟩2(q)⊤(q)⊤] [8 ε_f (1+ ε_d)+( ε_d+ ε_d )(4 ε_f+8 ε_f )]E [ (H_f^(q))^-1SHw^*,S^(q)x ^2S^(q)xx S^(q) ] ⪯ α0[8ϵ¯f′(1+ϵ¯d)+(ϵ¯d+ϵ¯d′)(4ϵ¯f+8ϵ¯f′)][tr((q)⊤(f(q))−1∗∗⊤⊤(f(q))−1(q))(q)(q)⊤]. _0[8 ε_f (1+ ε_d)+( ε_d+ ε_d )(4 ε_f+8 ε_f )]E [tr (S^(q) (H_f^(q))^-1SHw^*w^* HS (H_f^(q))^-1S^(q)H )S^(q)HS^(q) ]. Specifically, [tr((−(q)⊤(f(q))−1)∗∗⊤(−(q)⊤(f(q))−1)⊤)(q)(q)⊤] [tr (H (I-S^(q) (H_f^(q))^-1SH )w^*w^* (I-S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ] (143) ⪯ 2[tr((−⊤(f(q))−1)∗∗⊤(−⊤(f(q))−1)⊤)(q)(q)⊤] 2E [tr (H (I-S (H_f^(q))^-1SH )w^*w^* (I-S (H_f^(q))^-1SH ) )S^(q)HS^(q) ] + + 2[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)(q)(q)⊤] 2E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) )S^(q)HS^(q) ] ⪯ 8‖∗‖2[(q)(q)⊤]+4[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)⊤] 8\|w^*\|_H^2E [S^(q)HS^(q) ]+4E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) )SHS ] + + 4[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)ϵ(s)ϵ(s)⊤] 4E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) ) ε^(s)H ε^(s) ] ⪯ 8‖∗‖2[(q)(q)⊤]+4(ϵ¯s+ϵ¯s′)[tr((⊤(f(q))−1)∗∗⊤(⊤(f(q))−1)⊤)⊤] 8\|w^*\|_H^2E [S^(q)HS^(q) ]+4( ε_s+ ε_s )E [tr (H (S (H_f^(q))^-1SH )w^*w^* (S (H_f^(q))^-1SH ) )SHS ] ⪯ 8‖∗‖2[(q)(q)⊤]+4(ϵ¯s+ϵ¯s′)‖∗‖2⊤, 8\|w^*\|_H^2E [S^(q)HS^(q) ]+4( ε_s+ ε_s )\|w^*\|_H^2SHS , and [tr(((q)⊤(f(q))−1)∗∗⊤((q)⊤(f(q))−1)⊤)(q)(q)⊤] [tr (H (S^(q) (H_f^(q))^-1SH )w^*w^* (S^(q) (H_f^(q))^-1SH ) )S^(q)HS^(q) ] (144) ⪯ 2[tr((⊤(f(q))−1)∗∗⊤(⊤(f(q))−1)⊤)(q)(q)⊤] 2E [tr (H (S (H_f^(q))^-1SH )w^*w^* (S (H_f^(q))^-1SH ) )S^(q)HS^(q) ] + + 2[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)(q)(q)⊤] 2E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) )S^(q)HS^(q) ] ⪯ 2‖∗‖2[(q)(q)⊤]+4[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)⊤] 2\|w^*\|_H^2E [S^(q)HS^(q) ]+4E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) )SHS ] + + 4[tr((ϵ(s)⊤(f(q))−1)∗∗⊤(ϵ(s)⊤(f(q))−1)⊤)ϵ(s)ϵ(s)⊤] 4E [tr (H ( ε^(s) (H_f^(q))^-1SH )w^*w^* ( ε^(s) (H_f^(q))^-1SH ) ) ε^(s)H ε^(s) ] ⪯ 2‖∗‖2[(q)(q)⊤]+4(ϵ¯s+ϵ¯s′)‖∗‖2⊤. 2\|w^*\|_H^2E [S^(q)HS^(q) ]+4( ε_s+ ε_s )\|w^*\|_H^2SHS . Here we use two key inequalities: firstly, tr((⊤(f(q))−1)∗∗⊤(⊤(f(q))−1)⊤) (H (S (H_f^(q))^-1SH )w^*w^* (S (H_f^(q))^-1SH ) ) = = ∗⊤⊤(f(q))−1⊤(f(q))−1∗ w^* HS (H_f^(q))^-1SHS (H_f^(q))^-1SHw^* ≤ ≤ ∗⊤⊤(f(q))−1∗ w^* HS (H_f^(q))^-1SHw^* ≤ ≤ ‖∗‖2‖1/2⊤(f(q))−11/2‖ \|w^*\|_H^2\|H^1/2S (H_f^(q))^-1SH^1/2\| ≤ ≤ ‖∗‖2, \|w^*\|_H^2, and secondly, tr((−⊤(f(q))−1)∗∗⊤(−⊤(f(q))−1)⊤) (H (I-S (H_f^(q))^-1SH )w^*w^* (I-S (H_f^(q))^-1SH ) ) ≤ ≤ 2‖∗‖2+2tr((⊤(f(q))−1)∗∗⊤(⊤(f(q))−1)⊤) 2\|w^*\|_H^2+2tr (H (S (H_f^(q))^-1SH )w^*w^* (S (H_f^(q))^-1SH ) ) ≤ ≤ 4‖∗‖2. 4\|w^*\|_H^2. Overall, together with (130)-(144) and [(q)(q)]⪯(1+ϵ¯s)⊤E [S^(q)HS^(q) ] (1+ ε_s)SHS , it holds [ξ2~(q)(~(q))⊤] [ξ^2 x^(q)( x^(q)) ] (145) ≾ (1+α0‖∗‖2)(1+ϵ¯s+ϵ¯s′)[(1+ϵ¯d)(1+ϵ¯f)(ϵ¯lCy+σ¯02)+(1+ϵ¯d+ϵ¯d′)(1+ϵ¯f+ϵ¯f′)]⊤, (1+ _0\|w^*\|_H^2)(1+ ε_s+ ε_s ) [(1+ ε_d)(1+ ε_f)( ε_lC_y+ σ_0^2)+(1+ ε_d+ ε_d )(1+ ε_f+ ε_f ) ]SHS , together with f(q)⪰(1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s)⊤H_f^(q) (1+ ε_f)(1+ ε_d)(1+ ε_s)SHS , we have [ξ2~(q)(~(q))⊤]⪯σ¯M2f(q),E [ξ^2 x^(q)( x^(q)) ] σ_M^2H_f^(q), (146) where σ¯M2≲(1+α0‖∗‖2)(1+ϵ¯s+ϵ¯s′)[(1+ϵ¯d)(1+ϵ¯f)(ϵ¯lCy+σ¯02)+(1+ϵ¯d+ϵ¯d′)(1+ϵ¯f+ϵ¯f′)](1+ϵ¯f)(1+ϵ¯d)(1+ϵ¯s) σ_M^2 (1+ _0\|w^*\|_H^2)(1+ ε_s+ ε_s ) [(1+ ε_d)(1+ ε_f)( ε_lC_y+ σ_0^2)+(1+ ε_d+ ε_d )(1+ ε_f+ ε_f ) ](1+ ε_f)(1+ ε_d)(1+ ε_s). ∎