Paper deep dive
When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits
Tianhao Guo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Layer-wise post-training quantization methods let each layer compensate for the quantization error inherited from earlier layers. We ask under what assumptions such compensation guarantees a network output error no larger than without it. For linear networks of any depth, with the required calibration Gram matrices invertible, if the rounding error of each layer has zero conditional mean and a deterministic second-moment map common to both schemes, as under subtractive dither on an unbounded grid, then compensation with strengths in $[0,1]$ fixed in advance gives a mean-square output error no larger than no compensation, each scheme being followed along its own trajectory. The exact gain separates error removed directly from reduced excitation of later rounding noise. Outside this setting the comparison can fail. With rectified linear activations it fails for two independent reasons: noise changes activation states, and, even when no state changes, the correction moves error from samples on which a unit is inactive to samples on which it is active. With stable activations, commutators between the calibration projection and the activation patterns bound the excess. With a bounded codebook, even unique exact layer-wise minimizers with no overload can give a larger output error. For fixed rounding errors it already fails for two scalar linear layers, refuting a published universal guarantee. An exact residual identity gives the necessary and sufficient condition on a common layer input. All these comparisons hold the calibration input fixed. On two 8B models at 4 bits, fixed half-strength compensation beat uncompensated rounding in every model, text and draw tested, yet no rule for choosing the strength beat a layer-wise rule consistently, and lowering the full-network loss on the data used to choose did not reliably lower it on new text.
Tags
Links
- Source: https://arxiv.org/abs/2610.16020v1
- Canonical: https://arxiv.org/abs/2610.16020v1
Trouble viewing inline? Open PDF directly →
Full Text
138,035 characters extracted from source content.
Expand or collapse full text
When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits Tianhao Guo †thanks: Tianhao Guo is with the College of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China (e-mail: tianhao_guo@sxu.edu.cn). ORCID: 0000-0002-1317-868X. Code: https://github.com/tellsgo/quantization-compensation. 12 October 2026 Abstract Layer-wise post-training quantization methods let each layer compensate for the quantization error inherited from earlier layers. We ask under what assumptions such compensation guarantees a network output error no larger than without it. For linear networks of any depth, with the required calibration Gram matrices invertible, if the rounding error of each layer has zero conditional mean and a deterministic second-moment map common to both schemes, as under subtractive dither on an unbounded grid, then compensation with strengths in [0,1][0,1] fixed in advance gives a mean-square output error no larger than no compensation, each scheme being followed along its own trajectory. The exact gain separates error removed directly from reduced excitation of later rounding noise. Outside this setting the comparison can fail. With rectified linear activations it fails for two independent reasons: noise changes activation states, and, even when no state changes, the correction moves error from samples on which a unit is inactive to samples on which it is active. With stable activations, commutators between the calibration projection and the activation patterns bound the excess. With a bounded codebook, even unique exact layer-wise minimizers with no overload can give a larger output error. For fixed rounding errors it already fails for two scalar linear layers, refuting a published universal guarantee. An exact residual identity gives the necessary and sufficient condition on a common layer input. All these comparisons hold the calibration input fixed. On two 8B models at 4 bits, fixed half-strength compensation beat uncompensated rounding in every model, text and draw tested, yet no rule for choosing the strength beat a layer-wise rule consistently, and lowering the full-network loss on the data used to choose did not reliably lower it on new text. 1 Introduction Layer-wise post-training quantization processes the linear maps of a network one at a time, choosing for each a quantized weight matrix that reproduces the layer output on calibration data [2, 3, 4, 5]. Because every layer is treated separately, the error made in early layers reaches later layers through their inputs. Several recent methods change the layer objective so that a layer matches the output of the original network on the original input, while being fed the quantized input; the layer then compensates for the inherited error. The same objective appears under the names quantization error propagation [1], asymmetric calibration [6] and activation drift correction [17], and in [7, 16]. These methods share the motivation of reducing errors inherited from earlier quantized layers. This raises a stronger question: under what assumptions does layer-wise compensation guarantee that the network output error is no larger than without compensation? The answer is not obvious. Each layer solves a least-squares problem, and the solution is closer to the target than the uncorrected one before rounding. After rounding this need not be so, and a comparison on a common input says nothing yet about two schemes that, from the second layer on, see different inputs. That is the question of this paper. The comparison requires us to follow both the inherited error and the activation that multiplies the next rounding error. Reducing the first is not enough; the second must be controlled along the two trajectories as well. Throughout, the calibration input is fixed and the randomness is that of rounding; whether an improvement on calibration data carries over to other inputs is a separate question, on which Section 9 reports experiments. Related analyses exist. SPFQ [14] gives high-probability full-network error bounds for a stochastic path-following quantizer. The analysis of OPTQ and Qronos [8] gives layer-wise bounds for both algorithms and compares their error decompositions and upper bounds. These results do not establish an ordering of the realized total errors of a compensated and an uncompensated scheme. We establish a network-level comparison under explicit noise assumptions, give the gain exactly, and determine where the comparison fails. The fixed-error guarantee of [1] is one such failure. Its mechanism is simple: the inherited error and the new rounding error can cancel, and removing the first leaves the second without its counterpart. Whether compensation helps is therefore a question about the correlation between the two errors, and the positive results below are obtained by controlling it. We use the formulation of [1] to study a family of compensated layer objectives parameterized by the strength. The related methods motivate the analysis; their quantizers, solvers and regularization choices require separate treatment. QEP denotes the compensated scheme and BASE the uncompensated one in the model studied here. Contributions. 1. A comparison along two trajectories, and the gain it yields (Section 4). For linear networks of any depth, with rectangular layers allowed, Theorem 8 gives mean-square output error no larger with compensation than without it, for strengths in [0,1][0,1] fixed in advance, under the required Gram condition, zero conditional mean and a deterministic second-moment map common to both schemes. The proof carries two moments through the layers, that of the error and that of the quantized activation; the second governs how much the next rounding error is amplified. Proposition 9 separates the gain into error removed directly and reduced excitation of later noise. These compare compensation with no compensation. Ordering two non-zero strength schedules requires additional row-space conditions (Corollary 10); the closed form of Example 11 gives a ratio of expected errors in the specified random-teacher model, not a prediction for every fixed network. 2. Two obstructions for ReLU networks, and their geometric cost (Section 5). The comparison fails under the same dither when noise changes activation states (Example 12), and also when no state changes (Example 14). In the latter case the correction mixes calibration samples across activation patterns. Proposition 13 characterizes when this mixing is harmless for all inherited errors. Under stable activations, Theorem 15 bounds the excess by a term controlled by commutators of the calibration projection with the sample groups; Corollary 16 gives its small-error order. 3. The local condition and the limits of the network comparison (Sections 3, 6 and 7). An exact residual identity gives the necessary and sufficient condition for compensation to help on a common input. Its expectation covers subtractive dither with fixed inter-column feedback, without giving a network guarantee for GPTQ or QuIP. Fixed perturbations can instead cancel inherited error, invalidating the published universal guarantee and strength monotonicity. For an actual four-level codebook, an open family has a larger output error despite unique exact layer-wise minimizers and interior targets, with no overload (Theorem 22). Synthetic computations illustrate the positive results, and a scoped Lean formalization provides a complementary check (Sections 8 and 10). Experiments on two small and two 8B models address the question the theorems leave open, whether gains on calibration data carry over to new inputs (Section 9). The paper goes from one layer to the network: the per-layer identity and its expectation (Section 3), linear networks (Section 4), ReLU networks (Section 5), then fixed rounding errors and the published guarantee (Section 6) and actual quantizers (Section 7). Scope. The tools are standard: orthogonal projections, dithered rounding and order relations between second-moment matrices. The contribution is the comparison they yield and its limits. We propose no new quantizer and do not dispute reported empirical gains. We have not established stable activations in trained networks. The experiments of Section 9 do not satisfy our hypotheses and are not tests of them. The projection contraction, continuous optimum and each scheme’s own bound in [1] remain valid; Section 6 isolates the comparisons that fail. 2 Setting Consider a feedforward network, with the notation of [1, Appendix B.3], X1=X,Xl+1=σl(WlXl),l=1,…,L,X_1=X, X_l+1= _l(W_lX_l), l=1,…,L, where each σl _l is γl _l-Lipschitz in the Frobenius norm and σl(0)=0 _l(0)=0. A quantized forward pass uses weights W^l W_l: X^1=X X_1=X, X^l+1=σl(W^lX^l) X_l+1= _l( W_l X_l). Write δl=Xl−X^l _l=X_l- X_l and H^l=X^lX^l⊤ H_l= X_l X_l . We require H^l H_l to be invertible only at layers of positive strength: a layer with αl=0 _l=0 is not corrected, its target is WlW_l, and no Gram matrix is inverted there. The weights may be rectangular; invertibility of quantized weights is an additional assumption of the published fixed-error guarantee examined in Section 6, not of the positive results. The matrices ElE_l represent rounding errors. Sections 4 and 5 impose conditional moment assumptions on random errors; Section 6 instead fixes the same arbitrary matrices in both schemes. The uncorrected scheme uses W^lBASE=Wl+El W_l^BASE=W_l+E_l. QEP uses Wl∗(αl)=Wl+αlWlδlX^l⊤H^l−1,W^lQEP=Wl∗(αl)+El,αl∈[0,1].W_l^*( _l)=W_l+ _lW_l _l X_l H_l^-1, W_l^QEP=W_l^*( _l)+E_l, _l∈[0,1]. For M∈BASE,QEPM∈\BASE,QEP\ define the residuals and the bound RlM=WlXl−W^lMX^lM,UM=∑k=1L(∏s=k+1Lγs∥Ws∥2)γk∥RkM∥F.R_l^M=W_lX_l- W_l^M X_l^M, U^M= _k=1^L ( _s=k+1^L _s W_s _2 ) _k R_k^M _F. Summary of the results. Several different comparisons arise, and their answers differ according to whether they are read exactly, to first order in a fixed shared perturbation, or in mean square for errors with zero conditional mean and a common second-moment map, of which subtractive dither is the main instance. Table 1 records what this paper establishes; “open” means not settled here. Statement Exact, shared E First order, shared E Mean square, zero conditional mean UQEP≤UBASEU^QEP≤ U^BASE fails (Thm. 17) fails (Thm. 17) fails for some error laws allowed by Thm. 8 (Sec. 4); open for uniform dither Output error of QEP ≤ that of BASE fails (Thm. 17); for deterministic codebooks see Thm. 22 fails (Thm. 17) linear networks: holds (Thm. 8). ReLU: fails in general (Ex. 12, 14); holds in the commuting case of Thm. 15, otherwise up to the excess bounded there ∥δL+1QEP∥F≤UBASE _L+1^QEP _F≤ U^BASE fails (Ex. 27) holds, in the repaired form of App. B holds for linear networks (Thm. 8) and in the commuting case of Thm. 15; otherwise open Output error non-increasing in α fails (Thm. 19) fails (Thm. 19) holds for two linear scalar layers (Prop. 7); fails between non-zero schedules for sign-valued errors (Sec. 4) Layer residual on a common input, QEP ≤ BASE for α>0α>0, iff 2c≤(2−α)d2c≤(2-α)d (Cor. 2) same condition holds for fixed-step dither on an unbounded grid (Thm. 4); can fail under overload (Sec. 3.3) Table 1: What holds and what fails. The first four rows compare the two schemes on their own trajectories; the last row is a per-layer comparison on a common input. In the last column the bounds are compared through (UM)2E(U^M)^2, and entries established only for sign-valued errors say so. The network comparisons require the common deterministic second-moment map as well as zero conditional mean, and the additional hypotheses of the cited results. 3 One layer: when the correction helps We begin with a single layer on a common input. An exact identity decides whether the correction helps for a given rounding error; taking expectations gives the conditions under which it helps on average, and subtractive dither supplies them. The network-level results of Sections 4 and 5 build on this section. 3.1 An exact identity Fix one layer and one quantized input X X, and drop the layer index. Write A=Wδ,B=EX^,P=X^⊤H^−1X^,Rα=A(I−αP)−B.A=Wδ, B=E X, P= X H^-1 X, R_α=A(I-α P)-B. Thus R0R_0 is the BASE residual on this input and RαR_α the QEP residual with strength α. Let d=∥AP∥F2,c=⟨AP,B⟩F.d= AP _F^2, c= AP,B _F. Proposition 1 (Residual identity). Let A,BA,B be real matrices of the same shape and P a symmetric idempotent matrix. For all real α,βα,β, ∥Rα∥F2−∥R0∥F2 R_α _F^2- R_0 _F^2 =−α(2−α)d+2αc, =-α(2-α)\,d+2α\,c, (1) ∥Rβ∥F2−∥Rα∥F2 R_β _F^2- R_α _F^2 =(β−α)[(α+β−2)d+2c]. =(β-α) [(α+β-2)\,d+2c ]. (2) Proof. Rα=R0−αAPR_α=R_0-α AP, so ∥Rα∥F2=∥R0∥F2−2α⟨R0,AP⟩F+α2d R_α _F^2= R_0 _F^2-2α R_0,AP _F+α^2d. Because P is a symmetric idempotent, ⟨A,AP⟩F=∥AP∥F2=d A,AP _F= AP _F^2=d, hence ⟨R0,AP⟩F=d−c R_0,AP _F=d-c. This gives (1); (2) follows by subtraction. ∎ Corollary 2. With the notation above: (i) For α>0α>0: ∥Rα∥F≤∥R0∥F R_α _F≤ R_0 _F if and only if 2c≤(2−α)d2c≤(2-α)d. (i) α↦∥Rα∥Fα R_α _F is non-increasing on [0,1][0,1] if and only if c≤0c≤ 0. (i) ∥Rα∥F≤∥R0∥F R_α _F≤ R_0 _F for all α∈[0,1]α∈[0,1] if and only if 2c≤d2c≤ d. (iv) If d>0d>0, the minimizer of ∥Rα∥F R_α _F over [0,1][0,1] is α⋆=clip(1−c/d, 0, 1)α = (1-c/d,\,0,\,1). Proof. (i) is (1) divided by α>0α>0. For (i), if c≤0c≤ 0 the bracket in (2) is non-positive whenever α+β≤2α+β≤ 2. Conversely, if c>0c>0 then AP≠0AP≠ 0, so d>0d>0; take h=min(1,c/d)h= (1,c/d), and then ∥R1∥F2−∥R1−h∥F2=h(2c−hd)>0 R_1 _F^2- R_1-h _F^2=h(2c-hd)>0. (i) follows from (i) since (2−α)d≥d(2-α)d≥ d on [0,1][0,1], with equality at α=1α=1. (iv): ∥Rα∥F2 R_α _F^2 is a convex quadratic in α with vertex 1−c/d1-c/d. ∎ When 2c≤d2c≤ d but c>0c>0, full correction is no worse than no correction, yet the residual first decreases and then increases as α→1α→ 1. “No worse at α=1α=1” and “monotone in α” are different properties. Geometric reading. At α=1α=1 the condition 2c≤d2c≤ d is equivalent to ∥B−AP∥F≥∥B∥F B-AP _F≥ B _F. For B=EX^B=E X we have BP=BBP=B, so B and APAP lie in the same subspace, and the component A(I−P)A(I-P) is common to both residuals. The correction is harmful exactly when the new perturbation term B is closer to the inherited term APAP than to the origin, that is, when the two would largely cancel. In the counterexample of Theorem 17 in Section 6, AP=−εAP=- and B=−ε−ε2B=- - ^2. Remark 3 (Scope of α⋆α ). The formula for α⋆α holds B fixed. In an actual quantizer B depends on α through the rounding of W∗(α)W^*(α) (Section 7), so α⋆α is not by itself a rule for choosing the propagation strength. 3.2 In expectation Theorem 4 (Per-layer mean-square statement). Let A,B,PA,B,P be random, with P almost surely a symmetric idempotent and A,BA,B square integrable. Put D=∥AP∥F2D=E AP _F^2 and C=⟨AP,B⟩FC=E AP,B _F. For every fixed α, ∥Rα∥F2−∥R0∥F2=−α(2−α)D+2αC.E R_α _F^2-E R_0 _F^2=-α(2-α)D+2α C. In particular, if [B∣A,P]=0E[B A,P]=0 then C=0C=0 and ∥Rα∥F2=∥R0∥F2−α(2−α)D≤∥R0∥F2E R_α _F^2=E R_0 _F^2-α(2-α)D R_0 _F^2 for α∈[0,2]α∈[0,2], with ∥Rα∥F2E R_α _F^2 non-increasing on [0,1][0,1]. Proof. Take expectations in (1). Under the conditional-mean assumption, C=⟨AP,[B∣A,P]⟩F=0C=E\, AP,E[B A,P] _F=0. ∎ Remark 5 (A zero marginal mean is not enough). Let all quantities be scalars, P=1P=1 and A=B=±1A=B=± 1 with equal probability. Then B=0EB=0, but R0=0R_0=0 and Rα=−αAR_α=-α A, so Rα2=α2>0=R02ER_α^2=α^2>0=ER_0^2 for every α≠0α≠ 0. Remark 6 (The perturbation may depend on α). If the perturbation is BαB_α and [Bα∣A,P]=0E[B_α A,P]=0 for each α, the same computation gives ∥Rα∥F2−∥R0∥F2=−α(2−α)D+∥Bα∥F2−∥B0∥F2.E R_α _F^2-E R_0 _F^2=-α(2-α)D+E B_α _F^2-E B_0 _F^2. Zero conditional mean alone therefore does not suffice. Improvement holds exactly when ∥Bα∥F2−∥B0∥F2≤α(2−α)DE B_α _F^2-E B_0 _F^2≤α(2-α)D; a perturbation energy that does not grow with the correction is a sufficient condition. 3.3 Subtractive dither supplies both conditions Scope. The statements in this subsection concern rounding to an unbounded uniform grid with a step size fixed in advance. They apply to a bounded codebook only when no target falls outside its range; when targets do fall outside, the per-layer statement can fail, as shown at the end of the subsection. Section 9 reports how often corrected weights leave the range of the original row on two small models; we have no estimate beyond those, and we do not attribute the unfavourable 2-bit cases reported in [1] to overload. Let QΔQ_ denote rounding to the nearest point of Δℤ . The subtractively dithered quantizer maps t to QΔ(t+u)−uQ_ (t+u)-u, where u is uniform on [−Δ/2,Δ/2)[- /2, /2), independent of t, and known to the decoder. Its error QΔ(t+u)−u−tQ_ (t+u)-u-t is uniform on [−Δ/2,Δ/2)[- /2, /2) and independent of t [19, 20]; the vector analogue for lattices is in [21]. This holds for an unbounded grid; a bounded codebook requires in addition that no overload occurs. Suppose each entry of Wl∗(αl)W_l^*( _l) is quantized in this way, with mutually independent fresh dithers that are independent of all earlier layers, and a step size that is fixed in advance and does not depend on the correction strength. Then ElE_l has independent entries, uniform with variance σ2=Δ2/12σ^2= ^2/12, and is independent of (Wlδl,X^l)(W_l _l, X_l), which are functions of earlier dither only. Hence [ElX^l∣Wlδl,X^l]=0E[E_l X_l W_l _l, X_l]=0, and ∥ElX^l∥F2=σ2nl∥X^l∥F2E E_l X_l _F^2=σ^2n_l\,E X_l _F^2 does not depend on αl _l, where nln_l is the number of rows of WlW_l. Both conditions of Theorem 4 and Remark 6 hold, and on a common input ∥RlQEP∥F2=∥R~lBASE∥F2−αl(2−αl)∥WlδlPl∥F2.E R_l^QEP _F^2=E R_l^BASE _F^2- _l(2- _l)\,E W_l _lP_l _F^2. This is a statement about one layer on a common input. It does not by itself give a network-level comparison, because the two schemes produce different inputs for later layers. Section 4 treats the network level; we first compute a two-layer example exactly. Proposition 7 (Two-layer linear network with dither). Let L=2L=2, all matrices 1×11× 1, identity activations, teacher weights W1,W2W_1,W_2, input x, and subtractively dithered quantization with error variance σ2σ^2 in both layers, with X^2≠0 X_2≠ 0 almost surely. With α1 _1 arbitrary and α2=α _2=α, the mean-square output error of QEP is x2σ2[(1−α)2W22+W12+σ2].x^2σ^2 [(1-α)^2W_2^2+W_1^2+σ^2 ]. It is non-increasing in α on [0,1][0,1], and strictly decreasing when xW2≠0xW_2≠ 0; α=0α=0 gives the uncorrected scheme, and α=1α=1 removes the contribution x2σ2W22x^2σ^2W_2^2 of the first-layer error. Proof. Let r1,r2r_1,r_2 be the quantization errors. They are independent, zero mean, with variance σ2σ^2, and r2r_2 is independent of r1r_1 whatever the second-layer target is. The first-layer correction vanishes, so X^2=(W1+r1)x X_2=(W_1+r_1)x. The second-layer target is W2(1−αr1/(W1+r1))W_2 (1-α r_1/(W_1+r_1) ), hence the output is W2W1x+(1−α)W2r1x+r2(W1+r1)xW_2W_1x+(1-α)W_2r_1x+r_2(W_1+r_1)x. The output error is (1−α)W2r1x+r2(W1+r1)x(1-α)W_2r_1x+r_2(W_1+r_1)x; its two terms are uncorrelated and [r22(W1+r1)2]=σ2(W12+σ2)E[r_2^2(W_1+r_1)^2]=σ^2(W_1^2+σ^2). ∎ In a Monte Carlo check with Δ=1 =1, two parameter settings, five values of α and 3×1053× 10^5 samples each, the estimates differ from this formula by at most 0.00170.0017. This is a sanity check, not a proof. Rounding with feedback. Methods such as GPTQ and QuIP round the entries of a weight row in sequence, adding to each target a fixed linear combination of the errors already made: zj=wj+∑i<j(wi−qi)Uijz_j=w_j+ _i<j(w_i-q_i)U_ij, with U strictly upper triangular and determined by X X [4, 5]. Suppose each qjq_j is obtained from zjz_j by subtractive dither on an unbounded grid, with fresh independent dither and fixed step sizes. Each innovation nj=qj−zjn_j=q_j-z_j is then uniform and independent of the past, including its own target zjz_j, and the row error e=q−we=q-w satisfies e(I+U)=ne(I+U)=n, that is, e=n(I+U)−1e=n(I+U)^-1. Hence [EX^∣Wδ,X^]=0E[E X Wδ, X]=0, and ∥EX^∥F2E E X _F^2 does not depend on α provided U, the column order and the step sizes are not re-estimated when α changes. The per-layer statement therefore extends to dithered rounding with fixed feedback; the entries of E are now correlated, which the statement does not require them not to be. It does not extend to ordinary stochastic rounding, whose error variance depends on the position of the target within its cell. Bounded codebooks. For a bounded uniform codebook a,a+Δ,…,b\a,a+ ,…,b\ with nearest-point saturation, the dithered reconstruction of a target t is distributed as clip(t,a,b)+V (t,a,b)+V with V uniform on [−Δ/2,Δ/2)[- /2, /2). Overload thus introduces a bias clip(t,a,b)−t (t,a,b)-t and leaves the variance unchanged, and the bias can break the per-layer statement. Take the codebook 0,1,2,3\0,1,2,3\ with Δ=1 =1 and Z=(10112),W=(4201),T=(4101),X=W−1TZ,Z= pmatrix1&0\\ 1& 12 pmatrix, W= pmatrix4&2\\ 0&1 pmatrix, T= pmatrix4&1\\ 0&1 pmatrix, X=W^-1TZ, so that X^=Z X=Z, P=IP=I and W∗(α)=W+α(T−W)W^*(α)=W+α(T-W). With entrywise dithered rounding, ∥WX−W^(α)Z∥F2=58−α2+54α2,E WX- W(α)Z _F^2= 58- α2+ 54α^2, which equals 58 58 at α=0α=0 and 118 118 at α=1α=1; the minimum is at α=15α= 15. The entry 44 is clipped to 33 at every α, and correcting the neighbouring entry removes a term that partly cancelled this bias through the correlated inputs. Overload of the uncorrected targets is not needed: with the same T, W=(2201)W= ( smallmatrix2&2\\ 0&1 smallmatrix ), Z=(1021/2)Z= ( smallmatrix1&0\\ 2&1/2 smallmatrix ) and X=W−1TZX=W^-1TZ, every entry of W lies in the codebook, only the corrected entry 2+2α2+2α overloads, for α>12α> 12, and the loss 78+14(1−α)2+max(0,2α−1)2 78+ 14(1-α)^2+ (0,2α-1)^2 falls to 1516 1516 at α=12α= 12 and then rises to 158 158 at α=1α=1, above its value 98 98 at α=0α=0. These examples use entrywise rounding with a non-diagonal input Gram matrix; they do not contradict Remark 21, which concerns exact minimization over the codebook. Cost. The reconstruction QΔ(t+u)−uQ_ (t+u)-u is not a codebook point. The decoder must regenerate the dither, for instance from a seed, and subtract it. There is also an accuracy cost. On an unbounded grid the dithered error has mean square Δ2/12 ^2/12 whatever the input. For inputs that vary smoothly on the scale of Δ , deterministic rounding has the same mean-square error; on a coarse grid, deterministic rounding of concentrated inputs can have a smaller one. We have not measured either cost. The diagnostic of Section 9 includes dithered rounding of single layers; we have not evaluated dithered propagation end to end on any trained model; Section 8 does so on small synthetic networks. 4 Linear networks The per-layer statements compare the two schemes on a common input. In a network the two schemes produce different inputs from the second layer on, and each must be followed along its own trajectory. For linear networks the mean-square comparison holds for the whole network. Theorem 8 (Linear networks). Let all activations be the identity and let the strengths αl∈[0,1] _l∈[0,1] be fixed in advance. Let ℱlF_l denote the history before layer l is quantized. Suppose that, for both schemes and every l, [El∣ℱl]=0,[ElTEl⊤∣ℱl]=Ψl(T)E[E_l _l]=0, [E_lTE_l _l]= _l(T) for every bounded ℱlF_l-measurable positive semidefinite T, where Ψl _l is a deterministic positive linear map, the same for both schemes. The network, the input X1X_1 and the depth are fixed. Suppose also that, for every l with αl>0 _l>0, the Gram matrix H^l H_l along the QEP trajectory is almost surely invertible. Write ClM=[δlM(δlM)⊤]C_l^M=E [ _l^M( _l^M) ] and GlM=[X^lM(X^lM)⊤]G_l^M=E [ X_l^M( X_l^M) ]. Then these moments are finite, and for every l, ClQEP⪯ClBASE,GlQEP⪯GlBASE,C_l^QEP C_l^BASE, G_l^QEP G_l^BASE, and in particular ∥δL+1QEP∥F2≤∥δL+1BASE∥F2E _L+1^QEP _F^2 _L+1^BASE _F^2. Entrywise subtractive dither on an unbounded grid, with independent entries and a step fixed in advance, satisfies the hypotheses with Ψl(T)=σl2tr(T)I _l(T)= _l^2\,tr(T)\,I; with entry-dependent steps, Ψl _l is the corresponding fixed weighted trace map. The weight matrices may be rectangular, subject to the rank restriction discussed after the proof. Proof. Fix a layer of the QEP trajectory and write Z=X^lZ= X_l, X=XlX=X_l, P=PlP=P_l, W=WlW=W_l, α=αlα= _l. If α=0α=0 the correction is skipped, T=ZT=Z, and the two identities below hold trivially. Otherwise, since ZP=ZZP=Z, the next pre-activation is W^Z=WT+ElZ WZ=WT+E_lZ with T=(1−α)Z+αXPT=(1-α)Z+α XP, and T−X=(Z−X)(I−αP)T-X=(Z-X)(I-α P). Two pointwise identities follow from P=P⊤=P2P=P =P^2: (Z−X)(Z−X)⊤−(T−X)(T−X)⊤ (Z-X)(Z-X) -(T-X)(T-X) =α(2−α)(Z−X)P(Z−X)⊤⪰0, =α(2-α)\,(Z-X)P(Z-X) 0, (3) (1−α)ZZ⊤+αXX⊤−TT⊤ (1-α)Z +α X -T =αX(I−P)X⊤+α(1−α)(Z−XP)(Z−XP)⊤⪰0. =α\,X(I-P)X +α(1-α)(Z-XP)(Z-XP) 0. (4) The second-moment relation extends from bounded T to every ℱlF_l-measurable positive semidefinite T, as an identity in [0,∞][0,∞], by truncating T on ∥T∥≤n\ T ≤ n\ and monotone convergence. Taking the trace in (4) gives ∥T∥F2≤(1−α)∥Z∥F2+α∥X∥F2 T _F^2≤(1-α) Z _F^2+α X _F^2, and with κl=∥Ψl∗(I)∥2 _l= _l^*(I) _2 we have [∥ElZ∥F2∣ℱl]=trΨl(ZZ⊤)≤κl∥Z∥F2E[ E_lZ _F^2 _l]=tr\, _l(Z )≤ _l Z _F^2. Hence ∥X^l+1∥F2≤2∥W∥22[(1−α)∥X^l∥F2+α∥Xl∥F2]+2κl∥X^l∥F2E X_l+1 _F^2≤ 2 W _2^2 [(1-α)E X_l _F^2+α X_l _F^2 ]+2 _lE X_l _F^2, so all second moments of the activations and of the errors are finite by induction, for both schemes. The corrected weight itself need not be square integrable, since it contains H^l−1 H_l^-1; the output involves only the projection P. The two terms WTWT and ElZE_lZ are now square integrable, so the conditional-mean assumption removes their cross terms, and Cl+1QEP⪯WClQEPW⊤+Ψl(GlQEP),Gl+1QEP⪯W[(1−α)GlQEP+αXlXl⊤]W⊤+Ψl(GlQEP),C_l+1^QEP W\,C_l^QEP\,W + _l(G_l^QEP), G_l+1^QEP W [(1-α)G_l^QEP+α\,X_lX_l ]W + _l(G_l^QEP), using (3) and (4). For BASE, X^l+1=(W+El)X^l X_l+1=(W+E_l) X_l gives the exact recursions Cl+1BASE=WClBASEW⊤+Ψl(GlBASE)C_l+1^BASE=WC_l^BASEW + _l(G_l^BASE) and Gl+1BASE=WGlBASEW⊤+Ψl(GlBASE)G_l+1^BASE=WG_l^BASEW + _l(G_l^BASE). BASE is unbiased, X^lBASE=XlE X_l^BASE=X_l, hence GlBASE=XlXl⊤+ClBASE⪰XlXl⊤G_l^BASE=X_lX_l +C_l^BASE X_lX_l . Both orderings hold with equality at l=1l=1. If they hold at l, then, Ψl _l being positive, Cl+1QEP⪯WClBASEW⊤+Ψl(GlBASE)=Cl+1BASEC_l+1^QEP WC_l^BASEW + _l(G_l^BASE)=C_l+1^BASE, and, using XlXl⊤⪯GlBASEX_lX_l G_l^BASE, Gl+1QEP⪯WGlBASEW⊤+Ψl(GlBASE)=Gl+1BASEG_l+1^QEP WG_l^BASEW + _l(G_l^BASE)=G_l+1^BASE. Taking traces gives the last claim. ∎ The induction carries the pair (Cl,Gl)(C_l,G_l); the uncentred moment is needed because the new perturbation enters multiplied by the input. No inclusion between the row spaces of XlX_l and X^l X_l is assumed: the term X(I−P)X⊤X(I-P)X in (4) is exactly what the missing inclusion costs, and it has the right sign. At a layer of positive strength, invertibility of H^l H_l requires X^l X_l to have full row rank; layers of zero strength are not restricted. A linear map cannot increase rank, so a layer whose output feeds a later corrected layer may be narrower than its predecessors but not wider than the rank of its input; the last layer is not restricted. If H^l−1 H_l^-1 is replaced by the Moore–Penrose pseudo-inverse, PlP_l is still the orthogonal projection onto the row space of X^l X_l, the proof is unchanged and no restriction on widths remains; that is a different algorithm from the one in [1]. The two inequalities in the proof discard non-negative terms. Keeping them gives the gain exactly. Proposition 9 (Exact gain). Assume the hypotheses of Theorem 8. Along the QEP trajectory put, for layers with αl>0 _l>0, Sl S_l =αl(2−αl)Wl[δlPlδl⊤]Wl⊤, = _l(2- _l)\,W_l\,E [ _lP_l _l ]W_l , Bl B_l =Wl[αlXl(I−Pl)Xl⊤+αl(1−αl)(X^l−XlPl)(X^l−XlPl)⊤]Wl⊤, =W_l\,E [ _l\,X_l(I-P_l)X_l + _l(1- _l)( X_l-X_lP_l)( X_l-X_lP_l) ]W_l , and Sl=Bl=0S_l=B_l=0 when αl=0 _l=0. Then Δl=ClBASE−ClQEP _l=C_l^BASE-C_l^QEP and Λl=GlBASE−GlQEP _l=G_l^BASE-G_l^QEP satisfy Δ1=Λ1=0 _1= _1=0 and Δl+1 _l+1 =WlΔlWl⊤+Ψl(Λl)+Sl, =W_l _lW_l + _l( _l)+S_l, (5) Λl+1 _l+1 =(1−αl)WlΛlWl⊤+Ψl(Λl)+αlWlClBASEWl⊤+Bl, =(1- _l)\,W_l _lW_l + _l( _l)+ _l\,W_lC_l^BASEW_l +B_l, (6) in which every term is positive semidefinite. In particular ∥δL+1BASE∥F2−∥δL+1QEP∥F2≥∑l:αl>0αl(2−αl)∥WL⋯WlδlQEPPl∥F2,E _L+1^BASE _F^2-E _L+1^QEP _F^2\ ≥\ _l:\, _l>0 _l(2- _l)\,E W_L·s W_l\, _l^QEPP_l _F^2, (7) where the sum runs over the layers of positive strength only, the projection PlP_l not being defined elsewhere; the comparison of Theorem 8 is strict as soon as one term on the right is non-zero. Proof. The relations (3) and (4) are identities. Keeping their right-hand sides in the proof of Theorem 8 turns the two inequalities for QEP into Cl+1QEP=WlClQEPWl⊤+Ψl(GlQEP)−SlC_l+1^QEP=W_lC_l^QEPW_l + _l(G_l^QEP)-S_l and Gl+1QEP=Wl[(1−αl)GlQEP+αlXlXl⊤]Wl⊤+Ψl(GlQEP)−BlG_l+1^QEP=W_l[(1- _l)G_l^QEP+ _lX_lX_l ]W_l + _l(G_l^QEP)-B_l; the expectations are finite because PlP_l is a projection, and PlP_l is not separated from δl _l. Subtracting from the exact recursions for BASE gives (5) at once, and (6) after using GlBASE=XlXl⊤+ClBASEG_l^BASE=X_lX_l +C_l^BASE, which yields GlBASE−(1−αl)GlQEP−αlXlXl⊤=(1−αl)Λl+αlClBASEG_l^BASE-(1- _l)G_l^QEP- _lX_lX_l =(1- _l) _l+ _lC_l^BASE. Positivity of Δl _l and Λl _l is Theorem 8. Dropping Ψl(Λl)⪰0 _l( _l) 0 in (5) and iterating gives ΔL+1⪰∑lWL⋯Wl+1SlWl+1⊤⋯WL⊤ _L+1 _lW_L·s W_l+1S_lW_l+1 ·s W_L , whose trace is the right-hand side of (7), the terms with αl=0 _l=0 being zero. ∎ The two recursions separate two effects. SlS_l is the error removed at layer l. The source in (6) is a reduction of the energy that excites later rounding errors; it reaches the output only through the maps Ψ of later layers, and is not seen by a per-layer comparison. Since the recursions are linear with positive coefficients, the gain is a sum of non-negative contributions of the sources SlS_l and αlWlClBASEWl⊤+Bl _lW_lC_l^BASEW_l +B_l, each weighted by a positive semidefinite matrix that depends only on the teacher, the maps Ψl _l and the strengths; the bound (7) keeps the first kind only. A gain at an intermediate layer need not reach the output: later weights and noise maps may ignore the directions in which it occurs. Corollary 10 (Schedules with a common row space). Assume the hypotheses of Theorem 8 and let a and b be two schedules fixed in advance with 0≤al≤bl≤10≤ a_l≤ b_l≤ 1. Suppose that along both trajectories XlPl=XlX_lP_l=X_l almost surely at every layer of positive strength; this is the case when X1X_1 has full row rank and all teacher and quantized weight matrices are square and invertible. Then Clb⪯ClaC_l^b C_l^a and Glb⪯GlaG_l^b G_l^a for every l. For each such schedule, Cl+1=(1−αl)2WlClWl⊤+Ψl(Cl)+Ψl(XlXl⊤).C_l+1=(1- _l)^2\,W_lC_lW_l + _l(C_l)+ _l(X_lX_l ). In any class of fixed schedules that satisfy these hypotheses with the same maps Ψl _l and that contains the full-strength schedule, full strength minimizes the mean-square output error. Proof. With XlPl=XlX_lP_l=X_l the target is T=(1−αl)X^l+αlXlT=(1- _l) X_l+ _lX_l, so δl+1=(1−αl)Wlδl−ElX^l _l+1=(1- _l)W_l _l-E_l X_l. By induction every schedule is unbiased, δl=0E _l=0, hence Gl=XlXl⊤+ClG_l=X_lX_l +C_l and the displayed recursion holds. If Clb⪯ClaC_l^b C_l^a, positivity of the maps and (1−bl)2≤(1−al)2(1-b_l)^2≤(1-a_l)^2 give Cl+1b⪯Cl+1aC_l+1^b C_l+1^a. ∎ Example 11 (A closed form for the gain). Fix a calibration input X1X_1 with d rows and full row rank, and a schedule. Let all layers be square of size d and let the teacher weights be independent random matrices, almost surely invertible, with [Wl⊤Wl]=γlIE[W_l W_l]= _lI for deterministic constants 0<γl<∞0< _l<∞; for instance Wl=I+VlW_l=I+V_l with the entries of VlV_l independent and uniform on [−al,al][-a_l,a_l], al<1/(2d)a_l<1/(2d), where γl=1+dal2/3 _l=1+da_l^2/3. Suppose that, conditionally on all teacher weights and on the past, ElE_l has zero mean and [ElMEl⊤]=vltr(M)IE[E_lME_l ]=v_l\,tr(M)\,I with deterministic constants 0≤vl<∞0≤ v_l<∞ common to the two schemes, and that the quantized weights are almost surely invertible. Put ql=∥δl+1∥F2/∥Xl+1∥F2q_l=E _l+1 _F^2/E X_l+1 _F^2, with expectations over the teacher as well, and νl=dvl/γl _l=dv_l/ _l. Taking traces and the expectation over the teacher in Corollary 10 gives ql=[(1−αl)2+νl]ql−1+νlq_l=[(1- _l)^2+ _l]\,q_l-1+ _l with q0=0q_0=0, hence qLBASE=∏l=1L(1+νl)−1,qLQEP=∑j=1L∏i=jLνiat full strength.q_L^BASE= _l=1^L(1+ _l)-1, q_L^QEP= _j=1^L _i=j^L _i full strength. For νl=ν>0 _l=ν>0 the ratio qLBASE/qLQEP=∑k<L(1+ν)k/∑k<Lνkq_L^BASE/q_L^QEP= _k<L(1+ν)^k / _k<Lν^k increases with the depth and tends to L as ν↓0ν 0 (for ν=0ν=0 both errors vanish): at fine resolution full compensation leaves the rounding noise of the last layer only, a gain of 10log10L10 _10L dB. This is an average over random teachers and a ratio of expectations, in a class where all row spaces coincide; it is not a statement about a fixed network or about bottlenecks. Three limits should be stated. First, Theorem 8 compares any fixed schedule with the uncorrected scheme. Outside the setting of Corollary 10, monotonicity between two non-zero schedules can fail; the condition of that corollary is empty for the zero schedule and so does not exclude this. Take X=I4X=I_4, W1=[I2∣0]W_1=[\,I_2 0\,], W2=(1,0)W_2=(1,0), W3=1W_3=1, with errors E1=[ 0∣diag(u1,u2)]E_1=[\,0 (u_1,u_2)\,], E2=(0,η)E_2=(0,η), E3=0E_3=0, where u1,u2=±1u_1,u_2=± 1 and η=±12η=± 12 are independent fair signs, and α3=1 _3=1. The mean-square output error is 32 32 without correction, 1726 1726 at α2=12 _2= 12 and 34 34 at α2=1 _2=1. Second, the theorem does not cover feedback coefficients recomputed along each trajectory, which change Ψl _l; strengths chosen after seeing the rounding, which break the fixed schedule and the conditional mean; or overload, which introduces a conditional bias. Third, it does not extend to the bounds UMU^M. For three scalar linear layers with unit weights and input, strengths (0,1,0)(0,1,0), and independent errors ±t± t, ±4± 4, ±43± 43 with fair signs and 0<t<6670<t< 667, one finds (UQEP)2−(UBASE)2=t(6−67t)/9>0E(U^QEP)^2-E(U^BASE)^2=t(6-67t)/9>0. These errors satisfy the hypotheses of the theorem but are not a uniform dither; for uniform dither the comparison of the bounds is not settled here. 5 ReLU networks: two obstructions and a conditional guarantee The comparison of Theorem 8 does not survive a nonlinearity. We give the two ways in which it fails and a condition under which it is recovered up to an explicit excess. Example 12 (The mean-square comparison fails for ReLU). Let L=2L=2, all quantities scalar, both activations ReLU, X=1X=1, W1=1W_1=1, W2=−14W_2=- 14, and let both layers use subtractive dither on ℤZ, with errors u,vu,v independent and uniform on [−12,12)[- 12, 12). The teacher output is 00. With z=1+u∈[12,32)z=1+u∈[ 12, 32), the uncorrected output is z(v−14)+z\,(v- 14)_+; at α2=1 _2=1 the QEP target is −1/(4z)-1/(4z) and the QEP output is (vz−14)+(vz- 14)_+. Then [(X^3BASE)2]=132304≈0.00564,[(X^3QEP)2]=172−log3192≈0.00817.E [( X_3^BASE)^2 ]= 132304≈ 0.00564, [( X_3^QEP)^2 ]= 172- 3192≈ 0.00817. No target is out of range and all Gram matrices are positive. Given the first layer, the correction restores the conditional mean of the pre-activation to its teacher value −14- 14 (without it the conditional mean is −z/4-z/4). The conditional variance, z2/12z^2/12, is the same with and without the correction; the unconditional variance of the pre-activation drops from 55576 55576 to 13144 13144. The teacher’s ReLU is inactive there, so what reaches the output is the upper tail of the noise. The corrected output is positive less often, with probability 12−log34<14 12- 34< 14, but its positive tail carries more energy. A zero conditional mean before the activation does not survive the activation. Example 12 has noise crossing the activation boundary. That is not the only obstruction. A unit with a fixed 0/1 activation pattern across the calibration samples reads its pre-activation error through a diagonal mask D, while the correction mixes samples through P. The next proposition says when the correction is compatible with the mask for every possible inherited error. Proposition 13 (Masked contraction). Let P be an orthogonal projection, D a diagonal matrix with entries in 0,1\0,1\, and 0<α≤10<α≤ 1. Then ∥a(I−αP)D∥2≤∥aD∥2 a(I-α P)D _2≤ aD _2 for every row vector a if and only if PD=DPPD=DP. Proof. If PD=DPPD=DP then a(I−αP)D=aD(I−αP)a(I-α P)D=aD(I-α P) and I−αPI-α P is a contraction. Conversely, for every a with aD=0aD=0 the inequality forces aPD=0aPD=0, that is, (I−D)PD=0(I-D)PD=0; transposing, DP(I−D)=0DP(I-D)=0, hence PD=DPD=DPPD=DPD=DP. ∎ The proof uses only that D is a symmetric idempotent. The condition characterizes contraction for all inherited errors; in a given network the inherited error is constrained by the teacher and the input, so commutation is not necessary for the mean-square comparison in that network. The same computation gives a weighted form: for any symmetric positive semidefinite M, the inequality r(I−αP)M(I−αP)r⊤≤rMr⊤r(I-α P)M(I-α P)r ≤ rMr holds for every row vector r if and only if PM=MPPM=MP; in an orthonormal basis adapted to P, with M=(M11M12M12⊤M22)M= ( smallmatrixM_11&M_12\\ M_12 &M_22 smallmatrix ), the difference M−(I−αP)M(I−αP)M-(I-α P)M(I-α P) has blocks α(2−α)M11α(2-α)M_11, αM12α M_12 and 00, and is positive semidefinite exactly when M12=0M_12=0. The activation mask is one instance of a metric on the sample space. The defect of commutation can be computed from the calibration data: ∥PD−DP∥F2=2∑i:di=1∑j:dj=0Pij2 PD-DP _F^2=2 _i:\,d_i=1 _j:\,d_j=0P_ij^2, the squared coupling, through the fit, between samples on which the unit is active and samples on which it is inactive. Here P acts on the index space of the calibration samples; the coupling comes from fitting one weight matrix to all samples, not from any mixing of inputs at inference. Theorem 15 below uses the spectral norm of the commutator, which this Frobenius norm bounds from above. Example 14 (Failure without any change of activation). Let L=2L=2 with both activations ReLU, X=(1−11−111),W1=I2,W2=(1−2−1−1),X= pmatrix1&-1&1\\ -1&1&1 pmatrix, W_1=I_2, W_2= pmatrix1&-2\\ -1&-1 pmatrix, and let every weight of both layers be quantized with subtractive dither of the same step, with errors independent and uniform on [−t,t][-t,t]; α2=1 _2=1. The teacher pre-activations of the second layer are bounded away from zero by 11. For 0<t≤1100<t≤ 110, in both schemes and for every realization of the errors, every unit has the teacher’s activation state on every sample, and all Gram matrices and quantized weights are invertible. Nevertheless ∥δ3QEP∥F2−∥δ3BASE∥F2≥t2(227−42t(1+t))>0for 0<t≤1600,E _3^QEP _F^2-E _3^BASE _F^2\ ≥\ t^2 ( 227-42\,t(1+t) )>0 0<t≤ 1600, and the difference equals 227t2+O(t3) 227t^2+O(t^3); the proof is in Appendix A. Only the first output unit on the first sample is active. On that entry, with hijh_ij the normalized first-layer errors, the error inherited from the first layer is t(h11−h12)t\,(h_11-h_12) to first order without correction and t(−2h12+4h21)/3t\,(-2h_12+4h_21)/3 with it; the second-layer error contributes the same term to both. The correction has moved error from samples where the unit is closed onto the sample where it is open. Here P does not commute with the activation masks. When the projections respect the activation patterns the linear argument goes through, and when they nearly do the comparison holds up to an explicit cost. Partition the calibration samples into cells, with JcJ_c the diagonal 0/1 projection onto cell c, such that within each cell all samples have the same teacher activation pattern at every layer. Let Dl,cD_l,c be the diagonal 0/1 pattern of cell c at the output of layer l, and put Fl,c=Dl,cWlF_l,c=D_l,cW_l and Ψl,c(T)=Dl,cΨl(T)Dl,c _l,c(T)=D_l,c _l(T)D_l,c. Along the QEP trajectory let kl,c=∥PlJc−JcPl∥2k_l,c= P_lJ_c-J_cP_l _2 if l≥2l≥ 2 and αl>0 _l>0, and kl,c=0k_l,c=0 otherwise. Define deterministic positive semidefinite matrices by K1,c=V1,c=0K_1,c=V_1,c=0 and Kl+1,c K_l+1,c =Fl,cKl,cFl,c⊤+Ψl,c(Vl,c)+αlFl,c[kl,cδlQEP(δlQEP)⊤]Fl,c⊤, =F_l,cK_l,cF_l,c + _l,c(V_l,c)+ _l\,F_l,c\,E [k_l,c\, _l^QEP( _l^QEP) ]F_l,c , (8) Vl+1,c V_l+1,c =(1−αl)Fl,cVl,cFl,c⊤+Ψl,c(Vl,c)+αl[kl,c]Fl,cXlXl⊤Fl,c⊤. =(1- _l)F_l,cV_l,cF_l,c + _l,c(V_l,c)+ _l\,E[k_l,c]\,F_l,cX_lX_l F_l,c . (9) Theorem 15 (ReLU networks with stable activations). Let all activations be ReLU and let the strengths be fixed in advance. Assume: (a) almost surely both quantized trajectories have the teacher’s activation pattern at every layer and on every sample; (b) the hypotheses of Theorem 8 on ElE_l and H^l H_l. Then, for every l and c, [δlQEPJc(δlQEP)⊤] [ _l^QEPJ_c( _l^QEP) ] ⪯[δlBASEJc(δlBASE)⊤]+Kl,c, [ _l^BASEJ_c( _l^BASE) ]+K_l,c, [X^lQEPJc(X^lQEP)⊤] [ X_l^QEPJ_c( X_l^QEP) ] ⪯[X^lBASEJc(X^lBASE)⊤]+Vl,c, [ X_l^BASEJ_c( X_l^BASE) ]+V_l,c, and therefore ∥δL+1QEP∥F2≤∥δL+1BASE∥F2+∑ctrKL+1,c.E _L+1^QEP _F^2 _L+1^BASE _F^2+ _ctr\,K_L+1,c. In particular, if PlJc=JcPlP_lJ_c=J_cP_l almost surely for every c and every l≥2l≥ 2 with αl>0 _l>0, then ∥δL+1QEP∥F2≤∥δL+1BASE∥F2E _L+1^QEP _F^2 _L+1^BASE _F^2. Proof. Let P and J be orthogonal projections, 0≤α≤10≤α≤ 1 and k=∥PJ−JP∥2k= PJ-JP _2. Then (I−αP)J(I−αP)⪯J+αkI,PJP⪯J+kI.(I-α P)J(I-α P) J+α k\,I, PJP J+k\,I. (10) Indeed, write x=p+qx=p+q with p=Pxp=Px. Since PJ(I−P)=P(PJ−JP)(I−P)PJ(I-P)=P(PJ-JP)(I-P), we have |p⊤Jq|≤k∥p∥∥q∥≤12k∥x∥2 p Jq ≤ k p q ≤ 12k x ^2, and the two quadratic forms to be bounded equal −α(2−α)p⊤Jp−2αp⊤Jq-α(2-α)\,p Jp-2α\,p Jq and −q⊤Jq−2p⊤Jq-q Jq-2\,p Jq. Fix a layer and a cell, with the notation of the proof of Theorem 8 and J=JcJ=J_c, k=kl,ck=k_l,c. From T−X=(Z−X)(I−αP)T-X=(Z-X)(I-α P) and (10), (T−X)J(T−X)⊤⪯(Z−X)J(Z−X)⊤+αk(Z−X)(Z−X)⊤.(T-X)J(T-X) (Z-X)J(Z-X) +α k\,(Z-X)(Z-X) . Since (1−α)ZJZ⊤+αXPJPX⊤−TJT⊤=α(1−α)(Z−XP)J(Z−XP)⊤⪰0(1-α)ZJZ +α\,XPJPX -TJT =α(1-α)(Z-XP)J(Z-XP) 0, (10) also gives TJT⊤⪯(1−α)ZJZ⊤+αXJX⊤+αkXX⊤.TJT (1-α)ZJZ +α\,XJX +α k\,X . At the first layer and at layers with α=0α=0 we have T=ZT=Z and both hold with k=0k=0. No Gram matrix of ZJcZJ_c is inverted. By (a), on the samples of cell c the next activations are Fl,cT+Dl,cElZF_l,cT+D_l,cE_lZ for QEP, Fl,cZ+Dl,cElZF_l,cZ+D_l,cE_lZ for BASE and Fl,cXF_l,cX for the teacher. Write Cl,cMC_l,c^M and Gl,cMG_l,c^M for the two cell moments in the statement. Integrability is obtained as in Theorem 8, since ReLU does not increase norms. The conditional-mean assumption removes the cross terms, so Cl+1,cQEP C_l+1,c^QEP ⪯Fl,cCl,cQEPFl,c⊤+Ψl,c(Gl,cQEP)+αlFl,c[kl,cδlQEP(δlQEP)⊤]Fl,c⊤, F_l,cC_l,c^QEPF_l,c + _l,c(G_l,c^QEP)+ _lF_l,cE [k_l,c _l^QEP( _l^QEP) ]F_l,c , Gl+1,cQEP G_l+1,c^QEP ⪯Fl,c[(1−αl)Gl,cQEP+αlXlJcXl⊤]Fl,c⊤+Ψl,c(Gl,cQEP)+αl[kl,c]Fl,cXlXl⊤Fl,c⊤. F_l,c [(1- _l)G_l,c^QEP+ _lX_lJ_cX_l ]F_l,c + _l,c(G_l,c^QEP)+ _lE[k_l,c]F_l,cX_lX_l F_l,c . The commutator norm kl,ck_l,c is random and correlated with δlQEP _l^QEP, so the expectation of their product is not factored. For BASE the recursions are exact, Cl+1,cBASE=Fl,cCl,cBASEFl,c⊤+Ψl,c(Gl,cBASE)C_l+1,c^BASE=F_l,cC_l,c^BASEF_l,c + _l,c(G_l,c^BASE) and Gl+1,cBASE=Fl,cGl,cBASEFl,c⊤+Ψl,c(Gl,cBASE)G_l+1,c^BASE=F_l,cG_l,c^BASEF_l,c + _l,c(G_l,c^BASE), and by (a) BASE is unbiased on the cell, so Gl,cBASE⪰XlJcXl⊤G_l,c^BASE X_lJ_cX_l . Both claims hold with equality at l=1l=1. If they hold at l, inserting them in the two inequalities and using the positivity of Ψl,c _l,c gives them at l+1l+1, with (8) and (9). The squared Frobenius error is the sum over cells of trCL+1,ctr\,C_L+1,c. ∎ Condition (a) must hold almost surely, not on an event selected afterwards, since conditioning on the absence of activation changes does not in general preserve the zero conditional mean; it can be certified from the margins of the teacher pre-activations and a bound on the errors. Commutation says that the correction does not project samples of one activation pattern onto another. Stability and commutation are independent: with a single sample commutation is automatic, and Example 12 still fails because (a) does; Example 14 satisfies (a) and has non-commuting projections. Exact commutation is restrictive. Together with the invertibility of H^l H_l it is equivalent to ∑crank(X^lJc)=dl _crank( X_lJ_c)=d_l, where dld_l is the number of rows of X^l X_l: the activations of the different cells must span complementary subspaces, so the number of cells with non-zero activations at that layer is at most dld_l. It holds when all samples share one pattern, and it fails as soon as two cells have non-zero activations and one of them spans the whole space. It is not vacuous. With X=(12−1−1−1−112),W1=I2,W2=(111−1),W3=(1−311),X= pmatrix1&2&-1&-1\\ -1&-1&1&2 pmatrix, W_1=I_2, W_2= pmatrix1&1\\ 1&-1 pmatrix, W_3= pmatrix1&-3\\ 1&1 pmatrix, and cells 1,2\1,2\ and 3,4\3,4\, the two cells have different activation patterns at every layer, and all hypotheses hold with commuting projections for independent errors ±120± 120 on every weight. Exact enumeration of the 40964096 realizations gives mean-square output error 3287607/16000000≈0.20553287607/16000000≈ 0.2055 without correction and ≈0.0532≈ 0.0532 with α=(0,1,1)α=(0,1,1). The defect term makes the commuting case stable under perturbation. If Z0Z_0 has full row rank with smallest singular value at least s, P0P_0 is its row-space projection and ∥Z−Z0∥2≤η<s Z-Z_0 _2≤η<s, then the row-space projection P of Z satisfies ∥P−P0∥2≤2η/(s−η) P-P_0 _2≤ 2η/(s-η), and hence ∥PJ−JP∥2≤∥P0J−JP0∥2+2∥P−P0∥2 PJ-JP _2≤ P_0J-JP_0 _2+2 P-P_0 _2 for every projection J. A lower bound on the singular values is needed: the row-space projection of (1000η) ( smallmatrix1&0&0\\ 0&η&η smallmatrix ) has commutator norm 12 12 with diag(1,1,0)diag(1,1,0) for every η>0η>0. Corollary 16 (Small errors). Fix a ReLU network, the calibration input, the cells and the strengths. Suppose that all teacher pre-activations are non-zero, that XlX_l has full row rank whenever αl>0 _l>0, and that the teacher projections Pl0P_l^0 satisfy ∥Pl0Jc−JcPl0∥2≤ε0 P_l^0J_c-J_cP_l^0 _2≤ _0 for l≥2l≥ 2 with αl>0 _l>0. Let El=tNlE_l=tN_l with ∥Nl∥2 N_l _2 bounded, [Nl∣ℱl]=0E[N_l _l]=0 and [NlTNl⊤∣ℱl]=Ψl0(T)E[N_lTN_l _l]= _l^0(T) for a deterministic positive linear map common to both schemes. Then there are t0>0t_0>0 and a constant κ, depending on the network, the input and the bound on NlN_l, such that for 0<t<t00<t<t_0 ∥δL+1QEP∥F2≤∥δL+1BASE∥F2+κ(ε0t2+t3).E _L+1^QEP _F^2 _L+1^BASE _F^2+κ\,( _0t^2+t^3). Proof. For both schemes ∥δl+1∥F≤(∥Wl∥2+t∥Nl∥2)∥δl∥F+t∥Nl∥2∥Xl∥F _l+1 _F≤( W_l _2+t N_l _2) _l _F+t N_l _2 X_l _F on every realization, since the entrywise ReLU is 11-Lipschitz in the Frobenius norm and I−αPI-α P is a contraction; hence ∥δl∥F=O(t) _l _F=O(t) uniformly, and ∥δl∥2≤∥δl∥F _l _2≤ _l _F. For small t the margins of the teacher give (a), the quantized activations entering layers of positive strength have full row rank with singular values bounded below, and by the perturbation bound above kl,c≤ε0+O(t)k_l,c≤ _0+O(t). In (9) the source is O(ε0+t)O( _0+t), so Vl,c=O(ε0+t)V_l,c=O( _0+t). In (8) the source is O((ε0+t)t2)O(( _0+t)t^2) and Ψl,c(Vl,c)=t2Dl,cΨl0(Vl,c)Dl,c=O((ε0+t)t2) _l,c(V_l,c)=t^2\,D_l,c _l^0(V_l,c)D_l,c=O(( _0+t)t^2). The depth is fixed, so trKL+1,c=O(ε0t2+t3)tr\,K_L+1,c=O( _0t^2+t^3). ∎ When the teacher projections commute exactly, the excess is O(t3)O(t^3), an upper bound, while the errors themselves are in general of order t2t^2, although the projections along the quantized trajectory do not commute in general. An instance: X1X_1 with rows (2,2,1,1)(2,2,1,1), (1,1,2,2)(1,1,2,2), (0,0,η,−η)(0,0,η,-η), W1=[I2∣0]W_1=[\,I_2 0\,], W2=(1−1−13/2)W_2= ( smallmatrix1&-1\\ -1&3/2 smallmatrix ), cells 1,2\1,2\ and 3,4\3,4\, α=(0,1)α=(0,1), 0<η≤1100<η≤ 110 and independent errors uniform on [−t,t][-t,t] with t≤10−3t≤ 10^-3: the two cells have different patterns, (a) holds on the whole support, and 0<k2,c≤8ηt/(1−2ηt)0<k_2,c≤ 8η t/(1-2η t) outside a null set. The bound of Theorem 15 can be nearly attained and can be loose. Three saved instances with independent sign-valued additive errors have certificates combining exact rational output risks with rational intervals for the commutator norms. The stored binary parameters are embedded exactly as rationals. Every sign realization is checked for agreement with the teacher’s masks along both trajectories and for the required Gram inverses at corrected layers. The certificates use the stated recursion, with zero first-layer commutator contribution. They support a ratio of actual excess to ∑ctrKL+1,c _ctr\,K_L+1,c of approximately 0.9999940.999994, a three-layer case in which dropping Ψl,c(Vl,c) _l,c(V_l,c) makes the bound fail, and another ratio of approximately 0.740.74. The accompanying code repository’s numerics/saved_witnesses/ directory contains the configurations. The near-attainment case is checkG_climb_3_1.json; the omitted-term case is checkG_climb_variant_a_1.json; the third is checkG_search_seed5.json. Their rational risks and interval endpoints are indexed by the same filenames in SAVED_WITNESS_REPLAY.json; replay_fixed_witnesses.py replays only these saved inputs. These instances support near attainment and the need for the later-noise term, not optimality of the constant or a guarantee for actual quantizers. In Example 14 the bound exceeds the actual excess by more than an order of magnitude. Neither result removes condition (a), and we do not claim that trained networks satisfy it. 6 Fixed rounding errors and the published guarantee Sections 4 and 5 establish stochastic network comparisons under explicit assumptions. In contrast, the guarantee published with QEP is asserted for arbitrary fixed perturbation matrices shared by the two schemes. Under this deterministic convention, the comparison fails already for two scalar linear layers. Here the corrected scheme denotes the idealized correction in that guarantee; actual codebook minimizers are treated separately in Section 7. To our knowledge the only published statement of this kind is the guarantee given with Quantization Error Propagation (QEP) [1], which adds a closed-form correction to the weights before quantizing them, scaled by a propagation strength αl∈[0,1] _l∈[0,1]. Theorem 5.2 of [1] states that the output error of the propagated scheme is bounded by that of the uncorrected scheme; its formal version, Theorem B.5, states that a Lipschitz upper bound UQEPU^QEP on the output error satisfies UQEP≤UBASEU^QEP≤ U^BASE for any fixed perturbation matrices and every choice of strengths. Proposition 5.4 adds that the output error decreases monotonically as each αl _l approaches one. In the published discussion of the paper the authors describe Theorem 5.2 as ‘a rigorous guarantee that solving the layer-wise local optimization problem … reduces the final quantization error’11 1 Authors’ rebuttal of 30 July 2025 in the public discussion of [1], OpenReview forum a3l3K9khbL, under “Weakness and Question 2”; archived copy consulted on 4 October 2026. and use it to distinguish QEP from a concurrent method with the same local objective. Claims under examination. • [1, Theorem B.5]: for every choice of αl⊂[0,1]\ _l\⊂[0,1], UQEP≤UBASEU^QEP≤ U^BASE, via the per-layer inequality ∥RlQEP∥F≤∥RlBASE∥F R_l^QEP _F≤ R_l^BASE _F for all l. • [1, Theorem 5.2, informal]: ∥fθ(X)−fθ^QEP(X)∥F≤∥fθ(X)−fθ^BASE(X)∥F f_θ(X)-f_ θ_QEP(X) _F≤ f_θ(X)-f_ θ_BASE(X) _F for the quantizers defined by the two layer objectives. • [1, Proposition 5.4]: the output error of QEP decreases monotonically as each αl _l approaches 11. The appendix of [1] works to first order in El\E_l\ and writes the same symbol ElE_l for the perturbation in both schemes. The counterexample of Theorem 17 respects both conventions. Theorem 17 (Counterexample to Theorem B.5 of [1]). Let L=2L=2, all matrices 1×11× 1, σ1=σ2=id _1= _2=id, γ1=γ2=1 _1= _2=1, W1=W2=1W_1=W_2=1, X=1X=1, α1=α2=1 _1= _2=1, and E=(ε,−ε)E=( ,- ) with 0<ε<120< < 12. Then all hypotheses of Theorem B.5 hold, including invertibility of the Gram matrices and of all quantized weights, and R1BASE R_1^BASE =R1QEP=−ε, =R_1^QEP=- , X^2BASE X_2^BASE =X^2QEP=1+ε, = X_2^QEP=1+ , R2BASE R_2^BASE =ε2, = ^2, R2QEP R_2^QEP =ε+ε2, = + ^2, UBASE U^BASE =ε+ε2, = + ^2, UQEP U^QEP =2ε+ε2, =2 + ^2, |X3−X^3BASE| X_3- X_3^BASE =ε2, = ^2, |X3−X^3QEP| X_3- X_3^QEP =ε+ε2. = + ^2. Hence UQEP−UBASE=ε>0U^QEP-U^BASE= >0 and the QEP output error is strictly larger, for every 0<ε<120< < 12. Proof. In the first layer δ1=0 _1=0, so the correction vanishes and both schemes use W^1=1+ε W_1=1+ , giving X^2=1+ε X_2=1+ and R1=1−(1+ε)=−εR_1=1-(1+ )=- . In the second layer δ2=−ε _2=- and H^2=(1+ε)2>0 H_2=(1+ )^2>0. BASE. W^2=1−ε W_2=1- , output (1−ε)(1+ε)=1−ε2(1- )(1+ )=1- ^2, so R2=ε2R_2= ^2. QEP. W2∗=1+δ2X^2/X^22=1/(1+ε)W_2^*=1+ _2 X_2/ X_2^2=1/(1+ ), W^2=1/(1+ε)−ε W_2=1/(1+ )- , output 1−ε(1+ε)=1−ε−ε21- (1+ )=1- - ^2, so R2=ε+ε2R_2= + ^2. With γs=∥Ws∥2=1 _s= W_s _2=1, UM=|R1M|+|R2M|U^M= R_1^M + R_2^M . Since σ2 _2 is the identity, the output error equals |R2M| R_2^M . For 0<ε<120< < 12 the quantized weights 1+ε1+ , 1−ε1- and 1/(1+ε)−ε1/(1+ )- are positive. (The algebraic identities hold for every ε>0 >0; the QEP second-layer weight vanishes at ε=(5−1)/2 =( 5-1)/2 and the BASE one at ε=1 =1.) ∎ The gap is first order: maxk|Ek|=ε _k E_k = and (UQEP−UBASE)/ε=1(U^QEP-U^BASE)/ =1 throughout, while the two bounds and the two output errors differ from their first-order expressions by exactly ε2 ^2. The first-order convention of [1] therefore cannot absorb it. Both schemes see the same second-layer input, so the failure does not come from comparing different trajectories. Remark 18 (The sign matters). With E=(ε,ε)E=( , ) the same computation gives UBASE=3ε+ε2U^BASE=3 + ^2, UQEP=2ε+ε2U^QEP=2 + ^2 and output errors 2ε+ε22 + ^2 against ε+ε2 + ^2: QEP is better. The ordering depends on the relation between the inherited error and the new perturbation, which is what the identity of Section 3.1 makes precise. Theorem 19 (Failure of monotonicity in the shared-perturbation model). In the setting of Theorem 17, keep α1=1 _1=1 and let α2=α∈[0,1] _2=α∈[0,1]. Then the QEP output is 1−αε−ε21-α - ^2 and the output error is αε+ε2α + ^2. For 0≤α<β≤10≤α<β≤ 1 the error at β exceeds the error at α by (β−α)ε>0(β-α) >0. Proof. W2∗(α)=1−αε/(1+ε)W_2^*(α)=1-α /(1+ ), so W^2=1−αε/(1+ε)−ε W_2=1-α /(1+ )- and the output is (1+ε)W^2=1−αε−ε2(1+ ) W_2=1-α - ^2. ∎ This contradicts Proposition 5.4 of [1] read in the shared-perturbation model of its appendix. It does not contradict Corollary B.6 of [1]: that corollary bounds the residual by ∥Wlδl(I−αPl)∥F+∥El∥2∥X^l∥F W_l _l(I-α P_l) _F+ E_l _2 X_l _F, which here equals (2−α)ε+ε2(2-α) + ^2, is decreasing in α, and remains a valid upper bound on αε+ε2α + ^2 for α≤1α≤ 1. The bound is monotone; the error it bounds is not. Where the proof breaks. Let Pl=X^l⊤H^l−1X^lP_l= X_l H_l^-1 X_l, the orthogonal projection onto the row space of X^l X_l. The proof of Theorem B.5 correctly derives RlQEP=Wlδl(I−αlPl)−ElX^l,R~lBASE:=Wlδl−ElX^l,R_l^QEP=W_l _l(I- _lP_l)-E_l X_l, R_l^BASE:=W_l _l-E_l X_l, where R~lBASE R_l^BASE is the BASE residual on the same X^l X_l, and correctly notes ∥Wlδl(I−αlPl)∥F≤∥Wlδl∥F W_l _l(I- _lP_l) _F≤ W_l _l _F. It then asserts, “by the triangle inequality”, ∥Wlδl(I−αlPl)−ElX^l∥F≤∥R~lBASE∥F. W_l _l(I- _lP_l)-E_l X_l _F≤ R_l^BASE _F. This does not follow. Shrinking one term of a difference does not shrink the difference. In Theorem 17, W2δ2=−εW_2 _2=- and E2X^2=−ε−ε2E_2 X_2=- - ^2 nearly cancel in the BASE residual; QEP removes the first term and leaves the second intact. The proof contains a second unjustified step: having fixed X^l=X^lQEP X_l= X_l^QEP, it concludes by “taking X^l=X^lBASE X_l= X_l^BASE”, although the two trajectories differ in general. Theorem 17 isolates the first gap, since there the trajectories coincide up to the layer in question. The second gap causes failure on its own, as Example 26 shows. Three further facts about the shared-perturbation statements are established in Appendix B. The failure is not an artifact of P=IP=I or of scalar networks: it occurs with a proper projection, lifts to matrices, and occurs in a family that does not reduce to independent scalar channels (Example 25). The substitution of one trajectory for the other fails on its own, in a three-layer network where the first step causes no harm (Example 26). The composite inequality ∥δL+1QEP∥F≤UBASE _L+1^QEP _F≤ U^BASE, also asserted in Theorem B.5 of [1], fails for finite perturbations (Example 27); it holds once each factor ∥Ws∥2 W_s _2 in UBASEU^BASE is replaced by ∥Ws∥2+∥Es∥2 W_s _2+ E_s _2, and therefore to first order. 7 Actual nearest-codeword quantizers The shared-perturbation model fixes ElE_l independently of the target. Real quantizers do not behave this way. Remark 20 (The correction is piecewise constant). Let Q be a deterministic nearest-codeword map with a fixed codebook and scale, and W^(α)=Q(W+αK) W(α)=Q(W+α K) with K=WδX^⊤H^−1K=Wδ X H^-1. With the layer input and all other layers held fixed, on any interval of α on which no entry of W+αKW+α K crosses a decision boundary, W^(α) W(α) is constant, so the quantized layer and its loss are constant. Within such an interval the change in the continuous target is cancelled exactly by the change in rounding error: for α,α0α, _0 in the same interval, Bα=Bα0−(α−α0)KX^B_α=B_ _0-(α- _0)K X. The loss therefore changes only at boundary crossings, which are determined by the path of the target and by the codebook; the quadratic of Proposition 1, which holds B fixed, does not describe them. Remark 21 (No local harm at full strength with exact minimization). For any candidate Q and common input X X, ∥WX−QX^∥F2=∥Wδ(I−P)∥F2+∥(W∗(1)−Q)X^∥F2. WX-Q X _F^2= Wδ(I-P) _F^2+ (W^*(1)-Q) X _F^2. Hence an exact minimizer over the codebook of the QEP objective with α=1α=1 attains a layer residual no larger than any other codeword assignment on that input, including the one chosen by the uncorrected objective. This is the standard least-squares decomposition and is implicit in [1, Eq. (5)]. It shows that the shared-perturbation instance of Theorem 17 cannot arise from exact minimization of both objectives on a common input with a common codebook. It says nothing about the network output. The next result shows that output-level domination fails for exact minimizers. Theorem 22 (An actual finite-codebook family). Let the codebook be 0,1,2,3\0,1,2,3\ and consider a three-layer network with scalar weights a,b,c>0a,b,c>0, ReLU activations and two calibration samples X=(−1,1)X=(-1,1). Let each layer be quantized by exact minimization over the codebook, the uncorrected scheme using the objective of [1, Eq. (1)] with its own quantized inputs and QEP using [1, Eq. (3)] with α=1α=1 and no damping. Suppose 12<a<32,32<b<52,52<ab<3,12<c<32,32<abc<72. 12<a< 32, 32<b< 52, 52<ab<3, 12<c< 32, 32<abc< 72. Then every layer minimizer in both schemes is unique, the uncorrected scheme selects (1,2,1)(1,2,1) and QEP selects (1,3,1)(1,3,1), the continuous targets lie strictly inside (0,3)(0,3), and all input Gram matrices are positive. Writing t=abct=abc for the teacher output on the active sample, the output errors are |t−2| t-2 and |t−3| t-3 , so QEP is strictly worse⇔t<52,QEP is strictly better⇔t>52.QEP is strictly worse t< 52, is strictly better t> 52. Moreover, the better of the two attains the minimum output error over all 6464 codeword triples. Proof sketch. Since a>0a>0, the first sample is mapped to 00 by every layer and the second stays positive; the network acts on the second sample as the scalar chain x↦abcx abc\,x. The uncorrected scheme rounds a,b,ca,b,c to 1,2,11,2,1. QEP rounds a to 11; its second-layer target is ab∈(52,3)ab∈( 52,3), which rounds to 33; its third-layer target is abc/3∈(12,76)abc/3∈( 12, 76), which rounds to 11. The outputs are 22 and 33. Global optimality follows because every codeword triple outputs a non-negative integer and 22, respectively 33, is the nearest integer to t on the corresponding side of 52 52. ∎ We state plainly what this family is. It is a scalar chain: one sample is inactive throughout and the ReLU never acts on a sign change. The threshold t=52t= 52 is the midpoint of the two attainable outputs. The content of the theorem is that exact layer-wise minimization, with interior targets, unique minimizers and no clipping, produces a QEP output that is strictly worse on an open set of parameters. In the second layer QEP’s choice is locally better (the pre-activation 33 is closer to the teacher’s abab than 22 is), and the third layer’s grid is too coarse to exploit it. For instance, (a,b,c)=(75,2,1825)(a,b,c)=( 75,2, 1825) gives teacher output 2.0162.016, uncorrected error 0.0160.016 and QEP error 0.9840.984; changing c to 1110 1110 gives teacher output 3.083.08, and QEP is better. The same instance shows non-monotonicity in the propagation strength for an actual quantizer. At (75,2,1825)( 75,2, 1825), keeping α1=α3=1 _1= _3=1, with second-layer strength α2=12 _2= 12 the selected codewords are (1,2,1)(1,2,1) and the squared output error is 4/156254/15625; with α2=34 _2= 34 they are (1,3,1)(1,3,1) and it is 15129/1562515129/15625. The decision boundary is at α2=58 _2= 58. This instance is formalized, including the tie at the boundary, where the smaller codeword is selected. 8 Numerical illustrations The statements of Sections 4 and 5 are proved. The computations of this section show how large the quantities are in small synthetic networks. They are of three kinds. For Figure 1 and Table 2 the actual algorithm is run: the corrected target is computed with the Gram inverse, the weights are rounded once per run with entrywise subtractive dither on an unbounded grid, and the rounded weights are applied to all calibration samples. For the left panel of Figure 2 the rounding errors are sampled from the uniform law that this dither produces, and for the right panel sign-valued errors are enumerated; no quantizer is called in these two. Models, strengths, sample sizes and reported quantities were fixed before running. Scripts and raw outputs accompany the paper. How large the gain is. Figure 1 concerns the class of Example 11 with d=4d=4 and eight calibration samples, for depths up to 6464. The simulated normalized errors agree with the closed form for a fixed teacher and for random teachers: of the 168168 simulated values, 156156 lie within two standard errors of the prediction, and the largest deviation is 2.72.7 standard errors. Values at different depths come from the same runs and are not independent, so these counts are descriptive. Without compensation the error grows with the depth as (1+ν)L−1(1+ν)^L-1; with full compensation it stays close to ν, the rounding noise of the last layer; with α=12α= 12 it settles at an intermediate level. At fine resolution the gain follows 10log10L10 _10L over the depths shown. At the coarser resolutions and the largest depths shown it exceeds this value, since the uncorrected error then grows geometrically: at depth 6464 the simulated gains are 18.118.1, 18.218.2, 19.419.4 and 26.126.1 dB for ν=10−4,10−3,10−2ν=10^-4,10^-3,10^-2 and 5×10−25× 10^-2, against 10log1064≈18.110 _1064≈ 18.1. 124816326410−210^-210−110^-110010^0depth LLnormalized error qLq_Lno compensationα=12α= 12α=1α=112481632640010102020depth LLgain of full compensation (dB)10log10L10 _10Lν=10−4ν=10^-4ν=10−3ν=10^-3ν=10−2ν=10^-2ν=0.05ν=0.05 Figure 1: Gain as a function of depth in the class of Example 11, d=4d=4, with the actual algorithm and dithered rounding of the weights. Lines are the closed form; marks are simulations with two standard errors. Left: ν=10−2ν=10^-2; circles, fixed teacher Wl=IW_l=I (2×1042× 10^4 runs); crosses, random teachers (100100 teachers, 200200 runs each, errors clustered by teacher). Right: ratio of the two errors, random teachers, with standard errors paired by teacher. At depth one the two schemes coincide. Where the gain comes from. Table 2 concerns a linear network with two bottlenecks, where the row spaces of the original and quantized activations differ and Corollary 10 does not apply. The gain predicted by Proposition 9, from sources estimated on one batch of runs, agrees within two standard errors with the gain measured on an independent batch. In this network between 2%2\% and 3%3\% of the gain comes from the reduced excitation of later rounding noise, the part that a per-layer comparison does not see; the rest is error removed directly. The gain increases with the strength here, which Theorem 8 does not guarantee. α removed directly through later noise predicted gain measured gain error of QEP 0.250.25 0.08820.0882 0.00170.0017 0.08980.0898 0.0897±0.00220.0897± 0.0022 0.20930.2093 0.50.5 0.13560.1356 0.00300.0030 0.13860.1386 0.1392±0.00240.1392± 0.0024 0.15980.1598 0.750.75 0.15790.1579 0.00400.0040 0.16190.1619 0.1621±0.00240.1621± 0.0024 0.13690.1369 11 0.16400.1640 0.00460.0046 0.16860.1686 0.1686±0.00240.1686± 0.0024 0.13050.1305 Table 2: The two parts of the gain in Proposition 9, in a linear network 8→6→4→28→ 6→ 4→ 2 with dither of step 0.30.3 and strengths α2=α3=α _2= _3=α. The prediction is computed from the recursions (5)–(6) with sources estimated on 10410^4 runs; the measured gain comes from 10410^4 independent runs and is given with two standard errors. The mean-square error without compensation is 0.29910.2991. ReLU networks. The left panel of Figure 2 is Example 14. Over all runs no activation changed state, in either scheme and at either layer, and the excess of QEP over BASE divided by t2t^2 lies between 0.0710.071 and 0.0760.076, with a standard error of 0.0020.002, for t from 5×10−45× 10^-4 to 10−110^-1, against the leading coefficient 227≈0.074 227≈ 0.074. The harm is a second-order effect of constant relative size, present at error sizes well beyond the interval for which the lower bound is proved. The right panel is the bottleneck family described after Corollary 16, with sign-valued errors and exact enumeration. The projections do not commute with the cells, the largest commutator norm decreases as t, and the bound of Theorem 15 on the excess decreases as t3t^3; relative to the uncorrected error it is about 10−510^-5 at t=10−4t=10^-4 and 3×10−33× 10^-3 at t=3×10−2t=3× 10^-2. In this family the correction helps: the error of QEP is about 28%28\% of that of BASE at every t, so the bound is far from attained. No activation changed state. 10−310^-310−210^-210−110^-1005⋅10−25· 10^-20.10.1error size t(excess of QEP)/t2(excess of QEP)/t^22/272/27simulationproved lower bound10−410^-410−310^-310−210^-210−1210^-1210−610^-610010^010610^6error size tterror without compensationerror of QEPbound ∑ctrK _ctr\,K on the excesslargest commutator norm Figure 2: ReLU networks. Left: Example 14, rounding errors sampled from the uniform law of subtractive dither, 10610^6 independent runs per value of t, two standard errors; no activation changed state in any run. Right: the bottleneck family after Corollary 16 with η=110η= 110 and sign-valued errors, exact enumeration; here the correction helps, the bound on the excess decreases as t3t^3 and the commutator as t. These are small networks chosen to exhibit the statements; they do not test whether trained networks satisfy the hypotheses. The Monte Carlo values were computed twice, with two independent implementations and random generators, one of them free of any numerical library. On identical inputs the two agree to rounding error. Run independently they agree overall: of the 168168 values of Figure 1, 151151 differ by at most two combined standard errors and the largest difference is 3.063.06; these values are correlated and the comparison is descriptive. 9 Relation to experiments The comparisons of Sections 3–7 hold the calibration input fixed; the randomness is that of rounding. A quantized model is used on other inputs, and whether an improvement obtained on calibration data carries over to them is a different question, on which those results say nothing. This section reports what we observed on both questions: published observations, a layer-wise diagnostic on two small models, and experiments on two 8B models. None of the experiments satisfies the hypotheses of our theorems, and they are not tests of them. 9.1 Published observations Table 1 of [1] reports WikiText-2 perplexity for 6060 pairs (four base methods, five models, three bit widths) with and without QEP. In 5353 pairs QEP is better. In 77 pairs it is worse, all at 2 bits: RTN on Llama-2-7B and Llama-2-13B, GPTQ on Llama-2-13B and Llama-3-8B, and AWQ on Llama-2-7B, Llama-2-70B and Mistral-7B; in all seven the perplexity is above 10310^3 both with and without QEP. These experiments use αl=12 _l= 12 by default, except in the MLP layers of Llama-2-70B, where αl=0 _l=0. [17] uses the same propagated objective under the name activation drift correction, reports that it becomes “increasingly unstable in deeper layers” of larger models, and chooses a per-layer coefficient between drift-corrected and original calibration statistics by search; in most deeper layers of the Qwen3-8B configuration they report, the selected coefficient is close to the uncorrected end. In their reply of 30 July 2025 to reviewer pLsN in the same public discussion as above, under “Propagation Strength” and “Minor Question 1”, the authors of [1] report perplexities for GPTQ at 3 bits: on Llama-2-7B, 10.88110.881 without QEP, 7.8987.898 at α=12α= 12 and 8.1698.169 at α=1α=1; on Llama-2-70B, 4.8604.860, 4.1024.102 and 4.7714.771, where the middle entry uses α=0α=0 in the MLP down-projection matrices and 12 12 elsewhere. The authors attribute the loss at full strength to overfitting. Perplexity is not the quantity addressed by Proposition 5.4 of [1]. We have not run these configurations and do not attribute any of these observations to the mechanism of Section 3.1. 9.2 A layer-wise diagnostic on two small models If, for actual rounding errors, the ratio c/dc/d of Section 3.1 is typically far below the threshold of Corollary 2, the failure mode of Section 6 has little practical relevance. We measured this ratio on SmolLM2-135M [22] and Pythia-160M [23], each with two calibration texts, WikiText-2 and a public-domain novel. All linear maps inside the transformer blocks, 210210 and 4848 respectively, are first rounded to the nearest point of an unbounded uniform grid with step maxj|Wij|/7 _j W_ij /7 in row i. On this common quantized trajectory, and for each linear map separately, we compare the layer residual with and without the correction at α=12α= 12, using 40964096 calibration tokens; the corrected layer is not propagated further. The correction is computed from a truncated singular value decomposition of the quantized input, keeping singular values above 10−1010^-10 times the largest, without damping; all linear maps have full effective rank except one of the 4646 maps of Pythia-160M with d>0d>0, for each text. Here c is computed with the rounding error of the uncorrected layer. The labels in Table 3 use the following numerical conventions, with all energies divided by the number of tokens. A linear map enters the ratios when d>10−12max(∥Wδ∥F2,10−30)d>10^-12 ( Wδ _F^2,10^-30); three maps of SmolLM2-135M and two of Pythia-160M do not. With the rounding error held fixed, a map is counted as worse when 2c−(2−α)d>10−8max(d, 2|c|,∥B∥F2, 10−30)2c-(2-α)d>10^-8 (d,\,2 c ,\, B _F^2,\,10^-30), where B is the rounding error of the uncorrected layer applied to the quantized input. After re-rounding, on calibration or held-out tokens, the comparison uses the squared Frobenius residual per token, ∥WX−QX^∥F2/N WX-Q X _F^2/N for the quantized weight Q and N tokens: a map is counted as worse when this quantity with the correction exceeds its value without the correction by more than 10−810^-8 times the larger of the two values (or 10−3010^-30). On held-out tokens the correction fitted on the calibration tokens is reused without refitting. The protocol, the thresholds and the sample were fixed before any measurement. SmolLM2-135M Pythia-160M WikiText-2 novel WikiText-2 novel Linear maps with d>0d>0 207207 207207 4646 4646 Range of c/dc/d [−0.063, 0.070][-0.063,\,0.070] [−0.065, 0.065][-0.065,\,0.065] [−0.012, 0.025][-0.012,\,0.025] [−0.038, 0.023][-0.038,\,0.023] Worse, same rounding error 00 00 00 00 Worse, re-rounded, calibration 00 00 00 00 Worse, re-rounded, held-out 1919 00 1111 1212 Weights outside row range 0.19%0.19\% 0.21%0.21\% 9.1%9.1\% 9.3%9.3\% Table 3: Layer-wise diagnostic at α=12α= 12 with deterministic rounding. “Worse” counts linear maps whose squared residual is larger with the correction than without it, up to the numerical tolerances stated in the text. By Corollary 2, with the rounding error held fixed the correction is worse exactly when c/d>34c/d> 34. The held-out row applies the correction fitted on the calibration tokens to 20482048 other tokens. The first five rows come from one measurement on the unbounded grid. The last row comes from a second measurement, fixed in advance separately, on the same weights, corrections and quantized trajectory: it is the fraction, pooled over all weights of all 210210 and 4848 linear maps, of corrected weights W+αCW+α C, where C=WδX^⊤H^−1C=Wδ X H^-1 (with the truncated singular value inverse specified above), whose magnitude exceeds the largest magnitude of the original row. The ratio c/dc/d never exceeds 0.070.07 in absolute value and its median is below 10−310^-3 in absolute value: the cross term is small relative to the energy of the projected inherited error, and the condition of Corollary 2 is far from violated. This ratio is not a correlation coefficient, and a small value does not mean that the two errors are uncorrelated. Re-rounding the corrected weights does not change the counts on the calibration tokens. With subtractive dither and 1616 seeds, over the 80968096 pairs of a linear map and a seed, |c/d| c/d stays below 0.200.20; no pair is worse with the rounding error held fixed; two pairs, the same linear map of SmolLM2-135M on the novel, are worse after re-rounding on the calibration tokens; and on held-out tokens the number of linear maps that are worse ranges over the seeds from 1717 to 2020, from 00 to 22, and equals 1111 and 1212 in the four columns. On these models we did not observe a cancellation large enough to make the correction worse with the rounding error held fixed. What we did observe is of the other kind. On held-out tokens the correction fitted on the calibration tokens increases the residual of a sizeable fraction of the linear maps, up to 1212 of the 4848 maps of Pythia-160M. In addition, the correction moves weights out of the range of the original row, for 9%9\% of the weights of Pythia-160M at α=12α= 12 and 1717–18%18\% at α=1α=1, so a bounded codebook with a range set by the original weights would overload, which is the situation of Section 3.3. This diagnostic intervenes on one layer at a time on a common trajectory and does not run the two schemes to the output. 9.3 Choosing the strength on two 8B models Setting. We quantized Qwen3-8B and Meta-Llama-3-8B with our own implementation of a QEP-style correction on top of round-to-nearest; it is not a reproduction of QEP, GPTQ, AWQ or QuIP#. The seven linear maps of each transformer block are quantized to 4 bits with an asymmetric uniform codebook (integer codes 00–1515, a scale and zero point per row and group of 128128 input channels); embeddings, normalizations and the output head stay in BF16, and the dequantized weights are executed in BF16, so no speed is claimed. The maps are processed in dependency order in four groups (query–key–value, attention output, gate–up, down), maps that share an input being quantized together; each policy follows its own quantized trajectory, and the inputs of every group are taken from the actual modules. For a group with quantized input Z and teacher input X, the correction is fitted by ridge least squares with λ=tr(ZZ⊤)/dλ=tr(Z )/d, the mean eigenvalue of the Gram matrix, and every corrected candidate is re-quantized with recomputed scales and rounded deterministically. The ridge matters for reading the results: the matrix Pλ=Z⊤(ZZ⊤+λI)−1ZP_λ=Z (Z +λ I)^-1Z that replaces P is not a projection, and a direction in which the Gram matrix has eigenvalue s is compensated in proportion αs/(s+λ)α s/(s+λ). “Strength 11” below means external strength 11 with the ridge retained; on the fitting inputs and before re-quantization, in a direction with s=λs=λ it compensates half of the inherited error. For each model we drew, three times, twelve disjoint 512-token windows from WikiText-2 training text: four (20482048 tokens) to fit the corrections and eight (40964096 tokens) to choose strengths. Evaluation uses the first 1638416384 tokens of the WikiText-2 test text and of Pride and Prejudice (Project Gutenberg 1342, with the wrapper removed and the front matter kept), in non-overlapping 512-token windows with the context reset for each window; neither enters any decision. Each model uses its own tokenizer. The policies are: no correction (RTN); strength 12 12 in every group; strength 12 12 with the down projections uncorrected; and four rules that choose, for each group, a strength in 0,12,1\0, 12,1\ on the selection data, by the output error of the group (local), the error after the next block kept at RTN (block), the loss of the full network with the remaining blocks kept at RTN (network), or a first-order estimate of that loss from its gradient with respect to the block output, applied to the re-quantized candidates (gradient). The replication protocol, including all seven policies, was fixed after an exploratory round on validation text and before any of the reported evaluations. Model Text RTN α=12α= 12 local block network gradient Qwen3-8B WikiText-2 12.64312.643 12.32012.320 12.11912.119 12.199(1)12.199\ (1) 11.970(2)11.970\ (2) 11.870(3)11.870\ (3) novel 12.87512.875 12.61812.618 12.66512.665 12.552(3)12.552\ (3) 12.613(2)12.613\ (2) 12.582(1)12.582\ (1) Llama-3-8B WikiText-2 8.2648.264 8.0318.031 8.0188.018 8.017(2)8.017\ (2) 8.075(0)8.075\ (0) 8.028(0)8.028\ (0) novel 5.7865.786 5.6255.625 5.5865.586 5.627(1)5.627\ (1) 5.766(0)5.766\ (0) 5.689(1)5.689\ (1) Table 4: Perplexity at 4 bits, mean over three calibration draws. In parentheses, the number of draws, out of three, in which the rule has a lower perplexity than the local rule. RTN does not depend on the draw. What we observed. Every compensating policy has a lower mean perplexity than RTN in each model and text (Table 4), and fixed strength 12 12 is lower than RTN in each of the twelve combinations of model, text and draw; the network rule on the novel with Llama-3-8B is above RTN in one draw. No rule beats the local rule consistently. The gradient rule wins all three draws on Qwen3-8B with WikiText-2 and none on Llama-3-8B with WikiText-2. On Llama-3-8B the network rule loses to the local rule in all six comparisons, although, strength 00 being always a candidate, none of its 816816 decisions over both models and three draws increases the full-network loss on the selection data relative to leaving that group uncorrected. These two facts compare different things, the chosen candidate with no correction in the current state and one complete policy with another; the next paragraph compares the two policies directly. The gradient rule took 4.14.1–4.84.8 times the construction time of the local rule in our prototype. More selection data. To see whether the selection data were simply too few, we kept the 20482048 fitting tokens of two of the draws, enlarged the selection data in nested sets of 40964096, 1638416384 and 3276832768 tokens, added a separate 81928192-token diagnostic set from the same source that enters no decision, and evaluated on new windows (tokens 6553665536 to 8192081920) of both texts. These runs record the loss of both final policies on every data set, so the same pair can be compared with the same metric on different data. On Llama-3-8B with 40964096 selection tokens, averaged over the two draws, the network policy has a lower loss per token than the local policy on the selection data, by 0.02100.0210, and a higher one on the diagnostic data (+0.0084+0.0084), on WikiText-2 (+0.0069+0.0069) and on the novel (+0.075+0.075): the ordering of the two policies reverses when the data change. With more selection data the advantage on the selection data shrinks (−0.0116-0.0116 at 1638416384 tokens, −0.0087-0.0087 at 3276832768) and so does the deficit elsewhere, to +0.0025+0.0025 on WikiText-2 and +0.030+0.030 on the novel, behind in every draw on the novel. On Qwen3-8B the network rule is ahead at every size, most at 1638416384 tokens (−0.033-0.033 and −0.018-0.018) and less at 3276832768. Selection by the output error of the current block does not help consistently either. The candidates themselves were always fitted on the same 20482048 tokens, so these results do not separate the quality of the candidates from the accuracy of choosing among them. Where the overestimate lies. Measuring each policy against RTN separates the two rules. For a policy, take its gain over RTN on the selection data minus its gain over RTN on the diagnostic data; the difference between the two policies’ values is the change of their ordering reported above. With 40964096 selection tokens, averaged over the two draws, this overestimate is +0.022+0.022 on Llama-3-8B and +0.034+0.034 on Qwen3-8B for the network rule, and −0.007-0.007 and −0.006-0.006 for the local rule. A further run with the fitting data of the first draw, two new selection draws and 3232 source windows used by no policy gave the same ordering in all four cases, with 0.0290.029 to 0.0490.049 for the network rule and 0.0080.008 to 0.0220.022 for the local rule. The network rule chooses by the loss of the full network on the selection data, which is also the quantity reported on those data; the local rule chooses by a linear output error. A small standard error does not exclude the overestimate: on Llama-3-8B in the first draw the network rule is ahead of the local rule on the selection data by 0.0250.025, about six standard errors over the eight windows, and behind on the diagnostic data. The overestimate is comparable in the two models; the gains are not. On the selection data the network rule is ahead of the local rule by 0.0520.052 on Qwen3-8B and by 0.0210.021 on Llama-3-8B, and only the first survives. Fixed strength 12 12 makes no choice on the selection data and pays no such cost. Reusing the same eight windows for every decision is not the explanation. In the same run, giving each decision eight windows not used before in its path, with everything else fixed, did not reduce the overestimate of the network rule: summed over the decisions and measured on a fixed set of eight held-out source windows, it went from 0.0310.031–0.0430.043 with reused windows to 0.0700.070–0.0830.083 with fresh ones, and the mean final loss on held-out source windows did not improve. Fresh windows remove reuse across decisions, but not the choice, at each decision, of the candidate that scores best on the very windows on which its gain is then measured. This run used a protocol, predictions and a stopping rule fixed before inference; fresh windows also change the amount of text seen, so it does not isolate reuse at an equal data budget. Changing the suffix. Choosing group by group, each rule above scores a candidate with the later blocks still at RTN, and then changes those blocks. To see whether this explains the failures, we replayed 4848 recorded decisions (two draws per model; three blocks per draw, four groups each; the groups of a fourth, final block serve as a control) with the earlier state and the three candidates fixed, scoring them once with the RTN suffix and once with the corrected suffix reconstructed from the recorded decisions. In 1111 of the 4848 the preferred candidate on the selection data reverses by more than 10−410^-4 in loss; none of these 1111 reversals recurs, for the same pair and in the same direction, on the diagnostic data. Dependence on the suffix exists, but we found no evidence that it accounts for the losses on new text. The strength from the fixed-error quadratic. Remark 3 warns that the minimizer of a quadratic with the rounding error held fixed is not by itself a rule for an actual quantizer. We measured how far it goes at 2 bits (codes 00–33, same group structure), in 4848 slices (two draws per model, three blocks spread over the depth, four groups), each candidate re-quantized with recomputed scales and applied to a common RTN trajectory. With S the output of the ridge correction, the strength minimizing ∥R0−αS∥F2 R_0-α S _F^2 on the selection data, clipped to [0,1][0,1], ranks the re-quantized strengths with a mean rank correlation of 0.9770.977 to 0.9940.994 across models and draws, and on the diagnostic data it lowers the normalized residual of the group by 6.46.4 to 9.2%9.2\% relative to strength 12 12, on average within each model and draw (in 4747 of the 4848 slices). Seven of the 4848 values were above 11 before clipping, the largest 1.161.16; since the ridge has already shrunk the correction, these numbers say nothing about the best unshrunk strength. The fixed pair 12,1\ 12,1\, chosen between on the selection data with strength 00 retained as a fallback (as in the grid of ten non-zero strengths), already retains 98.298.2 to 99.6%99.6\% of the diagnostic gain of a ten-point grid. These are linear residuals, not perplexities. The scale is recomputed for each candidate, which places this setting outside Remark 20; a high rank correlation shows that the quadratic predicts the ordering, not that the loss is a smooth function of the strength. 9.4 What the experiments do and do not show On a fixed calibration input, Sections 4–7 give conditions under which compensation cannot hurt and mechanisms by which it can. In the experiments, compensation helped on average relative to no compensation, and on the small models the cancellation of Section 6 was not observed with the rounding error held fixed; we did not measure the cross term on the 8B models, so we neither confirm nor exclude it there. The held-out comparisons reveal a further limitation: gains measured on fitting or selection data need not persist on other inputs. A correction that lowers a layer residual on calibration tokens raises it on others, and a policy that is better than another on the selection data can be worse on new text. This is separate from the two increases on calibration tokens observed after re-rounding with dither in the small-model diagnostic, and from the fixed-input failures of Sections 6 and 7. We have not identified the cause of the held-out losses. The overestimate is larger for the rule that chooses by the reported quantity, and reusing the selection data across decisions does not explain it; the finite fitting data, the size and domain of the selection data, the ridge and the interaction between layers are not separated by these experiments. Calibrating quantized weights against the output of a whole block rather than a single layer is not new [28, 29]; our network rule goes further and scores candidates by the loss of the full network. What these experiments add is a measurement, in one setting, of how far such choices carry over to new text. The scope is narrow: two models, two or three draws, two texts that may appear in pretraining data, windows rather than independent documents, our own RTN-backed implementation and dequantized execution. 10 Formal verification The statements below are checked in Lean 4 (version 4.34.1) with Mathlib at commit d13f23b. The accompanying artifact contains 96 modules with 969 explicitly named theorem declarations, auxiliary lemmas included; every one of them depends only on the axioms propext, Classical.choice and Quot.sound. The number of declarations is not a count of results. The artifact also contains positive controls and ten deliberately false statements that are required to fail. The table lists the eleven modules for the statements of Sections 3.1, 6 and 7, with 153 declarations; the remaining modules are described after it. Module Content Decl. QepSharedE Theorem 17, the sign remark, arbitrarily small ε 15 QepSharedEExtended Exact parameter domain of Theorem 17: the quantized weights are non-zero for every ε>0 >0 except ε=1 =1 and ε2+ε=1 ^2+ =1 6 QepAlpha Theorem 19; validity and monotonicity of the bound of [1, Cor. B.6] 8 QepProjectionExample First part of Example 25 7 QepMatrixFamily Second part of Example 25 for D=εID= I and general inputs, and the instance X=[I2∣]X=[\,I_2 1\,] 25 QepCorrected Proposition 1, Corollary 2(i), finite weighted form of Theorem 4 10 QepCorollary Corollary 2(i)–(iv) 12 QepGram P=X^⊤H^−1X^P= X H^-1 X is a symmetric idempotent with X^P=X XP= X when X X has full row rank; Corollary 2(i) for the actual QEP residuals; the decomposition of Remark 21 13 QepDitherFinite Proposition 7 for independent finitely supported errors with zero mean and equal second moments 6 QepInterior Theorem 22 and its two numerical instances 43 QepInteriorAlpha The α2 _2 instance of Section 7, with its switch at 58 58 8 In QepSharedE the residuals and the bound U are computed from the recursion of Section 2, not assumed. QepInterior defines both encoders by an explicit four-way minimization and derives the selected codewords. The remaining modules were reviewed independently of their authors. They cover the following, each within the stated scope. • Theorem 8 for finitely supported error laws, with the actual Gram matrices, the expectation step, the joint induction on the two moments, and layers of zero strength. • Example 12 in full: the recursion with continuous dither, both integrals, the strict inequality and the probability of a positive output. • For the actual continuous dither on a fixed finite grid: the distribution of the saturating quantizer, the two bounded-codebook examples of Section 3.3, and per-layer energy and cross-term statements on a common input with an arbitrary history, under explicit integrability and rank assumptions. • Rounding with feedback, for the actual sequential process with continuous dither: with parameters fixed in advance, the relation between final errors and innovations, the covariance of the final error, the output energy on an unbounded grid and the overload term for a finite codebook; with measurable parameters depending on earlier randomness, the joint law of the standardized innovations only. • Examples 26 and 27 and the repaired composite bound; the non-commutative family of Example 25, with the spectral norm of D. • For Theorem 15, algebra only: the cell-wise identities; the two inequalities (10) and the two matrix bounds derived from them, under a bilinear bound on the commutator that is not identified with the spectral norm; and an order-propagation step with defect terms in which the per-layer inequalities are hypotheses. These are not linked to the actual Gram projections, the activations or the expectations. • Proposition 13; the schedule example and the example for the bounds in Section 4. Not formalized: Theorem 4 as an abstract conditional-expectation statement (the finite weighted version and the fixed-grid versions above are formalized); Theorem 8 for general error laws, where the integrability argument near singular Gram matrices is a paper proof; Proposition 9, Corollary 10 and Example 11, of which the first two were checked by exact enumeration on small networks and all three by the simulations of Section 8; the distributional property of subtractive dither on an unbounded grid for a general lattice, which we cite, and Proposition 7 for continuous dither; Theorem 15 as a statement about the actual stochastic network, Corollary 16, the perturbation bound for projections and the weighted form of Proposition 13; the expectation in Example 14, which is proved on paper in Appendix A; the energy statements for rounding with feedback when the parameters depend on earlier randomness; Remark 20; the consequence drawn in Remark 21 for an exact discrete minimizer; and the matrix family of Example 25 for general diagonal D. Verification consists of standard elaboration and kernel checking with printed axiom dependencies; no independent proof checker was run. 11 Limitations and open questions • The theorems compare the two schemes on a fixed calibration input. They say nothing about new inputs, where the largest losses we observed occur (Section 9); the overestimate we measured is larger for the rule that selects by full-network loss, but reuse of the selection data does not explain it and we have not established the cause. The cross term c/dc/d was measured only on two small models with single-layer interventions; how often 2c>(2−α)d2c>(2-α)d occurs in larger models, at lower bit widths or along each scheme’s own trajectory is not known to us. • The composite inequality ∥δL+1QEP∥F≤UBASE _L+1^QEP _F≤ U^BASE, also asserted in [1, Theorem B.5], fails as an exact statement (Example 27) and holds to first order, in the repaired form given in Appendix B. • At the network level, Theorem 8 covers linear networks and Example 12 shows that the mean-square comparison fails for ReLU networks. Theorem 15 bounds the cost of non-commuting projections for ReLU networks but requires stable activations, which we have not established for any trained network; a statement that tolerates a small probability of activation changes is open, and other activations are not treated. How (UQEP)2E(U^QEP)^2 and (UBASE)2E(U^BASE)^2 compare for linear networks under uniform dither is not settled in this paper. • We have not analyzed damping, although the experiments of Section 9 use it. With a ridge term, T−X=(Z−X)(I−αPλ)T-X=(Z-X)(I-α P_λ) still contracts the inherited error on the fitting inputs, before re-quantization, but the inequality for the activation moment in the proof of Theorem 8 does not carry over as stated, and a network comparison is not established. With a ridge term the matrix playing the role of P is not idempotent and Proposition 1 must be replaced by the general expansion ∥A−B−αS∥F2−∥A−B∥F2=α2∥S∥F2−2α⟨A−B,S⟩F A-B-α S _F^2- A-B _F^2=α^2 S _F^2-2α A-B,S _F for the correction direction S. • The counterexamples to the shared-perturbation theorem are small. Example 25 gives a matrix family and a non-commutative family; we have not looked for counterexamples with large or structured weights. 12 Related work RTN, GPTQ [2], AWQ [3] and QuIP [4] are the layer-wise methods to which QEP is applied in [1]; QuIP# [5] extends QuIP with lattice codebooks. GPTAQ [6] adopts an asymmetric calibration objective of the same form and is the method from which the authors of [1] distinguish their work by the guarantee examined here. Qronos [7], NestQuant [16] and WaterSIC [17] use the same propagated objective, and [15] studies the residual introduced by compensation itself. Error analyses that account for the quantization of earlier layers exist: SPFQ [14] gives full-network error bounds for a stochastic path-following quantizer, and [8] gives error bounds for OPTQ and Qronos and compares the two bounds. These are high-probability or worst-case bounds for specified algorithms; they do not assert that, realization by realization or for the whole network, correction is never worse than no correction, which is the statement examined here. For a single layer, the information-theoretic limits of quantizing for a matrix product are characterized in [18], with nested-lattice constructions, and under an i.i.d. Gaussian weight model at high rate [17] reports a rate gap of about 0.2550.255 bit to the information-theoretic limit. The independence of dithered quantization error from the input goes back to [19]; see [20, 21]. Subtractive dither has been applied to the compression of pretrained network weights [9], and stochastic rounding to low-precision training [10]; neither work compares corrected and uncorrected layer-wise quantization. Treating active and inactive ReLU samples differently in layer-wise reconstruction, with a cascade analysis, goes back to Net-Trim [11]. Quantizing a residual after predicting from an already quantized past is the structure of predictive coding and noise shaping, where the effect of noisy prediction and the limits on feedback gain are classical [12, 13]. Closer still are the classical analyses of finite-word-length effects. In roundoff-noise analysis of digital filters the output noise is computed through Gramians and minimized over realizations [24], and error feedback is designed to reduce it; the noise gain is then a quadratic function of the feedback, with a gain term and a cross term and an optimal strength [25]. The per-layer identity of Section 3.1 has this form and is elementary. The second-moment recursion of the uncorrected scheme is that of a linear system with multiplicative noise, and the propagation of orderings by positive maps is standard in that theory [26]. The theory of dither is surveyed in [27]. What is specific here is that the correction is a projection determined by the realized activations of the corrected scheme, so that it is neither a fixed feedback filter nor a fixed state transition, and that two trajectories are compared; the step to be proved is that this correction reduces both the error moment and the moment that excites later noise. That error feedback can amplify correlated errors is therefore not new; what we establish is a joint comparison of error and activation moments along two realized trajectories, its exact gain, and the limits imposed by nonlinear geometry and finite codebooks. 13 Conclusion Under the hypotheses of Theorem 8, compensating inherited quantization error with strengths fixed in advance does not increase the mean-square output error of a linear network. The proof compares the schemes along their own trajectories through both error and activation moments; the exact gain separates direct error removal from reduced excitation of later rounding noise. Stronger compensation is no worse under additional row-space conditions, with a closed-form depth gain in a specified model. Outside that setting we exhibit several ways in which the comparison fails. For ReLU networks, stable activations alone are insufficient; commutators between calibration projections and activation-pattern partitions bound the excess. For fixed perturbations, the per-layer comparison depends on the error cross term. With actual finite codebooks, even unique exact layer-wise minimizers without overload can yield a larger output error. Local improvement is therefore not a network guarantee. All of this holds the calibration input fixed; on two 8B models, the harder part was carrying an improvement on calibration data over to new text. Use of AI tools. Claude, ChatGPT and Codex assisted in developing counterexamples, parts of the analysis, numerical and Lean code, and manuscript revisions. The mathematical arguments are given in the paper and its appendices; Section 10 states the scope and limitations of the formal checks. Appendix A Proof for Example 14 Setting. Columns are samples. Let X1=(1−11−111),W1=I2,W2=(1−2−1−1),X_1= pmatrix1&-1&1\\ -1&1&1 pmatrix, W_1=I_2, W_2= pmatrix1&-2\\ -1&-1 pmatrix, so that X2=ReLU(W1X1)=(101011)X_2=ReLU(W_1X_1)= pmatrix1&0&1\\ 0&1&1 pmatrix, W2X2=(1−2−1−1−1−2)W_2X_2= pmatrix1&-2&-1\\ -1&-1&-2 pmatrix and X3=ReLU(W2X2)=(100000)X_3=ReLU(W_2X_2)= pmatrix1&0&0\\ 0&0&0 pmatrix. Fix t>0t>0 and let E1=tHE_1=tH, E2=tGE_2=tG, where the eight entries of H=(hij)H=(h_ij), G=(gij)G=(g_ij) are independent and uniform on [−1,1][-1,1]. Put Q1=W1+E1Q_1=W_1+E_1, Z2=ReLU(Q1X1)Z_2=ReLU(Q_1X_1) and Q2B=W2+E2,Q2Q=W2+W2(X2−Z2)Z2⊤(Z2Z2⊤)−1+E2.Q_2^B=W_2+E_2, Q_2^Q=W_2+W_2(X_2-Z_2)Z_2 (Z_2Z_2 )^-1+E_2. For either scheme Y=ReLU(Q2Z2)Y=ReLU(Q_2Z_2) and L=∥Y−X3∥F2L= Y-X_3 _F^2. Let Pt=Z2⊤(Z2Z2⊤)−1Z2P_t=Z_2 (Z_2Z_2 )^-1Z_2, w=(1,−2)w=(1,-2), e1=(1,0,0)⊤e_1=(1,0,0) , K=(h11−h120h11+h120−h21+h22h21+h22),K= pmatrixh_11-h_12&0&h_11+h_12\\ 0&-h_21+h_22&h_21+h_22 pmatrix, b=h11−h12,ct=wK(I−Pt)e1,c0=13(−2h12+4h21).b=h_11-h_12, c_t=wK(I-P_t)e_1, c_0= 13(-2h_12+4h_21). Theorem 23. (a) For 0<t≤1100<t≤ 110 and every realization of (H,G)(H,G): all layer-1 pre-activations and all layer-2 pre-activations of both schemes have the strict signs of the teacher’s; Z2=X2+tKZ_2=X_2+tK; σmin(Z2)≥1−8t≥1−3t>0 _ (Z_2)≥ 1- 8\,t≥ 1-3t>0; and Q1Q_1, Q2BQ_2^B, Q2Q_2^Q are invertible. (b) For 0<t≤1100<t≤ 110, LQEP−LBASE=t2(ct2−b2)\ EL_QEP-EL_BASE=t^2\,E(c_t^2-b^2), with b2=23Eb^2= 23 and c02=2027Ec_0^2= 2027. (c) For 0<t≤1100<t≤ 110 and every H: ‖Pt−P0‖2≤4t(1+t)\|P_t-P_0\|_2≤ 4t(1+t), |ct|≤6|c_t|≤ 6, |c0|≤2|c_0|≤ 2, |ct−c0|≤24t(1+t)|c_t-c_0|≤ 24t(1+t). (d) For 0<t≤1100<t≤ 110, LQEP−LBASE≥t2(227−42t(1+t)),EL_QEP-EL_BASE\;≥\;t^2 ( 227-42\,t(1+t) ), and the right-hand side is strictly positive for 0<t≤16000<t≤ 1600. Proof. Step 1 (layer-1 gates). Q1X1=X1+tHX1Q_1X_1=X_1+tHX_1 and row i of HX1HX_1 is (hi1−hi2,−hi1+hi2,hi1+hi2)(h_i1-h_i2,\,-h_i1+h_i2,\,h_i1+h_i2), with entries of modulus at most 22. The entries of X1X_1 are ±1± 1, so no sign changes when 2t<12t<1, and Z2=X2+tKZ_2=X_2+tK. Write k11=h11−h12k_11=h_11-h_12, k13=h11+h12k_13=h_11+h_12, k22=h22−h21k_22=h_22-h_21, k23=h21+h22k_23=h_21+h_22 (so |kij|≤2|k_ij|≤ 2) and a=1+tk11a=1+tk_11, c=1+tk13c=1+tk_13, b′=1+tk22b =1+tk_22, d=1+tk23d=1+tk_23; then Z2=(a0c0b′d)Z_2= pmatrixa&0&c\\ 0&b &d pmatrix with a,b′,c,d∈[1−2t,1+2t]a,b ,c,d∈[1-2t,1+2t]. Step 2 (Gram matrix). ∥K∥F2=2∑i,jhij2≤8 K _F^2=2 _i,jh_ij^2≤ 8. Since X2X2⊤=(2112)X_2X_2 = pmatrix2&1\\ 1&2 pmatrix has eigenvalues 1,31,3, σmin(X2)=1 _ (X_2)=1, and Weyl’s inequality gives σmin(Z2)≥1−t‖K‖2≥1−8t _ (Z_2)≥ 1-t\|K\|_2≥ 1- 8\,t. Hence Z2Z2⊤Z_2Z_2 is invertible and ‖Z2⊤(Z2Z2⊤)−1‖2≤(1−8t)−1\|Z_2 (Z_2Z_2 )^-1\|_2≤(1- 8\,t)^-1. Step 3 (layer-2 pre-activations). Q2BZ2=W2X2+tW2K+tGZ2Q_2^BZ_2=W_2X_2+tW_2K+tGZ_2, and Q2QZ2=W2Z2+W2(X2−Z2)Pt+tGZ2=W2X2+tW2K(I−Pt)+tGZ2.Q_2^QZ_2=W_2Z_2+W_2(X_2-Z_2)P_t+tGZ_2=W_2X_2+tW_2K(I-P_t)+tGZ_2. Step 4 (rows of W2KW_2K). The rows are r1=wK=(k11,−2k22,k13−2k23)r_1=wK=(k_11,-2k_22,k_13-2k_23) and r2=(−k11,−k22,−k13−k23)r_2=(-k_11, -k_22, -k_13-k_23). The functions ‖ri‖22\|r_i\|_2^2 are convex in H, so they are maximized at vertices of the cube, where (k11,k13)∈(±2,0),(0,±2)(k_11,k_13)∈\(± 2,0),(0,± 2)\ and independently (k22,k23)∈(±2,0),(0,±2)(k_22,k_23)∈\(± 2,0),(0,± 2)\. Enumerating the sign choices gives ‖r1‖22≤36\|r_1\|_2^2≤ 36 and ‖r2‖22≤16\|r_2\|_2^2≤ 16 at every vertex, so ‖wK‖2≤6,‖r2‖2≤4.\|wK\|_2≤ 6, \|r_2\|_2≤ 4. Step 5 (layer-2 gates). The (i,j)(i,j) entry of W2K(I−Pt)W_2K(I-P_t) is ri(I−Pt)ejr_i(I-P_t)e_j, of modulus at most ‖ri‖2≤6\|r_i\|_2≤ 6; the same bound holds for W2KW_2K. The (i,j)(i,j) entry of GZ2GZ_2 has modulus at most z1j+z2j≤2+4tz_1j+z_2j≤ 2+4t. Thus both pre-activations differ from W2X2W_2X_2 entrywise by at most t(8+4t)≤0.84<1t(8+4t)≤ 0.84<1 for t≤110t≤ 110, whereas every entry of W2X2W_2X_2 has modulus at least 11. Step 6 (invertibility). σmin(Q1)≥1−t∥H∥F≥1−2t>0 _ (Q_1)≥ 1-t H _F≥ 1-2t>0. The eigenvalues of W2W2⊤=(5112)W_2W_2 = pmatrix5&1\\ 1&2 pmatrix are 12(7±13) 12(7± 13), so σmin(W2)≥1.3 _ (W_2)≥ 1.3 and ‖W2‖2≤2.305\|W_2\|_2≤ 2.305; hence σmin(Q2B)≥1.3−2t>0 _ (Q_2^B)≥ 1.3-2t>0. Finally Q2Q=W2−tW2KZ2⊤(Z2Z2⊤)−1+tGQ_2^Q=W_2-tW_2KZ_2 (Z_2Z_2 )^-1+tG, and by Step 2 the perturbation has norm at most t(2.3058/(1−8t)+2)≤1.11<1.3t (2.305 8/(1- 8\,t)+2 )≤ 1.11<1.3 for t≤110t≤ 110. This proves (a). Step 7 (the loss is one scalar). By Step 5, Y has the support of X3X_3, so L=(Y11−1)2L=(Y_11-1)^2 with Y11Y_11 the (1,1)(1,1) pre-activation. As z21=0z_21=0 and z11=az_11=a, Step 3 gives Y11B−1=t(b+g11a),Y11Q−1=t(ct+g11a).Y^B_11-1=t\,(b+g_11a), Y^Q_11-1=t\,(c_t+g_11a). Step 8 (cancellation of the last-layer error). b, ctc_t, a depend on H only and g11g_11 is independent of H with mean zero, so for ξ∈b,ctξ∈\b,c_t\, (ξ+g11a)2=ξ2+13a2E(ξ+g_11a)^2=Eξ^2+ 13Ea^2. Subtracting gives the identity in (b). Step 9 (t=0t=0). The orthogonal complement of the row space of X2X_2 is spanned by m0=(1,1,−1)⊤m_0=(1,1,-1) , so I−P0=13m0m0⊤I-P_0= 13m_0m_0 and c0=13(k11−2k22−k13+2k23)=13(−2h12+4h21)c_0= 13(k_11-2k_22-k_13+2k_23)= 13(-2h_12+4h_21). With hij2=13Eh_ij^2= 13: b2=23Eb^2= 23, c02=4+1627=2027Ec_0^2= 4+1627= 2027, and |c0|≤2|c_0|≤ 2. Step 10 (projection bound). m=(cb′,ad,−ab′)⊤m=(cb ,\,ad,\,-ab ) is nonzero and orthogonal to both rows of Z2Z_2, so I−Pt=mm⊤/‖m‖22I-P_t=m /\|m\|_2^2. For unit vectors u,vu,v spanning lines at angle θ∈[0,π2]θ∈[0, π2], ‖uu⊤−vv⊤‖2=sinθ\|u -v \|_2= θ; therefore ‖Pt−P0‖2=sin∠(m,m0)=dist(m0,spanm)‖m0‖2≤‖m−m0‖23.\|P_t-P_0\|_2= (m,m_0)= dist(m_0,span\m\)\|m_0\|_2≤ \|m-m_0\|_2 3. Each coordinate of m−m0m-m_0 is ±((1+tα)(1+tβ)−1)± ((1+tα)(1+tβ)-1 ) with |α|,|β|≤2|α|,|β|≤ 2, of modulus at most 4t+4t24t+4t^2. Hence ‖Pt−P0‖2≤4t(1+t)\|P_t-P_0\|_2≤ 4t(1+t). Step 11. |ct|≤‖wK‖2‖(I−Pt)e1‖2≤6|c_t|≤\|wK\|_2\,\|(I-P_t)e_1\|_2≤ 6 and |ct−c0|=|wK(P0−Pt)e1|≤6⋅4t(1+t)=24t(1+t)|c_t-c_0|=|wK(P_0-P_t)e_1|≤ 6· 4t(1+t)=24t(1+t). This proves (c). Step 12 (lower bound). With δ=ct−c0δ=c_t-c_0, ct2−c02=2c0δ+δ2≥−2|c0||δ|c_t^2-c_0^2=2c_0δ+δ^2≥-2|c_0||δ|, so ct2−c02≥−48t(1+t)|c0|≥−48t(1+t)20/27≥−42t(1+t),Ec_t^2-Ec_0^2≥-48\,t(1+t)\,E|c_0|≥-48\,t(1+t) 20/27≥-42\,t(1+t), because 20/27≤78 20/27≤ 78. By Steps 8–9, LQEP−LBASE=t2(227+ct2−c02)≥t2(227−42t(1+t))EL_QEP-EL_BASE=t^2 ( 227+Ec_t^2-Ec_0^2 )≥ t^2 ( 227-42t(1+t) ). Since t↦42t(1+t)t 42t(1+t) is increasing and equals 42⋅601360000<0.0702<227 42· 601360000<0.0702< 227 at t=1600t= 1600, the bound is positive on (0,1600](0, 1600]. ∎ Remark 24. (i) Since 4t(1+t)≤12t4t(1+t)≤ 12t and 6≤406≤ 40, the cruder estimates ‖Pt−P0‖2≤12t\|P_t-P_0\|_2≤ 12t, |ct2−c02|≤960t|c_t^2-c_0^2|≤ 960t and positivity for 0<t≤10−50<t≤ 10^-5 also hold. (i) Apart from Steps 8 and 9 the argument is deterministic and uses only |hij|,|gij|≤1|h_ij|,|g_ij|≤ 1; Step 8 uses that G is independent of H with g11=0Eg_11=0, and Step 9 uses the second moments of the hijh_ij. If the hijh_ij are i.i.d. with mean zero and second moment s, the leading coefficient is 20s9−2s=2s9 20s9-2s= 2s9 (so 29 29 for symmetric signs ±1± 1). (i) ct(H)=−c−t(−H)c_t(H)=-c_-t(-H), so for a symmetric law t↦ct2t _t^2 is even and the true correction is O(t2)O(t^2); numerically (ct2−b2)≈227−0.283t2E(c_t^2-b^2)≈ 227-0.283\,t^2 and stays positive on all of (0,110](0, 110]; this is a numerical observation. The interval (0,1600](0, 1600] is the proved one and is not sharp. Appendix B Further counterexamples in the shared-perturbation model Example 25 (Not an artifact of P=IP=I). Take one output, one input dimension and two calibration samples: W=1W=1, X^=(1,0) X=(1,0), X=(2,1)X=(2,1), E=1E=1, α=1α=1. Then A:=Wδ=(1,1)A:=Wδ=(1,1), P=diag(1,0)P=diag(1,0), B:=EX^=(1,0)B:=E X=(1,0). The QEP residual is A(I−P)−B=(−1,1)A(I-P)-B=(-1,1) with squared norm 22; the BASE residual on the same input is A−B=(0,1)A-B=(0,1) with squared norm 11. The counterexample of Theorem 17 also lifts to matrices. Let d≥2d≥ 2, L=2L=2, identity activations, W1=W2=IdW_1=W_2=I_d, X=[Id∣]∈ℝd×(d+1)X=[\,I_d 1\,] ^d×(d+1), α1=α2=1 _1= _2=1, and E1=DE_1=D, E2=−DE_2=-D with D=diag(ε1,…,εd)D=diag( _1,…, _d), 0<εi<120< _i< 12. Both schemes use W^1=I+D W_1=I+D and X^2=(I+D)X X_2=(I+D)X, which has full row rank, and P2≠Id+1P_2≠ I_d+1. The correction is −D(I+D)−1-D(I+D)^-1, so W^2BASE=I−D W_2^BASE=I-D and W^2QEP=(I+D)−1−D W_2^QEP=(I+D)^-1-D, all invertible, and R1=−DX,R2BASE=D2X,R2QEP=(D+D2)X.R_1=-DX, R_2^BASE=D^2X, R_2^QEP=(D+D^2)X. Hence UQEP>UBASEU^QEP>U^BASE; for D=εID= I the gap is exactly ε∥X∥F X _F. The mechanism is the scalar one applied coordinate by coordinate. A family that does not reduce to independent scalar channels: let N=(0100)N= ( smallmatrix0&1\\ 0&0 smallmatrix ), D=diag(a,b)D=diag(a,b) with a>2b>0a>2b>0, W1=IW_1=I, W2=DW_2=D, X=IX=I, α=(1,1)α=(1,1), and E1=tNE_1=tN, E2=t(ND−DN)=(b−a)tNE_2=t(ND-DN)=(b-a)tN with t>0t>0. Both schemes have X^2=I+tN X_2=I+tN. The outputs are D+btND+btN for BASE and D+(b−a)tND+(b-a)tN for QEP, so the output errors are btbt and (a−b)t(a-b)t, and UBASE=(a+b)tU^BASE=(a+b)t, UQEP=(2a−b)tU^QEP=(2a-b)t. All weights and Gram matrices are invertible for every t. Here E2E_2 is a commutator; when a=ba=b it vanishes, and QEP is exact while BASE has error btbt. Example 26 (The trajectory substitution fails independently). Take L=3L=3, scalars, identity activations, W=(1,1,1)W=(1,1,1), X=1X=1, α=(1,1,1)α=(1,1,1) and E(t)=t(12,−110,−727)E(t)=t\,( 12,- 110,- 727) with 0<t≤10<t≤ 1. All quantized weights are positive. On QEP’s own inputs, the QEP residual is no larger in magnitude than the BASE residual evaluated on the same inputs, at every layer: the per-layer comparison on a common input holds throughout. Yet on each scheme’s own trajectory the third-layer residuals satisfy |R3QEP|−|R3BASE|=t(64+69t−14t2)540>0, R_3^QEP - R_3^BASE = t\,(64+69t-14t^2)540>0, with a positive first-order coefficient 16/13516/135. At t=1t=1 the BASE residual is 00 and the QEP residual is 119/540119/540. A layer-wise comparison on common inputs therefore does not transfer to the two actual trajectories, even to first order. Theorem B.5 of [1] also asserts the composite inequality ∥δL+1QEP∥F≤UBASE _L+1^QEP _F≤ U^BASE, which compares the actual QEP output error with the BASE bound. In Theorem 17 it holds with equality. As an exact statement it fails as well, although not to first order in the example we know. Example 27 (The composite inequality fails for finite perturbations). Take L=3L=3, scalars, identity activations, W=(1,1,1)W=(1,1,1), X=1X=1, α=(1,1,1)α=(1,1,1), and E=(−12,−12,3)E=(- 12,- 12,3). The BASE weights are (12,12,4)( 12, 12,4), with output 11 and residuals (12,34,0)( 12, 34,0), so UBASE=54U^BASE= 54. The QEP weights are (12,32,133)( 12, 32, 133), with output 134 134 and output error 94>UBASE 94>U^BASE. Along the ray E(t)=t(−12,−12,3)E(t)=t(- 12,- 12,3), 0<t≤10<t≤ 1, the difference between the QEP output error and UBASEU^BASE equals t(3t−1)/2t(3t-1)/2, which is positive only for t>13t> 13; the first-order coefficients are 33 and 72 72, so the inequality holds to first order along this ray. A repaired composite bound. In the shared-perturbation model, put wl=∥Wl∥2w_l= W_l _2 and el=∥El∥2e_l= E_l _2, and let U~BASE=∑k=1Lγk∥RkBASE∥F∏s=k+1Lγs(ws+es). U^BASE= _k=1^L _k R_k^BASE _F _s=k+1^L _s(w_s+e_s). For Lipschitz activations and strengths in [0,1][0,1], whenever the corrected recursion is defined, ∥δL+1QEP∥F≤U~BASE. _L+1^QEP _F≤ U^BASE. Only the Gram inverses at positive-strength layers are required; the weights need not be invertible. For a fixed finite teacher and input, if el(t)≤tfle_l(t)≤ tf_l with fixed finite flf_l, then, uniformly over defined corrected trajectories and strengths, 0≤U~BASE(t)−UBASE(t)≤Ct2,∥δL+1QEP(t)∥F≤UBASE(t)+Ct2. gathered0≤ U^BASE(t)-U^BASE(t)≤ Ct^2,\\ _L+1^QEP(t) _F≤ U^BASE(t)+Ct^2. gathered for some finite C and 0<t≤t00<t≤ t_0. This is an additive remainder, not an exact comparison with the original bound. Proof of the repaired composite bound. Write Zl=X^lQEPZ_l= X_l^QEP, Yl=X^lBASEY_l= X_l^BASE, Bl=Xl−YlB_l=X_l-Y_l, Dl=Zl−YlD_l=Z_l-Y_l, and bl=∥Bl∥Fb_l= B_l _F, dl=∥Dl∥Fd_l= D_l _F, hl=bl+dlh_l=b_l+d_l. Then Xl−Zl=Bl−DlX_l-Z_l=B_l-D_l, h1=0h_1=0 and ∥Xl−Zl∥F≤hl X_l-Z_l _F≤ h_l. Let PlP_l be the row-space projection of ZlZ_l when αl>0 _l>0, and set Pl=0P_l=0 otherwise. Both PlP_l and I−αlPlI- _lP_l have norm at most one. The exact difference of the pre-activations is W^lQEPZl−W^lBASEYl=WlDl(I−αlPl)+ElDl+αlWlBlPl. W_l^QEPZ_l- W_l^BASEY_l=W_lD_l(I- _lP_l)+E_lD_l+ _lW_lB_lP_l. The Lipschitz property therefore gives dl+1 d_l+1 ≤γl[(wl+el)dl+αlwlbl], ≤ _l[(w_l+e_l)d_l+ _lw_lb_l], bl+1 b_l+1 ≤γl∥RlBASE∥F. ≤ _l R_l^BASE _F. Since αlwl≤wl+el _lw_l≤ w_l+e_l, addition yields hl+1≤γl∥RlBASE∥F+γl(wl+el)hl.h_l+1≤ _l R_l^BASE _F+ _l(w_l+e_l)h_l. Unrolling from h1=0h_1=0 proves the claimed bound. The residuals and the projection have stayed on their respective trajectories throughout. For the remainder, on the uncorrected trajectory RlBASE(t)=(Wl+El(t))(Xl−Yl(t))−El(t)XlR_l^BASE(t)=(W_l+E_l(t))(X_l-Y_l(t))-E_l(t)X_l, so ∥RlBASE(t)∥F≤(wl+tfl)bl(t)+tfl∥Xl∥F. R_l^BASE(t) _F≤(w_l+tf_l)b_l(t)+tf_l X_l _F. Together with bl+1(t)≤γl∥RlBASE(t)∥Fb_l+1(t)≤ _l R_l^BASE(t) _F and b1=0b_1=0, finite induction gives bl(t)=O(t)b_l(t)=O(t) and ∥RlBASE(t)∥F=O(t) R_l^BASE(t) _F=O(t) uniformly. Each downstream product changes by O(t)O(t) when its factors wsw_s are replaced by ws+es(t)w_s+e_s(t): expand the finite product, with 0≤es(t)≤tfs0≤ e_s(t)≤ tf_s. This does not divide by wsw_s and also covers zero weights. Multiplying the product differences by the O(t)O(t) residuals and summing proves the nonnegative O(t2)O(t^2) remainder. ∎ Thus the composite inequality of [1] is valid to first order in this precise sense, unlike the ordering of the two original bounds. For Example 27, U~BASE=6 U^BASE=6. References [1] Y. Arai and Y. Ichikawa. Quantization error propagation: Revisiting layer-wise post-training quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2504.09629. [2] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023. [3] J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), volume 6, 2024. [4] J. Chee, Y. Cai, V. Kuleshov, and C. De Sa. QuIP: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [5] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa. QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. In International Conference on Machine Learning (ICML), PMLR 235:48630–48656, 2024. [6] Y. Li, R. Yin, D. Lee, S. Xiao, and P. Panda. GPTAQ: Efficient finetuning-free quantization for asymmetric calibration. In International Conference on Machine Learning (ICML), PMLR 267:36690–36706, 2025. arXiv:2504.02692. [7] S. Zhang, H. Zhang, I. Colbert, and R. Saab. Qronos: Correcting the past by shaping the future… in post-training quantization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2505.11695. [8] H. Zhang, S. Zhang, I. Colbert, and R. Saab. Provable post-training quantization: Theoretical analysis of OPTQ and Qronos. arXiv:2508.04853 (version 2, April 2026). [9] Y. Choi, M. El-Khamy, and J. Lee. Universal deep neural network compression. arXiv:1802.02271, 2018. [10] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning (ICML), 2015. [11] A. Aghasi, A. Abdi, N. Nguyen, and J. Romberg. Net-Trim: Convex pruning of deep neural networks with performance guarantee. In Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1611.05162. [12] R. Zamir, Y. Kochman, and U. Erez. Achieving the Gaussian rate–distortion function by prediction. IEEE Transactions on Information Theory, 54(7):3354–3364, 2008. [13] O. Ordentlich and U. Erez. Performance analysis and optimal filter design for sigma-delta modulation via duality with DPCM. arXiv:1501.01829. [14] J. Zhang and R. Saab. SPFQ: A stochastic algorithm and its error analysis for neural network quantization. arXiv:2309.10975, 2023. [15] S. Li, J. Deng, K. Xu, R. Deng, H. Gu, M. Jiang, H. Shen, and K. Huang. Rethinking residual errors in compensation-based LLM quantization. In International Conference on Learning Representations (ICLR), 2026. arXiv:2604.07955. [16] S. Savkin, E. Porat, O. Ordentlich, and Y. Polyanskiy. NestQuant: Nested lattice quantization for matrix products and LLMs. In International Conference on Machine Learning (ICML), PMLR 267:53042–53062, 2025. [17] E. Lifar, S. Savkin, O. Ordentlich, and Y. Polyanskiy. WaterSIC: Information-theoretically (near) optimal linear layer quantization. In International Conference on Machine Learning (ICML), 2026. arXiv:2603.04956. [18] O. Ordentlich and Y. Polyanskiy. Optimal quantization for matrix multiplication. IEEE Transactions on Information Theory, 72(3):1943–1972, 2026. [19] L. Schuchman. Dither signals and their effect on quantization noise. IEEE Transactions on Communication Technology, 12(4):162–165, 1964. [20] R. M. Gray and T. G. Stockham Jr. Dithered quantizers. IEEE Transactions on Information Theory, 39(3):805–812, 1993. [21] R. Zamir and M. Feder. On lattice quantization noise. IEEE Transactions on Information Theory, 42(4):1152–1159, 1996. [22] L. B. Allal et al. SmolLM2: When smol goes big — data-centric training of a small language model. arXiv:2502.02737, 2025. [23] S. Biderman et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), 2023. [24] C. T. Mullis and R. A. Roberts. Synthesis of minimum roundoff noise fixed point digital filters. IEEE Transactions on Circuits and Systems, 23(9):551–562, 1976. [25] T. Hinamoto, H. Ohnishi, and W.-S. Lu. Roundoff noise minimization of state-space digital filters using separate and joint error feedback/coordinate transformation optimization. IEEE Transactions on Circuits and Systems I, 50(1):23–33, 2003. [26] B. Bamieh and M. Filo. An input–output approach to structured stochastic uncertainty. arXiv:1806.07473, 2018. [27] S. P. Lipshitz, R. A. Wannamaker, and J. Vanderkooy. Quantization and dither: A theoretical survey. Journal of the Audio Engineering Society, 40(5):355–375, 1992. [28] Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021. arXiv:2102.05426. [29] W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), 2024. arXiv:2308.13137.