Paper deep dive
Memory-Efficient Integer-Only Training of Deep Neural Networks
Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural networks, in which every tensor (weights, gradients, and activations) is stored as INT8, and every operation (forward pass, backward pass, weight update, optimizer update) is performed in integer arithmetic. Specifically, tensors are stored as signed integer mantissas paired with explicit power-of-two exponents, on which integer-only operations are performed using five custom composable building blocks (products, sums, divide-by-constant, nonlinear functions, and weight updates). This allows complete implementation of trainable integer layers, including convolutions, batch and layer normalization, softmax, and GELU. A per-group velocity mechanism with independent exponent tracking preserves gradient precision across neurons and channels with diverse magnitudes. As every tensor fits in INT8, training memory is roughly 4x smaller than the FP32 baseline. We validate our training algorithm on CNNs (ResNet-18/34/50/101, MobileNetV2) and Vision Transformers (ViT-Ti / B), demonstrating for the first time that training neural networks with INT8 tensors and integer arithmetic can achieve competitive accuracy on CIFAR-10, CIFAR-100 and ImageNet at a reduced memory budget.
Tags
Links
- Source: https://arxiv.org/abs/2610.16298v1
- Canonical: https://arxiv.org/abs/2610.16298v1
Trouble viewing inline? Open PDF directly →
Full Text
70,869 characters extracted from source content.
Expand or collapse full text
Memory-Efficient Integer-Only Training of Deep Neural Networks Hazem Hesham Yousef Shalby †thanks: Corresponding author: hazemhesham.shalby@polimi.it Fabrizio Pittorino Manuel Roveri Affiliation: Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB) Affiliation: Politecnico di Milano, Milan, Italy Abstract Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural networks, in which every tensor (weights, gradients, and activations) is stored as INT8, and every operation (forward pass, backward pass, weight update, optimizer update) is performed in integer arithmetic. Specifically, tensors are stored as signed integer mantissas paired with explicit power-of-two exponents, on which integer-only operations are performed using five custom composable building blocks (products, sums, divide-by-constant, nonlinear functions, and weight updates). This allows complete implementation of trainable integer layers, including convolutions, batch and layer normalization, softmax, and GELU. A per-group velocity mechanism with independent exponent tracking preserves gradient precision across neurons and channels with diverse magnitudes. As every tensor fits in INT8, training memory is roughly 4×4× smaller than the FP32 baseline. We validate our training algorithm on CNNs (ResNet-18/34/50/101, MobileNetV2) and Vision Transformers (ViT-Ti / B), demonstrating for the first time that training neural networks with INT8 tensors and integer arithmetic can achieve competitive accuracy on CIFAR-10, CIFAR-100 and ImageNet at a reduced memory budget. 1 Introduction The cost of training deep neural networks continues to grow with model size and dataset scale. A natural way to reduce this cost is to lower numerical precision, replacing 32-bit floating-point arithmetic with narrower integer operations that are cheaper in energy, area, and latency. Mixed-precision training Micikevicius et al. (2018) showed that FP16 suffices for forward and backward propagation with minimal accuracy loss, and later methods pushed the datapath further to FP8 Micikevicius et al. (2022) and INT8 Zhu et al. (2020); Zhao et al. (2021). However, these approaches still rely on floating-point arithmetic in at least one critical component, such as the weight update, optimizer state, or nonlinear functions. This residual dependence on floating-point arithmetic remains a practical bottleneck for two complementary reasons. On GPUs, although floating-point computation is efficient, the persistent training state, including weights, gradients, and optimizer moments, still requires 12 bytes per parameter. As a result, model and batch size are often constrained by FP32 optimizer state rather than by the low-precision compute path itself. On integer-only platforms, including microcontrollers, fixed-function accelerators, and in-memory compute substrates, floating-point support may be absent entirely. In this setting, eliminating all floating-point operations is a fundamental prerequisite for on-device learning. Existing approaches that move toward fully integer training Wang et al. (2022); Hu et al. (2025) either do not include standard optimization mechanisms such as momentum and learning-rate schedules, or remain restricted to small models and tasks. We address these challenges through a unified integer-only training framework. Our key observation is that any tensor can be represented as a signed integer mantissa together with an explicit power-of-two exponent, so arithmetic remains in the integer domain while scale is tracked exactly through exponent addition. From this representation, we derive five reusable primitives: products, sums, divide-by-constant, nonlinear functions, and weight updates. These primitives compose into complete layers with integer-only forward and backward computation. To preserve gradient precision when magnitudes vary substantially within a weight tensor, we further introduce a per-group velocity mechanism that maintains separate exponent tracking for different parameter groups, such as output channels, and enables stable integer-only SGD with momentum. Together, these ideas define, to our knowledge, the first fully integer-only training framework spanning forward propagation, backward propagation, and parameter updates that scales to modern deep neural networks and standard computer vision benchmarks. The framework is general enough to realize standard CNN and ViT layers, including integer-only softmax, GELU, batch and layer normalization. 2 Related Work We organize prior work by how far each approach eliminates floating-point from training; Table 1 summarizes the comparison and Table 2 lists the per-tensor storage format of each method. Low-precision training. Mixed-precision training Micikevicius et al. (2018) showed that FP16 suffices for forward/backward passes if FP32 master weights are maintained. Subsequent methods reduced the datapath to FP8 Micikevicius et al. (2022). Quantization-aware training methods (PACT Choi et al. (2018), LSQ Esser et al. (2020), DoReFa-Net Zhou et al. (2018), F8Net Jin et al. (2022), DQT Shalby et al. (2026)) quantize the forward pass while keeping FP32 master weights and a floating-point backward; Jacob et al. (2018) introduced an integer-arithmetic-only inference framework adopted in mobile deployment. All methods retain floating-point for the weight update. Integer forward and backward passes. Banner et al. (2018) were the first to quantize weights, activations, and a substantial portion of the gradient stream to 8-bit on ImageNet, though the final weight-gradient accumulation remains in higher precision. Zhu et al. (2020) introduced gradient clipping strategies for full INT8 training on ImageNet-scale CNNs; Zhao et al. (2021) added per-channel scaling to close the accuracy gap. Fixed-point variants adapt bit-widths dynamically Zhang et al. (2020), while AMPA Ding et al. (2024) pushes inference below INT8 via mixed-precision allocation. All leave the optimizer in FP32. Quasi-integer training. SWALP Yang et al. (2019) trains in 8-bit block floating-point, quantizing weights, activations, gradients, and the optimizer state via stochastic rounding; a higher-precision stochastic-weight-average running mean remains in float. WAGE Wu et al. (2018) was among the first to discretize weights, activations, gradients, and errors to low-bit integers simultaneously, but replace batch normalization layers with constant scaling to avoid floating-point operations. NITI Wang et al. (2022) performs all three stages (forward, backward, weight update) in integer arithmetic with a single per-layer block exponent, but this shared exponent forces small-gradient channels to lose their effective integer range once the scale adapts to large-gradient channels; the resulting sub-less-significant-bit (sub-LSB) vanishing is the failure mode that our per-group velocity (Section 3.3.1) addresses. NITI also lacks momentum and LR scheduling, and in practice scales only to AlexNet. Ghaffari et al. (2022) propose INT8 mantissas with an integer weight update, but their reference implementation relies on additional operations to switch back-and-forth from the floating point representation to the integer one, and falls back to float softmax in the ViT attention block. Tin-Tin Hu et al. (2025) targets microcontrollers but has been applied only to tiny sensor tasks. Integer-only nonlinear functions. On the inference side, I-BERT Kim et al. (2021) demonstrated that polynomial approximations can replace all nonlinear functions in BERT with integer-only arithmetic, and I-ViT Li and Gu (2023) extended this to Vision Transformers with integer-only Softmax, GELU, and LayerNorm. We follow a different approach, by encoding all non-linear functions in look-up tables (LUTs), without significant memory overhead (see Section 3.2.4). Table 1: Comparison of low-precision training methods. ✓ = fully integer / supported; ✗ = floating-point / not supported; ∼ = partially integer (some float remains). CNN @ INet means ResNet-18–scale or CNNs trained/evaluated on ImageNet; ViT means training or fine-tuning of a Vision Transformer. Category Method Fwd/Bwd int Wt update int Wt storage int Momentum + LR CNN @ INet ViT Low-precision training Mixed-Prec. Micikevicius et al. (2018) ✗ ✗ ✗ ✓ ✓ ✓ FP8 Micikevicius et al. (2022) ✗ ✗ ✗ ✓ ✓ ✓ F8Net Jin et al. (2022) ✗ ✗ ✗ ✓ ✓ ✗ DQT Shalby et al. (2026) ✗ ✗ ✗ ✓ ✓ ✗ Integer fwd/bwd passes Banner et al. (2018) ✓ ✗ ✗ ✓ ✓ ✗ Zhu et al. (2020) ✓ ✗ ✗ ✓ ✓ ✗ Zhao et al. (2021) ✓ ✗ ✗ ✓ ✓ ✗ Zhang et al. (2020) ✓ ✗ ✗ ✓ ✓ ✗ AMPA Ding et al. (2024) ✓ ✗ ✗ ✓ ✓ ✗ Quasi-integer training WAGE Wu et al. (2018) ✓ ∼ ∼ ∼ ✗ ✗ SWALP Yang et al. (2019) ∼ ✓ ∼ ✓ ✓ ✗ NITI Wang et al. (2022) ✓ ✓ ✓ ✗ ✗ ✗ Ghaffari et al. (2022) ✓ ∼ ✗ ✓ ✓ ∼ Tin-Tin Hu et al. (2025) ✓ ✓ ✓ ✗ ✗ ✗ Integer-only Ours ✓ ✓ ✓ ✓ ✓ ✓ Table 2: Per-tensor storage format and arithmetic precision. Method Weights Gradient Velocity Activations FP32 FP32 FP32 FP32 FP32 FP16 Micikevicius et al. (2018) FP32 FP32 FP32 FP16 FP8 NVIDIA Micikevicius et al. (2022) FP32 FP32 FP32 FP16 FP8 IBM / HFP8 Sun et al. (2019) FP16 FP16 FP16 FP8 INT8 fwd/bwd Jin et al. (2022) FP32 FP32 FP32 FP32 Ghaffari et al. (2022) FP32 FP32 INT16 FP32 Ours INT8 INT8 INT8 INT8 3 Method In our framework, all persistent tensors (weights, gradients, velocity, saved activations) are stored as INT8 mantissas with power-of-two exponents (Section 3.1). Arithmetic operations (inner products, sums) accumulate in transient INT32 accumulators and are immediately compacted back to INT8 by applying a tensor range dependent shift. Whenever this shift moves bits to the right, we use stochastic rounding so that the compacted value remains unbiased while still using the available integer range effectively (Figure 1(a)). This representation lets standard layer operations such as multiplication, sum, division, and nonlinear transformations be implemented directly on mantissas, while the corresponding exponents are updated separately through addition (Section 3.2). For weight updates, our framework combines two mechanisms to maintain numerical efficiency and precision. First, per-group velocity assigns a separate exponent to each output neuron or channel, which improves precision when gradient magnitudes differ across channels (Figure 1(b) and Section 3.3). Second, reactive rescaling adjusts the weight exponent after every update step so that the mantissa stays well matched to the available intB range, avoiding both overflow and under-utilization. Figure 1 summarizes the full framework. (a) Integer-only layers. Each layer composes one or more integer primitives (product, sum/merge, divide-by-constant, nonlinear LUT) on INT8 inputs into an INT32 accumulator, then compacts back to INT8 via a single shift S with eventual stochastic rounding. (b) Integer-only weight update (Equation 9). Five integer steps replace the float SGD-with-momentum step. Per-group velocity exponents let every channel use the full INT8 range at <0.1%<0.1\% overhead (green), while a single exponent would let the largest-gradient channel dominate (red). Figure 1: Method overview. All training tensors are integer (C,e)(C,e) pairs (see Eq. (1)) with no floating-point storage or arithmetic at any stage. (a) Forward and backward of every layer. (b) Integer-only SGD with momentum and per-group velocity. 3.1 Mantissa-exponent tensor representation For a bit-width B we define qmax=2B−1−1q_ =2^B-1-1 and store every tensor as a pair (C,e)(C,e), where C∈[−qmax,qmax]shapeC∈[-q_ ,q_ ]^shape is a signed intB mantissa and e∈ℤe is a shared exponent (qmax=127q_ =127 for INT8). The real-valued tensor that this represents is C⋅2e.C· 2^e. (1) Because the exponent is a power of two, multiplying or dividing by 2p2^p with p∈ℕp is an exact integer shift on e, so no rounding is needed by scale changes that happen to be powers of two. Compaction. Each layer accumulates its output in an INT32 accumulator A and then stores it back in intB using a tensor range dependent shift S: Cout=clip(shift_round(A,S),−qmax,qmax),eout=ein+S.C_out= clip ( shift\_round(A,S),\ -q_ ,\ q_ ), e_out=e_in+S. (2) The shift is chosen so that the stored mantissa uses the available INT8 range as fully as possible without exceeding it. Let m=∥A∥∞m= A _∞ be the maximum magnitude in A. The shift is calculated as S=⌈log2(m/qmax)⌉,S= _2(m/q_ ) , (3) positive when A must be contracted and negative when it can be expanded. In practice we evaluate S as S^=bit_length(m)−bit_length(qmax)+1 S=bit\_length(m)-bit\_length(q_ )+1, where bit_length(⋅)bit\_length(·) is a count-leading-zeros instruction and bit_length(qmax)bit\_length(q_ ) is a compile-time constant. Stochastic rounding. Right shifts are the only lossy step in the entire framework. To prevent small but nonzero gradient updates from systematically rounding to zero (which would stall learning), every right shift during training uses stochastic rounding11 1 Training ResNet-18 on CIFAR-10 with our algorithm and deterministic rounding achieves only 89.3%, a 5.4% gap compared to the stochastic rounding version (94.7%), confirming the advantage of stochastic rounding into the integer-only training pipeline.: shift_round(x,S)=⌊x+U2S⌋,U∼Unif0,1,…,2S−1, shift\_round(x,S)= x+U2^S , U \0,1,…,2^S-1\, (4) where ⌊⋅⌋ · is the floor (rounding toward −∞-∞). The estimator is unbiased for all signed x Ghaffari et al. (2022). 3.2 Layer building blocks Given the representation above, the forward and backward passes of all standard layers reduce to four primitives on integer mantissas, with exponents propagated by addition: products, sums, divide-by-constant, and nonlinear lookup tables. The first three (Sections 3.2.1, 3.2.2 and 3.2.3) are arithmetic; the fourth (Section 3.2.4) tabulates each elementwise nonlinearity once at initialization, since under the (C,e)(C,e) representation each input (xint,ex)(x_int,e_x) takes only 2BNe2^BN_e distinct values, with Ne≈30N_e≈ 30 exponents reached in practice. Appendix B works through how these primitives compose into the layers of our models, namely convolutions, linear layers, BN/LN, the ViT attention block, and the output head. 3.2.1 Products Let (X,eX)(X,e_X) and (W,eW)(W,e_W) be an activation and weight tensor. An inner-dimension-K dot product is performed on the mantissas in an INT32 accumulator, with the exponent obtained by addition: yacc=∑i=1KXiWi,eacc=eX+eW,y_acc= _i=1^KX_i\,W_i, e_acc=e_X+e_W, (5) so that (yacc,eacc)(y_acc,e_acc) represents the real value yreal=yacc⋅2eaccy_real=y_acc· 2^e_acc. Compacting via Equation 3 yields (CY,eY)(C_Y,e_Y) with eY=eacc+Se_Y=e_acc+S. The backward products ∇Xacc=GW∇ X_acc=GW and ∇Wacc=G⊤X∇ W_acc=G X are computed in the same way. 3.2.2 Sums Residual connections and gradient stream merges add two tensors (C1,e1)(C_1,e_1) and (C2,e2)(C_2,e_2), which requires aligning them to a common exponent. Assume e1≥e2e_1≥ e_2 and let Δe=e1−e2 e=e_1-e_2. When Δe≤Δemax e≤ e_ we left-shift C1C_1 by Δe e (exact, no rounding) and set ecommon=e2e_common=e_2; otherwise we right-shift C2C_2 by Δe e with stochastic rounding and set ecommon=e1e_common=e_1. The cap Δemax e_ prevents the left shift from overflowing the wide accumulator. The aligned mantissas (C1′,C2′)(C _1,C _2) are summed and accumulated in INT32 and compacted back to intB: Cout=clip(shift_round(C1′+C2′,S),−qmax,qmax),eout=ecommon+S.C_out= clip\! ( shift\_round(C _1+C _2,\,S),\ -q_ ,\ q_ ), e_out=e_common+S. (6) 3.2.3 Divide-by-constant Average pooling and normalization statistics divide a tensor (C,e)(C,e) by a positive integer constant D. When D=2kD=2^k with k∈ℕk is a power of two, the division is absorbed into the exponent at no cost: (C,e)/ 2k=(C,e−k).(C,\,e)\;/\;2^k\;=\;(C,\;e-k). (7) For general D, we precompute the integer reciprocal R[D]=⌊2SR/D⌋R[D]= 2^S_R/D , so that R[D]⋅D≈2SRR[D]· D≈ 2^S_R and the division becomes a multiply followed by a shift by SRS_R: (C,e)/D≈(shift_round(C⋅R[D],SR),e).(C,\,e)\;/\;D\;≈\; ( shift\_round(C· R[D],\;S_R),\;e ). (8) 3.2.4 Nonlinear lookup tables Each nonlinearity is a finite map tabulated once at initialization and evaluated at runtime by an indexed lookup plus a handful of integer ops. The tables are shared across all instances of a given operator and require no polynomial approximation or floating-point fallback. Softmax. Softmax turns logits into a probability distribution via softmax(xi)=exp(xi)/∑jexp(xj),softmax(x_i)= (x_i)/ _j (x_j), which we realize in three integer steps: (i) calculate xi′=xi−maxi(xi)x _i=x_i- _i(x_i) so that exp(xi′)∈(0,1]∀i (x _i)∈(0,1]\,\,\,∀\,\,\,i; (i) read each exp(xi′) (x _i) from an INT8 LUT; (i) normalize by an integer reciprocal of the row sum (applied as in Eq. (8)). The probabilities are stochastically requantized to unsigned INT8 and cached for the backward pass, which takes the upstream gradient ∂y∂ y from the next layer and evaluates the softmax Jacobian ∂xi=pi(∂yi−∑j∂yjpj)∂ x_i=p_i(∂ y_i- _j∂ y_j\,p_j) on the cached probabilities pip_i. When followed by cross-entropy, the loss-gradient is ∂ℒ/∂XL=1N(−y) /∂ X_L= 1N(p-1_y), so backpropagation starts from an integer tensor with no need of float fallback for the loss. GELU. We tabulate GELU(x)=xΦ(x)GELU(x)=x\, (x) (where Φ is the cumulative distribution function for the Gaussian distribution) and its derivative into a pair of INT8 LUTs indexed by (xint,ex)(x_int,e_x). Forward is one lookup; backward is one lookup, one integer multiply, and one shift. 3.3 Integer-only weight update with scale alignment Our target update is SGD with momentum applied to each weight tensor W, using learning rate η, momentum coefficient β∈[0,1)β∈[0,1), a persistent velocity (momentum buffer) V of the same shape as W, and the loss gradient ∇W∇ W produced by backpropagation. Each step applies V←βV+∇W,W←W−ηV.V←β V+∇ W, W← W-η V. (9) Realizing Equation 9 entirely on integer mantissa-exponent pairs exposes two scale-alignment problems that are much less pronounced in the forward and backward passes. First, gradient magnitudes can vary substantially across output channels within the same weight tensor; with a single shared exponent, large-gradient channels dominate the available dynamic range and updates for smaller channels are heavily quantized or vanish entirely. Second, after many updates the integer weight mantissa drifts: ‖W‖∞\|W\|_∞ may overflow INT8, or shrink so that most of the available range goes unused, in either case losing precision at the next subtraction. We address the first with per-group velocity (Section 3.3.1) and the second with reactive rescaling (Section 3.3.3). 3.3.1 Per-group velocity Rather than storing the velocity V as one tensor with a single shared exponent, we partition it along the output-neurons/channels axis of the weights W into NgN_g groups, each with its own exponent: (Vk,eV,k),Vk∈[−qmax,qmax]shape(Wk),k=1,…,Ng,(V_k,\,e_V,k), V_k∈[-q_ ,q_ ]^shape(W_k), k=1,…,N_g, (10) where WkW_k denotes the corresponding slice of the weight tensor; the weights themselves keep a single shared exponent eWe_W, since their per-tensor scale changes only slowly across steps (Section 3.3.3). Each slice ∇Wk∇ W_k (along the same axis as the velocity) of the gradient ∇W∇ W is compacted via Eq. (2) with its own shift SkS_k read from ‖∇Wk‖∞\|∇ W_k\|_∞, yielding (Gk,eG,k)(G_k,e_G,k) with eG,k=e∇W+Ske_G,k=e_∇ W+S_k. Both VkV_k and GkG_k therefore use the full INT8 range regardless of the scales of other slices. The storage overhead is one exponent per velocity slice, below 0.1%0.1\% for a typical fully connected or convolution layer; all velocity mantissas are INT8 in our experiments. 3.3.2 Update algorithm We encode the scalar hyperparameters as integer (num, shift) pairs: β≈βnum 2−βshiftβ≈ _num\,2^- _shift (e.g., β=0.9≈230/28β=0.9≈ 230/2^8) and η≈ηnum 2−ηshiftη≈ _num\,2^- _shift, precomputing a sequence (ηnum(t),ηshift(t))( _num^(t), _shift^(t)) per iteration t to realize cosine, step, or warmup schedules (Appendix D). All arithmetic below is applied independently per group k. We first apply momentum decay to the previous velocity: Dk=shift_round(βnum⋅Vk,βshift),eD,k=eV,k.D_k= shift\_round( _num· V_k,\; _shift), e_D,k=e_V,k. (11) We then align the grouped gradient (Gk,eG,k)(G_k,e_G,k) to the velocity exponent eV,ke_V,k. With δk=eG,k−eV,k _k=e_G,k-e_V,k, the aligned gradient is G~k=shift_round(Gk,−δk). G_k= shift\_round(G_k,\,- _k). (12) The new velocity is obtained by accumulation followed by compaction back to INT8: Vk←clip(shift_round(Dk+G~k,Sk′),−qmax,qmax),eV,k←eV,k+Sk′,V_k← clip ( shift\_round(D_k+ G_k,\;S_k ),\;-q_ ,\;q_ ), e_V,k← e_V,k+S_k , (13) with Sk′S_k chosen by Equation 3 from ‖Dk+G~k‖∞\|D_k+ G_k\|_∞. Finally, we form the update direction ηnum⋅Vk _num\!·\!V_k at exponent −ηshift+eV,k- _shift+e_V,k, align it to the weight exponent eWe_W (as in Equation 12), and subtract from the current weights in a single shift-round: Wk←Wk−shift_round(ηnum⋅Vk,eW+ηshift−eV,k).W_k← W_k- shift\_round ( _num· V_k,\;e_W+ _shift-e_V,k ). (14) Weight decay is handled the same way as the momentum coefficient: λ is stored as (λnum,λshift)( _num, _shift) and λWkλ W_k is added to GkG_k before the velocity update (see Appendix D for details). 3.3.3 Reactive rescaling After the groupwise updates have been applied, the integer weight tensor may either exceed the INT8 representable range or use only a small fraction of it. Letting m=‖W‖∞m=\|W\|_∞, a single signed shift restores the mantissa to the band [qmax/4,qmax][q_ /4,\,q_ ]: W←shift_round(W,S),eW←eW+S,S=⌈log2(m/qmax)⌉,m>qmax,−⌊log2(qmax/(4m))⌋,0<m<qmax/4,0,otherwise. aligned W&← shift\_round(W,\,S),\\ e_W&← e_W+S, aligned S= cases _2(m/q_ ) ,&m>q_ ,\\[1.0pt] - _2\! (q_ /(4m) ) ,&0<m<q_ /4,\\[1.0pt] 0,&otherwise. cases (15) The first case (S>0S>0) rounds the overflow mantissa down by S bits, keeping the represented real value within one LSB; the second case (S≤0S≤ 0) is an exact left shift. 4 Experiments Setup. We evaluate three architecture classes across three datasets: ResNet He et al. (2016) on CIFAR-10, CIFAR-100 Krizhevsky (2009), and ImageNet Deng et al. (2009); MobileNetV2 Sandler et al. (2018) on CIFAR-10; ViT-Ti from scratch on CIFAR-10; and ViT-B Dosovitskiy et al. (2021) fine-tuned from a pretrained checkpoint on CIFAR-10. All runs use SGD with momentum β=0.9β=0.9 (encoded as (βnum,βshift)=(230,8)( _num, _shift)=(230,8) per Section 3.3.2) and a cosine LR schedule with linear warmup, precomputed as (ηnum,ηshift)( _num, _shift) pairs. The default configuration uses INT8 for activations, weights, gradients, and velocity, with INT32 accumulators. Full per-model hyperparameters, augmentation, input quantization, and the integer LR schedule are listed in Appendix D. Main results. Table 3 summarizes the comparison. Ours is the only method that holds every persistent tensor in the training pipeline at INT8 (weights, gradients, velocity, and saved activations), yielding a 4×4× reduction in training memory across every configuration while staying within a few percent of FP32 accuracy and within 0.10.1–0.7%0.7\% of Ghaffari et al. (2022), despite eliminating their FP32 master weights, FP32 optimizer state, and float softmax fallback. The memory gap to other low-precision methods is structural: training memory is dominated by saved-for-backward activations Chen et al. (2016) (80–99%80--99\% of the FP32 footprint across our configurations, per-config breakdown in Appendix C), and prior INT8 forward/backward methods leave activations in FP32 between layers. The Pareto front corresponding to each method in Table 3 is reported in Figure 2. Our algorithm is particularly advantageous at low memory budgets: e.g., on ImageNet it reaches 76.4%76.4\% at 5.205.20 GB memory on ResNet-50, matching the FP32 ResNet-18 (4.614.61 GB, 70.6%70.6\%) memory footprint within 13%13\% with an increase of +5.8%+5.8\% in accuracy. Figure 2: Accuracy vs training memory on CIFAR-10, CIFAR-100 and ImageNet. Each point is one (method, model) entry from Table 3. CNN results. On ResNet-18 we trail FP32 by 0.6%0.6\% on CIFAR-10 (94.794.7 vs. 95.395.3), 0.5%0.5\% on CIFAR-100 (75.575.5 vs. 76.076.0), and 1.2%1.2\% on ImageNet (69.469.4 vs. 70.670.6), at roughly 4×4× smaller training-memory footprint than the baselines. On ResNet-50/ImageNet our algorithm reaches 76.4%76.4\%, a 0.3%0.3\% gap from FP32 (76.7%76.7\%) and within 1.2%1.2\% of the best INT8 forward/backward result (Jin et al. (2022), 77.6%77.6\%), while running at 5.205.20 vs. 20.8220.82 GB (same 4×4× memory reduction). MobileNetV2/CIFAR-10 widens to −3.2%-3.2\% (91.591.5 vs. 94.794.7), reflecting the architecture known sensitivity to quantization. Two factor compound: depthwise separable convolutions decompose each spatial filter into per-channel kernels with very few weights, making each channel gradient signal small and quantization-sensitive, even with our per-output-channel velocity (which contributes +2.2%, see Table 5); and the inverted-residual blocks introduce a high density of sequential requantization points across depth. To our knowledge this is the first fully integer training of ResNet and MobileNet architectures. Table 3: Test accuracy (%) and training memory (GB) on CIFAR-10, CIFAR-100 and ImageNet. Training details and hyperparameters are reported in Appendix D, while the memory computation is detailed in Appendix C. Dataset Model Method Acc. (%) Training Precision mem. (GB) CIFAR-10 ResNet-18 FP32 Ghaffari et al. (2022) 95.3 1.36 FP32 Ghaffari et al. (2022) 94.8 1.33 INT8/16 + FP32 Ours 94.7 0.340.34 INT8 ResNet-34 FP32 Mert Bülent Sarıyıldız (2018) 95.5 2.16 FP32 Ours 95.1 0.550.55 INT8 MobileNetV2 FP32 Zhao et al. (2021) 94.7 1.82 FP32 Zhu et al. (2020) 93.3 1.82 INT8 + FP32 Zhao et al. (2021) 94.3 1.82 INT8 + FP32 Ours 91.5 0.450.45 INT8 ViT-Ti FP32 Aleptonic (2025) 90.9 7.60 FP32 Ours 88.0 1.901.90 INT8 ViT-B FP32 Ghaffari et al. (2022) 99.1 9.33 FP32 Ghaffari et al. (2022) 98.8 9.17 INT8/16 + FP32 (fine-tune) Ours 96.1 2.332.33 INT8 CIFAR-100 ResNet-18 FP32 Ghaffari et al. (2022) 76.0 1.36 FP32 Ghaffari et al. (2022) 75.4 1.33 INT8/16 + FP32 Ours 75.5 0.340.34 INT8 ResNet-34 FP32 weiaicunzai (2017) 76.8 2.16 FP32 Ours 76.5 0.550.55 INT8 ResNet-50 FP32 weiaicunzai (2017) 77.4 6.40 FP32 Ours 77.1 1.601.60 INT8 ImageNet ResNet-18 FP32 Micikevicius et al. (2022) 70.6 4.61 FP32 Micikevicius et al. (2022) 70.1 2.37 FP8 + FP32 Zhu et al. (2020) 69.7 4.61 INT8 + FP32 Zhao et al. (2021) 70.2 4.61 INT8 + FP32 Jin et al. (2022) 71.1 4.61 INT8 + FP32 Ghaffari et al. (2022) 69.3 4.59 INT8/16 + FP32 Ours 69.4 1.151.15 INT8 ResNet-50 FP32 Micikevicius et al. (2022) 76.7 20.82 FP32 Micikevicius et al. (2022) 76.8 10.55 FP8 + FP32 Zhu et al. (2020) 76.3 20.82 INT8 + FP32 Zhang et al. (2020) 76.2 20.82 INT8 + FP32 Zhao et al. (2021) 76.7 20.82 INT8 + FP32 Jin et al. (2022) 77.6 20.82 INT8 + FP32 Ours 76.4 5.205.20 INT8 ResNet-101 FP32 TorchVision (2016) 77.4 30.84 FP32 Ours 77.5 7.717.71 INT8 Vision Transformers. ViT-Ti from scratch on CIFAR-10 reaches 88.0%88.0\%, a −2.9%-2.9\% gap from FP32 (90.9%90.9\%). ViT-B fine-tuning on CIFAR-10 reaches 96.1%96.1\% in 50 epochs, a −3.0%-3.0\% gap from FP32 (99.1%99.1\%). To our knowledge, this is the first fully integer-only training or fine-tuning of a Vision Transformer: every layer (attention softmax, 1/dh1/ d_h scaling, LayerNorm, and GELU) runs on integer mantissa-exponent pairs with no floating-point fallback at any stage, (see Table 11 for the accuracy impact of quantizing each layer). The closest prior point, Ghaffari et al. (2022), retains FP32 master weights, an FP32 optimizer, and a float softmax in attention. ViTs are typically trained with Adam/AdamW, while our framework supports only SGD+momentum at the moment; extending the integer pipeline to adaptive optimizers could improve performance and is a natural next step, since their moment updates require integer square roots and divisions that complicate the scale bookkeeping. Lower-bit regimes. Table 4 sweeps the bit-width B below INT8 on ResNet-18/CIFAR-10 on our algorithm and on the algorithm from Ghaffari et al. (2022). Both methods retain their default optimizer state while weights and activations drops to B bits: our velocity stays at INT8, while Ghaffari et al. (2022) master weights stay at FP32 and their velocity at INT16. Notably, our algorithm retains 70.5%70.5\% at B=4B=4 where Ghaffari et al. (2022) diverges. Memory scales with B for ours (0.34→0.170.34→0.17 GB), while Ghaffari et al. (2022) retains the persistent FP32 state. Table 4: Accuracy (%) and training memory (GB) on ResNet-18 / CIFAR-10 (batch 256) across bit-widths B∈8,7,6,5,4B∈\8,7,6,5,4\. Method Metric INT8 INT7 INT6 INT5 INT4 Precision (B∈4,…,8B∈\4,…,8\) Acc (%) 94.8 94.7 94.47 88.5 Diverge Ghaffari et al. (2022) Mem (GB) 1.33 1.33 1.33 1.33 1.33 intB + FP32, velocity INT16 Acc (%) 94.7 94.1 91.8 83.3 70.5 Ours Mem (GB) 0.34 0.30 0.26 0.22 0.17 intB, velocity INT8 Per-group velocity ablation. Table 5 isolates the contribution of per-group velocity (Section 3.3.1) on CIFAR-10. Disabling it costs 0.9%0.9\% on ResNet-18, 2.2%2.2\% on MobileNetV2, and 2.7%2.7\% on ViT-Ti. Table 5: Effect of per-group velocity on test accuracy (%) and exponent memory. Dataset Model FP32 w/o per-group velocity w/ per-group velocity (Reference) Acc. Exp. mem. Acc. Exp. mem. CIFAR-10 ResNet-18 95.3 93.8 (−1.5-1.5) 0.020.02 KB 94.7 (−0.6-0.6) 4.74.7 KB MobileNetV2 94.7 89.3 (−5.4-5.4) 0.050.05 KB 91.5 (−3.2-3.2) 16.716.7 KB ViT-Ti 90.9 85.3 (−5.6-5.6) 0.050.05 KB 88.0 (−2.9-2.9) 20.520.5 KB 5 Conclusion We demonstrated that a full SGD-with-momentum training pipeline can be realized end to end on INT8 mantissa-exponent tensors with no floating-point fallback at any stage. The resulting algorithm is, to our knowledge, the first integer-only training framework that scales to standard computer-vision benchmarks at modern depth: it trains ResNet-18/34/50/101 and MobileNetV2 to within a few percent of FP32 across CIFAR-10, CIFAR-100, and ImageNet, fine-tunes ViT-B integer-only, and trains ViT-Ti from scratch. Methodologically, per-output-channel velocity exponents are essential for integer optimizer state. Because every persistent tensor is INT8, training memory is roughly 4×4× smaller than FP32 across every configuration we tested. This shifts the accuracy-memory frontier: at a fixed memory budget, the integer pipeline trains substantially deeper and more accurate architectures than FP32 can admit; at a fixed architecture, it trains within a few percent of the FP32 baseline at a quarter of the memory. The same pipeline can in principle run on hardware without floating-point support (e.g. microcontrollers, in-memory compute substrates) opening on-device training of ResNet- and ViT-scale models. Limitations and future work. Extending the framework to adaptive optimizers (an integer Adam, requiring integer square roots, integer division, and exponent bookkeeping for the bias-correction terms) is the most substantive open direction for moving to architectures that are typically trained with Adam or AdamW. Replacing power-of-two exponents with richer scaling schemes (learned scales, non-power-of-two block exponents) would use dynamic range more efficiently and likely narrow the remaining accuracy gap on MobileNetV2 and ViT. Operator fusion (e.g. reducing the requantization points per ViT block) is a possible engineering direction for requantization-heavy architectures. Finally, bringing the framework to LLM training and parameter-efficient fine-tuning (LoRA) is a natural next step, with the same structural advantage of keeping the optimizer state at INT8 rather than FP32. Acknowledgments and Disclosure of Funding This work was carried out in the EssilorLuxottica Smart Eyewear Lab, a Joint Research Center between EssilorLuxottica and Politecnico di Milano. References Aleptonic (2025) Aleptonic Vision transformer from scratch. Note: https://github.com/Aleptonic/Vision-Transformer-From-ScratchGitHub repository Cited by: Table 3. Banner et al. (2018) R. Banner, I. Hubara, E. Hoffer, and D. Soudry Scalable methods for 8-bit training of neural networks. External Links: 1805.11046, Link Cited by: §2, Table 1. Chen et al. (2016) T. Chen, B. Xu, C. Zhang, and C. Guestrin Training deep nets with sublinear memory cost. External Links: 1604.06174, Link Cited by: §4. Choi et al. (2018) J. Choi, Z. Wang, S. Venkataramani, P. I. Chuang, V. Srinivasan, and K. Gopalakrishnan PACT: parameterized clipping activation for quantized neural networks. External Links: 1805.06085, Link Cited by: §2. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , p. 248–255. External Links: Document Cited by: §4. Ding et al. (2024) L. Ding, W. Fei, Y. Huang, S. Ding, W. Dai, C. Li, J. Zou, and H. Xiong AMPA: adaptive mixed precision allocation for low-bit integer training. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 10960–10977. External Links: Link Cited by: §2, Table 1. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. External Links: 2010.11929, Link Cited by: §4. Esser et al. (2020) S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha Learned step size quantization. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. Ghaffari et al. (2022) A. Ghaffari, M. S. Tahaei, M. Tayaranian, M. Asgharian, and V. P. Nia Is integer arithmetic enough for deep learning training?. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: Appendix C, Table 11, §2, Table 1, Table 2, §3.1, §4, §4, §4, Table 3, Table 3, Table 3, Table 3, Table 3, Table 3, Table 3, Table 4. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 770–778. External Links: Document Cited by: §4. Hu et al. (2025) Y. Hu, J. Zuo, E. Zhang, B. Iannucci, and C. Joe-Wong Tin-tin: towards tiny learning on tiny devices with integer-based neural network training. External Links: 2504.09405, Link Cited by: §1, §2, Table 1. Jacob et al. (2018) B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Jin et al. (2022) Q. Jin, J. Ren, R. Zhuang, S. Hanumante, Z. Li, Z. Chen, Y. Wang, K. Yang, and S. Tulyakov F8Net: fixed-point 8-bit only multiplication for network quantization. External Links: 2202.05239, Link Cited by: Appendix C, §2, Table 1, Table 2, §4, Table 3, Table 3. Kim et al. (2021) S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer I-bert: integer-only bert quantization. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 5506–5518. External Links: Link Cited by: §2. Krizhevsky (2009) A. Krizhevsky Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §4. Li and Gu (2023) Z. Li and Q. Gu I-vit: integer-only quantization for efficient vision transformer inference. External Links: 2207.01405, Link Cited by: §2. Mert Bülent Sarıyıldız (2018) Mert Bülent Sarıyıldız resnet-pytorch: resnet implementation in pytorch with cifar-10 baselines. Note: https://github.com/mbsariyildiz/resnet-pytorchGitHub repository Cited by: Table 3. Micikevicius et al. (2018) P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu Mixed precision training. External Links: 1710.03740, Link Cited by: §1, §2, Table 1, Table 2. Micikevicius et al. (2022) P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu FP8 formats for deep learning. External Links: 2209.05433, Link Cited by: Appendix C, §1, §2, Table 1, Table 2, Table 3, Table 3, Table 3, Table 3. Sandler et al. (2018) M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen MobileNetV2: inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 4510–4520. External Links: Document Cited by: §4. Shalby et al. (2026) H. H. Y. Shalby, F. Pittorino, F. Palermo, D. Trojaniello, and M. Roveri DQT: dynamic quantization training via dequantization-free nested integer arithmetic. Proceedings of the AAAI Conference on Artificial Intelligence 40 (30), p. 25252–25259. External Links: Link, Document Cited by: §2, Table 1. Sun et al. (2019) X. Sun, J. Choi, C. Chen, N. Wang, S. Venkataramani, V. Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Cited by: Table 2. TorchVision (2016) TorchVision TorchVision: pytorch’s computer vision library, pretrained models. Note: https://github.com/pytorch/vision Cited by: Table 3. Wang et al. (2022) M. Wang, S. Rasoulinezhad, P. H. W. Leong, and H. K.-H. So NITI: training integer neural networks using integer-only arithmetic. IEEE Transactions on Parallel and Distributed Systems 33 (11), p. 3249–3261. External Links: ISSN 2161-9883, Link, Document Cited by: §1, §2, Table 1. weiaicunzai (2017) weiaicunzai Practice on cifar-100 with pytorch. Note: https://github.com/weiaicunzai/pytorch-cifar100GitHub repository Cited by: Table 3, Table 3. Wu et al. (2018) S. Wu, G. Li, F. Chen, and L. Shi Training and inference with integers in deep neural networks. External Links: 1802.04680, Link Cited by: §2, Table 1. Yang et al. (2019) G. Yang, T. Zhang, P. Kirichenko, J. Bai, A. G. Wilson, and C. D. Sa SWALP : stochastic weight averaging in low-precision training. External Links: 1904.11943, Link Cited by: §2, Table 1. Zhang et al. (2020) X. Zhang, S. Liu, R. Zhang, C. Liu, D. Huang, S. Zhou, J. Guo, Q. Guo, Z. Du, T. Zhi, and Y. Chen Fixed-point back-propagation training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 2327–2335. External Links: Document Cited by: §2, Table 1, Table 3. Zhao et al. (2021) K. Zhao, S. Huang, P. Pan, Y. Li, Y. Zhang, Z. Gu, and Y. Xu Distribution adaptive int8 quantization for training cnns. Proceedings of the AAAI Conference on Artificial Intelligence 35 (4), p. 3483–3491. External Links: Link, Document Cited by: Appendix C, §1, §2, Table 1, Table 3, Table 3, Table 3, Table 3. Zhou et al. (2018) S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou DoReFa-net: training low bitwidth convolutional neural networks with low bitwidth gradients. External Links: 1606.06160, Link Cited by: §2. Zhu et al. (2020) F. Zhu, R. Gong, F. Yu, X. Liu, Y. Wang, Z. Li, X. Yang, and J. Yan Towards unified int8 training for convolutional neural network. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 1966–1976. External Links: Document Cited by: Appendix C, §1, §2, Table 1, Table 3, Table 3, Table 3. Appendix A Integer primitives, shifts, and safety caps This appendix defines the low-level integer primitives used throughout the paper: the clip and signed shift-and-round operators, the stochastic-rounding operator, and the accumulator safety caps that bound the compaction shift of Equation 3. Clip and shift-round. The two primitives from which every rescale in the paper is built are saturation and signed shift-with-round: clip(x,a,b)=min(max(x,a),b),shift_round(x,S)=⌊(x+b(S))/2S⌋,S>0,x⋅2−S,S≤0, clip(x,a,b)= ( (x,a),b), shift\_round(x,S)= cases (x+b(S))/2^S ,&S>0,\\[2.0pt] x· 2^-S,&S≤ 0, cases (16) where ⌊⋅⌋ · denotes the floor toward −∞-∞ and b(S)≥0b(S)≥ 0 is an integer bias that selects the rounding mode. We use b(S)=U∼Unif0,…,2S−1b(S)=U \0,…,2^S-1\ during training (stochastic rounding, Equation 4) and b(S)=2S−1b(S)=2^S-1 at inference (deterministic midpoint). Left shifts (S≤0S≤ 0) are exact and involve no rounding choice. The compaction shift of Equation 3 uses ⌈log2m⌉ _2m , computed by integer bit-length (count-leading-zeros); we handle the edge case m=‖A‖∞=0m=\|A\|_∞=0 by setting S=0S=0. Stochastic rounding is integer-only. Stochastic rounding samples a uniform integer U per tensor element and computes shift_round(x,S)=⌊(x+U)/2S⌋,U∼Unif0,…,2S−1, shift\_round(x,S)\;=\; (x+U)/2^S , U \0,…,2^S-1\, (17) so every ingredient is integer: a xorshift or PCG generator produces U in 3–5 integer ops (shifts and xors), and the addition and right shift are native integer instructions, so the same kernel runs on hardware without an FPU. The per-element PRNG cost is negligible against the K-wide inner product that produces x: K ranges from 99 for a 3×33×3 depthwise convolution to 46084608 for a ViT-B MLP projection, so the PRNG adds well under 1%1\% to the per-layer arithmetic. We hold one PRNG state per tensor to avoid cross-thread contention; at inference U is replaced by the deterministic midpoint 2S−12^S-1, eliminating the PRNG entirely. Accumulator safety caps. The shift chosen by Equation 3 must not cause the next reduction to overflow. Let the wide accumulator be n-bit signed with maximum magnitude Amax=2n−1−1A_ =2^n-1-1. For a K-wide inner product with |xi|,|wi|≤qmax|x_i|,|w_i|≤ q_ we have |∑i=1Kxiwi|≤Kqmax2| _i=1^Kx_iw_i|≤ K\,q_ ^2, and for an M-way sum of aligned mantissas |∑ixi|≤Mqmax| _ix_i|≤ M\,q_ . The largest admissible expansions are therefore Scapprod=⌊log2(Amax/(Kqmax2))⌋,Scapsum=⌊log2(Amax/(Mqmax))⌋.S_cap^prod= _2\! (A_ /(K\,q_ ^2) ) , S_cap^sum= _2\! (A_ /(M\,q_ ) ) . (18) If a cap is negative the reduction must be tiled, the operands pre-shifted, or a wider accumulator used. With INT32 accumulators (Amax≈2.1⋅109A_ ≈ 2.1· 10^9) and qmax=127q_ =127 (INT8), both caps stay comfortably positive for every layer size in our models. Appendix B Concrete layer instantiations Each layer is built by composing the primitives of Sections 3.2 and 3.3. Table 6 maps each layer or operator used in our models to the primitives it reuses and any layer-specific lookup table; the subsections below fill in operator-specific details (LUT definitions, backward equations, and architectural constants). Residual adds and backward gradient merges reuse the sum operator directly (Equation 6), and average pooling reuses the divide-by-constant primitive (Equations 7 and 8). Table 6: Layer and operator instantiations used by our framework. Layer / operator Primitives used Ref. Convolution (incl. 1×1, depthwise) Product Equations 5 and 3 Linear / fully-connected Product Equations 5 and 3 Patch embedding (strided conv) Product Equations 5 and 3 Residual add, gradient merge Sum Equation 6 Average pool (K×K×K) Divide-by-constant Equations 7 and 8 ReLU / ReLU6 Clip Section B.1 GELU Product, LUT Section B.1 Batch / Layer normalization Product, sum, divide-by-constant Section B.2 Softmax Product, divide-by-constant, LUT Section B.3 Cross-entropy loss Softmax, divide-by-constant, LUT Section B.3 ViT attention (Q/K/V, score, context) Product, softmax, constant rescale Section B.4 B.1 Activations Our models use three pointwise activations: ReLU (ResNet-18), ReLU6 (MobileNetV2), and GELU (ViT-Ti/B). Table 7 compares their integer forward and backward; none changes the exponent e, and only GELU requires a lookup table. Table 7: Pointwise activations used in our models. All operations are integer on the intBB mantissa C; the exponent e is carried through unchanged. dYdY denotes the upstream gradient. Activation Forward (on C) Backward (grad) LUT ReLU max(C,0) (C,0) dY⋅[C>0]dY·1[C>0] — ReLU6 clip(C,0,c),c=min(6≫e,qmax) clip(C,0,c),\;c= (6 e,q_ ) dY⋅[0<C<c]dY·1[0<C<c] — GELU LUTfwd[e,C] LUT_fwd[e,C] (Equation 19) (dY⋅LUTbwd[e,C])≫BG(dY· LUT_bwd[e,C]) B_G INT8 [30,256][30,256] ReLU and ReLU6 are single-clip operations with no LUT. For ReLU6, expressing the clipping threshold 66 in the mantissa domain requires the per-exponent cap c=min(6≫e,qmax)c= (6 e,q_ ) (a plain signed shift: right by e for e>0e>0, left by |e||e| for e≤0e≤ 0; no rounding is needed because c is a deterministic bound, and truncation only shrinks the clip region). When 6≫e≥qmax6 e≥ q_ the clip reduces to ReLU because the mantissa already saturates below 66. GELU LUTs. Because GELU is pointwise, its entire INT8 behavior fits in a pair of INT8 tables indexed by (ex,xint)(e_x,x_int): LUTfwd[ex,x] LUT_fwd[e_x,x] =clip(round(GELU(x⋅2ex)/2ex),−qmax,qmax), = clip\! ( round(GELU(x· 2^e_x)/2^e_x),\,-q_ ,\,q_ ), (19) LUTbwd[ex,x] LUT_bwd[e_x,x] =round(GELU′(x⋅2ex)⋅2BG), = round\! (GELU (x· 2^e_x)· 2^B_G ), with BG=6B_G=6 so that GELU′∈[−0.13,1.13]GELU ∈[-0.13,1.13] fits in INT8 as [−8,72][-8,72]. Reducing BGB_G from the common 88 to 66 produces no measurable change in backward accuracy versus autograd, because the downstream compaction to INT8 already limits gradient precision to the input bit-width. The backward multiply in Table 7 stays in INT32 since |LUTbwd|≤72| LUT_bwd|≤ 72 and |dY|≤qmax|dY|≤ q_ . Cross-bit-width LUT reuse. For B<8B<8, an intBB mantissa xBx_B at exponent eBe_B equals the INT8 mantissa xB≪(8−B)x_B (8-B) at exponent eB−(8−B)e_B-(8-B); a single INT8 GELU table therefore serves all bit widths by shifting the input into the INT8 domain and stochastically shifting the result back. B.2 Batch and layer normalization BN and LN share the same integer pipeline; only the reduction axis differs (the batch for BN, the feature dimension for LN). Given an input mantissa CxC_x at exponent exe_x reduced over D=mD 2sDD=m_D\,2^s_D elements, the layer computes y=γ(x−μ)/σ2+ϵ+by=γ\,(x-μ)/ σ^2+ε+b in five integer stages. A single reciprocal table R[mD]=⌊2SR/mD⌋R[m_D]= 2^S_R/m_D (we use SR=30S_R=30) is precomputed once and shared by every BN/LN layer at that odd-part mDm_D. Every step below is integer-only: no float intermediate is constructed. (1) Mean. The elementwise sum Tμ=∑iCx,iT_μ= _iC_x,i is the integer sum primitive (Equation 6); because the operands already share exe_x, no alignment is needed. Dividing by D=mD 2sDD=m_D\,2^s_D is a multiply-and-shift through the reciprocal table: Cμ=shift_round(Tμ⋅R[mD],SR+sD),eμ=ex.C_μ= shift\_round\! (T_μ· R[m_D],\;S_R+s_D ), e_μ=e_x. (20) (2) Variance. The centered square sum Tσ2=∑i(Cx,i−Cμ)2T_σ^2= _i(C_x,i-C_μ)^2 reuses the product-accumulator (Equation 5); the safety cap (Equation 18) covers it because |Cx,i−Cμ|≤2qmax|C_x,i-C_μ|≤ 2q_ . One more multiply-and-shift by R[mD]R[m_D] yields the variance mantissa: Cσ2=shift_round(Tσ2⋅R[mD],SR+sD),eσ2=2ex.C_σ^2= shift\_round\! (T_σ^2· R[m_D],\;S_R+s_D ), e_σ^2=2e_x. (21) (3) Integer square root and reciprocal. We take σ=⌊Cσ2+ϵint⌋σ= C_σ^2+ _int by the standard Newton integer square-root (a fixed number of integer shifts, adds, and compares). Count-leading-zeros decomposes σ=mσ 2sσ=m_σ\,2^s_σ, and one INT32 divide per reduction row builds invσ=⌊2SR/mσ⌋inv_σ= 2^S_R/m_σ (the cost amortizes over the D normalizations below); eσ=exe_σ=e_x. (4) Normalize. Each centered value is multiplied by invσinv_σ in an INT32 accumulator and compacted back to intBB with stochastic rounding (Equation 3): (Cx^,i,ex^)=compact((Cx,i−Cμ)⋅invσ,−SR−sσ).(C_ x,i,\,e_ x)\;=\; compact\! ((C_x,i-C_μ)·inv_σ,\;-S_R-s_σ ). (22) The natural exponent passed to compact cancels both the 2SR2^S_R in invσinv_σ and the 2sσ2^s_σ in σ, leaving the real-valued (x−μ)/σ(x-μ)/σ as the interpretation of (Cx^,ex^)(C_ x,e_ x). (5) Affine. The learnable (γ,eγ)(γ,e_γ) and (b,eb)(b,e_b) are integer parameters updated by the weight-update pipeline of Section 3.3; the final output is formed by one product and one sum: y=sum(prod(x^,γ),b)y= sum ( prod( x,γ),\,b ) using Equations 5, 3 and 6. The backward pass caches (Cμ,Cσ2,invσ,sσ)(C_μ,C_σ^2,inv_σ,s_σ) and back-propagates through the five stages with only the product, sum, and shift-round primitives; no step needs floating point. In summary, the entire BN/LN layer uses integer sums, integer products, a single INT32 divide per reduction row, one integer square root, and shift-rounds. Running statistics. The per-channel running mean and variance used at evaluation are stored as integer mantissa-exponent pairs (μrun,eμrun)(μ^run,e_μ^run), (σ2,run,eσ2,run)(σ^2,run,e_σ^2,run) updated by an exponential moving average μrun←(1−ρ)μrun+ρμ.μ^run←(1-ρ)\,μ^run+ρ\,μ. (23) The EMA factor ρ is represented as ρ≈ρnum/2ρshiftρ≈ _num/2 _shift (e.g. ρ=0.1≈13/27ρ=0.1≈ 13/2^7) and the update uses the same bounded-shift alignment policy as the velocity update, so the running statistics remain integer throughout training and inference. They are not updated through gradient descent, so they do not affect the training-state accounting in Table 2. B.3 Output head: softmax and cross-entropy From the final logits (XL,eXL)(X_L,e_X_L) of a batch of N samples over K classes, with XL∈ℤN×KX_L ^N× K and label vector y∈0,…,K−1Ny∈\0,…,K-1\^N, the classification head produces three quantities: 1. fixed-point softmax probabilities p∈[0,2B−2]N×Kp∈[0,2^B-2]^N× K stored as unsigned intBB (B=8B=8, so p∈[0,254]p∈[0,254]), used wherever a probability is consumed (e.g. the attention scores of Section B.4); 2. the scalar cross-entropy loss ℒ=−1N∑nlog(pn,yn)L=- 1N _n (p_n,y_n), used only for logging; 3. the loss gradient ∂ℒ/∂XL=1N(p−y) /∂ X_L= 1N(p-1_y), which seeds the reverse-order layer loop of the training step. All three share the two precomputed INT8 tables of Table 8; every other operation is an integer sum, product, divide-by-constant, or shift-round. Table 8: Precomputed INT8 tables for the output head. Built once at initialization and shared across every output instance in the model. LUT Entry Domain Scale Shape Size LUTexp LUT_ round(exp(k 2ex)⋅2L) round\! ( (k\,2^e_x)· 2^L ) k∈[−255,0]k∈[-255,0], ∼30 30 exponents L=7L=7 [30,256][30,256] ∼8 8 KB LUTlog LUT_ round(−log(p/2B)⋅2Lℓ) round\! (- (p/2^B)· 2^L_ ) p∈[1,2B−2]p∈[1,2^B-2] Lℓ=5L_ =5 [256][256] ∼256 256 B Softmax forward. Four integer steps per row: 1. xi′=xi−maxjxj∈[−255,0]x _i=x_i- _jx_j∈[-255,0]. 2. Ei=LUTexp[ex,xi′]E_i= LUT_ [e_x,x _i]. 3. Z=∑jEjZ= _jE_j and invZ=⌊2SR/Z⌋inv_Z= 2^S_R/Z with SR=30S_R=30 (one INT32 divide per row; the K normalizations below are multiplies). 4. pi=shift_round(Ei⋅invZ,SR−B)p_i= shift\_round(E_i·inv_Z,\;S_R-B), yielding unsigned intBB in [0,2B−2][0,2^B-2] (the 2L2^L scale of EiE_i and Z cancels in the ratio; stochastic rounding is applied during training). The same cached intBB tensor p is reused by the backward and the loss below. Loss value (reporting only). For logging only. For each sample we index pn,yn∈[1,2B−2]p_n,y_n∈[1,2^B-2] (floored to ≥1≥ 1 so −log0- 0 never appears) into LUTlog LUT_ and take the integer batch mean ℒ=1N∑n=1NLUTlog[pn,yn],eℒ=−Lℓ,L\;=\; 1N _n=1^N LUT_ [p_n,y_n], e_L=-L_ , (24) with division by N via Equation 7 or Equation 8. Gradient (seed of backpropagation). At the loss head, softmax and cross-entropy telescope into (∂ℒ/∂XL)n,k=1N(pn,k−[k=yn]), ( /∂ X_L )_n,k\;=\; 1N (p_n,k-1[k=y_n] ), (25) a signed integer subtract on the intBB tensor p (promoted to int(B+1)(B+1) to carry the −1-1) followed by division by N. Away from the loss (softmax inside attention), we apply the Jacobian on the cached p: a=∑jpj∂yja= _jp_j\,∂ y_j (product + sum), then ∂xi=pi(∂yi−a)∂ x_i=p_i(∂ y_i-a) (subtract + product), with one compaction back to intBB. B.4 Vision Transformer block The ViT reuses the same primitives as the CNN models: patch embedding is a strided convolution (integer product), and the CLS token and positional embeddings are learned integer parameters merged with the sum primitive. Each pre-norm block is Y=X+MHA(LayerNorm(X)),Z=Y+MLP(LayerNorm(Y)),Y=X+ MHA( LayerNorm(X)), Z=Y+ MLP( LayerNorm(Y)), (26) with LayerNorm from Section B.2, integer GELU inside the MLP from Section B.1, and the residual adds folded into the sum primitive (Equation 6). Multi-head attention, forward. Per head (hidden size dhd_h, sequence length T): 1. Three integer linear projections on the layer-normed input produce (Q,eQ)(Q,e_Q), (K,eK)(K,e_K), (V,eV)(V,e_V) in ℤT×dhZ^T× d_h. 2. Score accumulator AQK=QK⊤∈ℤT×TA_QK=QK ^T× T (INT32 product, Equation 5), with natural exponent eQ+eKe_Q+e_K. 3. Scale by 1/dh1/ d_h and compact to intBB: (Cscore,escore)=compact(shift_round(AQK⋅rd,sd),eQ+eK),(C_score,\,e_score)= compact\! ( shift\_round(A_QK· r_d,\,s_d),\;e_Q+e_K ), (27) where (rd,sd)(r_d,s_d) encode 1/dh1/ d_h as a rational shift (see next paragraph). 4. Row-wise integer softmax (Section B.3) yields p∈[0,2Bp−1]T×Tp∈[0,2^B_p-1]^T× T. 5. Context accumulator Actx=pV∈ℤT×dhA_ctx=pV ^T× d_h, compacted to intBB with natural exponent −Bp+eV-B_p+e_V. 6. A final integer linear concatenates heads and projects back to the embedding dimension. Every intermediate is an (C,e)(C,e) pair; only p and V need to be cached for the backward. 1/dh1/ d_h as a rational shift. Since dhd_h is fixed at model-design time (commonly dh∈32,64,96d_h∈\32,64,96\), 1/dh1/ d_h is a compile-time constant. We represent it as rd/2sdr_d/2^s_d with rd=⌊2sd/dh⌋r_d= 2^s_d/ d_h : for ViT-B (dh=64d_h=64), sd=10s_d=10 and rd=128r_d=128 give 128/1024=1/64128/1024=1/ 64 exactly; at the same sds_d, dh=32d_h=32 and dh=96d_h=96 are lossless within one LSB. The backward applies the identical (rd,sd)(r_d,s_d) to the score gradient. Multi-head attention, backward. Given the upstream gradient ∂Z∂ Z and the cached (p,V)(p,V), the reverse pass walks the forward pipeline in reverse, each stage using only product, sum, and shift-round primitives: 1. Output projection. Back through the final linear yields the context gradient ∂Actx∈ℤT×dh∂ A_ctx ^T× d_h (one integer product) and the projection weight gradient. 2. Context matmul. The product Actx=pVA_ctx=pV splits into ∂p=∂ActxV⊤∂ p=∂ A_ctx\,V and ∂V=p⊤∂Actx∂ V=p ∂ A_ctx. 3. Softmax Jacobian. Row-by-row on the cached p: ∂Cscore,i=pi(∂pi−∑jpj∂pj)∂ C_score,i=p_i (∂ p_i- _jp_j\,∂ p_j ), exactly as in Section B.3. 4. Score scaling. The same rational shift is applied symmetrically: ∂AQK=shift_round(∂Cscore⋅rd,sd)∂ A_QK= shift\_round(∂ C_score· r_d,\,s_d). 5. Score matmul. AQK=QK⊤A_QK=QK splits into ∂Q=∂AQKK∂ Q=∂ A_QK\,K and ∂K=∂AQK⊤Q∂ K=∂ A_QK Q. 6. Q/K/V projections. Back through the three input projections produces input-gradient streams ∂X(Q),∂X(K),∂X(V)∂ X^(Q),∂ X^(K),∂ X^(V) and weight gradients ∂WQ,∂WK,∂WV∂ W_Q,∂ W_K,∂ W_V. 7. Stream merge. Two applications of the sum primitive (Equation 6) give ∂X=∂X(Q)+∂X(K)+∂X(V)∂ X=∂ X^(Q)+∂ X^(K)+∂ X^(V). Each INT32 accumulator is compacted back to intBB via Equation 3 before storage, so every tensor persisted across stages is an (C,e)(C,e) pair. Appendix C Training memory computation The Training memory (GB) column of Table 3 reports the total training memory of each method at the benchmark configuration of its row. This appendix lists the per-tensor storage format of each method and the formula we use to combine it with the model’s parameter count and activation footprint. Per-tensor storage format. The per-tensor formats assumed for each method are listed in Table 2 (main text). Summing the weight, gradient, and velocity widths gives the persistent state in bytes per parameter; the activation format scales with batch size and input resolution. Two points deserve comment. FP8 NVIDIA Micikevicius et al. [2022] quantizes to FP8 only at GEMM inputs; outputs and saved-for-backward activations remain FP16/BF16 (their §4.1 states “output tensors were left in higher precision…moving more tensors to FP8 is the subject of future study”). INT8 forward/backward methods Zhu et al. [2020], Zhao et al. [2021], Jin et al. [2022] and Ghaffari Ghaffari et al. [2022] are implemented as GPU emulations: the reduced-precision format is used at GEMM operand boundaries, while the tensor carried between layers (and saved for backward) is kept at FP32 in the implementation that produced their reported accuracy. The 8-bit claims in those papers refer to the arithmetic datapath, not to tensor storage. Computing total training memory. For a model with P parameters and a forward pass that saves A activation elements per sample (summed across layers), training at batch size N uses Mtrain=bparam⋅P+bact⋅A⋅N,M_train\;=\;b_param· P\;+\;b_act· A· N, (28) where bparamb_param is the persistent state in bytes per parameter from Table 2 and bactb_act is the activation bytes per element from the same table. For the FP32 baseline, bparam=12b_param=12 and bact=4b_act=4. Per-configuration values. Table 9 lists P, A, and N for every benchmark configuration used in Table 3. The values of P are standard for the corresponding reference implementations; the activation counts A are obtained from PyTorch autograd profiling on the respective architectures, counting only the tensors that autograd retains for the backward pass. Table 9: Per-configuration inputs to Equation 28: parameters P, saved-for-backward activations A per sample, batch size N. Configuration P (M) A (M/sample) N ResNet-18 / CIFAR-10 (322) 11.2 1.29 256 MobileNetV2 / CIFAR-10 (322) 2.24 1.88 256 ViT-Ti / CIFAR-10 (322) 5.36 7.91 256 ViT-B / CIFAR-10 fine-tune (2242) 86.0 35.1 64 ResNet-18 / CIFAR-100 (322) 11.2 1.29 256 ResNet-34 / CIFAR-100 (322) 21.3 2.01 256 ResNet-18 / ImageNet (2242) 11.7 4.7 256 ResNet-50 / ImageNet (2242) 25.6 21.5 256 ResNet-101 / ImageNet (2242) 44.5 31.8 256 Plugging each (method, configuration) pair into Equation 28 with the bytes from Table 2 reproduces the Training memory (GB) entries of Table 3. The computation is implemented as a short Python script released with the paper. Appendix D Training recipe and hyperparameters This appendix collects the training details omitted from the main text: how inputs and activations are quantized, how biases and weight decay are folded into the integer pipeline, the integer learning-rate schedule, and the full hyperparameter grid of Table 10. Hardware and software. All training runs use a single NVIDIA A100-SXM4 (40 GB) GPU. The integer pipeline is implemented on top of PyTorch 2.10 with CUDA 12.8 and cuDNN 9.1. Input quantization. The dataloader produces float RGB tensors in [0,1][0,1]; we normalize with standard per-channel statistics and then quantize once to INT8 at the framework boundary using the compaction primitive (Equation 3): (X0,eX0)=compact(⌊x⋅27⌋,−7).(X_0,\,e_X_0)\;=\; compact\! ( x· 2^7 ,\;-7 ). (29) The shift is data-independent because the normalized range is bounded, so no online calibration is required. This is the only float-to-integer conversion in the pipeline; at evaluation the INT8 logits XLX_L feed integer argmax directly (Section B.3), so no integer-to-float conversion is needed either. Activation quantization. Between layers, activations are re-quantized via the compaction primitive (Equation 3); the shift S is computed from the per-tensor ‖A‖∞\|A\|_∞ during the forward pass, so no separate calibration step is needed. For ViT-B we additionally clip activations at the 99.99th99.99^th percentile of absolute values (scale_percentile=0.9999) before computing the shift, preventing a single rare outlier from consuming the dynamic range. CNNs and ViT-Ti do not require this clipping at any of the bit widths we tested. Biases (ViT only). The CNN models in our setup (ResNet-18, MobileNetV2) omit convolution and linear biases because batch normalization absorbs them; ViT-B retains biases in its Q/K/V and MLP projections. Each bias is stored as a (b,eb)(b,e_b) pair at the weight bit width (INT8 by default) and aligned to the output exponent eXWe_XW via the bounded-shift policy of Section 3.2.2 before being added to the pre-activation. Momentum and weight decay. The momentum coefficient β is stored as the integer pair (βnum,βshift)( _num, _shift) with β≈βnum 2−βshiftβ≈ _num\,2^- _shift; the default β=0.9β=0.9 uses (βnum,βshift)=(230,8)( _num, _shift)=(230,8) giving β≈230/28β≈ 230/2^8. L2L_2 regularization is injected into the per-group gradient before the momentum update, so the decay term participates in the velocity accumulation: Gk←Gk+shift_round(λnum⋅Wk,λshift),G_k\;←\;G_k\,+\, shift\_round\! ( _num· W_k,\; _shift ), (30) with (λnum,λshift)( _num, _shift) precomputed for the target λ (e.g. λ=10−4λ=10^-4). The augmented GkG_k then feeds Equation 13, so no separate weight-decay step is applied at the weight update itself. Integer learning-rate schedule. The learning rate is stored as η≈ηnum 2−ηshiftη≈ _num\,2^- _shift. Since schedules are known before training, we precompute one (ηnum(t),ηshift(t))( _num^(t), _shift^(t)) pair per iteration t as the best-fit (ηnum(t),ηshift(t))=argminηnum,ηshift|ηnum 2−ηshift−η(t)|subject to1≤ηnum≤ηnum,max,( _num^(t),\, _shift^(t))\;=\; _ _num,\, _shift\, | _num\,2^- _shift-η^(t) | to 1≤ _num≤ _num, , (31) where η(t)η^(t) follows the regime in use: geometric decay γηγη at milestones for step schedules; ηmin+12(ηmax−ηmin)(1+cos(πt/T)) _ + 12( _ - _ )(1+ (π t/T)) for cosine; linear interpolation during warmup. Table 10: Per-model hyperparameters used for Table 3. All runs: SGD with momentum, cosine LR with linear warmup. RN-18 C10/C100 MNv2 C10 ViT-Ti C10 ViT-B C10 RN-18 INet Batch size 256 256 256 64 256 Epochs 200 200 400 50 90 Base LR 0.1 0.05 0.05 0.001 0.1 Warmup epochs 5 5 5 0 5 Momentum β 0.9 0.9 0.9 0.9 0.9 Weight decay 10−410^-4 10−410^-4 10−410^-4 10−410^-4 10−410^-4 Velocity bits BVB_V 8 8 8 8 8 Accumulator bits 32 32 32 32 32 Activation clip pct. — — — 0.9999 — Augmentation crop+flip crop+flip crop+flip RandAugment crop+flip +cutout +AutoAug +AutoAug Appendix E Additional experiments This appendix supplements Section 4 with: (i) training dynamics for ResNet-18/CIFAR-10 (Figure 3); (i) an operator-level breakdown of the ViT-B accuracy gap on CIFAR-10 (Table 11); and (i) a layer-by-layer cosine-similarity analysis of the INT8 ViT that motivates the need for fine-tuning. Training dynamics. Figure 3 shows that INT8 training tracks the FP32 baseline on ResNet-18/CIFAR-10 over 200 epochs; step-LR drops produce clean discrete transitions with no instability or divergence. Figure 3: ResNet-18 / CIFAR-10: integer-only (INT8) vs. FP32 baseline, 200 epochs, step LR at 60/120/180. Where the INT8 ViT drifts. Table 11 attributes the entire 3.2% pre-finetuning gap to the Linear projections: replacing only the Linear layers with INT8 (keeping LayerNorm, GELU, and softmax in float) costs 3.2%, while subsequently moving LayerNorm, GELU, and softmax into the integer pipeline costs nothing further (LayerNorm −0.1-0.1, GELU +0.1+0.1, softmax +0.2+0.2). In other words, the integer attention machinery introduced in this paper – LUT-based softmax, 1/dh1/ d_h as a rational shift, and the cached-p Jacobian backward – is essentially free in accuracy terms; the residual gap is the generic Linear-quantization error that any INT8 fwd/bwd method incurs. Table 11: Operator layer ablation on ViT-B fine-tuning (CIFAR-10). All rows use INT8 weights and activations. Softmax GELU LayerNorm Linear Acc. (%) Δ vs. float float float float float 99.1 (float baseline) float float float INT8 95.9 −3.2-3.2 float float INT8 INT8 95.8 −3.3-3.3 float INT8 INT8 INT8 95.9 −3.2-3.2 INT8 INT8 INT8 INT8 96.1 −3.0-3.0 Ghaffari et al. [2022] (float softmax, FP32 weights) 98.8 −0.3-0.3 A complementary cosine-similarity analysis of the INT8 ViT without fine-tuning shows where this Linear-quantization error compounds. The match to the float model degrades through the stack from 0.9770.977 after the patch embedding to 0.830.83 after block 0, 0.640.64 after block 2, and below 0.010.01 after block 5. The drop is concentrated inside the attention sublayer – 0.95→0.640.95→ 0.64 in a single block – but, consistent with Table 11, the source is not the attention-specific operators: it is the four Linear projections per block (QWQQW_Q, KWKKW_K, VWVVW_V, and the output projection), whose ∼72 72 INT8 compaction points compound multiplicatively across the 12-block stack. Fine-tuning then closes the gap as the weights adapt to the INT8 quantizer.