Paper deep dive
MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference
Romain Facq, Sami Ben Ali, Olivier Sentieys
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/26/2026, 3:44:12 AM
Summary
The paper introduces MicroQonv, a method to efficiently apply microscaling quantization to convolutional neural networks (CNNs) during both training and inference. It addresses the inefficiencies of naive microscaling approaches, which require multiple quantizations and excessive memory movement due to the im2col transformation. MicroQonv utilizes a 'channel-batch-first im2col' strategy to quantize tensors only once, significantly reducing memory movement, storage, and quantization overhead. The method demonstrates substantial performance gains in object detection models (YOLOV8nano, YOLOV26nano) and improves accuracy in continual learning scenarios using 4-bit microscaling.
Entities (16)
Relation Signals (9)
MicroQonv → reduces → memory movement
confidence 95% · It reduces memory movement and storage by up to ×7.53 compared to their full-precision counterparts.
MicroQonv → uses → channel-batch-first im2col
confidence 95% · MicroQonv reduces the quantization cost... by quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col.
Olivier Sentieys → affiliatedwith → Univ Rennes
confidence 90% · Romain Facq Sami Ben-Ali Olivier Sentieys Affiliation: Univ Rennes, Inria, CNRS, IRISA, Rennes, France
Romain Facq → affiliatedwith → Univ Rennes
confidence 90% · Romain Facq Sami Ben-Ali Olivier Sentieys Affiliation: Univ Rennes, Inria, CNRS, IRISA, Rennes, France
Sami Ben-Ali → affiliatedwith → Univ Rennes
confidence 90% · Romain Facq Sami Ben-Ali Olivier Sentieys Affiliation: Univ Rennes, Inria, CNRS, IRISA, Rennes, France
MicroQonv → appliedto → YOLOV8nano
confidence 90% · MicroQonv reduces microscaling-quantized activation memory movement by ×3.5 for state-of-the-art object detection models YOLOV8nano
MicroQonv → appliedto → YOLOV26nano
confidence 90% · and ×2.2 for YOLOV26nano.
MicroQonv → enables → Continual Learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers' forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of $\times2$ for weights and gradients, and by up to $\times9$ for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to $\times7.53$ compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by $\times3.5$ for state-of-the-art object detection models YOLOV8nano and $\times2.2$ for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%.
Tags
Links
- Source: https://arxiv.org/abs/2609.28358v1
- Canonical: https://arxiv.org/abs/2609.28358v1
Trouble viewing inline? Open PDF directly →
Full Text
72,290 characters extracted from source content.
Expand or collapse full text
MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference Romain Facq Sami Ben-Ali Olivier Sentieys Affiliation: Univ Rennes, Inria, CNRS, IRISA, Rennes, France Abstract Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers’ forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of ×2× 2 for weights and gradients, and by up to ×9× 9 for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to ×7.53× 7.53 compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by ×3.5× 3.5 for state-of-the-art object detection models YOLOV8nano and ×2.2× 2.2 for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%. Index Terms: Quantization, convolutional neural networks (CNNs), microscaling, hardware acceleration. I Introduction Recent progress in Artificial Intelligence (AI) has enabled substantial improvements in numerous fields, such as natural language processing or medical imaging. Some of its applications are executed on resource-constrained devices, e.g., UAVs [1] and autonomous driving systems, while models are mostly trained in the cloud. Whether the model is trained in the cloud or on an edge device, backpropagation is energy-intensive and time-consuming: reducing training costs will benefit all training scenarios. Moreover, giving embedded systems the ability to train locally is a huge improvement to their autonomy. Thanks to Continual Learning (CL) [2], models trained in the cloud can be adapted to the dynamic real-world environment encountered by each device, and can even learn new tasks and classes while maintaining performance on previously seen data. A simple yet very effective CL strategy is the replay buffer [3], where devices regularly store data from their environment to keep training on it while encountering new data. By reducing the number of bits per parameter for weights, activations, and gradients, quantization techniques [4] help reduce energy and latency cost in two complementary ways: they reduce memory movement [5] to bring data to the processing elements, and lower processing cost by using smaller hardware operators [6, 7]. Thus, combining training strategies with quantization — also known as Fully-Quantized Training (FQT) — is a good approach to enhancing device efficiency. Recent microscaling [8, 9] data formats stand out among quantization methods because they support 4-bit quantization, often using the 4-bit Floating-Point (FP4) format to maintain near-full-precision accuracy. For example, Large Language Models (LLMs) are trained from scratch using various flavors of FP4 and microscaling for weights, activations, and gradients [10, 11]. Microscaling partitions a tensor into line-shaped blocks whose elements share a single scaling factor. For efficiency, these lines must be aligned with the direction of the dot-product computation. However, weights, activations, and gradients participate in two distinct General Matrix Multiplications (GeMMs), and their computation directions differ. As a result, they must be quantized twice, which is suboptimal. The issue is particularly pressing for activations: the tensor is expanded via explicit or implicit im2col (image-to-column) [12, 13] before being quantized to the microscale format. Since the same full-precision element can be assigned different quantized values due to varying distributions inside the blocks, either the im2col-expanded quantized tensor or the original full-precision tensor must be saved for back-propagation, which results in excessive memory movement and memory storage compared to other quantization techniques. This contributes to the well-known ”AI memory wall” [5] problem. This paper addresses these issues of increased memory movement and quantization computation due to the microscaling format by proposing the following contributions. We first present MicroQonv, a tensor reordering method that quantizes each tensor only once during training, using the microscaling format. Activations are quantized before an implicit variant of im2col called channel-batch-first im2col is applied, significantly reducing memory movement, memory storage, and overhead due to quantization computations. MicroQonv is compared with other quantized training methods, showing its relevance by achieving higher accuracy at the same quantization level or by requiring much less computation with only a tiny accuracy loss, far outweighed by the memory size and memory movement reductions it provides. The gains are validated using a DRAM simulator [14], which simulates memory transfers between off-chip and on-chip memory. Notably, MicroQonv reduces microscaling-quantized activation memory movement in state-of-the-art object detection models, by ×3.5× 3.5 for YOLOV8nano and ×2.2× 2.2 for YOLOV26nano. We then apply 4-bit MicroQonv to a Continual Learning setting using quantized latent replay buffer strategy, to improve the state-of-the-art (SoTA) final model accuracy by up to +11%. The paper is organized as follows. Section I gives the background on training, im2col transformations, and quantization. In Section I, we describe the problems that arise when applying microscaling quantization to convolutional layers. Section IV describes the MicroQonv approach. Then, Section V presents the DRAM simulation results to quantify energy and latency gains. Section VI presents the experimental setup and results in a deep learning scenario, while Section VII shows improvements in a continual learning scenario. I Background and Related Work I-A CNN Training Following the emergence of the Transformer architecture for natural language processing, Vision Transformers (ViTs) [15] gained widespread popularity in computer vision. Because ViT achieves very good accuracy with the attention mechanism, it is worth asking whether convolutional layers will remain state-of-the-art. As an example, ConvNext [16] demonstrates that carefully designed convolutional networks can compete with vision transformers in accuracy. More importantly, embedded systems applications such as image classification and object detection not only require accuracy but also low latency and low energy consumption. In that domain, the high computational cost of attention mechanisms can be a disadvantage, as shown in [17, Fig. 1]. In object detection, as we write these lines, YOLO26 [18] and RF-DETR [19] achieve the best accuracy-latency tradeoff depending on the deployment platform. Like earlier YOLO versions, YOLO26 is a fully convolutional network. While RF-DETR mainly uses a transformer architecture, it also uses a YOLO convolutional projector to have richer encoder features. Another competitive model is RT-DETRv4 [20], which combines a convolutional backbone with an attention mechanism. Regarding real-time open vocabulary object detection, convolution-based YOLO26-E [18] is leading the way. Thus, in embedded-system applications, the convolutional layer is a core component, and improving it will benefit the overall system. The training process of convolutional neural networks (CNNs) involves three primary operations performed on each convolution layer: forward propagation, backward propagation, and weight update. In this subsection, we focus on standard convolution layers. For clarity, we assume a stride of 1 and no padding. However, the methods presented in this paper are independent of these assumptions. I-A1 Forward Pass Given a tensor representing a batch of N input feature maps X∈ℝN⋅Cin⋅Hin⋅WinX ^N· C_in· H_in· W_in,11 1 ”⋅·“ denotes starting a new dimension, e.g. here X is a 4D tensor. and convolutional weights W∈ℝCout⋅Cin⋅Kh⋅KwW ^C_out· C_in· K_h· K_w, where CinC_in and CoutC_out are the input and output channel dimensions, respectively, Hin⋅WinH_in· W_in the input dimensions, and Kh⋅KwK_h· K_w the convolution kernel dimensions, the output feature map tensor is computed using the convolution operation: Y=W⋆X,Y=W X, (1) where ⋆ denotes the standard convolution. The output tensor is defined as Y∈ℝN⋅Cout⋅Hout⋅Wout,Y ^N· C_out· H_out· W_out, with Hout=Hin−Kh+1H_out=H_in-K_h+1 and Wout=Win−Kw+1.W_out=W_in-K_w+1. I-A2 Backward Pass During backpropagation, compute two gradients: the gradient with respect to the input activations and the gradient with respect to the weights. Input Gradient: Given the gradient of the output feature maps GY∈ℝN⋅Cout⋅Hout⋅Wout,G_Y ^N· C_out· H_out· W_out, the gradient with respect to the input activations is computed by convolving GYG_Y with the transposed kernels: GX=WT⋆GY,G_X=W^T G_Y, (2) where WT∈ℝCin⋅Cout⋅Kh⋅KwW^T ^C_in· C_out· K_h· K_w is obtained by swapping the input and output channel dimensions of W. Weight Gradient: The gradient w.r.t the weights GW∈ℝCout⋅Cin⋅Kh⋅KwG_W ^C_out· C_in· K_h· K_w is computed by accumulating over the batch dimension: GW[cout,cin,:,:]=∑n=1N(GY[n,cout,:,:]⋆X[n,cin,:,:]),G_W[c_out,c_in,:,:]= _n=1^N (G_Y[n,c_out,:,:] X[n,c_in,:,:] ), (3) with cin∈[1…Cin]c_in∈[1… C_in] and cout∈[1…Cout]c_out∈[1… C_out]. Notations in this paper follow the principles of Python tensor iterators. I-A3 Weight Update After computing the gradients, the weights are updated using a learning rate η. A standard Stochastic Gradient Descent update step is computed as W=W−ηGW.W=W-η\,G_W. I-B Compute Architecture Model CNN training computation can run on different hardware targets, such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or a Systolic Array (SA). While these architectures differ in performance and use cases, they share one limitation: none can store the entire model in the memory of the chip performing the computations. Thus, they require frequent data movement to and from off-chip memory to load GeMM inputs and store GeMM outputs. The general and hardware-agnostic architecture considered in this paper is illustrated in Figure 1. Computations are performed on data (weights, activations, or gradients) in the on-chip memory. Data must be transferred to and from off-chip memory before computation. We consider optimal reuse of tiles and tensors once located in on-chip memory. Fig. 1: Generic compute architecture model considered. I-C Im2col Transformation The im2col transformation is the most widely used technique for mapping convolution operations to general matrix multiplications [12, 13]. This enables the use of highly optimized matrix-multiply libraries or dedicated hardware units (e.g., systolic arrays). In this subsection, we illustrate the forward and backward pass transformations for a standard convolution layer, a necessary step for understanding MicroQonv as detailed in Section IV. Given the input activation tensor X, the goal of the im2col operation is to rearrange all local receptive fields (i.e., patches) of size CinKhKwC_inK_hK_w from all N input feature maps into the columns of a large 2D matrix X′=im2col(X)X = im2col(X). I-C1 Input Rearrangement For each sample n∈1,…,Nn∈\1,…,N\ and for every spatial output position (h,w)(h,w), corresponding to h∈1,…,Houth∈\1,…,H_out\ and w∈1,…,Woutw∈\1,…,W_out\, the im2col operator extracts the patch of X[n,:,h:h+Kh−1,w:w+Kw−1]X[n,:,h:h+K_h-1,\;w:w+K_w-1]. This patch is then flattened into a vector of length CinKhKwC_inK_hK_w and placed as one column of the resulting matrix X′∈ℝ(CinKhKw)⋅(NHoutWout)X ^(C_inK_hK_w)·(NH_outW_out). Therefore, Rows of X′X correspond to all kernel positions across all input channels. Columns of X′X correspond to every spatial position of the sliding window across all samples in the batch. I-C2 Kernel Flattening The convolution kernel W is similarly reshaped into a 2D matrix Wf∈ℝCout⋅(CinKhKw)W_f ^C_out·(C_inK_hK_w) by flattening each filter W[cout,:,:,:]W[c_out,:,:,:] into a row vector of length CinKhKwC_inK_hK_w, preserving the channel-major layout: Wf[cout,:]=flatten(W[cout,:,:,:]).W_f[c_out,:]=flatten\! (W[c_out,:,:,:] ). I-C3 Matrix Multiplication Formulation After applying both transformations, the forward convolution in Equation 1 can be computed as a single matrix multiplication Yf=Wf⋅X′Y_f=W_f· X , where Yf∈ℝCout⋅(NHoutWout)Y_f ^C_out·(NH_outW_out) is a flattened representation of the output tensor Y. The im2col and flattened formats are used in the backward pass for computing the flattened weight gradients GWf∈ℝCout⋅(CinKhKw)G_W_f ^C_out·(C_inK_hK_w): GWf=GYf⋅X′TG_W_f=G_Y_f· X T where GYf∈ℝCout⋅(NHoutWout)G_Y_f ^C_out·(NH_outW_out) is a flattened 2D representation of the output gradient GYG_Y and X′TX T the transpose matrix of X′X . Consequently, the input gradient GX′G_X with the same format as X′X is also computed using matrix multiplication: GX′=WfT⋅GYf.G_X =W_f^T· G_Y_f. This classic im2col approach is central to high-performance CNN implementations, enabling convolutions to reuse efficient GeMM kernels and hardware acceleration. However, it incurs significant memory overhead, as the reduced feature matrix from the im2col transformation can use up to Kh×KwK_h× K_w times as much memory as the original feature map. I-C4 Channel-First Im2col In [13], the authors introduce a more efficient variant of the im2col transformation that relies on storing the input feature maps in a channel-first layout within off-chip memory. As illustrated in Figure 2, the input tensor is organized as XCin∈ℝCin⋅(NHinWin),X_C_in ^C_in·(NH_inW_in), rather than being stored directly in the full im2col format XCin′X _C_in. This data layout enables the hardware to contiguously load input activations into on-chip memory along the input channel dimension as the convolution window slides over the spatial domain. Consequently, the im2col columns can be generated on the fly, during the read operation. By avoiding explicit im2col storage and exploiting contiguous channel-wise access, this approach significantly reduces both off-chip memory usage and bandwidth requirements. However, this optimization applies only to the forward convolution pass. During backpropagation, the computation of the weight gradients requires loading the transposed activation matrix XCin′TX_C_in T, which inherently incurs discontinuous access patterns across the WinW_in dimension of the channel-first tensor XCinX_C_in stored in off-chip memory. This discontinuous access pattern is illustrated by the red arrow in Figure 2. As a consequence, the channel-first layout does not preserve the same efficiency for the backward pass, limiting this approach to inference or forward-only execution paths. Other im2col improvement methods exist, such as [21, 22]: they generate im2col elements as the computation unit needs them, avoiding storage of the im2col-expanded tensor. While this is efficient for FP32 tensors where elements are independent, these methods do not combine well with microscaling quantization. Because the same activation pixel repeated in im2col shares several microscaling blocks, each time with different neighboring pixels, it can have several different microscaling-quantized values in the im2col tensor. Thus, the same element must be re-quantized several times, as in classic im2col expansion. Furthermore, either the FP32 tensor must be stored in memory for reuse in backward propagation, or its microscaling-quantized im2colexpanded version must be stored, which significantly reduces the off-chip-to-on-chip memory-movement gains of microscaling quantization. Fig. 2: Example of the channel-first layout for a convolution layer with Kh=Kw=2K_h=K_w=2 and Hin=Hout=3H_in=H_out=3. The tensor is flattened along the spatial and batch dimensions, producing NHinWinNH_inW_in contiguous representation across columns, each containing all CinC_in channels. These columns are streamed to generate the im2col matrix. Forming the transposed representation requires reading across lines rather than columns. To address these limitations, Section IV introduces a new storage layout designed to maintain contiguous memory access in both the forward and backward passes of convolution layers, enabling efficient end-to-end training without relying on explicit im2col representations within off-chip memory.22 2 As shown in Figure 3, we keep the following notation for the rest of the paper: when a dimension of a 2D matrix is defined by a product of tensor dimensions, the last dimension of the product is the one along which memory access is performed. I-D Quantization Quantization strategies aim to reduce the quantization error between full-precision elements and their quantized versions. The goal is to balance accurate quantization of small values (i.e., rounding error) with accurate quantization of large values (i.e., clipping error). A small rounding error causes a large clipping error, and vice versa. This is where microscaling [9, 8] thrives. By performing per-block quantization that does not clip the block’s maximum value, microscaling improves the balance between rounding and clipping errors. In blocks with large values, the clipping error is minimized at the expense of the approximation of small elements of that block. In other blocks, rounding error is minimized. This balance between clipping and rounding errors makes microscaling a good technique for quantizing weights, activations, and gradients [10], enabling energy-efficient low-precision multiply-and-accumulate (MAC) units. In the original microscaling format [9], the block scale is a power of two encoded in E8M0 format. The NVFP4 format from Nvidia [11] further reduces quantization error by performing per-tensor scaling before the block-scale calculation in the 8-bit floating-point FP8 E4M3 (4-bit exponent, 3-bit mantissa plus one sign bit) format. The added complexity of per-tensor quantization and the mantissa in the block scale is offset by lower quantization error, enabling small-bitwidth formats and the hardware gains they provide while maintaining satisfactory accuracy. Elements are quantized in FP4 E2M1 as in the original microscaling format [9], organized as block-lines of B=16B=16 elements. In the experiments, we assume a master copy of the weights is kept in FP32 precision and used to update the weights, as in SoTA methods. I Motivations We first describe how to quantize a tensor in the microscaling format, then show its limitations in convolution operations, which prior work has not addressed. I-A Microscaling quantization To quantize following the microscaling format, a tensor X is decomposed into blocks XiX_i of B elements xi,1,…,xi,B\x_i,1,…,x_i,B\. Following the principles of the NVFP4 format, each value of X is scaled with a per-tensor scaling factor TscaleT_scale and a per-block scaling factor BscaleB_scale. Tscale=max|X|/(maxBsf×maxBef),T_scale= |X|/ ( B_sf× B_ef ), (4) where maxBsf B_sf is the maximum of the block-scale format (i.e., 448 for FP8 E4M3), and maxBef B_ef is the maximum of the element format (i.e., 6 for FP4 E2M1). Bscale=CastE4M3(max|Xi|/(Tscale×maxBef)),B_scale=CastE4M3 ( |X_i|/(T_scale× maxB_ef) ), (5) where CastE4M3(.)CastE4M3(.) rounds its argument to the nearest FP8 E4M3 value. Block elements XiX_i are turned into quantized format XiqX_i^q as Xiq=Round(Xi/(Bscale×Tscale)),X_i^q=Round (X_i/(B_scale× T_scale) ), (6) where Round(.)Round(.) denotes rounding, either round-to-nearest or stochastic, to an FP4 E2M1 format value. In simulation, fake quantization is obtained by multiplying XiqX_i^q by Bscale×TscaleB_scale× T_scale. Following [23], activations and weights are rounded to the nearest quantized value while stochastic rounding is performed on the gradient. I-B Issues with dot products of microscaled blocks We consider two vectors R=r1,…,rSR=\r_1,…,r_S\ and T=t1,…,tST=\t_1,…,t_S\ of size S, which belong to bigger matrices quantized using microscaling with block size B. If the quantization is performed in the same direction as the dot product, the dot product can be written as Q(R×T)=Q(∑i=1Sri×ti)=∑j=1⌈S/B⌉srj×stj×∑i=1Bri,jq×ti,jqQ(R× T)=Q ( _i=1^Sr_i× t_i )= _j=1 S/B sr_j× st_j× _i=1^Br^q_i,j× t^q_i,j (7) where ⌈.⌉ . rounds the content to the nearest superior integer, and Q(.)Q(.) represents quantization of both vectors involved in microscaling format. srjsr_j and stjst_j represent the scales of each microscale block, ri,jqr^q_i,j and ti,jqt^q_i,j are quantized elements of R and T vectors, respectively, in the i-th position of the j-th block. However, when the quantization direction is not the one of the dot product, scale blocks must be applied after every low-precision element multiplication, making the computation inefficient to the point where requantizing the whole tensor in the same direction is better. Ideally, each tensor would be quantized only once. In practice, as seen in the matrix multiplication formulation of Section I-C for GX′G_X , YfY_f and GWfG_W_f, each tensor is involved in two GeMM computations that happen in different directions: weight and activation matrices are transposed in the backward pass, whereas output gradients are on different sides of the GeMM product. I-C Limitations of related work The associated code with the DaCapo accelerator [24] shows that their convolutional layers load full precision tensors and then quantize them in line, in the forward as well as in the backward pass. To avoid requantization and extra memory movement, the accelerator of Cuyckens et al. [25] uses square quantization: instead of quantizing along a single dimension, it quantizes in a square shape across two dimensions of the tensor. Thus, tensors quantized in the forward pass can be reused in the backward pass. However, the authors consider only models with fully-connected layers. Extrapolating their method to convolutional layers involves memorizing the square-quantized version of an im2col-expanded tensor for activations during the forward pass, for its reuse during the backward pass. This reduces, if not cancels, the memory-movement and memory storage gains from quantization. Therefore, combining microscaling quantization and convolution layer operations remains an open question. IV MicroQonv: Reshaping Memory Layout for Efficient Microscaling Training In this section, we first introduce channel-batch-first im2col, a method that reorders weights, activations, and gradients so they can be quantized once during training while maintaining efficient computation. We then specify the selected quantization format, show the quantization and memory-movement gains, and compare MicroQonv’s accuracy with other methods. IV-A Reshaping memory layout with channel-batch-first im2col We store the input feature maps in a layout that enables contiguous data access along both the input-channel dimension CinC_in and the batch dimension N. To this end, we propose rearranging the input activations X, as presented in Figure 2, to XCin,N∈ℝCin⋅(HinWinN)X_C_in,N ^C_in·(H_inW_inN), as illustrated in Figure 3. In this representation, each spatial location (h,w)(h,w) contains a contiguous block of all channels across all batch elements. This organization ensures contiguous memory access during both major phases of convolutional training. During the forward pass, streaming the im2col matrix XCin,N′∈ℝ(KhKwCin)⋅(HoutWoutN)X _C_in,N ^(K_hK_wC_in)·(H_outW_outN) into on-chip memory requires sliding the convolution window and reading consecutive values along the CinC_in dimension. Since channel values are stored contiguously at each spatial position, the im2col blocks can be generated on the fly with minimal memory overhead. During the backward pass, computing input gradients requires reading the transposed activation matrix XCin,NTX^T_C_in,N along the batch dimension N. With batch elements stored adjacently, these accesses are also contiguous, eliminating the discontinuous striding that occurs in channel-first layouts. This layout preserves the memory-storage benefits of the channel-first representation while enabling optimal memory-bandwidth use for both forward and backward convolution operations. Furthermore, this layout enables data reuse: contiguous blocks of XCin,NX_C_in,N in either channel or batch dimension can be loaded only once from off-chip memory to on-chip memory and repeated on-chip to form channel-batch first im2col representation XCin,N′X _C_in,N, noticeably reducing memory movement. Fig. 3: Channel-batch-first layout: activations are stored compactly in off-chip memory for contiguous im2col and transpose access. Highlighted squares show MicroQconv quantization blocks across N and CinC_in dimensions. This layout also enables activation reuse, reducing redundant memory reads by avoiding repeated accesses to the same data. IV-B Efficient quantization directions Reordering input activations with channel-batch-first im2col requires changing the shapes of other tensors for GeMM computations. WfW_f must be reshaped as WfCout,Cin∈ℝCout⋅(KhKwCin)W_fC_out,C_in ^C_out·(K_hK_wC_in), and GYfG_Y_f reshaped as GYfCout,N∈ℝCout⋅(HoutWoutN)G_Y_fC_out,N ^C_out·(H_outW_outN). We now have a 2D matrix formulation of all tensors, which are compliant with all three GeMM computations: YfCout,N=WfCout,Cin⋅XCin,N′Y_fC_out,N=W_fC_out,C_in· X _C_in,N, GWfCout,Cin=GYfCout,N⋅XCin,N′TG_W_fC_out,C_in=G_Y_fC_out,N· X T_C_in,N and GX′Cin,N=WfCout,CinT⋅GYfCout,NG_X C_in,N=W^T_fC_out,C_in· G_Y_fC_out,N. Simple reordering of YfCout,NY_fC_out,N, GWfCout,CinG_W_fC_out,C_in and GX′Cin,NG_X C_in,N turns them into YfY_f, GWfG_W_f and GX′G_X respectively. Note that XCin,N′X _C_in,N is the im2col transformation of off-chip data layout XCin,NX_C_in,N, while WfCin,CoutW_fC_in,C_out and GYfCout,NG_Y_fC_out,N have the same representation off-chip and on-chip. With a 2D matrix formulation, we can quantize each tensor into a square block, so the computation direction always aligns with one dimension of the square. This ensures contiguous memory accesses and efficient computations as in Eq. 7 with a small variation: block scales must be applied every B B elements in both dimensions covered by the square. This way, XCin,NX_C_in,N is quantized along CinC_in and N directions, WfCin,CoutW_fC_in,C_out is quantized along CinC_in and CoutC_out directions and GYfCout,NG_Y_fC_out,N is quantized along CoutC_out and N directions. Figure 3 shows an example of MicroQonv quantization block for input activations in 2D shape, as stored in the memory: quantizing XCin,NX_C_in,N allows for keeping contiguous memory access for quantization blocks after im2col transformation into XCin,N′X _C_in,N. We combine these quantization directions with the NVFP4 format described in Section I-D, except the block shape is a square, not a line, and its size is B=64B=64 (8×8)(8× 8) as in [25], not 16. All experiments involving microscaling quantization are with a block size of B=64B=64. While we employ a specific block size and representation, MicroQonv is compliant with all block scales and element representation formats, and applies to all microscaling variants, including their early version, the block floating-point format. MicroQonv is hardware-agnostic: it applies to every hardware accelerator following the generic model depicted in Figure 1. Among others, it applies easily to custom fully-quantized-training accelerators with an explicit off-chip/on-chip hierarchy and programmable address generation/dataflow, the same class as Dacapo [24] and Cuyckens et al. [25]. MicroQonv is a design technique informing how such accelerators should lay out and quantize data, and it requires zero additional hardware circuitry. IV-C Gains in memory movement and per-element quantization Table I compares MicroQonv performance with Dacapo [24], and an adaptation of Cuyckens et al. [25] to convolutional layers, whose methods are available in Section I. The method described in Cuyckens et al. quantizes the im2col-expanded activation tensor in square 8×8 blocks and saves it for reuse during backward propagation. Dacapo quantizes the im2col-expanded tensor in line blocks of 64, following a constant line direction for forward propagation and a constant column direction for backward propagation, and stores the FP32 activation tensor. These theoretical results are obtained by applying a standard convolution using layer dimensions with parameters: KH=KW=3K_H=K_W=3, padding and stride of 1. Per-element (Per-elt.) quantization represents the number of times an element is quantized. Backward tensor size represents the size of the tensor that must be transferred and stored off-chip during forward propagation, to then be brought on-chip during backward propagation. The backward tensor size is compared to the MicroQonv tensor. As discussed previously, MicroQonv reordering allows for quantizing every tensor once, and the reordered activation tensor XCin,NX_C_in,N is quantized before its im2col expansion into XCin,N′X _C_in,N. Table I confirms the claims with reductions of the activation quantization overhead by ×9× 9 and the backward memory movement for activations by ×7.53× 7.53 compared to other methods, which would translate into significant energy gains for the hardware performing training. This is further analyzed in Section V. TABLE I: Quantization overhead and memory movement during backpropagation for Weights (W), Activations (A) and Gradients (G) of a convolutional layer with Kh=Kw=3K_h=K_w=3 quantized with 4-bit microscaling. Method Per elt. quantization Backward Mem. Move. A W G A W Dacapo [24] ×18× 18 ×2× 2 ×2× 2 ×7.53× 7.53 ×7.53× 7.53 Cuyckens et al. [25] ×9× 9 ×1× 1 ×1× 1 ×9× 9 ×1× 1 MicroQonv ×1×1 ×1×1 ×1×1 ×1×1 ×1×1 IV-D Accuracy analysis All methods use NVFP4 quantization format and a block size of 64, unless otherwise specified. Since MicroQonv introduces a new quantization pattern, the computation differs slightly from other methods. To compare accuracy, we train a ResNet-32 model on the CIFAR-100 dataset and a ResNet-18 model on the first 100 ImageNet classes. All methods use the same hyperparameters. Both training methods use a cosine-annealed scheduler and stochastic gradient descent with a momentum of 0.9. ResNet-32 is trained with a learning rate of 0.1 for 200 epochs, a weight decay of 5e−45e^-4, and a batch size of 128. ResNet-18 is trained with a learning rate of 0.05 for 150 epochs, a weight decay of 4e−54e^-5, and a batch size of 256. All layers are quantized except the first and last layers, which are kept at full precision. Results reported in Table I include the test accuracy mean and standard deviation (σ) on five training runs. All methods are within 0.6% of the FP32 baseline in both training scenarios; the different quantization methods are within a 0.25% σ window, which is roughly the margin of error. MicroQonv is slightly below Cuyckens et al., itself slightly below Dacapo [24]. TABLE I: Comparison of test accuracy and quantization MSE per element on ResNet-32 (R32) with CIFAR-100 (C100) and Resnet-18 (R18) with ImageNet (Inet100). Method Test acc % Normalized MSE R32-C100 R18-Inet100 A W G FP32 baseline 71.05±.2971.05±.29 75.55±.2275.55±.22 NANA NANA NANA Dacapo [24] 70.86±.2470.86±.24 75.19±.1275.19±.12 0.8910.891 0.9820.982 0.8270.827 Cuyck. et al. [25] 70.82±.1170.82±.11 75.20±.1875.20±.18 0.8750.875 0.9920.992 0.9960.996 MicroQonv 70.62±.1370.62±.13 75.08±.3675.08±.36 1.01.0 1.01.0 1.01.0 MicroQonv+ 71.10±.2871.10±.28 75.76±.1275.76±.12 0.2360.236 1.01.0 1.01.0 We further investigate this by measuring the average per-element quantization mean-square error (MSE) for Weights (W), Activations (A), and output Gradients (G), and normalize values by MicroQonv error for readability, on Resnet-32 with CIFAR-100. While all methods have similar weight errors, the activation MSE is slightly higher for MicroQonv. Since the same element is repeated several times in the im2col-expanded tensor, it is quantized several times, inducing a slightly lower average quantization error than when quantized once, as in MicroQonv. It is interesting to note that in-line (1D) gradient quantization error [24] is lower than its square (2D) counterpart of [25] and MicroQonv, which is probably why in-line quantization has moderately better accuracy than square quantization. However, this tiny accuracy loss must be balanced against the energy gains provided by MicroQonv shown in Section V. In energy-limited scenarios, spending less energy means training for more epochs, yielding better accuracy within the same energy budget. In these cases, it is likely that MicroQonv will close the accuracy gap with the other methods thanks to significant energy savings from reduced memory movement and quantization overhead. Furthermore, if accuracy is extremely important, we propose MicroQonv+, in which activations are quantized in NVFP6 (identical to the NVFP4 format, except that elements are quantized in FP6 E3M2). As shown in Table VI, while weights and gradients remain in NVFP4, its accuracy exceeds that of the other methods and matches the FP32 baseline. Although MicroQonv+ requires ≈ 50% more memory movement for activations because it uses 6 bits per element instead of 4 bits for MicroQonv, it still requires less memory movement and storage: roughly ×5.1 less than Dacapo and ×6 less than Cuyckens et al. extrapolation, while providing better accuracy, as shown in Table I. IV-E Viability in Object Detection We train YOLOV8nano and YOLOV26nano using Ultralytics33 3 https://github.com/ultralytics/ultralytics/releases/tag/v8.4.14 setup and native hyperparameters, except batch size, which is set to 8. Both models are pretrained on the COCO dataset and finetuned for 20 epochs on the PascalVOC dataset for YOLOV8nano [26], while YOLOV26nano is finetuned on the KITTI dataset [27] for 20 epochs. To maintain a mean Average Precision (mAP50 and mAP50-95 [28]) close to the baseline, all methods use the NVFP8 representation for weights, activations, and gradients, which is the same as NVFP4 except that elements are in FP8 E4M3 format instead of FP4 E2M1. For YOLOV8nano, we keep the first layer and the last Distribution Focal Loss (DFL) convolution layer [29] at full precision. For YOLOV26nano, only the first layer is not quantized. YOLOV26nano also includes depthwise convolution layers, which are more sensitive to quantization and account for less than 1% of YOLOV26nano’s MACs. Therefore, we keep them in BF16 precision. The mean and standard deviation for mAP50 and mAP50-95 across five runs are shown in Table I for YOLOV8nano and Table IV for YOLOV26nano. We append ’-8’ to the method names to indicate the use of the NVFP8 format. For YOLOV8nano, MicroQonv-8 is within 0.3 mAP of the FP32 baseline and less than 0.15 away from the Dacapo-8 and Cuyckens et al.-8 extrapolations, while offering much better energy efficiency, as we show in the following section. We further report in Table I an experiment in which the model is trained from scratch for 100 epochs on MicroQonv and the FP32 baseline only. Here, MicroQonv-8 is within 0.2 mAP of the baseline. Regarding YOLOV26nano, microscaling quantization slightly improves the FP32 baseline by providing regularization, which prevents overfitting. MicroQonv-8 is above Dacapo-8 and Cuyckens et al.-8. We also train MicroQonv from scratch for 100epochs and compare it to the FP32 baseline. There, MicroQonv-8 is .2 mAP away from the baseline with the mAP50 metric and less than 0.8mAP away from FP32 accuracy with the mAP50-95 metric. For both YOLO models, the close distance between MicroQonv-8 accuracy and FP32 accuracy indicates that MicroQonv can enhance the efficiency of SoTA object detection models. TABLE I: Object detection performance on YOLOV8nano model and PascalVOC dataset. Method Finetuning From scratch mAP50 mAP50-95 mAP50 mAP50-95 FP32 baseline 82.06±.0682.06±.06 60.89±.0660.89±.06 76.53±.1276.53±.12 55.56±.1055.56±.10 MicroQonv-8 81.87±.1581.87±.15 60.62±.1260.62±.12 76.52±.2776.52±.27 55.39±.1955.39±.19 Dacapo-8 [24] 81.88±.1081.88±.10 60.75±.1060.75±.10 - - Cuyck. et al.-8 81.85±.0681.85±.06 60.75±.1160.75±.11 - - TABLE IV: Object detection performance on YOLOV26nano model and KITTI dataset. Method Finetuning From scratch mAP50 mAP50-95 mAP50 mAP50-95 FP32 baseline 53.91±.4053.91±.40 33.75±.2433.75±.24 74.26±1.274.26± 1.2 49.44±.7349.44±.73 MicroQonv-8 55.10±1.355.10± 1.3 34.20±.7834.20±.78 74.11±1.274.11± 1.2 48.74±.5548.74±.55 Dacapo-8 [24] 54.44±.4754.44±.47 33.56±.8933.56±.89 - - Cuyck. et al.-8 54.88±1.354.88± 1.3 33.90±.7933.90±.79 - - V DRAM Simulation To measure latency reduction and energy gains from reduced memory movement, we simulate memory movement using DRAMsim3 [14]. Cuyckens et al. store the entire model on-chip, which is realistic for their use cases where models consist of a few fully-connected layers, but not for much larger convolutional networks. We assume that we can store input and output tensors for one layer on-chip, while other layer tensors must be stored off-chip. We consider memory movement during forward and backward pass phases, as shown in Figure 4. In the forward pass, the activation tensor is computed as the output of the last layer. Therefore, it is already on-chip. Because we quantize weights during the weight-update phase to reduce memory movement, we only need to transfer the quantized weights (if a quantization method is employed, as in Cuyckens et al. extrapolation and MicroQonv, but not in Dacapo). We then quantize the activations to perform the forward convolution, and save the quantized activations in memory. During backward propagation, the gradient tensor is already on-chip due to the previous layer’s computation. The system then loads weights and activations on-chip for the associated backward operations. We exclusively consider transfers between off-chip (DRAM) and on-chip (SRAM) memories, as memory movement between off-chip and on-chip memory is much more costly than internal on-chip memory movement and computations, as shown in [30]. Because all methods use the same convolution operations, internal on-chip memory transfer and computation costs will not differ much between methods, unlike off-chip-to-on-chip transfers. V-A Experimental Setup We simulate various model and dataset configurations: ResNet32 with CIFAR-100, ResNet18 with ImageNet, YOLOV8nano with PascalVOC, and YOLOV26nano with KITTI. Batch size is 32 for CIFAR100 (image size of 32×32×332× 32× 3) and ImageNet (224×224×3224× 224× 3), and 8 for PascalVOC and KITTI (640×480×3640× 480× 3). All reported results are obtained using the LPDDR4 (Low-Power Double Data Rate) memory model from the DRAMsim3 repository44 4 https://github.com/umd-memsys/DRAMsim3/releases/tag/1.0.0. Fig. 4: Memory transfers considered for DRAM simulation. V-B Results Table V reports the tensor size transferred between off-chip memory and on-chip memory, the transfer energy spent from DRAMsim3, and the memory transfer latency from DRAMsim3, for the whole model. As predicted by the analytical model in Table I, MicroQonv shows improvements over both methods. Given the same quantization method and the same model, the gains of these three parameters are the same. We have observed this not only with LPDDR4 but also with DDR5, DDR6, and HBM (High Bandwidth Memory) memory models. For simplicity, while we refer to energy in the following discussion, the same statements apply to tensor size and memory transfer latency. The gain of MicroQonv compared to saving full-precision activations like in Dacapo is more or less constant and close to ×7.53× 7.53 as in Table I for ResNet-32 and ResNet-18, close to ×3.93× 3.93 for YOLOV8n and close to ×3.4× 3.4 for YOLOV26n. Since the activation and weight tensors are of the same dimensions in both cases, the gain comes from the quantization format. DRAM overhead varies slightly across methods because of their internal components, which explains the small variation around that gain. The improvement factor is closer to ×3.93× 3.93 for YOLOV8n because elements are quantized in FP8 instead of FP4. Improvement factor is around ×3.4× 3.4 for YOLOV26n because of depthwise convolution layers which are quantized in BF16 for both methods. For Cuyckens et al. extrapolation, however, the im2col-expanded version is saved, and its dimensions heavily depend on the stride and kernel sizes of the tensor. Roughly, the expansion factor of im2col is Kh×KwK_h× K_w/stride. Thus, a smaller stride and a larger kernel size will increase the gains of MicroQonv over im2col expansion, and vice versa. In these conditions, all ResNet32 layers have kernel sizes Kh=Kw=3K_h=K_w=3. Most layers have a stride of 1, and a few have a stride of 2, which explains the energy gain of around ×8.5 in activation size, energy, and latency. ResNet18 has 17 convolutional layers with a kernel size of 3×3, 14 with a stride of 1, and 3 with a stride of 2, which explains why the energy gain is closer to ×7× 7. In the YOLOv8n model, most layers have a stride of 1; 40% of them use a kernel size of 1×1, and it regroups most layers with large activation tensors. Other layers use a 3×3 kernel, which explains the energy gain of about ×3.5× 3.5. In the YOLOv26n model, most layers use a stride of 1 too. The percentage of layers using a kernel size of 1 increases to 53%, and a few layers with a kernel size of 3 are depthwise convolution layers, which are quantized in BF16. That is why YOLOv26n has an energy gain of about ×2.2× 2.2. Regarding the weights, as there is no im2col expansion, Cuyckens et al. and MicroQonv thus quantize the same tensor. Only the quantization pattern varies, which explains the lack of difference. This simulation does not account for quantization cost: when the kernel size is larger than 1, the im2col-expanded tensor will require quantizing roughly Kh×KwK_h× K_w/stride more elements, which in turn requires more silicon area for quantization or adds extra latency, further improving the gains of MicroQonv. TABLE V: Energy (E) in mJ and Latency (L) in ms for all transfers in Fig. 4. Tensor Size (S) is in MegaBytes (MB). Method Activations Weights R32-Cifar100 S(MB) E(mJ) L(ms) S(MB) E(mJ) L(ms) Dacapo 38.67 92.56 46.63 1.87 4.402 1.95 Cuyck. et al. 42.57 100.40 50.89 0.244 0.529 0.164 MicroQonv 5.04 11.68 6.09 0.244 0.529 0.164 R18-Imagenet S(MB) E(mJ) L(ms) S(MB) E(mJ) L(ms) Dacapo 279.44 669.68 337.91 46.72 111.9 56.37 Cuyck. et al. 271.92 643.91 324.97 6.28 14.77 7.33 MicroQonv 38.56 91.12 45.97 6.28 14.77 7.33 YV8n-PVOC S(MB) E(mJ) L(ms) S(MB) E(mJ) L(ms) Dacapo-8 492.44 1179.85 595.19 12.58 29.9 14.76 Cuyck et al. -8 441.13 1057.15 533.46 3.195 7.44 3.35 MicroQonv-8 125.22 300.03 151.36 3.195 7.44 3.35 YV26n-KITTI S(MB) E(mJ) L(ms) S(MB) E(mJ) L(ms) Dacapo-8 634.82 1520.73 767.43 10.17 23.98 11.56 Cuyck et al. -8 420.39 1007.26 507.96 2.61 5.95 2.45 MicroQonv-8 187.16 448.33 225.85 2.61 5.95 2.45 VI Training Performance We first demonstrate the relevance of MicroQonv by comparing its performance with other fully-quantized training techniques in a classic learning scenario, which yields more stable results to facilitate comparison. Experiments are conducted on image classification, then on object detection. In all tables, quantization methods use the same setup, including learning rate and number of epochs, for a fair comparison. VI-A Image classification We train ResNet-32 and ResNet-18 using the same setup as in Table I. Table VI compares final test accuracy (mean and standard deviation over five training runs) between the FP32 baseline, LUQ [23], QLR [31], and MicroQonv. 4-bit per-tensor and per-channel quantization [4] showed more than 10% accuracy degradation or convergence issue, they are therefore not displayed. In LUQ [23], activations, weights, and gradients are quantized with a per-tensor scale. In ResNet-32 and ResNet-18, all convolutions except the first one occur after a ReLU activation function: the sign bit can be replaced by a mantissa/exponent bit for activations, or simply removed, leaving activations at 3-bit precision. We report LUQ results in two cases: keeping the original paper configuration at 4 bits (UINT4) or matching the MicroQonv effective bitwidth at 3 bits (INT4, LUQ† ). In the first case, MicroQonv shows better average performance with only half as many quantization bins. In the second case, MicroQonv beats LUQ by more than 2% for ResNet-32 and by 0.8% for ResNet-18. MicroQonv also uses a unified format, leading to better accelerator efficiency per silicon area than LUQ, which requires different computation formats for forward and backward propagation. Hence, MicroQonv delivers the best precision at 4-bit quantization. Its FP4 E2M1 activation format saves 3 bits per activation pixel instead of 4 (+8/64 = 3.125 with block scale), significantly increasing the amount of data we can store. In Table VI, we also compare to QLR [31], where training is done in FP32 for weights, activations, and gradients, then activations and weights are quantized to 8 bits for inference. QLR provides slightly better accuracy, at a much higher training cost than MicroQonv. TABLE VI: Performance comparison of quantization methods on ResNet-32 (R32) with CIFAR-100 (C100) and Resnet-18 (R18) with ImageNet (Inet100). FP4E corresponds to the E3M0 format. Technique Elements Format Test acc % A W G R32-C100 R18-Inet100 Baseline FP32 FP32 FP32 71.05±.2971.05±.29 75.55±.2275.55±.22 Dacapo [24] FP4 FP4 FP4 70.86±.2470.86±.24 75.19±.1275.19±.12 Cuyck. et al. [25] FP4 FP4 FP4 70.82±.1170.82±.11 75.20±.1875.20±.18 MicroQonv FP4 FP4 FP4 70.62±.1370.62±.13 75.08±.3675.08±.36 MicroQonv+ FP6 FP4 FP4 71.10±.2871.10±.28 75.76±.1275.76±.12 QLR [31] UINT8 INT8 FP32 70.95±.4170.95±.41 75.28±.2575.28±.25 LUQ [23] UINT4 INT4 FP4E 70.54±.3570.54±.35 74.77±.2274.77±.22 LUQ† INT4 INT4 FP4E 68.32±.0868.32±.08 74.19±.1074.19±.10 VI-B Object detection YOLOV8nano and YOLOV26nano are trained reusing Table I and Table IV setup. In addition to MicroQonv, Dacapo, and Cuyckens et al., we evaluate accuracy using symmetric per-tensor and per-channel quantization. Per-tensor quantization uses one scale per tensor, mapping the tensor’s absolute maximum value to the absolute maximum of the quantized element format for weights, activations, and gradients. Per-channel quantization attributes one scale per output channel for weight and gradient tensors, and one scale per input channel for activation tensors. The absolute maximum value of a channel is mapped to the quantized element absolute maximum. All methods use FP6 E2M3 for element quantization. We append ’-6’ to the method names to indicate the use of the FP6 format. This means microscaling-based quantization methods use NVFP6 representation, identical to NVFP4 except that elements are in FP6 E2M3 format instead of FP4 E2M1. Because the input and output layers are more sensitive to quantization, we keep the input layer and every output layer in higher precision. Because the YOLOV8 model aggregates 3 levels of feature maps and has two branches for prediction, one for localization and one for prediction, it has 6 output layers plus the DFL layer kept in full precision (out of 64 convolution layers in the model). YOLOV26 further divides each branch into one-to-one (used for training only) and one-to-many (used for training and inference) subbranches, in order to get rid of Non-Maximum-Suppression (NMS) step [18]: its 12 output layers (out of 126 convolution layers in the model) remain in full precision. Following Table IV setup The depthwise convolution layers of YOLOv26nano are set to BF16 precision. Results are shown in Table VII and Table VIII. For YOLOV8nano, MicroQonv-6 is within 0.3mAP of the baseline and less than .15mAP away from Dacapo-6 and Cuyckens et al.-6. More importantly, it is significantly above Per-tensor-6 and 0.4mAP to 0.8mAP higher than Per-channel-6. (a) Test of accuracy during training on YOLOV8nano using the PascalVOC dataset. (b) Test of accuracy during training on YOLOV26nano using the KITTI dataset. Fig. 5: Test of accuracy along training of MicroQonv-6 (in blue) and FP32 baseline (in green). A similar pattern emerges from the YOLOV26nano results: MicroQonv-6 is less than 0.3mAP from the FP32 baseline in the fine-tuning scenario and less than 1.3mAP away in the training-from-scratch scenario. It shows a 0.1mAP improvement over Dacapo-6 and is less than 0.85mAP away from Cucykens et al.-6, which is above the FP32 baseline. At the same time, it is way higher than per-tensor quantization and 1.2mAP to 3mAP higher than per-channel quantization. Figure 5 shows the test-of-accuracy training-curve comparison between MicroQonv-6 and the FP32 baseline in a training-from-scratch scenario for YOLOV8nano using the PascalVOC dataset (Fig. 5(a)) and YOLOV26nano using the KITTI dataset (Fig. 5(b)). As shown in the figure, MicroQonv-6 closely follows the FP32 curve across epochs. Section V has shown that MicroQonvis significantly more energy efficient than Dacapo and Cuyckens et al., and this section shows that it also maintains accuracy close to these methods and to the FP32 baseline. Moreover, MicroQonvis more accurate and offers a much more hardware-friendly computation pattern than per-channel quantization [4]. It therefore has a better accuracy-energy efficiency tradeoff than every other quantization method. TABLE VII: 6 bit Object detection performance on YOLOV8nano-PascalVOC. Method Finetuning From scratch mAP50 mAP50-95 mAP50 mAP50-95 FP32 baseline 82.33±.0282.33±.02 61.33±.0961.33±.09 77.11±.2977.11±.29 56.25±.2256.25±.22 MicroQonv-6 82.12±.2082.12±.20 61.08±.1361.08±.13 76.95±.2676.95±.26 55.98±.2655.98±.26 Per Channel-6 81.31±.1481.31±.14 60.15±.1260.15±.12 76.58±.2076.58±.20 55.55±.1555.55±.15 Per Tensor-6 65.88±.4265.88±.42 43.89±.3543.89±.35 - - Dacapo-6 [24] 82.19±.1082.19±.10 61.14±.0661.14±.06 - - Cuyck. et al.-6 82.18±.0982.18±.09 61.20±.1361.20±.13 - - TABLE VIII: 6 bit Object detection performance on YOLOV26nano-KITTI. Method Finetuning From scratch mAP50 mAP50-95 mAP50 mAP50-95 FP32 baseline 53.91±.7353.91±.73 33.86±.6133.86±.61 74.26±1.274.26± 1.2 49.44±.7349.44±.73 MicroQonv-6 54.11±.9354.11±.93 33.51±.5133.51±.51 73.02±.6273.02±.62 48.18±.5448.18±.54 Per Channel-6 51.22±.6451.22±.64 31.49±.3931.49±.39 71.96±1.171.96± 1.1 46.99±.8146.99±.81 Per Tensor-6 9.1±.769.1±.76 4.09±.424.09±.42 - - Dacapo-6 [24] 54.08±.1154.08±.11 33.36±.6333.36±.63 - - Cuyck. et al.-6 54.93±.6554.93±.65 34.02±.4634.02±.46 - - VII MicroQonv in a Continual Learning Setting Continual learning (CL) [2] aims to train AI models continually to adapt to changing data distributions. Because the complete history of data seen by the model is too large to be stored entirely, especially for on-device models, and because backpropagation optimizes only on currently available data, catastrophic forgetting occurs. Thus, the field aims to mitigate catastrophic forgetting. We focus on the Class Incremental Learning (CIL) scenario [32], where a model is trained sequentially on a series of tasks, each introducing new classes not seen in previous tasks. In this section, we present results in a (CL) setting that relies on latent replay [33]. VII-A Related Work Among existing CL strategies, the replay buffer [3] is one of the simplest and most effective ways to mitigate catastrophic forgetting. At the end of each task, a subset of the training data is selected and stored in the replay buffer memory. The replay buffer is then combined with data from subsequent tasks, allowing the model to learn new classes while mitigating catastrophic forgetting of previously seen classes. It can be used alone or in conjunction with other strategies to ensure optimal performance [34, 35]. The latent replay buffer [33] (LR), as shown in Figure 6, is a variant of the replay buffer: early layers of the feature extractor are frozen, and the output activations of the frozen feature extractor, rather than the original images, are stored in the replay buffer. This significantly reduces backpropagation computational cost. Moreover, early layers tend to learn generic features from large datasets [36], limiting accuracy degradation compared to whole-model retraining. Compressing the replay buffer items allows for storing more elements for the same memory budget, without LR [37, 38, 39] or with LR [40]. However, full-precision models used in these papers make these methods hard to adapt to energy-constrained devices. Few papers study the interaction between quantization and the latent replay buffer. Input activations from a convolutional or linear layer can be quantized and stored in memory to further reduce memory storage. LifeLearner [35] combines 8-bit quantization with a meta-learned feature extractor and product quantization. However, they require pre-training on the same data distribution as the test data, making them sensitive to distribution shifts. Quantized Latent Replay (QLR) [31] combines LR with integer 8-bit quantization for weights and activations during inference, while training is done in FP32. They use no compression technique other than quantization, which gives them good adaptability to unseen distributions. We seek to improve these results by using the ad hoc microscaling format proposed by MicroQonv. Fig. 6: Quantized Latent Replay (QLR) strategy overview. Feature extractor layers are also quantized. VII-B Experimental Results To run our experiments, we combine the PyTorch-based PyCIL library [41] with quantization methods, and we use the PyCIL hyperparameters for all methods. A ResNet18 model pretrained on ImageNet is quantized and full-precision accuracy retrieved through fine-tuning with Quantization-Aware Training [4]: activations and weights are quantized in the tested formats, while gradients remain in full-precision. Each quantization method is used with the Hindsight technique [42], which estimates tensor statistics, such as the maximum and mean, from previous batches. This helps load each tensor only once, reducing memory movement. All methods quantize the first and last layers to 8 bits for weights, gradients, and activations, as a common practice to mitigate accuracy degradation. Only convolutional layers without parallel residual connections can be treated as LR layers. For our experiments, we select the first convolutional layer of blocks 3.1, 4.0, and 4.1. We select samples for the replay buffer using the herding mechanism [43], a widely used strategy. Figure 7 shows the test accuracy after training on the last task, in the CIL scenario on CIFAR-100. MicroQonv accuracy is systematically and significantly above QLR across a range of layers and replay buffer sizes. Since 3.125 bits per activation pixel are saved instead of 8 bits, approximately ×2.5 more samples can be stored, which explains the accuracy delta. This would not be possible with other methods: Dacapo would save them using 32 bits per activation pixel, while Cuyckens et al. extrapolation would save the expanded im2col tensor, significantly reducing the number of samples that can be stored given a memory size. As LR at layer 4.1 shows the best performance, we keep this configuration for the rest of the experiments. Fig. 7: End-test accuracy vs replay buffer memory size. We now test MicroQonv on the CIFAR-100, CORE50 [44], and CUB-200 datasets. Each task has 10 classes for CIFAR-100, 5 for CORE50, and 40 for CUB-200. Table IX lists the results at various memory sizes. MicroQonv accuracy is always way above QLR, with test accuracy difference between +5.7% and +11% for CIFAR-100, +4.8% and +9.4% on CORE50, and +9.6% and +16.7% for CUB-200. The accuracy gap tends to be higher when there are fewer samples. This aligns with neural network scaling laws [45]: doubling the dataset size yields less accuracy improvement when the dataset is large than when it is small. We perform a final test to compare quantization performance on the CIFAR-100 dataset with a fixed memory budget of 2000 samples for all methods. The FP32 baseline achieves an accuracy of 48.83±.36%48.83±.36\% while QLR attains 48.77±.23%48.77±.23\% and MicroQonv reaches 48.16±.66%48.16±.66\%, less than 1% from baseline. TABLE IX: LR end test accuracy of ResNet18 for various LR memory sizes in MB and datasets. CIFAR-100 25MB 50MB 75MB 125MB QLR[31] 39.3±.739.3±.7 49.1±.749.1±.7 54.2±.554.2±.5 60.1±.760.1±.7 MicroQonv 50.3±.550.3±.5 59.0±.559.0±.5 62.4±.762.4±.7 65.8±.565.8±.5 CORE50 15MB 25MB 50MB 75MB QLR 59.1±.559.1±.5 64.0±.564.0±.5 70.7±.870.7±.8 73.2±1.173.2± 1.1 MicroQonv 68.5±1.068.5± 1.0 72.2±.472.2±.4 75.5±.575.5±.5 78.0±.878.0±.8 CUB200 10MB 20MB 30MB 40MB QLR 26.6±.626.6±.6 39.9.±.539.9.±.5 43.5±.743.5±.7 50.2±.350.2±.3 MicroQonv 43.3±.743.3±.7 52.8±.352.8±.3 57.1±.557.1±.5 59.8±.359.8±.3 VIII Conclusion MicroQonv uses in-memory tensor reshaping for convolution kernels to simplify microscaling quantization and reduce quantization overhead by a factor of ×2× 2 for weights and gradients, and up to ×9× 9 for activations. By enabling the storage of a quantized activation tensor instead of its im2col-expanded version, it reduces activation memory movement and storage by up to ×7.53× 7.53 under standard convolution parameters. Validation with a DRAM simulator shows significant memory and energy gains on various models, including state-of-the-art object detection models: ×3.5× 3.5 for YOLOV8nano and ×2.2× 2.2 for YOLOV26nano. When applied to CL with an LR strategy, 4-bit MicroQonv activations reduce memory usage and improve accuracy by +5.7% to +11% for the same memory budget. Future work includes building a hardware accelerator that leverages MicroQonv memory layout and studying the relationships between other lossy compression techniques (e.g., vector quantization) and microscaling. We will release all code used in this work as open source upon publication. Acknowledgment This work was supported by the PEPR IA - HOLIGRAIL project of the Agence Nationale de la Recherche, ANR-23-PEIA-0010. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://w.grid5000.fr) References [1] E. Cereda, A. Giusti, and D. Palossi (2024) Training on the Fly: On-Device Self-Supervised Learning Aboard Nano-Drones Within 20 mW. IEEE Trans. Comp.-Aided Des. Integ. Cir. Sys. 43 (11), p. 3685–3695. Cited by: §I. [2] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A Comprehensive Survey of Continual Learning: Theory, Method and Application . IEEE Trans. on Pattern Analysis & Machine Intelligence 46 (08), p. 5362–5383. External Links: ISSN 1939-3539 Cited by: §I, §VII. [3] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne (2019) Experience Replay for Continual Learning. Advances in Neural Information Processing Systems (NeurIPS) 32. Cited by: §I, §VII-A. [4] M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen, and T. Blankevoort (2021) A White Paper on Neural Network Quantization. arXiv:2106.08295. Cited by: §I, §VI-A, §VI-B, §VII-B. [5] A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer (2024) AI and Memory Wall. IEEE Micro 44 (3), p. 33–39. Cited by: §I, §I. [6] S. Ben Ali, S. Filip, and O. Sentieys (2024) A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training. In 27th IEEE/ACM Design, Automation and Test in Europe (DATE), p. 1–6. Cited by: §I. [7] L. Huang, Y. Liu, X. Lin, C. Wei, W. Sun, Z. Wang, B. Cao, C. Zhang, X. Fu, W. Zhao, and S. Zhang (2025) MPICC: Multiple-Precision Inter-Combined MAC Unit with Stochastic Rounding for Ultra-Low-Precision Training. In 30th Asia and South Pacific Design Automation Conference (ASPDAC), p. 554–559. External Links: ISBN 9798400706356 Cited by: §I. [8] B. D. Rouhani, R. Zhao, V. Elango, R. Shafipour, M. Hall, M. Mesmakhosroshahi, A. More, L. Melnick, M. Golub, G. Varatkar, L. Shao, G. Kolhe, D. Melts, J. Klar, R. L’Heureux, M. Perry, D. Burger, E. S. Chung, Z. Deng, S. Naghshineh, J. Park, and M. Naumov (2023) With Shared Microexponents, A Little Shifting Goes a Long Way. 50th Annual International Symposium on Computer Architecture (ISCA). Cited by: §I, §I-D. [9] B. D. Rouhani, R. Zhao, A. More, et al. (2023) Microscaling Data Formats for Deep Learning.. arXiv:2310.10537. External Links: 2310.10537 Cited by: §I, §I-D, §I-D. [10] B. Chmiel, M. Fishman, R. Banner, and D. Soudry (2025) FP4 All the Way: Fully Quantized Training of LLMs. arXiv:2505.19115. External Links: 2505.19115 Cited by: §I, §I-D. [11] F. Abecassis A. Agrusa et al. (2025) Pretraining Large Language Models with NVFP4. arXiv:2509.25149. External Links: 2509.25149 Cited by: §I, §I-D. [12] M. Dukhan (2020) Indirect Deconvolution Algorithm. In IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), p. 922–926. Cited by: §I, §I-C. [13] Y. Zhou, M. Yang, C. Guo, J. Leng, Y. Liang, Q. Chen, M. Guo, and Y. Zhu (2021) Characterizing and Demystifying the Implicit Convolution Algorithm on Commercial Matrix-Multiplication Accelerators. In IEEE International Symposium on Workload Characterization (IISWC), p. 214–225. External Links: ISBN 978-1-6654-4173-5 Cited by: §I, §I-C4, §I-C. [14] S. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob (2020) DRAMsim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator . IEEE Computer Architecture Letters 19 (02), p. 106–109. External Links: ISSN 1556-6064 Cited by: §I, §V. [15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), Cited by: §I-A. [16] S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. Kweon, and S. Xie (2023) ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16133–16142. Cited by: §I-A. [17] P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan (2023) MobileOne: An Improved One millisecond Mobile Backbone. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7907–7917. Cited by: §I-A. [18] G. Jocher, J. Qiu, M. Liu, S. Lyu, F. C. Akyon, and M. E. Kalfaoglu (2026) Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models. arXiv:2606.03748 [cs.CV]. External Links: 2606.03748 Cited by: §I-A, §VI-B. [19] I. Robinson, P. Robicheaux, M. Popov, D. Ramanan, and N. Peri (2026) RF-DETR: Real-Time Detection Transformer. In International Conference on Learning Representations (ICLR), Cited by: §I-A. [20] Z. Liao, Y. Zhao, X. Shan, Y. Yan, C. Liu, L. Lu, X. Ji, and J. Chen (2025) RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models. arXiv: 2510.25257 [cs.CV]. External Links: 2510.25257 Cited by: §I-A. [21] J. Fornt, P. Fontova-Musté, M. Caro, J. Abella, F. Moll, J. Altet, and C. Studer (2023) An Energy-Efficient GeMM-Based Convolution Accelerator With On-the-Fly im2col. IEEE Trans. Very Large Scale Integr. Syst. 31 (11), p. 1874–1878. External Links: ISSN 1063-8210 Cited by: §I-C4. [22] M. Cho and D. Brand (2017) MEC: memory-efficient convolution for deep neural network. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 815–824. Cited by: §I-C4. [23] B. Chmiel, R. Banner, E. Hoffer, H. Ben Yaacov, and D. Soudry (2023) Accurate Neural Training with 4-bit matrix multiplications at standard formats. In 11th International Conference on Learning Representations (ICLR), Cited by: §I-A, §VI-A, TABLE VI. [24] Y. Kim, C. Oh, J. Hwang, W. Kim, S. Oh, Y. Lee, H. Sharma, A. Yazdanbakhsh, and J. Park (2024) DaCapo: Accelerating Continuous Learning in Autonomous Systems for Video Analytics. In ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), p. 1246–1261. Cited by: §I-C, §IV-B, §IV-C, §IV-D, §IV-D, TABLE I, TABLE I, TABLE I, TABLE IV, TABLE VI, TABLE VII, TABLE VIII. [25] S. Cuyckens, X. Yi, N. S. Murthy, C. Fang, and M. Verhelst (2025) Efficient Precision-Scalable Hardware for Microscaling (MX) Processing in Robotics Learning. In IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), Cited by: §I-C, §IV-B, §IV-B, §IV-C, §IV-D, TABLE I, TABLE I, TABLE VI. [26] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman (2010) The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88 (2), p. 303–338. Cited by: §IV-E. [27] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun (2013) Vision meets Robotics: The KITTI Dataset. International Journal of Robotics Research (IJRR). Cited by: §IV-E. [28] P. Henderson and V. Ferrari (2016) End-to-End Training of Object Class Detectors for Mean Average Precision. In Computer Vision – ACCV, p. 198–213. Cited by: §IV-E. [29] X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, and J. Yang (2020) Generalized focal loss: learning qualified and distributed bounding boxes for dense object detection. In 34th International Conference on Neural Information Processing Systems (NeurIPS), External Links: ISBN 9781713829546 Cited by: §IV-E. [30] M. Horowitz (2014) Computing’s energy problem (and what we can do about it). In IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), Vol. , p. 10–14. Cited by: §V. [31] L. Ravaglia, M. Rusci, D. Nadalini, A. Capotondi, F. Conti, and L. Benini (2021) A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 11 (4), p. 789–802. External Links: ISSN 2156-3357, 2156-3365 Cited by: §VI-A, TABLE VI, §VII-A, TABLE IX. [32] G. M. van de Ven, T. Tuytelaars, and A. S. Tolias (2022) Three types of incremental learning. Nature Machine Intelligence 4 (12), p. 1185–1197. External Links: ISSN 2522-5839 Cited by: §VII. [33] L. Pellegrini, G. Graffieti, V. Lomonaco, and D. Maltoni (2020) Latent Replay for Real-Time Continual Learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 10203–10209. Cited by: §VII-A, §VII. [34] D. Zhou, Q. Wang, H. Ye, and D. Zhan (2023) A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. In International Conference on Learning Representations (ICLR), Cited by: §VII-A. [35] Y. D. Kwon, J. Chauhan, H. Jia, S. I. Venieris, and C. Mascolo (2024) LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms. In 21st ACM Conference on Embedded Networked Sensor Systems, SenSys’23, p. 138–151. Cited by: §VII-A, §VII-A. [36] I. Miro-Panades V. Lorrain et al. (2024) A 772uJ/frame ImageNet Feature Extractor Accelerator on HD Images at 30FPS. In IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), p. 25–29. External Links: ISBN 979-8-3503-7877-1 Cited by: §VII-A. [37] L. Wang, X. Zhang, K. Yang, L. Yu, C. Li, H. Lanqing, S. Zhang, Z. Li, Y. Zhong, and J. Zhu (2021) Memory Replay with Data Compression for Continual Learning. In International Conference on Learning Representations (ICLR), Cited by: §VII-A. [38] J. Yang, Z. Duan, A. Peng, Y. Huang, J. He, and F. Zhu (2024) Probing Image Compression For Class-Incremental Learning. arXiv:2403.06288. External Links: 2403.06288 Cited by: §VII-A. [39] Z. Luo, Y. Liu, B. Schiele, and Q. Sun (2023) Class-Incremental Exemplar Compression for Class-Incremental Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11371–11380. Cited by: §VII-A. [40] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, and C. Kanan (2020) REMIND Your Neural Network to Prevent Catastrophic Forgetting. In 16th European Conference (ECCV), p. 466–483. External Links: ISBN 978-3-030-58597-6 Cited by: §VII-A. [41] D. Zhou, F. Wang, H. Ye, and D. Zhan (2023) PyCIL: a Python toolbox for class-incremental learning. Science China Information Sciences 66 (9). External Links: ISSN 1674-733X, 1869-1919 Cited by: §VII-B. [42] M. Fournarakis and M. Nagel (2021) In-Hindsight Quantization Range Estimation for Quantized Training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, p. 3063–3070. Cited by: §VII-B. [43] S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert (2016) iCaRL: Incremental Classifier and Representation Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 5533–5542. Cited by: §VII-B. [44] V. Lomonaco and D. Maltoni (2017) CORe50: a New Dataset and Benchmark for Continuous Object Recognition. In 1st Annual Conference on Robot Learning, Proceedings of Machine Learning Research (PMLR), Vol. 78, p. 17–26. Cited by: §VII-B. [45] B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos (2022) Beyond neural scaling laws: beating power law scaling via data pruning. In 36th International Conference on Neural Information Processing Systems (NeurIPS), External Links: ISBN 9781713871088 Cited by: §VII-B.