Paper deep dive
DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNs
Daewon Chae, Hyunwon Chung, Changwoo Lee, Hun-Seok Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale foundation models achieve strong performance across diverse tasks, but their size makes inference costly, largely due to dense matrix multiplications. Prior work reduces this cost by replacing dense weight matrices with efficient structured forms such as low-rank factorizations. However, these methods approximate weights rather than the output activations that determine inference accuracy. Consequently, small weight-space errors can be amplified by input activations, producing large output errors. In this work, we propose DyRA, an input-adaptive method that improves structured matrix multiplication approximation by correcting residual output errors during inference. We show that matrix multiplication can be approximated more effectively by directly optimizing low-rank factors of the output. DyRA builds on this insight by dynamically approximating and correcting the output error introduced by structured weight approximations. This combines efficient structured computation with input-dependent correction, yielding a more faithful approximation of full matrix multiplication under the same computational budget. Across vision, speech, and language models, DyRA consistently improves the accuracy-efficiency trade-off over structured weight approximations alone. Notably, DyRA achieves a 1.5$\times$ end-to-end GPU speedup for DINOv3 while reducing accuracy degradation by more than 3$\times$ relative to weight-only baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.02882v1
- Canonical: https://arxiv.org/abs/2610.02882v1
Trouble viewing inline? Open PDF directly →
Full Text
68,065 characters extracted from source content.
Expand or collapse full text
DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNs Daewon Chae Affiliation: Department of EECS, University of Michigan Email: daewon@umich.edu Hyunwon Chung Affiliation: Department of EECS, University of Michigan Email: hyunwon@umich.edu Changwoo Lee †thanks: Contributed to this work while at the University of Michigan. Affiliation: Qualcomm AI Research Email: hunseok@umich.edu Hun-Seok Kim Affiliation: Department of EECS, University of Michigan Email: changle@qti.qualcomm.com Abstract Large-scale foundation models achieve strong performance across diverse tasks, but their size makes inference costly, largely due to dense matrix multiplications. Prior work reduces this cost by replacing dense weight matrices with efficient structured forms such as low-rank factorizations. However, these methods approximate weights rather than the output activations that determine inference accuracy. Consequently, small weight-space errors can be amplified by input activations, producing large output errors. In this work, we propose DyRA, an input-adaptive method that improves structured matrix multiplication approximation by correcting residual output errors during inference. We show that matrix multiplication can be approximated more effectively by directly optimizing low-rank factors of the output. DyRA builds on this insight by dynamically approximating and correcting the output error introduced by structured weight approximations. This combines efficient structured computation with input-dependent correction, yielding a more faithful approximation of full matrix multiplication under the same computational budget. Across vision, speech, and language models, DyRA consistently improves the accuracy–efficiency trade-off over structured weight approximations alone. Notably, DyRA achieves a 1.5×1.5× end-to-end GPU speedup for DINOv3 while reducing accuracy degradation by more than 3×3× relative to weight-only baselines. Our code is available at https://github.com/daewon88/DyRA. 1 Introduction Recent advances in deep learning have achieved state-of-the-art performance across diverse tasks, but at substantial computational cost. Dense matrix multiplications account for much of the computation in modern deep networks11 1 For example, in ViT-L/16 [14], linear layers account for approximately 96.7% of total FLOPs., motivating work on accelerating them by replacing dense weight matrices with efficient structured forms [22, 11, 21, 31, 5, 24]. Low-rank approximation is a representative example, while block-structured variants such as Monarch [11] and BLAST [22] offer more expressive alternatives. Given a pre-trained model, these methods approximate its weight matrices via closed-form decompositions or iterative optimization, enabling faster inference with reduced FLOPs. Despite their effectiveness, these methods are fundamentally weight-centric: they optimize approximations in weight space without explicitly accounting for the input activations seen at inference time. However, the actual computation is determined by the realized product between activations and weights, not by the weights alone. Thus, even small weight-space residuals can be amplified by the input, causing large output activation errors [40]. We illustrate this limitation on DINOv3 [35], a large-scale vision foundation model. As shown in Figure 1, replacing all dense linear layers with structured matrices such as low-rank, Monarch, and BLAST makes PCA feature visualizations noticeably noisier and less semantically organized than those of the original model. This demonstrates that approximating weights alone may fail to preserve model behavior. To address this limitation, we propose DyRA (Dynamic Residual Approximation), an input-adaptive method that improves structured matrix multiplication by correcting output errors on the fly. DyRA augments a static weight approximation with a dynamic correction adapted to the input activations. This correction is computed through nested optimization during inference to minimize residual output errors induced by the approximated weight matrix. Rather than evaluating the exact residual through an expensive dense multiplication, DyRA uses low-rank factors to approximate the residual directly in the output space, not the weight space. These factors are efficiently computed at inference time using alternating least squares, without explicitly forming the dense residual product. Consequently, DyRA provides input-adaptive output correction that more faithfully approximates the original matrix multiplication under a fixed computational budget. Figure 1 shows that this correction substantially preserves the semantic structure of DINOv3 features compared with weight-only approximations. We validate DyRA as a test-time model approximation and acceleration technique across pre-trained vision, language, and speech models. Combined with structured matrices such as low-rank, Monarch, and BLAST, DyRA consistently improves the accuracy–efficiency trade-off over weight-only baselines. For example, on semantic segmentation with DINOv3 at a 60% FLOPs reduction, DyRA limits the mIoU drop to 1.7 points, whereas the weight-only baseline drops by 54.1 points. These gains translate to practical runtime improvements: on DINOv3, it achieves a 1.52×1.52× GPU speedup while incurring only 1/3 of the accuracy drop of the weight-only baseline at the same speedup. Figure 1: DINOv3 feature visualizations under a 60% FLOPs reduction. After replacing all dense linear layers with structured weights, we visualize the resulting features using the DINO PCA protocol, mapping the first three principal components to RGB. Weight-only approximations produce noisy, semantically degraded features, while DyRA preserves structures close to the original model. 2 Proposed Method 2.1 Overview Starting from a pre-trained weight matrix W, matrix approximation or model compression methods approximate the weight with a computationally efficient structured form W (e.g., low-rank or block-structured approximation). The original weight matrix can be written as =^+res, W= W+ W_res, (1) where res=− W_res= W- W denotes the residual. Accordingly, the exact matrix product can be written as =^+res X W= X W+ X W_res (2) given input activation X. Conventional approximation methods use only the first term and approximate the full product as ≈ X W≈ X W. The approximation error is thus given by res X W_res, rather than the residual weight matrix res W_res alone. This implies that, even when W is chosen to minimize ‖res‖\| W_res\|, the resulting error remains input-dependent and may still be large in the output space due to its interaction with the input X. To address this limitation, DyRA explicitly accounts for the input-dependent residual product res X W_res by approximating it at inference time. This augments the static approximation X W with a dynamic correction, yielding a more faithful estimate of X W within a fixed computational budget. In the following, we introduce the dynamic residual approximation that enables this correction efficiently, and analyze its key properties. 2.2 Dynamic Residual Approximation Given an input matrix ∈ℝn×c1 X ^n× c_1 and a residual weight matrix res∈ℝc1×c2 W_res ^c_1× c_2, our goal is to efficiently approximate the residual product res X W_res at inference time without performing the full matrix multiplication. To this end, we formulate the residual output computation as an optimization problem over low-rank factors. Specifically, we seek a low-rank approximation res≈ X W_res≈ L R, where ∈ℝn×r L ^n× r and ∈ℝr×c2 R ^r× c_2 with r≪min(n,c2)r (n,c_2), by solving min∈ℝn×r,∈ℝr×c2‖res−‖F2. _ L ^n× r,\, R ^r× c_2 \| X W_res- L R \|_F^2. (3) This objective directly approximates the realized residual output res X W_res, rather than the residual weight matrix alone. Its optimal rank-r solution is given by the truncated singular value decomposition (SVD) of res X W_res [15]. However, computing this solution requires explicitly forming the full product res X W_res, which defeats the purpose of reducing computational cost. We therefore directly optimize the low-rank factors L and R without explicitly forming res X W_res. Specifically, we optimize the objective in Eq. 3 using alternating least squares (ALS). Starting from an initial right factor (0) R^(0), ALS alternates between least-squares subproblems obtained by fixing one factor and optimizing the other. This yields the following closed-form updates at iteration t: 22 2 In practice, we use a ridge-regularized version of these updates by adding a small λ≥0λ≥ 0 to the Gram matrices, i.e., (t)(t)⊤+λr R^(t) R^(t) +λ I_r and (t+1)⊤(t+1)+λr L^(t+1) L^(t+1)+λ I_r, for numerical stability. (t+1) L^(t+1) =(res(t)⊤)((t)(t)⊤)−1, = X ( W_res R^(t) ) ( R^(t) R^(t) )^-1, (4) (t+1) R^(t+1) =((t+1)⊤(t+1))−1((t+1)⊤)res. = ( L^(t+1) L^(t+1) )^-1 ( L^(t+1) X ) W_res. (5) A key efficiency advantage of this ALS formulation is that each update is performed through low-rank (r) intermediate computations. In the L update, res W_res is first projected onto the rank-r right factor, yielding res(t)⊤∈ℝc1×r W_res R^(t) ^c_1× r. This reduces the output dimension from c2c_2 to r, so that X is multiplied only with a slim c1×rc_1× r matrix. Similarly, in the R update, the input matrix is first projected onto the rank-r left factor as (t+1)⊤∈ℝr×c1 L^(t+1) X ^r× c_1, reducing the input side from n to r before being multiplied by res W_res. As a result, the residual output is estimated efficiently as ≈res L R≈ X W_res using low-dimensional intermediate products, without explicitly forming the full residual output. Figure 2: Convergence of ALS-based residual product approximation across ViT layers. Relative squared Frobenius error is measured for attention projections and MLP layers with target rank r=16r=16. Convergence of ALS Residual Approximation We empirically examine the convergence of the ALS-based residual product approximation using a Vision Transformer (ViT) [14] evaluated with ImageNet [13] samples. We apply ALS to approximate the residual product res X W_res at multiple linear layers of the model with a small target rank r=16r=16, initializing (0) R^(0) from a Gaussian distribution. In Figure 2, we report the relative squared Frobenius error across ALS steps for both attention projections and multi-layer perceptron (MLP) layers. Across layers, the approximation rapidly approaches the optimal rank-r solution within only one or two iterations. These results indicate that only a single update per factor is sufficient to effectively approximate the residual product, demonstrating the efficiency of ALS for the dynamic residual approximation. Further details on the experimental setup, along with additional results across various layers, are provided in Appendix E.3. 2.3 Analysis of DyRA with One-Step ALS Residual Approximation Given the rapid convergence observed above, we use only one ALS iteration as the default configuration of DyRA throughout this work.33 3 While more ALS iterations can further refine the approximation, one iteration captures most of the gain. Thus, under a fixed computation budget, we use a single iteration and allocate the saved computation to using a larger rank r. The resulting approximation is ≈^+ALS1(res)=^+(1)(1), X W≈ X W+ALS_1\! ( X W_res )= X W+ L^(1) R^(1), (6) where ALS1(⋅)ALS_1(·) denotes one ALS iteration, which updates L once and R once. Notably, the left-factor update admits an efficient implementation: (1)=(res(0)⊤)((0)(0)⊤)−1⏟:precomputable L^(1)= X ( W_res R^(0) ) ( R^(0) R^(0) )^-1_ P_ L\;:\;precomputable (7) where P_ L is precomputed offline as it depends only on res W_res and the fixed initialization (0) R^(0). Thus, computing (1) L^(1) at inference time requires only multiplying X by a slim c1×rc_1× r matrix P_ L, reducing the online cost of ALS1ALS_1 largely to the (1) R^(1) update. Subspace Comparison under the Same Rank Increase We now analyze this one-step variant to clarify what DyRA contributes beyond static approximation. While DyRA can be combined with any structured weight W such as Monarch [11] and BLAST [22], we focus on the low-rank case for analytical clarity. We first compare DyRA with simply increasing the rank of a static low-rank approximation. We then quantify the additional cost of DyRA and use it to construct a FLOPs-matched static baseline. Specifically, we compare DyRA at correction rank r with a static (weight-only) low-rank approximation whose rank is increased by the same amount, from k to k+rk+r. For DyRA, we take the rank-k truncated SVD of W as the static approximation and apply rank-r DyRA correction. The resulting DyRA is compared with the static rank-(k+r)(k+r) truncated SVD approximation of the weight. The following proposition characterizes this difference. Proposition 1. Let =⊤ W= U V be the SVD of W, and let ^=1:k1:k1:k⊤ W= U_1:k _1:k V_1:k be its rank-k truncated SVD approximation. Assume DyRA approximates res X W_res with rank r by one ALS iteration, initialized with (0)=k+1:k+r⊤ R^(0)= V_k+1:k+r . Then the resulting approximation can be written as X W ≈[1:k1:k+1:k+rk+1:k+r][1:k⊤(1)], ≈ X bmatrix U_1:k _1:k& U_k+1:k+r _k+1:k+r bmatrix bmatrix V_1:k \\ R^(1) bmatrix, (8) where (1)=(rres⊤⊤rres)−1rres⊤⊤res R^(1)=( A_r^res X X A_r^res)^-1 A_r^res X X W_res and rres=k+1:k+rk+1:k+r A_r^res= U_k+1:k+r _k+1:k+r. The proposition shows that DyRA uses the same additional left directions as the static rank-(k+r)(k+r) SVD approximation, k+1:k+rk+1:k+r U_k+1:k+r _k+1:k+r. The difference lies in how these directions are paired on the right side. A static rank-(k+r)(k+r) approximation uses the fixed weight-only basis k+1:k+r⊤ V_k+1:k+r . In contrast, DyRA replaces this basis with (1) R^(1), which is optimized for the current input through ⊤ X X. Thus, DyRA is not merely increasing the static rank; it constructs an input-adaptive residual subspace tailored to the realized product res X W_res. This distinction becomes clearer in the isotropic case ⊤= X X= I. In this case, (1)=k+1:k+r⊤ R^(1)= V_k+1:k+r , so DyRA reduces to the static rank-(k+r)(k+r) approximation. When ⊤≠ X X≠ I, however, the current input reweights feature directions, and (1) R^(1) generally differs from the weight-only basis. This is precisely where DyRA gains input adaptivity. The proof is provided in Appendix C. Computational Cost We now quantify the computational overhead of constructing the input-adaptive subspace. Let ∈ℝn×c1 X ^n× c_1, res∈ℝc1×c2 W_res ^c_1× c_2, and let r denote the rank of the dynamic residual correction. As shown in Eq. 7, the left-factor update multiplies X by a slim c1×rc_1× r matrix, costing O(nc1r)O(nc_1r). The right-factor update projects the residual product onto the r-dimensional subspace defined by (1) L^(1), with dominant cost O(nc1r+c1c2r)O(nc_1r+c_1c_2r). Including the final multiplication (1)(1) L^(1) R^(1), the dynamic residual correction costs O(nc1r+c1c2r+nc2r)O(nc_1r+c_1c_2r+nc_2r). Combined with the O(nk(c1+c2))O(nk(c_1+c_2)) cost of the rank-k static approximation, DyRA totals O(nk(c1+c2)+r(n(c1+c2)+c1c2))O (nk(c_1+c_2)+r (n(c_1+c_2)+c_1c_2 ) ). Since r is typically much smaller than c1c_1 and c2c_2, this remains substantially cheaper than the dense product X W, which costs O(nc1c2)O(nc_1c_2). Compared with simply increasing the static rank to k+rk+r, DyRA incurs an extra O(c1c2r)O(c_1c_2r) term, which is the cost of constructing the input-adaptive subspace. Figure 3: Remaining normalized Frobenius error of DyRA and a static low-rank baseline under a matched FLOPs budget. The error is measured between the original dense output and the approximated output across ViT layers. FLOPs-Matched Comparison Given this overhead, we examine whether DyRA’s gains persist under a matched computational budget. To this end, we compare DyRA against a FLOPs-matched static low-rank weight approximation baseline on ViT. Starting from a rank-k approximation of W, DyRA applies a rank-r dynamic correction to reduce the remaining output error. For the static baseline, we instead increase the rank from k to k+r+r′k+r+r , where the additional rank r′r accounts for the O(c1c2r)O(c_1c_2r) overhead of the dynamic correction, so that both methods use the same FLOPs budget 44 4 The matching is done at the level of exact FLOPs counts rather than asymptotic order, so r′r also absorbs the constant and lower-order terms omitted in the O(⋅)O(·) notation.. Figure 3 shows the remaining normalized Frobenius error across different ViT layers. Under this FLOPs-matched setting, DyRA consistently achieves lower error than the static low-rank baseline. These results show that, under the same computational budget, input-adaptive residual subspaces approximate the output more faithfully than higher-rank static approximations. Additional results for the remaining attention and MLP layers are provided in Appendix E.4. 2.4 Implementation of DyRA To minimize GPU memory traffic, we implement DyRA with custom Triton [36] kernels. Rather than materializing the reconstructed residual L R in HBM, we fuse the low-rank reconstruction with the output merge: given ∈ℝB×T×r L ^B× T× r and ∈ℝB×r×N R ^B× r× N, the kernel accumulates αα L R on-chip and directly writes base+α Y_base+α L R to the output tensor. For linear layers, we further fuse the base projection, bias addition, and residual correction, computing =tmp⊤++α Y= X_tmp A + b+α L R within a single Triton kernel. This eliminates intermediate tensors and reduces HBM accesses. The ALS update requires solving an r×r× r symmetric positive definite system for each batch instance. For small r, we use a Triton Cholesky solver, keeping the factorization and triangular solves in on-chip cache/registers. For a larger r, Cholesky becomes too sequential to fully utilize GPU parallelism, so we use a Newton–Schulz iterative solver. This replaces dependency-heavy factorization with matrix-multiplication-based updates, improving the GPU’s parallelism and tensor-core utilization. 3 Experimental Results Figure 4: ImageNet classification accuracy at different FLOPs reduction levels for ViT-L. DyRA consistently improves the accuracy–efficiency trade-off across low-rank, Monarch, and BLAST. We validate DyRA by integrating it with several structured weight approximations, including low-rank, Monarch [11], and BLAST [22]. We first evaluate DyRA as a test-time model approximation/acceleration technique applied to pre-trained models (without any fine-tuning) across vision, language, and speech tasks, demonstrating consistent improvements in downstream performance under matched computational budgets (Section 3.1). We then measure end-to-end latency to assess whether these computational savings translate into practical speedups on a GPU (Section 3.2). Finally, we study settings where additional adaptation is available, showing that DyRA can be combined with post-training calibration or fine-tuning to further improve performance (Section 3.3). All experimental details are provided in Appendix D. 3.1 Test-Time Model Approximation and Acceleration Across Domains Image Classification with ViT We evaluate DyRA on ImageNet [13] classification using a pre-trained ViT-L model [14]. We apply structured weight approximations to all dense linear layers in both the attention and MLP blocks, and measure top-1 accuracy under different FLOPs budgets. For DyRA, we fix the dynamic residual rank to r=8r=8 and vary the budget of the static weight approximation to control the overall FLOPs. As shown in Figure 4, DyRA consistently improves the accuracy–efficiency trade-off across all structured baselines. In particular, at around 55% FLOPs reduction, the low-rank and BLAST baselines incur accuracy drops of 66.1% and 17.4%, respectively, whereas combining them with DyRA reduces the drops to 2.4% and 1.6%. Dense Prediction Tasks with DINO Beyond image-level classification, we further show that DyRA preserves fine-grained representations required for dense prediction. To this end, we evaluate semantic segmentation on ADE20K [42] and depth estimation on NYUv2 [34] using DINOv3 [35]. We apply DyRA to all linear layers and compare it against the corresponding static weight approximation baselines. As shown in Figure 5, DyRA achieves a substantially better accuracy–efficiency trade-off across both tasks. In particular, for semantic segmentation at 60% FLOPs reduction, DyRA reduces mIoU by only 1.7 points relative to the original model, preserving most of its dense prediction performance. In contrast, the static weight approximation baseline suffers a 54.1 point drop in mIoU, indicating a severe degradation in fine-grained representations. (a) Semantic Segmentation on ADE20K. (b) Depth Estimation on NYUv2. Figure 5: DyRA vs. static weight approximations on dense prediction with DINOv3. DyRA achieves a superior accuracy–efficiency trade-off for semantic segmentation and depth estimation. Speech Recognition Table 1: WER under different FLOPs reduction ratios (FRR). FRR Method WER(↓ ) 0% Original 1.92 25% Low-rank 2.21 Low-rank + DyRA 2.08 BLAST 2.07 BLAST + DyRA 2.03 50% Low-rank 24.54 Low-rank + DyRA 4.89 BLAST 6.48 BLAST + DyRA 4.27 We evaluate DyRA on automatic speech recognition using a pre-trained Wav2Vec 2.0 Large model [2]. We apply DyRA to all MLP layers and the query/key projections, and measure word error rate (WER) under different FLOPs reduction ratios. As shown in Table 1 on the LibriSpeech Clean test set [28], DyRA consistently achieves lower WER than static weight approximation baselines under matched computational budgets. These results demonstrate that, similar to the vision domains, combining DyRA with structured weight approximations is more effective than using weight approximation alone, and generalizes well to the audio domain. Language Modeling We evaluate DyRA on language modeling using LLaDA-8B [27], a large language diffusion model. We apply DyRA to all MLP layers and the query/key projections, and evaluate performance on HumanEval [6] for code generation and GSM8K [10] for mathematical reasoning. As shown in Figure 7(a), the low-rank baseline performs poorly on both tasks, with accuracy below 4%. In contrast, DyRA improves performance by more than 20 percentage points in both cases under the same computational budget, achieving results substantially closer to the original model. These results demonstrate that DyRA makes structured weight approximation usable for large language diffusion models, even when the FLOPs-matched low-rank baseline largely fails. 3.2 Runtime Analysis Figure 6: End-to-end latency vs. accuracy on DINOv3 depth estimation. We evaluate whether the computational savings of DyRA translate into actual end-to-end speedups on a GPU. To this end, we apply DyRA to DINOv3 and compare it against static weight approximation baselines, including low-rank and BLAST. Following Section 2.4, we use the fused Triton kernels for DyRA. For the weight-only baselines, we use the faster option between the optimized structured kernels in [1] and torch.compile() optimization of PyTorch [29]. We measure the end-to-end latency of depth estimation on the NVIDIA A40 GPU. Figure 6 shows the relative speedup across different depth-estimation accuracy levels. DyRA achieves a 1.3×1.3× end-to-end speedup with only a small accuracy degradation of about 0.6%0.6\%, demonstrating that its FLOPs reduction translates into practical runtime improvement. Compared with the static low-rank baseline, DyRA provides a substantially better accuracy–latency trade-off. It achieves larger speedups at similar accuracy levels, while the low-rank baseline incurs a 3.7×3.7× larger accuracy drop to reach a 1.52×1.52× speedup. Overall, these results show that DyRA provides practical speedups on the NVIDIA A40 GPU while maintaining a more favorable balance between speed and accuracy than static weight approximations. 3.3 Further Evaluations with Augmented Methods Comparison with Input-aware Decomposition We compare DyRA with input-aware decomposition [40, 38], a post-training method that leverages calibration data to compute low-rank weight factors better aligned with output activations. Figure 7 reports results on LLaDA for HumanEval and GSM8K, and on ViT for ImageNet classification. Compared with vanilla low-rank decomposition, input-aware decomposition recovers part of the performance loss across these settings, confirming that activation information is useful for structured approximation. However, since the decomposed weights are fixed after calibration, input-aware decomposition cannot adapt to the activations of each test sample. DyRA provides this missing input-dependent correction at inference time, achieving a better accuracy–efficiency trade-off than input-aware decomposition alone. Moreover, because DyRA operates as a correction on top of the structured computation, it can be combined with input-aware decomposition. In LLaDA, combining DyRA with input-aware decomposition further improves performance on both HumanEval and GSM8K. Similarly, on ImageNet classification at around 70% FLOPs reduction, combining DyRA with input-aware decomposition further improves accuracy from 65.4% to 68.8% over DyRA with vanilla low-rank decomposition, showing that DyRA can further benefit from stronger calibrated static approximations. (a) LLaDA 8B on HumanEval and GSM8K (b) ViT-L on ImageNet-1K Figure 7: Comparison between DyRA and input-aware decomposition on LLaDA-8B and ViT-L. DyRA generally achieves better performance, and can further improve results by using input-aware decomposition as the static approximation. Fine-tuning Results Figure 8: Fine-tuning results on ImageNet classification with ViT-L. Beyond combining DyRA with input-aware decomposition, we further evaluate DyRA under direct fine-tuning on ImageNet classification. In this experiment, we replace all linear layers of a ViT-L model with DyRA-augmented low-rank approximations and fine-tune the resulting model on ImageNet for 15 epochs. For comparison, we fine-tune a low-rank weight-only baseline under the same setting. As shown in Figure 8, consistent with the test-time approximation evaluation in prior subsections, DyRA with fine-tuning outperforms the static weight approximation baseline across different FLOPs budgets. Notably, at around 60% FLOPs reduction, DyRA even slightly surpasses the original dense model while using substantially fewer FLOPs. These results suggest that DyRA remains effective after fine-tuning. Ablation Study We analyze key components of DyRA: the residual rank r and the number of ALS iterations t for an ablation study. We conduct experiments on ViT image classification by combining DyRA with a low-rank weight approximation and varying each hyperparameter. Figure 9 (left) reports the effect of the residual rank, evaluated with r∈4,8,16,24r∈\4,8,16,24\. DyRA consistently outperforms the corresponding low-rank static weight approximation baseline across all choices of r, showing that allocating even a small budget to the dynamic residual is beneficial and that performance is relatively robust to the choice of r. We then evaluate the effect of the ALS iteration count with t∈1,1.5,2t∈\1,1.5,2\ in Figure 9 (right), where t=1.5t=1.5 denotes the update sequence L,R,L,R,L. It shows that all configurations substantially improve over the static approximation baseline. Notably, t=1t=1 already achieves strong performance, suggesting that most of the gain comes from a single iteration. This supports our default choice of t=1t=1, which provides a favorable trade-off between accuracy and computational cost. Figure 9: Ablation study on ImageNet classification with ViT-L. Left: DyRA with different residual ranks r. Right: DyRA with different ALS iteration counts t. 4 Related Works Structured Matrices for Efficient DNN Inference Structured matrices have been widely used to reduce the cost of dense linear layers in deep neural networks. Prior studies [19, 16, 3, 18] have observed that pretrained weight matrices often exhibit low-rank structure, motivating low-rank approximation as a simple compression strategy [17]. More expressive block-structured variants, such as Monarch [11] and BLAST [22], improve the trade-off between efficiency and accuracy, while methods such as GBLR [21] and Einsum Linear [30] learn efficient weight structures directly. Another line of work incorporates activation information from calibration data to obtain input-aware weight decompositions [40, 38, 39, 7]. However, these methods still produce a static approximation before inference, and cannot adapt to the sample-dependent error induced by the current input. Our method addresses this limitation by approximating the realized output error on the fly for each input. Importantly, our method can be combined with advanced structured or input-aware decompositions to obtain a more faithful matrix multiplication under a fixed computational budget. Low-Rank Approximation via Alternating Least Squares Alternating least squares (ALS) has been studied as an efficient method for low-rank approximation and reconstruction, as it reduces a rank-constrained problem to a sequence of tractable least-squares subproblems. Prior work has used ALS for low-rank matrix reconstruction from underdetermined measurements [41], structured low-rank recovery with ADMM-style updates [23], and spectral analyses connected to orthogonal iteration [12]. In contrast, we use ALS to approximate matrix products directly, targeting efficient matrix multiplication rather than reconstruction of a single matrix. 5 Conclusion This work shows that the accuracy–efficiency trade-off of matrix multiplication in pre-trained DNNs can be improved by approximating the realized output product, rather than only the weight matrix. We formulate this output-domain approximation as a low-rank factorization problem and solve it efficiently with ALS during model inference. Building on this formulation, we propose DyRA, which augments static weight approximations with an input-adaptive residual correction. Our experiments show that this correction consistently improves the accuracy–efficiency trade-off across low-rank, Monarch, and BLAST approximations. In particular, DyRA preserves dense prediction performance under aggressive FLOPs reduction and translates these savings into practical speedups on a GPU. Limitations and Future Work. DyRA is designed to accelerate dense matrix multiplication by combining dynamic residual approximation with efficient structured weight approximations. As a result, it is most effective in regimes with large matrix–matrix multiplications, whereas token-by-token vector–matrix workloads such as autoregressive decoding are not its primary target application. Nevertheless, DyRA remains applicable to the prefill stage of autoregressive language models, where the full prompt induces large matrix multiplications. We provide supporting results in the supplementary material (Appendix E.1). Finally, as a test-time acceleration approach, DyRA offers additional flexibility compared to static approximation methods whose rank must be fixed offline. In principle, the dynamic residual rank can be adjusted based on the input or computational budget at inference time. Exploring such adaptive rank allocation is left for future work. Acknowledgments This work was in part supported by MSTAR automotive microelectronics research award #N036453, and also by the Center on Cognitive Multispectral Sensors (COGNISENSE), one of seven centers in Joint University Microelectronics Program (JUMP) 2.0, a Semiconductor Research Corporation (SRC) program sponsored by Defense Advanced Research Projects Agency (DARPA). Daewon Chae would like to acknowledge the support from the Hyundai Motor Chung Mong-Koo Foundation. References [1] P. Abillama, C. Lee, J. Dong, D. Blaauw, D. Sylvester, and H. Kim (2025) Memory-efficient acceleration of block low-rank foundation models on resource constrained gpus. arXiv preprint arXiv:2512.20861. Cited by: §3.2. [2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) Wav2vec 2.0: a framework for self-supervised learning of speech representations. Advances in neural information processing systems 33, p. 12449–12460. Cited by: Appendix D, §3.1. [3] L. Balzano, T. Ding, B. D. Haeffele, S. M. Kwon, Q. Qu, P. Wang, Z. Wang, and C. Yaras (2025) An overview of low-rank structures in the training and adaptation of large models. arXiv preprint arXiv:2503.19859. Cited by: §4. [4] Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. (2020) Piqa: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 7432–7439. Cited by: §E.1. [5] B. Chen, T. Dao, K. Liang, J. Yang, Z. Song, A. Rudra, and C. Re (2022) Pixelated butterfly: simple and efficient sparse training for neural network models. In International Conference on Learning Representations, Cited by: §1. [6] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: Appendix D, §3.1. [7] P. Chen, H. Yu, I. Dhillon, and C. Hsieh (2021) Drone: data-aware low-rank compression for large nlp models. Advances in neural information processing systems 34, p. 29321–29334. Cited by: §4. [8] C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova (2019) Boolq: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), p. 2924–2936. Cited by: §E.1. [9] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §E.1. [10] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix D, §3.1. [11] T. Dao, B. Chen, N. S. Sohoni, A. Desai, M. Poli, J. Grogan, A. Liu, A. Rao, A. Rudra, and C. Ré (2022) Monarch: expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, p. 4690–4721. Cited by: Appendix B, §1, §2.3, §3, §4. [12] A. Dax (2020) The equivalence between orthogonal iterations and alternating least squares. ADVANCES IN LINEAR ALGEBRA & MATRIX THEORY. 10 (2), p. 7–21. Cited by: §4. [13] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: §2.2, §3.1. [14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, Cited by: Appendix D, §2.2, §3.1, footnote 1. [15] C. Eckart and G. Young (1936) The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: §2.2. [16] T. Galanti, Z. S. Siegel, A. Gupte, and T. A. Poggio (2025) SGD with weight decay secretly minimizes the ranks of your neural networks. In The Second Conference on Parsimony and Learning (Proceedings Track), Cited by: §4. [17] H. Hajimolahoseini, W. Ahmed, M. Rezagholizadeh, V. Partovinia, and Y. Liu (2022) Strategies for applying low rank decomposition to transformer-based models. In 36th Conference on Neural Information Processing Systems (NeurIPS2022), Vol. 6. Cited by: §4. [18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: §4. [19] A. K. JAISWAL, Y. Wang, L. Yin, S. Liu, R. Chen, J. Zhao, A. Grama, Y. Tian, and Z. Wang (2025) From low rank gradient subspace stabilization to low-rank weights: observations, theories, and applications. In Forty-second International Conference on Machine Learning, Cited by: Appendix D, §4. [20] J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, et al. (2020) Libri-light: a benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 7669–7673. Cited by: Appendix D. [21] C. Lee and H. Kim (2024) Differentiable learning of generalized structured matrices for efficient deep neural networks. In The Twelfth International Conference on Learning Representations, Cited by: §1, §4. [22] C. Lee, S. M. Kwon, Q. Qu, and H. Kim (2024) BLAST: block-level adaptive structured matrices for efficient deep neural network inference. Advances in Neural Information Processing Systems 37, p. 14996–15027. Cited by: Appendix B, §1, §2.3, §3, §4. [23] K. Li, M. Sundin, C. R. Rojas, S. Chatterjee, and M. Jansson (2014) Alternating strategies are good for low-rank matrix reconstruction. arXiv preprint arXiv:1407.3410. Cited by: §4. [24] C. Lin, S. Gao, J. S. Smith, A. Patel, S. Tuli, Y. Shen, H. Jin, and Y. Hsu (2025) MoDeGPT: modular decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, Cited by: §1. [25] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: Appendix D. [26] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2381–2391. Cited by: §E.1. [27] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. ZHOU, Y. Lin, J. Wen, and C. Li (2026) Large language diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Appendix D, §3.1. [28] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 5206–5210. Cited by: Appendix D, §3.1. [29] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §3.2. [30] A. Potapczynski, S. Qiu, M. Finzi, C. Ferri, Z. Chen, M. Goldblum, C. B. Bruss, C. De, and A. G. Wilson (2024) Searching for efficient linear layers over a continuous space of structured matrices. Advances in Neural Information Processing Systems 37, p. 3857–3881. Cited by: §4. [31] S. Qiu, A. Potapczynski, M. A. Finzi, M. Goldblum, and A. G. Wilson (2024) Compute better spent: replacing dense layers with structured matrices. In Forty-first International Conference on Machine Learning, Cited by: §1. [32] T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor (2021) Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972. Cited by: Appendix D. [33] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi (2021) Winogrande: an adversarial winograd schema challenge at scale. Communications of the ACM 64 (9), p. 99–106. Cited by: §E.1. [34] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus (2012) Indoor segmentation and support inference from rgbd images. In European conference on computer vision, p. 746–760. Cited by: Appendix D, §3.1. [35] O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: Appendix D, §1, §3.1. [36] P. Tillet, H. Kung, and D. Cox (2019) Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, p. 10–19. Cited by: §2.4. [37] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §E.1. [38] X. Wang, Y. Zheng, Z. Wan, and M. Zhang (2025) SVD-LLM: truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, Cited by: Appendix B, Appendix D, §3.3, §4. [39] H. Yu and J. Wu (2023) Compressing transformers: features are low-rank, but weights are not!. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 11007–11015. Cited by: §4. [40] Z. Yuan, Y. Shang, Y. Song, D. Yang, Q. Wu, Y. Yan, and G. Sun (2023) Asvd: activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. Cited by: Appendix B, Appendix D, §1, §3.3, §4. [41] D. Zachariah, M. Sundin, M. Jansson, and S. Chatterjee (2012) Alternating least-squares for low-rank matrix reconstruction. IEEE Signal Processing Letters 19 (4), p. 231–234. Cited by: §4. [42] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba (2017) Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 633–641. Cited by: Appendix D, §3.1. Appendix A DyRA Algorithm Algorithm 1 summarizes the overall DyRA procedure, including offline preparation and inference-time dynamic residual correction. Algorithm 1 DyRA with One-Step ALS Residual Approximation 1: procedure Offline Preparation(,k,r,λ W,k,r,λ) 2: Compute a structured approximation W of W 3: res←− W_res← W- W 4: Initialize and fix (0)∈ℝr×c2 R^(0) ^r× c_2 5: Precompute L←(res(0)⊤)((0)(0)⊤+λr)−1 P_L← ( W_res R^(0) ) ( R^(0) R^(0) +λ I_r )^-1 6: return ^,res,L W, W_res, P_L 7: end procedure 8: 9: procedure Inference(,^,res,L,λ X, W, W_res, P_L,λ) 10: base← Y_base← X W 11: (1)←L L^(1)← X P_L 12: (1)←((1)⊤(1)+λr)−1((1)⊤)res R^(1)← ( L^(1) L^(1)+λ I_r )^-1 ( L^(1) X ) W_res 13: ^←base+(1)(1) Y← Y_base+ L^(1) R^(1) 14: return Y 15: end procedure Appendix B Background on Structured Matrices Low-Rank Approximation Low-rank approximation represents a weight matrix ∈ℝd×m W ^d× m as the product of two smaller matrices, ^=,∈ℝd×r,∈ℝr×m, W= L R, L ^d× r, R ^r× m, where r≪min(d,m)r (d,m). A standard choice is the rank-r truncated singular value decomposition (SVD). Given =⊤ W= U V , the approximation is ^=rrr⊤,=rr,=r⊤. W= U_r _r V_r , L= U_r _r, R= V_r . This approximation keeps the top-r singular components of W, providing the best rank-r approximation under the Frobenius norm. Monarch Matrix Monarch matrices [11] represent a weight matrix using two block-diagonal factors with an intermediate permutation. For ∈ℝN×N W ^N× N with N=b1b2N=b_1b_2, the approximation is written as ^=⊤,=diag(1,…,b2),=diag(1,…,b1), W= P L P R, L=diag( L_1,…, L_b_2), R=diag( R_1,…, R_b_1), where i∈ℝb1×b1 L_i ^b_1× b_1 for i∈[b2]i∈[b_2] and j∈ℝb2×b2 R_j ^b_2× b_2 for j∈[b1]j∈[b_1]. Here, ∈ℝN×N P ^N× N is a fixed permutation matrix that reorders the block structure, so that the overall product induces a structured block-wise pattern in W. In practice, the factors can be obtained by reshaping the dense weight into blocks, applying a truncated SVD to each block, and rearranging the resulting factors. BLAST BLAST [22] represents a weight matrix using shared block-row and block-column bases. For ∈ℝd×m W ^d× m, we partition the matrix into a grid of blocks i,j∈ℝp×q W_i,j ^p× q. Each block is parameterized as ^i,j=ii,jj⊤,i∈ℝp×r,i,j∈ℝr×r,j∈ℝq×r. W_i,j= U_i S_i,j V_j , U_i ^p× r, S_i,j ^r× r, V_j ^q× r. Here, i U_i is shared by all blocks in the same block row, and j V_j is shared by all blocks in the same block column. The middle factor i,j S_i,j is specific to each block and controls how the shared row and column bases are combined. This parameterization captures low-dimensional structure at the block level while reusing bases across the matrix. In practice, the BLAST factors are obtained by minimizing the reconstruction error between W and W. Input-Aware Decomposition Input-aware decomposition aims to better preserve output activations by incorporating calibration inputs into the weight approximation. Given a calibration set =ss=1||S=\ X_s\_s=1^|S|, these methods approximate W by reducing the output-space discrepancy measured on the calibration set, ∑s∈‖s−s^‖F2 _ X_s \| X_s W- X_s W\|_F^2. A common formulation introduces an invertible transformation matrix S and rewrites the weight as =()−1 W=( W S) S^-1. Truncated SVD is then applied to the transformed weight, ≈rrr⊤, W S≈ U_r _r V_r , yielding the approximation ^=rrr⊤−1. W= U_r _r V_r S^-1. (9) The transform S determines how calibration activations influence the decomposition. For example, S can be chosen as a diagonal scaling matrix based on channel-wise activation magnitudes [40], or as a Cholesky factor of the input covariance matrix computed from the calibration set [40, 38]. Appendix C Proof of Proposition 1 Proof. We start from the one-step DyRA approximation in Eq. 6, ≈^+(1)(1). X W≈ X W+ L^(1) R^(1). (10) In the low-rank case, the static approximation is ^=1:k1:k1:k⊤, W= U_1:k _1:k V_1:k , (11) and the residual is res=−^=k+1:k+1:k+1:⊤. W_res= W- W= U_k+1: _k+1: V_k+1: . (12) We initialize the right factor as (0)=k+1:k+r⊤ R^(0)= V_k+1:k+r . Since the singular vectors are orthonormal, (0)(0)⊤= R^(0) R^(0) = I. Substituting this into the left-factor update in Eq. 4 yields (1) L^(1) =(res(0)⊤)((0)(0)⊤)−1 = X ( W_res R^(0) ) ( R^(0) R^(0) )^-1 (13) =(k+1:k+1:k+1:⊤k+1:k+r) = X ( U_k+1: _k+1: V_k+1: V_k+1:k+r ) =k+1:k+rk+1:k+r. = X U_k+1:k+r _k+1:k+r. Let rres=k+1:k+rk+1:k+r A_r^res= U_k+1:k+r _k+1:k+r, so that (1)=rres L^(1)= X A_r^res. Substituting this into the right-factor update in Eq. 5, we obtain (1) R^(1) =((1)⊤(1))−1((1)⊤)res = ( L^(1) L^(1) )^-1 ( L^(1) X ) W_res (14) =(rres⊤⊤rres)−1rres⊤⊤res. = ( A_r^res X X A_r^res )^-1 A_r^res X X W_res. Plugging the resulting one-step factors into Eq. 6, we obtain X W ≈^+(1)(1) ≈ X W+ L^(1) R^(1) (15) =1:k1:k1:k⊤+k+1:k+rk+1:k+r(1) = X U_1:k _1:k V_1:k + X U_k+1:k+r _k+1:k+r R^(1) =(1:k1:k1:k⊤+k+1:k+rk+1:k+r(1)) = X ( U_1:k _1:k V_1:k + U_k+1:k+r _k+1:k+r R^(1) ) =[1:k1:k+1:k+rk+1:k+r][1:k⊤(1)]. = X bmatrix U_1:k _1:k& U_k+1:k+r _k+1:k+r bmatrix bmatrix V_1:k \\ R^(1) bmatrix. This proves the stated form. Finally, if ⊤= X X= I, then (1) R^(1) =(rres⊤rres)−1rres⊤res. = ( A_r^res A_r^res )^-1 A_r^res W_res. (16) Using orthogonality of singular vectors, rres⊤rres=k+1:k+r2,rres⊤res=k+1:k+r2k+1:k+r⊤, A_r^res A_r^res= _k+1:k+r^2, A_r^res W_res= _k+1:k+r^2 V_k+1:k+r , which gives (1)=k+1:k+r⊤ R^(1)= V_k+1:k+r . Thus, DyRA reduces to the static rank-(k+r)(k+r) truncated SVD in the isotropic case. ∎ Appendix D Experimental Details In this section, we provide additional details on the experimental setup. Unless otherwise specified, we use one ALS iteration for DyRA across all experiments. We initialize (0) R^(0) with the top right singular vectors of the residual weight res=− W_res= W- W, following the subspace interpretation in Section 2.3. This initialization is performed once offline before inference and does not introduce input-dependent overhead. Image Classification For image classification, we evaluate DyRA on ImageNet-1K using a ViT-L/32 model55 5 https://huggingface.co/google/vit-large-patch32-384 [14] pretrained on ImageNet-21K [32]. We apply DyRA to all linear layers in the model, including the MLP layers and the query, key, value, and output projections in the attention blocks. We fix the rank of the dynamic residual approximation to r=8r=8 and control the FLOPs by varying the configuration of the static weight approximation. For low-rank approximation, we vary the static rank over 100,150,200,250,300,350,400,450\100,150,200,250,300,350,400,450\. For Monarch, we vary the Monarch block size over 3,4,6,8\3,4,6,8\. For BLAST, we fix the block partition to 44 and vary the static rank over 100,150,200,250,300,350,400,450\100,150,200,250,300,350,400,450\. For each structured approximation, we construct the corresponding weight-only baseline by sweeping its configuration to match the FLOPs of DyRA. Dense Prediction For dense prediction, we evaluate DyRA on semantic segmentation and depth estimation using a ViT-7B-based DINOv3 model [35]. We evaluate semantic segmentation on ADE20K [42] and depth estimation on NYUv2 [34]. For both tasks, we follow the image preprocessing and decoding protocol of DINOv3 66 6 https://github.com/facebookresearch/dinov3. We apply DyRA uniformly to all linear layers in the model. We fix the rank of the dynamic residual approximation to r=48r=48 and vary the configuration of the static weight approximation to control the FLOPs reduction. For depth estimation, we sweep the low-rank approximation rank over 436,750,1000,1250,1500,1750\436,750,1000,1250,1500,1750\, the BLAST rank over 433,746,995,1244,1492,1741\433,746,995,1244,1492,1741\, and the Monarch block size over 2,3,4,6,8\2,3,4,6,8\. For semantic segmentation, we sweep the low-rank approximation rank over 750,1000,1250,1500,1750\750,1000,1250,1500,1750\, the BLAST rank over 746,995,1244,1492,1741\746,995,1244,1492,1741\, and the Monarch block size over 2,3,4,6,8\2,3,4,6,8\. The corresponding weight-only baselines are obtained by adjusting the same configuration parameters to match the FLOPs of DyRA. Speech Recognition For speech recognition, we evaluate DyRA on Wav2Vec2-Large77 7 https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self [2], pretrained and fine-tuned on 960 hours of Libri-Light [20] and LibriSpeech [28]. We report word error rate (WER) on the LibriSpeech test set, following the standard audio preprocessing and CTC-based decoding protocol used for Wav2Vec2. DyRA is applied to the query and key projections in the attention blocks and to the two fully connected layers in each MLP block. For both DyRA and the corresponding weight-only baselines, rank parameters are selected by grid search over a predefined set of configurations under target FLOPs reduction ratios of 25% and 50%. We summarize each DyRA configuration as (rMLP,rattn,rres)(r_MLP,r_attn,r_res), where rMLPr_MLP, rattnr_attn, and rresr_res denote the MLP rank, attention rank, and residual rank, respectively. The selected configurations are (370,210,48)(370,210,48) and (200,60,36)(200,60,36) for BLAST at 25% and 50% FLOPs reduction, respectively, and (440,160,44)(440,160,44) and (200,90,36)(200,90,36) for low-rank at 25% and 50% FLOPs reduction, respectively. Language Modeling For language modeling, we evaluate DyRA on LLaDA-8B88 8 https://huggingface.co/GSAI-ML/LLaDA-8B-Base [27], a large language diffusion model. We consider two benchmarks: HumanEval [6] for code generation and GSM8K [10] for mathematical reasoning. We apply low-rank-based DyRA to the query and key projections in the attention blocks and to the MLP layers. Following prior observations that query and key projections in LLMs are more amenable to compression [19], we use more aggressive FLOPs reduction for the query and key projections than for the MLP layers. For HumanEval, we use two DyRA configurations targeting 20% and 40% total FLOPs reduction. At 20% reduction, we set the query/key static rank to 840 with residual rank 40, and the MLP static rank to 1000 with residual rank 150. At 40% reduction, we set the query/key static rank to 300 with residual rank 50, and the MLP static rank to 300 with residual rank 150. For GSM8K, we again use two configurations targeting 20% and 40% total FLOPs reduction. At 20% reduction, we set the query/key static rank to 512 with residual rank 240, and the MLP static rank to 1024 with residual rank 380. At 40% reduction, we set the query/key static rank to 320 with residual rank 180, and the MLP static rank to 630 with residual rank 270. Fine-tuning ViT on ImageNet For the fine-tuning experiment, we use a ViT-L/32 model pretrained on ImageNet-21K and fine-tune it on ImageNet-1K. We apply DyRA uniformly to all linear layers, including the MLP layers and all attention projections. Each model is fine-tuned for 15 epochs using the AdamW optimizer [25] with a learning rate of 2×10−52× 10^-5 and a batch size of 1024. We fix the dynamic residual approximation rank to r=16r=16 and vary the static weight rank over 96,128,160,196\96,128,160,196\, training a separate DyRA model for each static rank. For each DyRA configuration, we construct a FLOPs-matched static low-rank baseline by adjusting its rank and fine-tune the corresponding baseline model using the same training setup. Comparison with Input-Aware Decomposition. For comparison with prior input-aware decomposition methods, we use the approaches proposed in [40, 38]. In Eq. 9, we instantiate S as the Cholesky factor of the input covariance matrix computed from calibration activations. Since input-aware decompositions can be sensitive to the choice of calibration data, we use the corresponding training split of each evaluation dataset to provide a strong baseline setting. Specifically, we use the ImageNet training set for ImageNet experiments and the GSM8K training set for GSM8K evaluation. Because HumanEval consists only of evaluation problems without a dedicated training split, we use the GSM8K training set as the calibration set for HumanEval experiments. For each target FLOPs reduction ratio, the ranks are kept identical to those of the corresponding DyRA configurations used in the language modeling experiments, except that the static low-rank factors are initialized using input-aware decomposition. Appendix E Additional Experiment Results Table 2: Perplexity comparison on language modeling benchmarks without fine-tuning. Lower is better. Bold indicates the best performance under the same compression ratio. CR Method WikiText PG-19 C4 0% Dense 9.38 10.07 7.01 20% Low-Rank 16.00(+6.62) 15.93(+5.86) 10.81(+3.80) BLAST 12.94(+3.56) 13.05(+2.98) 9.26(+2.25) Low-Rank + DyRA 12.03(+2.65) 12.48(+2.41) 8.75(+1.74) Table 3: Performance comparison across multiple-choice reasoning benchmarks. FR denotes the FLOP reduction ratio. Values in parentheses indicate accuracy changes relative to the dense model. Bold indicates the best performance under the same compression ratio. FR Method PIQA Wino BoolQ OBQA ARC-e ARC-c Avg 0% Dense 81.66 74.90 77.03 47.60 80.18 50.85 68.70 20% Low-Rank 76.66(-5.00) 69.38(-5.52) 69.42(-7.61) 45.20(-2.40) 73.53(-6.65) 41.72(-9.13) 62.65(-6.05) BLAST 78.56(-3.10) 67.64(-7.26) 66.91(-10.12) 44.80(-2.80) 75.42(-4.76) 43.43(-7.42) 62.79(-5.91) Low-Rank + DyRA 77.69(-3.97) 69.93(-4.97) 71.50(-5.53) 48.60(+1.00) 76.05(-4.13) 44.11(-6.74) 64.65(-4.05) E.1 Autoregressive LLM Prefilling In addition to the diffusion language model setting, we examine whether DyRA can also be applied to autoregressive LLMs, particularly in the prefilling regime where computation is dominated by large matrix multiplications over the input sequence. Rather than measuring decoding throughput, we evaluate this setting through language modeling perplexity and multiple-choice reasoning benchmarks [4, 33, 8, 26, 9], where each prompt and candidate continuation is processed in a forward pass. To better reflect non-trivial prefilling workloads, we use few-shot evaluation prompts that provide extended input contexts for multiple-choice reasoning benchmarks. We apply low-rank-based DyRA to the linear layers of LLaMA-7B [37], targeting a 20% reduction in matrix multiplication FLOPs. As shown in Tables 2 and 3, DyRA consistently improves over the corresponding low-rank approximation and also outperforms the BLAST baseline under the same FLOPs reduction. These results suggest that DyRA can be effectively applied to the prefilling stage of autoregressive LLMs. Figure 10: Top-1 accuracy versus FLOPs reduction on ViT-B/16. Across low-rank, Monarch, and BLAST approximations, combining DyRA consistently improves the accuracy–efficiency trade-off over static weight approximation baselines. E.2 Image Classification with ViT-B To further examine whether DyRA remains effective across different model scales, we additionally evaluate image classification on ViT-B/16 99 9 https://huggingface.co/google/vit-base-patch16-224. Using the same experimental setup as in Section D, we combine DyRA with static weight approximations and compare it against FLOPs-matched static baselines. As shown in Figure 10, DyRA consistently improves the accuracy–efficiency trade-off over the corresponding static weight approximations. These results demonstrate that DyRA’s gains are not specific to the larger ViT-L model used in the main paper, but also generalize to a smaller ViT backbone. E.3 Convergence of ALS Residual Approximation We provide additional details and results for the ALS convergence experiment discussed in Section 2.2. We randomly sample 4,096 images from the ImageNet validation set and use them to extract the input activations X at each linear layer of ViT-L. For each layer, we construct the residual weight res=− W_res= W- W, where W is obtained via a rank-196 truncated SVD of W. We then apply ALS to approximate the residual product res X W_res with target rank r=16r=16. At each ALS iteration t, we compute the relative Frobenius error between the residual product and its low-rank approximation: ‖res−(t)(t)‖F‖res‖F \| X W_res- L^(t) R^(t)\|_F\| X W_res\|_F. We compare this error with the optimal rank-r error obtained by applying SVD directly to res X W_res. Figure 12 reports the convergence results averaged over the sampled images across all ViT linear layers. ALS rapidly approaches the optimal rank-r error in nearly all layers, typically within two iterations. (a) Attention query and key projections. (b) Attention value projection and second MLP linear layer. Figure 11: Additional FLOPs-matched output approximation error results on ViT-L sublayers omitted from the main paper. DyRA consistently achieves lower normalized Frobenius output error than the static low-rank baseline. E.4 FLOPs-Matched Output Approximation Error We provide additional details and results for the FLOPs-matched output error comparison discussed in Section 2.3. Using the same ViT-L and ImageNet validation setup as in Section E.3, we collect input activations X from 4,096 validation images for each target linear layer. For each layer, we compute a rank-196 truncated SVD approximation W and apply a rank-16 DyRA correction to the residual product (−^) X( W- W) using one ALS iteration. For the FLOPs-matched static baseline, we increase the low-rank approximation rank so that its exact FLOPs match the total FLOPs of DyRA, including the dynamic correction overhead. We then compare the remaining normalized Frobenius output error, ‖−^‖F/‖‖F\| X W- Y\|_F/\| X W\|_F, where Y denotes the approximated output. Figure 11 reports additional results for the remaining sublayers not shown in the main text: the query, key, and value projections in attention, and the second MLP linear layer. Across these additional sublayers, DyRA consistently yields lower output approximation error than the FLOPs-matched higher-rank static low-rank baseline. Appendix F Broader Impacts DyRA is a test-time inference acceleration method designed to reduce the computational cost of neural network inference. Its potential positive societal impact lies in improving the resource efficiency of AI systems, enabling model deployment under limited computational budgets and reducing the energy costs associated with inference. Since DyRA does not introduce a new application domain or change the intended use of the underlying model beyond approximation, we do not identify negative societal impacts specific to the proposed method. However, by making inference more efficient, DyRA may facilitate broader deployment of existing AI systems, and therefore may inherit the societal risks associated with those systems and their applications. (a) Layers 0–7 of ViT-L. (b) Layers 8–15 of ViT-L. (c) Layers 16–23 of ViT-L. Figure 12: Convergence of ALS-based residual product approximation across ViT-L linear layers.