Paper deep dive

LRCC: Generalizing Low-Rank Compression with Conditional Computation

Thomas Vaitses Fontanari, Maximo Eduardo Rulli, Federico Alvetreti, Donatella Genovese, Simone Scardapane

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 91

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 10/8/2026, 6:14:35 AM

Summary

The paper introduces Low-Rank Conditional Computation (LRCC), a method that enhances low-rank compression of pretrained language models by adding token-dependent computation. Unlike static low-rank methods that use fixed rank allocations, LRCC trains lightweight routers per Transformer block to select among nested low-rank paths dynamically. This approach improves predictive performance and latency efficiency on models like Llama and Qwen without requiring specialized kernels.

Entities (10)

LRCC · method · 98%Llama-2-7b · model · 95%Llama 3.2-1B · model · 95%Qwen3-4B · model · 95%Router · component · 92%Transformer Block · component · 92%COALA · method · 90%SVD-LLM · method · 90%Swift-SVD · method · 90%WikiText-2 · dataset · 90%

Relation Signals (10)

LRCC → evaluatedon → Llama-2-7b

confidence 95% · We evaluate LRCC on Llama and Qwen models... including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B

LRCC → evaluatedon → Llama 3.2-1B

confidence 95% · At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B

LRCC → evaluatedon → Qwen3-4B

confidence 95% · We evaluate the performance of LRCC on Llama-3.2-1B, Llama-2-7B and Qwen3-4B models

LRCC → improves → Predictive Performance

confidence 95% · LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods.

Router → locatedin → Transformer Block

confidence 95% · training one lightweight router per Transformer block to select among a small set of nested low-rank paths.

LRCC → trains → Router

confidence 95% · A single router per Transformer block is then trained to select which subset of ranks to activate within the block

LRCC → uses → nested low-rank paths

confidence 93% · LRCC... select among a small set of nested low-rank paths.

LRCC → comparedwith → COALA

confidence 90% · We compare LRCC with several state-of-the-art low-rank compression methods, namely SVD-LLM, COALA and Swift-SVD.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers' path choices.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

87,754 characters extracted from source content.

Expand or collapse full text

LRCC: Generalizing Low-Rank Compression with Conditional Computation Thomas V. Fontanari* Maximo Rulli Federico Alvetreti Affiliation: Donatella Genovese Simone Scardapane Affiliation: Sapienza University of Rome Abstract Low-rank compression reduces the cost of pretrained language models by replacing linear transformations with low-rank factorizations. However, conventional methods use a fixed rank allocation during inference, assigning the same amount of compute regardless of the input token. We introduce Low-Rank Conditional Computation (LRCC), which adds token-dependent computation to pretrained models by training one lightweight router per Transformer block to select among a small set of nested low-rank paths. During training, the low-rank factors remain frozen, and only the routers are optimized. We evaluate LRCC on Llama and Qwen models for language modeling and zero-shot downstream tasks. Within the same average active-parameter budget, LRCC improves the predictive performance over static low-rank compression, including a 7.6 percentage-point gain in average downstream accuracy on Llama-2-7B over static methods. At matched batch-size-1 decoding latency, LRCC improves both perplexity and downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without specialized kernels. Finally, we assess the usefulness of assigning a token-wise path by analyzing the routers’ path choices. $*$$*$footnotetext: Correspondence to thomas.fontanari@uniroma1.it 1 Introduction Foundation models (Bommasani et al., 2021) achieve strong predictive performance across several tasks (Devlin et al., 2019; Brown et al., 2020; Radford et al., 2021) but require substantial computational and memory resources, motivating efficiency techniques. Low-rank compression methods (Chen et al., 2021) approximate the original weight matrices with low-rank products, typically obtained using SVD or its activation-aware variants. Given activations collected on a calibration dataset, these methods seek a rank-constrained approximation that minimizes the reconstruction error of each layer’s outputs. Layer-specific ranks are then selected to satisfy a global budget via fixed heuristics or learned allocations (Qi et al., 2026; Qinsi et al., 2025). Conditional computation (Scardapane et al., 2024; Pfeiffer et al., 2023) instead adapts the computational graph to input activations. Applying it to pretrained models, however, can require additional training and compute modules alongside lightweight routers. Previous work has demonstrated that SVD-like decompositions of linear layers can be used to produce many-in-one models, where the smaller ones correspond to a subset of the weights of larger ones (Yuan et al., 2025; Zaccone et al., 2026). The nested structure of these approaches provides an inexpensive basis for conditional computation. Routing over this hierarchy at every linear layer would nevertheless require many routers, each choosing among thousands of possible ranks. For example, a linear layer in Llama-2-7B can have up to 40964096 rank choices, and each Transformer block contains seven such layers. The challenge is to expose useful flexibility without making routing itself expensive. This work asks two questions: can token-dependent rank allocation improve predictive performance within the same active-parameter budget, and do these gains translate into better quality at matched decoding latency? We distinguish these questions because activating fewer parameters does not by itself guarantee lower latency. We address these questions by proposing Low-Rank Conditional Computation (LRCC). Through low-rank decomposition and rank allocation strategies, we construct a small set of models with shared weights and different computational costs. The smaller models correspond to a subset of the ranks used in the larger ones, leading to a nested family of models. A single router per Transformer block is then trained to select which subset of ranks to activate within the block, while the pretrained model and shared low-rank factors remain frozen. By using a small number of submodels and limiting the routers to the block level, we are able to show that token-dependent computation can be recovered from an existing low-rank hierarchy by training only lightweight routers, while still producing meaningful quality gains under controlled compute budgets. Our main contributions are summarized below: 1. We introduce LRCC, which combines shared low-rank factors with block-level routing to allocate ranks dynamically to each token. 2. We show that LRCC improves predictive performance over static low-rank compression within the same active-parameter budget, with gains of up to 7.67.6 percentage points in average downstream accuracy. It also achieves competitive quality at matched batch-size-1 decoding latency without specialized CUDA kernels. 3. We introduce a target-cost annealing scheme that produces a family of router checkpoints in one progressive training run, and analyze how the number of paths, routing choices, and deployment costs affect the resulting models. Figure 1: Overview of LRCC. First, (a) each linear weight matrix is decomposed into ordered rank-one components, and a rank-allocation algorithm constructs nested paths at different target costs. Then, (b) the low-rank factors are frozen and one router per block is trained to select a path for each token, balancing task and compression losses. 2 Method Conventional low-rank compression methods (Yuan et al., 2025; Wang et al., 2025; Parkina and Rakhuba, 2025; Qi et al., 2026; Qinsi et al., 2025) replace each linear weight matrix with a product of smaller matrices and retain only a fixed number of rank components in each layer. The number retained in each layer is chosen so that the compressed model meets a predefined inference-cost budget, typically expressed as a fraction of the original model’s cost. In contrast, LRCC dynamically determines the number of ranks activated in each linear layer of a Transformer block for each token. More precisely, we define multiple paths through each Transformer block, where each path specifies the number of ranks activated in every linear layer of the block. As in Matryoshka learning (Kusupati et al., 2022), the paths are ordered and nested: lower-cost paths retain subsets of the components used by higher-cost paths. Consequently, all paths share the same parameter base and can be selected using a single path index. We train only the block-level routers that choose among the possible paths for each token, while the remaining parameters are frozen. Figure 1 illustrates the framework. 2.1 Layer Decomposition For each linear weight matrix bℓ∈ℝdin×doutW_b ^d_in× d_out at layer ℓ in Transformer block b, we compute an activation-aware decomposition using a matrix of input activations bℓ∈ℝn×dinX_b ^n× d_in, where n is the number of token activations collected on a calibration dataset. More formally, we solve for each layer argmin~bℓ,r⁡‖bℓbℓ−bℓ~bℓ,r‖Fs.t.rank⁡(~bℓ,r)≤r. _ W_b ,r \|X_b W_b -X_b W_b ,r \|_F .t. ( W_b ,r)≤ r. (1) Let bℓ∈ℝdout×doutV_b ^d_out× d_out denote the matrix of right singular vectors of bℓbℓX_b W_b with columns ordered by decreasing singular value, and let bℓ,:rV_b ,:r denote its first r columns. Then, Equation 1 has an analytical solution given by ~bℓ,r=bℓbℓ,:rbℓ,:r⊤. W_b ,r=W_b V_b ,:rV_b ,:r . (2) Appendix B proves optimality and nestedness and Appendix E.1 describes how to compute the factors efficiently. Equation 2 has the important property that, similar to standard SVD, optimal solutions for different target ranks r are nested. Hence, we can obtain the best rank-r activation-aware approximation by indexing the first r columns of the precomputed matrices bℓ=bℓbℓA_b =W_b V_b and bℓ,=bℓB_b ,=V_b . More concretely, ~bℓ,r=(bℓbℓ,:r)bℓ,:r⊤=bℓ,:rbℓ,:r⊤. W_b ,r=(W_b V_b ,:r)V_b ,:r =A_b ,:rB_b ,:r . (3) Naively replacing a weight matrix with the product of its low-rank factors does not necessarily reduce computation. A dense linear transformation requires approximately 2dindout2d_ind_out FLOPs per token, whereas its rank-r factorization requires approximately 2r(din+dout)2r(d_in+d_out) FLOPs, yielding a computational benefit only when r<dindoutdin+doutr< d_ind_outd_in+d_out. 2.2 Path Construction In Section 2.1, we explained how to produce the decompositions of each linear transform in the model. Crucially, the decompositions form a nested structure, where retaining more components cannot increase reconstruction error in Equation 1, but induces a larger number of FLOPs. Routing over every possible rank at every layer would introduce many routing decisions and a large choice set for each router. Our path-count ablation in Section 4.1 shows that a small set of choices is sufficient to capture most of the quality gains. Instead, we simplify the routing problem by placing routers only at the beginning of Transformer blocks and restricting them to a small set of choices. We first construct T submodels at different target costs using the rank-allocation procedure of Swift-SVD (Qi et al., 2026). Given a target cost, Swift-SVD assigns a rank to each linear layer according to its estimated importance to model performance. More formally, submodel m with m=1,…,Tm=1,...,T is defined by the rank allocation ℛm=rbℓmb,ℓ,R_m=\r_b m\_b, , where rbℓmr_b m denotes the rank retained by linear layer ℓ in Transformer block b under submodel m. Equivalently, each weight matrix of the submodel is approximated as ~bℓ(m)=bℓ,:rbℓmbℓ,:rbℓm⊤. W_b ^(m)=A_b ,:r_b mB_b ,:r_b m . (4) Thus, each submodel specifies a different computational budget by assigning a particular rank to every linear layer. These whole-model allocations define the choices available to each block-level router. In block b, path pbmp_bm is the restriction of submodel m’s allocation to that block, pbm=rbℓmℓ.p_bm=\r_b m\_ . Selecting pbmp_bm therefore determines simultaneously how many ranks are activated in each linear layer of block b. We can count the number of parameters in a path based on its set of ranks. A factorized matrix of rank r activates r⁡(din+dout)r(d_in+d_out) parameters, since it is represented by two factors of sizes din×rd_in× r and dout×rd_out× r. Therefore, the parameter count of path pbmp_bm is Pbm=∑ℓrbℓm(din,bℓ+dout,bℓ).P_bm= _ r_b m (d_in,b +d_out,b ). (5) LRCC trains routers to assign each token to a corresponding path. Specifically, for each token entering a Transformer block b, the router gb:ℝdh→[0,1]Tg_b:R^d_h→[0,1]^T is a learnable linear transform that maps the input activations h∈ℝdhh ^d_h to a Gumbel-Softmax distribution over the T available paths. During both training and inference, a hard one-hot selection determines the executed path and hence the number of activated parameters. 2.3 Router Training The routers are optimized to minimize the task loss and a compression loss, which we define in this section. To preserve differentiability and retain the hard one-hot selection of the routers, we use the straight-through estimator (STE) (Jang et al., 2017). During training, all pretrained model parameters and low-rank factors remain frozen, and only the routers are updated. LRCC uses the ratio of activated parameters relative to the original model as the cost of processing a given token. Let cbc_b denote the average normalized cost of block b, and let CbC_b denote the normalized cost of its original non-factorized implementation. Because a low-rank factorization can activate more parameters than the original block, we softly cap its cost during training using a smooth approximate of min⁡(cb,Cb) (c_b,C_b) given by c¯b=cb−Cbκsoftplus⁡(κ⁡(cb−Cb)Cb), c_b=c_b- C_bκsoftplus ( κ(c_b-C_b)C_b ), (6) where κ is a hyperparameter that determines the smoothness of the approximation. This reflects that blocks whose factorized implementation is more expensive than the original one should be kept non-factorized when deployed, while avoiding the zero-gradient region introduced by a hard cap. The global cost and corresponding compression loss are then cglobal=∑b=1Bc¯b,ℒc=max⁡(0,cglobal−t),c_global= _b=1^B c_b, _c= (0,c_global-t ), (7) where t is the user-defined target cost. The routers are then trained to jointly minimize the task loss ℒtaskL_task and the computational cost: ℒ=ℒtask+λsℒc,L=L_task+ _sL_c, (8) where λs _s is an adaptive weight updated at training step s according to the current constraint violation, as described in Section 2.4. Further details on the computation of cbc_b and the soft cap are given in Appendix C. 2.4 Target-cost Annealing Because the paths are nested, LRCC naturally supports producing routers for multiple target costs in a single progressive training run, moving from a high-cost model to one with fewer activated parameters. We start with a high target cost, 0.950.95 in our experiments, and train the router until convergence (or until a maximum number of epochs is reached). Then, we reduce the target cost to a slightly lower value, 0.9250.925 in our case, and continue training the router weights to produce a new checkpoint. We stop when the final target cost is reached, or when convergence fails for some target, i.e., the training process failed to obtain a router inducing a cost within an acceptable distance of the current target. As compression becomes more aggressive, the task’s loss increases and can dominate the total cost, making it difficult to reach a checkpoint sufficiently close to the target. To avoid this, we introduce an adaptive compression-loss weight λs _s, which is updated after each optimization step based on the constraint violation: λs+1=max⁡(0,λs+α⁡(c−t)), _s+1= (0, _s+α(c_ global-t)), (9) where α is a hyperparameter. Intuitively, when c>tc_ global>t, the compression-loss weight increases, and when c<tc_ global<t, it decreases, allowing the router to activate more parameters. 3 Results Models. We evaluate the performance of LRCC on Llama-3.2-1B (Meta AI, 2024), Llama-2-7B (Touvron et al., 2023) and Qwen3-4B (Yang et al., 2025) models, as they allow us to compare with the low-rank baseline methods (Qi et al., 2026; Parkina and Rakhuba, 2025; Wang et al., 2025). LRCC setup. We decompose all linear layers in each Transformer block, including both attention and MLP projections, as described in Section 2.1, and compute the ranks assigned to each block path using a rank-allocation algorithm, as described in Section 2.2. In all cases, we follow previous work and use 256 2,048-token samples from the training set of WikiText-2 (Merity et al., 2017) to decompose the weights and train the router. We use the standard causal language-modeling loss as the task loss ℒtaskL_task. Training starts at a target cost of 0.975, and we progressively obtain checkpoints down to 0.3 in steps of 0.025 using the annealing scheme described in Section 2.4. Finally, during evaluation, we use 64 WikiText-2 validation samples to estimate the average activated parameter count of each block. Blocks for which this exceeds the parameter count of the corresponding block with non-factorized linear layers are deployed in non-factorized form. Datasets. The resulting models are evaluated on the WikiText-2 test set for language modeling and on several zero-shot downstream performance using ARC-Easy and ARC-Challenge (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), and WinoGrande (Sakaguchi et al., 2020). Static baselines. We compare LRCC with several state-of-the-art low-rank compression methods, namely SVD-LLM (Wang et al., 2025), COALA (Parkina and Rakhuba, 2025) and Swift-SVD (Qi et al., 2026). Since COALA and SVD-LLM do not propose a rank allocation method, we also evaluate variants using ranks obtained with the Swift-SVD rank allocation strategy. We omit SVD-LLM’s optional fine-tuning stage to isolate the effect of the compression method under a matched adaptation budget, since the other baselines do not use post-compression fine-tuning of the weights and an analogous refinement stage could likewise be applied to LRCC and the baselines. See Appendix D for details and additional results. PPL ↓ Commonsense reasoning accuracy (%) ↑ Ratio Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino Avg. 100% Base 5.49 44.80 73.95 79.42 76.08 44.00 78.62 69.30 66.60 90% COALA 7.27 35.5 ± 1.4 57.5 ± 1.0 49.3 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 70.6 ± 1.1 65.2 ± 1.3 54.2 ± 0.5 SVD-LLM 7.27 36.2 ± 1.4 57.4 ± 1.0 50.1 ± 0.9 62.8 ± 0.5 39.0 ± 2.2 70.8 ± 1.1 65.3 ± 1.3 54.5 ± 0.5 SVD-LLM+Swift 7.33 35.5 ± 1.4 58.3 ± 1.0 55.0 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 71.1 ± 1.1 62.9 ± 1.4 54.8 ± 0.5 COALA+Swift 7.27 35.1 ± 1.4 58.4 ± 1.0 57.4 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 70.9 ± 1.1 63.1 ± 1.4 55.1 ± 0.5 Swift-SVD 7.33 35.6 ± 1.4 58.7 ± 1.0 55.4 ± 0.9 62.8 ± 0.5 39.6 ± 2.2 70.7 ± 1.1 62.9 ± 1.4 55.1 ± 0.5 LRCC-5 6.17 41.8 ± 1.4 68.5 ± 1.0 74.4 ± 0.8 71.0 ± 0.5 41.4 ± 2.2 74.8 ± 1.0 66.3 ± 1.3 62.6 ± 0.5 LRCC-9 6.03 40.4 ± 1.4 68.6 ± 1.0 74.9 ± 0.8 71.7 ± 0.4 40.8 ± 2.2 74.8 ± 1.0 67.8 ± 1.3 62.7 ± 0.5 70% COALA 10.75 28.0 ± 1.3 43.9 ± 1.0 41.2 ± 0.9 46.1 ± 0.5 35.0 ± 2.1 62.1 ± 1.1 59.3 ± 1.4 45.1 ± 0.5 SVD-LLM 10.75 27.2 ± 1.3 44.2 ± 1.0 41.6 ± 0.9 46.1 ± 0.5 34.6 ± 2.1 61.8 ± 1.1 59.2 ± 1.4 45.0 ± 0.5 SVD-LLM+Swift 10.58 27.7 ± 1.3 44.1 ± 1.0 41.9 ± 0.9 46.6 ± 0.5 34.8 ± 2.1 61.9 ± 1.1 60.5 ± 1.4 45.4 ± 0.5 COALA+Swift 10.58 27.5 ± 1.3 44.1 ± 1.0 42.0 ± 0.9 46.4 ± 0.5 34.4 ± 2.1 61.6 ± 1.1 60.3 ± 1.4 45.2 ± 0.5 Swift-SVD 10.58 28.1 ± 1.3 44.2 ± 1.0 43.1 ± 0.9 46.5 ± 0.5 35.0 ± 2.1 61.1 ± 1.1 59.4 ± 1.4 45.4 ± 0.5 LRCC-5 9.49 31.4 ± 1.4 46.3 ± 1.0 46.6 ± 0.9 56.0 ± 0.5 34.6 ± 2.1 67.8 ± 1.1 61.6 ± 1.4 49.2 ± 0.5 LRCC-9 9.05 31.9 ± 1.4 46.4 ± 1.0 41.7 ± 0.9 56.5 ± 0.5 34.4 ± 2.1 65.1 ± 1.1 61.5 ± 1.4 48.2 ± 0.5 Table 1: Predictive performance of Llama-2-7B within the indicated active-parameter budgets. LRCC achieves the highest average accuracy at all compression ratios. Results for Llama-2-7B compressed using 256 samples of sequence length 2048. Avg. reports mean accuracy over the seven reasoning tasks; Wiki2 reports perplexity. 3.1 Predictive Performances Table 1 shows the predictive performance results across selected ratios for Llama-2-7B (see Appendix D.1 for results on Llama-3.2-1B and Qwen3-4B). Since LRCC is dynamic, its actual average ratio of activated parameters slightly differs from the specified target costs. In cases where the actual cost was higher on a task than the target cost, we show the performance obtained using a router with lower target costs, to ensure a fairer comparison. The actual average costs for each model in each task shown are specified in Table 7. In Section 4.3, we show that the actual cost of the LRCC models follow closely their target costs across several datasets not seen during training. At the 90%90\% budget, LRCC-9 reaches 62.7%62.7\% average accuracy, compared with 55.1%55.1\% for the best static baseline, a gain of 7.67.6 percentage points. At the 70%70\% budget, LRCC-5 improves average accuracy from 45.4%45.4\% to 49.2%49.2\%. Both variants also improve WikiText-2 perplexity at these budgets. The gains extend across most individual tasks, showcasing the beneficial effect of introducing conditional computation to the low-rank decompositions. Several static baselines obtain similar results among themselves, particularly on Llama-2-7B. This is consistent with their shared activation-aware reconstruction objective: in exact arithmetic and under the same calibration and rank allocation, their decompositions target equivalent optimal output approximations, while their numerical procedures differ. Among the evaluated combinations, COALA+Swift generally gives the strongest static results on Llama, so we use it for the trade-off curves. Figure 2: Predictive performance versus actual active-parameter ratio for LRCC-5 and COALA+Swift. Average accuracy is the mean over the seven downstream tasks. Figure 2 compares LRCC-5 with COALA+Swift over a wider range of measured active-parameter ratios. We evaluate checkpoints with target costs from 0.30.3 to 0.950.95 in steps of 0.050.05 and plot their actual average costs on each dataset. The path-count ablation in Section 4.1 motivates using five paths and Appendix D.4 gives additional results. LRCC improves the quality–cost trade-off in almost all plotted settings, with particularly large gains on the downstream tasks. We hypothesize that aggressive static truncation removes subspaces important for particular tasks, leading to sharp performance drops (Ahmad et al., 2025). LRCC can retain such components for selected tokens while meeting an average budget, providing a possible explanation for the gains. We also observe a greater language-modeling benefit on Llama-3.2-1B. One possible explanation is that larger models are more robust to compression (Yuan et al., 2025). To test whether the quality gains extend beyond Llama, we compare LRCC with Swift-SVD on Qwen3-4B. Appendix D.1 reports the results and the measured-cost curves in Figure 7. LRCC improves perplexity and average downstream accuracy across the reported budgets. 3.2 Computational Performance We also measured the latency for models with target costs from 0.30.3 to 0.90.9 on samples from the test sets. Both LRCC and the baseline were compiled with TorchDynamo and TorchInductor using identical settings. Following Lin et al. (2024), we measured batch-size-1 autoregressive decoding, which is a setting relevant to local, resource-constrained deployment. Each measurement was repeated 5 times on a DGX Spark (NVIDIA GB10 Grace Blackwell) with 128 GB of unified memory. We teacher-forced the ground-truth task tokens through the models to ensure identical inputs for both deployments and measured latency as the time required to decode each complete sequence divided by its number of decoded tokens. For multiple-choice tasks, we timed each candidate continuation and treated all continuations associated with the same document as one sample cluster. Figure 3 shows the resulting latency–quality trade-offs. Appendix D.2 gives the latency tables and measurement protocol; Appendix D.4 reports compilation and prefill times. LRCC-5 achieves better predictive performance at the same latency on all tasks for Llama-3.2-1B and on downstream tasks for Llama-2-7B at higher latencies. The results are mixed for Llama-2-7B on WikiText-2. Figure 3: Pareto curves of LRCC-5 and COALA+Swift across various decoding latencies (ms/token) applied to Llama-2-7B and Llama-3.2-1B. The average accuracy is obtained by averaging the results on the commonsense reasoning downstream tasks. 4 Analysis LRCC assigns variable amounts of compute to different input samples by choosing a different path among available options. We now ask how many paths are needed, whether the routers use them differently across tokens, and whether the learned budgets transfer beyond the router-training data. 4.1 Effect of the Number of Paths LRCC requires determining the number of available paths within each block. To better understand the influence of this hyperparameter, we trained models with T∈2,3,5,9,10,17,33T∈\2,3,5,9,10,17,33\ paths per block and measured perplexity on the WikiText-2 validation set. Figure 4 shows the validation perplexity obtained across several target costs and number of paths. Perplexity increases sharply when only two paths are available and remains notably higher with three paths. The effects are particularly pronounced at lower target costs, suggesting that the limited granularity in these settings makes it harder to find a feasible solution. Increasing the number of paths beyond five provides comparatively little improvement in most settings. Additional results for Llama-3.2-1B, and Torch compilation, prefill and decoding times across various path cardinalities are shown in Appendix D.4. (a) (b) Figure 4: Ablation on the number of paths for Llama-2-7B. (a) shows WikiText-2 validation perplexity as a function of the number of paths for selected target costs, while (b) shows the results for all target costs from 0.30.3 to 0.9750.975 in steps of 0.0250.025. Hyperparameter configurations for which no run converged are shown in white. 4.2 Token-dependent Routing To test whether the routers collapse to a single path for all tokens, we analyzed their choices for 64 samples from WikiText-2, each comprising 2,048 tokens. Specifically, we measured the average computational cost across each block and the effective fraction of paths at different target costs. For each router, we define the effective fraction of paths as Feff=exp⁡(H⁡(f))T,F_eff= (H(f))T, (10) where f denotes the empirical distribution of path selections across tokens, H⁡(f)H(f) is its entropy computed with natural logarithms, and T is the number of available paths. This quantity equals 11 when all paths are selected with equal frequency and 1/T1/T when all tokens are assigned to a single path. Figure 5(a) shows the results for Llama-2-7B. (a) (b) Figure 5: Router choices for Llama-2-7B on WikiText-2. (a) Average per-block cost (top) and effective fraction of paths (bottom) for each target. The dots indicate blocks that are deployed without factorization. (b) Distributions of the costs assigned to word and non-word tokens for each target. The results show a backbone of blocks retained in dense form, alongside blocks that route tokens through different low-rank paths. As the target cost decreases, the number of blocks that exhibit conditional routing increases. Multiple paths remain in use, with some Transformer blocks using almost all available paths. Together, these observations show that the routed blocks do not collectively collapse to a static rank allocation, even without an explicit path-balancing loss. We also analyze whether certain tokens were more often assigned to low-cost paths. As shown in Figure 5(b), non-word tokens (i.e., tokens containing no alphanumeric characters) are more frequently assigned to lower-cost paths. This provides evidence that compute allocation varies with token type. Appendix D.5 reports similar patterns for Llama-3.2-1B. 4.3 Target vs. Actual Deployment Cost LRCC routers are fitted to use a certain amount of compute on average, but this is not strictly enforced during inference. Hence, we measured the actual ratio of activated parameters of LRCC across different datasets used for testing. Figure 6 shows the results for Llama-2-7B. Figure 6: Target versus actual deployment cost for Llama-2-7B with LRCC-5. LRCC costs remain close to the target across different datasets. Although the routers were trained only on WikiText-2, actual deployment costs remain close to their targets across the evaluated datasets. Appendix D.3 presents the results for Llama-3.2-1B and Qwen3-4B. 5 Related Work Low-rank Compression Several low-rank compression methods have been proposed in recent years (Shihab et al., 2026; Qi et al., 2026; Parkina and Rakhuba, 2025; Wang et al., 2025; Yuan et al., 2025; Qinsi et al., 2025). Zaccone et al. (2026) and Rauba and van der Schaar (2026) construct nested low-rank submodels to support multiple compute budgets within a single model. Concurrent work by Zhu et al. (2026) selects prompt-dependent singular components under fixed per-layer rank budgets and reuses the selected subspaces across decoding steps. These approaches offer either selectable global budgets or prompt-dependent subspaces. LRCC instead varies the active rank at each block for each token, under an average cost target. Conditional Computation Mixture-of-Experts (Shazeer et al., 2017) has become standard in LLM pretraining, and several works have proposed architectures with dynamic computational graphs (e.g., (Raposo et al., 2024; Ainslie et al., 2023; Devvrit et al., 2024)). However, these approaches generally require training the model from scratch. Like LRCC, some approaches adapt pretrained checkpoints. Moefication and upcycling are arguably among the best known (Zhang et al., 2022; Komatsuzaki et al., 2023; Szatkowski et al., 2024), but other forms have been studied (e.g. (Wójcik et al., 2025; Cai et al., 2024)). These techniques are generally either difficult to apply to existing models (e.g., Moefication is tailored to ReLU-induced activation sparsity) or require training additional compute modules, whereas LRCC trains only lightweight routers over fixed factors. 6 Conclusion In this work, we introduced LRCC, a technique for adding conditional computation to pretrained Transformers by training block-level routers over shared low-rank factors. LRCC improves predictive performance within fixed active-parameter budgets on Llama models, with gains of up to 7.67.6 percentage points in average downstream accuracy; the Qwen experiments extend the quality gains to another model family. At matched batch-size-1 decoding latency, LRCC improves perplexity and average downstream accuracy on Llama-3.2-1B and remains competitive on Llama-2-7B, without custom CUDA kernels. Our analysis shows that a small set of paths captures most of the quality gains and that the routers retain token-dependent path choices without an explicit balancing loss. Together, these results establish shared low-rank hierarchies as a practical basis for introducing conditional computation into pretrained models. AI use statement We used generative AI tools, including coding agents, to discuss, critique, and refine our methodological ideas, improve the exposition of proof sketches, and assist with implementing code, given our specifications of the algorithms and requirements. We also used generative AI tools for literature search and to revise text passages for clarity and presentation. AI tools were not used to independently formulate the proposed method, design or run experiments, interpret experimental results, or generate experimental claims. AI-assisted code was manually inspected, mathematical arguments were independently verified, and citations were checked against the original sources. We take responsibility for the final content of the paper and all AI-assisted work. Ethics statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Reproducibility statement Section 2.1 describes the activation-aware low-rank decomposition, Section 2.2 the construction of block-level paths, Section 2.3 the router training objective, and Section 2.4 the target-cost annealing procedure. Appendix B proves the low-rank approximation result. Appendix C describes the cost computation and compression objective for training, and Appendix E reports implementation details, training settings, and baseline hyperparameters. Appendix D.2 describes the latency-measurement protocol. Acknowledgments We acknowledge ISCRA for awarding this project access to the LEONARDO supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CINECA (Italy). Donatella Genovese is supported by PNRR MUR project PE0000013-FAIR. References Ahmad et al. (2025) A. Ahmad, A. Joshi, and A. Modi Beyond components: singular vector-based interpretability of transformer circuits. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §3.1. Ainslie et al. (2023) J. Ainslie, T. Lei, M. de Jong, S. Ontanon, S. Brahma, Y. Zemlyanskiy, D. Uthus, M. Guo, J. Lee-Thorp, Y. Tay, Y. Sung, and S. Sanghai CoLT5: faster long-range transformers with conditional computation. In The 2023 Conference on Empirical Methods in Natural Language Processing, External Links: Link Cited by: §5. Bisk et al. (2020) Y. Bisk, R. Zellers, R. Le Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), p. 7432–7439. External Links: Document Cited by: §3. Bommasani et al. (2021) R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. Cited by: §1. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 1877–1901. External Links: Link Cited by: §1. Cai et al. (2024) R. Cai, S. Muralidharan, G. Heinrich, H. Yin, Z. Wang, J. Kautz, and P. Molchanov Flextron: many-in-one flexible large language model. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §C.1, §5. Chen et al. (2021) P. Chen, H. Yu, I. S. Dhillon, and C. Hsieh DRONE: data-aware low-rank compression for large NLP models. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: §1. Clark et al. (2019) C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova BoolQ: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 2924–2936. External Links: Link, Document Cited by: §3. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: §3. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: §1. Devvrit et al. (2024) F. Devvrit, S. Kudugunta, A. Kusupati, T. Dettmers, K. Chen, I. S. Dhillon, Y. Tsvetkov, H. Hajishirzi, S. M. Kakade, A. Farhadi, and P. Jain MatFormer: nested transformer for elastic inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §5. Izenman (1975) A. J. Izenman Reduced-rank regression for the multivariate linear model. Journal of Multivariate Analysis 5 (2), p. 248–264. External Links: Document Cited by: Appendix B, Appendix B. Jang et al. (2017) E. Jang, S. Gu, and B. Poole Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, Cited by: §2.3. Komatsuzaki et al. (2023) A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and N. Houlsby Sparse upcycling: training mixture-of-experts from dense checkpoints. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §5. Kusupati et al. (2022) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. M. Kakade, P. Jain, and A. Farhadi Matryoshka representation learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §2. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han Awq: activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems 6, p. 87–100. Cited by: §3.2. Merity et al. (2017) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. In International Conference on Learning Representations, External Links: Link Cited by: §3. Meta AI (2024) Meta AI Llama 3.2 model card. Note: https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md Cited by: §3. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, p. 2381–2391. External Links: Link, Document Cited by: §3. Parkina and Rakhuba (2025) U. Parkina and M. Rakhuba COALA: numerically stable and efficient framework for context-aware low-rank approximation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §E.1, §2, §3, §3, §5. Pfeiffer et al. (2023) J. Pfeiffer, S. Ruder, I. Vulić, and E. M. Ponti Modular deep learning. arXiv preprint arXiv:2302.11529. Cited by: §1. Qi et al. (2026) R. Qi, Y. Liu, X. Wu, X. Wang, M. Li, C. Chen, J. Chen, Y. Chen, and Q. Weng Swift-SVD: theoretical optimality meets practical efficiency in low-rank LLM compression. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §E.1, §E.1, §1, §2.2, §2, §3, §3, §5. Qinsi et al. (2025) W. Qinsi, J. Ke, M. Tomizuka, K. Keutzer, and C. Xu Dobi-SVD: differentiable SVD for LLM compression and some new perspectives. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §5. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1. Raposo et al. (2024) D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro Mixture-of-depths: dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258. Cited by: §5. Rauba and van der Schaar (2026) P. Rauba and M. van der Schaar Deep hierarchical learning with nested subspace networks for large language models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Sakaguchi et al. (2020) K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), p. 8732–8740. External Links: Document Cited by: §3. Scardapane et al. (2024) S. Scardapane, A. Baiocchi, A. Devoto, V. Marsocci, P. Minervini, and J. Pomponi Conditional computation in neural networks: principles and research trends. Intelligenza Artificiale 18 (1), p. 175–190. Cited by: §1. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, External Links: Link Cited by: §5. Shihab et al. (2026) I. F. Shihab, S. Akter, and A. Sharma Beyond variance: knowledge-aware LLM compression via fisher-aligned subspace diagnostics. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 47938–47954. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: §5. Szatkowski et al. (2024) F. Szatkowski, B. Wójcik, M. Piórczyński, and S. Scardapane Exploiting activation sparsity with dense to dynamic-k mixture-of-experts conversion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §5. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §3. Wang et al. (2025) X. Wang, Y. Zheng, Z. Wan, and M. Zhang SVD-LLM: truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2, §3, §3, §5. Wójcik et al. (2025) B. Wójcik, A. Devoto, K. Pustelnik, P. Minervini, and S. Scardapane Adaptive computation modules: granular conditional computation for efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 21510–21518. Cited by: §5. Yang et al. (2025) A. Yang, A. Li, B. Yang, et al. Qwen3 technical report. External Links: 2505.09388, Link Cited by: §3. Yuan et al. (2025) Z. Yuan, Y. Shang, Y. Song, D. Yang, Q. Wu, Y. Yan, and G. Sun ASVD: activation-aware singular value decomposition for compressing large language models. External Links: 2312.05821, Link Cited by: §1, §2, §3.1, §5. Zaccone et al. (2026) R. Zaccone, S. Laskaridis, M. Ciccone, and S. Horváth FlexRank: nested low-rank knowledge decomposition for adaptive model deployment. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Appendix A, §E.1, §1, §5. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 4791–4800. External Links: Link, Document Cited by: §3. Zhang et al. (2022) Z. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou Moefication: transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022, p. 877–890. Cited by: §5. Zhu et al. (2026) H. Zhu, Z. Mi, G. L. Zhang, and S. Huang Different prompts, different ranks: prompt-aware dynamic rank selection for SVD-based LLM compression. arXiv preprint arXiv:2605.08568. Cited by: §D.4, §5. Appendix A Limitations Our work focuses on demonstrating that conditioning the activated ranks on input activations can improve the quality–compute trade-off of low-rank compressed models, but several directions remain open. First, we have not evaluated LRCC on nested subspace models that refine the factorized weights through fine-tuning, such as FlexRank (Zaccone et al., 2026). Combining such refinement with conditional rank selection may yield further gains. Second, we demonstrated latency improvements only for batch-size-1 decoding, and future work should investigate optimized implementations for prefill and larger batch sizes. Finally, our paths are constructed before router training from a fixed set of nested rank allocations. Jointly learning the path structure and the routing policy, or exploring more flexible routing granularities, could further improve the allocation of computation across inputs. Appendix B Low-rank Approximation Results This section establishes the optimality and nestedness of the activation-aware rank-r approximation used in Section 2.1, and its connection to reduced-rank regression (Izenman, 1975). Efficient computation of the factors is discussed in Appendix E.1. Optimality and nestedness. For simplicity, we omit the block and layer subscripts. Let ∈ℝdin×doutW ^d_in× d_out be a linear weight matrix and let ∈ℝn×dinX ^n× d_in contain its calibration inputs. We consider the rank-constrained reconstruction problem min~r⁡‖−~r‖Fs.t.rank⁡(~r)≤r. _ W_r \|XW-X W_r \|_F .t. ( W_r)≤ r. (11) Let ==⊤Y=XW=U (12) be an SVD of the layer outputs, with singular values ordered in decreasing order, and let :rV_:r denote the first r columns of V. We define ~r=:r:r⊤. W_r=WV_:rV_:r . (13) Since rank(:r:r⊤)≤rrank(WV_:rV_:r )≤ r, this is a feasible solution to Equation 11. Its output on the calibration inputs is ~r W_r =:r:r⊤ =XWV_:rV_:r (14) =:r:r⊤ =YV_:rV_:r (15) =:r:r:r⊤=r, =U_:r _:rV_:r =Y_r, (16) where rY_r is the truncated rank-r SVD of Y. For any other feasible matrix ^r W_r, rank⁡(^r)≤rank⁡(^r)≤r.rank(X W_r) ( W_r)≤ r. (17) Therefore, by the Eckart–Young–Mirsky theorem, ‖−~r‖F=‖−r‖F≤‖−^r‖F. \|XW-X W_r \|_F= \|Y-Y_r \|_F≤ \|XW-X W_r \|_F. (18) Hence, ~r W_r is a global optimum of Equation 11. Moreover, define the full factors =,=.A=WV, =V. (19) For every target rank r, the optimal approximation can then be obtained by indexing the first r columns of these same factors: ~r=:r:r⊤=(:r):r⊤. W_r=A_:rB_:r =(WV_:r)V_:r . (20) Thus, a single pair of ordered factors A and B simultaneously contains a globally optimal solution for every target rank. In particular, the solutions are nested, since increasing the rank from r to r+1r+1 only adds the next rank-one component, ~r+1=~r+(r+1)r+1⊤. W_r+1= W_r+(Wv_r+1)v_r+1 . (21) Connection to reduced-rank regression. Equation 11 can also be viewed as a special instance of reduced-rank regression (R) (Izenman, 1975). Given predictors X and multivariate responses Y, R solves min⁡‖−‖Fs.t.rank⁡()≤r. _C\|Y-XC\|_F .t. (C)≤ r. (22) Let OLSC_OLS be an ordinary least-squares solution and let ^OLS=OLS Y_OLS=XC_OLS. If :rV_:r contains the leading r right singular vectors of ^OLS Y_OLS, a reduced-rank solution is r=OLS:r:r⊤.C_r=C_OLSV_:rV_:r . (23) In our setting the responses are exactly the outputs of the known pretrained linear transform, =Y=XW. Consequently, W itself is an ordinary least-squares solution with zero residual, so that ^OLS= Y_OLS=XW. Substituting OLS=C_OLS=W into the R solution therefore recovers r=:r:r⊤=~r,C_r=WV_:rV_:r = W_r, (24) providing an alternative interpretation of Equation 13. Appendix C Router Training Details C.1 Router Training For each token representation entering block b, the router produces a probability distribution over its T paths. During both training and inference, the forward pass selects a single path using the highest router probability. The executed path and its parameter count are computed from this hard one-hot selection, while the straight-through estimator (STE) propagates gradients through the soft probabilities during training. To encourage the router to meet a target activated-parameter ratio t∈[0,1]t∈[0,1], we use a compression loss similar to Cai et al. (2024), defined over a batch of N sequences of length S as ℒc=max⁡(0,c−t),L_c= (0,c_ global-t), (25) where c=∑b=1Bsoftcapb⁡(cb)c_ global= _b=1^Bsoftcap_b(c_b) (26) is a differentiable surrogate for the global activated-parameter ratio. The blockwise soft cap approximates the cost of retaining a dense (i.e., non-factorized) block when its factorized implementation would be more expensive. We define these terms below. Let yijbm∈0,1y_ijbm∈\0,1\ indicate whether token j in sequence i selects path m in block b. The normalized average cost of block b is cb=1NSP0∑i=1N∑j=1S∑m=1TyijbmPbm,c_b= 1NSP_0 _i=1^N _j=1^S _m=1^Ty_ijbmP_bm, (27) where P0=∑b=1BPbdenseP_0= _b=1^BP_b^dense (28) is the total parameter count of the original weight matrices across all Transformer blocks (the embedding matrix and language-model output head are not counted, as they are not compressed). Thus, cbc_b is the contribution of block b to the ratio of activated parameters relative to the original model. The normalized dense (non-factorized) cost of block b is Cb=PbdenseP0.C_b= P_b^denseP_0. (29) The blockwise soft cap is defined as softcapb⁡(cb)=cb−Cbκsoftplus⁡(κ⁡(cb−Cb)Cb),softcap_b(c_b)=c_b- C_bκsoftplus\! ( κ(c_b-C_b)C_b ), (30) where softplus⁡(x)=log⁡(1+ex)softplus(x)= (1+e^x) and κ controls the slope. This function smoothly approximates min⁡(cb,Cb) (c_b,C_b) and approaches CbC_b as the factorized block’s average cost increases above the dense cost. Since ∑bCb=1 _bC_b=1, all block costs use the original model’s Transformer weights as their common denominator. A hard cap would limit the maximum cost to CbC_b, reflecting that inefficient blocks (i.e., where cb≥Cbc_b≥ C_b) are retained in dense form at deployment. However, it would also create a zero-gradient region, preventing the compression loss from providing an optimization signal. The soft cap instead retains gradients above the dense cost, but is slightly lower than the hard-capped cost. We therefore report actual deployment costs separately in Sections 4.3 and D.3. During training, we freeze all model parameters except the routers and minimize the sum of the task and compression losses ℒ=ℒtask+λsℒc,L=L_task+ _sL_c, (31) where λs _s is an adaptive compression-loss weight, updated at each training step s according to the constraint violation as described in Section 2.4. The target-cost schedule and training hyperparameters are given in Appendix E. Appendix D Additional Experiments D.1 Additional Predictive Performance Results Similarly to the results presented in Section 3.1, we conduct an analysis of the downstream performance of LRCC compared to COALA, SVD-LLM, SVD-LLM, SVD-LLM+Swift, COALA+Swift, Swift-SVD (see Section 3.1 for more details). We present the complete results for Llama-2-7B, Llama-3.2-1B and Qwen3-4B respectively in Table 2, Table 3 and Table 4. As can be observed from the two tables, LRCC-5 and LRCC-9 continue to surpass static methods in nearly all evaluations. This demonstrates that models benefit from dynamically allocating different ranks depending on the tokens’ needed accuracy, highlighting the strong assumptions static approaches do about models’ computations. In Figure 7 we compare LRCC-5 with Swift-SVD on Qwen3-4B across measured active-parameter ratios. LRCC achieves lower perplexity at every ratio, with the largest gap under aggressive compression, where Swift-SVD degrades sharply. On the downstream tasks, LRCC matches or exceeds Swift-SVD throughout, with the largest gains at intermediate ratios. These trends mirror the Llama results in Figure 2, suggesting that the benefits of LRCC are not specific to one model family. Figure 7: Pareto curves of LRCC-5 and Swift-SVDacross various cost ratios applied to Qwen3-4B. The average accuracy is obtained by averaging the results on the commonsense reasoning downstream tasks. PPL ↓ Commonsense reasoning accuracy (%) ↑ Ratio Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino Avg. 100% Base 5.49 44.80 73.95 79.42 76.08 44.00 78.62 69.30 66.60 90% COALA 7.27 35.5 ± 1.4 57.5 ± 1.0 49.3 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 70.6 ± 1.1 65.2 ± 1.3 54.2 ± 0.5 SVD-LLM 7.27 36.2 ± 1.4 57.4 ± 1.0 50.1 ± 0.9 62.8 ± 0.5 39.0 ± 2.2 70.8 ± 1.1 65.3 ± 1.3 54.5 ± 0.5 SVD-LLM+Swift 7.33 35.5 ± 1.4 58.3 ± 1.0 55.0 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 71.1 ± 1.1 62.9 ± 1.4 54.8 ± 0.5 COALA+Swift 7.27 35.1 ± 1.4 58.4 ± 1.0 57.4 ± 0.9 62.8 ± 0.5 38.4 ± 2.2 70.9 ± 1.1 63.1 ± 1.4 55.1 ± 0.5 Swift-SVD 7.33 35.6 ± 1.4 58.7 ± 1.0 55.4 ± 0.9 62.8 ± 0.5 39.6 ± 2.2 70.7 ± 1.1 62.9 ± 1.4 55.1 ± 0.5 LRCC-5 6.17 41.8 ± 1.4 68.5 ± 1.0 74.4 ± 0.8 71.0 ± 0.5 41.4 ± 2.2 74.8 ± 1.0 66.3 ± 1.3 62.6 ± 0.5 LRCC-9 6.03 40.4 ± 1.4 68.6 ± 1.0 74.9 ± 0.8 71.7 ± 0.4 40.8 ± 2.2 74.8 ± 1.0 67.8 ± 1.3 62.7 ± 0.5 70% COALA 10.75 28.0 ± 1.3 43.9 ± 1.0 41.2 ± 0.9 46.1 ± 0.5 35.0 ± 2.1 62.1 ± 1.1 59.3 ± 1.4 45.1 ± 0.5 SVD-LLM 10.75 27.2 ± 1.3 44.2 ± 1.0 41.6 ± 0.9 46.1 ± 0.5 34.6 ± 2.1 61.8 ± 1.1 59.2 ± 1.4 45.0 ± 0.5 SVD-LLM+Swift 10.58 27.7 ± 1.3 44.1 ± 1.0 41.9 ± 0.9 46.6 ± 0.5 34.8 ± 2.1 61.9 ± 1.1 60.5 ± 1.4 45.4 ± 0.5 COALA+Swift 10.58 27.5 ± 1.3 44.1 ± 1.0 42.0 ± 0.9 46.4 ± 0.5 34.4 ± 2.1 61.6 ± 1.1 60.3 ± 1.4 45.2 ± 0.5 Swift-SVD 10.58 28.1 ± 1.3 44.2 ± 1.0 43.1 ± 0.9 46.5 ± 0.5 35.0 ± 2.1 61.1 ± 1.1 59.4 ± 1.4 45.4 ± 0.5 LRCC-5 9.49 31.4 ± 1.4 46.3 ± 1.0 46.6 ± 0.9 56.0 ± 0.5 34.6 ± 2.1 67.8 ± 1.1 61.6 ± 1.4 49.2 ± 0.5 LRCC-9 9.05 31.9 ± 1.4 46.4 ± 1.0 41.7 ± 0.9 56.5 ± 0.5 34.4 ± 2.1 65.1 ± 1.1 61.5 ± 1.4 48.2 ± 0.5 50% COALA 33.12 22.7 ± 1.2 30.9 ± 0.9 37.8 ± 0.8 31.0 ± 0.5 26.6 ± 2.0 53.2 ± 1.2 53.9 ± 1.4 36.6 ± 0.5 SVD-LLM 33.64 23.0 ± 1.2 30.9 ± 0.9 37.8 ± 0.8 31.1 ± 0.5 27.0 ± 2.0 53.0 ± 1.2 54.5 ± 1.4 36.8 ± 0.5 SVD-LLM+Swift 32.10 23.0 ± 1.2 30.8 ± 0.9 37.8 ± 0.8 31.3 ± 0.5 26.6 ± 2.0 53.1 ± 1.2 54.7 ± 1.4 36.8 ± 0.5 COALA+Swift 32.10 23.0 ± 1.2 30.5 ± 0.9 37.8 ± 0.8 31.4 ± 0.5 26.6 ± 2.0 52.8 ± 1.2 54.6 ± 1.4 36.7 ± 0.5 Swift-SVD 32.60 23.1 ± 1.2 30.9 ± 0.9 37.8 ± 0.8 31.4 ± 0.5 27.0 ± 2.0 52.4 ± 1.2 55.1 ± 1.4 36.8 ± 0.5 LRCC-5 41.86 24.8 ± 1.3 32.2 ± 1.0 37.8 ± 0.8 32.7 ± 0.5 27.8 ± 2.0 55.3 ± 1.2 52.9 ± 1.4 37.6 ± 0.5 LRCC-9 41.86 22.8 ± 1.2 33.2 ± 1.0 37.8 ± 0.8 32.1 ± 0.5 30.0 ± 2.1 56.1 ± 1.2 53.0 ± 1.4 37.8 ± 0.5 Table 2: Performance of Llama-2-7B compressed using 256 samples with sequence length 2048 on a language modeling dataset (measured using perplexity) and seven commonsense reasoning datasets (measured using accuracy). PPL ↓ Commonsense reasoning accuracy (%) ↑ Ratio Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino Avg. 100% Base 9.79 36.69 61.95 63.61 64.23 37.00 74.65 61.01 57.02 90% COALA 22.76 25.9 ± 1.3 38.4 ± 1.0 48.3 ± 0.9 35.0 ± 0.5 27.0 ± 2.0 56.4 ± 1.2 52.6 ± 1.4 40.5 ± 0.5 SVD-LLM 23.12 25.6 ± 1.3 38.7 ± 1.0 47.6 ± 0.9 34.8 ± 0.5 27.6 ± 2.0 56.4 ± 1.2 52.3 ± 1.4 40.4 ± 0.5 SVD-LLM+Swift 21.05 26.7 ± 1.3 41.3 ± 1.0 54.3 ± 0.9 38.9 ± 0.5 28.2 ± 2.0 59.4 ± 1.1 55.5 ± 1.4 43.5 ± 0.5 COALA+Swift 19.77 26.5 ± 1.3 44.4 ± 1.0 54.6 ± 0.9 42.1 ± 0.5 31.2 ± 2.1 62.2 ± 1.1 56.2 ± 1.4 45.3 ± 0.5 Swift-SVD 21.05 25.7 ± 1.3 38.8 ± 1.0 51.0 ± 0.9 36.7 ± 0.5 27.6 ± 2.0 57.7 ± 1.2 52.6 ± 1.4 41.4 ± 0.5 LRCC-5 11.81 32.6 ± 1.4 54.2 ± 1.0 49.6 ± 0.9 51.5 ± 0.5 34.2 ± 2.1 70.3 ± 1.1 56.7 ± 1.4 49.9 ± 0.5 LRCC-9 11.62 31.3 ± 1.4 53.1 ± 1.0 46.3 ± 0.9 52.5 ± 0.5 35.6 ± 2.1 70.5 ± 1.1 54.7 ± 1.4 49.1 ± 0.5 70% COALA 209.30 22.1 ± 1.2 28.4 ± 0.9 37.8 ± 0.8 27.7 ± 0.4 25.4 ± 1.9 52.5 ± 1.2 50.4 ± 1.4 34.9 ± 0.5 SVD-LLM 222.79 22.4 ± 1.2 28.2 ± 0.9 37.8 ± 0.8 28.0 ± 0.4 25.0 ± 1.9 52.0 ± 1.2 49.7 ± 1.4 34.7 ± 0.5 SVD-LLM+Swift 135.13 22.7 ± 1.2 29.0 ± 0.9 37.8 ± 0.8 28.1 ± 0.4 24.8 ± 1.9 53.0 ± 1.2 51.1 ± 1.4 35.2 ± 0.5 COALA+Swift 130.97 22.4 ± 1.2 28.8 ± 0.9 37.8 ± 0.8 28.1 ± 0.4 24.8 ± 1.9 52.6 ± 1.2 50.8 ± 1.4 35.1 ± 0.5 Swift-SVD 157.98 21.6 ± 1.2 28.5 ± 0.9 37.8 ± 0.8 28.0 ± 0.4 25.0 ± 1.9 52.3 ± 1.2 51.5 ± 1.4 35.0 ± 0.5 LRCC-5 24.23 22.7 ± 1.2 37.8 ± 1.0 38.6 ± 0.9 35.3 ± 0.5 27.4 ± 2.0 58.2 ± 1.2 50.3 ± 1.4 38.6 ± 0.5 LRCC-9 23.48 24.9 ± 1.3 38.7 ± 1.0 42.0 ± 0.9 34.3 ± 0.5 28.2 ± 2.0 59.6 ± 1.1 51.9 ± 1.4 40.0 ± 0.5 50% COALA 998.50 22.6 ± 1.2 25.6 ± 0.9 37.8 ± 0.8 27.0 ± 0.4 25.6 ± 2.0 49.6 ± 1.2 50.4 ± 1.4 34.1 ± 0.5 SVD-LLM 967.78 22.9 ± 1.2 25.8 ± 0.9 37.8 ± 0.8 27.0 ± 0.4 25.2 ± 1.9 49.9 ± 1.2 50.3 ± 1.4 34.1 ± 0.5 SVD-LLM+Swift 854.06 23.9 ± 1.2 26.1 ± 0.9 37.8 ± 0.8 26.7 ± 0.4 24.4 ± 1.9 50.1 ± 1.2 49.3 ± 1.4 34.1 ± 0.5 COALA+Swift 827.78 23.4 ± 1.2 26.2 ± 0.9 37.8 ± 0.8 26.7 ± 0.4 24.6 ± 1.9 50.1 ± 1.2 49.5 ± 1.4 34.1 ± 0.5 Swift-SVD 938.00 23.3 ± 1.2 26.3 ± 0.9 37.8 ± 0.8 26.5 ± 0.4 24.8 ± 1.9 51.1 ± 1.2 50.1 ± 1.4 34.3 ± 0.5 LRCC-5 79.44 21.8 ± 1.2 33.9 ± 1.0 37.8 ± 0.8 29.2 ± 0.5 26.8 ± 2.0 53.8 ± 1.2 50.6 ± 1.4 36.3 ± 0.5 LRCC-9 229.87 21.2 ± 1.2 33.7 ± 1.0 37.9 ± 0.8 27.8 ± 0.4 25.6 ± 2.0 54.5 ± 1.2 49.9 ± 1.4 35.8 ± 0.5 Table 3: Performance of Llama-3.2-1B compressed using 256 samples with sequence length 2048 on a language modeling dataset (measured using perplexity) and seven commonsense reasoning datasets (measured using accuracy). PPL ↓ Commonsense reasoning accuracy (%) ↑ Ratio Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino Avg. Swift-SVD 17.18 38.1 ± 1.4 56.4 ± 1.0 76.2 ± 0.7 58.1 ± 0.5 34.8 ± 2.1 68.4 ± 1.1 60.7 ± 1.4 56.1 ± 0.5 LRCC-5 13.17 41.1 ± 1.4 63.5 ± 1.0 77.3 ± 0.7 58.0 ± 0.5 35.2 ± 2.1 70.3 ± 1.1 61.3 ± 1.4 58.1 ± 0.5 90% LRCC-9 12.77 42.7 ± 1.4 65.5 ± 1.0 68.8 ± 0.8 59.0 ± 0.5 35.2 ± 2.1 71.7 ± 1.1 59.9 ± 1.4 57.5 ± 0.5 Swift-SVD 27.45 25.0 ± 1.3 37.9 ± 1.0 48.5 ± 0.9 37.1 ± 0.5 29.2 ± 2.0 57.9 ± 1.2 51.9 ± 1.4 41.1 ± 0.5 LRCC-5 19.17 27.9 ± 1.3 46.5 ± 1.0 60.8 ± 0.9 44.9 ± 0.5 30.8 ± 2.1 62.7 ± 1.1 53.0 ± 1.4 46.7 ± 0.5 70% LRCC-9 18.00 29.1 ± 1.3 48.1 ± 1.0 45.9 ± 0.9 42.2 ± 0.5 34.4 ± 2.1 65.2 ± 1.1 52.7 ± 1.4 45.4 ± 0.5 Swift-SVD 112.03 23.9 ± 1.2 28.2 ± 0.9 38.2 ± 0.8 29.4 ± 0.5 24.0 ± 1.9 52.9 ± 1.2 50.3 ± 1.4 35.3 ± 0.5 LRCC-5 52.92 23.6 ± 1.2 36.4 ± 1.0 37.8 ± 0.8 32.4 ± 0.5 27.4 ± 2.0 55.5 ± 1.2 51.2 ± 1.4 37.8 ± 0.5 50% LRCC-9 45.26 21.9 ± 1.2 32.9 ± 1.0 37.8 ± 0.8 32.5 ± 0.5 25.0 ± 1.9 54.0 ± 1.2 50.9 ± 1.4 36.4 ± 0.5 Table 4: Performance of Qwen3-4B compressed using 256 samples with sequence length 2048 on a language modeling dataset (measured using perplexity) and seven commonsense reasoning datasets (measured using accuracy). D.2 Computational Performance Results This section provides the measurement protocol and per-task latency results for Section 3.2. We measured the latency of LRCC compared with COALA+Swift, a representative low-rank compression approach, across several target costs and downstream tasks. We teacher-forced the ground-truth task tokens through the model to ensure identical inputs for both deployments and measured latency as the time required to decode each complete sequence divided by its number of decoded tokens. For multiple-choice tasks, we timed each candidate continuation and treated all continuations associated with the same document as one sample cluster. All measurements used a batch size of 1. The latency estimates are based on four randomly selected 2,048-token chunks from WikiText-2, 16 documents from WinoGrande, eight from OpenBookQA, and four from each remaining task: ARC-Challenge, ARC-Easy, BoolQ, HellaSwag, and PIQA. These sample counts were selected based on pilot validation runs, with additional samples allocated to datasets exhibiting greater document-level variability. Paired document-cluster bootstrap intervals for the final workloads had relative half-widths of at most 2.14%2.14\%. All candidate continuations from a sampled document were kept together during resampling to avoid treating correlated choices as independent observations. Both methods were compiled using TorchDynamo and TorchInductor with identical settings. CUDA graphs were disabled because we found their support limited for conditional methods and to ensure a fair comparison. We performed three 64-token warm-up runs before each measurement to exclude compilation and initialization from the timed region (compilation and prefill times are explored in Appendix D.4). Experiments were conducted on an NVIDIA GB10 Grace Blackwell with 128 GB of unified memory. We repeated each complete experiment five times to estimate run-to-run variability. Each table entry reports the mean latency in milliseconds per token across the five executions, and the subscripted ± term gives the half-width of a 95% Student-t confidence interval computed from the five execution-level measurements (n=5n=5, df=4df=4). Table 5 and Table 6 show the results. At the same nominal target cost, LRCC is slower on Llama-2-7B, while the ordering varies with the target cost on Llama-3.2-1B. LRCC introduces overhead from evaluating the router and dispatching the selected computational path, but its executed cost also depends on the routing decisions. As discussed in Section 3, at matched latencies, LRCC produces competitive predictive performance and outperforms the baseline in several settings. Furthermore, our implementation does not use specialized fused CUDA kernels, leaving opportunities for further speedups. For example, fusing the router with layer normalization could reduce the number of times the residual stream is read from memory. Target Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino 90% COALA+ 66.98±0.0866.98_ ± 0.08 67.01±0.0967.01_ ± 0.09 67.01±0.0967.01_ ± 0.09 66.99±0.0866.99_ ± 0.08 67.00±0.0967.00_ ± 0.09 67.02±0.1067.02_ ± 0.10 67.01±0.0967.01_ ± 0.09 67.01±0.0967.01_ ± 0.09 LRCC-5 70.83±0.1370.83_ ± 0.13 70.96±0.1270.96_ ± 0.12 70.98±0.1270.98_ ± 0.12 71.32±0.1371.32_ ± 0.13 71.51±0.1371.51_ ± 0.13 70.54±0.1170.54_ ± 0.11 71.37±0.1271.37_ ± 0.12 70.36±0.1170.36_ ± 0.11 80% COALA+ 60.29±0.1060.29_ ± 0.10 60.32±0.1160.32_ ± 0.11 60.33±0.1160.33_ ± 0.11 60.31±0.1060.31_ ± 0.10 60.32±0.1060.32_ ± 0.10 60.33±0.1160.33_ ± 0.11 60.35±0.1560.35_ ± 0.15 60.33±0.1160.33_ ± 0.11 LRCC-5 64.73±0.1064.73_ ± 0.10 64.86±0.1164.86_ ± 0.11 65.10±0.1165.10_ ± 0.11 65.00±0.1065.00_ ± 0.10 65.34±0.1165.34_ ± 0.11 65.21±0.1165.21_ ± 0.11 65.71±0.1165.71_ ± 0.11 65.01±0.1165.01_ ± 0.11 70% COALA+ 54.31±0.0754.31_ ± 0.07 54.34±0.0754.34_ ± 0.07 54.37±0.0654.37_ ± 0.06 54.35±0.0654.35_ ± 0.06 54.34±0.0654.34_ ± 0.06 54.38±0.0754.38_ ± 0.07 54.34±0.0654.34_ ± 0.06 54.36±0.0754.36_ ± 0.07 LRCC-5 57.96±0.0457.96_ ± 0.04 59.19±0.0759.19_ ± 0.07 59.53±0.0759.53_ ± 0.07 58.54±0.0658.54_ ± 0.06 59.25±0.0659.25_ ± 0.06 59.54±0.0859.54_ ± 0.08 60.30±0.0760.30_ ± 0.07 59.15±0.0759.15_ ± 0.07 60% COALA+ 48.09±0.0748.09_ ± 0.07 48.13±0.0948.13_ ± 0.09 48.14±0.1048.14_ ± 0.10 48.12±0.0848.12_ ± 0.08 48.12±0.0848.12_ ± 0.08 48.12±0.0848.12_ ± 0.08 48.12±0.0848.12_ ± 0.08 48.12±0.0848.12_ ± 0.08 LRCC-5 51.05±0.0551.05_ ± 0.05 53.06±0.0653.06_ ± 0.06 53.14±0.0653.14_ ± 0.06 51.87±0.0551.87_ ± 0.05 52.56±0.0552.56_ ± 0.05 53.12±0.0753.12_ ± 0.07 53.37±0.0653.37_ ± 0.06 52.34±0.0652.34_ ± 0.06 50% COALA+ 41.89±0.0441.89_ ± 0.04 41.91±0.0541.91_ ± 0.05 41.91±0.0541.91_ ± 0.05 41.92±0.0541.92_ ± 0.05 41.92±0.0541.92_ ± 0.05 41.89±0.0541.89_ ± 0.05 41.90±0.0441.90_ ± 0.04 41.89±0.0441.89_ ± 0.04 LRCC-5 44.39±0.0644.39_ ± 0.06 45.31±0.1145.31_ ± 0.11 45.28±0.0945.28_ ± 0.09 44.74±0.0844.74_ ± 0.08 45.00±0.0945.00_ ± 0.09 44.64±0.1144.64_ ± 0.11 45.43±0.1045.43_ ± 0.10 44.85±0.1144.85_ ± 0.11 40% COALA+ 35.76±0.0835.76_ ± 0.08 35.79±0.0735.79_ ± 0.07 35.78±0.0735.78_ ± 0.07 35.80±0.0735.80_ ± 0.07 35.79±0.0735.79_ ± 0.07 35.76±0.0735.76_ ± 0.07 35.78±0.0635.78_ ± 0.06 35.77±0.0735.77_ ± 0.07 LRCC-5 37.73±0.0837.73_ ± 0.08 38.63±0.1038.63_ ± 0.10 38.88±0.1038.88_ ± 0.10 38.30±0.0938.30_ ± 0.09 38.41±0.1038.41_ ± 0.10 38.12±0.1238.12_ ± 0.12 39.02±0.1039.02_ ± 0.10 38.12±0.1238.12_ ± 0.12 30% COALA+ 29.35±0.0829.35_ ± 0.08 29.41±0.0229.41_ ± 0.02 29.41±0.0229.41_ ± 0.02 29.42±0.0229.42_ ± 0.02 29.42±0.0229.42_ ± 0.02 29.41±0.0329.41_ ± 0.03 29.41±0.0229.41_ ± 0.02 29.41±0.0329.41_ ± 0.03 LRCC-5 30.82±0.0830.82_ ± 0.08 32.08±0.0932.08_ ± 0.09 32.43±0.1032.43_ ± 0.10 31.58±0.0831.58_ ± 0.08 31.81±0.0931.81_ ± 0.09 31.39±0.1131.39_ ± 0.11 32.69±0.1732.69_ ± 0.17 31.72±0.1231.72_ ± 0.12 Table 5: Decode latency (ms/token) for LRCC-5 and COALA+Swift (shown as COALA+) at matched target costs for Llama-2-7B at batch size 1. Each value corresponds to the mean latency across five benchmark executions. We also show the 95% Student-t confidence interval half-width (n=5n=5, df=4df=4). Both methods use identical tokenized inputs in each execution. Target Method Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino 90% COALA+ 15.34±0.0715.34_ ± 0.07 15.37±0.0715.37_ ± 0.07 15.35±0.0715.35_ ± 0.07 15.36±0.0715.36_ ± 0.07 15.36±0.0715.36_ ± 0.07 15.35±0.0715.35_ ± 0.07 15.35±0.0715.35_ ± 0.07 15.35±0.0715.35_ ± 0.07 LRCC-5 15.77±0.0915.77_ ± 0.09 15.98±0.0715.98_ ± 0.07 15.94±0.0715.94_ ± 0.07 15.90±0.0715.90_ ± 0.07 15.97±0.0715.97_ ± 0.07 16.02±0.0616.02_ ± 0.06 16.03±0.0716.03_ ± 0.07 15.96±0.0615.96_ ± 0.06 80% COALA+ 16.37±0.0716.37_ ± 0.07 16.34±0.0616.34_ ± 0.06 16.33±0.0616.33_ ± 0.06 16.38±0.0616.38_ ± 0.06 16.36±0.0616.36_ ± 0.06 16.28±0.0616.28_ ± 0.06 16.32±0.0616.32_ ± 0.06 16.29±0.0616.29_ ± 0.06 LRCC-5 14.90±0.0414.90_ ± 0.04 15.21±0.0515.21_ ± 0.05 15.27±0.0515.27_ ± 0.05 15.09±0.0515.09_ ± 0.05 15.18±0.0515.18_ ± 0.05 15.47±0.0415.47_ ± 0.04 15.27±0.0415.27_ ± 0.04 15.40±0.0415.40_ ± 0.04 70% COALA+ 15.15±0.0715.15_ ± 0.07 15.12±0.0715.12_ ± 0.07 15.11±0.0715.11_ ± 0.07 15.15±0.0715.15_ ± 0.07 15.13±0.0715.13_ ± 0.07 15.06±0.0715.06_ ± 0.07 15.10±0.0715.10_ ± 0.07 15.06±0.0715.06_ ± 0.07 LRCC-5 14.18±0.1114.18_ ± 0.11 14.61±0.1114.61_ ± 0.11 14.62±0.1114.62_ ± 0.11 14.42±0.1114.42_ ± 0.11 14.53±0.1114.53_ ± 0.11 15.06±0.1215.06_ ± 0.12 14.82±0.1114.82_ ± 0.11 14.74±0.1214.74_ ± 0.12 60% COALA+ 14.13±0.0814.13_ ± 0.08 14.10±0.0814.10_ ± 0.08 14.10±0.0814.10_ ± 0.08 14.14±0.0814.14_ ± 0.08 14.13±0.0814.13_ ± 0.08 14.05±0.0814.05_ ± 0.08 14.09±0.0814.09_ ± 0.08 14.05±0.0814.05_ ± 0.08 LRCC-5 13.22±0.0513.22_ ± 0.05 13.84±0.0713.84_ ± 0.07 13.96±0.0713.96_ ± 0.07 13.57±0.0713.57_ ± 0.07 13.64±0.0713.64_ ± 0.07 14.40±0.0714.40_ ± 0.07 14.00±0.0714.00_ ± 0.07 13.92±0.0713.92_ ± 0.07 50% COALA+ 13.41±0.0913.41_ ± 0.09 13.36±0.0813.36_ ± 0.08 13.35±0.0813.35_ ± 0.08 13.41±0.0813.41_ ± 0.08 13.39±0.0813.39_ ± 0.08 13.28±0.0813.28_ ± 0.08 13.34±0.0813.34_ ± 0.08 13.28±0.0813.28_ ± 0.08 LRCC-5 12.68±0.0412.68_ ± 0.04 13.55±0.0513.55_ ± 0.05 13.51±0.0513.51_ ± 0.05 13.03±0.0413.03_ ± 0.04 13.17±0.0513.17_ ± 0.05 14.03±0.0714.03_ ± 0.07 13.76±0.0613.76_ ± 0.06 13.38±0.0713.38_ ± 0.07 40% COALA+ 12.41±0.0812.41_ ± 0.08 12.38±0.0812.38_ ± 0.08 12.37±0.0812.37_ ± 0.08 12.41±0.0812.41_ ± 0.08 12.40±0.0812.40_ ± 0.08 12.32±0.0812.32_ ± 0.08 12.36±0.0812.36_ ± 0.08 12.33±0.0812.33_ ± 0.08 LRCC-5 13.56±0.0513.56_ ± 0.05 14.12±0.0514.12_ ± 0.05 14.23±0.0514.23_ ± 0.05 13.78±0.0713.78_ ± 0.07 13.74±0.0613.74_ ± 0.06 14.40±0.0714.40_ ± 0.07 14.31±0.0714.31_ ± 0.07 13.89±0.0613.89_ ± 0.06 30% COALA+ 11.59±0.0411.59_ ± 0.04 11.55±0.0411.55_ ± 0.04 11.54±0.0411.54_ ± 0.04 11.59±0.0411.59_ ± 0.04 11.57±0.0411.57_ ± 0.04 11.48±0.0411.48_ ± 0.04 11.53±0.0411.53_ ± 0.04 11.49±0.0411.49_ ± 0.04 LRCC-5 12.55±0.0512.55_ ± 0.05 12.95±0.0612.95_ ± 0.06 13.10±0.0513.10_ ± 0.05 12.73±0.0612.73_ ± 0.06 12.69±0.0712.69_ ± 0.07 13.17±0.0613.17_ ± 0.06 13.07±0.0513.07_ ± 0.05 12.85±0.0712.85_ ± 0.07 Table 6: Decode latency (ms/token) for LRCC-5 and COALA+Swift (shown as COALA+) at matched target costs for Llama-3.2-1B at batch size 1. Each value corresponds to the mean latency across five benchmark executions. We also show the 95% Student-t confidence interval half-width (n=5n=5, df=4df=4). Both methods use identical tokenized inputs in each execution. D.3 Additional Results on Target vs. Actual Deployment Cost Due to the input-dependence of the LRCCmethod, the average ratio of activated parameters is not strictly the same as the specified in the target cost. Table 7 shows the actual ratio of activated parameters in the different datasets, corresponding to the same checkpoints whose results were reported in Appendix D and for Llama-2-7B in Table 1. Ratio Method Model Wiki2 ARC-C ARC-E BoolQ Hella OBQA PIQA Wino Avg. 90% LRCC-5 Llama-2-7B 0.90 0.86 0.86 0.86 0.86 0.90 0.87 0.90 0.87 LRCC-9 Llama-2-7B 0.90 0.87 0.87 0.87 0.87 0.87 0.88 0.87 0.87 LRCC-5 Llama-3.2-1B 0.90 0.87 0.88 0.87 0.87 0.90 0.88 0.89 0.88 LRCC-9 Llama-3.2-1B 0.90 0.86 0.86 0.86 0.86 0.88 0.87 0.87 0.87 LRCC-5 Qwen3-4B 0.91 0.90 0.90 0.91 0.90 0.90 0.91 0.89 0.90 LRCC-9 Qwen3-4B 0.91 0.91 0.91 0.91 0.91 0.91 0.91 0.91 0.91 70% LRCC-5 Llama-2-7B 0.70 0.68 0.69 0.67 0.68 0.69 0.69 0.68 0.68 LRCC-9 Llama-2-7B 0.70 0.70 0.70 0.67 0.68 0.70 0.66 0.69 0.68 LRCC-5 Llama-3.2-1B 0.70 0.65 0.65 0.68 0.68 0.68 0.65 0.64 0.66 LRCC-9 Llama-3.2-1B 0.70 0.70 0.66 0.68 0.68 0.69 0.65 0.70 0.68 LRCC-5 Qwen3-4B 0.71 0.69 0.69 0.72 0.72 0.71 0.70 0.70 0.70 LRCC-9 Qwen3-4B 0.71 0.69 0.69 0.73 0.67 0.71 0.70 0.69 0.70 50% LRCC-5 Llama-2-7B 0.45 0.47 0.47 0.46 0.46 0.48 0.47 0.47 0.47 LRCC-9 Llama-2-7B 0.45 0.47 0.47 0.46 0.46 0.48 0.47 0.47 0.47 LRCC-5 Llama-3.2-1B 0.50 0.47 0.47 0.48 0.47 0.50 0.47 0.46 0.47 LRCC-9 Llama-3.2-1B 0.45 0.49 0.49 0.46 0.46 0.45 0.49 0.49 0.48 LRCC-5 Qwen3-4B 0.50 0.52 0.53 0.51 0.51 0.50 0.48 0.49 0.51 LRCC-9 Qwen3-4B 0.50 0.49 0.49 0.51 0.51 0.50 0.49 0.48 0.50 Table 7: Actual deployment cost (activated-parameter ratio) for the LRCC-5 and LRCC-9 configurations used during evaluation. Additionally, Figure 8 and Figure 9 show the actual deployment costs versus the target cost used during training for Llama-3.2-1B and Qwen3-4B, similarly to Section 4.3. Figure 8: Target versus actual deployment cost for Llama-3.2-1B with LRCC-5. Each panel shows one evaluation task; the dashed line indicates exact agreement between the target and actual deployment cost. Figure 9: Target versus actual deployment cost for Qwen3-4B with LRCC-5. Each panel shows one evaluation task; the dashed line indicates exact agreement between the target and actual deployment cost. D.4 Ablation on the Number of Paths As described in Section 4.1, we trained models with P∈2,3,5,9,10,17,33P∈\2,3,5,9,10,17,33\ paths per block to understand the influence of this hyperparameter. Each path corresponds to a rank assignment with a target cost uniformly spaced between 0.10.1 and 1.01.0. At each refinement, we retain the previous paths and add paths with target costs midway between them. For example, with P=2P=2, we use paths constructed by the rank-allocation algorithm with target costs of 0.10.1 and 1.01.0. For P=3P=3, we add a path with target cost 0.550.55. The results for Llama-2-7B are shown in Figure 4. Figure 10 shows the perplexities for Llama-3.2-1B. Perplexity increases sharply with only two paths, but increasing the number of choices beyond five provides comparatively little improvement. This agrees with the results for Llama-2-7B in Section 4.1. (a) (b) Figure 10: Ablation on the number of paths for Llama-3.2-1B. (a) shows WikiText-2 validation perplexity as a function of the number of paths for selected target costs, while (b) shows the results for all target costs from 0.30.3 to 0.9750.975 in steps of 0.0250.025. We also evaluate the effect of the number of paths on compilation, prefill with 1,024 tokens, and decoding for models with a target cost of 0.70.7. For this ablation, we decode 64 tokens after prefilling with samples from WikiText-2, rather than using the complete task sequences timed in Section D.2. Figure 11 shows that compilation time is the metric most affected by the number of paths. Compiling the 33-path models was infeasible because graph capture exceeded TorchDynamo’s recursion depth. Compilation also became substantially slower as the number of paths increased. Increasing the number of paths has little effect on the decoding and prefill latencies of Llama-2-7B but increases the decoding latency of Llama-3.2-1B. Overall, the 5-path models provide a good trade-off between predictive and computational performance. Finally, we note that the prefill implementation of LRCC computes the first factor multiplication at full rank, masks the intermediate representation according to the selected path, and then applies the second factor, thus performing computation for ranks that are subsequently masked. The prefill implementation therefore does not yet exploit all of the computation savings available from routing; we leave its optimization to future work. Other optimizations could improve the prefill performance of both static low-rank and LRCC models relative to the dense base model, for example by fusing operations that share the same inputs (Zhu et al., 2026). (a) Llama-2-7B (b) Llama-3.2-1B Figure 11: Effect of the number of paths on compilation, prefill, and decoding times. Increasing the number of paths leads to substantially higher compilation times and slightly higher decoding times. D.5 Router Choices We repeated the same experiments described in Section 4.2 using Llama-3.2-1B. The results are shown in Figure 12. As in Llama-2-7B, a backbone of blocks is retained in dense form, while other Transformer blocks use multiple paths across tokens. Figure 12(b) also suggests that Non-Word tokens are assigned less compute on average. (a) (b) Figure 12: Router choices for Llama-3.2-1B on WikiText-2. (a) Average per-block cost (top) and effective fraction of paths (bottom) for each target. The dots indicate blocks that are deployed without factorization. (b) Distributions of the costs assigned to word and non-word tokens for each target. Appendix E Implementation Details All training was conducted on A100 GPUs with 64GB of VRAM. We trained linear block-level routers on 256 WikiText-2 sequences of length 2,048 with BF16 parameters using Adam with a constant learning rate of 10−310^-3 and effective batch size of 4. Training annealed over 28 target costs, from 0.9750.975 to 0.3000.300 in steps of −0.025-0.025 with 12 epochs per target by default and up to 30 epochs when no valid checkpoint was found. A training epoch took, on average, 2 minutes 24 seconds for Llama-2-7B and 28 seconds for Llama-3.2-1B. Both models required approximately 350 epochs to produce all 28 checkpoints, corresponding to approximately 14 hours for Llama-2-7B and 2.3 hours for Llama-3.2-1B, or about 30 and 5 minutes per checkpoint, respectively. For the compression loss, we set the soft-cap slope to κ=20κ=20. We initialized the compression-loss weight at λ0=1 _0=1. We selected the dual learning rate α for λ and the Gumbel-Softmax temperature schedule using a model-specific grid of four configurations. For Llama-3.2-1B, we searched dual learning rates α∈0.4,0.8α∈\0.4,0.8\ and temperature schedules of a fixed τ=0.5τ=0.5 and →0.52.0\!→\!0.5. For Llama-2-7B, we searched α∈1.2,1.6α∈\1.2,1.6\ and temperature schedules of a fixed τ=2.0τ=2.0 and →0.52.0\!→\!0.5. Among the four runs, we selected the configuration with the lowest WikiText-2 validation loss at the largest number of the 28 target costs. We repeated this search independently for each path count in 2,3,5,9,10,17,33\2,3,5,9,10,17,33\. Using the authors’ source code, we reimplemented COALA, Swift-SVD, and SVD-LLM in our framework to ensure consistent experimental settings and evaluation. We omit SVD-LLM’s optional fine-tuning stage so that no method fine-tunes the pretrained weights after compression; LRCC additionally trains its routers. An analogous weight-refinement stage could be applied to LRCC and the baselines. For COALA, we searched accumulation values 2,4\2,4\ and regularization values μ∈0,1μ∈\0,1\, at each target cost in 0.3,0.5,0.7,0.9\0.3,0.5,0.7,0.9\. We found that an accumulation value of 2 and μ=0μ=0 gave the best WikiText-2 validation performance, although differences were small. We used these values for all COALA decompositions, including the base factorization of the Llama models in LRCC. For Swift-SVD, we used their recommended layer importance weight of 0.60.6 in Llama-2-7B. For Llama-3.2-1B, we observed that the best results were obtained with a layer importance weight of 1.01.0. We used the recommended defaults for the remaining hyperparameters. SVD-LLM required no hyperparameter tuning. Finally, COALA+Swift and SVD-LLM+Swift combine the respective decompositions with the rank mappings produced by Swift-SVD. We refer to the original papers for definitions of the baseline-specific hyperparameters. E.1 Efficient Computation The optimal factors require only the leading right singular vectors of the layer outputs =Y=XW, and different methods compute this subspace in different ways. Swift-SVD (Qi et al., 2026), for example, avoids storing the full output matrix by incrementally accumulating its Gram matrix over calibration batches, Y=⊤=∑j⊤j,j=j.G_Y=Y Y= _jY_j Y_j, _j=X_jW. (32) Since Y=2⊤,G_Y=V ^2V , (33) a single eigendecomposition of the accumulated matrix directly provides V and the squared singular values. FlexRank (Zaccone et al., 2026) uses a closely related strategy, but instead incrementally accumulates the second moment of the input activations, X=⊤=∑j⊤j.G_X=X X= _jX_j X_j. (34) Writing =X1/2S=G_X^1/2, the right singular vectors of SW are the same as those of the complete output matrix because ()⊤()=⊤⊤=⊤.(SW) (SW)=W X XW=Y Y. (35) This makes the memory required for calibration independent of the number of calibration tokens. COALA (Parkina and Rakhuba, 2025) instead proposes avoiding the explicit formation of a Gram matrix. In our matrix convention, let =X=QR (36) be a thin QR decomposition of the calibration activations, computed with tall-skinny QR (TSQR) when X is too large to materialize at once. Since ⊤=Q Q=I, ‖⁡(−~)‖F=‖⁡(−~)‖F. \|X(W- W) \|_F= \|R(W- W) \|_F. (37) Furthermore, ()⊤()=⊤⊤=⊤⊤=⊤.(RW) (RW)=W R RW=W X XW=Y Y. (38) Therefore, an SVD of the substantially smaller matrix RW yields the same right singular subspace V in exact arithmetic. Unlike approaches based on the normal equations, this procedure does not explicitly form ⊤X X or require its inversion. This is numerically advantageous for ill-conditioned calibration activations, since forming a Gram matrix squares the condition number and can amplify floating-point errors. In our experiments, we used COALA’s QR strategy to compute the decompositions for the Llama models. For Qwen3-4B, the COALA decomposition did not converge, so we accumulated the output Gram matrix over calibration batches following Qi et al. (2026).

Instant research discovery