Paper deep dive
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
Xiang Li, Pengcheng Wang, Huazheng Wang, Saurabh Bagchi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Conversely, ultra-low-rank adapters ($r \le 2$) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this problem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine-tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand. Across LLMs of varying scales, SALT recovers high-rank accuracy using $r \le 2$ residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per-adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B.
Tags
Links
- Source: https://arxiv.org/abs/2608.03579v1
- Canonical: https://arxiv.org/abs/2608.03579v1
Trouble viewing inline? Open PDF directly →
Full Text
58,184 characters extracted from source content.
Expand or collapse full text
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving Xiang Li 1 , Pengcheng Wang 1 , Huazheng Wang 2 , Saurabh Bagchi 1 1 Purdue University 2 Oregon State University Abstract Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task perfor- mance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Con- versely, ultra-low-rank adapters (r≤ 2) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this prob- lem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine- tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user’s task residual on demand. Across LLMs of vary- ing scales, SALT recovers high-rank accuracy using r≤ 2 residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per- adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B. 1 Introduction Low-Rank Adaptation (LoRA) (Hu et al. 2022; Chen et al. 2024b; Dettmers et al. 2023) has transformed the fine-tuning landscape of Large Language Models (LLMs). By confin- ing weight updates to low-rank matrices A ∈R m×r and B ∈R r×n , where r ≪ min(m,n), LoRA minimizes train- able parameters, lowering the barrier to injecting domain- specific knowledge into foundation models. This tuning ef- ficiency has driven the rapid growth of personalized LLM assistants and specialized agents (Kong et al. 2024; Zhang et al. 2024; Li, Yan, and Mortazavi 2026). Instead of de- ploying independent models, these specialized models are instantiated as distinct LoRA adapters attached to a shared foundation model. This structural shift has driven modern deployment toward LLM-as-a-Service (LLMaaS) architec- tures (Sheng et al. 2023), where centralized platforms (e.g., GSM8K (Math)MBPP (Code) 30 40 50 60 70 80 90 100 Accuracy (%) 53.9% 66.2% 49.0% 62.1% 38.6% 65.4% 56.9% 69.2% Std LoRA (r=16) CTS (r=16) VeRA (r=2024) SALT (r_res=1) Std. LoRA (r=16) CTS (r=16) VeRA (r=4096) SALT (r=1) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Adapter Swap Cost (MB) 13.61 MB 5.24 MB 0.52 MB 0.85 MB 16.0x Reduction 6.2x Reduction CTS pins 8.38 MB shared basis in HBM | VeRA pins 110 MB of shared random weights in HBM | SALT pins 13.6 MB in HBM (Centroid) Figure 1: SALT reduces the PCIe bandwidth and VRAM usage by up to 16x while maintaining the performance. Amazon Bedrock (Mohammed et al. 2026)) host thousands of custom, tenant-specific adapters simultaneously. To im- prove serving efficiency, Punica (Chen et al. 2024a) pro- posed Segmented Gather Matrix-Vector (SGMV) kernels that enable batched execution of heterogeneous adapters alongside a single frozen base model. However, multi-tenant LoRA serving remains bottlenecked by two hardware con- straints: the Host-to-Device (H2D) PCIe swapping of adapter weights (Li et al. 2025), and the GPU VRAM required to host adapter weights alongside the KV-cache (Sheng et al. 2023). This dual bottleneck creates a fundamental trade-off: higher rank adapters typically offer better downstream task performance but saturate VRAM capacity and PCIe band- width, whereas compressing these models to ultra-low ranks (r ≤ 2) reduces memory constraints but introduces drastic performance degradation. Recent work has attempted to resolve those bottlenecks through compressing the post-training adapter or decompos- ing adapters into shared and user-specific components. How- ever, they face several critical limitations that render them unsuitable for real-world deployment. Post-hoc Compression: "Compress then Serve" (Gabriels- son et al. 2025) applies joint diagonalization across hundreds of pre-trained LoRA adapters to extract a shared basis, re- constructing individual tasks via task-specific vectors. This approach may raise data isolation concerns in multi-tenant deployments (Asif and Amiri 2026) and requires recomput- ing the shared basis when boarding new tenants. Continual Updates and Multi-Tenant Interference: Meth- ods like Share (Kaushik et al. 2026) incrementally update a shared subspace as each new user’s task data arrives. This arXiv:2608.03579v1 [cs.LG] 4 Aug 2026 is structurally incompatible with multi-tenant deployment as each new user’s data modifies the shared subspace, silently shifting the representations of all existing users. Frozen Shared Bases and Geometric Misalignment: Meth- ods like VeRA (Kopiczko, Blankevoort, and Asano 2024) and COLA (Xia, Qin, and Hazan 2024)) which freeze a random, task-agnostic foundation provide no structural shortcut for downstream learning, forcing highly constrained scaling vec- tors to compensate for a non-semantic basis, which severely bottlenecks expressivity and provides no mechanism to align independent adapters into a shared geometric basis. To bridge these gaps, we propose Subspace-Aligned LoRA Training (SALT), a serving-efficient fine-tuning framework designed explicitly for multi-tenant serving architectures. Unlike existing works, which require tenants to indepen- dently learn domain logic from scratch, SALT intentionally shifts this heavy lifting to cloud providers. Our framework decomposes the LoRA weight update into two distinct com- ponents: a high-capacity, domain-shared centroid ( ̄ W) that aligns diverse intra-domain task adapters into a unified geo- metric basis, and an ultra-low-rank residual (δ i ) that captures task-specific distribution. To make this completely seamless for the user, SALT employs a novel activation profiling mech- anism that automatically routes unlabelled user data to the optimal domain centroid for fine-tuning. Cloud providers are highly incentivized to absorb this one-time training cost of those centroids: by anchoring users to shared foundations, tenants achieve higher rank performance using only ultra- low-rank residuals. During inference, the domain centroid (i.e., r = 16) is permanently pinned in GPU memory as a shared anchor, and the serving engine only swaps the ultra- low rank residuals (e.g., r = 1). SALT not only alleviates the PCIe bandwidth bottleneck but also increases static adapter density, enabling serving engines to host vastly more con- current tenants within the same hardware constraints, which significantly boosts the serving throughput. Our main contributions are summarized as follows: • To overcome the geometric divergence of independently trained adapters despite their shared domain knowledge, we introduce a matrix cosine regularizer that proactively aligns task subspaces into a unified basis. • We propose SALT, a serving-aware fine-tuning frame- work that decouples representational capacity from physi- cal memory costs. By anchoring ultra-low-rank task resid- uals to a pinned, domain-shared centroid, SALT recovers higher capacity adapter performance while minimizing the PCIe swapping footprint and GPU VRAM usage. • Across varying LLM scales, SALT largely recovers higher rank performance using only r = 1 task residuals. Fur- thermore, its up to 16x memory reduction boosts serving throughput by up to 51% under PCIe bandwidth pres- sure and 28% under GPU VRAM constraints with the Llama-3.2-3B model. 2 Related Work 2.1 Multi-Tenant LoRA Serving Recent systems (i.e., Punica, SLoRA) optimize serving massive LoRA collections via custom kernels (Segmented Gather Matrix-Vector (SGMV) kernel), memory manage- ment or request co-scheduling (Sheng et al. 2023; Chen et al. 2024a; Wu et al. 2024). Furthermore, CLIMB (Zhang et al. 2026) has formalized the "LoRA residency cliff", demon- strating how rapid H2D swapping causes system-wide con- gestion collapse when KV-cache demands peak. Contrary to those optimizing at the infrastructure layer, SALT operates at the algorithmic layer to produce ultra-low-rank (r ≤ 2) residuals natively compatible with any SGMV-based engine, drastically reducing the payload these systems must manage. 2.2 Shared Subspace and Adapter Compression Several works construct shared bases or manage adapter ge- ometry to reduce multi-tenant memory overhead and mitigate interference. At initialization, methods like PiSSA (Meng, Wang, and Zhang 2024) extract principal components di- rectly from the base model via SVD to form an initial sub- space. Post-training, "Compress then Serve" (Gabrielsson et al. 2025) applies joint diagonalization to the pre-trained adapter to extract a shared orthonormal basis. Similarly, EigenLoRAx (Kaushik et al. 2025) extracts a principal sub- space via SVD across a pre-trained adapter repository. How- ever, this paradigm involves three practical trade-offs: (1) it requires pre-trained high-rank (e.g., r = 16) adapters prior to compression; (2) the shared basis is derived directly from private task adapters, which may raise data isolation con- cerns in multi-tenant deployments (Asif and Amiri 2026); and (3) adding or removing a user requires recomputing the shared basis, which grows costly as the adapter pool scales. Incremental approaches like Share (Kaushik et al. 2026) and continual learning frameworks like LoDA (He et al. 2026) similarly entangle representations and destroy isolation by continually updating shared principal factors with new user data. SALT guarantees strict data isolation by training the shared centroid exclusively on provider-owned public data, and private data only updates the isolated task residual. 2.3 Hierarchical and Residual Adaptation Decomposing adapters into shared and task-specific com- ponents is common in multi-task and continual learning. HydraLoRA (Tian et al. 2024) employs a single shared A matrix alongside multiple task-specific B matrices via MoE routing, while LoraHub (Huang et al. 2023) compose inde- pendent LoRAs dynamically via few-shot inference. Shifting to sequential adaptation, LiLoRA (Che et al. 2026) mitigates catastrophic forgetting in continual learning by maintaining a sequentially updated shared LoRA module, constrained by a cosine stability loss. However, these shared modules are con- tinuously updated by diverse task data, which can entangle user representations and complicate the data isolation typi- cally required for multi-tenant LLMaaS. Conversely, meth- ods that use a frozen, randomly initialized base to ensure isolation (e.g., VeRA (Kopiczko, Blankevoort, and Asano 2024)) lack a semantically optimized multi-tenant anchor. COLA (Xia, Qin, and Hazan 2024) introduces additive resid- uals for single-task expansion but provides no mechanism to geometrically align independent adapters. Finally, methods like Align-LoRA (Liu et al. 2025) align low-dimensional representations produced by the shared LoRA projection via Public Math Corpus Pretrained Weights x + h Public Coding Corpus JavaC++ Python Centroid Co-Training to reduce individual task loss and alignment loss respect to centroid. PEFT Phase 1: Sub-space Aligned Centroids Training Pretrained Weights Coding Centroid Company A Data residual R=16/32 Company B DataCompany C Data residualresidual r=2 r=2 r=1 + Residual Pool Phase 2: Ultra-Low Rank Agent / User Specific Task Fine-Tuning GPU 1 VRAM Pretrained Weights + + Phase 3: Pin Once, Swap Light Coding Centroid GPU 2 VRAM Pretrained Weights + + Coding Centroid R=16/32 + Math Centroid R=16/32 Residual Swapping! Figure 2: The three phases of sub-space aligned fine-tuning for efficient LoRA serving. Phase 1: Training sub-space align centroids for different domains (i.e., Coding) with public corpus. Phase 2: Training ultra-low rank adapters using the domain centroids. Phase 3: Efficient multi-tenant serving with domain centroids and task-specific residuals. KL-divergence, which does not guarantee alignment of the underlying weight updates that our serving mechanism di- rectly composes. SALT bridges these gaps through an explicit training-time alignment that geometrically constrains task adapters to a public-data-trained centroid, enabling highly expressive rank-1 residuals natively suited for LLMaaS. 3 Methods SALT introduces a serving-aware fine-tuning framework de- signed to break the memory bandwidth and footprint bot- tlenecks in the multi-tenant LoRA serving systems by de- coupling the representational capacity of an adapter from its physical H2D transfer and GPU VRAM footprint. 3.1 The Centroid-Residual Decomposition In a standard multi-tenant setting, the serving engine hosts a collection of M distinct adapters, denoted as ∆W i M i=1 , where each adapter is fine-tuned on its corresponding task datasetD i . A standard LoRA parameterizes the task-specific weight update as ∆W i = B i A i . During inference, dynami- cally swapping hundreds of these standard higher rank ma- trices (i.e., r ≥ 16) saturates PCIe bandwidth. Conversely, permanently pinning them in HBM severely exacerbates KV- cache starvation, bottlenecking concurrent tenancy. To re- solve this, SALT structurally decomposes the weight update into two distinct components: ∆W i = ̄ W + δ i • The Shared Centroid ( ̄ W): A high-capacity, domain- specific, task-agnostic centroid (e.g., r = 16) that is pre- loaded and pinned in the GPU’s VRAM. • The Task Residual (δ i ): An ultra-low-rank dynamic adapter (δ i = B ′ i A ′ i , where r ≤ 2) that encapsulates task-specific knowledge. Only this minimal residual is dynamically swapped during multi-tenant inference. Phase 1: Joint Subspace Alignment A naive post-hoc averaging of independent adapters to establish ̄ W fails be- cause isolated adapters converge to distinct, unaligned local minima. While joint diagonalization bypasses this, it vio- lates multi-tenant data isolation and renders dynamic scaling intractable, as onboarding a new tenant requires recomput- ing the entire global basis. To force in-domain task adapters to map their learned features into a unified, geometrically aligned subspace, in Phase 1 training, SALT optimizes the individual task adapters (∆W i ) and the domain centroid ( ̄ W) jointly. Crucially, the centroid is not merely a passive geo- metric anchor; it is actively optimized on the multi-task data distribution. The joint objective is formulated as: min ∆W i , ̄ W 1 M M X i=1 w i h L task (W base + ∆W i ;D i )+ L task (W base + ̄ W ;D i ) + λL align (W base + ∆W i , ̄ W ) i (1) where λ controls the alignment penalty and w i ∝ |D i | reweights each task by its relative dataset size, preventing smaller, more frequently cycled datasets from being over- represented. To guarantee strict privacy, Phase 1 is executed by the cloud provider using only public, domain-specific datasets; user-private data is strictly reserved for Phase 2 fine- tuning. We implementL align as an epsilon-stabilized matrix cosine regularizer: L align (∆W i , ̄ W ) = 1− Tr(∆W T i ̄ W ) ∥∆W i ∥ F ∥ ̄ W∥ F + ε (2) The constant (ε = 10 −8 ) prevents numerical instability and division-by-zero artifacts at initialization ( ̄ W ≈ 0). Phase 2: Ultra-Low Rank Residual Tuning Once the geometrically aligned centroid ( ̄ W) is established, Phase 2 adapts the model to private, user-specific data. To guarantee strict multi-tenant data isolation, the optimization is exclu- sively confined to the ultra-low-rank task residual δ i : min δ i L task (W base + ̄ W + δ i ;D (i) private )(3) Constraining this frozen-anchor optimization to ultra-low ranks (r ≤ 2) is not merely a systems-level memory opti- mization; it also acts as a structural regularizer. Because the constrained residual lacks the parameter capacity to model the target distribution independently, it acts as a geometric correction vector. It is forced to utilize the pre-aligned cen- troid as a structural shortcut—achieving constructive align- ment with the generalized features. This dynamic allows SALT to successfully recover higher-rank standard LoRA performance while drastically reducing the dynamic H2D PCIe swapping footprint. 3.2 Automated Centroid Routing When a user submits an unlabelled datasetD for Phase 2 fine- tuning, the system must automatically select the optimal cen- troid from the Phase 1 centroid libraryC = ̄ W 1 ,..., ̄ W K , or fallback if the user data is entirely out-of-distribution (OOD). We achieve this robustly using a two-stage activation profiling algorithm. Using a minimal subset of D, we compute the centroid k-induced displacement norm N k = ∥h k − h base ∥ 2 for each centroid, where h is the final hidden state. To guarantee length invariance, this metric is token-normalized. The sys- tem evaluates these novel activations against baseline distri- butions (μ,σ). These baselines are established offline at the end of Phase 1 by passing each centroid’s own in-domain validation set through the model and recording the empirical mean and standard deviation of its activations. Using these pre-calibrated statistics, the system applies a two-stage filter: Stage 1: OOD Rejection. Building on representation- based OOD detection (Ren et al. 2021; Sun, Guo, and Li 2021), we compute the deviation Z N,k = |N k −μ N,k | σ N,k . If the minimum deviation across all centroids exceeds a strict threshold (min k Z N,k > τ N , e.g., τ N = 2.0 for 95% con- fidence interval (CI)), the data lacks geometric overlap with the available library. The system flags it as OOD and falls back to standard LoRA training. Stage 2: Expert Routing. For in-domain tasks, absolute norms naturally fluctuate based on prompt structure and vo- cabulary complexity. To isolate true domain alignment, we compute a dominance ratio R k , comparing a centroid’s acti- vation against the average of all others. The optimal centroid k ∗ is selected by minimizing the ratio deviation Z R,k : R k = N k 1 K−1 P j̸=k N j , k ∗ = arg min k |R k − μ R,k | σ R,k (4) If the minimum deviation still exceeds the routing thresh- old (Z R,k ∗ > τ R , e.g., τ R = 2.0), the task is considered a blended or ambiguous domain and sent to the fallback. Oth- erwise, the system automatically routes the user’s dataset to ̄ W k ∗ as the anchor for Phase 2 task residual tuning. 3.3 Multi-Tenant Serving During multi-tenant serving, the final weight matrix for a user request i is instantiated as: W final = W base + γW + δ i While the task residual δ i is optimized against the unscaled centroid during Phase 2, directly composing dense, multi- task representations at inference time can cause interference. Following established principles in task arithmetic (Ilharco et al. 2023), we introduce an inference-time scaling coef- ficient γ to dampen multi-task interference (γ < 1.0) or amplify target capabilities (γ > 1.0). To avoid the overhead of dynamically composing continuous γ values, SALT dis- cretizesγ into predefined bins (e.g.,γ ∈0.6, 0.8, 1.0). The serving frontend groups incoming requests by their assigned ( W,γ) pair. For high-frequency domain centroids, providers can fuse the scaled centroid directly into the base weights at node startup. During inference, these nodes only dynami- cally swap the ultra-low-rank residuals (δ i ). For the long tail of diverse requests, the scaled centroid and the residual can simply be concatenated offline into a single standard adapter (e.g., r = 17). This allows the serving engine to seamlessly handle rare tasks using standard heterogeneous batching. 4 Experiments 4.1 Evaluation Setup Datasets: We evaluate SALT across two domains — math- ematical reasoning and coding — using three base model families of varying scale (Llama-3.2-3B, Mistral-7B-v0.3, Pythia-12B). Math datasets include GSM8K (Cobbe et al. 2021), SVAMP (Patel, Bhattamishra, and Goyal 2021), Mul- tiArith (Roy and Roth 2015), and AQuA (Ling et al. 2017); coding tasks include MBPP (Austin et al. 2021), SPI- DER (Yu et al. 2018), APPS (Hendrycks et al. 2021), and HumanEval (Chen et al. 2021). Dataset details are listed in Appendix A.1. Evaluation Metrics: We report exact-match accuracy for math tasks, functional correctness (unit tests) for MBPP/Hu- manEval/Apps, and structural exact set match for Spider. All SALT results use the scaling factor γ selected via a small hold-out validation set, disjoint from the test sets in Table 1. Domain Centroids & Residuals: In Phase 1, the Math cen- troid is trained on GSM8K and SVAMP, and the Code cen- troid on MBPP and Spider. For Phase 2, we evaluate three settings: (1) Seen In-Domain: residuals tuned on the Phase 1 datasets (‡ in Table 1); (2) Unseen In-Domain: residuals tuned on held-out datasets (MultiArith, AQuA, APPs; † in Table 1) to test generalization; (3) Zero-Shot Transfer: the MBPP-tuned residual is evaluated directly on HumanEval without further training, testing whether the shared centroid enables generalization to a related, unseen task. Hardware: All throughput and latency experiments are con- ducted on NVIDIA H100 GPUs. 4.2 Baselines We compare against LoRA (isolated, uncompressed task- specific fine-tuning), VeRA (a frozen, randomly initialized shared projection with trainable scaling vectors), and Com- press then Serve (CTS) (post-training joint diagonalization of independent adapters into a shared basis). We restrict comparison to VeRA and CTS as they are the primary prior methods that, like SALT, produce a compressed per-tenant serving artifact; other related methods target multi-task or Memory (MB)Math Reasoning TasksCoding Tasks Method Rank Pin Swap GSM8K ‡ SVAMP ‡ Multiarith † AQuA † MBPP ‡ SPIDER ‡ APPs † HumanEval BaseNA–3.936.676.9024.514.500.008.7518.29 LoRA r = 1 0.00 0.85 49.6763.3311.0334.8015.7530.3416.025.58 r = 2 0.00 1.70 50.8063.7518.6236.7624.5029.5012.2512.40 r = 4 0.00 3.40 53.1365.0027.5838.2425.509.4721.2521.71 r = 8 0.00 6.81 53.7064.1726.9035.293.0036.2116.7524.81 r = 16 0.00 13.61 53.8866.2530.3534.8026.7521.3418.7521.71 LoRA (Merged Dataset) r = 1 0.00 0.85 48.0865.0–29.7550.95– r = 2 0.00 1.70 51.6367.08–28.7552.63– r = 4 0.00 3.40 51.6367.91–26.7552.52– VeRA r = 1k 27.26 0.26 40.5160.8336.5532.3524.2521.2217.018.60 r = 2k 54.52 0.52 38.6365.4231.0334.8024.7524.3417.519.37 r = 4k 109.05 1.05 40.7960.8332.4127.4526.5031.5317.7520.16 CTS (JD-Full) r = 1 0.52 0.339.1740.8318.6224.022.750.011.517.05 r = 2 1.05 0.66 10.2066.2527.5923.0414.250.1214.7513.17 r = 4 2.01 1.31 37.6964.5826.9025.495.000.08.2517.82 r = 8 4.19 2.62 47.0563.3328.2834.312.500.127.7517.05 r = 16 8.38 5.24 49.0262.0831.7229.909.253.4710.513.95 SALT (Ours) r = 1 13.63 0.85 56.88 (+7.2) 69.17 (+5.8) 86.21 (+73.9) 34.31 (-0.4) 27.25 (+11.5) 56.59 (+26.3) 19.25 (+3.2) 34.88 (+9.3) r = 2 13.63 1.70 57.81 (+7.0) 69.17 (+5.4) 85.52 (+66.4) 31.37 (-5.3) 27.80 (+3.3) 56.35 (+26.9) 17.75 (+5.5) 34.11 (+21.7) r = 4 13.63 3.40 53.23 (+0.1) 68.75 (+3.7) 86.21 (+59.6) 31.86 (-6.3) 27.25 (+1.8) 56.47 (+47.0) 18.25 (-3.0) 35.66 (+13.9) ‡ Datasets utilized for Phase 1 centroid training and Phase 2 residual tuning. † Datasets for Phase 2 residual tuning only. LoRA merged dataset refers to a single standard LoRA adapter trained on the concatenation of ‡ datasets. The numbers in parenthesis are improvement compared to the standard LoRA at the same rank. Table 1: Peak task performance and memory footprint for Mistral-7B-v0.3. SALT at r=1 largely recovers the performance of higher rank standard LoRA with significantly reduced memory footprint. continual-learning objectives without an explicit serving- efficiency mechanism are discussed in Section 2. 4.3 Task Residual Adapter Performance As shown in Table 1, SALT largely outperforms standard LoRA and state-of-the-art compression baselines in down- stream task performance while utilizing an ultra-low-rank residual. VeRA’s reported rank (1024-4096) is the dimen- sionality of the frozen random matrix, not a trainable adapter rank; since only diagonal scaling vectors are learned, a large basis is needed for competitive expressivity. Although high- capacity standard LoRA maintains a slight edge on a few spe- cific datasets (e.g., AQuA), with a dynamic footprint of just 0.85 MB (r = 1), SALT consistently exceeds both standard low-rank configurations and the state-of-the-art compression baselines. This suggests that anchoring residuals to a pre- aligned centroid decouples downstream task performance from multi-tenant PCIe bottlenecks. Moreover, SALT avoids the severe accuracy degradation and rank-sensitivity seen in baselines: standard LoRA and CTS exhibit volatile rank-to- rank fluctuations (e.g., LoRA drops to 3.00% at r = 8 on MBPP despite scoring 25.50% at r = 4), whereas SALT re- mains stable across r ∈1, 2, 4. This suggests that anchor- ing residuals to a shared centroid helps regularize against the instability that unconstrained low-rank adapters can exhibit. Figure 3 shows these gains are architecture-agnostic. Across Mistral-7B-v0.3, Pythia-12B, and Llama-3.2-3B, SALT con- sistently achieves an average of 80–95% downstream task performance recovery while delivering 9x to 11.5x reduc- tions in dynamic adapter rank. Detailed comparison results for SALT and baselines on each model can be found in the Appendix A.2. Mistral-7B-v0.3Pythia-12BLlama-3.2-3B 70 80 90 100 Recovery Rate (%) 100% Recovery BaselineRecovery Rate (%)Rank Reduction (x) 0 5 10 15 Rank Reduction (x) Figure 3: Average task performance recovery and adapter rank reduction for three evaluated models. 4.4 Serving Efficiency and Throughput To demonstrate serving efficiency, we evaluate SALT with vLLM and scaling the number of concurrent adapters (N) up to 512. We assess throughput across two extreme memory constraints - PCIe bandwidth and HBM capacity pressure: PCIeVRAM Adapters vLLM vLLM+SALT vLLM vLLM+SALT 12898.38 107.77 (+9.5%) 77.8282.50 (+6.0%) 25680.79 97.47 (+20.6%) 45.8249.91 (+8.9%) 51243.42 54.46 (+25.4%) 32.59 38.41 (+17.9%) Table 2: Throughput for Mistral-7B-v0.3 (req/s). 1. Constraining the adapter cache to force continuous H2D swapping, SALT yields +25.0% throughput over standard LoRA at N = 512. 2. When all N residual adapters instead reside in GPU VRAM, Standard LoRA’s large footprint rapidly con- sumes the KV-cache, forcing the scheduler to fragment batch sizes and preempt requests to avoid out-of-memory errors. SALT’s 16× footprint reduction avoids this, sus- taining larger continuous batches and improving peak throughput by up to 17.9% at N = 512. Since SALT relies on standard additive matrix operations, these throughput gains are achieved without custom ker- nels, ensuring compatibility with production engines. We omit VeRA and CTS from this end-to-end evaluation as their specialized matrices require non-standard serving pipelines and cannot be directly used with vLLM. Instead, Fig- ure 4 provides micro-benchmarks against all baselines. SALT achieves the lowest forward-pass latency while requiring an order-of-magnitude smaller swap footprint. Moreover, SALT matches VeRA’s minimal copy latency and substantially out- performs Standard LoRA and CTS. 05101520 Swap Footprint per Adapter (MB) 0 1 2 3 4 5 6 7 8 9 Adapter Forward Pass Latency (ms) Standard LoRA SALT VeRA CTS StandardCTSVeRASALT 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Adapter Copy Latency (ms) 0.25ms 0.10ms 0.03ms 0.02ms 0.01ms 0.01ms0.01ms0.01ms CPU → GPU (PCIe Swap) GPU → GPU (HBM Copy) Figure 4: Mistral: (Left) Swap footprint versus adapter for- ward pass latency. (Right) Latency breakdown for PCIe swap and HBM copy operations across different baselines. 4.5 Subspace Alignment Experiment We validate Phase 1 subspace alignment by analyzing layer- wise cosine similarity of full adapter updates (∆W = BA) for GSM8K and SVAMP. As Figure 5 (Left) shows, indepen- dently trained Standard LoRA updates for the two adapters exhibit near-zero similarity despite sharing domain seman- tics. However, under SALT’s alignment penalty, as λ in- creases, independently learned updates are forced to converge into a shared subspace. Moreover, this forces both adapters and the centroid into the same aligned subspace as shown in Figure 5(right). Ratio Dev. (Z R ) TaskZ N Math Code Routed To AQuA1.32 0.942.75Math MultiArith0.39 0.113.40Math CodeSearchNet 1.75 3.550.49Code APPs1.48 3.120.39Code xnli/german2.28 1.652.12OOD xnli/spanish2.32 1.692.19OOD Table 3: Zero-shot hybrid centroid routing validation on Mistral-7B-v0.3 (τ N = 2.0,τ R = 2.0, corresponding to standard 95% CI threshold). 051015202530 Layer Number 0.0 0.2 0.4 0.6 0.8 1.0 Weight Update Cosine Similarity A: Alignment Sweep Standard (Independent) λ= 0.5 λ= 1.0 λ= 1.5 λ= 2.0 051015202530 Layer Number B: Adapter-to-Centroid Alignment (λ= 1.0) GSM8K SVAMP Figure 5: Left: Cosine similarity between GSM8K and SVAMP weight updates across layers. Right: Cosine similar- ity of individual adapters to the shared centroid (λ = 1.0). 4.6 Phase 2 Centroid Routing We evaluate automated centroid routing on six datasets held out from Phase 1 training: AQuA and MultiArith (math), CodeSearchNet (Husain et al. 2019) and APPs (code), and two out-of-distribution (OOD) language splits from XNLI (Conneau et al. 2018) (German, Spanish). Follow- ing our protocol, we sample|D profile | = 100 examples from each dataset and extract adapter-induced displacement norms against both centroids. As shown in Table 3, the system correctly isolates OOD data while routing in-domain tasks accurately. For both language tasks, the adapter activation norm deviates sharply from both centroids’ expected ranges (Z N = 2.28 and 2.32, both > τ N ), this flags the data as OOD and triggers a fallback to Standard LoRA training. The remaining four datasets pass the OOD check (Z N ≤ 2.0) and are routed by minimum ratio deviation: AQuA and Mul- tiArith align with the Math centroid (Z R = 0.94 and 0.11), while CodeSearchNet and APPs route to the Code centroid (Z R = 0.49 and 0.39). These results show SALT can auto- mate centroid selection for in-domain tasks while gracefully handling out-of-distribution data. 5 Ablation Studies 5.1 Alignment Penalty λ Sensitivity We performed a sensitivity analysis on the alignment loss weight λ. We swept λ from 0.5 to 2.5, and show the corre- sponding downstream task accuracy. As shown in Figure 6, increasing λ monotonically improves the average adapter- centroid cosine similarity across all layers. Despite stronger 0.51.01.52.02.5 Alignment penalty weight λ 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Avg. Cosine Similarity 0.806 0.881 0.914 0.932 0.942 (a) Adapter–Centroid Alignment 0.51.01.52.02.5 Alignment penalty weight λ 30 40 50 60 70 80 Task Accuracy (%) (b) Downstream Accuracy GSM8K (Centroid+Residual) SVAMP (Centroid+Residual) Figure 6: Effect of alignment penalty λ. regularization, average downstream accuracy within the do- main remains stable, demonstrating that alignment incurs no significant performance penalty for in-domain tasks. 5.2 Centroid Scale γ Sensitivity Scale Factor (γ) Dataset0.40.60.81.01.2 GSM8K 53.04 58.46 58.65 55.29 48.83 SVAMP65.8370.068.3370.066.25 AQuA26.96 28.43 27.94 28.43 25.98 Table 4: Effect of centroid output scale factor (γ) on down- stream performance (%) for Mistral-7B-v0.3. As shown in Table 4, downstream performance follows an inverted-U pattern across the scale factor γ, peaking in the γ = 0.6–0.8 range and degrading at both extremes. No- tably, performance across γ = 0.6, 0.8, and 1.0 remains tightly clustered, suggesting the system is fairly insensitive to the exact scale factor within this middle range. This is practically useful for multi-tenant deployment: rather than fine-tuning γ, a provider can adopt few shared γ values (e.g., 0.8) across domains with only a marginal accuracy trade-off. Performance only degrades meaningfully at extreme bound- aries (γ = 0.4 or 1.2), indicating that precise fine-tuning is unnecessary as long as these boundary regions are avoided. Rank (r centroid ) GSM8K SVAMP MultiArith AQuA 855.4768.3385.5132.35 1656.8869.1786.2134.31 3255.2967.5089.6530.39 Table 5: Phase 1 Centroid Rank (r centroid ) analysis. 5.3 Centroid Rank Sensitivity We evaluate the impact of Phase 1 centroid capacity on down- stream performance. As shown in Table 5, increasing rank from r = 8 to r = 16 improves all four datasets, but scaling further to r = 32 degrades GSM8K, SVAMP, and AQuA, with only MultiArith continuing to improve (89.65%). This indicates r = 16 offers the best capacity–generalization bal- ance for this domain. Since higher-rank centroids also in- cur additional pinned VRAM, this result demonstrates that a moderate centroid rank is both the most accurate and memory-efficient choice in our settings. Centroid training remains efficient across this range (Appendix A.4), so ex- ploring rank during offline domain setup is inexpensive. 5.4 Effect of Subspace-Aligned Centroid We isolate the effect of subspace-aligned training against a centroid trained on naive dataset concatenation. Table 6 shows results at r = 2; subspace alignment consistently outperforms the mixed-data. Across 9 paired configurations (r ∈1, 2, 4 over GSM8K, SVAMP, and MBPP), the mean gain is +4.38% (paired t-test, p = 2.5× 10 −5 ), confirming a statistically significant and robust improvement. The result and a detailed analysis on SPIDER can be found separately in Appendix A.5, as it exhibits high variance across ranks. Accuracy % Centroid Method GSM8K SVAMP MBPP SPIDER Concatenated52.9463.7522.512.47 Subspace-aligned57.8169.1727.856.35 Table 6: Effect of subspace-aligned centroid training strategy (r = 2) on downstream tasks For Mistral-7B-v0.3. 6 Limitations While SALT demonstrates significant serving efficiency, it introduces a structural bias that may penalize specific in- domain tasks. Because Phase 1 subspace alignment enforces a strict geometric intersection, tasks with orthogonal for- mats like multiple-choice evaluation within a predominantly direct-generation domain can experience quality degrada- tion. In these cases, the highly constrained r = 1 residual lacks the capacity to overcome the centroid’s structural regu- larization, occasionally resulting in worse performance than standard LoRA like AQuA in Table 1. Moreover, our evalua- tion is restricted to two domains (math and coding); as shown in Appendix A.3, combining unrelated domains into a single centroid may degrade in-domain accuracy, indicating that SALT’s benefits depend on the provider partitioning tasks into sufficiently coherent domains. Finally, extending SALT to Mixture-of-Experts architectures remains open: MoE’s sparse, disjoint routing paths make establishing a unified geometric subspace across experts mathematically complex without additional memory cost. 7 Conclusion We introduced SALT, a serving-aware fine-tuning framework that anchors ultra-low-rank residuals to subspace-aligned do- main centroids to recover higher-rank task performance while reducing per-adapter memory by up to 16x, ultimately boost- ing serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints. References Asif, S.; and Amiri, M. M. 2026. Information-theoretic pri- vacy control for sequential multi-agent llm systems. arXiv preprint arXiv:2603.05520. Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Che, C.; Wang, Z.; Yang, P.; Wang, C.; Ma, H.; and Shi, Z. 2026. LoRA in LoRA: Towards parameter-efficient ar- chitecture expansion for continual visual instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 40, 19978–19986. Chen, L.; Ye, Z.; Wu, Y.; Zhuo, D.; Ceze, L.; and Krishna- murthy, A. 2024a. Punica: Multi-tenant lora serving. Pro- ceedings of Machine Learning and Systems, 6: 1–13. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Chen, Y.; Qian, S.; Tang, H.; Lai, X.; Liu, Z.; Han, S.; and Jia, J. 2024b. Longlora: Efficient fine-tuning of long- context large language models. In International Conference on Learning Representations, volume 2024, 8220–8238. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. Conneau, A.; Rinott, R.; Lample, G.; Williams, A.; Bowman, S. R.; Schwenk, H.; and Stoyanov, V. 2018. XNLI: Evaluat- ing Cross-lingual Sentence Representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Lin- guistics. Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36: 10088–10115. Gabrielsson, R. B.; Zhu, J.; Bhardwaj, O.; Choshen, L.; Gree- newald, K.; Yurochkin, M.; and Solomon, J. 2025. Compress then Serve: Serving Thousands of LoRA Adapters with Lit- tle Overhead. In Forty-second International Conference on Machine Learning. He, L.; Cheng, D.; Wang, H.; Yang, X.; Wang, N.; and Gao, X. 2026. Task-Driven Subspace Decomposition for Knowl- edge Sharing and Isolation in LoRA-based Continual Learn- ing. In Forty-third International Conference on Machine Learning. Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; and Steinhardt, J. 2021. Measuring Coding Challenge Compe- tence With APPS. NeurIPS. Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. Huang, C.; Liu, Q.; Lin, B. Y.; Pang, T.; Du, C.; and Lin, M. 2023. Lorahub: Efficient cross-task generalization via dy- namic lora composition. arXiv preprint arXiv:2307.13269. Husain, H.; Wu, H.-H.; Gazit, T.; Allamanis, M.; and Brockschmidt, M. 2019. Codesearchnet challenge: Eval- uating the state of semantic code search. arXiv preprint arXiv:1909.09436. Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Schmidt, L.; Ha- jishirzi, H.; and Farhadi, A. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations. Kaushik, P.; Vaidya, A.; Chaudhari, S.; Chellappa, R.; and Yuille, A. 2026. Shared LoRA Subspaces for almost Strict Continual Learning. arXiv preprint arXiv:2602.06043. Kaushik, P.; Vaidya, A.; Chaudhari, S.; and Yuille, A. 2025. EigenLoRAx: Recycling Adapters to Find Principal Sub- spaces for Resource-Efficient Adaptation and Inference. In Proceedings of the Computer Vision and Pattern Recognition Conference, 649–659. Kong, X.; Wu, J.; Zhang, A.; Sheng, L.; Lin, H.; Wang, X.; and He, X. 2024. Customizing Language Models with Instance-wise LoRA for Sequential Recommendation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Kopiczko, D. J.; Blankevoort, T.; and Asano, Y. M. 2024. VeRA: Vector-based Random Matrix Adaptation. In The Twelfth International Conference on Learning Representa- tions. Li, S.; Lu, H.; Wu, T.; Yu, M.; Weng, Q.; Chen, X.; Shan, Y.; Yuan, B.; and Wang, W. 2025. Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), 613–629. Boston, MA: USENIX Association. ISBN 978-1- 939133-48-9. Li, X.; Yan, N.; and Mortazavi, M. S. 2026. Embodied Task Planning via Graph-Informed Action Generation with Large Language Models. In Forty-third International Conference on Machine Learning. Ling, W.; Yogatama, D.; Dyer, C.; and Blunsom, P. 2017. Pro- gram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In Barzilay, R.; and Kan, M.-Y., eds., Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 158–167. Vancouver, Canada: Association for Computational Linguistics. Liu, J.; Cheng, B.; Chang, Y.; and Wu, Y. 2025. Align, Don’t Divide: Revisiting the LoRA Architecture in Multi- Task Learning. arXiv preprint arXiv:2508.05078. Meng, F.; Wang, Z.; and Zhang, M. 2024. PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Mohammed, B.; et al. 2026. Accelerate Custom LLM De- ployment: Fine-Tune with Oumi and Deploy to Amazon Bedrock. Patel, A.; Bhattamishra, S.; and Goyal, N. 2021. Are NLP Models really able to Solve Simple Math Word Problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2080–2094. Online: Asso- ciation for Computational Linguistics. Ren, J.; Fort, S.; Liu, J.; Roy, A. G.; Padhy, S.; and Lak- shminarayanan, B. 2021. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022. Roy, S.; and Roth, D. 2015. Solving General Arithmetic Word Problems. In Màrquez, L.; Callison-Burch, C.; and Su, J., eds., Proceedings of the 2015 Conference on Empir- ical Methods in Natural Language Processing, 1743–1752. Lisbon, Portugal: Association for Computational Linguistics. Sheng, Y.; Cao, S.; Li, D.; Hooper, C.; Lee, N.; Yang, S.; Chou, C.; Zhu, B.; Zheng, L.; Keutzer, K.; et al. 2023. S- lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285. Sun, Y.; Guo, C.; and Li, Y. 2021. React: Out-of-distribution detection with rectified activations. Advances in neural in- formation processing systems, 34: 144–157. Tian, C.; Shi, Z.; Guo, Z.; Li, L.; and Xu, C. 2024. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Ad- vances in Neural Information Processing Systems, 37: 9565– 9584. Wu, B.; Zhu, R.; Zhang, Z.; Sun, P.; Liu, X.; and Jin, X. 2024. dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 911–927. Santa Clara, CA: USENIX Association. ISBN 978-1-939133-40-3. Xia, W.; Qin, C.; and Hazan, E. 2024. Chain of LoRA: Ef- ficient Fine-tuning of Language Models via Residual Learn- ing. In ICML 2024 Workshop on LLMs and Cognition. Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; Zhang, Z.; and Radev, D. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to- SQL Task. In Riloff, E.; Chiang, D.; Hockenmaier, J.; and Tsujii, J., eds., Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, 3911–3921. Brussels, Belgium: Association for Computational Linguis- tics. Zhang, H.; Liang, Z.; Decheng, Z.; and Wang, H. 2026. CLIMB: Taming the LoRA Residency Cliff in Multi-LoRA Serving. In Forty-third International Conference on Machine Learning. Zhang, Z.; Rossi, R. A.; Kveton, B.; Shao, Y.; Yang, D.; Zamani, H.; Dernoncourt, F.; Barrow, J.; Yu, T.; Kim, S.; et al. 2024. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027. A Appendix A.1 Experiment Setup Details DatasetTrainValidTest GSM8K ‡ 74732501069 SVAMP ‡ 70060240 MultiArith † 42035145 AQuA † 500050204 MBPP ‡ 374100400 SPIDER ‡ 70002001034 APPs † 2000100400 HumanEval † –164 Table 7: Dataset details.‡: used for Phase 1 centroid training and Phase 2 residual tuning. †: used for Phase 2 residual tuning only (Unseen In-Domain). †: used exclusively for zero-shot evaluation (Section 4.1); no training split is used. Table 7 summarizes the statistics of the datasets used in our experiments. All datasets are used without further pro- cessing except AQuA and APPS. For AQuA, we subsampled a smaller subset from the original training set while keeping the full test set unchanged. For APPS, we restricted evalua- tion to introductory-level problems, as harder difficulty tiers were largely intractable for models in our 3B–12B param- eter range, and sampled 2,000 training and 500 evaluation examples from this subset. For every dataset, we further split the original test set into validation (20%) and test partitions (80%); the validation split is used exclusively for selecting the optimal centroid scale factor γ. Implementation Details: All models are trained using AdamW (weight decay 0.01) with a cosine decay schedule and a 3% linear warmup; gradients are clipped to a maximum norm of 1.0. Prompt Template: Mbpp | APPs | Humaneval Write a python function [Function Name] to solve the task described below. Task: [Instruction Text] Provide only the Python code inside a ”’python”’ block. Figure 7: The prompt template used for code generation. Prompt Template: SPIDER Write a SQL query to solve the task described below. [Instruction Text] Provide only the SQL query. Figure 8: The prompt template used for the SQL generation. Prompt Template: For the math tasks, we do not use any prompt template for the training or evaluation. For the cod- ing tasks, two different prompt templates are being used for the code generation (Figure 7) and SQL query generation (Figure 8) separately. A.2 Results across Diverse Architectures 0102030405060 Swap Footprint per Adapter (MB) 0 5 10 15 20 25 Adapter Forward Pass Latency (ms) Standard LoRA SALT VeRA CTS StandardCTSVeRASALT 0.0 0.2 0.4 0.6 0.8 1.0 Adapter Copy Latency (ms) 0.89ms 0.45ms 0.06ms 0.06ms 0.04ms 0.02ms 0.01ms 0.01ms CPU → GPU (PCIe Swap) GPU → GPU (HBM Copy) Figure 9: Llama-3.2-3B: (Left) Swap footprint versus adapter forward pass latency. (Right) Latency breakdown for PCIe swap and HBM copy operations across different baselines. 010203040 Swap Footprint per Adapter (MB) 0 2 4 6 8 10 12 Adapter Forward Pass Latency (ms) Standard LoRA SALT VeRA CTS StandardCTSVeRASALT 0.0 0.2 0.4 0.6 0.8 1.0 Adapter Copy Latency (ms) 0.88ms 0.45ms 0.06ms 0.06ms 0.04ms 0.02ms 0.01ms 0.01ms CPU → GPU (PCIe Swap) GPU → GPU (HBM Copy) Figure 10: Pythia-12B: (Left) Swap footprint versus adapter forward pass latency. (Right) Latency breakdown for PCIe swap and HBM copy operations across different baselines. To verify that the performance gains and efficiency of SALT are not uniquely tied to a specific model family, we extend our evaluation to two additional models with dis- tinct architectural designs and parameter scales: Llama-3.2- 3B and Pythia-12B. To accommodate their diverse archi- tectures, we adjusted the LoRA target modules accordingly. Unlike Mistral-7B-v0.3, where we only target the q_proj and v_proj modules, we target all linear layers for Llama- 3.2-3B and Pythia-12B. Following the two-phase training recipe described in Sec- tion 3, we independently trained mathematical and coding centroids for both models, followed by task-specific resid- ual fine-tuning in Phase 2. Detailed downstream accuracies for SALT on these architectures are presented in Tables 8 and 9. SALT demonstrates competitive, and in several cases superior, performance compared to standard LoRA at higher ranks, despite relying on a shared subspace and an ultra- low-rank task residual. While standard LoRA occasionally achieves higher accuracy on specific tasks at higher ranks (e.g., AQuA on Llama, GSM8K on Pythia), this is expected as standard LoRA dedicates its entire parameter budget to a sin- Memory (MB)Math Reasoning TasksCoding Tasks Method Rank Pin Swap GSM8K ‡ SVAMP ‡ Multiarith † AQuA † MBPP ‡ SPIDER ‡ APPs † HumanEval BaseNA0.00.012.1622.5032.4125.9831.7530.8214.7549.61 LoRA r = 1 0.03.043.2263.3315.1731.8630.7543.2924.535.66 r = 2 0.06.143.5963.7524.1334.8031.5042.4523.2534.88 r = 4 0.0 12.245.9369.5844.1334.3131.7544.8425.7537.21 r = 8 0.0 24.347.1573.3351.7234.8032.0044.3629.2535.66 r = 16 0.0 48.647.6173.754.1334.3133.2547.7229.034.88 LoRA ∗ (Merged Dataset) r = 1 0.03.042.1069.17–33.2543.76– r = 2 0.06.144.2568.75–33.2543.40– r = 4 0.0 12.244.8173.33–32.2544.12– VeRA r = 1k 111.2 0.827.1370.0066.8931.374.7540.3338.515.50 r = 2k 222.3 1.628.0670.0072.4134.804.5041.0141.7513.95 r = 4k 444.6 3.228.0672.5070.3433.824.7540.6044.7513.95 CTS (JD-Full) r = 1 1.51.53.1858.332.7622.5534.548.0835.2537.98 r = 2 3.03.16.6466.674.1422.5533.7546.435.7539.53 r = 4 5.96.232.9371.254.8232.8432.2542.4430.041.03 r = 8 11.9 12.440.7972.084.1329.4130.038.4827.7538.75 r = 16 23.9 24.843.2172.914.1333.8231.534.2928.033.33 SALT (Ours) r = 1 48.6 3.0 45.65 (+2.4) 74.17 (+10.8) 87.58 (+72.4) 31.86 (+0.0) 32.75 (+2.0) 46.04 (+2.7) 27.0 (+2.5) 41.86 (+6.2) r = 2 48.6 6.1 46.02 (+2.4) 73.33 (+9.5) 85.51 (+61.4) 31.86 (-2.9) 33.00 (+1.5) 44.72 (+2.2) 24.25 (+1.0) 41.08 (+6.2) r = 4 48.6 12.2 45.27 (-0.6) 73.75 (+3.7) 86.21 (+42.1) 30.88 (-3.4) 33.25 (+1.5) 45.44 (+0.6) 24.75 (-1.0) 42.63 (+5.4) ‡ Datasets utilized for Phase 1 centroid training and Phase 2 residual tuning. † Datasets for Phase 2 residual tuning only. Table 8: Task performance and memory footprint across different ranks and baselines on Llama-3.2-3B. Memory (MB)Math & Reasoning TasksCoding Tasks Method RankPin Swap GSM8K ‡ SVAMP ‡ MultiArith † AQuA † MBPP ‡ SPIDER ‡ APPs † HumanEval BaseNA–1.310.80.70.06.250.09.01.83 LoRA r = 10.00 2.21 13.2836.254.1317.6412.02.7612.54.27 r = 20.00 4.42 13.8440.07.5916.6711.750.012.257.93 r = 40.00 8.85 16.5641.676.2118.6211.753.7213.252.44 r = 80.00 17.70 17.1140.838.2721.5712.54.0811.758.54 r = 160.00 35.39 18.3343.336.8917.1612.251.7012.02.44 LoRA ∗ (Merged Dataset) r = 10.03.012.6223.75–13.00.0– r = 20.06.114.0331.67–11.250.12– r = 40.0 12.2 15.3431.25–15.250.0– VeRA r = 1024 62.92 0.292.1539.589.6517.645.8017.7914.500.61 r = 2048 125.83 0.592.7140.839.6515.685.6019.0515.500.00 r = 4096 251.66 1.182.9944.1710.3415.686.4017.9916.750.00 CTS (JD-Full) r = 10.74 1.481.782.924.820.011.250.09.250.0 r = 21.48 2.952.1530.834.130.4911.50.08.250.0 r = 42.95 5.904.5835.06.9010.789.50.012.50.0 r = 85.90 11.80 9.4539.586.9014.2211.250.011.750.0 r = 16 11.80 23.60 16.4640.04.1415.2011.250.2411.750.0 SALT (Ours) r = 135.39 2.21 17.68 (+4.4) 44.58 (+8.3) 57.24 (+53.1) 15.69 (-1.9) 12.25 (+0.2) 42.20 (+29.7) 13.25 (+0.8) 1.83 (-2.4) r = 235.39 4.42 16.93 (+3.1) 42.5 (+2.5) 51.72 (+44.1) 13.73 (-2.9) 12.0 (+0.2) 41.72 (+41.7) 14.25 (+2.0) 1.22 (-6.7) r = 435.39 8.84 16.65 (+0.1) 43.33 (+1.7) 49.66 (+43.5) 19.60 (+1.0) 12.75 (+1.0) 40.29 (+36.6) 14.50 (+1.3) 1.22 (-1.2) ‡ Datasets utilized for Phase 1 centroid training and Phase 2 residual tuning. † Datasets for Phase 2 residual tuning only. Table 9: Peak performance and memory footprint for Pythia-12B Listing 1: Representative failure case: the naively concatenated centroid answers correctly on its own, but a Phase 2 residual trained on top of it hallucinate an unnecessary table join. 1 / * ================================================================= 2 QUESTION: Return the document id, template id, and description for the document with the name Robbin CV. 3 ================================================================= * / 4 -- [1] GOLD QUERY (Reference Answer) 5 SELECT document_id, template_id, Document_Description FROM Documents WHERE document_name = "Robbin CV"; 6 -- [2] CONCATENATED CENTROID ONLY, NO RESIDUAL (Correct) 7 SELECT document_id, template_id, document_description FROM Documents WHERE document_name = "Robbin CV"; 8 -- [3] CONCATENATED CENTROID + RESIDUAL (Failed) 9 SELECT T1.document_id, T1.template_id, T2.document_description FROM Documents AS T1 JOIN Paragraphs AS T2 10 ON T1.document_id = T2.document_id WHERE T2.document_name = "Robbin CV"; 11 / * ----------------------------------------------------------------- 12 ERROR: Adding the residual hallucinates an unnecessary JOIN with the "Paragraphs" table and incorrectly aliases schema columns, overcomplicating a simple single-table lookup. 13 ----------------------------------------------------------------- * / PCIeVRAM Adapters vLLM vLLM+SALT vLLM vLLM+SALT 3273.82 79.98 (+8.3%) 73.73 76.97 (+4.4%) 6462.03 70.79 (+14.1%) 62.23 73.14 (+17.5%) 12851.65 65.91 (+27.6%) 50.24 62.10 (+23.6%) 25638.70 58.39 (+50.9%) 28.60 36.68 (+28.3%) Table 10: Throughput for Llama-3.2-3B (req/s) PCIeVRAM Adapters vLLM vLLM+SALT vLLM vLLM+SALT 1664.20 65.36 (+1.8%) 11.27 11.53 (+2.3%) 3262.67 64.74 (+3.3%) 10.47 11.18 (+6.8%) 6456.51 55.78 (-1.3%) 8.41 9.79 (+16.4%) 12847.40 53.89 (+13.7%) 5.11 7.27 (+42.3%) Table 11: Throughput for Pythia-12B (req/s). gle task, whereas SALT routes task representations through a unified centroid to enable efficient multi-tenant serving. Similar to our primary evaluation methodology, we con- ducted microbenchmarks and vLLM throughput tests to pro- file memory overhead and inference latency for SALT against all baselines on these models (Figures 9 and 10). SALT achieves either the lowest or second-lowest adapter mem- ory footprint and the lowest inference latency, while match- ing baseline downstream accuracy. As shown in Tables 10 and 11, plugging SALT directly into vLLM yields throughput improvements of up to 50.9% under PCIe bandwidth pres- sure for Llama-3.2-3B and up to 42.3% under HBM capacity contention for Pythia-12B. A.3 Multi-Domain Task Interference We compare the performance of task residuals fine-tuned over a domain-specific (Math) centroid and a multi-domain (Math + Code) centroid while fixing the centroid rank to 16. As shown in Table 12, integrating more data distributions into a single Phase 1 centroid may lead to degrading performance due to negative transfer. GSM8K SVAMP 2-Tasks 46.6574.17 4-Tasks 44.0372.33 6-Tasks 43.9270.0 ∆−2.73−4.17 Table 12: Impact of multi-domain centroid training on down- stream task performance. Expanding the shared centroid (Math) to include out-of-domain tasks (Code and Language) degrade the performance of downstream tasks. A.4 Training Resource Complexity # Adapters (M)1020304050 Peak HBM (GB) 31.14 34.76 38.38 42.01 45.63 Step Latency (s)2.906.079.86 13.49 18.24 Table 13: Phase 1 joint alignment training scalability on Llama-3.2-3B on an NVIDIA H100. In Phase 1, SALT jointly trains a shared centroid with M standard (r = 16) task adapters in the same domain. The acti- vation memory stays bounded because adapters are processed sequentially, and each compute graph is freed immediately after its backward pass. However, the weights and optimizer states for allM adapters remain resident throughout training, scales asO(M ), growing by∼360 MB per adapter (Table 13) for Llama-3.2-3B adapter targeting all linear layers. Per-step latency scales similarly. Even at M = 50, a single training step completes in ∼18.2s and fits comfortably within the 80 GB memory capacity of a single H100 GPU. A.5 Effect of Subspace-Aligned Training Accuracy % Rank Centroid Method GSM8K SVAMP MBPP SPIDER r = 1 Concatenated52.10 64.17 22.07.07 Subspace-aligned 56.88 69.17 27.25 56.69 r = 2 Concatenated52.94 63.75 22.5 12.47 Subspace-aligned 57.81 69.17 27.8 56.35 r = 4 Concatenated52.67 65.00 22.75 12.11 Subspace-aligned 53.23 68.75 27.25 56.47 Table 14: Effect of subspace-aligned centroid training on downstream task performance for Mistral-7B-v0.3. Table 14 shows the full results underlying the paired t-test in Section 5.4. Spider shows a distinct failure pattern. The LoRA (Merged Dataset) baseline in Table 1, where the cen- troid trained on concatenated data, without any Phase 2 resid- ual, already achieves∼50% accuracy, yet adding a Phase 2 residual on top of this same centroid degrades performance to as low as 7.07% at r = 1. Accuracy also remains unstable across rank for this configuration (7.07–12.47%), whereas the residual trained on the frozen subspace-aligned centroid consistently achieve high performance (56.35–56.69%) re- gardless of residual rank. Listing 1 shows an example of this failure. The concatenated centroid alone answers cor- rectly, but training a residual on top of it causes the residual to hallucinate a query (i.e. JOIN Paragraphs AS T2) rather than refine the centroid’s correct output. This suggests align- ment is a precondition for safe residual composition: on an unaligned centroid, the residual’s limited capacity produces arbitrary structural errors rather than refinements. A.6 Fine-grained Sweep of γ As introduced in Section 3.3, SALT discretizes the inference- time scaling coefficient γ into predefined bins to group re- quests that share the same centroid and enable efficient cen- troid fusion. To empirically validate the practicality of this design, we performed a fine-grained sweep of γ ∈ [0.2, 1.5] with a step size of 0.1, measuring downstream performance across seven datasets on Mistral-7B-v0.3. 0.20.30.40.50.60.70.80.91.01.11.21.31.41.5 Centroid Scale (γ) GSM8K MULTIARITH SVAMP AQUA APPS MBPP SPIDER 0.0 0.2 0.4 0.6 0.8 Normalized Accuracy Figure 11: Normalized downstream performance across a fine-grained sweep of the centroid scale factor γ. Figure 11 shows that three of the four mathematical reason- ing datasets achieve optimal performance within the 0.6–0.9 range before degrading sharply. AQuA remains an outlier, requiring a higher γ (near 1.3) to fully recover accuracy. Conversely, all three coding datasets reach peak accuracy when γ ∈ [0.7, 0.9]. Crucially, within this 0.6–0.9 optimal band, the average performance drop from the absolute peak is merely 3.23% for math tasks and 1.53% for coding tasks. This stability empirically confirms that adopting a small set of shared γ bins (e.g., γ ∈0.6, 0.70.8, 0.9) effectively op- timizes the downstream task performance while improving the serving throughput.