Paper deep dive
CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling
Maoqi Liu, Quan Fang, Yufei He
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.08312v1
- Canonical: https://arxiv.org/abs/2610.08312v1
Trouble viewing inline? Open PDF directly →
Full Text
71,523 characters extracted from source content.
Expand or collapse full text
CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling Maoqi Liu Affiliation: Beijing University of Posts and Telecommunications Quan Fang Affiliation: Beijing University of Posts and Telecommunications Yufei He Affiliation: National University of Singapore Abstract Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an “Orthogonality Dilemma”: rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA. 🖂🖂footnotetext: Corresponding author: qfang@bupt.edu.cn. Figure 1: Conceptual illustration of parameter space utilization. Traditional LoRA suffers from interference, while O-LoRA’s strict orthogonality hinders knowledge sharing. CoDe-LoRA leverages null space projection to consolidate shared knowledge. 1 Introduction Large Language Models (LLMs) have redefined the boundaries of natural language processing, exhibiting remarkable capabilities across a wide range of tasks Wei et al. (2021). However, adapting these models to a continuous stream of evolving tasks, known as Continual Learning (CL) De Lange et al. (2021), remains challenging. While parameter-efficient fine-tuning (PEFT) methods, particularly low-rank adaptation (LoRA) Hu et al. (2021), have significantly lowered the cost of adaptation, naively applying them to sequential tasks leads to catastrophic forgetting Li and Hoiem (2017), where learning new tasks degrades previously acquired capabilities on earlier tasks. To mitigate forgetting, recent work combines PEFT with orthogonal projection strategies Saha et al. (2021); Wang et al. (2023a). A representative example is O-LoRA Wang et al. (2023a), which isolates task-specific parameters in mutually orthogonal subspaces. Although effective at reducing inter-task interference, such rigid isolation introduces a trade-off between interference mitigation and knowledge transfer. When sequential tasks share semantic structure, their optimal parameter updates should naturally overlap. Enforcing strict orthogonality removes these shared components, reducing representation reuse and weakening cross-task transfer. As illustrated in Figure 1, this phenomenon reveals an “Orthogonality Dilemma”: the same geometric constraint that protects previous knowledge can also restrict transferable representation learning across related tasks. This limitation highlights a key challenge in continual adaptation: how to preserve prior knowledge while still accumulating transferable representations across related tasks. Motivated by this tension, we investigate two questions: Q1. Why does strict orthogonality limit generalization across related tasks, and under what conditions is this limitation most severe? Q2. How can we design a parameter-efficient framework that preserves transferable representations while maintaining task-specific expertise in a replay-free setting? In this paper, we introduce Consolidation and Decoupling LoRA (CoDe-LoRA), a dual-branch continual adaptation framework motivated by the orthogonality-induced transfer limitation. We provide a geometric characterization showing that strict orthogonal isolation becomes increasingly detrimental as task similarity grows. CoDe-LoRA separates transferable representation consolidation from task-specific specialization through two complementary branches. Unlike strict orthogonal isolation methods that suppress overlapping representations entirely, our framework preserves transferable components while constraining interference-prone directions. Specifically, the Consolidation Branch (Co-LoRA) maintains shared transferable representations through adaptive null-space projection with dynamic scaling to stabilize long-horizon continual adaptation, as empirically verified in Appendix A.2 (Table 5). Meanwhile, the Decoupling Branch (De-LoRA) maintains a pool of independent LoRA experts together with a prototype-based semantic routing mechanism for robust inference without explicit task identifiers. Importantly, CoDe-LoRA operates without relying on historical data replay, avoiding additional storage overhead and privacy concerns. Our main contributions are summarized as follows: • We identify and analyze the Orthogonality Dilemma in PEFT-based continual learning, showing that strict parameter isolation becomes increasingly detrimental as task similarity grows. • We propose CoDe-LoRA, a dual-branch continual adaptation framework that separates transferable representation consolidation from task-specific specialization through adaptive null-space projection and prototype-based expert decoupling. • Extensive experiments across four backbones and three continual learning benchmarks show that CoDe-LoRA achieves the best average accuracy while balancing knowledge transfer and interference mitigation. Figure 2: Preliminary evidence for the orthogonality dilemma and transferable representation accumulation. (a) Orthogonality-induced degradation under different task similarity levels. (b) Transfer trajectories over repeated training with and without orthogonal constraints. (c) Next-task zero-shot transfer comparison across continual adaptation methods. (d) Forward transfer dynamics across sequential tasks. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning Parameter-Efficient Fine-Tuning (PEFT) adapts Large Language Models (LLMs) with minimal additional parameters. Early approaches such as Adapters Houlsby et al. (2019) and Prompt/Prefix Tuning Li and Liang (2021); Lester et al. (2021) introduced lightweight modules while freezing the backbone. LoRA Hu et al. (2021) further improves efficiency through low-rank matrices. Although effective for static tasks, sequential adaptation with PEFT methods often leads to catastrophic forgetting of earlier tasks as training proceeds. 2.2 Continual Fine-Tuning for LLMs Continual fine-tuning methods mainly rely on regularization Kirkpatrick et al. (2017), replay Qin and Joty (2021); He et al. (2024), or parameter expansion Razdaibiedina et al. (2023). Expansion-based methods avoid interference through task-specific parameters, but their parameter cost grows with the number of tasks. Replay and regularization methods preserve prior knowledge through memory buffers or parameter constraints, yet often incur computational overhead or privacy concerns. Recently, mixture-of-experts (MoE) routing frameworks such as MoLE-CIE Wang et al. (2025) have been introduced for continual adaptation; however, they typically still rely on historical exemplar replay to prevent routing drift and maintain stable performance over long task sequences and domains. More recently, orthogonality-based methods such as O-LoRA Wang et al. (2023a), CLoRA Lu et al. (2025), and N-LoRA Yang et al. (2025b) mitigate forgetting by projecting updates into orthogonal or null subspaces. Although effective at reducing interference, strict orthogonal isolation may limit representation reuse across related tasks. In contrast, CoDe-LoRA separates shared representation consolidation from task-specific specialization, balancing knowledge transfer and interference mitigation through adaptive null-space projection without historical data replay. 3 Preliminary 3.1 Problem Definition We consider continual learning (CL) over a sequence of tasks (1),(2),…,(T)\T^(1),T^(2),…,T^(T)\. Each task (t)T^(t) is associated with a dataset (t)=(x,y)D^(t)=\(x,y)\, where x denotes the input prompt and y the target sequence. Under the PEFT setting, the backbone parameters θbase _base remain frozen while task-specific updates Δθ(t) θ^(t) are learned with a small number of trainable parameters. We consider the replay-free setting, where no data from earlier tasks is stored, and task identity is not given at inference time. Sequentially fine-tuning on new tasks often leads to catastrophic forgetting of previous ones. 3.2 Orthogonal Continual Adaptation To mitigate forgetting, O-LoRA Wang et al. (2023a) constrains each task update to be orthogonal to previous task subspaces. For task t, the LoRA update is parameterized as ΔWt=BtAt W_t=B_tA_t, where At∈ℝr×dinA_t ^r× d_in and Bt∈ℝdout×rB_t ^d_out× r. Orthogonality is imposed through: AiAt⊤=0,∀i<tA_iA_t =0, ∀ i<t (1) In practice it is enforced by a regularization penalty. 3.3 Orthogonality Dilemma Although orthogonal constraints reduce interference, they may also suppress transferable representations across closely related tasks. Let ΔWt∗ W_t^* be the update that unconstrained fine-tuning learns on task t, and let P<tP_<t be the orthogonal projector onto the row space of A<t=[A1;…;At−1]A_<t=[A_1;…;A_t-1]. We measure how much of this update is aligned with earlier tasks by ρt=‖ΔWt∗P<t‖F2‖ΔWt∗‖F2∈[0,1]. _t= \| W_t^*P_<t\|_F^2\| W_t^*\|_F^2∈[0,1]. (2) Enforcing AiAt⊤=0A_iA_t =0 for all i<ti<t forces ΔWtP<t=0 W_tP_<t=0, so the closest feasible update is ΔWt∗(I−P<t) W_t^*(I-P_<t), which discards exactly a ρt _t fraction of the update energy. The more related the tasks, the more reusable knowledge is lost. This creates an “Orthogonality Dilemma”: the constraint that protects previous knowledge also limits transfer between related tasks. 3.4 Preliminary Evidence and Mechanism Validation Figure 2 demonstrates these dynamics: (a) orthogonality-induced degradation worsens as task similarity grows; (b) unconstrained fine-tuning accumulates transferable knowledge over iterations, whereas strict orthogonality suppresses it; (c) preserving shared representations improves zero-shot transfer; and (d) transfer gains remain positive across task sequences. Mechanism Validation. To investigate the underlying cause, we define task similarity as the cosine between task centroids, sim((i),(j))=c((i))⋅c((j))sim(T^(i),T^(j))=c(T^(i))· c(T^(j)), where c()c(T) is the L2-normalized mean of mask-pooled, L2-normalized frozen T5-large embeddings of 200 training inputs, and measure update overlap by ρt _t (Eq. 2) on each task’s saved low-rank update. We compare O-LoRA (rank 10) with and without its orthogonality penalty, holding the Order-1 data, schedule, and seed fixed. The penalty (λ=0.5λ=0.5) suppresses the prior-subspace energy from 5.05–5.63% to 0.47–0.64% (89.0% on average), while removing it (λ=0λ=0) raises the mean next-task forward transfer from 43.70 to 49.45 (+5.75). This confirms that strict orthogonality suppresses update components that contribute to transfer, motivating our dual-branch design. Figure 3: The CoDe-LoRA framework. (Left) Training: Knowledge is disentangled into task-specific experts and a shared branch consolidated via null space projection. (Right) Inference: Inputs are routed to experts via prototype similarity, with a confidence-based fallback to the shared branch for robust prediction. 4 Methodology 4.1 Overall Framework To resolve the orthogonality dilemma (Section 3.3), we propose CoDe-LoRA (illustrated in Figure 3), a dual-branch framework designed to consolidate shared representations while decoupling task-specific expertise. Built upon a frozen backbone θbase _base, our architecture operates through two parallel pathways: the Consolidation Branch (Co-LoRA, θshared _shared), which accumulates transferable representations across tasks via null space projection (Sec. 4.2), and the Decoupling Branch (De-LoRA), which maintains a pool of independent task-specific LoRA experts θspeckk=1T\ _spec^k\_k=1^T that captures domain nuances without interference (Sec. 4.3). These components are ultimately integrated via a confidence-based fallback selection strategy (Sec. 4.4) to ensure robust inference. 4.2 Consolidating Shared Representations While strict parameter isolation (e.g., O-LoRA) prevents catastrophic forgetting, it often hinders the accumulation of shared representations across closely related tasks. Unlike strict orthogonal isolation methods that entirely discard overlapping update components, our consolidation branch preserves accumulated shared representations while constraining only newly introduced conflicting update directions. To achieve this objective, we propose a consolidation mechanism based on null space projection and dynamic scaling. Consolidation Algorithm. After training on each new task (t)T^(t), we update the parameters of the Consolidation Branch (Co-LoRA) through a three-step consolidation process. The Consolidation Branch is parameterized by low-rank weights Wshared=BsharedAsharedW_shared=B_sharedA_shared. Let Wacc(t−1)W_acc^(t-1) denote the accumulated shared weight matrix of Co-LoRA up to task t−1t-1, and let ΔWshared(t) W_shared^(t) represent the parameter update learned in the Consolidation Branch during the training of task t. Null Space Projection. We project ΔWshared(t) W_shared^(t) onto the null space of the existing representations represented by Wacc(t−1)W_acc^(t-1). Specifically, we perform Singular Value Decomposition (SVD) on Wacc(t−1)∈ℝdout×dinW_acc^(t-1) ^d_out× d_in to obtain its top-dvd_v left singular vectors U∈ℝdout×dvU ^d_out× d_v, which form an orthonormal basis for its column space. Here, dv≤rd_v≤ r is the rank of the accumulated column space (we use dv=rd_v=r). The new component ΔWnull W_null is then computed by removing the projection of ΔWshared(t) W_shared^(t) onto this subspace: ΔWnull=ΔWshared(t)−UU⊤ΔWshared(t) W_null= W_shared^(t)-U W_shared^(t) (3) Intuitively, the projection keeps only what is new in each task, so earlier representations are not overwritten. This projection reduces interference with previously consolidated representations. To improve scalability for large LLMs, we implement the projection using randomized low-rank SVD directly on factorized LoRA matrices. Details and efficiency analysis are provided in Appendix A.3. Dynamic Scaling Constraint. To maintain the stability of the parameter magnitude, we introduce time-varying scaling coefficients ct=t−1tc_t= t-1t and st=1ts_t= 1 t satisfying the normalization identity ct2+st2=1c_t^2+s_t^2=1. Since the null space projection ensures that each new update is orthogonal to the existing parameters, we scale the null space update component by sts_t. This constraint prevents the cumulative parameter magnitude from exploding or drifting as the number of sequential tasks increases. We provide the theoretical proof of the stable parameter bound and its empirical verification in Appendix A.2. The final consolidated weight matrix is updated by merging the scaled null space component with the previous state: Wacc(t)=ctWacc(t−1)+stΔWnullW_acc^(t)=c_tW_acc^(t-1)+s_t W_null (4) This recursive formulation ensures that the consolidated weight matrix represents a normalized accumulation of orthogonal task updates. The choice of scaling coefficients stabilizes the parameter magnitude across tasks as follows: Proposition 1. Given the recursive update rule defined in Equation (4), assuming the initial LoRA weights are zero (Wacc(0)=W(0)=0W_acc^(0)=W^(0)=0) and that U spans the column space of Wacc(t−1)W_acc^(t-1), the distance between the consolidated model and the pre-trained model remains bounded: ‖Wacc(t)−W(0)‖F2≤Mt \|W_acc^(t)-W^(0) \|^2_F≤ M_t (5) where Mt=maxi∈[1,t]‖ΔWshared(i)‖F2M_t= _i∈[1,t] \| W_shared^(i) \|^2_F. 4.3 Decoupling Task-Specific Expertise While the Consolidation Branch captures transferable representations, task-specific nuances are isolated within the Decoupling Branch (De-LoRA), consisting of a pool of task-specific experts θspeckk=1T\ _spec^k\_k=1^T. To enable task-agnostic inference, we introduce a semantic routing mechanism that leverages task prototypes to dynamically activate the most relevant De-LoRA expert for each input. Prototype Construction. For each task, we construct a prototype Pk∈ℝdmodelP_k ^d_model as the normalized mean of the frozen-backbone embeddings fθbase(xi)f_ _base(x_i) of N randomly selected training samples (fixed to N=10N=10 throughout experiments). Crucially, because prototypes and runtime query representations are extracted solely using the frozen base model θbase _base rather than adapter-modulated layers, the semantic matching space does not drift over sequential learning phases. Routing Procedure. During inference, the system identifies the optimal De-LoRA expert θspeck∗ _spec^k^* for a given input x by measuring the similarity between the input embedding ex=fθbase(x)e_x=f_ _base(x) and the stored prototypes Pkk=1T\P_k\_k=1^T. Since prototypes are pre-normalized, the routing index k∗=argmaxkex⋅Pk‖ex‖k^*= _k e_x· P_k\|e_x\| is the prototype with the highest cosine similarity. Routing stores only compact per-task statistics rather than raw samples, so it remains replay-free. The selected expert θspeck∗ _spec^k^* is then activated and integrated with the Consolidation Branch for inference via the fallback strategy described in Section 4.4. This similarity-based routing facilitates positive forward transfer: for inputs that bridge multiple task distributions, the mechanism naturally selects the most semantically related expert, enabling the model to leverage relevant prior knowledge. Appendix A.5 compares the routers and the cost of routing errors. Table 1: Performance comparison of different methods using the T5 model on Standard CL Benchmark and Large Number of Tasks. The average accuracy after training on the final task is reported. The top two results (excluding the oracle and MTL references) are bolded and underlined. Subscripts denote standard deviations across 3 random seeds. Method Standard CL Benchmark Large Number of Tasks Order-1 Order-2 Order-3 Avg Order-4 Order-5 Order-6 Avg Ref. Per-LoRA (Oracle) 81.4 81.3 81.4 81.4 77.6 77.4 77.5 77.5 MTL 80.0 80.0 80.0 80.0 76.5 76.5 76.5 76.5 Seq. FT SeqFT 18.9 24.9 41.7 28.5 7.4 7.4 7.5 7.4 SeqLoRA 44.6 32.7 53.7 43.7 0.6 1.9 1.6 1.4 IncLoRA 66.0 64.9 68.3 66.4 63.3 58.5 61.7 61.2 Classic CL Replay 55.2 56.9 61.3 57.8 55.0 54.6 53.1 54.2 EWC 48.7 47.7 54.5 50.3 45.3 44.5 45.6 45.1 LwF 54.4 53.1 49.6 52.3 50.1 43.1 47.4 46.9 L2P 60.3 61.7 61.1 61.0 57.5 53.8 56.9 56.1 LFPT5 67.6 72.6 77.9 72.7 70.4 68.2 69.1 69.2 Advanced CL O-LoRA 75.4 75.7 76.3 75.8 72.3 64.8 71.6 69.6 MIGU 77.1 77.0 75.6 76.6 67.3 68.5 74.2 70.0 N-LoRA 79.2 78.4 78.8 78.8 73.6 70.3 73.2 72.4 CLoRA 79.7 79.1 78.2 79.0 70.7 65.6 68.2 68.1 Co-LoRA 75.9± 0.2 75.5± 0.2 75.3± 0.7 75.6± 0.4 64.0± 1.7 67.4± 1.3 63.6± 1.8 65.0± 1.9 De-LoRA 79.4± 0.3 79.2± 0.1 79.3± 0.1 79.3± 0.2 76.7± 0.4 76.6± 0.8 76.2± 0.8 76.5± 0.7 CoDe-LoRA (Ours) 80.3± 0.2 80.4± 0.2 80.5± 0.3 80.4± 0.2 78.6± 0.5 78.8± 0.7 79.8± 0.3 79.1± 0.7 4.4 Knowledge Integration and Fallback Selection To produce the final prediction, we employ a confidence-based fallback selection strategy that prioritizes specialized expertise while leveraging the consolidated shared branch as a robust prior. Here, the routing confidence is defined as the maximum cosine similarity between the input embedding and task prototypes. If the routing confidence exceeds a threshold τ (a hyperparameter set to 0.750.75 by default), the system routes inference to the task-specific expert θspeck∗ _spec^k^*, executing y^=Model(x,θbase,θspeck∗) y=Model(x; _base, _spec^k^*). Otherwise, the model falls back to the consolidated shared branch, executing y^=Model(x,θbase,θshared) y=Model(x; _base, _shared). A higher τ trusts the router only on clear matches and sends the rest to the shared branch, whereas τ=0τ=0 always trusts the routed expert. This fallback strategy dynamically balances stability and plasticity by preserving high-capacity specialized expertise while utilizing the consolidated branch as a robust safety net for ambiguous or out-of-distribution inputs. For classification tasks, candidate-label scores are additionally PMI-calibrated. The complete training and consolidation workflow is detailed in Appendix A.3. Both branches share the same frozen backbone, so no backbone parameters are updated. 5 Experiments 5.1 Experimental Setup Datasets. We evaluate CoDe-LoRA on three benchmarks: the Standard CL Benchmark Zhang et al. (2015), the Large Number of Tasks Benchmark Razdaibiedina et al. (2023), and the TRACE benchmark Wang et al. (2023b) which contains diverse generative tasks. Following Qin and Joty (2021), we evaluate across three distinct task orders for each benchmark to ensure robustness. The complete specifications, including task sequences, dataset statistics, and natural language instructions, are detailed in Appendix A.8 and A.9. Table 2: Results across backbones. Std./Long: average accuracy on the Standard CL/Long Sequence benchmarks; OP/BWT: Trace metrics. Co/De-LoRA: single-branch ablations. Top two per column are bolded and underlined. Method Llama-2-7B Qwen3-0.6B Qwen3.5-4B Std. Long OP BWT Std. Long OP BWT Std. Long OP BWT CLoRA 78.0 67.8 45.3 −-6.0 68.0 61.6 39.8 −-10.6 80.7 82.1 60.8 −-4.0 O-LoRA 77.0 67.8 44.2 −-3.6 70.7 64.0 41.3 −-5.2 80.4 83.3 60.2 −-1.3 N-LoRA 73.9 57.4 44.0 −-1.6 65.9 58.9 41.8 −-4.3 79.0 81.6 58.4 −-1.4 MoLE-CIE 68.1 70.6 36.2 −-17.2 69.5 69.3 43.0 −-6.1 70.9 78.4 49.1 −-17.8 Co-LoRA 73.0 59.6 43.3 −-2.0 66.4 60.1 41.3 −-4.6 78.9 77.5 56.8 −-2.3 De-LoRA 81.1 73.7 50.0 0.0 77.3 70.2 52.2 0.0 81.7 84.9 64.2 0.0 CoDe-LoRA 81.8 74.4 50.1 0.0 79.7 73.0 51.8 0.0 82.1 85.1 64.0 0.0 (a) Task Order 1 (b) Task Order 2 (c) Task Order 3 (d) Task Order 4 (e) Task Order 5 (f) Task Order 6 Figure 4: Performance curves of different methods across six task orders on the Standard CL Benchmark (Orders 1-3) and Large Number of Tasks Benchmark (Orders 4-6). Results are averaged over three independent runs. Evaluation Metrics. Let ai,ja_i,j denote the performance on task i after training on task j. To assess continual learning performance, we report two primary metrics: (1) Average Performance (AP), computed as the average task performance at the end of the continual sequence (Average Accuracy for classification benchmarks and Overall Performance for generative benchmarks); and (2) Backward Transfer (BWT), which measures the retention of previously learned knowledge across tasks. We additionally analyze forward transfer behavior to evaluate the model’s ability to accumulate transferable representations across sequential tasks. Formal definitions are provided in Appendix A.4. Baselines. We compare CoDe-LoRA against four categories of continual learning baselines: (1) upper-reference baselines, including Multi-Task Learning (MTL) and Per-LoRA (Oracle); (2) sequential fine-tuning methods, including SeqFT de Masson D’Autume et al. (2019), SeqLoRA, and IncLoRA; (3) continual learning methods, including EWC Kirkpatrick et al. (2017), LwF Li and Hoiem (2017), Replay Lopez-Paz and Ranzato (2017), L2P Wang et al. (2022), and LFPT5 Qin and Joty (2021); and (4) parameter-efficient continual learning methods, including O-LoRA Wang et al. (2023a), MIGU Du et al. (2024), N-LoRA Yang et al. (2025b), CLoRA Lu et al. (2025), and the routing-based MoLE-CIE Wang et al. (2025). Implementation Details. We evaluate CoDe-LoRA across different model families and scales using T5 Raffel et al. (2020), Llama 2-7B Touvron et al. (2023), Qwen3-0.6B Yang et al. (2025a), and Qwen3.5-4B Team (2026). All models are optimized with AdamW Loshchilov and Hutter (2017), training each task for one epoch with LoRA rank 8. The learning rate is 10−310^-3 for T5 and 10−410^-4 for Llama 2-7B and Qwen3.5-4B; on Qwen3-0.6B, the per-task-expert methods (De-LoRA, CoDe-LoRA) use 10−310^-3 and the single-shared-adapter methods use 10−410^-4. Results are averaged over three independent runs. Hyperparameters, routing details, and implementation settings are provided in Appendix A.3. 5.2 Main Results On the Standard CL Benchmark (Table 1), CoDe-LoRA achieves the highest average accuracy (80.4%), outperforming prior PEFT-based CL methods and exceeding the MTL reference (80.0%). This result suggests that the decoupled expert pool provides task-specific capacity while avoiding the forgetting typical of sequential fine-tuning. On the Large Number of Tasks benchmark, the performance gap is more pronounced: CoDe-LoRA achieves 79.1% average accuracy, outperforming the strongest baseline, N-LoRA, by 6.7 points (79.1% vs. 72.4%). While O-LoRA reduces interference through strict orthogonal isolation, its performance degrades on longer sequences where transferable representations become increasingly important. These results indicate stronger scalability in long-horizon scenarios. Generalization to Generative Tasks. We further evaluate CoDe-LoRA on the TRACE benchmark Wang et al. (2023b), which contains generative tasks. As shown in Table 2, CoDe-LoRA achieves competitive generation quality (OP) and consistently improves knowledge retention (BWT) without relying on replay buffers, indicating improved retention in generative domains. Figure 5: Ablation analysis of CoDe-LoRA showing task-wise accuracy and BWT under different ensemble strategies (left) and routing accuracy across varying prototype sample sizes N (right). Figure 6: Zero-shot forward transfer on Qwen3-0.6B (Order 1). “N/A” indicates trained tasks. Generalization across Model Families. To evaluate generalization across model architectures and scales, we evaluate CoDe-LoRA on Qwen3-0.6B, Qwen3.5-4B, and Llama 2-7B against isolation-based PEFT (O-LoRA, N-LoRA) and routing-based MoE (MoLE-CIE Wang et al. (2025)) baselines (Table 2). CoDe-LoRA outperforms every baseline on every backbone and benchmark. On Qwen3-0.6B it improves over the strongest baseline by 9.0 points on standard CL (79.7 vs. 70.7) and by 3.7 points on the long-sequence benchmark (73.0 vs. 69.3); the corresponding margins are 3.8 and 3.8 on Llama 2-7B and 1.4 and 1.8 on Qwen3.5-4B, and TRACE OP improves by 3.2 to 8.8 points. Because every task retains its own frozen expert, both De-LoRA and CoDe-LoRA show no measurable backward transfer loss on TRACE (BWT =0.0=0.0 on all three backbones), whereas the strongest baseline per backbone still loses between 1.3 and 4.3 points, and MoLE-CIE loses up to 17.8. The single-branch ablations locate the gain: Co-LoRA alone is the weakest of the three, De-LoRA already accounts for most of the accuracy, and adding the consolidation branch adds a further 0.4 to 2.8 points on Std. CL and Long Seq. On TRACE, where the two branches cannot be combined, CoDe-LoRA matches De-LoRA. All of this is achieved without maintaining historical replay buffers. 5.3 Ablation Studies and Analysis Capacity Control. We re-train O-LoRA and N-LoRA with r=10r=10 on T5-Large so that their per-task adapter footprint matches CoDe-LoRA at r=8r=8, keeping every other setting identical (three task orders, three seeds each). Raising the baseline rank changes little: O-LoRA moves from 76.5 to 76.4 on the Standard CL Benchmark and from 70.2 to 70.4 on the Long Sequence Benchmark, and N-LoRA does not improve either. Against the strongest capacity-matched baseline, CoDe-LoRA leads by 4.0 points (80.4 vs. 76.4) and by 8.7 points (79.1 vs. 70.4), respectively. The gains therefore stem from decoupled knowledge consolidation rather than from parameter capacity. Ensemble Strategy. We compare CoDe-LoRA with LoRAHub Huang et al. (2023), which directly merges task-specific LoRA modules. As shown in Figure 5, LoRAHub exhibits severe catastrophic forgetting, yielding a BWT of -20.2% compared to -0.2% for CoDe-LoRA. This suggests that direct parameter merging introduces significantly stronger cross-task interference. Forward Transfer. Figure 6 shows the zero-shot forward transfer matrix on Qwen3-0.6B (Order 1). CoDe-LoRA achieves consistently higher forward transfer than O-LoRA, particularly after progressively accumulating knowledge across tasks (e.g., +19.1 points on Yahoo after training on dbpedia and amazon). This suggests that the consolidation branch effectively accumulates transferable representations that benefit future unseen tasks. Impact of Prototype Samples. We analyze the sensitivity of routing performance to the number of prototype samples N. Figure 5 (right) shows that routing accuracy improves as N increases, with performance saturating around N=10N=10. This indicates that only a small number of samples are required for reliable task routing. Contribution of Each Branch. To further analyze the necessity of the proposed dual-branch design, we compare the Consolidation Branch alone (Co-LoRA), the Decoupling Branch alone (De-LoRA), and the full CoDe-LoRA architecture on the Standard CL Benchmark, as reported in the corresponding rows of Table 1. Table 3: Efficiency on Order 1 (T5-large, one H20 GPU). Train and inference are seconds per task and per evaluated test set; CoDe-LoRA is a single seed, the others are means over three seeds. Params are the stored adapter parameters (CoDe-LoRA stores only the experts) and their share of the backbone. Method Train (s) Infer (s) Params O-LoRA 108.8 30.1 9.4M (1.28%) N-LoRA 98.0 29.9 9.4M (1.28%) MoLE-CIE 861.0 142.7 32.4M (4.40%) CoDe-LoRA 98.4 43.1 9.4M (1.28%) The consolidation branch (Co-LoRA) alone captures shared representations, while the decoupling branch (De-LoRA) alone preserves task-specific adaptation. Combining both branches yields the best overall performance across all task sequences (Table 1). Appendix A.6 reports the complementarity of the two branches and the attempts to exploit it. Efficiency and Scalability Analysis. CoDe-LoRA trains one adapter per task on top of the consolidated branch, so a task costs as much as for O-LoRA and N-LoRA (98.4 s vs. 108.8 s and 98.0 s on the Standard benchmark, 50.1 s vs. 53.2 s and 47.9 s on the Long benchmark) and about 9×9× less than MoLE-CIE (Table 3). It stores only the experts, the same 9.4M adapter parameters as O-LoRA and N-LoRA on four tasks (1.28% of the backbone, and 35.4M or 4.80% on the fifteen tasks of the Long benchmark; Appendix Table 11); the consolidated branch and the per-task contexts are rebuilt by replaying the folds. The price is inference: replaying the context of the routed expert makes an evaluation pass about 1.4×1.4× slower than O-LoRA (43.1 s vs. 30.1 s). 6 Conclusion We revisit orthogonal continual adaptation from the perspective of transferable representation learning. We show that while strict orthogonal isolation mitigates catastrophic forgetting, it can also suppress transferable representation accumulation across related tasks. To address this limitation, we propose CoDe-LoRA, a dual-branch continual adaptation framework that balances shared representation consolidation and task-specific adaptation. Extensive experiments demonstrate strong continual learning performance together with robust forward and backward transfer across diverse task sequences and model families. Limitations Although CoDe-LoRA demonstrates strong performance across both classification and generative continual learning benchmarks, our evaluation is limited to pre-trained language models. Extending the framework to multi-modal continual learning settings remains an important direction for future work. In addition, the effectiveness of the routing mechanism depends on the quality of task prototypes in the embedding space. Under severe task distribution shifts or highly overlapping task boundaries, routing reliability may degrade and require more adaptive prototype estimation strategies. Acknowledgements This work was supported by Beijing Natural Science Foundation (JQ24019). This work was supported by the National Natural Science Foundation of China (No. 62576047). This work is being sponsored by CIPS-SMP-Zhipu Large Model Fund (No. CIPS-SMP20250313) and supported by the Open Fund of the Key Laboratory for Civil Aviation Collaborative Air Traffic Management Technology and Applications (No. 2025-001). References De Lange et al. (2021) M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars A continual learning survey: defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence 44 (7), p. 3366–3385. Cited by: §1. de Masson D’Autume et al. (2019) C. de Masson D’Autume, S. Ruder, L. Kong, and D. Yogatama Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems 32. Cited by: §5.1. Du et al. (2024) W. Du, S. Cheng, T. Luo, Z. Qiu, Z. Huang, K. C. Cheung, R. Cheng, and J. Fu Unlocking continual learning abilities in language models. In Conference on Empirical Methods in Natural Language Processing, Cited by: §5.1. He et al. (2024) J. He, H. Guo, K. Zhu, Z. Zhao, M. Tang, and J. Wang Seekr: selective attention-guided knowledge retention for continual learning of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 3254–3266. Cited by: §2.2. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. Cited by: §2.1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §1, §2.1. Huang et al. (2023) C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin LoraHub: efficient cross-task generalization via dynamic lora composition. External Links: 2307.13269 Cited by: §5.3. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. External Links: Link Cited by: §2.2, §5.1. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. Cited by: §2.1. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. Cited by: §2.1. Li and Hoiem (2017) Z. Li and D. Hoiem Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40 (12), p. 2935–2947. Cited by: §1, §5.1. Lopez-Paz and Ranzato (2017) D. Lopez-Paz and M. Ranzato Gradient episodic memory for continual learning. Advances in neural information processing systems 30. Cited by: §A.4, §5.1. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §A.3, §5.1. Lu et al. (2025) Y. Lu, B. Qian, C. Yuan, H. Jiang, and X. Wang Controlled low-rank adaptation with subspace regularization for continued training on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 19165–19181. Cited by: §2.2, §5.1. Maas et al. (2011) A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, p. 142–150. Cited by: §A.8. Qin and Joty (2021) C. Qin and S. Joty Lfpt5: a unified framework for lifelong few-shot language learning based on prompt tuning of t5. arXiv preprint arXiv:2110.07298. Cited by: §A.3, §A.9, §2.2, §5.1, §5.1. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (1). Cited by: §5.1. Razdaibiedina et al. (2023) A. Razdaibiedina, Y. Mao, R. Hou, M. Khabsa, M. Lewis, and A. Almahairi Progressive prompts: continual learning for language models. arXiv preprint arXiv:2301.12314. Cited by: §2.2, §5.1. Saha et al. (2021) G. Saha, I. Garg, and K. Roy Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762. Cited by: §1. Team (2026) Q. Team Qwen3.5: accelerating productivity with native multimodal agents. External Links: Link Cited by: §5.1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. External Links: Link Cited by: §5.1. Wang et al. (2019) A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32. Cited by: §A.8. Wang et al. (2018) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, p. 353–355. External Links: Document, Link Cited by: §A.8. Wang et al. (2023a) X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152. Cited by: §A.3, §1, §2.2, §3.2, §5.1. Wang et al. (2023b) X. Wang, Y. Zhang, T. Chen, S. Gao, S. Jin, X. Yang, Z. Xi, R. Zheng, Y. Zou, T. Gui, et al. Trace: a comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762. Cited by: §A.8, §A.8, §5.1, §5.2. Wang et al. (2022) Z. Wang, Z. Zhang, C. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 139–149. Cited by: §5.1. Wang et al. (2025) Z. Wang, X. Wang, and W. Hu Mixture of LoRA experts for continual information extraction with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 13324–13339. External Links: Document, Link Cited by: §2.2, §5.1, §5.2. Wei et al. (2021) J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. Cited by: §1. Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §5.1. Yang et al. (2025b) S. Yang, K. Ning, Y. Liu, J. Yao, Y. Tian, Y. Song, and L. Yuan Is parameter collision hindering continual learning in LLMs?. In Proceedings of the 31st International Conference on Computational Linguistics, p. 4243–4259. External Links: Link Cited by: §A.3, §2.2, §5.1. Zhang et al. (2015) X. Zhang, J. Zhao, and Y. LeCun Character-level convolutional networks for text classification. Advances in neural information processing systems 28. Cited by: §A.8, §5.1. Appendix A Appendix A.1 Notation We summarize the key notations used throughout this paper in Table 4. Table 4: Summary of key notations. Notation Description (t)T^(t) The t-th task in the sequence (t)D^(t) Dataset associated with task (t)T^(t) θbase _base Parameters of the frozen backbone θshared _shared Parameters of consolidation branch θspeck _spec^k Parameters of k-th task expert Wacc(t)W_acc^(t) Accumulated shared weights up to task t ΔWshared(t) W_shared^(t) Consolidation branch update at task t ΔWnull W_null Null space projected update component ct,stc_t,s_t Dynamic scaling coefficients at task t PkP_k Semantic prototype for task k exe_x Embedding of input x from backbone τ Fallback confidence threshold A.2 Theoretical Justification for Dynamic Scaling In the model merging process, the dynamic time-varying scaling coefficients are introduced to maintain the deviation magnitude between the merged model and the pre-trained model. General Formulation of Scaling. For clarity, we denote the parameter deviation at step t as Dt=Wacc(t)−W(0)D_t=W_acc^(t)-W^(0), the null-space projected update component as Vt=ΔWnull(t)=PnullΔWshared(t)V_t= W_null^(t)=P_null W_shared^(t), and the running average norm of task updates as μt=1t∑i=1t‖ΔWshared(i)‖F _t= 1t _i=1^t\| W_shared^(i)\|_F. In our study, the generalized scaling factor λt _t is defined as the inverse of the scaling coefficient sts_t (i.e., λt=1/st _t=1/s_t), and can be computed as: λt=λt−1⋅‖Dt−1+Vt‖Fμt _t= _t-1· \|D_t-1+V_t\|_F _t (6) where PnullP_null denotes the null space projection operator. Derivation of Scaling Coefficients. Proof of Proposition 1. To simplify notation in this proof, we denote the consolidated weight matrix Wacc(t)W_acc^(t) as WtW_t, and the null-space projected update component ΔWnull(t) W_null^(t) as VtV_t. Recall that we assume W(0)=W0=0W^(0)=W_0=0. Under the update rule defined in Eq. (4), the consolidated weight matrix at task t is updated recursively as: Wt=ctWt−1+stVtW_t=c_tW_t-1+s_tV_t (7) where the scaling coefficients ct=(t−1)/tc_t= (t-1)/t and st=1/ts_t=1/ t satisfy the normalization identity ct2+st2=1c_t^2+s_t^2=1. By definition, the projected component VtV_t lies in the null space of the accumulated representations Wt−1W_t-1, which implies they are orthogonal under the Frobenius inner product (exactly when U spans the column space of Wt−1W_t-1, approximately for a truncated basis): ⟨Wt−1,Vt⟩F=0 W_t-1,V_t _F=0 (8) Using this orthogonality, and given the boundary condition W0=0W_0=0, we have ‖Wt−W0‖F2=‖Wt‖F2\|W_t-W_0\|^2_F=\|W_t\|^2_F. Applying the Pythagorean theorem to the update rule in Eq. (7) yields: ‖Wt‖F2=ct2‖Wt−1‖F2+st2‖Vt‖F2\|W_t\|^2_F=c_t^2\|W_t-1\|^2_F+s_t^2\|V_t\|^2_F (9) Let Mt=maxi∈[1,t]‖ΔWshared(i)‖F2M_t= _i∈[1,t]\| W_shared^(i)\|^2_F. We prove that ‖Wt‖F2≤Mt\|W_t\|^2_F≤ M_t by induction on t. For the base case t=1t=1, since c1=0c_1=0 and s1=1s_1=1, we have: ‖W1‖F2=‖V1‖F2≤M1\|W_1\|^2_F=\|V_1\|^2_F≤ M_1 (10) where the inequality holds because the projection operator is a contraction mapping, i.e., ‖Vt‖F≤‖ΔWshared(t)‖F\|V_t\|_F≤\| W_shared^(t)\|_F. Now, assume that the bound holds up to task t−1t-1, i.e., ‖Wt−1‖F2≤Mt−1\|W_t-1\|^2_F≤ M_t-1. For task t, substituting the induction hypothesis and the contraction property into Eq. (9): ‖Wt‖F2 \|W_t\|^2_F ≤ct2Mt−1+st2‖ΔWshared(t)‖F2 ≤ c_t^2M_t-1+s_t^2\| W_shared^(t)\|^2_F (11) ≤ct2Mt+st2Mt ≤ c_t^2M_t+s_t^2M_t (12) =(ct2+st2)Mt=Mt =(c_t^2+s_t^2)M_t=M_t (13) This completes the induction, showing that the Frobenius norm of the accumulated parameter deviation is bounded by the maximum individual task update norm: ‖Wacc(t)−W(0)‖F2≤Mt\|W_acc^(t)-W^(0)\|^2_F≤ M_t (14) This bound prevents the parameters of the consolidated branch from exploding or drifting away from the pre-trained manifold, mitigating catastrophic forgetting without sacrificing stability. ∎ Remark on Knowledge Dilution. One might be concerned that the st=1/ts_t=1/ t scaling in Eq. (4) dilutes the historical knowledge in the shared memory. We argue that this scaling acts as a variance normalization mechanism rather than information loss. Without scaling, the norm of the shared branch would grow linearly with t t (assuming orthogonal updates), eventually overwhelming the task-specific branch. The st=1/ts_t=1/ t factor ensures that the shared memory remains a stable “anchor” while the specific branch (Section 4.3) preserves the distinct decision boundaries for each task. This dual-branch design effectively decouples stability (via normalized shared memory) from plasticity (via preserved specific experts). Empirical Verification of 1/t1/ t Scaling. To empirically verify the stabilizing effect of our derived 1/t1/ t scaling factor against standard linear averaging (1/t1/t) or no scaling (11), we conduct a comparative analysis on the Standard CL Benchmark (Order-1, T5-Large). Table 5 reports the final average accuracy and the Frobenius norm of the accumulated parameters ‖Wacc(t)−W(0)‖F\|W_acc^(t)-W^(0)\|_F. Table 5: Scaling factors on the Standard CL Benchmark (Order-1, T5-Large), measured on the null-space consolidation path where this rule is defined. “Avg Accuracy” is the full model; “Shared Branch” evaluates the consolidated branch alone, which is what the scaling factor controls. Three seeds. Method ct,stc_t,\ s_t Avg Accuracy Shared Branch No Scaling 1, 11,\ 1 76.5±0.576.5_± 0.5 60.3±1.360.3_± 1.3 Linear Avg t−1t,1t t-1t,\ 1t 77.5±0.477.5_± 0.4 73.6±1.073.6_± 1.0 CoDe-LoRA −, t-1t,\ 1 t 77.8±0.477.8_± 0.4 75.6±0.675.6_± 0.6 Without scaling, the parameter norm explodes, pushing weights outside the LLM’s effective optimization manifold. Conversely, linear averaging (1/t1/t) overly dilutes the historical knowledge. Only our derived 1/t1/ t factor maintains a stable norm and achieves the peak accuracy. A.3 Implementation Details Following existing CL works Qin and Joty (2021); Wang et al. (2023a); Yang et al. (2025b), all methods are implemented with instruction tuning Qin and Joty (2021) and optimized with AdamW Loshchilov and Hutter (2017) (β=(0.9,0.999)β=(0.9,0.999), no weight decay, gradient clipping at 1.0) under a constant learning rate, in bfloat16. Every method trains each task for one epoch with an effective batch size of 64, and inputs are truncated to 512 source and 50 target tokens. The learning rate is set per backbone: 10−310^-3 for T5-large, and 10−410^-4 for Llama 2-7B and Qwen3.5-4B. On Qwen3-0.6B the per-task-expert methods (De-LoRA, CoDe-LoRA) use 10−310^-3, while the methods built on a single shared adapter use 10−410^-4. For all LoRA-based architectures we set rank r=8r=8, α=32α=32 and dropout 0.10.1, targeting the query and value projections of every Transformer block. Routing uses an LDA classifier on frozen-backbone embeddings estimated from 1000 support samples per task (Appendix A.5), the consolidation uses a randomized low-rank SVD with retained rank 88, and classification tasks are scored over their candidate labels with PMI calibration while TRACE is evaluated by generation. All experiments use NVIDIA H20 or A100 (80 GB) GPUs; the efficiency measurements in Appendix A.7 are taken on a single GPU to avoid inter-GPU communication effects. Each experiment is repeated over three independent runs with different random seeds. Learning rates. The learning rate is set per backbone because no single value works for all of them. At 10−310^-3 the methods built on a single shared adapter (LoRA, CLoRA) diverge on Llama 2-7B and Qwen3.5-4B (gradient norms of 100–340), and on Qwen3-0.6B they forget catastrophically (LoRA on the Long benchmark reaches an AP of only 10.5 with a BWT of −62-62). At 10−410^-4 the per-task experts of De-LoRA and CoDe-LoRA cannot learn the smallest tasks (CB and COPA, 4–6 optimizer steps per epoch) on Qwen3-0.6B, and De-LoRA falls from 71.2 to 59.5. Practical Implementation Details of SVD and Null Space Projection. To resolve the computational bottleneck of null space projection in large-scale LLMs, we adopt an optimized consolidation strategy. Specifically, we replace full Singular Value Decomposition (SVD) with a randomized low-rank SVD (torch.svd_lowrank) to efficiently extract singular vectors U from the column space of consolidated parameters. To further reduce complexity, the projection is performed directly on the low-rank factor B∈ℝdout×rB ^d_out× r rather than the full update ΔW∈ℝdout×din W ^d_out× d_in. The orthogonalized factor is computed as Borth=B−U(U⊤B)B_orth=B-U(U B), and the resulting update is ΔWnull=BorthA W_null=B_orthA. This optimization reduces the projection matrix dimensions from dout×dind_out× d_in to dout×rd_out× r, where r≪dinr d_in, achieving significant speedup while preserving mathematical accuracy. In practice, we set the number of randomized SVD iterations (svd_niter) to 1 to balance precision and performance. Computational Complexity and SVD Efficiency Bridge. As introduced in Section 4.2, standard SVD on the full weight matrix update ΔW∈ℝdout×din W ^d_out× d_in requires (doutdinmin(dout,din))O(d_outd_in (d_out,d_in)) operations, which is prohibitively expensive for large language models where dout,dind_out,d_in can scale to thousands or tens of thousands. In contrast, by projecting directly on the low-rank factor B∈ℝdout×rB ^d_out× r and leveraging the randomized low-rank SVD method (torch.svd_lowrank), the computational complexity is reduced to (doutr2)O(d_outr^2) where r is the LoRA rank (typically r∈[8,32]r∈[8,32]). Because r≪din,doutr d_in,d_out, this optimization achieves a speedup of several orders of magnitude. In practice, the SVD consolidation is negligible: over a whole T5-large run it takes 0.7 s for the four tasks of the Standard benchmark and 4.1 s for the fifteen tasks of the Long benchmark, while preserving the mathematical guarantee of null space projection. This provides a direct, highly efficient bridge between the theoretical null space projection in Section 4.2 and the practical scalability of our framework. Sensitivity to Approximate Orthogonality. While randomized SVD inevitably introduces approximation noise, CoDe-LoRA remains theoretically and architecturally robust to this relaxation. Theoretically, our dynamic scaling factor dampens noise accumulation, bounding the approximation error. Architecturally, the Decoupling Branch serves as a safety net; if approximation noise compromises the shared branch, routing dynamically isolates predictions to task-specific experts. To verify this, we vary the SVD retained rank on the Order-1 benchmark (T5-large, three seeds). Table 6 reports the accuracy of the consolidation branch alone and of the full model. Table 6: SVD retained rank on the Standard CL Benchmark (Order 1, T5-large). Rank 8 is the default. Retained rank Co-LoRA alone CoDe-LoRA 1 66.84 79.13 2 72.11 79.13 4 75.86 79.13 8 76.67 79.13 The consolidation branch alone improves monotonically with the retained rank, since a lower rank discards more of the accumulated update. The full model does not move because the routed expert answers every input, so the shared branch does not enter the prediction. The retained rank therefore matters for the quality of the consolidated weights, not for the routed accuracy. Step-by-Step Training and Consolidation Workflow. During the learning of each sequential task tT_t, CoDe-LoRA decouples and consolidates knowledge through a structured step-by-step procedure: 1. Initialization: The Consolidation Branch parameters θshared _shared are initialized from the previously consolidated state Wacc(t−1)W_acc^(t-1) to enable immediate knowledge reuse and positive transfer, while a new task-specific expert θspec(t) _spec^(t) is initialized to zero. 2. Joint Training: Both branches are optimized jointly on the current task dataset (t)D^(t) using instruction tuning. This joint optimization allows the Consolidation Branch to assimilate shared representation updates while the Decoupling Branch captures task-specific nuances. 3. Null Space Consolidation: Post-training, the parameter update learned in the Consolidation Branch, denoted as ΔWshared(t)=Wshared(t)−Wacc(t−1) W_shared^(t)=W_shared^(t)-W_acc^(t-1), is extracted. We project this update onto the null space of the accumulated representation history Wacc(t−1)W_acc^(t-1) using randomized low-rank SVD (as detailed above) to eliminate representational interference. 4. Dynamic Scaling & Archive: The projected update is scaled via our dynamic scaling coefficient st=1/ts_t=1/ t to prevent parameter norm explosion, and merged into the consolidated weight matrix Wacc(t)W_acc^(t) recursively using ct=(t−1)/tc_t= (t-1)/t. Concurrently, the trained task-specific expert θspec(t) _spec^(t) is archived in the expert pool, and its prototype PtP_t is computed and stored. Evaluation Protocol and Analysis Setup. For text classification tasks (e.g., Yahoo, DBpedia), candidate labels are obtained from task metadata and tokenized. The main performance results are reported using the generate() method; classification tasks are additionally scored over their candidate labels via forward(). For the ablation studies and prototype sensitivity analysis in Figure 5, we vary the number of support samples N to find the optimal balance between routing performance and inference speed. A.4 Evaluation Metrics We evaluate sequential learning performance using two standard metrics in continual learning: Average Performance (AP) represents the overall model capacity across all learned tasks after completing the continual learning sequence. For a sequence of T tasks, it is defined as: AP=1T∑i=1Tai,TAP= 1T _i=1^Ta_i,T (15) where ai,Ta_i,T denotes the evaluation performance (accuracy for classification benchmarks, or task-specific generative metrics like ROUGE/Exact Match for generative benchmarks) on task i after training on the final task T. In classification benchmarks, this corresponds to the standard Average Accuracy (ATA_T), while in generative benchmarks (such as TRACE), it is referred to as Overall Performance (OP). Backward Transfer (BWT) Lopez-Paz and Ranzato (2017) measures the influence of learning subsequent tasks on the performance of previously acquired knowledge: BWT=1T−1∑i=1T−1(ai,T−ai,i)BWT= 1T-1 _i=1^T-1(a_i,T-a_i,i) (16) where ai,ia_i,i is the performance on task i immediately after training on task i. A negative BWT value indicates catastrophic forgetting (performance degradation), whereas a value closer to 0 or positive indicates stable knowledge retention or positive backward transfer (where learning new tasks improves performance on older tasks). By employing BWT as our unified metric for backward retention across all benchmarks, we avoid the redundancy of using separate forgetting indicators for classification and generative tasks. Forward Transfer (FWT) measures the zero-shot accuracy on a task that has not been trained yet. After learning the first k tasks, FWTk→j=aj,kFWT_k→ j=a_j,k for j>kj>k, which we compare with the accuracy of the untrained backbone and of the other methods on the same task. A.5 Router Choice The main experiments route with a linear discriminant analysis (LDA) classifier on frozen-backbone sentence embeddings. For every task we store the mean embedding and one covariance matrix shared by all tasks, estimated from 1000 support samples per task, and an input goes to the task with the largest discriminant score. No raw sample is stored, so the method stays replay-free and the per-task storage is one vector, as for a prototype, plus a single shared matrix. The cosine prototype router of Section 4.3 is also implemented. Table 7: Routing errors and their cost on T5-large (three seeds). Loss is the AP of the true-task expert minus that of the routed expert. Standard (Order 1) Long (Order 4) Router Misrouted Loss Misrouted Loss Cosine prototype 2.03% 1.43 10.07% 4.74 LDA 0.15% 0.10 1.24% 0.20 The cosine router fails by confusing tasks of the same family, because it compares only the direction of the mean embedding and ignores the within-task variance. On Order 4 the largest confusions are MNLI→ (22.7% of MNLI inputs), MNLI→ (13.0%), Yahoo→ (6.9%), Yelp→ (5.1%) and Amazon→ (4.0%). LDA whitens with the shared within-task covariance and removes most of these errors. Figure 7 varies the number of support samples per task: cosine routing with N=10N=10 is still below its plateau and does not close the gap to LDA on the Long benchmark, and mean-centering the embeddings helps on the Standard benchmark but hurts on the Long benchmark. On the Long benchmark LDA already reaches 94.2% routing accuracy with N=10N=10, while the cosine router with N=100N=100 reaches 93.3%; LDA saturates beyond about 500 samples per task. Figure 7: Routing accuracy versus support samples per task N (T5-large). A.6 Co/De Synergy: Attempts and Limits We summarise what we tried to make the consolidation branch (Co) improve on the routed expert (De). All numbers are T5-large, three seeds, AP on the Standard (Order 1) and Long (Order 4) benchmarks, reported as Std / Long; rules are selected on a development subset and reported on test. The two branches are complementary, but no selection rule exploits it. Taking the better of Co and the routed expert per example (the oracle) exceeds the routed expert by 4.7 (Std) to 9.5 (Long) points (Table 9, Figure 8). The oracle is a diagnostic of how many examples Co answers correctly while the expert fails, not an attainable bound. Yet no rule chosen on development data captures it. With the LDA router the posterior is almost always above any useful threshold, so Co answers at most 0.3% of the inputs and CoDe-LoRA reduces to De-LoRA. With the cosine router the fallback does fire, but the routing itself first loses 4.7 points on the Long benchmark, more than Co recovers. The thresholds chosen on development data degenerate to “never use Co”. Figure 8: Per-task accuracy of Co, the routed expert and their oracle on the Long benchmark (Order 4). Inference-time schemes. We replayed twelve rounds of schemes on the same checkpoints, the same development subset and the same selection protocol (Table 8). Confidence-threshold fallbacks, per-expert gating, output selectors and a 1480-feature selector produced no development-qualified rule or gains below one point that did not hold across protocols. Fusing the two branches on candidate-label scores with a per-task weight was the only scheme that was positive across backbones, and it is the one used in the main tables. Without calibration, summing the two branches hurts on the Long benchmark (−1.0-1.0 to −1.6-1.6) because the label prior of Co is not removed. Table 8: Inference-time schemes for combining Co with the routed expert. Scheme family Outcome Confidence-threshold fallback best threshold on development is “never use Co” Per-expert branch choice ≈ routed expert Co as an extra expert no development-qualified candidate Output selectors (margin, entropy, agreement) below calibrated De, or kept De 1480-feature selectors none passed development; 26 rescues vs. 26 breaks on test Closed-form combinations (PoE, averaging, Dawid–Skene) do not solve the selection problem Calibrated label-score fusion (per-task weight) positive across backbones; used in the main tables Decoding-time mixing. Mixing the log-probabilities of Co and the routed expert at every decoding step, (1−w)logpDe+wlogpCo(1-w) p_De+w p_Co, does not help either. The weight chosen on development data is w=0w=0 on both benchmarks; a per-task weight gives +0.13+0.13 (Std) and +0.44+0.44 (Long), 2.8% and 5.2% of the oracle gap, at 2.0–2.25× the decoding cost. The branches agree on the top-1 token in 95% (Std) and 91% (Long) of the steps, with log-probability correlations of 0.94 and 0.92, so a mixture with w=0.3w=0.3 changes the argmax in only 1.4–1.6% of the steps, and the gains and losses cancel. The complementarity exists at the example level but not at the token level. Constructions of Co. Table 9 changes only how Co is merged. Projecting against the historical factor space, on one or both sides, or distilling after the SVD, does not improve Co. Merging experts that were trained in isolation from the base model into Co fails, because that shared block is never trained and the generation format collapses (41.87 / 11.06). The null-space merge of Section 4.2 gives a weaker Co that is the most complementary to the experts (oracle 84.53 / 86.00). Table 9: Constructions of Co (generative, test): Co alone and the oracle that takes the better of Co and the routed expert per example. The routed expert alone scores 79.13 / 76.3–76.5 (Std / Long). Construction Co alone Oracle Trained, SVD merge (main) 76.67 / 66.33 83.79 / 84.61 + null space, paper scaling 75.61 / 63.78 84.53 / 86.00 + distillation after SVD 74.77 / 66.34 83.66 / 84.51 + left-space projection 76.16 / 65.67 83.74 / 84.08 + two-sided projection 74.88 / 63.91 83.14 / 84.60 Isolated experts, SVD merge 41.87 / 11.06 81.84 / 73.99 + left projection 51.31 / 21.96 – + two-sided projection 58.03 / 46.96 – Using Co at training time. Initialising each new expert from Co, either by transferring the update or its input subspace, did not pass the development gate across both benchmarks. A family-level initialisation gives +2.30±0.31+2.30± 0.31 on the Long benchmark (positive in 18 of 18 runs over two independent seed sets), but 93% of it comes from the single task CB (+34.1+34.1); without CB and COPA it falls to +0.08+0.08, and it is no better than a global Co initialisation (+0.03+0.03). We therefore report it as a local effect. A measurement pitfall: routing compensation. With the cosine router (89.9% accurate), fusing paper-style Co with the experts improves over the same calibrated single-branch baseline by +0.49+0.49 (Std, 3/3 seeds) and +1.60+1.60 (Long, 3/3). Replacing the router by LDA (98.8% accurate) with Co unchanged reduces these to +0.13+0.13 (Std, 2/3) and +0.77+0.77 (Long, 3/3): about half of the apparent gain was the fusion repairing routing errors, not new information from Co, and 94% of the rest comes from CB and COPA (26 and 40 development examples). When evaluating a two-branch fusion, the routing quality must be fixed and a calibrated single-branch baseline must be reported. A.7 Detailed Efficiency Statistics Table 10 gives the training and inference seconds of every task on the Standard benchmark (Order 1, T5-large, one H20 GPU, seed 41). Inference is the evaluation stage after the task, and the last stage scores all four tasks. Stored adapter parameters are 9.4M (1.28%) for O-LoRA, N-LoRA and CoDe-LoRA and 32.4M (4.40%) for MoLE-CIE; peak GPU memory is 42.6, 42.6, 31.7 and 41.3 GiB for O-LoRA, N-LoRA, MoLE-CIE and CoDe-LoRA. Table 10: Training and inference seconds per task on the Standard benchmark (Order 1, T5-large, one H20 GPU, seed 41). Method Task Train (s) Infer (s) O-LoRA DBpedia 141 25 Amazon 65 27 Yahoo 186 43 AG News 44 117 N-LoRA DBpedia 128 26 Amazon 60 27 Yahoo 172 42 AG News 36 116 MoLE-CIE DBpedia 767 128 Amazon 626 110 Yahoo 1484 201 AG News 587 566 CoDe-LoRA DBpedia 124 39 Amazon 60 50 Yahoo 171 84 AG News 38 128 Long Sequence Efficiency (T=15). Table 11 reports the total training and inference minutes and the stored adapter parameters after the fifteen tasks of the Long Sequence Benchmark (Order 4, T5-large, one H20 GPU). Table 11: End-to-end efficiency on the Long Sequence Benchmark (T=15 tasks). CoDe-LoRA is a single seed, the others are means over three seeds. Method Train (min) Infer (min) Params O-LoRA 13.3 9.4 35.4M (4.80%) N-LoRA 12.0 9.4 35.4M (4.80%) MoLE-CIE 140.3 49.5 60.0M (8.14%) CoDe-LoRA 12.5 13.4 35.4M (4.80%) Because only the experts are stored, the parameter growth of CoDe-LoRA over the fifteen tasks is the same as for O-LoRA and N-LoRA. A.8 Datasets Table 12: Datasets and their tasks, categories, domains, and metrics. No. Dataset name Category Task Domain Metric 1 Yelp CL Benchmark sentiment analysis Yelp reviews accuracy 2 Amazon CL Benchmark sentiment analysis Amazon reviews accuracy 3 DBpedia CL Benchmark topic classification Wikipedia accuracy 4 Yahoo CL Benchmark topic classification Yahoo Q&A accuracy 5 AG News CL Benchmark topic classification news accuracy 6 MNLI GLUE NLI various accuracy 7 QQP GLUE paragraph detection Quora accuracy 8 RTE GLUE NLI news, Wikipedia accuracy 9 SST-2 GLUE sentiment analysis movie reviews accuracy 10 WiC SuperGLUE word sense disambiguation lexical databases accuracy 11 CB SuperGLUE NLI various accuracy 12 COPA SuperGLUE QA blogs, encyclopedia accuracy 13 BoolQA SuperGLUE boolean QA Wikipedia accuracy 14 MultiRC SuperGLUE QA various accuracy 15 IMDB SuperGLUE sentiment analysis movie reviews accuracy In our experiments, we utilize 15 datasets from standard classification and NLU benchmarks (as detailed in Table 12), as well as the generation-heavy TRACE benchmark Wang et al. (2023b). Specifically, the classification datasets are sourced from well-established suites such as the CL benchmark Zhang et al. (2015), GLUE Wang et al. (2018), and SuperGLUE Wang et al. (2019), as well as the IMDB movie review dataset Maas et al. (2011). These cover natural language inference (NLI), sentiment classification (SC), and topic classification (TC). Crucially, to evaluate generalization on more complex, generation-heavy tasks, we also incorporate the TRACE benchmark Wang et al. (2023b). Unlike standard classification tasks, TRACE features multi-turn, complex generative tasks that assess the structural reasoning and domain adaptation of LLMs. In particular, we evaluate on code generation (completion) using the Py150 dataset, where even a single syntax error can lead to a compilation failure, making it highly fragile under sequential fine-tuning. A.9 Task Sequence and Instructions To evaluate the robustness of our method, we employ a diverse set of task instructions and evaluate across multiple task orders. Table 13 summarizes the specific natural language prompts used for each task category, following the instruction-tuning format described in Qin and Joty (2021). The task sequences for the T5 and Llama 2-7B experiments, together with the TRACE benchmark sequence, are provided in Table 14. Table 13: Task descriptions and corresponding prompts. Task Prompts NLI What is the logical relationship between the “sentence 1” and the “sentence 2”? Choose one from the option. QQP Whether the “first sentence” and the “second sentence” have the same meaning? Choose one from the option. SC What is the sentiment of the following paragraph? Choose one from the option. TC What is the topic of the following paragraph? Choose one from the option. BoolQA According to the following passage, is the question true or false? Choose one from the option. MultiRC According to the following passage and question, is the candidate answer true or false? Choose one from the option. WiC Given a word and two sentences, whether the word is used with the same sense in both sentences? Choose one from the option. Table 14: Task sequences for different model and benchmark settings. Order Setting Task Sequence 1 T5, Llama 2-7B dbpedia → amazon → yahoo → ag 2 T5, Llama 2-7B dbpedia → amazon → ag → yahoo 3 T5, Llama 2-7B yahoo → amazon → ag → dbpedia 4 T5 mnli → cb → wic → copa → qqp → boolqa → rte → imdb → yelp → amazon → sst-2 → dbpedia → ag → multirc → yahoo 5 T5 multirc → boolqa → wic → mnli → cb → copa → qqp → rte → imdb → sst-2 → dbpedia → ag → yelp → amazon → yahoo 6 T5 yelp → amazon → mnli → cb → copa → qqp → rte → imdb → sst-2 → dbpedia → ag → yahoo → multirc → boolqa → wic 1 TRACE C-STANCE → FOMC → MeetingBank → Py150 → ScienceQA → NumGLUE-cm → NumGLUE-ds → 20Minuten