Paper deep dive
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning
Seunghyun Yoo, Kiseok Kim, Hyeontae Joo, Junyeop Bang, Hwangnam Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.34453v1
- Canonical: https://arxiv.org/abs/2609.34453v1
Trouble viewing inline? Open PDF directly →
Full Text
57,747 characters extracted from source content.
Expand or collapse full text
SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning Seunghyun Yoo †thanks: Equal contribution. Kiseok Kim11footnotemark: 1 Hyeontae Joo11footnotemark: 1 Junyeop Bang & Hwangnam Kim †thanks: Corresponding author. Affiliation: School of Electrical Engineering Affiliation: Korea University Affiliation: Seoul, Republic of Korea Email: seunghyunyoo,kisuk528,motern800,junyeop9981,hnkim@korea.ac.kr Abstract This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864. 1 INTRODUCTION Continual learning (CL) requires a model to sequentially learn new tasks while retaining knowledge acquired from old tasks, demanding a balance between stability and plasticity (Parisi et al., 2019). Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), enables CL by learning low-rank updates while keeping the pretrained backbone frozen (Hu et al., 2021). Existing LoRA-based methods address inter-task interference by separating the subspace for new task updates. O-LoRA uses an orthogonality regularization to separate current and previous LoRA subspaces (Wang et al., 2023), whereas InfLoRA determines the LoRA input factor from the interference-free subspace orthogonal to previous tasks before learning each task and keeps the projection fixed during optimization (Liang & Li, 2024). More recent methods further refine this approach. SplitLoRA determines the minor subspace dimension for each module through a first-order bound on aggregated old task gradients, balancing the old gradient energy falling into the subspace against the dimensionality available for new task learning (Qiu et al., 2026). JANUS-LoRA uses gradient rectification to decouple the LoRA factor updates, and reconstructs them through regularized least squares so that their composite weight update approximately follows the intended orthogonal direction (Chen et al., 2026). However, freezing previously learned parameters does not guarantee that old task behavior is preserved. Even when these parameters remain unchanged, a newly learned LoRA branch can induce non-zero residual outputs on old task activations, which propagate to downstream representations and predictions. Existing subspace-based methods reduce this interference by restricting new task updates relative to subspaces derived from old tasks. However, in these methods, the new branch either still responds to old task activations or stops responding only because one LoRA factor is frozen. Moreover, increasing protection reduces the subspace available for new task adaptation, creating a stability-plasticity trade-off in determining the protected subspace size. Since historical sensitivity from the accumulated tasks is distributed differently across activation directions in different modules, a fixed protection size may protect some modules not enough while unnecessarily constraining others. Moreover, sizing the protected subspace from aggregate old task gradients does not directly reflect this sensitivity associated with protected directions. To address these limitations, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA), which directly constrains the input factor of each new task LoRA branch with respect to selected old task activation directions and independently selects the protection size of each module. For each module, SPACE-LoRA selects the minimum number of directions needed to reach the target coverage of past-task sensitivity. Rather than using directions to determine the adaptation space for the new task, we directly constrain the LoRA input factor to the orthogonal complement of the protected subspace by projecting it after every optimizer step, while keeping both LoRA factors trainable. As a result, the response of the newly added LoRA branch is identically zero for any activation within the protected subspace, regardless of the output factor. Meanwhile, the remaining orthogonal complement remains available for new task adaptation. Because increasing the size of the protected subspace necessarily reduces the space available for new task adaptation, the protection capacity is chosen per module. Old task activations determine the candidate protection directions and their ordering within each module. Fisher information measures how much historical task sensitivity is captured as these directions are included. This coverage-based decision of SPACE-LoRA allows the protection dimensionality to vary according to the sensitivity distribution of each module. We evaluate SPACE-LoRA with a single coverage target across vision and language benchmarks under two training protocols, a wider range of setting than the methods we compare with. Mostly, SPACE-LoRA ranks first or close to first on every benchmark, whereas the strongest baseline changes from benchmark to benchmark. Our main contributions are summarized as follows: • We formulate continual low-rank adaptation as a residual-response control problem and introduce a hard activation-subspace constraint that directly projects the input factor of each new LoRA branch. This constraint guarantees zero residual response on the selected historical activation subspace while leaving the complementary input space available for new-task adaptation. • We introduce a Fisher-coverage scheme that adapts the protection size to each module’s sensitivity to previous tasks. By allocating stronger protection to modules that require greater preservation while avoiding unnecessary protection in less sensitive modules, SPACE-LoRA balances knowledge retention and adaptation capacity across the network. • We evaluate SPACE-LoRA on both vision and language models across a diverse set of continual learning benchmarks and against a broad range of competitive baselines. The results show that SPACE-LoRA consistently alleviates catastrophic forgetting while preserving adaptation to incoming tasks, thereby improving the trade-off between stability and plasticity across different model architectures and task domains. 2 RELATED WORK Continual learning (CL) studies how models learn from sequential tasks while retaining previously acquired knowledge from earlier tasks, and the main difficulty is catastrophic forgetting (De Lange et al., 2021). CL is categorized as task-incremental learning, where the task identity is given, class-incremental learning, where the model chooses among all classes seen, and domain-incremental learning, where the input distribution changes while the label space stays fixed (Van de Ven et al., 2022). Existing CL methods are commonly categorized into regularization-based, memory-based, and expansion-based approaches (Wang et al., 2024). Parameter-efficient fine-tuning (PEFT), which freezes a pretrained model and trains only a small number of additional parameters, can reduce the computational cost of adapting to new downstream tasks while maintaining competitive performance. Representative PEFT approaches include adapter-based tuning, which augments a frozen backbone with smaller and trainable modules (Houlsby et al., 2019), and prompt-based tuning, which optimizes learnable prompts while leaving the pretrained parameters fixed (Lester et al., 2021; Wang et al., 2022). Low-Rank Adaptation (LoRA) keeps the pretrained model frozen and parametrizes updates to selected weight matrices with two low-rank factors (Hu et al., 2021). It has been increasingly adopted in CL methods that mitigate interference among sequential task updates. O-LoRA reduces task interference by placing low-rank updates for different tasks in orthogonal parameter subspaces (Wang et al., 2023). InfLoRA constructs the input factor from a subspace designed to avoid interference with old tasks and keeps the factor fixed while learning the new task (Liang & Li, 2024). PLAN pre-assigns orthogonal basis vectors to each task and selects those least sensitive to interference with previously learned parameters (Wang et al., 2025). LB-CL (Qiao & Mahdavi, 2024) uses the sensitivity of low-rank parameters to identify knowledge-specific components and incorporates them into gradient projection. SplitLoRA partitions the old task gradient space into principal and minor subspaces and adjusts the partition size across modules to balance stability and plasticity (Qiu et al., 2026). JANUS-LoRA targets zero change in the layer output on previous activations (Chen et al., 2026). It projects the full-weight update onto the complement of a historical activation subspace and, because independent factor updates break this projection. reconstruct the factor updates by regularized least squares so that their composite update approximately follows the projected direction. These LoRA-based approaches are related to projection-based CL methods that restrict new updates with respect to subspaces associated with old task knowledge. GPM constructs important subspaces from old task representations, determining the size based on a layer-wise energy threshold, and projects new gradients onto their orthogonal complements (Saha et al., 2021). Energy-based rank selection has also shown its effectiveness in LoRA-based CL (Li et al., 2026). Adam-NSCL estimates a null space from old task input-feature covariance and constrains subsequent optimization to that space by projecting the optimizer update (Wang et al., 2021). Regularization-based methods such as EWC (Kirkpatrick et al., 2017) quantify parameter importance for old tasks using the Fisher information and restrict changes to parameters that are important for retaining old task knowledge. A separate line of work, such as AdaLoRA and OA-Adapter, allocate adaptation capacity selectively across components to improve parameter efficiency (Zhang et al., 2023; Wan et al., 2025). In subspace-based methods such as GPM and SplitLoRA, the protection size is determined by representation energy or averaged old task gradients, neither of which measures the per-example loss sensitivity along the activation directions that are actually protected. SPACE-LoRA treats selected old-task activation directions as a protected subspace and projects the input factor onto its orthogonal complement after each optimizer step, while keeping both LoRA factors trainable. As a result, the newly added LoRA branch produces zero response to activations within the protected subspace. In each module, historical activation directions are ordered by decreasing activation eigenvalue, and the protection size is selected as the minimum number of directions required to reach the target Fisher coverage. 3 PRELIMINARIES 3.1 LoRA-based Continual Learning Consider a task stream =t1,…,tTT=\t_1,…,t_T\. The objective is to learn new tasks while retaining performance on previously learned tasks (Zhou et al., 2024). We consider a setting in which task-specific LoRA adapters are accumulated (Wang et al., 2023). After learning task t, the effective weight of module m combines its pre-trained weight Wm0∈ℝdout,m×din,mW_m^0 ^d_out,m× d_in,m with the accumulated adapters: Wm(t)=Wm0+η∑τ=1tBm,τAm,τ,η=αr,W_m^(t)=W_m^0+η _τ=1^tB_m,τA_m,τ, η= αr, (1) where r denotes the LoRA rank, α is the scaling hyperparameter (Hu et al., 2021), and Am,t∈ℝr×din,m,Bm,t∈ℝdout,m×r.A_m,t ^r× d_in,m, B_m,t ^d_out,m× r. (2) During training on task t, only Am,tA_m,t and Bm,tB_m,t within module m are updated, while Wm0W_m^0 and all adapters from previous tasks remain frozen. 3.2 Adapter Interference and Catastrophic Forgetting Even with previously learned adapters frozen, a new adapter can change the output of a module. For a fixed input activation x∈ℝdin,mx ^d_in,m from a previous task, the output change induced by the new adapter at module m is Δhm(t)(x)=ηBm,tAm,tx. h_m^(t)(x)=η B_m,tA_m,tx. (3) This expression isolates the new adapter’s contribution at a fixed module input. Changes in preceding modules may additionally alter the input activation itself. Because each new adapter is optimized for the current task, its contribution may disrupt predictions on previous tasks. Such changes can lead to catastrophic forgetting. Let aj,ja_j,j denote the accuracy of task j immediately after learning it, and let at,ja_t,j denote its accuracy after learning through task t(t>j)t\;(t>j). Task-wise forgetting is defined as Fj(t)=aj,j−at,j.F_j^(t)=a_j,j-a_t,j. (4) Thus, Fj(t)>0F_j^(t)>0 indicates a decline in accuracy on task j following subsequent adaptation. Figure 1: Overview of SPACE-LoRA. 4 SPACE-LoRA We propose SPACE-LoRA as shown in Figure 1. SPACE-LoRA directly suppresses the response of a newly added residual adapter along selected past activation directions, while determining the protection size of each module using a common Fisher sensitivity coverage target. Under a fixed LoRA rank, this design suppresses interference along input directions that are sensitive to previous tasks while leaving the complementary input subspace available for learning the new task, thereby aiming to achieve a better trade-off between knowledge retention and adaptation. In the subsequent sections, we describe how to construct the protected activation subspace and how to determine the required protection size for each module. 4.1 Residual Response Nullification As discussed in Section 3, even when all previously learned adapters are frozen, a newly added residual branch can alter past predictions if it produces a non-zero response to past-task inputs. Therefore, continual adaptation should protect not only previously learned parameters but also the responses of the new adapter to input directions associated with old tasks. Recent LoRA-based continual learning methods address this issue by constraining new-task adaptation with respect to subspaces associated with previously learned tasks. InfLoRA constructs a low-interference adaptation subspace by removing current activation components aligned with historical gradient subspaces, whereas Janus-LoRA constrains the effective weight update to produce no change on historical activations. Motivated by these approaches, we instead directly constrain the response of the newly introduced residual branch by making its input factor insensitive to selected historical activation directions. At the endpoint of each task τ, we collect the input activations of each linear module m over the task training set DτD_τ using the model obtained after completing the task. Before learning task t, these activations are summarized by the Historical Gram matrix Hm<t=∑τ<t∑x∈Dτ∑qxm,qxm,q⊤,H_m^<t= _τ<t _x∈ D_τ _qx_m,qx_m,q , (5) where xm,qx_m,q denotes the activation vector at the q-th input position of module m. We decompose Hm<t=UmΛmUm⊤,H_m^<t=U_m _mU_m , (6) where the eigenvectors in Um=[um,1,…,um,dm]U_m=[u_m,1,…,u_m,d_m] are ordered by decreasing eigenvalue. These eigenvectors identify the principal input directions occupied by past activations. Protecting the full basis would leave no input direction available to the new adapter. We therefore protect only the first kmk_m activation directions and define Vm=[um,1,…,um,km],Pm=VmVm⊤.V_m=[u_m,1,…,u_m,k_m], P_m=V_mV_m . (7) Let AmA_m denote the input factor of the new LoRA branch. After each optimizer update, we apply the hard projection Am←Am(I−Pm),A_m← A_m(I-P_m), (8) which maintains AmVm=0.A_mV_m=0. (9) Hence, for any x∈span(Vm)x (V_m) with x=Vmcx=V_mc, ΔWmx=BmAmVmc=0. W_mx=B_mA_mV_mc=0. (10) Thus, the newly added branch produces no residual response along the protected historical directions, while the orthogonal complement remains available for learning the new task. This is a hard parameter constraint rather than a soft orthogonality regularizer. A detailed characterization of its exact nullification property and its behavior outside the protected subspace is provided in Appendix A. 4.2 Module-wise Fisher-Coverage Protection The number of protected activation directions need not be identical across modules, since their importance to previously learned tasks can differ substantially. Using a fixed protection size may therefore either leave sensitive directions insufficiently protected or unnecessarily reduce the input space available for new-task adaptation. We instead determine kmk_m independently for each module as the minimum number of activation-ordered directions required to capture a prescribed fraction of its historical loss sensitivity. To quantify this sensitivity, after completing task τ, we compute the gradient of each training example with respect to the effective weight of module m, Gm,i(τ)=∇Wmℓi(τ)∈ℝdout,m×din,m,G_m,i^(τ)= _W_m _i^(τ) ^d_out,m× d_in,m, (11) and define the input-space empirical Fisher Fm(τ)=1Nτ∑i=1Nτ(Gm,i(τ))⊤Gm,i(τ).F_m^(τ)= 1N_τ _i=1^N_τ (G_m,i^(τ) ) G_m,i^(τ). (12) This statistic is computed at the endpoint of task τ using the current task classifier and the observed ground-truth labels. The historical Fisher available before task t is Fm,past(t)=∑τ<tNτFm(τ)∑τ<tNτ.F_m,past^(t)= _τ<tN_τF_m^(τ) _τ<tN_τ. (13) Further details on the construction and accumulated historical Fisher are provided in Appendix B. Let Um=[um,1,…,um,dm]U_m=[u_m,1,…,u_m,d_m] be the full activation basis from the previous subsection, ordered by decreasing activation eigenvalue, where dm=din,md_m=d_in,m. We measure the historical sensitivity of each activation direction as pm,j(t)=um,j⊤Fm,past(t)um,j.p_m,j^(t)=u_m,j F_m,past^(t)u_m,j. (14) The activation statistics therefore determine the directions and their order, while the Fisher evaluates the loss sensitivity associated with those directions. For a prefix of length k, we define its Fisher coverage as Cm(k)=∑j=1kpm,j(t)tr(Fm,past(t)).C_m(k)= _j=1^kp_m,j^(t)tr (F_m,past^(t) ). (15) Since UmU_m is a complete orthonormal basis, tr(Fm,past(t))=∑j=1dmpm,j(t)tr(F_m,past^(t))= _j=1^d_mp_m,j^(t), so Cm(k)C_m(k) represents the fraction of total historical sensitivity captured by the first k activation directions. Given a common coverage target ρ, we select km=mink∈0,…,dm−r:Cm(k)≥ρ.k_m= \k∈\0,…,d_m-r\:C_m(k)≥ρ \. (16) The upper bound dm−rd_m-r leaves at least r input dimensions available to the rank-r adapter. If the total sensitivity is zero, we set km=0k_m=0. If no admissible prefix reaches the coverage target, we set km=dm−rk_m=d_m-r, corresponding to the maximum allowable protection size. The resulting first kmk_m activation directions are protected using the hard response constraint in Section 4.1, and the protected subspace remains fixed throughout the current task. 4.3 Update Procedure Algorithm 1 summarizes the overall training procedure of SPACE-LoRA. Before learning a new task, each module constructs its protected activation basis from the historical activation statistic and determines the protection size using the Fisher-coverage rule in Section 4.2. The resulting protected subspace is then fixed throughout the current task. A new LoRA branch is trained while its input factor is projected onto the orthogonal complement of the protected subspace after each optimizer update, as described in Section 4.1. After completing the task, the endpoint activation and Fisher statistics are computed and incorporated into the historical statistics used for subsequent tasks. The first task is trained without protection because no historical statistics are available. Algorithm 2 in the appendix provides the complete procedure, including the construction and incremental update of the historical statistics. Algorithm 1 Overview of SPACE-LoRA 1: Task datasets Dtt=1T\D_t\_t=1^T, LoRA rank r, coverage target ρ 2: for t=1,…,Tt=1,…,T do 3: Freeze the base model and all previously learned adapters 4: Add a new rank-r LoRA branch for task t 5: for each protected module m do 6: if t=1t=1 then 7: Set km←0k_m← 0 and Pm←0P_m← 0 8: else 9: Obtain the activation-ordered basis from Hm<tH_m^<t 10: Determine kmk_m using the Fisher-coverage target ρ 11: Construct Vm=[um,1,…,um,km]V_m=[u_m,1,…,u_m,k_m] and Pm=VmVm⊤P_m=V_mV_m 12: end if 13: end for 14: for each optimization step on DtD_t do 15: Update the new LoRA branch using the current-task loss 16: Project Am,t←Am,t(I−Pm)A_m,t← A_m,t(I-P_m) for each protected module m 17: end for 18: Update the historical activation and Fisher statistics using DtD_t 19: end for 5 EMPIRICAL STUDY 5.1 Experimental Setup Datasets and Baselines. We evaluate SPACE-LoRA on vision and language continual learning benchmarks. Our vision benchmarks comprise ImageNet-R, DomainNet, VTAB, and VDD. For language continual learning, we use a suite of 15 NLP classification datasets following the benchmark used by Progressive Prompts and O-LoRA (Razdaibiedina et al., 2023; Wang et al., 2023). We conduct systematic comparisons with representative LoRA-based continual learning methods: InfLoRA, SplitLoRA, and Janus-LoRA for vision, and these methods together with O-LoRA for language. The comparisons use a common experimental protocol within each benchmark. For methods originally developed for vision, their language experiments use T5 adaptations based on the official implementations. The adaptation details are provided in the appendix. Experimental Design. We evaluate ImageNet-R, DomainNet, and VTAB under class-incremental learning (CIL), where prediction is performed over all classes observed so far without access to the task identity at inference. For ImageNet-R, we follow the task-count configurations used by Qiu et al. (2026) and partition its 200 classes into 5, 10, and 20 tasks, with 40, 20, and 10 classes per task, respectively. For DomainNet, we use five domains excluding Quickdraw and partition its 345 classes into five tasks of 69 classes each. For VTAB, we adopt the 5-task cross-domain construction introduced by APER, with 10 classes per task. For VDD, we use a task-incremental learning (TIL) setting: ImageNet is excluded, and each of the remaining nine datasets constitutes a separate task, with task identity available at inference. For language continual learning, each dataset constitutes one task. We consider sequences of 5, 10, and 15 tasks, constructed from the corresponding prefixes of a fixed 15-dataset order. Shared tasks use the same data examples across these configurations. Evaluation uses the candidate label set of the dataset being evaluated, rather than a joint label space spanning all datasets. We use pretrained ViT-B/16 and T5-large backbones for the vision and language experiments, respectively. Detailed dataset statistics, data splits, task sequences, and implementation settings are provided in Appendix C. Evaluation Metrics and Implementation Details. We evaluate performance using average accuracy, ACCt=1t∑j=1tat,jACC_t= 1t _j=1^ta_t,j. Class-incremental evaluation considers all seen classes without task identity, whereas task-incremental evaluation uses task identity to select the corresponding head. For vision experiments, we use ViT-B/16 pretrained on ImageNet-21K, with rank-1010 LoRA adapters applied to the key and value projections in all 12 transformer blocks. All vision methods follow the same task splits and training protocol within each benchmark. We train for 50 epochs per task on ImageNet-R and VTAB, and 5 epochs on DomainNet. For VDD, we train for 1,500 optimization steps per task. For comparisons with the baselines, we report SPACE-LoRA with Fisher coverage thresholds of ρ=0.90ρ=0.90 and 0.950.95. For language experiments, we use T5-large as the primary backbone and follow the training settings of O-LoRA, including the number of training epochs, batch size, and learning rate. Table 1: Class-incremental learning results on ImageNet-R. ImageNet-R 5 Tasks 10 Tasks 20 Tasks Method Acc. ↑ F ↓ Acc. ↑ F ↓ Acc. ↑ F ↓ Joint-LoRA 82.68 - 82.68 - 82.68 - InfLoRA 78.89±0.5578.89± 0.55 6.02 75.90±0.2275.90± 0.22 6.28 72.23±0.5872.23± 0.58 6.82 SplitLoRA 74.46±0.2274.46± 0.22 10.80 70.25±0.3270.25± 0.32 7.98 59.84±0.2559.84± 0.25 4.81 Janus-LoRA 79.47±0.4779.47± 0.47 6.41 76.07±0.4576.07± 0.45 7.70 72.15±0.3672.15± 0.36 8.22 SPACE-LoRA (ρ=0.90ρ=0.90) 79.09±0.3179.09± 0.31 4.82 74.64±0.4474.64± 0.44 6.42 69.65±0.5069.65± 0.50 6.80 SPACE-LoRA (ρ=0.95ρ=0.95) 78.98±0.3378.98± 0.33 4.90 74.33±0.5174.33± 0.51 5.98 69.33±0.3169.33± 0.31 5.34 5.2 Main Results Results on ImageNet-R. We first evaluate SPACE-LoRA on ImageNet-R, where successive tasks introduce disjoint subsets of its classes. Table 1 reports the results for three task partitions of ImageNet-R. On ImageNet-R, both SPACE-LoRA configurations achieve slightly higher mean final accuracy and lower forgetting than InfLoRA in the 5-task setting, with ρ=0.90ρ=0.90 yielding the lowest forgetting among continual learning methods. In the 10-task setting, ρ=0.95ρ=0.95 achieves the lowest forgetting, whereas ρ=0.90ρ=0.90 exhibits slightly more forgetting than InfLoRA. However, both configurations achieve lower final accuracy than InfLoRA and Janus-LoRA in the 10- and 20-task settings. These results highlight the importance of balancing old task protection with new task adaptation. Table 2: Continual learning performance on DomainNet, VTAB, and VDD. DomainNet and VTAB use class-incremental evaluation, whereas VDD uses task-incremental evaluation. DomainNet: 5 Tasks VTAB: 5 Tasks VDD: 9 Tasks Method Acc. ↑ F ↓ Acc. ↑ F ↓ Acc. ↑ F ↓ InfLoRA 72.44 7.81 85.52 6.74 80.88 1.86 SplitLoRA 67.51 9.81 73.35 19.67 64.78 3.73 Janus-LoRA 73.41 7.82 80.47 10.59 78.53 4.16 SPACE-LoRA (ρ=0.90ρ=0.90) 73.26 6.06 86.05 5.94 86.26 3.32 SPACE-LoRA (ρ=0.95ρ=0.95) 73.19 5.54 85.20 6.51 80.41 2.66 Results on DomainNet, VTAB, and VDD. We consider benchmarks encompassing multiple visual domains or distinct classification tasks. Table 2 shows that SPACE-LoRA achieves the lowest forgetting on DomainNet with ρ=0.95ρ=0.95, while achieving slightly lower final accuracy than Janus-LoRA. On VTAB, SPACE-LoRA with ρ=0.90ρ=0.90 achieves the highest accuracy of 86.05% and the lowest forgetting of 5.94 percentage points. On VDD, SPACE-LoRA reaches 86.26% accuracy with ρ=0.90ρ=0.90, outperforming other baselines. Both coverage settings outperform Janus-LoRA and SplitLoRA in accuracy on VTAB and VDD. These results show that SPACE-LoRA achieves competitive performance across benchmarks involving multiple visual domains or heterogeneous task sequences. Table 3: Continual learning performance on the 5-, 10-, and 15-task language benchmarks. 5 Tasks 10 Tasks 15 Tasks Method Acc. ↑ F ↓ Acc. ↑ F ↓ Acc. ↑ F ↓ O-LoRA 48.76 27.70 54.56 22.21 37.52 32.48 InfLoRA 71.68 -0.90 70.83 0.48 67.54 3.91 SplitLoRA 61.60 0.40 58.72 0.54 54.85 0.50 Janus-LoRA 70.96 -0.75 66.97 2.96 63.08 2.26 SPACE-LoRA (ρ=0.90ρ=0.90) 71.88 -0.10 73.95 0.98 69.79 2.53 Figure 2: Gap to the best method on each benchmark (p). Darker indicate a smaller gap. Results on Language Continual Learning. Table 3 reports results on the 5-, 10-, and 15-task language benchmarks. SPACE-LoRA achieves the highest final accuracy in the 5- and 10-task settings, reaching 71.88% and 73.95%, respectively, with no forgetting on average in the 5-task setting and 0.98 points of forgetting in the 10-task setting. In the 15-task setting, SPACE-LoRA achieves 69.79% accuracy, exceeding InfLoRA by 2.25 points, with lower forgetting (2.53 versus 3.91 points). Although SPACE-LoRA does not achieve the lowest forgetting in any setting, it maintains competitive final accuracy across all three task sequences. These results complement the heterogeneous-task evaluation on VTAB and VDD with continual adaptation across distinct text classification tasks. Figure 2 shows that SPACE-LoRA stays close to the best method on every benchmark, whereas the strongest baseline changes from benchmark to benchmark. 5.3 Effect of Protection Coverage The Fisher coverage threshold ρ controls how much historical information is covered by the protected activation subspace. We examine ρ∈0.75,0.80,0.90,0.95,0.99ρ∈\0.75,0.80,0.90,0.95,0.99\ on the 5-task ImageNet-R benchmark while keeping the remaining training configuration fixed. In addition to final accuracy and forgetting, we report the average protected dimension k¯ k. Here, k¯ k averages the protected dimensions across modules and incremental tasks, excluding the first task. Table 4: Effect of Fisher coverage on 5-task ImageNet-R. ρ Acc. ↑ F ↓ k¯ k 0.75 78.12±0.2678.12± 0.26 7.41 90 0.80 78.35±0.3778.35± 0.37 7.34 113 0.90 79.09±0.3179.09± 0.31 4.82 211 0.95 78.98±0.3378.98± 0.33 4.90 322 0.99 78.30±0.3478.30± 0.34 3.50 517 Table 4 illustrates the trade-off between retaining previous knowledge and preserving capacity for subsequent adaptation. As ρ increases from 0.75 to 0.99, the average protected dimension grows from 90 to 517, while forgetting decreases overall from 7.41 to 3.50 percentage points. Final accuracy does not improve monotonically: it increases to 79.09% at ρ=0.90ρ=0.90 and then decreases to 78.30% at ρ=0.99ρ=0.99. Thus, stronger protection can improve retention without producing a corresponding improvement in final predictive performance. 5.4 Ablation on Protection Allocation We next examine the role of Fisher information in determining module-wise protection. We compare Fisher-guided protection with uniform allocation on 5-task ImageNet-R using ViT-B/16 and on the 10-task language benchmark using T5-large. The Fisher-guided configuration uses ρ=0.90ρ=0.90. This ablation complements the coverage sweep by examining how the protection rule affects final accuracy and forgetting. Table 5: Protection rule ablation on 5-task ImageNet-R and 10-task language continual learning with T5-large, with ρ=0.90ρ=0.90 for Fisher-based protection. ImageNet-R: 5 Tasks Language: 10 Tasks (T5-large) Protection rule Acc. ↑ F ↓ Acc. ↑ F ↓ Fisher (SPACE-LoRA) 79.09±0.3179.09± 0.31 4.82 73.95 0.98 Uniform 76.97 4.40 72.61 -0.12 Table 5 shows that Fisher-guided protection achieves higher final accuracy than uniform allocation in both settings. On ImageNet-R, it achieves 79.09% accuracy compared with 76.97% for uniform allocation, an improvement of 2.12 percentage points, while exhibiting slightly higher forgetting (4.82 versus 4.40 points). On the language benchmark, Fisher-guided protection achieves 73.95% accuracy compared with 72.61% for uniform allocation, an improvement of 1.34 percentage points. These results support the effectiveness of Fisher-guided protection in improving final predictive performance over uniform allocation, while showing that these gains do not arise from lower forgetting. 6 Conclusion In this work, we introduced SPACE-LoRA, a continual adaptation method that addresses catastrophic forgetting by suppressing the residual response of newly introduced LoRA branches. SPACE-LoRA projects the input factor of each new branch onto the orthogonal complement of selected historical activation directions, which guarantees zero residual response on the protected subspace while leaving the remaining directions for new task adaptation. A module-wise Fisher-coverage criterion further sets the protection size from the historical loss sensitivity along activation-ordered directions. Experiments on vision and language benchmarks show that SPACE-LoRA reduces forgetting while remaining competitive, and the coverage analysis shows the trade-off between protecting previous tasks and learning new ones. Although Fisher-coverage sizing does not consistently outperform uniform sizing, protecting important historical activation directions is effective for continual low-rank adaptation. AI Use Statement We used generative AI tools to assist with drafting and polishing the manuscript. All AI-assisted text was reviewed and revised by the authors, who take full responsibility for the final content. References Chen et al. (2026) Cheng Chen, Pengpeng Zeng, Yuyu Guo, Lianli Gao, Hengtao Shen, and Jingkuan Song. Janus-lora: A balanced low-rank adaptation for continual learning. arXiv preprint arXiv:2605.28495, 2026. De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021. Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. PMLR, 2019. Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 3045–3059, 2021. Li et al. (2026) Longhua Li, Lei Qi, Qi Tian, and Xin Geng. Energy-structured low-rank adaptation for continual learning. arXiv preprint arXiv:2605.27482, 2026. Liang & Li (2024) Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23638–23647. IEEE, 2024. Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019. Qiao & Mahdavi (2024) Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. Advances in Neural Information Processing Systems, 37:97476–97498, 2024. Qiu et al. (2026) Haomiao Qiu, Miao Zhang, Ziyue Qiao, Weili Guan, Min Zhang, and Liqiang Nie. Splitlora: Balancing stability and plasticity in continual learning through gradient space splitting. In International Conference on Learning Representations, volume 2026, p. 48458–48480, 2026. Razdaibiedina et al. (2023) Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314, 2023. Saha et al. (2021) Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762, 2021. Van de Ven et al. (2022) Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197, 2022. Wan et al. (2025) Zhiyi Wan, Wanrou Du, Liang Li, Miao Pan, and Xiaoqi Qin. Adaptive budget allocation for orthogonal-subspace adapter tuning in llms continual learning. arXiv preprint arXiv:2505.22358, 2025. Wang et al. (2024) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362–5383, 2024. Wang et al. (2021) Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 184–193. IEEE, 2021. Wang et al. (2023) Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 10658–10671, 2023. Wang et al. (2025) Xiequn Wang, Zhan Zhuang, and Yu Zhang. Plan: Proactive low-rank allocation for continual learning. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 2909–2918. IEEE, 2025. Wang et al. (2022) Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), p. 139–149. IEEE, 2022. Zhang et al. (2023) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023. Zhou et al. (2024) Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386, 2024. Appendix A Formal Properties of the Hard Protection Constraint This section provides a formal characterization of the hard input-space constraint used in SPACE-LoRA. We consider a linear module m whose newly introduced LoRA branch at task t is given by Δhm(t)(x)=ηBm,tAm,tx,η=αr, h_m^(t)(x)=η B_m,tA_m,tx, η= αr, (17) where Am,t∈ℝr×dm,Bm,t∈ℝom×r.A_m,t ^r× d_m, B_m,t ^o_m× r. (18) Let Vm=[vm,1,…,vm,km]∈ℝdm×km,Vm⊤Vm=I,V_m=[v_m,1,…,v_m,k_m] ^d_m× k_m, V_m V_m=I, (19) denote the protected basis selected for module m, and let Pm=VmVm⊤P_m=V_mV_m (20) be the corresponding orthogonal projector. After each optimizer update, the input factor is projected onto the orthogonal complement of the protected subspace as Am,t←Am,t(I−Pm).A_m,t← A_m,t(I-P_m). (21) A.1 Exact Nullification on the Protected Subspace The projection in Eq. equation 21 directly removes every component of Am,tA_m,t that acts on the protected directions. Since Pm=VmVm⊤P_m=V_mV_m and Vm⊤Vm=IV_m V_m=I, Am,t(I−Pm)Vm A_m,t(I-P_m)V_m =Am,t(Vm−VmVm⊤Vm) =A_m,t (V_m-V_mV_m V_m ) (22) =Am,t(Vm−Vm)=0. =A_m,t(V_m-V_m)=0. (23) Therefore, the projected input factor satisfies Am,tVm=0.A_m,tV_m=0. (24) This property directly determines the response of the new residual branch to inputs contained in the protected subspace. For any x∈span(Vm),x (V_m), (25) there exists c∈ℝkmc ^k_m such that x=Vmcx=V_mc. The corresponding residual response is then Δhm(t)(x) h_m^(t)(x) =ηBm,tAm,tx =η B_m,tA_m,tx (26) =ηBm,tAm,tVmc =η B_m,tA_m,tV_mc (27) =0. =0. (28) Thus, the new LoRA branch produces exactly zero response for inputs lying in the selected protected subspace, irrespective of the value of Bm,tB_m,t. This property follows directly from the parameter projection rather than from a soft regularization objective. It does not require a penalty term, a first-order approximation of the loss, or an assumption that the optimization gradient is approximately orthogonal to the protected directions. As long as Eq. equation 24 is maintained, the new branch cannot introduce an additional response to an input that lies entirely in span(Vm)span(V_m). The scope of this result is local to the corresponding module and its input. It guarantees that the newly added branch produces no additional response when the module receives an input within the protected subspace. It does not require that every old task activation lie exactly in that subspace, nor does it imply that downstream activations remain unchanged after adaptation at earlier modules. Therefore, the module-level property should not by itself be interpreted as a guarantee of zero forgetting at the network level. A.2 Interference Outside the Protected Subspace For an arbitrary input x∈ℝdmx ^d_m, the protected and unprotected components can be separated as x=Pmx+(I−Pm)x.x=P_mx+(I-P_m)x. (29) Because Eq. equation 24 also gives Am,tPm=Am,tVmVm⊤=0,A_m,tP_m=A_m,tV_mV_m =0, (30) the new adapter responds only to the component outside the protected subspace, Am,tx=Am,t(I−Pm)x.A_m,tx=A_m,t(I-P_m)x. (31) Consequently, ‖Δhm(t)(x)‖2 \| h_m^(t)(x) \|_2 =|η|‖Bm,tAm,t(I−Pm)x‖2 =|η| \|B_m,tA_m,t(I-P_m)x \|_2 (32) ≤|η|‖Bm,t‖2‖Am,t‖2‖(I−Pm)x‖2, ≤|η| \|B_m,t \|_2 \|A_m,t \|_2 \|(I-P_m)x \|_2, (33) where the inequality follows from submultiplicativity of the spectral norm. For fixed adapter norms, the magnitude of the module-output perturbation is therefore controlled by how much of the input remains outside the protected subspace. Inputs contained entirely in the protected subspace give zero residual response, while inputs close to that subspace can affect the new branch only through their remaining orthogonal component. The size of this residual component can be related to the activation statistics used to construct VmV_m. Let the accumulated historical activation Gram matrix be Hm=∑ixixi⊤=UmΛmUm⊤,H_m= _ix_ix_i =U_m _mU_m , (34) with eigenvalues ordered as λm,1≥λm,2≥⋯≥λm,dm≥0. _m,1≥ _m,2≥·s≥ _m,d_m≥ 0. (35) When Vm=[um,1,…,um,km],V_m=[u_m,1,…,u_m,k_m], (36) the total historical activation energy remaining outside the protected subspace is ∑i‖(I−Pm)xi‖22 _i \|(I-P_m)x_i \|_2^2 =tr[(I−Pm)Hm] =tr [(I-P_m)H_m ] (37) =∑j=km+1dmλm,j. = _j=k_m+1^d_m _m,j. (38) Therefore, for a fixed protection dimension kmk_m, selecting the leading eigenvectors of the historical activation Gram matrix minimizes the total squared residual energy of past activations outside the protected subspace. The activation statistics thus determine the ordering of directions to be protected, while the Fisher-based allocation described in the main text determines how many of these directions are retained for each module. Equation 33 characterizes the residual response at the module level rather than task-level forgetting. Its magnitude also depends on the learned norms of Am,tA_m,t and Bm,tB_m,t, which may vary across tasks and protection configurations. In addition, an output perturbation introduced at an earlier module can alter the input received by subsequent modules. For this reason, a small residual activation energy outside the protected subspace does not directly translate into an equivalent bound on the final network output. Appendix B Construction of Historical Statistics SPACE-LoRA uses two historical statistics to construct and evaluate the protected input directions: an activation Gram matrix and an input-space empirical Fisher matrix. Both statistics are collected at the endpoint of each task and accumulated across previously observed tasks. During later tasks, the protection subspace and its allocation are determined from these cached statistics without revisiting the raw training examples of previous tasks. B.1 Historical Activation Statistics For each module m, we summarize the input activations observed at the endpoint of task τ using the uncentered Gram matrix Hm(τ)=∑i∈Dτ∑qxm,i,qxm,i,q⊤,H_m^(τ)= _i∈ D_τ _qx_m,i,qx_m,i,q , (39) where xm,i,q∈ℝdmx_m,i,q ^d_m denotes the input activation at the q-th valid input position of example i. The historical activation statistic available before task t is Hm<t=∑τ<tHm(τ).H_m^<t= _τ<tH_m^(τ). (40) Hence, after completing task t, the cached activation statistic can be updated incrementally as Hm<t+1=Hm<t+Hm(t),H_m^<t+1=H_m^<t+H_m^(t), (41) without retaining activations or training examples from earlier tasks. Then, We compute its eigendecomposition Hm<t=UmΛmUm⊤,H_m^<t=U_m _mU_m , (42) where the eigenvalues are ordered in descending order. The corresponding eigenvectors Um=[um,1,…,um,dm]U_m=[u_m,1,…,u_m,d_m] (43) define the ordered candidate directions for protection. Importantly, the activation statistic determines both the directions and their prefix order. SPACE-LoRA does not reorder these directions according to the Fisher sensitivity. The Fisher is instead used only to determine how far this activation-ordered prefix should be extended for each module. Given a protection dimension kmk_m, the protected basis is therefore Vm=[um,1,…,um,km].V_m=[u_m,1,…,u_m,k_m]. (44) B.2 Historical Fisher Sensitivity While the activation Gram determines the candidate protection directions, SPACE-LoRA uses an input-space empirical Fisher statistic to evaluate how sensitive each direction is to previously learned tasks. For an example i from task τ, let Gm,i(τ)=∇Wmℓi(τ)∈ℝom×dmG_m,i^(τ)= _W_m _i^(τ) ^o_m× d_m (45) denote the gradient of the supervised loss with respect to the effective weight of module m. At the endpoint of task τ, we compute this statistic using the current task classifier and the observed ground-truth labels. We define the task-level input-space empirical Fisher as Fm(τ)=1Nτ∑i=1Nτ(Gm,i(τ))⊤Gm,i(τ).F_m^(τ)= 1N_τ _i=1^N_τ (G_m,i^(τ) ) G_m,i^(τ). (46) This matrix corresponds to the input-dimensional partial trace of the observed-label empirical Fisher associated with the effective weight WmW_m. The historical Fisher available before task t is accumulated using the number of examples in each previous task: Fm,past(t)=∑τ<tNτFm(τ)∑τ<tNτ.F_m,past^(t)= _τ<tN_τF_m^(τ) _τ<tN_τ. (47) In practice, this quantity can be maintained using the accumulated Fisher numerator together with the total number of examples, without storing task-specific training data. For the j-th activation direction um,ju_m,j, we define its historical sensitivity as pm,j(t)=um,j⊤Fm,past(t)um,j.p_m,j^(t)=u_m,j F_m,past^(t)u_m,j. (48) Equivalently, this quantity can be written as the average squared sensitivity along um,ju_m,j over all examples from previously observed tasks: pm,j(t)=1∑τ<tNτ∑τ<t∑i=1Nτ‖Gm,i(τ)um,j‖22.p_m,j^(t)= 1 _τ<tN_τ _τ<t _i=1^N_τ \|G_m,i^(τ)u_m,j \|_2^2. (49) Since Fm,past(t)F_m,past^(t) measures squared gradient responses along input directions, each directional sensitivity pm,j(t)p_m,j^(t) is nonnegative. Moreover, because Um=[um,1,…,um,dm]U_m=[u_m,1,…,u_m,d_m] forms a complete orthonormal basis, the sensitivities over all activation directions sum to the total historical sensitivity: ∑j=1dmpm,j(t) _j=1^d_mp_m,j^(t) =∑j=1dmum,j⊤Fm,past(t)um,j = _j=1^d_mu_m,j F_m,past^(t)u_m,j (50) =tr(Um⊤Fm,past(t)Um) =tr (U_m F_m,past^(t)U_m ) (51) =tr(Fm,past(t)). =tr (F_m,past^(t) ). (52) Therefore, the cumulative sensitivity of the first k activation directions, ∑j=1kpm,j(t) _j=1^kp_m,j^(t), represents the amount of historical sensitivity captured by that activation-ordered prefix. This identity provides the basis for the Fisher-coverage ratio used in Section 4.2. Appendix C Additional Experimental Details C.1 Vision Experimental Setup For ImageNet-R, the 5, 10, and 20 task settings use the same underlying image split and class ordering. This allows us to examine the effect of increasing the number of sequential learning stages while keeping the overall class set fixed. For DomainNet, we use five domains consisting of clipart, infograph, painting, real, and sketch, while excluding Quickdraw. The 345 classes are divided into five disjoint tasks with 69 classes per task. Images from all five retained domains are included in every task. For VTAB, we use a class incremental benchmark consisting of five datasets, Resisc45, DTD, Oxford IIIT Pets, EuroSAT, and Oxford Flowers102, in this order. Each dataset contributes 10 classes and constitutes a separate task. This results in five tasks and 50 classes in total. We use the officially released data and hold out 10% of the training images from each class as a validation set. This split yields 1,615 training images, 181 validation images, and 8,619 test images. For VDD, we use nine of the ten datasets in the Visual Domain Decathlon and exclude ImageNet. The nine datasets are Aircraft, CIFAR 100, Daimler Pedestrian Classification, DTD, GTSRB, Omniglot, SVHN, UCF101, and VGG Flowers. Each dataset constitutes one task in this order, such that every task introduces both a new domain and a new label set. The resulting benchmark contains 2,128 classes in total, with the number of classes per task ranging from 2 to 1,623. Since the labels of the official test split are not publicly available, we use the official validation split as the test set and hold out 10% of the training images from each class for validation. This results in 156,598 training images, 17,178 validation images, and 64,439 test images. C.2 Language Experimental Setup We evaluate continual language adaptation using a sequence of 15 text classification and natural language understanding datasets. Table 6 presents the datasets in the order in which they are introduced to the model. The 5 task setting uses the first five datasets, the 10 task setting uses the first ten datasets, and the 15 task setting uses the complete sequence. Therefore, each longer setting extends the preceding sequence without changing the order of previously introduced tasks. Table 6: Datasets used in the language continual learning experiments in their training order. Step Dataset Source Suite Task Type 1 AG News Standard CL Topic Classification 2 Amazon Standard CL Sentiment Analysis 3 Yelp Standard CL Sentiment Analysis 4 DBpedia Standard CL Topic Classification 5 Yahoo Standard CL Question Answering 6 MNLI GLUE Natural Language Inference 7 QQP GLUE Paraphrase Detection 8 RTE GLUE Natural Language Inference 9 SST-2 GLUE Sentiment Analysis 10 IMDB Independent Sentiment Analysis 11 WiC SuperGLUE Word Sense Disambiguation 12 CB SuperGLUE Natural Language Inference 13 COPA SuperGLUE Question Answering 14 MultiRC SuperGLUE Question Answering 15 BoolQ SuperGLUE Boolean Question Answering For most tasks, we construct a fixed dataset containing 1,000 training samples, 200 validation samples, and up to 500 test samples. The validation and test sets do not overlap whenever sufficient labeled data are available. For GLUE and SuperGLUE datasets whose public test labels are unavailable, we construct evaluation splits from the corresponding labeled validation data. The resulting test sets for RTE and WiC contain 77 and 438 samples, respectively. For smaller datasets such as CB and COPA, we use the available labeled evaluation samples for performance measurement. All experiments with T5 large use AdamW with a constant learning rate, a batch size of 8, and 3 epochs per task. We use LoRA with rank 8 and scaling parameter 16 for the query and value projections in all attention modules. A new adapter is introduced for each task, and the maximum input length is 128 tokens. We select the learning rate once using the five task setting with seed 0 and use the selected value for all methods and sequence lengths. No method specific learning rate tuning is performed. All compared methods use the same data, optimizer, and training schedule, while their method specific hyperparameters follow the default settings of their official implementations. C.3 Evaluation Protocol The model is evaluated after completing each task in the sequence. We report average accuracy after the final task to measure overall predictive performance and backward transfer to measure changes in performance on previously learned tasks. All methods are evaluated using the same task ordering, dataset splits, training schedule, and evaluation protocol. Algorithm 2 Continual Adaptation with SPACE-LoRA 1: Task datasets Dtt=1T\D_t\_t=1^T, protected modules ℳM, LoRA rank r, coverage target ρ 2: Initialize Hm<1←0H_m^<1← 0 and Fm,past(1)←0F_m,past^(1)← 0 for all m∈ℳm 3: Initialize the number of previous-task examples N<1←0N_<1← 0 4: for t=1,…,Tt=1,…,T do 5: Freeze the base model and all LoRA adapters learned from previous tasks 6: Introduce a new rank-r LoRA branch (Am,t,Bm,t)(A_m,t,B_m,t) for each m∈ℳm 7: for each module m∈ℳm do 8: if t=1t=1 then 9: km←0k_m← 0, Pm←0P_m← 0 10: else 11: Compute Hm<t=UmΛmUm⊤H_m^<t=U_m _mU_m with eigenvectors ordered by decreasing eigenvalue 12: for j=1,…,dmj=1,…,d_m do 13: pm,j(t)←um,j⊤Fm,past(t)um,jp_m,j^(t)← u_m,j F_m,past^(t)u_m,j 14: end for 15: if tr(Fm,past(t))=0tr(F_m,past^(t))=0 then 16: km←0k_m← 0 17: else 18: Find the smallest k∈0,…,dm−rk∈\0,…,d_m-r\ satisfying ∑j=1kpm,j(t)tr(Fm,past(t))≥ρ _j=1^kp_m,j^(t)tr (F_m,past^(t) )≥ρ 19: if such k exists then 20: km←k_m← k 21: else 22: km←dm−rk_m← d_m-r 23: end if 24: end if 25: Vm←[um,1,…,um,km]V_m←[u_m,1,…,u_m,k_m] 26: Pm←VmVm⊤P_m← V_mV_m 27: end if 28: end for 29: for each optimization step on DtD_t do 30: Compute the current-task loss and update the trainable parameters 31: for each module m∈ℳm do 32: Project the input factor Am,t←Am,t(I−Pm)A_m,t← A_m,t(I-P_m) 33: end for 34: end for 35: for each module m∈ℳm do 36: Collect endpoint activations on DtD_t and compute Hm(t)←∑i∈Dt∑qxm,i,qxm,i,q⊤H_m^(t)← _i∈ D_t _qx_m,i,qx_m,i,q 37: Using the current task classifier and ground-truth labels, compute Gm,i(t)←∇Wmℓi(t)G_m,i^(t)← _W_m _i^(t) for each i∈Dti∈ D_t 38: Compute Fm(t)←1Nt∑i=1Nt(Gm,i(t))⊤Gm,i(t)F_m^(t)← 1N_t _i=1^N_t (G_m,i^(t) ) G_m,i^(t) 39: Update the historical activation statistic Hm<t+1←Hm<t+Hm(t)H_m^<t+1← H_m^<t+H_m^(t) 40: Update the historical Fisher Fm,past(t+1)←N<tFm,past(t)+NtFm(t)N<t+NtF_m,past^(t+1)← N_<tF_m,past^(t)+N_tF_m^(t)N_<t+N_t 41: end for 42: N<t+1←N<t+NtN_<t+1← N_<t+N_t 43: end for