Paper deep dive
LCA: Local Classifier Alignment for Continual Learning
Tung Tran, Danilo Vasconcellos Vargas, Khoat Than
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 1:06:13 AM
Summary
The paper introduces Local Classifier Alignment (LCA), a novel loss function and continual learning framework designed to mitigate catastrophic forgetting and classifier-backbone mismatch in Class-Incremental Learning (CIL). By combining incremental model merging of PEFT modules with the LCA loss, the method aligns task-specific classifiers with an evolving backbone, achieving state-of-the-art performance across seven benchmarks.
Entities (5)
Relation Signals (3)
LCA → usedin → CIL
confidence 98% · we develop a complete solution for continual learning... using LCA.
LCA → addresses → Catastrophic Forgetting
confidence 95% · To address this issue [mismatch/forgetting], we propose a novel Local Classifier Alignment (LCA) loss
Model Merging → integratedwith → LCA
confidence 95% · we propose a complete CIL solution in which model merging... is used... and LCA serves as the main alignment loss
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A fundamental requirement for intelligent systems is the ability to learn continuously under changing environments. However, models trained in this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has recently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate forgetting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into a unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create a (potential) \textit{mismatch} between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel \textit{Local Classifier Alignment} (LCA) loss to better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop a complete solution for continual learning, following the model merging approach and using LCA. Extensive experiments on several standard benchmarks demonstrate that our method often achieves leading performance, sometimes surpasses the state-of-the-art methods with a large margin.
Tags
Links
- Source: https://arxiv.org/abs/2603.09888v2
- Canonical: https://arxiv.org/abs/2603.09888v2
Trouble viewing inline? Open PDF directly →
Full Text
62,734 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2026 LCA: LOCAL CLASSIFIER ALIGNMENT FOR CONTIN- UAL LEARNING Tung Tran 1 , Danilo Vasconcellos Vargas 1 , Khoat Than 2∗ 1 Kyushu University, Fukuoka, Japan 2 Hanoi University of Science and Technology, Hanoi, Vietnam tran.tung.son.949@s.kyushu-u.ac.jp vargas@inf.kyushu-u.ac.jp khoattq@soict.hust.edu.vn † ABSTRACT A fundamental requirement for intelligent systems is the ability to learn contin- uously under changing environments. However, models trained in this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has re- cently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate for- getting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into a unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create a (po- tential) mismatch between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel Local Classifier Alignment (LCA) loss to better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop a complete solution for con- tinual learning, following the model merging approach and using LCA. Extensive experiments on several standard benchmarks demonstrate that our method often achieves leading performance, sometimes surpasses the state-of-the-art methods with a large margin. 1INTRODUCTION In real-world scenarios, data arrives continuously, requiring deployed models to adapt to evolving distributions while preserving previously learned knowledge. This challenge, known as the stabil- ity–plasticity dilemma in continual learning, captures the difficulty of balancing adaptation with retention. A widely adopted benchmark approximates this setting by partitioning a dataset into dis- joint tasks and training a model sequentially without access to earlier data (Wang et al., 2024). At test time, the model must handle all tasks without being given the task identity. This setup is referred to as Class-Incremental Learning (CIL) (Van de Ven et al., 2022). Pre-trained models (PTMs) have recently emerged as a strong foundation for this setting, in contrast to earlier methods that trained networks from scratch (Li & Hoiem, 2017; Kirkpatrick et al., 2017). PTMs can be obtained through supervised or self-supervised training, and include large multimodal models such as CLIP (Radford et al., 2021) as well as vision backbones like the Vision Transformer (Dosovitskiy et al., 2020). Their broad generalization ability makes them effective feature extractors, requiring only lightweight adaptation and thereby reducing forgetting. Nevertheless, naive sequential adaptation still leads to degradation on past tasks. Restricting updates to the first task avoids forgetting but blocks useful transfer to later ones. Since each task introduces unique information, the final model must capture both shared and task-specific components. A natural approach is to incrementally adapt on each task and then merge the resulting models into a unified backbone. Insights from Linear Mode Connectivity and the Lottery Ticket Hypothesis ∗ Corresponding author. † Code is available at: this https URL 1 arXiv:2603.09888v2 [cs.AI] 11 Mar 2026 Published as a conference paper at ICLR 2026 (Frankle et al., 2020) suggest that task-specific solutions can be connected through low-loss paths and rely on sparse, critical parameters, making them amenable to combination. Advances in model merging (Ilharco et al., 2022; Yadav et al., 2023) further support this strategy, with both theoretical guarantees (Li et al., 2025) and empirical evidence on adaptation tasks (Akiba et al., 2025) showing that merged models can preserve and even enhance knowledge. Thus, it is reasonable to treat each task-specific model as a standalone expert containing complementary knowledge, whose consoli- dation yields a stronger overall backbone. However, merging backbones across tasks introduces a new challenge: classifiers trained independently may no longer align with the integrated backbone. Because these classifiers cannot be retrained without access to past data, even small parameter shifts can lead to severe drops in performance on earlier tasks. Addressing this misalignment is the central focus of our work. Our contributions in this work are as follows: • We introduce a novel loss, Local Classifier Alignment (LCA), for aligning CIL classifiers. This loss not only well aligns the backbone with the classifier, but also ensures robust- ness of the classifier. It also can contribute to reducing overlap between classes and hence improving classifier’s performance. • We provide theoretical analysis that decomposes the test error of a CIL model into three fundamental parts, including feature distribution shift, class-wise loss, and robustness. Those parts must be well controlled to assure a high performance for all observed tasks. Such a theory is crucial to support reliability of LCA and trustworthy CIL. • We propose a complete CIL solution in which model merging (for PEFT parameters only) is used to slightly adapt the backbone to new tasks and LCA serves as the main alignment loss to reduce mismatches between classifiers and the backbone. In our method, each class is represented as a Gaussian in the feature space, and LCA jointly optimizes all classifiers. • We conduct extensive experiments on seven benchmark datasets. Our results show that LCA consistently improves performance over baselines, enhances robustness under diverse scenarios, and can be integrated with other CIL methods to further boost their effectiveness. Figure 1 shows superiority of LCA on seven benchmark datasets. 2RELATED WORKS CIFAR100 IN-R IN-A CUB OB VTAB CARS Datasets 65 70 75 80 85 90 95 100 Accuracy (%) 92.8 84.3 66.5 86.7 81.1 84.6 70.1 94.8 85.8 75.0 90.8 81.4 95.2 76.2 IM IM+LCA Figure 1: A comparison between IM and IM+LCA. IM is the result after only done the Incremental Merging step, while IM+LCA has Local Classifier Alignment as the last step. Representation-Based Methods. This line of research leverages the representational power of large pre-trained models for continual learn- ing (Wang et al., 2024).Trained on mas- sive and diverse datasets, these models pro- vide strong transferability and inherent robust- ness against forgetting.One prominent di- rection adapts prompting techniques from nat- ural language processing, where prompts are modeled as learnable parameters attached to the inputs (Wang et al., 2022c;b; Tran et al., 2025). These prompts act as additional instruc- tions that guide model predictions during con- tinual learning. Another direction adapts the pre-trained backbone only once during the first task by using parameter-efficient fine-tuning (PEFT) modules and then relies on class pro- totypes for inference (Panos et al., 2023; Zhou et al., 2025; McDonnell et al., 2023; Perez et al., 2018). Such prototype-based approaches clas- sify via cosine similarity can avoid forgetting because accumulated prototypes across tasks are tasks order-invariant. However, empirical evidence shows that adaptation only in the first task is insuf- ficient, since data distributions in real-world scenarios vary substantially across tasks, which limits this method’s applicability in long training horizons. Incremental Backbone Evolution. A complementary line of research updates the backbone throughout the task sequence. For example, SLCA (Zhang et al., 2023) reduces forgetting by apply- 2 Published as a conference paper at ICLR 2026 ing a smaller learning rate to the backbone than to the classifier, while methods such as MagMax (Marczak et al., 2024), EASE (Zhou et al., 2024), and MOS (Sun et al., 2025b) focus on integrating task-specific components into a unified backbone. In all cases, past classifiers are typically frozen to avoid bias toward the current task’s data, which inevitably creates a mismatch between the evolv- ing backbone and the fixed classifiers. To mitigate this issue, EASE reweights old classifiers using semantic similarity between new and old prototypes, while MOS dynamically selects suitable back- bone adapters at inference time. Another line of research seeks to enrich past prototypes through augmentation, as in FeTrIL (Petit et al., 2023), PASS (Zhu et al., 2021b), IL2A (Zhu et al., 2021a), and CCFA (Kim et al., 2024), or to enhance the capacity of current classifiers through boosting, as in FOSTER (Wang et al., 2022a). Our approach retrains all classifiers after the unification step using samples drawn from simple Gaussian models, together with a novel loss that emphasizes lo- cal robustness. This procedure reduces the mismatch between the backbone and the classifiers and improves the stability of the overall model. Model Merging. Another related direction is model merging, which has recently gained attention for constructing a unified model from independently trained task-specific ones. Early methods such as FisherMerging (Matena & Raffel, 2022) use the Fisher Information Matrix to weight parameter importance, while Task Arithmetic (Ilharco et al., 2022) represents task updates as vectors, enabling explicit addition or subtraction of knowledge. More advanced approaches like TIES-Merge (Yadav et al., 2023) address task interference by resolving sign conflicts across task vectors. Although these methods show strong performance in out-of-domain generalization (Rame et al., 2022), applying them directly to continual learning often introduces a mismatch between the merged backbone and fixed classifiers, and incurs storage overhead since parameters from all past tasks must be retained. In contrast, our approach builds on the spirit of model merging by incrementally consolidating PEFT modules, which reduces memory requirements, and coupling this with a continuous training scheme that solidifies accumulated knowledge across tasks. 3METHODOLOGY This section presents our methodology for continual learning, which consists of two complemen- tary components. The first focuses on incrementally merging task-specific backbones into a unified model, while maintaining proximity across tasks by initializing new training from the latest merged backbone. The second addresses the mismatch that arises when frozen task-specific classifiers inter- act with the consolidated backbone, introducing an alignment mechanism based on class-wise local regions. Together, these components enable the model to effectively retain past knowledge while adapting to new tasks. 3.1PROBLEM FORMULATION In class-incremental learning (CIL), the objective is to train a single model sequentially on a series of (potentially infinite number of) tasks. Each task i is associated with a dataset D i =(x,y)| x∈X i , y ∈Y i , where x denotes an input sample and y its corresponding label. Here,X i represents the input space andY i the label space for task i. A defining characteristic of the CIL setting is that the label spaces of different tasks are strictly disjoint: ∀i̸= j, Y i ∩Y j =∅. This assumption implies that each new task introduces a set of novel classes that the model has not encountered before. Consequently, the model must continuously expand its knowledge while pre- serving performance on previously learned classes, making the prevention of catastrophic forgetting a central challenge in CIL. There are different strategies for constructing classifiers in continual learning, such as Nearest Class Mean (NCM) classifiers, which assign labels by comparing test features to stored class prototypes. In this work, however, we focus on the more general and widely used setup of progressively ex- panding Multi-Layer Perceptrons (MLPs) on top of the feature extractor. Instead of training a single unified classifier over all classes, a new MLP head is added for each task, with its output dimension matching the number of classes introduced by that task. This approach not only reduces storage but 3 Published as a conference paper at ICLR 2026 Algorithm 1 Incremental Merging (IM) Input: DatasetsD 1 ,...,D T , pretrained model θ pretrained , base PEFT θ peft 0 Output: Merged PEFT module θ merged 1: τ ← 0▷ Task vector accumulator 2: for i = 1 to T do▷ Sequentially process tasks 3: θ peft i ← finetune(θ peft i−1 ,D i )▷ Task-i adaptation 4: τ curr ← θ peft i − θ peft 0 ▷ Task vector 5:for k = 1 to d do▷ d = number of parameters 6:if|τ (k) curr |≥|τ (k) | then 7:τ (k) ← τ (k) curr ▷ Keep larger magnitude, preserve sign 8:end if 9:end for 10: θ merged ← θ peft 0 + α· τ▷ Update merged PEFT 11: end for also mitigates forgetting, since earlier classifiers remain frozen and are not modified during subse- quent training. After t tasks, the classifier consists of a set of task-specific headsθ cls 1 ,θ cls 2 ,...,θ cls t , and inference is performed by evaluating each head separately and concatenating their outputs: h(x) = concat h(x;θ cls 1 ), h(x;θ cls 2 ), ..., h(x;θ cls t ) . 3.2INCREMENTAL KNOWLEDGE CONSOLIDATION Although pre-trained models possess strong generalization, they still lack the domain-specific knowledge needed to serve as effective feature extractors. As a result, a fine-tuning stage is required. To prevent forgetting during continuous fine-tuning, early methods often assume that task distribu- tions remain close and restrict adaptation to the first task. However, as the number of tasks grows and their distributions diverge, performance on earlier tasks inevitably deteriorates. Inspired by re- search on model merging, we propose an incremental integration scheme that unifies task-specific backbones into a single consolidated backbone. We finetune the model—consisting of the pretrained backbone, the PEFT module, and the task classifier—parameterized by θ pretrained ,θ peft i ,θ cls i on task dataset D i . To limit drift across tasks, each task is initialized from the most recent PEFT parameters θ peft i−1 rather than the original base, keeping successive solutions close in parameter space—a property shown to be important for stable merging (Li et al., 2025). The finetuning step finetune uses SGD with cross-entropy loss. After finetuning task i, we flatten the PEFT parameters into a vector τ peft i . Assuming that final parameters should remain close to the pretrained initialization, we subtract the base vector τ peft 0 and select elements based on the largest absolute deviations. To avoid parameter growth, we retain only the previously merged vector and the current task vector during each merging step. The resulting update vector τ is added back to the base parameters to obtain the merged PEFT module for task i, as summarized in Algorithm 1. We further include an ablation in Appendix F, comparing Min, Max, and MaxAbs selection rules. All operators yield stable and competitive performance when applied to PEFT modules, reinforcing that merging only PEFT parameters is robust. To our knowledge, no prior work performs parameter- value-based merging without any trimming phase while still achieving effective results. 3.3LOCAL CLASSIFIER ALIGNMENT In contrast with a generalized feature extractor, task-specific classifiers should remain well separated to achieve strong classification performance. Because datasets from previous tasks are unavailable, updating earlier classifiers during new training would move them away from their previously opti- mized values. The standard pipeline therefore trains and merges the backbone while freezing the old classifiers, which introduces a mismatch between the unified backbone and those fixed heads. We address this discrepancy with an alignment step that bridges the two components. 4 Published as a conference paper at ICLR 2026 Consider the CIL approach where we use a pretrained model such as VIT to produce high-quality embeddings of the input samples, a Gaussian distribution (N or prototype) to represent a class, and a classifier. This approach has been investigated heavily and often high-performing CIL methods. At each time step t, one can use a learning method to train a classifier h t based on the classifier h t−1 which already fitted for prior tasks and a dataset D t for the current task. The overall performance of h t depends heavily on the employed learning algorithm. A good algorithm can well align the prototypes and classifier. However, this might not be always the case, and hence can degrade the overall CIL performance. We propose a simple finetuning step, called Local Classifier Alignment (LCA), to better align the classifier and prototypes. Specifically, LCA minimizes the following loss L(D,h t ) = 1 C t C t X i=1 L i (1) L i = E z∼D i [ℓ(h t , z)] + λE z,z ′ ∼D i ℓ(h t , z)− ℓ(h t , z ′ ) (2) where D i contains i.i.d. samples from the Gaussian distribution N i that represents class i in the feature space induced by the backbone, D =D 1 ,..., D t , and C t is the total number of observed classes upto task t. Basically, LCA tries to simultaneously minimize the loss for each class and keep the loss less sensi- tive to a small change in the input samples around the class prototypes. The class loss can be seen from the first term E z∼D i [ℓ(h t , z)], while the sensitivity comes from the second term in each L i . Such a term can be seen as a regularizer to penalize the classifier for unstable predictions. A larger value for λ suggests a stronger penalty for sensitivity of the loss. It is worth noting that the novelty of LCA comes from the regularization term. It not only helps improve robustness of the classifier, but also can reduce overlapping between classes. Indeed, when using the first term as the main objective for training the classifier, some samples randomly generated by N i of one class can lie far from the i-th class prototype and hence can be closer to some other class prototypes. This can harm the training for the classifier. The second term can help us reduce the negative effect from those potentially harmful samples, under a suitable choice of λ. 3.4THEORETICAL ANALYSIS We next analyze the generalization ability of the classifier h t . Until time step t, the classifier h t already has learned from C t classes. We want to estimate its test error for those learned tasks. To this end, the classical tradition is to estimate L(P,h t ) = 1 C t P C t i=1 E z∼N i [ℓ(h t , z)], which represents the overall expected error for the learned tasks. Let S t i=1 Z i be the decomposition of the data space into non-overlapping local areas, withZ i as the local area with centroid μ i , where μ i is the mean of the Gaussian distributionN i , for each index i. We have the following bound for the expected error of the overall classifier up to time step t, whose proof appears in Appendix A. Theorem 3.1. Consider a model h t learned from a dataset D = D 1 ,..., D t , where D i con- tains n i i.i.d. samples from distribution N i for each i ≤ C t , and a bounded loss ℓ. De- note P = 1 C t P C t i=1 N i as the overall distribution, n = P C t i=1 n i , ℓ max = sup z ℓ(h t , z), and ̄ε i (h t ) = E z∼N i ,s∼D i [|ℓ(h t , z) − ℓ(h t , s)| : z, s ∈ Z i ] for each index i. For any δ > 0, the following holds with probability at least 1− δ: L(P,h t )≤ L(D,h t ) + C t X i=1 n i n ̄ε i (h t ) + ℓ max r C t ln 4 + 2 ln(1/δ) n (3) This result shows that the test error of the classifier h t can be controlled by both the training error and the robustness term ̄ε = P C t i=1 n i n ̄ε i (h t ), which tells how robust is the loss w.r.t. to a small change in the input of the samples around the class prototypes. A stronger robustness (i.e., smaller ̄ε) can lead to a tighter bound on the test error, suggeting a better model. When both L(D,h t ) and ̄ε are small, the model must have small test error and hence generalize well on unseen data. On the other hand, a bad model will exhibit a large training error L(D,h t ) or large ̄ε. Therefore, Theorem 3.1 plays as 5 Published as a conference paper at ICLR 2026 the theoretical foundation for our LCA loss (1). Training by this loss would arguably improve both performance and robustness of the classifier, which is crucial for real-world CIL tasks. Corollary 1. Given the notations and assumptions as in Theorem 3.1, if n i = m for all i, then the following holds with probability at least 1− δ: L(P,h t )≤ L(D,h t ) + 1 C t C t X i=1 ̄ε i (h t ) + ℓ max s ln 4 m + 2 ln(1/δ) mC t (4) This is a direct consequence of Theorem 3.1. It suggests that, to assure high generalization, the number m of samples for each class should not be too small when doing alignment. If one use a small m, the uncertainty part in (4) will be large, meaning some randomness (by noise) can harm the alignment step. Remark 1. Although the LCA loss (1) is introduced to the CIL context, the loss is general enough to be employed in many other contexts. Indeed, one can use LCA loss to train a CIL classifier. Also, one can easily plug LCA to do a finetuning step for the existing CIL methods. In those cases, the theoretical benefits of LCA shown in Theorem 3.1 may remain valid. It is worth noting that the result in Theorem 3.1 holds when the feature distributions (or prototypes) are fixed. This means Theorem 3.1 may not directly apply for the cases of prototype changes. To fully understand the generalization dynamic of the overall CIL classifier, the previous results are insufficient. We need to take the backbone change into consideration. A change to the main backbone will lead to a change in the class prototypes that represent the GaussianN i . As a result, the overall distribution P in Theorem 3.1 will change after learning a new task. After training task t− 1, N i can accurately represent each new class i of task t− 1 and P t−1 = 1 C t−1 P C t−1 i=1 N i represents the overall distribution. After training a new task t, each distribution N i may change to b N i due to backbone change. Therefore, the overall distribution until task t can be rewritten as b P t = 1 C t P C t−1 i=1 b N i + 1 C t P C t j=C t−1 N j . However using this distribution to analyze generalization ability for all trained tasks may not be accurate. What we should analyze is for the distribution P t = 1 C t P C t i=1 N i . We have the following results. Theorem 3.2. Given the notations in Theorem 3.1, we have L(P t ,h t )≤ 2ℓ max TV(P t , b P t ) + L( b P t ,h t )(5) whereTV(·,·) denotes the total variation distance. Let ˆ D = S i≤C t−1 ˆ D i S C t j=C t−1 D j , with ˆ D i ∼ b N i and D j ∼ N j , contains i.i.d. samples. For any δ > 0, the following holds with probability at least 1− δ: L(P t ,h t )≤ 2ℓ max TV(P t , b P t ) + L( ˆ D,h t ) + C t X i=1 n i n ̄ε i (h t ) + ℓ max r C t ln 4 + 2 ln(1/δ) n (6) Basically, this theorem says that the test error of model h t will be small when one can ensure that the LCA loss (L( ˆ D,h t ) + P C t i=1 n i n ̄ε i (h t )) is small and the induced distribution b P t is close to P t (the accurate feature distribution). When distributions b P t and P t are far from each other, the backbone really causes catastrophic forgetting and thus can significantly increase the errors for past tasks. As a result, bound (6) suggests that a good CIL model should not significantly change the feature distributions of past tasks while being both robust and accurate for each class. Return to our CIL method, both components have their own important roles. IM can train the back- bone to adapt the feature distribution to the new task, but also reduces catastrophic forgetting for past tasks and hence keeps smallTV(P t , b P t ). The classifier alignment by LCA can simultaneously train the new part, adapt the old part while encouraging robustness of the overall classifier. Those roles are very crucial to ensure high performance for CIL. Therefore those analyses reveal the foundational support for LCA. 6 Published as a conference paper at ICLR 2026 Table 1: Average performance comparison on seven datasets with ViT-B/16-IN1K as the pretrained backbone. The best result is highlighted in bold, while the second best is highlighted in italic. MethodCIFAR100IN-RIN-ACUBOBVTABCARSOverall APER+Adapter90.8 ± 0.578.8 ± 0.658.9 ± 1.389.7 ± 1.380.3 ± 0.490.7 ± 0.650.6 ± 1.177.1 APER+Finetune81.7 ± 0.972.1 ± 0.858.7 ± 3.789.5 ± 1.477.8 ± 1.291.8 ± 1.453.2 ± 1.475.0 APER+SSF89.5 ± 1.078.1 ± 1.161.6 ± 0.589.6 ± 1.180.3 ± 0.691.8 ± 1.551.3 ± 1.177.5 APER+VPT-Deep89.0 ± 0.978.8 ± 0.757.0 ± 0.489.0 ± 1.279.8 ± 0.491.9 ± 1.450.6 ± 3.076.6 APER+VPT-Shallow88.1 ± 0.967.3 ± 3.556.9 ± 1.489.5 ± 1.479.7 ± 0.991.5 ± 0.850.9 ± 0.874.8 CODA-Prompt91.0 ± 0.278.2 ± 0.448.1 ± 0.975.6 ± 1.271.0 ± 0.165.6 ± 2.626.3 ± 0.665.1 DualPrompt86.7 ± 0.674.6 ± 0.555.3 ± 1.578.9 ± 1.074.4 ± 1.284.0 ± 5.949.4 ± 2.171.9 EASE91.7 ± 0.382.4 ± 0.567.8 ± 1.889.5 ± 1.280.8 ± 0.293.3 ± 0.148.1 ± 1.279.1 L2P87.7 ± 1.477.3 ± 0.652.6 ± 1.775.8 ± 1.873.8 ± 1.282.4 ± 2.853.4 ± 1.271.9 MOS94.3 ± 0.383.3 ± 0.667.6 ± 2.092.3 ± 0.686.1 ± 0.792.4 ± 0.571.4 ± 19.683.9 SLCA93.7 ± 0.385.1 ± 0.345.1 ± 19.890.2 ± 0.982.7 ± 0.691.1 ± 3.474.6 ± 2.280.4 IM92.8 ± 0.184.3 ± 1.066.5 ± 1.186.7 ± 0.881.1 ± 0.884.6 ± 4.970.1 ± 1.580.9 IM+LCA94.8 ± 0.385.8 ± 0.275.0 ± 0.590.8 ± 0.381.4 ± 0.595.2 ± 1.176.2 ± 1.485.6 4EXPERIMENTS This section presents the details of our experiments on seven benchmark datasets, along with an ablation study to examine some design aspects of our method. 4.1EXPERIMENT SETUP Dataset. We follow the setup in (McDonnell et al., 2023) to select datasets and define the number of classes in each incremental task. Specifically, we evaluate on CIFAR100 (Krizhevsky & Hinton, 2009), ImageNet-R (IN-R) (Hendrycks et al., 2021a), ImageNet-A (IN-A) (Hendrycks et al., 2021b), CUB-200 (CUB) (Welinder et al., 2010), OmniBenchmark (OB) (Zhang et al., 2022), VTAB (Zhai et al., 2019), and StanfordCars (CARS) (Krause et al., 2013). All datasets are split into 10 tasks, except VTAB which is divided into 5. Baselines. the following methods are used for comparison: • Ours: We consider two variants: IM (Incremental Merging) and IM+LCA, where the latter incorporates LCA in the alignment phase. • Pre-trained based CIL methods: CODA-Prompt (Smith et al., 2023), DualPrompt (Wang et al., 2022b), L2P (Wang et al., 2022c), EASE (Zhou et al., 2024), MOS (Sun et al., 2025b), SLCA (Zhang et al., 2023), APER (Zhou et al., 2025). For fair comparison and reproducibility, we adopt the implementation from (Sun et al., 2025a) for all baseline methods and use the same pre-trained backbone, ViT-B/16 trained on ImageNet-1K (ViT-B/16-IN1K) (Dosovitskiy et al., 2020). Training. We apply LoRA (Hu et al., 2022) and ensure a consistent computational budget by fixing the low-rank dimension to 64. All experiments are carried out on a single NVIDIA RTX 4090 GPU running Ubuntu 22.04. Detail on hyperparameters is mentioned in Appendix B. Evaluation Metrics. Following standard practice, we report the performance along incremental stages, defined as A = 1 T P T i=1 A t , where A t denotes the average accuracy after done training on task t. 4.2EXPERIMENT RESULTS 4.2.1OVERALL BENCHMARK Table 1 reports the mean and standard deviation of accuracy across three random seeds (1993, 1994, 1995). With LCA, the final model achieves the highest performance on five out of seven benchmark datasets, yielding an overall improvement of nearly 2%. Unlike methods such as EASE (Zhou et al., 2024) and MOS (Sun et al., 2025b), which either expand the backbone to integrate new tasks or rely on complex inference procedures, our approach requires no additional mechanism. The backbone is 7 Published as a conference paper at ICLR 2026 20406080100 Number of Classes 82.5 85.0 87.5 90.0 92.5 95.0 97.5 100.0 Accuracy (%) (a) CIFAR100 APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 50100150200 Number of Classes 60 65 70 75 80 85 90 95 100 Accuracy (%) (b) IN-R APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 50100150200 Number of Classes 30 40 50 60 70 80 90 Accuracy (%) (c) IN-A APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 50100150200 Number of Classes 70 75 80 85 90 95 100 Accuracy (%) (d) CUB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 50100150200 Number of Classes 70 75 80 85 90 95 100 Accuracy (%) (e) OB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 1020304050 Number of Classes 60 65 70 75 80 85 90 95 100 Accuracy (%) (f) VTAB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA 3676116156196 Number of Classes 30 40 50 60 70 80 90 Accuracy (%) (g) CARS APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA Figure 2: Performance curves of different methods across all tasks and datasets. All methods use ViT-B/16-IN1K as the pre-trained backbone without any additional exemplars. incrementally merged without storing past parameters or samples, and the only extra storage is the mean and covariance of each encountered class, which scales asO(n) with the number of classes. Figure 2 further shows that LCA consistently outperforms other methods across most datasets, with a notable improvement of 8% on ImageNet-A compared to the runner-up. 4.2.2ROBUSTNESS MEASUREMENT Following the standard (Hendrycks & Dietterich, 2019; Taori et al., 2020), we measure the per- formance of IM and IM+LCA on two robustness benchmarks, CIFAR100-C and CIFAR100-P. CIFAR100-C is constructed by applying 19 types of common corruptions (e.g., noise, blur, weather, and digital distortions) at 5 severity levels to the original CIFAR-100 test set, thereby evaluating model robustness under distribution shift. The final corruption robustness accuracy is computed as Acc C = 1 19× 5 19 X c=1 5 X s=1 A c,s , where A c,s is the accuracy under corruption type c with severity s. CIFAR100-P, in contrast, focuses on prediction stability under perturbations. Each image is per- turbed by transformations such as translations, rotations, or noise, and the model’s consistency is 8 Published as a conference paper at ICLR 2026 50100150200 Number of Classes 70 75 80 85 90 Accuracy (%) IN-A =0.0 =0.1 =1.0 =10.0 =100.0 1020304050 Number of Classes 90 92 94 96 98 100 Accuracy (%) VTAB =0.0 =0.1 =1.0 =10.0 =100.0 Figure 3: Effect of λ on the accuracy on two datasets. CIFAR100IN-RIN-ACUBOBVTABCARS Datasets 60 70 80 90 100 Accuracy (%) 91.9 65.2 59.8 82.2 79.0 89.2 59.7 93.1 81.9 68.5 90.4 81.5 91.0 68.7 90.4 81.6 63.5 83.8 78.7 80.2 62.4 92.0 83.6 67.4 89.0 80.7 87.4 66.7 MOS MOS - LCA SLCA SLCA - LCA (a) Performance for variants of MOS and SLCA. The postfix LCA means that method uses our alignment loss. CIFAR100-C Mean Accuracy CIFAR100-C Severity 5 CIFAR100-P Mean Accuracy Overall Robustness Score Robustness Metrics 0 20 40 60 80 100 Accuracy (%) 75.9 65.5 85.2 80.5 78.1 67.4 87.8 82.9 IM IM+LCA (b) Robustness comparison between IM and IM+LCA on CIFAR100-C and CIFAR100-P. Metrics include mean accuracy, accuracy at severity level 5, and overall robustness score. Figure 4: (a) Complementary evaluation of LCA when using LCA for MOS and SLCA. (b) Robust- ness performance of IM and IM+LCA on corruption and perturbation benchmarks. measured across perturbed versions. The perturbation robustness accuracy is defined as Acc P = 1 |P| X p∈P 1 N p N p X i=1 A p,i , where P denotes the set of perturbation types, N p is the number of perturbed samples for type p, and A p,i is the accuracy on the i-th perturbed sample. Finally, to summarize robustness across both benchmarks, we report an overall robustness score: Robustness = 1 2 (Acc C + Acc P ). Together, these benchmarks evaluate both accuracy under distributional corruption and resilience of predictions under small perturbations. Figures 4b and 5 summarize the robustness results. Figure 4b shows that when being trained with LCA, the model obtains a clear improvement on robustness, with more than +2% gain in mean accuracy on CIFAR100-C and a +2.5% gain on CIFAR100-P. The radar plots in Figure 5 further reveal that LCA consistently improves accuracy across all corruption and perturbation types. These results confirm that LCA strengthens robustness both on average and across diverse perturbations. 9 Published as a conference paper at ICLR 2026 20% 40% 60% 80% Gaussian Noise Shot Noise Impulse Noise Defocus Blur Glass Blur Motion Blur Zoom Blur Snow Frost Fog Brightne s Contrast Elastic Transfor m Pixelate JPEG Com pression Gaussian Blur Saturate Spatter Speckle Noise +3.2% +3.7% +3.0% +3.1% +2.7% +3.1% +3.3% +2.8% +3.4% +2.9% +3.0% +3.1% +3.2% +1.8% +3.6% +3.1% +3.1% +3.0% +3.6% IM IM+LCA (a) Performance on CIFAR100-C. 20% 40% 60% 80% Gaussian Noise Shot Noise Motion Blur Zoom Blur Snow Brightne s Translat e RotateTilt Scale +3.3% +3.5% +3.1% +3.1% +3.5% +3.1% +2.7% +2.8% +2.8% +2.8% IM IM+LCA (b) Performance on CIFAR100-P. Figure 5: Accuracy performance of IM and IM+LCA under different corruption and perturbation types. The relative difference between IM and IM+LCA is highlighted. 4.2.3ABLATION STUDY LCA as a complementary component. We further evaluate the applicability of LCA on SLCA (Zhang et al., 2023) and MOS (Sun et al., 2025b), two methods that also update the backbone pro- gressively. To asses the impact of our proposed method, we construct two baselines by omitting the final step of these methods, and retaining only the update backbone part. From these, we build their counterparts, SLCA-LCA and MOS-LCA, which include the aligment step as the final step. We measure the average performance and report in Figure 4a. We do not find the optimal hyper- parameters but fixing the value of λ at 0.1, yet the variants with LCA show improvements on all scenerios, notably in IN-A, CUB, VTAB, and CARS, in some cases even matching the reported re- sults of the original methods (SLCA-LCA achieves 89.0% in CUB compare to 90.0% of the original, MOS-LCA achieves 93.1% in CIFAR100 compare to 94.3%). Hyperparameter sensitivity. In our method, λ controls the strength of the robustness penalty in the loss function. While an appropriate choice of λ is important for achieving strong performance as Figure 3 shows that overly strong regularization can degrade results. In practice, we find that λ = 0.1 provides stable and reliable performance across all datasets. 5CONCLUSION This paper investigates the mismatch that may arise between a continuously updated backbone and the task-specific classifiers in continual learning. We propose a theoretically grounded loss function, with a provable bound on classification error, which enables the training of all classifiers using fea- tures generated from a Gaussian distribution. Extensive experiments on seven benchmark datasets confirm the effectiveness of our method, and additional robustness evaluations under various noisy conditions demonstrate consistent improvements. While this work primarily focuses on addressing the alignment phase, future research will explore integrating the proposed loss into the end-to-end training pipeline. Such an approach has the potential to further enhance the robustness of the back- bone itself and the performance of the overall CIL method. Despite having significant contributions, our work still remains some limitations. For example, we have not investigated the proposed LCA loss in other contexts. Furthermore, although providing a theoretical foundation and novel insights, the developed theory does not take the training of the whole backbone into consideration. Addressing those limitations can open interesting directions for future research. 10 Published as a conference paper at ICLR 2026 REPRODUCIBILITY STATEMENT We have taken several steps to ensure the reproducibility of our results. The hyperparameters re- quired for both training and inference are explicitly reported in Appendix B. The complete source code is provided as supplementary material, where users can download and unzip the package, fol- low the installation instructions, and run the main script to reproduce our experiments. All datasets used in this work are publicly available, implementation details and evaluation protocols are de- scribed in Section 4.1. Together, these resources are intended to make it straightforward for others to replicate and build upon our work. ACKNOWLEDGEMENTS This work was supported by JSPS Grant-in-Aid for Challenging Exploratory Research - Grant Number JP25K22833 and JSPS Research on Academic Transformation Areas (A) - Grant Num- ber JP22H05194. The authors also gratefully acknowledge the financial support provided by Hattori Hokokai Foundation and Telecommunications Advancement Foundation. REFERENCES Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195–204, 2025. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, p. 3259–3269. PMLR, 2020. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 8340–8349, 2021a. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174, 2021b. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. PMLR, 2019. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, p. 709–727. Springer, 2022. Taehoon Kim, Jaeyoo Park, and Bohyung Han. Cross-class feature augmentation for class incre- mental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 13168–13176, 2024. 11 Published as a conference paper at ICLR 2026 James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017. Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. Fine-grained categorization and dataset bootstrapping using deep learning with humans in the loop.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 580–587, 2013. doi: 10.1109/CVPR.2013.83.URL https://ai.stanford.edu/ ̃ jkrause/cars/car_ dataset.html. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009. Hongkang Li, Yihua Zhang, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. arXiv preprint arXiv:2504.10957, 2025. Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017. Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109–123, 2022. Daniel Marczak, Bartłomiej Twardowski, Tomasz Trzci ́ nski, and Sebastian Cygert. Magmax: Lever- aging model merging for seamless continual learning. In European Conference on Computer Vision, p. 379–395. Springer, 2024. Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022. Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36:12022–12053, 2023. Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 18820–18830, 2023. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Gr ́ egoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free class-incremental learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 3911–3920, 2023. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. PmLR, 2021. Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821–10836, 2022. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017. James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11909–11919, 2023. 12 Published as a conference paper at ICLR 2026 Hai-Long Sun, Da-Wei Zhou, De-Chuan Zhan, and Han-Jia Ye. Pilot: A pre-trained model-based continual learning toolbox. SCIENCE CHINA Information Sciences, 68(4):147101, 2025a. doi: https://doi.org/10.1007/s11432-024-4276-4. Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for pre-trained model-based class-incremental learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 39, p. 20699–20707, 2025b. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583–18599, 2020. Quyen Tran, Tung Lam Tran, Khanh Doan, Toan Tran, Dinh Phung, Khoat Than, and Trung Le. Boosting multiple views for pretrained-based continual learning. In The Thirteenth International Conference on Learning Representations, 2025. Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197, 2022. Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and com- pression for class-incremental learning. In European conference on computer vision, p. 398–414. Springer, 2022a. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362–5383, 2024. Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, p. 631–648. Springer, 2022b. Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 139–149, 2022c. Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical report, California Institute of Technology, 2010. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093–7115, 2023. Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019. Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 19148–19158, 2023. Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In European Conference on Computer Vision, p. 594–611. Springer, 2022. Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 23554–23564, 2024. Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu.Revisiting class- incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 133(3):1012–1032, 2025. 13 Published as a conference paper at ICLR 2026 Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. Advances in neural information processing systems, 34:14306–14318, 2021a. Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5871–5880, 2021b. 14 Published as a conference paper at ICLR 2026 APROOF FOR MAIN THEOREMS Proof of Theorem 3.1. Let p i = P (Z i ) be the probability measure of area Z i , and note that P t i=1 p i = 1. We first observe that L(P,h t ) = L(P,h t )− C t X i=1 n i n L(N i ,h t ) + C t X i=1 n i n L(N i ,h t )− L(D,h t ) + L(D,h t ) (7) Since L(P,h t ) = P C t i=1 p i L(N i ,h t ) and L(N i ,h t )≤ ℓ max , we observe that L(P,h t )− C t X i=1 n i n L(N i ,h t ) = C t X i=1 p i L(N i ,h t )− C t X i=1 n i n L(N i ,h t )(8) = C t X i=1 p i − n i n L(N i ,h t )(9) ≤ ℓ max C t X i=1 p i − n i n (10) Note that (n 1 ,...,n C t ) is a multinomial random variable with parameters n and (p 1 ,...,p C t ). For any ε > 0, Bretagnolle-Huber-Carol inequality shows Pr P C t i=1 p i − n i n ≥ 2ε ≤ 2 C t exp(−2nε 2 ). In other words, for any δ > 0, taking ε = q C t ln 2−lnδ 2n , the following holds true with probability at least 1− δ: L(P,h t )− C t X i=1 n i n L(N i ,h t ) ≤ C r C t ln 4− 2 lnδ n (11) Next we observe the second term: C t X i=1 n i n L(N i ,h t )− L(D,h t ) = C t X i=1 n i n L(N i ,h t )− C t X i=1 n i n L(D i ,h t )(12) = C t X i=1 n i n [L(N i ,h t )− L(D i ,h t )](13) = C t X i=1 n i n (E z∼N i [ℓ(h t , z)− L(D i ,h t ) : z ∈Z i ])(14) = C t X i=1 n i n (E z∼N i ,s∼D i [ℓ(h t , z)− ℓ(h t , s) : z, s∈Z i ]) (15) ≤ C t X i=1 n i n (E z∼N i ,s∼D i [|ℓ(h t , z)− ℓ(h t , s)| : z, s∈Z i ])(16) = C t X i=1 n i n ̄ε i (h t )(17) Combining (7) and (11) and (17) completes the proof. 15 Published as a conference paper at ICLR 2026 Proof of Theorem 3.2. Denote p t (z) and ˆp t (z) as the density of distributions P t and b P t , respec- tively. Observe that L(P t ,h t ) = L(P t ,h t )− L( b P t ,h t ) + L( b P t ,h t )(18) = E z∼P t [ℓ(h t , z)]− E z∼ b P t [ℓ(h t , z)] + L( b P t ,h t )(19) = Z ℓ(h t , z)p t (z)dz− Z ℓ(h t , z) ˆp t (z)dz + L( b P t ,h t )(20) ≤ Z |ℓ(h t , z)|·|p t (z)dz− ˆp t (z)|dz + L( b P t ,h t )(21) ≤ ℓ max Z |p t (z)dz− ˆp t (z)|dz + L( b P t ,h t )(22) = 2ℓ max TV(P t , b P t ) + L( b P t ,h t )(23) which shows the first statement. The second result thus follows using the result of Theorem 3.1. BTRAINING DETAILS Table 2: Training and merging hyperparameters used in all experiments of LCA. HyperparameterValue Training epochs10 Batch size64 Base learning rate1× 10 −2 Weight decay5× 10 −4 OptimizerSGD (momentum = 0.9) Learning rate schedulerCosineAnnealing (eta-min = 1× 10 −6 ) Merging coefficient α1.0 Alignment classifier epochs10 Number of samples per class512 Alignment batch size128 LoRA rank64 LoRA alpha128 LoRA dropout0.0 LoRA initializationGaussian CDETAILS OF EXAMPLE GENERATION In this section, we provide details on how Gaussian distributions are used to align the classifiers during the incremental training process. Pre-trained models typically produce well-structured rep- resentations, which allows us to approximate each class distribution using its empirical mean and covariance. Storing class statistics. For each class c at training stage t, we extract features using the backbone parameters θ t . The empirical mean μ c ∈ R d and covariance Σ c ∈ R d×d are computed as μ c = 1 K |D t | X i=1 1(y i = c)φ(x i ;θ t ),(24) Σ c = 1 K |D t | X i=1 φ(x i ;θ t )− μ c φ(x i ;θ t )− μ c ⊤ ,(25) whereφ(·;θ t ) denotes the feature extractor with backbone parametersθ t , andK = P |D t | i=1 1(y i = c). 16 Published as a conference paper at ICLR 2026 1020304050 Number of Classes 60 70 80 90 100 Accuracy (%) VTAB IN1K-SSF-IM+LCA IN21K-SSF-IM+LCA IN1K-SSF-IM IN21K-SSF-IM IN1K-VPT-IM+LCA IN21K-VPT-IM+LCA IN1K-VPT-IM IN21K-VPT-IM IN1K-Adapter-IM+LCA IN21K-Adapter-IM+LCA IN1K-Adapter-IM IN21K-Adapter-IM IN1K-LoRA-IM+LCA IN21K-LoRA-IM+LCA IN1K-LoRA-IM IN21K-LoRA-IM Figure 6: Ablation study on different PEFT strategies with two backbones, ViT-B/16-1K and ViT- B/16-21K. Feature replay via Gaussian sampling. Before each alignment step, we regenerate features for every class c∈Y t by sampling from a multivariate Gaussian distribution: ˆ z c ∼N (μ c , Σ c ).(26) We generate approximately five times the batch size of synthetic features per class, which provides sufficient diversity for classifier alignment. DFURTHER ANALYSIS Results with different fine-tuning strategies. We further examine the adaptability of incremental merging and LCA with different parameter-efficient fine-tuning (PEFT) strategies on the VTAB dataset. The strategies include SSF (Lian et al., 2022), Adapters (Rebuffi et al., 2017), VPT (Jia et al., 2022), and LoRA (Hu et al., 2022), evaluated with ViT-B/16 pretrained on both ImageNet-1K and ImageNet-21K. Figure 6 shows that LCA consistently improves performance across all PEFT methods. In particular, VPT without LCA suffers a sharp performance drop, suggesting that incremental merging alone is insufficient to prevent forgetting in some cases. Adding LCA, however, proves beneficial across all methods, demonstrating its adaptability. Notably, despite having relatively few trainable parameters, Adapters emerge as a strong candidate within our pipeline, achieving the highest accuracy when combining with LCA. EFURTHER RESULTS ON ACCURACY In this section, we report additional results of our proposed method when finetuning with the Adapter PEFT module (Houlsby et al., 2019), shown in Figure 7. We refer to this variant as IM+LCA Adapter, while the results in the main text correspond to IM+LCA LoRA. From the figure, we observe that several baseline methods suffer from strong instability under cer- tain settings (e.g., SLCA on IN-A and MOS on CARS), whereas our method remains consistently stable across all three random seeds (1993, 1994, 1995). Furthermore, with Adapter, our approach achieves state-of-the-art performance on datasets such as CUB and OB. This confirms that the lower performance reported in the main text for these datasets is due to our reproducibility-oriented con- straints on PEFT method choices, rather than limitations of the proposed method itself. 17 Published as a conference paper at ICLR 2026 Table 3: Average performance on different merge operators. CA means normal classifier alignment, while LCA is our proposed method. MethodIN-AVTABCARS IM Max66.26 ± 0.6584.76 ± 3.6969.90 ± 1.47 IM Max CA70.86 ± 1.1092.05 ± 2.4471.88 ± 0.49 IM Max LCA73.73 ± 1.1394.22 ± 1.0674.85 ± 1.50 IM Min66.49 ± 0.6785.08 ± 3.4469.79 ± 1.36 IM Min CA71.11 ± 1.1092.13 ± 2.3371.82 ± 0.50 IM Min LCA73.00 ± 1.2094.24 ± 1.1174.28 ± 1.29 IM MaxAbs66.5 ± 1.1084.6 ± 4.9070.1 ± 1.50 IM MaxAbs CA71.31 ± 0.8291.71 ± 2.2472.15 ± 0.52 IM MaxAbs LCA75.0 ± 0.5095.2 ± 1.1076.2 ± 1.40 20406080100 Number of Classes 82.5 85.0 87.5 90.0 92.5 95.0 97.5 100.0 Accuracy (%) (a) CIFAR100 APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 255075100125150175200 Number of Classes 60 65 70 75 80 85 90 95 100 Accuracy (%) (b) IN-R APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 255075100125150175200 Number of Classes 30 40 50 60 70 80 90 Accuracy (%) (c) IN-A APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 255075100125150175200 Number of Classes 70 75 80 85 90 95 100 Accuracy (%) (d) CUB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 255075100125150175200 Number of Classes 70 75 80 85 90 95 100 Accuracy (%) (e) OB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 1020304050 Number of Classes 60 65 70 75 80 85 90 95 100 Accuracy (%) (f) VTAB APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter 3676116156196 Number of Classes 30 40 50 60 70 80 90 Accuracy (%) (g) CARS APER + Adapter APER + SSF APER + VPT-Deep APER + VPT-Shallow DualPrompt CODA-Prompt L2P EASE SLCA MOS IM+LCA LoRA IM+LCA Adapter Figure 7: Additional results on Performance Curve. FDIFFERENT MERGING OPERATORS Table 3 shows the benefits of our parameter-selection design. We evaluate the generalization of our method using three merge operators: Min, which selects the parameter with the smallest value; 18 Published as a conference paper at ICLR 2026 Task 0Task 1Task 2Task 3Task 4Task 5Task 6Task 7Task 8Task 9 MaxAbs MAX MIN Merge Method 1.00000.99870.99660.99430.99210.99000.98850.98610.98450.9829 1.00000.99730.99320.98640.97810.96700.95690.94280.93090.9184 1.00000.99730.99320.98650.97810.96760.95780.94410.93260.9207 0.90 0.92 0.94 0.96 0.98 1.00 Cosine Similarity (a) IN-A Task 0Task 1Task 2Task 3Task 4 MaxAbs MAX MIN Merge Method 1.00000.99970.99890.99820.9970 1.00000.99930.99800.99610.9928 1.00000.99930.99700.99500.9915 0.90 0.92 0.94 0.96 0.98 1.00 Cosine Similarity (b) VTAB Task 0Task 1Task 2Task 3Task 4Task 5Task 6Task 7Task 8Task 9 MaxAbs MAX MIN Merge Method 1.00000.99840.99120.98830.98370.98050.97720.97420.97110.9674 1.00000.99650.98690.97830.96570.95290.93770.92330.90690.8877 1.00000.99640.98730.97880.96670.95410.93880.92460.90840.8895 0.90 0.92 0.94 0.96 0.98 1.00 Cosine Similarity (c) CARS Figure 8: Similarity measurements between backbone representations across tasks, computed rela- tive to the backbone obtained from the first task. Max, which selects the largest; and MaxAbs, which selects the parameter with the largest absolute magnitude. When we want to reduce the cost of storage by merging only two models, the MaxAbs operator can be viewed as a simplified form of TIES-Merging without the trimming phase in Yadav et al. (2023), and is equivalent to the merge rule proposed in Marczak et al. (2024). Importantly, unlike prior works such as Marczak et al. (2024), which apply merging to all model parameters, we show that merging only PEFT module parameters is substantially easier and more stable. Under this restricted setting, even very simple operators like Min or Max achieve compet- itive performance across benchmarks, demonstrating that PEFT-focused merging does not require 19 Published as a conference paper at ICLR 2026 CIFAR100IN-RIN-ACUBOBVTABCARS Datasets 0 20 40 60 80 100 Accuracy (%) 93.49 83.09 68.19 90.60 85.24 92.53 71.41 93.77 84.50 70.40 91.03 86.50 93.50 73.87 93.78 85.04 67.58 88.00 82.98 87.49 74.37 93.85 85.48 69.03 88.68 83.25 88.86 75.88 MOS MOS-LCA SLCA SLCA-LCA Figure 9: Accuracy of the original methods and their LCA variants across standard datasets. complex selection strategies. Moreover, we observe that the merge coefficient does not need careful tuning as a fixed value of 1.0 works reliably across all benchmarks. GABLATION ON ROBUSTNESS TERM We measure the similarity between the backbone obtained at each task and the backbone learned after the first task. The backbone at the first task serves as an anchor, as it is assumed to bridge the gap between the pre-trained dataset and the target domain. Figure 8 illustrates how incremental fine- tuning with merging enables the backbone to evolve gradually over time, supporting the theoretical intuition discussed in Section 3.4. Table 3 further shows the impact of including the robustness term during training. Across all merge operators, adding the robustness term yields consistent and clear improvements, demonstrating its effectiveness independent of the chosen merge strategy. HEFFECT OF LCA ON DIFFERENT METHODS In addition to the results reported in Figure 4a, where we replace the original alignment loss with our LCA loss, we conduct a more extensive ablation to evaluate the adaptability of LCA when attached to other alignment-based methods, specifically SLCA (Zhang et al., 2023) and MOS (Sun et al., 2025b). Both methods continually evolve the backbone representations over time, making them natural candidates for incorporating our proposed loss. Figure 9 shows that, across all datasets, the LCA-augmented variants consistently outperform their corresponding original methods, indicating a clear accuracy improvement. We further evaluate robustness following the same protocol used for IM. Specifically, we test on CIFAR100-C, the corrupted version of CIFAR100, and CIFAR100-P, the perturbed version. Details of these robustness benchmarks are provided in Section 4.2.2. Figure 10 summarizes the overall 20 Published as a conference paper at ICLR 2026 CIFAR100-C Mean Accuracy CIFAR100-C Severity 5 CIFAR100-P Mean Accuracy Overall Robustness Score Robustness Metrics 60 65 70 75 80 85 90 Accuracy (%) 74.87 62.34 84.98 79.93 76.51 64.49 86.35 81.43 75.57 65.01 85.18 80.38 77.03 66.35 86.93 81.98 MOS MOS-LCA SLCA SLCA-LCA Figure 10: Accuracy of the original methods and their LCA variants under various corruption and perturbation types. 20% 40% 60% 80% Gaussian Noise Shot Noise Impulse Noise Defocus Blur Glass Blur Motion Blur Zoom Blur Snow Frost Fog Brightne s Contrast Elastic Transfor m Pixelate JPEG Com pression Gaussian Blur Saturate Spatter Speckle Noise +1.9% +3.6% +0.6% +3.2% +2.9% +2.0% +2.2% +1.5% +1.4% +1.8% +1.4% +3.0% +1.8% +2.2% +1.8% +2.1% +2.5% +2.5% +3.4% MOS MOS-LCA 20% 40% 60% 80% Gaussian Noise Shot Noise Impulse Noise Defocus Blur Glass Blur Motion Blur Zoom Blur Snow Frost Fog Brightne s Contrast Elastic Transfor m Pixelate JPEG Com pression Gaussian Blur Saturate Spatter Speckle Noise +2.3% +2.0% +1.1% +2.2% +0.6% +2.4% +2.7% +1.6% +1.3% +1.5% +2.1% +2.3% +2.3% +1.5% +2.3% +2.3% +2.1% +1.6% +2.1% SLCA SLCA-LCA Figure 11: Accuracy of the original methods and their LCA variants on CIFAR100-C corruptions. robustness gains, while Figures 11 and 12 break down the improvements for each corruption and perturbation type, respectively. IEFFECT OF LCA WHEN SELECTIVELY UPDATING CLASSIFIERS Updating only half of the task-specific classifier heads (five in IN-A and three in VTAB) during alignment still yields competitive performance as demonstrated in figure 13, indicating that full classifier finetuning is not always necessary. 21 Published as a conference paper at ICLR 2026 20% 40% 60% 80% Gaussian Noise Shot Noise Motion Blur Zoom Blur Snow Brightne s Translat e RotateTilt Scale +1.5% +2.4% +3.3% +1.9% +0.6% +2.0% +1.2% +1.6% +1.4% MOS MOS-LCA 20% 40% 60% 80% Gaussian Noise Shot Noise Motion Blur Zoom Blur Snow Brightne s Translat e RotateTilt Scale +2.1% +2.1% +2.7% +1.9% +1.8% +1.8% +1.9% +2.2% +1.9% +2.2% SLCA SLCA-LCA Figure 12: Accuracy of the original methods and their LCA variants on CIFAR100-P perturbations. 50100150200 Number of Classes 60 65 70 75 80 85 90 95 Accuracy (%) (a) IN-A Full classifiers 1 classifier 3 classifiers 5 classifiers 7 classifiers 1020304050 Number of Classes 80 85 90 95 100 105 Accuracy (%) (b) VTAB Full classifiers 1 classifier 3 classifiers Figure 13: Accuracy of selectively finetuning a subset of classifiers. 22