Paper deep dive
Natural Gradient Descent for Online Continual Learning
Joe Khawand, David Colliaux
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/26/2026, 2:20:45 AM
Summary
This paper introduces the application of Natural Gradient Descent (NGD) with Kronecker-Factored Approximate Curvature (KFAC) to Online Continual Learning (OCL). By approximating the Fisher Information Matrix (FIM) using KFAC, the authors address the computational challenges of second-order optimization in deep learning. The method is shown to improve performance and convergence across various OCL benchmarks (Split CIFAR-100, CORE50, Split miniImageNet) and existing OCL strategies, effectively mitigating catastrophic forgetting.
Entities (6)
Relation Signals (3)
Natural Gradient Descent → uses → Kronecker Factored Approximate Curvature
confidence 100% · we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC).
Natural Gradient Descent → improves → Online Continual Learning
confidence 95% · This method demonstrates substantial improvements in performance across all OCL methods
ResNet18 → trainedusing → Natural Gradient Descent
confidence 90% · Using a Resnet18 He et al. (2016) we explore the use of different CL techniques in combination with the NGD-KFAC optimizer.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet.
Tags
Links
- Source: https://arxiv.org/abs/2603.20898v1
- Canonical: https://arxiv.org/abs/2603.20898v1
Trouble viewing inline? Open PDF directly →
Full Text
49,808 characters extracted from source content.
Expand or collapse full text
Natural Gradient Descent for Online Continual Learning Joe Khawand Ecole Polytechnique & Télécom Paris Paris, France joe.khawand@polytechnique.org David Colliaux Sony Computer Science Laboratories Paris, France 0000-0003-1898-4864 Abstract Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model’s performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet. 1 Introduction With the surge of embedded AI systems and the growing need for them, Continual Learning presents itself as an elegant solution to on-device learning. This paradigm refers to the ability of a model to learn by continuously assimilating knowledge over time as observed in biological entities Kudithipudi et al. (2022). Mathematically, this refers to the ability to train a model while breaking the i.i.d. assumption on the data, sending classes one after the others, for example, in the classification scenario. Naively training a model in this scenario leads to the phenomenon of catastrophic forgetting French (1999); McCloskey and Cohen (1989), where the model forgets previously learned representations and specializes on the last seen task. Most of the research focuses on scenarios in which the model can iterate for multiple epochs Van de Ven et al. (2022) and use batches of reasonable size. Unfortunately, doing so requires significant storage that is more often than not unavailable on devices. Online Continual Learning (OCL) Mai et al. (2022); Soutif-Cormerais et al. (2023) considers the more realistic but more difficult scenario where the model can only iterate once on the current task and has no access to the previous ones. The model is thus required to achieve satisfactory performance from a single pass over the online data stream using very small batch sizes. In this context, the model may encounter new categories (Online Class Incremental, OCI) or changes in the characteristics of the data such as alterations in the background, the introduction of blur or noise, changes in lighting, or the presence of obstructions (Online Domain Incremental, ODI) Van de Ven et al. (2022). We focus our attention on the methods that do not require access to task labels, as this does not represent a realistic OCL scenario. In this complex scenario, OCL methods struggle to achieve high accuracies as shown in the survey Mai et al. (2022). There is still a big gap between OCL-trained models and Offline trained models. In this paper, we explore the use of 2nd order optimization in Online Continual Learning. Motivated by the need for faster convergence, we investigate the use of the Natural Gradient Descent optimizer Amari (1998) with Kronecker Factored Approximate Curvature (KFAC) Martens and Grosse (2015), in the OCL scenario. We demonstrate that this method, presented as the Steepest Descent in the distribution space, boasts increased performance across all the OCL methods and tricks presented in Mai et al. (2022), on datasets such as Split CIFAR-100 Krizhevsky et al. (2009), CORE50 Lomonaco and Maltoni (2017), and Split MiniImageNet Mai et al. (2022). Figure 1: Average End Accuracy on Split-CIFAR-100 Krizhevsky et al. (2009) for 3 buffer sizes (1k-5k-10k) using ER Rolnick et al. (2019) in combination with OCL tricks Mai et al. (2022). Our paper makes the following contributions: 1. We propose the use of Natural Gradient descent with Kronecker Factored Approximate Curvature (NGD-KFAC) as an optimization method in Online Continual Learning (OCL) for image classification. This is a novel application of NGD-KFAC in the OCL domain. 2. Our experiments show that NGD-KFAC significantly improves the performance of OCL models on various OCI and ODI benchmarks. This method enables most existing online continual learning techniques to achieve better results with only a minor change to the optimizer, highlighting its effectiveness in reducing catastrophic forgetting and enhancing model convergence. 2 Related work Continual learning has been studied to address the dynamic nature of real-world data, where the assumption of a static data distribution often does not hold Van de Ven et al. (2022). A considerable portion of this research has been dedicated to mitigating the phenomenon of catastrophic forgetting French (1999); McCloskey and Cohen (1989), a significant challenge when models are sequentially trained on multiple tasks. A comprehensive survey by Mai et al. (2022) Mai et al. (2022) highlights various strategies used in Online Continual Learning (OCL), showing the complexity of training models that are constrained to learn from data streams with limited passes and without access to previous tasks. This survey shows the gap between traditional offline learning and the difficult requirements of OCL scenarios. The use of second-order optimization methods, specifically Natural Gradient Descent (NGD), has shown promise in machine learning for its ability to account for the underlying data distribution’s geometry. Amari (1998) Amari (1998) introduced NGD, leveraging the Fisher Information Matrix (FIM) to guide optimization in a manner that respects the information geometry of parameter space, offering a more principled update path compared to first-order methods like Stochastic Gradient Descent (SGD). Despite its theoretical advantages, the practical application of NGD in deep learning has been limited by computational challenges, particularly in calculating and inverting the FIM for large models. To overcome these challenges, Martens and Grosse (2015) Martens and Grosse (2015) proposed the Kronecker-Factored Approximate Curvature (KFAC) method, an approximation of NGD that reduces the computational overhead by approximating the FIM with a Kronecker product. EKFAC, introduced by George et al. (2018) George et al. (2018), enhances the KFAC approximation by rescaling the Kronecker factors with a diagonal matrix derived from singular value decomposition (SVD), aiming to mitigate some of these challenges. Conversely, the KBFGS Goldfarb et al. (2020) method seeks to approximate the inverse of the Kronecker factors through low-rank Broyden-Fletcher-Goldfarb-Shanno (BFGS) updates, offering a different approach to handling the inversion problem. The TENGraD method, developed by Soori et al. Soori et al. (2021), is accurate, fast, and needs less memory because it inverts blocks of the Fisher Information Matrix (FIM). It does this by efficiently breaking down the FIM and reusing certain calculations. Despite the advancements offered by EKFAC, KBFGS, and TenGrad, our tests showed that KFAC was more numerically stable, especially for our specific needs. We found KFAC to be more reliable and effective in this OCL setting, which is why we preferred it. Works Zenke et al. (2017); Aljundi et al. (2018); Kirkpatrick et al. (2017); Li and Hoiem (2017) have explored the potential of second-order methods and regularization techniques to mitigate catastrophic forgetting. These methods provide a promising direction for improving model robustness and adaptability in dynamic learning environments, but they still fall short of other methods like experience replay Rolnick et al. (2019), especially in the OCL scenarioMai et al. (2022). Additionally, Variational Continual Learning (VCL) Nguyen et al. (2017); Tseran et al. (2018) has been proposed as an improvement of those methods Kirkpatrick et al. (2017); Li and Hoiem (2017). However, to our knowledge, there are few direct comparisons of VCL with other methods in OCL settings, and as such, we do not include VCL in our evaluations. This paper builds upon these works by introducing an application of NGD-KFAC, within the OCL framework. Our approach leverages the strengths of second-order optimization to address the challenges of OCL, such as rapid convergence and efficient learning with data non-stationarity and task shifts. By integrating NGD with KFAC, our aim is to bridge the gap identified by Mai et al. (2022) Mai et al. (2022), enhancing the performance of OCL models on benchmarks such as Split CIFAR-100 Krizhevsky et al. (2009), CORE50 Lomonaco and Maltoni (2017), and Split MiniImageNet Mai et al. (2022). 3 Continual Learning Continual Learning (CL) addresses the challenge of training models on non-stationary data distributions, where the data =⋃t=1TtD= _t=1^TD_t is presented as sequential tasks tD_t, each drawn from a distinct distribution pt(X,y)p_t(X,y). In this setting, the i.i.d. assumption is violated, and training naively on tD_t results in catastrophic forgetting French (1999), where model parameters θ are optimized for tD_t at the cost of degrading performance on 1,…,t−1D_1,…,D_t-1. Mitigating forgetting has been approached using regularization methods that constrain updates to θ based on parameter importance Kirkpatrick et al. (2017), experience replay, which stores subsets of previous data to approximate p(X,y)p(X,y) Rolnick et al. (2019), or architectural growth to isolate task-specific parameters Rusu et al. (2016). However, these methods often assume multiple passes over data or large storage, making them unsuitable for online learning scenarios. 4 Online Continual Learning Online Continual Learning (OCL) extends CL by imposing stricter constraints: each data point is processed only once (T=1T=1) and prior tasks 1:t−1D_1:t-1 are inaccessible. Formally, at each step t, the model minimizes a streaming loss ℒt(θ)=(X,y)∼pt[ℓ(fθ(X),y)]L_t(θ)=E_(X,y) p_t[ (f_θ(X),y)] using small batch updates while adapting to the evolving ptp_t Mai et al. (2022). OCL can be divided into Online Class Incremental (OCI), where new classes are introduced over time, and Online Domain Incremental (ODI), where pt(X,y)p_t(X,y) shifts without introducing new classes Van de Ven et al. (2022). Without task labels or memory buffers, the challenge is to update θ efficiently while minimizing catastrophic forgetting and adapting to non-stationary data streams. This paper explores using Natural Gradient Descent (NGD) with Kronecker-Factored Approximate Curvature (KFAC) to improve convergence and robustness in these constrained settings. 5 Natural Gradient Descent Traditional Stochastic Gradient Descent (SGD) relies solely on the gradient of the loss function ℒL, as defined by the update rule: θt+1=θt−α∇θtℒ(θt|X) _t+1= _t-α _ _tL( _t|X) (1) where α is the learning rate. This method chooses the direction in the θ space so that the loss ℒL decreases the most i.e. highest reduction in loss with a unit change in the parameter θ. However, the Euclidean metric underlying this method may not always provide the most efficient path for updating parameters. Indeed, a slight adjustment in the parameter space can lead to a disproportionate change in the distribution. The conventional gradient descent method fails to account for this by not considering the curvature of the distribution space. To address those shortcomings, NGD proposes stepping in the direction that better respects the geometry of the distribution space Amari (1998); Shrestha (2023). It does so by employing the Fisher Information Matrix (FIM) to adjust the gradient, which accounts for the curvature of the parameter space as informed by the data distribution. Our intuition, for using this optimizer in Online Continual Learning stems from the need to achieve faster convergence, and avoid huge perturbation due to the non i.i.d constraint set on the dataset. The update rule for Natural Gradient Descent can be expressed as: θt+1=θt−α⋅F−1(θt)⋅∇ℒ(θt) _t+1= _t-α· F^-1( _t)· ( _t) (2) where F−1(θt)F^-1( _t) is the inverse of the Fisher Information Matrix (FIM) and ∇ℒ(θt) ( _t) is the gradient of the loss function, where the FIM is defined as follows. Definition 1. The Fisher Information Matrix is defined as: F(θt)=(x,y)∈train[∂ℓ(fθ(x),y)∂θ∂ℓ(fθ(x),y)∂θT]F( _t)=E_(x,y) _train[ ∂ (f_θ(x),y)∂θ ∂ (f_θ(x),y)∂θ^T] (3) where the expectation is taken over targets sampled from the model pθ=fθp_θ=f_θ But as it is formulated, the matrix F has a gigantic size nθ×nθn_θ× n_θ (with nθn_θ the number of parameters) which makes it too large to compute and invert in the context of modern deep neural networks with millions of parameters. That is why research in this domain gravitated towards approximation methods Heskes (2000); Desjardins et al. (2015); Martens and Grosse (2015); Fujimoto and Ohira (2018); Ba et al. (2017) for the Fisher information matrix. 5.1 Kronecker Factored Curvature Approximation Using the exact FIM, the storage required is proportional to the square of the number of parameters and the computation of the inverse of FIM is proportional to its cube. To mitigate this problem, most of the natural gradient methods approximate FIM using a block-diagonal matrix such that the elements corresponding to cross layers are 0 Shrestha (2023). But even with this approximation storage and computation requirements are still high. Martens and Grosse Martens and Grosse (2015) mitigate this by approximating the FIM by the Kronecker product of two small matrices. Considering the parameters θl _l of a layer l of size (n,m)(n,m), h the input of that layer and g=∂ℒ∂θlg= ∂ _l, the KFAC approximation of the FIM part associated to layer l, F(l)F^(l), is approximated as follows Martens and Grosse (2015): F(l)≈A⊗BF^(l)≈ A B (4) where : Ai,i′ A_i,i =[hihi′],∀(i,i′)∈⟦1,n⟧2 =E[h_ih_i ], ∀(i,i )∈ 1,n ^2 (5) Bj,j′ B_j,j =[gjgj′],∀(j,j′)∈⟦1,m⟧2 =E[g_jg_j ], ∀(j,j )∈ 1,m ^2 (6) 5.2 Regularization and numerical stability 2nd order optimization problems can be understood using a more traditional optimization perspective as presented by Martens Martens (2020). The idea is to compute the update δ to θ∈ℝnθ ^n by minimizing a local quadratic approximation (or "model" M) of the loss function centered around the current parameters at this iteration. We thus consider the model M: Definition 2. Mk(δ)=12δTFkδ+∇l(θk)Tδ+ℓ(θk),M_k(δ)= 12δ^TF_kδ+∇ l( _k)^Tδ+ ( _k), (7) where: • MkM_k is the model at iteration k. • Fk∈ℝnθ×nθF_k ^n_θ× n_θ is the curvature matrix which corresponds to the FIM in our case. • ℓ is our objective function, which consists of minimizing the average loss. This is defined by: ℓ(θ)=1||∑(x,y)∈(,)ℒ(y,f(x,θ)) (θ)= 1|X| _(x,y)∈(X,Y)L(y,f(x,θ)) (8) • δ is the update to θ With this definition, the natural gradient descent becomes the minimizer of M which is a convex approximation of the 2nd-order Taylor series of expansion of l(δ+θ)l(δ+θ). We can thus notice that this optimization method succeeds in generating a good local update as long as M(δ)M(δ) is a good local approximation of l(δ+θ)l(δ+θ). This is why Martens (2020); Martens and Grosse (2015) argue that it is necessary to use damping techniques such as the popular Tikhonov regularization Moré (2006) to prevent breakdowns in local quadratic approximations. These breakdowns usually tend to occur at the beginning of training since at first, the model tends to experience an initial "exploration phase" Darken and Moody (1990). This is especially relevant in our case since in OCL, the model is always stuck in this initial exploration phase thus requiring heavy damping. We showcase in section 7 that, in our experiments, using a high damping of 1.01.0 produces the best results across all datasets and techniques. For our experiments, we use the adapted Tikhonov regularization presented by Martens and Grosse (2015). For a layer l∈ℕl with dl∈ℕd_l units we have: F(l)=(Al−1,l−1+πl(λI))⊗(Bl,l+1πl(λI))F^(l)= (A_l-1,l-1+ _l ( λI ) ) (B_l,l+ 1 _l ( λI ) ) (9) Where λ∈ℝ+λ ^+ is our damping parameter and π is defined by: πl=tr(Al−1,l−1)/(dl−1+1)tr(Bl,l)/dl _l= tr(A_l-1,l-1)/(d_l-1+1)tr(B_l,l)/d_l (10) 6 Experimental Setting We consider sets X and Y, representing our datapoints and labels, respectively. We aim to train a model M with parameters θ to accurately classify classes in ⟦1,C⟧ 1,C , where C is a positive integer. This is done in an Online fashion where the model can only iterate for one epoch on the data. 6.1 Datasets We use the same datasets as the OCL survey Mai et al. (2022): 1. Split CIFAR-100: CIFAR100 Krizhevsky et al. (2009) divided into 20 tasks with disjoint classes, each containing 5 classes. 2. Split MiniImageNet: Splits the MiniImageNet Vinyals et al. (2016) dataset with 100 classes, into 20 disjoint tasks as in Chaudhry et al. (2019) containing each 5 classes. 3. NonStationary-MiniImageNet: Proposed by Mai et al. (2022) this dataset uses 3 nonstationary domain incremental scenarios: noise, blur, and occlusion. In our experiment, this is divided into 10 tasks for each type. 4. CORe50-NC: Lomonaco and Maltoni (2017) is a benchmark designed for class incremental learning with 9 tasks and 50 classes: 10 classes in the first task and 5 classes in the subsequent 8 tasks. 5. CORe50-NI: Lomonaco and Maltoni (2017) is a benchmark designed for assessing the domain incremental learning with 8 tasks, where each task has 50 classes with different types of nonstationarity including illumination, background, occlusion, pose and scale. Table 1: Summary of the datasets used. Dataset #Task #Train/task #Class Setting Split CIFAR-100 Krizhevsky et al. (2009) 20 2500 100 OCI Split MiniImageNet Chaudhry et al. (2019) 20 2500 100 OCI NS-MiniImageNet Mai et al. (2022) 10 5000 100 ODI CORE50-NC Lomonaco and Maltoni (2017) 9 12000∼ 24000 50 OCI CORE50-NI Lomonaco and Maltoni (2017) 8 15000 50 ODI For the NonStationary-MiniImageNet dataset Mai et al. (2022), we use the same strengths of nonstationarity as in Mai et al. (2022) to be able to better compare the results. The nonstationarity strength increases as the experiments goes, testing the model’s capability to adapt to noisier domains from the previous learning experience. Here are the values we used: • Noise: [0.0,0.4,0.8,1.2,1.6,2.0,2.4,2.8,3.2,3.6][0.0,0.4,0.8,1.2,1.6,2.0,2.4,2.8,3.2,3.6] • Occlusion: [0.0,0.07,0.13,0.2,0.27,0.33,0.4,0.47,0.53,0.6][0.0,0.07,0.13,0.2,0.27,0.33,0.4,0.47,0.53,0.6] • Blur: [0.0,0.28,0.56,0.83,1.11,1.39,1.67,1.94,2.22,2.5][0.0,0.28,0.56,0.83,1.11,1.39,1.67,1.94,2.22,2.5] 6.2 Models Using a Resnet18 He et al. (2016) we explore the use of different CL techniques in combination with the NGD-KFAC optimizer. We mainly focus on techniques using experience replay Rolnick et al. (2019) as it has been shown Mai et al. (2022); Soutif-Cormerais et al. (2023) that they offer the best performance in this OCL scenario: • ER Rolnick et al. (2019): Experience replay, one of the most popular CL techniques, proposes the use of a buffer to store previously seen experiences and reintegrate them into future training tasks. • A-GEM Hu et al. (2020); Lopez-Paz and Ranzato (2017): Similar to ER, A-GEM uses a buffer to store previously encountered datapoints but adds an optimization step ensuring that the average loss for all past tasks does not increase. • MIR Aljundi et al. (2019a): MIR is another replay technique that focuses on improving the buffer retrieval strategy. Where this is done randomly in ER, MIR chooses replay samples that will be the most affected by the upcoming parameter update, i.e. the samples for which the loss will increase the most after the update. • GSS Aljundi et al. (2019b): Similar to MIR, GSS tries to improve the buffer retrieval strategy by diversifying the gradient directions of the samples stored in the buffer. It does so by calculating a score for each sample in the buffer given by the maximal cosine similarity between the gradient of the sample and the gradients of a random subset of the buffer. We will use this technique only for the domain incremental or new instances settings, as it has been shown in this survey Mai et al. (2022) that this method falls short of the others shown here. • Finetune: Naively trains a model with no CL technique. This serves as a lower bound for our experiments. • Offline: We train the model offline for 70 epochs with a batch size of 128. We run our tests using a Resnet18 He et al. (2016) with a learning rate of 0.10.1, and a fixed batch size of 1010 to mimic a realistic Online Learning Scenario. Although it has been shown that it offers better performance lomonaco2020cvpr, we chose not to use pre-trained weights in our experiments to maintain consistency with the methodologies of the referenced papers. We use SGD without any parameters and add NGD-KFAC with a running average of 0.9 and a damping of 1.0. We do not use ADAM kingma2017adam for comparison because it yields worse results than SGD on this specific case of OCL with or without NGD-KFAC. 6.2.1 OCL tricks Recent works Masana et al. (2022); Hou et al. (2019); Wu et al. (2019); Ahn et al. (2021); Mai et al. (2022) have shown that the Softmax layer and its associated Fully-Connected layer suffer from task-recency bias, where those layers tend to be biased to the last encountered classes. This has prompted the creation of multiple tricks to alleviate this problem: • Labels Trick Zeno et al. (2021): Cross-entropy loss calculation considers only the classes present in the mini-batch, preventing excessive penalization of logits for classes absent from the mini-batch. The loss function is given as: ℒCE(xi,yi)=−logesyi∑j∈CcuresjL_CE(x_i,y_i)=- e^s_y_i _j∈ C_cure^s_j (11) where CcurC_cur denotes the classes in the current mini-batch. • Nearest Class Mean Classifier: Replaces the last biased fully connected classification layer by a nearest mean classifier such as in iCarl Rebuffi et al. (2017). The prototype vector for each class is given by: μy=1|My|∑xm∈Myϕ(xm) _y= 1|M_y| _x_m∈ M_yφ(x_m) (12) and the class label is assigned by: y∗=argminy=1,…,l‖ϕ(x)−μy‖y^*= *arg\,min_y=1,...,l\|φ(x)- _y\| (13) • Separated Softmax Ahn et al. (2021): Since one softmax layer results in a bias explained in Masana et al. (2022); Hou et al. (2019); Wu et al. (2019); Ahn et al. (2021); Mai et al. (2022), this technique employs two Softmax layers one for old classes and one for new classes. Thus training new classes will not overly penalize the old logits. The loss function can be calculated as below: ℒ(xi,yi)=−logesyi∑j∈Coldesj⋅1yi∈Cold−logesyi∑j∈Cnewesj⋅1yi∈Cnew splitL(x_i,y_i)=- e^s_y_i _j∈ C_olde^s_j· 1\y_i∈ C_old\-\\ e^s_y_i _j∈ C_newe^s_j· 1\y_i∈ C_new\ split (14) • Review trick Castro et al. (2018): Adds an additional fine-tuning step using a balanced subset of the memory buffer. 6.3 Metrics We evaluate our models using two different popular metrics: • Average accuracy: Average Accuracy is defined as Ai=1i∑j=1iai,jA_i= 1i _j=1^ia_i,j (15) where i=Ti=T, ATA_T represents the average accuracy by the end of training with the whole data sequence. • Average forgetting: Average Forgetting at task i is defined as Fi=1i−1∑j=1i−1fi,jF_i= 1i-1 _j=1^i-1f_i,j (16) where fk,j=maxℓ∈1,…,k−1(aℓ,j)−ak,j,∀j<kf_k,j= _ ∈\1,…,k-1\(a_ ,j)-a_k,j,\ ∀ j<k. fi,jf_i,j represents how much the model has forgot about task j after being trained on task i. Specifically, maxℓ∈1,…,k−1(aℓ,j) _ ∈\1,…,k-1\(a_ ,j) denotes the best test accuracy the model has ever achieved on task j before learning task k, and ak,ja_k,j is the test accuracy on task j after learning task k. We run every experiment 10 times and provide the mean and standard deviation for each experiment. For that, we use the same sequence of seeds for every model. In our case, the seed not only influences the initialisation parameters and the overall training of our model but also the order and organization of the datasets especially in the class incremental scenario, 7 Results & Discussion We notice an improvement in accuracy on most methods when we use NGD-KFAC. The results are provided in tables 2 to 5 and further discussed in this section for Online Class Incremental (OCI) learning and Online Domain Incremental (ODI) learning. Table 2: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on Split CIFAR-100. Finetune 3.7±0.33.7± 0.3 Offline 49.7±2.649.7± 2.6 Trick NGD-KFAC A-GEM ER MIR M=1k M=5k M=10k M=1k M=5k M=10k M=1k M=5k M=10k N/A × 5.6±0.45.6± 0.4 5.8±0.45.8± 0.4 5.6±0.45.6± 0.4 11.1±0.711.1± 0.7 21.4±1.221.4± 1.2 21.8±1.021.8± 1.0 11.5±0.6 11.5± 0.6 21.8±0.821.8± 0.8 24.7±1.324.7± 1.3 6.7±0.4 6.7± 0.4 6.9±0.2 6.9± 0.2 6.9±0.3 6.9± 0.3 11.4±0.4 11.4± 0.4 22.9±1.1 22.9± 1.1 23.9±0.8 23.9± 0.8 11.2±0.711.2± 0.7 23.1±0.7 23.1± 0.7 25.9±0.8 25.9± 0.8 LB Zeno et al. (2021) × 8.5±0.58.5± 0.5 8.5±0.68.5± 0.6 8.1±0.68.1± 0.6 17.0±0.917.0± 0.9 21.2±1.021.2± 1.0 22.5±0.722.5± 0.7 17.6±0.617.6± 0.6 22.4±0.922.4± 0.9 22.9±0.922.9± 0.9 10.7±0.6 10.7± 0.6 11.3±0.7 11.3± 0.7 10.4±0.8 10.4± 0.8 19.4±1.1 19.4± 1.1 25.6±0.6 25.6± 0.6 27.3±0.5 27.3± 0.5 19.2±0.7 19.2± 0.7 25.2±0.7 25.2± 0.7 27.2±0.7 27.2± 0.7 S Ahn et al. (2021) × 8.3±0.58.3± 0.5 8.7±0.58.7± 0.5 8.6±0.48.6± 0.4 16.9±0.816.9± 0.8 23.8±1.223.8± 1.2 25.5±1.125.5± 1.1 17.4±0.817.4± 0.8 25.0±0.925.0± 0.9 26.8±1.126.8± 1.1 11.9±0.9 11.9± 0.9 12.4±0.7 12.4± 0.7 11.9±0.7 11.9± 0.7 20.3±0.5 20.3± 0.5 28.8±0.7 28.8± 0.7 33.0±0.6 33.0± 0.6 20.3±0.6 20.3± 0.6 28.4±0.8 28.4± 0.8 32.6±0.8 32.6± 0.8 RV Castro et al. (2018) × 6.6±0.46.6± 0.4 15.7±1.5 15.7± 1.5 24.2±1.3 24.2± 1.3 14.6±0.4 14.6± 0.4 31.2±1.031.2± 1.0 36.8±1.136.8± 1.1 13.2±0.6 13.2± 0.6 30.8±0.830.8± 0.8 38.1±1.238.1± 1.2 6.9±0.3 6.9± 0.3 10.7±0.710.7± 0.7 22.4±1.022.4± 1.0 13.7±0.513.7± 0.5 34.2±0.3 34.2± 0.3 39.9±0.5 39.9± 0.5 12.3±0.512.3± 0.5 31.7±0.7 31.7± 0.7 39.6±0.5 39.6± 0.5 NCM Rebuffi et al. (2017) × 11.6±0.611.6± 0.6 13.8±0.713.8± 0.7 14.6±0.614.6± 0.6 16.9±0.616.9± 0.6 28.1±1.028.1± 1.0 30.1±1.030.1± 1.0 16.6±0.516.6± 0.5 27.6±0.727.6± 0.7 31.1±1.031.1± 1.0 15.3±0.7 15.3± 0.7 18.3±0.6 18.3± 0.6 18.7±0.7 18.7± 0.7 19.3±0.8 19.3± 0.8 31.9±0.9 31.9± 0.9 34.3±0.7 34.3± 0.7 17.7±0.7 17.7± 0.7 30.6±0.6 30.6± 0.6 35.0±0.6 35.0± 0.6 Table 3: Average End Accuracy for NonStationary-MiniImageNet (Noise, Occlusion, Blur) and CORe50-NI in the ODI setting. Method NGD-KFAC Mini-ImageNet-Noise Mini-ImageNet-Occlusion Mini-ImageNet-Blur CORe50-NI Finetune × 11.1±1.011.1± 1.0 13.8±1.613.8± 1.6 2.4±0.22.4± 0.2 14.0±2.814.0± 2.8 Offline × 37.3±0.837.3± 0.8 38.6±4.738.6± 4.7 11.9±1.011.9± 1.0 51.7±1.851.7± 1.8 Buffer Size M=1k M=5k M=10k M=1k M=5k M=10k M=1k M=5k M=10k M=1k M=5k M=10k ER × 19.4 ± 1.3 21.6 ± 1.1 24.3 ± 1.2 19.2 ± 1.5 23.4 ± 1.4 23.7 ± 1.1 5.3 ± 0.6 8.6 ± 0.8 9.4 ± 0.7 24.1 ± 4.2 28.3 ± 3.5 30.0 ± 2.8 21.0 ± 0.9 25.1 ± 0.9 26.7 ± 0.6 21.0 ± 0.9 25.1 ± 0.9 26.7 ± 0.6 5.3 ± 0.5 7.9 ± 0.7 10.0 ± 0.7 30.5 ± 1.0 32.5 ± 2.2 33.8 ± 1.6 MIR × 18.1 ± 1.1 22.5 ± 1.4 24.4 ± 0.9 17.6 ± 0.7 22.0 ± 1.1 23.8 ± 1.2 5.5 ± 0.5 8.1 ± 0.6 9.6 ± 1.0 26.5 ± 1.0 34.0 ± 1.0 33.3 ± 1.7 19.4 ± 1.0 22.4 ± 1.1 25.6 ± 1.1 21.5 ± 1.0 23.7 ± 1.1 26.2 ± 1.0 5.6 ± 0.6 7.6 ± 0.5 8.6 ± 0.5 30.3 ± 1.9 35.2 ± 1.4 36.8 ± 1.3 GSS × 18.9 ± 0.8 21.4 ± 0.9 23.2 ± 1.1 17.7 ± 0.8 21.0 ± 2.2 23.2 ± 1.4 5.2 ± 0.5 7.6 ± 0.6 8.0 ± 0.6 25.5 ± 2.1 27.2 ± 2.0 25.3 ± 2.1 20.9 ± 0.7 24.4 ± 0.9 24.9 ± 1.0 21.2 ± 0.9 24.4 ± 0.8 26.8 ± 0.9 5.6 ± 0.7 7.2 ± 0.7 7.8 ± 0.7 29.0 ± 1.6 29.5 ± 2.4 29.3 ± 1.8 A-GEM × 14.0 ± 1.3 14.6 ± 0.7 14.2 ± 1.4 16.4 ± 0.7 13.9 ± 2.6 14.4 ± 2.0 4.4 ± 0.4 4.4 ± 0.4 4.3 ± 0.5 12.4 ± 1.1 13.8 ± 1.2 15.0 ± 2.2 17.8 ± 0.6 18.5 ± 0.7 18.3 ± 0.6 19.2 ± 1.3 19.2 ± 0.9 19.6 ± 1.4 4.7 ± 0.4 4.8 ± 0.3 4.4 ± 0.5 19.4 ± 2.8 19.4 ± 2.2 19.3 ± 2.6 Table 4: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on Split MiniImageNet. Finetune 3.4±0.23.4± 0.2 Offline 51.9±0.551.9± 0.5 Trick NGD-KFAC A-GEM ER MIR M=1k M=5k M=10k M=1k M=5k M=10k M=1k M=5k M=10k N/A × 4.5±0.64.5± 0.6 5.0±0.45.0± 0.4 4.9±0.24.9± 0.2 10.9±0.9 10.9± 0.9 17.5±1.317.5± 1.3 17.1±1.817.1± 1.8 10.6±0.5 10.6± 0.5 18.4±1.9 18.4± 1.9 17.9±3.3 17.9± 3.3 5.8±0.4 5.8± 0.4 6.2±0.2 6.2± 0.2 6.2±0.4 6.2± 0.4 9.7±0.69.7± 0.6 18.2±1.2 18.2± 1.2 17.9±0.9 17.9± 0.9 8.9±0.78.9± 0.7 16.2±0.616.2± 0.6 16.6±1.316.6± 1.3 LB Zeno et al. (2021) × 9.9±0.99.9± 0.9 9.9±0.69.9± 0.6 9.9±0.69.9± 0.6 17.4±0.817.4± 0.8 18.6±2.218.6± 2.2 18.3±3.118.3± 3.1 17.9±1.317.9± 1.3 20.3±1.120.3± 1.1 21.9±1.221.9± 1.2 12.2±0.6 12.2± 0.6 12.2±0.5 12.2± 0.5 12.2±1.0 12.2± 1.0 20.5±0.8 20.5± 0.8 24.8±1.7 24.8± 1.7 27.1±0.9 27.1± 0.9 19.5±1.0 19.5± 1.0 25.4±0.6 25.4± 0.6 26.4±1.2 26.4± 1.2 S Ahn et al. (2021) × 11.2±0.511.2± 0.5 10.6±0.910.6± 0.9 10.2±0.910.2± 0.9 16.9±1.116.9± 1.1 20.7±2.320.7± 2.3 20.9±3.220.9± 3.2 18.5±0.518.5± 0.5 22.1±1.522.1± 1.5 21.8±1.521.8± 1.5 13.8±1.1 13.8± 1.1 13.8±0.9 13.8± 0.9 13.7±0.6 13.7± 0.6 21.0±0.9 21.0± 0.9 28.0±0.8 28.0± 0.8 30.8±0.8 30.8± 0.8 20.8±0.9 20.8± 0.9 26.9±1.2 26.9± 1.2 29.1±0.5 29.1± 0.5 RV Castro et al. (2018) × 8.5±0.7 8.5± 0.7 18.6±1.3 18.6± 1.3 23.7±1.0 23.7± 1.0 14.6±0.8 14.6± 0.8 29.3±1.329.3± 1.3 31.5±1.031.5± 1.0 12.5±0.6 12.5± 0.6 28.8±1.5 28.8± 1.5 32.7±1.432.7± 1.4 6.6±0.66.6± 0.6 13.6±1.013.6± 1.0 23.3±0.923.3± 0.9 11.5±0.611.5± 0.6 30.1±0.4 30.1± 0.4 33.9±0.6 33.9± 0.6 10.2±0.510.2± 0.5 27.4±0.927.4± 0.9 34.4±0.8 34.4± 0.8 NCM Rebuffi et al. (2017) × 9.9±0.49.9± 0.4 13.0±0.713.0± 0.7 13.4±0.513.4± 0.5 17.6±0.817.6± 0.8 22.7±1.322.7± 1.3 22.6±1.722.6± 1.7 17.6±0.417.6± 0.4 23.2±1.923.2± 1.9 23.7±2.223.7± 2.2 14.3±0.6 14.3± 0.6 18.4±0.7 18.4± 0.7 18.7±0.7 18.7± 0.7 18.4±0.6 18.4± 0.6 28.3±0.6 28.3± 0.6 29.5±0.7 29.5± 0.7 18.0±0.7 18.0± 0.7 26.2±0.7 26.2± 0.7 28.5±0.5 28.5± 0.5 Table 5: End Average Accuracy of methods and tricks with or without NGD-KFAC for the OCI setting on CORe50-NC. Finetune 7.7±1.07.7± 1.0 Offline 51.7±1.851.7± 1.8 Trick NGD-KFAC A-GEM ER MIR M=1k M=5k M=10k M=1k M=5k M=10k M=1k M=5k M=10k N/A × 9.2±0.89.2± 0.8 8.9±0.98.9± 0.9 8.8±1.28.8± 1.2 23.4±1.523.4± 1.5 27.2±1.827.2± 1.8 28.9±1.228.9± 1.2 24.2±2.024.2± 2.0 31.3±1.231.3± 1.2 31.9±1.431.9± 1.4 9.8±1.2 9.8± 1.2 9.4±1.0 9.4± 1.0 9.4±0.8 9.4± 0.8 24.4±0.8 24.4± 0.8 30.8±1.0 30.8± 1.0 32.2±1.4 32.2± 1.4 27.0±1.0 27.0± 1.0 34.2±1.0 34.2± 1.0 35.5±1.5 35.5± 1.5 LB Zeno et al. (2021) × 13.4±1.113.4± 1.1 13.4±1.013.4± 1.0 13.6±1.5 13.6± 1.5 22.4±1.222.4± 1.2 25.5±0.925.5± 0.9 25.6±1.825.6± 1.8 22.5±1.422.5± 1.4 25.7±1.225.7± 1.2 25.5±1.625.5± 1.6 14.2±1.2 14.2± 1.2 14.8±0.6 14.8± 0.6 13.5±0.713.5± 0.7 25.4±1.1 25.4± 1.1 27.8±1.1 27.8± 1.1 29.1±1.1 29.1± 1.1 26.4±1.2 26.4± 1.2 28.2±1.1 28.2± 1.1 28.8±1.4 28.8± 1.4 S Ahn et al. (2021) × 13.9±1.113.9± 1.1 14.9±1.514.9± 1.5 14.9±1.6 14.9± 1.6 22.4±1.222.4± 1.2 25.3±1.025.3± 1.0 25.2±1.625.2± 1.6 23.0±0.823.0± 0.8 26.6±0.826.6± 0.8 27.8±1.427.8± 1.4 15.4±1.1 15.4± 1.1 15.0±0.7 15.0± 0.7 14.5±1.214.5± 1.2 26.1±1.2 26.1± 1.2 30.5±1.5 30.5± 1.5 30.7±1.6 30.7± 1.6 26.8±1.5 26.8± 1.5 31.3±1.6 31.3± 1.6 31.9±1.7 31.9± 1.7 RV Castro et al. (2018) × 20.8±1.5 20.8± 1.5 24.1±0.824.1± 0.8 26.9±0.926.9± 0.9 24.4±1.124.4± 1.1 30.3±1.330.3± 1.3 32.4±1.132.4± 1.1 25.1±1.425.1± 1.4 32.3±0.932.3± 0.9 34.4±1.334.4± 1.3 15.1±1.215.1± 1.2 24.2±1.1 24.2± 1.1 30.4±0.7 30.4± 0.7 25.8±1.1 25.8± 1.1 35.1±1.7 35.1± 1.7 37.7±1.9 37.7± 1.9 27.0±1.1 27.0± 1.1 36.7±1.1 36.7± 1.1 38.2±1.1 38.2± 1.1 NCM Rebuffi et al. (2017) × 17.4±0.917.4± 0.9 18.9±1.018.9± 1.0 19.8±1.119.8± 1.1 21.7±1.321.7± 1.3 25.1±1.825.1± 1.8 26.2±1.226.2± 1.2 22.8±1.122.8± 1.1 28.6±1.128.6± 1.1 29.4±1.429.4± 1.4 20.4±1.5 20.4± 1.5 21.9±1.1 21.9± 1.1 21.2±1.3 21.2± 1.3 28.5±1.3 28.5± 1.3 34.8±2.1 34.8± 2.1 35.3±2.4 35.3± 2.4 29.5±1.6 29.5± 1.6 34.9±2.0 34.9± 2.0 35.5±2.4 35.5± 2.4 7.1 Class incremental 7.1.1 Overall Effect of NGD-KFAC • Without NGD-KFAC: The models achieved an average end accuracy of 19.58%19.58\% and an average forgetting of 18.05%18.05\%. • With NGD-KFAC: The models’ performance improved, with an average end accuracy of 22.20%22.20\% and an average forgetting of 24.58%24.58\%. This shows the benefit of using NGD-KFAC optimization in OCL, providing an overall increase in model performance by 2.62%2.62\%. However, while NGD-KFAC enhances performance, it slightly increases the tendency of models to forget previously learned information. 7.1.2 Effect of Memory Size • Memory Size 1000: Models without NGD-KFAC had an average end accuracy of 15.52%15.52\% and forgetting of 23.72%23.72\%, whereas those with NGD-KFAC optimization reached 17.09%17.09\% in accuracy and 30.05%30.05\% in forgetting. • Memory Size 5000: Performance increased significantly with memory size, with models achieving 20.95%20.95\% in accuracy and 15.74%15.74\% in forgetting without NGD-KFAC, and 23.71%23.71\% in accuracy and 22.78%22.78\% in forgetting with NGD-KFAC. • Memory Size 10000: The largest memory size showed the highest performance, with models achieving 22.22%22.22\% in accuracy and 14.70%14.70\% in forgetting without NGD-KFAC, and 25.80%25.80\% in accuracy and 20.44%20.44\% in forgetting with NGD-KFAC. The results show the importance of memory size in OCL, with larger buffers allowing for better performance. Additionally, the benefit of NGD-KFAC is consistent across different memory sizes, showing larger gains with increased buffer size. 7.1.3 Effect of OCL Tricks The analysis of different OCL tricks reveals varied improvements in performance when combined with NGD-KFAC: • Labels Trick: Improved from 17.77%17.77\% to 20.92%20.92\% in accuracy with NGD-KFAC, with forgetting increasing from 14.74%14.74\% to 18.07%18.07\%. • Nearest Class Mean (NCM) Trick: Showed a notable increase from 20.89%20.89\% to 25.31%25.31\% in accuracy with NGD-KFAC, with forgetting increasing from 11.49%11.49\% to 12.54%12.54\%. • Review Trick: Had a high baseline performance of 24.52%24.52\% which remained almost the same at 24.91%24.91\% with NGD-KFAC, but forgetting significantly increased from 17.24%17.24\% to 31.78%31.78\%. • Separated Softmax: Saw an improvement from 18.81%18.81\% to 22.95%22.95\% in accuracy with NGD-KFAC, with a modest increase in forgetting from 13.75%13.75\% to 16.72%16.72\%. • No Trick: Here the accuracy went from 15.92%15.92\% without NGD-KFAC to 16.94%16.94\% with NGD-KFAC, but with a substantial increase in forgetting from 33.08%33.08\% to 43.83%43.83\%. These results suggest that NGD-KFAC significantly enhances the effectiveness of various OCL tricks in improving model accuracy, with varied impacts on forgetting. The Nearest Class Mean (NCM) trick, combined with NGD-KFAC is particularly effective, significantly improving performance while reducing forgetting. It is interesting to notice that although the forgetting seems to increase with the use of NGD-KFAC, the accuracy increases with it. This increase in accuracy can be attributed to the faster convergence facilitated by NGD-KFAC in each learning task. As a result, although the initial decrease in performance when switching to a new task is greater compared to using SGD, the overall performance is still better. This might indicate that, with more robust OCL architectures, NGD-KFAC may be able to achieve better performance. We leave further exploration of this hypothesis to future research. Figure 2: Average End Accuracy by trick, averaged on all datasets in the OCI setting with regards to the damping parameter ϵε. 7.2 Domain incremental In the domain incremental setting (ODI), the benefit of NGD-KFAC is also clearly visible for most methods and datasets. For example, for experience replay with a memory buffer of 10000, it improves the accuracy from 24.3%24.3\% to 26.7%26.7\% in Mini-ImageNet with Noise, from 13.7%13.7\% to 26.7%26.7\% in Mini-ImageNet with occlusion, from 9.4%9.4\% to 10%10\% in Mini-ImageNet with Blur and from 30%30\% to 33.8%33.8\% in CORe50-NI. 7.3 Regularization The optimization is quite sensitive to the dampening coefficient ϵε of the regularisation. We show, in figure 2, the average end accuracy over all datasets in the class incremental setting. Higher dampening coefficients lead to higher accuracy for all the tricks implemented. 8 Conlusion In this paper, we proposed and explored the novel application of Natural Gradient Descent with Kronecker Factored Approximate Curvature (NGD-KFAC) as an optimization method in the domain of Online Continual Learning (OCL) for image classification. Our work demonstrates the advantage of using second-order optimization techniques, specifically NGD-KFAC, to address the challenges presented by OCL, such as catastrophic forgetting and the requirement for rapid convergence despite data non-stationarity and task shifts. Through our experiments across several datasets and OCL scenarios, we established that NGD-KFAC not only improves the performance of OCL models but also enhances the effectiveness of existing continual learning tricks. Our experiments across various benchmarks, including Split CIFAR-100, CORE50, and Split MiniImageNet, indicate that integrating NGD-KFAC with traditional OCL methods boosts model performance. Specifically, we observed in some cases, improvements in both average accuracy and reduction in average forgetting, showing NGD-KFAC’s potential to mitigate the effects of catastrophic forgetting more effectively than traditional first-order optimization approaches. This improvement is due to NGD-KFAC’s ability to account for the curvature of the parameter space informed by the data distribution, thereby offering a more principled and efficient update path. In conclusion, our findings suggest that second-order optimization methods, particularly the Natural Gradient Descent, hold promise for enhancing the performance and robustness of models in dynamic and evolving learning environments. References [1] H. Ahn, J. Kwak, S. Lim, H. Bang, H. Kim, and T. Moon (2021) Ss-il: separated softmax for incremental learning. In Proceedings of the IEEE/CVF International conference on computer vision, p. 844–853. Cited by: 3rd item, §6.2.1, Table 2, Table 4, Table 5. [2] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars (2018) Memory aware synapses: learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), p. 139–154. Cited by: §2. [3] R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, and L. Page-Caccia (2019) Online continual learning with maximal interfered retrieval. Advances in neural information processing systems 32. Cited by: 3rd item. [4] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio (2019) Gradient based sample selection for online continual learning. Advances in neural information processing systems 32. Cited by: 4th item. [5] S. Amari (1998) Natural gradient works efficiently in learning. Neural computation 10 (2), p. 251–276. Cited by: §1, §2, §5. [6] J. Ba, R. Grosse, and J. Martens (2017) Distributed second-order optimization using kronecker-factored approximations. In International conference on learning representations, Cited by: §5. [7] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari (2018) End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), p. 233–248. Cited by: 4th item, Table 2, Table 4, Table 5. [8] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato (2019) On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486. Cited by: item 2, Table 1. [9] C. Darken and J. Moody (1990) Note on learning rate schedules for stochastic optimization. Advances in neural information processing systems 3. Cited by: §5.2. [10] G. Desjardins, K. Simonyan, R. Pascanu, et al. (2015) Natural neural networks. Advances in neural information processing systems 28. Cited by: §5. [11] R. M. French (1999) Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3 (4), p. 128–135. Cited by: §1, §2, §3. [12] Y. Fujimoto and T. Ohira (2018) A neural network model with bidirectional whitening. In Artificial Intelligence and Soft Computing: 17th International Conference, ICAISC 2018, Zakopane, Poland, June 3-7, 2018, Proceedings, Part I 17, p. 47–57. Cited by: §5. [13] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent (2018) Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in neural information processing systems 31. Cited by: §2. [14] D. Goldfarb, Y. Ren, and A. Bahamou (2020) Practical quasi-newton methods for training deep neural networks. Advances in Neural Information Processing Systems 33, p. 2386–2396. Cited by: §2. [15] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778. Cited by: §6.2, §6.2. [16] T. Heskes (2000) On “natural” learning and pruning in multilayered perceptrons. Neural Computation 12 (4), p. 881–901. Cited by: §5. [17] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin (2019) Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 831–839. Cited by: 3rd item, §6.2.1. [18] G. Hu, W. Zhang, H. Ding, and W. Zhu (2020) Gradient episodic memory with a soft constraint for continual learning. arXiv preprint arXiv:2011.07801. Cited by: 2nd item. [19] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114 (13), p. 3521–3526. Cited by: §2, §3. [20] A. Krizhevsky, G. Hinton, et al. (2009) Learning multiple layers of features from tiny images. Cited by: Figure 1, §1, §2, item 1, Table 1. [21] D. Kudithipudi, M. Aguilar-Simon, J. Babb, M. Bazhenov, D. Blackiston, J. Bongard, A. P. Brna, S. Chakravarthi Raja, N. Cheney, J. Clune, et al. (2022) Biological underpinnings for lifelong learning machines. Nature Machine Intelligence 4 (3), p. 196–210. Cited by: §1. [22] Z. Li and D. Hoiem (2017) Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40 (12), p. 2935–2947. Cited by: §2. [23] V. Lomonaco and D. Maltoni (2017) Core50: a new dataset and benchmark for continuous object recognition. In Conference on robot learning, p. 17–26. Cited by: §1, §2, item 4, item 5, Table 1, Table 1. [24] D. Lopez-Paz and M. Ranzato (2017) Gradient episodic memory for continual learning. Advances in neural information processing systems 30. Cited by: 2nd item. [25] Z. Mai, R. Li, J. Jeong, D. Quispe, H. Kim, and S. Sanner (2022) Online continual learning in image classification: an empirical survey. Neurocomputing 469, p. 28–51. Cited by: Figure 1, §1, §1, §2, §2, §2, §4, item 3, 4th item, 3rd item, §6.1, §6.1, §6.2.1, §6.2, Table 1. [26] J. Martens and R. Grosse (2015) Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, p. 2408–2417. Cited by: §1, §2, §5.1, §5.1, §5.2, §5.2, §5. [27] J. Martens (2020) New insights and perspectives on the natural gradient method. Journal of Machine Learning Research 21 (146), p. 1–76. Cited by: §5.2, §5.2. [28] M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. Van De Weijer (2022) Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (5), p. 5513–5533. Cited by: 3rd item, §6.2.1. [29] M. McCloskey and N. J. Cohen (1989) Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of learning and motivation, Vol. 24, p. 109–165. Cited by: §1, §2. [30] J. J. Moré (2006) The levenberg-marquardt algorithm: implementation and theory. In Numerical analysis: proceedings of the biennial Conference held at Dundee, June 28–July 1, 1977, p. 105–116. Cited by: §5.2. [31] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner (2017) Variational continual learning. arXiv preprint arXiv:1710.10628. Cited by: §2. [32] S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert (2017) Icarl: incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, p. 2001–2010. Cited by: 2nd item, Table 2, Table 4, Table 5. [33] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne (2019) Experience replay for continual learning. Advances in neural information processing systems 32. Cited by: Figure 1, §2, §3, 1st item, §6.2. [34] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell (2016) Progressive neural networks. arXiv preprint arXiv:1606.04671. Cited by: §3. [35] R. Shrestha (2023) Natural gradient methods: perspectives, efficient-scalable approximations, and analysis. arXiv preprint arXiv:2303.05473. Cited by: §5.1, §5. [36] S. Soori, B. Can, B. Mu, M. Gürbüzbalaban, and M. M. Dehnavi (2021) TENGraD: time-efficient natural gradient descent with exact fisher-block inversion. arXiv preprint arXiv:2106.03947. Cited by: §2. [37] A. Soutif-Cormerais, A. Carta, A. Cossu, J. Hurtado, V. Lomonaco, J. Van de Weijer, and H. Hemati (2023) A comprehensive empirical evaluation on online continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3518–3528. Cited by: §1, §6.2. [38] H. Tseran, M. E. Khan, T. Harada, and T. D. Bui (2018) Natural variational continual learning. In Continual Learning Workshop@ NeurIPS, Vol. 2. Cited by: §2. [39] G. M. Van de Ven, T. Tuytelaars, and A. S. Tolias (2022) Three types of incremental learning. Nature Machine Intelligence 4 (12), p. 1185–1197. Cited by: §1, §2, §4. [40] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. (2016) Matching networks for one shot learning. Advances in neural information processing systems 29. Cited by: item 2. [41] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu (2019) Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 374–382. Cited by: 3rd item, §6.2.1. [42] F. Zenke, B. Poole, and S. Ganguli (2017) Continual learning through synaptic intelligence. In International conference on machine learning, p. 3987–3995. Cited by: §2. [43] C. Zeno, I. Golan, E. Hoffer, and D. Soudry (2021) Task-agnostic continual learning using online variational bayes with fixed-point updates. Neural Computation 33 (11), p. 3139–3177. Cited by: 1st item, Table 2, Table 4, Table 5.