Paper deep dive
Forgetting, plasticity, and co-observation: a third facet of continual learning
Timm Hess, Abhishek Jha, Gido M. van de Ven, Tinne Tuytelaars
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/20/2026, 5:09:10 AM
Summary
This paper identifies 'data co-observation' as a third critical factor in continual learning, distinct from catastrophic forgetting and loss of plasticity. The authors demonstrate that simultaneous observation of training data (joint training) provides representational benefits and improved generalization that cannot be achieved merely by retaining past knowledge (ensemble) or managing stability/plasticity. Using a diagnostic methodology involving linear probing and ensembles on 'chunking' scenarios, they show that memory replay methods succeed not just by preventing forgetting but by restoring co-observation benefits.
Entities (11)
Relation Signals (8)
Joint Training → enables → Co-observation
confidence 95% · simultaneous observation of training data (co-observation) yields benefits... joint training... allows this transfer of knowledge.
Co-observation → isdistinctfrom → Catastrophic Forgetting
confidence 95% · we highlight data co-observation as a distinct factor influencing continual learning performance... these two issues [forgetting/plasticity] cannot fully explain the performance gap
Co-observation → isdistinctfrom → Loss of Plasticity
confidence 95% · we highlight data co-observation as a distinct factor... decoupling the constraints of separate data access from stability and plasticity
Ensemble Baseline → mitigates → Catastrophic Forgetting
confidence 95% · The ensemble preserves the full representational state of the model after each sequential training phase, thereby avoiding forgetting by design.
Linear Probing → usedfor → Evaluating Representation Quality
confidence 93% · we evaluate continual learning at representation level via linear probing... decouples the model’s generalization from the confounding dynamics
Knowledge Distillation → actsas → Knowledge Retention Mechanism
confidence 92% · distillation-based approaches act only as effective knowledge retention mechanisms
Ensemble Baseline → lacks → Co-observation
confidence 90% · To eliminate forgetting... without explicitly breaking the separate observability constraint... we employ an ensemble approach.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.
Tags
Links
- Source: https://arxiv.org/abs/2608.18803v1
- Canonical: https://arxiv.org/abs/2608.18803v1
Trouble viewing inline? Open PDF directly →
Full Text
83,843 characters extracted from source content.
Expand or collapse full text
Forgetting, plasticity, and co-observation: a third facet of continual learning Timm Hess VIB.AI KU Leuven, BelgiumGido M. van de Ven Affiliation: ESAT KU Leuven, BelgiumAbhishek Jha Affiliation: ESAT KU Leuven, Belgium Bernoulli Institute University of Groningen, NetherlandsTinne Tuytelaars Abstract Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training. In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together. Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental “chunking” scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner’s generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift. Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process. †footnotetext: † timmfelix.hess, abhishek.jha, tinne.tuytelaars@kuleuven.be; ‡ .m.van.de.ven@rug.nl 1 Introduction Deep neural networks are predominantly trained under a joint training paradigm, where the model is optimized on independent and identically distributed (i.i.d.) batches drawn uniformly from a static dataset. In many real-world settings, however, it would be transformative if this paradigm need not be maintained. When new data arrives sequentially over time, retraining on all previously observed data becomes expensive or sometimes infeasible due to, e.g. computational, privacy, or storage constraints (47). The model could instead be sequentially trained on separate partitions of data as the data arrives. This process of incrementally updating a model from a non-stationary data stream is referred to as continual learning (CL) or lifelong learning (11; 38). A fundamental challenge in this regime stems from what is known as the stability-plasticity dilemma (19; 34). On the one hand, an effective continual learning system requires sufficient plasticity to adapt its representation based on new data. On the other hand, updating the model exclusively on new observations disrupts existing parameter configurations thereby severely impairing its stability, a.k.a. catastrophic forgetting (33; 38). Consequently, a large part of the continual learning literature focuses on algorithmic interventions designed to retain prior knowledge while remaining capable of incorporating new information (14; 50). Progress on these two desiderata is typically assessed relative to a pair of canonical reference points: a lower target, established by naive sequential training, typically interpreted as exposing the severity of forgetting in the absence of any intervention; and an upper target, established by (offline) joint training, interpreted as reflecting the maximum achievable performance on the full dataset. This evaluation framework forwards the implicit assumption that perfectly managing stability and plasticity during sequential training would close the gap to joint training. In our work, we question this assumption. We hypothesize that, under a sequential training regime, even a model with perfect stability and plasticity would still underperform a jointly trained model in terms of its generalization properties. This intuition can be linked to multi-task learning theory (9), which posits that simultaneous training provides a mutually beneficial inductive bias. Our work empirically investigates this dynamic in continual learning, and we demonstrate a benefit of simultaneous training even in identically distributed, data-incremental setups. To put this another way, we argue that the challenge of sequential training can be split into two parts: the temporal aspect, which underlies forgetting and plasticity, and the challenge of separate training, whereby cross-partition dependencies remain hidden. We refer to the influence of this phenomenon as the co-observation effect. An analogy, illustrated in Figure 1, provides a simple intuition for the challenge of separate training: assembling a puzzle is easier when all pieces are visible simultaneously, because overarching patterns are more readily recognized when the pieces that jointly encode them are co-observed. To empirically support our hypothesis, we report our results on data-incremental “chunking” scenarios (29), which avoid macroscopic distribution shifts, arguably making it one of the hardest settings to show this effect. Furthermore, we show that this co-observation effect persists across both supervised and self-supervised learning paradigms. We design a diagnostic methodology based on continual pre-training (22; 57) and model ensembles (49) to isolate the effect of co-observation as much as possible. Inspired by 21, we evaluate continual learning at representation level via linear probing (2), which decouples the model’s generalization from the confounding dynamics of a continually updated classifier head. Moreover, we employ an ensemble to control for forgetting. The ensemble preserves the full representational state of the model after each sequential training phase, thereby avoiding forgetting by design. Our results show that in the “chunking” setting, without distribution shift and class imbalance as additional confounds, the ensemble-baseline is consistently surpassed by incremental joint training. Because the impact of forgetting is controlled for, we attribute this residual difference in results to the absence of co-observation during optimization. Ultimately, our findings point towards an important realization for the conceptualization of continual learning mechanisms. Maintaining stability and plasticity is the fundamental requirement in continual learning. To approach the generalization capabilities of joint training, continual learning methods must also acknowledge and compensate for the representational deficit caused by the loss of co-observation in separate training. We summarize our contributions as follows: Figure 1: Intuitive illustration of the generalization properties of separate training in comparison to joint training. Finding a common structure across the presented information, here depicted as puzzle pieces, is easier when observing the information together, as in the joint training case (right). In contrast, we can think of separate training as placing information piece-by-piece while focusing on one piece of information at a time (left). Here, local fits emerge that struggle to map relationships across multiple pieces of information. • Conceptualization of the co-observation effect: We distinguish the impact of separate training from stability-plasticity, demonstrating co-observation as a distinct mechanism when it comes to explaining the gap between continual learning and joint learning. • Diagnostic methodology for separating forgetting from co-observation:† Code: https://github.com/TimmHess/Co-ob. We design an experimental setup based on continual pre-training, linear probing, and a full-retention ensemble baseline. This setup effectively disentangles the performance degradation caused by catastrophic forgetting from the representational deficit caused by the missing co-observation of the data during training. • Empirical evidence in the vision domain: We provide rigorous empirical evidence of the co-observation effect across both supervised and self-supervised continual pre-training paradigms, on standard CL vision benchmarks in the “chunking” setting, demonstrating that it is a fundamental limitation of separate training. • Contextualization of memory replay and knowledge distillation: We discuss the broader implications of our findings, marking that the empirical success of memory rehearsal methods stems not only from preventing forgetting but also from actively restoring the conditions necessary for data co-observation. And vice versa, knowledge distillation at a representation level alone can compensate forgetting, but not co-observation. 2 Isolating the co-observation effect To understand the impact of data co-observation in continual learning, it must be carefully detached from the impact of forgetting and loss of plasticity. In this section, we first make a conceptual argument for the co-observation effect, detailing how the lack of simultaneous data access implies a fundamental representational deficit. Subsequently, we introduce our methodology to demonstrate this phenomenon empirically. Because standard end-to-end evaluations conflate the effect from co-observation with effects from stability-plasticity, and come with additional challenges regarding confounding variables such as task-recency bias and class imbalance (51; 13), we design a diagnostic evaluation framework focused on continual representation learning to decouple these effects as cleanly as possible. 2.1 A conceptual explanation of the co-observation effect Assume a global data distribution is partitioned into sequential observations, 1D_1 and 2D_2. And, assume there exists a set of generalizable features, ZsharedZ_shared, which improve prediction of the target variable of interest across the joint distribution. Following multi-task learning theory (4), discovering ZsharedZ_shared is facilitated by minimizing risk across multiple data variations simultaneously. When optimizing solely on 1D_1, neural networks naturally favor simpler functions (simplicity bias) (45). Consequently, the model relies on task-specific shortcuts ZlocalZ_local instead of learning ZsharedZ_shared. Because ZsharedZ_shared was never necessary to minimize the loss in 1D_1, it becomes unidentifiable from the frozen parameters θ1 _1. Consider an ideal continual learner transitioning to 2D_2 with perfect stability (zero forgetting) and plasticity. While it perfectly retains θ1 _1, these parameters do not encode the discarded ZsharedZ_shared, and when optimizing exclusively on 2D_2, the model will again favor local shortcuts over discovering ZsharedZ_shared. We argue that discovering the overarching synergies of ZsharedZ_shared fundamentally requires updating features based on their simultaneous performance across both distributions. Without concurrent access to both 1D_1 and 2D_2, the model lacks the mutual constraints necessary to favor shared features. Thus, even a model that perfectly retains all past parameter states cannot necessarily synthesize cross-distribution dependencies. 2.2 Continual learning and linear probing We consider a standard separate training setting where a model is trained sequentially on a series of data subsets 1,2,…,TD_1,D_2,…,D_T. At each training stage t, by default, the model has access only to tD_t and cannot observe data from previous or future stages. Access to previous data is permitted only when a CL mechanism maintains a memory buffer. To evaluate the fundamental quality of the knowledge acquired during this process, we decouple the model into a feature extractor (backbone) and a classifier (head), focusing our analysis entirely on the feature extractor. Following standard practices in representation learning, we evaluate all approaches using linear probing (LP; 2). After completing a stage in the sequential training process, we freeze the learned feature extractor and train a linear classifier on top of the frozen representations using all available training data ⋃t=1Tt _t=1^TD_t. By evaluating the representation globally, we ensure that any differences in performance reflect its generalizability, rather than artifacts of a continually shifting decision boundary. 2.3 Four baselines To isolate the influence of co-observation, we compare the sequentially trained representation against an upper target while controlling for catastrophic forgetting. We achieve this by comparing four distinct baselines (c.f. Figure 2): • Joint (offline): A single model is trained with simultaneous access to all data ⋃i=1Ti _i=1^TD_i. This represents the standard offline machine learning paradigm, i.e. the common choice of upper target for representation quality. • Incremental joint: At each stage t, the model is trained with simultaneous access to all data encountered so far, i.e. ⋃i=1ti _i=1^tD_i. Because the model trains on all available past and present data simultaneously, it circumvents both fragmented data access and catastrophic forgetting. Because it is subject to loss of plasticity from repeated optimization, as other continual learning approaches, we utilize it for fair comparison. • Ensemble (perfect forgetting mitigation): To eliminate forgetting of previously learned knowledge without explicitly breaking the separate observability constraint of sequential training, we employ an ensemble approach. After each sequential training stage t, a copy of the feature extractor is stored. At evaluation time, the representations from all T stored checkpoints are concatenated, and a single linear probe is trained on this combined representation. • Naive sequential (lower target): A single model is trained on each data subset incrementally, with each stage initialized from the previous stage’s final checkpoint. This naive continual learning approach is susceptible to both catastrophic forgetting and the lack of co-observation. Because the ensemble preserves every historical state of the model, features learned at any stage remain available to the linear probe and its forgetting is mitigated by definition. Conversely, incremental joint training establishes an “upper target” that accounts for the loss of plasticity from the repeated optimization trajectory that the other sequential learning approaches are subject to. We included joint (offline) training to visualize that degradation. To the best of our knowledge, the ensemble serves as the most controlled approximation for a perfect-retention baseline to isolate the co-observation effect as cleanly as possible. Also, we do not aim to position the ensemble-baseline as a practical motivated choice of a continual learning approach, but use it as an analytical device. 2.4 Controlling for representational capacity A methodological challenge arises when comparing the ensemble to single-model baselines. First, the concatenated ensemble yields an expanded representation of dimension T×dT× d. To ensure a rigorous comparison of feature quality rather than raw capacity, we apply PCA where appropriate, retaining the d directions of greatest variance. This projection does not perfectly simulate a single model that has mitigated forgetting, but corrects for differences due to architectural bloat. Second, while our ensemble of frozen checkpoints prevents the loss of previously acquired information, i.e. forgetting, it inherently differs from a hypothetical “ideal” single model. Specifically, it does not fully account for the seamless consolidation of knowledge via forward and backward transfer. To gauge the influence of this potential confound regarding our experimental setup (Section 4), we empirically ablate it in Appendix C and find it to be minor. (a) (b) Forgetting Co-observation Plasticity Joint ✓ ✓ ✓ Incr. Joint ✓ ✓ ✗ Ensemble ✓ ✗ ✗ Naive ✗ ✗ ✗ (c) Figure 2: Overview of training and evaluation for joint and separate training and expected empirical decomposition of the co-observation effect. (a) In the (incremental) joint training settings (top), the backbone θBt _B_t is trained with simultaneous access to all data (accumulated up to stage t). The incremental joint model is continually updated from the previous stage’s initialization at each step, whereas the offline joint model is trained from scratch and depicted only at the final stage. In the separate training setting (bottom), the model observes only the current data chunk. To demonstrate the effect of co-observation, we eliminate forgetting without breaking the sequential visibility constraint using the ensemble baseline. The ensemble stores a frozen copy of the backbone after each task. During evaluation, representations from all t checkpoints are concatenated, optionally projected back to the original dimension d to control for representational capacity (dim. reduce), and evaluated via linear probing (LP). (b) Hypothesis of the expected results across four chunks. The performance difference between the naive sequential model and the ensemble quantifies forgetting, while the gap between the incremental joint model and the (offline) joint target highlights the loss of plasticity. The focus of this work lies on the remaining difference between the ensemble and the incremental joint model, which we attribute to the co-observation effect. (c) Summary of how the four baselines inherently account for the presence (✓) or absence (✗) of forgetting mitigation, data co-observation, and maintained plasticity. 3 Proof of principle To provide a proof-of-principle for the co-observation effect, we design an experiment where we can explicitly control the available information at each training stage. This allows us to trace how separate observation, in comparison to joint observation, affects the learned representation. We construct a sequence of two binary classification tasks from MNIST digits (28). Task 1 (1D_1) is odd vs. even classification on clean images. Task 2 (2D_2) is small (digits 0–4) vs. large (digits 5–9) classification on images corrupted with Gaussian noise (severity level 5, applied via the imgaug library (25)). We select the evaluation task to cover odd vs. even classification on noisy images (Figure 3). Success on this evaluation task requires combining elements from both training tasks, namely the decision boundary from Task 1 and the ability to handle noisy inputs as encountered in Task 2. The question is whether the features, learned in the context of Task 1, remain available and transferable when learned sequentially versus jointly. For the experiments in this section, we utilize a 3-layer MLP backbone with 400400 hidden units per layer. The models are trained for 1010 epochs per task using a batch size of 6464. Optimization is performed using Stochastic Gradient Descent (SGD) with a peak learning rate of 0.010.01, momentum of 0.90.9, and gradient clipping at 1.01.0, regulated by a cosine annealing schedule (5% warm-up phase). We report the average performance across three independent runs with varying random seeds. The results, displayed in Figure 3, show that the ensemble achieves higher LP-accuracy than naive sequential training, presenting the influence of forgetting. However, incremental joint training further improves accuracy over the ensemble, pointing to another source of knowledge gain that is not explained by mitigating forgetting, as the ensemble preserves all past knowledge, but by co-observation that distinguishes joint training and separate training. To elaborate, in separate training, the model learns features that are sufficient for the current task’s data, but in a form that generalizes primarily within that training context. As such, Task 2 does expose the model to noise and the resulting representation can handle noisy inputs for the small/large classification, but this capability does not transfer to the odd/even task when trained sequentially. In contrast, joint training, i.e. simultaneous observation, allows this transfer of knowledge. When data of both tasks are observed together during training, features that serve both objectives are realized. Figure 3: Proof-of-principle MNIST experiment, demonstrating the co-observation effect. Four approaches are compared: naive sequential training, ensemble, incremental joint, and (offline) joint training. Task 1 is binary classification of odd vs. even digits on clean images. Task 2 is binary classification of small (≤4≤ 4) vs. large (≥5≥ 5) digits on images augmented with Gaussian noise. The evaluation task is odd vs. even classification on images that exhibit similar Gaussian noise as Task 2. Both joint training variants achieve higher LP-accuracy, demonstrating that noise features transfer to the “odd vs. even”-evaluation task when noisy data is being observed simultaneously, even when associated with a different task. This is contrasted by the ensemble, which preserves all learned representations but still achieves lower LP-accuracy. The gap between ensemble and joint training we attribute to the co-observation effect. 4 Co-observation in continual learning While the controlled experiment in the previous section showcases the effect of co-observation in a controlled but artificially designed scenario, we now continue to demonstrate its influence as a generic property to be taken into account in sequential learning. Continual learning benchmarks typically induce a severe distribution shift between stages (e.g., class-incremental learning). To emphasize the generality of our findings beyond artifacts of such amplified distribution shift or class imbalance, we evaluate the co-observation gap in an environment of uniformly partitioned incremental data. Following 29, we employ a “chunking” protocol (data-incremental learning) where the global training dataset is randomly and uniformly partitioned into four identically distributed chunks (1,…,4D_1,…,D_4). In this setting, even without macroscopic distribution shifts between the sequential stages, we are able to observe representation deficits that can be directly attributed to the lack of co-observation. We first assess the influence of co-observation across both supervised and self-supervised paradigms by comparing our diagnostic baselines. Subsequently, we evaluate exemplar replay and knowledge distillation in light of the co-observation effect. To ensure reproducibility, full details on the experimental setup are provided in Appendix A. Further implementation specifics regarding the diagnostic ensemble and linear evaluation are detailed in Appendix B, while the configurations for the continual learning methods are outlined in Appendix D. Finally, comprehensive tables containing the numerical results for all experiments are available in Appendix F. 4.1 Supervised and self-supervised continual learning To establish the co-observation gap as a generic property of sequential optimization, we first compare our four baselines: naive sequential training, the ensemble, incremental joint, and (offline) joint. As explained in Section 2.4, to control for the increased representational capacity of the ensemble, we also report the ensemble baseline with original backbone dimension d reduced by PCA. We evaluate these baselines across both supervised and self-supervised paradigms using the 44-chunk partitioned setup. In particular, we evaluate on CIFAR-100 (27) and ImageNet-100 (43) (IN-100) using a ResNet-18 (20) backbone for supervised learning and Barlow Twins (54), and a ViT-B/16 (16) for I-JEPA (3). All models are trained from scratch. For fair comparison, all training is constrained to an identical, fixed iteration budget per chunk. This fixed number of optimization steps is calibrated to match the (offline) joint training budget of 100100 epochs on CIFAR-100 and 9090 epochs on IN-100. Similarly, for the self-supervised methods (Barlow Twins and I-JEPA), the fixed iteration budget is calibrated to match 400400 epochs, aligned with 18. (a) (b) Figure 4: The co-observation effect across supervised and self-supervised learning paradigms. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds (6 for I-JEPA). (a) Supervised learning (Cross-Entropy) on CIFAR-100 and ImageNet-100 (IN-100). (b) Self-supervised learning using Barlow Twins and I-JEPA on IN-100. In all settings, the beneficial effect of co-observation is clearly visible. The ensemble baseline prevents the forgetting seen in the Naive baseline, and consistently falls short of the incremental joint training. This residual difference between ensemble (separate training) and incremental joint highlights the impact of co-observation, demonstrating that simultaneously observing data is important for representation learning, regardless of the objective function. Supervised continual learning: As illustrated in Figure 4(a), and further extended in Appendix E, naive sequential training exhibits the lowest linear probing accuracy, affirming the impact of catastrophic forgetting on the representation even in the absence of explicit class-incremental distribution shifts. The ensemble baseline significantly improves upon naive sequential training and is not affected when controlling for representational capacity by PCA reduction, indicated by both lines overlapping. A clear benefit is visible from co-observation as the incremental joint training consistently improves over the ensemble. Because the ensemble retains all past representations, this residual difference highlights the co-observation effect, i.e. the representational benefit derived from simultaneously observing data across different chunks, is a property that is fundamentally lost in separate training. Self-Supervised continual learning: To investigate whether the co-observation effect is specific to supervised cross-entropy optimization, we extend our evaluation to self-supervised learning (SSL). Unlike supervised learning, which leverages explicit class boundaries, SSL methods rely on inferring diverse features within the training batches to form robust, invariant representations, which are thought to be more robust to forgetting (18). We evaluate two distinct families of SSL architectures on IN-100: Barlow Twins (54), which relies on redundancy reduction, and I-JEPA (3), a recent joint-embedding predictive framework. The results (Figure 4(b)) demonstrate that the co-observation gap is also pronounced in the self-supervised regime. Interestingly, applying PCA reduction to the ensemble yields varying effects depending on the SSL objective. For Barlow Twins it can occasionally improve the representation quality, while degrading it for I-JEPA. Yet, across both SSL algorithms and regardless of the capacity control, our ensemble baselines consistently evaluate lower than the representation quality obtained from the incremental joint model. This result indicates that the co-observation gap is a consequence of separate representation learning: synergistic features that span across data partitions cannot be fully realized without simultaneous observation, regardless of the underlying optimization objective. While it is difficult to perfectly isolate the co-observation effect from all sequential optimization dynamics, we account for the primary alternative explanation of a deficit in knowledge transfer in Appendix C. Supported by our controls, and this ablation, the persistent representational deficit observed across both supervised and distinct self-supervised paradigms strongly points to the structural lack of simultaneous data observation as the primary driver. 4.2 Relation to continual learning mechanisms Having confirmed the co-observation gap in generic continual learning scenarios, we now examine how replay and knowledge distillation compare to our established baselines, to decouple their ability to incrementally improve generalization in the model’s representation relative to the mitigation of forgetting alone. (a) (b) Figure 5: Interaction of continual learning mechanisms with the co-observation gap. Linear probing accuracy is reported across 4 sequential data chunks, showing the mean and standard error over 3 seeds for (a) Supervised Learning and (b) Self-Supervised Learning (Barlow Twins and I-JEPA). Comparing experience replay and knowledge distillation (LwF for SL; CaSSLe for Barlow) against the naive-, joint-incremental-, and ensemble-baselines, empirically demonstrates the relationship of both methods to mitigation of forgetting and obtained generalization from the representation learning perspective. Experience Replay: Replaying stored data is intuitively the most direct approach to bridging separate learning stages. Mechanistically, it creates a minibatch-level approximation of the joint data distribution by presenting previous and new data together. While replay is predominantly renowned as a forgetting-mitigation tool, we evaluate it here to explicitly contextualize this minibatch approximation as mechanism that enables data co-observation. We evaluate replay on IN-100 with buffers allowing the storage of 1250(∼3%)1250( 3\%), 6250(∼15%)6250( 15\%), and 12500(∼30%)12500( 30\%) exemplars of the data from each previously observed chunk, where new data and all data from the replay buffers are combined and sampled i.i.d. into mini-batches during optimization (40; 51; 12). In terms of raw values, as expected, the results presented in Figure 5 show that larger replay buffers yield higher accuracy. However, our interest is in the comparison to our baselines which allows to reason about a more nuanced dynamic. While sparse rehearsal (e.g., 3%3\%) has been shown to yield improvements when evaluated via end-to-end accuracy (10), our linear-probing evaluation reveals a different dynamic at representation level. Here, replay with tiny buffers struggles to consistently match the diagnostic ensemble. Storing approximately 15%15\% of the data appears sufficient for the replay mechanism to effectively match the performance of the ensemble. When the buffer size is increased to 30%30\% of the observed exemplars per stage, the linear probing accuracy pushes past the ensemble baseline, closing half the gap to incremental joint training. These results hold for supervised and self-supervised training with Barlow Twins and I-JEPA. The replay mechanism inherently conflates the mitigation of forgetting with the benefits of co-observation. However, our comparison against our ensemble baseline yields two crucial takeaways. First, our findings suggest that replay’s effectiveness is not bound merely by mitigating forgetting, but by actively reintroducing data co-observation. Because we strictly control the training budget across all setups, the gains from larger replay buffers are not an artifact of extended optimization on more data. Rather, the improved approximation of the joint distribution at the mini-batch level – instantiating the mechanism underlying co-observation – drives altered optimization trajectories that allow the model to synthesize cross-distribution features that perfect retention alone cannot recover. Second, there is a significant amount of stored exemplars necessary to improve beyond results achievable by mitigation of forgetting, although the specific number of samples required will inherently vary based on the domain and dataset diversity. Distillation: As a popular memory-free alternative to rehearsal, we evaluate knowledge distillation. We apply Learning without Forgetting (LwF) (30) for the supervised setup, and CaSSLe (18) for the self-supervised Barlow Twins setup. As shown in Figure 5, distillation achieves linear probing accuracies that closely mirror those of our ensemble and the 15%15\% replay buffer. Knowledge distillation has previously been shown to be an effective retention mechanism, successfully preventing representation-level forgetting (37). We deem it important to emphasize that knowledge distillation here is not failing to prevent forgetting, as indicated by its parity with the ensemble. Rather, distillation from the “incomplete” representation of the previous model does not facilitate the co-observation of information that would be required to recover the representational effectiveness found in large-scale replay. 5 Related Work 5.1 Forgetting, plasticity, and a third factor The continual learning literature has predominantly framed the challenges of sequential optimization along the stability-plasticity axis (34), typically addressed through mechanisms such as regularization (26; 55; 30), replay (40; 31; 7; 41), or architecture expansion (44; 52). Progress is measured against a joint-training upper target, implicitly assuming that a model with perfect stability (zero forgetting) and perfect plasticity (ability to incorporate new information) would naturally recover joint-training performance (50; 35). While recent work by 15 and others (48; 12) has established loss of plasticity as a distinct degradation independent of forgetting, we identify the co-observation effect as a third factor. Several recent studies have empirically encountered symptoms of the co-observation effect, though these observations have typically been isolated within specific continual learning setups: In class-incremental scenarios, 37 observed “inter-phase confusion” persisting after addressing forgetting, and 46 noted a deficit in cross-task features. While conceptually related, these observations have been confined to class-incremental setups; we demonstrate that they are part of a general deficit in continual representation learning, extending well beyond class-incremental considerations. Moving beyond class-incremental boundaries, 6 demonstrated that “continual confounders” prevent sequential learners from discovering generalizable rules. To clearly isolate this effect, the authors systematically introduce explicitly constructed, adversarial spurious correlations within a synthetic dataset. Our work highlights the broader implications of this result for separate training, demonstrating that such confounding effects can be considered to arise naturally simply from the sequence of the data itself. To do so, we build on the “chunking” scenario previously utilized by 29 to relate the impact of separate training to forgetting. We shift the focus to data co-observation, demonstrating that the co-observation effect emerges generically from random partitioning and affects generalization alongside forgetting, class imbalance, or adversarial structures. Finally, our work establishes that this effect on continual representation learning is not merely an artifact of supervised training dynamics or the narrow decision boundaries induced by cross-entropy. Self-supervised learning (SSL) methods are broadly considered to learn more diverse, task-agnostic features (18; 32). Despite these advantages, we show that the lack of co-observation affects learning similarly across the distinct SSL frameworks of Barlow Twins (54) and I-JEPA (3). This establishes the co-observation effect as a fundamental limitation of fragmented optimization, affecting both supervised and self-supervised sequential learning. 5.2 Representation evaluation and the ensemble baseline End-to-end accuracy in continual learning often conflates actual representation quality with the dynamics of a continually updated classifier head (51). To address this, 13 used linear probing (LP) (2) as an evaluation tool for CL, demonstrating that representation-level forgetting under naive finetuning is substantially milder than end-to-end metrics often suggest. We also adopt LP to evaluate the knowledge accumulated within our model at representation level. Thereby, our approach is more closely related to 21, who combined LP evaluation with an ensemble baseline to demonstrate that even seemingly “mild” representation-level forgetting in absolute numbers still exerts a detrimental effect on the effectiveness of continual representation learning. Instead of utilizing probing to highlight the penalty of forgetting, in the current work, we leverage the ensemble as a baseline to control for forgetting, enabling our novel analysis regarding the isolation of the co-observation effect. 5.3 Multi-task learning and joint optimization Historically, continual learning shares conceptual roots with multi-task learning (MTL). Both paradigms strive to accumulate and leverage knowledge across multiple objectives (11) and early continual learning works were differentiated from MTL by training the tasks incrementally instead of jointly. Regarding MTL, 9 showed that learning tasks in parallel can improve generalization because related tasks provide a mutually beneficial inductive bias, helping the model discover features it would not find when trained on either task in isolation. 42 articulated several mechanisms driving this benefit, including implicit data augmentation, attention focusing, and regularization through multi-objective optimization. 4 provided a theoretical foundation for this, showing that the risk of overfitting shared parameters decreases with the number of co-observed tasks. Furthermore, as demonstrated by 5, the specific composition of training data significantly influences optimization trajectories in non-convex landscapes. These works underscore that which data is observed simultaneously, fundamentally shapes the quality of the learned solutions. To the best of our knowledge the classical MTL desideratum of improving representation quality was largely left aside in continual learning, presumably because of the observed overwhelming impact of catastrophic forgetting. In this sense, our work can be understood as demonstrating that even within a single task (rather than multiple different tasks), fragmenting data into separate portions leads to reduced generalization of the model. Consequently, a limit is placed on the representational quality that can be achieved from strict separate learning, regardless of how well the aspect of forgetting is controlled. 6 Discussion 6.1 Limiting factors other than forgetting Continual learning algorithms are expected to improve over naive sequential training. However, one could argue that there are cases where reaching the joint training upper target is a rather theoretical construct because of strict limitations in compute (39; 47), memory (14; 10), privacy boundaries (17), the need to operate in an open world (36), etc. Nevertheless, achieving best possible results is the goal and a prevailing narrative heavily focuses on catastrophic forgetting as the primary culprit for the performance drop observed when training continually, with some recent pushes to also consider effects from plasticity losses (12; 48). Our work points out yet another, fundamental, effect where previous knowledge is perfectly retained or protected, but the model’s generalization falls short of the upper target. That is, without requiring any particularity of the distribution shift underlying the separate observations, apart from the samples being disjoint. Different instances of this effect have been studied, predominantly with respect to class-incremental learning, where intuitively observing instances of different classes together is required to construct discriminating features (51; 46; 37). But also regarding the potential of confounding factors in the separately observed training data, which inflict deficiencies on the continually training model that are not easy to recover (6). We demonstrate data co-observation as an inherent and distinct limiting factor in separate training, without additional prerequisites as studied in prior works. 6.2 Connection to large pre-trained models and LLMs Recent continual learning literature increasingly leverages pre-trained models. Approaches that leave the backbone frozen reduce continual learning to incremental prototype memorization (24). In such settings, the discovery of cross-distribution synergies is typically of minor importance and the co-observation effect is bypassed. Methods employing conservative fine-tuning, e.g. applying slow learning rates (56) or utilizing adapters (1), aim to make only slight adjustments to the pre-trained features. Arguably, the magnitude of new learning is inherently small relative to the initial pre-training phase, and optimization is heavily regularized to protect existing weights. Consequently, the dynamics of catastrophic forgetting and its mitigation largely overpower any representational gains that would theoretically emerge from co-observation. Continual pre-training to broadly update and improve the generalization of the backbone in a more foundational manner (22) is where we hypothesize co-observation effects to be more important. While our empirical analysis is conducted on standard vision benchmarks, evidence from the large language model (LLM) literature suggests that the interplay between data co-occurrence and representation quality extends to larger scales. 53 demonstrate that the composition of training batches during LLM pre-training, specifically, cross-domain interaction effects, quantitatively predicts model performance. Since their analysis concerns standard i.i.d. pre-training rather than sequential learning, this provides evidence that which data is observed together shapes learned representations independently of any forgetting dynamics. In the continual pre-training setting, 23 show that replaying previous data into new training batches is necessary to match the performance of retraining from scratch, a gap that learning rate adjustments alone cannot close. While this deficit is conventionally attributed to forgetting, our framework raises the complementary hypothesis that replay also restores co-observation conditions that enable the model to discover cross-distribution patterns. Disentangling these two roles of replay, forgetting prevention versus co-observation restoration, at LLM scale constitutes a natural direction of investigation. The diagnostic framework introduced in this work, comparing forgetting-free ensembles against joint training via linear probing, offers a principled starting point for this investigation. 7 Conclusion We empirically separated the effect of data co-observation from stability-plasticity considerations for the representation-level generalization of continually trained deep neural networks, and demonstrated the generic emergence of that effect across supervised and self-supervised training in data-incremental scenarios. While the representational synergies of joint training are well-established in the multi-task learning literature, we demonstrate such a performance gap persists generically in standard vision benchmarks even in “chunking” scenarios. We argue that making the influence of co-observation explicit and positioning it distinct from forgetting and plasticity, as a third facet of continual learning, is important for the community: assuming plasticity and successful forgetting mitigation whilst learning without co-observation appears to hit a performance ceiling that can be significantly lower than the upper target of a jointly trained model. We emphasize that we do not claim the performance penalty of co-observation universally rivals the sheer magnitude of catastrophic forgetting. As discussed in the context of pre-trained models, forgetting often remains the most striking source of degradation in many applications. Nevertheless, we deem explicitly isolating and understanding the co-observation effect to be conceptually vital for mapping the full dynamics and limits of continual learning. Improvement beyond requires continual learning algorithms to actively synthesize cross-data synergies. How to best facilitate such synergies, especially in compute-bounded settings, needs to be the subject of further research. 8 Limitations We see three limitations to this work. The first regards our diagnostic methodology, which intends to isolate the co-observation gap using linear probing and an ensemble baseline. While this effectively removes confounders like classifier deviation and forgetting, strictly controlling the ensemble’s capacity requires post-hoc dimensionality reduction (in our case using PCA). Although an ideal baseline would natively match the original model’s parameter count, we deem our capacity-controlled ensemble to serve as the most viable analytical proxy. Secondly, we employed uniform optimization schedules to ensure fair comparability between joint and separate training, rather than to maximize absolute performance. Other continual optimizers might yield higher accuracies, mitigate part of the forgetting or better control plasticity, but we expect the underlying trends due to data co-observation to remain robust. Finally, while we demonstrate the effect from co-observation across supervised and self-supervised paradigms on moderate-scale vision tasks, representational dynamics can shift at massive scales. Future work must investigate whether billion-parameter over-parameterization naturally mitigates or exacerbates the need for simultaneous data observation. Acknowledgments This paper is part of a project that has received funding from the European Union under the Horizon 2020 research and innovation program (ERC project KeepOnLearning, grant agreement No. 101021347), and KU Leuven C1 project (Reference: C14/23/100). Part of this work was conducted while Abhishek was affiliated with ESAT-PSI, KU Leuven, Belgium. The computational resources and services used in this work were partly provided by the VSC (Flemish Supercomputer Center). References Aghasanli et al. (2025) A. Aghasanli, Y. Li, and P. Angelov Prototype-based continual learning with label-free replay buffer and cluster preservation loss. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 6545–6554. Cited by: §6.2. Alain and Bengio (2017) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations (ICLR) workshop. Cited by: §1, §2.2, §5.2. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15619–15629. Cited by: Appendix D, §4.1, §4.1, §5.1. Baxter (2000) J. Baxter A model of inductive bias learning. Journal of artificial intelligence research 12, p. 149–198. Cited by: §2.1, §5.3. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML), p. 41–48. Cited by: §5.3. Busch et al. (2025) F. P. Busch, R. R. Kamath, R. Mitchell, W. Stammer, K. Kersting, and M. Mundt Where is the truth? The risk of getting confounded in a continual world. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 6046–6076. Cited by: §5.1, §6.1. Buzzega et al. (2020) P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems, Vol. 33, p. 15920–15930. Cited by: §5.1. Carta et al. (2023) A. Carta, L. Pellegrini, A. Cossu, H. Hemati, and V. Lomonaco Avalanche: a pytorch library for deep continual learning. Journal of Machine Learning Research 24 (363), p. 1–6. External Links: Link Cited by: Appendix D. Caruana (1997) R. Caruana Multitask learning. Machine Learning 28 (1), p. 41–75. Cited by: §1, §5.3. Chaudhry et al. (2019) A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486. Cited by: §4.2, §6.1. Chen and Liu (2018) Z. Chen and B. Liu Lifelong machine learning. Second edition, Synthesis Lectures on Artificial Intelligence and Machine Learning, Vol. 12, Morgan & Claypool Publishers. External Links: Document, Link Cited by: §1, §5.3. Cho et al. (2026) D. Cho, T. Moon, R. Chunara, K. Cho, and S. Cha Forget forgetting: continual learning in a world of abundant memory. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §D.1, §4.2, §5.1, §6.1. Davari et al. (2022) M. Davari, N. Asadi, S. Mudur, R. Aljundi, and E. Belilovsky Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16712–16721. Cited by: §2, §5.2. De Lange et al. (2021) M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars A continual learning survey: defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence 44 (7), p. 3366–3385. Cited by: §1, §6.1. Dohare et al. (2024) S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton Loss of plasticity in deep continual learning. Nature 632, p. 768–774. Cited by: §5.1. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Farquhar and Gal (2019) S. Farquhar and Y. Gal Differentially private continual learning. arXiv preprint arXiv:1902.06497. Cited by: §6.1. Fini et al. (2022) E. Fini, V. G. T. da Costa, X. Alameda-Pineda, E. Ricci, K. Alahari, and J. Mairal Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9621–9630. Cited by: Appendix A, §B.3, §C.2, §D.3, Appendix D, §4.1, §4.1, §4.2, §5.1. Grossberg (1980) S. Grossberg How does a brain build a cognitive code?. Psychological review 87 (1), p. 1. Cited by: §1. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778. Cited by: Appendix A, §4.1. Hess et al. (2024) T. Hess, E. Verwimp, G. M. van de Ven, and T. Tuytelaars Knowledge accumulation in continually learned representations and the issue of feature forgetting. Transactions on Machine Learning Research. Cited by: §1, §5.2. Hu et al. (2022) D. Hu, S. Yan, Q. Lu, L. Hong, H. Hu, Y. Zhang, Z. Li, X. Wang, and J. Feng How well does self-supervised pre-training perform with streaming data?. In International Conference on Learning Representations, External Links: Link Cited by: §1, §6.2. Ibrahim et al. (2024) A. Ibrahim, B. Thérien, K. Gupta, M. L. Richter, Q. Anthony, T. Lesort, E. Belilovsky, and I. Rish Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.08763. Cited by: §6.2. Janson et al. (2022) P. Janson, W. Zhang, R. Aljundi, and M. Elhoseiny A simple baseline that questions the use of pretrained-models in continual learning. arXiv preprint arXiv:2210.04428. External Links: Document Cited by: §6.2. Jung et al. (2020) A. B. Jung, K. Wada, J. Crall, S. Tanaka, J. Graving, C. Reinders, S. Yadav, J. Banerjee, G. Vecsei, A. Kraft, Z. Rui, J. Borovec, C. Vallentin, S. Zhydenko, K. Pfeiffer, B. Cook, I. Fernández, F. De Rainville, C. Weng, A. Ayala-Acevedo, R. Meudec, M. Laporte, et al. imgaug. Note: https://github.com/aleju/imgaugOnline; accessed 01-Feb-2020 Cited by: §3. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §5.1. Krizhevsky et al. (2009) A. Krizhevsky G. Hinton et al. Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §4.1. LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), p. 2278–2324. Cited by: §3. Lee and Storkey (2023) T. L. Lee and A. Storkey Chunking: continual learning is not just about distribution shift. CoLLAs. Cited by: §C.1, §1, §4, §5.1. Li and Hoiem (2016) Z. Li and D. Hoiem Learning without forgetting. In Proceedings of the European Conference on Computer Vision (ECCV), Vol. 2, p. 4. Cited by: §C.2, §D.2, §4.2, §5.1. Lopez-Paz and Ranzato (2017) D. Lopez-Paz and M. Ranzato Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, Vol. 30, p. 6467–6476. Cited by: §5.1. Madaan et al. (2022) D. Madaan, J. Yoon, Y. Li, Y. Liu, and S. J. Hwang Representational continuity for unsupervised continual learning. In International Conference on Learning Representations (ICLR), Cited by: §5.1. McCloskey and Cohen (1989) M. McCloskey and N. J. Cohen Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of learning and motivation, Vol. 24, p. 109–165. Cited by: §1. Mermillod et al. (2013) M. Mermillod, A. Bugaiska, and P. Bonin The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology Volume 4 - 2013. External Links: Document, ISSN 1664-1078 Cited by: §1, §5.1. Momeni and Liu (2025) S. Momeni and B. Liu Achieving upper bound accuracy of joint training in continual learning. arXiv preprint arXiv:2502.12388. Cited by: §5.1. Mundt et al. (2023) M. Mundt, Y. Hong, I. Pliushch, and V. Ramesh A wholistic view of continual learning with deep neural networks: forgotten lessons and the bridge to active and open world learning. Neural Networks 160, p. 306–336. Cited by: §6.1. Ni et al. (2023) Z. Ni, H. Shi, L. Wei, Y. Zhuang, Q. Tian, and S. Tang On representation-level forgetting in class incremental learning: what’s the bottleneck?. SSRN. External Links: Document Cited by: §4.2, §5.1, §6.1. Parisi et al. (2019) G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter Continual lifelong learning with neural networks: a review. Neural networks 113, p. 54–71. Cited by: §1. Prabhu et al. (2023) A. Prabhu, H. A. Al Kader Hammoud, P. K. Dokania, P. H. Torr, S. Lim, B. Ghanem, and A. Bibi Computationally budgeted continual learning: what does matter?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3698–3707. Cited by: §6.1. Rebuffi et al. (2017) S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert iCaRL: incremental classifier and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2001–2010. Cited by: §D.1, §4.2, §5.1. Rolnick et al. (2019) D. Rolnick, A. Ahuja, J. Schwarz, T. P. Lillicrap, and G. Wayne Experience replay for continual learning. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §5.1. Ruder (2017) S. Ruder An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. Cited by: §5.3. Russakovsky et al. (2015) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision 115 (3), p. 211–252. Cited by: §4.1. Rusu et al. (2016) A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell Progressive neural networks. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: §5.1. Shah et al. (2020) H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems 33, p. 9573–9585. Cited by: §2.1. Soutif-Cormerais et al. (2021) A. Soutif-Cormerais, M. Masana, J. Van De Weijer, and B. Twardowski On the importance of cross-task features for class-incremental learning. In ICML Workshop on Theory and Foundations of Continual Learning, External Links: Link Cited by: §5.1, §6.1. Verwimp et al. (2024) E. Verwimp, R. Aljundi, S. Ben-David, M. Bethge, A. Cossu, A. Gepperth, T. L. Hayes, E. Hüllermeier, C. Kanan, D. Kudithipudi, C. H. Lampert, M. Mundt, R. Pascanu, A. Popescu, A. S. Tolias, J. van de Weijer, B. Liu, V. Lomonaco, T. Tuytelaars, and G. M. van de Ven Continual learning: applications and the road forward. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §1, §6.1. Verwimp et al. (2025) E. Verwimp, G. Hacohen, and T. Tuytelaars Same accuracy, twice as fast: continuous training surpasses retraining from scratch. arXiv preprint arXiv:2502.21147. Cited by: §5.1, §6.1. Vogelstein et al. (2025) J. T. Vogelstein, J. Dey, H. S. Helm, W. LeVine, R. D. Mehta, T. M. Tomita, H. Xu, A. Geisa, Q. Wang, G. M. van de Ven, C. Gao, W. Yang, B. Tower, J. Larson, C. M. White, and C. E. Priebe Simple lifelong learning machines. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (11), p. 10033–10046. Cited by: §1. Wang et al. (2024) L. Wang, X. Zhang, H. Su, and J. Zhu A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5362–5383. External Links: Document Cited by: §1, §5.1. Wu et al. (2019) Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 374–382. Cited by: §D.1, §2, §4.2, §5.2, §6.1. Yan et al. (2021) S. Yan, J. Xie, and X. He DER: dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3014–3023. Cited by: §5.1. Ye et al. (2024) J. Ye, P. Liu, T. Sun, J. Zhan, Y. Zhou, and X. Qiu Data mixing laws: optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952. Cited by: §6.2. Zbontar et al. (2021) J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny Barlow twins: self-supervised learning via redundancy reduction. In International conference on machine learning, p. 12310–12320. Cited by: §D.3, §4.1, §4.1, §5.1. Zenke et al. (2017) F. Zenke, B. Poole, and S. Ganguli Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), p. 3987–3995. Cited by: §5.1. Zhang et al. (2023) G. Zhang, L. Wang, G. Kang, L. Chen, and Y. Wei Slca: slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 19148–19158. Cited by: §6.2. Zhou et al. (2025) D. Zhou, Z. Cai, H. Ye, D. Zhan, and Z. Liu Revisiting class-incremental learning with pre-trained models: generalizability and adaptivity are all you need. International Journal of Computer Vision 133 (3), p. 1012–1032. Cited by: §1. Appendix The following appendix provides supplementary details regarding our experimental setups, further details on our evaluation methodology, and quantitative results. The contents are structured as follows: • Appendix A details the optimization hyperparameters, dataset preprocessing, and augmentation pipelines for both the supervised and self-supervised pre-training regimes. • Appendix B expands upon our evaluation framework, describing the construction of the perfect-retention ensemble, the capacity-controlling dimensionality reduction via PCA, and the linear probing protocol, in full detail. • Appendix C provides empirical verification of the diagnostic ensemble, demonstrating its zero-forgetting properties on previously observed training data and ablating the potential confounding influence of forward and backward transfer using distillation-enhanced checkpoints. • Appendix D formally defines the continual learning mechanisms evaluated in this work, specifically our adaptations of Experience Replay and Knowledge Distillation (LwF and CaSSLe). • Appendix E presents additional results extending the supervised learning evaluation on ImageNet-100 to longer continual sequences of 88 and 1212 chunks, demonstrating the robustness of the co-observation gap under increased data fragmentation. • Appendix F provides the comprehensive numerical results, corresponding to the empirical evaluations and figures presented in the main text Section 4. Appendix A Detailed Experimental Setup Datasets and Augmentations. For CIFAR-100, training images (original resolution 32×3232× 32) are augmented using random cropping (padding of 44, reflect mode) and random horizontal flipping (p=0.5p=0.5). Images are normalized with μ=(0.485,0.456,0.406)μ=(0.485,0.456,0.406) and σ=(0.229,0.224,0.225)σ=(0.229,0.224,0.225). At evaluation, images are strictly evaluated at 32×3232× 32. For ImageNet-100 (IN-100), images are stored at 256256 pixels on their longer side. During training, we apply a RandomResizedCrop to 224×224224× 224, a random horizontal flip, and RandAugment (22 operations, magnitude 99). At evaluation, images are center cropped to 224×224224× 224, normalized with μ=(0.4914,0.4822,0.4465)μ=(0.4914,0.4822,0.4465) and σ=(0.2023,0.1994,0.2010)σ=(0.2023,0.1994,0.2010). Supervised Learning Configurations. For supervised training, we utilize a ResNet-18 backbone. For CIFAR-100, the backbone is adjusted to accommodate the smaller input size, as proposed by 20 (i.e., replacing the initial 7×77× 7 convolution with a 3×33× 3 convolution and removing the first max-pooling layer). The model is optimized using SGD with a momentum of 0.90.9, a weight decay of 5×10−45× 10^-4, and gradient clipping set to 1.01.0. The learning rate follows a warmup phase constituting 5%5\% of the total iterations, followed by a cosine annealing schedule with a peak learning rate of 0.10.1, and a final learning rate of 10−510^-5. To maintain consistent training dynamics across datasets, the total number of iterations is scaled to be equivalent to approximately 100100 epochs for CIFAR-100 and 9090 epochs for IN-100 on the respective full datasets. Specifically, training on CIFAR-100 proceeds for 9,7509,750 iterations per chunk with a mini-batch size of 128128, while training on IN-100 proceeds for 11,07011,070 iterations per chunk with a mini-batch size of 256256. The linear probe is trained mirroring the linear evaluation protocol of 18. We use a fixed number of 100100 epochs on the (optionally reduced) frozen representations, using SGD with a momentum of 0.90.9, an initial learning rate of 0.10.1, and no weight decay. The learning rate is decayed by a factor of 1010 at 6060 and 8080 epochs. Self-Supervised Learning Configurations. For the self-supervised continual learning experiments, we evaluate two distinct frameworks with their respective standard configurations. Barlow Twins. The ResNet-18 encoder is paired with a 33-layer MLP projector (hidden and output dimensions of 20482048) using BatchNorm and ReLU activations. The objective uses an off-diagonal scaling factor λ=5×10−3λ=5× 10^-3 and an overall loss scale of 0.10.1. Two views are generated per image using random resized cropping (scale [0.08,1.0][0.08,1.0]), horizontal flipping, color jitter, Gaussian blur, and random solarization. The model is optimized using LARS (learning rate 0.40.4, weight decay 10−410^-4, momentum 0.90.9) with a warmup-cosine schedule for 400400 epochs per chunk and a batch size of 256256. I-JEPA. The ViT-B/16 encoder (patch size 16, 224×224224× 224 input) is trained alongside an Exponential Moving Average (EMA) target encoder, with the momentum coefficient annealed from 0.9960.996 to 1.01.0. The predictor is a depth-6 ViT with an embedding dimension of 384384. A block-masking strategy samples one context mask (85–100%85--100\% of patches) and four target masks (15–20%15--20\% of patches). The model minimizes the smooth-ℓ1 _1 (Huber) loss and is optimized using Adam (learning rate 10−310^-3, weight decay 0.040.04) with a warmup-cosine schedule for 400400 epochs per chunk at a batch size of 512512. Appendix B Method details: Ensemble B.1 Ensemble Construction The ensemble-baseline introduced in Section 2 is designed to provide a perfect-retention baseline by storing the exact state of the continually training model after each training phase. After completing training on experience t, a frozen copy of the feature extractor (backbone) fθtf_ _t is appended to a persistent buffer ℬ=fθ1,fθ2,…,fθtB=\f_ _1,f_ _2,…,f_ _t\. Thereafter, these backbone weights are never updated. During evaluation, an input x is passed through all |ℬ||B| frozen backbones, as illustrated in Figure 2. The resulting features are concatenated along the channel dimension to form a single combined representation ∈ℝ|ℬ|⋅dz ^|B|· d, where d is the feature dimension of a single backbone. This concatenated representation is then used for downstream linear probing. B.2 Dimensionality Reduction Because the ensemble concatenates representations from |ℬ||B| checkpoints, its output dimension |ℬ|⋅d|B|· d scales linearly with the number of training phases. To ensure a fair comparison against single-model baselines and isolate feature quality from increased dimensional capacity, we apply an optional dimensionality reduction step to compress z back to a target dimension d′=d =d before linear probing. For the reduction a PCA is fitted exclusively on the concatenated representations of the downstream training set, retaining the d′d components with the largest variance. The fitted projection matrix is then fixed and applied to the evaluation set to prevent information leakage. This reduction step is only applied when evaluating the ensemble. B.3 Linear-Probe Evaluation At each evaluation point, a newly initialized linear classification head ∈ℝC×dinW ^C× d_in is instantiated and trained on the downstream training set, where C is the number of downstream classes and dind_in is the appropriate feature dimension. To optimize training efficiency, all training-set inputs are passed through the frozen backbone(s) once, and the resulting representations are cached. The linear probe is then trained on these cached features using a standard cross-entropy loss. Similarly to 18 the optimizer is an SGD (with momentum and either a multi-step or cosine annealing learning rate schedule). After training, the probe is evaluated on the downstream test set via a single forward pass, reporting top-11 accuracy. Appendix C Empirical verification of knowledge retention in the ensemble Our diagnostic ensemble serves as analytical device to isolate the co-observation effect from the standard stability-plasticity considerations as well as possible. To ensure the validity of this isolation, this section provides empirical verification that the ensemble reliably controls for catastrophic forgetting (perfect retention), and ablates potential confounding deficits regarding knowledge transfer. C.1 Empirical verification of perfect retention In Section 2.3, we introduce an ensemble baseline designed as an analytical device to achieve perfect retention (zero forgetting). To empirically corroborate that our ensemble baseline exhibits this stability, we mirror the diagnostic methodology utilized by 29 and conduct an evaluation directly on the training data. Because data-incremental “chunking” scenarios do not inherently possess a separate validation set per chunk, evaluating performance on the previously seen training data serves as a direct proxy for information retention. Figure 6: Empirical verification of zero forgetting in the diagnostic ensemble baseline. The linear probe accuracy is evaluated on the training sets of individual chunks (T0T_0 through T3T_3) across 4 sequential training stages. After a chunk is actively trained on, its train-set accuracy does not decrease in subsequent stages. As demonstrated in Figure 6, once the ensemble trains on a given chunk, its accuracy on that chunk’s training data never declines in any subsequent stage. Instead, we observe an increase in accuracy for chunk 00 and forward transfer w.r.t. later chunks. This occurs because the representations of past chunks are perfectly preserved, and concatenating newly frozen backbones from later stages only provides the global linear probe with strictly more features to combine, effectively enabling backward transfer at the readout level. This confirms the ensemble operates as a strict perfect-retention baseline. C.2 Ablation on knowledge transfer capabilities While the ensemble perfectly mitigates the loss of previously acquired information post-hoc, it is constructed from a naive sequential continual learner. Consequently, the individual checkpoints inherently suffer from forgetting during the sequential training process itself, which can impair forward transfer (the ability of past learned features to facilitate future learning). This means the ensemble does not fully account for the seamless consolidation of knowledge via ”ideal” forward and backward transfer that a perfect, single continually learning model would possess. To gauge this potential confounding influence, we conduct an ablation where we construct the ensemble using checkpoints from models equipped with stronger transfer and retention properties, rather than naive sequential checkpoints. Specifically, we utilize knowledge distillation methods, i.e. LwF (30) for the supervised setting, and CaSSLe (18) for the self-supervised setting. We choose distillation because it does not reintroduce data co-observation, preserving the validity of the ablation. Figure 7: Ablation of forward/backward transfer using distillation-enhanced checkpoints. Comparing the standard diagnostic ensemble (built from naive sequential checkpoints) against an ensemble built from distillation-enhanced checkpoints (LwF/CaSSLe) across sequential chunks. Even with enhanced transfer properties, the ensemble at best barely surpasses the standard baseline and fails to close the gap to joint-incremental training. As shown in Figure 7, substituting the naive checkpoints with distillation-enhanced checkpoints yields marginal (in case of CaSSLe) to no (in case of LwF) improvements. The performance of this enhanced ensemble tightly tracks the standard ensemble baseline and remains significantly below incremental joint training. Combined with our conceptual arguments in Section 2.1, this supports that the residual performance gap between the ensemble and joint training can be attributed to the representational deficit caused by the lack of simultaneous data co-observation, rather than being solely an artifact of poor transfer interaction between isolated checkpoints. Appendix D Method details: Continual Learning Mechanisms Implementation of the experiments builds on the Avalanche CL library (8)† https://github.com/ContinualAI/avalanche, the CaSSLe (18)† https://github.com/DonkeyShot21/cassle framework for self-supervised continual learning, and the official codebase of I-JEPA (3)† https://github.com/facebookresearch/ijepa. D.1 Experience Replay Our implementation of experience replay maintains a persistent memory buffer ℳM of past exemplars, with a buffer policy that stores m samples per observed task. Similar to a number of works that utilize exemplar replay in the literature (40; 51; 12), during optimization on experience t, the buffer ℳM is concatenated directly onto the current-task dataset tD_t to form a single combined dataset t+=t∪ℳD_t^+=D_t . Mini-batches of size B are sampled uniformly from t+D_t^+. Consequently, the expected fraction of buffer samples in any given training batch is given by |ℳ|/(|t|+|ℳ|)|M|/(|D_t|+|M|). This approach naturally mirrors the data balance observed during cumulative joint training and automatically adapts as the buffer fills over successive experiences. D.2 Knowledge Distillation (Supervised: LwF) For knowledge distillation in the supervised continual learning case, we utilize Learning without Forgetting (LwF) 30. After training on experience t, a frozen copy of the model θt−1θ^t-1 is saved as the teacher. When training on experience t+1t+1, the teacher’s soft predictions over previously seen classes serve as targets for the current student model. Let z and prevz^prev denote the logits of the student and teacher models, respectively. The distillation loss for a previous task τ is the Kullback-Leibler (KL) divergence between the softened distributions: ℒKDτ=KL(σ(τprev/T)∥σ(τ/T)),L_KD^τ=KL\! (σ\! (z^prev_A_τ/T )\;\|\;σ\! (z_A_τ/T ) ), (1) where σ(⋅)σ(·) is the softmax function, T=2T=2 is the temperature scaling as proposed by the original authors, and τA_τ is the set of active class indices for task τ. The total loss is a weighted sum of the cross-entropy loss on the current experience and the distillation loss over all previously encountered tasks. To ensure stability, distillation is applied strictly over the output units τA_τ corresponding to classes the model has actively trained on. D.3 Knowledge Distillation (Self-Supervised: CaSSLe) To adapt distillation to the self-supervised regime, we employ the CaSSLe framework 18 using the Barlow Twins 54 objective. Barlow Twins learns representations by driving the empirical cross-correlation matrix C of two augmented views toward the identity matrix, utilizing an invariance term (diagonal) and a redundancy-reduction term (off-diagonal). In the CaSSLe decorrelative variant, at the end of experience t, the encoder fθ~f_ θ and projector hψ~h_ ψ are frozen to act as the teacher. During experience t+1t+1, a lightweight, randomly initialized MLP predictor gϕg_φ is added to the student model. The student processes the input through its encoder, projector, and the new predictor to yield predictions P. The teacher processes the same input to yield target projections ~ Z. The distillation loss applies the Barlow Twins objective (ℒBTL_BT) to align the student’s predictions with the teacher’s frozen projections: ℒdistill=12[ℒBT(A,~A)+ℒBT(B,~B)]L_distill= 12 [L_BT(P^A,\, Z^A)+L_BT(P^B,\, Z^B) ] (2) The predictor gϕg_φ is applied exclusively to the student network. This asymmetry serves two functions: it prevents representation collapse, and it decouples the student’s current-task projector from the rigid distillation constraint, allowing the student to adapt to new data while the predictor bridges the representation spaces. The total training objective is the sum of the standard Barlow Twins loss on the student’s current projections and the distillation loss ℒdistillL_distill. Appendix E Extended chunking evaluation In the main text Section 4, our empirical evaluation utilizes a 44-chunk data-incremental setup. To demonstrate that the co-observation effect is not an artifact of this specific sequence length, but rather a robust property of the fragmented optimization, we extend our evaluation to longer sequences. We replicate the supervised learning setup on ImageNet-100, but partition the global training dataset uniformly into 88 and 1212 identically distributed chunks, respectively. The fixed iteration budget is scaled proportionally to ensure the total optimization steps across the entire sequence remain consistent with the 44-chunk and joint-training baselines. Figure 8: The co-observation effect across extended sequences. Linear probing accuracy, reported as the mean and standard error over 3 seeds, for supervised learning on ImageNet-100 partitioned into 88 chunks (left) and 1212 chunks (right). The performance gap between the diagnostic ensemble (perfect retention) and incremental joint (simultaneous observation) persists consistently across longer sequences. As illustrated in Figure 8, increasing the granularity of the data partitions does not alleviate the representational deficit. Across both the 8-chunk and 12-chunk sequences, the perfect-retention ensemble baseline successfully mitigates catastrophic forgetting relative to the naive sequential model, yet consistently falls short of incremental joint training. This sustained gap confirms that the co-observation effect remains a fundamental limiting factor regardless of the specific length or fragmentation of the continual learning sequence. Appendix F Numerical results Table 1: SL-CIFAR100 - LP-accuracy[%] CIFAR100. Runs reported as mean ± standard error over 33 runs. Method Chunk 1 Chunk 2 Chunk 3 Chunk 4 Naive 60.80±0.3160.80 ± 0.31 63.54±0.1763.54 ± 0.17 63.92±0.2063.92 ± 0.20 64.64±0.1664.64 ± 0.16 Ensemble 60.78±0.3060.78 ± 0.30 65.93±0.2265.93 ± 0.22 68.19±0.3768.19 ± 0.37 70.39±0.0470.39 ± 0.04 Ensemble PCA 60.83±0.3760.83 ± 0.37 65.81±0.1265.81 ± 0.12 68.12±0.3168.12 ± 0.31 70.25±0.0670.25 ± 0.06 Incremental Joint 61.15±0.1661.15 ± 0.16 68.22±0.0368.22 ± 0.03 72.45±0.0372.45 ± 0.03 74.29±0.3174.29 ± 0.31 Joint (offline) – – – 75.18±0.5575.18 ± 0.55 Table 2: SL-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 33 runs. Method Chunk 1 Chunk 2 Chunk 3 Chunk 4 Naive 73.24±0.4873.24 ± 0.48 76.66±0.1676.66 ± 0.16 77.98±0.2977.98 ± 0.29 78.27±0.2178.27 ± 0.21 Ensemble 73.22±0.4273.22 ± 0.42 77.48±0.1677.48 ± 0.16 79.20±0.2479.20 ± 0.24 80.55±0.1080.55 ± 0.10 Ensemble PCA 73.25±0.4373.25 ± 0.43 77.47±0.2177.47 ± 0.21 79.15±0.2079.15 ± 0.20 80.57±0.1180.57 ± 0.11 Incremental Joint 73.24±0.4873.24 ± 0.48 79.58±0.1279.58 ± 0.12 81.85±0.1081.85 ± 0.10 82.75±0.1582.75 ± 0.15 Joint (offline) – – – 83.68±0.1183.68 ± 0.11 Replay (mt=1250m_t\!=\!1250) 73.07±0.2273.07 ± 0.22 76.99±0.1476.99 ± 0.14 78.35±0.6378.35 ± 0.63 79.41±0.2379.41 ± 0.23 Replay (mt=6250m_t\!=\!6250) 73.06±0.2273.06 ± 0.22 77.67±0.1277.67 ± 0.12 79.54±0.1379.54 ± 0.13 80.55±0.2380.55 ± 0.23 Replay (mt=12500m_t\!=\!12500) 73.25±0.0773.25 ± 0.07 78.05±0.0778.05 ± 0.07 80.68±0.1580.68 ± 0.15 81.27±0.2481.27 ± 0.24 LwF 73.35±0.2373.35 ± 0.23 77.41±0.1577.41 ± 0.15 79.44±0.2279.44 ± 0.22 79.93±0.2579.93 ± 0.25 Table 3: Barlow-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 33 runs. Method Chunk 1 Chunk 2 Chunk 3 Chunk 4 Naive 67.47±0.3867.47 ± 0.38 69.04±0.4669.04 ± 0.46 69.64±0.1469.64 ± 0.14 69.25±0.0669.25 ± 0.06 Ensemble 67.56±0.2967.56 ± 0.29 69.81±0.2969.81 ± 0.29 70.37±0.4370.37 ± 0.43 71.25±0.1971.25 ± 0.19 Ensemble PCA 67.57±0.3767.57 ± 0.37 70.20±0.3070.20 ± 0.30 71.43±0.3071.43 ± 0.30 71.53±0.3171.53 ± 0.31 Incremental Joint 67.19±0.4567.19 ± 0.45 71.61±0.3671.61 ± 0.36 73.28±0.3473.28 ± 0.34 73.50±0.4173.50 ± 0.41 Joint (offline) – – – 76.42±0.4076.42 ± 0.40 Replay (mt=1250m_t\!=\!1250) 67.63±0.4267.63 ± 0.42 69.25±0.2469.25 ± 0.24 69.83±0.1469.83 ± 0.14 69.68±0.1669.68 ± 0.16 Replay (mt=6250m_t\!=\!6250) 67.47±0.2967.47 ± 0.29 69.69±0.1269.69 ± 0.12 70.93±0.1770.93 ± 0.17 71.63±0.1571.63 ± 0.15 Replay (mt=12500m_t\!=\!12500) 67.78±0.5067.78 ± 0.50 70.93±0.0470.93 ± 0.04 71.99±0.2871.99 ± 0.28 72.59±0.0972.59 ± 0.09 CaSSLe 67.62±0.2567.62 ± 0.25 69.96±0.3969.96 ± 0.39 71.66±0.2871.66 ± 0.28 72.07±0.2372.07 ± 0.23 Table 4: IJEPA-IN-100 - LP-accuracy[%] IN-100 Runs reported as mean ± standard error over 66 runs for Naive, Ensemble (PCA), and (Incremental) Joint. And over 33 runs for Replay. Method Chunk 1 Chunk 2 Chunk 3 Chunk 4 Naive 55.87±0.2755.87 ± 0.27 55.00±0.3855.00 ± 0.38 54.63±0.6354.63 ± 0.63 53.83±0.6053.83 ± 0.60 Ensemble 55.88±0.2655.88 ± 0.26 58.33±0.4158.33 ± 0.41 59.11±0.5159.11 ± 0.51 59.82±0.4659.82 ± 0.46 Ensemble PCA 55.91±0.2655.91 ± 0.26 57.08±0.5557.08 ± 0.55 58.12±0.6758.12 ± 0.67 58.83±0.6958.83 ± 0.69 Incremental Joint 55.80±0.2855.80 ± 0.28 59.59±0.3259.59 ± 0.32 61.83±0.6261.83 ± 0.62 62.64±0.5562.64 ± 0.55 Joint (offline) – – – 69.35±0.3869.35 ± 0.38 Replay (mt=1250m_t\!=\!1250) 55.93±0.6955.93 ± 0.69 56.18±0.3456.18 ± 0.34 56.28±0.8856.28 ± 0.88 54.79±0.9954.79 ± 0.99 Replay (mt=6250m_t\!=\!6250) 55.85±0.3155.85 ± 0.31 57.07±0.7157.07 ± 0.71 57.29±0.3757.29 ± 0.37 56.59±0.6556.59 ± 0.65 Replay (mt=12500m_t\!=\!12500) 56.56±0.1056.56 ± 0.10 58.72±0.0658.72 ± 0.06 59.41±0.6759.41 ± 0.67 60.21±0.9960.21 ± 0.99