Paper deep dive
CE$^4$L: Continual Ego, Exo, and Ego-Exo Learning
Hongwei Yan, Kanglei Zhou, Yuchen Liu, Qingyu Shi, Yi Zhong, Liyuan Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/27/2026, 2:46:37 AM
Summary
The paper introduces CE4L, a unified benchmark for Continual Ego, Exo, and Ego-Exo Learning, addressing the lack of multi-view continual learning evaluations in embodied agents. It proposes VISTA, a parameter-efficient method using task-specific adapters and training-free subspace routing to handle cross-view shifts and catastrophic forgetting. VISTA achieves state-of-the-art performance across four tasks: skill assessment, action segmentation, cross-view association, and action anticipation/planning.
Entities (10)
Relation Signals (10)
Hongwei Yan → affiliatedwith → Tsinghua University
confidence 95% · Hongwei Yan Affiliation: School of Life Sciences... Tsinghua University
CE4L → builtondataset → EgoExoLearn
confidence 95% · Built upon the EgoExoLearn (Huang et al., 2024) video dataset
CE4L → includestask → Cross-View Association
confidence 95% · spanning four representative tasks: ... cross-view association
CE4L → includestask → Action Anticipation
confidence 95% · spanning four representative tasks: ... action anticipation & planning
CE4L → includestask → action planning
confidence 95% · spanning four representative tasks: ... action anticipation & planning
CE4L → includestask → Skill Assessment
confidence 95% · CE4L highlights challenges... spanning four representative tasks: cross-view referenced skill assessment
CE4L → includestask → Action Segmentation
confidence 95% · spanning four representative tasks: ... temporal action segmentation
VisTA → usescomponent → Task Adapters
confidence 95% · VISTA stores task-specific updates in lightweight adapters
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Perception for embodied agents is video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shifts. Yet continual learning (CL) remains dominated by exo-only recognition tasks, obscuring behavior under these real-world coupled shifts. We introduce Continual Ego, E}xo, and Ego-Exo Learning (CE$^4$L), a unified multi-view CL benchmark spanning four representative tasks: cross-view referenced skill assessment, temporal action segmentation, cross-view association, and action anticipation & planning. CE$^4$L highlights challenges largely absent in prior CL benchmarks, including cross-view correspondence, view-dependent asynchrony, and heterogeneous semantic objectives. To this end, we propose Video Incremental Subspace-routed Task Adapters (VISTA), a parameter-efficient baseline method that stores task-specific updates in lightweight adapters and performs training-free routing via residual distance to task-specific whitened subspaces estimated from second-order statistics. Extensive experiments demonstrate the significantly varied efficacy of representative CL methods across CE$^4$L settings, while VISTA is consistently competitive and achieves state-of-the-art overall performance. Our source code for benchmarks and methods is available at this https URL .
Tags
Links
- Source: https://arxiv.org/abs/2609.23492v1
- Canonical: https://arxiv.org/abs/2609.23492v1
Trouble viewing inline? Open PDF directly →
Full Text
125,004 characters extracted from source content.
Expand or collapse full text
CE4L: Continual Ego, Exo, and Ego-Exo Learning Hongwei Yan Affiliation: School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China Affiliation: Tsinghua-Peking Center for Life Sciences Kanglei Zhou Affiliation: Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China Yuchen Liu Affiliation: Department of Computer Science, Tsinghua University, Beijing, China Qingyu Shi Affiliation: Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China Yi Zhong Affiliation: School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China Affiliation: Tsinghua-Peking Center for Life Sciences Liyuan Wang Affiliation: Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China Correspondence to: wly19@tsinghua.org.cn Abstract Perception for embodied agents is video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shifts. Yet continual learning (CL) remains dominated by exo-only recognition tasks, obscuring behavior under these real-world coupled shifts. We introduce Continual Ego, Exo, and Ego-Exo Learning (CE4L), a unified multi-view CL benchmark spanning four representative tasks: cross-view referenced skill assessment, temporal action segmentation, cross-view association, and action anticipation & planning. CE4L highlights challenges largely absent in prior CL benchmarks, including cross-view correspondence, view-dependent asynchrony, and heterogeneous semantic objectives. To this end, we propose Video Incremental Subspace-routed Task Adapters (VISTA), a parameter-efficient baseline method that stores task-specific updates in lightweight adapters and performs training-free routing via residual distance to task-specific whitened subspaces estimated from second-order statistics. Extensive experiments demonstrate the significantly varied efficacy of representative CL methods across CE4L settings, while VISTA is consistently competitive and achieves state-of-the-art overall performance. Our source code for benchmarks and methods is available at CE4L . Keywords: Continual Learning, Catastrophic Forgetting, Egocentric Video Learning Figure 1: CE4L benchmark overview. By constructing continuous task stream for both ego- and exocentric videos, CE4L comprehensively evaluates the within-view performance and cross-view transferability of continual learners through four diverse benchmarks. 1 Introduction The pursuit of embodied intelligence is gradually approaching real-world scenarios that mirror human life, requiring agents to learn continually from incoming experiences, adapt to new environments, acquire skills by observing expert demonstrations, and sustain mastered capabilities (Kim et al., 2024; Cheang et al., 2024; Black et al., 2026; Intelligence et al., 2025; Liu et al., 2025). However, the continual learning (CL) process faces a unique challenge in the embodied context: unlike the well-distributed and ideally static datasets typical of standard supervised training, the sensory signals received by an embodied agent, including visual stimuli, are temporally correlated and highly dependent on its physical viewpoint, a characteristic central to egocentric vision (Punamiya et al., 2025; Yang et al., 2025). While recent large-scale pretraining has driven remarkable progress in understanding such egocentric videos (Pramanick et al., 2023; Zhao et al., 2023; Chen et al., 2022), existing CL protocols have not kept pace for this domain. They often abstract away two properties that define embodied experience: strong viewpoint bias and heterogeneous semantic objectives (Park et al., 2021; Villa et al., 2022; Maraghi and Faez, 2022; Pian et al., 2023). Bridging this gap is not a straightforward matter of scaling existing CL paradigms from 2D images to video, where viewpoint bias fundamentally changes what should be preserved across time (Xu et al., 2023; Grauman et al., 2024; Plizzari et al., 2024). A learner can maintain performance on previously seen tasks within the training view while losing the cross-view invariance that makes its representation transferable, or conversely it can appear to forget simply because evaluation shifts to a mismatched view. This entanglement makes it difficult to interpret CL progress using single-view and single-task paradigms alone, and it obscures which algorithmic ingredients genuinely improve continual video learning rather than overfitting to the current view. To address these challenges, we introduce Continual Ego, Exo and Ego-Exo Learning (CE4L), a comprehensive benchmark suite designed to facilitate CL in the context of multi-view video understanding. Built upon the EgoExoLearn (Huang et al., 2024) video dataset, CE4L establishes a rigorous evaluation framework that explicitly disentangles within-view retention from cross-view generalization. We instantiate four complementary problems as a sequence of tasks: continual pairwise skill assessment, temporal action segmentation, cross-view association, and action anticipation & planning. Crucially, our protocol supports both single-view and multi-view training regimes, enabling a grounded assessment of a model’s cross-view transferability as well as its plasticity to viewpoint variations. Under CE4L, we perform a systematic study of representative continual paradigms and reveal a critical insight: retaining past performance is often hindered by the learner drifting toward view-specific shortcuts that fail to transfer. Concretely, the relative efficacy of replay, regularization, and ensemble-based methods vary substantially across objectives (Tables 3–5), and in several scenarios cross-view evaluation exposes viewpoint-conditioned mismatch that can persist even when within-view forgetting is reduced (Figs. 6–8). These findings motivate evaluation protocols that go beyond average accuracy on single-view recognition and make cross-view transfer explicit. Motivated by these findings, we propose Video Incremental Subspace-routed Task Adapters (VISTA), a parameter-efficient continual learner that serves as a strong replay-free baseline for continual multi-view video learning. VISTA stores task knowledge in lightweight adapters and performs training-free inference by routing inputs to a small adapter ensemble via residual distances to task-specific feature subspaces. Across CE4L, VISTA achieves the strongest overall performance among CL baselines while remaining efficient in memory and compute (Tables 3–5; Table 10). We summarize our contributions as follows: I. We introduce CE4L, a unified benchmark that evaluates CL across heterogeneous video tasks (ego, exo, and mixed-view), providing protocols that separate forgetting from cross-view transfer capability. I. We propose VISTA, an efficient adapter-based method that utilizes task subspace geometry for robust, training-free task inference in high-dimensional video representations. I. We provide a comprehensive evaluation of representative CL baselines under the CE4L protocols and diagnose failure modes related to view-conditioned interference, opening up a new avenue for embodied CL. 2 Related Work Egocentric and ego-exo video learning. Egocentric video understanding has progressed from curated first-person datasets to large-scale, multi-view procedural corpora that connect first-person observations with third-person demonstrations (Damen et al., 2022; Liu et al., 2022; Huang et al., 2024; Grauman et al., 2024). Existing benchmarks and representation models have enabled strong offline perception, cross-view association, and skill understanding (Pramanick et al., 2023; Zhao et al., 2023; Chen et al., 2022), but they rarely evaluate how such capabilities are retained when tasks and viewpoints evolve continually. Video continual learning. Video continual learning extends continual learning beyond static images by introducing temporal dynamics, high-dimensional inputs, and video-specific forgetting patterns (Park et al., 2021; Villa et al., 2022; Wang et al., 2024a). Prior work has explored replay, regularization, prompt tuning, and video-language continual adaptation (Pei et al., 2023; Villa et al., 2023; Tang et al., 2024; Tan et al., 2025), yet most protocols remain centered on single-view recognition rather than the coupled viewpoint and objective shifts faced by embodied agents. Pretraining-based continual learning. Recent continual learning methods increasingly adapt frozen pretrained models with lightweight prompts or adapters, often relying on task inference or routing to select task-specific components (Wang et al., 2022d; Wang et al., 2022c; Wang et al., 2022a; Wang et al., 2023b). In this work, we follow this parameter-efficient direction, but use training-free subspace routing for multi-view video streams where a learned router may itself be vulnerable to non-stationarity (McDonnell et al., 2024; Zhuang et al., 2022). A more detailed literature review is provided in Appendix A. Table 1: CE4L benchmark configurations. Details of each multi-view video continual learning problem constructed in CE4L. CE4L Scenario Info. Skill Assessment Action Segmentation Cross-View Association Action Anticipation Action Planning Number of Tasks 4 4 5 8 4 Task Division Action group Procedural task Procedural task Procedural task Procedural task Backbone Network I3D+RAAN I3D+MS-TCN TimeSformer-B + CLIP-text CLIP+TA3N CLIP+TA3N Evaluation Metrics Ranking Acc (↑ ) Acc/Edit/F1 (↑ ) Top-1 Acc (↑ ) Top-5 Recall (↑ ) ED@8 (↓ ) 3 CE4L Benchmark 3.1 General Setup Traditional CL benchmarks focus primarily on exocentric recognition with limited distribution shifts. This abstraction, however, fails under structured viewpoint changes inherent to embodied agents, where underlying procedure can appear substantially different across ego and exo perspectives. A learner may preserve within-view performance yet lose transferable cross-view knowledge. CE4L explicitly addresses this by coupling task-incremental streams with evaluation metrics that separately measure within-view retention, zero-shot cross-view generalization, and a co-training baseline regime that provides both views to distinguish stability from the lack of multi-view evidence. CE4L comprises four CL benchmarks that emphasize complementary challenges in cross-perspective video learning (see Fig. 1): cross-view referenced skill assessment, temporal action segmentation, cross-view association, and action anticipation & planning. These four benchmarks are built upon EgoExoLearn (Huang et al., 2024), which is collected in an ego-exo demonstration-following setting and covers eight high-level procedural tasks spanning both daily kitchens (five tasks) and professional laboratories (three tasks). The proposed CE4L involves two kinds of task streams: (I) tasks with full procedures leverage the inherent eight-task partition; and (I) tasks with trimmed clips are instead split by grouping action categories into disjoint subsets. Detailed configurations are summarized in Table 1. Formally, we denote a pretrained video encoder as fθf_θ, and a task head as gψg_ψ, yielding a predictor gψ∘fθg_ψ\! f_θ. A continual data stream comprises a sequence of task datasets tt=1T\D_t\_t=1^T presented in order11 1 In practice, the CL stream consists of sequential sessions formed by shuffling tasks. We use “task” for narrative convenience, reserving “session” only for disambiguation.. After training on task t, we evaluate on the union of seen tasks and record the performance matrix A, where At,jA_t,j denotes performance on task j≤tj≤ t after learning up to t. We record At,jj≤t\A_t,j\_j≤ t, and calculate final average task performance A¯T A_T and average task forgetting F¯T F_T for all benchmarks and corresponding metrics. More details of the benchmark setup are provided in Appendix B. 3.2 Continual Skill Assessment Problem setup. This task is formulated as a stream of action-specific pairwise rankings. Each training sample consists of two egocentric clips v1,v2\v_1,v_2\ of the same atomic action; the model must predict which demonstrates higher skill. When an exocentric reference is available, the model compares egocentric candidates against it, requiring robustness to asynchrony and viewpoint shifts. We partition action categories into disjoint subsets to form the continual data stream, using ranking accuracy as the primary metric. Backbone architecture. We adopt the ranking-based model RAAN (Doughty et al., 2019) built on pretrained I3D features (Carreira and Zisserman, 2017), which outputs a scalar score per clip (Huang et al., 2024); prediction is the score difference between the two egocentric candidates. We consider two reference-aware variants: a triplet-style objective (TL) that pulls the higher-skill egocentric embedding closer to the exocentric reference than the lower-skill one, and a relation-network variant (RN) that conditions scoring on the reference via feature interaction (Sung et al., 2018). This benchmark isolates CL behavior under pairwise supervision with an explicit cross-view reference. 3.3 Continual Action Segmentation Problem setup. Continual temporal action segmentation evaluates dense step parsing for long procedural videos. Given a video v, the model predicts an action label for each frame. We construct a task stream over procedural activities and train sequentially, where each task introduces a new activity with its own action distribution and temporal structure. We report within-view performance and the cross-view/co-training regimes defined in Sec. 3.1. Backbone architecture. We use MS-TCN (Farha and Gall, 2019) on per-frame I3D features (Carreira and Zisserman, 2017). Performance is measured by frame-wise accuracy, edit score, and segmental F1 at multiple overlap thresholds, summarized into task-wise aggregates for stable continual comparisons across different activity temporal resolution. Figure 2: VISTA method overview. VISTA employs task-specific feature adapters for CL and a training-free router for inference. The router assigns mixture weights by calculating the residual distance between input features and task subspaces, defined by accumulated second order statistics. Finally, adapted features are processed by the shared learner module which is frozen after the first task. 3.4 Continual Cross-View Association Problem setup. Cross-view association requires retrieving semantically corresponding clips across ego- and exocentric videos under temporal asynchrony and large viewpoint changes. We formulate it as multiple-choice retrieval: given a query clip from one view, select its match in the other view from a fixed-size candidate set (Huang et al., 2024; Xu et al., 2021; Lin et al., 2022; Pramanick et al., 2023). We report Top-1 accuracy for both Ego→ and Exo→ directions, and form a continual data stream by grouping queries by their underlying procedural activities. Backbone architecture. We use a pretrained dual-encoder video-text backbone with a time-aware visual encoder (TimeSformer (Bertasius et al., 2021)) and a CLIP text encoder (Radford et al., 2021) to obtain clip embeddings (Lin et al., 2022), trained with contrastive objectives under paired supervision. At inference, association reduces to similarity between query and candidate embeddings, directly probing whether sequential updates preserve cross-view alignment. 3.5 Continual Action Anticipation & Planning Problem setup. CE4L includes both short-horizon action anticipation and long-horizon action planning (Huang et al., 2024). For anticipation, the input is a short context window and the target is the next action category (verb and noun), treated as multi-label prediction and evaluated by Top-5 recall (Damen et al., 2022). For planning, the input is also a context window but the target is a fixed-length sequence of future coarse steps, evaluated by an edit-distance style metric aggregated over horizons (Grauman et al., 2022). In both problems, we define the continual data stream over procedural activities and again implement the within-view, cross-view, and co-training regimes described in Sec. 3. Backbone architecture. We adopt a lightweight temporal relation model TA3N (Chen et al., 2019) on top of a frozen CLIP encoder (Radford et al., 2021) at a fixed frame rate. The model aggregates temporal segments into a video-level representation and predicts either the next-step category (anticipation) or the future step sequence (planning). 4 VISTA Method In this section, we present the proposed VISTA for multi-view video CL and give detailed description of how it adapts the pretrained representation efficiently through adapters and assembles them by utilizing the corresponding subspace geometry. See Fig. 2 for an overview. 4.1 Task Aware Adapters We build VISTA on top of a pretrained video encoder fθf_θ, which maps an input clip v to a sequence of features x=fθ(v)∈ℝL×Cx=f_θ(v) ^L× C, where L is the temporal length and C is the channel dimension. In CL, directly updating θ of a large pretrained model is data intensive, and fine-tuning across a non-stationary data stream often causes severe interference, especially when the stream mixes heterogeneous tasks and viewpoints (Zhou et al., 2024a). VISTA therefore introduces a lightweight task-indexed adapter bank aϕtt=1T\a_ _t\_t=1^T that modulates intermediate features before converting them to task head gψg_ψ while keeping adaptation plasticity localized to each task in a stream of T tasks (Wang et al., 2025c; Li et al., 2025; Wang et al., 2025b). Each adapter is a residual bottleneck module applied independently at every time step (Rebuffi et al., 2017). Let xi∈ℝCx_i ^C denote the feature vector at time index i, aϕ(x)i=xi+αWupσ(WdownLN(xi)),a_φ(x)_i=x_i+α\,W_up\,σ\! (W_down\,LN(x_i) ), (1) where LN(⋅)LN(·) is layer normalization, σ(⋅)σ(·) is a GELU nonlinearity, Wdown∈ℝr×CW_down ^r× C and Wup∈ℝC×rW_up ^C× r are trainable linear maps with bottleneck dimension r≪Cr C, and α∈ℝα is a learnable residual scale initialized to 00 so that aϕa_φ starts from an identity mapping. This parameterization ensures that each newly introduced adapter can be optimized stably without disrupting the pretrained representation geometry at initialization, while still permitting task-specific specialization as training proceeds in an efficient way. During CL, when the data stream is at task t, aϕta_ _t is initialized with parameter of the last one aϕt−1a_ _t-1, supporting cross-task knowledge transfer with a warm-start. VISTA only updates the corresponding aϕta_ _t in each task, and freeze the shared head gψg_ψ after the first task to avoid disrupting the representation calibrated by the first adapter. At inference time, the task identity is not available. Instead, VISTA predicts a small set of candidate tasks and forms a mixture of adapters to produce an adapted feature sequence (Yu et al., 2024; Yu et al., 2025). Concretely, given a mixture specification (tℓ,πℓ)ℓ=1K\(t_ , _ )\_ =1^K with ∑ℓπℓ=1 _ _ =1 and πℓ≥0 _ ≥ 0, we compute x~=∑ℓ=1Kπℓaϕtℓ(x), x= _ =1^K _ \,a_ _t_ (x), (2) which reduces to the task-aware adapter when the router is confident (K=1K=1) and yields a soft interpolation when the input lies near task boundaries. The key remaining question is how to infer (tℓ,πℓ)\(t_ , _ )\ without labels, which is addressed by our subspace residual router below. Table 2: Performance of CL methods on the continual skill assessment benchmark. We report the final average ranking accuracy A¯T A_T (%,↑\%, ) and forgetting metrics F¯T F_T (%,↓\%, ). Method Final Average Accuracy A¯T(%,↑) A_T(\%, ) Final Average Forgetting F¯T(%,↓) F_T(\%, ) RAAN +RN +TL RAAN +RN +TL Joint 80.42±0.1480.42_± 0.14 80.17±0.0680.17_± 0.06 80.41±0.0880.41_± 0.08 −−-- −−-- −−-- Fine-tuning 72.94±2.1572.94_± 2.15 72.45±2.2172.45_± 2.21 71.63±2.1371.63_± 2.13 11.30±1.3511.30_± 1.35 10.97±1.6410.97_± 1.64 13.20±1.9413.20_± 1.94 ER 77.24±0.5677.24_± 0.56 77.39±0.5077.39_± 0.50 78.27±0.7378.27_± 0.73 3.94±2.30 03.94_± 2.30 3.13±2.83 03.13_± 2.83 2.39±0.95 02.39_± 0.95 DER++ 79.46¯±0.26 79.46_± 0.26 78.60±0.3078.60_± 0.30 79.47¯±0.07 79.47_± 0.07 0.98±1.24 00.98_± 1.24 1.17±1.70 01.17_± 1.70 1.09±0.81 01.09_± 0.81 EWC 73.34±2.2873.34_± 2.28 72.59±1.9572.59_± 1.95 71.05±2.0071.05_± 2.00 10.62±1.7510.62_± 1.75 11.32±2.2611.32_± 2.26 14.22±2.4414.22_± 2.44 LwF 72.61±1.7372.61_± 1.73 68.87±2.5868.87_± 2.58 71.83±2.1171.83_± 2.11 2.21±3.63 02.21_± 3.63 12.52±3.2012.52_± 3.20 0.49±2.24 00.49_± 2.24 L2P+ 72.03±1.4872.03_± 1.48 71.43±2.5671.43_± 2.56 71.38±0.9771.38_± 0.97 11.69±3.6411.69_± 3.64 12.52±1.2012.52_± 1.20 14.22±3.8114.22_± 3.81 S-Prompt+ 79.17±0.3079.17_± 0.30 77.75¯±0.41 77.75_± 0.41 78.05±0.7878.05_± 0.78 0.05¯±0.03 0 0.05_± 0.03 0.00±0.21 00.00_± 0.21 0.13¯±0.17 0 0.13_± 0.17 VISTA (ours) 80.38±0.3280.38_± 0.32 79.58±0.8179.58_± 0.81 80.41±0.4880.41_± 0.48 -0.19±0.51-0.19_± 0.51 0.19¯±0.19 0 0.19_± 0.19 0.12±0.14 00.12_± 0.14 Table 3: Performance of CL methods on the continual temporal action segmentation benchmark (test split). We report the final average frame-wise accuracy A¯T A_T (%,↑\%, ). Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 66.12±0.2666.12_± 0.26 22.66±3.8322.66_± 3.83 24.51±0.9224.51_± 0.92 36.98±0.6636.98_± 0.66 66.41±0.6666.41_± 0.66 38.26±1.1238.26_± 1.12 Fine-tuning 36.26±0.5836.26_± 0.58 10.16±1.9110.16_± 1.91 13.89±2.2013.89_± 2.20 20.05±1.1120.05_± 1.11 35.51±1.8235.51_± 1.82 20.32±1.7520.32_± 1.75 ER 45.19±3.0345.19_± 3.03 10.87±6.3410.87_± 6.34 18.43±2.2418.43_± 2.24 30.76±0.9530.76_± 0.95 43.93±5.2843.93_± 5.28 29.67±0.7929.67_± 0.79 DER++ 43.97±4.7643.97_± 4.76 10.02±2.0510.02_± 2.05 18.23±5.1218.23_± 5.12 30.71±2.7830.71_± 2.78 45.22±3.1845.22_± 3.18 30.04±3.9430.04_± 3.94 EWC 41.35±8.7341.35_± 8.73 8.15±2.77 08.15_± 2.77 19.67¯±5.05 19.67_± 5.05 30.88±2.4430.88_± 2.44 45.11±2.8545.11_± 2.85 30.35±2.5830.35_± 2.58 LwF 44.95±3.3944.95_± 3.39 12.65¯±1.65 12.65_± 1.65 19.07±2.4919.07_± 2.49 30.53±1.8630.53_± 1.86 45.30¯±2.83 45.30_± 2.83 30.35±4.5330.35_± 4.53 L2P+ 47.46¯±2.00 47.46_± 2.00 12.33±3.1012.33_± 3.10 18.67±3.2318.67_± 3.23 32.18±1.5732.18_± 1.57 44.88±3.0844.88_± 3.08 30.84¯±2.05 30.84_± 2.05 S-Prompt+ 38.69±0.9738.69_± 0.97 7.52±2.10 07.52_± 2.10 15.38±1.7415.38_± 1.74 28.01±1.0328.01_± 1.03 40.85±0.7040.85_± 0.70 23.72±0.5523.72_± 0.55 VISTA (ours) 47.72±0.7947.72_± 0.79 12.93±2.3312.93_± 2.33 20.03±1.1620.03_± 1.16 30.90¯±1.75 30.90_± 1.75 45.35±1.3445.35_± 1.34 31.41±0.8731.41_± 0.87 Table 4: Performance of CL methods on the continual cross-view association benchmark (test split). We report the final average Top-1 association accuracy A¯T(%,↑) A_T(\%, ) for both query directions. Method Train: Ego Train: Exo Train: Ego+Exo Ego→ Exo→ Ego→ Exo→ Ego→ Exo→ Joint 40.07±0.9040.07_± 0.90 37.97±1.0537.97_± 1.05 38.47±0.4138.47_± 0.41 45.87±0.5945.87_± 0.59 47.10±0.3647.10_± 0.36 46.53±2.1646.53_± 2.16 Fine-tuning 32.63±0.6132.63_± 0.61 25.30±2.4825.30_± 2.48 31.37±1.3331.37_± 1.33 36.47±2.0436.47_± 2.04 41.13±1.9941.13_± 1.99 35.57±0.1735.57_± 0.17 ER 34.20±1.2234.20_± 1.22 27.27±1.5227.27_± 1.52 32.53¯±1.67 32.53_± 1.67 36.80¯±1.71 36.80_± 1.71 43.47¯±0.25 43.47_± 0.25 38.83±1.4638.83_± 1.46 DER++ 33.90±2.2033.90_± 2.20 26.73±1.1426.73_± 1.14 31.90±1.6331.90_± 1.63 36.23±1.4236.23_± 1.42 43.10±1.2043.10_± 1.20 37.97±1.4437.97_± 1.44 EWC 32.40±0.6732.40_± 0.67 23.63±0.8523.63_± 0.85 29.83±0.2929.83_± 0.29 34.40±2.1334.40_± 2.13 41.77±0.9041.77_± 0.90 36.13±1.0236.13_± 1.02 LwF 30.90±3.8630.90_± 3.86 25.27±3.0825.27_± 3.08 31.50±0.7031.50_± 0.70 36.73±2.0736.73_± 2.07 41.20±1.2841.20_± 1.28 35.20±0.1435.20_± 0.14 L2P+ 33.77±0.8833.77_± 0.88 25.13±0.7825.13_± 0.78 29.70±1.7729.70_± 1.77 33.20±2.6833.20_± 2.68 39.90±0.8539.90_± 0.85 36.17±0.9736.17_± 0.97 S-Prompt+ 34.93¯±0.98 34.93_± 0.98 30.33¯±0.80 30.33_± 0.80 30.78±1.4230.78_± 1.42 31.26±1.2431.26_± 1.24 40.57±0.3340.57_± 0.33 36.60±1.1536.60_± 1.15 VISTA (ours) 34.97±1.1034.97_± 1.10 30.37±0.8330.37_± 0.83 34.60±1.8934.60_± 1.89 37.67±6.3837.67_± 6.38 44.74±1.1844.74_± 1.18 38.29¯±1.16 38.29_± 1.16 Table 5: Performance of CL methods on the continual action planning benchmark. We report the final average edit distance ED@8 (↓ ) over a fixed length of 8 steps. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 82.13±0.5482.13_± 0.54 84.54±0.3884.54_± 0.38 84.70±0.2284.70_± 0.22 76.15±0.4976.15_± 0.49 81.60±0.2481.60_± 0.24 76.14±0.9776.14_± 0.97 Fine-tuning 86.69±0.5386.69_± 0.53 87.91±0.3487.91_± 0.34 88.33±0.3688.33_± 0.36 83.23±0.4583.23_± 0.45 85.53±0.5685.53_± 0.56 82.68±1.2182.68_± 1.21 ER 83.14±0.3183.14_± 0.31 86.84±0.7286.84_± 0.72 88.01±0.7788.01_± 0.77 81.98±0.6181.98_± 0.61 83.47±0.6483.47_± 0.64 80.70±0.4780.70_± 0.47 DER++ 82.64¯±0.10 82.64_± 0.10 86.97±0.8086.97_± 0.80 87.16±0.8787.16_± 0.87 81.01¯±0.31 81.01_± 0.31 82.70¯±0.18 82.70_± 0.18 79.04±1.4079.04_± 1.40 EWC 83.81±0.6183.81_± 0.61 86.03±0.4386.03_± 0.43 88.67±0.3988.67_± 0.39 81.78±0.6281.78_± 0.62 84.60±0.3784.60_± 0.37 81.88±1.1781.88_± 1.17 LwF 82.42±0.6482.42_± 0.64 85.89¯±0.60 85.89_± 0.60 86.36±1.2086.36_± 1.20 82.31±0.6582.31_± 0.65 82.03±0.6182.03_± 0.61 80.67¯±1.31 80.67_± 1.31 L2P+ 83.51±1.1683.51_± 1.16 86.08±0.2886.08_± 0.28 86.61±1.0786.61_± 1.07 81.94±0.5381.94_± 0.53 83.76±1.1683.76_± 1.16 82.38±3.5882.38_± 3.58 S-Prompt+ 84.02±0.5184.02_± 0.51 87.33±1.0287.33_± 1.02 85.07¯±0.64 85.07_± 0.64 82.18±0.8382.18_± 0.83 85.40±0.5985.40_± 0.59 81.57±0.8081.57_± 0.80 VISTA (ours) 83.02±0.5283.02_± 0.52 85.46±0.7785.46_± 0.77 84.09±1.3484.09_± 1.34 80.36±0.7180.36_± 0.71 82.81±0.2182.81_± 0.21 80.74±0.9280.74_± 0.92 Table 6: Overall performance of representative CL methods on the continual action anticipation. We report the final average Top-5 recall (%,↑\%, ) for verb and noun prediction evaluated on egocentric and exocentric videos. -V, predict verb category. -N, predict noun category. Method Train: Ego Train: Exo Train: Ego+Exo Ego-V Ego-N Exo-V Exo-N Ego-V Ego-N Exo-V Exo-N Ego-V Ego-N Exo-V Exo-N Joint 33.40±0.0333.40_± 0.03 29.88±0.1629.88_± 0.16 32.68±0.0332.68_± 0.03 29.32±0.1129.32_± 0.11 35.15±0.0535.15_± 0.05 28.94±0.0528.94_± 0.05 33.09±0.0833.09_± 0.08 28.29±0.0928.29_± 0.09 35.50±0.4135.50_± 0.41 27.80±0.1727.80_± 0.17 33.12±0.2933.12_± 0.29 27.11±0.1227.11_± 0.12 Fine-tuning 29.83±1.0529.83_± 1.05 20.77±0.9920.77_± 0.99 30.47±0.8530.47_± 0.85 20.34±0.8120.34_± 0.81 31.55±1.4931.55_± 1.49 20.33±0.9020.33_± 0.90 30.87±0.9830.87_± 0.98 19.87±0.7719.87_± 0.77 31.25±1.1531.25_± 1.15 19.26±0.6719.26_± 0.67 30.30±0.9130.30_± 0.91 18.82±0.5418.82_± 0.54 ER 31.15±0.4931.15_± 0.49 26.47±0.4726.47_± 0.47 30.84¯±0.23 30.84_± 0.23 26.50±0.2126.50_± 0.21 34.39¯±0.39 34.39_± 0.39 26.01±0.3626.01_± 0.36 32.77¯±0.09 32.77_± 0.09 25.56¯±0.21 25.56_± 0.21 34.49¯±0.53 34.49_± 0.53 25.29±0.2425.29_± 0.24 32.70¯±0.15 32.70_± 0.15 24.75¯±0.23 24.75_± 0.23 DER++ 31.31¯±0.36 31.31_± 0.36 26.91¯±0.69 26.91_± 0.69 30.68±0.2230.68_± 0.22 26.16¯±0.56 26.16_± 0.56 34.02±0.3334.02_± 0.33 25.54±0.7825.54_± 0.78 32.44±0.0532.44_± 0.05 25.20±0.6025.20_± 0.60 33.70±0.3733.70_± 0.37 24.55±0.5324.55_± 0.53 32.37±0.2632.37_± 0.26 24.13±0.3524.13_± 0.35 EWC 29.72±1.1229.72_± 1.12 20.63±1.0020.63_± 1.00 30.16±0.3730.16_± 0.37 20.19±0.8420.19_± 0.84 31.69±1.4331.69_± 1.43 20.41±0.8920.41_± 0.89 30.63±1.1730.63_± 1.17 19.95±0.7519.95_± 0.75 31.29±1.0431.29_± 1.04 19.96±0.4819.96_± 0.48 30.64±0.9530.64_± 0.95 19.44±0.2519.44_± 0.25 LwF 29.84±0.7729.84_± 0.77 22.24±0.7722.24_± 0.77 29.75±0.7529.75_± 0.75 21.79±0.7421.79_± 0.74 30.47±0.7130.47_± 0.71 21.17±0.5521.17_± 0.55 30.27±0.3030.27_± 0.30 20.67±0.3520.67_± 0.35 30.57±0.2130.57_± 0.21 21.38±1.4621.38_± 1.46 31.19±1.3231.19_± 1.32 20.58±1.1620.58_± 1.16 L2P+ 28.76±1.1428.76_± 1.14 16.24±1.1816.24_± 1.18 27.88±1.2027.88_± 1.20 15.50±1.3415.50_± 1.34 30.03±2.2430.03_± 2.24 16.57±1.1416.57_± 1.14 28.06±1.8228.06_± 1.82 15.59±1.1715.59_± 1.17 28.51±1.8928.51_± 1.89 16.19±0.8016.19_± 0.80 26.12±0.9326.12_± 0.93 15.21±0.6215.21_± 0.62 S-Prompt+ 30.41±1.1130.41_± 1.11 18.36±1.7218.36_± 1.72 27.99±1.6227.99_± 1.62 18.23±1.3518.23_± 1.35 32.64±3.0732.64_± 3.07 16.80±2.6216.80_± 2.62 28.44±2.4128.44_± 2.41 16.82±2.4016.82_± 2.40 31.83±2.2431.83_± 2.24 16.49±2.4316.49_± 2.43 27.64±1.4627.64_± 1.46 16.36±2.0216.36_± 2.02 VISTA (ours) 32.37±4.7432.37_± 4.74 28.08±2.1528.08_± 2.15 34.83±6.7134.83_± 6.71 25.12±2.1525.12_± 2.15 35.37±3.6935.37_± 3.69 25.69¯±1.57 25.69_± 1.57 34.46±3.6334.46_± 3.63 26.40±1.5326.40_± 1.53 35.23±1.0535.23_± 1.05 25.23¯±1.33 25.23_± 1.33 32.77±3.3732.77_± 3.37 26.01±1.4826.01_± 1.48 Table 7: Ablation study of core design choices in VISTA on CE4L. We evaluate component combinations of subspace router and task adapters under the Ego+Exo setting across all four CE4L scenarios. All numbers are reported as average performance after the last task. Subspace Task Skill Segment (Ego+Exo) Association (Ego+Exo) Planning (Ego+Exo) Anticipation (Ego+Exo) Router Adapters RAAN (↑ ) Ego (↑ ) Exo (↑ ) Ego→ (↑ ) Exo→ (↑ ) Ego (↓ ) Exo (↓ ) Ego-V (↑ ) Ego-N (↑ ) Exo-V (↑ ) Exo-N (↑ ) × × 72.94±2.1572.94_± 2.15 35.51±1.8235.51_± 1.82 20.32±1.7520.32_± 1.75 41.13±1.9941.13_± 1.99 35.57±0.1735.57_± 0.17 85.53±0.5685.53_± 0.56 82.68±1.2182.68_± 1.21 31.25±1.1531.25_± 1.15 19.26±0.6719.26_± 0.67 30.30±0.9130.30_± 0.91 18.82±0.5418.82_± 0.54 Random ✓ 69.41±1.2769.41_± 1.27 38.03±1.8238.03_± 1.82 26.02±3.8026.02_± 3.80 41.00±0.3641.00_± 0.36 36.90±0.9236.90_± 0.92 84.06±0.0684.06_± 0.06 82.99±0.9882.99_± 0.98 28.66±1.7628.66_± 1.76 16.01±3.1016.01_± 3.10 28.15±1.9328.15_± 1.93 16.22±2.9616.22_± 2.96 ✓ ✓ 80.38±0.3280.38_± 0.32 45.35±1.3445.35_± 1.34 31.41±0.8731.41_± 0.87 44.74±1.1844.74_± 1.18 38.29±1.1638.29_± 1.16 82.81±0.2182.81_± 0.21 80.74±0.9280.74_± 0.92 35.23±1.0535.23_± 1.05 25.23±1.3325.23_± 1.33 32.77±3.3732.77_± 3.37 26.01±1.4826.01_± 1.48 Oracle ✓ 80.42±0.5180.42_± 0.51 47.17±3.1647.17_± 3.16 33.23±2.1133.23_± 2.11 45.07±0.5045.07_± 0.50 41.87±0.8741.87_± 0.87 82.35±0.2782.35_± 0.27 81.04±1.3581.04_± 1.35 35.35±4.1835.35_± 4.18 22.57±5.0622.57_± 5.06 35.45±3.7535.45_± 3.75 23.70±4.8323.70_± 4.83 4.2 Subspace Residual Router VISTA routes each input to a small subset of previously learned adapters by comparing its representation against a set of task-specific subspaces. The router is training-free at inference time: it only uses second-order statistics accumulated from past tasks, and does not require learning an additional classifier over task identities. This design benefits from the strong representation capabilities of pretrained models, and is crucial in CE4L, where task boundaries can be subtle and view shifts can dominate appearance cues. In mixed ego-exo streams, viewpoint changes can shift both feature centers and dominant variation directions; mean or prototype routers mainly capture the former and can become biased toward view-specific shortcuts. VISTA therefore normalizes task-conditioned variance through whitening and fits residual subspaces to capture task-specific variation directions, making routing better aligned with the multi-view difficulty exposed by CE4L. Router representation. Given a feature sequence x∈ℝL×Cx ^L× C produced by fθf_θ, we first extract a compact routing vector r(x)∈ℝdr(x) ^d. We split the temporal axis into M contiguous chunks and apply mean pooling within each chunk, then concatenate the pooled vectors to reduce the dimension: r(x) r(x) =[1|I1|∑i∈I1xi‖⋯‖1|IM|∑i∈IMxi]∈ℝd, = [ 1|I_1| _i∈ I_1x_i\; \|·s \| 1|I_M| _i∈ I_Mx_i ] ^d, (3) d d =MC, =MC, where Imm=1M\I_m\_m=1^M is a partition of 1,…,L\1,…,L\ and ‖⋯‖\|·s\| denotes concatenation. Choice of M depends on the input video length to balance the trade-off between information and efficiency, and such temporal pooling process provides a stable and low-variance routing signal. Whitened subspace construction. At the end of learning task t, we collect routing vectors rnn=1Nt\r_n\_n=1^N_t from the training data of task t and estimate their mean and diagonal variance: μt=1Nt∑n=1Ntrn,vart=1Nt∑n=1Ntrn⊙rn−μt⊙μt, _t= 1N_t _n=1^N_tr_n,\,\,\,\,var_t= 1N_t _n=1^N_tr_n r_n- _t _t, (4) where ⊙ is the Hadamard product. We then define an element-wise whitening weight vector wt=vart+ϵw_t= var_t+ε, with a small ϵ>0ε>0 for numerical stability. Using wtw_t, we compute centered whitened residuals zn=(rn−μt)⊙wtz_n=(r_n- _t) w_t and their covariance matrix in the whitened subspace: Σt=1Nt−1∑n=1Ntznzn⊤. _t= 1N_t-1 _n=1^N_tz_nz_n . (5) Let Ut∈ℝd×qU_t ^d× q denote the top-q eigenvectors of Σt _t, namely, the principal directions in the subspace. In addition to these variation directions, we explicitly include another whitened mean direction to better capture the space geometry mt=μt⊙wt‖μt⊙wt‖2.m_t= _t w_t \| _t w_t \|_2. (6) Finally, we build an augmented orthonormal basis Bt∈ℝd×(q+1)B_t ^d×(q+1) by QR factorization over the columns of [mtUt][m_t\,U_t]. This augmentation couples first-order shift (mean) and second-order variation (PCA) into a single subspace, enabling BtB_t to serve as an effective task signature. Residual-based routing score. Given a query routing vector r, we compute its task-conditional whitened coordinates r~t=r⊙wt r_t=r w_t and evaluate how well r~t r_t is explained by the augmented basis BtB_t. The whitened subspace residual score is defined as the complement of the explained energy ratio: et(r)=1−‖Bt⊤r~t‖22‖r~t‖22+ϵ.e_t(r)=1- \|B_t r_t \|_2^2 \| r_t \|_2^2+ε. (7) This score is non-negative and decreases when the query lies close to the task subspace in the whitened metric. It is also scale-stable due to the normalization by ‖r~t‖22\| r_t\|_2^2. When an input provides multiple correlated feature sequences (e.g., a pair of clips in skill assessment), we compute residual scores separately and average them to obtain mixture weights, which reduces variance without requiring supervision. Let S be the set of tasks observed so far. We convert the residual scores into a posterior over tasks via softmax: pt(r)=exp(−γet(r))∑j∈exp(−γej(r)),p_t(r)= \! (-γ\,e_t(r) ) _j \! (-γ\,e_j(r) ), (8) where γ>0γ>0 controls the sharpness of routing. We then select the top-K tasks with largest pt(r)p_t(r) and normalize their probabilities to obtain mixture weights πℓ=1K\ _ \_ =1^K. These weights are used to mix task adapters as in the adapter mixture above, yielding a task-agnostic inference rule that interpolates between hard routing and soft sharing depending on the confidence implied by residual gaps and it also mitigates performance degradation caused by misrouting. Table 8: Generalization to future tasks during continual learning. We report the upper-triangle average performance A~ A under the Ego+Exo setting, averaging over both views when applicable. Higher is better except planning ED@8 (↓ ). Method Skill Seg. Assoc. Plan. Ant.-V Ant.-N Fine-tuning 47.6947.69 18.9518.95 30.0930.09 88.5588.55 22.4822.48 12.5012.50 ER 47.3747.37 18.1118.11 28.8228.82 87.6987.69 24.2924.29 14.9014.90 DER++ 47.5147.51 18.8418.84 30.7030.70 87.9387.93 26.10¯ 26.10 14.8514.85 LwF 46.4346.43 20.02¯ 20.02 31.09¯ 31.09 87.52¯ 87.52 24.9424.94 14.96¯ 14.96 S-Prompt+ 44.6144.61 19.9219.92 26.2126.21 87.9787.97 20.9720.97 12.4512.45 VISTA (ours) 48.04 24.53 31.80 87.18 26.89 15.04 Table 9: Generalization to excluded tasks. We evaluate tasks not used in the default CE4L Ego+Exo pipeline. E, Ego. X, Exo. Method Segment Association Planning Ego Exo E→ X→ Ego Exo Fine-tuning 46.6046.60 36.5836.58 6.006.00 14.0014.00 84.0084.00 82.9682.96 ER 42.5442.54 38.2838.28 6.006.00 17.00¯ 17.00 84.1184.11 79.0079.00 DER++ 47.92¯ 47.92 40.7040.70 10.0010.00 15.0015.00 84.5584.55 78.15¯ 78.15 LwF 41.2941.29 41.18¯ 41.18 8.008.00 13.0013.00 83.45 78.1078.10 S-Prompt+ 47.5447.54 38.1338.13 17.00 17.00¯ 17.00 84.0184.01 80.1480.14 VISTA (ours) 48.36 42.58 17.00 19.00 83.97¯ 83.97 77.14 Figure 3: Hyper-parameter sensitivity and routing strategy analysis. (a) Different temperature parameter γ for residual score computation, (b) top-K choice of adapter mixture and (c) subspace dimension q. (d) Routing strategy comparison between kkNN, cosine similarity, whitened cosine similarity and whitened subspace router. We report the final average top-5 recall (%,↑\%, ) of the continual anticipation trained on Ego+Exo data, evaluated on egocentric verb prediction (a-c), and task-router accuracy (%,↑)(\%, ) of VISTA (d). Figure 4: Whitened-subspace distances across tasks. (a-d) We visualize pairwise distances between task-specific whitened subspaces (spanned by BtB_t with q=32q=32) used by the router (smaller distance means more similar) across 4 different scenarios. 5 Experiments 5.1 Experimental Setup Dataset. CE4L benchmarks are constructed on EgoExoLearn (Huang et al., 2024) (120 hours of paired ego/exo videos). We instantiate each benchmark as a sequential task stream and consider three training regimes: Ego-only, Exo-only, and Ego+Exo co-training. Configurations are summarized in Sec. 3.1 and Table 1. We report both within-view performance and zero-shot cross-view transfer. Baseline methods. We implement a variety of representative CL methods including replay-based ER (Rolnick et al., 2019) and DER++ (Buzzega et al., 2020); regularization-based EWC (Kirkpatrick et al., 2017) and LwF (Li and Hoiem, 2017); and ensemble-based L2P (Wang et al., 2022d) and S-Prompt (Wang et al., 2022a; Wang et al., 2023b). Note that the replay-based methods ER and DER++ explicitly reuse old training samples, which are typically unavailable under restricted CL settings and are potentially unfair when compared to other replay-free methods.We adapt L2P and S-Prompt to CE4L by replacing prompts with adapters (Wang et al., 2023b) while preserving their original routing strategies (i.e., learnable task embeddings and kkNN routing). We denote these variants as L2P+ and S-Prompt+. We also report joint training (upper bound) and sequential fine-tuning (lower bound). In all tables, we use light shading to group method families for readability. Scenario-specific details are provided in Appendix B. Evaluation. After each task, we evaluate on all seen tasks and summarize by final average performance A¯T A_T and average forgetting F¯T F_T when applicable. We use test splits when available (otherwise the official validation split; Appendix B) and average over 3 seeds (± std). Best/second-best are highlighted per column, excluding joint training. 5.2 Experimental Results Continual skill assessment. Table 3 reports ranking accuracy and forgetting for RAAN and two reference-aware variants. Fine-tuning forgets substantially and falls short of joint training. Replay helps, with DER++ the strongest replay baseline, but still does not fully recover joint training. Regularization-based baselines are less effective: EWC largely tracks fine-tuning with high forgetting, while LwF is inconsistent across variants, sometimes suppressing forgetting but not translating into stronger final accuracy. In contrast, VISTA achieves joint-level accuracy on all variants while maintaining near-zero forgetting. This suggests that, for pairwise ranking, stabilizing updates via parameter penalties or distillation alone is insufficient. It benefits from parameter isolation with robust test-time task routing. Continual action segmentation. Table 3 reveals a sharp separation between two aspects: for within-view, most CL methods improve clearly over fine-tuning, showing that interference is a major factor for dense parsing; for cross-view, zero-shot transfer remains low even for joint training, pointing to viewpoint-conditioned mismatch beyond CL. Ego+Exo co-training improves the cross-view evaluation substantially, suggesting that multi-view evidence is the primary lever for transferable segmentation problem, while current CL methods mainly control within-view retention and ignore such generalization ability. Continual cross-view association. Table 5 shows that Ego+Exo co-training improves both retrieval directions remarkably, highlighting the important role of paired cross-view evidence for alignment. Under single-view CL, replay and regularization provide only modest, direction-dependent gains over fine-tuning. The any-time curves (Figs. 6–8) show that S-Prompt+ and VISTA can improve over time with near-zero or negative forgetting, while curves for most other methods remain relatively flat, consistent with generally small forgetting observed for such contrastive objective (Cha et al., 2021; Ni et al., 2023; Tan et al., 2025), suggesting that task-wise statistics stabilize cross-view alignment while leveraging the evolving contrastive backbone. However, the remaining gap to joint training highlights the necessity of multi-view evidence for robust alignment, motivating future CL methods to better exploit such information. Continual action anticipation & planning. Table 6 reports Top-5 recall for verb and noun prediction. In anticipation, ensemble-based baselines do not translate to consistent gains: L2P+ and S-Prompt+ are generally weaker than replay and VISTA, and their degradation is especially visible on noun prediction, suggesting that their routing signals do not reliably capture the task structure. Replay-based methods improve performance but only partially offsets the degradation, while regularization-based methods largely track fine-tuning. VISTA is consistently competitive across training regimes and label heads, improving both within-view and cross-view accuracy, which suggests that task-conditioned adapters can preserve complementary temporal cues across tasks without an explicit learned task classifier. For planning (Table 5), regularization-based and replay-based baselines become competitive, and the gap to joint training is smaller. This suggests that, in our setup, planning is comparatively less dominated by catastrophic forgetting and more limited by the base representation of long-horizon procedural structure. VISTA remains consistently strong across views and training regimes, matching or improving upon other CL baselines while avoiding large replay buffers. Generalization beyond seen tasks. The default CE4L protocol evaluates the union of seen tasks, while table 9 reports performance on future tasks before they are introduced into the continual stream, using the upper-triangle average of the performance matrix. Table 9 evaluates EgoExoLearn tasks excluded from the default CE4L stream. Across both settings, VISTA achieves the strongest overall performance in most cases, consistent with its robust subspace-based routing under inputs outside the seen-task evaluation set. We report complementary metrics, validation results, additional forgetting analyses, backbone variants, and supplementary generality studies in Appendix B–D. Ablation and sensitivity study. Table 7 isolates the roles of task adapters and routing. Removing both components reduces VISTA to sequential fine-tuning and yields the weakest performance across CE4L. Adding task adapters with a random router often degrades performance, indicating that naive mixture selection can induce systematic mis-routing and reintroduce interference despite parameter isolation. In contrast, the whitened-subspace router combined with task adapters consistently recovers strong performance across all scenarios. An oracle router (100% routing accuracy) provides an upper bound and is close to our router on most metrics. Overall, these results indicate that CE4L requires not only storing task-specific adaptation, but also performing accurate, view-robust task inference at test time. Fig. 3 further examines hyper-parameter sensitivity and router design choices. Performance is stable around the default configuration used in all main experiments, with a clear optimum at intermediate routing temperature and a moderate top-K mixture size, indicating that VISTA benefits from confident (but not overly peaky) routing, and that mixing too many adapters can dilute task-specific updates. Increasing the task subspace dimension improves performance up to a moderate rank and then saturates, suggesting that a compact whitened subspace captures most task-discriminative variation needed for routing. Fig. 3(d) compares routing strategies and shows that whitening is consistently beneficial, while the whitened-subspace residual router achieves the highest router accuracy across benchmarks, validating the second-order task statistics over instance-level nearest neighbors for robust task inference under viewpoint shift. Figure 5: Task-wise subspace residual score. (a-d) We calculate the residual score of each test set feature in different task’s subspace of continual skill assessment benchmark (RAAN). Detailed analysis. We analyze the router behavior from two complementary angles: task separability in the saved subspace geometry and empirical routing accuracy. Fig. 5 visualizes the residual scores for skill assessment and shows that test samples attain systematically lower residuals under their ground-truth task subspace than under other tasks, explaining the near-perfect routing observed in this scenario. To quantify task-level geometry, we compute distances between tasks using a principal-angle discrepancy between their augmented bases (see Appendix C for details). Fig. 4 shows that separability varies substantially across benchmarks: skill tasks form well-separated subspaces, while segmentation and association exhibit much smaller inter-task distances, indicating a more entangled task geometry. Anticipation further reveals a clustered structure where tasks in either kitchen or lab scenarios are mutually similar. These geometric patterns align with routing difficulty in Fig. 3(d), but do not fully determine it. In the harder cases where task subspaces are closer, whitening and residual-energy scoring still recover informative task signals, improving router accuracy over cosine similarity and k-N baselines. This suggests that routing benefits from task signatures that capture both mean shift and variability (first and second order), and that routing quality depends on both task separability and the metric used to compare. Appendix Tables 25–26 further compare GMM/Mahalanobis routers and adapter mixture choices. Finally, Table 10 and Appendix Tables 28–30 compare and summarize methods’ efficiency. Replay methods incur substantial memory storage due to buffering raw samples and additional targets for DER++. Regularization methods avoid large buffers but introduce extra state and computation for Fisher estimation or teacher snapshots. In contrast, VISTA adds only lightweight adapters and compact per-task statistics, keeping storage overhead negligible and per-batch compute close to fine-tuning. This makes VISTA a practical baseline for long streams and large backbones where replay buffers may be impractical or where resources are limited. Table 10: Evaluation of efficiency on continual cross-view association benchmark. We report the number of trainable and total parameters in millions, storage cost in million floats, and average computation time cost in millisecond per training batch. Method Trainable (M) Total (M) Storage (M) Time (ms) Fine-tuning 177.66 177.66 0.00 1023.5 ER 177.66 177.66 1696.67 947.3 DER++ 177.66 177.66 1698.22 1042.6 EWC 177.66 177.66 355.32 1516.8 LwF 177.66 177.66 177.66 1047.3 L2P+ 177.83 177.83 0.00 1425.4 S-Prompt+ 177.83 177.83 0.00 730.4 VISTA 177.83 177.83 0.01 793.2 Limitations and Scope. CE4L is instantiated on one ego-exo corpus and assumes tasks with fixed boundaries, focusing on embodied perception and semantic/procedural cross-view understanding rather than full 3D spatial reasoning or end-to-end decision making. VISTA stores one adapter and compact routing statistics per task, so long streams may benefit from adapter merging, compression, or hierarchical routing despite the small per-task overhead in our setting. 6 Conclusion We introduced CE4L, a benchmark suite for continual ego, exo, and ego-exo learning that evaluates continual learners under heterogeneous video objectives while explicitly separating within-view retention from cross-view generalization and multi-view co-training. Our empirical study shows that the effectiveness of classic continual paradigms varies substantially across problems, and that cross-view evaluation exposes viewpoint-conditioned mismatch that can persist even when within-view forgetting is reduced. To provide a strong and practical baseline in this regime, we proposed VISTA, a parameter-efficient continual learner with training-free adapter routing, which consistently improves performance and efficiency over baselines across CE4L. As a general CL protocol, CE4L can be instantiated on future ego-exo corpora with broader task coverage, additional viewpoints and sensors, and new supervision types, enabling CL research to scale with the rapidly evolving data landscape of embodied vision. Acknowledgment. This work is supported by the NSFC Projects (Nos. 62406160, 62595773, 32530042), Beijing Natural Science Foundation (No. L247011), Beijing Major Science and Technology Project (No. Z251100008425003), and the STI2030-Major Projects (No. 2022ZD0204900). Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References Bansal et al. (2022) S. Bansal, C. Arora, and C. Jawahar My view is the best view: procedure learning from egocentric videos. In ECCV, p. 657–675. Cited by: §A.1. Bertasius et al. (2021) G. Bertasius, H. Wang, and L. Torresani Is space-time attention all you need for video understanding?. In ICML, M. Meila and T. Zhang (Eds.), Vol. 139, p. 813–824. Cited by: §B.3, §3.4. Black et al. (2026) K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky π0 _0: A vision-language-action flow model for general robot control. External Links: 2410.24164, Link Cited by: §1. Buzzega et al. (2020) P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara Dark experience for general continual learning: a strong, simple baseline. NeurIPS 33, p. 15920–15930. Cited by: Appendix B, §5.1. Carreira and Zisserman (2017) J. Carreira and A. Zisserman Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, p. 6299–6308. Cited by: §3.2, §3.3. Cha et al. (2021) H. Cha, J. Lee, and J. Shin Co2l: contrastive continual learning. In ICCV, p. 9516–9525. Cited by: §5.2. Cheang et al. (2024) C. Cheang, G. Chen, Y. Jing, T. Kong, H. Li, Y. Li, Y. Liu, H. Wu, J. Xu, Y. Yang, et al. Gr-2: a generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158. Cited by: §1. Chen et al. (2022) G. Chen, S. Xing, Z. Chen, Y. Wang, K. Li, Y. Li, Y. Liu, J. Wang, Y. Zheng, B. Huang, et al. Internvideo-ego4d: a pack of champion solutions to ego4d challenges. arXiv preprint arXiv:2211.09529. Cited by: §A.1, §1, §2. Chen et al. (2019) M. Chen, Z. Kira, G. AlRegib, J. Yoo, R. Chen, and J. Zheng Temporal attentive alignment for large-scale video domain adaptation. In ICCV, p. 6321–6330. Cited by: §B.4, §3.5. Chen et al. (2020) M. Chen, B. Li, Y. Bao, G. AlRegib, and Z. Kira Action segmentation with joint self-supervised temporal domain adaptation. In CVPR, p. 9454–9463. Cited by: §B.2. Cheng et al. (2024) H. Cheng, S. Yang, C. Wang, J. T. Zhou, A. C. Kot, and B. Wen Stsp: spatial-temporal subspace projection for video class-incremental learning. In ECCV, p. 374–391. Cited by: §A.2. Damen et al. (2018) D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: the epic-kitchens dataset. In ECCV, p. 720–736. Cited by: §A.1. Damen et al. (2022) D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Rescaling egocentric vision: collection, pipeline and challenges for epic-kitchens-100. IJCV 130 (1), p. 33–55. Cited by: §A.1, §B.4, §2, §3.5. De Lange et al. (2021) M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars A continual learning survey: defying forgetting in classification tasks. TPAMI 44 (7), p. 3366–3385. Cited by: §A.2. Doughty et al. (2019) H. Doughty, W. Mayol-Cuevas, and D. Damen The pros and cons: rank-aware temporal attention for skill determination in long videos. In CVPR, p. 7862–7871. Cited by: §B.1, §3.2. Farha and Gall (2019) Y. A. Farha and J. Gall Ms-tcn: multi-stage temporal convolutional network for action segmentation. In CVPR, p. 3575–3584. Cited by: §B.2, §3.3. Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: around the world in 3,000 hours of egocentric video. In CVPR, p. 18995–19012. Cited by: §B.3, §B.4, §3.5. Grauman et al. (2024) K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. Ego-exo4d: understanding skilled human activity from first-and third-person perspectives. In CVPR, p. 19383–19400. Cited by: §A.1, §1, §2. Grover et al. (2023) S. Grover, V. Vineet, and Y. Rawat Revealing the unseen: benchmarking video action recognition under occlusion. Advances in Neural Information Processing Systems 36, p. 65642–65664. Cited by: §A.1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §A.3. Huang et al. (2025) S. Huang, J. Wu, X. Wei, Y. Cai, D. Jiang, and Y. Wang Sound bridge: associating egocentric and exocentric videos via audio cues. In CVPR, p. 28942–28951. Cited by: §A.1. Huang et al. (2024) Y. Huang, G. Chen, J. Xu, M. Zhang, L. Yang, B. Pei, H. Zhang, L. Dong, Y. Wang, L. Wang, et al. Egoexolearn: a dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In CVPR, p. 22072–22086. Cited by: §A.1, §B.1, §B.2, §B.3, §B.4, §1, §2, §3.1, §3.2, §3.4, §3.5, §5.1. Intelligence et al. (2025) P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky π0.5 _0.5: A vision-language-action model with open-world generalization. External Links: 2504.16054, Link Cited by: §1. Kang et al. (2025) Z. Kang, L. Wang, X. Zhang, and K. Alahari Advancing prompt-based methods for replay-independent general continual learning. In ICLR, Cited by: Appendix D. Kim et al. (2022) G. Kim, C. Xiao, T. Konishi, Z. Ke, and B. Liu A theoretical study on solving continual learning. NeurIPS 35, p. 5065–5079. Cited by: §A.3. Kim et al. (2024) M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §1. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS. Cited by: Appendix B, §5.1. Kwon et al. (2021) T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys H2o: two hands manipulating objects for first person interaction recognition. In ICCV, p. 10138–10148. Cited by: §A.1. Lester et al. (2021) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. Cited by: §A.3. Li et al. (2025) J. Li, S. Wang, B. Qian, Y. He, X. Wei, Q. Wang, and Y. Gong Dynamic integration of task-specific adapters for class incremental learning. In CVPR, p. 30545–30555. Cited by: §4.1. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. Cited by: §A.3. Li et al. (2024) Y. Li, W. Huang, A. Wang, L. Zeng, J. Meng, and W. Zheng Egoexo-fitness: towards egocentric and exocentric full-body action understanding. In ECCV, p. 363–382. Cited by: §A.1. Li and Hoiem (2017) Z. Li and D. Hoiem Learning without forgetting. TPAMI. Cited by: Appendix B, §5.1. Lin et al. (2022) K. Q. Lin, J. Wang, M. Soldan, M. Wray, R. Yan, E. Z. Xu, D. Gao, R. Tu, W. Zhao, W. Kong, et al. Egocentric video-language pretraining. NeurIPS 35, p. 7575–7586. Cited by: §A.1, §B.3, §3.4, §3.4. Liu et al. (2023) B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, p. 44776–44791. Cited by: Appendix D. Liu et al. (2022) Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi Hoi4d: a 4d egocentric dataset for category-level human-object interaction. In CVPR, p. 21013–21022. Cited by: §A.1, §2. Liu et al. (2025) Z. Liu, G. Fu, C. Du, W. S. Lee, and M. Lin Continual reinforcement learning by planning with online world models. arXiv preprint arXiv:2507.09177. Cited by: §1. Ma et al. (2022) Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang Vip: towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030. Cited by: §A.1. Maharana et al. (2026) S. K. Maharana, S. S. Kushwaha, B. Zhang, A. Rodriguez, S. Wei, Y. Tian, and Y. Guo AVROBUSTBENCHAVROBUSTBENCH: Benchmarking the robustness of audio-visual recognition models at test-time. Advances in Neural Information Processing Systems 38. Cited by: §A.1. Mangalam et al. (2023) K. Mangalam, R. Akshulakov, and J. Malik Egoschema: a diagnostic benchmark for very long-form video language understanding. NeurIPS 36, p. 46212–46244. Cited by: §A.1. Maraghi and Faez (2022) V. O. Maraghi and K. Faez Class-incremental learning on video-based action recognition by distillation of various knowledge. Computational Intelligence and Neuroscience 2022 (1), p. 4879942. Cited by: §1. McDonnell et al. (2024) M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. van den Hengel Ranpac: random projections and pre-trained models for continual learning. NeurIPS 36. Cited by: §A.3, §2. Moon et al. (2023) J. Moon, K. Park, J. U. Kim, and G. Park Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. In ICCV, p. 11731–11741. Cited by: Appendix D. Nair et al. (2022) S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta R3m: a universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601. Cited by: §A.1. Ni et al. (2023) Z. Ni, L. Wei, S. Tang, Y. Zhuang, and Q. Tian Continual vision-language representation learning with off-diagonal information. In ICML, p. 26129–26149. Cited by: §5.2. Parisi et al. (2019) G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter Continual lifelong learning with neural networks: a review. Neural Networks 113, p. 54–71. Cited by: §A.3. Park et al. (2021) J. Park, M. Kang, and B. Han Class-incremental learning for action recognition in videos. In ICCV, p. 13698–13707. Cited by: §A.2, §1, §2. Pei et al. (2023) Y. Pei, Z. Qing, S. Zhang, X. Wang, Y. Zhang, D. Zhao, and X. Qian Space-time prompting for video class-incremental learning. In ICCV, p. 11932–11942. Cited by: §A.2, §2. Pian et al. (2023) W. Pian, S. Mo, Y. Guo, and Y. Tian Audio-visual class-incremental learning. In ICCV, p. 7799–7811. Cited by: §A.2, §1. Pirsiavash and Ramanan (2012) H. Pirsiavash and D. Ramanan Detecting activities of daily living in first-person camera views. In CVPR, p. 2847–2854. Cited by: §A.1. Plizzari et al. (2024) C. Plizzari, G. Goletto, A. Furnari, S. Bansal, F. Ragusa, G. M. Farinella, D. Damen, and T. Tommasi An outlook into the future of egocentric vision. IJCV 132 (11), p. 4880–4936. Cited by: §A.1, §1. Pramanick et al. (2023) S. Pramanick, Y. Song, S. Nag, K. Q. Lin, H. Shah, M. Z. Shou, R. Chellappa, and P. Zhang Egovlpv2: egocentric video-language pre-training with fusion in the backbone. In ICCV, p. 5285–5297. Cited by: §A.1, §B.5, §1, §2, §3.4. Punamiya et al. (2025) R. Punamiya, D. Patel, P. Aphiwetsa, P. Kuppili, L. Y. Zhu, S. Kareer, J. Hoffman, and D. Xu Egobridge: domain adaptation for generalizable imitation from egocentric human data. In Human to Robot: Workshop on Sensorizing, Modeling, and Learning from Humans, Cited by: §A.1, §1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, p. 8748–8763. Cited by: §B.3, §B.4, §3.4, §3.5. Ragusa et al. (2021) F. Ragusa, A. Furnari, S. Livatino, and G. M. Farinella The meccano dataset: understanding human-object interactions from egocentric videos in an industrial-like domain. In WACV, p. 1569–1578. Cited by: §A.1. Rebuffi et al. (2017) S. Rebuffi, H. Bilen, and A. Vedaldi Learning multiple visual domains with residual adapters. NeurIPS 30. Cited by: §A.3, §4.1. Rolnick et al. (2019) D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne Experience replay for continual learning. NeurIPS 32. Cited by: §A.3, Appendix B, §5.1. Römer et al. (2026) R. Römer, Y. Zhang, and A. P. Schoellig CLARE: continual learning for vision-language-action models via autonomous adapter routing and expansion. arXiv preprint arXiv:2601.09512. Cited by: Table 32, Table 32, Appendix D. Schiappa et al. (2023) M. C. Schiappa, N. Biyani, P. Kamtam, S. Vyas, H. Palangi, V. Vineet, and Y. S. Rawat A large-scale robustness analysis of video action recognition models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 14698–14708. Cited by: §A.1. Sener et al. (2022) F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao Assembly101: a large-scale multi-view video dataset for understanding procedural activities. In CVPR, p. 21096–21106. Cited by: §A.1. Sigurdsson et al. (2018) G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari Actor and observer: joint modeling of first and third-person videos. In CVPR, p. 7396–7404. Cited by: §A.1. Sun et al. (2026) G. Sun, H. Yan, L. Wang, Z. Kang, S. Cui, H. Su, J. Zhu, and Y. Zhong MePo: meta post-refinement for rehearsal-free general continual learning. arXiv preprint arXiv:2602.07940. Cited by: Appendix D. Sung et al. (2018) F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales Learning to compare: relation network for few-shot learning. In CVPR, p. 1199–1208. Cited by: §B.1, §3.2. Tan et al. (2025) Y. Tan, X. Hu, H. Xue, C. De Melo, and F. D. Salim Bisecle: binding and separation in continual learning for video language understanding. arXiv preprint arXiv:2507.00469. Cited by: §A.2, §2, §5.2. Tang et al. (2024) T. Tang, S. Deldari, H. Xue, C. De Melo, and F. Salim Vilco-bench: video language continual learning benchmark. NeurIPS 37, p. 70213–70229. Cited by: §A.2, §2. Thoker et al. (2022) F. M. Thoker, H. Doughty, P. Bagad, and C. G. Snoek How severe is benchmark-sensitivity in video self-supervised learning?. In European Conference on Computer Vision, p. 632–652. Cited by: §A.1. Villa et al. (2023) A. Villa, J. L. Alcázar, M. Alfarra, K. Alhamoud, J. Hurtado, F. C. Heilbron, A. Soto, and B. Ghanem Pivot: prompting for video continual learning. In CVPR, p. 24214–24223. Cited by: §A.2, §2. Villa et al. (2022) A. Villa, K. Alhamoud, V. Escorcia, F. Caba, J. L. Alcázar, and B. Ghanem Vclimb: a novel video class incremental learning benchmark. In CVPR, p. 19035–19044. Cited by: §A.2, §1, §2. Vyas et al. (2020) S. Vyas, Y. S. Rawat, and M. Shah Multi-view action recognition using cross-view video prediction. In European Conference on Computer Vision, p. 427–444. Cited by: §A.1. Wang et al. (2025a) C. Wang, Y. Zhu, Y. Xu, J. Yang, Z. Yan, Y. Wang, Y. Wang, and L. Wang InternVideo-next: towards general video foundation models without video-text supervision. arXiv preprint arXiv:2512.01342. Cited by: §A.1. Wang et al. (2025b) H. Wang, H. Lu, L. Yao, and D. Gong Self-expansion of pre-trained models with mixture of adapters for continual learning. In CVPR, p. 10087–10098. Cited by: §4.1. Wang et al. (2023a) L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao Videomae v2: scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 14549–14560. Cited by: §B.5. Wang et al. (2023b) L. Wang, J. Xie, X. Zhang, M. Huang, H. Su, and J. Zhu Hierarchical decomposition of prompt-based continual learning: rethinking obscured sub-optimality. NeurIPS. Cited by: §A.3, §2, §5.1. Wang et al. (2024a) L. Wang, X. Zhang, H. Su, and J. Zhu A comprehensive survey of continual learning: theory, method and application. TPAMI. Cited by: §A.2, §A.3, §2. Wang et al. (2022a) Y. Wang, Z. Huang, and X. Hong S-prompts learning with pre-trained transformers: an occam’s razor for domain incremental learning. NeurIPS 35, p. 5682–5695. Cited by: §A.3, Appendix B, §2, §5.1. Wang et al. (2025c) Y. Wang, D. Zhou, and H. Ye Integrating task-specific and universal adapters for pre-trained model-based class-incremental learning. In ICCV, p. 806–816. Cited by: §4.1. Wang et al. (2024b) Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng, Z. Wang, Y. Shi, et al. Internvideo2: scaling foundation models for multimodal video understanding. In ECCV, p. 396–416. Cited by: §A.1. Wang et al. (2022b) Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, et al. Internvideo: general video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191. Cited by: §A.1. Wang et al. (2025d) Y. Wang, X. Li, Z. Yan, Y. He, J. Yu, X. Zeng, C. Wang, C. Ma, H. Huang, J. Gao, et al. Internvideo2. 5: empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386. Cited by: §A.1. Wang et al. (2022c) Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: complementary prompting for rehearsal-free continual learning. In ECCV, p. 631–648. Cited by: §A.3, §2. Wang et al. (2022d) Z. Wang, Z. Zhang, C. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister Learning to prompt for continual learning. In CVPR, p. 139–149. Cited by: §A.3, Appendix B, §2, §5.1. Xu et al. (2021) H. Xu, G. Ghosh, P. Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer Videoclip: contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084. Cited by: §3.4. Xu et al. (2023) L. Xu, Q. Wu, L. Pan, F. Meng, H. Li, C. He, H. Wang, S. Cheng, and Y. Dai Towards continual egocentric activity recognition: a multi-modal egocentric activity dataset for continual learning. TMM 26, p. 2430–2443. Cited by: §1. Xue and Grauman (2023) Z. S. Xue and K. Grauman Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment. NeurIPS 36, p. 53688–53710. Cited by: §A.1. Yan et al. (2025) H. Yan, G. Sun, Z. Kang, Y. Zhong, and L. Wang Domain generalizable continual learning. arXiv preprint arXiv:2510.16914. Cited by: §A.3. Yan et al. (2026) H. Yan, G. Sun, K. Zhou, Q. Li, L. Wang, and Y. Zhong FlyPrompt: brain-inspired random-expanded routing with temporal-ensemble experts for general continual learning. ICLR. Cited by: Appendix D. Yang et al. (2025) R. Yang, Q. Yu, Y. Wu, R. Yan, B. Li, A. Cheng, X. Zou, Y. Fang, X. Cheng, R. Qiu, et al. Egovla: learning vision-language-action models from egocentric human videos. arXiv preprint arXiv:2507.12440. Cited by: §A.1, §1. Yu et al. (2025) J. Yu, Z. Huang, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He MoE-adapters++: towards more efficient continual learning of vision-language models via dynamic mixture-of-experts adapters. TPAMI. Cited by: §4.1. Yu et al. (2024) J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He Boosting continual learning of vision-language models via mixture-of-experts adapters. In CVPR, p. 23219–23230. Cited by: §4.1. Zhang et al. (2022) Y. Zhang, H. Doughty, L. Shao, and C. G. Snoek Audio-adaptive activity recognition across video domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 13791–13800. Cited by: §A.1. Zhao et al. (2023) Y. Zhao, I. Misra, P. Krähenbühl, and R. Girdhar Learning video representations from large language models. In CVPR, p. 6586–6597. Cited by: §A.1, §1, §2. Zhou et al. (2024a) D. Zhou, H. Sun, J. Ning, H. Ye, and D. Zhan Continual learning with pre-trained models: a survey. In IJCAI, p. 8363–8371. Cited by: §A.3, §4.1. Zhou et al. (2024b) D. Zhou, Q. Wang, Z. Qi, H. Ye, D. Zhan, and Z. Liu Class-incremental learning: a survey. TPAMI. Cited by: §A.2, §A.3. Zhou et al. (2026) K. Zhou, R. Cai, L. Wang, H. P. H. Shum, and X. Liang A comprehensive survey of action quality assessment: method and benchmark. Pattern Recognition 179, p. 113933. External Links: Document Cited by: §A.1. Zhou et al. (2025a) K. Zhou, Z. Hao, L. Wang, and X. Liang Adaptive score alignment learning for continual perceptual quality assessment of 360-degree videos in virtual reality. IEEE Transactions on Visualization and Computer Graphics 31 (5), p. 2880–2890. Cited by: §A.1. Zhou et al. (2025b) K. Zhou, Q. Pan, X. Zhang, H. P. Shum, F. W. Li, X. Liang, and L. Wang Continual action quality assessment via adaptive manifold-aligned graph regularization. arXiv preprint arXiv:2510.06842. Cited by: §A.1. Zhou et al. (2024c) K. Zhou, L. Wang, X. Zhang, H. P. Shum, F. W. Li, J. Li, and X. Liang Magr: manifold-aligned graph regularization for continual action quality assessment. In European Conference on Computer Vision, p. 375–392. Cited by: §A.1. Zhuang et al. (2022) H. Zhuang, Z. Weng, H. Wei, R. Xie, K. Toh, and Z. Lin Acil: analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS 35, p. 11602–11614. Cited by: §A.3, §2. Appendix A Background and Related Work A.1 Egocentric Video Learning Egocentric vision research has transitioned from limited datasets (Pirsiavash and Ramanan, 2012; Bansal et al., 2022) to large-scale unscripted collections capturing complex daily activities (Damen et al., 2018; Damen et al., 2022; Liu et al., 2022). This domain recently integrated third-person perspectives to ground first-person experiences, a crucial development for understanding skilled interactions and procedural demonstrations (Kwon et al., 2021; Sigurdsson et al., 2018). Specialized benchmarks now target challenges such as asynchronous cross-view association (Sener et al., 2022; Xue and Grauman, 2023; Huang et al., 2024; Grauman et al., 2024; Huang et al., 2025), action quality assessment (Li et al., 2024; Ragusa et al., 2021; Zhou et al., 2024c; Zhou et al., 2025a; Zhou et al., 2025b; Zhou et al., 2026), and skill transfer to embodied agents via Vision-Language-Action models (Nair et al., 2022; Ma et al., 2022; Punamiya et al., 2025; Yang et al., 2025). Concurrently, representation learning has advanced toward unified backbones that directly fuse multiple modalities for robust zero-shot transfer (Lin et al., 2022; Pramanick et al., 2023; Zhao et al., 2023; Chen et al., 2022; Wang et al., 2022b; Wang et al., 2024b; Wang et al., 2025d; Wang et al., 2025a). Related studies also examine static robustness and generalization under occlusions, corruptions, benchmark sensitivity, cross-view prediction, and audio-aided domain shifts (Grover et al., 2023; Schiappa et al., 2023; Thoker et al., 2022; Vyas et al., 2020; Zhang et al., 2022; Maharana et al., 2026). Despite these strides toward lifelong perception, current methods rely heavily on static offline training protocols (Plizzari et al., 2024; Mangalam et al., 2023). These standard approaches overlook the inherently non-stationary nature of visual streams where tasks and viewpoints evolve dynamically, a limitation this work directly addresses. A.2 Video Continual Learning Video CL imposes unique demands due to the high-dimensional temporal dynamics absent in static image benchmarks (De Lange et al., 2021; Wang et al., 2024a; Zhou et al., 2024b). Early approaches largely extended image-based paradigms, including regularization and replay, by incorporating temporal constraints to mitigate catastrophic forgetting: Park et al. (2021) proposed time-channel importance maps to guide knowledge distillation, while vCLIMB (Villa et al., 2022) introduces temporal consistency regularization to optimize the selection of frame-level exemplars. Recently, the field has shifted towards parameter-efficient adaptation of large-scale pre-trained models (e.g., CLIP) to avoid the computational burden of full fine-tuning. Methods such as Space-time Prompting (Pei et al., 2023) and PIVOT (Villa et al., 2023) employ learnable prompts to capture task-specific temporal semantics while freezing the backbone, thereby enhancing transferability with minimal storage costs. Alternative strategies like STSP (Cheng et al., 2024) explore geometric constraints, utilizing orthogonal subspace projections to separate class representations without relying on privacy-invasive exemplars. Furthermore, the scope of video CL is expanding beyond simple action classification: recent benchmarks like ViLCo (Tang et al., 2024) and methods like Bisecle (Tan et al., 2025) investigate continuous adaptation in complex video-language understanding tasks or incorporate audio streams (Pian et al., 2023). However, these works predominantly focus on single-view streams or isolated recognition tasks, neglecting the complex coupled shifts in viewpoint and long-horizon procedural dependencies that our CE4L seeks to bridge. A.3 Pretraining-based Continual Learning Traditional CL studies are mostly conducted by training artificial neural networks from scratch over non-stationary data streams (Parisi et al., 2019; Wang et al., 2024a; Zhou et al., 2024b; Rolnick et al., 2019). With the rise of pretrained models (PTMs), the research paradigm of CL has shifted from unlocking the potential of small models to leveraging the powerful representational capabilities and generalization abilities of PTMs (Zhou et al., 2024a; Wang et al., 2022d). Theoretical studies of canonical CL have proved that it is possible to decompose the task-free CL problem into two orthogonal subproblems: task identity inference and within-task prediction (Kim et al., 2022). Yet the use of PTMs has changed the relation between these two subproblems, given rich and general prior information from pretraining data (Wang et al., 2023b). Recent works focus on adapting the frozen PTM backbone via parameter-efficient tuning (PET) techniques, in which lightweight trainable modules are inserted into the backbone’s workflow (Wang et al., 2022d; Lester et al., 2021; Li and Liang, 2021; Rebuffi et al., 2017; Hu et al., 2021; Yan et al., 2025). Such PET components can be either task-shared (Wang et al., 2022c) or task-specific (Wang et al., 2022a; Wang et al., 2023b), and the latter often requires a routing strategy developed along the CL process. It is worth noting that a learnable router itself can suffer from catastrophic forgetting, and therefore a variety of work has proposed to address this issue using analytic or training-free algorithms (McDonnell et al., 2024; Zhuang et al., 2022). Following this direction, VISTA further investigates the efficacy of a non-parametric router for video data by describing the geometry of task-wise subspaces. Appendix B Additional Benchmark Details and Results Figure 6: CE4L performance curve (Ego + Exo). (a-e) We visualize pairwise the performance curve throughout the complete CL progress across 5 different scenarios. Trained on egocentric + exocentric data, and evaluated on egocentric test data, except continual skill assessment using RAAN and only egocentric data for training. Figure 7: CE4L performance curve (Ego). (a-e) We visualize pairwise the performance curve throughout the complete CL progress across 5 different scenarios. Trained on egocentric data, and evaluated on egocentric test data, except skill assessment using relation network (RN) and both egocentric training data and exocentric reference data. Figure 8: CE4L performance curve (Exo). (a-e) We visualize pairwise the performance curve throughout the complete CL progress across 5 different scenarios. Trained on exocentric data, and evaluated on egocentric test data, except continual skill assessment: triplet loss (TL) and both egocentric training data and exocentric reference data. Continual evaluation metrics. We evaluate continual learners by recording a performance matrix A∈ℝT×TA ^T× T, where At,jA_t,j denotes the performance on task j after learning up to task t (1≤j≤t≤T1≤ j≤ t≤ T). The final average performance is defined as A¯T=1T∑j=1TAT,j. A_T= 1T _j=1^TA_T,j. (9) The average forgetting summarizes how much performance on past tasks drops from its best historical value to the final value: F¯T=1T−1∑j=1T−1(maxt∈j,…,TAt,j−AT,j). F_T= 1T-1 _j=1^T-1 ( _t∈\j,…,T\A_t,j-A_T,j ). (10) We use the benchmark-specific metric At,jA_t,j (e.g., ranking accuracy, frame-wise accuracy, Top-1 association accuracy, Top-5 recall, and ED@8) and report A¯T A_T and F¯T F_T when forgetting is meaningful for the task. For action planning where ED@8 is lower-is-better, we use the monotone transformation 1−ED@81-ED@8 so that larger values always indicate better performance. We also report forward transfer as FWT¯=1T−1∑t=2T(At−1,t−A0,t), FWT= 1T-1 _t=2^T(A_t-1,t-A_0,t), (11) where A0,tA_0,t denotes the pretrained model performance on task t before continual training. Table 11: Detailed CE4L benchmark statistics. CE4L is built on EgoExoLearn, which contains 8 high-level procedural tasks and 745 videos. Split format is train / val / test unless otherwise noted. Skill assessment uses train / val pairs only; association evaluation uses 800 validation and 2,000 test direction-balanced multiple-choice questions. All videos are standardized to 25 fps and 224×224 resolution. Scenario Tasks Task Unit Ego Split Exo Split / Input Skill assessment 4 Action groups 13,722 / 10,518 pairs – / 10 segments per clip Action segmentation 4 Procedural tasks 173 / 28 / 55 videos 176 / 21 / 26 videos; full video, 5× downsampled Cross-view association 5 Procedural tasks 22.7k clips (314 videos) 5.4k clips (249 videos); 4 frames per clip Action anticipation 8 Procedural tasks 34.6k / 7.7k / 17.3k clips 6.1k / 2.1k / 4.7k clips; 2 s context, 5 frames at 5 fps Action planning 4 Procedural tasks 2,171 / 395 / 734 clips 1,847 / 205 / 270 clips; 2 s context, 5 frames at 5 fps Table 12: Forward transfer on CE4L. We report FWT¯ FWT under Ego+Exo training. Higher is better for all metrics. Method Skill Segmentation Association Planning Anticipation-V Anticipation-N Fine-tuning 3.70 19.89 9.85 0.09 2.83 1.57 ER 2.93 18.53 0.33 0.81 4.07 3.01 DER++ 3.10 20.04 2.80 1.44 3.28 2.36 LwF 2.64 20.55 9.88 0.99 2.61 2.32 S-Prompt+ 1.18 22.98 6.64 2.14 3.06 2.31 VISTA (ours) 3.11 27.68 11.66 1.57 3.61 2.48 We show the any-time average performance curve of all CL methods across all CE4L benchmarks in figures 6, 7 and 8. For the future-task evaluation used in Table 9, we compute A~=1T−1∑i=1T−1(1T−i∑j=i+1TAi,j) A= 1T-1 _i=1^T-1 ( 1T-i _j=i+1^TA_i,j ), the upper-triangle mean of the performance matrix before those tasks are learned. The excluded-task evaluation in Table 9 uses EgoExoLearn tasks not included in the default CE4L stream. In the following sections, we provide detailed descriptions of the specific training settings for each benchmark, the baseline frameworks, the adaptation of all CL algorithms, and additional results. Shared continual baseline settings. Unless otherwise noted, we use a single set of baseline hyper-parameters across all benchmarks. For ER (Rolnick et al., 2019) and DER++ (Buzzega et al., 2020), we set the buffer ratio to 0.2, meaning the replay memory capacity is 20% of the total number of training samples observed in the entire stream, and we set the replay batch ratio to 0.2, meaning 20% of each training mini-batch is replaced by replay samples while keeping the effective batch size unchanged. For DER++, we use a distillation weight of 0.5. For EWC (Kirkpatrick et al., 2017), we use a regularization weight of 10−210^-2, an online Fisher decay of 1.0, and estimate the diagonal Fisher from 50 mini-batches at the end of each task. For LwF (Li and Hoiem, 2017), we use an MSE distillation weight of 0.5. For L2P+ (Wang et al., 2022d), we use a top-2 adapter selection and an adapter bottleneck dimension of 64. For S-Prompt+ (Wang et al., 2022a), we use hard top-1 routing with 32 KMeans centroids per task. For VISTA, we use a top-2 adapter mixture, an adapter bottleneck dimension of 64, router pooling with a single temporal segment, a subspace rank of 32, and a routing temperature of 10.0. Benchmark-specific training hyper-parameters and method-specific targets are given below. B.1 Continual Skill Assessment Training setup. For the baseline architecture of continual skill assessment, we follow the framework of Huang et al. (2024), in which a margin ranking loss, a disparity loss and a rank-aware loss are utilized as the training objective for the ego-only baseline variant RAAN (Doughty et al., 2019); and for the ego+exo case given exocentric video reference, a triplet-loss based baseline variant TL is implemented by taking the reference video as anchor, and a relation-network baseline variant RN is introduced by respectively concatenating the two egocentric video features and exocentric feature (Sung et al., 2018). We train for 2000 epochs per task with Adam using a learning rate of 10−410^-4 and batch size 128. Baseline method details. Replay methods (ER and DER++). We follow the shared replay protocol above. In skill assessment, DER++ caches the pair of aggregated ranking scores (s1,s2)(s_1,s_2) produced at the time a sample is written to memory (averaged over RAAN branches and time steps), and uses an MSE distillation loss with weight 0.5 on replayed samples. Regularization methods (EWC and LwF). We follow the shared settings above. For EWC, we estimate a diagonal Fisher from 50 mini-batches at each task end and use a regularization weight of 10−210^-2 with online decay 1.0. For LwF, we distill toward a teacher snapshot via an MSE loss on ranking scores with weight 0.5. Ensemble-based baselines (L2P+ and S-Prompt+). L2P+ maintains a fixed pool of adapters and learns a key vector per adapter. For each input pair, it builds a query representation by temporally pooling the clip features into a single segment and selects the top-2 adapters by cosine match between the query and the keys. The selected adapters are applied and averaged, and an auxiliary similarity loss encourages stable key query matching. S-Prompt+ uses hard top-1 routing based on task-wise KMeans centroids: after each task, it clusters task features into 32 centers and assigns a test input to the single nearest task by the minimum average distance to that task’s centers, then applies only the selected task adapter. A key implementation difference is whether the shared predictor is frozen: for S-Prompt+ and VISTA, adapters are explicitly task-indexed and past adapters are not revisited once the stream moves on, so we freeze the shared learner (i.e., the prediction head) after the first task to keep earlier adapters calibrated; in contrast, L2P+ draws from a shared adapter pool whose members can continue to receive updates across tasks, so it does not require freezing the shared learner. Table 13: TAS results on the validation split (frame-wise accuracy). We report the final average frame-wise accuracy A¯T A_T (%,↑\%, ) for egocentric and exocentric evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 70.51±0.7370.51_± 0.73 21.40±0.9921.40_± 0.99 26.99±1.6726.99_± 1.67 40.06±1.3840.06_± 1.38 71.25±0.3571.25_± 0.35 40.60±1.1640.60_± 1.16 Fine-tuning 45.41±2.1545.41_± 2.15 8.21±3.158.21_± 3.15 22.85±1.5322.85_± 1.53 28.98±2.3128.98_± 2.31 46.14±2.1946.14_± 2.19 28.13±1.4128.13_± 1.41 ER 45.11±2.6245.11_± 2.62 10.94±5.7610.94_± 5.76 22.98¯±2.40 22.98_± 2.40 29.68±1.6829.68_± 1.68 44.23±5.0344.23_± 5.03 29.89±1.2929.89_± 1.29 DER++ 45.48±3.4345.48_± 3.43 7.46±2.937.46_± 2.93 18.65±5.5918.65_± 5.59 31.63¯±2.48 31.63_± 2.48 43.95±2.5543.95_± 2.55 30.59±3.2930.59_± 3.29 EWC 41.23±9.0941.23_± 9.09 6.86±0.316.86_± 0.31 21.31±5.8421.31_± 5.84 29.98±3.1229.98_± 3.12 45.88±0.8545.88_± 0.85 31.65±2.0431.65_± 2.04 LwF 46.04±3.5746.04_± 3.57 11.38¯±1.21 11.38_± 1.21 22.19±2.7622.19_± 2.76 31.13±0.9931.13_± 0.99 44.85±2.4944.85_± 2.49 32.76±2.2932.76_± 2.29 L2P+ 47.77¯±1.64 47.77_± 1.64 7.89±0.817.89_± 0.81 18.36±4.5218.36_± 4.52 29.24±1.6529.24_± 1.65 46.18¯±2.38 46.18_± 2.38 30.93±2.8330.93_± 2.83 S-Prompt+ 40.26±1.0640.26_± 1.06 6.52±0.756.52_± 0.75 15.42±2.4515.42_± 2.45 26.35±1.7426.35_± 1.74 42.62±1.5842.62_± 1.58 23.17±1.4023.17_± 1.40 VISTA (ours) 50.57±2.0850.57_± 2.08 12.80±3.1912.80_± 3.19 24.98±3.4624.98_± 3.46 33.73±0.8633.73_± 0.86 49.56±0.8149.56_± 0.81 32.13¯±2.66 32.13_± 2.66 Table 14: TAS forgetting on the test split (frame-wise accuracy). We report the final average forgetting F¯T F_T (%,↓\%, ) of frame-wise accuracy for both view evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint −−-- −−-- −−-- −−-- −−-- −−-- Fine-tuning 33.12±1.9433.12_± 1.94 12.98±9.5612.98_± 9.56 -0.55±1.19-0.55_± 1.19 14.21±2.2014.21_± 2.20 31.53±5.6931.53_± 5.69 15.61±3.0115.61_± 3.01 ER 34.12±4.5634.12_± 4.56 9.29±9.569.29_± 9.56 0.04±3.000.04_± 3.00 11.33±2.4711.33_± 2.47 33.94±6.5733.94_± 6.57 14.31±2.8414.31_± 2.84 DER++ 32.27±7.7732.27_± 7.77 8.88±2.628.88_± 2.62 5.08±5.185.08_± 5.18 12.01±3.4512.01_± 3.45 32.90±4.5532.90_± 4.55 14.39±5.7814.39_± 5.78 EWC 36.89±8.4836.89_± 8.48 11.45±4.4211.45_± 4.42 1.31±6.011.31_± 6.01 8.96±3.678.96_± 3.67 29.88±3.1629.88_± 3.16 13.74±2.1913.74_± 2.19 LwF 33.83±5.4433.83_± 5.44 8.53±0.758.53_± 0.75 1.46±4.321.46_± 4.32 12.04±2.6312.04_± 2.63 32.38±3.6332.38_± 3.63 13.77±2.7413.77_± 2.74 L2P+ 30.11±2.2030.11_± 2.20 13.41±2.2013.41_± 2.20 3.47±5.833.47_± 5.83 11.37±1.0411.37_± 1.04 29.49±2.4729.49_± 2.47 11.51±1.8811.51_± 1.88 S-Prompt+ 11.13¯±0.14 11.13_± 0.14 7.24¯±8.00 7.24_± 8.00 3.77±0.443.77_± 0.44 3.14±0.653.14_± 0.65 10.15¯±3.94 10.15_± 3.94 5.55¯±3.50 5.55_± 3.50 VISTA (ours) 3.56±0.223.56_± 0.22 6.42±4.666.42_± 4.66 −0.09¯±1.10 -0.09_± 1.10 3.70¯±0.40 3.70_± 0.40 2.81±0.742.81_± 0.74 1.68±0.621.68_± 0.62 Table 15: TAS results on the test split (edit score). We report the final average edit score A¯T A_T (↑ ) for egocentric and exocentric evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 46.34±0.5846.34_± 0.58 19.57±2.8319.57_± 2.83 33.93±3.4233.93_± 3.42 34.22±0.7234.22_± 0.72 43.41±0.4243.41_± 0.42 34.77±2.7134.77_± 2.71 Fine-tuning 42.04±0.8142.04_± 0.81 24.65±2.1424.65_± 2.14 27.68±4.5127.68_± 4.51 28.53±3.0428.53_± 3.04 40.38±1.5940.38_± 1.59 28.80±3.0828.80_± 3.08 ER 42.69±1.0042.69_± 1.00 22.95±2.1222.95_± 2.12 28.82±1.6428.82_± 1.64 32.95±3.3532.95_± 3.35 38.66±6.2938.66_± 6.29 30.88±3.7030.88_± 3.70 DER++ 42.51±2.2542.51_± 2.25 25.19±0.8625.19_± 0.86 27.79¯±2.77 27.79_± 2.77 35.29±2.3035.29_± 2.30 38.32±2.4338.32_± 2.43 29.02±1.1329.02_± 1.13 EWC 40.35±5.2140.35_± 5.21 25.07±1.1125.07_± 1.11 27.49±2.1827.49_± 2.18 33.44±0.5133.44_± 0.51 40.85¯±2.23 40.85_± 2.23 30.26±2.2430.26_± 2.24 LwF 43.10¯±2.11 43.10_± 2.11 26.77¯±0.50 26.77_± 0.50 26.31±1.4226.31_± 1.42 34.63±2.5734.63_± 2.57 41.19±1.1041.19_± 1.10 29.37±0.5029.37_± 0.50 L2P+ 42.85±0.6442.85_± 0.64 24.77±4.7724.77_± 4.77 11.03±0.6211.03_± 0.62 36.12±2.3236.12_± 2.32 39.59±4.6939.59_± 4.69 33.64±1.9133.64_± 1.91 S-Prompt+ 36.66±1.9736.66_± 1.97 15.46±6.8615.46_± 6.86 14.49±1.2614.49_± 1.26 36.76¯±1.43 36.76_± 1.43 31.79±1.6231.79_± 1.62 34.58¯±3.82 34.58_± 3.82 VISTA (ours) 44.88±0.9044.88_± 0.90 27.90±5.9127.90_± 5.91 27.74±3.4127.74_± 3.41 37.94±1.9337.94_± 1.93 39.83±6.2339.83_± 6.23 36.26±0.3736.26_± 0.37 Table 16: TAS results on the test split (segmental F1@AvgF1@Avg). We report the final average segmental F1@AvgF1@Avg A¯T A_T (↑ ) for egocentric and exocentric evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 41.60±0.1541.60_± 0.15 7.01±0.807.01_± 0.80 8.75±0.768.75_± 0.76 18.50±0.5618.50_± 0.56 38.91±0.8838.91_± 0.88 18.95±0.9218.95_± 0.92 Fine-tuning 30.63±1.0830.63_± 1.08 6.46±1.056.46_± 1.05 6.85±1.486.85_± 1.48 14.15±0.8614.15_± 0.86 27.19±1.5427.19_± 1.54 14.40±1.3714.40_± 1.37 ER 31.40±0.6631.40_± 0.66 5.35±0.545.35_± 0.54 7.67±0.917.67_± 0.91 14.76±1.2914.76_± 1.29 27.87±3.5027.87_± 3.50 13.79±1.0213.79_± 1.02 DER++ 32.02¯±0.64 32.02_± 0.64 6.44±0.746.44_± 0.74 7.09±1.007.09_± 1.00 15.25±1.5915.25_± 1.59 27.44±1.8027.44_± 1.80 14.65±1.5514.65_± 1.55 EWC 30.12±4.0130.12_± 4.01 5.75±1.395.75_± 1.39 7.19±1.147.19_± 1.14 15.49¯±1.11 15.49_± 1.11 28.49±1.9628.49_± 1.96 15.35¯±1.94 15.35_± 1.94 LwF 31.46±1.1031.46_± 1.10 7.43±0.317.43_± 0.31 7.42¯±0.57 7.42_± 0.57 15.38±1.0215.38_± 1.02 29.40±1.6429.40_± 1.64 13.74±1.8713.74_± 1.87 L2P+ 30.41±0.0930.41_± 0.09 7.07±1.567.07_± 1.56 4.45±0.094.45_± 0.09 13.19±1.0113.19_± 1.01 27.97±3.1627.97_± 3.16 14.21±1.3914.21_± 1.39 S-Prompt+ 22.49±0.6222.49_± 0.62 3.76±1.533.76_± 1.53 5.16±0.255.16_± 0.25 12.63±0.2212.63_± 0.22 19.17±1.0619.17_± 1.06 12.61±0.9412.61_± 0.94 VISTA (ours) 33.26±0.7833.26_± 0.78 7.23¯±1.27 7.23_± 1.27 6.19±0.476.19_± 0.47 15.68±0.5715.68_± 0.57 29.27¯±4.14 29.27_± 4.14 15.39±0.6815.39_± 0.68 Table 17: TAS results on the validation split (edit score). We report the final average edit score A¯T A_T (↑ ) for egocentric and exocentric evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 47.91±0.5247.91_± 0.52 22.44±1.9822.44_± 1.98 34.58±3.8234.58_± 3.82 37.32±0.6337.32_± 0.63 44.72±1.2744.72_± 1.27 37.64±1.8937.64_± 1.89 Fine-tuning 43.17±2.3143.17_± 2.31 25.41±2.5925.41_± 2.59 27.47±3.3527.47_± 3.35 31.11±2.4431.11_± 2.44 39.95±1.2839.95_± 1.28 29.56±1.1729.56_± 1.17 ER 44.74¯±1.80 44.74_± 1.80 24.15±2.0924.15_± 2.09 28.97±1.6928.97_± 1.69 35.42±2.8135.42_± 2.81 38.26±5.4738.26_± 5.47 30.12±2.2430.12_± 2.24 DER++ 43.72±1.5943.72_± 1.59 25.03±1.5225.03_± 1.52 26.31±1.8126.31_± 1.81 36.15¯±0.83 36.15_± 0.83 37.94±3.0037.94_± 3.00 30.04±2.8630.04_± 2.86 EWC 40.04±5.6840.04_± 5.68 24.75±1.2924.75_± 1.29 26.66±2.0426.66_± 2.04 32.69±0.2832.69_± 0.28 40.18±1.7440.18_± 1.74 34.25¯±1.33 34.25_± 1.33 LwF 44.62±0.8644.62_± 0.86 27.24¯±1.10 27.24_± 1.10 25.20±1.6625.20_± 1.66 35.66±3.5235.66_± 3.52 42.52±0.1842.52_± 0.18 32.70±1.6932.70_± 1.69 L2P+ 44.04±0.7844.04_± 0.78 26.49±2.4326.49_± 2.43 10.85±0.6910.85_± 0.69 31.81±0.9231.81_± 0.92 40.25±4.3040.25_± 4.30 33.73±2.4233.73_± 2.42 S-Prompt+ 36.66±0.3536.66_± 0.35 16.44±3.6616.44_± 3.66 14.38±1.4814.38_± 1.48 32.65±0.9632.65_± 0.96 31.35±1.7231.35_± 1.72 32.74±3.1032.74_± 3.10 VISTA (ours) 45.35±1.3845.35_± 1.38 28.87±3.5228.87_± 3.52 27.90¯±3.27 27.90_± 3.27 37.42±0.9937.42_± 0.99 42.51¯±6.58 42.51_± 6.58 37.11±1.4037.11_± 1.40 Table 18: TAS results on the validation split (segmental F1@AvgF1@Avg). We report the final average segmental F1@AvgF1@Avg A¯T A_T (↑ ) for egocentric and exocentric evaluation under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint 44.94±0.8544.94_± 0.85 8.61±0.378.61_± 0.37 8.56±0.528.56_± 0.52 21.07±0.3921.07_± 0.39 41.56±0.1441.56_± 0.14 22.34±0.9222.34_± 0.92 Fine-tuning 29.82±1.7129.82_± 1.71 5.64±0.885.64_± 0.88 5.94±0.885.94_± 0.88 16.46±2.4516.46_± 2.45 25.26±0.3225.26_± 0.32 15.62±1.9015.62_± 1.90 ER 32.05¯±1.44 32.05_± 1.44 5.37±0.695.37_± 0.69 6.42±0.846.42_± 0.84 16.14±0.8416.14_± 0.84 27.09±2.1527.09_± 2.15 14.55±0.6014.55_± 0.60 DER++ 31.86±0.7831.86_± 0.78 5.07±1.275.07_± 1.27 5.73±0.895.73_± 0.89 17.28¯±0.29 17.28_± 0.29 27.48±1.9727.48_± 1.97 15.30±1.4015.30_± 1.40 EWC 29.28±3.4029.28_± 3.40 4.56±0.224.56_± 0.22 6.41¯±1.03 6.41_± 1.03 15.62±1.4615.62_± 1.46 27.89±0.5727.89_± 0.57 16.55±1.6916.55_± 1.69 LwF 30.62±0.4230.62_± 0.42 6.78±1.096.78_± 1.09 6.07±0.126.07_± 0.12 15.85±0.8815.85_± 0.88 29.02¯±0.13 29.02_± 0.13 15.95±0.6815.95_± 0.68 L2P+ 30.86±1.2930.86_± 1.29 4.91±0.214.91_± 0.21 4.01±0.244.01_± 0.24 11.91±0.7911.91_± 0.79 27.29±2.0027.29_± 2.00 13.84±2.1913.84_± 2.19 S-Prompt+ 23.26±1.1823.26_± 1.18 3.57±1.123.57_± 1.12 4.59±0.234.59_± 0.23 10.98±1.0510.98_± 1.05 19.46±0.8519.46_± 0.85 12.06±0.5412.06_± 0.54 VISTA (ours) 34.89±1.4434.89_± 1.44 5.87¯±1.25 5.87_± 1.25 5.81±0.555.81_± 0.55 18.69±0.2018.69_± 0.20 30.72±3.6030.72_± 3.60 16.52¯±1.62 16.52_± 1.62 B.2 Continual Action Segmentation Training setup. For the baseline architecture of continual action segmentation, we follow the framework of Huang et al. (2024) and Chen et al. (2020), in which a per-stage training loss, consisting of a classification loss and a smoothing loss, is used as the objective for MS-TCN (Farha and Gall, 2019) model. For experimental efficiency, the frame sequence of both training and inference input video v is down-sampled by a factor of 5. We train for 150 epochs per task with Adam using a learning rate of 5×10−45× 10^-4 and batch size 1 video. Baseline method details. We follow the shared baseline settings above. A replay item stores the feature tensor, frame-wise labels, and validity mask; DER++ caches masked last-stage source logits and distills them on replayed samples. EWC and LwF are applied to the same masked prediction tensor. Adapters act on per-frame features before MS-TCN. And we set temporal pooling parameter M=2M=2 for VISTA because the input length of segmentation is longer. Additional results. We additionally report validation performance, test performance on complementary metrics, and test forgetting for continual temporal action segmentation in Tabs. 14–18. B.3 Continual Cross-View Association Training setup. For the baseline architecture of continual cross-view association benchmark of CE4L, we follow the framework of Huang et al. (2024), whose dual-encoder backbone consisting of a TimeSformer-B (Bertasius et al., 2021) video encoder and a CLIP text encoder (Radford et al., 2021) is initialized with EgoVLP (Lin et al., 2022) weights pretrained on Ego4D (Grauman et al., 2022) dataset. A contrastive loss between video embedding and annotation text embedding is used to fine-tune the backbone model during training, while upon inference, only the video encoder is used to extract features for query and candidate videos. The prediction is given by choosing one from 20 candidates, which has the highest cosine similarity with the query. We train for 5 epochs per task with AdamW using a learning rate of 10−510^-5 and batch size 32. Baseline method details. We follow the shared baseline settings above. For DER++, we cache per-sample embedding targets consisting of the video embedding and the text embedding, and distill them on replayed samples. LwF distills both embeddings toward the teacher snapshot with weight 0.5. Adapter-based routing methods act on the video embedding. And we set the temporal pooling hyperparameter M=2M=2 for VISTA because the input length of association videos is longer. Additional results. Additional validation results and test forgetting for continual association are reported in Tabs. 20 and 20. Table 19: Association results on the validation split. We report the final average Top-1 accuracy A¯T A_T (%,↑\%, ) for Ego→ and Exo→ retrieval under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Ego→ Exo→ Ego→ Exo→ Ego→ Exo→ Joint 35.17±0.7235.17_± 0.72 40.83±0.6240.83_± 0.62 39.00±1.0839.00_± 1.08 40.08±0.7240.08_± 0.72 43.08±1.2343.08_± 1.23 45.83±1.4545.83_± 1.45 Fine-tuning 25.25±1.7725.25_± 1.77 26.50±2.6526.50_± 2.65 33.25±1.8733.25_± 1.87 35.92±1.1635.92_± 1.16 36.25±0.7436.25_± 0.74 37.00±2.7637.00_± 2.76 ER 28.67±1.5628.67_± 1.56 30.75±0.4130.75_± 0.41 31.50±2.1331.50_± 2.13 35.83±0.4235.83_± 0.42 39.25¯±0.94 39.25_± 0.94 39.50±2.6739.50_± 2.67 DER++ 29.08¯±1.01 29.08_± 1.01 30.25±0.5430.25_± 0.54 31.58±2.8631.58_± 2.86 35.08±0.1235.08_± 0.12 38.17±0.6238.17_± 0.62 40.83±2.5740.83_± 2.57 EWC 27.17±2.5827.17_± 2.58 26.58±1.7126.58_± 1.71 29.25±2.5629.25_± 2.56 31.42±0.1231.42_± 0.12 36.75±1.8136.75_± 1.81 37.75±2.4137.75_± 2.41 LwF 25.67±2.6225.67_± 2.62 26.08±3.6026.08_± 3.60 33.75¯±1.06 33.75_± 1.06 37.08±1.5537.08_± 1.55 35.83±0.4735.83_± 0.47 36.67±2.7936.67_± 2.79 L2P+ 29.08¯±0.72 29.08_± 0.72 26.42±1.7826.42_± 1.78 32.75±1.4332.75_± 1.43 37.42±1.8537.42_± 1.85 36.92±2.4736.92_± 2.47 37.08±1.0137.08_± 1.01 S-Prompt+ 29.00±1.9529.00_± 1.95 32.33¯±0.77 32.33_± 0.77 31.08±2.0431.08_± 2.04 34.83±0.4734.83_± 0.47 35.58±0.5135.58_± 0.51 39.08±0.3139.08_± 0.31 VISTA (ours) 31.04±2.9531.04_± 2.95 32.70±0.2532.70_± 0.25 35.52±5.2435.52_± 5.24 37.33¯±4.71 37.33_± 4.71 40.70±5.2440.70_± 5.24 40.00¯±0.00 40.00_± 0.00 Table 20: Association forgetting on the test split. We report the final average forgetting F¯T F_T (%,↓\%, ) for Ego→ and Exo→ retrieval under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Ego→ Exo→ Ego→ Exo→ Ego→ Exo→ Joint −−-- −−-- −−-- −−-- −−-- −−-- Fine-tuning 3.09±2.463.09_± 2.46 8.25±3.598.25_± 3.59 6.54±1.536.54_± 1.53 2.50±3.022.50_± 3.02 2.27±1.962.27_± 1.96 2.69±1.552.69_± 1.55 ER 2.37±0.942.37_± 0.94 6.29±2.826.29_± 2.82 5.37±1.485.37_± 1.48 2.10±2.942.10_± 2.94 0.19±0.920.19_± 0.92 0.66±3.190.66_± 3.19 DER++ 2.98±0.802.98_± 0.80 6.96±1.786.96_± 1.78 6.04±1.286.04_± 1.28 3.04±2.523.04_± 2.52 1.23±2.811.23_± 2.81 2.85±1.632.85_± 1.63 EWC 3.32±1.583.32_± 1.58 9.92±1.849.92_± 1.84 8.07±3.058.07_± 3.05 4.40±3.374.40_± 3.37 1.94±2.151.94_± 2.15 1.49±1.991.49_± 1.99 LwF 5.63±5.185.63_± 5.18 8.32±4.328.32_± 4.32 6.44±2.256.44_± 2.25 1.79±3.771.79_± 3.77 2.50±2.342.50_± 2.34 3.00±1.263.00_± 1.26 L2P+ 2.68±0.652.68_± 0.65 8.58±1.618.58_± 1.61 7.16±2.177.16_± 2.17 5.80±2.445.80_± 2.44 4.42±1.034.42_± 1.03 2.96±1.922.96_± 1.92 S-Prompt+ −6.84¯±3.27 -6.84_± 3.27 -7.42±2.17-7.42_± 2.17 2.81¯±1.42 2.81_± 1.42 −3.75¯±1.43 -3.75_± 1.43 −6.54¯±4.73 -6.54_± 4.73 -3.70±3.68-3.70_± 3.68 VISTA (ours) -7.20±0.51-7.20_± 0.51 −6.39¯±2.78 -6.39_± 2.78 2.41±1.782.41_± 1.78 -4.05±0.82-4.05_± 0.82 -6.58±0.78-6.58_± 0.78 −3.40¯±0.70 -3.40_± 0.70 B.4 Continual Action Anticipation & Planning Training setup. For the baseline architecture of continual action anticipation and planning, we follow the framework of Damen et al. (2022); Grauman et al. (2022) and Huang et al. (2024), the model is structured as a trainable temporal relation module TA3N (Chen et al., 2019) over a pretrained CLIP encoder (Radford et al., 2021) which is frozen. For both anticipation and planning, the training objectives are cross-entropy based classification losses, but one is calculated over one action step and the other is evaluated over an action sequence. The evaluation metrics for anticipation and planning are Top-5 recall and edit distance at 8 steps (ED@8). We train for 40 epochs per task with SGD using a learning rate of 0.003. The batch size is 128 for anticipation and 16 for planning. Baseline method details. We follow the shared baseline settings above. DER++ caches the output logits as distillation targets and distills them on replayed samples. LwF distills output logits toward the teacher snapshot. Additional results. Table 21: Anticipation forgetting. We report the final average forgetting F¯T F_T (%,↓\%, ) of Top-5 recall for verb and noun prediction evaluated on egocentric and exocentric videos under each training regime. Method Train: Ego Train: Exo Train: Ego+Exo Ego-V Ego-N Exo-V Exo-N Ego-V Ego-N Exo-V Exo-N Ego-V Ego-N Exo-V Exo-N Joint −−-- −−-- −−-- −−-- −−-- −−-- −−-- −−-- −−-- −−-- −−-- −−-- Fine-tuning 2.82±0.802.82_± 0.80 10.41±0.2910.41_± 0.29 2.49±0.312.49_± 0.31 10.55±0.5510.55_± 0.55 5.79±0.765.79_± 0.76 8.36±0.288.36_± 0.28 4.49±1.554.49_± 1.55 8.41±0.748.41_± 0.74 5.76±0.505.76_± 0.50 8.61±0.268.61_± 0.26 4.62±1.354.62_± 1.35 8.47±0.518.47_± 0.51 ER 0.59±0.230.59_± 0.23 2.30±0.602.30_± 0.60 1.17±0.641.17_± 0.64 2.25±0.702.25_± 0.70 1.49±0.631.49_± 0.63 1.63±0.621.63_± 0.62 1.33±0.681.33_± 0.68 1.78±0.451.78_± 0.45 1.59±0.401.59_± 0.40 1.33±0.601.33_± 0.60 0.74±0.500.74_± 0.50 1.12±0.451.12_± 0.45 DER++ 0.19¯±0.35 0.19_± 0.35 1.43¯±0.55 1.43_± 0.55 0.46±0.480.46_± 0.48 1.57¯±0.76 1.57_± 0.76 0.88¯±0.29 0.88_± 0.29 1.04¯±0.45 1.04_± 0.45 0.98¯±0.36 0.98_± 0.36 1.29¯±0.57 1.29_± 0.57 0.79¯±0.40 0.79_± 0.40 0.53¯±0.45 0.53_± 0.45 0.76¯±0.38 0.76_± 0.38 0.62¯±0.52 0.62_± 0.52 EWC 3.02±1.063.02_± 1.06 10.34±0.5810.34_± 0.58 2.82±0.992.82_± 0.99 10.64±0.8310.64_± 0.83 5.67±0.945.67_± 0.94 8.47±0.328.47_± 0.32 4.82±1.314.82_± 1.31 8.43±0.638.43_± 0.63 5.64±0.545.64_± 0.54 7.85±0.297.85_± 0.29 3.88±1.653.88_± 1.65 7.79±0.247.79_± 0.24 LwF 1.41±0.891.41_± 0.89 7.63±0.237.63_± 0.23 2.36±0.542.36_± 0.54 7.83±0.457.83_± 0.45 3.61±0.283.61_± 0.28 7.08±0.867.08_± 0.86 3.91±0.613.91_± 0.61 7.04±0.907.04_± 0.90 3.37±0.733.37_± 0.73 5.68±0.915.68_± 0.91 3.60±0.983.60_± 0.98 5.76±0.805.76_± 0.80 L2P+ 1.99±1.031.99_± 1.03 3.30±2.143.30_± 2.14 2.99±0.712.99_± 0.71 3.73±1.933.73_± 1.93 2.01±0.902.01_± 0.90 3.04±0.923.04_± 0.92 2.22±0.652.22_± 0.65 3.61±1.053.61_± 1.05 2.10±0.852.10_± 0.85 4.44±1.004.44_± 1.00 2.83±1.252.83_± 1.25 4.87±1.054.87_± 1.05 S-Prompt+ 0.35±0.220.35_± 0.22 2.10±2.172.10_± 2.17 0.43¯±0.54 0.43_± 0.54 2.41±2.592.41_± 2.59 1.92±1.131.92_± 1.13 1.87±1.831.87_± 1.83 2.97±2.782.97_± 2.78 2.01±2.042.01_± 2.04 1.34±0.651.34_± 0.65 1.41±1.201.41_± 1.20 1.14±1.021.14_± 1.02 1.56±1.421.56_± 1.42 VISTA (ours) 0.07±0.050.07_± 0.05 0.76±0.530.76_± 0.53 0.37±0.220.37_± 0.22 0.80±0.590.80_± 0.59 0.44±0.170.44_± 0.17 0.53±0.450.53_± 0.45 0.85±0.960.85_± 0.96 0.62±0.510.62_± 0.51 0.33±0.200.33_± 0.20 0.46±0.540.46_± 0.54 1.24±0.931.24_± 0.93 0.49±0.570.49_± 0.57 Table 22: Planning forgetting. We report the final average forgetting F¯T F_T (%,↓\%, ) computed on 1−ED@81-ED@8. Method Train: Ego Train: Exo Train: Ego+Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Eval: Ego Eval: Exo Joint −−-- −−-- −−-- −−-- −−-- −−-- Fine-tuning 2.26±0.092.26_± 0.09 3.71±0.903.71_± 0.90 0.52±0.080.52_± 0.08 4.23±1.584.23_± 1.58 3.33±0.743.33_± 0.74 6.30±1.426.30_± 1.42 ER 2.63±0.892.63_± 0.89 2.58±0.462.58_± 0.46 1.30±0.751.30_± 0.75 3.40±0.953.40_± 0.95 3.60±0.693.60_± 0.69 2.78±0.532.78_± 0.53 DER++ 1.46±0.431.46_± 0.43 1.77±0.591.77_± 0.59 −0.04¯±0.69 -0.04_± 0.69 2.16±0.712.16_± 0.71 1.59±0.221.59_± 0.22 1.39±1.471.39_± 1.47 EWC 4.74±0.614.74_± 0.61 0.97±0.950.97_± 0.95 1.79±0.311.79_± 0.31 3.39±1.653.39_± 1.65 4.96±0.384.96_± 0.38 4.17±1.234.17_± 1.23 LwF 1.71±0.731.71_± 0.73 0.86±0.590.86_± 0.59 0.21±0.890.21_± 0.89 4.12±0.454.12_± 0.45 1.96±1.001.96_± 1.00 2.76±0.642.76_± 0.64 L2P+ 2.61±1.402.61_± 1.40 1.85±0.631.85_± 0.63 2.73±1.942.73_± 1.94 3.22±2.023.22_± 2.02 3.27±2.123.27_± 2.12 5.70±4.545.70_± 4.54 S-Prompt+ -0.04±0.95-0.04_± 0.95 0.37¯±0.19 0.37_± 0.19 1.60±0.371.60_± 0.37 0.45¯±1.60 0.45_± 1.60 1.25¯±0.48 1.25_± 0.48 1.02¯±0.91 1.02_± 0.91 VISTA (ours) 0.53¯±1.15 0.53_± 1.15 0.36±1.030.36_± 1.03 -0.26±1.16-0.26_± 1.16 0.15±1.150.15_± 1.15 0.87±0.490.87_± 0.49 0.66±1.140.66_± 1.14 B.5 Backbone Variants We further evaluate the same CE4L protocol with EgoVLPv2 (Pramanick et al., 2023) and VideoMAEv2 (Wang et al., 2023a) backbones. Tables 24 and 24 show that the relative behavior of CL methods and VISTA’s advantage are largely preserved across backbone choices, suggesting that the findings are not artifacts of a single encoder. Table 23: Additional results with EgoVLPv2 backbone. We report the final average performance under the Ego+Exo setting. Higher is better except planning ED@8 (↓ ). Method Skill Assoc. Planning Anticipation RAAN E→ X→ Ego Exo Ego-V Ego-N Exo-V Exo-N Fine-tuning 65.08 26.20 20.30 82.91 84.71 16.47 20.81 16.62 18.61 ER 76.92 29.30 23.40 81.43 82.21 21.72 26.90 21.83 23.81 DER++ 77.08 32.20 28.70 79.50 79.97 22.46 29.95 22.63 25.29 EWC 70.20 31.20 29.00 82.54 82.26 17.62 21.65 17.85 19.23 LwF 69.05 32.40 30.10 80.78 81.66 18.71 26.30 18.96 22.63 L2P+ 76.66 31.00 29.10 81.70 81.89 19.22 23.21 19.34 20.22 S-Prompt+ 77.36 32.60 28.40 81.81 81.15 19.86 22.94 19.77 20.52 VISTA (ours) 78.18 33.60 30.70 79.72 79.42 23.72 28.12 23.71 25.94 Table 24: Additional results with VideoMAEv2 backbone. We report the final average performance under the Ego+Exo setting. Method Skill Planning Anticipation RAAN Ego Exo Ego-V Ego-N Exo-V Exo-N Fine-tuning 71.15 81.59 82.04 20.38 26.24 20.35 22.69 ER 77.66 78.55 79.64 26.34 31.38 27.20 26.81 DER++ 78.46 77.72 78.64 26.78 31.26 27.67 26.82 EWC 76.02 80.79 80.50 21.26 28.18 22.29 24.07 LwF 77.96 78.54 81.18 23.05 31.76 23.02 28.33 L2P+ 79.88 78.90 79.72 22.49 28.59 23.72 24.30 S-Prompt+ 80.01 79.61 80.77 23.39 28.37 23.31 25.00 VISTA (ours) 81.67 78.36 78.31 27.87 32.12 28.84 27.83 Appendix C Additional VISTA Analysis C.1 Subspace Visualization Figure 9: Whitened-subspace distances across tasks. (a-e) We visualize pairwise distances between task-specific subspaces (spanned by BtB_t with q=32q=32) used by the router (smaller distance means more similar) across 5 different scenarios. Figure 10: Whitened-subspace distances across tasks. (a-e) We visualize pairwise distances between task-specific subspaces (spanned by BtB_t with q=32q=32) used by the router (smaller distance means more similar) across 5 different scenarios. We quantify task-level geometry by comparing augmented whitened-subspace bases through principal-angle discrepancy. Given two orthonormal bases Bi∈ℝd×riB_i ^d× r_i and Bj∈ℝd×rjB_j ^d× r_j, we compute the singular values σℓ\ _ \ of Bi⊤BjB_i B_j, which satisfy σℓ=cosθℓ _ = _ where θℓ\ _ \ are the principal angles between the two subspaces. We then define the discrepancy Dij=1m∑ℓ=1msin2θℓ,m=min(ri,rj),D_ij= 1m _ =1^m ^2 _ , m= (r_i,r_j), (12) so that smaller DijD_ij indicates more similar subspaces. C.2 Routing and Adapter Mixture Ablations Table 25 compares the whitened-subspace router with first-order and density-based alternatives. First-order methods are strong on some short-clip cases but degrade when longer inputs and view-induced variance dominate; Gaussian Mixture Model (GMM) and Mahalanobis modeling help in selected scenarios but remain less consistent overall. Table 26 compares hard and soft adapter selection. Top-1 routing is sufficient for well-separated skill subspaces, while residual-weighted top-2 mixing is more reliable on harder benchmarks. Table 25: Router comparison on CE4L. We report task-router accuracy (%,↑\%, ) using the same task-specific adapters and only changing the routing rule. “Whitened” denotes applying the same variance normalization used by VISTA. Router Skill Segmentation Association Planning Anticipation kkNN 88.50 55.95 57.96 44.56 54.94 Cosine 97.58 56.14 62.25 42.78 60.55 Whitened Cosine 98.31 57.14 70.96 42.78 61.15 GMM 88.06 64.29 59.25 44.81 54.23 Mahalanobis 88.78 65.00 62.88 50.63 53.58 Whitened Subspace (ours) 98.86 75.00 63.42 58.48 73.04 Table 26: Adapter mixture ablation across CE4L. We compare oracle routing, hard top-1 routing, unweighted top-2 averaging, and the residual-weighted top-2 mixture used by VISTA. Higher is better except planning ED@8 (↓ ). Mixture Rule Skill Segmentation Association Planning Anticipation RAAN Ego Exo Ego→ Exo→ Ego Exo Ego-V Ego-N Exo-V Exo-N Oracle 80.42 47.17 33.23 45.07 49.87 82.35 81.04 35.35 22.57 35.45 23.70 Top-1 79.62 42.70 27.74 41.50 38.10 83.85 82.72 34.68 20.81 32.37 20.80 Top-2 Avg. 76.05 42.15 30.91 40.50 37.55 83.96 81.14 32.82 15.10 31.13 15.30 Top-2 Mix (ours) 80.38 45.35 31.41 44.74 38.29 82.81 80.74 35.23 25.23 32.77 26.01 C.3 Computational Efficiency Table 27: Evaluation of computational efficiency on continual skill assessment benchmark. We report the number of trainable and total parameters in millions, storage cost in million floats, and average computation time cost in millisecond per training batch. Method Trainable (M) Total (M) Storage (M) Time (ms) Fine-tuning 1.84 1.84 0.00 32.7 ER 1.84 1.84 28.10 47.7 DER++ 1.84 1.84 28.10 49.6 EWC 1.84 1.84 3.68 126.8 LwF 1.84 1.84 1.84 71.9 L2P+ 2.38 2.38 0.00 67.6 S-Prompt+ 2.38 2.38 0.00 31.2 VISTA 2.38 2.38 0.04 34.7 Table 28: Evaluation of efficiency on continual temporal action segmentation benchmark. We report the number of trainable and total parameters in millions, storage cost in million floats, and average computation time cost in millisecond per training batch. Method Trainable (M) Total (M) Storage (M) Time (ms) Fine-tuning 0.74 0.74 0.00 395.0 ER 0.74 0.74 213.81 392.0 DER++ 0.74 0.74 219.50 364.4 EWC 0.74 0.74 1.48 1166.4 LwF 0.74 0.74 0.74 535.8 L2P+ 1.28 1.28 0.00 731.9 S-Prompt+ 1.28 1.28 0.00 359.7 VISTA 1.28 1.28 0.04 357.2 Table 29: Evaluation of computational efficiency on the continual action planning benchmark. We report the number of trainable and total parameters in millions, storage cost in million floats, and average computation time cost in millisecond per training batch. Method Trainable (M) Total (M) Storage (M) Time (ms) Fine-tuning 3.39 3.39 0.00 16.0 ER 3.39 3.39 6.97 91.5 DER++ 3.39 3.39 7.02 148.5 EWC 3.39 3.39 6.78 54.9 LwF 3.39 3.39 3.39 38.6 L2P+ 3.79 3.79 0.00 58.4 S-Prompt+ 3.79 3.79 0.00 25.0 VISTA 3.79 3.79 0.03 28.5 Table 30: Evaluation of computational efficiency on continual action anticipation benchmark. We report the number of trainable and total parameters in millions, storage cost in million floats, and average computation time cost in millisecond per training batch. Method Trainable (M) Total (M) Storage (M) Time (ms) Fine-tuning 3.24 3.24 0.00 19.3 ER 3.24 3.24 148.73 123.6 DER++ 3.24 3.24 149.46 169.0 EWC 3.24 3.24 6.47 83.4 LwF 3.24 3.24 3.24 29.8 L2P+ 4.05 4.05 0.00 37.7 S-Prompt+ 4.04 4.04 0.00 26.2 VISTA 4.04 4.04 0.03 19.9 All of our experiments are implemented using PyTorch on Ubuntu 22.04 LTS. The hardware environment consists of an Intel Xeon Gold 6530 CPU and 8 NVIDIA GeForce RTX 5090 GPUs. Notably, our proposed framework is efficient enough that each run requires only a single GPU for both training and inference. In the largest current VISTA setting, skill assessment uses C=d=1024C=d=1024, r=64r=64, and q=32q=32, corresponding to about 0.17M scalars per task (≈ 0.68 MB in fp32). This is far smaller than replay storage such as ER on skill assessment and anticipation, which stores 56.20M and 148.73M scalars respectively (Tables 28 and 30). Appendix D Additional Generality Analysis Beyond CE4L, we include two supplementary evaluations to investigate the applicability of VISTA. VISTA-Prompt replaces adapters with prompts on general CL image benchmarks under online blurry-boundary settings (Moon et al., 2023; Kang et al., 2025; Sun et al., 2026; Yan et al., 2026); results are shown in Table 32. We also report continual VLA results on LIBERO-10 (Liu et al., 2023; Römer et al., 2026) in Table 32, connecting the same design to embodied decision-making tasks. These experiments are not part of the main CE4L benchmark, but they indicate that the routing principle is applicable beyond continual multi-view video perception. Table 31: VISTA-Prompt on general continual learning benchmarks. We replace adapters with prompts and evaluate under online blurry-boundary settings. AAUCA_AUC denotes anytime accuracy AUC and AlastA_last denotes final average accuracy. The backbone is ViT-B/16 pretrained on ImageNet-21K. Method CIFAR-100 ImageNet-R CUB-200 AAUCA_AUC AlastA_last AAUCA_AUC AlastA_last AAUCA_AUC AlastA_last L2P 76.23 79.11 44.40 42.03 64.30 61.42 DualPrompt 76.04 76.62 46.13 40.80 65.03 62.43 CODA-Prompt 79.13 80.91 51.87 48.09 66.01 62.90 MISA 80.35 80.75 51.52 45.08 65.40 60.20 HiDe-Prompt 77.10 81.77 53.77 49.87 67.05 67.12 VISTA-Prompt (ours) 81.10 83.52 54.50 52.47 65.03 64.72 Table 32: Continual LIBERO-10 performance. We report success-rate AUC, forward transfer (FWT), and negative backward transfer (NBT). The model uses DINOv2, CLIP text features, and DiT flow matching, pretrained on LIBERO-90. We follow the standard pipeline of previous work (Römer et al., 2026). Method AUC (↑ ) FWT (↑ ) NBT (↓ ) SeqFFT 22.3722.37 76.13¯ 76.13 74.7074.70 SeqLoRA 21.3721.37 73.1073.10 71.6471.64 PackNet 4.844.84 37.2037.20 41.3441.34 ER 60.5460.54 76.60 22.7422.74 CLARE 74.07¯ 74.07 74.2774.27 0.18¯ 0.18 VISTA (ours) 74.92 74.1774.17 -0.39 D.1 Pseudo Code Algorithm 1 VISTA: Video Incremental Subspace-routed Task Adapters 1: Inputs: task stream tt=1T\D_t\_t=1^T, backbone fθf_θ, shared head gψg_ψ, subspace rank q, ϵ=10−6ε=10^-6, γ=10.0γ=10.0, top-K=2K=2. 2: 3: (1) Continual training and router fitting 4: Initialize seen task set ←∅S← 5: for t=1t=1 to T do 6: Train the current task adapter parameters ϕt _t on tD_t (freeze fθf_θ; freeze gψg_ψ after task 1). 7: ←∪tS ∪\t\ 8: Collect routing vectors rnn=1Nt←r(fθ(x)):x∈t\r_n\_n=1^N_t←\r(f_θ(x)):x _t\. 9: Compute mean and diagonal variance: μt←1Nt∑n=1Ntrn _t← 1N_t _n=1^N_tr_n, vart←1Nt∑n=1Ntrn⊙rn−μt⊙μtvar_t← 1N_t _n=1^N_tr_n r_n- _t _t. 10: Whitening weights: wt←vart+ϵw_t← var_t+ε. 11: Centered whitened residuals: zn←(rn−μt)⊙wtz_n←(r_n- _t) w_t. 12: Covariance: Σt←1Nt−1∑n=1Ntznzn⊤ _t← 1N_t-1 _n=1^N_tz_nz_n . 13: Top-q eigenvectors: Ut←eigq(Σt)∈ℝd×qU_t _q( _t) ^d× q. 14: Whitened mean direction: mt←μt⊙wt‖μt⊙wt‖2m_t← _t w_t\| _t w_t\|_2. 15: Augmented basis: Bt←QR([mtUt])∈ℝd×(q+1)B_t ([m_t\;U_t]) ^d×(q+1). 16: end for 17: 18: (2) Task-agnostic inference 19: for test input v do 20: x←fθ(v)x← f_θ(v); r←r(x)r← r(x). 21: for each t∈t do 22: r~t←r⊙wt r_t← r w_t. 23: et(r)←1−‖Bt⊤r~t‖22‖r~t‖22+ϵe_t(r)← 1- \|B_t r_t\|_2^2\| r_t\|_2^2+ε. 24: end for 25: pt(r)←softmax(−γe(r))p_t(r) (-γ\,e(r)). 26: Select top-K=2K=2 tasks tℓ=12\t_ \_ =1^2 with largest pt(r)p_t(r) and normalize πℓ\ _ \. 27: Adapter mixture: x~←∑ℓ=12πℓaϕtℓ(x) x← _ =1^2 _ \,a_ _t_ (x). 28: Prediction: y^←gψ(x~) y← g_ψ( x). 29: end for