Paper deep dive

MedCL-Bench: Benchmarking stability-efficiency trade-offs and scaling in biomedical continual learning

Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 57

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/22/2026, 5:45:52 AM

Summary

MedCL-Bench is a unified, task-diverse benchmark for evaluating continual learning (CL) strategies in biomedical NLP. It addresses catastrophic forgetting by streaming ten datasets across five task families (QA, fact checking, relation extraction, document classification, and multi-label topic classification) using eight task orders. The study evaluates eleven CL methods, finding that parameter-isolation (ADAPTER, TCL) and memory-based methods (GEM, REPLAY) provide the best stability-efficiency trade-offs, while sequential fine-tuning (VANILLA) suffers from severe performance regressions.

Entities (6)

ADAPTER · continual-learning-strategy · 100%Catastrophic forgetting · phenomenon · 100%GEM · continual-learning-strategy · 100%MedCL-Bench · benchmark · 100%Biomedical NLP · domain · 95%T5-base · language-model-backbone · 95%

Relation Signals (3)

Sequential fine-tuning → causes → Catastrophic forgetting

confidence 100% · direct sequential fine-tuning on incoming tasks induces catastrophic forgetting

MedCL-Bench → evaluates → Continual Learning Strategy

confidence 100% · evaluates eleven continual learning strategies across eight task orders

ADAPTER → mitigates → Catastrophic forgetting

confidence 90% · ADAPTER and TCL attain the strongest and most consistent AP

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Medical language models must be updated as evidence and terminology evolve, yet sequential updating can trigger catastrophic forgetting. Although biomedical NLP has many static benchmarks, no unified, task-diverse benchmark exists for evaluating continual learning under standardized protocols, robustness to task order and compute-aware reporting. We introduce MedCL-Bench, which streams ten biomedical NLP datasets spanning five task families and evaluates eleven continual learning strategies across eight task orders, reporting retention, transfer, and GPU-hour cost. Across backbones and task orders, direct sequential fine-tuning on incoming tasks induces catastrophic forgetting, causing update-induced performance regressions on prior tasks. Continual learning methods occupy distinct retention-compute frontiers: parameter-isolation provides the best retention per GPU-hour, replay offers strong protection at higher cost, and regularization yields limited benefit. Forgetting is task-dependent, with multi-label topic classification most vulnerable and constrained-output tasks more robust. MedCL-Bench provides a reproducible framework for auditing model updates before deployment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

56,588 characters extracted from source content.

Expand or collapse full text

MedCL-Bench: Benchmarking stability–efficiency trade-offs and scaling in biomedical continual learning Min Zeng, Shuang Zhou, Zaifu Zhan, Rui Zhang 1 Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA. *Corresponding author(s). E-mail(s): ruizhang@umn.edu; Abstract Medical language models must be updated as evidence and terminology evolve, yet sequential updating can trigger catastrophic forgetting. Although biomedi- cal NLP has many static benchmarks, no unified, task-diverse benchmark exists for evaluating continual learning under standardized protocols, robustness to task order and compute-aware reporting. We introduce MedCL-Bench, which streams ten biomedical NLP datasets spanning five task families and evaluates eleven continual learning strategies across eight task orders, reporting reten- tion, transfer, and GPU-hour cost. Across backbones and task orders, direct sequential fine-tuning on incoming tasks induces catastrophic forgetting, causing update-induced performance regressions on prior tasks. Continual learning meth- ods occupy distinct retention–compute frontiers: parameter-isolation provides the best retention per GPU-hour, replay offers strong protection at higher cost, and regularization yields limited benefit. Forgetting is task-dependent, with multi- label topic classification most vulnerable and constrained-output tasks more robust. MedCL-Bench provides a reproducible framework for auditing model updates before deployment. arXiv:2603.16738v1 [cs.AI] 17 Mar 2026 Large language models (LLMs) are increasingly used to support biomedical question answering, evidence retrieval, relation extraction, and document-level classi- fication. Yet biomedical knowledge is not static: new findings, revised clinical evidence, and evolving therapeutic guidance continually change what models should know. However, updating large models by full retraining is often computationally imprac- tical, while repeatedly fine-tuning on new datasets can erode previously acquired capabilities through catastrophic forgetting [1, 2]. The result is a tension between plas- ticity (incorporating new knowledge) and stability (preserving prior competencies) that is especially consequential in biomedical settings, where outdated or inconsistent behavior can directly undermine downstream research and decision-support. Con- tinual learning (CL) has therefore emerged as a promising paradigm for enabling models to acquire new knowledge over time while mitigating degradation of prior competencies [3]. In the clinical domain, these challenges are sharpened by three practical con- straints [4, 5]. First, privacy and data-governance constraints often limit the pooling of raw patient records across institutions, creating persistent “data walls” between sites [6, 7]. This motivates model-to-data workflows in which models are updated sequentially across hospitals (e.g., Hospital A → Hospital B) by transferring param- eters rather than sensitive primary data [8, 9]. Second, clinical data are inherently sequential and longitudinal[10, 11]: patient information accrues over time through repeated encounters, tests, treatments, and evolving diagnoses, while labeled datasets and task definitions often emerge incrementally rather than as a single static cor- pus. Third, clinical practice exhibits continual knowledge drift : diagnostic standards, treatment guidelines, and pathogen profiles can change over time [12–14], requiring models to incorporate new evidence without repeated full retraining. These constraints translate directly into deployment risk. Sequential updates can introduce silent regres- sions [13, 15]: a model may improve on newly introduced data while degrading on previously validated capabilities. In safety-critical workflows, such regressions increase the verification burden for clinicians and can undermine the reliability of downstream decision-support, including tasks such as adverse-event monitoring or drug–drug inter- action detection. This creates a practical need for rigorous evaluation of continual learning behavior under repeated updates. Despite growing interest in both biomedical Natural Language Processing (NLP) and continual learning, several questions remain unresolved in realistic biomedical deployment settings. In particular, it remains unclear how severe catastrophic forget- ting is in biomedical NLP, whether CL methods reliably mitigate forgetting and deliver meaningful benefits in biomedical NLP, which classes of CL strategies are most effec- tive for biomedical tasks, whether findings from general-domain CL transfer to settings with specialized biomedical language and heterogeneous data distributions, whether larger backbones consistently improve performance or instead exhibit non-monotonic, backbone-dependent trade-offs, and how these methods compare in terms of training cost, parameter efficiency, and update-time overhead. Biomedical NLP still lacks a unified, task-diverse benchmark for CL under a standardized training and evaluation protocol to answer these questions rigorously. To address these gaps, we introduce MedCL-Bench (Fig. 1), a unified contin- ual learning benchmark for biomedical NLP. MedCL-Bench is designed to answer three practical questions: (i) how severe catastrophic forgetting is in biomedical NLP, (i) which continual learning strategies offer the best trade-offs between end- of-stream performance and training cost, and (i) how sensitive conclusions are to task order and backbone choice under realistic resource constraints. Specifically, MedCL-Bench provides a standardized benchmark suite and evaluation pipeline for controlled comparison of continual learning strategies in biomedical NLP. First, we curate and standardize ten public biomedical NLP datasets into a standard- ized continual learning benchmark suite spanning five task families: biomedical question answering ( PubMedQA,BioASQ), scientific fact checking (SciFact, Pub- Health), relation extraction (GAD, ChemProt, DDI), document-level classification (PubmedRCT, DRUGLIB), and multi-label topic classification of biomedical liter- ature (LitCovid). Second, we evaluate sequential updates across eight pre-specified task orders with a unified preprocessing and evaluation protocol, enabling matched comparisons across continual learning strategies and backbone architectures. Third, we benchmark representative continual learning strategies—including naive fine-tuning, multi-task learning [16], regularization [17–19], rehearsal/gradient projection [20–22], generative replay [23–25], and parameter-efficient adaptation [26, 27]—and report end-of-stream performance together with compute- and parameter-efficiency (includ- ing GPU-hour cost) to characterize stability–efficiency trade-offs. Finally, we release the full benchmark codebase—preprocessing, training, and evaluation—to enable reproducible comparisons and further research. Using this benchmark, we further find that retention is not uniform across biomed- ical tasks. Tasks with more constrained output structure, such as multiple-choice question answering and multi-class relation extraction, are comparatively robust to sequential updates, whereas multi-label topic classification with overlapping label sets is markedly more vulnerable to forgetting. These patterns suggest that forgetting in biomedical continual learning depends not only on task difficulty, but also on task formulation and output structure. b) Problem: Catastrophic Forgetting a) Motivation: Sequential Updates in Biomedicaldomain e) Evaluation Metrics New Pubmed papersNew c linical trial evidenceNew dr ug-disease relations PubmedQA Task 1 SciF act Task 2 Task 3 Dr ugl ib RE Task 1 lost Task 2 lost Catastrophic forgetting Contin ual learnin g p reserves prior kn owled ge PubmedQA Task 1 SciF act Task 2 Task 3 Dr ugl ib RE Task 1 not lost Task 2 not lost Overall task performance Backward Transfer (how much previous task knowledge is forgotten) Forward Transfer (how much prior task knowledge is transferred) Realistic sequential updates Across institutions Within one institution Within one institution data can not be pooled Iterative updates; costly retraining 2023 2024 2025 2 Method Comparison 3 Order Sensitivity 4 Compute Efficiency 5 Scaling Behavior QA FactCheck RE DocClsMu lti Lab el PubMedQA BioAS Q SciFact PubHealth GAD ChemProt DDI PubMed_R CT DRUGLIB LitCovid c) MedCL-Bench: Datasets & Task Families 10 datasets 5 task families Language Model 푓 휃 0 Task Stream (푇 1 푇 2 푇 10 ) ⋯ QA QA Fact Fact Tr ain on 푇 1 Tr ain on 푇 2 Language Model 푓 휃 2 Tr ain on 푇 3 Language Model 푓 휃 3 Language Model 푓 휃 10 Language Model 푓 휃 1 Evaluate on 푇 1 Evaluate on 푇 1 , 푇 2 Evaluate on 푇 1 , 푇 2 , 푇 3 11Methods •EWC •ADAPTER •OLORA •REPLAY •GEM 8Tas k O r d er s (푇 1 푇 2 ⋯ 푇 3 (푇 2 푇 1 ⋯ 푇 3 Sequential updates d) Overview: Me dCL-Bench f) Key Questions 1 Forgetting Severity Fig. 1 | Overview of MedCL-Bench. (a) Biomedical knowledge and datasets evolve continuously (e.g., new literature and drug–disease relations), creating realistic sequential update streams—both across institutions (where data cannot be pooled) and within an institution over time. (b) Sequential sequential fine-tuning can overwrite previously acquired capabilities (catastrophic forgetting), whereas CL aims to retain prior knowledge while learning new tasks. (c) MedCL-Bench comprises ten biomedical NLP datasets grouped into five task families (QA, fact checking, relation extraction, document classification, and multi-label topic classification). (d) Benchmark workflow: a pretrained backbone is updated sequentially on a task stream under multiple task orders, and evaluated on all previously seen tasks after each stage. (e) CL metrics reported in this work: overall task performance (AP), backward transfer (BWT), and forward transfer (FWT). (f ) Key questions addressed: forgetting severity, method comparison, order sensitivity, compute efficiency, and scaling/backbone dependence. Icons are sourced from Flaticon.com (full attributions in Supplementary Note 1). Results We evaluate continual learning on MedCL-Bench, a stream of ten public biomedi- cal NLP datasets spanning five task types: biomedical question answering, scientific fact checking, relation extraction, document-level classification, and multi-label topic classification. A detailed overview of all datasets is provided in Extended Data Table 1. Tasks are presented sequentially under eight randomized task orders (Extended Data Table 2). Models are incrementally updated without access to future tasks. After each training stage, models are evaluated on all previously encountered tasks, enabling systematic assessment of both knowledge acquisition and retention. Unless stated oth- erwise, results use the T5-base backbone; additional backbones are considered only in the scaling experiments. Across all experiments, we report three continual learning metrics: average perfor- mance (AP; mean of per-task accuracies over all tasks after completing a task order), backward transfer (BWT; less negative indicates less forgetting), and forward transfer (FWT; less negative indicates better transfer) . Table 1 summarizes AP/BWT/FWT for each method under each of the eight task orders. Overall performance across methods Using Table 1, we highlight the main trends across methods. Multi-task learning (MULTI) provides an empirical upper bound, achieving consistently high AP (∼76%) across orders when all tasks are jointly optimized. Among continual learning meth- ods, ADAPTER and TCL attain the strongest and most consistent AP (72.01–73.27% and 69.75–71.37%, respectively), closely approaching the upper bound. GEM achieves higher AP but is more order-sensitive (66.83–73.69% across orders), whereas REPLAY improves over na ̈ıve sequential fine-tuning (VANILLA) yet remains lower overall (58.05–63.55%). VANILLA denotes standard sequential fine-tuning with no CL mechanism. For forgetting, BWT values reveal a sharp separation between method families. VANILLA shows severe forgetting with consistently negative BWT across all orders, whereas regularization baselines (EWC, L2) only partially mitigate forgetting and still yield negative BWT. In contrast, memory-based approaches—especially GEM and REPLAY—substantially mitigate forgetting, with BWT values markedly closer to zero. FWT is consistently negative across methods, indicating limited forward transfer. VANILLA forgets most (BWT down to −57.69), while replay/constraint methods improve retention (e.g., GEM: −6.88 to −1.33). Notably, OLORA remains highly forgetting-prone (BWT−43.92 to−29.33), underscoring that stability requires explicit retention mechanisms, not only parameter-efficient updates. Table 1 | Overall performance on MedCL-Bench. For each method, we report average task performance (AP), backward transfer (BWT), and forward transfer (FWT) across ten biomedical NLP tasks and eight randomized task orders. Higher is better; AP is highlighted in red. MethodTrain. Para. Metric ↑ Order1 Order2 Order3 Order4Order5Order6Order7Order8 VANILLA222M AP23.3836.2419.3734.4522.8734.3033.3224.21 BWT-52.07-31.72-57.69-41.95-49.95-41.78-42.80-50.65 FWT-22.75-21.22-26.05-23.55-28.19-23.23-32.98-27.80 EWC222M AP51.1750.9641.3448.8248.1345.3543.1656.19 BWT-17.07-25.10-33.82-27.31-25.32-30.72-31.65-19.65 FWT-22.10-21.51-25.92-23.49-28.00-23.68-32.88-27.81 L2222M AP53.3161.4261.0355.7351.8657.3362.1969.89 BWT-23.87-14.06-14.01-21.45-24.09-19.82-10.26-5.19 FWT-21.51-20.35-26.16-22.48-27.96-22.10-32.25-27.23 LAMOL222M AP54.0154.6547.6650.9249.5554.3351.1954.21 BWT-20.73-20.97-25.88-25.28-24.27-17.96-19.66-19.95 FWT-22.31-20.96-26.36-23.36-28.09-22.79-32.81-27.76 GEM222M AP69.7771.3570.8871.1071.1772.0666.8373.69 BWT-6.88-4.75-3.52-4.42-4.15-4.29-4.85-1.33 FWT-21.76-20.37-26.26-22.93-27.78-22.53-32.30-27.57 AGEM222M AP45.1350.9850.3059.1848.6852.8848.0959.16 BWT-32.21-24.05-26.41-17.90-26.88-24.13-26.53-15.93 FWT-22.59-21.17-26.33-23.33-27.81-23.46-32.81-27.69 REPLAY222M AP58.0561.9362.5763.5558.1158.7958.8760.89 BWT-15.10-10.81-9.41-9.12-16.43-5.29-12.68-6.52 FWT-22.38-21.28-26.31-23.33-27.94-23.23-33.01-27.65 OLORA2.5M AP39.0645.2646.8435.0539.3144.4539.3247.55 BWT-36.62-34.18-30.98-43.92-29.33-33.55-37.16-29.49 FWT-22.45-20.47-27.03-22.57-29.65-23.12-35.41-28.31 ADAPTER37.4M AP72.5572.0173.2772.4372.3672.5472.6972.02 BWT//////// FWT-22.92-21.31-26.60-23.57-28.05-23.72-32.91-27.72 TCL17.9 M AP70.2670.2770.6270.6371.1471.3770.1869.75 BWT//////// FWT-22.71-21.23-26.52-23.38-27.76-23.77-32.64-27.31 MULTI222M AP75.8875.7675.1576.0476.5076.2076.9776.12 BWT//////// FWT//////// Order robustness and statistical reliability Figure 2 quantifies sensitivity of final AP to task-order permutations. Fig. 2a reports mean final AP across eight orders with 95% bootstrap confidence intervals obtained by resampling task orders (n=8); MULTI provides an empirical upper bound with a narrow interval. Among CL methods, ADAPTER and TCL show the highest mean AP with the tightest intervals, whereas GEM achieves comparable mean AP but with wider inter- vals, indicating residual order sensitivity. REPLAY improves over VANILLA but remains lower on average. Complementing the CI-based view, Fig. 2b summarizes order sensitivity using the standard deviation of final AP across the eight orders. ADAPTER, TCL, and MULTI show the smallest observed variability, whereas GEM and REPLAY exhibit intermedi- ate sensitivity. Several baselines (notably VANILLA and L2) have the largest standard deviations, indicating that their outcomes can vary substantially under different task permutations. Per-order final AP values are shown in Supplementary Note 2. Treating task order as a matched block (n=8), paired exact sign-flip tests with Holm correc- tion confirm that ADAPTER and TCL outperform REPLAY, whereas L2 does not (Supplementary Note 3). 01020304050607080 Final AP VANILLA OLORA EWC AGEM LAMOL L2 REPLAY TCL GEM ADAPTER MULTI T5 Final AP across 8 task orders (mean ± 95% CI) (a) Mean with 95% bootstrap CI. Final AP aggregated over 8 task orders. ADAPTER TCL MULTI GEM REPLAY LAMOL OLORA EWC AGEM L2 VANILLA Method 0 1 2 3 4 5 6 7 STD(AP_final) across 8 task orders T5 order sensitivity (lower = more robust) (b) Order sensitivity. Standard deviation of final AP across orders (lower = more robust). Fig. 2 | Order robustness and statistical reliability on MedCL-Bench (T5- base). (a) Mean final AP across eight randomized task orders with 95% bootstrap confidence intervals for the mean obtained by resampling task orders (n=8). (b) Order sensitivity measured as the standard deviation (s.d.) of final AP across orders (lower indicates stronger robustness). Together, these panels summarize both average perfor- mance and sensitivity to task-order permutations. Forgetting Dynamics Across Task Orders To understand how task order affects stability during training, we examine stage- wise trajectories. Fig. 3a shows AP t , the mean accuracy over tasks seen up to stage t, under four task orders (Orders 1–4); trajectories for Orders 5–8 are provided in Supplementary Note 4. This metric provides an intuitive view of the stability–plasticity trade-off: sharp drops indicate substantial forgetting and interference with previously learned tasks, whereas flat trajectories indicate robust retention. Across all orders, VANILLA exhibits recurrent collapses as training progresses. Although the drops occur at different transition points depending on the permuta- tion, the overall pattern remains consistent: performance on previously learned tasks deteriorates under long-horizon sequential updates and only partially recovers. Memory-based methods provide the most consistent stabilization in these trajec- tories. In particular, REPLAY (experience replay) and GEM maintain comparatively smooth trajectories across orders, with markedly reduced drops when new tasks arrive. Notably, REPLAY interleaves a memory buffer of past samples during train- ing, whereas GEM additionally enforces gradient constraints—projecting updates to avoid increasing loss on stored examples—at the cost of extra computation. Regularization-based approaches (EWC, L2) provide only partial protection and still exhibit noticeable degradations at several transitions. Transition-level diagnosis of order sensitivity While the forgetting curves in Fig. 3a summarize the stage-wise evolution of AP t , they do not identify which task switches cause abrupt changes. We therefore analyze transition shock, defined as ∆AP t = AP t+1 − AP t (in percentage points), where AP t averages performance over all tasks observed up to stage t. Fig. 3b reports transition shocks for Order 1–4; Order 5–8 are provided in Supplementary Note 5. Across task orders, memory-based methods (REPLAY, GEM) and parameter- isolation methods (ADAPTER, TCL) tend to reduce the magnitude of negative transition shocks, whereas sequential fine-tuning (VANILLA) and regularization baselines (EWC, L2) exhibit larger drops at multiple switches, indicating stronger interference when new tasks are introduced. This transition-level view aligns with the aggregate trends in Table 1: methods with larger and more frequent negative shocks typically show more negative BWT (greater forgetting). Finally, while FWT captures transfer to a task before it is trained, ∆AP t reflects the net change after learning the next task (acquisition minus interference), so strong FWT does not necessarily imply small transition shocks. Taken together, these trajectory- and transition-level diagnostics show that method stability can depend on task permutations, motivating evaluation across multiple orders rather than relying on a single sequence. We next examine whether forgetting differs systematically across task families. BioASQ GAD Pubmed_RCT SciFact DRUGLIB LitCovid DDI PubMedQA PubHealth ChemProt 0 10 20 30 40 50 60 70 80 90 Average Performance (AP, %) Order 1 BioASQ GAD PubMedQA PubHealth ChemProt Pubmed_RCT SciFact DRUGLIB LitCovid DDI Order 2 PubMedQA PubHealth DRUGLIB BioASQ LitCovid DDI GAD ChemProt Pubmed_RCT SciFact 0 10 20 30 40 50 60 70 80 90 Average Performance (AP, %) Order 3 SciFact BioASQ DDI GAD PubMedQA LitCovid PubHealth DRUGLIB ChemProt Pubmed_RCT Order 4 Sequential learning (Task order →) VANILLA REPLAY LAMOL GEM AGEM EWC L2 ADAPTER TCL OLORA (a) Forgetting curves under four task orders. Trajectories of seen-task average accuracy across stages for Orders 1–4. Fig. 3 | Order-dependent forgetting dynamics in MedCL-Bench. (a) Forget- ting curves. (b) Transition shock heatmaps (next page). BioASQ→GAD GAD→Pubmed_RCT Pubmed_RCT→SciFact SciFact→DRUGLIB DRUGLIB→LitCovid LitCovid→DDI DDI→PubMedQA PubMedQA→PubHealth PubHealth→ChemProt Task transition (t → t+1) ADAPTER AGEM EWC GEM L2 LAMOL OLORA REPLAY TCL VANILLA -14.00.7-0.81.2-0.91.7-1.70.60.8 -41.627.18.7-10.9-10.414.5-0.9-11.8-7.6 -44.826.01.47.3-5.8-1.4-5.8-3.0-3.3 -9.1-4.24.58.6-1.80.3-2.9-0.8-2.8 -7.2-1.70.34.2-5.8-0.7-1.6-10.6-3.5 -3.31.1-3.33.9-3.8-7.6-4.5-2.2-0.7 -23.0-32.335.7-7.4-22.93.48.2-9.60.5 -13.81.5-1.05.6-3.80.4-5.6-0.4-2.6 -10.51.0-3.02.4-0.72.2-1.70.91.0 -43.523.1-0.312.2-6.0-4.20.4-33.2-5.1 Transition shock heatmap (ΔAP across task transitions) −40 −30 −20 −10 0 10 20 30 40 ΔAP (percentage points): AP(t+1) − AP(t) Order 1 BioASQ→GAD GAD→PubMedQA PubMedQA→PubHealth PubHealth→ChemProt ChemProt→Pubmed_RCT Pubmed_RCT→SciFact SciFact→DRUGLIB DRUGLIB→LitCovid LitCovid→DDI Task transition (t → t+1) ADAPTER AGEM EWC GEM L2 LAMOL OLORA REPLAY TCL VANILLA -13.6-2.33.02.3-0.20.01.2-0.51.2 -39.429.6-1.2-34.011.07.7-3.2-5.27.9 -45.233.7-9.8-29.5-14.816.88.13.77.8 -9.1-1.0-0.24.6-1.41.31.5-2.70.3 -7.2-4.0-0.2-1.2-11.86.24.4-4.2-0.5 -8.6-4.4-0.30.9-6.1-1.3-2.2-7.91.8 -11.0-5.0-13.2-28.1-8.917.3-5.9-5.018.0 -8.7-10.45.0-0.42.7-4.15.6-1.4-2.9 -10.5-4.33.22.50.2-1.31.1-0.61.4 -43.110.11.2-15.5-6.0-2.928.2-19.513.9 Transition shock heatmap (ΔAP across task transitions) −40 −20 0 20 40 ΔAP (percentage points): AP(t+1) − AP(t) Order 2 PubMedQA→PubHealth PubHealth→DRUGLIB DRUGLIB→BioASQ BioASQ→LitCovid LitCovid→DDI DDI→GAD GAD→ChemProt ChemProt→Pubmed_RCT Pubmed_RCT→SciFact Task transition (t → t+1) ADAPTER AGEM EWC GEM L2 LAMOL OLORA REPLAY TCL VANILLA 10.62.83.8-1.41.7-1.30.8-0.4-0.4 12.0-21.732.1-24.82.3-31.39.23.313.6 10.3-17.127.1-16.63.0-41.713.914.4-7.7 -1.315.76.1-2.42.2-0.91.0-2.3-2.9 12.610.4-1.4-6.80.1-16.43.4-1.34.6 10.7-9.01.31.74.4-0.5-1.2-7.7-7.8 4.5-28.634.4-26.5-16.9-6.423.01.6-0.4 9.710.5-2.1-5.24.4-6.10.1-4.21.7 11.03.42.8-1.32.0-2.01.0-0.1-1.4 4.4-7.317.6-9.9-15.3-33.815.5-9.2-3.3 Transition shock heatmap (ΔAP across task transitions) −40 −30 −20 −10 0 10 20 30 40 ΔAP (percentage points): AP(t+1) − AP(t) Order 3 SciFact→BioASQ BioASQ→DDI DDI→GAD GAD→PubMedQA PubMedQA→LitCovid LitCovid→PubHealth PubHealth→DRUGLIB DRUGLIB→ChemProt ChemProt→Pubmed_RCT Task transition (t → t+1) ADAPTER AGEM EWC GEM L2 LAMOL OLORA REPLAY TCL VANILLA 8.32.2-4.3-3.8-1.11.11.00.8-0.2 1.4-9.5-24.817.93.0-7.14.2-1.92.4 3.1-9.7-25.818.9-9.8-2.4-2.24.41.0 4.0-0.0-2.6-3.3-1.1-1.55.6-4.51.0 4.00.1-3.4-5.4-7.0-11.510.6-0.4-7.3 -15.66.5-4.71.3-4.4-5.70.30.4-3.8 -4.9-1.1-41.717.0-21.2-1.4-18.226.4-5.7 -6.42.2-3.1-4.0-12.39.55.0-2.9-0.5 8.93.8-4.9-3.2-0.51.31.21.1-0.0 -24.24.7-45.026.7-19.07.913.6-11.5-1.5 Transition shock heatmap (ΔAP across task transitions) −40 −30 −20 −10 0 10 20 30 40 ΔAP (percentage points): AP(t+1) − AP(t) Order 4 (b) Transition shock heatmaps under four task orders. ∆AP t = AP t+1 − AP t (per- centage points); cooler indicates interference. Task-family differences in forgetting To measure how much performance on each task degrades after subsequent updates, we compute per-task forgetting from the per-stage evaluations underlying Fig. 3a. Specifically, forgetting on task t is defined as ∆ = a post t − a end t (percentage points). Here, a post t is the accuracy on task t evaluated immediately after training on t (i.e., after stage t). a end t is the accuracy on the same task evaluated after completing the full 10-task stream (i.e., after the final stage). We then group the ten datasets into five task families using a fixed dataset-to-family mapping (Supplementary Note 6) and visualize the resulting forgetting distributions across task orders for a representative subset of methods (Fig. 4). For clarity, we omit MULTI because it is trained jointly and does not yield a sequential trajectory required to define forgetting, and we omit ADAPTER and TCL because their task-specific parameter isolation can yield near-zero forgetting, which would compress the scale and obscure differences among non-isolation methods. Across task orders, we observe clear family-level differences: MultiLabel tends to incur the largest forgetting, whereas QA and RE typically exhibit smaller losses, par- ticularly for replay-based and other stability-oriented methods (e.g., REPLAY, GEM, L2). Because ∆ can be negative (e.g., backward transfer or evaluation variability), we use the clipped measure max(∆, 0) as our primary forgetting metric, and report the raw (unclipped) distributions in Supplementary Note 7. QAFactCheckREDocClsMultiLabel Task family 0 20 40 60 80 100 Forgetting (percentage points) Family-level forgetting distribution by method (clipped at 0) VANILLA REPLAY L2 GEM AGEM EWC LAMOL OLORA Fig. 4| Task-family forgetting distributions by method. For each task, forget- ting is measured as ∆ = a post t − a end t (percentage points), i.e., the accuracy difference from immediately after learning the task to the end of the 10-task stream. Tasks are grouped into five families and distributions are aggregated across task orders. Boxes summarize the distribution over (order, task) instances and points show individual observations. We plot the clipped forgetting max(∆, 0) to isolate performance loss without allowing gains (e.g., backward transfer) to offset forgetting. Scaling to LLMs We next examine whether our findings generalize and scale across backbone archi- tectures, moving from an encoder–decoder model (T5-base) to decoder-only LLMs of different sizes. We additionally evaluate two decoder-only LLMs, Qwen-0.6B and Qwen-4B. Unless noted otherwise, scaling results use a single pre-specified task order (Order 1) and 5 epochs per task for tractability; the main benchmark uses 10 epochs per task (Table 1). We use the same unified task formulation and evaluation protocol as in the main experiments to ensure comparability across backbones. To contextualize scaling trends, Extended Data Table 3 reports the final AP and the number of trainable parameters for each method under each backbone. Fig. 5a visualizes the corresponding method-wise AP after completing the 10-task stream. Train counts parameters with requiresgrad=True. Backbone scaling exhibits method-dependent and non-monotonic effects Extended Data Table 3 and Fig. 5a show that scaling effects are strongly method- dependent and can be non-monotonic. Replacing T5 with a small decoder-only backbone (Qwen-0.6B) improves several baselines (e.g., VANILLA: 23.2→41.43; EWC: 30.45→44.98) but substantially degrades others (e.g., GEM: 68.83→38.22; ADAPTER: 68.43→59.67; TCL: 65.32→57.26), indicating that architectural changes alone do not uniformly mitigate catastrophic forgetting. Scaling further to Qwen-4B yields clearer improvements for many methods, particularly regularization-based approaches (EWC, L2) and replay-based training (REPLAY, LAMOL). In contrast, the multi-task upper bound increases only modestly (MULTI: 75.08→77.89), suggesting that additional capacity primarily reduces inter- ference in the sequential setting rather than dramatically raising the joint-training ceiling. Overall, backbone scaling reshapes the relative ranking of continual learning strategies rather than providing uniform gains across methods. Parameter-efficient methods benefit unevenly from scaling Parameter-efficient approaches exhibit nuanced scaling behavior. ADAPTER and TCL improve from Qwen-0.6B to Qwen-4B but do not recover the relative strength observed on T5, consistent with the fact that these methods freeze most backbone parameters and rely on limited task-specific capacity. As the backbone scales, the fraction of train- able parameters becomes increasingly small, potentially turning the adapter module into a bottleneck that limits effective adaptation. By contrast, the extremely compact OLORA shows only marginal improvement and remains substantially weaker than replay- or adapter-based methods, suggesting that aggressive parameter compression can undermine robustness to interference even on larger backbones. Gradient-projection methods are highly architecture-sensitive Gradient-projection-based GEM exhibits pronounced sensitivity to backbone archi- tecture. While GEM is among the strongest methods on the encoder–decoder T5 backbone, its performance drops sharply on decoder-only models (38.22% on Qwen-0.6B and 55.60% on Qwen-4B), falling below replay- and regularization-based approaches on the same backbones. A plausible explanation is that gradient-projection constraints can be more restrictive under decoder-only training dynamics, limiting effective updates compared to the encoder–decoder setting. We leave a mechanistic analysis of this architecture sensitivity to future work. Computational cost and efficiency trade-offs We measure training cost in GPU-hours, computed as wall-clock time multiplied by the number of GPUs used. Fig. 5b reports the absolute training cost of VANILLA for each backbone, while Figs. 5c–5e plot the average performance versus training-cost overhead normalized by the corresponding VANILLA run. Full per-method training costs for each backbone are reported in Supplementary Note 8. MULTI incurs a moderate overhead relative to VANILLA, ranging from roughly 2× on T5-base to ∼ 6× on Qwen-4B. In contrast, replay-based methods are consis- tently more expensive due to interleaving memory samples, requiring roughly 4–9× the GPU-hours of sequential fine-tuning across backbones. On T5-base, GEM is also substantially costlier (> 6×), reflecting its additional per-step gradient constraints. Taken together, these experiments reveal a nuanced scaling picture. (i) Moving from T5 to a small decoder-only LLM (Qwen-0.6B) does not automatically improve performance in biomedical NLP and can even degrade performance for some meth- ods. (i) Scaling further to Qwen-4B substantially mitigates catastrophic forgetting for many methods, particularly replay-based and regularization-based approaches. (i) Algorithmic design remains crucial: replay-based methods and adapter-style parame- ter isolation provide consistently strong stability–efficiency trade-offs, whereas GEM shows pronounced backbone sensitivity in our experiments. 0 20 40 60 80 100 VANILLA REPLAY GEM AGEM OLORA ADAPTER LAMOL EWC L2 TCL MULTI Average Performance T5Qwen3-0.6BQwen3-4B (a) Final AP across backbones. T5 Qwen-0.6B Qwen-4B 0 10 20 30 40 50 60 70 GPU-hours 0.79 15.14 63.33 VANILLA training cost per backbone (b) VANILLA cost (GPU-hours). 1234567 Relative training time (× VANILLA) 20 30 40 50 60 70 80 Average performance (AP, %) VANILLA REPLAY GEM ADAPTER TCL LAMOL EWC L2 OLORA AGEM MULTI (c) T5-BASE: cost vs. AP (× VANILLA time). 01234567 Relative training time (× VANILLA) 20 30 40 50 60 70 80 Average performance (AP, %) VANILLA REPLAY GEM ADAPTER TCL LAMOL EWC L2 OLORA AGEM MULTI (d) Qwen-0.6B: cost vs. AP (× VANILLA time). 0246810 Relative training time (× VANILLA) 20 30 40 50 60 70 80 Average performance (AP, %) VANILLA REPLAY GEM ADAPTER TCL LAMOL EWC L2 OLORA AGEM MULTI (e) Qwen-4B: cost vs. AP (× VANILLA time). Fig. 5 | Backbone scaling reshapes performance and efficiency trade-offs (Order 1). (a) Final AP after the 10-task stream on T5-base, Qwen-0.6B and Qwen- 4B. (b) Absolute training cost of VANILLA in GPU-hours for each backbone. (c–e) AP versus relative training cost for T5-base, Qwen-0.6B and Qwen-4B, where cost is normalized by the corresponding VANILLA run on the same backbone. Discussion In this study, we introduced MedCL-Bench, a continual learning benchmark that streams ten biomedical NLP datasets spanning five task families under a unified task formulation and evaluation protocol. Motivated by the need to update biomed- ical NLP models over time without retaining or retraining on all historical data, we use MedCL-Bench to characterize how CL strategies trade off adaptation to new tasks against retention of previously learned capabilities. Across task-order permu- tations, VANILLA shows substantial regression on earlier tasks, whereas memory- and parameter-isolation strategies yield more stable behavior, with regularization- based baselines providing more limited protection. More broadly, a central question is whether CL methods can reliably update biomedical models over evolving datasets while mitigating catastrophic forgetting, and which strategies offer the best trade-offs among end-of-stream performance, order robustness, parameter efficiency, and com- pute. MedCL-Bench is designed to address this question by evaluating methods across multiple task orders and complementary diagnostics that capture both end-of-stream outcomes and within-stream dynamics; we summarize the key findings below. First, MedCL-Bench exposes large and systematic differences in stability across CL strategies. Across eight randomized task orders, VANILLA exhibits severe catas- trophic forgetting, whereas replay/constraint-based methods and parameter-isolation approaches substantially improve retention (Table 1). Forward transfer remains lim- ited across methods on MedCL-Bench, underscoring that mitigating interference is the dominant challenge under long biomedical task streams. Second, robustness to task-order permutations is a necessary reporting dimension rather than a secondary diagnostic. Higher AP does not necessarily imply greater order robustness: some methods achieve competitive averages yet remain order-sensitive, whereas ADAPTER and TCL combine high performance with consistently low across- order variability (Fig. 2). Several baselines show pronounced variability across orders, so conclusions from a single permutation can be unreliable; ADAPTER and TCL com- bine high average performance with the narrowest uncertainty intervals, indicating stronger order robustness. Paired matched-order tests further show that key method differences persist across permutations under correction (Supplementary Note 3), supporting the use of task orders as matched blocks for statistical reliability. Third, forgetting can be concentrated at specific task transitions. We quantify these drops with transition shock ∆AP t (Fig. 3b), and larger or more frequent negative shocks are consistent with more negative BWT (Table 1). This transition-level view complements aggregate metrics by identifying when stability mechanisms succeed or fail during the stream. Fourth, forgetting is heterogeneous across task families, highlighting clinically rel- evant failure modes that can be masked by overall averages. When grouping datasets into five task families, we observe consistent family-level differences in forgetting, with some families exhibiting systematically larger loss across orders (Fig. 4). This hetero- geneity suggests that benchmark reporting should include task-family diagnostics in addition to a single averaged score, especially in biomedical settings where failure on specific task types may be unacceptable even if the overall mean appears competitive. Fifth, scaling to decoder-only LLMs reshapes the relative ranking of CL methods and reveals pronounced architecture dependence. Moving from an encoder–decoder backbone (T5-base) to decoder-only backbones (Qwen-0.6B/4B) does not uni- formly improve continual learning; instead, scaling effects are method-dependent and can be non-monotonic (Extended Data Table 3; Fig. 5a). Regularization- and replay- based approaches benefit more consistently from increased decoder-only capacity, whereas parameter-efficient and gradient-projection methods exhibit stronger back- bone dependence. In particular, GEM degrades on decoder-only models in our setting, indicating that the relative ranking of CL strategies can be backbone-dependent and motivating backbone-aware validation before deployment. Sixth, stability gains come with a clear efficiency trade-off that should be reported alongside performance. Replay/constraint-based methods often deliver strong reten- tion and lower transition shocks, but can incur substantial GPU-hour and memory overhead due to buffer maintenance and additional optimization constraints. In contrast, parameter-efficient approaches reduce the number of trainable parame- ters and can be more practical under limited compute, yet their benefits may saturate or become capacity-bottlenecked as backbones and task streams scale (Sup- plementary Note 8). These results highlight that method selection in biomedical settings should consider not only end-of-stream AP and robustness, but also the stability–compute–parameter trade-off under realistic update budgets. Together, these results support two deployment-relevant recommendations. (i) When continual updates are expected, evaluate and select methods under multiple task orders and include uncertainty, since single-order reporting can be misleading. (i) Choose methods based on both stability and compute: replay/constraint-based approaches offer strong retention but may incur substantial GPU-hour overhead, while parameter-efficient approaches reduce trainable parameters yet can become capacity-bottlenecked as backbones scale (Supplementary Note 8). This study has several limitations. First, while the main benchmark evaluates multiple task orders, scaling experiments use a single representative order and fewer training epochs for tractability; extending scaling analyses to multiple permutations and longer horizons will strengthen generality. Second, our largest decoder-only model is limited to 4B parameters; evaluating larger biomedical LLMs may reveal addi- tional scaling regimes. Third, we focus on accuracy under a unified task formulation; future work should incorporate additional deployment-facing criteria (e.g., calibra- tion, robustness, or clinically weighted errors). Finally, exploring hybrid strategies that combine selective replay with parameter isolation, as well as principled memory sizing and sampling, may further improve the stability–compute trade-off. In summary, MedCL-Bench fills a key gap in biomedical NLP by providing a unified continual learning benchmark that spans five task families, standardizes train- ing and evaluation, and evaluates methods under multiple pre-specified task orders with complementary diagnostics. Using MedCL-Bench, we answer under-explored questions that motivate continual updating in practice: which CL strategies best prevent regressions under long biomedical task streams, how sensitive conclusions are to task-order permutations, and whether forgetting and interference differ sys- tematically across task families. We further show that these conclusions depend on backbone architecture and are shaped by a clear stability–efficiency trade-off, motivat- ing backbone-aware and budget-aware method selection. Overall, our findings indicate that CL outcomes are jointly governed by algorithm design, task-order permutations, task-family heterogeneity, and backbone architecture, providing practical guidance and more robust evaluation standards for updating biomedical language models under realistic constraints. Methods Tasks and datasets MedCL-Bench benchmarks CL on ten publicly available biomedical NLP datasets spanning five task categories: biomedical question answering (BioASQ, PubMedQA), scientific fact checking (SciFact, PubHealth), relation extraction (GAD, ChemProt, DDI), document-level classification (PubMed-RCT, DRUGLIB), and multi-label topic classification of biomedical literature (LitCovid). We adopt the released train/valida- tion/test splits provided by the original datasets or the Zenodo benchmark release when applicable, and only construct missing splits when they are not provided. For SciFact, the official test split is unlabeled; we therefore use the original validation split as the test and create a new validation split from the training data. For DRUGLIB, which provides train/test only, we create a validation split by sampling 10% of the training set. For BioASQ (Task 7b), we retain only yes/no questions with gold answers inyes,no, use the first snippet as context, and create a deterministic train/validation split by hashing question IDs; the golden enriched set is used as test. For PubMed- RCT, we apply light text cleaning and low-information filtering within each split, and (for controlled experiments) subsample up to 1,000 training sentences per section label. For PubHealth, we convert the released parquet files into our unified JSONL format, keeping the claim text and the four-way labels (true/false/mixture/unproven) after filtering malformed entries. For GAD, we keep the provided splits and rewrite each example into a unified binary MCQ format, where the input asks whether a gene– disease relation is present and provides two options (A: has relation, B: norelation); we map the original labels to A/B and keep the placeholder-marked text (e.g., @GENE, @DISEASE) as-is. For the remaining datasets (PubMedQA, ChemProt, DDI, and LitCovid), we directly use the released train/validation/test splits from the Zenodo benchmark release (record 14025500) as-is. Experimental setup Model backbones Unless stated otherwise, all main benchmark experiments use T5-base as the pre- trained backbone (encoder–decoder). For scaling experiments, we additionally evaluate two decoder-only backbones, Qwen-0.6B and Qwen-4B. Continual learning protocol MedCL-Bench evaluates continual learning by exposing a pretrained backbone to a stream of T =10 tasks. Tasks are presented sequentially under eight pre-specified task- order permutations. We fix the set of orders and use the same orders for all methods to enable matched comparisons across algorithms. For the main benchmark results, we aggregate performance over the eight orders and report order sensitivity/uncertainty (e.g., s.d. and bootstrap CIs) to guard against order-specific conclusions. For the LLM- scaling experiments, we use a single order (Order 1) for computational tractability, as detailed in the Results section (“Scaling to LLMs”). Data curation and split caps To ensure a controlled and computationally consistent setting, we cap each dataset split on-the-fly to at most 1,000 training instances, 500 validation instances, and 500 test instances. When a split exceeds the cap, we apply fixed-seed stratified subsampling within that split to preserve label proportions. For datasets that do not provide all official splits (e.g., missing validation or a labeled test split), we construct the missing split(s) following the dataset-specific procedures described above. Unified input/output format We cast all datasets into a unified discriminative classification format. Each instance is represented as a single text input (the sentence field), and the model predicts a label from a task-specific closed label set. For QA tasks, the input provides a question and supporting context (and may optionally enumerate answer options), and the model predicts one answer label (BioASQ: yes/no; PubMedQA: yes/no/maybe). For rela- tion and classification tasks (e.g., ChemProt, DDI, GAD, PubMed-RCT, DRUGLIB, SciFact, PubHealth), the model predicts one categorical label. For LitCovid, labels are multi-valued and represented as a semicolon-separated set; we evaluate predictions by exact set match (subset accuracy) after splitting labels by ; in an order-insensitive manner. Evaluation metrics MedCL-Bench spans heterogeneous biomedical task types whose original papers use different evaluation measures (e.g., accuracy, micro-/macro-F1). Directly aggregating such task-specific metrics would make CL summaries ill-defined and can introduce scale inconsistencies when averaging across tasks and task orders. We therefore cast all datasets into a unified classification setting and use accuracy as the primary metric, enabling consistent cross-task comparison and statistically coherent continual learning aggregates. This design follows common practice in multi-task and continual learning benchmarks [26, 28, 29] that enforce a shared output space (e.g., mapping hetero- geneous tasks to a unified discriminative objective or to end-to-end generation) to support comparable aggregate measures. All results are reported as percentages. For single-label tasks, an example is correct if the predicted label matches the gold label. For LitCovid (multi-label), we use subset accuracy (exact match): an example is counted as correct if and only if the predicted label set exactly matches the gold label set (order-insensitive, split by ;). Additionally, to obtain overall continual learning comparisons, we follow standard definitions [20, 26, 28, 29] based on the task-wise accuracy matrix. Let R t,i denote the accuracy on task i after training up to task t (with i≤ t) in the sequence, with T total tasks. In our benchmark, T =10. (1) Average Performance (AP) is the average accuracy across all tasks after learning all the tasks: AP = 1 T T X i=1 R T,i . (2) Backward Transfer (BWT) [20, 30] quantifies the impact of new learning on previous tasks: BWT = 1 T − 1 T−1 X i=1 ( R T,i − R i,i ) . (3) Forward Transfer (FWT) measures the influence of previously learned tasks on future tasks before they are trained, relative to the initial pretrained (zero-shot) baseline: FWT = 1 T − 1 T X i=2 ( R i−1,i − R 0,i ) , where R 0,i denotes the performance on task i of the initial pretrained model before learning any tasks (zero-shot baseline). Baselines We compare representative baselines spanning sequential fine-tuning, multi-task learn- ing, regularization, rehearsal/gradient-projection, and generative replay. All methods share the same backbone, preprocessing, task orders, and evaluation protocol. Sequential fine-tuning (VANILLA) trains on tasks one by one with standard fine- tuning and no explicit mechanism to mitigate forgetting. Multi-task learning : MULTI jointly trains on the union of all tasks and serves as a non-continual reference. Regularization-based : EWC [17] adds a Fisher-based penalty to prevent changing parameters important for previous tasks. L2 anchors parameters to the previous-task solution via an ℓ 2 penalty. Rehearsal/gradient projection: REPLAY [31] maintains an episodic memory of past examples and mixes them with current-task data. GEM [20] and AGEM [21] use episodic memory to constrain updates and reduce interference with past tasks. Generative replay : LAMOL [25] performs replay by augmenting training with pseudo-samples generated from a model prior to learning the current task. Parameter-efficient adaptation: ADAPTER [26] inserts lightweight bottleneck adapters into the backbone and freezes all original backbone parameters, training only the adapter (and other explicitly enabled lightweight) parameters. TCL [27] and OLORA [32] are additional parameter-efficient baselines that restrict training to a small set of task-conditioned lightweight parameters (e.g., task embeddings / low-rank updates) while keeping the backbone frozen, following their standard practice. Hyperparameter control Unless otherwise specified, we keep shared optimization hyperparameters (e.g., epochs, learning rate, batch sizes, early stopping) consistent across baselines for a given back- bone. Method-specific hyperparameters (e.g., episodic memory size M and LAMOL augmentation ratio ρ) are fixed across methods whenever they represent a shared resource budget, to ensure fair comparisons. Hyperparameters: for rehearsal-based methods (REPLAY/GEM/AGEM), we use an episodic memory of size M = 5 per task in our implementation. For LAMOL, we set percentage LAM0L=0.1, meaning that for each task we generate pseudo-samples amounting to 10% of its training set using the model snapshot before learning that task. For ADAPTER, we use a bottleneck size of 48 on T5 and update only adapter- related parameters. For the T5-based ADAPTER, we use a bottleneck size of 48. For Qwen-based ADAPTER, we follow the same adapter-only training protocol and use a bottleneck size of 128/512 for 0.6B and 4B, respectively. We use OLoRA with rank r = 8 and α = 16. Implementation details All methods are implemented in PyTorch with HuggingFace Transformers and Deep- Speed. For each backbone, we use the same optimizer and training budget (epochs, learning rate, and batch sizes) across methods, and report results from the final check- point. After finishing each task, we evaluate the model on all tasks seen so far to form the accuracy matrix R t,i used to compute AP/BWT/FWT. The experiments were run on NVIDIA A100 40GB GPUs. Use of large language models ChatGPT was used for language polishing; all scientific content and final text were authored and verified by the authors. Data Availability All datasets used in this work are publicly available. • PubMedQA: https://zenodo.org/records/14025500 • BioASQ (Task 7b): the official training set (Training 7b) and test gold annotations (7b golden enriched) are available from the BioASQ Participants Area Datasets page (registration required for downloading the training set): https://participants-area. bioasq.org/datasets/ • PubHealth:https://huggingface.co/datasets/ImperialCollegeLondon/health fact • SciFact: https://github.com/allenai/scifact (also mirrored at https://huggingface. co/datasets/allenai/scifact). • GAD:https://huggingface.co/datasets/bigbio/gad • ChemProt: https://zenodo.org/records/14025500 • DDI: https://zenodo.org/records/14025500 • PubmedRCT:https://raw.githubusercontent.com/Franck-Dernoncourt/ pubmed-rct/master/PubMed20kRCT • DRUGLIB: https://archive.ics.uci.edu/dataset/461/drug%2Breview%2Bdataset% 2Bdruglib%2Bcom • LitCovid: https://zenodo.org/records/14025500 Code Availability The code for MedCL-Bench, including dataset preprocessing scripts, continual learning configurations, and baseline implementations, has been deposited in Zenodo for peer review and will be made publicly available upon publication. References [1] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation vol. 24, p. 109–165. Elsevier, ??? (1989) [2] He, F., Fei, R., Krull, J.E., Yu, Y., Zhang, X., Wang, X., Cheng, H., Gao, M., Su, L., Chen, Y., et al.: Harnessing the power of single-cell large language models with parameter-efficient fine-tuning using scpeft. Nature Machine Intelligence, 1–16 (2025) [3] Mendez, J.A., Eaton, E.: How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition. Trans. Mach. Learn. Res. 2023 (2023) [4] Kiyasseh, D., Zhu, T., Clifton, D.: A clinical deep learning framework for con- tinually learning from cardiac signals across diseases, time, modalities, and institutions. Nature Communications 12(1), 4221 (2021) [5] Lee, C.S., Lee, A.Y.: Clinical applications of continual learning machine learning. The Lancet Digital Health 2(6), 279–281 (2020) [6] Guinney, J., Saez-Rodriguez, J.: Alternative models for sharing confidential biomedical data. Nature Biotechnology 36(5), 391–392 (2018) https://doi.org/ 10.1038/nbt.4128 [7] A question of trust for AI research in medicine. Nature Machine Intelligence 6, 739 (2024) https://doi.org/10.1038/s42256-024-00880-0 . Editorial [8] Bergquist, T., et al.: Piloting a model-to-data approach to enable predictive analytics in health care through patient mortality prediction. Journal of the American Medical Informatics Association 27(9), 1393–1400 (2020) https://doi. org/10.1093/jamia/ocaa083 [9] McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.: Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Arti- ficialIntelligenceandStatistics(AISTATS),p.1273–1282(2017). https://proceedings.mlr.press/v54/mcmahan17a.html [10] Abul-Husn, N.S., Kenny, E.E.: Personalized medicine and the power of electronic health records. Cell 177(1), 58–69 (2019) https://doi.org/10.1016/j.cell.2019.02. 039 [11] Huguet, N., Kaufmann, J., O’Malley, J., Angier, H., Hoopes, M., DeVoe, J.E., Marino, M.: Using electronic health records in longitudinal studies: estimating patient attrition. Medical care 58, 46–52 (2020) [12] Sahiner, B., Chen, W., Samala, R.K., Petrick, N.: Data drift in medical machine learning: implications and potential remedies. British Journal of Radiology 96(1150), 20220878 (2023) https://doi.org/10.1259/bjr.20220878 [13] Finlayson, S.G., Subbaswamy, A., Singh, K., et al.: The clinician and dataset shift in artificial intelligence. New England Journal of Medicine 385(3), 283–286 (2021) https://doi.org/10.1056/NEJMc2104626 [14] Lasko, T.A., et al.: Why do probabilistic clinical models fail to transport between sites. npj Digital Medicine (2024) https://doi.org/10.1038/s41746-024-01037-4 [15] Kore, A., Abbasi Bavil, E., Subasri, V., et al.: Empirical data drift detection experiments on real-world medical imaging data. Nature Communications 15, 1887 (2024) https://doi.org/10.1038/s41467-024-46142-w [16] Caruana, R.: Multitask learning. Machine learning 28, 41–75 (1997) [17] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017) [18] Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence, p. 3987–3995 (2017) [19] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Proceedings of the European Conference on Computer Vision (ECCV), p. 139–154 (2018) [20] Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2017) [21] Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 (2018) [22] Mi, F., Chen, L., Zhao, M., Huang, M., Faltings, B.: Continual learning for natural language generation in task-oriented dialog systems, p. 3461–3474 (2020) [23] Zhao, Y., Zheng, Y., Tian, Z., Gao, C., Yu, B., Yu, H., Li, Y., Sun, J., Zhang, N.L.: Prompt conditioned vae: Enhancing generative replay for lifelong learning in task-oriented dialogue. (2022) [24] Zeng, M., Yang, H., Xue, W., Liu, Q., Guo, Y.: Dirichlet continual learning: Tackling catastrophic forgetting in nlp. In: The 40th Conference on Uncertainty in Artificial Intelligence (2024) [25] Sun, F.-K., Ho, C.-H., Lee, H.-Y.: Lamol: Language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329 (2019) [26] Madotto, A., Lin, Z., Zhou, Z., Moon, S., Crook, P., Liu, B., Yu, Z., Cho, E., Fung, P., Wang, Z.: Continual learning in task-oriented dialogue systems. (2021) [27] Zeng, M., Yang, H., Chen, X., Guo, Y.: Task-wrapped continual learning in task- oriented dialogue systems. In: Findings of the Association for Computational Linguistics: NAACL 2025, p. 3173–3183 (2025) [28] Zhang, Y., Wang, X., Yang, D.: Continual sequence generation with adaptive compositional modules. In: Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), p. 3653–3667 (2022) [29] Zeng, M., Chen, X., Yang, H., Guo, Y.: Sparse adapter fusion for continual learning in nlp. arXiv preprint arXiv:2602.02502 (2026) [30] Zhu, Q., Li, B., Mi, F., Zhu, X., Huang, M.: Continual prompt tuning for dialog state tracking, p. 1124–1137 (2022) [31] Robins, M.J.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science 7(2), 123–146 (1995) https://doi.org/10.1080/09540099550039318 [32] Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., Huang, X.-J.: Orthogonal subspace learning for language model continual learn- ing. In: Findings of the Association for Computational Linguistics: EMNLP 2023, p. 10658–10671 (2023)

Instant research discovery