Paper deep dive
Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection
Yuankun Xie, Xiaoxuan Guo, Xiaopeng Wang, Siqing Qin, Shaole Li, Kong Aik Lee
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continual audio deepfake detection requires learning newly emerging deepfake methods while retaining discrimination of previously encountered speech. Existing dataset-incremental evaluation changes both real-speech domains and deepfake mechanisms, making their effects difficult to distinguish. We construct five task organizations over identical training, development, and evaluation pools to study these factors under a controlled sample budget. Our proposed Real-Anchored Mechanism-Incremental (RAMI) protocol reflects the practical setting in which available real speech provides a recurring mixed-domain reference while new deepfake mechanisms arrive incrementally. We further propose RF-Prompt, an asymmetric continual prompt-learning method that preserves reusable real-speech knowledge through a shared real prompt and expands mechanism-specific knowledge through inherited fake experts with orthogonal residuals. Input-adaptive soft fusion combines the accumulated experts into a fixed number of injected tokens without requiring task identity at inference. On RAMI, RF-Prompt achieves 10.110% average EER and 10.370% pooled EER, outperforming all evaluated continual-learning baselines. Across the five controlled protocols, RAMI yields the lowest common-average and pooled EER. Component ablations, limited-data experiments, and cross-backbone evaluations further validate the proposed design.
Tags
Links
- Source: https://arxiv.org/abs/2609.37586v1
- Canonical: https://arxiv.org/abs/2609.37586v1
Trouble viewing inline? Open PDF directly →
Full Text
72,908 characters extracted from source content.
Expand or collapse full text
Preprint LEARNING AS DEEPFAKES EVOLVE: RF-PROMPT FOR CONTINUAL AUDIO DEEPFAKE DETECTION Yuankun Xie 1 , Xiaoxuan Guo 2 , Xiaopeng Wang 3 , Siqing Qin 1 , Shaole Li 1 , Kong Aik Lee 1 1 The Hong Kong Polytechnic University, Hong Kong SAR, China 2 Communication University of China, Beijing, China 3 Beijing Institute of Technology, Beijing, China ABSTRACT Speech generation methods are evolving rapidly, creating a moving target for audio deepfake detection (ADD). A deployed detector must incorporate newly emerging deepfake methods without forgetting previously learned real and deep- fake knowledge. Continual learning provides a natural solution, but existing con- tinual ADD evaluations commonly define tasks by dataset, coupling changes in real-speech sources with changes in deepfake mechanisms and obscuring what knowledge is being updated. We address task organization and detector adaptation jointly. First, we construct five protocols from identical training, development, and evaluation pools. Among them, the proposed real-anchored mechanism- incremental (RAMI) protocol reflects a practical detector-update scenario: real speech from known source domains is available, while newly arriving deepfake mechanisms must be learned continually without forgetting earlier knowledge. Second, we propose RF-Prompt (real–fake prompt learning), an asymmetric con- tinual prompt-learning method. A shared real prompt provides protected adapta- tion capacity, while task-specific fake experts expand as new mechanisms arrive. Parameter-level cosine anchoring stabilizes the real prompt; each new fake expert inherits a selected historical expert and learns a residual with soft orthogonal reg- ularization. Input-adaptive fusion integrates the accumulated fake experts with- out requiring task identity at inference. RF-Prompt obtains its lowest common- average and pooled EER under RAMI among the five tested protocols. On RAMI, it achieves 10.110% final average EER and 10.370% pooled EER, the lowest ag- gregate errors among the evaluated continual-learning baselines. Ablations assess the adaptation components, while limited-training-data and cross-backbone ex- periments examine their applicability across settings. Code is available online 1 . 1INTRODUCTION Speech generation is becoming increasingly accessible, expressive, and diverse. Industrial systems such as Alibaba’s Qwen3-TTS, ByteDance’s Seed-TTS, OpenAI’s GPT-Live, and Google’s Gemini Live are evolving rapidly, with new models and updates appearing on monthly or even weekly cycles (Hu et al., 2026; Anastassiou et al., 2024; OpenAI, 2026; Google, 2026). This rapid evolution creates a moving target for audio deepfake detection (ADD): a deployed detector must continually confront generators that were unavailable during training. Reliable ADD therefore requires not only generalization to unfamiliar deepfake methods but also adaptation to newly arriving methods without forgetting earlier ones. Existing research addresses this challenge through complementary data- and method-centric ap- proaches. Benchmark development, including ASVspoof 2019, the ADD challenge, ASVspoof 5, CodecFake, and AT-ADD, among others, broadens the available coverage of attacks and record- ing conditions (Todisco et al., 2019; Yi et al., 2022; Wang et al., 2025; Xie et al., 2024b; 2026). On the method side, approaches such as domain generalization aim to learn domain-invariant rep- resentations from limited training data, as exemplified by ASDG (Xie et al., 2024a) and related 1 https://github.com/xieyuankun/RF-Prompt 1 arXiv:2609.37586v1 [cs.SD] 29 Sep 2026 Preprint Continually Emerging Audio Deepfake Methods speech units / waveform DSP / Parametric Vocoder Text or linguistic units ... Neural Acoustic Model Continuous acoustic features Neural Vocoder / Decoder Raw waveform Neural Codec Encoder Discrete codec tokens ... Neural Codec Decoder Audio Token & Text Token Speech Language Model Acoustic Backend M1 Classical Pipeline 5 / 6 / 9 M2 Neural Acoustic Pipeline 24 / 23 / 32 M3 Neural Codec Pipeline 6 / 6 / 7 M4 Speech-LM Pipeline 5 / 5 / 13 Shared Real Prompt M1 Fake Prompt M2 Fake Prompt M3 Fake Prompt M4 Fake Prompt Soft MoE Fused Fake Prompt Continual Audio Deepfake Detector Real / Fake Real Speech Discrete token sequence ... ... Figure 1: Overview of mechanism-incremental audio deepfake detection. RF-Prompt preserves shared real-speech knowledge while expanding fake experts for emerging deepfake mechanisms. Counts below M1–M4 indicate the numbers of generators in the training, development, and evalua- tion splits. domain generalization approaches (Zhang et al., 2021; Kim et al., 2024; Huang et al., 2025). How- ever, a fixed training set cannot cover continually emerging forgery methods, motivating continual adaptation to new attacks while retaining previously acquired knowledge. Meanwhile, advances in pretrained audio models have substantially improved ADD performance. However, further scaling does not necessarily yield comparable gains in robustness under distribution shift, as recent scaling experiments demonstrate (Li et al., 2026). Generalization-oriented methods cannot guarantee reli- able detection of unseen forgery methods, nor do they explicitly address how to incorporate newly emerging attacks without forgetting previously acquired knowledge. This motivates continual adap- tation alongside generalization to unfamiliar generators. Continual learning offers a natural framework for addressing this need: incorporating newly emerg- ing attacks while retaining previously acquired knowledge. It enables learning from successive tasks without repeatedly retraining on the complete historical dataset. General approaches such as EWC and OWM protect earlier knowledge through parameter regularization or constrained updates (Kirk- patrick et al., 2017; Zeng et al., 2019). DFWF pioneered continual learning for ADD by combining knowledge distillation with real-embedding alignment (Ma et al., 2021). RAWM subsequently in- troduced adaptive weight modification and previous-model output regularization, followed by RWM and RegO, which further account for real–fake distribution differences and parameter-region impor- tance (Zhang et al., 2023; 2024c; Chen et al., 2025). Yet two key questions remain: how should continual ADD tasks be organized to reflect emerging deepfake generation methods, and how can a detector acquire new deepfake knowledge while preserving previously learned real and fake knowl- edge? Unlike conventional class-incremental recognition, the output classes in continual ADD re- main real and fake; what evolves is the distribution within those classes. Learning a new task must not sacrifice the ability to detect previously encountered attacks. Rethinking task organization. Prior continual ADD evaluations, including EVDA as used by RegO, commonly organize learning tasks by dataset (Zhang et al., 2024b; Chen et al., 2025). How- ever, new deepfake generators do not necessarily arrive with new real-speech domains: in practi- cal detector updates, an existing database may already contain abundant real speech from known sources, while additional deepfake methods continue to emerge. The objective is to incorporate these new attacks without losing detection capability on previously observed real and fake speech. Dataset boundaries may obscure this objective because different datasets can share generation mech- anisms and one dataset can contain several. We therefore propose the real-anchored mechanism- incremental (RAMI) protocol, which organizes fake samples by generation mechanism against a recurring mixed-domain real-speech background (Figure 1). This design aligns task construction with the need to learn new deepfake mechanisms while reusing available real-speech knowledge, 2 Preprint consistent with prior work on shared real characteristics and diverse fake representations (Zhang et al., 2021; Xie et al., 2024a; Huang et al., 2025). RAMI and four comparison protocols use iden- tical training, development, and evaluation pools to investigate how these task-organization choices affect continual-learning performance. Asymmetric prompt adaptation. This updating scenario calls for preserving reusable real-speech knowledge while accommodating increasingly diverse deepfake mechanisms. Updating a shared representation alone risks overwriting earlier knowledge, whereas learning each new task inde- pendently limits transfer between related deepfake methods. Prompt-based continual learning of- fers a parameter-efficient way to allocate shared and task-specific adaptation capacity (Wang et al., 2022b;a; Hong et al., 2025). We therefore propose RF-Prompt (real–fake prompt learning), which protects a shared real prompt and expands task-specific fake experts. Each new expert inherits his- torical knowledge and learns a complementary residual, allowing adaptation without overwriting stored experts. Cosine anchoring stabilizes the shared real prompt, while residual orthogonality en- courages distinct adaptation directions. Input-adaptive soft fusion draws on the accumulated experts while keeping the injected prompt length fixed, so additional stored knowledge does not require a longer prompt sequence. Our contributions are threefold: • RAMI protocol. We propose RAMI to model incrementally emerging deepfake mecha- nisms against a recurring mixed-domain real-speech background. Together with four other protocols, our evaluation covers five representative task organizations for continual ADD over identical training, development, and evaluation pools. • Asymmetric continual prompt learning. RF-Prompt combines cosine-anchored shared real parameters, inherited fake experts with orthogonal residual regularization, and input- adaptive fixed-length fusion. It retains and expands knowledge without storing historical training audio or adding a feature-distillation forward pass. • Multi-axis empirical analysis. We compare continual-learning and adapted prompt base- lines, task organizations, component ablations, limited-data settings, and speech back- bones. RF-Prompt obtains the lowest final average and pooled EER among the evaluated continual methods on RAMI: 10.110% and 10.370%. The five-protocol comparison ex- amines how real-speech arrival and fake-task organization affect RF-Prompt under a fixed global sample budget. 2RELATED WORK ADD datasets and generalization. ASVspoof 2019, ASVspoof 5, ADD, CodecFake, and AT-ADD broaden coverage of deepfake methods and recording conditions (Todisco et al., 2019; Wang et al., 2025; Yi et al., 2022; Xie et al., 2024b; 2026). ASDG and W2V-ASDG aggregate real represen- tations across domains while separating fake representations (Xie et al., 2024a; 2023); one-class and prototype-based approaches similarly exploit compact real structure and diverse fake patterns (Zhang et al., 2021; Kim et al., 2024; Huang et al., 2025). These findings motivate asymmetric modeling, while RAMI examines how source-domain arrival and mechanism organization affect continual adaptation. Continual ADD. DFWF combines knowledge distillation with real-embedding alignment (Ma et al., 2021). RAWM, RWM, and RegO develop adaptive update constraints and real–fake-aware knowl- edge protection (Zhang et al., 2023; 2024c; Chen et al., 2025). Oiso et al. use prompt tuning for target-domain adaptation (Oiso et al., 2024). We examine successive adaptation and retention under controlled task organizations, alongside these established approaches. Continual learning and prompt adaptation. Classical methods such as EWC and OWM protect historical knowledge through regularization and constrained updates (Kirkpatrick et al., 2017; Zeng et al., 2019). Prompt-based methods instead adapt lightweight tokens: L2P retrieves prompts, Du- alPrompt combines shared and task-specific prompts, and CODA-Prompt composes input-weighted components (Wang et al., 2022b;a; Smith et al., 2023). RainbowPrompt, KA-Prompt, SinglePrompt, and SMoPE further investigate diversity, alignment, sharing, and sparse expert selection (Hong et al., 2025; Xu et al., 2025; Park et al., 2026; Le et al., 2026). RF-Prompt assigns shared and expanding capacity to real and fake knowledge, respectively. Appendix F provides the detailed review. 3 Preprint Five Continual-Learning Protocols from One Locked Sample Pool Locked train / dev / eval pool ASV2019ASV5CodecFakeAT-ADD Real arrival strategy Dataset-wise RealSource-support-matched RealFour-domain mixed Real Fake organization Dataset-wise Fake Mechanism-wise Fake Protocol 1 T1 ASV19 ASV19 T2 ASV5 ASV5 T3 Codec Codec T4 AT-ADD AT-ADD Real Fake Protocol 4 T1 ASV19 T2 ASV5 T3 Codec T4 AT-ADD Real Fake Protocol 2 T1 ASV19 M1 T2 ASV5 M2 T3 Codec M3 T4 AT-ADD M4 Real Fake Protocol 3 T1 M1 T2 M2 T3 M3 T4 M4 Real Fake RAMI (Protocol 5) Primary setting T1 M1 T2 M2 T3 M3 T4 M4 Real Fake Real source domains:ASV2019ASV5CodecFakeAT-ADD Task Setup Fake Task Figure 2: Five task organizations from identical train, development, and evaluation pools. Columns vary real arrival, while rows vary fake organization. RAMI (Protocol 5) is the proposed protocol; recurring real domains use disjoint utterances across tasks. 3RAMI: A REAL-ANCHORED MECHANISM-INCREMENTAL PROTOCOL To reflect detector updates involving new deepfake mechanisms and available real speech from known domains, RAMI combines mechanism-incremental fake tasks with a recurring mixed-domain real background. Four comparison protocols over identical sample pools assess how task organiza- tion affects continual learning. 3.1LEARNING SETTING AND LOCKED SAMPLE POOLS We consider a sequence of datasetsD 1 ,...,D T , where each sample contains an utterance x and a binary label y ∈real, fake. When learning Task t, training accesses the current training partition but no historical training audio. After learning each task, we evaluate the detector on all task eval- uation partitions. Task boundaries are available during training; the task identity is unavailable at inference. The benchmark draws from ASVspoof 2019 LA, ASVspoof 5 Track 1, CodecFake, and the clean AT- ADD Track 2 Speech subset. We first lock training, development, and evaluation membership, then change only task assignment. The training and development pools each contain 9,600 real and 9,600 fake utterances; the evaluation pool contains 20,000 of each class. All five protocols therefore share 19,200 training, 19,200 development, and 40,000 evaluation samples. Dataset-qualified identities are retained to avoid merging unrelated generator identifiers with the same spelling. 3.2GENERATION MECHANISMS We organize fake speech into four groups according to the generation pathway. M1, Classical Pipeline, covers classical signal processing, traditional parametric vocoding, and speech-unit con- catenation. M2, Neural Acoustic Pipeline, covers neural acoustic modeling or vocoding that produces continuous acoustic features, latent representations, or waveforms. M3, Neural Codec Pipeline, encodes and reconstructs existing speech through a neural codec. M4, Speech-LM Pipeline, generates a target speech sequence using a speech language model, followed by acous- tic realization. M3 reconstructs existing speech, whereas M4 generates a new speech-token sequence. A down- stream neural vocoder does not change either assignment to M2. Source composition and generator assignments appear in Appendix A. 4 Preprint SSL Feature Transformer Encoder Layer퐿 1 Transformer Encoder Layer 퐿 z Transformer Encoder Layer 퐿 24 AASIST Tuned Frozen Real PromptFake PromptAudio Tokens ++ ++ ++ New expert + + + Cosine Sim + + + MOE MOE MOE MOE Orthogonal Residual Cosine Sim Cosine Sim RF-PromptContinual Learning for ADD Real Prompt Consistency Learning Fake Prompt Orthogonal Learning FPOL at Task 4 Historical Experts Selection Task 4 Fake audio tokens Signatures cosine + argmax Residual Separation Historical residuals Δ₁ Δ₂ Δ₃ New residual Δ₄ Δ₄ ⟂span(Δ₁, Δ₂, Δ₃) New Expert Construction Trainable Δ₄ New expert 푃 푓4 + = Historical experts Selected expert푝 푓푘 Soft MoE Fusion 푃 푓1 푃 푓2 푃 푓3 푃 푓4 Soft MOE 푎 1 (푥) Fake Prompt Orthogonal Residual Selected expert푃 푓푘 푎 2 (푥) 푎 3 (푥) 푎 4 (푥) Historical expert New expert Historical expertsHistorical experts New expert Orthogonal Residual Orthogonal Residual Figure 3: Overview of our proposed RF-Prompt. Left: the overall audio deepfake detection pipeline. Middle: continual learning of the shared real prompt and task-specific fake experts. Right: fake expert selection, residual learning, expert construction, and soft fusion at Task 4 as an example. 3.3FIVE TASK ORGANIZATIONS Figure 2 crosses real arrival with fake organization. Protocol 1 groups both classes by dataset; Pro- tocol 2 retains dataset-wise real arrival and groups fake speech by mechanism. Protocol 3 matches real-source support to each mechanism task. Protocols 4 and 5 supply a recurring four-domain real mixture, with dataset-wise and mechanism-wise fake tasks, respectively. RAMI is Protocol 5. Its training tasks each contain 600 fresh real utterances from each domain and 2,400 fake utterances from one mechanism. These real utterances are disjoint across tasks; full split counts are in Ap- pendix A. 4RF-PROMPT: ASYMMETRIC CONTINUAL PROMPT LEARNING To preserve shared real-speech knowledge while learning new deepfake patterns, RF-Prompt pro- tects a shared real prompt and expands fake experts through historical knowledge inheritance and complementary residual learning. Input-adaptive fusion combines these experts into a fixed-length prompt sequence. 4.1RF-PROMPT FRAMEWORK Figure 3 shows the framework. The frozen speech backbone has L transformer layers of width d. At task t, layer l maintains a shared real prompt R l t ∈ R m×d and complete fake expertsP l f,k t k=1 of the same shape. Defaults are L = 24, d = 1024, and m = 5. For audio tokens H l−1 t (x), the prompted transformation is H l t (x) = Audio E l [R l t ; e P l f,t (x); H l−1 t (x)] ,(1) where e P l f,t (x) is the fused fake prompt and Audio retains only the audio-token outputs. Prompts are introduced anew at each layer rather than accumulating in sequence length. A trainable AASIST backend maps the final audio representation to binary logits. The shared real prompt and backend remain trainable; historical fake experts are frozen. The real/fake names indicate the intended allocation of shared and expanding capacity, rather than class- exclusive supervision: both classes contribute to cross-entropy, and both prompt types participate in every prediction. The real tokens are not part of the fake mixture. 5 Preprint 4.2CONSISTENCY LEARNING FOR THE SHARED REAL PROMPT Sharing real parameters allows knowledge reuse but also exposes earlier real-speech knowledge to later updates. At the beginning of Task t > 1, we retain the previous selected real prompt ̄ R t−1 as a detached reference. We anchor corresponding token directions across the first K layers: L real = 1 Km K X l=1 m X j=1 1− cos R l t,j , sg( ̄ R l t−1,j ) .(2) Here sg denotes stop-gradient. We use K = L by default and set the loss to zero for Task 1. This parameter-level constraint protects shared knowledge without historical audio or a teacher-feature forward pass. 4.3ORTHOGONAL LEARNING FOR FAKE PROMPTS Selecting transferable knowledge. Let q(x)∈ R d be the normalized temporal mean of frozen SSL front-end representations, and let s k be the normalized mean of expert k’s prompt tokens across layers. At task initialization, we average queries from current-task fake speech to obtain ̄ q t and select k ⋆ = arg max k<t cos( ̄ q t , s k ),B t = sg(P f,k ⋆ ). This task-level inheritance occurs once; input-adaptive fusion below instead routes each utterance over all available experts. The new expert is initialized with a small perturbation projected away from the historical complete-Prompt space; Appendix B.4 provides the construction. Separating residual knowledge. During training, the complete current expert is optimized while its base stays fixed. Its residual is ∆ l t = P l f,t − B l t .(3) Let Q l ∆,t and Q l ∆,<t be row-orthonormal factors of the current residual and concatenated historical residuals, obtained by reduced QR of their transposes. If their row counts are r t and r h , we penalize their normalized overlap: L orth = 1 L L X l=1 ∥Q l ∆,t (Q l ∆,<t ) ⊤ ∥ 2 F r t r h .(4) Historical factors are detached, and the loss is zero for Task 1. Regularizing residuals preserves the inherited base while encouraging complementary new directions. Orthogonality is a soft training objective. 4.4INPUT-ADAPTIVE FUSION AND OPTIMIZATION For any input utterance, we compute weights over all available complete fake experts: α k (x) = exp(cos(q(x), s k )/τ ) P t r=1 exp(cos(q(x), s r )/τ ) , e P l f,t (x) = t X k=1 α k (x)P l f,k ,(5) We use τ = 0.1 and share routing weights across layers. Fusion combines complete experts and requires no inference-time task identity. The task objective combines binary cross-entropy with the two retention terms: L t = E (x,y)∼D t L cls (x,y) + λ real L real + λ orth L orth ,(6) 6 Preprint M1M2M3M4 After Task 1 After Task 2 After Task 3 After Task 4 4.66 ------ 8.009.38 ---- 12.8412.229.32 -- 14.509.1218.7810.12 Sequential M1M2M3M4 6.22 ------ 5.168.24 ---- 8.469.244.84 -- 10.408.4013.128.52 RF-Prompt 5 10 15 20 EER (%) Figure 4: Acquired-task EER trajectories of Sequential and RF-Prompt on RAMI (Protocol 5). Rows correspond to the checkpoint after each task and columns to acquired deepfake mechanisms; lower EER is better. Purple boxes mark the final checkpoint. Complete trajectories for all continual methods appear in Appendix C. We use λ real = 1 and λ orth = 0.1. Five fused fake tokens and five shared real tokens keep the injected length fixed at ten per layer. The expert bank grows with tasks. Appendix B gives the checkpoint-selection and training procedure. 5EXPERIMENTS We evaluate RF-Prompt against continual-learning and adapted prompt baselines, and examine the effect of task organization. Ablations, limited-data experiments, and alternative speech backbones further assess the contributions and applicability of the method. 5.1EXPERIMENTAL SETUP We use the frozen XLS-R 300M model 2 with a trainable AASIST backend, 50 epochs per task, batch size 32, and Adam with a cosine learning-rate schedule. Each task’s own development set selects its checkpoint by EER. Table 1 compares matched continual-learning and adapted prompt baselines; offline joint co-training is a separate reference. Experiments use seed 2026; complete settings appear in Appendix B. Let e t,j be the EER on task j after learning task t. We report final average EER, T −1 P j e T,j , and pooled EER obtained from all evaluation scores with a single threshold sweep. Average forgetting is AF = 1 T − 1 T−1 X j=1 e T,j −min t∈j,...,T e t,j .(7) For cross-protocol comparisons, pooled EER uses the same 40,000 utterances. We additionally regroup scores into the same four RAMI (Protocol 5) evaluation groups to compute a common average. Native averages from different task partitions are not interchangeable. All results below are single-seed observations; no statistical significance is implied. 5.2MAIN COMPARISON AND CONTINUAL RETENTION Our method achieves the lowest average and pooled EER among the continual methods, improving over the strongest prompt-based competitor, SinglePrompt, by 1.555% and 1.735%, respectively. It also improves average and pooled EER over EWC by 2.145% and 2.435% and reduces AF from 5.827% to 4.560%. Compared with the strongest ADD-specific competitor RAWM, average EER decreases from 11.295% to 10.110%, pooled EER from 11.730% to 10.370%, and AF from 5.753% to 4.560%. Oiso Prompt has lower AF but fails to acquire later mechanisms, yielding 25.92% and 30.10% EER on M3 and M4; its low forgetting therefore does not indicate stronger overall continual 2 https://huggingface.co/facebook/wav2vec2-xls-r-300m 7 Preprint Table 1: Final results on RAMI (Protocol 5) with XLS-R 300M. EER and AF are reported in %; lower is better. Category and venue identify each method’s original scope and publication. All scores are from our matched experiments;† marks methods adapted to continual ADD. Offline co- training is excluded when identifying the best continual method. MethodCategoryVenueM1M2M3M4 Avg EER Pool EERAF SequentialTraditional CL–14.509.12 18.78 10.1213.13013.445 6.433 EWC (Kirkpatrick et al., 2017)General CLPNAS’1713.408.30 17.889.4412.25512.805 5.827 OWM (Zeng et al., 2019)General CLNat. Mach. Intell.’19 13.808.68 17.649.9012.50513.080 5.993 RAWM (Zhang et al., 2023)ADD-specific CLICML’2310.228.02 14.96 11.9811.29511.730 5.753 RWM (Zhang et al., 2024c)ADD-specific CLAAAI’2413.408.32 17.469.8812.26512.845 5.700 RegO (Chen et al., 2025)ADD-specific CLAAAI’2514.468.22 17.729.6412.51013.000 6.060 Oiso Prompt † (Oiso et al., 2024)ADD domain adaptation Interspeech’247.34 13.46 25.92 30.1019.20520.350 1.067 SinglePrompt † (Park et al., 2026) Prompt-based CLCVPR Findings’2612.827.52 17.448.8811.66512.105 5.593 KA-Prompt † (Xu et al., 2025)Prompt-based CLICML’2512.007.80 18.669.3011.94012.415 5.440 SMoPE † (Le et al., 2026)Prompt-based CLICLR’2613.388.34 20.40 10.8013.23013.615 6.533 RF-Prompt (ours)ADD prompt-based CLOurs10.408.40 13.128.52 10.110 10.370 4.560 Joint (offline)Offline reference–7.207.56 11.52 10.609.2209.420— learning. The improvements are not uniform across tasks: several baselines achieve lower M2 EER. Joint co-training remains better in aggregate, with a pooled gap of 0.950%. Figure 4 compares retention with Sequential. RF-Prompt finishes with 10.40% EER on M1 and 13.12% on M3, versus Sequential’s 14.50% and 18.78%. Its M2 EER recovers from 9.24% after M3 to 8.40% after M4, close to 8.24% immediately after acquisition. Average forgetting decreases from Sequential’s 6.433% to 4.560%. Complete trajectories for all baselines appear in Appendix C. 5.3EFFECT OF TASK ORGANIZATION Table 2: Controlled protocol comparison for our RF-Prompt model. The common average uses identical RAMI (Protocol 5) evaluation groups. Protocolreal arrivalfake organizationCommon Avg EERPool EER 1Dataset-wiseDataset-wise16.40016.430 2Dataset-wiseMechanism-wise15.67515.900 3Source-support-matchedMechanism-wise13.23013.110 4Four-domain mixtureDataset-wise11.03511.330 5 (RAMI)Four-domain mixtureMechanism-wise10.11010.370 The five protocols cover two continual-learning regimes. Protocols 1–3 continually introduce both real and fake distributions, whereas Protocols 4 and 5 expose all four real source domains from the beginning and focus on the continual acquisition of emerging deepfake mechanisms. Accordingly, Protocols 1–3 evaluate joint real–fake continual learning, while Protocols 4 and 5 evaluate deepfake- incremental learning against a known mixed-domain real background. For RF-Prompt, changing from dataset-wise fake tasks in Protocol 1 to mechanism-wise tasks in Pro- tocol 2, with the same real-arrival strategy, lowers common-average EER from 16.400% to 15.675% and pooled EER from 16.430% to 15.900%. With mechanism-wise fake tasks retained, source- support-matched real arrival in Protocol 3 lowers these metrics further to 13.230% and 13.110%. Under the recurring mixed-domain real background, RAMI attains 10.110% and 10.370%, com- pared with 11.035% and 11.330% for Protocol 4. Thus, RAMI gives RF-Prompt its lowest aggregate errors among the five tested organizations. These comparisons characterize task-organization effects under a fixed global sample pool; per-task sizes and class proportions also change with the grouping. RAMI represents the practical setting of available real domains and arriving deepfake mechanisms. 5.4COMPONENT ABLATIONS In the uniform-mean control, we replace the input-adaptive weights in Eq. 5 with α k (x) = 1/t for all available complete fake experts. In the initialization control, we retain historical-expert selection and the residual perturbation scale, but skip removing the perturbation’s projection onto the historical complete-Prompt space. 8 Preprint Table 3: Ablations on RAMI. Each control retains the other selected settings; the initialization control removes only historical-space projection. ConfigurationAvg EERPool EERAF RF-Prompt (full)10.11010.370 4.560 w/o real cosine-anchoring loss10.75511.3655.293 w/o residual-orthogonality loss12.25512.8057.320 w/o adaptive fusion (uniform mean)11.52512.4805.480 w/o orthogonal initialization projection11.08511.4305.867 4812162024 Applied to first K layers 10.5 11.0 11.5 12.0 12.5 Pooled EER (%) real promptfake prompt 1005001,0002,400 Number of unique fake clips per task 10 12 14 16 18 Pooled EER (%) SequentialEWCRWMRegORF-Prompt Figure 5: Protection-depth sweeps (left) and limited-fake-data adaptation (right). The other con- straint remains active across 24 layers; the right axis counts unique fake clips per task. Removing any component increases both average and pooled EER. The largest pooled degradation, 2.435%, occurs when the residual orthogonal loss is removed while orthogonal initialization is re- tained. Initialization alone is therefore insufficient in this comparison. Replacing adaptive fusion with a uniform mean increases pooled EER by 2.110%, supporting input-dependent weighting. Re- moving real anchoring and initialization projection increases pooled EER by 0.995% and 1.060%, respectively. These are controlled single-seed ablations rather than estimates of independent additive component effects. Appendix D compares alternative real prompt protection and fake expert constructions. 5.5SENSITIVITY AND TRANSFER Protection depth. Figure 5 (left) varies the first K constrained layers under the selected loss weights, keeping the other constraint active across all 24 layers. Both sweeps are non-monotonic and attain their lowest pooled EER of 10.370% at K = 24, consistent with the all-layer configuration. Limited new fake training data. Figure 5 (right) restricts fake training speech to 100, 500, or 1,000 unique utterances per task, retaining 2,400 real utterances. Resampling preserves class balance and optimization steps; development and evaluation sets remain fixed, including 2,400 fake development utterances per task. RF-Prompt outperforms the four compared baselines at every budget. With 100 fake training utterances, pooled EER is 12.375%, versus 17.645% for the strongest compared baseline. Appendix H provides details. Transfer across backbones. RF-Prompt improves average EER, pooled EER, and AF over Se- quential on WavLM-Large, W2V-BERT 2.0, XLS-R 1B, and XLS-R 2B, attaining pooled EERs of 12.920%, 10.040%, 9.570%, and 9.550%, respectively. Appendix E reports the complete compar- isons. 6CONCLUSION We introduced RAMI, a real-anchored mechanism-incremental protocol, together with four con- trolled alternatives over the same sample pools. RF-Prompt achieves its lowest pooled and common- average EER under RAMI among these five organizations. Cosine-protected shared prompts, in- herited residual experts, and fixed-length adaptive fusion improve performance under RAMI and in matched low-resource and cross-backbone comparisons. Changing real domains, long task se- quences, and unseen generators remain open challenges. 9 Preprint AI USE STATEMENT Generative AI tools were used to assist with manuscript drafting and language polishing. All AI- assisted text was reviewed by the authors, and the experimental results, analyses, and scientific claims were verified by the authors. The authors take full responsibility for the final content of this work. REFERENCES Philip Anastassiou et al. Seed-TTS: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024. Yujie Chen et al. Region-based optimization in continual learning for audio deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 23651–23659, 2025. Trung-Anh Dang, Duy-Cuong Bui, Ngoc-Son Vu, Christel Vrain, and Vincent Nguyen. GAP-Prompt: Gated adaptive prompting for efficient continual learning.arXiv preprint arXiv:2608.23782, 2026. Google. Introducing Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking. https:// blog.google/innovation-and-ai/models-and-research/gemini-models/ gemini-3-8-live-gemini-3-8-live-extended-thinking/, September 2026. Kiseong Hong, Gyeong-hyeon Kim, and Eunwoo Kim.RainbowPrompt: Diversity-enhanced prompt-evolving for continual learning. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, p. 1130–1140, 2025. Hangrui Hu et al. Qwen3-TTS technical report. arXiv preprint arXiv:2601.15621, 2026. Wen Huang, Yanmei Gu, Zhiming Wang, Huijia Zhu, and Yanmin Qian. Generalizable audio deep- fake detection via latent space refinement and augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, p. 1–5, 2025. doi: 10.1109/ICASSP49660.2025. 10888328. Hyun Myung Kim, Kangwook Jang, and Hoirin Kim. One-class learning with adaptive centroid shift for audio deepfake detection. In Interspeech, p. 4853–4857, 2024. doi: 10.21437/Interspeech. 2024-177. James Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114. Minh Le, Bao-Ngoc Dao, Huy Nguyen, Quyen Tran, Anh Nguyen, and Nhat Ho. One-prompt strikes back: Sparse mixture of experts for prompt-based continual learning. In International Conference on Learning Representations, 2026. Xiang Li, Pin-Yu Chen, and Wenqi Wei. Scaling behavior in model fine-tuning for audio deepfake detection. In Proceedings of the 43rd International Conference on Machine Learning, 2026. Haoxin Ma, Jiangyan Yi, Jianhua Tao, Ye Bai, Zhengkun Tian, and Chenglong Wang. Continual learning for fake audio detection. In Interspeech 2021, p. 886–890, 2021. doi: 10.21437/ Interspeech.2021-794. Hideyuki Oiso, Yuto Matsunaga, Kazuya Kakizaki, and Taiki Miyagawa. Prompt tuning for audio deepfake detection: Computationally efficient test-time domain adaptation with limited target dataset. In Interspeech 2024, p. 2710–2714, 2024. doi: 10.21437/Interspeech.2024-81. OpenAI.IntroducingGPT-Live. https://openai.com/index/ introducing-gpt-live/, July 2026. Zihan Pan, Tianchi Liu, Hardik B. Sailor, and Qiongqiong Wang. Attentive merging of hidden embeddings from pre-trained speech model for anti-spoofing detection. In Interspeech, p. 2090– 2094, 2024. doi: 10.21437/Interspeech.2024-1472. 10 Preprint Seoyoung Park, Haemin Lee, and Hankook Lee. Is prompt selection necessary for task-free online continual learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings Track, 2026. James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11909–11919, 2023. Massimiliano Todisco et al. ASVspoof 2019: Future horizons in spoofed and fake audio detection. arXiv preprint arXiv:1904.05441, 2019. Xin Wang et al. ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech. Computer Speech & Language, 64:101114, 2020. doi: 10.1016/j.csl.2020. 101114. Xin Wang et al. ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech. arXiv preprint arXiv:2502.08857, 2025. Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. DualPrompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, 2022a. Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 139–149, 2022b. Yuankun Xie, Haonan Cheng, Yutian Wang, and Long Ye. Learning a self-supervised domain- invariant feature representation for generalized audio deepfake detection. In Interspeech, p. 2808–2812, 2023. doi: 10.21437/Interspeech.2023-1383. Yuankun Xie, Haonan Cheng, Yutian Wang, and Long Ye. Domain generalization via aggregation and separation for audio deepfake detection. IEEE Transactions on Information Forensics and Security, 19:344–358, 2024a. doi: 10.1109/TIFS.2023.3324724. Yuankun Xie et al. The Codecfake dataset and countermeasures for the universally detection of deepfake audio. arXiv preprint arXiv:2405.04880, 2024b. Yuankun Xie et al. AT-ADD: A benchmark and challenge for robust and all-type audio deepfake detection. arXiv preprint arXiv:2608.23437, 2026. Kunlun Xu, Xu Zou, Gang Hua, and Jiahuan Zhou. Componential prompt-knowledge alignment for domain incremental learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 70032–70046, 2025. Jiangyan Yi, Ruibo Fu, Jianhua Tao, et al. ADD 2022: The first audio deep synthesis detection challenge. In IEEE International Conference on Acoustics, Speech and Signal Processing, p. 9216–9220, 2022. Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1:364–372, 2019. doi: 10.1038/ s42256-019-0080-x. Qishan Zhang, Shuangbing Wen, and Tao Hu. Audio deepfake detection with self-supervised XLS- R and SLS classifier. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 6765–6773, 2024a. doi: 10.1145/3664647.3681345. Xiaohui Zhang, Jiangyan Yi, Jianhua Tao, Chenglong Wang, and Chu Yuan Zhang. Do you re- member? overcoming catastrophic forgetting for fake audio detection. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p. 41819–41831. PMLR, 2023. 11 Preprint Xiaohui Zhang, Jiangyan Yi, and Jianhua Tao. Towards robust audio deepfake detection: A evolving benchmark for continual learning. arXiv preprint arXiv:2405.08596, 2024b. Xiaohui Zhang, Jiangyan Yi, Chenglong Wang, Chu Yuan Zhang, Siding Zeng, and Jianhua Tao. What to remember: Self-adaptive continual learning for audio deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 19569–19577, 2024c. You Zhang, Fei Jiang, and Zhiyao Duan. One-class learning towards synthetic voice spoofing de- tection. IEEE Signal Processing Letters, 28:937–941, 2021. 12 Preprint ABENCHMARK CONSTRUCTION AND DATA COMPOSITION This appendix documents how one fixed sample pool is reorganized into the five continual-learning protocols. We distinguish three levels throughout the construction: the source dataset identifies the corpus from which an utterance is drawn, the generation mechanism assigns a fake utterance to M1–M4, and the continual task determines when that utterance becomes available for learning. A.1LOCKED SAMPLE POOLS The benchmark draws speech from ASVspoof 2019 LA, ASVspoof 5 Track 1, CodecFake, and AT- ADD Track 2 Speech. We use the speech-detection setting and exclude the music, singing, and environmental-sound subsets of AT-ADD. Sample membership is fixed independently for training, development, and evaluation before any continual task is constructed. All five protocols therefore contain the same utterances, labels, audio paths, and split membership; they differ only in how these samples are assigned to Tasks 1–4. Each split is globally class-balanced. Training and development each contain 9,600 real and 9,600 fake utterances, while evaluation contains 20,000 real and 20,000 fake utterances. Within the real pool, each of the four source domains contributes 2,400 training utterances, 2,400 development utter- ances, and 5,000 evaluation utterances. The training fake pool contains 2,500 ASVspoof 2019, 800 ASVspoof 5, 2,400 CodecFake, and 3,900 AT-ADD utterances. This locked construction enables task organizations to be compared without changing the global data available to the learner. A.2FIVE PROTOCOL ORGANIZATIONS The five protocols combine three real-arrival strategies with two fake-task organizations. Dataset- wise arrival introduces samples according to their source dataset; mechanism-wise arrival groups fake samples by M1–M4; mixed-domain real arrival repeatedly draws from all four known real- source domains. Protocol 1 organizes both real and fake speech by source dataset. Protocol 2 retains dataset-wise real arrival but organizes fake speech by generation mechanism. Protocol 3 uses mechanism-wise fake tasks and matches the real-source composition of each task to the source support of its fake data: within each split and source domain, its fixed real budget is allocated across tasks in proportion to that domain’s fake counts, with rounding remainders assigned by largest fractional share. Protocol 4 combines an equal four-domain real mixture with dataset-wise fake tasks. RAMI (Protocol 5) combines the same recurring real mixture with mechanism-wise fake tasks. Dataset-wise tasks follow ASVspoof 2019, ASVspoof 5, CodecFake, and AT-ADD, whereas mechanism-wise tasks follow M1–M4. Table 4 reports the resulting number of utterances in every task and split. Protocols 1 and 4 use 2,400 real utterances per training/development task and 5,000 per evaluation task; their total task sizes vary with the dataset-wise fake allocation. Protocols 2 and 5 contain 2,400 real and 2,400 fake utterances per training/development task, and 5,000 of each class per evaluation task. Protocol 3 fixes the same per-task fake budget as Protocols 2 and 5, while its real count varies according to source-support matching. Table 4: Total utterances per task. Each cell lists Train/Dev/Eval counts. ProtocolTask 1Task 2Task 3Task 4 14,900/4,505/9,8323,200/3,535/8,5254,800/4,800/10,0006,300/6,360/11,643 24,800/4,800/10,0004,800/4,800/10,0004,800/4,800/10,0004,800/4,800/10,000 34,704/5,526/10,6005,819/5,019/10,3804,800/4,800/10,0003,877/3,855/9,020 44,900/4,505/9,8323,200/3,535/8,5254,800/4,800/10,0006,300/6,360/11,643 5 (RAMI)4,800/4,800/10,0004,800/4,800/10,0004,800/4,800/10,0004,800/4,800/10,000 Protocols 4 and 5 represent continual adaptation with an existing multi-domain real-speech pool and newly arriving deepfake data. Each task receives a disjoint real subset containing 600 utterances from each source domain in training and development, and 1,250 from each domain in evaluation. RAMI uses this recurring real background to support continual acquisition of mechanism-organized fake knowledge. 13 Preprint A.3GENERATION MECHANISMS AND RAMI COMPOSITION We assign each fake generator to M1–M4 according to its dominant generation pathway, following Section 3.2. Speech-LM generation of a new speech-token sequence belongs to M4, whereas neural- codec reconstruction of existing speech belongs to M3; a downstream decoder or vocoder does not change these assignments. For hybrid classical/neural acoustic systems at the M1/M2 boundary, the waveform-generating or waveform-enhancing stage determines the assignment. For example, ASVspoof 2019 A07 uses WORLD synthesis followed by a WaveCycleGAN2 neural post-filter and is assigned to M2 (Wang et al., 2020). Table 6 reports the complete assignments. Table 5 gives the source composition of the four mechanism-defined fake tasks under RAMI. Every mechanism contributes 2,400 fake utterances to training and development and 5,000 to evaluation, while its source-dataset composition follows the available samples in the locked pool. Table 5: Source-dataset composition of the four mechanism-defined fake tasks under RAMI. Mechanism TrainDevEval M1ASV2019: 2,400ASV2019:2,000; ASV5: 400 ASV2019: 3,890; ASV5: 1,110 M2ASV2019: 100; ASV5: 800; AT-ADD: 1,500 ASV2019: 105; ASV5: 735; AT-ADD: 1,560 ASV2019: 942; ASV5: 2,030; AT-ADD: 2,028 M3CodecFake: 2,400CodecFake: 2,400CodecFake: 5,000 M4AT-ADD: 2,400AT-ADD: 2,400ASV5: 385; AT-ADD: 4,615 The numbers of represented generators in training/development/evaluation are 5/6/9 for M1, 24/23/32 for M2, 6/6/7 for M3, and 5/5/13 for M4. Generator identifiers are indexed jointly by source dataset and code, so Table 6 lists the complete assignment under both fields. In the locked training pool, M1, M3, and M4 each draw from one source dataset, while M2 spans three; mech- anism and source are therefore not fully crossed. M1 development and evaluation also include ASVspoof 5, and M4 evaluation includes ASVspoof 5, testing cross-source transfer within those mechanism groups. M1 development EER thus includes this source shift during checkpoint selec- tion. Table 6: Generation-mechanism assignment by source dataset. SourceMechanism Generator IDs in assignment catalog ASV2019M1A02, A03, A04, A05, A06, A11, A13, A14, A16, A17, A18, A19 ASV2019M2A01, A07, A08, A09, A10, A12, A15 ASV5M1A12, A19, A20 ASV5M2A01, A02, A03, A04, A05, A06, A07, A08, A09, A10, A11, A13, A14, A15, A16, A17, A18, A21, A22, A23, A24, A25, A26, A27, A28, A30, A31, A32 ASV5M4A29 AT-ADDM2BigVGAN, DiffGANTTS, DiffSpeech fastdiff, E2TTS, F5TTS, FastDiff, FastPitchfastdiff, FastSpeech2fastdiff, GlowTTS, GradTTS, HiFiGAN, Kokoro, MBMelGAN, MelGAN, MeloTTS, OpenVoice2, ParallelWaveGAN, PortaSpeechnormalfastdiff, SeedVC1, Star- GANv2VC, StyleMelGAN, StyleSpeech, StyleTTS2, Tacotron2, VITS, WaveGlow, WaveNet, prodiffteacherfastdiff AT-ADDM4ChatTTS, CosyVoice, CosyVoice2, CosyVoice3, FireRedTTS2, Fish, GPTSoVITS, IndexTTS1, IndexTTS1.5, IndexTTS2, Llasa1B, Llasa3B, Llasa8B, ParlerTTSmini, SparkTTS, StepAu- dioTTS, Tortoise CodecFake M3F01, F02, F03, F04, F05, F06, F07 BIMPLEMENTATION AND OPTIMIZATION DETAILS B.1TRAINING CONFIGURATION RF-Prompt freezes the SSL backbone and trains the AASIST backend, shared real parameters, and the current fake expert. Previous complete fake experts and inherited bases are detached. Both classes contribute to binary cross-entropy. The real consistency term operates directly on parameters and is sample-independent, despite its real-retention motivation. No historical audio is retained, and no teacher-feature forward pass is used by the selected prompt-cosine configuration. 14 Preprint Table 7: Selected XLS-R 300M configuration. SettingValue Sample rate / input length16 kHz / 64,600 samples Tasks / epochs per task4 / 50 Batch size / random seed32 / 2026 OptimizerAdam Adam betas / epsilon(0.9, 0.999) / 1e-8 Weight decay5e-4 Initial / minimum learning rate1e-4 / 1e-6 ScheduleCosine decay within each task Training / evaluation workers8 / 4 Development evaluationEvery epoch Checkpoint selectionLowest dev EER; ties: lowest dev loss real / fake tokens per layer5 / 5 Prompt dropout0.1 cosine coefficient1.0 Residual orthogonal coefficient0.1 Inherited perturbation scale0.1 Routing temperature0.1 B.2MATCHED BASELINE SETTINGS The recorded EWC configuration uses coefficient 100 and importance decay 1.0, without a Fisher- estimation batch cap. OWM uses its projection parameter α = 1.0. RegO uses importance quantile 0.75 and forgetting threshold 0.1, without an importance-estimation batch cap. The additional LwF reference uses distillation coefficient 1.0 and temperature 2.0. These values are extracted from each run’s own configuration; inactive defaults in another method’s configuration are not interpreted as active losses. All comparisons use the corresponding frozen-backbone training framework rather than reported scores from the original publications. SMoPE is adapted from its official prompt-expert implementation to XLS-R attention projections. The audio version uses 25 prefix experts per head, activates the top five in the first six encoder layers, and retains the binary AASIST classifier. Its results are from the same four-task RAMI pool and training budget as the other adapted prompt baselines, not from the original vision benchmarks. B.3EXPERT CONSTRUCTION AND TENSOR DIMENSIONS For the XLS-R 300M backbone used in our primary experiments, each complete fake expert has shape 24× 5× 1024, corresponding to 24 transformer layers, five prompt tokens per layer, and a hidden dimension of 1024. Parent selection returns one complete expert with this shape from the historical bank, rather than compressing fifteen historical tokens into five. At Task 4, the historical bank has shape 3× 24× 5× 1024; its three averaged signatures have shape 3× 1024. A current-task fake query has dimension 1024 and selects a single inherited expert. The trainable residual has the same shape as this expert. Their elementwise sum is the new complete expert. At inference, an utterance query produces four softmax weights at Task 4. For a minibatch of size b, a layer’s bank of shape 4× 5× 1024 is combined into b× 5× 1024 effective fake tokens. The five real tokens are injected separately. Routing uses all complete experts, including those not selected as the current expert’s base. Parent selection at task initialization and input-dependent soft routing are therefore different operations. B.4ORTHOGONAL INITIALIZATION AND RESIDUAL REGULARIZATION Initializing new directions. For each layer, let Q l P,<t contain orthonormal rows obtained from the historical complete Prompt bank. Starting from a random row-normalized matrix U l t , we remove its projection onto the historical space and orthonormalize the remaining rows: V l t = U l t − U l t (Q l P,<t ) ⊤ Q l P,<t , b U l t = RowQR(V l t ).(8) 15 Preprint The initialization scales each new direction relative to its inherited token: P l,init f,t,j = B l t,j + ε max(∥B l t,j ∥ 2 ,δ) b U l t,j , ε = 0.1,(9) with a small numerical floor δ. Thus the perturbation is small relative to the inherited prompt, rather than having a fixed absolute magnitude. For task 1, we initialize a standalone prompt and set its base to zero. Let RowQR(A) = qr(A ⊤ ).Q ⊤ using reduced QR. For each layer, the initialization removes a random matrix’s projection onto the historical complete-Prompt space, re-orthonormalizes the resulting rows, and scales them relative to the selected base tokens. Training instead applies the orthogonal loss to ∆ t = P f,t −B t and stored historical residuals. For Task 1, B 1 = 0, so ∆ 1 = P f,1 . The base remains fixed even if a different expert later receives the largest mixture weight. If Q h Q ⊤ h = I , then (U − UQ ⊤ h Q h )Q ⊤ h = 0 in exact arithmetic. Subsequent normalization within the projected span preserves this property when sufficient rank remains. Finite precision and QR rank deficiency can weaken the numerical interpretation; the implementation does not supply a rank-adaptive guarantee for arbitrarily long task sequences. During optimization, orthogonality is only encouraged by a soft loss, not enforced by repeated exact projection. Independent parameter directions also need not produce independent output features. The task-level parent query uses at most 512 current-task fake training examples, matching the recorded initialization sample cap. For smaller low-resource pools, this cap does not imply 512 unique examples. Expert signatures are normalized averages over layers and token slots. No separate learned keys are introduced in this response-based routing mode. B.5TASK-LEVEL TRAINING PROCEDURE Algorithm 1 summarizes RF-Prompt optimization across the task sequence. Each task adds one fake expert while updating the shared real prompt and detector backend; previously learned fake experts remain frozen, and development EER determines the checkpoint carried to the next task. Algorithm 1 Task-level training procedure of RF-Prompt Require: Task sequenceD t T t=1 ; frozen SSL backbone; real prompt P r ; fake expert bankB f Ensure: Selected detector and expanded fake expert bank after Task T 1: for t = 1,...,T do 2: if t = 1 then 3:Initialize P r and P f,1 ; set B 1 = 0 and ∆ 1 = P f,1 4: else 5:Load the best-development checkpoint from Task t− 1 6:Detach the preceding P r as its cosine reference and freezeB f 7:Average current-task fake queries and select the most similar historical expert B t 8:Initialize ∆ t by projecting a random perturbation away from the historical complete- expert subspace 9:Construct the current complete expert P f,t = B t + ∆ t 10: end if 11: for each training minibatch fromD t do 12:Compute input-dependent weights over all complete experts inB f ∪P f,t 13:Fuse them into five fake tokens per layer and inject them with five shared real tokens 14:Compute the binary classification loss 15:if t > 1 then 16:Add real prompt consistency and fake residual-orthogonality losses 17:end if 18:Update P r , ∆ t , and the detector backend 19: end for 20:Select by current-task development EER, breaking ties by development loss 21:Evaluate it on all evaluation partitions and append the frozen P f,t toB f 22: end for 16 Preprint CCOMPLETE CONTINUAL-LEARNING RESULTS C.1RETENTION MATRICES ON RAMI (PROTOCOL 5) Table 8 reports the complete acquired-task EER trajectories for the sequentially evaluated meth- ods in Table 1. Unacquired tasks are omitted because they do not measure retention. The results complement the final aggregate metrics by showing how each method changes after every update. RF-Prompt maintains the strongest final average and pooled EER: its final M3 EER is lower than ev- ery continual baseline, while its M2 EER remains close to the value obtained immediately after Task 2. Figure 4 presents the Sequential and RF-Prompt trajectories as paired heatmaps. Joint co-training is excluded because it is an offline reference rather than a sequential learner. Table 8: Acquired-task EER trajectories on RAMI (Protocol 5). The entry under “After Task t” lists EERs for M1/·/Mt in order; the last two columns report final average and pooled EER. All entries are percentages. MethodAfter Task 1 After Task 2After Task 3After Task 4Final Avg Final Pool Sequential4.668.00/9.3812.84/12.22/9.32 14.50/9.12/18.78/10.1213.13013.445 EWC4.668.28/9.6813.58/12.34/9.1413.40/8.30/17.88/9.4412.25512.805 OWM4.669.34/9.8014.08/12.50/8.8013.80/8.68/17.64/9.9012.50513.080 RAWM4.666.18/5.108.54/6.62/6.1810.22/8.02/14.96/11.9811.29511.730 RWM4.669.24/10.0813.80/12.38/9.1013.40/8.32/17.46/9.8812.26512.845 RegO4.668.82/10.1614.92/12.88/9.3414.46/8.22/17.72/9.6412.51013.000 Oiso Prompt6.507.96/13.166.66/12.98/24.04 7.34/13.46/25.92/30.1019.20520.350 SinglePrompt4.388.48/9.8412.52/11.00/9.1012.82/7.52/17.44/8.8811.66512.105 KA-Prompt4.568.28/9.7611.56/10.70/9.7812.00/7.80/18.66/9.3011.94012.415 SMoPE4.868.24/9.5811.94/11.66/9.32 13.38/8.34/20.40/10.8013.23013.615 RF-Prompt6.225.16/8.248.46/9.24/4.8410.40/8.40/13.12/8.5210.11010.370 C.2RF-PROMPT RETENTION MATRICES ON PROTOCOLS 1–4 Figure 6 reports the corresponding RF-Prompt trajectories for the four comparison protocols, fol- lowing the protocol order and task definitions in Table 2. Protocols 1 and 4 use dataset-wise fake tasks, whereas Protocols 2 and 3 use mechanism-wise fake tasks M1–M4. Each panel uses a shared color scale and reports the final native average and pooled EER beneath the matrix. T1T2T3T4 After T1 After T2 After T3 After T4 Dataset Real Dataset Fake 7.82 ------ 6.5913.46 ---- 8.2815.942.40 -- 17.2422.086.5212.74 Avg 14.644 | Pool 16.430 Protocol 1 T1T2T3T4 Dataset Real Mechanism Fake 0.18 ------ 0.8013.92 ---- 4.1026.482.56 -- 5.3230.366.244.22 Avg 11.535 | Pool 15.900 Protocol 2 T1T2T3T4 Source-matched Real Mechanism Fake 0.48 ------ 2.909.88 ---- 3.1429.302.82 -- 4.1626.025.224.30 Avg 9.925 | Pool 13.110 Protocol 3 T1T2T3T4 Mixed-domain Real Dataset Fake 7.64 ------ 6.944.48 ---- 8.666.615.94 -- 12.048.6815.188.80 Avg 11.176 | Pool 11.330 Protocol 4 0 10 20 30 EER (%) Figure 6: RF-Prompt acquired-task EER trajectories on Protocols 1–4. Rows denote the checkpoint after each task, columns denote the task-specific evaluation groups, and purple boxes mark the final checkpoint. All values are EER (%); lower is better. DDESIGN EXPLORATION FOR REAL AND FAKE PROMPTS We examine the two asymmetric design choices in RF-Prompt: how shared real knowledge is pro- tected and how new fake experts are constructed. All comparisons in this section use RAMI and the same XLS-R 300M backbone as the primary experiment. We conduct two complementary design studies. On the real side, we adapt Shared Prompt Dis- tillation (SPD) from GAP-Prompt, a prompt-based continual-learning method (Dang et al., 2026). 17 Preprint SPD stabilizes a continually updated shared prompt by maximizing the cosine similarity between intermediate representations produced with the current and previous shared prompts. We include it to examine whether this previously proposed feature-distillation mechanism transfers to continual ADD. In our audio implementation, SPD operates on real samples and aligns hidden representations at the audio-token positions of the protected transformer layers. It therefore provides a feature-level alternative to our parameter-level cosine anchor: SPD requires a reference forward pass with the previous real prompt, whereas Prompt cosine directly compares current and previous real prompt parameters. On the fake side, we use a targeted construction ablation to explain why orthogonality is imposed on residuals rather than complete prompts. The Full-Prompt variant learns a new complete fake prompt directly and regularizes the whole prompt against historical complete experts. RF-Prompt instead decomposes the new expert as P f,t = B t + ∆ t , where the selected historical expert B t transfers related deepfake knowledge and only the newly learned residual ∆ t is separated from historical residuals. Applying orthogonality to the complete expert would also push away the reusable struc- ture intentionally inherited through B t ; applying it only to ∆ t preserves this transferred base while encouraging task-specific knowledge to occupy a complementary direction. Figure 7 illustrates the two real-side alternatives and the fake-side ablation, and Table 9 reports their aggregate results. Parameter-level cosine anchoring reduces pooled EER from 11.180% to 10.370% relative to SPD, while inherited residual construction reduces it from 12.355% to 10.370% relative to Full-Prompt orthogonality. SPD cosine Current Task Previous Task Full-Prompt orth. + + Residual orth. Feature SPD Prompt cosine VS VS Real Prompt Protection Fake Prompt Construction Real prompt Audio token Fake promptFake prompt Residual Real prompt Audio token Figure 7: Comparison of real prompt protection and fake expert construction. Shared Prompt Distil- lation (SPD) aligns audio-token features produced with current and previous real prompts; Prompt cosine directly anchors their parameters. Full-Prompt orthogonality separates complete fake experts, whereas residual orthogonality separates only the newly learned residual while retaining the inher- ited base. Table 9: Comparison of real prompt protection and fake expert construction on RAMI. All entries are EER or AF (%). real protectionfake constructionAvgPoolAF NoneNone12.600 12.020 5.393 NoneInherited residual10.755 11.365 5.293 Shared Prompt Distillation (SPD) Inherited residual10.910 11.180 5.467 Prompt cosineFull-Prompt orthogonality 11.795 12.355 6.140 Prompt cosineInherited residual10.110 10.370 4.560 Figure 8 examines the learned fake-expert geometry at the final RAMI checkpoint. PCA is fit- ted jointly to row-normalized complete-expert and residual tokens from all transformer layers, with stars denoting task centroids. Because the first two components explain less than 5% of the variance, the 2D plots are illustrative; the quantitative subspace comparisons support the geometry analysis. The complete experts retain strongly overlapping subspaces because they share inherited knowl- edge, whereas the residuals occupy distinct directions. We quantify this structure using the nor- malized layer-wise overlap 1 m ∥Q l i (Q l j ) ⊤ ∥ 2 F , where Q l i is an orthonormal basis for expert i at layer l. Off-diagonal overlap is 0.817–0.951 for complete experts and 0.010–0.030 for residuals. These measurements describe parameter-space overlap under inherited residual construction; they do not directly measure functional or mechanism-specific knowledge separation. 18 Preprint −0.20.0 PC1 (2.6%) −0.3 −0.2 −0.1 0.0 0.1 0.2 PC2 (2.0%) (a) Complete Fake experts −0.50.0 PC1 (2.6%) −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 PC2 (2.0%) (b) Inherited residuals M1M2M3M4 M1 M2 M3 M4 1.000.920.880.82 0.921.000.950.89 0.880.951.000.93 0.820.890.931.00 (c) Complete experts M1M2M3M4 M1 M2 M3 M4 1.000.020.020.03 0.021.000.010.02 0.020.011.000.01 0.030.020.011.00 (d) Residuals 0.0 0.5 1.0 Overlap M1M2M3M4 Figure 8: Geometry of the learned fake experts on RAMI. Panels (a)–(b) show a joint PCA projection of complete-expert and residual tokens; translucent points are layer-wise prompt tokens and stars are mechanism centroids. Panels (c)–(d) report normalized layer-wise subspace overlap, where lower off-diagonal values indicate stronger separation. ELIMITED-DATA AND BACKBONE EXTENSIONS E.1UNIQUE-FAKE-UTTERANCE BUDGETS We restrict each task to 100, 500, or 1,000 unique fake utterances while retaining 2,400 unique real utterances. Sampling with repetition expands the chosen fake subset to 2,400 training rows per epoch. This isolates the amount of unique fake information while preserving class balance, the training-step budget, and the original development and evaluation sets. The 2,400-example setting uses the complete fake task. Table 10: Limited-data results on RAMI. All metrics are percentages; lower is better. Unique fake/taskMethodAvg EERPool EERAF 100Sequential17.32517.7506.887 EWC17.70518.0806.707 RWM17.07517.6456.067 RegO17.76518.2106.040 RF-Prompt12.59012.375 3.907 500Sequential15.22015.7157.367 EWC13.72014.0804.693 RWM14.22014.6506.460 RegO13.68013.9655.380 RF-Prompt10.1109.865 3.907 1,000Sequential13.81014.3856.993 EWC14.65515.0157.260 RWM14.17014.6407.193 RegO12.97513.4055.887 RF-Prompt11.39011.745 5.653 2,400Sequential13.13013.4456.433 EWC12.25512.8055.827 RWM12.26512.8455.700 RegO12.51013.0006.060 RF-Prompt10.11010.370 4.560 E.2PROMPT-TOKEN COUNT We vary the numbers of real and fused fake prompt tokens injected at each transformer layer while retaining the remaining RAMI training configuration. Figure 9 shows that the selected allocation of five tokens per prompt type gives the lowest average and pooled EER among the evaluated lengths. 19 Preprint The corresponding AF values for 1, 3, 5, and 10 tokens per type are 6.293%, 7.340%, 4.560%, and 6.073%, respectively. 13510 Prompt tokens per type and layer 10.0 10.5 11.0 11.5 12.0 12.5 13.0 EER (%) Selected: 5 Real + 5 Fake Average EERPooled EER Figure 9: Sensitivity to the number of real and fused fake prompt tokens per transformer layer on RAMI. The selected configuration uses five tokens for each prompt type, for ten injected tokens in total. E.3CROSS-BACKBONE CONFIGURATION AND RESULTS The primary XLS-R 300M model and WavLM-Large 3 have 24 layers and hidden width 1024. W2V- BERT 2.0 4 has approximately 600M parameters, while XLS-R 1B 5 and XLS-R 2B 6 have 48 lay- ers and widths 1280 and 1920, respectively. Prompt width and backend projection are adapted to each backbone. Every comparison uses four completed tasks, and Sequential and RF-Prompt are evaluated with the same backbone implementation. Figure 10 shows that RF-Prompt consistently improves average EER, pooled EER, and AF across all four alternative backbones. WavLM Large W2V-BERT 2.0 XLS-R 1B XLS-R 2B 0 5 10 15 20 Average EER (%) WavLM Large W2V-BERT 2.0 XLS-R 1B XLS-R 2B 0 5 10 15 20 25 Pooled EER (%) WavLM Large W2V-BERT 2.0 XLS-R 1B XLS-R 2B 0 2 4 6 8 10 AF (%) SequentialRF-Prompt Figure 10: Cross-backbone comparison of Sequential and RF-Prompt. Each panel reports one final continual-learning metric after four tasks; lower is better. Values above the bars are percentages. FEXTENDED RELATED WORK We review audio deepfake datasets and detection methods, followed by continual-learning ap- proaches with a focus on prompt-based adaptation. 3 https://huggingface.co/microsoft/wavlm-large 4 https://huggingface.co/facebook/w2v-bert-2.0 5 https://huggingface.co/facebook/wav2vec2-xls-r-1b 6 https://huggingface.co/facebook/wav2vec2-xls-r-2b 20 Preprint F.1AUDIO DEEPFAKE DATASETS Audio deepfake datasets provide complementary coverage of generation methods and recording conditions. ASVspoof 2019 includes logical-access attacks produced by speech synthesis and voice conversion, while ASVspoof 5 extends evaluation to more recent attacks and diverse recording con- ditions (Todisco et al., 2019; Wang et al., 2025). The ADD challenge complements this benchmark family with detection tasks addressing challenging acoustic conditions and partially manipulated audio (Yi et al., 2022). CodecFake focuses on audio produced through neural codec reconstruction, broadening coverage beyond conventional synthesis and conversion pipelines (Xie et al., 2024b). AT-ADD further expands evaluation across speech, singing, music, and environmental sound (Xie et al., 2026). These resources differ in both real-audio sources and deepfake generation mechanisms. EVDA evaluates continual ADD across eight dataset-defined tasks with changing sources and con- ditions, and RegO uses that benchmark (Zhang et al., 2024b; Chen et al., 2025). RAMI instead compares real-arrival and fake-organization choices over the same locked sample pool. F.2AUDIO DEEPFAKE DETECTION METHODS Domain generalization. ASDG aggregates real-speech representations across domains while sepa- rating fake representations, and W2V-ASDG integrates this objective with self-supervised speech features (Xie et al., 2024a; 2023). Related approaches encourage compact real representations through one-class learning and adaptive centroid shift (Zhang et al., 2021; Kim et al., 2024), or model fake diversity using multiple prototypes and latent-space augmentation (Huang et al., 2025). ADD systems also leverage pretrained XLS-R representations with layer-selective classification (Zhang et al., 2024a) and attentive merging of WavLM’s multi-layer representations (Pan et al., 2024), although scaling alone does not guarantee robustness under distribution shift (Li et al., 2026). Continual learning and domain adaptation. DFWF was the first study to apply continual learning to fake-audio detection, combining learning without forgetting with positive alignment of real em- beddings (Ma et al., 2021). RAWM then adapted weight-modification directions using the real/fake sample ratio and regularized the current detector with outputs from its preceding version (Zhang et al., 2023). RWM and RegO further address continual ADD through real–fake-aware and region- dependent optimization, respectively (Zhang et al., 2024c; Chen et al., 2025). Oiso et al. propose efficient prompt-based adaptation with limited target data (Oiso et al., 2024), focusing on target- domain performance rather than retention across successive tasks. RF-Prompt instead combines shared real-knowledge protection with incremental fake-knowledge expansion. F.3CONTINUAL LEARNING AND PROMPT-BASED ADAPTATION Classical continual-learning methods balance new-task adaptation with the preservation of previ- ously acquired knowledge. EWC penalizes changes to parameters estimated to be important for earlier tasks (Kirkpatrick et al., 2017). OWM instead projects updates away from previously learned input subspaces to reduce interference (Zeng et al., 2019). These approaches provide general mech- anisms for mitigating forgetting through parameter regularization or constrained optimization. Prompt-based continual learning adapts pretrained models through lightweight trainable tokens. L2P learns a prompt pool and retrieves relevant prompts without requiring task identity at inference (Wang et al., 2022b). DualPrompt combines task-invariant and task-specific prompts to accom- modate shared and specialized knowledge (Wang et al., 2022a). CODA-Prompt assembles decom- posed prompt components with input-conditioned attention weights (Smith et al., 2023). More re- cent methods investigate how prompt knowledge is reused and integrated: RainbowPrompt evolves task-specific prompts to enhance diversity (Hong et al., 2025), while KA-Prompt aligns knowledge components across domain-specific prompts to reduce interference during fusion (Xu et al., 2025). SinglePrompt examines whether prompt selection is necessary in task-free online continual learning (Park et al., 2026). SMoPE organizes a shared prefix prompt into sparse experts selected for each input (Le et al., 2026). These studies motivate prompt sharing, expansion, and selection as comple- mentary design choices. RF-Prompt investigates their asymmetric use for real and fake knowledge in continual ADD, where the output classes remain fixed while deepfake generation mechanisms accumulate. 21 Preprint GADDITIONAL METHOD DETAILS Selecting transferable knowledge. Let q(x) ∈ R d be a normalized temporal mean of the frozen SSL front-end representations. For each historical complete expert, define its signature by averaging over layers and token slots: s k = norm 1 Lm L X l=1 m X j=1 P l f,k,j .(10) At task initialization, we average up to 512 queries from current-task fake training examples to obtain ̄ q t . The selected historical expert and the frozen inherited base are k ⋆ = arg max k<t cos( ̄ q t , s k ),B t = sg(P f,k ⋆ ).(11) This selection occurs once when introducing a task. It differs from per-utterance soft routing at inference and requires no historical audio. We use the complete selected expert as the base, not a concatenation or mean of all historical experts. Initializing new directions. At each layer, we remove a random perturbation’s projection onto the historical complete-Prompt space and re-orthonormalize its rows. The perturbation is scaled by 0.1 times each inherited token’s norm before addition to the frozen base. Appendix B.4 provides the equations and numerical interpretation. Task 1 uses a standalone prompt with zero base. Our primary model uses frozen XLS-R 300M and a trainable AASIST backend. Audio is loaded at 16 kHz with a fixed input length of 64,600 samples. Each task is trained for 50 epochs with batch size 32, Adam, and a cosine learning-rate schedule from 10 −4 to 10 −6 . We evaluate the current task’s development set after every epoch, select the lowest development EER, and break ties using development loss. The selected model, rather than the final-epoch model, is evaluated and inherited by the next task. Reported experiments use seed 2026. We compare Sequential adaptation, EWC, OWM, RWM, and RegO within the same frozen- backbone setup. These are matched implementations in our training framework, not scores copied from their original papers. Offline joint co-training accesses all training tasks simultaneously and is reported separately. It is a useful reference but not a mathematical upper bound. HPROTECTION DEPTH AND LIMITED-DATA ANALYSIS The left panel of Figure 5 examines how many early transformer layers should receive each con- straint. We vary the first K layers for real prompt consistency or fake residual orthogonality while keeping the other constraint active across all 24 layers. Under the selected loss weights, both curves achieve their lowest pooled EER of 10.370% when the corresponding constraint is applied to all 24 layers. We therefore adopt all-layer real prompt consistency and fake residual orthogonality in RF-Prompt. The right panel evaluates adaptation with limited new fake training speech. Each task retains 2,400 unique real training utterances, while the number of unique fake training utterances is reduced to 100, 500, or 1,000. We resample each selected fake subset to 2,400 training instances per epoch, keeping class balance, optimization steps, and the learning-rate schedule unchanged. The original development pool, including 2,400 fake utterances per task, and evaluation sets are retained. Thus, the varied budget counts unique fake training utterances. RF-Prompt achieves the lowest pooled EER among all five continual-learning methods at every data budget. Relative to the strongest competing method at each budget, it reduces pooled EER by 5.270%, 4.100%, 1.660%, and 2.435% with 100, 500, 1,000, and 2,400 unique fake utterances per task, respectively. The particularly large improvement in the 100-shot setting demonstrates that the proposed asymmetric prompts remain effective when adapting to a newly arriving deepfake mechanism with scarce training audio. 22