Paper deep dive
Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models
Rongguang Ye, Zhan Zhuang, Yichen Wu, Ming Tang, Kede Ma
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Jointly adapting model parameters and task-specific soft prompts offers a promising solution, but faces two key limitations: historical prompts may become less effective as the model evolves, while their transferable cross-task relationships are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. LAPS first aligns historical prompts with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable Bézier control prompts. Once this anchored prompt space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the optimized prompt to draw on knowledge acquired from observed tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms distillation-based, prompt-based, and joint prompt--parameter adaptation baselines, improving average performance while reducing forgetting.
Tags
Links
- Source: https://arxiv.org/abs/2609.32499v1
- Canonical: https://arxiv.org/abs/2609.32499v1
Trouble viewing inline? Open PDF directly →
Full Text
83,568 characters extracted from source content.
Expand or collapse full text
Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models 1]City University of Hong Kong 2]Southern University of Science and Technology 3]Harvard University rongguaye2-c,zhazhuang3-c@my.cityu.edu.hk, , ://github.com/rG223/LAPS Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models Rongguang Ye Zhan Zhuang Yichen Wu Ming Tang Kede Ma Affiliation: [ Affiliation: [ Affiliation: [ Email: yiwu6@mgh.harvard.edu Email: tangm3@sustech.edu.cn Email: kede.ma@cityu.edu.hk September 26, 2026 Abstract Continually adapting large language models requires acquiring new knowledge while preserving previously learned capabilities. Jointly adapting model parameters and task-specific soft prompts offers a promising solution, but faces two key limitations: historical prompts may become less effective as the model evolves, while their transferable cross-task relationships are not explicitly learned. We propose Learning an Anchored Prompt Space (LAPS), which preserves historical prompt effectiveness and learns relationships among task-specific soft prompts to facilitate positive transfer. LAPS first aligns historical prompts with the updated model through self-distillation. LAPS then constructs an anchored prompt space whose vertices correspond to learned task-specific soft prompts and whose intermediate geometry is shaped by learnable Bézier control prompts. Once this anchored prompt space is learned, LAPS identifies the best-performing prompt for each task on its validation set, allowing the optimized prompt to draw on knowledge acquired from observed tasks. Experiments on the TRACE benchmark across three Qwen3 model scales show that LAPS consistently outperforms distillation-based, prompt-based, and joint prompt–parameter adaptation baselines, improving average performance while reducing forgetting. 1 Introduction Biological learning achieves efficient adaptation by combining plastic mechanisms that incorporate new evidence with stabilizing mechanisms that protect useful prior structure, rather than rewriting an entire system for each new experience (Grossberg, 1980; McCloskey and Cohen, 1989; French, 1999). Complementary learning systems and the stability–plasticity trade-off provide classic accounts of how learning can balance acquisition and retention (McClelland et al., 1995; Kirkpatrick et al., 2017). A similar pressure arises in large language model (LLM) deployment: after pretraining and instruction tuning, LLMs must adapt to new tasks, domains, policies, and user requirements without full retraining (Wang et al., 2023a; Ibrahim et al., 2024; Yang et al., 2025b). Practical adaptation must incorporate task-specific evidence from limited data while retaining capabilities acquired from previous tasks, making continual learning central to LLM adaptation. Recent studies on continual adaptation of LLMs have explored two complementary mechanisms: model parameter updates (Peng et al., 2024; Nayak et al., 2026) and task-specific soft prompts (Razdaibiedina et al., 2023; Qiu et al., 2025). Updating model parameters provides substantial capacity for acquiring task knowledge, but it can also interfere with prior knowledge. In contrast, prompt-based methods preserve task-specific behaviors without modifying the underlying model, although this limits adaptation capacity. To balance these strengths, later methods jointly adapt model parameters and prompts (Wang et al., 2024; Tiwari et al., 2026). Yet joint adaptation introduces an additional source of instability: historical prompts were optimized for earlier model parameters, and as model parameters evolve, these prompts may no longer induce their original behaviors. In addition to maintaining prior task behaviors, continual adaptation also creates opportunities for positive transfer across related tasks. If knowledge acquired from previous tasks can be organized and reused, later tasks may benefit from earlier adaptations rather than treating each task in isolation. Existing approaches have explored prompt retrieval, sharing, and composition to reuse knowledge across tasks (Wang et al., 2022b; Smith et al., 2023; Gao et al., 2024; Han et al., 2025; Hong et al., 2025). However, as the model evolves, these methods do not explicitly organize historical task-specific prompts into a learnable structure for discovering transferable cross-task relationships. To address these limitations, we propose Learning an Anchored Prompt Space (LAPS), a continual adaptation method that preserves historical task adaptations under evolving model parameters while identifying prompts that exploit positive transfer across tasks. LAPS contains two key elements. First, LAPS aligns historical prompts with the evolving LLM through on-policy self-distillation (Shenfeld et al., 2026a; Zhao et al., 2026). When model parameters are updated for a new task, historical prompts are adjusted so that they remain close to their original task behaviors under the updated model. By restoring prior behaviors through the prompts rather than constraining the model update, this alignment leaves greater flexibility for new-task adaptation. Second, LAPS organizes task-specific knowledge encoded by soft prompts into an anchored prompt space. Learned task-specific soft prompts serve as vertex anchors for each observed task, and a degree-k Bézier simplex introduces learnable control prompts that shape the geometry between these anchors. Rather than relying on direct prompt combination, LAPS learns this anchored space from cross-task interactions and then searches it to identify high-performing prompts that exploit positive transfer. Our contributions are summarized as follows: • We propose a continual adaptation method called LAPS that introduces historical prompt alignment and an anchored prompt space. The alignment preserves historical task behaviors under continued model updates without constraining backbone adaptation to new tasks • We introduce a two-stage optimization strategy for the anchored prompt space parameterized by a degree-k Bézier simplex: learning control prompts from cross-task interactions and selecting task-specific simplex coordinates by validation performance, enabling the discovery of positive transfer across observed tasks. • We evaluate LAPS on the TRACE benchmark (Wang et al., 2023b) across Qwen3 models from 1.7B to 8B. LAPS consistently achieves a favorable adaptation–retention trade-off against distillation-based, prompt-based, and joint prompt–parameter adaptation methods. 2 Related Work In this section, we review continual learning methods most closely related to LAPS: replay, knowledge distillation, and prompt-based adaptation. We focus on where reusable knowledge is stored, how prior behavior is maintained, and whether positive transfer is supported. Replay-based Continual Learning. Replay-based methods mitigate catastrophic forgetting by retaining and revisiting previous data when learning new tasks. Experience Replay (Isele and Cosgun, 2018; Rolnick et al., 2019; Fedus et al., 2020) shows that even a small memory of historical samples can help preserve previous knowledge. Subsequent work improves replay by storing additional training signals, such as historical logits (Buzzega et al., 2020), or by learning generative models that reproduce previous task knowledge (Gao and Liu, 2023; Maekawa et al., 2023). For LLM adaptation, OPR (Chen et al., 2026) further studies on-policy replay data for sequential updates. These methods rely on stored or regenerated historical information, while directly mixing replay data into model updates may interfere with current task adaptation. LAPS instead uses limited replay data to supervise historical prompt alignment and anchored prompt space learning, supporting retention and cross-task transfer while avoiding direct interference with new task model updates. Knowledge Distillation-based Continual Learning. Knowledge distillation mitigates forgetting by encouraging the updated model to match selected behaviors of an earlier model. LwF (Li and Hoiem, 2018) matches output distributions, while later continual learning methods distill soft predictions or intermediate representations (Buzzega et al., 2020; Douillard et al., 2020). Recent work further studies self-distillation for continual learning and restoration-style distillation for LLM behavior preservation (Dong et al., 2025; Shenfeld et al., 2026a). However, behavioral constraints that preserve earlier knowledge can also limit the acquisition of new task knowledge. LAPS instead applies self-distillation to historical prompts rather than constraining backbone updates. This leaves greater flexibility for new-task adaptation, while historical behaviors affected by model parameter updates are recovered through the aligned prompts. Prompt-based Continual Learning. Prompt-based continual learning stores task knowledge in learnable prompts. One common setting introduces task-specific prompts while typically keeping the pretrained backbone frozen. L2P (Wang et al., 2022b) learns a prompt pool for task-aware retrieval, enabling prompts to be reused across observed tasks. DualPrompt (Wang et al., 2022a) separates shared and task-specific prompts to capture common and task-dependent information. ProgPrompt (Razdaibiedina et al., 2023) learns a new prompt for each task and concatenates it with previous prompts, allowing later tasks to reuse earlier prompt knowledge. CODA-Prompt (Smith et al., 2023) decomposes prompts into reusable factors and learns attention-based combinations of these factors, providing a more flexible form of prompt composition. This prompt-only line of work reduces forgetting by storing task knowledge in learnable prompts, but keeping the backbone fixed limits the capacity for new-task adaptation through model parameter updates. Another line of work combines prompt learning with model parameter updates. ProMoT (Wang et al., 2024) first learns task-specific prompts and then updates model parameters. Subsequent methods combine meta prompting with low-rank parameter updates (Yu et al., 2025) or use fast–slow adaptation mechanisms (Tiwari et al., 2026). Jointly adapting prompts and model parameters is difficult because historical prompts are learned under earlier model states and may become less compatible as model parameters evolve. Existing prompt pools, prompt composition methods, and meta prompting approaches do not explicitly align historical prompts under changing model parameters or learn a controlled prompt space for positive transfer. LAPS addresses this setting by aligning historical prompts under the current model, learning Bézier control prompts to shape an anchored prompt space, and selecting task-specific simplex coordinates according to validation performance. 3 Learning an Anchored Prompt Space for Continual Adaptation In this section, we first formulate continual adaptation with joint prompt–parameter updates and identify the two difficulties that motivate LAPS. We then describe how LAPS aligns historical soft prompts, learns an anchored prompt space, and selects the final soft prompt for each observed task. 3.1 Problem Setup and Overview We study continual adaptation over a sequence of tasks 1:T=(1,…,T)T_1:T=(T_1,…,T_T). Each task tT_t is specified by a training set t=(t(i),yt(i))i=1NtD_t=\( x_t^(i),y_t^(i))\_i=1^N_t, where t(i) x_t^(i) and yt(i)y_t^(i) denote the input and target output of the i-th example from task t. The tasks are learned sequentially, with the full training data of previous tasks unavailable when adapting to tT_t. After each learning stage, the model is evaluated on all observed tasks. We follow the task-incremental setting (Van de Ven et al., 2022), where the identity of the current task is known at inference time and can therefore be used to select the corresponding task-specific prompt. Following recent studies (Wang et al., 2024; Tiwari et al., 2026), we consider two coupled adaptation components at stage t: the model parameters t θ_t and a task-specific soft prompt. We use j,tP_j,t to denote the soft prompt for task jT_j after adaptation stage t, with t,tP_t,t denoting the prompt newly learned for the current task tT_t. In the standard joint adaptation setting, each task-specific soft prompt is frozen after it is learned, , j,t=j,jP_j,t=P_j,j for all t≥jt≥ j. A basic joint adaptation objective therefore optimizes the model parameters t θ_t and the current task prompt t,tP_t,t on tD_t: t,t,t=argmin,(,y)∼t[ℓt(f(,,),y)],t=1,2,…,T. θ_t,P_t,t= _ θ,PE_( x,y) _t [ _t (f( x; θ,P),y ) ], t=1,2,...,T. (1) where f(,,)f( x; θ,P) denotes the model prediction for input x under parameters θ and soft prompt P, and ℓt _t denotes the task-specific supervised loss. LAPS instantiates this adaptation step through the alternating optimization strategy of ProMoT (Wang et al., 2024). For task tT_t, LAPS first learns the current task prompt t,tP_t,t while keeping the previous model parameters t−1 θ_t-1 fixed: t,t=argmin(,y)∼t[ℓt(f(,t−1,),y)].P_t,t= _PE_( x,y) _t [ _t (f( x; θ_t-1,P),y ) ]. (2) After solving Eq. (2), LAPS fixes t,tP_t,t and updates the model parameters on the current task: t=argmin(,y)∼t[ℓt(f(,,t,t),y)]. θ_t= _ θE_( x,y) _t [ _t (f( x; θ,P_t,t),y ) ]. (3) After this alternating update, LAPS has the current prompt and model state. However, the adaptation process still faces two challenges. First, updating model parameters from t−1 θ_t-1 to t θ_t for task t can cause model drift, making prompts learned for earlier tasks, j,t−1j<t\P_j,t-1\_j<t, less compatible with the updated model. Second, although task-specific soft prompts encode useful task knowledge, their transferable relationships are not explicitly organized as the model evolves. LAPS addresses these challenges through the three components illustrated in Fig. 1: historical prompt alignment, anchored prompt space learning, and validation-guided prompt selection. Figure 1: Overview of LAPS. (a) After adapting to the current task tT_t, LAPS aligns each historical prompt j,t−1P_j,t-1 to the updated model, producing j,tP_j,t through on-policy self-distillation. (b) The aligned prompts j,tj≤t\P_j,t\_j≤ t are fixed as simplex anchors, while non-vertex Bézier control prompts ,tC_ α,t are trained with cross-task replay losses to shape the space between anchors. (c) With the final model fixed, Bayesian optimization searches validation scores over simplex coordinates λ and returns the selected soft prompt T(j,T⋆)Z_T( λ_j,T ) for each observed task. 3.2 Historical Prompt Alignment via Self-Distillation As model parameters evolve, a prompt learned at an earlier stage may no longer induce the same behavior under the updated model. LAPS aligns historical prompts after each model update so that they remain compatible with the model, without constraining the model update in Eq. (3). Direct supervised fine-tuning of historical prompts on limited previous task replay data is insufficient because the replay set sparsely covers previous tasks, and ground-truth targets do not capture the full behavior of the original model–prompt pair, such as output distributions or responses to alternative continuations. LAPS therefore uses the previous model–prompt pair as the teacher, providing a richer behavioral target for aligning each historical prompt with the updated model. Let t⊂tS_t _t be a subset sampled from the current task. Prior work shows that the forward KL divergence from the original model to the updated model, evaluated on new task data, strongly predicts forgetting of previously learned capabilities (Shenfeld et al., 2026b). Motivated by this finding, we include tS_t alongside the stage-t replay set ℛtR_t when aligning historical prompts. For each task j=2,…,tj=2,…,t, the teacher is the previous pair (t−1,j,t−1)( θ_t-1,P_j,t-1), and the student is the pair (t,j,t)( θ_t,P_j,t). The teacher provides a behavioral reference for how the historical prompt functioned before the model update. Matching its predictive distribution allows the updated prompt to recover this behavior while keeping t θ_t fixed. LAPS aligns j,tP_j,t by minimizing the forward KL divergence: minj,t∼ℛt∪t<τ∼πt,j,t(⋅∣)[DKL(πt−1,j,t−1(⋅∣,<τ)∥πt,j,t(⋅∣,<τ))], _P_j,tE_ subarrayc x _t _t\\ y_<τ _ θ_t,P_j,t(· x) subarray [D_KL ( _ θ_t-1,P_j,t-1(· x, y_<τ)\| _ θ_t,P_j,t(· x, y_<τ) ) ], (4) where π,(⋅∣,<τ) _ θ,P(· x, y_<τ) denotes the next-token predictive distribution induced by parameters θ and prompt P, and <τ y_<τ is sampled on-policy from the student and used as the shared prefix for both teacher and student distributions. Optimizing Eq. (4) realigns earlier prompts to the updated model, so new task adaptation need not preserve every previous behavior in the model parameters. 3.3 Anchored Prompt Space Learning After alignment, LAPS has a set of task prompts 1,t,…,t,t\P_1,t,…,P_t,t\ that remain usable under the current model t θ_t. LAPS treats these task-specific soft prompts as anchors for their corresponding task knowledge. Using the anchors alone restricts inference to independently learned task solutions, leaving potential cross-task transfer unexplored. A natural alternative is convex interpolation, but its intermediate prompts are fixed linear combinations of the anchors and cannot be adapted through cross-task supervision. LAPS therefore introduces learnable non-vertex Bézier control prompts to shape the space between the fixed anchors using replay losses across observed tasks. At stage t, LAPS defines a simplex over the observed tasks: Δt−1=∈ℝ≥0t|∑j=1tλj=1. ^t-1= \ λ _≥ 0^t\; |\; _j=1^t _j=1 \. (5) Each component λj _j controls the relative emphasis on task jT_j. For any coordinate ∈Δt−1 λ∈ ^t-1, LAPS constructs a prompt t()Z_t( λ) with a degree-k Bézier simplex: t()=∑j=1tλjkj,t+∑k!∏j=1tαj!∏j=1tλjαj,t.Z_t( λ)=Σ _j=1^t _j^kP_j,t+Σ _ α k! _j=1^t _j! _j=1^t _j _jC_ α,t. -2.0pt (6) As shown in Fig. 1(b), the task prompts j,tP_j,t form the simplex vertices, while the non-vertex control prompts ,tC_ α,t shape the space between them. In the second sum, =(α1,…,αt) α=( _1,…, _t) ranges over the non-vertex degree-k multi-indices: each αj∈0,1,…,k _j∈\0,1,…,k\, ∑j=1tαj=k _j=1^t _j=k, and ∉k1,…,kt α∉\k e_1,…,k e_t\, where j e_j is the j-th unit vector. Each α indexes one learnable control prompt ,tC_ α,t, so LAPS adds (t+k−1k)−t t+k-1k-t control prompts at stage t. For coordinate λ, the Bernstein weight k!∏j=1tαj!∏j=1tλjαj k! _j=1^t _j! _j=1^t _j _j determines how strongly ,tC_ α,t contributes to t()Z_t( λ). When k=1k=1, no non-vertex control prompts exist, and Eq. (6) reduces to convex interpolation among task anchors. When k>1k>1, the control prompts add learnable degrees of freedom between the task anchors. Notably, these additional control prompts account for less than 0.5%0.5\% of the backbone parameters (see Appendix A.3). To learn these degrees of freedom, LAPS keeps the model parameters t θ_t and the task anchors fixed and optimizes only the non-vertex control prompts, collected as tC_t. These prompts are learned from supervision across all observed tasks. For each observed task j≤tj≤ t, the loss induced by the coordinate λ and control prompts tC_t is evaluated on examples from the stage-t replay set ℛtR_t: ℓj,t(,t)=(,y)∼ℛt[ℓj(f(,t,t()),y)]. _j,t( λ,C_t)=E_( x,y) _t [ _j (f( x; θ_t,Z_t( λ)),y ) ]. (7) Each λ specifies a desired trade-off among observed tasks. We compute the control prompt loss using a smooth Tchebycheff scalarization, which aggregates the λ-weighted losses and smoothly emphasizes the larger weighted loss: mint∼Δt−1[μlog∑j=1texp(λjℓj,t(,t)μ)], _C_tE_ λ ^t-1 [μ Σ _j=1^t ( _j _j,t( λ,C_t)μ ) ], (8) where μ>0μ>0 controls the smoothness of the scalarization. Optimizing Eq. (8) learns the non-vertex control prompts across sampled trade-offs λ, allowing cross-task interactions to shape the anchored prompt space for subsequent task-specific search. 3.4 Validation-Guided Prompt Selection After anchored prompt space learning, LAPS fixes the final model T θ_T, task anchors, and Bézier control prompts, and uses validation data to select a soft prompt from the learned anchored space for each observed task. For task jT_j, the prompt T()Z_T( λ), parameterized by the simplex coordinate ∈ΔT−1 λ∈ ^T-1, is directly optimized with the fixed model on the task’s validation set: j,T⋆=argmax∈ΔT−1Valj,T(),j=1,…,T, λ_j,T = _ λ∈ ^T-1Val_j,T( λ), j=1,…,T, (9) where Valj,T()Val_j,T( λ) denotes the resulting task-specific validation score, which may be discrete or otherwise unsuitable for gradient-based optimization ( , accuracy, F1 score, or exact match). LAPS therefore treats Valj,TVal_j,T as a black-box objective and approximates Eq. (9) with Bayesian optimization (Snoek et al., 2012; Snoek et al., 2014). For each task, a Gaussian process surrogate is fitted to the observed coordinate–score pairs, and Expected Improvement (Jones et al., 1998) proposes coordinates that balance high predicted validation scores with uncertainty. Each proposed coordinate λ is used to construct T()Z_T( λ), which is evaluated on the validation set to form a new coordinate–score pair for updating the surrogate. After the search, the leading candidate coordinates are reevaluated on the full validation set over repeated runs, with the task’s own anchor added if it is not already among the finalists. The coordinate with the highest mean validation score is mapped through Eq. (6) to obtain the final soft prompt for that task. Appendix A.2 provides further details on the search and evaluation. 4 Experiment In this section, we evaluate LAPS on continual instruction tuning. We first give the TRACE setup and comparison methods, then analyze aggregate performance, task-wise adaptation dynamics, and the learned prompt space. Finally, we present ablations and diagnostic studies of LAPS components. 4.1 Experimental Setup Benchmark. We conduct experiments on TRACE (Wang et al., 2023b), an eight-task continual instruction tuning benchmark consisting of C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, and 20Minuten. Following prior work (Chen et al., 2026), we subsample 5,000 training examples from each task and use the official evaluation set. Training. We evaluate all methods on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B (Yang et al., 2025a). Methods that update model parameters use full-parameter fine-tuning with AdamW, a learning rate of 1×10−51× 10^-5, a global batch size of 128, a maximum sequence length of 2,048, and bf16 precision. Unless otherwise stated, prompt-based methods use 3232 prompt tokens. Following the task order in TRACE, we train the eight tasks for [5,3,7,5,3,5,5,7][5,3,7,5,3,5,5,7] epochs, respectively. For replay-based methods, the replay ratio is set to 1%1\%. For LAPS, we use a cubic Bézier simplex with degree k=3k=3, resulting in 112 control prompts for eight tasks. The scalarization smoothing parameter is μ=0.1μ=0.1. We construct the alignment dataset by combining replay data from previous tasks and data from the current task at a :11\!:\!1 ratio. Bayesian optimization is performed on the official validation set with a budget of 100 evaluations per task, except for MeetingBank, where we use 50 evaluations. Baselines. We compare against baselines that stress different continual-adaptation mechanisms. SFT is a parameter-only adaptation baseline; vanilla replay (Rolnick et al., 2019) and on-policy replay (OPR-SC and OPR-RU, Chen et al., 2026) add stored or selected previous task examples; LwF (Li and Hoiem, 2018) uses distillation-based adaptation; ProgPrompt (Razdaibiedina et al., 2023) performs prompt-only adaptation; and ProMoT (Wang et al., 2024) implements joint prompt–parameter adaptation. Together, these baselines enable a systematic comparison of retention behavior across parameter-only adaptation, replay, distillation, frozen-backbone prompt tuning, and joint prompt–parameter adaptation under continual learning. Metrics. We report average performance (Average) across all tasks and backward transfer (BWT, Lopez-Paz and Ranzato, 2017) as a measure of forgetting. Let Rj,tR_j,t denote the performance on task jT_j after learning stage t. We define BWT=1T−1∑j=1T−1(Rj,T−Rj,j)BWT= 1T-1 _j=1^T-1(R_j,T-R_j,j). Higher Average indicates better final performance averaged across tasks, while higher BWT indicates stronger retention of earlier tasks. All results are averaged over eight inference runs with a sampling temperature of 0.10.1. Method characteristics and trainable parameter counts are provided in Appendix A.3; dataset statistics and evaluation metrics in Appendix A.4; and additional experimental analyses in Appendix A.5. Table 1: Performance across Qwen3 model scales. The best and second best results at each model scale are shown in bold and underlined, respectively. Method C-STANCE FOMC MeetingBank Py150 ScienceQA NG-cm NG-ds 20Minuten Average ↑ BWT ↑ Qwen3-1.7B Initial 36.336.3 32.732.7 18.418.4 2.52.5 63.463.4 70.270.2 30.230.2 35.835.8 36.236.2 N/A SFT 53.1¯ 53.1 58.558.5 20.120.1 52.652.6 76.776.7 46.346.3 53.853.8 38.838.8 50.050.0 −9.3-9.3 Replay 52.852.8 62.462.4 46.646.6 56.656.6 73.473.4 51.151.1 53.253.2 38.938.9 54.454.4 −4.1-4.1 OPR-RU 50.450.4 58.258.2 47.047.0 54.754.7 77.177.1 55.7¯ 55.7 51.751.7 38.838.8 54.254.2 −4.9-4.9 OPR-SC 50.750.7 51.951.9 38.638.6 55.055.0 77.777.7 47.547.5 52.652.6 38.738.7 51.651.6 −7.6-7.6 LwF 45.945.9 28.428.4 38.238.2 43.443.4 41.641.6 43.143.1 53.553.5 38.038.0 41.541.5 −13.8-13.8 ProgPrompt 53.553.5 60.660.6 47.747.7 49.849.8 73.873.8 45.445.4 47.647.6 38.438.4 52.152.1 0.00.0 ProMoT 50.950.9 62.7¯ 62.7 50.9¯ 50.9 59.2¯ 59.2 84.284.2 47.747.7 67.767.7 40.540.5 58.0¯ 58.0 −4.4-4.4 LAPS (Ours) 52.052.0 69.469.4 58.758.7 60.060.0 83.4¯ 83.4 56.056.0 64.8¯ 64.8 40.0¯ 40.0 60.560.5 −1.2¯ -1.2 Qwen3-4B Initial 42.742.7 5.05.0 20.620.6 2.42.4 64.664.6 76.476.4 38.238.2 35.935.9 35.735.7 N/A SFT 52.552.5 66.266.2 54.354.3 37.737.7 89.289.2 66.566.5 77.3¯ 77.3 40.840.8 60.660.6 −6.7-6.7 Replay 55.355.3 71.471.4 59.059.0 62.3¯ 62.3 91.991.9 64.064.0 75.075.0 40.540.5 64.9¯ 64.9 −2.7-2.7 OPR-RU 53.053.0 73.373.3 57.757.7 62.262.2 91.191.1 63.963.9 76.976.9 41.141.1 64.9¯ 64.9 −1.2-1.2 OPR-SC 52.852.8 61.961.9 58.058.0 60.460.4 82.782.7 62.862.8 77.877.8 40.440.4 62.162.1 −5.5-5.5 LwF 56.056.0 71.171.1 47.747.7 38.838.8 83.383.3 65.165.1 69.469.4 39.939.9 58.958.9 +0.2+0.2 ProgPrompt 57.8¯ 57.8 62.262.2 51.451.4 57.957.9 79.779.7 55.455.4 52.452.4 39.039.0 57.057.0 0.0¯ 0.0 ProMoT 51.951.9 59.359.3 62.962.9 63.063.0 93.493.4 63.163.1 72.772.7 41.0¯ 41.0 63.463.4 −2.0-2.0 LAPS (Ours) 59.559.5 71.9¯ 71.9 60.3¯ 60.3 61.661.6 92.4¯ 92.4 66.4¯ 66.4 76.976.9 40.640.6 66.266.2 −0.3-0.3 Qwen3-8B Initial 52.652.6 21.021.0 20.820.8 2.42.4 68.068.0 77.877.8 38.438.4 36.436.4 39.639.6 N/A SFT 47.347.3 63.563.5 50.050.0 50.050.0 73.473.4 67.967.9 77.777.7 41.541.5 58.958.9 −10.1-10.1 Replay 52.352.3 71.971.9 55.255.2 62.162.1 92.492.4 67.167.1 79.579.5 41.441.4 65.265.2 −2.9-2.9 OPR-RU 52.652.6 74.574.5 60.860.8 62.6¯ 62.6 92.092.0 69.169.1 78.5¯ 78.5 41.141.1 66.466.4 −2.7-2.7 OPR-SC 51.851.8 72.672.6 57.057.0 62.162.1 93.0¯ 93.0 69.869.8 76.776.7 41.541.5 65.665.6 −2.5-2.5 LwF 49.549.5 72.072.0 46.346.3 30.830.8 84.784.7 52.052.0 66.166.1 39.639.6 55.155.1 +4.3+4.3 ProgPrompt 58.658.6 64.964.9 54.954.9 60.760.7 84.984.9 47.547.5 56.056.0 39.339.3 58.458.4 0.00.0 ProMoT 55.955.9 69.869.8 64.5¯ 64.5 64.164.1 94.694.6 69.3¯ 69.3 78.5¯ 78.5 41.6¯ 41.6 67.3¯ 67.3 −0.2-0.2 LAPS (Ours) 57.6¯ 57.6 73.4¯ 73.4 66.466.4 64.164.1 92.492.4 66.266.2 78.378.3 41.841.8 67.567.5 +1.2¯ +1.2 4.2 Main Results Table 1 shows the adaptation–retention pattern across the three Qwen3 model scales. Parameter updating methods such as SFT, Replay, OPR-SC, OPR-RU, and ProMoT often improve new-task performance, but their BWT values show that earlier task behavior can still be overwritten. ProgPrompt has the opposite profile: freezing the backbone helps preserve performance on previous tasks, yet its Average remains lower than the best parameter-updating methods on several scales. LAPS attains the highest Average on all three model scales, with 60.5%, 66.2%, and 67.5% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively, while keeping BWT close to or above zero. The most directly comparable baseline is ProMoT, since it also combines soft prompts with model parameter updates. Relative to ProMoT, LAPS improves Average by 0.2%–2.8%0.2\%--2.8\% and BWT by 1.4%–3.2%1.4\%--3.2\%. This consistent pattern across model scales suggests a more favorable adaptation–retention trade-off that is not confined to a particular model size. Continual Adaptation Dynamics. Beyond the aggregate table, the task-wise trajectories in Fig. 2 reveal when forgetting occurs during the continual sequence. On Qwen3-1.7B, SFT illustrates the failure mode of sequential fine-tuning: later updates are followed by degradation on several earlier tasks. ProgPrompt keeps the backbone and previous prompts fixed, which helps retention, but this constraint leaves less room for new-task adaptation. ProMoT adapts both prompts and model parameters and improves performance on several new tasks, yet earlier tasks such as Py150 and NumGLUE-cm still decline as the model parameters continue to change across the task sequence. In contrast, LAPS exhibits a different trajectory. After sequential adaptation ends at 8T_8, indicated by the vertical gray dashed line, the model is fixed and Bayesian optimization selects a prompt for each task from the learned anchored prompt space. The rightmost point in each subplot shows performance with the selected prompt, compared with the preceding pre-search point. Improvements on most tasks suggest that useful task-specific regions exist beyond the original task anchors. Figure 2: Task-wise continual adaptation trajectories with Qwen3-1.7B. Each panel tracks one task after it is learned, and later points show its performance after subsequent adaptation stages. Drops indicate forgetting as the model learns new tasks. The gray dashed line marks the transition from sequential adaptation to the validation-guided prompt selection introduced in LAPS. Figure 3: PCA projection of learned anchored prompt spaces across Qwen3 model scales. Figure 4: Average performance and BWT across replay ratios on Qwen3-1.7B. Visualization of the Learned Prompt Space. Fig. 4 uses Principal Component Analysis (PCA) to examine where validation-selected prompts lie relative to the original per-task prompt anchors. The selected prompts are generally not located at the anchors and improve average performance by 0.78%0.78\%, 0.48%0.48\%, and 2.70%2.70\% over using the anchors on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively. This pattern suggests that the Bézier control prompts define a searchable prompt space in which useful task-specific prompts need not coincide with the original anchors. 4.3 Ablations and Diagnostic Analyses Effect of LAPS Components. Table 4.3 examines the contributions of historical prompt alignment, anchored prompt space learning, and validation-guided prompt selection. The full model outperforms each removal variant, indicating that the gains do not arise from any single component alone. Alignment has the clearest effect on retention: removing it causes the largest degradation in BWT while also lowering Average. This pattern is consistent with the role of alignment, which keeps historical prompts compatible with the updated model for each new task. Prompt space learning and selection provide additional benefits once alignment is present. Removing either component weakens the final trade-off, but the two ablations isolate different functions: prompt space learning tests whether the intermediate prompt geometry is learned, whereas prompt selection tests whether each task can choose a suitable prompt after training. The results suggest complementary roles: alignment is most closely tied to retaining historical behavior, while space learning and selection expand the learned prompt space and use it for task-specific prompt selection. Transferability of LAPS Components. Table 3 compares replay augmentation with integrating the prompt space learning and selection components of LAPS into ProgPrompt and ProMoT. Adding replay alone provides limited or inconsistent benefits and can even reduce performance. Adding prompt space learning and selection improves both Average and BWT across the reported model scales. These comparisons suggest that organizing and searching the prompt space provides additional adaptation and retention benefits beyond replay alone. Table 2: Ablation of the three LAPS components: historical prompt alignment, prompt space learning, and prompt selection. Components Qwen3-1.7B Qwen3-4B Alignment Space Learning Selection Average BWT Average BWT ✓ ✓ ✓ 60.560.5 −1.2-1.2 66.266.2 −0.3-0.3 ✗ ✓ ✓ 58.858.8 −4.0-4.0 65.665.6 −0.4-0.4 ✓ ✗ ✓ 59.659.6 −2.3-2.3 65.565.5 −0.5-0.5 ✓ ✓ ✗ 59.859.8 −2.1-2.1 65.865.8 −0.7-0.7 ✗ ✗ ✗ 57.357.3 −5.8-5.8 64.364.3 −1.9-1.9 Method Qwen3-1.7B Qwen3-4B Average BWT Average BWT ProgPrompt† 52.352.3 0.00.0 58.958.9 0.00.0 + Replay 50.850.8 0.00.0 55.155.1 0.00.0 + Space Learning & Selection 53.253.2 +1.0+1.0 60.560.5 +1.8+1.8 ProMoT 58.058.0 −4.4-4.4 63.463.4 −2.0-2.0 + Replay 54.254.2 −6.9-6.9 63.963.9 −2.8-2.8 + Space Learning & Selection 58.858.8 −4.0-4.0 65.665.6 −0.4-0.4 Table 3: Prompt-based baselines with replay and LAPS components. † denotes the variant without prompt concatenation. Table 4: Effect of the soft prompt length. Results are averaged over eight evaluations. Method Qwen3-1.7B Qwen3-4B Length = 32 Length = 64 Length = 128 Length = 32 Length = 64 Length = 128 Average BWT Average BWT Average BWT Average BWT Average BWT Average BWT ProgPrompt 52.152.1 0.00.0 49.249.2 0.00.0 51.251.2 0.00.0 57.057.0 0.00.0 57.457.4 0.00.0 57.557.5 0.00.0 ProMoT 58.058.0 −4.4-4.4 55.855.8 −4.6-4.6 48.648.6 −6.9-6.9 63.463.4 −2.0-2.0 65.565.5 −0.2-0.2 62.662.6 −3.2-3.2 LAPS (Ours) 60.560.5 −1.2-1.2 59.559.5 −1.6-1.6 58.558.5 0.70.7 66.266.2 −0.3-0.3 65.665.6 −0.4-0.4 64.764.7 −1.3-1.3 k # Control Prompts Qwen3-1.7B Qwen3-4B Average BWT Average BWT 11 00 59.659.6 −2.3-2.3 65.565.5 −0.5-0.5 22 2828 60.460.4 −2.1-2.1 65.765.7 −0.4-0.4 33 112112 60.560.5 −1.2-1.2 66.266.2 −0.3-0.3 44 322322 59.359.3 −2.4-2.4 65.265.2 −0.9-0.9 Table 5: Effect of the Bézier degree k. Method Qwen3-1.7B Qwen3-4B Average BWT Average BWT OPR-SC 51.651.6 −7.6-7.6 62.162.1 −5.5-5.5 OPR-SC + LAPS 60.060.0 −1.1-1.1 64.264.2 −1.5-1.5 OPR-RU 54.254.2 −4.9-4.9 64.9¯ 64.9 −1.2-1.2 OPR-RU + LAPS 60.060.0 −2.0-2.0 65.565.5 −0.4-0.4 Table 6: LAPS with on-policy replay. Effect of the Replay Ratio. Fig. 4 compares LAPS with replay-based methods under different replay ratios, testing whether retention is mainly explained by the number of previous-task examples. Increasing the replay ratio does not lead to monotonic improvements, and LAPS remains competitive with a much smaller replay budget. These results suggest that retention depends not only on replay volume but also on how historical samples are used for alignment and prompt space learning. Effect of the Soft Prompt Length. Table 4 varies the number of soft prompt tokens to test whether the observed gains depend on a particular prompt length. Increasing the prompt length does not consistently improve performance, and the baselines show greater sensitivity to this choice. Across all tested prompt lengths and model scales, LAPS maintains the highest Average, suggesting that its aggregate advantage does not depend on a narrowly tuned prompt length. Effect of the Bézier Degree k. Table 5 studies the capacity of the learned prompt space by varying the Bézier degree. At k=1k=1, prompt search is limited to convex combinations of the task anchors. Increasing k introduces learnable control prompts and consistently improves both Average and BWT up to k=3k=3 across model scales. Further increasing k to 44 increases the number of control prompts but brings no additional gain, suggesting that a moderate degree is sufficient in this setting. Compatibility of LAPS with On-Policy Replay. LAPS can be combined with on-policy replay methods. Table 6 shows that adding LAPS improves both Average and BWT for OPR-SC and OPR-RU across the reported model scales. These results indicate compatibility with on-policy replay: replay selects which historical examples are revisited, while LAPS provides an alignment and prompt selection mechanism for using the selected examples. 5 Conclusion and Discussion We presented LAPS for continual LLM adaptation by jointly updating prompts and model parameters. LAPS aligns historical prompts with the evolving model through self-distillation, learns an anchored prompt space that captures cross-task relationships, and selects task-specific soft prompts through validation-guided search. Experiments on TRACE across three Qwen3 scales show that LAPS achieves the highest average performance among compared methods while reducing forgetting. These results suggest that organizing task-specific soft prompts into a structured, searchable space can facilitate cross-task knowledge transfer during continual adaptation. The current formulation assumes known task identities and introduces additional control prompts as tasks accumulate. This becomes more challenging in open-ended learning, where task boundaries are unavailable and prompt count can grow continuously. A natural extension is to learn a task-free router that selects prompts from the input, without explicit task labels. Meanwhile, prompt capacity could be managed by adding new control prompts only when needed and consolidating redundant ones, thereby supporting flexible cross-task interactions while keeping memory overhead bounded. The learned anchored prompt space also provides a way to study which previous tasks are useful for a new task. By evaluating how task losses change across the anchored prompt space, LAPS could identify regions associated with positive transfer or interference and use this information to guide prompt selection. The learned anchored space could also provide better initializations for new task prompts by reusing nearby anchors or control prompts, reducing the amount of data and optimization needed for adaptation. These directions would enable LAPS to reuse cross-task structure more explicitly and improve the efficiency of future adaptation. References Buzzega et al. (2020) Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: A strong, simple baseline. In Advances in Neural Information Processing Systems, pages 15920–15930, 2020. Chen et al. (2026) Yan Chen, Taojie Zhu, Meng Zhang, Xin Chen, Jiaqi Huang, Dongyang Xu, and Yizhi Wang. On-policy replay for continual supervised fine-tuning. arXiv preprint arXiv:2605.29495, 2026. Dong et al. (2025) Zican Dong, Junyi Li, Jinhao Jiang, Mingyu Xu, Wayne Xin Zhao, Bingning Wang, and Weipeng Chen. LongReD: Mitigating short-text degradation of long-context large language models via restoration distillation. In Annual Meeting of the Association for Computational Linguistics (Long Papers), pages 10687–10707, 2025. Douillard et al. (2020) Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. PODNet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision, pages 86–102, 2020. Fedus et al. (2020) William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pages 3061–3071, 2020. French (1999) Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999. Gao and Liu (2023) Rui Gao and Weiwei Liu. DDGR: Continual learning with deep diffusion-based generative replay. In International Conference on Machine Learning, pages 10744–10763, 2023. Gao et al. (2024) Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal-free continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28463–28473, 2024. Grossberg (1980) Stephen Grossberg. How does a brain build a cognitive code? Psychological Review, 87(1):1–51, 1980. Han et al. (2025) Jisu Han, Jaemin Na, and Wonjun Hwang. Semantic prompting with image token for continual learning. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6987–6997, 2025. Hong et al. (2025) Kiseong Hong, Gyeong-hyeon Kim, and Eunwoo Kim. RainbowPrompt: Diversity-enhanced prompt-evolving for continual learning. In IEEE/CVF International Conference on Computer Vision, pages 1130–1140, 2025. Ibrahim et al. (2024) Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024. Isele and Cosgun (2018) David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI Conference on Artificial Intelligence, pages 3302–3309, 2018. Jones et al. (1998) Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455–492, 1998. Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. Levenshtein (1966) Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics-Doklady, 10(8):707–710, 1966. Li and Hoiem (2018) Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2018. Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, page 6470–6479, 2017. Maekawa et al. (2023) Aru Maekawa, Hidetaka Kamigaito, Kotaro Funakoshi, and Manabu Okumura. Generative replay inspired by hippocampal memory indexing for continual language learning. In Conference of the European Chapter of the Association for Computational Linguistics, pages 930–942, 2023. McClelland et al. (1995) James L. McClelland, Bruce L. McNaughton, and Randall C. O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419–457, 1995. McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pages 109–165. Academic Press, 1989. Nayak et al. (2026) Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar, Prateek Chanda, Kai Xu, Oleg Silkin, Mustafa Eyceoz, Hao Wang, Aldo Pareja, et al. Sculpting subspaces: Constrained full fine-tuning in LLMs for continual learning. In International Conference on Learning Representations, pages 20212–20243, 2026. Peng et al. (2024) Bohao Peng, Zhuotao Tian, Shu Liu, Ming-Chang Yang, and Jiaya Jia. Scalable language model with generalized continual learning. In International Conference on Learning Representations, pages 28412–28434, 2024. Qiu et al. (2025) Jiabao Qiu, Zixuan Ke, and Bing Liu. Continual learning using only large language model prompting. In International Conference on Computational Linguistics, pages 6014–6023, 2025. Razdaibiedina et al. (2023) Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In International Conference on Learning Representations, 2023. Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pages 348–358, 2019. Shenfeld et al. (2026a) Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. In International Conference on Machine Learning, 2026a. Shenfeld et al. (2026b) Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL’s razor: Why online reinforcement learning forgets less. In International Conference on Learning Representations, pages 59839–59864, 2026b. Smith et al. (2023) James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023. Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, pages 2951–2959, 2012. Snoek et al. (2014) Jasper Snoek, Kevin Swersky, Rich Zemel, and Ryan Adams. Input warping for Bayesian optimization of non-stationary functions. In International Conference on Machine Learning, pages 1674–1682, 2014. Tiwari et al. (2026) Rishabh Tiwari, Kusha Sareen, Lakshya A. Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S. Dhillon, Rishabh Agarwal, and Fnu Devvrit. Learning, fast and slow: Towards LLMs that adapt continually. In COLM Workshop on Context Beyond the Window: Persistent Knowledge in Language Models, 2026. Van de Ven et al. (2022) Gido M. Van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197, 2022. Wang et al. (2023a) Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of Empirical Methods in Natural Language Processing, pages 10658–10671, 2023a. Wang et al. (2023b) Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. TRACE: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023b. Wang et al. (2024) Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit Dhillon, and Sanjiv Kumar. Two-stage LLM fine-tuning with less specialization and more generalization. In International Conference on Learning Representations, pages 20380–20398, 2024. Wang et al. (2022a) Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. DualPrompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631–648, 2022a. Wang et al. (2022b) Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022b. Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415, 2016. Yang et al. (2025a) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a. Yang et al. (2025b) Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey. ACM Computing Surveys, 57(5):1–38, 2025b. Yu et al. (2025) Xiaobing Yu, Jin Yang, Xiao Wu, Peijie Qiu, and Xiaofeng Liu. FM-LoRA: Factorized low-rank meta-prompting for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 6399–6408, 2025. Zhao et al. (2026) Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. In International Conference on Machine Learning, 2026. Appendix A Appendix A.1 Properties of the Anchored Prompt Space We analyze the optimization of the shared Bézier parameters. During this stage, the LLM parameters and task-specific prompt anchors remain fixed, and only the non-vertex control prompts are updated. We adopt the following standard assumptions. Assumption 1 (Regularity of Task Losses). For every observed task i and simplex coordinate ∈Δt−1 λ∈ ^t-1, the replay loss ℓi(,) _i( λ,C) is continuously differentiable with respect to the non-vertex control prompts C. Moreover, there exist constants β>0β>0 and M>0M>0 such that, for all feasible C and ′C , ‖∇ℓi(,)−∇ℓi(,′)‖2 \| _C _i( λ,C)- _C _i( λ,C ) \|_2 ≤β‖−′‖2, ≤β \|C-C \|_2, (10) ‖∇ℓi(,)‖2 \| _C _i( λ,C) \|_2 ≤M. ≤ M. (11) We further assume that the expected smooth Tchebycheff objective is lower bounded, i.e., there exists a constant GinfG_ such that Gt()≥GinfG_t(C)≥ G_ for all feasible C. The regularity of the task losses first implies smoothness of the scalarized objective. Lemma 1 (Smoothness of the Smooth Tchebycheff objective). Under Assumption 1, for every fixed ∈Δt−1 λ∈ ^t-1, the pointwise objective gμ(,)g_μ( λ,C) has a Lipschitz-continuous gradient with respect to C for every μ>0μ>0. Consequently, Gt()=∼Δt−1[gμ(,)]G_t(C)=E_ λ ^t-1 [g_μ( λ;C) ] (12) is LGL_G-smooth for some finite constant LGL_G. Proof. For a fixed λ, the gradient of the smooth Tchebycheff objective gμg_μ is ∇gμ=∑i=1tqiλi∇ℓi. _Cg_μ= _i=1^tq_i _i _C _i. (13) The first condition in Assumption 1 ensures that the gradient of every task loss varies Lipschitz-continuously with C. The second condition bounds the variation introduced through the softmax coefficients qiq_i, whose derivatives are scaled by 1/μ1/μ. For any fixed μ>0μ>0, both contributions are therefore bounded, implying that ∇gμ _Cg_μ is Lipschitz continuous. The expectation over ∼Δt−1 λ ^t-1 preserves Lipschitz continuity of the gradient. Hence, Gt()G_t(C) is LGL_G-smooth for some finite LGL_G. ∎ Lemma 1 establishes the smoothness of the smooth Tchebycheff objective. We next exploit the Bézier parameterization to make its smoothness constant explicit, which will be used in the subsequent convergence analysis. Lemma 2 (Smoothness Bound for the Bézier–Smooth Tchebycheff Objective). Under Assumption 1, if λ is sampled uniformly from Δt−1 ^t-1, the smoothness constant in Lemma 1 satisfies LG≤Rt,k(β+M2μ),L_G≤ R_t,k (β+ M^2μ ), (14) where Rt,k=(k!)2(t−1)!(t+2k−1)![4k(t/2)k!−t(2k)],R_t,k= (k!)^2(t-1)!(t+2k-1)! [ 4^k(t/2)_kk!-t 2kk ], (15) and (a)k=a(a+1)⋯(a+k−1)(a)_k=a(a+1)·s(a+k-1) denotes the rising factorial. Proof. For fixed λ, the Bézier prompt can be written as t()=∑j=1tλjkj,t+∑Bk(),t.Z_t( λ)= _j=1^t _j^kP_j,t+ _ αB_ α^k( λ)C_ α,t. (16) Since the task anchors j,tj=1t\P_j,t\_j=1^t and λ are fixed during control prompt optimization, the first term is constant with respect to tC_t. Hence, t()Z_t( λ) is affine in the non-vertex control prompts. Define ρt,k2()=∑(Bk())2, _t,k^2( λ)= _ α (B_ α^k( λ) )^2, (17) where the sum is over the non-vertex degree-k multi-indices. Since the Bézier map is affine in the control prompts, its Jacobian satisfies ‖J,()‖op2=ρt,k2(). \|J_Z,C( λ) \|_op^2= _t,k^2( λ). (18) Therefore, the assumed gradient bound and β-smoothness of each replay loss imply ‖∇ℒi,tr‖F \| _CL_i,t^r \|_F ≤Mρt,k(), ≤ M _t,k( λ), (19) ‖∇2ℒi,tr‖op \| _C^2L_i,t^r \|_op ≤βρt,k2(). ≤β _t,k^2( λ). (20) For fixed λ, let gμ(,)=μlog∑i=1texp(λiℓi,tμ),g_μ( λ,C)=μ _i=1^t ( _i _i,tμ ), (21) and define the corresponding softmax weights qi=exp(λiℓi,t/μ)∑j=1texp(λjℓj,t/μ).q_i= ( _i _i,t/μ) _j=1^t ( _j _j,t/μ). (22) Its Hessian with respect to the control prompts can be written as ∇2gμ=∑i=1tqiλi∇2ℓi,t+1μCovq(λi∇ℓi,t). _C^2g_μ= _i=1^tq_i _i _C^2 _i,t+ 1μCov_q ( _i _C _i,t ). (23) Since 0≤λi≤10≤ _i≤ 1 and ∑iqi=1 _iq_i=1, applying Eq. (20) gives ‖∇2gμ(,)‖op≤ρt,k2()(β+M2μ). \| _C^2g_μ( λ,C) \|_op≤ _t,k^2( λ) (β+ M^2μ ). (24) Taking expectation over uniformly sampled ∈Δt−1 λ∈ ^t-1 therefore yields LG≤[ρt,k2()](β+M2μ).L_G _ λ [ _t,k^2( λ) ] (β+ M^2μ ). (25) It remains to evaluate the expectation. For uniform sampling over the simplex, the Dirichlet moment identity gives [∏j=1tλj2αj]=(t−1)!(t+2k−1)!∏j=1t(2αj)!.E_ λ [ _j=1^t _j^2 _j ]= (t-1)!(t+2k-1)! _j=1^t(2 _j)!. (26) Using Bk()=k!∏j=1tαj!∏j=1tλjαj,B_ α^k( λ)= k! _j=1^t _j! _j=1^t _j _j, (27) we obtain [ρt,k2()]=(k!)2(t−1)!(t+2k−1)!∑∏j=1t(2αjαj),E_ λ [ _t,k^2( λ) ]= (k!)^2(t-1)!(t+2k-1)! _ α _j=1^t 2 _j _j, (28) where the sum again excludes the vertex multi-indices. The generating function identity ∑≥0∑jαj=k∏j=1t(2αjαj)=4k(t/2)k! _ subarrayc α≥ 0\\ _j _j=k subarray _j=1^t 2 _j _j= 4^k(t/2)_kk! (29) sums over all degree-k multi-indices. Each of the t excluded vertex indices contributes (2k) 2kk. Hence, [ρt,k2()]=(k!)2(t−1)!(t+2k−1)![4k(t/2)k!−t(2k)]=Rt,k.E_ λ [ _t,k^2( λ) ]= (k!)^2(t-1)!(t+2k-1)! [ 4^k(t/2)_kk!-t 2kk ]=R_t,k. (30) Substituting this expression into Eq. (25) proves Eq. (14). ∎ We can now establish convergence of the shared control prompt optimization. Since the induced objective is generally non-convex, the appropriate guarantee is convergence to a first-order stationary point rather than global optimality. Theorem 1 (Convergence of Bézier–Smooth Tchebycheff Optimization). Given Lemma 2, suppose that Gt()G_t(C) is lower bounded by GinfG_ . Then, gradient descent with learning rate η=1/LGη=1/L_G satisfies 1K∑s=0K−1‖∇Gt((s))‖F2≤2Rt,k(β+M2μ)(Gt((0))−Ginf)K. 1K _s=0^K-1 \|∇ G_t(C^(s)) \|_F^2≤ 2R_t,k (β+ M^2μ ) (G_t(C^(0))-G_ )K. (31) Proof. By Lemma 1, GtG_t is LGL_G-smooth. Therefore, for the gradient descent update (s+1)=(s)−η∇Gt((s)),C^(s+1)=C^(s)-η∇ G_t(C^(s)), (32) the descent lemma gives Gt((s+1))≤Gt((s))+⟨∇Gt((s)),(s+1)−(s)⟩+LG2‖(s+1)−(s)‖F2.G_t(C^(s+1))≤ G_t(C^(s))+ ∇ G_t(C^(s)),C^(s+1)-C^(s) + L_G2 \|C^(s+1)-C^(s) \|_F^2. (33) Substituting the update yields Gt((s+1))≤Gt((s))−η(1−LGη2)‖∇Gt((s))‖F2.G_t(C^(s+1))≤ G_t(C^(s))-η (1- L_Gη2 ) \|∇ G_t(C^(s)) \|_F^2. (34) For η≤1/LGη≤ 1/L_G, Gt((s+1))≤Gt((s))−η2‖∇Gt((s))‖F2.G_t(C^(s+1))≤ G_t(C^(s))- η2 \|∇ G_t(C^(s)) \|_F^2. (35) Summing Eq. (35) over s=0,…,K−1s=0,…,K-1 gives η2∑s=0K−1‖∇Gt((s))‖F2≤Gt((0))−Gt((K)). η2 _s=0^K-1 \|∇ G_t(C^(s)) \|_F^2≤ G_t(C^(0))-G_t(C^(K)). (36) Since GtG_t is lower bounded by GinfG_ , 1K∑s=0K−1‖∇Gt((s))‖F2≤2(Gt((0))−Ginf)ηK. 1K _s=0^K-1 \|∇ G_t(C^(s)) \|_F^2≤ 2 (G_t(C^(0))-G_ )η K. (37) Choosing η=1/LGη=1/L_G gives 1K∑s=0K−1‖∇Gt((s))‖F2≤2LG(Gt((0))−Ginf)K. 1K _s=0^K-1 \|∇ G_t(C^(s)) \|_F^2≤ 2L_G (G_t(C^(0))-G_ )K. (38) Using Lemma 2, LG≤Rt,k(β+M2μ),L_G≤ R_t,k (β+ M^2μ ), (39) we obtain 1K∑s=0K−1‖∇Gt((s))‖F2≤2Rt,k(β+M2μ)(Gt((0))−Ginf)K, 1K _s=0^K-1 \|∇ G_t(C^(s)) \|_F^2≤ 2R_t,k (β+ M^2μ ) (G_t(C^(0))-G_ )K, (40) which proves Eq. (31). ∎ Remark 1 (Scope of the Convergence Guarantee). Theorem 1 characterizes optimization of the shared non-vertex Bézier control prompts while the LLM parameters and task-specific anchors remain fixed. The guarantee concerns convergence to first-order stationarity of the expected smooth Tchebycheff objective rather than global optimality of the underlying non-convex problem. Importantly, the bound makes the LAPS-specific dependence explicit: Rt,kR_t,k captures the sensitivity induced by the Bézier parameterization, while M2/μM^2/μ reflects the additional curvature introduced by the smooth Tchebycheff objective. A.2 Bayesian Optimization for Positive Transfer Discovery After constructing the final anchored prompt space, LAPS performs Bayesian optimization (BO) independently for each task. On the TRACE benchmark, the search variable is =(λ1,…,λ8)∈Δ7,λj≥0,∑j=18λj=1. λ=( _1,…, _8)∈ ^7, _j≥ 0, _j=1^8 _j=1. (41) Given λ, the cubic Bézier simplex deterministically constructs the corresponding prompt T()Z_T( λ). Task-wise Search. We perform an independent procedure for each task jT_j. Thus, each task obtains its own coordinate j,T⋆ λ_j,T rather than sharing a single global coordinate. The task-specific search objective is j,T⋆=argmax∈Δ7Valj(), λ_j,T = _ λ∈ ^7Val_j( λ), (42) where Valj()Val_j( λ) denotes the validation score of T()Z_T( λ) on task jT_j, measured by its task-specific evaluation metric; the metric for each task is reported in Table 8. Initial Design. Each run starts from 1616 simplex coordinates: all eight task vertices, the simplex centroid center=(1/8,…,1/8) λ_center=(1/8,…,1/8), and seven interior coordinates generated using a Sobol sequence and mapped onto the simplex. Explicitly including all task vertices ensures that the original task-specific prompts are evaluated during the search. Candidate Evaluation. For each candidate λ, we construct T()Z_T( λ) and evaluate it using the task-specific metric on at most 500500 validation examples with sampling temperature 0.10.1. Each candidate is evaluated over three independent inference runs, from which we record the empirical mean and variance. The empirical mean serves as the Bayesian optimization response, while the empirical variance estimates candidate-dependent observation noise. Simplex-Aware Gaussian Process. We fit a Gaussian process (GP) to the validation score Valj()Val_j( λ). Before GP fitting, each simplex coordinate is transformed elementwise as ~==(λ1,…,λ8). λ= λ= ( _1,…, _8 ). (43) Since ∑jλj=1 _j _j=1, the transformed coordinates satisfy ‖~‖2=1\| λ\|_2=1, mapping the simplex to the positive orthant of the unit sphere. Euclidean distance between the square root coordinates is proportional to the Hellinger distance between the corresponding simplex coordinates. We use a Constant Kernel multiplied by an RBF kernel: K(,′)=cexp(−‖−′‖222r2),K( λ, λ )=c (- \| λ- λ \|_2^22r^2 ), (44) where c and r denote the kernel amplitude and length scale, respectively. We initialize c=1.0c=1.0 and r=0.5r=0.5, with optimization ranges c∈[0.1,10]c∈[0.1,10] and r∈[0.05,5.0]r∈[0.05,5.0]. Both hyperparameters are optimized by maximizing the GP log marginal likelihood. The empirical variance from repeated inference is provided to the GP as heteroscedastic observation noise. Expected Improvement. At each Bayesian optimization iteration, we generate 4,0964,096 candidate simplex coordinates. Half are sampled from a standard Dirichlet distribution for broad simplex coverage, one quarter from a sparse Dirichlet distribution to emphasize boundaries and vertices, and the remaining quarter around the current best coordinate for local refinement. Let mj()m_j( λ) and sj()s_j( λ) denote the GP predictive mean and standard deviation for task jT_j, and let uj+u_j^+ denote the best validation score observed so far. Candidates are ranked using Expected Improvement (Jones et al., 1998): EIj() _j( λ) =(mj()−uj+−ξ)Φ(z)+sj()ϕ(z), = (m_j( λ)-u_j^+-ξ ) (z)+s_j( λ)φ(z), (45) z z =mj()−uj+−ξsj(), = m_j( λ)-u_j^+-ξs_j( λ), (46) where Φ(⋅) (·) and ϕ(⋅)φ(·) denote the cumulative distribution function and probability density function of the standard Gaussian, respectively. We set ξ=0.05ξ=0.05 and EIj()=0EI_j( λ)=0 whenever sj()=0s_j( λ)=0. We select at most four candidates per iteration. Within each batch, selected coordinates must satisfy a minimum pairwise distance to avoid spending evaluations on nearly identical locations. Each task is allocated at most 100100 candidate evaluations, except MeetingBank, whose budget is 5050. Search terminates early if the maximum EI remains below 0.0050.005 for three consecutive iterations. Final Candidate. After Bayesian optimization terminates, we retain the three coordinates with the highest observed validation scores. If the vertex corresponding to the target task is not among them, it is added as an additional candidate, yielding three or four finalists. Each finalist is reevaluated on the complete validation set using eight fresh inference runs. The coordinate with the highest mean validation score is selected as j,T⋆ λ_j,T , and T(j,T⋆)Z_T( λ_j,T ) is used as the final soft prompt for task jT_j. Table 7: Comparison of trainable components and parameter counts. Method Replay Backbone Soft Prompt # Trainable Parameters (M) Qwen3-1.7B Qwen3-4B Qwen3-8B SFT ✗ ✓ ✗ 1,720.61,720.6 4,022.54,022.5 8,190.78,190.7 Replay ✓ ✓ ✗ 1,720.61,720.6 4,022.54,022.5 8,190.78,190.7 LwF ✗ ✓ ✗ 1,720.61,720.6 4,022.54,022.5 8,190.78,190.7 OPR-RU ✓ ✓ ✗ 1,720.61,720.6 4,022.54,022.5 8,190.78,190.7 OPR-SC ✓ ✓ ✗ 1,720.61,720.6 4,022.54,022.5 8,190.78,190.7 ProgPrompt ✗ ✗ ✓ 0.50.5 0.70.7 1.01.0 ProMoT ✗ ✓ ✓ 1,721.11,721.1 4,023.14,023.1 8,191.78,191.7 LAPS (Ours) ✓ ✓ ✓ 1,727.91,727.9 4,031.74,031.7 8,205.58,205.5 A.3 Method Characteristics and Parameter Efficiency Table 7 compares the trainable components and parameter counts of different methods. Although LAPS jointly optimizes the backbone and soft prompts, it introduces only 7.37.3M, 9.29.2M, and 14.814.8M additional parameters for Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively. These additions account for only approximately 0.43%0.43\%, 0.23%0.23\%, and 0.18%0.18\% of the corresponding backbone sizes. A.4 Dataset Details We conduct experiments on the eight tasks from TRACE, covering diverse domains, languages, and evaluation metrics. Samples exceeding the maximum sequence length of 2,048 tokens are removed. Table 8 summarizes the statistics of each task. We use the official task-specific evaluation metrics from TRACE, including Accuracy, ROUGE-L, Edit Similarity, and SARI (Xu et al., 2016). Edit Similarity is computed from normalized Levenshtein distance (Levenshtein, 1966). Table 8: Statistics of the eight continual adaptation tasks used in our experiments. Dataset Source Average Length Metric Language Size C-STANCE Social Media 109 Accuracy Chinese 5,000 FOMC Finance 76 Accuracy English 5,000 MeetingBank Meeting 3,550 ROUGE-L English 5,000 Py150 GitHub 814 Edit Similarity English 5,000 ScienceQA Science 285 Accuracy English 5,000 NumGLUE-cm Math 50 Accuracy English 5,000 NumGLUE-ds Math 39 Accuracy English 5,000 20Minuten News 739 SARI German 5,000 The benchmark contains both classification and generation tasks and spans Chinese, English, and German. This diversity allows us to evaluate continual adaptation under heterogeneous task distributions rather than within a single domain or output format. Table 9: Task-specific simplex coordinates selected by Bayesian optimization. Model Target Task 1 λ_1 2 λ_2 3 λ_3 4 λ_4 5 λ_5 6 λ_6 7 λ_7 8 λ_8 Qwen3-1.7B C-STANCE 0.3313 0.2977 0.0001 0.0134 0.0646 0.2884 0.0000 0.0045 FOMC 0.2173 0.3009 0.0164 0.0002 0.4052 0.0170 0.0000 0.0430 MeetingBank 0.0000 0.0011 0.9967 0.0000 0.0011 0.0000 0.0003 0.0006 Py150 0.0000 0.0000 0.0000 0.9962 0.0004 0.0014 0.0000 0.0015 ScienceQA 0.0112 0.0000 0.0000 0.0000 0.9445 0.0012 0.0007 0.0424 NumGLUE-cm 0.0004 0.0260 0.0016 0.0059 0.0087 0.8868 0.0636 0.0070 NumGLUE-ds 0.0002 0.0042 0.0000 0.0168 0.0317 0.0000 0.9471 0.0001 20Minuten 0.1403 0.2323 0.0019 0.0368 0.0010 0.1004 0.0705 0.4168 Qwen3-4B C-STANCE 0.0767 0.0797 0.0957 0.4814 0.0292 0.0118 0.1497 0.0759 FOMC 0.0000 0.9310 0.0004 0.0000 0.0001 0.0084 0.0575 0.0027 MeetingBank 0.0249 0.0000 0.9634 0.0000 0.0000 0.0000 0.0032 0.0085 Py150 0.0607 0.0001 0.0379 0.8538 0.0054 0.0408 0.0000 0.0012 ScienceQA 0.0220 0.0000 0.0002 0.0048 0.9268 0.0079 0.0043 0.0339 NumGLUE-cm 0.1174 0.2825 0.0670 0.0016 0.0086 0.5036 0.0063 0.0129 NumGLUE-ds 0.0002 0.0345 0.0000 0.0160 0.0001 0.0025 0.9466 0.0002 20Minuten 0.1589 0.2461 0.1227 0.0508 0.0356 0.0511 0.1885 0.1464 Qwen3-8B C-STANCE 0.0660 0.2528 0.1278 0.0516 0.4995 0.0006 0.0010 0.0006 FOMC 0.0002 0.9137 0.0004 0.0002 0.0000 0.0465 0.0000 0.0389 MeetingBank 0.0003 0.0002 0.9884 0.0000 0.0002 0.0000 0.0080 0.0029 Py150 0.0372 0.0000 0.0034 0.9253 0.0198 0.0085 0.0041 0.0017 ScienceQA 0.0001 0.0005 0.0000 0.0323 0.9030 0.0639 0.0002 0.0000 NumGLUE-cm 0.0830 0.0588 0.0211 0.1582 0.0070 0.3813 0.2833 0.0073 NumGLUE-ds 0.0156 0.0054 0.0223 0.0002 0.0003 0.0001 0.9208 0.0353 20Minuten 0.0009 0.0254 0.0000 0.0012 0.1002 0.0000 0.1368 0.7355 A.5 Additional Experiments This subsection provides complementary analyses of LAPS, including search-selected task coordinates and convergence, robustness to task order, continual adaptation dynamics, and the effect of learnable control prompts on space geometry. Introducing learnable control prompts reshapes this fixed interpolation geometry and yields larger cumulative gains from space search on both model scales, supporting their role in creating more useful intermediate prompts. A.5.1 Task-Specific Coordinates Selected by BO As shown in Table 9, we further report the simplex coordinates selected by Bayesian optimization for every task. The selected coordinates reveal two characteristic behaviors. For several tasks, including MeetingBank, Py150, ScienceQA, and NumGLUE-ds, search-selected coordinates remains close to the corresponding task vertex across model scales. In contrast, C-STANCE, NumGLUE-cm, and 20Minuten often select more distributed interior coordinates, indicating that their high-performing prompts draw more strongly on other task directions. The selected coordinates also vary substantially across model scales, supporting task- and model-specific space search rather than a single fixed combination. Figure 5: Visualization of the learned anchored prompt space with and without non-vertex control prompts on Qwen3-1.7B and Qwen3-4B. The values in green indicate the cumulative performance improvement over the eight tasks after space search. Table 10: Continual learning performance under the reverse task order (20Minuten → NumGLUE-ds → NumGLUE-cm → ScienceQA → Py150 → MeetingBank → FOMC → C-STANCE). Method 20Minuten NG-ds NG-cm ScienceQA Py150 MeetingBank FOMC C-STANCE Average ↑ BWT ↑ Qwen3-1.7B SFT 39.139.1 64.264.2 54.254.2 18.218.2 58.658.6 43.843.8 52.352.3 50.850.8 47.647.6 −17.1-17.1 Replay 39.539.5 59.059.0 59.359.3 68.868.8 58.858.8 56.056.0 62.362.3 51.451.4 56.956.9 −7.7-7.7 ProgPrompt 38.538.5 49.249.2 37.737.7 70.970.9 50.550.5 47.747.7 60.660.6 50.150.1 50.650.6 0.00.0 ProMoT 39.739.7 61.461.4 30.630.6 75.675.6 59.459.4 61.961.9 64.764.7 52.252.2 55.755.7 −4.2-4.2 LAPS (Ours) 39.839.8 56.356.3 50.950.9 76.576.5 60.360.3 61.761.7 62.462.4 51.351.3 57.457.4 −2.7-2.7 Qwen3-4B SFT 38.638.6 71.271.2 65.765.7 9.89.8 63.163.1 61.461.4 53.953.9 56.356.3 52.552.5 −16.3-16.3 Replay 39.139.1 68.768.7 64.464.4 62.362.3 64.364.3 63.363.3 60.160.1 56.856.8 59.959.9 −7.7-7.7 ProgPrompt 39.439.4 52.352.3 56.556.5 81.681.6 57.457.4 49.449.4 61.461.4 54.854.8 56.656.6 0.00.0 ProMoT 40.240.2 73.273.2 69.169.1 40.740.7 63.663.6 65.565.5 67.467.4 55.355.3 59.459.4 −8.3-8.3 LAPS (Ours) 40.640.6 74.674.6 62.262.2 62.762.7 62.662.6 65.165.1 71.771.7 56.756.7 62.062.0 −5.7-5.7 A.5.2 Effect of Learnable Control Prompts on Space Geometry To further examine the role of non-vertex control prompts, we compare the learned anchored space with and without them on Qwen3-1.7B and Qwen3-4B, as shown in Fig. 5. Without control prompts, the space reduces to the convex simplex determined entirely by the task-specific prompts. In this case, anchored prompt space search is restricted to the fixed interpolation geometry spanned by the task anchors. A.5.3 Robustness to Task Order We further evaluate the robustness of LAPS under the reverse TRACE task order. As shown in Table 10, reversing the task sequence changes the difficulty of individual tasks and leads to noticeable performance shifts across all methods. Nevertheless, LAPS achieves the highest Average performance on both model scales, reaching 57.4%57.4\% on Qwen3-1.7B and 62.0%62.0\% on Qwen3-4B. It also obtains BWT scores of −2.7%-2.7\% and −5.7%-5.7\%, respectively, substantially reducing forgetting compared with the parameter-updating baselines SFT, Replay, and ProMoT. Although ProgPrompt achieves a BWT of 0.0%0.0\% by keeping the backbone and previously learned prompts fixed, its Average performance is considerably lower, at 50.6%50.6\% on Qwen3-1.7B and 56.6%56.6\% on Qwen3-4B. These results show that LAPS consistently provides a stronger balance between adaptation and retention across both model scales, indicating that its advantages are robust to changes in task order. A.5.4 Convergence of Bayesian Optimization Fig. 6 shows the convergence behavior of Bayesian optimization across the eight tasks and three model scales. In most cases, the best validation performance improves rapidly within the first few evaluations and then gradually stabilizes, indicating that effective regions of the learned anchored prompt space can be identified with a relatively small search budget. Figure 6: Bayesian optimization convergence across eight TRACE tasks and three Qwen3 model scales. Curves report the best validation performance observed so far over successive evaluations. Figure 7: Task-wise continual adaptation trajectories on TRACE with Qwen3-4B. Figure 8: Task-wise continual adaptation trajectories on TRACE with Qwen3-8B. A.5.5 Additional Continual Adaptation Dynamics The main text reports task-wise continual adaptation trajectories on Qwen3-1.7B. Figures 7 and 8 extend this analysis to Qwen3-4B and Qwen3-8B. Across both model scales, methods that update model parameters continue to exhibit noticeable fluctuations on previously learned tasks, whereas LAPS generally maintains more stable historical performance while preserving strong adaptation to newly introduced tasks. A.5.6 Qualitative Analysis of Space Search To further understand the effect of space search, we analyze qualitative examples on Qwen3-8B by comparing the outputs obtained from all eight task vertices with the search-selected prompt. The examples illustrate cases where searching the learned anchored prompt space yields behaviors that cannot be obtained from the task vertices alone. The MeetingBank example shows a different form of improvement. The original MeetingBank vertex already contains the relevant information, but substantially over-generates the transcript, resulting in a low ROUGE-L score. In contrast, the search-selected point remains close to the MeetingBank vertex while producing a much more concise summary, increasing the per-example ROUGE-L from 8.258.25 at the MeetingBank vertex to 45.2845.28. This suggests that the anchored space learning can refine not only task correctness but also the form in which task knowledge is expressed. For the FOMC task, all eight task vertices predict the FOMC example as neutral, whereas the search-selected interior point gives the correct hawkish prediction. Although the selected coordinate remains close to the FOMC vertex, its small contributions from other task directions are sufficient to induce a different model behavior. This illustrates that space search can identify useful prompts that are unavailable through vertex selection alone. Example for MeetingBank Query: Write a summary of the meeting transcript concerning access to low cost veterinary care in Boston. Reference: Order for a hearing regarding access to low cost veterinary care. Remains in the Committee on City Services and Innovation Technology. Responses: Prompt Generated Summary ROUGE-L C-STANCE vertex Councilor Braden proposed a hearing to address access to low cost veterinary care for residents in public and subsidized affordable housing. The response also discusses unequal access across neighborhoods and a One Health approach. 21.74 FOMC vertex The transcript discusses difficulty accessing low cost veterinary care, particularly for residents in Brighton, and calls for collaboration between city agencies and veterinary organizations. 16.16 MeetingBank vertex Order for a hearing regarding access to low cost veterinary care. The response then reproduces a long portion of the meeting discussion, including details on housing developments, veterinary providers, public health, and the One Health approach. 8.25 Py150 vertex Councilor Braden called for a hearing on the difficulty faced by low income residents in accessing affordable veterinary care and proposed considering a holistic One Health approach. 20.95 ScienceQA vertex The hearing will address the problem of low income Boston residents lacking access to affordable veterinary care and consider partnerships with veterinary services. 28.57 NumGLUE-cm vertex A hearing was called to address difficulty accessing affordable veterinary care in Boston, especially in Brighton, and to consider a One Health approach linking human and animal health. 23.26 NumGLUE-ds vertex The hearing addresses unequal access to low cost veterinary care for residents in public and subsidized housing and proposes a One Health framework for city services. 20.56 20Minuten vertex A Boston City Council hearing was held on access to low cost veterinary care. The response emphasizes unequal access and the potential use of a One Health approach. 24.24 Search-selected Order for a hearing regarding Access to Low Cost Veterinary Care. The Chair recognized Councilor Breadon. Councilor Breadon offered the following: Order for a hearing regarding access to low cost veterinary care. 45.28 BO coordinate: ⋆=[0.0003, 0.0002, 0.9884, 0.0000, 0.0002, 0.0000, 0.0080, 0.0029]. λ =[0.0003,\,0.0002,\,0.9884,\,0.0000,\,0.0002,\,0.0000,\,0.0080,\,0.0029]. Example for FOMC Query: What is the monetary policy stance for the following text? A. dovish, B. hawkish, C. neutral. Text: Of course, the basic problem in tackling the inflation of the 1970s is that we have one observation and many competing theories. Reference: B (hawkish). Predictions: Prompt Prediction Correct C-STANCE vertex C (neutral) No FOMC vertex C (neutral) No MeetingBank vertex C (neutral) No Py150 vertex C (neutral) No ScienceQA vertex C (neutral) No NumGLUE-cm vertex C (neutral) No NumGLUE-ds vertex C (neutral) No 20Minuten vertex C (neutral) No Search-selected B (hawkish) Yes BO coordinate: ⋆=[0.0002, 0.9137, 0.0004, 0.0002, 0.0000, 0.0465, 0.0000, 0.0389]. λ =[0.0002,\,0.9137,\,0.0004,\,0.0002,\,0.0000,\,0.0465,\,0.0000,\,0.0389].