Paper deep dive
Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving
Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 7/5/2026, 4:20:59 AM
Summary
The paper proposes Rollout-Retrieval Lifelong Policy Learning (R2LPL), a framework designed to enable autonomous driving policies to improve continually by learning from their own mistakes. Unlike traditional imitation or reinforcement learning, R2LPL uses a 'rollout-retrieval' (R2) mechanism to identify recoverable mistakes in closed-loop scenarios and convert them into compact, supervised corrective knowledge. To prevent catastrophic forgetting, the framework employs 'lifelong policy learning' (LPL) through a knowledge replay mechanism that combines newly retrieved corrective targets with historical corrections. The method was validated on the nuPlan benchmark, demonstrating state-of-the-art performance on challenging long-tail scenarios.
Entities (7)
Relation Signals (4)
R2LPL → contains → Rollout-Retrieval (R2)
confidence 100% · R2LPL first uses closed-loop rollouts... R2LPL performs rollout-retrieval (R2)
R2LPL → contains → Lifelong Policy Learning (LPL)
confidence 100% · R2LPL further performs lifelong policy learning (LPL)
R2LPL → evaluatedon → nuPlan
confidence 100% · We evaluate R2LPL on large-scale closed-loop nuPlan benchmarks.
Rollout-Retrieval (R2) → converts → mistakes
confidence 90% · R2 selectively mines mistake-related but recoverable states and retrieves feasible corrective targets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations. However, most learning-based policies are trained or fine-tuned on expert demonstrations and then rely largely on generalization to handle challenging closed-loop scenarios, lacking an explicit mechanism to correct and retain the mistakes exposed in these scenarios. This paper studies autonomous driving policy improvement from a lifelong learning perspective: Can a pretrained policy improve continually by accumulating corrective knowledge derived from its own mistakes, while retaining previously acquired driving competence? To answer this question, we propose Rollout-Retrieval Lifelong Policy Learning (R$^2$LPL), a policy learning framework that retrieves corrective targets from recoverable policy-induced mistakes and retains the resulting knowledge through lifelong policy learning. R^2LPL addresses a key bottleneck in continual policy improvement: closed-loop mistakes reveal where the policy is weak, but do not directly specify what the policy should learn. By filtering recoverable mistake-related states and retrieving feasible corrective targets, R$^2$LPL turns sparse failure evidence into compact supervised knowledge for stable and sample-efficient policy improvement. We evaluate R$^2$LPL on large-scale closed-loop nuPlan benchmarks. With only a few rollout and continual-learning cycles, R$^2$LPL elevates a learning-based planner with moderate initial performance to state-of-the-art performance across the evaluated benchmarks, especially on the challenging and long-tail Test14-hard split. These results demonstrate the effectiveness of R$^2$LPL in converting recoverable closed-loop mistakes into corrective knowledge for sustained policy improvement.
Tags
Links
- Source: https://arxiv.org/abs/2606.30537v1
- Canonical: https://arxiv.org/abs/2606.30537v1
Trouble viewing inline? Open PDF directly →
Full Text
87,915 characters extracted from source content.
Expand or collapse full text
Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong This work was supported in part by the National Natural Science Foundation of China under Grant 52372405, and in part by the National Key R&D Program of China under Grant 2022ZD0115503. Corresponding authors: Chao Lu and Jianwei Gong. Cheng Gong, Haoyang Wang, Chao Lu, and Jianwei Gong are with the School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China (e-mails: engibacter@gmail.com; haoyaw7@gmail.com; chaolu@bit.edu.cn; gongjianwei@bit.edu.cn). Zirui Li is with the School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798 (e-mail: zirui.li@ntu.edu.sg). Abstract Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations. However, most learning-based policies are trained or fine-tuned on expert demonstrations and then rely largely on generalization to handle challenging closed-loop scenarios, lacking an explicit mechanism to correct and retain the mistakes exposed in these scenarios. This paper studies autonomous driving policy improvement from a lifelong learning perspective: Can a pretrained policy improve continually by accumulating corrective knowledge derived from its own mistakes, while retaining previously acquired driving competence? To answer this question, we propose Rollout-Retrieval Lifelong Policy Learning (R2LPL), a new policy learning framework that retrieves corrective targets from recoverable policy-induced mistakes and retains the resulting knowledge through lifelong policy learning. R2LPL addresses a key bottleneck in continual policy improvement: closed-loop mistakes reveal where the policy is weak, but do not directly specify what the policy should learn. By filtering recoverable mistake-related states and retrieving feasible corrective targets, R2LPL turns sparse failure evidence into compact supervised knowledge for stable and sample-efficient policy improvement. We evaluate R2LPL on large-scale closed-loop nuPlan benchmarks. With only a few rollout and continual-learning cycles, R2LPL elevates a learning-based planner with moderate initial performance to state-of-the-art performance across the evaluated benchmarks, especially on the challenging and long-tail Test14-hard split. These results demonstrate the effectiveness of R2LPL in converting recoverable closed-loop mistakes into corrective knowledge for sustained policy improvement. Code available at: https://github.com/Engibacter/R2LPL. I Introduction Autonomous vehicles are expected to operate in open and continually evolving environments, where rare combinations of road geometry, traffic interactions, and agent behaviors inevitably give rise to long-tail situations [17, 10]. Recent advances in planning-oriented and end-to-end autonomous driving have substantially improved closed-loop performance [14, 8, 4] through increasingly powerful policy architectures [6, 38, 22], generative models [26, 55, 42], and large-scale foundation models [59, 60, 50]. Despite these advances, mistakes remain inevitable. Even highly capable driving policies may encounter situations in which their learned behaviors are insufficient, leading to suboptimal decisions or closed-loop failures [33, 12]. These failures may stem from rare interactions, imperfect generalization, or the accumulation of policy-induced deviations over closed-loop execution. This raises a fundamental question: if mistakes are inevitable, can a deployed driving policy learn from them and continually improve itself over time? Most existing learning-based planning policies address this challenge by exploiting richer information from pre-collected expert data. Imitation-based planners improve scene representations, model architectures, interaction reasoning, and training strategies to obtain stronger generalization from expert demonstrations [15, 7, 6, 5, 22]. Generative planners further model multimodal trajectory distributions using diffusion- or flow-based formulations [55, 39, 53]. Vision-language-action and language-augmented driving models introduce high-level reasoning and broader semantic priors to improve adaptability in complex scenarios [59, 60, 50, 57]. These approaches have significantly advanced learning-based planning through increasingly powerful policy models and stronger generalization from pre-collected expert data. However, they primarily focus on better utilizing existing knowledge, offering limited mechanisms to incorporate new knowledge revealed by policy failures and difficult scenarios. A promising direction is to continuously acquire new knowledge from policy interactions and exploit the resulting feedback to further improve policy behavior. Reinforcement learning and preference-based optimization can improve closed-loop behavior beyond expert demonstration, but reward design, sparse safety feedback, and exploration risk remain major challenges for safety-critical driving [46, 51, 35, 40, 11]. Dataset aggregation methods such as DAgger [33] improve policies by rolling out the current learner and querying an expert on visited states. However, obtaining expert supervision for arbitrary policy-induced states is often prohibitively expensive in large-scale driving scenarios. Recent closed-loop supervised fine-tuning methods, such as CAT-K [54] and RoaD [12], alleviate this burden by using logged ground-truth trajectories to guide closed-loop data generation and then fine-tuning driving policies on the recovered rollout experiences. Nevertheless, the knowledge recovered from closed-loop rollout is often heterogeneous in quality and informativeness. As policies deviate from expert states in rollout, some recovered knowledge provide valuable corrective information, whereas others may introduce biased or misleading updates. Consequently, effectively extracting useful corrective knowledge from closed-loop rollout remains an open challenge. We study this problem from a lifelong policy learning perspective. As shown in Fig. 1, in contrast to imitation learning, which relies on stronger generalization from fixed expert demonstrations, and reinforcement learning, which improves policies through reward-driven trial and error, our goal is to enable a pretrained planner to improve through repeated correction of its own rollout-induced mistakes. Unlike dataset aggregation, this process does not assume that a queryable expert can provide valid labels for arbitrary learner-induced states. Instead, it treats closed-loop rollouts as a stream of policy-dependent experience, from which the learner must selectively retrieve recoverable mistakes and convert them into reliable corrective knowledge. This perspective raises two central challenges. The first is mistake-to-knowledge conversion: how to accurately and efficiently extract reliable corrective knowledge from heterogeneous rollout experience. Potential policy mistakes may lead to failures, risks, and behavioral conflicts events in close-loop rollout. Yet such outcomes do not directly reveal the responsible mistakes or the supervision targets that should be retrieved. Since only part of the induced experience contains actionable corrective information, the learner must identify informative mistake-related states and retrieve valid targets while avoiding ambiguous or noisy supervision. The second is non-forgetting policy improvement: how to improve over repeated correction cycles without forgetting previous corrections. Once a policy is updated, it induces a new rollout distribution and may expose new mistakes. The challenge is to learn from these newly revealed mistakes while retaining the knowledge that prevents old mistakes from reappearing. Figure 1: Comparison of policy learning paradigms for autonomous driving. Existing paradigms improve policies from fixed expert demonstrations, reward/advantage signals, or expert/ground-truth guided supervision on policy-induced rollouts. In contrast, the proposed R2LPL converts recoverable policy-induced mistakes into corrective targets and accumulates the resulting knowledge through replay memory, providing a lifelong policy improvement route driven by the policy’s own mistakes. To tackle these challenges, this paper proposes R2LPL, short for Rollout-Retrieval Lifelong Policy Learning, as a paradigm for lifelong policy improvement. R2LPL first uses closed-loop rollouts to collect policy-induced experience associated with failures, risks, and conflicts. To address mistake-to-knowledge conversion, R2LPL performs rollout-retrieval (R2): candidate actions are evaluated at selected policy-induced states, feasible targets are retrieved from states that still admit meaningful correction, and misleading targets are avoided when no feasible correction exists. To address non-forgetting policy improvement, R2LPL further performs lifelong policy learning (LPL), which updates the planner with newly retrieved R2 knowledge while replaying previously learned corrections. Through repeated rollout-continual-learning (ROCL) cycles, R2LPL enables a pretrained planner to continuously improve from its own mistakes without relying on reward-driven exploration or online expert querying. The main contributions of this paper are: • We introduce R2LPL, a rollout-retrieval lifelong policy learning framework for autonomous driving. It formulates policy learning and improvement as a process of learning from the policy’s own mistakes, and organizes this process into mistake-related state mining, recoverability-aware corrective-knowledge construction, and lifelong policy update. • We propose R2, a rollout-retrieval mechanism that converts closed-loop mistakes into corrective supervision. Instead of directly using all rollout experiences, R2 selectively mines mistake-related but recoverable states and retrieves feasible corrective targets from candidate actions. • We formulate lifelong policy learning for planning as incremental accumulation of corrective knowledge. Instead of performing one-shot supervised fine-tuning on newly retrieved corrections, R2LPL updates the policy with both new R2 knowledge and replayed historical corrections, enabling multi-round improvement while reducing forgetting of previously corrective knowledge. • We validate R2LPL on large-scale closed-loop nuPlan benchmarks, showing that a learning-based planner with moderate initial performance can be improved to state-of-the-art performance within only a few ROCL rounds, especially on the hard long-tail Test14-hard split. I Related Work I-A Planning Policy Learning for Autonomous Driving In planning policy learning, autonomous-driving planners are commonly formulated as parameterized policies that map scene observations to future ego trajectories or distributions over candidate trajectories. Early workers utilize transformer-based, vectorized, and interaction-aware modeling to encode driving scene, and regress or classify to the logged ego future trajectory [3, 16]. GameFormer [15] formulates interactive prediction and planning with a game-theoretic transformer, while PLUTO [6] advances imitation-based planning through strong scene representations and data augmentation. Another line of work formulates planning as selection or scoring over multiple candidate actions. VADv2 [5] formulates end-to-end planning as a probabilistic distribution over a fixed set of trajectory anchors. Hydra-MDP [22] uses multi-target rule-distillation to learn multi-modal planning targets. Recent mixture-of-experts planning improves generalization by composing specialized motion-planning behaviors [37]. Compared with black-box trajectory regression, probabilistic or score-based planners expose alternative action hypotheses and their relative preferences, which provides a structured interface for analyzing and improving policy decisions at closed-loop induced states. Recent studies further improve learned planners through generative modeling and language-augmented policy architectures. Generative planners, including DiffusionDrive [26], Diffusion Planner [55], Flow Planner [39], Diffusion Forcing Planner [53], and MISTY [47], model multimodal and uncertain driving behaviors with diffusion, flow matching, or other distributional trajectory-generation mechanisms. Meanwhile, vision-language-action and language-augmented models such as OpenDriveVLA [59], AutoVLA [60], and Reasoning-VLA [50] introduce semantic reasoning and cross-scenario priors into end-to-end driving policies. These methods demonstrate the importance of scaling policy representations, trajectory generation, and reasoning capability for autonomous driving. However, stronger policy architectures do not by themselves provide a mechanism for stable and continual policy improvement from closed-loop evaluation experience. In particular, the increasing complexity of generative objectives, guidance terms, large reasoning models, and end-to-end action generation makes targeted policy refinement sensitive to data distribution, optimization stability, inference efficiency, and safety verification. I-B Closed-loop Policy Improvement Closed-loop policy improvement aims to refine a learned policy after initial training, especially when rollout execution exposes states that are not well handled by the pretrained policy. Classical dataset aggregation methods address this issue by learning from learner-induced states: DAgger [33] rolls out the learner, queries an expert at visited states, and aggregates newly labeled data, while DART [20], SafeDAgger [52], HG-DAgger [18], and EnsembleDAgger [30] improve robustness, query efficiency, or safety through noise injection, safety classifiers, human intervention, or uncertainty estimation. These methods show the value of policy-induced states, but rely on expert or intervention signals that can provide valid supervision during rollout, which is difficult to obtain for large-scale autonomous-driving scenarios. Recent driving-oriented methods adapt closed-loop improvement to modern learned planners: CAT-K [54] performs closed-loop supervised fine-tuning for tokenized traffic models, and RoaD [12] uses expert-guided rollouts as demonstrations for supervised fine-tuning. While these works show that rollout experience can reveal weaknesses beyond fixed expert logs, such experience is not uniformly informative: policy-induced states may deviate from logged behaviors, differ in recoverability, or provide noisy correction signals. Moreover, each policy update changes the future rollout distribution, so policy improvement requires more than one-shot adaptation. It requires repeatedly acquiring reliable corrective knowledge while retaining previously learned corrections. Reinforcement learning and preference-based optimization provide another route for policy improvement. Human-guided RL incorporates human knowledge to improve navigation behavior [46]; CarPlanner [51] applies large-scale reinforcement learning to autoregressive trajectory planning; DriveDPO [35] uses safety-oriented direct preference optimization for end-to-end driving; and Plan-R1 [40] further represents trajectories as motion tokens and applies rule-based GRPO to align planning behavior with driving objectives. Recent diffusion-based planners, including DIVER [36] and Hyper Diffusion Planner [56], further integrate reinforcement learning or reward optimization with generative trajectory modeling.These methods move beyond expert-log imitation and optimize policies toward closed-loop driving objectives. However, autonomous-driving rewards are sparse, multi-objective, and safety-critical, making exploration-intensive updates costly or unstable. Together with closed-loop supervised fine-tuning methods, these studies show that rollout experience is valuable for improving learned driving policies, but also highlight a central challenge: how to convert policy-induced experience into reliable and sustained policy improvement without depending on dense expert annotation or unstable exploration. I-C Lifelong Policy Learning Lifelong learning, often referred to as continual learning, studies how a model can acquire knowledge from sequential or incremental data while mitigating catastrophic forgetting [41, 43]. Representative strategies include regularization-based methods [19], distillation-based methods [23], replay-based methods [32, 28, 1], and architecture- or parameter-isolation methods [49, 29]. With large pretrained models, recent studies further explore prompt- and parameter-efficient continual learning [44, 45, 25, 58]. These methods mainly address how to preserve past knowledge while adapting to a new sequence of given tasks or data distributions. Policy learning poses an additional difficulty: the data distribution is induced by the policy itself, and each policy update can change the states that will be encountered in future rollouts. Moreover, policy-induced states usually do not come with explicit ground-truth actions, so useful supervision must first be constructed from closed-loop experience. Therefore, lifelong policy learning requires not only non-forgetting optimization, but also a mechanism for acquiring reliable corrective knowledge from the evolving behavior of the policy. Continual learning has also been explored in autonomous driving, including perception, prediction, and decision-making modules [24]. Prediction-oriented studies investigate case-level forgetting, dynamic expansion, and task-free identification for driving scene [21, 27]. These works extend continual learning to driving-specific data streams, but still largely follow the conventional setting where supervision from incremental data or task changes are given. Lifelong policy learning differs from conventional continual learning because the policy is updated through data that is partly induced by its own closed-loop behavior: its decisions affect future state visitation, and the resulting states may not come with explicit supervision. Existing studies have explored continual policy learning from different angles, including dynamic confidence-aware reinforcement learning for autonomous driving [2], lifelong skill preservation and recombination in robot manipulation [31], life-long policy learning for path tracking control [13], and human-guided continual learning for personalized driving decisions [48]. These works show that policies can be improved over time through interaction, accumulated knowledge, or external feedback. However, they mainly focus on policy update or adaptation, while mistake-driven acquisition of corrective knowledge remains less explored. In closed-loop planning, failures do not directly reveal responsible decisions, recoverability, or corrective targets. Therefore, lifelong policy learning for planning requires not only preserving previous knowledge, but also converting mistake-related experience into reliable supervision for avoiding repeated failures. I Problem Formulation I-A Planning Policy Learning for Autonomous Driving We consider autonomous-driving planning as trajectory-level policy learning. At time t, the planner observes a driving state st=htego,htagent,mt,ct,rt,s_t=\h_t^ego,h_t^agent,m_t,c_t,r_t\, (1) where htegoh_t^ego and htagenth_t^agent denote ego and agent histories, mtm_t denotes map and route geometry, ctc_t denotes traffic-control context, and rtr_t denotes the navigation goal. A learned planner maps sts_t to a future ego trajectory τt _t or, more generally, to a distribution over candidate trajectories. Because target retrieval requires candidate actions that can be explicitly searched, scored, and replayed as supervision, this paper focuses on score-based policies with a finite set of trajectory anchors as action space, similar to [5, 35]. Let =a1,a2,…,aKA=\a^1,a^2,…,a^K\ (2) be a finite anchor library, where each ai=(xℓi,yℓi,ψℓi)ℓ=1Ha^i=\(x_ ^i,y_ ^i, _ ^i)\_ =1^H is a future ego trajectory over horizon H. Given scene features ϕ(st)φ(s_t), the planner computes a trajectory score zθi(t)=fθ(ϕ(st),ai),z_θ^i(t)=f_θ(φ(s_t),a^i), (3) where θ denotes the policy parameters and zθi(t)z_θ^i(t) is the logit associated with action aia^i at state sts_t. The scores induce a categorical trajectory policy through softmax normalization: πθ(ai|st)=exp(zθi(t))∑j=1Kexp(zθj(t)). _θ(a^i|s_t)= (z_θ^i(t)) _j=1^K (z_θ^j(t)). (4) The executed action can be obtained by maximum-score selection, sampling, or downstream rule-based selection among top candidates. Rather than treating the expert future only as a continuous regression target, the score-based planner is trained with sparse anchor-score supervision. For a driving state s, let (s)=(i,yi)∣i∈s,yi>0Y(s)=\(i,y_i) i _s,y_i>0\ (5) denote the supervised anchor-score set, where s⊆1,…,KA_s 1,…,K is the subset of supervised anchor indices and yiy_i is the non-negative target score assigned to anchor aia^i. Depending on the training source, sA_s and yiy_i can be obtained from scoring, rollout retrieval, or candidate sampling. The best supervised anchor is i∗=argmax(i,yi)∈(s)yi.i^*= (i,y_i) (s)y_i. (6) The normalized target preference over the supervised subset is defined as qiY(s)=yi∑j∈syj,i∈s.q_i^Y(s)= y_i _j sy_j, i s. (7) Correspondingly, we restrict the policy distribution to the same supervised subset: qθi(s)=exp(zθi(s)/ϵ)∑j∈sexp(zθj(s)/ϵ),i∈s,q_θ^i(s)= (z_θ^i(s)/ε) _j s (zθ^j(s)/ε), i _s, (8) where ϵε is a temperature parameter. This restricted distribution is used only for training efficiency with sparse supervision loss. During inference, the planner scores the full anchor library. We define the anchor-score planning loss as ℒAS(s,;θ)=−logqθi∗(s)+λKL∑i∈sqiY(s)logqiY(s)qθi(s), _AS(s,Y;θ)=- qθ^i^*(s)\ + _KL _i sq_i^Y(s) q_i^Y(s)qθ^i(s), (9) where λKL _KL balances the best-anchor classification term and the preference-matching term. The first term pulls the policy toward the best supervised anchor, while the second term preserves the relative preference among multiple positive anchors. Standard pretraining can then be written as score-supervised planning over expert states, E=(stE,tE)t=1NE,D_E=\(s_t^E,Y_t^E)\_t=1^N_E, (10) where stEs_t^E is an expert-log state and tEY_t^E is the corresponding sparse anchor-score supervision. The initial policy is obtained by: θ=argminθ(stE,tE)∼E[ℒAS(stE,tE;θ)]θ= _(s_t^E,Yt^E) _E [L_AS (s_t^E,Y_t^E;θ ) ] (11) During inference, the planner can score the full anchor library, whereas training may supervise only selected anchors for efficiency. This score-based formulation is important for rollout retrieval: the finite candidate action space exposes alternative decisions and allows a retrieved corrective target to be represented as replayable supervised knowledge at an arbitrary policy-induced state. I-B Lifelong Policy Learning with Knowledge Replay Lifelong learning considers a sequence of learning tasks 1,…,K\D_1,…,D_K\, where kD_k contains the supervised knowledge newly available at task t. Let x denote a training sample, ℓ(x;θ) (x;θ) its task loss under model parameters θ, and θk−1θ^k-1 and θkθ^k the model parameters before and after learning task k, respectively. The objective is to learn the new knowledge in kD_k without overwriting useful knowledge acquired from 1,…,k−1D_1,…,D_k-1. In policy learning, the sequential tasks are not necessarily manually predefined. Instead, each task may correspond to a batch of incremental knowledge acquired from the current policy’s closed-loop behavior. After the policy is updated, its future rollout distribution changes, and new weaknesses may be exposed. Thus, the knowledge available at each round is policy-dependent and can only be acquired progressively. A trivial alternative would be to store all knowledge collected from previous rounds and repeatedly train on their union. However, this corresponds to an unbounded joint-training setting that assumes unlimited memory and increasing optimization cost. Training only on the latest knowledge, on the other hand, may cause the policy to forget previously knowledge. We therefore formulate lifelong policy learning as bounded incremental policy improvement with knowledge replay. Let ℳk−1⊆⋃j=1k−1j,|ℳk−1|≤C,M_k-1 _j=1^k-1D_j, |M_k-1|≤ C, (12) denote the replay memory before round k, where C is its maximum capacity. With replay mechanism, learning loss at task k can be defined as: ℒreplayk(θ)=x∼kℓ(x;θ)+λrepx∼ℳk−1ℓ(x;θ), splitL_replay^k(θ)=&E_x _k (x;θ)\\ &+ _repE_x _k-1 (x;θ), split (13) where the first term learns the current task, the second rehearses retained knowledge from historical tasks, and λrep≥0 _rep≥ 0 controls their relative influence. Replaying only the supervised target preserves the expert-preferred feasible action, but ignores the expert or scoring knowledge about infeasible or less preferred alternatives. To retain both positive and negative preferences in the action space, we store the historical policy response and use it as an auxiliary distillation target during replay. Following the idea of dark-experience replay [1], we store this response together with each replay sample. Let uxu_x be the stored output vector for sample x, and let oθ(x)o_θ(x) be the current model output over the same dimensions. We define the response-retention loss as ℒDER(x;θ)=KL(softmax(ux/ϵd)∥softmax(oθ(x)/ϵd)),L_DER(x;θ)=KL (softmax(u_x/ _d)\,\|\,softmax(o_θ(x)/ _d) ), (14) where ϵd>0 _d>0 is a temperature parameter. Combining supervised replay with response retention gives ℒKRk(θ)=ℒreplayk(θ)+λDERx∼ℳk−1ℒDER(x;θ),θk=argminθℒKRk(θ), splitL_KR^k(θ)=&L_replay^k(θ)+ _DERE_x _k-1L_DER(x;θ),\\ θ^k=& _θL_KR^k(θ), split (15) where λDER≥0 _DER≥ 0 controls retention of the historical model response. The replay term preserves stored supervision, while the DER term constrains changes in the richer output structure associated with that supervision. After each task, the memory is updated by ℳk=C(ℳk−1∪k),M_k=U_C (M_k-1 _k ), (16) where CU_C retains at most C records from the previous memory and current data. IV R2LPL: Rollout-Retrieval Lifelong Policy Learning IV-A Framework Overview Fig. 2 illustrates the overall pipeline of R2LPL. Each ROCL round follows a Rollout–Retrieval–Lifelong Policy Learning cycle. The current policy is first executed in a closed-loop simulator or world model to induce states under its own decisions and expose failure, risk, or conflict evidence. The retrieval stage then searches the rollout experience for recoverable mistake-related states and converts them into corrective targets. Finally, lifelong policy learning updates the planner with the newly retrieved knowledge while replaying previous corrections. After the update, the improved policy induces a new closed-loop state distribution, which starts the next ROCL round. The first two stages together instantiate rollout-retrieval (R2) target construction. Rollout is used diagnostically rather than as a demonstration generator: it reveals where the current policy becomes unsafe, risky, or conflict-related under closed-loop execution. Retrieval further determines whether the mined mistake-related states are recoverable by the available trajectory-action space and selectively retrieves feasible candidate actions as corrective targets, while discarding unrecoverable states to avoid propagating misguiding rollout histories into supervision. The resulting R2 knowledge is then used for lifelong policy learning, where the planner is updated with new R2 supervision, replayed historical corrections, and anchor-score retention. Through repeated ROCL rounds, R2LPL continually accumulates corrective knowledge from the policy’s own closed-loop experience while reducing forgetting of previously learned corrective knowledge. Figure 2: Overview of R2LPL. Each ROCL round follows a Rollout–Retrieval–Lifelong Policy Learning cycle: closed-loop rollout exposes failure, risk, and conflict evidence; retrieval identifies recoverable mistake-related states and constructs corrective targets; and lifelong policy learning updates the policy with new R2 knowledge and replayed memory. IV-B R2 Corrective Target Construction IV-B1 Policy-Induced State Mining Static expert data describe states visited by the expert, but do not reveal which states the current policy will induce or which of its earlier decisions will lead to undesirable closed-loop behavior. We therefore roll out the current policy to obtain policy-dependent experience from which corrective knowledge can be constructed. At round r, executing πr−1 _r-1 in closed-loop rollout produces: Ξr=Rollout(πr−1)=(sr,tπ,ar,tπ)t=1T. _r=Rollout( _r-1)=\(s_r,t^π,a_r,t^π)\_t=1^T. (17) where Ξr _r denotes the visited states collected from rollout in time T. Not every visited state is informative for correction. We use three complementary event detectors dg(Ξr,t)∈0,1d_g( _r,t)∈\0,1\, g∈F,R,Cg∈\F,R,C\, for Failure, Risk, and Conflict, respectively. Failure identifies states associated with an observed closed-loop failure; Risk captures safety-critical states that do not directly lead to failures; and Conflict captures inappropriate acceleration or yielding decisions that are inconsistent with logged expert behaviorally. Together, they cover terminal errors, non-terminal safety precursors, and decision-level inconsistencies. Observed events are not necessarily caused by the action selected at the exact event frame: erroneous decisions can begin several steps earlier and gradually drive the policy into danger. We therefore assign each event a preceding credit-assignment window tFt_F: tF=mint∣dF(Ξr,t)=1,t_F= \t d_F( _r,t)=1\, (18) For event type g, a state is retained in ℐr,gI_r,g, if a corresponding event occurs within the next WgW_g steps: ℐr,g=sr,tπ|t≤tF,∃k∈[t,min(t+Wg,tF)]s.t. dg(Ξr,k)=1.I_r,g= \s_r,t^π\; |\; subarrayct≤ t_F,\\ ∃ k∈[t, (t+W_g,t_F)]\\ s.t. d_g( _r,k)=1 subarray \. (19) The potentially informative policy-induced states are therefore: ℐr=⋃g∈F,R,Cℐr,g.I_r= _g∈\F,R,C\I_r,g. (20) The horizon WgW_g therefore links an observed event to the sequence of potentially responsible decisions rather than only supervising with its final frame. The constraint t≤tFt≤ t_F excludes post-failure states whose histories have already been corrupted by mistake related actions. Applying this definition across rollouts yields the complete state set ℐrI_r for target retrieval. IV-B2 Action Scoring and Target Recovering State mining identifies where correction may be useful, but does not directly provide a valid target. In particular, the logged expert future is anchored at the logged state stEs_t^E and may become infeasible after the learned policy reaches a different state stπs_t^π. R2 therefore first estimates the reliability of log-based guidance and then searches the structured action space for a feasible correction. For each sr,tπ∈ℐrs_r,t^π _r, we omit the round superscript for clarity and characterize its deviation from the logged state by Δt=(dxy(stπ,stE),dψ(stπ,stE),dv(stπ,stE),dp(stπ,stE)) _t= (d_xy(s_t^π,s_t^E),d_ψ(s_t^π,s_t^E),d_v(s_t^π,s_t^E),d_p(s_t^π,s_t^E) ) (21) We first form a geometrically admissible subset tpre⊆A_t^pre by removing actions that are clearly incompatible with the route, heading, or drivable area while retaining stop actions. This preselection only reduces the number of actions evaluated. Each remaining action receives a rule-based planning score qi=Q(misaf,midrv,miroute,miprog,micomf),q_i=Q\! (m_i^saf,m_i^drv,m_i^route,m_i^prog,m_i^comf ), (22) where the five terms measure safety, drivable-area compliance, route consistency, progress, and comfort, respectively, and Q denotes their rule-based aggregation. The feasible candidate set tvalidA_t^valid and the state recoverability R(stπ)R(s_t^π) of state stπs_t^π are: tvalid=ai∈tpre∣qi>0,R(stπ)=[tvalid≠∅]. splitA_t^valid&=\a_i _t^pre q_i>0\,\\ R(s_t^π)&=1[A_t^valid≠ ]. split (23) States with R(stπ)=0R(s_t^π)=0 are excluded because the available action space cannot provide valid supervision from them. For a recoverable state, its deviation determines the guidance class κt _t: κt=Γ(Δt)∈near,rec,far. _t= ( _t)∈\near,rec,far\. (24) where nearnear, recrec, and farfar denote three types of guidance class. Near-log states retain reliable expert guidance; recoverable off-log states use the expert only as a weak behavioral prior; and far-off-policy states no longer use the logged expert future for target retrieval. Thus, recoverability determines whether the state can be supervised, whereas κt _t determines which evidence should define that supervision. For recoverable states, action quality is described by three signals: normalized rule score q¯i q_i, reference consistency cirefc_i^ref, and expert consistency ciEc_i^E. Reference consistency measures geometric alignment with the route/reference path. Expert consistency measures behavioral alignment with the logged future: ciE=exp( c_i^E= ( −αHdH(ai,τtE)−αPdP(ai,τtE) - _Hd_H(a_i, _t^E)- _Pd_P(a_i, _t^E) (25) −αψdψ(ai,τtE)−αvdv(ai,τtE)), - _ψd_ψ(a_i, _t^E)- _vd_v(a_i, _t^E) ), where dHd_H is a Hausdorff-style shape distance, dPd_P measures progress mismatch along the reference path, and dψd_ψ and dvd_v measure heading and velocity disagreement, respectively. The progress term is symmetric: it penalizes both lagging and overshooting, so waiting and yielding behavior can be learned rather than overwritten by a progress-only scorer. The final target score yiy_i is computed as: yi=wqκtq¯i+wrefκtciref+wEκtciE.y_i=w_q _t q_i+w_ref _tc_i^ref+w_E _tc_i^E. (26) where wqκtw_q _t, wrefκtw_ref _t and wEκtw_E _t denote the respective weights for different guidance class κt _t. The class-dependent parameters express the different reliability of supervision. Near-log states assign greater weight to expert consistency; recoverable off-log states emphasize rule and reference scores while retaining weak expert guidance; and far-off-policy states set wEfar=0w_E^far=0. Hence the scoring rule transits from log-guided correction to rule-guided recovery as the deviation between policy-induced states and logged expert states increases. Thus the retrieved target is: a∗=argmaxai∈tvalidyi.a^*= _a_i _t^validy_i. (27) For training a score-based planner, the recovered sample stores the sparse target distribution: t=(i,yi)∣ai∈tvalid,yi>0,Y_t=\(i,y_i) a_i _t^valid,y_i>0\, (28) which supervises both the best anchor and the relative preference among recovered candidates. IV-C Lifelong Policy Learning with R2 Knowledge Let r=(stπ,t)D_r=\(s_t^π,Y_t)\ denote the recovered knowledge produced by R2 in round r. Each sample pairs a policy-induced driving state with a retrieved anchor-score target, so it can be used by the same score-based policy in Eq. (4). The key role of LPL is to incorporate this new R2 knowledge for policy learning while retaining knowledge learned in previous rounds. Thus both expert-log pretraining and rollout-induced lifelong learning are expressed through the same anchor-score planning loss in Eq. (9). Let ℳrM_r denote the bounded memory available before learning round r. The round-r update minimizes: ℒr=x∼rℒAS(x;θ)+λrepx∼ℳrℒAS(x;θ)+λDERx∼ℳrℒDER(x;θ)+λEx∼ErℒAS(x;θ). splitL_r=&E_x _rL_AS(x;θ)+ _repE_x _rL_AS(x;θ)\\ &+ _DERE_x _rL_DER(x;θ)+ _EE_x _E^rL_AS(x;θ). split (29) where x is shorthand for a state–target pair (s,)(s,Y). The first term learns from the current recovered failures. The second term constraints policy from forgetting replayed knowledge from previous rounds. The DER term preserves the stored teacher preference over replayed anchors through score-distribution matching. The final term mixes original expert-log samples from ErD_E^r to retain broad log-distribution behavior. After optimizing Eq. (29), LPL updates its bounded memory from both historical and newly recovered knowledge. The memory candidate pool is the duplicated union r=Unique(ℳr∪r).P_r=Unique (M_r _r ). (30) As memory capacity is limited, it is important to update and store only the most valuable knowledge. Each x∈rx _r is evaluated from two complementary aspects. The first is learning difficulty hr(x)h_r(x). For a new sample, it is measured by the mean anchor-score loss observed over NxN_x times it is sampled during round-r training. For a historical sample that is not revisited as current data, its stored priority ρr−1(x) _r-1(x) from the previous memory update is carried forward: hr(x)=1Nx∑n=1NxℒAS(n)(x;θ),x∈r,ρr−1(x),x∈ℳr∖r.h_r(x)= cases 1N_x _n=1^N_xL_AS^(n)(x;θ),&x _r,\\[6.0pt] _r-1(x),&x _r _r. cases (31) The second aspect is rollout utility u(x)u(x), which favors samples associated with more consequential or more difficult-to-recover closed-loop states: u(x)=∑g∈F,R,Cwg 1[g∈(x)]+ws(cx)+wttc[1−TTC(x)τttc]++wq[1−y∗(x)τq]+, splitu(x)=& _g∈\F,R,C\w_g\,1[g (x)]+w_s(c_x)\\ &+w_ttc [1- TTC(x) _ttc ]_++w_q [1- y^*(x) _q ]_+, split (32) where (x)G(x) records whether the sample is mined from Failure (F), Risk (R), or Conflict (C) evidence; cxc_x is its near-log, recoverable, or far-recoverable state class; TTC(x)TTC(x) is the minimum time to collision; and y∗(x)=maxiyiy^*(x)= _iy_i is the best recovered action score from Eq. (26). Because the scales of hrh_r and u vary across rounds, both are min–max normalized over the same candidate pool rP_r. Denoting this pool-wise normalization by r(⋅)N_P_r(·), the memory priority ρr(x) _r(x) is described as: ρr(x)=αr(hr(x))+βr(u(x)). _r(x)=α\,N_P_r\! (h_r(x) )+β\,N_P_r\! (u(x) ). (33) A global top-C selection rule may cause frequent scenarios or dominant failure types to occupy most of the memory. We therefore adopt a bucket-balanced memory update. Specifically, the candidate pool r=Unique(ℳr∪r)P_r=Unique(M_r _r) is partitioned according to the scene category, the state class cxc_x, and the primary Failure/Risk/Conflict reason. Within each bucket, samples are ranked by the memory priority ρr(x) _r(x), which combines learning difficulty and rollout utility. The limited memory budget is then allocated approximately uniformly across active buckets, and the highest-priority samples in each bucket are retained. If the candidate pool does not exceed the memory capacity, all candidates are kept. This stratified update preserves diverse corrective knowledge across scenarios and corrective knowledge types while still favoring high-priority old and newly retrieved R2 samples. The resulting memory ℳr+1M_r+1 serves as the replay set for the next ROCL round. Algorithm 1 R2LPL: Rollout-Retrieval Lifelong Policy Learning 1: Input: pretrained planner π0 _0, scenarios S, anchor library A, memory capacity C 2: Initialize replay memory ℳ0←∅M_0← 3: for round r=0,1,…,R−1r=0,1,…,R-1 do 4: Roll out πr _r on scenarios S 5: Mine ℐrI_r from Failure, Risk, and Conflict evidence; stop mining after failure boundaries 6: Initialize recovered dataset r←∅D_r← 7: for frame t∈ℐrt _r do 8: Restore scene at rollout ego state stπs_t^π 9: Construct admissible candidate anchors and score them by Eq. (26) 10: if no valid candidate exists then 11: discard frame as unrecoverable 12: else 13: retrieve sparse target tY_t by Eq. (28) 14: add (stπ,t)(s_t^π,Y_t) to rD_r 15: end if 16: end for 17: Update πr _r with LPL objective in Eq. (29) 18: Obtain next planner πr+1 _r+1 19: Form r=Unique(ℳr∪r)P_r=Unique(M_r _r) and compute ρr _r by Eqs. (31)–(33) 20: Select ℳr+1M_r+1 by the bucket-balanced rule 21: end for 22: Return: lifelong-improved planner πR _R V Experiments V-A Benchmarks and Metrics We evaluate R2LPL on the nuPlan closed-loop simulation benchmarks [17]. NuPlan provides real-world driving logs with rich scenario annotations, long-horizon closed-loop simulation, and standardized planning metrics. These properties allow us to expose policy-induced mistakes through rollout, and evaluate the capability of proposed R2LPL for continual policy improvement in large-scale real traffic scenarios. We use three commonly adopted nuPlan benchmarks: Val14, Test14-hard, and Test14-random. Val14 contains 1118 validation scenarios from 14 scenario types with an imbalanced category distribution. Test14-hard contains 272 challenging and long-tailed scenarios, while Test14-random contains 261 randomly sampled scenarios with a more balanced distribution over the same 14 scenario types. We consider two closed-loop protocols: non-reactive (NR) and reactive (R) simulation. In NR simulation, the ego vehicle is controlled by the planner while other agents follow logged trajectories. In R simulation, surrounding agents are controlled by an IDM-based reactive model. For each benchmark, ROCL data generation and policy updating are performed on the scenarios from that benchmark under the NR protocol. The R results are obtained by directly evaluating the NR-updated checkpoint under the R protocol, without any additional R-specific ROCL update. Planner performance is measured by the official nuPlan aggregate score, which combines safety, comfort, and progress-related metrics. V-B Implementation Details We instantiate R2LPL-base with an anchor-based planner with transformer encoder-decoder architecture. The planner encodes vectorized ego history, neighboring agents, road geometry, and route context, and predicts scores over a fixed library of K=4096K=4096 trajectory anchors. Each anchor covers a 4.04.0 s planning horizon at 0.20.2 s intervals. The model uses a hidden dimension of 256256, an encoder depth of 44, and a planning decoder depth of 88. We pretrain the base model on 1M samples from the nuPlan training set for 3030 epochs, using a batch size of 7272, a learning rate of 10−410^-4, 10001000 warm-up steps, λKL=0.1 _KL=0.1, and bf16-mixed precision on 4 NVIDIA RTX 4090 GPUs. Rollouts are performed with model-only planning without any post-processing. Rollout and retrieval are run on two AMD EPYC 7763 64-Core processors. Candidate mining uses a rollout time step of 0.10.1 s. Failure-related frames are collected from the 4040 steps preceding the failure frame. A failure frame is identified when collision or out-of-road is detected. Risk frames are identified by a minimum TTC below 1.01.0 s, together with a 1010-step pre-risk window. Conflict frames are selected according to misalignment between policy-induced states and logged expert states, using thresholds of 0.50.5 m/s for waiting speed, 3.03.0 m for lag-progress gap, and 1.01.0 m/s for moving speed. For R2 retrieval, we first prefilter at most 10241024 geometry-compatible anchors before scoring. Compatibility is determined by a maximum reference-path distance of 6.06.0 m and a maximum heading error of 60∘60 . A frame is discarded as unrecoverable if no valid anchor is found. Otherwise, the top 256256 recovered anchors are stored as sparse replay targets. State classes are defined by δnearxy=1.0 _ near^xy=1.0 m, δnearψ=15∘ _ near^ψ=15 , δnearv=2.0 _ near^v=2.0 m/s, δrecxy=5.0 _ rec^xy=5.0 m, and δrecψ=45∘ _ rec^ψ=45 . Near-log retrieval further requires the expert-consistency score to be within 0.050.05 of the best score and at least 0.750.75. The final anchor score in Eq. (26) uses (wsim,wref,wexp)=(0.10,0.10,0.80)(w_ sim,w_ ref,w_ exp)=(0.10,0.10,0.80) for near-log states, (0.65,0.30,0.05)(0.65,0.30,0.05) for recoverable states, and (0.80,0.20,0.00)(0.80,0.20,0.00) for far-off-policy states. For LPL, we use DER with scenario-aware rollout-priority replay as the default continual learning method. Each ROCL round is trained for 2020 epochs with a batch size 2424 and a memory capacity C=4096C=4096. In Eq. (29), the replay loss weight is λrep=1.0 _ rep=1.0, the DER distillation weight is λDER=0.25 _ DER=0.25, and the distillation temperature is 1.01.0. In Eq. (33), the normalized training-loss component and the rollout-utility component are both assigned a weight of 1.01.0. Expert-cache mixing uses a ratio of 0.250.25, a loss weight of λE=0.25 _E=0.25, and at most 20,00020,000 expert samples per task. V-C Comparison with Baselines We compare R2LPL with representative learning-based planners on the nuPlan benchmarks to evaluate the relative performance improvement and to assess its competitiveness among existing policy learning paradigms. The comparison focuses on planners whose reported performance mainly comes from learned policy modeling, including recent generative methods and reinforcement-learning fine-tuning methods. Since this experiment aims to compare learned policy capability rather than rule-assisted execution quality, we do not include methods whose final scores rely heavily on rule-based post-processing, trajectory repair, or hybrid rule-based refinement. Such methods are important for engineering robustness, but their performance may reflect both learned policy quality and external correction modules, making it difficult to isolate the capability of the learned planner. The compared methods include: ∙ UrbanDriver [34] is an early learning-based planner that optimizes a driving policy from real-world demonstrations using a policy-gradient objective. ∙ PDM-Open [9] predicts future way points using an IDM-based centerline and ego history. ∙ PlanTF [7] is an imitation-based transformer planner trained to predict expert future motion. ∙ PLUTO [6] improves imitation-based planning with stronger scene representation, data augmentation, and training design. ∙ Diffusion Planner [55] formulates planning as conditional diffusion trajectory generation with guidance. ∙ Flow Planner [39] applies flow matching and interaction-aware modeling for tokenized trajectory generation. ∙ DFP [53] uses diffusion forcing with history-annealed planning, and DFP-FM augments it with flow matching. ∙ Plan-R1 [40] casts trajectory planning as motion-token language modeling and applies rule-based GRPO post-training. We report our pretrained anchor-based planner as R2LPL-base, a fixed-budget five-round version as R2LPL-ROCL-5, and the best observed result within the ten-round search budget as R2LPL-ROCL-10-best. TABLE I: Main closed-loop results on nuPlan benchmarks. NR and R denote non-reactive and reactive simulation protocols, respectively. Scores are aggregated metrics, where higher values indicate better performance. Bold denote the best and underlined the second-best results among compared methods. Method Training Paradigm Val14 Test14-hard Test14-random NR R NR R NR R Expert Human log replay 93.53 80.32 85.96 68.80 94.03 75.86 UrbanDriver [34] Imitation learning 68.57 64.11 50.40 49.95 51.83 67.15 PDM-Open [9] Imitation learning 53.53 54.24 33.51 35.83 52.81 57.23 PlanTF [7] Imitation learning 84.27 76.95 69.70 61.61 85.62 79.58 PLUTO [6] Imitation learning 88.89 78.11 70.03 59.74 89.90 78.62 Diffusion Planner [55] Generative IL 89.87 82.80 75.99 69.22 89.19 82.93 Flow Planner [39] Generative IL 90.43 83.31 76.47 70.42 89.88 82.93 DFP [53] Generative IL 90.33 79.97 76.91 63.56 90.69 81.96 DFP-FM [53] Generative IL 92.68 81.30 79.43 67.94 90.62 83.59 Plan-R1 [40] IL + RL alignment 88.98 87.69 77.45 77.20 91.23 90.04 R2LPL-base (ours) Imitation learning 75.39 73.87 60.67 65.25 70.74 72.96 R2LPL-ROCL-5 (ours) IL + R²LPL 91.26 85.38 83.51 78.38 92.25 87.99 R2LPL-ROCL-10-best IL + R²LPL (envelope) 92.22 85.83 86.54 78.88 93.94 88.20 Table I reports the main closed-loop results on all three nuPlan benchmarks. The pretrained anchor-based planner is relatively weak compared with recent learning-based planners, achieving only 75.39/73.87, 60.67/65.25, and 70.74/72.96 on Val14, Test14-hard, and Test14-random under NR/R protocols. After five ROCL rounds, the same planner is improved to 91.26/85.38, 83.51/78.38, and 92.25/87.99, yielding absolute gains of +15.87+15.87, +22.84+22.84, and +21.51+21.51 points under NR and +11.51+11.51, +13.13+13.13, and +15.03+15.03 points under R. These improvements elevate a moderate base planner to the first or second place among all compared learning-based methods on every benchmark and protocol. Since ROCL updates the policy without changing the base architecture or adding deployment-time rule refinement, the results indicate that the proposed framework improves the learned policy itself rather than relying on a stronger planner design or external execution-time correction. Among the three splits, Test14-hard provides the clearest stress test for closed-loop policy improvement because it focuses on challenging and long-tailed scenarios. On this split, R2LPL-ROCL-5 achieves 83.51 NR and 78.38 R, outperforming the strongest listed learning-based baselines by 4.084.08 and 1.181.18 points, respectively. The larger margin under NR is expected, since both ROCL data generation and policy updating are conducted with NR rollouts. Nevertheless, the R score also increases from 65.25 to 78.38 without any R-specific ROCL update, reaching the best result among all compared methods. This cross-protocol improvement suggests that the retrieved corrective knowledge is not merely tied to the fixed-agent trajectories used during NR rollout. Since ROCL supervises recoverable mistake-related states rather than directly imitating a specific closed-loop trajectory, the learned corrections can encode more general decision preferences for avoiding unsafe or conflicting behaviors, which remain beneficial even when the interaction dynamics changes. The results on Val14 and Test14-random show a more moderate but consistent pattern. These splits cover broader evaluation scenarios rather than explicitly concentrating on hard long-tail cases, so the advantage of mistake-driven correction is less pronounced than on Test14-hard. Nevertheless, R2LPL-ROCL-5 still ranks second on Val14 under both protocols and first/second on Test14-random under NR/R. This indicates that correcting policy-induced mistakes does not sacrifice broader closed-loop competence. In other words, R2LPL is not only effective on failure-prone scenarios, but also retains strong general performance after lifelong updates. The learning-envelope result R2LPL-ROCL-10-best further shows that additional improvement is possible within the same ten-round search budget, reaching 92.22/85.83, 86.54/78.88, and 93.94/88.20 on the three splits. We use R2LPL-ROCL-5 as the primary fixed-budget result and report R2LPL-ROCL-10-best only to indicate the potential performance envelope of the iterative improvement process. TABLE I: Detailed Test14-hard closed-loop metrics across five ROCL updates. Scores are aggregated metrics, where higher values indicate better performance. Parenthesized values report absolute changes from the previous round. ROCL Round Score Collisions TTC Drivable Comfort Progress 0 (Base) 60.67 (+00.00) 75.55 (+00.00) 64.76 (+00.00) 90.81 (+00.00) 88.24 (+00.00) 94.81 (+00.00) 1 71.64 (+10.97) 91.36 (+15.81) 84.92 (+20.16) 95.95 (+5.14) 90.81 (+2.57) 74.68 (-20.13) 2 78.63 (+6.99) 91.54 (+0.18) 82.72 (-2.20) 95.59 (-0.36) 89.34 (-1.47) 87.78 (+13.1) 3 80.70 (+2.07) 94.85 (+3.31) 84.56 (+1.84) 95.22 (-0.37) 90.44 (+1.10) 88.87 (+1.09) 4 83.00 (+2.30) 94.49 (-0.36) 85.66 (+1.10) 97.43 (+2.21) 92.28 (+1.84) 90.16 (+1.29) 5 83.52 (+0.52) 93.20 (-1.29) 85.29 (-0.37) 97.43 (+0.00) 91.54 (-0.74) 91.80 (+1.64) Figure 3: Closed-loop performance over ROCL rounds on Val14, Test14-hard, and Test14-random. Solid curves report R2LPL scores across ROCL rounds, and dashed horizontal lines report the corresponding expert-log replay scores. Figure 4: Evolution of policy-induced failure data and replay memory across ROCL rounds. The three panels summarize: (a) retrieved data classified by recovery reason, (b) memory composition by recovery reason, and (c) memory source over each ROCL round. V-D Analysis on Iterative Improvement This subsection examines the iterative behavior of R2LPL. The analysis focuses on two empirical aspects: whether the planner continues to improve across repeated rollout-learning rounds, and what data and replay mechanisms account for this improvement. We use the round-wise score curves to address the former, and the mined-data and memory statistics to analyze the latter. Fig. 3 shows that the improvement is not limited to the first update. On Test14-hard, the NR score increases sharply in the early rounds and continues to improve more gradually afterwards, eventually exceeding the expert-log replay reference. Although the curve is not strictly monotonic, such fluctuations are expected because each policy update changes the induced closed-loop state distribution and may shift the balance among safety, comfort, and progress terms in the nuPlan aggregate score. Nevertheless, the overall upward trend indicates that R2LPL continues to accumulate useful corrective knowledge across ROCL rounds. The smoother curves on Val14 and Test14-random further indicate that iterative rollout learning does not only benefit the hardest subset, although the largest gain appears on Test14-hard, where the initial policy exposes more severe residual errors. To understand how the improvement is obtained, Table I reports the detailed metric evolution on Test14-hard. Parenthesized values denote absolute changes from the previous round, because the nuPlan aggregate and component metrics are bounded scores. The metric breakdown suggests a two-stage improvement pattern. The first round gives the largest aggregate gain, from 60.67 to 71.64, mainly through safety-related metrics: collision score improves by 15.81 points and TTC score improves by 20.16 points. This is consistent with the role of rollout mining, since the base policy exposes safety-critical states that can be converted into corrective targets when they are recoverable. Subsequent rounds provide smaller but still meaningful gains, with improvements distributed over drivable-area compliance, comfort, and progress. The temporary progress drop after the first round also illustrates the multi-objective nature of closed-loop planning: an update that strongly suppresses risky behavior can initially make the policy more conservative, and later rounds restore the progress score while preserving the safety gains. By ROCL round 5, the aggregate score reaches 83.52, with relatively high collision, TTC, drivable-area, comfort, and progress scores retained simultaneously. The remaining question is why repeated updates continue to provide useful supervision. Fig. 4 reports three statistics related to the data stream and memory: the retrieved training data mined in each round, the composition of the replay memory, and the source rounds preserved by the memory. These statistics indicate whether later policies still expose informative errors and whether the learner maintains access to previously recovered corrections. Fig. 4(a) shows that the recovered data are policy-dependent rather than fixed. Failure denotes frames in the temporal window before an actual rollout failure, Risk denotes high-risk context frames such as low TTC or dangerous near-miss states, and Conflict denotes model-expert disagreement frames where the policy behavior differs substantially from the logged expert. Early rounds contain many failure-window corrections, while later rounds still provide risk and conflict samples around the remaining difficult states. Thus, after the most obvious failures are corrected, rollout continues to reveal residual mistakes that can be used for further learning. Fig. 4(b) and Fig. 4(c) show how LPL stabilizes this evolving data stream. The memory keeps a mixed composition of Failure, Risk, and Conflict samples rather than being dominated by one category, and it preserves samples from multiple source rounds. Later updates are therefore not simple fine-tuning on the newest rollout cache. They combine newly exposed residual mistakes with replayed corrections from previous policies. This interaction between policy-dependent mining and replay explains the sustained improvement observed in Fig. 3: R2 supplies newly exploited recoverable mistakes as the policy updates, while LPL prevents earlier experience from being overwritten. V-E Ablation Studies We organize the ablation around two questions. The first is whether the improvement is attributed to overfitting target-scenario expert knowledge rather than learning from policy-induced mistakes. The second is whether recoverability-aware supervised fine-tuning is sufficient, or whether lifelong policy learning is needed for stable multi-round continuous improvement. TABLE I: Ablation on Test14-hard under a five-round ROCL budget. SFT denotes supervised fine-tuning, and w/o denotes without. Round 0 is the same pretrained base policy for all variants. The target expert-log SFT baseline is a one-shot adaptation baseline, so later ROCL rounds are not applicable. Method ROCL Score 0 1 2 3 4 5 Base + expert-log SFT 60.67 60.79 – – – – Base + RoaD-style SFT 60.67 66.36 67.34 66.62 65.57 63.58 Base + R2-style SFT 60.67 71.29 75.37 76.12 79.51 76.17 R2LPL w/o expert mix 60.67 72.44 78.34 80.89 83.18 82.64 R2LPL full 60.67 71.64 78.63 80.70 83.00 83.52 Table I separates the effects of target-scenario expert knowledge, closed-loop rollout data, recoverability-aware retrieval, and lifelong policy learning. Directly fine-tuning the base model on target-scenario expert logs provides almost no improvement, increasing the score from 60.67 to 60.79. This suggests that the main gain in R2LPL is not simply due to additional exposure to the target scenarios or memorization of their logged expert behavior. We further compare with a RoaD-style SFT baseline. This baseline uses closed-loop rollout states and the corresponding expert demonstration guided targets as supervised training samples, but does not perform R2 recoverability-aware target retrieval or LPL. It improves over expert-log SFT and reaches 67.34 after two rounds, showing that policy-induced states and targets recovered from expert logs do contain useful information. However, its gain is limited. A likely reason is the uneven quality of the induced supervision. When the policy has deviated substantially from the logged expert state, forcing the rollout state toward an expert-aligned demonstration can introduce misleading targets, because the logged expert motion may no longer be a feasible or appropriate correction from the learner-induced state. The later score decrease to 63.58 then reflects a stability and forgetting issue under repeated updates on such noisy rollout-derived targets. Replacing RoaD-style targets with R2 retrieval provides a stronger SFT baseline. Base model trained with R2-style SFT reaches 71.29 after the first round, close to the 71.64 obtained by full R2LPL, which indicates that recoverability-aware target construction provides the main initial corrective signal. However, this SFT baseline improves more slowly in later rounds and drops from 79.51 at round 4 to 76.17 at round 5. This behavior is consistent with learning under insufficient retention: while new R2 knowledge is being fitted, earlier correction knowledge are not explicitly preserved, and later updates cause the polity to forget more than it learns. In contrast, R2LPL maintains scores above 80 after round 3 and reaches 83.52 at round 5, showing the role of LPL in retaining previously acquired correction knowledge while incorporating new R2 knowledge. The variant without expert mix is close to the full model and is slightly higher in some intermediate rounds, indicating that expert-log mixing is not the primary source of the closed-loop performance gain. Overall, the ablation supports that R2LPL benefits from both high-quality recoverable supervision from R2 and non-forgetting policy update through LPL, while expert mixing mainly acts as a conservative regularizer. V-F Qualitative Analysis We visualize representative base-policy failures and the corresponding round-5 behaviors after R2LPL updates. The qualitative analysis is shown in three aspects: how R2 corrective knowledge is retrieved, which rollout states are used by R2 for recovery knowledge construction, and to illustrate how the improved policy avoids repeating the same failure. (a) High lateral acceleration (b) Changing lane (c) Near pedestrian on crosswalk Figure 5: Qualitative examples of R2 corrective-supervision construction, Blue boxes denote the rollout ego vehicle, and yellow boxes denote surrounding agents. Case (a) shows a high-lateral-acceleration maneuver that eventually leads to an off-road failure, case (b) shows a lane-changing collision failure, and case (c) shows a collision failure with pedestrians. Each row shows a recoverable state mined from a base-policy failure and the corresponding retrieved target in the same state. Fig. 5 provides a qualitative visualization of how R2 constructs corrective supervision from closed-loop failures. For each failure case, R2 first mines recoverable policy-induced states before the terminal failure. The left panels show the base-policy behavior around one of these states: although the original plan predicted at the selected frame may appear locally plausible, the subsequent closed-loop rollout follows this decision and eventually leads to an off-road or collision failure. Therefore, R2 focuses on the pre-failure window to identify recoverable states and correct the decisions that drive the policy toward failure, rather than simply labeling the terminal failure pose. Given the recoverable state, R2 then searches the structured action space for alternative corrective plans. As shown in the right panels, candidate plans are evaluated by the normalized target score yiy_i in Eq. (8), with darker candidates indicating higher-scoring plans under the evaluator. The selected recovered plan provides a safer and more feasible alternative to the original plan, and is therefore used as the retrieved target. This target is stored as R2 knowledge and used as corrective supervision in the subsequent lifelong policy update. (a) High lateral acceleration (b) Low magnitude speed (c) Traversing pickup-dropoff Figure 6: Qualitative recovery examples. Each row shows a base-policy failure and the corresponding ROCL round-5 R2LPL rollout results around the same critical frame. Case (a) corresponds to an off-road failure associated with high lateral acceleration, while cases (b) and (c) correspond to collision failures. Blue boxes denote the rollout ego vehicle, light-blue dashed boxes denote the log expert, yellow boxes denote surrounding agents, and purple boxes denote static obstacles. Black solid and gray dashed curves show rollout and expert histories, respectively. Green and red thick trajectory segments in the base rollout indicate recoverable and unrecoverable failure-window states. The round-5 panels at −2-2 s, 0 s, and +2+2 s show how the improved policy behaves before, at, and after the critical frame. Fig. 6 provides concrete examples of the rollout-retrieval process. In the base-policy rollout, R2 does not treat all states before failure equally. Green segments indicate states for which the action set and evaluator can still identify feasible recovery targets, and these states are used to construct R2 knowledge. Red segments indicate states that have become unrecoverable under the available action space or scene constraints, and are therefore excluded from training. This visualization corresponds directly to the recoverable/unrecoverable separation in the method. It also illustrates why R2LPL mines a temporal window before failure rather than only the final failure frame. In closed-loop planning, a collision or off-road event is often the consequence of erroneous decisions made several seconds earlier. When close to the final failure frame, the failure may already be irreversible. Corrective learning therefore needs to identify and repair the preceding recoverable decisions that lead the vehicle toward the dangerous state. The three cases in Fig. 6 cover different failure modes. In Fig. 6(a), the base policy drives the ego vehicle outside the drivable region while exhibiting high lateral acceleration. In Fig. 6(b) and Fig. 6(c), the base policy leads to collision-prone interactions, including collision with other agent in left turn and collision with a static obstacle. The round-5 panels show the learned policy at −2-2s, 0s, and +2+2s relative to the base policy’s failing moment. Compared with the base policy, the improved policy adjusts its behavior before reaching the failure state and no longer produces the same off-road or collision outcome. These examples illustrate how R2 supplies policy-specified recovery knowledge from selected failure-window states, and how LPL turns this knowledge into corrected closed-loop behavior in later rollouts. VI Conclusion We presented R2LPL, a rollout-retrieval lifelong policy learning paradigm for autonomous driving. Instead of treating closed-loop failures only as evaluation outcomes, R2LPL uses policy-induced rollouts to identify mistake-related states, retrieve recoverable corrective targets from a structured action space, and incorporate the retrieved knowledge through lifelong policy learning. The central insight is that the most useful knowledge for continual policy improvement is not the failure itself, but the corrective knowledge that helps the policy avoid repeating the same mistakes. Experiments on nuPlan show that R2LPL can substantially improve an anchor-based planner with moderate performance and elevate it to state-of-the-art performance among learning-based planners across multiple closed-loop benchmarks. The gains are most pronounced on hard and long-tailed scenarios, while the updated policy also maintains strong performance on broader validation and random test splits. Moreover, although the corrective knowledge is acquired from non-reactive rollouts, the improved policy generalizes to reactive simulation, suggesting that the retrieved targets encode reusable decision preferences rather than overfitting to non-reactive agent patterns. These results demonstrate that large-scale closed-loop policy improvement can be achieved with compact ROCL rounds through R2LPL, without relying solely on larger expert-log imitation, reward-driven exploration, or deployment-time rule refinement. The current instantiation of R2LPL focuses on decision policies with explicit or sampleable action candidates, where recoverability-aware target retrieval can be implemented through candidate evaluation. Its effectiveness therefore depends on the coverage of the action space and the alignment of the evaluator used to select corrective targets. Extending this idea to continuous generative planners may require retrieval or projection in latent trajectory space, together with stronger mechanisms for filtering noisy targets. Another important direction is to replace or complement simulation-based rollouts with learned physical world models with human-like agents, enabling policies to acquire corrective knowledge from a much larger and more diverse stream of closed-loop interactions. Acknowledgments Large language models, including models from DeepSeek and GitHub Copilot, were used for code debugging, language polishing, and readability improvement. The authors verified and take full responsibility for all presented contents, experimental results, and conclusions. All figures and corresponding experimental results can be reproduced using the code and instructions available in our public repository: https://github.com/Engibacter/R2LPL. References [1] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. CALDERARA (2020) Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems, Vol. 33, p. 15920–15930. Cited by: §I-C, §I-B. [2] Z. Cao, K. Jiang, W. Zhou, S. Xu, H. Peng, and D. Yang (2023) Continuous improvement of self-driving cars using dynamic confidence-aware reinforcement learning. Nature Machine Intelligence 5 (2), p. 145–158. External Links: ISSN 2522-5839, Document Cited by: §I-C. [3] S. Casas, A. Sadat, and R. Urtasun (2021) MP3: a unified model to map, perceive, predict and plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14398–14407. Cited by: §I-A. [4] L. Chen, P. Wu, K. Chitta, B. Jaeger, A. Geiger, and H. Li (2024) End-to-end autonomous driving: challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12), p. 10164–10183. External Links: Document Cited by: §I. [5] S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang (2024) Vadv2: end-to-end vectorized autonomous driving via probabilistic planning. In Proceedings of the International Conference on Learning Representations, Cited by: §I, §I-A, §I-A. [6] J. Cheng, Y. Chen, and Q. Chen (2024) PLUTO: pushing the limit of imitation learning-based planning for autonomous driving. arXiv preprint arXiv:2404.14327. Cited by: §I, §I, §I-A, §V-C, TABLE I. [7] J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu (2024) Rethinking imitation-based planner for autonomous driving. In Proceedings of the IEEE International Conference on Robotics and Automation, p. 14123–14130. Cited by: §I, §V-C, TABLE I. [8] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger (2023) TransFuser: imitation with transformer-based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (11), p. 12878–12895. External Links: Document Cited by: §I. [9] D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta (2023) Parting with misconceptions about learning-based vehicle motion planning. In Proceedings of The Conference on Robot Learning, Vol. 229, p. 1268–1281. Cited by: §V-C, TABLE I. [10] D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, A. Geiger, and K. Chitta (2024) NAVSIM: data-driven non-reactive autonomous vehicle simulation and benchmarking. In Advances in Neural Information Processing Systems, Vol. 37, p. 28706–28719. Cited by: §I. [11] S. Feng, H. Zhu, H. Sun, X. Yan, L. He, J. Yang, G. Su, B. Li, S. Li, L. Wang, S. Shen, and H. X. Liu (2026) Breaking through safety performance stagnation in autonomous vehicles with dense learning. Nature Communications 17 (3163). Cited by: §I. [12] G. Garcia-Cobo, M. Igl, P. Karkus, Z. Zhang, M. Watson, Y. Chen, B. Ivanovic, and M. Pavone (2026) RoaD: rollouts as demonstrations for closed-loop supervised fine-tuning of autonomous driving policies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Findings, p. 1000–1009. Cited by: §I, §I, §I-B. [13] C. Gong, C. Lu, Z. Li, Z. Liu, J. Gong, and X. Chen (2024) Beyond imitation: a life-long policy learning framework for path tracking control of autonomous driving. IEEE Transactions on Vehicular Technology 73 (7), p. 9786–9799. External Links: ISSN 1939-9359, Document Cited by: §I-C. [14] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li (2023) Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17853–17862. Cited by: §I. [15] Z. Huang, H. Liu, and C. Lv (2023) GameFormer: game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3903–3913. Cited by: §I, §I-A. [16] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang (2023) VAD: vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 8340–8350. Cited by: §I-A. [17] N. Karnchanachari, D. Geromichalos, K. S. Tan, N. Li, C. Eriksen, S. Yaghoubi, N. Mehdipour, G. Bernasconi, W. K. Fong, Y. Guo, and H. Caesar (2024) Towards learning-based planning: the nuplan benchmark for real-world autonomous driving. In Proceedings of the IEEE International Conference on Robotics and Automation, p. 629–636. Cited by: §I, §V-A. [18] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer (2019) HG-dagger: interactive imitation learning with human experts. In Proceedings of the International Conference on Robotics and Automation, p. 8077–8083. Cited by: §I-B. [19] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §I-C. [20] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg (2017) DART: noise injection for robust imitation learning. In Proceedings of the Conference on Robot Learning, Vol. 78, p. 143–156. Cited by: §I-B. [21] H. Li, X. Wu, J. Huang, and Z. Zhong (2026) Toward zero-forget continual learning for interactive trajectory prediction: a dynamically expandable approach. Communications in Transportation Research 6 (1), p. 9640015. External Links: Document Cited by: §I-C. [22] Z. Li, K. Li, S. Wang, S. Lan, Z. Yu, Y. Ji, Z. Li, Z. Zhu, J. Kautz, Z. Wu, et al. (2024) Hydra-mdp: end-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06978. Cited by: §I, §I, §I-A. [23] Z. Li and D. Hoiem (2016) Learning without forgetting. In Proceedings of the European Conference on Computer Vision, p. 614–629. Cited by: §I-C. [24] Z. Li, C. Gong, Y. Lin, G. Li, X. Wang, C. Lu, M. Wang, S. Chen, and J. Gong (2023) Continual driver behaviour learning for connected vehicles and intelligent transportation systems: framework, survey and challenges. Green Energy and Intelligent Transportation 2 (4), p. 100103. External Links: ISSN 27731537, Document Cited by: §I-C. [25] Y. Liang and W. Li (2024) InfLoRA: interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 23638–23647. Cited by: §I-C. [26] B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, and X. Wang (2025) DiffusionDrive: truncated diffusion model for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12037–12047. Cited by: §I, §I-A. [27] Y. Lin, Z. Li, G. Du, X. Zhao, C. Gong, X. Wang, C. Lu, and J. Gong (2026) H2C: hippocampal circuit-inspired continual learning for lifelong trajectory prediction in autonomous driving. IEEE Transactions on Intelligent Transportation Systems (), p. 1–18. External Links: Document Cited by: §I-C. [28] D. Lopez-Paz and M. Ranzato (2017) Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, p. 6470–6479. Cited by: §I-C. [29] A. Mallya and S. Lazebnik (2018) PackNet: adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7765–7773. Cited by: §I-C. [30] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer (2019) EnsembleDAgger: a bayesian approach to safe imitation learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5041–5048. Cited by: §I-B. [31] Y. Meng, Z. Bing, X. Yao, K. Chen, K. Huang, Y. Gao, F. Sun, and A. Knoll (2025) Preserving and combining knowledge in robotic lifelong reinforcement learning. Nature Machine Intelligence 7 (2), p. 256–269. External Links: ISSN 2522-5839, Document Cited by: §I-C. [32] S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert (2017) ICaRL: incremental classifier and representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2001–2010. Cited by: §I-C. [33] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, Vol. 15, p. 627–635. Cited by: §I, §I, §I-B. [34] O. Scheel, L. Bergamini, M. Wolczyk, B. Osiński, and P. Ondruska (2022) Urban driver: learning to drive from real-world demonstrations using policy gradients. In Proceedings of the Conference on Robot Learning, Vol. 164, p. 718–728. Cited by: §V-C, TABLE I. [35] S. Shang, Y. Chen, Y. Wang, Y. Li, and Z. ZHANG (2025) DriveDPO: policy learning via safety dpo for end-to-end autonomous driving. In Advances in Neural Information Processing Systems, Vol. 38, p. 81565–81585. Cited by: §I, §I-B, §I-A. [36] Z. Song, L. Liu, H. Pan, B. Liao, M. Guo, L. Yang, Y. Zhang, S. Xu, C. Jia, and Y. Luo (2026) DIVER: reinforced diffusion breaks imitation bottlenecks in end-to-end autonomous driving. arXiv preprint arXiv:2507.04049. Cited by: §I-B. [37] Q. Sun, H. Wang, J. Zhan, F. Nie, X. Wen, L. Xu, K. Zhan, P. Jia, X. Lang, and H. Zhao (2025) Generalizing motion planners with mixture of experts for autonomous driving. In Proceedings of the IEEE International Conference on Robotics and Automation, p. 6033–6039. Cited by: §I-A. [38] W. Sun, X. Lin, Y. Shi, C. Zhang, H. Wu, and S. Zheng (2025) Sparsedrive: end-to-end autonomous driving via sparse scene representation. In Proceedings of the IEEE International Conference on Robotics and Automation, p. 8795–8801. Cited by: §I. [39] T. Tan, Y. Zheng, R. Liang, Z. Wang, K. Zheng, J. Zheng, J. Li, X. Zhan, and J. Liu (2025) Flow matching-based autonomous driving planning with advanced interactive behavior modeling. In Advances in Neural Information Processing Systems, Vol. 38, p. 38310–38335. Cited by: §I, §I-A, §V-C, TABLE I. [40] X. Tang, M. Kan, S. Shan, and X. Chen (2026) Plan-r1: safe and feasible trajectory planning as language modeling. arXiv preprint arXiv:2505.17659. Cited by: §I, §I-B, §V-C, TABLE I. [41] G. M. van de Ven, T. Tuytelaars, and A. S. Tolias (2022) Three types of incremental learning. Nature Machine Intelligence 4 (12), p. 1185–1197. External Links: ISSN 2522-5839, Document Cited by: §I-C. [42] J. Wang, Y. Zheng, X. Liu, Z. Xing, P. Li, K. Ma, H. Ye, G. Chen, G. Li, L. Chen, Z. Xia, and Q. Zhang (2026) MeanFuser: fast one-step multi-modal trajectory generation and adaptive reconstruction via meanflow for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17884–17893. Cited by: §I. [43] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5362–5383. External Links: ISSN 1939-3539, Document Cited by: §I-C. [44] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C. Lee, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister (2022) Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 139–149. Cited by: §I-C. [45] Z. Wang, Z. Zhang, C. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister (2022) DualPrompt: complementary prompting for rehearsal-free continual learning. In Proceedings of the European Conference on Computer Vision, p. 631–648. Cited by: §I-C. [46] J. Wu, Y. Zhou, H. Yang, Z. Huang, and C. Lv (2023) Human-guided reinforcement learning with sim-to-real transfer for autonomous navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (12), p. 14745–14759. External Links: ISSN 1939-3539, Document Cited by: §I, §I-B. [47] Y. Xing, Z. Ke, Y. Tu, Z. Liu, W. Yu, and J. Wang (2026) MISTY: high-throughput motion planning via mixer-based single-step drifting. arXiv preprint arXiv:2604.21489. Cited by: §I-A. [48] H. Yang, Y. Zhou, J. Wu, H. Liu, L. Yang, and C. Lv (2025) Human-guided continual learning for personalized decision-making of autonomous driving. IEEE Transactions on Intelligent Transportation Systems 26 (4), p. 5435–5447. External Links: Document Cited by: §I-C. [49] J. Yoon, E. Yang, J. Lee, and S. J. Hwang (2018) Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference on Learning Representations, Cited by: §I-C. [50] D. Zhang, Z. Yuan, Z. Chen, C. Liao, Y. Chen, F. Shen, Q. Zhou, and T. Chua (2025) Reasoning-vla: a fast and general vision-language-action reasoning model for autonomous driving. arXiv preprint arXiv:2511.19912. Cited by: §I, §I, §I-A. [51] D. Zhang, J. Liang, K. Guo, S. Lu, Q. Wang, R. Xiong, Z. Miao, and Y. Wang (2025) CarPlanner: consistent auto-regressive trajectory planning for large-scale reinforcement learning in autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17239–17248. Cited by: §I, §I-B. [52] J. Zhang and K. Cho (2017) Query-efficient imitation learning for end-to-end autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, p. 2891–2897. Cited by: §I-B. [53] Z. Zhang, Y. Li, N. Zhang, and J. Cai (2026) Diffusion forcing planner: history-annealed planning with time-dependent guidance for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 39796–39805. Cited by: §I, §I-A, §V-C, TABLE I, TABLE I. [54] Z. Zhang, P. Karkus, M. Igl, W. Ding, Y. Chen, B. Ivanovic, and M. Pavone (2025) Closed-loop supervised fine-tuning of tokenized traffic models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 5422–5432. Cited by: §I, §I-B. [55] Y. Zheng, R. Liang, K. ZHENG, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu (2025) Diffusion-based planning for autonomous driving with flexible guidance. In Proceedings of the International Conference on Learning Representations, Cited by: §I, §I, §I-A, §V-C, TABLE I. [56] Y. Zheng, T. Tan, B. Huang, E. Liu, R. Liang, J. Zhang, J. Cui, G. Chen, K. Ma, H. Ye, L. Chen, Y. Zhang, X. Zhan, and J. Liu (2026) Unleashing the potential of diffusion models for end-to-end autonomous driving. arXiv preprint arXiv:2602.22801. Cited by: §I-B. [57] Y. Zheng, Z. Xing, Q. Zhang, B. Jin, P. Li, Y. Zheng, Z. Xia, Y. Chen, and D. Zhao (2026) PlanAgent: a multi-modal large language agent for closed-loop vehicle motion planning. IEEE Transactions on Cognitive and Developmental Systems (), p. 1–14. External Links: Document Cited by: §I. [58] D. Zhou, H. Sun, J. Ning, H. Ye, and D. Zhan (2024) Continual learning with pre-trained models: a survey. In Proceedings of the International Joint Conference on Artificial Intelligence, p. 8363–8371. Cited by: §I-C. [59] X. Zhou, X. Han, F. Yang, Y. Ma, V. Tresp, and A. Knoll (2026) OpenDriveVLA: towards end-to-end autonomous driving with large vision language action model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 13782–13790. Cited by: §I, §I, §I-A. [60] Z. Zhou, T. Cai, S. Zhao, Y. Zhang, Z. Huang, B. Zhou, and J. Ma (2025) AutoVLA: a vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In Advances in Neural Information Processing Systems, Vol. 38, p. 27920–27956. Cited by: §I, §I, §I-A.