Paper deep dive
Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving
Jiangxin Sun, Feng Xue, Teng Long, Chang Liu, Jian-Fang Hu, Wei-Shi Zheng, Nicu Sebe
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 8:58:08 AM
Summary
The paper proposes RaWMPC, a risk-aware world model predictive control framework for end-to-end autonomous driving that eliminates reliance on expert demonstrations. It utilizes a world model to predict consequences of candidate actions and selects low-risk behaviors through explicit risk evaluation, employing a risk-aware interaction strategy and self-evaluation distillation to handle long-tail scenarios and improve generalization.
Entities (9)
Relation Signals (8)
RaWMPC → addresses → End-to-End Autonomous Driving
confidence 95% · we propose a unified framework named Risk-aware World Model Predictive Control (RaWMPC) to address this generalization dilemma
RaWMPC → employs → Risk-Aware Interaction Strategy
confidence 93% · we design a risk-aware interaction strategy that systematically exposes the world model to hazardous behaviors
RaWMPC → uses → World Model
confidence 92% · RaWMPC leverages a world model to predict the consequences of multiple candidate actions
RaWMPC → employs → Self-Evaluation Distillation
confidence 91% · we introduce a self-evaluation distillation method to distill riskavoidance capabilities
RaWMPC → replaces → Imitation Learning
confidence 90% · RaWMPC ... replaces expert action supervision with risk-aware predictive control
RaWMPC → usessimulator → CARLA
confidence 88% · annotations ... are obtained from the simulator (e.g., CARLA)
RaWMPC → evaluatedon → Bench2Drive
confidence 85% · Experiments on Bench2Drive and NAVSIM show that RaWMPC
RaWMPC → evaluatedon → NAVSIM
confidence 85% · Experiments on Bench2Drive and NAVSIM show that RaWMPC
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:With advances in imitation learning (IL) and large-scale driving datasets, end-to-end autonomous driving (E2E-AD) has made great progress recently. Currently, IL-based methods have become a mainstream paradigm: models rely on standard driving behaviors given by experts, and learn to minimize the discrepancy between their actions and expert actions. However, this objective of "only driving like the expert" suffers from limited generalization: when encountering rare or unseen long-tail scenarios outside the distribution of expert demonstrations, models tend to produce unsafe decisions in the absence of prior experience. This raises a fundamental question: Can an E2E-AD system make reliable decisions without any expert action supervision? Motivated by this, we propose a unified framework named Risk-aware World Model Predictive Control (RaWMPC) to address this generalization dilemma through robust control, without reliance on expert demonstrations. Practically, RaWMPC leverages a world model to predict the consequences of multiple candidate actions and selects low-risk actions through explicit risk evaluation. To endow the world model with the ability to predict the outcomes of risky driving behaviors, we design a risk-aware interaction strategy that systematically exposes the world model to hazardous behaviors, making catastrophic outcomes predictable and thus avoidable. Furthermore, to generate low-risk candidate actions at test time, we introduce a self-evaluation distillation method to distill riskavoidance capabilities from the well-trained world model into a generative action proposal network without any expert demonstration. Extensive experiments show that RaWMPC outperforms state-of-the-art methods in both in-distribution and out-of-distribution scenarios, while providing superior decision interpretability.
Tags
Links
- Source: https://arxiv.org/abs/2602.23259v1
- Canonical: https://arxiv.org/abs/2602.23259v1
Trouble viewing inline? Open PDF directly →
Full Text
83,266 characters extracted from source content.
Expand or collapse full text
Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving Jiangxin Sun 1 , Feng Xue 1 , Teng Long 1 , Chang Liu 1 , Jian-Fang Hu 2 , Wei-Shi Zheng 2 , Nicu Sebe 1 1 University of Trento, Trento, Italy, 2 Sun Yat-sen University, Guangzhou, China jiangxin.sun@unitn.it, feng.xue@unitn.it, teng.long@unitn.it, chang.liu@unitn.it, hujianf@mail.sysu.edu.cn, wszheng@ieee.org, nicu.sebe@unitn.it Abstract With advances in imitation learning (IL) and large-scale driving datasets, end-to-end autonomous driving (E2E-AD) has made great progress recently. Currently, IL-based methods have become a mainstream paradigm: models rely on standard driving behaviors given by experts, and learn to minimize the discrepancy between their actions and expert actions. However, this objective of “only driving like the expert” suffers from limited generalization: when encountering rare or unseen long-tail scenarios outside the distribution of expert demonstrations, models tend to produce unsafe decisions in the absence of prior experience. This raises a fundamental question: Can an E2E-AD system make reliable decisions without any expert action supervision? Motivated by this, we propose a unified framework named Risk-aware World Model Predictive Control (RaWMPC) to address this generalization dilemma through robust control, without reliance on expert demonstrations. Practically, RaWMPC leverages a world model to predict the consequences of multiple candidate actions and selects low-risk actions through explicit risk evaluation. To endow the world model with the ability to predict the outcomes of risky driving behaviors, we design a risk-aware interaction strategy that systematically exposes the world model to hazardous behaviors, making catastrophic outcomes predictable and thus avoidable. Furthermore, to generate low-risk candidate actions at test time, we introduce a self-evaluation distillation method to distill risk- avoidance capabilities from the well-trained world model into a generative action proposal network without any expert demonstration. Extensive experiments show that RaWMPC outperforms state-of-the-art methods in both in-distribution and out-of-distribution scenarios, while providing superior decision interpretability. Date: February 27, 2026 Correspondence: Feng Xue at feng.xue@unitn.it 1 Introduction End-to-end autonomous driving (E2E-AD) [26,56, 62,63,81] aims to map sensor observations to control actions, e.g., steering, throttle, and brake, without relying on hand-crafted perception, prediction, or planning modules. Compared to traditional modular pipelines [48,51,53,74,76,101], E2E-AD offers a more unified representation of the driving task, en- abling the policy to reason about complex interactions between the ego vehicle and dynamic environments. As such, E2E-AD has attracted growing attention due to its potential for simplified system design, joint optimization, and real-time decision-making. 1 arXiv:2602.23259v1 [cs.CV] 26 Feb 2026 OurResultModel-basedReinforcementLearningResult GuidanceWorldModelEvaluation Behavior Cloning Agent Expert KnowledgeDistillation InputAction Actor Critic WorldModel RewardValue InputAction Imaginary Input CandidatesInputAction SafeRisk ContrastiveLearning RandomRisk Action (a)ImitationLearning(IL)(b)Model-basedReinforcementLearning(MBRL)(c)OurMethod ! OutputActionSequences High-costActionSequences × Crash ! Brake ImitationLearningResult 321 × ！ ！ Offroad Figure 1 Comparison between existing E2E-AD methods and RaWMPC. The first row shows the predicted trajectories, and the second row compares the core workflows. Black arrows denote test-time execution, while pink arrows indicate training-only steps. The comparison shows that prior methods often omit explicit hazard modeling and may trigger traffic violations, whereas RaWMPC uses a risk-aware world model to evaluate action consequences and select safe, compliant actions in critical scenes. Early studies on E2E-AD [4,6,75] primarily fo- cused on learning driving policies via reinforcement learning (RL) through online exploration. More re- cent research [5,95] demonstrated that rule-based or RL-based agents leveraging privileged informa- tion (e.g., bird’s-eye-view segmentation and high- definition maps) can produce superior driving deci- sions. Building upon these insights, state-of-the-art methods [28,62,63,81,90] generally follow an im- itation learning (IL) framework as shown in Fig. 1, where agents using only sensor inputs (e.g., RGB images and LiDAR) are trained to replicate the priv- ileged experts’ behavior through knowledge distilla- tion on both the driving policy and latent features. Although some works have attempted to enhance the driving performance via future motion model- ing [55,62,63,73], action-aware future prediction [23,28,38,39], and the integration of large language models [14,21,35,57], these approaches still adhere to the learning objective of “driving like an expert”, as demonstrated in Fig. 1 (a). They cannot funda- mentally resolve the inherent generalization dilemma of imitation learning: Since expert demonstrations cannot cover all scenarios and situations, imitation- based policies tend to produce unpredictable and often unsafe driving behavior when encountering unseen scenarios outside of expert demonstrations. More recently, model-based RL methods [37,85] have emerged and attempted to improve generalization by learning environmental dynamics and planning over them. However, as illustrated in Fig. 1 (b), most of them still aim to maximize the expected reward and lack explicit modeling and sampling of rare but high-risk situations, and thus continue to struggle to guarantee safety in these scenarios. In this paper, we argue that “enabling an E2E-AD system to learn and proactively avoid risky actions is more important than replicating expert driving behav- ior verbatim”. Motivated by this perspective, we pro- pose an E2E-AD framework that does not require any expert action supervision, called Risk-aware World Model Predictive Control (RaWMPC), as shown in Fig. 1(c). This framework discards expert demon- strations and instead leverages a risk-aware world model to drive predictive control to overcome the generalization challenge. Different from model-based RL that trains an actor to maximize reward from imagined rollouts of a world model, the world model in RaWMPC predicts near-future states for a set of “candidate” driving behaviors and explicitly evaluates their risk, so as to select the lowest-risk candidate. To endow our world model with risk-awareness, we introduce a risk-aware interaction strategy: Starting from scratch, the model selects self-identified high- risk actions to interact with the environment, from which our world model learns to predict the conse- quences of diverse risky behaviors. Without any ex- pert demonstration, our world model can reach strong performance from scratch, and can be further acceler- ated if a few video clips are provided by benchmarks 2 for a light warm-up. Finally, to efficiently provide low-risk candidates at test time, we further propose a self-evaluation distillation for driving policy learning. The well-trained world model is leveraged to identify safe and risky behaviors from the sampled action space, and to distill this knowledge, via safety–risk contrastive learning, into a generative action pro- posal network. Experiments on Bench2Drive and NAVSIM show that RaWMPC, even without expert demonstrations, surpasses previous state-of-the-art methods, while the optional light warm-up further helps to accelerate the convergence and improve per- formance. More importantly, since RaWMPC learns risk-awareness from interaction rather than expert labels, it achieves substantially higher driving per- formance in previously unseen scenarios. Our main contributions can be summarized as follows: •We propose RaWMPC, an E2E-AD framework with zero expert requirement. Unlike IL and MBRL methods, RaWMPC uses a world model to select low-risk behaviors from multiple can- didate actions, which naturally improves the interpretability and reliability of its decisions. •Within RaWMPC, we design a risk-aware inter- action learning strategy that enables the world model to acquire risk-awareness purely from envi- ronment interaction, without any expert demon- stration. •We introduce a self-evaluation distillation scheme for driving policy learning, which pro- vides high-quality candidate actions at test time, even outperforming policies learned directly from expert demonstrations. 2 Related Work 2.1 End-to-End Learning in Autonomous Driving Learning-based autonomous driving approaches typi- cally follow two main paradigms: imitation learning (IL) and reinforcement learning (RL) [7,9]. Early studies explored RL extensively [2,4,6,22,32,52, 58,59,75,93] for its natural capability to refine driving strategies through interactive feedback. Sub- sequent works [37,95] demonstrated that training sensor-based agents to imitate RL experts’ behavior could lead to superior performance, prompting a shift toward leveraging privileged information (e.g., BEV segmentation and HD maps) to train stronger RL experts. More recently, model-based RL has revis- ited this line by learning explicit dynamics/world models and performing look-ahead rollouts [85], im- proving consequence-aware evaluation and sample efficiency. Nevertheless, these RL approaches are commonly driven by maximizing expected return, and they rarely provide an explicit mechanism to sys- tematically discover and model rare-but-catastrophic outcomes, making reliable decisions in long-tail, high- risk scenarios still challenging. With large-scale driving datasets, imitation learning has attracted increased attention and achieved state- of-the-art performance in closed-loop autonomous driving [40,57,73,82]. Most IL approaches rely on collecting trajectory data and features from RL- based [37,95] or rule-based [10,65] experts using privileged information, and train sensor-based IL agents to replicate expert behaviors through knowl- edge distillation. A variety of research studies have explored ways to improve IL performance, such as multi-modal information fusion [10,26,54], object motion modeling [30,55,62,63], action-aware fu- ture prediction [23,28,38,39], feature alignment [27,90], and the integration of large language models [14,45,56,57,64,65,83,86]. Despite impressive results, the core objective of “driving like the expert” inherently limits generalization: expert demonstra- tions cannot cover all long-tail situations, and ex- perts typically avoid dangerous behaviors, leaving IL agents with limited supervision on how to recognize and proactively avoid high-risk actions. Moreover, pure imitation often provides limited interpretability because it outputs a single action without explic- itly comparing alternative actions via consequence evaluation. Motivated by these insights, we propose RaWMPC, a unified framework that replaces expert action supervision with risk-aware predictive control: it learns a risk-aware world model via a risk-aware interaction strategy that deliberately exposes the model to risky behaviors, and uses the learned model to predict and evaluate the consequences of multiple candidate actions, selecting low-risk behaviors with explicit risk evaluation. 2.2 World Models World models approximate environment transitions under the Markov Decision Process and have demon- strated success in RL [17–20,34,46,75,77,79] by predicting future states and rewards from cur- rent observations and actions. However, apply- ing these models to complex tasks such as au- tonomous driving remains challenging. Previous re- search [1,13,15,24,36,49,50,70–72,78,92,96– 98,102] has mainly leveraged world models to gener- ate controllable future driving trajectories (e.g., RGB 3 images and 3D/4D representations) conditioned on specific actions and scene descriptions. Such predic- tive models can enlarge training data and increase diversity, potentially benefiting downstream imita- tion learning, especially for rare scenes such as traffic accidents. Beyond prediction, a few recent attempts have utilized world models to improve closed-loop au- tonomous driving performance. In particular, some works have started to connect world modeling with planning and online evaluation in driving settings [8,39,91], while others provide high-fidelity gener- ative platforms that enable closed-loop evaluation [3,11,84]. In particular, model-based IL methods [23,38,39,61] employ world models to support bet- ter imitation: LAW [38] enhances end-to-end driving by predicting future information in a latent world model to assist policy learning, and WoTE [39] per- forms online trajectory evaluation via a BEV world model to score candidate trajectories. More recently, model-based RL methods have drawn increasing at- tention. Think2Drive [37] designed a model-based RL expert to forecast action-conditioned future re- wards for training a more effective critic network, and Raw2Drive [85] leveraged a world model pretrained from privileged experts to guide the learning of sensor agents. Despite these advances, most existing works still inherit supervision from experts or rewards, and they largely focus on imitation fidelity or expected- return maximization, lacking explicit mechanisms to systematically discover, model, and avoid rare-but- high-risk outcomes. In contrast, RaWMPC uses the world model as a risk evaluator within predictive control: we introduce a risk-aware interaction strat- egy to intentionally explore risky behaviors so that catastrophic consequences become predictable and avoidable, and we select actions by explicitly mini- mizing risk over multiple candidates, enhancing the interpretability, reliability, and generalization of the decision-making process. 3 Method To demonstrate our solution, Sec. 3.1 introduces the overall network structure and pipeline of our RaWMPC. Sec. 3.2 presents our training scheme, i.e., risk-aware interactive training, for efficient op- timization. To ensure the whole E2E-AD system runs efficiently during testing, Sec. 3.3 illustrates a self-evaluation distillation method to train an action proposal network. 3.1 Network Structure of RaWMPC 3.1.1 Problem Setup We consider a closed-loop end-to-end autonomous driving (E2E-AD) setting. At each time stept, the agent receives three inputs: a visual input I t (multi- view RGB images), an ego-centric measurement M t (velocity and position), and a set of candidate driving behaviorsA n t:t+H−1 N n=1 , whereNis the number of candidates andHis the planning horizon. Each step of action consists of three values: A = (steer∈ [−1,1], throttle ∈[0,1], brake ∈[0,1]). Based on the driving history (I 1:t ,M 1:t ,A 1:t−1 ), RaWMPC aims to select the best one A n ⋆ t:t+H−1 from the can- didates, so that the vehicle can move toward a des- tination while ensuring safety and compliance with traffic rules. 3.1.2 Overview As illustrated in Fig. 2, RaWMPC begins from input encoding. A visual encoder, an action encoder and an ego-state encoder are used to map I t ,A n t:t+H−1 N n=1 , and M t into embeddings i t ,a n t:t+H−1 N n=1 , and m t , respectively. Then, based on the observed states s 1:t = (i 1:t , m 1:t ), we use a world model to estimate its future state ˆ s n t+1:t+H conditioned on each action embeddings a n t:t+H−1 . Finally, we select the action that enables the ego vehicle to advance safely while avoiding traffic infractions, by decoding ˆ s n t+1:t+H and computing a cost value: A ⋆ t:t+H−1 = A n ⋆ t:t+H−1 , where n ⋆ = arg min n∈1,...,N C ˆ s n t+1:t+H . (1) whereC(·) denotes the cost function in the decod- ing process (will be detailed in Eq.(6)), andn ⋆ is the index of optimal action. Compared to imitation learning schemes, RaWMPC offers improved inter- pretability and introduces an explicit mechanism for decision validation and risk mitigation. 3.1.3 World Model In our pipeline, the world model, denoted asM, is employed to predict future states given an action. Specifically, conditioned on the observed states s 1:t = (i 1:t ,m 1:t ) and the potential next action a n t , the world modelMpredicts a near-future state ˆ s n t+1 . Then, for the further-future time step2,...,H, the world model recursively rolls out, which can be formalized as an autoregressive factorization: p M ( ˆ s n t+1:t+H |s 1:t , a n 1:t+H−1 ) = Y H k=1 p M ˆ s n t+k |(s 1:t , ˆ s n t+1:t+k−1 ) , a n 1:t+k−1 (2) 4 World Model Visual Encoder Action Encoder Ego State Encoder 퐢 ! 퐚 !:!#$%& ' 퐦 ! ̂ % !:!#$%& ' & 퐦 !#& SegmentationDecoder EgoStateDecoder ×퐻times SemanticClass PedVehStat ... TrafficEvent Metric EventDecoder 0.970.98 0.45 0.32 0.44 0.17 Evaluation SemanticGuidance ... Col Ped Col Veh Col Stat CandidateActions 퐀 !:!#$%& ' |푛∈[1,푁] Multi-ViewImages퐈 ! ✔ Speed Position EgoState퐌 ! ActionConsequenceEvaluator 6 퐬 !:!#$%& ' SpeedPosition OfflineWorld ModelWarm-up OnlineSimulator InteractionTraining 퐚 ! 0 Action Space Safe RiskyRandom 퐬 ! 퐬 ! Reward 6 퐬 !#& 퐬 !#& Simulator 퐬 !#& 6 퐬 !#& 퐬 ! 퐼 ! 퐼 !#& 휀 & 1-휀 & 휀 ( 1-휀 ( Figure 2 Overview of RaWMPC. Multi-view images I t , ego state M t , and candidate action sequencesA n t:t+H−1 N n=1 are encoded and rolled out by a world model over horizonH. Three decoders predict semantic segmentation, semantic- guided traffic events, and future ego states, enabling action evaluation for predictive control. Training combines offline warm-up on logged trajectories with online simulator interaction using world-model-guided exploration. wherep M denotes the conditional distribution defined by the world modelM. a n 1:t+k−1 = [a 1:t−1 ,a n t:t+k−1 ] denotes a candidate action sequence containing action history. 3.1.4 Semantic-Guided Decoding Once the world model predicts a sequence of future states, ˆ s n t+1:t+H , three transformer decoders sepa- rately map these states to task-specific outputs: se- mantic segmentation, potential traffic events (e.g., collision), and future ego-states (e.g., position). In what follows, we describe these decoders using the predicted state at time stept+k, i.e., ˆ s n t+k , where k ∈1,...,H. To enable a higher-level understanding of driving scenes and provide visual explanations for predicted traffic events, we inject semantic attention from the segmentation decoder into the event decoder. The segmentation decoder adopts a standard transformer attention: Att seg (Q c ,K c ,V c )=softmax(sim(Q c , K c ))·V c , (3) where Q c are learnable class queries, and K c ,V c are derived from visual tokens ˆ i n t+k ⊂ ˆ s n t+k . In the final layer, we follow SegViT [88] to predict the one-hot semantic segmentation map ˆ Y n t+k for each input class query. We then augment the event decoder by fusing its attention logits with the corresponding semantic attention logits from the last segmentation layer: Z e = Q e K ⊤ e , Z c = pad(Q c K ⊤ c ), ˆ E n t+k = sigmoid(softmax(W e ∗ [Z e , Z c ])V e ). (4) where Q e are learnable event queries, and K e ,V e are computed from predicted future states ˆ s n t+k .pad(·) zero-pads Z c to match the size of Z e , and W e is a 1×1 convolution that fuses the concatenated logits. The output of the event decoder, ˆ E n t+k ∈ [0,1] α , represents the probabilities ofαevent types. Finally, for the future ego-state prediction, we decode the ego token ˆ m n t+k ⊂ ˆ s n t+k to obtain the speed and position ˆ M n t+k . Guided by the semantic attention map, the event decoder draws more attention to regions critical to specific events. For instance, when recognizing the vehicle collision event, the model focuses more on vehicle areas, improving the accuracy and reliability of event predictions. 3.1.5 Action Selection and Predictive Control Given the decoder outputs, we perform predictive control by evaluating each candidate action sequence over the planning horizonHand selecting the one with the minimum predicted cost. Specifically, for then-th candidate A n t:t+H−1 , we con- sider (i) progress toward the target and (i) the risk of traffic-violation events. Let p ⋆ be the target 3D position and ˆ p n t+k ⊂ ˆ M n t+k the predicted ego position at stept+k. We define the progress as the reduction in target distance: ˆ D n t+k = p ⋆ − ˆ p n t+k−1 2 − p ⋆ − ˆ p n t+k 2 , (5) 5 We then define the predictive-control objective as: C( ˆ s n t+1:t+H ) = H X k=1 η k (− ˆ D n t+k + α X j=1 λ j ˆ E n t+k,j ), (6) whereη k =max(2 −k+1 ,1/8) down-weights distant predictions to account for increasing uncertainty. We floorη k at 1/8 to avoid vanishing contributions from distant steps, which stabilizes planning whenHis moderately large.λ j >0 reflects the severity of violation typej(e.g., pedestrian/vehicle collisions receive larger weights). Finally, we select the action sequence that minimizes the horizon cost in Eq.(6), i.e., a model-predictive control policy that favors faster progress while proactively reducing the proba- bility of predicted violations. 3.1.6 Overall Loss of RaWMPC The RaWMPC framework is trained in an end-to-end manner with a world model lossL world , a segmenta- tion lossL seg from SegViT [88], ego-state lossL ego , and event loss L event : L =L world +L seg +L ego +L event .(7) Following SegViT [88], the segmentation term in- cludes classification lossL cls (cross-entropy) and the binary mask loss. The mask loss consists of a focal lossL focal [43] and a dice lossL dice [47] for optimizing the segmentation accuracy: L seg =L cls +L focal +L dice .(8) For the other three losses, we use Mean Squared Error (MSE) to supervise the ego-state and world model supervision, and take Binary Cross Entropy (BCE) loss for the event decoder: L world = 1 H X H k=1 ∥ ˆ s t+k − s t+k ∥ 2 2 , L ego = 1 H X H k=1 ˆ M t+k − M t+k 2 2 , (9) L event = 1 H X H k=1 BCE( ˆ E t+k , E t+k ), whereBCE(·) denotes the BCE loss. All the anno- tations s t+k ,M t+k ,E t+k along the executed rollout under A t:t+H−1 are obtained from the simulator (e.g., CARLA). 3.2 Risk-aware Interactive Training To enable RaWMPC to evaluate diverse actions and identify risky scenarios, we propose a two-stage risk- aware interactive training scheme, as shown in Fig. 2. We first warm-start the world model from logged driving trajectories. Then, we refine it via online sim- ulator interaction, intentionally collecting both good CandidateActionSequences RankedbyCosts HighCostLowCost Randmode Bad modeGoodmode Figure 3 Different action-selection ranges under three driving modes in online simulator interaction. Red denotes high cost and green denotes low cost.rand samples uniformly from all candidates,badsamples from the high-cost region, andgoodsamples from the low-cost one. (safe, goal-directed) and bad (hazardous) rollouts to improve generalization under out-of-distribution controls and to learn rare but safety-critical events. Notably, RaWMPC does not rely on expert action la- bels for policy learning. The optional warm-up stage, when used, serves solely to initialize the predictive world model from observed state transitions, rather than to imitate expert actions. 3.2.1 Offline World Model Warm-up We bootstrap RaWMPC using a small set of logged trajectories to achieve simple and basic state- forecasting capability. Given state-action sequences (s t ,a t ) T t=1 from NAVSIM or CARLA, the world model predicts the next state ˆ s t+1 and is trained with L world to match the ground-truth s t+1 . We supervise the segmentation and ego-state decoders using the simulator-provided annotations. Since the warm-up trajectories contain no traffic violations, we train the event decoder with an all-zero target. In this way, only a small subset of training data (10%) is typically sufficient for warm-up, providing a reliable initialization for long-horizon rollouts and stabilizing subsequent world model optimization. 3.2.2 Online Simulator Interactive Training Offline warm-up data are mostly concentrated around human-like safe behaviors and thus provide limited coverage of hazardous or unconventional actions. To learn the consequences of risky behaviors, we perform world-model-guided exploration: selected simulator rollouts are fed back to refine the same world model, progressively improving prediction fidelity and risk sensitivity. Specifically, to ensure temporal continuity and avoid unrealistic control jitter, we sample horizon-Haction sequences (segment-wise) rather than single-step ac- tions (step-wise). The segment-wise sampling allows 6 sustained safe or risky behaviors to unfold and reveals their long-term consequences. At each training step, we sampleN s horizon-Hcandidate action sequences A n t:t+H−1 N s n=1 , roll out future states ˆ s n t+1:t+H with the current world model, evaluate their costsC n using Eq. (6), and rank candidates by costs. Modes for Interaction. We define three patterns for our risk-aware sampling strategy to select one candidate fromA n t:t+H−1 N s n=1 to execute, as shown in Fig. 3: • Rand samples uniformly from all candidates; • Bad samples from high-cost candidates; • Good samples from low-cost candidates. At the start of segmentr, the practical control mode is sampled according to probability: m r = rand, w.p. ε 1 , bad, w.p. (1− ε 1 )ε 2 , good, w.p. (1− ε 1 )(1− ε 2 ), (10) where “w.p.” means “with probability”.ε 1 con- trols broad action-space exploration, andε 2 controls the fraction of risk-seeking interaction within model- guided sampling. Soft Candidates Selection in Three Modes. Given sorted candidate actions with costsC n , we con- struct two cost-quantile sets:N good as the bottom-K candidates andN bad as the top-Kcandidates. To avoid low-information trajectories, we filterN bad by removing degenerate rollouts (e.g., those caused by unrealistic excessive control jumps), yielding ̃ N bad . In therandmode, we randomly select the executing action sequence from all candidates. In thegoodmode, rather than deterministically se- lecting the minimum-cost candidate, we sample from N good using a soft distribution to preserve diversity among low-cost plans and mitigate bias from imper- fect model predictions: P(n| good)∝ exp(−C n /τ g ), n∈N good . (11) whereτ g is a temperature hyper-parameter that con- trols the softness of the sampling distributions in thegoodmode, trading off greediness for diversity. This stochastic selection avoids repeatedly executing a single estimated optimum and encourages broader coverage of nominal behaviors. In thebadmode, instead of always executing the maximum-cost trajectory, we sample from high-cost Contrastivelearningondistribution 휇 c 휇 + 휇 - 휎 휇 휎 + 휇 + 휎 휇 휎 - 휇 - Shared Reconstruction Negative Action Sequences Positive Action Sequence Random Action Sequences Visual Encoder Ego State Encoder Condition Encoder 휎 c 휇 c Encoder ConditionPart Decoder Action Proposal Contrastive Supervision Test-onlyFlow Main Flow Encoder VAE Part Figure 4 Self-Evaluation Distillation for Policy Learn- ing. A cVAE is trained with RaWMPC-scored actions in a contrastive manner, pulling the condition prior toward positives and pushing it away from negatives. The well- trained decoder serves as the test-time action proposer. candidates to deliberately expose the model to a spec- trum of risky outcomes that are under-represented in safe logs: P(n| bad)∝ exp(C n /τ b ), n∈ ̃ N bad . (12) whereτ b is a temperature hyper-parameter. Com- pared to argmax selection, this soft sampling strategy prevents over-concentration on extreme or degenerate failures while still biasing interaction toward high-risk regions. In this way, segment-wise interaction and soft cost- based sampling bias exploration toward temporally coherent and informative safe and hazardous tra- jectories, enabling the world model to learn both reasonable dynamics and safety-critical consequences for risk-aware decision making. 7 3.3Self-Evaluation Distillation for Policy Learning After risk-aware interactive training, RaWMPC can reliably score candidate action sequences by predict- ing their long-horizon consequences. To reduce the cost of online optimization at test time, we distill this evaluation capability into a lightweight action proposal network, enabling efficient inference without expert demonstrations. The action proposal network corresponds to the “Guidance” module illustrated in Fig. 1(c), and is used to generate candidate action sequences for predictive control. Our key idea is to use RaWMPC as a self-evaluator to pseudo-label sampled actions and train a generative policy via contrastive learning. 3.3.1 Action Sampling and Pseudo-labeling Given a state history s 1:t (simplified as s), we randomly sampleN s horizon-Haction sequences A n t:t+H−1 N s n=1 (simplified asA n ) and compute their costsC n with the pretrained RaWMPC. We then form pseudo labels by ranking costs: the lowest- cost sequence is treated as a positive example A + , and the top-Khighest-cost sequences are treated as negativesA − j K j=1 . This construction transfers RaWMPC’s knowledge (low-risk / high-quality ac- tions) to the proposal network while avoiding any external supervision. 3.3.2 Action Proposal Network Following [60,87], we adopt a conditional VAE (cVAE) with an action encoderq θ (z|A,s), a con- ditional priorp γ (z|s), and a decoderp ψ (A|z,s). The decoder serves as the proposal policy at inference. For the positive action, we obtain a Gaussian pos- teriorq + =q θ (z|A + , s) =N(μ + , diag((σ + ) 2 )) and train the decoder to reconstruct A + . For each neg- ative action A − j , we computeq − j =q θ (z|A − j ,s) = N(μ − j , diag((σ − j ) 2 )), but do not reconstruct nega- tives to prevent the generator from imitating unsafe behaviors. The conditional prior isp c =p γ (z|s) = N(μ c , diag((σ c ) 2 )). 3.3.3 Contrastive Training Objective To address the lack of expert supervision in policy learning, we use an InfoNCE objective to make the conditional prior predictive of high-quality actions. Concretely, there are two potential contrastive for- mulations: •Usingp c as the anchor, pullingp c towardq + while pushing it away from q − K j=1 . •Usingq + as the anchor, pullingq + towardp c while pushing it away from q − K j=1 . We empirically found that the former often produces under-optimized trajectories. One possible reason is that negative samples are far more numerous and broadly cover the latent space, thereforep c is easily driven to a region that is far from most negatives yet not sufficiently close to the positive. In contrast, the latter explicitly pullsp c toward the statistical center of the positive posterior and, viaq + , indirectly sep- arates it from the negatives, leading to more stable learning and higher-quality trajectories. Therefore, we adopt the latter design choice to define our In- foNCE objective as: L c =− log exp(ℓ + ) exp(ℓ + ) + P K j=1 exp(ℓ − j ) , ℓ + =−D(q + ,p c )/τ, ℓ − j =−D(q + ,q − j )/τ, (13) whereD(·,·) is the Wasserstein-2 distance between Gaussians and τ is a temperature. 3.3.4Overall Loss of Action Proposal Network The total loss of our cVAE combines reconstruction, KL regularization, and contrastive loss: L total =E z∼q + − logp ψ (A + | z, s) + β D KL q + ∥p c + λL c . (14) This self-evaluation distillation trains a fast proposal policy that generates candidate action sequences con- sistent with RaWMPC’s evaluations, eliminating the need for expert demonstrations during policy learn- ing. 4 Experiments In this section, we present a comprehensive perfor- mance comparison between the proposed framework and state-of-the-art methods. We also conduct ex- tensive ablation studies to assess the effectiveness of our predictive control approach. 4.1 Benchmarks Following prior works [39,40,61], we evalu- ate RaWMPC on two widely used benchmarks: Bench2Drive [29] and NAVSIM [11]. They are com- plementary: Bench2Drive provides fully interactive 8 Table 1 Comparison with SOTA approaches on the closed-loop Bench2Drive benchmark on CARLA simulator.↑ means the higher the better. DS is taken as the primary metric in comparison and we rank all the methods accordingly, with bold indicating best performance. MethodVenueSchemeDS↑SR(%)↑ Efficiency↑ Comfortness↑ VAD [31]ICCV 2023IL42.35 15.00157.9446.01 SparseDrive [69]ICRA 2025IL44.54 16.71170.2148.63 GenAD [99]ECCV 2024IL44.81 15.90-- UniAD [25]CVPR 2023 IL45.81 16.36129.2143.58 MomAD [68]CVPR 2025IL47.91 18.11174.9151.20 UAD [16]T-PAMI 2025IL49.22 20.45189.5352.71 BridgeAD [89]CVPR 2025IL50.06 22.73-- TCP [81]NeurIPS 2022IL59.90 30.0076.5418.08 WoTE [39]ICCV 2025IL61.71 31.36-- DriveDPO [61]NeurIPS 2025IL & RL62.02 30.62166.8026.79 ThinkTwice [28]CVPR 2023 IL62.44 31.2369.3316.22 DriveTransformer [30]ICLR 2025IL63.46 35.01100.6420.78 DriveAdapter [27]ICCV 2023 IL64.22 33.0870.2216.01 Raw2Drive [85]NeurIPS 2025 RL71.36 50.24214.1722.42 Hydra-NeXt [40]ICCV 2025IL73.86 50.00197.7620.68 HiP-AD [73]ICCV 2025IL86.77 69.09203.1219.36 RaWMPC w/o Warm-up-PC87.3469.62203.2530.95 RaWMPC- PC88.3170.48206.8532.65 Pretrained VLM-based Approach ReAL-AD [44]ICCV 2025 IL41.17 11.36-- Dual-AEB [94]ICRA 2025 IL45.23 10.00-- ETA [21]ICCV 2025IL74.33 48.33186.0425.77 VLR-Drive [35]ICCV 2025IL75.01 50.00122.520.59 ORION [14]ICCV 2025 IL77.74 54.62151.4817.38 SimLingo [57]CVPR 2025IL85.94 66.82244.1830.76 closed-loop evaluation in CARLA [12] with dense annotations, while NAVSIM evaluates large-scale real-world planning via a data-driven, non-reactive simulation-based short-horizon rollout with safety- and progress-aware metrics. Bench2Drive. Bench2Drive is a CARLA Leadboard v2 closed-loop benchmark for multi-ability stress test- ing under complex interactions (e.g., cut-ins, over- takes, detours, emergency braking, and give-way). Its official dataset contains∼2M fully annotated frames from short clips spanning 44 scenarios, 23 weather conditions, and 12 towns; the commonly used Base training set contains 1K clips. Closed-loop evaluation is performed on 220 short routes (each focused on a single scenario), enabling stable and fine-grained comparison. We report four official metrics: Driv- ing Score (DS), Success Rate (SR), Efficiency, and Comfortness. DS is the primary aggregate score with penalties for safety and rule violations; SR measures successful completion; Efficiency reflects progress; and Comfortness captures motion smoothness. NAVSIM. NAVSIM benchmarks sensor-based plan- ning on large-scale real-world data built on Open- Scene (a planning-oriented reprocessing of nuPlan logs). The task predicts a 4-second future ego tra- jectory (typically 8 waypoints) given a short his- tory (e.g., 1.5 seconds) of observations. Following prior works [39,61], we use the official splits: Nav- train (∼103K samples) and Navtest (∼12K samples). We report NAVSIM metrics including NC, DAC, EP, TTC, C, and the primary score PDMS, where PDMS=NC· DAC·(5EP+ 5TTC+ 2C)/12. These metrics jointly capture safety, compliance, progress, and motion quality. All results are computed with the official toolkits and recommended splits. Our experiments combine offline warm-up with online simulator interaction, leveraging RaWMPC to learn predictive dynamics and risk-aware decision making. We adopt Bench2Drive for interactive closed-loop evaluation and NAVSIM for large-scale real-world generalization. We further conduct ablations to ana- lyze key training components and strategies. 9 Table 2 Comparison with the SOTA approaches on NAVSIM test set.↑means the higher the better. PDMS is taken as the primary metric in comparison and we rank all the methods accordingly, with bold indicating best performance. MethodVenueSchemeNC↑ DAC↑ EP↑ TTC↑C↑PDMS↑ Human--100 100 87.5 100 99.994.8 DrivingGPT [8]ICCV 2025IL98.990.7 79.7 94.9 100.082.4 UniAD [25]CVPR 2023IL97.8 91.9 78.8 92.9 100.083.4 Latent TransFuser [10] T-PAMI 2023 IL97.4 92.8 79.0 92.4 100.083.8 PARA-Drive [80]CVPR 2024IL97.9 92.4 79.3 93.0 99.884.0 TransFuser [10]T-PAMI 2023IL97.7 92.7 79.8 92.7 100.084.5 LAW [38]ICLR 2025 IL96.4 95.4 81.7 88.7 99.984.6 World4Drive [100]ICCV 2025IL97.4 94.3 79.9 92.8 100.085.1 DiffusionDrive [42]CVPR 2025IL98.2 96.288.2 94.7 100.088.1 WoTE [39]ICCV 2025 IL98.5 96.8 81.9 94.9 99.988.3 Hydra-NeXt [40]ICCV 2025IL98.1 97.7 81.8 94.6 100.088.6 UAD [16]T-PAMI 2025IL99.5 96.9 78.897.5 100.089.3 DriveDPO [61]NeurIPS 2025IL & RL98.5 98.1 84.3 94.8 99.990.0 GoalFlow [82]CVPR 2025IL98.498.3 85.0 94.6 100.090.3 RaWMPC w/o Warm-up-PC98.3 98.285.3 94.5 99.990.5 RaWMPC-PC98.998.3 86.195.699.991.3 4.2 Implementation Details Network architecture. We employ a pretrained ViT [66] as the vision encoder and use the SegViT segmentation head [88] as the segmentation decoder. BEV features are extracted from multi-view images using the query-based view transformer [41]. Fol- lowing previous works [10,39], the input image res- olution is 1024×256, and the BEV feature map resolution is 256×256. We use down-sampling fac- tors of 32 (front-view) and 16 (BEV), resulting in 512 visual tokens i t (256 per branch). Measurement inputs are encoded through an MLP-based encoder into 4 measurement tokens m t , and driving actions, represented as three scalar values (steer, throttle, brake), are transformed into 3 action tokens a t via a linear layer. We implement the world model as a transformer with 4 layers and 8 attention heads. RaWMPC predictsH=10 future steps conditioned on the past 5 observed steps, and evaluatesN=10 candidate action sequences proposed by the distilled action proposal network (described in Section 3.3) using the cost in Eq.(6)during inference. During action selection, we useη k =max(2 −k+1 ,1/8) to downweight distant predictions, and set the event severity weights toλ j =10, 15, 30for off-lane driving, traffic-sign violations, and collisions, respectively. Training. RaWMPC is trained with the two-stage risk-aware interactive training strategy described in Section 3.2. We first perform an offline warm-up using 10% of the training data (100 clips for Bench2Drive and 10K samples for NAVSIM), and then refine the model via online interaction using the proposed risk- aware training scheme. The random-sampling proba- bilityε 1 is linearly annealed from 1 to 0, while the risk- sampling probabilityε 2 is linearly increased from 0 to 0.3. We maintain a replay buffer of size 10K frames to store recent interaction data (e.g., RGB images, se- mantic segmentation, ego measurements, and traffic- event annotations). We use segment-wise sampling of horizon-H=10 action sequences to ensure temporal continuity. In good/bad modes, we rank the N s =50 candidates by cost and sample from the bottom/top- K=5 sets using temperaturesτ g = 0.5 andτ b = 1.0. An episode terminates when any of the following con- ditions is met: (1) the ego vehicle incurs 3 collisions, (2) the ego vehicle goes off-road or remains stuck for 100 consecutive steps, or (3) the ego vehicle success- fully completes the route. Across offline warm-up and online interaction, we train RaWMPC using a total of 1K clips on Bench2Drive and 100K samples on NAVSIM (comparable in scale to the official train- ing sets for fair comparison), on four NVIDIA A100 GPUs. We use Adam [33] with an initial learning rate of 10 −4 decayed to 10 −5 , and a batch size of 16. Self-evaluation distillation. For self-evaluation distil- lation (Section 3.3), the action proposal network is implemented as a cVAE [67] with a 32-dimensional latent space. During contrastive training, we sam- pleN s =50 action sequences, treat the minimum-cost 10 sequence as the positive example A + , and use the top-K=5 highest-cost sequences as negatives. We optimize the proposal network with the objective in Section 3.3, using temperatureτ=0.3, a KL weight scheduleβ: 0→0.1, and a contrastive weightλ=0.1. At inference, we sampleN=10 candidates from the cVAE decoder and select the final action by mini- mizing the predicted cost (Eq.(6)) under the world- model rollout. 4.3Comparison with the State-of-the- Art In this section, we compare RaWMPC with state-of- the-art end-to-end driving methods on the closed-loop Bench2Drive benchmark in CARLA and the NAVSIM test set. To reflect our main claim, we report two training settings: (i) w/o warm-up, where RaWMPC is trained without using any offline logged video, and (i) the full setting, where a small set of logged driving trajectories is used as an optional warm start. The warm start empirically accelerates convergence and further improves final performance, while RaWMPC already surpasses prior state-of-the-art even without it. 4.3.1 Evaluation on Bench2Drive Table 1 illustrates the closed-loop results on Bench2Drive. RaWMPC achieves the best overall performance among all compared methods, reaching 88.31 DS and 70.48% SR in the full setting. More importantly, even without warm-up (i.e., without logged trajectories), RaWMPC still attains87.34DS and 69.62% SR, surpassing strong recent baselines such as HiP-AD [73] (86.77 DS / 69.09% SR) and the pretrained-VLM method SimLingo [57] (85.94 DS / 66.82% SR). In addition, RaWMPC maintains com- petitive efficiency and achieves higher comfortness than most high-performing closed-loop agents, indi- cating that the gains are not obtained by aggressive maneuvers but by more reliable decision-making. 4.3.2 Evaluation on NAVSIM Table 2 summarizes the results on NAVSIM. RaWMPC achieves the highest PDMS of 91.3 among all learning-based methods. Without warm-up, RaWMPC still reaches90.5PDMS, already outper- forming previous best methods (e.g., GoalFlow [82]: 90.3). The warm start further improves PDMS (90.5→91.3), consistent with the observation that a small amount of logged trajectories can acceler- ate convergence and improve performance, while not being required to achieve state-of-the-art results. 4.3.3Generalization under weather-induced domain shift To evaluate robustness beyond the training distribu- tion, we conduct a weather-shift study where all meth- ods are evaluated exclusively on Rainy scenarios while being trained on either Sunny only or Sunny & Rainy data (Table 3). A key observation is that imitation- based methods are sensitive to training-domain cover- age: when rainy conditions are absent from training (Sunny-only), their performance drops notably, re- flecting limited transfer to previously unseen environ- ments. In contrast, RaWMPC achieves the best DS and SR under both training regimes, and notably still significantly outperforms strong IL baselines (LAW, WoTE) and SimLingo when trained on Sunny only (i.e., facing an unseen rainy target domain). More- over, compared with SimLingo, RaWMPC exhibits substantially smaller degradation when rainy data is removed from training, indicating stronger robustness to previously unseen conditions. Figure 5 shows a qualitative example of this Sunny- only→Rainy shift. LAW fails to recognize the lead vehicle under the altered visual conditions and results in a high-severity frontal collision. WoTE and SimLingo attempt evasive maneuvers that re- duce the impact relative to a direct frontal crash, yet the scene still ends in a side-swipe/rear collision. A plausible explanation is that the rainy shift degrades the reliability of the perception–decision stack, while the downstream policy does not explicitly optimize for minimum-risk clearance under uncertainty, yield- ing insufficient safety margins during close-proximity avoidance (e.g., inaccurate motion anticipation or mismatched ego response on wet roads). By con- trast, RaWMPC explicitly evaluates the predicted consequences of candidate action sequences with a risk-aware world model and selects the minimum-risk predictive-control behavior, maintaining safe clear- ance under uncertainty. We attribute this advantage to RaWMPC’s risk- aware predictive-control formulation: instead of re- producing expert actions, RaWMPC learns risk- awareness from interaction and selects actions by minimizing predicted risk via the learned world model. Such an objective encourages transferable decision principles (e.g., maintaining safe margins and acting conservatively under uncertainty) that remain effec- tive when appearance and dynamics change across do- mains. Thus, RaWMPC is less dependent on exhaus- tive expert action coverage for corner cases, which better matches the long-tail nature of real-world de- ployment where unseen scenarios are inevitable. 11 Make a direct left turnCollidewith the vehicle ahead LAW Make a slight left turnSideswipethe vehicle WoTE Make a wide left turnCollidewith the rear of the vehicle Simlingo Steer right one half turn, then turn leftAvoid the vehicle ahead and complete the left turn safely RaWMPC Front - view camera t+1t+2t+3t+4 t+6t+7t+8t+5 Vehicle Road Lanelines Sidewalk Comparison Figure 5 Qualitative comparison under weather-induced domain shift (Sunny-only→Rainy). All methods are trained on Sunny-only data and evaluated in Rainy conditions. LAW [38] misses the lead vehicle, causing a severe frontal collision. WoTE [39] and SimLingo [57] reduce severity by evasive maneuvers but still collide due to degraded perception–decision reliability and weak safety-margin enforcement. RaWMPC avoids collisions by selecting the minimum-risk predictive-control action under uncertainty. Table 3 Performance comparison under domain shift. All methods are trained on either Sunny only or Sunny & Rainy data, and evaluated exclusively on Rainy scenarios. ↑ indicates higher is better. MethodVenue Scheme Training Data Tested on Rainy DS↑SR(%)↑ LAWICLR 2025IL Sunny & Rainy 34.547.09 Sunny only 23.583.56 WoTEICCV 2025IL Sunny & Rainy 36.547.85 Sunny only 28.655.21 SimLingo (Pretrained-VLM) CVPR 2025IL Sunny & Rainy 51.69 13.68 Sunny only 33.498.97 RaWMPC–PC Sunny & Rainy 53.6714.96 Sunny only41.3610.83 4.3.4Qualitative visualization of predictive control We provide some visualization results of the predic- tive control procedure of RaWMPC in Fig. 6. Given RGB observations and high-level navigation com- mands (e.g., keep going straight or merge left), our generative policy proposes a small set of candidate action sequences (e.g., keep going straight, detour, brake, and lane change). For each candidate, the risk- aware world model predicts the near-future semantic traffic state and the decoding module evaluates its consequence from both task progress and safety per- spectives, including collision risk, off-lane/sidewalk intrusion risk, and progress-related penalties such as getting stuck in traffic. The final action is selected by comparing these predicted consequences and choosing the minimum-cost one under the navigation goal. In the first case, going straight collides with a crossing pedestrian, while detours either collide with an on- coming vehicle or drive onto the sidewalk; RaWMPC chooses slow down, straight briefly, then stop to safely stop in front of the pedestrian (instead of an overly conservative early brake). In the second case, go- ing straight or merging left immediately causes a collision, steering right hits a parked vehicle, and stopping leads to a deadlock; thus RaWMPC se- lects pause briefly, then merge left for a collision-free merge. These cases demonstrate that RaWMPC can proactively avoid risky behaviors by explicitly fore- casting and comparing action consequences, rather than merely following a single command or relying on a fixed fallback maneuver. 4.4 Ablation Study In this section, we provide comprehensive abla- tion studies of the proposed approach using the 12 t+1t+2t+3t+4t+5t Front - view camera Bird’s - eye view (BEV) and candidate action s Action Selection Keep going straight Slow down, straight briefly, then stop Detour forward-right ! Brake immediately Detour forward-left Collidewith the pedestrian Stop in front of the pedestrian Drive onto the sidewalk Stop in front of the pedestrian at a longer distance Collide with an oncoming vehicle Cost Candidates ✔ Pedestrian Front - view camera Bird’s - eye view (BEV) and candidate action s Action Selection Keep going straight Pause briefly, then merge left Merge into the adjacent left lane ! Come to a complete stop Steer right to avoid the oncoming vehicle from the left Collidewith the vehicle ahead-left Merge into the left lane without collision Collidewith the vehicle ahead-left Get stuck in traffic and cannot proceed Collidewith a vehicle parked along the roadside Cost Candidates t+1t+2t+3t+4t+5t ✔ At-risk vehicles Figure 6 Visualization of the predictive control procedure. At timet, we show the front-view and the BEV images (with segmentation). Dashed curves indicate candidate actions and highlighted agents denote key risks. Rollouts from t + 1 to t + 5 illustrate predicted consequences, and the bottom panel reports each action’s outcomes and costs (e.g., collision, sidewalk intrusion, stopping distance). Scenario 1: RaWMPC slows down, proceeds briefly, then stops for the pedestrian. Scenario 2: RaWMPC waits briefly, then turns left to avoid both the front-left vehicle and the parked car. Bench2Drive dataset. 4.4.1 Analysis of Framework Table 4 analyzes core components aligned with our model design (Sec. 3.1.1). w/o Semantic Guidance removes semantic-guided event decoding, i.e., the fusion of semantic attention from the segmentation decoder into the event decoder. This leads to a clear drop (DS 88.31→82.36, SR 70.48%→62.69%), show- ing that accurate safety-event prediction is crucial for risk-aware cost evaluation. w/o Segmentation Table 4 Ablation study on the proposed framework. Method Metrics DS↑SR(%)↑ Entire RaWMPC (Ours)88.3170.48 w/o Semantic Guidance82.36 -5.95 62.69 -7.79 w/o Segmentation Decoder 70.85 -17.46 48.95 -21.53 w/o Action Selection61.35 -26.96 30.98 -39.50 Decoder further removes the segmentation decoding branch and its supervision, resulting in a large degra- 13 Table 5 Ablation study on the risk-aware training. Method Metrics DS↑SR(%)↑ Risk-aware Sampling (Ours) 88.3170.48 ε-Greedy Sampling83.86 -4.45 61.74 -8.74 Random Sampling70.41 -17.90 46.82 -23.66 Table 6 Results obtained using different action supervi- sion in policy learning. Policy Learning Data Metrics DS↑SR(%)↑ Pos. & Neg. Actions (Ours) 88.3170.48 Expert Actions86.75 -1.56 68.25 -2.23 Only Positive Actions83.65 -4.66 66.52 -3.96 dation (DS 70.85 / SR 48.95%), which indicates that forecasting high-level semantics is essential for re- liable long-horizon rollouts. w/o Action Selection disables predictive control in Eq.(1)(bypassing cost- based ranking in Eq.(6)) and directly executes the proposal/guidance output, causing the most severe collapse (DS 61.35 / SR 30.98%). This confirms that selecting actions by explicitly evaluating predicted long-horizon consequences is the key to RaWMPC. 4.4.2 Analysis of Risk-Aware Training Table 5 evaluates the risk-aware interaction training strategy used to refine the world model. Our risk- aware sampling follows the design in Sec. 3.2: besides random exploration, it uses the current RaWMPC to score candidate action sequences and deliberately collects both good (low-cost) and bad (high-cost) rollouts, improving coverage of rare safety-critical outcomes. Replacing it withε-greedy sampling (ran- dom with probabilityε 1 , otherwise only selecting low-cost rollouts) reduces performance (DS 83.86 / SR 61.74%), showing that excluding high-cost fail- ures weakens learning of risky consequences. Pure random sampling further degrades results (DS 70.41 / SR 46.82%), indicating that unguided data collection is substantially less efficient for learning long-horizon consequences. 4.4.3 Analysis of Self-Evaluation Distillation Table 6 studies how to train the action proposal net- work in Sec. 3.3. Our default setting uses RaWMPC as a self-evaluator to pseudo-label actions: the lowest- cost sequence is treated as a positive, while high-cost sequences serve as negatives, which yields the best Table 7 Results obtained using different amounts of prediction horizon. Prediction Horizon Metrics DS↑SR(%)↑ H=157.85 -30.46 28.64 -41.84 H=574.98 -13.33 49.52 -20.96 H=10 (Ours)88.3170.48 H=1582.34 -5.97 62.38 -8.10 Table 8 Results obtained using different amounts of offline learning data in warm-up. Warm-up Data Metrics DS↑SR(%)↑ 0%87.34 -0.97 69.62 -0.86 10%88.3170.48 20%88.09 -0.22 70.32 -0.16 30%86.95 -1.36 68.52 -1.96 performance (DS 88.31 / SR 70.48%). Training the proposal network only with expert actions slightly degrades performance (DS 86.75 / SR 68.25%), sug- gesting that self-evaluated targets align better with the predictive-control objective than direct imitation targets. Using only positive actions further drops performance (DS 83.65 / SR 66.52%), indicating that explicitly contrasting against high-risk negatives is important for preventing unsafe candidates and im- proving downstream selection. 4.4.4 Discussion on Prediction Horizon Table 7 ablates the planning horizonHused in the world-model rollout and cost evaluation (Eq.(6)). Short horizons fail to capture delayed consequences, leading to poor performance (H=1: DS 57.85 / SR 28.64%; H=5: DS 74.98 / SR 49.52%). Increas- ing to H=10 yields the best results (DS 88.31 / SR 70.48%), as it provides sufficient look-ahead for risk assessment while keeping prediction uncertainty man- ageable. Further increasing to H=15 degrades perfor- mance (DS 82.34 / SR 62.38%), likely due to accu- mulated rollout errors that affect cost-based ranking. 4.4.5 Discussion on Warm-up Table 8 ablates the fraction of offline logged trajec- tories used for warm-up before interactive training. In this study, we keep the total number of training samples fixed and vary only the proportion allocated to offline warm-up data. Without warm-up (0%), 14 Table 9 Ablation study on the control pattern with learned world model. Method Metrics DS↑SR(%)↑ Predictive Control (Ours) 88.3170.48 Reinforcement Learning 73.58 -14.73 51.85 -18.63 performance drops (DS 87.34 / SR 69.62%), suggest- ing that training from scratch leads to less reliable rollouts and a less stable early optimization stage. Using a small amount of logged trajectories (10%) yields the best results (DS 88.31 / SR 70.48%), indi- cating that a light warm-up provides useful predictive priors (e.g., basic dynamics modeling and perception decoding) that improve long-horizon rollout quality and downstream control. However, further increasing the warm-up ratio begins to degrade performance (20%: DS 88.09 / SR 70.32%; 30%: DS 86.95 / SR 68.52%). We attribute this trend to the fact that offline logged trajectories are strongly biased toward safe, human-like behaviors and contain few hazardous events. Consequently, allocating too much data to offline warm-up reduces the opportunities for subse- quent online interaction to explore unconventional actions and collect safety-critical failures—especially in dangerous scenarios—which are essential for learn- ing robust risk awareness. 4.4.6 Discussion on Control Pattern Table 9 compares predictive control to directly op- timizing a policy with model-based reinforcement learning (RL). Predictive control achieves substan- tially higher performance (DS 88.31 / SR 70.48%) than model-based RL (DS 73.58 / SR 51.85%). This validates the benefit of evaluating candidate action se- quences via decoded future outcomes (segmentation, events, ego-states) and selecting the minimum-cost one, rather than relying on end-to-end policy opti- mization alone. 4.4.7 World-Model Prediction Accuracy Table 10 reports the event prediction quality used by the cost function in Eq.(6). The predictor achieves high accuracy across event types (0.91–0.96) and strong recall on collision-related events (e.g., pedes- trian collision recall 0.99), providing reliable signals for risk-aware evaluation. We observe lower precision for some rare events (e.g., pedestrian collision pre- cision 0.52), reflecting a conservative tendency with more false positives; in safety-critical driving, priori- Table 10 Prediction accuracy of future traffic events with learned world model. Collision Running Traffic Sign Pedestrian Vehicle Static Accuracy0.960.91 0.930.91 Recall0.990.84 0.890.84 Precision0.520.62 0.630.68 tizing recall can be preferable to missing hazards. 5 Conclusion In this work, we proposed RaWMPC, a risk-aware world-model predictive control framework for end- to-end autonomous driving that does not require expert action supervision. RaWMPC learns an action-conditioned world model to roll out multiple candidate behaviors, predicts future semantics and safety-critical events, and selects actions by explicitly minimizing a risk-aware cost. To make rare-but- catastrophic outcomes predictable and avoidable, we introduced a risk-aware interaction strategy that intentionally collects both safe and hazardous roll- outs, and we further proposed self-evaluation dis- tillation to train an efficient action proposal policy using RaWMPC as a self-evaluator. Extensive ex- periments on Bench2Drive and NAVSIM show that RaWMPC achieves state-of-the-art performance and stronger robustness under domain shift, even without offline warm-up, showing the potential to significantly reduce the reliance on costly real-world expert demon- strations. For future work, we will explore domain adaptation and more efficient planning to better sup- port real-world deployment and sim-to-real transfer. Statements and Declarations • Competing interests. The authors declare that they have no known competing financial inter- ests or personal relationships that could have appeared to influence the work reported in this paper. •Data availability. This work does not propose any new dataset. The datasets (Bench2Drive [29] and NAVSIM [11]) that support the findings of this study are openly available at the URLs: Bench2Drive and NAVSIM. 15 References [1]Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive en- vironments. In Forty-first International Conference on Machine Learning, 2024. [2]Jinkun Cao, Xin Wang, Trevor Darrell, and Fisher Yu. Instance-aware predictive navigation in multi- agent environments. In IEEE International Confer- ence on Robotics and Automation, pages 5096–5102, 2021. [3]Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, et al. Pseudo-simulation for autonomous driving. arXiv preprint arXiv:2506.04218, 2025. [4]Raphael Chekroun, Marin Toromanoff, Sascha Hor- nauer, and Fabien Moutarde. Gri: General rein- forced imitation and its application to vision-based autonomous driving. In NeurIPS 2021, Machine Learning for Autonomous Driving Workshop, 2021. [5]Dian Chen and Philipp Krähenbühl. Learning from all vehicles. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogni- tion, pages 17222–17231, 2022. [6]Dian Chen, Vladlen Koltun, and Philipp Krähen- bühl. Learning to drive from a world on rails. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15590–15599, 2021. [7]Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to- end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2024. [8] Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive trans- formers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 26890–26900, October 2025. [9]Pranav Singh Chib and Pravendra Singh. Recent advancements in end-to-end autonomous driving using deep learning: A survey. IEEE Transactions on Intelligent Vehicles, 9(1):103–118, 2023. [10]Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Trans- fuser: Imitation with transformer-based sensor fu- sion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. [11]Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xin- shuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706–28719, 2025. [12]Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning, pages 1–16, 2017. [13]Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In 2023 IEEE international conference on robotics and automation (ICRA), pages 3567–3575. IEEE, 2023. [14] Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jian- feng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action gen- eration. arXiv preprint arXiv:2503.19755, 2025. [15]Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile control- lability. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [16]Mingzhe Guo, Zhipeng Zhang, Yuan He, Ke Wang, Liping Jing, and Haibin Ling. End-to-end au- tonomous driving without costly modularization and 3d manual annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. [17]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018. [18]Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019. [19]Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555–2565. PMLR, 2019. [20]Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. [21]Shadi Hamdan, Chonghao Sima, Zetong Yang, Hongyang Li, and Fatma Güney. Eta: Effi- ciency through thinking ahead, a dual approach to self-driving with large models. In Proceedings of 16 the IEEE/CVF International Conference on Com- puter Vision, 2025. [22] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. In Inter- national Conference on Learning Representations, 2018. [23]Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. Advances in Neural Information Processing Systems, 35:20703–20716, 2022. [24]Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023. [25] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chong- hao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. [26]Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving mod- els. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8240–8249, 2023. [27]Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of per- ception and planning in end-to-end autonomous driving. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 7953–7963, 2023. [28]Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21983–21994, 2023. [29]Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: To- wards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877, 2024. [30] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. In In- ternational Conference on Learning Representations (ICLR), 2025. [31]Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vector- ized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 8340–8350, 2023. [32]Alex Kendall, Jeffrey Hawke, David Janz, Przemys- law Mazur, Daniele Reda, John-Mark Allen, Vinh- Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international confer- ence on robotics and automation (ICRA), pages 8248–8254. IEEE, 2019. [33]Diederik P Kingma. Adam: A method for stochas- tic optimization. arXiv preprint arXiv:1412.6980, 2014. [34] Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 14738–14748, 2021. [35]Fanjie Kong, Yitong Li, Weihuang Chen, Chen Min, Yizhe Li, Zhiqiang Gao, Haoyang Li, Zhongyu Guo, and Hongbin Sun. Vlr-driver: Large vision-language-reasoning models for embodied au- tonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 26966–26976, 2025. [36]Hanyang Kong, Dongze Lian, Michael Bi Mi, and Xinchao Wang. Dreamdrone: Text-to-image diffu- sion models are zero-shot perpetual view generators. In European Conference on Computer Vision, pages 324–341. Springer, 2024. [37]Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European Conference on Computer Vision, pages 142–158. Springer, 2024. [38] Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yun- tao Chen, Zhaoxiang Zhang, and Tieniu Tan. En- hancing end-to-end autonomous driving with latent world model. In The Thirteenth International Con- ference on Learning Representations, 2025. [39] Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 27137–27146, October 2025. [40]Zhenxin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Zuxuan Wu, and Jose M. Alvarez. Hydra-next: Ro- bust closed-loop driving with open-loop training. In Proceedings of the IEEE/CVF International Con- 17 ference on Computer Vision (ICCV), pages 27305– 27314, October 2025. [41]Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird’s-eye-view representation from lidar-camera via spatiotemporal transform- ers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. [42]Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xing- gang Wang. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12037–12047, June 2025. [43] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. [44]Yuhang Lu, Jiadong Tu, Yuexin Ma, and Xinge Zhu. Real-ad: Towards human-like reasoning in end-to- end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 27783–27793, 2025. [45]Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chi- dananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Vi- sual question answering for autonomous driving. In European Conference on Computer Vision, pages 252–269. Springer, 2024. [46]Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. In RSS, 2023. [47]Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 2016 fourth international conference on 3D vi- sion (3DV), pages 565–571. Ieee, 2016. [48] Michael Montemerlo, Jan Becker, Suhrid Bhat, Hen- drik Dahlkamp, Dmitri Dolgov, Scott Ettinger, Dirk Haehnel, Tim Hilden, Gabe Hoffmann, Burkhard Huhnke, et al. Junior: The stanford entry in the urban challenge. Journal of field Robotics, 25(9): 569–597, 2008. [49]Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan, Peng Jia, Xianpeng Lang, Xingang Wang, and Wenjun Mei. Recondreamer: Crafting world models for driving scene recon- struction via online restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1559–1569, June 2025. [50]Jingcheng Ni, Yuxin Guo, Yichen Liu, Rui Chen, Lewei Lu, and Zehuan Wu. Maskgwm: A gen- eralizable driving world model with video mask reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 22381–22391, June 2025. [51]Brian Paden, Michal Čáp, Sze Zheng Yong, Dmitry Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving ur- ban vehicles. IEEE Transactions on intelligent vehicles, 1(1):33–55, 2016. [52]Xinlei Pan, Xiangyu Chen, Qizhi Cai, John Canny, and Fisher Yu. Semantic predictive control for explainable and efficient policy learning. In Inter- national Conference on Robotics and Automation, pages 3203–3209, 2019. [53]Scott Drew Pendleton, Hans Andersen, Xinxin Du, Xiaotong Shen, Malika Meghjani, You Hong Eng, Daniela Rus, and Marcelo H Ang. Perception, plan- ning, control, and coordination for autonomous vehicles. Machines, 5(1):6, 2017. [54]Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end- to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 7077–7087, 2021. [55]Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mer- cea, A Sophia Koepke, Zeynep Akata, and Andreas Geiger. Plant: Explainable planning transformers via object-level representations. In Conference on Robot Learning, pages 459–470, 2023. [56]Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karn- sund, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Carllava: Vision language models for camera-only closed-loop driving. arXiv preprint arXiv:2406.10165, 2024. [57] Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop au- tonomous driving with language-action alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11993–12003, June 2025. [58]Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. In International Conference on Learning Representations, 2020. [59]Ahmad El Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. End-to-end deep re- 18 inforcement learning for lane keeping assist. arXiv preprint arXiv:1612.04340, 2016. [60]Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683–700. Springer, 2020. [61]ShuYao Shang, Yuntao Chen, Yuqi Wang, Yingyan Li, and Zhaoxiang Zhang. Drivedpo: Policy learning via safety dpo for end-to-end autonomous driving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [62]Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driv- ing using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 726–737, 2023. [63] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reason- net: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 13723–13733, 2023. [64]Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 15120–15130, 2024. [65]Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274. Springer, 2024. [66]Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Fed- erico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. [67]Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015. [68]Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang, and Yadan Luo. Don’t shake the wheel: Momentum-aware planning in end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22432–22441, 2025. [69]Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. In 2025 IEEE International Con- ference on Robotics and Automation (ICRA), pages 8795–8801. IEEE, 2025. [70]Shuhan Tan, Kelvin Wong, Shenlong Wang, Siv- abalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 892–901, 2021. [71]Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuehl. Language con- ditioned traffic generation. In Conference on Robot Learning, pages 2714–2752. PMLR, 2023. [72]Shuhan Tan, Boris Ivanovic, Yuxiao Chen, Boyi Li, Xinshuo Weng, Yulong Cao, Philipp Kraehen- buehl, and Marco Pavone. Promptable closed-loop traffic simulation. In Annual Conference on Robot Learning, 2024. [73]Yingqi Tang, Zhuoran Xu, Zhaotie Meng, and Erkang Cheng. Hip-ad: Hierarchical and multi- granularity planning with deformable attention for autonomous driving in a single decoder. arXiv preprint arXiv:2503.08612, 2025. [74] Sebastian Thrun, Mike Montemerlo, Hendrik Dahlkamp, David Stavens, Andrei Aron, James Diebel, Philip Fong, John Gale, Morgan Halpenny, Gabriel Hoffmann, et al. Stanley: The robot that won the darpa grand challenge. Journal of field Robotics, 23(9):661–692, 2006. [75]Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7153–7162, 2020. [76]Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, MN Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, et al. Autonomous driving in urban envi- ronments: Boss and the urban challenge. Journal of field Robotics, 25(8):425–466, 2008. [77] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. Dreamwalker: Mental planning for continuous vision-language navigation. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 10873–10883, 2023. [78]Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yun- tao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Pro- ceedings of the IEEE/CVF Conference on Computer 19 Vision and Pattern Recognition, pages 14749–14759, 2024. [79]Kehan Wen, Yutong Hu, Yao Mu, and Lei Ke. Mˆ 3pc: Test-time model predictive control using pre- trained masked trajectory model. In The Thirteenth International Conference on Learning Representa- tions, 2025. [80]Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2024. [81]Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided con- trol prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neu- ral Information Processing Systems, 35:6119–6132, 2022. [82]Zebin Xing, Xingyu Zhang, Yang Hu, Bo Jiang, Tong He, Qian Zhang, Xiaoxiao Long, and Wei Yin. Goalflow: Goal-driven flow matching for mul- timodal trajectories generation in end-to-end au- tonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1602–1611, 2025. [83]Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to- end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024. [84]Xuemeng Yang, Licheng Wen, Tiantian Wei, Yukai Ma, Jianbiao Mei, Xin Li, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, Liang He, Yong Liu, Botian Shi, and Yu Qiao. Drivearena: A closed- loop generative simulation platform for autonomous driving. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 26933–26943, October 2025. [85]Zhenjie Yang, Xiaosong Jia, Qifeng Li, Xue Yang, Maoqing Yao, and Junchi Yan. Raw2drive: Rein- forcement learning with aligned world models for end-to-end autonomous driving (in carla v2), 2025. Accepted by NeurIPS 2025. [86] Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Generalisable driving expla- nations with retrieval-augmented in-context learn- ing in multi-modal large language model. arXiv preprint arXiv:2402.10828, 2024. [87] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceed- ings of the IEEE/CVF international conference on computer vision, pages 9813–9823, 2021. [88]Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transform- ers. In Advances in Neural Information Processing Systems, 2022. [89]Bozhou Zhang, Nan Song, Xin Jin, and Li Zhang. Bridging past and future: End-to-end autonomous driving with historical prediction and planning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 6854–6863, 2025. [90]Jimuyang Zhang, Zanming Huang, and Eshed Ohn- Bar. Coaching a teachable student. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7805–7815, 2023. [91] Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, Xun Cao, and Wei Yin. Epona: Autoregressive diffusion world model for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 27220–27230, October 2025. [92]Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copi- lot4d: Learning unsupervised world models for au- tonomous driving via discrete diffusion. In ICLR, 2024. [93]Peizhi Zhang, Lu Xiong, Zhuoping Yu, Peiyuan Fang, Senwei Yan, Jie Yao, and Yi Zhou. Rein- forcement learning-based end-to-end parking for automatic parking system. Sensors, 19(18):3996, 2019. [94]Wei Zhang, Pengfei Li, Junli Wang, Bingchuan Sun, Qihao Jin, Guangjun Bao, Shibo Rui, Yang Yu, Wenchao Ding, Peng Li, et al. Dual-aeb: Syn- ergizing rule-based and multimodal large language models for effective emergency braking. In 2025 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 14888–14895. IEEE, 2025. [95]Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 15222–15232, 2021. [96]Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. Trafficbots: Towards world models for autonomous driving simulation and motion prediction. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1522–1529. IEEE, 2023. 20 [97]Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12015–12026, June 2025. [98]Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occ- world: Learning a 3d occupancy world model for autonomous driving. In European conference on computer vision, pages 55–72. Springer, 2024. [99] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chen- ming Zhang, and Long Chen. Genad: Gener- ative end-to-end autonomous driving. In Euro- pean Conference on Computer Vision, pages 87–104. Springer, 2024. [100] Yupeng Zheng, Pengxuan Yang, Zebin Xing, Qichao Zhang, Yuhang Zheng, Yinfeng Gao, Pengfei Li, Teng Zhang, Zhongpu Xia, Peng Jia, XianPeng Lang, and Dongbin Zhao. World4drive: End-to- end autonomous driving via intention-aware phys- ical latent world model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 28632–28642, October 2025. [101] Julius Ziegler, Philipp Bender, Markus Schreiber, Henning Lategahn, Tobias Strauss, Christoph Stiller, Thao Dang, Uwe Franke, Nils Appenrodt, Christoph G Keller, et al. Making bertha drive—an autonomous journey on a historic route. IEEE Intelligent transportation systems magazine, 6(2): 8–20, 2014. [102]Sicheng Zuo, Wenzhao Zheng, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Gaussianworld: Gaussian world model for streaming 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6772–6781, June 2025. 21