Paper deep dive
Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving
Chenglin Chen, Lujia Wang, Xinhu Zheng, Jun Ma, Haoang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/4/2026, 3:20:21 AM
Summary
The paper introduces EMPlan, an efficient multi-modal trajectory planning method for autonomous driving that combines a hybrid architecture (sparse anchors with offset refinement) with a two-stage training paradigm. The first stage involves pretraining with imitation and rule-based supervision, while the second stage employs reward-guided fine-tuning using unpaired preference data to enhance safety and planning accuracy without increasing inference latency.
Entities (8)
Relation Signals (6)
EMPlan → employs → Reward-Guided Fine-Tuning
confidence 95% · we propose EMPlan... powered by reward-guided fine-tuning
EMPlan → evaluatedon → NAVSIM
confidence 95% · We evaluate EMPlan on the non-reactive NAVSIM benchmark
EMPlan → uses → Sparse Anchors
confidence 95% · We design a hybrid architecture that combines sparse anchors with an offset refinement module
EMPlan → uses → Offset Refinement Module
confidence 95% · We design a hybrid architecture that combines sparse anchors with an offset refinement module
Reward-Guided Fine-Tuning → utilizes → Kahneman–Tversky Optimization
confidence 90% · we introduce Kahneman–Tversky Optimization (KTO)... To enable preference-based fine-tuning
Imitation Learning → suffersfrom → Causal Confusion
confidence 85% · Methods based on imitation learning suffer from causal confusion
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Safe and efficient trajectory planning is essential in autonomous driving. However, existing end-to-end approaches often fall short in both computational efficiency and safety guarantees. Methods based on imitation learning suffer from causal confusion, while rule-based scoring approaches often incur heavy computational overhead and suffer from objective misalignment. Additionally, preference-based methods rely on strict pairwise annotations, limiting data utilization. To overcome these limitations, we propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning. We design a hybrid architecture that combines sparse anchors with an offset refinement module for efficient multi-modal trajectory prediction. Sparse anchors provide coarse trajectory candidates with low latency, which are subsequently refined by the offset module for higher prediction accuracy. To enhance safety without incurring additional inference costs, we adopt a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. During fine-tuning, we leverage rule-based reward signals and unpaired preference supervision to refine the pretrained policy toward safer trajectory selection. We evaluate EMPlan on the non-reactive NAVSIM benchmark, where it strikes a favorable balance between planning accuracy and efficiency, demonstrating superior performance under real-time constraints.
Tags
Links
- Source: https://arxiv.org/abs/2609.38862v1
- Canonical: https://arxiv.org/abs/2609.38862v1
Trouble viewing inline? Open PDF directly →
Full Text
45,481 characters extracted from source content.
Expand or collapse full text
Efficient Multi-Modal Planning with Reward-Guided Preference Optimization for Autonomous Driving Chenglin Chen Lujia Wang Xinhu Zheng Jun Ma Haoang Li* Affiliation: The Hong Kong University of Science and Technology (Guangzhou) Abstract Safe and efficient trajectory planning is essential in autonomous driving. However, existing end-to-end approaches often fall short in both computational efficiency and safety guarantees. Methods based on imitation learning suffer from causal confusion, while rule-based scoring approaches often incur heavy computational overhead and suffer from objective misalignment. Additionally, preference-based methods rely on strict pairwise annotations, limiting data utilization. To overcome these limitations, we propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning. We design a hybrid architecture that combines sparse anchors with an offset refinement module for efficient multi-modal trajectory prediction. Sparse anchors provide coarse trajectory candidates with low latency, which are subsequently refined by the offset module for higher prediction accuracy. To enhance safety without incurring additional inference costs, we adopt a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. During fine-tuning, we leverage rule-based reward signals and unpaired preference supervision to refine the pretrained policy toward safer trajectory selection. We evaluate EMPlan on the non-reactive NAVSIM benchmark, where it strikes a favorable balance between planning accuracy and efficiency, demonstrating superior performance under real-time constraints. I Introduction Ensuring safety, comfort, and compliance with traffic rules are the fundamental objectives for an autonomous driving (AD) system. Recently, the end-to-end (E2E) AD has emerged as a promising solution to achieve these goals [1, 2, 3, 4]. This paradigm allows the entire system to be optimized jointly, thereby eliminating the information loss inherent in traditional modular designs. Nevertheless, the training objective of most existing E2E systems remains rooted in imitation learning (IL), where policies are trained in an open-loop manner by simply aligning predicted trajectories with expert data [5, 6]. IL-based methods often fail in long-tail situations due to causal confusion [7] and distribution shifts [8, 9, 10] in the training data. Furthermore, the objective of IL primarily aims to mimic expert data, often neglecting critical driving requirements such as safety, comfort, and adherence to traffic rules. Figure 1: Overview of our EMPlan, built upon a hybrid architecture and powered by reward-guided fine-tuning. (a) The hybrid architecture combines sparse anchors with an offset module for efficient multi-modal trajectory prediction. Reward-guided fine-tuning further aligns trajectory selection with safety preferences. (b) Our model achieves better performance in efficiency and accuracy compared with representative methods [2, 3, 11, 12]. (c) Compared to score-based methods (i.e., WoTE [12]), our model assigns lower scores to human-like but unsafe trajectories (red) and higher scores to safe ones (green). To alleviate these limitations, recent approaches attempt to refine IL-pretrained policies by incorporating auxiliary signals, such as rule-based scores [11, 12] or preference-based labels [13, 14, 15, 16, 17], to explicitly align the model with safe or logical driving behaviors. Although these approaches are generally more reliable than pure imitation, they still suffer from limitations in computational efficiency and safety guarantees. First, methods incorporating rule-based scores typically adopt anchor-based architecture. To ensure sufficient action space coverage, they require a dense set of anchors, resulting in prohibitive computational latency. Moreover, directly regressing rule-based scores often causes objective misalignment, as it encourages the model to fit absolute values rather than learning preference relationships necessary for reliable trajectory selection. Second, preference-based methods attempt to better capture driving logic through trajectory-level comparisons. However, they generally rely on strict pairwise annotations (e.g., chosen vs. rejected) within each scenario. Such pairwise supervision is sparse and data-inefficient, as learning is restricted to limited trajectory pairs while other valid trajectory candidates are ignored. To overcome these limitations, we propose EMPlan (see Fig. 1), a multi-modal trajectory planning method that unifies efficient trajectory generation with safety-aware trajectory selection. Our model is built upon a hybrid architecture and enhanced by reward-guided fine-tuning. First, we design a hybrid architecture to improve efficiency. Unlike traditional methods that exhaustively enumerate thousands of anchors, we utilize sparse anchors to generate coarse trajectory candidates, which are then refined by a lightweight offset module. This hybrid design preserves action space coverage while significantly reducing computational overhead. Second, to enhance safety without incurring additional inference costs, we employ a two-stage training paradigm consisting of pretraining and reward-guided fine-tuning. In the pretraining stage, we design a score-based policy learning objective that integrates trajectory imitation, rule-based supervision, and auxiliary BEV segmentation to better handle data imbalance and improve representation learning. In the fine-tuning stage, we optimize the pretrained policy using reward signals instead of rule-based scores. This aligns the training objective with the trajectory selection criterion at inference, reducing the mismatch between optimization and decision-making. However, reward supervision typically operates on individual trajectories and fails to explicitly model relative comparisons between candidates. We therefore incorporate preference-based signals to capture relative relationships within each scenario. To alleviate the reliance on strict pairwise annotations in preference learning, we utilize unpaired trajectory samples with preference annotations (e.g., desirable and undesirable samples). This approach improves the utilization of available trajectory data, aligning the policy with safe driving behaviors while avoiding the constraints of strict pairwise supervision. In summary, our main contributions are as follows: • We propose EMPlan, an efficient multi-modal trajectory planning method powered by reward-guided fine-tuning, which can achieve real-time inference and improve overall planning accuracy. • We design a hybrid architecture combining sparse anchors with an offset module to refine trajectories, overcoming discrete coverage gaps at low computational cost. • We introduce an unpaired, reward-guided fine-tuning strategy that adapts preference learning to anchor-based trajectory evaluation, enabling effective policy alignment with multi-objective driving criteria. I Related Works I-A End-to-End Autonomous Driving Benefiting from large-scale driving datasets and advances in transformer architectures [18], E2E AD has progressed rapidly in recent years. Early works, such as Transfuser [2], emphasize multi-modal sensor fusion to learn stronger scene representations. Moving towards planning-oriented design, UniAD [3] unifies full-stack tasks within a single network to enhance planning performance. To further improve efficiency, VAD [19] explores compact vectorized scene representations. Despite architectural differences, these approaches typically regress a single-mode trajectory conditioned on the ego vehicle. To address the inherent multimodality of driving behaviors, VADv2 [4] introduces a predefined anchor vocabulary and learns to assign scores to candidate trajectories, enabling multi-mode prediction. Building upon this idea, HydraMDP [11] distills rule-based knowledge from simulators into the model to enhance trajectory scoring. WoTE [12] leverages a world model to generate future BEV representations as intermediate supervision, improving downstream trajectory prediction. Nevertheless, such score-based models are optimized by assigning supervision scores to each anchor in isolation, while ignoring the relative preference relations among different trajectory candidates. More recently, diffusion models have been explored for multi-modal trajectory prediction. Despite their strong generative capability, diffusion-based approaches [20, 21] typically incur high memory overhead and inference latency, which hinder their deployment in real-time safety-critical AD systems. In contrast, we propose a lightweight hybrid architecture model for multimodal trajectory prediction, further enhanced by reward-guided fine-tuning to encourage safer planning behaviors. I-B Policy Fine-Tuning in Autonomous Driving Recently, policy fine-tuning has emerged as a promising direction for improving E2E AD, mainly through closed-loop reinforcement learning (RL) and preference-based optimization. RL-based methods benefit from interactive environment signals but are still in an early stage for AD, as they rely heavily on domain-specific reward engineering and often exhibit unstable training. To bridge the simulation-to-reality gap, RAD [9] introduces a 3D Gaussian Splatting-based RL framework for training in high-fidelity environments. In parallel, both AlphaDrive [22] and TrajHF [23] adopt Group Relative Policy Optimization [24] for policy optimization, where the former incorporates vision-language reasoning, and the latter aligns diffusion planners with human driving preferences. Furthermore, ReCogDrive [25] leverages simulator-assisted RL to refine trajectory generation toward safer and more human-like behaviors. More recently, DriveDPO [17] fine-tunes anchor-based planners via DPO, but it relies on strict pairwise preference annotations, which may leave valuable unpaired trajectories underutilized. In contrast, our method adopts unpaired preference data, allowing each candidate trajectory to contribute independently to preference optimization. Our goal is to improve the alignment between predicted scores and actual driving quality without requiring explicit pair construction. This is particularly beneficial under a multi-candidate selection setting, where candidate trajectories can be independently evaluated rather than selectively paired. Figure 2: Overview of the network architecture. Given multi-view images, the perception module extracts BEV representations, which are fed into a trajectory decoder to predict trajectory offsets and reward scores based on ego states and sparse anchors. The offset head refines anchor trajectories, and the trajectory with the highest aggregated reward is chosen as the final output. During training, the model is first pretrained with imitation and rule-based supervision and subsequently optimized via reward-guided fine-tuning, which integrates reward loss and preference loss to align trajectory selection with safety preferences. I Method Given the sensor observation O, the goal of the model is to forecast a T-step ego-vehicle’s future trajectory τ=(xt,yt,θt)t=1Tτ=\(x_t,y_t, _t)\_t=1^T, where each waypoint denotes the vehicle’s position and headings at time step t. To handle the multi-modal nature of real-world driving, we follow the paradigm in VADv2 [4] and treat motion planning as a discrete selection problem over a predefined vocabulary of trajectory anchors =τi∈ℝT×3i=1NV=\ _i ^T× 3\_i=1^N, where each anchor τi _i represents a T-step waypoint sequence. The policy assigns a score to each anchor, and the anchor with the highest score is selected as the predicted trajectory. Standard supervised training optimizes these scores independently for each anchor, which ignores relative quality among candidates and requires a dense anchor set to cover the action space, leading to high computational cost. To address these limitations, we aim to 1) efficiently generate trajectories without exhaustive anchor enumeration and 2) leverage relative reward signals to refine the policy toward safer trajectory selection. The following sections detail the overall architecture and key optimization strategies. I-A Hybrid Architecture of Our Policy Model As shown in Fig. 2, our method is structured into two networks: a Perception module for scene understanding and a Trajectory Decoder for multimodal trajectory generation. For the perception module, we extract features from multi-view RGB images and transform them into BEV representations at two distinct scales. One scale is utilized for BEV semantic segmentation with auxiliary supervision to enhance spatial awareness. The other is combined with learnable positional embeddings to form scene tokens EenvE_ env, which provide rich, scene-level context for the subsequent trajectory decoder. The core of our architecture lies in the design of the trajectory decoder. To balance computational efficiency and planning accuracy, we decompose trajectory generation into coarse discrete mode representation and continuous offset refinement. The sparse anchors represent distinct maneuver modes, while the offset head predicts a scene-conditioned residual for each anchor, reducing the quantization error imposed by a finite anchor vocabulary. In contrast, dense-anchor methods obtain finer discrete coverage by increasing the number of anchors, which enlarges the candidate vocabulary and raises the computation of trajectory decoding. Unlike diffusion planners, our refinement is completed in a single feed-forward pass without iterative denoising. Specifically, we begin by constructing a sparse set of trajectory anchors =τi∈ℝT×3i=1NV=\ _i ^T× 3\_i=1^N by applying k-means clustering to the NAVSIM training dataset, following prior works [4, 11, 12]. These anchors are transformed into a high-dimensional feature space through Fourier positional encoding to obtain the anchor embeddings: Γ()=[,ϕ0(),ϕ1(),⋯,ϕL−1()]∈ℝN×T×(3+6L) (V)= [V, _0(V), _1(V),·s, _L-1(V) ] ^N× T×(3+6L) (1a) ϕk()=[sin(2kπ),cos(2kπ)] _k(V)= [ (2^k ), (2^k ) ] (1b) In addition, the ego state and high-level driving command are also encoded and fused with these anchor embeddings to form the final planning tokens EplanE_ plan for trajectory decoding. The scene tokens EenvE_ env and planning tokens EplanE_ plan are then fed into a Planning Transformer to perform cross-attention, producing a set of scene-conditioned planning tokens: E^plan=TransformerDecoder(Q=Eplan,K,V=Eenv) E_ plan=TransformerDecoder (Q=E_ plan,K,V=E_ env ) (2) Finally, the context-aware planning tokens E^plan E_ plan are passed to two separate prediction heads: an offset head and a scorer head. Since the predefined trajectory anchors form a sparse and discrete motion prior, the ground-truth trajectories may not perfectly align with them. To address this, an offset head predicts per-anchor offsets to refine the coarse trajectories. The refined trajectories are given by: ^=+OffsetHead(E^plan) V=V+OffsetHead ( E_ plan ) (3) In parallel with trajectory refinement, the scorer head evaluates the quality of each refined trajectory. Specifically, it outputs an imitation-based score SimS_ im to measure human-likeness and a set of rule-based scores SsimS_ sim to assess safety and comfort constraints. These per-anchor scores are further unified via a reward aggregation module into a single reward for each trajectory. During inference, the optimal trajectory is chosen by maximizing the predicted reward. During training, the predicted scores together with the refined trajectories are supervised to optimize the policy in the first stage. I-B Score-Based Policy Learning This section details the supervision and loss formulation for first-stage policy learning. Following established practices [11, 26], we employ a trajectory regression loss ℒtrajL_ traj with a winner-takes-all strategy and an imitation-based score loss ℒimL_ im to encourage human-like trajectory prediction. BEV Segmentation Loss. We introduce an auxiliary BEV semantic segmentation task to enhance scene representation. The BEV map contains multiple categories (e.g., background, road, lane centerline, vehicles) with severe class imbalance, as background and road dominate the scene. Unlike prior work [2, 11] that adopts cross-entropy loss, we supervise the predicted map using focal loss ℒmapL_ map to emphasize hard and minority categories. Rule-Based Score Loss. To enforce safety and physical constraints beyond imitation, we supervise the predicted rule-based scores SsimS_ sim using precomputed outputs Ssim∗S_ sim^* from NAVSIM simulator [27]. The simulator provides five rule-based metrics: No At-Fault Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Comfort (C), and Ego Progress (EP). The distribution of rule-based scores is heavily skewed, as the majority of anchors correspond to low-quality behaviors. To mitigate this imbalance, we adopt a Sigmoid Focal Loss (SFLoss) for supervision. Notably, for the EP score, we introduce a conditional progress mask. Since EP measures agent progress but is set to zero when either NC or DAC is zero, the mask ensures that EP loss is applied only for trajectories that are safe and legal. This prevents the model from being misled by unsafe behavior while still learning meaningful progress signals. As a result, the loss is defined as: ℒsim=MaskEP⋅SFLoss(Ssim,Ssim∗),ifsim=EPSFLoss(Ssim,Ssim∗),otherwise.L_ sim= casesMask_ EP· SFLoss (S_ sim,S_ sim^* ),&if\ sim= EP\\ SFLoss (S_ sim,S_ sim^* ),&otherwise. cases\\ (4) where MaskEP=(SNC>0.5)∧(SDAC>0.5)Mask_ EP=(S_ NC>0.5) (S_ DAC>0.5). Finally, the total loss in the first stage is defined as: ℒtotal=λmapℒmap+λtrajℒtraj+λimℒim+λsimℒsimL_total= _ mapL_ map+ _ trajL_ traj+ _ imL_ im+ _ simL_ sim (5) where λmap,λtraj,λim,λsim _ map, _ traj, _ im, _ sim balance the contributions of the respective loss terms. I-C Reward-Guided Fine-Tuning The first-stage policy learning yields a pretrained reference policy πref _ ref, which predicts per-anchor scores and refined trajectories. However, regressed absolute scores can be noisy and fail to explicitly model relative preferences among candidate trajectories. In reality, driving decisions rely more on relative preferences than absolute scoring. To bridge this gap, we introduce a reward-guided fine-tuning strategy (see Fig. 3) that combines two supervision signals: i) reward supervision over all candidate trajectories and i) preference-based optimization by unpaired trajectory samples with preference annotations. Preference-based optimization captures trajectory preferences but is prone to training instability and distribution drift when applied alone. Inspired by [8], we introduce reward supervision as an auxiliary loss to mitigate this issue, which also helps to keep the training objective consistent with trajectory selection. In particular, we break down this process into two key components: reward design and preference-based optimization in Stage I. Figure 3: Illustration of our two-stage training pipeline. In Stage I, πθ _θ is pretrained using expert demonstrations and rule-based scores to learn human-like driving priors. In Stage I, rule-based scores are combined to compute a trajectory-level reward loss across all candidates. In addition, unpaired preference data are used to compute preference loss (see Eq. (10)), enabling the model to learn safer and more reliable trajectory selection. Reward Design. During reward-guided fine-tuning, we employ the reward aggregation module to unify the per-anchor scores predicted by the policy model into a single scalar reward for each trajectory. For each candidate trajectory y under scene x, the reward R(x,y)R(x,y) is computed as a weighted combination of the imitation score SimS_ im and rule-based scores Ssim∈SNC,SDAC,SEP,STTC,SCS_ sim∈\S_ NC,S_ DAC,S_ EP,S_ TTC,S_ C\ in log space: R(x,y)= R(x,y)= ω1logSim+ω2logSNC+ω3logSDAC _1 S_ im+ _2 S_ NC+ _3 S_ DAC +ω4log5SEP+5STTC+2SC12 + _4 5S_ EP+5S_ TTC+2S_ C12 (6) where ωii=14\ _i\_i=1^4 are the hyperparameters that balance driving style and safety. We use log-space aggregation to reflect the conjunctive nature of the safety-related criteria, so that high EP, TTC, or C scores cannot compensate for near-zero NC or DAC scores. Unlike a conventional weighted sum, which allows compensation across all components, the weighted sum of logarithms is equivalent to the logarithm of a weighted geometric product. Consequently, when any individually logged factor approaches zero, the aggregated score decreases sharply, strongly penalizing the corresponding trajectory. These scores serve as unified reward signals for both direct supervised training and subsequent preference-based optimization. Preference-Based Optimization. To further align the policy with robust driving behaviors, we introduce Kahneman–Tversky Optimization (KTO) [28], which encourages desirable outputs while penalizing undesirable ones using unpaired samples rather than strict pairwise data. To enable preference-based fine-tuning, our first step is to construct preference samples. As illustrated in Fig. 4, we utilize the pretrained model πref _ ref to generate multiple trajectories and feed them into an offline evaluator to obtain trajectory-level imitation scores, rule-based scores, and aggregated rewards. We restrict preference partitioning to the top 25% of candidates ranked by L2 similarity to the expert demonstration, since the remaining trajectories deviate substantially from expert behavior and are therefore excluded from fine-grained preference construction. These human-like trajectories are then partitioned into three disjoint sets according to two empirically defined reward thresholds, τhigh=−0.6 _ high=-0.6 and τlow=−1.0 _ low=-1.0. Trajectories with rewards above the threshold τhigh _ high form the desirable set +D^+, corresponding to strictly safe behaviors. Those with rewards below the threshold τlow _ low form the undesirable set −D^-, corresponding to safety-violating behaviors. The remaining trajectories constitute the reference set refD ref, which contains safe yet suboptimal behaviors and is used to estimate the reference point as the average performance of the reference model. This partition enables the model to capture finer-grained preferences among trajectories with similar quality, providing more informative supervision for trajectory ranking than rule-based scores. Figure 4: Illustration of our construction of reward signal and preference data. The policy model pretrained in Stage I first generates candidate trajectories, which are then evaluated by an offline evaluator to obtain reward scores. The resulting rewards are used both for dense reward supervision and for constructing unpaired preference data by selecting +D^+, −D^-, refD ref. Second, we define a relative reward and formulate the corresponding loss accordingly. We use the relative reward rθ(x,y)r_θ(x,y) to measure the preference of the current policy for output y relative to the reference policy under scene x. We denote πθ(y∣x) _θ(y x) as the probability assigned to the trajectory y by the current policy, obtained by normalizing the aggregated rewards R(x,y)R(x,y) over the set of trajectory anchors V using a softmax operation: πθ(y∣x)=exp(R(x,y′))∑y′∈exp(R(x,y′)) _θ(y x)= (R(x,y ) ) _y (R(x,y ) ) (7) Similarly, πref(y∣x) _ ref(y x) denotes the corresponding probability assigned by the frozen reference policy. Thus, the relative reward is given by: rθ(x,y)=logπθ(y∣x)−logπref(y∣x)r_θ(x,y)= _θ(y x)- _ ref(y x) (8) A larger rθ(x,y)r_θ(x,y) indicates that πθ _θ assigns a higher preference to y compared to πref _ ref, while a smaller value suggests the opposite. We further estimate reference points to stabilize training. KTO was originally proposed for text generation, where the output space is continuous and many desirable samples are available. In contrast, anchor-based trajectory prediction operates over a discrete candidate set, in which a large portion of sampled trajectories may be infeasible or unsafe. Consequently, computing the reference point over all predicted trajectories can be biased, as low-quality samples may dominate the expectation. To mitigate this, we estimate the reference point z0z_0 as the average relative reward over the reference set: z0=y′∼ref[rθ(x,y′)]z_0=E_y ref [r_θ(x,y ) ] (9) Finally, we formulate the preference loss objective: ℒKTO _ KTO =(x,y)[w(y)(1−v(x,y))], =E_(x,y) [w(y) (1-v(x,y) ) ], (10) v(x,y) v(x,y) =σ(β(rθ(x,y)−z0)),if y∈+σ(β(z0−rθ(x,y))),if y∈− = casesσ (β (r_θ(x,y)-z_0 ) ),if y ^+\\ σ (β (z_0-r_θ(x,y) ) ),if y ^- cases where w(y)w(y) denotes the sample weight and is set to λD _D or λU _U for desirable or undesirable samples, respectively. σ(⋅)σ(·) denotes the sigmoid function, and β>0β>0 controls the sensitivity of the objective to deviations of the relative reward from the reference point. This objective encourages the policy to increase the relative likelihood of desirable trajectories while suppressing undesirable ones with respect to the reference distribution. TABLE I: Performance comparison with representative methods on the navtest split. C: camera. L: LiDAR. Method Input Supervision Anchor NC↑ DAC↑ EP↑ TTC↑ C↑ PDMS↑ (Overall) Human - - - 100.0 100.0 87.5 100.0 99.9 94.8 PDM-closed Perception GT - - 94.6 99.8 89.9 86.9 99.9 89.1 Transfuser [2] C & L Human - 97.7 92.8 79.2 92.8 100.0 84.0 UniAD [3] C Human - 97.8 91.9 78.8 92.9 100.0 83.4 VADv2 [4] C Human - 97.9 91.7 77.6 92.9 100.0 83.0 PARA-Drive [29] C Human - 97.9 92.4 79.3 93.0 99.8 84.0 LAW [30] C Human - 96.4 95.4 81.7 88.7 99.9 84.6 DRAMA [31] C & L Human - 98.0 93.1 80.1 94.8 100.0 86.5 DiffusionDrive [21] C & L Human - 98.2 96.2 94.7 82.2 100.0 88.1 Hydra-MDP [11] C & L Human & Rule 8192 98.3 96.0 78.7 94.6 100.0 86.5 WoTE [12] C & L Human & Rule 256 98.5 96.8 81.9 94.9 99.9 88.3 EMPlan (Ours) C Human & Rule 256 98.4 97.5 84.2 94.9 100.0 89.6 Figure 5: Qualitative comparison of predicted trajectories across challenging scenarios. We present qualitative comparisons with TransFuser [2], Hydra-MDP [11], and the SOTA WoTE [12]. In (a), both TransFuser and Hydra-MDP fail to respond properly to the red traffic signal, causing rear-end collisions with the leading vehicle. In addition, all baselines exhibit lane deviations in (b). WoTE tends to be overly conservative, resulting in stagnation behaviors in (c). In (d), TransFuser and Hydra-MDP proceed straight toward the construction obstruction, while WoTE avoids collision but fails to commit to the required right turn. In contrast, our EMPlan successfully executes all maneuvers. IV Experiment IV-A Experimental Setup Dataset. We conduct experiments on NAVSIM [27], an E2E driving benchmark built from the OpenScene [32] redistributed version of nuPlan [33]. NAVSIM provides 2 Hz sensor inputs, along with semantic BEV maps and 3D bounding box annotations of objects. NAVSIM provides 1192 scenes in the navtrain split for training and validation, and 136 scenes in the navtest split for testing. Metrics. Following prior work [11, 12, 21], we use the Predictive Driving Model Score (PDMS) as the primary evaluation metric under a non-reactive simulation setting. PDMS is computed from five sub-metrics: NC, DAC, TTC, C, and EP. The final score is defined as PDMS=NC×DAC×(5EP+5TTC+2C)/12PDMS=NC×DAC×(5EP+5TTC+2C)/12. Implementation Details. We use concatenated 1024×2561024× 256 multi-view RGB images for perception input, following the commonly adopted setting in Transfuser [2]. We use V2-99 [34] as the backbone to extract image features. We set N=256N=256 predefined trajectory anchors. The reward weights w2=w3=0.5w_2=w_3=0.5, w4=1.0w_4=1.0 are set informed by the relative importance of components in the PDMS metric, and a smaller weight w1=0.1w_1=0.1 is assigned to the imitation term to prioritize safety over style imitation. Training is performed on a platform with two NVIDIA RTX A6000 GPUs. For the first-stage training, we set λmap=10.0 _ map=10.0, λtraj=1.0 _ traj=1.0, λim=1.0 _ im=1.0, and λsim=NC:5.0,DAC:6.0,EP:3.0,TTC:3.0,C:1.0 _ sim=\ NC:5.0, DAC:6.0, EP:3.0, TTC:3.0, C:1.0\ to balance the loss term. We pretrain the model for 35 epochs with AdamW at a learning rate of 1×10−41× 10^-4 and a batch size of 16 per GPU. In the fine-tuning stage, we further optimize the policy for 10 epochs with a learning rate of 5×10−55× 10^-5 and a batch size of 16 per GPU in an iterative manner. During inference, the model predicts 8-waypoint trajectories over a 4-second horizon. IV-B Comparisons with State-of-the-art Methods TABLE I: Efficiency comparison between anchor-based methods. All models were tested on an NVIDIA RTX A6000. Method PDMS↑ Latency↓ Hydra-MDP [11] 86.5 104.7 ms WoTE [12] 88.2 28.3 ms EMPlan (Ours) 89.6 21.4 ms TABLE I: Ablation study for offset refinement and SFLoss. offset SFLoss PDMS↑ ✗ ✗ 81.6 ✓ ✗ 87.8 ✓ ✓ 88.3 TABLE IV: Analysis on number of predefined anchors N. N PDMS↑ Latency↓ Param. 128 88.2 17.5 ms 256 88.3 21.4 ms 113 M 512 88.4 24.8 ms TABLE V: Ablation study of KTO with auxiliary supervision (Sup.). Method PDMS↑ w/o Fine-tuning 88.3 KTO + Score Sup. 89.0 KTO + Reward Sup. 89.6 To evaluate the effectiveness of EMPlan, we perform experiments on the NAVSIM navtest split against open-sourced representative planning baselines. These include the imitation-based Transfuser [2], the diffusion-based DiffusionDrive [21], and the scoring-based Hydra-MDP [11] and WoTE [12], where WoTE represents the previous state-of-the-art (SOTA). As shown in Table I, our model achieves the best PDMS of 89.6, indicating superior overall driving performance. Among the baselines, Transfuser (84.0) suffers from causal confusion and often produces unsafe behaviors in complex interactions, including collisions and drivable-area violations, as shown in Figs. 5(a, b). DiffusionDrive (88.1) achieves the highest EP but underperforms our method on critical safety indicators, particularly NC and DAC. This highlights that PDMS provides a more reliable evaluation of driving performance, whereas a sub-metric alone can be misleading. Scoring-based methods such as Hydra-MDP (86.5) and WoTE (88.3) reduce collision rates but still struggle with consistent lane-keeping (see Fig. 5(b)), making them highly prone to off-road incidents. More importantly, their safety gains largely stem from overly conservative behavior, resulting in unnecessary braking and stagnated behavior as reflected by lower EP. As illustrated in the U-turn scenario in Fig. 5(c), both Hydra-MDP and WoTE fail to complete the maneuver and remain stalled, whereas our EMPlan successfully executes the turn. As further illustrated in the right-turn scenario with road construction in Fig. 5(d), Transfuser and Hydra-MDP fail to recognize the construction cones and proceed straight, resulting in collisions. Although WoTE avoids an immediate collision, continuing straight remains potentially unsafe given the obstruction ahead. In contrast, our EMPlan executes a smooth and decisive right turn, demonstrating its ability to suppress trajectories that appear human-like but carry latent safety risks. Table I compares our EMPlan with publicly available representative anchor-based methods with respect to inference efficiency. Notably, our method achieves the lowest latency (21.4 ms) while maintaining the highest PDMS score (89.6). This efficiency gain is primarily attributed to our hybrid architecture, which bypasses the heavy computational burden from dense anchors. In addition, EMPlan relies solely on camera inputs and operates at real-time inference speed, further enhancing its suitability for real-world deployment. IV-C Ablation Study Table V presents the ablation results of offset-based trajectory refinement and SFLoss. Adding offset results in a notable performance gain, improving PDMS from 81.6 to 87.8. This is primarily because the base model relies on sparse predefined anchors that cannot fully cover diverse driving behaviors, while offset refinement enables fine-grained trajectory adjustments beyond discrete anchor limitations. Incorporating SFLoss further increases PDMS to 88.3 by emphasizing hard samples during training. Table V analyzes the effect of the number of predefined anchors N. As Nincreases from 128 to 512, PDMS improves slightly from 88.2 to 88.4, indicating that a denser anchor set provides better coverage of diverse driving behaviors. However, increasing N also leads to noticeably higher inference latency (from 17.5 ms to 24.8 ms), which may hinder efficiency in closed-loop evaluation. Considering both accuracy and efficiency, we adopt N=256N=256, as it provides a favorable trade-off between accuracy and efficiency. Figure 6: Visualization of reward distributions over candidate trajectories, colored by reward value. Unsafe trajectories (e.g., collisions or off-road driving, red circles) are consistently assigned low rewards, while feasible trajectories near the ground truth (yellow circles) receive higher rewards, demonstrating that the reward-guided policy effectively suppresses unsafe behaviors while preserving multimodal diversity. Table V evaluates different fine-tuning objectives. Compared to the model without fine-tuning, incorporating KTO with auxiliary supervision consistently improves PDMS. In particular, KTO complements the supervision loss by encouraging the policy to favor more desirable trajectories over undesirable ones, while the auxiliary supervision provides dense training signals for stable optimization. Moreover, reward supervision achieves the best performance (89.6), outperforming score supervision (89.0) used in Stage I. This suggests that the aggregated reward signal offers more task-aligned guidance, as it better matches the final trajectory selection criterion used during inference. To better understand how the model ranks trajectory candidates, Fig. 6 presents the reward distributions over predicted trajectories across diverse scenarios. The learned reward exhibits a clear separation between safe and unsafe trajectories while preserving multimodal feasible solutions. For instance, in the left-turn scenario (see Fig. 6(c)), trajectories corresponding to the intended turning behavior receive higher rewards, whereas alternative non-turning trajectories are consistently suppressed. Similar patterns are observed in Figs. 6(a, b, d), where trajectories leading to collision or off-road driving are assigned low rewards, while those closer to the ground truth obtain relatively higher rewards. V Conclusion We present EMPlan, an efficient multi-modal trajectory planning framework based on a hybrid architecture and reward-guided fine-tuning. Our method enables real-time multi-modal trajectory generation while improving planning performance beyond pure imitation learning. Experiments on the non-reactive NAVSIM benchmark demonstrate that EMPlan achieves state-of-the-art performance with competitive inference efficiency. A limitation, however, is that our reward-guided training relies on simulator-based evaluation, which prevents us from directly scaling to benchmarks that lack compatible simulation environments. In future work, we plan to explore simulator-free reward modeling strategies to improve training scalability, and we will also explore reactive closed-loop evaluation to better assess how the model behaves when other agents respond to its actions. References [1] S. Hu, L. Chen, P. Wu, H. Li, J. Yan, and D. Tao (2022) ST-P3: end-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, p. 533–549. Cited by: §I. [2] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger (2022) TransFuser: imitation with transformer-based sensor fusion for autonomous driving. IEEE Trans. Pattern Anal. Mach. Intell. 45 (11), p. 12878–12895. Cited by: Fig. 1, §I, §I-A, Fig. 5, §I-B, TABLE I, §IV-A, §IV-B. [3] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. (2023) Planning-oriented autonomous driving. In CVPR, p. 17853–17862. Cited by: Fig. 1, §I, §I-A, TABLE I. [4] S. Chen, B. Jiang, H. Gao, B. Liao, Q. Xu, Q. Zhang, C. Huang, W. Liu, and X. Wang (2024) VADv2: end-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243. Cited by: §I, §I-A, §I-A, TABLE I, §I. [5] D. Chen and P. Krähenbühl (2022) Learning from all vehicles. In CVPR, p. 17222–17231. Cited by: §I. [6] M. Bansal, A. Krizhevsky, and A. Ogale (2018) ChauffeurNet: learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079. Cited by: §I. [7] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann (2020) Shortcut learning in deep neural networks. Nat. Mach. Intell. 2 (11), p. 665–673. Cited by: §I. [8] Y. Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, R. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson, et al. (2023) Imitation is not enough: robustifying imitation with reinforcement learning for challenging driving scenarios. In IROS, p. 7553–7560. Cited by: §I, §I-C. [9] H. Gao, S. Chen, B. Jiang, B. Liao, Y. Shi, X. Guo, Y. Pu, H. Yin, X. Li, X. Zhang, et al. (2025) RAD: training an end-to-end driving policy via large-scale 3DGS-based reinforcement learning. arXiv preprint arXiv:2502.13144. Cited by: §I, §I-B. [10] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, p. 627–635. Cited by: §I. [11] Z. Li, K. Li, S. Wang, S. Lan, Z. Yu, Y. Ji, Z. Li, Z. Zhu, J. Kautz, Z. Wu, et al. (2024) Hydra-MDP: end-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06978. Cited by: Fig. 1, §I, §I-A, Fig. 5, §I-A, §I-B, §I-B, TABLE I, §IV-A, §IV-B, TABLE I. [12] Y. Li, Y. Wang, Y. Liu, J. He, L. Fan, and Z. Zhang (2025) End-to-end driving with online trajectory evaluation via BEV world model. arXiv preprint arXiv:2504.01941. Cited by: Fig. 1, §I, §I-A, Fig. 5, §I-A, TABLE I, §IV-A, §IV-B, TABLE I. [13] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. NeurIPS 30. Cited by: §I. [14] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. NeurIPS 35, p. 27730–27744. Cited by: §I. [15] W. Xiao, Z. Wang, L. Gan, S. Zhao, Z. Li, R. Lei, W. He, L. A. Tuan, L. Chen, H. Jiang, et al. (2024) A comprehensive survey of direct preference optimization: datasets, theories, variants, and applications. arXiv preprint arXiv:2410.15595. Cited by: §I. [16] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. NeurIPS 36, p. 53728–53741. Cited by: §I. [17] S. Shang, Y. Chen, Y. Wang, Y. Li, and Z. Zhang (2025) DriveDPO: policy learning via safety dpo for end-to-end autonomous driving. arXiv preprint arXiv:2509.17940. Cited by: §I, §I-B. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. NeurIPS 30. Cited by: §I-A. [19] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang (2023) VAD: vectorized scene representation for efficient autonomous driving. In ICCV, p. 8340–8350. Cited by: §I-A. [20] X. Li, Y. Zhang, and X. Ye (2024) DrivingDiffusion: layout-guided multi-view driving scenarios video generation with latent diffusion model. In ECCV, p. 469–485. Cited by: §I-A. [21] B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, et al. (2025) DiffusionDrive: truncated diffusion model for end-to-end autonomous driving. In CVPR, p. 12037–12047. Cited by: §I-A, TABLE I, §IV-A, §IV-B. [22] B. Jiang, S. Chen, Q. Zhang, W. Liu, and X. Wang (2025) AlphaDrive: unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608. Cited by: §I-B. [23] D. Li, J. Ren, Y. Wang, X. Wen, P. Li, L. Xu, K. Zhan, Z. Xia, P. Jia, X. Lang, et al. (2025) Finetuning generative trajectory model with reinforcement learning from human feedback. arXiv preprint arXiv:2503.10434. Cited by: §I-B. [24] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §I-B. [25] Y. Li, K. Xiong, X. Guo, F. Li, S. Yan, G. Xu, L. Zhou, L. Chen, H. Sun, B. Wang, et al. (2025) ReCogDrive: a reinforced cognitive framework for end-to-end autonomous driving. arXiv preprint arXiv:2506.08052. Cited by: §I-B. [26] Z. Zhou, J. Wang, Y. Li, and Y. Huang (2023) Query-centric trajectory prediction. In CVPR, p. 17863–17873. Cited by: §I-B. [27] D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, et al. (2024) NAVSIM: data-driven non-reactive autonomous vehicle simulation and benchmarking. NeurIPS 37, p. 28706–28719. Cited by: §I-B, §IV-A. [28] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) Model alignment as prospect theoretic optimization. In ICML, Cited by: §I-C. [29] X. Weng, B. Ivanovic, Y. Wang, Y. Wang, and M. Pavone (2024) Para-Drive: parallelized architecture for real-time autonomous driving. In CVPR, p. 15449–15458. Cited by: TABLE I. [30] Y. Li, L. Fan, J. He, Y. Wang, Y. Chen, Z. Zhang, and T. Tan (2024) Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.08481. Cited by: TABLE I. [31] C. Yuan, Z. Zhang, J. Sun, S. Sun, Z. Huang, C. D. W. Lee, D. Li, Y. Han, A. Wong, K. P. Tee, et al. (2024) DRAMA: an efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.03601. Cited by: TABLE I. [32] S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser, et al. (2023) OpenScene: 3D scene understanding with open vocabularies. In CVPR, p. 815–824. Cited by: §IV-A. [33] H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari (2021) nuPlan: a closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810. Cited by: §IV-A. [34] Y. Lee, J. Hwang, S. Lee, Y. Bae, and J. Park (2019) An energy and gpu-computation efficient backbone network for real-time object detection. In CVPRW, Cited by: §IV-A.