Paper deep dive
Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning
Alessandro Canevaro, Hang Yu, Julian Schmidt, Peizheng Li, Silvan Lindner, Wilhelm Stork, Georg Martius, Julian Jordan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/10/2026, 2:28:19 AM
Summary
The paper introduces Shift & Drift, a novel dual-track benchmark for evaluating the generalization and robustness of autonomous driving motion planners. The Semantic Shift Track utilizes the DeepPlan dataset (converted from the DSC3D aerial dataset to nuPlan format) to test zero-shot performance across 1,182 scenarios in German and US cities featuring dense pedestrian-cyclist interactions. The State-Distribution Drift Track injects stochastic noise (AWGN and Ornstein-Uhlenbeck processes) into vehicle dynamics to measure robustness against compounding execution errors. Evaluations reveal that imitation learning-based planners struggle with semantic shifts and drift, while reinforcement learning-based planners (e.g., CaRL) demonstrate superior resilience and graceful degradation, highlighting a trade-off between imitation fidelity and closed-loop robustness.
Entities (10)
Relation Signals (12)
Shift & Drift → hastrack → Semantic Shift Track
confidence 97% · we present Shift & Drift, a novel dual-track benchmark designed to rigorously stress-test motion planners across two critical axes of distribution shift: (1) The Semantic Shift Track... (2) The State-Distribution Drift Track
Shift & Drift → hastrack → State-Distribution Drift Track
confidence 97% · we present Shift & Drift, a novel dual-track benchmark designed to rigorously stress-test motion planners across two critical axes of distribution shift: ... (2) The State-Distribution Drift Track
DeepPlan → coverslocation → San Francisco
confidence 96% · The finalized evaluation set comprises 1182 scenarios distributed as follows: 205 for Munich, 206 for Stuttgart, 490 for Sindelfingen, 101 for Berlin and 180 for San Francisco.
DeepPlan → coverslocation → Munich
confidence 96% · The finalized evaluation set comprises 1182 scenarios distributed as follows: 205 for Munich, 206 for Stuttgart, 490 for Sindelfingen, 101 for Berlin and 180 for San Francisco.
Semantic Shift Track → usesdataset → DeepPlan
confidence 95% · The foundation of this track is DeepPlan, a novel benchmark generated via a pipeline that converts the high-precision, real-world aerial DSC3D [6] data into a format compatible with the standardized nuPlan [7] simulator.
DeepPlan → derivedfrom → DSC3D
confidence 94% · We present DeepPlan, a suite of nuPlan-compatible scenarios created by mapping the DeepScenario Open 3D Dataset (DSC3D) [6] into the nuPlan framework.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While closed-loop motion planners trained on large-scale, object-level datasets, e.g., nuPlan, demonstrate strong in-distribution (ID) performance, their generalization to novel urban topologies and recovery mechanisms following execution perturbations remain under-explored. To address this, we present Shift & Drift, a novel dual-track benchmark designed to rigorously stress-test motion planners across two critical axes of distribution shift: (1) The Semantic Shift Track leverages a novel conversion pipeline that transforms the aerial, DeepScenario Open 3D dataset into the nuPlan simulation framework. This enables zero-shot evaluation of planners trained on North American and Singaporean data against 1,182 scenarios spanning four German cities and the US city of San Francisco featuring dense pedestrian-cyclist interactions. (2) The State-Distribution Drift Track injects stochastic perturbations into the ego vehicle's dynamics to quantify robustness against compounding execution errors. Based on this, we systematically evaluate the failure modes of diverse planning paradigms under semantic and state-distribution shifts. While imitation learning methods achieve high scores in ID benchmarks, they exhibit significant failures under semantic shift, particularly in pedestrian-dense environments, and suffer from persistent drift when subjected to temporally correlated actuation noise. In contrast, the evaluated reinforcement-learning-based planner demonstrates more graceful degradation, maintaining higher safety and progress metrics across both tracks. Our findings reveal an empirical trade-off between imitation fidelity and closed-loop resilience, providing the community with a rigorous benchmark to evaluate progress toward reliable deployment.
Tags
Links
- Source: https://arxiv.org/abs/2607.07844v1
- Canonical: https://arxiv.org/abs/2607.07844v1
Trouble viewing inline? Open PDF directly →
Full Text
44,997 characters extracted from source content.
Expand or collapse full text
Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning Alessandro Canevaro 1,2,∗,† , Hang Yu 1,3,∗ , Julian Schmidt 1 , Peizheng Li 1,2 , Silvan Lindner 1 , Wilhelm Stork 3 , Georg Martius 2 , and Julian Jordan 1 Abstract— While closed-loop motion planners trained on large-scale, object-level datasets, e.g., nuPlan, demonstrate strong in-distribution (ID) performance, their generalization to novel urban topologies and recovery mechanisms following ex- ecution perturbations remain under-explored. To address this, we present Shift & Drift, a novel dual-track benchmark designed to rigorously stress-test motion planners across two critical axes of distribution shift: (1) The Semantic Shift Track leverages a novel conversion pipeline that transforms the aerial, DeepSce- nario Open 3D dataset into the nuPlan simulation framework. This enables zero-shot evaluation of planners trained on North American and Singaporean data against 1,182 scenarios span- ning four German cities and the US city of San Francisco featuring dense pedestrian-cyclist interactions. (2) The State- Distribution Drift Track injects stochastic perturbations into the ego vehicle’s dynamics to quantify robustness against compounding execution errors. Based on this, we systematically evaluate the failure modes of diverse planning paradigms under semantic and state-distribution shifts. While imitation learning methods achieve high scores in ID benchmarks, they exhibit significant failures under semantic shift, particularly in pedestrian-dense environments, and suffer from persistent drift when subjected to temporally correlated actuation noise. In contrast, the evaluated reinforcement-learning-based planner demonstrates more graceful degradation, maintaining higher safety and progress metrics across both tracks. Our findings reveal an empirical trade-off between imitation fidelity and closed-loop resilience, providing the community with a rigorous benchmark to evaluate progress toward reliable deployment. I. INTRODUCTION The advancement of learning-based Autonomous Driving (AD) planners has been primarily driven by large-scale datasets and standardized simulators [1]. However, a signif- icant gap remains between performance in familiar training environments and reliability in the real world. Current State- of-the-Art (SOTA), Imitation Learning (IL), Reinforcement Learning (RL), and rule-based planners, are predominantly evaluated on the same geographic distributions found in their training sets [2], [3]. This reliance on i.i.d. (independent and identically distributed) data assumptions often masks a critical lack of generalization. As noted by Codevilla et al. This work is a result of the joint research project STADT:up (19A22006O). The project is supported by the German Federal Ministry for Economic Affairs and Energy (BMWE), based on a decision of the German Bundestag. The authors are solely responsible for the content of this publication. Code and data are available at: https://github.com/alessandro-canevaro/Shift-Drift. 1 Mercedes-Benz AG, Research & Development, Sindelfingen, Germany 2 University of T ̈ ubingen, T ̈ ubingen, Germany 3 Karlsruhe Institute of Technology, ITIV, Karlsruhe, Germany † alessandro.canevaro@mercedes-benz.com ∗ Equal contribution. State-Dristribution Drift Track Standardized nuPlan evaluation format Semantic Shift Track DSC3D urban drone data Stochastic noise injection Drift Noise Fig. 1: Overview of our dual-track benchmark: Shift & Drift. Track 1: Semantic Shift Track converts aerial DSC3D data [6] into nuPlan [7] standard for zero-shot evaluation. Track 2: State-Distribution Drift Track injects different types of noise into actuation. [4], models that excel in known urban layouts often suffer from geographic overfitting, where the policy implicitly memorizes map-specific features or localized traffic patterns rather than learning universal driving priors. This challenge is further compounded by causal confusion [5], where planners may latch onto spurious correlations rather than transferable driving priors. Similarly, rule-based planners can face scala- bility challenges when cost-function tuning must account for diverse long-tail scenarios. Specifically, a manifestation of these challenges is the significant degradation in planning capability when mod- els encounter semantic shift, such as novel urban layouts or intricate multi-agent interactions featuring high-density pedestrians and cyclists. While benchmarks like nuPlan [7] and Waymo open dataset [8] have scaled the volume of data, they still fall short of covering all potential edge cases [9], [10], particularly complex, high-density interaction scenarios. Furthermore, even when a planner exhibits high fidelity in nominal conditions, it often lacks robustness against state-distribution shift. This compounding error problem, fundamentally identified in [11], was later formalized in the work [12]. It is observed that a model trained strictly on expert trajectories lacks the recovery behaviors necessary to handle the drift caused by small, cumulative execution error. While closed-loop evaluation has become the standard for AD planners, current benchmarks like CARLA [13] or nuPlan often utilize simplified vehicle dynamics models. In real-world execution, errors in the dynamic model, sen- sor latency, and stochastic actuation errors are inevitable. arXiv:2607.07844v1 [cs.RO] 8 Jul 2026 Without mechanisms to force a planner to recover from these off-policy states, closed-loop success scores remain an optimistic upper bound rather than a measure of true deployment readiness. In this paper, we introduce Shift & Drift, a new benchmark designed to stress-test the generalization and robustness of motion planners. Our benchmark, as shown in Fig. 1, focuses on two primary axes of distribution shift relevant for safety- critical deployment: Semantic Shift through Cross-Dataset Generalization. We present DeepPlan, a suite of nuPlan-compatible scenarios created by mapping the DeepScenario Open 3D Dataset (DSC3D) [6] into the nuPlan framework. This allows us to leverage the high-precision, occlusion-free, drone-recorded aerial trajectories of DSC3D, captured across five diverse international locations, to conduct a rigorous zero-shot eval- uation of models trained on the original nuPlan dataset. Beyond quantifying policy generalization versus map mem- orization, this benchmark evaluates planner performance in more challenging, high-density environments characterized by extensive pedestrian-cyclist interactions. State-Distribution Drift through Dynamic Noise Injec- tion. We formalize the state distributional drift by inject- ing stochastic perturbations into the ego-vehicle’s transition dynamics during closed-loop simulation. To model different noise characteristics, we employ Additive White Gaussian Noise (AWGN) to represent high-frequency uncorrelated jit- ter and the Ornstein-Uhlenbeck (OU) process [14] to capture low-frequency systematic errors. This forces the planner to recover from off-policy states, providing a quantitative measure of robustness and recovery capability. We evaluate a comprehensive suite of SOTA planners, including PDM-Closed [15], PlanTF [16], PLUTO [17], Diffusion Planner [18] and CaRL [19]. Our results pro- vide empirical evidence for the inherent trade-offs between different learning paradigms: while IL-based models often excel in nominal conditions, they exhibit significant fragility in both evaluation tracks, with overall score performance dropping up to 76% in novel urban environments. Rule-based models also drop in performance significantly, typically with a decrease in progress, but without compromising safety, achieving the lowest number of collisions. Conversely, our evaluation shows that CaRL, an RL-based method, exhibits superior resilience across all tracks, in particular exhibiting at most an 8% performance decay even under high-intensity state-distribution drift. I. RELATED WORK A. Large-Scale Autonomous Driving Datasets Large-scale datasets serve as the cornerstone for modern learning-based motion planners. While early benchmarks primarily supported perception tasks [20], recent datasets such as Argoverse 2 [21], Waymo Open [8] and nuPlan [7] provide the large-scale, multi-agent trajectories required for planning. Recent efforts like ScenarioNet [22] have further expanded this landscape by providing a unified platform to aggregate these heterogeneous datasets into a common simulation format. However, these datasets are derived from ego-centric sensor logs, which are inherently subject to occlusion [23], [24]. While occlusion is often treated as a perception nuisance, it introduces a fundamental challenge in closed-loop simu- lation. When a planner deviates from the recorded expert trajectory, an inevitability in closed-loop evaluation, it may enter spatial regions that were blind spots for the original ego-vehicle. In these off-policy states, the simulator may fail to account for actors that were physically present but occluded during the initial data capture. Consequently, the agent may navigate through phantom empty space, leading to an overestimation of safety and performance. Aerial datasets such as highD [25], INTERACTION [26], DSC3D [6], and DeepUrban [27] mitigate this problem by offering occlusion-free trajectories. However, they are predominantly designed for trajectory forecasting tasks and they lack the standardized simulation interfaces and map formats required for closed-loop planning evaluation. Our work transforms the DSC3D dataset into the nuPlan framework, ensuring that the multi-agent interaction context remains complete regardless of the ego-vehicle’s displace- ment, providing a more rigorous and realistic constraint on the planner’s behavior. B. Motion Planning Paradigms Current SOTA motion planners can be broadly categorized into three paradigms, each with distinct trade-offs regarding generalization and robustness. ILparadigms,includingthetransformer-based PlanTF [16] and the generative Diffusion Planner [18], leverage human data but are limited by their open-loop nature and causal confusion [5], often resulting in a lack of robust recovery behaviors. Hybrid models like PLUTO [17] attempt to bridge this gap using a rule-based refinement layer for traffic law adherence. While interactive techniques like DAgger [12] address distributional drift via expert querying, they scale poorly to real-world datasets where experts are unavailable post-recording. RL approaches, such as CaRL [19], a SOTA nuPlan agent trained via PPO [28] to maximize progress, safety, and comfort, mitigate these shifts through closed-loop explo- ration. However, RL remains computationally expensive and requires extensive reward engineering due to poor sample efficiency. Finally, rule-based planners like PDM-Closed [15] serve as deterministic baselines, utilizing IDM [29] and MOBIL [30] for longitudinal and lateral control. While providing strong safety guarantees and interpretability, they require exhaustive tuning and often fail to generalize to complex long-tail multi- agent interactions. C. Closed-Loop Benchmarking and Generalization Pioneered by simulators like CARLA [13] and bench- marks including Bench2Drive [31], nuPlan [7] and NAVSIM [32], the transition from open-loop displacement metrics to closed-loop simulation has become the standard TABLE I: Comparison of motion planning benchmarks. AttributenuPlanInterPlanDeepPlan (Ours) Data SourceVehicleVehicleAerial Occlusion-Free Tracking ×✓ Geographic CoverageUS + SGUS + SGUS + DE Semantic ShiftLimitedScenario✓ Interaction DensityModerateHighVery High Train–Test Overlap RiskHighModerateLow Scenario Count1118801182 for evaluating sequential decision-making. Recent advance- ments have focused on improving the realism of background agents. For example, nuPlan-R [33] and the integration of SMART agents [34] replace standard rule-based models with learned, reactive agents, complementing our work by improving the realism of simulated traffic agents. However, most existing benchmarks rely on i.i.d. evaluation, where models are tested on held-out data from the same geographic and semantic distribution as the training set. This often masks a critical lack of generalization, as models may implicitly memorize map-specific features or localized traffic patterns. A recent effort to address these deficiencies is interPlan [35], which challenges the optimistic upper bound of the nuPlan benchmark. This refers to the observation that many planners achieve high scores in nuPlan by excelling at standard cruising tasks in simple scenarios, failing to reveal their fragility in complex interactions. While interPlan effectively probes behavioral reliability in rare scenarios, its 80 handcrafted cases do not provide the statistical scale required for comprehensive validation. Our work advances this frontier by enabling large-scale cross-dataset evaluation. By converting 1,100+ diverse real- world recordings from DSC3D, we move beyond manual scenario design to provide a data-driven measure of semantic generalization across international geographic domains (see Table I for a comparative analysis). D. Robustness and Sim-to-Real Gaps The disparity between simulation and reality, often termed the sim-to-real gap, remains a critical hurdle. Prior work explored this through adversarial attacks [36], [37] or by varying visual domains in perception-based pipelines. How- ever, for object-based planning, a subtle but critical shift arises from imperfect low-level control. While robust control theory is well-established, its integration into AD planning benchmarks is under-explored. Unlike adversarial approaches seeking worst-case perturbations, our State-Distribution Drift Track utilizes Gaussian and temporally-correlated noise mod- els to simulate execution errors, providing a practical proxy for a planner’s stability on imperfect hardware. I. THE BENCHMARK Our benchmark is structured into two independent evalua- tion tracks designed to probe the critical limitations of current AD planners along two distinct axes: semantic generalization and robustness to state-distribution shifts. A. The Semantic Shift Track The foundation of this track is DeepPlan, a novel bench- mark generated via a pipeline that converts the high- precision, real-world aerial DSC3D [6] data into a for- mat compatible with the standardized nuPlan [7] simulator. Thus, models trained exclusively on the North American and Singaporean locations within the nuPlan dataset can be evaluated on zero-shot basis in newly converted urban scenes such as Munich, Stuttgart, and others. This track is designed to answer a critical question: have planners learned generalizable driving policies, or have they merely memorized the topology and traffic patterns of their training distribution? The DSC3D scenarios present a significant semantic shift, characterized by: • Novel Urban Topologies: Planners must navigate road layouts and intersection types not present in the original nuPlan dataset like pedestrian areas and parking lots. • Dense Social Interactions: The high density of pedes- trians and cyclists in the selected scenarios significantly exceeds that of standard nuPlan logs, as illustrated in Fig. 2, thereby testing the capacity of planners for com- plex multi-agent reasoning and safe navigation among vulnerable road users. • Regional Driving Styles: The evaluation exposes plan- ners to distinct and localized driving behaviors, and norms inherent to European locations. Our conversion process transforms the DSC3D raw data into nuPlan-compatible log files and vector maps through a multi-stage pipeline: 1) Vector Map Conversion: We process the accompany- ing OpenDRIVE [38] map files for each DSC3D location to extract semantic and geometric primitives (lane bound- aries, centerlines, roadblocks, speed limits). This process includes: (i) automatic intersection detection via spatial overlap analysis of lane polygons, (i) coordinate system alignment from GPS/OpenDRIVE to UTM projections with ego pose registration, and (i) R-tree spatial indexing for efficient map queries during simulation. Outputs are layered in GeoPackage format required by nuPlan, ensuring planners use the same high-level semantic representations consistent with their training. 2) Episode Windowing and Ego Vehicle Selection: Raw 12.5 Hz DSC3D recordings are segmented into 18-second sliding windows (225 frames) with a 6-second overlap to maximize data coverage while maintaining temporal diver- sity. Since DSC3D is a drone-recorded, ego-less dataset, a candidate ego must be identified programmatically. To qualify, every tracked agent is evaluated against the following criteria: (i) it must be classified as a vehicle (excluding pedestrians, cyclists, and trucks); (i) its peak speed over the episode must exceed 1.0 m/s; and (i) it must not experience any bounding-box overlap with surrounding agents. Each qualifying vehicle in a window yields an independent nuPlan database file, so a single window can produce multiple episodes. Each episode is then upsampled to nuPlan’s 20 nuPlan Val14InterPlanDeepPlan (Ours) 0 20 40 60 80 Average agents per scenario 29.5 9.1 74.9 28.5 0.3 49.0 0.2 0.0 9.2 Vehicles Pedestrians Bicycles Fig. 2: Average number of vehicles, pedestrian and bicycles per scenario on nuPlan val14, InterPlan and DeepPlan. Hz via linear interpolation, producing 360-frame episodes. Within each episode, frame 60 is defined as the current timestep (t = 0) delineating 3 seconds of history and 15 seconds of future context. 3) Agent States and Log Generation: For each episode, we compute comprehensive kinematic states (position, ve- locity, acceleration, heading) for all tracked agents. These states, coupled with ego odometry, are serialized into the nuPlan SQLite schema for closed-loop replay. 4) Post-Processing: The raw conversion output is subject to two stages of post-processing: (i) an automated filter evaluates each generated scenario under the nuPlan closed- loop simulator using the log playback planner, and dis- cards database file where the ego exhibits at-fault collisions, drivable-area violations, or insufficient forward progress, indicating a malformed or degenerate scenario. (i) a manual inspection tool renders each remaining scenario with its semantic map layers and recorded ego trajectory, allowing a human reviewer to visually inspect individual scenarios and to correct route roadblock assignments where necessary. Dataset Characteristics: The finalized evaluation set com- prises 1182 scenarios distributed as follows: 205 for Munich, 206 for Stuttgart, 490 for Sindelfingen, 101 for Berlin and 180 for San Francisco. Fig. 2 reports the average number of vehicles, pedestrians and bicycles in the nuPlan Val14, InterPlan and DeepPlan datasets. B. The State-Distribution Drift Track To quantify the robustness of the evaluated planners against compounding errors, a phenomenon where small execution inaccuracies push the agent into out-of-distribution states, we introduce the State-Distribution Drift Track. Rather than faithfully replicating specific hardware imperfections, we employ controlled stochastic models as standardized stress-test proxies. We perturb the ego-vehicle’s command vector u cmd = [a cmd , ̇ δ cmd ] ⊤ , where a cmd is the longitudinal acceleration command and ̇ δ cmd is the steering rate command, using two distinct stochastic mechanisms. 1) Stochastic Actuation Perturbations: We model high- frequency mechanical jitter and control inaccuracies by in- jecting Additive White Gaussian Noise (AWGN) directly into the command outputs: a applied ̇ δ applied = a cmd ̇ δ cmd +ε,ε∼N 0, diag σ 2 a , σ 2 ̇ δ . (1) Becauseε is drawn independently at each timestep, errors are temporally uncorrelated. This tests the planner’s ability to maintain a smooth trajectory despite instantaneous, high- frequency jitter. We evaluate three intensity levels: • Low: σ ̇ δ = 0.1 rad/s, σ a = 0.5 m/s 2 . • Mid: σ ̇ δ = 0.2 rad/s, σ a = 1.0 m/s 2 . • High: σ ̇ δ = 0.3 rad/s, σ a = 1.5 m/s 2 . 2) Correlated Drift: To simulate realistic, low-frequency systematic errors, such as wheel misalignment or environ- mental forces like crosswinds, we perturb the planner’s com- manded acceleration a cmd and steering rate ̇ δ cmd using two independent Ornstein-Uhlenbeck (OU) processes [14]. Given the noise vector x k = [x a k , x ̇ δ k ] ⊤ , the applied commands are: a applied ̇ δ applied = a cmd ̇ δ cmd + x k .(2) Each component x (·) k evolves according to the continuous- time OU SDE dx t = θ (μ− x t )dt + σ dW t ,(3) which is integrated numerically at every simulation step ∆t via the Euler-Maruyama scheme: x k+1 = x k +θ (μ−x k ) ∆t +σ √ ∆t ε k , ε k ∼N (0, 1). (4) The two processes share the same mean-reversion rate θ = 2.0 s −1 and long-term mean μ = 0, but differ in volatility, using the same three σ intensity levels as the AWGN track for direct comparison. The mean-reversion term θ(μ − x k )∆t prevents unbounded random-walk behavior, while the stochastic term σ √ ∆tε k injects new randomness scaled to preserve the correct per-unit-time variance. The re- sulting noise exhibits an autocorrelation that decays as e −θτ , yielding a characteristic correlation time of 1/θ = 0.5 s. This timescale is sufficiently long to produce multi-frame drifts that represent a genuine disturbance (see Fig. 4.4), yet bounded in expectation. Consequently, the planner’s receding-horizon feedback must actively compensate for ac- cumulated error rather than merely reacting to instantaneous perturbations. C. Evaluation Metrics We adopt the standard closed-loop evaluation protocol of nuPlan, which assesses performance across safety, rule compliance, progress, and passenger comfort. The metric suite includes collision avoidance, driving direction compli- ance, drivable area adherence, time-to-collision (TTC), speed limit compliance, and comfort metrics such as acceleration and jerk. Following the original framework, safety-critical violations act as hard constraints where exceeding predefined thresholds results in a scenario score of zero, while the remaining metrics are aggregated into a weighted average to compute the final score. For the DeepPlan evaluation, we adjust a subset of these thresholds (Table I) to account for the unique character- istics of dense, pedestrian-heavy European environments. Unlike the predominantly vehicle-centric, wide-lane traffic in TABLE I: Modified nuPlan evaluation thresholds for Deep- Plan scenarios. MetricDefaultOurs Driving Direction Compliance (m)2.04.0 Driving Direction Violation (m)6.010.0 Drivable Area Violation (m)0.33.0 Max Longitudinal Jerk (m/s 3 )4.136.0 Min Longitudinal Accel. (m/s 2 )-4.05-6.0 Max Longitudinal Accel. (m/s 2 )2.404.0 Minimum TTC (s)1.00.5 nuPlan’s North American logs, DeepPlan scenarios feature significantly higher interaction density and frequent nego- tiation of shared spaces. In these settings, safe, human- like navigation can often require tighter TTC margins or temporary deviations from nominal lane boundaries during yielding maneuvers. To validate these adjustments, we conducted a sanity check by evaluating the original log playback (the human driver) using the default nuPlan thresholds on the DeepPlan set. The log’s score dropped from approximately 90 to 65, primarily due to spurious safety violations in scenarios that were, in reality, safe human maneuvers. Applying the default thresholds would therefore systematically penalize reasonable behavior rather than actual planning failures. Compared to existing benchmarks like interPlan that modify the underlying metric logic, our approach maintains the original nuPlan scoring equations, requiring only adjusting threshold values to ensure the benchmark remains physically meaningful across geographic domains. IV. EXPERIMENTS A. Experimental Setup 1) Baselines: To ensure a comprehensive evaluation across the current planning landscape, we select five SOTA planners representing three dominant paradigms: Rule-Based. We include the nuPlan challenge winner PDM [15]. Imitation Learning. We compare against PlanTF [16], PLUTO [17], and Diffusion Planner [18]. Reinforcement Learning. We evaluate CaRL [19], a recent SOTA closed-loop learning approach. 2) Metrics: Adhering to the standard nuPlan evaluation protocol, we assess planner performance across several met- rics. The final performance is distilled into a composite Closed-Loop Score (CLS), representing the weighted mean of the following: Drivable Area Compliance (DAC): Quantifies adherence to road boundaries and penalizes off-road excursions. Driving Direction Compliance (DDC): Penalizes driving against the flow of traffic. Ego is Comfortable (EIC): Measures kinematic smoothness by penalizing excessive jerk and centripetal acceleration. Ego is Making Progress (EMP): Binary metric validating minimum 20% progress relative to the expert. Ego Progress Along Expert Route (PER): Measures the ratio of ego-to-expert progress along the designated route. Non-Collision Rate (NCR): Safety metric calculating the percentage of scenarios completed without at-fault collision. Speed Limit Compliance (SLC): Penalizes any exceedance of the posted speed limit. Time To Collision (TTC): Measures the temporal buffer to lead agents to penalize near-misses and tailgating. In the following sections we report results on the two nuPlan simulation modes: Non-Reactive (NR) where agents are replayed from the log, and Reactive (R) where neighbor vehicles are controlled by the IDM model. B. Track 1: Semantic Shift Standard closed-loop benchmarks, such as nuPlan Val14, evaluate planners on unseen scenarios drawn from the same geographical regions as the training data. In this within- domain regime, architecturally diverse methods, from rule- based controllers to diffusion transformers, often converge to a narrow performance band (Table I, Val14). Our zero- shot benchmark exposes fundamental differences in gener- alization that these standard evaluations conceal. Table I reports the CLSs for our novel scenarios in Sindelfingen (SIFI), Munich (MUC), Stuttgart (STR), Berlin (BER), San Francisco (SFO) and a metric breakdown of the overall CLS. Results and Analysis. CaRL achieves the highest zero-shot performance with a 72.57 CLS, outperforming the next best baseline (PDM-Closed) by 24.5 points. A possible explanation of its robustness can be identified into three fundamental differences in training methodology: (i) compared to IL that seeks to minimize a divergence from expert trajectories, CaRL optimizes for a reward signal (progress, safety, and comfort) that is directly aligned with the nuPlan evaluation metrics, making it inherently region- agnostic. (i) Furthermore, the stochastic exploration required during RL training forces the agent to visit and recover from off-policy states, effectively acting as a form of implicit data augmentation. (i) CaRL is trained in both NR and R simulations exposing the policy to multiple behaviors. IL methods exhibit severe sensitivity to semantic shift. Diffusion Planner’s CLS collapses to 21.50 in Munich, where complex pedestrian interactions absent from the nu- Plan training distribution dominates the scenario set. This represents a 76% degradation from its Val14 baseline (89.87). PlanTF’s CLS drops from 84.27 (Val14) to 34.00, making it the worst model in terms of progress (PER). PLUTO, which augments IL with a rule-based post-processing layer, achieves comparable overall CLS to Diffusion Planner in NR mode but exhibits an intriguing improvement under R simulation. Its CLS increases from 36.74 (NR) to 63.03 (R), a 71.5% improvement, whereas PDM-Closed and PlanTF improve by only ∼4-15 points. This suggests that PLUTO’s rule-based refinement module benefits from IDM-controlled agents that may provide larger safety margin compared to human log replay, effectively compensating for its IL backbone’s distributional mismatch. PDM-Closed, the purely rule-based planner, occupies a revealing middle ground. While it shares RL’s goal-oriented objective, it lacks the functional flexibility to navigate dense, TABLE I: Zero-Shot Generalization Performance. Comparison of planners on the standard nuPlan validation set (Val14) versus our proposed semantic shift benchmark. nuPlanCities (CLS)Metrics MethodVal14AllSIFIMUCSTRBERSFODACDDCEICEMPPERNCRSLCTTC Non-Reactive Log96.3990.7188.3590.0096.9585.0393.97100.0094.8390.7799.8399.13100.0099.9790.77 PDM-Closed92.8448.0727.2257.0259.5954.5877.7798.8199.1172.1766.6750.2189.2599.9977.75 PlanTF 84.2734.0028.8429.2440.5844.1440.2898.8198.7778.9352.3636.2286.0499.9878.34 PLUTO 92.8836.7411.9445.5755.4461.9859.5399.8299.5169.8952.6939.1488.9899.9980.91 Diffusion planner89.8737.4949.1621.5031.5726.5736.8097.6397.2961.9263.2848.6578.9399.9774.11 CaRL93.8772.5773.5157.5972.7373.3386.4799.7497.0377.6691.3783.2989.2599.9878.08 Reactive Log82.0271.6366.6178.3668.5058.2288.75100.0094.8390.7799.8399.1378.2599.9773.35 PDM-Closed92.1249.9525.2760.7265.4555.4084.0597.8098.9472.0867.8553.0390.1099.9980.29 PlanTF76.9549.4442.5738.2255.9857.4768.9096.7095.3583.0875.2158.7183.7699.9976.31 PLUTO92.0663.0350.2762.6966.4276.8388.2899.9199.0983.2282.5860.5789.3399.9980.67 Diffusion planner82.8049.3948.2834.3062.5941.3659.0089.8494.5469.0390.1074.3371.9999.9966.92 CaRL93.1270.2761.1968.9076.0474.2987.6799.8396.6174.7893.2384.1284.4399.9977.74 푡=2푠푡=4푠푡=6푠푡=8푠푡=10푠 푡=1푠푡=2푠푡=3푠푡=4푠푡=5푠 Pedestrians Cyclists Neighbor Vehicle Fig. 3: Qualitative comparison of planner trajectories across DeepPlan scenarios. We show the ground truth (Log) alongside PlanTF, PDM-Closed, PLUTO, Diffusion Planner, and CaRL. Top row: During a right turn with high pedestrian density, Diffusion Planner and PDM-Closed proceed aggressively, resulting in collisions, while PLUTO and CaRL successfully yield. PlanTF fails to commit to forward progress. Middle row: In an unprotected left turn, several planners fail to account for oncoming traffic. Bottom row (left to right): Diffusion Planner exhibits freezing behavior on an empty road; most planners fail to overtake parked vehicles using the opposite lane, whereas CaRL successfully navigates the obstacle; CaRL invades opposite lane to avoid pedestrian; all planners struggle to predict bus motion, leading to suboptimal stopping; finally, extreme urban density causes universal failure across all models, highlighting the complexity of the DeepPlan environment. non-linear social interactions. It avoids the catastrophic collisions of IL, reaching the highest safety score (NCR) among the evaluated planners (89.25 in NR and 90.10 in R). However, it suffers from low progress with a PER of 50 points, reflecting a tendency to prioritize safety margins. This demonstrates that our benchmark probes planning capabilities beyond what rule-based heuristics can provide, such as adaptive social reasoning in dense interactions. A broader takeaway is that the failure modes differ funda- mentally across paradigms: IL planners more frequently exhibited safety-critical failures (low NCR, low DDC), rule- based planners fail conservatively (low progress, high safety), and the RL planner degrades gracefully, maintaining both safety and progress under significant distributional shift. The difference of behaviors and some challenging DeepPlan scenarios are exemplified in Fig. 3. C. Track 2: State-Distribution Drift While Track 1 evaluates generalization across environ- ments, Track 2 tests resilience against execution-time per- turbations. On the noise-free Val14 set, most planners ap- pear equally capable. However, the introduction of actuation noise reveals a significant divergence in control stability (Table IV). TABLE IV: Noise Robustness Performance. Comparison of planners on nuPlan (Val14) and DeepPlan validation sets with different type and level of noise on dynamic model. Noise levels and OU parameters are defined in Section I-B. Results are reported as mean std over three random seeds. nuPlanDeepPlan AWGNOUAWGNOU MethodVal14LowMidHighLowMidHighBaseLowMidHighLowMidHigh Non-Reactive PDM-Closed92.8490.920.6087.212.7376.463.2988.492.8780.053.3670.453.5448.0746.910.4044.902.7541.371.9545.740.3143.460.3740.931.67 PlanTF84.2783.021.2368.830.9651.972.9379.291.2754.462.3342.690.9934.0031.921.1929.764.5329.230.9731.930.3527.611.8823.071.70 PLUTO92.8890.392.0879.773.5967.872.1987.760.6672.082.7463.392.6536.7435.471.1732.042.0630.663.0132.013.8031.531.8029.772.03 Diffusion planner89.8787.220.6177.525.2863.407.5384.312.4868.037.1452.907.5237.4936.820.3535.671.9832.971.7535.910.8433.261.7930.032.82 CaRL93.8793.310.3092.230.8688.403.0493.190.3092.390.7787.084.8972.5770.690.2770.091.1869.570.9370.440.3369.890.5068.631.00 Reactive PDM-Closed92.1290.670.9788.011.0376.462.3989.961.5781.931.9770.244.8949.9548.920.9848.720.8346.791.6049.051.1448.203.3544.961.38 PlanTF76.9573.971.7764.961.7755.121.5670.060.4555.902.7638.707.0649.4448.390.6446.912.0945.612.1446.381.6044.053.1337.513.28 PLUTO92.0685.962.9582.731.0172.602.8484.031.3374.614.6863.973.5863.0362.381.9162.032.1160.930.6060.262.6659.913.0155.101.98 Diffusion planner82.8081.220.5075.442.0365.593.9479.691.0468.783.0956.524.8949.3949.340.2948.041.1346.611.6648.660.6147.382.2443.153.23 CaRL93.1292.970.3392.071.0388.302.5492.540.3891.141.5685.765.6470.2769.910.5569.020.6067.880.4969.590.4368.180.5667.080.80 Fig. 4: Qualitative analysis of trajectory stability under actuation perturbations. From left to right: (1) Reference trajectories for Log, CaRL, and Diffusion Planner under nominal conditions; (2) CaRL under nominal conditions (solid) vs. high-intensity OU drift (dashed); (3) Diffusion Planner under nominal conditions (solid) vs. high-intensity AWGN jitter (dashed); (4) Diffusion Planner under nominal conditions (solid) vs. high-intensity OU drift (dashed). Results and Analysis. CaRL maintains high resilience: under AWGN and OU noise, it retains all CLSs above 85, which is a minimal drop compared to the other planners. This robustness likely stems from the stochastic nature of RL training: by optimizing under exploration noise, the policy internalizes the vehicle’s dynamic constraints and learns to recover from off-policy states. Consequently, CaRL functions as a dynamics-aware governor that can adjust its control strategy to counteract drift. Despite being also trained to recover from perturbed states through data augmentation, IL methods degrade sharply, and the pattern is noise-type-dependent. Under AWGN, which adds independent perturbations at each timestep, planners can partially self-correct on subsequent steps. However, the CLS performance still significantly decrease by up to 20- 30 points on the high noise settings. Under the temporally correlated OU process errors accumulate as a persistent drift that IL policies, trained to match expert state-action pairs without closed-loop correction, cannot recover from. PlanTF and Diffusion Planner CLS falls by approximately an addi- tional 10 points compared to Gaussian noise, highlighting a failure to recover from multi-frame errors. Fig. 4 shows a characteristic failure: the Diffusion Planner, subjected to OU noise, gradually drifts from the lane center over several seconds before crossing into oncoming traffic, while CaRL remains centered throughout. PDM-Closed maintains its position as a middle-ground performer. Its rule-based lane-following logic allows it to better compensate for deviations, preventing the drift-to- collision failures of IL methods. While safer than pure IL, this intermediate position highlights a fundamental limitation of reactive rule-based control under noisy actuation. The most revealing result emerges from combining the two benchmark tracks. On the DeepPlan set (right half of Table IV), noise compounds the generalization challenge non-additively: planners already struggling with the semantic shift see their remaining performance eroded further by actuation noise. CaRL’s advantage is even more pronounced in this combined-stress regime, suggesting that RL-trained robustness generalizes across both distributional and dynamic perturbations simultaneously. V. CONCLUSIONS In this work, we introduced Shift & Drift, a novel two- track benchmark evaluating AD planners beyond conven- tional i.i.d. assumptions. By integrating high-fidelity aerial datasets into a standard simulation framework and intro- ducing actuation noise, we expose the significant fragility of SOTA motion planners under semantic shifts and state- distribution drifts. Our empirical analysis yields three critical insights: • The Generalization Gap: We quantify the general- ization capabilities of IL-based models. Despite high fidelity in training domains, they exhibit brittleness in novel environments, often failing to negotiate dense pedestrian-cyclist interactions. • The Robustness Paradox: Pure IL paradigms lack reactive closed-loop correction mechanisms. Thus, they fail to recover from temporally correlated execution errors (e.g., OU noise), exposing control instabilities entirely masked by standard noise-free evaluations. • Functional Flexibility vs. Rule-Based Heuristics: Purely rule-based planners fail despite structurally aligning with the evaluation metrics, indicating manual heuristics are overly rigid for complex interactions. The evaluated RL agent maintained higher robustness through learned adaptability in complex environments where manual heuristics become excessively restric- tive. Disentangling the contributions of reward design, closed-loop training, and policy flexibility remains an open challenge for future work. Ultimately, Shift & Drift provides the community with a rigorous framework for stress-testing social intelligence and dynamic stability of object-level planners. Future work will focus on scaling the semantic shift track to include a wider variety of global driving cultures, isolating the individual contributions of topology, interaction density, and regional driving norms to the observed performance degradation, and investigating hybrid architectures that combine the safety guarantees of rule-based systems with the resilient explo- ration of RL. REFERENCES [1] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics, 2019. [2] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paix ̃ ao, F. Mutz, L. de Paula Veronese, T. Oliveira-Santos, and A. F. De Souza, “Self-driving cars: A survey,” Expert Systems with Applications, 2021. [3] Z. Zhu and H. Zhao, “A survey of deep rl and il for autonomous driving policy learning,” IEEE T-ITS, 2022. [4] F. Codevilla, E. Santana, A. Lopez, and A. Gaidon, “Exploring the limitations of behavior cloning for autonomous driving,” in ICCV, 2019. [5] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” NeurIPS, 2019. [6] O. Dhaouadi, J. Meier, L. Wahl, J. Kaiser, L. Scalerandi, N. Wan- delburg, Z. Zhou, N. Berinpanathan, H. Banzhaf, and D. Cremers, “Highly accurate and diverse traffic data: The deepscenario open 3d dataset,” in IEEE IV, 2025. [7] H. Caesar, J. Kabzan, K. Tan, and et al., “Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles,” in CVPR ADP3 Workshop, 2021. [8] R. Xu, H. Lin, W. Jeon, H. Feng, Y. Zou, L. Sun, J. Gorman, E. Tolstaya, S. Tang, B. White, B. Sapp, M. Tan, J.-J. Hwang, and D. Anguelov, “Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios,” arXiv preprint arxiv:2510.26125, 2025. [9] R. Chandra, U. Bhattacharya, A. Bera, and D. Manocha, “Traphic: Tra- jectory prediction in dense and heterogeneous traffic using weighted interactions,” in CVPR, 2019. [10] M. Liu, E. Yurtsever, J. Fossaert, X. Zhou, W. Zimmer, Y. Cui, B. L. Zagar, and A. C. Knoll, “A survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook,” IEEE T-IV, 2024. [11] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in NeurIPS, 1988. [12] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in AISTATS, 2011. [13] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “CARLA: An open urban driving simulator,” in CoRL, 2017, p. 1–16. [14] G. E. Uhlenbeck and L. S. Ornstein, “On the theory of the brownian motion,” Phys. Rev., 1930. [15] D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta, “Parting with misconceptions about learning-based vehicle motion planning,” in CoRL, 2023. [16] J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu, “Rethink- ing imitation-based planner for autonomous driving,” arXiv preprint arXiv:2309.10443, 2023. [17] J. Cheng, Y. Chen, and Q. Chen, “Pluto: Pushing the limit of imita- tion learning-based planning for autonomous driving,” arXiv preprint arXiv:2404.14327, 2024. [18] Y. Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, and J. Liu, “Diffusion-based planning for autonomous driving with flexible guidance,” in ICLR, 2025. [19] B. Jaeger, D. Dauner, J. Beißwenger, S. Gerstenecker, K. Chitta, and A. Geiger, “Carl: Learning scalable planning policies with simple rewards,” in CoRL, 2025. [20] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR, 2012. [21] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays, “Argoverse 2: Next generation datasets for self-driving perception and forecasting,” in NeurIPS Datasets and Benchmarks, 2021. [22] Q. Li, Z. Peng, L. Feng, Z. Liu, C. Duan, W. Mo, and B. Zhou, “Scenarionet: Open-source platform for large-scale traffic scenario simulation and modeling,” NeurIPS, 2023. [23] A. Breuer, J.-A. Term ̈ ohlen, S. Homoceanu, and T. Fingscheidt, “opendd: A large-scale roundabout drone dataset,” in ITSC, 2020. [24] O. Zheng, M. Abdel-Aty, L. Yue, A. Abdelraouf, Z. Wang, and N. Mahmoud, “Citysim: A drone-based vehicle trajectory dataset for safety-oriented research and digital twins,” TRR, 2024. [25] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems,” in ITSC, 2018. [26] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. K ̈ ummerle, H. K ̈ onigshof, C. Stiller, A. de La Fortelle, and M. Tomizuka, “INTERACTION dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps,” CoRR, vol. abs/1910.03088, 2019. [27] C. Selzer and F. Flohr, “Deepurban: Interaction-aware trajectory pre- diction and planning for automated driving by aerial imagery,” in ITSC, 2024. [28] J.Schulman,F.Wolski,P.Dhariwal,A.Radford,and O. Klimov, “Proximal policy optimization algorithms.” CoRR, vol. abs/1707.06347, 2017. [29] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” Physical Review E, 2000. [30] A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,” TRR, 2007. [31] X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan, “Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,” in NeurIPS Datasets and Benchmarks Track, 2024. [32] W. Cao, M. Hallgarten, T. Li, D. Dauner, X. Gu, C. Wang, Y. Miron, M. Aiello, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, A. Geiger, and K. Chitta, “Pseudo-simulation for autonomous driving,” in CoRL, 2025. [33] M. Peng, R. Yao, X. Guo, and J. Ma, “nuplan-r: A closed-loop planning benchmark for autonomous driving via reactive multi-agent simulation,” arXiv preprint arxiv:2511.10403, 2025. [34] S. Hagedorn, L. Donkov, A. Distelzweig, and A. P. Condurache, “When planners meet reality: How learned, reactive traffic agents shift nuplan benchmarks,” arXiv preprint arxiv:2510.14677, 2025. [35] M. Hallgarten, J. Zapata, M. Stoll, K. Renz, and A. Zell, “Can vehicle motion planning generalize to realistic long-tail scenarios?” arXiv preprint arXiv:2404.07569, 2024. [36] C. Xu, W. Ding, W. Lyu, Z. Liu, S. Wang, Y. He, H. Hu, D. Zhao, and B. Li, “Safebench: A benchmarking platform for safety evaluation of autonomous vehicles,” in NeurIPS Datasets and Benchmarks, 2022. [37] J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun, “Advsim: Generating safety-critical scenarios for self- driving vehicles,” in CVPR, 2021. [38] OpenDrive, “Opendrive: Open dynamic road information for vehicle environment,” 2000. [Online]. Available: https://w.opendrive.com