Paper deep dive
ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving
Ziyi Luo, Zhe Sun, Yehao Lu, Lei Zhou, Lisheng Wu, Xuewei Li, Zequn Qin, Xi Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDrive, a counterfactual planning benchmark that uses VLM-assisted screening, localized multi-view editing, and quality auditing to insert hazards into real nuScenes scenes while preserving their context. Its 21 tasks span six safety families and define hazard or conflict regions, local safety constraints, and acceptable responses. Because hazard insertion can invalidate the recorded human trajectory, our reference-free protocol evaluates edited predictions using Unsafe Rate (UR), Hazard Clearance Compliance (HCC), Hazard Proximity Response (HPR), and Counterfactual Trajectory Shift (CTS), which measure core-region intrusion, clearance compliance, clearance relative to a prescribed margin, and counterfactual trajectory change. Seven representative planners frequently intrude into hazard regions or provide insufficient clearance. We also develop a Reminder Agent that, without sample-specific task labels, converts visual evidence and the shared taxonomy into structured records of hazard presence, type, and a recommended high-level strategy. The agent neither predicts trajectories nor controls the vehicle; its records guide a VLM-based decision agent. In zero-shot experiments, the reminders improve strategy accuracy and reduce under-warning.
Tags
Links
- Source: https://arxiv.org/abs/2609.37871v1
- Canonical: https://arxiv.org/abs/2609.37871v1
Trouble viewing inline? Open PDF directly →
Full Text
39,422 characters extracted from source content.
Expand or collapse full text
ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving Ziyi Luo 1 , Zhe Sun 1 , Yehao Lu 1 , Lei Zhou 2 , Lisheng Wu 2 , Xuewei Li 1 , Zequn Qin 1 , and Xi Li ∗1 1 College of Computer Science and Technology, Zhejiang University, Hangzhou, China 2 Yinwang Intelligent Technology Co., Ltd., Shenzhen, China Abstract Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDrive, a counterfactual planning benchmark that uses VLM-assisted screening, localized multi-view editing, and quality auditing to insert hazards into real nuScenes scenes while preserving their context. Its 21 tasks span six safety families and define hazard or conflict regions, local safety constraints, and acceptable responses. Because hazard insertion can invalidate the recorded human trajectory, our reference-free protocol evaluates edited predictions using Unsafe Rate (UR), Hazard Clearance Compliance (HCC), Hazard Proximity Response (HPR), and Counterfactual Trajectory Shift (CTS), which measure core-region intrusion, clearance compliance, clearance relative to a prescribed margin, and counterfactual trajectory change. Seven representative planners frequently intrude into hazard regions or provide insufficient clearance. We also develop a Reminder Agent that, without sample-specific task labels, converts visual evidence and the shared taxonomy into structured records of hazard presence, type, and a recommended high-level strategy. The agent neither predicts trajectories nor controls the vehicle; its records guide a VLM-based decision agent. In zero-shot experiments, the reminders improve strategy accuracy and reduce under-warning. ∗ Corresponding author: xilizju@zju.edu.cn. 1 arXiv:2609.37871v1 [cs.RO] 29 Sep 2026 Edit AgentEdit Agent Image Distribution 6 Safety Corner Case Families 25.56% 19.75% 24.96% 3.60% 9.10% 17.03% Generation from Real Data Controllable Hazard Customization Diverse Risk Scenarios Planning Safety Evaluation Key Advantages Figure 1: Overview of ExceptionDrive. Our framework generates localized counterfactual hazards in real driving scenes, evaluates planner responses without requiring an expert reference trajectory for the edited scene, and produces structured safety-aware reminders. The benchmark includes 21 planning tasks across six planning-safety families. 1 Introduction Autonomous-driving planners have achieved strong performance on standard benchmarks, yet these benchmarks remain dominated by routine, low-risk scenes. Average-case performance therefore does not establish reliability under rare, safety-critical hazards, such as an occluded pedestrian entering the ego path or a collapsed road surface blocking the lane. Existing resources provide limited support for controlled planner evaluation in such cases. Real-world datasets preserve realistic observations but contain few controllable hazards, whereas simulation introduces a domain gap. Corner-case datasets often target perception, anomaly recognition, or semantic reasoning rather than trajectory compliance. A planning benchmark should instead combine realistic scenes, controlled hazard insertion, paired source–edited observations, spatial hazard annotations, and trajectory-level evaluation. Hazard insertion also invalidates conventional trajectory supervision: the recorded human trajectory may intersect the new hazard and no longer provide a valid reference. Moreover, slowing, stopping, yielding, and bypassing may all be acceptable under different constraints. Each task should therefore define local safety constraints and a set of acceptable responses rather than prescribe a unique expert trajectory. We proposed ExceptionDrive, a planning-oriented counterfactual benchmark built by inserting localized hazards into real nuScenes observations. Its construction pipeline combines VLM-assisted scene screening, localized multi-view editing, quality review, and BEV hazard localization while preserving the surrounding context. The benchmark contains 21 tasks in six safety families. Overall, ExceptionDrive comprises 12,440 source–edited pairs: 3,181 quality-approved benchmark pairs and 9,259 auxiliary pairs. Source–edited pairs use identical model settings and non-visual inputs, allowing the effect of the localized edit to be isolated. Our hazard-centric protocol evaluates edited trajectories without expert references. Unsafe Rate (UR) measures core-region intrusion, and Hazard Clearance Compliance (HCC) measures compliance with 2 the required clearance. Hazard Proximity Response (HPR) diagnoses clearance relative to the prescribed margin, while Counterfactual Trajectory Shift (CTS) measures the change between source and edited predictions. HCC and UR are the primary safety metrics; HPR and CTS are complementary diagnostics. The protocol evaluates local hazard compliance rather than complete route validity or closed-loop safety. Experiments on representative planners reveal frequent intrusion and insufficient clearance. We also develop a lightweight Reminder Agent that maps visual evidence and the shared task taxonomy to a structured record containing hazard status, type, relative location, risk level, supporting evidence, the applicable safety constraint, and a recommended high-level strategy. Without receiving sample-specific task labels or target strategies, the reminder supports high-level decision making but does not predict trajectories or control the vehicle. Our main contributions are threefold: 1.We proposed ExceptionDrive, a planning-oriented counterfactual corner-case benchmark created through localized multi-view editing of real driving scenes. It enables controlled insertion of safety- critical hazards while preserving the surrounding driving context as much as possible. 2. We established a planning-oriented task taxonomy and hazard-centric evaluation framework. The framework organizes 21 tasks into six planning-safety families, defines each task through local safety constraints and acceptable risk-mitigation responses, and evaluates edited-scene trajectories without requiring expert reference trajectories. HCC, UR, HPR, and CTS respectively characterize clearance compliance, core-region intrusion, hazard-proximity response, and counterfactual trajectory change. 3.We systematically evaluated representative planners using the proposed benchmark and reveal frequent failures in local hazard compliance. As an auxiliary application, we also introduced a lightweight Reminder Agent and found that structured task-grounded risk records improve high-level strategy accuracy and reduce under-warning in zero-shot VLM experiments. 2 Related Work 2.1 VLM-Assisted Data Generation for Autonomous Driving Autonomous-driving data come from real-world logs, simulation, and generative models. Real-world datasets preserve authentic distributions but are costly and dominated by routine driving Caesar et al. [2020], Wilson et al. [2023], Sun et al. [2020], Geiger et al. [2012]; simulation offers controllability but introduces domain gaps Dosovitskiy et al. [2017], Jia et al. [2024], Gong et al. [2024]; and generative models synthesize scenes conditioned on text, layout, or geometry Gao et al. [2024], Wen et al. [2024], Li et al. [2024], Wang et al. [2024]. VLMs support scene captioning, semantic annotation, prompt generation, and quality assessment Sima et al. [2024], Chen et al. [2025], Wang et al. [2025], Hao et al. [2026]. Rather than synthesizing entire scenes, ExceptionDrive uses VLMs to select task-compatible real scenes, generate location-aware editing instructions, and review edited samples. This localized approach retains the source scene as a within-scene control while seeking to isolate the intended hazard. 2.2 Corner-Case Taxonomies and Dataset Construction Corner cases span multiple levels, from sensor or appearance shifts and unusual objects to rare scene configurations and safety-critical interactions Breitenstein et al. [2020], Heidecker et al. [2021], Rösch et al. [2022]. Existing benchmarks can be broadly grouped by their evaluation targets. Perception-oriented datasets assess road-anomaly segmentation or unknown-object detection Pinggera et al. [2016], Blum et al. [2021], 3 Chan et al. [2021], Li et al. [2022a], Gong et al. [2024]; event- and reasoning-oriented datasets collect accidents or annotate rare scenes with semantic descriptions and driving suggestions Fang et al. [2019], Chen et al. [2025]; and system-level benchmarks evaluate driving policies in interactive simulation Dosovitskiy et al. [2017], Jaeger et al. [2023], Xu et al. [2022], Jia et al. [2024]. Real-world datasets preserve visual realism but provide limited control over rare hazards, whereas simulation offers controllability at the cost of a domain gap. ExceptionDrive complements these settings with planner-level counterfactual testing on paired real scenes. 3 Method Edit Assistant Positive Prompt Should Do Nagative Prompt Should Not Do Driving Editor Review Assistant "Qualification Status": Pass "Task Correct": True "Local Physics Valid": True "Background Preserved": True "Task Confidence": 0.78 "Realism Score": 3.0 "Action Determinacy Score": 3.0 "Short Reason": The edited hazard is visible and the driving response is understandable. Skill: Acceptance_Standard.md Skill: Task_Description.md Task: Large Vehicle Blocking Lane Output: Qwen-Image-Edit Positive PromptNagative Prompt Positive Prompt: Keep the result photorealistic. Preserve the original camera viewpoint, lens perspective, road geometry, lane markings, lighting, weather, shadows, traffic context, buildings, vegetation, road edges, and all unrelated objects. Required edit location: about 5-8 meters ahead of the ego vehicle, in the ego lane. Scene-specific rewrite: Create a realistic autonomous-driving corner case where a stopped large vehicle blocks the ego lane at a medium-to-near distance ahead. Nagative Prompt: Do not change the camera viewpoint, road layout, lane topology, weather, time of day, background buildings, unrelated vehicles, unrelated pedestrians, or traffic signs unless explicitly required. Do not overpaint the whole image. Do not create a cartoon, illustration, CGI look, blur, low-resolution artifacts, distorted lane lines, warped vehicles, floating objects, duplicated objects, unrealistic scale, unnatural shadows, or inconsistent lighting. Figure 2: Counterfactual corner-case construction pipeline. An edit assistant uses the task definition and real-scene context to produce task-aware positive and negative prompts. A localized driving-scene editor inserts the target hazard while preserving viewpoint, road geometry, lighting, traffic context, and unrelated background. A review assistant then audits task correctness, physical plausibility, background preservation, realism, and action determinacy. Accepted source–edit pairs provide hazard-localized, planning-oriented samples for evaluation. 3.1 Task Taxonomy An object- or appearance-based corner-case taxonomy is insufficient for planner evaluation because risk depends on the hazard’s location, its conflict with the ego vehicle, and the feasible responses. For example, the same vehicle may require stopping when it blocks the ego lane, yielding when it occupies an intersection conflict region, or performing a constrained bypass when sufficient adjacent space is available. Conversely, visually different hazards may impose the same planning constraint, such as avoiding a non-drivable region. Constructing planning-oriented counterfactual cases therefore requires an operational task specification that jointly defines the applicable driving context, the localized hazard intervention, the spatial safety constraint, and the range of acceptable risk-mitigation responses. We define 21 tasks and organize them into six planning-safety families: Sudden Emergence, Lane 4 Blockage, Bypass and Gap Conflict, Signal and Rule Conflict, Intersection Conflict, and Road-Surface Hazard. For each task, we specify scene-suitability conditions, rejection rules, edit content, target-location constraints, the corresponding hazard or conflict region, and acceptable risk-mitigation responses. These responses do not prescribe a single mandatory maneuver. Instead, they describe task-level options such as avoiding a localized risk region, stopping before a constraint boundary, yielding to conflicting traffic, or bypassing only when sufficient space is available. The resulting specifications provide a common basis for scene screening, localized editing, quality review, hazard-region construction, geometric planner evaluation, and high-level strategy evaluation. 3.2 Dataset Construction ExceptionDrive uses task-conditioned local counterfactual editing to introduce planning-relevant hazards while preserving the surrounding real-world context. The task specifications in Section 3.1 guide four stages: scene screening, localized multi-view editing, quality review, and hazard-region localization. For each scene–task combination, a VLM assesses whether the synchronized nuScenes views and scene description satisfy the task-specific suitability conditions. It examines the road layout, available space, occlusion pattern, and potential insertion location. For suitable combinations, the VLM selects the relevant views, specifies an ego-relative target location, and generates view-specific editing instructions. A combination is rejected if the hazard would violate task constraints, create an ambiguous planning conflict, or require substantial changes to the original scene. An instruction-driven editor then combines global preservation requirements with the task-level hazard description and scene-specific location instructions. It modifies only the target region while preserving the viewpoint, road geometry, lane markings, illumination, weather, unrelated traffic participants, and background. For hazards visible in multiple synchronized views, view-specific instructions promote cross-view coherence, which is subsequently assessed during VLM review; no separate geometric multi-view consistency module is used. The resulting source–edited pair is intended to differ primarily in the inserted hazard. Each edit undergoes task-aware quality review. A VLM reviewer assesses hazard correctness, physical plausibility, visibility, cross-view consistency, source-scene preservation, and clarity of the planning conflict. Low-scoring edits and a random subset of accepted edits are also inspected manually. Only pairs satisfying both visual-quality and task-validity requirements are retained. Finally, we construct a hazard region퐻 푖 in ego coordinates for each accepted sample. BEVFormer-based localization Li et al. [2022b] provides four ground-plane corners defining퐻 푖 as a closed BEV quadrilateral. For physical hazards,퐻 푖 represents the occupied or non-drivable region; for signal, right-of-way, and intersection tasks, it represents the task-defined conflict or no-go region. Each region is verified against the intended edit location, multi-view evidence, camera calibration, and task definition, and serves as the spatial reference for hazard-centric evaluation. 3.3 Paired Counterfactual Planner Evaluation After a safety-critical hazard is inserted into a real driving scene, the original human trajectory may pass through the newly introduced hazard and therefore cannot serve as a valid reference trajectory for the edited scene. We therefore evaluate whether the predicted trajectory satisfies the task-defined local safety constraints, rather than using an unique trajectory label. Let the corresponding source and edited trajectories,휏 표 푖 and휏 푒 푖 , for scene 푖 be: 휏 표 푖 = (푥 표 푖,푡 , 푦 표 푖,푡 ) 푇 푡=0 ,휏 푒 푖 = (푥 푒 푖,푡 , 푦 푒 푖,푡 ) 푇 푡=0 .(1) The source prediction휏 표 푖 is used only as a counterfactual reference for diagnosing behavioral changes, and the safety of 휏 푒 푖 is evaluated by its geometric relationship with the task-defined hazard region 퐻 푖 . 5 (a) Hazard Clearance Compliance (HCC) (b) Hazard Proximity Response (HPR) Hᵢ dᵢ d_safe primary metric Clearance Regimes (if no intrusion, urᵢ = 0) Intrusion ur i = 1 Insufficient clearance 0 ≤ dᵢ < d_safe Sufficient clearance dᵢ ≥ d_safe HCC = 0 dᵢ 0 < HCC < 1 dᵢ HCC = 1 (saturates) Too Close dᵢ < d_safe Ideal (at clearance) dᵢ = d_safe Too far dᵢ > d_safe dᵢ Lower HPR dᵢ Highest HPR dᵢ Lower HPR HPR curve 1 0 d_safe dᵢ Hᵢ Hᵢ Hᵢ HᵢHᵢ Hᵢ minimum sufficient response Figure 3: Illustration of the core hazard-centric evaluation metrics used in ExceptionDrive. The blue collision tube is evaluated relative to the task-defined hazard퐻 푖 and its required clearance boundary푑 safe . (a) Hazard Clearance Compliance (HCC) quantifies whether the trajectory maintains the prescribed clearance margin. (b) Hazard Proximity Response (HPR) characterizes whether the avoidance response is close to the minimum sufficient clearance. 6 Counterfactual Trajectory Shift (CTS) measures how strongly the planner changes its predicted trajectory after the hazard is introduced: 퐶푇푆= 1 푁 푁 ∑︁ 푖=1 1 푇 + 1 푇 ∑︁ 푡=0 √︂ 푥 푒 푖,푡 − 푥 표 푖,푡 2 + 푦 푒 푖,푡 − 푦 표 푖,푡 2 .(2) and its longitudinal and lateral components are given in the supplementary material. CTS diagnoses the planner’s sensitivity to the counterfactual hazard, but does not by itself determine whether the resulting response is safe or directionally appropriate. Unsafe Rate (UR) measures direct intrusion into the task-defined core hazard region. We construct an ego collision tubecoll 푖 from the edited trajectory using oriented vehicle footprints, with details given in the supplementary material. Let퐻 푖 denote the point set of the hazard region defined for scene푖. We define the unsafe-region intrusion indicator as: 푢푟 푖 = I [ coll 푖 ∩ 퐻 푖 ≠∅ ] ,(3) whereI[·]is the indicator function, which equals one when the enclosed condition holds and zero otherwise. The dataset-level Unsafe Rate is: 푈푅= 1 푁 푁 ∑︁ 푖=1 푢푟 푖 .(4) A lower UR indicates that the planner enters the localized hazard or no-go region in fewer edited cases. Avoiding direct intrusion is necessary but insufficient, because a safe trajectory may still pass unacceptably close to the hazard. Therefore, Hazard Clearance Compliance (HCC) is used as our primary metric to measure geometric safety-compliance . Let푑 푖 = 푑(coll 푖 , 퐻 푖 ), 푑 푖 ≥ 0denote the minimum Euclidean distance between the ego collision tube and the hazard region. Given the required safety clearance푑 safe , HCC is defined as: 퐻퐶= 1 푁 푁 ∑︁ 푖=1 (1− 푢푟 푖 )· min 푑 푖 푑 safe , 1 .(5) HCC is bounded in[0, 1]. It is zero when the trajectory intrudes into the hazard region, increases with clearance for non-overlapping trajectories, and saturates once the required safety clearance is reached. Hazard Proximity Response (HPR): although a trajectory should maintain sufficient clearance, its distance from the hazard also provides information about the magnitude of the avoidance response. We define HPR as: 퐻푃푅= 1 푁 푁 ∑︁ 푖=1 (1− 푢푟 푖 )· 푑 safe | 푑 푖 − 푑 safe | + 푑 safe .(6) HPR is zero when the trajectory enters the hazard region. Among safety-compliant trajectories, it reaches one when the minimum clearance equals푑 safe and decreases as the trajectory deviates farther from this clearance. Thus, HPR is used to diagnose whether safe avoidance is close to the minimum sufficient response. In particular, a high HPR does not by itself imply safety compliance, because trajectories with푑 푖 < 푑 safe may still obtain a positive HPR value; their insufficient clearance is explicitly captured by HCC. Illustrated in Fig. 3, CTS, UR, HCC, and HPR together characterize counterfactual sensitivity, direct hazard-region violation, spatial-clearance compliance, and clearance proportionality, respectively. Safety comparisons primarily rely on high HCC and low UR, whereas CTS and HPR provide complementary diagnostic information and should not be interpreted independently as monotonic safety scores. 7 3.4 Task-Grounded Risk Reminder The planner evaluation described above diagnoses whether a predicted trajectory responds safely to a localized counterfactual hazard, but it does not directly provide an interpretable explanation of the detected risk. We therefore explore a secondary use of the ExceptionDrive taxonomy by constructing a lightweight task-grounded risk Reminder Agent for VLM-based driving agents. The Reminder Agent neither predicts trajectories nor controls the vehicle. Instead, it converts task-relevant visual evidence into a structured risk record that supports high-level driving strategy selection. Given a driving scene, the reminder first identifies potential hazard evidence and associates it with the applicable risk definitions in the task taxonomy. It then produces a structured record containing hazard presence, hazard type, and a recommended high-level strategy. The reminder has access to the task taxonomy as a collection of risk definitions, but is not provided with the ground-truth task identity or its target strategy for the current sample. The final driving strategy is predicted by a VLM-based driving agent from a predefined strategy space, including actions such as maintaining the current path, slowing down, stopping, yielding, or preparing to bypass. 4 Experiments 1. Sudden Emergence (a). Original(b). Edited 2. Lane Blockage (a). Original(b). Edited 3. Bypass and Gap Conflict (a). Original(b). Edited 4.Intersection Conflict (a). Original(b). Edited 5. Signal and Rule Conflict (a). Original(b). Edited 6. Road-Surface Hazard (a). Original(b). Edited Figure 4: Representative samples from ExceptionDrive across six safety families. Each example pairs an original image with its counterfactual edit. Red dashed boxes mark the inserted local hazards. The edits preserve the original viewpoint, road layout, illumination, weather, and surrounding traffic context while modifying only the task-relevant risk factor. 4.1 Dataset Statistics and Quality Analysis Starting from 850 nuScenes scenes and 21 predefined tasks, we obtain 17,850 scene–task combinations. Scene applicability filtering identifies 3,446 suitable pairs, accounting for 19.31% of all combinations. After localized editing and VLM-based quality review, 3,181 samples are retained, corresponding to an acceptance rate of 92.31%, and form the final benchmark. To assess the reliability of the automated review, we randomly inspect 168 VLM-approved samples, covering 5.28% of the benchmark. Among them, 163 pass manual inspection, yielding a 97.02% acceptance rate and indicating a high manual confirmation rate among the audited VLM-approved samples. 8 ModelHCC%↑ UR%↓ HPR%↑ CTS CTS 푥 CTS 푦 Impromptu VLA Chi et al. [2026]19.64 78.9218.87 2.20 1.86 0.70 OmniDrive Wang et al. [2025]9.52 83.369.84 1.40 1.34 0.22 AutoVLA Zhou et al. [2026]10.37 81.9311.26 1.97 1.67 0.64 DrivoR Kirby et al. [2026]4.86 92.135.42 1.25 1.19 0.22 LightEMMA Qiao et al. [2025]18.66 79.5118.04 1.68 1.61 0.23 ST-P3 Hu et al. [2022]21.03 78.1118.59 1.57 1.55 0.24 VAD Jiang et al. [2023]13.89 80.2613.47 1.68 1.47 0.51 Table 1: Main planner evaluation results on the proposed corner-case benchmark. HCC is the primary metric and measures whether the planner produces the task-defined safe response in edited hazardous scenes. UR measures unsafe response into the annotated hazard region. HPR measures the efficiency of hazard avoidance. CTS reports the magnitude of counterfactual trajectory shift between source and edited inputs, with longitudinal and lateral components denoted by CTS 푥 and CTS 푦 . PlannerEmergence Blockage Bypass Signal Inter. Surface autoVLA9.9012.8610.22 9.78 9.929.56 Impromptu VLA19.6320.2819.98 20.14 17.86 19.95 DrivoR4.314.592.615.49 3.948.27 ST-P320.0219.8617.48 22.92 24.75 21.17 Table 2: Family-level HCC. The table reports planner response correctness across the six safety families, revealing which types of corner cases are most challenging for each model. Higher HCC indicates a larger fraction of edited hazardous scenes in which the planner produces the task-defined safe response. Fig. 1 shows the distribution across six safety families. Sudden Emergence, Lane Blockage, Bypass and Gap Conflict, Signal and Rule Conflict, Intersection Conflict, and Road-Surface Hazard account for 25.56%, 19.75%, 24.96%, 3.60%, 9.10%, and 17.03%, respectively. Signal and Rule Conflict is the least frequent because it requires specific intersection layouts, traffic-light states, and right-of-way contexts. To support failure analysis, qualitative studies, and future expansion, we construct an auxiliary set in addition to the benchmark evaluation set. Following the task distribution of the final benchmark, we select 8,994 additional scene–task pairs from outside the 3,446 screened benchmark candidates and process them directly with the editing model. These pairs are combined with the 265 edits rejected during benchmark quality review, yielding an auxiliary set of 9,259 pairs. The complete ExceptionDrive collection therefore contains 12,440 source–edited pairs: 3,181 quality-approved pairs in the benchmark evaluation set and 9,259 pairs in the auxiliary set. The auxiliary set is not used for the reported benchmark evaluation. Fig. 4 shows representative original–edited pairs. The approved samples introduce localized task-relevant hazards while largely preserving irrelevant scene content, enabling controlled counterfactual evaluation. 4.2 Planner Evaluation on Corner Cases We evaluate seven representative planners on the final benchmark: Impromptu VLA, OmniDrive, AutoVLA, DrivoR, LightEMMA, ST-P3, and VAD. For each original–edited pair, the navigation command, historical information, and inference configuration remain unchanged. Model-specific input modalities and adaptations to the edited views are described in the supplementary material. Table 1 shows limited local hazard compliance across all planners: the highest HCC is 21.03%, and every UR exceeds 78%. ST-P3 performs best, with the highest HCC (21.03%) and lowest UR (78.11%), whereas DrivoR obtains the lowest HCC (4.86%) and highest UR (92.13%). Impromptu VLA produces the largest CTS (2.20) without achieving the best safety performance, confirming that trajectory change indicates sensitivity to the edit but not necessarily a safe response. We therefore use HCC and UR as the primary safety metrics and treat HPR and CTS as complementary diagnostics. Table 2 shows that ST-P3 achieves the highest HCC in most families, while DrivoR reaches only 2.61% on Bypass and Gap Conflict. For Signal and Rule Conflict and Intersection Conflict, HCC measures spatial 9 compliance with task-defined regions rather than complete traffic-rule understanding. 4.3 Qualitative Analysis Hazard Area 1. Sudden Emergence (a). Original(b). Edited 2. Lane Blockage (a). Original(b). Edited 3. Bypass and Gap Conflict (a). Original(b). Edited 4.Intersection Conflict (a). Original(b). Edited 5. Signal and Rule Conflict (a). Original(b). Edited 6. Road-Surface Hazard (a). Original(b). Edited Planner：AutoVLAPlanner：InpromptuPlanner：DrivoR Figure 5: Qualitative planner responses across the six ExceptionDrive safety families. For each family, (a) shows the source scene and (b) the edited counterfactual scene. Red dashed boxes denote the inserted hazards, while blue, orange, and green curves show trajectories from AutoVLA, Impromptu VLA, and DrivoR, respectively. Fig. 5 presents representative planning cases and illustrates three common failure patterns. First, some models follow trajectories close to those predicted in the original scene and directly enter the risk region, resulting in increasing UR and decreaseing HCC. Second, some models react to the edit but shift in an inappropriate direction or by an insufficient amount, producing a large CTS without achieving safe avoidance. Third, some trajectories avoid direct intrusion but remain too close to the risk boundary, leading to a relatively lower UR but limited HCC. These cases show that intrusion, insufficient clearance, and counterfactual trajectory change are related but non-equivalent phenomena. The qualitative results therefore complement, rather than replace, the quantitative metrics. 4.4 Reminder Agent Evaluation We further evaluate task-grounded reminders as a downstream application of ExceptionDrive, using Qwen2.5- VL-7B Bai et al. [2023] as the final driving agent. The baseline and reminder-augmented settings use identical visual inputs, candidate strategies, and base instructions. The baseline predicts directly from the scene, whereas the augmented setting additionally receives the structured reminder defined in Section 3.4, generated by an auxiliary VLM from the same visual input and shared task taxonomy. Neither setting receives human-annotated risk regions, sample-specific safe actions, or quality-review labels. To avoid ambiguity in free-form evaluation, Qwen2.5-VL-7B selects one of five predefined strategies, 10 denoted A–E. A prediction is correct if it belongs to the task-defined acceptable response set. We report strategy accuracy and under warning rate, where the latter measures the fraction of hazardous cases in which the selected action is insufficiently cautious. MethodAccuracy↑ Under-warning Rate↓ Qwen2.5VL-7B51.8645.53 Qwen2.5VL-7B + Reminder 70.7129.18 Table 3: High-level strategy evaluation for the zero-shot VLM driving agent. Table 3 shows that the reminder improves strategy accuracy from 51.86% to 70.71% and reduces the miss rate from 45.53% to 29.18%. These results demonstrate the value of ExceptionDrive for both planner diagnosis and structured risk communication to VLM-based driving agents. 5 Conclusion In this paper, we proposed ExceptionDrive, a planning oriented counterfactual benchmark for evaluating autonomous-driving planners under rare and safety-critical local hazards. ExceptionDrive constructs 21 tasks across six planning-safety families by inserting localized hazards into compatible real-world driving observations while preserving the original scene context. To address the absence of valid expert trajectories after hazard insertion, we proposed a hazard-centric evaluation protocol that assesses hazard intrusion, clearance compliance, avoidance response, and counterfactual behavioral changes. In addition, we developed a lightweight Reminder Agent to transform visual hazard evidence into structured risk records for high-level decision making. Experiments with representative planners demonstrate that existing systems frequently fail to maintain sufficient safety clearance, while the Reminder Agent improves zero-shot strategy prediction and reduces under-warning. These results highlight the value of ExceptionDrive for systematically analyzing planner behavior under localized visual interventions that are critical to safety. References Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation. International Journal of Computer Vision, 129(11):3119–3135, 2021. Jasmin Breitenstein, Jan-Aike Termöhlen, Daniel Lipinski, and Tim Fingscheidt. Systematization of corner cases for visual perception in automated driving. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1257–1264. IEEE, 2020. Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann. Segmentmeifyoucan: A benchmark for anomaly segmentation. arXiv preprint arXiv:2104.14812, 2021. 11 Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7817–7826. IEEE, 2025. Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, et al. Impromptu vla: Open weights and open data for driving vision-language-action models. Advances in Neural Information Processing Systems, 38, 2026. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017. Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, He Wang, and Sen Li. Dada-2000: Can driving accident be predicted by driver attentionƒanalyzed by a benchmark. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 4303–4309. IEEE, 2019. Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. In International Conference on Learning Representations, volume 2024, pages 22841–22860, 2024. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012. Lei Gong, Yu Zhang, Yingqing Xia, Yanyong Zhang, and Jianmin Ji. Sdac: a multimodal synthetic dataset for anomaly and corner case detection in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1914–1922, 2024. Ruiyang Hao, Bowen Jing, Haibao Yu, and Zaiqing Nie. Styledrive: Towards driving-style aware benchmarking of end-to-end autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 4627–4635, 2026. Florian Heidecker, Jasmin Breitenstein, Kevin Rösch, Jonas Löhdefink, Maarten Bieshaar, Christoph Stiller, Tim Fingscheidt, and Bernhard Sick. An application-driven conceptualization of corner cases for perception in highly automated driving. arXiv preprint arXiv:2103.03678, 2021. Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pages 533–549. Springer, 2022. Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8240–8249, 2023. Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. Advances in Neural Information Processing Systems, 37:819–844, 2024. Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8350, 2023. 12 Ellington Kirby, Alexandre Boulch, Yihong Xu, Yuan Yin, Gilles Puy, Éloi Zablocki, Andrei Bursuc, Spyros Gidaris, Renaud Marlet, Florent Bartoccioni, et al. Driving on registers. arXiv preprint arXiv:2601.05083, 2026. Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In European Conference on Computer Vision, pages 406–423. Springer, 2022a. Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scenarios video generation with latent diffusion model. In European Conference on Computer Vision, pages 469–485. Springer, 2024. Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.17270, 2022b. Peter Pinggera, Sebastian Ramos, Stefan Gehrig, Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1099–1106. IEEE, 2016. Zhijie Qiao, Haowei Li, Zhong Cao, and Henry X Liu. Lightemma: Lightweight end-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2505.00284, 2025. Kevin Rösch, Florian Heidecker, Julian Truetsch, Kamil Kowol, Clemens Schicktanz, Maarten Bieshaare, Bernhard Sick, and Christoph Stiller. Space, time, and interaction: A taxonomy of corner cases in trajectory datasets for automated driving. In 2022 IEEE Symposium Series on Computational Intelligence (SSCI), pages 86–93. IEEE, 2022. Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European conference on computer vision, pages 256–274. Springer, 2024. Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020. Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In Proceedings of the computer vision and pattern recognition conference, pages 22442–22452, 2025. Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. In European conference on computer vision, pages 55–72. Springer, 2024. Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6902–6912, 2024. 13 Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493, 2023. Chejian Xu, Wenhao Ding, Weijie Lyu, Zuxin Liu, Shuai Wang, Yihan He, Hanjiang Hu, Ding Zhao, and Bo Li. Safebench: A benchmarking platform for safety evaluation of autonomous vehicles. Advances in Neural Information Processing Systems, 35:25667–25682, 2022. Zewei Zhou, Tianhui Cai, Seth Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Autovla: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. Advances in Neural Information Processing Systems, 38:27920–27956, 2026. 14