Paper deep dive
Beyond the Leaderboard: Counterfactual Diagnosis of End-to-End and VLA Driving Policies Under Domain Shift
Ruolin Yang, Zilin Huang, Buoyue Wang, Zhengyang Wan, Yuhao Luo, Zihao Sheng, Sikai Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/22/2026, 4:55:17 AM
Summary
The paper introduces 'Check-Up', a counterfactual diagnosis framework for evaluating end-to-end and Vision-Language-Action (VLA) driving policies under domain shift. It argues that leaderboard rankings (e.g., nuScenes, NAVSIM) fail to predict policy behavior in new domains, particularly regarding pedestrian interactions. The framework uses pixel-level interventions (pedestrian removal and night-style lighting perturbation) to compute two causal axes: faithfulness displacement (response to pedestrian) and interference displacement (response to irrelevant changes). Five exams assess driving distance, safety benefit, hazard scaling, plan stability, and lighting sensitivity. Results on six policies show that genuine pedestrian avoidance is rare (1.9% of critical scenes), and leaderboard ranks do not correlate with safety behaviors in new domains.
Entities (16)
Relation Signals (16)
Check-Up → computes → Interference Displacement
confidence 95% · and the interference displacement II, the change the same policy produces under an edit that carries no driving-relevant information
Check-Up → computes → Faithfulness Displacement
confidence 95% · From these edits two causal axes are read... faithfulness displacement FF
Check-Up → uses → Pedestrian Removal
confidence 95% · We propose a counterfactual check-up: a few hundred real frames, each edited two ways (pedestrian removed...)
Check-Up → uses → Night-style Perturbation
confidence 95% · each edited two ways (pedestrian removed, or re-lit by a night-style perturbation)
Pedestrian Removal → measures → Faithfulness Displacement
confidence 93% · FF, the plan displacement caused by removing the target pedestrian
Night-style Perturbation → measures → Interference Displacement
confidence 93% · II, that caused by the night-style perturbation
Leaderboard Rank → failstopredict → New Domain Behavior
confidence 92% · rank on nuScenes open-loop error or on NAVSIM's leaderboard does not carry over to scenes with a pedestrian near the ego corridor at a new site.
Check-Up → assesses → Causal Faithfulness Ratio
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:End-to-end and vision-language-action (VLA) driving policies are compared by leaderboard rank, but a rank reports an outcome, not the behaviour behind it, so it predicts poorly how a policy will behave at a new site. On six released policies, rank on nuScenes open-loop error or on NAVSIM's leaderboard does not carry over to scenes with a pedestrian near the ego corridor at a new site. We propose a counterfactual check-up: a few hundred real frames, each edited two ways (pedestrian removed, or re-lit by a night-style perturbation), every edit verified by an independent detector, and the change in the planned trajectory read as a diagnosis rather than a score. From these edits two causal axes are read, and five exams built on them separate what a score merges: how far the policy plans to drive, whether seeing the pedestrian buys safety, whether that response scales with danger, whether the plan moves when nothing requires it, and how much an irrelevant lighting change moves it. On 246 NAVSIM near-pedestrian scenes, in the cells where the pedestrian lies on the planned path only 1.9% of responses are genuine avoidance, and under our open-loop protocol the median clearance change is at most 0.03 m and the median change in planned distance at most 0.08 m for every policy. In a pre-registered test from left- to right-hand drive, the exposure and specificity orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not. Read as a selection report, the profiles say which policy is safe because it plans short, which covers a human-like distance without yielding, and which is unsteady under a change that requires no reaction, and they price each verdict: most settle within a few dozen frames, hazard sensitivity needs hundreds. Code and edited frames will be released.
Tags
Links
- Source: https://arxiv.org/abs/2609.22582v1
- Canonical: https://arxiv.org/abs/2609.22582v1
Trouble viewing inline? Open PDF directly →
Full Text
50,156 characters extracted from source content.
Expand or collapse full text
Beyond the Leaderboard: Counterfactual Diagnosis of End-to-End and VLA Driving Policies Under Domain Shift Ruolin Yang Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Zilin Huang Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Buoyue Wang Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Zhengyang Wan Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Yuhao Luo Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Zihao Sheng Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Sikai Chen †thanks: This research was supported by the U.S. Department of Transportation (No. 69A3552348305) and by grants from NVIDIA utilizing NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition GPUs.†thanks: *Corresponding author: Sikai Chen. Affiliation: Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison, WI 53706, USA. Abstract End-to-end and vision-language-action (VLA) driving policies are compared by leaderboard rank, but a rank reports an outcome, not the behaviour behind it, so it predicts poorly how a policy will behave at a new site. On six released policies, rank on nuScenes open-loop error or on NAVSIM’s leaderboard does not carry over to scenes with a pedestrian near the ego corridor at a new site. We propose a counterfactual check-up: a few hundred real frames, each edited two ways (pedestrian removed, or re-lit by a night-style perturbation), every edit verified by an independent detector, and the change in the planned trajectory read as a diagnosis rather than a score. From these edits two causal axes are read, and five exams built on them separate what a score merges: how far the policy plans to drive, whether seeing the pedestrian buys safety, whether that response scales with danger, whether the plan moves when nothing requires it, and how much an irrelevant lighting change moves it. On 246 NAVSIM near-pedestrian scenes, in the cells where the pedestrian lies on the planned path only 1.9% of responses are genuine avoidance, and under our open-loop protocol the median clearance change is at most 0.03 m and the median change in planned distance at most 0.08 m for every policy. In a pre-registered test from left- to right-hand drive, the exposure and specificity orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not. Read as a selection report, the profiles say which policy is safe because it plans short, which covers a human-like distance without yielding, and which is unsteady under a change that requires no reaction, and they price each verdict: most settle within a few dozen frames, hazard sensitivity needs hundreds. Code and edited frames will be released. I Introduction A leaderboard rank does not tell you how a driving policy will behave in a new domain. End-to-end and vision-language-action (VLA) driving policies are compared by benchmark scores, and a deployment decision is, implicitly, a bet that the rank those scores produce will still hold once the policy is driving at the new site, not just on someone else’s roads. On scenes where a pedestrian is near the ego corridor, that bet fails for the six released policies we test: their rank on nuScenes’ [1] open-loop error or on NAVSIM’s [2] leaderboard score does not carry over to the deployment site (rank correlation |ρ|≤0.43|ρ|≤ 0.43 against near-pedestrian behaviour there, Section IV-E), a gap that open-loop scores are known to open [3, 4, 5]. The reason is not noise but mechanism. A score is an outcome measure, and an outcome can be right for the wrong reason: a policy that never approaches a pedestrian because it plans short trajectories scores as well as one that sees the pedestrian and yields, and the two behave very differently the first time the scene changes. Telling them apart means asking why the score came out as it did, usually from little data, because a closed-loop simulator for the target domain or a fleet test is exactly what the decision is meant to save. Deployment safety cases still rest on target-domain miles [6], and post-training on a policy’s own deployments [7] presumes the same access. For a selection decision, that cost is prohibitive [8]. Cross-domain work in domain adaptation [9] and generalisation [10] solves the shift through training and does not assess how a policy behaves in the target domain, so it leaves the selection problem at deployment open. Closer to our setting, [11] improves a VLA’s out-of-distribution generalisation by aligning its visual representations, but it presumes an already-chosen vision–language model and offers no procedure for selecting among policies, vision-only ones included. Representation engineering [12] locates and steers concepts in a language model’s activations, but it is built around text prompts. Function vectors [13] show that a task can be read out of, and transferred through, a language model’s hidden states. We know of no counterpart of these instruments for the cross-domain evaluation of driving policies. Fig. 1: The Check-Up framework. Pixel-level paired data (a) yield the two causal axes (b) and the five exams built on them (c). Diagnosed on one driving side and tested on the other, they give a priced prognosis, the check-up report (d). To address this, we propose the Check-Up, a counterfactual diagnosis in four stages (Fig. 1). (a) We construct pixel-level paired data: every logged frame is queried with and without the target pedestrian and under a night-style photometric perturbation, and an independent detector verifies every edit. (b) From each pair we distil two axes of causal attribution—the faithfulness displacement F, the plan change caused by the pedestrian itself, and the interference displacement I, the change the same policy produces under an edit that carries no driving-relevant information (3). (c) Five exams, each with a control and a reference threshold, are built on these paired plans and axes (Fig. 1c): how far the policy plans to drive, whether seeing the pedestrian buys safety, whether that benefit scales with the hazard, whether the plan moves when nothing requires it, and how much the irrelevant edit moves it. (d) Read on one driving side and tested on the other, the exams yield a prognosis for the target domain: which verdicts and orderings carry over, what each verdict costs in frames, and a per-policy selection report. The contribution is the Check-Up, a counterfactual diagnosis of end-to-end and VLA driving policies under domain shift, in three parts. 1. Pixel-level paired data. Every logged frame is queried with and without the target pedestrian and under a night-style perturbation, each edit verified by an independent detector. Six released policies are read on two driving sides. 2. Two causal axes. A response is read as the faithfulness displacement F and the interference displacement I (3), and counts as avoidance only if it moves and buys distance, and exceeds the interference where a re-lit plan exists. Where the pedestrian lies on the planned path, 1.9% of responses qualify, and the lighting edit moves every plan further than the pedestrian does. 3. A priced prognosis. Diagnosed on one driving side and tested on the other, the exposure and specificity orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not. Each verdict is priced in frames, lighting within 14, exposure within 51, and the profiles read as a selection report that says what each policy needs before deployment (Section IV-E). I Related Work Driving policies and their benchmarks We examine bird’s-eye-view (BEV) planners of the TransFuser lineage [14] (LTF, DiffusionDrive [15], DiffusionDriveV2 [16]) and vision–language–action models (SimLingo [17], AutoVLA [18], Alpamayo-1.5 [19, 20]). NAVSIM [2] scores plans by multiplying hard gates with a weighted average of soft terms, open-loop scores are known to reward proxies [4], end-to-end policies to exploit biases in their inputs [21], and closed-loop benchmarks [22] score a policy in simulation. NAVSIM’s pseudo-simulation [23] moves the score toward closed loop. We ask a different question, whether the pedestrian entered the plan, and read released policies on real frames. Behavioural tests and causal confusion Checklists [24] and corruption benchmarks [25] probe models beyond aggregate accuracy, but a metric drop does not say whether a policy became more cautious or less. Imitation learners respond to correlates of the expert’s action rather than to its causes [26, 5, 27]. Our exams intervene on a cause (a pedestrian) and a nuisance (lighting) directly and read signed plan changes. Counterfactual editing Counterfactual explanations edit the scene [28], and CausalAgents [29] deletes non-causal agents to test motion forecasters. Closed-loop simulators insert or re-render agents in reconstructed scenes [30, 31, 32]. Our removal edit applies the same idea at pixel level to real frames, every edit is verified by a detector, and six released policies are compared on the same edits. I The Check-Up: Interventions, Axes and Exams I-A Units, interventions and plans Write a policy as a map π from a sensory-and-state input to a planned trajectory X=(Xt)t∈[0,2.5]X=(X_t)_t∈[0,2.5], Xt∈ℝ2X_t ^2 in the ego frame, sampled at 0.10.1 s. A unit is a pair u=(f,v)u=(f,v): a logged frame f and an input ego speed v. We take v from the log for every frame and, wherever the pedestrian’s logged future is available, additionally v∈4,8m/sv∈\4,8\\,m/s, which places the pedestrian on the planned path in scenes where the logged ego was stationary. The same grid is used on both corpora so that all six policies are read on one unit pool. The readouts are insensitive to it (widening the grid to 2,4,6,8\2,4,6,8\ leaves the specificity and scaling orderings identical, ρ=1.00ρ=1.00). Three interventions act on the input, leaving the policy untouched: ℰO=id,ℰR:delete the pedestrian, _O=id, _R:\ delete the pedestrian, (1) ℰN:re-light at night, _N:\ re-light at night, and yield the paired plans XO=π(ℰOu)X^O=π(E_Ou), XR=π(ℰRu)X^R=π(E_Ru), XN=π(ℰNu)X^N=π(E_Nu). The letters O,R,NO,R,N name the three conditions only: an intervention carries one as a subscript, a plan as a superscript. All quantities below are functionals of these plans and of the pedestrian’s logged future PtP_t. The horizon 2.52.5 s is the one all six policies share. The displacement between two plans is D¯(X,Y)=1||∑t∈‖Xt−Yt‖, D(X,Y)= 1|T| _t \|X_t-Y_t\|, (2) with =0,0.1,…,2.5T=\0,0.1,…,2.5\, and the two quantities the rest of the paper is written in are F=D¯(XO,XR),I=D¯(XO,XN).F= D(X^O,X^R), I= D(X^O,X^N). (3) We call F the faithfulness displacement—the plan change attributable to the pedestrian itself—and I the interference displacement—the change the same policy produces under a large edit that should not change the correct plan, which bounds from above how far the plan moves for reasons unrelated to the pedestrian. A policy that uses the pedestrian and ignores the lighting has F≫IF I. The reverse ordering means the plan is driven by something the road does not care about. Realising ℰRE_R The target is localised by projecting its annotated 3D box into the camera rather than by taking the dataset’s 2D boxes, whose association to the tracked object is unreliable for the short-range, partially occluded pedestrians of these scenes. It is segmented with SAM [33] (Fig. 1a) (box and torso prompts, clipped to the box, dilated by 99 px) and inpainted with LaMa [34]. For the LiDAR policy the points inside the box are deleted. VLA policies read several cameras and only the front one is edited, so we retain only frames in which the target appears in no other view. Realising ℰNE_N Each image ∈[0,1]h×w×3f∈[0,1]^h× w× 3 the policy consumes is mapped by a closed-form photometric transform ℰN()=clip[g(sats)γ⊙+σϵ],ϵ∼(0,1),E_N(f)=clip\! [\,g\, (sat_sf )^γ τ+σ ε\, ], ε (0,1), (4) with satssat_s the saturation scaling, τ a per-channel tint and the noise seed fixed per frame. This is a photometric perturbation (gamma, gain, saturation, noise), not a captured night scene. We call it the night-style perturbation and use (γ,g,s,σ)=(2.2,0.45,0.45,7/255)(γ,g,s,σ)=(2.2,0.45,0.45,7/255) with a blue tint (Fig. 1a). Validity of the interventions An independent detector (Faster R-CNN [35], COCO, score ≥0.5≥ 0.5) recovers the target in 100% of the original images, in 8.5% under ℰRE_R and in 89.9% under ℰNE_N (IoU ≥0.5≥ 0.5): removal suppresses the pedestrian for an observer outside the policy, re-lighting leaves it visible. Pasting inpainting patches on empty road bounds the residual artefact at 55–17%17\% of the removal effect. I-B Readouts With ct(X)=‖Xt−Pt‖−re−rpc_t(X)=\|X_t-P_t\|-r_e-r_p the clearance net of the ego half-width re=1.0r_e=1.0 m and the pedestrian radius rp=0.4r_p=0.4 m, a plan is summarised by four functionals, S(X) S(X) =⟨min(ct,c¯)⟩t,A(X)=[mintct>0], = (c_t, c) _t,\;A(X)=1[ _tc_t>0], (5) C(X) C(X) =clip[0,1](mintct/d0),T(X)=inft:ct<d0/2.5, =clip_[0,1]( _tc_t/d_0),\;T(X)= \t\!:c_t<d_0\/2.5, (6) with T=1T=1 if the set is empty. The constants are physical, not tuned: rer_e is the half-width of the nuScenes ego box (4.08×1.854.08× 1.85 m), rpr_p the disc a standing adult occupies, d0=1d_0=1 m the outer edge of personal space [36], and c¯=10 c=10 m caps the contribution of scenes in which the pedestrian is far away. Equations (5)–(6) are the standard surrogate-safety measures [37], read as distance to a person over the whole trajectory. Speed is deliberately not a readout: the same deceleration is safe or unsafe depending on where the person is. I-C Exams Let L(X)=∑t‖Xt+δ−Xt‖L(X)= _t\|X_t+δ-X_t\| denote arc length and H the logged human trajectory. Exposure Exp=medL(XO)/L(H)Exp=med\,L(X^O)/L(H) over frames with L(H)≥0.5L(H)≥ 0.5 m. Hazard sensitivity A unit needs a reaction when the pedestrian-blind plan enters personal space, C(XR)<1C(X^R)<1. On those units, writing Δ(⋅)=(⋅)(XO)−(⋅)(XR) (·)=(·)(X^O)-(·)(X^R), HS=14[ΔA+ΔC+ΔT+tanh(ΔS/d0)]∈[−1,1].HS= 14 [ A+ C+ T+ ( S/d_0) ]∈[-1,1]. (7) Each term lies in [−1,1][-1,1] by construction and tanh squashes the single unbounded one at the d0d_0 scale. The composite is ours, its ingredients are not. Its four terms are the surrogate-safety measures of (5)–(6), and averaging normalised sub-scores into one index is how the benchmark we compare against forms its extended predictive driver model score (EPDMS) [2], keeping the two commensurable. What the check-up adds is that every sub-score is a paired difference across ℰRE_R—the ablation form used to attribute behaviour to an input in interpretability work [12]. HSHS is read against the model-independent hazard areq=v22max(d−2, 0.5),TTC0=d−2v,a_req= v^22 (d-2,\,0.5), _0= d-2v, (8) the deceleration required to stop short of a pedestrian at distance d and the corresponding time to collision (TTC) [37]. Units with v<1v<1 m/s are excluded: a stationary ego can move neither toward nor away from anyone. Scaling Sc=ρ(HS,areq)Sc=ρ (HS,\,a_req ) over needed units, with ρ Spearman’s rank correlation. Specificity On units where the pedestrian requires no reaction (C(XR)≥1C(X^R)≥ 1) the correct plan does not change when it is removed, and a plan that changes anyway over-reacts: SP=exp(−D¯(XO,XR)/0.5m)∈(0,1],SP= \! (- D(X^O,X^R)/0.5\,m )∈(0,1], (9) so SP→1SP→ 1 iff removing a pedestrian that required no reaction leaves the plan where it was. SPSP is read together with HSHS: a policy blind to pedestrians scores SP=1SP=1 and HS=0HS=0. Lighting The causal faithfulness ratio (CFRCFR) weighs the two displacements of (3) against each other, CFR=D¯(XO,XR)D¯(XO,XN)=FI,CFR= E\, D(X^O,X^R)E\, D(X^O,X^N)= EFEI, (10) A policy for which lighting is irrelevant has CFR≫1CFR 1. The thresholds (HS≥0.15HS≥ 0.15, Sc>0Sc>0, SP≥0.8SP≥ 0.8, exposure within [0.8,1.2][0.8,1.2], CFR>1CFR>1) are reference values we set, not population norms. IV Experiments IV-A Setup We examine LTF and DiffusionDrive (front camera and ego status), DiffusionDriveV2 (camera and LiDAR), SimLingo (InternVL2-1B), AutoVLA (Qwen2.5-VL-3B, three cameras × four frames) and Alpamayo-1.5 (four cameras, a diffusion head over the vision–language model cache), all with released weights. The three BEV planners are trained on NAVSIM (nuPlan) data, AutoVLA’s released checkpoint on a mixture of nuPlan and nuScenes [18], SimLingo in CARLA, and Alpamayo-1.5 on NVIDIA’s internal fleet data together with public sets that include NAVSIM-Traj and the nuScenes-derived nuScenesQA, DriveLM and OmniDrive [20]. None is fine-tuned here. On the nuScenes frames the BEV planners and SimLingo are therefore out of distribution and the two VLA policies are not, and on the NAVSIM scenes only SimLingo is. F<IF<I on average for every policy (Section IV-B, Section IV-D). For the BEV planners the nuScenes front image is centre-cropped to the 4:1 aspect of NAVSIM’s stitched front view with the sky removed and resized to the network’s input size, the logged speed and acceleration fill the status vector, and DiffusionDriveV2 receives the LiDAR sweep. The VLA policies receive the nuScenes cameras mapped approximately onto the views they expect. DiffusionDrive’s diffusion seed is fixed so that paired queries share their noise. The primary corpus is nuScenes [1] keyframes: corridor pedestrians (lateral clearance to the ego corridor ≤1≤ 1 m) at 55–1212 m, their off-road counterparts, and 3030 approach sequences at 5.85.8–14.614.6 m with logged pedestrian futures. The near-pedestrian units come from 8888 Boston (left-hand-drive vehicles, right-hand traffic) and 148148 Singapore (right-hand-drive vehicles, left-hand traffic) front-only frames. Alpamayo-1.5 covers 170170 of them, needing 1.61.6 s of history and 6.46.4 s of future within the scene. Section IV-B repeats ℰRE_R and ℰNE_N on a second, right-hand-drive corpus, the 246 NAVSIM test-split [2] Singapore scenes with a pedestrian near the ego corridor and a complete logged pedestrian future, used for the transfer and decomposition results and for the NAVSIM columns of Table I. The exam rows of Table I are measured on the nuScenes frames. IV-B The Two Axes The diagnosis reads a policy on the two axes of (3). We apply both interventions to the 246 NAVSIM Singapore near-pedestrian scenes and read, per scene, F, the plan displacement caused by removing the target pedestrian, and I, that caused by the night-style perturbation. Fig. 2: The two axes. (a) Genuine avoidance: with the pedestrian in view (XOX^O, solid) DiffusionDrive stays short of its logged future (dashed from the star), while blind to it (XRX^R, dashed) it reaches it. XNX^N gives I. (b) Moved towards: DiffusionDriveV2 plans further with the pedestrian than without. (c) Cells in the avoidance region (F≥0.5F≥ 0.5 m, F>IF>I) per policy, by ΔS S: blue ≥0.5≥ 0.5 m, navy ≤−0.5≤-0.5 m, grey between. Label: blue over all. (d) All 1,4761,476 cells (246 scenes × six policies) on the axes of (3), by policy, with means. F measures how far a plan moves, not where it moves to. We therefore decompose it (Fig. 2d), reading F against I of (3)—the same policy’s interference under an unrelated edit—and against ΔS S, the change in clearance to the pedestrian’s logged future. A response counts as avoidance only if the plan moved (F≥0.5F≥ 0.5 m) and the movement bought distance (ΔS≥0.5 S≥ 0.5 m), and, where a re-lit plan exists, only if it moved more than the policy’s own interference (F>IF>I). Four readings follow. First, the pedestrian outweighs the irrelevant edit in only 4–23% of scenes, so most of what F registers sits below the interference of an irrelevant edit, the F=IF=I line of Fig. 2d. Second, avoidance is rare where it is called for. At the logged speed, where all three conditions apply, 35 of the 1,4761,476 policy–scene cells qualify (Fig. 2c). The logged ego is stationary through most of this corpus, so a plan that does not move is usually the correct one. We therefore replay every scene at input speeds of 4 and 8 m/s, as in Section I, and read only the cells in which the pedestrian-blind plan does come within 1 m of the pedestrian’s logged future, the same criterion as C(XR)<1C(X^R)<1 in Section I. No re-lit plan exists at these speeds, so the F>IF>I condition is not applied there. Across 6 policies that is 787 of 4428 cells (11–30% per policy), and 15 of them—1.9%—are genuine avoidance. On those cells the median clearance changes by 0.00–0.03 m and the median planned distance by −-0.02–0.08 m: facing a pedestrian it would otherwise reach, the median plan neither diverts nor slows under our open-loop protocol. At the logged speed alone the need set has 53 cells and one genuine avoidance, so the counterfactual speeds add cells without changing the rate. Third, the displacement that does occur is not aimed: of the 121 cells in which the plan moved, 43 increased the clearance and 42 decreased it. Per policy the split differs sharply (Fig. 2c): DiffusionDrive reacts in few scenes but in the right direction (median ΔS=2.16 S=2.16 m), while SimLingo moves in more scenes than the other five combined and moves towards the pedestrian more often than away (32 against 13, median −-0.59 m), and it also has the highest rate of sub-1.5 s time-to-proximity (Table I). SimLingo is trained in simulation and run here on real images, so part of what we measure is a sim-to-real gap, which is exactly what a check-up at a new site should surface. Finally, what a policy yields does not grow with the hazard: the rank correlation between ΔS S and the deceleration the scene demands is between −-0.37 and 0.07. An exam that rewarded responsiveness alone would therefore rank first the policy with the most sub-1.5 s encounters. A score merges these four readings into one number. The edited frames keep them apart. IV-C The Five Exams TABLE I: The full report for the six policies. Rows are readings, columns policies. Brackets: 95% CI. Arrows: outside the reference threshold of Section I. Bold: best per row (p<0.01p<0.01 for the two rows of N). Human: the logged trajectory scored against each policy’s blind plan. EPDMS on all 783 and on the 246 near-pedestrian NAVSIM scenes. Reading \ Policy D LTF DDv2 SimLingo AutoVLA Alpamayo 1.5 Human Exposure (ratio to human) ≈1≈ 1 0.50↓ 0.79↓ 0.88 2.19↑ 1.01 1.13 1.00 95% CI [0.46, 0.55] [0.75, 0.84] [0.77, 0.95] [1.98, 2.65] [0.97, 1.11] [1.06, 1.21] Hazard sens. HS∈[−1,1]HS∈[-1,1] ↑ 0.06↓ 0.03↓ 0.10↓ 0.00↓ 0.02↓ 0.01↓ 0.57 95% CI [−-0.01, 0.14] [0.00, 0.06] [0.05, 0.16] [−-0.03, 0.03] [0.00, 0.05] [−-0.01, 0.02] Scaling (rank corr.) ↑ −-0.28↓ −-0.13↓ 0.23 −-0.22↓ −-0.18↓ −-0.23↓ 0.13 95% CI [−-0.50, −-0.05] [−-0.34, 0.07] [−-0.02, 0.50] [−-0.38, −-0.08] [−-0.34, 0.01] [−-0.49, 0.03] Specificity SP∈(0,1]SP∈(0,1] ↑ 0.76↓ 0.78↓ 0.65↓ 0.44↓ 0.76↓ 0.84 – 95% CI [0.73, 0.80] [0.75, 0.81] [0.61, 0.70] [0.38, 0.50] [0.72, 0.79] [0.81, 0.87] Lighting CFRCFR (ratio) ↑ 0.23↓ 0.23↓ 0.47↓ 0.23↓ 0.44↓ 0.25↓ – 95% CI [0.17, 0.30] [0.19, 0.28] [0.37, 0.58] [0.18, 0.29] [0.33, 0.55] [0.16, 0.35] Collision at 8 m/s, vis. / rm. (%) ↓ 7.6 / 9.2 9.9 / 10.7 3.1 / 6.1 40.5 / 36.6 32.1 / 29.8 11.2 / 11.2 – speed of N (%) ↓ %37\% %12\% −%-8\% %16\% −0.3%-0.3\% −2%-2\% – clearance of N, ΔNS _NS (m) ↑ −0.37-0.37 −0.24-0.24 0.150.15 −0.47-0.47 0.0050.005 0.030.03 – Standard scores (rank) L2 (m), vis. / rm. ↓ 0.98 / 0.99 0.88 / 0.91 1.35 / 1.31 3.28 / 3.35 0.68 / 0.66 0.77 / 0.72 – EPDMS, all scenes ↑ 0.535 (4) 0.509 (5) 0.652 (3) 0.359 (6) 0.683 (1) 0.663 (2) – EPDMS, near-ped. ↑ 0.775 (2) 0.774 (3) 0.371 (6) 0.617 (4) 0.414 (5) 0.788 (1) – TTC<<1.5 s (%), LHD ↓ 3.2 (1) 5.4 (2) 18.3 (3) 61.3 (6) 19.4 (4) 25.4 (5) – TTC<<1.5 s (%), RHD ↓ 13.4 (1) 15.9 (2) 22.0 (5) 56.1 (6) 20.7 (4) 19.5 (3) – Table I gives the six profiles, read under our open-loop protocol on the nuScenes frames: the behaviour behind the ranks of Section IV-E. Exposure The six policies differ first in how far they plan, and this alone will turn out to explain more of the collision picture below than perception does: exposure spans a factor of four, from DiffusionDrive at half the human’s distance to SimLingo at more than twice it, with AutoVLA and Alpamayo-1.5 near the human’s own. Hazard sensitivity On these frames the plans respond little to the pedestrian: on units that need a reaction, seeing it buys only 0.00–0.10 of safety on a [−1,1][-1,1] scale, against 0.57 when the logged human trajectory H takes the place of XOX^O in (7) (Table I), a reference rather than a ceiling, since H and XRX^R differ for reasons beyond the pedestrian. The benefit is also unreliable in direction: counting only cases where seeing the pedestrian changes the outcome, it turns a collision into a clean pass about as often as it turns a pass into a collision, for every policy (DiffusionDrive 9 vs. 5, SimLingo 21 vs. 30, the rest at most 5 either way, all exact-binomial p≥0.26p≥ 0.26) — what reads as a safety margin in Table I does not carry over to the individual case. Scaling The benefit does not grow with the hazard: its rank correlation with areqa_req is negative for 5 policies, down to −-0.28 for DiffusionDrive. DiffusionDriveV2 is the only exception (0.23), and its benefit stays below 0.150.15. Replacing the hazard axis by the time-based TTC0TTC_0, whose sign is reversed by construction, leaves the picture unchanged (−-0.07 to 0.32, against −-0.28 to 0.23). The human reference is 0.130.13: a driver who already yields at every hazard level has little room left to scale, so this exam is informative where HSHS is small, which is where all six policies sit. Specificity A policy whose plan moves when nothing requires it is not cautious but unsteady, and steadiness is what the exploratory analysis of Section IV-E finds to travel across driving sides. SimLingo’s plan moves most when nothing requires it (SP=0.44SP=0.44), while the other five stay steadier, at 0.65–0.84. Lighting Removing the pedestrian moves every plan less than the night-style perturbation does (CFR<1CFR<1 for all six, Table I). Section IV-D reads this exam in depth. Collisions Collisions with the pedestrian’s logged future follow how far a policy plans to drive, not what it perceives. At the logged speed, the visible and removed rates differ by at most 0.4 points for every policy, and at 8 m/s by at most 3.9. SimLingo, which plans the longest trajectories, collides most. The low-exposure policies almost never reach the pedestrian. DiffusionDrive is safe because it stays short, AutoVLA and Alpamayo-1.5 cover a human-like distance without yielding, and SimLingo covers the most ground with the lowest specificity. IV-D The Lighting Exam in Depth Lighting should not change what a driver does, so this exam asks how far a policy is moved by an irrelevant change. It is the verdict that transfers between driving sides for every policy (Section IV-E). Magnitude Removing a pedestrian from the corridor moves every policy’s plan only 0.23–0.47 times as far as the night-style perturbation does, with every 95% interval below one (Table I). On detector-verified edits CFR=0.30CFR=0.30–0.640.64. Direction: faster, and therefore closer Three policies plan faster under the night-style perturbation, DiffusionDrive by 37%, SimLingo by 16% and LTF by 12%, and their plans pass 0.240.24–0.470.47 m closer to the pedestrian, while DiffusionDriveV2, the only policy that also reads LiDAR, slows (−8%-8\%) and keeps more distance. Whether these profiles and orderings persist across driving sides is tested in Section IV-E. IV-E Prognosis: What Transfers, and at What Cost Fig. 3: What a verdict costs. One frame is 5 s of near-pedestrian driving. (a) Spread of each estimate against sample size with its fitted law, one line per policy. (b) Driving needed per verdict, one dot per policy, off-scale values at the axis end. Two questions decide whether a few hundred edited frames can stand in for a rank: which verdicts carry over to the other driving side, and how many frames each verdict needs. A pre-registered transfer test Does a diagnosis made on one driving side describe the same policy on the other? Before computing any right-hand-drive statistic, we fixed seven predictions, P1–P7 in Table I, and their decision rules. They were derived from the profile of the 88 Boston frames, and we tested them on the near-pedestrian units of the 148 Singapore frames (Table I). The registration covered the six policies public at that time. NVIDIA subsequently released Alpamayo 1.5, which accepts navigation instructions, and we use it in place of Alpamayo-R1 throughout this paper. Because both driving sides of Alpamayo 1.5 were measured together, its numbers do not satisfy the registration discipline, so the table is evaluated on the five policies that do. Retaining the replaced policy changes none of the seven verdicts. TABLE I: Pre-registered transfer test. Predictions fixed on 88 Boston frames, tested on 148 Singapore frames, five policies (the replaced sixth changes no verdict). No. Prediction Holds Evidence P1 Lighting outweighs the pedestrian (CFR <1<1) for every policy yes max upper CI 0.70 P2 Hazard sensitivity stays below 0.15 (upper CI) no D, DDv2 above P3 Exposure ordering transfers (ρ≥0.6ρ≥ 0.6; SimLingo top, D bottom) yes ρ=0.90ρ=0.90 P4 Specificity ordering transfers (ρ≥0.6ρ≥ 0.6; SimLingo lowest SPSP) yes ρ=0.60ρ=0.60 P5 Point values stay within the left-hand CI (≥ 70% of 15 cells) no 60% P6 Night- and removal-induced plan changes more orthogonal, less shift between sides (auxiliary) no ρ=−0.90ρ=-0.90 (opposite) P7 SimLingo collides most, visible vs. removed within 3 pts yes gap 2.7 pts What transferred 4 of the seven predictions hold: the lighting verdict, the two orderings and the collision outcome. The night-style perturbation outweighs the pedestrian in every policy. Exposure keeps its order (ρ=0.90ρ=0.90), and so does specificity, with SimLingo the lowest, at the registered threshold (ρ=0.60ρ=0.60). SimLingo collides most, with the visible and removed rates within 3 points of each other. What did not Three predictions fail. • Point values drift: only 60% of the model–exam cells stay inside their Boston interval. • In Singapore, DiffusionDrive and DiffusionDriveV2 show a small but non-zero hazard sensitivity (HS=0.09HS=0.09 and 0.110.11) whose upper confidence limits, 0.180.18 and 0.200.20, cross the registered 0.150.15 line. The failure is one of precision rather than of sign: HSHS stays below 0.120.12 on both sides for every policy, but 148 frames do not pin it below 0.150.15. • P6, on the alignment between the night-induced and the removal-induced plan changes, predicted that policies with more orthogonal changes would shift less between sides. It failed (ρ=−0.90ρ=-0.90, opposite to the prediction). We no longer use this quantity and report its verdict for completeness. Reading A small check-up can therefore report an ordering and a verdict for use in a new domain, but not a calibrated value. The standard scores on the same frames Running the standard scores on the same six policies (Table I) shows what a rank cannot see. Removing the pedestrian changes none of the nuScenes open-loop numbers, the L2 error by at most 0.07 m and the frontal and pedestrian collision rates by at most 1.7 and 3.0 points, so the metric cannot tell whether a policy saw the pedestrian, and its ordering follows how far a policy’s planned distance departs from the human’s (0.66 with |exposure−1||exposure-1|). On NAVSIM the EPDMS column follows the official protocol on the Singapore part of the test split and stands in for the leaderboard here. Its ordering over all 783 Singapore scenes is unrelated to the one on the 246 near-pedestrian scenes (ρ=−0.09ρ=-0.09). Its first, AutoVLA, falls to fifth and its third, DiffusionDriveV2, to last, because the full-set score is carried by the map-compliance gates (95% of the spread) and none of the exams above is visible in it. Pedestrian TTC, by contrast, orders the policies the same way on both driving sides (ρ=0.77ρ=0.77) and coincides with exposure (1.00, 0.77). Which exam travels (exploratory) The target is how often each policy keeps clear of the pedestrian on the Singapore scenes, the share of moving-ego scenes with minimum time-to-proximity ≥1.5≥ 1.5 s, the complement of the TTC column of Table I. This target changes dataset and city as well as driving side, whereas the pre-registered test above stays within nuScenes and changes city and side, but not dataset or sensor rig. Each predictor is a six-policy ordering obtained without that data, and we read the rank correlations as orderings, not as tests. The leaderboard score and the open-loop error agree with the target at |ρ|≤0.43|ρ|≤ 0.43 (NAVSIM leaderboard 0.09, nuScenes L2 0.43). Pedestrian-centred readouts that need no counterfactual edit do better: the open-loop pedestrian-collision rate on the nuScenes near-pedestrian frames at 0.81 (ties averaged) and the TTC column of Table I taken on the left-hand-drive side at 0.77. The exams are not a substitute for such measurements. Of the five, read on the 88 Boston frames as a pre-deployment diagnosis would be, specificity matches the target best, ρ=0.83ρ=0.83 (D 0.87, LTF 0.78, DDv2 0.66, SimLingo 0.37, AutoVLA 0.79, Alpamayo 1.5 0.85), against −0.77-0.77 for exposure and −0.37-0.37 for CFRCFR, where specificity and CFRCFR are higher-is-better and exposure’s negative value says that policies that plan further keep clear less often. What the exams add is the reason a policy orders where it does: displacement that is not aimed away from the pedestrian is not protection, and a policy producing a lot of it comes close to the pedestrian on both sides. The selection report Read together, the profiles of Table I give each policy a prognosis under our open-loop protocol. SimLingo plans furthest, moves its plan when a pedestrian requires no reaction (SP=0.44SP=0.44), moves towards the pedestrian more often than away, and collides most on both sides. Trained in simulation, it would need target-domain post-training before deployment, and the check-up prices the re-check at a few dozen frames for exposure and specificity. AutoVLA and Alpamayo-1.5 keep a human-like exposure and a steady plan but show no hazard response. Before deployment, post-training should add the response to the pedestrian, and only hundreds of frames with a genuine need to react can verify that it did. DiffusionDrive and LTF are safe because they plan short, which holds only while exposure stays low, so any post-training that lengthens their plans should be re-examined for hazard sensitivity. DiffusionDriveV2 is the only policy with a positive scaling estimate (0.23, interval crossing zero) and the only one that slows under the night-style perturbation, at the cost of the lowest specificity among the BEV planners. What a verdict costs For every exam and policy we subsample n of the 170 frames that all six policies share, 400 times, and fit the spread of the estimate with sd(n)=a1/n−1/Msd(n)=a 1/n-1/M, M=170M=170 (Fig. 3). Well-fitting exams The law is the finite-population sampling form, so the fit is not a test. What it yields is the spread constant a, which converts into the number of frames a verdict needs. For exposure, the lighting ratio and specificity the fit is tight (R2≥0.97R^2≥ 0.97) and the verdicts settle inside the pool. • Lighting outweighs the pedestrian (CFR<1CFR<1) is settled for every policy within 14 frames. • Exposure differs from the human’s is settled within 51 frames for the four policies whose exposure is far from one. For AutoVLA and Alpamayo-1.5 it would take hundreds to thousands of frames, which is itself a finding, since they plan a human-like distance. • Orderings of the six policies stabilise quickly: exposure with 10 frames (96% of subsamples reach ρ≥0.6ρ≥ 0.6 with the full ordering) and specificity with 40 (99%). Poorly fitting exams Hazard sensitivity behaves differently. Its fits are poor (R2R^2 as low as 0.26) wherever few units need a reaction. Deciding that HS≠0HS≠ 0 needs 57–1,326 frames, an extrapolation beyond the 170-frame pool for four policies, except for SimLingo, whose estimate sits on the threshold, and the ordering reaches 84% stability only at 90 frames. The division is therefore clear. A few dozen frames, 3 to 13 logged scenes, diagnose exposure, specificity and lighting. Whether a policy manages hazard needs hundreds of frames that contain a genuine need to react. V Discussion Limitations The exams are open-loop, replayed from logs and re-anchored to the logged pose. The night-style perturbation is photometric, not captured. The counterfactual pairs are edited, not captured: 8.5% of removals leave a detectable residue, the inpainting artefact is 5–17% of the effect, and the VLA policies are edited only through their front camera. Alpamayo-1.5 covers 170 of the nuScenes frames, SimLingo’s 2.5 s plans are extrapolated to NAVSIM’s 4 s (holding position instead gives EPDMS 0.47, still the lowest), and cross-policy correlations rest on six policies. Next steps The check-up is the first half of a benchmark-to-deployment loop. The paired data will next be captured rather than constructed, the same scene with and without a pedestrian recorded on our vehicle and re-rendered with 3D Gaussian splatting. The day–night pairs will move from a photometric transform to captured or physically rendered lighting. And the VLA policies will be post-trained on target-domain data and re-examined, so that the diagnosis measures what post-training recovers. The sample-size laws above price that collection. Beyond these steps, we intend to refine and systematise the check-up into a standing procedure for evaluating and selecting learned driving policies, so that a deployment decision can rest on a diagnosis rather than on a rank. VI Conclusion A benchmark score reports an outcome, not the behaviour behind it, so a rank cannot be trusted to hold at a new site. The check-up reads that behaviour from a few hundred logged frames through verified, pixel-level counterfactual edits, reduced to two causal axes, F and I, on which five exams are built. Three results carry the paper. On the cells in which a pedestrian lies on the planned path, only 1.9% of responses are genuine avoidance and, in open loop, the median clearance change is at most 0.03 m, while the night-style perturbation moves every plan further than the pedestrian does. Diagnosed on one driving side and tested on the other, the exposure and specificity orderings, the lighting verdict and the collision outcome transfer, while point values and the hazard-sensitivity verdict do not, and the profiles read as a selection report: which policy is safe because it plans short, which covers a human-like distance without yielding, and which is unsteady under a change that requires no reaction. Each verdict is priced in advance, lighting within 14 frames and exposure within 51, so that, for the verdicts that transfer, each diagnosis is priced at tens of frames rather than a fleet test. Acknowledgment This research was supported by the U.S. Department of Transportation (No. 69A3552348305) and by grants from NVIDIA utilizing NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition GPUs. References [1] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. [2] D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, A. Geiger, and K. Chitta, “NAVSIM: Data-driven non-reactive autonomous vehicle simulation and benchmarking,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. [3] S. Z. Zhao, L. Wang, H. Ruan, Y. Bao, Y. Chen, Z. Leng, A. Ravichandran, H. He, Z. Zhou, X. Han, A. Peri, Z. Huang, P. Desai, H. Christensen, J. Ma, and B. Zhou, “BridgeSim: Unveiling the OL–CL gap in end-to-end autonomous driving,” arXiv preprint arXiv:2604.10856, 2026. [4] Z. Li, Z. Yu, S. Lan, J. Li, J. Kautz, T. Lu, and J. M. Alvarez, “Is ego status all you need for open-loop end-to-end autonomous driving?” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [5] F. Codevilla, E. Santana, A. M. López, and A. Gaidon, “Exploring the limitations of behavior cloning for autonomous driving,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2019. [6] K. D. Kusano, J. M. Scanlon, Y.-H. Chen, T. L. McMurry, T. Gode, and T. Victor, “Comparison of Waymo rider-only crash rates by crash type to human benchmarks at 56.7 million miles,” Traffic Injury Prevention, vol. 26, no. sup1, 2025. [7] Physical Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levine, A. Li-Bell, Y. Lu, V. Mano, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, C. Sharma, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, A. Swerdlow, J. Tanner, M. Torne, Q. Vuong, A. Walling, H. Wang, B. Williams, S. Yoo, L. Yu, U. Zhilinsky, and Z. Zhou, “π0.6∗π^*_0.6: A VLA that learns from experience,” arXiv preprint arXiv:2511.14759, 2025. [8] N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” RAND Corporation, Tech. Rep. R-1478, 2016. [9] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International Conference on Machine Learning (ICML), 2015. [10] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in International Conference on Learning Representations (ICLR), 2021. [11] N. Kachaev, M. Kolosov, D. Zelezetsky, A. K. Kovalev, and A. I. Panov, “Don’t blind your VLA: Aligning visual representations for OOD generalization,” arXiv preprint arXiv:2510.25616, 2025. [12] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski et al., “Representation engineering: A top-down approach to AI transparency,” arXiv preprint arXiv:2310.01405, 2023. [13] E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau, “Function vectors in large language models,” in International Conference on Learning Representations (ICLR), 2024. [14] K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger, “TransFuser: Imitation with transformer-based sensor fusion for autonomous driving,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. [15] B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, and X. Wang, “DiffusionDrive: Truncated diffusion model for end-to-end autonomous driving,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [16] J. Zou, S. Chen, B. Liao, Z. Zheng, Y. Song, L. Zhang, Q. Zhang, W. Liu, and X. Wang, “DiffusionDriveV2: Reinforcement learning-constrained truncated diffusion modeling in end-to-end autonomous driving,” arXiv preprint arXiv:2512.07745, 2025. [17] K. Renz, L. Chen, E. Arani, and O. Sinavski, “SimLingo: Vision-only closed-loop autonomous driving with language-action alignment,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [18] Z. Zhou, T. Cai, S. Z. Zhao, Y. Zhang, Z. Huang, B. Zhou, and J. Ma, “AutoVLA: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning,” arXiv preprint arXiv:2506.13757, 2025. [19] NVIDIA, Y. Wang, W. Luo, J. Bai et al., “Alpamayo-R1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail,” arXiv preprint arXiv:2511.00088, 2025. [20] NVIDIA, “Alpamayo 1.5: An open reasoning vision-language-action model with navigation guidance,” 2026, model card https://huggingface.co/nvidia/Alpamayo-1.5-10B and code https://github.com/NVlabs/alpamayo1.5. [21] B. Jaeger, K. Chitta, and A. Geiger, “Hidden biases of end-to-end driving models,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [22] X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan, “Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. [23] W. Cao, M. Hallgarten, T. Li, D. Dauner, X. Gu, C. Wang, Y. Miron, M. Aiello, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, A. Geiger, and K. Chitta, “Pseudo-simulation for autonomous driving,” in Conference on Robot Learning (CoRL), 2025. [24] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Annual Meeting of the Association for Computational Linguistics (ACL), 2020. [25] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in International Conference on Learning Representations (ICLR), 2019. [26] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2019. [27] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, p. 665–673, 2020. [28] P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Pérez, and M. Cord, “STEEX: Steering counterfactual explanations with semantics,” in European Conference on Computer Vision (ECCV), 2022. [29] R. Roelofs, L. Sun, B. Caine, K. S. Refaat, B. Sapp, S. Ettinger, and W. Chai, “CausalAgents: A robustness benchmark for motion forecasting using causal relationships,” arXiv preprint arXiv:2207.03586, 2022. [30] W. Ljungbergh, A. Tonderski, J. Johnander, H. Caesar, K. Åström, M. Felsberg, and C. Petersson, “NeuroNCAP: Photorealistic closed-loop safety testing for autonomous driving,” in European Conference on Computer Vision (ECCV), 2024. [31] Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, A. J. Yang, and R. Urtasun, “UniSim: A neural closed-loop sensor simulator,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. [32] H. Zhou, L. Lin, J. Wang, Y. Lu, D. Bai, B. Liu, Y. Wang, A. Geiger, and Y. Liao, “HUGSIM: A real-time, photo-realistic and closed-loop simulator for autonomous driving,” arXiv preprint arXiv:2412.01718, 2024. [33] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [34] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with fourier convolutions,” in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022. [35] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2015. [36] E. T. Hall, The Hidden Dimension. Doubleday, 1966. [37] D. Gettman and L. Head, “Surrogate safety measures from traffic simulation models,” Transportation Research Record, vol. 1840, no. 1, p. 104–115, 2003.