Paper deep dive
Collision Snapshot Guided Time-Reversed Safety-Critical Scenario Generation
Taehyung Kim, Jongeun Choi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/12/2026, 2:37:28 AM
Summary
The paper introduces COSTER, a framework for generating safety-critical traffic scenarios for autonomous vehicle training. COSTER uses learned traffic priors to determine plausible collision times and locations by constructing a 'collision snapshot' where a new adversarial vehicle is placed in contact with a target vehicle. A conditional variational autoencoder (CVAE) then performs a time-reversed rollout to reconstruct the adversarial vehicle's trajectory backward from the collision point. Experiments on the Waymo Open Motion Dataset show COSTER outperforms existing methods in plausibility, diversity, and data efficiency, and agents trained on COSTER-generated scenarios reduce collision rates by 31%.
Entities (9)
Relation Signals (7)
Taehyung Kim → affiliatedwith → Yonsei University
confidence 95% · Taehyung Kim and Jongeun Choi are with Yonsei University, Seoul, South Korea.
Jongeun Choi → affiliatedwith → Yonsei University
confidence 95% · Taehyung Kim and Jongeun Choi are with Yonsei University, Seoul, South Korea.
COSTER → evaluatedon → Waymo Open Motion Dataset
confidence 95% · We train the models described in Sec. III-B and Sec. III-D using the Waymo Open Motion Dataset v1.1 (WOMD)
COSTER → uses → Conditional Variational Autoencoder
confidence 95% · Starting from this collision snapshot, a conditional variational autoencoder is used to perform a time-reversed rollout
COSTER → outperforms → Adv-BMT
confidence 90% · COSTER outperforms existing methods in plausibility, diversity, and data efficiency.
COSTER → outperforms → TrafficGen
confidence 90% · COSTER outperforms existing methods in plausibility, diversity, and data efficiency.
COSTER → outperforms → LCTGen
confidence 90% · COSTER outperforms existing methods in plausibility, diversity, and data efficiency.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The generation of safety-critical traffic scenarios is essential for training and evaluating autonomous vehicles. Prior approaches typically perturb the trajectories of existing agents in a traffic scenario using simplified adversarial objectives to induce safety-critical interactions, which can limit the plausibility and diversity of the generated scenarios. Although inserting new adversarial vehicles can alleviate this limitation, determining when and where to introduce them in a scenario-specific manner remains challenging. In this work, we introduce \underline{CO}llision \underline{S}napshot guided \underline{T}im\underline{E}-\underline{R}eversed safety-critical scenario generation (COSTER), a framework that leverages learned traffic priors to determine plausible collision times and locations. COSTER first constructs a collision snapshot by inserting a new vehicle in contact with the target vehicle at the identified collision state within a traffic scenario. Starting from this collision snapshot, a conditional variational autoencoder is used to perform a time-reversed rollout, reconstructing the trajectory of the inserted vehicle backward toward earlier timesteps. Experiments show that COSTER outperforms existing methods in plausibility, diversity, and data efficiency. Moreover, agents trained on COSTER-generated scenarios reduce collision rates by 31\% on safety-critical scenarios from the Waymo Open Motion Dataset while also improving ego task completion. The project website is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.06433v1
- Canonical: https://arxiv.org/abs/2609.06433v1
Trouble viewing inline? Open PDF directly →
Full Text
45,668 characters extracted from source content.
Expand or collapse full text
Collision Snapshot Guided Time-Reversed Safety-Critical Scenario Generation Taehyung Kim Affiliation: Taehyung Kim and Jongeun Choi are with Yonsei University, Seoul, South Korea. kth8606, jongeunchoi@yonsei.ac.kr Jongeun Choi Affiliation: Taehyung Kim and Jongeun Choi are with Yonsei University, Seoul, South Korea. kth8606, jongeunchoi@yonsei.ac.kr Abstract The generation of safety-critical traffic scenarios is essential for training and evaluating autonomous vehicles. Prior approaches typically perturb the trajectories of existing agents in a traffic scenario using simplified adversarial objectives to induce safety-critical interactions, which can limit the plausibility and diversity of the generated scenarios. Although inserting new adversarial vehicles can alleviate this limitation, determining when and where to introduce them in a scenario-specific manner remains challenging. In this work, we introduce COllision Snapshot guided TimE-Reversed safety-critical scenario generation (COSTER), a framework that leverages learned traffic priors to determine plausible collision times and locations. COSTER first constructs a collision snapshot by inserting a new vehicle in contact with the target vehicle at the identified collision state within a traffic scenario. Starting from this collision snapshot, a conditional variational autoencoder is used to perform a time-reversed rollout, reconstructing the trajectory of the inserted vehicle backward toward earlier timesteps. Experiments show that COSTER outperforms existing methods in plausibility, diversity, and data efficiency. Moreover, agents trained on COSTER-generated scenarios reduce collision rates by 31% on safety-critical scenarios from the Waymo Open Motion Dataset while also improving ego task completion. The project website is available at https://anonym-121.github.io/COSTER/. I Introduction One of the major obstacles to progress toward fully autonomous driving is well illustrated by the concept of the ‘curse of rarity’ proposed by Liu and Feng [1]. The curse of rarity refers to the phenomenon where safety-critical events, resulting from complex combinations of diverse environmental factors, occur too infrequently in real-world scenarios. This scarcity poses a challenge for data-driven algorithms to model and generalize such events. One promising approach to addressing these challenges is to leverage simulation to generate large volumes of safety-critical scenarios for training [2] and validation [3]. Therefore, the generation of diverse and plausible safety-critical scenarios has been actively studied from multiple perspectives [4, 5]. Among the various approaches to safety-critical scenario generation, methods targeting the planning module of autonomous vehicles have primarily focused on introducing adversarial vehicles within the nominal traffic scenario to threaten the target vehicle. A nominal traffic scenario can be transformed into a safety-critical scenario by selecting an existing vehicle and perturbing its trajectory so that it poses a threat to the target vehicle. However, this approach requires hand-crafted adversarial objectives, making it costly to design and limiting the diversity of collision scenarios. It also strongly depends on the original scenario configuration: unless the perturbed agent is already close enough to approach the target vehicle, the resulting behavior tends to be unrealistically aggressive or fails to produce a safety-critical scenario. To address these limitations, recent approaches insert a new adversarial vehicle into the nominal scenario [6]. To ensure a collision with the target vehicle, the adversarial vehicle is initialized at a selected time with a pose that collide with the target, and trajectory of adversarial vehicle is then generated backward in time. This reframing avoids the need to hand-craft an adversarial objective and removes the dependence on the base scene configuration, while guaranteeing a collision with the target vehicle. Despite its effectiveness, prior work determines the inserted vehicle using random pose offsets and a fixed collision time. Random pose placement may fail to account for the surrounding context, leading to overly aggressive scenarios or providing the time-reversed trajectory generation model with out-of-distribution initial conditions. Moreover, a fixed collision time may overlook more favorable opportunities for generating diverse and plausible collisions within the nominal scenario. Fig. 1: The overview of COSTER. COSTER predicts surrounding vehicle states at each timestep (Sec. I-B), identifies the highest-risk timestep to construct a collision snapshot by inserting a contextually plausible adversarial vehicle in contact with the target vehicle (Sec. I-C), and generates physically feasible safety-critical scenarios via time-reversed trajectory rollout with clustering-based selection and refinement (Sec. I-D). To address this limitation, we introduce COllision Snapshot-guided TimE-Reversed scenario generation (COSTER). COSTER leverages direct access to collision snapshots in time-reversed generation, using traffic priors learned from the nominal traffic data to determine both the collision configuration and collision time. Compared with prior approaches that rely on random pose initialization and a fixed collision time, this traffic-prior-based design provides three key advantages: (i) it accounts for the surrounding traffic context to construct scenario-specific collision configurations; (i) it reduces the likelihood of placing the inserted vehicle in an out-of-distribution state, enabling the subsequent generative module to reconstruct plausible trajectories from more realistic initial conditions; and (i) it selects collision times according to the evolving traffic context, creating more diverse and plausible opportunities for safety-critical interactions. In addition, we propose a conditional variational autoencoder (CVAE) [7] tailored for time-reversed trajectory generation, which updates surrounding-agent information at each timestep during the backward rollout from a collision snapshot. With these components, we aim to achieve diverse, and context-aware generation of time reversed trajectory. The overall framework is illustrated in Fig. 1. Our main contributions can be summarized as follows: • We propose COSTER, a collision-snapshot-guided time-reversed generation framework that directly incorporates scenario-specific traffic priors into collision configurations and time. • We introduce a velocity-entropy-based strategy for collision-time selection and a CVAE-based time-reversed rollout that updates surrounding-agent information at each timestep, enabling multimodal and context-aware adversarial trajectory generation. • We show that COSTER outperforms state-of-the-art methods in scenario quality and diversity, while agents trained on COSTER-generated scenarios achieve improved downstream driving safety. I Related Work Early importance sampling [8] and adaptive stress testing [9] methods accelerate rare-event discovery but operate at the parameter level. Subsequent work shifted to trajectory-level perturbation, modifying recorded motions under kinematic constraints [10] or differentiable model constraints [11, 12]. STRIVE [13] encodes nominal scenarios into a CVAE latent space and optimizes adversarial objectives therein, while CAT [14] decomposes the objective into motion-prediction sub-problems. However, these methods depend on simplified adversarial objectives (e.g., distance minimization [13], marginal collision probability [14]) that do not account for scenario-specific context, resulting in limited diversity, overly aggressive and map-violating trajectories. Several recent methods attempt to address these limitations. CaDRE [15] uses quality-diversity optimization to broaden collision configurations, while GOOSE [16] and SEAL [17] improve behavioral realism via goal-conditioned and skill-based hierarchical RL, respectively. Despite these advances, each method retains notable limitations. CaDRE relies on a simplified distance-based objective with diversity bounded by its predefined measure space. GOOSE requires manual design and selection of an interaction mode (e.g., cut-in, deceleration), while SEAL depends on a pretrained CAT model. Both, along with CaDRE, modify existing agents, making their generation capacity heavily dependent on the initial configuration of the base scenario. To address these limitations, Adv-BMT [6] inserts a new adversarial vehicle into the given scenario. The vehicle is initialized at a future state that collides with the target vehicle, and its trajectory is then generated backward in time with bidirectional transformer. However, the adversarial vehicle is randomly initialized relative to the target vehicle, without explicitly accounting for the scenario-specific traffic context. Moreover, such random initialization can place the learned generative model under out-of-distribution initial conditions, often resulting in implausible trajectories and requiring extensive filtering. These limitations restrict the diversity and plausibility of the generated scenarios. In addition, Adv-BMT uses a fixed collision time rather than explicitly selecting a collision time suited to each scenario. In contrast, COSTER leverages traffic priors to construct scenario-specific collision snapshots by estimating contextually plausible adversarial vehicle states in contact with the target vehicle. This allows the subsequent CVAE-based time-reversed rollout to start from an in-distribution initial state, avoiding map-inconsistent or physically infeasible initialization and enabling stable trajectory generation using a CVAE trained on nominal data. COSTER further selects the collision time for each scenario based on velocity entropy, aiming to improve data efficiency while promoting diverse and contextually plausible collision scenarios. I Proposed Method I-A Problem Formulation A traffic scenario is denoted as S1:T=(M,1:T)S_1:T=(M,T_1:T), where M represents the vectorized map and 1:T=τ1:T1,…,τ1:TNT_1:T=\ _1:T^1,…, _1:T^N\ denotes the set of trajectories for N vehicles over T timesteps. Each trajectory τ1:Tn _1:T^n consists of the vehicle states across time, represented as s1n,…,sTn\s_1^n,…,s_T^n\. COSTER aims to generate a collision scenario S′=(M,1:tcol′∪τ1:tcoladv)S = (M,\;T_1:t_col ∪ _1:t_col^adv ), where tcolt_col is the collision time and τ1:tcoladv _1:t_col^adv is the trajectory of an inserted adversarial vehicle that induces safety-critical events with the target vehicle in the original scenario. COSTER constructs S′S in two stages. First, it selects a collision snapshot χ=(tcol,stcoladv,stcoltar)χ=(t_col,\,s_t_col^adv,\,s_t_col^tar), where stcoladvs_t_col^adv and stcoltars_t_col^tar denote the states of the adversarial and target vehicles at tcolt_col. Second, a time-reversed trajectory distribution pφp_ is trained and used to sample trajectories of the adversarial vehicle τ1:tcol−1adv _1:t_col-1^adv from pφ(⋅∣stcoladv,S1:tcol)p_ (\,· s_t_col^adv,\,S_1:t_col). I-B Local Region-Based Vehicle State Modeling To contextually initialize the adversarial vehicle, we partition M into local regions m1,…,mK\m_1,…,m_K\ and estimate the expected vehicle state within each region. Large-scale real-world traffic datasets often provide vectorized M with approximations of curved road elements, including lane boundaries, lane centerlines, and crosswalks. To partition the region, we uniformly sample points along the vectorized lane lines in the given M, from which two adjacent points, qk1,qk2∈ℝ2q_k^1,q_k^2 ^2, are selected as shown in Fig. 2 (a). We define a local region coordinate by setting the origin at qk1q_k^1, aligning the y-axis with the direction of qk2−qk1q_k^2-q_k^1, and constructing a rectangular region with a width of wrw_r along the x-axis and a height of ∥qk2−qk1∥ q_k^2-q_k^1 along the y-axis. For each local region mkm_k, the states of traffic agents passing through mkm_k are represented in the region’s local coordinate system as a vector rkr_k, defined as rk=(qk1,qk2,lk,σk,pk,vk,bk,δk)r_k=(q_k^1,q_k^2,l_k, _k,p_k,v_k,b_k, _k). (a)(b) Fig. 2: Visualization of a sample scenario with overlaid local regions. (a) Region-wise representation. (b) Adversarial vehicle placement. At the given collision time, the peak-probability region within the elliptical area (red dashed line) is selected. The heading is then estimated at the red region, and the adversarial vehicle is translated into contact with the target vehicle to be placed at its final high-probability position. In this representation, lkl_k denotes the lane type and σk _k indicates the traffic light state. pk,vk,bk∈ℝ2p_k,v_k,b_k ^2 represent the position, velocity and size of the traffic agent, respectively. δk _k is an indicator variable that is 0 when no traffic agent is present in the region and 1 otherwise. The regional representations r1,…,rK\r_1,…,r_K\ are first projected into a higher-dimensional feature space and then refined via multi-context gating (MCG) [18], which approximates cross-attention [19] by conditioning on a single global context vector, offering significantly reduced computational cost while preserving expressive power [20, 21]. The resulting context-aware features rk′r_k are decoded by independent MLP heads: one parameterizes a Bernoulli distribution over δk _k for placement prediction, while the others model pkp_k, vkv_k, and bkb_k as discretized categorical distributions, from which continuous values are recovered via bin-center decoding at inference. Following prior works [20, 21], we model each vehicle attribute with a separate distribution conditioned on shared contextual features, avoiding explicit modeling of the high-dimensional joint distribution. Let yx,ky_x,k and p^x,k p_x,k denote the discretized ground-truth bin index and the predicted categorical distribution for each state variable x∈p,v,bx∈\p,v,b\, corresponding to position, velocity, and size of region k, respectively. The model is trained end-to-end with the following loss: ℒ=1K∑k=1K[BCE(δk,δ^k)+δk∑x∈p,v,bCE(yx,k,p^x,k)]. = 1K _k=1^K [BCE( _k, δ_k)+ _k _x∈\p,v,b\CE\! (y_x,k, p_x,k ) ]. (1) I-C Collision Snapshot Determination We now determine a collision snapshot χ=(tcol,stcoladv,stcoltar)χ=(t_col,s_t_col^adv,s_t_col^tar) based on the estimated traffic agent states in the local regions. Unlike prior methods where collision time is implicitly determined as a byproduct of adversarial objectives, COSTER’s snapshot-first design enables explicit selection of tcolt_col via any suitable risk criterion. In this work, we adopt velocity entropy H as the selection criterion, motivated by its strong empirical correlation with collision risk [22, 23, 24, 25]. The regional traffic modeling described in Sec. I-B is applied independently at each timestep in S, thereby characterizing a single scene. In this section, we extend the notation to incorporate time, and denote the velocity distribution of region mkm_k at timestep t as p^v,k,t p_v,k,t. At each timestep, the set of local regions ttarN_t^tar within an elliptical neighborhood centered at the target vehicle, with longitudinal and lateral axis lengths ρlong _long and ρlat _lat, respectively, is selected as shown in Fig. 2 (b), and p^v,k,t p_v,k,t is estimated for each local region. We then compute HtH_t from the normalized state distribution p¯v,t p_v,t obtained by uniformly averaging the distributions p^v,k,t p_v,k,t over the |ttar||N_t^tar| regions. The velocity entropy at timestep t is then computed as Ht=−∑ip¯v,t(i)logp¯v,t(i),H_t=- _i p_v,t(i) p_v,t(i), where i indexes the discretized velocity bins. The timestep with the largest HtH_t is selected as tcolt_col, and the state of the corresponding vehicle at that timestep is taken as stcoltars_t_col^tar. Among the |tcoltar||N_t_col^tar| local regions around the target vehicle, we first identify the region with the highest Bernpla,k,tcolBern_pla,k,t_col, as described in Sec. I-B. Within the selected region, the heading, velocity, and size of the adversarial vehicle are sampled from the predicted categorical distributions. The adversarial vehicle candidate is then translated toward the target vehicle while maintaining a fixed relative heading, until the two vehicles are in contact. Since multiple center positions may satisfy the contact condition, we select the final placement based on the maximum-probability position under the predicted categorical position distribution. This determines stcoladvs_t_col^adv, completing χ. I-D Adversarial Trajectory Generation Based on χ, we reconstruct the adversarial vehicle trajectory prior to tcolt_col using a model trained on a large-scale traffic dataset to recover vehicle trajectories backward from a generic reverse endpoint trevt_rev to t1t_1, with trev=tcolt_rev=t_col at generation time. Motivated by the success of CVAEs in generating multimodal trajectories [26, 27], we adopt a CVAE-based framework for COSTER. The CVAE in COSTER takes χ, M, T and a latent variable as input, all expressed in coordinates relative to the vehicle being reconstructed. M is encoded into map embeddings, which serve as keys and values in cross-attention queried by the reconstructed vehicle state. The resulting representation is concatenated with the vehicle-state embedding to form the initial decoder input htrevh_t_rev. However, the context at trevt_rev alone is insufficient for accurate prediction of the subsequent trajectory. We therefore propose dynamic surrounding attention (DSA) in the CVAE decoder to update the surrounding-agent context at every timestep. Specifically, the model predicts the reverse displacement at timestep t using the decoder input representation ht+1h_t+1. Based on the predicted state, the surrounding-agent context is dynamically updated and incorporated into ht+1h_t+1 through an RNN. The surrounding-agent context is computed via a cross-attention module between the vehicle being reconstructed and its neighboring agents, where ht+1h_t+1 serves as the query, and the relative position and velocity differences between the vehicle and its neighbors serve as both keys and values. The resulting attention output is concatenated with the state of the vehicle being reconstructed at the corresponding time step to form the surrounding-agent context. Despite the limited information in χ, DSA enables efficient generation of multimodal, context-aware adversarial-vehicle trajectories, while retaining the computational advantage of sampling-based generation. Refer to Table I for the ablation and performance results of DSA. During training, reconstructs trajectories from trevt_rev to t1t_1. At each decoding step, a latent variable ztz_t is sampled from the posterior distribution, and the decoder, conditioned on both the decoder input representation ht+1h_t+1 and ztz_t, is trained to predict the displacement dtd_t. For posterior estimation, we additionally use gtg_t, an RNN-encoded summary of the ground-truth trajectory segment available only during training, enabling the model to better capture long-horizon strategic behaviors. The conditional prior pθp_θ, posterior qϕq_φ, and likelihood pξp_ξ are jointly optimized by maximizing the following evidence lower bound (ELBO), averaged over L=trev−t1L=t_rev-t_1: maxϕ,θ,ξℒ= _φ,θ,ξ\;L= i[1L∑t=t1trev−1qϕ[logpξ(dt∣zt,ht+1)⏟1st term _i [ 1L _t=t_1^t_rev-1E_q_φ [ p_ξ(d_t z_t,h_t+1)_1st term −logqϕ(zt∣gt,ht+1)pθ(zt∣ht+1)⏟2nd term]]. 40.00006pt- q_φ(z_t g_t,h_t+1)p_θ(z_t h_t+1)_2nd term ] ]. (2) The first term of Eq. (2) maximizes the reconstruction log-likelihood, while the second term regularizes the posterior toward the prior via KL divergence. During inference, the decoder utilizes ztz_t sampled from the prior distribution. The trajectories of adversarial vehicle are then oversampled from the CVAE and refined through two stages: (i) K-means clustering on lateral-displacement and arc-length features for representative selection, efficiently summarizing multimodal behavior while reducing sensitivity to outliers [27], and (i) projecting each state transition of the selected representatives onto its nearest motion token in a motion vocabulary constructed following [28] to ensure physical feasibility. IV Experiments IV-A Experiment Setting We train the models described in Sec. I-B and Sec. I-D using the Waymo Open Motion Dataset v1.1 (WOMD), which contains 9-second scenarios (530,000 scenes). The local region width is set to wr=5mw_r=5\,m. The longitudinal and lateral semi-axis lengths, ρlong _long and ρlat _lat, are set to twice the ego vehicle’s length and width, respectively. The trajectory generation horizon in Sec. I-D is adaptively determined based on tcolt_col. The original dataset, recorded at 10 Hz, is downsampled to 2 Hz, and the motion vocabulary is constructed accordingly. TABLE I: Quantitative Comparison on Past Trajectory Prediction Accuracy minADE (m) ↓ minFDE (m) ↓ avgADE (m) ↓ avgFDE (m) ↓ Inference Time (ms) ↓ Constant Velocity 1.695 4.961 1.695 4.961 - Constant Lane 14.860 17.442 14.860 17.442 - BMT [6] 0.417 1.022 0.838 2.190 536.2 COSTER (w/o DSA) 0.557 1.545 1.212 3.460 5.7 COSTER (w/ DSA) 0.259 0.828 0.570 1.966 39.4 TABLE I: MMD Comparison on Vehicle State Inference. Position ↓ Heading ↓ Speed ↓ Size ↓ Mean Value 0.2249 0.4237 0.1564 0.5451 LCTGen [21] 0.1466 0.1555 0.2286 0.1052 TrafficGen [20] 0.1367 0.1277 0.1704 0.0937 COSTER 0.1242 0.1116 0.1399 0.0764 IV-B Vehicle State Prediction Assessment We evaluate the implementation described in Sec. I-B against other baselines. For each method, we performed state inference on an identical set of 1,000 WOMD scenarios and measured the respective accuracy. Baseline. Since the component implemented in Sec. I-B has primarily been used as a submodule in prior frameworks, we adopt the corresponding modules from existing works as baselines. TrafficGen [20] predicts regional vehicle states using an MLP-based Gaussian mixture model (GMM) over position, heading, speed, and size. LCTGen [21] similarly adopts GMM-based modeling but incorporates interaction-aware agent features through self- and cross-attention. For a fair comparison, we remove the language conditioning from the original LCTGen and rely only on map and agent features. COSTER represents each state variable as a discretized categorical distribution over bins. We additionally include a Mean Value baseline that predicts the average vehicle states computed from the same WOMD scenarios. Evaluation metrics. We evaluate the predicted vehicle-state distributions using maximum mean discrepancy (MMD) [29], following prior traffic regional modeling studies [20, 21]. Lower MMD indicates a closer match to the ground-truth distribution. Analysis. As shown in Table I, the bin-based approach achieves consistently lower MMD than the GMM-based baselines across all state attributes. This suggests that discretized categorical modeling better captures complex vehicle-state distributions without imposing Gaussian assumptions or a fixed number of mixture components. In contrast, both TrafficGen and LCTGen are limited to five Gaussian components, which may limit their ability to represent irregular or highly multimodal distributions. Moreover, the discrete formulation enables the velocity entropy in Sec. I-C to be computed in closed form, avoiding the numerical approximation required for GMM entropy estimation. IV-C Time-Reversed Trajectory Generation Assessment We evaluate the time-reversed trajectory generation module by reconstructing past 5-second trajectories of existing vehicles in the WOMD. Baseline. We compare COSTER without DSA (COSTER (w/o DSA)) and with DSA (COSTER (w/ DSA)) against the following baselines: Constant Velocity: linearly extrapolates the vehicle state at trevt_rev backward assuming constant velocity. Constant Lane: extrapolates the vehicle backward along the nearest lane centerline at constant speed. BMT [6]: a bidirectional motion transformer for trajectory reconstruction. Evaluation metrics. We report (i) minADE, (i) minFDE [30], the minimum average and final displacement errors over 5 samples, (i) avgADE, (iv) avgFDE [31], the mean average and final displacement errors, and (v) Inference Time per trajectory. Analysis. As shown in Table I, COSTER (w/ DSA) achieves the lowest displacement errors across all metrics, confirming its advantage over existing methods in both overall trajectory accuracy and final position prediction. Ablating DSA confirms that dynamically updating surrounding-agent context at each decoding step is essential, particularly for reducing long-horizon drift as reflected in minFDE and avgFDE. Both COSTER variants also achieve significantly faster inference than BMT. IV-D Safety-Critical Scenario Assessment We evaluate the full scenario generation pipeline by inserting adversarial vehicles into WOMD. TABLE I: Quantitative Comparison of Generated Safety-Critical Scenarios. Data Efficiency Adversariality Plausibility Diversity SUR (%) ↑ CR (%) ↑ ORR (%) ↓ VDD ↓ ADD ↓ CSD (m/s) ↑ CPD (° ) ↑ CAT [14] 8.9 53.2 12.3 0.2775 0.2370 3.14 80.1 SEAL [17] 8.7 60.8 19.1 0.2828 0.2419 3.31 70.8 Adv-BMT [6] 61.2 100 62.3 0.2394 0.3019 3.37 29.9 COSTER (Ours) 74.1 100 10.2 0.2485 0.0983 4.75 86.2 Random + CVAE 63.5 100 51.8 0.2451 0.1147 3.42 31.6 Snapshot + BMT 71.8 100 19.4 0.2437 0.2864 4.68 84.1 Original ScenarioCAT [14]SEAL [17]Adv-BMT [6]COSTER Fig. 3: Qualitative comparison of generated safety-critical scenarios. Baseline. We conduct both quantitative and qualitative comparisons with the following baselines: CAT [14]: Perturbs existing traffic agents by selecting collision-prone trajectories to generate collision scenarios based on a simplified collision objective. SEAL [17]: Perturbs existing traffic agents using skill-based policies to generate collision. Adv-BMT [6]: Inserts new adversarial vehicles and employs a bidirectional motion transformer to reconstruct trajectories. For a fair comparison, we fix the target vehicle as the ego vehicle across all methods. Additional variants. COSTER and Adv-BMT differ in two components: the collision snapshot and the reverse rollout model. To disentangle them, we evaluate the two hybrid configurations. Random + CVAE replaces our snapshot construction with the ego-pose-offset initialization of Adv-BMT; Snapshot + BMT feeds our snapshot to the BMT with its native post-processing. TABLE IV: Comparison of Collision Time Determination. CSD (m/s) ↑ CPD (° ) ↑ Random timestep 4.12 70.2 Last timestep 4.29 73.3 Velocity entropy timestep 4.75 86.2 TABLE V: Downstream Evaluation of Policy Safety under Normal and Hard Scenarios WOMD-Normal WOMD-Hard [32] Training Data RCR ↑ CR ↓ ORR ↓ RCR ↑ CR ↓ ORR ↓ WOMD-Only 0.68±0.03 0.17±0.01 0.24±0.02 0.30±0.02 0.32±0.03 0.32±0.04 CAT [14] 0.67±0.05 0.18±0.01 0.26±0.02 0.29±0.04 0.33±0.03 0.34±0.01 SEAL [17] 0.69±0.03 0.16±0.01 0.26±0.02 0.35±0.01 0.27±0.02 0.30±0.03 Adv-BMT [6] 0.64±0.07 0.20±0.02 0.29±0.02 0.26±0.02 0.38±0.04 0.36±0.01 COSTER (w/o Aug) 0.71±0.06 0.15±0.03 0.23±0.04 0.38±0.04 0.24±0.01 0.31±0.02 COSTER (w/ Aug) 0.73±0.04 0.15±0.01 0.22±0.02 0.41±0.02 0.22±0.02 0.30±0.03 Evaluation metrics. We adopt seven evaluation metrics based on prior studies [13, 33]. (i) Scenario utilization rate (SUR), which measures the proportion of raw scenarios from which a collision scenario can be generated. (i) Collision rate (CR), which measures the probability of a collision between the target and the adversarial vehicle. (i) Collision point diversity (CPD), which measures the standard deviation of normalized collision points in the ego-centric coordinate frame. (iv) Collision speed diversity (CSD), which measures the standard deviation of relative collision speeds at the moment of impact across scenarios. (v) Off road rate (ORR), which measures the proportion of trajectories in which the adversarial vehicle crosses the road boundary. (vi) Velocity distribution divergence (VDD) and (vii) acceleration distribution divergence (ADD), which measure the Jensen–Shannon divergence between the velocity and acceleration distributions of trajectories from raw scenarios and generated trajectories, respectively. Quantitative analysis. As shown in Table I, COSTER achieves the highest SUR while matching the best CR. CAT and SEAL show low SUR due to their reliance on perturbing existing vehicles, limiting generation to scenarios that satisfy specific conditions. While Adv-BMT attains comparable SUR and CR, it relies on randomly initialized collision snapshots. As a result, the trajectory generation module trained on nominal data is frequently initialized in implausible or out-of-distribution states, leading to degraded realism, as evidenced by its high ORR. Similar limitations are observed in diversity. CAT and SEAL are inherently constrained by raw scenario configuration, while Adv-BMT uses random initialization constructed via ego-pose offsets, which yields limited variations in collision configurations. In contrast, COSTER achieves the highest CPD and CSD, indicating substantially richer diversity in both collision location and dynamics, while maintaining strong plausibility with the lowest ORR and ADD. These results suggest that constructing probabilistically grounded collision snapshots, followed by generative rollout, provides an effective approach for synthesizing safety-critical scenarios as out-of-distribution events from nominal data. The hybrid variants further clarify the respective roles of collision initialization and trajectory rollout. Although BMT demonstrates strong trajectory reconstruction capability under nominal conditions, as shown in Table I, Random + CVAE still exhibits substantially degraded plausibility, particularly in terms of ORR. This suggests that random collision initialization can expose the rollout model to implausible or out-of-distribution initial conditions, leading to unrealistic trajectories. In contrast, Snapshot + BMT markedly alleviates these failures, highlighting the importance of context-aware collision initialization, while the full COSTER further improves trajectory plausibility through its CVAE-based rollout. This failure mode is also qualitatively evident in Fig. 3, where the randomly initialized Adv-BMT produces implausible trajectories. Qualitative analysis. Figure 3 compares safety-critical scenarios generated from the same original scenario. In (a) and (c), CAT and SEAL produce perturbed trajectories that leave the map, indicating that the adversarial objective based method often fails to account for diverse and context-aware situations. In (b), CAT fails to induce a collision under the given raw scenario configuration, while SEAL generates unrealistic behavior by failing to respond properly to target vehicle motion. Adv-BMT also shows some limitations. In (a) and (b), implausible collision snapshots from random initialization cause the generative model to fail, as discussed in the quantitative analysis section. In (c), it produces scenarios that do not adequately consider surrounding vehicles. In contrast, COSTER generates multimodal and plausible collision scenarios that account for interactions with surrounding agents. Beyond the five representative trajectories, the distribution of remaining generated trajectories is visualized as a heatmap using Gaussian kernel density estimation. IV-E Analysis on Velocity Entropy-based Selection We conduct an additional ablation study to more precisely isolate the impact of the collision time selection method. Baseline. We compare three collision time determination strategies: Random timestep, which randomly samples a timestep from the base scenario, Last timestep, which fixes the collision time to the last timestep, and Velocity entropy timestep, which selects the timestep with the highest velocity entropy. For each strategy, safety-critical scenarios are generated using the COSTER pipeline from the same set of 1,000 WOMD scenarios. Evaluation metrics. Among the metrics used in Sec. IV-D, we focus on CPD and CSD, as these are most directly influenced by the choice of collision time. Metrics that are largely insensitive to collision time determination (e.g., SUR, CR) are not reported for this experiment. Quantitative analysis. As shown in Table IV, both the random timestep and last timestep strategies yield lower CPD and CSD compared to velocity entropy-based selection, indicating more limited collision configuration diversity. This result suggests that velocity entropy-based timestep selection more effectively identifies regions of complex traffic interaction, where a wider variety of candidate adversarial vehicle states are available. The richer candidate pool in such regions naturally translates into a broader range of collision types in terms of both impact location and relative speed. IV-F Downstream Evaluation on Policy Safety We evaluate whether scenarios generated by COSTER lead to greater improvements in downstream task performance compared to those generated by CAT, SEAL, and Adv-BMT. To this end, RL-based traffic agents are trained in closed-loop simulation using MetaDrive [34] with 1,000 scenarios produced by each method. We follow the RL training configuration of Liu et al. [6]. COSTER generates multi-modal trajectories from a single collision snapshot. We consider two variants: COSTER (w/ Aug), where 1000 trajectories are oversampled and five clustered trajectories are used for training and COSTER (w/o Aug), where a single representative trajectory is selected from the five trajectories. Additionally, we introduce a control baseline, WOMD-Only, where the agent is trained on 1000 unmodified WOMD scenarios without any adversarial perturbations. We evaluate under two settings. (i) WOMD-Normal: A generalization setting with 200 randomly sampled scenarios from the WOMD not used in training, to assess potential performance degradation in normal conditions. (i) WOMD-Hard: A safety-critical setting consisting of scenarios selected using SafeShift [32] hierarchical scoring (top 20 percentile) from the WOMD, as used in previous work [17], to measure performance in high-risk environments. As evaluation metrics, we use route completion rate (RCR), defined as the proportion of scenarios in which the trained agent successfully completes the route without collision or off-road deviation, along with CR and ORR. Analysis. As shown in Table V, COSTER consistently outperforms prior methods across both evaluation sets. On WOMD-Normal, COSTER achieves the highest RCR while maintaining low CR and ORR, indicating that it preserves performance in standard scenarios without degradation. More importantly, on the more challenging WOMD-Hard split, COSTER substantially improves RCR, outperforming baseline methods while also achieving the lowest CR. Notably, COSTER (w/ Aug) demonstrates superior performance compared to COSTER (w/o Aug) in SafeShift scenarios. Overall, these results suggest that the scenarios generated by COSTER not only contribute effectively to training compared to baseline methods, but also facilitate more efficient learning by providing diverse variants of safety-critical scenarios derived from a single raw scenario. V Conclusion We presented COSTER, a safety-critical scenario generation framework that constructs collision time and configuration from learned traffic priors and generates adversarial trajectories via time-reversed CVAE rollout. By controlling collision configurations, COSTER improves plausibility, diversity, data efficiency, and downstream driving safety. Despite these benefits, COSTER is currently limited to vehicle-to-vehicle collisions and may overlook hazardous low-entropy scenarios. Future work will extend the framework to diverse road users and incorporate richer semantic priors, such as vision–language models. References [1] H. X. Liu and S. Feng (2024) Curse of rarity for autonomous vehicles. nature communications 15 (1), p. 4808. Cited by: §I. [2] Z. Wang, S. Lan, X. Sun, N. Chang, Z. Li, Z. Yu, and J. M. Alvarez (2025) Enhancing autonomous driving safety with collision scenario integration. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 10116–10123. Cited by: §I. [3] Z. Ghodsi, S. K. S. Hari, I. Frosio, T. Tsai, A. Troccoli, S. W. Keckler, S. Garg, and A. Anandkumar (2021) Generating and characterizing scenarios for safety testing of autonomous vehicles. In 2021 IEEE Intelligent Vehicles Symposium (IV), p. 157–164. Cited by: §I. [4] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao (2023) A survey on safety-critical driving scenario generation—a methodological perspective. IEEE Transactions on Intelligent Transportation Systems 24 (7), p. 6971–6988. Cited by: §I. [5] K. Chen, Y. Lei, H. Cheng, H. Wu, W. Sun, and S. Zheng (2024) FREA: feasibility-guided generation of safety-critical scenarios with reasonable adversariality. In 8th Annual Conference on Robot Learning (CoRL), Cited by: §I. [6] Y. Liu, Z. M. Peng, X. Cui, and B. Zhou (2026) Adv-bmt: bidirectional motion transformer for safety-critical traffic scenario generation. Advances in Neural Information Processing Systems 38, p. 55310–55335. Cited by: §I, §I, Fig. 3, §IV-C, §IV-D, §IV-F, TABLE I, TABLE I, TABLE V. [7] K. Sohn, H. Lee, and X. Yan (2015) Learning structured output representation using deep conditional generative models. Advances in neural information processing systems 28. Cited by: §I. [8] M. O’Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi (2018) Scalable end-to-end autonomous vehicle testing via rare-event simulation. Advances in neural information processing systems 31. Cited by: §I. [9] M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer (2018) Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), p. 1–7. Cited by: §I. [10] J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun (2021) Advsim: generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9909–9918. Cited by: §I. [11] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger (2022) King: generating safety-critical driving scenarios for robust imitation via kinematics gradients. In European Conference on Computer Vision, p. 335–352. Cited by: §I. [12] Y. Cao, C. Xiao, A. Anandkumar, D. Xu, and M. Pavone (2022) Advdo: realistic adversarial attacks for trajectory prediction. In European Conference on Computer Vision, p. 36–52. Cited by: §I. [13] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany (2022) Generating useful accident-prone driving scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17305–17315. Cited by: §I, §IV-D. [14] L. Zhang, Z. Peng, Q. Li, and B. Zhou (2023) CAT: closed-loop adversarial training for safe end-to-end driving. In 7th Annual Conference on Robot Learning (CoRL), Cited by: §I, Fig. 3, §IV-D, TABLE I, TABLE V. [15] P. Huang, W. Ding, B. Stoler, J. Francis, B. Chen, and D. Zhao (2025) Cadre: controllable and diverse generation of safety-critical driving scenarios using real-world trajectories. In 2025 IEEE international conference on robotics and automation (ICRA), Cited by: §I. [16] J. Ransiek, J. Plaum, J. Langner, and E. Sax (2024) Goose: goal-conditioned reinforcement learning for safety-critical scenario generation. In 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), p. 2651–2658. Cited by: §I. [17] B. Stoler, I. Navarro, J. Francis, and J. Oh (2025) Seal: towards safe autonomous driving via skill-enabled adversary learning for closed-loop scenario generation. IEEE Robotics and Automation Letters (99), p. 1–8. Cited by: §I, Fig. 3, §IV-D, §IV-F, TABLE I, TABLE V. [18] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cornman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. (2022) Multipath++: efficient information fusion and trajectory aggregation for behavior prediction. In 2022 International Conference on Robotics and Automation (ICRA), p. 7814–7821. Cited by: §I-B. [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §I-B. [20] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou (2023) Trafficgen: learning to generate diverse and realistic traffic scenarios. In 2023 IEEE international conference on robotics and automation (ICRA), p. 3567–3575. Cited by: §I-B, §IV-B, §IV-B, TABLE I. [21] S. Tan, B. Ivanovic, X. Weng, M. Pavone, and P. Kraehenbuehl (2023) Language conditioned traffic generation. In Conference on Robot Learning, p. 2714–2752. Cited by: §I-B, §IV-B, §IV-B, TABLE I. [22] Z. Liu, Y. Wang, Q. Cheng, and H. Yang (2022) Analysis of the information entropy on traffic flows. IEEE Transactions on Intelligent Transportation Systems 23 (10), p. 18012–18023. Cited by: §I-C. [23] W. Ye, Y. Xu, X. Shi, N. Shiwakoti, Z. Ye, and Y. Zheng (2024) A macroscopic safety indicator for road segment: application of entropy theory. Physica A: Statistical Mechanics and its Applications 642, p. 129787. Cited by: §I-C. [24] Z. Zheng, S. Ahn, and C. M. Monsere (2010) Impact of traffic oscillations on freeway crash occurrences. Accident Analysis & Prevention 42 (2), p. 626–636. Cited by: §I-C. [25] C. Xu, P. Liu, W. Wang, and Z. Li (2012) Evaluation of the impacts of traffic states on crash risks on freeways. Accident Analysis & Prevention 47, p. 162–171. Cited by: §I-C. [26] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, p. 683–700. Cited by: §I-D. [27] P. Xu, J. Hayet, and I. Karamouzas (2022) Socialvae: human trajectory prediction using timewise latents. In European Conference on Computer Vision, p. 511–528. Cited by: §I-D, §I-D. [28] J. Philion, X. B. Peng, and S. Fidler (2024) Trajeglish: traffic modeling as next-token prediction. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §I-D. [29] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola (2012) A kernel two-sample test. The journal of machine learning research 13 (1), p. 723–773. Cited by: §IV-B. [30] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, et al. (2021) Tnt: target-driven trajectory prediction. In Conference on robot learning, p. 895–904. Cited by: §IV-C. [31] D. Zhu, M. Zahran, L. E. Li, and M. Elhoseiny (2022) Motion forecasting with unlikelihood training in continuous space. In Conference on Robot Learning, p. 1003–1012. Cited by: §IV-C. [32] B. Stoler, I. Navarro, M. Jana, S. Hwang, J. Francis, and J. Oh (2024) Safeshift: safety-informed distribution shifts for robust trajectory prediction in autonomous driving. In 2024 IEEE Intelligent Vehicles Symposium (IV), p. 1179–1186. Cited by: §IV-F, TABLE V. [33] Z. Huang, X. Gao, G. Zheng, L. Wen, X. Yang, and X. Sun (2025) Safety-critical traffic simulation with adversarial transfer of driving intentions. In 2025 IEEE international conference on robotics and automation (ICRA), Cited by: §IV-D. [34] Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou (2022) Metadrive: composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence 45 (3), p. 3461–3475. Cited by: §IV-F.