Paper deep dive
Top-down Traffic Scenario Generation via Joint Initial-Goal Diffusion and Trajectory Infilling
Da Saem Lee, Yash Vardhan Pant, Sebastian Fischmeister
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/13/2026, 3:38:02 AM
Summary
The paper introduces TrafficDiffuser, a top-down traffic scenario generation framework that jointly models initial and goal states using diffusion models to create interpretable high-level scenarios. This approach simplifies trajectory generation into a goal-conditioned infilling problem. Evaluated on the Argoverse 2 dataset, TrafficDiffuser outperforms existing methods in agent initialization, significantly reducing speed distribution distance and off-road rates.
Entities (8)
Relation Signals (7)
TrafficDiffuser → evaluatedon → Argoverse 2
confidence 95% · We conduct extensive experiments on the Argoverse 2 motion prediction dataset to evaluate how well the generated outputs capture real-world distributions.
TrafficDiffuser → uses → DDPM
confidence 95% · As in the agent initialization model in PathDiffuser [8], we adopt the differential transformer... To learn the real-world distribution, we leverage a DDPM [5] formulation
TrafficDiffuser → outperforms → PathDiffuser
confidence 90% · Table I shows TrafficDiffuser has lower JSD and collision rates compared to PD-init (PathDiffuser initialization).
TrafficDiffuser → outperforms → SceneControl
confidence 90% · In addition to generating goal states, TrafficDiffuser outperforms the next-best approach for agent initialization... SceneControl [10] leverages diffusion models...
SceneControl → uses → DDPM
confidence 90% · SceneControl [10] leverages diffusion models with transformer decoder layers to generate initial states...
Ctrl-Sim → runson → Nocturne
confidence 85% · The real-world driving logs are replayed on the Nocturne simulator [18] to generate an offline RL dataset... Ctrl-Sim [12] introduces a simulator...
TrafficDiffuser → usescomponent → QCNet
confidence 85% · To efficiently encode the vectorized map structure, we adopt and modify the map encoder from QCNet [23].
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robust traffic simulators are crucial for developing and testing autonomous vehicles to reduce the costly, labor-intensive real-world data collection process and the need for physical presence on the road. However, existing simulators require agents' initial states to generate trajectories, which limits scalability and diversity due to restrictions on the given initial states. While data-driven agent initialization has been widely studied, the generated initial states are not interpretable in terms of why the agents are initialized at those specific locations. Given known initial states, trajectory generation is also a challenging problem, as the model must learn the variability of the destination and how agents should reach it over time. In this paper, we propose TrafficDiffuser, a top-down traffic scenario generation framework that generates high-level traffic scenarios, defined by initial and goal state pairs, by jointly modeling them. The high-level scenario generation makes initial states better interpretable and reduces trajectory generation into as simple as an infilling problem. We demonstrate how the generated high-level traffic scenarios can be used, including constraining based on different trajectory modes and integrating them with existing trajectory generation models. We conduct extensive experiments on the Argoverse 2 motion prediction dataset to evaluate how well the generated outputs capture real-world distributions. In addition to generating goal states, TrafficDiffuser outperforms the next-best approach for agent initialization, reducing speed distribution distance by 55.3% and the off-road rate by 2.8%.
Tags
Links
- Source: https://arxiv.org/abs/2608.11407v1
- Canonical: https://arxiv.org/abs/2608.11407v1
Trouble viewing inline? Open PDF directly →
Full Text
40,022 characters extracted from source content.
Expand or collapse full text
Top-down Traffic Scenario Generation via Joint Initial-Goal Diffusion and Trajectory Infilling Da Saem Lee Yash Vardhan Pant Sebastian Fischmeister Thanks: This work is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), Canada Foundation for Innovation - John R. Evans Leaders Fund (CFI JELF), Mitacs, and Intact Financial Corporation. Thanks: The authors are with the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, Canada. ds3lee@uwaterloo.ca, yash.pant@uwaterloo.ca, sfischme@uwaterloo.ca Abstract Robust traffic simulators are crucial for developing and testing autonomous vehicles to reduce the costly, labor-intensive real-world data collection process and the need for physical presence on the road. However, existing simulators require agents’ initial states to generate trajectories, which limits scalability and diversity due to restrictions on the given initial states. While data-driven agent initialization has been widely studied, the generated initial states are not interpretable in terms of why the agents are initialized at those specific locations. Given known initial states, trajectory generation is also a challenging problem, as the model must learn the variability of the destination and how agents should reach it over time. In this paper, we propose TrafficDiffuser, a top-down traffic scenario generation framework that generates high-level traffic scenarios, defined by initial and goal state pairs, by jointly modeling them. The high-level scenario generation makes initial states better interpretable and reduces trajectory generation into as simple as an infilling problem. We demonstrate how the generated high-level traffic scenarios can be used, including constraining based on different trajectory modes and integrating them with existing trajectory generation models. We conduct extensive experiments on the Argoverse 2 motion prediction dataset to evaluate how well the generated outputs capture real-world distributions. In addition to generating goal states, TrafficDiffuser outperforms the next-best approach for agent initialization, reducing speed distribution distance by 55.3% and the off-road rate by 2.8%. 11 1 https://github.com/CL2-UWaterloo/TrafficDiffuser I Introduction When developing the autonomous system, exposure to diverse traffic scenarios is crucial to ensure safety and adaptability. However, obtaining and manually annotating real-world datasets that capture such diversity is both time-consuming and costly [20]. To mitigate this, data-driven simulators [6, 14] focuses on generating synthetic traffic scenarios to create data samples. Typically, this process is decoupled into two tasks: generating initial agent states [10, 17] and then synthesizing the corresponding trajectories [6]. For multi-agent scenarios, generating a realistic set of initial agent states is important for diversifying the scenario. While prior data-driven approaches [10, 17] model the initial states and interactions conditioned on the map, they often lack interpretability, as it is difficult to understand the underlying intent of the generated states. By treating initialization as a task to capture a single timestep of scenarios, these models provide limited context to trajectory generation. With the given initial traffic scene, trajectory generation has been studied to complete the scenario generation process [6, 12]. Effective trajectory generation requires the model to capture agents’ intentions and behaviors. In trajectory prediction tasks, given the initial states and past trajectories, prior works [4, 22] focus on predicting goal points and motions between the initial and predicted goal points to improve prediction accuracy in unseen map structures. Fig. 1: Illustration of proposed approach. (a) Generated high-level traffic scenario. (b) Infilled trajectories conditioned on the high-level scenario. In this paper, we introduce TrafficDiffuser, a diffusion-based framework that generates traffic scenarios in a top-down, goal-conditioned manner. As illustrated in Fig. 1(a), the initial and goal states are jointly modeled to generate high-level scenarios which allows inherently learning feasible paths between endpoints. This high-level scenario simplifies the complex trajectory generation into a goal-conditioned infilling task as shown in Fig. 1(b). Fig. 2 provides an overview of the proposed framework, which integrates high-level scenario generation with trajectory infilling. The contributions of our approaches are as follows: Fig. 2: Overview of TrafficDiffuser. In A, a noisy scenario state, map summary, and map are processed by the TrafficGenerator ϵθ _θ to produce a denoised high-level scenario ^0 s^0, which consists of initial and goal states. In B, the denoised scenario is then passed to infiller fϕf_φ to connect between the generated initial and goal states with kinematically feasible trajectories τ. • Interpretability: Jointly generated start-goal pair provides better interpretability of the initial traffic scenes and agent intentions. • Feasible path generation: The model synthesizes feasible start-goal configurations for multi-agent scenarios. • Simplified Trajectory Generation: Leveraging high-level traffic scenarios enables simplified trajectory generation through goal-conditioned infilling. The remainder of the paper is organized as follows. Section I reviews prior works, Section I introduces notation, problem setup, and preliminaries, Section IV details our approach, and Section V presents experimental results and comparisons. Finally, Section VI concludes the paper and discusses future work. I Related Works In this section, we review existing works for traffic scenario generation. We specifically focus on data-driven approaches that initialize agents in the scene, predict or generate their future trajectories, and end-to-end approaches. Earlier works [1, 16] represented traffic scenarios using rasterized images, by encoding individual features in each channel. To improve computational efficiency by eliminating redundancy, recent works employ vectorized representations of the map structure [9, 3]. SceneControl [10] leverages diffusion models with transformer decoder layers to generate initial states of the agents in the scene. While SceneControl demonstrates controllable agent initialization on diverse scenarios, it requires guidance sampling [2] to enforce realism, such as collision avoidance and road compliance. With known initial states, trajectory prediction or generation tasks are widely studied using vectorized map representations. For trajectory generation, Ctrl-Sim [12] introduces a simulator trained using offline reinforcement learning (RL). The real-world driving logs are replayed on the Nocturne simulator [18] to generate an offline RL dataset. In the trajectory prediction domain, Trajeglish [11] formulates trajectory prediction as a next token prediction problem using predefined motion tokens. To enhance generalizability, Holigraph Goal [4] proposes a goal-prediction and goal-based trajectory-prediction approach that leverages the lane structure. While these approaches predict or generate feasible trajectories, they require past trajectories and initial states, which makes it difficult to generate or predict the trajectories. To further enhance diversity, generating both agent initialization and trajectories has been widely explored. PathDiffuser [8] proposes a two-stage diffusion-based approach for traffic scenario generation. For agent initial scene generation, the differential transformer [21] is used to narrow down the attention to the map components. For trajectory generation, candidate trajectories are defined in the Frenet coordinate system while reducing reliance on past motions and improving map compliance in adversarial scenarios. ScenarioDreamer [14] also proposes a two-stage approach to traffic scenario generation. The scene initialization employs a latent diffusion model that uses latent embeddings captured from a VAE model to represent lane segments and agents’ states. Then, the closed-loop simulator from Ctrl-Sim [12] is leveraged for trajectory generation. SceneDiffuser [6] utilizes v-prediction [15] to train the diffusion model by using a control mask for controllable generation in both scenario generation and behavior prediction. By using the proposed amortized diffusion, a single denoising function generates future trajectory predictions by refining buffered predictions from previous steps. While these approaches enable the traffic scenario generation, agents’ high-level intent is not modeled during initialization, which limits the interpretability of the initial states of the resulting traffic scenario. Most existing works generate diverse traffic scenarios by factorizing the problem into two stages: generating initial states and then generating trajectories. In contrast, we propose a top-down diffusion-based approach that first generates high-level scenarios, then infills trajectories. I Problem Statements and Preliminaries This section presents the mathematical notations, the problem formulation, and a brief overview of the diffusion process underlying our approach. I-A Notation The vectorized map consists of road elements, including map polygons and map points, whose embeddings are denoted by ℳM. We consider a scene with N agents and a time horizon H, with intermediate timestep h∈0,1,…,Hh∈\0,1,…,H\. The state of agent i at a given timestep is defined as ai=[xi,yi,vi,ci]∈ℝ4a_i=[x_i,y_i,v_i,c_i] ^4, where (xi,yi)∈ℝ2(x_i,y_i) ^2 denotes position, vi∈ℝ+v_i ^+ the speed, and ci∈ℤ+c_i ^+ the agent type. The joint state of all agents in the scene at time step h is h=[a1,…,aN]∈ℝN×4s_h=[a_1,…,a_N] ^N× 4. The high-level traffic scenarios are denoted by HL=[0,H]s_HL=[s_0,s_H]. We denote a positional trajectory of agent i over the time as τi=[(x,y)i,1,…,(x,y)i,H] _i=[(x,y)_i,1,…,(x,y)_i,H], trajectories for all agents as =[τ1,…,τN] τ=[ _1,…, _N], and trajectory at time h as h τ_h. The real-world distribution is denoted as q(.)q(.), and the learned distribution is denoted as pθ(.)p_θ(.). We denote generated states and predicted trajectories using the hat notation, e.g., τ and s. I-B Problem Statement Given a map ℳM and the number of agents N, our goal is to develop a generative model that approximates the distribution of traffic scenarios from a real-world dataset q(|ℳ)q( τ|M), so that the sampled data from learned distribution pθ(|ℳ)p_θ( τ|M) resembles the real-world data. Unlike existing approaches, we propose decomposing the problem using the concept of a high-level traffic scenario. We define a high-level traffic scenario HL=[0,H]s_HL=[s_0,s_H], which consists of the initial states 0s_0 and the goal states Hs_H for all N agents in the map. Using this concept, the problem is decomposed as follows: Problem 1 (High-level Traffic Scenario Generation) Given a map ℳM and the number of agents N, develop a generative model which enables sampling high-level scenarios HLs_HL from pθ(HL|ℳ)p_θ(s_HL|M) that resembles samples from the true distribution q(HL|ℳ)q(s_HL|M). Problem 2 (Trajectory Infilling) Given a map ℳM and high-level scenarios HLs_HL, develop a model for goal-conditioned trajectory prediction that infills trajectories τ between the sampled initial 0s_0 and goal states Hs_H. I-C Diffusion Model Among existing generative models, denoising diffusion probabilistic models (DDPM) [5] have shown promising results across a variety of domains. DDPM corrupts data with Gaussian noise in a forward pass based on the schedule, and learns to iteratively reverse this process to recover the samples from the original distribution. Forward Process. The distribution of the agent’s initial and goal states is denoted by q(HL)q(s_HL). Over the diffusion steps T, Gaussian noise is iteratively added as follows: q(HLt|HLt−1,ℳ):=(HLt,1−βtHLt−1,βt)q(s_HL^t|s_HL^t-1,M):=N (s_HL^t; 1- _t\,s_HL^t-1,\, _t\,I ) where t∈0,1,…,Tt∈\0,1,…,T\ is the diffusion steps, βt _t is the predefined variance schedule at step t. Using the Markov property, samples at any step t can be written as: q(HLt|HL0)=(HLt,α¯tHL0,(1−α¯t))q(s^t_HL|s_HL^0)=N (s^t_HL; α_t\,s_HL^0,(1- α_t)\,I ) in which αt=1−βt _t=1- _t represents the variance preserving term and α¯t=∏s=1tαs α_t= _s=1^t _s controls the noise schedule. Reverse Process. In the reverse process pθ(HL0:T)p_θ(s_HL^0:T), noisy data is initially drawn from the prior distribution pθ(HLT)=(HLT,,)p_θ(s_HL^T)=N(s^T_HL;0,I). Then, the noisy sample is gradually denoised to recover a noise-free sample, which can be considered as the transition from HLts_HL^t to HLt−1s_HL^t-1 and expressed as follows: pθ(HLt−1|HLt)=(HLt−1,θ(HLt,t),σt2)p_θ(s_HL^t-1|s_HL^t)=N(s_HL^t-1; μ_θ(s_HL^t,t), _t^2I) where σt2=βt1−α¯t−11−α¯t _t^2= _t 1- α_t-11- α_t. Moreover, estimating added noise ϵε is mathematically equivalent to estimating the clean data HLs_HL but offers better stability. Guidance Sampling. The reverse process can be guided to steer denoising towards the desired output. While some approaches introduce an additional classifier network to approximate the guidance [2], they require additional training and inference costs. To reduce the computational cost, some works [10, 19, 7] define the guidance cost function (x0)J(x^0) to reflect the desired characteristics, where x0x^0 is the clean data. Then, during the reverse process, the denoised data x^0(xt) x^0(x_t) is reconstructed by estimating the noise. By taking the gradient ∇xt(x^0(xt)) _x^tJ( x^0(x_t)), it is injected to bias the denoising process. For instance, as in SceneControl [10], during the reverse process, we can steer the agent’s location by biasing with the gradient of the guidance cost function. In our task, we apply this principle to guide HLs_HL by perturbing the noise prediction as in Eq. 1. ϵ^θ(HLt,ℳ)←ϵθ(HLt,ℳ)−λguide∇HLt(^HL0), ε_θ(s_HL^t,M)← ε_θ(s_HL^t,M)- _guide _s^t_HLJ( s^0_HL), (1) where λguide∈ℝ+ _guide ^+ is the guidance scale. IV TrafficDiffuser: Approach Building upon the preliminaries and notations in Section I, we propose a hierarchical diffusion framework for top-down traffic scenario generation. IV-A High-Level Traffic Scenario Generation As formulated in Problem 1, we first aim to generate the high-level traffic scenarios that resemble the true distribution by jointly modeling the initial and goal states. To learn the real-world distribution, we leverage a DDPM [5] formulation, and the reverse process ϵθ(0,Ht,ℳ,t) ε_θ(s^t_0,H,M,t) is parameterized to estimate noise at each diffusion step t, given the map ℳM and the number of agents N. IV-A1 Architecture Based on the DDPM formulation, we now describe the detailed architecture of our framework. As illustrated in Fig. 2A, the high-level scenario generation model consists of a map encoder and a traffic generator. Map Encoder To efficiently encode the vectorized map structure, we adopt and modify the map encoder from QCNet [23]. Since the goal is to generate the initial states on the map, the map point and map polygon nodes are encoded with their positions and orientations. Then, as illustrated in Fig. 3, map polygon nodes attend map point nodes to learn how the lane is internally structured, and other polygons to learn how they are connected. Moreover, we introduce a map summary layer to summarize the map by attending to encoded map polygons. Fig. 3: Map encoder architecture. Map polygons are encoded using a heterogeneous message passing (HMP) layer that attends to corresponding map points and surrounding map polygons. Then, the learnable embedding ℳsumM_sum attends the encoded map polygons to summarize the map structure. Traffic Generator When generating realistic high-level traffic scenarios, the model needs to generate a valid path for each agent while not colliding with other agents. As in the agent initialization model in PathDiffuser [8], we adopt the differential transformer [21] to promote the sparse map attention. Unlike PathDiffuser [8] and SceneControl [10], our model attends to map polygon embeddings to reduce the number of attention operations to learn the map structure. An overview of the traffic generator is shown in Fig 2A. To encode the high-level scenario, we consider the problem as encoding a sequence of length two, i.e., initial and goal states. Each states are independently encoded with time and agent identity embeddings to maintain temporal and social distinction during the attention operations. The denoising process follows a sequence of attention layers to fuse multi-modal constraints. First, noisy samples attend to a map summary to provide a global spatial context of the given map. This is followed by temporal self-attention to ensure consistency between an agent’s own initial and goal states. Then, the sample attends to other agents through a social cross-attention layer to capture the interaction among agents. Lastly, it attends the local map embeddings to condition on the geometry of the given map structure. IV-A2 Loss The training objective is to maximize the Variational Lower Bound, but, as shown in [5], it can be reduced to predicting noise. By reparameterizing the objective for our high-level scenario HLs_HL generation, the training objective reduces to the following loss function: ℒℍ(θ)=[‖ϵ−ϵθ(HLt,ℳ,t)‖2]L_HL(θ)=E [ \| ε- ε_θ (s^t_HL,M,t ) \|^2 ] (2) To further promote sparser map attention, an additional loss term is introduced to minimize the entropy of the attention weights, as in Eq. 3. As the differential transformer [21] can output negative attention weights, the attention weights are normalized for entropy computation. ℒent(θ)=[−w~ilog(w~i)]L_ent(θ)=E [- w_i ( w_i) ] (3) where w~i=|wi,j|‖wi‖2 w_i= |w_i,j|||w_i||_2 represents the normalized attention weights of map token j for agent i, wi,jw_i,j denotes the attention weight between agent i and map token j, |.||.| denotes element-wise absolute value, and ||.||2||.||_2 denotes l2-norm of attention weights for agent i across all map tokens. By combining the Eq. 2 and Eq. 3, the model is trained using the following loss function: ℒ(θ)=ℒℍ(θ)+λentℒent(θ),L(θ)=L_HL(θ)+ _entL_ent(θ), where λent∈ℝ+ _ent ^+ is a hyperparameter. IV-B Trajectory Infilling In existing trajectory prediction or generation problem formulations, the model must capture how the agent’s trajectory rolls out from the initial states or the past trajectories, which requires learning the agent’s destination and its behavior along the way. However, with the generated high-level scenarios, the trajectory generation problem can be simplified into a goal-conditioned infilling problem, and the model only needs to know how the agent travels to the goal states. IV-B1 Architecture Given the high-level traffic scenarios, we propose using a line between them as a bridge and designing a model in Fig. 2B to predict the distance from the bridge over time, as illustrated in Fig. 4. Fig. 4: Illustration of Trajectory Infilling. Given a “bridge” (see Sec. IV-B1) between the initial and goal states, a trajectory is produced by refining the predicted distance from the bridge on each time step. As Algorithm 1 describes, to introduce more flexibility to our approach, the infilling model uses a unified architecture that accommodates different constraint settings within a single framework. Specifically, we consider the following cases: (i) hard constraints (HC) on both initial and goal states, (i) HC on initial states only, (i) HC on goal states only, and (iv) no HC. Hard constraints set the predicted distance values to zero at the corresponding timestep. Given the initial and goal states, the bridge ℬB is constructed between them and treated as an anchor. Based on the randomly sampled training mode m, the model learns to handle different modes while predicting the distance to the bridge ℬB. After the initial distance estimation, the trajectory is refined by attending to the map and other agents. The weights γ are multiplied to apply the constraint depending on the sampled mode. Algorithm 1 Trajectory Infilling 1: map ℳM, high-level scenario HLs_HL, time horizon H, mode m∼HCboth,HCinit,HCgoal,HCnonem \HC_both,HC_init,HC_goal,HC_none\ 2: ←[i−1H−1]i=1Hh←[ i-1H-1]_i=1^H ⊳ Linearly spaced time vector in [0,1][0,1] 3: ℬ←(H0−00)B←(s^0_H-s^0_0)h ⊳ Create evenly spaced bridge 4: ←(1−)if m=HCboth1−if m=HCinitif m=HCgoal1otherwise γ← casesh(1-h)&if m=HC_both\\ 1-h&if m=HC_init\\ h&if m=HC_goal\\ 1&otherwise cases⊳ Mode-based weight 5: ^←ℬ+fϕ(ℳ,ℬ,m) τ + γf_φ(M,B,m) 6: return τ Fig. 5: Diversity in Generated High-level Traffic Scenario. Each row contains an identical map, and the number of agents, and each column shows different scenarios. Blue dots indicate the initial positions, and red stars indicate the final positions. The line connecting the two points denotes an initial-goal pair. Generated high-level scenarios show reachable pairs. More visualization is available at https://github.com/CL2-UWaterloo/TrafficDiffuser. TABLE I: Performance Comparison of High-level Traffic Scenario Generation Common Sense Metrics ↓ [rgb]1,0,0 Distributional JSD ↓ [rgb]1,0,0 Collision rate (%) Off road rate (%) Near. edge (m) Speed Lat. Dev. Local Density Near. Dist. Initial States GT 0.50 1.70 1.58 - - - - SC [10] 4.40 5.25 1.63 0.16 0.212 0.177 0.092 PD-init [8] 4.22 5.36 1.64 0.1524 0.213 0.175 0.089 Ours 9.80 5.10 1.81 0.068 0.200 0.201 0.101 Goal States GT 0.50 2.80 1.86 - - - - Ours 10.6 7.21 2.02 0.08 0.198 0.206 0.103 IV-B2 Loss The trajectory infilling model fϕ(.)f_φ(.) learns to predict the distance from the bridge ℬB, by using the following loss function: ℒtraj(ϕ)=ℒrecon(ϕ)+λsℒsmooth(^),L_traj(φ)=L_recon(φ)+ _sL_smooth( τ), where λs _s is a hyperparameter to balance temporal smoothing and accuracy. The reconstruction loss is defined as ℒrecon(ϕ)=‖^−‖2L_recon(φ)=|| τ- τ||^2, where the predicted trajectory is reconstructed as ^=ℬ+fϕ(ℳ,ℬ,m) τ=B+ γf_φ(M,B,m). To avoid abrupt motions, a smoothness penalty is introduced asℒsmooth(^)=∑h=0H‖^h+1−^h‖2L_smooth( τ)= _h=0^H|| τ_h+1- τ_h||_2, which computes the mean squared first-order difference over the trajectory horizon. V Experiments In this section, we demonstrate the generation of high-level traffic scenarios. We show how the jointly generated initial and goal states improve the interpretability by providing more context for each agent’s initial placement. Also, we show how the generated high-level traffic scenarios can be leveraged for trajectory generation, demonstrating their compatibility through experiments. V-A Experimental Setup V-A1 Dataset Argoverse 2 Motion Forecasting Dataset [20], which is a large-scale real-world dataset for motion planning and prediction, is used to capture the distribution of real-world traffic scenarios. As explained in Section I, for diffusion models, unit-variance Gaussian noise is added based on the predefined schedule during the forward process. For training, the input data should be scaled to have a variance comparable to that of the noise, ensuring meaningful signal-to-noise ratios. Therefore, during data preprocessing, the speed and positions of the agents and map components are z-normalized using the mean and standard deviation of the agents’ initial and final states. We consider 5 agent categories: vehicle, bus, pedestrian, bicycle, and motorcycle. V-B High-level Traffic Scenario V-B1 Baselines Although we generate both initial and goal states of the agents in the scene, we compare our initialization results against the most relevant agent initialization approaches in the literature. Specifically, we evaluate against SceneControl(SC) [10] replica, as it is not publicly available, and PathDiffuser’s agent initialization (PD-Init) [8]. V-B2 Performance Measure The generated high-level traffic scenarios contain two time steps, and each time step is evaluated separately. As in [10, 8, 13], we assess the realism of the generated scenarios using Common Sense and Jensen-Shannon Divergence (JSD). Common Sense includes (i) Collision Rate, which measures the ratio of collided agents, (i) Off-road Rate, which measures the ratio of off-road agents, and (i) the distance to the nearest lane center point(Near. Edge). To measure the distributional distance between ground truth and generated output, the distributions are estimated using histograms. Then, using JSD, the distance is measured for (i) Speed, (i) distance to the nearest lane center (Lat. Dev.), (i) distance to 5 nearest agents (Local Density), and (iv) distance to the nearest agent (Near. Dist.). V-B3 Quantitative Results Table I shows the evaluation results for initial and goal states in separate rows. Although the task became more complex by modeling the high-level traffic scenario, we observe that the off-road rate is reduced by 2.8% compared to the PD-init. In terms of JSD, speed distribution is closer to the ground truth by 55.3%, compared to the speed output distribution of SC. Also, we observe that TrafficDiffuser shows an increase in the collision rate and the distance to the nearest edge, which is expected as we use encodings of map polygons instead of map points. For fair comparison, the results reported in Table I were obtained without the use of guidance sampling. As shown in SceneControl [10], guidance sampling can improve realism, which improves quantitative and qualitative results. V-B4 Qualitative Results Generated high-level traffic scenarios are visualized in Fig. 5. We observe that the generated high-level scenario consists of pairs of points, the initial and goal states of the same agent, with reachable paths on the map. To show the diversity of generated scenarios, we generate multiple scenarios from the identical map. Each row in Fig. 5 shares the same map, while each column displays variations with a different number of agents. To leverage the benefit of diffusion models, guidance sampling, explained in Sec. I-C, is used to bias the reverse process. As guidance functions, the distances to neighboring agents and to the lane center are used. V-B5 Interpretability One of the main contributions of the proposed framework is that the generated initial states are better interpretable. While baseline approaches only generate agent initial states, our approach provides agents’ initial states with their intent by indicating where the vehicle is trying to reach. For instance, as shown in Fig. 5, each agent is explicitly paired with a corresponding goal state. By comparing the initial and goal states, we can infer the vehicle’s intention. A close distance between them indicates the vehicle is stationary, whereas a larger distance between them indicates a moving vehicle. Compared to existing approaches, initial-goal pairing enhances scenario interpretability by showing the underlying intent of each agent and provides a preview of the traffic scenario. V-C Trajectory Infilling Given the map and high-level traffic scenarios, trajectory generation can be simplified to a goal-conditioned trajectory prediction by connecting the initial and end states. V-C1 Metrics As the trajectories are infilled by fixing the initial and goal states, we evaluate the trajectories by comparing with the groundtruth using standard metric for motion prediction [12, 4], which are (1) Average Displacement Error (ADE) that measures average displacement across the prediction horizon H, (2) Final Displacement Error (FDE) that measures the displacement error of the final states, and (3) Miss Rate (MR) that evaluates the rate of agents with FDE is above 2 meters. V-C2 Quantitative Results As shown in Table I, MR is all zero, and ADE is close to groundtruth, which is expected since we use the bridge ℬB between initial and goal position as an anchor. Also, as the weight γ forces the distance to zero under the given constraints, we observe that FDE is zero if the goal states are fixed. V-C3 Qualitative Results Fig. 6 shows infilled trajectories between the high-level scenarios. We can observe that, given high-level scenarios, the model produces trajectories between the initial and goal states while following the map structure, even when changing lanes or turning left. Fig. 6: Visualization of Trajectory Infilling. Given the ground-truth initial and goal states, the model infills the states by connecting them with a kinematically feasible trajectory. TABLE I: Quantitative Evaluation of Trajectory Infilling Model Hard Constraints ADE (m)↓ [rgb]1,0,0 FDE (m) ↓ [rgb]1,0,0 MR (%) ↓ [rgb]1,0,0 Ours Initial and Goal States 0.52 0 0 Initial States 0.53 0.16 0.19 Goal States 0.53 0 0 None 0.53 0.1 0.04 PD-init [8] Initial States 1.86 7.16 43.84 Fig. 7: Visualization of End-to-end Traffic Scenario Generation. Each row has an identical map structure to demonstrate trajectory generation given a high-level scenario. (a) visualizes the sampled high-level traffic scenarios containing the initial and goal states, and (b) illustrates the infilled trajectory between them with hard constraints (HC). (c) demonstrates the compatibility of the framework with the PD-Traj model to generate trajectories conditioned on the given initial states. V-D End-to-end Process for Complete Scenario Generation We now evaluate the entire pipeline, in which the generated multi-agent high-level traffic scenarios are used to produce trajectories. As illustrated in Fig. 7(a), the high-level scenario consists of synthesized pairs of agents. Then, Fig. 7(b) shows the predicted trajectory using our trajectory infilling model. Compatibility with Existing Trajectory Generator To demonstrate how the generated high-level scenarios can be used, we integrate them with the trajectory generation model from PathDiffuser (PD-Traj) [8]. As Fig. 7(c) shows, our high-level traffic scenario generator can be combined with an existing trajectory generation model to promote diverse traffic scenario generation. VI Discussion Summary. We presented TrafficDiffuser, a framework that factorizes the multi-agent traffic scenario generation into a high-level traffic scenario generation and a goal-conditioned trajectory infilling. By jointly modeling the initial and goal states, our approach captures reachable initial and goal states, as well as social interactions, from a real-world driving dataset. This hierarchical structure provides enhanced interpretability from high-level traffic scenarios and simplifies the trajectory generation task into a goal-conditioned prediction problem. Our framework not only optimizes the generative process but also maintains modular compatibility with existing models in the trajectory generation domain. Limitations and Future Work. Despite these contributions, our approach has certain limitations. In the proposed framework, stochasticity is confined to the high-level scenario generation, while the trajectory generation is formulated as a deterministic, goal-conditioned infilling task rather than a probabilistic process. While this accelerates inference, it limits the diversity of trajectories for a given high-level scenario. To address this, we plan to integrate generative models into trajectory infilling to better capture the multimodality of agent behavior and extend it to conditional generation based on partially observed agent trajectories. Conclusion. TrafficDiffuser demonstrates that modeling traffic scene initialization with agents’ intent enhances the interpretability of synthesized traffic scenarios. By factorizing traffic scenario generation into high-level scenario generation and goal-conditioned trajectory infilling problems, we provide a modular and efficient pipeline for synthesizing complex multi-agent traffic scenarios. Ultimately, this framework provides an efficient, interpretable foundation for developing and validating autonomous systems in realistic traffic scenarios. References [1] K. Chitta, D. Dauner, and A. Geiger (2024) SLEDGE: synthesizing driving environments with generative models and rule-based traffic. In ECCV, Cited by: §I. [2] P. Dhariwal and A. Q. Nichol (2021) Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Cited by: §I, §I-C. [3] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid (2020) VectorNet: encoding hd maps and agent dynamics from vectorized representation. External Links: 2005.04259, Link Cited by: §I. [4] D. Grimm, A. Abouelazm, and J. M. Zöllner (2025) Goal-based trajectory prediction for improved cross-dataset generalization. In 2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC), Vol. , p. 3007–3013. External Links: Document Cited by: §I, §I, §V-C1. [5] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In NeurIPS, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), Cited by: §I-C, §IV-A2, §IV-A. [6] C. M. Jiang, Y. Bai, A. Cornman, C. Davis, X. Huang, H. Jeon, S. Kulshrestha, J. W. Lambert, S. Li, X. Zhou, C. Fuertes, C. Yuan, M. Tan, Y. Zhou, and D. Anguelov (2024) SceneDiffuser: efficient and controllable driving simulation initialization and rollout. In NeurIPS, External Links: Link Cited by: §I, §I, §I. [7] C. M. Jiang, A. Cornman, C. Park, B. Sapp, Y. Zhou, and D. Anguelov (2023) MotionDiffuser: controllable multi-agent motion prediction using diffusion. In CVPR, p. 9644–9653. Cited by: §I-C. [8] D. S. Lee, A. Karthikeyan, Y. V. Pant, and S. Fischmeister (2025) Path diffuser: diffusion model for data-driven traffic simulator. In 2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC), Vol. , p. 569–576. External Links: Document Cited by: §I, §IV-A1, TABLE I, §V-B1, §V-B2, §V-D, TABLE I. [9] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun (2020) Learning lane graph representations for motion forecasting. External Links: 2007.13732, Link Cited by: §I. [10] J. Lu, K. Wong, C. Zhang, S. Suo, and R. Urtasun (2024) Scenecontrol: diffusion for controllable traffic scene generation. In ICRA, p. 16908–16914. Cited by: §I, §I, §I, §I-C, §IV-A1, TABLE I, §V-B1, §V-B2, §V-B3. [11] J. Philion, X. B. Peng, and S. Fidler (2024) Trajeglish: traffic modeling as next-token prediction. In ICLR, External Links: Link Cited by: §I. [12] L. Rowe, R. Girgis, A. Gosselin, B. Carrez, F. Golemo, F. Heide, L. Paull, and C. Pal (2024) CtRL-sim: reactive and controllable driving agents with offline reinforcement learning. External Links: 2403.19918, Link Cited by: §I, §I, §I, §V-C1. [13] L. Rowe, R. Girgis, A. Gosselin, L. Paull, C. Pal, and F. Heide (2025) Scenario dreamer: vectorized latent diffusion for generating driving simulation environments. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17207–17218. Cited by: §V-B2. [14] L. Rowe, R. Girgis, A. Gosselin, L. Paull, C. Pal, and F. Heide (2025) Scenario dreamer: vectorized latent diffusion for generating driving simulation environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §I. [15] T. Salimans and J. Ho (2022) Progressive distillation for fast sampling of diffusion models. External Links: 2202.00512, Link Cited by: §I. [16] S. Sun, Z. Gu, T. Sun, J. Sun, C. Yuan, Y. Han, D. Li, and M. H. Ang (2024) DriveSceneGen: generating diverse and realistic driving scenarios from scratch. IEEE RA-L. Cited by: §I. [17] S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun (2021) SceneGen: learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 892–901. Cited by: §I, §I. [18] E. Vinitsky, N. Lichtlé, X. Yang, B. Amos, and J. Foerster (2023) Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. External Links: 2206.09889, Link Cited by: §I. [19] Y. Wang, C. Tang, L. Sun, S. Rossi, Y. Xie, C. Peng, T. Hannagan, S. Sabatini, N. Poerio, M. Tomizuka, et al. (2025) Optimizing diffusion models for joint trajectory prediction and controllable generation. In ECCV, p. 324–341. Cited by: §I-C. [20] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays (2021) Argoverse 2: next generation datasets for self-driving perception and forecasting. In NeurIPS Datasets and Benchmarks Track, External Links: Link Cited by: §I, §V-A1. [21] T. Ye, L. Dong, Y. Xia, Y. Sun, Y. Zhu, G. Huang, and F. Wei (2025) Differential transformer. In ICLR, External Links: Link Cited by: §I, §IV-A1, §IV-A2. [22] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov (2020) TNT: target-driven trajectory prediction. External Links: 2008.08294, Link Cited by: §I. [23] Z. Zhou, J. Wang, Y. Li, and Y. Huang (2023) Query-centric trajectory prediction. In CVPR, Cited by: §IV-A1.