Paper deep dive
OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models
Tianran Liu, Shengwen Zhao, Mozhgan Pourkeshavarz, Weican Li, Nicholas Rhinehart
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/1/2026, 1:16:57 AM
Summary
OccSim is a novel 3D autonomous driving simulator that utilizes an occupancy world model (W-DiT) to generate large-scale, multi-kilometer static environments and dynamic traffic layouts without relying on HD maps or continuous driving logs. By leveraging rigid transformations and a mask-injected conditioning mechanism, OccSim achieves stable rollouts over 3,000 frames, significantly outperforming existing state-of-the-art models in long-horizon consistency and downstream semantic occupancy forecasting.
Entities (5)
Relation Signals (4)
OccSim → poweredby → W-DiT
confidence 100% · OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator.
OccSim → poweredby → Layout Generator
confidence 100% · OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator.
W-DiT → generates → 3D occupancy maps
confidence 95% · W-DiT handles the ultra-long-horizon generation of static environments
OccSim → improvesperformanceof → 4D semantic occupancy forecasting
confidence 95% · data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80x improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5x the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%.
Tags
Links
- Source: https://arxiv.org/abs/2603.28887v1
- Canonical: https://arxiv.org/abs/2603.28887v1
Trouble viewing inline? Open PDF directly →
Full Text
79,449 characters extracted from source content.
Expand or collapse full text
]Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models Tianran Liu⋆ Shengwen Zhao Mozhgan Pourkeshavarz Weican Li Nicholas Rhinehart† [ Abstract Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80× improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5×5× the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%. [ Project Website]https://orbis36.github.io/OccSim/ Figure 1: Comparison of OccSim and workflow. We overcome all of the mentioned drawback of previous method, with only 1 single frame and future actions, the W-DiT we proposed is able to build a multi-kilometer consistent occupancy map for autonomous driving simulation. The agent initial poses are generated via the layout generator and then forward-simulated. OccSim is compatible with multi-agent forward simulation methods as plug-and-play modules; in our experiments, we use an IDM variant [kesting2010enhanced]. 1 Introduction Closed-loop evaluation is indispensable for the advancement of Level 4 autonomous driving systems [caesar2021nuplan, dauner2024navsim, codevilla2018offline]. Although open-loop replay of recorded data provides authentic data streams, it inherently lacks reactive responses to ego behaviors. In contrast, geometry-based simulators (e.g., CARLA[Dosovitskiy17]), despite supporting interactivity, suffer from a significant sim-to-real gap regarding sensor realism and content diversity [yang2023unisim, zhang2025epona]. This creates a fundamental trade-off: current approaches are often forced to compromise between realism, diversity, and interactivity. Although emerging data-driven simulations [yan2025drivingsphere, zhou2024hugsim] promise to overcome this dilemma, existing methods exhibit notable deficiencies when deployed as reliable simulators. ∗: Contact: tianran.liu@mail.utoronto.ca. † : Project Manager. Specifically, we argue that a comprehensive and scalable data-driven simulator must satisfy the following desiderata: 1. Sensor realism & semantic richness: The simulator should not only approximate the distribution of real-world sensor data (high-fidelity sensor realism) but also inherently generate dense semantic labels, eliminating the reliance on manual annotation. 2. Reactive & diverse agents: Simulator should provide an API or direct learn non-ego vehicles (agents) interactive behaviors from data, reflecting human-like distributions rather than following heuristic rules or simple trajectory playback. 3. 3D-Consistent World Dynamics: The simulator should maintain global spatial persistence, overcoming the “memoryless visual hallucination” prevalent in standard generative models. During long-horizon closed-loop driving, static elements (e.g., road topology, buildings) must preserve strict spatiotemporal consistency. For instance, when the ego-vehicle completes a loop and revisits a previously observed location, the static environment must remain structurally identical. 4. Log-independent Simulation: The simulator’s operation should not be overly-constrained by pre-recorded driving logs. Ideally, it possesses the capability for open-ended generation in novel, unseen scenarios in order to overcome the scale limitations of collected data. In this paper, we aim to address the overarching question: “How can we construct a data-driven generative simulator that efficiently satisfies these desiderata?” To this end, we present OccSim. To the best of our knowledge, OccSim is the first autonomous driving simulator driven by an occupancy world model. As illustrated in Figure 1, OccSim establishes a new paradigm: it uniquely satisfies all the aforementioned desiderata by conditioning solely on a single static frame and future ego-actions, while being trained entirely on publicly-available data. Specifically, we adopt a decoupling strategy for static and dynamic voxels. We model the simulation process as two synergistic generative tasks: the multi-kilometer rollout from world model to consist large-scale static scenes and the conditional generation of dynamic traffic flow. For static-world generation, the core difficulty is to ensure the stable high-quality output of the world model over ultra-long horizons. Prior work lacks this stability — given future actions, several SOTA occupancy world models[shi2025come, liao2025i2] can only stably generate fewer than 50 frames (shown later in Fig. 5). Although this horizon is sufficient for vehicle control decisions, when the problem extends to larger-scale map construction (e.g., multiple city blocks, ≥2km2≥\!2km^2) , 20–50 frames often cover a range of only about 60-150m (given 2Hz sampling rate with around 3 meters moving of ego vehicle per frame), which is far from real street size. For the task of dynamic generation, the model needs to add reasonable agents within the given range of static occupancy and simulate realistic motion through the static map. We attribute the short horizon rollout of occupancy world model to the network architecture design of current stochastic 4D occupancy world models, where almost all works directly borrow typical architectures from the video generation field: stacking multiple 3D occupancy latents in the temporal dimension and then aggregating features via DiTs in both temporal and spatial dimensions [gu2024dome, ma2024latte]. This paradigm neglects the inherent geometric rigidity and SE(3) equivariance of 3D space, forcing the network to implicitly learn complex camera motions: that is, most areas in space can be obtained through explicit rigid rotation matrices. This characteristic easily leads to model collapse during the auto-regressive based rollout process, where errors accumulate progressively. Based on this observation, we propose “W-DiT”, the first world model capable of diverse, at least 3k-step rollouts of static maps under complex trajectories (tested with over 3,000 frames generated, >80× improvement in stable rollout length), which we have used to generate static scenes of over 4km in length. The core mechanism of W-DiT is a pipeline designed for effective conditioning on the previous frame: W-DiT explicitly utilizes 3D geometric priors through a unique Mask-injected Conditioning mechanism and an SNR-weighted occupancy perception loss. Extensive experiments prove that W-DiT is the first model capable of achieving 3k-step horizon static map rollout, while maintaining extremely high scene diversity. After obtaining single-frame outputs, we design a keyframe-based map fusion method and a graph-search-based lane topology extraction scheme, which allows us to obtain large-scale static road maps for agents to drive on. After acquiring the large-scale static road network, how to populate interactive agents without relying on historical logs? We discard rule-based heuristic algorithms and propose a layout generator based on Latent Flow Matching. The layout generator produces road structure-conditioned initial conditions of traffic. The inference logic of OccSim is shown in figure˜1: first, generate a large-scale static road network at once, then randomly specify the ego vehicle’s position, and finally complete the population and control of all agents. In summary, our contributions can be summarized as follows: 1. We propose a new pipeline for data-driven occupancy-based simulation: it consists of an occupancy world model responsible for large-scale static scene generation, a layout generation model controlling the initial position and a IDM-based agents control engine. The pipeline also compatible with any current trajectory forecasting algorithms to provide even more realistic agent behavior control. 2. To solve the ultra-long rollout problem of the occupancy world model for road network construction, we propose W-DiT and corresponding optimization schemes. Our W-DiT based backbone can achieve a stable rollout length over 80×80× longer than that of previous models using the same amount of training data. Furthermore, we provide a set of keyframe-based frame-to-map fusion schemes to obtain large-scale road networks. To our best knowledge, W-DiT is currently the first model capable of rolling out over 3,000 frames static occupancy and forming complex road topologies without relying on HD maps in any form as condition. 3. Unlike previous rule-based agent addition, our agent layout generation module can learn potential agent positions and states directly from semantic occupancy, allowing our model to provide more realistic initial traffic flows. 4. Combining all modules mentioned, extensive experiments confirm that with the data collected from OccSim, we can directly train existing semantic occupancy forecasting models and achieve up to 67% zero-shot performance in terms of average IoU/mIoU, which 11% higher than previous asset-based method. When scaling the collected OccSim dataset to 5× the size, the zero-shot performance further increases to up to 74%, while the improvement over asset-based simulators expands to 22%. 2 Related Work 2.1 Long horizon RGB world models Driven by the abundance of data resources, numerous efforts to construct world models for autonomous driving have utilized RGB signals as their primary input to achieve robust long-horizon rollouts. For instance, GAIA-1 [hu2023gaia] employs a multi-modal transformer architecture with language and action conditioning to autoregressively generate minute-long video clips. GAIA-2 [russell2025gaia] extended this capability to multi-view RGB generation by transitioning the underlying backbone from a deterministic model to flow matching. Furthermore, Cosmos-Drive-Dreams [ren2025cosmos] finetuning the Cosmos [agarwal2025cosmos] foundation model to enhance the roll out of autonomous driving scene generation. In the indoor navigation task, Ego2Memory [zhang2025mem2ego] explicitly uses VLM to map 2D features with Frontier Map for long-term persistent rollout. By introducing a two-stage training and fine-tuning pipeline alongside better utilization of dynamic priors, Vista [gao2024vista] significantly improves the resolution and realism of generated videos, as evidenced by FID and FVD metrics. Subsequently, DrivingWorld [hu2024drivingworld] adopted a purely auto-regressive (AR) based approach. More recently, Epona [zhang2025epona] provided the first open-source implementation for long video rollouts, utilizing an AR diffusion method. 2.2 Occupancy World Models in Autonomous Driving While RGB images offer rich semantic features, accurately capturing and predicting the complex 3D geometry and physical depth of dynamic environments remains a significant challenge. Consequently, there is a growing trend towards developing world models based on 3D occupancy representations, which are typically derived from voxelized and densified LiDAR data. Earlier works such as OccWorld [zheng2024occworld] and Occllama [wei2024occllama] addressed this by combining discrete encoding techniques (e.g., VQ-VAE [van2017neural]) with deterministic auto-regressive (AR) transformer architectures, with Occllama further incorporating Large Language Models (LLMs) to enhance scene comprehension. Currently, generative modeling approaches have gained significant traction. For instance, DOME [gu2024dome] utilizes a continuous VAE alongside a high-performance diffusion model, enabling fine-grained controllability. Recent advancements have further extended these capabilities: COME [shi2025come] introduces a post-trained ControlNet [zhang2023adding] approach, DynamicCity [bian2024dynamiccity] leverages hex-plane decomposition, and OccFM [liu2025towards] employs a Flow Matching-based multi-scale DiT [peebles2023scalable] architecture. Expanding the scope, UniScene [li2025uniscene] explores a conditional pipeline that uses BEV layouts to generate comprehensive multi-sensor data, ranging from RGB images to LiDAR point clouds. 2.3 Data driven simulation with generative models High-fidelity simulation is crucial for bridging the sim-to-real gap in autonomous driving, yet conventional physics-based simulators like CARLA [Dosovitskiy17] often struggle with domain discrepancies. Recent research has shifted towards data-driven approaches to leverage large-scale real-world datasets. Nocturne [vinitsky2022nocturne] facilitates high-efficiency simulation by replaying trajectories from the Waymo dataset. Building upon this, SLEDGE [chitta2024sledge] and Scenario Dreamer[rowe2025scenario] extends simulation to synthesized scenarios by generatively modeling road layouts and vehicle agents. In parallel, efforts have been made to generate more realistic sensory observations. ReSim [yang2025resim] and DriveArena [yang2025drivearena] utilize video generation techniques and HD maps to render high-resolution RGB observations for closed-loop policy evaluation. The raising of 3D Gaussian Splatting rendering in unbounded scene [chen2024lidar, zhou2024hugsim, xiao2025splatco] allow us reconstruct static backgrounds more realistic. While, the SemCity[lee2024semcity] and InifiCube[lu2025infinicube] follow the out-painting solution introduced by Repainting[lugmayr2022repaint] while combine it with tri-plane decomposition diffusion. Most recently, X-scene [yang2025x] advances the field by proposing an external diffusion module that generates BEV layouts from natural language prompts, thus mitigating dependency on recorded driving logs. 3 Long-horizon static Occupancy World Models In this part, we introduce the W-DiT architecture and its corresponding optimization methods in Section 3.1, which form the foundation of our infinite-horizon static scene generation. In Section 3.2, we briefly describe how to merge single-frame static scenes generated by W-DiT and infinitely extend them into closed-loop city-scale environments. Furthermore, in Section 3.3, we will demonstrate how to learn the possible distribution of agents layout from data and incorporate them into the scene to interact with test ego vehicle. 3.1 Long-horizon stable generation with W-DiT Figure 2: Illustration of structure to generate road map of t+1t+1 from condition at timestep t. Here, t represents the sequence frame index, and τ denotes the probability flow timestep. Different from classic temporal concatenation in the input, the core insight of this paradigm design is to transform temporal generation into scene completion at a single time point. This token-wise scale and shift condition injection method allow us can precisely control spatial rigid transformation during long-horizon generation. I(−)I(-) stand for rasterization process of given future trajectory Jt+1t+nJ_t+1^t+n. The latent representations ztz_t and zt+1z_t+1 (∈ℝH×W×Cz ^H× W× C_z) at bottom right corner are visualized as decoded BEV occupancy maps for intuitive illustration. A fundamental distinction between occupancy sequences and RGB sequences is that perspective shifts of occupancy sequences can be represented with rigid transformations. Despite this advantage, most existing occupancy world models naïvely inherit the 2+1D information aggregation approach from video generation. We found that a geometry-aware alternative yields substantial improvements in long-term stability, perspective controllability, and spatial consistency. Formally, using an occupancy VAE [liu2025towards] (consisting of spatial compressor ℰE and decoder D), we can map the occupancy frames t,t+1O_t,O_t+1 to their respective latent representations t,t+1∈ℝH×W×Czz_t,z_t+1 ^H× W× C_z, where t denotes time index. Our aim is to model the joint predictive distribution over an extended future horizon: given an initial latent state 0z_0 and a sequence of future ego-trajectories J0:K+σJ_0:K+σ, our goal is to auto-regressively estimate: P(1:K|0,J0:K+σ)=∏i=1KPθ(i|i−1,Ji−1:i+σ−1),P(z_1:K|z_0,J_0:K+σ)= _i=1^KP_θ(z_i|z_i-1,J_i-1:i+σ-1), (1) where K is the overall forecast horizon and K+σK+σ stand for the length of trajectory. When K≥103K≥ 10^3, standard autoregressive formulations inevitably suffer from severe compounding errors, leading to structural drift and spatial inconsistency over time. To enforce ultra-long-term stability and fidelity, we propose the W-DiT (Warp-DiT) block (Figure 2). Unlike conventional RGB-based world models, which require the network to implicitly hallucinate perspective changes, the core design of W-DiT lies in explicitly utilizing a sequence of deterministic rigid transformation matrices—derived from the trajectory J—to warp the spatial latent representations, thereby bounding geometric accumulation error. Specifically, to have the rigid transformation matrices, we assume a planar motion on the XYXY-plane, we constrain the vertical velocity and roll/pitch rotations to zero. Therefore, we can define t+1=exp(ξ^tΔt)T_t^t+1= ( ξ_t t), where exp(⋅) (·) denotes the matrix exponential, ξt=ℳ(t) _t=M( a_t) is the twist vector corresponding to t=[vx,t,vy,t,ωt]⊤ a_t=[v_x,t,v_y,t, _t] , the hat operator ⋅ · converts the twist ξt=[vx,t,vy,t,0,0,0,ωt]⊤ _t=[v_x,t,v_y,t,0,0,0, _t] into its matrix form ξ^t=[[]×,;0]∈ℝ4×4 ξ_t=[[ ω]_×,v;0] ^4× 4, with the planar velocity vector =[vx,t,vy,t,0]⊤v=[v_x,t,v_y,t,0] and yaw rate vector =[0,0,ωt]⊤ ω=[0,0, _t] . Here, [⋅]×[·]_× denotes the skew-symmetric matrix operator. Subsequently, by applying the rigid transformation t+1T_t^t+1, we forward-warp tz_t to the viewpoint at t+1t+1 to obtain the warped latent ^t+1 z_t+1. Due to the ego-vehicle’s motion, certain spatial regions in this warped latent inevitably fall outside the sensor’s field of view or become occluded. To explicitly account for this, we can deterministically compute a binary visibility mask vis∈0,1H×WM_vis∈\0,1\^H× W derived purely from the known geometry, where observable regions are set to 1 and unobservable ones to 0. However, since the physical displacement between adjacent frames is relatively small (e.g., 3 to 4 meters given Δt=0.5s t=0.5s), relying solely on visM_vis is insufficient and encourages the model to degenerate into a trivial shortcut of directly copying the inputs, which severely hinders robust representation learning. Therefore, following MAE [he2022masked], we introduce an additional random mask rand∈0,1H×WM_rand∈\0,1\^H× W, whose elements are sampled from a Bernoulli distribution ℬ(1−p)B(1-p), with p being the given masking ratio. The final conditioned latent ~t+1 z_t+1 is then formulated as ~t+1=(vis⊙rand)⊙^t+1 z_t+1=(M_vis _rand) z_t+1, where ⊙ denotes the Hadamard product, and the spatial masks are broadcast across the channel dimension CzC_z. Next, instead of fusing the future action with τ directly [gu2024dome, shi2025come], we rasterized these action to waypoints which align with time index t+1t+1. Specifically, we represent the sequence of future actions as a set of physical waypoints t=t+i∈ℝ2i=1σP_t=\p_t+i ^2\_i=1^σ projected onto the BEV plane. These waypoints are anchored in the ego coordinate system at time t+1t+1, derived by cumulatively applying the previously defined rigid transformation matrices t+1t+σT_t+1^t+σ. By sequentially connecting these waypoints, we construct a continuous geometric polyline that explicitly traces the intended path from the current ego position towards the future road surface and then rasterize it to lt+1∈0,1H,W,1l_t+1∈\0,1\^H,W,1. Here we use function I(−)I(-) to represent this projection and rasterization process. Considering the interpolation error caused by the warping operation, we use a U-Net to refine the feature formed by concatenation of lt+1l_t+1 and ~t+1 z_t+1, before injecting them to the backbone. Here, we use ft+1′∈ℝB,N,Cwf _t+1 ^B,N,C_w to represent token from condition at time t and trajectory, where N=H×WN=H× W. After obtaining these features f^t+1 f_t+1 aligned to time index t+1t+1, we perform feature injection by applying token-wise scaling and shifting. Specifically, for global scale, shift, and gate regression, we still follow the same setting in original DiT, use timestep τ as input. Simultaneously, we employ a token-wise projection layer to regress the spatial control signal from f^t+1 f_t+1: [γglobal,βglobal,αglobal]=MLPtime(τ),[token,token,token]=MLPcond(f^t+1)[ _global, _global, _global]=MLP_time(τ),[ _ token,B_token,A_token]=MLP_cond( f_t+1) From the main stream side, we obtain zt+1τz_t+1^τ via a linear interpolation with the sampled gaussian noise ϵε: zt+1τ=(1−τ)ϵ+τzt+1z_t+1^τ=(1-τ)ε+τ z_t+1 and use ft+1∈ℝB×N×Cwf_t+1 ^B× N× C_w to represent the tokenized feature feeded to W-DiT block. For the i-th token in the sequence, the modulation parameters are obtained via: γ(i)=γglobal+token(i),β(i)=βglobal+token(i),α(i)=αglobal+token(i)γ^(i)= _global+ _ token^(i), β^(i)= _global+B_token^(i), α^(i)= _global+A_token^(i) With these dual-conditioned designs, the forward pass of the Multi-Head Self-Attention (MHSA) module in our W-DiT block is formulated as: ft+1′=ft+1+1⊙MHSA(1⊙LN()+1)f_t+1 =f_t+1+ α_1 ( γ_1 (h)+ β_1 ), where 1=[γ(1),γ(2),…,γ(N)]⊤∈ℝN×Cw γ_1=[γ^(1),γ^(2),…,γ^(N)] ^N× C_w denotes the stacked sequence of token-wise scale parameters, with 1 β_1 and 1 α_1 defined analogously. We adopt the identical setting for (2,2,2)( γ_2, β_2, α_2) in subsequent MLP layer. Finally, after processing through all W-DiT blocks, the network outputs a prediction of the instantaneous velocity field at the probability flow timestep τ. Let vθ(zt+1τ,τ,)v_θ(z_t+1^τ,τ,c) denote our W-DiT network parameterized by θ, where c compactly represents all the conditions we mentioned. Following the recent success of flow models in achieving high-quality generation with fewer sampling steps [liu2025towards, song2025hume, zhang2025epona], we also optimize W-DiT by adopting a velocity loss objective. However, original MSE loss in flow matching provides a theoretically sound objective, our empirical investigations reveal that optimizing solely with it in the continuous latent space severely impedes model convergence. This phenomenon is a notorious challenge in continuous VAE-based generative models, where the MSE objective often forces the network into a “mean trap”. To overcome this bottleneck, we propose a novel SNR-scaled Perception Loss. Our core insight is to leverage the inherently discrete and semantic nature of 3D occupancy. Rather than constraining the loss entirely within the continuous latent manifold, we explicitly project the denoised predictions back into the discrete occupancy space for direct supervision. Our overall objective function is: ℒtotal(θ)=τ,ϵ,zt+1,[‖vθ−(zt+1−ϵ)‖22+λτ2mask|mask|CE(t+1,(zt+1τ+(1−τ)vθ))]L_total(θ)=E_τ,ε,z_t+1,c [ \|v_θ-(z_t+1-ε) \|_2^2+λτ^2 M_mask|M_mask|CE (O_t+1,D(z_t+1^τ+(1-τ)v_θ) ) ] (2) where mask=vis⊙randM_mask=M_vis _rand and λ were used to balance different components. In the second term, at flow timestep τ, we first recover the denoised latent representation z^t+1=zt+1τ+(1−τ)vθ(zt+1τ,τ,) z_t+1=z_t+1^τ+(1-τ)v_θ(z_t+1^τ,τ,c) using the current velocity field presented by vθv_θ without any sampling. Then we feed z^t+1 z_t+1 into the frozen VAE decoder D to reconstruct the occupancy logits ^t+1=(z^t+1) O_t+1=D( z_t+1) for mask-region only Cross-Entropy (CE) loss between ^t+1 O_t+1 and t+1O_t+1. Crucially, we introduce a time-dependent scaling factor τ2τ^2 to this perception loss, which we ablate in Sec. 4.3. This term guarantees that, when τ is large (indicating a low-noise regime closer to the target data), the weight exponentially increases. This encourages the network to refine fine-grained geometric boundaries and semantic details. 3.2 Multi-kilometer road map fusion from single frames As detailed in Sec. 3.1, our W-DiT is capable of stably rolling out thousands of static map frames. While the stochastic nature of flow models effectively bypasses the data scarcity bottleneck of traditional HD maps, the generated outputs remain discrete, ego-centric local scenes. Furthermore, naive fusion strategies, such as temporal averaging, tend to aggressively over-smooth crucial high-frequency details during sequence aggregation, because of the temporal inconsistency in the forecasted frames. Our strategy consists of two components: key-frame based fusion and branch road detection: We first propose a two-pass keyframe-based fusion heuristic (detailed in Alg. 1 in Appendix 9). Given local occupancy predictions and their global poses, we first aggregate a minimally overlapping keyframe subset to establish a sharp foundational map ℳglobalM_global. Subsequently, after a gravity alignment step, the redundant non-keyframes are used to robustly inpaint spatial gaps via threshold-based voting and morphological cleaning. To enable unbounded map expansion, we systematically identify valid extensible frontiers within ℳglobalM_global. By applying morphological skeletonization to the road manifold, we extract a topological graph to isolate candidate branch points (Alg. 2 in Appendix 9). A Dual-Probing Filtering mechanism—comprising topological and 3D semantic probes—is then employed to reject collision-prone directions. By selecting two valid disjoint endpoints and connecting them via a collision-free spline, we prompt W-DiT to autoregressively synthesize the missing intermediate occupancy. This autoregressive bridging paradigm elegantly closes the loop, enabling the unbounded, procedural generation of diverse autonomous driving scenarios from a single prompt occupancy. Figure 3: Agent initialization within generated static maps. Each panel displays a 200x200 voxel crop centered on the (unrendered) ego-vehicle from ℳglobalM_global. The first two panels show distinct generated scenes, the third and fourth panels demonstrate the model’s multimodality: two different plausible layouts generated for the same revisited location. Orange vehicles (second panel) are stationary. 3.3 Agent generation and control with static road map Having constructed the city-scale static map ℳglobalM_global (with explicit lane graphs extracted via Alg. 3), we next populate the environment with plausible initial agent states. Unlike log-replay methods, our procedurally generated topologies are generated, necessitating a generative approach for traffic layouts. Formally, our objective is to model the conditional probability distribution of the agent layout given 1 frame of static environment: P(|t)P(A|z_t), where =(xi,yi,si)i=1NA=\(x_i,y_i,s_i)\_i=1^N denotes the set of vehicle centers (xi,yi)(x_i,y_i) motion states si∈dynamic,statics_i∈\dynamic,static\ and staticz_static is cropped from a previously fused large map given a random pose. To make this distribution tractable for our flow-based generative framework, we map the discrete set A into a continuous 2D spatial heatmap ℋH, encoding static and dynamic vehicles as positive and negative standard Gaussian kernels, respectively. Then we train a compact Diffusion Transformer (DiT-S) to learn the continuous distribution P(ℋ|t)P(H|z_t). To prevent the DiT-S from overfitting and memorizing small-scale dataset layouts, we apply multiple spatial augmentations during training (detailed in Appendix) During inference, the sampled heatmap ℋH is discretized via Non-Maximum Suppression (NMS) to extract individual vehicle instances. Each vehicle is then projected onto the nearest lane to establish a kinematically feasible initial yaw. In figure 3, we visualize several samples after add the agents to their initial pose. Finally, we employ A* routing and the extended 2D Intelligent Driver Model (2D-IDM) [kesting2010enhanced] to roll out microscopic kinematic trajectories toward randomly selected junctions in ℳglobalM_global (Alg. 5 in Appendix 9). While we utilize classic heuristics to validate simulation feasibility, our initialized map topology and agent poses guarantee plug-and-play compatibility with any modern learning-based trajectory forecasting model. 4 Experiment We organized our experiments to answer the following 3 questions: Q1. Static Realism & Stability: Can the proposed W-DiT structure generate highly realistic and spatiotemporally consistent static environments, especially over extended horizons? Q2. Generation Diversity: Does OccSim exhibit sufficient multimodality and diversity in long-horizon rollout? Q3. Dynamic Fidelity & Downstream Utility: Can OccSim be used to train temporal-based tasks like 4D semantic occupancy forecasting more effectively than data collected from CARLA? 4.1 Metrics and datasets To answer Q1, we assess its performance across two primary dimensions: the realism of the generated static environments and the dynamic fidelity of the traffic simulation. Following previous works [bian2024dynamiccity, yang2025x], we evaluate static realism via 2D/3D-FID, KID, and MMD (denoted by Dis below). To formalize realism, we define two evaluation settings to measure the performance of model under different context: 1. Conditional fidelity: Dis(Pθ(^t|0,J),Pdata(t|0,J)) Dis (P_θ( O_t|O_0,J),P_data(O_t|O_0,J) ). This evaluates the structural accuracy at each specific time index t∈1,…,t∈\1,…,K\ (K equal to the length of GT sequence) under the strict constraint of the initial observation 0O_0 and ego-trajectory J. 2. Unconditional realism: Dis(Pθ(^t|0,J),Pdata()) Dis (P_θ( O_t|O_0,J),P_data(O) ), where t depend on the rollout length K. Although the generation inherently relies on the initial observation 0O_0 and trajectory J, long-horizon rollouts can reasonably diverge into various valid structures. Therefore, instead of comparing against the strictly paired future frame, we measure the distance between the conditionally generated samples at time t and the global marginal distribution of the entire ground truth training set O. This ensures long-term rollouts universally reside on the valid data manifold. In Q2, we introduce the Vendi score [friedman2022vendi] and Semantic IoU diverse score. Specifically, the Semantic IoU diverse score evaluates structural divergence in the decoded semantic space by calculating the complement of the average Intersection over Union across multiple stochastic rollouts generated from the same initial scene. Meanwhile, the Vendi score operates in the latent feature space—extracted via a pretrained encoder—to quantify the effective sample size of the generated distributions. These metrics reflect the model’s multimodality, where higher scores indicate a richer variety of plausible future layouts, demonstrating the simulator’s capability to model environmental uncertainty. To address Q3, we evaluate the dynamic simulation performance through a zero-shot transfer setting. To the best of our knowledge, we are the first to demonstrate the downstream utility of a data-driven simulator by evaluating its pre-training efficacy on the 4D semantic occupancy forecasting task. Specifically, we utilize continuous data streams collected directly from the ego-vehicle’s perspective within our generated simulation. Performance on this downstream task serves as a proxy, directly reflecting the simulator’s capability to faithfully emulate complex, real-world traffic dynamics. Regarding datasets, we utilize Occ3D-nuScenes [tian2023occ3d] for all static generation realism comparisons to ensure fairness in training data volume. For the remaining experiments, we employ UniOcc-Waymo [wang2025uniocc] or UniOcc-nuScenes due to their unified voxel categorization. Additional simulation metrics and training/implementation details are provided in Appendix 6.3. 4.2 Main results comparisons We first evaluate the realism and stability of the generated static road maps. In Fig. 4, we compare our results with previous occupancy world models in terms of both conditional fidelity and unconditional realism. For a comprehensive analysis, our evaluations are conducted across two distinct VAE latent spaces (derived from UniScene [li2025uniscene] and OccFM [liu2025towards]). As illustrated in the figure, our method (shown in red) demonstrates superior long-term stability and structural realism, significantly outperforming baseline models that suffer from rapid error accumulation. Furthermore, due to the limited number of ground truth (GT) samples (fewer than 1000 per timestep t), we specifically selected unbiased metrics—namely KID and MMD with two different kernels—to assess conditional fidelity. Here, 1000 frames rollout generated conditioned on straight trajectory after GT’s, with constant velocity. Figure 4: Realism evaluation of generated 3D static occupancy. The left three columns report the conditional fidelity (MMD and KID) under a 30-frame constraint. The right two columns showcase the unconditional realism (KID and FID) over a 1000-frame long-horizon rollout. Our W-DiT-based method obtains the best long-horizon stability and conditional realism. This performance gap observed in the 3D occupancy metrics becomes even more pronounced when evaluating realism on the 2D plane. As illustrated in Fig. 5, to increase the rigor of our evaluation, we selected three distinct trajectories to compare our method against the previous state-of-the-art model, COME (details regarding the specific trajectory shapes are provided in Fig. 8 of the Appendix). Notably, even when evaluated on the most challenging curved and closed-loop trajectories, our approach still outperforms the previous state-of-the-art method’s results on simpler straight-line trajectories. Furthermore, to establish a lower-bound reference, we constructed a 1000-frame occupancy map collection to serve as a “chaos” baseline. This was achieved by randomly selecting structurally collapsed samples from the model’s output that lack coherent semantic information. Qualitatively, any metric score surpassing this baseline threshold indicates a completely unusable state. Visualizations of this chaotic baseline collection are available in Appendix 8.3. Figure 5: Long-horizon 2D realism under varying trajectories. FID and KID comparisons for 2D projections of the 3D generated occupancy. We evaluate three ego-actions: straight, closed-loop, and continuous turning. The dashed reference line indicates the “chaos” threshold characterizing scene collapse. In all scenarios, our model (red) remains strictly below this threshold, vastly outperforming the previous SOTA model(which exceed the “chaos” threshold in 35/40-th frame generated) and ensuring long-term structural integrity regardless of the driving action. To evaluate the generation diversity over long-term rollouts, we analyze the Pairwise mIoU Diversity and Vendi Scores across first 100 timesteps, as illustrated in Fig. 6. Our proposed method consistently outperforms prior generative baselines. Notably, at early timesteps, our approach demonstrates a rapid increase in diversity, establishing a significant margin over both COME and DOME. As the autoregressive generation extends, our method maintains a high and stable level of diversity without performance degradation. In both OccFM and UniScenes latent spaces, our method achieves the highest Vendi Scores, indicating a richer variety of generated geometric and semantic structures. Figure 6: Generation diversity comparison. Pairwise mIoU diversity and Vendi scores evaluated over 100 timesteps using 10 random seeds. Solid lines and shaded areas denote the mean and the 25th-75th percentiles, respectively. Our methods performed the best diversity in the rollout among all tested generative method. To further validate the practical utility of our generated world, we evaluate OccSim on the downstream task of 4D semantic forecasting. For this experiment, we configure the model to unroll along a curved, closed-loop trajectory, expanding the generation horizon from 1,000 to 3,000 frames to construct a significantly larger map. Based on this extended map, we build the simulation environment utilizing the methodologies detailed in Sec. 3.2 and Sec. 3.3. As demonstrated in Tab. 1, even when relying on a relatively straightforward dynamics algorithm such as the 2D IDM to control agents, OccSim provides better training signals for simulating overall traffic flow and background dynamics than CARLA. Notably, models trained on data generated by OccSim achieve better performance compared to those trained on the CARLA simulator (Uni-C), showing an improvement of at least 14%14\% relative in average mIoU across the two semantic forecasting models. When we collect 5x more data from UniOcc to train the downstream model, this gap further expands to 52%52\%. In terms of 3s average IoU, our method reaches 67% and 72% of the upper-bound’s performance. Table 1: The performance of different semantic forecasting model trained with occupancy collected from UniOcc-Carla(Uni-C) or OccSim, test on UniOcc-NuScenes(Uni-N) validation set. mIOU measured over all background occupancy categories and car category for foreground. † : We collected the same amount of data on OccSim as that provided by UniOcc for Carla for fairness. ⋆ : Use ×5× 5 amount of data collected from OccSim to train downstream models. Stoch.: whether the model is stochastic. The gray-shaded rows represent the performance upper bound, as the models are trained and tested on the same domain without domain shift. Model Train Data mIoU IoU IoUvehIoU_veh Stoch. 1s 2s 3s 1s 2s 3s OccWorld[zheng2024occworld] Uni-N 26.53 16.73 12.33 32.17 22.88 18.41 17.10 ✗ OccWorld Uni-C 11.79 8.35 6.75 18.31 14.09 12.03 6.58 ✗ OccWorld†OccWorld OccSim 13.51 9.56 7.61 20.53 15.48 13.24 7.46 ✗ OccWorld⋆OccWorld OccSim 19.54 12.31 9.08 23.87 16.92 13.47 12.41 ✗ OccFM[liu2025towards] Uni-N 36.28 25.10 19.47 41.22 30.90 25.02 28.79 ✓ OccFM Uni-C 6.99 5.34 4.70 8.63 6.50 5.71 6.42 ✓ OccFM†OccFM OccSim 15.93 11.76 9.26 18.31 14.10 11.75 13.28 ✓ OccFM⋆OccFM OccSim 25.32 18.96 13.10 29.19 21.08 13.93 15.82 ✓ 4.3 Ablation study In Tab. 2, we conduct a detailed analysis of the perception loss, network architecture, and SNR weighting proposed in our W-DiT design: The success of W-DiT is inextricably linked to these three components. It is evident that without any one of them, achieving stable quality output at the 500-frame scale becomes challenging. Among these, the feature injection structure plays the most significant role. Without this design, the model’s output degrades rapidly. Table 2: W-DiT design ablation: we use 3D unconditional realism FID score. When not using the “local-global” form of feature aggregation we propose, we directly concatenate the masked condition with the input. The latent here is encoded by AE from UniScenes. Percep. Loss Dual-head injection SNR scaled 1st 5th 10th 30th 50th 100th 500th ✗ ✗ ✗ 663.89 712.56 789.91 1023.51 1281.99 1587.21 1612.40 ✗ ✓ ✗ 637.52 708.91 924.11 992.72 1206.50 1387.55 1672.43 ✓ ✓ ✗ 676.98 652.91 711.66 782.25 801.67 852.30 901.57 ✓ ✓ ✓ 665.23 630.34 645.96 687.90 698.87 712.65 744.71 5 Conclusion In this paper, we present OccSim, the first occupancy world model-driven simulator capable of generating diverse and realistic rollouts spanning thousands of frames without any BEV layout or HD map as input. We demonstrate that a comprehensive simulator built upon such generated road topologies can replicate the spatiotemporal dynamics of driving scenarios with significantly higher fidelity than traditional simulators. As our framework is inherently compatible with complex learning-based policies, future work could study replacing IDM with other data-driven control strategies, alongside extending the simulator to support high-fidelity multi-modal generation. Acknowledgments This research was enabled in part by the Digital Research Alliance of Canada (alliancecan.ca), the NVIDIA Academic Grant Program, and Google TPU Research Cloud (TRC). Appendix In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper. 6 Training and metric details 6.1 Training configuration The training of OccSim’s two components—W-DiT and the Layout Generator—was conducted separately as follows. For W-DiT, we trained for 200 epochs on 4 Blackwell RTX Pro 6000 GPUs with a batch size of 32. Training on the Nuscenes dataset took approximately 100 GPU hours. For the Layout Generator, we trained for 500 epochs on 8 A100 80G GPUs, taking approximately 160 GPU hours. All of the experiments used AdamW with cosine annealing, decaying the learning rate from 3.2e-5 to 3.2e-6. In Eq. 2, we set λ=2λ=2, with 20% possiblilty to drop condition during training for classifier free guidance [ho2022classifier] in sampling stage. When generating randM_rand, we control the maximum of overall masked region maskM_mask between 10% to 50% of the full latent size. ϵε in Eq. 2 is sampled from sigmoid((0,I))sigmoid(N(0,I)). 6.2 Augmentation in Layout generator Theoretically, the distribution of valid vehicle behaviors given a specific map representation, denoted P(ℋ|t)P(H|z_t), is inherently highly multimodal, as various driving intentions can perfectly align with the same static environment. However, direct optimization on a finite training dataset =(t(i),ℋ(i))i=1ND=\(z_t^(i),H^(i))\_i=1^N poses a severe risk of overfitting, as the number of samples is only 28K to train the DiT-S. In practice, for any specific scene t(i)z_t^(i) in the log, we only observe a single deterministic realization of the vehicles’ behavior ℋ(i)H^(i). Consequently, without proper regularization, model tend to memorize these exact map-action pairs, effectively collapsing the learned conditional distribution into a Dirac delta function, Pθ(ℋ|t(i))≈δ(ℋ−ℋ(i))P_θ(H|z_t^(i))≈δ(H-H^(i)). To prevent the model from memorizing spurious correlations between specific voxel arrangements and deterministic position, we introduce the data augmentation in layout generator training. By applying paired spatial transformations T to both the condition and the target, i.e., ((t),(ℋ))(T(z_t),T(H)), we artificially expand the support of the empirical distribution. Specifically, we introduce a two-stage data augmentation pipeline comprising agent-level perturbations and global spatial transformations. First, to manage scene density and prevent the model from overfitting to overly crowded environments, we cap the maximum number of vehicles per frame at 10; if a frame contains N>10N>10 vehicles, we uniformly sample a subset of exactly 1010 vehicles without replacement, ensuring each vehicle has an equal retention probability of 10/N10/N. For each retained vehicle, we project it onto a 2D spatial heatmap ℋ∈ℝ200×200×1H ^200× 200× 1 as a Gaussian distribution. Specifically, the Gaussian is centered at the vehicle’s spatial coordinates with a kernel radius of 3. To simulate positional uncertainty and enrich the local behavioral manifold, we independently perturb the center of each Gaussian, randomly shifting it within a local 5×55× 5 neighborhood restricted to the drivable surface area. Following this localized perturbation, we apply rigid global transformations to the entire augmented scene (including both the occupancy map and the constructed heatmap ℋH). We uniformly sample and apply discrete rotations ℛ∈90∘,180∘,270∘R∈\90 ,180 ,270 \ and axis-aligned reflections and axis-aligned reflections (flipping along the x- or y-axis). Crucially, identical spatial transformations are applied to the corresponding latent representations to maintain spatial alignment with ℋH. While such reflections fundamentally invert the handedness of the traffic rules (effectively synthesizing valid left-hand traffic scenarios from right-hand data), this joint transformation ensures that all local topological constraints and relative vehicle dynamics remain physically coherent. This composite strategy effectively artificially expands the empirical distribution, forcing the network to learn robust, viewpoint-invariant geometric representations. 6.3 Additional metrics details To evaluate the diversity of the generated stochastic rollouts, we employ two distinct metrics: Pairwise mIoU Diversity in the decoded semantic space, and the Vendi Score [friedman2022vendi] in the latent feature space. For the decoded semantic space, we generate N independent sequences over T frames using different random seeds. At any given timestep t, we compute the mean Intersection over Union (mIoU) across all semantic categories for every unique pair of generated occupancy grids. The Pairwise mIoU Diversity at time t is then defined as the complement of the average pairwise similarity: Dt=1−1||∑(i,j)∈mIoUt(i,j)D_t=1- 1|P| _(i,j) mIoU_t^(i,j) where =(i,j):1≤i<j≤NP=\(i,j):1≤ i<j≤ N\ denotes the set of all unordered seed pairs. The rollout-level diversity is obtained by averaging DtD_t over time. To capture structural diversity in the latent space, we report the Vendi Score, which measures the effective sample size of the generated distributions. First, we extract frame-level features xii=1N\x_i\_i=1^N using pretrained encoders (i.e., the OccFM VAE encoder and the UniScenes AE encoder). We then compute a cosine similarity kernel matrix Q, where its elements are given by ij=(1+x~i⊤x~j)/2Q_ij=(1+ x_i x_j)/2, with x~ x being the ℓ2 _2-normalized feature. The Vendi Score is calculated as the exponential of the von Neumann entropy of the normalized kernel matrix ^=/tr() Q=Q/tr(Q): V=exp(−∑ℓλℓlogλℓ)V= (- _ _ _ ) where λℓ\ _ \ are the non-negative eigenvalues of Q. Intuitively, the Vendi Score quantifies the effective number of distinct modes within a given set of generations. By evaluating the entropy of the similarity eigenspectrum, it properly penalizes highly correlated samples. A higher Vendi Score indicates that the generated occupancy features are widely distributed across the latent space, reflecting true structural and geometric diversity rather than minor variations of a single collapsed mode. 7 Additional ablation experiments 7.1 Inference speed and condition comparison In the previous section, we demonstrated a comparison between our method and prior approaches in terms of realism. However, it is worth noting that our method also achieves significant improvements in real-time performance compared to them. In Tab. 3, we compares the time and FLOPS required for our method to generate a single frame. Table 3: Comparison of efficiency and computational cost among different methods. When evaluating the time taken by our method, we distribute the post-processing time evenly across each frame of the sequence generation. Each result was computed independently using an RTX4090. DOME and COME generate 6 frames together, ours is 1 single frame per step, for fair comparison, we keep measure the GFLOPS / inference time needed per single frame. Method WM Input #Params Framessecond Framessecond GFLOPSframe GFLOPSframe DOME ✓ 4 frames sem. occ.&traj 444.07M 5.48 8891.98 COME ✓ 4 frames sem. occ.&traj 204.53M 2.33 41333.17 InfiniCubes ✕ 1 frames sem. occ. 831.75M 0.02 990927.28 OccSim (Ours) ✓ 1 frames sem. occ. & traj 181.24M 1.47 15570.18 8 Additional visualization 8.1 Visualization of long-horizon rollouts From Fig. 7, we can clearly observe that our proposed W-DiT significantly extends the duration of stable rollout. Even after 3000 frames (1500s, approximately 25 minutes), the generated occupancy remains stable. Figure 7: Qualitative comparison of two rollouts for two models: previous SOTA occupancy world model (first and third rows), and our model (second and fourth rows) at different horizons. Ours exhibits long-horizon stability under different trajectory input. 8.2 Visualization of trajectory shapes used in Fig. 5 To systematically evaluate the robustness of our model in long-horizon generation across diverse driving behaviors, we configure three distinct spatial trajectory profiles: Loop, Straight, and Turn. As illustrated in Fig. 8, the trajectory generation process conditioned on is consist of the initial ground-truth (GT) trajectory (highlighted in blue, encompassing frames 1–40) and extended trajectory with 3 modes (red lines). Figure 8: Visualization of trajectory shapes (Loop, Straight, and Turn) evaluated in Fig. 5. The green star indicates the starting point. The blue solid line denotes the initial ground-truth (GT) trajectory snippet (frames 1–40) used as the condition prompt, while the red dashed line represents the extended trajectory explicitly designed to guide the long-horizon generation. For visual clarity the displayed trajectories are truncated at 500 frames, whereas our actual synthesis spans up to 3000 frames. 8.3 Visualization of chaos baseline constructed As shown in Fig. 9, the constructed “chaos” baseline primarily comprises severely fragmented road segments, topologically disconnected layouts, and large-scale empty regions. Consequently, as we mentioned in Fig. 5, any model yielding FID/KID scores above this baseline threshold can be deemed completely incapable of synthesizing large-scale, valid road networks. Figure 9: Visualization of frames from the “chaos” baseline. We randomly visualize 20 frames from the 1000 samples pool. The semantic categories represented by colors here are consistent with those in Fig. 3. 9 Algorithm details 9.1 Heuristic for Single frame fusion. After we obtained the raw forecast observation iiK\O_i\_i^K from the W-DiT based world model, we than use the following algorithm 1 to fuse all frames to ℳglobalM_global. Algorithm 1 Two-pass Keyframe-based Occupancy Fusion 1:Local predictions ^tt=1T\ O_t\_t=1^T, Ego-poses tt=1T\T_t\_t=1^T, max step dmaxd_ , vote threshold τ, number of semantic categories C. 2:Fused global occupancy map ℳglobal∈0,1,…,CX×Y×ZM_global∈\0,1,…,C\^X× Y× Z. 3:// Phase 1: SE(2) Projection & Keyframe Selection 4:t(x,y)←ProjSE(2)(t),∀t∈1…TT_t(x,y) _SE(2)(T_t), ∀ t∈\1… T\ ⊳ Align to BEV plane 5:←1,last←1(x,y)K←\1\, _last _1(x,y) 6:for t=2t=2 to T do 7: if ‖t(x,y)−last‖2>dmax\|T_t(x,y)-p_last\|_2>d_ then 8: ←∪t,last←t(x,y)K ∪\t\, _last _t(x,y) 9:// Phase 2: Pass 1 - Keyframe Sinking & Fusion 10:Initialize ℳglobal()←∅,∀∈ℤX×Y×ZM_global(v)← , ^X× Y× Z ⊳ ∅ denotes unassigned. 11:for k∈k do 12: k′←(k^,k)O_k ( O_k,T_k) ⊳ Spatial warping to global coordination 13: ⊳ Assign key-frame voxels from local map to global map. 14: ℳglobal()←k′(),∀ s.t. ℳglobal()=∅M_global(v) _k(v), s.t. M_global(v)= 15:Δz(x,y)←minz∣ℳglobal(x,y,z)∈ground z(x,y)← \z _global(x,y,z) _ground\ 16:ℳglobal(x,y,z)←ℳglobal(x,y,z+Δz)M_global(x,y,z) _global(x,y,z+ z) ⊳ Column sinking to lowest ground 17:ℳglobal()←Mode(3×3())M_global(v) (N_3× 3(v)) for v where ℳglobal()=∅,z=0M_global(v)= ,z=0 18:// Phase 3: Pass 2 - Non-Keyframe Voting Inpainting 19:Initialize vote tensor ∈ℕX×Y×Z×C←V ^X× Y× Z× C 0 20:for t∉t do 21: t′←(^t,t)O_t ( O_t,T_t) 22: (,c)←(,c)+[t′()=c],∀ s.t. ℳglobal()=∅V(v,c) (v,c)+I[O _t(v)=c], s.t. M_global(v)= 23:ℳglobal()←argmaxc(,c)M_global(v)← _cV(v,c) if maxc(,c)≥τ _cV(v,c)≥τ 24:// Phase 4: Morphological Refinement 25:ℳglobal←CloseOp(ℳglobal,csidewalk)M_global (M_global,c_sidewalk) ⊳ Binary closing on sidewalk 26:ℳglobal←FilterAreaOp(ℳglobal,croad,A<2m2)M_global (M_global,c_road,A<2m^2) ⊳ Remove isolated noise 27:return ℳglobalM_global The pipeline consists of four primary phases. First, to manage computational overhead and redundancy, we subsample the generated sequences into keyframes based on a spatial distance threshold, projecting them onto a unified SE(2) bird’s-eye-view (BEV) plane. Second, during the initial fusion pass, we introduce a deterministic column sinking operation. This forces all predicted ground-level voxels to align with the absolute zero-elevation plane (z=0z=0), effectively rectifying pitch and roll artifacts or elevation drifts accumulated over long-horizon rollouts. Third, to address occlusions and local generation artifacts, we leverage the temporal redundancy of non-keyframes; by applying a spatial majority voting mechanism, we robustly inpaint the remaining gaps on the z=0z=0 plane. Finally, a morphological refinement stage—comprising binary closing and connected-component area filtering—is applied to enforce spatial continuity on sidewalks and systematically remove isolated noise clusters (e.g., artifacts smaller than 2m22m^2). This pipeline ensures a connected and smooth road topology, which is strictly required for downstream 2D multi-agent simulations. 9.2 Branch road detection and topology extraction. Then, to bridge the gap between discrete occupancy grids ℳglobalM_global and vectorized road networks required for downstream multi-agent simulation, we propose a robust topology extraction and graph refinement algorithm. We first extract the 2D drivable surface mask, apply connected-component filtering, and perform strict single-pixel skeletonization to initialize the graph structure. To mitigate rasterization artifacts, we systematically eliminate spurious triangular loops by removing the longest edges within 3-cliques. Subsequently, a rigorous graph cleaning protocol is executed: we iteratively prune short, noisy spurs and introduce a node contraction mechanism to merge spatially dense junction clusters into unified super-nodes, accurately reflecting complex real-world intersections. Finally, to precisely isolate valid map ingress and egress points, we design a novel Dual-Probe filtering mechanism. A topology probe identifies and discards artificial endpoints caused by internal fragmentation, while a semantic probe leverages the 3D elevation profile of the occupancy map to detect physical occlusions or obstacles along the projected path. Ultimately, this pipeline translates the generated voxel map into a high-fidelity, deadlock-free topological graph, providing a reliable foundation for downstream closed-loop planning and control. Algorithm 2 Branch point detection / topology extraction 1:Fused global map ℳglobal∈0,…,CX×Y×ZM_global∈\0,…,C\^X× Y× Z, lane width wlanew_lane, pruning threshold τprune _prune. valid←∅V_valid← for valid branch node set in the graph. 2:Vectorized road graph =(V,E)G=(V,E), valid endpoints validP_valid. 3:// Phase 1: Binary Masking & Skeletonization 4:ℬ(x,y)←[ℳglobal(x,y,0)=croad],∀(x,y)∈ℤ2B(x,y) [M_global(x,y,0)=c_road], ∀(x,y) ^2 5:←Skeletonize(ℬ)S (B) 6:(x+1,y+1)←0,∀(x,y) s.t. ∑i=01∑j=01(x+i,y+j)=4S(x+1,y+1)← 0, ∀(x,y) s.t. _i=0^1 _j=0^1S(x+i,y+j)=4 7:// Phase 2: Graph Initialization & Artifact Removal 8:V←∈ℤ2∣()=1V←\v ^2 (v)=1\ 9:E←(,)∣,∈V,‖−‖∞=1E←\(u,v) ,v∈ V,\|u-v\|_∞=1\, with weight w(,)=‖−‖2w(u,v)=\|u-v\|_2 10:Initialize graph ←(V,E)G←(V,E) 11:for each 3-clique (triangle) K3∈K_3 do 12: emax←argmaxe∈K3w(e)e_max← _e∈ K_3w(e) 13: E←E∖emaxE← E \e_max\ ⊳ Break loops to expose leaf nodes 14:// Phase 3: Iterative Spur Pruning & Junction Contraction 15:repeat 16: leaf←v∈V∣deg(v)=1V_leaf←\v∈ V (v)=1\ 17: Find branch path Pl→jP_l→ j from l∈leafl _leaf to nearest junction j (deg(j)>2deg(j)>2) 18: if ∑e∈Pw(e)<τprune _e∈ Pw(e)< _prune then 19: V←V∖(Pl→j∖j)V← V (P_l→ j \j\) ⊳ Remove short spurs 20:until no spurs are removed 21:junc←v∈V∣deg(v)>2V_junc←\v∈ V (v)>2\ 22:for ,∈juncu,v _junc s.t. ‖−‖2<2⋅wlane\|u-v\|_2<2· w_lane do 23: ←ContractNodes(,,)G (G,u,v) ⊳ Merge clustered junctions 24:// Phase 4: Dual-Probe Endpoint Filtering 25:obstacle←(ℳglobal(x,y,z≥1)∉croad,csidewalk,∅)O_obstacle←(M_global(x,y,z≥ 1)∉\c_road,c_sidewalk, \) ⊳ Get 3D obstacle occ above ground layer 26:end←v∈V∣deg(v)=1V_end←\v∈ V (v)=1\ 27:for v∈endv _end do 28: Let d→ d be the outward normalized direction vector from parent junction to v. 29: Topology Probe: ptopo←v+(1.5⋅wlane)d→p_topo← v+(1.5· w_lane) d 30: Semantic Probe: Generate 15m×3.6m15m× 3.6m bounding box BsemB_sem along d→ d from v. 31: if ℬ(ptopo)==0B(p_topo)==0 and ∑∈Bsem[3D()]<τobs _p∈ B_semI[O_3D(p)]< _obs then 32: valid←valid∪vV_valid _valid∪\v\ ⊳ Endpoint is neither internal nor blocked 33:return G, validV_valid 9.3 Lane topology extraction. After we have G, validV_valid, we use the retained graph information to create lane graphs. As shown in Alg. 3, based on the extracted skeleton, i.e., E∈E , we introduce a two-phase lane topology extraction algorithm (Algorithm 3). In the first phase, we convert the raw skeletal segments into kinematically smooth reference centerlines using 3rd-degree B-Spline interpolation with a fixed spatial resolution Δsstep s_step. To reconstruct multi-lane structures, we generate explicit parallel lanes via orthogonal offsetting along the normal vectors of the reference trajectories. To guarantee physical feasibility, we strictly enforce a topological boundary constraint: the interior of any newly generated lane must be fully bounded by the drivable surface mask ℬB. In the second phase, we address the geometric overlaps that inevitably occur at complex intersections where multiple segments converge. We formulate this as a spatial intersection resolution problem. For each candidate lane, we identify conflict points that fall within a minimal distance threshold ϵε of any other independent lane. We then sever the trajectories at these overlapping junctions and exclusively retain the longest continuous sub-trajectory. Finally, a secondary B-Spline interpolation is applied to eliminate any geometric artifacts or non-differentiable kinks introduced by the splitting process. This rigorous pipeline yields a highly precise, non-overlapping lane network, providing a kinematically reliable foundation for multi-agent closed-loop simulation. Algorithm 3 Lane Topology Extraction 1:Extracted segments S from G, drivable mask ℬB, lane width wlanew_lane, distance threshold ϵε, resampling step Δsstep s_step. 2:Vectorized explicit lanes ℒfinalL_final. 3:// Phase 1: B-Spline Smoothing & Parallel Offsetting 4:ℒcand←∅L_cand← 5:for each S∈S s.t. |S|≥10|S|≥ 10 do 6: c←BSpline(Spath,deg=3,Δsstep)P_c (S_path,deg=3, s_step) 7: ←NormalVectors(c)N (P_c), n←⌊Swidth/wlane⌋−1n← S_width/w_lane -1 8: ←(i−n−12)wlane|i∈0,…,n−1O← \ (i- n-12 )w_lane\; |\;i∈\0,…,n-1\ \ 9: for o∈o do 10: l←c+o⋅P_l _c+o·N ⊳ Generate parallel lane curves 11: if l∘⊂ℬP_l then ⊳ l∘P_l denotes the interior of the trajectory 12: ℒcand←ℒcand∪lL_cand _cand∪\P_l\ 13:// Phase 2: Spatial Intersection Resolution & Resmoothing 14:for each L∈ℒcandL _cand do 15: L←∈L∣∃L′∈ℒcand∖L,dist(,L′)<ϵC_L←\p∈ L ∃ L _cand \L\,dist(p,L )<ε\ ⊳ Identify cross-lane overlaps 16: L←argmaxLsub⊆(L∖L)|Lsub|L← _L_sub (L _L)|L_sub| ⊳ Retain the longest continuous sub-trajectory 17: L←BSpline(L,deg=3,Δsstep)L (L,deg=3, s_step) ⊳ Smooth artifacts caused by splitting 18:ℒfinal←L∈ℒcand∣|L|≥5L_final←\L _cand |L|≥ 5\ ⊳ Remove all lanes short than 5 pixels 19:return ℒfinalL_final Algorithm 4 Spatially-conditioned agent generation and routing initialization 1:Query anchor pose ∈SE(2)T∈ SE(2) with spatial coordinate p_T, boolean flag begob_ego indicating whether to explicitly initialize the ego vehicle at T, mean/standard of initial velocity μv,σv _v, _v, ℳglobalM_global as global map, validV_valid from Alg. 2, Agents asset pool X. 2:Crop local occupancy O_T from ℳglobalM_global by pose T. 3:new←LayoutGenerator()∪( if bego else ∅)A_new (O_T)∪ (\p_T\ if b_ego else ) 4:for each agent a∈newa _new parameterized by pose ap_a do 5: Initialize velocity va←vdes∼(μv,σv)v_a← v_des ( _v, _v) 6: target∼Uniform(valid)p_target (V_valid) ⊳ Random valid endpoint 7: a∼Uniform()x_a (X) ⊳ Random agent asset 8: πtraj←A*(ℒfinal,a,target) _traj *(L_final,p_a,p_target) ⊳ Shortest valid path 9: a←Tangent(πtraj,a)h_a ( _traj,p_a) ⊳ Generate initial headings from trajectory 10: a←(a,a,va,πtraj,a)a←(p_a,h_a,v_a, _traj,x_a) ⊳ Assign initial states to agent 11:return newA_new 9.4 2D-IDM based simulation. With all geometric information obtained, we introduce a 2D-IDM based simulation in Alg. 5. The algorithm is structurally divided into two phases: global pre-computation (Phase 1) and dynamic step-by-step execution (Phase 2). Phase 1 employs a proactive spatial pre-computation strategy. Rather than solely initializing traffic around the starting ego-pose (0egoT_0^ego), the engine seamlessly injects background agents at forward and backward anchor poses (fwdT_fwd, bwdT_bwd) extending up to a predefined horizon dpred_pre. The unified SpawnAgents procedure handles this layout generation, simultaneously sampling valid endpoints, predicting spatial coordinates, and assigning initial shortest paths via A* routing. In the simulation part, algorithm˜5 Line 22 defines the spatial identification of candidate leading vehicles (aC_a). Instead of relying on heuristic bounding box checks, an agent a filters surrounding vehicles using a normalized vector dot product, ((a′−a)⋅a‖a′−a‖2>0.5) ( (p_a -p_a)·h_a\|p_a -p_a\|_2>0.5 ), to ensure the target is strictly within a 60∘60 forward field-of-view cone. This is coupled with a trajectory distance constraint (Dist(a′,πtraj)<dlatDist(p_a , _traj)<d_lat) to confirm the target is actively blocking the planned route. Furthermore, algorithm˜5 Line 25 explicitly shows the lane-changing logic. An agent initiates a Bezier-smoothed lane transition and A* re-routing only when the longitudinal distance falls below a critical threshold (s<dlcs<d_lc) under two specific risks: either the ego agent is approaching a slower leader (Δv>0 v>0), or it encounters a wrong-way/head-on vehicle. The latter scenario is elegantly captured by the negative dot product of their respective heading unit vectors (a⋅alead<−0.5h_a·h_a_lead<-0.5). Finally, the kinematic states are updated via IDM, and the dynamic 3D assets are sequentially rendered onto the cropped static background using a spatial overwrite operator (⊛ ) to handle semantic Z-buffering (Step 2.3), ultimately producing a seamless sequence of dynamic local environments. Algorithm 5 A* routing + 2D-IDM based control engine pseudo codes 1:A pool of global maps M generated, IDM parameters, rolling horizon threshold drolld_roll, simulation horizon ϕφ, Lane topology ℒfinalL_final, Δt t for Inter-frame time interval, FOV()FOV(T): The set of all spatial coordinates within the perception range of the ego vehicle at pose T, Agents asset pool X, dlcd_lc for distance threshold of lane change. SpawnAgents: method generate initial state of agents, details in Alg. 4. 2:Rendered sequence of dynamic local environments ℳlocal(t)t=1ϕ\M_local^(t)\_t=1^φ. 3:// Phase 1: Global Initialization & Pre-computation 4:Randomly select a global map ℳglobal∼Uniform()M_global (M). 5:Get ego-poses tt=1T\T_t\_t=1^T used to build ℳglobalM_global in Alg. 1. 6:Randomly select initial ego pose 0ego∼Uniform(tt=1T)T_0^ego (\T_t\_t=1^T). 7:Find fwdT_fwd and bwdT_bwd at distance ±dpre± d_pre along tt=1T\T_t\_t=1^T from 0egoT_0^ego. 8:←SpawnAgents(0ego,True)∪SpawnAgents(fwd)∪SpawnAgents(bwd)A (T_0^ego,True) (T_fwd) (T_bwd) 9:// Phase 2: Closed-Loop Simulation 10:Δdego←0,out←∅ d_ego← 0, _out← ⊳ Init movement tracker and empty sequence for frames 11:for t=1…ϕt=1…φ do 12: // Step 2.1: Rolling Horizon Update 13: Δdego←Δdego+vego(t)Δt d_ego← d_ego+v_ego^(t) t 14: if Δdego≥droll d_ego≥ d_roll then 15: ←a∈∣a∈FOV(tego)A←\a _a (T_t^ego)\ ⊳ Remove out-of-view agents 16: fwd←Pose ahead of tego by dpreT_fwd ahead of T_t^ego by d_pre, bwd←Pose behind of tego by dpreT_bwd behind of T_t^ego by d_pre 17: ←∪SpawnAgents(fwd)∪SpawnAgents(bwd)A (T_fwd) (T_bwd) 18: Δdego←0 d_ego← 0 19: // Step 2.2: IDM Kinematics & Route Execution 20: for each agent a∈a parameterized by state (a,a,va,πtraj,target)(p_a,h_a,v_a, _traj,p_target) do 21: Let ah_a denote the heading unit vector of agent a. 22: a←a′∈∖a|(a′−a)⋅a‖a′−a‖2>0.5∧Dist(a′,πtraj)<dlatC_a← \a \a\\; |\; (p_a -p_a)·h_a\|p_a -p_a\|_2>0.5 (p_a , _traj)<d_lat \ 23: alead←argmina′∈a‖a′−a‖2a_lead← _a _a\|p_a -p_a\|_2 ⊳ Select out the agent ahead of current a 24: s←‖alead−a‖2s←\|p_a_lead-p_a\|_2, Δv←va−vlead v← v_a-v_lead ⊳ adjusted if head-on 25: if s<dlcs<d_lc and (Δv>0 or a⋅alead<−0.5)( v>0 or h_a·h_a_lead<-0.5) then 26: adj←SearchParallelLane(a,ℒfinal)p_adj (p_a,L_final) 27: if adj≠∅p_adj≠ then 28: πbezier←BezierCurve(a,adj) _bezier (p_a,p_adj) ⊳ Generate smooth transition 29: πnew←A-Star(ℒfinal,adj,target) _new -Star(L_final,p_adj,p_target) ⊳ Re-route from adjacent lane 30: πtraj←πbezier⊕πnew _traj← _bezier _new ⊳ Concatenate trajectories 31: aacc←IDM(va,vdes,Δv,s)a_acc (v_a,v_des, v,s) ⊳ Compute acceleration 32: a←Advance(a,πtraj,vaΔt)p_a (p_a, _traj,v_a t) ⊳ Move along trajectory 33: // Step 2.3: Rendering 34: vis←a∈∣a∈FOV(tego)A_vis←\a _a (T_t^ego)\ ⊳ Filter visible agents in current view 35: (t)←Crop(ℳglobal,tego)O_T^(t) (M_global,T_t^ego) ⊳ Extract static local background 36: (t)←⋃a∈visΦrigid(§a,a,a)X^(t)← _a _vis _rigid( _a,p_a,h_a) ⊳ Transform and aggregate dynamic assets 37: ℳlocal(t)←(t)⊛(t)M_local^(t) _T^(t) ^(t) ⊳ Overlay foreground onto static background 38: out←out∪ℳlocal(t)S_out _out∪\M_local^(t)\ ⊳ Append rendered frame to sequence 39:return outS_out 10 Potential limitation and Future work Although OccSim provide the first implementation of using occupancy world model in simulation, we still observe the following limitation in design and evaluation period of OccSim: • Heuristic based fusion and lane graph extraction: Although the robustness of our multi-frame fusion/lane graph extraction algorithm has been qualitatively demonstrated in numerous practical applications, given the occasional instability in the quality of the raw occupancy frames output by the model, we still seek an end-to-end fusion approach rather than frame-by-frame fusion. • Reliance on the rotation in-variance of the latent space: As mentioned in the previous discussion, one of the assumptions underlying W-DiT is the rotational invariance of the latent space. Although our experiments demonstrate that the minor errors introduced by the Warp+interpolation operation do not cause the model to collapse after U-Net correction, we still look forward to future work that can perform lossless rotational operations directly in the occupancy space without relying on the latent. • Limited training data: Compared to image data, semantic occupancy data requires manual semantic annotation, which limits the total amount of available training data to fewer than 100,000 frames. This volume is significantly smaller than the amount of available RGB video data, and it also restricts our ability to explore the upper limits of the model’s performance. References