Paper deep dive
ReWorld: An Interactive World Model with Long-Horizon Memory
Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang, Tianyi Gui, Lianghua Huang, Yingcong Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An interactive world model must follow the user's actions, remember the places it has shown, and stream in real time. The tension is structural: control wants a short horizon, memory wants an unbounded one. ReWorld separates the two during training and bounds them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sources -- Unreal-rendered fly-throughs, game roaming, and real-world footage -- on one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704x1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity ($11.95^\circ$ rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts ($64$\,s, $384$ latents), its fixed 12-chunk cache still regenerates the starting view -- at rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory.
Tags
Links
- Source: https://arxiv.org/abs/2608.23565v1
- Canonical: https://arxiv.org/abs/2608.23565v1
Trouble viewing inline? Open PDF directly ā
Full Text
74,133 characters extracted from source content.
Expand or collapse full text
1]HKUST(GZ) 2]ATH, Alibaba [*]Equal contribution [ā ]Corresponding author [Website]https://zhifeichen097.github.io/ReWorld/ Figure 1: ReWorld follows camera intents (top), regenerates a revisited view after an out-and-back excursion (middle), and streams long rollouts (bottom). ReWorld: An Interactive World Model with Long-Horizon Memory Zhifei Chen Luozhou Wang Guibao Shen Dongyu Yan Shuai Yang Tianshuo Xu Yihua Du Wei Wang Tianyi Gui Lianghua Huang Yingcong Chen Affiliation: [ Affiliation: [ Abstract An interactive world model must do three things at once: follow the userās actions, remember the places it has already shown, and keep streaming in real time. The tension is structuralācontrol wants a short horizon, memory wants an unbounded one. ReWorld resolves it by separating the two during training and bounding them at inference. Mixed per-head attention windows confine most heads to the recent past while a small set of global heads attends over the entire history, and random head routing keeps either capability from binding to particular heads; random chunk dropping makes sparse histories in-distribution. At inference the whole past lives under a fixed budget: a bounded KV cache backed by a pose-indexed landmark bank, from which the model retrieves the landmarks nearest the current pose. A metric-scale-aligned data engine places eight sourcesāUnreal-rendered fly-throughs, game roaming, and real-world footageāon one physical action scale, so the same key press moves the camera the same distance in every source, and palindrome trajectories supply the revisit evidence that memory training needs. Distribution-matching distillation confined to a LoRA adapter then compresses sampling to four steps: one backbone serves both a high-fidelity multi-step mode and a real-time interactive one, streaming 704Ć1280704Ć 1280 video across photorealistic, game-style, and stylized worlds. Under a three-axis protocol covering action following, long-horizon recall, and video quality, against six recent interactive world models it attains the best control fidelity (11.95ā11.95 rotation error and the best camera-motion consistency) and the best generation quality; and on minute-long out-and-back rollouts (6464 s, 384384 latents), its fixed 12-chunk cache still regenerates the starting viewāat rollout lengths where a sliding window has long evicted the evidence and full-KV attention runs out of memory. ā footnotetext: The project was done during an internship at Alibaba. 1 Introduction A world model simulates an environment an agent can act in [10]. Recent interactive systems turn a stream of user actions into video of a coherent, explorable environment [2, 27, 30, 35, 29]. Expectations have converged: the model should react, reflecting a key press in the very next frames; remember, so that a revisited place looks the way it did before; and stream, generating at interactive rates over unbounded rollouts. Current systems pursue these goals along two axes. For control, one route folds camera pose into attention, so attention logits depend on relative pose [39, 7]; the other directly injects an explicit action signal [30, 35]. For memory, camera pose is the retrieval key of choice, in three forms: selecting which past frames re-enter the context [44], indexing an external memory bank [38, 36], or entering the attention index itself [39]. Learning a world model comes down to learning two abilities: control, so that the next frames follow the current action, and memory, so that a revisited place looks the way it did before. When the two are trained together, control learns well but memory does not: adding direct action injection improves every control metric while revisit fidelity drops (Sec. 4.5, Table 7). The two abilities also ask for different attention windows. Control should not depend on the window size: at inference the model must respond to the current action correctly whatever window it is given, and what the response requiresāthe current scene and the current commandāsits inside a short window. Memory is the opposite: retrieval can only be learned under a long window, because a model that cannot see the far past has nothing to retrieve from. The learning of the two abilities can therefore be split by window: control learned under short windows, memory under long ones. This paper presents ReWorld, an action-controllable streaming world model designed in two steps: split the training of control and memory by window, then consolidate memory under a fixed KV budget at inference. We keep both conditioning channels: pose-indexed attention (MRoPE) finds cached content by pose, and direct action injection states the command. What we separate is the window each attention head trains under. Mixed attention windows give the model both window lengths in every iteration: most heads are local and attend only to a short recent window, while a small set of global heads attends over the entire causal past. Random head routing then switches which heads are global at every step, cycling through a fixed pool of random partitions. The routing is what makes the split workable. The split itself cannot be kept at inference: the full history is gone, and all that remains is one bounded cache that every head reads, so there is no way to give some heads a long window and others a short one. A fixed partition would train heads to specialize in windows that deployment cannot provide. Routing removes this dependence: which heads are global changes at every step, every head trains under both windows, and neither ability binds to particular headsāat inference any head can read the shared bounded cache. The recipe adds no parameters, losses, or modules, and it makes a testable prediction: swapping cache-compression policies at inference should leave control error unchanged (Sec. 4.5). With control insulated, the second step strengthens memory at inference: the rollout is unbounded but the cache is not. ReWorld therefore consolidates, keeping few chunks but keeping them intact: a chunk aging out of the recent window is stored in a bounded landmark bank only if the camera has travelled far enough since the last stored landmarkāone full-resolution snapshot per stretch of camera travelāand once the bank is full, each admission evicts the member most spatially redundant with the rest. Retrieval then fills the fixed cache with a sink chunk [37], a recent window, and the landmarks closest to the current camera pose. Inference thus reads a sparse, non-contiguous history, while standard training sees only a complete, contiguous prefixāa trainātest mismatch [14]. Chunk-drop training closes the gap: a random subset of past chunks is masked at every step, teaching the model to reconstruct scene state from incomplete memory and making the spliced caches of deployment in-distribution (Secs. 2.3 and 2.3). Two components complete the system. A metric-scale-aligned pipeline places synthetic, real, and game footage on a single physical action scale, so the same key press moves the camera the same physical distance in every source, with palindrome trajectories supplying the revisit evidence memory training needs (Sec. 3). Distribution-matching distillation with self-forcing rollouts [43, 42, 14] compresses sampling into a few denoising steps inside a LoRA adapter [11], so one backbone serves both a high-fidelity multi-step operating point and a real-time interactive one (Sec. 2.4). In summary, our contributions are: ā¢ Window-split training of control and memory. Mixed per-head attention windows train control under short windows and memory under long ones, and random head routing keeps control in every head complete within the short window, so cache compression at deployment does not disturb action following (Secs. 2.2 and 4.5). ā¢ Chunk-drop training with consolidation-based inference. A training augmentation that makes sparse, non-contiguous KV caches in-distribution, paired with an inference mechanism that consolidates aged chunks into a landmark bank kept bounded by redundancy-based eviction and retrieves them by pose proximity, so spatial memory persists over unbounded rollouts at a fixed KV budget (Secs. 2.3 and 2.3). ā¢ A metric-aligned multi-source data pipeline. Synthetic, real, and game footage aligned to one physical action scale, with palindrome augmentation providing revisit supervision (Sec. 3). ā¢ Real-time deployment and evaluation. LoRA-confined few-step distillation that gives a single backbone both a high-fidelity and a real-time operating point, together with an evaluation protocol covering action following, long-horizon recall, and general video quality (Secs. 2.4 and 4). 2 Method 2.1 Overview ReWorld is an action-controllable streaming world model: given a text prompt, an optional reference image, and a stream of 6-DoF camera actions, a causal flow-matching diffusion transformer [22, 8] generates video one latent chunk at a time, each chunk driven by a per-chunk camera action. The generated stream must both follow the commanded trajectory and remain spatially consistent with everything already generatedāfor instance, when the camera revisits a location seen long ago. The two requirements are learned under different conditions: action following needs only what a short window holds, while spatial memory can be learned only when distant content is visible in the attention window. Deployment adds a hard constraint: the KV cache holds a limited number of chunks, so the model cannot simply attend to its full history. ReWorld meets both requirements with a small set of co-designed components. The camera-control section builds up from pose-indexed attention (MRoPE) to direct action injection, then trains control under short windows and memory under long ones with mixed per-head attention windows and random head routing (Sec. 2.2); memory consolidation then strengthens what the decoupling protectsāat inference a pose-retrieved landmark cache fills the fixed budget with the most relevant old chunks (Sec. 2.3), and chunk-drop training makes attention robust to the sparse caches this policy produces (Sec. 2.3); four-step LoRA distillation makes the pipeline real time (Sec. 2.4)āthe overall development route, AR-training the base model then plugging in a step-distilled LoRA, follows LongLive-2.0 [5]; and a metric-aligned, eight-source data mixture lets a single action space transfer across synthetic and real footage (Sec. 3). Figure 2 summarizes the design. Figure 2: Overview of ReWorld. Left: a metric-aligned data pipeline places UE-rendered, real-world, and game footage on a single physical action scale, and palindrome routesāthe camera retracing its own pathāsupply revisit supervision (Sec. 3). Middle: teacher-forcing training turns the bidirectional backbone into a streaming world model, with a DMD LoRA trained alongside for few-step real-time inference (Secs. 2.2 and 2.4); control (action injection, local short-window heads) and memory (MRoPE pose-indexed attention, global long-window heads) are decoupled, and the partition into global and local heads is switched every step through a fixed pool of random partitions so that neither capability binds to particular heads (Sec. 2.2); chunk-drop training masks random parts of the history, matching the sparse cache the model will read at inference (Sec. 2.3). Right: at inference, a fixed cache budget holds a sink chunk, pose-retrieved landmarks, and a recent window next to the chunk being generated; chunks aging out of the recent window are consolidated into a bounded landmark bank and retrieved by pose proximity upon revisits, so spatial memory persists over unbounded rollouts at constant cost (Sec. 2.3). 2.2 Camera-Controllable Streaming Generation Backbone and chunked causal generation. ReWorld builds on Wan2.2-TI2V-5B [33], a video diffusion transformer [28] operating in the latent space of a causal VAE. We turn this bidirectional backbone into a streaming generator by imposing chunkwise causality: video is generated as a sequence of latent chunks (L=12L=12 chunks of four latent frames per window), with full attention within a chunk and causal attention across chunks, so at inference the model emits one chunk per denoising pass and appends its keys and values to the cache C. Pose-indexed attention as implicit memory. Standard RoPE indexes attention by time and space, so a revisited location is, positionally, just a distant timestampāretrieval must be inferred from content alone. Our first component therefore builds spatial memory into the attention index itself. We adopt Memory-RoPE (MRoPE), following the camera-phase RoPE design of PM-RoPE [39]: each latent frame f carries a relative camera-to-world pose PfāSEā”(3)P_f (3) (anchored to the first frame) with descriptor cf=[vecā”(Rf);tf]āā12c_f=[vec(R_f);\,t_f] ^12, which a zero-initialized MLP maps to a phase offset Ī“f _f applied on top of RoPE to queries and keys within the same attention pass, q~=RoPEā”(q)āeiāĪ“fā”(q),k~=RoPEā”(k)āeiāĪ“fā”(k),āØq~,k~ā©āeiā”(Ī“fā”(q)āĪ“fā”(k)), q\;=\;RoPE(q)\,e^\,i\, _f(q), k\;=\;RoPE(k)\,e^\,i\, _f(k), q, k \; \;e^\,i\,( _f(q)- _f(k)), (1) so attention depends on pose difference rather than temporal distance: similar viewpoints are pulled together no matter how far apart in time. The cache thus acts as an implicit spatial memoryāeach chunk is stored with the pose it was seen from, and a revisit retrieves it by pose proximity. Zero-initialized SEā”(3)SE(3) residuals on the values and outputs (v~=v+Wvā(Pfā1āv) v=v+W_v(P_f^-1 v), out=Woāy+Wpā(Pfāy)out=W_oy+W_p(P_f y)) complete the conditioning. Unlike the two-pass designs in Table 1, which add a separate camera-aware attention pass so that pose can steer generation, MRoPE uses pose as a retrieval index in the existing pass: the cost is one small MLP plus two linears, and the attention kernel and per-head windows stay untouched. MRoPE alone gives strong revisit memory but imprecise control (Sec. 4.5, Table 7). Table 1: Pose conditioning in attention: design comparison. Two-pass designs such as HY-World 1.5 add a camera-aware attention pass next to the temporal-RoPE pass and fuse the outputs; E-PRoPE trims the second pass to spatially reduced tokens; MRoPE folds pose into the same pass, leaving the attention kernel, mask layout, and per-head windows unchanged. Design Attention index Where pose enters Attn. passes Extra cost Temporal RoPE time & space ā 1 ā HY-World 1.5 (two-pass) [30] time ++ projective pose separate camera-aware pass, fused 2 ā2Ćā2Ć attention FLOPs E-PRoPE [7] time ++ projective pose separate reduced-token camera pass, added back 2 one reduced pass (ā¼4.5Ć 4.5Ć fewer tokens) MRoPE (ours) time, space & pose Q/K phase ++ zero-init V/O residuals 1 1 small MLP ++ 2 linears Action injection. For control authority, we therefore inject the camera command directly. The commanded pose of each latent frame is expanded into a PlĆ¼cker ray map: every spatial position receives the 6-D PlĆ¼cker coordinates [d,oĆd][d,\;oĆ d] of its viewing ray under that pose (direction d, camera center o, expressed in the same first-frame coordinate system as the MRoPE poses). An MLP projects the map to the model width, and the result is added token-wise to the patch embeddings at the transformer input (Figure 2). The two channels now have clearly separated jobs: the ray map tells each token of the current chunk where its camera should look, while MRoPE poses tell attention where every cached token was seen. They still encode the same trajectory, however, and when every head receives both, the action signal crowds out pose-keyed retrieval: control improves, but long-horizon memory degrades (Sec. 4.5). Mixed per-head attention windows. The two abilities need different windows to be learned. Action following does not depend on the window size: the next chunk must answer the current command however much history is visible, and the current scene plus the command fit in a short window. Retrieval, by contrast, can only be learned when distant content is visible. We therefore train under both windows at once (Fig. 3, left): we split the H=24H=24 attention heads of every block into 18 local heads, which attend only to the last w=12w=12 latent frames (three chunks), and a set G of ||=6|G|=6 global heads, which attend to the entire causal history: contextā”(h)=full causal history,hāā(6 global heads),last āw=12ā frames,otherwise (18 local heads).context(h)\;=\; casesfull causal history,&h \ (6 global heads),\\ last w=12 frames,&otherwise \ (18 local heads). cases (2) Every iteration thus trains both abilities at once: local heads learn control under the short window, and global heads are the only heads that can see distant content, so they are where retrieval is learned (a fixed 1:31:3 global-to-local budget). Random head routing. A fixed partition would not survive deployment. At inference the per-head split is hard to realize: the full history is gone, and all heads read the same bounded cache (Sec. 2.3), so no head may be trained to depend on a particular window. We therefore draw a pool P of ||=12|P|=12 random six-head global sets, and every optimizer step switches to the next set in the pool (Fig. 3, right); the 1:31:3 global-to-local ratio is unchanged. Every head thus spends steps in both roles: neither ability binds to particular heads, and since a head is local on most steps, the control it learns must be complete within the short window; retrieval keeps living in whichever heads currently see far. Changing the inference cache policy should then leave control error unchangedāa prediction Sec. 4.5 tests. Figure 3: Mixed per-head attention windows, routed from a pattern pool. Left: local heads attend to the last w=12w=12 frames (reactive control); global heads attend to the full causal history (long-range memory). Right: each optimizer step switches to the next of ||=12|P|=12 random head partitions, so neither ability binds to particular heads. This is the control-side design of ReWorld: pose-indexed attention supplies the memory index, action injection supplies the command, and mixed-window training with random head routing lets each ability learn under the window it needs. Its memory-side counterpart, chunk-drop training and landmark retrieval, follows in the next sections. 2.3 Memory Consolidation under a Bounded KV Budget ReWorld generates video autoregressively over latent chunks zkz_k, so its KV cache C grows linearly with the rollout horizon: at 720p, an unbounded cache exhausts device memory well before the horizons at which spatial memory is actually tested. A deployable interactive world model must therefore read from a cache of constant size while still recalling scene content observed arbitrarily far in the past [44, 30, 35, 36]. ReWorld addresses this by separating the write side of memory from the read side. On the write side, consolidation keeps few chunks and keeps them intact and diverse: aged chunks are admitted selectively into a bounded landmark bank at full resolutionāone snapshot per stretch of camera travelāand the bank is held at its capacity by evicting the member most spatially redundant with the restārather than keeping the whole history at degraded fidelity. On the read side, retrieval spends a fixed budget of B=12B=12 chunks on a pose-indexed working set. A training-time augmentation, chunk drop, makes the backbone robust to the sparse caches this policy produces. The pose-indexed positional encoding of Sec. 2.2 is what makes retrieval effective: because attention keys carry camera-pose structure, a retrieved chunk remains addressable by pose regardless of how distant it is in time. Figure 4: Memory consolidation under a fixed KV budget. Left: at inference the cache holds one sink, six pose-retrieved landmarks, and five recent chunks (B=12B=12); a chunk aging out of the recent window is consolidated into the bounded landmark bankākept sparse and diverseāand re-enters the cache when the camera returns nearby (a full bank evicts its most redundant member). Right: chunk-drop training randomly masks past chunks at every step, so such sparse caches are in-distribution. Landmark consolidation and retrieval. At deployment (Fig. 4, left), ReWorld maintains a bounded cache whose active set at chunk k is k= 1āsinkāŖkā5,ā¦,kā1ārecent windowāŖkāretrieved landmarks,|k|=6,|k|=B=12,C_k\;=\; \\,1\,\_sink\;āŖ\; \\,k-5,ā¦,k-1\,\_recent window\;āŖ\; T_k_retrieved landmarks, |T_k|=6, |C_k|=B=12, (3) where the sink chunk anchors the global scene layout in the spirit of attention sinks [37], and every retained chunk is stored at full resolutionāno pooling, merging, or token-level pruning is applied. The retrieved landmarks are drawn from a bounded bank ā³M (|ā³|ā¤K|M|ā¤ K, K=30K=30) held in a tiered store: a pinned host-memory master copy of all K members, with only the retrieved working set staged on device. Each chunk carries a single camera pose PkāSEā”(3)P_k (3) (first-frame rotation with block-mean translation), and the bank is managed by pose alone. Admission keeps the bank diverse with an odometer rule: a chunk aging out of the recent window is stored only if the camera has travelled at least a stride Ī“ (measured in median step lengths) since the last stored landmark; the chunks in between are the densest, most redundant samples along a path, and are dropped. Crucially, the rule measures distance travelledānever whether a place is already bankedāso a revisit pass is stored just like the first visit, whereas gating against the whole bank would discard exactly the chunk a later revisit needs (only ā¼64% 64\% far-revisit coverage in simulation). Once the bank is full, each admission evicts the member most redundant in pose space (the one whose nearest neighbour is closest), and the first two members are never evicted, since the earliest region is the one revisited across the longest gaps. After every generated chunk, the six landmarks nearest to the current pose re-enter the cache as kT_k, where they are read through pose-indexed attention. Device memory thus stays at Oā”(B)O(B) chunks regardless of rollout length, with the full bank in host memory; landmark transfers are prefetched on a separate stream and overlap with denoising. Chunk-drop training. The cache above hands the model a sparse, non-contiguous subset of its history at inference, whereas standard teacher forcing conditions every denoising query on a contiguous, complete causal prefixāa trainātest gap that manifests as blur and drift as soon as chunks are evicted. We close it by randomly dropping cached KV chunks during training (Fig. 4, right): for a training clip of L=12L=12 chunks (Sec. 2.1), the sink chunk is always kept and five further survivors are drawn at random at every optimization step, so each step exposes the model to a random half of its history (66 of 1212 chunks). The attention mask is restricted accordingly: the query for chunk zkz_k attends only to the clean keys of surviving earlier chunks and to its own chunk, and queries are never dropped, meaning that every chunk still receives its denoising loss. The keep-set is resampled independently at every step and broadcast so that all data- and sequence-parallel ranks apply an identical mask. We compare this deployment policy against four alternatives under the identical budget B=12B=12: an unbounded full-KV cache (undeployable upper bound), a sliding-window cache with sink (forgetting baseline), mean-pooled compression of aged chunks, and a retrieval-free variant that replaces pose retrieval with a static bank of six pose-deduplicated landmarks attended in full; results are reported in Sec. 4.5 (Tab. 7). 2.4 Real-Time Distillation with a Lightweight Adapter Figure 5: Real-time distillation. A caption-domain DMD LoRA plugs into the frozen backbone; the control path carries no LoRA. To run in real time, ReWorld generates with only four denoising steps, obtained by step distillation confined to a plug-in LoRA in the AR-train-then-distill route of the LongLive series [40, 5] (Fig. 5): the multi-step model is the teacher, and the student is the same backbone, frozen, plus a rank-128128 LoRA on the attention and feed-forward layersāthe LoRA is the only thing trained (the DMD critic is also a LoRA over the frozen weights). Training follows distribution-matching distillation with self-forcing rollouts [14, 43, 42]: the student streams chunks from its own KV cache, just as at deployment, so distillation needs only text promptsāno video. The prompts are generic long captions from another domain (248K, extended from VidProM), and the action and pose injection path carries no LoRA, so the modelās control is untouched by distillation. One backbone thus has two modes: adapter off is the original multi-step model; adapter on streams in real time at four steps. The 2.6 GB adapter also transfers across base checkpoints, re-distilling in 2kā4k steps when needed. Two practical rules: distill at the deployment resolution, with prompts that resemble deployment prompts; the production student uses block size 16 with a single rollout and folds classifier-free guidance into the adapter. 3 Data Pipeline ReWorld is trained on an eight-source joint corpus of 220,724220,724 pose-annotated clips (Fig. 6). Two of the sources are rendered by our own Unreal Engine (UE) pipeline and play a double role: they anchor the metric scale to which all other sources are aligned, and their trajectories are generated specifically for camera control. Figure 6: The four-stage data pipeline. Eight UE, real-world, and game sources; an automated UE service renders 337337 environments with NavMesh-based trajectory planning (the control pattern anchors each path at two points of interest and fills the segment between them with direction-balanced random motion); a curation funnel filters each pool and evens out on-screen speed by dropping too-fast and too-slow clips; unification then rescales the pose numbers to one shared metric scale ā the same action moves the camera the same distance ā before clip-level sampling. 3.1 Sources Table 2 summarizes the eight sources; each clip is stored as an RGB video, a per-frame camera trajectory, and a text caption. The sources fall into three groups ā UE-rendered (a control set and an expansion set, produced by the pipeline of Sec. 3.2), real-world footage (DL3DV [21], RealEstate10K [47], and Sekai real-walking-hq [19]), and game footage (game roaming, OmniWorld-Game [16], and Sekai game-walking) ā with per-source construction details in Appendix A. Table 2: The eight-source joint training corpus. #Clips are counts after per-source filtering (Sec. 3.4); Duration is the per-clip length; Pose gives the camera-trajectory provenance; Caption gives the text-annotation provenance. Source #Clips Duration Pose Caption UE control (ours) 95,89595,895 1515 s engine GT (c2w, cm) pipeline-generated UE expansion (ours) 42,53642,536 3030ā6060 s engine GT (c2w, cm) scene summary + camera DL3DV 29,86429,864 1515 s VIPE Qwen-VL RealEstate10K 12,06512,065 ā„8ā„ 8 s VIPE (re-estimated) pre-computed Sekai real-walking-hq 14,73014,730 6060 s MegaSaM (c2w) official CSV Game roaming (7979 games) 18,38718,387 1515 s VIPE (metric) Qwen-VL OmniWorld-Game 5,6295,629 ā¼ 1212 s engine (w2cā 2w) per-clip JSON Sekai game-walking 1,6181,618 6060 s engine GT (UE5, c2w) official CSV Total 220,724220,724 3.2 UE Rendering Pipeline An automated service inside the UE editor drives scene loading, geometry survey, navigation-mesh construction, point-of-interest discovery, trajectory planning, and Movie Render Queue rendering, and a batch orchestrator sweeps this chain over a library of 337337 environments (Fig. 6). Each clip is exported with RGB frames, exact engine ground-truth camera poses (pinhole intrinsics and per-frame camera-to-world matrices), and scene and trajectory metadata; control clips additionally carry per-frame action labels. Trajectories are planned for control, not only for scene coverage: they are drawn from six planned modes, each walking at a constant, mode-specific speed. The control mode anchors each trajectory at two points of interest and fills the path between them with randomized motion segments, so that backward, strafing, and diagonal motion are sampled as evenly as forward motion (Fig. 6). The action-following accuracy reported in Sec. 4 owes as much to these direction-balanced trajectories as to the model design, since forward-biased footage gives little supervision for the rarer directions. 3.3 Metric-Scale Alignment The action akāā6a_k ^6 that conditions each latent chunk is a frame-to-frame 6-DoF camera increment, so the same commanded action must correspond to the same motion in every source ā both on screen and in the pose numbers (Fig. 6). The curation stage already settles the on-screen half: its speed filter drops clips that move too fast or too slow, leaving the survivors at a similar pixel speed (Sec. 3.4). What still disagrees is the numbers attached to that motion ā estimated poses (VIPE, MegaSaM) are defined only up to scale, and engine poses differ in units. We therefore rescale the translations of each source s by a single divisor ā the ratio of pooled per-latent-step translation-increment medians against the UE anchor, measured by replaying the exact training-time windowing with augmentation disabled: Ļs=medsā”[ā„Īātā„]medUEā”[ā„Īātā„],medUEā”[ā„Īātā„]=0.3667,tāt/Ļs. _s\;=\; med_s\! [ t ]med_UE\! [ t ], _UE\! [ t ]=0.3667, t\;ā\;t/ _s. (4) A single global scalar per source suffices because the per-clip scale spread within each source is bounded; all sources are further unified to camera-to-world matrices in a common camera basis, and per-source divisors, spread statistics, alignment audits, and convention verification are given in Appendix A. 3.4 Filtering and Sampling Filtering. The UE control renders pass the funnel of Fig. 6 ā splitting, a speed filter that drops clips moving too fast or too slow, captioning, and a darkness/wall filter ā reducing 110,204110,204 clips to 95,89595,895, while the expansion set is drawn from the raw render pool by scene round-robin and darkness-filtered from 47,72647,726 to 42,53642,536 clips. The game-roaming source is curated from 494.7494.7 hours of gameplay across 168168 games down to 18,38718,387 free-roaming clips from 7979 games. Filters for the remaining sources are simple duration and completeness checks; all thresholds and per-source details are given in Appendix A. Sampling. The eight sources are concatenated and sampled uniformly at the clip level, so the training mixture equals the clip-count proportions of Table 2 ā 63%63\% UE, 26%26\% real, and 11%11\% game ā independent of clip duration. Each dataset access draws a random 189189-frame window resampled to 2424 fps (ā8ā 8 s, one training window; Sec. 2.1); with probability 0.20.2 on the two UE sources the window is replaced by a palindrome ā a random half of the window concatenated with its temporal reverse ā injecting the explicit revisit evidence that long-horizon spatial memory requires. From each window the loader emits the RGB frames, the per-chunk 6-DoF actions aka_k, and the relative SE(3) trajectory PkP_k consumed by MRoPE (Appendix A). 4 Experiments 4.1 Setup Implementation. ReWorld is trained in two resolution stages on top of the Wan2.2-TI2V-5B backbone [33]: a 480p (384Ć640384Ć 640) pre-training stage, followed by a 720p (704Ć1280704Ć 1280) warm-start with interpolated spatial RoPE positions; optimizer, guidance, and parallelism settings follow the training details of Sec. 2.2. Each training window covers L=12L=12 latent chunks zkz_k, conditioned on per-chunk actions aka_k and poses PkP_k as defined in Sec. 2.1. Of the H=24H=24 attention heads, the global set G (||=6|G|=6) attends over the full causal history while the remaining heads use a local window of w=12w=12 frames; during training, G cycles through a fixed pool of 12 random six-head partitions, switching every optimizer step (random head routing, Sec. 2.2), and KV chunks are randomly dropped down to 6 kept chunks plus one sink chunk (chunk drop, Sec. 2.3). Real-time inference. Unless otherwise stated, all deployed and timed results are produced with a 4-step DMD-distilled [43, 42] LoRA (rank 128, block size 16, single-rollout student) applied to the EMA weights of the multi-step model, following the self-forcing style distillation recipe [14]. Inference runs at 720p with 4 denoising steps, CFG scale 1, and 16 latent frames per block; per-head attention windows are a training-time construct and are not applied at evaluationāall heads attend over the bounded cache. Bounded-memory arms operate under a KV-cache chunk budget of B=12B=12 on the cache C; the landmark bank ā³M is capped at |ā³|ā¤K|M|ā¤ K with K=30K=30 (Sec. 2.3). Evaluation suite. We evaluate four axes: (i) camera controllability against six interactive world-model and camera-controlled video generation baselines on a shared trajectory benchmark (Sec. 4.2); (i) long-horizon memory with a needle-in-a-haystack (NIAH) protocol built from palindromic revisit trajectories of up to 384 latents (Sec. 4.3); (i) video quality on seven video-intrinsic VBench [15] dimensions (Sec. 4.4); and (iv) ablations isolating the training recipe (chunk drop, random head routing), the inference-time cache policy, and the action/pose fusion design (Sec. 4.5). 4.2 Camera Controllability Protocol. We construct a controllability benchmark of 40 start images (drawn from the six baselinesā official repositories at 704Ć1280704Ć 1280, so that no single method is favored) Ć 6 canonical trajectories, i.e., 240 clips per method. The trajectories are dolly (sustained forward), strafe (lateral leftāright), arc_yaw (forward with yaw), arc_pitch (forward with pitch), s_curve (forward with alternating yaw), and palindrome (forward then exact return); every trajectory contains forward translation, since pure-rotation ground truth has zero displacement and degenerates the similarity alignment. Each method receives the same camera intent translated into its native control interface (action keys, pose sequences, or text, as appropriate). Generated clips are re-tracked with ViPE [12], and the estimated trajectory is registered to the intended trajectory by a Simā”(3)Sim(3) position alignment together with a global orientation alignment (restricted to det=+1 =+1), which absorbs per-method conventions in step magnitude, handedness, and coordinate frame so that the comparison measures whether the trajectory shape follows the intent. We report per-trajectory rotation error RotErr (geodesic, degrees) and aggregate translation error TransErr and camera motion consistency CamMC. We compare against SANA-WM [48], DreamX [7], HY-WorldPlay [30] (its lightweight Wan-based variant on this benchmark), Matrix-Game 3.0 [35], LingBot-World [29], and Yume-1.5 [26]. Table 3: Camera controllability on the 40-image Ć 6-trajectory benchmark (240 clips per method). Overall RotErr, TransErr, and CamMC (ā ) averaged over all trajectories, followed by per-trajectory RotErr (ā, ā ) after Simā”(3)Sim(3) and global orientation alignment. Best per column in bold. Overall RotErrā ā per trajectory Method RotErrā ā TransErr ā CamMC ā dolly strafe arc_yaw arc_pitch s_curve palindrome SANA-WM [48] 13.02 0.123 0.388 4.95 1.51 32.01 21.33 17.54 0.80 DreamX [7] 13.10 0.114 0.381 2.50 1.21 33.00 20.71 18.06 3.13 HY-WorldPlay [30] 14.66 0.114 0.427 0.26 0.24 43.53 22.00 21.73 0.20 Matrix-Game 3.0 [35] 15.45 0.075 0.394 8.19 1.00 39.93 20.59 17.75 5.24 LingBot-World [29] 12.59 0.107 0.354 3.10 6.01 26.21 26.83 12.23 1.16 Yume-1.5 [26] 14.24 0.131 0.428 0.84 2.01 29.05 24.57 25.60 3.38 ReWorld (ours) 11.95 0.102 0.332 2.69 1.07 27.63 23.61 16.06 0.64 Table 3 reports per-trajectory rotation error together with aggregate metrics for all seven methods. Because all methods are aligned with the same Simā”(3)Sim(3) and orientation registration against a method-agnostic ground-truth intent, the comparison isolates trajectory-following fidelity from per-method magnitude conventions. ReWorld attains the best overall RotErr and CamMC, sits in the first tier on the translation-dominant trajectories (dolly, strafe, palindrome), and remains competitive on the rotation-heavy arcs, which are the hardest regime for every method. 4.3 Long-Horizon Memory Protocol. We probe spatial memory with a needle-in-a-haystack protocol built on palindrome trajectories: the camera moves out and then retraces its path, so views generated early in the clip must be reproduced after a long temporal gapāthese early views are the needles. The baseline benchmark pairs 12 start images with 3 canonical palindromes at two rollout lengths kā48,96kā\48,96\ latents (ā 8 s and ā 16 s), giving 36 clips per method and length. The palindromes are strafe, dolly, and yaw (Fig. 7). Every method receives the same camera intent through its native control interface, with per-image matched prompts and a fixed seed. Scored revisit pairs are the mirror pairs of the palindrome, anchored in the earliest fifth of the clip and kept only when their gap spans at least half the rollout, up to five pairs per clip. For our own arms we additionally run a longer version of the memory test: grouped-action explorations completed into palindromes, at nested lengths kā96,192,288,384kā\96,192,288,384\ latents; each length is a prefix of the same trajectory, so scores are comparable across k. This longer test feeds the ablations of Sec. 4.5. Figure 7: Constructed palindrome trajectories. The camera moves out (blue) and retraces its path (orange), so the needle view at t=0t=0 must be regenerated at t=kā1t=k-1; scored mirror pairs (t,kā1āt)(t,\,k-1-t) span at least half the rollout. Memory metrics. For each revisit pair we compare the predicted frame at the needle with the predicted frame generated when the camera returns to that pose, and score their similarity four ways. SSIM measures pixel-level structure. LPIPS is a perceptual distance between deep features (lower is better). DINO is the cosine similarity of DINO ViT features and asks whether the two frames show the same scene layout. ORB is the fraction of matched ORB keypoints and asks whether specific landmarks reappear in place. Each score is averaged over revisit pairs, then over clips, and reported as an absolute value. A rollout that barely moves scores high on any similarity metric, so the table also reports each methodās executed path length: the accumulated median optical flow of the outbound half, i.e., how much visual ground the rollout actually covers before turning back. Comparison to baselines. Table 4 reports all four scores at both lengths for the same six methods as Sec. 4.2. For HY-WorldPlay we evaluate its flagship autoregressive model, as in Sec. 4.4. ReWorld runs its default deployment configuration (consolidation with the landmark bank, B=12B=12). The two lengths play different roles. At k=48k=48 the revisit gap reaches 47 latents, and several baselines still hold the start view in contextāLingBot-World keeps a 52-latent window, while SANA-WM (linear attention) and Yume-1.5 are unboundedāso k=48k=48 serves as a sanity check. At k=96k=96 the gap reaches 95 latents: the start view has left every bounded window and must be recalled from memory, making this the discriminative length. HY-WorldPlay posts the strongest baseline scores, but it also moves the least: at k=96k=96 its median outbound path length is 210 px, versus 332ā834 px for the other baselines (Table 4), and a shorter path is easier to reproduce from memory. Figure 8 shows this regime qualitatively on a k=96k=96 strafe-and-return rollout: the final chunks must reproduce the input view after a long round trip, and ReWorld returns to the start with the landmark layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry. The effect of the KV budget and cache policy on our model, including its per-length scaling, is isolated on the long-rollout test in Sec. 4.5 (Table 7). Table 4: Long-horizon memory against the six interactive world-model baselines on the palindromic revisit bench: absolute revisit-similarity scores (SSIM, LPIPS, DINO, ORB) at rollout lengths k=48k=48 and k=96k=96 latents, alongside the executed path length (accumulated median optical flow of the outbound half, px). Every method receives the same action commands, but how far it actually travels differs, and a shorter path generally yields higher revisit scores. k=48k=48 is a sanity length that several baselines can solve from context alone; k=96k=96 forces recall from beyond every bounded window. Best per column in bold, second best underlined; path length is descriptive, not ranked. k=48k=48 (ā 8 s) k=96k=96 (ā 16 s) Method Path Length SSIMā LPIPSā DINOā ORBā Path Length SSIMā LPIPSā DINOā ORBā SANA-WM [48] 296 0.318 0.433 0.810 0.174 499 0.313 0.528 0.700 0.161 DreamX [7] 279 0.214 0.553 0.740 0.180 454 0.194 0.627 0.603 0.159 HY-WorldPlay [30] 113ā 0.448 0.163 0.969 0.464 210ā 0.427 0.247 0.942 0.460 Matrix-Game 3.0 [35] 312 0.318 0.369 0.891 0.292 724 0.275 0.478 0.850 0.247 LingBot-World [29] 352 0.274 0.557 0.663 0.177 834 0.251 0.635 0.509 0.162 Yume-1.5 [26] 165 0.268 0.533 0.747 0.179 332 0.269 0.586 0.661 0.152 ReWorld (ours) 286 0.349 0.282 0.913 0.325 615 0.384 0.332 0.932 0.379 ā HY-WorldPlay executes by far the shortest paths: 113/210 px vs. 279ā352/454ā834 px for all other methods. Figure 8: Qualitative memory comparison on a strafe-and-return rollout. All methods start from the same input frame and execute the same lateral palindromeāthe camera strafes left, then retraces the same line back to the start, without ever turningāso the last column should reproduce the first: the same landmarks, in the same places. Columns are latents 0/8/28/48/72/940/8/28/48/72/94 of a k=96k=96 rollout; each baseline is driven through its native control interface. ReWorld (top row; bounded cache B=12B=12 with the landmark bank, four-step distilled) returns to the input view with the layout intact, whereas the baselines either drift off the commanded path or regenerate the revisited region with altered geometry. 4.4 Generation Quality We assess perceptual quality on the seven video-intrinsic VBench [15] dimensions that require no text prompt: Imaging Quality, Aesthetic Quality, Subject Consistency, Background Consistency, Temporal Flickering, Dynamic Degree, and Motion Smoothness, computed on the same clip set as the controllability benchmark. Native outputs differ across methods in length, resolution, and frame rate, so all clips are normalized to a common specification before scoringā32 uniformly sampled frames spanning the full clip, resized to 1280Ć7041280Ć 704, re-encoded at 16 fps; the comparison is therefore internally controlled, but not directly comparable to numbers reported on native-length outputs. For HY-WorldPlay, this benchmark evaluates its flagship autoregressive model. Table 5: VBench quality on the seven video-intrinsic dimensions (higher is better), computed on the shared benchmark clips normalized to 32 frames / 1280Ć7041280Ć 704 / 16 fps. Best per column in bold. Method Mean ā Imaging Aesthetic Subject Cons. Background Cons. Temporal Flick. Dynamic Degree Motion Smooth. SANA-WM [48] 0.835 0.668 0.537 0.908 0.919 0.942 0.900 0.969 DreamX [7] 0.828 0.670 0.542 0.880 0.904 0.909 0.946 0.946 HY-WorldPlay [30] 0.842 0.670 0.601 0.976 0.957 0.973 0.733 0.987 Matrix-Game 3.0 [35] 0.836 0.713 0.461 0.894 0.919 0.921 0.983 0.962 LingBot-World [29] 0.841 0.692 0.577 0.905 0.915 0.918 0.933 0.947 Yume-1.5 [26] 0.844 0.669 0.565 0.878 0.911 0.934 0.988 0.964 ReWorld (ours) 0.850 0.665 0.579 0.929 0.929 0.952 0.912 0.979 Table 5 summarizes the seven quality dimensions. ReWorld attains the best mean: no single dimension is dominant, but it is in the first tier on the temporal axes (Motion Smoothness, Temporal Flickering) while sustaining a high Dynamic Degree, whereas the per-dimension leader HY-WorldPlay pays for its consistency scores with markedly lower motion. We note that Dynamic Degree should be read jointly with the revisit metrics of Sec. 4.3, as consistency and revisit metrics alike favor low-motion rollouts. 4.5 Ablations Figure 9: Inference cache policies, qualitatively. On out-and-back trajectories, sliding window and KV compression drift or reinvent the revisited scene; the landmark cache (ours) returns to the start view. Training recipe and cache policy. Table 7 ablates two axes on the long-rollout memory test of Sec. 4.3, reporting revisit SSIM at rollout lengths kā96,192,288,384kā\96,192,288,384\ latents. The upper block varies the training recipeāchunk drop only (CD), random head routing only (RH), and both combined (ours). All variants are trained from scratch under identical settings (720p, 20k steps) and evaluated with the same consolidation arm, so only the recipe differs. The lower block fixes the final recipe and varies only the inference cache policy under the same budget B=12B=12. Every bounded arm keeps the same sink chunk and differs only in how the rest of the budget is filled: window uses a sliding window of recent chunks; naive-merge pools aged chunks instead of evicting them; consolidation w/o bank adds six static landmarks (no retrieval); consolidation w/ bank retrieves the six landmarks nearest the current pose from the bounded bank (Sec. 2.3); full-KV keeps everything and is the undeployable upper bound (OOM beyond k=192k=192). The far-gap lengths kā288,384kā\288,384\, where a sliding window has necessarily evicted the needle, are the discriminative setting; Fig. 9 shows the contrast qualitatively. Controlāmemory decoupling. Table 7 tests whether the routed head structure decouples action-following from long-range recall, comparing three fusion variants trained under the same protocol. MRoPE only drops the action embedding and sets all H=24H=24 heads global, so control must be inferred solely from pose-indexed positional encodings; Action + MRoPE keeps the additive action embedding with all heads global, i.e., fusion without routing; Routing is our full configuration with the mixed global/local head structure and per-step head routing. We report the control metrics of Sec. 4.2 on the long-rollout memory test. Adding action injection improves every control metric over MRoPE only (RotErr 17.66āā13.21ā17.66 \!ā\!13.21 , TransErr ā0.1070.204\!ā\!0.107, CamMC ā0.3710.394\!ā\!0.371) but drops revisit SSIM from 0.3898 to 0.3376āthe interference that motivates the window split. Routing matches the unrouted variant on control (12.94ā12.94 , 0.093, 0.346) while restoring revisit SSIM to 0.3752: the structural constraint costs no control authority and preserves long-gap recall. 5 Related Works Streaming video generation. Table 6: Revisit SSIMā on the long-rollout memory test (Sec. 4.3) at four rollout lengths k. Upper block: the training recipe is varied with the inference arm fixed to our consolidation. Lower block: the inference cache policy is varied on the final recipe; full-KV is the unbounded upper bound (OOM beyond k=192k=192). Revisit SSIMā Method k=96k=96 k=192k=192 k=288k=288 k=384k=384 Training recipe (consolidation arm) base 0.4019 0.3747 0.3698 0.3387 + CD 0.4043 0.3941 0.3846 0.3463 + RH 0.4319 0.3787 0.3836 0.3565 + CD + RH (ours) 0.4358 0.3929 0.3943 0.3752 Inference cache policy full-KV 0.4463 0.4231 (OOM) (OOM) window 0.4129 0.3687 0.3505 0.3476 naive-merge 0.3741 0.3371 0.3467 0.3541 consolidation w/o bank 0.4376 0.3758 0.3743 0.3629 consolidation w/ bank 0.4358 0.3929 0.3943 0.3752 Table 7: Controlāmemory decoupling study: control metrics on the long-rollout memory test (Sec. 4.3) for the three fusion variants. Method RotErrāā TransErrā CamMCā Revisit SSIMā MRoPE only 17.66 0.204 0.394 0.3898 Action + MRoPE 13.21 0.107 0.371 0.3376 Routing (ours) 12.94 0.093 0.346 0.3752 Large video diffusion models denoise a clip as a single bidirectional block [33, 18, 41], an interface at odds with interaction: no frame can appear before the whole clip is finished. Autoregressive reformulations restore a temporal arrow. Per-frame noise schedules [3], chunked causal attention with KV caching [32], context packing [45], and streaming pipelines [17] realize it, while few-step distillation [43, 42, 34, 25], training-free attention reuse [46], self-rollout training that closes the exposure gap of AR [14, 23], and AR-train-then-distill systems scaled to long video [5] make the streams real-time. ReWorld follows this recipeāchunked causal generation over a bidirectional backbone, DMD distillation with self-forcing rolloutsābut confines distillation to a LoRA [11], so one backbone retains a high-fidelity multi-step mode beside the real-time one. Interactive world models. From recurrent latent simulators [10] through latent-action pretraining [2] and its frontier successors [27], interactive generation now spans action-conditioned diffusion trained on game and UE footage [35, 29, 48, 13], explorable scene generation [26, 31, 6, 24], and low-latency streamed deployments [4, 7]. The two prevailing control signals differ in kind: relative pose folded into attention localizes past content [7, 39], whereas an injected action supplies the motion command directly [35], and some recent systems carry both [30]. ReWorld likewise keeps both channels, but treats their interference as a training problem, splitting the window each attention head trains under so that control is learned within short windows and memory under long ones. Memory in world models. Long rollouts stay consistent only if views that have left the context remain recallable [9]. Existing designs differ chiefly in what they store and what that costs: archives of past frames that grow with the rollout, from which pose-relevant views are pulled back as conditioning [44, 38]; an explicit 3D reconstruction that keeps memory outside the generatorās own representation [36]; or pose-dependent attention over the retained history [39]. Sinks and sliding windows [37] bound the cache instead, at the price of forgetting everything beyond the window. ReWorld takes the bounded route without the forgetting: chunks aging out of the recent window consolidate into a fixed-capacity landmark bank inside the modelās own KV space, redundancy-based eviction keeps the bank diverse, pose-proximity retrieval fills the cache, and chunk-drop training teaches the model to read the spliced result. 6 Conclusion We presented ReWorld, an interactive streaming world model built in two steps: split the training of control and memory by windowāmixed per-head attention windows with random head routingāthen consolidate memory at inference, where a bounded cache backed by a pose-indexed landmark bank holds the entire past under a fixed KV budget and chunk-drop training makes its sparse caches in-distribution. Trained on metrically aligned multi-source data and distilled to a few-step LoRA student, ReWorld streams high-resolution video in real time and leads recent interactive world models on control fidelity and visual quality, while revisit fidelity persists at rollout lengths where a sliding window has long evicted the evidence. Memory is still keyed on camera pose alone; extending consolidation to dynamic scenes and richer, non-navigational interaction is the natural next step. References [1] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: Ā§A.1. [2] J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. (2024) Genie: generative interactive environments. In International Conference on Machine Learning (ICML), Note: arXiv:2402.15391 Cited by: Ā§1, Ā§5. [3] B. Chen, D. Marti MonsĆ³, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann (2024) Diffusion forcing: next-token prediction meets full-sequence diffusion. In Advances in Neural Information Processing Systems, Cited by: Ā§5. [4] M. Chen, L. Cui, W. Zhang, H. Zhang, Y. Zhou, X. Li, S. Tang, J. Liu, B. Liao, H. Chen, et al. (2025) Midas: multimodal interactive digital-human synthesis via real-time autoregressive video generation. arXiv preprint arXiv:2508.19320. Cited by: Ā§5. [5] Y. Chen, L. Wang, W. Huang, S. Yang, B. Zhang, Y. Xiao, R. Chu, W. Mao, Q. Hu, S. Liu, Y. Zhao, H. Mao, Y. Chen, E. Xie, X. Qi, and S. Han (2026) LongLive-2.0: an nvfp4 parallel infrastructure for long video generation. arXiv preprint arXiv:2605.18739. Cited by: Ā§2.1, Ā§2.4, Ā§5. [6] X. Cheng, H. Zhou, W. Yu, T. Jia, B. Lin, Y. Ge, W. Li, and L. Yuan (2026) 360Explorer: exploring 4d controllable world in panoramic videos. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 3300ā3308. Cited by: Ā§5. [7] DreamX Team (2026) DreamX-world 1.0: a general-purpose interactive world model. arXiv preprint arXiv:2606.16993. Cited by: Ā§1, Table 1, Ā§4.2, Table 3, Table 4, Table 5, Ā§5. [8] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. MĆ¼ller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach (2024) Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206. Cited by: Ā§2.1. [9] Y. Gu, W. Mao, and M. Z. Shou (2025) Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325. Cited by: Ā§5. [10] D. Ha and J. Schmidhuber (2018) World models. arXiv preprint arXiv:1803.10122. Cited by: Ā§1, Ā§5. [11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2106.09685 Cited by: Ā§1, Ā§5. [12] J. Huang, Q. Zhou, H. Rabeti, A. Korovko, H. Ling, X. Ren, T. Shen, J. Gao, D. Slepichev, C. Lin, et al. (2025) ViPE: video pose engine for 3D geometric perception. arXiv preprint arXiv:2508.10934. Cited by: Ā§A.1, Ā§4.2. [13] S. Huang, J. Wu, Q. Zhou, S. Miao, and M. Long (2025) Vid2World: crafting video diffusion models to interactive world models. arXiv preprint arXiv: 2505.14357. Cited by: Ā§5. [14] X. Huang, Z. Li, G. He, M. Zhou, and E. Shechtman (2025) Self forcing: bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009. Cited by: Ā§1, Ā§1, Ā§2.4, Ā§4.1, Ā§5. [15] Z. Huang, Y. He, J. Yu, F. Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q. Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu (2024) VBench: comprehensive benchmark suite for video generative models. In CVPR, Cited by: Ā§4.1, Ā§4.4. [16] InternRobotics Team (2025) OmniWorld: a multi-domain and multi-modal dataset for 4D world modeling. arXiv preprint arXiv:2509.12201. Cited by: Ā§A.1, Ā§3.1. [17] A. Kodaira, T. Hou, J. Hou, M. Georgopoulos, F. Juefei-Xu, M. Tomizuka, and Y. Zhao (2026) Streamdit: real-time streaming text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 29200ā29210. Cited by: Ā§5. [18] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) Hunyuanvideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: Ā§5. [19] Z. Li, C. Chen, H. Yang, Y. Han, B. Zhu, Z. Zhou, et al. (2025) Sekai: a video dataset towards world exploration. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2506.15675 Cited by: Ā§A.1, Ā§3.1. [20] Z. Li, R. Tucker, F. Cole, Q. Wang, L. Jin, V. Ye, A. Kanazawa, A. Holynski, and N. Snavely (2025) MegaSaM: accurate, fast, and robust structure and motion from casual dynamic videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Ā§A.1. [21] L. Ling, Y. Sheng, Z. Tu, W. Zhao, C. Xin, K. Wan, L. Yu, Q. Guo, Z. Yu, Y. Lu, et al. (2024) DL3DV-10K: a large-scale scene dataset for deep learning-based 3D vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: Ā§A.1, Ā§3.1. [22] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In International Conference on Learning Representations, Cited by: Ā§2.1. [23] K. Liu, W. Hu, J. Xu, Y. Shan, and S. Lu (2025) Rolling forcing: autoregressive long video diffusion in real time. arXiv preprint arXiv:2509.25161. Cited by: Ā§5. [24] W. Liu, Z. Chen, Z. Li, Y. Wang, H. Yu, and J. Wu (2026) RealWonder: real-time physical action-conditioned video generation. arXiv preprint arXiv:2603.05449. Cited by: Ā§5. [25] Z. Lv, C. Si, T. Pan, Z. Chen, K. K. Wong, Y. Qiao, and Z. Liu (2025) Dual-expert consistency model for efficient and high-quality video generation. https://arxiv.org/abs/2506.03123. Cited by: Ā§5. [26] X. Mao, Z. Li, C. Li, X. Xu, K. Ying, T. He, J. Pang, Y. Qiao, and K. Zhang (2025) Yume-1.5: a text-controlled interactive world generation model. arXiv preprint arXiv:2512.22096. Cited by: Ā§4.2, Table 3, Table 4, Table 5, Ā§5. [27] J. Parker-Holder and S. Fruchter (2025) Genie 3: a new frontier for world models. Note: Google DeepMind Blog, https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/Announced August 5, 2025 Cited by: Ā§1, Ā§5. [28] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In ICCV, p. 4195ā4205. Cited by: Ā§2.2. [29] Robbyant Team, Z. Gao, Q. Wang, Y. Zeng, J. Zhu, K. L. Cheng, Y. Li, H. Wang, Y. Xu, et al. (2026) Advancing open-source world models. arXiv preprint arXiv:2601.20540. Cited by: Ā§1, Ā§4.2, Table 3, Table 4, Table 5, Ā§5. [30] W. Sun, H. Zhang, H. Wang, J. Wu, Z. Wang, Z. Wang, Y. Wang, J. Zhang, T. Wang, and C. Guo (2025) WorldPlay: towards long-term geometric consistency for real-time interactive world modeling. arXiv preprint arXiv:2512.14614. Cited by: Ā§1, Ā§1, Ā§2.3, Table 1, Ā§4.2, Table 3, Table 4, Table 5, Ā§5. [31] H. Team, Z. Wang, Y. Liu, J. Wu, Z. Gu, H. Wang, X. Zuo, T. Huang, W. Li, S. Zhang, et al. (2025) HunyuanWorld 1.0: generating immersive, explorable, and interactive 3d worlds from words or pixels. arXiv preprint arXiv:2507.21809. Cited by: Ā§5. [32] H. Teng, H. Jia, L. Sun, L. Li, M. Li, M. Tang, S. Han, T. Zhang, W. Zhang, W. Luo, et al. (2025) MAGI-1: autoregressive video generation at scale. arXiv preprint arXiv:2505.13211. Cited by: Ā§5. [33] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: Ā§2.2, Ā§4.1, Ā§5. [34] F. Wang, Z. Huang, A. W. Bergman, D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, et al. (2024) Phased consistency model. arXiv preprint arXiv:2405.18407. Cited by: Ā§5. [35] Z. Wang, Z. Liu, J. Li, K. Huang, B. Xu, F. Kang, M. An, et al. (2026) Matrix-game 3.0: real-time and streaming interactive world model with long-horizon memory. arXiv preprint arXiv:2604.08995. Cited by: Ā§1, Ā§1, Ā§2.3, Ā§4.2, Table 3, Table 4, Table 5, Ā§5. [36] T. Wu, S. Yang, R. Po, Y. Xu, Z. Liu, D. Lin, and G. Wetzstein (2025) Video world models with long-term spatial memory. arXiv preprint arXiv:2506.05284. Cited by: Ā§1, Ā§2.3, Ā§5. [37] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICLR), Cited by: Ā§1, Ā§2.3, Ā§5. [38] Z. Xiao, Y. Lan, Y. Zhou, W. Ouyang, S. Yang, Y. Zeng, and X. Pan (2025) WorldMem: long-term consistent world simulation with memory. arXiv preprint arXiv:2504.12369. Cited by: Ā§1, Ā§5. [39] T. Xu, Y. Xie, D. Meng, C. Peng, Q. Herau, B. Jiang, Y. Hu, and W. Zhan (2026) Teaching video generators to remember: eliciting dynamic memory for out-of-sight state evolution. arXiv preprint arXiv:2605.25333. Cited by: Ā§1, Ā§2.2, Ā§5, Ā§5. [40] S. Yang, W. Huang, R. Chu, Y. Xiao, Y. Zhao, X. Wang, M. Li, E. Xie, Y. Chen, Y. Lu, S. Han, and Y. Chen (2025) LongLive: real-time interactive long video generation. arXiv preprint arXiv:2509.22622. Cited by: Ā§2.4. [41] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2025) CogVideoX: text-to-video diffusion models with an expert transformer. In ICLR, Cited by: Ā§5. [42] T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and W. T. Freeman (2024) Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867. Cited by: Ā§1, Ā§2.4, Ā§4.1, Ā§5. [43] T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park (2024) One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828. Cited by: Ā§1, Ā§2.4, Ā§4.1, Ā§5. [44] J. Yu, J. Bai, Y. Qin, Q. Liu, X. Wang, P. Wan, D. Zhang, and X. Liu (2025) Context as memory: scene-consistent interactive long video generation with memory retrieval. ICCV. Cited by: Ā§1, Ā§2.3, Ā§5. [45] L. Zhang and M. Agrawala (2025) Packing input frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504.12626. Cited by: Ā§5. [46] X. Zhao, X. Jin, K. Wang, and Y. You (2025) Real-time video generation with pyramid attention broadcast. In ICLR, Cited by: Ā§5. [47] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018) Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics (SIGGRAPH) 37 (4). Cited by: Ā§A.1, Ā§3.1. [48] H. Zhu, H. Liu, Y. Zhao, T. Ye, J. Chen, J. Yu, T. He, S. Han, and E. Xie (2026) SANA-WM: efficient minute-scale world modeling with hybrid linear diffusion transformer. arXiv preprint arXiv:2605.15178. Cited by: Ā§4.2, Table 3, Table 4, Table 5, Ā§5. Appendix Appendix A Data Pipeline Details This appendix expands Sec. 3 with per-source construction details, the per-source scale divisors and their audits, and the loader-level sampling and conditioning specifics. A.1 Per-Source Construction UE-rendered fly-throughs (metric anchor). Our Unreal Engine rendering pipeline produces camera fly-throughs with exact ground-truth trajectories: each clip directory contains the rendered video, a camera.json with per-frame absolute 4Ć44Ć 4 camera-to-world matrices (right-handed, Y-up, centimeters) together with intrinsics, and a summary.json recording the scene and trajectory metadata (plus an overhead trajectory visualization). The full rendering run yields 130,986130,986 raw clips over 337337 purchased and in-house environments (indoor, urban, and landscape scenes), drawn from six trajectory modes planned at generation time ā local_explore (ā¼ 29%), reveal_pan (ā¼ 26%), aerial (ā¼ 15.5%), control (ā¼ 15%), wandering (ā¼ 8%), and boomerang (ā¼ 6%) ā each walking at a constant, mode-specific speed (180180 cm/s for control, 120120 cm/s for the exploratory modes). Because raw mode and scene frequencies are skewed, we curate a balanced subset by scene round-robin: for each mode we target ā¼ 8,000 clips while cycling over environment and level identifiers, which yields 47,72647,726 clips spanning 248248ā324324 environments per mode; darkness filtering (Sec. 3.4) reduces this to the 42,53642,536-clip expansion set of Table 2, whose captions combine the pipeline scene summary with a camera-motion sentence. The 95,89595,895-clip control set of Table 2 comes from an earlier, control-heavy rendering run (27,55127,551 renders of 6060 s, split into 110,204110,204 clips) and is obtained from 109,132109,132 captioned clips by the same darkness filter (removing black-frame and wall-facing clips); its trajectories are generated from the explicit discrete action space of the control mode, with per-frame action labels kept consistent with the realized motion, and its captions are generated by the rendering pipeline itself. Because UE poses are exact and metric, these sources serve as the scale anchor in Sec. 3.3. UE luminance and motion filtering. UE renders fail in a characteristic way: when the camera clips into geometry or faces an unlit surface, frames are near-black, so both UE sets pass through a luminance filter. Each clip is probed with 1616 uniformly sampled grayscale frames and dropped if its mean luminance is below 3535 (on 00ā255255) or if more than half of the sampled frames are dark; a lightweight motion probe during captioning separately removes frozen clips whose frames barely change. This filter reduces the control set from 109,132109,132 captioned clips to 95,89595,895 (ā12.1%-12.1\%) and the round-robin-selected expansion set from 47,72647,726 clips to 42,53642,536 (ā10.9%-10.9\%). DL3DV. From DL3DV-10K real-scene walkthroughs [21] we cut frame-accurate 1515 s slices whose pose sub-segments are aligned to the video slice with zero drift, yielding 29,86429,864 clips with no missing modality. Poses are estimated with VIPE [12] (near-metric). Each clip is captioned with Qwen-VL [1] (average 149149 words per caption, no duplicates or empties). RealEstate10K. We use RealEstate10K [47] indoor/real-estate footage (watermark-inpainted video), retaining clips of at least 88 s, which leaves 12,06512,065 of the original 22,27522,275. Rather than the released SLAM trajectories, poses are re-estimated with VIPE so that pose provenance matches DL3DV; captions come from a pre-computed caption store. Game roaming. The game-roaming source is curated from a 494.7494.7-hour collection of gameplay recordings spanning 168168 games. A roaming whitelist keeps 8888 titles suited to free exploration and walking (pure shooters are excluded), which are sliced into 24,80124,801 clips of 1515 s at 1920Ć10801920Ć 1080, 3030 fps. Each clip receives VIPE metric camera-to-world poses (OpenCV convention, first frame anchored to the identity) and a Qwen-VL caption; successive cleaning passes remove pose-quality outliers, dark clips, and full-screen menu or loading content, leaving 18,38718,387 clips across 7979 games (nine whitelisted titles yield no surviving clips). Because the VIPE poses of this source are already metric, it enters Sec. 3.3 with Ļ=1Ļ=1. OmniWorld-Game. From the simulator/game domain of OmniWorld [16] (479479 scenes) we normalize the released quaternion extrinsics, invert the world-to-camera matrices to camera-to-world at build time, and merge only frame-contiguous reconstruction splits into runs (splits across index gaps are independent reconstructions ā largely teleports ā and are never merged). Runs are cut into 1212 s windows with 99 s stride (25%25\% overlap), discarding runs shorter than 8.58.5 s, and each clip is re-anchored so that its first pose is the identity. This yields 5,6295,629 clips (median 12.012.0 s) with per-clip JSON captions. Sekai real-walking-hq and game-walking. From the Sekai corpus [19] of first-person walking videos we intersect available 720720p/6060 s videos with released pose files, obtaining 14,73014,730 real-walking clips; clips are kept as uncut 6060 s segments. Poses are per-frame MegaSaM [20] camera-to-world estimates (the c2w convention was verified empirically by forward-direction alignment, and poseāvideo synchronization was verified by cross-correlating optical-flow speed with pose angular speed, with median lag 00). The companion game-walking split contributes 1,6181,618 clips rendered in UE5 with engine ground-truth c2w trajectories, downscaled to 720720p without re-cutting. Both splits use the official CSV captions (caption plus location, weather, crowd, and time-of-day tags). A.2 Alignment Details and Audits Per-source divisors. The divisor of Eq. (4) is measured by replaying the exact training-time windowing (a 189189-frame window resampled to 2424 fps, i.e. 0.16406250.1640625 s per latent step, with all trajectory augmentation disabled), converting poses to per-latent translation increments Īāt t, and pooling the increment magnitudes over sampled clips. The resulting divisors are: UE Ļ=100Ļ=100 (exact cmā conversion; anchor), DL3DV 0.0709/0.3667=0.19230.0709/0.3667=0.1923, RealEstate10K 0.0825/0.3667=0.22730.0825/0.3667=0.2273, OmniWorld-Game 0.641/0.3667=1.750.641/0.3667=1.75 (pooled over the locomotion subset, since the UE anchor is locomotion-dominated and including the 29%29\% vehicle clips would inflate the divisor), Sekai real-walking 0.0032/0.3667=0.00860.0032/0.3667=0.0086, and Sekai game-walking 0.0039/0.3667=0.01070.0039/0.3667=0.0107; the game-roaming source is already metric under VIPE and uses Ļ=1Ļ=1. The divisor is applied to the translation column of every pose in the loader. The UE expansion set moves genuinely more slowly than the control set (its raw per-latent-step translation median is ā20ā 20 cm versus ā37ā 37 cm); we deliberately do not renormalize it separately, since the speed difference is real rather than a scale error. Spread and post-hoc audits. A single global scalar per source suffices because the per-clip scale spread is bounded: the p90/p10p_90/p_10 ratio of per-clip median increments is 3.653.65 (DL3DV), 2.662.66 (RealEstate10K), and 2.862.86/2.672.67 (Sekai real/game) ā under one order of magnitude and attributable to genuine capture-speed variation ā so neither per-scene normalization nor metric-depth rescaling is needed. A post-hoc audit of the model-facing increments after rescaling confirms alignment: the per-source median ā„Īātā„ t relative to UE is 1.001.00 (UE, by construction), 0.920.92 (DL3DV), 1.001.00 (RealEstate10K), 1.211.21 (OmniWorld-Game, a vehicle-tail effect), 0.960.96 (Sekai real), and 0.940.94 (Sekai game). Coordinate-convention normalization. Coordinate conventions are normalized jointly with scale. All sources are brought to camera-to-world matrices in the OpenCV camera basis (forward +z+z, down +y+y): the UE camera basis (forward +x+x, up +y+y) is conjugated by a non-trivial 90ā90 rotation about the vertical axis (a naive diagā”(1,ā1,ā1)diag(1,-1,-1) sign flip is incorrect and was ruled out by an empirical derivation against engine trajectories); OmniWorld world-to-camera extrinsics are inverted at clip-build time; Sekai extrinsics are already c2w and are ingested without inversion; the legacy camera-referenced (w2c) RealEstate10K mode is not used in the joint configuration. The audit further verifies that forward motion maps to a dominant +tz+t_z in every source (DL3DV is txt_x-dominated in aggregate, but this reflects its orbital/strafing capture style rather than a convention error) and that yaw maps to the same Euler index with consistent sign. Rotations are already dimensionless (radians) and are never rescaled; only translation passes through Eq. (4). A.3 Sampling and Conditioning Details Window extraction. Within a long clip the 189189-frame window ā the native training-window length of 4848 latent frames, i.e. L=12L=12 chunks of 44 ā starts at a uniformly random frame, and frames are resampled to 2424 fps by uniform striding (clips shorter than the required span are stretched by linear index interpolation). A 6060 s Sekai clip therefore contributes a different random ā8ā 8 s excerpt (7.887.88 s) on every epoch, which converts long uncut footage into diverse window-level supervision without offline re-slicing. Palindrome augmentation. The palindrome augmentation of Sec. 3.4 forces the camera to retrace its own path within a single training window and additionally provides reverse-motion coverage. Under this augmentation, pose downsampling is SE(3)-aware (rotation slerp), since arithmetic averaging across the palindrome mirror point produces singular matrices. A whole-window reversal probability exists in the loader but is disabled in the joint configuration. Emitted conditioning. From each window the loader emits the RGB frames, the per-chunk actions, and the pose context consumed by MRoPE. The 189189 frame poses are downsampled to the 4848 latent steps, re-anchored so the first camera is the identity, and differenced into per-step 6-DoF increments (three translations plus three Euler angles), which are grouped per latent chunk into the actions akāā6a_k ^6. The MRoPE context is the relative camera-to-world trajectory PkāSEā”(3)P_k (3) itself, with translations additionally normalized by the windowās maximum radius (a pure rescaling that preserves the identity anchor), so that pose-indexed attention operates on a bounded, source-agnostic coordinate frame.