Paper deep dive
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Zexuan Yan, Yuzhou Wu, Yue Ma, Zonghang He, Kaibo Yin, Xiaobing Tu, Yinggui Wang, Jinkui Ren, Xiantao Zhang, Shijian Wang, Jinghong Liu, Linfeng Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/3/2026, 9:56:41 AM
Summary
The paper introduces EgoGenesis, an egocentric world-action simulator that synthesizes high-quality manipulation videos to augment scarce real-world training data for embodied AI. It addresses geometric failures in existing generators by introducing two key components: Online Anchored Projective Memory (OAPM), which maintains a stable 3D scene anchor while updating recent states, and Action-3D Rotary Position Embedding (A3D-RoPE), which encodes end-effector motion with camera-aware 3D coordinates to ensure precise action alignment. Experiments demonstrate that augmenting real trajectories with EgoGenesis-generated data significantly improves out-of-distribution success rates on single-arm and dual-arm real-robot tasks.
Entities (9)
Relation Signals (7)
EgoGenesis → usescomponent → Online Anchored Projective Memory
confidence 95% · EgoGenesis builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM)...
EgoGenesis → usescomponent → Action-3D Rotary Position Embedding
confidence 95% · Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates...
EgoGenesis → augments → real trajectories
confidence 90% · augmenting 400 real trajectories with 400 EgoGenesis-generated trajectories improves out-of-distribution real-robot success
Action-3D Rotary Position Embedding → improves → action alignment
confidence 90% · Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts.
Online Anchored Projective Memory → improves → geometric stability
confidence 90% · Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts.
EgoGenesis → outperforms → Wan2.1 14B
confidence 85% · EgoGenesis ranks first on six of seven metrics... compared with general image-to-video and controllable video baselines, including Wan2.1-Fun-14B-Inp
EgoGenesis → outperforms → RynnWorld-TeleOp
confidence 85% · EgoGenesis ranks first on six of seven metrics... compared with ... RynnWorld-TeleOp
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77\% to 84\% on single-arm tasks and from 53\% to 70\% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.
Tags
Links
- Source: https://arxiv.org/abs/2607.28243v1
- Canonical: https://arxiv.org/abs/2607.28243v1
Trouble viewing inline? Open PDF directly →
Full Text
66,895 characters extracted from source content.
Expand or collapse full text
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE Zexuan Yan1,2 Yuzhou Wu3 Yue Ma4 Zonghang He1 Kaibo Yin7 Xiaobing Tu2 Yinggui Wang2 Jinkui Ren2 Xiantao Zhang2 Shijian Wang5 Jinghong Liu6 Linfeng Zhang1† 1Shanghai Jiao Tong University 2Alibaba Group 3Tianji KernalMind Co., Ltd. 4The Hong Kong University of Science and Technology 5Southeast University 6Renmin University of China 7The University of Tokyo Project: https://egogenesis.github.io/ Abstract Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present EgoGenesis, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. EgoGenesis builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 EgoGenesis-generated trajectories improves out-of-distribution real-robot success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization. 22footnotetext: Corresponding author: Linfeng Zhang. 1 Introduction In the current field of embodied AI, training World Action Models has increasingly become a promising and popular paradigm. Training these models requires egocentric observations paired with action trajectories across a diverse range of scenes, objects, and embodiments [40, 20, 18]. Collecting such data with real robots is expensive: each new trajectory requires physical execution, reset, and supervision, while failures can damage objects or hardware. Large egocentric corpora demonstrate the value of this perspective for capturing hand-object contact and transferable skills, yet diverse, geometry-calibrated video–action pairs remain scarce [13, 14, 17, 19]. Recent video generation models and egocentric data engines offer a promising alternative: starting from limited demonstrations, they can generate additional egocentric interactions with varied appearances and environments [7, 8, 34, 23, 38, 16]. These synthetic rollouts are useful for downstream training when their visual observations remain consistent with the associated action labels. Figure 1: Motivation from diagnosing failures in existing egocentric video generators. Existing egocentric video generators still struggle to simulate coherent, controllable interactions, as evidenced by the two representative failure modes in Figure 1. Wan2.1-14B [34], a strong general video prior, produces uncontrolled camera motion and a gripper trajectory that departs from the intended execution because neither trajectory is provided as an explicit geometric condition. Although controllable video methods introduce 2D skeletons, masks, point trajectories, or generic control images [37, 43, 44, 9], these signals do not fully constrain metric camera and end-effector motion. RynnWorld-TeleOp [47], by contrast, is action-conditioned, yet its generated scene and manipulated object drift over time, which we attribute to insufficient first-frame anchoring as occlusions and overfitted end-effector for training distribution [45]. These complementary failures motivate a central question: how can we simulate an embodied interaction with controllable camera and end-effector trajectories while preserving a stable scene and consistent object identity? To address these geometric failures, we propose EgoGenesis, an egocentric world-action model that generates controllable video–action pairs for embodied data augmentation, as summarized in Figure LABEL:fig:teaser. EgoGenesis innovatively involves decoupling the two modalities and processing them separately. For the world modeling, Online Anchored Projective Memory (OAPM) keeps an immutable first-frame 3D scene anchor while refreshing a compact, confidence-weighted recent snapshot during autoregressive generation. Camera-aware projective attention reads both states in the target view, allowing generated history to update the current scene without overwriting its stable anchor. For the action encoding, Action-3D Rotary Position Embedding (A3D-RoPE) rasterizes skeleton or end-effector joints onto the latent patch grid, unprojects them with camera trajectories , and encodes their reference-frame metric coordinates as rotary phases in skeleton-to-video cross-attention. Together, these components improve scene persistence and action alignment over long egocentric generation. To improve the generalization of our approach and prevent overfitting to a specific scene or end-effector, we train EgoGenesis on a source-balanced collection spanning human hands, dexterous hands, parallel grippers, and robot-arm end-effectors. The generated video–action pairs augment real demonstrations for tuning downstream world-action models and manipulation policies. On four bimanual and four single-arm real-robot skills, adding 400 generated trajectories to 400 real trajectories raises OOD success from 53% to 70% for bimanual tasks and from 77% to 84% for single-arm tasks (Table 3 and Figure 8). These gains demonstrate improved synthetic rollout quality from the use of EgoGenesis: expanding scarce embodied data to improve real-world generalization. Figure 2: EgoGenesis architecture. Text, noisy video, and skeleton embeddings provide the conditioning inputs. Autoregressive DiT generates and appends video frames. OAPM maintains anchored and recent 3D scene slots and refreshes the recent state online. A3D-RoPE injects metric action geometry into skeleton-to-video cross-attention. In summary, our contributions are: • We introduce OAPM, an anchored projective 3D scene memory with online refresh for autoregressive egocentric generation. • We introduce A3D-RoPE, which injects camera-aware metric skeleton and end-effector geometry into cross-attention for controllable action-aligned video generation. • We show that augmenting real-robot data with EgoGenesis-generated trajectories improves OOD generalization on bimanual and single-arm manipulation tasks. 2 Related Work Video and World Action Models Large video diffusion and causal generators provide reusable priors for high-quality and long-horizon synthesis [7, 8, 34, 28, 51, 24], while editing and control methods introduce masks, trajectories, camera motion, and entity-level conditions [12, 37, 43, 39, 2, 3]. World-action models further use pixels, features, edited images, or imagined rollouts for prediction and robot control [42, 41, 49, 4, 20, 46, 15, 33, 40, 50, 18, 11]. Recent frontier video models including generic priors (Wan and Cosmos), interaction-conditioned generators (EgoHOI, Mask2IV, and CosHand), and egocentric models (EgoSim and RynnWorld-Teleop) [21, 22, 32, 16, 47] are unsatisfying in preventing geometric failures such as drifting. Besides, all of them treats end-effector simply without carefully modeling the geometric relationship between the end-effector and the scene in camera coordinate axes. Egocentric Interaction Data Generation Large egocentric corpora and human-to-robot pipelines provide egocentric hand–object observations, 3D geometry, and transferable action cues [13, 14, 17, 5, 30, 35, 19]. Synthetic data engines complement them by generating egocentric interactions from environments, video–action corpora, tasks, or explicit hand and camera controls [23, 38, 48, 45, 44, 9, 21, 22]. Camera and hand geometry can improve such control [25, 36, 29], while generated or semantically varied observations can improve policy generalization [6, 10, 26, 27, 1]. EgoGenesis connects these directions by synthesizing action-aligned egocentric rollouts and using them to improve downstream WAM generalization with flexible switching between various environments, object manipulation, end effector observation attributes as synthetic video-action pairs. 3 Preliminaries Problem setup and latent video modeling. Given an egocentric manipulation sample, we observe an initial RGB frame I0I_0, a language instruction y, camera intrinsics and extrinsics bb=1B\C_b\_b=1^B over B temporal blocks, an initial scene representation, and the corresponding action geometry. The action may be a MANO hand skeleton, a dexterous-hand skeleton, or a robot end-effector/gripper trajectory in a unified keypoint format. Our goal is to generate a rollout V V that follows the specified action geometry while preserving the scene, camera motion, object identity, and plausible contact dynamics; V V can serve as both a world-model prediction and synthetic embodied data. During training, let V be the clean target video, EvaeE_vae and DvaeD_vae the frozen VAE encoder and decoder, and Z=Evae(V)Z=E_vae(V) the video latent, partitioned into B temporal blocks Z=[Z1,…,ZB]Z=[Z_1,…,Z_B]. Let bc_b collect the language, camera, scene-memory, and action conditions for block b. The causal key–value cache <bK_<b contains the committed blocks preceding b. At flow time t∈[0,1]t∈[0,1], a clean block ZbZ_b is interpolated with Gaussian noise as Zbt=(1−σt)Zb+σtεb,εb∼(0,),Z_b^t=(1- _t)Z_b+ _t _b, _b (0,I), (1) where I is the identity matrix and we use the linear schedule σt=t _t=t. The DiT predicts the flow field v^bt v_b^t =Fθ(Zbt,t;b,<b), =F_θ\! (Z_b^t,t;c_b,K_<b ), (2) ℒFM _FM =[‖v^bt−(εb−Zb)‖22]. =E [\| v_b^t-( _b-Z_b)\|_2^2 ]. Sampling integrates the learned flow from noise to data with step size Δt t, e.g. using the Euler update Zbt−Δt=Zbt−Δtv^bt.Z_b^t- t=Z_b^t- t\, v_b^t. (3) Once Zb0Z_b^0 is denoised, it is committed as ZbZ_b and appended to <b+1K_<b+1; the next block is generated from this committed history and its block-aligned conditions. 4 Method Model Overview EgoGenesis instantiates the chunkwise DiT model of Section 3 with three conditioning streams, as illustrated in Figure 2. The first is the noisy video block ZbtZ_b^t, with its first frame pinned during generation. The second is scene memory: compact 3D patch tokens with metric coordinates and confidence. The third is action control, denoted by bS_b: a dense skeleton or end-effector latent with metric 3D coordinates. Text and image encoders provide task and appearance context. After flow integration reaches t=0t=0, Zb0Z_b^0 is committed to the causal cache <b+1K_<b+1 and may refresh the recent scene memory before block b+1b+1. Online Anchored Projective Memory Static first-frame conditioning becomes stale as the camera moves and objects change state. OAPM therefore maintains two abstract scene slots: an immutable anchor ℳaM_a from the clean first frame and a replace-only recent slot ℳrbM_r^b from generated history. Before each block generation, we encode the two slots with the pretrained VGGT-Ω and directly use its 3D scene reconstruction features as the scene embedding: b=SceneEncodeΩ(ℳa⊕ℳrb),M_b=SceneEncode_ \! (M_a _r^b ), (4) where ⊕ denotes slot concatenation and SceneEncodeΩSceneEncode_ extracts the pretrained VGGT-Ω features. Selected DiT layers then use the resulting memory bank in a gated cross-attention update: Q=WQHK=WKbV=WVb Q=W_QH K=W_KM_b V=W_VM_b (5) H←H+GatedCrossAttn(Q,K,V). H← H+GatedCrossAttn(Q,K,V). This lets the hidden states read both the stable scene anchor and the current interaction state through the gated pathway in Figure 2. After every generated block, the result is decoded by the pretrained VAE and committed to the causal KV cache <b+1K_<b+1. With refresh interval srs_r, OAPM decodes the causally visible generated prefix, encodes its latest RGB frame into a new snapshot, and replaces only ℳrbM_r^b: I^b I_b =RecentFrame(Dvae(Z≤b)), =RecentFrame\! (D_vae(Z_≤ b) ), (6) Z≤b Z_≤ b =[Z1,…,Zb], =[Z_1,…,Z_b], ℳrb+1 _r^b+1 =E3D(I^b),b≡0(modsr),ℳrb,otherwise. = The anchor is never overwritten since the initial frame provides the clearest and most stable conditions, and new generated frames are used to update the established portions of the scene memory by applying learned gating parameters. This update mechanism ensures scene consistency during the dynamic refreshing and generation process, effectively mitigating the issue of scene or object drifting. Action-3D Rotary Position Embedding Figure 3: Qualitative comparison on flattening shorts and assembling a square table; EgoGenesis better preserves embodiment, scene layout, and contact-driven object evolution. Category Method PSNR↑ SSIM↑ LPIPS↓ Kpt.Err↓ Phys.Faith↑ Subj. Cons.↑ Bg. Cons.↑ Generic video models Wan2.1-14B-InP [34] 19.991 0.8288 0.3356 0.1295 0.7444 0.8786 0.9437 Wan2.2-5B-Control [34] 16.5896 0.7542 0.4323 0.0813 0.7296 0.7761 0.8956 Cosmos3-Nano [28] 19.0996 0.8052 0.3810 0.1115 0.8093 0.8905 0.9392 Egocentric video models EgoHOI [21] 20.223 0.7883 0.3326 0.1884 0.7537 0.8590 0.9193 Mask2IV∗ [22] 18.6821 0.8102 0.3789 0.2086 0.6018 0.8786 0.9353 EgoSim-14B [16] 19.4114 0.8317 0.2750 0.0811 0.7759 0.9101 0.9491 CosHand [32] 18.7549 0.8033 0.4039 0.1202 0.4667 0.8089 0.9104 RynnWorld-TeleOp [47] 18.8247 0.8215 0.3913 0.2107 0.7870 0.8887 0.9466 Ours EgoGenesis 21.8609 0.8509 0.2399 0.0501 0.8278 0.8923 0.9546 Table 1: Generation quality on held-out trajectories under identical scene and action conditioning. Blue bold cells denote the best results, and green underlined cells denote the second-best results for each metric. Rendered skeleton videos provide visual control but do not expose metric depth. Our implementation keeps the pretrained WAN self-attention positional path unchanged and adds A3D-RoPE to the frame-local skeleton-to-video cross-attention adapters. The skeleton or end-effector latent is patchified on the video grid. Let ℐbI_b denote the patches covered by the rendered skeleton in block b; bX_b collects their anchor-frame metric 3D coordinates, while XaX_a denotes the scalar coordinate along axis a∈x,y,za∈\x,y,z\. Metric rotary cross-attention. At an A3D-RoPE adapter, queries originate from the video hidden states, whereas keys and values originate from the encoded skeleton tokens. In the following, Q, K, and V refer only to the QKV entries indexed by ℐbI_b after their standard linear projections, rather than to QKV over the full patch grid. This limits metric rotations to patches with valid action coordinates and leaves background tokens unchanged. A3D-RoPE splits the selected query and key channels into three groups, Q=[Qx‖Qy‖Qz],K=[Kx‖Ky‖Kz].Q=[Q^x\,\|\,Q^y\,\|\,Q^z], K=[K^x\,\|\,K^y\,\|\,K^z]. (7) Within each group, adjacent channels form standard two-dimensional RoPE pairs. If MaM_a pairs are assigned to axis a∈x,y,za∈\x,y,z\, pair m uses the channel pair (u,v)(u,v): θa,m(Xa) _a,m(X_a) =sXaκ−m/Ma, =sX_aκ^-m/M_a, (8) [u′v′] bmatrixu \\ v bmatrix =[cosθa,m(Xa)−sinθa,m(Xa)sinθa,m(Xa)cosθa,m(Xa)][uv], = bmatrix _a,m(X_a)&- _a,m(X_a)\\ _a,m(X_a)& _a,m(X_a) bmatrix bmatrixu\\ v bmatrix, Here, θa,m(Xa) _a,m(X_a) is the rotation angle induced by the metric coordinate XaX_a at the m-th RoPE frequency, with s=4s=4 and κ=104κ=10^4. For each supported patch in block b, its entry in bX_b is represented by the anchor-frame coordinates (Xx,Xy,Xz)(X_x,X_y,X_z); thus, XaX_a is one axis component of bX_b, rather than a separate coordinate. Applying Eq. (8) along all three axes and to every supported patch yields the blockwise rotation RbR_X_b, which is applied directly to Q and K: Q~=Rb(Q),K~=Rb(K). Q=R_X_b(Q), K=R_X_b(K). (9) The rotated Q/KQ/K and selected V update only HℐbH_I_b through the gated cross-attention pathway shown in Figure 2. Coordinate construction and implementation details are provided in the supplementary material. 5 Experiments Experimental Setup We evaluate EgoGenesis as both a generative world-action model and a data augmentation engine on a mixed held-out benchmark containing test subsets conditioned on robot grippers, gripper end-effector (EEF) trajectories, and human-hand skeletons. All training and test splits are strictly disjoint, with no clip or trajectory overlap. Every video is standardized to 81 frames at 16 FPS and a spatial resolution of 832×480832× 480. Training uses a source-balanced 210K-clip corpus drawn from EgoDex [17], AgiBot [1], RoboTwin [27], Real-world Ego, and DexJoCo. In the first stage, we conducted 6k steps of SFT using pre-processed momory slots ℳaM_a and ℳrM_r. For the second stage for autoregressive training, we utilized generated frame as ℳrM_r for auto-regressive training, which also ran for 6k steps. We report PSNR, SSIM, and LPIPS for frame-level fidelity; Kpt.Err (Hand Keypoint End-Point Error) for action alignment; Phys.Faith, assessed by Kimi K2.7, for plausible contact and object motion; and Subj. Cons. (Subject Consistency) and Bg. Cons. (Background Consistency) for temporal stability. Kpt.Err is computed only on the 50-clip EgoDex subset. EgoGenesis is trained on 8 NVIDIA A100 GPUs, while downstream real-robot inference and LingBot-VA training and testing are performed on 8 NVIDIA H100 GPUs. We compare with general image-to-video and controllable video baselines, including Wan2.1-Fun-14B-Inp, Wan2.2-5B-Control, EgoHOI, Mask2IV, and EgoSim-14B. Further details are provided in the Supplementary Material. Qualitative Results Generalization across embodiments and tasks remains challenging, as evidenced by Figure 3, where we compare the AgiBot-World shorts-flattening task and the EgoDex square-table assembly task. RynnWorld-TeleOp exhibits strong gripper overfitting, replacing the human hand with gripper-like morphology, while its assembled object also deviates substantially from the reference. Cosmos3 and EgoHOI follow the shorts-flattening instruction poorly and produce weak contact interactions; Cosmos3 additionally hallucinates an unintended object during table assembly. Wan2.2-5B-Control suffers severe drift and loses hand and end-effector cues. In contrast, EgoGenesis remains close to the ground truth in both tasks while preserving coherent hand–object and gripper–object interactions. Quantitative Results To test whether geometry-aware conditioning improves interaction quality without sacrificing appearance, we compare EgoGenesis with generic and egocentric generators under identical scene and action conditions. EgoGenesis ranks first on six of seven metrics and second on subject consistency, as reported in Table 1. In particular, its Kpt.Err of 0.0501 and Phys.Faith of 0.8278 provide direct evidence of stronger action alignment and contact plausibility, while the best PSNR (21.8609), LPIPS (0.2399), and background consistency (0.9546) rule out a trade-off in visual fidelity or temporal stability. The joint gains therefore support more reliable action-conditioned interaction rather than appearance alone. Ablation Study To verify that the gains arise from both OAPM and A3D-RoPE, we vary one component at a time while fixing the Wan2.2-5B-Control backbone, training protocol, and complementary component. With A3D-RoPE fixed, adding the recent slot ℳrM_r to the anchor ℳaM_a raises PSNR from 20.4135 to 21.8609 and reduces LPIPS from 0.2533 to 0.2399, as shown in Table 2; this supports the value of tracking the evolving scene rather than relying only on the first frame. With OAPM fixed, A3D-RoPE outperforms both RoPE and PRoPE on all three compact fidelity metrics, including an SSIM increase from 0.8198 and 0.8408 to 0.8509. These controlled improvements attribute the full-model gains to both scene-state maintenance and metric action encoding. Furthermore, we tested a baseline AR model without any of our proposed methods. The results showed that it underperformed across all relevant metrics, demonstrating that each of our proposed techniques is individually effective. Component Setting Setting PSNR↑ SSIM↑ LPIPS↓ Wan2.2-5B-Control-AR None 19.9238 0.7812 0.3028 ℳaM_a only A3D-RoPE fixed 20.4135 0.8385 0.2533 +ℳrM_r (OAPM) 21.8609 0.8509 0.2399 RoPE [31] OAPM fixed 21.4250 0.8198 0.2838 PRoPE [25] 21.8421 0.8408 0.2481 A3D-RoPE 21.8609 0.8509 0.2399 Table 2: Compact core-component ablation on Wan2.2-5B-Control. The complete action, physical-faithfulness, and consistency metrics are reported in the supplementary material. Figure 4: Accumulated depth and camera errors over an 80-frame rollout; A3D-RoPE consistently reduces geometric drift. Geometric drift. A3D-RoPE substantially reduces long-horizon geometric drift, as demonstrated by the controlled comparison in Figure 4, which keeps OAPM and the training setup fixed. Depth-ERR is clip-scale-aligned AbsRel within action-keypoint regions, and Cam-ERR is the per-pixel ℓ2 _2 error on first-camera-relative Plücker coordinates; both are computed with VGGT-Ω on 15 sampled non-anchor frames per clip. RoPE provides no metric action geometry, while PRoPE improves camera-aware video self-attention but does not encode the skeleton trajectory in metric 3D within action cross-attention. A3D-RoPE aligns the action keys and video queries with reference-frame 3D rotary coordinates, substantially reducing both local depth drift around the hand or end-effector and global camera-ray inconsistency. At frame 80, A3D-RoPE reduces Depth-ERR and Cam-ERR by 79.30% and 78.26% relative to RoPE, and by 66.46% and 49.04% relative to PRoPE, respectively. This sustained reduction shows that A3D-RoPE preserves action geometry throughout long autoregressive rollouts rather than improving only frame-level appearance; the corresponding Kpt.Err comparison is reported in the supplementary component-ablation table. Figure 5: A3D-RoPE concentrates spatial influence on end-effectors and contact-relevant regions across embodiments. Spatial influence of A3D-RoPE. A3D-RoPE concentrates its spatial influence on end-effectors and nearby contact regions, as visualized in Figure 5; warmer regions indicate larger changes in the cross-attention output when metric 3D encoding is enabled. By injecting end-effector information, object-interaction depth, and camera displacement relative to the hand frame, A3D-RoPE consistently focuses the response on interaction-relevant regions rather than the background across EgoDex, AgiBot, and RoboTwin. Figure 6: OAPM preserves persistent scene content and updates the current interaction state periodically. Qualitative scene-memory ablation. The anchor and recent slots in OAPM play complementary roles, as demonstrated in Figure 6. The full model in (a) keeps the immutable ℳaM_a to preserve the clean scene layout and persistent object identity while replacing ℳrM_r from the most recent committed block to track the current interaction state. The anchor-only variant in (b) remains tied to stale object states, whereas removing OAPM in (c) allows objects to disappear or drift, as highlighted by the red boxes. Maintaining both slots therefore avoids overwriting the stable scene prior without sacrificing online state updates. Downstream WAM Generalization Figure 7: Eight real-robot tasks used to evaluate downstream WAM generalization. Our eight-task real-robot suite spans precise contact, multi-stage execution, and coordination across embodiments, as summarized in Figure 7, where arrows indicate the intended motion. The bimanual tasks test coordinated folding, sequential pick-and-place, object handoff, and sequential place-and-push, while the single-arm tasks test stacking, pick-and-place, target selection, and pushing. To test whether the generated rollouts provide useful supervision beyond perceptual gains, we fix the downstream WAM architecture and training schedule and vary only the training-data composition. Each real trajectory is converted into a first-frame scene anchor, camera trajectory, action skeleton or end-effector track, and language prompt; EgoGenesis then resimulates it under edited appearance or scene conditions while preserving the action-label space. Augmenting 400 real trajectories with 400 synthetic trajectories raises OOD success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, as shown in Table 3. It also reduces the corresponding ID-to-OOD losses from 7 to 4 points and from 19 to 6 points. Because the policy and optimization are held fixed, these gains demonstrate that EgoGenesis supplies complementary variation that improves real-robot generalization under held-out appearances and layouts. Training data Split Single-arm SR↑ SR Loss↓ Dual-arm SR↑ SR Loss↓ 400 real ID 84.0 ↓ 7.0 72.0 ↓ 19.0 OOD 77.0 53.0 400 synth. ID 84.0 ↓ 8.0 69.0 ↓ 13.0 OOD 76.0 56.0 400 real + 400 synth. ID 88.0 ↓ 4.0 76.0 ↓ 6.0 OOD 84.0 70.0 Table 3: Real-robot task success rates (%). SR denotes Success Rate; ID and OOD denote in-distribution and out-of-distribution evaluation, respectively. OOD stage progress. Synthetic augmentation improves not only binary OOD success but also how far failed rollouts progress, as demonstrated in Figure 8. The analysis resolves each rollout into its furthest stage over 25 trials per task; the rightmost mode is full task completion, and modes to its left denote approach, grasp, transport, and placement progress before failure. With 400 real trajectories alone, failures are distributed across the intermediate stages. Fine-tuning the downstream WAM with 400 additional EgoGenesis-generated trajectories shifts both distributions toward later stages and complete executions: average completion rises from 53% to 70% for the bimanual suite and from 77% to 84% for the single-arm suite. This stage-wise shift independently corroborates the success-rate evidence in Table 3, showing that augmentation improves partial execution even when a rollout does not fully succeed. Figure 8: OOD task progress with and without EgoGenesis-generated training data. 6 Conclusion We introduced EgoGenesis as an autoregressive video generation model with OAPM and A3D-RoPE to address the scarcity of diverse egocentric manipulation data for embodied AI. OAPM combines a persistent 3D scene anchor with online recent-state refresh, while A3D-RoPE injects 3D geometry of the end-effector into gated cross-attention. Together, they produce high-quality egocentric rollouts with stronger visual fidelity, geometric stability, and action alignment. It allows for spanding current scarce video-action pairs with various attributes for egocentric data augmentation. Beyond generation quality, current real data with our EgoGenesis-augmented data, can be used to train the downstream World Action Model to improve its generalization performance on both single- and dual-arm real-robot tasks. These results show that high-quality egocentric synthesis can expand limited real data and provide useful supervision for downstream WAM generalization. References [1] AgiBot-World-Contributors (2025) AgiBot world colosseo: a large-scale manipulation platform for scalable and intelligent embodied systems. External Links: Link Cited by: §2, §5. [2] R. Akkerman, H. Feng, M. J. Black, D. Tzionas, and V. Fernández Abrevaya (2024) InterDyn: controllable interactive dynamics with video diffusion models. External Links: Link Cited by: §2. [3] H. Alzayer, W. Huang, H. Chen, C. Luey, L. Zhang, M. Agrawala, G. Wetzstein, L. Fei-Fei, Y. Du, J. Wu, and J. Huang (2026) Masked visual actions for unified world modeling. External Links: Link Cited by: §2. [4] M. Assran, A. Bardes, D. P. Fan, et al. (2025) V-JEPA 2: self-supervised video models enable understanding, prediction and planning. Cited by: §2. [5] P. Banerjee et al. (2025) HOT3D: hand and object tracking in 3d from egocentric multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Document Cited by: §2. [6] H. Bharadhwaj et al. (2024) Gen2Act: human video generation in novel scenarios enables generalizable robot manipulation. Cited by: §2. [7] A. Blattmann, T. Dockhorn, S. Kulal, et al. (2023) Stable video diffusion: scaling latent video diffusion models to large datasets. Cited by: §1, §2. [8] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan (2024) VideoCrafter2: overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Document Cited by: §1, §2. [9] Y. Chen, X. Shi, X. Wu, X. Wang, P. Wan, and Y. Liu (2026) HandsOnWorld: unconstrained egocentric video generation with camera-disentangled hand control. Cited by: §1, §2. [10] Z. Q. Chen, Z. Mandi, H. Bharadhwaj, M. Sharma, S. Song, A. Gupta, and V. Kumar (2024) Semantically controllable augmentations for generalizable robot learning. The International Journal of Robotics Research. External Links: Document Cited by: §2. [11] Y. Du et al. (2025) Large video planner enables generalizable robot control. Note: arXiv preprint Cited by: §2. [12] M. Geyer, O. Bar-Tal, S. Bagon, and T. Dekel (2023) TokenFlow: consistent diffusion features for consistent video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Cited by: §2. [13] K. Grauman, A. Westbury, E. H. Byrne, et al. (2022) Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Document Cited by: §1, §2. [14] K. Grauman, A. Westbury, L. Torresani, et al. (2024) Ego-Exo4D: understanding skilled human activity from first- and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Document Cited by: §1, §2. [15] Y. Guo, L. X. Shi, J. Chen, and C. Finn (2025) Ctrl-world: a controllable generative world model for robot manipulation. External Links: Link Cited by: §2. [16] J. Hao, M. Jia, R. Wang, H. Zhu, J. Cao, X. Liu, R. Yi, L. Ma, J. Pang, and X. Xu (2026) EgoSim: egocentric world simulator for embodied interaction generation. External Links: Link Cited by: §1, §2, Table 1. [17] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang (2026) EgoDex: learning dexterous manipulation from large-scale egocentric video. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §5. [18] Y. Hu, Y. Guo, P. Wang, X. Chen, Y. Wang, J. Zhang, K. Sreenath, C. Lu, and J. Chen (2025) Video prediction policy: a generalist robot policy with predictive visual representations. In Proceedings of the International Conference on Machine Learning, Cited by: §1, §2. [19] S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu (2025) EgoMimic: scaling imitation learning via egocentric video. In Proceedings of the IEEE International Conference on Robotics and Automation, External Links: Document Cited by: §1, §2. [20] B. Li, X. Yin, M. Lin, Y. Zhang, and D. Xu (2026) EgoWAM: world action models beyond pixels with in-the-wild egocentric human data. External Links: Link Cited by: §1, §2. [21] D. Li, L. Liu, B. Liu, S. Zhou, J. Feng, Z. Lu, M. Zheng, C. You, and Z. Fan (2026) Egocentric world model for photorealistic hand-object interaction synthesis. External Links: Link Cited by: §2, §2, Table 1. [22] G. Li, B. Zhao, J. Yang, and L. Sevilla-Lara (2026) Mask2IV: interaction-centric video generation via mask trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 6091–6099. External Links: Document, Link Cited by: §2, §2, Table 1. [23] G. Li, K. Zhao, S. Zhang, X. Lyu, M. Dusmanu, Y. Zhang, M. Pollefeys, and S. Tang (2024) EgoGen: an egocentric synthetic data generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Document Cited by: §1, §2. [24] L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu (2026) Causal world modeling for robot control. External Links: Link Cited by: §2. [25] R. Li, B. Yi, J. Liu, H. Gao, Y. Ma, and A. Kanazawa (2025) Cameras as relative positional encoding. External Links: Link Cited by: §2, Table 2. [26] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, Cited by: §2. [27] Y. Mu, T. Chen, Z. Chen, S. Peng, Z. Lan, Z. Gao, Z. Liang, Q. Yu, Y. Zou, M. Xu, L. Lin, Z. Xie, M. Ding, and P. Luo (2025) RoboTwin: dual-arm robot benchmark with generative digital twins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, External Links: Link Cited by: §2, §5. [28] NVIDIA, Aditi, N. Agarwal, et al. (2026) Cosmos 3: omnimodal world models for physical AI. External Links: Link Cited by: §2, Table 1. [29] G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik (2024) Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2. [30] K. Shaw et al. (2024) HumanEgo: zero-shot robot learning from minutes of human egocentric videos. Note: arXiv preprint Cited by: §2. [31] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) RoFormer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. External Links: Document, Link Cited by: Table 2. [32] S. Sudhakar, R. Liu, B. Van Hoorick, C. Vondrick, and R. S. Zemel (2024) Controlling the world by sleight of hand. In Computer Vision – ECCV 2024, p. 414–430. External Links: Document, Link Cited by: §2, Table 1. [33] J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen (2026) VLA-JEPA: enhancing vision-language-action model with latent world model. External Links: Link Cited by: §2. [34] Wan Team (2025) Wan: open and advanced large-scale video generative models. External Links: Link Cited by: §1, §1, §2, Table 1, Table 1. [35] G. Wang, K. Ren, A. Morgan, Y. Chen, H. H. Qian, P. Chanrungmaneekul, and K. Hang (2026) EgoInfinity: a web-scale 4d hand-object interaction data engine for any-view robot retargeting and video-to-action robot learning. External Links: Link Cited by: §2. [36] J. Wang et al. (2025) VGGT: visual geometry grounded transformer. Note: arXiv preprint Cited by: §2. [37] X. Wang, H. Yuan, S. Zhang, D. Chen, J. Wang, Y. Zhang, Y. Shen, D. Zhao, and J. Zhou (2023) VideoComposer: compositional video synthesis with motion controllability. In Advances in Neural Information Processing Systems, Cited by: §1, §2. [38] X. Wang, K. Zhao, F. Liu, J. Wang, G. Zhao, X. Bao, Z. Zhu, Y. Zhang, and X. Wang (2024) EgoVid-5M: a large-scale video-action dataset for egocentric video generation. Cited by: §1, §2. [39] Z. Wang, Z. Yuan, X. Wang, Y. Li, T. Chen, M. Xia, P. Luo, and Y. Shan (2024) MotionCtrl: a unified and flexible motion controller for video generation. In ACM SIGGRAPH Conference Papers, External Links: Document Cited by: §2. [40] H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong (2024) Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, Cited by: §1, §2. [41] J. Wu, S. Yin, N. Feng, X. He, D. Li, J. Hao, and M. Long (2024) iVideoGPT: interactive VideoGPTs are scalable world models. In Advances in Neural Information Processing Systems, Cited by: §2. [42] S. Yang, Y. Du, K. Ghasemipour, J. Tompson, L. P. Kaelbling, D. Schuurmans, and P. Abbeel (2024) Learning interactive real-world simulators. In International Conference on Learning Representations, Cited by: §2. [43] S. Yin, C. Wu, J. Liang, J. Shi, H. Li, M. Gong, and N. Duan (2024) DragNUWA: fine-grained control in video generation by integrating text, image, and trajectory. In International Conference on Learning Representations, Cited by: §1, §2. [44] C. Zhang, B. Ye, B. Chen, A. Delitzas, F. Wang, M. Pollefeys, and X. Wang (2026) Controllable egocentric video generation via occlusion-aware sparse 3d hand joints. Cited by: §1, §2. [45] L. Zhang, J. Ye, Y. Wang, M. Zhong, M. Cao, W. Xia, B. Zeng, Z. Zhang, and H. Tang (2025) EgoLCD: egocentric video generation with long context diffusion. Cited by: §1, §2. [46] Y. Zhang, W. Zhang, Z. Qi, H. Zhang, H. Lin, J. Zhang, Y. Mu, X. Yang, W. Zeng, and X. Jin (2026) ImageWAM: do world action models really need video generation, or just image editing?. External Links: Link Cited by: §2. [47] H. Zhao, X. Zhao, H. Li, B. Gong, K. Li, S. Huang, X. Li, D. Zhao, and Z. Li (2026) RynnWorld-teleop: an action-conditioned world model for digital teleoperation. External Links: Link Cited by: §1, §2, Table 1. [48] H. Zhao et al. (2025) TASTE-Rob: advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §2. [49] G. Zhou, H. Pan, Y. LeCun, and L. Pinto (2025) DINO-WM: world models on pre-trained visual features enable zero-shot planning. In Proceedings of the International Conference on Machine Learning, Cited by: §2. [50] S. Zhou, Y. Du, J. Chen, Y. Li, D. Yeung, and C. Gan (2024) RoboDreamer: learning compositional world models for robot imagination. Cited by: §2. [51] H. Zhu et al. (2026) Causal forcing: autoregressive video generation with causal diffusion models. Note: arXiv preprint External Links: Link Cited by: §2. Supplementary Material for EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE Zexuan Yan, Yuzhou Wu, Yue Ma, Zonghang He, Kaibo Yin, Xiaobing Tu Yinggui Wang, Jinkui Ren, Xiantao Zhang, Shijian Wang, Jinghong Liu, Linfeng Zhang† Contents 1 Introduction 2 Related Work Video and World Action Models Egocentric Interaction Data Generation 3 Preliminaries 4 Method Model Overview Online Anchored Projective Memory Action-3D Rotary Position Embedding 5 Experiments Experimental Setup Qualitative Results Quantitative Results Ablation Study Downstream WAM Generalization 6 Conclusion References A Technical Details for OAPM and A3D-RoPE OAPM Encoding, Read, and Refresh A3D-RoPE Encoding and Coordinate Construction B Training Corpus and Autoregressive Procedure Training Data Autoregressive Training and Inference AR Generation with OAPM and A3D-RoPE C Complete Component Ablation D Detailed Downstream Real-Robot Results Tianji M6 Platform, Tasks, and Evaluation Protocol Task Stages and Completion Criteria Downstream WAM Training and Inference Details E Additional Qualitative Comparisons Cross-Embodiment Simulation in an Unseen Environment F Evaluation Metrics and Kimi K2.7 Prompt Main-Table Metrics Kimi K2.7 Prompt for Physical Faithfulness Appendix A Technical Details for OAPM and A3D-RoPE OAPM Encoding, Read, and Refresh OAPM treats the immutable anchor ℳaM_a and online recent memory ℳrbM_r^b as abstract slots. Before block b, the pretrained VGGT-Ω encodes their concatenation, and its 3D scene reconstruction features are used directly as the scene embedding: b=SceneEncodeΩ(ℳa⊕ℳrb).M_b=SceneEncode_ \! (M_a _r^b ). (10) The resulting tokens carry scene features, reference-frame 3D coordinates, and confidence. The memory read uses the same gated cross-attention form as the main paper: Q Q =WQH, =W_QH, K K =WKb, =W_KM_b, (11) V V =WVb, =W_VM_b, H H ←H+ ← H+ GatedCrossAttn(Q,K,V). (Q,K,V). In online mode, after every srs_r committed AR blocks, the pipeline decodes the causally visible latent prefix and uses its most recent RGB frame to construct a new snapshot: ℳrb+1 _r^b+1 =E3D(RecentFrame( =E_3D\! (RecentFrame ( (12) Dvae(Z≤b))), D_vae(Z_≤ b) ) ), b b ≡0(modsr). ≡ 0 s_r. The anchor is unchanged and the recent snapshot is replaced, combining stable reference geometry with the latest scene state. A3D-RoPE Encoding and Coordinate Construction The rendered-skeleton VAE latent is patchified on the video grid. Let ℐbI_b denote the patches covered by the skeleton in block b; bX_b collects their anchor-frame metric 3D coordinates, and XaX_a is one component along axis a∈x,y,za∈\x,y,z\. Queries originate from the video hidden states, whereas keys and values originate from the patch-aligned skeleton tokens. In the following, Q, K, and V refer only to the QKV entries indexed by ℐbI_b after their standard linear projections, rather than to QKV over the full patch grid. A3D-RoPE splits the selected query and key channels into three spatial groups, Q=[Qx‖Qy‖Qz],K=[Kx‖Ky‖Kz].Q=[Q^x\,\|\,Q^y\,\|\,Q^z], K=[K^x\,\|\,K^y\,\|\,K^z]. (13) Within each group, adjacent channels form standard two-dimensional RoPE pairs. If MaM_a pairs are assigned to axis a∈x,y,za∈\x,y,z\, pair m uses θa,m(Xa) _a,m(X_a) =sXaκ−m/Ma, =sX_aκ^-m/M_a, (14) [u′v′] bmatrixu \\ v bmatrix =[cosθa,m(Xa)−sinθa,m(Xa)sinθa,m(Xa)cosθa,m(Xa)][uv], = bmatrix _a,m(X_a)&- _a,m(X_a)\\ _a,m(X_a)& _a,m(X_a) bmatrix bmatrixu\\ v bmatrix, Here, θa,m(Xa) _a,m(X_a) is the rotation angle induced by XaX_a at the m-th RoPE frequency, with s=4s=4 and κ=104κ=10^4. For each supported patch in block b, its entry in bX_b is represented by (Xx,Xy,Xz)(X_x,X_y,X_z) in the anchor frame. Therefore, XaX_a is one axis component of bX_b, and applying the axis-wise rotation to every supported patch yields RbR_X_b. The video and action features are then rotated before gated cross-attention: Q~ Q =Rb(Q), =R_X_b(Q), K~ K =Rb(K), =R_X_b(K), (15) Hℐb H_I_b ←Hℐb+GatedCrossAttn(Q~,K~,V). ← H_I_b+GatedCrossAttn( Q, K,V). The update is written back only at ℐbI_b because these are the patches with valid action coordinates; unselected background states remain unchanged. We next detail how the skeleton-supported coordinates tpX_tp are constructed. For skeleton edge e=(j,k)e=(j,k), let tju_tj and tku_tk be normalized image coordinates and dtj,dtkd_tj,d_tk their valid depths. The projection of patch center pu_p onto the edge is αtpe=clip[0,1](p−tj)⊤(tk−tj)‖tk−tj‖22, _tpe=clip_[0,1] (u_p-u_tj) (u_tk-u_tj)\|u_tk-u_tj\|_2^2, (16) where the numerator denotes the inner product. The perspective-correct edge depth and Gaussian support are dtpe d_tpe =dtjdtk(1−αtpe)dtk+αtpedtj, = d_tjd_tk(1- _tpe)d_tk+ _tped_tj, (17) wtpe w_tpe =eexp(−‖p−[(1−αtpe)tj+αtpetk]‖222rte2), =1_e \! (- \|u_p-[(1- _tpe)u_tj+ _tpeu_tk]\|_2^22r_te^2 ), dtp d_tp =∑ewtpedtpe∑ewtpe. = _ew_tped_tpe _ew_tpe. Here e1_e requires valid endpoints, and rte=r0+0.2‖tk−tj‖2r_te=r_0+0.2\|u_tk-u_tj\|_2 is the adaptive tube radius (with r0=0.10r_0=0.10). The supported patch ray is unprojected and expressed in the reference-camera frame as tp=[reft−1[dtpt−1¯p1]]1:3.X_tp= [V_refV_t^-1 bmatrixd_tpK_t^-1 u_p\\ 1 bmatrix ]_1:3. (18) Patches without valid skeleton support are excluded from A3D-RoPE cross-attention. Appendix B Training Corpus and Autoregressive Procedure Training Data We train on a source-balanced 210K-clip egocentric corpus. EgoDex and AgiBot contribute 100K clips each, complemented by 4K RoboTwin clips, 5K Real-world Ego clips, and 1K DexJoCo clips. Real-world Ego combines teleoperated rollouts in the downstream task environments, egocentric human-hand interactions, and first-person recordings collected on different robot embodiments. Every source is converted to a unified camera-and-pose conditioning interface. All training and test splits are disjoint at the clip and trajectory levels, and videos are standardized to 81 frames at 16 FPS and 832×480832× 480. All EgoGenesis training runs use NVIDIA A100 GPUs. Figure 9: Composition of the 210K-clip egocentric training corpus. Every source is converted to a unified camera-and-pose conditioning interface. Autoregressive Training and Inference Starting from the pretrained video prior, we train EgoGenesis with block-causal attention: two context blocks are teacher-forced and one target block is denoised at a time, with random target windows and online OAPM refresh. Training uses the source-balanced mixture above. For AR training, the clean context blocks are written into <bK_<b and the target block is optimized with the flow-matching objective ℒAR=b,t[‖v^bt−(εb−Zb)‖22],L_AR=E_b,t [ \| v_b^t-( _b-Z_b) \|_2^2 ], (19) which exposes the model to the same causal context used at inference. At inference, each chunk is sampled from noise, committed to the cache, and used to condition the next chunk: Z^b=FlowSample(b,<b),<b+1=<b⊕Z^b. Z_b=FlowSample(c_b,K_<b), _<b+1=K_<b Z_b. (20) The initial frame initializes ℳaM_a, the recent memory ℳrM_r starts empty, and Eq. (12) refreshes it at the configured stride. The final rollout is decoded after all chunks are committed. AR Generation with OAPM and A3D-RoPE The complete blockwise generation procedure is given in Algorithm 1. Algorithm 1 AR Generation with OAPM and A3D-RoPE 1:Initial frame I0I_0, prompt y, block conditions b,Sb,bb=1B\C_b,S_b,X_b\_b=1^B, refresh stride srs_r. 2:Generated rollout V V. 3:ℳa←E3D(I0)M_a← E_3D(I_0); ℳr1←∅M_r^1← ; <1←∅K_<1← 4:⊳ Stage 1: Condition initialization 5:for autoregressive block b=1,…,Bb=1,…,B 6: Zb1∼(0,)Z_b^1 (0,I) 7: b←SceneEncodeΩ(ℳa⊕ℳrb)M_b _ (M_a _r^b) 8: ℐb←SkeletonSupport(Sb)I_b (S_b) 9:⊳ Stage 2: Blockwise flow integration 10: for flow time t from 11 to 0 11: H←PatchEmbed(Zbt)+TimeEmbed(t)H (Z_b^t)+TimeEmbed(t) 12: for each DiT layer 13: H←DiTLayer(H;y,b,<b)H (H;y,C_b,K_<b) 14: if the layer contains an OAPM adapter 15: H←H+GatedCrossAttnOAPM(H,b)H← H+GatedCrossAttn_OAPM(H,M_b) 16: end if 17: if the layer contains an A3D-RoPE adapter 18: Hℐb←Hℐb+GatedCrossAttnA3D(Hℐb,Sb,ℐb,b)H_I_b← H_I_b+GatedCrossAttn_A3D(H_I_b,S_b,I_b,X_b) 19: end if 20: end for 21: v^bt←Head(H) v_b^t (H); Zbt−Δt←Zbt−Δtv^btZ_b^t- t← Z_b^t- t\, v_b^t 22: end for 23: Zb←Zb0Z_b← Z_b^0; append ZbZ_b to <b+1K_<b+1 24:⊳ Stage 3: Causal commit and memory refresh 25: ℳrb+1←ℳrbM_r^b+1 _r^b 26: if bmodsr=0b s_r=0 27: I^brec←RecentFrame(Dvae(Z≤b)) I_b^rec (D_vae(Z_≤ b)) 28: ℳrb+1←E3D(I^brec)M_r^b+1← E_3D( I_b^rec) 29: end if 30:end for 31:return V^←Dvae([Z1,…,ZB]) V← D_vae([Z_1,…,Z_B]) Appendix C Complete Component Ablation The full ablation reports action alignment, physical faithfulness, and temporal consistency in addition to the compact visual-fidelity columns retained in the main paper. Component Setting PSNR↑ SSIM↑ LPIPS↓ Kpt.Err↓ Phys.Faith↑ Subj. Cons.↑ Bg. Cons.↑ Wan2.2-5B-Control-AR 19.9238 0.7812 0.3028 0.07723 0.7796 0.8337 0.9316 Scene memory (A3D-RoPE fixed) First-frame anchor only 20.4135 0.8385 0.2533 0.0532 0.8037 0.8847 0.9532 ++ Recent refresh 21.8609 0.8509 0.2399 0.0501 0.8278 0.8923 0.9546 Positional encoding (OAPM fixed) RoPE 21.4250 0.8198 0.2838 0.07719 0.8182 0.8837 0.9353 PRoPE 21.8421 0.8408 0.2481 0.06135 0.8255 0.8919 0.9481 A3D-RoPE 21.8609 0.8509 0.2399 0.0501 0.8278 0.8923 0.9546 Table 4: Core-component ablations on Wan2.2-5B-Control. Within each block, the complementary component is fixed to its full configuration. Appendix D Detailed Downstream Real-Robot Results The complete execution sequences in Figure 10 provide a task-level view of the real-robot evaluation summarized in main-paper Table 3. Each row contains five checkpoints spanning initialization, approach, contact, intermediate transition, and the terminal state. Bimanual success requires a stable towel fold, placing the pen-like object before pushing its receiving cup, placing both objects into the bin, or completing the bottle handoff. Single-arm success requires a stable two-cube stack, placing the cube inside the bin, selecting the instructed target without disturbing distractors, or pushing the cube fully into the marked region. Figure 10: Detailed execution sequences for four bimanual and four single-arm real-robot tasks. Five checkpoints per row show the progression from the initial configuration to the task-specific success state. Main-paper Table 3 reports aggregate success rates for the single-arm and dual-arm suites, while Supplementary Table 6 provides the corresponding per-task results under the same policy architecture and training schedule. We compare 400 real trajectories, 400 synthetic trajectories, and their combination. ID trials retain training-time object appearances and layouts, whereas OOD trials hold out object appearances and initial or goal layouts. Each task is evaluated over 25 trials, so its success rate changes in 4-point increments; each aggregate is the exact mean over four tasks, or equivalently the success rate over 100 trials. Tianji M6 Platform, Tasks, and Evaluation Protocol All real-robot trials use the Tianji M6 platform shown in Figure 11. Its two gripper-equipped arms operate over an overlapping tabletop workspace, while the head-mounted camera provides the egocentric observation used by the policy. Single-arm trials activate one arm and keep the other outside the task workspace; bimanual trials coordinate both arms. The robot, camera mounting, and workspace remain fixed across training-data settings and ID/OOD evaluation, so the comparison isolates changes in object appearance, initial configuration, and target layout. Figure 11: Tianji M6 real-robot environment from front, side, and back views. The two gripper-equipped arms share a tabletop workspace observed by the head-mounted egocentric camera. The single-arm suite includes Cube Stacking into a stable tower, Pick and Place into a bin, Pointing and Select without disturbing distractors, and Cube Pushing into a taped region. The dual-arm suite includes Towel Folding with one arm anchoring the cloth, Pick and Place of two objects into a central bin, Bottle Handoff followed by upright placement, and Place and Push of a pen-like object and its receiving cup. Task Stages and Completion Criteria Each task is divided into ordered interaction stages to make the execution progress and terminal success criterion explicit. The percentages below indicate normalized trajectory progress rather than success rates; Done denotes the task-specific terminal state at 100%. Task Stage Progress Completion criterion Towel Folding S1: Approach and grasp 25% Both grippers approach the towel, stabilize one side, and grasp the side to be folded. S2: Lift and fold 62% The right side of the towel is lifted and folded toward the left. Done: Align and flatten 100% The folded edge is adjusted and flattened to form a stable, aligned fold. Bimanual Pick and Place S1: Grasp both cubes 28% The left and right grippers approach and grasp the cubes on their respective sides. S2: Place the blue cube 58% The blue cube is transported into the central bin. S3: Transport the red cube 80% The red cube is lifted and moved above the bin. Done: Place the red cube 100% The red cube is placed in the bin so that both cubes reach their target location. Bottle Handoff S1: Grasp the bottle 22% The delivering gripper grasps the horizontally placed bottle. S2: Lift and rotate 48% The bottle is lifted from the table and rotated toward an upright pose. S3: Transfer control 70% The receiving gripper secures the bottle and completes the handoff. Done: Place upright 100% The receiving gripper places the bottle upright at the target location. Place and Push S1: Grasp the pen and cup 23% The two grippers take control of the pen and cup, respectively. S2: Lift and align 52% The pen is moved above the cup opening while the other gripper stabilizes the cup. S3: Insert the pen 72% The pen is inserted into the cup and reaches a stable state. Done: Push the cup 100% The cup containing the pen is pushed to the target location. Cube Stacking S1: Approach and grasp 24% The gripper approaches and grasps the yellow cube. S2: Lift and transport 56% The yellow cube is moved above the blue cube. Done: Align and stack 100% The cubes are aligned to form a stable two-level stack. Single-Arm Pick and Place S1: Approach and grasp 26% The gripper approaches and grasps the cube outside the bin. S2: Lift and transport 58% The cube is moved above the bin and aligned with its opening. Done: Place in the bin 100% The cube is released into the bin and reaches the target state. Point and Select S1: Approach the target 30% The gripper moves toward the specified red target without contacting the other cubes. S2: Align the pointing pose 68% The pointing direction and position are aligned above the target. Done: Point and hold 100% A clear and stable pointing pose is maintained near the red target. Cube Pushing S1: Approach and contact 30% The gripper approaches the red cube and establishes pushing contact. S2: Push toward the target 72% Continuous contact moves the cube toward the black marked region. Done: Stabilize in the target 100% The cube lies fully within the target region and remains stable. Table 5: Stage-wise execution protocol for the eight downstream real-robot tasks. (a) Single-arm tasks on Tianji M6 Training data Cube Stacking Pick&Place Pointing & Select Cube Pushing Avg. SR↑ In-distribution (ID) 400 real 76 84 92 84 84.0 400 real + 400 synth. 80 88 96 88 88.0 400 synth. 72 84 96 84 84.0 Out-of-distribution (OOD) 400 real 56 80 92 80 77.0 400 real + 400 synth. 72 84 96 84 84.0 400 synth. 52 80 92 80 76.0 (b) Dual-arm tasks on Tianji M6 Training data Towel Folding Pick&Place Bottle Handoff Place & Push Avg. SR↑ In-distribution (ID) 400 real 72 76 68 72 72.0 400 real + 400 synth. 76 80 72 76 76.0 400 synth. 68 72 68 68 69.0 Out-of-distribution (OOD) 400 real 52 56 52 52 53.0 400 real + 400 synth. 68 80 68 64 70.0 400 synth. 52 64 56 52 56.0 Table 6: Per-task real-robot success rates (%), with 25 trials per task. Panels (a) and (b) exactly decompose the Single-arm SR and Dual-arm SR columns in main-paper Table 3; bold denotes the best training budget within each split. Downstream WAM Training and Inference Details For each training-data setting, we independently initialize LingBot-VA from its official released checkpoint and fine-tune it on the corresponding real, synthetic, or mixed trajectory set. The downstream WAM therefore does not inherit weights from our trained Wan2.2-5B-Control generator. We keep the model architecture, optimization configuration, and inference settings identical across data settings, changing only the composition of the fine-tuning trajectories. Each sample contains three synchronized RGB observations from the head-mounted camera, the left-wrist camera, and the right-wrist camera. All observations are resized to 256×256256× 256. The temporal attention window is set to 30, and the video frame chunk size is set to 2. The model maintains a canonical 30-dimensional action space. For the bimanual joint-control experiments, we select 16 active channels corresponding to seven left-arm joint dimensions, the left gripper, seven right-arm joint dimensions, and the right gripper. Using zero-based indexing, the selected channels are ordered as [14:20, 28, 21:27, 29].[14:20,\;28,\;21:27,\;29]. (21) The end-effector channels are therefore not used in these experiments. The action-per-frame factor is set to 16. Each active action dimension is normalized using its 1st and 99th percentiles computed from the corresponding training data. We optimize the model using AdamW with a learning rate of 1×10−51× 10^-5, coefficients (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), and weight decay of 0.10.1. The first 200 training steps are used for learning-rate warmup. The per-GPU batch size is 1, and gradients are accumulated over four iterations. LingBot-VA training and testing, including downstream real-robot inference, use NVIDIA H100 GPUs; training is distributed over eight H100 GPUs, resulting in an effective global batch size of 32. Each model is trained for 20,000 steps. During training, the language-condition embedding is replaced by the empty-text embedding with probability 0.1 for classifier-free guidance training. Checkpoints are saved every 1,000 steps. A complete training run takes approximately 96 hours on eight H100 GPUs. During inference, the classifier-free guidance scales are set to 5 for the video branch and 1 for the action branch. We use 5 denoising steps for video generation and 10 denoising steps for action generation. The video and action SNR-shift parameters are set to 5.0 and 1.0, respectively. The video denoising process is not truncated. Table 7 summarizes the complete configuration. Table 7: Training and inference configuration of the downstream WAM. The same configuration is used for the real, synthetic, and mixed training-data settings. Hyperparameter Setting Pretrained initialization Official LingBot-VA release checkpoint; independent initialization for each data setting Observation cameras Head, left wrist, and right wrist Input resolution 256×256256× 256 Temporal attention window 30 Video frame chunk size 2 Canonical action dimension 30 Active action dimension 16 Active action channels [14:20, 28, 21:27, 29][14:20,\,28,\,21:27,\,29] Action-per-frame factor 16 Action normalization Per-dimension 1st/99th percentiles Optimizer AdamW Learning rate 1×10−51× 10^-5 Adam coefficients (0.9,0.95)(0.9,0.95) Weight decay 0.10.1 Warmup steps 200 Per-GPU batch size 1 Gradient accumulation steps 4 Training hardware 8 NVIDIA H100 GPUs Effective global batch size 32 Training steps 20,000 Text-condition dropout probability 0.1 Checkpoint interval 1,000 steps Training time Approximately 96 hours Video CFG scale 5 Action CFG scale 1 Video denoising steps 5 Action denoising steps 10 Video SNR shift 5.0 Action SNR shift 1.0 Video denoising truncation None Figure 12: Additional comparisons on egg transfer and cup-lid removal; EgoGenesis better preserves scene and hand identity while following the commanded contact sequence. Appendix E Additional Qualitative Comparisons Figure 12 extends the main-paper comparison to egg transfer and cup-lid removal. In the egg task, EgoGenesis preserves the tabletop objects and follows the hand from the teacup toward the bowl, whereas several baselines alter the setup, drift in hand/object geometry, or under-execute the transfer. In the lid-removal task, EgoGenesis retains the cup arrangement and red-sleeved hand identity while producing successive contact-driven changes; competing generations more often merge hands with lids, deform cups, or show limited task progression. Cross-Embodiment Simulation in an Unseen Environment Figure 13 shows that EgoGenesis can simulate manipulation in an environment not observed during training while preserving the specified action. Starting from the same unseen scene and instruction, the model generates both human-hand and robot-gripper executions. The gripper control skeleton in the lower example is obtained by extracting the index-finger and thumb trajectories from the full hand skeleton in the upper example, providing a compact action condition for cross-embodiment simulation. Appendix F Evaluation Metrics and Kimi K2.7 Prompt Main-Table Metrics All methods are evaluated on temporally aligned generated and reference clips under the same scene and action conditions. Frame-level values are first averaged within each sample; table entries are arithmetic means over non-empty per-sample values. The seven metrics reported in the main results and the supplementary component ablation are defined as follows. Metric Definition PSNR↑ Peak Signal-to-Noise Ratio between aligned generated and reference RGB frames in [0,1][0,1], averaged over time. SSIM↑ Structural Similarity Index between aligned frames, computed with an 11×1111× 11 Gaussian window and averaged over time. LPIPS↓ Learned Perceptual Image Patch Similarity using the AlexNet backbone, averaged over aligned frame pairs. Kpt.Err↓ Hand Keypoint End-Point Error: evaluated on 50 clips selected from EgoDex. We compute the mean 2D Euclidean error between matched generated and reference hand keypoints, normalized by the image diagonal. We use ground-truth 2D joints when available; otherwise, the same MediaPipe 21-keypoint detector is applied to both generated and reference frames, followed by centroid-based hand matching. Phys.Faith↑ Physical faithfulness, measured by the normalized single-axis score from Kimi K2.7 as defined below. It judges whether grasps, support, pushes, contact-driven object motion, non-penetration, and gravity are physically credible. Subj. Cons.↑ Subject Consistency: VBench-style temporal consistency of DINO ViT-S/16 frame features, averaging cosine similarity to both the first and previous frames. Bg. Cons.↑ Background Consistency: VBench-style temporal consistency of CLIP ViT-B/32 image features, using the same first-frame and adjacent-frame cosine-similarity aggregation. Figure 13: Cross-embodiment simulation in an unseen environment. Given the same initial scene and instruction, EgoGenesis follows a full hand skeleton to generate human-hand manipulation (top) and a compact gripper skeleton to generate robot-gripper manipulation (bottom). The gripper skeleton is extracted from the index-finger and thumb trajectories of the hand skeleton. Kimi K2.7 Prompt for Physical Faithfulness We use Kimi K2.7 as the judge. Frames are sampled uniformly from each generated video and presented in temporal order. The judge evaluates only whether contact and object dynamics are physically credible for the depicted manipulator type. Prompt: Physical Faithfulness You are evaluating a generated video of manipulator–object interaction for the task “[TASK PROMPT].” You are shown k frames sampled uniformly in temporal order. Treat a human hand, dexterous hand, gripper, or robot end effector according to its own embodiment. Evaluate physical faithfulness considering whether: 1. contacts, grasps, support, and pushes are credible; 2. object motion is caused by plausible manipulator contact; 3. objects avoid penetration, floating, and violations of gravity; 4. contact and object dynamics remain coherent over time. Rate from 0–5, respond with only an integer. For the returned score sphys∈0,…,5s_phys∈\0,…,5\, we report Phys.Faith=sphys5.Phys.Faith= s_phys5. (22)