Paper deep dive

GameWAM: A World Action Model for Video Games

Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Weijia Li

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 152

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/28/2026, 3:19:57 AM

Summary

The paper introduces GameWAM, the first World-Action Model (WAM) designed for native closed-loop gameplay and GUI control in video games. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories using parallel Video and Action Diffusion Transformers (DiTs) with block-causal conditioning and flow matching. It addresses heterogeneous controls by predicting gameplay/GUI modes and uses a block-cycle control strategy to decouple long-horizon prediction from short-horizon execution, preserving temporal continuity via hierarchical history. Experiments on Minecraft and ViZDoom show competitive task success with fewer native actions. The authors also identify a failure mode called Low-Frequency Action Source Imprinting (LASI), where low-frequency components of the action source bias camera motion.

Entities (9)

GameWAM · model · 100%Minecraft · environment · 95%VizDoom · environment · 95%World Action Model · concept · 95%Action DiT · component · 90%Block-Cycle Control · method · 90%Flow Matching · method · 90%Low-Frequency Action Source Imprinting · phenomenon · 90%Video DiT · component · 90%

Relation Signals (8)

GameWAM → evaluatedon → Minecraft

confidence 95% · Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance

GameWAM → evaluatedon → VizDoom

confidence 95% · Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance

GameWAM → isa → World Action Model

confidence 95% · We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control.

GameWAM → exhibits → Low-Frequency Action Source Imprinting

confidence 90% · We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion

GameWAM → uses → Flow Matching

confidence 90% · GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching.

GameWAM → uses → Action DiT

confidence 90% · GameWAM models their joint visual and control dynamics with parallel Video and Action DiTs

GameWAM → uses → Video DiT

confidence 90% · GameWAM models their joint visual and control dynamics with parallel Video and Action DiTs

GameWAM → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

149,089 characters extracted from source content.

Expand or collapse full text

GameWAM: A World Action Model for Video Games Yuncheng Guo Affiliation: Fudan University Zhanqiu Zhang Affiliation: LIGHTSPEED Yiwen Guo Affiliation: Independent Researcher Weijia Li Affiliation: Tsinghua Shenzhen International Graduate School†Corresponding authors. Abstract Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World–Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard–mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world–action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode at each action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control predicts beyond the committed horizon, executes only a short action prefix, and replans from new observations, while fine-grained within-cycle context and hierarchical cross-cycle history preserve temporal continuity. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/. 1 Introduction Modern video games demand pixel-based native control over rapidly changing first-person observations, keyboard–mouse inputs, and the consequences of prior actions. Existing solutions span behavioral pretraining, vision-language-action models, hierarchical agent architectures, and large-scale interaction learning (Baker et al., 2022; Lifshitz et al., 2023; Wang et al., 2024a; Wang et al., 2024b; Wang et al., 2026b; Wang et al., 2025). Despite different system designs, their control ultimately maps visual and task context to native or abstract actions, often through discretized, semantic, or temporally compressed representations. While effective for behavior learning, these objectives do not explicitly model how the visual world evolves under executed actions, and their action abstractions may weaken the fine temporal and metric structure needed for concurrent key control and continuous camera motion. Interactive game world models provide the complementary capability: they learn action-conditioned visual dynamics, but rely on actions supplied by a player or external controller rather than selecting task-directed behavior themselves (Bruce et al., 2024; Alonso et al., 2024; Che et al., 2025; Guo et al., 2025; Zhang et al., 2025). Control models therefore choose behavior without explicitly modeling its visual consequences, while interactive world models predict those consequences without serving as task policies. This motivates a unified model of task-directed actions and their visual consequences. Figure 1: Overview of GameWAM. GameWAM jointly models future visual observations and native actions for closed-loop gameplay and GUI control. World–Action Models (WAMs) jointly generate future visual observations and executable action trajectories, using visual prediction to provide dynamics-aware supervision for control. Recent WAMs have shown promising results mainly in tabletop or bounded indoor robot manipulation (Ye et al., 2026; Yuan et al., 2026), while their behavior under rapidly changing first-person observations, persistent world state, and repeated closed-loop interaction remains less understood. Modern video games provide a scalable and controllable testbed for this regime: ego-motion, dynamic entities, heterogeneous interaction modes, and persistent state create diverse and abrupt visual changes, while environments remain precisely instrumented, reliably reset, and repeatable under controlled interventions (Fan et al., 2022; Raad et al., 2024). Although games do not reproduce every aspect of physical embodiment, they expose control models to rapidly evolving interaction at a scale difficult to obtain in current physical testbeds (Khazatsky et al., 2024; Pumacay et al., 2024). We therefore introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard–mouse trajectories from visual, instruction, and other condition signals. Realizing a WAM in games introduces challenges along action and temporal dimensions. The action challenge is particularly prominent in native game control: gameplay combines concurrent discrete keys, continuous camera motion, and sparse mouse events, while GUI interaction reuses the same physical channels for cursor motion, clicks, scrolling, and interface shortcuts. Thus, identical action dimensions can have different semantics, scales, and conditional distributions across regimes, and treating them as one undifferentiated distribution can mix incompatible camera and cursor statistics. GameWAM generates continuous and concurrent discrete dimensions within a shared Action-DiT flow, while a learned per-action router selects between gameplay- and GUI-specific action-flow predictions at each action timestep. The two regimes use separate normalization for their distinct control distributions while retaining the same underlying physical controls. The temporal challenge is amplified in rapidly changing interactive environments by visual subsampling relative to native actions. Recent embodied WAMs use temporal subsampling to control video-token cost, but coarse temporal resolution can miss rapid ego-motion, transient targets, and brief interaction events in games. Denser sampling captures these changes, yet under fixed token and cache budgets the same resources cover less interaction time, shortening both future look-ahead and retained past context. This trade-off especially affects chunk-wise autoregressive WAMs like DreamZero (Ye et al., 2026), where prediction, execution, and transient KV accumulation proceed along a finite block chain. On the future side, denser sampling shortens the real-time horizon covered by a fixed chain. Extending the chain to recover longer look-ahead can also lengthen the committed action span and delay feedback from updated observations. GameWAM instead decouples prediction from commitment: each planning unit predicts P actions but commits only an E-step execution block, with E<PE<P, before observing and replanning. This predict-long/execute-short organization provides longer action look-ahead while preserving frequent closed-loop feedback. During training, overlapping P-step plans are anchored every E steps, extending supervision beyond each execution block while reusing the same E-spaced temporal organization rather than lengthening the autoregressive chain. On the past side, denser sampling similarly shortens the interaction span represented by a fixed-size KV cache. Retaining more blocks increases memory, while the transient cache is reset when the finite chain ends, leaving subsequent chains without persistent access to earlier executed observations. GameWAM therefore combines a bounded within-cycle KV cache with persistent hierarchical visual history across cycle boundaries: recent executed observations remain explicitly represented, while older information is compressed into long-term history. This preserves fine-grained current-cycle context alongside persistent multi-cycle conditioning under bounded memory. To support WAM training in this setting, we construct synchronized, WAM-ready gameplay and GUI trajectories. GameWAM models their joint visual and control dynamics with parallel Video and Action DiTs, block-causal attention, and joint flow matching, while event-anchored sampling emphasizes event-rich segments. Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance; in Minecraft, it does so with fewer executed native actions than the compared agents across all evaluated task categories. Our closed-loop evaluation further reveals an unexpected failure mode: when the same sampled action source is reused across replanning steps, some source realizations induce a persistent directional camera bias, including repeated in-place rotation. Controlled frequency-domain interventions reveal that low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning. We term this phenomenon Low-Frequency Action Source Imprinting (LASI). LASI distinguishes ordinary source-dependent sampling from a pathological low-frequency control bias that can accumulate through repeated closed-loop interaction. Our contributions are: • We introduce GameWAM, to our knowledge the first World–Action Model for native closed-loop gameplay and GUI control. Built on parallel Video and Action DiTs, block-causal attention, and joint flow matching, GameWAM jointly generates future visual observations and executable keyboard–mouse action trajectories. • We construct synchronized gameplay and GUI trajectories for joint world–action learning, combining standardized gameplay data with generated interface interactions. • We develop a unified native-action and block–cycle formulation for heterogeneous, long-horizon game interaction. GameWAM supports gameplay and GUI control in a shared keyboard–mouse action space, while block–cycle control decouples long-horizon prediction from short-horizon execution and combines fine-grained within-cycle context with hierarchical cross-cycle history. Experiments demonstrate competitive closed-loop performance with fewer executed native actions than the compared agents. • We uncover Low-Frequency Action Source Imprinting (LASI), a source-sensitivity failure mode in which low-frequency components of the sampled action source steer coarse generated camera motion under fixed conditioning. Reuse across replanning steps can accumulate into persistent directional bias in closed-loop interaction. Figure 2: GameWAM architecture for training and inference. Parallel Video and Action DiTs jointly model future observations and native actions, with block-causal interaction and gameplay/GUI-specific action generation. 2 Related Work 2.1 Game Agents and Game World Models Video games provide scalable settings for native control and long-horizon interaction. Earlier work established large-scale game environments, demonstrations, behavioral learning, and open-ended evaluation (Guss et al., 2019; Fan et al., 2022; Baker et al., 2022; Lifshitz et al., 2023; Cai et al., 2024; Zheng et al., 2025; Kempka et al., 2016; Petrenko et al., 2020). Subsequent game agents increasingly emphasize multimodal reasoning, memory, hierarchical control, and generalist pretraining (Wang et al., 2023; Wang et al., 2024a; Wang et al., 2024b; Raad et al., 2024). Recent work more directly addresses action representation and scalable native control. OpenHA (Wang et al., 2026b) systematically studies action abstractions and introduces Chain of Action, using abstract actions as intermediate reasoning steps before executable control. Game-TARS (Wang et al., 2025) scales a unified native keyboard–mouse action space through large-scale continual pretraining across heterogeneous game and computer-use domains, while Lumine (Tan et al., 2025) unifies perception, reasoning, and real-time keyboard–mouse control for long-horizon 3D open-world interaction. Game world models learn controllable visual dynamics of interactive environments. This line spans latent-action generation, diffusion world modeling, real-time neural game simulation, and increasingly open-world action-conditioned generation (Bruce et al., 2024; Alonso et al., 2024; Valevski et al., 2025; Che et al., 2025; Guo et al., 2025; Zhang et al., 2025). WHAM (Kanervisto et al., 2025) further models gameplay video together with human controller actions. These developments build on broader advances in video diffusion and token-based video generation (Ho et al., 2022; Voleti et al., 2022; Harvey et al., 2022; Villegas et al., 2022; Yu et al., 2023). 2.2 World–Action Models World–Action Models connect predictive world modeling with generative control. Learned world models have progressed from latent planning and imagination to scalable transformer and continuous-control formulations (Hafner et al., 2019b; Hafner et al., 2019a; Hafner et al., 2020; Schrittwieser et al., 2020; Micheli et al., 2023; Zhang et al., 2023; Hansen et al., 2024; Hafner et al., 2025). Generative control has similarly developed structured action prediction through chunking, diffusion, latent actions, and large-scale vision–language–action learning (Zhao et al., 2023; Chi et al., 2023; Lee et al., 2024; Brohan et al., 2023b; Brohan et al., 2023a; O’Neill et al., 2024; Ghosh et al., 2024), while video- and latent-action methods connect predicted trajectories to control (Du et al., 2023; Du et al., 2024; Wu et al., 2024; Ye et al., 2025). Diffusion Transformers and flow-based objectives provide scalable generative backbones (Peebles & Xie, 2023; Ma et al., 2024; Lipman et al., 2023). Related world–action formulations have also been explored in offline model-based RL (Cheng et al., 2025). More recent generative WAMs directly couple visual prediction with executable action generation and increasingly revisit inference-time execution. DreamZero (Ye et al., 2026) jointly predicts future video and actions for closed-loop control, Fast-WAM (Yuan et al., 2026) retains video co-training while removing future-video generation at test time, and FFDC-WAM (Wang et al., 2026a) adapts execution by verifying predicted futures against realized observations. GameWAM studies this emerging paradigm in native closed-loop gameplay and GUI control. 3 Data Construction and Training Sampling Joint world–action learning requires aligned observations and native controls. For Minecraft, we construct three complementary WAM-ready training streams. Regular VPT trajectories (Baker et al., 2022) provide broad naturalistic interaction coverage and long temporal context. From the same recordings, we additionally construct an Event-Anchored VPT dataset by identifying MineStudio-style interaction events (Cai et al., 2024) from state transitions and forming instruction-aligned event-centered trajectories. We further generate MineStudio-based scripted GUI trajectories to broaden interface-interaction coverage. All streams are standardized on a common interaction timeline into synchronized observations, state, and native keyboard–mouse actions with a control-mode label. Dataset construction details are provided in Appendix B. Event-anchored training clip sampling. Dataset construction determines the event-centered trajectories available for training, whereas clip sampling separately determines their temporal sampling density. Long interaction trajectories contain highly uneven supervision density: transitions near an annotated event often contain the visual and action changes most relevant to the instruction, while distant portions may consist largely of traversal, incidental camera motion, or other weakly related behavior. Uniform sampling would therefore devote substantial training capacity to less informative regions. Within the Event-Anchored VPT stream, we instead sample training clips densely around the event anchor and more sparsely away from it, increasing exposure to event-relevant transitions while retaining surrounding interaction context. The sampling configuration is in Appendix C. 4 Method GameWAM is a block-causal world–action model for native gameplay and GUI control. Figs. 1, 2 and 3 summarize the model and rollout. 4.1 Block-Causal World–Action Modeling At block k of cycle c, define the conditioning context as Γc,k=(Cc,k,ℓ,Hc,sc,k), _c,k=(C_c,k, ,H_c,s_c,k), (1) where Cc,kC_c,k denotes the clean within-cycle observation context, ℓ the language or task instruction, HcH_c the persistent cross-cycle visual history, and sc,ks_c,k the optional proprioceptive state. The complete clean visual prefix used by block-causal attention is c,k=Hc∪Cc,k.P_c,k=H_c∪ C_c,k. (2) A world–action block is Bj=(j,j)B_j=(V_j,A_j), with video latents jV_j and native actions j=[jcont;jdisc]A_j=[A_j^cont;A_j^disc]. For a plan spanning R blocks, GameWAM models the block-causal joint distribution pθ(Bk:k+R−1∣Γc,k)=∏r=0R−1pθ(Bk+r∣Γc,k,Bk:k+r−1).p_θ\! (B_k:k+R-1 _c,k )= _r=0^R-1p_θ\! (B_k+r _c,k,B_k:k+r-1 ). (3) The attention mask determines how the two modalities exchange information within each conditional factor. We parameterize visual and action processes with joint flow matching (Lipman et al., 2023) using parallel Video- and Action-DiT branches (Peebles & Xie, 2023; Ma et al., 2024). Continuous and discrete action coordinates remain components of one native action vector; discrete ones are converted to executable binary decisions only after generation. For modality m∈v,am∈\v,a\, Xσm=(1−σm)X0m+σmϵm,Um=ϵm−X0m.X_ _m^m=(1- _m)X_0^m+ _mε^m, U^m=ε^m-X_0^m. (4) The parallel DiTs estimate (U^θv,U^θa)( U_θ^v, U_θ^a) from the noisy modalities, their noise levels, and the causal context. At inference, X1m∼⁡(0,I)X_1^m (0,I) and the learned ODE is integrated from σ=1σ=1 to 00. Block-causal video–action masking. For target block j, let c,jP_c,j denote its causally available clean visual prefix, and let jV_j and jA_j denote the noisy video and action variables of that block. We adopt a Fast-WAM-style (Yuan et al., 2026) modality-decoupled mask, under which both modalities share the same clean prefix while their noisy variables do not condition one another: Vis⁡(j)=c,j∪j,Vis⁡(j)=c,j∪j.Vis(A_j)=P_c,j _j, (V_j)=P_c,j _j. (5) The Video DiT encodes c,jP_c,j into layer-wise K/V states shared by both branches, so video and action supervision jointly shape the context used for action generation. Instruction and proprioception remain available through their corresponding conditioning paths. This preserves video co-training while avoiding future-video denoising during action-only online inference. We compare this default design with joint within-block video–action attention in Section D.2. Per-action routing. Gameplay and GUI interaction share the same physical keyboard–mouse coordinates but induce different conditional action distributions. GameWAM therefore uses a timestep-wise router with gameplay- and GUI-specific prediction branches. Let rτ∗∈0,1r_τ^*∈\0,1\ denote the observed interaction mode and ρτ _τ the routing logit. At rollout, the predicted route r^τ r_τ selects the action-flow prediction for the entire native action vector at timestep τ: r^τ=[sigmoid(ρτ)>12],U^τa=(1−r^τ)U^τgame+r^τU^τgui. r_τ=1\! [sigmoid( _τ)> 12 ], U_τ^a=(1- r_τ) U_τ^game+ r_τ U_τ^gui. (6) During training, rτ∗r_τ^* selects the supervised prediction branch and also provides the routing target; at rollout, the predicted route is used. Continuous action coordinates are normalized with gameplay- or GUI-specific statistics, whereas discrete coordinates retain the shared normalization. Validity masks exclude controls unavailable in a given environment. Figure 3: Block–cycle control and hierarchical history. GameWAM predicts beyond the committed horizon while executing only a short prefix, and combines a transient cycle-local K/V cache with persistent cross-cycle visual history. 4.2 Block-Cycle Control and Hierarchical History As illustrated in Fig. 3, GameWAM separates prediction from commitment. A plan contains P actions, whereas only its first E actions form the committed execution block: ^c,kplan∈ℝP×da,E<P,^c,kexec=^c,kplan[1:E]. A_c,k^plan ^P× d_a, E<P, A_c,k^exec= A_c,k^plan[1:E]. (7) The plan spans R=⌈P/E⌉R= P/E execution intervals. After executing one block, the agent receives a new observation and replans the overlapping P−EP-E suffix. Only observations realized through executed interaction become clean causal context; unexecuted lookahead is discarded. During training, overlapping P-step plans anchored every E steps are formed from the same trajectory and teacher-forced in parallel under block-causal visibility, avoiding sequential rollout of the additional look-ahead. GameWAM separates within-cycle realized context from persistent cross-cycle history. Online, both form the clean prefix and its transient K/V cache; persistent history is summarized by recent and long-term visual memory. Cycle KV cache. During online rollout, c,kP_c,k is materialized as layer-wise K/V states. At cycle start, the cache is initialized from cross-cycle history and current context; newly realized observations then extend only its current-cycle portion. The cache is discarded at the cycle boundary and reinitialized from c+1,0P_c+1,0; persistent information survives through Hc+1H_c+1. Recent history. At a cycle boundary, the realized visual segment produced by executed actions is encoded by the video VAE and compressed by a Conv3D tokenizer: Sc=Conv3D⁡(VAE⁡(Ocexec)),(Rc+1,S¯c)=FIFOKR⁡(Rc,Sc).S_c=Conv3D\! (VAE(O_c^exec) ), (R_c+1, S_c)=FIFO_K_R(R_c,S_c). (8) The buffer RcR_c retains the latest KRK_R segments; S¯c S_c denotes the evicted segment when the buffer is full. Recent history therefore remains explicitly addressable at a compressed spatial resolution. Long-term history. An evicted recent segment updates a fixed set of long-term memory slots through attention pooling and a learned slot-wise gate: M~c=Attn⁡(Mc,S¯c,S¯c),Mc+1=LN⁡[Mc+gc⊙(M~c−Mc)]. M_c=Attn(M_c, S_c, S_c), M_c+1=LN\! [M_c+g_c ( M_c-M_c) ]. (9) If no segment is evicted, Mc+1=McM_c+1=M_c. Learned slot and level embeddings assign distinct temporal roles to different memory slots. The resulting persistent history Hc=[Mc;Rc]H_c=[M_c;R_c] is retained across interaction-cycle boundaries, unlike the transient cycle-local K/V state. 4.3 Training Objective Training teacher-forces completed blocks with rollout block-causal visibility. The video flow loss is ℒv=⁡[wv(σv)∥U^v−Uv∥22].L_v=E\! [w_v( _v)\, U^v-U^v _2^2 ]. (10) For the selected interaction branch, the Action DiT jointly predicts the complete native action vector. To prevent either continuous or discrete coordinates from dominating, they are reduced separately. For g∈cont,discg∈\cont,disc\, ℒag=[wa(σa)∑τ,d∈gmτd(U^τda−Uτda)2∑τ,d∈gmτd],g∈cont,disc;ℒa=λcontℒacont+λdiscℒadisc. gatheredL_a^g=E\! [w_a( _a) _τ,d _gm_τ d( U_τ d^a-U_τ d^a)^2 _τ,d _gm_τ d ], g∈\cont,disc\;\\ L_a= _contL_a^cont+ _discL_a^disc. gathered (11) Here mτdm_τ d masks padded or unavailable action coordinates, and U^a U^a is selected by the observed interaction mode during training. Routing is supervised via masked binary cross-entropy, ℒmode=∑τντBCE⁡(ρτ,rτ∗)∑τντ,L_mode= _τ _τ\,BCE( _τ,r_τ^*) _τ _τ, (12) where ντ _τ indicates whether the interaction-mode label is valid. To keep compressed history informative, we predict the current clean visual feature qcq_c from persistent history: ℒhist=Dpred(F⁡(Pool⁡(Hc)),sg⁡(qc)),L_hist=D_pred\! (F(Pool(H_c)),sg(q_c) ), (13) where DpredD_pred combines cosine and squared-error distances. This regularizes HcH_c to retain cross-cycle predictive visual information, while stop-gradient keeps the target fixed. ℒ=λvℒv+λaℒa+λmℒmode+λhℒhist.L= _vL_v+ _aL_a+ _mL_mode+ _hL_hist. (14) 5 Experiments 5.1 Evaluation Setup We evaluate GameWAM on two complementary closed-loop benchmarks: Minecraft Universe (MCU) and the four-map ViZDoom suite. MCU evaluates long-horizon task completion under native control using task success and executed environment actions, jointly measuring control effectiveness and interaction efficiency. ViZDoom adds fast visual dynamics and continuous interaction, evaluated by average episode reward. We compare with published baselines under their reported evaluation protocols. Further details are provided in Appendix C. 5.2 Main Results on MCU Table 1: Evaluation results of Minecraft agents on the MCU benchmark with over 800 tasks. For each category, we report average native interaction steps over successful episodes, ASR on the Mini subset (ASR Mini), and ASR over all tasks (ASR All). The ASR Mini is calculated by averaging 10 runs per task, and the ASR All is calculated by averaging 5 runs per task. Game PT. denotes large-scale policy pretraining or continual pretraining on interaction data spanning many game environments; single-game training and generic foundation-model pretraining are excluded. Results highlighted in red correspond to the state-of-the-art performance for each metric across all comparable agents, while those in blue represent the second-best performance. Embodied Tasks GUI Tasks Combat Tasks Avg Model Game PT. Steps ↓ ASR Mini ↑ ASR All ↑ Steps ↓ ASR Mini ↑ ASR All ↑ Steps ↓ ASR Mini ↑ ASR All ↑ Mini ↑ All ↑ Instruction-Conditioned Policies VPT × 377 10.1±3.610.1^± 3.6 6.0±11.46.0^± 11.4 398 0.7±0.10.7^± 0.1 0.8±3.30.8^± 3.3 396 3.6±7.73.6^± 7.7 3.6±7.73.6^± 7.7 4.8 3.5 STEVE-1 × 384 8.4±3.08.4^± 3.0 8.0±17.08.0^± 17.0 391 0.00.0 3.2±8.43.2^± 8.4 395 4.9±1.84.9^± 1.8 3.9±12.03.9^± 12.0 4.4 5.0 ROCKET-1 × 392 19.2±6.119.2^± 6.1 18.9±24.318.9^± 24.3 – 0.00.0 0.00.0 320 29.8±9.029.8^± 9.0 27.9±29.327.9^± 29.3 16.3 15.6 JARVIS-VLA × 305 31.0±12.731.0^± 12.7 30.0±35.430.0^± 35.4 339 25.3±5.725.3^± 5.7 25.1±23.925.1^± 23.9 352 18.3±5.218.3^± 5.2 18.5±22.718.5^± 22.7 24.9 24.5 VLM-based Agents LatentHA × 363 27.3±37.427.3^± 37.4 24.4±31.124.4^± 31.1 393 3.5±8.73.5^± 8.7 3.0±7.53.0^± 7.5 371 8.2±15.68.2^± 15.6 8.5±17.98.5^± 17.9 13.0 12.0 MotionHA × 336 31.6±10.131.6^± 10.1 27.4±35.227.4^± 35.2 – 0.00.0 0.00.0 392 9.1±3.99.1^± 3.9 4.3±10.84.3^± 10.8 13.6 10.6 GroundingHA × 290 39.7±13.739.7^± 13.7 37.1±38.537.1^± 38.5 380 3.7±2.33.7^± 2.3 6.7±10.86.7^± 10.8 346 28.2±6.228.2^± 6.2 26.5±23.426.5^± 23.4 23.9 23.4 SkillHA × 365 13.8±7.713.8^± 7.7 11.3±14.511.3^± 14.5 397 3.4±0.83.4^± 0.8 6.3±9.26.3^± 9.2 393 3.4±0.83.4^± 0.8 6.5±9.36.5^± 9.3 6.9 8.0 TextVLA × 321 23.9±8.923.9^± 8.9 27.0±17.027.0^± 17.0 291 14.0±4.114.0^± 4.1 25.8±14.325.8^± 14.3 317 27.1±11.827.1^± 11.8 10.0±6.110.0^± 6.1 21.7 20.9 OpenHA × 287 37.0±15.937.0^± 15.9 30.1±13.930.1^± 13.9 314 33.3±13.333.3^± 13.3 32.5±9.232.5^± 9.2 316 40.0±19.640.0^± 19.6 31.9±13.731.9^± 13.7 36.8 31.5 Game-TARS ✓ 373 – 50.4±20.750.4^± 20.7 406 – 39.1±27.539.1^± 27.5 372 – 38.1±24.638.1^± 24.6 – 42.5 Ours GameWAM × 138 70.0±32.270.0^± 32.2 47.5±36.047.5^± 36.0 155 43.0±32.943.0^± 32.9 60.0±38.660.0^± 38.6 203 39.0±25.939.0^± 25.9 32.2±30.232.2^± 30.2 50.7 46.6 Table 1 shows that GameWAM achieves the highest average success rates (ASR) on both the Mini and full task sets while requiring substantially fewer steps per successful episode across all task categories. The interaction-efficiency gains are particularly pronounced on embodied and GUI tasks. 5.3 Main Results on ViZDoom Figure 4: Evaluation results in FPS ViZDoom benchmark across four maps. The reported average reward for each map is calculated over 50 episodes. Fig. 4 evaluates GameWAM under ViZDoom’s distinct dynamics and native control. GameWAM consistently improves over Game-TARS and achieves competitive or leading average rewards relative to the compared multimodal agents across all four scenarios. Together with the MCU results, this demonstrates the formulation’s effectiveness across diverse game environments. 5.4 Ablation Analysis Table 2 shows that removing future-video supervision or using coarser temporal sampling produces the largest performance degradation. Event-anchored clip sampling also contributes substantially, consistent with emphasizing behaviorally informative transitions during training. A unified gameplay/GUI action distribution underperforms the mode-specific formulation, supporting separate prediction and normalization for heterogeneous native controls. Matching the prediction and execution horizons (P=EP=E) further degrades ASR, indicating a benefit from predicting beyond the committed horizon while retaining frequent feedback. Cross-cycle history has a smaller and task-dependent effect: GUI and combat performance declines without it, whereas the embodied point estimate increases slightly, suggesting greater value for tasks that rely on information across replanning cycles. Table 2: GameWAM ablations on MCU Mini. Variant Embodied GUI Combat Avg Full GameWAM 70.0±32.270.0^± 32.2 43.0±32.943.0^± 32.9 39.0±25.939.0^± 25.9 50.750.7 Action-only supervision 60.0±27.660.0^± 27.6 34.0±35.834.0^± 35.8 13.0±13.513.0^± 13.5 35.7 Coarser temporal sampling 60.0±29.760.0^± 29.7 34.0±40.034.0^± 40.0 16.0±36.716.0^± 36.7 36.7 No event-anchored clip sampling 64.0±28.764.0^± 28.7 33.0±30.033.0^± 30.0 17.0±14.917.0^± 14.9 38.0 Unified action distribution 59.0±25.559.0^± 25.5 35.0±27.735.0^± 27.7 21.0±29.121.0^± 29.1 38.3 Matched prediction–execution horizon (P=EP=E) 63.0±33.863.0^± 33.8 32.0±36.832.0^± 36.8 29.0±28.129.0^± 28.1 41.3 No cross-cycle history 75.0±30.175.0^± 30.1 31.0±29.831.0^± 29.8 34.0±31.034.0^± 31.0 46.7 5.5 Low-Frequency Action Source Imprinting Figure 5: Controlled evidence for LASI. Fixed-condition association, low-frequency source replacement, and source zeroing quantify the influence of sampled low-frequency components on generated camera motion. Our investigation of action-source sensitivity arose from a closed-loop failure observed during evaluation. Reusing the same sampled source across replanning cycles caused some realizations to induce persistent directional camera bias and repeated in-place rotation, in severe cases leaving almost no task completion. Resampling between cycles largely removed this pattern, motivating a controlled study of how source variation shapes generated actions. Source dependence is expected in diffusion and flow-based generation because the sampled source initializes the generative trajectory and selects among possible outputs. LASI is more structured: under fixed conditioning, low-temporal-frequency source components coherently steer coarse camera motion. We therefore fix the visual, historical, and proprioceptive context. Let X0,Z∈ℝHa×DaX_0,Z ^H_a× D_a denote a generated continuous-action chunk and its sampled source. We apply an orthonormal DCT along the action horizon, X~0=CX0,Z~=CZ,C⊤C=I, X_0=CX_0, Z=CZ, C C=I, (15) where modes 00–22 form the low-frequency band. Fig. 5 provides three complementary tests. Under fixed conditioning, matching low-frequency source and action coefficients remain strongly associated, with yaw DCT0 reaching r=0.890r=0.890. Replacing only source modes 00–22 makes the corresponding yaw DCT0 output follow the donor in 94.8%94.8\% of trials, while zeroing the same band removes 99.25%99.25\% of the associated output variance. Together, these interventions establish causal evidence that low-frequency action-source components steer coarse generated camera motion rather than a correlation induced by changing conditions. LASI is therefore distinct from ordinary sampling diversity: a sampled source can acquire a coherent low-frequency control bias that becomes harmful when reused through repeated closed-loop interaction. In our closed-loop evaluation, we consequently resample the action source at each replanning step instead of reusing a fixed source throughout the rollout. This mitigates coherent episode-level accumulation without removing the underlying source sensitivity. Frequency selectivity, single-forward transfer, model-path amplification, and closed-loop alignment are in Section D.5. 6 Conclusion We introduced GameWAM, a World–Action Model for native closed-loop gameplay and GUI control that jointly models future visual observations and executable keyboard–mouse actions. Across Minecraft and ViZDoom, GameWAM achieves competitive closed-loop performance, while using fewer executed native actions than the compared agents in Minecraft. We further identify Low-Frequency Action Source Imprinting (LASI), showing that sampled low-frequency action-source structure can induce persistent control bias when reused across replanning steps. Together, these results support joint world–action modeling as a promising direction for interactive game control while exposing a source-sensitivity failure mode specific to generative action policies. AI Use Statement Generative AI tools were used for language polishing and editorial assistance, including improving grammar, clarity, concision, and academic phrasing, as well as refining the presentation of author-produced experimental results. The core research questions, method, model design, data construction, implementation, experimental results, and numerical measurements were developed or produced by the authors. All AI-assisted text was manually reviewed and checked against the implementation, experimental records, and underlying results, and was revised by the authors where necessary. The authors take full responsibility for the final content of this work, including all claims, analyses, and artifacts. Ethics Statement This work studies learned closed-loop control in simulated video-game environments. We conduct no new human-subject experiments and collect no sensitive personal data as part of this study. The reported experiments are restricted to digital game environments and do not involve deployment in physical systems. While GameWAM is developed for research on native game control, models capable of operating general keyboard–mouse interfaces could in principle be adapted to other interactive software; deployment beyond the controlled environments studied here should therefore consider authorization, safety, and potential misuse. Reproducibility Statement We provide detailed descriptions of the model formulation, training objectives, and closed-loop control procedure in the main text and Appendix A. Appendix B documents trajectory alignment, native action interfaces, training data construction, and training-sequence semantics, while Appendix C reports the architecture, temporal and history configurations, training-data mixture and clip-sampling settings, optimization details, objective weights, sampling configuration, and evaluation protocols. Additional quantitative and qualitative analyses are provided in Appendix D. We will publicly release the training and evaluation code, data-processing scripts, constructed datasets, trained GameWAM model weights, and associated configuration files. The released resources will include the complete procedures required to reproduce the reported experiments, subject to applicable third-party licensing and redistribution requirements. Together with the detailed methodological descriptions and experimental protocols provided in this paper, these resources are intended to facilitate reproduction of our results and further research on world–action models for interactive game control. References Alonso et al. (2024) Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024. Baker et al. (2022) Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022. Brohan et al. (2023a) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023a. Brohan et al. (2023b) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. In Proceedings of Robotics: Science and Systems, 2023b. doi: 10.15607/RSS.2023.XIX.025. Bruce et al. (2024) Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first international conference on machine learning, 2024. Cai et al. (2024) Shaofei Cai, Zhancun Mu, Kaichen He, Bowei Zhang, Xinyue Zheng, Anji Liu, and Yitao Liang. Minestudio: A streamlined package for minecraft ai agent development. arXiv preprint arXiv:2412.18293, 2024. Che et al. (2025) Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. In International Conference on Learning Representations, volume 2025, p. 37546–37593, 2025. Cheng et al. (2025) Jie Cheng, Ruixi Qiao, Yingwei Ma, Binhua Li, Gang Xiong, Qinghai Miao, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via jointly-optimized world-action model pretraining. In International Conference on Learning Representations, 2025. Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.026. Du et al. (2023) Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems, volume 36, 2023. Du et al. (2024) Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning. In International Conference on Learning Representations, 2024. Fan et al. (2022) Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343–18362, 2022. Ghosh et al. (2024) Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, et al. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, 2024. Guo et al. (2025) Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft. arXiv preprint arXiv:2504.08388, 2025. Guss et al. (2019) William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440, 2019. Hafner et al. (2019a) Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019a. Hafner et al. (2019b) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, p. 2555–2565. PMLR, 2019b. Hafner et al. (2020) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. Hafner et al. (2025) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647–653, 2025. doi: 10.1038/s41586-025-08744-2. Hansen et al. (2024) Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations, 2024. Harvey et al. (2022) William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in neural information processing systems, 35:27953–27965, 2022. Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022. Kanervisto et al. (2025) Anssi Kanervisto, Dave Bignell, Linda Yilin Wen, Martin Grayson, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Tabish Rashid, Tim Pearce, Yuhan Cao, et al. World and human action models towards gameplay ideation. Nature, 638(8051):656–663, 2025. Kempka et al. (2016) Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG), p. 1–8. IEEE, 2016. Khazatsky et al. (2024) Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. doi: 10.15607/RSS.2024.X.120. Lee et al. (2024) Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 26991–27008. PMLR, 2024. Lifshitz et al. (2023) Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36:69900–69929, 2023. Lipman et al. (2023) Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023. Ma et al. (2024) Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, p. 23–40. Springer, 2024. Micheli et al. (2023) Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In International Conference on Learning Representations, 2023. O’Neill et al. (2024) Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation. IEEE, 2024. Peebles & Xie (2023) William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4195–4205, 2023. Petrenko et al. (2020) Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning, p. 7652–7662. PMLR, 2020. Pumacay et al. (2024) Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. THE COLOSSEUM: A benchmark for evaluating generalization for robotic manipulation. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. doi: 10.15607/RSS.2024.X.133. Raad et al. (2024) Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404.10179, 2024. Schrittwieser et al. (2020) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. Tan et al. (2025) Weihao Tan, Xiangyang Li, Yunhao Fang, Heyuan Yao, Shi Yan, Hao Luo, Tenglong Ao, Huihui Li, Hongbin Ren, Bairen Yi, Yujia Qin, Bo An, Libin Liu, and Guang Shi. Lumine: An open recipe for building generalist agents in 3d open worlds. arXiv preprint arXiv:2511.08892, 2025. Valevski et al. (2025) Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In International Conference on Learning Representations, volume 2025, p. 73754–73776, 2025. Villegas et al. (2022) Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022. Voleti et al. (2022) Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35:23371–23385, 2022. Wang et al. (2023) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. Wang et al. (2026a) Rui Wang, Yue Zhang, Jiehong Lin, Kuncheng Luo, Jianan Wang, Zhongrui Wang, and Xiaojuan Qi. When to trust imagination: Adaptive action execution for world action models. arXiv preprint arXiv:2605.06222, 2026a. Wang et al. (2024a) Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3):1894–1907, 2024a. Wang et al. (2024b) Zihao Wang, Shaofei Cai, Zhancun Mu, Haowei Lin, Ceyao Zhang, Xuejie Liu, Qing Li, Anji Liu, Xiaojian Ma, and Yitao Liang. Omnijarvis: Unified vision-language-action tokenization enables open-world instruction following agents. Advances in Neural Information Processing Systems, 37:73278–73308, 2024b. Wang et al. (2025) Zihao Wang, Xujing Li, Yining Ye, Junjie Fang, Haoming Wang, Longxiang Liu, Shihao Liang, Junting Lu, Zhiyong Wu, Jiazhan Feng, Wanjun Zhong, Zili Li, Yu Wang, Yu Miao, Bo Zhou, Yuanfan Li, Hao Wang, Zhongkai Zhao, Faming Wu, Zhengxuan Jiang, Weihao Tan, Heyuan Yao, Shi Yan, Xiangyang Li, Yitao Liang, Yujia Qin, and Guang Shi. Game-TARS: Pretrained foundation models for scalable generalist multimodal game agents. arXiv preprint arXiv:2510.23691, 2025. Wang et al. (2026b) Zihao Wang, Muyao Li, Kaichen He, Xiangyu Wang, Zhancun Mu, Minghao Liu, Anji Liu, and Yitao Liang. OpenHA: A series of open-source hierarchical agentic models in minecraft. In Proceedings of the 43rd International Conference on Machine Learning, 2026b. Wu et al. (2024) Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024. Ye et al. (2025) Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. In International Conference on Learning Representations, 2025. Ye et al. (2026) Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, Ayaan Malik, Kyungmin Lee, William Liang, Nadun Ranawaka, Jiasheng Gu, Yinzhen Xu, Guanzhi Wang, Fengyuan Hu, Avnish Narayan, Johan Bjorck, Jing Wang, Gwanghyun Kim, Dantong Niu, Ruijie Zheng, Yuqi Xie, Jimmy Wu, Qi Wang, Ryan Julian, Danfei Xu, Yilun Du, Yevgen Chebotar, Scott Reed, Jan Kautz, Yuke Zhu, Linxi Fan, and Joel Jang. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026. Yu et al. (2023) Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10459–10469. IEEE, 2023. Yuan et al. (2026) Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-WAM: Do world action models need test-time future imagination? arXiv preprint arXiv:2603.16666, 2026. Zhang et al. (2023) Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. STORM: Efficient stochastic transformer based world models for reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023. Zhang et al. (2025) Yifan Zhang, Chunli Peng, Boyang Wang, Puyi Wang, Qingcheng Zhu, Fei Kang, Biao Jiang, Zedong Gao, Eric Li, Yang Liu, et al. Matrix-game: Interactive world foundation model. arXiv preprint arXiv:2506.18701, 2025. Zhao et al. (2023) Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.016. Zheng et al. (2025) Xinyue Zheng, Haowei Lin, Kaichen He, Zihao Wang, Qiang Fu, Haobo Fu, Zilong Zheng, and Yitao Liang. Mcu: an evaluation framework for open-ended game agents. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25. JMLR.org, 2025. Appendix Contents Appendix A Extended Method The main text defines the GameWAM formulation, block-causal visibility, prediction–execution decomposition, hierarchical history, and learning objective. This section expands their computational realization under teacher-forced training and online interaction. We first clarify how the same block-causal information boundary is instantiated under the two execution settings and how realized context is organized within and across interaction cycles. We then detail the cross-cycle history update, transformer-level multimodal computation and masking variants, timestep-wise gameplay/GUI action routing, and the supervision semantics of the training objective. Concrete architectural dimensions, rollout horizons, memory capacities, and training configurations are reported separately in Appendix C. A.1 Teacher-Forced Training and Online Rollout Teacher-forced training and online rollout share the same block-causal information boundary but differ in how the clean causal prefix becomes available. During training, complete trajectories are available from the dataset. We therefore construct prediction anchors from the E-spaced execution-aligned temporal grid and provide each anchor with its corresponding ground-truth causal prefix. Future visual and action variables within the prediction horizon remain generation targets and are corrupted according to the flow objective, while observations beyond the prediction boundary remain masked. Teacher forcing therefore changes the computational realization of the clean prefix, rather than the conditional information available to the model. This organization preserves the realized-versus-future separation used during online interaction. Although multiple prediction anchors may be constructed from the same trajectory, each anchor only receives observations that are causally available at its own prediction boundary. Ground-truth observations that occur later in the trajectory do not become additional context for earlier anchors, and predicted future variables from one anchor are never promoted into the clean prefix of another anchor. The resulting training computation therefore preserves the causal semantics of closed-loop execution while allowing supervision to be formed directly from complete trajectories. Online rollout realizes the same dependency structure sequentially. At each decision point, GameWAM predicts a P-step action plan but commits only the first E actions. The environment is then advanced by this committed execution block, producing a new realized observation for the next decision. The model subsequently replans from the updated causal context, while the unexecuted suffix of the previous prediction is discarded. Consequently, only observations obtained through executed interaction can enter the clean prefix or persistent history. The prediction horizon and execution horizon play different roles in this process. The block–cycle temporal organization is defined by the execution horizon E: prediction anchors, committed blocks, and interaction updates are aligned to the same E-spaced temporal grid. In contrast, the prediction horizon P determines how far into the future each anchor is supervised. For the j-th prediction anchor, aj=jE,Yj=A[aj+1:aj+P],a_j=jE, Y_j=A[a_j+1:a_j+P], (16) where YjY_j denotes the corresponding future action target. When P>EP>E, neighboring prediction targets overlap by P−EP-E actions. Increasing P therefore extends the predictive supervision attached to each execution-aligned anchor without changing the underlying block or cycle boundaries. This execution-aligned overlapping supervision increases the predictive coverage extracted from each interaction trajectory. Because the execution-aligned anchors remain fixed by E, increasing P extends the forecast range associated with each anchor rather than requiring additional execution-aligned samples. Realized transitions in overlapping regions can therefore be reused across multiple forecast offsets and supervised from different causally valid prefixes while maintaining the same decision cadence. For example, a transition that is farther in the future for one anchor may become a near-term prediction target for the next anchor. GameWAM therefore combines frequent E-spaced decision points with longer P-step predictive supervision, rather than requiring the training sequence to adopt a longer execution interval. Because complete trajectories are available during training, these overlapping targets do not require autoregressive rollout of intermediate predicted blocks. Each prediction anchor is supervised using its own ground-truth causal prefix, avoiding the need to generate previous blocks merely to construct later conditioning context. This distinction does not make larger prediction horizons computationally free: increasing P still introduces additional future targets and associated prediction losses. Instead, the benefit of the organization is that longer-horizon supervision and frequent execution-aligned anchors coexist without extending the autoregressive commitment chain. During action-only online evaluation, iterative denoising of future visual variables is omitted. The realized visual prefix remains available as the context for action generation, while only the unobserved future video generation process is removed. The resulting computation preserves the same closed-loop execution semantics while removing future-video generation that is not required for immediate control. A.2 Temporal Context Within and Across Interaction Cycles The preceding training and rollout schedules expose two temporal levels of context. The cross-cycle state HcH_c is already defined in the main text and is the historical representation that survives an interaction-cycle boundary. Within an active cycle, we additionally denote by Cc,jC_c,j the realized visual context available before decision j. It contains current-cycle observations that have actually been reached through executed actions. As defined in the main text, the complete clean visual prefix combines persistent cross-cycle history with the realized context accumulated within the active cycle: c,j=Hc∪Cc,j.P_c,j=H_c∪ C_c,j. (17) This decomposition is informational rather than architectural: HcH_c specifies what remains accessible from earlier cycles, whereas Cc,jC_c,j preserves the fine-grained context accumulated inside the current cycle. Predicted but unexecuted future variables belong to neither term. Under teacher forcing, Cc,jC_c,j is obtained directly from the corresponding ground-truth prefix of the sampled trajectory. During online rollout it can only grow after the environment has returned a new realized observation, Cc,j+1inf=Cc,jinf∪oc,j+1env.C^inf_c,j+1=C^inf_c,j∪\o^env_c,j+1\. (18) Hence a predicted future observation is not treated as history merely because it has been generated; only the observation reached after execution can extend the within-cycle context. The online KV cache is a computational representation of the instantiated clean prefix c,jP_c,j, rather than a third source of temporal context. At the beginning of a cycle, the persistent cross-cycle history HcH_c and the initial realized within-cycle context are processed together to form the clean layer-wise K/V states. As interaction continues, only newly realized observations extend the current-cycle portion of this cache. Denoting the retained clean-prefix representation by c,jKVZ^KV_c,j, c,jKV=Ψθ(c,jinf)=Ψθ(Hc∪Cc,jinf),Z^KV_c,j= _θ\! (P^inf_c,j )= _θ\! (H_c∪ C^inf_c,j ), (19) where Ψθ _θ denotes the layer-wise transformation of the clean prefix. During teacher-forced training, the analogous clean representations are formed transiently inside the parallel forward computation. During online inference, they are retained and incrementally extended as new observations are realized. At a cycle boundary the transient K/V state is discarded, while the completed interaction updates the persistent history used to construct c+1,0P_c+1,0 for the next cycle. Thus, HcH_c is the persistent state across cycles, Cc,jC_c,j is the fine-grained realized context accumulated within the active cycle, and c,jKVZ^KV_c,j is their transient processed representation during online computation. Training and inference differ in the computational lifetime of this representation, not in the semantic source of historical information. A.3 Cross-Cycle Hierarchical Memory The main text defines the persistent history as Hc=[Mc;Rc]H_c=[M_c;R_c], where RcR_c retains recent executed segments and McM_c summarizes older segments evicted from the recent buffer. Here we specify the temporal encoding and long-term update used to construct these two components. As in Eq. 8, the representation Sc=Conv3D⁡(VAE⁡(Ocexec))S_c=Conv3D (VAE(O_c^exec) ) (20) is formed only from observations realized through execution. The Conv3D tokenizer reduces spatial resolution while preserving the temporal ordering within the segment. Temporally encoded segment tokens. Both recent-history readout and long-term memory updates operate on the same compressed segment representation ScS_c. Let sc,τ,ks_c,τ,k denote the token at temporal position τ and compressed spatial position k. Each token is augmented with embeddings for its within-segment temporal position and spatial token identity, together with a relative-time embedding. For a segment spanning raw temporal indices [b,e][b,e] and a reference time t, we use ηc(t)=[log⁡(1+t−bTcyc),log⁡(1+(t−e)+Tcyc),e−bTcyc], _c(t)= [ \! (1+ t-bT_cyc ),\; \! (1+ (t-e)_+T_cyc ),\; e-bT_cyc ], (21) where TcycT_cyc is the cycle stride in the native interaction timeline. A small embedding network ψ maps this descriptor to the model dimension. The resulting token representation has the form uc,τ,k(t)=sc,τ,k+eτtime+ekspace+ψ⁡(ηc(t)).u_c,τ,k(t)=s_c,τ,k+e^time_τ+e^space_k+ψ\! ( _c(t) ). (22) Recent-history readout and long-term writing use separate role embeddings and normalization parameters on top of uc,τ,ku_c,τ,k, while sharing the underlying compressed segment tokens and temporal descriptor. For recent segments, t is the current interaction time, so their relative-age encoding evolves as the rollout proceeds. For a segment entering long-term memory, the reference is its overflow time from the recent buffer. Recent buffer and overflow. The recent branch retains the latest KRK_R completed segments explicitly: (Rc+1,S¯c)=FIFOKR⁡(Rc,Sc).(R_c+1, S_c)=FIFO_K_R(R_c,S_c). (23) No long-term write occurs while the buffer has free capacity. Once it is full, the oldest segment S¯c S_c is evicted and becomes the source of a long-term memory update. Thus, recent observations remain individually addressable until they leave the recent buffer; hierarchical summarization is triggered by overflow rather than applied to every completed cycle. Multi-timescale long-term update. Let Mc∈ℝL×KM×dM_c ^L× K_M× d contain KMK_M persistent slots at each of L temporal levels. The evicted segment is first converted to long-memory source tokens BcB_c using the long-term role encoding described above. Each memory slot attends directly to the complete overflow segment: M~ℓ,k=Attn⁡(Mℓ,k+eℓlevel+ekslot,Bc,Bc). M_ ,k=Attn (M_ ,k+e^level_ +e^slot_k,\;B_c,\;B_c ). (24) The level and slot embeddings distinguish the roles of different persistent states, while the update rate varies explicitly across temporal levels. Let hℓh_ denote the characteristic half-life of level ℓ and let Δ denote the duration represented by the incoming segment. The corresponding update prior is αℓ(Δ)=1−2−Δ/hℓ. _ ( )=1-2^- /h_ . (25) Shorter-timescale levels therefore place a larger prior weight on a new segment, whereas longer-timescale levels change more conservatively. The prior is combined with a content-dependent correction computed from the existing slot and its attention candidate: gℓ,k g_ ,k =sigmoid⁡(logit⁡(αℓ(Δ))+G⁡([Mℓ,k;M~ℓ,k])), =sigmoid (logit\! ( _ ( ) )+G\! ([M_ ,k; M_ ,k] ) ), (26) Mℓ,k′ M _ ,k =LN⁡(Mℓ,k+gℓ,k⊙(M~ℓ,k−Mℓ,k)), =LN (M_ ,k+g_ ,k ( M_ ,k-M_ ,k) ), where gℓ,k∈(0,1)dg_ ,k∈(0,1)^d provides a feature-wise gated interpolation. Consequently, the temporal prior determines the nominal update rate of each level, while the learned correction allows the content of a particular overflow segment to modulate that update. The long-term branch is itself temporally bounded. Only a fixed number of the most recent overflow segments, determined by the largest configured memory timescale, contribute to the persistent state. Writing this bounded ordered set as cQ_c, the effective memory can be viewed as applying Eq. 26 sequentially over cQ_c in temporal order. Hence both the number of explicit recent tokens and the number of long-term memory slots remain independent of episode length. At readout, the long-term slots are flattened and concatenated with the temporally encoded recent segments to form the persistent history supplied to the next cycle: Hc=[Mc;Rc].H_c=[M_c;R_c]. (27) This construction preserves explicit access to the most recent realized interaction while progressively summarizing older executed context at multiple temporal scales. Predicted but unexecuted observations never enter either branch of the persistent history. A.4 Block-Causal Multimodal Computation Figure 6: Block-causal visibility for the default modality-decoupled world–action formulation. The training view (left) and autoregressive rollout view (right) show three overlapping plans. Visual and action predictions receive the same clean causal prefix, while simultaneously corrupted future visual and action tokens are not used as cross-modal conditions. Completed blocks are teacher-forced during training; at rollout, only observations reached through committed execution enter the clean prefix. Planned but unexecuted suffixes are discarded before the next decision. Colored cells indicate visible entries, gray cells are masked, green cells denote clean retained context, and hatched cells denote predicted but unexecuted variables. Figure 7: Block-causal visibility with joint current-block video–action coupling. The joint alternative allows aligned noisy visual and action variables to exchange information inside the current planning block while retaining the same clean causal prefix and the same exclusion of later blocks. Training and rollout otherwise follow the same teacher-forcing and execution-grounded context rules as in Fig. 6. The main text introduces the modality-decoupled block-causal mask used by the reported GameWAM model. Here we make its transformer-level realization explicit and contrast it with a joint within-block video–action alternative. For target block j, let c,jP_c,j denote the causally available clean visual prefix, and let jV_j and jA_j denote the corresponding noisy video and action variables. Instruction, proprioceptive context, and other non-prefix conditions remain available through their respective conditioning paths and are omitted below for clarity. At the transformer level, the Video and Action DiTs retain modality-specific hidden states, prediction parameters, and flow-time modulation. Their interaction is instead determined by the key/value states made visible to each query stream. For modality m∈v,am∈\v,a\ at layer ℓ , a generic update can be written as h~m,ℓ h^m, =hm,ℓ+Gattnm,ℓ(Attn⁡(Qm,ℓ,c,jm,ℓ,c,jm,ℓ),σm), =h^m, +G^m, _attn\! (Attn (Q^m, ,K^m, _c,j,V^m, _c,j ); _m ), (28) hm,ℓ+1 h^m, +1 =h~m,ℓ+Gffnm,ℓ(h~m,ℓ,σm), = h^m, +G^m, _ffn\! ( h^m, ; _m ), where the modality-specific gates absorb the corresponding flow-time modulation, and (c,jm,ℓ,c,jm,ℓ)(K^m, _c,j,V^m, _c,j) denotes the layer-ℓ attention context assembled according to the block-causal visibility rule. For the default modality-decoupled mask, Vis⁡(j)=c,j∪j,Vis⁡(j)=c,j∪j.Vis(A_j)=P_c,j _j, (V_j)=P_c,j _j. (29) Importantly, this decoupling applies to the simultaneously corrupted future variables; it does not separate action generation from the learned representation of the realized visual world. Concretely, at each transformer layer the complete clean visual prefix c,jP_c,j is processed by the Video DiT to produce layer-wise clean-prefix key/value states (c,jp,ℓ,c,jp,ℓ)(K^p, _c,j,V^p, _c,j). The same states are reused by both prediction branches. Let (jv,ℓ,jv,ℓ)(K^v, _j,V^v, _j) and (ja,ℓ,ja,ℓ)(K^a, _j,V^a, _j) denote the key/value states of the current noisy video and action variables, respectively. Omitting the additional conditioning paths above, the default attention contexts have the schematic form c,jv,ℓ ^v, _c,j =[c,jp,ℓ;jv,ℓ], = [K^p, _c,j;K^v, _j ], c,jv,ℓ ^v, _c,j =[c,jp,ℓ;jv,ℓ], = [V^p, _c,j;V^v, _j ], (30) c,ja,ℓ ^a, _c,j =[c,jp,ℓ;ja,ℓ], = [K^p, _c,j;K^a, _j ], c,ja,ℓ ^a, _c,j =[c,jp,ℓ;ja,ℓ]. = [V^p, _c,j;V^a, _j ]. Thus, video queries attend to the realized clean prefix and the current noisy video states, whereas action queries attend to the same realized clean prefix and the current noisy action states. The action branch does not read current-block noisy-video K/V states, and the video branch does not read current-block noisy-action K/V states. The two branches therefore remain coupled through a shared learned representation of the realized clean prefix, rather than through direct exchange between simultaneously denoised future variables. This distinction is also important for understanding the role of video supervision. The clean-context K/V states are produced by the Video DiT and remain in the training computation graph when they are consumed by both branches. Consequently, gradients from both the video and action objectives propagate through the Video-DiT path that constructs this shared context. Video prediction can therefore shape the realized-world representation used by action generation even though the action prediction does not condition on a denoised future video. In this sense, the default formulation couples world and action learning through the shared representation of realized dynamics, while deliberately decoupling their current noisy future variables. Fig. 6 visualizes this computation under teacher-forced training and autoregressive rollout. During training, completed blocks provide clean causal context, but future targets that are not yet causally available remain excluded from the corresponding clean prefix. During online interaction, only observations reached after committed environment actions are promoted into the realized context; predicted but unexecuted suffixes never become clean conditioning states. The same dependency structure is retained during action-only online inference. The instantiated clean prefix c,jP_c,j is processed by the Video DiT to construct layer-wise clean-prefix K/V states, which are cached and consumed by Action DiT denoising. As new observations are obtained through execution, only the newly realized within-cycle context extends this cache. What is omitted during action-only inference is the iterative denoising of future video variables, not the Video-DiT encoding of the realized clean prefix. The default mask therefore preserves the learned visual-context pathway used for action generation while avoiding the additional online cost of future-video denoising. For the joint within-block alternative, Vis⁡(j)=Vis⁡(j)=c,j∪j∪j.Vis(A_j)=Vis(V_j)=P_c,j _j _j. (31) The corresponding attention contexts contain both current noisy modalities: c,jv,ℓ=c,ja,ℓ ^v, _c,j=K^a, _c,j =[c,jp,ℓ;jv,ℓ;ja,ℓ], = [K^p, _c,j;K^v, _j;K^a, _j ], (32) c,jv,ℓ=c,ja,ℓ ^v, _c,j=V^a, _c,j =[c,jp,ℓ;jv,ℓ;ja,ℓ]. = [V^p, _c,j;V^v, _j;V^a, _j ]. The joint formulation therefore introduces direct bidirectional interaction between the current noisy future-video and action variables during denoising. The temporal causal boundary is otherwise unchanged: both formulations use the same realized clean prefix and exclude information from later blocks. This additional within-block coupling also changes online computation. Because the action queries depend on the simultaneously denoised future-video states, joint action generation requires the future visual stream to be denoised alongside the action stream. By contrast, the default modality-decoupled model retains the shared realized-context pathway while allowing future-video denoising to be omitted during action-only deployment. We compare the two formulations under matched training and evaluation settings in Section D.2. A.5 Interaction-Conditioned Native Action Generation Minecraft gameplay and GUI interaction use the same physical keyboard–mouse interface but induce substantially different conditional control distributions. GameWAM therefore retains a single native action representation while specializing action prediction by interaction mode. The specialization is applied independently at each native action timestep rather than once to an entire prediction horizon, allowing a generated trajectory to transition between gameplay and GUI interaction without switching models or invoking a separate high-level controller. The Action DiT produces gameplay- and GUI-specific flow predictions together with a timestep-wise routing prediction. During training, the observed interaction-mode label selects which action branch is supervised at each timestep and simultaneously provides the target for the routing loss. During rollout, the predicted route instead selects the branch used to generate the complete native action vector at that timestep. Routing therefore chooses between interaction-specific conditional predictions while preserving a shared action dimensionality and a single low-level control interface. This distinction is particularly important for continuous control. The same continuous coordinates represent camera motion during gameplay and cursor motion during GUI interaction, but their numerical distributions differ substantially. We therefore normalize continuous action coordinates with interaction-specific statistics before flow modeling and invert the corresponding normalization after generation. Binary keyboard and mouse coordinates retain their shared normalization, while validity masks exclude coordinates unavailable for a particular environment or interaction state. Environment-specific thresholding, hotbar constraints, and coordinate definitions are given in Section B.2. The router changes the conditional distribution used to predict an action; it does not decompose the controller into independent gameplay and GUI policies. Both branches operate on the same Action-DiT representation and predict the same native action vector, and the route can change from one action timestep to the next. Consequently, heterogeneous interaction modes are handled within a single world–action trajectory while retaining mode-appropriate continuous control statistics. A.6 Supervision Semantics Beyond the Compact Objective The losses in Section 4.3 already define the optimized quantities. Three details clarify how those terms are applied. First, valid continuous and binary action coordinates are reduced separately before being combined. This normalizes heterogeneous action coordinates but does not imply independent action policies; both groups belong to the same generated native action trajectory. Validity masks exclude padding and coordinates unavailable in a particular environment or interaction state. Second, teacher forcing applies clean context only to blocks that are causally complete at the prediction boundary. The current block’s future visual and action variables remain generation targets and are corrupted independently, while completed visual blocks provide clean conditioning. This preserves the realized-versus-uncertain partition used by online rollout. Finally, the predictive-history objective in Eq. 13 operates only on the bounded cross-cycle history. Its pooled representation predicts the clean visual feature at the current observation boundary, which is treated as a fixed target. This auxiliary constraint discourages information loss during history compression and encourages HcH_c to retain visual information predictive across cycle boundaries, without changing the main world–action flow objective. Numerical weights are given in Appendix C. Appendix B Data and Training Construction B.1 Trajectory Alignment All data sources are converted to a common observation–state–action timeline. The state paired with an action is the state observed immediately before that action is executed. Binary controls, continuous camera/cursor values, padding indicators, and validity masks for unavailable state or action fields are retained before normalization. The same pre-action convention determines whether a Minecraft step belongs to gameplay or GUI interaction. For VPT, video observations are synchronized with the corresponding per-frame mouse, keyboard, and Minecraft state records before conversion to the shared trajectory representation. A null-action filtering stage removes redundant null controls together with their paired frames while preserving alignment between retained observations, state, actions, and event metadata. The resulting episodes store aligned action and state records, episode-aligned visual observations, and the metadata required for training. B.2 Native Action Interfaces The main text defines GameWAM over hybrid environment actions. Tables 3 and 4 summarize the action coordinates used in our experiments. Table 3: Minecraft action interface. The first two coordinates are continuous camera deltas; the remaining coordinates are binary keyboard or mouse controls. Index Control Type Index Control Type 0 Camera pitch continuous 11 Drop item binary 1 Camera yaw continuous 12 Open inventory binary 2 Move forward binary 13 Hotbar slot 1 binary 3 Move backward binary 14 Hotbar slot 2 binary 4 Move left binary 15 Hotbar slot 3 binary 5 Move right binary 16 Hotbar slot 4 binary 6 Jump binary 17 Hotbar slot 5 binary 7 Sneak binary 18 Hotbar slot 6 binary 8 Sprint binary 19 Hotbar slot 7 binary 9 Attack binary 20 Hotbar slot 8 binary 10 Use/interact binary 21 Hotbar slot 9 binary Table 4: ViZDoom action interface used in our experiments. A unified 9-D representation covers both map families. Battle maps use continuous horizontal turning, whereas the two defense maps use binary left/right turning. Index Control Type Used in 0 Fire binary all maps 1 Move forward binary Battle maps 2 Move backward binary Battle maps 3 Strafe left binary Battle maps 4 Strafe right binary Battle maps 5 Speed modifier binary Battle maps 6 Turn left binary defense maps 7 Turn right binary defense maps 8 Horizontal turn delta continuous Battle maps The camera coordinates are normalized continuously. Binary controls are thresholded after generation, and the hotbar coordinates are constrained to select at most one slot. Gameplay and GUI interaction share the same action layout but use separate normalization statistics and mode-specific decoding. The Minecraft state input contains view orientation, cursor location, selected hotbar slot, and GUI state; unavailable fields are explicitly masked. For Battle maps, the continuous turn coordinate is normalized to [−1,1][-1,1], corresponding to a horizontal rotation of up to 10∘10 per interaction step. No separate continuous pitch control is used in the reported ViZDoom experiments. The proprioceptive input contains planar position, heading, and health, with unavailable fields treated as missing rather than observed zeros. Figure 8: Construction of WAM-ready game trajectories. VPT recordings are standardized into regular and Event-Anchored datasets; the event branch detects interaction events, extracts anchored windows, filters temporally redundant candidates, and applies max–min fair allocation across event types. MineStudio-based scripted GUI trajectories broaden interface-interaction coverage. All streams share the same native action space and trajectory representation for joint world–action training. B.3 Training Data Sources Minecraft. Minecraft training uses three complementary streams. Regular VPT trajectories (Baker et al., 2022) provide broad naturalistic interaction coverage and long temporal context. From the same source recordings, we construct an Event-Anchored VPT dataset around identifiable MineStudio-style interaction events (Cai et al., 2024), providing instruction-conditioned supervision around behaviorally meaningful transitions. MineStudio-based scripted GUI trajectories form the third stream and broaden interface-interaction coverage. All three streams follow the same observation–state–action alignment and native action representation. The two VPT streams differ at the dataset level. Regular VPT preserves the standardized long trajectories, whereas the Event-Anchored stream applies the offline event-selection procedure described below to construct anchored sub-trajectories. Event-anchored clip sampling is subsequently applied during training and is therefore distinct from construction of the Event-Anchored Dataset. ViZDoom. We collect expert trajectories using Asynchronous Proximal Policy Optimization (APPO) (Petrenko et al., 2020) on the two Battle maps and the Defend the Center and Defend the Line maps. The action actually executed at each recorded interaction step is preserved and expressed in the unified 9-D representation of Table 4. Continuous horizontal turning is retained on the Battle maps rather than discretized. ViZDoom does not require a separate gameplay/GUI interaction mode. B.4 Event-Anchored VPT Dataset Construction The Event-Anchored VPT Dataset is constructed offline from the same source recordings as Regular VPT. Its construction proceeds through event detection, anchored window extraction, temporal redundancy filtering, and allocation across event types. The concrete construction parameters used for the reported dataset are provided with the sampling configuration in Section C.3. Event detection. We identify MineStudio-style interaction events from changes in the recorded Minecraft state statistics. The retained event set covers identifiable interactions such as mining, entity interaction, crafting, and related state transitions. Each detected event is associated with its temporal location and corresponding instruction metadata, providing an anchor aligned with the original visual and native-action trajectory. Anchored window extraction. For an event occurring at temporal index tit_i, we extract a surrounding trajectory window Wi=X[ti−L:ti+R],W_i=X[t_i-L:t_i+R], (33) where X denotes the aligned observation–state–action trajectory and L and R specify the retained context around the event. The event position is preserved within the extracted sub-trajectory as anchor metadata used by the subsequent filtering and training-sampling stages. Dual-level clip filtering. Event detection can yield temporally redundant candidates, either because different detected events produce strongly overlapping windows or because multiple occurrences of the same event type appear in close succession. We therefore apply two temporal filtering criteria: one limits overlap among candidate clips, while the other suppresses nearby repetitions of the same event type. This reduces redundant supervision before event-type allocation while retaining distinct interactions and their surrounding context. Max–min fair allocation. The filtered candidate pool remains imbalanced because event types occur with substantially different frequencies in natural trajectories. We therefore allocate the retained dataset budget across event types using max–min fairness. Event types with fewer available candidates saturate at their available count, while the remaining capacity is redistributed among event types with larger candidate pools. This limits domination by highly frequent events while preserving long-tail interactions whenever sufficient candidates are available. The retained windows, together with their event anchors, instruction metadata, aligned observations, state, and native actions, constitute the Event-Anchored VPT Dataset. B.5 Scripted GUI Trajectory Construction The scripted GUI stream is generated in MineStudio through rule-scripted native keyboard–mouse interaction. The collection covers diverse interface procedures and uses multiple collection modes to vary the interaction trajectories. The scripts are used only to generate demonstrations: the resulting training data contains rendered observations, proprioceptive state, and native actions in the same representation used by the VPT streams. The collected episodes undergo the same frame-wise parsing, state preservation, action mapping, and trajectory alignment as the other Minecraft data sources. Consequently, scripted GUI data broadens interface-interaction coverage without introducing a separate action representation for training. B.6 Event-Anchored Training Clip Sampling Event-Anchored Dataset construction determines which anchored sub-trajectories are retained, whereas event-anchored clip sampling separately determines the temporal density of training windows drawn from them. Long interaction trajectories contain highly uneven supervision density. Transitions near an annotated event often contain the visual changes and actions most directly related to the corresponding instruction, whereas more distant portions may consist largely of traversal, incidental camera motion, or other weakly related behavior. Uniform window sampling can therefore devote substantial training capacity to comparatively less informative regions. For the Event-Anchored VPT stream, we increase the density of candidate training windows whose observation span overlaps the neighborhood of the stored event anchor, while sampling the remaining regions more sparsely. This increases exposure to event-relevant state transitions while retaining surrounding interaction context. Regular VPT and scripted GUI trajectories use fixed sampling intervals rather than anchor-dependent sampling. The mixture weights, anchor neighborhood, and clip-sampling intervals used in the reported training run are given in Section C.3. Appendix C Implementation and Evaluation Protocol C.1 Model Architecture and Initialization The visual predictor is initialized from Wan2.2-TI2V-5B. The Action DiT follows the same 30-layer, 24-head transformer layout but uses a reduced hidden width of 1024, compared with 3072 for the Video DiT. To retain the pretrained video-transformer prior despite this width reduction, we adapt the Wan2.2 Video-DiT weights to initialize the Action-DiT backbone, while action-specific input/output projections, gameplay/GUI prediction heads, and the interaction-mode router are initialized separately and learned during GameWAM training. Diffusion-time modulation is applied independently to the two streams. The video VAE is kept frozen, and text-conditioning features are produced by a fixed text encoder. The Video DiT, Action DiT (including the interaction-specific action prediction heads and router), hierarchical history module, and proprioceptive encoder are optimized during GameWAM training. C.2 Observation, Planning, and Temporal-Context Configuration Both environments use 224×224224× 224 RGB observations. Visual observations are sampled once every two native environment actions. The prediction horizon is P=16P=16 native actions and the execution horizon is E=8E=8, giving an eight-action overlap between neighboring predictions. Three committed execution blocks form one 24-action cycle for the hierarchical temporal context. For each completed cycle, the cross-cycle representation preserves four latent time positions and projects each position to four spatial tokens, giving 16 tokens per cycle-level segment. The two most recent completed segments are kept explicitly. Older context is summarized at two temporal scales with four memory tokens per scale; the corresponding scale half-lives are two and four completed cycles. The resulting cross-cycle history contains at most 40 tokens. During training, the available cross-cycle history is shortened with probability 0.10.1 by retaining a randomly selected valid suffix, including the possibility of no cross-cycle history. The environment-specific action interfaces are given in Tables 3 and 4. Minecraft uses timestep-wise interaction conditioning for gameplay and GUI behavior, whereas the evaluated ViZDoom tasks use a single interaction regime. C.3 Training Data Mixture and Clip-Sampling Configuration Event-Anchored VPT construction. The reported Event-Anchored VPT dataset uses 96-frame sub-trajectories. For an event at frame t, the extraction window ends eight frames after the event, giving the interval [t−87,t+8][t-87,t+8] and retaining both substantial pre-event context and a short post-event continuation. Null-action filtering is applied before event-window construction while preserving detected event anchors. During dual-level filtering, retained clip starts are separated by at least 64 frames, and anchors of the same event type are separated by more than 96 frames. The filtered candidates from the VPT partitions are pooled and subject to a global max–min fair allocation with a total budget of 200,000 Event-Anchored sub-trajectories. Candidate selection is shuffled with random seed 42. VPT data are processed at their native 20 FPS. Training mixture and temporal sampling. The reported Minecraft training run uses sample-mixture weights of 80% for Event-Anchored VPT, 5% for Regular VPT, and 15% for scripted GUI trajectories. These weights specify the training-sample distribution rather than the raw corpus composition. The three streams use different temporal intervals for constructing training clips. For Event-Anchored VPT, candidate windows whose observation span intersects a radius-88 neighborhood of the stored event anchor are sampled with stride 2, while the remaining regions use stride 16. Regular VPT uses stride 32 throughout the trajectory, whereas scripted GUI trajectories use stride 8. Thus, only the Event-Anchored stream varies its training-window density according to temporal proximity to an interaction event. C.4 Optimization and Objective Weighting GameWAM is trained for two epochs using fused AdamW with β1=0.9 _1=0.9, β2=0.95 _2=0.95, weight decay 0.010.01, and gradient clipping at a maximum norm of 1.01.0. The learning rate is linearly warmed up over the first 5% of optimization, corresponding to 1,095 steps, to a peak value of 4×10−54× 10^-5, and then follows cosine decay to a minimum of 4×10−74× 10^-7. Each epoch contains 10,950 optimizer steps, for 21,900 steps in total. Training uses BF16 precision and DeepSpeed ZeRO-2 on eight NVIDIA H200 GPUs. The per-GPU batch size is 44 with gradient accumulation of one, giving an effective global batch size of 352. The full two-epoch training run takes approximately 22 hours. For the objective in Eq. 14, the reported model uses λv=1.0 _v=1.0, λa=1.0 _a=1.0, λm=0.05 _m=0.05, and λh=0.5 _h=0.5. Within the grouped action objective in Eq. 11, continuous and discrete action losses are weighted equally with λcont=λdisc=1.0 _cont= _disc=1.0. C.5 Flow-Training and Sampling Configuration The visual and action flow times are sampled independently. For u∼⁡[0,1]u [0,1], the reported training runs use the shifted time mapping σ=5u1+4u.σ= 5u1+4u. (34) The corresponding flow residuals are weighted by w⁡(σ)=exp⁡[−2(σ−12)2]−exp⁡(−12)Z,w(σ)= [-2(σ- 12)^2]- (- 12)Z, (35) where Z normalizes the mean weight under the sampling schedule. At inference, sampling starts from Gaussian sources and follows the learned vector field from σ=1σ=1 to σ=0σ=0 with first-order integration. Standard closed-loop evaluation uses ten denoising steps. C.6 Closed-Loop Evaluation Protocols All reported task-performance evaluations use the corresponding final checkpoint after the second training epoch, without validation- or MCU-based checkpoint selection. Unless otherwise stated, the action source is independently resampled at each replanning step rather than reused across an episode. This mitigates coherent episode-level accumulation from repeated reuse of a single source; the corresponding LASI failure mode and source-resampling analysis are detailed in Section D.5. Minecraft MCU. We use the MCU protocol reported in the main paper, including the Mini subset of 30 tasks (10 mining, 10 crafting, and 10 combat tasks). Success is determined by the environment’s task-completion signals. Interaction steps count native environment actions actually executed during closed-loop rollout, excluding internal denoising evaluations. The reported step metric is averaged over successful episodes within each task category and is undefined when no successful episode is observed. The reported task-wise standard deviation is computed across per-task success rates. Because the systems compared in Table 1 differ substantially in training data, task-specific supervision, model class, action representation, and available inputs, the table should primarily be interpreted as a system-level benchmark comparison rather than a controlled comparison of architecture or data efficiency. ViZDoom. We use the four-map protocol reported in the main paper and measure average episode reward. The controller uses the action interface in Table 4 and the same P=16P=16, E=8E=8 overlapping-plan geometry as Minecraft. Online execution frequency. For the inference-efficiency comparison in Section D.2 and Table 6, online execution frequency is evaluated separately from task-performance evaluation on a single NVIDIA H200 GPU under the standard inference configuration. For each model variant, we report the mean frequency over 10 closed-loop episodes. Timing includes all model-side computation performed during online replanning and excludes environment simulation and execution time. Appendix D Additional Experimental Analysis D.1 Training Scale and Token Consumption Table 5 places the training scale of GameWAM in context with OpenHA (Wang et al., 2026b) and Game-TARS (Wang et al., 2025). We retain the token categories reported by each method rather than forcing heterogeneous training pipelines into a shared modality definition. GameWAM uses a single training stage, whereas OpenHA and the reported Game-TARS training recipe use two stages. Table 5: Model scale and reported training-token consumption. Token categories follow the accounting reported by each work. For two-stage methods, modality-level counts are shown as Stage 1 / Stage 2. A dash indicates that the corresponding count is not separately reported or is not applicable. GameWAM OpenHA Game-TARS Model initialization and scale Backbone(s) Video: Wan2.2-TI2V-5B Action: Wan2.2-initialized Action-DiT Qwen2-VL-7B Qwen2.5-VL-7B-Instruct Backbone scale / params. Video: 5B Action-DiT: 1B 7B 7B Training-token exposure Stage 1 Single stage 3.40B 526B Stage 2 0.22B 40B Video tokens 2.27B – – Image tokens – – 208B / 10B Action tokens 0.37B – – Text tokens 0.15B – 326B / 29B Total consumed tokens 2.79B 3.62B 566B For GameWAM, the reported counts follow the exact two-epoch sampling schedule rather than an estimate based on average sequence length. Video and action tokens count target-token presentations participating in the training objective, including overlapping autoregressive planning targets. Text tokens count non-padding tokenizer tokens under the same sampling schedule; zero-padded text slots are excluded. Across the two epochs, GameWAM consumes 2.27B video-target tokens, 0.37B action-target tokens, and 0.15B effective text-conditioning tokens, for 2.79B total consumed tokens. The exact schedule presents 7,708,800 training samples over the two epochs. The 150,723,927 non-padding text tokens therefore correspond to an average of 19.55 effective text tokens per presented sample. The implementation uses a fixed 128-slot text context, corresponding to 986,726,400 text slots over the same schedule, but most of the additional positions are zero padding. We therefore use non-padding tokenizer tokens for the reported text-token consumption rather than counting padded computational slots. For OpenHA, Stage 1 and Stage 2 correspond to its mixed-action pre-training and CoA fine-tuning stages, respectively. The reported consumed-token counts are 3.40B and 0.22B, giving 3.62B in total. A modality decomposition directly comparable to the video, action, and text accounting used for GameWAM is not separately reported, so the corresponding entries are left unspecified. For Game-TARS, we reproduce the stage-wise quantities in its reported training recipe without redefining their token accounting. Pre-training reports 526B total tokens, including 208B image tokens and 326B text tokens, while post-training reports 40B total tokens, with 10B image tokens and 29B text tokens. We retain these values exactly as reported rather than reconstructing the totals from the modality columns. Our experimental comparison uses the Game-TARS-Dense model, whose reported initialization is Qwen2.5-VL-7B-Instruct; because a separate exact stage-wise token accounting for this Dense variant is not provided, the Game-TARS token entries in Table 5 should be interpreted as the training scale reported for the Game-TARS model family. At the reported stage level, OpenHA has approximately 1.30×1.30× the total token exposure of GameWAM, while the paper-reported Game-TARS recipe exceeds the GameWAM total by more than 200×200×. These quantities provide context for training-data scale rather than a direct measure of computational cost. The models differ in tokenizer design, modality representation, sequence construction, objectives, and architecture, so token exposure should not be interpreted as FLOP-equivalent training compute. D.2 Cross-Modal Masking: Control, World Modeling, and Inference Efficiency Section A.4 defines and visualizes the default modality-decoupled mask and the joint within-block alternative in Figs. 6 and 7. We compare the two under otherwise matched model and evaluation settings. This comparison examines whether direct interaction between noisy video and action variables provides a control benefit sufficient to justify the additional computation required by joint video–action denoising. Table 6: Cross-modal masking, closed-loop performance, and online inference efficiency. We report MCU Mini and MCU All ASR together with executed environment steps averaged over successful episodes under the closed-loop protocols in Section C.6, and separately report online execution frequency. Embodied GUI Combat Avg Exec. Freq. Mask Steps ↓ Mini ↑ All ↑ Steps ↓ Mini ↑ All ↑ Steps ↓ Mini ↑ All ↑ Mini ↑ All ↑ (Hz) ↑ Modality-decoupled (default) 138 70.0±32.270.0^± 32.2 47.5±36.047.5^± 36.0 155 43.0±32.943.0^± 32.9 60.0±38.660.0^± 38.6 203 39.0±25.939.0^± 25.9 32.2±30.232.2^± 30.2 50.750.7 46.646.6 12.5112.51 Joint video–action 141 70.0±24.170.0^± 24.1 40.3±35.540.3^± 35.5 145 45.0±34.145.0^± 34.1 58.8±38.558.8^± 38.5 175 24.0±25.024.0^± 25.0 19.6±24.719.6^± 24.7 46.346.3 39.639.6 8.128.12 Table 6 shows that the modality-decoupled design provides stronger overall closed-loop performance. The two variants match on Embodied Mini, while the joint model is slightly higher on GUI Mini. On the broader MCU All evaluation, however, the default model performs better in all three categories, with the largest difference on Combat. Overall ASR increases from 46.3 to 50.7 on MCU Mini and from 39.6 to 46.6 on MCU All with the modality-decoupled mask. The larger difference on MCU All suggests that the advantage of modality decoupling becomes more apparent across the broader task set rather than being confined to the Mini subset. The step results provide a complementary view of interaction cost. The modality-decoupled model requires fewer steps on Embodied tasks, whereas the joint model reports fewer steps on GUI and Combat. These values should be read together with task success: most notably, the lower Combat step count of the joint model coincides with a substantial reduction in both Mini and All ASR. The step results therefore do not change the overall pattern that the default mask provides substantially stronger Combat control. The computational advantage of modality decoupling is more pronounced. Joint video–action generation operates at 8.12 Hz under the reported inference configuration, whereas the modality-decoupled model reaches 12.51 Hz, corresponding to approximately a 1.54×1.54× higher online execution frequency. This gain follows directly from removing iterative future-video denoising from the online action-generation path: video prediction remains available as a training signal, while deployment does not require the future visual stream to be jointly denoised with the actions. Interestingly, the optimization dynamics do not simply mirror the downstream control results. During training, the joint model fits the action objective faster, while optimization of the video branch progresses more slowly. Qualitative inspection of validation predictions reveals the same asymmetry: future-video predictions from the joint model are generally less sharp than those produced by the modality-decoupled model. Faster fitting of the action objective therefore does not translate into stronger overall closed-loop control or better future-video prediction. The magnitude of this difference also appears to depend on the visual dynamics of the interaction regime. GUI trajectories typically contain comparatively stable viewpoints and relatively limited frame-to-frame visual change, making their future observations easier to predict than gameplay sequences involving ego-motion, moving entities, and rapid scene transitions. Consistent with this distinction, the joint variant shows little degradation on GUI: its GUI Mini ASR is slightly higher than that of the default model (45.0 versus 43.0), while GUI All remains close (58.8 versus 60.0). This may partly explain why joint coupling does not produce the substantial GUI degradation observed in more visually dynamic settings. The difference is considerably larger on Combat, where rapid camera motion, moving targets, and abrupt changes in scene content make the visual future more difficult to predict. We likewise observe qualitatively that prediction blur becomes more pronounced in fast-motion gameplay sequences, particularly for the joint model. Together with the weaker video-side optimization, this suggests that the consequences of cross-modal coupling may become more significant as the visual prediction problem becomes harder. One possible interpretation is asymmetric optimization interference between action fitting and visual-dynamics learning. Allowing noisy future-video and action variables to interact directly may make the action objective easier to fit while simultaneously weakening optimization of the visual predictive process, particularly when the visual future is intrinsically difficult to model. This interpretation is consistent with the faster action-loss fitting, slower video-side optimization, blurrier validation predictions, and larger performance degradation in visually dynamic gameplay. We nevertheless treat this as a hypothesis rather than a mechanistic conclusion, since the present experiments do not directly isolate gradient interference or identify where such competition arises inside the model. This observation may also have implications beyond games. Future visual prediction in physical embodied environments can be challenging under ego-motion, clutter, partial observability, sensor variation, stochastic dynamics, and other sources of visual uncertainty. Strongly coupling action learning to an already difficult visual prediction process may therefore not always be beneficial, and optimization pressure from action learning could potentially interfere with learning useful visual dynamics. The modality-decoupled formulation provides an alternative in which video prediction continues to supply dynamics-aware training supervision without making online action generation directly depend on uncertain future-video variables. Whether a similar optimization asymmetry arises in physical embodied systems remains an important direction for future investigation. D.3 Qualitative Closed-Loop Rollout Analysis Aggregate task metrics summarize whether an interaction ultimately succeeds, but provide limited information about how the controller reaches that outcome. We therefore visualize representative closed-loop rollouts from both Minecraft and ViZDoom. These examples are intended as qualitative diagnostics rather than additional benchmark results. They expose behaviors that are compressed by aggregate success rates or episode rewards, including target search, sustained low-level control, persistent tracking, precise GUI manipulation, recovery from intermediate errors, survival-oriented navigation, and reactive combat. All examples are generated by the same low-level GameWAM policy used in the corresponding quantitative evaluations. The model receives the task instruction and repeatedly replans from newly realized observations; the displayed trajectories therefore reflect closed-loop execution rather than a pre-specified open-loop action sequence. D.3.1 Minecraft Closed-Loop Rollouts Local exploration for target acquisition. Fig. 9 visualizes the task “Mine the white bed.” The target object is not initially visible in the agent’s immediate observation. To complete the task, GameWAM first explores the surrounding local environment through native movement and viewpoint adjustment. The agent subsequently enters a nearby structure, where the white bed becomes visible, and then performs the required interaction. This rollout demonstrates that GameWAM can acquire targets that are not directly presented in the initial view by continuously coupling action selection with newly received visual feedback. We interpret this behavior as a form of closed-loop local exploration and target acquisition, rather than evidence of explicit search algorithms or symbolic planning. GameWAM does not maintain an object-level map or a predefined exploration policy; instead, exploratory behavior emerges from repeated native-action prediction conditioned on evolving visual observations. Figure 9: Closed-loop rollout for “Mine the white bed.” The white bed is not visible in the initial observation. GameWAM explores the nearby environment, enters a structure, acquires visual access to the target, and completes the mining interaction. The sequence illustrates visually guided local exploration and target acquisition under closed-loop interaction. Maintaining a continuous interaction over an extended duration. Fig. 10 shows “Collect obsidian for crafting.” Mining obsidian differs from a short attack or click interaction because the mining action must remain consistently applied for a comparatively long interval before the block breaks. A controller that frequently changes its action, loses alignment with the target, or prematurely interrupts the interaction will fail to obtain the block. GameWAM first approaches and aligns with the obsidian and then maintains the required mining behavior until the block is successfully broken. This rollout therefore highlights temporal consistency at the native-action level: repeated replanning does not necessarily fragment a behavior that must remain stable over many consecutive environment steps. Figure 10: Closed-loop rollout for “Collect obsidian for crafting.” Obtaining obsidian requires a sustained mining interaction rather than a brief action. After aligning with the target block, GameWAM maintains the mining behavior across successive replanning steps until the block breaks, illustrating temporal consistency of low-level control. Long-horizon pursuit of a resilient moving target. Fig. 11 shows the task “Hunt a horse.” We choose this example because horses combine two challenging properties for low-level control. Compared with common low-health animals such as cows, sheep, and chickens, horses require substantially more successful attacks before task completion. Meanwhile, the target continuously changes its relative position during the encounter. These two factors combine to create a longer interaction horizon, requiring the controller to repeatedly update camera orientation, movement, and attack behavior based on newly observed visual feedback rather than relying on a short reactive sequence. Across the rollout, GameWAM repeatedly reacquires the horse, adjusts its position and viewpoint, and continues attacking until task completion. The example illustrates persistent goal-directed behavior under a changing target state: the high-level objective remains stable while the native actions needed for completion continuously adapt throughout the interaction. This highlights GameWAM’s ability to maintain consistent behavior across multiple observation-action cycles. Figure 11: Closed-loop rollout for “Hunt a horse.” The horse requires repeated successful attacks before task completion and changes its relative position throughout the interaction. GameWAM repeatedly reacquires the target, adjusts movement and viewpoint, and continues the interaction until completion. The sequence highlights persistent closed-loop control over an extended moving-target interaction, where consistent task intent must be maintained across multiple observation–action cycles. Figure 12: Closed-loop rollout for “Kill the salmon.” GameWAM tracks and attacks a moving target in an underwater environment, where both visual appearance and locomotion differ from ordinary land-based gameplay. The rollout demonstrates successful closed-loop control under a qualitatively different gameplay condition. Control under underwater visual and motion dynamics. Fig. 12 provides a qualitatively different interaction condition through the task “Kill the salmon.” The entire engagement takes place underwater. Compared with ordinary terrestrial gameplay, the scene has different appearance, depth cues, target motion, and player locomotion. The target also moves through three-dimensional space rather than remaining approximately constrained to a ground plane. GameWAM remains able to orient toward the salmon, track its changing location, and execute the required attack sequence until completion. This example does not by itself establish broad robustness to arbitrary visual distribution shift, but it shows that successful native control is not restricted to the most common land-based interaction pattern represented by the other examples. Figure 13: Closed-loop rollout for “craft item purple concrete powder.” GameWAM progressively moves the required ingredients from the inventory into their corresponding crafting-grid locations and completes the target recipe. The sequence illustrates precise cursor control and structured multi-step placement through the native GUI action interface. Figure 14: Closed-loop rollout for “Create a dispenser in the crafting table.” GameWAM makes an intermediate placement error while constructing the recipe. After observing the resulting GUI state, subsequent actions adjust the crafting-grid configuration and complete the target item. The rollout illustrates error recovery enabled by repeated closed-loop observation and replanning. Precise spatial placement in the crafting grid. Gameplay and GUI interaction place substantially different demands on the same native keyboard–mouse interface. Gameplay primarily requires camera-centered movement and interaction, whereas crafting requires the cursor to manipulate small spatial targets under recipe-specific placement constraints. Fig. 13 shows “craft item purple concrete powder.” Successful execution requires multiple ingredient types to be moved from the inventory into specific locations in the crafting grid. The task therefore depends not only on selecting the correct objects, but also on controlling their relative positions and completing a multi-step placement sequence through continuous cursor motion and discrete mouse actions. The rollout shows GameWAM progressively constructing the required arrangement and obtaining the target output. This example highlights the role of mode-specific GUI action modeling: the same physical mouse channels used for camera control during gameplay must here support precise cursor displacement, selection, and placement inside a structured interface. Recovery from an intermediate placement error. Fig. 14 shows “Create a dispenser in the crafting table.” This trajectory is particularly informative because successful execution is not obtained through a perfectly correct initial sequence. During crafting, GameWAM makes an intermediate placement error that leaves the grid in an incorrect state. Subsequent observations expose that state to the controller, after which the policy modifies the placement and eventually constructs the valid recipe. The example illustrates a basic but important consequence of closed-loop GUI control. An intermediate low-level error need not irreversibly determine the remainder of the trajectory: because the model replans from the realized interface state, later actions can compensate for an earlier mistake. We refer to this behavior as closed-loop error recovery rather than high-level self-reflection, since no explicit symbolic diagnosis or textual reasoning process is assumed. D.3.2 ViZDoom Closed-Loop Rollouts The ViZDoom examples complement Minecraft by examining GameWAM under faster, more reactive combat dynamics. Unlike the Minecraft cases above, these tasks primarily reward continuous survival and combat effectiveness rather than completion of a discrete object-manipulation goal. The four maps also impose different spatial structures: the two Battle scenarios require locomotion and target engagement through navigable environments, whereas the two defense scenarios emphasize rapid target selection under constrained movement. Reactive combat and survival in Battle 1. Fig. 15 visualizes the instruction “Fight off the monsters and stay alive as long as you can.” The rollout alternates between navigation and combat as enemies enter the agent’s field of view. GameWAM changes orientation to bring nearby threats into the firing direction and engages them while continuing to move through the environment. This task differs from a single-target Minecraft combat interaction because there is no terminal target whose elimination completes the instruction. Instead, the controller must repeatedly react to newly encountered threats while preserving its ability to continue the episode. The sequence therefore illustrates sustained closed-loop combat in which movement, target acquisition, and firing are repeatedly recomputed from new observations. Figure 15: Closed-loop rollout on ViZDoom Battle 1 for “Fight off the monsters and stay alive as long as you can.” GameWAM navigates through the environment, reacts to newly visible enemies, adjusts its view toward threats, and repeatedly engages them while continuing the survival-oriented rollout. Figure 16: Closed-loop combat and health recovery on ViZDoom Battle 2 for “Fight off the monsters in the maze and stay alive as long as you can.” GameWAM navigates and fights through the maze. After taking damage, the policy approaches and collects a health pack, recovers health, and subsequently continues the survival-oriented combat rollout. The sequence illustrates adaptation to realized agent state in addition to reactive enemy engagement. Combat with health-aware recovery in Battle 2. Fig. 16 shows the more spatially complex instruction “Fight off the monsters in the maze and stay alive as long as you can.” Here the agent must combine combat with navigation through a maze-like environment, where walls and corridors repeatedly change which enemies and resources are visible. The rollout also exposes an important form of closed-loop adaptation that is not captured by the final episode reward alone. After taking damage during an engagement, GameWAM does not simply continue attacking until failure. It navigates toward an available health pack, collects it to recover health, and then continues through the environment and resumes combat. The sequence therefore demonstrates that the policy can condition its subsequent behavior on the realized consequences of earlier interaction, including changes in its own state. We interpret this behavior as state-sensitive closed-loop recovery rather than explicit resource planning. No symbolic survival planner is provided; the change in behavior occurs through repeated prediction from the updated visual and proprioceptive state. Maintaining a frontal defense line. Fig. 17 visualizes “Defend the line: shoot the monsters advancing toward you from across the room.” Unlike the Battle maps, the spatial objective is comparatively constrained: enemies approach primarily from the forward region, and successful behavior requires repeated target acquisition and firing as successive threats advance. GameWAM continually adjusts its horizontal aim between approaching enemies and engages targets before they reach the defended position. The rollout illustrates rapid visual–action feedback in a setting where the relevant target can change from one replanning step to the next and delayed correction would allow enemies to continue advancing. Multi-directional threat response in Defend the Center. Fig. 18 considers “Defend the center: shoot the enemies closing in from all directions.” The control challenge differs from Defend the Line because threats are not restricted to a single frontal direction. The policy must repeatedly change view orientation to locate and engage enemies appearing around the central position. Across the rollout, GameWAM redirects its aim between different parts of the arena and fires on enemies as they enter actionable views. This example therefore emphasizes repeated orientation changes and target switching rather than pursuit through the environment. Together with Defend the Line, it shows that the same learned controller can realize qualitatively different combat patterns under different language-conditioned objectives. Figure 17: Closed-loop rollout for “Defend the line: shoot the monsters advancing toward you from across the room.” GameWAM repeatedly acquires approaching enemies, adjusts horizontal aim, and fires as the active threat changes. The sequence highlights reactive closed-loop control under a frontal-defense objective. Qualitative interpretation. The Minecraft and ViZDoom rollouts expose complementary properties of the same world–action policy. Minecraft emphasizes longer-horizon interaction: searching for an initially non-visible object, maintaining a long mining action, pursuing moving entities, manipulating structured interfaces, and recovering from an intermediate GUI error. ViZDoom instead emphasizes rapid feedback under continuously changing combat states: repeated target acquisition, survival-oriented navigation, adaptation after damage, and switching between threats under different spatial constraints. These examples also illustrate why closed-loop replanning is important beyond aggregate task success. Useful behavior need not correspond to a single unchanging action pattern. Obsidian mining requires maintaining an action despite repeated replanning, horse hunting requires preserving task intent while continually changing movement and attack commands, the dispenser example requires correcting an earlier interaction, and Battle 2 requires changing behavior after the agent’s health state changes. In each case, subsequent actions are conditioned on realized environment feedback rather than on an unexecuted prediction of what should have happened. As with any qualitative visualization, these trajectories should not be interpreted as estimates of how frequently each behavior occurs. They are selected rollouts intended to expose interaction patterns that are difficult to convey through aggregate success rate or reward alone. Quantitative Minecraft and ViZDoom performance remains reported through the evaluation results in the main paper. Figure 18: Closed-loop rollout for “Defend the center: shoot the enemies closing in from all directions.” Threats can appear from different directions around the agent. GameWAM repeatedly redirects its view and switches targets as enemies approach, illustrating reactive multi-directional defense from a central position. D.4 Cross-Game Zero-Shot Transfer to VoxeLibre We further probe cross-game transfer by directly deploying the Minecraft-trained GameWAM checkpoint in VoxeLibre without any environment-specific training or fine-tuning. The final checkpoint after the second Minecraft training epoch is used unchanged. Model-side configurations, including the 224×224224× 224 observation resolution and temporal settings, remain identical to the Minecraft evaluation. We only apply a deterministic interface mapping between VoxeLibre keyboard–mouse controls and the native action representation expected by GameWAM. Because the GUI conventions of the two games differ substantially, this diagnostic focuses on basic gameplay interactions rather than interface tasks. We define six simple tasks spanning resource collection, combat, and block placement and provide each through a short natural-language instruction (e.g., “Chop down the oak log.”). Each task is evaluated over 20 episodes. The primary metric is success rate; average executed environment steps are reported over successful episodes. Table 7: Zero-shot cross-game transfer from Minecraft to VoxeLibre. The Minecraft-trained GameWAM checkpoint is evaluated without VoxeLibre training or fine-tuning. Each task is evaluated over 20 episodes; steps are averaged over successful episodes. Task Success Rate Steps / success Chop Tree 17/20 85.0% 61.8 Mine Stone 14/20 70.0% 41.9 Mine Iron Ore 10/20 50.0% 49.4 Kill Cow 2/20 10.0% 46.5 Kill Zombie 12/20 60.0% 112.6 Place Block 16/20 80.0% 67.9 Overall 71/120 59.2% – As shown in Table 7, GameWAM succeeds in 71 of 120 episodes, corresponding to a 59.2% micro-averaged success rate. Success rates vary substantially across tasks, ranging from 10.0% on Kill Cow to 85.0% on Chop Tree, with strong transfer also observed on Place Block (80.0%) and Mine Stone (70.0%). These results suggest that part of the learned visual–action control behavior transfers to a related but distinct game despite changes in rendering and interaction dynamics. This experiment is intended as a diagnostic of cross-game generalization rather than as a new benchmark or a claim of broad out-of-domain transfer. Minecraft and VoxeLibre share related voxel-world structure and basic gameplay semantics, and the evaluated tasks deliberately focus on low-level gameplay skills for which a deterministic native-control interface mapping is possible. D.5 Extended LASI Analysis We use the LASI notation, source variable, and DCT convention defined in Section 5.5. This section provides additional source-space controls, frequency-resolved analyses, generative-trajectory diagnostics, and closed-loop measurements underlying the main-text findings. D.5.1 Evaluation Setup and Frequency Decomposition The qualitative LASI failure pattern was observed across multiple GameWAM checkpoints spanning different training stages, hyperparameter settings, and training configurations. To keep the controlled intervention suite internally consistent, all quantitative diagnostics reported below are performed on the final checkpoint after the second Minecraft training epoch, used here as the representative checkpoint, so every comparison uses the same underlying policy. Controlled source interventions hold the conditioning context fixed and vary only the explicit action source. For analyses that resolve source sensitivity along the generative trajectory, we use a deterministic 20-step integration path, providing finer denoising-time resolution than the standard closed-loop sampling configuration. This diagnostic setting is used only for the LASI analysis and does not alter the reported task-performance evaluation. The controlled explicit-source matrix contains 24 fixed conditions and 16 action sources per condition (24×16=38424× 16=384 samples). Low-frequency donor swapping and low-frequency zeroing each use the same controlled matrix; the antithetic analysis uses 24×8=19224× 8=192 source pairs. Confidence intervals reported below use 10,000 condition-cluster bootstrap resamples where available. For each analyzed eight-step continuous camera chunk, we apply an orthonormal DCT along time. Modes q=0,1,2q=0,1,2 form the low-frequency band and modes q=3,…,7q=3,…,7 form the high-frequency band. P0–P2 and Y0–Y2 denote pitch and yaw DCT modes 0–2, respectively. D.5.2 Within-Condition Source Association For condition c, source j, action dimension d, and DCT mode q, define rc,d,q=Corrj⁡(Z~c,j,d,q,X~^0,c,j,d,q).r_c,d,q=Corr_j ( Z_c,j,d,q, X_0,c,j,d,q ). (36) The six low-frequency modes show substantial within-condition association. Averaging rc,d,qr_c,d,q across the fixed conditions, pitch DCT0–2 have mean correlations 0.7760.776, 0.6130.613, and 0.7460.746, while yaw DCT0–2 have 0.8900.890, 0.8240.824, and 0.7460.746. The yaw DCT0 95% confidence interval is [0.835,0.936][0.835,0.936]. Because the condition is fixed within each group, this analysis isolates source-associated variation, but correlation alone is not used as the causal claim. D.5.3 Donor, Zeroing, and Source-Space Controls For a base source ZbZ^b and donor ZdZ^d, the low-frequency intervention is Z~qswap=Z~qd,q∈0,1,2,Z~qb,q∈3,…,7. Z_q^swap= cases Z_q^d,&q∈\0,1,2\,\\ Z_q^b,&q∈\3,…,7\. cases (37) The donor-follow rate asks whether the intervened output moves closer to the corresponding donor output than to the base output. For yaw DCT0, the donor-follow rate is 94.8% (95% CI [92.2,97.1][92.2,97.1]%), and the correlation between donor-induced source and output changes is r=0.921r=0.921. For low-frequency zeroing, Z~q=0 Z_q=0 for q∈0,1,2q∈\0,1,2\. Writing NcN_c and NsN_s for the numbers of fixed conditions and sources, respectively, define the condition-averaged within-condition source variances Vd,qbase=1Nc∑c=1NcVarj⁡[X~^0,c,j,d,qbase],Vd,qzero=1Nc∑c=1NcVarj⁡[X~^0,c,j,d,qzero].V_d,q^base= 1N_c _c=1^N_cVar_j [ X^base_0,c,j,d,q ], V_d,q^zero= 1N_c _c=1^N_cVar_j [ X^zero_0,c,j,d,q ]. (38) The fraction of source-induced variance removed is Rd,qzero=1−Vd,qzeroVd,qbase.R_d,q^zero=1- V_d,q^zeroV_d,q^base. (39) Yaw DCT0 loses 99.25% of its source-induced variance (95% CI [98.93,99.50][98.93,99.50]%). The corresponding reductions for pitch DCT0–2 are 95.95%, 91.44%, and 93.21%; for yaw DCT1–2 they are 98.45% and 96.99%. To compare variation across conditions with variation induced by the source, we additionally define Rd,qcond/src=Varc⁡[1Ns∑j=1NsX~^0,c,j,d,qbase]Vd,qbase.R_d,q^cond/src= Var_c [ 1N_s _j=1^N_s X^base_0,c,j,d,q ]V_d,q^base. (40) We also use antithetic source pairs Z and −Z-Z: X^odd(Z)=12[X^(Z)−X^(−Z)],X^even(Z)=12[X^(Z)+X^(−Z)]. X_odd(Z)= 12[ X(Z)- X(-Z)], X_even(Z)= 12[ X(Z)+ X(-Z)]. (41) Let ±J_± denote the subset of base sources paired with their negatives. The antithetic cancellation ratio for the corresponding DCT coefficient is Rd,qanti=1−1Nc∑cVarj∈±⁡[X~^even,c,j,d,q]1Nc∑cVarj∈±⁡[X~^0,c,j,d,qbase].R_d,q^anti=1- 1N_c _cVar_j _± [ X_even,c,j,d,q ] 1N_c _cVar_j _± [ X^base_0,c,j,d,q ]. (42) Fig. 19 summarizes the source-space controls. The condition/source variance ratio should not be interpreted as source dominance in every mode: pitch DCT0 is 1.062, slightly above one, whereas the remaining reported ratios are 0.940, 0.329, 0.361, 0.239, and 0.086. Figure 19: LASI source-space controls. (a) Correlation between the change imposed on a low-frequency source coefficient and the corresponding change in the predicted clean-action coefficient. (b) Condition-induced variance divided by source-induced variance; values below one indicate larger source variation for the measured mode. (c) Antithetic cancellation ratio for paired Z and −Z-Z sources, measuring source-odd structure. P0–P2 and Y0–Y2 denote pitch and yaw DCT modes 0–2. D.5.4 Frequency Selectivity We compare three unit-norm perturbation directions: a low-frequency direction aligned with the leading principal direction of clean camera actions in DCT modes 0–2, an orthogonal direction within the same low-frequency subspace, and a high-frequency direction in modes 3–7. Thus, differences in response are not attributable to perturbation magnitude. For a fixed condition c and perturbation direction q, write X^0(x,σ,c) X_0(x;σ,c) for the reconstructed clean-action estimate obtained from noisy action state x at noise level σ. We estimate the local response by the centered finite difference Jσ,c(q)≈X^0(Xσ,c+ϵq,σ,c)−X^0(Xσ,c−ϵq,σ,c)2ϵ.J_σ,c(q)≈ X_0(X_σ,c+ε q;σ,c)- X_0(X_σ,c-ε q;σ,c)2ε. (43) We measure the induced low-frequency response magnitude as ρσ,c(q)=‖PlowJσ,c(q)‖2, _σ,c(q)= \|P_lowJ_σ,c(q) \|_2, (44) where PlowP_low projects the continuous camera trajectory onto DCT modes 0–2. The reported comparison uses ϵ=0.05ε=0.05. The primary frequency-selectivity statistic compares the aligned low-frequency perturbation with the high-frequency perturbation through the condition-paired log-response difference Δσlow/high=1Nc∑c=1Nc[log⁡ρσ,c(qlow)−log⁡ρσ,c(qhigh)],Gσlow/high=exp⁡(Δσlow/high). _σ^low/high= 1N_c _c=1^N_c [ _σ,c(q_low)- _σ,c(q_high) ], G_σ^low/high= \! ( _σ^low/high ). (45) Across the tested σ∈0.20,0.40,0.55,0.68,0.80σ∈\0.20,0.40,0.55,0.68,0.80\, the geometric low/high response ratios are 4.35, 4.20, 3.53, 4.58, and 4.43. The aligned low-frequency direction is also compared with the orthogonal low-frequency direction; this difference is substantially smaller and depends on noise level. The supported conclusion is therefore strong selectivity between the tested low- and high-frequency perturbations, while evidence for additional selectivity among directions within the low-frequency subspace is weaker and noise-dependent. D.5.5 Single-Forward Source Transfer To separate one-call source transfer from iterative path effects, we directly construct Xσana=(1−σ)X0+σZX_σ^ana=(1-σ)X_0+σ Z (46) and evaluate the denoiser once at that fixed noise level, without taking an integration step. To remove between-condition variation, source transfer is measured from within-condition source-pair differences. For two sources j and k under the same condition, ΔZ~c,jk,d,q=Z~c,k,d,q−Z~c,j,d,q,ΔX~^0,c,jk,d,qana=X~^0,c,k,d,qana−X~^0,c,j,d,qana. Z_c,jk,d,q= Z_c,k,d,q- Z_c,j,d,q, X_0,c,jk,d,q^ana= X_0,c,k,d,q^ana- X_0,c,j,d,q^ana. (47) The source-to-output gain for a DCT coefficient is the regression slope gana(σ)=Covc,j<k⁡(ΔZ~c,jk,d,q,ΔX~^0,c,jk,d,qana)Varc,j<k⁡(ΔZ~c,jk,d,q).g_ana(σ)= Cov_c,j<k ( Z_c,jk,d,q, X_0,c,jk,d,q^ana )Var_c,j<k ( Z_c,jk,d,q ). (48) Fig. 20 shows that source transfer is already present before iterative denoising. For yaw DCT0, the gain peaks at 0.709 at σ=0.68σ=0.68 in this sweep; the corresponding pairwise Pearson correlation increases from 0.480 at σ=1σ=1 to 0.853 at σ=0.2σ=0.2, while sign agreement is approximately 0.87–0.91 over most measured noise levels after the initial point. Figure 20: Single-forward LASI across analytic noise levels. (a) Regression gain from within-condition yaw-DCT0 source differences to the corresponding reconstructed clean-action differences. (b) Pearson correlation between the same source and reconstructed action differences. (c) Sign agreement of the paired differences. Each point uses one denoiser evaluation at an analytically constructed noisy state, with no integration update before measurement. D.5.6 Model-Path Amplification For a state reached by the iterative model path at the same numeric noise level, define gpath(σ)g_path(σ) analogously using within-condition source-pair differences. The absolute and relative amplification statistics are A⁡(σ)=gpath(σ)−gana(σ),Rg(σ)=gpath(σ)|gana(σ)|.A(σ)=g_path(σ)-g_ana(σ), R_g(σ)= g_path(σ)|g_ana(σ)|. (49) Fig. 21 combines the frequency-selectivity result with the matched-σ analytic/model-path comparison. At early denoising steps, the two gains are nearly identical. A positive gap appears in the mid/low-noise region; at step 19 (σ=0.208σ=0.208), yaw DCT0 has analytic gain 0.162 and model-path gain 1.117, giving a 6.897× ratio. Figure 21: LASI frequency selectivity and model-path amplification. (a) Geometric response ratio of aligned low-frequency versus high-frequency perturbations over tested noise levels. (b) Yaw-DCT0 source gain for an analytically constructed noisy state and for the state reached along the iterative model path at the matched σ shown in parentheses. (c) Model-path gain divided by the magnitude of the analytic gain; the endpoint ratio reaches 6.897× at step 19 (σ=0.208σ=0.208). Fig. 22 reports the same comparison as an absolute difference with uncertainty. The measured gap is essentially zero at steps 0–9, becomes positive at step 14, and increases to 0.955 at step 19. Figure 22: LASI model-path amplification gap. Difference gpath−ganag_path-g_ana for yaw DCT0 at matched noise levels. Error bars are 95% confidence intervals; the gap is approximately zero early in the trajectory and increases to 0.955 at the final measured step. D.5.7 Closed-Loop Alignment The closed-loop analysis contains 300 traces spanning 30 tasks and 16,707 valid replanning steps. Let ze,t,mz_e,t,m and ye,t,my_e,t,m denote the source and executed-action DCT coefficients for episode e, replanning step t, and low-frequency camera mode m∈ℳ=P0,P1,P2,Y0,Y1,Y2m =\P0,P1,P2,Y0,Y1,Y2\. To remove episode-level offsets, let z¯e,m z_e,m and y¯e,m y_e,m denote the corresponding within-episode means and define Align⁡(z,y)=1|ℳ|∑m∈ℳCorre,t⁡(ze,t,m−z¯e,m,ye,t,m−y¯e,m).Align(z,y)= 1|M| _m Corr_e,t (z_e,t,m- z_e,m,y_e,t,m- y_e,m ). (50) To test whether the action source remains aligned with later behavior rather than merely reflecting a generic trajectory statistic, we circularly shift the source schedule relative to the rollout. For the 75-position replan schedule, define S⁡(δ)=Align⁡(ze,(t+δ)mod75,m,ye,t,m).S(δ)=Align (\z_e,(t+δ) 75,m\,\y_e,t,m\ ). (51) The unshifted statistic is 0.33763 and ranks first among all 75 circular shifts. As a separate permutation control, we randomly permute the source schedule B=999B=999 times. No permuted statistic exceeds the observed value, and the maximum permutation-null statistic is 0.06466. The corresponding finite-sample permutation value is p=1+∑b=1B[Sbperm≥Sobs]B+1=0.001.p= 1+ _b=1^B1[S_b^perm≥ S_obs]B+1=0.001. (52) For the 197 complete 75-replan episodes, we additionally average each action DCT coefficient across episodes at every replan position and correlate the resulting mean trajectory with the explicit source schedule. The correlations are 0.800,0.738,0.7270.800,0.738,0.727 for pitch DCT0–2 and 0.803,0.808,0.6390.803,0.808,0.639 for yaw DCT0–2. Figure 23: Closed-loop LASI alignment. (a) Episode-adjusted low-frequency association at the correct source/replan alignment compared with the maximum permutation-null statistic; the correct alignment additionally ranks first among all 75 circular shifts. (b) Correlation between the explicit source schedule and the cross-episode mean trajectory for complete 75-replan episodes, shown for pitch and yaw DCT modes 0–2. D.5.8 Closed-Loop Accumulation and Source Resampling The practical failure motivating LASI was most apparent when a single sampled action source was reused across successive replanning steps. Closed-loop behavior was strongly dependent on the reused source realization: for some sources, a persistent directional bias accumulated into repeated in-place rotation, and in severe cases almost no tasks were completed. Resampling the action source between replanning steps largely removed this episode-level failure pattern. To characterize why repeated-source reuse can produce such behavior, for a coarse camera quantity Δc _c at replanning step c, consider the decomposition Δc=μ⁡(c)+b⁡(Zc)+εc, _c=μ(C_c)+b(Z_c)+ _c, (53) where μ⁡(c)μ(C_c) collects condition-dependent behavior and b⁡(Zc)b(Z_c) is a source-dependent low-frequency bias. This decomposition is explanatory rather than a claim that the measured trajectory is fully described by these terms. If one source Z is reused for K cycles, the coherent component contributes ∑c=1Kb⁡(Z)=Kb(Z), _c=1^Kb(Z)=K\,b(Z), (54) so a turning bias can be repeatedly reinforced. By contrast, if sources are resampled independently and b⁡(Zc)b(Z_c) is approximately centered, Zc∼i.i.d.(0,I),⁡[∑c=1Kb⁡(Zc)]≈0,Z_c i.i.d. N(0,I), [ _c=1^Kb(Z_c) ]≈ 0, (55) and Var⁡[∑c=1Kb⁡(Zc)]=KVar⁡[b⁡(Zc)].Var [ _c=1^Kb(Z_c) ]=KVar[b(Z_c)]. (56) Thus, source resampling removes the coherent episode-level reuse of a single source bias even though stochastic action variation remains. Consistent with this analysis, our closed-loop evaluation resamples the action source at every replanning step. We treat this as an evaluation-time mitigation of coherent accumulation rather than a mechanism-level correction of the underlying LASI sensitivity. Appendix E Scope, Limitations, and Interpretation Boundaries E.1 Method and Evaluation Scope GameWAM is designed as a low-level closed-loop controller rather than a standalone high-level planner. Its policy is conditioned on current visual, language, historical, and proprioceptive context, but it does not explicitly maintain a symbolic task graph, inventory plan, recipe representation, or long-horizon search process. Failures that require semantic decomposition, exploration, or multi-stage resource planning therefore remain outside the intended scope of the block-cycle controller. A natural extension is to pair GameWAM with a higher-level planner while retaining GameWAM as the native keyboard–mouse policy. This distinction is particularly relevant to GUI interaction. Crafting, smelting, and inventory manipulation combine low-level cursor and keyboard control with structured procedural knowledge. Higher-level agents can externalize such procedures through symbolic actions, explicit plans, or dedicated executors, whereas GameWAM must realize them through the same low-level native-action policy. The scripted GUI data provides procedural supervision for these interactions, but the current model does not separately represent recipes or explicitly plan multi-step GUI procedures. Future systems could combine GameWAM with structured procedural representations or a higher-level planner while preserving closed-loop visual feedback at the execution level. The current persistent history is also predominantly visual. Its auxiliary objective encourages the compressed cross-cycle representation to remain predictive of the current clean visual feature, but it does not directly supervise which historical information should be retrieved or used for downstream action generation and task completion. Future work could introduce task- or action-aware history objectives, explicit retrieval supervision, cross-cycle credit assignment, or multimodal memory that jointly represents visual observations, actions, inventory state, and higher-level task context. Our empirical evaluation is restricted to digital game environments. Minecraft and ViZDoom provide diverse visual dynamics and native action spaces, but they do not establish transfer to physical control, where sensing delay, actuation noise, safety constraints, and contact dynamics differ substantially. The Minecraft results also depend on the available VPT/MineStudio data mixture; the current experiments do not isolate every possible dataset-composition factor. More broadly, current Minecraft Computer Use (MCU) training and evaluation settings largely focus on atomic task completion and individual interaction events. While useful for measuring specific capabilities, they provide limited supervision and evaluation for temporally extended behaviors where intermediate decisions influence future states and actions. Constructing high-quality chain-of- interaction trajectories with coherent intent, evolving state context, and aligned native actions remains challenging, and represents an important direction for future game-agent datasets and evaluations. E.2 Interpretation of the LASI Diagnostics The combined LASI evidence supports four specific statements. First, an explicit intervention on low-frequency source coefficients causally changes the matching low-frequency action output under fixed conditioning. Second, the effect is substantially stronger at low than at high temporal frequencies. Third, source transfer is already measurable in a single analytic forward pass. Fourth, the iterative model path amplifies the effect later along the generative trajectory, and a statistically aligned component remains detectable in closed-loop trajectories. These results remain diagnostic rather than a complete mechanistic account. They do not identify a unique internal layer or training-time pathway, imply that conditioning information is absent, or establish that source variation explains the complete closed-loop trajectory. Empirically, the clearest LASI effect appears in continuous camera control. A plausible contributor is the pronounced low-frequency structure of camera trajectories in the training data: meaningful camera motion often evolves smoothly over short horizons, potentially making low-frequency source structure especially easy to align with and propagate through the generated action trajectory. This provides a possible data-level explanation for the concentration of the observed effect in camera motion. Resampling the action source across replanning steps breaks repeated exposure to the same source bias and largely removes its harmful coherent accumulation in closed-loop interaction, but it does not remove the underlying source sensitivity. We therefore also explored training-time interventions aimed at the formation of LASI itself. These included frequency-selective objectives that directly discourage low-frequency source–action coupling, consistency objectives that reduce dependence on particular source realizations under the same conditioning, and additional supervision that strengthens the dependence of generated actions on conditioning. Across these directions, stronger suppression could interfere with normal action learning or degrade closed-loop performance, whereas other variants preserved task behavior but left substantial low-frequency source dependence. None therefore provided a robust mechanism-level correction without an accompanying trade-off. We consequently use source resampling as an effective deployment-time mitigation while treating mechanism-level removal of LASI as an open problem. A desirable solution should prevent stochastic source structure from becoming a persistent low-frequency control bias without suppressing legitimate smooth motion, condition-compatible action diversity, or closed-loop responsiveness. Developing such objectives remains an important direction for generative action policies.

Instant research discovery