Paper deep dive

Decoupling Planning and Control for Instructable Agents

Zineng Tang, Kelsey R. Allen, Sjoerd van Steenkiste, Ishita Dasgupta, Alane Suhr

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 82

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/28/2026, 4:12:32 AM

Summary

The paper introduces 'Instruct-to-Act', a framework that decouples high-level planning from low-level control for embodied agents. It combines pre-trained Vision-Language Models (VLMs) as planners, which generate sparse, high-level text instructions, with a lightweight, environment-specific world-model controller that executes these instructions at high frequency. The controller is trained using post-hoc instruction supervision on synthetic labels derived from VLM summaries of policy rollouts, optimizing for behavior cloning, reward maximization, and world modeling. This approach allows for plug-and-play integration of various VLM planners without fine-tuning them to specific action spaces, enabling efficient multi-agent coordination through language while maintaining low-latency control.

Entities (10)

Instruct-to-Act · methodframework · 98%VLM Planner · model-component · 95%World-Model Controller · model-component · 95%GPT-4o · modelplanner · 92%Atari · environment · 90%Behavior Cloning · training-objective · 90%DreamerV3 · baselinearchitecture · 90%MineRL · environment · 90%Pico Park · environment · 90%RSSM · architecture · 90%

Relation Signals (9)

Instruct-to-Act → uses → World-Model Controller

confidence 95% · Instruct-to-Act... where we train a world-model controller to act autonomously at high frequency

Instruct-to-Act → uses → VLM Planner

confidence 95% · Instruct-to-Act... where we train a world-model controller to act autonomously... conditioned on... text instructions generated by a VLM planner.

GPT-4o → usedas → VLM Planner

confidence 92% · Using GPT-4o to generate sparse instructions executed by a trained controller... improves performance

Instruct-to-Act → evaluatedon → Pico Park

confidence 90% · We evaluate our proposed approach across seven embodied environments, including... Pico Park

Instruct-to-Act → evaluatedon → Atari

confidence 90% · We evaluate our proposed approach across seven embodied environments, including... Atari

Instruct-to-Act → evaluatedon → MineRL

confidence 90% · We evaluate our proposed approach across seven embodied environments, including... MineRL ObtainDiamond

World-Model Controller → isbasedon → RSSM

confidence 90% · The controller π c is an instruction-conditioned RSSM

World-Model Controller → isbasedon → DreamerV3

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,027 characters extracted from source content.

Expand or collapse full text

Published as a conference paper at COLM 2026 Decoupling Planning and Control for Instructable Agents Zineng Tang terran@berkeley.edu UC Berkeley Kelsey R. Allen ∗ krallen@cs.ubc.ca UBC Sjoerd van Steenkiste svansteenkiste@google.com Google DeepMind Ishita Dasgupta idg@google.com Google DeepMind Alane Suhr suhr@berkeley.edu UC Berkeley Abstract Recent work shows that pre-trained, instruction-tuned vision-language mod- els (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world- model controllers excel at fast observation-to-action control, but lack open- ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act au- tonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train con- trollers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling ob- jectives. We evaluate our proposed approach across seven embodied envi- ronments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actua- tors. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks. 1 1 Introduction Embodied agents increasingly operate in settings that demand both high-frequency, low-latency continuous control and intermittent, high-level reasoning and planning. For example, consider a long-horizon, multi-player game like Minecraft, where agents can communicate with one another to achieve shared goals. There have been two dominant approaches in building such agents. Learning world models through reinforcement learning (RL) has demonstrated that compact modeling of latent environment dynamics can support sample-efficient, low-level control from pixels (Hafner et al., 2020; 2025). In parallel, instruction-following large language and vision-language(-action) (LLM/VLM/VLA) models (Zitkovich et al., 2023; Black et al., 2025; Kim et al., 2024; Bjorck et al., 2025) can synthesize useful decompositions, subgoals, and plans from high-level task descriptions and visual context. Yet these two capabilities are typically available in isolation. VLM-only agents plan well, but struggle to realize plans as precise action streams under tight real-time budgets. Moreover, because VLM outputs are text tokens rather than time-critical action streams, eval- uating them as embodied agents across diverse domains is challenging. Low-level controllers act smoothly and quickly but are difficult to steer with abstract, open-ended instructions. ∗ Work done at Google DeepMind. 1 Code and a demo for our project is available at https://zinengtang.github.io/speak-to-act/. 1 arXiv:2608.26788v1 [cs.AI] 27 Aug 2026 Published as a conference paper at COLM 2026 Figure 1: Illustration of inference with a VLM planner and trained controller. In this work, we study the problem of grounding language into policy for real-time control coupled with high-level abstraction. We propose Instruct-to-Act, a plug-and-play paradigm that decomposes the problem of embodied decision-making into planning and control. In this paradigm, a pre-trained VLM planner maps environment observations to plans and instruc- tions; these instructions are sent to a controller, which processes a stream of observations and produces low-level actions in real time. The VLM planner plans and issues instructions con- currently with low-level execution, without the need for finetuning. Our core contributions center around a framework for efficiently training environment-specific controllers that can be paired with any planner to support real-time planning and control. Each controller is built as a language-aware world model and policy, where both representation learning (on the world model) and control (actor and critic heads) are conditioned on a latent representation of an instruction sent by a planner. In contrast with single-task models like Dreamer (Hafner et al., 2020; 2025), where a policy is learned indirectly through rewards, our setting allows latent imagination and value learning to incorporate a wide range of abstract goals specified using a language instruction. To train controllers to be conditioned on instructions, we perform post- hoc instruction supervision, summarizing replay segments into language instructions with a VLM and using them to perform behavior cloning during training. Finally, during inference, we perform asynchronous instruction: the planner continuously reasons and plans in text space while the controller simultaneously executes low-level actions in the environment. Because planning is decoupled from acting, inference scales well in both performance and efficiency: any number of language planners can each be coupled with identical controller instances, enabling efficient multi-agent communication in real time. This design also lets us evaluate both (a) how well individual VLMs act as embodied single agents and (b) how well they reason about, coordinate with, and communicate with other agents under real-time constraints. We organize evaluation into two parts. First, we verify that without instructions, trained controllers match the performance of standard world-model RL baselines, and that they achieve an average instruction-following accuracy of 92.8% across instructions generated from a diverse set of VLM planners. Then, we use the controllers to evaluate VLM planners’ ability to (a) act as agents across diverse single-player embodied environments by mapping high-level task descriptions to mid-level instructions, and (b) plan and coordinate with other VLM planners in multi-agent tasks. Pairing VLM planners with our trained controllers performs significantly better than mapping directly from high-level task to low-level actions, in both task score and inference efficiency, achieving competitive performance in 6 of the 7 environments; overall, decoupling planning and control supports strong task performance, low latency, and convenient modularity for evaluating VLMs as embodied agents in arbitrary heterogeneous environments. In contrast to prior work (Section 5), no VLM is fine-tuned to an environment’s low-level action space—only the lightweight controller is environment-specific; the controller, unlike pure world-model RL agents, is explicitly trained to execute open-ended language instructions at control frequency; and the stable language interface supports asynchronous planner swaps and direct multi-agent coordination with a shared controller architecture. 2 Published as a conference paper at COLM 2026 2 Overview and Experimental Setup We study embodied agents that map observations to low-level actions, optionally conditioned on natural-language task specifications. Our goal is to adapt existing vision-language models (VLMs) into embodied, language-using agents out-of-the-box, without fine-tuning them to each environment’s low-level action space. At inference time, we decompose decision making into a planner and controller. The planner maps observations (and optional task context) to a natural-language instruction, and the controller maps the instruction and visual observations to low-level actions. This contrasts with approaches that require end-to-end low-level action generation by fine-tuned VLM/VLA models (Driess et al., 2023; Zitkovich et al., 2023; Black et al., 2025; Kim et al., 2024; Bjorck et al., 2025; Hafner et al., 2020; 2025), and follows language-conditioned control paradigms in robotics (Ahn et al., 2022); it allows us to plug in any instruction-tuned VLM as a planner with a lightweight environment-specific controller. Task. We consider partially-observable environments(S,A,Ω,O,P,R,γ)with state space S, action spaceA, observation spaceΩ, observation kernelO :S→∆ Ω , transition kernel P :S×A→∆ S , reward functionR, and discount factorγ. Pretrained VLMs struggle when Ais high-frequency and environment-specific (e.g., repeated camera/control commands in Minecraft or Atari), making direct low-level action sequence emission impractical. In our target domains, the challenge is not only choosing the right subgoal, but sustaining a long, low-level action stream at control rate. At each environment step, an agent policyπ :Ω →∆ A maps from observations to a distribution over actions. We study agents which are compositions of independent planner and controller agents, respectivelyπ p :Ω→∆ X andπ c :Ω×X →∆ A , whereXis the set of all possible natural language utterances. As illustrated in Fig. 1, this planner–controller split is closely related to hierarchical RL and the options framework, where higher levels issue temporally extended commands (Dayan & Hinton, 1992; Sutton et al., 1999; Bacon et al., 2017; Nachum et al., 2018; Levy et al., 2019). We sample actions fromπgiven an observationo by first sampling an instructionx∼ π p (·| o), then sampling a sequence of low-level actionsa fromπ c , wherea i ∼ π c (·| o i ,x). We augment the action space with an instruction-completion action; when selected, a new instruction is sampled fromπ p . Controllers need not be con- ditioned on an instruction; when none is available, the controller can continue autonomously between planner updates. In experiments,π p is an instruction-tuned VLM andπ c is a lightweight planner-agnostic environment-specific recurrent state space model (RSSM). Controller.The controllerπ c is an instruction-conditioned RSSM (Hafner et al., 2025) with an additional head predicting instruction completion (Sec. 3.2). It is trained by relabeling segments of its own on-policy rollouts with VLM-generated instructions and adding a behavior-cloning objective to the standard Dreamer world-model, actor, and value losses (Sec. 3.4). No expert demonstrations are used. Evaluation. We evaluate on seven environments spanning classic single-agent RL bench- marks (e.g., Atari, Bellemare et al., 2013) and multi-agent tasks based on real cooperative video games (e.g., Pico Park); tasks, metrics, and analyses are detailed in Section 4.2. 3 Method 3.1 Decoupling Planning and Control in Inference Our agent policyπcomposes a plannerπ p and a controllerπ c (Section 2). In practice, we run inference onπ p andπ c in independent parallel processes, which exploitsπ p ’s capabilities beyond high-level decision-making—reasoning and communication with other agents—while low-level actions execute in the environment, and extends naturally to multi-agent settings by spawning planner and controller threads per agent. 3 Published as a conference paper at COLM 2026 Asynchronous Online Inference. Trading off inference betweenπ p andπ c , whereπ p must wait forπ c to complete its current instruction before it starts to plan a new one, wastes wall-clock time. Instead, we allow plannersπ p to run inference asynchronously in the background, given a live observation stream, continuously updating their reasoning, illustrated by Alg. 1 (pseudocode in Appendix C). Upon receiving a message that the controller has completed its current instruction, planners are prompted to immediately emit the next instruction from a draft plan. Given a draft plan, emitting an instruction requires generating only a few tokens. This simultaneous inference supports both the relatively slow language reasoning of VLMs beneficial to generating optimal instructions, and the fast low-level execution provided by the controller. In the offline variant (Alg 2), by contrast, the controller waits until the planner finishes producing a new instruction, so throughput directly inherits VLM latency at every instruction boundary. Multi-agent Inference. We extend the framework to cooperative tasks withnagents n π (i) = π (i) p ,π c o n i=1 . 2 Each agent shares controller parameters, but at inference time, these controllers operate on different observation streams, maintain different latent states, and execute different actions. Planners need not be the same type of agent. Unlike classic centralized-training/decentralized-execution (CTDE, Kraemer & Banerjee, 2016), we do not train a separate centralized critic or impose structured role labels. Instead, coordination happens purely in language, with planners specifying and negotiating high-level goals. Agents communicate in discrete rounds aligned with the control step. At each step, the agents can (i) receive messages from the shared chatroomChatroom[i] t ; (i) instruct the controller to act; (i) optionally speak. The chatroomChatroom[i]concatenates all peer messages at any given step. All agents share the same planner system prompt, which instructs them to process incoming messages and reply as if in a dialogue. 3 Qualitatively, agents use messages to claim subgoals, negotiate roles early in an episode, and report bottlenecks when progress stalls. Controllers do not process messages sent between planner agents. 3.2 Controller Architecture Our controller is based on the recurrent state space model (RSSM; Hafner et al., 2019; 2020), which maintains a latent states t computed from previous observationso <t and actionsa <t , with decoder heads estimating the next observation, reward, and action. Following Hafner et al. (2025), we train the observation decoder with a KL-regularized reconstruction objective and optimize Dreamer-style actor and value heads on imagined rollouts. 4 We adapt this base architecture to optionally condition on a planner instructionx t : a frozen, pretrained contrastive language encoder mapsx t toe t ∈R d ℓ , which is projected and concatenated with the RSSM features used by the policy and value heads, yielding π(a| s t ,e t )= Softmax ℓ(s t ,e t )/τ a , ℓ(s t ,e t )∈R |A| . When no instruction is available,e t is a learned null embedding, so the controller can act autonomously. We additionally augment the output space of the controller to include an indicator of whether an instruction has been completed. At each stept, the actor emitsp stop t = σ g(s t ,e t ) where g is a small head on top of the latent state and σ is the logistic function. 3.3 Planner Our framework supports any plannerπ p :Ω →∆ X that maps from observations to instructions, where observations can include visual input and, for multi-task environ- ments, an optional natural-language task specification. We implement planners using vision-language models (VLMs), prompted to additionally maintain, during inference, a 2 We include more details on the multi-agent setup in Appendix C. 3 Appendix A.11 contains all system prompts. 4 Full equations and losses can be seen in Appendix A.1. 4 Published as a conference paper at COLM 2026 memory bankMrand a partial planPn. Concretely, at inference stept ′ , the VLM maps (S p,Mr t ′ −1 ,Pn t ′ −1 ,O <t ′ ,p stop t ′ −1 ) —system prompt, memory bank, partial plan, observation history, and the controller’s most recent instruction-completion probability—to an updated (Mr t ′ ,Pn t ′ )and, ifp stop t ′ −1 indicates completion, an instructionxsent to the controllerπ c . Crucially, as discussed in Sec. 3.1, the VLM stepst ′ need not be synchronous with the controller’s timestepst, supporting online planning of the next instruction while the agent executes its current one; in the synchronous variant, each instruction is generated only after the previous one completes, so t ′ aligns with instruction boundaries. In multi-agent settings, planners additionally communicate in one of two modes: decentralized, where at each VLM inference step any planner may initiate at most one message to a peer, or centralized, where a fixed hub agent sends messages to all other agents (formal definitions in Appendix D.1). Relative to the single-agent setting, the planner input is augmented with the agent’s inbox and recent outgoing messages, and the planner output can additionally include a short message for another agent. 3.4 Training We train the controller parameters by alternating between collecting rollouts conditioned on the current policy, and optimizing the policy using both reward, world model, and instruction-following behavior-cloning objectives. The losses combine two objectives: (a) instruction following, via behavior cloning on on-policy action sequences post-hoc labeled with VLM-generated instructions; and (b) reward maximization and world modeling, which, following the Dreamer approach (Hafner et al., 2025), align the labeled behavior distribution with what the controller executes at inference time. Behavior cloning with post-hoc instruction annotation. During learning, we use a FIFO replay buffer (Mnih et al., 2015)D(capacity|D|=1024) that stores tuples τ t ≡(o t ,a t ,r t ,e t ,complete t ,done t ), wheree t is an optional instruction embedding,complete t indicates whethera t completed the instruction, anddone t marks episode termination. To acquiree t , we annotate the buffer online: we sample a set of non-overlapping, variable-length intervalsI = [u k ,v k ] K k=1 and, for each interval, prompt a VLM summarizer with task-specific prompts, equally spaced frames, and the packed action-sequence text to produce a high-level instructionx. We encodexintoe(Section 3.2), pair all tuplest∈ [u k ,v k ]withe, and set complete t =1 only at t=v k . 5 During training, we treata u k :v k as a label for the instructionxconditioned on observations o u k :v k , and optimize a behavior cloning loss over labeled intervals: L BC =−λ BC ∑ [u k ,v k ]∈I v k ∑ t=u k logπ a t | s t ,e t . To learn to mark instructions as complete, we also apply a binary cross-entropy loss onp sto p t using the label complete t : L stop =− ∑ t complete t log p stop t + ( 1−complete t ) log 1− p stop t . We annotate intervals covering 50% of the buffer, leaving the remaining unannotated so that the controller also learns to act autonomously from the environment rewardR. Annotation reuses logged trajectories and runs in parallel with optimization, adding roughly 17% GPU-hours and a 12% training slowdown (Section 4). When no instruction is provided, we use the learned null embedding from Section 3.2. We (optionally) support a strict instruction-required mode that architecturally guarantees no actions are executed without a valid, non-completed instruction. 5 Further segment sampling details and per-timestep bookkeeping can be referred in Appendix D.1. 5 Published as a conference paper at COLM 2026 Final Training Objective.The overall objective augments the base objective with behavior cloning: L =L model + λ V L value + λ A L actor +L BC +L stop whereL model ,L value , andL actor are defined by RSSM training algorithm (Hafner et al., 2025). 4 Experiments 4.1 Architecture and Training Our controller is a language-conditioned DreamerV3-style RSSM (Hafner et al., 2025) with CLIP-base-224 (Radford et al., 2021) and DINOv2-base (Oquab et al., 2024) visual features, and MiniLM-L6-H384-uncased (Wang et al., 2020) language embeddings. Main results use the 800M controller variant. During data collection, rollouts are generated with empty language (e t = 0), then post-hoc instruction annotation labels random replay segments (length 1 to 20) while unlabeled gaps carry no behavior-cloning loss. GPT-4o is the annotator and 50% of replay segments are annotated. 6 4.2 Evaluation Each evaluation consists of 100 episodes (or the task’s standard protocol), with no exploration noise (ε=0) and temperatureτ=0.8 for stochastic policies. We follow each environment’s official protocol and metric: Atari (Kaiser et al., 2020) (sticky actions, no-op starts), MineRL ObtainDiamond (Guss et al., 2019), Crafter (Hafner, 2022), DMLab explore_goal_locations(Beattie et al., 2016), Overcooked (Carroll et al., 2019) (scripted partners for single-agent tests, cross-play for multi-agent), Pico Park (cooperative puzzle completion with synchronized actions), 7 and MindCraft (White et al., 2025) (cooking/construction/crafting success). 8 4.3 VLMs as Planners Baselines Setup. We group baselines into three categories. Reproduced (DreamerV3, QMIX, MAPPO): trained and evaluated by us under our observation and action spaces, giving frameworks that require extensive language reasoning additional training and inference budget until convergence. Domain-adapted: RT-2, fully fine-tuned on demonstrations from a trained DreamerV3 policy with non-pixel observations tokenized alongside pixels; as this departs from its original real-robot setting, these numbers reflect a domain adaptation. Contextual (Voyager, DEPS, JARVIS-1, LS-Imagine): evaluated with released code on Minecraft Diamond and adapted code on MindCraft, as their prompting infrastructure is Minecraft-specific and new prompts on new environments would introduce confounds. Results. Table 1 shows the main results, comparing different VLMs as planners with several baseline agents. We train one controller per environment, so training does not require planner-specific fine-tuning or VLM reasoning in the loop. The table includes: (i) our planner+controller agents using four VLM planners; (i) ablations that remove the controller or language interface; (i) prior environment-specific baselines; and (iv) recent VLA/MARL baselines when available. The main pattern is consistent across tasks: decomposition into planning and control improves over direct VLM action generation, and adding language guidance improves over the controller-only baseline. Nearly all tested planners significantly outperform the end-to-end GPT-4o agent. Using GPT-4o to generate sparse instructions executed by a trained controller, rather than directly producing low-level actions, improves performance on every task. The strongest planners (GPT-4o and Qwen-VL-2.5-72B) achieve the best overall results, while our method remains competitive with stronger domain-specialized systems such as JARVIS-1 in Minecraft. 6 Hyperparameters, model scales, and compute/runtime overhead are shown in Appendix A.3, Appendix A.5, and Appendix A.4 respectively. 7 Pico Park (https://picoparkgame.com/en/) is a 2-8 player 2d cooperative platformer puzzle game. 8 Detailed tasks descriptions are in Appendix B. We also include a lightweight human experiment in Appendix D, a multi-agent failure mode analysis in Appendix A.7, an instruction-length robustness study in Appendix A.9, and a taxonomy of generated instructions in Appendix A.8. 6 Published as a conference paper at COLM 2026 Single-Agent TasksMulti-agent Tasks AtariMC-DiaCraftDMLOCookPicoMindC Ours (planner swap) Gemma-3-27B (Gemma Team et al., 2025) 8809.713.871187.468.953.0 llava-v1.6-34b (Liu et al., 2024)8629.912.874187.263.551.6 Qwen-VL-2.5-72B (Bai et al., 2025) 87811.113.477192.368.458.5 GPT-4o (Hurst et al., 2024)89111.714.176193.270.170.2 Ablation Qwen-VL-2.5-72B (Direct VLM Finetune) 58110.17.645150.130.638.2 GPT-4o (w/o controller)67010.48.756180.450.350.2 Controller-Only8098.212.667170.230.740.0 Reproduced Baselines Dreamerv3 (Hafner et al., 2025)8118.610.565--- MAPPO (Yu et al., 2022)----182.550.844.9 QMIX (Rashid et al., 2020)----187.258.553.2 Domain- Adapted (§4.3) RT-2 (Zitkovich et al., 2023)4576.24.736124.734.033.1 Contextual Comparisons Voyager (Wang et al., 2024)-11.8----- DEPS (Wang et al., 2023)-9.4----45.6 LS-Imagine (Li et al., 2025)-9.6----50.1 JARVIS-1 (Wang et al., 2025)-12.3----54.1 Mindcraft (White et al., 2025)------49.0 Table 1: Main results. Each cell reports the task’s official metric where available. Baselines are grouped by comparability into reproduced, domain-adapted, and contextual categories, detailed in Section 4.3; our controlled claim rests on the matched comparison between our planner-controller variants and the ablation rows. Abbr: MC-Dia (Minecraft Diamond), Craft (Crafter), DML (DMLab), OCook (Overcooked), Pico (Pico Park), and MindC (MindCraft). 248 Number of Agents 64 66 Score Performance 248 Number of Agents 35 40 45 Throughput Efficiency DecentralizedCentralized (a) Agent scaling (Pico Park) offlineonline Planning Mode 10 11 Score Performance offlineonline Planning Mode 20 40 Throughput Efficiency SmallMediumLarge (b) Planning modes (Minecraft Diamond) Figure 2: Per-setting performance (left) and efficiency (right) results for (a) agent scaling and (b) planning modes. Model-size and reasoning scaling curves are shown in Figure 3 (Ap- pendix A.6). 4.4 Scalability and Efficiency We evaluate the framework’s performance and efficiency (throughput as environment steps per second) under multiple conditions. 9 In Figure 2a, we compare decentralized with cen- tralized control in the multi-agent Pico Park environment. Each agent is implemented as a Qwen-VL-2.5-72B planner paired with an 800M-parameter controller. We measure task perfor- mance and throughput as we vary the number of agents from 2 to 8. We find that decentralized communication results in improved task performance while also maintaining a throughput equivalent to centralized control. This is consistent with the motivation for centralized- training/decentralized-execution schemes in MARL, where routing all decisions through a central entity scales poorly with the number of agents (Kraemer & Banerjee, 2016; Rashid et al., 2020); in our framework the analogous bottleneck appears at the hub planner in centralized mode, whose failure modes we analyze in Appendix A.7. Unlike CTDE, however, our agents coordinate through language at the planner level rather than through a centralized critic. 9 Additional conditions are described in Appendix A.6 and Figure 3. 7 Published as a conference paper at COLM 2026 Train-time instruction typeDiamond↑ VLM-generated11.1 Template-based instructions9.9 Clustered action labels8.9 Random strings8.3 No instructions8.2 Environment-derived labels11.4 Table 2: Impact of post-hoc instruction source on Minecraft Diamond, using a Qwen-VL-2.5-72B planner and an 800M-parameter controller. Eval Planner↑ AnnotatorGemmaQwenGPT Gemma9.69.210.2 Qwen9.79.99.9 GPT9.89.610.0 Table 3: Cross-planner train×eval matrix (Minecraft Diamond). Each cell is score of a controller trained on instructions annotated by the row planner and evaluated by the column planner. In Figure 2b, we compare online with offline planning modes on Minecraft Diamond. Here we use a Qwen-VL-2.5 (Bai et al., 2025) planner with three sizes: small (50M controller with 2B planner), medium (200M controller with 7B planner), and large (800M controller with 72B planner). As model sizes increase, online planning offers increased task performance compared to offline planning, while also remaining an order of magnitude faster in terms of throughput. 4.5 Generalization We also study how far the framework generalizes beyond the exact configuration it was trained with: does a controller trained on one planner ’s annotations transfer to other planners and to different instruction-timing policies, do the gains depend on our specific controller architecture, and how reliably does the controller execute held-out planner instructions? Across these studies, 10 planners that decide instruction timing outperform fixed cadence on Minecraft Diamond (e.g., GPT-4o: 10.75 for VLM-decided proactiveness vs. 9.60 fixed cadence), and train×eval cross-planner transfer remains stable (Table 3). The framework is also not tied to RSSM specifically: a transformer world model is comparable to ours (11.2 vs. 11.0), while weaker controller variants degrade performance (RNN policy: 9.7; value-only world model: 10.2). Controller reliability remains high across domains and planners, with instruction-following accuracy between 172/200 and 194/200 (86% to 97%). Annotation quality. Table 2 shows the influence of annotation quality during controller training on controller quality. We compare VLM-generated instructions to template- generated and cluster-based instructions, as well as environment-derived symbolic labels. Environment-derived symbolic labels slightly outperform VLM-generated instructions (11.4 vs. 11.1), but these require access to environment metadata, whereas VLM-based annotation needs only raw observations and thus generalizes to domains without such metadata. The results show that controllers benefit from semantic content even if instructions are of lower quality; using random strings as instructions, or no instructions at all, significantly underperform template- or cluster-based instructions. Cross-planner generalization. Train×eval transfer remains stable across all planner pairs in Table 3 (9.2 to 10.2), indicating the controller does not overfit to the phrasing of its annotating planner. Controller cost and scope.Our main results use one 800M-parameter controller trained per environment, a one-time cost of∼23 GPU-hours on 4 RTX A6000 GPUs (∼5.5M environment steps; full compute and annotation overhead in App. A.4). In early experiments, a single con- troller trained jointly on all environments lost 1.12 points on Minecraft Diamond (≈10% rela- tive), so lightweight domain-general instruction-conditioned controllers remain future work. 10 Full tables are included in Appendix A.5. 8 Published as a conference paper at COLM 2026 5 Related Work Our controller builds on RSSMs and imagination-based control: PlaNet introduced stochastic–deterministic latent transitions for planning from pixels (Hafner et al., 2019), and Dreamer learns policies by backpropagating value gradients through imagined latent trajectories (Hafner et al., 2020; 2025); we instantiate this line of work with language conditioning and post-hoc instruction supervision. Decoupling slow, deliberate reasoning from fast, reactive control is a dominant paradigm across embodied domains, including autonomous driving (Tian et al., 2024; Qian et al., 2024; Liu et al., 2025), robot manipulation (Zhang et al., 2024; Shi et al., 2025; Bjorck et al., 2025; Figure AI, 2025; Song et al., 2025; Chen et al., 2025), and vision-and-language navigation (Wei et al., 2025); HiRT (Zhang et al., 2024) in particular runs an asynchronous planner-controller scheme communicating through latent representations. We do not claim the slow/fast decomposition as novel; we differ in that (i) the planner-controller interface is natural language rather than latents or waypoints, making the planner plug-and-play and enabling language-based multi-agent coordination; (i) the fast system is a world-model controller rather than a domain-specific stack or waypoint follower, so it can generalize across heterogeneous domains; and (i) our controller requires no expert data—all behavior-cloning supervision comes from post-hoc VLM relabeling of its own on-policy rollouts. Language-conditioned robot-control systems motivate our planner-controller decomposition: SayCan, Code as Policies, and Inner Monologue compose language-model reasoning with low-level skills (Ahn et al., 2022; Liang et al., 2023; Huang et al., 2022), while end-to-end VLA approaches such as RT-1, RT-2, RT-H, OpenVLA,π 0 , GR00T, and Octo map observation- language inputs directly to actions (Brohan et al., 2023; Zitkovich et al., 2023; Belkhale et al., 2024; Kim et al., 2024; Black et al., 2025; Bjorck et al., 2025; Octo Model Team et al., 2024). Unlike monolithic VLA adaptation, we train only the smaller environment-specific controller and keep the planner plug-and-play, enabling planner swaps without running full VLM inference at control frequency. For multi-agent control, we replicate one shared-parameter controller across agents with separate recurrent states, similar to prior MARL work (Terry et al., 2020; Chu & Ye, 2017; Rashid et al., 2020); unlike CTDE-style methods with centralized critics or value factorization, our low-level controllers stay decentralized and coordination happens in language at the planner level. Embodied planner–controller agents in open-ended domains (Voyager, DEPS, LS-Imagine, JARVIS-1 (Wang et al., 2024; 2023; Li et al., 2025; Wang et al., 2025)) and iterative-reasoning LM agents (ReAct, Reflexion (Yao et al., 2023; Shinn et al., 2023)) demonstrate the promise of language-guided planning, but are typically tightly coupled to particular domains, planner backbones, or action-generation pipelines; we instead emphasize a reusable language interface between planner and controller, asynchronous inference, and evaluation across both single-agent and multi-agent domains. Language is not the only possible conditioning interface: latent goal or skill abstractions such as LISA (Garg et al., 2022) provide an alternative, but language lets humans and off-the-shelf VLMs act as planners without retraining the controller-side interface, which is central to our plug-and-play evaluation setup. 6 Conclusion We introduced Instruct-to-Act, a framework that combines the low latency of world-model controllers with the high-level reasoning ability of VLMs. Across diverse embodied tasks, the resulting planner+controller agents achieve strong performance while preserving high control-rate throughput, and the same interface extends naturally to multi-agent coordination without requiring specialized MARL training. Important next steps include studying human planners directly in the loop, extending the framework to asymmetric multi-agent roles and multi-task controller training, and improving planner–controller alignment through prompting or lightweight planner adaptation. More broadly, the framework suggests a practical path for deploying general-purpose language models in domains where abstract guidance must be translated into long, low-level action sequences. 9 Published as a conference paper at COLM 2026 References Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, p. 1726–1734, 2017. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, et al. DeepMind Lab. arXiv preprint arXiv:1612.03801, 2016. Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quan Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: Action hierarchies using language. In Robotics: Science and Systems, 2024. Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013. Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π 0 : A vision-language-action flow model for general robot control. In Robotics: Science and Systems, 2025. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems, 2023. Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-AI coordination. In Advances in Neural Information Processing Systems, volume 32, 2019. Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. TransDreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.09481, 2022. Hao Chen, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Renrui Zhang, Xiaoqi Li, Xiao He, et al. Fast-in-slow: A dual-system VLA model unifying fast manipulation within slow reasoning. In Advances in Neural Information Processing Systems, volume 38, 2025. Xiangxiang Chu and Hangjun Ye. Parameter sharing deep deterministic policy gradient for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:1710.00336, 2017. Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, volume 5, 1992. Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. In International Conference on Machine Learning, p. 8469–8488, 2023. Figure AI.Helix: A vision-language-action model for generalist humanoid control. https://w.figure.ai/news/helix, 2025. Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. LISA: Learning interpretable skill abstractions from language. In Advances in Neural Information Processing Systems, volume 35, 2022. 10 Published as a conference paper at COLM 2026 Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, p. 2442–2448, 2019. Danijar Hafner. Benchmarking the spectrum of agent capabilities. In International Conference on Learning Representations, 2022. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, p. 2555–2565, 2019. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647–653, 2025. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, 2022. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A. J. Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for Atari. In International Conference on Learning Representations, 2020. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning, 2024. Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82–94, 2016. Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019. Jiajian Li, Qi Wang, Yunbo Wang, Xin Jin, Yang Li, Wenjun Zeng, and Xiaokang Yang. Open-world reinforcement learning over long short-term imagination. In International Conference on Learning Representations, 2025. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, p. 9493–9500, 2023. Haichao Liu, Haoren Guo, Pei Liu, Benshan Ma, Yuxiang Zhang, Jun Ma, and Tong Heng Lee. VLM-UDMC: VLM-enhanced unified decision-making and motion control for urban autonomous driving. arXiv preprint arXiv:2507.15266, 2025. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024. 11 Published as a conference paper at COLM 2026 Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523–562, 2018. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine.Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, 2018. Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. In Robotics: Science and Systems, 2024. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Kangan Qian, Zhikun Ma, Yangfan He, Ziang Luo, Tianyu Shi, Tianze Zhu, Jiayin Li, et al. FASIONAD: FAst and slow FusION thinking systems for human-like autonomous driving with adaptive feedback. arXiv preprint arXiv:2411.18013, 2024. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763, 2021. Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1–51, 2020. Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. In International Conference on Machine Learning, 2025. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, p. 8634–8652, 2023. Haoming Song, Delin Qu, Yuanqi Yao, Qizhi Chen, Qi Lv, Yiwen Tang, Modi Shi, et al. Hume: Introducing System-2 thinking in visual-language-action model. arXiv preprint arXiv:2505.21432, 2025. Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112 (1-2):181–211, 1999. J. K. Terry, Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005.13625, 2020. Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. DriveVLM: The convergence of autonomous driving and large vision-language models. In Conference on Robot Learning, 2024. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. 12 Published as a conference paper at COLM 2026 Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, p. 5776–5788, 2020. Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. In Advances in Neural Information Processing Systems, volume 36, p. 34153–34189, 2023. Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3):1894–1907, 2025. Meng Wei, Chenyang Wan, Jiaqi Peng, Xiqian Yu, Yuqiang Yang, Delin Feng, Wenzhe Cai, et al. Ground slow, move fast: A dual-system foundation model for generalizable vision-and-language navigation. arXiv preprint arXiv:2512.08186, 2025. Isadora White, Kolby Nottingham, Ayush Maniar, Max Robinson, Hansen Lillemark, Mehul Maheshwari, Lianhui Qin, and Prithviraj Ammanabrolu. Collaborating action by action: A multi-agent LLM framework for embodied reasoning. arXiv preprint arXiv:2504.17950, 2025. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 35:24611–24624, 2022. Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, and Jianyu Chen. HiRT: Enhancing robotic control with hierarchical robot transformers. In Conference on Robot Learning, 2024. Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183, 2023. 13 Published as a conference paper at COLM 2026 A Model Details A.1 World-model and control details RSSM transition and inference. Conditioning on language, the deterministic transition and stochastic prior/posterior are h t = f θ ( h t−1 ,z t−1 ,a t−1 ,e t−1 ) , p θ (z t | h t )=N μ θ (h t ),diagσ 2 θ (h t ) , q φ (z t | h t ,o t ,e t )=N μ φ (h t ,o t ,e t ),diagσ 2 φ (h t ,o t ,e t ) . We denote the model state ass t =(h t ,z t )withz t ∼ q φ (·)during training andz t ∼ p θ (·)during imagination. Decoders and predictors. From s t we decode/predict: p θ (o t | s t )(observation decoder), p θ (r t | s t )(reward model), p θ (c t | s t )(continuation/discount model with c t ∈(0,1)), p θ (x t+1 | s t )(next-instruction predictor; optional teacher forcing on x t ). The instruction head is trained only when m t+1 = 1; otherwise its loss is masked out. Model learning objective. Over a sequencet=1:T, the model loss combines reconstruc- tion/prediction terms with a KL regularizer: L model = T ∑ t=1 E q φ h −logp θ (o t | s t )− λ r logp θ (r t | s t )− λ c logp θ (c t | s t )− λ x m t+1 logp θ (x t+1 | s t ) i + β T ∑ t=1 KL q φ (z t | h t ,o t ,e t ) p θ (z t | h t ) , with weights λ r ,λ c ,λ x ≥ 0 and KL scale β. A.2 Dreamer-style Control in Latent Space Given a learned world model, we learn a policy and value on latent states conditioned on language: π c (a t | s t ,e t ),V ψ (s t ,e t ). Imagined rollouts.Starting from posterior statess t on real trajectories, we ‘imagine’H-step futures using the prior dynamics and current policy: ̃ s t+1 ∼ p θ (·| ̃ h t+1 ), ̃ h t+1 = f θ ( ̃ h t , ̃ z t , ̃ a t , ̃ e t ), ̃ a t ∼ π η (·| ̃ s t , ̃ e t ), ̃ r t ∼ p θ (·| ̃ s t ), ̃ c t ∼ p θ (·| ̃ s t ), where ̃ e t is the instruction embedding available to the agent during imagination (e.g., last known instruction embedding, or an imagined instruction from p θ (x t+1 | s t )). Value targets viaλ-returns with continuation. Define the continuation as the learned discount γ ̃ c τ ∈[0,1]. The λ-return is ˆ G λ τ = ̃ r τ + γ ̃ c τ (1− λ)V ψ ( ̃ s τ+1 , ̃ e τ+1 )+ λ ˆ G λ τ+1 , ˆ G λ t+H = V ψ ( ̃ s t+H , ̃ e t+H ). Actor and value losses. We optimize the value to regress to the return and the actor to maximize it through imagined trajectories: L value = t+H−1 ∑ τ=t V ψ ( ̃ s τ , ̃ e τ )−stopgrad( ˆ G λ τ ) 2 2 , L actor =− t+H−1 ∑ τ=t E ̃ a τ ∼π η h stopgrad( ˆ G λ τ ) i − α t+H−1 ∑ τ=t H π η (·| ̃ s τ , ̃ e τ ) , 14 Published as a conference paper at COLM 2026 Table 4: Model-scale configurations. Scale (tag) rssm.deter rssm.hidden rssm.classes depth units 50M (size50m)40965123232512 200M (size200m)8192102464641024 800M (size800m)2457630721921923072 MethodThroughput RT-2 (Zitkovich et al., 2023)16.2 DEPS (Wang et al., 2023)20.5 LS-Imagine (Li et al., 2025)23.7 JARVIS-1 (Wang et al., 2025)26.0 Instruct-to-Act (Qwen-2.5-VL-72B)31.7 Table 5: Inference throughput (environment steps/s) for representative VLA baselines and our framework with a Qwen-based planner. with entropy scaleα≥0. Gradients backpropagate through the imagined dynamics (world model) as in Dreamer. A.3 Model Hyperparameters We use MiniLM-L6-H384-uncased (Wang et al., 2020) as the language encoder. For small controller as an example, we instantiate a language-conditioned DreamerV3-style (Hafner et al., 2025) agent with a CNN encoder (4 convolutional blocks; channels[32,64,128,256], stride 2, SILU), an RSSM with a deterministic GRU core of size 512 and a diagonal-Gaussian stochastic latentz t ∈R 32 , and decoders for observation/reward/continuation as MLPs (two layers, 512 units, SILU). The actor and value heads operate in latent space and use three-layer MLPs with 512 units (SILU). The policy head produces action logits for discrete control or mean/scale for continuous control, depending on the environment action space. We follow dreamer to use KL balancing with scaleβ=1.0 and free-nats 1.5, reconstruction/prediction weights(λ r ,λ c ,λ x )=(1,1,1), imagined horizonH=15 (as above), and continuation-based λ-returns. The language pathway pools a frozen text encoder into an embeddinge t that conditions both the world model and control; when absent, a learned null embedding is used. Optimizer and update schedule follow the hyperparameters above (Adam, 3e−4, clip 40, 1:1 world-model/control updates). Table. 4 show controller scales from 50M, 200M, to 800M. These configs are adapted from Dreamerv3 (Hafner et al., 2025) and re-used the 50M and 200M configuration. A.4 Training Compute and Annotation Overhead Unless otherwise stated, GPT-4o is used as the replay annotator and 50% of replay segments are instruction-annotated. Training uses 4×NVIDIA RTX A6000 GPUs and an Intel Xeon Gold 6338 CPU. Across all tasks, the maximum training time for one environment is approximately 36 wall-clock hours (23 hours on average). Relative to controller-only training, post-hoc annotation adds roughly 17% GPU hours and a 12% training slowdown. A.5 Additional Results Efficiency comparison with other works: Table 5 complements the throughput plots with a direct comparison against representative end-to-end or tightly coupled baselines. Because our controller executes at every step while the planner is queried only sparsely, Instruct-to-Act achieves the best throughput among these methods. 15 Published as a conference paper at COLM 2026 VLM Planner Instruction ModeGemmaQwenGPT Fixed Cadence8.869.329.60 Fully controlled by VLM9.8910.1210.52 Proactiveness decided by VLM10.1210.2310.75 Table 6: Single-agent tests under different instruction modes. (Minecraft Diamond) Table 7: Controller architecture ablation on Minecraft Diamond. The gains are not specific to an RSSM controller, though world-model controllers perform best. Controller architectureDiamond Dreamer-V3-style RSSM (ours)11.0 Transformer world model (Chen et al., 2022)11.2 RNN policy (model-free)9.7 Value-only world model10.2 Table 8: Instruction-following accuracy measured on 200 held-out planner instructions per domain. TaskGemmaQwenGPT-4o Minecraft Diamond193/200190/200194/200 Pico Park186/200180/200189/200 Atari181/200172/200185/200 2B7B72B VLM Size 10 11 Score Scaling Perf. 50M 200M 800M Small MediumLarge Model Scale 20 30 40 50 60 Throughput Scaling Eff. w/ VLM w/o VLM (a) Model size scaling (Minecraft Diamond) 832128 # Avg. Tokens 10 11 Score Token Scaling Perf. Qwen-VL-2.5 Gemma3 GPT-4o 832128 # Avg. Tokens 20 30 40 50 60 Throughput Token Scaling Eff. Small Medium Large (b) Reasoning scaling (Minecraft Diamond) Figure 3: Per-setting performance (left) and efficiency (right) results for the model-size and reasoning scaling analyses in Sec. 4.4. Agent scaling and planning modes are shown in Figure 2 in the main paper. Model scales for controller and VLM are small (50M, 2B), medium (200M, 7B), and large (800M, 72B). Instruction modes (Minecraft): We contrast three instruction regimes (Table 6): (1) Fixed cadence: planners issue instructions everyK=16 steps; (2) VLM-decided proactiveness (the VLM emits instructions only when its uncertainty exceeds a threshold, with a minimum gap of 32 steps); (3) Fully controlled by VLM. We report achievement scores for three planners. Controller architecture: Table 7 shows that the broader framework is not tied to Dreamer specifically. A transformer world-model controller yields similar trends, whereas removing the world-model structure degrades performance. Human Evaluated Instruction Following Success:We perform human eval in Table 8 that quantifies controller reliability under planner instructions. Across domains and planners, instruction-following accuracy stays around 86–97%. The most common failures arise from vague or underspecified instructions such as “explore a bit more” or “try something different”, which can lead to locally suboptimal behavior until a clearer instruction arrives. 16 Published as a conference paper at COLM 2026 Minecraft DiamondOvercooked OnlineOfflineOnlineOffline VLM latency mean (ms)1037.41221.91106.31010.4 VLM latency p95 (ms)1701.12035.7 1872.51523.7 Staleness mean (steps)51.20.0165.10.0 Blocking steps mean0.035.40.00.0 Throughput (env steps/sec)37.513.4169.992.7 Table 9: Latency, instruction staleness, and throughput for online (asynchronous) versus offline (blocking) planning. Overcooked staleness and latency are averaged over both agents. A.6 Scalability and Efficiency Curves Figure 2 (main paper) and Figure 3 report paired curves for each scaling axis, with perfor- mance on the left and throughput (environment steps/s) on the right. Across all four settings, we observe a favorable frontier: configurations that improve task performance usually retain most of the efficiency gains from decoupled control rather than collapsing to the latency profile of end-to-end VLM action generation. Regarding relative planner performance, Figure 3(a) provides a controlled scaling study in which every planner-controller combination shares matched settings; there, performance on Minecraft Diamond improves monotonically as either planner or controller size grows. Deviations from a pure parameter-count ordering in Table 1 reflect differences in planner pre-training rather than the framework. We also note that Voyager is engineered specifically for Minecraft Diamond, which explains its strong performance on that single task, whereas our framework targets robustness across heterogeneous environments with a single recipe. Asynchronous planning efficiency. Table 9 quantifies the benefit of asynchrony beyond the two-bar comparison in Figure 2(b). Online planning achieves 2.8×higher throughput on Minecraft Diamond and 1.8×on Overcooked (whose shorter episode horizon limits the gain). Asynchrony introduces instruction staleness of roughly 51 steps (Minecraft) and 165 steps (Overcooked) between a stop signal and the arriving instruction, but because the controller continues executing the previous instruction during this window, staleness affects only the next high-level instruction and never low-level control. A.7 Multi-agent Failure Mode Centralized settings tend to bottleneck on the hub planner, while decentralized settings most often fail through conflicting or duplicated subgoal allocation. To quantify this, we manually examined 100 failure cases in Overcooked, defining controller-level failures as instruction-following errors or stale loops, and planner-level failures as duplicated task assignment, unmet preconditions (e.g., serving a soup that is not yet made), or pipeline stalls. Planner-level coordination failure dominates: 96 of 100 failure windows show near-duplicate sub-task assignments across agents (instruction-embedding cosine similarity above 0.7), while only 3 show controller instruction-following failures. A.8 Taxonomy of Generated Instructions Across our Minecraft experiments we collected 47,547 instruction instances (3,477 unique). The set is dominated by local action descriptions (82.8%,n=39,363), short imperatives describing a single immediate action such as “Mine stone blocks with a wooden pickaxe” or “Swim across the river towards the trees.” The remainder (17.2%,n=8,184) are high-level subgoals that chain a primary action to a downstream task, e.g., “Mine stone to craft a stone pickaxe.” We observed no retrospective summaries of behavior. 17 Published as a conference paper at COLM 2026 A.9 Instruction-length Robustness We generated 100 Minecraft Diamond instructions across 25 length checkpoints (1 to 100 words, 4 per length) spanning navigation, collection, crafting, and survival categories, and measured whether the controller terminates at an appropriate time (judged by an API model), averaged over 5 seeds. The stop head is robust fromL=2 throughL=50, with no cliff at the training-distribution boundary (L=11), indicating generalization beyond training lengths. Degradation begins gradually aboveL=50, reaching∼0.55 success atL=100; the only sharp drop is atL=1, where a single word carries insufficient semantic content to condition behavior. A.10 Training Dynamics of the Composite Loss Figure 4 shows how each sub-objective evolves over training. Early in training, the composite loss is dominated by the world-model reward term, as the controller has not yet learned to predict environment returns. As training progresses and instruction-relabeled segments accumulate in the replay buffer, the behavior-cloning loss becomes the dominant component. The dynamics loss exhibits an initial spike followed by a steady decrease, reflecting that the policy first encounters many unexpected observations during early exploration and subsequently learns the environment dynamics. Notably, the policy and behavior-cloning objectives continue to improve even as the expanding behavior distribution exposes the world model to new surprises, and we observed no single component collapsing or diverging. Training was stable across this run without loss re-weighting or scheduling beyond the fixed linear combination described in Section 3.2. A.11 Prompt Details In this section, we list the prompts we used for training and inference including multi-agents. The summarizer prompt for GPT-4o as shown in Fig. 5 is used for post-hoc training language annotation in Sec. 3.4. The inference prompt as shown in Fig. 6 is used for online reasoning and issuing instructions as mentioned in Sec. 3.1. The multi-agent prompt as shown in Fig. 7 is used for prompting multi-agents communication. The full prompt is simply appending this to the single-agent inference prompt. Note that the math notations are all replaced by actual inputs or in-context examples. B Tasks Details We list and describe the evaluation tasks in details here totaling 7 environments. Every environment can be discussed from its goal, the observation space, evaluation protocol, and the metric. Arcade Learning Environment (ALE) • Goal: Achieve high scores across Atari 2600 titles using standard ALE evaluation. • Obs/Act: Raw visual frames with common preprocessing (frame skip/stack); full primitive action set. • Protocol: Follow the evaluation practices popularized in recent Atari work, e.g., Kaiser et al. (2020); Machado et al. (2018), including fixed evaluation episodes and capped frames per episode; when using sticky actions or ALE “game flavours,” also follow the ALE protocol paper. • Metric: Game score per episode; report means (and variance) over seeds. MineRL ObtainDiamond • Goal: Obtain a diamond in survival Minecraft starting from scratch. •Obs/Act: 64×64 first-person RGB plus discrete inventory observations; actions com- bine continuous camera control with discrete navigation, mining/crafting/smelting interactions. 18 Published as a conference paper at COLM 2026 Figure 4: Training curves of the individual loss components for a 200M-parameter con- troller on Minecraft Diamond (seed 0). The overall objective is a linear combination of the world-model losses (reconstruction, dynamics, and reward prediction), the actor and value objectives, the behavior-cloning (BC) loss on instruction-relabeled segments, and the stop- token loss. •Protocol: Episode terminates on death, diamond obtained, or 18,000-frame (15 min) limit. Competition evaluation averages performance over 500 episodes on fixed but unseen seeds; strict train-from-scratch compliance. •Metric: Shaped milestone reward (e.g., logs, planks, pickaxes, iron, diamond), summed per episode; tie-break by fewest episodes to last milestone. Crafter • Goal: Survive and progress in a procedurally generated 2D open world by unlocking semantically meaningful achievements (e.g., find water, craft tools, defeat enemies). 19 Published as a conference paper at COLM 2026 You are a concise action summarizer for training labels. Read inputs IN THIS ORDER: 1. TaskGuidance G (what succeeds in this environment) and a short Message M (any extra note), 2. sampled visual Frames F[t : t+L-1] with timestamps, 3. the executed Actions A[t : t+L-1] with timestamps. Goal produce ONE short imperative instruction x that, if given before t, would make a competent controller reproduce A[t : t+L-1]. Keep it atomic and task-grounded. Rules Focus on intent, not low-level joystick/button spam. Refer only to what is visible/achieved in Frames and consistent with Actions. No hallucinations; if ambiguous, pick the MINIMAL instruction that explains A. 6–18 tokens; avoid conjunction chains; one clause; present tense verb first word. If the segment ends at completion, align x to that subgoal's completion boundary. Output JSON ONLY "instruction": "<single-line imperative>", "justification": "<8-20 token rationale tied to frames/actions>", "segment": "t_start": t, "t_end": t+L-1 [USER] TaskGuidance (G): G Message (M): M Frames F[t:t+L-1]: FRAME_LIST_OR_CAPTIONS Actions A[t:t+L-1]: ACTION_LIST .description font-size: 14px; margin-bottom: 15px; h2 font-size: 14px; font-weight: normal; margin-top: 15px; margin-bottom: 5px; .user-heading font-size: 16px; margin-top: 15px; margin-bottom: 5px; .steps font-size: 14px; margin-bottom: 10px; .steps ol margin-left: 20px; margin-bottom: 10px; .steps li margin-bottom: 2px; .goal font-size: 14px; margin-bottom: 10px; .rules ul margin-left: 20px; margin-bottom: 10px; .rules li margin-bottom: 2px; font-size: 14px; .json-block font-size: 12px; margin: 10px 0; white-space: pre; .template-vars font-size: 14px; margin-bottom: 10px; .var-item 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 HTML Online Viewer (https://html.onlineviewer.net/)  Preview (Full page)  Highlight (Full page)  Format  Expand all  Collapse all  Sample  Clear  Import  Export  Privacy settings  About © 2025 OnlineViewer.net (https://w.onlineviewer.net) 1 of 1 streams All + .*Aa 9/25/25, 1:41 AMHTML Online Viewer https://html.onlineviewer.net1/1 Figure 5: Training summarization prompt. •Obs/Act: Local top-down pixel observations showing surroundings and inventory status; discrete actions for movement, interaction, crafting, sleeping, placing. •Protocol: Two tracks: with extrinsic rewards and reward-free. Agents get a fixed budget of 1M environment steps (commonly also reported at 5M in baselines). Success rates computed across the entire training run to emphasize sample efficiency. •Metric: Crafter score is the geometric mean of the 22 achievement success rates, emphasizing breadth/depth of capabilities. DeepMind Lab: explore_goal_locations •Goal: First-person 3D navigation in maze-like levels to discover and reach goal locations under partial observability. • Obs/Act: RGB first-person frames; continuous look and movement controls. • Protocol: Standard DM-Lab evaluation with fixed episode caps; commonly eval- uated on held-out maps (unseen layouts), sometimes with environment-provided debug info only for visualization/analysis (not agent inputs). •Metric: Episode return (goals found/reached), success rate, and path efficiency; report means over seeds/maps. Overcooked-AI • Goal: Two-player cooperative cooking (deliver soups quickly) with strong zero-shot coordination (ZSC) to unseen partners and layouts. 20 Published as a conference paper at COLM 2026 [SYSTEM = Sp] You are the planner module π_p running ASYNCHRONOUSLY with a low-latency controller π_c. Maintain Reasoning memory M_r (brief scratch notes). Partial plan P_n (bullet list of next subgoals → micro-steps). You receive a live observation stream O_<t' and a stop signal probability p_stop_t'-1 from π_c. Emit a NEW instruction x ONLY when either (a) p_stop_t'-1 is true/high (≥ 0.5), or (b) you proactively decide the current plan needs revision. Otherwise ADVANCE your background plan silently. Constraints Instructions are 6–18 tokens, single-clause, imperative, specific and verifiable from observations. Never block control; if unsure, issue the safest non-destructive next step. Keep M_r and P_n ultra-compact; prefer deltas over full rewrites. Output JSON ONLY "emit": <true|false>, "instruction": "<if emit=true, the instruction; else empty>", "reasoning_update": "<<=40 tokens delta for M_r>", "plan_update": ["<<=12 tokens step 1>", "<step 2>", "..."], "confidence": <0.0-1.0> [USER] Previous state - M_r(t'-1): M_r_prev - P_n(t'-1): P_n_prev - p_stop_t'-1: p_stop_prev New context - O_<t' (latest frames/summaries): OBS_STREAM_SNIPPET - Task spec (if any): TASK_SPEC .var-item font-size: 12px; margin: 2px 0; .emphasis .highlight </style> </head> <body> <div class="container"> <div class="description"> [SYSTEM = Sp]<br> You are the planner module π_p running ASYNCHRONOUSLY with a low-late π_c. </div> <h2>Maintain</h2> <div class="maintain"> <ul> <li>Reasoning memory M_r (brief scratch notes).</li> <li>Partial plan P_n (bullet list of next subgoals → micro-steps) </ul> </div> <div class="receive"> You receive a live observation stream O_<t' and a stop signal pr p_stop_t'-1 from π_c. </div> <div class="emit"> Emit a NEW instruction x ONLY when either (a) p_stop_t'-1 is true/h or (b) you proactively decide the current plan needs revision. Ot ADVANCE your background plan silently. </div> <h2>Constraints</h2> <div class="constraints"> <ul> <li>Instructions are 6–18 tokens, single-clause, imperative, spec verifiable from observations.</li> <li>Never block control; if unsure, issue the safest non-destruct .</li> <li>Keep M_r and P_n ultra-compact; prefer deltas over full rewri </ul> </div> <h2>Output JSON ONLY</h2> <div class="json-block"> "emit": <true|false>, "instruction": "<if emit=true, the instruction; else empty>", "reasoning_update": "<<=40 tokens delta for M_r>", "plan_update": ["<<=12 tokens step 1>", "<step 2>", "..."], "confidence": <0.0-1.0> </div> <div class="user-heading">[USER]</div> <h2>Previous state</h2> 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 HTML Online Viewer (https://html.onlineviewer.net/)  Preview (Full page)  Highlight (Full page)  Format  Expand all  Collapse all  Sample  Clear  Import  Export  Privacy settings  About © 2025 OnlineViewer.net (https://w.onlineviewer.net) 1 of 1 streams All + .*Aa 9/25/25, 1:44 AMHTML Online Viewer https://html.onlineviewer.net1/1 Figure 6: Single-agent prompt. •Obs/Act: Gridworld state or egocentric features; discrete actions (move, interact, pick/place). •Protocol: Evaluate on canonical layouts (Cramped Room, Asymmetric Advantages, Coor- dination Ring, etc.) and out-of-distribution layouts; cross-play with behavior-cloned human models and held-out agents; human studies where applicable. •Metric: Team return (deliveries/time), success rate, and ZSC scores (cross-play averages across unseen partners/layouts). Pico Park (Co-op Puzzles) •Goal: Complete short, cooperative puzzle-platforming levels that demand synchronized actions (e.g., stacking, shared switches, tethered movement). •Obs/Act: Platformer state with simple discrete controls for 2–8 players (levels scale with player count). 21 Published as a conference paper at COLM 2026 [SYSTEM = Sp_multi_agent_addon] You are planner i in a cooperative team of n agents. You may send at MOST ONE short message per step. Inbox processing first read Inbox[i] (messages addressed to you or broadcast). Update M_r and P_n accordingly. Communication modes Decentralized: you decide when to message. Centralized (hub=h): only agent h broadcasts role/assignment messages; others reply sparsely. Messaging rules Keep messages ≤ 25 tokens. Prefer structured intents: claim:<subgoal>, status:<brief>, block:<issue>, request: <help on X>, handoff:<asset/role>. Addressing: use @all for broadcast or @j for a specific agent j. Do not restate observations verbatim; send actionable deltas that change teammates' plans. Augmented OUTPUT (append to (2)'s JSON) "comm": "send": <true|false>, "to": "@all" | "@j" | "@hub", "message": "<one line following the intent tags above>" margin-bottom: 2px; font-size: 14px; .json-block font-size: 12px; margin: 10px 0; white-space: pre; .emphasis .highlight </style> </head> <body> <div class="container"> <div class="description"> [SYSTEM = Sp_multi_agent_addon]<br> You are planner i in a cooperative team of n agents. You may send at message per step. </div> <h2>Inbox processing</h2> <div class="inbox"> first read Inbox[i] (messages addressed to you or broadcast). Update accordingly. </div> <h2>Communication modes</h2> <div class="modes"> <ul> <li>Decentralized: you decide when to message.</li> <li>Centralized (hub=h): only agent h broadcasts role/assignment others reply sparsely.</li> </ul> </div> <h2>Messaging rules</h2> <div class="rules"> <ul> <li>Keep messages ≤ 25 tokens.</li> <li>Prefer structured intents: claim:<subgoal>, status:& block:<issue>, request:<help on X>, handoff: /role>.</li> <li>Addressing: use @all for broadcast or @j for a specific agent <li>Do not restate observations verbatim; send actionable deltas teammates' plans.</li> </ul> </div> <h2>Augmented OUTPUT (append to (2)'s JSON)</h2> <div class="json-block">"comm": "send": <true|false>, "to": "@all" | "@j" | "@hub", "message": "<one line following the intent tags above>" </div> </div> </body> 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 HTML Online Viewer (https://html.onlineviewer.net/)  Preview (Full page)  Highlight (Full page)  Format  Expand all  Collapse all  Sample  Clear  Import  Export  Privacy settings  About © 2025 OnlineViewer.net (https://w.onlineviewer.net) 1 of 1 streams All + .*Aa 9/25/25, 1:47 AMHTML Online Viewer https://html.onlineviewer.net1/1 Figure 7: Multi-agent prompt (to be appended to single-agent prompt). •Protocol: Curate a fixed subset of multiplayer levels (e.g., 48 classic levels) and evaluate multiple seeds/player configurations; require that all agents reach the goal to clear a level. •Metric: Level completion rate and median completion time across the suite; optionally, coordination error counts (drops, desyncs). MindCraft (Minecraft Multi-Agent Collaboration) • Goal: Multi-agent embodied collaboration on Cooking, Crafting, and Construction tasks via language-enabled coordination. •Obs/Act: First-person Minecraft control with inventories; agents exchange natural- language messages; tasks provide recipes/blueprints and split resources/knowledge across teammates. •Protocol: Procedurally generated tasks per category; for construction, initialize agents with blueprints and disjoint materials/skills; for cooking/crafting, vary recipe complexity and information asymmetry (“Hell’s Kitchen” variants). •Metric: Average success rate for Cooking & Crafting; Construction uses an edit-distance alignment between the built structure and the target blueprint; overall score averages category scores. 22 Published as a conference paper at COLM 2026 C Algorithm Algorithm 1 Online planning at timestep t 1:Input: environmentP, controllerπ c , VLM plannerπ p ,Stop ∈ 0, 1,Inbox ∈ X,Plan ∈X , latest observation o t . 2: thread Controller(): 3:while running do 4: ( a t ,s t ) ← π c .step ( o t ,x t ) 5:P.step(a t ) 6:if s t then Stop← 1; x t+1 ←∅ 7:if Inbox then x t+1 ← Inbox 8: thread VLM(): 9:while running do 10:if Stop then 11: Inbox← π p .emit(o t ,Plan) 12: Stop← 0; Plan←∅ 13:else 14: Plan← π p .advance(o t ,Plan) Algorithm 2 Offline planning at timestep t 1:Input: environmentP, controllerπ c , VLM plannerπ p ,Stop∈0,1, latest observation o t , plan state Pn t . 2: while running do 3: ▷Add to the current plan, potentially emitting an instruction. 4: (x t ,Pn t+1 )← π p .step(o t ,Pn t ,Stop) 5:if x t then Stop← 0 6: ▷Execute the existing instruction. 7: (a t ,s t )← π c .step(o t ,x t ) 8:P.step(a t ) 9:if s t then Stop← 1 Shared-state semantics. Inbox, Stop, and Plan are synchronized through step-indexed buffers rather than arbitrary cross-thread writes. The controller writes a completion event whenp stop crosses its threshold. If this happens while the planner is mid-generation, the planner finishes its current ADVANCE call using the latest available observation context and stores the resulting draft plan in Plan. At the planner ’s next scheduling point, the completion event triggers a short EMIT call that converts the draft into the next executable instruction and writes it atomically to Inbox; the controller takes in the latest instruction at the next control step. Observation staleness is therefore bounded by the duration of the planner’s current background reasoning call plus one control-step boundary, and the controller continues executing its previous instruction throughout, so staleness affects only the next high-level instruction and never low-level control. In terms of token cost, ADVANCE carries most of the reasoning tokens, as it updates the planner’s memory and partial plan while the controller acts, whereas EMIT only decodes a short instruction from the existing draft. This asymmetry is why asynchronous planning reduces instruction-boundary latency relative to blocking inference (quantified in Table 9). Multi-agent chatroom synchronization. Each agent runs its own asynchronous planner thread and controller thread. Planner outputs, including chat messages and instructions, are never injected into the environment at arbitrary wall-clock times. Instead, they are written to a per-agent mailbox and consumed at the next synchronized communication/control boundary, at which point each agent reads the latest incoming messages and updates its planner context. Communication thus proceeds in discrete rounds aligned with control steps while planner reasoning proceeds asynchronously in the background. Planner observability. In all environments, the planner receives only the observations available to the agent itself. No privileged simulator state, full-state access, or hidden information is provided, ensuring the framework applies to settings where privileged information is unavailable. D Instruction-following evaluation protocol. Instruction-following accuracy was judged by one of the authors on 200 held-out planner- generated instructions per domain. The evaluator was shown the instruction, the correspond- ing observation/trajectory segment, and a real-time video of the controller’s execution, and 23 Published as a conference paper at COLM 2026 judged whether the behavior completed the intended instruction. As a sanity check, we re- peated the same annotation task with an API model (GPT-5.5), which agreed with the human annotator on 93.2% of cases across tasks. We emphasize that single-annotator judgment com- bined with model agreement serves as a sanity check rather than a substitute for independent multi-annotator human evaluation, and we present these accuracies with that caveat. D.1 Additional Training and Communication Details When collecting a rollout of lengthT, different timestepstmay be associated with different instruction embeddingse t and completion indicatorscomplete t , whereasdone t is true if and only ift = T. To sample annotation segments, a segment lengthLis first randomized within an integer interval, then a start index is drawn ast∼ Uniform1,...,|D|−L; sampled segments are constrained to be non-overlapping. Formally, for multi-agent communication withnplanner agents, at each VLM inference step any plannerimay initiate at most one messagem ij to plannerjin the decentralized mode. In the centralized mode, the sender is a fixed hub agenththat sends messagesm hi to alli∈[n]\ h. 24

Instant research discovery