Paper deep dive
WCM: World-Cognition Model for Generalizable Human-Robot Interaction
Yuzhen Chen, KC Zhou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language agents can now interact fluently with users in software, but robots still struggle to bring comparable interaction to physical tasks. Current robot-control paradigms, including vision-language-action policies and world-model-based planners, are mainly optimized for instruction execution, leaving users with little visibility into why an action is chosen and few mechanisms to redirect, correct, or teach the robot through interaction. To solve this problem, we present the World-Cognition Model (WCM), a human-centered embodied agent built on the SLAK architecture (Sensing, Logic, Action, and Knowledge) and an asynchronous runtime. SLAK separates perception, reasoning, control, and memory, while the runtime allows reasoning, dialogue, and execution to proceed concurrently. WCM further introduces a human-in-the-loop teaching mode that enables users to interactively teach the robot difficult or long-horizon tasks. Teaching episodes and autonomous task rollouts are refined into chain-of-thought supervision to continually improve the model. WCM achieves a 73.8% average success rate across nine real-world human-robot interaction tasks, including tasks held out from CoT fine-tuning and a long-horizon task learned through teaching.
Tags
Links
- Source: https://arxiv.org/abs/2607.22999v1
- Canonical: https://arxiv.org/abs/2607.22999v1
Trouble viewing inline? Open PDF directly →
Full Text
35,428 characters extracted from source content.
Expand or collapse full text
WCM: World-Cognition Model for Generalizable Human–Robot Interaction Yuzhen Chen1, KC Zhou2 Abstract Language agents can now interact fluently with users in software, but robots still struggle to bring comparable interaction to physical tasks. Current robot-control paradigms, including vision-language-action policies and world-model-based planners, are mainly optimized for instruction execution, leaving users with little visibility into why an action is chosen and few mechanisms to redirect, correct, or teach the robot through interaction. To solve this problem, we present the World-Cognition Model (WCM), a human-centered embodied agent built on the SLAK architecture (Sensing, Logic, Action, and Knowledge) and an asynchronous runtime. SLAK separates perception, reasoning, control, and memory, while the runtime allows reasoning, dialogue, and execution to proceed concurrently. WCM further introduces a human-in-the-loop teaching mode that enables users to interactively teach the robot difficult or long-horizon tasks. Teaching episodes and autonomous task rollouts are refined into chain-of-thought supervision to continually improve the model. WCM achieves a 73.8% average success rate across nine real-world human–robot interaction tasks, including tasks held out from CoT fine-tuning and a long-horizon task learned through teaching. I Introduction Language has become a common interface for robot control, but following language instructions is not the same as human–robot interaction. In software, language agents allow users to refine goals, ask for explanations, and steer a task as it unfolds. In embodied systems, however, language is still often treated as an input condition for action generation rather than a persistent channel for interaction, explanation, and feedback-driven learning. As a result, even robots that can execute language instructions offer limited support for mid-execution redirection, human-readable explanations, or learning from user feedback. Vision-language-action (VLA) and robot foundation models [6, 37, 22, 15, 4, 3, 2] and recent world-model systems for physical AI, robot imagination, and manipulation [16, 32, 36, 7, 1, 21, 34] expose this gap in different ways. VLA policies can accept language commands, but language usually serves as a conditioning signal for action prediction rather than a persistent channel through which a user can intervene, revise the goal, or teach the robot during execution. Their decisions are also difficult to inspect: even when a pretrained vision-language backbone provides semantic understanding, the final motion is produced by an action decoder or policy, leaving the link between high-level reasoning and physical behavior implicit. World-model-based planners face a related limitation. They reason through predicted future states or imagined rollouts, which can support planning, but these rollouts are not by themselves human-readable explanations, nor do they naturally provide a mechanism for real-time conversational correction. Thus, both families advance language-conditioned execution, but neither makes interaction, legibility, and feedback-driven learning a first-class part of the control loop. Interactive robots therefore require more than a stronger language-conditioned policy: perception, reasoning, action, memory, and human feedback must remain connected throughout execution. The robot should ground a user’s instruction in the current scene, expose the reason behind its next action, revise that action when the user intervenes, and turn interaction itself into future training signal [10, 17]. We instantiate this view in the World-Cognition Model (WCM), a human-centered embodied agent built on the SLAK architecture and an asynchronous runtime. SLAK separates Sensing, Logic, Action, and Knowledge, making perception, reasoning, control, and memory explicit rather than hidden inside a single opaque policy. The runtime allows reasoning, dialogue, and execution to proceed concurrently, while WCM converts autonomous rollouts and human teaching episodes into chain-of-thought supervision for continual improvement [31, 18]. We evaluate WCM on a low-cost functional mobile-manipulation platform and propose nine real-world human–robot interaction tasks covering object retrieval, handover, tool use, drawer manipulation, and trash disposal. WCM achieves a 73.8% average success rate, including tasks held out from CoT fine-tuning and a long-horizon task learned through teaching. Our contributions are threefold. First, we introduce WCM, an interactive embodied system with explicit SLAK layers and asynchronous execution. Second, we propose a human-in-the-loop teaching and CoT distillation pipeline that turns autonomous rollouts and teaching episodes into training signal. Third, we build a low-cost mobile-manipulation platform and introduce a nine-task real-world HRI suite for evaluating language-guided interaction, real-time correction, and task learning. I Method: The World-Cognition Model I-A System Architecture and Runtime I-A1 SLAK Architecture Figure 1: The SLAK architecture. WCM organizes sensing, logic, action, and knowledge as explicit layers connected to a CoT distillation pipeline. Autonomous executions and human teaching episodes are refined into CoT training data for continual improvement. SLAK is the core system architecture of WCM, consisting of four explicit layers: Sensing, Logic, Action, and Knowledge (Fig. 1). These layers separate scene understanding, reasoning, robot control, and memory while communicating through explicit state exchange. This design lets WCM update the scene, goal, memory, and action outcomes during execution, rather than hiding the full perception-to-action process inside a single end-to-end policy [37, 22, 4]. Sensing layer The Sensing layer converts multimodal sensor streams into an interaction-aware scene state. The right side of Fig. 2 visualizes this conversion from raw RGB-D observations to a structured scene state with semantic and part-level interaction cues. It integrates vision, depth, audio, IMU, and force/torque signals to maintain local 3D structure, object relations, occupancy, and semantic cues around the robot. Unlike streaming 3D perception methods that primarily reconstruct geometry from image sequences [33, 35], or open-vocabulary grounding and segmentation models that localize language-specified concepts as boxes or masks [25, 27, 8], WCM estimates the scene properties needed for robot interaction. These include pixel- and part-level cues for grasping, opening, placement, and disposal, as well as coarse physical and material attributes such as rigidity, fragility, and grasp suitability [9, 14]. The resulting state informs Logic about available local interactions and the physical constraints that should shape the next action. Logic layer The Logic layer is WCM’s reasoning, planning, and decision-making module. It conditions each decision on the user’s current instruction, the structured scene state from Sensing, and the relevant memory retrieved from Knowledge. Rather than committing to a complete task script upfront, Logic follows an incremental plan–reason–validate loop. At step t, Logic forms a decision context xt=(st,ut,mt)x_t=(s_t,u_t,m_t), where sts_t denotes the structured scene state, utu_t the user’s current instruction or goal, and mtm_t the relevant task memory retrieved from Knowledge. Given this context, Logic produces a reasoning trace, the next action, and its expected outcome: (rt,at,o^t)=fθ(xt),vt=1,Δ(ot,o^t)≤ϵ,0,otherwise.(r_t,a_t, o_t)=f_θ(x_t), v_t= cases1,& (o_t, o_t)≤ε,\\ 0,&otherwise. cases (1) Here, fθf_θ is the Logic model, θ denotes its parameters, oto_t is the realized outcome returned after execution, Δ measures the mismatch between expected and realized outcomes, and ϵε is a validation threshold. If vt=1v_t=1, the task proceeds; otherwise Logic re-reasons from the updated state or asks the user for clarification. This stepwise structure keeps decisions grounded in the current scene, exposes the robot’s intent and expected outcome in natural language, and limits error propagation when the environment changes or the user redirects the task [20, 13, 19]. Knowledge layer The Knowledge layer stores and updates the state and experience WCM accumulates during task execution and long-term use. It maintains short-term task memory, including the current goal, observed objects, spatial context, recent actions, outcomes, and user feedback, as well as longer-term knowledge such as task experience, object-interaction history, and skill memory. This memory allows WCM to preserve context across a continuous interaction rather than treating each decision as an isolated response to the current observation. Action layer The Action layer converts Logic’s selected step into executable robot behavior. Given a step and its expected outcome, Action binds it to the appropriate low-level skill or policy, sequences the required motion commands, and executes them on the robot hardware. These skills cover navigation, reaching, grasping, placing, handover, and simple articulated-object interactions, and are reused across tasks rather than trained as task-specific policies. During execution, Action reports status, failures, and realized outcomes back to Logic and Knowledge for validation and replanning when necessary. This separation keeps high-level reasoning independent from the underlying controller, allowing the action library or hardware stack to be improved without changing the reasoning interface [23, 28]. During execution, Sensing, Logic, and Action write scene changes, active goals, reasoning states, action outcomes, failures, and user feedback into Knowledge as a shared task state. Logic retrieves this state when deciding the next step, keeping execution, interaction feedback, and later learning connected within the same system loop. Taken together, SLAK makes WCM’s execution state explicit across sensing, reasoning, action, and memory. This explicit state interface provides the basis for the asynchronous runtime, which coordinates concurrent reasoning, execution, and state updates while preserving a shared task context. I-A2 Asynchronous Runtime Building on SLAK’s explicit state interface, the asynchronous runtime decouples reasoning, execution, and state updates during task execution. In a conventional robot-control pipeline, perception, reasoning, action, and feedback are processed sequentially, so model inference can block the robot from acting or delay its response to new user commands. WCM instead allows the reasoner to propose actions, the executor to carry out committed actions on the hardware, and the state updater to incorporate new observations, outcomes, and user feedback without forcing these processes to run in a single blocking sequence. This asynchronous design does not mean that WCM blindly follows an outdated action queue. Before committing queued actions and after execution, the runtime applies the validation check in Eq. (1) using the latest scene state, user input, and realized outcome. If the context no longer matches the expectation, stale actions are discarded and Logic re-reasons from the updated state. This decoupling preserves responsiveness in dynamic environments, allowing the robot to move, explain, listen for corrections, and update the task plan as the situation changes. I-B CoT Distillation from Rollouts and Teaching To connect execution with learning, WCM converts autonomous rollouts and human-guided teaching episodes into step-level CoT supervision. For an episode of length T, each decision record stores the scene, instruction, memory, reasoning, selected action, expected outcome, and realized outcome: e=dtt=1T,dt=(st,ut,mt,rt,at,o^t,ot),e=\d_t\_t=1^T, d_t=(s_t,u_t,m_t,r_t,a_t, o_t,o_t), (2) where sts_t is the scene state, utu_t the user instruction, mtm_t the retrieved memory, rtr_t the Logic reasoning, ata_t the selected action, o^t o_t the expected outcome, and oto_t the realized outcome. These records come from two sources: autonomous executions, including both successes and failures, and human-guided teaching episodes for difficult or long-horizon tasks. Let autoD_auto and teachD_teach denote the corresponding decision records. The offline refinement pipeline produces refined CoT examples auto′=ℛ(auto)D _auto=R(D_auto) and teach′=ℛ(teach)D _teach=R(D_teach), where ℛR denotes the refinement procedure. Because teaching episodes contain direct human guidance and clearer task decomposition, we weight them more heavily during distillation. Let ℒauto(θ)L_auto(θ) and ℒteach(θ)L_teach(θ) denote the CoT negative log-likelihood losses over auto′D _auto and teach′D _teach, respectively. The weighted distillation objective is ℒdistill(θ)=ℒauto(θ)+λℒteach(θ),L_distill(θ)=L_auto(θ)+ _teach(θ), (3) where λ≥1λ≥ 1 controls the relative weight of human-guided teaching examples. WCM minimizes this objective to update the reasoning model. I-B1 Autonomous Data Flywheel During autonomous execution, WCM records the reasoning-action process behind each completed or failed task. Successful episodes provide positive evidence for effective task decomposition, grounding, action selection, and outcome validation, but their raw traces may still contain redundant reasoning, indirect steps, or unnecessary recovery attempts. The refinement pipeline therefore turns successful executions into cleaner and more direct CoT traces, making the underlying logic easier for the model to imitate. Failed episodes are also useful, but not as positive demonstrations directly. Instead, they expose where the model misunderstood the scene, selected an ineffective action, predicted the wrong outcome, or repeated an unnecessary recovery step. These failure records contribute in two ways: they are stored in Knowledge as reusable experience for future reasoning, and they are sent to the offline refinement pipeline, where the failed reasoning path is converted into a more targeted corrected CoT trace for distillation. Unlike VLA-style training on teleoperated state-action trajectories [37, 22, 4], WCM uses the decision records in Eq. (2) as reasoning supervision rather than motor-action supervision. Raw execution records are therefore not used directly for training. Real robot runs can contain redundant reasoning, repeated attempts, indirect action choices, or ineffective intermediate actions, even when the final task succeeds. A stronger model ensemble reviews the recorded reasoning and outcomes, removes unnecessary detours, corrects inconsistent decision paths when needed, and produces cleaner reasoning traces through voting or consensus. The result preserves the causal structure of real execution while turning noisy robot behavior into higher-quality CoT supervision [31, 26, 24, 29]. The refined autonomous examples are distilled under the shared objective in Eq. (3), closing the autonomous data flywheel [5, 30]: execution produces decision records, refinement improves the supervision, and distillation improves later executions. Over time, WCM accumulates reusable reasoning patterns for object interaction, action sequencing, outcome prediction, and failure recovery, rather than memorizing a single task script. I-B2 Human-in-the-Loop Teaching Mode The autonomous data flywheel improves WCM from its own successes and failures, but some unseen or long-horizon tasks remain inefficient to discover through trial and error alone. For such cases, WCM introduces a human-in-the-loop teaching mode. Rather than requiring low-level teleoperation, the user teaches through natural language, guiding the robot step by step by indicating what to attend to, which object or part to act on, and when to advance to the next stage. Figure 2: Human-in-the-loop teaching and sensing output. Left: natural-language teaching of a long-horizon task. Right: raw RGB-D observations and the corresponding interaction-aware scene state. The episode is refined and distilled with autonomous rollouts. During teaching, WCM records the same step-level decision format as autonomous execution, augmented with explicit human guidance and feedback. These examples are often more informative than unguided rollouts because they provide clearer task decomposition, direct correction signals, and human choices about how a difficult task should be staged. This makes them especially useful for long-horizon or previously unseen tasks, where autonomous exploration alone may be slow or brittle. Teaching episodes enter the same refinement pipeline and shared distillation objective as autonomous execution records, but are weighted by λ in Eq. (3) because they provide direct human guidance. In the short term, taught procedures are stored in the Knowledge layer for reuse on similar tasks. In the long term, refined teaching examples are distilled into the reasoning model, allowing the robot to internalize human guidance as reusable task competence [31, 18, 11, 12]. I Experiments We evaluate WCM on a low-cost mobile-manipulation platform and a nine-task real-world HRI suite, measuring task success, held-out transfer, teaching-based skill acquisition, system efficiency, and ablated components. I-A Hardware WCM runs on a low-cost mobile-manipulation platform: a Mecanum-wheel base, a 5-DOF arm (∼ 1.3 m reach), and a 640×480640× 480 RGB-D camera. The complete robot costs under $2,000, while reasoning runs off-board on a shared NVIDIA RTX 5090 over the local network; we therefore report robot hardware cost separately from compute. I-B Nine-Task Human–Robot Interaction Suite We propose a nine-task real-world human–robot interaction suite to evaluate WCM beyond isolated manipulation primitives. Each task begins with a spoken instruction and requires language understanding, scene grounding, and physical execution. The suite covers object retrieval, handover, tool use, drawer manipulation, and trash disposal. Per-task success rates, trials, and run times are reported in Table I, with frame-by-frame demonstrations in Figs. 3–11 (Appendix A). The robot communicates in real time during these tasks: it listens, confirms, asks, and explains while acting. Four tasks are held out from CoT fine-tuning yet still succeed, demonstrating transfer beyond the training tasks. The hardest task, “Take the screwdriver to the drawer,” cannot be solved reliably before teaching; after interactive teaching it reaches 69%69\% success, and after distillation it rises to 82%82\%, testing the full teaching-to-distillation loop. I-C Case-Study Comparison We present a case-study comparison with X-Square’s WALL-OSS, reproducing its task suite as closely as our platform allows. Because the robots differ in body, kinematics, sensors, and execution stack, this comparison is illustrative rather than a controlled benchmark. Per-task speed, GPU occupancy, and success rates appear in Appendix B (Table I). I-D Ablations Finally, we ablate the asynchronous runtime and the interaction-aware Sensing layer on a shared five-task subset. Removing asynchrony forces the robot to wait between reasoning and action, increasing end-to-end run time by 1.41.4–1.7×1.7×. Replacing Sensing with a plain VLM and 2D grounding sharply reduces success on part-level interaction tasks; Screwdriver-to-Drawer falls from 69%69\% to 0%0\%. Results are reported in Appendix C (Table I). IV Conclusion We present WCM, a human-centered embodied agent that combines SLAK, asynchronous execution, and CoT distillation from rollouts and teaching. WCM maintains task context, explains intent, responds to mid-execution corrections, and improves from interaction. On a low-cost mobile-manipulation platform, WCM achieves a 73.8% average success rate across nine real-world tasks, including held-out tasks and a long-horizon task learned through teaching. While currently limited by hardware and a case-study evaluation, WCM points toward robots that can act, explain, be corrected, and improve with people in the loop. Acknowledgments We thank T. Lihe and Andrew Hoang for implementation support. This work was funded by Cyberbrain, Inc.; the system, datasets, and related IP are owned by Cyberbrain, Inc. References Agarwal et al. [2025] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025. Bjorck et al. [2025] Johan Bjorck, Fernando Castaneda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. Black et al. [2025a] Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, et al. π0.5 _0.5: A vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 17–40. PMLR, 2025a. Black et al. [2025b] Kevin Black, Noah Brown, Danny Driess, et al. π0 _0: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025b. arXiv:2410.24164. Bousmalis et al. [2023] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, et al. Robocat: A self-improving generalist agent for robotic manipulation. Transactions on Machine Learning Research, 2023. arXiv:2306.11706. Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023. arXiv:2212.06817. Bruce et al. [2024] Jake Bruce, Michael D Dennis, Ashley Edwards, et al. Genie: Generative interactive environments. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 4603–4623. PMLR, 2024. arXiv:2402.15391. Carion et al. [2026] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. Sam 3: Segment anything with concepts. In International Conference on Learning Representations (ICLR), 2026. arXiv:2511.16719. Chen et al. [2025] Yuzhen Chen, Hojun Son, and Arpan Kusari. Matpredict: A dataset and benchmark for learning material properties of diverse indoor objects. arXiv preprint arXiv:2505.13201, 2025. Clark and Brennan [1991] Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley, editors, Perspectives on Socially Shared Cognition, pages 127–149. American Psychological Association, 1991. Deng et al. [2023] Yuntian Deng, Kiran Prasad, Roland Fernandez, et al. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460, 2023. Deng et al. [2024] Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838, 2024. Driess et al. [2023] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, et al. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8469–8488. PMLR, 2023. arXiv:2303.03378. Ewen et al. [2024] Parker Ewen, Hao Chen, Yuzhen Chen, Anran Li, Anup Bagali, Gitesh Gunjal, and Ram Vasudevan. You’ve got to feel it to believe it: Multi-modal bayesian inference for semantic and property prediction. In Robotics: Science and Systems (RSS), 2024. arXiv:2402.05872. Ghosh et al. [2024] Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, et al. Octo: An open-source generalist robot policy. In Robotics: Science and Systems (RSS), 2024. arXiv:2405.12213. Ha and Schmidhuber [2018] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. Horvitz [1999] Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pages 159–166, 1999. Hsieh et al. [2023] Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, et al. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics (ACL), pages 8003–8017, 2023. arXiv:2305.02301. Huang et al. [2023] Wenlong Huang, Fei Xia, Ted Xiao, et al. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1769–1782. PMLR, 2023. arXiv:2207.05608. Ichter et al. [2023] Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 287–318. PMLR, 2023. arXiv:2204.01691. Jang et al. [2025] Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, et al. Dreamgen: Unlocking generalization in robot learning through video world models. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 5170–5194. PMLR, 2025. Kim et al. [2025] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, et al. OpenVLA: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2679–2713. PMLR, 2025. arXiv:2406.09246. Liang et al. [2023] Jacky Liang, Wenlong Huang, Fei Xia, et al. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500, 2023. arXiv:2209.07753. Lightman et al. [2024] Hunter Lightman, Vineet Kosaraju, Yura Burda, et al. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050. Liu et al. [2024] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), pages 38–55, 2024. arXiv:2303.05499. Nye et al. [2021] Maxwell Nye, Anders Andreassen, Guy Gur-Ari, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. Ren et al. [2024] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024. Sutton et al. [1999] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2):181–211, 1999. Uesato et al. [2022] Jonathan Uesato, Nate Kushman, Ramana Kumar, et al. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. Wang et al. [2024] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 51936–51983. PMLR, 2024. Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837, 2022. arXiv:2201.11903. Wu et al. [2023] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 2226–2240. PMLR, 2023. Zhou et al. [2026a] Kaichen Zhou, Zeyang Bai, Xinhai Chang, Mengyu Wang, Paul Pu Liang, and Fangneng Zhan. Stream3d: Sequential multi-view 3d generation via evidential memory. arXiv preprint arXiv:2605.21472, 2026a. Zhou et al. [2026b] Kaichen Zhou, Yuzhen Chen, Fangneng Zhan, Hang Hua, Grace Chen, Xinhai Chang, Ao Qu, Yilun Du, Zhuang Liu, Paul Pu Liang, et al. Gem-4d: Geometry-enhanced video world models for robot manipulation. arXiv preprint arXiv:2605.22882, 2026b. Zhou et al. [2026c] Kaichen Zhou, Yuhan Wang, Grace Chen, Gaspard Beaudouin, Fangneng Zhan, Paul Pu Liang, and Mengyu Wang. Page-4d: Disentangled pose and geometry estimation for vggt-4d perception. In International Conference on Learning Representations (ICLR), 2026c. arXiv:2510.17568. Zhou et al. [2024] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61885–61896. PMLR, 2024. Zitkovich et al. [2023] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR, 2023. arXiv:2307.15818. Appendix A Human–Robot Interaction Demos TABLE I: Human–robot interaction demos. Per-task success rate (%) of WCM on nine live interaction tasks, each coupling dialogue with physical handover to a person. Task labels are the spoken instructions of the matching demo figures (Figs. 3–11). Because conventional VLA policies typically do not treat continuous dialogue, interpretable interruption, and teaching mode as core closed-loop capabilities, we report WCM only. Spoken instruction SR (%) Trials Time (s) “Grab me the popcorn.” 85 11/13 72 “Hand me the honey.” 79 11/14 34 “Hand me the jacket.” 67 6/9 54 “Hand me the green ratchet.” 64 7/11 36 “Hold the BBQ tongs.” 86 12/14 48 “Grab me the water bottle.” 831 10/12 30 “Take the screwdriver to the drawer.” 692 9/13 165 “Put the cup in the trash can.” (Outdoor) 671 10/15 46 “Put the bottle in the trash can.” 641 9/14 34 1 Held out from the CoT training set; solved zero-shot by transfer. 2 Held out and acquired through teaching mode: the reported 69%69\% is post-teaching but pre-distillation (below 20%20\% before teaching, 82%82\% after CoT distillation). Trials reports successful / total attempts; SR is their ratio. Time is a typical end-to-end run time (seconds), from robot-motion onset to task completion; values are approximate. Figures 3–11 show representative frame sequences from the live demos summarized in Table I, each capturing the dialogue-driven handover from the robot’s first look at the scene through grasp and delivery to the person. Figure 3: “Grab me the popcorn.” Object retrieval and handover. Figure 4: “Hand me the honey.” The robot locates the honey, grasps it, and hands it to the person. Figure 5: “Hand me the jacket.” Grasping a deformable object and delivering it to the person. Figure 6: “Hand me the green ratchet.” Retrieving a tool in a garage setting. Figure 7: “Hold the BBQ tongs.” Grasping and presenting the tongs to the person. Figure 8: “Grab me the water bottle.” Object retrieval and handover. Figure 9: “Take the screwdriver to the drawer.” A two-stage manipulation: open the drawer, then place the screwdriver inside. Figure 10: “Put the cup in the trash can.” (Outdoor) A multi-step task: pick up the cup, navigate outside, and dispose of it. Figure 11: “Put the bottle in the trash can.” A multi-step task: pick up the bottle, locate the trash can, and dispose of it. Appendix B Performance Comparison Results TABLE I: Case-study comparison. Per-task comparison of WCM (Ours) against X-Sq. (X-Square Robot; model: WALL-OSS) on six tasks across three metrics. Arrows give the better direction; bold marks the better method per compared cell. Movement time (s) ↓ GPU busy time (s) ↓ Success (%) ↑ Task WCM X-Sq. WCM X-Sq. (est.) WCM ColorPlateSort 25.0 109.0 19.4 109.0 92 LabelRecognition 36.0 116.0 18.4 116.0 90 PickUpFruits 16.6 18.5 6.4 18.5 90 ThrowTrash 80.5 258.0 22.5 258.0 86 PickUpCube 16.1 43.2 5.6 43.2 92 SolveMathProblem 57.0 76.0 59.9 76.0 78 Average 38.5 103.5 22.0 103.5 88.0 Movement time is measured from the onset of robot motion to completion of the target task, matching the reference videos of the baseline (which start when the robot begins to move). GPU busy time is the wall-clock time the GPU spends computing for the task. A VLA runs on the GPU at every control step, so for X-Square we estimate its GPU busy time as the full movement time (continuous occupancy); WCM uses the GPU only intermittently, freeing it for other robots. WCM also spends 33–55\,s on a post-task summary, so its GPU stays busy slightly past the end of motion, which is why GPU busy time can exceed movement time (e.g., SolveMathProblem, 59.959.9 vs 57.057.0\,s). Success. X-Square did not provide per-task success rates, so the success column reports WCM only. Appendix C Ablations TABLE I: Ablations. Removing the asynchronous runtime (the robot must halt to think between actions) inflates end-to-end run time; replacing INTERACTION-AWARE SENSING LAYER with a plain VLM and 2D grounding (no part-level interaction cues) reduces the success rate. Both are evaluated on the same five tasks. “WCM” is the full system—its times and success rates match Table I; arrows give the better direction and bold marks the full system. Time (s) ↓ Success (%) ↑ Task WCM w/o Async WCM w/o Sensing “Hand me the green ratchet.” 36 61 64 18 “Grab me the water bottle.” 30 47 83 27 “Take the screwdriver to the drawer.” 165 254 69 0 “Put the cup in the trash can.” (Outdoor) 46 63 67 18 “Put the bottle in the trash can.” 34 49 64 9