Paper deep dive

Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method

Yunzhe Xu, Zhe Liu

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 193

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 9/30/2026, 7:40:50 AM

Summary

This paper introduces Systematic Multi-Agent Vision-and-Language Navigation (MAVLN), a new benchmark and task formulation for coordinating multiple agents in unseen environments. It addresses limitations in existing multi-agent VLN by introducing complex dependency and resource constraints (presence locks, holding chains). The authors present MAVLN, a benchmark with 11,724 episodes across 145 scenes, and TRISS, a coordination-ready navigation system utilizing an LLM-based scheduler, shared topological memory, and conflict-aware execution.

Entities (8)

MAVLN · benchmark · 98%TRISS · methodsystem · 97%Vision-and-Language Navigation · task · 95%Habitat 3 · platformenvironment · 90%HM3D · dataset · 90%Gemini-3 · model · 85%CoNavBench · benchmark · 80%DeCoNav · method · 80%

Relation Signals (10)

MAVLN → contains → 11,724 episodes

confidence 95% · MAVLN, comprising 11,724 episodes across 145 scenes

MAVLN → isformulationof → Systematic Multi-Agent Vision-and-Language Navigation

confidence 95% · We present Systematic Multi-Agent Vision-and-Language Navigation... instantiates the task as MAVLN

TRISS → isproposedfor → MAVLN

confidence 95% · We further present TRISS... Extensive experiments establish TRISS as a comprehensive baseline

MAVLN → hasconstraint → holding chains

confidence 90% · each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains)

MAVLN → hasconstraint → presence locks

confidence 90% · each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains)

TRISS → usescomponent → LLM-based subtask scheduler

confidence 90% · TRISS, a coordination-ready navigation system coupling an LLM-based subtask scheduler

TRISS → usescomponent → shared topological memory

confidence 90% · coupling... a shared topological memory that turns each agent's exploration into team knowledge

TRISS → usescomponent → conflict-aware execution mechanism

confidence 90% · and a conflict-aware execution mechanism that realizes simultaneous intentions as collision-free routes

MAVLN → usesdataset → HM3D

confidence 90% · Our platform is built on Habitat 3 with the HM3D dataset

MAVLN → usesplatform → Habitat 3

confidence 90% · a synchronized multi-agent VLN platform built on Habitat 3

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Vision-and-Language Navigation (VLN) has largely focused on a single agent following a single instruction, yet many real-world applications require teams of robots to tackle tasks beyond the capabilities of any individual agent. We present Systematic Multi-Agent Vision-and-Language Navigation, providing, to our knowledge, the first systematic formalization of multi-agent VLN as a constrained coordination problem: each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains). A verified four-stage crafting pipeline instantiates the task as MAVLN, comprising 11,724 episodes across 145 scenes with teams of up to four agents under three instruction regimes, accompanied by tailored constraint-aware metrics. We further present TRISS, a coordination-ready navigation system coupling an LLM-based subtask scheduler, a shared topological memory that turns each agent's exploration into team knowledge, and a conflict-aware execution mechanism that realizes simultaneous intentions as collision-free routes. Extensive experiments establish TRISS as a comprehensive baseline and reveal substantial room for improvement across scheduling, planning, and execution, highlighting the challenges of coordinating under MAVLN task constraints. Project page: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

190,181 characters extracted from source content.

Expand or collapse full text

1 Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method Yunzhe Xu and Zhe Liu Abstract—Vision-and-Language Navigation (VLN) has largely focused on a single agent following a single instruction, yet many real-world applications require teams of robots to tackle tasks beyond the capabilities of any individual agent. We present Systematic Multi-Agent Vision-and-Language Navigation, providing, to our knowledge, the first systematic formalization of multi-agent VLN as a constrained coordination problem: each mission consists of subtasks carrying dependency and resource constraints (presence locks and holding chains). A verified four-stage crafting pipeline instantiates the task as MAVLN, comprising 11,724 episodes across 145 scenes with teams of up to four agents under three instruction regimes, accompanied by tailored constraint-aware metrics. We further present TRISS, a coordination-ready navigation system coupling an LLM-based subtask scheduler, a shared topological memory that turns each agent’s exploration into team knowledge, and a conflict-aware execution mechanism that realizes simultaneous intentions as collision-free routes. Extensive experiments establish TRISS as a comprehensive baseline and reveal substantial room for improvement across scheduling, planning, and execution, highlighting the challenges of coordinating under MAVLN task constraints. Project page: https://xyz9911.github.io/mavln. Index Terms—Vision-and-language navigation, multi-agent systems, embodied AI, benchmark, memory mechanisms. ✦ 1 INTRODUCTION V ISION-and-Language Navigation [1], a cornerstone challenge in Embodied AI, has witnessed rapid progress [2], [3], [4]. Under the original VLN setting, an agent follows an instruction and navigates within a pre- defined topological graph to reach a nearby goal. Though the paradigm’s simplicity eases the integration of vision- and-language models, it limits practical relevance. Navi- gation in continuous environments [5] was therefore pro- posed to narrow this gap, and has been further extended to physically realistic settings [6], [7]. Moreover, long-horizon navigation [8] and memory-persistent navigation [9], [10] have been explored to carry out complex tasks comprising successive subtasks and to improve over time. Extending VLN to multi-agent scenarios is a natural next step toward general intelligence in the real world, as team collaboration reaches beyond single-agent capacity. For embodied multi-agent navigation, early attempts include collaborative exploration [11], scene understanding [12], manipulation [13], and single-target [14] or multi-target navigation [15]. Importantly, CoNavBench [16] presents a major step toward multi-agent VLN, treating two agents as a coordinated team to exploit parallelism and save wall- clock time under relay-based coordination, and DeCoNav [17] further propels this paradigm with dynamic task al- location and re-planning for real-time, adaptive coordina- tion. Meanwhile, Co-VLN [18] reveals that peer observa- tion sharing can significantly boost navigation performance, • The authors are with the School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: xyz9911@sjtu.edu.cn; liuzhesjtu@sjtu.edu.cn). demonstrating the benefit of memory-wise coordination. In task-planning domains, LLMs are widely used to allocate subtasks among multiple agents [19] or to facilitate inter- agent communication for informed decisions [20], while relying on oracle simulator functions [21] or pre-explored semantics [22] for perception and low-level control. Despite these advances, existing paradigms exhibit two critical limitations. First, current embodied multi-agent nav- igation tasks impose over-simplified constraints: the re- quired coordination rarely goes beyond chain-style depen- dencies, limited environmental coverage curtails the expe- rience reuse that benefits generalization, and inter-agent conflict is left unmeasured, undermining real-world feasi- bility. Second, planning-oriented systems are confined to known environments: although their task dependencies are commonly represented as directed acyclic graphs (DAGs), the reliance on pre-built reconstructions or simulator APIs sidesteps the challenge of generalizing to unseen environ- ments. The two lines have thus matured on complementary halves of the same real-world problem, while their intersec- tion remains unexplored. To this end, we propose Systematic Multi-Agent Vision- and-Language Navigation, an embodied navigation task re- quiring sophisticated coordination in unseen environments. As shown in Figure 1, a team of agents must first identify the dependency and resource constraints implied by the instruction (e.g., the magenta agent’s first subtask can only complete after the blue agent’s counterpart, while the blue agent must hold its position until the magenta agent’s com- pletion releases it), and then execute the subtasks separately to fulfill the mission, benefiting from shared experiences and minimizing physical conflicts. We further propose a veri- fied task crafting pipeline for multi-agent VLN, comprising arXiv:2609.35965v1 [cs.CV] 28 Sep 2026 2 Existing Paradigm 1: Multi-Agent Embodied Navigation Existing Paradigm 2: LLM-based Multi-Agent Planning Unseen Generalizable Simple Coordination No Resource Constraints No Conflict Formulation Dependency- Aware Goals Relies on APIs / Pre-Built Maps No Perception Grounding Limited Generalizability Systematic Multi-Agent Vision-and-Language Navigation Coordination in Unseen Environment Task Structure Proceed to the entrance and remain there to verify the manifest with the courier 1 until your partner arrives 1 ... Clear the nightstand in the central bedroom for incoming inventory 1 ... Once the package is opened, navigate to the living room table 2 , while the books should be delivered to the nightstand 2 ... 1: Dynamic Scheduling 2: Experience Sharing 3: Conflict Resolution Task DAG (example) success Proceed to the entrance, clear the nightstand and open the ... 1 Find the paper towel roll on the living room table and bring .... 2 3 45 go to entrance open package nightstand clearance go to living room deliver books Oracle Skills open package clean surface navigate to pick books 1 2 3 45 Dependency (must precede) Presence Lock (waiting) Holding Chain (continuity) 11 1 1 2 3 2 4 5 3 3 2 5 3 2 5 Holding chain / presence lock break Dependency / resource- respecting execution Observation reusing for informed planning Collisions due to temporal conflict Conflict-free path finding and execution Isolated exploration in unseen environments Fig. 1. Overview of Systematic Multi-Agent Vision-and-Language Navigation. Existing settings either pursue simple goals with underused cross- agent experience and no measure of physical conflict, or plan rich task structures through atomic navigation APIs in known environments. Our task grounds a single mission into a team of VLN agents in an unseen environment. It requires dynamic scheduling under dependency and resource constraints, experience sharing to help teammates plan in unexplored spaces, and conflict resolution to turn simultaneous intentions into collision- free routes. grounded scene understanding for multi-granularity scene annotation, diverse mission synthesis driven by user intents and zone features, mission instantiation and schedule re- finement for optimal task efficiency, and verified instruction rendering and filtering for fidelity control and reliable gen- eration. The pipeline yields MAVLN, a Systematic Multi- Agent VLN benchmark featuring DAG-structured subtask dependencies, resource constraints and conflict awareness. The benchmark comprises 11,724 episodes with flexible team sizes and three instruction regimes, and is accompa- nied by tailored constraint-aware evaluation metrics that measure coordination effectiveness. We identify three key challenges of the proposed task: dynamic subtask scheduling under dependency and re- source constraints, simultaneous experience sharing, and inter-agent conflict resolution. To tackle these challenges, we present Topological Memory-based Collaborative Nav- igation and Subtask Scheduling (TRISS), a coordination- ready navigation system that pairs a compact LLM with an efficient planner model: 1) an LLM-based subtask scheduler that identifies task constraints and iteratively dispatches atomic instructions to downstream navigators; 2) a topo- logical memory-based navigation planner that facilitates navigation planning with shared topological memory; and 3) a conflict-aware execution mechanism that couples opti- mal target allocation with conflict-free path finding while maintaining topological consistency. The experiments validate TRISS’s effectiveness and es- tablish it as a comprehensive baseline. Nevertheless, sub- stantial room for improvement remains across scheduling, planning, and execution, outlining concrete directions for future work. Our key contributions include: •Systematic Formulation of Multi-Agent VLN: To the best of our knowledge, we provide the first systematic formalization of multi-agent VLN as a dependency- and resource-constrained coordination problem. •Verified Multi-Agent Task Crafting: We present an automated generation pipeline that yields reliable multi-agent tasks through grounded perception, mis- sion synthesis, schedule refinement, and round-trip verification. •Comprehensive Benchmark and Platform: We re- lease MAVLN, comprising 11,724 episodes under three instruction regimes across 145 scenes, together with a synchronized multi-agent VLN platform built on Habitat 3 and tailored constraint-aware evalua- tion metrics. •Coordination-ReadyNavigationSystem:We present TRISS, a system coupling a compact LLM and an efficient planner through scheduling, shared memory, and conflict-aware execution. Experiments demonstrate its effectiveness while highlighting substantial headroom. 2 RELATED WORK 2.1 Vision-and-Language Navigation Traditional Vision-and-Language Navigation (VLN) [1], [5], [23] requires an agent to navigate to a destination based on an instruction, and methods have evolved through im- provements including memory mechanisms [2], [24], [25], navigation planning [10], [26] and data augmentation [3], [27]. The focus has shifted to LLM integration, training 3 large-scale navigation foundation models [4], [28] as VLN planners [29] to boost generalization capabilities in indoor [30], [31] and outdoor environments [32]. LLMs’ zero-shot capabilities [33], [34], [35] enable collaboration between LLMs and expert VLN models [36], [37]. However, tra- ditional VLN cannot represent real-world complexity in addressing human needs, and this limitation has motivated multi-target navigation [38], [39], long-horizon navigation [8], [9], [10] and physically realistic navigation [6], [7]. Multi- agent VLN has also been explored, ranging from early at- tempts at multi-expert debates for decisions [40] and collab- orative aerial search [41], [42] to memory-wise coordination [18] and relay-style collaboration [16], [17]. In these tasks, agents either pursue independent goals or cooperate on loosely structured team goals. Moreover, the collaboration benefit is often confined to either makespan optimization [16] or shared memory [18], leaving challenges including conflict resolution, experience reuse and scheduling either unaddressed or studied in isolation. 2.2 Multi-Agent Embodied Navigation Embodied collaborative navigation has shown promising results in various scenarios including exploration [11], scene understanding [12], instance localization [15] and manipu- lation [13]. Agents commonly collaborate to find multiple targets [15] or a shared single target [14]. In particular, lever- aging LLMs in collaborative navigation is becoming increas- ingly common, with protocols ranging from centralized co- ordination [14] and communication-based coordination [43] to memory-based decentralized coordination [44]. However, these tasks all focus on simple task goals, limiting their applicability to real-world scenarios. Task decomposition has been explored [45], but the dependencies are generally predefined and restricted to a fixed team size. Additionally, collision avoidance [46], [47] is seldom considered in em- bodied scenarios. 2.3 LLM-based Multi-Agent Planning SMART-LLM [19] pioneered the application of LLMs to multi-agent task planning for robotic tasks. These plan- ning styles range from leveraging linear programming [48], PDDL [49] and prompt optimization [50] to enhancing sys- tem capacity in task allocation and dependency identifica- tion [51]. CoELA [20] explores decentralized protocols in LLM-based agent coordination, and ProAgent [52] further enables cooperation without explicit communication. EMOS [21] extends predefined agent roles to physical composi- tions by proposing a heterogeneous multi-agent collabora- tion system. Importantly, LLaMAR [53] extends LLM-based multi-agent systems (MAS) to partially observable envi- ronments, but these MAS rely on either oracle perception to capture environment layouts or oracle skills to carry out low-level actions through simulator APIs, leaving the embodiment addressed only at the surface level. RoboOS- NeXT [22] integrates planning with low-level execution, yet it relies heavily on spatial memory initialization, forfeiting its generalization to unseen environments. 3 TASK FORMULATION & BENCHMARK 3.1 Systematic Multi-Agent VLN Vision-and-Language Navigation in Continuous Environ- ments (VLN-CE) [5] requires an agent to follow an in- struction ℓ and navigate using a set of low-level actions (move forward 0.25 m, turn left or right 15 degrees, or stop). At each timestep t, the agent perceives a panoramic observation O t = (o rgb i ,o dep i ) 12 i=1 comprising RGB and depth images, captured from 12 equally spaced horizontal heading directions. Inheriting the perception and action spaces of VLN-CE, we formalize multi-agent VLN as follows. An episode places a team of agents A = a 1 ,...,a M at distinct start poses and provides instructions L in one of three regimes (Sec- tion 3.3). The task is a DAG over subtasksT =τ 1 ,...,τ N , where each subtask τ k = (g k ,A k ,D k ,h k ,l k ,U k ) is visible only to the evaluator: g k is the goal grounded to one object instance, A k ⊆ A the agents permitted to execute it, D k ⊆ T its prerequisite subtasks, and (h k ,l k ,U k ) its resource attributes. If h k holds, the executing agent carries an object after completing τ k and may complete no subtask other than the consumer of τ k , by the same agent, forming a holding chain; if l k holds, the agent acquires a presence lock: the agent must maintain presence at g k until a teammate completes some τ j with τ k ∈ U j . Dependency and resource constraints are thus orthogonal: the former govern when subtasks may complete, the latter govern what an agent may do with itself while they are live. A subtask succeeds when a permitted agent declares arrival within the success radius of g k with all of D k completed and no holding or locking restriction violated; the mission succeeds if every subtask succeeds. 3.2 Verified Task Crafting We leverage the Gemini-3 series as the backbone LLM across four curation stages (Figure 2). Grounded Scene Understanding. Our platform is built on Habitat 3 [54] with the HM3D dataset [55]. We first render observations at each viewpoint based on the navigation graphs [3], and gather surrounding objects through the se- mantic sensor. We then prompt a VLM with RGB panorama splits and object cues to obtain descriptions containing ver- ified details of observable objects. Based on optimized zone partitions of HM3D scenes, we prompt an LLM with the aggregated viewpoint descriptions of each zone to obtain a general zone description along with a list of the objects available in it. Diverse Mission Synthesis. We first generate household profiles for diverse scenarios using an LLM. For each anno- tated zone, we extract its description and properties together with those of its neighbors, and prompt an LLM to pick the most suitable profile. Conditioned on the household characteristics and zone information, the LLM brainstorms candidate mission scenarios, each comprising a task de- mand and a list of subtasks with targets and constraints. Synthesized missions violating a set of well-formedness principles are filtered out. Mission Refinement. Based on the synthesized task infor- mation, we further prompt an LLM with the available view- points containing the targets of each subtask to obtain the 4 (a) Grounded Scene Understanding Offline Semantic Cues Object classes Visibility Position Geometry VLM RGB Panorama Viewpoint Card ·Visible objects within each view ·Grounded view descriptions ·Zone properties (object, viewpoint) ·Zone description (type, function) Zone Card Viewpoint-to-Zone Aggregation (b) Diverse Mission Synthesis Zone Cards User Profile Pool Mission Demand ·Task goals and purposes · Task constraints ·Required items / targets ·Involved Agents (c) Mission Refinement (d) Verified Instruction Rendering Subtask Trajectory Family with young children White-collar workers Professional roommates Subtasks w/ Constraints #agenttargetdeplockhold 1Adesk- ✔ 2Bbook[1] 3 Ccabinet[1] ... ✔ ✘ ✘ ✔ ✘ Mission Subtasks 1 desk ... #targetdemand check the laptop on ... sofa cabinet door timeline cabinet timeline sofadoor LLM Schedule Refinement LLM Instance Grounding TargetGoal vp12 Instruction Navigate to the desk and ... Refined Subtasks B desk ... agenttargetinstruction Navigate to the ...vp12 goal Subtask Instruction Navigate to the desk in the study and check the unread emails with the laptop on it. LLM Verified / refined atomic instruction Instruction Rendering Goal arrival sequence reconstruction Symbolic Constraint Checker Reject Pass Retry DecentralizedCentralizedCentralized implicit 1. 2. 3. Fig. 2. The crafting pipeline of MAVLN. (a) Descriptions are generated at viewpoint granularity and summarized into zone-level descriptions. (b) Mission scenarios are synthesized from a household profile and zone summaries, emitting constraint-annotated subtasks. (c) Subtask targets are grounded to viewpoints to generate atomic instructions, and the mission makespan is optimized. (d) A VLM verifies and refines each subtask instruction; consolidated instructions are rendered in three different styles. Crafted instructions pass a closed-loop filter comparing LLM- reconstructed arrival sequences against task constraints. destination viewpoint together with a fine-grained atomic instruction per subtask. The instantiated task then under- goes a makespan optimization phase that searches over constraint-consistent execution orders and agent allocations, mutating the drafted allocation to optimize the schedule. A subsequent filtering step eliminates consecutive subtasks that share the same viewpoint or lie too close to each other. Verified Instruction Rendering. We prompt a VLM with each subtask’s atomic instruction and observations rendered along the agent’s trajectory from start locations to subtask destinations to verify and refine the instruction. An LLM then consolidates the refined atomic instructions into task instructions of decentralized, centralized and centralized- implicit styles. Finally, an LLM reconstructs an ordered arrival sequence based on the instructions, which is checked against the ground-truth task constraints to certify fidelity; failed cases are given a limited number of retries before being filtered out. We further assess instruction reliability through human evaluation. 3.3 Dataset Statistics Table 1 compares MAVLN, our proposed Systematic Multi- Agent VLN benchmark, with existing collaborative em- bodied benchmarks. MAVLN is the first multi-agent VLN benchmark supporting flexible team sizes, and the first em- bodied navigation benchmark to introduce complex depen- dency and resource constraints as well as inter-agent con- flict measurement. It contains 11,724 collaborative episodes spanning 145 scenes with an average path length of 35.9 meters, providing sufficient area coverage in unseen envi- ronments. The episodes derive from 3,908 unique missions, each rendered under the three instruction regimes below; missions are partitioned scene-disjointly into 3,079 training (128 scenes), 386 validation-unseen (7 scenes), and 443 test- unseen (10 scenes) missions. The episodes are divided into three splits by instruction regime, each with val-unseen and test-unseen scenarios. The decentralized split distributes an explicit instruction to each agent, and every subtask permits only its designated agent as the executor. The centralized split provides one consoli- dated instruction specifying the whole team’s mission; un- like the decentralized split, every subtask permits any agent as an allowed executor. The centralized-implicit split further removes agent attribution from the centralized instruction, and the team must additionally infer who performs what. As illustrated in Figure 3, episodes carry five subtasks on average and up to eight, over half of them field three or more agents, and subtask counts grow with team size. Dependency edges are universal by construction, while holding chains appear in 92.9% of missions and presence locks in 28.3%, with 26.3% exercising both primitives at once and only 5.1% reducing to a pure DAG. Across suc- cessive subtasks, a team spends approximately a quarter of its total travel distance revisiting previously traversed paths, with this proportion increasing for longer trajectories. Such retracing quantifies the headroom available to persistent and shared memory. The optimal multi-agent makespan is shorter than the minimum single-agent completion time in all but several resource-constrained cases. This gap widens with team size, demonstrating that larger teams enable greater parallelism. 3.4 Evaluation Metrics We extend standard VLN evaluation to the multi-agent, constraint-aware setting. Arrivals are credited within a 3 m radius of any navigable anchor of the target instance. Suc- cess Rate (SR) is the fraction of missions in which every subtask succeeds under the task’s constraint semantics. Suc- cess weighted by Path Length (SPL) weights SR by the ratio 5 TABLE 1 Comparison with related benchmarks on collaborative embodied tasks. BenchmarkMax AgentsTaskPerception Dependency Resource Conflict Path Length Scenes Episodes R2R [1]1VLN✓---9.49021,567 REVERIE [23]1VLN✓---9.79021,702 IR2R [9]1VLN✓Chain--9.49021,567 LHPR-VLN [8]1VLN✓Chain--25.22163,260 CollaVN [15]4Img.-Goal Nav.✓-- ×-5721,180K PARTNR [56]2Planning×DAG × ×-60100K Habitat-MAS [21]4Planning×DAG × ×-611,113 CoNavBench [16]2VLN✓Chain × ×19.41284,048 MAVLN (Ours)4VLN (w/ Planning)✓DAG✓35.914511,724 2 3 4 Agents per episode 648714399– –155134225245– –5116223714718 2345678 0 200 400 600 800 Episodes 6 63 816 903 489 192 18 (a) Subtasks per episode NoYes Presence Lock No Yes Holding 5.1% (126) 2.1% (51) 66.6% (1656) 26.3% (654) (b) Constraint decomposition 0 200 Episodes 20406080100 0% 20% 40% 60% Duplicated path length Bin median and IQR (c) Trajectory length 1020304050 20 40 60 80 100 120 Single-agent serialization (m) Parity 2 agents 3 agents 4 agents (d) Multi-agent makespan Fig. 3. Statistics of 2,487 evaluation episodes. (a) Joint distribution of team size (rows) and subtask count (columns), with the marginal over subtask counts below. (b) Decomposition of resource constraints including holding chains and presence locks. (c) Team trajectory length (top) against the fraction of it duplicated by more than one subtask (bottom). (d) Optimal multi-agent makespan against the best single-agent serialization of the same mission. between the length of the optimal path and total traversed path of all agents. Conditional Subtask Success Rate (CSR) is the fraction of subtasks whose arrival lies within the success radius, honoring all dependency, locking, and hold- ing constraints. Independent / Conditional Subtask SPL (ISPL / CSPL) measure per-subtask path efficiency, gated by success without and with task constraints, respectively. Both normalize by the length the agent traverses within the segment; ISPL takes as reference the shortest path from the terminal position of its preceding subtask to the current goal, while CSPL takes the subtask’s fragment of the optimal reference path. Task Completion (TC) is the fraction of sub- tasks the team consumes irrespective of whether the arrival is correct. Time Steps (TS) counts synchronized simula- tion steps until termination. Unlike planned makespan [16], this realized makespan captures congestion, waiting, and deadlock during execution. Multi-Agent Conflict (MAC) is the number of time steps in which any two agents lie within twice the radius of each other, divided by the product of TS and team size. Arrival-to-subtask matching can be ambiguous: multiple target instances may lie within the success radius, and an incorrect stop may fall near an- other target. We pair the embeddings of scheduler’s emitted atomic instructions (Section 4.1) with the ground-truth ones one-to-one through all-MiniLM-L6-v2 to map from atomic instructions to subtask IDs. This requires exposing a pool of atomic instructions; matching for systems that consume the mission instruction undecomposed is left to future work. 4 METHOD This section presents TRISS for the MAVLN benchmark. As shown in Figure 4, our method comprises three modules: a subtask scheduler that dynamically dispatches subtasks, a shared memory-based navigation planner for viewpoint selection, and a conflict-aware mechanism that executes paths while maintaining topological consistency. 4.1 LLM-based Subtask Scheduler To facilitate subtask scheduling, the task instructions L are first processed by an LLM-based atomic instruction extractor, which identifies subtasks and yields a shared pool of atomic navigation instructions I = ℓ i L i=1 , where L denotes the number of subtasks identified by the ex- tractor. An LLM-based atomic instruction scheduler then allocates unspent atomic instructions in successive schedul- ing rounds, reasoning about the dependency and resource constraints implied by the raw task instructionsL. A round is triggered once every previously dispatched navigation has concluded, at which point the scheduler emits an as- signment I n ∈ (I ∪ ∅) M granting each agent at most one atomic instruction. A blank assignment is reserved for agents that are locked, blocked or finished, and the mission is completed once I n is blank for every agent. In practice, although one agent’s next subtask may de- pend on another’s, the agent can travel toward its destina- tion in advance as long as no presence lock pins it elsewhere, because arrival alone does not complete a subtask. A stop 6 Look for the multi-panel calendar on ... Please coordinate to prepare for the upcoming travel by visiting ... Navigate to the large, multi-panel calendar ... Additionally, head into the bedroom to find a dark wooden ... Retrieve the research notes ... State-Aware Scheduler SharedTopological Memory Coarse-Scale Encoder Waypoint Predictor Waypoint Predictor Waypoint Predictor merge topology Navigation Instruction Head into the bedroom, where you ... Make your way to the laundry alcove ... Look for the large, multi-panel calendar mounted on the ... Agent A Head into the bedroom, where you will find a wooden desk ... Make your way to the laundry alcove in the garage to find ... Atomic Instruction Generator ... Fine-Scale Encoder Instruction Encoder Logits RGB Agent B Agent C v 1 v 2 v 3 ... 0.4 ... ... ... 0.50.1 0.10.80.1 0.10.10.8 v 3 v 2 v 1 v 3 v 2 v 1 beforeafter Conflict-Free Target Allocation DepthDepthDepth Conflict-Based Path Finding Failure- Verified Exclusion Traversal- Verified Connection beforeafter (b) Topological Memory-based Navigation Planner(a) LLM-based Subtask Scheduler(c) Conflict-Aware Execution Mechanism Topology Maintenance Fire Fire Pending Fig. 4. Details of Topological Memory-based Collaborative Navigation and Subtask Scheduling. (a) Atomic instructions are extracted and allocated by an atomic instruction scheduler. (b) A navigation planner predicts viewpoint-level actions over the shared topology for each agent. (c) We deploy conflict-free target allocation and plan agent paths to avoid collisions. A low-level controller outputs actions while the overall topology is protected. action with no unmet dependency fires automatically upon arrival, whereas a gated action waits for an explicit firing signal. Such pre-allocation overlaps travel with waiting, shortening the realized makespan and directly improving TS. We therefore maintain a pending list P n ⊆ A of agents whose newly allocated subtask will be waiting to be fired, and an ordered firing list F n of previously pending agents whose actions fire at the beginning of this round. The scheduler jointly emits: I n ,P n ,F n = LLM sched L,I,I m ,P m ,F m m<n .(1) 4.2 Topological Memory-based Navigation Planner To facilitate experience sharing to support informed deci- sions, we first construct a shared topological memory, con- sisting of a topological representation of the environment and visual features accumulated through viewpoint predic- tion and merging. On top of this memory, a DUET-style [2] cross-modal planner combines each agent’s atomic instruc- tion and observation to produce navigation decisions. Shared Topological Memory. To facilitate cooperative rea- soning, all agents jointly construct one topological map on the fly. Similar to prior works [25], the graph G t = (V t ,E t ) represents the constructed topology after t navigation steps, with nodes divided into explored viewpointsV e t , containing all nodes previously visited by any agent, and unexplored viewpointsV u t , proposed by viewpoint prediction but never visited. A depth-only waypoint predictor [57] consumes the depth view representations d t = d (i) t 12 i=1 and proposes K neighboring viewpoints Y t = y i K i=1 around the agent. Every position the team produces is localized against the shared node set through: v ∗ (q) = arg min v∈V t dist(v,q).(2) A viewpoint y i ∈ Y t is matched if dist(v ∗ (y i ),y i ) ≤ ε loc , yielding the matched viewpoints V ′ t and the unmatched remainder Y ′ t ; letting v t denote the agent’s current view- point, the map is updated in one pass as V t+1 = V t ∪ Y ′ t and E t+1 = E t ∪ (v t ,v) | v ∈ V ′ t ∪ Y ′ t , v ̸= v t . Since all agents apply the same operator to the same node set, duplicate proposals from different teammates merge rather than fork. Unlike single-agent topological mapping, which assumes each agent occupies its intended viewpoint before the next planning step, we include an additional localization phase before viewpoint prediction: the agent’s measured position q t is localized through Equation (2), and its current viewpoint is set to v t = v ∗ (q t ) if the match lies within ε loc . This ensures that, when backtracking fails, the intended viewpoint is not mistakenly treated as the agent’s new viewpoint or updated using observations collected from the agent’s actual position. To obtain the viewpoint representations, we feed the view representations r t , d t and the orientation encodings captured at viewpoint v t into a panorama encoder to obtain the contextual panoramic embeddings ˆr t = ˆr (i) t 12 i=1 . The representation of the current viewpoint v t is set to the aver- age pooling of ˆr t . A newly instantiated viewpoint v ∈ Y ′ t adopts ˆr (i) t of the view through which it was proposed, while a matched unexplored viewpoint v ∈ V ′ t ∩ V u t ap- pends ˆr (i) t to its accumulated embeddings and updates its representation to their mean. Cross-Modal Planning. Each agent receives an atomic in- struction ℓ ∈ I n from the subtask scheduler and encodes it into textual representations ˆ ℓ. Following DUET [2], we employ a coarse-scale encoder for node-level reasoning and a fine-scale encoder for neighborhood-level reasoning. To bound the planning space, for agent a, we denote by V (a) t ⊆ V e t the nodes it has visited in the current round and construct the candidate set ˆ V (a) t = V e t ∪ N V (a) t 7 containing all explored nodes and the neighboring frontiers of its current-round trajectory. This restriction prevents can- didate explosion in continuous environments, particularly as the team size grows. The coarse-scale encoder processes candidate viewpoint features X = [x 0 ,x 1 ,...,x | ˆ V (a) t | ] to- gether with ˆ ℓ through a graph transformer, where x 0 is the stop token. Global action logits s (c) i are computed from ˆx i for each viewpoint, where ˆx i ∈ ˆ X denotes the output of the coarse-scale encoder, providing node-level navigation preferences. Comparatively, fine-scale planning processes the immediate contextual panoramic embeddings ˆr t . We prepend a stop token r 0 and feed the instruction embedding ˆ ℓ and the sequence R = [r 0 ; ˆr t ] into the fine-scale encoder, yielding local action logits s (f) i for the current neighbors v i ∈ Y ′ t ∪V ′ t , which are converted to the global action space: s (f ′ ) i = ( s (f) i ,if v i ∈ Y ′ t ∪V ′ t , s back , otherwise, (3) where s back aggregates the logits of the visited neighbors of v t . To support dual-scale navigation planning, we dy- namically fuse the two granularities using a scalar gate σ t inferred through the concatenated features [ˆr 0 ; ˆx 0 ]. The final logit for each candidate is computed as s i = σ t s (f ′ ) i + (1− σ t )s (c) i . 4.3 Conflict-Aware Execution Mechanism To facilitate conflict-free execution, we first adopt a conflict- free target allocation algorithm to prevent multiple agents from selecting the same viewpoint as their destination. Subsequently, we leverage a conflict-based path finding algorithm to plan conflict-free path sequences toward the targets, carried out by low-level controllers. Finally, we introduce a topology maintenance mechanism to avoid topology corruption. Conflict-Free Target Allocation. Multi-agent navigation lets several agents select target viewpoints on a shared graph, introducing potential conflicts where agents jockey for the same viewpoint, leading to collisions. We therefore model multi-agent viewpoint allocation as a maximum-score bipar- tite assignment problem, which is solved by the Hungarian algorithm. Every agent is thus dispatched to a distinct view- point, maximizing the team’s total policy score; an agent whose preferred frontier is claimed by a higher-scoring teammate is rerouted to its best remaining alternative within the same solve. Conflict-Based Path Finding. Agents traverse multiple viewpoints before arriving at their destinations, which reg- ularly gives rise to temporal conflicts when several of them occupy the same viewpoint along their paths. We leverage Conflict-Based Search [46] to resolve them over the shared topology. A space-time A* first plans each agent a short- horizon trajectory across explored nodes, with unexplored nodes enterable only as goals and waiting admitted as a move, requiring goal occupancy only at the end of the horizon so that an agent may pause or temporarily vacate its goal to let a teammate pass. A bounded search then resolves the remaining conflicts on the synchronized timeline, return- ing the least-conflict route set once its expansion budget is exhausted. The resulting routes are executed jointly through a low-level controller [25]. Topology Maintenance. When executing low-level actions along the searched H -step path, agents commonly en- counter two types of failures: mapping failures and drifting issues, both of which corrupt a topology that every team- mate subsequently consumes. We propose failure-verified exclusion to tackle mapping failures. A predicted viewpoint v H ∈ V u t might be unreachable, so that navigation ter- minates without any position change while the waypoint predictor keeps proposing the same location, forming a mapping failure. Therefore we propose a collective ban list B = (u,y), pairing the viewpoint u from which the failed attempt departed with the position y of the proposal. Whenever a navigation from u fails to produce a position change, we remove y from the memory and insert (u,y) into B. Subsequent proposals are then filtered as Y t ←y i ∈ Y t | dist(y,y i )≥ ε ban ,∀(v t ,y)∈ B,(4) where ε ban indicates the radius of the banned area. More- over, we propose traversal-verified connection to tackle drifting issues. During the localization phase, a naive ap- proach connects the viewpoint v t with the penultimate viewpoint v H−1 of the path. However, the controller might drift away from the explored viewpoint or exit early upon collision, so that v H−1 is not where the agent physically came from and the added edge would assert a traversal that never happened. Instead, we identify the last viewpoint of the searched path that the agent verifiably passed through without early exit, reducing the risk of spurious shortcuts caused by detours or recovery. 4.4 Student-Forced Multi-Agent Learning The navigation planner is trained by imitation learning with the ground-truth atomic navigation instructions. To avoid dependency-induced waiting and multi-agent sim- ulation overhead during training, we augment the single- agent VLN pipeline with shared topological memory. We group the training set so that episodes assigned to one team take pairwise distinct starts, and preferentially exhibit path overlap between teammates’ reference trajectories to reward reading teammate experience. We use student-forcing to encourage exploration during training and compute the loss at each step through: L IL =− M X a=1 X t log p v ⋆ a,t | ℓ a ,G t ,O a,t , (5) where v ⋆ a,t denotes the viewpoint closest to the agent’s goal. 5 EXPERIMENTS 5.1 Experimental Setup Training Settings. We adopt a pretraining stage for the planner following ETPNav [25]. We construct pretraining data from the NavRAG [27] training set together with ground-truth pairs of atomic instructions and subtask tra- jectories extracted from the MAVLN training split. Since 8 TABLE 2 Performance on MAVLN under oracle and LLM-based subtask schedulers across the three instruction regimes Val-UnseenTest-Unseen Sched.Nav. SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ TS↓ MAC↓ SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ TS↓ MAC↓ Ora. (SA) Ora.67.163.174.968.591.6100.04120.0069.164.478.070.890.4100.04190.00 Ora. (LE) Ora. 100.099.7100.099.592.6100.02611.16100.099.6100.099.492.0100.02481.39 Ora.Ora. 100.099.8100.099.692.6100.02151.75100.099.6100.099.492.0100.02062.10 Ora.ETP2.81.927.221.630.5100.03940.875.64.228.722.430.6100.04030.85 Ora. (SA) Ours5.43.326.015.325.2100.020560.005.93.825.217.427.7100.016650.00 Ora.Ours9.14.036.419.726.9100.015880.829.35.233.320.027.9100.014941.42 Rand.Ours0.30.17.94.323.6100.016210.910.20.16.74.124.1100.015471.12 Seq.Ours3.41.623.612.824.9100.015930.902.51.221.812.626.5100.014431.28 Decentralized Instruction Style Gemma4 Ours 7.0±0.9 2.9±0.4 33.6±0.9 18.2±0.6 24.0±0.4 94.6±0.5 1570±37 0.95±0.1 8.3±0.1 4.4±0.0 33.2±0.3 19.5±0.3 26.5±0.2 95.8±0.2 1447±13 1.22±0.0 Qwen3.6 Ours 7.3±0.7 2.8±0.3 34.7±0.4 18.2±0.1 24.9±0.3 97.0±0.3 1613±59 0.87±0.1 7.9±0.8 4.1±0.4 32.1±0.3 18.7±0.2 25.9±0.4 97.4±0.2 1521±25 1.26±0.1 Centralized Instruction Style Gemma4 Ours 7.5±1.2 3.1±0.5 34.3±1.0 18.1±0.5 23.9±0.5 94.4±0.5 1570±40 0.83±0.0 6.8±0.1 3.9±0.1 31.6±0.1 18.6±0.1 25.4±0.1 95.0±0.2 1475±10 1.13±0.0 Qwen3.6 Ours 7.6±0.1 3.1±0.2 34.4±0.3 18.2±0.2 24.0±0.4 95.7±0.1 1619±29 0.85±0.0 8.4±0.9 4.2±0.5 32.4±0.8 19.1±0.7 26.3±0.5 96.5±0.4 1470±7 1.03±0.0 Centralized Implicit Instruction Style Gemma4 Ours 5.5±0.1 2.3±0.1 32.1±0.8 16.5±0.2 22.5±0.2 92.5±0.4 1611±44 0.76±0.0 7.1±0.3 3.7±0.2 29.6±0.6 17.0±0.5 23.3±0.3 92.8±0.4 1512±37 0.97±0.1 Qwen3.6 Ours 6.2±0.6 2.7±0.3 31.5±0.2 16.6±0.0 23.2±0.2 96.8±0.2 1662±16 0.78±0.0 6.8±0.7 3.5±0.4 28.8±0.6 16.5±0.5 24.1±0.5 96.5±0.5 1562±26 1.03±0.1 Sched.: scheduler; Nav.: navigator; Ora.: oracle; (SA): single agent variant; (LE): legacy variant without pre-allocation; ETP: ETPNav trained on MAVLN; Rand./Seq.: ground-truth atomic instructions dispatched to free agents in random / sequential order. Bold: best among LLM schedulers. NavRAG is also built on HM3D, we remove all trajectories belonging to the unseen splits of MAVLN beforehand to prevent data leakage. We pretrain for 200,000 steps and select the checkpoint with the best navigation performance on val-unseen. For imitation learning, we set the maximum trajectory length to 30, let four agents share the topological memory, and train for 30,000 steps, selecting the checkpoint that maximizes the sum of SR and SPL. Training runs on a single A100 GPU. Inference Settings. Given the size and complexity of the evaluation splits, we adopt two open-source VLMs to keep evaluation tractable: Qwen3.6-27b, a lightweight yet capable model on agentic tasks, and Gemma4-31b, the latest open- source release from Google. Both serve as subtask sched- ulers within our framework, and all LLM-dependent results are averaged over three independent runs. Unless stated otherwise, ablation studies replace the LLM scheduler with an oracle scheduler that emits ground-truth atomic instruc- tions, which isolates navigation quality from scheduling variance and improves time efficiency. Inference runs on a single A100 GPU. 5.2 Quantitative Analysis Table 2 compares navigation performance across schedulers and instruction regimes. With oracle scheduling and nav- igation, TRISS achieves 100.0 SR with 215 and 206 time steps on val-unseen and test-unseen, respectively. Removing pre-allocation increases TS to 261 and 248, demonstrating the benefit of overlapping travel with dependency-induced waiting. Restricting execution to one agent reduces SR to 67.1 and 69.1 and increases TS to 412 and 419, highlighting the feasibility and parallelism benefits of multi-agent execu- tion. Replacing oracle navigation with our navigator reduces test-unseen SPL from 99.6 to 5.2. The gap is not specific to our navigator: ETPNav trained on MAVLN reaches only 2.8 and 5.6 SR, although it attains higher CSPL and ISPL by reaching fewer subtasks along more direct paths. Multi- agent execution still outperforms its single-agent counter- TABLE 3 Ablation of system components and training recipes Val-Unseen ComponentVariant SR↑ SPL↑ CSR↑ CSPL↑ TS↓ MAC↓ Framework Shared Memory Disabled6.23.033.419.5 1065 0.7 Enabled9.14.036.419.7 1588 0.8 Planning Space Unbounded 7.83.735.020.2 1316 0.9 Bounded9.14.036.419.7 1588 0.8 Planner Coarse-Scale 0.00.08.51.22468 1.9 Dual-Scale9.14.036.419.7 1588 0.8 Step Limit 206.02.634.720.0 1272 0.9 309.14.036.419.7 1588 0.8 Training Teacher-Forcing 50%6.02.833.121.6 1010 1.1 0%9.14.036.419.7 1588 0.8 Concurrent Agents 24.92.532.318.2 1589 1.1 49.14.036.419.7 1588 0.8 Accum. Rounds 19.14.036.419.7 1588 0.8 28.53.437.319.4 1760 0.9 Pretrain Source MAVLN5.72.430.018.1 1404 1.0 +REVERIE 7.33.431.718.0 1667 1.1 +NavRAG9.14.036.419.7 1588 0.8 part (3.8 SPL). Replacing the oracle scheduler with an LLM further reduces SPL to 4.4 (Gemma4, decentralized), yet it clearly outperforms random and sequential dispatch, whose ISPL stays close to the oracle scheduler’s while CSR collapses, confirming that failures stem from violated constraints rather than navigation. Centralized-implicit in- structions are hardest, as allocation must be inferred on top of constraint reasoning (3.5 SPL for Qwen3.6). Lower MAC does not necessarily accompany better task performance, and Qwen3.6 outperforms Gemma4 on SR in four of the six scenarios. 5.3 Ablation Studies & Analyses Framework Components. Table 3 ablates the components of TRISS. Disabling the shared topological memory reduces SR from 9.1 to 6.2 and SPL from 4.0 to 3.0, validating 9 TABLE 4 Ablation of the conflict-resolving mechanisms Val-Unseen CFTA CBPF SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TS↓ MAC↓ 8.84.134.919.727.212302.2 ✓8.33.835.219.827.512651.4 ✓9.14.036.419.726.915880.8 CFTA: Conflict-Free Target Allocation; CBPF: Conflict-Based Path Finding. TABLE 5 Ablation of topology mapping and maintenance Val-Unseen Localization FVE TVC SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ MAC↓ 6.52.734.018.325.41.1 ✓7.83.536.019.626.10.8 ✓7.33.034.118.625.11.0 ✓9.14.036.419.726.90.8 FVE: Failure-Verified Exclusion; TVC: Traversal-Verified Connec- tion. memory sharing as the central mechanism of the frame- work. The same mechanism raises TS from 1065 to 1588 because conflict-resolving takes time. Bounding the plan- ning space during cross-modal reasoning raises SR from 7.8 to 9.1 by filtering noisy distant candidates, while the unbounded variant retains marginally better CSPL from its wider horizon. Dual-scale planning has the largest observed effect: the coarse-scale variant achieves 0.0 SR and 8.5 CSR, compared with 9.1 and 36.4 for the dual-scale planner. A shared map results in far more candidate nodes per step and the waypoint predictor contributes additional noise, and therefore dual-scale reasoning anchors attention on the local neighborhood. This is not a stylistic inheritance from DUET [2] but a requirement of the multi-agent regime. Finally, reducing navigation step limit to 20 saves time but lowers SR from 9.1 to 6.0, as exploration is curtailed before decisions can be grounded. Planner Training. Table 3 also compares training recipes. Fully student-forced training outperforms 50% teacher forc- ing in SR (9.1 vs. 6.0), although teacher forcing yields lower TS and higher CSPL, indicating that exposure to demonstra- tor trajectories narrows the exploration the student must ultimately perform alone. Training with four rather than two agents improves SR by 4.2 points and SPL by 1.5, supporting training with broader shared-memory exposure. We further ask whether pre-filling that memory helps, by persisting the topology across rounds; SR instead drops slightly when accumulation rises from one round to two, suggesting that an over-rich memory encourages reliance on inherited structure, which hurts precisely when little expe- rience is available. Pretraining with REVERIE [23] improves SPL from 2.4 to 3.4 over MAVLN-only pretraining, while the NavRAG configuration [27] performs best at 4.0, consistent with its broader HM3D scene coverage. Basic navigation competence therefore scales with the knowledge acquired in pretraining. Conflict Resolving. Table 4 isolates conflict-free target allo- cation (CFTA) and conflict-based path finding (CBPF). The vanilla framework attains the highest MAC at 2.2, together TABLE 6 Comparison of arrival-sequence matching strategies. Val-Unseen Instruction StyleMatching SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ Decentralizedgreedy8.33.335.318.522.598.1 embed8.33.335.218.425.097.3 ∆+0.0+0.0-0.1-0.1+2.5-0.8 Centralizedgreedy7.32.837.819.422.897.3 embed7.53.134.218.023.695.6 ∆+0.2+0.3-3.6-1.4+0.8-1.7 Centralized Implicit greedy5.12.135.218.322.398.1 embed5.42.331.516.623.196.6 ∆+0.3+0.2-3.7-1.7+0.8-1.5 02040 Time per scheduling turn (s) LLM scheduling Inference Simulation 33.52 4.56 2.79 01020 GPU memory (GB) Inference Training (batch = 4) 2.11.7 3.8 12.42.2 × 4 21.1 PlannerSimulator Fig. 5. Latency breakdown per scheduling round and GPU memory consumption during training and inference. with the lowest TS and the highest SPL, since agents drive straight at their goals. Enabling CFTA lowers MAC to 1.4 at a slight cost in success, reflecting the compromise imposed by mutually exclusive target selection. Adding CBPF further reduces MAC to 0.8, indicating its effectiveness in avoiding interference at the price of 323 additional time steps. Topology Mapping and Maintenance. Table 5 examines the additional localization phase together with failure-verified exclusion (FVE) and traversal-verified connection (TVC). Localization lets an agent recover the most plausible view- point after control deviation, rather than inheriting the as- sumed viewpoint. Enabling either mechanism on top of lo- calization improves performance, by 0.8 and 0.3 SPL respec- tively. The gain is larger for FVE, since control deadlocks against obstacles are more common. The full configuration performs best, and these results support verifying both failed proposals and traversed connections before updating shared topology. Metric Calculation. Table 6 compares our embedding-based instruction-to-subtask matching with a greedy strategy that maximizes task completion and uses conditional success to break ties. Mission-level metrics change little, but subtask metrics are more sensitive: greedy matching reports higher CSR by 0.1, 3.6 and 3.7 points under decentralized, central- ized and centralized-implicit instructions respectively, while reporting lower ISPL throughout. The inflation tracks allo- cation freedom: it is negligible when each subtask admits only its designated agent, and grows once any agent may execute any subtask and the space of consistent interpre- tations widens accordingly. Faithful measurement therefore requires an explicit instruction-based mapping rather than a maximizing search. Computational Efficiency. Figure 5 reports inference la- tency and GPU memory. LLM scheduling dominates the re- ported per-round latency at 33.52 s, compared with 4.56 s for 10 Navigation Instruction Work together to set up for a weekend movie night. One of you should make your way into the kitchen to find the large, stainless steel double-door refrigerator, which is ... Retrieve the cold movie snacks from here. Simultaneously, another should head into the kitchen and locate the sleek electric coffee machinesitting on ... Prepare the warm beverages here. Meanwhile, a third should navigate to the living room and approach the large, textured grey, fabric-upholstered sectional couch... Arrange this seating area. Once the snacks are retrieved and the seating area is fully prepared, navigate to the living area to locate the round wooden side table... and place the snacks there. After the snacks have been placed on the side table and the beverages are ready, head to the dining area to locate the large dining table... Place the drinks on the table to complete the preparations. Scheduling Round 1 LLM Scheduler Output (atomic instructions) Make your way into the kitchen to find the .... refrigerator, ... positioned right next to the open doorway that leads to the adjacent dining area. Head into the kitchen and locate the electric coffee machine... between the stovetop and the integrated stack containing the microwave and ... Navigate to the living room and approach the ... couch..., positioned directly against a wall featuring three framed pieces of artwork, ... Trajectory Map Agent B (to coffee machine) Agent Observations (selected steps) Agent C (to couch) Agent A (to refrigerator) hallway corridor doorway 1 1 1 doorway doorway living room 2 2 2 refrigerator coffee machine couch 3 3 3 Scheduling Round 2 LLM Scheduler Output (atomic instructions) Navigate to the living area to locate the round wooden side table..., recognizable by the stack of magazines resting on its surface. Head to the dining area to locate the large dining table. It is positioned on a light, neutral-colored rug directly beneath a large, dark, ... Trajectory Map Agent B (to dining table) Agent Observations (selected steps) Agent A (to wooden table) kitchen kitchen living area dining area wooden table dining table 1 1 3 3 2 2 Round 1 Team State carrying snacks carrying drinks arrived at couch Round 2 Team State snacks placed drinks placed success Fig. 6. Scheduling and navigation process of TRISS on a representative episode. Agents A and B enter the holding state after the first scheduling round, and the mission completes at the end of the second. Subtask goals are drawn as flags in each agent’s color. TABLE 7 Performance of observation-guided scheduler Val-UnseenTest-Unseen Obs. SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ Decentralized Instruction Style 8.33.335.218.425.07.03.931.718.425.6 ✓7.83.034.718.525.37.74.033.019.523.6 Centralized Instruction Style 7.53.134.218.023.67.23.631.818.525.7 ✓7.33.333.017.723.16.53.531.819.325.2 Centralized Implicit Instruction Style 5.42.331.516.623.17.44.028.916.724.3 ✓5.72.430.616.822.27.94.231.218.623.9 TABLE 8 Zero-shot navigation performance on a 50-episode subset Val-Unseen Instruction StylePlanner SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TS↓ MAC↓ DecentralizedDefault 10.02.837.216.923.214531.2 Hybrid4.01.433.220.424.313431.2 VLM2.00.720.813.217.39770.9 CentralizedDefault 10.02.534.916.823.615020.9 Hybrid6.02.929.817.922.512220.7 VLM2.01.320.612.716.18820.7 Centralized Implicit Default6.02.336.018.024.816491.0 Hybrid4.02.026.816.021.611720.8 VLM2.01.112.69.114.08620.4 navigation planning and 2.79 s for simulation, identifying scheduling as the principal target for future optimization and confirming that topological planning is efficient. The planner requires only 2.1 GB of GPU memory at inference, comparable to the simulator itself, and training fits within 21.1 GB in total, keeping the system reproducible on widely TABLE 9 Comparison against an oracle planner Val-UnseenTest-Unseen Planner SR↑ SPL↑ CSR↑ CSPL↑ TS↓ SR↑ SPL↑ CSR↑ CSPL↑ TS↓ Ours9.14.036.419.71588 9.35.233.320.01494 Oracle89.1 73.496.381.8640 81.0 64.091.074.9769 available hardware. Observation-Guided Scheduler. Decentralized instructions distribute the agent assignment explicitly, whereas cen- tralized and centralized-implicit instructions progressively withhold it, leaving the scheduler to infer an allocation. We analyze whether scheduling benefits from observa- tions that hint at promising locations. Table 7 compares the language-only scheduler against an observation-guided variant on Qwen3.6. Centralized-implicit instructions show small gains. Centralized / decentralized instructions show no consistent improvement, suggesting that perceptual con- text can also distract reasoning. Closing this gap charts a robust scheduler that grounds allocation in the team’s spatial state and approaches optimality. Zero-Shot Navigation. Table 8 compares the trained planner with Qwen3.6 as a zero-shot VLM planner and a hybrid that queries the VLM only when the trained planner is uncertain. The trained planner outperforms the zero-shot VLM in SR across all three instruction regimes, by 8.0, 8.0, and 4.0 points. The hybrid variant recovers part of the gap but still trails the trained planner. Notably, the VLM planner attains the lowest TS and MAC while failing most missions, which reflects early termination rather than efficiency. Executing coarse-grained atomic instructions over a shared topology therefore remains beyond current zero-shot VLMs. 11 Navigation Instruction ... Head into the small tanning booth to the solarium, a large, curved tanning bed ... Right next to the side of the large solarium unit, locate the electric hub... The agent at the solarium must maintain its position at the diagnostic panel until the breaker is reset at the electric hub. Only after the solarium panel is being held can the electric hub be reset, which will then release the agent at the solarium... LLM Scheduler Output (Turn 1) Go to the administrative area to find the small handwritten note containing ... Head into the small tanning booth to the solarium, a large, curved tanning bed ... LLM Scheduler Output (Turn 2) Right next to the side of the large solarium unit, locate the electric hub ... Find the air conditioner mounted on the wall within the workspace, ... (a) Scheduler: Constraint Violation air- conditioner electric hub tanning bed Handwrit -ten note maintain presenceunlock at next round Atomic Navigation Instruction Navigate to the long, dark wooden workstation desk against the wall, identifiable by a large cork bulletin board above it, two sets of modular transparent storage boxes on top, and a white stool underneath, located near a window with light floor-length curtains. Trajectory Sequence (selected steps) (b) Planner: Inefficiency & Mis-grounding 1 3 5 7 2 4 6 8 Target Desk (insufficiently explored) Distractor (semantically similar) Execution Failure from Barriers (c) Controller: Failure & Topology Corruption 1 3 2 4 Misled Execution from Erroneous Topology Predicted Viewpoint (on Floor 1) Agent Location (on Floor 2) Observation (predicted viewpoint) Observation (current viewpoint) connected going round in circles Fig. 7. Failure modes. The scheduler case violates a presence lock by re-dispatching an agent that should have held position; the planner case stops at a distractor while leaving the target insufficiently explored; the controller case shows an execution failure and an erroneous edge misleading later execution. TABLE 10 Failure statistics of conflict-based path finding Val-UnseenTest-Unseen SchedulerTotalUnres.↓ Inter.↓TotalUnres.↓ Inter.↓ Qwen3.618,5083.781.0221,0574.411.26 Oracle18,0873.981.1921,0625.051.55 Topological Planning Limitations. Table 9 bounds the cur- rent paradigm with an oracle planner, raising SR from 9.1 to 89.1 on val-unseen and from 9.3 to 81.0 on test-unseen, iden- tifying planning as a major bottleneck. Even granted perfect node selection, topological planning is thus capped well be- low saturation. Viewpoint prediction noise compounds over an episode, so a target may simply never be proposed, and the resulting spurious nodes corrupt memory. This charts future work on robust mapping modules, or on planning paradigms that dispense with predicted viewpoints while preserving memory-wise coordination. Conflict Resolving Limitations. Table 10 reflects failures of the bounded search to return conflict-free routes. 3.78- 3.98% of transitions remain unresolved on val-unseen and 4.41-5.05% on test-unseen. Geometrically intersecting tran- sitions occur in 1.02-1.55% of cases. Such residual rates would remain unacceptable in physical deployments, chart- ing stronger search and geometric checks during execution. 5.4 Qualitative Analysis Figure 6 illustrates successful coordination across five sub- tasks and two scheduling rounds. In the first round, agents A and B reach the refrigerator and coffee machine, entering holding states associated with carrying snacks and drinks, while agent C reaches the couch. In the second round, A and B are assigned the side table and dining table, respectively, and complete the mission under the task con- straints. This example illustrates how TRISS preserves agent assignments across holding chains while coordinating de- pendent subtasks. Figure 7 highlights three complementary failure modes. The scheduler violates an explicit presence constraint by re-dispatching the agent at the solarium before its teammate resets the electric hub. The planner repeatedly revisits candidate areas and stops at a semantically similar distractor, leaving the target desk insufficiently explored. The controller encounters execution failures at obstacles that produce deadlocks, and these failures can lead to nonex- istent connections in the shared map, while an erroneous connection between viewpoints on different floors misleads subsequent navigation. Across the analyses above, the headroom MAVLN leaves is attributable to no single module. Schedulers fail to keep language-stated constraints live through execution, which neither perceptual context nor their dominant share of la- tency currently redeems; planners stay capped by waypoint proposal even under oracle node selection, and the task remains well beyond zero-shot VLMs; controllers still suffer from failures, leaving residual conflicts unresolved. 6 CONCLUSION This work introduces Systematic Multi-Agent Vision-and- Language Navigation, formulating embodied multi-agent navigation as a coordination problem in which a mission de- composes into subtasks carrying dependency and resource constraints. We present an automatic generation engine that yields MAVLN released with constraint-aware metrics. We further present TRISS, a coordination-ready navigation sys- tem integrating an LLM-based subtask scheduler, a shared 12 topological memory and a conflict-aware execution mech- anism. Experiments establish TRISS as a comprehensive baseline while measuring how far the task remains from solved, as absolute success rates stay in the single digits, and the headroom is distributed across scheduling, planning, and execution rather than concentrated in any one. We release the benchmark, platform and code to support the work of closing it. REFERENCES [1]P. Anderson et al., “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in CVPR, 2018, p. 3674–3683. [2]S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and- language navigation,” in CVPR, 2022, p. 16 537–16 547. [3]Z. Wang et al., “Scaling data generation in vision-and-language navigation,” in ICCV, 2023, p. 12 009–12 020. [4]J. Zhang et al., “Navid: Video-based vlm plans the next step for vision-and-language navigation,” in RSS, 2024, p. 1–16. [5]J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous en- vironments,” in ECCV, 2020, p. 104–120. [6]L. Wang et al., “Rethinking the embodied gap in vision-and- language navigation: A holistic study of physical and visual disparities,” in ICCV, 2025, p. 9455–9465. [7]S. Lin et al., “Vlnverse: A benchmark for vision-language naviga- tion with versatile, embodied, realistic simulation and evaluation,” arXiv preprint arXiv:2512.19021, 2025. [8]X. Song et al., “Towards long-horizon vision-language navigation: Platform, benchmark and method,” in CVPR, 2025, p. 12 078– 12 088. [9]J. Krantz et al., “Iterative vision-and-language navigation,” in CVPR, 2023, p. 14 921–14 930. [10] Y. Xu, Y. Pan, and Z. Liu, “Dream to recall: Imagination-guided experience retrieval for memory-persistent vision-and-language navigation,” IEEE TPAMI, vol. 48, no. 8, p. 9035–9049, 2026. [11] C. Yu et al., “Learning efficient multi-agent cooperative visual exploration,” in ECCV, 2022, p. 497–515. [12] S. Tan, W. Xiang, H. Liu, D. Guo, and F. Sun, “Multi-agent embodied question answering in interactive environments,” in ECCV, 2020, p. 663–678. [13] U. Jain et al., “A cordial sync: Going beyond marginal policies for multi-agent embodied tasks,” in ECCV, 2020, p. 471–490. [14] B. Yu, H. Kasaei, and M. Cao, “Co-navgpt: Multi-robot cooperative visual semantic navigation using large language models,” arXiv preprint arXiv:2310.07937, 2023. [15] H. Wang, W. Wang, X. Zhu, J. Dai, and L. Wang, “Collaborative visual navigation,” arXiv preprint arXiv:2107.01151, 2021. [16] T. Wang et al., “Conavbench: Collaborative long-horizon vision- language navigation benchmark,” in ICLR, 2026, p. 1–13. [17] S. Zhou et al., “Deconav: Dialog enhanced long-horizon collabora- tive vision-language navigation,” arXiv preprint arXiv:2604.12486, 2026. [18] Q. Jin, Y. Song, and Q. Wu, “Does peer observation help? vision- sharing collaboration for vision-language navigation,” arXiv preprint arXiv:2603.20804, 2026. [19] S. S. Kannan, V. L. Venkatesh, and B.-C. Min, “Smart-llm: Smart multi-agent robot task planning using large language models,” in IROS, 2024, p. 12 140–12 147. [20] H. Zhang et al., “Building cooperative embodied agents modularly with large language models,” in ICLR, 2024, p. 1–16. [21] J. Chen et al., “Emos: Embodiment-aware heterogeneous multi- robot operating system with llm agents,” in ICLR, 2025, p. 1–13. [22] H. Tan et al., “Roboos-next: A unified memory-based framework for lifelong, scalable, and robust multi-robot collaboration,” arXiv preprint arXiv:2510.26536, 2025. [23] Y. Qi et al., “Reverie: Remote embodied visual referring expression in real indoor environments,” in CVPR, 2020, p. 9982–9991. [24] S. Chen, P.-L. Guhur, C. Schmid, and I. Laptev, “History aware multimodal transformer for vision-and-language navigation,” in NeurIPS, vol. 34, 2021, p. 5834–5847. [25] D. An et al., “Etpnav: Evolving topological planning for vision- language navigation in continuous environments,” IEEE TPAMI, vol. 47, no. 7, p. 5130–5145, 2025. [26] Y. Pan, Y. Xu, Z. Liu, and H. Wang, “Planning from imagina- tion: Episodic simulation and episodic memory for vision-and- language navigation,” in AAAI, vol. 39, no. 6, 2025, p. 6345–6353. [27] Z. Wang, Y. Zhu, G. H. Lee, and Y. Fan, “Navrag: Generating user demand instructions for embodied navigation through retrieval- augmented llm,” in ACL Findings, 2025, p. 8430–8440. [28] J. Zhang et al., “Embodied navigation foundation model,” arXiv preprint arXiv:2509.12129, 2025. [29] A.-C. Cheng et al., “Navila: Legged robot vision-language-action model for navigation,” in RSS, 2025, p. 1–13. [30] S. Zeng et al., “Janusvln: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation,” arXiv preprint arXiv:2509.22548, 2025. [31] M. Wei et al., “Streamvln: Streaming vision-and-language navigation via slowfast context modeling,” arXiv preprint arXiv:2507.05240, 2025. [32] Y. Xu, Y. Pan, Z. Liu, and H. Wang, “Flame: Learning to navigate with multimodal llm in urban environments,” in AAAI, vol. 39, no. 9, 2025, p. 9005–9013. [33] G. Zhou, Y. Hong, and Q. Wu, “Navgpt: Explicit reasoning in vision-and-language navigation with large language models,” in AAAI, vol. 38, no. 7, 2024, p. 7641–7649. [34] K. Chen et al., “Constraint-aware zero-shot vision-language navi- gation in continuous environments,” IEEE TPAMI, vol. 47, no. 11, p. 10 441–10 456, 2025. [35] X. Shi et al., “Smartway: Enhanced waypoint prediction and back- tracking for zero-shot vision-and-language navigation,” in IROS, 2025, p. 16 923–16 930. [36] Y. Qiao, Y. Qi, Z. Yu, J. Liu, and Q. Wu, “March in chat: Interactive prompting for remote embodied referring expression,” in CVPR, 2023, p. 15 758–15 767. [37] Y. Qiao, Q. Liu, J. Liu, J. Liu, and Q. Wu, “Llm as copilot for coarse-grained vision-and-language navigation,” in ECCV, 2024, p. 459–476. [38] S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva, “Multion: Benchmarking semantic map memory using multi-object naviga- tion,” in NeurIPS, vol. 33, 2020, p. 9700–9712. [39] Y. Pan, Y. Xu, Z. Liu, and H. Wang, “Seeing through uncer- tainty: Robust task-oriented optimization in visual navigation,” in NeurIPS, vol. 38, 2026, p. 25 259–25 286. [40] Y. Long, X. Li, W. Cai, and H. Dong, “Discuss before moving: Vi- sual language navigation via multi-expert discussions,” in ICRA, 2024, p. 17 380–17 387. [41] R. Wu et al., “Aeroduo: Aerial duo for uav-based vision and language navigation,” in ACM M, 2025, p. 2576–2585. [42] J. Song et al., “Conav-uav: Cooperative dual-altitude aerial nav- igation via stackelberg learning,” arXiv preprint arXiv:2608.01802, 2026. [43] P.Wu et al.,“Camon:Cooperativeagentsformulti- object navigation with llm-based conversations,” arXiv preprint arXiv:2407.00632, 2024. [44] Z. Shen, H. Luo, K. Chen, F. Lv, and T. Li, “Enhancing multi-robot semantic navigation through multimodal chain-of-thought score collaboration,” in AAAI, vol. 39, no. 14, 2025, p. 14 664–14 672. [45] X. Liu et al., “Embodied multi-agent task planning from ambigu- ous instruction,” in RSS, 2022, p. 1–14. [46] G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant, “Conflict- based search for optimal multi-agent pathfinding,” AIJ, vol. 219, p. 40–66, 2015. [47] G. Sartoretti et al., “Primal: Pathfinding via reinforcement and imitation multi-agent learning,” IEEE RA-L, vol. 4, no. 3, p. 2378– 2385, 2019. [48] K. Obata, T. Aoki, T. Horii, T. Taniguchi, and T. Nagai, “Lip- llm: Integrating linear programming and dependency graph with large language models for multi-robot task planning,” IEEE RA-L, vol. 10, no. 2, p. 1122–1129, 2024. [49] X. Zhang, H. Qin, F. Wang, Y. Dong, and J. Li, “Lamma-p: Gen- eralizable multi-agent long-horizon task allocation and planning with lm-driven pddl planner,” in ICRA, 2025, p. 10 221–10 221. [50] Y. Xu, Z. Zhang, and Z. Liu, “Beyond elicitation: Provision-based prompt optimization for knowledge-intensive tasks,” IEEE TASLP, vol. 34, p. 2128–2139, 2026. 13 [51] T. Kawabe and R. Takano, “Hierarchical llm-based multi-agent framework with prompt optimization for multi-robot task plan- ning,” arXiv preprint arXiv:2602.21670, 2026. [52] C. Zhang et al., “Proagent: Building proactive cooperative agents with large language models,” in AAAI, vol. 38, no. 16, 2024, p. 17 591–17 599. [53] S. Nayak et al., “Long-horizon planning for multi-agent robots in partially observable environments,” in NeurIPS, vol. 37, 2024, p. 67 929–67 967. [54] X. Puig et al., “Habitat 3.0: A co-habitat for humans, avatars, and robots,” in ICLR, 2024, p. 1–15. [55] S. K. Ramakrishnan et al., “Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,” in NeurIPS D&B, 2021. [56] M. Chang et al., “Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks,” in ICLR, 2025, p. 1–14. [57] Y. Hong, Z. Wang, Q. Wu, and S. Gould, “Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation,” in CVPR, 2022, p. 15 439– 15 449. 7 BIOGRAPHY SECTION Yunzhe Xu received the bachelor’s degree in software engineering from Harbin Institute of Technology in 2022. He is currently pursuing the Ph.D. degree in computer science and tech- nology with Shanghai Jiao Tong University. His research interests include embodied navigation systems, robotic learning and large language model agents. He has published papers in top- tier AI venues, including IEEE T-PAMI, AAAI and NeurIPS. Zhe Liu received the Ph.D. degree in control technology and control engineering from Shang- hai Jiao Tong University, Shanghai, China, in 2016. From 2017 to 2020, he was a Post- Doctoral Fellow with the Department of Mechan- ical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong. From 2020 to 2022, he was a Research Associate with the Department of Computer Science and Technol- ogy, University of Cambridge, Cambridge, U.K. From 2022 to 2025, he has been an Associate Professor with the AI institute, Shanghai Jiao Tong University, where he is currently an Associate Professor with the Department of Automation. His current research interests include multi-robot cooperation and au- tonomous driving system. He is a Senior Editor of IEEE Transactions on Automation Science and Engineering, an Associate Editor of IEEE Transactions on Robotics, and an Editor of IEEE International Confer- ence on Robotics and Automation. 1 Supplementary Material for “Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method” ✦ This supplementary material is organized as follows. Section A documents the simulation platform, perception backbone, pretraining, imitation learning and inference runtime. Section B analyzes the generation funnel and the composition of MAVLN. Section C reports additional experiments. Section D formalizes the goal anchors, the constraint-aware metrics and the instruction-to-subtask matching. Section E unfolds the three modules of TRISS, including the full episode loop, the scheduler protocol, the shared-topology maintenance, the planner architecture, conflict-aware execution and the training stream. Section F details verified task crafting. Table S6 consolidates the constants of the generation pipeline, the evaluation protocol and TRISS. The prompt templates are reproduced verbatim on the last pages of this document, grouped by the pipeline step they serve: Figure S7 for grounded scene understanding and mission synthesis, Figure S8 for mission refinement, Figure S9 for verified instruction rendering, Figure S10 for the subtask scheduler of TRISS, and Figure S11 for its VLM-assisted planners. A IMPLEMENTATION DETAILS A.1 Simulation Platform Inference and evaluation run on a synchronized multi-agent platform that we build on Habitat 3 [1] (habitat-lab 0.3.1) with HM3D scenes [2], whereas planner training reuses the single-agent VLN-CE stack of ETPNav [3] (habitat-lab 0.1.7) and shares the topological memory across simulator instances (Section E.6). Each episode instantiates up to four Fetch articulated agents in kinematic mode. Every agent carries twelve RGB cameras (224× 224) and twelve depth cameras (256× 256, range [0, 10] m, normalized), all with a 90 ◦ horizontal field of view and oriented at 30 ◦ heading increments, so that each step yields the panoramic observation. Low-level motion is realized by a per-agent velocity controller integrated over a 1 s time step: a unit linear command translates the agent by 0.25 m and a unit angular command rotates it by 15 ◦ , reproducing the action granularity of VLN-CE [4]. A single environment step advances all agents simultaneously, so the step counter defines one synchronized clock for the whole team. The Time Steps (TS) metric counts these steps, and the Multi-Agent Conflict (MAC) metric treats each agent as a disc of radius ρ = 0.25 m. Subtask arrivals are credited within a 3 m geodesic radius of the nearest navigable anchor of the target instance (Section D.1). A.2 Perception Backbone and Waypoint Prediction Following ETPNav [3], RGB views are encoded by a CLIP image encoder into 512-dimensional vectors, and depth views are encoded by a ResNet-50 pretrained with D-PPO point-goal navigation on Gibson, whose 128× 4× 4 feature maps are average-pooled into 128-dimensional vectors. Both encoders are kept frozen and in evaluation mode throughout training. Neighboring viewpoints are proposed by the transformer-based waypoint predictor of [5], using the depth-driven checkpoint released with ETPNav for a 90 ◦ field of view, which is likewise frozen. The predictor outputs a heatmap over 120 heading bins (3 ◦ each) and 12 distance bins (0.25 m to 3 m in 0.25 m increments), which exposes the planner to plausible topologies. A.3 Pretraining The cross-modal planner is initialized from BERT-base-uncased and pretrained on discrete navigation graphs following DUET [6] and ETPNav [3]. The corpus merges the NavRAG [7] training set with the ground-truth pairs of atomic instructions and subtask trajectories extracted from the MAVLN training split: every subtask yields one pair whose instruction is its verified atomic instruction, whose path is the navigation-graph shortest path from the goal of the previous subtask of the same agent (or its start pose) to the current goal viewpoint, and whose positive endpoint is that goal viewpoint. All NavRAG trajectories located in the val-unseen and test-unseen scenes of MAVLN are removed beforehand. Two proxy tasks are optimized jointly. Masked Language Modeling (MLM) randomly masks instruction tokens and recovers them from the cross-modal text states conditioned on the topological graph built along the full demonstration. 2 STAGE 1 Grounded scene understanding 20,910 Viewpoints 1,628 Zones Scene context for mission synthesis GENERATION STEPMISSIONSRETAINED Mission synthesis 13,747 — Instance grounding 8,946 65.08% −4,801 missions Atomic instruction generation 8,945 99.99% Schedule refinement 4,060 45.39% −4,885 missions Atomic instruction refinement 3,956 97.44% Instruction rendering 3,956 100.00% Validity filter 3,908 98.79% STAGE 2 Diverse mission synthesis STAGE 3 Mission refinement STAGE 4 Verified instruction rendering DRAFT-TO-FINAL RETENTION 28.43% Fig. S1. Generation funnel of MAVLN. Stage 1 produces the viewpoint and zone annotations consumed by mission synthesis; subsequent rows report the missions retained after each generation step and the retention ratio relative to the preceding step. Single-step Action Prediction (SAP) truncates the demonstration at a random step, builds the graph from the observed prefix, and supervises the expert next node with fused logits (Section E.4). Pretraining runs for 200,000 steps, and the checkpoint with the best navigation performance on val-unseen (step 180,000) initializes imitation learning. A.4 Imitation Learning Imitation learning uses the per-subtask episodes of the MAVLN training split: every subtask of the decentralized rendering becomes one single-agent episode whose instruction is the verified atomic instruction, whose start pose is the goal pose of the previous subtask of the same agent (or the mission start pose for its first subtask), whose initial heading faces the final view of that previous trajectory, and whose goal is the subtask goal viewpoint with a 3 m success radius. Four simulator instances run concurrently, each hosting one agent, and instances whose current episodes belong to the same scene share one topological memory (Section E.6). Each iteration rolls out one episode per instance for at most T max = 30 planning steps under student forcing, and performs one AdamW update with learning rate 10 −5 . Training lasts 30,000 iterations with a checkpoint every 1,000 iterations, and the checkpoint maximizing the sum of SR and SPL on val-unseen is retained. The visual encoders and the waypoint predictor stay frozen, while the language, panorama and cross-modal encoders are updated. Training fits on a single A100 GPU. A.5 Inference Runtime The LLM scheduler is served locally through an OpenAI-compatible endpoint with JSON-formatted responses, using Qwen3.6-27b (FP8) or Gemma4-31b (W4A16 quantization-aware checkpoint); the same backbone performs instruction de- composition, scheduling and, when enabled, observation description and zero-shot planning (Section E.4). The observation- guided and VLM-assisted variants additionally tag each candidate view with the RAM++ image tagger (Swin-L backbone) at 384× 384 resolution. All LLM-dependent results are averaged over three runs. B DATASET DETAILS B.1 Generation Funnel Figure S1 traces every mission draft through the four crafting stages detailed in Section F. Grounded scene understanding annotates 20,910 viewpoints and aggregates them into 1,628 repaired zones, which form the scene context of mission 3 050100150200 Evaluation episodes Work & Collaboration Setup 190 Daily Routine & Transition Management 187 Homeschooling & Education Setup 151 Household Event & Guest Preparation 132 Personal Care & Wellness 79 Allergy-Safe Meal Preparation 44 Maintenance & Repair 32 Childcare & Supervision 14 Scenario domains 829 evaluation episodes · 8 categories COUNT (a) Scenario domains 05101520 Unique targets (%) Fixture/Architectural 195 Container/Holder 187 Storage Unit 166 Surface/Table 154 Appliance 113 Seating 96 Textile/Soft Furnishing 62 Electronic Device 56 Navigation targets 1,029 unique instances · 8 categories COUNT (b) Target categories Fig. S2. Composition of the evaluation splits. (a) Scenario domains of the 829 evaluation missions. (b) Categories of the 1,029 unique navigation target instances. synthesis. From a synthesis budget of 14,000 missions distributed over scenes and zones in proportion to their grounded viewpoints, 13,747 drafts are returned in a valid format. Instance grounding retains 8,946 of them (65.08%): the 4,801 discarded drafts violating the well-formedness rules (Table S5). Atomic instruction generation loses a single draft. Schedule refinement is the most selective step, retaining 4,060 of 8,945 missions (45.39%), because it rewrites the allocation by the makespan-optimal schedule and then enforces the navigation-graph and continuous-space leg bounds, the start-pose constraints and the start-to-target separation on every candidate instance combination; a mission is released only if at least one combination satisfies all of them. Perception-verified refinement of atomic instructions retains 3,956 missions (97.44%), dropping those for which the VLM cannot bind the instruction to a candidate instance with a visible mask. All 3,956 missions are rendered in the three instruction regimes, and the validity filter finally retains 3,908 (98.79%), discarding every mission whose instructions fail round-trip reconstruction or human assessment (val-unseen and test-unseen split) in any regime. The draft-to-final retention is therefore 28.43%. Two observations support the reliability of MAVLN. First, the heaviest attrition is caused by programmatic geometric and scheduling checks rather than by LLM-judged quality, so the released distribution is shaped by verifiable constraints. Second, the last filter removes only 1.21% of the rendered missions, which indicates that instructions produced from perception-verified atomic instructions under our rendering rules are already faithful to the constraint structure before filtering, and that the filter acts as a safeguard rather than as the primary source of quality. B.2 Scenario and Target Composition Figure S2 characterizes the 829 evaluation missions (val-unseen and test-unseen) and their navigation targets. The scenario domains are led by work and collaboration setup (190, 22.9%), daily routine and transition management (187, 22.6%), homeschooling and education setup (151, 18.2%), and household event and guest preparation (132, 15.9%), followed by personal care and wellness (79, 9.5%), allergy-safe meal preparation (44, 5.3%), maintenance and repair (32, 3.9%), and childcare and supervision (14, 1.7%). The four leading domains cover 79.6% of the evaluation missions, reflecting the household profiles that most often match the evaluation scenes, while the long tail keeps constraint-heavy scenarios such as supervision and care, in which continuous presence is a natural source of presence locks. The 1,029 unique target instances are spread over eight categories without a dominant one: fixtures and architectural elements (195, 19.0%), containers and holders (187, 18.2%), storage units (166, 16.1%), surfaces and tables (154, 15.0%), appliances (113, 11.0%), seating (96, 4 150200250300350400450500550 Words per mission (evaluation splits) 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative fraction Decentralized Centralized Implicit Fig. S3. Empirical cumulative distribution of words per mission on the evaluation splits under the decentralized, centralized and centralized-implicit regimes. To handle the evening transition and the plumbing emergency, follow this coordinated plan. First, locate the dark wooden vanity cabinet in the bathroom. This sink unit is mounted against the light-colored tiled wall, positioned directly beneath the window. You must maintain your position there to physically hold and stabilize the leaking pipe until the water supply is shut off. At the same time, navigate to the kitchen sink, which is integrated into the white countertop directly beneath the large window overlooking the outdoors. A silverdishwasher sits right next to the sink, while the dark wood cabinets and open shelving are located along the same wall. Wait until the leaking pipe at the bathroom vanity is secured before shutting off the main water supply valve. Shutting off this valve will release the person holding the pipe in the bathroom. Additionally, head over to the lounge area to find the large flat-screen television. The TV sits on a dark console, positioned directly beside the glass sliding doors that lead out to the deck. Wait until the water supply at the kitchen sink has been shut off before turning off the television to begin the tech-freeevening. Meanwhile, to prepare the living area, continue into the kitchen and make your way to the large stainless steeldouble-door refrigerator to retrieve the evening snacks. It is a tall, prominent appliance set within the dark wooden cabinetry, located right beside the white wall section that leads to the adjacent room. Also, navigate to the living room and look for the dark, round coffee table sitting centrally on the light-colored rug. It is placed directly in front of the white sectional sofa, which is decorated with yellow and grey throw pillows. You can identify the table by the small decorative item resting on itssurface, with the dark wood sideboard standing nearby. Wait until the snacks have been fetched from the refrigerator before placing them on the coffee table. Centralized Implicit To handle the evening transition and plumbing emergency, one agent should locate the dark wooden vanity cabinet in the bathroom,mounted against the light-colored tiled wall directly beneath the window, and maintain position there to physically hold and stabilize the leaking pipe. Simultaneously, a second agent must navigate to the kitchen sink, which is integrated into the white countertop directly beneath the large window overlooking the outdoors next to a silver dishwasher. The second agent must wait until the pipe is secured by the first agent, then shut off the main water supply valve, resolving the emergency and releasing the first agent. Meanwhile, a third agent should make their way to the large stainless steeldouble-door refrigerator, set within the dark wooden cabinetry right beside the white wall section, to retrieve the evening snacks. The third agent must carry the snacks to the living room and place them on the dark,round coffee table sitting centrally on the light-colored rug directly in front of the white sectional sofa with yellow and grey throw pillows. Finally, once the first agent is released from holding the leaking pipe, they should head over to the lounge area to find the large flat-screen television sitting on a dark console directly beside the glass sliding doors that lead out to the deck, andshut down the TV to begin the tech-free evening. Centralized Agent 1: Locate the dark wooden vanity cabinet in the bathroom, which is mounted against the light-colored tiled wall directly beneath the window. You must maintain your position here to physically hold and stabilize the leaking pipe to prevent flooding. Wait until your partner shuts off the main water supply before leaving this spot. Once you are released, head over to the lounge area to find the large flat- screen television sitting on a dark console directly beside the glass sliding doors that lead out to the deck, andshut down the TV to prepare for the tech-free evening. Agent 2: Navigate to the kitchen sink, which is integrated into the white countertop directly beneath the large window overlooking the outdoors, with a silver dishwasher sitting right next to it and dark wood cabinets along the same wall. Wait until your partner has secured the leak at the bathroom sink cabinet, andthen shut off the main water supply valve to safely stop the leak and release your partner. Agent 3: Make your way to the large stainless steeldouble-door refrigerator, a tall prominent appliance set within the dark wooden cabinetry located right beside the white wall section that leads to the adjacent room, andretrieve the evening snacks. After retrieving the snacks, carry them to the living room and look for the dark, round coffee table sitting centrally on the light-colored rug directly in front of the white sectional sofa decorated with yellow and grey throw pillows. You can identify the table by the small decorative item resting on its surface and the nearby dark wood sideboard; place the snacks there to complete your task. Decentralized Fig. S4. Mission instructions of the example episode, rendered under the decentralized, centralized and centralized-implicit regimes. 9.3%), textiles and soft furnishings (62, 6.0%), and electronic devices (56, 5.4%). No category exceeds a fifth of the targets, and a substantial portion consists of small or visually ambiguous objects such as containers, holders and devices, which must be distinguished from same-category distractors through the fine-grained attributes injected by perception-verified refinement. B.3 Instruction Length Figure S3 plots the empirical cumulative distribution of words per mission on the evaluation splits for the three regimes. Mission instructions run to several hundred words, far longer than typical single-agent VLN instructions, because every atomic destination is described with the perception-verified attributes that separate it from nearby distractors and because each regime additionally verbalizes the coordination structure. The decentralized regime distributes the mission over per- agent instructions, each of which restates the cross-agent conditions gating its subtasks; the centralized regime consolidates the plan into one instruction; and the centralized-implicit regime removes agent attribution while keeping every completion dependency, so its length reflects the constraint content rather than the allocation. Consequently, the regimes differ in how the constraint structure is expressed rather than in the amount of navigational content, which is shared across them by construction. 5 Locate the dark wooden vanity cabinet in the bathroom. This sink unit is mounted against the light-colored tiled wall, positioned directly beneath the window. You are at the kitchen sink, which is integrated into the white countertop directly beneath the large window overlooking the outdoors. A silver dishwasher sits right next to the sink, while the dark wood cabinets and open shelving are located along the same wall. Continue into the kitchen and make your way to the large stainless steeldouble-door refrigerator. It is a tall, prominent appliance set within the dark wooden cabinetry, located right beside the white wall section that leads to the adjacent room. In the living room, look for the dark, round coffee table sitting centrally on the light-colored rug. It is placed directly in front of the white sectional sofa, which is decorated with yellow and grey throw pillows. You can identify the table by the small decorative item resting on its surface,with the dark wood sideboard standing nearby. Head over to the lounge area to find the large flat-screen television. The TV sits on a dark console, positioned directly besidethe glass sliding doors that lead out to the deck. Subtask id: 1 Agent ids: [1] Locked:true Unlock: [] Holding: false, Depends on: [] Subtask id: 2 Agent ids: [2] Locked:false Unlock: [1] Holding: false, Depends on: [1] Subtask id: 3 Agent ids: [3] Locked:false Unlock: [] Holding: true, Depends on: [] Subtask id: 4 Agent ids: [3] Locked:false Unlock: [] Holding: false, Depends on: [3] Subtask id: 5 Agent ids: [1] Locked:false Unlock: [] Holding: false, Depends on: [2] Fig. S5. Constraint structure, per-subtask atomic instructions and rendered agent trajectories of the same example episode. B.4 Episode Visualization Figures S4 and S5 illustrate one released episode end to end, so that the artifacts described above can be inspected on a concrete mission. Figure S4 shows the three renderings of the same mission. The decentralized rendering assigns one instruction to every agent and restates, inside each of them, the cross-agent conditions that gate its subtasks; the centralized rendering merges the plan into a single instruction that still names the executor of every destination; and the centralized- implicit rendering removes all agent attribution while preserving every completion dependency, which is exactly the information a system must recover in order to allocate the subtasks itself. Figure S5 shows the underlying structure of the same episode: the subtask graph with its dependency, holding and locking attributes, the atomic instruction of every subtask, and the rendered trajectory of every agent through its goal viewpoints. Reading the two figures together exposes the correspondence that verified task crafting establishes and that evaluation relies upon. C ADDITIONAL EXPERIMENTS C.1 Scheduling with Oracle Navigation Table S1 isolates the LLM scheduler by replacing planning and control with the oracle navigator of Section D.5. Under the oracle scheduler, oracle navigation completes every mission, so any deficit in this table is attributable to scheduling. Scheduling alone already costs a substantial share of missions. Under decentralized instructions, Qwen3.6 reaches 92.3 and 92.7 SR on val-unseen and test-unseen, and Gemma4 reaches 82.6 and 84.2. Under centralized instructions, Qwen3.6 drops to 86.9 and 90.7 SR, while Gemma4 remains at 82.0 and 84.2. The centralized-implicit regime is the hardest, with 80.0 and 82.1 SR for Qwen3.6 and 70.2 and 71.4 for Gemma4, confirming that inferring the allocation on top of the constraint structure is a distinct source of difficulty. 6 TABLE S1 Performance on MAVLN under oracle navigation with LLM scheduling. Val-UnseenTest-Unseen Scheduler Navigator SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ Decentralized Instruction Style Gemma4Oracle82.6±1.3 82.5±1.3 94.5±0.5 94.4±0.5 87.6±0.5 94.5±0.5 84.2±0.8 84.2±0.8 95.5±0.1 95.5±0.1 88.3±0.2 95.8±0.2 Qwen3.6Oracle92.3±0.7 92.2±0.7 96.6±0.3 96.4±0.3 89.6±0.3 96.9±0.4 92.7±0.6 92.7±0.6 97.0±0.4 96.8±0.4 89.7±0.3 97.4±0.3 Centralized Instruction Style Gemma4Oracle82.0±1.0 81.4±1.1 93.9±0.6 92.7±0.6 87.6±0.4 94.6±0.5 84.2±1.0 83.1±1.0 94.5±0.3 92.8±0.3 87.5±0.1 95.1±0.2 Qwen3.6Oracle86.9±0.7 86.1±0.9 94.0±0.2 92.5±0.4 88.5±0.0 95.8±0.0 90.7±0.2 89.4±0.2 95.3±0.4 93.5±0.4 88.9±0.4 96.8±0.4 Centralized Implicit Instruction Style Gemma4Oracle70.2±0.9 67.2±0.9 89.4±0.7 82.4±0.8 85.0±0.3 92.6±0.3 71.4±0.7 65.9±0.6 89.9±0.6 81.2±0.4 84.9±0.4 93.1±0.5 Qwen3.6Oracle80.0±1.2 76.2±1.2 91.1±0.9 83.8±0.9 88.8±0.5 97.0±0.3 82.1±0.1 75.7±0.1 92.0±0.2 82.7±0.1 87.9±0.4 96.8±0.5 TABLE S2 Comparison of embedding-based and LLM-annotated instruction-to-subtask matching. Val-UnseenTest-Unseen Instruction StyleMatching SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ SR↑ SPL↑ CSR↑ CSPL↑ ISPL↑ TC↑ Decentralizedembed8.33.335.218.425.097.37.03.931.718.425.697.0 annotated 8.33.335.118.325.097.37.03.831.718.425.697.0 ∆0.00.0+0.1+0.1+0.1+0.10.0+0.00.00.00.00.0 Centralizedembed7.53.134.218.023.695.67.23.631.818.525.795.9 annotated 7.53.134.318.023.695.87.43.731.918.625.696.2 ∆0.00.0-0.10.00.0-0.2-0.2-0.1-0.1-0.1+0.1-0.3 Centralized Implicit embed5.42.331.516.623.196.67.44.028.916.724.396.7 annotated 5.42.331.416.723.196.67.23.928.816.624.396.9 ∆0.00.0+0.10.00.0-0.1+0.2+0.1+0.2+0.10.0-0.2 Embed denotes sentence-embedding matching; annotated denotes Qwen3.6 annotation. The subtask-level metrics locate the failures. TC stays between 92.6 and 97.4 and CSR between 89.4 and 97.0 in all configurations, so most failures are not missing subtasks but a few constraint violations or premature terminations that invalidate an otherwise completed mission. Path efficiency degrades with allocation freedom: under decentralized instructions SPL is within 0.1 of SR, whereas under centralized-implicit instructions the gap grows to 3.0–6.4 points and CSPL falls 7.0–9.3 points below CSR, indicating that when the scheduler must choose executors, it often chooses feasible but longer allocations than the makespan-optimal reference. ISPL remains between 84.9 and 89.7 throughout, below the 92.6 and 92.0 attained under oracle scheduling, which shows that scheduling also affects how far agents travel between declarations. Qwen3.6 outperforms Gemma4 in SR in every regime and split under oracle navigation, although the two backbones perform comparably with the learned navigator; the navigation bottleneck therefore masks scheduling differences that become visible once navigation is solved. Consequently, improvements in scheduling are necessary but not sufficient: their benefit is bounded by navigation quality, while their absence caps performance even under perfect navigation. C.2 Robustness of Instruction-to-Subtask Matching Table S2 compares the embedding-based matching of Section D.4 with an LLM-annotated mapping, in which Qwen3.6 reads each emitted atomic instruction together with the ground-truth pool and annotates the subtask it refers to, on the same Qwen3.6 scheduling runs. Mission-level metrics are identical on val-unseen and differ by at most 0.2 points on test-unseen, and every subtask-level difference lies within ±0.3 points. Since the two matchers rely on entirely different mechanisms, this agreement indicates that the reported metrics do not hinge on the particular matcher, and it justifies the lightweight sentence-embedding matcher as a reproducible default. The largest differences appear in TC under the centralized and centralized-implicit regimes, where any agent may execute any subtask, so that surplus or paraphrased declarations are plausibly harder to attribute. D EVALUATION DETAILS D.1 Navigable Goal Anchors Measuring arrivals against a single object center systematically mis-scores large furniture: the axis-aligned bounding-box (AABB) center of a bed or cabinet may project to an interior or unreachable point, while an agent legitimately standing near another side of the object would still be penalized. We therefore associate each target instance with a small set of navigable 7 (a)(b)(c) Fig. S6. Navigable goal anchors rendered as blue cylinders in panoramas captured at goal viewpoints. (a) Anchors around a suitcase in a walk-in closet. (b) Anchors around an armchair in a bedroom. (c) Anchors of a wooden chest in a bedroom, where deduplication and reachability leave two anchors. anchors. Specifically, we take the AABB center and its six face centers as candidate points and project each candidate onto the navigation mesh. Using a viewpoint that certifies the instance, we discard projected candidates that are not geodesically reachable from that viewpoint. We then scan the remaining projections in order and remove duplicates, retaining a point only if it is at least 0.5 m from all previously retained anchors. Instances for which no anchor remains are not grounded at that viewpoint. Let d(x,o) denote the distance from a position x to an instance; and the set of instances reached from x is R(x) = o∈g 1 ,...,g N : d(x,o) < 3 m .(S1) Figure S6 visualizes the anchors of three evaluation targets. The anchors surround each object on its accessible sides and collapse to a few points for compact objects, so that an agent approaching from any accessible direction is credited consistently. The same anchors define the grounding radius at generation time (Section F.1), the geodesic supervision of the oracle components (Section D.5) and the arrival test at evaluation time, so generation and evaluation share one notion of reaching a target. D.2 Arrival Declarations and Metric Definitions An agent declares an arrival when its allocated atomic instruction completes (Section E.1). The j-th declaration of an episode is the tuple δ j = (a j ,t j ,R j , ˆ ℓ j ,κ j ), where a j is the declaring agent, t j the synchronized time step of the declaration, R j = R(x t j a j ) the set of instances reached at that moment, ˆ ℓ j the atomic instruction the agent was executing, and κ j the ground-truth subtask matched to ˆ ℓ j (Section D.4), with κ j = ∅ when no matching is used. The evaluator of Section D.3 consumes the declaration sequence and returns three subtask sets: the completed subtasksC ⊆ T , which were consumed by some declaration; the atomically successful subtasksS ◦ ⊆C, whose target was reached irrespective of constraints; and the successful subtasksS ⊆S ◦ , whose declaration additionally honors all dependency, holding and locking constraints and whose dependencies are themselves successful. For the declaration δ j consumed by τ k , let j − be the previous declaration of the same agent (the episode start if none), let p k = P t j t=t j − +1 ∥x t a j −x t−1 a j ∥ be the distance the agent traveled in between, let λ ◦ k = d(x t j − a j ,g k ) be the geodesic distance from its position at t j − to the target, and let λ ⋆ k be the precomputed length of the fragment of τ k in the optimal reference execution. With the total traveled distance P = P a∈A P T t=1 ∥x t a − x t−1 a ∥ over the T synchronized steps of the episode, the metrics are SR = I S =T ,SPL = SR· P k λ ⋆ k max P, P k λ ⋆ k , CSR = |S| N ,CSPL = 1 N N X k=1 I τ k ∈S λ ⋆ k max(λ ⋆ k ,p k ) , TC = |C| N ,ISPL = 1 N N X k=1 I τ k ∈S ◦ λ ◦ k max(λ ◦ k ,p k ) , TS = T,MAC = 1 T M T X t=1 I h ∃a̸= a ′ :∥x t a − x t a ′ ∥ < 2ρ i , (S2) where fractions are set to zero when their denominator vanishes, and all metrics are averaged over episodes. ISPL rewards efficient approach to the target from wherever the agent actually was, independent of whether it should have gone there, whereas CSPL compares against the optimal team execution and rewards only constraint-consistent arrivals. TS is the realized makespan, which, unlike a planned makespan, absorbs congestion, waiting and deadlock during execution. MAC counts the synchronized steps with at least one pair of agents closer than 2ρ and normalizes by the team size, so that values remain comparable across team sizes. Presence locks are verified on the realized trajectory. If τ k ∈ S is locked and was consumed by δ j , the mission fails if the executing agent leaves the success radius of any instance credited at δ j before the release. 8 Algorithm S1: Constraint-Aware Discrete-Event Evaluation Input : TSubtask DAG with attributes (g k ,A k ,D k ,h k ,l k ,U k ) δ j J j=1 Declarations (a j ,t j ,R j , ˆ ℓ j ,κ j ) 1 H←h 0 with all subtasks incomplete and last(a)← ∅ for all a 2 for j = 1 to J do 3 H ′ ←∅ 4for hypothesis h∈H do 5Φ←τ k incomplete in h : D k completed in h, a j ∈ A k , κ j ∈∅,k // ready subtasks 6 R ′ ←g κ j if κ j ̸= ∅ and g κ j ∈R j ;∅ if κ j ̸= ∅ otherwise;R j (or∅ if empty) if κ j = ∅ 7for target r ∈R ′ do 8Φ r ←τ k ∈ Φ : g k = r 9if Φ r ̸=∅ then 10for τ k ∈ Φ r doH ′ ←H ′ ∪Commit(h,j,τ k , true) 11else if Φ̸=∅ then 12for τ k ∈ Φ doH ′ ←H ′ ∪Commit(h,j,τ k , false) 13elseH ′ ←H ′ ∪Commit(h,j, ∅, false) 14 H←H ′ Output: Hypothesis selected lexicographically, its setsC,S ◦ ,S after failure propagation, and the next oracle assignment 15 Function Commit(h,j,τ, correct): 16 h ′ ← copy of h; τ − ← last(a j ) in h ′ 17if τ − ̸= ∅ and τ − is still locked then mark τ − failed, and mark τ failed as well if τ ̸= ∅ and correct 18if τ = ∅ then return h ′ 19mark τ completed; mark τ atomically successful and successful if correct , failed otherwise 20if correct then 21if τ − is holding and (τ − /∈ D τ or the drafted executors of τ − and τ differ) then mark τ failed 22if some τ d ∈ D τ is incomplete or failed then mark τ failed 23for u∈ U τ that is the last subtask of another agent do unlock u 24last(a j )← τ ; return h ′ D.3 Discrete-Event Evaluator Algorithm S1 evaluates a declaration sequence under the task semantics. Because several instances may lie within the success radius and a declaration may be attributable to several ready subtasks, the evaluator maintains a set of hypotheses, each holding a copy of the DAG state and the last subtask of every agent. For every declaration it branches over all consistent interpretations: a ready subtask of the declaring agent whose target was reached is consumed as a correct arrival; otherwise, any ready subtask of that agent may be consumed as an incorrect arrival, which counts toward completion but not toward success; if the agent has no ready subtask, the declaration consumes nothing but still breaks a presence lock held by that agent. When a subtask identifier κ j is available from matching, the branching collapses to that subtask, which is consumed correctly if and only if its own target is inR j . Failure propagates upward: a subtask fails whenever any of its dependencies fails, and the mission succeeds only if every subtask succeeds. Among the final hypotheses, the evaluator retains those whose pending assignments are consistent with the task state and that admit a complete future execution, and selects the one that lexicographically maximizes TC, then maximizes CSR, then minimizes the number of future rounds, with the order of consumed subtasks as the final tie-breaker; when no hypothesis admits a complete future, the same key without the future rounds is applied to all hypotheses. The future execution is computed by a branch-and-bound search over round-wise assignments with pre-allocation (Section D.5), which also yields the next assignment of the oracle scheduler. Without matching, the evaluator thus resolves ambiguity in favor of the team, which corresponds to the greedy strategy that maximizes task completion and breaks ties by conditional success. D.4 Instruction-to-Subtask Matching Since the scheduler emits atomic instructions that it extracted itself, a declaration does not identify which ground-truth subtask the agent attempted. We recover this mapping by text similarity. We use all-MiniLM-L6-v2 to obtain sentence embeddings of a case-folded, whitespace-normalized text. Let ˆ ℓ j J j=1 be the instructions of the declarations of an episode, and letℓ ⋆ k N k=1 be the ground-truth atomic instructions. Every declaration is a separate row, even when two declarations carry identical text. The mapping solves the rectangular assignment problem over one-to-one partial assignments of maximum cardinality, solved by the Hungarian algorithm. Each matched declaration receives κ j = k; unmatched surplus declarations are recorded for auditing and removed from the declaration sequence before the metrics are recomputed. The 9 Algorithm S2: Episode Loop of TRISS Input : L Task instructions A Team of M agents T max Planning step limit π θ Trained planner 1 I ← LLM dec (L);G ←∅; P 0 ←∅ // atomic pool, shared topology, pending set 2 for round n = 1, 2,... do 3Reset visitation records and stop scores; localize all agents inG (Algorithm S3) 4(I n ,P n ,F n )← SCHEDULE(L,I,n) 5for agent a∈ F n in order do declare the pending arrival of a; update the task state 6if I (a) n = ∅ for all a and F n =∅ then break 7Encode ˆ ℓ (a) for every agent with I (a) n ̸= ∅; mark these agents active 8for step t = 0,...,T max − 1 while some agent is active do 9for every active agent a do 10Localize a and updateG from its panorama (Algorithm S3) 11encode its panorama, update node features, and compute fused logits s (a) over ˆ V (a) t 12Assign targets to active agents by maximum-score bipartite matching overs (a) (Equation (S7)) 13for every active agent a do 14if a selects stop or t = T max − 1 or no candidate is selectable then 15goal← node with the highest stop score of a in this round; mark a idle 16else goal← selected node; remove it fromG if unexplored and record the attempt 17Idle agents hold their current node; routes← CBS(G, starts, goals) (Algorithm S4) 18Execute all routes synchronously with the low-level controller (Section E.5); collect verified anchors 19for agent a with I (a) n ̸= ∅ and a /∈ P n do declare the arrival of a; update the task state Output: Declaration sequence evaluated by Algorithm S1 mapping prevents an evaluator from crediting an arrival to whichever subtask happens to be satisfied, and it is robust to the choice of matcher (Table S2). It presupposes that the evaluated system exposes a pool of atomic instructions. D.5 Oracle Components Oracle scheduler. The oracle scheduler reveals the ground-truth atomic instructions together with the next round of the evaluator’s optimal future execution. This execution is found by a branch-and-bound search over rounds: in each round, pending subtasks whose dependencies are complete fire first in subtask order; each free agent (neither locked nor pending) receives at most one undispatched subtask it is permitted to execute, where a holding agent may only receive the consumer of its held subtask with the same drafted executor, and a subtask consuming a held object is reserved for the holder once the object has been acquired; assigned subtasks with complete dependencies fire at the end of the round, whereas the others are pre-allocated and become pending; and at least one assigned subtask must be dependency-ready. Assignments are expanded in order of decreasing size and decreasing number of ready subtasks, and a branch is pruned when the rounds used cannot improve on the best complete plan. The legacy variant (LE) restricts dispatch to dependency-ready subtasks and thus disables pre-allocation. Oracle navigator. The oracle navigator replaces planning and control by a geodesic follower that drives the agent toward the goal position of its allocated subtask along the navigation-mesh shortest path, moving 0.2 m per step when aligned with the next path point within 0.1 rad and turning by 10 ◦ otherwise. It terminates when the agent is within 0.2 m of the goal on the horizontal plane, after 500 steps, or immediately if the goal is unreachable. Oracle planner. The oracle planner keeps the shared topology, conflict-based path finding and control of TRISS but replaces the policy decision, and hence the target allocation, by the expert action used for imitation learning (Section E.6): it stops when the geodesic distance to the goal of the allocated subtask is below 1.5 m, and otherwise selects the selectable candidate whose simulated position is geodesically closest to that goal. E METHOD DETAILS E.1 Episode Loop of TRISS Algorithm S2 combines the three modules of TRISS into one episode. An episode alternates scheduling rounds and navigation rounds on the synchronized platform. At the beginning of round n, every agent is localized in the shared topology, the scheduler emits the assignment I n , the newly pre-allocated agents P n and the ordered firing list F n , and the previously pending actions listed in F n fire one by one, each as a separate declaration so that a release caused by an earlier firing is visible to later ones. The agents holding a non-blank instruction then navigate for at most T max planning steps. At every step, the active agents update and read the shared topology, the planner scores their candidate nodes, the conflict-free target allocation assigns distinct targets, the conflict-based path finder converts the targets into synchronized 10 routes over the shared topology, and the low-level controllers execute these routes. An agent becomes idle once it stops, reaches the step limit, or has no selectable candidate left. When all agents are idle, every allocated agent that is not pending declares its arrival, the evaluator updates the task state, and the next round begins. The topology persists across rounds, whereas per-agent visitation records and stop scores are reset at every round. The episode terminates when a round yields a blank assignment for every agent without firing any pending action. E.2 LLM-based Subtask Scheduler Atomic instruction extraction. At the start of an episode, the extractor LLM decomposes L into the shared pool I = ℓ i L i=1 , addressed by integer indices. Its prompt defines a navigation destination as the specific target the agent interacts with together with everything that identifies it (its room, appearance, placement, surroundings and markers), so that a sentence introducing a room and a subsequent sentence describing a target within that room form a single destination. The extractor must preserve all identifying detail and the original wording, and may only remove on-arrival actions and coordination or dependency clauses. The call is repeated until it returns a non-empty list of strings. Scheduling protocol. At each round, the scheduler receives the instruction(s)L, the poolI , the per-round history of issued indices, the per-round history of the pending set, its own previous scheduling plans, and the list of agent identifiers, and returns an updated scheduling plan describing future rounds only, the per-agent indices I n , the ordered firing list F n and the newly pre-allocated agents P n . Observation-guided variant. The observation-guided scheduler additionally receives a situation report S n = a, v (a) n , desc(v (a) n ) a∈A (S3) that pairs each agent’s current viewpoint v (a) n with a description of the place, and its prompt instructs the scheduler to use this report as the primary evidence for allocation whenever the instruction does not bind a subtask to an agent. The description is generated once per viewpoint at the start of a round by the same VLM from the views in the directions of the current waypoint proposals, their headings relative to the agent, and RAM++ landmark tags. Its prompt requires an orientation-agnostic description that anchors objects to architectural elements and to each other rather than to the viewer, emphasizes cues that distinguish the place from nearby alternatives, and, given the atomic pool as background, notes which targets are plausibly correlated with the current location while marking such inferences as uncertain. Round execution. The environment realizes the protocol as follows. At the beginning of round n, each agent in F n fires its previously pending subtask as an individual declaration in the listed order, and is removed from the pending set only at its own firing, so that later firings in the same list are evaluated after earlier releases. The new pending agents P n are registered only after all firings of the round, and the instructions of I n are installed afterwards. At the end of the round, allocated agents outside the pending set declare their arrival, whereas pending agents remain at their destinations until they appear in a later firing list. Under the oracle scheduler, the same procedure is driven by the evaluator’s optimal future execution (Section D.5) and additionally carries ground-truth subtask identifiers, so that each declaration is evaluated against its intended subtask. E.3 Shared Topology Maintenance Representation. The shared topologyG t = (V t ,E t ) stores, for every explored node v ∈V e t , a measured position ˆp v , and for every unexplored node v ∈ V u t the list of position estimates at which it has been proposed together with their mean ˆp v . The explored set is shared by the team, whereas each agent a additionally holds the setV (a) t ⊆V e t of nodes it has visited in the current round. Edges carry metric lengths, and the memory maintains the all-pairs shortest distance matrix E together with a predecessor table from which shortest node sequences are reconstructed; E is consumed as the distance bias of the coarse-scale encoder, in the positional features of the planner, and as transition cost and heuristic of the space-time search. The memory further stores node features (Section E.4), per-agent stop scores for the current round, and the collective ban list B. Localization and node identity. Two radii govern node identity. The identification radius ε loc = 0.5 m decides whether a position is an existing node: before viewpoint prediction, the measured position q t of an agent is localized by Equation (S4) restricted to explored nodes, and if the nearest one lies within ε loc it becomes the current viewpoint v t . Otherwise, a new explored node is created at ˆp v t = q t . Waypoint proposals are localized against all nodes except v t , using the mean estimate for unexplored nodes, so that proposals from different agents and vantage points merge instead of forking, v ∗ (q) = argmin v∈V t ∥ ˆp v − q∥, q is identified with v ∗ (q) ⇐⇒ ∥ ˆp v ∗ (q) − q∥≤ ε loc . (S4) The neighbor radius ε nbr = 1.5 m governs stitching at the beginning of a round: an agent whose start position localizes to no explored node is instantiated as a new node and connected by metric edges to all explored nodes within ε nbr , so that a freshly placed agent inherits the connectivity of its teammates rather than starting an isolated component. Within a round, a newly created node is connected only to the verified anchors returned by the controller (traversal-verified connection); a localized node is connected to these anchors if it differs from them. The localized v t is also connected to its predecessor. Unexplored-node lifecycle. A proposal y i that matches an unexplored node appends its estimate to that node, updates the mean, and inherits no new edge from the matching itself. If the node is already connected, the geodesics are relaxed 11 Algorithm S3: Shared Topology Update for One Agent Input : G t Shared topology with E, B q t ,φ t Measured pose of agent a Y t Viewpoint proposals ΛVerified anchors from the controller attPending attempt (u,y) of a ε loc ,ε nbr ,ε ban Radii // Localization 1 v e ← argmin v∈V e t ∥ ˆp v − q t ∥ 2 if∥ ˆp v e − q t ∥≤ ε loc then 3 v t ← v e ; if att ̸= ∅ then B ← B∪(u,y) // failure-verified 4connect v t to every λ∈ Λ\v t 5 else 6create explored node v t with ˆp v t = q t 7if first step of the round then connect v t to all v ∈V e t with∥ ˆp v − q t ∥ < ε nbr 8else connect v t to every λ∈ Λ // traversal-verified 9 att ← ∅ // Proposal filtering and identification 10 if v t has ban entries then remove every y i violating Equation (S9) from Y t 11 for y i ∈ Y t do 12 v ∗ ← argmin v∈V t \v t ∥ ˆp v − y i ∥ 13if∥ ˆp v ∗ − y i ∥≤ ε loc then 14if v ∗ ∈V u t then append y i to the estimates of v ∗ and update ˆp v ∗ 15 V ′ t ←V ′ t ∪v ∗ 16else create unexplored node v with ˆp v = y i ; Y ′ t ← Y ′ t ∪v 17 for v ∈V ′ t ∪ Y ′ t do 18if v ∈V e t then 19add edge (v t ,v) if absent 20else 21if v was connected then re-insert its edges with lengths to ˆp v and relax through each neighbor 22add edge (v t ,v) 23 Add v t toV (a) t andV e t 24 Prune unexplored neighbors of v t not proposed in the current step Output: UpdatedG t+1 , current node v t , candidate neighborsV ′ t ∪ Y ′ t through each previously connected node; a proposal matching an explored node yields a direct edge to it. A proposal that matches nothing creates a new unexplored node, and every matched or created node is connected to v t . After the update, unexplored neighbors of v t that were not proposed by any agent in the current step are pruned, so that stale frontiers do not accumulate. When an agent selects an unexplored node as its target, the node is removed from the memory at dispatch; the position the agent actually reaches is inserted as an explored node at the next localization, and its identity is decided by Equation (S4). Hence explored positions are always measured, never extrapolated from proposals. E.4 Topological Memory-based Navigation Planner Text encoder. An atomic instruction ℓ is tokenized and embedded by a stack of transformer layers into ˆ ℓ ∈ R |ℓ|×D with hidden size D = 768. Panorama encoder. At viewpoint v t , each of the twelve views is represented by its RGB feature r (i) t , depth feature d (i) t , orientation feature o (i) = [sinθ i , cosθ i , sinφ i , cosφ i ] and a binary type n (i) indicating whether the view carries a waypoint proposal. Views carrying proposals are listed first, one entry per proposal, followed by the remaining views. The panorama encoder yields the contextual panoramic embeddings ˆr t = ˆr (i) t . The representation of v t is overwritten by the average of ˆr t over valid views, whereas an unexplored node proposed through view i accumulates ˆr (i) t and is represented by the mean of its accumulated embeddings. A node that has become explored no longer accumulates proposal embeddings. Coarse-scale encoder. The candidate set ˆ V (a) t = V e t ∪ N V (a) t is realized within the connected component of v t as the union of three groups: the nodes V (a) t visited by the agent in the current round, the other explored nodes, and the unexplored nodes directly adjacent toV (a) t . The first group provides context but is masked from selection. The coarse-scale encoder augments the candidate node features x i for v i ∈ ˆ V (a) t with positional encodings and a learned stop token x 0 , and 12 processes X = [x 0 ,x 1 ,...,x | ˆ V (a) t | ] together with ˆ ℓ through cross-modal layers. Each layer first attends from X to ˆ ℓ, then applies Graph-Aware Self-Attention (GASA) [6], GASA(X) = Softmax XW q (XW k ) ⊤ √ d + Z XW v , Z = E w e + b e , (S5) and finally a feed-forward block, where the distance bias Z applies a learned scalar affine map to the pairwise shortest distances E restricted to the candidate set (with zero distance for the stop token) and is shared across attention heads. Because E is computed on the shared topology, the bias spans the union of the team’s exploration. Global action logits are computed as s (c) i = FFN c (ˆx i ), where ˆx i ∈ ˆ X denotes the output of the coarse-scale encoder, providing node-level navigation preferences. Fine-scale encoder. The fine-scale encoder processes R = [r 0 ; ˆr t ], where the stop token r 0 is a zero vector, with fourteen- dimensional positional features that concatenate the positional feature of the agent’s start node of the round, outputting contextual panoramic embeddings ˆr i . Local action logits are computed as s (f) i = FFN f (ˆr i ). Local logits are converted to the global action space by Equation (S6), where s back sums the local logits of the candidate views whose proposals localize to nodes already visited by the agent, s (f ′ ) i = ( s (f) i ,if v i ∈ Y ′ t ∪V ′ t , s back , otherwise. (S6) Dynamic fusion and stop selection. The scalar gate is σ t = Sigmoid FFN σ ([ˆr 0 ; ˆx 0 ]) , and the fused logit of every candidate, including the stop action i = 0, is s i = σ t s (f ′ ) i + (1− σ t )s (c) i . The stop probability softmax(s) 0 is recorded as the stop score of v t for the agent. When the agent stops, it does not necessarily stop at v t : following DUET, it travels to the node with the highest stop score among those it has scored in the current round. VLM-assisted variants. The zero-shot VLM planner and the hybrid planner share the shared topology and conflict- aware execution. For the zero-shot planner, the VLM receives the atomic instruction, the per-round instruction history of all agents, the targets reached in previous rounds, the agent’s own step history and planning notes, the images of the candidate views, textual action options and the list of admissible action strings, and returns a thought, a one-sentence plan and exactly one admissible action. Action options are verbalized as “go forward”, “turn slight left/right”, “turn sharp left/right” or “turn around” toward a viewpoint according to its relative heading (sectors bounded at ±30 ◦ , ±90 ◦ and ±150 ◦ ), or as “backtrack to viewpoint” for nodes already visited by the agent; nodes outside the current neighborhood are offered as backtracking options only if they are on the trained planner’s shortlist (defined below) or were final targets of previous rounds. Each option is enriched with the RAM++ landmark tags observed toward the corresponding node, and the zero-shot planner stops at its current node. The hybrid planner queries the VLM only when the trained planner is uncertain. Let p (1) ≥ p (2) ≥ · be the softmax probabilities of the fused logits over selectable candidates and the stop action. The expert is considered confused if p (1) − p (2) ≤ κ, and it recommends the ranked shortlist c : p c ≥ γ p (1) truncated to n max entries, which is provided to the VLM as a strong prior. Confident agents keep the conflict-free allocation of Equation (S7), whereas confused agents query the VLM sequentially, with the targets already claimed by idle agents, by confident agents and by previously queried agents removed from their options; stopping is admissible after the first step if the current node is unclaimed, or whenever no option remains. We use κ = 0.03, γ = 0.5 and n max = 5. E.5 Conflict-Aware Execution Conflict-free target allocation. Within an episode, letA act be the active agents, letV I be the current nodes of idle agents, and let W = S a∈A act v ∈ ˆ V (a) t (a) t : v /∈ V I be the pool of selectable nodes. Each agent additionally owns a private stop column ς a . Allocation solves the maximum-score bipartite assignment max z∈0,1 A act ×(W∪ς a ) X a,c z ac S ac s.t. X c z ac = 1 ∀a, X a z ac ≤ 1 ∀c, (S7) where S av = s (a) v for selectable nodes of agent a, S aς a = s (a) 0 . The problem is solved by the Hungarian algorithm; the private stop columns guarantee feasibility, and an agent assigned to its stop column executes the stop action. Every agent is thus dispatched to a distinct node, and an agent whose preferred frontier is claimed by a higher-scoring teammate is rerouted to its best remaining alternative within the same solve. Conflict-based path finding. Given the start node s i and goal node g i of every agent (idle agents use s i = g i ), the planner first verifies by breadth-first search over admissible moves that every goal is reachable and derives the horizon T h = max 5, max i:s i ̸=g i hops BFS (s i ,g i ) + 1 + M , (S8) which reserves one additional waiting slot per agent for conflict resolution. The low-level space-time A* plans each agent a sequence of exactly T h states starting at t = 0. A move follows a direct edge to an explored node, or to the agent’s own goal; an unexplored goal can only be entered at the final step, and no move departs from an unexplored node. Waiting is always admissible. The transition cost is the metric length of the move (zero for waiting), the heuristic is the stored shortest distance to the goal (zero if unknown), and a state is terminal only if t≥ T h − 1 and the agent occupies its goal, so an agent 13 Algorithm S4: Conflict-Based Path Finding on the Shared Topology Input : GShared topology with E s 1:M Start nodes g 1:M Allocated goal nodes δCollision distance B cbs Expansion budget // Reachability and horizon 1 if some g i is unreachable from s i by admissible moves then return failure 2 Compute T h by Equation (S8) // Root: independent space-time plans 3 for agent i = 1 to M do 4 π i ← SpaceTimeAStar(s i ,g i ,T h ,∅); if π i does not exist then return failure 5 R← (π i ,C i =∅); push R keyed by (#conflicts, cost); h ⋆ ← R; n exp ← 0 6 while open list nonempty do 7Pop best node h; if h has fewer conflicts than h ⋆ , or as many at lower cost then h ⋆ ← h 8 n exp ← n exp + 1; if n exp > B cbs then break 9 c← earliest conflict in h: vertex or proximity (< δ) at t≥ 1, then swap or segment (< δ) on [t,t + 1] 10if c = ∅ then return routes of h, which are conflict-free 11for agent i involved in c do 12h ′ ← h; add toC i the vertex constraint (t,π i (t)) or the edge constraint (t,π i (t),π i (t + 1)) 13Replan with the updated constraints; if feasible then count conflicts and cost of h ′ ; update h ⋆ ; push h ′ Output: Routes of h ⋆ with trailing waits trimmed, conflict-free if the search closed TABLE S3 States of the low-level controller. Transitions follow the listed conditions. StateActionNext state SELECTPop the next route node of the macro-step; rotate toward it; set⌊d/0.25⌋ forward stepsTRACK, or DONE if no node remains TRACKExecute one forward step of 0.25 m per synchronized stepSELECT when steps are exhausted; RECOVERINIT upon collision RECOVERINITRefund the collided step; turn 90 ◦ to a random side; load the six probe headingsSWEEP SWEEPRotate to the next probe heading and attempt one forward stepPROBE, or SELECT if no heading remains PROBEIf displaced, rotate back to the blocked headingRETRY if displaced, otherwise SWEEP RETRYExecute the remaining forward stepsSELECT when exhausted or upon collision DONEIdle until all agents finish the macro-step— may temporarily vacate its goal to let a teammate pass and return before the horizon. Paths shorter than T h are padded by waiting at the goal. The high-level search (Algorithm S4) scans the synchronized timeline for the earliest conflict. At every step t ≥ 1 it checks vertex conflicts, where two agents occupy the same node, and proximity conflicts, where their nodes are closer than the collision distance δ. For every transition from t to t + 1 it checks swap conflicts and segment conflicts, where the minimum distance between the two agents over eleven synchronized interpolation points is below δ; conflicts at t = 0 between agents that localized to the same node are ignored, since several agents may legitimately depart from one abstract node. A vertex or proximity conflict at time t adds the vertex constraint (t, own node) to each of the two agents in two child nodes, and a segment or swap conflict adds the edge constraint (t, own move). Children whose replanning succeeds are pushed into an open list ordered by the number of conflicts and then by total metric cost. The search returns the first conflict-free node; after B cbs expansions, it returns the incumbent with the fewest conflicts and, among those, the lowest cost, trading completeness for real-time operation. Returned paths are trimmed of trailing waits; all intermediate nodes are explored, and the goal is addressed by its measured position if explored and by its mean estimate otherwise. For agents moving to an unexplored target, the last node before the target on the returned route replaces the planned front node when recording the attempt. Failure-verified exclusion. When an agent is dispatched toward an unexplored target, the memory records the attempt as the pair of the last explored node u on its route and the target position y (Algorithm S3). At the next localization, if the agent’s measured position is identified with an existing explored node instead of instantiating a new one, the navigation did not reach new ground, and (u,y) is appended to the collective ban list B; otherwise the attempt is cleared. Before the proposals Y t generated at v t enter the memory, every proposal lying within the ban radius of a failed target recorded from v t is excluded, Y t ← y i ∈ Y t : ∥y− y i ∥≥ ε ban , ∀ (v t ,y)∈ B ,(S9) with ε ban = 0.5 m. An excluded proposal is discarded and creates no node. Because B is shared by the team, the exclusion is collective: a frontier that one agent proved unreachable from u is not re-proposed at u by any teammate that later localizes to the same node. Low-level controller. Routes are executed in synchronized macro-steps: in each macro-step, every agent that still has a route node tracks exactly that node, agents without one idle, and the next macro-step starts only after all agents have 14 Algorithm S5: Team-Correlated Episode Sorting Input : Episode listE = (e 1 ,...,e |E| ) with scene, start node and reference path; team size N 1 ShuffleE ; i← 1 2 while i≤|E| do 3 R←j > i : scene(e j ) = scene(e i ) // same scene 4 C ←j ∈R : start(e j ) /∈start(e k ) i k=max(i−N,1) // distinct start 5 F ←j ∈C : path(e j )∩ path(e i )̸=∅ // overlapping path 6ifF ̸=∅ then 7swap e i+1 with e j , j ∼U (F ) 8else ifC ̸=∅ then 9swap e i+1 with e j , j ∼U (C) 10else ifR̸=∅ then 11if fewer than 100 relocation attempts were made at position i then 12C ′ ← positions j < i in the contiguous same-scene block preceding e i whose episodes at positions j− N,...,j + N − 1 do not share the start node of e i 13F ′ ←j ∈C ′ : path(e j )∩ path(e i )̸=∅ or path(e j−1 )∩ path(e i )̸=∅ 14ifF ′ ∪C ′ ̸=∅ then 15move e i to position j ∼U (F ′ ), or j ∼U (C ′ ) ifF ′ =∅; i← i− 1 16else swap e i+1 with e j , j ∼U (R) 17else swap e i+1 with e j , j ∼U (R) 18 i← i + 1 19 SplitE into streamsS 1 ,...,S N with e i ∈S (i mod N)+1 Output: StreamsS n , consumed one episode per stream per iteration with one memory per scene finished, which preserves the shared timeline. To track a node, the controller of Table S3 rotates the agent toward it in 15 ◦ increments and issues ⌊d/0.25⌋ forward commands of 0.25 m, where d is the horizontal distance to the node. If a forward command collides, the consumed step is refunded and a recovery sweep starts: the agent turns 90 ◦ to a uniformly chosen side and probes the headings 90 ◦ , 60 ◦ , 30 ◦ ,−30 ◦ ,−60 ◦ ,−90 ◦ relative to the blocked heading, sweeping from the chosen side to the opposite one, by attempting one forward step at each. At the first heading that yields a displacement, it turns back to the blocked heading and executes the remaining forward steps; a further collision abandons the node. If no probe yields a displacement, the node is abandoned as well. In both cases the controller proceeds to the next route node without reporting the abandoned one. The controller reports as verified anchor the last explored route node that it completed without abandonment, and the departure node if no route node was completed. These anchors are the only nodes to which a newly created node is connected (Algorithm S3), which realizes traversal-verified connection: an edge is added only for a traversal that the controller completed, so detours and recoveries cannot introduce spurious shortcuts. Conversely, a dispatch that ends at an already known position is caught by failure-verified exclusion, so control failures are converted into persistent, team-visible map knowledge. E.6 Student-Forced Multi-Agent Learning Rollout and supervision. During imitation learning, each simulator instance executes a single-agent episode (Section A.4), and instances in the same scene share one topological memory, in which the agents are indexed by their order within the batch. The loss of the main paper is normalized by the number n act of executed decisions in the iteration. Actions are drawn from the categorical distribution of the fused logits; with probability β the expert action is executed instead, decaying with the training iteration. Our student-forced recipe uses β 0 = 0, and the teacher-forcing ablation uses β 0 = 0.5. Stop and backtracking moves are executed by teleportation to explored nodes, as in ETPNav, whereas moves toward unexplored nodes are executed physically from the front node with the recovery sweep of Section E.5. The localization phase, failure-verified exclusion, candidate restriction and the consumption of selected unexplored nodes are identical to inference. Memory persistence. A per-scene counter controls how long a memory survives: a new memory is created for a scene whenever the counter reaches the number K traj of accumulation rounds, and is otherwise reused after clearing the visitation records and detaching the stored node features from the computation graph. Our final recipe uses K traj = 1, i.e., every iteration starts from an empty memory that is then built jointly by the concurrent agents, while the accumulation ablation uses K traj = 2. Team-correlated episode sorting. Sharing memory during training is useful only if concurrently simulated agents operate in the same scene, start from distinct poses, and traverse overlapping space. Algorithm S5 sorts the training stream accordingly. Starting from a shuffled list, the algorithm fills the position following each episode e i with a same-scene episode whose start node differs from the start nodes of the episodes at positions i−N,...,i, preferring candidates whose reference path shares a node with that of e i . If every same-scene candidate conflicts on the start node, the algorithm 15 TABLE S4 Model assignment of the crafting pipeline. StageStepModelReasoning 1 Viewpoint annotationGemini-3.1-flash-litehigh Zone annotationGemini-3-flashmedium 2 Profile selectionGemini-3.1-flash-litedefault Mission synthesisGemini-3.1-prolow 3 Mission instantiationGemini-3-flashlow Atomic instruction generationGemini-3-flashlow 4 Perception-verified refinementGemini-3.1-flash-litehigh Instruction renderingGemini-3.1-prolow Arrival-sequence reconstructionGemini-3-flashmedium attempts, up to 100 times per position, to move e i backward into the contiguous same-scene block, to a position whose neighborhood of N episodes on either side contains no start conflict, preferring positions adjacent to an episode with an overlapping path, and then revisits the vacated position; otherwise it accepts a same-scene episode despite the conflict. The sorted list is split into N streams in round-robin order, and each training iteration advances every stream by one episode, so that the N concurrently simulated agents are consecutive episodes of the sorted list. The list is reshuffled and re-sorted whenever a stream is exhausted. F VERIFIED TASK CRAFTING This section unfolds the four-stage crafting pipeline, including the zone-repair and schedule-refinement algorithms, the well-formedness rules, and the design rules governing every synthesis prompt. We leverage different VLMs and LLMs of the Gemini-3 family across the stages, matching model capacity and reasoning effort to the difficulty of each step (Table S4): Gemini-3.1-flash-lite handles viewpoint-level annotation, profile selection, and perception-verified refinement of atomic instructions; Gemini-3-flash handles zone-level annotation, mission instantiation, atomic instruction generation, and arrival-sequence reconstruction; and Gemini-3.1-pro handles mission synthesis and instruction rendering. F.1 Stage 1: Grounded Scene Understanding Scene parsing and rendering. Viewpoints are taken from the navigation graphs of HM3D scenes [7], [8]. For each scene, we parse the semantic annotations into object instances, discarding instances whose category is empty or contains any of the non-target terms wall, frame, floor, sheet, void, stairs, unknown, ceiling, window, curtain, beam and decoration. At every viewpoint, we render twelve RGB and pixel-aligned semantic views at 30 ◦ heading increments with a 90 ◦ vertical field of view at 512× 512 resolution, and certify which instances are observable in each view by the protocol below. In the same pass, we compute the navigable anchors of every certified instance together with their geodesic distances from the viewpoint. Observability certification. An instance–view pair (o,i) at viewpoint u is certified observable if and only if the following tests hold. (i) Proximity: the distance from u to the oriented bounding box of o is below θ obs = 1.5 m, which admits objects in the immediate surroundings of the viewpoint that a subtask can meaningfully target. (i) Semantic visibility: the semantic identifier of o occurs in the pixel-aligned semantic rendering of view i, which rejects nearby objects hidden behind walls or outside the view. (i) Mask sufficiency: the bounding box of the visible mask of o in view i covers at least a fraction θ area = 10 −4 of the frame, which rejects empty or degenerate masks at frame borders. (iv) Reachability: the anchor set is non-empty, which rejects objects none of whose sides can be reached from the viewpoint. The categories of instances passing tests (i) and (i) form the instance lists given to the viewpoint annotator, while the grounded instances used for synthesis must pass all four tests, be confirmed by the annotator, and have anchors within θ gnd = 3 m geodesic distance of the viewpoint. The anchors of a released target are those computed at a certifying viewpoint, and they are stored with the episode as the instance locations consumed by Equation (S1). Hence every grounded instance in MAVLN is visible from the viewpoint that grounds it, describable by the annotator, and reachable by a navigating agent, and generation, training supervision and evaluation share one consistent notion of reaching a target. Viewpoint annotation. A VLM annotates each viewpoint from six directional views at 60 ◦ spacing (forward, front-right, back-right, backward, back-left and front-left), each paired with an instance list of the categories of instances that pass the proximity and semantic-visibility tests in that view. For each view, the VLM describes the layout and spatial relationships and lists all observable objects with a description and a function. Objects from the potential list must be named exactly, with a numeric suffix, and only if actually visible, since the list is treated as noisy candidates rather than ground truth; all other objects also receive numeric suffixes. Suffixes must reflect physical identity across views: the same physical object seen in two directions receives one suffix, whereas distinct look-alikes, objects co-occurring in a view, and objects in different views without evidence of identity receive distinct suffixes. The VLM then writes a summary of the viewpoint centered on the agent’s position, emphasizing navigation-relevant structure and estimated distances, and a list of consolidated objects 16 Algorithm S6: Zone-Connectivity Repair Input : Initial zonesZ j from region annotations; navigation graph G nav ; round cap 20 1 for every empty zone do move the least-connected viewpoint of the largest zone into it 2 for round = 1 to 20 do 3for every zone Z j with more than one connected component on G nav do 4C 0 ← largest component of Z j 5for every other component C of Z j do // Phase 1: bridge 6B ←u∈ Z j ′ , j ′ ̸= j : u adjacent to C 0 and to C, |Z j ′ | > 1, Z j ′ \u connected 7ifB ̸=∅ then move some u∈B into Z j ; C 0 ← C 0 ∪u∪ C ; continue // Phase 2: safe relocation 8for u∈ C do 9j ∗ ← argmax j ′ ̸=j |w ∈ Z j ′ : (u,w)∈ G nav | subject to Z j ′ ∪u connected and a positive score 10if j ∗ exists then move u to Z j ∗ // Phase 3: forced relocation 11for u∈ C still in Z j do 12j ∗ ← argmax j ′ ̸=j |w ∈ Z j ′ : (u,w)∈ G nav | 13if the score of j ∗ is positive then move u to Z j ∗ 14else leave u in Z j and log a warning 15if no viewpoint moved in this round then break Output: Repaired zonesZ j and the list of unresolved zones that merges all per-view entries by category without suffixes. A response is accepted only if every consolidated category appears among the per-view object names. Grounded instances. An instance is grounded at a viewpoint in a given direction if it passes all certification tests, its category matches the name of an object listed by the VLM in that direction, its category appears among the consolidated objects, and its anchors are within θ gnd = 3 m geodesic distance of the viewpoint. The grounded instances of a viewpoint are the union over its six directions, and their category descriptions are taken from the consolidated list. This rule makes every textual reference traceable to pixels: a subtask can only name a category that the VLM verifiably described in a specific view of a specific viewpoint and that corresponds to an observable, reachable instance there. Zone construction and repair. Zones are initialized from the region annotations of HM3D. Region names are obtained from per-scene weighted votes over region labels, and regions labeled as unknown rooms are discarded. Each viewpoint is assigned to the region whose floor object has the smallest oriented-bounding-box distance to it, and empty regions are dropped. These raw assignments are unreliable planning units, because region boundaries frequently cut through navigable space. We therefore repair them by Algorithm S6, which iterates for at most 20 rounds. In each round, every disconnected zone keeps its largest connected component on the navigation graph and processes each remaining fragment in three phases. The bridge phase borrows a single viewpoint from another zone that is adjacent to both the main component and the fragment, provided that its removal does not disconnect its source zone and does not empty it. If no bridge exists, the relocation phase moves each fragment viewpoint to the adjacent zone with the largest number of navigation-graph edges to it, provided that the receiving zone remains connected. Remaining viewpoints are force-relocated to the best-connected adjacent zone regardless of the receiving zone’s connectivity, and viewpoints without any adjacent zone are left in place with a warning. The iteration stops when a round changes nothing. For mission synthesis, each zone is finally restricted to its largest connected component on the navigation graph, and two zones are adjacent if any navigation-graph edge joins them. Zone annotation. An LLM aggregates the viewpoints of each zone, i.e., their summaries, grounded-instance descriptions and positions, into a zone description. The prompt supplies the voted region name as a reference, requires the zone to be characterized by its primary single room type while discounting objects of adjacent rooms visible near boundaries, and requires a description of every grounded instance of the zone under its exact name, merging observations from different viewpoints. A response is accepted only if the described instances coincide exactly with the zone’s grounded instances. This stage yields 20,910 annotated viewpoints aggregated into 1,628 zone descriptions. F.2 Stage 2: Diverse Mission Synthesis Household profiles. Mission narratives are conditioned on one of 20 structured household profiles, such as a family with young children, a multi-generational home, professional roommates, a household with disability accommodations, home- based healthcare, a household with rotating shift workers, or a household with special dietary needs. Each profile specifies a household type, its members with role, age, occupation, mobility, schedule and task preferences, and, where relevant, medical, accessibility, dietary or special needs, together with the household’s coordination style, common activities and typical challenges. For each scene, a profile-selection pass receives the scene description and all profiles, and returns 17 TABLE S5 Well-formedness rules enforced before and after instantiation. RuleCondition for rejection Unreleased lockA locked subtask has a later subtask of the same agent depending on it, and no subtask releases it Release by another agentA released subtask is not a transitive dependency of the releasing subtask, or is found among its dependencies with the same drafted agent Single consumerA holding subtask has more than one dependent subtask of the same agent Single held objectThe consumer of a holding subtask also depends on another holding subtask Distinct consecutive goalsA subtask shares its goal viewpoint with a same-agent dependency (after instantiation) the indices of the profiles whose needs are best accommodated by the layout, considering zone functions and contained instances, zone connectivity, mobility requirements, schedule compatibility, special needs, age-appropriate spaces and supported activities, and prioritizing profiles in which several members’ needs are met simultaneously. Profiles are the mechanism through which constraints arise organically: restricted mobility motivates fetch chains, overlapping schedules motivate parallelism, and continuous-care needs are the canonical source of presence locks. Mission budget and team size. A budget of 14,000 missions is distributed over scenes in proportion to the number of viewpoints in their zones that contain at least one grounded instance, and within each scene over zones in proportion to their viewpoints by largest-remainder rounding, zones whose share is below one mission receiving none. The admissible team size of a mission centered on a zone depends on the zone’s number of viewpoints: at most two agents below 47 viewpoints, at most three below 70, and at most four otherwise. The team size of each mission is drawn uniformly between two and this maximum, and the selected profiles are cycled over the missions of a zone in a shuffled order. Mission drafting. Conditioned on a profile, the center zone (its number of viewpoints, summary and grounded-instance descriptions), the adjacent zones with grounded instances, the textual zone connectivity and the team size, the LLM drafts a task demand and a list of subtasks, each with an identifier, a drafted agent, a zone, a navigation instruction, an exact target name, the dependency list, the lock and holding flags, the release list, and a natural-language description justifying every non-default attribute. A draft is accepted only if it contains a task demand, every target is a grounded instance of its zone, every zone is the center zone or one of its adjacent zones, at least one target lies in the center zone, subtask identifiers are unique, and every dependency refers to a previously listed subtask. Well-formedness. Before instantiation, every release list is closed under dependencies, i.e., each subtask in U k is added to D k , and the draft must pass the static checker of Table S5. The checker operates on the DAG in which an edge points from each dependency to its dependent subtask, and it considers the drafted agents. F.3 Stage 3: Mission Refinement Instance grounding. For every subtask, the candidate goal viewpoints are the viewpoints of its zone at which its target is grounded. For every drafted agent, candidate viewpoint sequences over its subtasks are composed such that consecutive goals differ and their navigation-graph distance lies in (3, 20) m; a draft is discarded if any agent has no valid sequence. The grounding mode depends on the size of the joint combination space. If the product of the per-agent sequence counts is below 100, every joint combination is instantiated, scheduled and ranked by Algorithm S7, rewritten by its best schedule, validated against the leg bounds below, and retained only if distinct start viewpoints can be allocated; the LLM then selects one of the retained complete combinations, preferring the one whose viewpoints best support the subtasks’ purposes and spatial context. Otherwise, agents with fewer than 100 sequences are offered their complete sequences, while the subtasks of the remaining agents are offered their candidate viewpoints individually; the LLM must then select exactly one sequence for each constrained agent, assign one candidate viewpoint to each remaining subtask without reusing a viewpoint for consecutive subtasks of the same agent, and prioritize coverage and reasonable distances over contextual fit when concurrent subtasks have few options. In both modes, the LLM also rewrites every subtask instruction with visual details of the assigned viewpoint and a reference to the room, and its output is accepted only if subtask identifiers and drafted agents are preserved, each viewpoint belongs to the subtask’s zone and grounds its target, and the instantiated draft passes Table S5. Atomic instruction generation. For every subtask, an LLM receives the grounded instruction and the directional views of the goal viewpoint in which the target is grounded, each with its layout description and target annotations, selects the view whose annotation is most distinctive and spatially precise, and rewrites the instruction into a destination-only atomic instruction. The candidate instances of the subtask are the grounded instances of the target category in the selected direction. Makespan-optimal schedule refinement. The instantiated mission is then scheduled by Algorithm S7. A schedule is a sequence of single agent–subtask commitments. At every state, the admissible commitments are determined by the task semantics: a locked agent receives nothing until a teammate completes a subtask that releases it; a holding agent may only commit to the consumer of its held subtask with the same drafted executor, and this parent is reserved for it; and every other agent may commit to any ready subtask, i.e., an incomplete subtask whose dependencies are complete, regardless of the drafted allocation. Each commitment completes the subtask, records its executor, releases the locks listed in its release list, and appends to the executor’s itinerary the navigation-graph shortest path from its current viewpoint to the goal. Every complete commitment sequence is then replayed on a synchronized clock. At commitment c, the clock advances one 18 Algorithm S7: Makespan-Optimal Schedule Refinement Input : TInstantiated subtask DAG ADrafted agents s 1:M Initial viewpoints SP(·,·) Graph shortest paths C max Enumeration cap 1 S ←∅ 2 Function Traverse(state): 3 Z ← admissible single commitments (a,τ ) at state 4for (a,τ )∈Z do 5state ′ ← copy of state ; complete τ with executor a; release every u∈ U τ held by a teammate 6append SP(pos(a), vp(τ )) to the itinerary of a and empty legs to all other agents; pos(a)← vp(τ ) 7if|S|≤ C max then Traverse(state ′ ) 8if no recursion took place and every subtask is complete in state thenS ←S∪state 9 Traverse(initial state at s 1:M ) 10 for candidateP ∈S with commitments c = 1,...,N do 11 m← 0; all cursors at c = 1 12for c = 1 to N do 13advance lagging cursors to c; refresh the lock flag of the agent committing at c 14while the leg of commitment c is not exhausted do 15for agent a∈A do 16if a has an exhausted current leg and a is unlocked then move the cursor of a to its next non-empty leg 17if the current leg of a is non-empty then advance a by one node 18m← m + 1 19key(P)← m, #deviations(P), − P depth(deviations(P)) Output:P ⋆ ← argmin P∈S key(P); allocation rewritten byP ⋆ node per step until the committing agent completes its leg, while every other agent whose current leg is exhausted and which is not locked pulls its next leg forward and advances in lockstep; lock flags are refreshed from the state recorded at each commitment, which reflects the releases performed so far. The global makespan is the number of steps of the replay. Candidates are ranked lexicographically by makespan, by the number of subtasks whose executor deviates from the drafted agent, and by the negated sum of the DAG depths of the deviating subtasks, and the top-ranked schedule is adopted: when it reassigns a subtask, the episode is rewritten accordingly, so that the released allocation is makespan-justified rather than LLM-asserted. Continuous-space validation and start allocation. After rewriting, every consecutive leg of every agent must have a navigation-graph distance in [3, 20] m and a geodesic distance of at most 20 m in the continuous simulator, since graph distances systematically underestimate continuous detours around furniture and through doorways. Missions failing these checks are discarded rather than repaired. Start viewpoints are then allocated by enumerating the combinations of candidate instances over all subtasks in random order. For each combination, a viewpoint is eligible as a starting location for an agent if it is not the first goal viewpoint of any agent, its navigation distance to the agent’s first goal viewpoint is between 4 and 15 m, the shortest path contains at least four viewpoints, and its distance to the oriented bounding box of the agent’s first target instance is between 3 and 15 m. Distinct starting viewpoints are then selected through a minimum-cost assignment with independent, uniformly sampled random costs for eligible agent–viewpoint pairs, randomizing the selection while ensuring that no two agents share the same start. The combination is rejected if some agent has no eligible start. Each agent’s trajectory is then rendered along the graph shortest paths through its goals: at every node, the agent faces the next node, the view index is the corresponding 30 ◦ heading bin, and at the goal the view of the selected direction is appended. A leg is rejected if its starting position lies within 3 m geodesic distance of any anchor of its target, so that every subtask requires navigation to be credited. The first combination passing all checks is released, together with the start positions and the start orientations facing the next node on each agent’s first path. F.4 Stage 4: Verified Instruction Rendering Perception-verified refinement. For every subtask, the candidate instances at its goal viewpoint are painted with distinct colors on darkened copies of the six even-indexed views, and only the views in which some candidate is visible are kept; these composites serve for auditing. A VLM receives the trajectory frames labeled by navigation step, the clean goal views in which a candidate is visible, and a map listing each candidate’s name, color and bounding boxes (restricted to boxes covering at least a fraction θ area of the frame) in each goal view. It must first select the candidate that best matches the instruction by reasoning about its position, size, appearance and surroundings, and then rewrite the target description from what is actually visible for the selected instance, overriding any wrong or hallucinated attributes, specifying which part of the object the agent arrives at, and grounding it by its immediate neighbors at the arrival position. The output is 19 accepted only if the selected name is one of the candidates, which fixes the target instance g k of the subtask; subtasks whose candidates have no valid bounding box, or whose refinement fails, invalidate the mission. This step converts generation- time language into perception-verified language: the released atomic instruction describes the instance as a navigating agent will observe it. Variant rendering. Each mission is rendered by one call producing the decentralized and centralized instructions and by a separate call producing the centralized-implicit instruction. Both calls receive the task demand, the agent identifiers, the per-subtask information, viewpoint and zone descriptions, pairwise navigation-graph distances between goal viewpoints in steps, and the zone affiliation of each goal. The decentralized and centralized call additionally receives the drafted and final executor of each subtask and the subtask descriptions, and must organize the instructions according to the final executors whenever schedule refinement reassigned a subtask. The centralized-implicit call receives no executor information, so its single instruction treats all subtasks as one coordinated plan without attribution. In the released episodes, a decentralized subtask permits only its final executor, whereas centralized and centralized-implicit subtasks permit every agent of the team. Closed-loop filtering. Every rendered instruction set faces a round-trip filter. An LLM receives only the instruction(s), the agent identifiers and the subtasks presented as an unordered list with their permitted agents, atomic instructions and goal viewpoints, and must return an interleaved sequence of (agent, viewpoint) arrivals in which each agent’s destinations follow the order implied by its instruction and the interleaving respects the stated waiting and coordination conditions. The sequence is replayed by a generation-time version of the evaluator in which an arrival consumes the first ready subtask of the agent whose goal viewpoint matches, lock and holding violations fail the involved subtasks as in Algorithm S1, and the mission is valid if and only if every subtask succeeds. Instructions failing the filter are regenerated and re-filtered, and a mission is released only if its instructions are valid in all three regimes, so that the three regimes share the same missions. The filter certifies the property that evaluation relies upon: the language alone, under the task’s discrete-event semantics, determines a successful execution. Scene-disjoint splits. Scenes are grouped by the largest team size among their valid missions, and scenes hosting at least one four-agent mission are sorted by their number of missions. All scenes without four-agent missions and six of the four- agent scenes form the training split of 128 scenes, while seven four-agent scenes form val-unseen and ten form test-unseen. Interleaving the ranks balances scenes with many and few missions across splits, and every unseen scene hosts four-agent missions. Finally, we apply human assessment to the val-unseen and test-unseen split to remove erroneous missions and therefore ensure the dataset quality. 20 TABLE S6 Consolidated hyperparameters. SymbolMeaningValue Generation θ obs Oriented-bounding-box proximity for observability1.5 m θ area Minimum mask bounding-box area ratio10 −4 —Anchor deduplication radius0.5 m θ gnd Maximum geodesic distance from a viewpoint to the anchors of a grounded instance3 m —Mission synthesis budget; zone-size thresholds for team sizes≤ 2/≤ 3/≤ 414,000; < 47 / < 70 /≥ 70 viewpoints —Consecutive-goal band on the navigation graph; continuous-space leg bound(3, 20) m;≤ 20 m —Start-to-first-goal band; minimum start path length(4, 15) m; 4 nodes —Start-to-first-target bounding-box band; minimum start-to-anchor geodesic(3, 15) m; 3 m C max Schedule enumeration cap10 5 —Joint combinations below which grounding is exhaustive10 2 —Zone repair rounds20 Evaluation —Success radius to navigable anchors3 m ρAgent radius for MAC0.25 m Scheduling —Rounds after which an all-blank round always terminates20 Shared topology and planning KMaximum waypoint proposals per step5 ε loc Node identification radius0.5 m ε nbr Round-start stitching radius1.5 m ε ban Ban radius0.5 m T max Planning step limit per round30 Conflict-aware execution δCollision distance in path finding0.4 m T h Space-time horizonat least 5 B cbs Expansion budget10 3 —Segment interpolation points11 —Forward quantum; turn quantum; recovery probes0.25 m; 15 ◦ ;±30 ◦ ,±60 ◦ ,±90 ◦ Learning —Pretraining steps (selected checkpoint)200,000 (180,000) —Optimizer; learning rateAdamW; 10 −5 —Imitation-learning iterations; checkpoint interval30,000; 1,000 MConcurrent agents sharing memory during training4 β 0 Initial teacher-forcing ratio (decay interval)0 (5,000 iterations) K traj Memory accumulation rounds1 —Expert stop radius1.5 m —Maximum instruction length512 tokens 21 REFERENCES [1] X. Puig et al., “Habitat 3.0: A co-habitat for humans, avatars, and robots,” in ICLR, 2024, p. 1–15. [2] S. K. Ramakrishnan et al., “Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,” in NeurIPS D&B, 2021. [3] D. An et al., “Etpnav: Evolving topological planning for vision-language navigation in continuous environments,” IEEE TPAMI, vol. 47, no. 7, p. 5130–5145, 2025. [4] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee, “Beyond the nav-graph: Vision-and-language navigation in continuous environ- ments,” in ECCV, 2020, p. 104–120. [5] Y. Hong, Z. Wang, Q. Wu, and S. Gould, “Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation,” in CVPR, 2022, p. 15 439–15 449. [6] S. Chen, P.-L. Guhur, M. Tapaswi, C. Schmid, and I. Laptev, “Think global, act local: Dual-scale graph transformer for vision-and-language navigation,” in CVPR, 2022, p. 16 537–16 547. [7] Z. Wang, Y. Zhu, G. H. Lee, and Y. Fan, “Navrag: Generating user demand instructions for embodied navigation through retrieval-augmented llm,” in ACL Findings, 2025, p. 8430–8440. [8] Z. Wang et al., “Scaling data generation in vision-and-language navigation,” in ICCV, 2023, p. 12 009–12 020. 22 You are a household assistant robot that analyzes indoor scenes. Your task is to observe the environment from the specific position, generate structured annotations for each view, and consolidate multiple observations into a comprehensive summary and organized object list. ### Input Format You will receive images captured from six directional views ... ### Instructions 1. For each view, describe the room layout and spatial relationships in the "detail" field. 2. For each view, list all observable objects in the "objects" field. Pay special attention to objects in `potential_instances`. If the image is obscured, use an empty array `[]`. 3. Create a comprehensive "summary" that synthesizes viewpoint-specific information from all directional views. 4. Create a "consolidated_objects" list in the summary. This list merges all objects observed across all per-view annotations by category: objects of the same type (e.g., `book_1`, `book_2` from various views) are merged into a single entry describing all instances. The list must align with the objects actually listedin the per-view annotations. ... ### Output Format ```json "forward": "detail": ..., "objects": ..., ... "summary": "detail": ..., "consolidated_objects": ... ``` ### Input Potential Instances (within 1.5 meters): ```json "forward": forward_potential_objects, ... ``` Output: (a) Viewpoint annotation. You are a household assistant robot tasked with creating comprehensive zone descriptions from multiple viewpoint summaries. ### Input Format: You'l receive: 1. Multiple viewpoint entries, each containing: -"summary": Description of the viewpoint -"grounded_instances_description": List of objects with detailed descriptions -"position": [x, y, z], where x denotes left/right, y denotes vertical height, and z denotes forward/backward (negative z is forward) 2. A master list of "grounded_instances" that should be included in your output ### Output Format: ```json "summary": ..., "grounded_instances": ... ``` ### Guidelines: -IMPORTANT: Identify the PRIMARY SINGLE ROOM TYPE based on key functional features. Some viewpoints near zone boundaries may not fully capture the main layout of a specific zone, as objects from adjacent rooms may be visible. When categorizing zones, focus on the main function of each space and apply common knowledge of residential layouts. -Begin with identifying the zone type ("This zone is a...") ... Input: The possible name (for reference) for this zone is: zone_name The Python dictionary of viewpoints within the zone: zone_dict The grounded instances in this zone are: grounded_instances Output: (b) Zone annotation. Multi-agent cooperation is essential for navigating complex household environments. Your task is to consider the provided house layout and select the most suitable user profiles for this specific home environment. The house contains multiple connected zones with distinct functions and features. Each zone contains specific objects and serves particular purposes(bedroom, bathroom, storage, etc.). User profiles represent diverse household compositions with varying mobility needs, schedules, and coordination styles. Selection criteria: 1. Analyze each zone's features and contained instances 2. Consider zone connectivity patterns (which rooms connect to others) 3. Consider how each user profile's needs align with the house layout: -Mobility requirements (e.g., elderly using walkers need accessible layouts) -Schedule compatibility (e.g., homes with shared bathrooms need households with compatible schedules) -Special needs match (e.g., homes with study areas for children requiring homework assistance) -Age-appropriate spaces (e.g., bunk beds for households with children) -Activity support (e.g., gym equipment for fitness-oriented households) 4. Prioritize profiles where multiple household members' needs are simultaneously accommodated Output ONLY a valid JSON object containing a list with at least one indice(begin from 0) of suitable user profiles, with NO text outside the JSON structure ### Output Format: ```json [ profile_indice, ... ] ``` Input: House description: house_prompt User profiles: user_profiles Output: (c) Household-profile selection. You are designing collaborative navigation tasks for household assistant robots. Your house is divided into several zones, with a part of zones' connectivity and environmental description provided. Each zone description contains a zone summary and description of various types of grounded instance within the zone. You should focus on letting robots collaborate primarily in the center zone based on your household needs. ## Task Creation Requirements: 1. NAVIGATION-FOCUSED COLLABORATION: ... 2. MEANINGFUL DEPENDENCIES: ... 3. TEMPORAL CONSTRAINTS: ... 4. INFORMATION SHARING: ... 5. SUBTASK ATTRIBUTES: Define key attributes for each subtask: -**depends_on**: ... -**locked**: ... -**unlock**: ... -**holding**: ... 6. ZONE-APPROPRIATE COORDINATION: ... 7. CLEAR SPECIFICATIONS: ... ### Input Format: ... ### Output Format: ```json "task_demand": ..., "subtasks": [ "id": <subtask_id>, "agent": <agent_number>, "zone": <zone_id>, "instruction": "Detailed instruction for this specific subtask", "target": "<exact_name_from_grounded_instances>", "depends_on": [<list_of_subtask_ids_that_must_complete_before_this_one>], "locked": <boolean>, "unlock": [<list_of_subtask_ids_being_unlocked>], "holding": <boolean> ... , ... ] ``` Input: Your profile information: profile -Center zone details: center_zone_info -Neighboring zones: neighbour_zones_info -Zone connectivity: zone_connectivity The task should involve agent_num agents. Output: (d) Mission drafting. Fig. S7. Prompt templates of grounded scene understanding (Section F.1) and diverse mission synthesis (Section F.2). (a) Per-viewpoint annotation from six directional views, with the potential-instance list of every view. (b) Aggregation of the annotated viewpoints of a zone into a zone description. (c) Selection of the household profiles that a scene can accommodate. (d) Drafting of the task demand and of the attributed subtask graph for one center zone, one profile and one team size. 23 You are assigning viewpoints to subtasks in a multi-agent indoor navigation system and refining each subtask's instruction with grounded visual details. ## Your Two Jobs: 1. Select one complete viewpoint allocation from the provided list of valid allocations 2. Refine each subtask's instruction to include specific visual details from the assigned viewpoint, preserving the original goal and purpose exactly -Spatial cues -Any distinguishing features of the target instance's immediate environment ### Requirements: 1. Strict Allocation Constraint ... 2. Exact Instance Match ... 3. Contextual Fit ... 4. Zone Clarification ... 5. Complete Coverage ... 6. Never use technical terms: ... ... ### Output Format: ```json [ "id": <subtask_id>, "agent": <agent_number>, "viewpoint": "<viewpoint_id>", "instruction": ..., "target": "<exact_name_from_subtask>" , ... ] ``` Input: The crafted collaborative navigation task: task_info The viewpoint candidates: viewpoint_candidates Valid subtask-to-viewpoint allocations: combination_candidates Output: (a) Instance grounding, sequence mode. You are assigning viewpoints to subtasks in a multi-agent indoor navigation system and refining each subtask's instruction with grounded visual details. ## Your Two Jobs: 1. Assign exactly one viewpoint to each subtask, following the constraints below 2. Refine each subtask's instruction to include specific visual details from the assigned viewpoint and preserve the original goal and purpose exactly -Spatial cues -Any distinguishing features of the target instance's immediate environment ### Requirements: 1. Two-Tier Viewpoint Assignment ... 2. No Consecutive Viewpoint Reuse (Same Agent) ... 3. Exact Instance Match ... 4. Contextual Fit ... 5. Zone Clarification ... 6. Complete Coverage ... 7. Never use technical terms: ... ... ### Output Format: ```json [ "id": <subtask_id>, "agent": <agent_number>, "viewpoint": "<viewpoint_id>”, "instruction": ..., "target": "<exact_name_from_subtask>" , ... ] ``` Input: The crafted collaborative navigation task: task_info The viewpoint candidates: viewpoint_candidates Valid per-agent allocations: agent_combination_candidates Available viewpoints per subtask: viewpoints_available Output: (b) Instance grounding, viewpoint mode. You are selecting the best view direction for a navigation instruction and rewriting it with precise visual detail. Given: 1. A navigation instruction describing a goal to reach a specific target instance, possibly followed by behavioral context 2. A list of view candidates, each containing a scene description and target annotations Select the view whose target annotation provides the most distinctive and spatially precise description of the target —prioritizing views where the target's appearance, position, and surrounding context are most clearly described. Your refined instruction must: -Describe only the navigation destination —what to navigate to, where it is, and what it looks like -Strip all behavioral context that follows the navigation goal: this includes waiting conditions, locked constraints, holding or delivery instructions, dependency clauses, and any instruction about what to do after arriving -Preserve the room information from the original instruction -Enrich the visual description with details from the selected view ... If multiple views describe the target equally well, prefer the view with richer spatial context ### Output Format: "view_direction": "<exact direction name from candidates>", "instruction": "Refined navigation instruction grounded in visual details from the selected view" Input: The task instruction is: instruction The description of views is: view_info Output: (c) Atomic instruction generation. Fig. S8. Prompt templates of mission refinement (Section F.3). (a) Instantiation when the joint combination space is enumerable, so that every agent is offered complete viewpoint sequences of already validated joint combinations. (b) Instantiation when it is not, so that the subtasks of the unconstrained agents are offered their candidate viewpoints individually. (c) Rewriting of a grounded instruction into a destination-only atomic instruction from the directional views of its goal viewpoint. 24 You are selecting a target object and enriching a navigation instruction for a household robot using visual perception data from its trajectory toward a target. Given: 1. An original navigation instruction describing a goal to reach a specific target 2. A bounding box map listing each candidate object's name and bounding box (x, y, w, h) per view direction at the destination viewpoint 3. Sequential images labeled "navigation step N" —the agent's trajectory toward the destination 4. Images labeled "goal view N" —clean views from multiple directions at the destination viewpoint ## Step 1 —Select Use the bounding box coordinates to reason about its spatial position and size in the corresponding goal views and locate each candidate. Assess each candidate's visual appearance, position, and immediate surroundings against the instruction. Select the best matching candidate and return its object_name. ## Step 2 —Enrich Correct the target description in the original instruction based on the selected instance's precise visual grounding as observed in the semantic annotations and trajectory data. Your job is to add specific details that make the destination more identifiable. Guidelines: ... ### Output Format: ```json "object": "object_name", "instruction": "Enriched instruction with visual grounding integrated" ``` Input: Task instruction: instruction Object names and bounding boxes: color_map Output: (a) Perception-verified refinement. Generate a centralized-style instruction and a decentralized-style instruction for a household collaborative navigation task based on the provided scenario and subtask details. Guidelines: ... ### Input Format: You will receive: 1. An original task demand describing the overall scenario and requirements 2. Involved agent ids of this collaborative task 3. Subtask information containing: ... 4. Viewpoint descriptions 5. Zone descriptions 6. Viewpoint pairwise distances for included viewpoints 7. Zone affiliation for each viewpoint ### Output Format: ```json "decentralized": [ "agent": <agent_id>, "instruction": "Your detailed, consolidated instruction for this agent" , ... ], "centralized": "instruction": "Your detailed, consolidated instruction here" ``` Input: original task demand: task_demand, agent ids: agent_ids, subtasks: subtask_info, viewpoint descriptions: viewpoint_info, zone descriptions: zone_info, viewpoint pairwise distances: viewpoint_pairwise_distances, zone affiliation: zone_affiliation Output: (b) Decentralized and centralized rendering. Generate a centralized-style instruction for a household collaborative navigation task based on the provided scenario and subtask details. Guidelines: -Create ONE consolidated instruction that combines all subtasks' navigation goals and requirements, treating all subtasks as a single coordinated plan regardless of which agent is assigned to each -For each subtask's navigation content, construct the instruction based on the atomic_instruction. The specific navigation goals must preserve atomic instruction details. Never compress or omit visual details on a target's first mention. The task demand and raw_instruction/ description provide scenario context only, but you should incorporate the action after arrival from them naturally into the instruction ... ### Input Format: You will receive: 1. An original task demand describing the overall scenario and requirements 2. Involved agent ids of this collaborative task 3. Subtask information containing: ... 4. Viewpoint descriptions 5. Zone descriptions 6. Viewpoint pairwise distances for included viewpoints 7. Zone affiliation for each viewpoint ### Output Format: ```json "centralized": "instruction": "Your detailed, consolidated instruction here" ``` Input: original task demand: task_demand, agent ids: agent_ids, subtasks: subtask_info, viewpoint descriptions: viewpoint_info, zone descriptions: zone_info, viewpoint pairwise distances: viewpoint_pairwise_distances, zone affiliation: zone_affiliation Output: (c) Centralized-implicit rendering. You are verifying whether a collaborative navigation instruction is clear and unambiguous. Your task: read the full instruction(s) and match each described destination to the corresponding subtask. Then determine the correct interleaved arrival sequence across all agents, respecting the dependencies and coordination constraints described in the instruction. The sequence format is [[agent_id, viewpoint_id], ...], where each entry represents one agent completing a subtask at one destination. Each agent's destinations must appear in the order implied by their instruction. The interleaving across agents must respect any waiting or coordination conditions mentioned. Return ONLY a valid JSON object: "sequence": [ [<agent_id>, "<viewpoint_id>"], ... ] Input: full instruction(s): instruction_info agent ids: agent_ids subtasks in random order (use these to match each destination in the instruction to a viewpoint): subtask_info Output: (d) Arrival-sequence reconstruction. Fig. S9. Prompt templates of verified instruction rendering (Section F.4). (a) Selection of the target instance among the colored candidates and rewriting of its description from what is visible at the arrival position. (b) Rendering of the decentralized and centralized instructions, organized by the final executors. (c) Rendering of the centralized-implicit instruction, which receives no executor information. (d) Reconstruction of the interleaved arrival sequence used by the closed-loop validity filter. 25 You are decomposing a robot navigation instruction into atomic instructions. An instruction may chain several navigation destinations together, interleaved with actions to perform at each one and with coordination or dependency language. Your job is to decompose it into a list of atomic instructions, one per distinct navigation destination. A navigation destination is the specific TARGET the robot will interact with —usually an object at a specific location within a room —together with everything that identifies it: the room it sits in, the target's appearance, where it sits within the room, what surrounds it, what marks it. This entire descriptive bundle is ONE destination. When the instruction first names a room and then describes a target within it ("Continue into the living area... Approach the large white sectional sofa..."), that is ONE destination, not two —the room sentence is framing for the target description that follows. Keep them together. Preserve ALL the target's identifying detail. Do not strip the description of what the object looks like, where it sits, or what surrounds it —those features are what the low-level navigator uses to recognize the destination on arrival. Remove only: -What to do upon arrival -Coordination and dependency clauses Try to keep the original sentences exactly as written; do not compact or paraphrase. Sound natural and conversational. ## Output Format ```json "atomic_instructions": ["<atomic instruction 1>", "<atomic instruction 2>", ...] ``` (a) Atomic instruction extraction. You are the global scheduler for a team of embodied robots collaborating on navigation tasks. Each turn you assign each robot an atomic instruction by its index in the shared atomic-instruction pool, respecting dependencies and avoiding repeated allocations. ## Inputs -"instruction(s)": ... -"atomic instructions pool": ... -"atomic instructions history": ... -"pending history": ... -"previous scheduling": ... ## Atomic Instruction Assignment Each atomic instruction is NAVIGATION ONLY: it directs a robot to a place to carry out an action there. The index covers only the navigation and completes the moment the robot arrives. The on-arrival action is NOT part of the index —it is governed by the task's dependencies (see Pending below). Choose whichever index best fits a robot's position and the dependency structure. Each index is issued at most once across the episode: once issued to any robot it is spent and never assigned again. ... ## Dependencies Before assigning anything, read each robot's instruction and identify every inter- subtask dependency. Distinguish the two patterns and tag each gate with which it is: ... ## Action Firing An on-arrival action with NO unmet dependency fires automatically the turn the robot arrives —including actions that put the robot into a LOCKED state. No pending_executionentry is created for ungated firings; pending_executionis exclusively for previously-pending actions whose gate just cleared. An on-arrival action WITH an unmet dependency does NOT fire on arrival. The robot is dispatched with its index marked in 'pending'; it arrives and waits. On the turn the gating event happens, place the robot in 'pending_execution' to fire it. ... ## Pending vs. Pending Execution -'pending' (PRE-ALLOCATION): ... -'pending_execution' (EXECUTION FIRING): ... ## Output Format ```json "thought": ..., "scheduling": ..., "atomic_instructions_indices": ..., "pending_execution": ..., "pending": ... ``` (b) Round-wise scheduling. You are the global scheduler for a team of embodied robots collaborating on navigation tasks. Each turn you assign each robot an atomic instruction by its index in the shared atomic-instruction pool, respecting dependencies and avoiding repeated allocations. ## Inputs -"instruction(s)": ... -"atomic instructions pool": ... -"atomic instructions history": ... -"pending history": ... -"current observation”: ... -"previous scheduling": ... ## Atomic Instruction Assignment Each atomic instruction is NAVIGATION ONLY: it directs a robot to a place to carry out an action there. The index covers only the navigation and completes the moment the robot arrives. The on-arrival action is NOT part of the index —it is governed by the task's dependencies (see Pending below). Choose whichever index best fits a robot's position and the dependency structure. Each index is issued at most once across the episode: once issued to any robot it is spent and never assigned again. ... ## Dependencies Before assigning anything, read each robot's instruction and identify every inter- subtask dependency. Distinguish the two patterns and tag each gate with which it is: ... ## Action Firing An on-arrival action with NO unmet dependency fires automatically the turn the robot arrives —including actions that put the robot into a LOCKED state. No pending_executionentry is created for ungated firings; pending_executionis exclusively for previously-pending actions whose gate just cleared. An on-arrival action WITH an unmet dependency does NOT fire on arrival. The robot is dispatched with its index marked in 'pending'; it arrives and waits. On the turn the gating event happens, place the robot in 'pending_execution' to fire it. ... ## Pending vs. Pending Execution -'pending' (PRE-ALLOCATION): ... -'pending_execution' (EXECUTION FIRING): ... ## Output Format ```json "thought": ..., "scheduling": ..., "atomic_instructions_indices": ..., "pending_execution": ..., "pending": ... ``` (c) Observation-guided scheduling. You are a viewpoint description expert supporting multi-agent navigation scheduling. Your task is to describe the current place. Produce a description of the viewpoint and beware which targets in the instructions may be more appropriate to schedule to the robot at this location. The description must be orientation-agnostic: an agent arriving here from any facing direction should be able to interpret it without knowing which way they are oriented. Rules: -Emphasize features that distinguish this place from nearby alternatives, such as room category cues, major furniture clusters, doors/openings, counters, appliances, seating, tables, beds, stairs, or other fixed architectural anchors. -Use the navigation instructions as background context for inferring the potential correlation between targets and this location. Mention if any target has evidence for scheduling the robot toward there. -The target of an atomic instruction may not be visible from this position. Use place recognition, commonsense, and target descriptions to infer plausible nearby target types when helpful, butmark such inferences as uncertain. -Describe observed facts from the landmark tags and images, with cautious functional cues when supported by visible evidence. -Focus on visible objects, their relationships, the likely room or functional area, and information that could help the scheduler distinguish this viewpoint from others. -Do not anchor any spatial term to the viewer or the location itself. Forbidden phrasings include: 'in front of you', 'behind you', 'on your left/right', 'to the side', 'ahead', and structuring the description by the four input views. -Horizontal terms (left, right, beside,opposite, across from, ...) ARE allowed when they describe the relationship between named objects or layout elements. -Distance-from-here terms (nearby, a few steps away, ...) for named objects and layout elements are encouraged. -Place each object relative to fixed architectural anchors and to other objects, rather than relative to the viewer. -Merge objects that appear across multiple views and belong to the same sub-space into one coherent description. -Keep the output concise. (d) Viewpoint description. Fig. S10. Prompt templates of the subtask scheduler of TRISS (Section E.2). (a) Decomposition of the task instructions into the shared pool of atomic instructions. (b) Emission of the per-agent assignment, the firing list and the pre-allocated agents of one round. (c) The same protocol extended by the situation report of Equation (S3). (d) Orientation-agnostic description of an agent’s current viewpoint, which forms that report. 26 You are one of a team of embodied robots navigating in the real world. At each step you choose where to move next, using neighboring-viewpoint images, until you reach the destination and stop. ## Inputs -'atomic instruction’: ... -'atomic instruction history’: ... -'lifelong history’: ... -'history’: ... -'previous planning’: ... -'action descriptions’: ... -'action candidates’: ... ## Reasoning each turn 1. Parse the atomic instruction into: primary target, likely room/area, and 2-3 distinctive cues. Treat the room/area as route guidance, not as the destination by itself. 2. Decide whether to stop before considering movement. Stop only when the current viewpoint clearly matches the primary target and at least one distinctive cue, with no contradictory room/type evidence. 3. If not stopping, rank movement evidence: direct target/cue visibility first; then a doorway, hallway, opening, staircase, or frontier toward the likely room/area; then object-list hints as a tiebreaker. 4. Use history to avoid redundant checks. Do not re-inspect a room/view already found without target evidence. Backtrack only to reach a new branch/frontier, return from a dead end, or recover from a wrong room/type. 5. In planning, keep a short action chain: target/cue -> chosen waypoint/action -> next confirmation. ## Output Format ```json "thought": A concise reasoning for navigation action, "planning": One short sentence: target/cue -> chosen waypoint/action -> next confirmation. Refine 'previous planning' without restating completed checks, "action": Exactly one element from 'action candidates' ``` (a) Zero-shot VLM planner. You are one of a team of embodied robots navigating in the real world. At each step you choose where to move next, using neighboring-viewpoint images, until you reach the destination and stop. ## Inputs -'atomic instruction’: ... -'atomic instruction history’: ... -'lifelong history’: ... -'history’: ... -'previous planning’: ... -'action descriptions’: ... -'action candidates’: ... -'expert recommendation’: ... ## Reasoning each turn 1. Parse the atomic instruction into: primary target, likely room/area, and 2-3 distinctive cues. Treat the room/area as route guidance, not as the destination by itself. 2. Decide whether to stop before considering movement. Stop only when the current viewpoint clearly matches the primary target and at least one distinctive cue, with no contradictory room/type evidence. 3. If not stopping, rank movement evidence: direct target/cue visibility first; then a doorway, hallway, opening, staircase, or frontier toward the likely room/area; then object-list hints as a tiebreaker. 4. Use history to avoid redundant checks. Do not re-inspect a room/view already found without target evidence. Backtrack only to reach a new branch/frontier, return from a dead end, or recover from a wrong room/type. 5. In planning, keep a short action chain: target/cue -> chosen waypoint/action -> next confirmation. ## Output Format ```json "thought": A concise reasoning for navigation action, "planning": One short sentence: target/cue -> chosen waypoint/action -> next confirmation. Refine 'previous planning' without restating completed checks, "action": Exactly one element from 'action candidates' ``` (b) Hybrid planner. Fig. S11. Prompt templates of the VLM-assisted planners (Section E.4). (a) Zero-shot planning from the candidate views, the verbalized action options and the agent’s own step history. (b) Hybrid planning, queried only when the trained planner is uncertain, which additionally receives its ranked shortlist as a strong prior and the targets already claimed by teammates.

Instant research discovery