Paper deep dive

Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 84

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $\pi_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

80,392 characters extracted from source content.

Expand or collapse full text

Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation Youpeng Wen 1 , Gang Wang 1 , Tianyu Huang 1 , Weiwei Zhang 1 , Zhijian Huang 2 , Wei Chen 1 , Xieyuanli Chen 3 , Yunhui Liu 1 1 The Chinese University of Hong Kong 2 Xiaomi Corporation 3 National University of Defense Technology Abstract A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and ex- ecution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destina- tions during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a bench- mark named RMBench-Compose. Orchestra achieves a mean success rate of68.22% across nine RM- Bench tasks, outperforming Mem-0 by26.22percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of7.35subtasks,2.46times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by25.0% over Orchestra using the baseπ 0.5 policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra. 1 Introduction Generalist robot policies and vision-language-action (VLA) models have expanded language-conditioned manipulation [ Brohan et al.,2023,Octo Model Team et al.,2024,Kim et al.,2024,Physical Intelligence et al. ,2025]. While these advances improve local action capabilities, executing long-horizon tasks remains challenging because task progress must stay aligned with changing object states and goals. This raises a system-level question:how can an action policy execute a complete user instruction across a long sequence of interactions?Addressing this question requires coordination beyond local action generation, including maintaining the requested operation order, grounding objects and destinations, and managing subtask tran- sitions. The first challenge is maintaining task continuity by tracking completed and pending operations while retaining relevant observations. Memory-based approaches combine short-horizon visual history with task- level information [ Torne et al.,2026], and RMBench studies memory-dependent manipulation through its benchmark and Mem-0 policy [Chen et al.,2026]. However, these memory mechanisms alone do not ensure that perceptual history and task records remain aligned with the active subtask throughout execution of a complete instruction. 1 arXiv:2610.18352v1 [cs.RO] 13 Oct 2026 (a) Long-horizon instruction following “Arrange the nameplates on the blue mat in the order 3, 1, 2.” Task continuity Repeat or skip a subtask Instruction grounding Wrong object or destination 312 Correct order + entities (b) Orchestra overview Instruction Language + visual cues Task Agent Plan and track subtasks Visual grounding Localize + track Observation RGB + robot state Shared Context Active subtask + grounding Recent visual history Working memory Action Policy Spatial + temporal Progress Evaluator Subtask progress Robot execution Advance to next subtask (c) Benchmarks and real-world tasks RMBench / RMBench-Compose 9 tasks / 5 Compose environments Train on atomic skills Evaluate 6–18-subtask compositions Real-world 4 manipulation tasks Evaluate 3–5 subtasks 20 matched instructions per task (d) Orchestra performance RMBench Success rate (%) +26.22 p 0 50 100 42.00 Mem-0 68.22 Ours RMBench-Compose ASL (subtasks) 2.46×ASL 0 5 10 2.99 HarnessVLA 7.35 Ours Real-world ASL (subtasks) +25.0% ASL 0 2 4 2.20 Orchestra with π₀.₅ 2.75 Ours Figure 1:Overview of Orchestra. (a)Long-horizon instruction following requires task continuity and correct grounding. (b) Orchestra coordinates action generation and subtask transitions through a shared execution context.(c)Evaluation spans RMBench, RMBench-Compose, and real-world tasks.(d)Orchestra improves success rate and Average Success Length over the indicated baselines. The second challenge is preserving instruction grounding: a mechanically successful action can still target the wrong object or destination [Hou and Zhao,2026]. Multimodal prompts and spatial representations support task specification and grounding [Jiang et al.,2022,Huang et al.,2023], but task-relevant grounding must be refreshed as objects move and goals change. Grounding and continuity are therefore coupled: each operation must target the intended entity at the correct point in the sequence. Robotic systems have increasingly drawn on language-model agent architectures to organize task execu- tion. These architectures coordinate reasoning, action, and feedback [ Yao et al.,2023], while their robotic applicationsincorporateaffordances, executionfeedback, andprogrammaticcontrol[ Ahnetal.,2022,Huang et al. ,2022,Liang et al.,2022]. More recently, Harness VLA uses a memory-guided agent to compose a frozen VLA with analytic manipulation primitives [ Zhang et al.,2026]. Building on this perspective, we focus on a shared execution context that aligns the active subtask, spatial grounding, observation history, and progress-driven transitions (Figure 1). Guided by the principle“Let the mind conceive, let the hand achieve”, we introduceOrchestra, an agent harness for task-level interpretation and execution management. Orchestra separates these functions from local action generation through a shared execution context. Given a complete multimodal instruction, it constructs ordered subtasks and maintains this context for the active subtask. Perception tools update task- relevantspatialgrounding, whileworkingmemorypreservestaskrecordsandrecentobservations. Moreover, we instantiate the action policy with an adaptedπ 0.5 model [ Physical Intelligence et al.,2025], conditioned on the active subtask, spatial prompts, recent RGB observations, and current proprioception. We also train an independent progress evaluator that uses aligned context to estimate subtask progress and guide transitions. At each subtask transition, the harness preserves task records while resetting local observation buffers and spatial grounding. We evaluate Orchestra on both simulated benchmarks and real-world manipulation tasks. The simulated benchmarks include RMBench [ Chen et al.,2026] and RMBench-Compose, which we introduce by adapt- 2 ing five RMBench environments to support variable-length, state-feasible compositions of atomic skills. On RMBench-Compose, the action policy is fine-tuned only on atomic instructions and short trajectories, while evaluation requires executing longer subtask sequences from complete instructions as object states change. Together, these evaluations assess memory-dependent manipulation, composition beyond the training hori- zon, and sustained execution on real hardware. Our contributions are threefold: •We introduceOrchestra, an agent harness that maintains a shared execution context to coordinate task interpretation, spatial grounding, working memory, and subtask transitions throughout complete instructions. •We adapt a VLA policy to combine subtask instructions, spatial prompts, and recent visual history, and train an independent progress evaluator to separate action generation from subtask-completion assessment. •We introduceRMBench-Composeto evaluate long-horizon task execution and evaluate Orchestra on RMBench, RMBench-Compose, and real-world manipulation tasks. Orchestra improves mean RM- Bench success rate by26.22percentage points over Mem-0 and achieves2.46×the Average Success Length of HarnessVLA on RMBench-Compose. On real-world tasks, it improves mean Average Suc- cess Length by25.0% over Orchestra with the baseπ 0.5 policy. 2 Related Work Imitation-based robot policies.Recent robot learning has increasingly focused on building generalist ma- nipulation policies from large-scale and heterogeneous demonstrations [ Kim et al.,2024,Black et al.,2025, Physical Intelligence et al.,2025]. Early vision-language-action (VLA) models transfer semantic representa- tions from pre-trained vision-language models to robot control; they typically discretize continuous actions and predict them as tokens [ Brohan et al.,2023,Kim et al.,2024]. More recent approaches instead condition dedicated action experts on vision-language representations, enabling more expressive continuous control. For example,π 0 uses a flow-matching action expert to generate temporally chunked actions [Black et al., 2025], whileπ 0.5 combines heterogeneous co-training with semantic subtask prediction for open-world gen- eralization[PhysicalIntelligenceetal.,2025]. Thesemodelsmainlyimprovemanipulationcapabilitythrough policy scaling, broader training data, and stronger action representations. In this work, we take a comple- mentary approach: We retain the pretrained VLA policy as the action policy and externalize the planning, memory, and command-mapping functions to the agent-level system. Long-horizon manipulation.Long-horizon manipulation requires tracking task progress and relevant history beyond the current observation. Recent approaches address this through hierarchical task decompo- sition and temporal memory. Hierarchical methods separate coarse task semantics from low-level control; for example, MimicPlay uses high-level guidance derived from human play [Wang et al.,2023], whileπ 0.5 predicts semantic subtasks that condition subsequent action generation [Physical Intelligence et al.,2025]. More recent work incorporates historical context into VLA policies. HAMLET aggregates compact moment tokens through a lightweight memory module, allowing pretrained VLAs to eﬀiciently use past observa- tions [ Koo et al.,2026]. MEM further introduces memory at multiple temporal scales, combining short-term perceptual context with long-term semantic task information [Torne et al.,2026]. RMBench conducted a systematic study of this configuration and proposed Mem-0 to enable memory-based operations using ex- plicit memory components [ Chen et al.,2026]. These methods primarily incorporate temporal context into the policy or its inference architecture. In contrast, our framework combines policy-level visual memory with explicit task records and subtask-transition control maintained by the agent harness. 3 Robot agents.Recent robot-agent systems use foundation models to coordinate reusable robot skills, separating high-level reasoning from low-level execution. Early approaches such as Code as Policies gener- ate executable programs over perception and control APIs, providing a modular interface between language- model reasoning and robot control [ Liang et al.,2022]. Later systems extend this idea to large-scale em- bodied orchestration. AutoRT combines VLM-based scene understanding with LLM-based task generation to coordinate robot behaviors in real-world environments [Ahn et al.,2024]. More recent work integrates VLAs directly into the agent hierarchy. VLAs-as-Tools treats specialized VLA policies as executable tools selected by a high-level VLM agent. It also uses execution feedback and event-triggered replanning to sup- port long-horizon tasks [Lei et al.,2026]. Harness VLA takes a different approach. It uses a frozen VLA as a contact-rich manipulation primitive and coordinates it with analytic primitives through a memory-guided agent [Zhang et al.,2026]. These methods shift robot agents from predefined symbolic skills toward learned VLApolicies. Ourframeworkfollowsthisdirection, emphasizingexternaltaskcoordination, multi-timescale memory, and multimodal instruction anchoring around a pretrained VLA. 3 Method a Let the MindConceiveb LettheHandAchieve Multi-modal instruction ℓ Agent Conceive Subtasksg₁ ... gᵢ ... gₖ “Place the bread inthe bowl.” Grounding Localize + track Observation oₜ RGBState Shared Agent Context Cₜ Observation Working Memory Mᴬ Parallel inference Action Capability π Spatial + temporal conditioning Progress Evaluator e Predict subtask progress Action chunk Robot Subtask Controller Completion →next subtask Shared input xₜ= Π(Cₜ) Shared Agent Context State Action Capability π Short-horizon video encoder STST spatial / temporal features Action chunk Progress Evaluator e Progress network Separate temporal features 0100200 0 50 100 Frame Progress (%) “Arrange the objects as shown.” “Place the bread in the bowl.” Subtask gᵢGrounding Fₜ Pre-trained Policy Action expert “Place the bread in the bowl.” Subtask Grounding Short-term visual memory Figure 2:Orchestra framework. (a) Let the Mind Conceive.The agent decomposes multimodal instruc- tions into subtasks, grounds relevant entities, and maintains task records and recent RGB observations.(b) Let the Hand Achieve.An action policy and an independent progress evaluator use the active subtask, observations, and spatial grounding to generate action chunks and predict progress. A terminal-category prediction triggers the next subtask. 3.1 Problem Formulation and Overview We consider long-horizon manipulation specified by a multimodal instructionℓ. At timet, the robot observes o t = (I t , s t ), comprising multi-camera images and proprioception. Orchestra convertsℓinto an ordered plan G= (g 1 , . . . , g N )andgroundstheactivesubtaskg i t inthecurrentscene. Anactionpolicyπ θ andaseparately 4 trained progress evaluatore φ operate on a shared context: C t = ( o t , g i t , M A t , F t ) , A t =π θ (C t ),q t =e φ (C t ), (1) whereF t is the spatial grounding,A t an action chunk, andq t a distribution overBwithin-subtask progress categories. The shared context provides a common execution interface, with model-specific adapters select- ing the inputs consumed by the action policy and progress evaluator. Agent working memory is M A t = (D t ,H t , i t ),(2) containing completed-subtask records, a bounded history of timestamped multi-camera observations, and the active index. The controller tracks completed subtasks inD t , while the policy and progress evaluator use recent observations fromH t . At each subtask transition, the controller retains the task record and reinitializes the visual history and spatial grounding for the next subtask. 3.2 Task Orchestration and Visual Grounding Task interpretation.Orchestra supports three input forms: (i) language paired with a reference image, e.g., “Arrange the objects in the corresponding positions shown in the reference image”; (i) language-only instructions, e.g., “Arrange the numbered nameplates on the blue mat in the order 3, 2, 1”; and (i) referring instructions paired with user-provided points or bounding boxes, e.g., “Put this on that,” where annotations identify the referenced source object and target destination. We use DeepSeek-V4.1-Flash [DeepSeek-AI, 2026]asthedefaulttask-orchestrationagentandevaluateQwen3.7-Plus[QwenTeam,2026]asareplacement (Section 5.3). The module converts each input into ordered subtasks with executable instructions and entity bindings. Reference images and user annotations are resolved at this stage; the action policy receives the active subtask and its scene grounding. Subtask-conditioned grounding.MolmoPoint-8B [Clark et al.,2026] locates task-relevant entities in the current head-camera image, providing point prompts for SAM3 [ Carion et al.,2026] instance segmenta- tion. Grounding is initialized at subtask boundaries; SAM3 updates the selected regions at subsequent action replanning calls. The resulting masks provide bounding boxes, sampled points, or segmentation contours for downstream conditioning. 3.3 Spatially and Temporally Conditioned Action Policy We augmentπ 0.5 [Physical Intelligence et al.,2025] with spatial conditioning and short-term visual memory. Its PaliGemma [ Beyer et al.,2024] backbone and action expert generate actions from the active subtask, spa- tial prompts, proprioception, and recent RGB observations. We use factorized spatial and temporal attention to incorporate visual history. Spatial conditioning.During training, each demonstration is paired with spatial annotations for entities relevant to the active subtask. Bounding boxes and points are serialized as entity names with quantized co- ordinates, e.g.,<bbox, "brown loaf", [x1,y1,x2,y2]>or<point, "brown loaf", [x,y]>, and appended to the language prefix alongside the subtask instruction. For mask-based conditioning, segmenta- tion contours are overlaid on the corresponding objects in the head-camera image. Temporal visual encoding. For each camera c , we sample a window from H t : W c t = ( I c t−(K−1)∆ , . . . , I c t−∆ , I c t ) ,(3) whereKis the window length and∆the sampling stride; unavailable history is masked. Spatial atten- tion operates within frames, and temporal attention mixes features at corresponding patch positions across 5 frames. Onlycurrent-framefeaturesareretainedforthelanguage-modelprefixaftertemporalfusion. History therefore adds temporal computation without increasing the number of visual tokens passed to the language model. Action learning. The policy predicts action chunks of length H p using the flow-matching objective of the base policy [Lipman et al.,2023], conditioned on the subtask instruction, spatial prompts, and recent visual observations. 3.4 Progress Estimation and Closed-Loop Execution Progressprediction.AnindependentevaluatorwithaGemma-300M[SchechterVeraetal.,2025]backbone receives the active subtask, spatial conditions, proprioception, and recent visual history. A learnable progress token aggregates these inputs intoh t , yielding q t =softmax ( W prog LN(h t ) +b prog ) .(4) The predicted category is ˆ b t =argmax j∈0,...,B−1 q t,j . Progress supervision.Demonstrations are segmented into language-labeled subtasks. For frameuin segment[a i , e i ), temporal progress and its discrete label are ρ i (u) = 1− max(e i −1−u,0) max(e i −a i ,1) , b i (u) =clip ( round ( (B−1)ρ i (u) ) ,0, B−1 ) . (5) We train the evaluator with cross-entropy: L prog =−E (u,i) logq u,b i (u) .(6) Theselabelsdescribetemporalprogress, andtheirterminal-categoryinterpretationisdetailedinAppendixB.5. Closed-loop execution.At each action-chunk replanning call, the agent updates grounding and invokes the policy and evaluator in parallel. The agent switches subtasks when the most probable progress category is terminal: c t =1[ ˆ b t =B−1].(7) Otherwise, it executes the action chunk and continues the current subtask. A transition records completion, discards the old subtask’s action chunk, advances the plan, and resets local histories and grounding. The final transition terminates the plan; task success is measured separately. Intermediate observations update visual histories without invoking grounding or progress evaluation. 4 RMBench-Compose Benchmark We introduce RMBench-Compose by adapting five RMBench [Chen et al.,2026] environments for long- horizon instruction following. Whereas the original tasks use predefined instruction templates and execution sequences, RMBench-Compose supports varying the length and order of feasible operations within each environment. The benchmark evaluates whether systems can follow complete language instructions over sequences longer than those seen during atomic-skill training. Successful execution requires maintaining the requested operation order and selecting the intended objects and destinations throughout the sequence. The five environments arePress Button,Pick Up Object,Cover and Uncover Block,Swap Block, and Place Block on Mat. Each environment defines atomic subtasks with associated language instructions, rele- vant entities, feasibility conditions, and success predicates. RMBench-Compose also provides segmentation 6 masks, bounding boxes, and points for objects and spatial targets relevant to the active subtask. Feasibil- ity depends on the current scene state; for example, an uncovering operation requires the target block to be covered. Let U e be the atomic subtask library for environment e , and F e ( x ) ⊆U e its feasible subset at state x . A composed task is G= (g 1 , . . . , g L ),g j ∈F e (x j−1 ),(8) wherex j−1 is the scene state after successful execution of the preceding subtasks. Sequence length and subtask order are configurable, subject to the environment’s available skills and state-dependent feasibility conditions. Evaluation protocol.Training demonstrations contain atomic instructions and short trajectories. At evaluation time, each episode begins with a complete language instructionℓspecifying the ordered subtask sequence. Each subtask starts from the scene state left by the preceding subtask and has an environment- specific action budget. Episodes end when the sequence is completed or a subtask exhausts its action budget. We report Atomic Success Rate (Atomic SR) on isolated atomic tasks and two metrics for composed-task execution. Average Success Length (ASL) measures the mean number of consecutive subtasks completed in the requested order. Subtask Success Rate (SSR) measures the fraction of attempted subtasks completed successfully, excluding subtasks not reached after an earlier failure. 5 Experiments We evaluate sustained instruction following on RMBench [Chen et al.,2026], RMBench-Compose, and a physical robot, with ablations examining the components of Orchestra. 5.1 Experimental Setup Benchmarks and task inputs.We evaluate nine RMBench tasks and five RMBench-Compose environ- ments under the atomic-training, composed-evaluation protocol in Section4. Evaluation lengths aren= 10,12,6,14,18for Pick Up Object, Cover and Uncover Block, Press Button, Swap Block, and Place Block on Mat, respectively. We conduct 100 evaluation rollouts for each task. Training uses atomic instructions and short demonstrations, whereas evaluation begins with the complete ordered instruction and preserves scene states across subtask boundaries. Compared configurations.Table 1combines nine oﬀicially reported RMBench [Chen et al.,2026] baselines with our evaluations of HarnessVLA and Orchestra. RMBench-Compose comparesπ 0.5 [Physical Intelligence et al. ,2025], HarnessVLA [Zhang et al.,2026], Orchestra withπ 0.5 , the complete Orchestra system, and its Qwen3.7-Plus-agent variant.Orchestra withπ 0.5 uses the base action policy instead of the spatially and temporally conditioned policy in Section3.3. The Qwen-agent variant replaces only the task- orchestration agent. AppendixDdetails the sources of baseline results and model configurations. Implementation details.In simulation, spatial inputs use environment-provided ground-truth segmen- tation masks. For real-world training data, we use an automatic annotation pipeline combining Qwen3.7- Plus [ Qwen Team,2026] for subtask decomposition, MolmoPoint-8B [Clark et al.,2026] for object pointing, and SAM3 [Carion et al.,2026] for segmentation. During real-world deployment, pointing and segmentation initialize spatial grounding at subtask boundaries; masks are subsequently tracked at action-chunk replanning calls (Section3). The temporally conditioned action policy uses a visual history window ofK= 6frames with a sampling stride of∆ = 3. The progress evaluator usesB= 11discrete within-subtask progress categories. 7 Table 1:Success rates on RMBench.The best result in each column is shown in bold. Method Observe and Pick Up Rearrange Blocks Put Back Block Swap Blocks Swap T Battery Try Blocks Ranking Try Cover Blocks Press Button Total average Base policies DP1%0%0%11%20%10%10%0% 0% 5.78% ACT1%29%0%2% 2% 19%0%0% 0% 5.89% π 0.5 9%13%11% 24% 15% 16%6%0% 0% 10.44% X-VLA9%13%18% 16% 3% 26%1%2% 0% 9.78% QwenOFT0%0%0%0% 0% 14%37%0% 0% 5.67% Dual-system models MemER7%17%0%14% 7% 27%0%6% 0% 8.67% Mem-04%89%90%67% 14% 28%18%68% 0% 42.00% End-to-end memory models MemoryVLA (OpenVLA)0%22%50% 17% 9% 25%12%40% 0% 19.44% MemoryVLA (QwenOFT)2%53%81% 76% 9%33%53%69%0% 41.78% Agent-based systems HarnessVLA10%32%62% 18% 5% 24%20%26% 44% 26.78% Orchestra81%95%81%86%10%30%65%69%97%68.22% 5.2 Evaluation on RMBench Orchestra achieved the highest mean success rate among the evaluated methods, reaching68.22% versus Mem-0’s42.00%, a26.22-percentage-point gain (Table 1). On Observe and Pick Up and Press Button, it achieved81% and97%, compared with HarnessVLA’s10% and44%, respectively. However, Put Back Block remained below Mem-0 (81% versus90%), and Swap T success was only10%. Thus, strong average performance coexisted with task-specific weaknesses; aggregate scores do not separate manipulation failures from task-management errors. 5.3 Compositional Execution on RMBench-Compose Atomic competence and sustained execution.The baseπ 0.5 policy achieved85.0% Atomic SR but only 47.36% SSR and0.932ASL during composed execution (Fig.3). Orchestra with the same policy retained 85.0% Atomic SR while increasing SSR to70.78% and ASL to2.884(3.09×). This comparison supports organizing local actions around explicit subtasks and transitions. Complete-system performance.Complete Orchestra achieved95.2% Atomic SR,86.69% SSR, and 7.350ASL, or2.46×HarnessVLA’s2.986ASL. Its ASL was also2.55×that of Orchestra withπ 0.5 , sup- porting the complete configuration combining the harness with the spatially and temporally conditioned policy. Orchestra withπ 0.5 alone remained below HarnessVLA in mean ASL, so adding the harness alone does not explain the complete system’s advantage. Orchestra exceeded HarnessVLA in ASL across all five environments, with the largest absolute gain on Place Block on Mat (4.35to13.06,n= 18). On Press Button, Orchestra completed an average of5.79of the6requested subtasks. Pick Up Object remained dif- ficult, reaching only2.11ASL (n= 10) despite81% Atomic SR. These results expose a remaining gap between isolated competence and sustained execution, while differences in task diﬀiculty prevent attributing cross-environment variation solely to sequence length. Alternative orchestration agent.Replacing the agent with Qwen yielded85.74% SSR and7.188ASL, close to the default configuration’s86.69% and7.350; AtomicSR remained95.2%. Qwenperformed slightly better on Cover and Uncover Block (5.90versus5.82ASL), while the default agent was higher in the other four environments. Strong performance was retained under this substitution, without establishing robustness to arbitrary agent choices. Fig. 10provides per-environment Atomic SR and SSR. 8 0%50%100% π₀.₅ HarnessVLA Orchestra + π₀.₅ Orchestra (Qwen agent) Orchestra 85.0 87.8 85.0 95.2 95.2 0%50%100% 47.4 72.1 70.8 85.7 86.7 048 0.93 2.99 2.88 7.19 7.35 0 5 10 2.11 Pick up object 0 6 12 5.82 Cover / uncover block 0 3 6 5.79 Press button 0 7 14 9.97 Swap block 0 9 18 13.06 Place block on mat (a)Atomic SR(b)Subtask success(c)Mean ASL (d) Task-level ASL (dashed lines: task lengths) π₀.₅ HarnessVLA Orchestra + π₀.₅ Orchestra (Qwen agent) Orchestra Figure 3:Atomic and compositional performance on RMBench-Compose. (a–c)Unweighted five- environment averages of Atomic SR, SSR, and ASL.(d)Per-environment ASL; dashed segments indicate task lengths. Orchestra withπ 0.5 changes the action policy; the Qwen variant changes the orchestration agent. Spatially conditioned configurations use environment-provided grounding. 5.4 Ablation Studies (a) Component ablations Mean ASLTask-level ASL Pick up Cover / uncover PressSwapPlace Orchestra 7.350 2.115.825.799.9713.06 w/o spatial grounding 5.814 2.002.815.027.2911.95 w/o visual memory 2.500 1.692.471.284.872.19 w/o task orchestration 1.502 1.071.891.202.301.05 Oracle transitions 7.374 2.205.675.9110.1112.98 048 Average success length ASL / n 0.00.51.0 (b) Simulation examples Cover / uncover Open left coverCover left blockCover right block Swap blocks Middle to left trayRight to middle trayMiddle to right tray Figure 4:Component ablations and simulation examples on RMBench-Compose. (a)Mean and per- environment ASL for component removals and state-based oracle transitions. Heatmap cells report raw ASL; colors encode ASL/n.(b)Selected subtask frames illustrating Cover and Uncover Block and Swap Block. Componentcontributions.Removingtaskorchestration,short-horizonvisualmemory,orspatialground- ing reduced mean ASL from7.350to1.502,2.500, and5.814, respectively (Fig.4a). These correspond to decreases of79.6%,66.0%, and20.9%. Task orchestration made the largest contribution: directly condition- ing the policy on the complete instruction reduced mean ASL by nearly80%, demonstrating the importance of structured subtask execution. Short-horizon visual memory was also critical, especially on Place Block 9 on Mat (13.06to2.19ASL) and Press Button (5.79to1.28). Spatial grounding provided substantial gains on visually demanding tasks, with Cover and Uncover Block increasing from2.81to5.82ASL when grounding was enabled. Together, these ablations show that task orchestration, temporal memory, and spatial grounding each contribute to sustained compositional execution. Learned versus state-based transitions.Replacing the progress evaluator with the simulator’s state- based completion indicator yielded7.374mean ASL, compared with7.350for Orchestra’s learned evaluator. The mean gap was only0.024subtasks, and absolute per-environment differences were at most0.15. This small aggregate gap suggests similar rollout performance, without establishing equivalent completion detec- tion. 5.5 Real-World Evaluation (a) Real-world performance Orchestra + π₀.₅Orchestra Subtask success · SSR (%) 0 25 50 75 100 68.2 74.0 Breakfast 77.4 91.2 Fruit bowl 71.2 79.6 Number ordering 74.3 83.5 Trash bin Sustained execution · ASL 0 1 2 3 4 5 2.15 2.70 Breakfast n = 4 2.05 2.60 Fruit bowl n = 3 1.85 2.15 Number ordering n = 3 2.75 3.55 Trash bin n = 5 (b) Grounded subtask examples Fruit Bowl Placement Subtask 1Subtask 2Subtask 3 Number Ordering Subtask 1Subtask 2Subtask 3 Figure 5:Real-world performance and grounded subtask examples. (a)SSR and ASL on four tasks over20episodes per task and configuration. Dashed lines mark task lengths (n= 4,3,3,5).(b)Selected subtask examples for Fruit Bowl Placement and Number Ordering, with corresponding object masks overlaid on RGB frames. We evaluated Breakfast Preparation, Fruit Bowl Placement, Number Ordering, and Trash Bin Placement, with instruction lengths of4,3,3, and5subtasks, respectively. We ran20evaluation rollouts per task for each configuration, totaling80rollouts per configuration. Both configurations used the same set of randomly selected task instructions. Execution used predicted and tracked grounding; AppendixD.4describes the robot platform. Complete Orchestra improved SSR and ASL on all four tasks (Fig. 5a). Mean SSR increased from 72.77% to82.09%, and mean ASL increased from2.20to2.75(25.0%). Fruit Bowl Placement showed the largest SSR gain, increasing from77.36% to91.22%. Trash Bin Placement showed the largest ASL gain (2.75to3.55), with improvements observed across all evaluated instruction lengths. With the same Orchestra harness used in both configurations, these consistent gains highlight the practical contribution of the spatially and temporally conditioned policy under predicted real-world grounding. Additional qualitative real-world results and failure analyses are provided in Appendices E.3andE.4, respectively. 6 Conclusion We introduced Orchestra, an agent harness that coordinates task orchestration, spatial grounding, working memory, and progress-driven subtask transitions through a shared execution context. We also introduced 10 RMBench-Compose to evaluate long-horizon composition after atomic-skill training. Orchestra achieved 68.22% mean success on RMBench and7.35Average Success Length (ASL) on RMBench-Compose,2.46× that of HarnessVLA. Across four real-world tasks, it improved mean ASL by25.0% over Orchestra with the base π 0.5 policy. These results support shared-context coordination as a complement to local action capabilities. Performance remains task-dependent, and simulation relies on reference grounding, motivating evaluation on longer real-world sequences with predicted grounding. References Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can and not as I say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022. URLhttps://arxiv.org/abs/2204. 01691. MichaelAhn, DebidattaDwibedi, ChelseaFinn, MontseGonzalezArenas, KeerthanaGopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Ser- manet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, and Zhuo Xu. AutoRT: Em- bodied foundation models for large scale orchestration of robotic agents.arXiv preprint arXiv:2401.12963, 2024. URL https://arxiv.org/abs/2401.12963. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalam- pidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. URL https://arxiv.org/abs/2407.07726. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.π 0 : A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2025. URLhttps://arxiv. org/abs/2410.24164 . Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Are- nas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kan- ishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Ste- fan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna 11 Zitkovich. RT-2: Vision-language-actionmodelstransferwebknowledgetoroboticcontrol.arXiv preprint arXiv:2307.15818, 2023. URLhttps://arxiv.org/abs/2307.15818. Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chai- tanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Radle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollar, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment anything with concepts.International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2511.16719. Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang, Junjie Wang, Tianhang Yang, Renjing Xu, Ruihai Wu, Yao Mu, Yaodong Yang, Hao Dong, and Ping Luo. RMBench: Memory-dependent robotic manipulation benchmark with insights into policy design.arXiv preprint arXiv:2603.01229, 2026. URLhttps:// arxiv.org/abs/2603.01229. Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.arXiv preprint arXiv:2303.04137, 2023. URLhttps://arxiv.org/abs/2303.04137. Christopher Clark, Yue Yang, Jae Sung Park, Zixian Ma, Jieyu Zhang, Rohun Tripathi, Mohammadreza Salehi, Sangho Lee, Taira Anderson, Winson Han, and Ranjay Krishna. MolmoPoint: Better pointing for VLMs with grounding tokens.arXiv preprint arXiv:2603.28069, 2026. URL https://arxiv.org/ abs/2603.28069 . DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression.arXiv preprint arXiv:2609.19969, 2026. URLhttps://arxiv.org/abs/2609.19969. Yuchen Hou and Lin Zhao. LangGap: Diagnosing and closing the language gap in vision-language-action models.arXiv preprint arXiv:2603.00592, 2026. URL https://arxiv.org/abs/2603.00592. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models.arXiv preprint arXiv:2207.05608, 2022. URL https://arxiv.org/abs/2207. 05608. Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models.arXiv preprint arXiv:2307.05973, 2023. URL https://arxiv.org/abs/2307.05973. YunfanJiang, AgrimGupta, ZichenZhang, GuanzhiWang, YongqiangDou, YanjunChen, LiFei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2022. URLhttps://arxiv.org/abs/2210.03094. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024. URL https://arxiv.org/ abs/2406.09246. 12 Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, and Jinwoo Shin. HAMLET: Switch your vision-language-action model into a history-aware policy.arXiv preprint arXiv:2510.00695, 2026. URLhttps://arxiv.org/abs/2510.00695. Zixing Lei, Changxing Liu, Yichen Xiong, Minhao Xiong, Yuanzhuo Ding, Zhipeng Zhang, Weixin Li, and Siheng Chen. Towards long-horizon embodied agents with tool-aligned vision-language-action models. arXiv preprint arXiv:2605.13119, 2026. URLhttps://arxiv.org/abs/2605.13119. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control.arXiv preprint arXiv:2209.07753, 2022. URLhttps://arxiv.org/abs/2209.07753. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.International Conference on Learning Representations, 2023. URLhttps:// arxiv.org/abs/2210.02747. Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024. URLhttps://arxiv.org/abs/2405. 12213. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky.π 0.5 : A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025. URL https://arxiv.org/abs/ 2504.16054 . Qwen Team. Qwen3.7-Plus model documentation. Alibaba Cloud Model Studio Documentation, 2026. URL https://help.aliyun.com/zh/model-studio/qwen3-7-plus. Accessed: 2026-09-24. Henrique Schechter Vera, Sahil Dua, and EmbeddingGemma Team. Embeddinggemma: Powerful and lightweight text representations. 2025. URLhttps://arxiv.org/abs/2509.20354. Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manipulation.arXiv preprint arXiv:2508.19236, 2025. URL https://arxiv.org/abs/ 2508.19236. Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. MemER: Scaling up memory for robot con- trol via experience retrieval.arXiv preprint arXiv:2510.20328, 2025. URLhttps://arxiv.org/abs/ 2510.20328. StarVLA Community. StarVLA: A lego-like codebase for vision-language-action model developing.arXiv preprint arXiv:2604.05014, 2026. URLhttps://arxiv.org/abs/2604.05014. Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Sprin- genberg, Sergey Levine, Chelsea Finn, and Danny Driess. MEM: Multi-scale embodied memory for 13 vision language action models.arXiv preprint arXiv:2603.03596, 2026. URLhttps://arxiv.org/ abs/2603.03596. Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. MimicPlay: Long-horizon imitation learning by watching human play.arXiv preprint arXiv:2302.12422, 2023. URLhttps://arxiv.org/abs/2302.12422. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Repre- sentations, 2023. URLhttps://arxiv.org/abs/2210.03629. Yixian Zhang, Huanming Zhang, Feng Gao, Xiao Li, Zhihao Liu, Chunyang Zhu, Jiaxing Qiu, Yuchen Yan, Jiyuan Liu, Wenhao Tang, Zhengru Fang, Yi Nie, Changxu Wei, Yu Wang, Wenbo Ding, and Chao Yu. Harness VLA: Steering frozen VLAs into reliable manipulation primitives via memory-guided agents. arXiv preprint arXiv:2607.08448, 2026. URLhttps://arxiv.org/abs/2607.08448. Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipula- tion with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023. URLhttps://arxiv.org/abs/ 2304.13705. Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, Ya-Qin Zhang, Jiangmiao Pang, Jingjing Liu, Tai Wang, and Xianyuan Zhan. X-VLA: Soft-prompted transformer as scalable cross-embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025. URL https://arxiv.org/abs/2510.10274. 14 Appendix Overview The appendix is organized as follows: •AppendixA: Agent Framework and Execution Details.Describes task decomposition, the agent prompt and output schema, shared context, memory management, and progress-driven execution. •AppendixB: Policy and Evaluator Implementation.Details the action policy, spatial conditioning, short-horizon visual memory, and progress evaluator, including progress-target generation and subtask transitions. •AppendixC: Data Collection and Training.Summarizes real-world and simulation demonstration collection, annotation procedures, and training configurations. •AppendixD: Evaluation Protocols.Specifies composed instructions, execution budgets, evaluation metrics, baseline sources, and the physical robot platform. •AppendixE: Additional Results and Analysis.Presents per-environment compositional results, ab- lation implementation details, qualitative execution examples, and failure analysis. A Agent Framework and Execution Details A.1 Task interpretation and registered capabilities Orchestra receives the complete user instruction and the current scene, together with the registered action- policy capabilities. The instruction may be language-only or include a reference image, points, or bounding boxes. The agent converts these inputs into an ordered sequence of executable subtasks and identifies the entities relevant to each subtask. The complete instruction is available at the beginning of an episode. Unless otherwise specified, the task-orchestration agent is DeepSeek-V4.1-Flash. The Qwen3.7-Plus variant replaces this orchestration agent, as evaluated in Section 5.3. The framework exposes action gener- ation and progress detection as separate capabilities. The experiments instantiate them with the policy and evaluator described below; the shared-context interface permits other input configurations without changing the overall orchestration structure. A.2 Agent prompt and output schema Figure6presents the prompt template used by the task-orchestration agent to decompose a complete user instruction into an ordered sequence of manipulation subtasks. The agent receives the current scene image, the user instruction with any supplied visual references, and the registered action-policy capabilities. The prompt directs the agent to preserve explicit operation order, account for task dependencies and expected scene changes, and identify relevant entities using consistent, visually distinguishable descriptions. It also requires coverage of all requested operations, including explicitly specified repetitions, without introducing unsupported objects, destinations, or capabilities. The output is a JSON array in execution order. Each entry contains asubtaskfield specifying a ma- nipulation instruction and anactivate objectfield listing the relevant entity descriptions. This interface supplies the subtask sequence and textual entity descriptions to downstream components, which perform spatial localization, action generation, and completion detection. 15 Task agent prompt You are a robot agent responsible for decomposing a complete user instruction into an ordered sequence of executable manipulation subtasks. Inputs You are provided with: Current scene image input_image Complete user instruction and any supplied visual references instruction Registered action-policy capabilities registered_action_policy_capabilities Planning rules Construct a subtask sequence that fulfills the complete instruction using the registered capabilities: 1. Preserve the operation order specified by the user. If no order is specified, choose an order consistent with task dependencies and the expected scene changes. 2. Express each subtask as a concrete manipulation instruction executable by a registered capability. 3. Account for the expected effects of preceding subtasks when describing later operations. 4. Identify each subtask's relevant objects. Use consistent, visually distinguishable descriptions across subtasks. 5. Resolve referring expressions using the current scene and any supplied reference image, points, or bounding boxes. 6. Cover all requested operations. Avoid unintended omissions or duplicates, while preserving repetitions explicitly required by the instruction. 7. Do not invent objects, destinations, or capabilities unsupported by the inputs. Output schema Return a JSON array in execution order. Each item contains "subtask", an executable instruction, and "activate object", a list of relevant entity descriptions. [ "subtask": "<executable instruction>", "activate object": ["<object>"] ] Produce the plan and entity descriptions only; spatial localization, action generation, and completion detection are handled by downstream components. Figure 6:Task-orchestrationagentpromptandoutputschema.The template specifies the scene, instruc- tion, and capability inputs; seven rules for subtask decomposition and entity identification; and the required JSON output format. Each output entry pairs an executable manipulation instruction with descriptions of its relevant entities. Spatial localization, action generation, and completion detection are delegated to down- stream components. 16 A.3 Shared context and spatial grounding Role of the shared execution context.Orchestra uses the shared execution context as a common interface for executing long-horizon tasks specified by multimodal instructions. The harness maintains execution information separately from the architectures of the action policy and progress evaluator. This organization allows the downstream models to consume the information required by their respective interfaces while the harness manages task interpretation and subtask execution. Thesharedcontextcontainstheactiveinstruction,currentobservations,recentvisualhistory,taskrecords, and grounded entities. At a subtask boundary, MolmoPoint-8B localizes the relevant entities in the head- camera image and SAM3 initializes their masks. At subsequent action-chunk replanning calls, the segmen- tation session tracks those entities. A subtask transition resets the grounding session for the next instruction. Boththeactionpolicyandtheprogressevaluatoruseobservationsfromthesharedcontext. Theymaintain independent model weights and temporal buffers. Spatial representation choices are detailed in AppendixB. Contextinterfacesandinformationflow.The shared context is a logical record rather than a single tensor supplied unchanged to both models. In Algorithm1,Π π andΠ e denote the policy-specific and evaluator- specific input construction functions. Each selects the active subtask, current proprioception, recent camera observations, and spatial conditions fromC t . The model-specific inputs can be written as x π t = Π π (C t ),x e t = Π e (C t ). These adapters select and format context fields for each consumer, separating context maintenance from model-specific input construction. Substituting policies and evaluators.Compatible policies and evaluators can be substituted through their respective adapters while preserving the overall orchestration structure. A replacement policy must provide actions compatible with the robot interface, and a replacement evaluator must provide an output that can be mapped to the controller’s subtask-transition rule. Such substitution may require model-specific preprocess- ing, training, or output conversion. Our experiments evaluate the particular policy and evaluator instantiated in this work. A.4 What the agent remembers Memory contentRoleBoundary Completed-subtask names and start/end frames Explicit execution recordsRetained across subtasks; cleared for a new episode Current plan and active indexIdentify the current and remaining goals Advance at terminal-bin prediction Timestamped multi-view RGB his- tory Supply recent local visual observa- tions Bounded; the next subtask’s local window is reinitialized Grounded entity dataAssociate the active goal with current image regions Recomputed for the next subtask Policy/evaluator observation buffers Construct model-specific temporal input Separate buffers reset by the agent The agent explicitly stores the name and the start and end frames of each completed subtask. These records are retained across subtask transitions and cleared for a new episode. Together, they provide an explicit episode-level execution trace of completed goals and their temporal boundaries. 17 The recent observation history supplies temporal visual inputs to the policy and evaluator. The current proprioceptivestateispassedwiththecurrentobservation. Groundingsessionsandmodel-sidebuffersfollow the active subtask’s lifecycle. A.5 Progress-driven runtime algorithm As shown in Algorithm1, Orchestra executes the subtasks in order and uses predicted progress to determine when to advance to the next subtask. At each replanning call, the policy predicts an action chunk while the evaluator estimates progress from the same observation context. If the predicted category is terminal, the controller discards the chunk and advances the plan. Otherwise, it executes up toH e actions before replanning. Intermediate observations update the RGB histories without triggering grounding or model inference. Episode and observation indices ensure that each image is appended only once. Proprioception is supplied as the current state and is not stored in the visual history. At each subtask transition, the controller resets the local histories and grounding, while retaining completed-subtask records for bookkeeping. The evaluator assesses the observed subtask stage before the newly predicted action chunk is executed. A terminal prediction therefore triggers a subtask transition based on the current observations. After the final subtask, the controller terminates the plan. 18 Algorithm 1Progress-driven execution in Orchestra Require:Instructionℓ, observation stream, policyπ θ , evaluatore φ Ensure:Termination status and completed-subtask recordsD 1:Decomposeℓinto an ordered planG= (g 1 , . . . , g N ) 2:D ←∅;i←1 3:Clear visual history, model buffers, and grounding 4:Record the start frame ofg i 5:whilethe execution budget remainsdo 6:Obtain current RGB images and proprioceptive state 7:Update visual history and model buffers 8:Initialize or update groundingFforg i 9:Construct the shared contextCforg i 10:Run in parallel:A←π θ (Π π (C)),q←e φ (Π e (C)) 11:ifargmax b∈0,...,10 q b = 10then 12:Append(g i ,start frame,current frame)toD 13:DiscardAand any pending actions forg i 14:ifi=Nthen 15:return(PLAN_TERMINATED,D) 16:end if 17:i←i+ 1 18:Clear visual history, model buffers, and grounding 19:Record the current frame as the start frame ofg i 20:Seed the new visual history with the current images 21:continuewith a new inference call forg i 22:else 23:Execute up toH e actions fromAwithin the remaining budget 24:Update visual history and model buffers as observations arrive 25:end if 26:end while 27:return(BUDGET_EXHAUSTED,D) B Policy and Evaluator Implementation B.1 Action-policy architecture The action policy is initialized from theπ 0.5 base checkpoint and combines the Gemma-2B language back- bone within PaliGemma with a Gemma-300M action expert. It predicts action chunks from the active sub- task, spatial conditions, recent visual observations, and current proprioception. Temporal attention reuses spatial-attention parameters, and the temporal positional encodings are fixed buffers. Thus, the memory pathway adds no new learned memory parameters. Training and execution settings are listed in Appendix C. B.2 Entity selection and spatial prompts During training, per-frame active-entity annotations select the objects relevant to the current language sub- task. End-effector entities are excluded. When spatial conditioning is retained, the transform samples one of three representations with weights 0.34, 0.33, and 0.33: 19 RepresentationImage processingText processing ContourOverlay object contours on the cur- rent head-camera image Append entity bounding boxes Bounding boxLeave RGB unchangedAppend named bounding-box coordi- nates PointLeave RGB unchangedAppend named point coordinates Coordinates are quantized to 256 bins. The contour configuration uses green boundaries with blend- ing coeﬀicient 0.4. The policy drops all spatial conditions with probability 0.2 to encourage robustness to missing or inaccurate grounding information. Policy training also uses image dropout with probability 0.1 to encourage greater reliance on grounding information rather than solely on image observations. In the image-drop branch, wrist views are retained with probability 0.9. These augmentations vary the availability of visual and spatial inputs during training. At inference, the evaluation configuration selects contour-plus-bounding-box conditioning deterministi- cally and disables these training dropouts. Grounding is predicted online in real-world execution and pro- vided by the environment in spatially conditioned simulation experiments. Current-frame and historical-image preprocessing.For both the action policy and the progress evalua- tor, contour conditioning replaces the current head-camera image in the temporal window with its contour- overlaid version. Historical frames remain original RGB images. Thus, both models receive the current spatial cue together with unmodified visual history; current contours are not copied onto past frames. The contour representation also supplies the corresponding entity bounding boxes in the language prefix, as spec- ified above. B.3 Temporal window and encoder Figure 7details our instantiation of the space–time factorization used in the policy’s visual encoder. Each camera contributes six observations separated by three observation indices, including the current image. We report the window configuration in observation indices. Frames are embedded spatially before temporal mixing. Temporal attention is inserted after layers 4, 8, 12, 16, and 20 of the visual encoder, using one-based layer numbering. For each patch position, causal atten- tion operates across frame indices, masks unavailable history, and reuses the corresponding layer’s attention parameters. Fixed sinusoidal temporal encodings express frame age. Following layer 20, past-frame tokens are discarded, leaving the current-frame patches for subsequent visual layers. At the documented resolution and patch size, the language model receives 256 visual tokens per cam- era, as in the single-frame baseline. History increases the visual encoder’s work, while leaving this visual prefix count unchanged. Stride changes the sampled temporal span without changing the number of frames processed. 20 Figure 7:Short-horizon visual memory inside SigLIP. (a)Each camera supplies a six-frame window with stride three, spanning observation indicest−15throught. Patch embedding and spatial positional encoding are applied independently to each frame. Four spatial layers (S) alternate with a causal temporal-attention operation (T), giving temporal mixing after layers 4, 8, 12, 16, and 20. After layer 20, historical tokens are discarded and only current-frame features pass through the remaining spatial layers. The VLM receives 256 visual tokens per camera, matching the single-frame baseline.(b)Spatial attention is bidirectional within a frame; temporal attention operates across frames at the same patch index, with causal and valid-history masks. The temporal pathway reuses the corresponding spatial-attention weights and adds fixed sinusoidal age encodings, with zero encoding for the current frame. This adaptation introduces no new learned memory parameters. Patch grids are schematic. Figure 8:Architecture of the progress evaluator.A learned query aggregates the multimodal visual and language prefix, and a normalized linear head predicts one of 11 progress categories. 21 B.4 Progress-evaluator configuration The progress evaluator uses a Gemma-300M language backbone with a visual encoder. A learned query of width 1024 aggregates information from the multimodal prefix. Layer normalization and a linear projection produce the 11 progress logits. The evaluator uses the same six-frame, stride-three observation window as the policy and has independent weights. The deployed subtask transition uses only Eq. equation7, while Eq. equation6specifies the progress classification objective. B.5 Automatically generated progress targets and transitions Progress targets are generated automatically from the step index within each demonstration subtask; no man- ual progress-category labels are required. For a subtask segment ofT i =e i −a i ≥1frames, Eq. equation5 reduces to ρ i (u) = 1− max(e i −1−u,0) T i .(9) The first frame hasρ i = 1/T i , and the last hasρ i = 1. An 11-way label is obtained by rounding10ρ i and clipping it to[0,10]. Thus the final bin includes a terminal interval near the end of the demonstration, rather than exclusively its final frame. In particular, values above 0.95 round to bin 10, with exact-half behavior determinedbytheroundingimplementation. Forexample, ina100-framesegment,ρ= 0.96alreadybelongs to the final bin. During deployment, the controller uses the argmax class directly. A prediction of bin 10 triggers the transition. For visualization, an optional expected progress value can be computed as ̄ρ t = 10 ∑ j=0 j 10 q t,j . (10) Relationship between terminal labels and subtask completion.We examined demonstration subtask endings across all tasks and found that completed-state frames occupy approximately the final1/11to1/9 ofthe demonstrationsegments, correspondingto approximately9.1%–11.1%of their duration. These frames depict an already completed subtask. This completed-state interval begins before the terminal-category boundaryinducedbyrounding10ρ i (u), whichliesatapproximatelyρ i (u) = 0.95. Thisobservationsupports the use of the terminal category as a completion target in our demonstration data. C Data Collection and Training C.1 Real-world training data We collected demonstrations for Breakfast Preparation, Fruit Bowl Placement, Number Ordering, and Trash Bin Placement at 20 Hz. Demonstrations from all four tasks were combined for joint training. Collection was performed at the subtask level; Table2reports the amounts using a complete-task accounting unit, where one unit corresponds to one completion of the task. For Number Ordering, collection covered all six permutations of the three numbers, with 24 demonstra- tionsperorderunderthesameaccountingconvention. Thereal-worldannotationpipelineusesQwen3.7-Plus for subtask decomposition, MolmoPoint for object pointing, and SAM3 for segmentation. Progress cate- gories are generated automatically from the step position within each demonstration subtask, as described in Appendix B.5. 22 Table 2:Real-world data collection and evaluation. Collection amounts use complete-task accounting units assembled from subtask-level collection. Evaluation counts are per configuration. TaskCollection units Eval. episodes Eval. length Breakfast Preparation120204 Fruit Bowl Placement120203 Number Ordering144203 Trash Bin Placement120205 C.2 RMBench-Compose training data For each environment, letmdenote the number of task steps required to complete one operation for ev- ery object in the environment. We use this quantity to set the collection scale: we collected50matomic demonstrations and set the composed evaluation length to2m. Training uses atomic instructions and subtask demonstrations, whereasevaluationsuppliesacompleteinstruction. Table3reportstheenvironment-specific amounts and evaluation budgets. C.3 Training configuration Unless otherwise specified, the action policy and progress evaluator use the same training hyperparameters, including 50,000 training steps and a global batch size of 20 across 2 Nvidia A6000 Pro GPUs. They have independent weights and distinct training objectives: flow matching for action prediction and cross-entropy for progress classification. Architecture-specific settings and action horizons apply only to the corresponding model. SettingConfiguration Policy initializationπ 0.5 base checkpoint Policy language backboneGemma-2B within PaliGemma Policy action expertGemma-300M Maximum text sequence length300 Policy prediction / execution horizons50 / 30 steps Precisionbfloat16 Global batch size (across both GPUs)20 Training steps50,000 OptimizerAdamW, betas 0.9 and 0.95, epsilon10 −8 Peak / final learning rate2.5×10 −5 /2.5×10 −6 Warmup / schedule decay steps1,000 / 30,000 Weight decay / gradient clipping10 −10 / global norm 1.0 D Evaluation Protocols D.1 Composed instructions and execution budgets For each RMBench-Compose environment and evaluated configuration, we ran 100 episodes. Each episode began with a randomly selected complete instruction of the environment’s specified maximum evaluation length. Orchestra received the entire instruction and decomposed it into subtasks before sequential execution. Scene states persisted across subtask boundaries. These experiments use one specified evaluation length per environment. IncompleteOrchestra,theevaluator’sterminal-categorypredictiontriggerssubtaskadvancement. Environment- statecompletionreplacesthislearnedsignalonlyintheoracle-transitionablationinSection 5.4. Environment 23 Table 3:RMBench-Compose collection and evaluation settings.mis the minimum task length used for collectionscaling;n= 2mdefinesadeliberatelylongevaluationsequencedesignedtostress-testthepolicy’s limits in sustained execution. Budgets are maximum executed action steps per subtask, not inference-call counts. Each environment is evaluated for 100 episodes per configuration. EnvironmentmDemos (50m)nBudget Pick Up Object5250 10250 Cover and Uncover Block6300 12300 Press Button3150 6150 Swap Block7350 14300 Place Block on Mat9450 18250 success predicates are used for external scoring and should not be interpreted as an additional switching con- dition in the complete system. Spatially conditioned simulation configurations receive environment-provided grounding. Real-world execution instead uses predicted and tracked grounding. D.2 Metrics and real-world evaluation Atomic Success Rate (Atomic SR) measures isolated atomic-task execution. Subtask Success Rate (SSR) is the fraction of attempted subtasks completed successfully, excluding subtasks not reached after an earlier failure. Average Success Length (ASL) measures the mean number of consecutive subtasks completed in the requested order and is bounded by the instruction length. Cross-environment averages are unweighted arithmetic means. Any subtask failure terminates the corresponding evaluation rollout. For each real-world task, we randomly selected 20 instructions and evaluated both policy configurations on the same instruction set, with one evaluation per selected instruction and configuration. This gives 80 evaluations per configuration across the four tasks. Data collection and robot execution both operate at 20 Hz. Any subtask failure terminates the corresponding evaluation rollout. D.3 Baseline comparability For the RMBench simulation comparison in Table1, we use the oﬀicially reported RMBench results for nine baselines. These comprise DP [Chi et al.,2023], ACT [Zhao et al.,2023],π 0.5 [Physical Intelligence et al., 2025], X-VLA [Zheng et al.,2025], QwenOFT [StarVLA Community,2026], MemER [Sridhar et al.,2025], Mem-0 [ Chen et al.,2026], MemoryVLA (OpenVLA) [Shi et al.,2025], and MemoryVLA (QwenOFT) [Shi et al.,2025]. For HarnessVLA (Codex) [Zhang et al.,2026], we reproduce the baseline on RMBench using the HarnessVLA source code. HarnessVLA uses Codex as its default agent. The table therefore combines oﬀicial benchmark results with our evaluations of HarnessVLA (Codex) and Orchestra. For Orchestra on RMBench, the task-orchestration agent receives the complete episode instruction, the associated task description, and the initial scene image. It generates an ordered sequence of subtask instruc- tions together with descriptions of the relevant entities. For example, for the generated subtask “open the left cover,” the agent identifies “the left cover” as the relevant entity. We then resolve this description to an object instance in the simulator and retrieve its ground-truth segmentation mask to construct the spatial conditions for the downstream action policy and progress evaluator. Thus, the subtask sequence and entity descriptions are generated by Orchestra, whereas the corresponding spatial masks are supplied by the environment. D.4 Physical robot platform 24 RealsenseL515 7-DOF Arm 180°FOV Fisheye Camera Figure 9:Custom dual-arm robot platform used in the real-world experiments. Our real-world experiments use the custom dual-arm robot platform shown in Fig.9. The robot is equipped with two 7-DoF arms and parallel grippers. A head- mounted Intel RealSense L515 camera provides a global viewoftheworkspace, whilea180-degreefisheyecamera mounted on each wrist provides close-range observations during manipulation. Together, these three viewpoints capture both scene-level context and local arm–object in- teractions. All RGB images are resized to224×224be- fore being passed to the models. The proprioceptive state has 16 dimensions, compris- ingsevenjointvaluesandonegrippervalueperarm. Joint actions are represented as offsets from the current state, whereas gripper actions are specified as absolute values. The model pads the action representation to 32 dimen- sions and converts its predictions back to the robot con- trol interface for execution. Both policy configurations use the same observation and action interfaces. E Additional Results and Analysis E.1 Per-environment compositional results Pick up object Cover / uncover block Press button Swap block Place block on mat 0 25 50 75 100 Success rate (%) a Atomic-task success Pick up object Cover / uncover block Press button Swap block Place block on mat 0 25 50 75 100 Success rate (%) b Composed-task subtask success π₀.₅ HarnessVLA Orchestra + π₀.₅ Orchestra (Qwen agent) Orchestra Figure 10:Per-environment success rates on RMBench-Compose. (a)Atomic SR measures isolated atomic-task execution.(b)SSR measures success among attempted subtasks in composed episodes; un- reached subtasks are excluded. Results are shown for all five evaluated configurations and all five environ- ments. Task-level ASL is reported in Fig.3. Figure10compares isolated Atomic SR and composed-task SSR across all five evaluated configurations and environments, complementing the task-level ASL results in Fig.3. For example, the baseπ 0.5 policy achieved98.0% Atomic SR on Swap Block, while its composed-task SSR was53.27% and its ASL was1.14. Orchestra with the same base policy retained98.0% Atomic SR but achieved86.09% SSR and6.19ASL. 25 These metrics describe different evaluation contexts; an isolated-task success rate should not be interpreted as a per-step success probability along a composed rollout. E.2 Ablation implementation details For the no-orchestration ablation in Section5.4, we retrain the policy on long-horizon instructions and cor- responding demonstrations while retaining short-horizon visual memory and spatial grounding. This de- sign removes subtask decomposition while keeping the training and evaluation distributions aligned. The visual-memory ablation disables short-horizon visual memory, and the spatial-grounding ablation omits task-relevant spatial conditions. The oracle-transition configuration replaces the learned completion detector with an environment-state indicator while retaining the rest of the system. E.3 Qualitative visualizations We provide qualitative visualizations of Orchestra on RMBench-Compose and the real-world tasks, comple- menting the quantitative results in Sections5.3and5.5. E.3.1 RMBench-Compose Figure11presents selected execution sequences from the five RMBench-Compose environments. The ex- amples illustrate how successive manipulation subtasks change the scene, with later actions operating on the object configurations produced by earlier actions. Frames are arranged chronologically from left to right and show selected portions of the composed rollouts. Figure 11:Qualitative execution examples on RMBench-Compose.From top to bottom: Pick Up Object, Cover and Uncover Block, Swap Block, Press Button, and Place Block on Mat. Each row shows selected observations from a composed manipulation sequence, with blue arrows indicating temporal progression. The examples visualize object manipulation and scene changes across successive subtasks; they do not depict every step of the full evaluation instructions. 26 (a) Breakfast Preparation 4 subtasks RGB Mask overlay Progress Subtask 1 0 1 0240 Step Subtask 2 0 1 240360 Step Subtask 3 0 1 360570 Step Subtask 4 0 1 570720 Step (b) Fruit Bowl Placement 3 subtasks RGB Mask overlay Progress Subtask 1 0 1 0210 Step Subtask 2 0 1 210420 Step Subtask 3 0 1 420630 Step (c) Number Ordering 3 subtasks RGB Mask overlay Progress Subtask 1 0 1 0210 Step Subtask 2 0 1 210480 Step Subtask 3 0 1 480690 Step (d) Trash Bin Placement 5 subtasks RGB Mask overlay Progress Subtask 1 0 1 0150 Step Subtask 2 0 1 150420 Step Subtask 3 0 1 420600 Step Subtask 4 0 1 600750 Step Subtask 5 0 1 750840 Step Figure 12:Qualitative real-world execution with spatial grounding and progress visualization. (a) Breakfast Preparation (four subtasks).(b)Fruit Bowl Placement (three subtasks).(c)Number Ordering (three subtasks).(d)Trash Bin Placement (five subtasks). Each subtask contains selected RGB observations with pointing markers, corresponding mask overlays, and a progress trace on a normalized scale from 0 to 1. Subtasks are arranged from left to right, and the horizontal axes of the progress plots indicate execution steps. 27 E.3.2 Real-world execution Figure12shows real-world execution examples for Breakfast Preparation, Fruit Bowl Placement, Number Ordering, andTrashBinPlacement. Foreachsubtask, thevisualizationcombinesselectedRGBobservations, spatialmaskoverlays, andawithin-subtaskprogresstrace. Theseviewsillustratetherelationshipbetweenthe activemanipulationgoal, itsgroundedentities, andexecutionprogress. Objectpointingandsegmentationare initialized at subtask boundaries, and the initialized masks are tracked at subsequent action-chunk replanning calls. The plotted progress values provide a visualization of within-subtask evolution; the deployed transition rule uses the terminal-bin argmax decision. E.4 Failure case analysis We analyze 65 real-world failure samples from the complete Orchestra system to identify limitations at dif- ferent stages of execution. These samples include both subtask-level failures and failures of the overall task. In some trials, changes in the environment necessitated a restart; these interrupted trials were excluded from the counted evaluation rollouts but retained in the failure-case analysis. Fig.13presents errors in task de- composition, object localization, segmentation, progress estimation, and action execution, together with their shares of the recorded failures. (a) Task decomposition failure Instruction Place all objects shown in the image into the trash bin. Given the available robot skills, decompose this task into executable subtasks and output the resulting subtask sequence. Generated subtask sequence 1. Place the orange can inside the trash bin. 2. Place the white-and-blue box inside the trash bin. 3. Place the yellow-edged snack bag inside the trash bin. 4. Place the yellow-edged snack bag inside the trash bin. 5. Place the plastic bottle with a green bandinside the trash bin. Repeated object in subtask 4; the white-and-green box is omitted. Correct subtask 4: Place the white-and-green box inside the trash bin. (b) Incorrect object localization Instruction: "Point to the snack bag with yellow edge." Predicted locationIntended location The predicted point selects a different object, rather than the requested yellow-edged snack bag. (c) Incomplete segmentation The mask covers only a local patch of the central target region on the blue mat, leaving the rest unsegmented. (d) Progress estimation failure Observed execution The bread has been placed, but completion is not recognized in time to trigger the next subtask. 210240270300330360390420 0.0 0.5 1.0 Predicted progress Step (e) Action execution failures Placement offset The fork misses its designated square support. Incorrect grasp pose The robot fails to grasp the bottle with the correct pose. Collision during grasping The gripper knocks the apple away during the grasp attempt. Toppling during placement The number plate falls over during placement. (f) Failure distribution Share of recorded failures Task decomposition 1.5% Localization 4.6% Segmentation 0.0% Progress estimation 6.1% Action execution 87.7% Figure 13:Failure analysis of Orchestra in real-world manipulation. (a)Task decomposition repeats the yellow-edged snack bag and omits the white-and-green box.(b)Object localization selects a different object from the requested snack bag.(c)Segmentation captures only part of the central target region on the blue mat.(d)The evaluator fails to recognize completion promptly after bread placement, delaying the subtask transition.(e)Action execution fails through inaccurate placement, an incorrect grasp pose, unintended contact, or unstable placement.(f)Distribution of 65 recorded failures of the complete Orchestra system across the five categories. 28 Task decomposition.The orchestration agent can produce individually executable subtasks that collec- tively fail to cover the instruction. In Fig.13(a), the instruction requires placing all objects in the trash bin, but the generated sequence includes the yellow-edged snack bag twice and omits the white-and-green box. Executing this sequence successfully would therefore still leave the overall task incomplete. This case highlights the need to check object coverage and duplicate entity assignments when constructing a plan. Objectlocalizationandsegmentation.Spatial grounding can fail through either incorrect entity selection or incomplete region extraction. In Fig.13(b), the localization module points to another object instead of the requested yellow-edged snack bag, providing spatial context inconsistent with the active instruction. In Fig.13(c), thesegmentationoutputcoversonlyalocalpatchofthecentraltargetregiononthebluemat. These examples show that obtaining a valid point or a nonempty mask does not guarantee semantically correct and spatially complete grounding. Progressestimation.Physicalcompletionandtheevaluator’scompletiondecisioncandiverge. InFig.13(d), the robot has placed the bread, but the evaluator does not recognize completion in time to trigger the next subtask. The controller advances only when the predicted progress category is the terminal bin. This fail- ure illustrates how delayed completion detection can prevent a locally successful action from advancing the overall task. Action execution.Fig.13(e) illustrates limitations in grasping and placement. The robot places the fork away from its designated square support, approaches the bottle with an incorrect grasp pose, knocks the apple away during a grasp attempt, and topples the number plate during placement. These failures indicate that task orchestration and spatial conditioning do not eliminate the need for precise motion generation and stable contact handling. Failure distribution and implications.As shown in Fig.13(f), action execution accounts for87.7% of the 65 recorded failures, followed by progress estimation (6.1%), localization (4.6%), and task decomposition (1.5%). Segmentation accounts for0.0% in this tally, despite the incomplete-mask example in panel (c); observing a segmentation defect does not by itself establish that it caused a task failure. Within this evalua- tion, the distribution identifies action execution as the dominant remaining limitation, while the qualitative examples also motivate better plan validation, grounding checks, and completion detection. 29

Instant research discovery