Paper deep dive
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning
Yuxing Long, Lei Kang, Ziyan Yu, Yuzheng Gao, Bin Cheng, Jiyao Zhang, Xiaoqi Li, Haolin Yang, Dongjiang Li, Hui Shen, Hao Dong
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 3:10:40 AM
Summary
This paper introduces MAGE, a scalable data synthesis pipeline that uses a Hierarchical Appliance Graph (HAG) to generate training data from appliance manuals. Using MAGE, the authors create UseAppliance, a large-scale dataset for manual-grounded appliance manipulation. They also propose AppliancePlan, an end-to-end model that leverages this dataset to achieve state-of-the-art performance on the RealAppliance-Bench, demonstrating effective sim-to-real transfer in household robotics.
Entities (8)
Relation Signals (6)
MAGE → creates → UseAppliance
confidence 98% · With MAGE, we build UseAppliance, the first large-scale dataset
AppliancePlan → trainedon → UseAppliance
confidence 97% · Built on UseAppliance, we develop AppliancePlan
AppliancePlan → evaluatedon → RealAppliance-Bench
confidence 96% · On RealAppliance-Bench, AppliancePlan... achieves over 10x the best baseline
MAGE → uses → Hierarchical Appliance Graph
confidence 95% · MAGE... introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate... data
AppliancePlan → outperforms → GPT-5
confidence 90% · AppliancePlan... consistently outperforms proprietary MLLMs such as GPT-5
AppliancePlan → builton → Qwen2
confidence 85% · AppliancePlan is built on Qwen2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10x the best baseline on open-loop planning and consistently outperforms state-of-the-art models across all tasks. Real-robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics.
Tags
Links
- Source: https://arxiv.org/abs/2608.15863v1
- Canonical: https://arxiv.org/abs/2608.15863v1
Trouble viewing inline? Open PDF directly →
Full Text
58,154 characters extracted from source content.
Expand or collapse full text
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning Yuxing Long ∗ Center on Frontiers of Computing Studies, School of Computer Science, Peking University Lei Kang ∗ Center on Frontiers of Computing Studies, School of Computer Science, Peking University Ziyan Yu Center on Frontiers of Computing Studies, School of Computer Science, Peking University Yuzheng Gao Center on Frontiers of Computing Studies, School of Computer Science, Peking University Bin Cheng Beijing University of Aeronautics and Astronautics Jiyao Zhang Center on Frontiers of Computing Studies, School of Computer Science, Peking University Xiaoqi Li Center on Frontiers of Computing Studies, School of Computer Science, Peking University Haolin Yang Center on Frontiers of Computing Studies, School of Computer Science, Peking University Dongjiang Li Jingdong Technology Information Technology Co., Ltd Hui Shen Jingdong Technology Information Technology Co., Ltd Hao Dong † Center on Frontiers of Computing Studies, School of Computer Science, Peking University hao.dong@pku.edu.cn Abstract Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manip- ulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop ad- justment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10×the best baseline on open-loop plan- ning and consistently outperforms state-of-the-art models across all tasks. Real-robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics. ∗ These authors contributed equally to this work. † Corresponding author. This work is licensed under a Creative Commons Attribution 4.0 International License. M ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3836196 CCS Concepts • Computing methodologies→Robotic planning; Vision for robotics;• Computer systems organization→ Robotics. Keywords Appliance manipulation planning, household robotics, multimodal foundation models ACM Reference Format: Yuxing Long, Lei Kang, Ziyan Yu, Yuzheng Gao, Bin Cheng, Jiyao Zhang, Xiaoqi Li, Haolin Yang, Dongjiang Li, Hui Shen, and Hao Dong. 2026. Scal- ing Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning. In Proceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3836196 1 Introduction Enabling robots to operate household appliances is a key prereq- uisite for practical home autonomy. Appliance manipulation is substantially harder than generic pick-and-place: appliances vary widely in interface layout, operation logic, and state transitions, and successful execution requires long-horizon, state-dependent procedures that must remain robust to disturbances. Humans address this by consulting manuals—an approach that naturally decomposes the task into three sub-capabilities: appliance part grounding, open-loop manipulation planning, and closed-loop planning adjustment. RealAppliance-Bench [9] provides a unified evaluation protocol for these capabilities, pairing 100 real appliance manuals with high-fidelity digital assets. Yet existing MLLMs and arXiv:2608.15863v1 [cs.RO] 16 Aug 2026 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilLong et al. ...... Planning Adjustment for Button Interference Touch Control Panel Operation Planning Adjustment for Door Interference Accurate Knob Angle Control ...... ...... ...... Figure 1: Capabilities of our AppliancePlan model. Left: AppliancePlan plans manipulation for diverse appliances and updates the plan when interrupted. Right: AppliancePlan outperforms strong large-model baselines on RealAppliance-Bench [9]. embodied planning models still perform poorly on RealAppliance- Bench. Prior methods either focus on simple interactions (e.g., press- ing a single button) [4,14,20] or rely on multi-stage zero-shot pipelines built upon proprietary models, incurring high inference latency and compounding errors [17,20,30]. The fundamental bottleneck is the lack of large-scale, high-quality training data tailored to manual-grounded appliance manipulation. To address this bottleneck, we introduce MAGE (Manual-grounded Appliance data GEneration pipeline), a scalable data synthesis pipeline built on a novel Hierarchical Appliance Graph (HAG). HAG represents each appliance manual as a four-level hierarchy—from document and page structure down to individual parts and state- transition graphs. MAGE collects and classifies manuals via au- tonomous agents to form the upper layers of each HAG, then com- pletes the lower layers through model-assisted annotation. With the HAG in place, data generation reduces to combinatorial graph traversal—sampling state pairs for plan synthesis and injecting dis- turbances for closed-loop recovery—with human verification at every stage. This yields three data types: (i) multi-view bounding- box for part grounding; (i) long-horizon manipulation plans with systematic state-space coverage; and (i) planning-aligned obser- vation images with closed-loop recovery data. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, with four key properties: (i) high quality—Core annotations are human-labeled and double-verified; (i) rich diversity—spanning 22 appliance cate- gories from manufacturers worldwide with diverse control inter- faces and operation procedures; (i) multi-task coverage—providing unified supervision for all three capabilities with 89K+ part bound- ing boxes, 53K+ manipulation tasks, and 33K+ closed-loop adjust- ment steps; and (iv) extensibility—the MAGE pipeline is fully de- coupled from specific appliances, allowing new training data to be generated from any appliance manual. Furthermore, we propose AppliancePlan, a unified end-to-end model for manual-based appliance manipulation planning. Beyond the three main planning objectives, AppliancePlan incorporates three complementary auxiliary tasks—bidirectional manual-part alignment, key-step action prediction, and part state judgment—that provide targeted supervision for spatial grounding, parameter preci- sion, and state-aware plan revision, respectively. On RealAppliance- Bench, AppliancePlan with only 7B parameters consistently outper- forms proprietary MLLMs such as GPT-5 and embodied planning baselines across all tasks: it achieves over 10×the best baseline on open-loop planning (31.36% vs. 2.68% task success) and leads the most realistic sequential setting by a wide margin (28.07% vs. 4.08%). As shown in Figure 1, real-robot experiments on six house- hold appliances further confirm effective physical transfer, with 40.00% task success compared to 3.33% for GPT-5. In this work, our main contributions are: •MAGE, a novel scalable data synthesis pipeline based on a Hierarchical Appliance Graph (HAG) representation, which automatically generates part grounding, long-horizon plan- ning, and closed-loop recovery data at scale from appliance manuals with strategic human verification. Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified PlanningMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil •UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, spanning 22 appliance cat- egories with 89K+ part annotations, 53K+ appliance manip- ulation tasks, and 33K+ closed-loop adjustment steps. • AppliancePlan, the first end-to-end model for long-horizon appliance manipualtion planning, surpassing all baselines on every RealAppliance-Bench track—notably over 10×on open-loop planning (31.36% vs. 2.68% task success). Real- robot experiments further confirm substantial gains (40.00% vs. 3.33% for GPT-5). 2 Related Work 2.1 Appliance Manipulation Enabling robots to operate household appliances has been explored along two directions. The first targets direct interaction with house- hold devices [4,14,20], but mainly handles short-horizon behaviors (e.g., opening a microwave door) without manual-grounded opera- tion. The second direction explicitly uses manuals for action plan- ning. CheckManual [17] and ApBot [30] show the value of manual information, but both rely on multi-stage zero-shot pipelines with cascaded external modules, incurring high latency and compound- ing errors. ApBot’s action space is further limited to direct state transitions and incremental adjustments, precluding single-step prediction of continuous parameters such as rotation angles. Our work addresses this gap with large-scale training data containing fully parameterized atomic actions and a unified end-to-end model. 2.2 Large Models in Embodied Planning Multimodal foundation models [2,13,15,18,22] have greatly im- proved long-context understanding and multimodal reasoning, spurring embodied planning research. On the action execution side, Say- Can [1] combines language models with skill affordances for grounded decision making, PaLM-E [7] integrates multimodal perception and language modeling in a single embodied model, RT-2 [5] casts ro- bot actions as text tokens to enable transfer from web-scale data, and OpenVLA [12] makes large-scale VLA training broadly acces- sible. On the planning side, RoboBrain [11] and RoboBrain 2.0 [25] leverage embodied planning data [6] to improve long-horizon plan generation with stronger spatial and temporal reasoning. However, these models are evaluated primarily on generic object manipula- tion and fall short on manual-grounded appliance operation. The fundamental bottleneck remains the lack of large-scale, high-quality training data tailored to these capabilities. 3 MAGE: Data Synthesis Pipeline The fundamental bottleneck for manual-grounded appliance ma- nipulation is the lack of large-scale training data. We introduce MAGE (Manual-grounded Appliance data GEneration pipeline), a scalable data synthesis pipeline built on a novel Hierarchical Appli- ance Graph (HAG). MAGE automates both HAG construction and data generation: the first two stages (Secs. 3.2–3.3) build each HAG with automated collection agents and human-verified annotation, while the latter two (Secs. 3.4–3.5) consume the completed HAG to synthesize appliance manipulation planning data with human verification gates after each stage (Figure 2). 3.1 Hierarchical Appliance Graph Formulation We formalize appliance manual knowledge as a Hierarchical Ap- pliance Graph (HAG)G 푚 = (V,E), where nodesVrepresent semantic entities at four granularity levels and edgesEencode containment, reference, and state-transition relations. A HAG organizes knowledge into four layers: (i) Document layer: the manual root푣 푚 representing the entire document; (i) Page layer: typed page nodes푣 (푘) 푝 connected to the root by containment edges, such as product-overview and operating-instructions pages; (i) Part layer: part nodes푣 (푗) 푐 linked to their describing pages, each annotated with a visual bounding box in the corresponding appli- ance view; (iv) State layer: state nodes푣 (푖) 푠 representing discrete or continuous part configurations, such as knob angles and button states. Directed action edges푣 푠 푎 −→ 푣 푠 ′ encode which atomic action 푎 transitions the part from state 푠 to 푠 ′ . This formulation provides three useful properties: (1) Combi- natorial task coverage—sampling initial–goal state pairs yields diverse tasks; (2) Locally verifiable annotations—each layer can be independently audited; (3) Domain-agnostic extensibility— the four-layer schema generalizes to any product manual. 3.2 Appliance Manual Collection This stage builds the document and page layers of each HAG. We design a VLM-powered manual collection agent that operates in a ReAct-style loop [29] to autonomously search manufacturer websites and manual databases, download candidate manuals, and filter out low-quality ones based on page count, file size, and reso- lution, each instantiated as a document-layer node푣 푚 . The agent then classifies each page into one of nine semantic types (product overview, operating instructions, etc.) to populate the page layer. 3.3 Appliance Part and State Annotation This stage completes the part and state layers of each HAG. Part Grounding Annotation. Since manual diagrams and real ap- pliance appearances differ substantially, we construct multi-view observation images to bridge this domain gap. For manuals with clear cover photos, we use an open-vocabulary object detector [16] with the appliance category as the prompt to detect the appliance region; if a high-confidence bounding box is returned, we crop it from the cover and then synthesize three canonical views (front, oblique front-side, and oblique top-down) [28]. Human annotators label visible parts with bounding boxes on these views guided by product-overview pages, further enriched with spatial augmenta- tion. State & Transition Annotation. For each part node푣 (푗) 푐 , we assign a state-type label and construct a directed state-transition graph within the state layer. A VLM reads the relevant manual pages and proposes candidate states together with inter-state transitions; human annotators verify and correct the proposals against the manual. All parts are unified into discrete state graphs: binary parts (e.g., doors) yield two-node graphs with complementary action edges, discrete parts (e.g., mode selectors) are enumerated by mode with button-press edges, and continuous parts (e.g., knobs) are discretized at manual-specified granularity with rotation edges. Edges may be unidirectional when the manual specifies irreversible M ’26, November 10–14, 2026, Rio de Janeiro, BrazilLong et al. 2. Appliance Part and State Annotation 1. Appliance Manual Collection Product Overview Safety Information Operating Instructions Troubleshooting 4. Observation Generation and Interruption Injection 3. Manipulation Task and Planning Creation Appliance Multi-view PhotosAppliance Part Bbox & Label “There is a mixer and a plate of food_name on the table. Please Blend food_name at speed speed with the mixer for time.” “speed_lever_manip”: “OFF/0”: “0”, “STIR”: “STIR”, “Speed 2”: “2”, “Speed 4”: “4”, “Speed 6”: “6”, “Speed 8”: “8”, “Speed 10”: “10” ...... Step 1 Lift(motor head) Step 5 Lower(motor head) Step 2 Pick(watermelon) Step 6 Set(speed lever, 2) Step 1 - Lift(motor head) Step 2 - Pick(watermelon) Step 3 - Move(watermelon, plate, bowl) Step 4 - Place(watermelon, bowl) Step 5 - Lower(motor head) Step 6 - Set(speed lever, 2) Step 7 - Wait(2min) Step 8 - Lift(motor head) Close-loop Step Lift(motor head) ... ... Close-loop Step Lower(motor, head) “There is a mixer and a plate of watermelon on the table. Please Blend watermelon at speed 2 with the mixer for 2min.” Step 1 - Lift(motor head) Step 2 - Pick(food_name) Step 3 - Move(food_name, plate, bowl) Step 4 - Place(food_name, bowl) Step 5 - Lower(motor, head) Step 6 - Set(speed control lever, speed_lever_manip_value) Step 7 - Wait(time) Step 8 - Lift(motor head) Cover Page Cropped Appliance Photo Grounding DINO [banana, dough, watermelon, kale, spinach ...] Appliance Part Candidate States Hierarchical Appliance Graph (HAG) Motor Head is lowered down Motor Head is lifted No Disturbance No Disturbance Hierarchical Appliance Graph Food List Document Root Page 1 Page 2 Page Layer Page 3 Part Layer ... Motor Head Speed Lever Knob State Layer Lifted Lowered Speed 0 Speed 2 Speed 8 ... Clockwise 0° Clockwise 180° Set lower Lift Speed 10 Set Rotate Clockwise Rotate Anti-clockwise Document Layer Step 3 Move(watermelon, plate, bowl) Time List [30s, 1min,2min, 3min, 4min, ...] LLM-based Rewriter ... “There is a mixer and a plate of watermelon on the table . Task Description Appliance Photo + Figure 2: Overview of MAGE. The first two stages (Secs. 3.2–3.3) construct the Hierarchical Appliance Graph (HAG) via automated collection and human-verified annotation; the latter two (Secs. 3.4–3.5) consume the completed HAG to synthesize three types of data—part grounding annotations, open-loop manipulation plans, and observation images with closed-loop recovery data. operations. The resulting state nodes푣 (푖) 푠 and directed action edges 푣 푠 푎 −→ 푣 푠 ′ complete the HAG for downstream plan synthesis. 3.4 Manipualtion Task and Planning Creation This stage consumes the completed HAG to synthesize open-loop task– plan pairs, producing long-horizon manipulation planning data. The key idea is to treat task generation as structured graph traversal: each task corresponds to a path in the state-transition graph, and plan generation is driven by traversing the state-transition graph and instantiating templates with sampled parameters (Figure 2). Task and Plan Templates. For each appliance, we manually de- sign a set of task and plan templates based on its manual and the corresponding HAG. Task templates contain typed placeholders for the target state, duration, and interacting objects (e.g., “Heat the object for duration at target_state”). Plan templates are ordered sequences of atomic actions drawn from the action vocabulary defined in RealAppliance-Bench, including appliance manipulation (Press,Rotate, etc.), object manipulation (Pick,Place, etc.), and timing (Wait). Every appliance action spec- ifies the target part and an operation placeholder (rotation angle, press count, etc.); every object action specifies the target object and relevant locations;Waitonly takes a duration parameter. Task and plan templates are paired: each task template maps to a corre- sponding plan template that realizes the described goal, with shared parameters kept consistent. Template-based Generation. Given a task–plan template pair, we generate concrete instances by sampling an initial–goal state pair Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified PlanningMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil a. Appliance Part Category Distributionb. Action Transition Flow Figure 3: Statistics of UseAppliance. (a) Long-tail distribution of 89K+ part bounding boxes. (b) Action-transition flow of 53K+ open-loop planning tasks. (푠 0 ,푠 ∗ )from the HAG state layer and drawing task attributes (inter- acting objects, durations, etc.) from appliance-specific lists. These jointly fill the task template placeholders. The plan template already defines the action sequence skeleton; we fill operation placehold- ers from the state difference between푠 0 and푠 ∗ together with the sampled task attributes, yielding an open-loop plan: 푎 푡 =T plan 푡 Δ(푠 0 , 푠 ∗ ), A , 푌 표 = 푎 1 , 푎 2 , . . ., 푎 푇 ,(1) whereT plan 푡 denotes the푡-th step of the plan template,Δ(푠 0 ,푠 ∗ ) encodes the state difference (rotation angle훼 ∗ − 훼 0 , discrete step count, etc.), andAdenotes the sampled task attributes (objects, durations, etc.). An LLM rewrites the filled task templates into diverse natural-language instructions while preserving semantic fidelity. Uniform sampling of(푠 0 ,푠 ∗ )pairs ensures systematic state- space coverage, avoiding the bias of human demonstrations toward common configurations. 3.5 Observation Generation and Interruption Injection This stage generates planning-aligned observation images and injects structured disturbances to produce closed-loop recovery data. Visual State Prediction. We use Seedream 4.0 [24] as the genera- tive world modelM gen to iteratively generate observation images. Given the appliance photo and task description, we first generate the initial observation퐼 0 ; at each subsequent step푡we convert the action푎 푡 into a textual prompt describing the expected visual change and generate: 퐼 푡+1 =M gen (퐼 푡 , prompt(푎 푡 )),(2) producing an observation sequence(퐼 0 ,퐼 1 , . . .,퐼 푇 )aligned with the action sequence. We apply cross-step consistency checking and human verification to mitigate visual drift. Closed-loop Interruption Injection. To create closed-loop adjust- ment supervision, we inject hypothetical disturbances into inter- mediate steps of the generated observation sequences. We define three categories of interruptions aligned with the three part types: (i) binary-state errors—a part unexpectedly changes state; (i) but- ton mis-touches—an incorrect number of presses is executed; and (i) rotation deviations—a knob is turned to an incorrect angle. For each injected interruption at step푡, we generate a post- disturbance observation image퐼 ′ 푡 using the generative world model and annotate the corrective action푎 corr 푡+1 conditioned on the task context and disturbed observation. This yields closed-loop training triples(퐼 ′ 푡 , context,푎 corr 푡+1 )that teach the model to detect deviations and generate appropriate recovery actions. 4 UseAppliance Dataset Applying the MAGE pipeline to 22 appliance categories, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning. 4.1 Key Characteristics As the first large-scale training dataset for manual-grounded appli- ance manipulation planning, UseAppliance offers four key charac- teristics: •High quality—All part bounding boxes and state-transition graphs are human-labeled; task and plan templates are man- ually authored per appliance; every annotation passes at least one round of independent human verification. •Multi-task coverage—The dataset provides unified supervi- sion for part grounding with 89K+ bounding boxes, open- loop planning with 53K+ task–plan pairs, and closed-loop adjustment with 33K+ recovery steps, enabling joint training of all three capabilities. •Rich diversity—Spanning 22 appliance categories from man- ufacturers worldwide, UseAppliance covers a wide range of control interfaces such as knobs, buttons, touch panels, and levers, as well as diverse operation procedures including heating, blending, and brewing. •Extensibility—Since the MAGE pipeline takes only an ap- pliance manual as input, new appliance categories can be onboarded without modifying the pipeline itself. 4.2 Data Statistics Quantitative statistics are summarized in Table 1. As shown in Fig- ure 3(a), part annotations follow a long-tail distribution: common parts (e.g., doors, buttons, knobs) receive dense annotations while appliance-specific controls appear infrequently, encouraging robust- ness to rare structures. Figure 3(b) visualizes the action-transition flow across appliance categories: tasks average 8.15 open-loop steps, with action distributions varying significantly by category—e.g., M ’26, November 10–14, 2026, Rio de Janeiro, BrazilLong et al. knob-heavy appliances are dominated byRotate, while panel- based appliances rely more onPress—confirming that the dataset captures diverse operation patterns. Table 1: Summary statistics of UseAppliance. MetricValue Total # Appliance manuals2,120 Total # appliance manual pages50,727 Total # Appliance parts1,584 Total # Part bounding boxes89,088 Total # Appliance manipulation tasks53,854 Average # Open-loop manipulation steps8.15 Median # Open-loop manipulation steps8 Total # Close-loop manipulation steps33,815 5 AppliancePlan Model 5.1 Model Architecture AppliancePlan is built on Qwen2.5-VL-7B-Instruct [3]. Given visual inputsI= 퐼 1 , . . .,퐼 푡 (manual pages and observation images) and a textual task instruction퐿, the vision encoder and vision- language merger project the inputs into the LLM token space, and the language model predicts the output autoregressively: 푃 휃 (푌 | 퐻 푣 ,푋)= 푁 Ö 푛=1 푃 휃 ( 푦 푛 | 푦 <푛 ,퐻 푣 ,푋 ) .(3) where퐻 푣 denotes the projected visual tokens,푋= Tok(퐿)the text tokens,푁the output sequence length, and푌= (푦 1 , . . .,푦 푁 )the output sequence. 5.2 Model Training AppliancePlan is trained with 3 main objectives and 3 auxiliary objectives constructed from UseAppliance. All objectives are for- mulated as next-token prediction and optimized with cross-entropy loss. 5.2.1 Main Planning Objectives. The three main objectives corre- spond to the three core capabilities. LetD 푔 ,D 표 , andD 푐 denote the training splits for part grounding, open-loop planning, and closed-loop adjustment, respectively. Appliance Part Grounding. This objective bridges the domain gap between symbolic part descriptions in manuals and their pixel-level appearance in observation images. Given a multimodal input u 푔 = (I manual ,퐼 obs ,퐿 part ), the model predicts the bounding-box coordinate sequence b=(푥 1 ,푦 1 ,푥 2 ,푦 2 ). L 푔 =E (u 푔 ,b)∼D 푔 − log푃 휃 b | u 푔 .(4) Open-loop Manipulation Planning. While grounding localizes where to act, this objective trains the model to reason about what to do and in what order. From the manual pagesI manual , an initial ob- servation퐼 0 , a task instruction퐿 task , and the atomic action setA, the model generates the full open-loop plan푌 표 =(푎 1 (휓 1 ), . . .,푎 푇 (휓 푇 )) as a single autoregressive sequence, with one parameterized action per line. L 표 =E (u 표 ,푌 표 )∼D 표 " − |푌 표 | ∑︁ 푛=1 log푃 휃 푦 표 푛 | 푦 표 <푛 , u 표 # ,(5) where u 표 =(I manual ,퐼 0 ,퐿 task ,A) denotes the full multimodal input. Closed-loop Planning Adjustment. This objective trains the model to predict corrective actions by comparing the expected plan with the actual execution state. At step푡, the input comprises the current observation퐼 푡 , the task instruction퐿 task , the reference plan 푃 1:푇 , and the executed action history퐴 exec 1:푡 ; the model predicts the next corrective action 푎 ★ 푡+1 : L 푐 =E (u 푐 ,푎 ★ 푡+1 )∼D 푐 − log푃 휃 푎 ★ 푡+1 | u 푐 ,(6) where u 푐 =(퐼 푡 ,퐿 task ,푃 1:푇 ,퐴 exec 1:푡 ). The overall main objective combines these three losses: L main =L 푔 +L 표 +L 푐 .(7) 5.2.2 Auxiliary Training Objectives. The three auxiliary objectives target complementary dimensions of the planning problem. Let D align ,D key , andD state denote the corresponding training splits. Manual-Appliance Part Alignment. The main grounding objec- tive only maps part names to locations. This auxiliary objective strengthens bidirectional part–manual correspondence with three complementary sub-objectives, all receiving manual pages and an observation image as input: (i) Manual-side part identification—the target part is highlighted by a bounding box b on the observation image; the model predicts the part name푛 ★ ; (i) Observation-side part identification—the target part is indicated by its center-point co- ordinate c; the model predicts푛 ★ ; (i) Part localization—a part name 퐿 part is provided; the model predicts the center-point coordinate c on the observation image. The alignment loss is: L align =E D align − log푃 휃 (푛 ★ | u mid )−log푃 휃 (푛 ★ | u oid )−log푃 휃 (c| u loc ) , (8) where u mid = (I manual ,퐼 obs ,b), u oid = (I manual ,퐼 obs ,c), and u loc = (I manual ,퐼 obs ,퐿 part ). Key-Step Action Prediction. This objective focuses supervision on steps involving precise action parameter prediction (e.g., rotation angles, button-press counts). We extract such steps from open-loop plans to form a key-step subsetD key and train the model with three complementary prediction targets: (i) Next-step prediction—from manual pages, the current observation, the task instruction, and the executed action history, the model predicts the next key action; (i) Remaining-step prediction—with the same context, predict all remaining actions from the current step; (i) Parameter VQA—the model answers a parameter question (e.g., target angle, press count) based on the current observation. L key =E (u key ,푎 ★ key )∼D key h − log푃 휃 푎 ★ key | u key i ,(9) where u key =(I manual ,퐼 푡 ,퐿 task ,퐴 1:푡 ). Part State Judgment. This objective provides explicit state-recognition supervision through VQA: the model receives manual pages, an observation image퐼 obs , and a question about a specific part’s state, and predicts the ground-truth state label 푠 ★ . L state =E (u state ,푠 ★ )∼D state − log푃 휃 푠 ★ | u state ,(10) Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified PlanningMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil AppliancePlan Model ... Locate “Attachment Knob”... “bbox_2d”: [x1, y1, x2, y2] ... Please Blend watermelon at speed 2 with the mixer for 2min. Step 1 - Lift(motor head); Step 2 - Pick(watermelon); Step 3 - Move(watermelon, plate, bowl); Step 4 - Place(watermelon, bowl); Step 5 - Lower(motor, head); Step 6 - Set(speed control lever, 4); Step 7 - Wait(2min); Step 8 - Lift(motor head); ... Blend watermelon at speed 2 with for 2min. Grounding Truth: ...... Execution Actions: Lift(motor head); Pick(watermelon); Move(watermelon, plate, bowl); Place(watermelon, bowl); Lower(motor, head); What should you do next? Lower(motor, head) Appliance Part Grounding Open-loop Manipulation Planning Close-loop Planning Adjustment Attachment Knob“point_2d”: [x, y] Attachment Knob What is the part name? What is the part name? Localize “Attachment Knob” ... Please Blend watermelon at speed 2 with the mixer for 2min. What should do after completing “Lift(motor head); Pick(watermelon); Move(watermelon, plate, bowl); Place(watermelon, bowl); Lower(motor, head); ” ? Set(speed control lever, 4) Wait(2min) Lift(motor head) What is the state of the mixer’s motor head? From the given appliance image, the motor head of the mixer is lifted. Set the speed control lever to 2 Predict Remaining Steps Set the speed control lever to 2 Predict Next Step How to adjust the speed to 2? Appliance Manipulation Planning Ta s ks Manual-Appliance Part Alignment Key-step Action Prediction Part State Judgement Auxiliary Training Tasks Figure 4: Overview of AppliancePlan model. AppliancePlan is an end-to-end model for manual-grounded appliance manipula- tion planning, trained with three main objectives and three auxiliary objectives. where u state = (I manual ,퐼 obs ,퐿 state_query ) and퐿 state_query is the state question. The overall auxiliary objective is: L aux =L align +L key +L state .(11) The full training objective is: L=L main +L aux .(12) 6 Experiments 6.1 Implementation Details Training runs for 96 GPU-hours on NVIDIA H200 with a global batch size of 512, a learning rate of 1×10 −5 with cosine decay, and a frozen vision encoder. We use DeepSpeed ZeRO-3 [23] with bfloat16 mixed precision. All loss weights are set to 1.0. During inference, we use a temperature of 0.1 for stable plan generation. Training uses only UseAppliance, which has no overlap with RealAppliance- Bench, making all evaluations out-of-distribution. 6.2 Evaluation on RealAppliance-Bench 6.2.1 Evaluation Tracks and Metrics. RealAppliance-Bench [9] is a benchmark for manual-grounded appliance manipulation built on 100 high-fidelity digital appliance assets in NVIDIA Isaac Sim, span- ning 14 common appliance categories. Each asset is aligned with its real-world manual in appearance, physical mechanisms, electronic mechanisms, and program logic. We consider the following four evaluation tracks: •Appliance Part Grounding: Given relevant manual pages, observation, and part name, the model localizes the target appliance part by predicting its bounding box. We report average Intersection-over-Union (IoU) and mean Average Precision at IoU 0.5 (mAP@0.5). •Open-loop Manipulation Planning: Given relevant man- ual pages, an initial observation image, a task instruction, and the available atomic action set, the model predicts the full manipulation sequence without feedback during exe- cution. Following the benchmark protocol, we report task completion rate and task success rate. •Closed-loop Planning Adjustment: Given relevant man- ual pages, a task instruction, a reference plan, the executed action history, the current observation image, and the avail- able atomic action set, the model predicts the next corrective action under disturbances. Performance is measured by step- wise success rate. •Sequential Planning and Adjustment: This evaluation- only setting combines the previous two tracks: the model first generates an open-loop plan from the manual, initial observation, task instruction, and available atomic action set, and then revises subsequent actions online from execution M ’26, November 10–14, 2026, Rio de Janeiro, BrazilLong et al. Table 2: Results on RealAppliance-Bench across three standard evaluation tracks. Baseline ModelTotal Appliance Part Grounding (Average / mAP@0.5) Proprietary MLLMs GPT-5 [19]12.71/7.694.84/0.0012.00/8.5116.33/17.8513.23/9.6728.83/28.5720.24/11.9410.42/4.3432.97/33.3310.01/9.674.72/1.3319.17/18.7511.17/9.674.80/0.0012.15/8.59 GPT-5 Mini [19]9.08/7.461.85/0.007.18/3.199.71/0.004.43/3.2210.83/0.0013.95/14.922.45/0.0019.56/0.001.82/0.002.48/0.007.59/6.255.33/0.004.94/0.006.51/3.49 Gemini 2.5 Pro [10]7.77/4.473.75/0.0010.74/8.519.47/10.716.74/6.457.23/0.0010.46/8.955.87/4.3432.82/41.669.70/11.292.92/0.006.64/6.256.70/6.456.16/0.008.16/6.64 Gemini 2.5 Flash [10] 8.21/4.472.46/0.007.94/7.4412.14/17.852.28/3.2216.67/0.0011.22/7.462.18/0.0021.16/16.666.06/4.831.52/1.335.06/0.004.81/6.458.38/0.006.67/5.06 Open-source MLLMs Qwen3-VL 8B Thinking [21]5.55/1.490.25/0.002.14/1.062.02/0.001.11/0.005.29/0.006.09/0.003.93/0.005.25/0.002.52/1.610.40/0.000.00/0.003.03/3.229.93/0.002.92/0.69 Qwen3-VL 8B Instruct [21]2.27/0.000.00/0.000.44/0.000.63/0.001.28/0.000.15/0.002.83/0.001.56/0.004.58/0.000.85/0.000.32/0.000.00/0.001.46/0.008.32/0.001.32/0.00 Qwen3-VL 235B Thinking [21]5.19/2.980.99/0.003.41/3.191.01/0.001.96/0.000.25/0.006.78/0.001.58/0.006.01/0.001.08/0.000.12/0.000.00/0.001.50/0.008.64/0.002.80/0.87 Qwen3-VL 235B Instruct [21]4.58/4.470.00/0.000.61/0.000.61/0.001.15/0.000.46/0.004.64/0.001.61/0.006.28/0.000.86/0.000.00/0.000.00/0.000.44/0.007.35/0.001.75/0.52 GLM-4.1V-Thinking [26]3.88/1.490.00/0.000.54/0.000.98/0.001.67/0.000.42/0.004.01/0.001.55/0.004.71/0.000.93/0.000.00/0.000.00/0.001.33/0.0011.63/0.001.74/0.17 GLM-4.5V [26]3.59/2.980.00/0.000.75/0.000.99/0.002.50/0.000.32/0.004.09/0.001.56/0.006.15/0.000.66/0.000.00/0.000.00/0.000.62/0.0011.00/0.001.76/0.35 Embodied Planning Models RoboBrain 2.0-7B [25]0.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.00 RoboBrain 2.0-32B [25]0.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.00 ManualPlan [17]3.56/0.000.00/0.001.01/0.002.46/0.002.37/0.003.08/0.005.61/0.001.25/0.002.79/0.000.20/0.000.15/0.001.66/0.001.73/0.005.07/0.001.92/0.00 ApBot [30]23.60/28.4010.50/12.308.50/7.604.70/4.5015.70/20.0017.70/20.0010.20/11.200.00/0.004.50/4.3011.70/12.0011.20/14.700.00/0.0012.20/13.604.30/5.3010.60/12.10 AppliancePlan (Ours)13.25/10.4519.03/19.1514.19/12.7735.04/35.7115.25/16.1316.39/14.2919.77/19.7039.65/39.1372.06/66.6727.82/27.4231.98/33.336.77/6.2520.53/19.3539.56/33.3322.96/22.24 - Manual-App. Part Alignment17.74/11.945.77/2.1313.42/10.6418.09/21.4322.43/19.3525.78/14.2923.75/21.2119.56/21.7452.06/58.3318.46/14.5226.93/32.0016.63/18.7514.04/6.4532.73/25.0019.27/17.34 - Key-step Action Prediction26.66/26.8714.50/12.7715.26/11.7027.57/28.5732.56/29.0329.50/42.8625.56/21.2122.94/26.0950.59/58.3321.41/22.5826.35/28.0022.39/18.7514.45/9.6825.44/16.6722.99/21.89 - Part State Judgement24.62/22.3919.69/19.1518.34/15.9624.41/25.0033.44/32.2629.31/42.8629.00/27.2722.98/26.0941.72/33.3318.64/16.1326.59/32.0033.08/31.259.50/3.2320.89/16.6723.45/22.59 Open-loop Manipulation Planning (Task Completion Rate / Task Success Rate) Proprietary MLLMs GPT-5 [19]2.60/1.112.57/0.001.55/0.000.00/0.007.77/2.005.92/0.004.70/0.830.95/0.004.80/2.003.02/0.006.44/2.027.77/0.0014.59/6.0010.66/10.004.30/1.22 GPT-5 Mini [19]2.22/2.225.23/1.250.11/0.000.00/0.003.58/0.001.26/0.003.52/1.666.45/0.004.09/2.002.34/0.004.08/0.005.55/0.009.29/4.009.66/6.663.27/1.02 Gemini 2.5 Pro [10]0.59/0.003.47/1.253.70/1.530.00/0.003.18/0.002.01/0.004.51/4.160.47/0.000.50/0.001.07/0.0010.72/8.088.88/0.0015.80/12.008.00/6.664.08/2.45 Gemini 2.5 Flash [10]0.23/0.002.00/0.006.12/3.840.00/0.002.79/0.001.26/0.002.34/1.667.82/3.332.16/2.002.34/0.8810.55/8.085.55/0.0015.51/12.006.66/6.664.26/2.65 Open-source MLLMs Qwen3-VL 8B Thinking [21] 1.55/1.112.82/0.001.53/1.530.00/0.003.88/2.004.23/3.701.79/1.667.42/3.335.73/2.000.88/0.886.39/5.052.77/0.007.22/4.006.66/6.663.01/1.94 Qwen3-VL 8B Instruct [21]0.00/0.004.41/0.000.76/0.760.00/0.000.00/0.000.00/0.001.66/1.660.00/0.002.00/2.000.88/0.881.68/0.0012.22/0.006.71/4.653.33/3.331.70/0.82 Qwen3-VL 235B Thinking [21]1.56/1.115.95/1.251.59/0.000.00/0.009.46/4.007.83/3.702.08/1.6610.79/3.335.40/2.004.18/0.886.68/4.048.33/0.007.51/2.008.00/6.664.36/1.73 Qwen3-VL 235B Instruct [21]0.00/0.006.86/1.253.23/3.070.00/0.008.01/4.002.32/0.004.50/4.163.14/0.003.13/2.002.47/1.766.58/3.038.88/0.009.95/6.006.66/6.664.11/2.34 GLM-4.1V-Thinking [26]0.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.00 GLM-4.5V [26] 2.22/2.223.30/1.252.41/2.301.11/1.115.21/4.004.44/3.701.30/0.831.33/0.004.43/2.001.50/0.978.41/6.063.33/0.0010.11/8.0012.66/10.003.73/2.68 Embodied Planning Models RoboBrain 2.0-7B [25]0.00/0.000.84/0.000.06/0.000.00/0.000.17/0.000.00/0.000.16/0.000.00/0.000.00/0.000.27/0.000.00/0.000.55/0.000.22/0.000.00/0.000.16/0.00 RoboBrain 2.0-32B [25]0.00/0.001.99/0.000.00/0.000.00/0.000.71/0.000.00/0.000.12/0.000.00/0.000.00/0.000.17/0.000.67/0.000.00/0.001.53/0.000.00/0.000.37/0.00 ManualPlan [17] 6.12/0.008.76/0.002.003/0.000.92/0.003.20/0.005.75/0.0011.27/0.0010.44/0.006.44/0.004.51/1.762.20/0.0011.32/0.0011.50/3.992.17/0.005.61/0.40 ApBot [30] 0.00/0.000.00/0.000.00/0.000.00/0.002.00/2.008.70/7.401.60/0.800.00/0.000.00/0.007.10/7.107.10/7.100.00/0.004.40/4.000.00/0.002.30/2.10 AppliancePlan (Ours)45.70/27.8042.80/23.7048.00/36.2049.00/21.1029.92/14.0043.40/40.7048.20/31.7020.00/13.3041.58/24.0247.30/32.7061.30/49.5073.30/60.0061.82/45.9852.20/13.3047.86/31.36 -Manual-App. Part Alignment45.50/28.9034.30/16.3045.00/34.6048.70/24.4036.68/15.9863.40/55.6036.60/15.8028.70/6.7043.22/23.9853.10/41.6052.80/41.40100.00/100.0066.26/43.9853.70/16.7047.17/30.33 -Key-step Action Prediction36.10/17.8034.40/17.5032.10/21.5046.70/18.9020.68/12.0242.20/40.7036.50/17.5023.40/6.7038.94/14.0253.60/34.5046.60/20.2083.30/70.0069.26/42.0254.60/16.7041.90/22.57 -Part State Judgement45.90/26.7034.00/15.0033.00/25.4054.30/33.3034.70/18.0253.30/51.9041.30/20.808.70/6.7026.28/22.0253.50/41.6051.00/34.3095.00/95.0049.86/27.9851.20/16.7043.67/28.50 Close-loop Planning Adjustment (Step-wise Success Rate) Proprietary MLLMs GPT-5 [19] 28.845.7938.510.0031.7412.5039.436.6641.6630.5520.5810.5234.1445.4529.61 GPT-5 Mini [19] 14.425.1718.240.0015.783.8425.1915.0037.839.0920.5842.100.0027.2716.33 Gemini 2.5 Pro [10]26.1313.0445.620.0042.8511.5338.005.0027.0235.4523.5215.7841.4631.8131.73 Gemini 2.5 Flash [10]31.0610.1434.050.0046.0319.2337.9015.0024.3237.0326.4742.1039.0233.3331.61 Open-source MLLMs Qwen3-VL 8B Thinking [21]23.0713.0430.430.0030.1511.5336.3615.0013.5137.2720.5836.840.0022.7225.58 Qwen3-VL 8B Instruct [21]28.8424.6336.950.0038.0919.2339.8610.008.1038.1823.5242.1026.864.5430.65 Qwen3-VL 235B Thinking [21]22.1114.4933.330.0041.2623.0738.5615.0035.1338.1832.3542.1029.2640.9031.23 Qwen3-VL 235B Instruct [21] 25.0024.6339.130.0042.8526.9235.9410.005.4028.1826.4742.1024.399.0929.13 GLM-4.1V-Thinking [26]0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00 GLM-4.5V [26]0.000.000.740.000.000.000.000.000.000.000.000.000.000.000.12 Embodied Planning Models RoboBrain 2.0-7B [25]27.2725.6734.310.0045.1653.3334.8622.725.2640.7727.7736.8437.039.0931.77 RoboBrain 2.0-32B [25]23.1420.2728.990.0022.5813.3326.3118.187.8918.4419.4426.3125.929.0921.96 ApBot [30]0.802.604.400.0018.2025.003.702.805.809.9016.7026.305.900.007.00 AppliancePlan (Ours)31.4022.9734.910.0054.8456.6742.7618.1831.5863.1122.2231.5825.934.5537.12 - Manual-App. Part Alignment34.7132.4336.090.0053.2320.0043.429.0910.5347.5713.8947.3737.040.0034.67 - Key-step Action Prediction40.5028.3836.090.0051.6153.3348.6827.2721.0540.7838.8942.1137.049.0938.24 - Part State Judgement33.0627.0333.730.0056.4523.3342.7613.6413.1648.548.3347.3737.044.5534.00 history and updated observations. We use task completion rate and task success rate as metrics. The first three tracks are defined by RealAppliance-Bench. The fourth track based on RealAppliance-Bench is introduced by us. GPT 5: Open(busket) GPT 5: wait(2min) GPT 5: pick(bread), place(黑麦面包, 面包机烤槽), rotate(旋钮, 2档, 0°), press(按制, 开始, 1), wait(20s), pick(黑麦面包), place(黑麦面包, 盘子) GPT 5: To p-leftcorner(x, y):(150,180) Width(w):40,Height(h)40 Ours: [420, 709, 524, 828] Ours: pick(黑麦面包), move(黑麦面包, 木桌, 卡槽), place(黑麦面包, 卡槽), rotate(旋钮, 2档, 顺时针旋转45°), push(按置, 1), wait(55s), pick(黑麦面包), move(黑麦面包, 卡槽, 木桌), place(黑麦面包, 木桌) Ours: rotate(temp knob, correcting mistake, rotate counterclockwise 36°) Ours: close(microwave door) GPT Ours Prompt: Given the manual, locate 'on/off' in the appliance observation image and output the bounding box coordinates in JSON format. Prompt: you are a professional document analysis and embodied planning assistant. based on the manual images...answer the question:now there is a toaster and bread on the table, please plan how to use the toaster to heat bread for 55s at level 2 Prompt: Now carry out the closed-loop planning task. Task instruction: .... All planning steps: open(busket), pick(chicken), move(chicken, table, basket), place(chicken, busket), close(busket), rotate(temp knob, 100°C, rotate clockwise 36°), rotate(time knob, 10min, rotate clockwise 90°), wait(10min), open(busket), pick(chicken), move(chicken, basket, table), place(chicken, table). Already taken atomic action: open(busket), pick(chicken), move(chicken, table, basket), place(chicken, busket), close(busket), rotate(temp knob, 100°C, 36°) Prompt: Now carry out the closed-loop planning task. All planning steps: open(door), pick(popcorn), move(popcorn, table, turntable), place(popcorn, turntable), close(door), rotate(power knob, MidHigh, rotate clockwise135°), rotate(time knob, 2min, 40°), wait(2min), open(door), pick(popcorn), move(popcorn, turntable, table), place(popcorn, table). Already taken atomic action:open(door), pick(popcorn), move(popcorn, table, turntable), place(popcorn, turntable), close(door), rotate(power knob, MidHigh, rotate clockwise135°) Appliance Part Grounding Open-loop Manipulation Planning Close-loop Planning Adjustment Close-loop Planning Adjustment Figure 5: Real-robot execution with AppliancePlan. Repre- sentative examples on household appliances. 6.2.2 Comparison with Baselines. Table 2 compares Appliance- Plan with proprietary MLLMs, open-source MLLMs, and embodied planning baselines. Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified PlanningMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 3: Sequential planning and adjustment results based on RealAppliance-Bench. This evaluation-only setting combines open-loop planning with online closed-loop correction. Entries report task completion rate / task success rate. Baseline ModelTotal Sequential Planning and Adjustment (Task Completion Rate/Task Success Rate) Proprietary MLLMs GPT-5 [19]2.70/1.064.42/1.131.68/0.000.00/0.007.07/1.694.70/0.004.39/0.730.69/0.004.44/1.852.38/0.005.90/1.855.18/0.0014.55/7.0114.00/13.334.26/1.35 GPT-5 Mini [19]2.22/2.228.02/2.500.11/0.000.00/0.003.58/0.004.44/0.003.46/1.635.86/0.006.10/4.002.31/0.004.99/0.995.55/0.009.29/4.0016.99/13.333.98/1.51 Gemini 2.5 Pro [10]0.58/0.004.32/1.123.51/1.450.00/0.002.94/0.004.44/0.005.14/4.680.37/0.000.47/0.001.98/0.8011.17/8.415.92/0.0016.48/12.9614.66/13.334.58/2.81 Gemini 2.5 Flash [10] 1.34/1.112.68/0.006.29/3.780.00/0.004.94/2.002.33/0.003.08/2.386.90/2.944.00/3.842.30/0.8611.11/8.735.55/0.0015.20/11.7613.33/13.334.97/3.25 Open-source MLLMs Qwen3-VL 235B Thinking [21]2.64/2.199.24/4.701.57/0.000.00/0.009.46/4.007.83/3.702.82/2.418.99/2.777.25/3.924.03/0.856.61/4.008.33/0.008.97/3.7711.33/10.005.05/2.48 Qwen3-VL 235B Instruct [21]3.33/3.336.86/1.254.00/3.840.00/0.0010.00/6.006.03/3.705.28/4.953.14/0.005.13/4.002.47/1.769.62/6.068.88/0.0019.95/16.0010.00/10.005.84/4.08 Qwen3-VL 8B Thinking [21]1.55/1.116.23/2.501.51/1.510.00/0.008.17/6.004.76/3.701.84/1.657.42/3.338.36/3.920.88/0.887.07/5.052.77/0.0011.22/8.0010.00/10.004.03/2.74 Qwen3-VL 8B Instruct [21]0.00/0.007.26/0.003.07/3.070.00/0.002.00/2.000.00/0.001.66/1.660.00/0.004.00/4.000.88/0.882.69/1.0112.22/0.006.71/4.656.66/6.662.65/1.54 GLM-4.1V-Thinking [26]0.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.000.00/0.00 GLM-4.5V [26] 2.22/2.223.30/1.252.39/2.291.11/1.115.21/4.004.44/3.701.30/0.831.33/0.004.43/2.001.50/0.978.33/6.003.33/0.0010.11/8.0012.66/10.003.72/2.67 Embodied Planning Models RoboBrain 2.0-7B [25]0.00/0.000.00/0.000.06/0.000.00/0.000.00/0.000.00/0.000.16/0.000.00/0.000.00/0.000.27/0.000.00/0.000.52/0.000.20/0.000.00/0.000.08/0.00 RoboBrain 2.0-32B [25]0.00/0.000.41/0.000.00/0.000.00/0.002.43/2.430.00/0.000.00/0.000.00/0.000.00/0.000.00/0.001.96/1.960.00/0.000.00/0.000.00/0.000.37/0.30 ApBot [30]0.00/0.000.00/0.000.00/0.000.00/0.002.000/2.009.20/7.401.30/0.800.00/0.000.00/0.005.30/5.305.10/5.100.00/0.002.40/2.000.00/0.001.80/1.60 AppliancePlan (Ours)43.94/24.4435.23/21.2548.19/36.2246.83/16.6729.46/14.0044.44/44.4440.96/24.1717.92/13.3332.71/10.0045.29/30.0959.45/46.4665.00/50.0055.65/36.0052.17/30.0044.59/28.07 - Manual-App. Part Alignment43.49/25.5631.31/16.2543.87/33.0747.14/22.2234.72/16.0046.46/25.9331.71/12.5023.75/6.6737.21/18.0048.96/35.4051.44/39.3975.00/65.0061.31/34.0053.75/36.6743.45/26.54 - Key-step Action Prediction34.65/14.4430.68/16.2530.48/19.6945.48/17.7820.13/12.0027.41/14.8131.86/14.1721.27/6.6736.29/10.0049.69/30.9746.20/20.2061.67/45.0064.84/36.0054.58/40.0038.79/19.98 - Part State Judgement43.99/23.3330.71/15.0032.91/25.2052.38/30.0031.73/16.0036.40/22.2235.49/15.838.75/6.6723.38/14.0048.37/33.6351.53/35.3570.00/60.0053.62/30.0051.25/36.6740.73/25.10 Appliance Part Grounding. AppliancePlan achieves the best over- all grounding performance (22.96% IoU, 22.24% mAP@0.5), out- performing the strongest embodied baseline ApBot by 12.36 IoU and 10.14 mAP@0.5 points with a simpler end-to-end pipeline. The gains stem from explicit manual-conditioned training: without man- ual context, models cannot reliably associate part descriptions with visual appearance, especially for appliance-specific controls absent from web-crawled data. Open-loop Manipulation Planning. AppliancePlan achieves 47.86% task completion and 31.36% task success—over 10×the best base- line (4.36%/2.68%)—and ranks first on every appliance category. The near-zero baseline performance confirms that general language and vision capabilities do not transfer to structured multi-step planning with precise parameters; training on (initial-state, goal-state, plan) triples with fully parameterized atomic actions bridges this gap. Closed-loop Planning Adjustment. AppliancePlan achieves the highest step-wise success rate (37.12%), outperforming the best open-source baseline RoboBrain 2.0-7B by 5.35 points and the best proprietary MLLM Gemini 2.5 Pro by 5.39 points, indicat- ing stronger robustness to execution deviations across appliance categories. Sequential Planning and Adjustment. This end-to-end setting is the most realistic, as both the initial plan and subsequent corrections are model-generated. AppliancePlan achieves 44.59% completion and 28.07% success, versus 5.84%/4.08% for the best baseline (Qwen3- VL 235B Instruct), confirming that planning and correction gains transfer to long-horizon execution, including recovery from errors introduced by the model’s own predictions. 6.2.3 Ablation of Auxiliary Training Objectives. We ablate the three auxiliary objectives by removing one at a time; results appear in the lower rows of Table 2 (rows marked “−”). Removing Manual- Appliance Part Alignment causes the largest grounding degra- dation, confirming that bidirectional part–manual correspondence is essential for accurate spatial localization. Removing Key-Step Action Prediction produces the largest decline on long-horizon metrics—particularly on the realistic sequential setting—indicating that focused supervision on parameter-critical steps is the primary driver of precise action prediction. Removing Part State Judgment mainly harms closed-loop recovery, validating its role in enabling state-aware plan revision. Overall, each objective contributes to its targeted capability as designed, and their combination yields the best balance across grounding, planning, and closed-loop correc- tion. 6.3 Real-Robot Evaluation To test whether benchmark gains transfer to physical execution, we deploy AppliancePlan on a local workstation with an RTX 4090 GPU and evaluate it on a Franka Emika Panda robot. The system uses an eye-in-hand RealSense D415 camera together with an third- view D415 camera. We test six household appliances: microwave, air fryer, toaster, rice cooker, coffee machine, and blender. For each ap- pliance, we build a CAD/URDF model; FoundationPose [27] uses the CAD model for 6D pose estimation, from which appliance-specific atomic actions are instantiated. For Pick actions, we integrate Any- Grasp [8]. During each episode, AppliancePlan first predicts an open-loop plan from the manual, observation, and task instruction before execution, and then revises the next action online using real-time perception feedback. We evaluate 10 manipulation tasks for each appliance. Quantitative results are reported in Table 4, and qualitative examples are shown in Figure 5. As shown in Table 4, AppliancePlan outperforms all baselines on every appliance. Aver- aged over the six appliances, AppliancePlan achieves 49.87% task completion and 40.00% task success, compared with 19.35%/3.33% for GPT-5 and 9.12%/3.33% for ApBot. The much larger margin on task success indicates that general-purpose MLLMs and pipeline methods may execute early steps correctly, but often fail at the state-dependent decisions required to complete the full task. Table 4: Real-robot results on household appliances. Entries report task completion rate/task success rate. Microwave Air-fryer Ricecooker Coffee Machine Toaster Blender GPT-516.67/0.0011.33/0.0028.30/10.0018.30/0.0014.90/0.00 26.60/10.00 ApBot [30]11.80/0.0014.70/0.003.30/0.008.30/10.003.30/0.0013.30/10.00 AppliancePlan (Ours)50.00/40.0049.10/40.0026.80/20.0052.50/40.0033.33/20.0087.50/80.00 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilLong et al. Acknowledgments This work was supported by the Beijing Natural Science Foundation (L2608141) and the National Natural Science Foundation of China (62136001). References [1]Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Ricardo Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, et al.2022. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691 (2022). [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. NeurIPS (2022). [3]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025). [4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.2024. pi0: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164 (2024). [5]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Danny Driess, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Her- zog, et al.2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818 (2023). [6]Open X-Embodiment Collaboration. 2023. Open X-Embodiment: Robotic Learn- ing Datasets and RT-X Models. https://arxiv.org/abs/2310.08864. [7]Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. PaLM-E: An Embodied Multimodal Language Model. arXiv preprint arXiv:2303.03378 (2023). [8]Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. 2023. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics 39, 5 (2023), 3929–3945. [9] Yuzheng Gao, Yuxing Long, Lei Kang, Yuchong Guo, Ziyan Yu, Shangqing Mao, Jiyao Zhang, Ruihai Wu, Dongjiang Li, Hui Shen, and Hao Dong. 2026. RealAppli- ance: Let High-fidelity Appliance Assets Controllable and Workable as Aligned Real Manuals. [10] Google AI. 2025. Gemini 2.5 API. https://ai.google.dev/gemini-api/docs [11] Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al.2025. Robobrain: A uni- fied brain model for robotic manipulation from abstract to concrete. In Proceedings of the Computer Vision and Pattern Recognition Conference. 1724–1734. [12]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al.2024. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint arXiv:2406.09246 (2024). [13]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models. In ICML. [14]Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. 2024. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18061–18070. [15]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruc- tion tuning. In NeurIPS. [16] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023). [17]Yuxing Long, Jiyao Zhang, Mingjie Pan, Tianshu Wu, Taewhan Kim, and Hao Dong. 2025. CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [18] OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774 (2023). [19] OpenAI. 2025. Gpt5. https://openai.com/index/introducing-gpt-5/ [20]Mingjie Pan, Jiyao Zhang, Tianshu Wu, Yinghao Zhao, Wenlong Gao, and Hao Dong. 2025. OmniManip: Towards General Robotic Manipulation via Object- Centric Interaction Primitives as Spatial Constraints. arXiv:2501.03841 [cs.RO] https://arxiv.org/abs/2501.03841 [21]Qwen Team. 2025. Qwen3-VL API. https://modelstudio.console.alibabacloud. com/?tab=doc#/doc/?type=model&url=2840914_2&modelId=qwen3-vl-plus [22]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning. 8748–8763. [23]Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- Speed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506. [24]Team Seedream, Yunpeng Chen, Yu Gao, Lixue Gong, Meng Guo, Qiushan Guo, Zhiyao Guo, Xiaoxia Hou, Weilin Huang, Yixuan Huang, et al.2025. See- dream 4.0: Toward next-generation multimodal image generation. arXiv preprint arXiv:2509.20427 (2025). [25]BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Xiansheng Chen, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, et al.2025. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02029 (2025). [26] GLM-V Team. 2025.GLM-4.5V and GLM-4.1V-Thinking: Towards Ver- satile Multimodal Reasoning with Scalable Reinforcement Learning. arXiv:2507.01006 [cs.CV] https://arxiv.org/abs/2507.01006 [27]Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. 2024. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17868–17879. [28]Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng- ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al.2025. Qwen-image technical report. arXiv preprint arXiv:2508.02324 (2025). [29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). [30]Jian Zhang, Hanbo Zhang, Anxing Xiao, and David Hsu. 2025. Robot Operation of Home Appliances by Reading User Manuals. arXiv preprint arXiv:2505.20424 (2025).