Paper deep dive
RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset
Yongzhong Wang, Keyu Zhu, Yong Zhong, Liqiong Wang, Jinyu Yang, Feng Zheng
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/22/2026, 6:28:12 AM
Summary
RADAR is a fully autonomous, closed-loop robotic data generation pipeline that eliminates human intervention by integrating a Vision-Language Model (VLM) for semantic planning, a Graph Neural Network (GNN) for in-context imitation learning, and a Finite State Machine (FSM) for autonomous environment resets using a Last-In, First-Out (LIFO) causal sequence.
Entities (5)
Relation Signals (4)
RADAR → utilizes → Vision-Language Model
confidence 100% · the VLM acts as the cognitive “brain” responsible for high-level semantic reasoning
RADAR → utilizes → Graph Neural Network
confidence 100% · the GNN-based policy functions as the “cerebellum” to execute sub-millimeter, high-frequency physical control
RADAR → utilizes → Finite State Machine
confidence 100% · a Finite State Machine (FSM) governs the system to actively compute and execute a causal inverse sequence
Vision-Language Model → queries → Affordance Library
confidence 90% · the VLM planner identifies an atomic action a_t, retrieves the most semantically and geometrically congruent demonstration d_t from the affordance library L
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The acquisition of large-scale physical interaction data, a critical prerequisite for modern robot learning, is severely bottlenecked by the prohibitive cost and scalability limits of human-in-the-loop collection paradigms. To break this barrier, we introduce Robust Autonomous Data Acquisition for Robotics (RADAR), a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle. RADAR elegantly divides the cognitive load into a four-module pipeline. Anchored by 2-5 3D human demonstrations as geometric priors, a Vision-Language Model first orchestrates scene-relevant task generation via precise semantic object grounding and skill retrieval. Next, a Graph Neural Network policy translates these subtasks into physical actions via in-context imitation learning. Following execution, the VLM performs automated success evaluation using a structured Visual Question Answering pipeline. Finally, to shatter the bottleneck of manual resets, a Finite State Machine orchestrates an autonomous environment reset and asymmetric data routing mechanism. Driven by simultaneous forward-reverse planning with a strict Last-In, First-Out causal sequence, the system seamlessly restores unstructured workspaces and robustly recovers from execution failures. This continuous brain-cerebellum synergy transforms data collection into a self-sustaining process. Extensive evaluations highlight RADAR's exceptional versatility. In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks, effortlessly solving challenges where traditional baselines plummet to near-zero performance. In real-world deployments, the system reliably executes diverse, contact-rich skills (e.g., deformable object manipulation) via few-shot adaptation without domain-specific fine-tuning, providing a highly scalable paradigm for robotic data acquisition.
Tags
Links
- Source: https://arxiv.org/abs/2603.11811v1
- Canonical: https://arxiv.org/abs/2603.11811v1
Trouble viewing inline? Open PDF directly →
Full Text
45,222 characters extracted from source content.
Expand or collapse full text
RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset Yongzhong Wang 1∗ , Keyu Zhu 1∗ , Yong Zhong 1∗ , Liqiong Wang 1 , Jinyu Yang 2 , Feng Zheng 1,3 Abstract— The acquisition of large-scale physical interaction data—a critical prerequisite for modern robot learning—is severely bottlenecked by the prohibitive cost and scalability limits of human-in-the-loop collection paradigms. To break this barrier, we introduce Robust Autonomous Data Acquisition for Robotics (RADAR), a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle. RADAR elegantly divides the cognitive load into a four-module pipeline. Anchored by merely 2-5 3D human demonstrations as geometric priors, a Vision-Language Model (VLM) first orchestrates scene-relevant task generation via precise semantic object grounding and skill retrieval. Next, a Graph Neural Network (GNN) policy translates these subtasks into robust physical actions via in-context imitation learning. Following execution, the VLM performs automated success evaluation using a structured Visual Question An- swering (VQA) pipeline. Finally, to shatter the bottleneck of manual resets, a Finite State Machine (FSM) orchestrates an autonomous environment reset and asymmetric data routing mechanism. Driven by simultaneous forward-reverse planning with a strict Last-In, First-Out (LIFO) causal sequence, the system seamlessly restores unstructured workspaces and ro- bustly recovers from execution failures. This continuous brain- cerebellum synergy transforms data collection into a self- sustaining process. Extensive evaluations highlight RADAR’s exceptional versatility. In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks, effort- lessly solving challenges where traditional baselines plummet to near-zero performance. During real-world deployments, the sys- tem reliably executes diverse, contact-rich atomic skills—such as deformable object manipulation—via few-shot adaptation without domain-specific fine-tuning, providing a highly scalable paradigm to democratize robotic data acquisition. I. INTRODUCTION Recent end-to-end embodied intelligence models have demonstrated remarkable generalization and few-shot learn- ing capabilities, driving significant progress in robotic ma- nipulation [1], [2], [3]. However, the scaling of such em- bodied intelligence is fundamentally bottlenecked by the acquisition of large-scale, high-fidelity physical interaction data. Existing solutions to this data bottleneck face a frus- trating dichotomy. Simulation-based methods [4], [5], [6], [7] offer tremendous scalability but struggle with persistent sim-to-real gaps and limited behavioral diversity. Conversely, teleoperation-based methods [8], [9], [10], [11] provide high- ∗ Equal contribution 1 Southern University of Science and Technology. 2 Harbin Institute of Technology, Shenzhen. 3 Spatialtemporal AI. quality demonstrations but remain prohibitively expensive and fundamentally unscalable due to the slow and serial nature of human control. To bypass this dilemma, recent efforts have explored autonomous data collection pipelines and advanced execution policies, yet they consistently fail to bridge the semantic- to-physical divide due to three critical limitations. First, in terms of visual prompting and task planning, systems often rely on fragile 2D pixel-level guessing that lacks strict 3D kinematic constraints [12], [13], or employ intermediate image generation that succumbs to catastrophic geometric hallucinations [14], [15]. Second, while recent In-Context Imitation Learning (ICIL) and diffusion-based architectures achieve exceptional low-level action precision [16], [17], [18], they fundamentally remain passive, isolated execu- tion engines incapable of autonomously orchestrating tasks or verifying outcomes. Finally, current autonomous frame- works [14], [19] suffer from unreliable success evaluation and lack the vital cognitive mechanism to autonomously reset the physical environment. This inability to form a true closed loop inevitably traps human operators back in the data collection cycle. To address these limitations, we introduce Robust Autonomous Data Acquisition for Robotics (RADAR), a fully automated, closed-loop pipeline, as conceptually out- lined in Fig. 1. Instead of forcing Vision-Language Models (VLMs) to hallucinate intermediate pixels and guess fragile 3D coordinates, or relying on disjointed heuristic scripts for physical environment resets, RADAR elegantly divides the cognitive load, establishing a robust brain-cerebellum synergy. Specifically, the VLM acts as the cognitive “brain” responsible for high-level semantic reasoning, task genera- tion, and success evaluation, while the GNN-based policy functions as the “cerebellum” to execute sub-millimeter, high-frequency physical control based on 3D geometric priors. Our key insight is that recent advances in In-Context Learning (ICL) [16], [20] offer a direct path to amplify a handful of human demonstrations into large-scale physical execution. By leveraging these human demonstrations as a 3D physical prior, our framework effectively bypasses the sim-to-real gap of simulations, the teleoperation bottleneck of human data collection, and the hallucination risks of pure VLM control. Enabled by RADAR, high-fidelity robotic data can be continuously acquired with strictly mini- mal human intervention. By automating the entire data- arXiv:2603.11811v1 [cs.RO] 12 Mar 2026 Execution Forward Failure Reset Success Repeat the same task E DataExecute STORAGE Reset Failure Affordance Library Forward Success Reverse Execution C Task Planning A (1)Semantic ObjectGrounding (2)Hierarchical Task Planning (3)In-Context Skill Retrieval ForwardExecution B Step 1: Put the Block in the Box Step 2:Close the Box Step 3:Push the Box to the Right Step 1’: Push the Box to the Left Step 2’: Open the Box Step 3’: Take out the Block Single Storage Forward Reverse D Dual Storage Forward Reverse Fig. 1.Overview of the RADAR pipeline and the state transition diagram of its decoupled Finite State Machine (FSM). To ensure logical clarity, the architecture strictly separates physical execution loops (States A, B, C) from concurrent data routing actions (States D, E). A fully successful execution forms a continuous loop (B → C → B), concurrently triggering Dual Storage (D) to repeatedly harvest trajectory variations without re-planning. In contrast, an asymmetric recovery loop (B → C → A) bypasses reset failures by selectively saving the valid forward trajectory via Single Storage (E) and initiating a novel planning cycle on the altered workspace. This architecture guarantees a truly self-sustaining, human-out-of-the-loop engine. generation pipeline, we accordingly structure the process into a four-module framework: (1) Scene-Relevant Task Generation: We employ a Vision-Language Model (VLM) to autonomously construct scene-relevant tasks and extract object segmentation masks based on the current observation. Complex, long-horizon tasks are decomposed into a sequence of atomic subtasks, each matched with a relevant demonstration from a small affordance library as a behavioral prior. (2) Task Execution via In-Context Imitation Learning: The robot performs the assigned subtasks using a Graph Neural Network (GNN)-based in-context imitation learning framework, which maps the selected demonstrations and current observations to executable continuous trajectories. (3) Automated Success Evaluation: The VLM acts as an embodied evaluator to determine the outcome of the execution through a structured Visual Question Answering (VQA) formulation, filtering out failed trajectories. (4) Autonomous Environment Reset: Upon completion of a task, a Finite State Machine (FSM) governs the system to actively compute and execute a causal inverse sequence of the forward actions. This strictly adheres to a Last-In, First-Out (LIFO) logical constraint to restore the environ- ment, facilitating truly continuous and robust data generation without human assistance. Despite drastically reducing the reliance on human labor, extensive evaluations demonstrate that our pipeline maintains highly robust performance and execution efficacy across both simulation and real-world deployments. In the simu- lation environment, we rigorously assess the framework’s capability to orchestrate and execute complex, long-horizon manipulation tasks, achieving high success rates and robust autonomous resetting. To validate its practical applicability, we further deploy the pipeline directly onto a physical robotic system. Remarkably, by relying merely on a single or a few visual demonstrations (i.e., one-shot or few-shot adaptation), the system successfully executes a variety of challenging atomic skills—such as manipulating deformable objects (e.g., folding towels) and performing high-precision alignments (e.g., inserting paper rolls)—without requiring any domain- specific fine-tuning. Together, these results confirm the ro- bustness and versatility of our closed-loop pipeline across different domains, providing a highly scalable and domain- agnostic data generation engine for physical robot learning. Our main contributions are highlighted as follows: • We introduce RADAR, a fully automated, closed-loop pipeline for real-world robot manipulation data collec- tion. Our system requires only 2-5 manually collected atomic demonstrations and scales them into diverse, task-relevant datasets with strictly minimal human in- tervention. • We propose a scene-relevant task generation frame- work, which effectively translates complex, long- horizon tasks in cluttered environments into sequentially executable atomic skills. • We design a novel autonomous environment reset mechanism to achieve autonomous environment re- setting. This empowers the robot with causal self- correction and scene-restoration capabilities, unlocking continuous, human-out-of-the-loop data streaming. • ExtensiveEmpiricalValidation:Wevalidate RADAR’s capabilities across simulated and physical domains. Our pipeline achieves up to 90% success rates on complex, long-horizon simulated tasks, demonstrating highly robust forward planning and execution.Furthermore,real-worlddeployments establish the system as a powerful proof-of-concept for human-out-of-the-loop data generation, reliably executing diverse, contact-rich physical skills (e.g., deformable object manipulation) via few-shot in-context learning. I. RELATED WORK A. Autonomous Data Collection and Evaluation High-quality robot demonstration data is the cornerstone of robust imitation learning [8], [21]. To overcome manual collection bottlenecks, recent works explore autonomous policy improvement [19]. For instance, the SOAR frame- work [14] utilizes pre-trained VLMs for task proposal and success detection, employing an image-editing diffusion model, SuSIE [15], to generate visual subgoals for a goal- conditioned policy [22]. However, SOAR and similar autonomous systems face a fundamental bottleneck: environment resets. Once a robot alters the environment, continuous collection breaks without human intervention. Furthermore, SOAR’s single-stage VLM evaluation is highly susceptible to conversational redun- dancies and visual hallucinations, leading to false-positive success labels [14]. In contrast, our pipeline achieves true closed-loop auton- omy via a Simultaneous Forward-Reverse Planning mecha- nism. The VLM constructs a strict Last-In, First-Out (LIFO) causal sequence to automatically restore the environment. Coupled with an asymmetric failure handling logic, un- restored scenes seamlessly become novel initial states. More- over, we replace unreliable single-stage evaluation with a ro- bust three-stage Vision-Question-Answering (VQA) pipeline, strictly decoupling VLM visual reasoning from deterministic logic to ensure the absolute fidelity of collected demonstra- tions. B. Visual Prompting and Affordance Reasoning Integrating VLMs into control architectures requires effec- tive affordance representations [23], [24]. Inspired by Set-of- Mark prompting [25], MOKA [12] advanced this via a mark- based visual prompting framework, overlaying 2D candidate keypoints on RGB images to formulate affordance reasoning as a VQA problem [12], [26]. While simplifying VLM reasoning, this 2D paradigm exhibits significant geometric vulnerability. Predicting affor- dances entirely in 2D pixel space forces MOKA to rely on noisy depth heuristics to back-project points into 3D SE(3) space [12]. Consequently, executions involving complex con- tact dynamics (e.g., tight insertions) inevitably fail, as 2D pixels cannot encapsulate precise kinematic constraints. Our approach firmly rejects this fragile 2D guessing. We introduce a 3D prior-based Affordance Library derived from a small set of real human demonstrations. Instead of forcing the VLM to generate coordinates from scratch, we constrain it to perform semantic object grounding and In-Context Skill Retrieval. By retrieving the most geometrically and semanti- cally consistent 3D demonstration as a contextual prior, we shift the physical precision burden to human-demonstrated trajectories. This preserves VLM semantic generalization while achieving strictly feasible, high-fidelity kinematics. C. In-Context Learning and Graph Diffusion In-Context Imitation Learning (ICIL) has emerged as a promising paradigm to execute VLM-planned subtasks without exhaustive fine-tuning [18], [20]. Bridging ICIL with multimodal diffusion models [17], Instant Policy [16] formu- lates ICIL as a conditional graph generation problem using Graph Neural Networks [27]. It employs a graph transformer- based reverse diffusion process to iteratively denoise future action nodes [16]. While the underlying graph diffusion architecture is highly effective for low-level action generation, it fundamentally operates as an isolated execution policy. It heavily relies on human-provided task definitions and demonstrations, lacking the cognitive mechanisms required to autonomously propose novel tasks, chain long-horizon behaviors, or verify execution success. Conversely, autonomous frameworks like SOAR attempt to close this loop using image-editing diffusion mod- els (SuSIE) to generate intermediate visual subgoals [14], [15]. However, generating intermediate pixels introduces high computational latency and severe physical hallucina- tions (e.g., floating objects), inevitably causing catastrophic execution failures. Our work resolves this dichotomy by embedding the graph diffusion model as a subordinate execution engine within a fully autonomous, VLM-driven closed loop. We replace error-prone pixel generation with our robust front-end hier- archical task planning, which autonomously grounds objects and translates high-level semantic goals into precise atomic contexts. Following execution, our back-end VQA evaluation strictly verifies task completion and triggers environment resets. By wrapping the sub-millimeter SE(3) precision of the ICIL graph architecture with these autonomous cognitive layers, we successfully elevate it from a passive execution policy into a continuous, human-out-of-the-loop data gener- ation pipeline. I. METHOD As orchestrated by the continuous control flow illustrated in Fig. 1, our fully automated data collection framework operates as a self-sustaining closed loop. To ground the system’s physical execution capabilities, we first formalize a foundational set of semantic skills—depicted as the vital data prior in our architecture—as an Affordance Library. Building upon this library, our pipeline is elegantly structured into a four-module framework that explicitly mirrors the brain-cerebellum division of labor: (1) Scene-Relevant Task Generation, (2) Task Execution via In-Context Imitation Learning, (3) Automated Success Evaluation, and (4) Au- tonomous Environment Reset. In the following subsections, we first define the prerequi- site Affordance Library, and subsequently detail the technical formulation of each of the four pipeline modules. In the following subsections, we first define the prerequi- site Affordance Library, and subsequently detail the technical formulation of each of the four pipeline phases. A. Preliminaries: Affordance Library To facilitate downstream in-context imitation learning pol- icy (detailed in Section I-C), we construct an Affordance Library of manually collected demonstrations. We denote the (a) Atomic Task in Simple Environments (b) Atomic Task in Complex Environments (c) Long-Horizon Skill Chaining Forward task: Pick up the lemon Demo: Pick up the grip ball Reverse task: Put down the lemon Demo: Put down the grip ball (towel, deformable) (lemon, oval) (strawberry, conical) (Rubik’s cube, cube ) (yellow block, cube) (red block, cube) (pink block, cube) Forward task: Push the yellow block in the white area and stack the red block on the yellow block Forward subtask 1: Push the yellow block in the white area Demo: Push the earbud case to the right Forward subtask 2: Stack the red block on the yellow block Demo: Put the glass case on the shelf Reverse task: Put the red block on the table and push the yellow block out of the white area Reverse subtask 1: Put the red block on table Demo: Move the glass case from the shelf to the table Reverse subtask 2: Push the yellow block out of the white area Demo: Push the earbud case to the left Forward task: Fold the towel Demo: Close the box Reverse task: Unfold the towel Demo : Open the box Fig. 2.Overview of our hierarchical Scene-Relevant Task Generation framework across varying complexities. (a) Atomic Task in Simple Environments: The VLM performs direct affordance matching, mapping a deformable object task (folding a towel) to a geometrically congruent 3D prior (closing a box). (b) Atomic Task in Complex Environments: Through selective attention, the planner actively masks out distractors (e.g., strawberry, Rubik’s cube) to precisely ground the target object (lemon) and retrieve a robust prior. (c) Long-Horizon Skill Chaining: For multi-step tasks, the VLM orchestrates a forward skill chain (pushing and stacking blocks) while concurrently generating a strict Last-In, First-Out (LIFO) causal sequence to autonomously construct executable environment-resetting plans. library as L. Following instant policy [16], each demon- stration d ∈ L is a trajectory of local graph capturing the state of execution over a temporal horizon T . Formally, a demonstration d is defined as a sequence of states: d =(P t ,T t WE ,g t ,ξ) T t=1 (1) At each timestamp t, our observation at time consists of segmented point cloud P t , the homogeneous transformation matrix from the world frame to the end-effector frame T t WE ∈ SE(3) and g t ∈ 0, 1 indicates the gripper state, where 0 and 1 represent open and closed status, respectively. Specific to simulation, we add a list of object identified by id ξ sim = x 1 ,x 2 ,...,x n to d for masking. Moreover, in order to dependency on ground-truth trackers, the real-world demonstrations omit these identifiers, relying instead on the VLM-based object grounding described in our task planning module. B. Module 1: Scene-Relevant Task Generation As illustrated in Fig. 2, to generate physically feasible and semantically meaningful tasks in unstructured environments, we propose a hierarchical visual task planning framework. To leverage the reasoning capabilities of VLM (denoted as M), we propose a 2-stage process: (1) semantic object grounding, where we utilize VLM to ground relevant ob- jects; (2) hierarchical task planning, where we translate visual observations into structured task plans. 1) Semantic Object Grounding: To get an accurate under- standing of the scene, instead of feeding raw images directly into the planning module (which may lead to hallucinations), we first employ a VLM-based object detector to extract a structured semantic representation of the current scene. The process can be formulated as follows: ξ scene =M(p ground ,s 0 )(2) Given an RGB image of the initial scene s 0 with a grounding language prompt p ground , the pipeline queries the VLM to identify all candidate objects and their geometric attributes (e.g., identifying a lemon as “oval” or a strawberry as “con- ical”, as shown in Fig. 2b). We denote this structured output as ξ scene = x 1 ,x 2 ,...,x n ,x i = (name,shape). This output serves as a hard constraint for subsequent planning, ensuring the robot only attempts to interact with objects physically present in the workspace. 2) Hierarchical Task Planning: We formalize the hier- archical task planning as a conditional generation problem. Given the current scene observation s and a task-level prompt p level , the VLM M generates a structured response y: y =M(p level ,s)(3) where y encapsulates a set of task-relevant objects ξ mask for precise visual grounding and a sequence of executable subtasks T = (a t ,d t ,c t ) T t=1 . For each step t, the VLM planner identifies an atomic action a t , retrieves the most semantically and geometrically congruent demonstration d t from the affordance library L, and provides a textual descrip- tion c t to guide the downstream imitation learning policy. Based on scene complexity and task horizon, our plan- ner dynamically adapts across three scenarios, as compre- hensively detailed in Fig. 2: 1) Atomic Task in Simple Environments: For uncluttered scenes, the VLM performs Direct Affordance Matching (Fig. 2a). For example, it maps the task of “folding a deformable towel” directly to the geometrically similar demonstration of “closing a box”, thereby generating its reverse task (“unfolding the towel”) based on the “opening a box” demonstration. 2) Atomic Task in Complex Environments: When distractors are present, a specific prompt enforces Selective Attention (Fig. 2b). The VLM identifies a target subset ξ mask (e.g., focusing purely on the target lemon while actively ignoring the strawberry and Rubik’s cube) to prevent hallucinated interactions, re- trieving the “pick up grip ball” skill as a robust 3D prior. 3) Long-Horizon Skill Chaining: For multi-step tasks, the VLM logically chains atomic skills (Fig. 2c) without de- composing them into smaller primitives. The system prompt enforces physical causality and the simultaneous generation of environment-resetting plans. For example, generating a forward sequence (“push the yellow block into the white area”, “stack the red block”) concurrently produces a strict Last-In, First-Out (LIFO) constrained reverse sequence (“put the red block on the table”, “push the yellow block out”). This explicitly ensures the multi-step plan is both physically executable and inherently self-reversible. 3) In-Context Skill Retrieval via Affordance Library: To bridge high-level planning with low-level control, both atomic and long-horizon planning modes utilize an in-context retrieval mechanism. For every generated subtask, the VLM estimates a similarity rank based on action semantics and ob- ject geometry, returning an ordered list of reference demon- strations D = d 1 ,d 2 ,...,d r . Specifically, our prompting framework instructs the VLM to evaluate similarities across two distinct dimensions: Action Similarity (e.g., ensuring the motion trajectories of folding and closing align) and Geometric/Functional Similarity (e.g., recognizing that a “lemon” is geometrically congruent to an oval “grip ball”). This dual-criteria retrieval eliminates the need for hallu- cinating arbitrary 3D waypoints, allowing the downstream imitation learning policy to leverage robust geometric priors for zero-shot generalization to novel objects. C. Module 2: Task Execution via In-Context Imitation Learning To instantly execute novel tasks without fine-tuning, our system builds upon the Instant Policy framework [16]. This approach formulates In-Context Imitation Learning (ICIL) as a graph-based diffusion generation problem [16]. Specifically, we construct a heterogeneous graph G that jointly expresses the context G c (derived from the provided demonstrations), the current point cloud observation G t l , and the future actionsG a l (a). The closed-loop control actions are inferred through a reverse diffusion process. Starting from a noise-altered graph constructed with sampled random actions a K ∼ N(0,I), a parameterized graph transformer model predicts the gradient field ε θ to iteratively denoise the actions. The reverse denoising step at iteration k is defined as: G k−1 =G G a l α(a k − γε θ (G k ,k)) +N(0,σ 2 I) ,G c (4) where G k represents the graph at diffusion step k, and ε θ (G k ,k) acts as the predicted gradient field that guides the adjustment of nodes representing the robot actions. The term α controls the step size, and γ scales the noise prediction. By repeating this process for K steps, the graph transitions into the true data distribution, yielding the final executable action a 0 . D. Module 3: Automated Success Evaluation To close the loop of our autonomous data collection pipeline, it is imperative to evaluate the outcome of the executed policy without requiring human-in-the-loop ver- ification. To this end, we introduce an Automated Suc- cess Evaluation module. This module leverages a Vision- Language Model (VLM) to visually inspect the post- execution workspace and determine whether the physical state aligns with the semantic goal of the commanded subtask. To mitigate the inherent variability and conversational ver- bosity of raw VLM outputs, we formulate the success detec- tion as a structured, three-stage Visual Question Answering (VQA) pipeline: (1) Semantic Task-to-Query Translation, (2) Vision-Language Assessment, and (3) Robust Boolean Decoding. 1) Semantic Task-to-Query Translation: Standard VLMs often struggle to evaluate imperative task commands (e.g., “put the yellow ball on the blue plate”) directly. To optimize the VLM’s reasoning, we first translate the task description c t into a targeted, interrogative VQA query q vqa . We employ a Large Language Model (LLM), denoted as M trans , guided by a few-shot in-context prompt containing diverse manipulation examples (e.g., mapping “move the red object from the cloth to the table” to “Is the red object on the cloth or the table?”). This process translates the action- oriented command into a state-oriented visual query: q vqa =M trans (c t ,p vqa )(5) where p vqa represents the few-shot prompt template designed to elicit specific spatial and relational questions. 2) Vision-Language Assessment: Following the execution of the policy over the temporal horizon T , the robot captures the final visual observation of the workspace, denoted as s T . The generated query q vqa and the image s T are subsequently fed into a state-of-the-art VLM (e.g., GPT-4V or CogVLM), denoted as M vlm . The VLM acts as an embodied evaluator, analyzing the spatial relationships and object states within s T to generate a raw textual assessment r vlm : r vlm =M vlm (s T ,q vqa )(6) 3) Robust Boolean Decoding: Since the raw response r vlm from the VLM may contain auxiliary reasoning or conversational fillers (e.g., “Yes, I can see that the object is...” instead of a strict boolean), directly parsing this output is error-prone. To ensure a deterministic control flow for our autonomous pipeline, we introduce a final decoding step. We utilize a parsing LLM M parse to distill the ver- bose evaluation into a strict binary success signal b succ ∈ True,False. The parser is conditioned on the original task c t , the generated query q vqa , and the VLM’s response r vlm : b succ =M parse (c t ,q vqa ,r vlm )(7) Instead of relying on heuristic error handling, this extracted binary signal b succ directly serves as the deterministic trig- ger for state transitions in our downstream pipeline. These signals dictate whether the system advances to the reverse resetting module or aborts to initiate a new planning cycle, as detailed in Section I-E. Furthermore, these discrete success metrics are logged to maintain historical task statistics, enabling the system to track the reliability of specific skills over time. E. Module 4: Autonomous Environment Reset To achieve truly continuous and autonomous data col- lection, minimizing human intervention between episodes is critical. We introduce an Autonomous Environment Reset mechanism, which leverages the reasoning capabilities of the VLM to automatically restore the workspace to its initial state s 0 following the completion of a generated task. Instead of relying on hard-coded or heuristic reset scripts, our framework formulates the environment reset as an inverse task planning problem tightly coupled with the forward generation. 1) Simultaneous Forward-Reverse Planning: During the hierarchical task planning phase, the VLM is prompted to act as a causal reasoning engine. It simultaneously generates the primary executable plan (denoted as the forward task) and its exact causal inverse (denoted as the reverse task). For atomic tasks, the VLM directly infers the inverse affordance based on the object’s geometric properties. For example, if the forward atomic action is “pick up the cup”, the VLM proposes “place the cup down” as the reverse task and retrieves the most semantically congruent demonstration from the affordance library L. For long-horizon multi-step tasks, the environment reset strictly adheres to a Last-In, First-Out (LIFO) causal se- quence constraint. Given a forward plan consisting of N subtasks T fwd = (a f i ,d f i ,c f i ) N i=1 , the VLM constructs a reverse plan T rev = (a r j ,d r j ,c r j ) N j=1 . Crucially, each reverse subtask at step j must explicitly undo the physical state changes introduced by the forward subtask at step i = N − j + 1. This ensures physical feasibility during the restoration phase (e.g., a box must be opened before the cube placed inside it can be extracted). 2) Continuous Collection via Finite State Machine: To seamlessly integrate physical actions and data logging into a continuous, human-out-of-the-loop pipeline, we orchestrate the process using a Finite State Machine (FSM), as visualized in Fig. 1. To ensure logical clarity, our FSM explicitly decou- ples the physical execution states—Task Planning (State A), Forward Execution (State B), and Reverse Execution (State C)—from the concurrent data routing actions—Dual Storage (State D) and Single Storage (State E). Governed by the binary success signal b succ from the VQA evaluator, the system executes the following operational loops, with data storage acting as triggered side-effects: • Continuous Success Loop (B → C → B): If both the forward task and the reverse reset succeed, the physical environment is perfectly restored. The control flow seamlessly loops from State C directly back to State B to repeatedly execute the same assigned task. Concurrently, this successful cycle triggers the Dual Storage (State D) mechanism, validating and saving both the forward and reverse trajectories. This cyclic execution allows the system to continuously harvest diverse trajectory variations of a specific skill without the computational overhead of VLM re-planning. • Asymmetric Recovery Loop (B → C → A): If the forward task succeeds but the reverse reset fails, the physical environment is left un-restored. To recover, the control flow forcibly breaks the loop and routes back to Task Planning (State A), treating the altered workspace as a novel initial scene. Concurrently, this transition triggers the Single Storage (State E) mechanism, se- lectively retaining the perfectly valid forward trajectory while discarding the failed reset attempt. • Forward Abort (B → A): If the initial forward execution fails, the invalid trajectory is immediately discarded (no storage triggered). The control flow aborts the current execution queue and transitions back to State A to re-observe and re-plan. As depicted in the global architecture (Fig. 1), this state- decoupled orchestration guarantees the continuous streaming of high-fidelity data into the Affordance Library. By strictly separating the execution loops from the asymmetric data routing, the pipeline effectively bypasses reset failures and eliminates the need for manual scene restoration, transform- ing it into a truly self-sustaining engine. IV. EXPERIMENTS A. Experimental Setup We evaluate our automated pipeline in the RLBench simulation [28]. We select 7 atomic tasks and 3 complex long-horizon tasks, comparing our system against two state- of-the-art vision-language manipulation baselines: MOKA [12] and ReKep [13]. Implementation Details & Evaluation Protocol: To rigorously isolate and evaluate the forward task generation and execution capabilities, the environments are reset using simulation ground truth between rollouts. We report the success rate over 10 independent rollouts per task with randomized initial states. Empirically, we provide a single high-quality demonstration (1-shot) as the context, as in- creasing demonstrations did not yield proportional gains. Furthermore, we utilized a VLM instead of CLIP for skill retrieval, as our preliminary tests showed CLIP embeddings are heavily noun-biased and fail to distinguish fine-grained actionable semantics. TABLE I SUCCESS RATES IN RLBENCH SIMULATION (10 ROLLOUTS PER TASK) TaskReKepMOKAOurs Atomic Tasks Large Container (Cup)0.200.200.80 Large Container (Block)<0.100.300.80 Large Container (Laptop)<0.100.100.90 Push Block0.400.401.00 Stack Block0.400.100.80 Close Box0.400.301.00 Open Box0.200.200.70 Long-Horizon Tasks Put Laptop & Cup into Tray0.100.000.80 Push & Stack Blocks0.000.000.40 Close then Open Box0.200.100.90 T=1T=2T=3 T=6 T=1T=2T=3 T=4T=5 T=4T=5T=6 Task Planning 2Open the BoxClosethe Box 12 Task Planning 1Put Cup in TrayPut Laptop in Tray 12 Fig. 3.Visualizations of long-horizon tasks in the RLBench simulation. (a) Put Laptop & Cup into Tray: The robot successfully executes a multi- step sequence requiring interactions with multiple distinct objects. (b) Close then Open Box: The pipeline reliably performs state-dependent articulated actions, demonstrating its robust skill chaining capability. B. Main Results As shown in Table I, our pipeline significantly outper- forms ReKep and MOKA. While baselines exhibit moderate performance on simple atomic tasks, their success rates plummet to near-zero in long-horizon scenarios (e.g., “Push & Stack Blocks”, as illustrated in Fig. 4c). In contrast, our system maintains robust performance, achieving 80%-90% success rates on demanding multi-step tasks. Specifically, the pipeline effectively orchestrates sequential skill chains and dependent articulated actions (e.g., opening a previously closed box, as visualized in Fig. 3), proving its capability to autonomously gather high-quality data for complex behav- iors. C. Ablation: The Necessity of Point Cloud Masking TABLE I ABLATION ON POINT CLOUD MASKING TaskW/o MaskingOurs (With Masking) Large Container (Cup)0.100.80 Large Container (Block)0.000.80 Push Block0.001.00 To evaluate our VLM-driven selective attention, we ab- late the masking module and feed raw scene point clouds into the execution policy. As shown in Table I, removing Atomic Task Planning in Complex Scenarios: Place strawberry in real Hierarchical Task Planning Stack the BlockPush the Block 12 Marked Observation Selective Attention Long-Horizon Task Planning via Skill Chaining: Push the block & stack the block Atomic Task Planning in Simple Scenarios: Fold towels in real a b c Fig. 4.Qualitative results of our automated data collection pipeline across different scenarios. (a) Simple Atomic Scenario: The robot executes deformable object manipulation (folding a towel) in a real-world setting without distractors. (b) Complex Atomic Scenario: The pipeline utilizes selective attention to isolate a target object (a strawberry) among visual distractors for precise grasping. (c) Long-Horizon Scenario: In simulation, the VLM decomposes a complex instruction into a sequential skill chain (e.g., first pushing, then stacking a block). semantic masking causes catastrophic failures, with success rates for grasping and pushing dropping to near zero due to susceptibility to distractors. This highlights that explicit semantic masking is crucial for execution robustness in cluttered scenes (a real-world qualitative example is depicted in Fig. 4b). D. Real-World Deployment and Limitations To validate practical applicability, we deployed RADAR on a physical Realman RM65-B arm with a RealSense D435i camera, using SAM [29] and XMem++ [30] for real-time 3D object segmentation. Qualitative Feasibility: Operating under a 1-shot adapta- tion paradigm without domain-specific fine-tuning, the robot successfully executed intricate tasks. In simple environments, it handled deformable object manipulation (e.g., folding a towel, Fig. 4a). In complex scenarios, it utilized selective attention for precise grasping among distractors (e.g., target- ing a strawberry, Fig. 4b). These qualitative results strongly indicate RADAR’s immense potential as a scalable, human- out-of-the-loop engine for physical robot learning. Limitations and Future Work (The Reset Challenge): While our pipeline demonstrates the feasibility of au- tonomous data generation, fully 100% reliable environment resetting remains an open challenge. Probabilistically, chain- ing forward execution with a causal reverse reset inherently compounds failure rates (p total ≈ p forward × p reverse ). Currently, our FSM acts as a robust proof-of-concept for simple-to-moderate scenes. Overcoming this compounding error in highly unstructured environments—perhaps via multi-modal tactile feedback or high-frequency visual ser- voing—represents an exciting frontier for future work. V. CONCLUSION We present Robust Autonomous Data Acquisition for Robotics (RADAR), a self-sustaining, human-out-of-the- loop data generation engine. Orchestrated by a decoupled Finite State Machine (FSM), RADAR achieves a seamless brain-cerebellum synergy by coupling VLM-based cogni- tive planning and LIFO-constrained autonomous environ- ment resetting with the sub-millimeter precision of GNN- based in-context imitation learning. Anchored by 3D human demonstrations, it circumvents the geometric hallucinations of purely 2D pipelines, while its asymmetric data routing guarantees continuous data streaming despite execution fail- ures. RADAR achieves up to 90% success on complex simu- lated tasks and reliably executes contact-rich physical skills via few-shot adaptation without domain-specific fine-tuning. Looking ahead, our future work will proceed along two ex- citing frontiers: tackling the open challenge of compounding reset errors via multi-modal sensory integration, and leverag- ing the generated high-fidelity datasets to train downstream foundational visuomotor policies (e.g., Diffusion Policy) for dynamic real-world environments. REFERENCES [1] K. Black et al., “π 0 : A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024. [2] P. Intelligence, K. Black, et al., “π 0.5 : a vision-language-action model with open-world generalization,” arXiv preprint arXiv:2504.16054, 2025. [3] S. Liu et al., “Rdt-1b: a diffusion foundation model for bimanual ma- nipulation,” in The Thirteenth International Conference on Learning Representations, 2024. [4] M. Dalal, A. Mandlekar, C. R. Garrett, A. Handa, R. Salakhutdinov, and D. Fox, “Imitating task and motion planning with visuomotor transformers,” in Conference on Robot Learning.PMLR, 2023, p. 2565–2593. [5] Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan, “Robogen: towards unleashing infinite data for automated robot learning via generative simulation,” in Proceedings of the 41st International Conference on Machine Learning, 2024, p. 51 936–51 983. [6] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox, “Mimicgen: A data generation system for scalable robot learning using human demonstrations,” in Conference on Robot Learning. PMLR, 2023, p. 1820–1864. [7] Z. Jiang, Y. Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. J. Fan, and Y. Zhu, “Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, p. 16 923–16 930. [8] A. Brohan et al., “Rt-1: Robotics transformer for real-world control at scale,” Robotics: Science and Systems XIX, 2023. [9] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine, “Bridge data: Boosting gener- alization of robotic skills with cross-domain datasets,” in Proceedings of Robotics: Science and Systems, New York City, NY, USA, 6 2022. [10] A. Brohan et al., “Do as i can, not as i say: Grounding language in robotic affordances,” in Conference on Robot Learning. PMLR, 2023, p. 287–318. [11] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, “Interactive language: Talking to robots in real time,” IEEE Robotics and Automation Letters, 2023. [12] K. Fang, F. Liu, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,” in Robotics: Science and Systems (RSS), 2024. [13] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei, “Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation,” in Conference on Robot Learning. PMLR, 2025, p. 4573–4602. [14] Z. Zhou, P. Atreya, A. Lee, H. R. Walke, O. Mees, and S. Levine, “Autonomous improvement of instruction following skills via founda- tion models,” in Conference on Robot Learning.PMLR, 2025, p. 4805–4825. [15] K. Black, M. Nakamoto, P. Atreya, H. R. Walke, C. Finn, A. Ku- mar, and S. Levine, “Zero-shot robotic manipulation with pre-trained image-editing diffusion models,” in The Twelfth International Confer- ence on Learning Representations, 2023. [16] V. Vosylius and E. Johns, “Instant policy: In-context imitation learning via graph diffusion,” in The Thirteenth International Conference on Learning Representations, 2024. [17] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023. [18] V. Jain, M. Attarian, N. J. Joshi, A. Wahid, D. Driess, Q. Vuong, P. R. Sanketi, P. Sermanet, S. Welker, C. Chan, I. Gilitschenski, Y. Bisk, and D. Dwibedi, “Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers,” in Proceedings of (RSS) Robotics Science and Systems. Proceedings of Robotics: Science and Systems, May 2024. [19] K. Bousmalis et al., “Robocat: A self-improving generalist agent for robotic manipulation,” Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id= vsCpILiWHu [20] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning. PMLR, 2022, p. 991–1002. [21] J. J. Lim, “Open x-embodiment: Robotic learning datasets and rt-x models,” in IEEE International Conference on Robotics and Automa- tion. IEEE, 2024. [22] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hind- sight experience replay,” Advances in neural information processing systems, vol. 30, 2017. [23] J. J. Gibson, “The theory of affordances,” Hilldale, USA, vol. 1, no. 2, p. 67–82, 1977. [24] L. Manuelli, W. Gao, P. Florence, and R. Tedrake, “kpam: Keypoint affordances for category-level robotic manipulation,” in International Symposium on Robotics Research (ISRR), 2019. [25] J. Yang et al., “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” arXiv preprint arXiv:2310.11441, 2023. [26] S. Liu et al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” in European conference on computer vision. Springer, 2024, p. 38–55. [27] T. Wang, R. Liao, J. Ba, and S. Fidler, “Nervenet: Learning structured policy with graph neural networks,” in International Conference on Learning Representations (ICLR), 2018. [28] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” IEEE Robotics and Automation Letters, vol. 5, no. 2, p. 3019–3026, 2020. [29] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 4015–4026. [30] M. Bekuzarov, A. Bermudez, J.-Y. Lee, and H. Li, “Xmem++: Production-level video segmentation from few annotated frames,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 635–644.