Paper deep dive
RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation
Yixue Zhang, Kun Wu, Zhi Gao, Zhen Zhao, Pei Ren, Zhiyuan Xu, Fei Liao, Xinhua Wang, Shichao Fan, Di Wu, Qiuxuan Feng, Meng Li, Zhengping Che, Chang Liu, Jian Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 1:39:45 AM
Summary
The paper introduces RoboGene, an agentic framework for automating the generation of diverse, physically plausible robotic manipulation tasks to address data scarcity in VLA pre-training. It employs diversity-driven sampling (LFU), self-reflection via specialized evaluators, and human-in-the-loop refinement with long-term memory. RoboGene outperforms state-of-the-art models like GPT-4o and Gemini 2.5 Pro, enabling VLA models to achieve higher success rates and generalization in real-world experiments.
Entities (8)
Relation Signals (7)
RoboGene → outperforms → GPT-4o
confidence 95% · Results demonstrate that RoboGene significantly outperforms state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro).
RoboGene → outperforms → Gemini 2.5 Pro
confidence 95% · Results demonstrate that RoboGene significantly outperforms state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro).
RoboGene → uses → LFU
confidence 95% · we employ a diversity-driven sampling mechanism based on a Least Frequently Used (LFU) strategy.
RoboGene → uses → Self-Reflection
confidence 95% · RoboGene integrates three core components: ... self-reflection mechanisms to enforce physical constraints
RoboGene → uses → Human-in-the-Loop
confidence 95% · RoboGene incorporates a long-term memory module that consolidates Human-in-the-Loop (HITL) feedback.
RoboGene → improves → VLA
confidence 90% · VLA models pre-trained with RoboGene achieve higher success rates and superior generalization
RoboGene → trains → π 0
confidence 85% · Experiments indicate that π0 pre-trained on RoboGene-generated task exhibit superior generalization capabilities
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The pursuit of general-purpose robotic manipulation is hindered by the scarcity of diverse, real-world interaction data. Unlike data collection from web in vision or language, robotic data collection is an active process incurring prohibitive physical costs. Consequently, automated task curation to maximize data value remains a critical yet under-explored challenge. Existing manual methods are unscalable and biased toward common tasks, while off-the-shelf foundation models often hallucinate physically infeasible instructions. To address this, we introduce RoboGene, an agentic framework designed to automate the generation of diverse, physically plausible manipulation tasks across single-arm, dual-arm, and mobile robots. RoboGene integrates three core components: diversity-driven sampling for broad task coverage, self-reflection mechanisms to enforce physical constraints, and human-in-the-loop refinement for continuous improvement. We conduct extensive quantitative analysis and large-scale real-world experiments, collecting datasets of 18k trajectories and introducing novel metrics to assess task quality, feasibility, and diversity. Results demonstrate that RoboGene significantly outperforms state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro). Furthermore, real-world experiments show that VLA models pre-trained with RoboGene achieve higher success rates and superior generalization, underscoring the importance of high-quality task generation. Our project is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.16444v2
- Canonical: https://arxiv.org/abs/2602.16444v2
Trouble viewing inline? Open PDF directly →
Full Text
94,890 characters extracted from source content.
Expand or collapse full text
RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation Yixue Zhang 1,2,∗ , Kun Wu 1,∗ , Zhi Gao 3 , Zhen Zhao 1 , Pei Ren 1 , Zhiyuan Xu 1 , Fei Liao 1 , Xinhua Wang 1 , Shichao Fan 1,4 , Di Wu 1,5 , Qiuxuan Feng 1,5 , Meng Li 1 , Zhengping Che 1,†, , Chang Liu 2, , Jian Tang 1, 1 Beijing Innovation Center of Humanoid Robotics 2 The School of Advanced Manufacturing and Robotics, Peking University 3 Beijing Institute of Technology 4 The School of Mechanical Engineering and Automation, Beihang University 5 State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University * Co-first authors; † Project leader; Corresponding authors. Generator Human-in-the-Loop Retrieve Self-Reflection Long-Term-Memory Task Diverse Real-World Tasks Human-GeneratedRoboGene-Generated Scene Balance (Human vs RoboGene) Evaluating Object and Skill Diversity Sampling Generate K-V Pair RoboGeneHumanGemini 2.5 ProGPT-4o RoboGene’s Enhanced Generalization Background Variations Novel Objects Static Distractors In Distribution Instruction Changes Illumination Changes Figure 1: RoboGene is an agentic framework designed to automate the generation of diverse, physically plausible manipulation tasks, thereby enhancing the foundational capability and generalizability of VLA model pre-training. Abstract The pursuit of general-purpose robotic manipula- tion is hindered by the scarcity of diverse, real- world interaction data.Unlike data collection from web in vision or language, robotic data col- lection is an active process incurring prohibitive physical costs. Consequently, automated task cu- ration to maximize data value remains a critical yet under-explored challenge.Existing manual methods are unscalable and biased toward com- mon tasks, while off-the-shelf foundation mod- els often hallucinate physically infeasible instruc- tions. To address this, we introduce RoboGene, an agentic framework designed to automate the generation of diverse, physically plausible ma- nipulation tasks across single-arm, dual-arm, and mobile robots.RoboGene integrates three core components: diversity-driven sampling for broad task coverage, self-reflection mechanisms to en- force physical constraints, and human-in-the-loop refinement for continuous improvement. We con- duct extensive quantitative analysis and large-scale real-world experiments, collecting datasets of 18k trajectories and introducing novel metrics to as- sess task quality, feasibility, and diversity. Re- sults demonstrate that RoboGene significantly out- performs state-of-the-art foundation models (e.g., GPT-4o, Gemini 2.5 Pro). Furthermore, real-world experiments show that VLA models pre-trained arXiv:2602.16444v2 [cs.RO] 19 Feb 2026 with RoboGene achieve higher success rates and superior generalization, underscoring the impor- tance of high-quality task generation. Our project is available at https://robogene-boost-vla.github.io/. 1 Introduction The advent of foundation models trained on vast, web-scale data has established a new paradigm in artificial intelligence, demonstrating remarkable generalization across a wide array of tasks [ Pfeifer and Iida, 2004 ] . Following this trajectory, the research community is increasingly focused on facilitating the development of general-purpose embodied agents capa- ble of performing diverse manipulation tasks within unstruc- tured environments [ Zitkovich et al., 2023; Kim et al., 2024; Liu et al., 2025a; Cheang et al., 2025; Lee et al., 2025; Intelligence et al., 2025; Bu et al., 2025b ] . Central to this vision is the curation of comprehensive datasets [ O’Neill et al., 2024; Wu et al., 2025a; Hou et al., 2025 ] that endow models with a broad foundation of physical knowledge and manipulation skills. However, robotics faces a distinct and formidable embodied data bottleneck. Unlike text or image data, robotic data requires resource-intensive physical collec- tion through real-world interaction. While recent initiatives have focused on scaling hardware infrastructure to increase raw data volume, a critical question regarding data utility re- mains largely unaddressed: How can we automate the design of high-quality, diverse data collection tasks to enhance pre- training efficacy and generalist capabilities? The prevalent approach [ Walke et al., 2023; O’Neill et al., 2024; Khazatsky et al., 2024 ] relies on human operators to explicitly specify tasks for data collection. This method suf- fers from inherent scalability limits and severe cognitive bi- ases. Human designers typically gravitate towards simplistic, repetitive primitives, such as “pick the apple” or “push the block”, while neglecting complex, multi-stage, or nuanced interactions. This bias induces a skewed dataset character- ized by a severe long-tail distribution, where a narrow subset of common objects and skills is disproportionately overrep- resented. Consequently, this lack of diversity in the training distribution acts as a barrier, preventing models from acquir- ing a sufficiently broad physical understanding and limiting their generalization to novel instructions and scenarios. To mitigate scalability and diversity constraints, one might consider leveraging Large Language Models (LLMs) for au- tomated task generation [ Wang et al., 2023b; Wang et al., 2023a; Ha et al., 2023; Hua et al., 2024; Dai et al., 2025; Gong et al., 2025 ] .Although this offers scalability, the naive application of LLMs introduces significant impedi- ments. First, due to an absence of perception regarding global dataset statistics, these models fail to rectify the long-tail ef- fects. Second, current LLMs lack physical grounding and are susceptible to hallucination, frequently yielding tasks that diverge from user constraints or reference non-existent ob- jects. They cannot inherently assess whether a specific robot embodiment possesses the kinematic feasibility to execute a generated instruction. Ensuring logical consistency and feasi- bility in this context thus requires substantial expert interven- tion. Finally, the generation is typically open-loop, failing to utilize valuable feedback from real-world execution, such as object substitution needs or motion failures, to refine subse- quent generations. In this paper, we introduce RoboGene, a novel agentic framework designed for the automated, large-scale genera- tion of diverse, physically grounded, and high-quality robotic manipulation tasks.To address the long-tail distribution problem, we employ a diversity-driven sampling mechanism based on a Least Frequently Used (LFU) strategy. This mech- anism actively guides the agent toward under-explored re- gions of the task space, prioritizing interactions with rare ob- jects and skills. To suppress hallucinations and ensure ro- bust physical grounding, we introduce a self-reflection mech- anism where generated task proposals are rigorously scruti- nized by three specialized evaluators, independently assess- ing constraint adherence, novelty, and physical feasibility. Furthermore, RoboGene incorporates a long-term memory module that consolidates Human-in-the-Loop (HITL) feed- back. By assimilating corrections from task modifications and execution failures encountered during real-world collec- tion, the system progressively refines its understanding of physical constraints and continuously enhances generation quality over time. Ultimately, RoboGene obviates the re- liance on extensive expert manual labor while ensuring that the resulting task distribution is both physically executable and statistically balanced. To validate the effectiveness of our framework, we con- duct extensive evaluations ranging from quantitative analy- sis to large-scale real-world experiments. We introduce a novel suite of metrics to quantify both the quality of indi- vidual tasks, such as the physical feasibility, and the global properties of the generated dataset, such as object and skill di- versity. Empirical results demonstrate that RoboGene signif- icantly outperforms state-of-the-art foundation models (e.g., GPT-4o [ Hurst et al., 2024 ] , Gemini 2.5 Pro [ Comanici et al., 2025 ] ) and manual design methods. For real-world valida- tion, we collected large-scale datasets comprising over 18k trajectories, spanning 1200 distinct tasks with 15 demonstra- tions per task, to benchmark the pre-training performance of VLA models. Experiments indicate that π 0 [ Black et al., 2024 ] pre-trained on RoboGene-generated task exhibit supe- rior generalization capabilities when facing unseen scenarios, including novel objects, background variations, static distrac- tors, illumination shifts, and instruction changes. These find- ings confirm that the diversity induced by RoboGene is crit- ical for the effective pre-training of generalist policies. Our main contributions are summarized as follows: • We identify the lack of diversity and physical grounding in task generation as a fundamental barrier to learning generalist policies. • We propose RoboGene, an agentic framework that auto- mates task generation through diversity sampling, self- reflection, and memory consolidation. • We provide extensive empirical evidence showing that our generated data facilitates robust policy learning. • We will open-source the high-quality task dataset and the large-scale real-world manipulation dataset. 2 Related Work 2.1 Generalist Vision-Language-Action Models The paradigm of robotic learning has progressively shifted from training specialized experts [ Zhao et al., 2023a; Chi et al., 2023; Bharadhwaj et al., 2024; Xia et al., 2025b; Wu et al., 2025b; Buamanee et al., 2024; Xia et al., 2025a; Wu et al., 2025c; Fu et al., 2024; Ze et al., 2024 ] on narrow domains to developing general-purpose Vision-Language- Action (VLA) models, which demonstrate superior general- ization and transfer capabilities. This fundamental transition has been substantially driven by the consolidation of large- scale, diverse robotic datasets [ Ebert et al., 2022; Walke et al., 2023; Khazatsky et al., 2024; O’Neill et al., 2024 ] , in- cluding recent contributions such as RoboMIND 1.0/2.0 [ Wu et al., 2025a; Hou et al., 2025 ] , AgiBot World [ Bu et al., 2025a ] , and RoboCoin [ Wu et al., 2025d ] . Building upon this robust data infrastructure, landmark VLA model [ Zitkovich et al., 2023; Black et al., 2024; Liu et al., 2025a; Wen et al., 2025a ] have emerged, enabling agents to interpret natural language commands to execute var- ious tasks. Moreover, recent studies [ Intelligence et al., 2025; Wen et al., 2025b; Liu et al., 2025b; Qu et al., 2025; Zheng et al., 2025; Fan et al., 2025 ] have focused on further augmenting these foundational policies with specialized cog- nitive and physical capabilities. Despite these architectural advancements, the performance of VLA remains fundamen- tally bounded by the quality and diversity of the training dis- tribution within datasets. While existing datasets have scaled significantly in volume, they frequently suffer from long-tail distributions dominated by repetitive, human-curated primi- tives. Consequently, RoboGene addresses this bottleneck not by proposing a new model architecture, but by autonomously generating high-quality, diverse, and physically grounded training tasks that unlock the full potential of these generalist policies. 2.2 Automated Task Generation Automating the synthesis of training tasks has evolved from rigid procedural templates [ Shridhar et al., 2020 ] to scalable, semantic-rich generation driven by Large Language Mod- els (LLMs). Pioneering frameworks [ Wang et al., 2023a; Ha et al., 2023; Chen et al., 2025 ] , such as RoboGen [ Wang et al., 2023b ] and GenSim 2 [ Hua et al., 2024 ] , leverage the code-generation and reasoning capabilities of LLMs to au- tonomously propose task descriptions, synthesize simulation environments, and formulate reward functions. This gener- ative paradigm has been further expanded to optimize cur- riculum design [ Ryu et al., 2025; Hu et al., 2025 ] , scale realistic task and scene synthesis [ Nasiriany et al., 2024; Ko et al., 2025; Zook et al., 2025; Ye et al., 2025 ] , and facili- tate sim-to-real transfer [ Gong et al., 2025; Jiang et al., 2025; Dai et al., 2025 ] . Despite these advancements, the primary objective of these frameworks lies in establishing automated simulation pipelines, where task generation is treated merely as a sub-component. Consequently, they often overlook the diversity of task generation, neglecting the critical role of data diversity in the large-scale pre-training of Vision-Language- Action (VLA) models. Additionally, the naive application of LLMs to robotics often suffers from grounding failures, where the generator produces tasks that violate physical laws, reference hallucinated assets, or exceed the kinematic con- straints of the robot embodiment.Furthermore, existing methods typically operate in an open-loop manner, lacking mechanisms to incorporate execution feedback or actively op- timize the statistical diversity of the generated dataset. In con- trast, RoboGene introduces a closed-loop agentic framework that integrates a diversity-driven sampling mechanism with a self-reflective evaluation loop. This ensures that generated tasks are not only semantically novel but also physically exe- cutable and valid, providing a robust and balanced data foun- dation for pre-training generalist agents. 3 Methodology 3.1 Overview We address the challenge of automating diverse task design for real-world robotic data collection. Formally, we define the task space as a tuple T = ⟨r,e,c,O,S⟩, where r ∈ R denotes the robot type (e.g., dual-arm), e ∈ E represents the scenario category, and c provides the task description con- text. The sets O ⊆ O and S ⊆ S denote the objects and ma- nipulation skills, respectively. Our objective is to synthesize physically valid and diverse task instances T ∈T to facilitate foundation model pre-training. To achieve this, we introduce RoboGene, an agentic frame- work that synthesizes tasks through a closed-loop process in- volving diversity-driven sampling, self-reflective generation, and memory-augmented refinement. As illustrated in Fig- ure 2, the generation process Φ transforms user prompt inputs into concrete task specifications. The overall task generation can be formalized as: T = Φ refine Φ gen Φ sample (E,O,S | H),R |M .(1) Here, the process initiates with the sampling function Φ sample , which selects a constraint set based on the global definition spaces (E,O,S ) and historical usage statistics H to ensure distribution diversity. The generator Φ gen then produces a task proposal based on these constraints and robot type R. Finally, the refiner Φ refine optimizes the task by leveraging a long-term memory moduleM that consolidates human feed- back knowledge. We provide the detailed algorithmic work- flow of RoboGene in Appendix A. 3.2 Diversity-Driven Task Space Sampling To mitigate the long-tail distribution issues and ensure bal- anced coverage of the task space, we employ a Least Fre- quently Used (LFU) sampling strategy guided by historical statistics H . We maintain usage counters u(·) for scenarios, objects, and skills, incrementing them upon the successful generation of a valid task. The sampling process operates hierarchically to construct a context for the current generation cycle t. First, we select the scenario category e t with the minimum usage count to prioritize under-explored environments, formulated as e t = arg min e∈E u(e). Following scenario selection, we address object and skill sampling. To prevent semantic incoherence Constraint Adherence Evaluator Statistical Information Proposal Generator Initial Task Descriptions Critique Feedback Long-Te r m M e m o r y Self-Reflection Refiner Retrieve scenesobjectsskills Classification: Single-arm robot twist open the oil bottle 푇 !"# 푇 !$% Critique: Entity OOD; substitute the oil bottle with beverage bottle . Ta sk N o v e lty Evaluator Classification: Critique: High novelty; introduces complex rotational primitives and rare contact mechanics . Real-World Robot Data Collection Physical Feasibility Evaluator Classification: Critique: Kinematically infeasible; needs dual-arm for stable counter-force.. RoboGene Human-in-the-Loop Cycles=10 Generate Key-Value Pair K: Task Description V: Refined Experience Generated Task Task Name: gpt_franka_dual_twist_cap Task Description: Twist the cap of the beverage bottle. Object Name: beverage bottle Skill Name: twist Location Information: Centered in bimanual workspace. Figure 2: Overview of RoboGene. The framework comprises three key components: (1) a diversity-driven sampling mechanism based on a Least Frequently Used (LFU) strategy; (2) a self-reflection mechanism that enhances task generation quality; and (3) a long-term memory module that consolidates Human-in-the-Loop (HITL) feedback. (e.g., “folding a microwave in the factory”), we do not sam- ple randomly from the global space. Instead, we first filter a relevant subsetO e t ⊂ O based on semantic similarity to the selected scenario e t . From this subset, we sample a fixed-size candidate set O t ⊂ O e t using the LFU strategy, favoring ob- jects with lower usage frequencies. A similar logic is applied to obtain the candidate skill set S t . Finally, the resulting tuple (e t ,O t ,S t ) is integrated as context into the prompt template to facilitate subsequent real-world task generation. By pro- viding a candidate set rather than a single forced choice, we endow the generative agent with the flexibility to compose logically sound tasks while maintaining high diversity. 3.3 Self-Reflection and Task Generation Upon determining the sampling constraints (e t ,O t ,S t ), the framework enters a generator-evaluator-improvement loop to produce a high-quality task specification T . Task Proposal Generation. The initial task T raw is synthe- sized by a Proposal Generator G(·), implemented via a Large Language Model (LLM). We construct a structured prompt injecting the sampled constraints, robot type r, and agent role definitions. For embodiments requiring spatial aware- ness, such as mobile manipulators, G(·) is instantiated as a Vision-Language Model (VLM). This allows the system to process scene images I 1 ,...,I k as supplementary input, grounding the generated task instructions in the physical lay- out to ensure spatial plausibility. The output T raw is formatted as a standardized JSON containing the robot type, scenario category, task description, object, and skill lists. Multi-Faceted Self-Reflection. Since T raw may contain hallucinations or physical infeasibilities, we employ a bank of specialized LLM-based evaluators to scrutinize the proposal. First, the Physical Feasibility Evaluator E phy (·) assesses whether the task is achievable given the robot typeR and the laws of physics. It critically analyzes the task from several dimensions, including kinematic feasibility (e.g., the overlap of dual-arm workspaces to ensure collaborative operations are possible). Furthermore, E phy (·) validates the logic of task de- composition to prevent incoherent actions and ensures com- pliance with fundamental physical laws. It also reviews syn- chronization control to address potential failures in time syn- chronization or force distribution. Based on this assessment, it provides a critique f phy . Second, the Novelty Evaluator E nov (·) measures the task’s complexity and novelty across four dimensions: object com- plexity, such as deformable objects, fluids, or optical chal- lenges; the precision of contact and interaction, including in-hand manipulation and tool use; the temporal and logical depth of the task, favoring long-horizon tasks with partial ob- servability; and the unstructured nature of the environment, such as dynamic targets or unpredictable disturbances. This node outputs feedback f nov to ensure the task presents a suf- ficient novelty. Third, the Constraint Adherence Evaluator E con (·) verifies that the task description aligns with the assigned scene type and correctly utilizes objects and skills from the sampled sets O t and S t without hallucination. Its output, f con , contains specific suggestions on modifying the task description, object names, and robotic skills to fit the scenario constraints better. These evaluators produce natural language critiques f phy ,f nov ,f con , which are passed to the Self-Reflection Re- finer E ref (·) to produce the revised task T ref . 3.4 Integrating Human Feedback via Memory To enable continuous system evolution and prevent the recur- rence of errors, we integrate a Human-in-the-Loop (HITL) mechanism coupled with a semantic Long-Term Memory moduleM. Human-in-the-Loop Feedback Consolidation.When tasks are dispatched for real-world execution, human oper- ators provide feedback not just as binary labels, but as rich natural language explanations detailing infeasibility reasons (e.g., “drawer handle requires left-hand actuation due to oc- clusion”). We employ an LLM-based summarizer to periodi- cally distill these specific feedback instances into generalized heuristic knowledge. These heuristics are stored inM as key- value pairs (K,V ), where the key is the semantic embedding of the task context and the value is the actionable guideline. Memory-Augmented Refinement. During the refinement phase, the system retrieves relevant knowledge from M to augment the self-reflection critiques. Specifically, we com- pute the embedding of the current task proposal and retrieve the top-k most similar heuristic guidelines via cosine similar- ity. These retrieved insights are injected into the context win- dow of the refiner E ref (·). This Retrieval-Augmented Gen- eration (RAG) approach [ Lewis et al., 2020 ] allows the agent to dynamically access a growing knowledge base of physical constraints and manipulation strategies, significantly enhanc- ing the qualities of generated tasks over time. We present task instances generated by RoboGene in Appendix B, and prompt templates for each node in Appendix C. 4 Experiments We evaluate RoboGene by addressing three pivotal research questions regarding the validity, diversity, and utility of the generated tasks. Specifically, we investigate: (1) whether individual tasks are linguistically comprehensible and phys- ically executable; (2) if the collective dataset exhibits suffi- cient coverage across scenarios, objects, and skills; and (3) the effectiveness of the collected data for downstream policy pre-training. To this end, we benchmark RoboGene against human experts, the rule-based method, and state-of-the-art Large Foundation Models (LFMs), specifically GPT-4o and Gemini 2.5 Pro. For each method, we generated a suite of 900 tasks spanning single-arm, dual-arm, and mobile manip- ulation domains, comprising 300 tasks for each robot cate- gory. We provide the implementation details for the baseline methods in Appendix E. 4.1 Individual Task Evaluation Evaluation Metrics. Quantifying the quality of open-ended robotic tasks remains an open challenge. We introduce six metrics to assess semantic and physical validity. Task Clarity and Type Consistency check for linguistic clarity and robot type alignment (e.g., dual-arm constraints). Logical Validity ensures semantic plausibility, penalizing contextually inap- propriate actions (e.g., industrial assembly in a kitchen). To measure grounding accuracy, we define Object Coverage and Skill Coverage as the ratio of generated entities present in the valid configuration setsO andS , respectively. Finally, Phys- ical Feasibility reports the average success rate of tasks exe- Table 1: Individual task evaluation results on the 900 tasks generated by each method. Higher scores indicate better task quality. Method Task (↑) Type (↑) Logical (↑) Obj. (↑) Skill (↑) Phys. (↑) Clarity Consist.ValidityCov.Cov.Feas. Rule-based0.55330.39000.40440.21450.48400.4811 Human0.96440.84110.94890.35800.17690.9478 GPT-4o 0.79220.85550.76220.31050.25420.7467 Gemini 2.5 Pro0.87910.86710.66910.21720.24580.6790 RoboGene0.99100.98760.98990.63230.91520.9899 Figure 3: Scenario diversity results across 8 pre-defined scenario categories. RoboGene demonstrates a highly balanced distribution. cuted via human teleoperation (5 trials per task). The scores for task clarity, type consistency, and logical validity are de- rived by averaging ratings from human evaluators, GPT-4o, and Gemini 2.5 Pro. We provide more detailed explanations of the metrics and their calculation methods in Appendix D. Evaluation Results. Table 1 demonstrates that Robo- Gene establishes a new state-of-the-art across all dimensions. While standard LFMs like GPT-4o and Gemini 2.5 Pro ex- hibit high task clarity, they suffer severely from hallucination, proposing objects and skills absent from the robot’s physical configuration. For instance, their object coverage scores are merely 0.3105 and 0.2172, respectively. In contrast, Robo- Gene effectively grounds generation within the available as- sets, achieving an object coverage of 0.6323 and a skill cov- erage of 0.8307. Furthermore, our self-reflection mechanism ensures physical grounding, outperforming the logical valid- ity and physical feasibility of human-designed tasks, whereas rule-based baselines fail to produce coherent physical exe- cutable behaviors. 4.2 Dataset Diversity Analysis We further evaluate the distributional properties of the gen- erated datasets, as diversity is critical for the generalization capability of VLA models. We analyze coverage across sce- narios, objects, skills, and task semantic descriptions. We in- clude ablation studies and additional experimental results in Appendix F and Appendix G, respectively. Results on Scenario Diversity. Figure 3 depicts the task distribution across eight semantic scenariosE . LFMs exhibit Figure 4: Skill diversity results given a total inventory of 118 skills. RoboGene significantly outperforms baselines in skill coverage and demonstrates a highly balanced distribution. 151015 Object Rank 0% 3% 6% 9% Proportion (%) (a) Object Usage Frequency RoboGene Human GPT-4o Gemini 2.5 Pro RoboGeneHumanGPT-4oGemini 2.5 Pro Method Name 0 300 600 900 Number of Objects 719 63.2% 407 35.8% 353 31.0% 247 21.7% (b) Total Object Coverage Figure 5: (a) Usage frequency of the top-15 objects across the 900 tasks generated by each method. (b) Total object count for tasks generated by each method, given a total inventory of 1,137 objects. a strong bias towards domestic settings, with GPT-4o concen- trating nearly 90% of tasks in home, kitchen, and office sce- narios, likely reflecting biases in their pre-training corpora. RoboGene rectifies this imbalance, achieving a uniform dis- tribution where no single category exceeds 20%. By explic- itly sampling under-represented environments (e.g., medical, industrial), our framework ensures the robotic agent encoun- ters diverse physical constraints, from rigid industrial tools to delicate laboratory equipment. Results on Skill Diversity. We analyzed the coverage of manipulation skills within the generated tasks in Figure 4. RoboGene generates tasks encompassing 118 distinct skills, achieving a 91.5% coverage of the skill spaceS , substantially outperforming GPT-4o (25.4%) and Gemini (24.6%) base- lines. The skill distribution in baseline methods is heavily skewed. For instance, simple primitives like pick and place dominate the human-designed dataset (>40%). Conversely, RoboGene maintains a balanced distribution where top skills constitute less than 5% each. This indicates our method suc- cessfully synthesizes complex, long-horizon behaviors (e.g., clean, arrange) alongside fundamental actions, fostering ro- bust policy learning. Results on Object Diversity. Figure 5 illustrates the fre- quency distribution of objects in generated tasks. Human- curated datasets (Blue dotted line) exhibit a sharp long-tail distribution, where a small set of dominant objects accounts for the majority of tasks, leaving the most objects under- represented. While direct usage of GPT-4o and Gemini 2.5 Pro partially mitigates this, they still suffer from “generation Table 2: Results on task semantic diversity. Lower scores indicate lower similarity between task and greater dataset diversity. Metric / MethodRoboGeneGPT-4oGemini 2.5 ProRule-basedHuman BLEU-1 (↓)28.9336.6138.9780.3272.80 BLEU-2 (↓) 6.629.849.7667.8265.62 BLEU-3 (↓)3.154.825.7362.1160.19 BLEU-4 (↓) 1.752.713.2356.4555.55 ROUGE-L (↓)0.19180.25120.27670.57860.4886 Cosine Similarity (↓)29.0734.6434.6369.8468.88 Camera DFR-OpenMedicalBoxDFR-RinseLettuce SUR-SortButtonsSUR-PourWater DFR-HandoverTape SUR-ReplenishShelves AGX-WaterPlantAGX-AssemblingCrucible Orbbec Gemini 335L AGX-OrganizeDesk AgileX Cobot Magic V2.0 Dual-Arm Franka Single-Arm UR-5e Orbbec Gemini 335L Orbbec Gemini 335L Figure 6: Real-world experimental setup. We employ three robots: the Single-arm UR-5e (SUR), the Dual-arm Franka (DFR), and the AgileX Cobot Magic V2.0 (AGX). saturation,” plateauing at approximately 350 and 250 unique objects. RoboGene overcomes this bottleneck through our diversity-driven sampling strategy, covering 719 distinct ob- jects, a 1.7× expansion over the strongest baseline. The re- sulting distribution is significantly flatter, ensuring that the VLA policy is exposed to a broad spectrum of physical enti- ties rather than overfitting to head classes. Results on Task Semantic Diversity. Finally, we quan- tify task description diversity using BLEU [ Papineni et al., 2002 ] , ROUGE-L [ Lin, 2004 ] , and cosine similarity met- rics in Table 2. Lower scores in this context denote less lexical overlap and higher diversity. RoboGene consistently records the lowest similarity scores, such as a BLEU-4 score of 1.75 compared to 2.71 for GPT-4o, indicating that our tasks are described using distinct phrasing and structures. In con- trast, rule-based and human-designed tasks exhibit high rep- etition (BLEU-1 > 70), reflecting a lack of task variety. This Grasp the yellow cup Place the pink cup on the holder Task:DFR-HangCupsVaried MugsPatterned TableclothAdded ClutterColored LightingParaphrased Commands Novel ObjectsIn-DistributionBackground VariationsStatic DistractorsIllumination ChangesInstruction Changes Insert them. Figure 7: Experimental setup for generalization evaluation on the DFR-HangCups task, including novel objects, background variations, static distractors, illumination changes, and language instruction changes. Table 3: Validation of data collection across three robots. We trained on three tasks and reported the success rates of various models. Method SUR-SortSUR-PourSUR-Replenish ButtonWaterShelves ACT90%55%60% DFR-HandDFR-OpenDFR-Rinse TapeBoxLettuce ACT90%85%60% AGX-Organize AGX-TranferAGX-Water CrucibleCupPlant π 0.5 60%70%45% confirms that RoboGene successfully leverages the linguistic generative power of LFMs to produce diverse instructions, effectively mitigating mode collapse. 4.3 Real-World Manipulation Task Evaluation In this section, we evaluate the effectiveness of our pipeline through extensive real-world experiments. We first validate the feasibility of the generated tasks on physical robots. Sub- sequently, we assess the contribution of RoboGene to large- scale pre-training and its generalization capabilities in un- structured environments. Validation of Generated Tasks. To verify the physi- cal feasibility of our task generation pipeline, we conducted single-task evaluations on three distinct robots: a single-arm UR-5e, a dual-arm Franka, and an AgileX mobile robot as shown in Figure 6. For each robot type, we generated three representative tasks using RoboGene and collected 250 tra- jectories per task. We train ACT [ Zhao et al., 2023b ] for the stationary robots and π 0.5 [ Intelligence et al., 2025 ] for the mobile robot and report the average success rates over 20 rollouts for each task. As reported in Table 3, the system achieved robust performance, including a 90% success rate on the SUR-SortButton task. These results confirm that RoboGene produces high- quality, physically executable tasks. Effectiveness of Pre-training Datasets. Building on this validation, we investigated the impact of large-scale pre- training on downstream performance.We constructed a dataset of 150 single-arm and 150 dual-arm tasks generated from each method. As shown in Figure 8, we pre-trained the π 0 [ Black et al., 2024 ] on these datasets and subse- quently fine-tuned them on five unseen dual-arm tasks with 15 demonstrations per task, reporting the average success rate over 20 rollouts. Our results indicate a significant perfor- Figure 8: Success rates on 5 tasks after fine-tuning. Models were first pre-trained using data collected from the 300 tasks generated by each method. The number appended to each method name indicates the number of fine-tuning epochs. Table 4: Generalization results on the DFR-HangCups task. Method NovelBackgroundStaticIlluminationInstruction Avg. ObjectsVariationsDistractorsChangesChanges Human10%20%20%15%10%15% GPT-4o 10%5%10%5%10%8% Gemini 2.5 Pro10%15%10%10%15%12% RoboGene30%25%30%25%35%29% mance disparity between RoboGene and LLM-based base- lines. Policies pre-trained on data generated by LFMs (GPT- 4o, Gemini) struggled to adapt to new tasks, often yielding success rates below 20% or failing completely. This sug- gests that LFMs fail to generate tasks with sufficient physi- cal feasibility or diversity. In contrast, scaling the diversity of RoboGene tasks directly correlated with improved transfer learning. The RoboGene with 30 epochs consistently outper- formed baselines, achieving 40% success on the challenging DFR-GrillSkewers task. Generalization to Unseen Scenarios. To rigorously eval- uate the robustness of the learned policies, we conducted evaluation for task DFR-HangCups under five unseen envi- ronmental conditions: (1) novel objects, (2) background vari- ations, (3) static distractors, (4) illumination changes, and (5) instruction changes. As reported in Table 4, policies pre-trained on the RoboGene dataset demonstrate superior generalization capabilities compared to the results based on GPT-4o and Gemini 2.5 Pro. In the illumination changes scenario, RoboGene maintains a 35% success rate, approx- imately tripling the performance of Gemini 2.5 Pro (10%). These results suggest that RoboGene facilitates the acqui- sition of robust, transferable features, significantly mitigat- ing the performance degradation typically observed when de- ploying policies in real-world, unseen environments. 5 Conclusion In this work, we address the critical data diveristy bot- tleneck in robotic learning by introducing RoboGene, an agentic framework that automates the generation of diverse, physically grounded manipulation tasks. By integrating a diversity-driven Least Frequently Used (LFU) sampling strat- egy with a multi-faceted self-reflection mechanism, Robo- Gene effectively overcomes the hallucination issues inher- ent in large foundation models. Our extensive quantitative analysis and real-world experiments demonstrate that Robo- Gene not only generates datasets with significantly higher ob- ject and skill coverage than state-of-the-art baselines but also facilitates the training of VLA policies with superior zero- shot generalization to unseen environments. We believe this scalable, automated pipeline offers a promising path toward general-purpose embodied intelligence and will open-source our resources to support future research. References [ Bharadhwaj et al., 2024 ] Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In ICRA. IEEE, 2024. [ Black et al., 2024 ] Kevin Black, Noah Brown, Danny Driess, Ad- nan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π 0 : A vision- language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024. [ Bu et al., 2025a ] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. Agibot world colosseo: A large-scale manip- ulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025. [ Bu et al., 2025b ] Qingwen Bu,Yanting Yang,Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. In Proceedings of Robotics: Science and Systems (RSS), 2025. [ Buamanee et al., 2024 ] ThanpimonBuamanee,Masato Kobayashi, Yuki Uranishi, and Haruo Takemura.Bi-act: Bilateral control-based imitation learning via action chunking with transformer. In AIM. IEEE, 2024. [ Cheang et al., 2025 ] Chilam Cheang, Sijin Chen, Zhongren Cui, Yingdong Hu, Liqun Huang, Tao Kong, Hang Li, Yifeng Li, Yux- iao Liu, Xiao Ma, et al. Gr-3 technical report. arXiv preprint arXiv:2507.15493, 2025. [ Chen et al., 2025 ] Shuaixing Chen, Ruolin Ye, Saurabh Dingwani, Pooyan Fazli, Hasti Seifi, and Tapomayukh Bhattacharjee. Rcare- gen: An interface for scene and task generation in rcareworld. In 2025 20th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 1274–1278. IEEE, 2025. [ Chi et al., 2023 ] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. RSS, 2023. [ Comanici et al., 2025 ] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. [ Dai et al., 2025 ] Liu Dai, Haina Wang, Weikang Wan, and Hao Su. Manitaskgen: A comprehensive task generator for benchmark- ing and improving vision-language agents on embodied decision- making. arXiv preprint arXiv:2505.20726, 2025. [ Ebert et al., 2022 ] Frederik Ebert, Yanlai Yang, Karl Schmeck- peper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In RSS, 2022. [ Fan et al., 2025 ] Shichao Fan, Kun Wu, Zhengping Che, Xinhua Wang, Di Wu, Fei Liao, Ning Liu, Yixue Zhang, Zhen Zhao, Zhiyuan Xu, et al. Xr-1: Towards versatile vision-language- action models via learning unified vision-motion representations. arXiv preprint arXiv:2511.02776, 2025. [ Fu et al., 2024 ] Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mo- bile ALOHA: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. In 8th CoRL, 2024. [ Gong et al., 2025 ] Ran Gong, Xiaohan Zhang, Jinghuan Shang, Maria Vittoria Minniti, Jigarkumar Patel, Valerio Pepe, Riedana Yan, Ahmet Gundogdu, Ivan Kapelyukh, Ali Abbas, et al. Anytask: an automated task and data generation framework for advancing sim-to-real policy learning.arXiv preprint arXiv:2512.17853, 2025. [ Ha et al., 2023 ] Huy Ha, Pete Florence, and Shuran Song. Scal- ing up and distilling down: Language-guided robot skill acqui- sition. In Conference on Robot Learning, pages 3766–3777. PMLR, 2023. [ Hou et al., 2025 ] Chengkai Hou, Kun Wu, Jiaming Liu, Zheng- ping Che, Di Wu, Fei Liao, Guangrun Li, Jingyang He, Qiuxuan Feng, Zhao Jin, et al. Robomind 2.0: A multimodal, bimanual mobile manipulation dataset for generalizable embodied intelli- gence. arXiv preprint arXiv:2512.24653, 2025. [ Hu et al., 2025 ] Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian-Guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. In KDD, 2025. [ Hua et al., 2024 ] Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, and Lirui Wang. Gen- sim2: Scaling robot data generation with multi-modal and rea- soning llms. arXiv preprint arXiv:2410.03645, 2024. [ Hurst et al., 2024 ] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [ Intelligence et al., 2025 ] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.π 0.5 : a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [ Jiang et al., 2025 ] Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Jim Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexter- ous manipulation via imitation learning. In ICRA. IEEE, 2025. [ Khazatsky et al., 2024 ] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun- liang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024. [ Kim et al., 2024 ] Moo Jin Kim, Karl Pertsch, Siddharth Karam- cheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Open- vla: An open-source vision-language-action model. In Confer- ence on Robot Learning (CoRL), 2024. [ Ko et al., 2025 ] Dogyu Ko, Chanyoung Yeo, Daeho Kim, Jaeho Kim, and Hyoseok Hwang. Gaia: Generating task instruction aware simulation grounded in real contexts using vision-language models. RAL, 2025. [ Lee et al., 2025 ] Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, et al. Molmoact: Action reasoning models that can reason in space. In CoRL 2025 Robot Data Workshop, 2025. [ Lewis et al., 2020 ] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küt- tler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Ad- vances in neural information processing systems, 33:9459–9474, 2020. [ Lin, 2004 ] Chin-Yew Lin. Rouge: A package for automatic evalu- ation of summaries. In Text summarization branches out, 2004. [ Liu et al., 2025a ] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual ma- nipulation. In ICLR, 2025. [ Liu et al., 2025b ] Zhuoyang Liu, Jiaming Liu, Jiadong Xu, Nuowei Han, Chenyang Gu, Hao Chen, Kaichen Zhou, Ren- rui Zhang, Kai Chin Hsieh, Kun Wu, et al.Mla: A mul- tisensory language-action model for multimodal understand- ing and forecasting in robotic manipulation.arXiv preprint arXiv:2509.26642, 2025. [ Nasiriany et al., 2024 ] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu.Robocasa: Large-scale simula- tion of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024. [ O’Neill et al., 2024 ] Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In ICRA. IEEE, 2024. [ Papineni et al., 2002 ] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In ACL, 2002. [ Pfeifer and Iida, 2004 ] Rolf Pfeifer and Fumiya Iida. Embodied artificial intelligence: Trends and challenges. Lecture notes in computer science, pages 1–26, 2004. [ Qu et al., 2025 ] Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. In RSS, 2025. [ Ryu et al., 2025 ] Kanghyun Ryu, Qiayuan Liao, Zhongyu Li, Payam Delgosha, Koushil Sreenath, and Negar Mehr.Cur- ricullm: Automatic task curricula design for learning complex robot skills using large language models. In ICRA. IEEE, 2025. [ Shridhar et al., 2020 ] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpret- ing grounded instructions for everyday tasks. In CVPR, 2020. [ Walke et al., 2023 ] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In CoRL. PMLR, 2023. [ Wang et al., 2023a ] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang.Gensim: Generating robotic sim- ulation tasks via large language models.arXiv preprint arXiv:2310.01361, 2023. [ Wang et al., 2023b ] Yufei Wang, Zhou Xian, Feng Chen, Tsun- Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erick- son, David Held, and Chuang Gan. Robogen: Towards unleash- ing infinite data for automated robot learning via generative sim- ulation. arXiv preprint arXiv:2311.01455, 2023. [ Wen et al., 2025a ] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient vision- language-action models for robotic manipulation. IEEE Robotics and Automation Letters, 2025. [ Wen et al., 2025b ] Junjie Wen, Yichen Zhu, Minjie Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Xiaoyu Liu, Chaomin Shen, Yaxin Peng, and Feifei Feng. Diffusionvla: Scaling robot foun- dation models via unified diffusion and autoregression. In ICML, 2025. [ Wu et al., 2025a ] Kun Wu, Chengkai Hou, Jiaming Liu, Zheng- ping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manip- ulation. In RSS, 2025. [ Wu et al., 2025b ] Kun Wu, Ning Liu, Zhen Zhao, Di Qiu, Jinming Li, Zhengping Che, Zhiyuan Xu, and Jian Tang. Learning from imperfect demonstrations with self-supervision for robotic ma- nipulation. In ICRA, 2025. [ Wu et al., 2025c ] Kun Wu, Yichen Zhu, Jinming Li, Junjie Wen, Ning Liu, Zhiyuan Xu, and Jian Tang. Discrete policy: Learning disentangled action space for multi-task robotic manipulation. In ICRA, 2025. [ Wu et al., 2025d ] Shihan Wu, Xuecheng Liu, Shaoxuan Xie, Pengwei Wang, Xinghang Li, Bowen Yang, Zhe Li, Kai Zhu, Hongyu Wu, Yiheng Liu, et al. Robocoin: An open-sourced bi- manual robotic data collection for integrated manipulation. arXiv preprint arXiv:2511.17441, 2025. [ Xia et al., 2025a ] Wenke Xia, Ruoxuan Feng, Dong Wang, and Di Hu. Phoenix: A motion-based self-reflection framework for fine-grained robotic action correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 6981–6990, June 2025. [ Xia et al., 2025b ] Wenke Xia, Yichu Yang, Hongtao Wu, Xiao Ma, Tao Kong, and Di Hu.Robotic policy learning via human-assisted action preference optimization. arXiv preprint arXiv:2506.07127, 2025. [ Ye et al., 2025 ] Weirui Ye, Fangchen Liu, Zheng Ding, Yang Gao, Oleh Rybkin, and Pieter Abbeel. Video2policy: Scaling up ma- nipulation tasks in simulation through internet videos. arXiv preprint arXiv:2502.09886, 2025. [ Ze et al., 2024 ] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: General- izable visuomotor policy learning via simple 3d representations. In RSS, 2024. [ Zhao et al., 2023a ] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. [ Zhao et al., 2023b ] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023. [ Zheng et al., 2025 ] Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, et al. X-vla: Soft-prompted trans- former as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274, 2025. [ Zitkovich et al., 2023 ] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action mod- els transfer web knowledge to robotic control. In CoRL. PMLR, 2023. [ Zook et al., 2025 ] Alex Zook, Fan-Yun Sun, Josef Spjut, Valts Blukis, Stan Birchfield, and Jonathan Tremblay. Grs: Generat- ing robotic simulation tasks from real-world images. In CVPR, 2025. A Implementation Details of RoboGene RoboGene operates as a closed-loop agentic framework de- signed to autonomously generate high-quality, physically grounded, and diverse robotic manipulation tasks within real- world environments. The framework is founded upon three synergistic components devised to address the inherent chal- lenges of automated data generation. First, we employ a diversity-driven sampling mechanism based on a Least Fre- quently Used (LFU) strategy to mitigate the long-tail dis- tribution problem often observed in robotic datasets. This mechanism actively steers the generative agent toward under- explored regions of the task space, prioritizing interactions with rarely utilized objects and skills. Second, to suppress hallucinations and ensure robust physical grounding, we in- corporate a self-reflection mechanism. This module utilizes three specialized evaluators built on foundation models to in- dependently assess constraint adherence, novelty, and physi- cal feasibility. These evaluators rigorously scrutinize gener- ated task proposals to provide actionable feedback. Third, the framework features a long-term memory module that consol- idates Human-in-the-Loop (HITL) feedback. By assimilat- ing corrections derived from task modifications and execution failures encountered during real-world collection, the system progressively refines its understanding of physical constraints and continuously enhances generation quality over time. Formally, the generation process requires the specification of a robot embodiment, denoted as r ∈ R, which is selected from three primary categories: single-arm robots, dual-arm robots, and mobile dual-arm robots. To define the seman- tic boundaries of the generation space, the system accepts a global scenario setE , an object libraryO, and a skill library S . These definitions can be provided by the user as structured text files to align with specific hardware availability or data collection needs. In our specific implementation, we define |E| = 8 distinct scenario categories, encompassing Domes- tic, Office, Education, Laboratory, Kitchen, Industry, Retail, and Medical environments. The object library O comprises 1,137 distinct physical entities, while the skill libraryS con- tains 118 manipulation primitives. This task space allows for high flexibility; users can customize the libraries based on their specific experimental setup or leverage Large Founda- tion Models (LFMs) to autonomously expand the semantic space. Additionally, RoboGene provides a remark interface for natural language constraints, allowing users to input spe- cific requirements such as “generate long-horizon tasks com- posed of 3-4 sub-goals” or “prioritize tasks requiring dexter- ous manipulation,” which function as soft prompts during the generation phase. The core execution logic of RoboGene is formalized in Al- gorithm 1. The process is structured as an iterative loop that transitions through sampling, generation, reflection, and re- finement phases to produce valid task instances. We initialize the system by setting usage counters u(·) and history statistics H to zero. In the Diversity-Driven Sampling phase, the sys- tem identifies the target scenario e t with the minimum usage count to prioritize under-represented environments. It then filters context-relevant objects O e t and skills S e t based on semantic similarity, subsequently sampling candidate sets O t Algorithm 1 RoboGene Iterative Task Generation Process Require: Robot Type r, Scenario Set E , Object Library O, Skill LibraryS , History Statistics H , Long-Term Mem- oryM 1: Initialize usage counters u(e),u(o),u(s)← 0 for all e∈ E,o∈O,s∈S 2: Initialize history statistics H = u(e),u(o),u(s) 3: while Generation Loop is Active do 4:# Diversity-Driven Sampling 5:Select target scenario e t ← arg min e∈E u(e) to priori- tize under-explored environments 6:Filter context-relevant subsetsO e t ⊂ O andS e t ⊂ S based on semantic similarity to e t 7:Sample candidate sets O t ⊂ O e t and S t ⊂ S e t using the LFU strategy based on H 8:# Task Proposal Generation 9:Generateinitialtaskproposal T raw ← Φ gen (r,e t ,O t ,S t ) 10:# Self-Reflection Evaluation 11: f nov ← E nov (T raw ) Assess novelty and complexity 12: f con ← E con (T raw ) Verify adherence to sampled constraints 13: f phy ← E phy (T raw ) Check kinematic and physical feasibility 14:# Memory-Augmented Refinement 15:Retrieverelevantheuristicguidelines v ← Retrieve(M,T raw ) 16:Refinetaskproposal T ref ← Φ refine (T raw ,f nov ,f con ,f phy ,v) 17:if T ref is valid then 18:Add T ref to the dataset 19:Update history statistics H 20:end if 21:# Human-in-the-Loop Update 22:if Real-world execution feedback is available then 23:Consolidate feedback and update MemoryM with new failure cases or insights 24:end if 25: end while and S t using the LFU strategy. Following sampling, the generator Φ gen synthesizes an ini- tial task proposal T raw based on the robot type r and sam- pled constraints. This proposal immediately enters the Self- Reflection Evaluation phase, where it is scrutinized by three evaluators to produce specific critiques: f nov for novelty, f con for constraint adherence, and f phy for physical feasibil- ity. In the Memory-Augmented Refinement phase, the system retrieves relevant heuristic guidelines v from the long-term memory M. A refiner module Φ refine then synthesizes a finalized task T ref by integrating the original proposal, the evaluator critiques, and the retrieved memory. The task T ref is added to the dataset, and the history H is updated only if it passes the validity check. Finally, if real-world execution feedback is available, it is consolidated into the memoryM to prevent future failures. The resulting tasks are structured in a standardized JSON format, facilitating direct translation into execution code for both simulation and real-world de- ployment. B Generated Task Instances To demonstrate the versatility of RoboGene, we present rep- resentative JSON outputs for single-arm, dual-arm, and mo- bile manipulation tasks. These examples illustrate how the system generates specific object layouts and skill sequences. Instance 1: Single-Arm Task This long-horizon task in- volves a sequence of cleaning operations, requiring the robot to manage state changes (dirty to clean) and precise pick-and- place actions. Single-Arm Task "task_name": "robogene_single_clean_spill", "task_description": "The robot uses a tissue to clean spilled drink under the display unit: first, throw the used tissue into the trash bin, then take a new tissue to wipe the table .", "language_instruction": "Clean the spilled drink under the display unit: throw the used tissue into the trash bin first, then wipe the table with a new tissue.", "objects": [ "C&S Tissue Pack", "Countertop Display Unit", "Beverage Bottle", "Trash Bin" ], "skills": ["clean"], "scene_layout": "C&S Tissue Pack": "Lower Right", "Countertop Display Unit": "Upper Center", "Beverage Bottle": "Toppled on the table beside the display unit", "Trash Bin": "Lower Right, near the tissue" , "task_context": "The used tissue is first deposited into the bin. Subsequently, a new tissue is retrieved to wipe the liquid on the desktop and the area beneath the shelf." Instance 2: Dual-Arm Task This task highlights the sys- tem’s ability to generate bimanual coordination tasks where one arm acts as a stabilizer while the other performs a preci- sion tool-use operation. Dual-Arm Task "task_name": "robogene_dual_rotate_screwdriver", "task_description": "The left arm stabilizes a wooden block containing a large Phillips screw. The right arm grasps a screwdriver, aligns its tip with the screw head groove, inserts it, and rotates the wrist 90 degrees clockwise while applying downward pressure to tighten.", "language_instruction": "Hold the wooden block firmly with your left arm. Pick up the screwdriver with your right arm, insert its tip into the screw head, and then rotate it clockwise 90 degrees to tighten it.", "objects": [ "Screwdriver", "Wooden Block with Screw" ], "skills": ["rotate"], "scene_layout": "Screwdriver": "Right Arm Workspace: Middle Right", "Wooden Block with Screw": "Left Arm Workspace : Middle Left" , "task_context": "The block is placed flat with the screw head facing up and loose. The screwdriver handle is oriented towards the right arm, placed horizontally." Instance 3: Mobile Manipulation Task This task ne- cessitates navigation and logical reasoning to sort objects based on size, demonstrating spatial planning across a larger workspace. Mobile Manipulation Task "task_name": "robogene_sort_items", "task_description": "Classify cardboard boxes of varying sizes and deposit them into the corresponding blue storage bins.", "language_instruction": "Sort the cardboard boxes by size and place them into the corresponding blue storage bins.", "objects": [ "Cardboard Box (No White Edge) 4#", "Blue Storage Bins" ] "skills": [ "pick", "place", "identify", "navigate" ], "scene_image": "industry_scenario.jpg", "scene_layout": "Small Cardboard Box": "On conveyor belt, near left side", "Large Cardboard Box": "On conveyor belt, near right side", "Small Blue Storage Bins": "On the right side of the table", "Large Blue Storage Bins": "On the right side of the table", , "steps": [ "step": 1, "skill": "pick", "action": "Pick up a cardboard box", "requirement": "Select a target box" , "step": 2, "skill": "identify", "action": "Identify box size", "requirement": "Classify based on dimensions ." , "step": 3, "skill": "navigate", "action": "Navigate to corresponding bin", "requirement": "Move to correct storage location" , "step": 4, "skill": "place", "action": "Place box in the blue storage bin ", "requirement": "Ensure stable placement" ], C Prompt Structure for Agentic Nodes The efficacy of RoboGene relies on carefully engineered prompts that guide the Large Foundation Models (LFMs) within each node of the framework. Below, we outline the structural composition of the prompts used for each node. Proposal Generator Prompt. The generator prompt is de- signed to synthesize the initial task draft T raw . It integrates the diversity constraints derived from the sampling module. The prompt structure includes: (1) Role Definition: You are an expert robotic task designer specializing in [Robot Type] manipulation. (2) Constraint Injection: You must generate a task within the [Scenario Name] environment. You are re- stricted to using objects from this candidate list: [List O t ] and skills from: [List S t ]. (3) Format Specification: Out- put the task strictly in the defined JSON schema, ensuring all fields such as spatial layout and linguistic instructions are populated. Proposal Generator Prompt "system": "You are an expert in bimanual robotic manipulation and task design. Your role is to generate novel, physically feasible, and diverse manipulation tasks for tabletop environments. You must ensure all tasks are kinematically plausible, stable, and involve meaningful bimanual coordination.", "user": "Task: Design diverse dual-arm manipulation scenarios based on the following specifications: 1. Operational Constraints and Resources: (1) Output format requirements in JSON Structure: Each task must be a JSON object containing: - 'task_name': Following the pattern ' RoboGene_robot_action'. - 'task_description': A detailed description. - 'language_instruction': A precise English command passed into the VLA model. - 'objects' - 'skills' - 'scene_layout': The tabletop is divided into two symmetrical 3x3 grids corresponding to the left and right arm workspaces. The indexing follows a row-major order. - "task_context": explains how the object is positioned and its physical state. 2. Permissible Assets: Only items listed in 'asset.csv' and actions defined in 'Skills.txt' are allowed. 3. Reference Example: "task_name": "robogene_dual_pick_and _place_white_black_cup", "task_description": "Both robotic arms are used to manipulate two cups of different colors. A white cup and a black cup are initially placed in the upper-right area of the table. The robot visually calibrates the objects, grasps the cups, and places them sequentially on the table in a specified order, ensuring a rim-to-rim distance of 15 centimeters between the two cups.", "language_instruction": "Using both arms , pick up the white cup and the black cup from the upper-right area of the table and place them on the table in the specified order, keeping a 15 cm distance between the rims of the two cups.", "objects": [ "White cup", "Black cup" ], "skills": [ "pick_and_place" ], "scene_layout": "White cup": "Right Arm Workspace: Upper Right", "Black cup": "Right Arm Workspace: Upper Right" , "task_context": "Both cups are upright and stable on the tabletop. They are initially placed close to each other in the upper-right area. The table surface is flat and unobstructed, allowing sequential grasping and precise placement with a fixed rim-to-rim distance." 4. Safety and Feasibility Constraints: (1) Avoid trivial tasks: Avoid creating tasks that can be completed with just one arm. (2) Moderated Force: Contact-based actions (e.g., pressing) must use moderated force to ensure stability and avoid hardware damage. (3) Collision Avoidance: Strictly avoid high-impact trajectories . 5. Final Checklist: - Asset Consistency: Strictly adhere to provided asset and skill taxonomies. - Logical Flow: Tasks must be physically executable and logically sequenced for dual-arm coordination. - Diversity: Maximize the variety of object-skill combinations." Task Novelty Evaluator Prompt This evaluator ensures the generated task contributes to the dataset’s diversity. The prompt instructs the model to: (1) Analyze Complexity: As- sess the proposed task for interaction richness (e.g., tool use, deformability) versus simple pick-and-place primitives. (2) Check Redundancy: Compare the task logic against common patterns to identify trivial or repetitive designs. (3) Output Decision: Return a binary decision (Yes / No) and a critique describing the novelty level. Task Novelty Evaluator Prompt "system": "You are an expert evaluator for robotic manipulation task datasets. Your role is to analyze task distributions, identify potential imbalances in skill usage, and assess whether a given set of tasks is overly biased toward common manipulation paradigms or lacks novelty and rare skill types. Your analysis should be objective, concise, and suitable for inclusion in an academic context .", "user": "You will be provided with structured task descriptions. Based on the given information, analyze whether there exists a task distribution imbalance, with particular attention to skill diversity, novelty, and paradigm bias. 1. Task Analysis Procedure: (1)Task Comprehension: Carefully examine each task, including the task name, task description, language instruction, skill name and so on. (2)Skill Distribution Analysis: Count the frequency of each skill across the task set. Evaluate whether skill usage is balanced or dominated by a small number of common skills. Consider whether additional, less frequent skills could be introduced to improve overall balance. (3)Novelty and Rarity Assessment: Determine whether the tasks involve novel or rare skill types, or whether they rely exclusively on well-established manipulation skills without introducing new operational challenges. (4)Paradigm Bias Evaluation: Analyze whether the tasks are biased toward conventional manipulation paradigms (e.g., repetitive pick-and-place patterns), thereby lacking diversity or uniqueness. (5)Final Judgment and Explanation: If a task distribution imbalance is identified, mark the result as "No". If no imbalance is found, mark the result as "Yes". Provide a brief explanation supporting your judgment, including key observations and reasoning. 2. Input Data Structure: The input will be provided in a JSON-like format and may include: - 'task_name': Following the pattern ' RoboGene_robot_action'. - 'task_description': A detailed description. - 'language_instruction': A precise English command passed into the VLA model. - 'objects' - 'skills' - 'scene_layout': The tabletop is divided into two symmetrical 3x3 grids corresponding to the left and right arm workspaces. The indexing follows a row-major order. - "task_context": explains how the object is positioned and its physical state. 3. Output Format: After completing your analysis, produce a response strictly in the following format: - Feasibility: [Yes / No] - Analysis: [A concise justification grounded in the evaluation dimensions above] 4. Examples: Example 1 Input: "task_name": "RoboGene_dual_Franka_pick_and _place_white_and_black_cup", "task_description": "Both robotic arms are used to manipulate two cups of different colors. A white cup and a black cup are initially placed in the upper-right area of the table. The robot visually calibrates the objects, grasps the cups using coordinated dual-arm manipulation, and places them sequentially on the table in a specified order. The final placement must ensure a rim- to-rim distance of 15 centimeters between the two cups.", "language_instruction": "Use both arms to pick up the white cup and the black cup from the upper-right area of the table and place them on the table in the specified order, ensuring a 15 cm distance between the rims of the two cups.", "objects": [ "White cup", "Black cup" ], "skills": [ "pick_and_place" ], "scene_layout": "White cup": "Right Arm Workspace: Upper Right", "Black cup": "Right Arm Workspace: Upper Right" , "task_context": "Both cups are upright and stable on a flat tabletop surface. They are initially positioned close to each other in the upper-right area. No external obstacles are present. During placement, the cups must remain upright, and the rim-to-rim distance between the white cup and the black cup must be maintained at 15 centimeters." Output: - Feasibility: No - Analysis: The task set is overly dominated by the pick_and_place skill, leading to a pronounced imbalance in skill utilization. More advanced manipulation skills, including assembly, rotation, and force control, are entirely absent. This lack of skill diversity limits the representational richness and execution difficulty of the task set, and incorporating non-pick-and-place skills would be essential for achieving a more balanced and realistic evaluation." Constraint Adherence Evaluator Prompt This node en- forces the strict boundaries set by the sampling strategy. The prompt requires the model to: (1) Verify Entities: Confirm that all objects and skills mentioned in the task description exist strictly within the provided candidate sets O t and S t . (2) Check Hallucination: Identify any invented objects or phys- ical properties not present in the input metadata. (3) Consis- tency Check: Ensure the task description logically aligns with the selected scenario category. Constraint Adherence Evaluator Prompt "system": "You are an expert task consistency reviewer for structured dual-arm robotic manipulation tasks. Your role is to critically evaluate, validate , and refine task definitions by strictly verifying their alignment with provided object and skill inventories. You must operate as a deterministic critic, not a creative generator. You must: - Rely only on the provided reference lists . - Avoid assumptions, commonsense inference, or synonym expansion. - Apply exact string matching when validating objects and skills. - Produce clear, actionable, and academically rigorous feedback suitable for downstream automated correction or benchmarking.", "user": "You will be provided with structured task descriptions. Based on the given information, follow the requirements below. 1. Consistency Verification (1) Skill Validation: - Verify whether the skill name exactly matches an entry in the skill list. (2) Object Validation: - Verify whether the object name exactly matches an entry in the object asset list. (3) JSON Validation: - Check whether the JSON-like format is strictly followed. Validation Rules: - Any object or skill not found verbatim in the reference lists must be marked as invalid. - Do not assume common objects, aliases, translations, or semantic equivalents are acceptable. - Exact string-level correspondence is mandatory. 2. Input Data Structure: The input will be provided in a JSON-like format and may include: - 'task_name': Following the pattern ' RoboGene_robot_action'. - 'task_description': A detailed description. - 'language_instruction': A precise English command passed into the VLA model. - 'objects' - 'skills' - 'scene_layout': The tabletop is divided into two symmetrical 3x3 grids corresponding to the left and right arm workspaces. The indexing follows a row-major order. - "task_context": explains how the object is positioned and its physical state. 3. Output Format: After completing your analysis, produce a response strictly in the following format: - Feasibility: [Yes / No] - Analysis: [A concise justification grounded in the evaluation dimensions above] 4. Examples: Example 1 Input: "task_name":"RoboGene_dual_Franka _rotate_ and_place_blackboard", "task_description": "The robot manipulates a wooden blackboard by rotating it 90 degrees from its initial orientation and placing it at the center of the table. After rotation and placement, the long side of the board must be aligned parallel to the table edge.", "language_instruction": "Rotate the wooden board by 90 degrees and place it at the center of the table with its long side parallel to the table edge.", "objects": [ "Blackboard" ], "skills": [ "rotate", "place" ], "scene_layout": "Blackboard": "Center of the table" , "task_context": "The blackboard is initially placed flat on the table with its long side perpendicular to the table edge. The board is stable, unobstructed, and graspable. During manipulation, the board must remain in contact with the tabletop, and the final placement requires the board's long edge to be parallel to the table edge." Output: - Feasibility: Yes - Analysis: Both the skill name and the object name exactly match entries in the corresponding reference lists. No inconsistencies were detected. Example 2 Input: "task_name":"RoboGene_dual_Franka _rotate _and_place_wooden_board_correctly", "task_description": "The robot manipulates a wooden board by grasping it, rotating it by 90 degrees from its initial orientation, and placing it at the center of the table. After placement, the long side of the wooden board must be aligned parallel to the table edge.", "language_instruction": "Rotate the wooden board by 90 degrees and place it at the center of the table, ensuring that its long side is parallel to the table edge.", "objects": [ "Wooden board" ], "skills": [ "Rotate and Place" ], "scene_layout": "Wooden board": "Center of the table" , "task_context": "The wooden board is initially placed flat on the tabletop with its long side perpendicular to the table edge . The board is stable, unobstructed, and fully graspable. The task requires controlled rotation followed by precise placement so that the final orientation has the long side parallel to the table edge." Output: - Feasibility: No - Analysis: The skill name "Rotate and Place" does not exist in the provided skill list. Suggested replacements include: "rotate " from the reference list. The object name "Wooden Board" is invalid and does not appear in the object asset list. Suggested replacements include: " blackboard" from the reference list." Physical Feasibility Evaluator Prompt This evaluator fo- cuses on kinematic and dynamic plausibility. The prompt di- rects the model to: (1) Kinematic Validation: Determine if the [Robot Type] is physically capable of the described motions (e.g., reachability, payload). (2) Interaction Logic: Analyze if the object interactions are physically sound (e.g., a single arm cannot unscrew a floating object; it requires a fixture or a second arm). (3) Safety and Stability: Flag potential collision risks or unstable states defined in the generated layout. Robotic Task Feasibility Evaluator Prompt "system": "You are a rigorous Robotic Task Feasibility Analyst with expertise in dual-arm manipulation systems. Your role is to evaluate the physical and logical feasibility of structured robotic task sequences. You must adopt a strictly hardware-first perspective, prioritizing mechanical constraints, kinematic limits, and physical laws over theoretical or symbolic task feasibility.", "user": "You will be provided with structured task descriptions in a JSON-like format. Your task is to critically analyze whether the described robotic task can be executed by a real dual-arm robotic system. 1. Evaluation Criteria: You must assess the task along the following five dimensions: (1) Kinematic Feasibility: The task must admit valid inverse kinematics solutions. Target poses must lie within the robot's reachable workspace and must not induce singular configurations or joint limit violations. (2) Workspace Coordination: For dual-arm tasks, the shared workspace of both manipulators must sufficiently cover the task region. If coordinated manipulation is required but the objects lie outside the common interaction zone, the task should be considered infeasible. (3) Logical Decomposition: Multi-step task sequences must be logically coherent and physically connected. For example, a grasping action must be followed by a placement or manipulation step with adequate spatial clearance and consistent object state transitions. (4) Physical Laws and Constraints: The task must respect fundamental physical principles, including gravity, friction, stability, and contact mechanics. Examples of infeasible tasks include attempting to stably grasp a spherical object with parallel-jaw grippers or stacking objects with inherently unstable geometries. (5) Synchronous Control and Force Limits: The task must fall within the operational limits of dual-arm synchronization and force control. Tasks requiring unrealistically precise force balancing, excessive contact forces, or near- perfect temporal synchronization should be judged infeasible. 2. Input Data Structure: The input will be provided in a JSON-like format and may include: - Sub-task names - Natural language instructions - Skill identifiers - Object attributes and spatial or positional information 3. Output Format: After completing your analysis, produce a response strictly in the following format: - Feasibility: [Yes / No] - Analysis: [A concise justification grounded in the evaluation dimensions above] 4. Examples: Example 1 Input: "task_name": "RoboGene_dual_Franka _rotate_and_place_board", "task_description": "The robot manipulates a wooden board by rotating it 90 degrees from its initial orientation and placing it at the center of the table. After placement, the long side of the board must be aligned parallel to the table edge.", "language_instruction": "Rotate the wooden board by 90 degrees and place it at the center of the table with its long side parallel to the table edge.", "objects": [ "Wooden board" ], "skills": [ "rotate", "place" ], "scene_layout": "Wooden board": "Center of the table" , "task_context": "The wooden board is initially placed flat on the tabletop with its long side perpendicular to the table edge . The board is stable, unobstructed, and fully graspable. The task requires a controlled 90-degree rotation followed by precise placement so that the final orientation is parallel to the table edge." Output: - Feasibility: Yes - Analysis: The rotation and placement operations are kinematically feasible within the robot's reachable workspace. The task sequence is logically coherent and does not violate physical or mechanical constraints. Example 2 Input: "task_name": "RoboGene_dual_Franka _stack_marker_on_pencil", "task_description": "The robot performs a stacking manipulation task in which a marker is placed on top of a pencil. The marker is initially located in the lower-left area of the table, while the pencil is positioned in the center-right area. The robot must grasp the marker, align it above the pencil, and carefully place it on top, ensuring stable contact without rolling or slipping.", "language_instruction": "Pick up the marker from the lower-left area of the table and stack it carefully on top of the pencil located at the center-right area.", "objects": [ "Marker", "Pencil" ], "skills": [ "stack" ], "scene_layout": "Marker": "Lower Left", "Pencil": "Center Right" , "task_context": "The marker and pencil are both resting on a flat tabletop surface. The pencil is placed horizontally and remains stationary throughout the task. The marker is fully graspable and must be placed on top of the pencil such that it remains balanced and does not roll off after stacking." Output: - Feasibility: No - Analysis: The task violates physical stability constraints. The cylindrical geometries and minimal contact surface between a marker and a pencil make stable stacking infeasible under gravity and realistic friction conditions." Self-Reflection Refiner Prompt The refiner synthesizes critiques to produce the improved task T ref . The prompt in- cludes: (1) Context Aggregation: Here is the original task T raw and the critiques from the evaluators. (2) Memory Re- trieval: Consider these heuristic guidelines retrieved from historical execution failures: [Retrieved Knowledge k]. (3) Revision Instruction: Modify T raw to resolve all identified issues while maintaining the original semantic intent. Self-Reflection Refiner Prompt "system": "You are an expert in dual-arm robotic task design and refinement, with strong expertise in synthesizing feedback from multiple evaluators. Your responsibility is to generate a final, improved version of a dual- arm robotic task specification by integrating : (1) the originally generated task, (2) feasibility and quality critiques provided by three independent reviewer LLMs. Your refinement must ensure that the resulting task is physically executable, logically coherent, and suitable for high- quality robotic policy learning.", "user": "You will be provided with structured task descriptions in a JSON-like format and the judgements and criteria from three critic models. Your task is to act as a task refiner to synthesize feedback from a 'Critic' module and the 'Original Task' definition to generate high-fidelity, physically plausible, and diverse dual-arm robotic manipulation tasks. 1. Evaluation Criteria: You will be provided with a structured task description in a JSON-like format, along with evaluation results from three critic models. The critic models are: (1) Constraint Adherence Evaluator as a hard constraint (2) Physical Feasibility Evaluator as a hard constraint (3) Task Novelty Evaluator as a soft constraint You must process and resolve critic feedback in the following strict order: Step 1. Constraint Adherence Resolution: This step has the highest priority. If the Constraint Adherence Evaluator returns Feasibility: No, you must revise the task to ensure: - All object names exactly match entries in asset.csv - All skill names exactly match entries in Skills.txt - No hallucinated entities or properties are present Exact string matching is mandatory. You are not allowed to introduce new objects or skills beyond the provided candidate sets. Step 2. Physical Feasibility Resolution: This step is processed only after all constraint violations are resolved. If the Physical Feasibility Evaluator returns Feasibility: No, you must revise the task to satisfy: - Kinematic reachability - Stable contact and support conditions - Realistic dual-arm coordination requirements Prefer the following revision strategies in order: (a) Adjust spatial layout or object positions (b) Decompose or reorder sub-actions (c) Substitute objects only if absolutely necessary Step 3. Task Novelty Optimization: This step is applied only if the task passes Steps 1 and 2. If the Task Novelty Evaluator returns Fail, you may revise the task to improve diversity by: - Replacing overused skills with less frequent primitives from Skills.txt - Introducing multi-stage or tool- mediated interactions Step 4. Global Consistency Audit: After all revisions, ensure that: - The task is logically coherent - Object states, scene layout, and language instructions are mutually consistent - Dual-arm coordination is explicit and non-trivial 2. Input Data Structure: (1) Initial tasks: Each task must be a JSON object containing: - 'task_name': Following the pattern ' RoboGene_robot_action'. - 'task_description': A detailed description. - 'language_instruction': A precise command for the VLA model. - 'objects': List of objects. - 'skills': List of skills. - 'scene_layout': 3x3 grids for left/ right arm workspaces. - 'task_context': Explains physical state and positioning. (2) Feedback from three critic models: The input will be provided in the following format: - Feasibility: [Yes / No] - Analysis: [A concise justification grounded in evaluation dimensions] 3. Output Format: The refined tasks will be provided in a JSON-like format and must include: - 'task_name', 'task_description', ' language_instruction', 'objects', 'skills', ' scene_layout', 'task_context'." D Evaluation Metrics and Protocol To rigorously validate the efficacy of the tasks generated by our framework, we establish a comprehensive benchmarking protocol. We compare RoboGene against four distinct base- lines: a standard Rule-based method, Human expert design, and two state-of-the-art Large Foundation Models (LFMs), specifically GPT-4o [ Hurst et al., 2024 ] and Gemini 2.5 Pro [ Comanici et al., 2025 ] . To ensure statistical significance, we generated a diverse evaluation corpus consisting of 900 tasks for each method, balanced evenly across three embodi- ment categories: 300 single-arm, 300 dual-arm, and 300 mo- bile manipulation tasks. Given that quantifying the quality of open-ended robotic tasks is an under-explored challenge, we propose a novel suite of multi-dimensional metrics designed to assess linguistic clarity, logical coherence, asset grounding, and physical executability. The reported results represent the mean scores averaged across the 900 generated tasks for each method. Task Clarity. This metric evaluates the linguistic quality of the generated instruction. It assesses whether the task de- scription is concise, unambiguous, and free from excessive rhetoric, ensuring interpretability by human operators during data collection. To mitigate subjective bias, we employ a hy- brid evaluation strategy. We aggregate binary judgments (0 or 1) from three distinct evaluators, including GPT-4o, Gemini 2.5 Pro, and human experts, and calculate the final score as the average of these three independent assessments. Robot Type Consistency. This metric verifies the align- ment between the generated task description and the speci- fied robot embodiment constraints r. A penalty is assigned if the task requirements contradict the robot’s kinematic con- figuration, such as assigning a bimanual coordination task to a single-arm robot. Similar to task clarity, this score is de- rived by averaging binary judgments from GPT-4o, Gemini 2.5 Pro, and human evaluators. Logical Validity. This metric determines if the task de- scription adheres to real-world semantic and common-sense constraints. It penalizes contextually inappropriate actions, such as performing industrial machining operations within a domestic dining environment. This metric is also calculated as the average of binary assessments from GPT-4o, Gemini 2.5 Pro, and human evaluators. Object Coverage.To quantify the precision of asset grounding, we define object coverage as the ratio of generated objects that exist within the valid object libraryO to the total number of unique objects proposed by the method. We utilize string matching algorithms to compute this metric. Formally, let O gen be the set of unique objects generated by a method; the coverage is calculated as|O gen ∩O|/|O|. A higher score indicates that the method effectively utilizes available assets rather than hallucinating non-existent items. Skill Coverage. Similar to object coverage, this metric measures the grounding of manipulation skills. It is defined as the proportion of generated skills that align with the pre- defined skill library S relative to the total number of pre- defined skills. Using string matching, we calculate this as |S gen ∩S|/|S|. This metric reflects the diversity and validity of the actions proposed by the generator. Physical Feasibility. This is the most critical metric, eval- uating the practical executability of the task in the real world. Unlike the semantic metrics, feasibility is assessed through physical interaction. We recruited human operators to attempt each task via teleoperation on the physical robot hardware. We report the average success rate based on five independent teleoperation trials per task. A trial is considered successful only if the human operator can successfully complete the ob- jective defined in the instruction. E Implementation Details of Baselines To ensure a rigorous comparative analysis, we provide de- tailed implementations of the baseline methods employed in our evaluation. These baselines represent the spectrum of task generation approaches, ranging from rule-based and human- based algorithms to state-of-the-art foundation models oper- ating without agentic scaffolding. Rule-based Method. This baseline utilizes a deterministic algorithm structured around a triple-nested loop. The sys- tem systematically iterates through the alphabetized lists of valid scenarios E , objects O, and skills S to mechanically Table 5: Ablation study of RoboGene. We evaluate the impact of the Reflection Mechanism, Skill/Object Sampling, and Memory Module on task diversity (Unique Skills/Objects) and Physical Feasibility. Exp. ReflectionSkillObjectMemoryUniqueUniquePhysical MechanismSamplingSamplingModuleSkillsObjectsFeasibility 1303530.75 2 ✓585120.88 3✓1015240.91 4 ✓966850.85 RoboGene✓1087190.99 synthesize task tuples. While this exhaustive combinatorial approach maximizes the theoretical coverage of the search space, it lacks semantic filtering capabilities. Consequently, the method produces a high volume of logically incoherent tasks, such as “cooking a wrench,” which are physically in- feasible. Furthermore, the generated instructions are primar- ily limited to simple, single-step primitive actions, failing to capture the complexity required for generalist robot learning. Human-based Method. To evaluate manual curation, we recruited expert annotators to generate tasks, granting them access to the complete history of previously designed entries to encourage diversity in distribution. However, our observa- tions identified two critical limitations in this approach. First, human designers exhibited significant cognitive bias, dispro- portionately favoring familiar settings such as Domestic and Kitchen environments while neglecting specialized domains like Industry or Medical scenarios. This tendency reflects an inherent difficulty for humans to conceive tasks within unfa- miliar physical contexts. Second, the designers rapidly en- countered a creativity plateau, where subsequent generations devolved into minor variations of existing tasks, such as sim- ple object substitution or positional adjustments, rather than introducing structurally novel behaviors. GPT-4o.For the GPT-4o baseline, we leveraged the prompt engineering capabilities of the model by directly in- jecting the complete sets of definitions for scenarios E , ob- jects O, and skills S into the context window. The model was instructed to function as an autonomous task generator, outputting tasks strictly in the required JSON format. Cru- cially, this baseline operates in an open-loop manner, lacking the feedback mechanisms, self-reflection, or external memory modules present in our framework. Gemini 2.5 Pro.Similarly, we designed a dedicated prompt for the Gemini 2.5 Pro baseline, embedding the req- uisite definition lists and formatting constraints akin to the GPT-4o setup. This baseline serves to evaluate the intrin- sic raw generation capabilities of state-of-the-art foundation models when deprived of the agentic scaffolding provided by RoboGene. By comparing these results, we isolate the spe- cific contributions of our diversity-driven sampling and self- reflection mechanisms. F Ablation Study To assess the individual contributions of the components within the RoboGene framework, we conducted a systematic ablation study. We focus on how the Self-Reflection Mecha- nism, Diversity-Driven Sampling (Skill and Object), and the Long-Term Memory module affect both the diversity of the generated dataset and the physical executability of the tasks. The quantitative results are summarized in Table 5. Impact of Self-Reflection. We first investigate the ne- cessity of the Self-Reflection mechanism by comparing Ex- periment 1 (GPT-4o Baseline) and Experiment 2. The base- line, which relies solely on a Large Foundation Model (LMF) GPT-4o for generation, exhibits limited diversity and a rela- tively low physical feasibility score of 0.75 due to frequent hallucinations. Integrating the self-reflection mechanism sig- nificantly mitigates these issues, improving feasibility to 0.88 and nearly doubling the number of valid unique skills from 30 to 58. This indicates that scrutinizing proposals against mul- tiple constraints is a prerequisite for high-quality task gener- ation. Improving Task Diversity via LFU Sampling. To over- come the long-tail distribution problem inherent in human and LLM data, we incrementally introduce the Least Fre- quently Used (LFU) strategies. In Experiment 3, enabling Skill Sampling drastically expands the action space, increas- ing unique skills from 58 to 101. Subsequently, Experiment 4 incorporates Object Sampling, which pushes the system to in- teract with under-represented assets, raising the unique object count to 685. However, we observe that enforcing high object diversity in Experiment 4 leads to a slight decline in physical feasibility (dropping from 0.91 to 0.85). This suggests that blindly maximizing diversity can introduce complex object- skill combinations that are challenging to ground physically without historical context. Consolidating Human Feedback with Long-Term Memory. The RoboGene framework (Experiment 5) ad- dresses the feasibility drop observed in the diversity-focused variants by incorporating the Long-Term Memory module. By retrieving and utilizing heuristic knowledge from Human- in-the-Loop feedback, the system effectively refines com- plex tasks that were previously error-prone. Consequently, RoboGene achieves the highest performance across all met- rics, reaching near-perfect physical feasibility (0.99) while maintaining the broadest coverage of the task space with 719 unique objects and 108 unique skills. These results con- firm that the synergy between diversity-driven sampling and memory-augmented refinement is essential for synthesizing datasets that are both expansive and executable. Table 6: Comparative results of downstream fine-tuning performance on unseen dual-arm tasks. We report the success rates (%) of theπ 0 model pre-trained on different datasets and fine-tuned with 50 demonstrations per task. Method DFR-GrillDFR-HangDFR-BuildDFR-LubricateDFR-Weigh Average SkewersCupsBlocksGearsBeaker Human75405452538 GPT-4o554015502036 Gemini 2.5 Pro 803015351535 RoboGene805030552548 G Additional Experiments To further evaluate the efficacy of the data generated by our framework, we conducted additional experiments focusing on the downstream transfer capabilities of Vision-Language- Action (VLA) models. Specifically, we investigated whether pre-training on diverse, synthetically generated tasks facili- tates better adaptation to unseen real-world scenarios com- pared to baselines.We constructed distinct pre-training datasets, each comprising 150 single-arm and 150 dual-arm tasks generated by Human experts, GPT-4o, Gemini 2.5 Pro, and RoboGene, respectively. We employed the π 0 [ Black et al., 2024 ] as the back- bone policy for these experiments. Following the pre-training phase on each respective dataset, we fine-tuned the models on a set of five unseen dual-arm tasks: DFR-GrillSkewers, DFR-HangCups, DFR-BuildBlocks, DFR-LubricateGears, and DFR-WeighBeaker. To vigorously test data efficiency and adaptation speed, we utilized 50 human demonstrations per task for fine-tuning. The evaluation metric reports the av- erage success rate calculated over 20 real-world rollouts for each task. The quantitative results are summarized in Table 6. The policy pre-trained on the RoboGene-generated tasks consis- tently outperforms the baselines, achieving the highest av- erage success rate of 48%.Notably, RoboGene demon- strates a significant advantage in complex tasks such as DFR- BuildBlocks, where it achieves a 30% success rate compared to 5% for Human-curated data and 15% for other Foundation Models. This performance gap highlights that while human data is physically valid, it often lacks the structural diversity required to generalize to complex multi-stage manipulations. Furthermore, while GPT-4o and Gemini 2.5 Pro provide some level of diversity, their lack of physical grounding often leads to suboptimal pre-training priors. In contrast, RoboGene suc- cessfully balances diversity with physical feasibility, enabling the VLA model to learn robust representations that transfer effectively to novel, unseen tasks.