Paper deep dive

AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation

Mingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui, Zeyang Liu, Xingyu Chen, Xuguang Lan

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 38

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/14/2026, 2:37:42 AM

Summary

AffordSim is a novel simulation framework for robotic manipulation that integrates open-vocabulary 3D affordance prediction (via the VoxAfford model) into the data generation pipeline. By guiding grasp pose estimation toward task-relevant functional regions, it enables the automated generation of semantically correct trajectories for complex tasks like pouring and hanging. The framework supports cross-embodiment robotic platforms, utilizes VLM-powered task generation, and employs DA3-based domain randomization for effective sim-to-real transfer.

Entities (5)

AffordSim · simulation-framework · 100%NVIDIA Isaac Sim · simulation-platform · 100%VoxAfford · ai-model · 100%Franka FR3 · robotic-platform · 95%DA3 · reconstruction-method · 90%

Relation Signals (3)

AffordSim → builton → NVIDIA Isaac Sim

confidence 100% · Built on NVIDIA Isaac Sim with cross-embodiment support

AffordSim → integrates → VoxAfford

confidence 100% · AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector

VoxAfford → guides → Grasp Pose Estimation

confidence 95% · VoxAfford... predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Simulation-based data generation has become a dominant paradigm for training robotic manipulation policies, yet existing platforms do not incorporate object affordance information into trajectory generation. As a result, tasks requiring precise interaction with specific functional regions--grasping a mug by its handle, pouring from a cup's rim, or hanging a mug on a hook--cannot be automatically generated with semantically correct trajectories. We introduce AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into the manipulation data generation pipeline. AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector that enhances MLLM output tokens with multi-scale geometric features, to predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions. Built on NVIDIA Isaac Sim with cross-embodiment support (Franka FR3, Panda, UR5e, Kinova), VLM-powered task generation, and novel domain randomization using DA3-based 3D Gaussian reconstruction from real photographs, AffordSim enables automated, scalable generation of affordance-aware manipulation data. We establish a benchmark of 50 tasks across 7 categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) and evaluate 4 imitation learning baselines (BC, Diffusion Policy, ACT, Pi 0.5). Our results reveal that while grasping is largely solved (53-93% success), affordance-demanding tasks such as pouring into narrow containers (1-43%) and mug hanging (0-47%) remain significantly more challenging for current imitation learning methods, highlighting the need for affordance-aware data generation. Zero-shot sim-to-real experiments on a real Franka FR3 validate the transferability of the generated data.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

37,234 characters extracted from source content.

Expand or collapse full text

11footnotetext: Equal contribution.22footnotetext: Corresponding author. AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation Mingyang Li∗, Haofan Xu∗, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Qiongjie Cui†, Zeyang Liu†, Xingyu Chen†, Xuguang Lan† School of Artificial Intelligence, Xi’an Jiaotong University Abstract Simulation-based data generation has become a dominant paradigm for training robotic manipulation policies, yet existing platforms do not incorporate object affordance information into trajectory generation. As a result, tasks requiring precise interaction with specific functional regions—grasping a mug by its handle, pouring from a cup’s rim, or hanging a mug on a hook—cannot be automatically generated with semantically correct trajectories. We introduce AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into the manipulation data generation pipeline. AffordSim uses our VoxAfford model, an open-vocabulary 3D affordance detector that enhances MLLM output tokens with multi-scale geometric features, to predict affordance maps on object point clouds, guiding grasp pose estimation toward task-relevant functional regions. Built on NVIDIA Isaac Sim with cross-embodiment support (Franka FR3, Panda, UR5e, Kinova), VLM-powered task generation, and novel domain randomization using DA3-based 3D Gaussian reconstruction from real photographs, AffordSim enables automated, scalable generation of affordance-aware manipulation data. We establish a benchmark of 50 tasks across 7 categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) and evaluate 4 imitation learning baselines (BC, Diffusion Policy, ACT, Pi 0.5). Our results reveal that while grasping is largely solved (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging for current imitation learning methods, highlighting the need for affordance-aware data generation. Zero-shot sim-to-real experiments on a real Franka FR3 validate the transferability of the generated data. Figure 1: Overview of AffordSim. Natural language task descriptions are processed by a VLM to generate simulation scenes in Isaac Sim. VoxAfford predicts 3D affordance maps on object point clouds, guiding grasp pose estimation and motion planning to produce semantically correct manipulation trajectories. Comprehensive domain randomization, including DA3-reconstructed real backgrounds, enables zero-shot sim-to-real transfer. 1 Introduction Simulation-based data generation has become a dominant paradigm for scaling robotic manipulation training data (Mu et al., 2024; Nasiriany et al., 2024; Gu et al., 2023; RoboVerse Team, 2025). By collecting demonstrations in simulation, researchers can generate thousands of diverse trajectories at low cost, train visuomotor policies via imitation learning, and deploy them on real robots through sim-to-real transfer. However, a fundamental limitation persists across existing simulation platforms: none of them incorporates object affordance information into the trajectory generation pipeline. Object affordances—the functional regions through which objects are meant to be manipulated—are critical for tasks requiring precise, semantically correct interactions. Pouring from a cup requires grasping its handle and tilting from the rim; hanging a mug demands locating the hook-compatible opening. Current simulation platforms handle grasp pose estimation through one of two approaches, both of which fail for affordance-demanding tasks. Manual grasp design requires human engineers to hand-specify grasp poses for each object and task, which is labor-intensive and does not scale. Generic grasp estimation methods such as AnyGrasp (Fang et al., 2023) predict stable grasps without considering task semantics, often selecting poses that are physically stable but functionally incorrect (e.g., grasping a mug by its body when the task requires pouring). In our experiments, affordance-demanding tasks such as mug hanging achieve at most 10% policy success when trajectories are generated with generic grasp methods, compared to meaningful improvements with affordance guidance. We propose AffordSim, the first simulation framework that integrates affordance prediction into the robotic manipulation data generation pipeline. AffordSim uses our VoxAfford model (Yang and others, 2024b), an open-vocabulary 3D affordance detection method that enhances MLLM output tokens with multi-scale 3D geometric features, to predict affordance maps indicating where and how a robot should interact with objects. These affordance predictions guide grasp pose estimation and motion planning, producing trajectories that are both physically plausible and semantically aligned with task objectives. Built on NVIDIA Isaac Sim (NVIDIA, 2023), AffordSim features: • Affordance-aware trajectory generation. VoxAfford predicts 3D affordance heatmaps on object point clouds, guiding grasp pose selection toward task-relevant functional regions. • Automated task-scene generation. A VLM-powered pipeline converts natural language task descriptions into simulation scenes, supporting 4 robot embodiments (Franka FR3, Panda, UR5e, Kinova) without task-specific engineering. • Photorealistic domain randomization. Five randomization axes—including DA3-based (Yang and others, 2024a) 3D Gaussian reconstruction from real photographs—reduce the visual domain gap for sim-to-real transfer. • A 50-task benchmark across 7 manipulation categories (grasping, placing, stacking, pushing/pulling, pouring, mug hanging, long-horizon composite) with evaluations of 4 imitation learning baselines (BC, Diffusion Policy (Chi et al., 2023), ACT (Zhao et al., 2023), Pi 0.5 (Black et al., 2024)). Our benchmark reveals a key finding: while grasping is largely solved by current imitation learning methods (53–93% success), affordance-demanding tasks such as pouring into narrow containers (1–43%) and mug hanging (0–47%) remain significantly more challenging. This performance gap highlights the need for affordance-aware data generation to address these challenging manipulation tasks. Zero-shot sim-to-real experiments on a real Franka FR3 validate that policies trained exclusively on AffordSim data transfer to the real world, with success rates ranging from 60% on grasping to 10% on mug hanging, mirroring the affordance-complexity gradient observed in simulation. 2 Related Work 2.1 Simulation Platforms for Robotic Manipulation A growing number of simulation platforms provide environments for generating robotic manipulation data. RoboTwin (Mu et al., 2024) combines digital twin environments with LLM-driven task generation and multi-embodiment support. RoboCasa (Nasiriany et al., 2024) focuses on household environments with large-scale scene generation. ManiSkill2 (Gu et al., 2023) provides GPU-parallelized environments for contact-rich manipulation. Meta-World (Yu et al., 2020) and CALVIN (Mees et al., 2022) offer standardized multi-task benchmarks for policy learning. LIBERO (Liu et al., 2024) introduces lifelong learning benchmarks with procedural task generation. RoboVerse (RoboVerse Team, 2025) unifies multiple simulators under a common interface. GenManip (Geng and others, 2025) uses LLM-driven simulation for generalizable instruction-following manipulation, focusing on language grounding rather than affordance-aware trajectory generation. However, none of these platforms integrates object affordance information into the trajectory generation process. Grasp poses are either manually designed or estimated by generic methods, limiting the range of tasks that can be automatically generated with semantically correct interactions. 2.2 3D Affordance Prediction Affordance prediction has been studied extensively as a standalone perception problem. Where2Act (Mo et al., 2021) predicts per-point actionability for articulated objects. 3D AffordanceNet (Deng et al., 2021) learns affordance representations from 3D point clouds. GAPartNet (Geng et al., 2023) detects generalizable and actionable parts across object categories. Our concurrent work, VoxAfford (Yang and others, 2024b), advances this line by injecting multi-scale geometric features from a frozen 3D VQVAE encoder into MLLM output tokens via cross-attention, achieving state-of-the-art open-vocabulary 3D affordance detection (∼8% 8\% mIoU improvement) with zero-shot transfer to real robots. Despite progress in affordance prediction, these models have been studied in isolation from simulation data generation pipelines. AffordSim bridges this gap by integrating VoxAfford directly into the trajectory generation loop, enabling affordance-aware data collection at scale. 2.3 Automated Data Generation and Domain Randomization LLM- and VLM-driven approaches have enabled automated task and scene generation. GenSim2 (Katara et al., 2024) uses LLMs to generate task code and reward functions. RoboGen (Wang et al., 2023) automates the generation of diverse manipulation tasks through LLM-based decomposition. These approaches focus on task diversity but remain agnostic to object affordances during trajectory collection. Domain randomization (Tobin et al., 2017; Peng et al., 2018) is a widely adopted strategy for closing the sim-to-real gap by diversifying visual appearance during training. We extend standard domain randomization with DA3-based (Yang and others, 2024a) 3D Gaussian reconstruction from real scene photographs, providing photorealistic background rendering from arbitrary viewpoints—a complement to texture-based randomization that better preserves real-world visual statistics. 3 AffordSim Framework AffordSim is a simulation framework for generating affordance-aware robotic manipulation data. Given a natural language task description T (e.g., “pick up the mug by its handle and pour into the pan”), AffordSim automatically generates a simulation scene, predicts affordance maps on target objects, plans grasp poses guided by affordance predictions, executes motion plans to collect trajectories, and applies domain randomization for sim-to-real transfer. Figure 1 provides an overview of the full pipeline. 3.1 System Overview The AffordSim pipeline consists of five stages: (1) A VLM converts T into a scene configuration S, specifying objects, their poses, the robot embodiment, and goal conditions. (2) For each target object, a point cloud ∈ℝN×3P ^N× 3 is captured from the simulation scene. (3) VoxAfford predicts an affordance map :→[0,1]A:P→[0,1], where ai=(pi)a_i=A(p_i) indicates the suitability of point pip_i for the specified interaction. (4) Affordance-guided grasp pose estimation selects a grasp ∗∈SE(3)g^* (3) from the affordance-weighted candidate set. (5) A motion planner generates and executes a trajectory τ, and domain randomization is applied to the rendered observations. The system is built on NVIDIA Isaac Sim and supports four robot embodiments: Franka FR3, Franka Panda, UR5e, and Kinova. 3.2 Affordance-Aware Trajectory Generation The core contribution of AffordSim is the integration of affordance prediction into grasp planning. Given the task description T and an object point cloud P, our VoxAfford model (Yang and others, 2024b) predicts a per-point affordance score ai∈[0,1]a_i∈[0,1] indicating how suitable each surface point is for the specified manipulation. VoxAfford enhances MLLM output tokens with multi-scale geometric features from a frozen 3D VQVAE encoder via cross-attention, producing spatially-aware affordance masks. It operates in an open-vocabulary setting: it accepts arbitrary natural language affordance queries (e.g., “graspable handle,” “pourable rim”) and generalizes across object categories without task-specific fine-tuning. Affordance-guided grasp selection. Grasp pose estimation proceeds in two stages. First, the predicted affordance map A identifies the high-affordance region on the object surface. We sample multiple approach directions around this region and generate a set of candidate grasps =1,…,KG=\g_1,…,g_K\, each oriented along a different approach vector toward the affordance peak. Second, each candidate kg_k is scored by jointly considering (i) the affordance values of the contacted surface and (i) the kinematic feasibility with respect to the robot’s current configuration q: sk=1|k|∑pi∈kai⏟skaff⋅f(k,)⏟skkin,s_k= 1|C_k| _p_i _ka_i_s_k^aff\;·\; f(g_k,q)_s_k^kin, (1) where k⊂C_k denotes the set of points in contact with grasp kg_k, skaffs_k^aff measures the mean affordance value of the contact region, and skkin=f(k,)∈[0,1]s_k^kin=f(g_k,q)∈[0,1] evaluates the reachability and collision-free feasibility of the grasp given the robot arm’s actual position. The selected grasp is ∗=argmaxkskg^*=g_ argmax_k\,s_k. This two-stage design ensures that the final grasp is both semantically correct (targeting functional regions such as a mug handle or cup rim) and physically executable by the specific robot embodiment. Motion planning. Given the selected grasp pose ∗g^*, we use cuRobo (Sundaralingam et al., 2023), a GPU-accelerated motion planning library, to generate a collision-free trajectory from the robot’s home configuration to the pre-grasp pose, execute the grasp, and plan the subsequent manipulation motion (e.g., lifting, tilting for pouring, placing). The full trajectory =(ot,at)t=1T τ=\(o_t,a_t)\_t=1^T consists of observation-action pairs, where oto_t includes RGB-D images and proprioceptive state, and ata_t is the end-effector pose command. 3.3 VLM-Powered Task and Scene Generation AffordSim automates the task-to-scene pipeline using a VLM. Given a natural language task description T, the VLM generates: (1) a list of required objects with their USD asset paths from the object library (500+ objects, 50+ real-scanned), (2) initial poses for each object on the workspace, (3) the target robot embodiment, and (4) task-specific goal conditions (e.g., “object A is inside container B”). This eliminates manual scene engineering and enables rapid scaling to new tasks. The system supports four robot embodiments—Franka FR3, Franka Panda, UR5e, and Kinova—with automatic kinematic adaptation (Figure 4). Figure 2: Domain randomization in AffordSim. Each column shows the cumulative effect of adding one randomization axis: object pose, lighting, background texture, and image noise. The rightmost column shows example background textures used for randomization. Three representative tasks are shown across rows. 3.4 Domain Randomization for Sim-to-Real Transfer To close the visual gap between simulation and the real world, AffordSim applies domain randomization along five axes (Figure 2): (1) Background texture: random textures from a diverse texture library are applied to the workspace and surrounding surfaces. (2) Lighting: randomized number, position, intensity, and color of light sources. (3) Object texture: PBR material properties (albedo, roughness, metallicity) are perturbed. (4) Object pose: small random perturbations to initial object positions and orientations. (5) DA3 Gaussian background: we reconstruct the real deployment scene using DA3-based 3D Gaussian Splatting (Yang and others, 2024a) from 10–20 photographs. The reconstructed Gaussian field renders photorealistic background images from arbitrary viewpoints, which are composited behind the simulation scene. This approach preserves the visual statistics of the real environment (spatial layout, textures, lighting gradients) rather than sampling from a generic texture distribution, significantly reducing the domain gap. Figure 3: Task gallery. Representative tasks from the AffordSim benchmark across seven manipulation categories: grasping, placing, stacking, pushing/pulling, pouring, mug hanging, and long-horizon composite tasks. 3.5 Benchmark Design We design a benchmark of 50 tasks organized into 7 manipulation categories of increasing affordance complexity (Figure 3): 1. Grasping (10 tasks): Pick up objects with varied geometries. Requires basic affordance (graspable surfaces). 2. Placing (10 tasks): Pick and place objects onto target surfaces or into containers. Requires release-point affordance. 3. Stacking (5 tasks): Stack objects in specified orders. Requires stable-surface affordance and sequencing. 4. Pushing/Pulling (6 tasks): Push objects to targets or pull open drawers/doors. Requires contact-surface affordance. 5. Pouring (8 tasks): Pour contents between containers of varying geometry. Requires rim affordance and tilt control. 6. Mug Hanging (3 tasks): Hang mugs on racks or hooks. Requires handle-opening affordance and precise alignment. 7. Long-Horizon Composite (8 tasks): Multi-step tasks that chain primitives from different categories, e.g., pick-pour-place or open-and-place. Requires sequential affordance reasoning across multiple objects and interactions. Each task is evaluated with 30 simulation rollouts. The primary metric is task success rate (SR), defined as the fraction of rollouts that satisfy all goal conditions. For multi-step tasks, we additionally report sub-task success rates to localize failure modes (e.g., pick succeeds at 90% but pour fails at 3%). 4 Experiments We evaluate AffordSim along four axes: (1) benchmark performance of imitation learning baselines, (2) cross-embodiment generalization, (3) the contribution of affordance integration, and (4) zero-shot sim-to-real transfer. Baselines. We evaluate four imitation learning methods: BC (behavioral cloning with an MLP policy), Diffusion Policy (DP) (Chi et al., 2023), ACT (Zhao et al., 2023), and Pi 0.5 (Black et al., 2024). All policies are trained on 300 demonstrations per task generated by AffordSim, using RGB-D observations and proprioceptive state as input. Training details are provided in Appendix A. 4.1 Benchmark Evaluation Table 1: Benchmark evaluation results (Success Rate %) on representative tasks across six categories. Each task is evaluated over 30 rollouts. Sub-task success rates are shown where applicable. Best results per task in bold. Category Task BC ACT DP Pi 0.5 Grasping pick_banana 53 63 87 93 pick_red_cup 17 77 63 80 Placing pick_banana_place_plate 37 60 84 93 pick_cup_place_shelf 5 7 3 47 pick_kettle_place_coffee machine 27 43 49 65 Stacking stack_two_blocks 30 43 47 77 stack_three_blocks 13 27 28 53 Push/Pull push_box_to_target 23 63 67 78 pull_drawer_open 37 53 58 86 Pouring pour_basket_into_bowl 20 86 94 99 pour_pan_into_bowl 7 33 87 92 pour_cup_into_bowl 1 24 36 43 Mug Hanging hang_mug_on_rack 0 10 17 47 hang_mug_on_hook 0 7 13 33 Long Horizon pick_cup_pour_place_coffee_machine 0 3 8 16 pick_cup_pour_hang_on_rack 0 1 3 21 open_microwave_place_can 0 0 2 13 Average (all 17 tasks) 16 35 44 61 Each task is trained on 300 demonstrations with object pose randomization applied during data collection; all evaluation rollouts also include object pose randomization to test generalization. Task-specific randomization ranges are detailed in Appendix A. Table 1 presents results on representative tasks from each category. Across all baselines, grasping tasks achieve the highest success rates (53–93%), confirming that basic pick-up manipulation is largely solved by current methods. Placing and pushing/pulling tasks show moderate-to-high success (3–93%), with failures primarily occurring during the precision-placement or target-reaching phase rather than the initial grasp. The most striking finding is the sharp performance drop on affordance-demanding tasks. Pouring into a wide basket achieves 20–99% success, but pouring into a narrow cup drops to 1–43%—a significant degradation for the same manipulation primitive (pour) when the target container demands higher positional precision. Mug hanging, which requires the policy to align the mug handle with a hook or rack peg, achieves at most 47% success even for the best baseline. These results highlight that affordance-demanding tasks—where success depends on the precision of functional-region interaction—remain significantly more challenging for current methods, underscoring the importance of affordance-aware data generation. Pi 0.5 achieves the highest average performance (61%), followed by DP (44%) and ACT (35%). BC lags significantly behind (16%), suggesting that foundation-model-based, diffusion-based, and action-chunking architectures better capture the multimodal action distributions characteristic of affordance-guided trajectories. Sub-task analysis. The sub-task breakdown reveals that grasping (the first sub-task) succeeds at 63–100% across pouring tasks, while the downstream pouring action fails dramatically for small containers. This localizes the bottleneck: current policies can learn affordance-guided grasps from data but struggle to execute the fine-grained, affordance-dependent manipulation that follows. 4.2 Cross-Embodiment Evaluation Figure 4: Cross-embodiment support. Three representative tasks executed by four robot embodiments: Franka FR3, Franka Panda, UR5e, and Kinova. AffordSim generates affordance-guided trajectories for each embodiment without task-specific tuning. We evaluate trajectory generation success across four robot embodiments on 10 representative tasks. AffordSim successfully generates affordance-guided trajectories for all embodiments, with generation success rates of 94% (Franka FR3), 92% (Franka Panda), 83% (UR5e), and 95% (Kinova). The variation reflects kinematic differences rather than affordance prediction failures, as the VoxAfford module is embodiment-agnostic. Notably, the UR5e exhibits lower success on orientation-sensitive tasks such as pouring, because its 6-DOF kinematics (versus 7-DOF for the other arms) limits wrist dexterity and reduces the feasible set of end-effector orientations. This confirms that AffordSim’s affordance-aware pipeline generalizes across robot platforms without task-specific tuning. 4.3 Affordance Integration Ablation Table 2: Affordance integration ablation. Trajectory generation success rate (%) under four grasp strategies. Manual: human-specified grasp poses without affordance. AnyGrasp: generic grasp estimation. VoxAfford: predicted affordance guidance (ours). Human Aff.: human-annotated affordance ground truth (oracle upper bound). Evaluated with Pi 0.5. Task Manual AnyGrasp VoxAfford Human Aff. pick_banana_place_plate 100 47 98 100 pour_into_pan 67 20 80 93 pour_into_cup 83 0 63 80 hang_mug_on_rack 87 0 10 90 hang_mug_on_hook 83 0 17 87 pick_screwdriver 100 53 97 100 Average 87 20 61 92 Table 2 compares four grasp strategies on affordance-demanding tasks: Manual (human-specified grasp poses without affordance reasoning), AnyGrasp (Fang et al., 2023) (generic grasp estimation), VoxAfford (our predicted affordance guidance), and Human Aff. (human-annotated affordance ground truth, serving as an oracle upper bound). Manual grasp design achieves the highest baseline performance (87%), yet each new object or task instance requires re-designing grasp poses from scratch—a labor-intensive process that does not scale. AnyGrasp (20%) performs poorly on affordance-demanding tasks, as generic grasp estimation cannot reason about task-relevant contact regions. VoxAfford (61%) substantially outperforms AnyGrasp, demonstrating the value of learned affordance guidance. On pouring tasks, VoxAfford achieves 80% (pan) and 63% (cup) vs. AnyGrasp’s 20% and 0%, because affordance guidance ensures the robot grasps the cup at a functionally correct position for tilting. On mug hanging, however, VoxAfford achieves only 10–17%, because the VoxAfford model was not trained with hanging-specific affordances: hanging a mug on a rack requires grasping the cup body rather than the handle, which differs from the conventional grasp affordances in the training data. AnyGrasp completely fails (0%) on these tasks as well. The Human Aff. column (92%) establishes the oracle upper bound: when ground-truth affordance labels replace VoxAfford predictions, success rates improve by ∼ 31 percentage points. This gap is largely driven by the mug-hanging tasks where VoxAfford lacks training coverage, indicating that expanding the affordance training set to include non-standard grasp regions is the primary avenue for improvement. On tasks where VoxAfford has adequate training data (e.g., pick-and-place, pouring, screwdriver), it already recovers over 80% of the oracle performance. 4.4 Zero-Shot Sim-to-Real Transfer Table 3: Zero-shot sim-to-real transfer on a real Franka FR3. Policies trained exclusively on AffordSim data (Pi 0.5). Each task evaluated over 10 trials. Category Task SR (%) Grasping pick_banana 60 Placing pick_cup_place_shelf 30 Stacking stack_two_blocks 20 Push/Pull push_box_to_target 40 Pouring pour_into_pan 20 Mug Hanging hang_mug_on_rack 10 Average 30 We deploy Pi 0.5 trained exclusively on AffordSim data to a real Franka FR3, evaluating on one representative task from each category (Table 3). The sim-to-real transfer success rate mirrors the affordance-complexity gradient observed in simulation: grasping achieves 60%, placing 30%, push/pull 40%, and stacking 20%, while pouring (20%) and mug hanging (10%) remain challenging. These results confirm two findings: (1) AffordSim-generated data supports effective sim-to-real transfer, and (2) the affordance-complexity bottleneck is not an artifact of simulation but reflects genuine difficulty in learning fine-grained functional interactions. 4.5 Domain Randomization Robustness Table 4: Domain randomization robustness. Sim-to-real transfer success rate (%) on a real Franka FR3 under different training and evaluation conditions. Standard: original deployment scene. Novel scene: modified tablecloth and background objects. Each task evaluated over 10 trials. DP Pi 0.5 Setting pick place pour pick place pour w/o DR (standard) 10 0 0 40 10 0 w/ DR (standard) 20 0 0 50 20 10 w/ DR (novel scene) 20 0 0 50 20 10 To evaluate whether domain randomization equips policies with robustness to environmental variations, we train DP and Pi 0.5 on three representative tasks (pick_banana, pick_cup_place_shelf, pour_pan_into_bowl), each with 300 demonstrations, under two settings—with and without domain randomization—and evaluate on the real Franka FR3 in both the standard deployment scene and a novel scene with modified tablecloth and background objects (Table 4). Specific randomization parameters for lighting, texture, and tablecloth are provided in Appendix A. Policies trained without domain randomization achieve very low success rates in the standard scene (DP: 3% avg, Pi 0.5: 17% avg). Adding domain randomization during training improves performance notably, with Pi 0.5 increasing from 17% to 27% average success. Crucially, DR-trained policies maintain their performance even when the real scene is visually altered: Pi 0.5 achieves identical success (50/20/10%) in both standard and novel scenes, and DP likewise shows consistent results across scene conditions. This demonstrates that AffordSim’s domain randomization pipeline effectively endows policies with robustness to environmental perturbations, enabling deployment across varying real-world conditions without retraining. 5 Discussion and Limitations Affordance prediction accuracy. AffordSim’s trajectory quality depends on VoxAfford’s affordance predictions. While VoxAfford generalizes well to common object categories, prediction accuracy degrades for novel objects with unusual geometries or under heavy occlusion. Failure cases include thin tool handles, transparent objects, and objects with ambiguous functional regions (e.g., symmetric containers with no clearly defined pouring side). Scope of manipulation. The current framework is limited to rigid-body manipulation. Deformable objects (cloth, rope, dough) require different simulation backends and affordance representations, which we leave for future work. The benchmark also does not cover in-hand manipulation or bimanual tasks, both of which introduce additional affordance reasoning challenges (e.g., coordinated functional region contact across two grippers). Background reconstruction. The DA3 Gaussian reconstruction requires 10–20 photographs of the real deployment scene. While this is a one-time effort per workspace, it limits portability to new environments without re-collection. An alternative would be diffusion-based style transfer, though at the cost of geometric accuracy. Broader impact. AffordSim accelerates the development of robotic manipulation systems by reducing the cost of training data generation. While this benefits applications in manufacturing, logistics, and assistive robotics, autonomous manipulation capabilities could be misused in contexts where physical interactions should require human oversight. We release our code and benchmark under a permissive license to support reproducible research. 6 Conclusion We introduced AffordSim, the first simulation framework that integrates open-vocabulary 3D affordance prediction into robotic manipulation data generation. By combining VoxAfford for affordance-guided grasp planning, VLM-powered task generation, and photorealistic domain randomization with DA3-reconstructed real backgrounds, AffordSim produces semantically correct manipulation trajectories at scale. Our 50-task benchmark across 6 categories reveals that affordance-demanding tasks—pouring and mug hanging—remain a critical open challenge for current imitation learning methods, exposing affordance complexity as a missing evaluation dimension. Zero-shot sim-to-real experiments validate the practical utility of the generated data. We hope that AffordSim and its benchmark will serve as a catalyst for developing manipulation policies that reason about object affordances. References K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: 4th item, §4. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. Robotics: Science and Systems (RSS). Cited by: 4th item, §4. S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia (2021) 3D AffordanceNet: a benchmark for visual object affordance understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1778–1787. Cited by: §2.2. H. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu (2023) AnyGrasp: robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics. Cited by: §1, §4.3. H. Geng et al. (2025) GENMANIP: LLM-driven simulation for generalizable instruction-following manipulation. arXiv preprint. Cited by: §2.1. H. Geng, H. Xu, C. Zhao, C. Xu, L. Yi, S. Huang, and H. Wang (2023) GAPartNet: cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Cited by: §2.2. J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, et al. (2023) ManiSkill2: a unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.04659. Cited by: §1, §2.1. P. Katara, Z. Xian, and K. Fragkiadaki (2024) GenSim2: scaling robot data generation with multi-modal and reasoning LLMs. arXiv preprint arXiv:2410.03645. Cited by: §2.3. B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2024) LIBERO: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems. Cited by: §2.1. O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard (2022) CALVIN: a benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters 7 (3), p. 7327–7334. Cited by: §2.1. K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani (2021) Where2Act: from pixels to actions for articulated 3D objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6813–6823. Cited by: §2.2. Y. Mu, T. Chen, S. Peng, Z. Chen, Z. Gao, Y. Zou, L. Lin, Z. Xie, and P. Luo (2024) RoboTwin: dual-arm robot benchmark with generative digital twins. arXiv preprint arXiv:2409.02920. Cited by: §1, §2.1. S. Nasiriany, A. Maddukuri, L. Zhang, A. Parber, T. Lo, A. Joshi, H. Welborn, and Y. Zhu (2024) RoboCasa: large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02523. Cited by: §1, §2.1. NVIDIA (2023) NVIDIA Isaac Sim. Note: https://developer.nvidia.com/isaac-simVersion 5.1 Cited by: §1. X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel (2018) Sim-to-real transfer of robotic control with dynamics randomization. IEEE International Conference on Robotics and Automation (ICRA), p. 3803–3810. Cited by: §2.3. RoboVerse Team (2025) RoboVerse: towards a unified platform for scalable and generalizable robot learning. arXiv preprint arXiv:2504.09837. Cited by: §1, §2.1. B. Sundaralingam, S. K. Ber, T. Hermans, and B. Boots (2023) cuRobo: parallelized collision-free robot motion generation. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §3.2. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 23–30. Cited by: §2.3. Y. Wang, Z. Fan, Z. Jia, S. Srinivasa, and D. Xu (2023) RoboGen: towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455. Cited by: §2.3. T. Yang et al. (2024a) DA3: dense affordance annotation from 3D gaussian splatting. arXiv preprint. Cited by: 3rd item, §2.3, §3.4. T. Yang et al. (2024b) VoxAfford: open-vocabulary affordance detection in 3D point clouds. arXiv preprint. Cited by: §1, §2.2, §3.2. T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine (2020) Meta-World: a benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL). Cited by: §2.1. T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems (RSS). Cited by: 4th item, §4. Appendix A Training Details All imitation learning policies are trained on 300 demonstrations per task. Observations consist of two RGB-D camera views (wrist-mounted and third-person, 256×256 resolution) and 7-DoF proprioceptive state (joint positions). Actions are 7-DoF end-effector pose commands (position + quaternion orientation + gripper). BC. MLP with 3 hidden layers (256 units each), ReLU activations. ResNet-18 visual encoder. Trained for 500 epochs with batch size 64, learning rate 10−410^-4 (Adam). Diffusion Policy (DP). U-Net denoiser with 256-dim latent, 100 diffusion steps at training, 10 DDIM steps at inference. Observation horizon To=2T_o=2, action horizon Ta=8T_a=8, prediction horizon Tp=16T_p=16. Trained for 500 epochs, batch size 64, learning rate 10−410^-4. ACT. CVAE architecture with d=512d=512 latent dimension, k=100k=100 action chunks. Transformer encoder-decoder with 4 layers, 8 heads. Trained for 500 epochs, batch size 64, learning rate 10−510^-5. Pi 0.5. Pretrained Pi 0.5 backbone fine-tuned on AffordSim data. Full fine-tuning of action head, frozen vision-language backbone. Fine-tuned for 100 epochs, batch size 32, learning rate 10−510^-5. All experiments are conducted on a single NVIDIA A100 80GB GPU. Training time ranges from 2 hours (BC) to 12 hours (Pi 0.5) per task. Appendix B Full Benchmark Results The full 50-task benchmark results are available in the supplementary material and project website. Table 1 in the main paper shows representative tasks from each category. Appendix C Sim-to-Real Setup The real-world evaluation uses a Franka FR3 robot arm with a Franka Hand gripper, mounted on a tabletop workspace. Two Intel RealSense D435 cameras provide RGB-D observations (wrist-mounted and third-person view at 45∘45 elevation). The DA3 Gaussian background is reconstructed from 15 photographs of the workspace taken from diverse viewpoints. All policies are evaluated zero-shot without any real-world fine-tuning, with 10 trials per task.

Instant research discovery