Paper deep dive
HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments
Shivani Kamtikar, Kendall Koe, Justin Wasserman, Samhita Marri, Benjamin Walt, Naveen Kumar Uppalapati, Girish Krishnan, Girish Chowdhary
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 2:28:14 AM
Summary
HyReach is a real-time, vision-guided hybrid rigid-soft continuum manipulator system designed for robust object reaching in cluttered, unseen environments. It integrates multi-view RGB reconstruction, shape-aware motion planning using a Constant Curvature model, and a learning-based controller to navigate complex scenes without environment-specific retraining, achieving sub-2 cm reaching accuracy.
Entities (5)
Relation Signals (3)
HyReach → incorporates → B3 soft continuum arm
confidence 95% · The platform combines a standard 6-DoF industrial robotic arm with a tri-chambered bending (B3) SCA
HyReach → uses → Mast3r
confidence 95% · The images are processed using Mast3r to estimate metric depth and reconstruct the scene.
HyReach → uses → YOLO-World
confidence 95% · Goal objects are localized using YOLO-World
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As robotic systems increasingly operate in unstructured, cluttered, and previously unseen environments, there is a growing need for manipulators that combine compliance, adaptability, and precise control. This work presents a real-time hybrid rigid-soft continuum manipulator system designed for robust open-world object reaching in such challenging environments. The system integrates vision-based perception and 3D scene reconstruction with shape-aware motion planning to generate safe trajectories. A learning-based controller drives the hybrid arm to arbitrary target poses, leveraging the flexibility of the soft segment while maintaining the precision of the rigid segment. The system operates without environment-specific retraining, enabling direct generalization to new scenes. Extensive real-world experiments demonstrate consistent reaching performance with errors below 2 cm across diverse cluttered setups, highlighting the potential of hybrid manipulators for adaptive and reliable operation in unstructured environments.
Tags
Links
- Source: https://arxiv.org/abs/2603.21421v1
- Canonical: https://arxiv.org/abs/2603.21421v1
Trouble viewing inline? Open PDF directly →
Full Text
47,862 characters extracted from source content.
Expand or collapse full text
HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments Shivani Kamtikar‘∗1, Kendall Koe∗1, Justin Wasserman2, Samhita Marri1, Benjamin Walt1, Naveen Kumar Uppalapati1, Girish Krishnan1, Girish Chowdhary1 *Equal Contribution, ‘Corresponding Author 1University of Illinois at Urbana-Champaign, 2Skild AI (skk7, kfkoe2, jbwasse2, marri2, walt, uppalap2, gkrishna, girishc)@illinois.edu Abstract As robotic systems increasingly operate in unstructured, cluttered, and previously unseen environments, there is a growing need for manipulators that combine compliance, adaptability, and precise control. This work presents a real-time hybrid rigid–soft continuum manipulator system designed for robust open-world object reaching in such challenging environments. The system integrates vision-based perception and 3D scene reconstruction with shape-aware motion planning to generate safe trajectories. A learning-based controller drives the hybrid arm to arbitrary target poses, leveraging the flexibility of the soft segment while maintaining the precision of the rigid segment. The system operates without environment-specific retraining, enabling direct generalization to new scenes. Extensive real-world experiments demonstrate consistent reaching performance with errors below 2cm2\,cm across diverse cluttered setups, highlighting the potential of hybrid manipulators for adaptive and reliable operation in unstructured environments. I Introduction Unstructured and cluttered environments present substantial challenges for robotic manipulation due to variability, occlusion, and constrained accessibility. In domains like agriculture, environmental monitoring, and disaster relief, robots must operate in complex scenes containing diverse obstacles, ranging from dense foliage to collapsed infrastructure. These settings often violate the assumptions of controlled environments [26, 27], demanding systems that adjust to clutter, avoid collisions, and perform goal-directed reaching with minimal prior knowledge of its surroundings. Many recent advances, however, target relatively uncluttered environments, such as tabletop settings [6, 34, 15], where rigid manipulators with depth cameras struggle in tight or cluttered spaces due to limited dexterity and absence of passive compliance. Soft continuum arms (SCAs) provide greater adaptability and safe interaction [33, 17], but their effectiveness is limited by a restricted workspace and challenges in accurate modeling, planning, and control. Hybrid continuum manipulators offer a promising alternative, combining the reach and stability of rigid arms with the compliance and distal flexibility of SCAs [32, 37, 19]. These manipulators consist of rigid segments connected to a soft distal section. They have potential in scenarios requiring a unique combination of dexterity, adaptability, and precision, qualities that conventional rigid and soft manipulators struggle to achieve on their own. Recent studies have explored the design and modeling of hybrid manipulators [32, 42], along with control and learning frameworks [19, 25, 14]. While some works address obstacle avoidance, they typically assume perfect prior knowledge of the environment or obstacles [37, 21, 33]. Depth cameras can aid perception and scene modeling in novel environments; however, they are often impractical due to the limited payload capacity of SCAs. These limitations expose a critical gap in existing soft and hybrid manipulators: the lack of a unified, real-time framework that enables safe, goal-directed reaching in cluttered, previously unseen environments without relying on environment-specific modeling, calibration, or retraining. In particular, two key questions remain unresolved: (1) how hybrid systems can perceive and avoid obstacles while reasoning over their deformable geometry, and (2) how such capabilities can generalize across diverse unseen environments. Figure 1: Our system solves reaching tasks while using a hybrid rigid-soft continuum arm system. Setup consists of a B3 (three bending actuators) soft continuum arm with a small RGB camera mounted on a 6DOF rigid manipulator. The setup also has a magnetic sensor (used only for data collection) that measures the pose of the end effector. We show two overlayed snapshots of the manipulator reaching toward a goal object through a cluttered environment. To address this gap, we present a real-time, vision-guided hybrid manipulation framework that enables reliable object reaching in cluttered, unstructured, and previously unseen environments. The platform combines a standard 6-DoF industrial robotic arm with a tri-chambered bending (B3) SCA mounted at its distal end (Fig. 1). The proposed framework couples multi-view reconstruction with shape-aware motion planning, explicitly reasoning over the deformable backbone of the hybrid manipulator to ensure safe and feasible motion, and pairs this with a learning-based controller for accurate execution in visually occluded, obstacle-rich scenes. We evaluate the framework across four increasingly challenging real-world environments, demonstrating reliable and robust performance under clutter and unseen scene geometry. This paper introduces the proposed framework, details its sensing, planning, and control components, and presents extensive real-world experiments that validate the feasibility of open-world, shape-informed hybrid reaching, while also discussing current limitations. Summary of contributions: 1. Real-world demonstration of hybrid reaching. To our knowledge, this is the first hybrid manipulator system capable of open-world object reaching in cluttered, unseen environments. The system achieves sub-2 cm accuracy and high success rates across multiple test environments without environment-specific fine-tuning. 2. Multi-view RGB reconstruction for obstacle-aware planning. We develop a lightweight multi-view perception pipeline that enables obstacle-aware planning without relying on depth sensors or environment-specific retraining, making the system suitable for deployment on payload-limited hybrid manipulators. 3. Role of shape estimation in safe manipulation. We demonstrate that explicitly incorporating shape estimation into the planning loop is critical for safe and reliable hybrid manipulation, significantly improving both safety and task success in cluttered environments. I Related Work Figure 2: Our pipeline for real-time reaching and control of a hybrid manipulator in complex, unstructured environments. The pipeline comprises goal detection, 3D reconstruction, shape-informed path planning, and a learned controller for hybrid manipulators. 3D reconstruction, integrated with an occupancy grid, enhances scene understanding and identifies traversable areas. Shape-informed path planning optimizes paths by effectively navigating around obstacles. Additionally, our hybrid manipulator controller enables actuation to any arbitrary pose within the workspace. Vision Guided Reaching. Goal-based reaching with rigid manipulators has been extensively studied in both structured and semi-structured environments as it supports core capabilities in robotic manipulation, from pick-and-place operations [12, 36] to assembly [41, 40] and inspection [1]. Recent vision-guided approaches leverage dense visual representations, such as multi-view reconstructions or point clouds, to enable reaching in cluttered scenes [44, 4]. These perception-driven techniques enable manipulators to reason about geometry and occlusion, improving safety and precision in motion planning. Although existing learning-based frameworks have made significant progress in rigid manipulation, they typically rely on fixed kinematics, calibrated depth sensors, or known goal states [10, 4, 5]. In contrast, our system overcomes these limitations by leveraging low-cost RGB cameras for 3D reconstruction and geometry-aware planning, enabling real-time open-world reaching with hybrid, deformable kinematics - without requiring prior demonstrations or explicit goal poses. Hybrid and Soft-Arm Systems in Clutter. Planning in cluttered environments presents unique challenges for hybrid and soft manipulators, as their deformable geometries must navigate confined spaces while avoiding damage or excessive strain. Sampling-based algorithms such as RRT* and its variants [21, 23, 22] have been applied to SCAs, but often assume full environmental knowledge or rely on third-person observations. More broadly, only a small number of studies have addressed hybrid manipulator planning [21, 22, 16, 19], and even fewer have reported consistent performance on real-world hardware [22, 16]. Classical planners such as RRT* can effectively explore the configuration space and generate feasible paths. However, they depend on complementary modules such as shape sensing and modeling to handle the complex nonlinear dynamics of hybrid manipulators. Frameworks such as SOFA [7], Elastica [24], and Sorotoki [2] have enabled high-fidelity modeling of SCAs for tasks such as obstacle navigation, shape estimation, and shape control. However, these studies are typically confined to simulation and lack validation on real hardware, which limits their applicability in realistic, sensor-noisy environments. Shape estimation in real-world systems remains equally challenging. Although commercial fiber optic sensors provide accurate deformation feedback [8], their high cost restricts practical deployment. Recent research on SCA modeling and control has shown promise by leveraging learning-based frameworks [17, 37, 26, 27, 19]. However, these methods are typically validated in structured or obstacle-free settings, and their reliance on reinforcement learning or simulation-based training often leads to challenges in sim-to-real transfer and reward tuning. In this work, we demonstrate a learning-based hybrid manipulator system that operates in cluttered, previously unseen real-world environments. By reasoning directly from a first-person visual perspective and accounting for the geometry of both rigid and soft segments, the system enables safe and adaptive motion generation in complex real-world settings. Hybrid-Arm Robot Control. Various numerical methodologies, including Piecewise Constant Curvature (PCC) [14, 38], Cosserat Rod Theory [31], and Variable-Strain Model [30], have been employed to model and control the dynamics of SCAs. For instance, He et al. [14] combined a PCC model with a data-driven module to control a hybrid manipulator. Similarly, Xu et al. [39] demonstrated a two-stage visual servoing strategy for pushing objects linearly with a hybrid system. Despite these advances, the inherently flexible, deformable, and high-DOF nature of soft and hybrid manipulators introduces significant complexity, making accurate modeling and control with traditional methods particularly challenging. Learning-based control offers a model-free alternative that has proven effective for purely soft systems [17, 26, 27], as it eliminates the need for precise analytical modeling of hyper-redundant structures. Building on this paradigm, we employ a learning-based controller that enables our hybrid manipulator to reach arbitrary 6-DoF poses. This approach achieves precise control across a wide range of configurations without explicit system modeling, demonstrating the practicality of learning-based strategies for hybrid manipulation in real-world settings. I Method To achieve reliable reaching with hybrid manipulators in cluttered environments, we propose a three-stage pipeline. First, the perception module integrates 3D scene reconstruction with open-world object detection, eliminating the common assumption that the target must be visible from the robot’s initial pose. Second, a shape-informed path planner leverages the 3D-reconstruction to compute obstacle-aware trajectories toward the goal pose. Finally, our learning-based controller executes these trajectories by actuating both rigid and soft segments of the manipulator to accurately reach the target. I-A Perception The perception module’s role within the pipeline is illustrated in Fig. 2. It reconstructs the environment to provide a structured scene representation, forming the geometric basis for obstacle-aware planning and goal-directed control. The perception module begins by capturing RGB images from a monocular tip-mounted camera while the hybrid manipulator explores a small region around its home position to obtain multiple viewpoints. This multi-view strategy improves goal detection even when the target is initially occluded. The images are processed using Mast3r [20] to estimate metric depth and reconstruct the scene. If the goal remains undetected, the manipulator performs additional exploratory motions to acquire new viewpoints, refining the reconstruction. This is challenging since depth cues are inferred solely from a low-resolution monocular stream, increasing depth-estimation uncertainty. Goal objects are localized using YOLO-World [3], an open-world detector that enables natural-language queries and avoids dependence on fixed object categories. This design choice enables open-world flexibility and avoids dependence on a fixed set of object categories. YOLO-World is further chosen for its lightweight inference speed and ease of integration with real-time pipelines, making it well-suited for on-board robotic perception. Detected object locations are fused with the depth map to estimate 3D target positions. The target with the highest confidence is then paired with four candidate approach directions to generate corresponding candidate poses. The reconstructed point cloud is discretized into an occupancy grid encoding traversable and occupied regions. This grid serves as the environment model for the downstream shape-informed path planner, which generates safe trajectories for the hybrid manipulator to reach the target object. Details of the planning process follow in the next section. I-B Path Planning with Shape Estimation Path planning is crucial for enabling hybrid manipulators to navigate around clutter and reliably reach a target, especially with occlusions. A key contribution of this paper is a shape-aware planning formulation that explicitly reasons over the deformable backbone of the soft segment during planning. Unlike conventional approaches that perform collision checking only at the end-effector [37], our planner evaluates collisions along the entire hybrid backbone and enforces asymmetric feasibility constraints: strictly collision-free motion for the rigid links while allowing bounded, controlled contact for the soft segment. This formulation exploits compliance as a planning primitive rather than treating deformation as a disturbance. Shape Estimation: To incorporate the SCA’s geometry into planning, we use a Constant Curvature (C) model [13, 29, 35]. Although C-based modeling is less accurate than higher-fidelity alternatives (e.g., Cosserat rod solvers), it is computationally efficient and suitable for online planning. Estimating the SCA’s shape during path generation reduces the likelihood of collisions while maintaining real-time feasibility. Importantly, shape-informed planning ensures that candidate paths are not only kinematically valid but also safe with respect to the manipulator’s deformable geometry. To compute the shape of the hybrid manipulator for path planning, the homogeneous transformation matrix of the shape of the soft distal link, TsT_s, is calculated as follows: Ts=[Rt01][cos(ϕ)cos(κl)−sin(ϕ)−cos(ϕ)sin(κl)−1κcos(ϕ)(1−cos(κl))sin(ϕ)sin(κl)cos(ϕ)−sin(ϕ)sin(κl)−1κsin(ϕ)(1−cos(κl))sin(κl)0cos(κl)1κsin(κl)0001]T_s= bmatrixR&t\\ 0&1 bmatrix bmatrix (φ) (κ l)&- (φ)&- (φ) (κ l)&- 1κ (φ)(1- (κ l))\\ (φ) (κ l)& (φ)&- (φ) (κ l)&- 1κ (φ)(1- (κ l))\\ (κ l)&0& (κ l)& 1κ (κ l)\\ 0&0&0&1 bmatrix (1) R and t represent the orientation and position of the rigid link to which the SCA is attached, κ represents curvature of the SCA, ϕφ represents rotation about the vector tangent to the SCA base, and l is the arc length. The soft-arm shape is estimated by discretizing l to compute poses along the manipulator. The rigid segment shape is determined using joint angles and known rigid body geometry. Path Planning: We implement a modified Rapidly-exploring Random Tree Star (RRT*) algorithm [18] for path planning. The planner incrementally builds a tree by sampling random states, steering toward samples from the nearest node, and evaluating candidate edges for feasibility. The core modifications are: 1) Collision-aware expansion: Candidate trajectories are evaluated against the occupancy grid by checking collisions along the entire arm backbone. To guarantee safe motion, the rigid segment must remain collision-free, while the soft segment is allowed limited contact within a predefined threshold. 2) Collision thresholding: A tunable hyperparameter, τ, defines the maximum allowable number of collisions along a candidate path. This threshold is empirically determined and allows tailoring the planner to different environments, for example, tolerating higher contact rates in deformable environments, and enforcing stricter constraints in environments with rigid or safety-critical obstacles. Let q∈Qq∈ Q denote a candidate arm configuration, and let P(q)=pii=1N,pi∈ℝ3P(q)=\p_i\_i=1^N,p_i ^3 be a set of N uniformly sampled points along the manipulator’s backbone under the configuration q, obtained via shape estimation described above. N was set to 50. Let the occupancy grid be represented by an indicator occ:ℝ3 )0,1occ:R^3 [2.3pt]5.0pt0.4pt -4.0mu 41 \0,1\, that returns 1 if and only if the point lies in an occupied voxel and 0 otherwise. The total collision count for configuration q is defined as: C(q)=∑i=1Nocc(pi)C(q)= _i=1^Nocc(p_i) (2) A configuration is considered feasible if and only if C(q)≤τC(q)≤τ, where τ∈0,…,Nτ∈\0,...,N\. The feasibility condition for the rigid component is given by Cr(q)=0C_r(q)=0, and the feasibility condition for the soft segment is given by Cs≤τC_s≤τ, enforcing strict collision-free motion for the rigid section, while allowing limited allowable contact for the soft segment based on the predefined threshold. 3) Dynamic shape updates: The C model is recomputed for each steering step to ensure that shape-dependent collisions are accurately represented during tree expansion. Feasible paths are post-processed with a shortcutting heuristic [28] to remove redundant waypoints and produce smoother trajectories. If no solution is found within a maximum number of iterations, the planner terminates and returns failure, prompting replanning. The trajectory is generated before execution, and the waypoints are then passed to the controller. Combining occupancy-based feasibility with online shape estimation allows efficient and safe trajectory generation by leveraging the SCA’s compliance and limiting collisions. Figure 3: The hybrid arm controller takes as input the start pose, psp_s, and relative pose to the goal prelp_rel and outputs hybrid arm actuations. This fully learned controller successfully actuates the hybrid system to an arbitrary pose, avoiding the need for complex modeling of the hybrid system while enabling closed-loop control. Figure 4: (a) Experimental Setups: The four experimental setups include No~Obstacles , Obstacles , Clutter , and Hole . An image of the setup, along with the reconstruction obtained from Mast3r [20] is visualized. (b) Experimental results (one example test run for each setup): Shows initial view (start of the test), two intermediate views, and the final view (end of test) obtained from the tip camera. The final column shows the query/goal object that the hybrid arm was tasked to reach. I-C Learned Hybrid Arm Controller In previous work, it has been reported that controlling a hybrid manipulator is difficult due to its virtually infinite degrees of freedom, non-linear characteristics arising from material properties, coupling of rigid and soft segments, and the variety of designs and actuation techniques. To address this, we employ a learning-based closed-loop controller to actuate the hybrid manipulator (soft and rigid components) to a target pose. During deployment, the controller takes waypoints from the path planner and actuates the system incorporating the pose as feedback (Fig. 3). The controller inputs are the start pose and the relative pose to the goal, and the outputs are the soft-actuator commands and target joint angles for the rigid segment. Data Collection and Training. Data collection is performed by systematically incrementing each actuation dimension to ensure the hybrid manipulator explores its entire 9DOF workspace. For each configuration, the pose (position and orientation) and corresponding control inputs are recorded after oscillations settle, yielding 95369536 data points. The model is trained using a mean-squared error (MSE) loss between the predicted actuations u u and ground truth controls ugtu_ gt: L(x)=1‖D‖∑i=1D‖u^i−ugti‖2L(x)= 1||D|| _i=1^D|| u^i-u^i_ gt||^2 -2.84544pt (3) The input, x, is a concatenation of the current pose and the relative pose to the goal. The inputs were passed through a 20-layer MLP network designed with a bottleneck architecture and ResNet blocks to compute the desired normalized actuations. The model is trained with a batch size of 2000, learning rate of 1×10−41× 10^-4, and hidden size of 15,000. TABLE I: Across all environments, our method consistently outperforms the baselines in cluttered scenes. 11 trials are performed for each setup. Intuitively, the success rate decreases with the complexity of the environment. The results show the importance of the perception pipeline and the shape-informed planner in successfully navigating obstacles. Furthermore, the data demonstrates the need for the soft segment in improving the success rate in cluttered, complex environments. No~Obstacles Obstacles Clutter Hole # Method SR@2cm (%) ↑ SR@Touch (%) ↑ SR@2cm (%) ↑ SR@Touch (%) ↑ SR@2cm (%) ↑ SR@Touch (%) ↑ SR@2cm (%) ↑ SR@Touch (%) ↑ 1 Rigid only 100.0 72.7 45.5 36.3 54.5 18.2 18.2 9.1 2 Img2Act 81.8 45.5 54.5 27.3 45.5 9.1 36.4 9.1 3 Ours 90.9 63.6 75.0 66.7 90.9 72.7 54.5 27.3 IV Experiments and Results IV-A Task Setup Experiments are performed in four test environments (Fig. 4(a)) with increasing difficulty that requires using the unique shape of the soft part of the manipulator. The physical setup description is given in the supplemental material. The first environment is No~Obstacles , which acts as a basic table-top test environment. Next is the Obstacles environment, where obstacles are placed in the environment to block the immediate path to the goal. Following this is the Clutter environment, where a plant is placed in the scene. This requires the methods to be able to reach a target fruit on the plant even when faced with heavy occlusion. Finally, the Hole environment, where the goal is placed on the other side of a wall that contains a hole. The goal is fully occluded at the start of each episode, and the SCA needs to either look through the hole or around the wall to reach the goal. Metrics: Each trial is evaluated with three different success metrics. Here, SR refers to the success rate: 1. SR@2cm: The trial is a success if the end effector reaches within 2 cm of the query object and the object is in the line of sight. This metric is applicable for scenarios where the manipulator needs to observe and survey its surroundings. 2. SR@Touch: The trial is a success if the end effector touches the object. This metric is used for scenarios where it is necessary to touch/interact with the target objects. 3. Trans. Err: Translation error with respect to the distance between the initial end effector position and the target position. This metric demonstrates precise reaching across larger distances and the range of motion of the system. Baselines: As a baseline, we adopt a learned visual servoing approach [17] that maps images from a tip-mounted camera directly to actuator commands for the SCA. This method, Img2Act, has robust performance in structured environments and serves as a representative visual servoing baseline. It was trained by collecting images in each experimental scene according to the data collection procedure of the prior work [17], identical to that used for our controller. Separate models were trained for the four scenes using the same network architecture. The method requires environment access for training and a goal image at inference, limiting its ability to reach objects in unseen environments. The model was trained on 5184 image–actuation pairs for the No~Obstacles setup, then fine-tuned with 1344 pairs for Obstacles and another 1344 for Hole . Pretrained weights enabled efficient adaptation through limited fine-tuning because Obstacles and Hole share similar visual features with No~Obstacles . A separate model was trained for the visually distinct Clutter setup using 5696 data pairs. Hyperparameters were tuned for each environment, yielding distinct, environment-specific models. As another baseline, Rigid Only, we evaluate a purely rigid manipulator by replacing the soft segment with a rigid link of equivalent length. This configuration preserves the overall kinematic reach but eliminates compliance, serving as a comparison that highlights the advantages of the soft segment in navigating cluttered environments. Baseline results are detailed in Table I and I. Ablations: To further evaluate our method, we perform the following ablation experiments. 1) Effect of shape-informed planning: We assess the contributions of the shape-informed planner within our pipeline. This is done by evaluating our pipeline with and without the shape-informed planner across all environments (11 trials for every environment), shown in Table I 2) Effect of Collision Thresholds We also study the effects of the collision threshold hyperparameter (τ), comparing strict (τ=0τ=0), moderate (τ=4−6τ=4-6), and lax (τ=10−15τ=10-15) settings, which highlights the trade-off between safety (fewer collisions) and feasibility (higher path success rates). We compare results across all environments, with five trials per environment. Results are summarized in Table IV. 3) Effect of controller input format on reaching accuracy: We evaluate four input configurations to assess how state and goal representations affect controller performance: (1) current pose and transform to goal, (2) goal pose only, (3) current and goal poses, and (4) transform to goal only. Results are shown in Table LABEL:tab:pose2act. IV-B Experimental Results TABLE I: Our method enables precisely reaching objects across larger distances than previous purely soft and hybrid manipulators. Translation errors (Mean ± STD) measuring the distance from the end effector to the target object across all environments and methods are shown. Additionally, the initial distance from the target averaged across the 11 trials (Init. Δ (cm)) is also given. No~Obstacles Obstacles Clutter Hole # Method Trans. Err (cm) ↓ Init. Δ (cm) Trans. Err (cm) ↓ Init. Δ (cm) Trans. Err (cm) ↓ Init. Δ (cm) Trans. Err (cm) ↓ Init. Δ (cm) 1 Rigid Only 1.0± 1.2 33.0± 2.8 7.6± 6.2 31.7± 2.0 5.2± 5.5 31.2± 1.9 9.8± 8.1 31.5± 1.8 2 Img2Act 1.6± 1.8 33.0± 2.8 4.8± 2.7 31.7± 2.0 9.7± 9.3 32.0± 2.0 7.0± 4.5 32.3± 1.9 3 Ours 1.2± 1.7 33.2± 2.0 1.6± 2.3 32.5± 2.0 1.0± 1.9 31.3± 1.9 2.9± 2.4 31.6± 1.8 TABLE I: Shape estimation is essential for reliable operation in cluttered environments. Our experiments demonstrate that incorporating shape-informed path planning is critical for generating feasible trajectories and avoiding severe collisions. No~Obstacles Obstacles Clutter Hole # Method SR@2cm SR@Touch Trans. Err SR@2cm SR@Touch Trans. Err SR@2cm SR@Touch Trans. Err SR@2cm SR@Touch Trans. Err (%) ↑ (%) ↑ (cm) ↓ (%) ↑ (%) ↑ (cm) ↓ (%) ↑ (%) ↑ (cm) ↓ %) ↑ (%) ↑ (cm) ↓ 1 No-Shape 90.9 63.6 1.6± 2.9 45.5 36.4 6.0± 6.8 54.5 27.3 4.0± 5.5 45.5 18.2 5.0± 4.5 2 Ours 90.9 63.6 1.2± 1.7 75.0 66.7 1.6± 2.3 90.9 72.7 1.0± 1.9 54.5 27.3 2.9± 2.4 Result 1: Our hybrid manipulator system enables reliable object reaching in complex environments (Table I) and I. Experiments demonstrate that our hybrid manipulator reliably performs open-world reaching in cluttered, unseen environments. These results highlight the manipulator’s ability to reach target objects while safely navigating around obstacles. In the No~Obstacles setup, it achieved 90.9% for SR@2cm and 63.6% for SR@Touch. As expected, success rates decline with increasing environmental complexity; however, our system maintains robust performance even in the most cluttered setting, achieving 90.9% for SR@2cm and 72.7% for SR@Touch in the Clutter setup and 75% for SR@2cm and 66.7% for SR@Touch in the Obstacles setup. The Hole scenario, requiring precise motion through a narrow aperture, remains the most challenging: success rates reached 54.5% (SR@2cm) and 27.3% (SR@Touch), demonstrating feasibility in this highly constrained case. Representative successful trials across all environments are shown in Fig. 4(b). For the Rigid Only baseline, where the soft segment was replaced by a rigid link, the system performed well in No~Obstacles (100% and 72.7%) but suffered frequent collisions in confined spaces, dropping to 45.5% (SR@2cm) in Obstacles and 54.5% in Clutter . This is due to the absence of the compliance and dexterity provided by the soft continuum actuator, which are essential for navigating clutter and accessing partially occluded targets. Differences between SR@2cm and SR@Touch metrics arise from depth-estimation errors in Mast3R. Overall, results validate the use of a hybrid design to achieve robust reaching in unstructured, cluttered settings where existing methods fall short. Result 2: Multi-view reconstruction enables real-time, generalizable, and obstacle-aware reaching (Table I). Integrating multi-view reconstruction into the perception pipeline markedly improves performance and generalization. The Img2Act baseline [17] performs well in simple, uncluttered scenes, achieving 81.8% for SR@2cm and 45.5% for SR@Touch in the No~Obstacles setup, but fails to avoid obstacles and requires retraining for each environment. The lower performance on the SR@Touch metric is primarily due to inaccuracies in the learned controller, which come from the non-linearities present in the SCA. In contrast, our system leverages 3D reconstruction for real-time reaching in unseen environments without retraining, explicitly reasoning about obstacles to ensure safe trajectories. These results confirm that combining perception, planning, and control yields reliable, generalizable reaching performance. Figure 5: Rigid Only baseline hardware setup. The soft segment in Fig. 1 was replaced with an equivalent length rigid link. Result 3: Shape estimation improves path safety and efficiency (Table I). The No~Obstacles setup is not significantly affected by SCA shape estimation with both methods achieving 90.9% (SR@2cm), as there were no collisions in this setup. However, in more complex scenarios, the absence of shape estimation significantly compromises both the safety of the environment and the SCA. In the Obstacles setup, lack of shape estimation led to frequent, sometimes severe, collisions with obstacles, reducing the success rate to 45.5% as compared to our method, which gave 75%. While the arm often reached its target in the Clutter setup, it occasionally became entangled with surrounding leaves and branches. Here too, our method achieved a success rate of 90.9%, surpassing the 54.5% success rate of the no-shape method. In Hole , lack of shape estimation caused repeated collisions and overbending, reducing success to 45.5%, while our method achieved 54.5%. Although higher-fidelity shape estimation methods exist [43, 11], our approximate model provides real-time performance with substantial gains in safety. Even with shape estimation, limited contact is permitted due to the inherent resilience of the SCA. Analysis of collision thresholds (Table IV) shows a clear trade-off between feasibility and safety: a strict threshold (τ=0τ=0) prevents collisions but limits feasible paths, while a lax threshold (τ=10τ=10–1515) increases contact and reduces success. A moderate range (τ=4τ=4–66) achieves the best balance, enabling safe interactions while maintaining high path feasibility in diverse environments. Result 4: Hybrid manipulators achieve precise reaching across extended workspaces (Table I). Our system enables accurate reaching over larger distances than prior soft-only [9] or hybrid systems [37]. The mean translation error remains below 2 cm for No~Obstacles , Obstacles , and Clutter setups, and below 3 cm for the challenging Hole case. This expanded workspace makes hybrid manipulators more practical for applications such as agriculture and medical intervention, where soft systems are limited by reach. Result 5: The learned controller reliably actuates the hybrid manipulator to arbitrary 6-DoF goals. Given the current and relative goal poses, the controller outputs actuations for both rigid and soft components in a closed-loop fashion, achieving an average position error of 1.7 cm across all environments. Ablation experiments on the No~Obstacles setup (15 trials for every input type), shown in Table LABEL:tab:pose2act, confirm that using both current and relative poses yields the most accurate results among all input configurations tested. TABLE IV: Effect of collision threshold on goal reaching Strict thresholds (τ=0τ=0) yield safer but less feasible paths, while lax thresholds (τ=10−15τ=10-15) come at the cost of increased collisions. Moderate threshold (τ=4−6τ=4-6) strikes a good balance between safety and feasibility. Metric τ=0τ=0 τ=10−15τ=10-15 τ=4−6τ=4-6 SR@2cm 47.7% 59.1% 77.8% SR@Touch 29.5% 36.3% 57.5% Trans. Err (cm) 5.9± 5.6 4.1± 4.8 1.6± 2.0 TABLE V: Controller ablations show that giving the current pose and the relative pose between the current and target pose improves prediction of the target actuations Input Type Trans. Err (cm) ↓ Current Pose + Relative Pose to Goal 1.2± 1.7 Goal Pose 9.1± 7.8 Current Pose + Goal Pose 11.4± 7.3 Relative Pose to Goal 15.9± 10.5 V Limitations and Future Work The experiments demonstrate that open-world reaching of unseen objects in cluttered environments is achievable with a hybrid manipulator. However, the controller does not include an explicit objective to minimize control effort or actuation changes. Consequently, the predicted actuation commands reflect the learned data distribution rather than an energy-optimal solution. Even though this doesn’t impact the achieved reaching accuracy, incorporating effort-aware regularization could improve trajectory efficiency. Another important direction for future work is evaluating payload-dependent behavior, enabling hybrid manipulators to support load-bearing tasks through load-aware planning. Dynamic environments with moving obstacles or targets are also not considered, which would require continuous replanning. Finally, future work could explore how obstacles might be leveraged, not just avoided, for reaching tasks. VI Conclusion We present a vision-guided framework for hybrid rigid–soft manipulators operating in cluttered, unstructured environments. The system integrates multi-view 3D reconstruction and shape-informed path planning for reliable obstacle avoidance, paired with a learning-based controller that accurately actuates the manipulator to arbitrary poses. Experiments show consistent improvements over baselines in both obstacle avoidance and target reaching, particularly in settings where rigid manipulators fail. These results highlight the effectiveness of combining shape-aware, perception-driven planning with learning-based control for robust real-world deployment of hybrid continuum manipulators. References [1] H. Ben Abdallah, I. Jovančević, J. Orteu, and L. Brèthes (2019) Automatic inspection of aeronautical mechanical assemblies by matching the 3d cad model and real 2d images. Journal of Imaging 5 (10), p. 81. Cited by: §I. [2] B. J. Caasenbrood, A. Y. Pogromsky, and H. Nijmeijer (2024) Sorotoki: a matlab toolkit for design, modeling, and control of soft robots. IEEE Access. Cited by: §I. [3] T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, and Y. Shan (2024) Yolo-world: real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16901–16911. Cited by: §I-A. [4] E. Chisari, N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada (2024) Learning robotic manipulation policies from point clouds with conditional flow matching. arXiv preprint arXiv:2409.07343. Cited by: §I, §I. [5] S. Christen, W. Yang, C. Pérez-D’Arpino, O. Hilliges, D. Fox, and Y. Chao (2023) Learning human-to-robot handovers from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9654–9664. Cited by: §I. [6] K. Fang, F. Liu, P. Abbeel, and S. Levine (2024) MOKA: open-world robotic manipulation through mark-based visual prompting. Proceedings of Robotics: Science and Systems, Delft, Netherlands. Cited by: §I. [7] F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, et al. (2012) Sofa: a multi-model framework for interactive physical simulation. Soft tissue biomechanical modeling for computer assisted surgery, p. 283–321. Cited by: §I. [8] K. C. Galloway, Y. Chen, E. Templeton, B. Rife, I. S. Godage, and E. J. Barth (2019) Fiber optic shape sensing for soft robotics. Soft robotics 6 (5), p. 671–684. Cited by: §I. [9] Y. Gan, P. Li, H. Jiang, G. Wang, Y. Jin, X. Chen, and J. Ji (2022) A reinforcement learning method for motion control with constraints on an hpn arm. IEEE Robotics and Automation Letters 7 (4), p. 12006–12013. Cited by: §IV-B. [10] A. Goyal, A. Mousavian, C. Paxton, Y. Chao, B. Okorn, J. Deng, and D. Fox (2022) Ifor: iterative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14787–14797. Cited by: §I. [11] A. M. Gruebele, A. C. Zerbe, M. M. Coad, A. M. Okamura, and M. R. Cutkosky (2021) Distributed sensor networks deployed using soft growing robots. In 2021 IEEE 4th International Conference on Soft Robotics (RoboSoft), p. 66–73. Cited by: §IV-B. [12] A. Hammoud, M. Khoramshahi, Q. Huet, and V. Perdereau (2025) Online object localization in a robotic hand by tactile sensing. In 2025 IEEE/SICE International Symposium on System Integration (SII), p. 645–652. Cited by: §I. [13] M. W. Hannan and I. D. Walker (2003) Kinematics and the implementation of an elephant’s trunk manipulator and other continuum style robots. Journal of robotic systems 20 (2), p. 45–63. Cited by: §I-B. [14] S. He, L. Sun, Y. Xu, and D. Li (2023) A modeling and data-driven control framework for rigid-soft hybrid robot with visual servoing. IEEE Robotics and Automation Letters. Cited by: §I, §I. [15] B. Huang, X. Zhang, and J. Yu (2024) Toward optimal tabletop rearrangement with multiple manipulation primitives. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 10860–10866. Cited by: §I. [16] H. Huang, H. Wang, C. Fang, M. Yan, R. Xu, Y. Zhang, Z. Wang, F. Ying, J. Liu, C. Laschi, et al. (2025) Grasping by spiraling: reproducing elephant movements with rigid-soft robot synergy. npj Robotics 3 (1), p. 18. Cited by: §I. [17] S. Kamtikar, S. Marri, B. Walt, N. K. Uppalapati, G. Krishnan, and G. Chowdhary (2022) Visual servoing for pose control of soft continuum arm in a structured environment. IEEE Robotics and Automation Letters 7 (2), p. 5504–5511. Cited by: §I, §I, §I, §IV-A, §IV-B. [18] S. Karaman and E. Frazzoli (2011) Sampling-based algorithms for optimal motion planning. The international journal of robotics research 30 (7), p. 846–894. Cited by: §I-B. [19] K. Koe, S. Marri, B. Walt, S. Kamtikar, N. K. Uppalapati, G. Krishnan, and G. Chowdhary (2025) Learning-based position and orientation control of a hybrid rigid-soft arm manipulator. Journal of Mechanisms and Robotics 17 (7), p. 071010. Cited by: §I, §I, §I. [20] V. Leroy, Y. Cabon, and J. Revaud (2024) Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, p. 71–91. Cited by: Figure 4, Figure 4, §I-A. [21] Y. Li, T. Miyazaki, Y. Yamamoto, and K. Kawashima (2024) S-rrt*-based obstacle avoidance autonomous motion planner for continuum-rigid manipulator. arXiv preprint arXiv:2409.19110. Cited by: §I, §I. [22] P. Luo, S. Yao, Y. Yue, J. Wang, H. Yan, and M. Q. Meng (2024) Efficient rrt*-based safety-constrained motion planning for continuum robots in dynamic environments. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 9328–9334. Cited by: §I. [23] B. H. Meng, I. S. Godage, and I. Kanj (2022) RRT*-based path planning for continuum arms. IEEE Robotics and Automation Letters 7 (3), p. 6830–6837. Cited by: §I. [24] N. Naughton, J. Sun, A. Tekinalp, T. Parthasarathy, G. Chowdhary, and M. Gazzola (2021) Elastica: a compliant mechanics environment for soft robotic control. IEEE Robotics and Automation Letters 6 (2), p. 3389–3396. Cited by: §I. [25] M. S. Nazeer, Y. T. Ansari, E. Falotico, and C. Laschi (2024) A comparison of model-free controllers for trajectory tracking in a plant-inspired soft arm. In Conference on Biomimetic and Biohybrid Systems, p. 208–220. Cited by: §I. [26] M. S. Nazeer, C. Laschi, and E. Falotico (2023) Soft dagger: sample-efficient imitation learning for control of soft robots. Sensors 23 (19), p. 8278. Cited by: §I, §I, §I. [27] M. S. Nazeer, C. Laschi, and E. Falotico (2024) Rl-based adaptive controller for high precision reaching in a soft robot arm. IEEE Transactions on Robotics 40, p. 2498–2512. Cited by: §I, §I, §I. [28] L. Petit and A. L. Desbiens (2021) RRT-rope: a deterministic shortening approach for fast near-optimal path planning in large-scale uncluttered 3d environments. In 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), p. 1111–1118. Cited by: §I-B. [29] P. Rao, Q. Peyron, S. Lilge, and J. Burgner-Kahrs (2021) How to model tendon-driven continuum robots and benchmark modelling performance. Frontiers in Robotics and AI 7, p. 630245. Cited by: §I-B. [30] F. Renda, C. Armanini, V. Lebastard, F. Candelier, and F. Boyer (2020) A geometric variable-strain approach for static modeling of soft manipulators with tendon and fluidic actuation. IEEE Robotics and Automation Letters 5 (3), p. 4006–4013. External Links: Document Cited by: §I. [31] J. Shi, S. Abad Guaman, J. Dai, and H. Wurdemann (2023) Position and orientation control for hyper-elastic multi-segment continuum robots. IEEE/ASME Transactions on Mechatronics. Cited by: §I. [32] N. K. Uppalapati and G. Krishnan (2020) VaLeNS: design of a novel variable length nested soft arm. IEEE Robotics and Automation Letters 5 (2), p. 1135–1142. Cited by: §I. [33] C. Veil, M. Flaschel, and E. Kuhl (2025) Shape-space graphs: fast and collision-free path planning for soft robots. arXiv preprint arXiv:2510.03547. Cited by: §I, §I. [34] K. Wang, Z. Wang, K. Nakagaki, and K. Perlin (2024) “Push-that-there”: tabletop multi-robot object manipulation via multimodal’object-level instruction’. In Proceedings of the 2024 ACM Designing Interactive Systems Conference, p. 2497–2513. Cited by: §I. [35] X. Wang, Y. Li, and K. Kwok (2021) A survey for machine learning-based control of continuum robots. Frontiers in Robotics and AI 8, p. 730330. Cited by: §I-B. [36] Z. Wang, J. Chen, Z. Chen, P. Xie, R. Chen, and L. Yi (2024) GenH2R: learning generalizable human-to-robot handover via scalable simulation demonstration and imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16362–16372. Cited by: §I. [37] F. Xu, X. Kang, and H. Wang (2024) Hybrid visual servoing control of a soft robot with compliant obstacle avoidance. IEEE/ASME Transactions on Mechatronics. Cited by: §I, §I, §I-B, §IV-B. [38] F. Xu, H. Wang, W. Chen, and Y. Miao (2021) Visual servoing of a cable-driven soft robot manipulator with shape feature. IEEE Robotics and Automation Letters 6 (3), p. 4281–4288. Cited by: §I. [39] F. Xu, H. Wang, Z. Liu, W. Chen, and Y. Wang (2022) Visual servoing pushing control of the soft robot with active pushing force regulation. Soft Robotics 9 (4), p. 690–704. Cited by: §I. [40] J. Xu, K. Liu, Y. Pei, C. Yang, Y. Cheng, and Z. Liu (2022) A noncontact control strategy for circular peg-in-hole assembly guided by the 6-dof robot based on hybrid vision. IEEE Transactions on Instrumentation and Measurement 71, p. 1–15. Cited by: §I. [41] S. Yan, X. Tao, and D. Xu (2021) High-precision robotic assembly system using three-dimensional vision. International Journal of Advanced Robotic Systems 18 (3), p. 17298814211027029. Cited by: §I. [42] S. Zhang, X. Li, D. Sui, Q. Zhang, Z. Wang, T. Zheng, J. Zhao, and Y. Zhu (2024) Design, modeling and implementation of a novel rigid-flexible hybrid robotic arm. In International Conference on Intelligent Robotics and Applications, p. 229–243. Cited by: §I. [43] T. Zheng, C. McFarland, M. Coad, and H. Lin (2024) Estimating infinite-dimensional continuum robot states from the tip. In 2024 IEEE 7th International Conference on Soft Robotics (RoboSoft), p. 572–578. Cited by: §IV-B. [44] H. Zhu, Y. Wang, D. Huang, W. Ye, W. Ouyang, and T. He (2024) Point cloud matters: rethinking the impact of different observation spaces on robot learning. Advances in Neural Information Processing Systems 37, p. 77799–77830. Cited by: §I.