Paper deep dive

Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics

Hendrik Chiche, Antoine Jamme, Trevor Rigoberto Martinez

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 34

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/22/2026, 6:22:39 AM

Summary

The paper presents an offline hand-shadowing and retargeting pipeline that maps human hand articulations from egocentric RGB-D camera data to joint commands for a 6-DOF SO-ARM101 robotic manipulator. The system utilizes MediaPipe for hand landmark detection, depth deprojection, and a damped-least-squares inverse kinematics solver in PyBullet. The pipeline achieves a 90% success rate on a structured pick-and-place benchmark without training, outperforming several vision-language-action policies, though it faces significant challenges with hand occlusion in unstructured environments.

Entities (6)

Intel RealSense D400 · hardware-sensor · 100%LeRobot · software-framework · 100%MediaPipe Hands · software-library · 100%PyBullet · physics-simulation-engine · 100%SO-ARM101 · robotic-manipulator · 100%ACT · vision-language-action-policy · 90%

Relation Signals (4)

LeRobot → controls → SO-ARM101

confidence 100% · replay on the physical robot through the LeRobot framework

MediaPipe Hands → detects → Hand Landmarks

confidence 100% · detects 21 hand landmarks per hand using MediaPipe Hands

Intel RealSense D400 → providesinputto → Hand Shadowing Pipeline

confidence 100% · pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses

PyBullet → solvesikfor → SO-ARM101

confidence 100% · solves a damped-least-squares inverse kinematics problem in PyBullet to produce joint commands for the 6-DOF SO-ARM101 robot

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Teleoperation of low-cost robotic manipulators remains challenging due to the complexity of mapping human hand articulations to robot joint commands. We present an offline hand-shadowing and retargeting pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses. The pipeline detects 21 hand landmarks per hand using MediaPipe Hands, deprojects them into 3D via depth sensing, transforms them into the robot coordinate frame, and solves a damped-least-squares inverse kinematics problem in PyBullet to produce joint commands for the 6-DOF SO-ARM101 robot. A gripper controller maps thumb-index finger geometry to grasp aperture with a four-level fallback hierarchy. Actions are first previewed in a physics simulation before replay on the physical robot through the LeRobot framework. We evaluate the IK retargeting pipeline on a structured pick-and-place benchmark (5-tile grid, 10 grasps per tile) achieving a 90% success rate, and compare it against four vision-language-action policies (ACT, SmolVLA, pi0.5, GR00T N1.5) trained on leader-follower teleoperation data. We also test the IK pipeline in unstructured real-world environments (grocery store, pharmacy), where hand occlusion by surrounding objects reduces success to 9.3% (N=75), highlighting both the promise and current limitations of marker-free analytical retargeting.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

34,063 characters extracted from source content.

Expand or collapse full text

Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics Hendrik Chiche, Antoine Jamme, and Trevor Rigoberto Martinez Hendrik Chiche and Antoine Jamme are with OMGrab Inc. and affiliated with the University of California, Berkeley through the Capstone Project program (e-mail: hendrik_chiche@berkeley.edu; antoine_jamme@berkeley.edu). Trevor Rigoberto Martinez is with the Department of Mechanical Engineering, University of California, Berkeley (e-mail: tre_mart@berkeley.edu). Abstract Teleoperation of low-cost robotic manipulators remains challenging due to the complexity of mapping human hand articulations to robot joint commands. We present an offline hand-shadowing and retargeting pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses. The pipeline detects 21 hand landmarks per hand using MediaPipe Hands, deprojects them into 3D via depth sensing, transforms them into the robot coordinate frame, and solves a damped-least-squares inverse kinematics problem in PyBullet to produce joint commands for the 6-DOF SO-ARM101 robot. A gripper controller maps thumb–index finger geometry to grasp aperture with a four-level fallback hierarchy. Actions are first previewed in a physics simulation before replay on the physical robot through the LeRobot framework. We evaluate the IK retargeting pipeline on a structured pick-and-place benchmark (5-tile grid, 10 grasps per tile) achieving a 90% success rate, and compare it against four vision-language-action policies (ACT, SmolVLA, π0.5 _0.5, GR00T N1.5) trained on leader–follower teleoperation data. We also test the IK pipeline in unstructured real-world environments (grocery store, pharmacy), where hand occlusion by surrounding objects reduces success to 9.3% (N=75N=75), highlighting both the promise and current limitations of marker-free analytical retargeting. Index Terms: Hand tracking, inverse kinematics, MediaPipe, PyBullet, robot teleoperation, retargeting, SO-ARM101, vision-language-action models. Project resources: Video demonstration: https://youtu.be/mKtVg_gYwf8 Datasets and fine-tuned models: https://huggingface.co/Chichonnade Source code: https://github.com/chichonnade/Vision-Based-Hand-Shadowing I Introduction Teaching robots to manipulate objects the way humans do is a central goal of robotics research. Two dominant paradigms have emerged: teleoperation, where a human operator directly controls the robot in real time, and imitation learning, where a policy is trained from recorded demonstrations [4, 5]. Teleoperation provides immediate, interpretable control but traditionally requires expensive hardware such as exoskeletons, leader–follower arm pairs, or VR headsets [7, 8]. Imitation learning can reduce the demonstration burden but demands careful data collection, GPU-based training, and policy evaluation. This work studies how an analytical inverse-kinematics pipeline can bridge these paradigms by converting first-person RGB-D recordings into robot trajectories that can be replayed on hardware and reused as training data for imitation-learning frameworks. Our contributions are: 1. An end-to-end pipeline from egocentric RGB-D video to single-arm robot trajectory generation via analytical inverse kinematics. 2. A sim-to-real transfer workflow using PyBullet for trajectory preview and validation before physical deployment on the SO-ARM101 robot. 3. A quantitative comparison of analytical IK retargeting with four VLA policies on a structured pick-and-place benchmark. 4. An in-the-wild evaluation in grocery store and pharmacy environments to assess robustness under clutter and occlusion. Across these studies, the IK pipeline reaches 90% success on the structured benchmark with zero training, while hand occlusion emerges as the main limitation in unstructured scenes. I Related Work I-A Hand Pose Estimation Real-time hand tracking has advanced rapidly. MediaPipe Hands [2] runs on-device at 30 Hz using a two-stage BlazePalm + landmark pipeline predicting 21 keypoints, and requires no GPU. We adopt MediaPipe for its CPU efficiency and cross-platform availability. WiLoR [1] improves accuracy with a DarkNet localiser followed by a ViT-based 3D reconstructor, achieving state-of-the-art results on FreiHAND and HO3D benchmarks at over 130 FPS but requires GPU inference. Both systems output 21 keypoints compatible with the MANO hand model topology [3]. I-B Teleoperation Systems Open-TeleVision [7] provides stereoscopic VR-based teleoperation for humanoid robots. Bunny-VisionPro [8] uses Apple Vision Pro for bimanual dexterous control with haptic feedback. These systems achieve high fidelity but require specialised VR hardware. Our approach uses only an RGB-D depth camera and 3D-printed accessories. I-C Imitation Learning ACT [4] introduced action chunking with transformers for fine-grained bimanual manipulation, achieving 80–90% success with 10 minutes of demonstrations. SmolVLA [5] is a 450M-parameter vision-language-action model trainable on a single GPU. π0 _0 [6] is a generalist robot foundation model using flow matching over a VLM backbone, trained across 7 robot embodiments and 68 tasks. I-D Low-Cost Robotics The SO-ARM100/101 is a 6-DOF arm using STS3215 bus servos with 30 kg⋅·cm torque [12]. LeRobot [11] provides a hardware-agnostic Python framework that standardises data collection, training, and deployment across robot platforms. I-E Physics Simulation PyBullet [10] provides real-time rigid body dynamics with built-in inverse kinematics via the Bullet Physics SDK. It supports URDF loading, position/velocity/torque control, and GPU-accelerated rendering, making it suitable for rapid prototyping and sim-to-real transfer. I System Overview The system converts egocentric hand observations into robot commands through a multi-stage transformation pipeline (Fig. 1). It requires an Intel RealSense D400 camera [9] mounted on 3D-printed glasses and a single SO-ARM101 follower arm. At the software level, each pipeline stage is implemented as a Transformation[InT, OutT] subclass that wraps a _transform method with automatic latency tracking. Table I lists each stage with its input and output types; Fig. 1 shows the overall data flow. 1. RGB-D Capture (RealSense D400)2. Hand Detection (MediaPipe)3. Depth Deprojection4. Coord. Transform (,R,t)5. Inverse Kinematics (PyBullet)Sim PreviewSO-ARM101 Figure 1: System architecture. The six-stage pipeline converts egocentric RGB-D observations into robot joint commands. Stages 1–5 run sequentially; the output drives either the PyBullet simulation for preview or the physical SO-ARM101 for deployment. TABLE I: Pipeline Stages With Input/Output Types # Stage Input Output 1 Camera Input None CameraFrame 2 Hand Detection CameraFrame ImageSpace 3 Depth Deprojection (ImageSpace, CameraSpace CameraFrame) 4 Coord. Transform CameraSpace RobotSpace 5 IK + Gripper RobotSpace ControlCmds 6 Sim / Robot ControlCmds — I-A Hardware The egocentric sensor platform consists of an Intel RealSense D400-series stereo depth camera mounted on 3D-printed glasses using three PLA/ABS parts (frame, two temple branches, and a camera mount bracket), secured with M1.5, M2, and M3 screws. The camera connects via USB-C 3.1 Gen 1 and streams synchronised RGB and depth at 640×480 resolution and 30 FPS. The robot platform uses a single SO-ARM101 follower arm with 6 revolute joints (5 arm + 1 gripper) driven by STS3215 Feetech bus servos. The complete lab bench setup, including the robot and egocentric camera stand, is shown in Fig. 2. Figure 2: Lab bench setup. The SO-ARM101 robot arm is mounted on a wooden base with the Intel RealSense D400 camera on a stand above, oriented in the same egocentric direction as the glasses-mounted camera. IV Methods IV-A RGB-D Capture and Camera Model The RealSense camera provides aligned RGB-D frames via the pyrealsense2 SDK. We model the camera using the pinhole projection: [uv]=[fx0cx0fycy][X/ZY/Z1], bmatrixu\\ v bmatrix= bmatrixf_x&0&c_x\\ 0&f_y&c_y bmatrix bmatrixX/Z\\ Y/Z\\ 1 bmatrix, (1) where (fx,fy)(f_x,f_y) are focal lengths and (cx,cy)(c_x,c_y) is the principal point, all extracted automatically from the camera stream at initialisation. The camera supports recording to .bag files for offline processing and to MP4 for archival. IV-B Hand Pose Estimation We use MediaPipe Hands [2] for 2D hand detection and landmark localisation. MediaPipe’s two-stage architecture—a BlazePalm detector that locates hands via oriented bounding boxes, followed by a lightweight landmark regression network—predicts 21 keypoints per hand in real time on CPU. The detector outputs left/right hand classification and 2D keypoint coordinates (ui,vi)i=020\(u_i,v_i)\_i=0^20 covering the wrist, thumb, index, middle, ring, and pinky finger joints. To reduce temporal jitter, we apply exponential moving average (EMA) smoothing to the 2D landmarks: p^t=αpt+(1−α)p^t−1,α=0.8, p_t=α\,p_t+(1-α)\, p_t-1, α=0.8, (2) where ptp_t is the raw detection and p^t p_t is the smoothed estimate. IV-C Depth-Based 3D Reconstruction Each 2D landmark (ui,vi)(u_i,v_i) is deprojected to 3D camera coordinates using the depth image D and camera intrinsics: [XYZ]=D[vi,ui]s[(ui−cx)/fx(vi−cy)/fy1], bmatrixX\\ Y\\ Z bmatrix= D[v_i,u_i]s bmatrix(u_i-c_x)/f_x\\ (v_i-c_y)/f_y\\ 1 bmatrix, (3) where s=1000s=1000 is the depth scale (millimetres to metres). Depth values are validated against the range [dmin,dmax]=[0.1,5.0][d_ ,d_ ]=[0.1,5.0] m; landmarks with out-of-range depth are marked invalid. A depth fallback mechanism handles the critical case where one of the two gripper-defining landmarks (THUMB_MCP or INDEX_FINGER_MCP) has invalid depth but the other does not. In this case, the valid landmark’s depth is substituted for the failed one, since these adjacent landmarks are typically at similar depths. A hand pose is rejected entirely if fewer than 50% of its 21 landmarks have valid depth. IV-D Camera-to-Robot Coordinate Transform Landmarks in camera coordinates camp_cam are transformed to the robot base frame robp_rob by a rigid-body transformation composed of a camera-to-robot transform and an optional URDF offset: rob=finalcam+final,p_rob=R_final\,p_cam+t_final, (4) where final=urdf⋅cam,final=urdf⋅cam+urdf.R_final=R_urdf·R_cam, _final=R_urdf·t_cam+t_urdf. (5) The camera rotation matrix camR_cam accounts for the camera mounting angle θ=50∘θ=50 : cam=[−1000sin⁡θ−cos⁡θ0−cos⁡θ−sin⁡θ],R_cam= bmatrix-1&0&0\\ 0& θ&- θ\\ 0&- θ&- θ bmatrix, (6) with translation cam=(0.04,−0.049, 0.48)⊤t_cam=(0.04,\;-0.049,\;0.48) metres. These parameters—including the 50∘50 mounting angle and translation offsets—are extracted directly from the SolidWorks CAD assembly that models both the robot and the camera mount bracket at their exact physical positions. The x-axis negation mirrors the image horizontally (the camera observes hands from a first-person perspective, while the robot faces the operator), and the rotation about the x-axis maps the downward camera view to the robot’s forward–up coordinate system. IV-E Target Pose Computation The robot end-effector target position is the midpoint of the two gripper-defining hand landmarks: target=12(THUMB_MCP+INDEX_FINGER_MCP).p_target= 12 (p_THUMB\_MCP+p_INDEX\_FINGER\_MCP ). (7) The target orientation is constructed as a rotation matrix from three orthogonal axes derived from hand geometry: 1 _1 =INDEX_FINGER_MCP−THUMB_MCP‖INDEX_FINGER_MCP−THUMB_MCP‖, = p_INDEX\_FINGER\_MCP-p_THUMB\_MCP\|p_INDEX\_FINGER\_MCP-p_THUMB\_MCP\|, (8) =12((THUMB_TIP−THUMB_MCP) = 12 ((p_THUMB\_TIP-p_THUMB\_MCP) +(INDEX_FINGER_TIP−INDEX_FINGER_MCP)), +(p_INDEX\_FINGER\_TIP-p_INDEX\_FINGER\_MCP) ), (9) d =‖, = d\|d\|, 3 _3 =1×^‖1×^‖, = e_1× d\|e_1× d\|, (10) 2 _2 =3×1. =e_3×e_1. (11) The resulting rotation matrix =[123]R=[e_1\;\;e_2\;\;e_3] is converted to a quaternion via scipy.spatial.transform for the IK solver. When fingertip landmarks are unavailable (due to occlusion or depth failure), a fallback orientation is computed using only THUMB_MCP, INDEX_FINGER_MCP, and WRIST, replacing the average finger direction d with the wrist-to-gripper vector. IV-F Inverse Kinematics Given the target position targetp_target and orientation quaternion targetq_target, we solve for joint angles θ=(θ1,…,θ5)θ=( _1,…, _5) of the 5-DOF arm using PyBullet’s calculateInverseKinematics: θ∗=arg⁡minθ⁡‖FK(θ)−(target,target)‖2+λ‖θ−θrest‖2,θ^*= _θ \|FK(θ)-(p_target,q_target) \|^2+λ\|θ- _rest\|^2, (12) where FK(⋅)FK(·) is the forward kinematics function, θrest _rest is the current joint state (used as the rest pose), and λ captures per-joint damping extracted from the URDF (clamped to ≥0.001≥ 0.001 to prevent solver instability). The solver runs up to 100 iterations with a residual threshold of 10−410^-4. EMA smoothing (α=0.5α=0.5) is applied to the IK solution to suppress temporal jitter: θ^t=0.5θt∗+0.5θ^t−1. θ_t=0.5\,θ^*_t+0.5\, θ_t-1. (13) A safety check rejects any target with z<0.05z<0.05 m to prevent ground-plane collisions. IV-G Gripper Control The gripper angle ϕφ is computed from the angular separation between vectors from the gripper base (target position) to the thumb and index finger: ϕ=arccos⁡((thumb−target)⋅(index−target)‖thumb−target‖‖index−target‖),φ= \! ( (p_thumb-p_target)·(p_index-p_target)\|p_thumb-p_target\|\;\|p_index-p_target\| ), (14) clamped to the acute range [0,π/2][0,π/2] and then to the gripper joint limits [ϕmin,ϕmax]=[0.087,1.658][ _ , _ ]=[0.087,1.658] rad after applying an offset of −0.175-0.175 rad for tighter grip calibration. A four-level fallback hierarchy ensures robust gripper control: 1. Use THUMB_TIP and INDEX_FINGER_TIP (primary). 2. Use THUMB_IP and INDEX_FINGER_DIP (knuckle fallback). 3. Use the last valid gripper angle (temporal persistence). 4. Use the mid-open default (ϕmin+ϕmax)/2( _ + _ )/2. IV-H Simulation Preview Before deploying on the physical robot, all IK-generated trajectories are previewed in a PyBullet simulation environment. The simulation loads the SO-ARM101 URDF with 7 joints per arm (1 fixed base + 5 revolute arm + 1 revolute gripper); only the right arm is used in our experiments. The simulation runs at 240 Hz with gravity g=9.81g=9.81 m/s2. Joint commands are executed via position control with uniform force limits of 6.0 N, position gains of 0.2, and velocity gains of 1.0. Fig. 3 shows the simulation preview interface. The top-left panel displays the RGB frame from the egocentric camera with the operator’s hand visible; the top-right panel shows the corresponding depth map colour-coded by distance; and the bottom panel renders the robot in PyBullet tracking the hand-derived IK targets. This three-panel view allows the operator to verify that the computed joint trajectories faithfully reproduce the intended hand motion before committing to physical execution. Beyond trajectory verification, the simulation also exports demonstration data (.npy action files) that can be used to train imitation-learning policies. Figure 3: PyBullet simulation preview. Top-left: RGB frame from the egocentric camera showing the operator’s hand. Top-right: depth colour map. Bottom: robot arm in PyBullet with debug joint labels and IK target markers (green/red spheres), tracking the hand-derived trajectory. IV-I Physical Deployment Validated actions are deployed on the physical SO-ARM101 via the LeRobot [11] framework. Joint angles from the IK solver (in radians) are normalised to [0,1][0,1] using the joint range limits (Table I) and converted to motor commands: amotor=(anorm−0.5)×200arm joints,anorm×100gripper joint.a_motor= cases(a_norm-0.5)× 200&arm joints,\\ a_norm× 100&gripper joint. cases (15) Servo PID coefficients are configured for smooth motion (Table I), and acceleration/velocity limits prevent abrupt movements. Three gripper modes are supported: Normal (raw angle pass-through), Binary (threshold at 60∘60 ; outputs fully open or closed), and Offset (adds a configurable offset for tighter grasps). Fig. 4 shows the hand-shadowing result: the operator (left) grasps an object while wearing the camera glasses, and the robot (right) mirrors the grasp posture via the IK pipeline after offline processing. Figure 4: Hand shadowing. Left: the operator wearing the RealSense glasses performs a grasp. Right: the SO-ARM101 robot mirrors the hand pose through the IK pipeline. V Experimental Setup V-A Hardware and Software • Camera: Intel RealSense D400, 640×480 RGB+D at 30 FPS. • Robot: Single SO-ARM101 follower arm (STS3215 servos, 30 kg⋅·cm torque). • Glasses: Custom 3D-printed PLA mount (3 parts: frame, 2× branch, mount bracket). • Compute: Apple M-series or x86 laptop (MediaPipe runs on CPU; PyBullet IK requires no GPU). • Software: Python 3.10, PyBullet ≥ 3.2.5, MediaPipe ≥ 0.10, LeRobot ≥ 0.4.1, OpenCV ≥ 4.8. Table I lists the joint angle ranges that define the normalisation bounds for physical deployment and the IK solver’s feasible set. Table I gives the servo PID and motion-limit parameters. TABLE I: SO-ARM101 Joint Angle Ranges (Radians) Joint Min Max Range (∘) Shoulder pan −1.920-1.920 +1.920+1.920 220.0220.0 Shoulder lift −1.745-1.745 +1.745+1.745 200.0200.0 Elbow flex −1.745-1.745 +1.571+1.571 190.0190.0 Wrist flex −1.658-1.658 +1.658+1.658 190.0190.0 Wrist roll −2.793-2.793 +2.793+2.793 320.0320.0 Gripper −0.175-0.175 +1.745+1.745 110.0110.0 TABLE I: Physical Robot Servo Parameters Parameter Value P coefficient 12 I coefficient 0 D coefficient 24 Arm acceleration limit 50 / 254 Gripper acceleration limit 100 / 254 Arm velocity limit 1500 RPM Gripper velocity limit 3000 RPM V-B Task and Protocol The benchmark task is: “Grab the purple cube and drop it in the box.” The cube is a soft foam block (5×5×55× 5× 5 cm) placed on a 3×33× 3 tile grid (tiles numbered #1–#9), with a 25×2525× 25 cm cardboard box positioned to the left of the grid. Fig. 5 shows the robot executing this task; the tile grid is visible as orange tape markers on the bench surface. For the IK pipeline, the operator wears the glasses and performs the task naturally. The system records 30 FPS RGB-D to a .bag file, processes it offline to extract joint-angle actions, previews the trajectory in PyBullet (Section IV-H), and then deploys on the physical robot. The reported benchmark uses tiles #1–#5 with 10 grasps per tile, yielding 10×5=5010× 5=50 episodes per approach. Pilot trials on tiles #6–#9 revealed two issues: • IK pipeline: tiles #6–#9 (closer to the robot base) require the operator to reach downward and behind, changing the hand orientation relative to the egocentric camera such that the thumb and index finger become partially self-occluded, preventing reliable gripper-angle computation. • VLA policies: the robot’s own gripper could occlude the cube in the policy input view at certain grid positions (discussed further in Section VI-B). We therefore excluded tiles #6–#9 from the final evaluation. Figure 5: Pick-and-place benchmark task. The SO-ARM101 robot grasps the purple cube and places it in the box. The tile grid (#1–#9) is marked with orange tape; the box sits to the left. The RealSense camera is visible on the stand above. V-C VLA Policy Training For the imitation-learning comparison, 50 demonstration episodes (×40 s each, tiles #1–#5) are collected via leader–follower teleoperation using a second SO-ARM101 as the leader arm (standard LeRobot teleoperation setup) while the same external RealSense camera records RGB and depth at 640×480, 30 FPS. The demonstration data includes synchronised RGB, depth, and joint-angle recordings, uploaded to the Hugging Face Hub in LeRobot dataset format. Four policies are fine-tuned on Google Colab (T4 GPU, 16 GB): • ACT [4]: batch size 4–6, 50 k steps. Learns a generative model over action chunks conditioned on image and proprioceptive observations. • SmolVLA [5]: batch size 64, 20 k steps. A 450M-parameter vision-language-action model. • π0.5 _0.5 [6]: batch size 32, 3 k steps, bfloat16. Fine-tuned from a pretrained VLM backbone. • GR00T N1.5: batch size 32, 3 k steps. A cross-embodiment foundation model. VI Results and Discussion VI-A Pipeline Latency Fig. 6 shows the per-stage latency breakdown. The three dominant stages are MediaPipe hand detection (23 ms), RGB/depth frame overlay and visualisation (110 ms), and PyBullet inverse kinematics solving (80 ms). The total per-frame processing time is 23+110+80=21323+110+80=213 ms, yielding an effective throughput of approximately ∼ 5 FPS. The pipeline therefore does not operate in real time at the camera’s native 30 FPS: frames are recorded to a .bag file at 30 FPS, then processed offline at ∼ 5 FPS to produce joint-angle trajectories, which are subsequently replayed on the robot at the target frame rate. Figure 6: Per-stage latency breakdown. The RGB/depth frame overlay (110 ms) and PyBullet IK solver (80 ms) dominate, yielding a total of 213 ms per frame (∼ 5 FPS). Processing is performed offline on recorded .bag files. VI-B Pick-and-Place Success Rates Table IV reports the pick-and-place success rates on tiles #1–#5 (50 episodes per approach, 10 grasps per tile). The IK retargeting pipeline achieves a 90% success rate (45/50) with zero training, since it analytically maps every operator hand motion to the robot. Its failures stem exclusively from thumb/index landmark detection issues at certain hand orientations, which prevent the gripper angle from being computed (the four-level fallback hierarchy recovers in most but not all cases). Fig. 7 shows the per-tile breakdown for the IK pipeline. Tiles #1 and #2 (farthest from the robot, most natural hand orientation) achieve 10/10 and 10/10 respectively. Success degrades toward tile #5 (closest to the robot base, requiring downward hand orientation) at 7/10, consistent with the hand self-occlusion failure mode identified in Section V-B. Figure 7: IK retargeting per-tile success (out of 10 grasps each). Tiles #1–#2 (farthest from robot, natural hand pose) achieve perfect scores. Performance degrades toward tile #5 (closest to robot base) where the operator’s hand orientation causes partial thumb/index self-occlusion from the egocentric camera. Total: 45/50 = 90%. The four VLA policies are trained on 50 leader–follower teleoperation episodes and evaluated on the same tile set. ACT achieves a 92% success rate—slightly exceeding the IK pipeline. One possible explanation is that leader–follower demonstrations provide physically successful trajectories, whereas the IK pipeline must reconstruct the motion from vision at every frame and can fail when hand landmarks are poorly detected. ACT may also benefit from the larger training budget (50 k steps) on this structured task. SmolVLA reaches 50%, competitive given its language-conditioning capability but limited by the small number of training episodes. π0.5 _0.5 achieves 40%, limited by its short fine-tuning (3 k steps) from the generalist pretrained checkpoint. GR00T N1.5 reaches 35%, consistent with the challenge of adapting a cross-embodiment foundation model to a specific low-cost arm. A likely failure mode for SmolVLA, π0.5 _0.5, and GR00T N1.5 is self-occlusion: during the approach phase, the robot’s gripper can partially hide the cube in the camera view, reducing visibility of the grasp target at a critical moment. ACT appears more robust to this issue, possibly because action chunking can carry the policy through short periods of occlusion, although this may also make the policy more task-specific. TABLE IV: Pick-and-Place Success Rates (tiles #1–#5, 10 grasps per tile = 50 episodes). The IK pipeline requires no training; VLA policies are trained on leader–follower teleoperation demonstrations. Approach Training Inference Success Steps Hz Rate IK Retargeting (ours) — ∼ 5 (offline) 90% ACT [4] 50 k ∼ 10 92% SmolVLA [5] 20 k ∼ 10 50% π0.5 _0.5 [6] 3 k ∼ 10 40% GR00T N1.5 3 k ∼ 10 35% VI-C Approach Comparison Table V compares the IK pipeline with the learned policy approaches across key dimensions. TABLE V: Comparison of Teleoperation Approaches Property IK (ours) Learned Policies Training data None 50 episodes (33 min) Training time 0 2–8 h on T4 GPU Inference rate ∼ 5 Hz (offline) ∼ 10 Hz Generalisation Task-agnostic Task-specific (ACT) or language-conditioned Failure mode Hand occlusion Gripper self-occlusion VI-D In-the-Wild Evaluation To assess the generality of the IK retargeting approach beyond the structured lab bench, we deployed the system in two unstructured real-world environments: a grocery store and a pharmacy. The robot and supporting compute setup were placed on a standard shopping basket, which provides a repeatable height across trials and positions the robot arm within ∼ 30 cm of shelf objects—a necessary constraint since the human arm (∼ 60 cm reach) is significantly longer than the SO-ARM101 (∼ 30 cm reach). Fig. 8 shows the in-the-wild setup: the operator, still wearing the glasses-mounted camera, reaches for items on the store shelf (left) while the robot mounted in the shopping basket reproduces the motion via IK retargeting after offline processing (right). The task involves grasping various store items (cans, bottles, boxes) from the shelf and placing them in the basket. Over 75 grasp attempts across both locations, only 7 succeeded, yielding a success rate of 9.3%. The primary failure mode is hand occlusion by surrounding objects: in a cluttered shelf environment, the operator’s hand is frequently occluded by adjacent products, shelf edges, and price tags from the egocentric camera’s perspective. This causes MediaPipe to lose track of the thumb and index finger landmarks, preventing both the gripper angle computation and the IK target position calculation. Despite the low overall success rate, the 7 successful grasps—shown in the mosaic in Fig. 9—indicate that the pipeline can transfer to unstructured environments when hand visibility is preserved, suggesting that occlusion handling is a key area for improvement. Figure 8: In-the-wild deployment in a grocery store. The robot and supporting compute setup are placed on a shopping basket for repeatable height. Left: the operator reaches for shelf items while wearing the glasses-mounted camera. Right: the robot mirrors the motion; the PyBullet preview is visible on the laptop screen. Figure 9: Mosaic of successful in-the-wild grasps (7 out of 75 attempts). Each pair shows the human hand capture (left) and the corresponding robot IK retargeting (right) for various store items across grocery and pharmacy environments. Despite an overall 9.3% success rate, successful transfers demonstrate the pipeline’s potential in unstructured scenes. VI-E Failure Modes We identify the following primary failure modes, separated by context. VI-E1 Structured Lab Environment 1. Thumb/index detection failure: At certain hand orientations (particularly toward tiles #6–#9), the thumb and index finger landmarks are poorly detected, preventing gripper angle computation. This is the dominant IK failure mode (accounting for all 10% of lab failures). 2. Gripper self-occlusion (VLAs): The robot’s gripper can hide the cube in the camera view during the approach phase. This effect appears most pronounced for SmolVLA, π0.5 _0.5, and GR00T N1.5, where the vision model loses sight of the grasp target at a critical moment. 3. Gripper jitter: Rapid open-close oscillations occur when the thumb-index angle is near the binary threshold. The binary gripper mode with a 60∘60 threshold resolves this. VI-E2 Unstructured In-the-Wild Environments 1. Hand occlusion by scene objects: Shelf products, price tags, and adjacent items frequently occlude the operator’s hand from the egocentric camera, causing MediaPipe to lose landmark tracking. This is the dominant failure mode, accounting for the drop from 90% to 9.3% success. 2. Workspace mismatch: The human arm (∼ 60 cm) reaches further than the robot (∼ 30 cm), requiring careful positioning of the shopping basket to keep objects within the robot’s reachable volume. 3. Depth noise: Reflective packaging (foil, plastic wrap) produces unreliable depth readings, causing spurious 3D deprojection results. VII Conclusion We have presented a vision-based hand-shadowing system for offline robot trajectory generation via egocentric hand tracking and analytical inverse kinematics. The pipeline processes recorded RGB-D frames at ∼ 5 FPS (213 ms per frame) using MediaPipe Hands for landmark detection, depth-based 3D deprojection, and PyBullet for IK solving and trajectory preview, with the LeRobot framework for physical deployment on the SO-ARM101. On a structured pick-and-place benchmark (tiles #1–#5, 50 episodes), the IK retargeting pipeline achieves 90% success with zero training. Among the teleoperation-trained VLA policies, ACT reaches 92%, while SmolVLA (50%), π0.5 _0.5 (40%), and GR00T N1.5 (35%) are substantially lower. The gap is consistent with the structured nature of the benchmark and with visibility challenges caused by gripper self-occlusion in the camera view. In-the-wild evaluation in grocery store and pharmacy environments (7/75 successful grasps, 9.3%) indicates that hand occlusion by surrounding objects is the primary limitation of the current system. VII-A Future Work • Occlusion-robust hand tracking: Temporal hand tracking, learned depth completion, or multi-camera setups to maintain landmark visibility in cluttered environments. • Dual-arm extension: The current system controls a single arm; extending to simultaneous bimanual control. • IK demonstrations for VLA training: Using IK-collected trajectories as training data for VLA policies, potentially avoiding the gripper self-occlusion issue present in leader–follower teleoperation data. • Larger workspace: Adaptive camera-to-robot calibration for dynamic operator positioning. • Force feedback: Integrating gripper force sensing for haptic feedback to the operator. Acknowledgment The authors thank the UC Berkeley Department of Mechanical Engineering and the Fung Institute for Engineering Leadership for providing access to lab facilities, equipment, and student collaboration. References [1] R. A. Potamias, J. Zhang, J. Deng, and S. Zafeiriou, “WiLoR: End-to-end 3D hand localization and reconstruction in-the-wild,” arXiv preprint arXiv:2409.12259, 2024. [2] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “MediaPipe Hands: On-device real-time hand tracking,” arXiv preprint arXiv:2006.10214, 2020. [3] J. Romero, D. Tzionas, and M. J. Black, “Embodied hands: Modeling and capturing hands and bodies together,” ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 36, no. 6, p. 245:1–245:17, 2017. [4] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” arXiv preprint arXiv:2304.13705, 2023. [5] F. Cadena et al., “SmolVLA: A vision-language-action model for affordable and efficient robotics,” arXiv preprint arXiv:2506.01844, 2025. [6] K. Black et al., “π0 _0: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024. [7] X. Cheng et al., “Open-TeleVision: Teleoperation with immersive active visual feedback,” in Proc. CoRL, 2025. [8] Y. Ding et al., “Bunny-VisionPro: Real-time bimanual dexterous teleoperation for imitation learning,” arXiv preprint arXiv:2407.03162, 2024. [9] L. Keselman, J. I. Woodfill, A. Grunnet-Jepsen, and A. Bhowmik, “Intel RealSense stereoscopic depth cameras,” in Proc. CCD Workshop, CVPR, 2017. [10] E. Coumans and Y. Bai, “PyBullet, a Python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021. [11] Hugging Face, “LeRobot: State-of-the-art machine learning for real-world robotics,” https://github.com/huggingface/lerobot, 2024. [12] The Robot Studio, “SO-ARM100: Open-source 6-DOF robotic arm,” https://github.com/TheRobotStudio/SO-ARM100, 2024.

Instant research discovery