Paper deep dive
EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras
Kush Hari, Justin Kerr, Nidhya Shivakumar, Samarth Mahapatra, Carmelo Sferrazza, Jiahui Lei, Jitendra Malik, C. Karen Liu, Ken Goldberg, Angjoo Kanazawa
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
Tags
Links
- Source: https://arxiv.org/abs/2610.03710v1
- Canonical: https://arxiv.org/abs/2610.03710v1
Trouble viewing inline? Open PDF directly →
Full Text
77,044 characters extracted from source content.
Expand or collapse full text
EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras Kush Hari ∗1 Justin Kerr ∗1,2 Nidhya Shivakumar 1 Samarth Mahapatra 1 Carmelo Sferrazza 2 Jiahui Lei 1 Jitendra Malik 1,2 C. Karen Liu 2,3 Ken Goldberg †1 Angjoo Kanazawa †1,2 https://eyerobot2.github.io 1 UC Berkeley 2 Amazon FAR 3 Stanford University ∗ Equal contribution, random order † Equal advising Figure 1: EyeRobot 2.0 enables fine-grained bimanual manipulation like inserting a straw into a boba cup using only a fixed stereo camera. It coordinates a sequence of gaze targets, for example here first at the boba cup, then the straw, then the insertion point. Gaze targets are output from a semantic object selector, which conditions a low-level gaze policy to servo precisely. Its design draws inspiration from human foveated vision by leveraging multi-resolution image crops to provide high detail where the policy gazes. Abstract: Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing com- putation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierar- chically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper pol- icy which allows it to discover fixation sequences that can resemble a human’s fixation sequence while performing the task (Fig. 1). EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper proprioception and actions into a rotating fixation-relative SE(3) frame, which compacts the size of the action dis- tribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials compar- ing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap without any wrist cameras, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects arXiv:2610.03710v1 [cs.RO] 2 Oct 2026 occlude the wrist cameras (48% vs. 22%). These results suggest active gaze as a single-camera alternative to wrist cameras for fine-grained manipulation. 1 Introduction Consider the task of picking up a cup with a lid, grasping a straw, and inserting the straw into a small opening in the lid (Fig. 1). As humans carry out this task, our eyes flit around the scene, filtering information with coordinated look-ahead movements that anticipate hand motion [1]. This task- relevant visual sampling, called fixation, directs high-resolution foveal vision to relevant parts of the visual field. Fixation also provides 3D anchor points for manipulation, simplifying the challenge of spatially reasoning about the scene [2] and implicitly breaking larger tasks into concrete sub-goals as gaze switches between targets. We hypothesize that this behavior could also benefit robot manipulation. Fixating on a dynamic task- relevant 3D point acts as a form of physical attention which centers input observations, compacts the data distribution, filters irrelevant visual regions, and allocates additional resolution where it matters for fine-grained tasks without the cost of processing the full panorama at high resolution. Some of these benefits can be offered by wrist-mounted cameras, which are used by nearly all existing bimanual manipulation systems [3, 4, 5, 6]. However, colocating vision with the gripper ties precise visual input to the manipulator, which is limiting in tool use, whole-body manipulation, or when grasped objects occlude the camera. In addition, these systems constrain manipulator design because of their direct mounting to the gripper, which adds complexity, often occludes the ego camera, introduces motion blur, and prevents sleeker gripper designs. A purely ego-centric behavior cloning approach has the potential to ease constraints on end effector design, make better use of ego-centric human data, and improve tasks which benefit from vision-manipulator decoupling like fast motion, reaching into tight spaces, or searching large spaces [7]. We present EyeRobot 2.0, a robot learning framework which introduces Active Visual Fixation (AVF). Using input only from one fixed stereo camera, EyeRobot 2.0 can carry out precise manip- ulation tasks like inserting a straw, capping a marker, zipping a bag, or extracting a wrench from a toolkit, and multi-stage tasks like opening a toaster oven, pulling the tray out, extracting a pan, pushing the tray back, and closing the door again. EyeRobot 2.0 couples a fixation policy with a fixation-centric gripper policy. The fixation policy has two modules: a low-level gaze policy that coordinates two eye viewpoints to rapidly find and track 3D objects in the scene, and a high-level target selector to choose what object to look at and when to look at it during the task. Visual inputs are represented as foveated stereo multi- crops centered at the fixation point, which allocate more resolution towards the direction of interest. This physical attention not only increases visual acuity, but also reduces computation allocated to distractors. In addition, EyeRobot 2.0 trains a fixation-centric gripper policy via behavior cloning, by representing SE(3) proprioception and action chunks canonicalized into the rotated fixation frame. This effectively constrains the input and output space for the gripper model with respect to task- relevant local regions. The gripper policy additionally receives the distance of the 3D fixation point as an observation, which acts as an anchor point for the policy to understand depth. Both the gaze policy and target selector are trained with reinforcement learning (RL), with gaze emerging from a BC-RL loop as in Kerr et al. [7]. We evaluate EyeRobot 2.0 on 7 physical tasks and 6 simulated MuJoCo tasks, collecting data and training single-task policies from scratch. Across over 1000 physical trials, it outperforms static stereo baselines by 40%, and by 20% on simulated pick-place tasks, with a larger performance improvement on fine-precision tasks. Compared to the widely adopted SOTA ego + wrist setup, EyeRobot 2.0 without wrist cameras matches task performance in physical trials, and on tasks with heavy occlusion of wrist cameras maintains strong performance while wrist baselines regress to static stereo performance. In ablations we find that fixation-centric actions account for a significant portion of EyeRobot 2.0’s improvement, causing a 20% performance boost across sim tasks, and foveation contributes strongly to real-world success on fine-grained tasks, boosting by 20% overall. 2 Figure 2: Gaze target outputs over an autonomous physical trial which picks up a marker cap and body, caps it, repositions a bag, inserts the marker, and zips the bag. The target selector outputs goals that are relevant to the current stage of the task: for example the policy looks at the marker when capping or the bag when repositioning. Wrist cameras are present but unused. 2 Related Work 2.1 Learning manipulation from human demonstrations Behavior Cloning (BC) allows robots to learn complex behaviors from teleoperated demonstra- tions without requiring handcrafted motion primitives and reward functions [8, 9, 10, 11]. BC has been extended to work in complex settings including mobile manipulation [12, 6] and bimanual manipulation [3, 4], and several recent efforts have been directed towards scaling teleoperation datasets [13, 14, 5, 15, 16]. Another enticing direction for scaling is learning from ego-centric human data, either by instrumenting humans or scraping internet videos [17, 18, 19, 20, 21]. How- ever, learning from human videos presents a significant domain gap between ego-centric sensors in first-person datasets and the wrist cameras that our robot policies so frequently rely on for fine- grained performance. EyeRobot 2.0 aims to provide a route towards lessening this gap by lever- aging human-like fixation for precise bimanual manipulation. We are also inspired by task condi- tioning approaches in BC which condition a low-level policy on the output of a higher-level task sequencer [22, 23, 24, 25]. Similarly, EyeRobot 2.0 learns a target selector to control gaze, but does not require human gaze supervision and instead trains gaze with RL via a BC accuracy reward. 2.2 Active vision for manipulation Active vision deliberately moves sensors to improve observability, filter information, and reduce occlusions for better environment understanding [26, 27, 28, 29, 30, 31, 32, 33]. Previous systems have used active vision to help with tasks such as visual search [34, 35, 36, 37, 38], object detection and tracking [39, 40, 41, 42], and 3D reconstruction [43, 44, 45, 46]. Similarly, active vision has proven beneficial for robot manipulation with a “robot neck” moving the camera based on VR head- set movement during task teleoperation to reveal better viewpoints [47, 48, 49, 50, 51, 52]. Active visual search can also benefit from training on ego-centric human data [53, 54, 55]. EyeRobot 2.0 in contrast does not attempt to address viewpoint selection which has been explored in prior works, but rather investigates how coordinated fine-grained eye movements can boost performance even with a well-positioned viewpoint. Related to learning more precise gaze, some papers collect human eye tracking data during teleop- eration to predict a gaze point as an intermediate output to condition a gripper policy with [56, 57] or crop a high-resolution finger-tip image [58]. Others [59] learn gaze in simulation through on- policy task-based rewards; however, these methods struggle with sim-to-real transfer. EyeRobot [7] overcomes the need for gaze demonstrations by training a gaze policy on top of real-world behavior cloning data from 360 ◦ video by co-training the eye and arm policies with an action feedback loop for reward. Building on this, EyeRobot 2.0 uses a similar co-training loop but tackles much more fine-grained manipulation tasks by utilizing active stereo fixation, a fixation-centric action represen- tation, and a hierarchical target selector to precisely coordinate gaze throughout execution. 3 Figure 3: Learning gaze control. EyeRobot 2.0 trains a goal-conditioned gaze policy to continu- ously servo two eyes’ gaze in 3D. We train this behavior via RL, with a spatial reward derived from detecting the object and estimating its 3D centroid with stereo depth. 3 Manipulation with Active Visual Fixation Problem Statement: We study visuomotor policy learning from human teleoperation demonstra- tions for bimanual manipulation with parallel-jaw grippers. Given a set of demonstrations specifying a task, our goal is to train task-specific policies that can reproduce the demonstrated behavior with bounded variations in pose. In particular, EyeRobot 2.0 considers an ego-centric setting in which observations are provided solely through an approximately head-mounted stereo camera. The policy is evaluated using task-specific success criteria. Overview: AVF trains separate gripper and fixation policies that work together to achieve a task. The two eyes’ viewpoints actively pivot to converge on a shared 3D fixation point, whose location is updated at each timestep by the fixation policy. The resulting fixation-centered observations are fed into the downstream gripper policy, which produces action chunks for robot execution. These SE(3) action chunks are canonicalized into a dynamically shifting fixation frame pointing towards the current fixation point (Fig. 4). Each policy uses a foveated transformer-decoder architecture similar to [7], which we fully describe in the Appendix. 3.1 Learning Gaze Patterns with RL We train AVF in two phases. First, we train a goal-conditioned fixation policy capable of fixating on any task-relevant objects. Next, we train a hierarchical target selection policy that sequences fix- ations to optimize task performance. In the first phase, the fixation policy is trained with RL using a geometric reward from detected object positions, enabling accurate 3D tracking of prompted ob- jects. In the second phase, the target selector is trained with RL in a BC-RL loop as in EyeRobot [7]: a gripper BC policy is co-trained on the resulting fixated views, and the selector is rewarded for fixa- tions that maximize the BC policy’s action prediction accuracy. This hierarchical formulation allows task-dependent gaze behavior to naturally arise without the need for manual human specification. To learn gaze, we need a way to counterfactually render it on top of existing robot demonstrations that were collected before any gaze policy exists. We therefore need to render what the robot would have seen had it looked somewhere. EyeRobot [7] addressed this by collecting demonstrations with a 360 ◦ camera and rendering novel views from it. We transfer this idea to stereo, using a calibrated stereo camera to synthesize fixation: rotating a camera about its optical center only requires warping its image, without any 3D reconstruction. This lets AVF use the same stereo camera for data collec- tion, training, and deployment. We simulate fixation by rendering viewpoints from a stereo camera to replicate the appearance of two eyes converging their gaze on a common 3D point in the scene. Goal-Conditioned Stereo Fixation First, we train a standalone goal-conditioned fixation policy. This policy takes as input a language prompt specifying the goal, the current fixated observations, and the current fixation point in polar coordinates. Though in principle this language prompt could be anything, in practice we limit it to a task-specific annotated set of target objects which the robot interacts with in the scene. The gaze policy outputs local fixation deltas for azimuth and elevation ∆θ, ∆φ, and fixation distance ∆d represented as a discretized distribution. 4 Figure 4: Sequencing fixation for manipulation. (Left) EyeRobot 2.0 trains a lightweight per-task target selector to switch between objects in the scene. This target selector is trained with RL to maximize the prediction accuracy of a gripper BC policy, which is co-trained from scratch at the same time on viewpoints generated from the fixation rollout. Action chunks are supervised with teleop demos, and negative prediction error is supplied as the reward for the selector to optimize. (Right) Action chunks are represented as SE(3) chunks in the eye frame, aligned with the current fixation point. We train this policy with PPO [60] on rollouts from randomly selected stereo still-frames from our teleop demonstration set. To compute a goal-conditioned reward r for each rollout, we first detect candidate objects by querying SAM3 [61] with the task’s language prompts, then randomly select one detected prompt. To localize the object in 3D, we extract depth from the calibrated stereo pair with FoundationStereo [62] and set the ground truth position to be the 3D centroid of depth intersected with the object mask, p gt . At each step of RL, the policy is conditioned on the current prompt and rewarded based on the current fixation point p cur with r =−||p cur − p gt || 2 . The result is a lightweight policy that can rapidly fixate on all task-relevant objects at 30 Hz. Learning to Sequence Fixations via a BC-RL Loop While the first phase enables low-level gaze behavior capable of fixating on any object, we now address the high-level decision of where and when to focus. As in EyeRobot [7], to circumvent the burden of hand-annotated task stages, we learn this decision process via RL by jointly training gaze alongside the BC policy itself. This naturally encourages gaze sequences that optimize task performance. To do this, we freeze the goal- conditioned fixation policy, and train a lightweight target selector with RL, which is conditioned on robot proprioception and outputs a categorical distribution over per-task prompts (Fig. 4, top left). The resulting sampled prompt is used to condition the fixation policy, whose behavior is used during PPO rollout as observations to compute gripper action chunk predictions, which are then used to compute reward for the target selector to optimize (Fig. 4, middle). During training, the target selector emits a new target once every 15 frames to allow gaze to settle before sampling a new target. During train rollouts we uniformly sample a start frame from the demonstration, with special 10% weighting given to the first frame in the demo which often contains the most critical decision for gaze (what to reach for first). The first 30 frames of the agent’s observations operate on a paused frame to allow gaze to settle before being forced to act. We adopt a similar per-step reward as [7], which extracts a task performance proxy from the BC policy’s prediction error. Given predicted and ground truth bimanual action chunks transformed into the same frame A pred , A gt , we first identify an “active arm” whose end effector moves a greater distance in A gt , then compute the arc- length Fr ́ echet distance between A gt and A pred for this arm. The negative of this distance is used as reward to optimize the target selector, encouraging fixation which aids BC prediction accuracy. Limiting reward computation to the active arm avoids sensitivity to spurious correlations in bimanual demonstrations, and Fr ́ echet distance provides a more stable velocity-invariant metric than L2 error. 3.2 Fixation-Centric Action Chunking Knowing where to focus not only reduces perceptual complexity but also improves action predic- tion. Specifically, we represent the robot’s proprioception and action chunks relative to the fixation 5 Figure 5: We collect teleop demonstrations for 7 real and 6 simulated tasks. Selected images show the raw perspective of the left stereo camera. Real-world tasks contain precise manipulation re- quirements, multi-stage behavior, articulated object interaction, and manipulation across the entire workspace. See the Appendix for a full breakdown of task stages and success criteria. frame. Such a representation standardizes the input about the fixation direction, making the training distribution more compact. Model input proprioception and actions are represented as SE(3) end effector poses transformed into the fixation frame, defined as the cyclopean (virtual center-eye) ro- tation towards the fixation point in 3D. This canonicalized action representation moves dynamically with fixation, which results in a significant performance improvement over a fixed world frame. We jointly train the BC policy alongside gaze post-training on perspectives sampled from the fixations induced by RL policy rollout. We maintain a ring buffer of the past 100 rollouts seen, and sample BC train batches independently from this buffer to maintain diversity across demonstrations. The BC architecture is a transformer decoder resembling that of [7]; see the Appendix for architecture and hyperparameter details. 4 Experiments EyeRobot 2.0 is implemented using a bimanual, 14-DoF I2RT YAM manipulator with parallel-jaw grippers. We mount a static stereo camera between the arms 35 cm above the base joint, angled 45 ◦ downwards for full observability of the table. Each eye captures a 120 ◦ diagonal FOV at 1600×1200 resolution. We use a custom leader arm modified from GELLO [63] and TRLC [64] to teleop the robot. Simulation Environment We build a simulation environment in MuJoCo [65] to conduct repeat- able, large-volume experiments. Note that our objective is not sim-to-real transfer, nor do we pro- pose to use physics simulation to learn eye gaze, but rather we strive for a digital twin with which to conduct careful method comparisons using the same training pipeline as in real. We replicate the di- mensions of the YAM manipulators, and use the public calibrated PD and mass-inertial constants in MJ-Playground [66]. We implement non-pinhole camera rendering in MuJoCo by rendering pinhole images and warping them to exactly match the camera characteristics of each camera. We collect teleoperation with GELLOs through a custom VR interface which streams visuals to the operator in a Quest 3 headset, similar to prior VR teleop systems. All simulation code will be made public. Tasks and Data In total we collected teleoperation data for 7 physical tasks and 6 simulated tasks as shown in Figure 5. We focus on 5 multi-stage real tasks which involve bimanual coordination (e.g. object handover, coordinated grasping), fine-grained tolerance requirements (e.g. marker capping, zipping, boba straw insertion), and long-horizon behavior (e.g. opening a toaster oven, taking out a pan, pushing the shelf back in, closing the door). Due to the challenges of simulating contact- rich physical interactions in real-time, our simulated tasks aim to evaluate a range of basic pick and place behavior. We collect between 10 and 53 minutes of data per task. Full task definitions, stage visualizations, train distribution visualizations, and dataset size details are provided in the Appendix. 5 Results We evaluate how well EyeRobot 2.0 performs compared to SOTA baselines. To construct carefully controlled comparisons, we choose to collect all data ourselves and train all policies from scratch on the exact same demonstrations. This avoids obfuscation from unknown pretraining distributions of 6 Sim Avg.Real Avg.bobamarkerpotteatoastertapewrenchtapepottigertrayplatespoon 0% 25% 50% 75% 100% Success Rate 54 27 8 4 2828 36 53 32 55 69 87 44 41 26 75 52 12 56 32 80 52 66 68 81 90 96 85 74 27 74 67 44 68 52 80 52 94 76 79 91 98 78 70 30 AveragesReal (640 trials)Sim (1800 trials) No Gaze (Stereo Only)Ego + Wrist (3 Cameras)EyeRobot 2.0 (Stereo Only) Figure 6: End-to-end success on real and sim tasks. Each real task is evaluated on 25 trials for each policy, all sim tasks over 100. Full per-stage conditional success is reported in the Appendix. off-the-shelf models, reduces compute demand, and increases the tractable volume of experiments by an order of magnitude. In real, we train every policy with 3 seeds and randomly select one for evaluation. For each task, we evaluate all policies on 25 scene configurations selected by placing objects within the training data region, but not exactly matching train datapoints. During evaluation, we overlay the live camera stream on the test positions and manually align objects so all policies share the exact same test positions. See the Appendix for a visualization of these test distributions. Across all real-world physical trials of baselines and ablations, we evaluate over 1000 policy rollouts. In sim, every baseline and EyeRobot 2.0 policy is trained across 3 seeds, of which we report the maximum performance to prune failed seeds. See Appendix for full statistics. All sim evaluations are run on the same 100 random configurations, all unseen during training but drawn from the same distribution. Each policy is trained from scratch on single-task data, and executed using the same temporal ensembling weights and inference code. The gaze policy is given 1 second to allow gaze to settle before acting. Per-stage success is programmatically judged using manually specified MuJoCo physics requirements, and each task has a failure timeout based on the average length of demonstrations. Baselines We collect all our data with stereo ego and wrist cameras, meaning EyeRobot 2.0 and baselines share the same training data. No Gaze takes in the raw stereo input stream, which has the exact same observability but without fixation, and Ego + Wrist mimics the prevailing bimanual setup using two wrist cameras and the left image from the stereo pair. All baselines are trained with ACT [3] and share the same architecture and training hyperparameters as EyeRobot 2.0. We augment baseline input images with 95% center-crop, and 10% translation and rotation randomization to make them less sensitive to placement error, as in [5]. To select the best action space, we test baselines with end-effector (E) relative SE(3), world-absolute SE(3), joint chunk-relative, and joint chunk-absolute. Since E-relative SE(3) performs the best in sim (see Appendix), we use that for real evaluations. Baselines are trained with a chunk size of 1 second as suggested in ACT [3] and EyeRobot 2.0 is trained with 1.5 seconds. We found a slight performance gain from 1.5s chunks with fixation but no such gain for baselines when swept in simulated tasks. How does EyeRobot 2.0 compare to gaze-free stereo? This baseline corresponds to training stan- dard policies with static ego-only stereo inputs without wrist cameras. EyeRobot 2.0 outperforms static stereo policies significantly across all 13 tasks, on average by +20% in sim and +40% in real (Fig. 6). Ego-only policies have a strong tendency to overfit to the training positions, an effect which can particularly be seen in the tape handover task, as data was collected in a regular 8× 8 grid, or the tea task which requires servoing around the entire workspace. In addition, ego policies lack the visual acuity to competently accomplish precise tasks like marker insertion (4%) and straw insertion (8%). Despite having access to the exact same stereo stream, by foveating on relevant objects EyeRobot 2.0 achieves 68% and 44% end-to-end success on these two tasks. The failure modes of static stereo policies tend to cluster around task stages that require precision (see Sankey diagrams in Appendix), as well as significantly more overfitting behavior in tasks with wide spatial distribution (tape handover). 7 wrenchteatape Policy1st graspsuccess1st graspsuccess1st graspsuccess Ego-only3/250/253/250/256/250/25 Ego + Wrist10/254/2511/254/2514/250/25 EyeRobot 2.020/255/2513/254/2523/258/25 Table 1: Real-world performance with distractors. EyeRobot 2.0 is better able to ignore distractor objects because of its physical attention directed towards relevant objects. PolicybobamarkerpotteatoastertapewrenchMean EyeRobot 2.0−foveation12/255/2510/2512/259/2547/6415/2546.5% EyeRobot 2.0−stereo10/258/2510/2515/256/2551/6416/2548.5% EyeRobot 2.011/2517/2513/2520/2513/2560/6419/2566.5% Table 2: Real-world ablations show that foveation and stereo inputs are both important. boba pot 0 20 40 60 8 28 12 32 44 52 Success (%) Ego-onlyEgo+WristEyeRobot 2.0 Figure 7: Wrist occlusion.When grasped objects block the wrist cameras, ego+wrist falls to near the stereo base- line while EyeRobot 2.0 maintains per- formance. How does EyeRobot 2.0 compare against SOTA ego + wrist policies? On precise physical tasks where the wrist camera maintains full visibility (marker, tea, toaster, wrench), both perform similarly (Fig. 6). In tasks where grasped objects occlude the wrist cameras (boba, pot), ego+wrist performance collapses to near the gaze-free stereo baseline, while EyeRobot 2.0 maintains strong per- formance, overall a 2× improvement (Fig. 7). In the tape handover task, wrist camera policies tend to over- fit to the grid training distribution similarly to ego-only policies because the wrist camera on the gripper holding the yellow tape is occluded until just before placement, by which point the tape is already in an unrecoverable position. Does fixation improve robustness to visual distractors? We evaluate all policies in real on tea, wrench, and tape with colorful distractor objects randomly scattered on the table and report their end-to-end success along with first stage success in Table 1. All comparisons use the exact same object positions to ensure comparability. The wrist camera policy often reaches for the largest or most distinct object in its view. Once it begins reaching, the policy cannot recover since the wrist camera loses sight of the target. In contrast, EyeRobot 2.0 reaches for the correct object when its gaze is correctly directed at it, and mostly misses grasps when it is looking in the wrong place. This shows most clearly in first-grasp success, which is significantly higher with active gaze. Decoupling vision and action could improve robustness in cluttered scenes by literally ignoring irrelevant information. Gaze analysis See Fig. 2 for a visualization of target selector probabilities over the full course of the marker task. EyeRobot 2.0 first fixates on the marker cap, then switches to the marker just before picking it up and keeps its gaze there throughout capping. Overall, it learns intuitive gaze switching, for example looking at an object until just after it has been grasped. Interestingly, it sometimes uses static objects as a sort of anchor during manipulation, for example during spans of the marker task it gazes at the pencil pouch which during data collection was always in a roughly straight-ahead position. 5.1 Ablations No foveated image architecture We ablate the multi-scale image pyramid and train both gaze and gripper policies on one peripheral view for each eye (Tab. 2). Overall this harms performance by 20%, with the largest gap on precise physical tasks which require detailed image understanding. Interestingly, in simulation this ablation performs the same as EyeRobot 2.0. This may be because 8 robot execution in simulation has little to no variance, whereas real-world execution varies from trial to trial and so demands more closed-loop visual servoing. Fixation-centric actions We test an ablation of EyeRobot 2.0 in simulation which uses SE(3) ac- tions and proprioception in the world frame rather than a fixation-relative frame. This ablation performs on par with the gaze-free ego baseline (−21%). Including gaze while still allowing a global output space for the policy seems to still afford overfitting to specific trajectory positions in this low-data regime. Stereo vs. Mono We train an ablation which only has one input image and no input fixation depth. This ablation performs on average 18% worse than EyeRobot 2.0 across all real-world tasks, indi- cating that the model benefits from access to binocular information. 6 Limitations and Future Work EyeRobot 2.0 currently only trains task-specific gaze and gripper policies. An interesting direction for future extension is to investigate how to adapt the framework to maintain multi-task performance. The usage of a target selector means that gaze would need to be trained on a sufficiently large corpus to allow generic prompts, which may necessitate different architectures for generating prompts such as a vision-language model. Separately, in this work we do not control head or neck movement, and only propose how to coordinate fine-grained eye movements provided a good head position is already available. Finally, while object-level pretraining is sufficient for many manipulation tasks, it cannot yet train sub-object-level gaze which may provide more benefits for manipulation. 7 Conclusion This work presents EyeRobot 2.0, a robot learning framework for fine-grained bimanual manipu- lation using only a fixed stereo camera that leverages active visual fixation to direct attention to- wards task-relevant features. Our experiments suggest that active fixation more effectively leverages stereo data than non-active stereo baselines, strongly outperforming them across a range of high- precision physical tasks and matching or exceeding the performance of wrist cameras. Ablating any of its components results in a performance drop, indicating it utilizes multi-resolution image in- puts, fixation-centric actions, and stereo information to effectively facilitate fine-grained real-world manipulation. This suggests exciting potential for systems which use active visual fixation for fine- grained manipulation instead of wrist cameras. 9 Appendix A Additional Results Failure modes analysis Please see the included Sankey diagrams (Figs. 9, 10, 11) for a visualization of per-stage success breakdown for AVF and baselines across all physical trials. The primary failure cases align with the most difficult, precision-requiring stages of each task, for example inserting the boba straw or capping the marker. In comparison, the ego-only baseline struggles more with these precise stages, and also has considerably more failures in “easy” portions of each task, for example picking up objects initially. With only a fixed input view the policy struggles to achieve the precision required to succeed, and even struggles more at basic picking, for example in the tea task only grasping the small teabag in 32% of trials, while AVF always grasped the teabag. The ego+wrist baseline recovers this grasping precision, but fails severely on task stages (pot on lid, boba insertion) where wrists become occluded by the grasped object. Even in tasks with unoccluded views from the wrist camera, wrist cameras have similar conditional success rates to AVF, indicating that failures stem from a common modeling or data insufficiency which wrist cameras provide no advantage for. How well do policy rankings in sim and real correlate? To measure sim-real correlation we collect 2 similar tasks (pot and tape) in sim and in real. While the absolute success rate in sim is not predictive of real-world success, policy rankings are maintained on these tasks between all baselines, and results indicate that sim may favor wrist cameras over physical trials, perhaps because E- relative chunks are more sensitive to compounding action errors with imperfect real physics than simulated. Pot real-world success is notably lower overall because simulation has more forgiving physics for pot lid closure. What fixation patterns does AVF learn? Probabilities while inserting the marker into the bag are more ambiguous as all objects are in the same location. Quantitatively, we analyze per-stage gaze in simulation and report full task results in the Appendix. During the first task stage, 100% of policies learned to look at the first object the robot grasps. In tape handover, dishrack, and tray the gaze switches to the second object, while in the pot and spoon tasks it remains on the first object. How accurately can AVF fixate in 3D? Overall, the average fixation error after gaze pre-training is 3.5 cm computed over 800 random frames from all trajectories. Fixation quality is best seen in our included policy rollout videos, which visualize real-time fixation from the policy during execution. Despite being trained only on still frames, the policy can reliably track moving objects during robot execution. B Model Architecture and Training Details Periphery (cm)↓Foveal (cm)↓ Sim Tasks4.733.51 Real Tasks4.443.55 Table 3: 3D fixation Euclidean error during fixa- tion pretraining, averaged over the final 100 train- ing iterations of the best seed per task. Both the fixation and gripper policies use a transformer architecture which decodes actions from input image, proprioception, and goal prompt observations. The input propriocep- tion includes the current fixation point encoded as a unit vector direction and distance. Im- age tokens are extracted using a frozen DI- NOv3 [67] ViT-S/16 backbone in a foveated manner: a pyramid of centered crops is inde- pendently passed through the encoder and concatenated along the sequence dimension, allowing the model to attend to tokens from multiple resolutions as it desires. Each input token is given a RoPE [68] (x,y,t) position. For image tokens, x,y corresponds to the location of the token within the global image, and all other tokens are given a value of the image center ( h 2 , w 2 ). Each token is assigned t corresponding to the current observation timestep, except 10 0%25%50%75%100% Tape: pick Tape: handover Tape: place Plate: pick Plate: place Spoon: pick Spoon: hang Tray: pick Tray: place Tray: bottle Pot: pick Pot: place yellow tape 100% grey tape 67% yellow tape 33% grey tape 100% plate 100% dish rack 67% plate 33% spoon 100% spoon 100% tray 100% tray 67% blue cap bottle 33% blue cap bottle 100% lid 100% lid 100% Figure 8: Per-stage gaze selection in simulated evaluations. For each policy trained on sim data, we compute the mean gaze pointing error to each object in the scene, and consider the object with the lowest mean error as the “chosen” object during that stage. EyeRobot 2.0 reliably learns to pre-attend grasping, as evidenced by the fact it looks at the first object 100% of the time. In sim, it sometimes does not look at the second stage of the task, i.e., the place action, for example in the pot and spoon hanging tasks. The tiger task is omitted as it only has one stage. all 25 pick cup 25 · 100% pick straw 25 · 100% insert straw 11 · 44% place on coaster 11 · 100% × 14no insert straw all 25 grab cap 25 · 100% grab marker 25 · 100% cap marker 19 · 76% grab bag 19 · 100% place marker 17 · 89% zip bag 17 · 100% × 2no place marker × 6no cap marker all 25 grab lid 25 · 100% optimal place 10 · 40% suboptimal place3 · 12%× 12failed place all 64 pick yellow 61 · 95% handoff 61 · 100% place yellow 60 · 98% × 1no place yellow × 3no pick yellow all 25 grab tea 25 · 100% tea in cup 20 · 80% × 5no tea in cup all 25 open oven 24 · 96% pull tray 21 · 88% pick pan 20 · 95% place pan 18 · 90% push tray 16 · 89% close oven 13 · 81% × 3no close oven × 2no push tray × 2no place pan × 1no pick pan × 3no pull tray × 1no open oven all 25 grab tab 22 · 88% open box 19 · 86% grab wrench 19 · 100% × 3no open box × 3no grab tab BobaMarkerPot PlaceTapeTeaToasterWrench Figure 9: Per-stage conditional success for EyeRobot 2.0 across 25 physical trials per task. action chunks which are assigned t consistent with their future time. We use a θ value of 1, 000 for sinusoidal frequencies, lower than the default 10, 000 for RoPE to adjust for the shorter sequence length. The BC gripper policy takes in history-free single-frame observations, while the gaze policy is allowed to attend to the gaze direction of the previous 3 frames, which stabilizes gaze. The transformer uses bi-directional attention between observation and query tokens, to improve compute efficiency over full self-attention. Each input and output modality is projected to the same token stream and decoded into outputs with a 3-layer MLP. The fixation policy is an actor-critic agent which outputs a categorical distribution over actions alongside a continuous value function estimate v. The target selector is a 3-layer, 256-wide MLP with GELU [69] activations which takes in proprioceptive input and outputs a categorical distribution over target prompts along with a critic estimate. We apply 10% dropout to the hidden layers of the MLP to prevent overfitting to proprioceptive data, along with 10% proprioceptive token drop during BC batch sampling. The 11 all 25 pick cup 22 · 88% pick straw 20 · 91% insert straw 2 · 10% place on coaster 2 · 100% × 18no insert straw × 2no pick straw × 3no pick cup all 25 grab cap 17 · 68% grab marker 16 · 94% cap marker 3 · 19% grab bag 2 · 67% place marker 2 · 100% zip bag 1 · 50% × 1no zip bag × 1no grab bag × 13no cap marker × 1no grab marker × 8no grab cap all 25 grab lid 18 · 72% suboptimal place 4 · 22% optimal place3 · 17%× 11failed place × 7no grab lid all 64 pick yellow 48 · 75% handoff 42 · 88% place yellow 34 · 81% × 8no place yellow × 6no handoff × 16no pick yellow all 25 grab tea 17 · 68% tea in cup 7 · 41% × 10no tea in cup × 8no grab tea all 25 open oven 25 · 100% pull tray 17 · 68% pick pan 13 · 76% place pan 13 · 100% push tray 9 · 69% close oven 9 · 100% × 4no push tray × 4no pick pan × 8no pull tray all 25 grab tab 15 · 60% open box 9 · 60% grab wrench 8 · 89% × 1no grab wrench × 6no open box × 10no grab tab BobaMarkerPot PlaceTapeTeaToasterWrench Figure 10: Per-stage conditional success for Ego-only gaze-free baseline across 25 physical trials per task. all 25 pick cup 25 · 100% pick straw 25 · 100% insert straw 3 · 12% place on coaster 3 · 100% × 22no insert straw all 25 grab cap 25 · 100% grab marker 24 · 96% cap marker 17 · 71% grab bag 17 · 100% place marker 15 · 88% zip bag 14 · 93% × 1no zip bag × 2no place marker × 7no cap marker × 1no grab marker all 25 grab lid 25 · 100% optimal place 5 · 20% suboptimal place3 · 12%× 17failed place all 64 pick yellow 63 · 98% handoff 52 · 83% place yellow 42 · 81% × 10no place yellow × 11no handoff × 1no pick yellow all 25 grab tea 24 · 96% tea in cup 20 · 83% × 4no tea in cup × 1no grab tea all 25 open oven 25 · 100% pull tray 19 · 76% pick pan 17 · 89% place pan 13 · 76% push tray 13 · 100% close oven 13 · 100% × 4no place pan × 2no pick pan × 6no pull tray all 25 grab tab 22 · 88% open box 20 · 91% grab wrench 17 · 85% × 3no grab wrench × 2no open box × 3no grab tab BobaMarkerPot PlaceTapeTeaToasterWrench Figure 11: Per-stage conditional success for Ego+Wrist baseline across 25 physical trials per task. attention and FFN hidden layers of the transformer MLPs also have 10% dropout. Each transformer uses 6 transformer blocks, with any learnable query tokens initialized as standard normal embedding vectors with a standard deviation of 0.02 in each dimension. The token dimension of the transformer is 384, and it has an MLP expansion ratio of 4.0, with 12 attention heads. Of each head’s 128 dimensions, 32 are used for temporal RoPE, 48 for x, and 48 for y. B.1 Hyperparameters All training code will be released for reproducibility, but we report selected important hyperparam- eters here. General RL parameters We utilize the standard PPO update rules with value bootstrapping and discount factor γ = 0.95. We utilize generalized advantage estimation [70] with λ = 0.95. We do on-policy RL updates for 10 epochs, early-terminating if KL-divergence on the discrete policy output reaches more than 0.02. The gaze policy discretizes its action space and outputs a cross-product categorical distribution. Azimuth and elevation are binned into 8 polar directions, with 2 magnitudes for each direction (10 ◦ 12 Figure 12: Baseline action space grid-search over all options in 5 simulated tasks. Note that faded bars are evaluated on the exact same positional distributions as the training set to understand train- test overfitting in the simulated environments. Main-paper results are reported with the “Test Set” distribution, which is a different random seed from teleop demonstrations. Across the board, E- relative (rightmost in each section) outperforms other action spaces, so we use it for all real-world baselines. All baseline models exhibit signs of overfitting to the train positions on some tasks (par- ticularly tape and plate), even for tasks where teleop positions were randomly initialized. and 3 ◦ ) along with a zero action. Depth deltas are binned with 3 magnitudes (10 cm, 5 cm, 1 cm) and a zero action. The resulting cross-producted action space has 119 bins for the policy output. We prefer discretized action spaces as they help break ties in ambiguous actions (i.e., duplicate objects). Gaze pre-training Each policy rollout consists of 120 timesteps, all over a paused stereo frame. The policy is optimized with the AdamW [71] optimizer, with a learning rate of 10 −4 , β 1 = 0.9,β 2 = 0.95, weight decay of 0.1, with a linear ramp over the first roughly 5% of train steps and cosine decay over the last 30%. Each rollout uses 64 parallel environments, with a minibatch size of 16 environment rollouts. Policies are optimized for 15M total rollout steps, which corresponds to 1, 950 iterations, or 125, 000 unique rollouts total. We use an entropy regularization term of 0.01 during training, which decays to 0.005 alongside cosine decay. Gaze post-training and behavior cloning During BC-RL training, the behavior cloning policy is completely isolated from RL gradients by having its own set of projection and decoder MLPs. Both the target selector and BC policy are optimized with Muon [72], with the learning rate for the BC policy set to 5× 10 −4 and the target selector 10 −4 . The target selector entropy is regularized with a SAC-style entropy regularizer [73] which tunes the regularization to reach a target average entropy of 70% of the theoretical maximum policy entropy. During post-training, we maintain a buffer of the past 100 episodes of experience, from which we IID sample minibatches of 256 frames for the BC policy to use for SGD. We perform 4 minibatches of BC optimization for each update of the rolling experience buffer. During post-training we roll out 64 parallel environments for 30 steps with paused demonstration time, and 170 steps with time playing forward. During post-training, the video is played at 2x speed (skipping every other frame) so that each episode provides more coverage of the demonstration. In total, we roll out 10, 000, 000 environment steps, which corresponds to 50, 000 unique episode rollouts, 781 iterations, or 3, 124 BC gradient steps. Training details Each train run uses 8 NVIDIA L40S GPUs, training with data-parallel process coordination with environments evenly split among GPUs. All reported batch sizes are for the total number, not per-GPU. Training is done in bfloat16 mixed precision, except RoPE which is computed at float32 resolution. Baseline details To decide the action space to use for all baselines, we considered the classically known options of joint-space and end-effector SE(3) space, both chunk-relative and chunk-absolute. While community wisdom has recently drifted towards increasingly using chunk-relative represen- tations, we wanted to ensure that in our data-frugal scenario we thoroughly calibrated the perfor- mance of these. Testing such a large grid-search is infeasible in physical trials, so we ran a subset of our simulated tasks with all action spaces (Fig. 12). Across the board, we found that end-effector SE(3) relative to the first E frame in the chunk (“E-relative”) performed the best, so we use this 13 Figure 13: Real tasks and their stages. The tasks that we study in EyeRobot 2.0 represent a suite of complex manipulation behaviors, including bimanual coordination (tape handover, grasping and opening a lid), fine-grained tolerance requirements (capping a marker, grasping a zipper, inserting a straw into a hole, inserting a marker into a pencil pouch), challenging visual scenarios (picking up a white teabag on a white table), and multi-stage tasks (6 for both marker and toaster oven tasks). action space for all baseline experiments. To doubly verify this, we ran full baseline evaluations with world-relative SE(3) actions on the wrench and toaster tasks, and observed no difference in performance. Baselines are trained at 256 image resolution. C Robot inference Action chunks are predicted as 30 Hz open-loop trajectories, which we cubically interpolate at 200 Hz to obtain the commanded robot positions. We apply temporal ensembling during inference with a k value of 0.01. We additionally compute the temporally-ensembled velocity across predicted action chunks and provide this as a feedforward term for the YAM follower arms, which increases fluidity of the robot movements. See included videos for unedited videos of robot execution. 14 Figure 14: Simulated tasks and their stages. Simulated tasks represent a range of simple pick- place behaviors including single-arm (pot, spoon), single-arm-with-handover (tape), and bimanual coordination (tray, spoon). All models are queried as quickly as compute will allow, with end-to-end throughput of around 40 Hz for baselines and 20 Hz for AVF policies. 15 Figure 15: Training distribution of objects in the training set. Each dot represents an actual com- puted ground truth 3D location of an object which has been projected onto the table surface for visualization. Shadows of the YAM arms have been provided for scale. The distribution of objects is approximately uniformly random, except in tape where the distribution is a grid. D Data Details For stage-by-stage visualizations of all task stages, see Figs. 13 and 14. Figure 15 shows the ex- act training distribution for all sim and real tasks, as calculated by detecting each object’s starting position in 3D and plotting them projected onto the tabletop surface. As described in the Results section, notice how the tape task is collected in a grid, while other tasks are randomly initialized within a certain region for each object. Baseline policies are surprisingly sensitive to this positional distribution, with a large gap in performance between AVF and wrist policies on this grid-collected task. This gap is lessened by collecting randomly distributed initial states. See Tab. 4 for dataset size statistics for all tasks. E Simulation Details We use MuJoCo’s Newton solver with an elliptic friction cone, implicit-fast integration at a 0.5 ms timestep, and 10 noslip iterations to ensure stable contact-rich grasping behavior in simulation. Collision meshes are generated with a convex decomposition from CoACD [74], and we use the default YAM collision meshes and physical properties from MuJoCo Playground [66]. Simulation environments have hand-coded initial object distributions and physical success measures, and the same parameters are used across all models. The same seed is used for environment initializations to ensure direct comparability. F Evaluation Details F.1 Test Distribution See Fig. 16 for a visualization of the test distributions that we use for policy evaluation. All poli- cies are evaluated on the exact same positions, within human placement error, by aligning object positions with an overlay of the current position vs test position. F.2 Distractor objects We follow the same procedure for distractor experiments as full evals, by recording 25 positions of test objects and distractor objects and exactly repeating them for all models. See Fig. 17 for a random subsample of initial positions of distractor objects in the wrench task. 16 TeaToasterWrenchTape MarkerPot PlaceBoba Figure 16: Test distributions, visualized by average-compositing all 25 test overlay images on top of each other. During each evaluation we standardize reset positions across policies using these. Figure 17: Subsample of distractor positions during the wrench task (25 total). F.3 Per-stage success criteria for real tasks For all stages, we allow 3 retries before counting a stage as a failure, and count a failure if the robot stops moving for more than 10 seconds. Tape Pick yellow: the robot must grasp the yellow tape and lift it fully off the table without dropping it before the next stage. Handover: the robot must successfully transfer the yellow tape to the right gripper without dropping it. Place: the robot must place the yellow tape on top of the grey tape without it falling off or tilting. The static position must be stable with the center of gravity fully supported by the grey tape roll. Pot Pick lid: the robot must fully grasp and lift the pot lid off the table without dropping it. Place lid: the robot must place the pot lid on the pot with no gaps visible. This success criterion is quite strict, as simply resting the lid on top of the pot is insufficient. The lid must fully seal the rim of the pot, which is partially why the real success of this task is significantly lower than the simulated version of the task. Toaster Open door: the robot must fully open the door to expose the internal oven. Pull rack: the robot must pull out the internal wire rack from the toaster oven, and success is only marked if the rack is fully extended to reveal the pan inside, and the tray is not over-extended so far as to physically prevent pushing it back in. Pick up tray: the robot must grab the tray inside the toaster oven with both hands (failure if only one grasps), and gently place it on the table (failure counted if the tray is dropped from more than about 3 inches or twisted during placement). Push rack in: the robot must push the wire rack back into the oven, with failure marked if it insufficiently pushes the rack to allow door closure. Close oven door: the robot must fully close the oven door and end without touching the door. Marker Pick cap: the robot must pick up the cap and reach the capping position without dropping it (mid-air drop counts as a cap failure). Pick marker: the robot must pick up the marker from the table and reach the cap position without dropping it. Cap marker: the robot must insert the marker 17 SimulationReal platetapetigerhangspoonmedicaltraypotlidtapebobawrenchmarkerbagpotlidteatoasterTotal # Demos10064506463646410811412695213891278 Minutes21.610.84.08.118.47.59.531.126.353.411.326.552.0287.7 Table 4: Per-task demonstration counts and total duration of demonstrations. EyeRobot 2.0 (full) EyeRobot 2.0 (−fix. relative) Object gaze (worst obj.) Object gaze (best obj.) EyeRobot 2.0 −stereo EyeRobot 2.0 −foveation 0.740.530.500.690.740.72 Table 5: Simulated Ablations. Getting rid of fixation-relative actions (“−fix. relative”) severely harms performance even on relatively simple pick and place tasks, dropping by 21% overall. Re- straining gaze to one constant object during training produces a 24% drop for the worst option, and even for the best constant-gaze choice results in a 5% reduction. Interestingly, both stereo and foveation ablations have little effect in simulated tasks, which we hypothesize is because they do not possess the visual complexity or fine-grained tolerance to require precise visual features or spatial understanding. into the cap such that the cap reaches a friction-fit over the marker. If the cap is partially inserted and loosely dangling, this counts as a failure. Grab bag: the robot needs to grasp the pencil bag and place it upright close to the robot base (failure if not grasped or if the bag falls over). Insert marker: the robot must fully insert the marker into the pencil pouch zipper enclosure, with no part of the marker sticking out. Zip bag: the robot must fully zip the bag and let go with both grippers (zipping is considered reached when the robot fully closes the bag, with the zipper within 1cm of the edge of the bag). Boba Pick cup: the robot must lift the boba cup off the workspace without dropping it. Pick straw: the robot must pick up the straw without dropping it. Insert straw: the robot must insert the straw into the hole in the cup, with success counted if the straw is fully seated in the hole after releasing (partially inside and tilting outside is counted as a failure). Place on coaster: the robot must place the cup on the coaster, with the cup resting flat fully supported by the coaster at the end. Teabag Pick teabag: the robot must grasp and lift the teabag off the table without dropping it. Place teabag in cup: after releasing, the teabag must be fully contained inside the cup; if it hits the teacup wall and bounces out this is a failure. Wrench Grasp box tab: the robot must grasp the black box tab with the left hand securely without slippage during the lid movement. Open toolbox: the toolbox lid must be fully open more than 90 ◦ . Extract wrench: the robot must fully grasp and lift the wrench out of the toolbox, and reach a steady state above the wrench without dropping the wrench. If the wrench falls before reaching steady state, this is considered a failure. 18 PolicySubtask 1Subtask 2Subtask 3Success wrenchgrab tabopen boxgrab wrenchcomplete Ego-only3/250/250/250/25 Ego + Wrist10/256/254/254/25 EyeRobot 2.020/257/255/255/25 teagrab teatea in cup–complete Ego-only3/250/25–0/25 Ego + Wrist11/254/25–4/25 EyeRobot 2.013/254/25–4/25 tapepick yellowhandoffplace yellowcomplete Ego-only6/251/250/250/25 Ego + Wrist14/250/250/250/25 EyeRobot 2.023/2512/258/258/25 Table 6: Task success with distractors. Each cell reports the number of trials out of 25 reaching the corresponding subtask. Success denotes end-to-end task completion. Bold indicates the best policy for each task-subtask. PolicyPlateTapeTiger med mn±sd max med mn±sd max med mn±sd max BC ego-mono5452±5553838±5417678±583 BC ego-stereo4141±1415252±4558686±287 BC ego+wrist7171±3747979±3819695±296 EyeRobot 2.06767±3707576±2799797±198 PolicySpoonTrayPot med mn±sd max med mn±sd max med mn±sd max BC ego-mono2221±5265048±3506969±170 BC ego-stereo2324±2264039±6446264±569 BC ego+wrist2727±1278583±3858989±190 EyeRobot 2.02627±2307776±2789089±391 Table 7: Full sim success statistics (%). Median, mean, and std are reported over 3 seeds for each policy/task combination. Each evaluation of a seed is done over 100 trials. 19 References [1] M. F. Land and M. Hayhoe. In what ways do eye movements contribute to everyday activi- ties? Vision Research, 41(25):3559–3565, 2001. ISSN 0042-6989. doi:https://doi.org/10.1016/ S0042-6989(01)00102-X. URL https://w.sciencedirect.com/science/article/ pii/S004269890100102X. [2] J. A. Feldman. Four frames suffice: A provisional model of vision and space. Behavioral and Brain Sciences, 8(2):265–289, 1985. doi:10.1017/S0140525X00020707. [3] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. [4] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402.10329, 2024. [5] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control, 2024. URL https://arxiv. org/abs/2410.24164. [6] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024. [7] J. Kerr, K. Hari, E. Weber, C. M. Kim, B. Yi, T. Bonnen, K. Goldberg, and A. Kanazawa. Eye, robot: Learning to look to act with a bc-rl perception-action loop. arXiv preprint arXiv:2506.10968, 2025. [8] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988. [9] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017. [10] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3 (1):297–330, 2020. [11] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025. [12] Y. Du, D. Ho, A. Alemi, E. Jang, and M. Khansari. Bayesian imitation learning for end-to-end mobile manipulation. In International Conference on Machine Learning, pages 5531–5546. PMLR, 2022. [13] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. [14] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, 20 Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Mart ́ ın-Mart ́ ın, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024. [15] O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Her- zog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakr- ishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Sch ̈ olkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. B ̈ uchler, D. Ja- yaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Ra- dosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silv ́ erio, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. J. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sun- daresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart’in-Mart’in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Ju- lian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mir- chandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X mod- els. https://arxiv.org/abs/2310.08864, 2023. [16] T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid. Aloha unleashed: A simple recipe for robot dexterity, 2024. URL https://arxiv.org/abs/2410. 13126. [17] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sug- ano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Ki- 21 tani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022. URL https://arxiv.org/abs/2110.07058. [18] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383–19400, 2024. [19] Z. Lv, N. Charron, P. Moulon, A. Gamino, C. Peng, C. Sweeney, E. Miller, H. Tang, J. Meiss- ner, J. Dong, K. Somasundaram, L. Pesqueira, M. Schwesinger, O. Parkhi, Q. Gu, R. D. Nardi, S. Cheng, S. Saarinen, V. Baiyya, Y. Zou, R. Newcombe, J. J. Engel, X. Pan, and C. Ren. Aria everyday activities dataset, 2024. URL https://arxiv.org/abs/2402.13349. [20] Y. Liu, Y. Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022. [21] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M.-Y. Liu, Y. Zhu, J. Jang, and L. J. Fan. Dreamdojo: A generalist robot world model from large-scale human videos, 2026. URL https://arxiv.org/abs/2602.06949. [22] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113–1132. Pmlr, 2020. [23] C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422, 2023. [24] L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv:2403.12910, 2024. [25] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. In https://arxiv.org/abs/2403.01823, 2024. [26] K. Y. Goldberg and R. Bajcsy. Active touch and robot perception. Cognition and Brain Theory, 7(2):199–214, 1984. [27] R. Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988. [28] J. Aloimonos, I. Weiss, and A. Bandyopadhyay. Active vision. International journal of com- puter vision, 1(4):333–356, 1988. [29] R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42(2):177–196, 2018. [30] D. Jayaraman and K. Grauman. Learning to look around: Intelligently exploring unseen envi- ronments for unknown tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1238–1247, 2018. [31] T. Van de Maele, T. Verbelen, O. C ̧ atal, C. De Boom, and B. Dhoedt. Active vision for robot manipulators using the free energy principle. Frontiers in neurorobotics, 15:642780, 2021. [32] D. H. Ballard. Animate vision. Artificial intelligence, 48(1):57–86, 1991. 22 [33] J. K. Tsotsos. On the relative complexity of active vs. passive visual search. International journal of computer vision, 7(2):127–141, 1992. [34] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE interna- tional conference on robotics and automation (ICRA), pages 3357–3364. ieee, 2017. [35] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017. [36] X. Ye, Z. Lin, H. Li, S. Zheng, and Y. Yang. Active object perceiver: Recognition-guided pol- icy learning for object searching on mobile robots. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6857–6863. IEEE, 2018. [37] A. Rasouli, P. Lanillos, G. Cheng, and J. K. Tsotsos. Attention-based active visual search for mobile robots. Autonomous Robots, 44(2):131–146, 2020. [38] S. Dass, J. Hu, B. Abbatematteo, P. Stone, and R. Mart ́ ın-Mart ́ ın. Learning to look: Seeking information for decision making via policy factorization, 2024. URL https://arxiv.org/ abs/2410.18964. [39] N. P. Papanikolopoulos, P. K. Khosla, and T. Kanade. Visual tracking of a moving target by a camera mounted on a robot: A combination of control and vision. IEEE transactions on robotics and automation, 9(1):14–35, 1993. [40] D. Wilkes and J. K. Tsotsos. Active object recognition, volume 3. University of Toronto, 1994. [41] S. Gould, J. Arfvidsson, A. Kaehler, B. Sapp, M. Messner, G. R. Bradski, P. Baumstarck, S. Chung, A. Y. Ng, et al. Peripheral-foveal vision for real-time object recognition and tracking in video. In Ijcai, volume 7, pages 2115–2121, 2007. [42] A. Jonnalagadda, W. Y. Wang, B. Manjunath, and M. P. Eckstein. Foveater: Foveated trans- former for image classification. arXiv preprint arXiv:2105.14173, 2021. [43] J. E. Banta, Y. Zhien, X. Z. Wang, G. Zhang, M. Smith, and M. A. Abidi. Best-next-view algorithm for three-dimensional scene reconstruction using range images. In Intelligent Robots and Computer Vision XIV: Algorithms, Techniques, Active Vision, and Materials Handling, volume 2588, pages 418–429. SPIE, 1995. [44] S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza. An information gain formulation for active volumetric 3d reconstruction. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3477–3484. IEEE, 2016. [45] M. Mendoza, J. I. Vasquez-Gomez, H. Taud, L. E. Sucar, and C. Reta. Supervised learning of the next-best-view for 3d object reconstruction. Pattern Recognition Letters, 133:224–231, 2020. [46] R. Zeng, Y. Wen, W. Zhao, and Y.-J. Liu. View planning in robot active vision: A survey of systems, algorithms, and applications. Computational Visual Media, 6(3):225–245, 2020. [47] X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immer- sive active visual feedback. arXiv preprint arXiv:2407.01512, 2024. [48] I. Chuang, A. Lee, D. Gao, M.-M. Naddaf-Sh, and I. Soltani. Active vision might be all you need: Exploring active vision in bimanual robotic manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7952–7959. IEEE, 2025. 23 [49] Q. Zeng, C. Li, J. S. John, Z. Zhou, J. Wen, G. Feng, Y. Zhu, and Y. Xu. Activeumi: Robotic manipulation with active perception from robot-free human demonstrations. arXiv preprint arXiv:2510.01607, 2025. [50] H. Xiong, X. Xu, J. Wu, Y. Hou, J. Bohg, and S. Song. Vision in action: Learning active perception from human demonstrations. arXiv preprint arXiv:2506.15666, 2025. [51] J. Yu, Y. Shentu, D. Wu, P. Abbeel, K. Goldberg, and P. Wu. Egomi: Learning active vi- sion and whole-body manipulation from egocentric human demonstrations. arXiv preprint arXiv:2511.00153, 2025. [52] G. Spigler. Tavis: A benchmark for egocentric active vision and anticipatory gaze in imitation learning. arXiv preprint arXiv:2605.07943, 2026. [53] Y. Li, M. Liu, and J. M. Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE transactions on pattern analysis and machine intelligence, 45(6):6731–6747, 2021. [54] A. Pani and Y. Yang. Gaze-regularized vision-language-action models for robotic manipula- tion. arXiv preprint arXiv:2603.23202, 2026. [55] J. Li, Y. Qiao, Y. Guo, C. Chen, and W. Lian. Act, sense, act: Learning non-markovian active perception strategies from large-scale egocentric human data, 2026. URL https://arxiv. org/abs/2602.04600. [56] I. Chuang, J. Zou, A. Lee, D. Gao, and I. Soltani. Look, focus, act: Efficient and robust robot learning via human gaze and foveated vision transformers. arXiv preprint arXiv:2507.15833, 2025. [57] C. Li, K. Xiong, Y. Xu, L. Qian, Y. Wang, and W. Zhu. Gazevla: Learning human intention for robotic manipulation. arXiv preprint arXiv:2604.22615, 2026. [58] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Multi-task robot data for dual-arm fine manipulation. arXiv preprint arXiv:2401.07603, 2024. [59] A. Lee, I. Chuang, D. Gao, K. Fukazawa, and I. Soltani. Gaze on the prize: Shaping visual attention with return-guided contrastive learning. arXiv preprint arXiv:2510.08442, 2025. [60] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347. [61] N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. R ̈ adle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y. Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Doll ́ ar, N. Ravi, K. Saenko, P. Zhang, and C. Feichtenhofer. Sam 3: Segment anything with concepts, 2025. URL https: //arxiv.org/abs/2511.16719. [62] B. Wen, M. Trepte, J. Aribido, J. Kautz, O. Gallo, and S. Birchfield. Foundationstereo: Zero- shot stereo matching. CVPR, 2025. [63] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023. [64] The Robot Learning Company. TRLC-DK1: An Open Source Dev Kit for AI-native Robotics. https://w.robot-learning.co/, 2026. Accessed: 2026-05-26. [65] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026– 5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109. 24 [66] K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel. Mujoco playground, 2025. URL https://arxiv.org/abs/2502.08844. [67] O. Sim ́ eoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Cou- prie, J. Mairal, H. J ́ egou, P. Labatut, and P. Bojanowski. DINOv3, 2025. URL https: //arxiv.org/abs/2508.10104. [68] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2023.127063. URL https://w.sciencedirect.com/ science/article/pii/S0925231223011864. [69] D. Hendrycks and K. Gimpel. Bridging nonlinearities and stochastic regularizers with gaus- sian error linear units. CoRR, abs/1606.08415, 2016. URL http://arxiv.org/abs/1606. 08415. [70] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [71] I. Loshchilov and F. Hutter.Fixing weight decay regularization in adam.CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101. [72] K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan. github.io/posts/muon/. [73] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018. [74] X. Wei, M. Liu, Z. Ling, and H. Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG), 41(4):1–18, 2022. 25