Paper deep dive
Differential Amplifier-Inspired AmpAttention for Multi-View Robotic Manipulation
Jin Yang, Ping Wei, Nanning Zheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/7/2026, 4:49:03 AM
Summary
The paper introduces AmpAttention, a novel attention mechanism inspired by differential amplifiers, designed to suppress noise and mitigate attention drift in multi-view robotic manipulation. It proposes the RVAF model, which integrates task-guided intra- and inter-view AmpAttention, and extends it to RVAF++ by incorporating the SAM2 image encoder. Extensive experiments on RLBench and real-world scenarios demonstrate that RVAF achieves state-of-the-art performance with 33.3% less training time, while RVAF++ significantly improves high-precision task success rates.
Entities (7)
Relation Signals (7)
RVAF → uses → AmpAttention
confidence 97% · RVAF integrates task-guided intra-view and inter-view AmpAttention.
RVAF → extends → RVAF
confidence 96% · we extend RVAF to RVAF++ by incorporating the SAM2 image encoder.
RVAF → evaluatedon → RLBench
confidence 95% · Extensive experiments in both the RLBench simulation (18 RLBench tasks, 249 variations) and real-world settings demonstrate the effectiveness.
AmpAttention → inspiredby → Differential Amplifier
confidence 95% · AmpAttention is a novel attention mechanism inspired by the differential amplifier in analog circuits.
RVAF → incorporates → SAM2
confidence 94% · RVAF++ leverages the SAM2 image encoder to inject rich visual priors.
RVAF → outperforms → RVT-2
confidence 93% · RVAF achieves higher success rates than RVT-2 with 33% less training time.
CMRR Loss → optimizes → AmpAttention
confidence 92% · optimizing attention learning through a Common Mode Rejection Ratio (CMRR) loss.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-view robotic manipulation methods with the attention mechanism have recently achieved significant progress in both training efficiency and task performance. However, the inherent redundancy, occlusion, and viewpoint dependency in robotic view images often lead to severe attention drift. To address this challenge, we propose AmpAttention, a novel attention mechanism inspired by differential amplifiers in analog circuits. It aims to suppress attention noise and capture high signal-to-noise ratio signals for more reliable perception. Based on this, we introduce the RVAF model, which integrates task-guided intra-view and inter-view AmpAttention. Compared to previous state-of-the-art methods, RVAF achieves the optimal average success rate across 18 RLBench tasks (249 variations) while reducing training time by 33.3\%. RVAF also demonstrates strong potential in real-world high-precision tasks, exemplified by its ability to pick up a dart and accurately insert it into the red bullseye. Furthermore, we extend RVAF to RVAF++ by incorporating the SAM2 image encoder. RVAF++ achieves substantial gains on high-precision tasks, achieving a 91\% success rate on the `insert peg' task. More qualitative results are provided at the anonymous project website this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.02845v1
- Canonical: https://arxiv.org/abs/2607.02845v1
Trouble viewing inline? Open PDF directly →
Full Text
44,031 characters extracted from source content.
Expand or collapse full text
Differential Amplifier-Inspired AmpAttention for Multi-View Robotic Manipulation Jin Yang, Ping Wei∗, and Nanning Zheng National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University* Corresponding author. pingwei@xjtu.edu.cn Abstract Multi-view robotic manipulation methods with the attention mechanism have recently achieved significant progress in both training efficiency and task performance. However, the inherent redundancy, occlusion, and viewpoint dependency in robotic view images often lead to severe attention drift. To address this challenge, we propose AmpAttention, a novel attention mechanism inspired by differential amplifiers in analog circuits. It aims to suppress attention noise and capture high signal-to-noise ratio signals for more reliable perception. Based on this, we introduce the RVAF model, which integrates task-guided intra-view and inter-view AmpAttention. Compared to previous state-of-the-art methods, RVAF achieves the optimal average success rate across 18 RLBench tasks (249 variations) while reducing training time by 33.3%. RVAF also demonstrates strong potential in real-world high-precision tasks, exemplified by its ability to pick up a dart and accurately insert it into the red bullseye. Furthermore, we extend RVAF to RVAF++ by incorporating the SAM2 image encoder. RVAF++ achieves substantial gains on high-precision tasks, achieving a 91% success rate on the ‘insert peg’ task. More qualitative results are provided at the anonymous project website https://anonymous.4open.science/w/RVAF-Anonymization. I INTRODUCTION Robotic manipulation in unstructured 3D environments requires both precise reasoning and adaptability. Multi-view based methods have recently emerged as a powerful paradigm, as they provide diverse visual cues that are critical for accurate 3D understanding and task execution [23, 8, 7, 5]. Compared with voxel-based representations, view-based methods such as RVT [8] and RVT-2 [7] achieve superior performance with dramatically reduced training costs, demonstrating their potential for scalable deployment. However, effectively exploiting multi-view observations remains challenging. Existing approaches typically rely on Transformer-based attention mechanisms [8, 7, 30] to localize task-relevant regions (e.g. the red box in Fig. 1.(a)). But they often suffer from attention drift, where focus shifts from meaningful cues to redundant or noisy content (e.g., background clutter, occlusions, or repeated patterns), as illustrated in the heatmap on the left of Fig. 1.(b). This arises because robotic views contain highly redundant and viewpoint-dependent information, while standard attention has limited capacity to suppress irrelevant signals, ultimately degrading reasoning and action accuracy. Figure 1: Comparison of attention distributions between standard attention and AmpAttention in robotic view images. (a) Task scenario of open the top drawer. (b) Visualization of attention distributions under different attention mechanisms. left: standard attention. right: AmpAttention. (c) Quantitative comparison of both attention mechanisms. Given these challenges, there is a clear motivation to develop a reliable attention mechanism that can effectively capture task-relevant cues while suppressing task-irrelevant or noisy cues. To this end, we propose AmpAttention, a novel attention mechanism inspired by the differential amplifier in analog circuits. It focuses on the differential components of the signal while suppressing common-mode noise. This is achieved by jointly modeling the differential and common-mode components and optimizing attention learning through a Common Mode Rejection Ratio (CMRR) loss. By maintaining a high signal-to-noise ratio, AmpAttention ensures robust performance across multiple manipulation tasks. As illustrated on the right of Fig. 1.(b), AmpAttention produces more concentrated and discriminative activations around the manipulation target. Quantitative analysis in Fig. 1.(c) shows that AmpAttention achieves a 93.4% relative improvement in attention allocation to task-relevant regions (TRR) while reducing attention to task-irrelevant regions (TIR) by 40.6%. This results in a 226% relative improvement in the TRR–TIR ratio, significantly enhancing the model’s ability to focus on the most relevant cues. Building on AmpAttention, we propose the RVAF (Robotic View AmpFormer) model. It integrates intra-view AmpAttention to highlight task cues within each view, and inter-view AmpAttention to aggregate complementary information across views. Furthermore, we develop RVAF++, which leverages the SAM2 image encoder [20] to inject rich visual priors from large-scale pretraining [5, 30]. Extensive experiments in both the RLBench simulation (18 RLBench tasks, 249 variations) and real-world settings demonstrate the effectiveness of our approach. RVAF achieves higher success rates than RVT-2 with 33% less training time, while RVAF++ delivers remarkable gains on high-precision tasks such as ‘insert peg’ (91%). Extensive ablation studies validate the effectiveness of the components in the model. We further show that RVAF/RVAF++ outperforms the mainstream VLA baselines on some tasks. In real-world scenarios, we evaluate RVAF on five tasks using only a single third-person view camera. With only 50 collected demonstrations per task, RVAF performs well on high-precision manipulation tasks. In addition, RVAF demonstrates strong generalizability and robustness to environmental variations, maintaining stable performance under unseen objects and changing lighting conditions. We summarize our four key contributions as follows. 1. We propose AmpAttention, a differential-amplifier-inspired mechanism that extracts high signal-to-noise task cues from robotic views. 2. We design RVAF, which integrates intra- and inter-view AmpAttention for multi-view manipulation. 3. We extend RVAF to RVAF++ by incorporating the SAM2 image encoder, significantly improving high-precision manipulation performance. 4. We achieve optimal results in both simulation and the real world, demonstrating task generalization, efficiency, and scalability. I Related Work I-A Vision-based Robotic Manipulation. Vision-based robotic manipulation policies have gained significant attention for their ability to provide high-dimensional state information. Researchers have explored various visual inputs and training paradigms for vision-based policies. Some methods [27] encode RGB-D observations into latent states for reinforcement learning, but training remained sample-inefficient and slow [12]. Vision-language-action models [17, 16, 26] integrate RGB observations with natural language instructions for end-to-end action prediction. While improving generalization, they typically rely on large-scale pretraining, making them resource-intensive. Some methods [11, 23] voxelize point clouds to obtain strong structural priors. However, the precision of voxelization directly impacts task accuracy. Higher precision voxels require more memory and longer training times. An alternative line of work leverages multi-view representations to balance efficiency and accuracy. Instead of operating directly on point clouds, RVT [8] and RVT-2 [7] re-render multiple virtual view images from reconstructed point clouds and extracts scene features with the Transformer. Our method follows this multi-view paradigm but focuses on enhancing training efficiency and robust extraction of task-relevant cues with a high signal-to-noise ratio. Additionally, with the advancement of visual foundation models, methods like SAM-E [30] and SAM2Act [5] have demonstrated that large-scale vision models provide richer visual representations, further enhancing model performance. I-B Transformers for Robotic Manipulation. Transformers have been widely used in robotic manipulation, e.g., 3D scene representation [6], intent prediction [2], and long-horizon control [4, 25]. However, many of these methods still require hundreds of real-world demonstrations, which limits their scalability and practicality. Attention is central to Transformers, enabling modeling of dependencies and fusion of heterogeneous inputs, which has driven multimodal robot learning [31, 18, 24, 9]. However, standard attention often suffers from attention drift in vision-based manipulation (Fig. 1), diverting focus from task-relevant to irrelevant cues and reducing stability. This limitation motivates our design of a new attention mechanism to improve robustness. Figure 2: Illustration of the differential amplifier. I Methodology I-A Preliminaries Differential Amplifier. A differential amplifier amplifies the difference between two input signals (differential component) while suppressing their identical part (common-mode component), which is typically noise. Fig. 2 shows the circuit diagram of a differential amplifier. When R1=R2,R3=R4R_1=R_2,R_3=R_4, the ideal output (zero common-mode gain) is given by Vout=Ad(V2−V1),V_out=A_d (V_2-V_1 ), (1) where AdA_d is the differential gain. In practical circuits, component mismatches make common-mode signals unavoidable [21], leading to Vout=AdVd+AcVc=Ad(V2−V1)+Ac2(V1+V2),V_out=A_dV_d+A_cV_c=A_d (V_2-V_1 )+ A_c2 (V_1+V_2 ), (2) where AcA_c is the common-mode gain and AdAc A_dA_c refers to the common-mode rejection ratio (CMRR). A higher CMRR indicates that the circuit preserves more useful information while effectively rejecting interference. I-B Overview Problem Statement. We aim to train a robotic manipulation model capable of handling various tasks. The model takes as input a natural language description of the task along with the current visual observation and the current gripper state. The output is the next action, represented by the 6-DoF end-effector pose (3-DoF for position translation and 3-DoF for rotation orientation), a 1-DoF gripper state (open or close), and a 1-DoF collision state (a binary flag indicating whether collision is permissible during motion planning [23]). Figure 3: Overview of the RVAF architecture. Given a task instruction and multi-view images, RVAF first encodes them into task embeddings and view-specific embeddings. These embeddings are then processed through intra-view and inter-view AmpAttention to capture view representations. The resulting representations are used to predict the next key-frame pose. We construct a dataset D=D1,D2,…,DmD= \D_1,D_2,...,D_m \ comprising m expert demonstrations across various tasks for model training. Each demonstration Di=(Oi,Ri,li)D_i= (O_i,R_i,l_i ) represents a successful trajectory of length tit_i. It consists of a sequence of RGB-D observations Oi=o1i,o2i,…,otiiO_i= \o_1^i,o_2^i,...,o_t_i^i \, paired with corresponding robot actions Ri=r1i,r2i,…,rtiiR_i= \r_1^i,r_2^i,...,r_t_i^i \, and a task instruction lil_i in natural language. We adopt a key-frame based paradigm for manipulation learning [23], where key-frames denote critical gripper motions corresponding to meaningful state transitions. Following RVT [8], we extract key-frames from raw demonstrations to construct training samples. Architecture Overview. We illustrate the architecture of RVAF in Fig. 3. The input consists of multi-view orthographic RGB images, which are preprocessed and patchified into view-specific embeddings. Task instructions in natural language are encoded into task embeddings using the pre-trained CLIP [19]. These embeddings are first processed by the proposed intra-view AmpAttention, which guides each view to focus on task-relevant cues. Inter-view AmpAttention then aggregates complementary cues across views to form enriched view representations. The view representations are first processed with a convex upsampling layer to produce per-view heatmaps, which are back-projected into 3D space and aggregated to determine the target gripper position. Following RVT-2, we extract local features from each view based on the predicted position. These features are concatenated and processed by the MLP to predict the gripper orientation, collision state, and gripper state. Figure 4: Overview of the AmpAttention. I-C AmpAttention Existing view-based robotic manipulation methods rely on standard attention to extract task-relevant cues [8, 30]. However, the complexity and variability of manipulation scenarios present significant challenges in attention allocation. For example, in scenes with multiple visually similar objects, distractors can easily misdirect attention to irrelevant regions. Under such conditions, the standard attention mechanism is prone to attention drift (Fig. 1) due to its limited noise suppression capacity. To address this, we propose AmpAttention. It is designed to suppress noise and emphasize task-relevant cues with a higher signal-to-noise ratio. The computational pipeline of AmpAttention is depicted in Fig. 4. Given an input sequence embedding X∈ℝNseq×dX ^N_seq× d, it is first projected into the query, key, and value through three separate ‘Linear’ layers. Each projected embedding is then evenly split into two groups, resulting in Q1,Q2,K1,K2,V1,V2∈ℝNseq×d2Q_1,Q_2,K_1,K_2,V_1,V_2 ^N_seq× d2. [Q1;Q2]=XWQ,[K1;K2]=XWK,[V1;V2]=XWV, [Q_1;Q_2 ]=XW^Q, [K_1;K_2 ]=XW^K, [V_1;V_2 ]=XW^V, (3) where WQ,WK,WV∈ℝd×dW^Q,W^K,W^V ^d× d are trainable parameters. Then, we compute the attention output for each group as follows. A1(X)=[softmax(Q1K1TdQ1)Vi]i=1,2,A2(X)=[softmax(Q2K2TdQ2)Vi]i=1,2, splitA_1 (X )\,\,&=\,\, [soft ( Q_1K_1^T d_Q_1)V_i ]_i=1,2,\\ A_2 (X )\,\,&=\,\, [soft ( Q_2K_2^T d_Q_2)V_i ]_i=1,2, split (4) where [⋅]i=1,2 [· ]_i=1,2 denotes the concatenation of the results computed for indices i=1,2i=1,2. The dimensions dQ1d_Q_1 and dQ2d_Q_2 correspond to those of Q1Q_1 and Q2Q_2, respectively. Moreover, the softmax attention formulation in Equation 4 enables us to leverage efficient attention computation frameworks, such as FlashAttention [3], thereby improving training efficiency. Inspired by Equations 1 and 2, we define the differential attention operator AdiffA_diff as the difference between two softmax attention outputs, and the common-mode attention operator AcmA_cm as the average of these two outputs. Adiff(X)=A2−λA1,Acm(X)=12(A1+A2), splitA_diff (X )&=A_2-λ A_1,\\ A_cm (X )&= 12 (A_1+A_2 ), split (5) where A1A_1 and A2A_2 represent the softmax attention values computed from the Q, K, and V for each group. λ is a learnable coefficient that balances the two attention branches in the differential computation, parameterized following [28]. As analyzed in Equation 1, AdiffA_diff can directly serve as the output of AmpAttention in the ideal case. However, due to the inevitable existence of common-mode signals, Equation 2 motivates a more principled formulation of AmpAttention. One straightforward solution is to use a weighted sum of AdiffA_diff and AcmA_cm as the final AmpAttention output. This design introduces additional hyperparameters that could potentially undermine model stability and generalization. To mitigate these issues, we propose a CMRR-inspired loss function LCMRRL_CMRR, which optimizes differential attention learning. To avoid numerical instability arising from the division by zero during training, we approximate the CMRR by subtracting logarithms instead of directly performing the division. LCMRRL_CMRR is computed as follows. ℒCMRR=1Nseq∑n=1NseqS(Acmn,Adiffn),S(a,b)=max(0,log(1+‖a‖2)−log(1+‖b‖2)), splitL_CMRR&= 1N_seq _n=1^N_seqS (A_cm^n,A_diff^n ),\\ S (a,b )&= (0, (1+ \|a \|_2 )- (1+ \|b \|_2 ) ), split (6) where ∥⋅∥2 \|· \|_2 denotes the Frobenius norm. The hinge operation with max(0,⋅) (0,·) ensures that the loss imposes an effective constraint only when common-mode noise suppression is insufficient. This design prevents unnecessary regularization from disrupting well-learned differential characteristics. S(Acmn,Adiffn)S (A_cm^n,A_diff^n ) refers to the CMRR loss value at the n-th position in the input sequence. The ℒCMRRL_CMRR computes the mean of these CMRR loss values across all positions in the sequence of length NseqN_seq. We also adopt the multi-head attention (MHA) design where each head’s output is individually normalized and scaled. The processed outputs from all heads are then concatenated to produce the final results as follows. headh=Adiffh(X;WhQ,WhK,WhV),s.t.ℒCMRR(Adiffh,Acmh)headh′=(1−λinit)⋅LN(headh)AmpAttention(X)=Concat[headh′]h=1,2,…,HWO split&head_h=A_diff^h (X;W_h^Q,W_h^K,W_h^V ),s.t.\,L_CMRR (A_diff^h,A_cm^h )\\ &head_h = (1- _init )·LN (head_h )\,\,\\ &AmpAttention (X )=Concat [head_h ]_h=1,2,...,HW^O split (7) where λinit∈(0,1) _init∈(0,1) initializes λ with a strategy consistent with [28]. LN(⋅)LN (· ) is the layer normalization operation, we adopt RMSNorm for each head in this work. (1−λinit) (1- _init ) scales the normalized output of each head [28], ensuring that the gradient dynamics of AmpAttention approximate those of standard MHA. This design preserves training stability and enables direct hyperparameter transfer. H is the number of heads. Concat[⋅]Concat [· ] refers to concatenate all heads along the channel dimension. WO∈ℝd×dW^O ^d× d denotes the learnable output projection. We set H to half the number of heads in standard MHA, ensuring that the head dimension of AmpAttention matches that of standard MHA. This configuration maintains computational complexity comparable to standard MHA while avoiding performance degradation. I-D Task-guided Intra-view AmpAttention A key aspect of multi-view-based robotic manipulation is the extraction of task-relevant visual cues from multiple views. Prior methods apply self-attention within views and then fuse with task instructions, which can cause early focus on irrelevant regions. We instead adopt a “task-guided first” design. We first apply cross-AmpAttention between task instructions and each view to guide feature extraction, and then use inter-view AmpAttention to aggregate visual cues. The computation process of Task-guided Intra-view AmpAttention is illustrated in the upper-right part of Fig. 3. The task embeddings are first concatenated with each view embedding. These concatenated embeddings are then fed into a Linear layer for mapping, which helps eliminate the modality gap. The mapped features are subsequently processed through a stack of L modules (L is set to 4), resulting in task-guided view embeddings. Each module contains AmpAttention and FeedForward submodules, with residual connections incorporated to stabilize training. The FeedForward submodule consists of two Linear layers, with a GEGLU [22] activation function applied between them. I-E Inter-view AmpAttention The Inter-view AmpAttention plays a critical role in integrating task-relevant cues across multiple views. Its computation process is illustrated in the lower-right part of Fig. 3. Specifically, it begins with task-guided view embeddings, which are flattened into a single sequence. These flattened embeddings are then processed through a stack of modules containing both AmpAttention and FeedForward submodules. The configuration of these modules is identical to that of the Intra-view AmpAttention, consisting of L modules. For action prediction, we follow RVT-2 [7], which employs convex upsampling for position estimation, location-conditioned rotation prediction, and simple classifiers for gripper and collision states. I-F RVAF++: SAM2-Enhanced Visual Representation Previous works [5, 30] have demonstrated that utilizing visual foundation models pre-trained on large-scale datasets to extract visual embeddings significantly improves the performance of view-based robotic manipulation methods. To this end, we integrate SAM2’s image encoder [20] into RVAF to extract view image embeddings, resulting in RVAF++. For model efficiency, we employ low-rank adaptation [10] with a default rank of 16 to fine-tune the image encoder of SAM2. I-G Training To train the model, we define the model loss function ℒmodelL_model, which consists of ℒCMRR−totalL_CMRR-total, ℒtransL_trans, ℒrotaL_rota, ℒgripperL_gripper, and ℒcollisionL_collision. The total CMRR loss ℒCMRR−totalL_CMRR-total is employed to regulate the learning of all AmpAttention modules within the model, ensuring the provision of high signal-to-noise ratio visual cues. It is computed as the average of the CMRR losses across all AmpAttention modules, and further averaged across N samples. ℒCMRR−total=1N∑i=1N(12L∑l=12LℒCMRRl(Adiffi,Acmi))L_CMRR-total=\,\, 1N _i=1^N ( 12L _l=1^2LL_CMRR^l (A_diff^i,A_cm^i ) ) (8) where 2L2L is the total number of AmpAttention modules in the model (Fig. 3). The CMRR loss for each AmpAttention module is computed as described in Equation 6. We follow prior works [8, 7] and apply standard cross-entropy losses for action prediction, including translation loss ℒtransL_trans, rotation loss ℒrotaL_rota, gripper state loss ℒgripperL_gripper, and collision state loss ℒcollisionL_collision. Thus, the model loss function ℒmodelL_model is defined as follows. ℒmodel= _model= α1(ℒtrans+ℒrota+ℒgripper+ℒcollision) _1 (L_trans+L_rota+L_gripper+L_collision ) (9) +α2ℒCMRR−total, + _2L_CMRR-total, where α1 _1 and α2 _2 are balance hyperparameters. In this work, we set α1 _1 to 1.0 and α2 _2 to 0.01, respectively. TABLE I: Multi-task performance (%) on RLBench. Models Avg. Success↑ Train time (days)↓ Close Jar Drag Stick Insert Peg Meat off Grill Open Drawer Place Cups Place Wine Push Buttons ACt3D [6] 65.0 5.0(V100) 92.0 92.0 27.0 94.0 93.0 3.0 80.0 99.0 RVT [8] 62.9 1.0(V100) 52.0± 2.5 99.2± 1.6 11.2± 3.0 88.0± 2.5 71.2± 6.9 4.0± 2.5 91.0± 5.2 100.0± 0.0 RVT-2 [7] 81.4 0.83(V100) 100.0± 0.0 99.0± 1.7 40.0± 0.0 99.0± 1.7 74.0± 11.8 38.0± 4.5 95.0± 3.3 100.0± 0.0 RVT-2∗ [7] 80.3 0.36(4090) 100.0± 0.0 100.0± 0.0 38.0± 5.2 98.0± 4.0 81.0± 3.8 36.0± 3.3 94.0± 2.3 92.0± 0.0 SAM-E [30] 70.6 - 82.4± 3.6 100.0± 0.0 18.4± 4.6 95.2± 3.3 95.2± 5.2 0.0± 0.0 94.4± 4.6 100.0± 0.0 SAM2Act [5] 86.8 -(H100) 99.0± 2.0 99.0± 2.0 84.0± 5.7 98.0± 2.3 83.0± 6.0 47.0± 6.0 93.0± 3.8 100.0± 0.0 RVAF (ours) 83.1 0.24(4090) 100.0± 0.0 100.0± 0.0 30.0± 4.0 100.0± 0.0 86.0± 2.3 31.0± 3.8 96.0± 3.3 100.0± 0.0 RVAF++ (w. SAM2) 87.0 0.58(4090) 99.0± 2.0 100.0± 0.0 91.0± 3.8 100.0± 0.0 85.0± 2.0 36.0± 3.3 95.0± 3.8 100.0± 0.0 Models Put in Cupboard Put in Drawer Put in Safe Screw Bulb Slide Block Sort Shape Stack Blocks Stack Cups Sweep to Dustpan Turn Tap ACt3D [6] 51.0 90.0 95.0 47.0 93.0 8.0 12.0 9.0 92.0 94.0 RVT [8] 49.6± 3.2 88.0± 5.7 91.2± 3.0 48.0± 5.7 81.6± 5.4 36.0± 2.5 28.8± 3.9 26.4± 8.2 72.0± 0.0 93.6± 4.1 RVT-2 [7] 66.0± 4.5 96.0± 0.0 96.0± 2.8 88.0± 4.9 92.0± 2.8 35.0± 7.1 80.0± 2.8 69.0± 5.9 100.0± 0.0 99.0± 1.7 RVT-2∗ [7] 68.0± 3.3 99.0± 2.0 94.0± 4.0 92.0± 0.0 51.0± 3.8 49.0± 7.6 83.0± 3.8 78.0± 2.3 99.0± 2.0 93.0± 5.0 SAM-E [30] 64.0± 2.8 92.0± 5.7 95.0± 3.3 78.4± 3.6 95.2± 1.8 34.4± 6.1 26.4± 4.6 0.0± 0.0 100.0± 0.0 100.0± 0.0 SAM2Act [5] 75.0± 3.8 99.0± 2.0 98.0± 2.3 89.0± 2.0 86.0± 4.0 64.0± 4.6 76.0± 8.6 78.0± 4.0 99.0± 2.0 96.0± 5.7 RVAF (ours) 76.0± 0.0 100.0± 0.0 94.0± 2.3 93.0± 2.0 80.0± 0.0 47.0± 2.0 72.0± 3.3 91.0± 3.8 100.0± 0.0 99.0± 2.0 RVAF++ (w. SAM2) 73.0± 7.6 100.0± 0.0 99.0± 2.0 95.0± 2.0 74.0± 2.3 61.0± 7.6 76.0± 3.3 84.0± 4.6 99.0± 2.0 99.0± 2.0 IV Experiments IV-A Simulation Experiments Setup Simulation Benchmark. Following prior works [8, 7, 5], we evaluate on RLBench [13], a simulation suite built on CoppeliaSim with a Franka Panda arm and parallel gripper. The robot is equipped with four RGB-D cameras (front, left shoulder, right shoulder, and wrist) at a resolution of 128×128. RLBench provides 18 manipulation tasks with 249 variations, ranging from simple actions to common pick-and-place and high-precision operations. Training and Evaluation Details. Following existing baselines [7, 30], we utilize 1800 expert demonstrations (100 demonstrations per task) from the RLBench [13] for training and 450 unseen demonstrations (25 demonstrations per task) for evaluation. The resolution of all virtual images is set to 224 × 224. All models are trained using 8 NVIDIA 4090 (24 GB) GPUs. We train the RVAF model for approximately 50K steps, with a batch size of 160 (20 × 8) and a learning rate of 2e-3. The RVAF++ model is trained for approximately 110K steps, with a batch size of 64 (8 × 8) and a learning rate of 3.2e-3. Both RVAF and RVAF++ utilize the LAMB [29] optimizer and employ cosine learning rate decay with a warm-up period for the first 2,000 steps. We evaluate all tasks using the model from the final epoch. Due to the randomness introduced by the sampling-based motion planner used in RLBench [13], we run each model four times on each task and report the average success rate and standard deviation. Figure 5: Real-world experimental setup. (a) Real-world task scenario setting. (b) Pick dart and insert bullseye: the red/green bullseye has a radius of 6/14 m. (c) Press the toy switch: pressing a 12m × 6m control switch. (d) Cable grasping task. (e) Pick and place toy bear task. IV-B Real-World Experiments Setup Environment Setup and Tasks. We construct a tabletop experimental platform as illustrated in Fig. 5(a). The system consists of a Realman RM65 6DoF robotic arm equipped with the Realman RMG24 two-finger parallel gripper. The scene is perceived by a statically mounted Intel RealSense D435i camera from a third-person perspective. The extrinsic transformation between the camera and the robot base frame is obtained via a checkerboard-based calibration method. To evaluate precision and generalization, we design five real-robot tasks: four challenging tasks (‘press the toy switch’, ‘pick dart and insert red/green bullseye’ and ‘cable grasping’) and one easier ‘pick and place toy bear’ task. Each task supports diverse natural language instructions and includes randomized object placements to test robustness. The task scene details are shown in Fig. 5(b)-(e). Data Collection. We collect paired visual and action data using a manual teaching device. The visual data consists of RGB and depth images captured by a single camera, while the action data includes the robot arm’s joint states and the gripper status. The visual data is recorded at 20 Hz and action data at 200 Hz. We align the two data streams using their timestamps to ensure synchronization. For each task, we collected 50 demonstration sequences. Training and Evaluation Details. To ensure a fair comparison, we train both the RVT-2 [7] and RVAF models on real-world data using the same training settings. For different tasks, we select different model checkpoints saved during training. During evaluation, we deploy the models on a server using the Flask framework to create an HTTP-based inference API. Observation data collected on a local device is sent to the server via POST requests. Upon receiving the data, the server performs inference using the deployed model and returns the predictions in real time. IV-C Quantitative Experiments Results Simulation Multi-task Performance. Tab. I compares the performance of our RVAF-based models with previous state-of-the-art methods. To ensure fairness, we re-train RVT-2 [7], using our hardware setup and refer to this result as RVT-2∗. Without incorporating vision foundation models, RVAF achieves a 3.5% relative improvement over RVT-2 on the same device, while also reducing training time by roughly 33.3% (from 0.36 days to 0.24 days). When leveraging SAM2, RVAF++ surpasses the prior method SAM-E [30] by 16% absolute improvement. Overall, RVAF-based models outperform previous methods in 12 out of 18 tasks. Further analysis indicates significant performance improvements in high-precision tasks after the introduction of the SAM2 [20]. For example, in the insert peg task, success rates increased substantially from 84% (prior best) to 91%. We attribute these improvements to the powerful representational capabilities provided by SAM2, which enables more high-precision visual cues. However, RVAF++ does not achieve performance comparable to previous best methods on the slide blocks task. We argue that this discrepancy is due to overfitting on the simple task or inherent trade-offs in multi-task learning, where improvements on complex tasks negatively impact performance on simpler ones. TABLE I: Ablation study (%) on RLBench. We quantify the impact of different components in RVAF. Row ID AmpAttn. AmpAttn. w/o ℒCMRRL_CMRR Attn. Intra-Inter Coarse-to- Fine Scale of Zoom in #\# of Views Train time (hours) Train time (%\% of base) Avg. Success Avg. Success diff. wrt. base 1 ✓ × × ✓ ✓ 4 3 5.876 100.0%\% 83.1 0 2 × ✓ × ✓ ✓ 4 3 5.849 99.5%\% 81.6 -1.5 3 × × ✓ ✓ ✓ 4 3 5.150 87.6%\% 81.2 -1.9 4 ✓ × × × ✓ 4 3 5.870 99.9%\% 80.2 -2.9 5 ✓ × × ✓ × - 3 4.172 71.0%\% 67.2 -15.9 6 ✓ × × ✓ ✓ 2 3 5.879 100.1%\% 82.8 -0.3 7 ✓ × × ✓ ✓ 3 3 5.868 99.9%\% 81.8 -1.3 8 ✓ × × ✓ ✓ 5 3 6.007 102.2%\% 79.9 -3.2 9 ✓ × × ✓ ✓ 4 2 5.238 89.1%\% 76.2 -6.9 10 ✓ × × ✓ ✓ 4 4 7.315 124.5%\% 80.7 -2.4 Simulation Ablation Study. Tab. I presents extensive ablation studies of components in RVAF within the simulation environment. We examine the impact of several design choices: a) The type of attention mechanism used in the RVAF model. ‘AmpAttn.’ is the full AmpAttention mechanism, ‘AmpAttn. w/o ℒCMRRL_CMRR’ is a variant that considers only differential gain (as defined in Equation 1), and ‘Attn.’ refers to the standard attention mechanism; b) whether to apply task-guided intra-view AmpAttention before inter-view AmpAttention (‘Intra-Inter’); c) whether to use the coarse-to-fine design [14, 7] (‘Coarse-to-Fine’); d) varying the zoom-in scale used in the fine stage of coarse-to-fine rendering (‘Scale of Zoom in’); e) the number of rendered virtual views (‘#\# of Views’). For each setting, we report both the average success rate and training time. (a) Comparing rows 1, 2, and 3, the RVAF model with the full AmpAttention mechanism achieves the optimal performance. This improvement is attributed to its ability to suppress attention noise, allowing the model to focus more effectively on high signal-to-noise visual cues. The performance drop from row 1 to row 2, where the ℒCMRRL_CMRR loss is removed, indicates that common-mode suppression contributes to more stable and precise attention. (b) The comparison between rows 1 and 4 highlights the benefit of applying task-guided intra-view AmpAttention before inter-view AmpAttention, resulting in improved average success rate. (c) As expected, adopting the coarse-to-fine design leads to significant gains in performance (row 1 vs. row 5). (d) The scale of zoom-in in the fine stage also plays a critical role. Comparing rows 1, 6, 7, and 8 reveals that appropriate scaling provides a more informative field of view, which helps to improve model performance. However, larger zoom levels do not guarantee better performance and even reduce efficiency by increasing training time and limiting contexts. (e) The number of rendered virtual views is a critical design factor in multi-view models. Comparing rows 1, 9, and 10, we observe that increasing the number of views does not necessarily improve performance. Excessive views may introduce redundancy and amplify attention noise, leading to higher computational cost without performance gains. TABLE I: Comparison of RVAF series with VLAs on four tasks. Tasks 1 to 4 correspond to close jar, insert onto square peg, open drawer, and put item in drawer, respectively. Method Task1 Task2 Task3 Task4 Avg. RVAF 100.0 30.0 86.0 100.0 79.0 RVAF++ 99.0 91.0 85.0 100.0 93.8 π0-FAST _0-FAST[1] 62.1 8.9 75.0 68.3 53.6 OpenVLA-OFT[15] 68.3 12.3 81.2 70.1 58.0 TABLE IV: Performance of the model in real-world scenarios. Models Avg. Success↑ (%) Press the toy switch Pick dart and insert red bullseye Pick dart and insert green bullseye Cable grasping Pick and place toy bear RVT-2 [7] 44.0 3/10 3/10 5/10 2/10 9/10 OpenVLA-OFT[15] 44.0 3/10 2/10 4/10 3/10 10/10 RVAF (ours) 64.0 6/10 5/10 8/10 4/10 9/10 Figure 6: Robustness validation of RVAF under the real-world environment. Figure 7: Illustration of attention heatmaps for different models across various tasks. The highlighted regions indicate the robot’s next action-relevant area. (a) Basic manipulation tasks. (b) High-precision manipulation tasks. Compared to VLAs. Tab. I compares RVAF++ with two advanced VLAs (π0 _0-FAST and OpenVLA-OFT) on four tasks. RVAF++ achieves the best overall performance, with an average success rate of 93.8%, substantially outperforming π0 _0-FAST and OpenVLA-OFT. Notably, on the high-precision assembly task (Task2), RVAF++ is significantly higher than both VLA models. These results suggest that the multi-view imitation learning paradigm still offers clear advantages on certain tasks. Real-World Performance. Tab. IV compares RVAF with RVT-2 and OpenVLA-OFT on five real-world tasks (10 trials per task). RVAF consistently performs better on the four high-precision tasks, and achieves the highest average success rate (64.0%). Fig. 6 validates the robustness of RVAF in real-world environments. The first three images show that the robotic arm successfully executes tasks despite interference from previously unseen objects. The last image demonstrates successful task execution under changing lighting conditions. IV-D Qualitative Results Fig. 7 compares attention heatmaps of RVAF, RVAF++, and RVT-2 on both basic tasks and high-precision tasks. The heatmaps from the front and right views reveal how each model allocates attention, offering intuitive insights into the behavioral differences observed in Tab.I. RVT-2, based on standard attention, shows diffuse patterns vulnerable to background interference, leading to attention drift. RVAF with AmpAttention yields more concentrated focus, enabling accurate localization of task-relevant regions. RVAF++ further improves precision and semantic consistency in high-precision tasks, explaining its substantial performance gains in these scenarios. V Conclusion In this work, we propose RVAF and RVAF++, which incorporate task-guided intra-view and inter-view AmpAttention to improve perception accuracy and training efficiency for multi-view robotic manipulation. At the core is AmpAttention, a novel attention mechanism inspired by differential amplifiers. It facilitates the extraction of task-relevant visual cues while suppressing irrelevant cues, thereby mitigating attention drift. Extensive simulation and real-world experiments demonstrate strong effectiveness, generalization, and robustness. Furthermore, our results confirm that leveraging large-scale pretrained visual foundation models significantly enhances the performance of robotic manipulation in high-precision tasks. References [1] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: TABLE I. [2] H. M. Clever, A. Handa, H. Mazhar, K. Parker, O. Shapira, Q. Wan, Y. Narang, I. Akinola, M. Cakmak, and D. Fox (2021) Assistive tele-op: leveraging transformers to collect robotic task demonstrations. arXiv preprint arXiv:2112.05129. Cited by: §I-B. [3] T. Dao (2023) Flashattention-2: faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. Cited by: §I-C. [4] S. Dasari, O. Mees, S. Zhao, M. K. Srirama, and S. Levine (2024) The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410.10088. Cited by: §I-B. [5] H. Fang, M. Grotz, W. Pumacay, Y. R. Wang, D. Fox, R. Krishna, and J. Duan (2025) SAM2Act: integrating visual foundation model with a memory architecture for robotic manipulation. arXiv preprint arXiv:2501.18564. Cited by: §I, §I, §I-A, §I-F, TABLE I, TABLE I, §IV-A. [6] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki (2023) Act3d: 3d feature field transformers for multi-task robotic manipulation. arXiv preprint arXiv:2306.17817. Cited by: §I-B, TABLE I, TABLE I. [7] A. Goyal, V. Blukis, J. Xu, Y. Guo, Y. Chao, and D. Fox (2024) Rvt-2: learning precise manipulation from few demonstrations. arXiv preprint arXiv:2406.08545. Cited by: §I, §I, §I-A, §I-E, §I-G, TABLE I, TABLE I, TABLE I, TABLE I, §IV-A, §IV-A, §IV-B, §IV-C, §IV-C, TABLE IV. [8] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y. Chao, and D. Fox (2023) Rvt: robotic view transformer for 3d object manipulation. In Conference on Robot Learning, p. 694–710. Cited by: §I, §I, §I-A, §I-B, §I-C, §I-G, TABLE I, TABLE I, §IV-A. [9] P. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid (2023) Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning, p. 175–187. Cited by: §I-B. [10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), p. 3. Cited by: §I-F. [11] H. Huang, O. Howell, D. Wang, X. Zhu, R. Walters, and R. Platt (2024) Fourier transporter: bi-equivariant robotic manipulation in 3d. arXiv preprint arXiv:2401.12046. Cited by: §I-A. [12] S. James and A. J. Davison (2022) Q-attention: enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters 7 (2), p. 1612–1619. Cited by: §I-A. [13] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison (2020) Rlbench: the robot learning benchmark & learning environment. IEEE Robotics and Automation Letters 5 (2), p. 3019–3026. Cited by: §IV-A, §IV-A, §IV-A. [14] S. James, K. Wada, T. Laidlow, and A. J. Davison (2022) Coarse-to-fine q-attention: efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13739–13748. Cited by: §IV-C. [15] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. arXiv preprint arXiv:2502.19645. Cited by: TABLE I, TABLE IV. [16] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: §I-A. [17] Q. Li, Y. Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y. Deng, S. Xu, Y. Zhang, et al. (2024) Cogact: a foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650. Cited by: §I-A. [18] X. Li, M. Zhang, Y. Geng, H. Geng, Y. Long, Y. Shen, R. Zhang, J. Liu, and H. Dong (2024) Manipllm: embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18061–18070. Cited by: §I-B. [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §I-B. [20] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. (2024) Sam 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: §I, §I-F, §IV-C. [21] W. M. Sansen (2007) Analog design essentials. Vol. 859, Springer Science & Business Media. Cited by: §I-A. [22] N. Shazeer (2020) Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §I-D. [23] M. Shridhar, L. Manuelli, and D. Fox (2023) Perceiver-actor: a multi-task transformer for robotic manipulation. In Conference on Robot Learning, p. 785–799. Cited by: §I, §I-A, §I-B, §I-B. [24] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg (2023) Progprompt: generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), p. 11523–11530. Cited by: §I-B. [25] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: §I-B. [26] J. Wen, Y. Zhu, J. Li, M. Zhu, Z. Tang, K. Wu, Z. Xu, N. Liu, R. Cheng, C. Shen, et al. (2025) Tinyvla: towards fast, data-efficient vision-language-action models for robotic manipulation. IEEE Robotics and Automation Letters. Cited by: §I-A. [27] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto (2021) Mastering visual continuous control: improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645. Cited by: §I-A. [28] T. Ye, L. Dong, Y. Xia, Y. Sun, Y. Zhu, G. Huang, and F. Wei (2024) Differential transformer. arXiv preprint arXiv:2410.05258. Cited by: §I-C, §I-C. [29] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C. Hsieh (2019) Large batch optimization for deep learning: training bert in 76 minutes. arXiv preprint arXiv:1904.00962. Cited by: §IV-A. [30] J. Zhang, C. Bai, H. He, Z. Wang, B. Zhao, X. Li, and X. Li (2024) SAM-e: leveraging visual foundation model with sequence imitation for embodied manipulation. In International Conference on Machine Learning, p. 58579–58598. Cited by: §I, §I, §I-A, §I-C, §I-F, TABLE I, TABLE I, §IV-A, §IV-C. [31] W. Zhao, P. Ding, M. Zhang, Z. Gong, S. Bai, H. Zhao, and D. Wang (2025) Vision-language-action model with speech instructions for customized robot manipulation. arXiv preprint arXiv:2502.13508. Cited by: §I-B.