Paper deep dive
SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning
David D. Yuan, Tony Z. Zhao, Kaylee Burns, Chelsea Finn
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While learned robotic policies hold promise for advancing generalizable manipulation, their practical deployment is often hindered by suboptimal execution speeds. Imitation learning policies are inherently limited by hardware constraints and the speed of the operator during data collection. In addition, there are no established methods for accelerating policies learned via imitation, and the empirical relationship between execution speed and task success remains underexplored. To address these issues, we introduce SpeedTuning, a reinforcement learning framework specifically designed to enhance the speed of manipulation policies. SpeedTuning learns to predict the optimal execution speed for actions, thereby complementing a base policy without necessitating additional data collection. We provide empirical evidence that SpeedTuning achieves substantial improvements in execution speed, exceeding 2.4x speed-up, while preserving an adequate success rate compared to both the original task policy and straightforward speed-up methods such as linear interpolation at a fixed speed. We evaluate our approach across a diverse set of dynamic and precise tasks, including pouring, throwing, and picking, demonstrating its effectiveness and robustness in enhancing real-world robotic manipulation.
Tags
Links
- Source: https://arxiv.org/abs/2608.09138v1
- Canonical: https://arxiv.org/abs/2608.09138v1
Trouble viewing inline? Open PDF directly →
Full Text
40,830 characters extracted from source content.
Expand or collapse full text
SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning David D. Yuan, Tony Z. Zhao, Kaylee Burns, and Chelsea Finn Stanford University Abstract While learned robotic policies hold promise for advancing generalizable manipulation, their practical deployment is often hindered by suboptimal execution speeds. Imitation learning policies are inherently limited by hardware constraints and the speed of the operator during data collection. In addition, there are no established methods for accelerating policies learned via imitation, and the empirical relationship between execution speed and task success remains underexplored. To address these issues, we introduce SpeedTuning, a reinforcement learning framework specifically designed to enhance the speed of manipulation policies. SpeedTuning learns to predict the optimal execution speed for actions, thereby complementing a base policy without necessitating additional data collection. We provide empirical evidence that SpeedTuning achieves substantial improvements in execution speed, exceeding 2.4x speed-up, while preserving an adequate success rate compared to both the original task policy and straightforward speed-up methods such as linear interpolation at a fixed speed. We evaluate our approach across a diverse set of dynamic and precise tasks, including pouring, throwing, and picking, demonstrating its effectiveness and robustness in enhancing real-world robotic manipulation. Videos and code are available at https://daivdyuan.github.io/speed-tuning/ †publicationid: pubid: © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. DOI: 10.1109/ICRA55743.2025.11128753 I INTRODUCTION Speed is critical in real-world robotic tasks, as faster policies can increase throughput and enhance the user experience. For example, in robotic assembly, high-speed and precise manipulation is essential for tasks like inserting screws or fitting components, where faster execution directly boosts production efficiency. Even though speed is essential in many real-world settings, imitation learning methods largely ignore speed both when training and evaluating a policy. Training robots to perform tasks quickly and successfully through imitation is challenging. Imitation learning is typically concerned with matching the behavior in the demonstration dataset, while it is hard to collect fast and accurate teleoperation data even from experts. Take ALOHA [40], a widely adopted and performant teleoperation hardware as an example, cutting a piece of tape and sticking it to a box takes more than 15 seconds, and putting a velcro shoe on a foot takes more than 22 seconds. In contrast, humans can perform either of these tasks in under 5 seconds, so simply matching the speed of demonstrations won’t result in “human-level” performance. Moreover, many real-world tasks are dynamic, requiring precise coordination of actions over time. For instance, tasks like tossing and pouring demand fine velocity control, while actions such as placing a tea bag into a cup require careful timing to synchronize with the bag’s swing. Consequently, naive approaches—such as increasing the underlying speed of the robot by a constant multiplier—are insufficient, as they fail to account for the nuanced relationship between speed and task success. An ideal speed-up system for imitation learning would (1) not require modifications to demonstration collection mechanisms, (2) retrofit existing policies to work at faster speeds, and (3) learn the interaction between execution speed and policy success. Modeling the relationship between speed and performance is especially important in dynamic tasks, where the speed may need to be increased or decreased depending on the stage of the task. For example, when pulling a tea-bag out of a cup, most of the episode can be accelerated, but grasping the tag of the tea-bag must be timed with the swing of the string, as shown in Fig. LABEL:fig:teaser. We present SpeedTuning, a method for accelerating manipulation policies by learning a “speed-policy” that predicts a speed factor for a chunk of future actions. The speed-policy is built on top of a base imitation learning policy. As actions are predicted in the environment, the speed policy takes in the observation history and outputs the velocity of the subsequent action. The speed policy is trained with reinforcement learning and optimizes a task and speed reward. To make this practical and compatible with modern imitation learning methods, most of which predict action chunks at:t+ka_t:t+k, we linearly interpolate predicted chunks by the factor predicted by the speed policy. Because the speed policy is trained entirely with reinforcement learning, this acceleration process requires no extra data-collection. We evaluate SpeedTuning across a diverse set of dynamic tasks, including pouring, throwing, and picking, in both simulated and real-world environments. We compare our method against reinforcement learning from scratch and a naive speed-up baseline that uniformly accelerates actions. The results demonstrate that SpeedTuning achieves substantial speed-ups (over 2.4x) compared to the baseline imitation-learning policies. Additionally, we conduct ablation studies to analyze the impact of various design choices, such as the choice of task policy, the use of image observations, and the specific image encoder employed, further validating the effectiveness and robustness of our approach. I Related Work Imitation learning. Imitation learning provides a straightforward way to acquire robotic manipulation skills by learning from expert demonstrations [23, 24]. Our work builds on a recent paradigm for imitation learning that combines a puppeteering setup [12, 16, 26, 33], transformer architecture [30], and action chunking to learn dexterous skills [40, 4]. Recent advancements in imitation learning highlight the surprising effectiveness of imitation learning for rapid skill acquisition [6, 14, 4]. However, the quality of learned skills remains inherently tied to the quality of demonstrations, which can be limited by unintuitive teleoperation interfaces and demonstrator sub-optimality [19]. Furthermore, the current paradigm lacks mechanisms to modify demonstrations or learn policies post-hoc that better optimize for task demands. We focus specifically on optimizing for execution speed and introduce a novel framework to accelerate existing imitation-learned policies for higher-speed manipulation tasks without requiring additional demonstrations or sacrificing performance. Reinforcement learning in robotics. A reinforcement learning (RL) agent takes actions in an environment to maximize cumulative reward [32, 28, 29, 18]. RL in robotic environments [20, 7, 1] evades the issues of sub-optimal demonstration data, but state-of-the art RL algorithms still require extensive real-world interactions to learn complex control policies from scratch [13]. Roboticists have successfully fine-tuned policies pre-trained with imitation learning [15, 37]. Unlike these works, we train a new policy that predicts the velocity at which the base policy runs. Fast robot manipulation. The pursuit of fast robot manipulation, particularly in dynamic tasks, has traditionally centered around optimizing the robot’s hardware or integrating speed into the reward function. These approaches have been successful in highly dynamic scenariossuch as table tennis [3, 5]. Dynamic manipulation tasks, such as cable manipulation [34, 35, 36, 39], throwing objects [38, 8], and high-throughput pick-and-place [21, 9, 22] also benefit from speed-focused objectives and hardware solutions. Unlike these works, SpeedTuning is compatible with many base policy learning methods, which means that it can be incorporated on top of already performant robotic policies. I Speed Tuning We introduce SpeedTuning, a method for accelerating manipulation policies learned from demonstrations while optimizing for both execution speed and task success. We achieve this by decoupling the policy into two components: a task policy trained with imitation learning to mimic expert demonstrations, which is summarized in Section I-A, and a speed policy trained with reinforcement learning to predict optimal action velocities that balance speed and success. The SpeedTuning objective is described in Section I-B. We then present a baseline approach in Section I-F that uniformly accelerates actions to establish the empirical trade-off between speed and success. In Section I-E, we detail how SpeedTuning uses RL to predict the velocity of executed actions based on current state to achieve a better combined success and speed than naive speed-up techniques. I-A Preliminaries We assume access to a set of expert demonstrations =(τ1,…,τN)D=\( _1,…, _N)\ where τn _n is a trajectory defined as a sequence of state-action pairs: τ=[(s1,a1)…(sT,aT)]τ=[(s_1,a_1)…(s_T,a_T)]. Under the standard imitation learning paradigm, learning a policy, π, parameterized by θ is a matter of minimizing the below loss function to maximize the likelihood of the expert actions: Lθ L_θ =[d(πθ(si),ai)] =E_D [d( _θ(s_i),a_i) ] (1) where d is any distance metric, usually Manhattan or Euclidean. These demonstrations are considered “expert” in that they are sampled from a behavior policy optimizing for task reward, rtaskr_task, which we assume to be a binary reward indicating whether the task was completed during a given transition. This reward signal can be derived either from the structure of the environment or through human-in-the-loop feedback, particularly in real-world reinforcement learning scenarios where automated reward formulation is insufficient. rtask(st,at) r_task(s_t,a_t) =1st+1∈Ssuccess =1_\s_t+1∈ S_success\ (2) Absent from this framework is an understanding of how to speed up policy execution. Typically, we assume actions are executed at a constant velocity, v. In the next section, we consider an updated objective where the velocity of actions is a component of the action space. Algorithm 1 SpeedTuning Training 1:Given: Demonstration dataset taskD_ task, speed reward function rspeed(v)r_speed(v) speed reward weight α, frame skip constant kskipk_skip, frame stack constant kstackk_stack, action chunk size kck_c 2:Initialize task policy πθ(at:t+kc|st) _θ(a_t:t+k_c|s_t) 3:Initialize speed policy πφ(vt|st) _ (v_t|s_t) 4:Initialize replay buffer replayD_replay 5:Initialize observation buffer O 6:Train the task policy πθ(at:t+kc|st) _θ(a_t:t+k_c|s_t) using the task dataset taskD_task 7:for each episode do 8: Clear observation buffer =[]O=[] 9: for each timestep t do 10: Obtain current observation s(t,0)s_(t,0) 11: Append current observation to buffer =∪s(t,0)O=O∪ s_(t,0) 12: Predict the speed vt=πφ([−kstack:])v_t= _ (O[-k_stack:]) 13: Predict the task action sequence At:=at:t+kc=πθ(s(t,0))A_t:=a_t:t+k_c= _θ(s_(t,0)) 14: Interpolate the action based on speed a(At,vt)a^(A_t,v_t) 15: for iteration i=1,2,…,kskipi=1,2,…,k_skip do 16: Apply interpolated action ai′:=ai(At,vt)a _i:=a_i^(A_t,v_t) 17: Obtain next observation s(t,i)s_(t,i) 18: Append new observation to buffer =∪s(t,i)O=O∪ s_(t,i) 19: Obtain task reward rtask(s(t,i−1),ai′)r_task(s_(t,i-1),a _i) 20: Compute the combined reward: rST=α⋅rspeed(vt)+rtask(s(t,i−1),ai′)r_ST=α· r_speed(v_t)+r_task(s_(t,i-1),a _i) 21: Store transition in the replay buffer: replay=replay∪(s(t,i−1),ai′,rST,s(t,i))D_replay=D_replay∪(s_(t,i-1),a _i,r_ST,s_(t,i)) 22: end for 23: end for 24: Sample mini-batches from the replay buffer replayD_replay to update the speed policy πφ(vt|st) _ (v_t|s_t) 25:end for I-B SpeedTuning Objective To extend our framework to account for action execution speed, we introduce an updated objective function that incorporates velocity as a component of the action space. We define the objective J(ψ)J(ψ) as the expected cumulative reward over trajectories sampled from a policy πψ(at,vt|st) _ψ(a_t,v_t|s_t), where the reward at each time step t consists of a weighted speed reward α⋅rspeed(vt)α· r_speed(v_t) and the task reward rtask(st,at)r_task(s_t,a_t). We restrict the selection of velocities to a discrete set: V=v1,v2,…,vKV=\v_1,v_2,…,v_K\ to simplify policy optimization. J(ψ) J(ψ) =τ∼πψ[∑t=1T(α⋅rspeed(vt)+rtask(st,at))] =E_τ _ψ [ _t=1^T (α· r_speed(v_t)+r_task(s_t,a_t) ) ] (3) By introducing the weighting term α≥0α≥ 0 to the velocity reward, we can adjust the trade-off between execution speed and task success. This allows us to describe the Pareto curve of optimality, optimizing for either faster execution (higher α) or higher task success rate (lower α). I-C Factorizing Task and Speed We decompose the execution policy into two distinct sub-policies: a task policy, πθ(at|st) _θ(a_t|s_t), and a speed policy, πφ(vt|st) _ (v_t|s_t). This not only makes optimizing for speed and success more tractable, but also enables us to accelerate any parameterization of πθ _θ. In our method, the task policy is trained to regress actions with imitation learning. For the speed policy, we explore both constant and non-linear speed policies, which are described in the next two sections. I-D Constant Velocity In scenarios where the velocity v is constant, the objective simplifies to: J(ψ) J(ψ) =α⋅vT+τ∼πψ[∑t=1Trtask(st,at)]. =α· vT+E_τ _ψ [ _t=1^Tr_task(s_t,a_t) ]. (4) This formulation enables us to explore different points along the Pareto frontier by varying the weighting term α, thereby balancing the trade-off between speed and task performance. However, in practice, more complex non-linear speed reward functions calculated at each transition are often preferred, which we discuss in the next section. Figure 1: Initialization and key actions for three real-world tasks (top to bottom): Tea Bag Disposal, Food Preparation, and Almond Pouring. I-E SpeedTuning: Learning a Speed-Adaptive Policy While uniformly increasing the velocity provides a simple method for accelerating action sequences, it lacks adaptability to different task contexts. Our aim is to develop a more general closed-loop policy where acceleration is conditioned on the current state. For instance, in a pick-and-place scenario, an optimal speed policy would likely slow down during critical actions, such as grasping or placing, to ensure successful execution, and speed up during less critical movements to maximize overall efficiency. To achieve this, SpeedTuning seeks to balance swift task execution with maintaining a high success rate. Specifically, we use reinforcement learning with the objective in Equation 3 to train a speed policy πφ(vt|st) _ (v_t|s_t), which predicts an optimal speed vtv_t for a fixed task policy πθ _θ: J(φ)=at∼πθvt∼πφ[∑t=1T(α⋅rspeed(vt)+rtask(st,at))] J( )=E_ subarrayca_t _θ\\ v_t _ subarray [ _t=1^T (α· r_speed(v_t)+r_task(s_t,a_t) ) ] (5) To optimize the objective in Equation 5 we employ Rainbow DQN [11], which is a value-based off-policy algorithm for discrete action spaces, because of its strong performance and high sample efficiency. Adapting the Rainbow DQN objective for our SpeedTuning gives the following optimal speed policy and Bellman equation: πφ∗(st,at) π^*_ (s_t,a_t) =argmaxvtQ∗(st,at,vt), = _v_tQ^*(s_t,a_t,v_t), (6) rST(st,at,vt) r_ST(s_t,a_t,v_t) =α⋅rspeed(vt)+rtask(st,at), =α· r_speed(v_t)+r_task(s_t,a_t), (7) Qπφ(st,at,vt) Q _ (s_t,a_t,v_t) =rST(st,at,vt)+ =r_ST(s_t,a_t,v_t)+ γst+1[Qπφ(st+1,at+1,vt+1)], γ\,E_s_t+1 [Q _ (s_t+1,a_t+1,v_t+1 ) ], (8) where γ is the discount factor. We made several careful design choices for the speed network. As described in Section I-B, we limit the output speed v to a discrete set of possible values. We also introduce an additional hyperparameter, β, to further scale the speed reward: rspeed(v)=vβr_speed(v)=v^β. To manage complexity in long-horizon tasks, we use a frame skip constant kskip∈ℤ+k_skip ^+, querying the speed policy only once every kskipk_skip steps, while continuing executing the accelerated policy with the latest speed between queries. To better preserve the Markovian properties, we adopt a frame stack of kstackk_stack recent observations as input. However, the most important design decision we make is to predict an action chunk at each time step. This makes the implementation of the velocity change more straightforward and reduces the effective horizon of the task. We describe the concrete interpretation of velocity when predicting action chunks in the next section. I-F Interpolation over Action Chunks Instead of manually scaling the velocity at which actions are run, we assume that the base policy predicts chunks of actions of size k for joint positions: πθ(At|st) _θ(A_t|s_t) where At=at:t+kA_t=a_t:t+k. To modulate the execution speed of actions generated by the task policy, we employ linear temporal interpolation. With linear temporal interpolation, the i-th accelerated action at speed v for time step t is denoted as ai(At,v)a^(A_t,v)_i, with a1(At,v)a^(A_t,v)_1 selected as the next action to be executed at the current time step. Formally, we define linear temporal interpolation over a function f:ℤ↦ℝnf:Z ^n with a step length of v at point t as: interpf,v(t)=f(⌊vt⌋)+vt−⌊vt⌋v(f(⌊vt+1⌋)−f(⌊vt⌋)). _f,v(t)=f ( vt )+ vt- vt v (f ( vt+1 )-f ( vt ) ). (9) This linear temporal interpolation effectively accelerates the execution of the function f by a factor of v. For simplicity, we denote a trajectory of actions as A=a0:kA=a_0:k. The temporal action function fa(A):ℤ↦ℝnf^(A)_a:Z ^n is defined as: fa(A)(i)=ai(A),0≤i<k+1.f^(A)_a(i)=a^(A)_i, 0≤ i<k+1. (10) The accelerated action at speed v over the trajectory A at step i is then: ai(A,v)=interpfa(A),v(i),0≤i≤⌊k+1v⌋.a^(A,v)_i=interp_f^(A)_a,v(i), 0≤ i≤ k+1v . (11) Figure 2: Results showing the SpeedTuning policy on simulated (top row) and real world (bottom row) tasks. For simulated tasks, the baseline curve is obtained by evaluating 100 episodes across various speeds at 0.1 intervals, while the distribution and mean of the SpeedTuning policy is calculated over 2000 episodes. For the real-world tasks, the baseline curve is obtained by evaluating 20 episodes across various speeds at 0.25 intervals, while the mean and error of the SpeedTuning policy is calculated over 20 episodes. Due to the sample efficiency constraints of reinforcement learning in real-world settings, the policy was not trained to full convergence, which may account for the slightly lower observed success rate. Figure 3: Comparison of the SpeedTuning and vanilla ACT policy trajectories for the Almond Pouring task, featuring key action captures and speed-up values throughout the task. The results highlight SpeedTuning’s task understanding, significantly accelerating overall execution by 2.5x (200 time steps vs. 500 time steps), while deliberately slowing down during the critical, dynamic pouring phase. In summary, this interpolation method allows us to integrate the speed policy’s output, v, with the action chunks generated by a task policy by decreasing the effective chunk-size down to k+1v k+1v. IV Experiments In this section, we present a comprehensive evaluation of SpeedTuning to demonstrate its effectiveness in accelerating policy execution. In the following sections, we discuss the implementation details and describe our simulated and real-world dynamic manipulation tasks. Then, we present our key results, which show that SpeedTuning achieves significant speed-ups while maintaining solid policy performance. Finally, we ablate the design choices such as the base action policy and the parameterization of the reward function. Overall, we find that our method robustly produces the policy with the best tradeoff between speed and success. IV-A Implementation Details Imitation Learning Details. Action Chunking with Transformers (ACT) [40] models the policy distribution p(At|st)p(A_t|s_t) as a transformer-based Conditional Variational Autoencoder (CVAE) [17, 27]. The decoder outputs a chunk of actions of size kck_c. Chunking is particularly important for high-frequency, fine-grained manipulation tasks, with chunk sizes kck_c being as high as 100 [40]. 3) The input observations oto_t includes both proprioception joint positions and camera image embedding through a pretrained image encoder, which enables the algorithm to be effectively used on real robots while not adding too much computing overhead. RL details RainbowDQN [11] models the Q function in discrete action space Q(qt|st,at)Q(q_t|s_t,a_t), enhancing DQN with several improvements: double Q-learning [10] reduces overestimation bias, prioritized experience replay [25] increases sampling efficiency, and the dueling architecture [31] improves performance in environments where action values are state-independent. Additionally, distributional Q-learning [2] captures reward uncertainty by modeling a categorical distribution over Q-values, making it well-suited for high-dimensional, dynamic tasks in fine-grained manipulation. Figure 4: Ablation study on task policy, speed rewards, and frame skip values. All configurations are evaluated over 2000 episodes on Simulated Tea Bag Transfer. IV-B Robot Setup and Tasks We conducted all our experiments based on the ALOHA setup for simulated and real-world tasks. The bimanual hardware setup operates at 50Hz, consists of a 14-dimensional action space, and the observation states include proprioception (joint position and speed) and 4 RGB cameras. We conducted evaluations across six fine-grained manipulation tasks, divided equally between simulated and real-world environments. These tasks encompass long-horizon, contact-rich, and dynamic manipulation scenarios, reflecting the complexity of daily robotic applications. Simulated Cube Transfer: This task requires the robot to pick up and transfer a cube between two grippers, assessing the system’s ability to handle basic object manipulation. Simulated Peg Insertion: In this task, the robot must pick up and accurately insert a peg into a cube with a hole, demanding precise and bimanual manipulation. Simulated Tea Bag Transfer: This task involves picking up and transferring a tea bag from a desktop into a cup. The dynamic nature of this task, including the swinging motion of the tea bag during transfer, presents significant challenges for our approach. Tea Bag Disposal: Extending the simulated tea bag transfer to the real world, this task requires more precise bimanual and dynamic manipulation. The robot must first pick up a mug containing a tea bag and then carefully dispose of the tea bag into a garbage bin. Food Preparation: This task involves a sequence of precise maneuvers: picking up a plate with one gripper, picking up and throwing two pieces of chicken from the desktop onto the plate, and finally placing the served food at a designated location on the desktop. Pouring Almond: This highly precise and dynamic task involves opening the lid of a container and pouring almonds from a can into the container. IV-C Key Results SpeedTuning significantly accelerates the policy. As illustrated in Fig. 2 and 2, SpeedTuning achieves over a 2.4×2.4× speed-up across all six tasks compared to the original ACT task policy. We observed a trend where higher speed-ups are attained in tasks requiring less precision, such as Simulated Cube Transfer and Simulated Tea Bag Transfer. This demonstrates that achieving higher speed-ups necessitates a substantial level of precision in task execution. SpeedTuning effectively retains the success rate at high speed. While accelerating the base policy, SpeedTuning maintains a high success rate even at increased speeds, significantly outperforming the universal interpolation baseline. Notably, SpeedTuning emerges as an outlying point beyond the Pareto curve defined by the baseline. This trend is particularly pronounced in more dynamic tasks, such as Simulated Tea Bag Transfer, real-world Tea Bag Disposal, and Almond Pouring. This highlights that naive acceleration, as in the baseline, is insufficient for dynamic tasks, requiring a deeper understanding of task-specific dynamics—such as selecting the appropriate speed for transferring a swinging tea bag—something SpeedTuning effectively accomplishes through reinforcement learning. SpeedTuning learns the critical parts of the tasks to act accordingly. To achieve high acceleration while maintaining success rates, SpeedTuning intelligently segments the task based on dynamic and contact-rich phases, adjusting its speed accordingly. For instance, as shown in Fig. 3, the resulting policy for Almond Pouring demonstrates this capability: SpeedTuning learns to maintain a high speed during the lid-opening and can-picking phases, deliberately slows down during the transfer and pouring stages to prevent spilling, and then resumes high speed to place the can back. V Ablations In this section, we conduct ablation studies on various design choices for SpeedTuning. All experiments are carried out using Simulated Tea Bag Transfer due to its precise and dynamic nature, as well as its efficiency in simulation. Choice of Task Policy: Replacing the learned ACT task policy in SpeedTuning with an open-loop, scripted policy eliminates the learned generalization capabilities of the task policy. As shown in Fig. 4, the scripted policy results in lower acceleration and a slightly higher success rate. This underscores the importance of generalizing across various speeds for the underlying task policy in SpeedTuning. Image Observations: The speed policy uses both proprioceptive and visual observations for speed predictions. To evaluate whether proprioceptive data alone suffices to represent the environment state, we removed the image input from the policy. Shown in Fig. 4, omitting image observations led to less stable training and lower success rates and acceleration. These results highlight the necessity of incorporating visual information for more efficient task execution. Degree of Speed Rewards: The speed reward is formulated as rspeed(v)=vβr_speed(v)=v^β. As shown in Fig. 4, varying the degree of the speed reward, β, has a significant impact on SpeedTuning performance. When β is large, the speed policy prioritizes velocity over task completion, resulting in lower success rates. Conversely, when β=0β=0, there is no direct incentive for speed, and acceleration relies solely on the standard discount factor γ, leading to unstable training. Frame Skip: In SpeedTuning, a frame skip of 10 is used to shorten the horizon of the MDP process. While smaller frame skips provide finer granularity, the extended horizon diminishes the emphasis on terminal success rewards. Conversely, larger frame skips improve short-horizon properties but reduce control precision over speed adjustments. As shown in Fig. 4, smaller frame skips with longer horizons lead to suboptimal speed policies, while larger skips sacrifice the granularity needed for achieving higher acceleration. VI CONCLUSION We introduced SpeedTuning, a reinforcement learning framework designed to optimize both execution speed and task success for learned manipulation policies. By training a speed policy to dynamically adjust action velocities based on the current state, SpeedTuning achieved over 2.4x speed-ups across various dynamic and precise tasks while maintaining adequate success rates compared to baseline methods. Overall, we hope our solution provides valuable insights into the relationship between execution speed and task success in robotic manipulation. Future work should focus on enhancing sample efficiency to make the method more practical for real-world applications. References [1] M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba (2018) Learning dexterous in-hand manipulation. The International Journal of Robotics Research 39, p. 20 – 3. Cited by: §I. [2] M. G. Bellemare, W. Dabney, and R. Munos (2017) A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 449–458. Cited by: §IV-A. [3] D. Büchler, S. Guist, R. Calandra, V. Berenz, B. Scholkopf, and J. Peters (2020) Learning to play table tennis from scratch using muscular robots. IEEE Transactions on Robotics 38, p. 3850–3860. Cited by: §I. [4] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137. Cited by: §I. [5] D. B. D’Ambrosio, S. Abeyruwan, L. Graesser, A. Iscen, H. B. Amor, A. Bewley, B. J. Reed, K. Reymann, L. Takayama, Y. Tassa, K. Choromanski, E. Coumans, D. Jain, N. Jaitly, N. Jaques, S. Kataoka, Y. Kuang, N. Lazic, R. Mahjourian, S. Moore, K. Oslund, A. Shankar, V. Sindhwani, V. Vanhoucke, G. Vesom, P. Xu, and P. R. Sanketi (2024) Achieving human level competitive robot table tennis. Cited by: §I. [6] P. R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. S. Wong, J. Lee, I. Mordatch, and J. Tompson (2021) Implicit behavioral cloning. ArXiv abs/2109.00137. Cited by: §I. [7] S. S. Gu, E. Holly, T. P. Lillicrap, and S. Levine (2016) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA), p. 3389–3396. Cited by: §I. [8] H. Ha and S. Song (2021) FlingBot: the unreasonable effectiveness of dynamic manipulation for cloth unfolding. ArXiv abs/2105.03655. Cited by: §I. [9] S. D. Han, S. W. Feng, and J. Yu (2019) Toward fast and optimal robotic pick-and-place on a moving conveyor. IEEE Robotics and Automation Letters 5, p. 446–453. Cited by: §I. [10] H. v. Hasselt, A. Guez, and D. Silver (2016) Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, p. 2094–2100. Cited by: §IV-A. [11] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver (2018) Rainbow: combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §I-E, §IV-A. [12] T. Hulin, K. Hertkorn, P. Kremer, S. Schätzle, J. Artigas, M. Sagardia, F. Zacharias, and C. Preusche (2011) The dlr bimanual haptic device with optimized workspace. 2011 IEEE International Conference on Robotics and Automation, p. 3441–3442. Cited by: §I. [13] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine (2021) How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research 40, p. 698 – 721. Cited by: §I. [14] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn (2022) BC-z: zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, Cited by: §I. [15] R. C. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman (2020) Never stop learning: the effectiveness of fine-tuning in robotic reinforcement learning. In Conference on Robot Learning, Cited by: §I. [16] B. Katz (2018) A low cost modular actuator for dynamic robots. Cited by: §I. [17] D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §IV-A. [18] V. R. Konda and J. N. Tsitsiklis (1999) Actor-critic algorithms. In Neural Information Processing Systems, Cited by: §I. [19] A. Kumar, J. Hong, A. Singh, and S. Levine (2022) Should i run offline reinforcement learning or behavioral cloning?. In International Conference on Learning Representations, External Links: Link Cited by: §I. [20] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. M. O. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2015) Continuous control with deep reinforcement learning. CoRR abs/1509.02971. Cited by: §I. [21] V. Nabat, M. de la O Rodríguez, O. Company, S. Krut, and V. Pierrot (2005) Par4: very high speed parallel robot for pick-and-place. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 553–558. Cited by: §I. [22] H. Pham and Q. Pham (2018) Critically fast pick-and-place with suction cups. 2019 International Conference on Robotics and Automation (ICRA), p. 3045–3051. Cited by: §I. [23] D. A. Pomerleau (1988) ALVINN: an autonomous land vehicle in a neural network. In NIPS, Cited by: §I. [24] S. Schaal (1999) Is imitation learning the route to humanoid robots?. Trends in Cognitive Sciences 3, p. 233–242. External Links: Link Cited by: §I. [25] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2015) Prioritized experience replay. CoRR abs/1511.05952. External Links: Link Cited by: §IV-A. [26] M. Schwarz, C. Lenz, A. Rochow, M. Schreiber, and S. Behnke (2021) NimbRo avatar: interactive immersive telepresence with force-feedback telemanipulation. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 5312–5319. Cited by: §I. [27] K. Sohn, H. Lee, and X. Yan (2015) Learning structured output representation using deep conditional generative models. Advances in neural information processing systems 28. Cited by: §IV-A. [28] R. S. Sutton and A. G. Barto (1998) Reinforcement learning - an introduction. Adaptive computation and machine learning, MIT Press. External Links: ISBN 978-0-262-19398-6 Cited by: §I. [29] R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour (1999) Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems, Cited by: §I. [30] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. ArXiv abs/1706.03762. Cited by: §I. [31] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas (2016) Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 1995–2003. Cited by: §IV-A. [32] C. Watkins and P. Dayan (1992) Q-learning. Machine Learning 8, p. 279–292. Cited by: §I. [33] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel (2023) GELLO: a general, low-cost, and intuitive teleoperation framework for robot manipulators. Cited by: §I. [34] Y. Yamakawa, A. Namiki, M. Ishikawa, and M. Shimojo (2009) Knotting manipulation of a flexible rope using a high-speed multifingered hand and high-speed visual and tactile sensory feedback. Journal of the Robotics Society of Japan 27, p. 1016–1024. Cited by: §I. [35] Y. Yamakawa, A. Namiki, and M. Ishikawa (2010) Motion planning for dynamic knotting of a flexible rope with a high-speed robot arm. 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 49–54. Cited by: §I. [36] Y. Yamakawa, A. Namiki, and M. Ishikawa (2013) Dynamic high-speed knotting of a rope by a manipulator. International Journal of Advanced Robotic Systems 10. Cited by: §I. [37] J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn (2023) Robot fine-tuning made easy: pre-training rewards and policies for autonomous real-world reinforcement learning. 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 4804–4811. Cited by: §I. [38] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser (2019) TossingBot: learning to throw arbitrary objects with residual physics. Ieee Transactions On Robotics. External Links: Document Cited by: §I. [39] H. Zhang, J. Ichnowski, D. Seita, J. Wang, and K. Goldberg (2020) Robots of the lost arc: learning to dynamically manipulate fixed-endpoint ropes and cables. ArXiv abs/2011.04840. Cited by: §I. [40] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. RSS. Cited by: §I, §I, §IV-A. [Simulation Rollouts and Additional Ablations] -A Simulated Rollout Progressions Figure 5 provides a qualitative view of SpeedTuning on the three simulated tasks. Cube Transfer proceeds from grasping to a bimanual handoff; Peg Insertion requires grasping, alignment, and insertion; and Tea Bag Transfer requires transporting the suspended tea bag into the receptacle. All frames are taken from the original evaluation rollouts. Figure 5: Representative SpeedTuning rollouts in simulation. Rows show Cube Transfer, Peg Insertion, and Tea Bag Transfer; columns show the initial state followed by four successive task stages. -B Task Policy The following plots show the original training curves underlying the endpoint ablations in Fig. 4; all configurations are evaluated on Simulated Tea Bag Transfer. In Fig. 6, learned ACT supports substantially higher acceleration, whereas the scripted policy maintains a higher success rate. Figure 6: Base-policy ablation: acceleration (left) and success rate (right). Learned ACT is compared with an open-loop scripted policy. -C Image Observations Removing image observations produces high nominal acceleration but a much lower success rate (Fig. 7). Visual input is therefore necessary for identifying task phases in which aggressive acceleration would compromise completion. Figure 7: Visual-observation ablation: acceleration (left) and success rate (right). Both variants retain proprioception. -D Speed-Reward Exponent The exponent β controls how strongly the reward favors increased speed. Figure 8 shows that β=2β=2 provides the strongest combined acceleration and final success; β=0β=0 is less stable, while β=1β=1 and β=3β=3 favor lower-acceleration operating points. Figure 8: Speed-reward ablation: acceleration (left) and success rate (right). The main experiments use β=2β=2. -E Frame Skip Frame skip fsf_s sets the interval between speed-policy decisions. In Fig. 9, fs=5f_s=5 favors acceleration, fs=25f_s=25 favors success, and the selected value fs=10f_s=10 balances the two objectives. Figure 9: Frame-skip ablation: acceleration (left) and success rate (right). The main experiments use fs=10f_s=10. -F Image Encoder All three encoders reach similar final acceleration (Fig. 10). The randomly initialized encoder learns success more slowly, while the pretrained ResNet-18 attains the highest final success, showing that the speed policy need not reuse the task-policy encoder. Figure 10: Image-encoder ablation: acceleration (left) and success rate (right).