Paper deep dive
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation
Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 6/21/2026, 2:38:15 AM
Summary
This study benchmarks four action-space representations (pose increment, pose velocity, joint position increment, and joint velocity) for vision-based robotic manipulation tasks (object picking and pushing) using a Franka Emika Panda robot. The research evaluates these spaces in both simulation (MuJoCo) and real-world settings via sim-to-real transfer. The findings demonstrate that the choice of action space significantly impacts motion smoothness and sim-to-real performance, with joint velocity being identified as the optimal representation for both the PandaPickCuboid and PandaPushCuboid tasks.
Entities (10)
Relation Signals (3)
joint velocity → isbestfor → PandaPickCuboid
confidence 100% · we find that the joint velocity action space is best for the vision-based picking and pushing tasks
PPO → usedtotrain → PandaPickCuboid
confidence 100% · Policies are trained in simulation using PPO and evaluated on a physical Franka Emika Panda robot via sim-to-real transfer.
MuJoCo → simulates → Franka Emika Panda
confidence 90% · We utilize MuJoCo [24] to train our policies in simulation. ... We study this problem using two vision-based manipulation tasks on the Franka Emika Panda robot arm.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.
Tags
Links
- Source: https://arxiv.org/abs/2606.18594v1
- Canonical: https://arxiv.org/abs/2606.18594v1
Trouble viewing inline? Open PDF directly →
Full Text
48,107 characters extracted from source content.
Expand or collapse full text
Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation Seyed Alireza Azimi1,∗ Homayoon Farrahi1 Abhishek Naik2 Colin Bellinger3,4,† A. Rupam Mahmood1,5,6 1Department of Computing Science, University of Alberta 2National Research Council Canada 3School of Electrical Engineering and Computer Science, University of Ottawa 4Vector Institute 5Alberta Machine Intelligence Institute (Amii) 6Canada CIFAR AI Chair †This work was conducted while at the National Research Council Canada. ∗Corresponding author: sazimi@ualberta.ca Demo video: https://youtu.be/MmXEexVRa18 Code: https://github.com/RL-Sim-to-Real/training Abstract In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments. I Introduction Reinforcement learning (RL) is an effective framework for learning control policies for robotic tasks. Despite this, RL has faced challenges with real-world practicality, safety, and motion smoothness in robotic manipulation [10, 11, 14]. A central design choice in any RL formulation is the action space, which has been shown to affect factors such as motion smoothness, sample efficiency, and sim-to-real transfer performance [1, 7, 12]. Despite the importance of action-space design, prior work has largely conducted this study without vision-based observations and has often limited its study to simulation [11, 1, 26, 12]. Vision-based manipulation introduces partial observability and perceptual noise, which can substantially alter learning dynamics and real-world behavior. These challenges are further amplified in sim-to-real transfer, where mismatches in dynamics and sensing can expose sensitivities that remain hidden in idealized simulation environments. In this work, we investigate the impact of action-space design in vision-based robotic manipulation. Vision provides rich sensory input, enabling robots to perceive and interact with their environment. We study this problem using two vision-based manipulation tasks on the Franka Emika Panda robot arm: PandaPickCuboid and PandaPushCuboid. PandaPickCuboid is a prehensile manipulation task in which the agent must pick up a red cuboid using visual input from a wrist-mounted camera. Object picking is a fundamental robotic manipulation task [28, 27, 9, 8] that naturally decomposes into reaching, grasping, and lifting, making it both challenging and important. PandaPushCuboid is a non-prehensile manipulation task in which the agent must push the same cuboid within a specified boundary. Pushing is another core manipulation task that enables object displacement without grasping and involves interactions with friction and mass [14, 1]. Policies are trained in simulation using PPO and evaluated on a physical Franka Emika Panda robot via sim-to-real transfer. Simulation training allows us to safely exploit the exploratory nature of RL, particularly in contact-rich tasks, while enabling rapid experimentation and large-scale data collection. To mitigate discrepancies between simulated and real-world dynamics and visual appearance, we employ domain randomization [28, 23]. We adopt PPO due to its compatibility with vision-based actor–critic architectures [18, 5], and demonstrated effectiveness in sim-to-real robotic learning [12, 1, 21, 2, 13]. Additionally, PPO’s ability to leverage large numbers of parallel simulation environments makes it time-efficient for training. We compare four commonly used action-space representations: pose increment, pose velocity, joint velocity, and joint position increment. Our results show that the choice of action space affects final sim-to-real performance, and that joint velocity performs best in terms of smoothness and final task performance. I Related Work Action-space design plays a critical role in reinforcement learning for robotic manipulation, influencing learning efficiency, motion quality, and robustness to sim-to-real transfer. A growing body of work has investigated how different action representations affect policy learning and performance across a variety of robotic tasks and embodiments. Aljalbout et al. [1] conducted a comprehensive study of action-space design for sim-to-real robotic manipulation, comparing action representations across joint-space, Cartesian-space, and incremental formulations. Using Proximal Policy Optimization (PPO), they trained policies in simulation on a Franka Emika Panda robot and evaluated them on reaching and pushing tasks, considering metrics such as episodic return, constraint violations, task accuracy, and trajectory error. The authors found that action-space choice significantly affects both learning dynamics and transfer performance, with joint-velocity control achieving the strongest overall sim-to-real results while requiring minimal actuator tuning. Our work builds on these findings by extending the evaluation of action spaces to vision-based manipulation and contact-rich picking and pushing tasks. Martín-Martín et al. [12] examined the effect of action-space design on contact-rich manipulation in simulation, comparing joint torque, joint velocity, joint position, and Cartesian impedance control across path following, door opening, and surface wiping tasks. Using PPO, they demonstrated that action-space choice impacts sample efficiency, energy consumption, and safety, with Cartesian impedance actions performing favorably across tasks. While their study focuses on simulation-only settings with state-based observations, we extend this line of work to real-world evaluation via sim-to-real transfer and vision-based sensing. Similarly, Varin et al. [26] compared several action representations—including torque control, joint-space proportional–derivative (PD) control, inverse dynamics control, and task-space impedance control—across simulated manipulation tasks such as peg insertion, hammering, and pushing. Policies were trained using both PPO and Soft Actor-Critic (SAC), with task-space impedance control consistently achieving high performance and strong sample efficiency. In contrast to their simulation-focused evaluation, our study evaluates action spaces in the real world under partial observability induced by visual input. Beyond manipulation, action-space design has also been studied in the context of locomotion and control of articulated figures. Peng et al. [15] investigated action representations for planar articulated figures in simulation, comparing torque control, musculotendon unit activations, PD target angles, and joint velocity commands. They showed that action spaces incorporating intrinsic feedback—such as PD targets and velocity control—improve learning speed and robustness compared to direct torque control. However, their experiments were limited to two-dimensional simulated systems and did not include real-world evaluation. Chen et al. [3] demonstrated that torque control outperforms joint position control in quadruped locomotion tasks under varying gain parameters, although position-based control exhibited superior sample efficiency. Note that sample efficiency would primarily be a concern if it affects training time. Similarly, Kim et al. [7] showed that torque-based policies achieve stronger sim-to-real transfer for bipedal locomotion compared to position-based control, albeit at the cost of reduced sample efficiency. They mitigate this issue through pretraining with gravity compensation and highlight the drawback of position control in requiring careful gain tuning across tasks and embodiments—a challenge we also encountered. Schneider et al. [17] studied the effect of action representations on policy gradient methods across several simulated environments. For the Gymnasium Reacher task involving a simple two-joint robotic arm, they found that joint velocity control outperformed both torque and joint position control, with torque control performing the worst. Their results further demonstrate that action-space choice can substantially affect learning outcomes across a range of control tasks, including Pendulum and Walker-Walk [25]. I Sim-to-Real System overview In this section, we describe critical components in our sim-to-real pipeline. In particular, the learning algorithm, simulator, and the real-robot control system. I-A Proximal Policy Optimization Proximal Policy Optimization (PPO) is an on-policy deep reinforcement learning method designed on the idea of a clipped surrogate objective that confines policy updates to the boundaries of a trust region [18]: LCLIP():=π[min(ρt()Ht,clip(ρt(),1−ϵ,1+ϵ)Ht)], L^CLIP( θ):=E_π\! [\, \! ( _t( θ)\,H_t,\;clip\! ( _t( θ),1-ε,1+ε )H_t ) ], (1) where ρt()=π(At∣St,)π(At∣St,old) _t( θ)= π(A_t S_t, θ)π(A_t S_t, θ_old) and HtH_t is the generalized advantage estimate. We utilize PPO for three reasons: (1) it is a deep reinforcement learning algorithm well suited to continuous state and action spaces, as well as high-dimensional observations such as images; (2) it is widely adopted in prior sim-to-real work [12, 1, 21, 2, 13]; and (3) it is time efficient as it can leverage a large number of parallel environments and perform simple updates [18]. In our experiments, we use PPO implemented by [5]. I-B Dynamics-based Simulation We utilize MuJoCo [24] to train our policies in simulation. MuJoCo is a torque-based simulator that faithfully captures real-world dynamics through three important computational functions that are used sequentially: actuator, dynamics, and integrator. Actuators are functions that map actuation to torque ( τ), the dynamics function maps from applied torque to acceleration, and the integrator updates the state of the simulator by computing next positions and velocities (Fig. 1). Actuator. In our work, we consider two actuation models that are described as follows: fposac f^ac_pos :=p(′−)−v, (position actuator) := K_p( q - q)- K_v v, (position actuator) (2) fvelac f^ac_vel :=v(′−), (velocity actuator), := K_v( v - v), (velocity actuator), (3) where p K_p denotes the diagonal matrix of proportional gains or stiffness, and v K_v represents the diagonal matrix of damping coefficients. Here, ′ q and ′ v are the target joint position and velocity, while q and v are the current joint position and velocity. fposacf_pos^ac is a PD-controller with target velocity ′=0 v =0 and fvelacf_vel^ac is a P-controller. Dynamics Function. The dynamics function is then used to compute joint accelerations t a_t from applied torque and other forces in the system using the following equation: t=−1(+⊤−), a_t= M^-1( τ+ J f- c), (4) where M is inertia in joint space, τ is the applied force consisting of actuator torque, passive force, and external forces, f is the constraint force, and c is the bias force consisting of Coriolis, centrifugal, and gravitational forces. T J^T maps forces from the constraint space of the simulator to the joint space of the robot. Integrator. MuJoCo advances the simulator from time step t to t+1t+1 using one of several numerical integrators (semi-implicit Euler, implicit-in-velocity, fast implicit-in-velocity, or RK4). In all experiments, we use the fast implicit-in-velocity integrator, which provides a practical trade-off between stability and computational cost [24]. This integrator performs an implicit velocity update that can be written as t+1=t+δt^−1t,^:=−δt, v_t+1\;=\; v_t\;+\;δ t\, M^-1Ma_t, M:=M-δ t\,D, (5) where M is the joint-space inertia matrix and :=∂/∂D:= /∂ v. Under the fast implicit-in-velocity integration scheme, the velocity-dependent term is approximated as ≈∂,D≈ ∂ τ∂ v, where D denotes the Jacobian of the applied torques with respect to joint velocities. In all experiments, we employ MuJoCo’s fast implicit-in-velocity integrator. Joint positions are then updated via t+1=t+δtt+1. q_t+1\;=\; q_t\;+\;δ t\, v_t+1. (6) Figure 1: MuJoCo state transition. actionaction is the action produced by the policy. ′t q _t and ′t v _t are the target joint positions and velocities, respectively. t q_t and t v_t are the current joint positions and velocities. t τ_t is the applied torque and t a_t is the current acceleration. t+1 q_t+1, t+1 v_t+1 are the updated joint positions and velocities. I-C Real Robot Control System For real-world experiments, we control the Franka Emika Panda using the Franka ROS Interface via the Python API wrapper introduced by [20]. We utilize two actuation interfaces: set_joint_positions_velocities and set_joint_velocities. The set_joint_positions_velocities interface employs a joint impedance controller fimpac:=c+p(′−)+v(′−),f_imp^ac:= τ_c+ K_p( q - q)+ K_v( v - v), (7) where c τ_c denotes the Coriolis torque, q and ′ q are the current and target joint positions, v and ′ v are the current and target joint velocities, and p K_p and v K_v are diagonal matrices of stiffness and damping gains. Setting ′=0 v =0 yields fposac:=c+p(′−)−v,f_pos^ac:= τ_c+ K_p( q - q)- K_v v, (8) which closely matches the PD position controller used in simulation (Eq. 2). The main difference is the omission of c τ_c, which is implicitly handled by the dynamics function in MuJoCo (Eq. 4). The set_joint_velocities interface commands target joint velocities directly. Although its actuator model is not explicitly documented, it can be inferred to correspond to velocity P-control of the form fvelac:=c+v(′−).f^ac_vel:= τ_c+ K_v( v - v). The computed torque values are then mapped to electrical currents used to move the joint servos of the physical robot. Figure 2: Top row: PandaPickCuboid. Bottom row: PandaPushCuboid. IV Tasks We evaluate our action spaces on two tasks: PandaPickCuboid and PandaPushCuboid. PandaPickCuboid is a prehensile manipulation task in which the robot must reach, grasp, and lift a red cuboid to a minimum height of 17 cm. Object picking is a fundamental manipulation task that requires precise, multi-stage execution of reaching, grasping, and lifting. In simulation, the episode terminates upon successful lift or timeout. In the real world, episodes additionally terminate if the end-effector exits predefined workspace boundaries or exceeds 10 unsuccessful grasp attempts. PandaPushCuboid is a non-prehensile manipulation task where the agent must push the same red cuboid across a white base (Fig. 2) as much as possible, emphasizing directional control and interaction with object friction and mass. Episodes terminate after a 12 s timeout in both simulation and real-world settings, and additionally if the end-effector exits workspace boundaries in the real world. IV-A Reward Functions PandaPickCuboid Reward. The reward function we use to train the robot is as follows: rt r_t :=max0,gtsum−max1≤i≤t−1gisum, := \0\;,\;g_t^sum- _1≤ i≤ t-1g_i^sum \, (9) where gtsum:=∑i=14wi⋅gt(i)g_t^sum:= _i=1^4w_i· g^(i)_t is a weighted sum of the reward components with wiw_i being the weight for the iith component (Table I). This formulation ensures that if the reward at time step t is lower than the maximum reward observed in previous steps, the reward for that step is set to zero, effectively considering only improvements in the reward. This formulation was introduced by [16], and we adapted our implementation from [28]. TABLE I: Reward components for PandaPickCuboid. δtarget _target denotes the distance between the block and a fixed target position located vertically above its initial pose at 0.2m. δrot _rot is the block upright orientation error. δgripper _gripper is the gripper–block distance. hblockh_block is the block height. Reward W Description g(1)=1−tanh(5(0.9δtarget+0.1δrot))g^(1)=1- \! (5(0.9\, _target+0.1\, _rot) ) 4 Pose tracking g(2)=1−tanh(5δgripper)g^(2)=1- \! (5 _gripper ) 8 Gripper-block distance penalty g(3)=[no floor collision]g^(3)=I[no floor collision] 0.25 Collision penalty g(4)=[|hblock−0.2|<0.03]g^(4)=I[|h_block-0.2|<0.03] 2.0 Lift success PandaPushCuboid Reward. We utilize the following reward function to train policies for the PandaPushCuboid task: rt=5tinrtdisp− 0.5rtdist− 0.1tfloor,r_t=5\,I^in_t\,r^disp_t\;-\;0.5\,r^dist_t\;-\;0.1\,I^floor_t, (10) where rtdispr^disp_t denotes the planar displacement of the block at time step t. The indicator function tinI^in_t returns 11 if the block remains within the boundaries of the white base and 0 otherwise; this is to ensure displacement reward is only given when the block is within the boundaries of the base. The term rtdistr^dist_t is the Euclidean distance between the block and the end-effector, designed to encourage the agent to reach the block, while tfloorI^floor_t returns 11 if the end-effector makes contact with the floor and 0 otherwise. IV-B Observation Spaces The observation space comprises four components: (1) an RGB image of size 64×64×364× 64× 3, (2) a proprioception vector containing joint positions q, joint velocities v, and grasp status, (3) the end-effector height, and (4) the previous action. Joint positions and velocities are linearly scaled to [−1,1][-1,1] using =2−minmax−min−1 q=2 q- q_ q_ - q_ -1 and =2−minmax−min−1 v=2 v- v_ v_ - v_ -1. We empirically observed that scaling improved sim-to-real performance. For PandaPushCuboid, the grasp-status signal is omitted, as grasping is not required. IV-C Action Spaces The Franka Emika Panda has 7 joints, allowing for 7 degrees of freedom. In the PandaPickCuboid task, our action space has dimensionality 8, with the final dimension reserved for controlling the grasping mechanism. In the PandaPushCuboid task, the dimensionality is 7. All action values are bounded to the range (-1,1), and all action spaces have a cycle time of δt=40msδ t=40ms. We consider four action spaces: pose increment (δδ x), pose velocity (˙ x), joint velocities ( v), and joint position increments ( δ q). These action spaces were selected because they are agnostic to the absolute joint and end-effector positions, making them suitable for our partially observable setup and commonly used in robotic manipulation [28, 10, 1]. TABLE I: Action scaling factors for PandaPickCuboid (Pick) and PandaPushCuboid (Push) Pick Push Action Sim Real Sim Real Pose inc. (δδ x) 0.05 0.02 0.05 0.02 Pose vel. (˙ x) 0.05 0.01 0.05 0.01 Joint pos. inc. (δδ q) 0.05 0.06 0.05 0.05 Joint vel. ( v) 1.0 0.20 1.0 0.15 Joint Space. Actions are vector outputs from the policy, and the actuator gives them meaning. We define actuators as functions that map actuation to torque. In v, we utilize velocity actuators and map the target velocities scaled by a constant (c) to torque (fvelac:c.target↦f^ac_vel:c. v^target τ) (Table I). For δδ q, target joint positions are computed as target=+c.δ q^target= q+c.δ q and passed to a position actuator (fposac:target↦f_pos^ac: q^target τ). Cartesian Space. For our Cartesian action spaces, the δδ x representation is actuated using position actuators, whereas ˙ x applies incremental commands through velocity actuators. In the δδ x action space, the first three components correspond to position increments, which specify the desired change in the end-effector’s current position. The next three components correspond to rotational increments about the three Cartesian axes. The seventh component specifies the rotation increment of the robot’s seventh degree of freedom, which is common between all action spaces. Target pose is then computed as target=⊕c.δ x^target= x c.δ x. This target pose is then passed to an analytical inverse kinematics solver proposed by [6] and we use the implementation by [28], which maps target x^target to target joint positions (IK:target↦targetIK: x^target q^target), then a position actuator is used (fposac:target↦f^ac_pos: q^target τ). The primary difference between δδ x and ˙ x lies in how the resulting joint commands are actuated. For ˙ x, we reuse the same target joint positions target q^target produced by the inverse kinematics solver, but approximate joint velocities using the current joint positions and the action cycle time, given by target≈target−δt v^target≈ q^target- qδ t which then uses a velocity actuator that maps from target velocity to torque (fvelac:target↦f^ac_vel: v^target τ). Gripper Control In PandaPickCuboid, once a policy outputs an action vector a, the final value of this vector is used to command the gripper into a closed or open state to grasp the object. Gripper activation at a given time step t can be expressed using the following step-wise function: graspt:=1,if ag<λg1,0,if ag≥λg2,graspt−1,otherwise. _t:= cases1,&if a_g<λ^1_g,\\[4.0pt] 0,&if a_g≥λ^2_g,\\[4.0pt] grasp_t-1,&otherwise. cases (11) where aga_g is the scalar gripper action output and λg _g is the grasp activation threshold. A value of 1 corresponds to closing the gripper, and a value of 0 corresponds to opening it. During simulation training, we set λg1=λg2=0λ^1_g=λ^2_g=0 and we adjusted the thresholds during real-time trials to improve grasping performance by setting them to λg1=−0.2λ^1_g=-0.2 and λg2=0.9λ^2_g=0.9. In the case of δδ x, we set λg1=−0.1λ^1_g=-0.1 to improve grasping performance and reduce false grasp attempts. Figure 3: PandaPickCuboid simulation training results consisting of episodic return, success rate, and episodic length. 10 independent runs are shown with the median curve highlighted. V Experimental Setup In this section, we discuss the real-world setup of the camera and arm, the neural networks, and domain randomization used in our experiments. V-A Real World Setup Our real-world setup consists of a red block with dimensions 4×4×64× 4× 6 cm, an Intel RealSense D405 camera, and the Franka Emika Panda robot arm mounted to the robot’s wrist (Fig. 4). In all real-world trials, the block is placed well within the arm’s dexterous workspace and within the camera’s field of view. Figure 4: Real-world setup and wrist-mounted camera. TABLE I: Real-world evaluation results for PandaPickCuboid and PandaPushCuboid. For Pick, success reports mean ± standard error, and other metrics report median ± IQR. For Push, values are reported as median ± IQR. ↑ indicates a higher value is preferred. ↓ indicates a lower value is preferred. Best performers are bolded. PandaPickCuboid PandaPushCuboid Action Space Success (%) ↑ Time (s) ↓ Collision ↓ Jerk (m/s3m/s^3) ↓ Distance (m) ↑ Collision ↓ Jerk (m/s3m/s^3) ↓ Pose Inc. (δδ x) 41.67±14.2341.67± 14.23 14.78±11.3814.78± 11.38 0.0±0.00.0± 0.0 26.88±17.9626.88± 17.96 0.050±0.0510.050± 0.051 0.0±0.0 0.0± 0.0 60.27±13.6160.27± 13.61 Pose Vel. (˙ x) 0.00±0.000.00± 0.00 8.98±17.658.98± 17.65 0.0±0.00.0± 0.0 24.55±10.3324.55± 10.33 0.000±0.0030.000± 0.003 2.5±22.02.5± 22.0 24.85±5.4724.85± 5.47 Joint Pos. Inc. (δδ q) ±0.00 100± 0.00 2.12±0.59 2.12± 0.59 0.0±0.00.0± 0.0 52.59±15.4152.59± 15.41 0.104±0.0710.104± 0.071 0.0±0.0 0.0± 0.0 24.37±2.3824.37± 2.38 Joint Vel. ( v) ±0.00 100± 0.00 3.58±0.233.58± 0.23 0.0±0.00.0± 0.0 16.03±1.34 16.03± 1.34 0.405±0.289 0.405± 0.289 8.0±14.58.0± 14.5 14.25±0.99 14.25± 0.99 Pick Script ±0.00 100± 0.00 15.91±1.5715.91± 1.57 0.0±0.00.0± 0.0 11.43±0.88 11.43± 0.88 – – – V-B Neural Networks We employ vision-based actor–critic networks that jointly process RGB image observations and vector-valued observations. Visual features from the image are extracted using a convolutional encoder implemented by [5] composed of three convolutional layers with ReLU activations. The visual feature vector is concatenated with additional non-visual vector-valued observations, including proprioception, the previous action, the end-effector height, and the grasp status (depending on the task: pick vs push). The critic network consists of the convolutional encoder and concatenated observations, followed by a fully connected multilayer perceptron (MLP) head that outputs a scalar state-value estimate. The actor network shares the same structure but differs in its output head, where two fully connected layers parameterize a Gaussian policy by predicting the mean (μ) and standard deviation (σ) of the action distribution. V-C Domain Randomization To improve robustness and facilitate sim-to-real transfer, we apply domain randomization [23, 14, 4]. During training, we randomize both visual and dynamic parameters. Visual perturbations include camera pose, lighting position and intensity, and image-level properties such as contrast and saturation. We also randomize the block’s initial pose and overlay randomized wooden textures to better match the real-world surface. We inject Gaussian noise into joint position and velocity readings and randomize stiffness and damping. V-D Actuator Tuning When using impedance control on the real robot, appropriately tuned stiffness and damping gains are critical for safe, stable operation. Poorly tuned gains can lead to excessive vibrations, unstable motion, and safety-triggered shutdowns. In our experiments, we reduced stiffness to reduce vibrations and prevent erratic motion. VI Real-World Results PandaPickCuboid. We train each policy in simulation using PPO over 10 independent seeds with 1024 parallel environments. Fig. 3 presents 10 deterministic evaluation runs performed during simulation training. Individual runs are displayed for episodic return, success rate, and episode length. Each action space is denoted by a unique color, and the median run is highlighted. All individual runs are displayed to better portray variation in performance [22]. In simulation, all action spaces achieve comparable final performance, with success rates in the range [0.98,1.00][0.98,1.00]. The primary distinction lies in the episodic length, which, in this case, is the time required to complete the task. The pose increment action space (δδ x) achieves the shortest episodes, followed by joint position increment (δδ q). For real-world evaluation, we conduct 12 deterministic trials per action space (Table I). For each action space, we select the top two policies with the highest final episodic return and report results from the policy with the best real-world success. A trial is considered successful if the cuboid reaches at least a target height of 17 cm. Trials terminate upon 10 failed grasp attempts, violation of workspace constraints, or a timeout of 30 s. Joint velocity ( v) achieves a 100% success rate with a median completion time of 3.58 s. It consistently avoids collisions and exhibits the lowest measured jerk among the evaluated action spaces. Joint position increments (δδ q) also achieve a 100% success rate and the fastest median completion time of 2.12 s; however, stable real-world deployment required careful impedance tuning. We also tested our scripted pick policy, which achieves a 100%100\% success rate and a median trial time of 15.9115.91 seconds. In contrast, pose increments (δδ x) demonstrate degraded real-world performance. The policy frequently dragged the cuboid toward the robot base before lifting. To mitigate this behavior, we imposed a lower bound of 0.45 m on the end-effector x position during evaluation. Despite this modification, the action space achieved a 41.67% success rate. Pose velocity (˙ x) performed worst in real-world trials, often drifting into unstable configurations or exiting the workspace. It did not successfully complete the task in any trial. Overall, the simulation results did not fully translate to the real-world experiments. In simulation, δδ x lifted the block more rapidly, whereas on the real robot, it frequently failed to complete the lifting motion. Likewise, ˙ x failed in all real-world trials. These findings suggest that our Cartesian action spaces are more sensitive to sim-to-real discrepancies. A key difference is that the Cartesian actions rely on an inverse kinematics (IK) mapping to generate joint commands. Even when analytical IK is used, this introduces a kinematic transformation layer. Any discrepancies in link parameters between simulation and reality will propagate through the IK mapping, generating inaccurate joint targets. Furthermore, IK-generated joint trajectories can be highly demanding or abrupt—particularly near singularities. While the simulator’s idealized actuators can execute these commands, real-world actuators bounded by physical joint friction, latency, and imperfect impedance gains may struggle to track them accurately, leading to amplified errors. PandaPushCuboid. We train each policy in simulation using PPO over 10 independent seeds with 1024 parallel environments. Fig. 5 reports episodic return for 10 deterministic evaluations collected throughout simulation training. All individual runs are displayed. Each action space is represented using a unique color. The median run is highlighted. In simulation, joint position increment (δδ q) and pose increment (δδ x) achieve higher median episodic returns than joint velocity ( v) and pose velocity (˙ x), probably due to generating faster trajectories that displace the block more from point-to-point. δδ x and ˙ x exhibit some failed runs, potentially due to the emergence of singularities in the IK solution that stall the robot’s motion. Figure 5: PandaPushCuboid episodic return in simulation training. 10 independent runs are displayed. The median curve is highlighted. For real-world evaluation, we conduct 10 trials per action space using deterministic policies (Table I). For each action space, we select the policy with the highest final episodic return and evaluate zero-shot sim-to-real performance. Each trial runs for 12s or terminates if the end-effector exits predefined workspace boundaries that are larger than the white base’s boundaries. Aggregate block distance over the trial duration is used as the performance metric. Joint velocity ( v) achieves the largest median aggregate block distance of 0.405 m. Although this action space produces the lowest jerk, we observe multiple collisions, primarily due to forceful contact with the cuboid edges. Joint position increments (δδ q) achieve the second-highest displacement (median 0.104 m). In contrast, pose increment (δδ x) and pose velocity (˙ x) perform poorly in terms of block distance. However, δδ x and δδ q exhibit fewer forceful contact events and appear more compliant during interaction. This compliance does not arise from explicit contact avoidance, but rather from the tuned joint-impedance controller used for position-based actuation, which reduces stiffness and attenuates excessive forces during contact. Figure 6: Simulation training metrics: floor collisions and jerk per step for picking and pushing tasks. 10 independent runs are displayed, and the median curve is highlighted for each action space. VII Training Results on Jerk and Collision Figure 6 reports episodic floor collisions and average jerk per time step during simulation training, averaged over 1024 parallel environments. These results can inform us of what can happen during learning in the real world. A collision is registered when the gripper body contacts the floor in simulation. Across both PandaPickCuboid and PandaPushCuboid, joint velocity yields the lowest floor-collision counts at convergence, exhibiting more precision during simulation training and exhibiting the smallest jerk. We approximate jerk at time step t using a finite difference of acceleration of consecutive time steps, Jt≈‖t−t−1δt‖2.J_t≈|| a_t- a_t-1δ t||_2. Since joint acceleration is proportional to the applied torque, jerk reflects step-to-step changes in torque. Consequently, larger torque variations across consecutive time steps lead to larger observed jerk. We will now provide a plausible explanation for why joint velocity has the lowest jerk. Under position actuation, torque is generated by a position actuator acting on the position error. The resulting torque variation depends on changes in both the position error and joint velocity. In particular, fluctuations in the desired position directly perturb the position error, which is scaled by the proportional gains p K_p. When p K_p values are large, variations in the commanded position can produce substantial torque changes and, therefore, larger jerk. Under velocity actuation, torque is instead driven by the velocity tracking error and scaled by the velocity gain v K_v. In our experiments, kp>kvk_p>k_v for all joints. Hence, for comparable command variations, position control can induce larger torque differences than joint velocity. Additionally, with position actuation, the policy directly specifies a desired position (or position increment), so high-frequency fluctuations in the policy output immediately affect the position error and are amplified by the proportional gain. In contrast, velocity actuation produces target velocities, and joint positions evolve via integration, t+1=t+t′δt. q_t+1= q_t+ v _tδ t. This integration acts as a low-pass filter: high-frequency components in the velocity command are attenuated when mapped to position, introducing temporal smoothing. VIII Conclusion In this work, we benchmarked four commonly used action space representations in vision-based robotic manipulation: joint position increments (δδ q), joint velocity ( v), pose increments (δδ x), and pose velocity (˙ x). Policies were trained in simulation on two vision-based tasks, PandaPickCuboid and PandaPushCuboid, and deployed on a physical Franka Emika Panda using sim-to-real transfer. Across both tasks, joint-space action spaces demonstrated better real-world performance. In particular, joint velocity ( v) achieved the strongest zero-shot sim-to-real results, reaching a 100%100\% success rate in PandaPickCuboid and the highest cuboid displacement in PandaPushCuboid. Joint position increments (δδ q) achieved comparable picking performance, although stable deployment required impedance tuning. In terms of safety and compliance, action spaces operating through the tuned impedance controller (δδ q and δδ x) exhibited more compliant behavior and reduced collision severity during contact. However, joint velocity produced the smoothest trajectories overall, achieving the lowest measured jerk and yielding the most natural motion profiles. These results suggest that joint velocity provides a strong balance of robustness, smoothness, and sim-to-real performance. IX Limitation and Future Work A limitation of this work is that we consider single-object manipulation tasks involving a single cuboid. Extending the proposed sim-to-real pipeline to diverse objects with varying geometries, sizes, and appearances, as well as to multi-object or cluttered scenes, and complex manipulation tasks like screwing, hammering, and manipulating objects with significant variation in mass and friction, remains future work. In particular, object selection in cluttered environments presents a scalability challenge. One possible direction is to incorporate object detection to define regions of interest (ROIs) for target selection [23]. Alternatively, mask-based goal conditioning and ROI selection could enable specification and manipulation of target objects in cluttered scenes [19]. This paper considers four commonly used action spaces for robots; there are more, such as joint position or torque control. We did in fact experiment with torque-based control, but found it was more difficult to train and led to unstable behavior in simulation. In real-world deployment, torque control required explicit gravity and Coriolis compensation, complicating stable lifting during the picking task. A systematic evaluation of torque control for contact-rich manipulation is left for future investigation. Finally, this study focuses on object manipulation; a line of future work is to perform a similar action-space comparison in other classes of robotic tasks, such as locomotion and other embodiments, such as drones and humanoid robots. Acknowledgement We would like to thank anonymous reviewers for their constructive comments and feedback, which helped improve this work. We would like to thank Shivam Garg, Mustafa Heidarbhai, and Haruto Tanaka for helpful discussions. This research was supported in part by the National Research Council Canada (NRC) and by the CIFAR AI Chair program. We are also appreciative of the computing resources provided by the Digital Research Alliance of Canada and the financial support from the RLAI laboratory and Amii. References [1] E. Aljalbout, F. Frank, M. Karl, and P. van der Smagt (2024) On the role of the action space in robot manipulation learning and sim-to-real transfer. IEEE Robotics and Automation Letters 9 (6), p. 5895–5902. Cited by: §I, §I, §I, §I, §I, §I-A, §IV-C. [2] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox (2019) Closing the sim-to-real loop: adapting simulation randomization with real world experience. In IEEE International Conference on Robotics and Automation (ICRA), p. 8973–8979. Cited by: §I, §I-A. [3] S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath (2023) Learning torque control for quadrupedal locomotion. In IEEE-RAS International Conference on Humanoid Robots (Humanoids), p. 1–8. Cited by: §I. [4] X. Chen, J. Hu, C. Jin, L. Li, and L. Wang (2022) Understanding domain randomization for sim‐to‐real transfer. In Tenth International Conference on Learning Representations (ICLR), Cited by: §V-C. [5] Brax: a differentiable physics engine for large-scale rigid-body simulation External Links: Link Cited by: §I, §I-A, §V-B. [6] Y. He and S. Liu (2021) Analytical inverse kinematics for franka emika panda: a geometrical solver for 7-dof manipulators with unconventional design. In International Conference on Control, Mechatronics and Automation (ICCMA), p. 194–199. Cited by: §IV-C. [7] D. Kim, G. Berseth, M. Schwartz, and J. Park (2023) Torque-based deep reinforcement learning for task- and robot-agnostic learning on bipedal robots using sim-to-real transfer. IEEE Robotics and Automation Letters 8 (10), p. 6251–6258. Cited by: §I, §I. [8] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg (2019) Making sense of vision and touch: self-supervised learning of multimodal representations for contact-rich tasks. In IEEE International Conference on Robotics and Automation (ICRA), p. 8943–8950. Cited by: §I. [9] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. International Journal of Robotics Research 37 (4-5), p. 421–436. Cited by: §I. [10] A. R. Mahmood, D. Korenkevych, B. J. Komer, and J. Bergstra (2018) Setting up a reinforcement learning task with a real-world robot. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: §I, §IV-C. [11] A. R. Mahmood, D. Korenkevych, G. Vasan, W. Ma, and J. Bergstra (2018) Benchmarking reinforcement learning algorithms on real-world robots. In Conference on Robot Learning (CoRL), p. 561–591. Cited by: §I, §I. [12] R. Martín-Martín, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg (2019) Variable impedance control in end-effector space: an action space for reinforcement learning in contact-rich tasks. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1010–1017. Cited by: §I, §I, §I, §I, §I-A. [13] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang (2019) Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113. Cited by: §I, §I-A. [14] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel (2018) Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation (ICRA), p. 1–8. Cited by: §I, §I, §V-C. [15] X. B. Peng and M. van de Panne (2017) Learning locomotion skills using deeprl: does the choice of action space matter?. In ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA), p. 12:1–12:13. Cited by: §I. [16] A. Petrenko, A. Allshire, G. State, A. Handa, and V. Makoviychuk (2023) DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §IV-A. [17] J. Schneider, P. Schumacher, D. F. B. Häufle, B. Schölkopf, and D. Büchler (2023) Investigating the impact of action representations in policy gradient algorithms. arXiv preprint arXiv:2309.06921. Cited by: §I. [18] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §I, §I-A, §I-A. [19] F. Shahriar, C. Wang, A. Azimi, G. Vasan, H. H. Elanwar, A. R. Mahmood, and C. Bellinger (2025) General and efficient visual goal-conditioned reinforcement learning using object-agnostic masks. arXiv preprint arXiv:2510.06277. Cited by: §IX. [20] Franka ros interface: a ros/python api for controlling and managing the franka emika panda robot (real and simulated). Cited by: §I-C. [21] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke (2018) Sim-to-real: learning agile locomotion for quadruped robots. In Robotics: Science and Systems (RSS), Cited by: §I, §I-A. [22] H. Tanaka and A. R. Mahmood (2026) Performance variation in deep reinforcement learning. arXiv preprint arXiv:2606.06746. Cited by: §VI. [23] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 23–30. Cited by: §I, §V-C, §IX. [24] E. Todorov, T. Erez, and Y. Tassa (2012) MuJoCo: a physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, p. 5026–5033. Cited by: §I-B, §I-B. [25] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa (2020) Dm_control: software and tasks for continuous control. Software Impacts 6, p. 100022. External Links: ISSN 2665-9638 Cited by: §I. [26] P. Varin, L. Grossman, and S. Kuindersma (2019) A comparison of action spaces for learning manipulation tasks. arXiv preprint arXiv:1908.08659. Cited by: §I, §I. [27] Z. Xu, Y. Li, X. Yang, Z. Zhao, L. Zhuang, and J. Zhao (2024) Open-source reinforcement learning environments implemented in MuJoCo with franka manipulator. In IEEE International Conference on Advanced Intelligent Mechatronics (AIM), p. 709–714. Cited by: §I. [28] K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel (2025) MuJoCo playground. arXiv preprint arXiv:2502.08844. Cited by: §I, §I, §IV-A, §IV-C, §IV-C.