Paper deep dive
Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped
Jagannath Prasad Sahoo, Saurabh Kumar, Surya Prakash S. K., Samiran Datta, Abhay Dwivedi, Amit Shukla
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A bipedal robot cannot deviate from its path to avoid an obstacle without disturbing its balance, and this coupling is most severe on underactuated platforms such as the biped considered here, which has four actuated joints per leg and no hip or ankle roll. This paper presents a Hierarchical Reinforcement Learning (HRL) framework in which a High-Level (HL) policy observes the robot pose, 36 raycast proximity measurements, moving-obstacle states, and a receding-horizon local goal, and outputs a body-velocity command $(v_x, v_y, \omega_{yaw})$ every ten control steps, while a velocity-conditioned Low-Level (LL) policy tracks each command through PD-controlled joint targets. Both policies are trained jointly with Soft Actor-Critic (SAC). Because the converged gait is task-agnostic, it is frozen and driven by classical planners over the same command interface, yielding three controlled baselines: SAC+A*, SAC+RRT*, and SAC+APF. Across 100 evaluation trials per method in randomized PyBullet environments, the proposed method reaches the goal in 98.0% of static and 88.0% of dynamic trials, against at most 78.0% and 68.0% for the planner hybrids, with path lengths within 4% of the A* reference, and ablations confirm that each observation channel and reward term contributes materially to this performance.
Tags
Links
- Source: https://arxiv.org/abs/2610.05855v1
- Canonical: https://arxiv.org/abs/2610.05855v1
Trouble viewing inline? Open PDF directly →
Full Text
30,462 characters extracted from source content.
Expand or collapse full text
Hierarchical Reinforcement Learning for Collision-Free Locomotion of an Underactuated Biped Jagannath Prasad Sahoo Saurabh Kumar Affiliation: CAIR, Indian Institute of Technology Mandi, Mandi, India S23118,s25059,s24037@students.iitmandi.ac.in, abhaydwivedi10122005@gmail.com and amitshukla@iitmandi.ac.in Surya Prakash S.K Affiliation: SMME, Indian Institute of Technology Mandi, Mandi, India d21077@students.iitmandi.ac.in Samiran Datta Affiliation: CAIR, Indian Institute of Technology Mandi, Mandi, India S23118,s25059,s24037@students.iitmandi.ac.in, abhaydwivedi10122005@gmail.com and amitshukla@iitmandi.ac.in Abhay Dwivedi Affiliation: CAIR, Indian Institute of Technology Mandi, Mandi, India S23118,s25059,s24037@students.iitmandi.ac.in, abhaydwivedi10122005@gmail.com and amitshukla@iitmandi.ac.in Amit Shukla †thanks: Video Link: https://drive.google.com/file/d/1efTTfWlWawnvGmUdufkGtoBgXsjYLPRd/view?usp=sharing Affiliation: CAIR, Indian Institute of Technology Mandi, Mandi, India S23118,s25059,s24037@students.iitmandi.ac.in, abhaydwivedi10122005@gmail.com and amitshukla@iitmandi.ac.in Abstract A bipedal robot cannot deviate from its path to avoid an obstacle without disturbing its balance, and this coupling is most severe on underactuated platforms such as the biped considered here, which has four actuated joints per leg and no hip or ankle roll. This paper presents a Hierarchical Reinforcement Learning (HRL) framework in which a High-Level (HL) policy observes the robot pose, 36 raycast proximity measurements, moving-obstacle states, and a receding-horizon local goal, and outputs a body-velocity command (vx,vy,ωyaw)(v_x,v_y, _yaw) every ten control steps, while a velocity-conditioned Low-Level (L) policy tracks each command through PD-controlled joint targets. Both policies are trained jointly with Soft Actor-Critic (SAC). Because the converged gait is task-agnostic, it is frozen and driven by classical planners over the same command interface, yielding three controlled baselines, SAC+A⋆, SAC+RRT⋆, and SAC+APF. Across 100 evaluation trials per method in randomized PyBullet environments, the proposed method reaches the goal in 98.0% of static and 88.0% of dynamic trials, against at most 78.0% and 68.0% for the planner hybrids, with path lengths within 4% of the A⋆ reference, and ablations confirm that each observation channel and reward term contributes materially to this performance. This present project information and videos are available at HRL-Biped. Index Terms: Bipedal locomotion, hierarchical reinforcement learning, soft actor–critic, obstacle avoidance, underactuated robots. I Introduction Obstacle-aware walking remains an open problem for three compounding reasons. A biped is underactuated, so its floating base cannot be commanded directly and every avoidance maneuver perturbs balance. Its velocity set is gait-limited and asymmetric, since a biped needs several steps to change speed or heading, and on our platform the absence of roll joints further restricts lateral motion. Moving obstacles demand anticipation, since by the time proximity alone signals danger a walking robot may no longer have time to step aside. Fig. 1: Control paradigms for obstacle-aware bipedal locomotion. Flat RL entangles navigation with gait, decoupled planning ignores balance coupling, and the proposed hierarchy learns a velocity-command policy over a learned velocity-conditioned gait. Existing approaches occupy two extremes (Fig. 1). Flat end-to-end RL policies map observations directly to joint targets [1], train slowly, generalize poorly to unseen layouts, and must relearn locomotion whenever the task changes. Decoupled pipelines instead track a geometrically planned path with an independent gait controller [2, 3, 4], ignoring balance coupling and, for global planners, moving obstacles. Neither extreme addresses all three difficulties at once, which motivates separating navigation from gait generation without severing the feedback between them. Given an underactuated biped with eight actuated leg joints, a known static occupancy map, and moving obstacles whose trajectories are unknown in advance, the controller must reach a goal at least 6.0 m away without collision or fall, sensing static geometry through proximity measurements and moving obstacles through their relative position and velocity, while generating joint-level actions at 50 Hz. We solve this with a two-level hierarchy in which both levels are learned and trained jointly. A lightweight planning layer computes a global path on the known static map and extracts a receding-horizon local goal. The high-level (HL) policy decides where to go next, emitting a body-velocity command =(vx,vy,ωyaw)u=(v_x,v_y, _yaw) from the robot pose, raycast distances, moving-obstacle states, and the local goal. The low-level (L) policy decides how to walk, converting each command into leg-joint targets and executing it for H=10H=10 consecutive control steps before the HL policy is queried again. This abstraction shortens the effective navigation horizon tenfold, and joint training lets the gait adapt to the command distribution the navigation layer actually issues. Because the converged L policy is a pure velocity-conditioned gait skill, it can be frozen and driven by classical planners over the same interface, which we exploit to build three controlled baselines, SAC+A⋆, SAC+RRT⋆, and SAC+APF, in which only the command generator differs. The contributions of this work are as follows. • A jointly trained two-level HRL framework in which an HL SAC policy outputs body-velocity commands from pose, raycast, dynamic-obstacle, and local-goal observations, and a velocity-conditioned L SAC policy executes each command for ten control steps. • An HL observation and reward design covering both static layouts and moving obstacles, including a time-to-collision (TTC) term inducing anticipatory avoidance. • A controlled benchmark against SAC+A⋆, SAC+RRT⋆, and SAC+APF hybrids driving the identical frozen gait, isolating the navigation layer’s contribution. • An ablation over HL observation channels and reward components quantifying their effect on success, collision, and failure rates. I Related Work Obstacle avoidance began with classical planning. Global methods such as A⋆ [2] and RRT⋆ [3] compute near-optimal paths on a known map, and reactive methods such as artificial potential fields [4] and the dynamic window approach [5] respond to sensed obstacles online. These assume a base that executes commanded velocities faithfully, the reactive ones suffer local minima and oscillation, and applied to a biped they decouple avoidance from balance. CLF-CBF quadratic programs [6] restore real-time safety guarantees but still treat the gait as a separate tracking problem. Reinforcement learning then removed the need for hand-designed reactive rules, maturing first on wheeled robots for mapless navigation [7], goal-driven exploration [8], and crowd navigation [9]. In parallel, model-free RL produced robust legged locomotion, including periodic-reward gaits on Cassie [10], transformer-based humanoid locomotion [11], footstep-conditioned walking [12], and massively parallel training for quadrupeds [13, 14]. In our own prior work we trained multi-skill locomotion for the same underactuated platform using waypoint-based reward shaping [15]. The two lines rarely meet: navigation policies assume a statically stable base, locomotion policies do not avoid obstacles, and the one flat-RL attempt at bipedal avoidance [1] is limited to fixed, known obstacle positions. Hierarchical RL bridges the two. Temporal abstraction via options [16] and goal-conditioned hierarchies such as HIRO [17] showed that a coarse-timescale high-level policy simplifies long-horizon tasks. DeepLoco [18] learned a footstep-level planner over a stepping controller, hierarchical commands have been used for quadruped navigation [19], and recent bipedal social navigation adopts a related decomposition [20]. Our work targets an underactuated biped without roll joints, combines raycasts with explicit dynamic-obstacle states at the HL, trains both levels jointly, and benchmarks the learned HL against classical planners driving the identical learned gait, which prior HRL locomotion studies do not. I Preliminaries and Problem Formulation I-A Two-timescale decision process The L control problem is a discrete-time Markov decision process (ℓ,ℓ,pℓ,rℓ,γℓ)(S_ ,A_ ,p_ ,r_ , _ ) at fc=50f_c=50 Hz, with γℓ=0.99 _ =0.99 (Table I). Because the HL policy acts every H=10H=10 L steps with the action held constant, the HL problem is a semi-MDP [16] whose reward accumulates the per-step task reward over the macro-step, rh(sk,k)=∑t=kH(k+1)H−1γℓt−kHr(st,at),r_h(s_k,u_k)= _t=kH^(k+1)H-1 _ ^\,t-kH\,r(s_t,a_t), (1) where sks_k is the HL state at macro-step k, ku_k the held command, sts_t the per-step L state, and at∼πℓ(⋅∣st,k)a_t _ (· s_t,u_k) the L action, with r(st,at)r(s_t,a_t) the task reward of Sec. IV-C. Eq. (1) shrinks the effective navigation horizon by a factor of H, stabilizing credit assignment for long-range goal reaching. I-B Soft Actor–Critic Both levels are trained with SAC [21], maximizing the entropy-regularized return J(π)=∑t(st,at)∼ρπ[r(st,at)+αℋ(π(⋅∣st))],J(π)= _tE_(s_t,a_t) _π\! [r(s_t,a_t)+α\,H\! (π(· s_t) ) ], (2) where ρπ _π is the induced state–action distribution, ℋH the policy entropy, and α the temperature, auto-tuned at both levels from 1.01.0 (Table I). Eq. (2) aids exploration in the contact-rich L space and the HL navigation problem alike, and off-policy SAC with twin critics is more sample-efficient than on-policy alternatives such as PPO [22]. IV Method Fig. 2: System architecture: a global path supplies a receding-horizon local goal; every H=10H=10 steps the HL policy outputs a velocity command executed by the L policy through PD-tracked joint targets; baselines replace only the HL block. The control structure is shown in Fig. 2. IV-A Planning layer and local-goal generation A global path is computed once per episode on the known static map. Dynamic obstacles are excluded from the map, since handling them is the HL policy’s job. A receding-horizon module extracts a local goal, the lookahead waypoint at arc-length dlad_la (a tuned constant) ahead of the robot’s closest path point, advancing with the robot and replaced by the final goal once within dlad_la. This is the g~ g term of Eq. (4), and the same path defines the tangent, cross-track error, and waypoint progress of Eq. (7). The layer issues no motion commands. IV-B Low-level velocity-conditioned gait policy The L observation sℓ=[q,q˙,ϕbase,ωbase,ct,]s_ =[\,q, q, _base, _base,c_t,u\,] comprises the eight leg-joint positions q and velocities q˙ q, base orientation ϕbase _base and angular velocity ωbase _base from the IMU, a clock signal ct=[sinωt,cosωt]c_t=[ ω t, ω t], and the held command u; it has no obstacle or goal information. The action is the target position for each of the eight actuated joints, added to the half-sitting configuration and tracked by PD controllers. With no hip or ankle roll, lateral commands vyv_y are realized through coordinated hip-yaw and sagittal motion, so the command box is asymmetric with |vy|≤vymax≪vxmax|v_y|≤ v_y v_x . The L reward rewards accurate tracking with a stable gait, rℓ= r_ = wve−∥xybase−xy∥2/σv+wωe−(ωzbase−ωyaw)2/σω w_v\,e^- ^xy_base-u^xy ^2/ _v+w_ω\,e^-(ω^z_base- _yaw)^2/ _ω +rgait, +r_gait, (3) with measured planar velocity basexyv^xy_base and yaw rate ωbasezω^z_base, empirically tuned weights wv,wωw_v,w_ω and tolerances σv,σω _v, _ω, and rgaitr_gait aggregating contact-timing consistency [10], posture regularization, action-rate smoothness, and torque penalties. IV-C High-level navigation policy The HL observation is sh=[pbase,ψ,ray,(p~j,v~j)j=1Nd,g~],s_h= [\,p_base,ψ,\;d_ray,\;\( p_j, v_j)\_j=1^N_d,\; g\, ], (4) where pbase,ψp_base,ψ are the base position and heading, ray∈ℝ36d_ray ^36 collects 36 raycasts over 180∘180 clipped to dmax=3.0d_ =3.0 m, p~j,v~j p_j, v_j are the position and velocity of the j-th of Nd=5N_d=5 nearest dynamic obstacles, and g~ g is the local goal in the robot frame. Both raycasts and obstacle states are privileged simulator quantities standing in for a LiDAR and object tracker (Sec. VI). The HL action =(vx,vy,ωyaw)u=(v_x,v_y, _yaw) is tanh -squashed to the L command box and held for H=10H=10 steps, so the HL policy runs at 55 Hz. The task reward accumulates into Eq. (1) as a shaping sum clipped at −cs-c_s (cs=15c_s=15) plus terminal terms added after the clip, rt=max(rshape,−cs)+rterm.r_t= \! (r_shape,\,-c_s )+r_term. (5) rshape=rpath+rposture+rsafe+rreg.r_shape=r_path+r_posture+r_safe+r_reg. (6) The path term rewards progress along the reference, rpath=w1Δdwp+w2v∥+w3(^⋅^)−w4ect2+w9dgoal+ϵ,r_path=w_1 d_wp+w_2v_ +w_3 ( h\!·\! t )-w_4e_ct^2+ w_9d_goal+ε, (7) with waypoint progress Δdwp d_wp, path tangent t, base heading h, forward velocity v∥v_ , cross-track error ecte_ct, goal distance dgoald_goal, and ϵ=10−5ε=10^-5. The posture term makes the HL accountable for gait feasibility, rposture= r_posture= −w5(|ϕr|+|ϕp|)−w6|zb−zref|−w7v⟂2 -w_5 (| _r|+| _p| )-w_6|z_b-z_ref|-w_7v_ ^2 −w8∑i=18|τi|+w12∑f∈swingmin(hf,hcap), -w_8 _i=1^8| _i|+w_12\!\! _f \! (h_f,h_cap), (8) with roll/pitch ϕr,ϕp _r, _p, base height zbz_b (zref=0.45z_ref=0.45 m), lateral drift v⟂v_ , torques τi _i, and swing clearance hfh_f capped at hcaph_cap. The safety term acts at two ranges, rsafe=−w14max(0,dsafe−dobs)dsafe−w15max(0,Th−ttc)Th,r_safe=-w_14 \! (0,\,d_safe-d_obs )d_safe-w_15 \! (0,\,T_h-t_tc )T_h, (9) with nearest-obstacle distance dobsd_obs, safety radius dsafe=0.55d_safe=0.55 m, time to collision (TTC) ttct_tc from the nearest dynamic obstacle’s relative state, and TTC horizon Th=1.5T_h=1.5 s. The TTC term penalizes predicted collisions before proximity alone would trigger, which is the key mechanism behind dynamic-obstacle avoidance. The regularization term removes degenerate optima, rreg=−w10 1stuck−w11 1slow−w13,r_reg=-w_10\, 1_stuck-w_11\, 1_slow-w_13, (10) where stuck 1_stuck activates after tstuck=0.7t_stuck=0.7 s without progress, slow 1_slow when v∥<vmin=0.02v_ <v_ =0.02 m/s, and w13w_13 is a constant time penalty. Without it, standing still near obstacles is favored. The terminal term is rterm=−Rc 1collision−Rf 1fall+Rg 1goal,r_term=-R_c\, 1_collision-R_f\, 1_fall+R_g\, 1_goal, (11) with Rc=Rf=50R_c=R_f=50, Rg=30R_g=30. Collisions incur −Rc-R_c without terminating the episode, letting the policy learn recovery. Falls and goal arrival terminate it. The clip csc_s prevents a transient penalty pile-up from dominating the value function, and adding Eq. (11) after the clip keeps collision and fall penalties undiluted. Weights w1w_1–w15w_15 were tuned manually and are omitted for space. Deviating from the path to avoid an obstacle costs the cross-track term of Eq. (7) but is recouped through Eq. (9). Resolving this trade-off is what the HL policy learns. IV-D Joint two-timescale training Both policies train simultaneously in the same episodes (Algorithm 1). The L policy is updated at 50 Hz from Eq. (3). The HL policy is updated at 5 Hz from the macro-step return of Eqs. (1) and (5). Joint training couples the two levels: the gait adapts to the navigation layer’s actual command distribution, and the navigation layer learns the gait’s evolving feasibility limits through Eq. (8), with the HL policy acting as an automatic curriculum. Episodes terminate on goal arrival, fall, or timeout. After convergence the L policy is frozen and reused by all baselines of Sec. IV-E. Algorithm 1 Joint two-timescale HRL training 1: initialize πh _h, πℓ _ , buffers hD_h, ℓD_ 2: for each episode do 3: randomize layout/obstacles/start/goal; plan global path; init. local goal (Sec. IV-A) 4: for each macro-step k do 5: advance local goal; observe shs_h (Eq. (4)); sample k∼πhu_k _h 6: for t=kHt=kH to (k+1)H−1(k+1)H-1 do 7: at∼πℓ(⋅∣sℓ,t,k)a_t _ (· s_ ,t,u_k); step environment; store in ℓD_ ; update πℓ _ 8: end for 9: accumulate rhr_h; store in hD_h; update πh _h 10: end for 11: end for 12: freeze πℓ _ for reuse by all baselines IV-E Planner-based baselines over the shared interface All baselines drive the frozen co-trained L policy, differing only in how ku_k is generated. In SAC+A⋆, the planner [2] plans on the inflated static map and a pure-pursuit tracker converts the path to commands, with periodic replanning from ground-truth obstacle positions in dynamic scenarios as the strongest reasonable variant. SAC+RRT⋆ [3] plans on the same map and is tracked identically. SAC+APF computes attractive and repulsive forces from the same raycast and obstacle observations available to our HL policy [4] and clips the resultant to the command box. APF is fully reactive but susceptible to local minima. Every method commands the same gait through the same interface at the same rate, so Table I compares navigation intelligence only. V Experiments All experiments are conducted in simulation. V-A Setup The robot has four actuated joints per leg (hip yaw, hip pitch, knee pitch, ankle pitch, eight DoF total), thigh and shank links of 99.4599.45 m, height 542542 m, feet 146×66.95146× 66.95 m, and a base-mounted IMU. Simulation is in PyBullet (seed 42). Both levels use 2×2562× 256 MLP actors and critics (Table I), trained on a single NVIDIA RTX A4000 GPU. Metrics are success, collision, and fall/timeout rate, path length, and time to goal, over 10 unseen layouts per condition with 10 trials each (100 trials per method per condition). TABLE I: SAC training hyperparameters, shared by both levels. Parameter Value Discount factor γℓ _ 0.990.99 Learning rate 1×10−41× 10^-4 Replay buffer size 1×1061× 10^6 Batch size 256256 Entropy temperature α auto-tuned (init. 1.01.0) Soft update coefficient 0.0050.005 Gradient steps per env. step 11 Max episode length 50005000 steps (100100 s) Control frequency fcf_c 5050 Hz HL decision period H 1010 steps (55 Hz) Actor/critic networks MLP, 2×2562× 256 Simulator PyBullet (seed 42) V-B Static and dynamic scenarios Static evaluations use unseen randomized layouts with start and goal ≥6.0≥ 6.0 m apart (Fig. 3). Dynamic obstacles move along straight and lateral trajectories at 0.100.10 to 0.300.30 m/s with up to Nd=5N_d=5 simultaneous movers, including crossing and head-on encounters (Fig. 4). The HL policy reduces vxv_x and issues a yaw-dominant command to let an obstacle pass, then re-accelerates, while the L policy preserves gait stability. Fig. 3: Static-obstacle navigation scenario. Fig. 4: Dynamic-obstacle navigation scenario. Fig. 5 shows joint training curves under three difficulty conditions. Each curve is the mean evaluation return with a shaded band denoting per-evaluation variability of the single run (seed 42). The dynamic-obstacle condition converges more slowly since the policy must additionally learn velocity-aware avoidance through the TTC term of Eq. (9). Fig. 5: Joint training curves under three conditions of increasing difficulty. Shaded bands denote per-evaluation variability. TABLE I: Comparison with classical navigation baselines over 100 trials per method per condition. Path length and time are averaged over successful episodes only. Static Dynamic Method Succ. (%) Coll. (%) Path (m) Time (s) Succ. (%) Coll. (%) Path (m) Time (s) SAC+A⋆ [2] 76.0 2.0 7.64 13.25 68.0 20.0 7.76 13.55 SAC+RRT⋆ [3] 78.0 22.0 7.98 13.95 68.0 32.0 7.27 12.72 SAC+APF [4] 63.0 37.0 6.87 14.27 55.0 45.0 6.80 13.12 HRL (ours) 98.0 2.0 7.92 13.74 88.0 12.0 7.90 13.48 V-C Comparison with planner-based hybrids Table I compares the learned HL policy against the three hybrids. On static maps the proposed method succeeds in 98.0% of trials against a range of 76.0% to 78.0% for the global planners. In dynamic scenarios the gap widens: the learned policy retains 88.0% success, whereas A⋆ and RRT⋆ collide in 20.0% and 32.0% of trials and APF drops to 55.0% success with 45.0% collisions. Path lengths (7.92 m static, 7.90 m dynamic) remain within 4% of the A⋆ reference, so the safety gains are not bought with detours. Per-episode averages are computed over successful episodes only, so low-success methods report optimistically short paths. V-D Ablation study Fig. 6 reports success, collision, and fall or timeout rates for the full model and each ablated variant. Removing the dynamic-obstacle velocity channel or obstacle positions from Eq. (4) degrades both regimes severely. Removing the TTC or proximity penalty of Eq. (9) has a comparable effect, confirming anticipatory and proximal safety signals are complementary. Removing the posture terms of Eq. (8) shifts failures toward falls, and removing the shaping clip of Eq. (5) destabilizes training outright (90.0% fall/timeout, static). Across variants the dominant failure mode is fall or timeout rather than collision, indicating each ablation harms the navigation-balance coupling as a whole. Fig. 6: Ablation of observation channels and reward terms in (a) static and (b) dynamic environments. VI Discussion and Limitations The learned navigation layer’s advantage rests on three mechanisms. First, it commands only what the gait can execute: joint training and Eq. (8) let it internalize the platform’s gait-limited, asymmetric command set, whereas the pure-pursuit trackers of A⋆ and RRT⋆ demand infeasible turns, surfacing as falls and timeouts even on static maps. Second, it anticipates rather than reacts: the TTC term of Eq. (9) yields before proximity alone would trigger, whereas global planners see moving obstacles only at replanning instants, too late for a robot needing several steps to change heading, explaining their dynamic collision rates of 20.0% and 32.0%, respectively. APF reacts instantly but on instantaneous positions only, so it oscillates and stalls in local minima, consistent with its 45.0% dynamic collision rate.explaining their dynamic collision rates of 20.0% and 32.0%, respectively. APF reacts instantly but on instantaneous positions only, so it oscillates and stalls in local minima, consistent with its 45.0% dynamic collision rate. Third, the hierarchy assigns each problem to its natural timescale, and the ablation, in which every removed component shifts failures toward falls or timeouts rather than collisions, shows the advantage comes from this coupled structure rather than any single term. Several limitations remain in this work. As this study is simulation-only. Sim-to-real transfer would require addressing model mismatch, sensing noise, and inter-level latency. The raycasts and dynamic-obstacle states of Eq. (4) are privileged simulator quantities standing in for a LiDAR and object tracker, chosen to make that substitution direct, though onboard perception remains future work. The planning layer assumes a known static map, shared symmetrically by our method and the baselines. Because the L gait is co-trained with the HL policy, the frozen gait reused by baselines may track their commands slightly less faithfully. Results use a single training seed, so seed variance is unquantified, and the policy actuates only leg joints, leaving upper-body and multi-contact avoidance open. VII Conclusion This paper presented a hierarchical reinforcement learning framework for collision-free locomotion of an underactuated bipedal robot in static and dynamic environments. A high-level SAC policy maps pose, raycast, dynamic-obstacle, and local-goal observations to body-velocity commands, and a velocity-conditioned low-level SAC gait policy executes each command over ten control steps, with both trained jointly at their respective timescales. The shared command interface enabled a controlled comparison against SAC+A⋆, SAC+RRT⋆, and SAC+APF hybrids driving the identical learned gait, in which the proposed method attained 98.0% success in static and 88.0% in dynamic scenarios within 4% of the shortest-path reference in path length. Ablations established that the TTC term, proximity penalty, posture shaping, and reward clip each contribute materially to this performance. Future work will address onboard perception, seed-variance quantification, and sim-to-real transfer. References [1] M. Hamze, M. Morisawa, and E. Yoshida, “Learning bipedal walking for humanoid robots in challenging environments with obstacle avoidance,” in Proc. JSME Conf. Robot. Mechatron. (ROBOMECH), 2024, arXiv:2410.08212. [2] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” IEEE Trans. Syst. Sci. Cybern., vol. 4, no. 2, p. 100–107, 1968. [3] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” Int. J. Robot. Res., vol. 30, no. 7, p. 846–894, 2011. [4] O. Khatib, “Real-time obstacle avoidance for manipulators and mobile robots,” Int. J. Robot. Res., vol. 5, no. 1, p. 90–98, 1986. [5] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,” IEEE Robot. Autom. Mag., vol. 4, no. 1, p. 23–33, 1997. [6] J. Liu, M. Li, J. W. Grizzle, and J.-K. Huang, “CLF-CBF constraints for real-time avoidance of multiple obstacles in bipedal locomotion and navigation,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2023, p. 10 497–10 504. [7] L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2017, p. 31–36. [8] R. Cimurs, I. H. Suh, and J. H. Lee, “Goal-driven autonomous exploration through deep reinforcement learning,” IEEE Robot. Autom. Lett., vol. 7, no. 2, p. 730–737, 2022. [9] Y. F. Chen, M. Everett, M. Liu, and J. P. How, “Socially aware motion planning with deep reinforcement learning,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2017, p. 1343–1350. [10] J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2021, p. 7309–7315. [11] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Real-world humanoid locomotion with reinforcement learning,” Science Robotics, vol. 9, no. 89, p. eadi9579, 2024. [12] R. P. Singh et al., “Learning bipedal walking on planned footsteps for humanoid robots,” in Proc. IEEE-RAS Int. Conf. Humanoid Robots (Humanoids), 2022, p. 686–693. [13] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Proc. Conf. Robot Learn. (CoRL), ser. PMLR, vol. 164, 2021, p. 91–100. [14] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022. [15] J. P. Sahoo, S. P. S.K., D. Prajapati, K. Pant, A. Dwivedi, and A. Shukla, “Learning multi-skill locomotion in underactuated biped: A waypoint-based reward shaping approach,” in Proc. Eleventh Indian Control Conference (ICC), Bangalore, India, 2025, p. 370–375. [16] R. S. Sutton, D. Precup, and S. Singh, “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” Artif. Intell., vol. 112, no. 1–2, p. 181–211, 1999. [17] O. Nachum, S. Gu, H. Lee, and S. Levine, “Data-efficient hierarchical reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2018, p. 3303–3313. [18] X. B. Peng, G. Berseth, K. Yin, and M. van de Panne, “DeepLoco: Dynamic locomotion skills using hierarchical deep reinforcement learning,” ACM Trans. Graph., vol. 36, no. 4, p. 1–13, 2017. [19] D. Jain, A. Iscen, and K. Caluwaerts, “Hierarchical reinforcement learning for quadruped locomotion,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2019, p. 7551–7557. [20] A. Shamsah et al., “EmoBipedNav: Emotion-aware social navigation for bipedal robots with deep reinforcement learning,” 2025, arXiv:2503.12538. [21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proc. Int. Conf. Machine Learning (ICML), ser. PMLR, vol. 80, 2018, p. 1856–1865. [22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017, arXiv:1707.06347.