Paper deep dive
PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots
Amr Mousa, Rifny Rachman, Neil Karavis, Michele Caprio, Richard Allmendinger
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/3/2026, 2:31:02 AM
Summary
The paper introduces PROMO, a Preference-Conditioned Multi-Objective Reinforcement Learning framework for quadrupedal robots. It separates semantic objectives (tracking, stability, efficiency) from fixed locomotion priors, allowing a single policy to adapt behavior at runtime via preference inputs. PROMO demonstrates robust zero-shot transfer to a Unitree Go2 robot, significantly improving energy efficiency and tracking accuracy compared to fixed-objective controllers.
Entities (8)
Relation Signals (7)
PROMO → transfersto → Unitree Go2
confidence 98% · The same policy transfers zero-shot to a Unitree Go2
PROMO → achieves → zero-shot transfer
confidence 95% · C3 Zero-shot semantic morl on a real quadruped. We provide... the first real-robot demonstration... A single policy transfers zero-shot to a Unitree Go2
PROMO → uses → Multi-Objective Reinforcement Learning
confidence 95% · We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning)...
PROMO → optimizes → Energy Efficiency
confidence 90% · the semantic reward vector is ... where the objectives correspond to ... energetic efficiency.
PROMO → optimizes → Tracking
confidence 90% · the semantic reward vector is ... where the objectives correspond to command tracking...
PROMO → optimizes → Stability
confidence 90% · the semantic reward vector is ... where the objectives correspond to ... body stability...
PROMO → approximates → Pareto Front
confidence 85% · promo instead learns a weight-conditioned approximation of the supported Pareto front
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.01260v1
- Canonical: https://arxiv.org/abs/2610.01260v1
Trouble viewing inline? Open PDF directly →
Full Text
94,092 characters extracted from source content.
Expand or collapse full text
PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots Amr Mousa Affiliation: The University of Manchester, United Kingdom. Rifny Rachman Affiliation: The University of Manchester, United Kingdom. Neil Karavis Affiliation: BAE Systems, United Kingdom. Michele Caprio Affiliation: The University of Manchester, United Kingdom. Affiliation: University of Warwick, United Kingdom. and Richard Allmendinger †thanks: Project website, code, and videos: https://amrmousa.com/promo/.This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Affiliation: The University of Manchester, United Kingdom. Abstract Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hard-codes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective framework that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment-facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward-shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100100 sampled preferences in simulation, 6767 behaviors are non-dominated under exact Pareto dominance, with a mean preference–objective correlation of 0.8430.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%30.4\%, position error by 38.7%38.7\%, and peak body-attitude deviation by 59.0%59.0\% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/. Index Terms: Legged robots, multi-objective reinforcement learning, quadrupedal locomotion, semantic objectives, preference conditioning, interpretable control, sim-to-real transfer. †aftertitle: I Introduction Robotic locomotion via rl (rl) has moved quadrupeds from controlled demonstrations toward agile and deployable systems [1, 2, 3]. Most learned controllers, however, optimize a scalar reward that fixes the trade-off among command tracking, stability, energy efficiency, and locomotion regularization. This is restrictive in long-horizon autonomy, where terrain, payload, mission urgency, operator intent, or hardware state can change which objective should dominate. After training, a robot can receive new motion commands, but the objective trade-off governing how those commands are executed remains locked. Changing that trade-off usually requires a reward redesign, policy retraining, or controller switching. The deployment challenge is therefore to learn robust locomotion while making the underlying objective priorities selectable after training. morl (morl) provides a principled basis for adapting trade-offs by representing distinct objectives through vector-valued rewards and optimizing their utility under a preference [4, 5, 6]. However, directly treating locomotion reward terms as independent objectives creates a fundamental mismatch with learned locomotion. Robotic rewards typically combine many shaping terms for posture, contact timing, smoothness, joint limits, and other requirements for viable motion [2, 3, 7]. Making each term an objective yields a high-dimensional Pareto space that becomes harder to optimize as dimensionality grows [8], while exposing low-level coefficients that do not express behavioral intent. This objective-definition problem is central to deployable robotic morl: the optimized dimensions need not coincide with the components used to shape a robust controller. We therefore separate a compact semantic objective vector—tracking, stability, and efficiency—from fixed locomotion priors. This keeps embodiment-specific requirements internal to the controller and makes the preference vector a meaningful runtime interface rather than a reformulation of the full reward function. Behavior-conditioned locomotion addresses flexibility through a different interface. Methods such as Walk These Ways [9] expose gait descriptors that specify how the robot should move through morphology, timing, or stance parameters. Preference-conditioned control instead specifies what the robot should prioritize, leaving the physical gait realization to the policy. This distinction matters for deployment: semantic preferences should retain their meaning across commands, disturbances, and operating conditions rather than become handcrafted gait-tuning variables. These observations motivate promo (promo), a preference-conditioned morl framework that turns a fixed reward trade-off into a runtime semantic behavior interface. A single policy is conditioned on preferences over three deployment-facing objectives—tracking, stability, and efficiency—while embodiment-specific locomotion priors remain fixed. The resulting preference vector is both a deployment-time control input and an interpretable representation of behavioral intent. Rather than manipulating latent gait codes or implementation-level reward weights, the operator specifies why behavior should change—for example, by prioritizing stability over tracking—and the policy learns how to realize that trade-off. We evaluate whether this interface is controllable in simulation and remains preference-responsive after zero-shot transfer to hardware. In simulation, promo remains competitive at the balanced operating point while providing tracking, stability, and efficiency specialization from a single policy, with lower failure rates than the evaluated preference-conditioned and specialist baselines. A dense 100-preference study shows that these anchor behaviors lie within a broader controllable interface. On a Unitree Go2, changing only the preference produces large, directionally consistent changes in tracking, stability, and energy efficiency across five trials per condition. To our knowledge, this is the first real demonstration of semantic preference-conditioned morl on a robotic platform, moving morl from offline Pareto optimization toward an operator-facing mechanism for adaptive robot behavior. Our core contributions are: C1 A semantic locomotion interface. We formulate runtime-adjustable quadruped control as preference-conditioned morl over tracking, stability, and efficiency, making the objective trade-off an explicit deployment-time input that represents operator intent. C2 Objective/prior factorization for controllable locomotion. We address the mismatch between vector-valued morl objectives and dense robotic reward design by separating operator-facing semantic objectives from fixed embodiment-level locomotion priors. Ablations show that this factorization is critical for controllability and reliability: the fixed-prior formulation attains a 6%6\% fall rate under challenging evaluation conditions, whereas alternative designs increase it to as much as 21.8%21.8\% and degrade preference controllability. C3 Zero-shot semantic morl on a real quadruped. We provide, to our knowledge, the first real-robot demonstration of semantic preference-conditioned morl for locomotion. A single policy transfers zero-shot to a Unitree Go2 and modulates tracking, stability, and efficiency through preference alone. Relative to a balanced configuration, preference modulation reduces specific energy by up to 30.4%30.4\%, position error by up to 38.7%38.7\%, and peak body-attitude deviation by up to 59.0%59.0\%. C4 Open-source implementation and complete experimental suite. We release the implementation of promo together with all evaluated baselines, experimental tasks, evaluation protocols, and ablation studies, enabling reproduction of the complete experimental pipeline and facilitating future benchmarking and extensions. The remainder of this paper is organized as follows. Section I reviews related work. Section I formulates promo and describes its training and deployment. Sections IV and V evaluate the framework in simulation and on the Unitree Go2 robot, respectively. Section VI discusses the implications and limitations, and Section VII concludes the paper. I Related Work This section positions promo relative to learned quadruped locomotion, reward design and runtime behavioral interfaces, preference-conditioned morl, and the use of Pareto coverage for behavioral control. I-A Learned Quadruped Locomotion rl-based locomotion has made quadruped control increasingly robust to sim-to-real transfer, partial observability, and environmental uncertainty. Actuator modeling [10], domain randomization [11], privileged adaptation [1], massively parallel training [2], perception-aware control [3, 12], implicit adaptation [13, 14], and teacher-aligned representations [7] have enabled reliable deployment under challenging conditions. Most controllers nevertheless remain sorl (sorl) policies: commands and observations may change at runtime, but the reward-level priority among tracking, stability, efficiency, and regularization is fixed. This raises a question: which parts of the locomotion objective should remain fixed, and which should be exposed for runtime adaptation? I-B Reward Design and Runtime Behavioral Interfaces Modern locomotion rewards mix task objectives with posture regulation, contact shaping, actuator smoothness, joint-limit penalties, and other embodiment-specific terms [2, 3, 7]. These dense components suppress different failure modes and collectively shape viable motion, but they do not all represent objectives over which an operator should express a preference. Treating contact regularity, posture, or action smoothness as independent objectives would expose reward-engineering decisions as deployment controls and enlarge the preference space whenever another shaping term is introduced. Reward shaping [15], constrained rl [16], constrained morl [17], and inverse reward design [18] similarly distinguish task objectives from auxiliary signals or constraints. promo operationalizes this distinction through semantic objective/prior factorization: task-level trade-offs define the preference vector, whereas embodiment-level requirements remain fixed locomotion priors. Runtime flexibility can also be achieved through behavior conditioning [9, 19, 20]. These methods show that one policy can span diverse behaviors through operator-controlled gait variables, but the interface primarily specifies how the robot should move through morphology or timing. Semantic preference conditioning instead specifies what the robot should prioritize. This shifts the problem from commanding behaviors to trading off objectives: how can a single policy realize a continuum of semantic preferences? I-C MORL Preference Conditioning morl formalizes control with multiple conflicting objectives through vector-valued returns [4], scalarization and utility theory [6], and Pareto optimality [5]. The algorithmic literature includes Pareto-front approximation [21], decomposition [22], constrained optimization [23], diversity-driven coverage [24], and preference-conditioned policy learning [25, 26]. Standardized frameworks such as MO-Gymnasium and MORL-Baselines support systematic comparison [27, 28]. Preferences may be introduced a priori, a posteriori, or interactively, and expressed through weights, reference points, or solution comparisons [29]. A fixed a priori weight vector yields one scalarized solution; promo instead learns a weight-conditioned approximation of the supported Pareto front, enabling a posteriori behavior selection at deployment. Preference-conditioned single-policy methods are attractive for deployment because one policy can represent a continuum of trade-offs. They are based on universal value function approximation [30] and reward-conditioned rl [31, 32]. In our evaluation, moppo (moppo) provides a single-policy preference-conditioned ppo (ppo) baseline [33], whereas dpmorl (dpmorl) represents a multi-policy route to Pareto-front approximation [34]. Robotics applications remain comparatively limited, including prediction-guided multi-objective continuous control [21], constrained robot control [35], and gradient-conflict resolution [36]. These works establish the relevance of morl to robotics, but do not address how preference-conditioned optimization should be formulated and deployed for contact-rich legged robots. This leaves an interface-level question: does learning a broad Pareto set make preferences useful as deployment-time controls? I-D From Pareto Coverage to Behavioral Control Standard morl benchmarks ignore failures, dense reward structure, embodiment-specific priors, sim-to-real transfer, and hardware deployment. Strong Pareto performance therefore does not necessarily imply that preferences provide a reliable control interface on a physical robot. Hypervolume is the principal Pareto-front approximation metric used in our study. It is Pareto compliant and measures the objective-space volume dominated by an approximation set and bounded by a reference point, summarizing convergence and diversity [37]. However, it does not establish whether changing a preference predictably changes behavior. Recent work therefore considers preference sensitivity, monotonicity, objective alignment, and behavioral consistency [38, 39, 40]. A policy may approximate a broad Pareto front while still providing an unreliable interface if preferences map inconsistently to realized behaviors. For physical deployment, Pareto quality must be evaluated together with preference controllability, robustness, and semantic interpretability. promo combines semantic objective specification, preference-conditioned control, locomotion-specific priors, interface-oriented evaluation, and physical deployment in a single framework. I Methodology: Semantic Preference-Conditioned Locomotion This section formulates promo and describes its architecture, semantic objective factorization, preference-conditioned optimization, and mechanisms for responsive deployment. I-A Problem Formulation and Semantic Interface We model locomotion as a partially observed multi-objective decision process. Let oto_t denote the current deployable observation, including proprioception and the velocity command, let ot−H:to_t-H:t denote its history, and let stprivs_t^priv collect the corresponding simulator-only state used during training. The preference is wt∈ΔK−1w_t∈ ^K-1. The history encoder EHE_H and velocity estimator produce ztH=EH(ot−H:t,wt),v^tb=EV(ztH,ot−4:t),z_t^H=E_H(o_t-H:t,w_t), v_t^b=E_V(z_t^H,o_t-4:t), (1) where v^tb v_t^b estimates the three-dimensional base linear velocity. The deployed policy outputs ata_t according to πθ(at∣ot,ztH,v^tb,wt) _θ(a_t o_t,z_t^H, v_t^b,w_t). The stop-gradient operations used during optimization are stated in Section I-B. For quadrupedal locomotion, the semantic reward vector is tsem=[rttrack,rtstab,rteff]⊤,r_t^sem= [r_t^track,r_t^stab,r_t^eff ] , (2) where the objectives correspond to command tracking, body stability, and energetic efficiency. Let ℐ=track,stab,effI=\track,stab,eff\ denote the semantic objective index set, with K=|ℐ|=3K=|I|=3, and let ri,tr_i,t denote component i∈ℐi of tsemr_t^sem. Semantic utility is defined as U(tsem,wt)=wt⊤tsem.U(r_t^sem,w_t)=w_t r_t^sem. (3) During training, semantic utility is augmented by a fixed locomotion prior, ut(wt)=wt⊤tsem+λpriorrtprior,u_t(w_t)=w_t r_t^sem+ _priorr_t^prior, (4) where rtpriorr_t^prior contains embodiment-level regularization terms shared across all preferences and λprior _prior is its fixed coefficient. For the discount factor γ, the semantic and prior returns are Jsem(πθ,w)=πθ,w[∑t≥0γttsem],J_sem( _θ,w)=E_ _θ,w [ _t≥ 0γ^tr_t^sem ], (5) Jprior(πθ,w)=πθ,w[∑t≥0γtrtprior].J_prior( _θ,w)=E_ _θ,w [ _t≥ 0γ^tr_t^prior ]. (6) Because all semantic objectives are maximized, we use the following Pareto terminology [8, 5]. Let Π be the feasible policy class and let (π)∈ℝKJ(π) ^K denote the semantic return vector of π∈Ππ∈ as defined in Eq. (5). A policy πaπ^a Pareto dominates πbπ^b, written πa≻Pπbπ^a _Pπ^b, if Ji(πa) J_i(π^a) ≥Ji(πb) ≥ J_i(π^b) ∀i∈ℐ, ∀ i , (7) Jj(πa) J_j(π^a) >Jj(πb) >J_j(π^b) for some j∈ℐ. some j . A policy is Pareto optimal if no feasible policy dominates it. The Pareto set ⊆ΠP contains these policies in the policy space, whereas the Pareto front ℱ=(π):π∈F=\J(π):π \ contains their return vectors in the semantic objective space. Within a finite evaluated set, a policy is non-dominated if no other evaluated policy dominates it; this finite set is therefore a sampled approximation to, rather than proof of, the unknown true Pareto front. For the learned conditional actor, each preference induces πw(a∣o,zH,v^b)=πθ(a∣o,zH,v^b,w) _w(a o,z^H, v^b)= _θ(a o,z^H, v^b,w) and the corresponding outcome (πw)=Jsem(πθ,w)J( _w)=J_sem( _θ,w). A single preference is sampled per environment from the truncated-simplex distribution and kept fixed within each rollout. Preference changes occur only during deployment or evaluation. Implementation details are provided in Section IV-A. The semantic objective space contains only deployment-facing dimensions whose relative importance is specified by the user. Embodiment-level regularizers, including contact timing, contact balance, and posture constraints, are represented by the fixed prior rtpriorr_t^prior. This factorization prevents dense reward components from becoming artificial objective dimensions. Thus, w represents behavioral intent rather than implementation-level reward weights, and Pareto and preference-controllability analyses are defined exclusively over tsemr_t^sem. At deployment, a human operator, planner, or supervisory controller supplies wt∈ΔK−1w_t∈ ^K-1 to the fixed policy πθ(at∣ot,ztH,v^tb,wt) _θ(a_t o_t,z_t^H, v_t^b,w_t). In all reported evaluations, each preference is kept fixed throughout a rollout, directly associating the realized behavior with the requested semantic trade-off. Appendix A gives the full reward decomposition. I-B Architecture Overview promo extends the locomotion architecture of tar (tar) [7] to preference-conditioned morl with semantic objectives. Its privileged encoder EPE_P and history encoder EHE_H produce distinct latents: ztP=EP(stpriv,wt),ztH=EH(ot−H:t,wt).z_t^P=E_P(s_t^priv,w_t), z_t^H=E_H(o_t-H:t,w_t). (8) Here, stprivs_t^priv comprises the current deployable observation, ground-truth base velocity, and the additional privileged simulator signals. Both encoders are preference-conditioned, as shown in Fig. 2. During training, the critic receives (ot,vtb,ztP)(o_t,v_t^b,z_t^P) and predicts one value for each semantic objective. It does not receive ztHz_t^H; the preference enters its representation through ztPz_t^P and is applied explicitly when the objective-wise values are scalarized. In contrast, the actor receives (ot,ztH,v^tb,wt)(o_t,z_t^H, v_t^b,w_t) and outputs 1212 joint-position targets. The history representation is aligned with a detached privileged target using ℒalign=t[‖ztH−sg[ztP]‖22],L_align=E_t\! [ z_t^H-sg[z_t^P] _2^2 ], (9) where sg[⋅]sg[·] denotes stop-gradient. The critic loss updates the privileged encoder, whereas Eq. (9) does not propagate into it. The velocity estimator uses the history latent and the latest five observations, as defined in Eq. (1), and minimizes ℒvel=t[‖v^tb−vtb‖22].L_vel=E_t\! [ v_t^b-v_t^b _2^2 ]. (10) The history encoder is updated by λalignℒalign+ℒvel _alignL_align+L_vel with λalign=1.0 _align=1.0, and the velocity estimator is updated by ℒvelL_vel. Actor-side inputs use sg[ztH]sg[z_t^H] and sg[v^tb]sg[ v_t^b], preventing policy gradients from entering either auxiliary module. The privileged encoder nevertheless receives gradients from the semantic critic loss through ztPz_t^P. Compared with tar, promo replaces the recurrent history encoder with a multilayer-perceptron encoder, removes the forward-dynamics auxiliary model, and introduces preference conditioning throughout the actor–critic architecture. The result is a lightweight architecture for preference-conditioned flat-terrain locomotion that retains privileged representation alignment. Figure 2 summarizes training and deployment; network, simulator, and optimization details are provided in Appendix B. Fig. 2: promo architecture and training loop. The privileged latent ztPz_t^P enters the critic, whereas the history latent ztHz_t^H and estimated velocity v^tb v_t^b enter the actor through stop-gradient operations. Both encoders receive wtw_t. The semantic critic values are scalarized by wtw_t before the clipped ppo update. The shaded lower path contains the modules retained at deployment: the history encoder, velocity estimator, and actor; the privileged encoder and critic are discarded. Algorithm 1 summarizes the resulting update paths. It separates rollout inference from the privileged quantities and auxiliary targets used only during optimization. Algorithm 1 promo training and deployment 0: Actor πθ _θ, critic VψV_ψ, encoders EP,EHE_P,E_H, velocity estimator EVE_V 1: for each training rollout do 2: Sample w and hold it fixed for the rollout 3: for each step t do 4: ztH←EH(ot−H:t,w)z_t^H← E_H(o_t-H:t,w); v^tb←EV(ztH,ot−4:t) v_t^b← E_V(z_t^H,o_t-4:t) 5: at∼πθ(⋅∣ot,sg[ztH],sg[v^tb],w)a_t _θ(· o_t,sg[z_t^H],sg[ v_t^b],w) 6: Step the environment and store ot,stpriv,vtb,tsem,rtprioro_t,s_t^priv,v_t^b,r_t^sem,r_t^prior 7: end for 8: ztP←EP(stpriv,w)z_t^P← E_P(s_t^priv,w) and evaluate Viψ(ot,vtb,ztP)V_i^ψ(o_t,v_t^b,z_t^P) for all i∈ℐi 9: Form semantic values, GAE advantages, ℒsemL_sem, ℒPPOL_PPO, and ℒdivactL_div^act 10: Form ℒalign=∥ztH−sg[ztP]∥22L_align= z_t^H-sg[z_t^P] _2^2 and ℒvel=∥v^tb−vtb∥22L_vel= v_t^b-v_t^b _2^2 11: Update (ψ,EP)(ψ,E_P) with ℒsemL_sem 12: Update EHE_H with λalignℒalign+ℒvel _alignL_align+L_vel 13: Update EVE_V with ℒvelL_vel 14: Update θ with ℒPPO+λdivℒdivactL_PPO+ _divL_div^act 15: end for 16: Deploy: retain (EH,EV,πθ)(E_H,E_V, _θ); discard (EP,Vψ)(E_P,V_ψ) I-C Semantic Objective Factorization and Value Learning Using the returns in Eqs. (5)–(6), the training objective for preference wtw_t is Jtrain(θ,wt)=wt⊤Jsem(πθ,wt)+λpriorJprior(πθ,wt),J_train(θ,w_t)=w_t J_sem( _θ,w_t)+ _priorJ_prior( _θ,w_t), (11) where λprior=1 _prior=1 in all experiments and is independent of wtw_t. The individual prior coefficients are listed in Appendix A. Pareto, controllability, and preference-response evaluations are computed only from JsemJ_sem; the prior regularizes locomotion but is not exposed as a preference axis. Appendix D states the corresponding design property formally. The semantic critic has one learned value head per semantic objective, Viψ(ot,vtb,ztP),i∈ℐ,V^ψ_i(o_t,v_t^b,z_t^P), i , (12) where ψ denotes the critic parameters and ViψV^ψ_i approximates the expected discounted return for semantic objective i. The heads depend on the preference through the privileged latent ztP=EP(stpriv,wt)z_t^P=E_P(s_t^priv,w_t), but estimate only semantic returns. For the terminal indicator dtd_t, the one-step ppo critic target is yi,t=ri,t+γ(1−dt)Viψ(ot+1,vt+1b,zt+1P),y_i,t=r_i,t+γ(1-d_t)V^ψ_i(o_t+1,v_t+1^b,z_t+1^P), (13) and the semantic critic minimizes the objective-wise regression loss ℒsem(ψ)=1K∑i∈ℐ(yi,t−Viψ(ot,vtb,ztP))2.L_sem(ψ)= 1K _i (y_i,t-V^ψ_i(o_t,v_t^b,z_t^P) )^2. (14) The fixed prior has no value head; see the objective/prior ablation in Appendix D and Table D.1. I-D Preference-Conditioned Policy Optimization For preference wtw_t, promo scalarizes the semantic value heads as Vwtψ(ot,vtb,ztP)=∑i∈ℐwt,iViψ(ot,vtb,ztP).V_w_t^ψ(o_t,v_t^b,z_t^P)= _i w_t,iV^ψ_i(o_t,v_t^b,z_t^P). (15) For actor optimization, the semantic reward is scalarized as rtsem(wt)=wt⊤tsemr^sem_t(w_t)=w_t r_t^sem. promo then combines this reward with the fixed prior and the preference-scalarized semantic critic to estimate the actor advantage with gae (gae) parameter λGAE _GAE, where ℓ indexes future rollout steps: At A_t =∑ℓ≥0(γλGAE)ℓ(δt+ℓ), = _ ≥ 0(γ _GAE) \! ( _t+ ), (16) δt _t =rtsem(wt)+λpriorrtprior =r^sem_t(w_t)+ _priorr_t^prior +γ(1−dt)Vwt+1ψ(ot+1,vt+1b,zt+1P) +γ(1-d_t)V_w_t+1^ψ(o_t+1,v_t+1^b,z_t+1^P) −Vwtψ(ot,vtb,ztP). -V_w_t^ψ(o_t,v_t^b,z_t^P). Because no prior value function is learned, the prior term enters this residual with a zero baseline; the only baseline subtraction is the scalarized semantic critic. Its contribution is therefore estimated through the λGAE _GAE-discounted gae trace. For λGAE=1 _GAE=1, this reduces to the Monte-Carlo prior return; for the setting used here (0.950.95), it is the standard biased, lower-variance gae estimator rather than an unbiased estimate of the full prior return. The actor update uses the advantage in Eq. (16). Define the complete actor input as xtA=(ot,sg[ztH],sg[v^tb],wt)x_t^A=(o_t,sg[z_t^H],sg[ v_t^b],w_t). Then ϱt(θ)=πθ(at∣xtA)πθold(at∣xtA),ϱ¯t=clip(ϱt,1−ϵ,1+ϵ), _t(θ)= _θ(a_t x_t^A) _ _old(a_t x_t^A), _t=clip( _t,1-ε,1+ε), where θold _old denotes the behavior-policy parameters used to collect the rollout and ϵε is the ppo clipping width. The actor minimizes the negative clipped ppo surrogate ℒPPO=−t[min(ϱtAt,ϱ¯tAt)].L_PPO=-E_t [ ( _tA_t, _tA_t ) ]. (17) The optimizer constants are listed in Appendix B. I-E Preference Responsiveness and Deployment Conditioning the actor on wtw_t enables preference-dependent behavior, but does not guarantee visible use of the preference input. promo therefore adds a small action-diversity regularizer. For two preferences wtw_t and wt′w _t evaluated at the same observation, detached history latent, and detached velocity estimate, let x~tA=(ot,sg[ztH],sg[v^tb]) x_t^A=(o_t,sg[z_t^H],sg[ v_t^b]) and define πt=πθ(⋅∣x~tA,wt),πt′=πθ(⋅∣x~tA,wt′). _t= _θ(· x_t^A,w_t), π _t= _θ(· x_t^A,w _t). The regularizer is ℒdivact=−[∥wt−wt′∥Dsym(πt,πt′)],L_div^act=-E [ w_t-w _t D_sym( _t,π _t) ], (18) where DsymD_sym is the symmetric KL divergence between the two diagonal-Gaussian action distributions, Dsym(πt,πt′)=12[DKL(πt∥πt′)+DKL(πt′∥πt)],D_sym( _t,π _t)= 12 [D_KL( _t\|π _t)+D_KL(π _t\| _t) ], (19) summed across action dimensions. The resulting actor objective is ℒactor=ℒPPO+λdivℒdivact,L_actor=L_PPO+ _divL_div^act, (20) with λdiv=10−5 _div=10^-5. Appendix D evaluates the coefficient sensitivity and the latent-diversity alternative. At deployment, the history encoder EHE_H, velocity estimator EVE_V, and actor πθ _θ are retained. The privileged encoder, privileged state, ground-truth base velocity, and critic are discarded. The retained modules require only (ot−H:t,wt)(o_t-H:t,w_t), and wtw_t is the sole runtime mechanism for behavior modulation. IV Experimental Evaluation This section evaluates whether one preference-conditioned policy can provide a controllable semantic locomotion interface while retaining robust quadrupedal behavior. We organize the experiments around four questions: Q1: Training comparison. How does promo optimize locomotion relative to single-objective controllers, existing morl methods, and independently trained specialists under training metrics available to all methods? (Section IV-C) Q2: Common semantic evaluation. Under common semantic metrics, does promo preserve balanced locomotion quality and provide a useful multi-objective interface across requested behaviors? (Section IV-D) Q3: Preference-interface structure. For the deployed promo policy, does varying the preference produce predictable semantic responses, and is Pareto coverage sufficient to characterize the interface? (Section IV-E and Appendix D) Q4: Hardware deployment. Can the same preference-conditioned policy modulate physical locomotion through changes in priorities over the semantic objectives alone, and which failure modes appear on hardware? (Section V) IV-A Setup and Baselines All policies are trained in Isaac Sim/IsaacLab with a 1212- dof (dof) Unitree Go2 quadruped and 40964096 parallel flat-terrain environments. Domain randomization covers friction, base mass and inertia, center-of-mass offsets, external perturbations, and joint-state initialization to support robustness and zero-shot hardware transfer. The complete network, simulation, control, and domain-randomization configuration is provided in Appendix B. We compare promo against three baseline groups: • Single-objective locomotion controllers: rma (rma) [1] and tar [7], trained using the balanced semantic objective configuration. This group represents the default locomotion setting in which a single reward defines a fixed trade-off among conflicting objectives. • morl methods: moppo [33], a single-policy preference-conditioned baseline, and dpmorl [34], a multi-policy baseline that approximates the Pareto front using a set of specialized policies. • promo-sorl: Three per-objective scalar specialists derived from the promo architecture. They serve as a reference for the cost of amortizing multiple behaviors into a single preference-conditioned policy. These comparisons test whether one promo policy can provide controllable multi-objective locomotion while remaining competitive with established locomotion controllers, existing morl approaches, and specialized scalar policies. Appendix E summarizes the baseline implementations and shared controls used to keep the comparison methodologically consistent. IV-B Evaluation Metrics We evaluate training performance, semantic multi-objective performance, and preference controllability. Semantic evaluations use shared scripted commands and disturbances so preference-dependent differences are compared under matched rollout conditions; Fig. C.1 illustrates the protocol, with full details in Appendix C. Training performance. We report the mean training reward and episode length over three independent seeds. These metrics characterize optimization under each method’s native training objective and are therefore reported separately from the common semantic evaluation. Semantic performance and robustness. All methods are evaluated on the same three semantic objectives: tracking, stability, and efficiency. The balanced evaluation reports semantic rewards, fall rate, and lower-tail body-attitude risk; the four-anchor comparison reports requested-objective scores, aggregate fall rate, and scalarized return. In Appendix C, Table C.1, lower-tail robustness is measured by cvar (cvar)10 and Pareto quality by hv (hv) after objective-range normalization. For method/run m, let m⊂ℝ3Y_m ^3 denote its semantic outcomes. We define common empirical bounds from the union of run-level non-dominated sets, ℛ=⋃mND(m),Jimin/max=min/max∈ℛJi,R= _mND(Y_m), J_i / = / _J J_i, (21) and normalize all methods using J^i=Ji−JiminJimax−Jimin,ri=Jimin−0.05(Jimax−Jimin). J_i= J_i-J_i J_i -J_i , r_i=J_i -0.05(J_i -J_i ). (22) Since all objectives are maximized, the common reference maps to ^=(−0.05,−0.05,−0.05) r=(-0.05,-0.05,-0.05). We compute hv from each normalized run-level non-dominated set in this common 3-D space. Only the objective ranges are normalized; hv itself is not, giving a bounding-box upper bound of 1.053=1.1576251.05^3=1.157625. Larger hv indicates greater Pareto coverage. Preference controllability. Beyond Pareto quality, controllability measures whether changing a preference produces the corresponding semantic response. For objective i, we compute ρi=Spearman(wi,Jisem) _i=Spearman(w_i,J_i^sem) [41] and average across objectives. We distinguish four-anchor controllability for baseline comparison from dense controllability for the 100-preference promo evaluation. Finally, promo is evaluated over 100100 preferences to characterize response trends, Pareto structure, and locomotion failures. Higher values are better for reward, utility, controllability, and range-normalized-objective hv; lower values are better for fall rate and body-attitude tail risk. IV-C Training and Baseline Comparison The simulation evaluation establishes the interface properties needed before hardware deployment, rather than treating high training reward as sufficient evidence. The results combine optimization behavior, common semantic metrics, and a standalone analysis of the policy’s controllability and Pareto coverage. The training comparison evaluates mean reward and episode length over three independent seeds, with one-standard-deviation bands (Fig. 3). These curves measure optimization under each method’s native objective; deployable interface quality is assessed separately in Tables I and I. Locomotion baselines. Against tar, rma, and the promo-sorl specialists, all methods rapidly acquire stable locomotion and reach the 10001000-step episode horizon. The single-objective controllers reach that horizon slightly earlier, but promo continues improving after locomotion has stabilized, converging to 27.527.5 reward with low seed variance, compared with 25.325.3 for promo-sorl, 24.824.8 for rma, and 24.224.2 for tar. This indicates that the preference-conditioned formulation can preserve stable locomotion while amortizing multiple semantic behaviors into one policy. morl baselines. promo and the single-policy moppo baseline learn similarly early, but promo continues to 27.527.5 reward compared with 23.523.5 for moppo, with lower variance. dpmorl shows four distinct optimization phases, one per separately trained policy, each transition resetting optimization. promo instead learns the preference space in one continuous run with a single deployable actor. Amortization against specialists. Specialist reward curves are not directly comparable because each policy optimizes a differently scaled scalar objective; episode length, with identical terminations, is the common axis. promo reaches the horizon after 550550–650650 steps, close to the balanced specialist (600600–700700) and tracking specialist (400400–500500), whereas the efficiency specialist converges much later (17001700–19001900) because effort minimization is a weak gait-discovery signal. Thus, promo learns the four semantic behaviors at roughly the optimization cost of one balanced controller, instead of training and deploying four separate policies. (a) Locomotion baselines: reward (b) morl baselines: reward (c) Specialists: reward (d) Locomotion baselines: episode length (e) morl baselines: episode length (f) Specialists: episode length Fig. 3: Training optimization over three seeds (mean, ±1± 1 s.d.). The top row reports mean reward and the bottom row reports mean episode length for locomotion baselines (a,d), morl baselines (b,e), and independently trained specialists (c,f). promo converges to a higher reward than the shared-scale locomotion and morl references while matching the episode-length convergence of the balanced specialist, learning the preference space at roughly the optimization cost of a single controller. Specialist rewards are shown for completeness but use differently scaled scalar objectives, so episode length is the more informative cross-specialist comparison. IV-D Common Semantic Evaluation Against Baselines Training reward is insufficient to validate a preference-conditioned locomotion interface. We therefore separate balanced locomotion quality from preference-interface quality across requested behaviors. Balanced locomotion quality. Table I evaluates the balanced operating point. promo records the highest mean tracking reward (18.70318.703), compared with 16.22516.225 for the closest method, promo-sorl. It also has a lower mean fall rate than tar and rma (7.4%7.4\% vs. 14.5%14.5\% and 18.2%18.2\%) and the lowest mean cvar tilt (0.1050.105). The reward rankings remain objective-dependent: tar has the highest mean aggregate stability reward, whereas rma has the highest mean efficiency reward. The results also suggest a potential benefit of preference-conditioned training. By training a shared actor across tracking-, stability-, and efficiency-biased preferences, promo is exposed to a broader behavioral repertoire that may benefit its balanced operating point. TABLE I: Common balanced locomotion evaluation. Higher is better for semantic rewards; lower is better for fall and cvar tilt. Fall values are percentages. Bold denotes the best mean in each column. Method Tracking ↑ Stability ↑ Efficiency ↑ Fall ↓ cvar tilt ↓ tar 14.92114.921 −2.218-2.218 −3.643-3.643 14.5%14.5\% 0.1160.116 rma 13.74213.742 −2.920-2.920 −3.423-3.423 18.2%18.2\% 0.1250.125 promo-sorl 16.22516.225 −2.804-2.804 −4.157-4.157 11.0%11.0\% 0.1090.109 promo (ours) 18.70318.703 −2.855-2.855 −5.366-5.366 7.4%7.4\% 0.1050.105 Four-anchor preference interface. Table I evaluates the balanced and three single-objective anchor requests. promo records higher mean requested tracking and efficiency than the specialists (20.09920.099 vs. 17.83317.833 and −2.189-2.189 vs. −2.942-2.942, respectively), while the stability specialist retains the higher mean stability score (−1.916-1.916 vs. −2.245-2.245). promo also records a 4.24.2-percentage-point lower aggregate fall mean relative to the specialist family (8.4%8.4\% vs. 12.6%12.6\%). The four-anchor results therefore show no clear numerical amortization penalty while demonstrating that one actor can express multiple semantic behaviors without matching every specialist extremum. Compared with moppo, promo records a higher mean scalarized return (3.9643.964 vs. 2.2122.212), an 11.011.0-percentage-point lower mean fall rate, and higher mean four-anchor controllability (0.6620.662 vs. 0.4920.492). dpmorl and promo have comparable mean requested efficiency (−2.133-2.133 vs. −2.189-2.189). dpmorl, however, records a substantially higher mean fall rate of 26.6%26.6\%. The comparison supports the intended deployment profile through the observed combination of objective specialization, robustness, preference response, and one-policy execution. TABLE I: Four-anchor semantic evaluation against morl and specialist baselines. Bold denotes the best mean in each column. Method Tracking T↑T Stability S↑S Efficiency E↑E Fall ↓ Scalarized return ↑ 4-anchor Ctrl. ↑ dpmorl 14.43014.430 −2.352-2.352 −2.133-2.133 26.6%26.6\% 2.9862.986 −−-- moppo 15.32615.326 −2.305-2.305 −2.794-2.794 19.4%19.4\% 2.2122.212 0.4920.492 Specialists 17.83317.833 −1.916-1.916 −2.942-2.942 12.6%12.6\% 3.2553.255 −−-- promo 20.09920.099 −2.245-2.245 −2.189-2.189 8.4%8.4\% 3.9643.964 0.6620.662 IV-E Pareto Coverage and Preference Controllability We evaluate promo over 100100 preference vectors sampled uniformly over the simplex (Figure C.2), with each preference rolled out across 10241024 parallel environments for 2020 s under external disturbances. Appendix C, Table C.1 summarizes the resulting Pareto quality and preference controllability. Pareto quality. The achieved semantic objective set provides a dense sampled Pareto-front approximation: 6767 of the 100100 preference-induced policies are non-dominated within the evaluated set under Eq. (7). Applying ε -dominance pruning following [42] (Appendix C-A, Table C.2) retains 3737 solutions at ε=1% =1\%, indicating a densely populated trade-off surface rather than isolated anchors. Tracking and efficiency form the clearest conflict, while stability is partly complementary to both. Preference controllability. The learned interface is strongly aligned with the requested preference: increasing a semantic weight generally corresponds to a higher realized return. The dense evaluation records 0.84340.8434 controllability and a 7.50%7.50\% mean fall rate, with failures concentrated near aggressive preference boundaries. Figure 4 summarizes the Pareto geometry and preference-to-behavior mappings for our promo policy. (a) Tracking vs. efficiency front approximation (b) Tracking vs. stability front approximation (c) Stability vs. efficiency front approximation (d) Tracking response (e) Stability response (f) Efficiency response Fig. 4: Standalone simulation results for promo policy evaluated at 100 sampled preference vectors. Each sampled preference produces one policy evaluation; panels (a)–(c) show different 2-D projections of these same 100 evaluations, with Pareto-nondominated and dominated samples distinguished visually. Tracking and efficiency form the clearest conflict, while stability and efficiency are largely complementary. Panels (d)–(f) show marginal per-objective preference-to-behavior mappings: each realized objective return rises near-monotonically with its corresponding preference weight, while the spread reflects variation in the other two simplex weights. The per-objective correlations ρT,ρS,ρE _T, _S, _E show the largest response on the efficiency axis, while tracking and stability remain strongly aligned with their preference weights despite tighter coupling to gait dynamics. Together, the dense results show that the four anchor behaviors are samples from a broader controllable interface rather than isolated operating points. V Real-Robot Evaluation The hardware study tests whether the simulated semantic interface transfers to a physical robot (Q4). A single promo policy trained in simulation is deployed on hardware without retraining or controller switching, and preference changes alone modulate measured tracking, stability, and efficiency behavior. V-A Protocol All hardware experiments use a single deployed promo checkpoint on a Unitree Go2 [43]. Experiments are conducted indoors with Vicon motion capture providing ground-truth position and velocity. We evaluate the four anchors in Fig. C.2: balanced (13,13,13)( 13, 13, 13), tracking-heavy (0.8,0.1,0.1)(0.8,0.1,0.1), stability-heavy (0.1,0.8,0.1)(0.1,0.8,0.1), and efficiency-heavy (0.1,0.1,0.8)(0.1,0.1,0.8), across three command regimes. The replay regime uses the same recorded trajectory for every preference and is the command-matched comparison. Fast and slow regimes use different route-speed profiles and are reported through aggregate outcomes. This gives 1212 conditions with five trials each. The policy runs at approximately 50 Hz over a 200 Hz low-level control loop; the observation stack, command interface, and network weights remain fixed. Table I reports means, and Fig. 6 shows run-to-run variability. V-B Preference-Conditioned Behavior Figure 5 shows the physical gait changes induced by preference, providing a glimpse of how the learned optimal actions adapt with changing preferences. The tracking-heavy preference produces larger foot excursions, whereas the efficiency-heavy preference yields compact, low-clearance trajectories. The stability-heavy behavior exhibits an intermediate gait with more pronounced foot clearance in this sequence. These differences emerge without prescribing gait parameters. (a) Tracking-heavy (b) Stability-heavy (c) Efficiency-heavy Fig. 5: Preference-induced gait adaptation on hardware. Chronophotographic overlays with 2-s tracked foot trajectories for tracking-, stability-, and efficiency-heavy preferences (left to right) using the same deployed promo policy. Tracking produces larger foot excursions, efficiency produces compact low-clearance trajectories, and stability exhibits an intermediate gait with increased clearance in this sequence. Table I and Fig. 6 quantify these differences. The clearest effect is energetic: the efficiency preference records lower mean specific energy in all three regimes, by 17.9%17.9\%, 30.4%30.4\%, and 11.3%11.3\% relative to balanced in replay, fast, and slow, respectively. Action- and torque-rate RMS are also approximately 29%29\% lower on average, indicating a large, consistent shift toward less aggressive actuation. The tracking preference records lower mean position-error mae (mae) than balanced in all three regimes, by 24.2%24.2\%, 25.5%25.5\%, and 38.7%38.7\% in replay, fast, and slow, respectively. This tracking shift is accompanied by approximately 2424–49%49\% higher action- and torque-rate RMS. In the fast regime, the stability preference also records a lower mean position mae of 0.9190.919 m, indicating that body regulation can improve trajectory progress under dynamic commands. The stability preference reduces peak attitude deviation in every regime, by 19.4%19.4\%, 59.0%59.0\%, and 8.7%8.7\% relative to balanced in replay, fast, and slow, respectively. The effect is largest during dynamic locomotion and weaker near standstill. The balanced preference provides a compromise when no objective is prioritized, including a lower velocity RMSE in fast and slow operation. The slow-regime stability setting has high aggregate velocity RMSE (2.2032.203 m/s) despite low peak attitude deviation, showing that velocity regulation and body-attitude regulation capture different near-standstill failure modes. These results establish the central deployment property of promo: preference changes systematically modulate hardware behavior with a fixed policy. The objectives are coupled. For example, the tracking preference improves accumulated position error while slightly worsening velocity RMSE relative to balanced, consistent with more aggressive transient tracking that reduces long-horizon positional drift. We therefore report both measures rather than treating velocity RMSE alone as tracking performance. TABLE I: Real-robot evaluation of a single promo policy under four semantic preferences and three command regimes on the Unitree Go2. Values are means over n=5n=5 trials per condition; bold denotes the preferred value within each regime. Preference Vel. RMSE [m/s] Pos. MAE [m] Energy [J/m] cot [-] Peak att. [rad] Act. rate [rad/s] Trq. rate [N m/s] Replay-round Balanced 0.4440.444 0.5340.534 124.6124.6 1.0581.058 0.1960.196 8.458.45 240.7240.7 Efficiency 0.4440.444 0.6240.624 102.3102.3 0.8690.869 0.1870.187 5.425.42 153.4153.4 Stability 0.4280.428 0.5380.538 132.4132.4 1.1251.125 0.1580.158 9.269.26 263.0263.0 Tracking 0.4930.493 0.4050.405 169.6169.6 1.4411.441 0.2810.281 11.9811.98 342.9342.9 Move-fast Balanced 0.4370.437 1.5941.594 180.5180.5 1.5331.533 0.2660.266 9.729.72 276.3276.3 Efficiency 0.7370.737 1.6411.641 125.7125.7 1.0681.068 0.1680.168 6.226.22 175.8175.8 Stability 0.8410.841 0.9190.919 208.7208.7 1.6101.610 0.1090.109 9.899.89 280.3280.3 Tracking 0.7530.753 1.1871.187 289.3289.3 2.4582.458 0.2130.213 14.0914.09 411.1411.1 Move-slow Balanced 0.3320.332 0.8760.876 129.6129.6 1.1011.101 0.1270.127 6.506.50 182.7182.7 Efficiency 0.7700.770 1.2391.239 114.9114.9 0.9760.976 0.1750.175 5.535.53 156.6156.6 Stability 2.2032.203 0.9310.931 135.8135.8 1.1541.154 0.1160.116 5.965.96 165.8165.8 Tracking 0.4380.438 0.5370.537 146.5146.5 1.2451.245 0.1460.146 8.118.11 224.2224.2 Fig. 6: Preference-conditioned hardware behavior. Tracking mae, peak attitude deviation, and specific energy for the same deployed promo policy under four semantic preferences across replay, fast, and slow regimes. Bars show means over five hardware trials and error bars denote one standard deviation. Lower is better for all metrics. The corresponding preference generally shifts behavior toward its intended semantic objective, while cross-objective differences expose the resulting physical trade-offs. Fig. 7: Disturbance recovery on a low-traction surface. Representative sequences using the balanced preference on a low-traction surface. Top is whole-body recovery following a disturbance at the head. Middle is recovery from a rear-left hip disturbance inducing approximately 230∘230 of yaw rotation. Bottom is recovery from direct disturbance of the front-right leg, requiring adjustment of the support configuration. Figure 7 examines low-traction recovery using the balanced preference. Despite substantial slip and displacement, the policy restores viable locomotion after whole-body, rotational, and direct limb disturbances; the most severe rotational example produces approximately 230∘230 of yaw rotation before recovery. Because disturbance forces were not instrumented, these sequences are qualitative stress tests rather than force-normalized robustness measurements. Complete recovery trajectories are provided in the accompanying videos. VI Discussion This section interprets promo as a deployment interface, distinguishes behavioral control from offline Pareto optimization, and discusses the limitations of the present study. VI-A Semantic Preferences as a Deployment Interface promo treats preference-conditioned morl as a runtime locomotion interface for reward non-stationarity. The non-stationary quantity is the scalar reward specification induced by wtw_t, not the transition dynamics. This framing targets a practical setting in which the relevant objective classes are known, but their desired trade-off changes during operation. The hardware results show that this trade-off can be exposed as a semantic input to one real robot policy rather than resolved offline into a fixed controller. The semantic objective design is more than reward engineering. The policy remains a neural controller, but the operator-facing cause of a behavioral change is expressed through meaningful objective weights rather than an opaque scalar reward or latent gait code. This is not a causal explanation of every joint command; it is a transparent interface at the level where deployment decisions are made, allowing behavior to be inspected against the intended tracking, stability, and efficiency emphasis. VI-B From Pareto Optimization to Deployable Control The results and ablations in Appendix D support two design conclusions. First, model selection for a deployed preference interface should balance utility, lower-tail rollout outcomes, controllability, and fall rate rather than optimize any single aggregate metric. Second, locomotion priors should remain fixed auxiliary terms: removing them, embedding them in the semantic objectives, or exposing them as a fourth objective can improve an aggregate metric while degrading controllability or reliability. The baseline evaluation makes the same distinction at the method-comparison level. Individual objective extrema and deployment-interface quality are different criteria: several baselines retain an advantage in one semantic score, whereas promo provides a stronger combination of objective specialization, lower failures, preference response, and one-policy deployment. The results also suggest a second role for preference coverage during training. Sampling a broad semantic simplex appears to diversify the behaviors used to solve locomotion, so the shared policy is not simply interpolating among independently learned fixed-objective solutions. This may explain why the balanced promo preference outperforms fixed-objective baselines on tracking, fall rate, and tail attitude risk. The evidence supports this as a plausible mechanism; a causal ablation of preference-coverage width remains future work. The ablations show that reward-optimization quality and deployable-control quality are separable. Variants that match or exceed the baseline in training reward or range-normalized-objective hv in the three semantic dimensions can underperform in controllability and falls, including variants that equalize objective magnitudes and suppress the tracking signal that drives gait acquisition (Appendix A). Robotic morl should therefore evaluate Pareto quality together with whether preferences remain meaningful, controllable, and robust in the deployed system. VI-C Limitations promo targets operator-induced objective non-stationarity rather than non-stationary mdp with changing transition dynamics, hidden regime shifts, or online adaptation. Linear scalarization focuses the policy on supported Pareto solutions and does not recover unsupported non-convex regions of the Pareto front. The critic estimates expected semantic returns, so lower-tail quantities such as cvar10 are rollout diagnostics rather than directly optimized critic objectives. The hardware study uses five trials per condition and four anchor preferences, demonstrating preference modulation rather than dense hardware controllability or statistical significance. The shared baseline evaluation likewise uses representative balanced and anchor settings; dense 100-preference baseline comparisons and ablations over training preference coverage remain future work. VII Conclusion We presented promo, a semantic preference-conditioned morl framework that exposes tracking, stability, and efficiency as runtime objectives for a single quadruped locomotion policy. By separating these semantic objectives from fixed locomotion priors, promo provides an interpretable preference input for deployment. In simulation, the policy achieves 0.84340.8434 dense preference–objective correlation with a 7.50%7.50\% mean fall rate over 100 sampled preferences, and the baseline comparisons show that individual objective extrema and deployment-interface quality are distinct criteria. On hardware, the same Unitree Go2 policy modulates energy use, trajectory error, and body attitude through preference changes alone, reducing specific energy, position error, and peak attitude deviation by up to 30.4%30.4\%, 38.7%38.7\%, and 59.0%59.0\% relative to the balanced setting. These results support semantic morl as a deployment mechanism in which the robot exposes the objective trade-off itself as a runtime control input, rather than only producing an offline Pareto set after training. References [1] A. Kumar, Z. Fu, D. Pathak, and J. Malik (2021) RMA: rapid motor adaptation for legged robots. In Proc. Robot.: Sci. Syst. (RSS), Note: doi: 10.15607/RSS.2021.XVII.011 External Links: Document Cited by: §I, §I-A, 1st item. [2] N. Rudin, D. Hoeller, P. Reist, and M. Hutter (2022) Learning to walk in minutes using massively parallel deep reinforcement learning. In Proc. 5th Conf. Robot Learn. (CoRL), Vol. 164, p. 91–100. Cited by: §I, §I, §I-A, §I-B. [3] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter (2022) Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci. Robot. 7 (62, Art. no. eabk2822). Note: doi: 10.1126/scirobotics.abk2822 External Links: Document Cited by: §I, §I, §I-A, §I-B. [4] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley (2013) A survey of multi-objective sequential decision-making. J. Artif. Intell. Res. 48, p. 67–113. Note: doi: 10.1613/jair.3987 External Links: Document Cited by: §I, §I-C. [5] C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers (2022) A practical guide to multi-objective reinforcement learning and planning. Auton. Agents Multi-Agent Syst. 36 (1, Art. no. 26). Note: doi: 10.1007/s10458-022-09552-y External Links: Document Cited by: §I, §I-C, §I-A. [6] R. Rădulescu, P. Mannion, D. M. Roijers, and A. Nowé (2020) Multi-objective multi-agent decision making: a utility-based analysis and survey. Auton. Agents Multi-Agent Syst. 34 (1, Art. no. 10). Note: doi: 10.1007/s10458-019-09433-x External Links: Document Cited by: §I, §I-C. [7] A. Mousa, N. Karavis, M. Caprio, W. Pan, and R. Allmendinger (2025) TAR: teacher-aligned representations via contrastive learning for quadrupedal locomotion. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), p. 11669–11676. Note: doi: 10.1109/IROS60139.2025.11247281 External Links: Document Cited by: §I, §I-A, §I-B, §I-B, 1st item. [8] R. Allmendinger, A. Jaszkiewicz, A. Liefooghe, and C. Tammer (2022) What if we increase the number of objectives? Theoretical and empirical implications for many-objective combinatorial optimization. Comput. Oper. Res. 145. Note: Art. no. 105857, doi: 10.1016/j.cor.2022.105857 External Links: Document Cited by: §I, §I-A. [9] G. B. Margolis and P. Agrawal (2023) Walk these ways: tuning robot control for generalization with multiplicity of behavior. In Proc. 6th Conf. Robot Learn. (CoRL), Vol. 205, p. 22–31. Cited by: §I, §I-B. [10] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter (2019) Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4 (26, Art. no. eaau5872). Note: doi: 10.1126/scirobotics.aau5872 External Links: Document Cited by: §I-A. [11] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke (2018) Sim-to-real: learning agile locomotion for quadruped robots. In Proc. Robot.: Sci. Syst. (RSS), Note: doi: 10.15607/RSS.2018.XIV.010 External Links: Document Cited by: §I-A. [12] I. M. A. Nahrendra, B. Yu, and H. Myung (2023) DreamWaQ: learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 5078–5084. Note: doi: 10.1109/ICRA48891.2023.10161144 External Links: Document Cited by: §I-A. [13] J. Long, Z. Wang, Q. Li, L. Cao, J. Gao, and J. Pang (2024) Hybrid internal model: learning agile legged locomotion with simulated robot response. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-A. [14] J. Long, W. Yu, Q. Li, Z. Wang, D. Lin, and J. Pang (2025) Learning H-Infinity locomotion control. In Proc. 8th Conf. Robot Learn. (CoRL), Vol. 270, p. 1094–1108. Cited by: §I-A. [15] A. Y. Ng, D. Harada, and S. Russell (1999) Policy invariance under reward transformations: theory and application to reward shaping. In Proc. 16th Int. Conf. Mach. Learn. (ICML), Bled, Slovenia, p. 278–287. Cited by: §I-B. [16] J. Achiam, D. Held, A. Tamar, and P. Abbeel (2017) Constrained policy optimization. In Proc. 34th Int. Conf. Mach. Learn. (ICML), Vol. 70, p. 22–31. Cited by: §I-B. [17] S. Gu, B. Sel, Y. Ding, L. Wang, Q. Lin, A. Knoll, and M. Jin (2025) Safe and balanced: a framework for constrained multi-objective reinforcement learning. IEEE Trans. Pattern Anal. Mach. Intell. 47 (5), p. 3322–3331. Note: doi: 10.1109/TPAMI.2025.3528944 External Links: Document Cited by: §I-B. [18] D. Hadfield-Menell, S. Milli, P. Abbeel, S. Russell, and A. Dragan (2017) Inverse reward design. In Adv. Neural Inf. Process. Syst. (NeurIPS), Vol. 30. Cited by: §I-B. [19] J. Wu, Y. Xue, and C. Qi (2023) Learning multiple gaits within latent space for quadruped robots. Note: arXiv:2308.03014 External Links: 2308.03014 Cited by: §I-B. [20] S. Lee, J. Kim, S. Park, J. T. Kim, Y. Jeong, H. J. Cho, H. R. Choi, and J. Cho (2026) Gait-parameterized reinforcement learning for a hydraulic quadruped robot. IEEE Robot. Autom. Lett. 11 (5), p. 5725–5732. Note: doi: 10.1109/LRA.2026.3673991 External Links: Document Cited by: §I-B. [21] J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik (2020) Prediction-guided multi-objective reinforcement learning for continuous robot control. In Proc. 37th Int. Conf. Mach. Learn. (ICML), Vol. 119, p. 10607–10616. Cited by: §I-C, §I-C. [22] T. Basaklar, S. Gumussoy, and U. Y. Ogras (2023) PD-MORL: preference-driven multi-objective reinforcement learning algorithm. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-C. [23] H. Lu, D. Herman, and Y. Yu (2023) Multi-objective reinforcement learning: convexity, stationarity and Pareto optimality. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-C. [24] T. Ambadkar, S. Panda, S. Kale, J. Dodge, and A. Verma (2026) Preference conditioned multi-objective reinforcement learning: decomposed, diversity-driven policy optimization. Note: arXiv:2602.07764 External Links: 2602.07764 Cited by: §D-C, §I-C. [25] R. Liu, Y. Pan, L. Xu, L. Song, J. Bian, P. You, and Y. Chen (2025) Efficient discovery of Pareto front for multi-objective reinforcement learning. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-C. [26] E. Liu, Y. Wu, X. Huang, C. Gao, R. Wang, K. Xue, and C. Qian (2025) Pareto set learning for multi-objective reinforcement learning. In Proc. AAAI Conf. Artif. Intell., Vol. 39, p. 18789–18797. Note: doi: 10.1609/aaai.v39i18.34068 External Links: Document Cited by: §I-C. [27] F. Felten, L. N. Alegre, A. Nowé, A. L. C. Bazzan, E. Talbi, G. Danoy, and B. C. da Silva (2023) A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Adv. Neural Inf. Process. Syst. (NeurIPS), Datasets and Benchmarks Track, Vol. 36, p. 23671–23700. Note: doi: 10.52202/075280-1028 External Links: Document Cited by: §I-C. [28] F. Felten, E. Talbi, and G. Danoy (2024) Multi-objective reinforcement learning based on decomposition: a taxonomy and framework. J. Artif. Intell. Res. 79, p. 679–723. Note: doi: 10.1613/jair.1.15702 External Links: Document Cited by: §I-C. [29] K. Miettinen (1999) Nonlinear multiobjective optimization. International Series in Operations Research & Management Science, Vol. 12, Kluwer Academic Publishers, Boston, MA, USA. External Links: ISBN 0-7923-8278-1 Cited by: §I-C. [30] T. Schaul, D. Horgan, K. Gregor, and D. Silver (2015) Universal value function approximators. In Proc. 32nd Int. Conf. Mach. Learn. (ICML), Vol. 37, p. 1312–1320. Cited by: §I-C. [31] A. Kumar, X. B. Peng, and S. Levine (2019) Reward-conditioned policies. Note: arXiv:1912.13465 External Links: 1912.13465 Cited by: §I-C. [32] M. Reymond, E. Bargiacchi, and A. Nowé (2022) Pareto conditioned networks. In Proc. 21st Int. Conf. Auton. Agents Multiagent Syst. (AAMAS), p. 1110–1118. Cited by: §I-C. [33] M. Terekhov and C. Gulcehre (2024) In search for architectures and loss functions in multi-objective reinforcement learning. Note: arXiv:2407.16807 External Links: 2407.16807 Cited by: §I-C, 2nd item. [34] X. Cai, P. Zhang, L. Zhao, J. Bian, M. Sugiyama, and A. J. Llorens (2023) Distributional Pareto-optimal multi-objective reinforcement learning. In Adv. Neural Inf. Process. Syst. (NeurIPS), Vol. 36, p. 15593–15613. Note: doi: 10.52202/075280-0686 External Links: Document Cited by: §I-C, 2nd item. [35] S. H. Huang, A. Abdolmaleki, G. Vezzani, P. Brakel, D. J. Mankowitz, M. Neunert, S. Bohez, Y. Tassa, N. Heess, M. Riedmiller, and R. Hadsell (2022) A constrained multi-objective reinforcement learning framework. In Proc. 5th Conf. Robot Learn. (CoRL), Vol. 164, p. 883–893. Cited by: §I-C. [36] H. Munn, B. Tidd, P. Böhm, M. Gallagher, and D. Howard (2025) Scalable multi-objective robot reinforcement learning through gradient conflict resolution. Note: arXiv:2509.14816 External Links: 2509.14816 Cited by: §I-C. [37] J. D. Knowles, L. Thiele, and E. Zitzler (2006) A tutorial on the performance assessment of stochastic multiobjective optimizers. TIK Rep. Technical Report 214, Computer Engineering and Networks Laboratory (TIK), ETH Zurich, Zurich, Switzerland. Note: Revised version, doi: 10.3929/ethz-b-000023822 External Links: Document Cited by: §I-D. [38] Y. Yang, T. Zhou, M. Pechenizkiy, and M. Fang (2025) Preference controllable reinforcement learning with advanced multi-objective optimization. In Proc. 42nd Int. Conf. Mach. Learn. (ICML), Vol. 267, p. 71612–71647. Cited by: §I-D. [39] P. de las Heras Molins, B. Yalcinkaya, L. Peters, D. Fridovich-Keil, and G. Bakirtzis (2026) Controllability in preference-conditioned multi-objective reinforcement learning. Note: arXiv:2605.10585 External Links: 2605.10585 Cited by: §I-D. [40] Z. Jiang, Y. Wang, R. Marr, E. Novoseller, B. T. Files, and V. Ustun (2026) GraphAllocBench: a flexible benchmark for preference-conditioned multi-objective policy learning. Note: arXiv:2601.20753 External Links: 2601.20753, Link Cited by: §I-D. [41] C. Spearman (1904) The proof and measurement of association between two things. Amer. J. Psychol. 15 (1), p. 72–101. Note: doi: 10.2307/1412159 External Links: Document Cited by: §IV-B. [42] M. Laumanns, L. Thiele, K. Deb, and E. Zitzler (2002) Combining convergence and diversity in evolutionary multiobjective optimization. Evol. Comput. 10 (3), p. 263–282. External Links: Document Cited by: §IV-E. [43] Unitree Robotics Unitree Go2. Note: Accessed: Aug. 7, 2026. [Online]. Available: https://w.unitree.com/go2/ Cited by: §V-A. [44] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine (2019) Diversity is all you need: learning skills without a reward function. In Proc. Int. Conf. Learn. Represent. (ICLR), External Links: Link Cited by: §D-C. Appendix A Reward and Preference Specification This appendix defines the reward quantities used by promo. Semantic terms are grouped into tracking, stability, and efficiency objectives; fixed locomotion priors remain auxiliary terms outside the deployment preference vector. A-A Semantic Reward Decomposition Tracking measures command following, stability measures body regulation, and efficiency measures energetic and control effort. Fixed locomotion priors encode embodiment-level regularity needed for viable legged motion. Table A.1 lists the active reward terms and environment coefficients. TABLE A.1: Semantic objectives and fixed locomotion priors used by promo. Term Group Coefficient Physical interpretation track_lin_vel_xy_exp Tracking 1.51.5 Rewards planar velocity tracking with kernel standard deviation 0.50.5 track_ang_vel_z_exp Tracking 0.750.75 Rewards yaw-rate tracking with kernel standard deviation 0.50.5 lin_vel_z_l2 Stability −2.0-2.0 Penalizes vertical base motion ang_vel_xy_l2 Stability −0.05-0.05 Penalizes roll and pitch angular velocity flat_orientation_l2 Stability −2.5-2.5 Penalizes body tilt away from upright posture joint_torques_l2 Efficiency −2.0×10−4-2.0× 10^-4 Penalizes squared applied torque joint_acc_l2 Efficiency −2.5×10−7-2.5× 10^-7 Penalizes squared joint acceleration action_rate_l2 Efficiency −0.01-0.01 Penalizes rapid changes in action commands feet_air_time Fixed loco. prior 0.250.25 Promotes viable stepping with minimum swing time 0.50.5 s foot_contact_balance_penalty Fixed loco. prior −0.5-0.5 Penalizes imbalanced foot contact over a 1.01.0 s window joint_pos_penalty Fixed loco. prior −0.1-0.1 Regularizes joint posture, with stronger stand-still weighting The scalar semantic reward used for actor optimization is rtsem(wt)=wt,trackrttrack+wt,stabrtstab+wt,effrteff,r_t^sem(w_t)=w_t,trackr_t^track+w_t,stabr_t^stab+w_t,effr_t^eff, (23) with w∈Δ2w∈ ^2. The fixed locomotion priors form rtpriorr_t^prior and are added to the actor-side utility, but they are not entries of wtw_t and do not define critic heads. All reported promo experiments use λprior=1 _prior=1 in Eq. (11), so the actor reward is rtactor=rtsem(wt)+rtprior.r_t^actor=r_t^sem(w_t)+r_t^prior. (24) A-B Per-Step Objective Definitions Let ct=(ctx,cty,ctψ)c_t=(c^x_t,c^y_t,c^ψ_t) be the commanded planar velocity and yaw rate, vtbv^b_t and ωtbω^b_t the base linear and angular velocity in the body frame, gtbg^b_t the projected gravity vector, qtq_t, q˙t q_t, q¨t q_t, and τt _t the joint state and applied torque, and ata_t the policy action. The raw quantities grouped in Table A.1 are defined below. The tracking objective uses exponential kernels, rtlin r^lin_t =exp(−(ctx−vtb,x)2+(cty−vtb,y)2σv2), = (- (c^x_t-v^b,x_t)^2+(c^y_t-v^b,y_t)^2 _v^2 ), (25) rtyaw r^yaw_t =exp(−(ctψ−ωtb,z)2σψ2), = (- (c^ψ_t-ω^b,z_t)^2 _ψ^2 ), (26) with σv=σψ=0.5 _v= _ψ=0.5. The stability penalties are ptz-vel p^z-vel_t =(vtb,z)2, =(v^b,z_t)^2, (27) ptxy-ang p^xy-ang_t =(ωtb,x)2+(ωtb,y)2, =(ω^b,x_t)^2+(ω^b,y_t)^2, (28) ptorient p^orient_t =(gtb,x)2+(gtb,y)2. =(g^b,x_t)^2+(g^b,y_t)^2. (29) The efficiency penalties are ptτ p^τ_t =∑j(τtj)2, = _j(τ^j_t)^2, (30) ptq¨ p q_t =∑j(q¨tj)2, = _j( q^j_t)^2, (31) ptΔa p a_t =∑j(atj−at−1j)2. = _j(a^j_t-a^j_t-1)^2. (32) These terms are implemented as sums over the configured joints or action dimensions, not as dimension-normalized means. The semantic objectives used in rtsemr_t^sem are therefore rttrack r_t^track =1.5rtlin+0.75rtyaw, =1.5\,r^lin_t+0.75\,r^yaw_t, (33) rtstab r_t^stab =−2.0ptz-vel−0.05ptxy-ang−2.5ptorient, =-2.0\,p^z-vel_t-0.05\,p^xy-ang_t-2.5\,p^orient_t, (34) rteff r_t^eff =−2.0×10−4pτt−2.5×10−7pq¨t−0.01pΔat. =-2.0×10^-4\,p^τ_t-2.5×10^-7\,p q_t-0.01\,p a_t. (35) Fixed locomotion priors are optimized during training but remain independent of the semantic preference space. The feet air-time prior rewards sufficiently long swing phases upon first contact, rtair=[∥ctxy∥2>0.1]∑f∈ℱ(Tair,tf−Tmin)firstcontact,tf,r^air_t=1\! [\|c^xy_t\|_2>0.1 ] _f (T^f_air,t-T_ )1^f_first\ contact,t, (36) where Tmin=0.5T_ =0.5 s and the reward is active only under a non-negligible planar command. The contact-balance prior discourages persistent imbalance in foot usage over a rolling window. For each foot f, the contact fraction is ϕtf=1Nt∑τ∈t[∥Fτf∥2>1.0N],φ^f_t= 1N_t _τ _t1\! [\|F^f_τ\|_2>1.0\,N ], (37) and define ptcontact=Varf∈ℱ(ϕtf)+1|ℱ|∑f∈ℱ[ϕtf<ϕmin].p^contact_t=Var_f (φ^f_t)+ 1|F| _f 1\! [φ^f_t< _ ]. (38) The first term penalizes unequal contact utilization across feet, while the second penalizes feet that are rarely used. We use a 1.01.0 s window and ϕmin=0.05 _ =0.05. The joint-position prior regularizes the robot toward its nominal posture, ptjpos=ηt‖qt−q0‖2,p^jpos_t= _t \|q_t-q^0 \|_2, (39) where ηt=1 _t=1 during locomotion and ηt=5 _t=5 when standing still, imposing stronger posture regularization near stationary operation. The recovered configuration defines locomotion as active when either the command norm exceeds 0.10.1 or the planar body-speed norm exceeds 0.50.5 m/s. A-C Preference Space and Sampling Training rollouts use fixed sampled preferences. The default training distribution samples x∼Dirichlet(1,1,1)x (1,1,1) and maps it affinely to the interior of the semantic simplex, wt,i=wmin+(1−Kwmin)xi,wmin=0.1,K=3.w_t,i=w_ +(1-Kw_ )x_i, w_ =0.1, K=3. (40) Thus wt,i≥0.1w_t,i≥ 0.1 and ∑iwt,i=1 _iw_t,i=1 automatically; no clipping or projection is applied after sampling. Because the Dirichlet density is uniform on the full simplex and the affine map has constant Jacobian, this procedure uniformly samples the truncated simplex with minimum component 0.10.1. The minimum prevents any semantic objective from disappearing during training; in particular, zero tracking weight can remove the command-following signal needed to learn locomotion. The sampled preference is held fixed for the rollout used to estimate critic targets and actor advantages. Table A.2 lists the training and evaluation preference configurations. TABLE A.2: Preference configurations used for training and evaluation. Protocol Preference specification Purpose Training distribution wi=0.1+0.7xiw_i=0.1+0.7x_i, x∼Dirichlet(1,1,1)x (1,1,1) Uniform truncated-simplex coverage without zeroed objectives Balanced (1/3,1/3,1/3)(1/3,1/3,1/3) Nominal semantic trade-off Tracking-heavy (0.8,0.1,0.1)(0.8,0.1,0.1) Tracking-focused anchor Stability-heavy (0.1,0.8,0.1)(0.1,0.8,0.1) Stability-focused anchor Efficiency-heavy (0.1,0.1,0.8)(0.1,0.1,0.8) Efficiency-focused anchor Two-objective mixtures (0.45,0.45,0.1)(0.45,0.45,0.1), (0.45,0.1,0.45)(0.45,0.1,0.45), (0.1,0.45,0.45)(0.1,0.45,0.45) Representative simplex edges Dense 100-preference evaluation Stored set of 100100 full-simplex preferences Pareto and controllability evaluation beyond truncated training support A-D Objective Scaling and Semantic Interpretation promo retains the nominal reward scales of the locomotion environment instead of normalizing the semantic objectives to equal numerical magnitudes before scalarization. The balanced anchor (13,13,13)( 13, 13, 13) therefore corresponds to the nominal scalar reward already tuned for viable locomotion, and other preferences act as relative semantic modifiers around that operating point. The objectives do not play symmetric roles during gait acquisition. Tracking supplies the main locomotion-driving signal, while stability and efficiency act mainly as regularizing pressures. Consistent with this interpretation, scaling stability and efficiency by 22–5×5× relative to tracking degraded locomotion quality and preference-conditioned behavior. Retaining nominal scales lets the balanced preference recover the default trade-off while preference changes shift emphasis toward the named objectives. Consequently, absolute scalar utility values are tied to these nominal scales and should not be interpreted as scale-normalized objective importance. Appendix B Architecture and Training Configuration This appendix provides the reproducibility configuration for promo: network and control settings, domain randomization, rollout collection, and optimizer constants. B-A Network, Simulation, and Control Configuration Table B.1 defines the architecture, simulator, and control stack used by promo. Table B.2 defines the randomized training environment used for robustness and zero-shot transfer. TABLE B.1: promo network, simulation, and control configuration. Item Value Robot Unitree Go2, 1212-dof quadruped Simulator Isaac Sim / IsaacLab Actor hidden sizes [512,256,128][512,256,128] Critic shared trunk [512,256,128][512,256,128] Semantic critic heads K=3K=3 scalar heads Privileged encoder MLP over privileged state and preference History encoder MLP over 1010-step history and preference Privileged latent dimension 4545 Actor observation history 1010 control steps Velocity estimator input ztHz_t^H and ot−4:to_t-4:t Velocity estimator target Three-dimensional base linear velocity vtbv_t^b Action dimension 1212 joint-position targets Command ranges vx,vy∈[−1,1]v_x,v_y\!∈\![-1,1] m/s, ψ˙∈[−1,1] ψ\!∈\![-1,1] rad/s Sim timestep 55 ms (200200 Hz); control decimation 44 Policy / control rate 5050 Hz Episode length 2020 s (10001000 policy steps) Terrain Flat TABLE B.2: Domain randomization and perturbation ranges used during training. Parameter Range / setting Static and dynamic friction [0.6, 1.2][0.6,\,1.2] Restitution coefficient [0, 1][0,\,1] Base mass offset [0, 2][0,\,2] kg Force perturbation ±10± 10 N Torque perturbation ±5± 5 N m Velocity perturbation ±2± 2 m/s every 22–44 s Initial linear velocity ±0.5± 0.5 m/s Initial angular velocity ±0.5± 0.5 rad/s B-B Rollout and Optimization Configuration The semantic critic uses objective-wise TD targets. The actor advantage uses the scalarized semantic critic as its baseline, while the fixed prior reward enters the gae residual directly without a separate prior value head. Advantages are normalized per batch before the ppo update. Table B.3 lists the rollout and optimizer constants. TABLE B.3: Training and optimization configuration. Parameter Value Parallel environments 40964096 Rollout horizon 5050 policy steps per env Discount factor γ 0.990.99 gae parameter λGAE _GAE 0.950.95 ppo clipping ϵε 0.120.12 Entropy coefficient 0.0050.005 Learning rate Adaptive, min: 5×10−65×10^-6, max: 2×10−42×10^-4 Target KL divergence 0.0060.006 Gradient-norm clipping 0.50.5 Optimization epochs 33 Minibatches 44 Advantage normalization Per batch Optimizer Adam Latent-alignment coefficient λalign _align 1.01.0 Diversity coefficient λdiv _div 10−510^-5 Appendix C Evaluation Protocol This appendix defines the simulation and hardware execution protocols. C-A Simulation Evaluation Protocol Fig. C.1: Simulation evaluation protocol. Representative preferences reuse the same command-burst and friction/contact schedule, controlling for test-sequence variation when comparing preference-conditioned behavior. Simulation evaluation uses two modes. The representative-preference protocol evaluates the seven anchor and mixed preferences in Table A.2, using 10241024 environments per preference over 2020 s episodes. To isolate preference-induced differences, every representative evaluation uses the same disturbance script: the contact model alternates between predefined friction pairs every 22 s while commanded forward, lateral, and yaw velocities follow a fixed burst sequence. The dense protocol evaluates 100100 sampled preferences, again with 10241024 environments per preference over 2020 s episodes under standardized conditions. Each preference is executed independently, producing preference-performance maps for Pareto coverage, controllability, expected utility, lower-tail utility, and fall rate. Table C.2 gives the corresponding ε -dominance pruning. The drop from 6767 policies that are non-dominated within the sampled set under exact Pareto dominance to 3737 at ε=1% =1\% indicates a dense approximation of the trade-off surface rather than isolated anchors. The retained percentage is computed relative to the original exact non-dominated set. TABLE C.1: Key interface metrics for promo over 100100 preferences. Metric Value Mean utility per step 0.01490.0149 cvar10 utility 0.01200.0120 Fall rate 7.50%7.50\% Preference–objective correlation 0.84340.8434 tracking ρT _T 0.77480.7748 stability ρS _S 0.82890.8289 efficiency ρE _E 0.92660.9266 Range-normalized-objective hv 0.82660.8266 Non-dominated samples (exact) (67)/100(67)/100 TABLE C.2: ε -dominance pruning of the 100-preference evaluation. ε threshold 00 0.0050.005 0.010.01 0.020.02 0.030.03 Non-dominated solutions 6767 5050 3737 2525 1414 Non-dominated set retained (%) 100100 74.674.6 55.255.2 37.337.3 20.920.9 Figures C.1 and C.2 summarize the scripted evaluation protocol and semantic preference simplex used in training, simulation evaluation, and hardware anchor tests. Fig. C.2: Semantic preference simplex for promo. The simplex defines the deployment-facing trade-off among tracking, stability, and efficiency; the marked anchors are used for representative simulation and hardware evaluations, while the dense evaluation samples the interior. C-B Hardware Evaluation Protocol Section V gives the shared hardware setup, preference anchors, and trial counts. Table C.3 distinguishes the three command regimes: replay provides the command-matched comparison, fast tests dynamic joystick-driven locomotion, and slow emphasizes low-speed operation with near-standstill transitions. Figure E.1 provides the detailed command-matched diagnostic for the Replay regime. Because all preferences receive the same recorded command sequence, differences in trajectory tracking, energy use, and body regulation are attributable to the deployed semantic preference rather than operator-input variation. TABLE C.3: Real-robot evaluation regimes. Regime Command protocol Evaluation focus Replay Recorded, identical Command-matched comparison Fast Manual joystick Dynamic locomotion Slow Manual joystick Near-standstill locomotion Appendix D Design Analysis and Ablations This appendix analyzes the design choices behind promo: separating semantic objectives from fixed locomotion priors, scalarizing before the clipped ppo surrogate, and regularizing preference responsiveness at the action level. D-A Objective/Prior Factorization Proposition 1. Objective/prior separation. The aggregate locomotion-prior term does not define a preference axis. Specifically, the preference vector wtw_t does not weight or scalarize JpriorJ_prior, although Jprior(πθ,wt)J_prior( _θ,w_t) may change indirectly through the preference-dependent state visitation distribution. Only tracking, stability, and efficiency define the deployment interface and receive value heads, Pareto analysis, and controllability evaluation. The prior remains a fixed regularizer. All hv values use these three outcomes and the common within-study normalization in Section IV. Table D.1 tests four treatments of the locomotion priors. We report mean reward and episode length instead of utility and cvar10 because the fourth-objective variant changes both the utility definition and the preference dimension. Keeping the priors fixed and outside the semantic objectives gives the selected balance of controllability and reliability. Moving priors inside the semantic objectives slows learning and nearly triples the fall rate (0.060→0.1650.060→ 0.165), while removing priors increases hypervolume but reduces controllability and produces non-viable efficiency-heavy behavior. Exposing locomotion as a fourth semantic objective gives high aggregate utility but degrades controllability and fall rate. Across variants, aggregate metrics can improve while the operator-facing interface degrades. TABLE D.1: Effect of objective/prior factorization on comparable training and interface metrics. Variant Reward ↑ Ep. len. ↑ Ctrl. ↑ Fall ↓ hv ↑ Fixed priors (ours) 27.0627.06 997.56997.56 0.7410.741 0.0600.060 0.6930.693 Prior removed 25.6325.63 956.54956.54 0.6380.638 0.1310.131 0.9060.906 Priors inside objectives 26.7226.72 996.60996.60 0.6990.699 0.1650.165 0.6260.626 Prior as 4th objective 26.8626.86 996.72996.72 0.4470.447 0.2180.218 0.5120.512 (a) Mean reward (b) Mean episode length Fig. D.1: Training behavior under alternative objective/prior formulations. Keeping the locomotion priors as fixed auxiliary shaping reaches stable locomotion and high reward fastest; embedding the priors in the semantic objectives or exposing them as a fourth objective delays convergence. D-B Scalarization Before ppo Clipping The clipped ppo surrogate is nonlinear in the advantage. For a scalar advantage, define Lϵ(A,ϱt)=min(ϱtA,clip(ϱt,1−ϵ,1+ϵ)A).L_ε(A; _t)= \! ( _tA,\,clip( _t,1-ε,1+ε)A ). (41) Semantic scalarization and clipping do not generally commute: Lϵ(∑iwt,iAi,ϱt)≠∑iwt,iLϵ(Ai,ϱt),L_ε\! ( _iw_t,iA_i; _t )≠ _iw_t,iL_ε(A_i; _t), (42) because the clipped branch selected by LϵL_ε depends on the sign and magnitude of the scalarized advantage. promo therefore estimates semantic objective values Vi(st,wt)V_i(s_t,w_t), forms the preference-scalarized semantic baseline Vw(st,wt)=∑iwt,iVi(st,wt)V_w(s_t,w_t)= _iw_t,iV_i(s_t,w_t), includes the fixed-locomotion-prior reward directly in the scalar actor residual, normalizes scalar advantages over the batch, and then applies the clipped ppo surrogate. The fixed locomotion priors are not assigned critic heads. D-C Diversity Regularization TABLE D.2: Diversity-regularization ablation. Training metrics are final-window means; evaluation metrics are averaged over three seeds. Type λ Reward ↑ Ep. len. ↑ Ctrl. ↑ cvar10 ↑ Fall ↓ hv ↑ None – 26.8 997.3 0.80 0.012 0.10 0.56 Action 10−610^-6 26.8 996.9 0.88 0.012 0.07 0.55 3×10−63×10^-6 26.6 994.9 0.78 0.012 0.07 0.74 −10^-5 26.9 997.8 0.79 0.012 0.08 0.66 3×10−53×10^-5 26.8 996.4 0.86 0.011 0.11 0.59 10−410^-4 26.5 996.7 0.48 0.009 0.13 0.85 Latent 10−610^-6 26.3 997.1 0.86 0.011 0.07 0.62 3×10−63×10^-6 26.5 997.1 0.78 0.011 0.18 0.35 10−510^-5 26.4 997.6 0.79 0.012 0.08 0.55 3×10−53×10^-5 26.0 997.2 0.65 0.011 0.12 0.65 10−410^-4 26.2 996.4 0.81 0.011 0.11 0.44 Fig. D.2: Sensitivity of deployment metrics to diversity regularization. Results are averaged over three seeds and shown as relative improvement over no diversity, exposing both metric-specific optima and coefficient sensitivity. Diversity regularization, in the spirit of diversity-regularized morl [24] and unsupervised skill discovery [44], is useful but coefficient-sensitive. We compare two mechanisms. Action diversity uses the training loss in (18), directly separating the output distributions induced by different preferences at the same observation and detached history latent. The latent alternative regularizes an internal preference-conditioned representation hθ(ot,ztH,wt)h_θ(o_t,z_t^H,w_t), ℒdivlat=[(∥hθ(ot,ztH,wt)−hθ(ot,ztH,w′t)∥−α∥wt−w′t∥)2], splitL_div^lat&=E [ ( h_θ(o_t,z_t^H,w_t)-h_θ(o_t,z_t^H,w _t) \\ & -α w_t-w _t )^2 ], split (43) where α sets the target scale between representation distance and preference distance. Action diversity acts on the deployed policy distribution itself, whereas latent diversity shapes the representation from which the policy is produced. Table D.2 summarizes the action/latent sweep, including reward and episode length over the final 10%10\% of training, while Fig. D.2 shows the corresponding improvement over no diversity. No coefficient dominates every metric: action diversity at 10−610^-6 gives the largest controllability gain (0.803→0.8790.803→ 0.879) and a low fall rate (0.0670.067), whereas the selected 10−510^-5 setting broadens the Pareto spread (0.6570.657 vs. 0.5610.561 hypervolume), reduces falls (0.098→0.0830.098→ 0.083), and slightly improves final reward and episode length. Latent diversity can improve individual metrics, but its response is less consistent across coefficients. Taken together, the sweep favors action-level regularization as the more reliable interface-level mechanism and shows that its coefficient should be selected using controllability, robustness, and Pareto coverage rather than training reward alone. Appendix E Baseline Implementations All simulation baselines use the common flat-terrain evaluation used for promo: a 1212-dof quadruped with joint-position actions at 5050 Hz, 2020 s episodes, planar velocity commands in [−1,1][-1,1] m/s, yaw-rate commands in [−1,1][-1,1] rad/s, and matched termination conditions. All methods use on-policy ppo-style training and are evaluated with the semantic decomposition in Appendix A. E-A Single-Objective Locomotion Baselines tar and rma are fixed-trade-off controllers trained with the balanced scalar semantic reward plus fixed locomotion priors. Their deployment actors do not receive a preference input. tar uses a history-based actor and aligns it with privileged critic information through auxiliary transition and velocity-prediction losses. rma trains a privileged environment encoder and distills its latent into a history-based adaptation module for deployment. E-B morl Baselines moppo uses a single preference-conditioned ppo policy. The actor and critic receive a three-dimensional preference sampled from the same truncated simplex as promo and held fixed throughout each rollout. The critic predicts one return per semantic objective, while the actor optimizes their preference-weighted combination together with the fixed locomotion priors. dpmorl uses a four-policy archive trained with fixed utilities: equal weighting and one utility emphasizing each of tracking, stability, and efficiency. Each policy observes the cumulative discounted semantic-objective return state and is trained with scalar ppo on the incremental utility reward. The environment-interaction budget is divided equally across archive members, so each receives one quarter of the single full-budget run. E-C promo-sorl Specialists promo-sorl comprises independently trained scalar policies using the promo actor-critic architecture without preference conditioning, objective-wise semantic critics, or parameter sharing. The actor observes the same proprioceptive history used by promo but receives no preference vector. We train balanced, tracking-, stability-, and efficiency-focused specialists by varying the semantic coefficients while retaining fixed locomotion priors. This baseline compares one preference-conditioned promo policy against separately trained fixed scalarizations. Fig. E.1: Replay command grid. Identical recorded command sequence is replayed for all preference anchors, isolating preference-induced differences in trajectory tracking, energetic cost, and body regulation under matched command conditions.