Paper deep dive

Progressively Learning Heterogeneous Skills in a Unified Latent Space

Yue-Yi Zhang, Ming Gong, Linpu He, Wei-Shi Zheng, Zhilin Zhao

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 131

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/26/2026, 4:15:25 AM

Summary

The paper introduces HetSkills, a framework for physics-based character control that progressively learns heterogeneous skills (tracking, text-to-motion, completion, adaptation) within a unified latent space. It utilizes a shared motion decoder and part-wise decomposition to enable skill reuse and composition without retraining separate controllers. Key innovations include Motion Intuition Distillation (MID) for grounding text-to-motion in language semantics and a task-guidance module for downstream adaptation.

Entities (7)

HetSkills · framework · 100%Motion Intuition Distillation · method · 95%Sun Yat-sen University · organization · 95%Unified Latent Space · concept · 95%Motion Tracking · skill · 90%Physics-based Character Control · task · 90%Text-to-Motion Generation · skill · 90%

Relation Signals (5)

Authors → affiliatedwith → Sun Yat-sen University

confidence 95% · Affiliation: Sun Yat-sen University

Motion Intuition Distillation → iscomponentof → HetSkills

confidence 95% · we introduce motion intuition distillation to ground text-to-motion generation in language semantics

HetSkills → uses → Unified Latent Space

confidence 95% · HetSkills is a novel framework designed to progressively learn heterogeneous skills within a unified latent space

HetSkills → supports → Text-to-Motion Generation

confidence 90% · Experimental results demonstrate the effectiveness in ... text-to-motion generation

HetSkills → supports → Motion Tracking

confidence 90% · Experimental results demonstrate the effectiveness in motion tracking

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We propose HetSkills, a novel framework designed to progressively learn heterogeneous skills within a unified latent space for physics-based character control. The core idea is to treat this latent space as a shared executable interface, enabling seamless integration of skills learned from diverse data sources, supervision forms, and tasks. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

127,951 characters extracted from source content.

Expand or collapse full text

Progressively Learning Heterogeneous Skills in a Unified Latent Space Yue-Yi Zhang email: zhangyy696@mail2.sysu.edu.cn Affiliation: Sun Yat-sen University , China , Ming Gong email: t2085443@gmail.com Affiliation: Sun Yat-sen University , China , Linpu He email: linpuhe@163.com Affiliation: Sun Yat-sen University , China , Wei-Shi Zheng email: wszheng@isee.org Affiliation: Sun Yat-sen University , China and Zhilin Zhao email: zhaozhlin@mail.sysu.edu.cn Affiliation: Sun Yat-sen University , China Figure 1. Overview of HetSkills. From left to right, the results illustrate four representative skill categories: motion tracking, text-to-motion generation, motion completion, and downstream task adaptation. Although these skills are learned from heterogeneous data sources, supervision signals, and task objectives, they are all represented in a unified executable latent space and decoded by a shared physics-based motion controller, enabling progressive skill accumulation, reuse, and composition. Abstract. We propose HetSkills, a novel framework designed to progressively learn heterogeneous skills within a unified latent space for physics-based character control. The core idea is to treat this latent space as a shared executable interface, enabling seamless integration of skills learned from diverse data sources, supervision forms, and tasks. HetSkills begins by learning a tracking skill that establishes a strong foundation in motion control and creates a shared motion decoder, which can be reused across tasks without the need for retraining or separate controllers. To prevent the text-to-motion skill from exploiting shortcut pathways instead of learning language semantics, we introduce motion intuition distillation to ground text-to-motion generation in language semantics and a task-guidance module that dynamically adjusts actions based on high-level language instructions. This enables HetSkills to preserve natural motion while continuously expanding its skill repertoire, making it highly adaptable for long-horizon tasks. Experimental results demonstrate the effectiveness in motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, achieving impressive success rates even under challenging conditions. Keywords: Reinforcement learning, character animation, motion imitation 1. Introduction Learning physics-based character skills that are natural, composable, and reusable is a fundamental goal in animation, gaming, and robotics. Early approaches (41; 57; 77) typically learn task-specific controllers by imitating reference motions from small, narrowly curated motion datasets. While effective for individual tasks, such methods offer limited reusability and compositionality. More recent efforts substantially expand the coverage of motion priors and behavior models by leveraging large-scale data (66). However, despite this progress, physics-based character control still lacks a unified and executable skill representation that can support diverse behaviors within a single framework. As a result, it remains difficult to accumulate skills from heterogeneous data sources, compose them flexibly, and transfer them efficiently to new downstream tasks while preserving natural human-like motion. Large-scale motion priors and language-conditioned control (55; 19) suggest that language can provide a unified and flexible interface for accessing rich human motion knowledge. This creates the possibility of reusing previously acquired motion capabilities without collecting new demonstrations or training a dedicated controller for every new behavior. Existing methods (31; 37) partially move in this direction by leveraging pretrained motion priors to guide downstream control. Yet these approaches typically apply pretrained motion models as external guidance or regularization, rather than as a shared skill representation in which newly acquired capabilities can be progressively integrated and directly reused. Consequently, they remain limited when tasks involve heterogeneous objectives, diverse interaction patterns, or behaviors that go beyond the support of the original motion data. To address the lack of a unified and executable skill representation for progressively accumulating and reusing heterogeneous skills in physics-based character control, we propose HetSkills, a physics-based character control framework that progressively learns heterogeneous skills within a unified latent space. Specifically, heterogeneous skills refer to capabilities acquired from different data sources, supervision forms, and training stages, including motion tracking, text-to-motion generation, motion completion, and downstream task adaptation. Instead of training a separate controller for each capability, HetSkills continuously accumulates these skills into a shared latent representation that serves as a common executable interface for skill acquisition, composition, and reuse. This unified design enables the controller to grow with newly introduced abilities while maintaining a coherent motion prior and a consistent control space. More specifically, HetSkills is built around a unified latent space that serves as a shared interface for acquiring, composing, and reusing heterogeneous skills. To improve compositional control and generalization, we adopt a part-wise character decomposition (2; 1), which allows different body parts to receive specialized yet coordinated control signals within a common latent architecture. The latent space is anchored by an end-to-end tracking skill that is preserved and directly reused throughout subsequent learning stages, thereby avoiding an additional distillation step and its associated performance degradation. To support robust text-to-motion generation, we further introduce Motion Intuition Distillation (MID), which encourages the model to ground its predictions in language semantics rather than shortcut pathways. For downstream tasks, HetSkills dispenses with reference motions and task-specific demonstrations. It performs lightweight latent-space adaptation, with language specifying high-level behavioral objectives for different body parts within a coordinated whole-body control framework. Experiments demonstrate the effectiveness of HetSkills in three key aspects. First, HetSkills supports stable accumulation and reuse of heterogeneous skills within a shared latent space, enabling reliable compositional control across behaviors learned from different sources and supervision forms. Second, it achieves robust text-to-motion generation under both standard and challenging initializations, reaching a success rate of 96.9%96.9\% under standard initialization and 81.7%81.7\% from a neutral pose. Third, HetSkills transfers effectively to novel downstream tasks through lightweight latent-space adaptation, without requiring task-specific demonstrations or reference motions. These results indicate that a unified latent space can serve as an effective substrate for progressively expanding physics-based character capabilities while preserving natural motion quality. 2. Related Work Physics-based character control is a long-standing focus of research in animation, robotics, and gaming, aiming to generate realistic, reusable, and composable character behaviors (72; 63; 26). Over the years, methods have addressed different aspects of motion learning, including imitation, motion tracking, language-conditioned control, and downstream task adaptation. However, existing methods often struggle with integrating heterogeneous skills into a single, unified framework, requiring either retraining or relying on fragmented modules for new tasks. We summarize the key lines of related work below, situating them with respect to our goal of progressively integrating heterogeneous skills within a unified framework. Physics-based Motion Imitation and Skill Learning. Physics-based character control traditionally relies on motion capture data and reinforcement learning to produce physically plausible behaviors. Early work (40) establishes the foundational paradigm of training tracking policies against reference motion clips, enabling robust imitation of a wide range of physically plausible skills. To reduce reliance on hand-crafted tracking rewards, adversarial approaches (42; 41; 57; 75; 65; 6; 11) are introduced to encourage stylistic realism without explicit motion matching, with some also learning reusable latent skill spaces from unstructured motion data for high-level task reuse. In parallel, VAE-based methods (25; 70; 77; 35) provide complementary benefits through probabilistic latent modeling, learning structured and diverse skill representations that support behavioral diversity and downstream task reuse. Diffusion-based methods (37; 59; 13; 49) further broaden the generative toolkit by leveraging pretrained motion diffusion models as either reusable behavioral priors or direct policy parameterizations for physics-based control. Despite these advances, both families of methods are typically designed around a fixed training corpus with homogeneous supervision, making it challenging to incrementally expand the skill repertoire as new data sources or conditioning modalities become available. Large-scale Motion Priors and Language-conditioned Control. To improve coverage and generalization beyond small task-specific datasets, a line of work (30; 31) scales motion tracking to large corpora and distills the acquired motor skills into a universal physics-based latent space, enabling diverse downstream tasks to reuse a shared motor representation. Building on this, methods such as (55; 66) unify motion tracking with richer conditioning signals such as language and kinematic constraints within a single model, significantly broadening the interface through which users can direct character behavior. Similarly, works like (18; 19; 46) train physics-based controllers directly conditioned on language commands, scaling to thousands of diverse skills. However, distillation-based approaches may introduce some capability loss relative to the original tracking experts, and adapting these models to complex downstream tasks can remain challenging, as the breadth of the learned skill space makes it difficult to reliably activate task-relevant behaviors without additional guidance. Figure 2. HetSkills progressively integrates four representative heterogeneous skills, including motion tracking ℱtrc F^trc, text-to-motion ℱt2m F^t2m, motion completion ℱmoc F^moc, and downstream task adaptation ℱtask∗ F^task^*, within a unified latent space. Each skill maps its heterogeneous goal signals to part-wise arm and body latents via its own task encoder. After motion tracking learns the shared state encoder and motion decoder, these modules are frozen and reused by all subsequent skills. The downstream task adaptation adopts a language-guided compositional module to blend frozen motion priors with task-conditioned residual corrections, enabling lightweight adaptation without task-specific demonstrations or reference motions. Part-Wise Motion Learning. Part-wise decomposition is widely explored in kinematics-based motion generation to improve motion controllability, diversity, and compositionality (17; 62). In physics-based character control, related works further leverage part-wise structure to compose partial motion priors, decouple imitation objectives, or facilitate part-wise planning, enabling the synthesis of more diverse and interaction-rich behaviors (2; 69; 21). Another line introduces part-wise latent priors or modular skill representations within hierarchical control pipelines, enabling more structured composition and improved task adaptability (1; 14). While these works collectively demonstrate the benefits of part-wise decomposition for compositional control and generalization, they typically instantiate part-wise structure within a specific motion-prior, imitation, planning, or modular-control framework. This makes their skill spaces effective for composing behaviors under a fixed interface, but less suited for progressively accommodating heterogeneous skill types, supervision signals, and conditioning modalities within a unified representation. Downstream Task Adaptation with Motion Priors. A natural paradigm for downstream task learning is to freeze a pretrained motion prior and train a high-level policy that operates in the learned latent space (41; 57; 31; 77; 11), allowing task policies to inherit motion naturalness without retraining the low-level controller. To better handle complex or contact-rich scenarios, token-based adaptation methods (39; 60) introduce task-specific tokens as a lightweight interface between a frozen pretrained policy and new task objectives. Another direction repurposes pretrained generative models (37; 58) as reusable behavioral priors that provide motion naturalness constraints or planning guidance during task optimization. While these approaches demonstrate the value of structured motion priors for downstream learning, the underlying skill representation is generally fixed after pretraining, and the semantic interface it exposes is limited, making it difficult to flexibly compose and steer behavioral distributions toward diverse downstream objectives without additional supervision or data. 3. Problem Formulation We formulate physics-based character control as a goal-conditioned Markov decision process (27), where a policy acts according to the current state sts_t and a goal signal gtg_t. At each timestep t, the character executes an action ata_t, transitions to the next state st+1∼p(⋅∣st,at)s_t+1 p(· s_t,a_t), and receives a scalar reward rtr_t. The objective is to maximize the expected discounted return (1) J⁡(π)=⁡[∑t=0Tγtrt],J(π)=E\! [ _t=0^Tγ^tr_t ], where γ∈[0,1)γ∈[0,1) is the discount factor (53). In our setting, actions are target joint rotations that are converted to torques through proportional-derivative (PD) controllers (54) in the physics simulator. The key challenge is that the goal signal takes heterogeneous forms across skills. Depending on the task, gtg_t may correspond to future reference poses for motion tracking, language descriptions for text-to-motion generation, sparse temporal constraints for motion completion, or task-specific observations and language conditions for downstream adaptation. These goals differ in modality, supervision, and data distribution, making it difficult to consolidate them under a single controller. Our objective is therefore to learn a unified latent space that serves as a shared control interface across heterogeneous skills. Instead of training a separate low-level controller for each task, each skill predicts a latent ztz_t from its own goal specification, and a shared decoder maps ztz_t and sts_t to the final physical action. Under this formulation, the central problem is to progressively construct a latent control space that can incorporate new skills from different training stages while preserving previously learned behaviors and supporting efficient transfer to downstream tasks. 4. HetSkills: Progressive Heterogeneous Skill Learning The proposed HetSkills is a physics-based character control framework that progressively learns heterogeneous skills within a unified latent space. As shown in Fig. 2, the key idea is to treat this latent space as a shared executable interface for skill acquisition, composition, and reuse. Accordingly, skills learned from different data sources, supervision forms, and training stages can be incorporated into a common control representation. To support scalable reuse and downstream transfer, HetSkills further combines this unified latent space with a standardized task interface, allowing newly introduced skills to be integrated without retraining separate low-level controllers. Specifically, HetSkills begins by learning a tracking skill ℱtrc F^trc, which simultaneously constructs the unified latent space and produces the shared motion decoder reused across all subsequent stages. To improve compositional control and generalization, ℱtrc F^trc adopts a part-wise character decomposition that factorizes the humanoid into coordinated body partitions, each governed by a dedicated latent command within a common control architecture. After training, the state encoder and part-wise decoders are frozen and directly reused, avoiding an additional distillation step and its associated performance degradation. The remaining skills are built on top of this fixed control substrate, each introducing a distinct form of goal conditioning over the shared latent space. ℱt2m F^t2m grounds natural language descriptions into latent motion commands through Motion Intuition Distillation (MID), improving semantic robustness under challenging initializations. ℱmoc F^moc handles sparse or partial observations under a unified sparse-goal formulation, covering VR-driven tracking, motion in-betweening, and human-scene interaction. ℱtsk∗ F^tsk* reuses the frozen text-conditioned prior for downstream tasks, composing part-wise language instructions through a learned routing mechanism without requiring task-specific demonstrations or retraining. Across all stages, skills are formulated under a standardized task description, which provides a skill-agnostic interface for consistent skill integration and flexible sequential composition in long-horizon tasks. 5. ℱtrc F^trc: Tracking Skill We begin by learning a tracking skill ℱtrc F^trc to construct the unified latent space that underlies all subsequent skills. The key idea is to learn a general tracking controller with a compact and reusable latent space that remains expressive for diverse later skills. Therefore, subsequent skills can operate in the same control space without relearning low-level dynamics. To support compositionality, we adopt a part-wise architecture that factorizes the character into coordinated body partitions, while preserving a shared global context. Fig. 3 provides an overview of the tracking architecture. The tracker consists of a shared state encoder ℰsE_s, two part-wise future encoders ℱatrc F^trc_a and ℱbtrc F^trc_b, and two part-wise low-level controllers aD_a and bD_b. The future encoders predict compact deterministic latent commands zatz_a^t and zbtz_b^t for the arm and body partitions, which are decoded into joint-level actions by the controllers conditioned on the shared state feature. After training, ℰsE_s, aD_a, and bD_b are frozen and reused as a shared motion controller across all subsequent stages, avoiding an additional distillation step and its associated performance degradation. The following subsections detail the model representation, part-wise architecture, and training objective in turn. Figure 3. Architecture of the tracking skill ℱtrc F^trc. The tracking skill takes proprioception, the previous action, and future target poses as input, then predicts separate latent commands for the arm and body branches. A shared state encoder extracts the current physical context, while part-wise controllers decode the arm and body latents into joint-level target actions. After tracking training, the shared state encoder and part-wise controllers are frozen as the common motion decoder for all later skills, while ℱtrc F^trc itself is retained as one heterogeneous skill. 5.1. Model Representation As shown in Fig. 4, we use a physically simulated humanoid based on the neutral SMPL (29) body model with 6969 degrees of freedom, following prior physics-based character control works (33; 30; 55). All rotations are represented using the continuous 6D parameterization (76). For notational convenience, we apply δr(⋅) _r(·) to denote a quantity expressed relative to the current root frame, and δg(⋅) _g(·) to denote the difference between a target quantity and its current counterpart. Proprioception. The policy observes the current body configuration as (2) st=(δr(θt),θ˙t,δr(vt),hroott),s^t= (\, _r(θ^t),\; θ^t,\; _r(v^t),\;h^t_root\, ), where θtθ^t and θ˙t θ^t denote the joint rotations and angular velocities, vtv^t denotes the linear velocities, and hrootth^t_root is the root height above the ground. Goal observation. The goal input consists of the next K target poses from the reference motion, gt=[f^t+1,…,f^t+K]g^t=[ f^t+1,…, f^t+K]. Each target pose f f is represented as (3) f^=(δr(p^),δg(p),δr(θ^),δg(θ),δg(v),δg(θ˙)), f= (\, _r( p),\; _g(p),\; _r( θ),\; _g(θ),\; _g(v),\; _g( θ)\, ), where the first two terms denote the target body positions and position errors in the current root frame, the next two denote the target rotations and rotation errors, and the last two denote the linear and angular velocity differences between the current and target poses. Figure 4. Visualization models of different HetSkills stages. The leftmost character is the physically simulated humanoid used for control, while the other four styled characters visualize the learned skill families, including a cowboy for motion tracking ℱtrc F^trc, a Santa for text-to-motion ℱt2m F^t2m, a gamer for motion completion ℱmoc F^moc, and a courier for downstream task adaptation ℱtask∗ F^task^*. These visual appearances distinguish skills in demonstrations. All skills share the same underlying physical character and latent controller. Action Space. The tracking skill takes the current proprioception sts^t, future reference goal gtg^t and the previous action at−1a^t-1 as input, and predicts part-wise latent actions in the corresponding latent space: (4) zpt=ℱptrc(st,gt,at−1),p∈a,b,z_p^t= F^trc_p(s^t,g^t,a^t-1), p∈\a,b\, where zatz_a^t and zbtz_b^t denote the latent actions for the arm and body partitions, respectively. These latent actions are then decoded into joint-level target actions through the corresponding state feature: (5) ct=ℰs(st),at=[a(ct,zat),b(ct,zbt)],c^t=E_s(s^t), a^t= [D_a(c^t,z_a^t),\;D_b(c^t,z_b^t) ], where ℰsE_s is the shared state encoder, and aD_a and bD_b are the part-wise low-level controllers for the arm and body partitions. The final action ata^t consists of target joint rotations for all actuated degrees of freedom, which are converted to joint torques via PD controllers (54). PD controllers are widely used in physics-based character animation (68; 73; 40) because they provide a stable and intuitive interface between learned policies and physical simulation, abstracting away low-level torque computation while remaining responsive to perturbations. 5.2. Part-wise Architecture Part-wise architecture is designed to improve compositionality and latent controllability while preserving coordinated whole-body behavior. The core idea is to factorize the humanoid into a small number of semantically meaningful body partitions, and to let each partition predict its own latent action within a shared control framework. This design encourages specialization for different motion patterns, while maintaining global consistency through a common state representation. In addition, we construct the latent space to be deterministic and compact. Therefore, it can serve as a stable substrate for subsequent skill learning and downstream composition. Part-wise Decomposition. We decompose the humanoid into two kinematic partitions: an arm part that groups both hands and arms, and a body part that contains all remaining joints. This decomposition is motivated by the observation that motion datasets often include fine-grained upper-limb behaviors, such as gesturing, punching, and object interaction, whose high-frequency and low-inertia dynamics differ substantially from those of locomotion and balance. A dedicated arm branch therefore allows the model to specialize in these behaviors without entangling them with lower-body control. Compared with finer-grained decompositions (14; 1), the two-part split provides a favorable trade-off between expressiveness and simplicity. It achieves strong tracking quality across diverse motion categories while avoiding redundant parameters and excessive latent channels that would complicate downstream skill composition. We validate this design choice in Section 10.1. Network Structure. With this factorization, the tracking policy consists of five modules, including a shared state encoder ℰsE_s, two part-wise future encoders ℱatrc F_a^trc and ℱbtrc F_b^trc, and two part-wise low-level controllers aD_a and bD_b. At each timestep, ℰsE_s maps the current observation to a shared state feature ctc^t that captures the full-body context. In parallel, ℱatrc F_a^trc and ℱbtrc F_b^trc each take a two-frame window of future reference motion and predict compact latent commands zatz_a^t and zbtz_b^t for the arm and body partitions, respectively. We adopt two future frames because this improves tracking accuracy and provides stronger supervision when the tracker later serves as an expert. At inference time, a single reference frame is sufficient by duplicating it to form the two-frame input. The shared feature ctc^t is concatenated with each part-wise latent and passed to the corresponding controller to produce joint-level actions. Because both controllers condition on the same shared state feature, the two branches retain access to a consistent global body context and can coordinate without explicit cross-branch communication. At the same time, the separate latent commands and decoders allow each branch to specialize in its own body partition. Therefore, the latent action captures the intended motion while the shared feature anchors it to the current physical state. Figure 5. Learned body and arm latent spaces. t-SNE projections show that motions from different categories, including locomotion, gesture, acrobatics, and other behaviors, form distinguishable distributions, even though no explicit category labels are used during latent learning. This indicates that the deterministic and compact latent representation captures semantically meaningful motion structure, which supports later skill reuse and composition. Deterministic Latent Space. Existing prior latent-variable policies (28; 25; 55) typically model the latent space as a stochastic distribution, while our future encoders produce deterministic latent representations (12; 74). In stochastic formulations, exploration noise is injected directly into the latent space (44), which encourages the encoder to increase latent magnitudes in order to preserve discriminability among different skills. This can degrade tracking precision, destabilize the latent scale, and introduce an additional sensitive KL-divergence coefficient (24; 3). We instead keep the latent space deterministic and inject exploration noise only at the final action space, where it promotes exploration without distorting the learned latent geometry. To further regularize the latent representation, we apply two complementary objectives: a latent magnitude penalty that encourages compact representations, and an AR(1) temporal smoothness penalty (35) that discourages abrupt latent changes between consecutive timesteps: (6) ℒmr=∑p∈a,b⁡[‖zpt‖22],ℒar=∑p∈a,b⁡[‖zpt−ϕzpt−1‖2].L_mr= _p∈\a,b\E [\|z_p^t\|_2^2 ], _ar= _p∈\a,b\E [\|z_p^t-φ z_p^t-1\|_2 ]. Together, the deterministic design and these regularizers yield a compact and well-structured latent space. As illustrated by the t-SNE visualization in Fig. 5, motions from different categories, including locomotion, gesture, acrobatics, and other behaviors, exhibit distinct distributional tendencies in the learned latent space, suggesting that the representation captures semantically meaningful structure without explicit category supervision. This structure is also beneficial for downstream skill composition, since later modules only need to predict a latent point per part rather than match an entire latent distribution. 5.3. Training Objective The tracking skill is trained to imitate reference motions while simultaneously shaping the latent space to be compact, smooth, and reusable. To this end, we optimize the controller with reinforcement learning using motion-tracking rewards, and augment the policy objective with latent-space regularization. We further adopt sampling and termination strategies that improve training efficiency on large and diverse motion datasets. We train the tracking policy with proximal policy optimization (PPO) (48) to imitate reference motions. Each reward term takes the form r⁡(x,k)=exp⁡(−k‖x‖)r(x,k)= (-k\|x\|). The full tracking reward is defined as (7) rt=∑j∈wjr(δg(xt,j),kj)+wctrt,ct+wsmrt,sm+wegrt,eg,r^t= _j w^j\,r\! ( _g(x^t,j),\,k^j )+w^ctr^t,ct+w^smr^t,sm+w^egr^t,eg, where =gp,gr,jv,jav,rhJ=\gp,gr,jv,jav,rh\ corresponds to global joint positions, global joint rotations, joint velocities, joint angular velocities, and root height, respectively. Here, rt,ctr^t,ct denotes the contact reward, which encourages correct foot contact behavior. The final two terms correspond to an action smoothness penalty and an energy penalty, which jointly encourage smoother motions. Detailed reward weights and coefficients are provided in the supplementary material. The overall training objective combines the PPO loss with the latent regularization terms introduced in Eq. 6: (8) ℒtrc=ℒppo+λmrℒmr+λarℒar.L_trc=L_ppo+ _mrL_mr+ _arL_ar. This objective improves tracking fidelity and encourages the learned latent actions to remain compact and temporally coherent, which is important for later skill reuse and downstream composition. To improve training efficiency on diverse motion datasets, we additionally adopt early termination (40) and prioritized motion sampling (31). An episode is terminated when any joint position deviates from the reference beyond a predefined threshold, preventing training from being dominated by undesirable states. Motions with higher failure rates are sampled more frequently, allowing training to focus on challenging and underrepresented behaviors. After training, we freeze ℰsE_s, aD_a, and bD_b and reuse them as the shared decoder in subsequent stages. 6. ℱt2m F^t2m: Text-to-Motion Skill Figure 6. Architecture of the text-to-motion skill ℱt2m F^t2m. The frozen TMR text encoder provides a semantic language embedding, while the current proprioception-action input and recent history states are projected into tokens. Two part-wise Transformer encoders predict arm and body latents in the shared latent space, which are then decoded by the frozen state encoder and part-wise controllers into physical actions. This design isolates language grounding from low-level control and allows text-conditioned motion generation to reuse the tracking-learned controller. We learn a text-to-motion skill ℱt2m F^t2m that maps natural language descriptions to part-wise latent actions within the unified latent space, with the low-level controller frozen from the tracking skill. The goal is to ground natural language descriptions into reusable latent actions while preserving the motion quality and controllability provided by the tracking decoder. A major challenge is that text-conditioned models can easily exploit shortcut pathways, such as privileged future-motion cues or overly regular training initialization, instead of learning meaningful language semantics. Our design therefore emphasizes semantic grounding under imperfect context, so that the learned skill remains robust under changed initial states, mismatched histories, and other challenging inference conditions. The architecture of the text-to-motion skill is illustrated in Fig. 6. It consists of two part-wise text encoders ℱat2m F^t2m_a and ℱbt2m F^t2m_b, which predict latent actions for the arm and body partitions, respectively, while reusing the frozen state encoder ℰsE_s and part-wise controllers aD_a and bD_b from the tracking stage. The following subsections detail MID, the model architecture, and the training objective in turn. 6.1. Motion Intuition Distillation The key idea of Motion Intuition Distillation(MID) is to remove those training shortcuts that would otherwise allow the model to bypass semantic understanding. Instead of predicting motion under near-perfect future guidance or always starting from a matched initial state, the model must infer the intended behavior from language, the current state, and imperfect historical context. This encourages the text-conditioned policy to acquire a more robust motion intuition that generalizes beyond the training distribution. Shortcut Issue. A major challenge in learning text-to-motion skills within a unified latent space is that the model can easily rely on shortcut pathways instead of learning meaningful language semantics. When future motion information is indirectly accessible, whether through residual branches, future-motion conditions, or other forms of privileged information, the model tends to shift the main modeling burden to these easier pathways, since predicting the next latent from nearby future frames requires far less abstraction than grounding language descriptions into motion. As a result, the residual structure may acquire overly strong compensatory behavior that effectively bypasses the text encoder, while the component responsible for semantic understanding remains insufficiently trained. In addition, when training episodes always start from the ground-truth initial frame together with its matching motion history, the model can overfit to local state-transition patterns and achieve high in-distribution accuracy without learning to handle changed initial states, mismatched histories, or motion transitions. Formulation. To address this issue, we propose MID, which removes dependence on future motion and standard initialization. The text-to-motion encoders ℱat2m F^t2m_a and ℱbt2m F^t2m_b receive only the current state, past history, and the text embedding. As a result, the model infers the intended motion from the language condition, the current state, and imperfect historical context. We first adopt Reference State Initialization (RSI) (40), where, with probability prsi=0.7p_rsi=0.7, the initial state of a training episode is sampled uniformly from any frame along the reference clip rather than taken from the first frame of the target motion. We further introduce Randomized Memory Initialization (RMI). With probability prmi=0.2p_rmi=0.2, we uniformly sample another motion sequence from the motion library, use its terminal frame as the initial state, and fill the history buffer with a real history segment from the end of that randomly sampled sequence. Under this training scheme, the model cannot always rely on an initial frame and history prefix that strictly match the target motion. Instead, it learns to infer the subsequent behavior from language semantics and the current observation. We validate the necessity of each component in Section 10.2. 6.2. Model Architecture The text-to-motion architecture reuses the latent space and shared decoder learned by the tracking skill, and only learns the mapping from text-conditioned observations to latent actions. This design keeps the low-level control substrate fixed and shifts the learning burden to semantic grounding in latent space. As a result, the model can leverage a strong motion prior learned from large unstructured motion data while adapting it to language supervision with comparatively limited text-annotated data. Concretely, at each step the skill receives the proprioception sts^t, the previous action at−1a^t-1, an observation history HtH^t, and a text embedding e, and maps them to part-wise latent actions: (9) zpt=ℱpt2m(st,at−1,Ht,e),p∈a,b,z_p^t= F^t2m_p(s^t,a^t-1,H^t,e), p∈\a,b\, where zatz_a^t and zbtz_b^t denote the latent actions predicted for the arm and body partitions, respectively. These latent actions are then decoded into the final joint-level action through the frozen shared decoder: (10) at=[a(ℰs(st),zat),b(ℰs(st),zbt)].a^t= [D_a(E_s(s^t),z_a^t),\;D_b(E_s(s^t),z_b^t) ]. Input Representation. The proprioception sts^t follows the same definition as in the tracking skill. The text embedding e is obtained from a frozen Text-to-Motion Retrieval (TMR) text encoder (43), which is trained through contrastive learning to align language and motion in a shared embedding space. This provides a semantically structured representation that facilitates grounding text descriptions into latent motion commands. The observation history HtH^t consists of six frames uniformly subsampled from the past two seconds of simulation, providing sufficient temporal context for the model to infer the current motion phase and dynamics. Skill Encoders. Two part-wise Transformer (61) encoders ℱat2m F^t2m_a and ℱbt2m F^t2m_b take sts^t, at−1a^t-1, e, and HtH^t as input and predict latent actions zatz_a^t and zbtz_b^t in the arm and body latent spaces, respectively. The two encoders share the same architecture but maintain separate parameters, enabling each branch to specialize in the dynamics of its corresponding body partition. The predicted latents are then decoded by the frozen state encoder ℰsE_s and part-wise controllers aD_a and bD_b to produce joint-level actions. Shared Decoder Reuse. Freezing the decoder provides two practical benefits. First, it removes the need to jointly learn low-level control during text-to-motion training, reducing the optimization problem to latent prediction alone. Second, it allows the text-to-motion skill to build on top of a controller already trained on a large and diverse motion dataset. Therefore, even a comparatively small text-annotated dataset can suffice to learn semantically grounded motion skills on top of a general-purpose motor foundation. 6.3. Training Objective We train the text-to-motion encoders using the frozen tracking skill ℱtrc F^trc as the expert teacher (47). During training, the text-to-motion skill autonomously interacts with the environment, and at each time step, ℱtrc F^trc observes the ground-truth future reference frames to provide the expert action a^t a^t for the current state. Rather than supervising in the latent space, we compare the final joint-level actions ata^t and a^t a^t produced after decoding through the shared controllers. Therefore, the loss naturally accounts for the nonlinear mapping from latent to action space. The overall training objective is (11) ℒt2m=⁡[‖at−a^t‖2]+λmrℒmr+λarℒar.L_t2m=E\! [\|a^t- a^t\|_2 ]+ _mrL_mr+ _arL_ar. The two regularization terms follow the same form as in Eq. 6, penalizing latent magnitude and encouraging temporal smoothness. These regularizers help maintain a compact and well-structured latent distribution. Since our downstream module predicts residual latent adjustments on top of the text-to-motion output, a well-regularized base distribution makes such residual learning more stable and effective. 7. ℱmoc F^moc: Motion Completion Skill We introduce ℱmoc F^moc as a motion completion skill that extends the latent space to tasks requiring the recovery of physically plausible full-body motion from sparse or partial observations (8; 5; 45; 38; 15). These tasks differ in the form of their conditioning signals. However, they all require the controller to infer coherent whole-body behavior from incomplete information. Some involve spatially sparse observations, as in VR-driven body tracking where only a small set of end-effector trajectories is available, while others involve temporally sparse observations, as in motion in-betweening where only scattered keyframe poses are provided. The key idea of this stage is to express these diverse signals uniformly as partial goal specifications and handle them within the same latent space. The motion completion skill therefore serves as a unified control interface for sparse-observation tasks. Instead of introducing a separate controller for each conditioning type, we learn a single skill family that maps the current proprioception together with sparse target observations to latent actions compatible with the shared decoder. In this work, we instantiate this framework in three representative settings, including VR tracking, motion in-betweening, and human-scene interaction. These cases cover both spatially sparse and temporally sparse conditioning, and together illustrate that the shared latent space can support a broader family of goal specifications beyond motion tracking and language-guided generation. 7.1. Unified Formulation Let t=(tk,g~kt)k=1KO^t=\(t_k, g_k^t)\_k=1^K denote the sparse target observations, where each tkt_k is a relative time offset and g~kt g_k^t is the corresponding partial goal feature. The motion completion skill takes the current proprioception sts^t, the previous action at−1a^t-1, and the sparse observation set tO^t as input, and predicts part-wise latent actions: (12) zpt=ℱpmoc(st,at−1,t),p∈a,b,z_p^t= F^moc_p(s^t,a^t-1,O^t), p∈\a,b\, where zatz_a^t and zbtz_b^t denote the latent actions for the arm and body partitions, respectively. The resulting latent actions are decoded into the final joint-level target action through the shared decoder: (13) ct=ℰs(st),at=[a(ct,zat),b(ct,zbt)].c^t=E_s(s^t), a^t= [D_a(c^t,z_a^t),\;D_b(c^t,z_b^t) ]. VR Tracking. In VR tracking, the conditioning signal consists of the full kinematic state of three end-effector bodies, namely the head and both hands, from the next reference frame. This corresponds to a spatially sparse observation setting, where only a small subset of body parts is directly specified and the controller infers the remaining full-body motion. We train the skill with a behavior cloning objective, (14) ℒBC=⁡[‖at−a^t‖2],L_BC=E\! [\|a^t- a^t\|_2 ], using the stage-1 tracker as the expert teacher. We additionally apply the latent magnitude regularization term ℒmrL_mr from Eq. 6. The AR(1) smoothness term ℒarL_ar is omitted in this case, since strong immediate fidelity to sparse spatial targets is more important than long-horizon temporal smoothing. Motion In-betweening. In motion in-betweening (51; 22; 20), the conditioning signal consists of a future full-body pose together with its time offset τtτ^t. The target frame is uniformly sampled from a future horizon of 55 to 3030 frames, and once the character reaches it, a new target frame is resampled from the same range. This corresponds to a temporally sparse observation setting, where the controller synthesizes plausible intermediate motion that connects scattered target poses. Training follows the same behavior cloning objective ℒBCL_BC, while both ℒmrL_mr and ℒarL_ar are applied to encourage compact latent representations and temporally coherent transitions across keyframes. Figure 7. Examples of human-scene interaction generated by HetSkills. The characters interact with diverse everyday objects such as chairs, sofas, and tables, producing sitting, reclining, and other object-conditioned motions within the same shared control framework. Human-scene Interaction. Human-scene interaction (52; 10) can be cast under the same formulation as motion in-betweening (16). The conditioning signal consists of a future target interaction state together with its time offset, sampled from the same rolling future horizon. We train this skill on the SAMP dataset (9), which contains motions of characters interacting with everyday objects such as chairs and sofas, shown in Fig. 7. SAMP lies outside the distribution of the tracker training data. However, the generalization capacity of the shared latent space allows the same control framework to be reused without modification. For this setting, we adopt reinforcement learning instead of behavior cloning, as direct optimization against a tracking-style reward yields stronger performance under distribution shift. A key advantage of the unified latent space is that each motion completion skill can be trained independently on its own motion distribution while remaining fully compatible with the same shared control interface. 8. ℱtsk∗ F^tsk^*: Language-Guided Downstream Adaptation We introduce ℱtsk∗ F^tsk^* as a language-guided downstream adaptation skill that reuses the language-conditioned motion distribution induced by ℱt2m F^t2m for new task objectives. Rather than retraining the motion prior or the low-level controllers, ℱtsk∗ F^tsk^* learns lightweight task-specific guidance on top of the shared latent space. Therefore, downstream behaviors remain natural and human-like while adapting to novel tasks. To accommodate heterogeneous downstream objectives, this stage combines a standardized task interface with compositional guidance based on part-wise language priors and task-conditioned residual correction. Concretely, ℱtsk∗ F^tsk^* consists of three components: a standardized task description that provides a unified interface for representing and organizing tasks, a compositional task guidance module that combines multiple part-wise language priors under the current task context, and a lightweight adaptation objective that optimizes only the downstream guidance modules while keeping the pretrained motion prior fixed. The following subsections detail these three components in turn. 8.1. Standardized Task Description To support heterogeneous skill learning and composition, HetSkills represents all skills through a unified task interface. Specifically, both previously learned skills and downstream task skills are formulated using a standardized task description that is independent of their specific training procedures. This abstraction provides a consistent way to specify, organize, compose, and extend skills, and further serves as the basis for long-horizon task execution in the shared latent space. Task Unit. We abstract the execution of each skill as a standardized task unit (15) i=(Initializationi,Conditioni,Terminatei),T_i= (Initialization_i,\ Condition_i,\ Terminate_i ), where each component plays a distinct role. Initialization specifies the initialization protocol for the skill, including any required state resets, memory initialization, or environment configuration that must be established before execution begins. Condition encodes the task-specific guidance that governs the behavior during execution. Depending on the skill type, this may take the form of a language instruction, a target goal state, a reference trajectory, or another modality that parameterizes the desired behavior. Terminate defines the criterion under which the skill is considered complete, such as a fixed execution horizon, a goal-reaching condition, or a learned termination signal. Task composition. This interface is deliberately agnostic to skill type. A language-conditioned motion generation skill, a human-scene interaction skill, and a goal-conditioned locomotion skill can all be expressed under the same iT_i abstraction, differing only in how each component is instantiated. Since all skills share this interface, they can be executed independently or organized sequentially as (16) Seq⁡(1,2,…,N),Seq(T_1,T_2,…,T_N), which provides a unified mechanism for sequential scheduling and long-horizon skill composition. 8.2. Compositional Task Guidance A single language condition is often insufficient to represent the full complexity of a downstream task, especially when the desired behavior involves blending multiple motion styles or switching between behaviors over time. Our solution is to combine multiple part-wise language priors through a learned routing mechanism and then refine the resulting latent action with a task-conditioned residual. This produces a flexible adaptation module that remains grounded in the pretrained motion prior while retaining task-specific expressiveness. Instruction Set. For each body part, we consider a set of M language conditions (17) ℐp=epmm=1M,p∈a,b,I_p=\e_p^m\_m=1^M, p∈\a,b\, where each instruction epme_p^m provides a distinct semantic description of the target behavior. Conditioned on these part-wise language inputs, the corresponding frozen language priors produce multiple latent candidates: (18) zpt,m=ℱpt2m(st,at−1,Ht,epm),p∈a,b.z_p^t,m= F_p^t2m(s^t,a^t-1,H^t,e_p^m), p∈\a,b\. Latent Routing. Since the most appropriate motion style depends on the current task context, we learn a gating function pG_p for each body part. It takes the current proprioceptive state and a task-specific observation otaskto^t_task as input and outputs normalized routing weights over the M instruction branches: (19) pt=softmax⁡(p(st,at−1,otaskt)),p∈a,b. α_p^t=softmax\ \! (G_p(s^t,a^t-1,o^t_task) ), p∈\a,b\. Here, pt=αpt,mm=1M α_p^t=\ _p^t,m\_m=1^M denotes the routing weights, where αpt,m _p^t,m is the weight assigned to the m-th branch for part p. Residual Refinement. While a weighted combination of language-prior latents approximates the target motion style, the resulting latent remains loosely coupled to task-specific objectives, as the language priors encode general motion distributions without direct awareness of task constraints. We therefore introduce a task-conditioned residual module ℛpR_p for each body part and define the final part-wise latent as (20) z^pt=∑m=1Mαpt,mzpt,m+ℛp(st,at−1,otaskt),p∈a,b. z_p^t= _m=1^M _p^t,mz_p^t,m+R_p(s^t,a^t-1,o^t_task), p∈\a,b\. The resulting latents z^at z_a^t and z^bt z_b^t are then decoded by the frozen part-wise controllers using the same procedure as in the pretrained model. Input: Frozen priors ℱpt2m F_p^t2m, frozen decoder D; instruction sets ℐp=epmm=1MI_p=\e_p^m\_m=1^M, p∈a,bp∈\a,b\; gating networks pG_p, residual networks ℛpR_p, critic VϕV_φ; fixed regularization coefficients λlmp,λsmooth,zbound _lmp, _smooth,z_bound Output: Trained MoE gating networks pG_p, part-wise residual networks ℛpR_p, and value network VϕV_φ for each training iteration do for each environment i (parallel) do Observe sts^t, at−1a^t-1, HtH^t, otaskto^t_task; for each language condition epm∈ℐpe_p^m _p do Compute zpt,mz_p^t,m via Eq. (18); end for Compute gating weights pt α_p^t via Eq. (19); Compute aggregated latent z^pt z_p^t via Eq. (20); z^t←[z^at;z^bt] z^t←[ z_a^t;\, z_b^t]; Sample zt∼⁡(z^t,diag⁡(σ2))z^t ( z^t,diag(σ^2)), store log⁡πθ(zt) _θ(z^t); at←⁡(st,zt)a^t (s^t,z^t), collect reward rtr^t, store transition; end for Compute advantages A^t A^t and returns R^t R^t via GAE; for each mini-batch from rollout buffer do Compute ℒtskL_tsk via Eq. (24); Update p,ℛp,VϕG_p,R_p,V_φ via ∇ℒtsk _tsk; end for end for return ℱtsk∗=ℐp,p,ℛpp∈a,b F^tsk^*=\I_p,G_p,R_p\_p∈\a,b\; ALGORITHM 1 Language-Guided Downstream Adaptation 8.3. Task Adaptation Downstream adaptation optimizes only the lightweight guidance modules while keeping all pretrained priors and decoders fixed. This design preserves the motion naturalness encoded in the shared latent space and restricts task learning to the level of latent composition and correction. In practice, however, optimizing only task rewards can still drive the latent actions away from the motion-prior distribution. We therefore regularize the adapted latents to maintain stable and natural behavior. Fig. 8 shows representative downstream task examples under this adaptation setting. Trainable modules. We train the compositional task guidance modules pG_p and ℛpR_p using PPO (48), while keeping all pretrained priors and the decoder frozen. The full procedure is summarized in Algorithm 1. After training, a new downstream skill is characterized by its instruction set and learned guidance module: (21) ℱtsk∗=ℐp,p,ℛp,p∈a,b. F^tsk^*=\I_p,\,G_p,\,R_p\, p∈\a,b\. Latent regularization. Empirically, optimizing only the task reward tends to drive the latent action away from the motion-prior distribution, resulting in motion jitter and degraded naturalness. To mitigate this effect, we introduce two regularization terms. The latent magnitude penalty ℒlmpL_lmp is activated only when z z exceeds a predefined threshold zboundz_bound: (22) ℒlmp=⁡[(max⁡(|z^|−zbound,0))2].L_lmp=E [ ( (| z|-z_bound,0 ) )^2 ]. The latent smoothness penalty ℒsmoothL_smooth encourages temporal consistency by penalizing large differences between latent actions at consecutive timesteps: (23) ℒsmooth=⁡[‖z^t−z^t−1‖2].L_smooth=E [\| z^t- z^t-1\|^2 ]. Optimization objective. The full downstream training objective combines the PPO loss with the two regularization terms: (24) ℒtsk=ℒppo+λlmpℒlmp+λsmoothℒsmooth.L_tsk=L_ppo+ _lmpL_lmp+ _smoothL_smooth. Together, these terms keep the adapted latent actions within a stable region of the shared control space and improve the naturalness and robustness of the resulting motions. 9. Experimental Setup All experiments are conducted in Isaac Lab (36) using ProtoMotions (56), with physics simulation running at 120 Hz and control policy execution at 30 Hz. Our framework is trained in four progressive stages using two consumer-grade NVIDIA RTX 5090 GPUs. Detailed hyperparameters and implementation specifics are provided in the appendix. 9.1. Datasets Our progressive training paradigm naturally supports heterogeneous data sources across different stages, eliminating the need to unify all data under a single annotation format. This approach allows each stage to leverage the data best suited to its task objectives. Specifically, for motion tracking, we use AMASS (34) as the base motion dataset and follow the filtering pipeline of PHC (30) to clean the data. This pipeline removes clips that exhibit non-physical artifacts, such as limb penetration, body floating, and interactions with unmodeled objects, yielding a high-quality set of training sequences. This filtered set is also reused for VR tracking and motion in-betweening, where sparse conditioning signals are constructed directly from the same clips. For VR tracking, only the kinematic states of the head and both hands are retained as spatially sparse end-effector observations. For motion in-betweening, a future full-body pose is sampled from a rolling horizon of 55 to 3030 frames ahead, providing temporally sparse keyframe targets. For text-to-motion, we use the HumanML3D (7) dataset to introduce natural language annotations for motion clips. Following SuperPADL (19), we discard clips shorter than 22 seconds or longer than 99 seconds, as such clips tend to be dominated by idle poses, redundant pauses, or compound actions, all of which degrade training stability and supervision quality. For human-object interaction, we incorporate the SAMP (9) dataset, which provides high-quality motion capture data covering typical furniture interactions, such as sitting and lying down. The dataset records both body motion and object spatial information simultaneously, which is essential for training models that can understand and predict interaction-based behaviors. (a) Path Follow (b) Strike Kick (c) Strike Push (d) Pick-and-Place Figure 8. Downstream task examples solved by HetSkills. The same frozen latent controller is adapted to diverse downstream tasks: (a) path follow, (b) strike with a kick, (c) strike with a push, and (d) pick-and-place. No task-specific motion demonstrations are used. Each task is learned through reward-driven lightweight adaptation that composes language-conditioned latent priors with task-conditioned residual corrections. 9.2. Tasks To evaluate the downstream task adaptation module, we design three tasks that do not require task-specific demonstration data. For all tasks, we apply an energy penalty λ⋅λ·P (λ=10−5λ=10^-5) to suppress unnecessary high-power motions. In the path follow task, this penalty is applied only to the leg joints. Full reward specifications and hyperparameters are provided in the appendix. Specifically: Path Follow The agent tracks an online-generated random path through complex terrain. At each timestep, the agent observes the next 1010 waypoints in its local coordinate frame, and the reward is defined as: (25) r=exp⁡(−‖ptargetxy−prootxy‖2)−λ⋅legr= (-\|p_target^xy-p_root^xy\|^2 )-λ·P_leg where the first term encourages the root to stay close to the current path target in the horizontal plane. Strike The agent approaches a randomly placed target and knocks it over using designated body parts, either hands for pushing or leg segments for kicking. The reward is defined as (26) r=0.6rrot+0.2rvel+0.1rprog+0.1rtoward−λ⋅r=0.6\,r_rot+0.2\,r_vel+0.1\,r_prog+0.1\,r_toward-λ·P where rrotr_rot rewards target tilt, rvelr_vel encourages approaching the target at a desired speed, rprogr_prog rewards reducing the distance to the target, and rtowardr_toward rewards facing the target. Pick-and-Place The task consists of three stages, including pick-up, carry-to, and put-down. In the pick-up stage, the agent must approach a box randomly placed on a source platform and lift it to a target height using both hands. The reward is: (27) r=0.4rlift+0.3rgrasp+0.2rforce+0.1rface−λ⋅r=0.4\,r_lift+0.3\,r_grasp+0.2\,r_force+0.1\,r_face-λ·P where rliftr_lift rewards lifting the box, rgraspr_grasp rewards approaching the grasp points, rforcer_force rewards effective bilateral contact, and rfacer_face rewards facing the box. The carry-to and put-down stages require the agent to walk along a path while maintaining the grasp, and to place the box onto a target platform, respectively. We use pick-up as the baseline comparison task, as it is the most challenging of the three stages. 9.3. Evaluation Dimensions To tackle a wide variety of tasks, including precise motion tracking, generating realistic motions from high-level language inputs, and adapting to new tasks without requiring retraining, we evaluate the performance of HetSkills across three key dimensions, with an additional long-horizon skill composition demonstration. Specifically: Tracking and Motion Completion. We evaluate the motion tracking skill ℱtrc F^trc on both the training and test splits of AMASS, and report the success rate and MPJPE (Mean Per Joint Position Error, in m) (31; 55). We also conduct ablation studies to justify our part-wise decomposition design. For the motion completion skill ℱmoc F^moc, we evaluate three instantiations that cover both spatially and temporally sparse conditioning. In VR-driven body tracking, only the head and both hands are provided as visible goal constraints, while all other body segments are masked. In motion in-betweening, the success rate is reported on the training and test splits of AMASS under the same MPJPE failure threshold. For human-object interaction, we provide qualitative examples to demonstrate the ability of the framework to handle complex scenarios. Text-to-Motion. We evaluate the text-to-motion skill ℱt2m F^t2m from two complementary aspects. First, we evaluate pose-level robustness under two initialization protocols: starting from the ground-truth first frame of each target clip and starting from a neutral pose. A rollout is considered a failure if the MPJPE exceeds a predefined threshold at any frame, and we report the success rates under both protocols. Second, we evaluate semantic alignment using the HumanML3D retrieval protocol (7) with a pretrained TMR model (43). In this evaluation, the generated motion is used to retrieve its corresponding language description from candidate texts in the shared text-motion embedding space. We report R-Precision (R@N) and MedR, where higher R@N and lower MedR indicate better semantic alignment between the generated motion and the input language. This query-based evaluation complements MPJPE by measuring whether the generated motion is recognizable as the intended textual action. Downstream Task Adaptation. We train and evaluate ℱtsk∗ F^tsk^* on the three tasks described in Section 9.2, comparing our results against relevant baselines. The language instructions used to condition each task are provided in Appendix D. 10. Experimental Results HetSkills progressively builds a unified latent space that supports motion tracking, text-to-motion generation, motion completion, and downstream task adaptation within a single shared architecture. In this section, we present the experimental results to demonstrate the effectiveness of each stage in the HetSkills pipeline. We also highlight the utility of our part-wise decomposition design, which enhances both performance and interpretability, and show how the unified latent space generalizes across heterogeneous skills and tasks. Specifically, we evaluate HetSkills on motion tracking, text-to-motion generation, and downstream task adaptation, and analyze how these capabilities contribute to the overall flexibility and robustness of the model. Building on these skills, we further provide a long-horizon composition example to show that heterogeneous abilities can be organized through the standardized task interface and executed within the same shared model. Figure 9. Qualitative results of motion tracking. Each row shows temporally ordered rollout frames generated by the tracking skill ℱtrc F^trc. The examples demonstrate that the shared decoder can reproduce diverse reference motions with stable balance, plausible contacts, and coordinated full-body control. Table 1. Motion-tracking performance on AMASS. Success rate reports the percentage of rollouts that remain below the 0.5 m MPJPE failure threshold on flat terrain. The upper block compares HetSkills with MaskedMimic and PULSE, including one-frame and two-frame inference variants of HetSkills. The lower block ablates the number of latent body partitions, showing the trade-off between tracking accuracy, part-wise interpretability, and computational efficiency. Train Test Method Success MPJPE Success MPJPE MaskedMimic 99.4% 32.9 99.2% 35.1 PULSE 99.8% 39.2 97.1% 54.1 HetSkills-1step 99.9% 29.2 99.3% 43.6 HetSkills-2step 99.9% 28.2 100% 39.4 HetSkills-1part 99.9% 31.8 100% 42.3 HetSkills-5part 100% 26.0 100% 35.5 10.1. Tracking and Motion Completion We evaluate HetSkills on motion tracking and motion completion. These tasks assess the ability of the model to generate precise motions and handle sparse conditioning. First, we present motion tracking results and compare HetSkills with baselines. Next, we evaluate motion completion using partial input data, highlighting the ability to handle varying partial observations. (a) Human-Scene Interaction: the character naturally interacts with diverse everyday objects in a variety of poses. (b) Motion In-Betweening: the character synthesizes physically plausible intermediate frames (white) between sparse keyframe poses (colored). (c) VR Tracking: physically plausible full-body motion completed from only head and hand constraints. Figure 10. HetSkills supports diverse motion completion tasks including (a) human-scene interaction, (b) motion in-betweening, and (c) VR tracking, recovering coherent full-body motion from spatially or temporally sparse observations within a single shared controller. Motion Tracking. Table 1 reports motion tracking results on the training and test splits of AMASS (34). We evaluate each method using success rate and MPJPE (Mean Per Joint Position Error, in m). Following prior work, a rollout is considered a failure if the mean per-joint position error exceeds 0.5 m at any frame (31; 55). MPJPE measures the average Euclidean distance between the predicted and reference joint positions over all joints and frames. Additional tracking rollouts are shown in Fig. 9. The upper block of Table 1 reports results for our two-part decomposition. Since ℱtrc F^trc is trained with a two-frame future window, we report two inference variants: HetSkills-2step uses two frames as during training, while HetSkills-1step duplicates a single observed frame to fill the two-frame input. HetSkills-1step achieves the highest success rate among all methods under the fair inference setting. Its MPJPE is higher than MaskedMimic only on the test set, which is expected: the distribution shift introduced by removing one future frame at inference, combined with the prioritization of success on challenging motions, leads to a slight MPJPE increase. The small gap between the two variants nonetheless confirms that duplicating a single frame suffices in practice. The lower block of Table 1 investigates the effect of part granularity. Increasing the number of parts from one to five consistently reduces the MPJPE. Specifically, when increasing from the one-part configuration (HetSkills-1part) to the five-part configuration (HetSkills-5part), the MPJPE on the test set decreases from 42.3 m to 35.5 m, showing a clear improvement in tracking accuracy. This reduction in MPJPE confirms that part-wise decomposition improves tracking precision. However, finer decompositions reduce the semantic interpretability of each part latent and increase computational overhead. Based on these trade-offs, we adopt the two-part configuration (arm part and body part) for all subsequent experiments, as it strikes a good balance between accuracy and efficiency. Motion Completion. The VR-driven body tracking task, where only the head and both hands are observable as goal constraints, is shown in Table 2. A rollout is considered a failure if the mean tracking error of the head and both hands exceeds 0.5 m at any frame. HetSkills achieves success rates of 99.9%99.9\% and 97.8%97.8\% on the training and test splits, respectively, demonstrating that the unified latent space has the capacity to support this sparse-conditioning skill along with other heterogeneous skills. HetSkills significantly outperforms other large-scale motion priors in terms of success rates, suggesting that the structured latent space provides a more robust foundation for generalizing to new conditioning forms. The higher MPJPE on the test set is expected, as the lower body is underdetermined with only three upper-body endpoints provided as constraints, allowing many plausible configurations and positional errors that do not necessarily reflect the motion quality. For motion in-betweening, HetSkills achieves success rates of 99.9%99.9\% on the training split and 100%100\% on the test split of AMASS, with both tasks using a 0.5 m full-body MPJPE failure threshold. However, success rate alone is not a comprehensive measure, as motion in-betweening allows for a wide range of plausible intermediate trajectories. A rollout that deviates from the reference is not necessarily incorrect. For human-scene interaction, HetSkills achieves a success rate of 96.1%96.1\% on the SAMP dataset, demonstrating the ability of the unified latent space to generalize to out-of-distribution interaction patterns without modifying the shared decoder. This result shows that each skill can be trained independently on its respective motion distribution while remaining fully compatible with the common control space. These results indicate that the unified latent space can accommodate heterogeneous skills from various conditioning modalities and motion distributions within a single shared framework. As quantitative metrics alone do not fully capture motion naturalness and plausibility, we encourage viewers to refer to the supplementary video for a qualitative demonstration of the capabilities of the system. Table 2. VR-driven body tracking from sparse end-effector observations. Only the head and both hands are provided as goal constraints, while the remaining full-body motion must be inferred by the controller. Success rate measures whether the visible end-effectors remain within the 0.5 m tracking-error threshold, and MPJPE reports full-body reconstruction error in millimeters on the AMASS train and test splits. Train Test Method Success MPJPE Success MPJPE HetSkills 99.9% 39.9 97.8% 82.3 PULSE 99.5% 57.8 93.4% 88.6 MaskedMimic 98.6% 50.0 98.1% 58.1 ASE 79.8% 103.0 37.6% 120.5 10.2. Text-to-Motion. (a) A person does a backflip. (b) A person does a cartwheel. (c) A person kicks forward. (d) A person walks in a circle. (e) A person dances the waltz. (f) A person crouches and then lies down. (g) A person runs forward quickly. (h) A person dances ballet. (i) A person waves both hands. Figure 11. Qualitative results of text-to-motion generation. Given only natural-language descriptions, ℱt2m F^t2m generates diverse full-body motions from a neutral-pose initialization, including acrobatic movements, locomotion, dance, crouching, lying down, running, and hand waving. These results show that the text-to-motion skill can produce semantically aligned and physically plausible motions even when the initial state does not match the target motion. We evaluate the text-to-motion skill ℱt2m F^t2m from two complementary aspects. First, we evaluate pose-level robustness under two initialization protocols: starting from the ground-truth first frame of each target clip and starting from a neutral pose. A rollout is considered a failure if the mean per-joint position error (MPJPE) exceeds a predefined threshold at any frame (32; 55), and we report success rates at thresholds of 0.30.3 m and 0.50.5 m in Table 3. Second, we evaluate semantic alignment following the HumanML3D retrieval protocol (7), where retrieval scores are computed using a pretrained TMR model (43). Specifically, each generated motion is used as a query to retrieve its corresponding text description from a set of candidate texts in the shared TMR embedding space. We report R-Precision (R@N) and median rank (MedR), where R@N measures whether the correct match is ranked within the top N retrieved results, and MedR denotes the median rank of the correct match. Higher R@N and lower MedR indicate better semantic alignment between the generated motion and the input language. Table 3. Text-to-motion success rates on HumanML3D under two initialization protocols and two MPJPE failure thresholds. The upper block reports the main results, where rollouts start from either the ground-truth first pose or a neutral pose. The lower block ablates MID components, highlighting the importance of Randomized Memory Initialization and showing that the residual shortcut weakens language grounding. threshold: 0.3 m threshold: 0.5 m Method First frame Neutral First frame Neutral MaskedMimic 76.1% 19.8% 83.5% 34.0% CLoSD 30.9% 27.1% 44.2% 38.5% HetSkills 92.2% 60.2% 96.9% 81.7% w/o RMI 92.6% 48.5% 97.7% 70.0% w/ residual 0% 0% 0% 0% HetSkills outperforms both MaskedMimic (55) and CLoSD (58) across both initialization protocols in Table 3. At the 0.50.5 m threshold, HetSkills achieves 96.9%96.9\% success with the first-frame initialization and 81.7%81.7\% with the neutral-pose initialization, significantly surpassing all baselines. This advantage is especially noticeable under neutral-pose initialization, where other methods suffer from a sharp decline. This robustness is a direct result of MID, which removes the reliance on matched initial states and future context, forcing the model to ground motion generation solely in language semantics and current observations. The ablation results in Table 3 further support this design: removing RMI causes a clear performance drop under neutral-pose initialization, while adding a residual branch alongside the text encoder leads to failure under both protocols due to the shortcut issue described in Section 6.1. These results indicate that RMI is critical for preventing overfitting to matched motion histories and for maintaining effective language conditioning. In contrast to diffusion-based methods, the next-token prediction approach of HetSkills allows real-time closed-loop corrections, preventing positional errors from accumulating across motion segments. Qualitative results across diverse action categories are shown in Fig. 11, with animated demonstrations provided in the supplementary video. The retrieval results in Table 4 further show that HetSkills preserves strong text-motion alignment under the neutral-pose initialization protocol. Compared with the baselines, HetSkills achieves higher R@N scores and a lower MedR, indicating that the generated motions are more consistently retrieved as their corresponding language descriptions in the TMR embedding space. This result complements the MPJPE-based success rates: while MPJPE measures whether the generated motion follows the target pose sequence, R@N and MedR evaluate whether the motion remains semantically recognizable as the intended textual action. Together, these results suggest that HetSkills improves not only initialization robustness, but also language-motion consistency. Table 4. Text-to-motion retrieval performance under the HumanML3D evaluation protocol (7) with neutral-pose initialization. Retrieval metrics are computed using the pretrained TMR model. R@N measures the fraction of correct text-motion matches ranked within the top N retrieved results, while MedR denotes the median rank of the correct match. Higher R@N and lower MedR indicate better text-motion alignment. Method R@1↑ R@2↑ R@3↑ R@5↑ MedR↓ Ground Truth 71% 86% 91% 96% 1.00 HetSkills 65% 82% 88% 93% 1.03 CLoSD 41% 58% 67% 77% 2.08 MaskedMimic 38% 54% 62% 72% 2.41 Figure 12. MoE gating dynamics during the pick-and-place task. The rendered sequence at the top shows rollout frames sampled every 2020 frames across the pick-up, carry-to, and put-down stages. The heatmap below shows how the Arm and Body branches assign time-varying weights to different language-conditioned priors. The changing weights indicate that the task-guidance module dynamically selects and blends different semantic priors as the task stage evolves; the full language prompts are listed in Table 5. Figure 13. Downstream training curves on path follow, strike push, and pick-up. Episode rewards are plotted against environment samples and averaged over three random seeds. Compared with Scratch, ASE, and PULSE, HetSkills converges stably to high task rewards across all tasks, while preserving natural motion through the frozen language-conditioned prior. 10.3. Downstream Task Adaptation HetSkills adopts the language-conditioned prior to restrict downstream exploration to a semantically meaningful distribution of natural human motions. This design improves motion naturalness and makes task-relevant behaviors easier to discover, especially when the desired behavior occupies only a small region of the latent space. We evaluate these properties through task performance across three downstream tasks. In particular, the pick-and-place task serves as a representative compositional setting, where multiple language-conditioned skill priors are routed and combined to accomplish a single object-interaction objective; we further analyze the MoE gating behavior to show how the model dynamically selects and composes these priors during execution. Comparison Across Tasks. The training curves in Fig. 13 show that HetSkills converges stably across all three downstream tasks. On path follow, all methods achieve comparable task rewards, but ASE and PULSE occasionally generate backward-walking gaits because their priors do not impose semantic constraints on locomotion direction. On strike, all methods can knock over the target, while the baselines often produce awkward postures due to insufficient constraints on the action distribution. In contrast, HetSkills generates more natural kicking and pushing motions. The largest difference appears on pick-up, where only HetSkills converges reliably. This task requires coordinated bimanual lifting, which occupies a small and sparse region of the latent space. Without semantic guidance, the policy struggles to find this region within the limited sample budget and can fall into local optima. These results highlight two main advantages of HetSkills for downstream adaptation. First, the semantic constraint from the language-conditioned prior helps the policy preserve natural human-like motion while optimizing task rewards. More importantly, it narrows exploration toward task-relevant regions of the latent space, making sparse and coordinated behaviors easier to discover. Second, adapting to a new task only requires changing the language instructions, without task-specific motion data or prior retraining. This provides a simple and flexible interface for reusing the learned skill space across different downstream objectives. Additional rollout examples are included in the supplementary video. Compositional Task Guidance. Fig. 12 visualizes the gating weights assigned to different language instructions during the pick-and-place task. The visualization shows how the MoE module changes the active semantic prior across the pick-up, carry-to, and put-down stages. We observe three common patterns. Dominance. The gating network assigns most of the weight to one instruction that best matches the current task stage. This indicates that the model can identify the most relevant semantic prior and use it as the main behavior source. Periodicity. The weight of the dominant instruction changes periodically over time. This pattern reflects the intrinsic rhythm of the corresponding motion prior, such as the gait cycle in walking or the preparation, execution, and recovery phases in grasping. Complementarity. When the dominant instruction enters a low-activity phase, the gating network shifts part of the weight to a semantically related instruction. This compensation helps fill behavioral gaps and maintain smooth transitions over long-horizon execution. Overall, the MoE module provides an adaptive mechanism for semantic prior selection rather than assigning a fixed instruction to each task stage. It selects the most relevant prior according to the current task context, aligns the generated behavior with the corresponding temporal structure, and integrates complementary instructions when necessary. This mechanism enables a compact set of language priors to support coherent and coordinated long-horizon downstream behaviors. 10.4. Long-Horizon Skill Composition Figure 14. Long-horizon composition of heterogeneous skills. A high-level task is decomposed into a sequence of task units, including pick-up, carry-to, put-down, path follow, human-scene interaction, and text-to-motion, with each unit executed by its corresponding skill module within the same unified latent space. This demonstrates that skills learned from different data sources, supervision forms, and task objectives can be composed seamlessly without modifying the shared controller or manually designing low-level transition rules. Prompt: Long-Horizon Skill Composition Please follow the Standardized Task Description to generate an executable long-horizon skill program for a SMPL character. The task depicts a humanoid that retrieves a box from a source table, transports it to a nearby target platform, moves toward a chair, interacts with the scene, and finally performs an open-ended text-conditioned motion. The detailed skill sequence is specified below. Pick-up Skill. Initialization: Spawn the humanoid next to the source table with the box. Condition: Provide the box position. Terminate: End when the humanoid stably lifts the box for several consecutive frames. Carry-to Skill. Initialization: Use the final pick-up state as the initial state. Condition: Generate and follow a path from the current position to the front of the target platform. Terminate: End when the humanoid reaches the neighborhood of the target platform. Put-down Skill. Initialization: Use the final carry-to state as the initial state. Condition: Provide the target platform position to the put-down policy. Terminate: End when the box is successfully placed for several consecutive frames. Path Follow Skill. Initialization: Start from the final state of the put-down Skill. Condition: Generate and follow a walking path toward a target location in front of the chair. Terminate: End when the humanoid reaches the target location in front of the chair. Human-Scene Interaction Skill. Initialization: Start from the final state of the path follow skill. Condition: Specify a scheduled interaction sequence, including turning toward the chair, sitting down, and standing up, and assign a duration to each target pose. Terminate: End when the interaction sequence is completed. Text-to-Motion Skill. Initialization: Start from the final state of the Human-scene Interaction Skill. Condition: Set both prompts to “a person doing cartwheel”. Terminate: Continue until the episode is externally stopped. Figure 15. Example LLM prompt for long-horizon skill composition. To further demonstrate the compositional capability of HetSkills, we construct a long-horizon demonstration that sequentially combines multiple heterogeneous skills into one continuous behavior. The demonstration includes object manipulation, goal-directed locomotion, human-scene interaction, and text-conditioned motion generation, where the character picks up a box, places it on a target platform, walks toward a chair, sits down, and then rises from the chair to perform an open-ended text-conditioned motion. The long-horizon behavior is generated through the standardized task interface introduced in Section 8.1. As shown in the prompt example in Fig. 15, the user only needs to describe the task at the skill level, where each stage is represented by its initialization, condition, and termination criterion. The large language model (LLM) (4; 50) then organizes the high-level instruction into a structured sequence of executable task units, making it easier to compose different skills without manually designing low-level transition logic. Importantly, the LLM is not used to generate motion trajectories directly. Instead, it serves as a convenient planning interface (64; 71; 67) that translates a natural task description into task units executed by the corresponding skill modules. This design makes long-horizon composition more flexible and easier to specify. Since all skills share the same latent control space and standardized task format, skills learned from different supervision forms and task settings can be combined within a single framework. New behaviors can be expressed by changing the task description, target conditions, or skill ordering, without manually designing detailed controller-switching logic. As shown in the long-horizon demonstration in Fig. 14, the generated behavior remains coherent across different stages, even though the underlying skills involve different objectives and conditioning modalities. This suggests that HetSkills provides a practical interface for organizing heterogeneous skills into reusable and extensible long-horizon behavior programs. To further illustrate the compositional flexibility of HetSkills, we provide additional examples with different skill combinations and task sequences in the appendix E. 11. Limitations and Future Work Our experiments reveal two main limitations. First, although HetSkills demonstrates progressive integration across several representative skill categories, the current experiments do not yet fully cover larger-scale skill accumulation, more complex long-horizon composition, or more open-ended human-object and human-scene interactions. Second, text-driven motion generation still struggles with high-difficulty actions and ambiguous language descriptions. MID improves robustness to different initial states and motion histories. However, the text-to-motion skill remains affected by the quality of language-motion annotations. In particular, semantically similar descriptions may correspond to substantially different motions, which makes fine-grained language-motion alignment more difficult. Future work includes improving HetSkills along three directions. First, incorporating more diverse contact-rich and scene-aware motion data during the tracking stage could help the unified latent space better cover interaction behaviors. Second, contact-aware modules and higher-quality text-motion annotations could improve interaction stability and text-driven generation fidelity. Finally, extending HetSkills to a broader range of skills and real-world robot control is a promising direction. Another interesting direction is to further explore the role of large language models in high-level planning. In this work, an LLM is used to help organize high-level instructions into task sequences, illustrating the potential of combining standardized task interfaces with language-based planning. Future work could further investigate more robust task decomposition, automatic skill selection, and failure recovery for more open-ended long-horizon tasks. 12. Conclusion We presented HetSkills, a physics-based character control framework that progressively learns heterogeneous skills within a unified latent space. The key idea is to treat the latent space as a shared executable interface, allowing skills from different data sources, supervision forms, and training stages to extend the same control substrate rather than requiring dedicated controllers. HetSkills integrates motion tracking, text-to-motion generation, motion completion, and downstream task adaptation, while supporting natural language as a flexible interface for both motion generation and task guidance. With part-wise latent control, motion intuition distillation, and language-guided latent composition, the framework preserves natural motion quality, enables robust skill reuse, and adapts to new tasks without task-specific demonstrations, motion-prior retraining, or shared-controller modification. Experiments show that HetSkills can effectively accommodate diverse skills and task objectives, suggesting that a unified latent space is a practical foundation for scalable, reusable, and progressively extensible character control. References Bae et al. (2025) J. Bae, Y. Lee, D. Lim, and Y. M. Kim PLT: part-wise latent tokens as adaptable motion priors for physically simulated characters. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–10. Cited by: §1, §2, §5.2. Bae et al. (2023) J. Bae, J. Won, D. Lim, C. Min, and Y. M. Kim PMP: learning to physically interact with environments using part-wise motion priors. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–10. Cited by: §1, §2. Burgess et al. (2018) C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner Understanding disentangling in β-vae. arXiv preprint arXiv:1804.03599. Cited by: §5.2. Chang et al. (2024) Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15 (3), p. 1–45. Cited by: §10.4. Cohan et al. (2024) S. Cohan, G. Tevet, D. Reda, X. B. Peng, and M. van de Panne Flexible motion in-betweening with diffusion models. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–9. Cited by: §7. Dou et al. (2023) Z. Dou, X. Chen, Q. Fan, T. Komura, and W. Wang C·ase: learning conditional adversarial skill embeddings for physics-based characters. In ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), p. 1–11. Cited by: §2. Guo et al. (2022) C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng Generating diverse and natural 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5152–5161. Cited by: §10.2, Table 4, Table 4, §9.1, §9.3. Harvey et al. (2020) F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal Robust motion in-betweening. ACM Transactions on Graphics (TOG) 39 (4), p. 60–1. Cited by: §7. Hassan et al. (2021a) M. Hassan, D. Ceylan, R. Villegas, J. Saito, J. Yang, Y. Zhou, and M. J. Black Stochastic scene-aware motion prediction. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 11374–11384. Cited by: §7.1, §9.1. Hassan et al. (2021b) M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black Populating 3d scenes by learning human-scene interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14708–14718. Cited by: §7.1. Hassan et al. (2023) M. Hassan, Y. Guo, T. Wang, M. Black, S. Fidler, and X. B. Peng Synthesizing physical character-scene interactions. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–9. Cited by: §2, §2. Holden et al. (2017) D. Holden, T. Komura, and J. Saito Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG) 36 (4), p. 1–13. Cited by: §5.2. Huang et al. (2025a) X. Huang, T. Truong, Y. Zhang, F. Yu, J. P. Sleiman, J. Hodgins, K. Sreenath, and F. Farshidian Diffuse-cloc: guided diffusion for physics-based character look-ahead control. ACM Transactions on Graphics (TOG) 44 (4), p. 1–12. Cited by: §2. Huang et al. (2025b) Y. Huang, Z. Dou, and L. Liu ModSkill: physical character skill modularization. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 12394–12404. Cited by: §2, §5.2. Hwang et al. (2025a) I. Hwang, J. Bae, D. Lim, and Y. M. Kim Motion synthesis with sparse and flexible keyjoint control. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 13203–13213. Cited by: §7. Hwang et al. (2025b) I. Hwang, B. Zhou, Y. M. Kim, J. Wang, and C. Guo SceneMI: motion in-betweening for modeling human-scene interaction. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 6034–6045. Cited by: §7.1. Jang et al. (2022) D. Jang, S. Park, and S. Lee Motion puzzle: arbitrary motion style transfer by body part. ACM Transactions on Graphics (TOG) 41 (3), p. 1–16. Cited by: §2. Juravsky et al. (2022) J. Juravsky, Y. Guo, S. Fidler, and X. B. Peng PADL: language-directed physics-based character control. In ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), p. 1–9. Cited by: §2. Juravsky et al. (2024) J. Juravsky, Y. Guo, S. Fidler, and X. B. Peng SuperPADL: scaling language-directed physics-based control with progressive supervised distillation. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–11. Cited by: §1, §2, §9.1. Kaufmann et al. (2020) M. Kaufmann, E. Aksan, J. Song, F. Pece, R. Ziegler, and O. Hilliges Convolutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), p. 918–927. Cited by: §7.1. Khoshsiyar et al. (2024) N. Khoshsiyar, R. Gou, T. Zhou, S. Andrews, and M. van de Panne PartwiseMPC: interactive control of contact-guided motions. In Computer Graphics Forum (CGF), Vol. 43, p. e15174. Cited by: §2. Kim et al. (2022) J. Kim, T. Byun, S. Shin, J. Won, and S. Choi Conditional motion in-betweening. Pattern Recognition 132, p. 108894. Cited by: §7.1. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: Appendix A. Kingma and Welling (2013) D. P. Kingma and M. Welling Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §5.2. Ling et al. (2020) H. Y. Ling, F. Zinno, G. Cheng, and M. Van De Panne Character controllers using motion vaes. ACM Transactions on Graphics (TOG) 39 (4), p. 40–1. Cited by: §2, §5.2. Liu and Hodgins (2017) L. Liu and J. Hodgins Learning to schedule control fragments for physics-based characters ssing deep q-learning. ACM Transactions on Graphics (TOG) 36 (3), p. 1–14. Cited by: §2. Liu et al. (2022a) M. Liu, M. Zhu, and W. Zhang Goal-conditioned reinforcement learning: problems and solutions. arXiv preprint arXiv:2201.08299. Cited by: §3. Liu et al. (2022b) S. Liu, G. Lever, Z. Wang, J. Merel, S. A. Eslami, D. Hennes, W. M. Czarnecki, Y. Tassa, S. Omidshafiei, A. Abdolmaleki, et al. From motor control to team play in simulated humanoid football. Science Robotics 7 (69), p. eabo0235. Cited by: §5.2. Loper et al. (2015) M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black SMPL: a skinned multi-person linear model. ACM Transactions On Graphics (TOG) 34 (6), p. 248:1–248:16. Cited by: §5.1. Luo et al. (2023a) Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 10895–10904. Cited by: §2, §5.1, §9.1. Luo et al. (2023b) Z. Luo, J. Cao, J. Merel, A. Winkler, J. Huang, K. Kitani, and W. Xu Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582. Cited by: §1, §10.1, §2, §2, §5.3, §9.3. Luo et al. (2021) Z. Luo, R. Hachiuma, Y. Yuan, and K. Kitani Dynamics-regulated kinematic policy for egocentric pose estimation. Conference on Neural Information Processing Systems (NeurIPS) 34, p. 25019–25032. Cited by: §10.2. Luo et al. (2022) Z. Luo, Y. Yuan, and K. M. Kitani From universal humanoid control to automatic physically valid character creation. arXiv preprint arXiv:2206.09286. Cited by: §5.1. Mahmood et al. (2019) N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black AMASS: archive of motion capture as surface shapes. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 5442–5451. Cited by: §10.1, §9.1. Merel et al. (2018) J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv:1811.11711. Cited by: §2, §5.2. Mittal et al. (2025) M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Munoz, X. Yao, R. Zurbrüg, N. Rudin, et al. Isaac lab: a gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831. Cited by: §9. Mu et al. (2025) Y. Mu, Z. Zhang, Y. Shi, M. Matsumoto, K. Imamura, G. Tevet, C. Guo, M. Taylor, C. Shu, P. Xi, et al. SMP: reusable score-matching motion priors for physics-based character control. arXiv preprint arXiv:2512.03028. Cited by: §1, §2, §2. Oreshkin et al. (2023) B. N. Oreshkin, A. Valkanas, F. G. Harvey, L. Ménard, F. Bocquelet, and M. J. Coates Motion in-betweening via deep Δ -interpolator. IEEE Transactions on Visualization and Computer Graphics (TVCG) 30 (8), p. 5693–5704. Cited by: §7. Pan et al. (2025) L. Pan, Z. Yang, Z. Dou, W. Wang, B. Huang, B. Dai, T. Komura, and J. Wang TokenHSI: unified synthesis of physical human-scene interactions through task tokenization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5379–5391. Cited by: §2. Peng et al. (2018) X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne DeepMimic: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37 (4), p. 1–14. Cited by: §2, §5.1, §5.3, §6.1. Peng et al. (2022) X. B. Peng, Y. Guo, L. Halper, S. Levine, and S. Fidler ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG) 41 (4), p. 1–17. Cited by: §1, §2, §2. Peng et al. (2021) X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa AMP: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG) 40 (4), p. 1–20. Cited by: §2. Petrovich et al. (2023) M. Petrovich, M. J. Black, and G. Varol TMR: text-to-motion retrieval using contrastive 3d human motion synthesis. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 9488–9497. Cited by: §10.2, §6.2, §9.3. Plappert et al. (2017) M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz Parameter space noise for exploration. arXiv preprint arXiv:1706.01905. Cited by: §5.2. Qin et al. (2022) J. Qin, Y. Zheng, and K. Zhou Motion in-betweening via two-stage transformers.. ACM Transactions On Graphics (TOG) 41 (6), p. 184–1. Cited by: §7. Ren et al. (2023) J. Ren, M. Zhang, C. Yu, X. Ma, L. Pan, and Z. Liu InsActor: instruction-driven bhysics-based characters. Conference on Neural Information Processing Systems (NeurIPS) 36, p. 59911–59923. Cited by: §2. Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), p. 627–635. Cited by: §6.3. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §5.3, §8.3. Serifi et al. (2024) A. Serifi, R. Grandia, E. Knoop, M. Gross, and M. Bächer Robot motion diffusion model: motion generation for robotic characters. In ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), p. 1–9. Cited by: §2. Singh et al. (2025) A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. OpenAI gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: §10.4. Starke et al. (2023) P. Starke, S. Starke, T. Komura, and F. Steinicke Motion in-betweening with phase manifolds. Proceedings of the ACM on Computer Graphics and Interactive Techniques 6 (3), p. 1–17. Cited by: §7.1. Starke et al. (2019) S. Starke, H. Zhang, T. Komura, and J. Saito Neural state machine for character-scene interactions. ACM Transactions On Graphics (TOG) 38 (6), p. 178. Cited by: §7.1. Sutton et al. (1998) R. S. Sutton A. G. Barto et al. Reinforcement learning: an introduction. Vol. 1, MIT press Cambridge. Cited by: §3. Tan et al. (2011) J. Tan, K. Liu, and G. Turk Stable proportional-derivative controllers. IEEE Computer Graphics and Applications 31 (4), p. 34–44. Cited by: §3, §5.1. Tessler et al. (2024) C. Tessler, Y. Guo, O. Nabati, G. Chechik, and X. B. Peng MaskedMimic: unified physics-based character control through masked motion inpainting. ACM Transactions On Graphics (TOG) 43 (6), p. 1–21. Cited by: §1, §10.1, §10.2, §10.2, §2, §5.1, §5.2, §9.3. Tessler et al. (2025) C. Tessler, Y. Jiang, X. B. Peng, E. Coumans, Y. Shi, H. Zhang, D. Rempe, G. Chechik, and S. Fidler ProtoMotions3: an open-source framework for humanoid simulation and control. GitHub. Note: https://github.com/NVLabs/ProtoMotions/ Cited by: §9. Tessler et al. (2023) C. Tessler, Y. Kasten, Y. Guo, S. Mannor, G. Chechik, and X. B. Peng CALM: conditional adversarial latent models for directable virtual characters. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–9. Cited by: §1, §2, §2. Tevet et al. (2024) G. Tevet, S. Raab, S. Cohan, D. Reda, Z. Luo, X. B. Peng, A. H. Bermano, and M. van de Panne CLoSD: closing the loop between simulation and diffusion for multi-task character control. arXiv preprint arXiv:2410.03441. Cited by: §10.2, §2. Truong et al. (2024) T. E. Truong, M. Piseno, Z. Xie, and K. Liu PDP: physics-based character animation via diffusion policy. In ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), p. 1–10. Cited by: §2. Vainshtein et al. (2025) R. Vainshtein, Z. Rimon, S. Mannor, and C. Tessler Task tokens: a flexible approach to adapting behavior foundation models. arXiv preprint arXiv:2503.22886. Cited by: §2. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Conference on Neural Information Processing Systems (NeurIPS) 30. Cited by: §6.2. Wan et al. (2024) W. Wan, Z. Dou, T. Komura, W. Wang, D. Jayaraman, and L. Liu TLControl: trajectory and language control for human motion synthesis. In European Conference on Computer Vision (ECCV), p. 37–54. Cited by: §2. Wang et al. (2010) J. M. Wang, D. J. Fleet, and A. Hertzmann Optimizing walking controllers. ACM Transactions on Graphics (TOG) 29 (4), p. 1–8. Cited by: §2. Wang et al. (2025) W. Wang, L. Pan, Z. Dou, J. Mei, Z. Liao, Y. Lou, Y. Wu, L. Yang, J. Wang, and T. Komura SIMS: simulating stylized human-scene interactions with retrieval-augmented script generation. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 14117–14127. Cited by: §10.4. Won et al. (2020) J. Won, D. Gopinath, and J. Hodgins A scalable approach to control diverse behaviors for physically simulated characters. ACM Transactions on Graphics (TOG) 39 (4), p. 33–1. Cited by: §2. Wu et al. (2025a) Y. Wu, K. Karunratanakul, Z. Luo, and S. Tang UniPhys: unified planner and controller with diffusion for flexible physics-based character control. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 13214–13224. Cited by: §1, §2. Wu et al. (2025b) Z. Wu, J. Li, P. Xu, and C. K. Liu Human-object interaction from human-level instructions. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 11176–11186. Cited by: §10.4. Xu et al. (2025) M. Xu, Y. Shi, K. Yin, and X. B. Peng PARC: physics-based augmentation with reinforcement learning for character controllers. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–11. Cited by: §5.1. Xu et al. (2023) P. Xu, X. Shang, V. Zordan, and I. Karamouzas Composite motion learning with task control. ACM Transactions on Graphics (TOG) 42 (4), p. 1–16. Cited by: §2. Yao et al. (2022) H. Yao, Z. Song, B. Chen, and L. Liu ControlVAE: model-based learning of generative controllers for physics-based characters. ACM Transactions on Graphics (TOG) 41 (6), p. 1–16. Cited by: §2. Yao et al. (2024) H. Yao, Z. Song, Y. Zhou, T. Ao, B. Chen, and L. Liu MoConVQ: unified physics-based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG) 43 (4), p. 1–21. Cited by: §10.4. Yin et al. (2007) K. Yin, K. Loken, and M. Van de Panne SIMBICON: simple biped locomotion control. ACM Transactions on Graphics (TOG) 26 (3), p. 105–es. Cited by: §2. Yu et al. (2025) R. Yu, Y. Wang, Q. Zhao, H. W. Tsui, J. Wang, P. Tan, and Q. Chen SkillMimic-v2: learning robust and generalizable interaction skills from sparse and noisy demonstrations. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 1–11. Cited by: §5.1. Zhang et al. (2018) H. Zhang, S. Starke, T. Komura, and J. Saito Mode-adaptive neural networks for quadruped motion control. ACM Transactions on Graphics (TOG) 37 (4), p. 1–11. Cited by: §5.2. Zhang et al. (2025) Z. Zhang, S. Bashkirov, D. Yang, Y. Shi, M. Taylor, and X. B. Peng Physics-based motion imitation with adversarial differential discriminators. In ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), p. 1–12. Cited by: §2. Zhou et al. (2019) Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li On the continuity of rotation representations in neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 5745–5753. Cited by: §5.1. Zhu et al. (2023) Q. Zhu, H. Zhang, M. Lan, and L. Han Neural categorical priors for physics-based character control. ACM Transactions on Graphics (TOG) 42 (6), p. 1–16. Cited by: §1, §2, §2. Appendix A Implementation Details Unless otherwise specified, all trainable policy and prior modules are optimized with Adam (23). We use a learning rate of 2×10−52× 10^-5 for policy and prior networks, and a learning rate of 10−410^-4 for critic networks. The arm and body latent dimensions are both set to 6464. Transformer-based prior modules embed all input tokens into 256256-dimensional features. The default latent regularization coefficients are λmr=1×10−3 _mr=1× 10^-3, λar=5×10−3 _ar=5× 10^-3, and ϕ=0.99φ=0.99. Tracking skill ℱtrc F^trc. We train the part-wise tracking policy with PPO. The shared state encoder outputs a 512512-dimensional feature. Exploration is applied only in the final action space using a fixed diagonal Gaussian distribution with log standard deviation −2.9-2.9. We use a tracking-error termination threshold of 0.5m0.5\,m. Reference State Initialization (RSI) is applied with probability 0.80.8; otherwise, the episode starts from the first frame of the reference motion. The tracking reward includes global joint position, global joint rotation, joint velocity, joint angular velocity, and root-height terms, with weights wgp=0.5w_gp=0.5, wgr=0.3w_gr=0.3, wjv=0.1w_jv=0.1, wjav=0.1w_jav=0.1, and wrh=0.2w_rh=0.2, and exponential coefficients kgp=100k_gp=100, kgr=5k_gr=5, kjv=0.5k_jv=0.5, kjav=0.1k_jav=0.1, and krh=100k_rh=100. We further use a contact mismatch penalty, an action smoothness penalty, and an energy penalty with weights wct=−0.1w_ct=-0.1, wsm=−0.02w_sm=-0.02, and weg=−10−5w_eg=-10^-5, respectively. The energy penalty is clipped from below at −0.5-0.5, and the reference contact signal is smoothed with a 77-frame temporal window. Text-to-motion skill ℱt2m F^t2m. The text-to-motion prior uses two part-wise Transformer branches, one for the arm latent and one for the body latent. The input tokens consist of one current-state token, one language token, and 66 history tokens uniformly sampled from a 6060-step history window. The language token is obtained from the frozen text encoder used for language-motion alignment. During training, we apply RSI with probability prsi=0.7p_rsi=0.7 and Randomized Memory Initialization (RMI) with probability prmi=0.2p_rmi=0.2. This initialization scheme encourages the policy to rely on language semantics and current observations rather than matched initial states or memorized motion histories. Motion completion skill ℱmoc F^moc. We instantiate the motion completion skill in three sparse-observation settings: (1) VR tracking. The sparse target contains one future step of head and both hands states. The prior is a four-layer MLP with 10241024 hidden units, and outputs a 128128-dimensional latent vector split into za,zb∈ℝ64z_a,z_b ^64; (2) Motion in-betweening. The prior uses two part-wise Transformer branches. The input consists of the current state, a future target keyframe pose, its time offset, and 33 historical poses uniformly sampled from a 3030-step history window. The target offset is uniformly sampled from 55 to 3030 future frames; (3) Human-scene interaction. We train the human-scene interaction prior with PPO. The input consists of the current state, a future target keyframe pose, its time offset, and 33 historical poses uniformly sampled from a 3030-step history window. The target offset is uniformly sampled from 55 to 3030 future frames. We use 66 PPO mini-epochs per update. The reward uses the same global joint position, global joint rotation, joint velocity, joint angular velocity, action smoothness, and energy terms as the tracking skill, but removes the root-height and contact mismatch terms. Downstream task adaptation ℱtsk∗ F^tsk^*. For downstream adaptation, the pretrained motion prior and low-level decoder are kept frozen, and only the task-guidance modules are optimized. The arm and body residual latent dimensions are both set to 6464. We use rollouts of length 3232 and 44 PPO mini-epochs per update. The latent policy log standard deviation is annealed from −2.5-2.5 to −3.0-3.0 between epochs 500500 and 10001000. Adapted latents outside the interval [−0.1,0.1][-0.1,0.1] are penalized with coefficient 10.010.0, and the latent smoothness penalty coefficient is set to 0.10.1. Appendix B Training Time and Compute The progressive design of HetSkills allows different skills to be trained independently on top of the shared latent space, instead of jointly optimizing all skills in a single monolithic model. This design keeps each training stage relatively lightweight. After the tracking skill learns the shared motion decoder, later stages only need to learn skill-specific latent mappings or task-guidance modules while reusing the frozen low-level controller. In our implementation, the tracking skill is trained on two RTX 5090 GPUs for approximately 22 days, and the text-to-motion skill is trained on two RTX 5090 GPUs for approximately 55 days. For motion completion, the three instantiations are trained independently on a single RTX 5090 GPU. Human-scene interaction takes approximately 22 days, VR tracking takes approximately 11 day, and motion in-betweening takes approximately 22 days. For downstream task adaptation, each body part may be conditioned on multiple language instructions. These instruction branches are evaluated in parallel by batching them together, so increasing the number of instructions does not lead to a proportional increase in wall-clock training time. In our experiments, each downstream task can be trained within 55 hours on a single RTX 5090 GPU. Appendix C Downstream Task Rewards and Settings This section describes the reward functions and environment settings used for downstream task adaptation. All rewards are designed to encourage task completion while regularizing the motion with energy penalties, so that the adapted behaviors remain physically plausible and compatible with the pretrained motion prior. Path Follow. The path follow reward encourages the character root to track an online-generated target path in the horizontal plane: (28) r=exp⁡(−‖ptargetxy−prootxy‖2)−λ⋅legr= \! (-\|p_target^xy-p_root^xy\|^2 )-λ·P_leg where ptargetxyp_target^xy and prootxyp_root^xy denote the horizontal positions of the current path target and the character root, respectively. The leg energy term is defined as (29) leg=∑i|τiq˙i|,P_leg= _i| _i q_i|, where the summation is taken over the 2424 leg DoFs, including the bilateral hip, knee, ankle, and toe joints. We set λ=10−5λ=10^-5 and clip the resulting energy penalty to [−0.1,0][-0.1,0]. The path is generated online with a maximum speed of 5m/s5\,m/s and a maximum acceleration of 2m/s22\,m/s^2. An episode terminates if the horizontal distance between the root and the current path target exceeds 1.5m1.5\,m. Strike. The strike reward encourages the character to approach the target, face it, move toward it, and apply a motion that tilts the target: (30) r=0.6rrot+0.2rvel+0.1rprog+0.1rtoward−λ.r=0.6\,r_rot+0.2\,r_vel+0.1\,r_prog+0.1\,r_toward- . The reward components are defined as (31) rrot r_rot =max(1−⊤Rtarget, 0), = (1-u R_targetu,\ 0), rvel r_vel =exp⁡(−4(2.5−valong)2), = \! (-4(2.5-v_along)^2 ), rprog r_prog =clamp(D0−DtD0+ϵ, 0, 1), =clamp\! ( D_0-D_tD_0+ε,\ 0,\ 1 ), rtoward r_toward =clamp⁡(face⋅target, 0, 1). =clamp\! (d_face·d_target,\ 0,\ 1 ). Here, =[0,0,1]⊤u=[0,0,1] is the world up vector, RtargetR_target is the target orientation, and valongv_along is the root velocity projected onto the horizontal direction toward the target. D0D_0 and DtD_t denote the initial and current horizontal distances to the target, respectively. faced_face is the character heading direction, and targetd_target is the direction from the character to the target in the horizontal plane. Once the target tilt exceeds approximately 78∘78 , the strike reward is set to 11. The energy term is defined as =∑i|τiq˙i|P= _i| _i q_i| over all joints. We set λ=10−5λ=10^-5 and clip the resulting energy penalty to [−0.1,0][-0.1,0]. The target is initialized at a random horizontal distance between 0.5m0.5\,m and 10.0m10.0\,m from the character. Pick-and-Place. The pick-and-place task is decomposed into three stages, including pick-up, carry-to, and put-down. Each stage uses a stage-specific reward while sharing the same energy regularization form. The pick-up reward encourages the character to face the box, grasp it with both hands, apply sufficient contact force, and lift it from the source platform: (32) r=0.4rlift+0.3rgrasp+0.2rforce+0.1rface−λ⋅r=0.4\,r_lift+0.3\,r_grasp+0.2\,r_force+0.1\,r_face-λ·P The reward components are (33) rlift r_lift =clamp(zmin−hsrchtarget−hsrc, 0, 1), =clamp\! ( z_ -h_srch_target-h_src,\ 0,\ 1 ), rgrasp r_grasp =12(e−3dR+e−3dL), = 12 (e^-3d_R+e^-3d_L ), rforce r_force =clamp(min⁡(‖FR‖,‖FL‖)10, 0, 1)⋅[both hands close], =clamp\! ( (\|F_R\|,\|F_L\|)10,\ 0,\ 1 )·1[both hands close], rface r_face =clamp⁡(face⋅box, 0, 1). =clamp\! (d_face·d_box,\ 0,\ 1 ). Here, zminz_ is the lowest corner height of the box, hsrch_src is the source platform surface height, and htarget=0.8mh_target=0.8\,m is the target lifting height. dRd_R and dLd_L are the distances from the right and left hands to their nearest grasp points on opposite sides of the box. FRF_R and FLF_L are the right and left hand contact forces. The indicator [both hands close]1[both hands close] equals 11 only when both hands are within 0.1m0.1\,m of their corresponding grasp points. The energy term is computed over all joints with λ=10−5λ=10^-5, and the resulting energy penalty is clipped to [−0.1,0][-0.1,0]. The source platform is initialized at a random horizontal distance between 0.5m0.5\,m and 1.5m1.5\,m, with a surface height between 0.1m0.1\,m and 0.5m0.5\,m. The carry-to reward maintains the grasp while guiding the character toward the target platform: (34) r=0.4rpath+0.1rface+0.2rgrasp+0.15rforce+0.15rlift′−λ⋅r=0.4\,r_path+0.1\,r_face+0.2\,r_grasp+0.15\,r_force+0.15\,r_lift -λ·P The path follow and lift-maintenance terms are defined as (35) rpath=exp⁡(−2‖ptargetxy−prootxy‖2),rlift′=clamp⁡(zmin1.0, 0, 1).r_path= \! (-2\|p_target^xy-p_root^xy\|^2 ), r_lift =clamp\! ( z_ 1.0,\ 0,\ 1 ). The grasp and force terms are reused from the pick-up reward. We set λ=2×10−5λ=2× 10^-5 and clip the resulting energy penalty to [−0.2,0][-0.2,0]. The put-down reward guides the character to place the box on the target platform and release it after stable placement: (36) r=0.3rprog+0.2rplace+0.2rsucc+0.1rgrasp+0.1rforce+0.1rlift′−λ.r=0.3\,r_prog+0.2\,r_place+0.2\,r_succ+0.1\,r_grasp+0.1\,r_force+0.1\,r_lift - . The progress and placement terms are defined as (37) rprog=clamp⁡(D0−DtD0+ϵ, 0, 1),rplace=exp⁡(−5|zbox−ztarget|).r_prog=clamp\! ( D_0-D_tD_0+ε,\ 0,\ 1 ), r_place= \! (-5|z_box-z_target| ). Here, rprogr_prog rewards horizontal progress toward the target platform, and rplacer_place is activated within 0.5m0.5\,m of the target platform to encourage vertical alignment. The success term rsuccr_succ is a binary reward triggered when the box is stably placed on the target platform and both hands have released it. Upon success, all sub-rewards are frozen at their maximum values. We set λ=2×10−5λ=2× 10^-5 and clip the resulting energy penalty to [−0.2,0][-0.2,0]. Appendix D Language Instruction Sets for Downstream Tasks For downstream task adaptation, we provide fixed language instruction sets for the arm and body branches separately. Each instruction specifies a candidate semantic motion prior in the text-to-motion latent space, such as lifting, carrying, standing, or moving in a particular direction. Given the current task context, the gating network pG_p predicts part-wise mixture weights over these candidate instructions and dynamically blends the corresponding motion priors. This design allows the downstream policy to compose task-relevant behaviors from reusable language-conditioned priors, while keeping the pretrained motion decoder fixed. The instruction sets used for each downstream task are listed in Table 5. Table 5. Language-prior instruction sets for downstream adaptation. The table lists the natural-language prompts used by ℱt2m F^t2m to construct candidate motion priors for each downstream task and body branch. The arm and body prompt sets provide part-specific semantic priors for tasks such as pick-and place, strike, and path follow, while the MoE gating network dynamically blends these priors according to the current task observation. Task Branch Instructions Pick-and-Place Arm “a man lifts something on his left and places it down on his right” (Lift) (all stages) “the person is moving or carrying something” (Carry) Body “a person walks to his right” (Right) “the person was walking forward then turn left” (Left) “a man stands still” (Stand) Strike (kick) Arm & Body “a person does a high kick with their right leg” “this person kicked with their right leg” “a man runs forward fast” Strike (push) Arm & Body “a person pushes something forward with both hands” “a person walks up to an object and shoves it” “a man runs forward fast” Path Follow Arm & Body “a person walks to his right” “the person was walking forward then turn left” “a person runs to the right then runs to the left then back to the middle” Appendix E Long-Horizon Skill Composition Examples More long-horizon skill composition examples are provided to show different combinations and task sequences. Each example follows the same standardized task interface, where the overall behavior is decomposed into a sequence of executable skill units with specified initialization, condition, and termination criterion. The examples involve different combinations and orderings of heterogeneous skills, further illustrating that HetSkills can flexibly organize skills learned from different supervision forms and task settings into coherent long-horizon behaviors. Prompt A: Two-Box Sequential Placement Please follow the Standardized Task Description to generate an executable long-horizon skill program for a SMPL character. The task depicts a humanoid that retrieves a box from the first source table, transports it to a target desk, walks to a second source table, retrieves the second box, and transports it to the same target desk. The detailed skill sequence is specified below. Pick-up Skill (Box 1). Initialization: Spawn the humanoid next to the first source table with the first box. Condition: Provide the box position of the first source table. Terminate: End when the humanoid stably lifts the first box for several consecutive frames. Carry-to Skill (Box 1). Initialization: Use the final pick-up state as the initial state. Condition: Generate and follow a path from the current position toward the target desk while carrying the first box. Terminate: End when the humanoid reaches the neighborhood of the target desk. Put-down Skill (Box 1). Initialization: Use the final carry-to state as the initial state. Condition: Provide the first target placement position on the desk. Terminate: End when the first box is successfully placed for several consecutive frames. Path Follow Skill. Initialization: Start from the final state of the first put-down skill. Condition: Generate and follow a walking path toward the second source table. Terminate: End when the humanoid reaches the vicinity of the second source table. Pick-and-Place Skill (Box 2). Initialization: Start from the final state of the path follow skill, with the second box on the second source table. Condition: Reuse the same pick-up, carry-to, and put-down procedure for Box 2 by providing the second box position, the path from the second source table to the desk, and the second target placement position. Terminate: End when the second box is successfully placed on the desk. Figure 16. Long-horizon skill composition results for the two-box sequential placement task. The humanoid first picks up the first box from the source table and places it on the target desk, then walks to the second source table, picks up the second box, carries it back to the desk, and places it at the target position. Prompt B: From Sitting to Striking and Celebration Please follow the Standardized Task Description to generate an executable long-horizon skill program for a SMPL character. The task depicts a humanoid that walks to a chair, sits down and stands up, moves toward a target object and strikes it down, and then celebrates with an open-ended text-conditioned jumping motion. The detailed skill sequence is specified below. Path Follow Skill. Initialization: Spawn the humanoid at the scene origin. Condition: Generate and follow a straight walking path toward a target location in front of the chair. Terminate: End when the humanoid reaches the target location in front of the chair. Human-Scene Interaction Skill. Initialization: Start from the final state of the path follow skill. Condition: Provide a scheduled inpainting target sequence, including sitting down on the chair, holding the seated pose, and standing back up, with a specified duration for each target frame. Terminate: End when the full interaction schedule is completed. Strike Skill. Initialization: Start from the final state of the human-scene interaction skill. Condition: Provide the target object position, and generate a full-body striking motion that moves the humanoid toward the target object and knocks it down. Terminate: End when the target object remains in a knocked-down orientation. Text-to-Motion Skill. Initialization: Start from the final state of the strike skill. Condition: Set the text prompt to “a person jumping up and down, with their hands above their head.” Terminate: Continue until the episode is externally stopped. Figure 17. Long-horizon skill composition results for the sit-strike-celebrate task. The humanoid walks to the chair, sits down and stands up, approaches the target object, strikes it down, and then performs a jumping celebration motion. Prompt: Text-Conditioned Motion Sequence Please follow the Standardized Task Description to generate an executable long-horizon skill program for a SMPL character. The task depicts a humanoid that sequentially performs three text-conditioned motions, including punching, kicking, and cartwheeling, each for a specified duration. The detailed skill sequence is specified below. Text-to-Motion: Punch. Condition: Set the language embedding to “person was fighting with a left punch”. Terminate: End after 5 seconds. Text-to-Motion: Forward Kick. Condition: Switch the language embedding to “a person kicks forward.” Terminate: End after 3 seconds. Text-to-Motion: Cartwheel. Condition: Switch the language embedding to “a person does a cartwheel.” Terminate: End after 5 seconds. Figure 18. Long-horizon rollout of sequential text-conditioned motion generation. The humanoid switches across three language conditions and performs a left punch, a forward kick, and a cartwheel in sequence.

Instant research discovery