Paper deep dive
Learning Transferable Skills using Goal-Conditioned Bisimulation
Mohammad Amin Abbasfar, Farbod Azimmohseni, Mohammad Hossein Rohban
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/2/2026, 9:29:55 AM
Summary
The paper proposes USB+ATR, a method for learning transferable skills in reinforcement learning by combining an Action-Aware Temporal Representation (ATR) with Unsupervised Skill Discovery using Bisimulation (USB). ATR enforces functional equivariance and preserves local temporal structure, while USB conditions skill policies on dynamically similar state features rather than full states, enabling robust generalization to unseen environment layouts.
Entities (8)
Relation Signals (6)
USB+ATR → uses → ATR
confidence 95% · Our method consists of two components. First, we introduce an objective for training an action-aware temporal representation (ATR)... Second, we build upon this representation to propose unsupervised skill discovery using bisimulation (USB)
USB+ATR → uses → USB
confidence 95% · we further propose unsupervised skill discovery using bisimulation (USB)... Building upon this embedding
ATR → enforces → Functional Equivariance
confidence 92% · ATR... aims to learn a structured state embedding with functional equivariance property
USB → improves → Generalization
confidence 90% · enabling skills to transfer effectively to other configurations... demonstrating strong out-of-distribution generalization
HILP → outperformedby → USB+ATR
confidence 88% · USB+ATR learns effective and transferable skills across different layouts and offers strong out-of-distribution generalization... HILP... struggles to generalize to out-of-distribution layouts
USB+ATR → evaluatedon → OGBench
confidence 85% · We extend the popular OGBench... Through experiments in several maze environments
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free datasets to pretrain general-purpose policies. However, current skill discovery methods either require access to expert data or exhibit limited generalization, failing to transfer effectively to previously unseen layouts. A key challenge is to learn representations that capture the temporal structure of the environment while remaining robust to variations across layouts. To address this issue, we present an objective for learning action-aware temporal representations that satisfy the functional equivariance property while preserving the local temporal structure of the environment. Building upon this embedding, we further propose unsupervised skill discovery using bisimulation, which learns transferable skills by conditioning the behavior of skills exclusively on the subset of state features that directly affect their execution. This enforces invariant behavior across different layouts, enabling skills to transfer effectively to other configurations. Finally, through comprehensive empirical evaluations, we show that skills learned in a given environment can be effectively applied to solve downstream tasks in various environment layouts, demonstrating strong out-of-distribution generalization.
Tags
Links
- Source: https://arxiv.org/abs/2610.00676v1
- Canonical: https://arxiv.org/abs/2610.00676v1
Trouble viewing inline? Open PDF directly →
Full Text
71,487 characters extracted from source content.
Expand or collapse full text
Learning Transferable Skills using Goal-Conditioned Bisimulation Mohammad Amin Abbasfar †thanks: Corresponding author Affiliation: Department of Computer Engineering Farbod Azimmohseni Affiliation: Sharif University of Technology Mohammad Hossein Rohban Affiliation: amin.abbasfar01, far.azimmohseni.82, rohban@sharif.edu Abstract Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free datasets to pretrain general-purpose policies. However, current skill discovery methods either require access to expert data or exhibit limited generalization, failing to transfer effectively to previously unseen layouts. A key challenge is to learn representations that capture the temporal structure of the environment while remaining robust to variations across layouts. To address this issue, we present an objective for learning action-aware temporal representations that satisfy the functional equivariance property while preserving the local temporal structure of the environment. Building upon this embedding, we further propose unsupervised skill discovery using bisimulation, which learns transferable skills by conditioning the behavior of skills exclusively on the subset of state features that directly affect their execution. This enforces invariant behavior across different layouts, enabling skills to transfer effectively to other configurations. Finally, through comprehensive empirical evaluations, we show that skills learned in a given environment can be effectively applied to solve downstream tasks in various environment layouts, demonstrating strong out-of-distribution generalization. 11 1 The code and videos are available at https://sites.google.com/view/usb-atr/. 1 Introduction Leveraging large unlabeled datasets has substantially contributed to the advancement of deep learning in recent years. Unsupervised and self-supervised techniques such as masked language modeling in natural language processing (Devlin et al., 2019) and contrastive losses in computer vision (Chen et al., 2020) have enabled deep neural networks to benefit from vast and low-cost unlabeled data. Numerous attempts have been made to apply the same strategy in the context of reinforcement learning (RL), where the goal is to pretrain the agent’s policy using only pre-collected offline datasets (Chebotar et al., 2021; Reed et al., 2022; Park et al., 2023a; Park et al., 2024a; O. X.-E. Collaboration et al., 2024). Unsupervised skill discovery has been recognized as a promising approach for pretraining such policies. These methods aim to learn a versatile set of latent-conditioned policies, each demonstrating a distinct behavior, such that together they cover the state space. Prior work has proposed various objectives to train such skill policies, either via behavioral cloning (Pertsch et al., 2021; Ajay et al., 2021) or by diversifying their state visitation distribution (Eysenbach et al., 2019; Park et al., 2023b; Park et al., 2024a). These skills can then be used to perform downstream tasks in a zero-shot manner, or harnessed as low-level policies within a hierarchical approach (Eysenbach et al., 2019; Sharma et al., 2020). Many applications of reinforcement learning in locomotion and robotic manipulation involve frequently occurring temporally-extended actions that appear during the accomplishment of a wide range of tasks (Hauskrecht et al., 1998; Sutton et al., 1999; Pateria et al., 2021). For instance, in the AntMaze environment (Fu et al., 2020; Park et al., 2025), where a quadruped agent must navigate to different goals within a maze, the ability to move in the four cardinal directions constitutes a sufficient set of skills for performing any task, regardless of maze layout. Similarly, in the Kitchen environment (Gupta et al., 2019), which consists of several long-horizon manipulation tasks, a temporally-extended action such as placing the kettle from the cupboard onto the oven top should be performed independently of other environmental features, such as whether the microwave door is open or closed. In such structured environments, learning reusable skills is essential for generalizing to previously unseen configurations. Despite the importance of such reusable behaviors in structured domains, existing skill discovery methods are not explicitly designed to account for this structure. For instance, the Hilbert foundation policy (HILP) (Park et al., 2024a), which is a state-of-the-art distance-maximizing skill discovery method, learns skills that span the state space reasonably well. However, it struggles to generalize to out-of-distribution layouts (Sobal et al., 2025). To illustrate, suppose HILP learns to move a kettle only when the microwave door is closed. In a scenario where the microwave door is open, despite being irrelevant to the task, the behavior of the kettle-moving skill may change drastically, making it incapable of completing the task. In contrast to distance-maximizing skill discovery methods, behavioral cloning-based methods offer better out-of-distribution generalization (Pertsch et al., 2021). Nevertheless, they fall short when using suboptimal datasets, as they tend to replicate the exact action sequences that exist in the dataset. (a) Skills discovered by HILP (b) Skills discovered by USB+ATR (ours) Figure 1: (Left) HILP’s skills overfit to the training layout despite spanning the state space, limiting their ability to generalize to new layouts. (Right) USB+ATR extracts more fine-grained and layout-agnostic skills, which transfer effectively to unseen layouts.33 3 This figure is adapted from Park et al. (2024a). Motivated by this, we aim to extract skills that are transferable to out-of-distribution layouts, even when only a single training layout and low-quality datasets are available during training. Our method consists of two components. First, we introduce an objective for training an action-aware temporal representation (ATR) that aims to learn a structured state embedding with functional equivariance (Hansen-Estruch et al., 2022) property, while preserving local temporal distances. This allows us to enforce each skill to show monotonic behavior while retaining the benefits of temporal representations when learning from suboptimal data. To train such representations, we employ bisimulation metrics (Ferns and Precup, 2014), which have demonstrated well-established capabilities for improving generalization (Zhang et al., 2021; Agarwal et al., 2021). In particular, we use goal-conditioned bisimulation (Hansen-Estruch et al., 2022) to enforce this property for state-goal pairs that share similar underlying action sequences. Second, we build upon this representation to propose unsupervised skill discovery using bisimulation (USB), a modification of HILP in which the behavior of skills is conditioned only on the state features that are relevant to their execution, rather than on the full state. To achieve this, we again employ bisimulation metrics to learn a state representation in which states with similar transition dynamics are embedded close to one another. Conditioning each skill on this representation encourages the learned skills to exhibit consistent behavior across dynamically similar states, even when the corresponding action sequences are not present in the dataset for all such states. In this way, we encourage skills to disregard layout-specific features of the environment and to acquire generic behaviors that improve their transferability. These generic skills can then be equipped with a planning approach or composed by a high-level controller to accomplish downstream tasks across diverse layouts. Finally, to evaluate the out-of-distribution generalization capability of our method, we assess the learned skills in both a test-time planning setup and a hierarchical approach. By conducting experiments across environments with different layout maps, we demonstrate that our method offers strong generalization and outperforms existing state-of-the-art offline goal-conditioned RL methods. In summary, our main contributions in this work are as follows: 1) We propose ATR, a novel structured temporal representation for the states with the functional equivariance property. 2) Building on this representation, we propose USB, an unsupervised skill discovery method for pretraining latent-conditioned policies that learn generic and transferable skills. 3) We extend the popular OGBench (Park et al., 2025), a standard offline goal-conditioned RL benchmark, by adding a feature for generating random maze layouts and datasets, enabling a more thorough evaluation of the generalization ability of different methods. 4) Through experiments in several maze environments, we empirically demonstrate that USB+ATR learns effective and transferable skills across different layouts and offers strong out-of-distribution generalization. 2 Related Work Offline goal-conditioned RL. Goal-conditioned objectives have been widely used in unsupervised RL to learn latent representations for states that preserve temporal distances (i.e., the minimum number of steps required to reach a goal from a given state under an optimal policy). Under certain assumptions, the optimal goal-conditioned value function is equivalent to the discounted temporal distance between pairs of states (Kaelbling, 1993). Inspired by this, prior work has explored learning universal value functions as a proxy for learning temporal representations (Ghosh et al., 2023; Park et al., 2023a). Another line of work has sought to learn such representations by incorporating contrastive losses (Eysenbach et al., 2022; Ma et al., 2023; Myers et al., 2024). Since optimal goal-conditioned value functions satisfy the triangle inequality (Liu et al., 2023; Pitis et al., 2020), several model architectures (Wang and Isola, 2022; Liu et al., 2023) and objective functions (Wang et al., 2023; Myers et al., 2025) have been proposed to leverage this inductive bias in order to learn temporal embeddings more efficiently. Unsupervised skill discovery. Skill discovery is a powerful approach in unsupervised RL aiming to pretrain a latent-conditioned policy that encodes a diverse set of behaviors. These skills are often trained by behavioral cloning (Pertsch et al., 2021; Ajay et al., 2021), maximizing mutual information between skills and their future state distributions (Gregor et al., 2017; Eysenbach et al., 2019; Sharma et al., 2020), or maximizing the distance between these distributions (Park et al., 2022; Park et al., 2023b; Park et al., 2024b; Park et al., 2024a). The closest to our work is HILP (Park et al., 2024a), which incorporates a temporal representation to train each skill by encouraging movement along a random unit vector in that representation. This is achieved by defining directional rewards for each skill, computed as the inner product between the temporal displacement caused by executing this skill and its corresponding target vector. Bisimulation. Many approaches have been proposed for clustering states in general Markov decision processes (MDPs) to make a form of state abstraction. Bisimulation (Larsen and Skou, 1991) is the strictest approach, which groups only states that are indistinguishable and behaviorally equivalent under the optimal value function. Bisimulation metrics (Ferns et al., 2011; Ferns and Precup, 2014) relax this assumption by defining a notion of behavioral similarity between states. Castro (2020) introduces on-policy bisimulation, which is a scalable and computationally efficient approach for approximating bisimulation metrics by using the similarity between on-policy behaviors of states, for deterministic MDPs. Zhang et al. (2021) propose a gradient-based method for learning a latent representation for high-dimensional state spaces in general MDPs with stochastic transitions. Finally, our action-aware temporal representation is based on the goal-conditioned extension of bisimulation metrics (Hansen-Estruch et al., 2022), where an embedding of state-goal pairs is learned to cluster such pairs with similar underlying reward or action sequences. 3 Preliminaries Problem setting. We assume our environment is an MDP defined by (,,,ℛ,γ)(S,A,P,R,γ), where S is the state space, A is the action space, :×→Δ()P:S×A→ (S) is the transition dynamics, and ℛ:→ℝR:S is the reward function. Following prior works in offline goal-conditioned RL (Park et al., 2023a; Wang et al., 2023), we also assume that the MDP is deterministic. We are also given an offline unlabeled dataset D containing trajectories τ=(s0,a0,s1,…,sT)τ=(s_0,a_0,s_1,…,s_T) collected from an unknown behavior policy πbπ^b. We want to learn a skill-conditioned policy πl(a|s,z)π^l(a|s,z), where z∈z denotes the latent embedding of skills. We aim for the policy πlπ^l to learn a diverse range of behaviors for spanning the state space, while also being transferable to previously unseen layouts. Representation learning using bisimulation metrics. On-policy bisimulation metrics can be expressed in the following recursive formulation (Castro, 2020): d(si;sj)=ai∼π(⋅|si),aj∼π(⋅|sj)[|ℛ(si)−ℛ(sj)|+γ1((⋅|si,ai),(⋅|sj,aj);d)],d(s_i;s_j)=E_a_i π(·|s_i),a_j π(·|s_j) [|R(s_i)-R(s_j)|+ _1(P(·|s_i,a_i),P(·|s_j,a_j);d) ], (1) where 1W_1 denotes the 1-Wasserstein distance. As proposed by Zhang et al. (2021), replacing the 1-Wasserstein distance with the 2-Wasserstein distance allows this recursive formulation to be used as an objective for training a latent representation h. We adopt this objective for our offline setting: ℒh=(∥h(si)−h(sj)∥1−|ℛ(si)−ℛ(sj)|−γ2((⋅|h¯(si),ai),(⋅|h¯(sj),aj)))2,L_h= (\|h(s_i)-h(s_j)\|_1-|R(s_i)-R(s_j)|- _2(P(·| h(s_i),a_i),P(·| h(s_j),a_j)) )^2, (2) where ⋅¯ · denotes a stop-gradient. Under the deterministic MDP assumption, the 2-Wasserstein distance term simplifies to: 2((⋅|h¯(si),ai),(⋅|h¯(sj),aj))W_2(P(·| h(s_i),a_i),P(·| h(s_j),a_j)) = ‖h¯(si′)−h¯(sj′)‖2\| h(s_i )- h(s_j )\|_2. 4 Learning Transferable Skills using Bisimulation (a) (b) (c) (d) Figure 2: (a) We first train a state-goal pair encoder ψ using goal-conditioned bisimulation, which encodes pairs with similar underlying action sequences close to each other. (b) Next, we train an action-aware temporal representation ϕφ that preserves local temporal dependencies while respecting functional equivariance enforced by ψ. (c) Based on the similarity between the local structures shaped in representation ϕφ, we train a representation β using bisimulation to encode dynamically similar states to spatially close latent vectors. (d) Lastly, we train the skill policy using HILP directional state-spanning rewards by conditioning these skills on the β latents. Unlike HILPs, which exploit the layout structure to greedily maximize the reward (dashed line), USB instead learns monotonic behaviors across dynamically similar states, making the resulting skills transferable to unseen layouts (solid line). 4.1 Motivation Current unsupervised skill discovery methods that are based on temporal representations, such as HILP, tend to generalize poorly to unseen layouts (Sobal et al., 2025). In the absence of structural constraints, temporal embeddings are shaped by the specific layout of the environment and its corresponding temporal distances. Because the temporal relationships among states vary across layouts, skill discovery methods that rely solely on covering the temporal structure of a particular layout often fail to transfer to new environment configurations. To achieve better transferability, it is necessary to learn more primitive yet frequently occurring action sequences that remain invariant across different layouts (Figure 1). Inspired by this observation, we argue that a generalizable skill discovery method must satisfy two desiderata. First, each skill should be capable of executing one or more temporally extended actions, and collectively cover the entire state space. Second, the behavior of each skill should be conditioned only on those state features that are necessary for its execution. The first property ensures the versatility and effectiveness of the learned skills within the training layout, while the second enforces behavioral consistency across dynamically similar states, thereby enabling transfer even to previously unseen layouts. 4.2 Action-Aware Temporal Representation To train a skill policy with the properties described in the previous section, we first learn an action-aware temporal representation that preserves local temporal relationships while respecting functional equivariance. By definition, a universal temporal representation must (1) maximally spread out states while (2) preserving their local temporal distance constraints (Wang et al., 2023). However, freely maximizing the distances between states can eventually lead the representation to overfit to the specific layout, as the representation is encouraged to spread states according to layout-dependent features. Therefore, rather than freely maximizing distances between states, we aim to spread states while simultaneously enforcing functional equivariance when learning temporal distances. Although this design sacrifices some properties of a temporal representation, as our representation is no longer a universal temporal representation, it provides the additional benefit of generalizing across different layouts. To enforce functional equivariance in the representation, we adopt goal-conditioned bisimulation (Hansen-Estruch et al., 2022). Goal-conditioned bisimulation extends the classical notion of bisimulation metrics to goal-conditioned settings by grouping together state-goal pairs that correspond to analogous tasks. Inspired by Agarwal et al. (2021), and in contrast to the typical practice in the bisimulation literature, we define our bisimilarity metric as the ℓ1 _1 distance between actions rather than rewards. However, unlike Agarwal et al. (2021), who measure distances using optimal actions, we sample actions from our dataset for which we do not have optimality assumptions. Instead, the goal-conditioned formulation allows us to exploit such potentially suboptimal actions, since similarity is defined with respect to achieving the specified goals rather than relying on optimal behavior. Following the approach used in Hansen-Estruch et al. (2022) and similar to Equation (2), we define an embedding ψ(s,g)ψ(s,g) 44 4 For consistency with the notation used in other skill discovery methods, we reserve ϕφ for the final temporal representation and ψ for the goal-conditioned embedding, even though this differs from the notation in Hansen-Estruch et al. (2022). such that the ℓ1 _1 distance between embeddings of state-goal pairs reflects their bisimilarity. Accordingly, the loss function for learning the representation ψ is given by: ℒψ=(‖ψ(si,gi)−ψ(sj,gj)‖1−‖ai−aj‖1−γψ‖ψ¯(si′,gi)−ψ¯(sj′,gj)‖2)2,L_ψ= (\|ψ(s_i,g_i)-ψ(s_j,g_j)\|_1-\|a_i-a_j\|_1- _ψ\| ψ(s_i ,g_i)- ψ(s_j ,g_j)\|_2 )^2, (3) where (si,ai,si′,gi)(s_i,a_i,s_i ,g_i) and (sj,aj,sj′,gj)(s_j,a_j,s_j ,g_j) denote state, next state, action, and goal tuples sampled from the dataset D. As a result of this objective, ψ embeds bisimilar state-goal pairs close to one another in the representation space (Figure 2(a)). Note that this representation captures similarities between state-goal pairs based on their associated action sequences and does not encode any notion of temporal distance. Indeed, the same action sequence can correspond to different temporal distances when executed from different initial states. Therefore, we use ψ only to spread states across the temporal representation in a structured manner. We can now define our action-aware temporal representation ϕφ as: ϕ(si)−ϕ(si′):=ψ(si,si′)−s∈[ψ(s,s)],φ(s_i)-φ(s_i ):=ψ(s_i,s_i )-E_s [ψ(s,s)], (4) in which s∈[ψ(s,s)]E_s [ψ(s,s)] acts as a normalization term to maximally spread the action sequences in different directions. The choice of the normalizer was determined empirically, and an ablation study examining different choices of the normalizer is provided in Appendix C. Using this objective, we encourage ATR to capture the encoded directions learned by ψ, while the constraints imposed by the metric space ensure that it satisfies the triangle inequality (Figure 2(b)). For learning such a representation, the resulting objective function is defined as: ℒϕ=‖(ϕ(si)−ϕ(si′))−(ψ¯(si,si′)−s∼[ψ¯(s,s)])‖22,L_φ= \|(φ(s_i)-φ(s _i))-( ψ(s_i,s _i)-E_s [ ψ(s,s)]) \|_2^2, (5) As a consequence of enforcing the triangle inequality on the representation of states, the displacement in temporal representation caused by a given action sequence is no longer identical across all states but is instead influenced by the underlying dynamics of the surrounding states as well. This property allows us to identify similarities in state dynamics by comparing the temporal differences produced by the same action sequence across states. We leverage this characteristic in Section 4.3 to capture dynamically similar states. Additional visualizations of ATR along with comparisons to the temporal embeddings used by HILP are provided in Appendix A. We emphasize this point as particularly important. Although Hansen-Estruch et al. (2022) learn a very similar embedding, our definition of ϕφ differs in two key aspects. First, unlike Hansen-Estruch et al. (2022), who uses goal-conditioned rewards as a metric and therefore learns a temporal representation, we use actions as the metric. This choice fundamentally changes the properties of ϕφ and provides a stronger notion of functional equivariance. Second, because ψ does not provide temporal information and only the direction of vectors is informative in our case, we minimize ℒϕL_φ (Equation 5) only on adjacent states. The full embedding is then constructed from these local relationships rather than from distant state-goal pairs. Algorithm 1 Training Algorithm for USB+ATR 1: while not converged do 2: Sample (si,ai,si′,gi),(sj,aj,sj′,gj)∼(s_i,a_i,s_i ,g_i),(s_j,a_j,s_j ,g_j) 3: Update ψ to minimize ℒψ((si,ai,si′,gi),(sj,aj,sj′,gj))L_ψ((s_i,a_i,s_i ,g_i),(s_j,a_j,s_j ,g_j)) 4: Update ϕφ to minimize ℒϕ((si,si′),(sj,sj′))L_φ((s_i,s_i ),(s_j,s_j )) 5: Sample ak∼a_k 6: Update β to minimize ℒβ(si,sj,ak)L_β(s_i,s_j,a_k) 7: Update f to minimize ‖f(si,ai)−si′‖22\|f(s_i,a_i)-s_i \|^2_2 8: Sample uniformly z∼z s.t. ‖z‖2=1\|z\|_2=1 9: Compute directional reward ℛ(si,z,si′)R(s_i,z,s_i ) 10: Train πl(⋅|β¯(si),z)π^l(·| β(s_i),z) with any off-the-shelf offline RL algorithm 11: end while 4.3 Unsupervised Skill Discovery using Bisimulation Using ATRs instead of universal temporal representations reduces the variance in the action sequences encoded along each direction because of their functional equivariance property. As a result, when skills are trained with directional reward functions, they tend to exhibit more monotonic behavior. However, because each skill greedily maximizes its cumulative reward, it still may exploit the environment’s layout-specific structure by deviating from its target direction to obtain distant rewards. Such deviations conflict with the intended monotonic behavior of the skills. To prevent skills from memorizing the environment layout, we condition the skill policy only on the subset of state features that determine their dynamics, as these provide the information necessary for executing the skill. This constraint enforces skills to behave consistently across dynamically similar states and improves generalization to out-of-distribution states by reducing the influence of irrelevant state features. To this end, once again we employ bisimulation metrics to learn a representation β(s)β(s) in which states with similar transition dynamics are embedded spatially close to each other (Figure 2(c)). To define a metric that captures this relationship between states, we leverage the emergent characteristic of ATRs discussed in Section 4.2. ATRs maintain functional equivariance as long as it does not disrupt the temporal dependencies among neighboring states. Therefore, any variation in how a specific action sequence affects the temporal embedding must arise from differences in the underlying state dynamics. Based on this property, we define our metric as the dissimilarity between the temporal displacement resulting from executing the same action from two distinct states. Using this definition, the recursive formulation of bisimulation can be expressed as: d(si;sj)=ak∼Unif(),si′∼(⋅|si,ak),sj′∼(⋅|sj,ak)[ d(s_i;s_j)=E_a_k (A),s_i (·|s_i,a_k),s_j (·|s_j,a_k) [ ‖(ϕ(si′)−ϕ(si))−(ϕ(sj′)−ϕ(sj))‖1+ \|(φ(s_i )-φ(s_i))-(φ(s_j )-φ(s_j))\|_1+ γ2((⋅|si,ak),(⋅|sj,ak);d)], _2(P(·|s_i,a_k),P(·|s_j,a_k);d) ], (6) where Unif()Unif(A) denotes the uniform distribution over the action space A. Since our metric measures the difference between the temporal displacement produced by applying the same action aka_k to both sis_i and sjs_j, converting the on-policy formulation to the offline setting is challenging, as such matching actions may not be available in the dataset. To address this issue, we first learn a dynamics model f that predicts the next state. We then construct matching action pairs by sampling two states and a shared action, and generating their corresponding next states using f. The final loss for β(s)β(s) is then defined as: ℒβ=(∥β(si)−β(sj)∥1−∥[ϕ¯(f¯(si,ak))−ϕ¯(si)] _β= (\|β(s_i)-β(s_j)\|_1-\|[ φ( f(s_i,a_k))- φ(s_i)] −[ϕ¯(f¯(sj,ak))−ϕ¯(sj)]∥1− -[ φ( f(s_j,a_k))- φ(s_j)]\|_1- OPENγβ‖β¯(f¯(si,ak))−β¯(f¯(sj,ak))‖2)2. _β|| β( f(s_i,a_k))- β( f(s_j,a_k))||_2 )^2. (7) Lastly, we train the skill policy πl(⋅|β¯(s),z)π^l(·| β(s),z) which is conditioned on the β-representation of states. Following HILP, we uniformly sample skill embeddings from z∈:‖z‖2=1\z :\|z\|_2=1\, and train the latent-conditioned policy using the following directional reward: ℛ(s,z,s′)=⟨ϕ(s′)−ϕ(s),z⟩,R(s,z,s )= φ(s )-φ(s),z , (8) where ⟨⋅,⋅⟩ ·,· denotes the inner product between vectors. By training USB on ATR, each skill acquires monotonic and robust behaviors, enabling transferability to previously unseen layouts (Figure 2(d)). Bringing all components together, the complete method is summarized in Algorithm 1. Table 1: Evaluation of transferability to new layouts with a high-level controller. The average success rates with standard errors are calculated across 3 layouts and 5 seeds for each (15 runs in total). Task Method Environment Dataset Size Flat OPAL HILP USB+ATR pointmaze navigate medium 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 40.0±7.440.0 ± 7.4 91.3±4.291.3 ± 4.2 large 0.0±0.00.0 ± 0.0 30.7±6.930.7 ± 6.9 44.4±7.544.4 ± 7.5 100.0±0.0100.0 ± 0.0 stitch medium 0.0±0.00.0 ± 0.0 2.1±2.12.1 ± 2.1 11.1±4.711.1 ± 4.7 75.6±6.575.6 ± 6.5 large 0.0±0.00.0 ± 0.0 20.0±6.020.0 ± 6.0 28.9±6.828.9 ± 6.8 81.9±5.881.9 ± 5.8 antmaze navigate medium 2.1±0.72.1 ± 0.7 99.0±0.299.0 ± 0.2 57.7±6.357.7 ± 6.3 92.3±1.992.3 ± 1.9 large 0.0±0.00.0 ± 0.0 89.4±4.289.4 ± 4.2 44.9±5.444.9 ± 5.4 83.9±2.983.9 ± 2.9 stitch medium 2.1±0.72.1 ± 0.7 99.2±0.299.2 ± 0.2 59.0±6.859.0 ± 6.8 88.3±1.088.3 ± 1.0 large 0.0±0.00.0 ± 0.0 98.0±0.498.0 ± 0.4 65.2±6.165.2 ± 6.1 78.9±2.278.9 ± 2.2 explore medium 2.1±0.72.1 ± 0.7 2.1±2.12.1 ± 2.1 78.8±5.978.8 ± 5.9 84.6±5.084.6 ± 5.0 large 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 93.3±3.193.3 ± 3.1 51.7±7.251.7 ± 7.2 Average 0.630.63 44.0544.05 52.3352.33 82.8582.85 5 Experiments Table 2: Evaluation of transferability to new layouts using test-time planning. The average success rates with standard errors are calculated across 5 layouts and 5 seeds for each (25 runs in total). Task Method Environment Dataset Size GCIQL QRL CRL HIQL HILP USB+ATR pointmaze navigate medium 89.1±4.289.1 ± 4.2 90.7±5.590.7 ± 5.5 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 36.9±9.036.9 ± 9.0 large 72.2±9.172.2 ± 9.1 80.0±8.280.0 ± 8.2 0.0±0.00.0 ± 0.0 0.5±0.50.5 ± 0.5 0.0±0.00.0 ± 0.0 42.0±9.442.0 ± 9.4 stitch medium 99.3±0.699.3 ± 0.6 57.0±9.557.0 ± 9.5 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 33.7±9.333.7 ± 9.3 large 99.7±0.299.7 ± 0.2 80.0±8.280.0 ± 8.2 0.0±0.00.0 ± 0.0 14.2±6.814.2 ± 6.8 0.0±0.00.0 ± 0.0 52.5±9.852.5 ± 9.8 antmaze navigate medium 6.3±2.36.3 ± 2.3 0.4±0.40.4 ± 0.4 28.5±5.828.5 ± 5.8 5.0±1.05.0 ± 1.0 18.2±5.118.2 ± 5.1 75.7±2.275.7 ± 2.2 large 0.3±0.30.3 ± 0.3 0.0±0.00.0 ± 0.0 14.0±3.414.0 ± 3.4 2.2±0.72.2 ± 0.7 1.2±0.71.2 ± 0.7 66.3±3.966.3 ± 3.9 stitch medium 8.3±3.18.3 ± 3.1 0.0±0.00.0 ± 0.0 3.2±2.23.2 ± 2.2 9.1±3.29.1 ± 3.2 17.2±5.617.2 ± 5.6 68.6±3.068.6 ± 3.0 large 9.2±3.09.2 ± 3.0 0.0±0.00.0 ± 0.0 27.8±6.427.8 ± 6.4 0.3±0.10.3 ± 0.1 3.0±2.13.0 ± 2.1 54.8±4.354.8 ± 4.3 explore medium 57.7±5.557.7 ± 5.5 0.0±0.00.0 ± 0.0 60.4±6.260.4 ± 6.2 0.0±0.00.0 ± 0.0 20.1±7.320.1 ± 7.3 69.8±4.169.8 ± 4.1 large 13.8±4.613.8 ± 4.6 0.0±0.00.0 ± 0.0 24.9±6.324.9 ± 6.3 0.0±0.00.0 ± 0.0 13.1±5.113.1 ± 5.1 69.8±4.169.8 ± 4.1 Average 45.5945.59 30.8130.81 15.8815.88 3.133.13 7.287.28 52.1952.19 5.1 Experimental Setup We conduct our experiments on OGBench (Park et al., 2025), a recently proposed benchmark for offline goal-conditioned RL. In particular, we evaluate our method on two maze environments: pointmaze and antmaze, where the point and ant agents must navigate to a random goal location in mazes with different sizes. To study the sensitivity of each method to dataset quality, we use three dataset types provided by OGBench: navigate, stitch, and explore. Finally, to evaluate out-of-distribution generalization, we extend OGBench with a new feature for generating random maze layouts and datasets. Additional details about the environments are provided in Appendix D. 5.2 Layout Generalization with Hierarchical RL In the first experiment, we evaluate each method in a hierarchical setup. To this end, we freeze the low-level skill policy and train a high-level controller πh(z|s)π^h(z|s) to solve a downstream task with sparse rewards using online interaction. The high-level policy is invoked every H steps, where H denotes the horizon hyperparameter. During each H-step interval, the low-level policy takes actions conditioned on the current state and the skill selected by the high-level controller. For training the high-level policy, we employ Soft Actor-Critic (SAC) (Haarnoja et al., 2018). In this experiment, we compare our method against three baselines: OPAL (Ajay et al., 2021), a strong behavioral cloning-based skill discovery method; HILP, a state-of-the-art method for distance-maximizing skill discovery; and a flat policy. For the flat baseline, we train a high-level SAC agent to output primitive actions directly rather than skills, allowing us to evaluate whether the learned skills accelerate learning on downstream tasks. (See Appendix E for implementation details). Table 1 reports the overall results for each environment. USB+ATR either outperforms or performs competitively with the other baselines across most environments in this setup. Although OPAL performs well on the navigate and stitch datasets in antmaze, it completely fails on the low-quality explore dataset. This strong sensitivity to dataset quality is expected from a behavioral cloning-based method, which limits its applicability in many scenarios. In contrast, HILP exhibits weaker generalization when trained on near-optimal trajectories but benefits from suboptimal, high-coverage datasets. Consequently, its performance improves significantly when provided with more diverse state coverage. The average performance on unseen layouts demonstrates that USB+ATR generalizes effectively to novel layouts, outperforming existing state-of-the-art skill discovery methods. 5.3 Layout Generalization with Test-Time Planning Next, we evaluate the transferability of our method without any policy optimization in the test environment. Specifically, each method is trained on a single training layout, and the resulting policy is then transferred to an unseen layout to perform a downstream task. To provide the policy with information about the target layout, we additionally make a set of sampled states from the target layout available at test time. We compare our approach against several baselines: GCIQL (Park et al., 2023a), QRL (Wang et al., 2023), CRL (Eysenbach et al., 2022), HIQL (Park et al., 2023a), and HILP. GCIQL is the goal-conditioned variant of Implicit Q-Learning (Kostrikov et al., 2022), a widely adopted method in offline RL. QRL learns a quasimetric value function by optimizing a dual objective. CRL leverages contrastive learning to estimate the value function and subsequently improve the policy. HIQL is a hierarchical approach that learns a two-level policy. Since evaluating downstream tasks in previously unseen layouts without further policy optimization requires test-time planning, we equip each method with a planning procedure. HILP uses the planning approach introduced in its original framework. For USB+ATR, we perform graph-search-based planning over the ϕφ-embeddings of sampled states provided for each environment. For the remaining methods, we perform a similar graph-search-based planning procedure over the raw sampled states. Further details on graph construction and the planning approaches are provided in Appendix E. The results in Table 2 indicate that USB+ATR outperforms the other methods across most environments. Although GCIQL and QRL perform well in the pointmaze environment, this is primarily because its 2D state space provides sufficient information for planning directly over the raw state space. In contrast, all baselines fail when evaluated in the more challenging antmaze environment. However, USB+ATR maintains consistently strong performance regardless of the quality of the training dataset. 5.4 Ablation Study Table 3: Ablation study of the proposed components in the hierarchical setup. We evaluate the task and layout generalization capabilities of each variant in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 3 layouts/tasks, and 5 random seeds per setting (15 runs in total). Method Task Generalization Layout Generalization HILP+UTR 71.7±4.871.7 ± 4.8 44.9±5.444.9 ± 5.4 HILP+ATR 57.6±5.357.6 ± 5.3 25.1±4.325.1 ± 4.3 USB+UTR 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 USB+GCB 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 USB+Raw 44.2±5.544.2 ± 5.5 28.4±4.128.4 ± 4.1 USB+ATR 66.4±6.466.4 ± 6.4 83.9±2.983.9 ± 2.9 Our method consists of several components, and we hypothesize that each is necessary to achieve effective layout generalization. We therefore investigate the contribution of each component through an ablation study in the antmaze-large-navigate environment using the hierarchical evaluation setup. Specifically, we isolate each component by replacing the other components with alternative choices to assess their individual contributions. We evaluate the task and layout generalization capabilities of the following variants: HILP skills trained using our proposed ATR representation (HILP+ATR); USB directional skills trained using the universal temporal representation used in HILP (USB+UTR); USB skills trained using the representation proposed by Hansen-Estruch et al. (2022) (USB+GCB); USB skills trained directly on the raw state representation (USB+Raw). We also compare with HILP, which we denote as HILP+UTR to distinguish the skill learning method from the temporal representation it uses. The results in Table 3 support our hypothesis that USB is responsible for enabling layout generalization, while ATR provides the functional equivariance necessary to learn such transferable skills. Thus, the two components are complementary, with neither component alone being sufficient to achieve strong out-of-distribution layout generalization. For more details on this ablation study and additional ablations, please refer to Appendix C. 6 Conclusion In this work, we proposed ATR, a structured temporal representation with the functional equivariance property. We also introduced USB, which trains skills that span ATR by conditioning skill behavior on only a subset of relevant state features. Over a set of experiments on an offline goal-conditioned benchmark, we demonstrated that our method offers strong generalization to previously unseen tasks and layouts. Limitations and future work. One limitation of our work is that we model ATR using a symmetric space. Extending the goal-conditioned representation by enforcing the triangle inequality with more general spaces, such as quasimetric architectures (Wang and Isola, 2022; Liu et al., 2023), is a promising direction for future research. Moreover, although our focus in this work is on deterministic environments, extending ATR to stochastic settings remains an open problem. Finally, although we only considered locomotion maze tasks in our evaluations, our method does not make any task-specific assumptions that inherently limit its applicability to this setting. Defining new benchmarks for layout generalization in other tasks, such as robotic manipulation, and evaluating existing methods on these benchmarks could be an interesting direction for future work. References Agarwal et al. (2021) R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §1, §4.2. Ajay et al. (2021) A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum OPAL: offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §5.2. Bradbury et al. (2018) J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, et al. JAX: composable transformations of python+ numpy programs. Cited by: Appendix E. Castro (2020) P. S. Castro Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 10069–10076. Cited by: §2, §3. Chebotar et al. (2021) Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, et al. Actionable models: unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, Cited by: §1. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, p. 1597–1607. Cited by: §1. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §1. Eysenbach et al. (2019) B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine Diversity is all you need: learning skills without a reward function. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Eysenbach et al. (2022) B. Eysenbach, T. Zhang, S. Levine, and R. Salakhutdinov Contrastive learning as goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §2, §5.3. Ferns et al. (2011) N. Ferns, P. Panangaden, and D. Precup Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing 40 (6), p. 1662–1714. Cited by: §2. Ferns and Precup (2014) N. Ferns and D. Precup Bisimulation metrics are optimal value functions.. In UAI, p. 210–219. Cited by: §1, §2. Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Cited by: §1. Ghosh et al. (2023) D. Ghosh, C. A. Bhateja, and S. Levine Reinforcement learning from passive data via latent intentions. In International Conference on Machine Learning, p. 11321–11339. Cited by: §2. Gregor et al. (2017) K. Gregor, D. J. Rezende, and D. Wierstra Variational intrinsic control. In International Conference on Learning Representations (ICLR) Workshop, Cited by: §2. Gupta et al. (2019) A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman Relay policy learning: solving long horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, Cited by: §1. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, Cited by: Appendix E, §5.2. Hansen-Estruch et al. (2022) P. Hansen-Estruch, A. Zhang, A. Nair, P. Yin, and S. Levine Bisimulation makes analogies in goal-conditioned reinforcement learning. In International Conference on Machine Learning, p. 8407–8426. Cited by: §C.1, Table 5, §1, §2, §4.2, §4.2, §4.2, §5.4, footnote 4. Hauskrecht et al. (1998) M. Hauskrecht, N. Meuleau, L. P. Kaelbling, T. L. Dean, and C. Boutilier Hierarchical solution of markov decision processes using macro-actions. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, p. 220–229. Cited by: §1. Kaelbling (1993) L. P. Kaelbling Learning to achieve goals. In IJCAI, Vol. 2, p. 1094–8. Cited by: §2. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations, Cited by: Table 7. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, Cited by: Appendix E, §5.3. Larsen and Skou (1991) K. G. Larsen and A. Skou Bisimulation through probabilistic testing. Information and Computation 94 (1), p. 1–28. External Links: Document Cited by: §2. Liu et al. (2023) B. Liu, Y. Feng, Q. Liu, and P. Stone Metric residual network for sample efficient goal-conditioned reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 8799–8806. Cited by: §2, §6. Ma et al. (2023) Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang VIP: towards universal visual reward and representation via value-implicit pre-training. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. Myers et al. (2025) V. Myers, B. Zheng, B. Eysenbach, and S. Levine Offline goal-conditioned reinforcement learning with quasimetric representations. In Advances in Neural Information Processing Systems, Cited by: §2. Myers et al. (2024) V. Myers, C. Zheng, A. Dragan, S. Levine, and B. Eysenbach Learning temporal distances: contrastive successor features can provide a metric structure for decision-making. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §2. O. X.-E. Collaboration et al. (2024) O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, H. Su, H. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Luo, J. Gu, J. Tan, J. Oh, J. Malik, J. Tompson, J. Yang, J. J. Lim, J. Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Zhang, K. Rana, K. P. Srinivasan, L. Y. Chen, L. Pinto, L. Tan, L. Ott, L. Lee, M. Tomizuka, M. Du, M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. H. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Moore, S. Bahl, S. Dass, S. D. Sonawani, S. Song, S. Xu, S. Haldar, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Dasari, S. Belkhale, T. Osa, T. Harada, T. Matsushima, T. Xiao, T. Yu, T. Ding, T. Davchev, T. Zhao, T. Armstrong, T. Darrell, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang, X. Zhu, X. Li, Y. Lu, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. Wu, Y. Tang, Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, Z. J. Cui, et al. Open x-embodiment: robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. External Links: Document Cited by: §1. Park et al. (2022) S. Park, J. Choi, J. Kim, H. Lee, and G. Kim Lipschitz-constrained unsupervised skill discovery. In International Conference on Learning Representations, External Links: Link Cited by: §2. Park et al. (2025) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl. In The Thirteenth International Conference on Learning Representations (ICLR), Cited by: §1, §1, §5.1. Park et al. (2023a) S. Park, D. Ghosh, B. Eysenbach, and S. Levine HIQL: offline goal-conditioned RL with latent states as actions. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2, §3, §5.3. Park et al. (2024a) S. Park, T. Kreiman, and S. Levine Foundation policies with hilbert representations. In Proceedings of the 41st International Conference on Machine Learning, p. 39737–39761. Cited by: §1, §1, §1, §2, footnote 3. Park et al. (2023b) S. Park, K. Lee, Y. Lee, and P. Abbeel Controllability-aware unsupervised skill discovery. In International Conference on Machine Learning, Cited by: §1, §2. Park et al. (2024b) S. Park, O. Rybkin, and S. Levine METRA: scalable unsupervised RL with metric-aware abstraction. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. Pateria et al. (2021) S. Pateria, B. Subagdja, A. Tan, and C. Quek Hierarchical reinforcement learning: a comprehensive survey. ACM Computing Surveys 54 (5), p. 109:1–109:35. External Links: Document Cited by: §1. Peng et al. (2019) X. B. Peng, A. Kumar, G. Zhang, and S. Levine Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: Appendix E. Pertsch et al. (2021) K. Pertsch, Y. Lee, and J. Lim Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, p. 188–204. Cited by: §1, §1, §2. Pitis et al. (2020) S. Pitis, H. Chan, K. Jamali, and J. Ba An inductive bias for distances: neural nets that respect the triangle inequality. In International Conference on Learning Representations, External Links: Link Cited by: §2. Reed et al. (2022) S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. Transactions on Machine Learning Research. Cited by: §1. Sharma et al. (2020) A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Sobal et al. (2025) U. Sobal, W. Zhang, K. Cho, R. Balestriero, T. G. J. Rudner, and Y. LeCun Learning from reward-free offline data: A case for planning with latent dynamics models. In Advances in Neural Information Processing Systems, Cited by: §1, §4.1. Sutton et al. (1999) R. S. Sutton, D. Precup, and S. Singh Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial intelligence 112 (1-2), p. 181–211. Cited by: §1. Wang and Isola (2022) T. Wang and P. Isola Improved representation of asymmetrical distances with interval quasimetric embeddings. In Workshop on Symmetry and Geometry in Neural Representations at Conference on Neural Information Processing Systems, Cited by: §2, §6. Wang et al. (2023) T. Wang, A. Torralba, P. Isola, and A. Zhang Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, p. 36411–36430. Cited by: §2, §3, §4.2, §5.3. Wilcoxson et al. (2025) M. Wilcoxson, Q. Li, K. Frans, and S. Levine Leveraging skills from unlabeled prior data for efficient online exploration. In Proceedings of the 42nd International Conference on Machine Learning, p. 66833–66860. Cited by: Appendix E. Zhang et al. (2021) A. Zhang, R. T. McAllister, R. Calandra, Y. Gal, and S. Levine Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2, §3. Appendix A Additional Visualizations A.1 Visualization of ATR To illustrate how ATR enables directional skills to show monotonic behavior, we visualize a trained 32-dimensional ATR on pointmaze, antmaze-large-navigate datasets and compare it with its counterpart learned by HILP. For visualization, we apply principal component analysis (PCA) to project the 32-dimensional representations into two dimensions. PCA is used because it is a linear transformation that preserves the geometric structure of the latent space. This allows the sequence of states visited by each skill to be visualized as movements along an arbitrary vector. Figure 3 shows that the functional equivariance property of ATR allows it to accurately capture the layout of the environment, rather than overfitting to its global temporal structure. Since similar action sequences lie on approximately parallel directions in the latent space, ATR enables USB to learn monotonic and transferable skills. In contrast, HILP learns global temporal distances between all pairs of states, which disrupts the underlying structure of the environment. As a result, it learns skills whose behaviors change consistently over time rather than following recurring action sequences. Maze layout HILP’s temporal representation ATR antmaze-large-navigate Maze layout HILP’s temporal representation ATR pointmaze-large-navigate Figure 3: We visualize the universal temporal representation learned by HILP and ATR on a sample maze layout in two environments. ATR’s functional equivariance preserves the environment’s structural layout and aligns similar action sequences along parallel directions. In contrast, HILP distorts this structure by encoding global temporal distances and learns consistently changing behaviors. A.2 Visualization of β Representation Figure 4: Visualization of the β representation in the antmaze-medium-navigate environment. This embedding maps states with similar underlying dynamics to nearby representations while disregarding their temporal proximity. To further evaluate the qualitative properties of our β representation, we visualize it in the antmaze-medium-navigate environment. Figure 4 illustrates that temporally close states can have distant embeddings in β when their underlying dynamics differ, whereas temporally distant states with similar dynamic postures are mapped to nearby points in the representation space. Appendix B Additional Experiments B.1 Qualitative Analysis of Skills using Human High-Level Control We qualitatively analyze the skills discovered by USB to assess their monotonicity and reusability, and to examine the extent to which the learned skills resemble those illustrated in Figure 1(b). To this end, we first select four skills that span the first two principal components of ϕφ in the antmaze-large-navigate environment. We then map these four skills, corresponding to two directions along each principal axis, to the four arrow keys. A human operator can subsequently control the ant by pressing these keys to select the corresponding skills and navigate through a previously unseen maze layout. The human operator was able to successfully navigate the ant through the maze using only these four skills. This demonstration highlights the ability of our skill policy to generalize across layouts and execute the same skills in states that were never encountered during training. To the best of our knowledge, our method is the first to learn fully unsupervised skills that enable intuitive human control while transferring effectively to out-of-distribution maze layouts. A video of this experiment is available on the paper website. B.2 Task Generalization One natural question is whether improving transferability across different layouts comes at the cost of reduced performance on tasks within the same layout distribution. To investigate this trade-off, we also evaluate USB+ATR on in-distribution tasks using the test-time planning setup and compare its performance with that of other methods. The results in Table 4 show that USB+ATR maintains strong performance on standard in-distribution tasks and, on average, outperforms offline RL and skill discovery methods in the configurations for which they are specifically designed. Table 4: Evaluation of in-distribution task generalization using test-time planning. The average success rates with standard errors are calculated across 5 tasks and 5 seeds for each (25 runs in total). Task Method Environment Dataset Size GCIQL QRL CRL HIQL HILP USB+ATR pointmaze navigate medium 53.0±9.953.0 ± 9.9 94.3±4.394.3 ± 4.3 16.0±6.616.0 ± 6.6 87.6±6.287.6 ± 6.2 51.9±9.951.9 ± 9.9 66.1±8.666.1 ± 8.6 large 30.1±8.330.1 ± 8.3 84.0±7.584.0 ± 7.5 34.9±9.534.9 ± 9.5 54.9±8.854.9 ± 8.8 36.5±9.336.5 ± 9.3 70.1±8.170.1 ± 8.1 stitch medium 18.9±5.918.9 ± 5.9 79.0±7.279.0 ± 7.2 7.5±5.27.5 ± 5.2 85.0±4.685.0 ± 4.6 22.2±7.922.2 ± 7.9 76.4±8.376.4 ± 8.3 large 34.2±8.834.2 ± 8.8 87.4±6.687.4 ± 6.6 0.0±0.00.0 ± 0.0 10.2±5.810.2 ± 5.8 58.4±9.458.4 ± 9.4 76.0±8.776.0 ± 8.7 antmaze navigate medium 78.3±2.278.3 ± 2.2 89.1±1.889.1 ± 1.8 95.5±0.795.5 ± 0.7 95.0±0.795.0 ± 0.7 77.0±3.677.0 ± 3.6 83.8±3.783.8 ± 3.7 large 34.3±5.234.3 ± 5.2 77.9±2.277.9 ± 2.2 86.4±2.486.4 ± 2.4 88.4±3.188.4 ± 3.1 55.0±6.655.0 ± 6.6 64.6±4.864.6 ± 4.8 stitch medium 34.2±4.134.2 ± 4.1 68.1±5.468.1 ± 5.4 41.9±6.741.9 ± 6.7 93.8±1.193.8 ± 1.1 68.2±5.268.2 ± 5.2 72.1±5.672.1 ± 5.6 large 8.5±3.78.5 ± 3.7 18.4±6.118.4 ± 6.1 17.6±5.317.6 ± 5.3 67.3±6.067.3 ± 6.0 61.0±6.761.0 ± 6.7 49.2±6.149.2 ± 6.1 explore medium 11.5±4.011.5 ± 4.0 1.0±0.31.0 ± 0.3 1.9±0.71.9 ± 0.7 42.4±7.742.4 ± 7.7 94.6±1.594.6 ± 1.5 84.2±2.484.2 ± 2.4 large 0.7±0.40.7 ± 0.4 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 1.4±1.11.4 ± 1.1 35.0±7.135.0 ± 7.1 18.6±4.418.6 ± 4.4 Average 30.3730.37 59.9259.92 30.1730.17 62.6062.60 55.9855.98 66.1166.11 Appendix C Additional Ablation Studies C.1 Ablation Study of Method Components In this section, we provide a deeper analysis of the results presented in Section 5.4. The results show that HILP+ATR performs worse than HILP+UTR on unseen layouts, indicating that ATR alone does not improve layout generalization. USB+UTR fails completely because the universal temporal representation does not satisfy functional equivariance. Consequently, even for two states with identical dynamics, the same action sequence may correspond to completely different directions in UTR, making it impossible to learn a β representation with the desired properties. USB+GCB exhibits similar behavior, as the equivariance proposed by Hansen-Estruch et al. (2022) is defined with respect to consistency in goal-conditioned rewards rather than action consistency and therefore does not provide the functional equivariance required by USB. In contrast, USB+Raw performs substantially better than USB+UTR and USB+GCB because the raw state representation in the OGBench maze environment partially satisfies the functional equivariance property. However, because it lacks the temporal information required for learning effective skills, it is unable to achieve performance competitive with USB+ATR. Finally, USB+ATR significantly outperforms the other variants. Taken together, these results provide strong empirical evidence that ATR and USB are synergistic: functional equivariance enables consistent β-conditioned skills, while temporal embedding makes these skills effective, and only their combination yields the observed improvements in out-of-distribution layout generalization. C.2 Ablation Study of ψ normalizing term In Equation 4, we use a normalization term to maximally spread the states in the representation space. We consider several normalization options to investigate which one yields the most effective representation ϕφ. We evaluate these options on the OGBench antmaze-large-navigate environment in the test-time planning setup. The results in Table 5 indicate that s∈[ψ(s,s)]E_s [ψ(s,s)] performs best. These results also highlight the importance of selecting an appropriate normalizer to achieve maximal state spread along the ATR. Table 5: Ablation study of the ψ normalization term in the test-time planning setup. We evaluate the layout generalization of USB+ATR using different choices of the normalization term in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 5 layouts and 5 random seeds per setting (25 runs in total). Normalizing Term Success Rate ψ(si,si)ψ(s_i,s_i) 0.0±0.00.0 ± 0.0 ψ(si′,si′)ψ(s_i ,s_i ) (Hansen-Estruch et al., 2022) 0.4±0.20.4 ± 0.2 s∈[ψ(s,s)]E_s [ψ(s,s)] (Ours) 66.3±3.966.3 ± 3.9 C.3 Ablation Study of Sensitivity to Dynamics Model Learning an accurate dynamics model can be challenging in many real-world applications and may result in inaccurate models. In this ablation, we investigate whether our method relies on an accurate dynamics model by evaluating its robustness to model approximation errors. Specifically, we conduct an additional ablation study in which we systematically reduce the capacity of the dynamics model and evaluate the resulting performance on the antmaze-large-navigate environment in the hierarchical setup. We consider the following choices for the dynamics model: • Small model. A low-capacity model with hidden dimension (64)(64), resulting in approximately 10× higher MSE than the original model. • Medium model. A medium-capacity model with hidden dimensions (128,128)(128,128), resulting in approximately 4× higher MSE than the original model. • Large model. Our original model with hidden dimensions (512,512,512)(512,512,512). Table 6: Ablation study of sensitivity to the dynamics model in the hierarchical setup. We evaluate the layout generalization of USB+ATR using dynamics models with different capacities in the antmaze-large-navigate environment. Average success rates with standard errors are reported across 3 layouts and 5 random seeds per setting (15 runs in total). Model Size Success Rate Small 59.1±6.459.1 ± 6.4 Medium 69.2±5.569.2 ± 5.5 Large (Ours) 83.9±2.983.9 ± 2.9 The results in Table 6 show that our method achieves strong out-of-distribution generalization even when using the small dynamics model, which has approximately 10× higher MSE than the original model. Appendix D Environment Details Our experiments are conducted on antmaze and pointmaze environments, along with datasets provided by OGBench. Each environment contains an agent (with 2 degrees of freedom for point and 8 degrees of freedom for ant) with the task of reaching a specified goal. We consider two environment scales: medium and large. For each environment, OGBench provides offline trajectory datasets with different qualities, which we use for pretraining skill discovery methods. The navigate dataset consists of trajectories collected from a noisy expert, the stitch dataset is composed of shorter trajectories that must be concatenated to reach distant goals, and the explore dataset contains extremely low-quality but high-coverage trajectories. To evaluate the generalization capabilities of the methods, we extended OGBench with a new feature that generates random layouts and their corresponding tasks for each environment. The generated layout with seed 0 matches the default layout and its corresponding tasks provided by OGBench, which we use to assess task-generalization performance (Appendix B.2). To evaluate the transferability of learned skills across different layouts, we use the generated layouts with seeds 1,2,3,4,5\1,2,3,4,5\ for the test-time planning evaluation (Section 5.3) and 1,2,3\1,2,3\ for hierarchical evaluation (Section 5.2). These layouts, along with their tasks, are shown in Figure 5. seed=0 (default) seed=1 seed=2 seed=3 seed=4 seed=5 antmaze, pointmaze-medium seed=0 (default) seed=1 seed=2 seed=3 seed=4 seed=5 antmaze, pointmaze-large Figure 5: Randomly generated layouts in two sizes, used for evaluating the transferability of skills across different layouts Appendix E Implementation Details Offline skill discovery. All our implementations are in JAX (Bradbury et al., 2018) and are based on the reference implementation codebase provided by the OGBench. For ATR, we use our own implementation of goal-conditioned bisimulation. We tuned the hyperparameter γψ _ψ by sweeping over 0.0,0.5,0.99\0.0,0.5,0.99\ and found that smaller values of γψ _ψ yield better performance on lower-quality datasets. For USB, we use the open-source HILP implementation for the offline goal-conditioned setting, which trains skill policies using IQL (Kostrikov et al., 2022) and AWR (Peng et al., 2019). Unlike the universal temporal representation used by HILP, ATR does not constrain the norm of the difference vector between adjacent states. Thus, to prevent training instability arising from very large or very small rewards, we found that normalizing and standardizing the directional rewards is useful in some environments. We tuned γβ _β via a sweep over 0.9,0.95,0.99\0.9,0.95,0.99\, and also performed a sweep over the dimension of β in 8,16,32\8,16,32\. To preserve fairness across methods, we do not tune any other hyperparameters shared with HILP and instead use their suggested values. A complete report of hyperparameters used for USB+ATR is provided in Table 7. For OPAL, since no official implementation is available, we use the implementation provided by SUPE (Wilcoxson et al., 2025). We use the same hyperparameter values as those provided in that implementation. Online hierarchical RL. To evaluate the learned skills on downstream tasks in hierarchical setup, we use SAC (Haarnoja et al., 2018) to train a high-level policy on top of each skill discovery method through online interaction. We adopt the implementation provided by SUPE for this purpose. To determine the optimal horizon length for each method, we perform a sweep over 4,8,16\4,8,16\ and find that a horizon of 4 works best for OPAL, while 8 performs best for HILP and USB. All other hyperparameters remain unchanged compared to SUPE. We train the high-level model for up to 500K environment steps on the medium maze layout and 800K steps on the large layout. Evaluations are conducted at 300K, 400K, and 500K steps for the medium layout, and at 600K, 700K, and 800K steps for the large layout. For each evaluation step, results are averaged over 50 rollouts, and we report the mean performance across these evaluation steps. For a fair evaluation, the high-level policy is updated every 8 environment steps in each method to keep the total number of gradient steps identical for all. Test-time planning. To evaluate the performance of the methods on unseen layouts using test-time planning, we generated a dataset for each layout and randomly sampled N=50,000N=50,000 states from each dataset, which were made available to the methods at test time. For USB+ATR, we then computed the corresponding embeddings of these states in the ϕφ space. We constructed a planning graph using a simple K-nearest-neighbor approach: for each state, we identified its K nearest neighbors and connected them with edges weighted by their Euclidean distance. We tuned K over 4,8,16\4,8,16\ and found that K=4K=4 performed best. We then applied Dijkstra’s algorithm to compute the shortest-path distance from each sampled state to the goal. With this preprocessing, we can dynamically identify the shortest path to the goal during execution without running Dijkstra’s algorithm over the entire graph at every time step. We then select the L-th node along the resulting path as the subgoal, where L denotes the number of lookahead steps (e.g., L=1L=1 corresponds to the immediate next node), and perform the task toward this subgoal. We tune L over 1,2,4,8\1,2,4,8\ and find that L=1L=1 performs best in antmaze, while L=4L=4 performs best in pointmaze. For all baselines other than HILP, which has its own planning procedure, we used the same planning approach but constructed the graph in the raw state space rather than in a learned representation space. Table 7: Hyperparameters for USB+ATR Hyperparameter Value # gradient steps 10610^6 Learning rate 0.00030.0003 Optimizer Adam (Kingma and Ba, 2015) Batch size 10241024 MLP hidden dimensions (512,512,512)(512,512,512) Target smoothing coefficient τ 0.0050.005 ψ discount factor γψ _ψ 0.990.99 (default), 0.00.0 (explore) ϕφ latent dimension 3232 β discount factor γβ _β 0.990.99 (default), 0.90.9 (explore) β Latent dimension 1616 (antmaze), 88 (pointmaze) Skills discount factor γ 0.990.99 Skills IQL expectile 0.90.9 Skills AWR temperature 1010 Normalize rewards False (antmaze), True (pointmaze)