Paper deep dive
Q-Shaped Options for Hierarchical Reinforcement Learning
Clarisse Wibault, Antoine Gorceix, Antonio Léon Villares, Alexey Zakharov, Evangelos Chatzaroulas, Michael Matthews, Eduardo Pignatelli, Jakob Foerster
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/10/2026, 3:10:33 AM
Summary
The paper introduces Q-Shaped Options (QSO), a Hierarchical Reinforcement Learning (HRL) algorithm that learns action abstractions via Q-functions to balance control sufficiency and data aggregation. QSO uses a shared encoder shaped by low-level Q-functions (to retain necessary distinctions) and high-level Q-functions (to discard unnecessary ones), outperforming baselines in offline goal-conditioned locomotion and manipulation tasks.
Entities (10)
Relation Signals (8)
Q-Shaped Options → isa → Hierarchical Reinforcement Learning
confidence 95% · We introduce Q-Shaped Options (QSO) to address all three... In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO)...
Q-Shaped Options → uses → Low-level Q function
confidence 90% · The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control.
Q-Shaped Options → uses → High-level Q function
confidence 90% · The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded.
High-level Q function → encourages → Discarding Unnecessary Distinctions
confidence 88% · The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded.
Low-level Q function → ensures → Control Sufficiency
confidence 88% · Proposition 1. Low-level Q-shaping implies control sufficiency.
Q-Shaped Options → outperforms → OPAL
confidence 85% · QSO... outperforms baselines... achieving non-zero performance in tasks where all other evaluated algorithms fail.
Q-Shaped Options → outperforms → SHARSA
confidence 85% · QSO... outperforms baselines... achieving non-zero performance in tasks where all other evaluated algorithms fail.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. In principle, Hierarchical Reinforcement Learning (HRL) addresses both challenges through the interaction between action (temporal) and state (spatial) abstraction. First, using an action abstraction to represent temporally extended behaviour as options reduces the effective decision horizon. Second, enabling different state abstractions at each level of the decision process permits greater data aggregation for learning. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. Current HRL algorithms fail in one of two ways. Some discard distinctions between options needed for optimal control, undermining hierarchy altogether. Others retain unnecessary distinctions, preserving horizon reduction, but forfeiting coarser state abstraction. In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstraction between consecutive levels as a shared encoder shaped by their respective Q functions. The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control. The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail.
Tags
Links
- Source: https://arxiv.org/abs/2610.12135v1
- Canonical: https://arxiv.org/abs/2610.12135v1
Trouble viewing inline? Open PDF directly →
Full Text
98,568 characters extracted from source content.
Expand or collapse full text
Preprint, October 2026. Q-SHAPED OPTIONS FOR HIERARCHICAL REINFORCEMENT LEARNING Clarisse Wibault Antoine GorceixAntonio L ́ eon VillaresAlexey ZakharovEvangelos Chatzaroulas Michael MatthewsEduardo PignatelliJakob Foerster BOLD University of Oxford clarisse.wibault@magd.ox.ac.uk ABSTRACT Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. In principle, Hi- erarchical Reinforcement Learning (HRL) addresses both challenges through the interaction between action (temporal) and state (spatial) abstraction. First, using an action abstraction to represent temporally extended behaviour as options reduces the effective decision horizon. Second, enabling different state abstractions at each level of the decision process permits greater data aggregation for learning. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. Current HRL algorithms fail in one of two ways. Some discard distinctions between options needed for optimal control, undermining hierarchy altogether. Others retain unnecessary distinctions, preserving horizon re- duction, but forfeiting coarser state abstraction. In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstrac- tion between consecutive levels as a shared encoder shaped by their respective Q functions. The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control. The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail. Our code is open sourced at https://github.com/CWibault/QSO. 1INTRODUCTION Learning to tackle long-horizon, goal-conditioned (Kaelbling, 1993; Schaul et al., 2015) tasks requires an agent to reason over extended timescales and act across a broad range of states (Sutton et al., 1999; Park et al., 2026). In goal-conditioned two-tiered Hierarchical Reinforcement Learning (HRL) a high-level policy outputs a high-level action or option (Sutton et al., 1999) conditioned on the current state and goal; a low-level policy then conditions on this option as well as the current state to output a primitive action (Figure 1a). In maze navigation, for example, the option might be to reach “a point two metres down”, while the primitive action could be applying torques to an individual joint. In principle, HRL (Barto and Mahadevan, 2003) addresses both challenges of long-horizon, goal- conditioned tasks through the interaction between action (temporal) and state (spatial) abstraction (Konidaris, 2019). First, by using an action abstraction to represent temporally extended behaviour as options, a hierarchical policy reduces the effective decision horizon (Park et al., 2026). The high-level policy reasons over options instead of primitive actions, while the low-level policy reasons to pursue the option rather than the distant goal. Second, by enabling different state abstractions at each level of the hierarchy (Figure 1a), a hierarchical policy can achieve coarser state abstraction than a non-hierarchical policy if each level discards different information irrelevant to its part of the decision process (Konidaris, 2019; Konidaris et al., 2018). Intuitively, this permits greater data aggregation for 1 arXiv:2610.12135v1 [cs.AI] 8 Oct 2026 Preprint, October 2026. Figure 1: (a) HRL permits different state abstractionsφ h (s t ,g)andφ l (s t ,ω t )at each level of the decision process. (b) Decoupled policy training. (c) Different methods for learning the action abstraction φ ω (τ t:t+k ). learning. For example, the high-level can aggregate all examples at a certain coordinate irrespective of pose, which is too fine-grained for high-level control, while the low-level can aggregate all examples with a certain pose irrespective of coordinates, since reaching “a point two metres down” requires the same actions wherever the agent is. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. To improve stability and avoid collapsing to degenerate solutions (Bacon et al., 2017), decoupled approaches (Ajay et al., 2020; Park et al., 2023; 2026) train each policy separately (Figure 1b) using fixed-length trajectory segments. The action abstraction maps these segments to the option space, serving as actions for the high-level policy and goals for the low-level policy during training. The issue is that current HRL algorithms learn abstractions that fail in one of two ways. Some discard distinctions between options that are needed for optimal control (Park et al., 2023; Li et al., 2006). This limits the quality of the recoverable policy and undermines hierarchy altogether. Others retain distinctions between options that are irrelevant for control (Park et al., 2026; Ajay et al., 2020; Ravindran and Barto, 2002). This preserves horizon reduction but forfeits coarser state abstraction, as functionally equivalent behaviours remain distinct outputs of the high-level policy and distinct inputs to the low-level one. In this work, we characterise three desiderata for an action abstraction to enable horizon reduction and data aggregation. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture (Park et al., 2026) with separate state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstraction between consecutive levels as an encoder shared between the two levels and shaped by their respective Q functions. To our knowledge, QSO is the first HRL algorithm to learn an action abstraction through a Q function. We encode only the current and target states to represent the outcome of a trajectory segment rather than the exact state-action sequence that produced it. We shape the abstraction via the low-level Q function to separate options with different low-level Q values (Li et al., 2006), retaining distinctions necessary for optimal control. Finally, we shape the abstraction via the high-level Q function to provide an inductive bias for discarding unnecessary distinctions between functionally equivalent behaviour. Across offline (Levine et al., 2020; Lange et al., 2012) goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces (Figures 2, 6 and 7) and outperforms baselines (Table 2), achieving non-zero performance in tasks where all other evaluated algorithms fail (Table 7). We summarise our contributions as follows. 1. We provide a formal definition of a control-sufficient action abstraction and prove that exact low-level Q-shaping satisfies control sufficiency. 2.We extend a Probably Approximately Correct bound on offline error in a finite-state and finite- action MDP to motivate discarding unnecessary distinctions between options. 3.We develop Q-Shaped Options (QSO), to our knowledge the first HRL algorithm to learn the action abstraction through a Q function, yielding a control-sufficient action abstraction that also discards unnecessary distinctions between options. 2RELATED WORKS As summarised in Table 1 and shown in Figure 1c, prior algorithms using decoupled training approaches differ substantially both in what the action abstraction encodes, ranging from a single 2 Preprint, October 2026. AlgorithmEncodesMethodPreservesDiscards HIQLs t ,s t+k VNoYes OPALτ t:t+k β-VAE (Surrogate Objective)YesNo SHARSA s t+k -YesNo DSHARSA s t+k -YesNo ASOs t+k -YesNo QSOs t ,s t+k Q h and Q l YesYes Table 1: Properties of the action abstractionφ ω : what the abstraction encodes; the method used to learn the abstraction; whether the objective encourages preserving distinctions necessary for optimal control and whether the abstraction empirically exhibits discarding unnecessary distinctions. target states t+k to the fullk-step state-action trajectory segmentτ t:t+k , and the method used to learn the action abstraction. HIQL (Park et al., 2023) learns the action abstraction through its single action-free state-value functionV (s t ,φ ω (s t ,g)). However, value equivalence does not imply action equivalence (Li et al., 2006), as two state-goal pairs can share a value, and hence option, while requiring entirely different immediate actions (Figure 2a). This means that the abstraction can discard information necessary for optimal control. SHARSA and DSHARSA (Park et al., 2026) simply bypass learning an action abstraction by using subgoal optionsω t = s t+k (Konidaris, 2019), which specify the entire state that the low-level policy should reach. The “encoder” is just an identity mapping. While these “options” retain the information required for low-level control, they distinguish functionally equivalent behaviour occurring at different points in the state space (Figure 2c), limiting coarser state abstraction and hence data aggregation for learning. OPAL (Ajay et al., 2020) uses a surrogate objective to learn an option spaceω t ∼ φ ω (ω t | τ t:t+k ) with aβ-VAE trained to reconstruct the entire action sequence. This means that the option encodes how a segment was executed, instead of its outcome. Similarly to SHARSA and DSHARSA, such an encoding hinders coarser state abstraction as two trajectory segments with the same outcome may be represented differently just because their executions differ (Figure 2b). QC (Li et al., 2026), which builds on action chunking (Zhao et al., 2023), is closely related to our work as it also provides temporal abstraction, but rather than having a multi-level structure, a non- hierarchical policy outputs an action chunkπ(a t:t+k | s t ,g). Because a single level is responsible for the entire decision process, this prevents coarser state abstraction. 3PRELIMINARIES Problem Setting We consider a goal-conditioned Markov Decision Process (MDP) (Sutton et al., 1998; Schaul et al., 2015) defined by the tupleM := (p s 0 ,p g ,S,G,A,T ,r, ̄γ), wherep s 0 is the initial state distribution,p g is the goal distribution,S,GandArespectively denote the state, goal and action spaces withG = S, andTis the transition function.r(s,g) := −(1− β(s,g))and ̄γ(s,g) := γ· (1−β(s,g)), whereγ ∈ [0, 1)is a constant, are the reward and discount or continuation functions (Andrychowicz et al., 2017; Sutton et al., 2011; White et al., 2012) defined through a goal-achievement indicatorβ :S×G →0, 1(specified per environment in Appendix H): the agent receives−1per step until the goal is reached, after which the return terminates. At the beginning of the episode, a states 0 is sampled fromp s 0 , and a goalgis sampled fromp g . The goal is fixed for the entire episode. At each timestept≥ 0, an agent takes an actiona t conditioned on its current state s t and goal stateg, and deterministically transitions to a new states t+1 =T (s t ,a t ); we adopt the fully observed deterministic MDP framework as used in Park et al. (2025) for simplicity. Our aim is to learn a policyπ :S×G → ∆(A), where∆(A)denotes the space of probability distributions overA, from a fixed datasetDofNstate-action trajectories of lengthH, collected by an arbitrary policy, that maximisesE π [ P ∞ t=0 ( Q t−1 i=0 ̄γ(s i ,g))r(s t ,g)]. The return is bounded in[−1/(1− γ), 0] and is maximised by minimising the discounted time to reach the goal. In what follows we write the constant γ in place of ̄γ(s,g) and omit goal-termination masking for clarity. 3 Preprint, October 2026. Hierarchical Reinforcement Learning In two-tiered goal-conditioned HRL (Barto and Mahade- van, 2003; Park et al., 2023) the aim is to learn a high-level policyπ h (ω t | s t ,g)that outputs an optionω t ∈ Ω, whereΩdenotes the option space, conditioned on the current states t ∈Sand goal g ∈G; a low-level policyπ l (a t | s t ,ω t )then conditions on this option as well as the current state to output a low-level actiona t ∈A(Figure 1a), such that the overall policy is defined by the following: π(a t | s t ,g) := Z ω t ∈Ω π l (a t | s t ,ω t )π h (ω t | s t ,g)dω t . While options may have fixed or variable temporal extent, in this work we focus on fixed-length options, which do not collapse to degenerate solutions (Bacon et al., 2017) and simplify the separation of the two decision processes with a fixed temporal and discount relationship. During training, an action abstractionφ ω is learned and used to encode (a subset of) thek-step state-action trajectory segmentτ t:t+k = (s t ,a t ,· ,s t+k )into an optionω t = φ ω (τ t:t+k ). The high-level policy is trained to output this option, while the low-level policy is trained to condition on this option. At deployment, the action abstraction is no longer required, since the two levels communicate entirely within option space. 4Q-SHAPED OPTIONS Our aim is to learn an action abstraction that represents the outcome of a trajectory segment rather than the exact state-action sequence that produced it, retains distinctions between options necessary for optimal control, and discards unnecessary distinctions. 4.1DESIDERATA The action abstraction should represent the outcome of a trajectory segment rather than the exact state-action sequence that produced it. We want to prevent potentially suboptimal dataset execution from becoming a part of the option’s representation. To enable two different executions with the same outcome to map to the same option, we restrict the abstraction to be a function of the state-achieved-state pair(s t ,s t+k ), rather than the entire state-action trajectory segment. We include the current state as well as the achieved state to enable representing relative rather than absolute outcome (see Desideratum 3). HIQL also encodes (s t ,s t+k ), but motivates this choice empirically. Desideratum 1. Represent outcome rather than exact execution. QSO: Action abstraction encodes (s t ,s t+k ) instead of τ t:t+k . The action abstraction should be control-sufficient i.e. preserve distinctions between options necessary for optimal control. We want two target states that require different actions from the same current state map to different options, so that a policy conditioning on the representation instead of the target state can still be optimal for both. To address control sufficiency, we shape the abstraction through the low-level Q function. If two admissible target states require different actions from the same current state, the low-level Q function must represent different sets of action values. Minimising the Bellman error therefore pressures the action abstraction to map the current and target state pairs to different points in the option space. Conversely, if two trajectory segments have functionally equivalent relative outcomes but different dataset executions, they can share an option if their sets of action values are compatible, even if one was executed well and the other poorly. This can enable the low-level policy to draw on the better execution at deployment. Definition 1 formalises this notion of a control-sufficient action abstraction, while Proposition 1 connects this property to the low-level Q shaping objective. We refer to Appendix B for the proof. 4 Preprint, October 2026. Definition 1. Control-sufficient action abstraction. LetG k (s)⊆Sdenote the set of admissible target states for thek-step low-level decision problem from states, and letΠ ⋆ (s,g s ) := argmax a Q ⋆ (s,g s ,a)denote the set of optimal actions of the original MDP. An action abstractionφ ω is control-sufficient if, for alls∈Sandg s ,g ′ s ∈G k (s), φ ω (s,g s ) = φ ω (s,g ′ s )=⇒Π ⋆ (s,g s ) = Π ⋆ (s,g ′ s ). In particular, there exists a low-level policyπ l such thatπ l (· | s,φ ω (s,g s ))is supported on Π ⋆ (s,g s ) for every admissible pair (s,g s ). Note that Q-shaping is stronger than control sufficiency, as it distinguishes between options whose action values differ only by an additive offset or in the ranking of suboptimal actions, even if these differences do not affect the optimal action. Proposition 1. Low-level Q-shaping implies control sufficiency. Suppose there exists a low-level criticQ l such thatQ l s,φ ω (s,g s ),a = Q ⋆ (s,g s ,a) for all s ∈ S,g s ∈ G k (s)anda ∈ A, i.e.φ ω is aQ ⋆ -irrelevance abstraction in the sense of Li et al. (2006). Then φ ω is control-sufficient. Desideratum 2. Preserve distinctions between options necessary for optimal control. QSO: Learn action abstraction through low-level Q function: Q l (s t ,φ ω (s t ,g s ),a t ). The action abstraction should discard unnecessary distinctions between options. We want trajectory segments with similar outcomes to map to the same option, so that experience can be aggregated across the state space. This enables conservative extrapolation: aggregating different examples across different levels of the hierarchy to produce an action not observed for a particular state-goal pair. Compressing a finite option space or organising a continuous option space in a semantically meaningful way reduces the effective complexity of the hierarchical decision problem and can reduce the distribution mismatch between the optimal policy and the dataset by aggregating occupancy mass across equivalent state-goal and state-option pairs. We refer to Appendix C for an example, and Appendix D for a proposition that formalises this intuition by deriving and analysing the Probably Approximately Correct bound on offline error for a hierarchical policy with joint state and action abstraction. To encourage discarding unnecessary distinctions between options, we additionally shape the option space through the high-level Q function. Functionally equivalent state-target pairs induce similar high-level action values, so distinguishing between them provides little predictive benefit toQ h . Note that, unlike control sufficiency, this invariance is not explicitly enforced by the objective, instead simply providing an inductive bias towards it under shared function approximation and gradient- based optimisation. Empirically, we find that this is sufficient to form coherent clusters in the option space. Discarding unnecessary distinctions between options also motivates conditioning the action abstraction on the current state, as then the abstraction only needs to separate target states if they require different actions from the same current state rather than if they require different actions from any admissible current state. Desideratum 3. Discard unnecessary distinctions between options. QSO: Learn action abstraction φ ω through the high-level Q function: Q h (s t ,g,φ ω (s t ,s t+k )). 4.2PRACTICAL IMPLEMENTATION Joint Action Abstraction and Value LearningWe learn the action abstraction between consecutive levels of the hierarchy as an encoder shared between the two levels and jointly shaped by their 5 Preprint, October 2026. respective Q functions: min Q l ,Q h ,φ ω L Q l ,φ ω +L Q h ,φ ω where the critic losses are the following, conservative Implicit Q Learning (Kostrikov et al., 2021) critic losses: L Q l ,φ ω = E (s t ,a t ,s t+1 ,g s )∼D ω t+1 =φ ω (s t+1 ,g s ) " Q l s t ,φ ω (s t ,g s ),a t − r l (s t ,g s )− γV l (s t+1 ,ω t+1 ) 2 # , L Q h ,φ ω = E (s t ,s t+k ,g)∼D " Q h s t ,g,φ ω (s t ,s t+k ) − R h (s t:t+k ,g)− γV h (s t+k ,g) 2 # , and the normalised k-step option return is R h (s t:t+k ,g) = 1−γ 1−γ k P k−1 t ′ =0 γ t ′ r h (s t+t ′ ,g). We use a continuous option space, parameterising the action abstraction with an MLP followed by length normalisation (Park et al., 2023) onto a hypersphere. Unlike a discrete option space, this avoids imposing arbitrary boundaries between semantically similar options, while the length normalisation keeps proposed options in-distribution for the low-level policy. At deployment, the high-level policy resamples an option at every timestep, allowing the agent’s intent to slowly shift across the option space. We use the same discount factor at both levels so that each level looks1/(1− γ)of its own decisions ahead, and normalise thek-step option return so that a full-length option has the same cost as a single primitive action. This keeps the two Q functions on comparable scales and hence avoids a hyperparameter to balance their losses. Ideally,k = 1/(1− γ)so that one option spans exactly the horizon that the low-level reasons over. In practice,kis bounded by the data (Park et al., 2023; 2026; Li et al., 2026). Algorithm 1 Q-Shaped Options (QSO) Training Initialise option representationφ ω (s t ,g s ), high-level valueV h (s t ,g), criticQ h (s t ,g,ω)and flow policy π h (ω | s,g), low-level valueV l (s t ,ω), criticQ l (s t ,ω,a t )and flow policyπ l (a t | s,ω)and target option and critic networks ̃ φ ω , ̃ Q h , ̃ Q l . repeat ▷ Dataset Sampling and Reward Labelling. Sample states fromD. Determine a t ,s t+1 ,s t+k . Sample high-level goals g and low-level targets g s using Equations 8 and 9. Determine rewards r l (s t ,g s ) and r h (s t:t+k ,g). ▷ Joint Action Abstraction and Value Learning. Encode high-level options ω ← φ ω (s t ,s t+k ) and target options ̃ω ← ̃ φ ω (s t ,s t+k ). Encode low-level options φ ω (s t ,g s ) and target options ̃ φ ω (s t ,g s ). Train V h and V l using Equations 4 and 6. Train Q h and Q l and φ ω using Equations 5 and 7. Update ̃ φ ω , ̃ Q h , ̃ Q l using exponential moving averages. ▷ Policy Extraction. Train π AWR h and π BC l using Equations 10 and 11. until convergence Policy Extraction Given the learned action abstraction and value functions, we extract a policy at each level using conservative policy extraction algorithms (Peng et al., 2019; Ghasemipour et al., 2021). The high-level policy uses Rejection Sampling (RS) over an advantage-weighted behaviour cloning (BC) flow policy; the low-level uses BC alone. We use advantage-weighting at the high-level because its dataset contains a large proportion of random goals, needed to cover the otherwise out-of-distribution state-goal pairs faced at deployment. While pure BC learns to ignore the goals, advantage-weighting restores goal-conditioning and leads to better proposals for RS. This issue does not arise at the low-level, for which we sample almost no random target states. In fact, we empirically found that advantage-weighting and RS degraded low-level performance by substantially reducing the effective batch size and exploiting errors in the Q function. Even though the low-level is trained with BC, we still get conservative extrapolation, as the high-level policy can pair a state with a better option than observed at that state during training; the low-level then pursues the option with behaviour learned elsewhere in the dataset. Figure 17 and Table 9 in Appendix H show performance and goal sensitivity when ablating high-level policy extraction. 6 Preprint, October 2026. Algorithm 2 Q-Shaped Options (QSO) Deployment while episode not terminated do ▷ High-level Rejection Sampling. Sample ω 1 ·ω N ∼ π AWR h (·| s t ,g). Select ω ← argmax ω 1 ·ω N Q h (s t ,g,ω i ). ▷ Low-level Behaviour Cloning. Sample a t ∼ π BC l (·| s t ,ω). end while We provide pseudocode for QSO training and deployment in Algorithms 1 and 2, and full equations and dataset sampling methods in Appendix G. We refer to Appendix A for our choice of IQL and policy extraction algorithms. 5EXPERIMENTS We first evaluate QSO’s downstream performance from a fixed offline dataset to isolate policy learning from data collection. To assess our second and third desiderata, we introduce ASO, which shares QSO’s policy extraction but, like SHARSA and DSHARSA, uses absolute subgoal options instead of a learned action abstraction. This isolates the effect of QSO’s action abstraction: discarding distinctions necessary for control should yield a worse policy than ASO, whereas discarding only unnecessary ones should improve performance through greater data aggregation. We then analyse the learned option space directly, testing whether functionally equivalent behaviours map to similar points while functionally distinct ones remain separated. Finally, we ablate QSO’s components to identify their individual contributions. Experimental Protocol We evaluate on the most challenging OGBench (Park et al., 2025) envi- ronments using the default datasets, as OGBench is a standard offline RL benchmark with complex, long-horizon, goal-conditioned tasks. We compare against hierarchical algorithms from Section 2. As non-hierarchical baselines, we include IQL with DDPGBC (Fujimoto and Gu, 2021) policy extraction, the strongest non-hierarchical baseline (Park et al., 2025; 2024), and QC. All baselines use the hyperparameters from their original implementations. Apart from the option dimension, for which we use the OGBench default, QSO introduces no additional hyperparameters, so we use those of SHARSA and DSHARSA. Environment, baseline and hyperparameter details are in Appendix H. Non-Hierarchical Hierarchical QSO / Best Mean Environment IQL QCHIQLOPALSHARSADSHARSAASOQSO antmaze-giant-stitch-v0 0 [0, 0] 0 [0, 0]54 [48, 60]53 [52, 55]68 [61, 76]72 [67, 77]68 [66, 70]68 [66, 70]94% antmaze-teleport-stitch-v0 47 [44, 51] 37 [32, 40]36 [34, 39]69 [64, 74]53 [52, 55]58 [55, 62]66 [62, 69]54 [52, 57]79% humanoidmaze-giant-stitch-v0 1 [1, 1] 0 [0, 0]6 [4, 8]82 [78, 85]30 [27, 33]30 [27, 32]44 [38, 48]90 [88, 92]100% cube-double-play-v0 20 [17, 23] 72 [70, 74]7 [5, 9]29 [26, 33]72 [68, 77]80 [75, 84]68 [65, 70]70 [65, 74]87% cube-triple-play-v0 6 [6, 7] 23 [19, 27]16 [14, 18]3 [2, 4]14 [10, 18]14 [13, 16]4 [4, 4]22 [20, 23]92% cube-quadruple-play-v0 0 [0, 0] 2 [1, 2]3 [1, 5]0 [0, 0]0 [0, 1]0 [0, 1]0 [0, 0]4 [4, 5]100% scene-play-v0 48 [45, 51] 82 [81, 83]41 [35, 46]84 [78, 87]92 [90, 93]89 [87, 91]94 [91, 95]85 [82, 90]91% puzzle-3x3-play-v0 100 [99, 100] 81 [46, 99]48 [46, 49]70 [65, 74]78 [71, 87]85 [81, 90]76 [72, 80]72 [69, 75]72% puzzle-4x4-play-v0 71 [61, 78] 25 [23, 27]0 [0, 1]64 [61, 68]74 [67, 79]72 [69, 75]69 [65, 75]92 [81, 100]100% puzzle-4x5-play-v0 14 [11, 15] 16 [13, 19] 4 [3, 5]24 [22, 25]17 [15, 20]18 [18, 19]14 [11, 18]15 [11, 20]65% puzzle-4x6-play-v0 8 [5, 10] 16 [14, 19]5 [4, 6]17 [15, 18]14 [10, 16]16 [14, 17]11 [8, 15]18 [15, 22]100% Average over 11 Environments 29 [27, 30] 32 [29, 34]20 [20, 20]45 [44, 46]46 [44, 49]49 [48, 49]47 [46, 48]54 [51, 56]100% Table 2: Summary of results on selected OGBench environments using the default datasets. Success rate averaged over 5 evaluation tasks after 1M training steps. Mean and 95% bootstrap confidence interval over 4 seeds. Results Table 2 shows that QSO achieves the highest average success rate across the 11 environ- ments, 54% against 49% for the next best method (DSHARSA). Particularly notable is its performance in humanoidmaze, which has the largest action space dimension (21, against at most 8 in the other environments) where it achieves 90% against 82% for the next best method and three times that of DSHARSA. In puzzle-4x4 it achieves 92% against at most 74% for any baseline. In cube-quadruple, the hardest manipulation environment, QSO is the only method with non-zero success on two of the five tasks (Table 7 in Appendix H). QSO performs consistently across environments, achieving at 7 Preprint, October 2026. Figure 2: PCA (top) of the action abstraction for antmaze-giant fitted on random segments, showing different groups of segments with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) in units of median distance between two random segments. QSO is the only algorithm that consistently groups functionally equivalent behaviours (Desideratum 3) while separating behaviour groups (Desideratum 2). least 65% of the best success rate in all environments, while every other algorithm drops below 35% of the best in at least one. Is the action abstraction semantically meaningful? We test whether QSO maps functionally equivalent behaviours to similar parts of the option space by identifying groups of semantically similar behaviours in antmaze-giant, cube-double, and cube-quadruple (Figures 2, 6, and 7). For each behaviour, we randomly sample trajectory sequences from the validation dataset and compare PCA projections and whitened distances of their encodings. An invariant encoder should produce a clear block-diagonal similarity structure. QSO exhibits this structure, whereas HIQL also assigns high similarity between behaviour groups, and OPAL and absolute subgoal options produce low similarity within behaviour groups. How necessary is joint option shaping by both the high and low-level Q functions? Table 3 and Figures 3 and 9 indicate that joint shaping is important for both the learned action abstraction and downstream performance. Neither Q function alone recovers the clean semantic structure obtained when both are used. Figure 8 shows that shaping with the high-level only performs well in antmaze-giant but degrades significantly in cube-double. We hypothesise that for cube-double continued high-level shaping increasingly removes distinctions that are irrelevant to high-level value but necessary for fine-grained control; without the low-level Q function to preserve this information, the resulting abstractions become insufficient for the low-level policy. 8 Preprint, October 2026. Figure 3: PCA (top) of QSO’s action abstractions for gradient flow ablations for antmaze-giant fitted on random trajectory sequences, showing different groups of three k-step trajectory sequences with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) for those groups in units of median distance between two random trajectory sequences. Both high and low-level gradient flows are required to group functionally equivalent behaviours while separating the behaviour groups. Do the decoupled levels form distinct state abstractions? For the high and low-level decision processes, we measure how much each network’s output varies with slow and fast moving dimensions; a larger percentage share of the output variance being associated to a particular dimension means that the network is more dependent on that dimension. We would expect the low-level decision process to focus more on high frequency (fast-moving) dimensions, and the high-level decision process on low frequency (slow-moving) ones. As expected, the two levels focus on different states (Table 8 in Appendix H). For example, the high-level state-value function depends almost entirely on slow-moving dimensions (88% for antmaze-giant and 74% for cube-double), while the low-level policy depends much more on fast-moving ones (59% and 77% respectively). We refer to Appendix H for further discussion. DefaultQ Shaping AblationOption Resampling Ablation EnvironmentQSOLow-Level OnlyHigh-Level OnlyEvery Step 5% (2 Steps) 25% (7 Steps) 50% (13 Steps) antmaze-giant-stitch-v068 [66, 70]38 [38, 40]69 [67, 72]68 [66, 70] 68 [66, 70] 62 [61, 64] 54 [47, 60] cube-double-play-v0 70 [65, 74]70 [67, 73]47 [42, 50]70 [65, 74] 69 [63, 73] 50 [44, 55] 14 [10, 18] Table 3: Q shaping and option resampling ablations of QSO. Success rate averaged over 5 evaluation tasks after 1M training steps. Mean and 95% bootstrap confidence interval over 4 seeds. 9 Preprint, October 2026. Figure 4: QSO’s sampled options gradually shift across the option space during deployment. L2 distance between two consecutively sampled options (every 10 steps), whitened against the distance between two randomly sampled options. Dotted blue shows the distance between two random trajectory sequence embeddings whitened against the distance between two randomly sampled options. Mean and 95% confidence interval over 4 seeds, cut off where the third quartile of successful episodes has ended. How does the option vary during deployment? Figure 4 shows that sampled options gradually shift across the option space during deployment: the distance between consecutively sampled options is roughly 30% of that between two randomly sampled options in antmaze-giant, and 25% of that in cube-double. Although consecutive options may partly be similar due to smoothness of the high-level policy, the above analysis showing that option-space proximity reflects semantic similarity suggests that it also corresponds to gradual changes in intent. The distance between two random trajectory sequence embeddings whitened against the distance between two randomly sampled options (dotted blue) being approximately 1 confirms that the small consecutive distances reflect genuinely slow variation rather than a collapsed policy. Table 3 ablates how long the option is held for at deployment: the rapid decrease in performance for cube-double suggests that the option encodes significant information about the immediate next action. We provide ablations on the option representation input and dimension, goal sampling, reward labelling, policy extraction, and temporal horizon in Appendix H. 6DISCUSSION AND FUTURE WORK We introduce QSO, which, unlike prior algorithms, learns a control-sufficient action abstraction that discards unnecessary distinctions between options. Future work will extend QSO to do targeted exploration at the option-level in online RL. A limitation of QSO arises in stochastic environments: by representing an option retrospectively, the high-level critic evaluates the option conditioned on its realised outcome, which can lead to optimistic bias. Although we note that this issue is shared by all hindsight-based hierarchical methods including HIQL, OPAL, SHARSA and DSHARSA, future work could address this through progress-adaptive high-level control. ETHICS STATEMENT This work aims to advance machine learning, specifically hierarchical reinforcement learning, and is evaluated only in simulated environments using public datasets. It may have broader societal consequences, but none of them must be specifically highlighted here. ACKNOWLEDGEMENTS The authors would like to thank Darius Muglich for helpful advice and comments on the project. Clarisse Wibault is funded by EPSRC grant EP/W524311/1. Antoine Gorceix is funded by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines grant EP/Y035070/1 and Iconic Interactive Ltd. Alexey Zakharov is supported by the Clarendon Fund in partnership with the Oxford-Basil Reeve Graduate Scholarship at the University of Oxford. Eduardo Pignatelli is funded by University College London. Jakob Foerster is partially funded by the UKRI grant EP/Y028481/1, as well as being supported by the JPMC and Amazon Research Awards. 10 Preprint, October 2026. REPRODUCIBILITY STATEMENT Our code is open sourced athttps://github.com/CWibault/QSO. Hyperparameters and exact implementation details can be found in Appendices G and H. AI DISCLOSURE We used AI to render figures, debug code, and check spelling and grammar, but all ideas, analysis, and written content are our own. We take full responsibility for the final submission. REFERENCES Leslie Pack Kaelbling. Learning to achieve goals. In Ijcai, volume 2, pages 1094–8, 1993. Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312–1320. PMLR, 2015. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999. Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. Advances in Neural Information Processing Systems, 38:8350–8389, 2026. Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(4):341–379, 2003. George Konidaris. On the necessity of abstraction. Current opinion in behavioral sciences, 29:1–7, 2019. George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learn- ing symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215–289, 2018. Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020. Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36:34866– 34891, 2023. Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. AI&M, 1(2):3, 2006. Balaraman Ravindran and Andrew G Barto. Model minimization in hierarchical reinforcement learning. In International symposium on abstraction, reformulation, and approximation, pages 196–211. Springer, 2002. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art, pages 45–73. Springer, 2012. Qiyang Li, Zhiyuan Paul Zhou, and Sergey Levine. Reinforcement learning with action chunking. Advances in Neural Information Processing Systems, 38:55518–55553, 2026. Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. 11 Preprint, October 2026. Richard S Sutton, Andrew G Barto, and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017. Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th international conference on autonomous agents and multiagent systems-volume 2, pages 761–768, 2011. Adam White, Joseph Modayil, and Richard S Sutton. Scaling life-long off-policy learning. In 2012 ieee international conference on development and learning and epigenetic robotics (icdl), pages 1–6. IEEE, 2012. Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, volume 2025, pages 94937–94982, 2025. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682–3691. PMLR, 2021. Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021. Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? Advances in Neural Information Processing Systems, 37:79029–79056, 2024. Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279–292, 1992. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. Pmlr, 2014. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587–1596. Pmlr, 2018. David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, pages 2915–2923. PMLR, 2016. Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. arXiv preprint arXiv:1207.4114, 2012. Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020. 12 Preprint, October 2026. R ́ emi Munos and Csaba Szepesv ́ ari. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008. Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International conference on machine learning, pages 1042–1051. PMLR, 2019. Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003. Tor Lattimore and Marcus Hutter. Pac bounds for discounted mdps. In International Conference on Algorithmic Learning Theory, pages 320–334. Springer, 2012. Arnaud Robert, Ciara Pike-Burke, and Aldo Faisal. Sample complexity of goal-conditioned hierarchi- cal reinforcement learning. Advances in Neural Information Processing Systems, 36:62696–62712, 2023. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Dan Hendrycks and Kevin Gimpel.Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016. 13 Preprint, October 2026. AEXTENDED PRELIMINARIES Value LearningTo learn our state-value and Q functions, we adopt Implicit Q-Learning (Kostrikov et al., 2021), a conservative value learning algorithm that evaluates the state-value function under the dataset action that leads to the maximum value rather than evaluating the Q-value under the current policy or using the maximum over actions, as is typically done in online RL (Watkins and Dayan, 1992; Mnih et al., 2015). This avoids querying the Q function on out-of-distribution actions, to which the Q function can assign erroneously high values (Levine et al., 2020). The state-value function and critic are trained with the following losses: L V = E (s t ,a t ,g)∼D h ℓ 2 ν V (s t ,g)− ̃ Q(s t ,g,a t ) i , L Q = E (s t ,a t ,s t+1 ,g)∼D h (Q(s t ,g,a t )− r(s t ,g)− γV (s t+1 ,g)) 2 i , whereℓ 2 ν denotes the expectile lossℓ 2 ν (x) :=| ν− 1· (x < 0)|·x 2 . Forν > 0.5, underestimating a high Q value is penalised more strongly than overestimating a low one, soV (s t ,g)is pulled to the upper end of the dataset Q values. Policy Extraction To extract our policies, we use Advantage Weighted Regression (AWR) (Peng et al., 2019) and Rejection Sampling (RS) (Ghasemipour et al., 2021), which are conservative policy extraction algorithms that constrain the policy to sample the highest quality actions within the dataset. As the name suggests, AWR does advantage-weighted behaviour cloning of dataset actions: L π =−E (s t ,a t ,g)∼D exp Q(s t ,g,a t )− V (s t ,g) α · logπ(a t | s t ,g) , while RS learns a behaviour-cloned policy to imitate the dataset and then defines the policy by choosing the action that maximises the Q function: π := argmax a t ∈a i t ,a i t ∼π BC (·|s t ,g) Q(s t ,g,a i t ). Unlike algorithms that optimise the policy by differentiating through the learned Q-function (Silver et al., 2014; Lillicrap et al., 2015; Fujimoto et al., 2018; Fujimoto and Gu, 2021), these conservative policy extraction algorithms avoid a direct source of Q function exploitation, whereby errors in the learned Q values are propagated into the policy. However, a non-hierarchical policy extracted with AWR or RS is bounded by the quality of the data at that state: where a region of the state space contains only low-quality actions, there is nothing “better” among the proposals to upweight. 14 Preprint, October 2026. BLOW-LEVEL Q-SHAPING IMPLIES CONTROL SUFFICIENCY Proof 1. Low-level Q-shaping implies control sufficiency. Ifφ ω (s,g s ) = φ ω (s,g ′ s ) =: ω, then for everya,Q ⋆ (s,g s ,a) = Q l (s,ω,a) = Q ⋆ (s,g ′ s ,a), so the two targets induce the same action-value function atsand hence the same set of maximisers. Takingπ l (· | s,ω)to be any distribution supported onargmax a Q l (s,ω,a)gives the policy required by Definition 1. 15 Preprint, October 2026. CCONSERVATIVE EXTRAPOLATION To provide intuition on how joint state and action abstractions can enable conservative extrapolation, consider an agent learning a locomotion task, where its state can be decomposed into the agent’s pose and coordinate s = (p,c). Imagine three state-goal pairs: (s 1 ,g,a 1 ) = (p,c) | z s 1 , g,a 1 ,(s 2 ,g,a 2 ) = (p ′ ,c) | z s 2 , g,a 2 ,(s 3 ,g ′ ,a 3 ) = (p,c ′ ) | z s 3 , g ′ ,a 3 . The first state-goal pair is the one that the agent must act at, and its dataset action is poor (it might just dither on the spot), while the dataset actions at the second and third pair are good. Imagine that in all situations, the agent must move two metres down to reach its respective goal because the goals lie in the same direction from their respective coordinates. A non-hierarchical policy extracted with AWR or RS never learns a useful action at(s 1 ,g)because the only action the dataset offers there is the poor one. We now show how a hierarchical policy with abstract relative options (such as stay “on the spot” or reach “a point two metres down”) can output a good action for the first state-goal pair using conservative extrapolation (where each level only selects among actions its own data supports). The second state-goal pair shares the coordinate and goal of the first, so the high-level abstraction (which ignores low-level pose information) groups them. This means that the high-level decision is rescued by the second state-goal pair: conservative extraction has the good option to reach “a point two metres down” available from the second state-goal pair, rather than only to stay “on the spot” from the first: φ h (s 1 ,g) = φ h (s 2 ,g) = (c,g) π h (ω | s 1 ,g) = π h (ω | s 2 ,g) = δ ω=ω down Note that the second pair supplies the right option, but its action is executed from a different pose, so it cannot be copied directly. Similarly, the third state-goal pair then shares the pose and option of the first, so the low-level abstraction (which ignores high-level coordinate information) groups them. The low-level decision is then rescued by the third state-goal pair: conservative extraction has the good actiona 3 available from the third pair, rather than only a 1 from the first: φ l (s 1 ,ω down ) = φ l (s 3 ,ω down ) = (p,ω down ) π l (a| s 1 ,ω down ) = π l (a| s 3 ,ω down ) = δ a=a 3 Under the hierarchical policy, the agent therefore takesa 3 at(s 1 ,g), an action never observed for that state-goal pair. Note that this example hinges on the learning of abstract relative options so that reach “a point two metres down” is a single option rather than, for example, two coordinate-specific ones. 16 Preprint, October 2026. DOPTION SPACE COMPRESSION BOUNDS OFFLINE ERROR Proposition 2. Offline error under joint state and action abstraction. For a finite-state, finite-action MDP with at most two possible next-state transitions, with probability of at least1−δ, the suboptimality in expected return,ε := max s∈S,g∈G V ⋆ (s,g)− V π (s,g) of a goal-conditioned hierarchical policy learned from a dataset ofNtransitions satisfies ε∝ s |C h ||Ω|· ρ rep h (1− γ k ) 3 N + r |C l ||A|k 3 · ρ rep l N , whereφ h : S ×G → C h aggregates state-goal pairs requiring the same option such thatC h denotes the state-abstracted high-level space, andφ l :S× Ω→C l aggregates state-option pairs requiring the same action such thatC l denotes the state-abstracted low-level space. Constants and logarithmic factors are suppressed andρ rep h andρ rep l are the concentrability coefficients at each level: ρ rep h =sup c h ∈C h ,ω∈Ω d π ⋆ h (c h ,ω) d π BC h (c h ,ω) , ρ rep l =sup c l ∈C l ,a∈A d π ⋆ l (c l ,a) d π BC l (c l ,a) , whered π is underlying discounted occupancy measure of a policyπand occupancy measures are aggregated over each abstraction: d π h (c h ,ω) = X (s,g)∈φ −1 h (c h ) d π h (s,g,ω), d π l (c l ,a) = X (s,ω)∈φ −1 l (c l ) d π l (s,ω,a). We refer to Appendices E and F for full definitions and derivations. The two terms respectively bound the high and low-level error terms; the concentrability coefficientsρ rep measure the distribution shift between the occupancy of the optimal policy and that of the dataset: as suprema of occupancy ratios, a single state-goal or state-option pair that the optimal policy visits but the dataset does not can cause the term to go to infinity, unless aggregation lets that pair borrow mass from equivalent ones. Compressing the option space by discarding unnecessary distinctions between options reduces the bound in three ways. It reduces the high-level action space|Ω|, tightens the bound|C l |≤|S||Ω| on the low-level state-abstraction space, and aggregates occupancy mass across a larger number of equivalent state-goal and state-option pairs, thereby reducing both concentrability coefficients (“conservative extrapolation”). In a continuous space, rather than making|Ω|as small as possible, this equates to mapping functionally equivalent behaviour to similar points in the option space (Abel et al., 2016; Ferns et al., 2012; Zhang et al., 2020), which reduces the effective complexities of both error terms (Munos and Szepesv ́ ari, 2008; Chen and Jiang, 2019). 17 Preprint, October 2026. ESTATE AND ACTION ABSTRACTION SAMPLE COMPLEXITY IN ONLINE GOAL-CONDITIONED RL In this section, we show how joint state and action abstraction can reduce the sample complexity for finding an optimal goal-conditioned policy such that the error in expected return is smaller than a fixed constant ε from N ∝ |S||G||A| (1− γ) 3 ε 2 , for a policy with no state or action abstraction, to N ∝ |C h ||Ω| (1− γ k ) 3 ε 2 + |C l ||A|k 3 ε 2 , for a policy with both state and action abstraction. We build on the works of Kakade (2003); Munos and Szepesv ́ ari (2008); Lattimore and Hutter (2012); Robert et al. (2023); Li et al. (2006). Sample Complexity in Online GCRLIn online GCRL, the aim is to find the optimal policy with the smallest number of samples or online interactions,N, such that the error in optimal return is smaller than a fixed constantε. Consider the case of a discrete, discounted horizon MDP with a finite state spaceS, action spaceAand discount factorγ ∈ [0, 1). The Probably-Approximately Correct (PAC) Learning (Kakade, 2003) upper-bound sample complexity to find anεoptimal policy reaching a unique goal-state optimally (assuming at most two possible next-states for each state/action pair) with constant probability of 1− δ is given by Lattimore and Hutter (2012): N ∝ |S||A| (1− γ) 3 ε 2 . Hence, the minimax sample complexity required to find anεoptimal policy reaching any given state (such that we have|G| unique goal-states) is given by: N ∝ |S||G||A| (1− γ) 3 ε 2 .(1) Consider the case of a discrete finite horizon (of lengthH) MDP with a finite state spaceS, action spaceA and discount factor γ ∈ [0, 1). The minimax sample complexity to find an ε optimal policy reaching a unique goal-state optimally is given by: N ∝ |S||G||A|H 3 ε 2 . Action Abstraction Sample Complexity in Online GCRL Using a hierarchical policy, we can break the distant goalgfrom our current statesinto options defined over an option spaceΩ. Hence, the high-level policy becomes N ∝ |S||G||Ω| (1− γ k ) 3 ε 2 , where we have substituted the option-space to Equation 1, and use the environment’s discount factor raised to a factor of k, assuming that the high-level policy acts every k steps. The low-level policy has a sample complexity of N ∝ |S||Ω||A|k 3 ε 2 , since|Ω|options can be executed. Hence, the overall sample complexity of the hierarchical policy is (Robert et al., 2023): N ∝ |S||G||Ω| (1− γ k ) 3 ε 2 + |S||Ω||A|k 3 ε 2 .(2) Note that in the case of no temporal abstraction, whenk = 1, we approximately recover the original sample complexity of a non-hierarchical policy. Since the option just becomes a single primitive action, Equation 2 becomes: N ∝ |S||G||A| (1− γ) 3 ε 2 + |S||A| 2 ε 2 ≈ |S||G||A| (1− γ) 3 ε 2 , 18 Preprint, October 2026. where the approximation follows due to the dominating1/(1− γ) 3 factor in the first term, assuming long-horizon problems such thatγ → 1and that the goal-space is larger or equal to the size of the action space|G|≥|A|. Even though this might not be the case, usually the goal-space is unknown, so we train the policy to reach any state within the state-space i.e. such thatG =S. Assuming that |S|≫|A| is a standard assumption. In most real-world settings we expect|Ω| ≪ |A| k . The option space does not have to perfectly recreate the sequence of low-level actions, only contain the sufficient information to approximate the high-level value and choose the low-level action. State and Action Abstraction Sample Complexity in Online GCRLFinally, since the high-level decision process can discard irrelevant low-level information, and the low-level decision process can discard irrelevant high-level information, state abstraction reduces the effective complexity of the state space at each level of the hierarchy such that Equation 2 becomes: N ∝ |C h ||Ω| (1− γ k ) 3 ε 2 + |C l ||A|k 3 ε 2 ,(3) where|C h |is the effective state-space required for the high-level decision process and|C l |is the effective state-space for the low-level decision process. By definition,|C h |≤|S||G|and|C l |≤|S||Ω|. 19 Preprint, October 2026. FSTATE AND ACTION ABSTRACTION ERROR IN OFFLINE GOAL-CONDITIONED RL In offline GCRL, the aim is to bound the error ε given an offline datasetD of fixed size N . Unlike in the online setting, where it is assumed that the agent can sample any state-action pair to learn the environment’s dynamics, in offline RL, a concentrability coefficientρis incorporated to account for the distribution shift in data collected by the policyπ BC , and the data that would have been collected induced under the optimal policyπ ⋆ . Intuitively, if the dataset does not include the states required to learn the optimal policy, the algorithm may never learn that optimal policy. The concentrability coefficient ρ is defined as: ρ =sup s∈S,g∈G,a∈A d π ⋆ (s,g,a) d π BC (s,g,a) , where d π (s,g,a) is the discounted occupancy measure (i.e. stationary distribution) of the policy π: d π (s,g,a) = (1− γ)E τ∼π,s 0 ,g 0 ∼Unif(G) " ∞ X t=0 γ t 1(s t = s,a t = a,g 0 = g) # , and where the trajectory is induced by the MDP and following the policyπ(a| s,g). If the dataset is highly exploratory and covers the optimal paths well,ρwill be small. If the dataset is narrow, or misses critical regions of the state-action space, ρ will be large. Rearranging Equation 1 and incorporating the concentrability coefficient, the offline bound for a goal-conditioned non-hierarchical policy is given by: ε∝ s |S||G||A|· ρ (1− γ) 3 N . Action Abstraction Error in Offline GCRLSince the size of the offline dataset is fixed, one way to reduce the error ε is to use a hierarchical policy with distinct state-goal representations. Using a hierarchical policy, the error term becomes: ε∝ s |S||G||Ω|· ρ h (1− γ k ) 3 N + r |S||Ω||A|k 3 · ρ l N , where the concentrability coefficients are defined as: ρ h =sup s∈S,g∈G,ω∈Ω d π ⋆ h (s,g,ω) d π BC h (s,g,ω) and ρ l =sup s∈S,ω∈Ω,a∈A d π ⋆ l (s,ω,a) d π BC l (s,ω,a) . State and Action Abstraction Error in Offline GCRL Then, ignoring irrelevant low-level in- formation for the high-level decision process and high-level information for the low-level decision process, the error term is bounded by: ε∝ s |C h ||Ω|· ρ rep h (1− γ k ) 3 N + r |C l ||A|k 3 · ρ rep l N . Again,|C h |≤|S||G| and|C l |≤|S||Ω|. Because the concentrability coefficients are now defined over the constrained spacesC h andC l , i.e. ρ rep h =sup c h ∈C h ,ω∈Ω d π ⋆ h (c h ,ω) d π BC h (c h ,ω) , ρ rep l =sup c l ∈C l ,a∈A d π ⋆ l (c l ,a) d π BC l (c l ,a) , the probability mass of the offline dataset is aggregated across state-goal or state-option pairs that are equivalent under this information loss (i.e. for which c h = φ h (s,g) and c l = φ l (s,ω)): d π h (c h ,ω) = X (s,g)∈φ −1 h (c h ) d π h (s,g,ω), d π l (c l ,a) = X (s,ω)∈φ −1 l (c l ) d π l (s,ω,a). This reduces the likelihood of a support mismatch, where the optimal policy requires a state transition on which the dataset places zero mass. Hence, ρ rep h ≤ ρ h and ρ rep l ≤ ρ l . 20 Preprint, October 2026. GQ-SHAPED OPTIONS G.1ACTION ABSTRACTION Joint Action Abstraction and Value Learning The action abstractions are learned jointly with the high-level and low-level Q functions, acting as a goal-space for the low-level Q function and an action space for the high-level Q function. The high-level value and critic losses are given by the following IQL loss: L V h = E (s t ,s t+k ,g)∼D h ℓ 2 ν V h (s t ,g)− ̃ Q h s t ,g, ̃ φ ω (s t ,s t+k ) i (4) L Q h ,φ ω = E (s t ,s t+k ,g)∼D " Q h s t ,g,φ ω (s t ,s t+k ) − R h (s t:t+k ,g)− γV h (s t+k ,g) 2 # ,(5) where the normalised k-step option return is R h (s t:t+k ,g) = 1− γ 1− γ k k−1 X t ′ =0 γ t ′ r h (s t+t ′ ,g). The low-level value and critic losses are given by the following IQL loss: L V l = E (s t ,a t ,g s )∼D h ℓ 2 ν V l (s t ,φ ω (s t ,g s ))− ̃ Q l s t , ̃ φ ω (s t ,g s ),a t i (6) L Q l ,φ ω = E (s t ,a t ,s t+1 ,g s )∼D ω t+1 =φ ω (s t+1 ,g s ) " Q l s t ,φ ω (s t ,g s ),a t − r l (s t ,g s )− γV l (s t+1 ,ω t+1 ) 2 # .(7) Continuous Option SpaceWe use a continuous option space encoded using a feed-forward MLP followed by length-normalisation so that the options lie on a hypersphere of radius √ d ω inR d ω , preventing any proposed option from being significantly out-of-distribution for the low-level policy: φ ω (s,g) = p d ω f ω (s,g) ∥f ω (s,g)∥ 2 , wheref ω :S×G → R d ω denotes the feed-forward MLP encoder. At deployment, the options are continually resampled at every timestep. DiscountingThe same discount factorγis used at each level of the hierarchy, so that each level has the same effective horizon. For a two-level hierarchy, this means that the low-level decision process looks1/(1− γ)actions ahead, and the higher-level decision process looks1/(1− γ)options ahead. Since one high-level option corresponds to k low-level actions, we would ideally set k = 1 1− γ , so that the low-level decision process reasons over exactly the span of a single option. The effective horizons then compose multiplicatively: in other words, the higher level decision process coversk 2 actions, while only reasoning over 1/(1− γ) decisions. In practice (and like prior work on hierarchical RL and QC) we find thatkis bounded by the data. Since options are sampled ask-steps within a single trajectory,kmust be small relative to the trajectory lengthHfor the dataset to contain enough distinct options. For example, with our target discount factor of0.99, we would want to use100-step options, but antmaze-giant’s trajectories are only 200 steps. Reward CalibrationAccumulated option rewards are normalised so that a full-length option costs the same as a single action in the low-level decision process. This means that the Q functions work on similar scales, preventing one Q function from dominating the other in magnitude (bypassing the need to introduce any hyperparameters), while options terminating early incur a proportionally smaller cost. The accumulated option reward is normalised by 1− γ 1− γ k , so that a full-length option (i.e.β(s t+t ′ ,g) = 0fort ′ ∈0,· ,k− 1) costs exactly−1, identical to the cost of a single action in the low-level decision process. 21 Preprint, October 2026. G.2STATE ABSTRACTIONS Rather than introducing an explicit representation-learning objective, we induce meaningful state abstractions through level-specific datasets. At the high-level, we sample a broad mixture of local, trajectory, and random goals, encouraging it to generalise across a broad combinatorial coverage of state-goal pairs; at the low-level, we predominantly sample nearby trajectory target states, encouraging precise control over the option horizon. We replace the stricter reward-labelling criterion used by prior work (Park et al., 2025), which requires an exact match with the goal on every state dimension, with the environment success criterion at the high-level, avoiding distinctions between states that are equivalent for the task. Note that, to isolate the effect of action abstraction and because we find that it improves all hierarchical baselines (Figure 13 in Appendix H) we apply this change consistently in our experiments. At the low-level, where we want to focus on fine-grained control, we retain the stricter criterion. We ablate using the environment criterion at both levels: in Appendix H, Figure 11 shows performance ablating high-level goal sampling, while Figures 12, 15 and 16 show performance and option embeddings when ablating reward labelling. Table 8 quantifies state abstraction. Goal Sampling During training, goals are sampled according to the following, whereτ = (s 0 ,...,s H ) denotes theH-length trajectory containings t andUnif D denotes a uniform distri- bution over all states in the dataset, and the weightswof each of the three sampling components sum to 1. As usual, k denotes the fixed option length. High-level Decision Process For the high-level decision process, goals are sampled according to p h (g | s t ) = w opt h p opt h (g | s t ) + w traj h p traj h (g | s t ) + w rand h Unif D (g),(8) where the first component samples a goal uniformly from within the option window, p opt (g | s t ) =U min(t + 1, H), ..., min(t + k− 1, H) , the second component samples the goal uniformly from the remainder of the trajectory beyond the option window, p traj h (g | s t ) =U min(t + k, H), ..., H , and the third samples goals uniformly from the entire dataset. The first component ensures that a fraction of every batch returns success, the second component provides goals that are distant but reachable by dataset stitching, and the third component provides examples of the out-of-distribution combinations of state-goal pairs that the high-level policy will face at deployment. Low-level Decision Process For the low-level decision process, goals are sampled according to p l (g s | s t ) = w cur l δ s t (g s ) + w traj l p traj l (g s | s t ) + w rand l Unif D (g s ),(9) where the first component places all mass on the current state, the second geometrically samples a state a number of steps ahead, p traj l (g s | s t ) : g s = s min(t+κ, H) , κ∼ Geom 1 k , so that the expected look-ahead is k steps and the third samples uniformly from the entire dataset. The first component again ensures that a fraction of every batch returns a success, the second component acts in accordance with the option length of the high-level decision process, and the third component calibrates the value functions across the state space, making the learned values neater to visualise, but is unnecessary for algorithm functioning, since, even at deployment, the low-level value functions are never queried at such states. Reward LabellingWe use the environment success criterion (goal-achievement indicator) for the high-level β h (s,g) = β(s,g), specified per-environment in Appendix H. For the low-level, we follow prior work and use a stricter success criterion that requires an exact match on every state dimension: β l (s,g s ) = 1s = g s . 22 Preprint, October 2026. G.3POLICY EXTRACTION The high-level policy is defined using Rejection Sampling of an advantage-weighted behaviour cloning policy, and the low-level simply as a behaviour cloning policy: π h (ω | s t ,g) := argmax ω∈ω i ,ω i ∼π AWR h (·|s t ,g) Q h (s t ,g,ω i ), π l (a t | s t ,ω) := π BC l (a t | s t ,ω) High-Level Policy Extraction The high-level policy is fitted using the following advantage- weighted flow-matching loss: L π AWR h = E (s t ,s t+k ,g)∼D, z∼N (0,I d ω ) u∼Unif([0,1]), ω=φ ω (s t ,s t+k ), ω u =(1−u)z+uω " exp Q h (s t ,g,ω)− V h (s t ,g) α ! · v π AWR h s t ,g,u,ω u − (ω− z) 2 2 # . (10) Low-Level Policy ExtractionThe low-level policy is fitted using the following behaviour-cloning flow-matching loss: L π BC l = E (s t ,a t ,s t+k )∼D, z∼N (0,I d ) u∼Unif([0,1]), a u =(1−u)z+ua t h v BC π l s t ,φ ω (s t ,s t+k ),u,a u − (a t − z) 2 2 i .(11) 23 Preprint, October 2026. HEXPERIMENTS H.1ENVIRONMENTS EnvironmentState Dim.Action Dim.Data Trajectory LengthDataset SizeMax. Episode Length antmaze-giant-stitch-v02982001M1000 antmaze-teleport-stitch-v02982001M1000 humanoidmaze-giant-stitch-v069214004M4000 cube-double-play-v037510001M500 cube-triple-play-v046510003M1000 cube-quadruple-play-v055510005M1000 puzzle-3x3-play-v055510001M500 puzzle-4x4-play-v083510001M500 puzzle-4x5-play-v099510003M1000 puzzle-4x6-play-v0115510005M1000 scene-play-v040510001M750 Table 4: OGBench environment and dataset characteristics. Maze Environments For antmaze-giant, antmaze-teleport and humanoidmaze-giant, high-level success is defined as the agent lying withinεof the goal’s xy coordinates, wherex(s)∈ R 2 denotes a states’s xy coordinates: β(s,g) = 1∥x(s)− x(g)∥ 2 ≤ ε. Cube Environments For cube-double, cube-triple and cube-quadruple, high-level success is defined as every cube being withinεof its goal xyz position, wherec i (s)∈ R 3 denotes cubei’s xyz coordinates in states, andn is the number of cubes in the environment: β(s,g) = n Y i=1 1∥c i (s)− c i (g)∥ 2 ≤ ε. Puzzle Environments For puzzle-3x3, puzzle-4x4, puzzle-4x5 and puzzle-4x6, high-level success is defined as every button matching its goal configuration, whereb i (s)∈ R 3 denotes buttoni’s configuration in states, andn is the number of buttons in the environment: β(s,g) = n Y i=1 1b i (s) = b i (g). Scene Environment For scene-play, high-level success is defined as then c cubes, drawer and window being withinε c ,ε d and ε w of their goal positions and the n b buttons matching their goal configurations: β(s,g) = n c Y i=1 1∥c i (s)− c i (g)∥ 2 ≤ ε c · n b Y i=1 1b i (s) = b i (g) · 1|d(s)− d(g)|≤ ε d · 1|w(s)− w(g)|≤ ε w . 24 Preprint, October 2026. H.2BASELINES H.2.1ALGORITHMS Absolute Subgoal Options: DSHARSA For DSHARSA (Park et al., 2026), both the high and low-level value and critic functions are learned using IQL: L V h = E (s t ,s t+k ,g)∼D h ℓ 2 ν V h (s t ,g)− ̃ Q h (s t ,g,s t+k ) i , L Q h = E (s t ,s t+k ,g)∼D Q h (s t ,g,s t+k )− k−1 X t ′ =0 γ t ′ r h (s t+t ′ ,g)− γ k V h (s t+k ,g) ! 2 , and L V l = E (s t ,a t ,g s )∼D h ℓ 2 ν V l (s t ,g s )− ̃ Q l (s t ,g s ,a t ) i , L Q l = E (s t ,a t ,s t+1 ,g s )∼D h (Q l (s t ,g s ,a t )− r l (s t ,g s )− γV l (s t+1 ,g s )) 2 i , The high-level and low-level policies are fitted using behaviour-cloning flow-matching losses: L π BC h = E (s t ,s t+k ,g)∼D, z∼N (0,I d s ) u∼Unif([0,1]), g s =s t+k , g u s =(1−u)z+ug s " v π BC h s t ,g,u,g u s − (g s − z) 2 2 # , and L π BC l = E (s t ,a t ,s t+k )∼D, z∼N (0,I d ) u∼Unif([0,1]), a u =(1−u)z+ua t " v π l s t ,s t+k ,u,a u − (a t − z) 2 2 # . At deployment, both the high and low-level policies are defined with Rejection Sampling: π h (g s | s t ,g) := argmax g s ∈g i s ,g i s ∼π BC h (·|s t ,g) Q h (s t ,g,g i s ), π l (a t | s t ,g s ) := argmax a t ∈a i t ,a i t ∼π BC l (·|s t ,g s ) Q l (s t ,g s ,a i t ). Absolute Subgoal Options: SHARSA SHARSA (Park et al., 2026) is almost identical to DSHARSA, but only uses Rejection Sampling for the high-level policy. The low-level policy is defined using pure behaviour cloning: π h (g s | s t ,g) := argmax g s ∈g i s ,g i s ∼π BC h (·|s t ,g) Q h (s t ,g,g i s ), π l (a t | s t ,g s ) := π BC l (a t | s t ,g s ). Note that conservative extrapolation is still possible as the low-level policy might be queried at an absolute subgoal option (proposed by the high-level policy) that was unseen during training. Absolute Subgoal Options: ASOAs explained in Section 4, for QSO we use advantage-weighting to train the high-level policy. Therefore, as a baseline, we introduce a new variant of SHARSA that, like QSO, uses AWR to train the high-level policy: L π AWR h = E (s t ,s t+k ,g)∼D, z∼N (0,I d s ) u∼Unif([0,1]), g s =s t+k , g u s =(1−u)z+ug s " exp Q h (s t ,g,g s )− V h (s t ,g) α ! · v π AWR h s t ,g,u,g u s − (g s − z) 2 2 # . At deployment, the high-level policy is still defined using Rejection Sampling and the low-level with behaviour cloning: π h (g s | s t ,g) := argmax g s ∈g i s ,g i s ∼π AWR h (·|s t ,g) Q h (s t ,g,g i s ), π l (a t | s t ,g s ) := π BC l (a t | s t ,g s ). 25 Preprint, October 2026. Abstract Relative Options: HIQL HIQL (Park et al., 2023) shapes the abstract subgoal option space through its single, action-free state-value functionV (s,φ ω (s,g)), which is trained using Implicit V-Learning (IVL) (Park et al., 2025): L V,φ ω = E (s t ,a t ,g)∼D,g∼p D (·|s,a) h ℓ 2 ν V (s t ,φ ω (s t ,g))− r(s t ,g)− γ ̃ V (s t+1 , ̃ φ ω (s t+1 ,g)) i , where ̃ Vand ̃ φ ω denote the target network and representations andℓ 2 ν the expectile loss. HIQL extracts high and low-level Gaussian policies using AWR-like objectives: L π h =−E (s t ,s t+k ,g)∼D,ω=φ ω (s t ,s t+k ) exp V (s t+k ,g)− V (s t ,g) α · logπ h (ω | s t ,g) , L π l =−E (s t ,s t+k ,g)∼D,ω=φ ω (s t ,s t+k ) exp V (s t+1 ,s t+k )− V (s t ,s t+k ) α · logπ l (a t | s t ,ω) . Abstract Options: OPALOPAL (Ajay et al., 2020) shapes the abstract subgoal option space using a β-VAE trained to encode state-action trajectory sequences: min π l ,p ω ,q ω −E τ∼D,ω∼q ω (·|τ ) " k−1 X t=0 logπ BC l (a t | s t ,ω) # + β· D KL (q ω (ω | τ )|| p ω (ω | s 0 )), whereq ω andp ω respectively denote the option-space encoder and option-space prior. Using this learned option space to encode trajectory sequences, state-value and Q functions are trained using IQL: L V = E (τ,g)∼D,ω∼q ω (·|τ ) h ℓ 2 ν V (s t ,g)− ̃ Q(s t ,g,ω) i , L Q = E (τ,g)∼D,ω∼q ω (·|τ ) Q(s t ,g,ω)− k−1 X t ′ =0 γ t ′ r(s t+t ′ ,g)− γ k V (s t+k ,g) ! 2 , and a high-level flow-matching policy is fitted using the following behaviour-cloning flow-matching loss: L π BC h = E (τ,g)∼D, z∼N (0,I d ω ) u∼Unif([0,1]), ω∼q ω (·|τ ), ω u =(1−u)z+uω " v π BC h s t ,g,u,ω u − (ω− z) 2 2 # , At deployment, the high-level policy is defined using Rejection Sampling, and the low-level with behaviour cloning: π h (ω | s t ,g) := argmax ω∈ω i ,ω i ∼π BC h (·|s t ,g) Q h (s t ,g,ω i ), π l (a t | s t ,ω) := π BC l (a t | s t ,ω). In practice, the low-level policy is finetuned to output the immediate action. Non-hierarchical: IQL IQL (Kostrikov et al., 2021) trains a value and Q function with the following losses: L V = E (s t ,a t ,g)∼D h ℓ 2 ν V (s t ,g)− ̃ Q(s t ,g,a t ) i , L Q = E (s t ,a t ,s t+1 ,g)∼D h (Q(s t ,g,a t )− r(s t ,g)− γV (s t+1 ,g)) 2 i , We then extract a Gaussian policy by maximising the Deep Deterministic Policy Gradient objective (Fujimoto and Gu, 2021): L π =−E (s t ,a t ,g)∼D " Q s,g,μ π (s,g) + α logπ(a| s,g) # , whereμ π denotes the mean of the Gaussian policy. Following Park et al. (2024), we use DDPGBC rather than AWR for policy extraction. 26 Preprint, October 2026. Action Chunking: QC QC (Li et al., 2026) trains an action-chunked Q function: L Q = E (s t ,a t:t+k ,s t+k ,g)∼D a t+k:t+2k ∼π(·|s t+k ,g) " Q s t ,g,a t:t+k − k−1 X t ′ =0 γ t ′ r(s t+t ′ ,g)− γ k Q s t+k ,g,a t+k:t+2k 2 # . It fits a flow-matching policy using behaviour-cloning: L π BC = E (s t ,a t:t+k ,g)∼D, z∼N (0,I d ) u∼Unif([0,1]), a u =(1−u)z+ua t:t+k " v π l s t ,g,u,a u − (a t:t+k − z) 2 2 # . The policy is defined using Rejection Sampling: π(a t:t+k | s t ,g) := argmax a t:t+k ∈a i t:t+k ,a t:t+k i ∼π BC (·|s t ,g) Q s t ,g,a i t:t+k . H.2.2DATASET SAMPLING FOR NON-HIERARCHICAL AND ACTION CHUNKING BASELINES The dataset sampling intends to mirror the hierarchical ones as closely as possible, to enable the isolation of the hierarchical option space on performance as opposed to dataset sampling. Non-hierarchical: IQLThe flat baseline has no option structure, so there is no distinction between a near and a distant goal and the option window collapses to the current state. Goals are sampled according to: p flat (g | s t ) = w cur δ s t (g) + w traj p traj flat (g | s t ) + w rand Unif D (g), where the second component samples the goal uniformly from the remainder of the trajectory beyond the current state, p traj flat (g | s t ) =U min(t + 1, H), ..., H . and, like for hierarchical high-level dataset sampling, the third component samples goals uniformly from the entire dataset. We use the same weighting as for the hierarchical high-level dataset sampling as opposed to the low-level one, since the single non-hierarchical policy must cover the full range of state-goal possibilities with one distribution, in contrast to the hierarchical case where the near and far regimes can be split across two levels. Action Chunking: QCThe action chunking baseline is flat but temporally extended, so it inherits the high-level goal distribution of Equation 8 and additionally requires the length-kaction sequence beginning at s t , a t:t+k = a min(t,H) , ..., a min(t+k−1,H) , where indices are clipped to the end of the trajectory. If an agent reaches its target state before the end of theklength option chunk, actions past the target state are masked out and set to zero, so that the critic is not trained on behaviour occurring past termination. Following original implementations of non-hierarchical methods, we use strict reward labelling. 27 Preprint, October 2026. H.3HYPERPARAMETERS We use the following default hyperparameters for all value networks (both Value and Q functions), flow-matching and Gaussian policies, and Rejection Sampling methods. Default learning rate, batch size, target update rate, layer normalisation, non-linearities, and representation latent dimensions follow Park et al. (2025). According to Park et al. (2023), we use128for HIQL. Rejection Sampling Nfollows Park et al. (2026). Default network sizes and actor flow steps were chosen according to available compute time and memory. For Gaussian policies, following Park et al. (2023), we use a network that outputs a mean and constant standard deviation. We use the number of steps from Park et al. (2023), apart from for QC, where we use their suggestion of 5. Table 5: Hyperparameters. HyperparameterValue OptimiserAdam (Kingma and Ba, 2014) Learning Rate0.0003 Batch Size1024 Target Update Rate0.005 Representation Latent Dimension10 (default), 128 (HIQL) Representation Layer NormalisationTrue Representation Non-linearityGELU (Hendrycks and Gimpel, 2016) Representation MLP[512, 512, 512] Value Layer NormalisationTrue Value Non-linearityGELU (Hendrycks and Gimpel, 2016) Value MLP[512, 512, 512, 512] Value Expectile ν (IQL)0.7 Actor Layer NormalisationTrue Actor Non-linearityGELU (Hendrycks and Gimpel, 2016) Actor MLP[512, 512, 512, 512] Actor Flow Steps6 Actor Rejection Sampling N32 Actor AWR α3.0 Actor DDPGBC α1.0 (IQL default), 0.3 (IQL ant), 0.1 (IQL humanoid) Subgoal Steps k25 (default), 100 (humanoid), 5 (QC) Hierarchical Goal Sampling (w opt h ,w traj h ,w rand h )(0.1, 0.5, 0.4) Hierarchical Goal Sampling (w cur l ,w traj l ,w rand l )(0.1, 0.85, 0.05) Non-hierarchical Goal Sampling (w cur ,w traj ,w rand )(0.1, 0.5, 0.4) Discount γ0.99 (default), 0.995/0.999 (HIQL & IQL ant/humanoid) 28 Preprint, October 2026. Non-Hierarchical Hierarchical Environment Task IQL QCHIQLOPALSHARSADSHARSAASOQSO antmaze-giant-stitch-v0 Task 1 0 [0, 0] 0 [0, 0]35 [29, 39]35 [33, 37]59 [42, 71]72 [63, 86]57 [47, 67]57 [50, 63] Task 2 0 [0, 0] 0 [0, 0]66 [60, 70]83 [77, 90]74 [65, 87]67 [60, 73]67 [64, 69]75 [69, 79] Task 3 0 [0, 0] 0 [0, 0] 42 [38, 45]62 [52, 71]53 [39, 66]72 [66, 78]66 [58, 72]73 [66, 81] Task 4 0 [0, 0] 0 [0, 0]57 [50, 62]17 [12, 22]65 [57, 72]67 [57, 77]65 [60, 72]52 [43, 61] Task 5 0 [0, 0] 0 [0, 0]69 [53, 86]69 [65, 74]90 [85, 95]82 [80, 85]84 [70, 94]82 [78, 87] Overall 0 [0, 0] 0 [0, 0]54 [48, 60]53 [52, 55]68 [61, 76]72 [67, 77]68 [66, 70]68 [66, 70] antmaze-teleport-stitch-v0 Task 1 35 [27, 44] 27 [18, 33]44 [40, 48]61 [50, 72]18 [12, 28]28 [17, 38]54 [46, 65]34 [28, 40] Task 2 47 [42, 52] 38 [32, 43]44 [41, 48]85 [81, 91]77 [70, 83]87 [83, 93]75 [70, 80]75 [73, 78] Task 3 57 [52, 62] 40 [33, 47] 34 [27, 42]87 [82, 90]80 [76, 83]74 [71, 78]82 [78, 88]68 [53, 79] Task 4 51 [40, 60] 42 [36, 50]26 [13, 41]50 [47, 54]55 [51, 61]51 [45, 58]59 [51, 65]50 [45, 57] Task 5 48 [38, 57] 37 [27, 48]32 [26, 38]61 [52, 71]33 [26, 42]48 [38, 58]58 [50, 67]46 [43, 48] Overall 47 [44, 51] 37 [32, 40]36 [34, 39]69 [64, 74]53 [52, 54]58 [55, 62]66 [62, 69]54 [52, 57] humanoidmaze-giant-stitch-v0 Task 1 0 [0, 0] 0 [0, 0]5 [0, 10]73 [63, 83]12 [8, 18]17 [12, 22]22 [18, 28]82 [77, 89] Task 2 3 [1, 6] 0 [0, 0]18 [8, 27]78 [65, 89]41 [33, 51]47 [32, 69]32 [18, 43]98 [95, 100] Task 3 1 [0, 2] 0 [0, 0]6 [4, 7]85 [72, 92]42 [28, 61]61 [58, 65]55 [48, 62]91 [88, 93] Task 4 0 [0, 0] 0 [0, 0]2 [0, 3]78 [73, 83]11 [10, 12]8 [3, 12]19 [17, 22]85 [82, 87] Task 5 0 [0, 0] 0 [0, 0]1 [0, 2]95 [92, 98]43 [32, 61]17 [5, 28]92 [84, 98]97 [94, 99] Overall 1 [1, 1] 0 [0, 0]6 [4, 8]82 [78, 85]30 [27, 33]30 [27, 32]44 [38, 48]90 [88, 92] Table 6: Full results on selected OGBench locomotion environments using default datasets. Success rate (%) on each evaluation task after 1M offline training steps. Mean and 95% bootstrap confidence interval over 4 seeds. H.4FURTHER RESULTS Figure 5: Training on selected OGBench environments using the default datasets. Success rate averaged over 5 evaluation tasks. Mean and 95% bootstrap confidence interval over 4 seeds. 29 Preprint, October 2026. Non-Hierarchical Hierarchical Environment Task IQL QCHIQLOPALSHARSADSHARSAASOQSO cube-double-play-v0 Task 1 62 [53, 70] 90 [86, 93]26 [20, 32]82 [78, 87]95 [92, 98]98 [97, 100]96 [92, 99]99 [98, 100] Task 2 11 [3, 18] 88 [83, 96]2 [0, 5]12 [4, 18]82 [78, 86]93 [91, 96]88 [87, 90]68 [53, 78] Task 3 20 [16, 23] 88 [85, 90]4 [1, 8]11 [6, 15]89 [87, 92]91 [86, 95]85 [83, 88]68 [62, 76] Task 4 2 [0, 5] 42 [38, 45]2 [0, 3]15 [8, 22]41 [33, 48]59 [48, 68]20 [13, 27]53 [50, 57] Task 5 6 [3, 8] 54 [45, 63]2 [0, 3]25 [18, 32]53 [43, 67]58 [50, 64]48 [40, 57]61 [55, 69] Overall 20 [17, 23] 72 [70, 74]7 [5, 9]29 [26, 33]72 [68, 77]80 [75, 84]68 [65, 70]70 [65, 74] cube-triple-play-v0 Task 1 29 [27, 32] 57 [48, 68]58 [51, 65]8 [5, 12]45 [29, 58]44 [40, 48]16 [13, 18]45 [42, 48] Task 2 1 [0, 2] 31 [18, 42]7 [7, 7]1 [0, 2]15 [11, 21]16 [11, 23]2 [1, 3]28 [27, 30] Task 3 0 [0, 0] 20 [17, 23]12 [10, 13]2 [1, 3]5 [3, 7]5 [0, 10]1 [0, 2]17 [12, 22] Task 4 0 [0, 0] 4 [2, 7] 3 [0, 8]2 [0, 5]3 [1, 6]6 [2, 9]0 [0, 0]15 [12, 18] Task 5 0 [0, 0] 6 [4, 7]1 [0, 2]1 [0, 2]2 [0, 5]1 [0, 2]0 [0, 0]3 [3, 3] Overall 6 [6, 6] 24 [19, 27]16 [14, 18]3 [2, 4]14 [10, 18]14 [13, 16]4 [4, 4]22 [20, 23] cube-quadruple-play-v0 Task 1 0 [0, 0] 4 [3, 6]11 [5, 19]0 [0, 0]2 [0, 3]2 [0, 3]0 [0, 0]9 [8, 10] Task 2 0 [0, 0] 4 [3, 6]4 [2, 7]0 [0, 0]1 [0, 2]1 [0, 2]0 [0, 0]10 [7, 13] Task 3 0 [0, 0] 0 [0, 0]1 [0, 2]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]1 [0, 2] Task 4 0 [0, 0] 0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]2 [0, 5] Task 5 0 [0, 0] 0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]1 [0, 2] Overall 0 [0, 0] 2 [1, 2]3 [1, 5]0 [0, 0]0 [0, 1]0 [0, 1]0 [0, 0]4 [4, 5] scene-play-v0 Task 1 86 [78, 93] 100 [100, 100]55 [40, 65]99 [98, 100]100 [100, 100]100 [100, 100]100 [100, 100]100 [100, 100] Task 2 54 [49, 58] 98 [97, 99]59 [52, 71]92 [88, 98]98 [97, 100]96 [93, 98]98 [95, 100]99 [98, 100] Task 3 64 [57, 72] 97 [94, 99]22 [20, 23]80 [67, 88]98 [97, 100]100 [100, 100]100 [100, 100]82 [78, 88] Task 4 33 [23, 43] 65 [60, 70]47 [38, 55]76 [68, 80]85 [75, 92]84 [82, 87]86 [78, 93]75 [65, 82] Task 5 3 [1, 6] 52 [46, 56]22 [13, 30]70 [59, 78]78 [72, 85]64 [52, 77]85 [81, 91]69 [60, 80] Overall 48 [45, 51] 82 [81, 83]41 [35, 46]84 [78, 87]92 [90, 93]89 [87, 91]94 [91, 95]85 [82, 90] puzzle-3x3-play-v0 Task 1 100 [100, 100] 94 [82, 100]63 [57, 68]94 [90, 98]100 [100, 100]100 [100, 100]99 [98, 100]100 [100, 100] Task 2 100 [100, 100] 91 [75, 100]58 [42, 73]72 [59, 81]74 [55, 93]85 [79, 91]78 [70, 87]81 [73, 88] Task 3 100 [100, 100] 74 [25, 100]32 [30, 35]58 [52, 62]67 [58, 76]78 [70, 87]70 [68, 72]66 [59, 73] Task 4 99 [98, 100] 72 [24, 98]36 [32, 41]59 [57, 64]69 [58, 80]76 [67, 86]52 [40, 62]48 [38, 58] Task 5 100 [100, 100] 72 [25, 100]49 [43, 55]65 [55, 75]78 [68, 90]84 [78, 90]79 [75, 84]64 [62, 67] Overall 100 [100, 100] 81 [46, 99]48 [46, 49]70 [65, 74]78 [71, 87]85 [81, 90]76 [72, 80]72 [69, 75] puzzle-4x4-play-v0 Task 1 85 [77, 92] 94 [90, 98]0 [0, 0]70 [63, 81]85 [82, 88]76 [68, 83]81 [77, 85]96 [92, 100] Task 2 47 [33, 60] 0 [0, 0]1 [0, 2]63 [48, 78]72 [66, 78]67 [62, 70]68 [61, 75]92 [80, 100] Task 3 78 [64, 90] 32 [20, 43]0 [0, 0]65 [56, 78]74 [63, 83]78 [75, 80]60 [51, 76]92 [80, 100] Task 4 72 [64, 78] 0 [0, 0]1 [0, 2]52 [50, 53]71 [62, 80]68 [57, 78]71 [68, 75]89 [75, 100] Task 5 72 [56, 88] 0 [0, 0]0 [0, 0]72 [67, 78]68 [53, 82]73 [68, 82]67 [62, 73]92 [78, 100] Overall 71 [61, 78] 25 [23, 27]0 [0, 1]64 [61, 68]74 [67, 79]72 [69, 75]69 [65, 75]92 [81, 100] puzzle-4x5-play-v0 Task 1 68 [53, 76] 81 [63, 97]18 [13, 23]43 [39, 47]46 [38, 55]45 [40, 52]46 [36, 60]36 [29, 42] Task 2 0 [0, 0] 0 [0, 0]0 [0, 0]36 [27, 46]17 [10, 23]27 [18, 35]19 [7, 32]20 [15, 25] Task 3 0 [0, 0] 0 [0, 0]0 [0, 0]7 [2, 12]20 [12, 27]21 [18, 23]6 [3, 8]3 [0, 10] Task 4 0 [0, 0] 0 [0, 0]1 [0, 2]30 [23, 37]1 [0, 2]0 [0, 0]1 [0, 2]17 [5, 28] Task 5 0 [0, 0] 0 [0, 0]0 [0, 0]2 [0, 3]1 [0, 2]0 [0, 0]0 [0, 0]1 [0, 2] Overall 14 [11, 15] 16 [13, 19]4 [3, 5]24 [22, 25]17 [15, 20]18 [18, 19]14 [11, 18]15 [11, 20] puzzle-4x6-play-v0 Task 1 32 [20, 45] 80 [68, 92]20 [18, 22]39 [33, 43]39 [30, 49]33 [27, 38]38 [33, 43]45 [41, 51] Task 2 10 [3, 17] 0 [0, 0]5 [2, 8]34 [31, 38]21 [15, 28]30 [27, 34]14 [5, 24]37 [28, 50] Task 3 0 [0, 0] 0 [0, 0]0 [0, 0]8 [5, 10]7 [3, 11]14 [7, 22]4 [2, 7]8 [3, 13] Task 4 0 [0, 0] 0 [0, 0]0 [0, 0]2 [0, 5]1 [0, 2]1 [0, 2]0 [0, 0]1 [0, 2] Task 5 0 [0, 0] 0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0]0 [0, 0] Overall 8 [5, 10] 16 [14, 18]5 [4, 6]17 [15, 18]14 [10, 16]16 [14, 17]11 [8, 15]18 [15, 22] Average over 55 Tasks 29 [27, 30] 32 [29, 34]20 [20, 20]45 [44, 46]46 [44, 49]49 [48, 49]47 [46, 48]54 [51, 56] Table 7: Full results on selected OGBench manipulation environments using default datasets. Success rate (%) on each evaluation task after 1M offline training steps. Mean and 95% bootstrap confidence interval over 4 seeds. Last row averages each seed over all 55 tasks, locomotion and manipulation (Table 6). 30 Preprint, October 2026. H.5ANALYSIS AND ABLATIONS Figure 6: PCA (top) of the action abstraction for cube-double fitted on random segments, showing different groups of segments with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) in units of median distance between two random segments. QSO is the only algorithm that consistently groups functionally equivalent behaviours (Desideratum 3) while separating behaviour groups (Desideratum 2). 31 Preprint, October 2026. Figure 7: PCA (top) of the action abstraction for cube-quadruple fitted on random segments, showing different groups of segments with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) in units of median distance between two random segments. QSO is the only algorithm that consistently groups functionally equivalent behaviours (Desideratum 3) while separating behaviour groups (Desideratum 2). 32 Preprint, October 2026. Reward Labelling High-level Policy High-level Value Option Encoder Low-level Policy Agent High LowFastSlow Interaction/ Noise Fast Slow Interaction FastSlowInteraction Fast Slow Interaction/ Noise antmaze-giant-stitch-v0 QSO E S35 [31, 39]14 [13, 16]50 [47, 54] 6 [4, 7] 88 [86, 90] 7 [6, 7]45 [42, 47]45 [42, 47]11 [10, 11] 59 [57, 60] 4 [4, 4] 37 [36, 38] E E 28 [26, 30]18 [17, 19]54 [52, 55] 6 [4, 7] 88 [86, 90] 6 [6, 7]23 [21, 25]72 [70, 74]5 [5, 6] 55 [54, 57] 5 [4, 5] 40 [39, 41] S S34 [30, 38]14 [13, 15]52 [49, 56] 8 [6, 10] 84 [81, 87] 8 [7, 9]46 [43, 48]45 [43, 48]9 [9, 9] 54 [53, 56] 5 [4, 5] 41 [40, 42] S E29 [26, 32]19 [18, 20]52 [51, 54] 8 [6, 11] 84 [81, 87] 8 [7, 8]28 [25, 30]68 [65, 70]5 [5, 5] 56 [54, 57] 4 [4, 5] 40 [39, 41] ASO E S21 [18, 24]13 [12, 14]67 [64, 69] 5 [4, 6] 88 [86, 90] 7 [7, 8]------ 53 [51, 54] 5 [5, 5] 42 [41, 44] E E21 [18, 24]13 [12, 14]67 [64, 69] 5 [4, 6] 88 [86, 90] 7 [7, 8]------ 53 [51, 54] 5 [5, 5] 42 [41, 44] S S19 [16, 22]13 [12, 14]68 [66, 70] 8 [6, 10] 83 [80, 85] 9 [8, 10]------ 54 [53, 55] 5 [4, 5] 41 [40, 43] S E19 [16, 22]13 [12, 14]68 [66, 70] 8 [6, 10] 83 [80, 85] 9 [8, 10]------ 54 [53, 55] 5 [4, 5] 41 [40, 43] cube-double-play-v0 QSO E S73 [71, 74]5 [5, 5]22 [21, 24] 3 [2, 4] 74 [70, 77] 23 [20, 26]79 [77, 80]15 [14, 16]6 [6, 7] 77 [75, 78] 4 [4, 5] 19 [18, 20] E E25 [23, 27]23 [21, 24]52 [51, 53] 3 [3, 4] 75 [72, 78] 21 [19, 24]8 [8, 9]84 [83, 85]8 [7, 8] 71 [69, 72] 8 [7, 8] 22 [21, 23] S S71 [70, 72]7 [6, 7]22 [21, 23] 4 [3, 5] 70 [67, 72] 26 [24, 28]76 [75, 78]15 [14, 16]9 [8, 9] 81 [80, 83] 5 [4, 5] 14 [13, 15] S E18 [17, 20]39 [37, 41]42 [41, 44] 4 [3, 5] 73 [71, 76] 23 [21, 25]8 [7, 9]83 [82, 84]9 [9, 9] 75 [74, 77] 5 [4, 5] 20 [19, 21] ASO E S 26 [25, 27]52 [51, 53]22 [21, 23] 3 [3, 4] 75 [71, 78] 22 [19, 25]------ 88 [87, 89] 2 [2, 2] 10 [9, 11] E E 26 [25, 26]52 [51, 53]23 [22, 24] 3 [3, 4] 74 [70, 78] 22 [19, 26]------ 88 [87, 89] 2 [2, 2] 10 [9, 11] S S 26 [25, 27]52 [51, 53]22 [21, 23] 4 [3, 5] 66 [63, 69] 30 [28, 32]------ 88 [87, 89] 3 [2, 3] 9 [8, 10] S E26 [25, 27]52 [51, 53]22 [21, 23] 4 [3, 5] 66 [63, 69] 30 [28, 32]------ 88 [87, 89] 3 [2, 3] 9 [8, 10] Table 8: QSO and ASO high and low-level decision process state abstractions. Percentage share of the variance of each output explained by slow-moving dimensions (state and goal XY coordinates for antmaze-giant; state and goal cube positions and orientations for cube-double) and by the fast-moving dimensions (respectively state and goal XY height, orientation, joints and velocities; and state and goal effector, gripper, joints and velocities). The reward labelling columns indicate whether the high and low-level decision processes are trained using environment (E) or strict (S) reward labelling. Inputs are built on grids that combine the slow-moving dimensions of 16 random validation state-goal (or state-target) pairs with the fast-moving dimensions of 16 others, each under 8 flow-noise samples. For QSO, the state-target pairs are encoded before passing them to the low-level policy; the option encoder columns give the variance of these encoded options (ASO has no encoder). For ASO’s high-level policy and both ASO’s and QSO’s low-level policies, each dimension of the output is divided by the larger of its standard deviations over the dataset and over the sampled absolute subgoal options or actions. The variance is attributed exactly using two-way ANOVA, so that the attributions sum to 100. Mean and 95% bootstrap confidence interval (2000 resamples of the 64 grids) for one seed after 1M training steps. Further discussion on the coherence of state abstractions.The high-level policy remains depen- dent on fast-moving dimensions (with 35% of the output variance being explained by fast-moving dimensions in antmaze-giant and 73% in cube-double). We attribute this to our method of Q-shaping: because the option is shaped to retain information about the required immediate action, if the target’s fast-moving dimensions form part of the representation that the option encodes, the high-level policy inherits this dependence. The following ablation confirms this hypothesis. We ablate using the environment success criterion (which only focusses on slow-moving dimensions) rather than the stricter success criterion to also label the reward in the dataset for the low-level decision process. Doing so makes the option encoder and hence also high-level policy substantially less dependent on fast-moving dimensions: percentage shares for the option encoder drop from 45% to 23% and from 79% to 8% respectively, while for the high-level policy, they drop from 35% to 28% and 73% to 25%. Because we see no loss in downstream performance (Figure 12) and although we use the stricter success criterion for labelling reward in the low-level decision process across all experiments, we hypothesise that the environment success criterion might be preferable at both levels. 33 Preprint, October 2026. Figure 8: High and low-level critic Q-function shaping ablation. Figure 9: PCA (top) of QSO’s action abstractions for gradient flow ablations for cube-double fitted on random trajectory sequences, showing different groups of three k-step trajectory sequences with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) for those groups in units of median distance between two random trajectory sequences. Both high and low-level gradient flows are required to group functionally equivalent behaviours while separating the behaviour groups. 34 Preprint, October 2026. How sensitive is QSO to option representation input? Although our reasoning in Section 4 motivates representing options as relative achievements, we test whether this design choice improves downstream performance. Figure 10 compares performance with absolute achievementφ ω (s t+k )with relative achievement encodingsφ ω (s t ,s t+k ). Relative encoding substantially improves performance in antmaze-giant, but has little effect in cube-double. We hypothesise this to be because relative encoding provides the largest benefit when many absolute state-goal pairs(s,g s )can be collapsed into the same context-dependent achievement, which is the case for the locomotion task, but not necessarily for the manipulation one, where the above state-abstraction analysis suggests that the option retains much more fine-grained information for immediate control. Figure 10: φ ω (s t+k ) rather than φ ω (s t ,s t+k ) option encoding ablation. How sensitive is QSO to goal sampling? Both QSO and ASO are relatively robust to the proportion of random high-level goals (Figure 11). In both antmaze-giant and cube-double, all three sampling ratios converge to similar performance in both algorithms. Figure 11: Random goal sampling ablation for the high-level decision process for QSO (blue) and ASO (grey). 35 Preprint, October 2026. How sensitive is QSO to reward labelling? Environment reward labelling for the high-level decision process substantially improves performance for both QSO and ASO in cube-double, while on antmaze-giant the effect is small (Figure 12). For ASO, this gain comes from the high-level critic: ranking samples from the environment-labelled high-level policy using the strict-labelled high-level critic recovers the full performance (of 70% rather than 20%), while the reverse swap results in a corresponding drop in performance. We hypothesise that the environment reward labelling enables the high-level decision process to form better state abstractions: consistently, the percentage share of the output variance for slow-moving dimensions in the high-level state-value function increases by almost 10% (66% to 75%) for cube-double (Table 8). This is not the case for antmaze-giant, where even strict labelling yields a value dominated by the slow-moving dimensions. Note that we analyse percentage shares of output variance for the state-value function rather than the critic, which would add a confounding factor of how to sample the subgoal/option for the crossed state-goal pairs. For QSO, reward labelling also shapes the option space itself, as the option encoder is trained through the critics. Downstream performance is most sensitive to using high-level reward labelling, which, like for ASO, we attribute to the high-level critic. Interestingly, however, low-level environment labelling makes the encoder markedly less sensitive to fast-moving dimensions, dropping from 79% to 8% for cube-double and 45% to 23% for antmaze-giant; the high-level policy becomes correspondingly more dependent on slow-moving dimensions, respectively increasing from 5% to 23% and 14% to 18%. Hence, while the option space remains similarly clean under both variants (Figures 15 and 16), using low-level environment reward labelling could be considered better, as it retains less information about fast-moving state dimensions without sacrificing downstream performance. Figure 12: Strict versus environment dataset reward labelling ablation for QSO (blue) and ASO (grey). Figure 13: Change in performance when using the environment success criterion rather than the stricter success criterion used by prior work for reward labelling in the high-level decision process. Performance either does not change significantly, or improves for all algorithms. 36 Preprint, October 2026. Figure 14: Performance with different option resampling frequencies for low-level dataset reward labelling ablations for QSO (blue) and ASO (grey). Figure 15: PCA (top) of QSO’s action abstractions for reward labelling ablations for antmaze-giant fitted on random trajectory sequences, showing different groups of three k-step trajectory sequences with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) for those groups in units of median distance between two random trajectory sequences. All reward labelling seems to encourage grouping functionally equivalent behaviours while separating the behaviour groups. 37 Preprint, October 2026. Figure 16: PCA (top) of QSO’s action abstractions for reward labelling ablations for cube-double fitted on random trajectory sequences, showing different groups of three k-step trajectory sequences with semantically similar behaviour (randomly selected from the validation dataset). L2 distances (bottom) for those groups in units of median distance between two random trajectory sequences. All reward labelling seems to encourage grouping functionally equivalent behaviours while separating the behaviour groups. 38 Preprint, October 2026. How sensitive is QSO to policy extraction? Both advantage weighting and RS lead to more consistent performance for ASO and QSO (Figure 17). As motivated, advantage weighting increases the high-level policy’s dependence on the goal: compared with BC, the goal’s share of output variance increases from 1% to 14% for QSO and 1% to 5% for ASO in antmaze-giant, and, respectively, from 0% to 1% and 1% to 3% in cube-double. Note that RS on top of this further increases goal-dependency (Table 9). Figure 17: High-level policy extraction ablation for QSO (blue) and ASO (grey). Flow Policy Deployed Policy (best of 32) Agent Policy ExtractionStateGoalNoise State Goal Noise antmaze-giant-stitch-v0 QSO BC72 [69, 74]1 [1, 2]27 [25, 30] 60 [57, 62] 24 [23, 26] 16 [15, 17] AWR64 [61, 67]14 [13, 15]22 [21, 24] 54 [52, 57] 34 [32, 36] 12 [11, 12] SHARSA BC38 [35, 41]1 [1, 2]61 [58, 64] 34 [31, 36] 15 [15, 16] 51 [49, 53] ASO AWR35 [32, 38]5 [5, 5]60 [57, 62] 33 [31, 36] 19 [18, 20] 48 [46, 50] cube-double-play-v0 QSO BC98 [98, 98]0 [0, 0]2 [2, 2] 97 [96, 97] 2 [2, 2] 2 [1, 2] AWR93 [92, 93]1 [1, 1]6 [6, 7] 88 [88, 89] 6 [6, 7] 5 [5, 6] SHARSA BC85 [85, 86]1 [1, 1]14 [13, 15] 80 [79, 81] 9 [8, 9] 11 [10, 13] ASO AWR84 [83, 85]3 [3, 4]13 [12, 13] 79 [78, 79] 12 [11, 12] 10 [9, 10] Table 9: QSO, SHARSA and ASO high-level goal sensitivity with AWR and BC policy extraction. Percentage share of the variance of the high-level policy’s options explained by the state and goal. Inputs are built on grids that combine 16 random validation states with 16 random validation goals, each under 8 flow-noise samples. For SHARSA and ASO, each dimension of the output is divided by the larger of its standard deviations over the dataset and over the sampled absolute subgoal options. The variance is attributed exactly using two-way ANOVA, so that the attributions sum to 100 (“Goal” includes the state-goal interaction). The flow policy returns a single flow sample; the deployed policy returns the best of 32 flow samples under the high-level critic. Mean and 95% bootstrap confidence interval (2000 resamples of the 64 grids) for one seed after 1M training steps. 39 Preprint, October 2026. How sensitive is QSO to option dimension? QSO is largely insensitive to option dimension from 5 to 20, which all lead to similar downstream performance (Figure 18). We use a fixed default of 10. Figure 18: d option-space dimensionality ablation. How sensitive is QSO to temporal horizon? Like ASO and prior work (Park et al., 2026; Ajay et al., 2020; Park et al., 2023; Li et al., 2026), QSO is sensitive to temporal abstraction horizon (Figure 19). For example, option horizons of 100 steps substantially degrades performance for both QSO and ASO in antmaze-giant; this is unsurprising, however, considering that the maximum number of steps in the antmaze-giant dataset is 200. Figure 19: k temporal horizon ablation for QSO (blue) and ASO (grey). 40