Paper deep dive
An Informational Curse of Horizon in Goal-Conditioned Policy Learning
John L. Zhou, Yuxuan Dong, Jonathan C. Kao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/8/2026, 7:23:58 AM
Summary
This paper identifies an 'informational curse of horizon' in goal-conditioned policy learning, where increasing the goal relabeling horizon during training degrades policy performance and generalization, particularly for Behavioral Cloning (BC) policies. The authors attribute this to a decrease in conditional mutual information between actions and hindsight-relabeled goals, leading to reduced policy sensitivity to goal information. They propose 'Sobolev bootstrapping,' a method that distills input Jacobians from short-horizon policies into long-horizon policies to restore sensitivity and improve performance, especially in combinatorial manipulation tasks.
Entities (8)
Relation Signals (6)
Goal Relabeling Horizon → causes → Performance Degradation
confidence 95% · increasing the goal relabeling horizon can significantly reduce policy generalization and performance.
Behavioral Cloning → suffersfrom → Informational Curse of Horizon
confidence 95% · goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation
Sobolev Bootstrapping → mitigates → Performance Degradation
confidence 94% · distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains
Goal Relabeling Horizon → decreases → Conditional mutual information
confidence 92% · horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals
Sobolev Bootstrapping → uses → Input Jacobians
confidence 90% · distills the input Jacobians of a short-horizon teacher policy into a long-horizon student policy.
Long-horizon Training → shiftssensitivityto → State Information
confidence 88% · both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.
Tags
Links
- Source: https://arxiv.org/abs/2610.09247v1
- Canonical: https://arxiv.org/abs/2610.09247v1
Trouble viewing inline? Open PDF directly →
Full Text
85,609 characters extracted from source content.
Expand or collapse full text
An Informational Curse of Horizon in Goal-Conditioned Policy Learning John L. Zhou †thanks: Equal contribution Yuxuan Dong11footnotemark: 1 Jonathan C. Kao Affiliation: University of California, Los Angeles Affiliation: john.ly.zhou@gmail.com, dyx723@g.ucla.edu Abstract The difficulty of learning goal-reaching policies is often attributed to a “curse of horizon” that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy’s input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data. 1 Introduction Learning to reach any state from any other state is an elegantly simple problem that allows an agent to acquire diverse behaviors from data, without supervision (Kaelbling, 1993). A core component of standard goal-conditioned policy learning involves hindsight goal relabeling (Andrychowicz et al., 2017), which allows an agent to learn from arbitrary experiences regardless of the actor’s original intentions. However, simply providing a policy with auxiliary input in the form of goal observations or natural language instructions does not necessarily produce a policy that is capable of following such instructions. A goal-reaching policy must learn to meaningfully utilize information about the desired goal to generate actions, and we argue that this dependence can only emerge when knowing the goal provides meaningful additional information about the policy learning objective. We formalize this additional goal-related information as the conditional mutual information (MI) ℐ(A;G∣S)I(A;G S) between the goal G and action A conditioned on the current state S, which we refer to as the action-goal mutual information. Under standard hindsight relabeling, action-goal MI is non-increasing with the relabeling horizon [Proposition 3.2]. This points to a distinct informational curse of horizon in goal-conditioned policy learning: increasing the horizon of relabeled training goals not only introduces well-characterized learning challenges (Park et al., 2025), but may also alter how the policy is learned. Specifically, as distant goals provide progressively less information about which action should be taken, the policy may converge to a solution that is globally less sensitive to goal information—even when conditioned on nearby goals. However, disentangling these two effects is surprisingly difficult in standard goal-reaching benchmarks (Park et al., 2024). Long-horizon evaluation goals are typically more challenging and require longer sequences of correct actions, while policies must also be trained on distant goals to simply keep them in-distribution at evaluation time. This makes it virtually impossible to distinguish failures due to increased task difficulty from failures due to weaker goal conditioning. To isolate the effect of training horizon HtrainH_train on performance, we construct a series of controlled experiments using an oracle hierarchical planner that “prompts” a learned, goal-conditioned policy with optimal, fixed-distance (sub)goals at test time. We find that behavioral cloning (BC) policies degrade severely as HtrainH_train increases, even when the distance of goals presented at test time HplanH_plan remains fixed. On the other hand, reinforcement learning (RL) policies are substantially more robust to increased training goal horizons, although this robustness is closely tied to the RL objective (weighted BC or reparameterized policy gradient) and policy class (Gaussian or conditional flow matching) [Section 3.1]. These results suggest that poor policy performance is not only due to higher task difficulty, but also a degradation in the learned policy’s ability to achieve simpler goals. Our theoretical analysis predicts that longer-horizon goals provide less action-relevant information, but how might this be reflected in a policy parameterized by a neural network? In our empirical studies, we use the Jacobian norm of the action with respect to the state or goal as a local measure of policy sensitivity. For certain parameterizations, e.g., fixed-variance Gaussians, the Jacobian norm can be directly related to Fisher information [Box 3.3]. Across BC and RL objectives, as well as Gaussian and flow-matching policy classes, we find that increasing the training goal horizon systematically shifts sensitivity away from goal conditioning towards the current state. Furthermore, explicitly reducing the action-goal MI by training on some proportion of arbitrary, unrelated goal states produces a similar reduction in goal sensitivity. To determine if there is a causal relationship between goal sensitivity and performance, we then ask if preserving the sensitivity structure of policies trained exclusively on short-horizon goals can improve the performance of policies trained on longer-horizon goals. We propose Sobolev bootstrapping, a simple modification of existing policy bootstrapping methods that additionally distills the input Jacobians of a short-horizon teacher policy into a long-horizon student policy. This simple intervention significantly boosts performance in a suite of 16 diverse locomotion and manipulation environments, with especially large gains in combinatorial manipulation tasks. 2 Preliminaries We consider the problem of offline goal-conditioned RL, described by a Markov decision process (MDP) ℳ=(,,ℛ,,)M=(S,A,R,P,G) where S is the state space, A the action space, ℛ:×→ℝR:S×S the goal-conditioned reward function, :×→P:S×A the transition function, and G the goal space. In this work, we assume the goal space is equivalent to the state space. In the offline setting, we are given a reward-free dataset of environment interactions D and asked to learn a policy π(⋅∣s,g)π(· s,g) to reach a specified goal g. At test time, the learned policy is deployed into the environment for a set of evaluation goals drawn from some goal distribution p(g)p(g). Then, we can write the objective as (π)=g∼p(g),τ∼pπ(τ)[∑t=0∞γtr(st,g)], (π)=E_g p(g),τ p^π(τ) [ _t=0^∞γ^tr (s_t,g ) ], (1) where pπ(τ)p^π(τ) is the distribution of trajectories generated by the policy π and the transition function P during (online) evaluation. Importantly, we do not assume access to p(g)p(g) and must learn a general policy that is capable of reaching any goal encountered in the dataset. 2.1 Input Jacobians and policy sensitivity Our analyses draw inspiration from prior work that studies input gradients, defined simply as the gradient of the model output with respect to its inputs, from the perspective of classifier robustness in computer vision applications. In our work, we extend these ideas to the parameters of a policy action distribution π. For exposition, we first study policies modeled as fixed-variance Gaussian distributions, where a∼(μθ(s,g),)a ( _θ(s,g),I). Differentiating this output with respect to the state or goal yields the policy’s state and goal Jacobians, Jπ(s)=∂μθ(a∣s,g)∂sandJπ(g)=∂μθ(a∣s,g)∂g. J_π(s)= ∂ _θ(a s,g)∂ s J_π(g)= ∂ _θ(a s,g)∂ g. (2) Prior work has established that properties of input gradients such as the magnitude of the ℓ1 _1 norm and degree of alignment with human perception strongly correlate with model robustness (Tsipras et al., 2018; Ganz et al., 2023), and either regularizing the input gradient norm (Drucker and LeCun, 1992; Ross and Doshi-Velez, 2018; Rodríguez-Muñoz et al., 2024) or distilling input gradients from a robust teacher through Sobolev training (Czarnecki et al., 2017; Chan et al., 2020) can significantly improve generalization under perturbations. 3 An Informational Curse of Horizon Figure 1: Training on more distant relabeled goals harms nearby goal-reaching capabilities. We record the success rates across policies trained on a geometric distribution of relabeling goal offsets with discount γ, where γ=1γ=1 corresponds to a uniform distribution over future states. At evaluation, policies are given optimal HplanH_plan-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints, three seeds, and four environments: cube-double, cube-triple, cube-quadruple, and scene. Horizon has many opportunities to influence the performance of a goal-reaching policy in subtly different ways. Longer evaluation horizons allow compounding error to accumulate over time, reducing the success rate of a policy with a fixed per-step error rate ϵε (Ross and Bagnell, 2010). Furthermore, directly learning to reach distant goals is simply harder in the first place, due to noisy advantage estimates and compounding bias in value learning [Appendix A.2]. We identify a third challenge in which increasing the horizon of relabeled training goals, HtrainH_train, can degrade the policy’s ability to reach nearby goals as well. Said differently, the difficulties induced by long-horizon training are not confined to reaching distant goals. To study this effect, we introduce a third horizon: the planning horizon, HplanH_plan, which separates the total evaluation horizon HevalH_eval from the horizon of the goals presented to the learned policy. Using an oracle planner, we decouple HplanH_plan from HtrainH_train and HevalH_eval. This reveals severe, training horizon-dependent degradation for goal-conditioned BC policies and a milder, albeit objective-dependent form of the same phenomenon in RL policies. We provide an information-theoretic explanation for these findings and study how it manifests in policies parameterized by neural networks. 3.1 How does horizon affect performance? To isolate policy quality from task difficulty, we use an oracle planner that presents the policy with optimal subgoals at a fixed HplanH_plan while leaving the underlying evaluation task unchanged. HplanH_plan is similar in spirit to the subgoal horizon used in hierarchical approaches, but here we use it only at evaluation rather than learning a separate high-level policy. We vary the distribution over training goal horizons in three complementary ways. First, we sample future in-trajectory goals from a geometric distribution with increasing discount factors γ, which assign more probability mass to longer horizons while maintaining the same goal support set [Figures 1 & 6]. Then, we directly control the range of goal distances the actor sees by truncating uniform distributions to different maximum offsets [Figure 5]. Finally, we mix in arbitrary, unrelated dataset states as goals, directly weakening the predictive relationship between actions and goals [Figure 7]. Across both the fixed- and geometric-horizon experiments, we observe a clear degradation in nearby goal-reaching competence as BC policies are trained on increasingly distant relabeled goals. Because the evaluation task and HplanH_plan are fixed, this effect cannot simply be attributed to the policy being asked to reach more distant goals. Instead, it is clear that increasing HtrainH_train changes the quality of the learned actor itself. RL policies exhibit a milder version of the same phenomenon, where the degree of robustness to horizon depends heavily on the objective. Our results demonstrate more nuance than a monotonic relationship between training goal horizon and performance. Geometric relabeling can retain strong performance even with exposure to long-horizon goals, and providing unrelated dataset states as goals results in consistent performance decline across all algorithms. Taken together, our experiments suggest that temporal or spatial training goal distance alone cannot fully account for policy degradation. To resolve this discrepancy, we hypothesize that the informational relationship between actions and goals is the critical factor. Next, we formalize this intuition and investigate how training on extended horizons alters the learned policy for nearby goals. 3.2 Why should training horizon matter? To understand why training on extended horizons can degrade nearby goal-reaching performance [Section 3.1], we theoretically analyze the divergence between a goal-conditioned policy and its unconditional behavioral prior, focusing on how sources of goal-dependent action information change with relabeling horizon. To make the analysis tractable, we focus on the KL-regularized objective corresponding to Peng et al. (2019). We use uppercase letters (e.g., StS_t) to denote random variables, and lower-case letters (e.g., sts_t) to denote their instantiations. The full proof is provided in Appendix B. Theorem 3.1 (Divergence Bound from the Goal-Agnostic Prior). Let π∗(a∣s,g)∝pHR(a∣s,g)exp(Q(s,a,g)/α)π^*(a s,g) p_HR(a s,g) (Q(s,a,g)/α) be the target policy induced by a KL-regularized policy update, where α is a temperature parameter and pHRp_HR is the distribution of tuples (S,A,G)=(St,At,St+K)(S,A,G)= (S_t,A_t,S_t+K ) induced by the dataset trajectory distribution and a hindsight relabeling rule K∼qHRK q_HR. Let β(a∣s)β(a s) be the goal-agnostic behavior policy induced by the dataset, and let the critic value span be ΔQ(s,g)=supaQ(s,a,g)−infaQ(s,a,g) _Q(s,g)= _aQ(s,a,g)- _aQ(s,a,g) over the support of pHRp_HR. Then, the expected squared total variation distance between the two policies is bounded by: S,G[DTV(π∗(⋅∣S,G),β(⋅∣S))2]≤S,G[ΔQ(S,G)2]8α2+IHR(A;G∣S)E_S,G [D_TV (π^*(· S,G),β(· S) )^2 ]≤ E_S,G [ _Q(S,G)^2 ]8α^2+I_HR(A;G S) This bound reveals that π∗π^* degenerates toward the goal-agnostic prior when the value function weakly distinguishes among supported actions relative to α, and the relabeled goal contains little additional information about the action when conditioned on the current state. Critically, both terms decay over the horizon. Notably, the first value-based term applies only to RL, whereas the second is applicable to both BC and RL. The diminishing action gap. The action gap quantifies the difference in value between the optimal action and sub-optimal actions at a given state (Farahmand, 2011). In goal-conditioned RL with sparse rewards and a discount factor γ<1γ<1, the relative advantage of any single primitive action decays exponentially as the goal distance d(s,g)d(s,g) increases. Consequently, the expected value span ΔQ(S,G) _Q(S,G) shrinks, and the Q-values provide progressively less information about the optimal action distribution. As d(s,g)→∞d(s,g)→∞, the first term approaches zero. Proposition 3.2 (Monotonic Decay of Mutual Information). Let dataset trajectories be generated by a Markovian behavior policy and transition kernel. For any time horizon (relabeling offset) k′>k>0k >k>0, given the current state StS_t, the sequence At→St+k→St+k′A_t→ S_t+k→ S_t+k forms a conditional Markov chain. The mutual information between the action and future states monotonically decreases over time: I(At;St+k∣St) I (A_t;S_t+k S_t ) −I(At;St+k′∣St) -I (A_t;S_t+k S_t ) =St,St+k,St+k′[DKL(p(At∣St,St+k)∥p(At∣St,St+k′))]≥0 =E_S_t,S_t+k,S_t+k [D_KL (p(A_t S_t,S_t+k)\|p(A_t S_t,S_t+k ) ) ]≥ 0 This inequality is strict when there exist at least two intermediate states, st+k(1)s_t+k^(1) and st+k(2)s_t+k^(2), that share a common future successor state st+k′s_t+k with non-zero probability, but yield distinct intermediate action posteriors p(At∣st,st+k(1))≠p(At∣st,st+k(2))p(A_t s_t,s_t+k^(1))≠ p(A_t s_t,s_t+k^(2)). Non-increasing mutual information. BC suffers a catastrophic performance degradation when trained on more distant relabeled goals (see GCBC and GCFBC in Figure 1), which is unsurprising as it does not benefit from the additional goal-related information imparted by a goal-conditioned value function. We formalize this as a value-independent reduction in mutual information IHR(A;G∣S)I_HR(A;G S) as the relabeling horizon grows [Proposition 3.2]. An extended discussion is included in Appendix B. This bound only guarantees non-increasing information. Strictly decreasing information occurs in environments and datasets where distinct trajectories eventually merge at a shared future state, which can occur in two common scenarios. First, when learning from a sub-optimal policy, subsequent behavior can “overwrite” the effects of earlier actions. An agent might make a mistake and later correct it, reaching the same future state as a trajectory that took an optimal path. Second, merging occurs when there are multiple valid ways to accomplish the same goal. For example, in a sequential pick-and-place task over three objects, there are up to six valid orderings that all lead to the same final configuration. In both scenarios, longer task and relabeling horizons provide more opportunities for these action-distinguishable trajectories to merge. Thus, temporally distant relabeled goals provide progressively less information about the immediate action. 3.3 From action-goal mutual information to policy sensitivity While our theoretical results characterize goal dependence as a property of the full distribution, it is difficult to measure this quantity in a policy parameterized by a neural network. One local measure of sensitivity is the input gradient (or Jacobian, for vector-valued outputs). As introduced in Section 2.1, we use the Jacobian of the policy action with respect to the state and goal as a first-order measure of policy sensitivity. A large goal Jacobian indicates that small changes in the goal induce large changes in the action, whereas a small goal Jacobian indicates local invariance to the goal. It is important to acknowledge that this quantity is not an estimator of I(A;G∣S)I(A;G S). Rather, it allows us to locally probe the action-goal dependence whose global behavior is captured by mutual information. For fixed-variance Gaussian policies, the squared Frobenius norm of the input Jacobian corresponds to the trace of the Fisher information matrix (FIM), and directly measures how rapidly the policy distribution changes with respect to changes in goal information [Box 3.3]. To investigate whether the goal Jacobian norm reflects the predicted decrease in action-goal MI, we measure the ℓ1 _1 goal Jacobian norm proportion for policies trained on the goal distributions described in Section 3.1. The geometric [Figures 2 & 8] and truncated uniform distributions [Figure 9] vary the proportion of temporally distant goals, which we expect to be less informative about the immediate action. The random goal mixture more directly reduces action-goal MI by introducing unrelated target states [Figure 10]. Across BC and RL objectives, as well as Gaussian and flow-matching policy classes, we observe a consistent, global shift in sensitivity away from the goal towards the current state across matched state-goal distances as the training horizon increases. As predicted, we see that goal sensitivity collapses for the BC policies, which do not utilize additional value-related information [Section 3.2]. Figure 2: Increasing training goal horizon globally decreases policy goal sensitivity. We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with log2 _2 scaling in cube-triple. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon, with additional environments in Figure 8. Intuition: Fisher Information Regularization When π is a Gaussian distribution with fixed variance σ2σ^2, the state Jacobian Jπ(s)J_π(s) is the square root of the Fisher information matrix (FIM) ℐ(s)=πθ[∇slogπθ(a∣s,g)⋅∇slogπθ(a∣s,g)T] (s)=E_ _θ [ _s _θ(a s,g)· _s _θ(a s,g)^T ] with respect to the state. Consequently, the squared Frobenius norm of the state Jacobian, ‖Js(s,g)‖F2\|J_s(s,g)\|_F^2, when scaled by σ2σ^2, corresponds exactly to the trace of the FIM and serves as a scalar measure of the local information the policy extracts from the state input s. Viewed through this lens, our Sobolev bootstrapping objective [Equation 3] acts as a dynamic form of Fisher information regularization. By penalizing deviations between the student’s state Jacobian and that of the short-horizon teacher, we enforce the local information geometry of the long-horizon policy to be similar to that of the short-horizon policy. This also provides a mechanical explanation for the findings in Section 5.3 that regularizing the state Jacobian also increases the norm of the goal Jacobian. By regularizing local state dependence, Sobolev bootstrapping prevents the network from converging to a solution which exploits state-only optimization shortcuts and ignores goal conditioning inputs [Section 3]. 4 Sobolev Bootstrapping The above analysis establishes a strong relationship between action-goal information and local policy sensitivity, where policies trained on longer-horizon goals exhibit reduced sensitivity to goal conditioning and a corresponding increase in sensitivity to the current state. This motivates us to investigate whether intervening on the policy input Jacobian can mitigate performance degradation. Direct ℓ1 _1 and ℓ2 _2 penalties on the state Jacobian yield modest gains on simpler tasks [Table 2], providing evidence for a causal link between the policy sensitivity and downstream performance. However, these naive penalties fail to scale to more complex, multi-object environments. We argue that a more principled Jacobian intervention should be context-dependent, and propose Sobolev bootstrapping. Our approach uses a policy trained only on short-horizon goals as a first-order teacher for the full goal-conditioned “student” policy. Rather than globally suppressing state sensitivity via a uniform norm penalty, we regularize the long-horizon policy’s state Jacobian towards that of the short-horizon policy, explicitly distilling the desirable sensitivity structure learned in a regime of high action-goal information. We additionally note that, for fixed-variance Gaussian policies, this objective can be interpreted as a form of Fisher information regularization [Box 3.3]. 4.1 Using short-horizon policies as teachers As the name suggests, Sobolev bootstrapping builds on a class of policy extraction algorithms which perform policy bootstrapping (Chane-Sane et al., 2021; Zhou and Kao, 2025). These algorithms provide a natural foundation for our proposed method, since they already maintain a separate one-step policy trained only on short-horizon goals. Due to space constraints, we provide only a brief intuition here and defer detailed descriptions of the value learning and policy extraction algorithms to Appendix D. Policy bootstrapping methods leverage the insight that, if an action a is optimal for reaching a subgoal sgs_g along the optimal path to goal g, then a is also optimal for reaching g. Additionally, goal-conditioned policies are more likely to be accurate for short-horizon subgoals sgs_g, due to the decreasing “signal-to-noise” ratio of advantage estimates over the horizon (Park et al., 2023). Policy bootstrapping therefore uses π(a∣s,sg)π(a s,s_g) as a stronger, short-horizon supervisory signal than pure value-based policy extraction, by minimizing the Kullback-Leibler (KL) divergence between their action distributions argminθDKL(πθ(a∣s,g)∥πsub(a∣s,sg)). _θD_KL ( _θ(a s,g)\|π^sub(a s,s_g) ). But how do we find good subgoals for a given goal? Existing approaches such as Reinforcement Learning with Imagined Subgoals (Chane-Sane et al., 2021, RIS) and Subgoal Advantage-Weighted Policy Bootstrapping (Zhou and Kao, 2025, SAW) differ primarily in how they answer this question. Similar to a bilevel hierarchical policy, RIS learns a generative model πh(sg∣s,g)π^h(s_g s,g) to produce “imagined” subgoals. However, explicitly learning a hierarchical policy is known to suffer from compounding error and bottlenecks performance (Ahn et al., 2025). On the other hand, SAW avoids these optimization difficulties by sampling only from in-trajectory waypoints sg∼p(st+k∣st,g)s_g p^D(s_t+k s_t,g). It then re-weights the bootstrapping loss by exp(A(s,sg,g)) (A(s,s_g,g)), using a subgoal advantage estimate to prioritize high-quality waypoints [Equation 12]. The advantage A(s,sg,g)A(s,s_g,g) is approximated by the learned value function as V(sg,g)−V(s,g)V(s_g,g)-V(s,g), following Park et al. (2023). 4.2 Input Jacobian distillation While standard policy bootstrapping only distills the action distribution of the short-horizon teacher, our experiments in Section 3 suggest that the input Jacobian of a short-horizon teacher may also provide useful supervisory signals. Drawing inspiration from Sobolev training (Czarnecki et al., 2017), we augment the standard zeroth-order distillation with a first-order objective that matches the student’s state Jacobian to that of the teacher policy πsub(a∣s,sg)π^sub(a s,s_g): SB=s,a,sg,g[‖∂πθ(a∣s,g)∂s−∂πsub(a∣s,sg)∂s‖22]. _SB=E_s,a,s_g,g [ \| ∂ _θ(a s,g)∂ s- ∂π^sub(a s,s_g)∂ s \|_2^2 ]. (3) Intuitively, the short-horizon teacher provides a prior not only for what the long-horizon policy should do, but also what it should pay attention to when deciding what to do. We match only the state Jacobian because the two policies receive different goal inputs: πsubπ^sub is conditioned on a nearby subgoal sgs_g and π on the final goal g. Interestingly, we find that regularizing the state Jacobian induces a corresponding change in the goal Jacobian, which we discuss in more detail in Section 5.3. Table 1: Evaluating Sobolev bootstrapping on offline goal-conditioned RL tasks. We compare our method’s average (binary) success rate (%\%) against the numbers reported in Park et al. (2024) and Zhou and Kao (2025) across the five test-time goals for each environment, averaged over the last three checkpoints and eight seeds ± standard deviations. Numbers within 5%5\% of the best value in the row are in bold. Environment Dataset GCBC GCIVL GCIQL QRL CRL HIQL SAW S2AW pointmaze pointmaze-medium-navigate-v0 99 ±6± 6 6363 ±6± 6 5353 ±8± 8 8282 ±5± 5 2929 ±7± 7 7979 ±5± 5 97 ±2± 2 96 ±3± 3 pointmaze-large-navigate-v0 2929 ±6± 6 4545 ±5± 5 3434 ±3± 3 86 ±9± 9 3939 ±7± 7 5858 ±5± 5 85 ±10± 10 83 ±8± 8 pointmaze-giant-navigate-v0 11 ±2± 2 00 ±0± 0 00 ±0± 0 6868 ±7± 7 2727 ±10± 10 4646 ±9± 9 6868 ±8± 8 73 ±9± 9 antmaze antmaze-medium-navigate-v0 2929 ±4± 4 7272 ±8± 8 7171 ±4± 4 8888 ±3± 3 95 ±1± 1 96 ±1± 1 97 ±1± 1 97 ±1± 1 antmaze-large-navigate-v0 2424 ±2± 2 1616 ±5± 5 3434 ±4± 4 7575 ±6± 6 8383 ±4± 4 91 ±2± 2 90 ±3± 3 93 ±2± 2 antmaze-giant-navigate-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 1414 ±3± 3 1616 ±3± 3 6565 ±5± 5 7373 ±4± 4 87 ±3± 3 humanoidmaze humanoidmaze-medium-navigate-v0 88 ±2± 2 2424 ±2± 2 2727 ±2± 2 2121 ±8± 8 6060 ±4± 4 89 ±2± 2 88 ±3± 3 90 ±2± 2 humanoidmaze-large-navigate-v0 11 ±0± 0 22 ±1± 1 22 ±1± 1 55 ±1± 1 2424 ±4± 4 4949 ±4± 4 4646 ±4± 4 57 ±5± 5 humanoidmaze-giant-navigate-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 11 ±0± 0 33 ±2± 2 1212 ±4± 4 3535 ±4± 4 39 ±5± 5 cube cube-single-play-v0 66 ±2± 2 5353 ±4± 4 6868 ±6± 6 55 ±1± 1 1919 ±2± 2 4444 ±9± 9 7272 ±5± 5 86 ±5± 5 cube-double-play-v0 11 ±1± 1 3636 ±3± 3 4040 ±5± 5 11 ±0± 0 1010 ±2± 2 66 ±2± 2 4040 ±7± 7 57 ±6± 6 cube-triple-play-v0 11 ±1± 1 11 ±0± 0 33 ±1± 1 00 ±0± 0 44 ±1± 1 33 ±1± 1 44 ±2± 2 30 ±5± 5 cube-quadruple-play-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 3 ±2± 2 scene scene-play-v0 55 ±1± 1 4242 ±4± 4 5151 ±4± 4 55 ±1± 1 1919 ±2± 2 3838 ±3± 3 6363 ±6± 6 74 ±6± 6 puzzle puzzle-3x3-play-v0 22 ±0± 0 66 ±1± 1 95 ±1± 1 11 ±0± 0 33 ±1± 1 1212 ±2± 2 99 ±3± 3 1515 ±4± 4 puzzle-4x4-play-v0 00 ±0± 0 1313 ±2± 2 2626 ±3± 3 00 ±0± 0 00 ±0± 0 77 ±2± 2 2626 ±3± 3 41 ±3± 3 4.3 A practical policy extraction algorithm Sobolev bootstrapping requires access to differentiable subgoal-conditioned “teacher” policies, making existing policy bootstrapping algorithms a natural foundation for our approach. For our primary results in Table 1, we instantiate Sobolev bootstrapping on top of SAW, which we denote as S2AW [Algorithm 1], due to its simplicity and strong performance. We modify the objective in Equation 3 to add the same subgoal advantage-weighted coefficient as in SAW [Appendix D], resulting in the objective SB=s,a,sg,g[eA(s,sg,g)‖Jπθ(s)−Jπsub(s)‖22]. _SB=E_s,a,s_g,g [e^A(s,s_g,g) \|J_ _θ(s)-J_π^sub(s) \|_2^2 ]. (4) While we focus our main analysis on S2AW, we also demonstrate the effectiveness of Sobolev bootstrapping on RIS, achieving substantial gains over the baseline algorithm [Section H.2]. 5 Experiments We evaluate Sobolev bootstrapping on the OGBench suite (Park et al., 2024), which spans long-horizon locomotion and combinatorial manipulation tasks. Our experiments address two central questions: (1) does Sobolev bootstrapping improve goal-reaching performance, and (2) does it reverse the horizon-induced shift in policy sensitivity identified in our previous experiments? Interestingly, although Sobolev bootstrapping only distills the state Jacobian, we find that it also restores goal-conditioning sensitivity [Section 5.3]. 5.1 Sobolev bootstrapping boosts long-horizon performance Adding Sobolev bootstrapping to SAW (S2AW) improves performance across most environments [Table 1], with the largest gains in combinatorial manipulation tasks such as the cube and puzzle environments. In particular, S2AW improves from 40%40\% to 57%57\% on cube-double, 4%4\% to 30%30\% on cube-triple, and 26%26\% to 41%41\% on puzzle-4x4. We also observe substantial improvements in long-horizon locomotion, although these results may be partially attributed to architectural differences [Appendix H.1]. Additional horizon reduction baselines are provided in Section H.3. The improvements in the discrete puzzle environments demonstrate that first-order distillation can be practically useful even when the true decision boundary is non-smooth, since the short-horizon teacher policy provides a (piecewise) smooth, differentiable approximation of these hard boundaries. 5.2 Effects on state sensitivity Figure 3: Sobolev bootstrapping exhibits more interpretable sensitivity structure. We plot the proportion of the total input Jacobian norm corresponding to each cube during a triple pick-and-place on-policy rollout for SAW without (top) and with (bottom) Sobolev bootstrapping. For fairness, we also analyze a scripted optimal trajectory in Figure 11. To examine how Sobolev bootstrapping qualitatively changes policy sensitivity to state information, we decompose the state Jacobian norm by object over the course of a cube-triple rollout [Figure 3], allowing us to visualize where the policy’s “attention” is allocated. Vanilla SAW exhibits diffuse sensitivity across all cubes and immediately fails at the first task stage, whereas S2AW dynamically allocates sensitivity to the focal cube at each stage and successfully completes the full triple pick-and-place task. To control for differences in policy competence, we repeat the analysis on an independent, scripted optimal trajectory in Figure 11 and observe the same pattern. We hypothesize that the observed shift in sensitivity away from goal information and towards state inputs may also result in policies that overfit to task-irrelevant “distractors.” To test this, we evaluate policies trained in multi-cube environments on single pick-and-place tasks. The first evaluation task in cube-double and triple already tests this capability, and we add a similar task to cube-quadruple, which we refer to as Task 6. While naive ℓ1 _1 and ℓ2 _2 Jacobian norm penalties produce small improvements, context-dependent Sobolev bootstrapping significantly improves performance even on cube-triple and quadruple [Table 2]. Figure 4: Sobolev bootstrapping recovers policy goal sensitivity. We compare the absolute goal Jacobian norm between S2AW and SAW, averaging over batches of goals sampled at fixed intervals from the current state up to the maximum horizon. Results are averaged across four seeds and 1024 state-goal pairs for each horizon, with shaded standard deviations. Table 2: Sobolev bootstrapping improves single pick-and-place performance in the presence of distractor objects. We evaluate policies trained in multi-cube environments on single pick-and-place tasks. While naive ℓ1 _1 or ℓ2 _2 Jacobian regularization [Appendix H.4] yields modest improvements on simpler tasks, Sobolev bootstrapping matches or outperforms all baselines. Other settings are identical to Table 1. Environment GCIQL GCIQLℓ1_ _1 SAW SAWℓ1_ _1 SAWℓ2_ _2 S2AW cube-single 6868 ±6± 6 7474 ±4± 4 7676 ±8± 8 85 ±6± 6 85 ±4± 4 86 ±5± 5 cube-double-task1 7474 ±8± 8 7676 ±8± 8 6464 ±9± 9 81 ±9± 9 6565 ±14± 14 78 ±12± 12 cube-triple-task1 1313 ±3± 3 1616 ±10± 10 1818 ±11± 11 2424 ±9± 9 1717 ±12± 12 74 ±9± 9 cube-quadruple-task6 00 ±1± 1 00 ±1± 1 22 ±2± 2 22 ±2± 2 11 ±1± 1 35 ±14± 14 5.3 Effects on goal conditioning Although Sobolev bootstrapping explicitly matches only the state Jacobian, we find that it induces an increase in the policy’s absolute goal Jacobian norm [Figure 4]. This suggests that the policy has converged to a different, more goal-sensitive solution instead of simply smoothing out state sensitivity. We note that a decrease in sensitivity to distant goals is unsurprising—for instance, perturbing the target placement in a pick-and-place task should not affect the policy before grasping the cube. However, standard long-horizon policies exhibit global insensitivity across all goal distances, including nearby goals [Figure 4], and it is this pathology that Sobolev bootstrapping addresses. 6 Discussion In this work, we identify an informational curse of horizon in goal-conditioned policy learning. By decoupling the training goal horizon from the subgoal distance presented at test time, we show that policies trained on distant relabeled goals can become worse at reaching nearby goals. We explain this effect through the decay of action-goal mutual information under hindsight relabeling: as training goals become more distant, they provide progressively less information about the immediate action. This is accompanied by a global shift in sensitivity away from the goal and toward the current state in neural network policies, across all goal distances. Although input Jacobians are only a local probe of this dependence, explicitly weakening action-goal information produces a corresponding collapse in goal sensitivity. We then propose Sobolev bootstrapping, which exploits this observation by distilling the sensitivity structure of short-horizon policies into long-horizon policies and achieves large gains in combinatorial manipulation tasks. Overall, our results suggest that the goal-relabeling horizon should be treated as an important design choice, rather than a mere consequence of task horizon. Our work does have several limitations. The optimal goal-horizon distribution depends strongly on the policy objective, prompting further study on how different policy extraction methods utilize goal information. Additionally, input Jacobians capture only local sensitivity and are not direct estimators of mutual information, and the Sobolev bootstrapping algorithm derived from those insights introduces significant additional computation. These limitations motivate future work on information-aware relabeling and policy-learning objectives that preserve goal dependence without explicit Jacobian distillation. Acknowledgments This work was supported by NIH R01NS121097 awarded to JCK. References Ahn et al. (2025) H. Ahn, H. Choi, J. Han, and T. Moon Option-aware temporally abstracted value for offline goal-conditioned reinforcement learning. External Links: Link Cited by: §A.2, Figure 11, §H.2, §H.3, §H.3, §4.1. Amit et al. (2020) R. Amit, R. Meir, and K. Ciosek Discount factor as a regularizer in reinforcement learning. In International Conference on Machine Learning, p. 269–278. Note: ISSN: 2640-3498 External Links: Link Cited by: §A.2. Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba Hindsight experience replay. In Neural Information Processing Systems, Vol. 30. External Links: Link Cited by: §1. Boucheron et al. (2013) S. Boucheron, G. Lugosi, and P. Massart Concentration inequalities : a non asymptotic theory of independence. Oxford University Press. Cited by: Appendix B. Chan et al. (2020) A. Chan, Y. Tay, and Y. Ong What it thinks is important is important: robustness transfers through input gradients. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 329–338. Note: ISSN: 2575-7075 External Links: Link, Document Cited by: §2.1. Chane-Sane et al. (2021) E. Chane-Sane, C. Schmid, and I. Laptev Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, p. 1430–1440. Note: ISSN: 2640-3498 External Links: Link Cited by: §D.3, §D.3, §H.2, §4.1, §4.1. Czarnecki et al. (2017) W. M. Czarnecki, S. Osindero, M. Jaderberg, G. Świrszcz, and R. Pascanu Sobolev training for neural networks. In Neural Information Processing Systems (NeurIPS), External Links: Link, Document, 1706.04859 [cs] Cited by: §2.1, §4.2. Drucker and LeCun (1992) H. Drucker and Y. LeCun Improving generalization performance using double backpropagation. IEEE Transactions on Neural Networks 3 (6), p. 991–997. External Links: ISSN 1941-0093, Link, Document Cited by: §2.1. Du et al. (2023) Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel Learning universal policies via text-guided video generation. External Links: 2302.00111, Link Cited by: §A.3. Eysenbach et al. (2022) B. Eysenbach, T. Zhang, R. Salakhutdinov, and S. Levine Contrastive learning as goal-conditioned reinforcement learning. External Links: Link, Document, 2206.07568 [cs] Cited by: §A.1. Farahmand (2011) A. Farahmand Action-gap phenomenon in reinforcement learning. In Neural Information Processing Systems, Vol. 24. External Links: Link Cited by: §3.2. Fujimoto and Gu (2021) S. Fujimoto and S. Gu A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems, External Links: Link Cited by: §A.1. Ganz et al. (2023) R. Ganz, B. Kawar, and M. Elad Do perceptually aligned gradients imply robustness?. In International Conference on Machine Learning, p. 10628–10648. External Links: ISSN 2640-3498, Link Cited by: §2.1. Hu et al. (2022) H. Hu, Y. Yang, Q. Zhao, and C. Zhang On the role of discount factor in offline reinforcement learning. In International Conference on Machine Learning, p. 9072–9098. Note: ISSN: 2640-3498 External Links: Link Cited by: §A.2. Jiang et al. (2015) N. Jiang, A. Kulesza, S. Singh, and R. Lewis The dependence of effective planning horizon on model accuracy. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Cited by: §A.2. Jiang et al. (2016) N. Jiang, S. Singh, and A. Tewari On structural properties of MDPs that bound loss due to shallow planning. Cited by: §A.2. Kaelbling (1993) L. P. Kaelbling Learning to achieve goals. In IJCAI, Vol. 2, p. 1094–8. External Links: Link Cited by: §A.1, §1. Korniak et al. (2026) M. Korniak, K. Dybek, B. Eysenbach, M. Bagatella, and M. Bortkiewicz Three steps at a time: learning representations from action sequences in contrastive rl. External Links: 2608.30640, Link Cited by: §A.3. Kostrikov et al. (2021) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. External Links: Link Cited by: §A.1, §D.1. Levine (2018) S. Levine Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv. External Links: Link, Document, 1805.00909 [cs] Cited by: §D.3. Li et al. (2026a) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, Y. Shen, and Y. Xu Causal world modeling for robot control. External Links: 2601.21998, Link Cited by: §A.3. Li et al. (2026b) Q. Li, S. Park, and S. Levine Decoupled q-chunking. External Links: Link Cited by: §A.2, §H.3, §H.3. Li et al. (2025) Q. Li, Z. Zhou, and S. Levine Reinforcement learning with action chunking. In Neural Information Processing Systems, External Links: Link, Document, 2507.07969 [cs] Cited by: §A.2, §H.3. Mishra et al. (2026) U. A. Mishra, Y. Chen, D. Xu, Y. Liu, X. Chen, and J. Mao Understanding and mitigating the video-action generalization gap via temporal ratio. External Links: 2607.08127, Link Cited by: §A.3. Park et al. (2024) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned RL. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: Appendix F, §1, Table 1, §5. Park et al. (2025) S. Park, K. Frans, D. Mann, B. Eysenbach, A. Kumar, and S. Levine Horizon reduction makes RL scalable. External Links: Link Cited by: §A.2, §H.3, §H.3, §1. Park et al. (2023) S. Park, D. Ghosh, B. Eysenbach, and S. Levine HIQL: offline goal-conditioned RL with latent states as actions. External Links: Link Cited by: §D.1, §D.3, §H.2, §H.3, §4.1, §4.1. Park et al. (2026) S. Park, A. Oberai, P. Atreya, and S. Levine Transitive RL: value learning via divide and conquer. External Links: Link Cited by: §A.1. Peng et al. (2019) X. B. Peng, A. Kumar, G. Zhang, and S. Levine Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv. External Links: Link, Document, 1910.00177 Cited by: §A.1, §D.2, §3.2. Physical Intelligence et al. (2026) Physical Intelligence, B. Ai, A. Amin, R. Aniceto, A. Balakrishna, G. Balke, K. Black, G. Bokinsky, S. Cao, T. Charbonnier, V. Choudhary, F. Collins, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, M. Dhaka, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Habeeb, H. Hancock, K. Hausman, G. Hussein, V. Hwang, B. Ichter, C. Jacobsen, S. Jakubczak, R. Jen, T. Jones, G. Kammerer, B. Katz, L. Ke, M. Khadikov, C. Kuchi, M. Lamb, D. LeBlanc, B. LeCount, S. Levine, X. Li, A. Li-Bell, V. Lialin, Z. Liang, W. Lim, Y. Lu, E. Luo, V. Mano, N. Marwaha, A. Mongush, L. Murphy, S. Nair, T. Patterson, K. Pertsch, A. Z. Ren, G. Schelske, C. Sharma, B. Shi, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, J. Tang, J. Tanner, S. Tekeste, M. Torne, K. Vedder, Q. Vuong, A. Walling, H. Wang, J. Wang, X. Wang, C. Whalen, S. Whitmore, B. Williams, C. Xu, S. Yoo, L. Yu, W. Zhang, Z. Zhang, and U. Zhilinsky π0.7π_0.7: A steerable generalist robotic foundation model with emergent capabilities. External Links: 2604.15483, Link Cited by: §A.3. Rodríguez-Muñoz et al. (2024) A. Rodríguez-Muñoz, T. Wang, and A. Torralba Characterizing model robustness via natural input gradients. In European Conference on Computer Vision, p. 161–178. External Links: ISBN 978-3-031-73116-7, Document Cited by: §2.1. Ross and Doshi-Velez (2018) A. Ross and F. Doshi-Velez Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In AAAI Conference on Artificial Intelligence, Vol. 32. External Links: ISSN 2374-3468, 2159-5399, Link, Document Cited by: §2.1. Ross and Bagnell (2010) S. Ross and D. Bagnell Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics, p. 661–668. Note: ISSN: 1938-7228 External Links: Link Cited by: §A.2, §3. Tsipras et al. (2018) D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry Robustness may be at odds with accuracy. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. Wang et al. (2023) T. Wang, A. Torralba, P. Isola, and A. Zhang Optimal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learning, p. 36411–36430. External Links: ISSN 2640-3498, Link Cited by: §A.1. Zheng et al. (2026) B. Zheng, V. Myers, B. Eysenbach, and S. Levine Scaling goal-conditioned reinforcement learning with multistep quasimetric distances. External Links: Link Cited by: §A.2. Zhou and Kao (2025) J. L. Zhou and J. Kao Flattening hierarchies with policy bootstrapping. In Neural Information Processing Systems, External Links: Link Cited by: §D.3, §D.3, Appendix D, Appendix F, §H.2, §H.3, Table 7, §4.1, §4.1, Table 1. Zhou et al. (2025) P. Zhou, L. Chen, S. Chen, D. Chen, W. Zhao, R. Jin, G. Ren, and J. Luo Act2goal: from world model to general goal-conditioned policy. arXiv preprint arXiv:2512.23541. Cited by: §A.3. Appendix A Related Work A.1 Offline goal-conditioned policy learning The objective of offline goal-conditioned policy learning is to learn a policy which can reach any state from any other state from a fixed dataset of offline trajectories. This can be accomplished with goal-conditioned behavioral cloning (GCBC) that relabels state-action pairs with future trajectory states as goals; or goal-conditioned reinforcement learning (GCRL), which makes use of learned goal-conditioned value functions. It combines the challenges of conservative offline RL with the sparse rewards and recursive structure of goal-reaching tasks (Kaelbling, 1993). Standard algorithms typically use behavior-regularized actor-critic methods or learn an implicit value function with in-sample value iteration (Kostrikov et al., 2021), then extract a policy using weighted behavioral cloning (Peng et al., 2019) or reparameterized policy gradient with a behavioral cloning penalty (Fujimoto and Gu, 2021). The recursive structure of goal-conditioned RL also admits a variety of non-traditional value learning algorithms based on successor measures (Eysenbach et al., 2022) and quasimetric structure (Wang et al., 2023; Park et al., 2026). A.2 The curse of horizon The challenges introduced by horizon in sequential decision-making tasks are well known. In addition to the error bounds of Ross and Bagnell (2010) in imitation learning, reinforcement learning introduces a whole host of additional horizon-related difficulties. The effective task horizon, set by the discount factor γ, is known to control the complexity of the policy class (Jiang et al., 2015; Jiang et al., 2016), and lower discount factors can provide pessimistic (Hu et al., 2022) and regularizing effects (Amit et al., 2020). However, artificially decreasing the discount factor is not feasible when the true task horizon is long and reward signals are sparse, as in goal-reaching tasks. Increasing the discount factor in off-policy RL leads to a host of additional problems, typically attributed to bias accumulation in value learning (Park et al., 2025) and difficulty in advantage estimation for primitive actions in policy extraction. To mitigate these difficulties, recent approaches have employed various techniques to reduce the effective horizon relative to decision-making frequency, both for value learning (Ahn et al., 2025; Zheng et al., 2026) and policy extraction (Li et al., 2025; Li et al., 2026b). Our work specifically focuses on the effects of horizon on goal-conditioned policy learning, where we identify a particular informational curse of horizon: as the horizon increases, the informative signal between the policy and the goal diminishes, causing the policy to overfit to spurious state features at the expense of goal conditioning. A.3 Steerable robot policies Effective goal conditioning is increasingly critical for modern robotic systems, including subgoal-conditioned policies (Zhou et al., 2025; Physical Intelligence et al., 2026) and inverse-kinematic-style World Action Models (Du et al., 2023; Li et al., 2026a). Recent works demonstrate the significance of effectively utilizing goal information: Mishra et al. (2026) links video-action model (VAM) performance to a model’s reliance on the generated future frames, as measured through cross-attention ratios. Korniak et al. (2026) shows that action chunking increases the mutual information between the state-action pair and the goal, which facilitates critic learning in goal-conditioned RL. Appendix B Extended Proofs Below, we use uppercase letters (e.g., StS_t) to denote random variables, and lower-case letters (e.g., sts_t) to denote their instantiations. Lemma B.1. Let P be a base probability distribution and let f(x)f(x) be a bounded scalar function such that f(x)∈[a,b]f(x)∈[a,b] for all x in the support of P. Define the exponentially reweighted distribution PfP_f as Pf(x)=P(x)ef(x)ZP_f(x)= P(x)e^f(x)Z, where Z=X∼P[ef(X)]Z=E_X P[e^f(X)] is the partition function. Then, the Kullback-Leibler divergence is bounded by: DKL(Pf∥P)≤(b−a)28D_KL(P_f\|P)≤ (b-a)^28 Proof. The KL divergence between a base distribution P and its exponentially reweighted version Pf∝PefP_f Pe^f can be expressed in terms of the log-partition function ψ(λ)=logX∼P[eλf(X)]ψ(λ)= _X P[e^λ f(X)], where λ∈[0,1]λ∈[0,1] is a scalar parameter that defines the intermediate distributions between the base distribution and the exponentially reweighted target. By taking a second-order Taylor expansion of ψ(λ)ψ(λ), we can rewrite this KL divergence as the remainder 12ψ′(c) 12ψ (c) for some c∈(0,1)c∈(0,1). Because f(X)f(X) is bounded in [a,b][a,b], we can apply Hoeffding’s Lemma (see Lemma 2.2 in Boucheron et al. (2013)) to bound this second derivative, yielding a maximum KL divergence of (b−a)28 (b-a)^28. ∎ Theorem B.2 (Divergence Bound from the Goal-Agnostic Prior). Let π∗(a∣s,g)∝pHR(a∣s,g)eQ(s,a,g)/απ^*(a s,g) p_HR(a s,g)e^Q(s,a,g)/α be the target distribution induced by the KL-regularized policy update, where α is a temperature parameter and pHRp_HR is the distribution of tuples (S,A,G)=(St,At,St+K)(S,A,G)= (S_t,A_t,S_t+K ) induced by the dataset trajectory distribution and a hindsight relabeling rule K∼qHRK q_HR. Let β(a∣s)β(a s) be the goal-agnostic behavior policy induced by the dataset, and let the critic value span be ΔQ(s,g)=supaQ(s,a,g)−infaQ(s,a,g) _Q(s,g)= _aQ(s,a,g)- _aQ(s,a,g) over the support of pHRp_HR. Then, the expected squared total variation distance between the two policies is bounded by: S,G[DTV(π∗(⋅∣S,G),β(⋅∣S))2]≤S,G[ΔQ(S,G)2]8α2+IHR(A;G∣S)E_S,G [D_TV (π^*(· S,G),β(· S) )^2 ]≤ E_S,G [ _Q(S,G)^2 ]8α^2+I_HR(A;G S) Proof. We bound the total variation distance (TVD) between the target policy π∗(⋅∣s,g)π^*(· s,g) and the behavioral policy β(⋅∣s)β(· s) by introducing the hindsight-relabeled distribution pHR(⋅∣s,g)p_HR(· s,g) as an intermediate distribution. We denote π∗(⋅∣s,g)π^*(· s,g) as π∗π^* below for brevity. The same notation applies to β(⋅∣s)β(· s) and pHR(⋅∣s,g)p_HR(· s,g). Because TVD is a metric, it satisfies the triangle inequality. For any given state s and goal g: DTV(π∗,β)≤DTV(π∗,pHR)+DTV(pHR,β).D_TV(π^*,β)≤ D_TV(π^*,p_HR)+D_TV(p_HR,β). After squaring both sides and applying the inequality (x+y)2≤2x2+2y2(x+y)^2≤ 2x^2+2y^2, we have DTV(π∗,β)2≤2DTV(π∗,pHR)2+2DTV(pHR,β)2.D_TV(π^*,β)^2≤ 2D_TV(π^*,p_HR)^2+2D_TV(p_HR,β)^2. Pinsker’s inequality, DTV(P,Q)2≤12DKL(P∥Q)D_TV(P,Q)^2≤ 12D_KL(P\|Q), applied to 2DTV(π∗,pHR)22D_TV(π^*,p_HR)^2, gives us 2DTV(π∗,pHR)2≤DKL(π∗∥pHR).2D_TV(π^*,p_HR)^2≤ D_KL(π^*\|p_HR). Because π∗π^* is an exponentially reweighted version of pHRp_HR, we can apply Lemma 1 to DKL(π∗∥pHR)D_KL(π^*\|p_HR) with f(a)=Q(s,a,g)/αf(a)=Q(s,a,g)/α, yielding DKL(π∗∥pHR)≤ΔQ(s,g)28α2.D_KL(π^*\|p_HR)≤ _Q(s,g)^28α^2. Now we have shown DTV(π∗,β)2≤ΔQ(s,g)28α2+2DTV(pHR,β)2.D_TV(π^*,β)^2≤ _Q(s,g)^28α^2+2D_TV(p_HR,β)^2. (5) Next, we apply Pinsker’s inequality again to 2DTV(pHR,β)22D_TV(p_HR,β)^2, leading to 2DTV(pHR,β)2≤DKL(pHR(⋅∣s,g)∥β(⋅∣s)).2D_TV(p_HR,β)^2≤ D_KL(p_HR(· s,g)\|β(· s)). Note that hindsight relabeling does not change the marginal state distribution, implying β(a∣s)=pHR(a∣s)β(a s)=p_HR(a s). Then taking the expectation of this KL divergence over S and G yields the conditional mutual information between actions and goals given the state: S,G[DKL(pHR(⋅∣S,G)∥pHR(⋅∣S))]=IHR(A;G∣S)E_S,G [D_KL(p_HR(· S,G)\|p_HR(· S)) ]=I_HR(A;G S) (6) Taking the expectation of equation 5 over S and G and combining it with equation 6 concludes the proof. ∎ Proposition B.3 (Monotonic Decay of Mutual Information). Let dataset trajectories be generated by a Markovian behavior policy and transition kernel. For any time horizon (relabeling offset) k′>k>0k >k>0, given the current state StS_t, the sequence At→St+k→St+k′A_t→ S_t+k→ S_t+k forms a conditional Markov chain. The mutual information between the action and future states monotonically decreases over time: I(At;St+k∣St) I (A_t;S_t+k S_t ) −I(At;St+k′∣St) -I (A_t;S_t+k S_t ) =St,St+k,St+k′[DKL(p(At∣St,St+k)∥p(At∣St,St+k′))]≥0 =E_S_t,S_t+k,S_t+k [D_KL (p(A_t S_t,S_t+k)\|p(A_t S_t,S_t+k ) ) ]≥ 0 This inequality is strict when there exist at least two intermediate states, st+k(1)s_t+k^(1) and st+k(2)s_t+k^(2), that share a common future successor state st+k′s_t+k with non-zero probability, but yield distinct intermediate action posteriors p(At∣st,st+k(1))≠p(At∣st,st+k(2))p(A_t s_t,s_t+k^(1))≠ p(A_t s_t,s_t+k^(2)). Proof. Expand I(At;St+k,St+k′∣St)I(A_t;S_t+k,S_t+k S_t) with mutual information chain rule in both ways and leverage I(At;St+k′∣St,St+k)=0I (A_t;S_t+k S_t,S_t+k )=0 implied by the Markov chain, we have I(At;St+k∣St)−I(At;St+k′∣St)=I(At;St+k∣St,St+k′)≥0.I (A_t;S_t+k S_t )-I (A_t;S_t+k S_t )=I (A_t;S_t+k S_t,S_t+k )≥ 0. Next, we rewrite I(At;St+k∣St,St+k′)I (A_t;S_t+k S_t,S_t+k ) as an expected KL divergence I(At;St+k∣St,St+k′)=St,St+k′[DKL(p(At,St+k∣St,St+k′)∥p(At∣St,St+k′)p(St+k∣St,St+k′))]. split&I(A_t;S_t+k S_t,S_t+k )\\ & =E_S_t,S_t+k [D_KL (p(A_t,S_t+k S_t,S_t+k )\|p(A_t S_t,S_t+k )p(S_t+k S_t,S_t+k ) ) ]. split Since p(At,St+k∣St,St+k′)=p(St+k∣St,St+k′)p(At∣St,St+k)p(A_t,S_t+k S_t,S_t+k )=p(S_t+k S_t,S_t+k )p(A_t S_t,S_t+k), we can simplify the KL term I(At;St+k∣St,St+k′)=St,St+k′,(St+k∣St,St+k′)[DKL(p(At∣St,St+k)∥p(At∣St,St+k′))]. split&I(A_t;S_t+k S_t,S_t+k )\\ & =E_S_t,S_t+k ,(S_t+k S_t,S_t+k ) [D_KL (p (A_t S_t,S_t+k )\|p (A_t S_t,S_t+k ) ) ]. split To see when this inequality is strict (i.e., when mutual information strictly decreases with an increase in relabeling horizon), we compare the intermediate action posterior p(At∣St,St+k)p(A_t S_t,S_t+k) and the future action posterior p(At∣St,St+k′)p(A_t S_t,S_t+k ) within the KL term. We instantiate some start state sts_t and future state st+k′s_t+k to focus on the KL term inside the expectation. By the law of total probability, the future posterior is the marginalization of the intermediate posteriors over all possible intermediate states: p(At∣st,st+k′)=St+k|st,st+k′[p(At∣st,St+k)].p(A_t s_t,s_t+k )=E_S_t+k s_t,s_t+k [p(A_t s_t,S_t+k)]. Now, to make the KL term non-zero, we need St+k|st,st+k′[p(At∣st,St+k)]≠p(At∣st,st+k)E_S_t+k s_t,s_t+k [p(A_t s_t,S_t+k)]≠ p(A_t s_t,s_t+k) for some instantiation st+ks_t+k. In other words, the intermediate posterior p(At∣st,St+k)p(A_t s_t,S_t+k) cannot be almost-surely constant with respect to St+kS_t+k. For the intermediate posterior p(At∣st,St+k)p(A_t s_t,S_t+k) to vary depending on St+kS_t+k, we need at least two instantiations, st+k(1)s_t+k^(1) and st+k(2)s_t+k^(2), that simultaneously satisfy the two conditions below: Merging paths: both intermediate states must be viable nodes on the path from the start state sts_t to the future state st+k′s_t+k . In other words, they must both occur with non-zero probability given the endpoints: p(st+k(1)∣st,st+k′)>0andp(st+k(2)∣st,st+k′)>0p(s_t+k^(1) s_t,s_t+k )>0 p(s_t+k^(2) s_t,s_t+k )>0 Different actions: knowing which of these intermediate states the agent passed through must change our belief about the action taken at time t: p(At∣st,st+k(1))≠p(At∣st,st+k(2))p(A_t s_t,s_t+k^(1))≠ p(A_t s_t,s_t+k^(2)) The two conditions are satisfied and strict loss of information occurs in environments and datasets where distinct trajectories eventually “merge” at a shared future state. Intuitively, this can occur in two common scenarios. First, when learning from a sub-optimal policy, subsequent behavior can overwrite the effects of earlier actions. An agent might make a mistake and later correct it, reaching the same future state as a trajectory that took an optimal path. Second, merging occurs when there are multiple valid ways to accomplish the same goal. For example, in a task requiring an agent to pick and place three cubes, there are six valid orderings that all lead to the same final state. In both scenarios, longer task and relabeling horizons provide more opportunities for these action-distinguishable trajectories to merge. Thus, temporally distant relabeled goals become progressively less informative about the original action. Our analysis extends to distributions over relabeling offsets when K∼qHRK q_HR is sampled independently of the dataset trajectory. The chain rule for mutual information gives IHR(A;G∣S)=K∼qHR[I(At;St+K∣St)]−IHR(A;K∣S,G)≤K∼qHR[I(At;St+K∣St)]I_HR(A;G S)=E_K q_HR[I(A_t;S_t+K S_t)]-I_HR(A;K S,G) _K q_HR[I(A_t;S_t+K S_t)]. Therefore, by Proposition B.3, shifting qHRq_HR towards longer offsets decreases this upper bound on the action-goal mutual information. ∎ Appendix C Algorithm Pseudocode Algorithm 1 Sobolev Bootstrapping 1: Input: offline dataset D, goal distribution p(g)p(g). 2: Initialize value function VβV_β, target subpolicy πϕsubπ^sub_φ, and policy πθ _θ. 3: while not converged do 4: Train value function: β←β−λ∇βℒGCIVL(β)β←β-λ _βL_GCIVL(β) with (st,st+1)∼p,g∼p(g)(s_t,s_t+1) p^D,g p(g) [Equation 9] 5: end while 6: while not converged do 7: Train target subpolicy: ϕ←ϕ+λ∇ϕAWR(ϕ)φ←φ+λ _φJ_AWR(φ) with (st,a,sg)∼p(s_t,a,s_g) p^D [Equation 13] 8: end while 9: while not converged do 10: Train policy: θ←θ−λ∇θ(SB(θ)−SAW(θ))θ←θ-λ _θ (J_SB(θ)-J_SAW(θ) ) with (st,a,sg)∼p,g∼p(g)(s_t,a,s_g) p^D,g p(g) [Equation 4 & 12] 11: end while Appendix D Algorithmic Details In this section, we provide full details of the value and policy learning objectives for our Sobolev bootstrapping algorithm, which is instantiated on top of Subgoal Advantage-Weighted Policy Bootstrapping (Zhou and Kao, 2025, SAW). D.1 Goal-conditioned implicit value learning (GCIVL) To learn a value function V(s,g)V(s,g) that can operate both on single- and multi-step transitions, we use a goal-conditioned variant of implicit Q-learning (IQL) (Kostrikov et al., 2021), which approximates the max operator with an expectile regression on in-sample state-action pairs: ℒV(βV)=(s,a,g)∼D[L2τ(Qβ¯Q(s,a,g)−VβV(s,g))] _V( _V)=E_(s,a,g) D [L_2^τ(Q_ β_Q(s,a,g)-V_ _V(s,g)) ] (7) ℒQ(βQ)=(s,a,s′,g)∼D[(r(s,g)+γVβV(s′,g)−QβQ(s,a,g))2], _Q( _Q)=E_(s,a,s ,g) D [(r(s,g)+γ V_β_V(s ,g)-Q_ _Q(s,a,g))^2 ], (8) where L2τ(x)=|τ−(x<0)|x2L_2^τ(x)=|τ- 1(x<0)|x^2 is the expectile loss and τ∈[0.5,1)τ∈[0.5,1) is the expectile parameter. Following previous works (Park et al., 2023), we adopt an action-free variant of this objective, which estimates the state-action value function Q(s,a,g)Q(s,a,g) as r(s,g)+γV¯(s′,g)r(s,g)+γ V(s ,g), leading to ℒGCIVL(β)=(s,s′,g)∼D[L2τ(r(s,g)+γVβ¯(s′,g)−Vβ(s,g))].L_GCIVL(β)=E_(s,s ,g) D [L_2^τ(r(s,g)+γ V_ β(s ,g)-V_β(s,g)) ]. (9) Note that using the action-free estimator with expectile regression is optimistically biased in stochastic environments, since it directly regresses towards high-value transitions without using Q-values to marginalize over environment stochasticity. However, it performs well in practice and can be used to measure the relative advantages of intermediate subgoals without modification. D.2 Offline policy extraction The above value iteration algorithm already performs implicit policy improvement. Then, we simply need to extract the optimal policy from the learned value function. To remain inside the convex hull of dataset actions and avoid querying the value function for out-of-distribution actions, we use a form of weighted behavioral cloning called Advantage-Weighted Regression (Peng et al., 2019, AWR): πθ(θ)=(s,a,g)∼D[exp(α(Q¯(s,a,g)−V(s,g)))logπθ(a∣s,g)], _ _θ(θ)=E_(s,a,g) D [ (α( Q(s,a,g)-V(s,g))) _θ(a s,g) ], (10) where α∈[0,∞)α∈[0,∞) is an inverse temperature parameter. Because we do not directly parameterize a critic Q(s,a,g)Q(s,a,g) in GCIVL [Equation 9], we instead estimate the “one-step” advantage Q¯(s,a,g)−V(s,g) Q(s,a,g)-V(s,g) as V(s′,g)−V(s,g)V(s ,g)-V(s,g) to reweight the corresponding action in the transition tuple (s,a,s′)(s,a,s ). This estimate is biased in stochastic environments, since it optimistically assumes that the transition from s to s′s was caused by action a. D.3 Policy bootstrapping algorithms Policy bootstrapping algorithms are based on the insight that, if an action a is optimal for reaching a subgoal sgs_g along the optimal trajectory toward goal g, then a should also be optimal for reaching g. To evaluate the optimality of subgoals sgs_g, we can use a multistep variant of the action-free advantage estimator for GCIVL, where we approximate A(s,sg,g)A(s,s_g,g) as V(sg,g)−V(s,g)V(s_g,g)-V(s,g). Reinforcement Learning with Imagined Subgoals [RIS] (Chane-Sane et al., 2021): To produce subgoals, RIS first learns a generative model πψh(sg∣s,g) _ψ^h(s_g s,g) to produce “imagined” subgoals sgs_g, and then bootstraps a goal-conditioned flat policy πθ(a∣s,g) _θ(a s,g) on a short-horizon policy πsub(a∣s,sg)π^sub(a s,s_g) conditioned on the imaginary way-points. The generative model πψh(sg∣s,g) _ψ^h(s_g s,g) is optimized to produce subgoals with high advantages while remaining close to the dataset state distribution μD _D, in a manner similar to AWR: πψh(ψ)=(s,sg,g)∼D[exp(αA(sg∣s,g))logπψh(sg∣s,g)].J_ _ψ^h(ψ)=E_(s,s_g,g) D [ (α A(s_g s,g)) _ψ^h(s_g s,g) ]. (11) Subgoal Advantage-Weighted Policy Bootstrapping [SAW] (Zhou and Kao, 2025): SAW introduces a unifying view of RIS and a related hierarchical method (Park et al., 2023) under the “RL as inference” framework (Levine, 2018). Rather than modeling the optimal subgoal distribution p∗(sg∣s,g)p^*(s_g s,g). SAW samples in-trajectory waypoints in dataset and uses a similar multistep advantage weight to prioritize bootstrapping on high-advantage subgoals. p∗(sg∣s,g)∝pD(sg∣s)exp(A(sg∣s,g)).p^*(s_g s,g) p^D(s_g s) (A(s_g s,g)). This eliminates the need for a parametric subgoal generator and replaces Equation 11 and 14 with πθSAW(θ)=(s,a,sg,g)∼D[eα1A(a∣s,g)logπθ(a∣s,g)−eα2A(sg∣s,g)DKL(πθ∥πsub)],J^SAW_ _θ(θ)=E_(s,a,s_g,g) D [e _1A(a s,g) _θ(a s,g)-e _2A(s_g s,g)D_KL( _θ\|π^sub) ], (12) where the target subpolicy πsubπ^sub is extracted with Equation 13, πθ _θ is a shorthand for πθf(⋅∣s,g) _θ^f(· s,g), and πsubπ^sub denotes πsub(⋅∣s,sg)π^sub(· s_,s_g). Following Zhou and Kao (2025), we implement the offline variant of RIS and SAW with an independently parameterized short-horizon policy with AWR, instead of using an exponential moving average of the long-horizon policy πθπ^θ as in the original implementation by Chane-Sane et al. (2021): πsub(ϕ)=(s,a,sg)∼D[exp(αA(a∣s,sg))logπsub(a∣s,sg)].J_π^sub(φ)=E_(s,a,s_g) D [ (α A(a s,s_g)) π^sub(a s,s_g) ]. (13) Finally, we train the long-horizon policy πθ _θ with “one-step” transitions while regressing it toward the short-horizon policy πsubπ^sub conditioned on generated subgoals: πθRIS(θ)=(s,a,g)∼D,sg∼πψh(⋅∣s,g)[exp(αA(a∣s,g))logπθ(a∣s,g)−DKL(πθ∥πsub)],J^RIS_ _θ(θ)=E_(s,a,g) D,s_g _ψ^h(· s,g) [ (α A(a s,g)) _θ(a s,g)-D_KL( _θ\|π^sub) ], (14) where the “one-step” advantage A(a∣s,g)A(a s,g) is estimated by VβV(s′,g)−VβV(s,g)V_ _V(s ,g)-V_ _V(s,g), πθ _θ is a shorthand for πθf(⋅∣s,g) _θ^f(· s,g), and πsubπ^sub denotes πsub(⋅∣s,sg)π^sub(· s_,s_g). Appendix E Additional Figures Figure 5: Training on distant relabeled goals harms nearby goal-reaching capabilities. We record the success rates across learned policies trained on a uniform distribution of relabeling goal offsets [1,Htrain][1,H_train], where Htrain=∞H_train=∞ corresponds to a uniform distribution over all future in-trajectory states. At evaluation, policies are given optimal HplanH_plan-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds. Figure 6: Training on a larger proportion of distant relabeled goals harms nearby goal-reaching capabilities. We record the success rates across learned policies trained on a geometric distribution of relabeling goal offsets with discount γ, where γ=1γ=1 corresponds to a uniform distribution over future states. At evaluation, policies are given optimal HplanH_plan-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds. Figure 7: Training on more random relabeled goals harms nearby goal-reaching capabilities. We record the success rates across learned policies trained on a mixture of goals either sampled from a uniform distribution over future states or from an arbitrary dataset state with probability prand∈0.25,0.50,0.75p_rand∈\0.25,0.50,0.75\. At evaluation, policies are given optimal HplanH_plan-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds. Figure 8: Increasing training goal horizon globally decreases policy goal sensitivity. We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with log2 _2 scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon. Figure 9: Training on distant relabeled goals globally decreases policy goal sensitivity. We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with log2 _2 scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon. Figure 10: Random goal sampling globally decreases policy goal sensitivity. We directly reduce the action-goal MI by replacing in-trajectory goals with randomly sampled dataset states with probability prandom∈0.25,0.5,0.75p_random∈\0.25,0.5,0.75\, and measure the goal Jacobian norm proportion at fixed state-goal offsets. Figure 11: Sobolev bootstrapping pays attention to the correct object during optimal trajectories. To enable fair comparisons across later task stages, we plot the input Jacobian norms with respect to each object over an independent scripted optimal trajectory Ahn et al. (2025) for the cube-triple triple pick-and-place task. S2AW (top) allocates a higher proportion of the Jacobian norm to the focal cube both during approach and grasp, whereas SAW’s (bottom) Jacobian norm only changes after the cube is already grasped. The shaded area denotes when the cube is actively grasped. Appendix F Hyperparameters We provide hyperparameters related to our Sobolev bootstrapping objective in Table 3. We use value learning hyperparameters from Zhou and Kao (2025) for HIQL, SAW, and S2AW in cube-single, which significantly improves performance. If not listed, all other hyperparameters follow the default values from Park et al. (2024) and Zhou and Kao (2025). Table 3: Hyperparameters for SAW with Sobolev bootstrapping. Dataset Expectile τ AWR α KLD β Subgoal steps k Sobolev weight λ pointmaze-medium-navigate-v0 0.70.7 3.03.0 3.03.0 2525 0.030.03 pointmaze-large-navigate-v0 0.70.7 3.03.0 3.03.0 2525 1.001.00 pointmaze-giant-navigate-v0 0.70.7 3.03.0 3.03.0 2525 1.001.00 antmaze-medium-navigate-v0 0.70.7 3.03.0 3.03.0 2525 0.030.03 antmaze-large-navigate-v0 0.70.7 3.03.0 3.03.0 2525 0.030.03 antmaze-giant-navigate-v0 0.70.7 3.03.0 3.03.0 2525 0.300.30 humanoidmaze-medium-navigate-v0 0.70.7 3.03.0 3.03.0 100100 0.030.03 humanoidmaze-large-navigate-v0 0.70.7 3.03.0 3.03.0 100100 0.030.03 humanoidmaze-giant-navigate-v0 0.70.7 3.03.0 3.03.0 100100 0.030.03 cube-single-play-v0 0.90.9 3.03.0 0.30.3 1010 0.020.02 cube-double-play-v0 0.70.7 3.03.0 1.01.0 1010 0.020.02 cube-triple-play-v0 0.70.7 3.03.0 1.01.0 1010 0.030.03 cube-quadruple-play-v0 0.70.7 3.03.0 1.01.0 1010 0.030.03 scene-play-v0 0.70.7 3.03.0 1.01.0 1010 0.020.02 F.1 Sensitivity Analysis We examine sensitivity to the choice of Sobolev weight λ in Table 4. We do not perform additional tuning of the expectile τ, AWR α, subgoal steps k, and KLD β, which are common hyperparameters shared by HIQL and SAW. Our results show that smaller weights can be insufficient to realize the benefits of Sobolev training, while larger weights may “oversmooth” in some manipulation environments. Nevertheless, we observe consistently substantial gains in cube-triple and antmaze-giant across all values of λ. Table 4: Sobolev loss weight sensitivity analysis. Sobolev bootstrapping is robust to different weighting factors within a reasonable scale (around 0.01 for cube-double and scene). Notably, improvements on antmaze-giant and cube-triple are robust across various weighting scales. Experimental results are averaged across four seeds. Dataset λ=λ=0 0.0010.001 0.010.01 0.10.1 cube-double-play-v0 4040 ±7± 7 4040 ±11± 11 5555 ±7± 7 3535 ±9± 9 cube-triple-play-v0 44 ±2± 2 99 ±2± 2 3030 ±7± 7 2323 ±10± 10 scene-play-v0 6363 ±6± 6 5454 ±10± 10 7171 ±5± 5 6262 ±5± 5 antmaze-giant-navigate-v0 7373 ±4± 4 7676 ±4± 4 8282 ±4± 4 8585 ±4± 4 Appendix G Computational Resources All experiments were run on an internal cluster of NVIDIA 5090 and NVIDIA PRO 6000 Blackwell GPUs. In Table 5, We select the highest-dimensional environments and report the hours required to complete a standard 1M-step training run on an NVIDIA 5090 GPU. Table 5: Training time comparison. We select the highest-dimensional environments and report the hours required to complete a standard 1M-step training run on an NVIDIA 5090 GPU. hours / 1M iters GCIQL SAW S2AW cube-quadruple-play 0.26 0.46 0.75 scene-play 0.25 0.45 0.74 humanoidmaze-giant-navigate 0.26 0.46 1.75 Appendix H Additional Experiments H.1 Architectural ablations While we have made the case that differences in state-goal distributions are responsible for the improved properties of the target policy πsubπ^sub, we note that the experiments in Table 1 use different architectures for each policy. The full goal-conditioned policy π consists of a three-layer MLP, followed by a length-normalized bottleneck of dimension 10, and another three-layer MLP; the target policy consists only of a three-layer MLP. Then, our results may be explained by a natural simplicity bias introduced by smaller network sizes. To control for this potential confound, we train Sobolev SAW using identical architectures for πsubπ^sub and π, which we denote as S2AWℓ. Table 6: Architectural ablations to the Sobolev SAW target actor. We report the performance of Sobolev SAW where the target actor has the same architecture as the full policy, averaged over 44 seeds with standard deviations after the ± sign. Numbers within 5%5\% of the best value in the row are in bold. Environment Dataset GCBC GCIVL GCIQL QRL CRL HIQL SAW S2AW S2AWℓ maze-giant antmaze-giant-navigate-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 1414 ±3± 3 1616 ±3± 3 6565 ±5± 5 7373 ±4± 4 87 ±3± 3 8080 ±5± 5 humanoidmaze-giant-navigate-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 11 ±0± 0 33 ±2± 2 1212 ±4± 4 3535 ±4± 4 38 ±6± 6 3535 ±5± 5 cube cube-single-play-v0 66 ±2± 2 5353 ±4± 4 6868 ±6± 6 55 ±1± 1 1919 ±2± 2 4444 ±9± 9 7272 ±5± 5 86 ±5± 5 7979 ±8± 8 cube-double-play-v0 11 ±1± 1 3636 ±3± 3 4040 ±5± 5 11 ±0± 0 1010 ±2± 2 66 ±2± 2 4040 ±7± 7 5050 ±6± 6 55 ±8± 8 cube-triple-play-v0 11 ±1± 1 11 ±0± 0 33 ±1± 1 00 ±0± 0 4 ±1± 1 33 ±1± 1 44 ±2± 2 30 ±5± 5 30 ±4± 4 cube-quadruple-play-v0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 00 ±0± 0 3 ±2± 2 22 ±1± 1 scene scene-play-v0 55 ±1± 1 4242 ±4± 4 5151 ±4± 4 55 ±1± 1 1919 ±2± 2 3838 ±3± 3 6363 ±6± 6 73 ±6± 6 72 ±5± 5 We find that results remain consistent across all manipulation environments where combinatorial generalization is most important, supporting the primary conclusion that the difference in training curricula, and not architectural differences, produces desirable properties of input gradients. Performance does drop slightly in locomotion environments compared to Sobolev SAW with the smaller target actor, suggesting that a smaller actor network does provide some beneficial regularization. Indeed, we observe that Sobolev bootstrapping with a smaller target policy achieves lower SAW validation loss than its counterpart. H.2 Extending Sobolev bootstrapping to RIS Adding Sobolev bootstrapping to RIS (Chane-Sane et al., 2021) also yields significant gains, albeit less consistently. In our implementation of RIS, subgoals generated by πψh _ψ^h lie in a lower-dimensional latent space rather than the raw observation space, making it impractical to compute advantages directly from the learned value function VβV_ _V. Thus, we adopt a constant weighting scheme for the Sobolev bootstrapping loss, under the assumption that all generated subgoals are near-optimal. If the subgoal generator proposes suboptimal waypoints, then the uniformly weighted Sobolev term in RIS can force the flat policy to match noisy input gradients, which can degrade its performance. Table 7: Evaluating RIS with Sobolev bootstrapping. We evaluate the effectiveness of Sobolev bootstrapping on RIS by comparing it against the numbers reported in Zhou and Kao (2025). Results are averaged over 4 seeds, and other settings are identical to Table 1. Dataset RIS Sobolev RIS pointmaze-medium-navigate-v0 8888 ±6± 6 96 ±1± 1 pointmaze-large-navigate-v0 6363 ±13± 13 75 ±3± 3 pointmaze-giant-navigate-v0 5757 ±12± 12 67 ±5± 5 antmaze-medium-navigate-v0 96 ±1± 1 97 ±1± 1 antmaze-large-navigate-v0 89 ±3± 3 90 ±1± 1 antmaze-giant-navigate-v0 65 ±4± 4 69 ±0± 0 humanoidmaze-medium-navigate-v0 7373 ±5± 5 84 ±7± 7 humanoidmaze-large-navigate-v0 2121 ±7± 7 47 ±2± 2 humanoidmaze-giant-navigate-v0 33 ±2± 2 13 ±3± 3 cube-single-play-v0 81 ±6± 6 5959 ±3± 3 cube-double-play-v0 3636 ±4± 4 41 ±2± 2 cube-triple-play-v0 3 ±2± 2 3 ±0± 0 cube-quadruple-play-v0 0 ±0± 0 0 ±0± 0 scene-play-v0 64 ±7± 7 66 ±1± 1 Furthermore, it is well-known that planning with a latent subgoal representation extracted from an intermediate layer of the value function, as done in Park et al. (2023) and our implementation of RIS, is suboptimal and can significantly hamper performance in larger environments such as antmaze-giant and humanoidmaze-giant (Zhou and Kao, 2025; Ahn et al., 2025). Although not a focus of our work, we expect that improvements to the high-level policy, such as modern generative models and new representation learning objectives, may significantly improve RIS’s performance. H.3 Additional horizon reduction baselines Existing horizon-reduction approaches address one of two issues: compounding bias in off-policy value learning, e.g., n-step TD (Park et al., 2025), QC (Li et al., 2025; Li et al., 2026b), OTA (Ahn et al., 2025); or noisy advantage estimates during policy extraction, e.g., HIQL (Park et al., 2023), SAW (Zhou and Kao, 2025). In contrast, our work identifies an additional bottleneck within the actor itself, related to decreased policy sensitivity to goal information scaling with horizon. For completeness, we compare our method against OTA (Ahn et al., 2025), DQC (Li et al., 2026b), and SHARSA (Park et al., 2025). Because the original OTA paper did not report results on state-based manipulation tasks, we use the official implementation and sweep the abstraction factor n∈5,10,20n∈\5,10,20\. Furthermore, adapting DQC and SHARSA to our standard dataset is non-trivial, as they were evaluated using larger datasets with oracle representations. To ensure a fair comparison, we adopt the documented hyperparameters and use the standard batch and model sizes consistent with the OGBench suite. Our results demonstrate that OTA, DQC, and SHARSA underperform with the standard datasets. Table 8: Additional horizon reduction baselines. The additional baselines do not readily generalize to the standard datasets and perform poorly. All results are averaged across 4 seeds. Environment OTA n=5 OTA n=10 OTA n=20 SHARSA DQC S2AW antmaze-giant-navigate 77±4± 4 - - 72±1± 1 25±6± 6 87±3± 3 cube-double-play 2±0± 0 3±1± 1 2±1± 1 5±1± 1 24±4± 4 57±6± 6 cube-triple-play 1±0± 0 0±0± 0 1±0± 0 2±1± 1 11±4± 4 30±5± 5 H.4 Uniform Jacobian regularization Table 9: Uniform Jacobian regularization. We apply ℓ1 _1 and ℓ2 _2 norm penalization with coefficient 0.30.3 to the SAW baseline. We observe that ℓ1 _1 penalization is more effective, leading to notable performance gains in cube-double and scene environments. Results are averaged across four seeds and five evaluation tasks. SAW SAWℓ1_ _1 SAWℓ2_ _2 S2AW antmaze-giant-navigate 73±4± 4 72±3± 3 74±2± 2 87±3± 3 cube-double-play 40±7± 7 47±10± 10 37±7± 7 57±6± 6 cube-triple-play 4±2± 2 5±4± 4 4±1± 1 30±5± 5 scene-play 63±6± 6 69±8± 8 62±2± 2 74±6± 6