Paper deep dive
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
Tingrui Huang, Ai Bo, Junzhe Zhang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
Tags
Links
- Source: https://arxiv.org/abs/2610.14953v1
- Canonical: https://arxiv.org/abs/2610.14953v1
Trouble viewing inline? Open PDF directly →
Full Text
108,180 characters extracted from source content.
Expand or collapse full text
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning Tingrui Huang1 Ai Bo2 Junzhe Zhang2 1Eindhoven University of Technology, Eindhoven, The Netherlands 2Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY, USA Abstract Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state–action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action–outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks. 1 Introduction ×+D^+−D^-positive demonstrationsUUnegative demonstrationspessimistic counterfactualtransitionη(s,a,a′,s′)T_η(s,a,a ,s )ℒ+L^+ℒ−L^-ssaacounterfactual rollout×s+s^+observed future (confounded)φ(s,a) (s,a)ψ(⋅)ψ(·)×ℒ(s,a,s~+,s−)L(s,a, s^+,s^-)s~+ s^+(s,a)(s,a)negative sample s−∼P−(s)s^- P^-(s) Figure 1: Overview of CauCRL. A pessimistic counterfactual transition ηT_η is learned from positive demonstrations +D^+ and negative demonstrations −D^-, confounded by U. Its rollout from (s,a)(s,a) yields a counterfactual future s~+ s^+ that replaces the observed future s+s^+ as the positive sample, contrasted against a negative sample s−∼P−(s)s^- P^-(s). Learning to make good decisions from sparse feedback is a long-standing challenge in reinforcement learning (RL) (Sutton and Barto, 2018). In many real-world applications, the agent receives little or no reward signal: it may only observe expert demonstrations, or be told whether a task has been accomplished. Among various paradigms of learning from sparse feedback, imitation learning (Pomerleau, 1989; Ross et al., 2011; Ho and Ermon, 2016), which learns from demonstrations without any reward signal, and goal-conditioned RL (Kaelbling, 1993; Schaul et al., 2015), which learns to reach goals specified at test time, have emerged as crucial and popular directions. Goal-conditioned RL is particularly appealing, since a single learned policy can be reused across many tasks (Duan et al., 2016). A variety of algorithms have been proposed for goal-conditioned RL, including reward shaping (Ng et al., 1999), hindsight relabeling (Andrychowicz et al., 2017), goal-conditioned supervised learning (Ghosh et al., 2021), and universal value function approximation (Schaul et al., 2015), to name a few; see Liu et al. (2022) for a survey. Among them, contrastive RL is a simple and effective framework that trains a critic to distinguish future states of the same trajectory from states of unrelated trajectories, and uses this critic as a goal-conditioned value function (Eysenbach et al., 2021; Eysenbach et al., 2022). While successful in practice, these methods rely on the crucial assumption that the actions observed in the offline data, along with the subsequent states they produce, are not simultaneously influenced by unobserved confounders. If this assumption is violated, the probability of reaching the goal following a candidate policy becomes non-identifiable, i.e., it cannot be uniquely determined from the available data and model assumptions. The following example illustrates such challenges. Example 1 (Challenges of Unobserved Confounding). Consider the windy-corridor task described in Fig. 2 (left), where the agent must reach the goal without touching the lava (Bo et al., 2026). A short route passes through a corridor next to the lava, while a longer route is safe. At each time step t, a wind U(t)U^(t) blows through the corridor; when it is unfavorable, moving forward along the short route pushes the agent into the lava. The learner cannot observe the wind. Instead, it observes demonstrations from an expert who can: the expert moves forward along the short route when the wind is favorable and pauses otherwise, while a small fraction of demonstrations follow the longer route. The wind U(t)U^(t) thus becomes an unobserved confounder affecting both the observed action and the next state. We apply standard contrastive RL (Eysenbach et al., 2022) to these demonstrations. As shown in Fig. 2 (right), it fails to achieve a high success rate: since forward moves along the short route are almost always followed by reaching the goal in the demonstrations, the critic overvalues them, even though the learner cannot determine when they are safe. ∎ Figure 2: Windy corridor (left) and success rate over training (right): under confounded demonstrations, all baselines stay below the expert success rate of 1.01.0. Recently, a growing body of literature has explored the nuanced interactions between causal inference (Pearl, 2009; Bareinboim et al., 2022) and RL to address data biases in decision-making under uncertainty, known as causal reinforcement learning. Several algorithms have been proposed for various policy learning settings, including off-policy evaluation and learning (Kallus and Zhou, 2018; Namkoong et al., 2020; Kallus and Zhou, 2020; Bennett et al., 2021; Zhang and Bareinboim, 2025), offline-to-online learning (Zhang and Bareinboim, 2019), robust offline and deep RL (Joshi et al., 2024; Li et al., 2025; Li et al., 2026), meta-RL (Bo et al., 2026), and imitation learning (de Haan et al., 2019; Zhang et al., 2020; Ruan et al., 2024; Balazadeh et al., 2024), to name a few. Despite these progresses, a systematic approach for goal-conditioned RL under unobserved confounding is still missing. It is unclear how one can learn the goal-reaching occupancy underlying contrastive RL when the demonstrations that provide its positive samples are confounded. This paper aims to address this gap by proposing Causal Contrastive Reinforcement Learning (CauCRL; Fig. 1), which trains contrastive critics on pessimistic counterfactual rollouts that remain consistent with the confounded demonstrations. More specifically, our contributions are summarized as follows. (1) We formulate goal-conditioned RL from confounded demonstrations and introduce counterfactual transitions together with a worst-case occupancy measure that lower-bounds the true probability of reaching the goal (Sec. 2.1); (2) We develop a computational procedure for CauCRL that learns a pessimistic counterfactual transition from positive and negative demonstrations and uses its counterfactual rollouts as positive samples for contrastive critic learning (Sec. 3); (3) Finally, we validate CauCRL through simulations on goal-reaching tasks with confounded demonstrations, ranging from discrete navigation to high-dimensional continuous control (Sec. 4). Due to space constraints, all proofs are provided in Appendix A. Notations. We use capital letters to denote random variables (X), lowercase letters for their values (x), and calligraphic letters for their domains (X). For a set X, |||X| denotes its cardinality. We write P(X)P(X) for a probability distribution over X and P(Y∣X)P(Y X) for the set of conditional distributions P(Y∣X=x)P(Y X=x) over all realizations x. We abbreviate P(X=x)P(X=x) as P(x)P(x) and P(Y=y∣X=x)P(Y=y X=x) as P(y∣x)P(y x). Finally, Z=z 1\Z=z\ is an indicator that equals 11 if Z=zZ=z and 00 otherwise. 2 Goal-Conditioned RL with Unobserved Confounding We consider goal-conditioned RL (Eysenbach et al., 2022) from offline trajectories of a demonstrator that may use information unavailable to the learner, modeled by a confounded Markov decision process (Zhang and Bareinboim, 2016; Bo et al., 2026). Definition 1. A confounded goal-conditioned Markov decision process is a tuple ℳ=⟨,,,,ℱ,P⟩M= ,A,Y,U,F,P , where (1) S, A, and =0,1Y=\0,1\ are, respectively, the spaces of observed states, actions, and rewards; (2) U is the space of unobserved latent variables; (3) ℱF is a set of structural mechanisms fS:×→f_S:S×A×U and fA:×→f_A:S×U , where fSf_S determines the state transition and fAf_A determines the behavioral policy of the demonstrator; and (4) P is an exogenous distribution over U. Goals are states, s∗∈s^* , and the reward indicates whether the goal is reached, Y=S=s∗Y= 1\S=s^*\. The process starts from an initial state S(0)∼P(S(0))S^(0) P(S^(0)). At each time step t, a latent variable U(t)U^(t) is drawn independently from P(U)P(U), an action A(t)A^(t) is taken, and the environment moves to S(t+1)←fS(S(t),A(t),U(t))S^(t+1)← f_S(S^(t),A^(t),U^(t)) with reward Y(t)=S(t)=s∗Y^(t)= 1\S^(t)=s^*\. Since the structural functions and exogenous distributions are invariant with regard to time steps, we write (S,A,S′)(S,A,S ) for (S(t),A(t),S(t+1))(S^(t),A^(t),S^(t+1)) when the time index is clear. The learner acts through a policy π(a∣s)π(a s). Following a policy is an intervention do(π)do(π) that replaces the behavior mechanism fAf_A by π while leaving the remaining mechanisms unchanged (Pearl, 2009); similarly, do(A(t)=a)do(A^(t)=a) sets a single action to a. A subscript on a random variable denotes the corresponding counterfactual: Sa′S _a is the next state had the action been set to a. The resulting interventional transition is Pa(s′∣s)=P(Sa′=s′∣S=s)=∫fS(s,a,u)=s′dP(u)P_a(s s)=P(S _a=s S=s)= _U 1\f_S(s,a,u)=s \\,dP(u), and a policy π induces the interventional distribution over trajectories (s¯,a¯)=(s(0),a(0),s(1),a(1),…)( s, a)=(s^(0),a^(0),s^(1),a^(1),…), Pπ(s¯,a¯) P_π( s, a) =P(s(0))∏t≥0π(a(t)∣s(t))Pa(t)(s(t+1)∣s(t)). =P(s^(0)) _t≥ 0π(a^(t) s^(t))P_a^(t)(s^(t+1) s^(t)). For a state–action pair (s,a)(s,a) and goal s∗s^*, we define the discounted goal-reaching occupancy as dπ(s∗∣s,a) d^π(s^* s,a) =(1−γ)∑k=0∞γk =(1-γ) _k=0^∞γ^k (1) ×Pa,π(S(t+k)=s∗∣S(t)=s), × P_a,π\! (S^(t+k)=s^* S^(t)=s ), where Pa,πP_a,π sets A(t)A^(t) to a and lets subsequent actions follow π. Since Y(t+k)=S(t+k)=s∗Y^(t+k)= 1\S^(t+k)=s^*\, this occupancy coincides with the Q-function of π, dπ(s∗∣s,a)=Qπ(s,a):=(1−γ)a,π[∑k=0∞γkY(t+k)∣S(t)=s]d^π(s^* s,a)=Q^π(s,a):=(1-γ)\,E_a,π[ _k=0^∞γ^kY^(t+k) S^(t)=s] (Eysenbach et al., 2022). The learner seeks a policy π that maximizes this goal-reaching probability. Contrastive RL (Eysenbach et al., 2022) learns this occupancy with a critic fω(s,a,s∗)=φω(s,a)⊤ψω(s∗)f_ω(s,a,s^*)= _ω(s,a) _ω(s^*) by contrasting positive and negative samples. For a state–action pair (s,a)(s,a), a positive sample s+s^+ is a state from the discounted future of a trajectory under π, while a negative sample s−∼P−(s)s^- P^-(s) is drawn from a marginal state distribution that differs from the state distribution of the current policy estimate π. The critic is trained with the binary NCE objective maxω(s,a,s+)∼Pπs−∼P−[ _ω _ subarrayc(s,a,s^+) P_π\\ s^- P^- subarray [ logσ(fω(s,a,s+)) σ\! (f_ω(s,a,s^+) ) (2) +log(1−σ(fω(s,a,s−)))]. + \! (1-σ\! (f_ω(s,a,s^-) ) ) ]. At the optimum, the critic recovers the unnormalized occupancy of the positive samples (Eysenbach et al., 2022): expf∗(s,a,s∗)=dπ(s∗∣s,a)/P−(s∗) f^*(s,a,s^*)=d^π(s^* s,a)/P^-(s^*), i.e., QπQ^π up to an action-independent factor, so the policy can be improved by favoring actions with high critic values. In the offline setting, on the other hand, the learner cannot intervene and must instead learn from recorded trajectories of a demonstrator, who observes the latent variable and selects actions by the behavioral policy A(t)←fA(S(t),U(t))A^(t)← f_A(S^(t),U^(t)). Demonstrations therefore follow the observational distribution P(s¯,a¯) P( s, a) =P(s(0))∏t≥0∫fA(s(t),u)=a(t) =P(s^(0)) _t≥ 0 _U 1\f_A(s^(t),u)=a^(t)\ ×fS(s(t),a(t),u)=s(t+1)dP(u), × 1\f_S(s^(t),a^(t),u)=s^(t+1)\\,dP(u), in which the same latent realization u drives both the action and the transition. From these data, the learner can estimate the observational transition P(s′∣s,a)=P(S′=s′∣S=s,A=a)P(s s,a)=P(S =s S=s,A=a), but not the latent variable. The offline data consist of two parts. Positive demonstrations follow an observational distribution P+(s¯,a¯)P^+( s, a) of this form, induced by the expert behavioral policy fAf_A, and contain trajectories reaching the goal. Negative demonstrations are generated by a non-expert behavioral policy, such as a randomized one, but record the eventual states it reaches, such as the lava in Example 1. We denote their state distribution by P−(s)P^-(s), which serves as the negative distribution in Eq. (2).11 1 Unlike the original CRL (Eysenbach et al., 2022), negatives here are drawn from the non-expert demonstrations P−P^-, which we expect to sufficiently cover undesirable states for the task; Sec. 3 uses them as an inductive bias toward negative outcomes for counterfactual transitions. Offline CRL trains Eq. (2) on these data, taking the positive sample (s,a,s+)∼P+(s,a,s^+) P^+ from the discounted future of the same demonstration instead of from PπP_π. This change of data is where the guarantee breaks: because U affects both the demonstrator’s action and the next state, generally P(s′∣s,a)≠Pa(s′∣s)P(s s,a)≠ P_a(s s), so positive samples follow the observational counterpart d(s∗∣s,a)d(s^* s,a) of Eq. (1), with Pa,πP_a,π replaced by P(⋅∣S(t)=s,A(t)=a)P(· S^(t)=s,A^(t)=a). The critic then recovers d(s∗∣s,a)/P−(s∗)d(s^* s,a)/P^-(s^*) rather than QπQ^π, biasing both the critic and the policy improved from it. The following example illustrates this challenge. Example 2 (Example 1 continued). In the windy corridor, U(t)U^(t) is the wind, which the demonstrator observes but the learner does not. The expert policy fAf_A moves forward along the short route only when the wind is favorable, so recorded forward moves are almost always followed by reaching the goal, and a contrastive critic trained on these continuations values them highly. A learner that cannot observe the wind, however, would move forward regardless of it: conditioning on a forward action selects favorable wind, so the observational transition P(s′∣s,forward)P(s s,forward) never leads into the lava, whereas the interventional transition Pforward(s′∣s)P_forward(s s) also covers unfavorable wind. ∎ 2.1 Counterfactual Modeling Evaluating a policy requires the interventional transition Pa(s′∣s)P_a(s s), whereas the offline data provide only the observational transition P(s′∣s,a)P(s s,a). Under unobserved confounding, the former is generally not identifiable from the latter (Pearl, 2009; Bareinboim et al., 2022): different confounded MDPs can induce the same observational distribution but different interventional transitions. Consequently, the occupancy dπ(s∗∣s,a)d^π(s^* s,a) in Eq. (1) is generally not identifiable either (Kallus and Zhou, 2020; Zhang and Bareinboim, 2025). Rather than treating Pa(s′∣s)P_a(s s) as a single unknown, we use counterfactual reasoning to separate its identified and unidentified components. For an action a∈a , the counterfactual (potential outcome) variable SA(t)=a(t+1)=fS(S(t),a,U(t))S^(t+1)_A^(t)=a=f_S(S^(t),a,U^(t)) denotes the next state that would have been observed had the action at time t been set to a, with the state and the unobserved confounder held at their realized values (Pearl, 2009). Unlike interventional quantities, counterfactual variables can be evaluated jointly with the factual ones under the same realization of U(t)U^(t), relating the outcome of action a to the action a′a actually selected by the demonstrator. Conditioning on the factual state and action yields the counterfactual transition probability (s,a,a′,s′)=P(SA(t)=a(t+1)=s′∣S(t)=s,A(t)=a′).T(s,a,a ,s )=P(S^(t+1)_A^(t)=a=s S^(t)=s,A^(t)=a ). (3) It answers the counterfactual query: “What would the next state have been had the action at state s been set to a, given that the demonstrator in fact selected a′a ?” This construction follows the same counterfactual reasoning as the effect of treatment on the treated (ETT) (Shpitser and Pearl, 2009), which we extend here to state transitions in MDPs. Conditioning on A(t)=a′A^(t)=a retains information about the confounder under which the demonstrator acted. The counterfactual transition connects the observational and interventional transitions. When a=a′a=a , the consistency axiom (Galles and Pearl, 1998; Pearl, 2009) gives (s,a,a,s′)=P(s′∣s,a)T(s,a,a,s )=P(s s,a), which is identified from the offline data. Averaging over the demonstrator’s action instead recovers the interventional transition by the law of total probability, ∑a′∈(s,a,a′,s′)P(a′∣s)=Pa(s′∣s), _a T(s,a,a ,s )\,P(a s)=P_a(s s), (4) where the sum becomes an integral for continuous actions. A counterfactual transition is therefore compatible with the offline data as long as it coincides with the observational transition along the direction a=a′a=a ; the remaining entries, with a≠a′a≠ a , describe the outcomes of actions the demonstrator did not take and are free parameters. Thus, the compatible set is =:(s,a,a,⋅)=P(⋅∣s,a)T=\T:T(s,a,a,·)=P(· s,a) for all s,as,a\, where each (s,a,a′,⋅)T(s,a,a ,·) is a distribution over S. Building on the counterfactual transition, we define the counterfactual occupancy measure Qπ(s,a,a′) Q^π(s,a,a ) =(1−γ)∑k=0∞γkP(SA(t)=a,π(t+k)=s∗CLOSE =(1-γ) _k=0^∞γ^k\,P (S^(t+k)_A^(t)=a,π=s^* (5) ∣S(t)=s,A(t)=a′), S^(t)=s,A^(t)=a ), where SA(t)=a,π(t+k)S^(t+k)_A^(t)=a,π is the state that would have been reached at time t+kt+k had A(t)A^(t) been set to a and subsequent actions followed π, given that the demonstrator in fact selected a′a . Averaging over the demonstrator’s action recovers the Q-function, Qπ(s,a)=∑a′Qπ(s,a,a′)P(a′∣s)Q^π(s,a)= _a Q^π(s,a,a )\,P(a s), as in Eq. (4). Since the unobserved confounder is drawn independently at each step, the counterfactual occupancy satisfies the counterfactual Bellman equation Qπ(s,a,a′) Q^π(s,a,a ) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ (6) +γ∑s′(s,a,a′,s′)∑a~,a~′π(a~∣s′) +γ _s T(s,a,a ,s ) _ a, a π( a s ) ×P(a~′∣s′)Qπ(s′,a~,a~′). × P( a s )\,Q^π(s , a, a ). That is, the next state follows the counterfactual transition, after which the policy selects a~ a while the demonstrator’s action a~′ a is again drawn from P(⋅∣s′)P(· s ). The counterfactual Bellman equation depends on the unobserved confounder only through the counterfactual transition. Replacing the true counterfactual transition with a candidate T yields a candidate MDP ℳM_T, and we use the subscript T for quantities computed in ℳM_T. In particular, solving Eq. (6) with any compatible ∈T gives the counterfactual occupancy measure Qπ(s,a,a′)Q^π_T(s,a,a ), whose average Qπ(s,a)=∑a′Qπ(s,a,a′)P(a′∣s)Q^π_T(s,a)= _a Q^π_T(s,a,a )\,P(a s) is the Q-function of π in ℳM_T (Lemma A.1). By consistency, T contains the true counterfactual transition of ℳM. Since the offline data cannot distinguish it from the other compatible ones, minimizing over T gives a worst-case occupancy measure, which lower-bounds the true one. Proposition 1. For every counterfactual transition ∈T , the counterfactual Bellman equation (6) has a unique solution Qπ(s,a,a′)Q^π_T(s,a,a ), and the worst-case occupancy measure for a policy π(a∣s)π(a s), Q¯π(s,a)=inf∈∑a′∈Qπ(s,a,a′)P(a′∣s) Q^π(s,a)= _T _a Q^π_T(s,a,a )\,P(a s) satisfies Q¯π(s,a)≤Qπ(s,a) Q^π(s,a)≤ Q^π(s,a) for every (s,a)(s,a). The proof is given in Appendix A. Since T and P(a′∣s)P(a s) are determined by the observational data, Q¯π Q^π lower-bounds the probability of reaching the goal without access to the unobserved confounder. 3 Causal Contrastive Reinforcement Learning We now present Causal Contrastive Reinforcement Learning (CauCRL), which approximates the pessimistic construction of Sec. 2.1 with learned models. CauCRL retains the observed dynamics for the factual component and models the unidentified component pessimistically. It learns a pessimistic counterfactual transition model (Sec. 3.1), generates counterfactual rollouts under learner-selected actions whose states serve as positive samples for the contrastive critic, and improves the policy with this critic (Sec. 3.2). Algorithm 1 summarizes the procedure. 3.1 Learning Counterfactual Transitions Computing the worst-case occupancy measure in Proposition 1 requires characterizing the worst-case counterfactual transitions within T. In discrete domains, the entries with a≠a′a≠ a are unrestricted, and the worst case is given by the Manski bounds (Manski, 1990): the interventional transition follows the observational one with probability P(a∣s)P(a s) and is otherwise unconstrained. Such bounds have been applied to off-policy evaluation (Zhang and Bareinboim, 2025) and deep Q-learning (Li et al., 2025). In continuous domains, however, the demonstrator selects any particular action with probability zero, so the constraint (s,a,a,⋅)=P(⋅∣s,a)T(s,a,a,·)=P(· s,a) alone does not restrict nearby actions, and consistent bound estimation requires additional smoothness constraints (Tan et al., 2024). We therefore assume that the counterfactual transition is Lipschitz continuous in all of its conditioning arguments: for some constant L<∞L<∞, W((s1,a1,a′1,⋅),(s2,a2,a′2,⋅)) W (T(s_1,a_1,a _1,·),\,T(s_2,a_2,a _2,·) ) ≤L(d(s1,s2)+d(a1,a2)+d(a1′,a2′)) ≤ L (d_S(s_1,s_2)+d_A(a_1,a_2)+d_A(a _1,a _2) ) for all s1,s2∈s_1,s_2 and a1,a2,a1′,a2′∈a_1,a_2,a _1,a _2 , where W denotes the Wasserstein-1 distance between distributions over S. Along the action arguments, this assumption mirrors the smoothness conditions of Kim et al. (2018). Continuity in the evaluated action a corresponds to smooth treatment response (STR): the next-state distribution changes gradually with the intervened action. Continuity in the demonstrator’s action a′a corresponds to smooth treatment selection (STS): situations in which the demonstrator selects nearby actions lead to similar counterfactual outcomes. Continuity in the current state s encodes the locality of the transition dynamics typical of RL. For example, in grid-world navigation, agents moving in neighboring cells are likely to lead to a collision. As illustrated in Fig. 3, consistency identifies T along the diagonal a=a′a=a . An unidentified entry (s,a,a′)(s,a,a ) with a≠a′a≠ a reaches the diagonal by moving a and a′a to a common value x between them, a path of total length d(a,a′)d_A(a,a ). Lipschitz continuity limits how far probability mass can be transported along this path, W((s,a,a′,⋅),P(⋅∣s,x))≤Ld(a,a′).W (T(s,a,a ,·),\,P(· s,x) )≤ L\,d_A(a,a ). The counterfactual transition thus lies in a Wasserstein ball around the identified transitions: the outcomes of actions close to the demonstrator’s remain close to the observed ones, while the ball widens as the evaluated action moves away. The worst-case occupancy measure is obtained by restricting the minimization in Proposition 1 to Lipschitz-continuous elements of T. a′a s′s a=a′a=a Ld(a,a′)L\,d_A(a,a )P(⋅∣s,x)P(· s,x)(s,a,a′,⋅)T(s,a,a ,·) Figure 3: Lipschitz continuity of the counterfactual transition. Along the diagonal a=a′a=a (thick line), (s,a,a,⋅)=P(⋅∣s,a)T(s,a,a,·)=P(· s,a) is identified (filled densities over s′s ). An off-diagonal entry (s,a,a′,⋅)T(s,a,a ,·) is not identified (dashed densities). The shortest transport to the diagonal is L-shaped (arrow): it moves the evaluated action from a and the demonstrator’s action from a′a to a common value x, with total length d(a,a′)d_A(a,a ). We model a pessimistic counterfactual transition with a conditional generative model ηT_η: a neural network gηg_η maps the state s, the evaluated action a, the demonstrator’s action a′a , and noise ε∼(0,I) (0,I) to a next state s~′=gη(s,a,a′,ε) s =g_η(s,a,a , ). Spectral normalization of its weights (Miyato et al., 2018) bounds the Lipschitz constant of gηg_η in (s,a,a′)(s,a,a ) for every ε ; with shared noise, this bound carries over to the Wasserstein distance between next-state distributions. The model is trained with two objectives. The observational objective fits the entries with a=a′a=a to the positive demonstrations +D^+ drawn from P+P^+, ℒ+(η)=(s,a,s′)∼+[ℓ(η(s,a,a,⋅),s′)],L^+(η)=E_(s,a,s ) ^+ [ \! (T_η(s,a,a,·),s ) ], where ℓ is a proper scoring rule, such as the negative log-likelihood or the energy score. Minimizing ℒ+L^+ enforces the consistency constraint that defines T; the remaining entries are chosen pessimistically. Since the negative demonstrations record only states, the interventional objective pairs each state–action pair (s,a′)(s,a ) from +D^+ with an alternative action a≠a′a≠ a drawn from a proposal ρ(⋅∣s)ρ(· s), such as a uniform distribution, and with an independent negative state s−s^- from −D^-: ℒ−(η)=(s,a′)∼+,a∼ρ,s−∼−[ℓ(η(s,a,a′,⋅),s−)].L^-(η)=E_(s,a ) ^+,\,a ρ,\,s^- ^- [ \! (T_η(s,a,a ,·),s^- ) ]. Minimizing ℒ−L^- steers the entries with a≠a′a≠ a toward states reached by non-expert behavior; without further constraints, its minimizer is (s,a,a′,⋅)=P−T(s,a,a ,·)=P^-. When P−P^- concentrates on absorbing failure states such as the lava, this coincides with the worst case of the Manski bounds, which places all unidentified mass on states from which the goal cannot be reached, and thus approximates the minimizer in Proposition 1. The combined objective is given by ℒ(η)=ℒ+(η)+ℒ−(η)L_T(η)=L^+(η)+L^-(η) (7) is minimized over spectrally normalized networks: ℒ+L^+ anchors the model to the identified transitions, ℒ−L^- selects pessimistic outcomes among the unidentified ones, and the Lipschitz constraint keeps ηT_η within the Wasserstein ball of radius Ld(a,a′)L\,d_A(a,a ). This prevents arbitrarily adverse predictions and lets the model generalize to action pairs not covered by the data. Implementation details are given in Appendix B. 3.2 Counterfactual Rollouts Evaluating a policy π requires next states drawn from the interventional transition Pa(s′∣s)P_a(s s), which by Eq. (4) averages the counterfactual transition over the demonstrator’s action. Since P(A∣S)P(A S) is identified from the demonstrations, we estimate it from +D^+ with a statistical model μϕ(a′∣s) _φ(a s), the nominal behavioral policy. Any conditional density estimator can be used; our choices are detailed in Appendix B. For a counterfactual transition T, averaging over the demonstrator’s action defines the induced transition Pa(s′∣s)=∑a′∈(s,a,a′,s′)P(a′∣s),P^T_a(s s)= _a T(s,a,a ,s )\,P(a s), (8) which coincides with Pa(s′∣s)P_a(s s) for the true counterfactual transition. As previously defined, we write ℳM_T for the MDP with transition PaP^T_a (Sec. 2.1). Algorithm 1 Causal Contrastive RL (CauCRL) 1: +D^+, −D^-, rounds R, horizon H 2: Fit nominal policy μϕ _φ on +D^+ 3: Fit ηT_η by minimizing Eq. (7) 4: Initialize critic fωf_ω and actor πθ _θ 5: for r=1,…,Rr=1,…,R do 6: ~←CtfRollout(πθ,η,μϕ) D← CtfRollout( _θ,T_η, _φ) 7: πθ←PolicyImprovement(~,πθ) _θ← PolicyImprovement( D, _θ) 8: end for 9: return πθ _θ 10: procedure CtfRollout(π,,μπ,T,μ) 11: Sample (s,a)∼+(s,a) ^+; set s~(t)=s s^(t)=s, a~(t)=a a^(t)=a 12: for h=t,…,t+H−1h=t,…,t+H-1 do 13: Sample a′(h)∼μ(⋅∣s~(h))a (h) μ(· s^(h)) 14: Sample s~(h+1)∼(s~(h),a~(h),a′(h),⋅) s^(h+1) ( s^(h), a^(h),a (h),·) 15: Sample a~(h+1)∼π(⋅∣s~(h+1)) a^(h+1) π(· s^(h+1)) 16: end for 17: return rollouts ~ D 18: end procedure 19: procedure PolicyImprovement(~,π D,π) 20: for each gradient step do 21: Sample (s,a,s~+)(s,a, s^+) from ~ D and s−∼−s^- ^- 22: Update fωf_ω by Eq. (9) and π by Eq. (10) 23: end for 24: return π 25: end procedure A counterfactual rollout samples from PaηP^T_η_a under the policy π, with μϕ _φ in place of P(a′∣s)P(a s) (CtfRollout in Algorithm 1). It starts from a state–action pair (s,a)(s,a) at time t, with s~(t)=s s^(t)=s and a~(t)=a a^(t)=a; at each step h≥th≥ t, the demonstrator’s action a′(h)a (h) is drawn from the nominal policy, the next state s~(h+1) s^(h+1) from ηT_η, and the learner’s next action from π; a′a is resampled even at the first step, since the recorded action would give the confounded (s,a,a,⋅)=P(⋅∣s,a)T(s,a,a,·)=P(· s,a). Algorithm 1 uses the current actor as the rollout policy, which may also be kept fixed across rounds. Replacing the positive samples from P+P^+ with these rollouts yields the counterfactual NCE objective maxω(s,a,s~+)∼CtfRollouts−∼P− _ω\;E_ subarrayc(s,a, s^+) CtfRollout\\ s^- P^- subarray [logσ(fω(s,a,s~+)) [ σ\! (f_ω(s,a, s^+) ) (9) +log(1−σ(fω(s,a,s−)))], + \! (1-σ\! (f_ω(s,a,s^-) ) ) ], where each pair (s,a)(s,a) from +D^+ takes the rollout state s~+=S~(t+K) s^+= S^(t+K) as its positive goal, with P(K=k)=(1−γ)γkP(K=k)=(1-γ)γ^k for k≥0k≥ 0, and the negative goals are unchanged. The following proposition shows that the optimal critic recovers the Q-function of π in the MDP induced by the counterfactual transition.22 2 The constant 1/P−(s∗)1/P^-(s^*) requires P−(s∗)>0P^-(s^*)>0. This can always be ensured by mixing random states from the whole dataset into the negative samples, as in the original CRL (Eysenbach et al., 2022). These additional states serve only as negatives for the critic in Eq. (9); they are not used to train the counterfactual transition in Eq. (7). Proposition 2. Suppose that CtfRollout in Eq. (9) uses a counterfactual transition T and the nominal policy μ=P(⋅∣s)μ=P(· s). The critic that optimizes Eq. (9) is the Q-function of π in ℳM_T, up to a multiplicative constant 1/P−(s∗)1/P^-(s^*): expf∗(s,a,s∗)=1P−(s∗)⋅Qπ(s,a). f^*(s,a,s^*)= 1P^-(s^*)· Q^π_T(s,a). The proof is given in Appendix A. Since P−(s∗)P^-(s^*) does not depend on the action, argmaxaf∗(s,a,s∗)=argmaxaQπ(s,a) _af^*(s,a,s^*)= _aQ^π_T(s,a): the critic retains the value interpretation of contrastive RL, now in the MDP induced by the counterfactual transition. In practice, rollouts are truncated at a horizon H; sampling K only up to H changes the distribution of positive goals by at most γH+1γ^H+1 in total variation. Details of the rollout construction are given in Appendix B. Policy improvement. CauCRL retains the actor update used in standard offline CRL. The actor selects actions with high critic values while remaining close to the actions in the offline demonstrations: maxθ(s,a′)∼+a∼πθ(⋅∣s)[fω(s,a,s∗)+λBClogπθ(a′∣s)].\! _θ\;E_ subarrayc(s,a ) ^+\\ a _θ(· s) subarray [f_ω(s,a,s^*)+ _BC _θ(a s) ]. (10) The critic evaluates the policy’s own action a, while the behavioral-cloning term, weighted by λBC _BC, keeps πθ _θ close to the recorded action a′a . By Proposition 2, higher critic values mean higher Q-values in ℳM_T, so this update improves the policy in that MDP. Proposition 3. Assume that states and actions are tabular, μ=P(⋅∣s)μ=P(· s), and P−(s∗)>0P^-(s^*)>0. Let the critic f be δ-accurate for rollouts of π under T, i.e., |P−(s∗)expf(s,a,s∗)−Qπ(s,a)|≤δ|P^-(s^*) f(s,a,s^*)-Q^π_T(s,a)|≤δ for all (s,a)(s,a), and let π′(s)∈argmaxaf(s,a,s∗)π (s)∈ _af(s,a,s^*) be the policy obtained after one iteration of PolicyImprovement with λBC=0 _BC=0. Then π′π achieves a higher goal-reaching occupancy than π in ℳM_T, up to the critic error: Qπ′(s,a)≥Qπ(s,a)−2γδ1−γfor all (s,a).Q^π _T(s,a)≥ Q^π_T(s,a)- 2γδ1-γ all (s,a). The proof (Appendix A) mirrors the approximate policy improvement result of Eysenbach et al. (2022, Lemma 4.2), with ℳM_T in place of the true environment. For the Bayes-optimal critic of Proposition 2, δ=0δ=0 and π′π improves upon π exactly, so alternating CtfRollout and PolicyImprovement in Algorithm 1 performs approximate policy iteration in ℳηM_T_η. Proposition 3 guarantees improvement in ℳM_T. When T is the worst-case counterfactual transition of Proposition 1, QπQ^π_T lower-bounds the true occupancy. With the pessimistic ηT_η learned in Sec. 3.1, CauCRL improves an approximation of this lower bound as a proxy for policy improvement in the real environment. 4 Experiments Figure 4: Continuous-control environments: (a) PointMaze with a stochastic swamp region (yellow) and (b) AntMaze with falling-rock hazards (red). We evaluate CauCRL on three goal-conditioned navigation tasks with unobserved confounding and increasing control complexity: Windy Corridor with discrete actions, PointMaze with continuous planar control, and AntMaze with eight-dimensional continuous actions (Fig. 4). In each environment, the demonstrator observes latent information that affects both its actions and subsequent transitions but is hidden from the learner and all evaluated methods. We compare CauCRL against four goal-conditioned offline learning baselines. CRL is standard contrastive reinforcement learning (Eysenbach et al., 2022) and our primary baseline; it uses the same demonstrations and actor regularization as CauCRL but trains its critic on recorded rather than counterfactual future goals, so comparing the two isolates the effect of counterfactual rollouts. GCBC performs behavioral cloning on hindsight-relabeled trajectories (Lynch et al., 2020; Ghosh et al., 2021). GCIQL is a goal-conditioned variant of implicit Q-learning (Kostrikov et al., 2022), and HIQL uses latent future states as high-level actions for long-horizon goal-conditioned control (Park et al., 2023). All baselines treat recorded next states as outcomes of the recorded actions and do not model the confounder. For continuous-control tasks, CRL and CauCRL share the actor–critic architecture and optimizer; full setup and training details are in Appendices C.2–C.4. We report mean goal-reaching success and, where applicable, robust-route rates over five seeds, with ± s.e.m. over the final 10% of evaluation checkpoints; per-seed results are in Appendix C.5. Figure 5: Success rates over training on Windy Corridor (left), PointMaze (middle), and AntMaze (right). Curves show means over five random seeds, with shaded regions indicating standard errors. Experiment 1: Windy Corridor. We revisit the 15×1515× 15 Windy Corridor task of Example 1 (Appendix C.1), with a short passage adjacent to lava and a longer safe route. The demonstrator primarily takes the shortcut and uses the observed wind to pause when forward motion is unsafe, while a smaller fraction of demonstrations follows the safe route. Because the wind is resampled at every step and hidden from the learner, only the safe route reaches the goal reliably. In Fig. 5 (left), CauCRL achieves a mean success rate of 0.919±0.0360.919± 0.036 and a robust-route rate of 1.001.00, despite the safe route requiring about 39 steps instead of 13. Although its success varies across seeds (0.7970.797 to 1.0001.000), every evaluation episode takes the safe route. All baselines remain below 0.600.60 success; CRL, GCIQL, and HIQL reach 0.5130.513 on every seed and converge to the confounded shortcut. Because demonstrations pair shortcut actions almost exclusively with successful outcomes, confounding-unaware critics overvalue this route. CauCRL assigns pessimistic outcomes to moving forward where the demonstrator might have waited, so its critic prefers the safe route. Experiment 2: PointMaze. The agent observes a four-frame history of planar positions and the goal (16 dimensions) and selects two-dimensional velocities (Appendix C.2). The shortcut crosses three swamp cells, each active with probability 0.300.30 at every step; entering an active cell freezes the agent for the remainder of the 50-step episode. The demonstrator observes the swamp condition and primarily follows the shortcut, waiting when necessary, while a smaller set of demonstrations takes the detour. In Fig. 5 (middle), CauCRL achieves both a mean success rate and detour rate of 1.001.00. CRL, GCBC, and GCIQL each achieve 0.3470.347 success and never take the detour, close to the probability (1−0.30)3≈0.343(1-0.30)^3≈ 0.343 that all three cells are inactive when crossed. HIQL performs worse (0.176±0.0130.176± 0.013). Since the swamp is hidden and resampled at every step, waiting cannot improve these odds for the learner; CauCRL instead identifies the detour as the reliable route. Experiment 3: AntMaze. The Ant has a 31-dimensional observation, an eight-dimensional action space, and an 800-step horizon (Appendix C.3). The direct route crosses two rockfall zones, each activated with probability 0.400.40 per episode, while a longer route avoids them; contact with a falling rock terminates the episode. The demonstrator observes the rockfall state, but the learner observes only the Ant state and goal. Because the demonstrator crosses only when the route is safe, its trajectories contain no rock-contact failures and make direct-route actions appear safe. CauCRL additionally trains its counterfactual transition on forced crossings that override a demonstrator decision to wait and expose the resulting rock collisions. In Fig. 5 (right), CauCRL achieves 0.815±0.0300.815± 0.030 success, compared with 0.629±0.0150.629± 0.015 for CRL, an absolute improvement of 0.1860.186. GCIQL reaches 0.616±0.0760.616± 0.076, while GCBC and HIQL achieve 0.498±0.0220.498± 0.022 and 0.409±0.0060.409± 0.006, respectively. CauCRL’s lowest seed (0.7120.712) exceeds CRL’s highest (0.6770.677). 5 Conclusion We study offline goal-conditioned reinforcement learning under unobserved confounding, where demonstrators may use information unavailable to the learner. Standard contrastive RL can then learn spurious action–outcome associations from observational future states. To address this problem, we introduce Causal Contrastive Reinforcement Learning (CauCRL), which learns a pessimistic counterfactual transition from positive and negative demonstrations and uses counterfactual rollouts for contrastive critic learning. Our causal formulation separates identified factual transitions from unidentified counterfactual outcomes and yields a worst-case goal-reaching occupancy that lower-bounds the true occupancy. Experiments across discrete and continuous-control tasks show that CauCRL consistently improves robustness over confounding-unaware baselines. These results highlight the value of counterfactual future-state supervision under confounding. Future work will evaluate CauCRL beyond maze-style navigation, including goal-conditioned manipulation and more diverse sources of unobserved confounding. AI Use Statement In this work, we used generative AI tools to assist with research ideation, methodological discussions, experimental design and diagnostics, code development and debugging, and checking mathematical arguments. We also used these tools for retrieving and summarizing related work and for drafting and revising portions of the manuscript. All AI-assisted outputs were reviewed and verified by the authors. Mathematical arguments were checked against the stated assumptions and derivations, code was inspected and validated through experiments, and literature summaries were verified against the original sources. The authors made all final decisions regarding the methodology, experiments, interpretations, and presentation, and take full responsibility for the content of this work. References Andrychowicz et al. (2017) M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba Hindsight experience replay. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §1. Balazadeh et al. (2024) V. Balazadeh, K. Chidambaram, V. Nguyen, R. G. Krishnan, and V. Syrgkanis Sequential decision making with expert demonstrations under unobserved heterogeneity. In Advances in Neural Information Processing Systems, Vol. 37, p. 65476–65498. Cited by: §1. Bareinboim et al. (2022) E. Bareinboim, J. D. Correa, D. Ibeling, and T. Icard On Pearl’s hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, H. Geffner, R. Dechter, and J. Y. Halpern (Eds.), p. 507–556. External Links: Document Cited by: §1, §2.1. Bennett et al. (2021) A. Bennett, N. Kallus, L. Li, and A. Mousavi Off-policy evaluation in infinite-horizon reinforcement learning with latent confounders. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, A. Banerjee and K. Fukumizu (Eds.), Proceedings of Machine Learning Research, Vol. 130, p. 1999–2007. External Links: Link Cited by: §1. Bo et al. (2026) A. Bo, J. Zhang, and M. C. Gursoy Counterfactual bootstrap for robust meta-reinforcement learning. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 8707–8731. Cited by: §1, §2, Example 1. Bortkiewicz et al. (2025) M. Bortkiewicz, W. Pałucki, V. Myers, T. Dziarmaga, T. Arczewski, Ł. Kuciński, and B. Eysenbach Accelerating goal-conditioned reinforcement learning algorithms and research. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 15732–15754. External Links: Link Cited by: Appendix D. Chevalier-Boisvert et al. (2023) M. Chevalier-Boisvert, B. Dai, M. Towers, R. de Lazcano, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry Minigrid & Miniworld: modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §C.1. de Haan et al. (2019) P. de Haan, D. Jayaraman, and S. Levine Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §1. Duan et al. (2016) Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel RL2^2: fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779. Cited by: §1. Eysenbach et al. (2021) B. Eysenbach, R. Salakhutdinov, and S. Levine C-Learning: learning to achieve goals via recursive classification. In International Conference on Learning Representations, External Links: Link Cited by: §1. Eysenbach et al. (2022) B. Eysenbach, T. Zhang, S. Levine, and R. R. Salakhutdinov Contrastive learning as goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 35603–35620. External Links: Document, Link Cited by: Appendix A, Appendix D, §1, §2, §2, §2, §2, §3.2, §4, Example 1, footnote 1, footnote 2. Galles and Pearl (1998) D. Galles and J. Pearl An axiomatic characterization of causal counterfactuals. Foundations of Science 3 (1), p. 151–182. Cited by: §2.1. Ghosh et al. (2021) D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, Cited by: §1, §4. Ho and Ermon (2016) J. Ho and S. Ermon Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: §1. Joshi et al. (2024) S. Joshi, J. Zhang, and E. Bareinboim Towards safe policy learning under partial identifiability: a causal approach. In AAAI Conference on Artificial Intelligence, p. 13004–13012. External Links: Document, Link Cited by: §1. Kaelbling (1993) L. P. Kaelbling Learning to achieve goals. In International Joint Conference on Artificial Intelligence, p. 1094–1099. External Links: Link Cited by: §1. Kallus and Zhou (2018) N. Kallus and A. Zhou Confounding-robust policy improvement. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §1. Kallus and Zhou (2020) N. Kallus and A. Zhou Confounding-robust policy evaluation in infinite-horizon reinforcement learning. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 22293–22304. External Links: Link Cited by: §1, §2.1. Kim et al. (2018) W. Kim, K. Kwon, S. Kwon, and S. Lee The identification power of smoothness assumptions in models with counterfactual outcomes. Quantitative Economics 9 (2), p. 617–642. External Links: Document Cited by: §3.1. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, Cited by: §4. Li et al. (2025) M. Li, J. Zhang, and E. Bareinboim Confounding robust deep reinforcement learning: a causal approach. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 138292–138325. External Links: Document, Link Cited by: §1, §3.1. Li et al. (2026) M. Li, J. Zhang, and E. Bareinboim Causal flow Q-learning for robust offline reinforcement learning. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1. Liu et al. (2022) M. Liu, M. Zhu, and W. Zhang Goal-conditioned reinforcement learning: problems and solutions. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, p. 5502–5511. Cited by: §1. Lynch et al. (2020) C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet Learning latent plans from play. In Proceedings of the Conference on Robot Learning, Proceedings of Machine Learning Research, Vol. 100, p. 1113–1132. Cited by: §4. Manski (1990) C. F. Manski Nonparametric bounds on treatment effects. American Economic Review 80 (2), p. 319–323. Cited by: §3.1. Miyato et al. (2018) T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, Cited by: §3.1. Myers et al. (2025) V. Myers, B. Zheng, B. Eysenbach, and S. Levine Offline goal-conditioned reinforcement learning with quasimetric representations. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 19654–19679. External Links: Document, Link Cited by: Appendix D. Namkoong et al. (2020) H. Namkoong, R. Keramati, S. Yadlowsky, and E. Brunskill Off-policy policy evaluation for sequential decisions under unobserved confounding. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 18819–18831. External Links: Link Cited by: §1. Ng et al. (1999) A. Y. Ng, D. Harada, and S. Russell Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, p. 278–287. Cited by: §1. Park et al. (2025) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned RL. In International Conference on Learning Representations, Cited by: §C.4. Park et al. (2023) S. Park, D. Ghosh, B. Eysenbach, and S. Levine HIQL: offline goal-conditioned RL with latent states as actions. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 34866–34891. External Links: Document, Link Cited by: §4. Pearl (2009) J. Pearl Causality: models, reasoning and inference. 2 edition, Cambridge University Press. External Links: Document Cited by: §1, §2.1, §2.1, §2. Pomerleau (1989) D. A. Pomerleau ALVINN: an autonomous land vehicle in a neural network. In Advances in Neural Information Processing Systems, Vol. 1. Cited by: §1. Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, p. 627–635. Cited by: §1. Ruan et al. (2024) K. Ruan, J. Zhang, X. Di, and E. Bareinboim Causal imitation for Markov decision processes: a partial identification approach. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 87592–87620. External Links: Document, Link Cited by: §1. Schaul et al. (2015) T. Schaul, D. Horgan, K. Gregor, and D. Silver Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei (Eds.), Proceedings of Machine Learning Research, Vol. 37, Lille, France, p. 1312–1320. External Links: Link Cited by: §1, §1. Shpitser and Pearl (2009) I. Shpitser and J. Pearl Effects of treatment on the treated: identification and generalization. In Conference on Uncertainty in Artificial Intelligence, p. 514–521. External Links: Link Cited by: §2.1. Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2 edition, MIT Press. Cited by: §1. Tan et al. (2024) J. Tan, J. Blanchet, and V. Syrgkanis Consistency of neural causal partial identification. In Advances in Neural Information Processing Systems, Vol. 37, p. 68956–68999. External Links: Link Cited by: §3.1. Zhang and Bareinboim (2016) J. Zhang and E. Bareinboim Markov decision processes with unobserved confounders: a causal approach. Technical report Technical Report R-23, Purdue AI Lab. Cited by: §2. Zhang and Bareinboim (2019) J. Zhang and E. Bareinboim Near-optimal reinforcement learning in dynamic treatment regimes. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . External Links: Link Cited by: §1. Zhang and Bareinboim (2025) J. Zhang and E. Bareinboim Causal eligibility traces for confounding robust off-policy evaluation. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, S. Chiappa and S. Magliacane (Eds.), Proceedings of Machine Learning Research, Vol. 286, p. 4933–4942. External Links: Link Cited by: §1, §2.1, §3.1. Zhang et al. (2020) J. Zhang, D. Kumor, and E. Bareinboim Causal imitation learning with unobserved confounders. In Advances in Neural Information Processing Systems, Vol. 33, p. 12263–12274. Cited by: §1. Zheng et al. (2024a) C. Zheng, B. Eysenbach, H. Walke, P. Yin, K. Fang, R. Salakhutdinov, and S. Levine Stabilizing contrastive RL: techniques for robotic goal reaching from offline data. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 28236–28264. External Links: Link Cited by: Appendix D. Zheng et al. (2024b) C. Zheng, R. Salakhutdinov, and B. Eysenbach Contrastive difference predictive coding. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 47577–47601. External Links: Link Cited by: Appendix D. Appendix A Proofs This appendix collects the proofs of the three propositions in the main text. Proposition 1 shows that minimizing the counterfactual occupancy over all compatible counterfactual transitions lower-bounds the true goal-reaching occupancy, which underlies the pessimistic construction of Sec. 2.1; Proposition 2 identifies the Bayes-optimal critic trained on counterfactual rollouts, which underlies the counterfactual NCE objective of Sec. 3.2; and Proposition 3 shows that greedy updates with an approximately optimal critic improve the policy in the MDP induced by the counterfactual transition. All three proofs follow the standard dynamic-programming playbook: each quantity of interest is characterized as the unique fixed point of a γ-contractive Bellman operator, and identities and inequalities between these quantities are obtained by verifying them for the corresponding operators. Throughout, we fix a goal s∗∈s^* and a discount factor γ∈(0,1)γ∈(0,1), and all functions are bounded and equipped with the supremum norm ∥⋅∥∞\|·\|_∞. Sums over states and actions become integrals for continuous spaces, and all arguments carry over verbatim. Entries (s,a,a′)(s,a,a ) with P(a′∣s)=0P(a s)=0 receive zero weight in Eq. (6) and in the definition of Q¯π Q^π, so they play no role and can be fixed arbitrarily. For a policy π and a counterfactual transition ∈T , we define the counterfactual Bellman operator, acting on functions q(s,a,a′)q(s,a,a ), as the right-hand side of Eq. (6), (πq)(s,a,a′) (K^π_Tq)(s,a,a ) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′(s,a,a′,s′)∑a~,a~′π(a~∣s′) +γ _s T(s,a,a ,s ) _ a, a π( a s ) ×P(a~′∣s′)q(s′,a~,a~′), × P( a s )\,q(s , a, a ), and the Bellman operator of π in ℳM_T, acting on functions q(s,a)q(s,a), as (ℬπq)(s,a) (B^π_Tq)(s,a) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′Pa(s′∣s)∑a~π(a~∣s′)q(s′,a~), +γ _s P^T_a(s s) _ aπ( a s )\,q(s , a), where PaP^T_a is the induced transition in Eq. (8). Finally, dπ(s∗∣s,a)d^π_T(s^* s,a) denotes the goal-reaching occupancy of π in ℳM_T, i.e., Eq. (1) with PaP_a replaced by PaP^T_a, and Pa,πP^T_a,π denotes the trajectory distribution in ℳM_T that sets A(t)A^(t) to a and lets subsequent actions follow π. The operator facts shared by the three proofs are encapsulated in the following lemma. Lemma A.1. Let π be a policy and ∈T , and let B denote either πK^π_T or ℬπB^π_T. (a) B is monotone, satisfies B(q−c)=Bq−γcB(q-c)=Bq-γ c for every constant c, and is a γ-contraction in ∥⋅∥∞\|·\|_∞. Consequently, B has a unique bounded fixed point, which is the limit of BnqB^nq as n→∞n→∞ for every bounded q. (b) The fixed point of ℬπB^π_T is dπ(s∗∣s,a)d^π_T(s^* s,a). (c) If Qπ(s,a,a′)Q^π_T(s,a,a ) is the fixed point of πK^π_T, then Qπ(s,a)=∑a′Qπ(s,a,a′)P(a′∣s)Q^π_T(s,a)= _a Q^π_T(s,a,a )\,P(a s) is the fixed point of ℬπB^π_T. In particular, Qπ(s,a)=dπ(s∗∣s,a)Q^π_T(s,a)=d^π_T(s^* s,a), i.e., QπQ^π_T is the Q-function of π in ℳM_T. Proof. We first prove (a). Both operators are of the form (Bq)(x)=(1−γ)s=s∗+γ∑yK(x,y)q(y),(Bq)(x)=(1-γ) 1\s=s^*\+γ _yK(x,y)\,q(y), where s is the state component of x. For πK^π_T, we have x=(s,a,a′)x=(s,a,a ), y=(s′,a~,a~′)y=(s , a, a ), and K(x,y)=(s,a,a′,s′)π(a~∣s′)P(a~′∣s′),K(x,y)=T(s,a,a ,s )\,π( a s )\,P( a s ), while for ℬπB^π_T, we have x=(s,a)x=(s,a), y=(s′,a~)y=(s , a), and K(x,y)=Pa(s′∣s)π(a~∣s′).K(x,y)=P^T_a(s s)\,π( a s ). In both cases, K(x,⋅)K(x,·) is a probability distribution, since (s,a,a′,⋅)T(s,a,a ,·), π(⋅∣s′)π(· s ), P(⋅∣s′)P(· s ), and Pa(⋅∣s)P^T_a(· s) are. Since K≥0K≥ 0, B is monotone, and since ∑yK(x,y)=1 _yK(x,y)=1, B(q−c)=Bq−γcB(q-c)=Bq-γ c. Moreover, |(Bq1)(x)−(Bq2)(x)| |(Bq_1)(x)-(Bq_2)(x)| ≤γ∑yK(x,y)|q1(y)−q2(y)| ≤γ _yK(x,y)\,|q_1(y)-q_2(y)| ≤γ‖q1−q2‖∞, ≤γ\|q_1-q_2\|_∞, so B is a γ-contraction. The Banach fixed-point theorem gives the remaining claims of (a). We next prove (b). The k=0k=0 term of Eq. (1) in ℳM_T equals (1−γ)s=s∗(1-γ) 1\s=s^*\, since S(t)=sS^(t)=s. For k≥1k≥ 1, conditioning on the next state S(t+1)=s′S^(t+1)=s and the next action A(t+1)=a~A^(t+1)= a, the Markov property of ℳM_T gives Pa,π(S(t+k)=s∗∣S(t)=s) P^T_a,π (S^(t+k)=s^* S^(t)=s ) =∑s′Pa(s′∣s)∑a~π(a~∣s′) = _s P^T_a(s s) _ aπ( a s ) ×Pa~,π(S(t+k)=s∗∣S(t+1)=s′). × P^T_ a,π (S^(t+k)=s^* S^(t+1)=s ). Multiplying by (1−γ)γk(1-γ)γ^k, summing over k≥1k≥ 1, and using time-homogeneity to shift the time index by one, we obtain dπ(s∗∣s,a) d^π_T(s^* s,a) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′Pa(s′∣s)∑a~π(a~∣s′) +γ _s P^T_a(s s) _ aπ( a s ) ×dπ(s∗∣s′,a~). × d^π_T(s^* s , a). Hence dπd^π_T is a fixed point of ℬπB^π_T, and by (a) it is the unique one. It remains to prove (c). Multiplying the fixed-point equation Qπ=πQπQ^π_T=K^π_TQ^π_T by P(a′∣s)P(a s) and summing over a′a , the first term is unchanged, since ∑a′P(a′∣s)=1 _a P(a s)=1. In the second term, Eq. (8) gives ∑a′(s,a,a′,s′)P(a′∣s)=Pa(s′∣s), _a T(s,a,a ,s )\,P(a s)=P^T_a(s s), while the inner sum over a~′ a gives ∑a~′P(a~′∣s′)Qπ(s′,a~,a~′)=Qπ(s′,a~). _ a P( a s )\,Q^π_T(s , a, a )=Q^π_T(s , a). Therefore Qπ(s,a) Q^π_T(s,a) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′Pa(s′∣s)∑a~π(a~∣s′)Qπ(s′,a~), +γ _s P^T_a(s s) _ aπ( a s )\,Q^π_T(s , a), that is, Qπ=ℬπQπQ^π_T=B^π_TQ^π_T. By (a) and (b), Qπ(s,a)=dπ(s∗∣s,a)Q^π_T(s,a)=d^π_T(s^* s,a). This completes the proof. ∎ Proof of Proposition 1. The proof proceeds in three steps. We first note that the counterfactual Bellman equation is well posed for every compatible counterfactual transition. We then show that the true counterfactual transition of ℳM is compatible, and that the corresponding solution is the true counterfactual occupancy (5). Averaging over the demonstrator’s action then recovers the true occupancy, and minimizing over T can only decrease it. For every ∈T , Eq. (6) is the fixed-point equation q=πq=K^π_Tq. By Lemma A.1(a), it has a unique bounded solution Qπ(s,a,a′)Q^π_T(s,a,a ). We next show that the true counterfactual transition of ℳM is compatible. Let ℳ(s,a,a′,s′)=P(SA(t)=a(t+1)=s′∣S(t)=s,A(t)=a′)T_M(s,a,a ,s )=P (S^(t+1)_A^(t)=a=s S^(t)=s,A^(t)=a ) denote the counterfactual transition (3) of ℳM. On the event A(t)=a\A^(t)=a\, the consistency axiom gives SA(t)=a(t+1)=S(t+1)S^(t+1)_A^(t)=a=S^(t+1). Hence ℳ(s,a,a,s′) _M(s,a,a,s ) =P(S(t+1)=s′∣S(t)=s,A(t)=a) =P (S^(t+1)=s S^(t)=s,A^(t)=a ) =P(s′∣s,a), =P(s s,a), so ℳ∈T_M . We now show that the true counterfactual occupancy (5) solves Eq. (6) with ℳT_M. Fix (s,a,a′)(s,a,a ) with P(a′∣s)>0P(a s)>0, and let E=S(t)=s,A(t)=a′E=\S^(t)=s,A^(t)=a \. The k=0k=0 term of Eq. (5) equals (1−γ)s=s∗(1-γ) 1\s=s^*\, since SA(t)=a,π(t)=S(t)=sS^(t)_A^(t)=a,π=S^(t)=s. For k≥1k≥ 1, conditioning on the counterfactual next state gives P(SA(t)=a,π(t+k)=s∗∣E) P (S^(t+k)_A^(t)=a,π=s^* E ) =∑s′ℳ(s,a,a′,s′)P(SA(t)=a,π(t+k)=s∗∣Es′), = _s T_M(s,a,a ,s )\,P (S^(t+k)_A^(t)=a,π=s^* E_s ), where Es′=E∩SA(t)=a(t+1)=s′E_s =E∩\S^(t+1)_A^(t)=a=s \. Since A(t)=fA(S(t),U(t))A^(t)=f_A(S^(t),U^(t)) and SA(t)=a(t+1)=fS(S(t),a,U(t))S^(t+1)_A^(t)=a=f_S(S^(t),a,U^(t)), the event Es′E_s is determined by (S(t),U(t))(S^(t),U^(t)). From time t+1t+1 onward, the counterfactual trajectory starts at s′s , draws a~∼π(⋅∣s′) a π(· s ), and then evolves through fSf_S and π, driven by U(t+1),U(t+2),…U^(t+1),U^(t+2),…, which are drawn independently of (S(t),U(t))(S^(t),U^(t)). Conditional on Es′E_s , it therefore has the same law as a trajectory that starts at S(t+1)=s′S^(t+1)=s , sets A(t+1)A^(t+1) to a~∼π(⋅∣s′) a π(· s ), and follows π thereafter. Since the structural mechanisms are invariant over time, it follows that (1−γ)∑k≥1γkP(SA(t)=a,π(t+k)=s∗∣Es′) (1-γ) _k≥ 1γ^k\,P (S^(t+k)_A^(t)=a,π=s^* E_s ) =γ∑a~π(a~∣s′)dπ(s∗∣s′,a~). =γ _ aπ( a s )\,d^π(s^* s , a). Applying the law of total probability to the demonstrator’s action at s′s , as below Eq. (5), gives dπ(s∗∣s′,a~)=∑a~′P(a~′∣s′)Qπ(s′,a~,a~′).d^π(s^* s , a)= _ a P( a s )\,Q^π(s , a, a ). Combining the three displays, we obtain Qπ(s,a,a′) Q^π(s,a,a ) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′ℳ(s,a,a′,s′)∑a~,a~′π(a~∣s′) +γ _s T_M(s,a,a ,s ) _ a, a π( a s ) ×P(a~′∣s′)Qπ(s′,a~,a~′), × P( a s )\,Q^π(s , a, a ), that is, Qπ=ℳπQπQ^π=K^π_T_MQ^π. By uniqueness of this solution, Qℳπ(s,a,a′)=Qπ(s,a,a′)Q^π_T_M(s,a,a )=Q^π(s,a,a ). Finally, averaging over a′∼P(⋅∣s)a P(· s) and applying the law of total probability as below Eq. (5) gives ∑a′Qℳπ(s,a,a′)P(a′∣s)=Qπ(s,a). _a Q^π_T_M(s,a,a )\,P(a s)=Q^π(s,a). Since ℳ∈T_M , the infimum defining Q¯π(s,a) Q^π(s,a) is at most its value at ℳT_M, and hence Q¯π(s,a)≤Qπ(s,a)for every (s,a). Q^π(s,a)≤ Q^π(s,a) every (s,a). This completes the proof. ∎ Proof of Proposition 2. The proof has two steps. We first show that the positive goals generated by CtfRollout follow the goal-reaching occupancy of π in ℳM_T; the claim then follows by computing the Bayes-optimal critic of Eq. (9) and identifying it through Lemma A.1(c). Throughout, we consider untruncated rollouts, i.e., H=∞H=∞ (the effect of truncation is discussed after the proposition), and optimize Eq. (9) over all functions f. The critic is then determined on pairs (s,a)(s,a) in the support of the distribution ν of training pairs drawn from +D^+, and on goals with P−(s∗)>0P^-(s^*)>0. Consider the iteration h of CtfRollout. The demonstrator’s action a′(h)a (h) is drawn from μ(⋅∣s~(h))μ(· s^(h)) independently of the past given s~(h) s^(h). Marginalizing it out and using μ=P(⋅∣s)μ=P(· s), the next state satisfies P(s~(h+1)=s′∣s~(t:h),a~(t:h)) P ( s^(h+1)=s s^(t:h), a^(t:h) ) =∑a′(s~(h),a~(h),a′,s′)P(a′∣s~(h)) = _a T( s^(h), a^(h),a ,s )\,P(a s^(h)) =Pa~(h)(s′∣s~(h)), =P^T_ a^(h)(s s^(h)), by Eq. (8), and the next action is drawn from π(⋅∣s~(h+1))π(· s^(h+1)). Hence the rollout from (s,a)(s,a) has the trajectory distribution Pa,πP^T_a,π. Since K is drawn independently with P(K=k)=(1−γ)γkP(K=k)=(1-γ)γ^k, the positive goal s~+=S~(t+K) s^+= S^(t+K) satisfies P(s~+=s∗∣s,a) P( s^+=s^* s,a) =(1−γ)∑k=0∞γk =(1-γ) _k=0^∞γ^k ×Pa,π(S(t+k)=s∗∣S(t)=s) × P^T_a,π (S^(t+k)=s^* S^(t)=s ) =dπ(s∗∣s,a). =d^π_T(s^* s,a). We now compute the critic that maximizes Eq. (9). By the previous display, positive triples (s,a,s~+)(s,a, s^+) are distributed as q+(s,a,s∗)=ν(s,a)dπ(s∗∣s,a)q^+(s,a,s^*)=ν(s,a)\,d^π_T(s^* s,a). Since s−s^- is drawn independently of (s,a)(s,a), negative triples (s,a,s−)(s,a,s^-) are distributed as q−(s,a,s∗)=ν(s,a)P−(s∗)q^-(s,a,s^*)=ν(s,a)\,P^-(s^*). The objective in Eq. (9) therefore equals ∑s,a,s∗[q+logσ(f)+q−log(1−σ(f))], _s,a,s^* [q^+ σ(f)+q^- (1-σ(f) ) ], which decouples across (s,a,s∗)(s,a,s^*). For each triple with ν(s,a)>0ν(s,a)>0 and P−(s∗)>0P^-(s^*)>0, the map x↦q+logσ(x)+q−log(1−σ(x))x q^+ σ(x)+q^- (1-σ(x)) is strictly concave and is maximized at σ(x)=q+/(q++q−)σ(x)=q^+/(q^++q^-), i.e., at expx=q+/q− x=q^+/q^-, with the convention expx=0 x=0 when q+=0q^+=0. The factor ν(s,a)ν(s,a) cancels, so the optimal critic satisfies expf∗(s,a,s∗)=dπ(s∗∣s,a)P−(s∗). f^*(s,a,s^*)= d^π_T(s^* s,a)P^-(s^*). By Lemma A.1(c), dπ(s∗∣s,a)=Qπ(s,a)d^π_T(s^* s,a)=Q^π_T(s,a), and therefore expf∗(s,a,s∗)=1P−(s∗)⋅Qπ(s,a). f^*(s,a,s^*)= 1P^-(s^*)· Q^π_T(s,a). This proves the claim. ∎ Proof of Proposition 3. The proof has two steps and follows the approximate policy improvement argument of Eysenbach et al. (2022, Lemma 4.2), with ℳM_T in place of the true environment. We first show that the greedy policy π′π is 2δ2δ-greedy with respect to QπQ^π_T; the claim then follows by propagating this inequality through the Bellman operator of π′π . By Lemma A.1(c), QπQ^π_T and Qπ′Q^π _T are the fixed points of ℬπB^π_T and ℬπ′B^π _T, respectively. Let Q^(s,a)=P−(s∗)expf(s,a,s∗) Q(s,a)=P^-(s^*) f(s,a,s^*), so that |Q^−Qπ|≤δ| Q-Q^π_T|≤δ pointwise by assumption, and let Vπ(s)=∑aπ(a∣s)Qπ(s,a)V^π(s)= _aπ(a s)\,Q^π_T(s,a). Since P−(s∗)>0P^-(s^*)>0, the map x↦P−(s∗)expx P^-(s^*) x is increasing, so π′(s)∈argmaxaf(s,a,s∗)π (s)∈ _af(s,a,s^*) also maximizes Q^(s,⋅) Q(s,·). Hence, for every s, Qπ(s,π′(s)) Q^π_T(s,π (s)) ≥Q^(s,π′(s))−δ ≥ Q(s,π (s))-δ ≥∑aπ(a∣s)Q^(s,a)−δ ≥ _aπ(a s)\, Q(s,a)-δ ≥Vπ(s)−2δ, ≥ V^π(s)-2δ, where the first and last inequalities use |Q^−Qπ|≤δ| Q-Q^π_T|≤δ, and the middle one uses that a maximum dominates any average. We next propagate this bound. Since π′π is deterministic, (ℬπ′Qπ)(s,a) (B^π _TQ^π_T)(s,a) =(1−γ)s=s∗ =(1-γ) 1\s=s^*\ +γ∑s′Pa(s′∣s)Qπ(s′,π′(s′)). +γ _s P^T_a(s s)\,Q^π_T(s ,π (s )). Applying the previous inequality at every s′s and using that QπQ^π_T is the fixed point of ℬπB^π_T, we obtain ℬπ′Qπ≥ℬπQπ−2γδ=Qπ−2γδB^π _TQ^π_T ^π_TQ^π_T-2γδ=Q^π_T-2γδ pointwise. We now show by induction that (ℬπ′)nQπ≥Qπ−2γδ∑k=0n−1γk(B^π _T)^nQ^π_T≥ Q^π_T-2γδ _k=0^n-1γ^k for every n≥1n≥ 1. The case n=1n=1 is the display above. If the bound holds for n, then the monotonicity of ℬπ′B^π _T and the identity ℬπ′(q−c)=ℬπ′q−γcB^π _T(q-c)=B^π _Tq-γ c from Lemma A.1(a) give (ℬπ′)n+1Qπ (B^π _T)^n+1Q^π_T ≥ℬπ′Qπ−2γ2δ∑k=0n−1γk ^π _TQ^π_T-2γ^2δ _k=0^n-1γ^k ≥Qπ−2γδ∑k=0nγk. ≥ Q^π_T-2γδ _k=0^nγ^k. Letting n→∞n→∞, Lemma A.1(a) gives (ℬπ′)nQπ→Qπ′(B^π _T)^nQ^π_T→ Q^π _T, and therefore Qπ′(s,a)≥Qπ(s,a)−2γδ1−γfor all (s,a).Q^π _T(s,a)≥ Q^π_T(s,a)- 2γδ1-γ all (s,a). This completes the proof. ∎ Appendix B Implementation Details This section provides implementation details for CauCRL beyond the algorithmic description in Sec. 3. The continuous-control experiments use environment-specific parameterizations of the nominal behavioral policy and counterfactual transition model. Network architectures and optimization hyperparameters are reported in Appendix C.4. B.1 Nominal Behavioral Policy For both continuous-control environments, we parameterize the nominal behavioral policy as a K-component censored Gaussian mixture density network with K=5K=5 and diagonal covariance, μϕ(a∣c)=∑k=1Kwk(c)cens(a,μk(c),σk(c)), _φ(a c)= _k=1^Kw_k(c)\,N^cens_A (a; _k(c), _k(c) ), where censN^cens_A denotes the Gaussian distribution censored to the action box A, with continuous density in the interior and point mass at clipped boundaries. The conditioning context is c=sc=s for AntMaze and c=(s,g)c=(s,g) for PointMaze. The PointMaze context contains only learner-visible information; privileged hazard and demonstrator metadata are excluded. For Windy Corridor, the action space is discrete, and we parameterize the nominal behavioral policy as a categorical distribution with context c=sc=s, μϕ(a∣c) _φ(a c) =softmax(fϕ(c))a, =softmax (f_φ(c) )_a, a a ∈left,right,forward,wait, ∈\ left, right, forward, wait\, where fϕf_φ is an MLP with two hidden layers of 128 ReLU units applied to a one-hot encoding of the agent’s x-coordinate, y-coordinate, and heading. The wait action is the demonstrator’s no-op, which it uses to pause when the wind is unfavorable. It is kept in the support of μϕ _φ because it occurs in the demonstrations, even though the learned policy selects only among the three movement actions. In all three environments, the nominal-policy likelihood is ℒμ(ϕ)=−(c,a)∼μ[logμϕ(a∣c)].L_μ(φ)=-E_(c,a) _μ [ _φ(a c) ]. (11) The environments use different fitting procedures. In PointMaze, μϕ _φ is trained separately on the 4,8004,800 expert episodes, excluding the uniform-coverage and blind-demonstrator subsets, and the checkpoint is selected on a validation split before the model is frozen. In AntMaze, ℒμL_μ is optimized jointly with the counterfactual-transition objective with coefficient λμ=1 _μ=1, and the final iterate is used. In Windy Corridor, μD_μ consists of the 1,0001,000 demonstrations in +D^+, on which ℒμL_μ reduces to the four-way cross-entropy. We optimize it separately with Adam (learning rate 10−310^-3, batch size 256) for 3,0003,000 updates and use the final iterate. Because the state space is small (15×1515× 15 cells and four headings), we evaluate the frozen μϕ _φ once on every reachable state and store the resulting table, from which the counterfactual rollouts sample the nominal action. For AntMaze, the learned mixture weights are also used by the rollout construction in Appendix B.3. Network and optimization hyperparameters are reported in Appendix C.4. B.2 Counterfactual Transition Model The three environments use different parameterizations of the counterfactual transition model Tη(s,a,a′,⋅)T_η(s,a,a ,·) introduced in Sec. 3.1, where a denotes the target action and a′a the demonstrator action. Windy Corridor uses a finite categorical model with locally restricted support, while PointMaze and AntMaze use neural transition models with different spectral-normalization schemes. Windy Corridor. Because the state and action spaces are finite, the counterfactual transition is represented explicitly over states and action pairs. Actions other than forward have deterministic one-step dynamics and are represented by point masses at their corresponding successors. Factual forward transitions use the empirical one-step distribution P^(⋅∣s,a) P(· s,a). For off-diagonal forward transitions, we use a categorical kernel κη _η to combine the empirical transition distributions associated with the two actions: Tη(s,a,a′,y) T_η(s,a,a ,y) =(1−ϵ)∑x,x′P^(x∣s,a)P^(x′∣s,a′) =(1-ε) _x,x P(x s,a)\, P(x s,a ) (12) ×κη(y∣x,x′,s,a,a′)+ϵ||, × _η(y x,x ,s,a,a )+ ε|S|, with ϵ=10−3ε=10^-3. The kernel is categorical over 7575 local offsets around the deterministic successor of the target action, corresponding to Δx,Δy∈−2,…,2 x, y∈\-2,…,2\ and heading changes in 0,±1\0,± 1\. Its support is restricted to candidate successors within radius r=2r=2 of both factual anchors under the state metric used by the model. The kernel is parameterized using translation-invariant local features and does not receive the latent wind variable. It is trained by cross-entropy using the counterfactual supervision described in Sec. 3.1. PointMaze. PointMaze uses a stochastic generator that predicts the two-dimensional planar displacement of the agent conditioned on the current state, target action a, and demonstrator action a′a . Let Δx=x′−x x=x -x denote the observed planar displacement. For each input, the model produces M=8M=8 samples Δx~1,…,Δx~M x_1,…, x_M and is trained with the Monte Carlo energy score ℓ^ES _ES =1M∑m=1M‖Δx~m−Δx‖2 = 1M _m=1^M\| x_m- x\|_2 (13) −12M(M−1)∑m≠n∥Δx~m−Δx~n∥2. - 12M(M-1) _m≠ n\| x_m- x_n\|_2. The sampled displacement is clipped before constructing the next stacked state by shifting the position history and inserting the generated planar position. To control sensitivity to the target action, the linear maps on the target-action pathway are projected to spectral norm at most 11. Together with the 11-Lipschitz nonlinearities and clipping operation, this gives a Lipschitz bound of 11 with respect to the target action in raw action units. An auxiliary classifier is trained jointly to predict the hazard-transition indicator. Let diagD_diag and offD_off denote the diagonal and off-diagonal subsets of a data partition D. For each regime r∈diag,offr∈\diag,off\, we define ℒr()=1|r|∑i∈rℓ^ES,i+λon||∑i∈rℓonset,iBCE,L_r(D)= 1|D_r| _i _r _ES,i+ _on|D| _i _r ^BCE_onset,i, (14) where λon=0.25 _on=0.25. The complete PointMaze transition-model objective is ℒETT()=ℒdiag()+ℒoff().L_ETT(D)=L_diag(D)+L_off(D). (15) The onset component is estimated using class-stratified sampling and reweighted to the empirical event prevalence. AntMaze. AntMaze uses a deterministic predictor of the full state displacement, conditioned on the current state, target action a, and demonstrator action a′a . Its predictive distribution is a Dirac measure, for which the energy score reduces to ℓES=‖gη(s,a,a′)−(s′−s)‖2. _ES=\|g_η(s,a,a )-(s -s)\|_2. (16) Spectral normalization is applied to every layer of the displacement network using 1010 power iterations and coefficient 2.02.0. This coefficient controls the per-layer spectral scale and is used as a regularizer; it is not interpreted as a global Lipschitz constant of the transition network. Additional architecture and optimization hyperparameters are reported in Appendix C.4. B.3 Counterfactual Rollout and Replay Construction Model-generated continuations are stored separately from the original offline demonstrations, which remain fixed throughout training. Generated trajectories preserve their temporal structure so that future-goal relabeling is performed within the same recorded or generated trajectory. Windy Corridor. For Windy Corridor, each continuation starts from a recorded state–action pair (s,a)(s,a) drawn uniformly from +D^+, and the recorded action is kept as the first target action. At each generated state, a nominal demonstrator action a′a is sampled from the tabulated μϕ _φ, the next state is sampled from the tabulated Tη(s,a,a′,⋅)T_η(s,a,a ,·), and the next target action is sampled from the current learner. Because the target action always comes from the learner, the factual branch is used exactly when the learner’s action coincides with the sampled demonstrator action, and no separate retention probability is needed. A continuation ends at lava, at the goal, or at the rollout horizon. The augmentation buffer is regenerated with the current policy at the start of each training round. Contrastive positives are relabeled from future states of the generated continuations, while behavioral cloning continues to use the recorded actions in +D^+. PointMaze. For PointMaze, we retain a recorded state–action pair and replace its subsequent continuation with a rollout from the learned counterfactual transition model. The generated suffix supplies future states for contrastive goal relabeling, while behavioral-cloning supervision continues to use the recorded actions from the offline demonstrations. AntMaze. For AntMaze, the recorded prefix is retained up to the route decision and the remaining trajectory is replaced by a model-generated continuation. At each generated state, a nominal demonstrator action is sampled from μϕ _φ and a target action is sampled from the current learner. The probability of retaining the factual branch is pdiag(s)=∑k=1Kwk(s)2,p_diag(s)= _k=1^Kw_k(s)^2, computed from the mixture weights of the nominal behavioral policy; otherwise the transition model is evaluated using the learner-selected action. We generate continuations of length 9090 and refresh the augmentation buffer throughout actor–critic training using the current policy. Each refresh contains 2,0002,000 generated continuations, and generated and recorded trajectories each contribute half of the training minibatch. Additional rollout parameters are listed in Appendix C.4. B.4 Contrastive Critic and Actor Training CauCRL uses the same actor–critic parameterization as the corresponding CRL baseline within each environment. PointMaze represents goals using the full stacked state, whereas AntMaze uses the planar (x,y)(x,y) position. The implementation uses in-batch negative sampling for contrastive critic training. PointMaze uses a single critic, while AntMaze uses two independently parameterized critics and evaluates actor actions using the minimum of their scores. The actor is a tanh-Gaussian goal-conditioned policy. Behavioral-cloning regularization uses recorded offline actions; PointMaze draws these actions from a separately balanced BC minibatch, whereas AntMaze uses the standard training minibatch. Network dimensions and optimization hyperparameters are reported in Appendix C.4. Appendix C Experimental Setup C.1 Windy Corridor Geometry. Windy Corridor is a 15×1515× 15 MiniGrid (Chevalier-Boisvert et al., 2023) environment with two routes from the start (1,1)(1,1) to the goal (13,1)(13,1). The short route runs east along row y=1y=1, directly above a row of lava cells at x∈3,4,5x∈\3,4,5\ and x∈8,9,10x∈\8,9,10\. The safe route descends along column x=1x=1, follows row y=13y=13, and returns north along column x=13x=13; it takes about 3939 steps, compared with about 1313 on the short route. Entering a lava cell terminates the episode as a failure. Dynamics and wind. The agent’s actions are left, right, and forward; the demonstrator can additionally wait. A wind direction is drawn independently at every step, east, south, west, or north with probability 0.10.1 each and still with probability 0.60.6. It is observed by the demonstrator before acting and resampled after the transition. Wind affects only forward: a tailwind moves the agent two cells, a headwind cancels the move, and a crosswind moves it one cell forward and then one cell in the wind direction. On the short route, a south wind therefore pushes the agent into the lava from six positions along row y=1y=1; the safe route has no such positions. Observation. The learner observes its cell and heading, st=(xt,yt,ht)s_t=(x_t,y_t,h_t), and the goal is the target cell (x,y)(x,y). The wind is never included in the learner observation. Demonstrator and dataset. At the beginning of each episode, the demonstrator selects the short route with probability 0.950.95 and the safe route otherwise. On the short route, it observes the current wind and waits whenever it stands at one of the six exposed positions under a south wind; on the safe route, it follows the corridor without consulting the wind. The demonstration dataset +D^+ contains 1,0001,000 episodes and 14,02814,028 transitions, of which 952952 follow the short route and 4848 the safe route. Every episode reaches the goal, so the dataset contains no lava failures. All offline RL methods use +D^+. A separate pool of 1,0001,000 trajectories of a uniformly random policy is used only to construct P−P^- for the counterfactual transition model (Appendix B.2) and is excluded from actor–critic replay. Evaluation. Each evaluation checkpoint uses 300300 episodes with a fixed set of environment seeds, deterministic policy actions, and a horizon of 400400 steps under the natural wind distribution. Success is recorded when the agent reaches the goal cell. We additionally report the robust-route rate, the fraction of episodes that enter the safe-route row y=13y=13. C.2 PointMaze Geometry. PointMaze is a 9×59× 5 grid maze with two start-to-goal routes. The agent starts at (0.5,3.5)(0.5,3.5) and the goal is at (8.5,3.5)(8.5,3.5). The short route runs along row y=3y=3 and crosses three hazard cells at (3,3)(3,3), (4,3)(4,3), and (5,3)(5,3). The safe route branches downward at x=1x=1, follows row y=1y=1, and rejoins the short route at x=7x=7. Episodes have a horizon of 5050 steps. Dynamics and hazard. Actions are two-dimensional velocities clipped to [−1,1]2[-1,1]^2, with additive Gaussian noise of standard deviation 0.010.01. Each environment step is integrated over 1010 simulation substeps with per-axis wall collision. Each of the three hazard cells is independently active with probability 0.300.30 at every step. The hazard indicators are observed by the demonstrator before acting and are resampled after the transition. If the agent ends a step inside an active hazard cell, it enters an absorbing failure state: its position remains fixed, subsequent actions have no effect, and the goal reward can no longer be obtained. Observation. The learner observes the four most recent planar positions, st=[xt,yt,…,xt−3,yt−3],s_t=[x_t,y_t,…,x_t-3,y_t-3], giving an 88-dimensional state representation. The goal is represented in the corresponding stacked form. Hazard indicators are never included in the learner observation. Demonstrator and dataset. At the beginning of each episode, the demonstrator selects either the safe route or the short route. Safe-route demonstrations follow a shortest path computed with the hazard cells treated as blocked. On the short route, the demonstrator observes the current hazard indicators and, before entering a hazard cell, avoids actions predicted to terminate in an active cell. The offline dataset contains 6,6006,600 episodes and 330,000330,000 transitions: Behavior Episodes Wait at the hazard, then cross 3,0733,073 Cross immediately 1,4701,470 Uniform coverage 1,2001,200 Blind demonstrator 600600 Forced safe detour 257257 The resulting dataset contains 4,5144,514 successful episodes, 1,4511,451 absorbed failures, and 635635 timeouts. All offline RL methods use the complete dataset. The nominal behavioral policy described in Appendix B.1 is fitted only on the 4,8004,800 demonstrator episodes, excluding the uniform-coverage and blind-demonstrator subsets. Evaluation. Each evaluation checkpoint uses 300300 episodes with a fixed set of environment seeds and deterministic policy actions. Success is recorded when the agent reaches the task goal according to the environment reward. We additionally report the safe-detour rate. An episode is classified as taking the detour if it enters the lower route before entering any hazard cell; otherwise, entering a hazard cell first is classified as taking the shortcut. C.3 AntMaze Geometry. AntMaze is a continuous-control ring maze with two routes from the start to the goal. The Ant starts at (0,0)(0,0) and the goal is at (24,0)(24,0). The direct route crosses two stochastic rockfall zones, while a longer perimeter route avoids both. Episodes have a horizon of 800800 steps. The two rockfall zones lie on the direct route. Zone 1 spans x∈[6.6,9.4]x∈[6.6,9.4] and Zone 2 spans x∈[14.6,17.4]x∈[14.6,17.4], with both covering y∈[−2,2]y∈[-2,2]. The learner observes a 2929-dimensional Ant state and selects 88-dimensional joint torques; the goal is represented by its planar (x,y)(x,y) position. Rockfall states and activation schedules are hidden from the learner. Hazard. Each rockfall zone is independently activated with probability pactive=0.40p_active=0.40 at the beginning of an episode. Conditional on activation, the start times are sampled independently as t0(1)∼10,…,45,t0(2)∼90,…,125.t_0^(1) \10,…,45\, t_0^(2) \90,…,125\. An activated zone remains active for 7272 steps and drops four rocks. Physical contact between a rock and the Ant produces an absorbing failure: the agent loses control for the remainder of the episode and can no longer receive the goal reward. Demonstrator and dataset. The demonstrator first chooses between the direct and detour routes using an episode-level route variable independent of the rockfall activations. Detour trajectories follow the safe perimeter route without consulting the rockfall schedules. On direct-route trajectories, the demonstrator observes the activation schedule of each zone and decides whether to enter or wait. A crossing is attempted only when the predicted crossing interval, using a 5050-step crossing estimate and an 88-step safety margin, does not overlap the corresponding rockfall interval. The demonstration dataset contains 1,0001,000 episodes and 311,005311,005 transitions, collected with pactive=0.40p_active=0.40. Direct and detour routes each account for 50%50\% of the demonstrations. The dataset contains 994994 successful episodes and 66 timeouts, with no rock-contact failures. A separate pool of 300300 episodes is used only for fitting the counterfactual transition model and is excluded from actor–critic replay. This pool is collected in the same environment by overriding a demonstrator decision to wait at Zone 1 with a go decision and retaining trajectories in which the override produces a realized Zone 1 rock-contact failure. Evaluation. Each evaluation checkpoint uses 300300 episodes with fixed environment seeds and deterministic policy actions over the full 800800-step horizon. Evaluation uses the same rockfall activation probability pactive=0.40p_active=0.40 as data collection. Success is recorded when the Ant reaches the task goal at (24,0)(24,0) according to the environment reward. C.4 Network Architectures and Training Hyperparameters Table 1 summarizes the actor–critic configurations used for all three environments. Within each environment, CRL and CauCRL use the same actor and critic architectures. All experiments were run on NVIDIA GeForce RTX 5090 GPUs. Table 1: Actor–critic architectures and training hyperparameters. Windy Corridor PointMaze AntMaze State dimension 3 8 29 Action dimension 3 (discrete) 2 8 Goal dimension 2 8 2 Episode horizon 400 50 800 Batch size 64 256 1024 Actor hidden layers [128][128] [256,256][256,256] [1024,1024][1024,1024] Critic hidden layers [128][128] [256,256][256,256] [1024,1024][1024,1024] Activation ReLU ReLU ReLU Representation dimension 64 64 16 Number of critics 1 1 2 Discount γ 0.99 0.95 0.999 BC coefficient λBC _BC 1 0.05 0.05 Entropy coefficient 0 0 0 Actor learning rate 2×10−32× 10^-3 3×10−43× 10^-4 3×10−43× 10^-4 Critic learning rate 2×10−32× 10^-3 3×10−43× 10^-4 3×10−43× 10^-4 Optimizer SGD Adam Adam Main updates 100k 300k 100k Critic warm-up – 30k – Evaluation interval 250 1k 400 Evaluation episodes 300 300 300 For Windy Corridor, the critic encodes the state–action pair and the goal with one-hot features over grid states and cells, and the contrastive score uses temperature 0.070.07. The actor receives the raw (x,y,h)(x,y,h) state. Its objective (1−λ)π[f]+λlogπ(a∣s,g)(1-λ)\,E_π[f]+λ π(a s,g) with λ=0.5λ=0.5 corresponds to λBC=1 _BC=1. Actor minibatches are drawn uniformly over distinct recorded state–action pairs rather than over transitions, so the rare safe-route decisions are not outweighed by the frequent short-route ones. Table 2 reports the nominal behavioral-policy configurations. As described in Appendix B.1, the PointMaze and Windy Corridor models are trained separately and then frozen, whereas the AntMaze model is optimized jointly with the counterfactual transition model. Table 2: Nominal behavioral-policy architectures and training settings. Windy Corridor PointMaze AntMaze Conditioning context state (one-hot, 34-D) state + goal (16-D) state (29-D) Action dimension 4 (discrete) 2 8 Mixture components K – (categorical) 5 5 Hidden layers [128,128][128,128] [128,128][128,128] [512,512][512,512] Activation ReLU SiLU ReLU Covariance – diagonal diagonal Action likelihood categorical censored on [−1,1]2[-1,1]^2 censored on empirical action box Scale parameterization – σ∈[0.02,2.0]σ∈[0.02,2.0], |μ|≤3|μ|≤ 3 logσ∈[−5,2] σ∈[-5,2] Spectral normalization none none all layers, coef. 2.02.0, 10 power iterations Batch size 256 512 1024 Learning rate 10−310^-3 3×10−43× 10^-4 10−310^-3 Gradient clipping – 10.0 – Training updates 3k 20k 20k Optimization separate, then frozen separate, then frozen joint with transition model, λμ=1 _μ=1 Checkpoint selection final iterate best validation NLL final iterate Training data 900 expert episodes 4,800 expert episodes full demonstration pool Validation split – 10% by episode episode-level holdout shared with the counterfactual transition model For PointMaze, the 4,8004,800 nominal-policy episodes are split by complete episode into 4,3214,321 training and 479479 validation episodes. Validation NLL is evaluated every 500500 updates. Context normalization statistics are fit on the training split only. For AntMaze, the nominal-policy head uses the same positive-sample normalization statistics and episode-level holdout as the transition model. For Windy Corridor, the nominal policy is trained on a random 90%90\% episode split of +D^+ and is tabulated over all grid states after training. Table 3 summarizes the learned counterfactual-transition models. Table 3: Counterfactual transition-model architectures and training settings. Windy Corridor PointMaze AntMaze Prediction target next state (75 local offsets) planar displacement (2-D) full-state displacement (29-D) Hidden layers [128,128][128,128] [128,128][128,128] + generator [512,512][512,512] Activation ReLU ReLU ReLU Lipschitz control locality restriction, r=2r=2 target-action spectral projection, La≤1L_a≤ 1 all-layer spectral normalization, coef. 2.02.0 Power iterations – – 10 Predictive form categorical (tabulated) stochastic, M=8M=8 deterministic (Dirac) Batch size 256 256 1024 Training updates 6k 2.5k 20k Learning rate 10−310^-3 3×10−43× 10^-4 10−310^-3 Optimizer Adam Adam Adam Checkpoint selection final iterate earliest validation minimum final iterate For PointMaze, checkpoint selection uses the earliest minimum of ℒETT()=ℒdiag()+ℒoff(),L_ETT(V)=L_diag(V)+L_off(V), evaluated on the held-out validation partition V. AntMaze and Windy Corridor use the final transition-model iterate without validation-based checkpoint selection. For Windy Corridor, the factual next-state network and the counterfactual kernel κη _η share the settings in Table 3, and the background weight is ϵ=10−3ε=10^-3. The AntMaze counterfactual-replay configuration is summarized in Table 4. Table 4: AntMaze counterfactual rollout and replay hyperparameters. Hyperparameter Value Rollout start route-dependent split point Rollout horizon 90 Rollouts per refresh 2,000 Refresh interval every training round Generated-data fraction 0.5 State feasibility state bounds and maze free space The Windy Corridor configuration is summarized in Table 5. Table 5: Windy Corridor counterfactual rollout and replay hyperparameters. Hyperparameter Value Rollout start recorded (s,a)(s,a), uniform over +D^+ Rollout horizon 400 Rollouts per refresh 10,000 Refresh interval every training round (5k updates) Training rounds 20 Generated-data fraction 1.0 (critic); actor BC uses +D^+ State feasibility tabulated over legal grid states Baseline configurations. For GCIQL, both continuous-control environments use expectile 0.90.9 and Polyak coefficient 0.0050.005. The PointMaze actor uses AWR with inverse temperature 3.03.0 and weights clipped at 100100, whereas AntMaze uses the DDPG+BC-style actor with coefficient 0.30.3. For HIQL, both continuous-control environments use three hidden layers of size 512512, layer normalization, representation dimension 1010, expectile 0.70.7, low- and high-level actor coefficients of 3.03.0, Polyak coefficient 0.0050.005, learning rate 3×10−43× 10^-4, and batch size 10241024. PointMaze uses discount 0.950.95 and a 1010-step subgoal horizon, whereas AntMaze uses discount 0.990.99 and a 2525-step subgoal horizon. HIQL is trained for 330,000330,000 updates with checkpoints every 1,0001,000 updates. The PointMaze subgoal horizon differs from the OGBench (Park et al., 2025) default of 2525. We evaluated horizons in 5,10,25\5,10,25\ together with actor coefficients in 3,10\3,10\; the 55-step setting produced nearly degenerate subgoal advantages because most sampled subgoals already lay within the success radius, and we retained a horizon of 1010. For Windy Corridor, all baselines are trained on +D^+ for 100,000100,000 updates and evaluated every 250250 updates with the protocol of Appendix C.1. Their minibatches are drawn uniformly over distinct recorded state–action pairs, as for CRL. GCBC uses the CRL actor trained only with the behavioral-cloning loss on future-state relabeled goals. GCIQL uses expectile 0.90.9, Polyak coefficient 0.0050.005, and the AWR actor with inverse temperature 3.03.0 and weights clipped at 100100. Its Q and V networks use one hidden layer of 128128 units on the critic’s one-hot features and are trained with Adam at learning rate 3×10−43× 10^-4. HIQL uses the continuous-control configuration above with a categorical low-level actor, discount 0.990.99, and a 33-step subgoal horizon, since Windy Corridor demonstrations are only 1010 to 4040 steps long. For both GCIQL and HIQL, goal relabeling uses a binary 1/01/0 reward for the first non-failure arrival at the relabeled goal rather than the 0/−10/-1 convention used by the reference OGBench implementations. Absorbing failure states use a zero bootstrap mask, while time-limit truncations continue to bootstrap. Other baseline settings follow the corresponding reference implementations. C.5 Result Details Table 6 reports the complete per-seed results for all three benchmarks: for each method and random seed, the success rate averaged over the final 10%10\% of evaluation checkpoints. These are the per-seed values behind the means reported in Section 4 and plotted in Figures 2 and 5. Table 6: Per-seed final success rates (mean over the final 10%10\% of evaluation checkpoints) on all three benchmarks. Benchmark Method Per-seed Windy CauCRL 0.797 0.984 0.919 0.895 1.000 CRL 0.513 0.513 0.513 0.513 0.513 GCIQL 0.513 0.513 0.513 0.513 0.513 GCBC 0.586 0.562 0.635 0.574 0.574 HIQL 0.513 0.513 0.513 0.513 0.513 PointMaze CauCRL 1.000 1.000 1.000 1.000 1.000 CRL 0.347 0.347 0.347 0.347 0.347 GCIQL 0.347 0.347 0.347 0.347 0.347 GCBC 0.347 0.347 0.347 0.347 0.347 HIQL 0.194 0.173 0.166 0.213 0.135 AntMaze CauCRL 0.712 0.869 0.798 0.820 0.879 CRL 0.596 0.620 0.646 0.677 0.605 GCIQL 0.532 0.612 0.516 0.913 0.508 GCBC 0.476 0.499 0.429 0.529 0.559 HIQL 0.401 0.390 0.421 0.417 0.418 Appendix D Related Work Contrastive goal-conditioned reinforcement learning. Contrastive reinforcement learning (CRL) casts goal-conditioned control as learning a critic that distinguishes future achieved states from unrelated states, thereby connecting contrastive representation learning with goal-reaching value functions (Eysenbach et al., 2022). Subsequent work has extended this framework along several directions. Stable CRL studies architectural and optimization choices that make contrastive goal-conditioned policies effective on offline, image-based robotic manipulation tasks (Zheng et al., 2024a). TD InfoNCE replaces Monte-Carlo future-state estimation with a temporal-difference contrastive objective, enabling trajectory stitching and more sample-efficient long-horizon prediction (Zheng et al., 2024b). More recently, JaxGCRL systematically studies modern CRL design choices and finds strong performance with symmetric InfoNCE objectives and distance-based critics (Bortkiewicz et al., 2025), while quasimetric representations combine contrastive successor representations with temporal-distance structure for offline goal-conditioned learning (Myers et al., 2025). Our approach is complementary to these developments: CauCRL modifies the transition and future-outcome supervision used to train a goal-conditioned critic, rather than relying on a particular contrastive objective. For example, InfoNCE-based CRL can use counterfactually augmented future states as positive pairs, while a TD-based contrastive method can apply the same robust transition construction to the one-step successors used in its bootstrapped targets.