Paper deep dive
Scalable Causal Imitation Learning
Eylam Tagor, Mingxuan Li, Elias Bareinboim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/21/2026, 4:26:20 AM
Summary
The paper introduces Causal Soft Q Imitation Learning (Causal SQIL) and Causal Inverse soft-Q Learning (Causal IQ-Learn), two off-policy algorithms designed to address the limitations of existing Causal Imitation Learning (CIL) methods in long-horizon, high-dimensional continuous control tasks. By combining causal adjustment frameworks with inverse reinforcement learning objectives and approximating the sequential pi-backdoor criterion via a sliding window, these methods effectively handle unobserved confounders and outperform prior CIL and non-causal imitation methods.
Entities (12)
Relation Signals (11)
Eylam Tagor → authored → Scalable Causal Imitation Learning
confidence 99% · Scalable Causal Imitation Learning Eylam Tagor, Mingxuan Li, Elias Bareinboim
Sequential pi-backdoor criterion → addresses → unobserved confounders
confidence 95% · By identifying appropriate adjustment sets via the sequential $π$-backdoor criterion, causal imitation learning (CIL) provides a framework for approximating the expert's policy from confounded data.
Causal BC → suffersfrom → compounding errors
confidence 95% · Causal BC suffers from compounding errors
Causal GAIL → suffersfrom → instability
confidence 95% · Causal GAIL is unstable and sample-inefficient
Causal SQIL → uses → Sequential pi-backdoor criterion
confidence 95% · Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential $π$-backdoor criterion
Causal IQ-Learn → uses → Sequential pi-backdoor criterion
confidence 95% · Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential $π$-backdoor criterion
Causal GAIL → suffersfrom → Instability
confidence 92% · Causal GAIL is unstable and sample-inefficient
Causal BC → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations, but it struggles when the imitator's and expert's observations are mismatched and unobserved confounders are present in expert demonstrations. By identifying appropriate adjustment sets via the sequential $\pi$-backdoor criterion, causal imitation learning (CIL) provides a framework for approximating the expert's policy from confounded data. However, existing CIL methods, Causal Behavioral Cloning (Causal BC) and Causal Generative Adversarial Imitation Learning (Causal GAIL), are designed for short-horizon, low-dimensional settings. When applied to continuous control tasks with long horizons and high-dimensional state-action spaces, these methods exhibit poor performance: Causal BC suffers from compounding errors, Causal GAIL is unstable and sample-inefficient, and sequential $\pi$-backdoor adjustment becomes impractical. We introduce Causal Soft Q Imitation Learning (SQIL) and Causal Inverse soft-Q Learning (IQ-Learn), two off-policy causal imitation learning algorithms that combine the causal adjustment framework with state-of-the-art inverse reinforcement learning objectives. Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential $\pi$-backdoor criterion, exploiting the causal structure of continuous control environments to reduce the full-horizon adjustment to a fixed-size sliding window. We evaluate all methods in a suite of confounded environments and find that Causal SQIL and Causal IQ-Learn substantially outperform prior CIL algorithms on long-horizon tasks, sometimes surpassing the expert, whereas all causally unaware imitation methods fail to learn meaningful behavior.
Tags
Links
- Source: https://arxiv.org/abs/2607.17003v1
- Canonical: https://arxiv.org/abs/2607.17003v1
Trouble viewing inline? Open PDF directly →
Full Text
140,977 characters extracted from source content.
Expand or collapse full text
Scalable Causal Imitation Learning Eylam Tagor, Mingxuan Li, Elias Bareinboim Keywords: Causality, Imitation Learning, Inverse Reinforcement learning, Unobserved Confounders Summary Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations, but it struggles when the imitator’s and expert’s observations are mismatched and unobserved confounders are present in expert demonstrations. By identifying appropriate adjustment sets via the sequential π-backdoor criterion, causal imitation learning (CIL) provides a framework for approximating the expert’s policy from confounded data. However, existing CIL methods, Causal Behavioral Cloning (Causal BC) and Causal Generative Adversarial Imitation Learning (Causal GAIL), are designed for short-horizon, low-dimensional settings. When applied to continuous control tasks with long horizons and high-dimensional state-action spaces, these methods exhibit poor performance: Causal BC suffers from compounding errors, Causal GAIL is unstable and sample-inefficient, and sequential π-backdoor adjustment becomes impractical. We introduce Causal Soft Q Imitation Learning (SQIL) and Causal Inverse soft-Q Learning (IQ-Learn), two off-policy causal imitation learning algorithms that combine the causal adjustment framework with state-of-the-art inverse reinforcement learning objectives. Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential π-backdoor criterion, exploiting the causal structure of continuous control environments to reduce the full-horizon adjustment to a fixed-size sliding window. We evaluate all methods in a suite of confounded environments and find that Causal SQIL and Causal IQ-Learn substantially outperform prior CIL algorithms on long-horizon tasks, sometimes surpassing the expert, whereas all causally unaware imitation methods fail to learn meaningful behavior. Contribution(s) 1. We introduce Causal SQIL and Causal IQ-Learn, two CIL algorithms for continuous-control settings with unobserved confounders. Both algorithms apply state-of-the-art inverse soft Q-learning objectives to causally-adjusted state representations. Context: Prior CIL algorithms (zhang2020causal; kumor2021sequential; ruan2023causal; ruan2024partial) are only suitable for short-horizon, low-dimensional domains. Standard (non-causal) SQIL (reddy2020sqil) and IQ-Learn (garg2021iqlearn) assume no confounding. 2. We develop an approximation of the sequential π-backdoor criterion that reduces the adjustment set computation from the full horizon to a fixed-size sliding window, making causal adjustment tractable for environments with horizons of a thousand or more steps. Context: The sequential π-backdoor criterion (kumor2021sequential) provides a complete graphical characterization of imitability given knowledge of the causal diagram but is infeasible to compute and bloats dimensionality; our approximate adjustment is correct assuming bounded temporal influence and endogenous time-homogeneity. 3. We provide an empirical evaluation across confounded high-dimensional long-horizon control tasks, demonstrating that Causal SQIL and Causal IQ-Learn achieve competitive success rates on tasks where Causal BC and Causal GAIL degrade and all causally unaware methods fail to learn meaningful policies. Context: These environments are augmentations of OGBench (park2025ogbench) tasks with latent confounders and partial observability modeled by structural causal models to mimic real-world imitation tasks. Prior empirical evaluations of CIL use short-horizon discrete MDPs or low-dimensional continuous environments. Abstract Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations, but it struggles when the imitator’s and expert’s observations are mismatched and unobserved confounders are present in expert demonstrations. By identifying appropriate adjustment sets via the sequential π-backdoor criterion, causal imitation learning (CIL) provides a framework for approximating the expert’s policy from confounded data. However, existing CIL methods, Causal Behavioral Cloning (Causal BC) and Causal Generative Adversarial Imitation Learning (Causal GAIL), are designed for short-horizon, low-dimensional settings. When applied to continuous control tasks with long horizons and high-dimensional state-action spaces, these methods exhibit poor performance: Causal BC suffers from compounding errors, Causal GAIL is unstable and sample-inefficient, and sequential π-backdoor adjustment becomes impractical. We introduce Causal Soft Q Imitation Learning (SQIL) and Causal Inverse soft-Q Learning (IQ-Learn), two off-policy causal imitation learning algorithms that combine the causal adjustment framework with state-of-the-art inverse reinforcement learning objectives. Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential π-backdoor criterion, exploiting the causal structure of continuous control environments to reduce the full-horizon adjustment to a fixed-size sliding window. We evaluate all methods in a suite of confounded environments and find that Causal SQIL and Causal IQ-Learn substantially outperform prior CIL algorithms on long-horizon tasks, sometimes surpassing the expert, whereas all causally unaware imitation methods fail to learn meaningful behavior. Repository: https://github.com/CausalAILab/Scalable-Causal-Imitation-Learning 1 Introduction Imitation learning (IL) has become a central paradigm in robotics and control tasks as an alternative to reinforcement learning (RL) in domains where reward signals are unavailable, sparse, or difficult to engineer (osa2018; zare2023survey). Rather than optimizing a task-specific reward, IL seeks to learn from demonstrations collected as state-action trajectories from an expert deployed in the environment. This framework underlies a large body of work in behavioral cloning, dataset aggregation, and inverse RL (ross2011reduction; ziebart2008maxent; ho2016generative; fu2018learning; chi2023diffusion; zhao2023aloha), and is widely used in offline RL and robotics applications (levine2020offline; fu2021d4rl; prudencio2023survey; brohan2023rt2). Traditionally, IL methods assume that the expert and imitator operate with the same sensory capabilities, meaning every variable that the expert can observe is also observable to the imitator. Under this No Unobserved Confounders (NUC) assumption, the expert policy is identifiable from observational data and standard IL can recover it given sufficient demonstrations. However, realistic decision-making systems rarely satisfy NUC. In practice, experts often have access to additional sensors or context unavailable to the imitator, and latent conditions such as wind, friction, and payload may unpredictably change their distribution. These phenomena introduce unobserved confounding to the imitation task that may jointly affect the state transition, expert’s action and rewards. Figure 1: Performance of various imitation learning algorithms on the Confounded AntMaze Large task. (a) One of our proposed algorithms, Causal IQ-Learn. (b) Existing CIL methods such as Causal GAIL struggle to scale to high-dimensional long-horizon tasks, even when equipped with our windowed adjustment (Section 3.2) to bypass their infeasibility (see Appendix F.3). (c) Non-causal methods fail by overfitting to spurious correlations corrupted by unobserved confounding. (d) Success rates for all algorithms tested aggregated across all tasks evaluated; Causal SQIL and Causal IQ-Learn perform best through leveraging causal knowledge and scalable policy learning. By falsely assuming NUC or ignoring unobserved confounding, standard IL methods overfit to spurious correlations during training, thus failing to generalize when these correlations shift at runtime (dehaan2019causal; lu2023imitation). Figure 1 illustrates this on a confounded maze navigation task: the observation contains quantities that correlate with expert actions but, due to influence from unobserved confounders, suffer distribution shift at runtime. Causally unaware methods mistake these spurious correlations for causal signals during training and incorporate them into their decision-making, ultimately failing to exhibit coherent movement during runtime and achieve near-0%0\% success (Figure 1c,d). In safety-critical tasks in which IL is often applied due to its independence from reward engineering, such as autonomous driving (chen2024endtoend; codevilla2019exploring) and robotic manipulation (chi2023diffusion; zheng2024imitation), such failures are only revealed during deployment and thus pose an unaffordable risk which severely limits the utility of IL. Causal imitation learning (CIL) addresses this by leveraging structural causal knowledge to identify which observed variables are safe to condition on (zhang2020causal; kumor2021sequential; ruan2023causal; ruan2024partial). However, existing CIL methods have remained restricted to low-dimensional, short-horizon tasks: in continuous control benchmarks (todorov2012mujoco; park2025ogbench) where observation spaces are high-dimensional and episodes span thousands of steps, they suffer from compounding error and training instability (Figure 1b). On the same large maze task, these methods recover between 13−71%13-71\% of expert performance despite recovering 84−89%84-89\% on the medium maze (Table 1). A review of the CIL and IL literature is provided in Appendix A. Collectively, we identify the gap in the current state of imitation learning literature: scalable methods can imitate long-term expert policies, but are fragile when NUC is violated; causal methods are robust under confounding, but struggle to scale. Our contributions address this gap by introducing scalable CIL through soft Q-learning methods augmented by an efficient approximation to the sequential π-backdoor adjustment sets that leverages structural properties of the environment. For evaluation, we introduce a suite of confounded control tasks based on OGBench (park2025ogbench) where confounding biases are designed to mirror what is naturally found in real-world scenarios. We find that in these environments, existing CIL algorithms struggle to recover a consistent policy and non-causal IL algorithms fail altogether, whereas our proposed algorithms are able to achieve 90%~90\% of the expert’s success rate on average (Figure 1d) and even surpass it on some tasks. Notations. We will consistently use capital letters (X) to represent variables either in the observation or in the causal diagram, and lowercase (x) for their values. We bold capital letters (X) for sets of variables. We denote the parents of the variable X in a causal diagram pa(X)pa(X) and its children ch(X)ch(X) (with Pa(X)Pa(X) and Ch(X)Ch(X) including X) and the C-component of X in G as (X)C(X). For a topological order ≺ and the subgraph (X)G(X) induced by X and its predecessors, the extended parent set pa+(X)=pa((X))∖Xpa^+(X)=pa(C(X)) \X\; pa+(X)pa^+(X) contains every variable collider-connected to X within (X)G(X) together with the parents of those variables. We write ch+(X)ch^+(X) for the effective children of X, or every variable reachable from X by a directed path whose internal nodes are all latent. XtX_t and XHX_H denote the instance of X at timestep t and at the last step, respectively. We use P(X)P(X) as the probability distribution over X, π(X∣)π(X ) as the behavioral policy distribution conditioning on Z, and do(x)do(x) as the intervention fixing X to take values x. 2 The Challenge of Imitation under Unobserved Confounding We model the joint expert–environment system as a structural causal model (SCM). Definition 1 (Structural Causal Model (pearl2009causality; bareinboim2022pearl)). A structural causal model (SCM) is a tuple ℳ=⟨,,ℱ,P()⟩M= ,V, F,P(u) , where: • U is a set of exogenous variables determined by factors outside the model; • =V1,…,VnV=\V_1,…,V_n\ is a set of endogenous variables determined by variables in ∪U ; • ℱ=fV:V∈ F=\f_V:V \ is a set of structural functions such that each V∈V is assigned via V←fV(pa(V),UV)V← f_V(pa(V),U_V), where pa(V)⊆∖Vpa(V) \V\ are endogenous parents of V and UV⊆U_V ; • P()P(u) is a joint probability distribution over the exogenous variables U. Each SCM ℳM induces a causal diagram G with one node for each V∈V , directed edges pa(V)→Vpa(V)~→~V, and bidirected edges between variables that share an unobserved parent in U. In our setting, V includes states, actions, and latent environment variables, ⊆X is the action set, and Y∈Y is the latent reward. To differentiate between variables that are unobserved, observed, and observed by the expert only, we partition the endogenous variables into O⊆(observed to the imitator),L=∖O(latent to the imitator).V^O (observed to the imitator), ^L=V ^O (latent to the imitator). The expert demonstrations reflect the joint observational distribution P(O)P(V^O), whereas the imitator, operating under its own policy, induces the interventional distribution P(∣do(π))P(V do(π)). In the presence of latent variables LV^L, these two distributions may differ substantially: correlations between observed variables and expert actions may be driven by unobserved confounders rather than causal paths. The goal of CIL is therefore to determine, for each time step t, the sufficient subset of observed variables t⊆OZ_t ^O for constructing an unbiased approximation of the expert’s decision mechanism, πt(xt∣t)≈P(xt∣t), _t(x_t _t)≈ P(x_t _t), in a way that is stable to the removal of latent confounding. Figure 2: (a) Generic causal diagram for sequential imitation learning with unobserved confounders, as perceived by the expert. (b) Expanded single-timestep slice of a diagram with a spurious variable WtW_t, a useful state variable ZtZ_t, and latent state tLV^L_t. (c) Imitator perception of diagram (b), demonstrating how latent observation variables become additional unobserved confounders. Example 1 (Confounded AntMaze). Consider an ant robot navigating a maze toward a goal region, receiving a terminal reward Y upon success (see Figure 1 for visualization). The expert observes the full state V, including its torso orientation O, and selects joint torques X accordingly to move itself; Figure 2a shows this sequential structure. The imitator, however, does not observe O (i.e. ∈LO ^L). A compass sensor W serves as a noisy surrogate for the ant’s bearing. The environment is also subject to a latent wind U (shown implicitly through bidirectional edges) that applies an external force to the dynamics, affecting the compass reading and the difficulty of reaching the goal. Figure 2b shows a single-timestep slice of this structure. The compass WtW_t receives incoming edges from both the hidden orientation Ot∈tLO_t ^L_t and the latent wind UtU_t, but has no outgoing edges: it is a collider that plays no causal role in determining future states, actions, or rewards. The useful state variables (position, joint angles, velocities) are represented by ZtZ_t, which does causally influence the expert’s action due to its edge into XtX_t. From the imitator’s perspective (Figure 2c), the hidden tLV^L_t becomes an additional unobserved confounder, introducing a bidirected edge between tOV^O_t and XtX_t. A causally unaware imitator that conditions on all observed variables, including WtW_t, inadvertently opens the spurious path Y←Ut→Wt←Ot→XtY← U_t→ W_t← O_t→ X_t. During training, the compass correlates with the expert’s turning behavior because both are influenced by the wind. The imitator mistakes this for a causal signal: it learns, for instance, that when WtW_t points east the expert turns right, not realizing both facts are driven by an eastward gust. Once deployed under a different P(U)P(U), these associations become actively harmful: the imitator turns into walls whenever the wind changes direction. ■ 2.1 Sequential π-Backdoor Criterion Formally, the sequential π-backdoor criterion graphically determines what each tZ_t must contain so that conditioning on tZ_t blocks all noncausal paths from XtX_t to the final outcome Y. Definition 2 (Sequential π-Backdoor Criterion (kumor2021sequential)). Let G be the causal diagram induced by the SCM. For each action XtX_t, define a manipulated graph t′G_t^ obtained by: (i) removing all incoming edges into future actions t+1:HX_t+1:H, and (i) replacing each future action XjX_j (j>tj>t) by a node whose parents are restricted to jZ_j. A family of sets tt=0H\Z_t\_t=0^H satisfies the sequential π-backdoor for (,,Y)(G,X,Y) if, for every t, either (Xt⟂Y∣t)(t′)XtorXt∉Ant′(Y).(X_t \!\!\! Y _t)_(G_t^ )_X_t X_t _G_t^ (Y). Here (t′)Xt(G_t^ )_X_t denotes the graph obtained from t′G_t^ by deleting outgoing edges from XtX_t. When t\Z_t\ satisfies Definition 2, conditioning on tZ_t removes all confounding and noncausal dependencies between XtX_t and Y that arise from shared latent parents in LV^L, proxy variables, or unobserved factors. Crucially, tZ_t is restricted to the observable set OV^O. Returning to Example 1 and Figure 2, applying the sequential π-backdoor to the diagram yields an adjustment set tZ_t that contains the useful state variables (position, joint angles, velocities) but excludes the compass WtW_t. By conditioning only on tZ_t, the imitator’s policy π(xt∣t)π(x_t _t) is indifferent to the wind-driven distributional shift in WtW_t and instead relies exclusively on variables that causally determine the expert’s actions. Even when the orientation OtO_t is unobserved and the full imitability condition is broken, learning a policy over t⊆OZ_t ^O can still approximate the expert behavior because the remaining observed variables carry sufficient causal signal for navigation. 3 Scalable Causal Imitation Learning The sequential π-backdoor criterion identifies the correct conditioning set, but correct adjustment sets alone are not sufficient for imitation at scale. Prior CIL work has instantiated the criterion within two algorithmic paradigms: Causal Behavioral Cloning (Causal BC), or supervised cloning of the expert’s conditional policy on the adjustment sets π^t(xt∣t)=P(Xt∣t) π_t(x_t _t)=P(X_t _t) (kumor2021sequential), and Causal Generative Adversarial Imitation Learning (Causal GAIL), which matches occupancy measures over the adjustment sets via adversarial training (ruan2023causal). Both exhibit fundamental scaling limitations, which we illustrate on the Confounded AntMaze task from Example 1. Failure of Causal BC in long-horizon tasks. Even when supplied with correct adjustment sets, Causal BC remains a simple supervised learner. In long-horizon tasks, small prediction errors compound and the policy drifts into states not covered by the expert demonstrations, where its predictions are unreliable. While Causal BC achieves a respectable 88.9%88.9\% normalized to the expert in AntMaze-Medium (H≈300H≈ 300 effective steps for successful solves), it drops to 71.2%71.2\% in AntMaze-Large (H≈700H≈ 700 for successful episodes) and to 30.8%30.8\% in HumanoidMaze-Medium (H>1000H>1000 for successful episodes). This degradation is shown in Figure 3, where Causal BC easily navigates a medium maze; on a large maze, however, the agent initially follows the expert path but gradually drifts off course due to compounding error, entering unseen states from which it cannot recover. Figure 3: Causal BC on Confounded AntMaze. On Medium (left), the agent reaches the goal. On Large (right), the agent follows the expert path initially but drifts off course partway through; once outside the demonstration support, it cannot recover and fails to reach the goal. Failure of Causal GAIL in high-dimensional domains. While Causal GAIL addresses compounding error by matching occupancy measures via adversarial training, it introduces its own scaling difficulties through its reliance on on-policy rollouts (typically via PPO) and a discriminator to distinguish expert from imitator trajectories. In long-horizon tasks, the agent must discover the complete path to the goal through exploration before the discriminator can provide a useful learning signal for later portions of the trajectory. In AntMaze-Medium, this exploration is still somewhat feasible and Causal GAIL achieves 84.6%84.6\% success, but performance drops to 13.1%13.1\% in AntMaze-Large and collapses to 0%0\% by HumanoidMaze-Medium. Figure 1b shows that even in AntMaze-Large, the agent learns only coherent navigation for roughly the first two-thirds of the maze. This indicates a credit-assignment failure, as on-policy training collects too few complete traversals to propagate reward signal to later stages of navigation and prevents scaling to long-horizon tasks. These failure modes point to a clear algorithmic requirement: off-policy methods that treat expert demonstrations as a static buffer, learn from self-collected transitions, and propagate reward signal across long horizons via temporal-difference learning. SQIL (reddy2020sqil) and IQ-Learn (garg2021iqlearn), two recent off-policy imitation learning methods built on soft Q-learning, satisfy all three properties but assume unconfounded environments. We now describe how to combine them with the causal adjustment framework. 3.1 Causal SQIL and Causal IQ-Learn Given a π-backdoor admissible scope =⟨Xt,t⟩t=0H−1S=\ X_t,Z_t \_t=0^H-1, a straightforward causally-adjusted state representation would be tz_t that concatenates the values of the variables in tZ_t at each timestep. Both Causal SQIL and Causal IQ-Learn then operate on (t,xt)(z_t,x_t) pairs in place of the standard (st,at)(s_t,a_t) pairs, applying their respective objectives to the adjusted representation. Causal SQIL. SQIL (reddy2020sqil) assigns a fixed reward of r=1r=1 to expert transitions and r=0r=0 to policy transitions, then trains an SAC agent on the combined replay buffer. We apply this to causally-adjusted inputs (t,xt)(z_t,x_t): the critic minimizes the soft Bellman residual y=r+γ(minj=1,2Qθ¯j(′,a′)−αlogπϕ(a′∣′)),ℒQ=[(Qθ(,x)−y)2],y=r+γ ( _j=1,2Q_ θ_j(z ,a )-α _φ(a ) ), _Q=E [ (Q_θ(z,x)-y )^2 ], (1) with a′∼πϕ(⋅∣′)a _φ(· ), and the actor maximizes the entropy-regularized objective ℒπ=[αlogπϕ(x∣)−minj=1,2Qθj(,x)],x∼πϕ(⋅∣).L_π=E_z [α _φ(x )- _j=1,2Q_ _j(z,x) ], x _φ(· ). (2) Because the adjustment set tZ_t satisfies the π-backdoor criterion, the Q-function learns value estimates that are not confounded by latent variables, while temporal-difference learning propagates the expert signal across the full horizon. Full pseudocode is given in Algorithm 2 (Appendix C). Causal IQ-Learn. IQ-Learn (garg2021iqlearn) learns a Q-function whose implicit reward is consistent with expert behavior. The critic enforces the soft Bellman equation on expert data, ℒexpert=(,x,′)∼exp[(Qθ(,x)−γVθ¯(′))2],L_expert=E_(z,x,z ) _exp [ (Q_θ(z,x)-γ\,V_ θ(z ) )^2 ], (3) where V()=logx∼π[expQ(,x)]V(z)= _x π[ Q(z,x)], and applies a policy-consistency regularizer on policy data, ℒreg=(,x)∼π[(logπϕ(x∣)−Qθ(,x)+Vθ())2].L_reg=E_(z,x) π [ ( _φ(x )-Q_θ(z,x)+V_θ(z) )^2 ]. (4) The combined loss ℒQ=ℒexpert+λℒregL_Q=L_expert+λ\,L_reg trains the critic, grounding the implicit reward r(,x)=Q(,x)−γV(′)r(z,x)=Q(z,x)-γ V(z ) in deconfounded state-action associations rather than spurious correlations. The actor uses the same entropy-regularized objective as Causal SQIL. Full pseudocode is given in Algorithm 3 (Appendix C). The causal adjustment layer is algorithm agnostic. The downstream RL algorithm is unmodified; rather, it is the input representation that is changed. This property means that any future IL algorithm built on the soft Q learning approach can be made causal by the same procedure. 3.2 Scalable Causal Adjustment Although the sequential π-backdoor yields a principled solution to the causal imitation problem, applying it swiftly becomes intractable as horizon and dimensionality increase. In a sequential decision-making problem with a terminal reward Y and a horizon H≥1000H≥ 1000, the FindOX algorithm of kumor2021sequential (which returns the maximal admissible set XOV^O_X for sequential π-backdoor adjustment; reproduced in Appendix C) operates over thousands of nodes and results in |t| |Z_t | that linearly increases with t. In an SCM where the full histories of V and X have at least a potential causal effect on XtX_t, adjustment is inefficient and bloats observational dimensionality. To make adjustment feasible, we propose an approximation of the true adjustment sets by generalizing structural properties of confounded control environments. Our assumption is stated in terms of the extended parent set pa+pa^+, which rests on the notion of a confounded component. Definition 3 (Confounded Component (tian2002ccomp)). Given a causal diagram G over variables V, a confounded component (C-component) is a maximal set ⊆C such that every pair of variables in C is joined by a path consisting entirely of bidirected edges. For V∈V , (V)C(V) denotes the C-component containing V. Fixing a topological order and letting (V)G(V) be the subgraph induced by V and its predecessors, recall that the extended parent set is pa+(V)=pa((V))∖Vpa^+(V)=pa(C(V)) \V\, essentially comprising of all variables collider-connected to V and their parents. The C-component generalizes the parent set to the semi-Markovian setting: under unobserved confounding a variable can remain dependent on a non-descendant joined to it by a bidirected edge even after conditioning on its directed parents, which the extended parents set accounts for. Throughout, pa+pa^+ is computed on the diagram’s latent projection ~ G onto O∪YV^O∪\Y\ (verma1990equivalence; tian2002ccomp), in which latent-mediated paths appear as single edges (formal definition in Appendix B). We use this to bound the temporal reach of confounding: Assumption 1 (k-Bounded Time-Homogeneous Confounding). Let HG_H be the causal diagram induced by the SCM unrolled over horizon H, with endogenous variables tV_t at each timestep t, and let ~H G_H be its latent projection onto O∪YV^O∪\Y\, with Y treated as an element of HV_H: (i) k-bounded influence. There exists a constant k≥1k≥ 1 such that for every t, every node in pa+(t)pa^+(V_t) occurs at a timestep no earlier than t−kt-k. (i) Endogenous time-homogeneity. The structural functions ℱ F are identical at every timestep: for all t,t′t,t and for any fixed j≤kj≤ k, the local causal structure among (t−j,…,t,Xt)(V_t-j,…,V_t,X_t) is isomorphic to that among (t′−j,…,t′,Xt′)(V_t -j,…,V_t ,X_t ). Condition (i) ensures that all causal influence on XtX_t, whether from observed state variables or unobserved confounders, is fully captured within a window of length k timesteps. Condition (i) ensures that this local causal structure applies to any timestep in the unrolled graph, such that computing adjustment sets for one k-length window yields sufficient information of the relevant causal relationships to apply for all timesteps. Both conditions of Assumption 1 are naturally satisfied in physics-based continuous control environments such as MuJoCo, where dynamics depend solely on the immediate state and external forces from confounders have temporally localized effects. Algorithm 1 Windowed Sequential π-Backdoor Adjustment 0: Window size k, full horizon H. 1: Construct proxy environment ℰkE_k with horizon h=k+1h=k+1 and extract its causal graph kG_k. 2: Run FindOX(k,,Y)(G_k,X,Y) → XO^X. If ⊈XX ^X, return not imitable. 3: Compute ancestral graph kYG_k^Y. Compute Markov boundary MB←Pa+((Ch+(X)))∖XMB ^+(C(Ch^+(O^X))) ^X and boundary actions BA←Xi∈∩X∣ch+(Xi)⊈XBA←\X_i ^X ^+(X_i) ^X\ in kYG_k^Y. 4: for each action XtX_t in kG_k do 5: if t∈BAX_t then 6: tk←(MB∪BA)∩before(Xt)Z_t^k←(MB ) (X_t) 7: else 8: tk←∅Z_t^k← // Xt∉An(Y)X_t (Y) in t′G_t ; satisfies condition (2) of Def. 2 9: end if 10: end for 11: for t=0,…,H−1t=0,…,H-1 do 12: tH←(v,τ)∈tk∣τ≥t−kZ_t^H←\(v,τ) _t^k τ≥ t-k\ // Clip to window of width k 13: end for 14: Build sliding window specification S: for each observed variable type V appearing in ⋃ttk _tZ_t^k, enumerate lags −1,…,−k\-1,…,-k\ with dimension dVd_V. 15: return tHt=0H−1\Z_t^H\_t=0^H-1, S. Algorithm 1 implements this approximation in two stages. The first stage (Lines 1–10) solves the exact sequential π-backdoor on a short-horizon proxy graph kG_k with horizon k+1k+1. FindOX (kumor2021sequential) identifies the maximal admissible set XO^X; if ⊈XX ^X, the problem is not imitable. The algorithm then constructs per-action adjustment sets from the Markov boundary MBMB of XO^X in the ancestral graph kYG_k^Y (the minimal observed set blocking XO^X from all other variables) and the boundary actions BABA (actions whose causal effect on Y persists regardless of future actions) (kumor2021sequential, Definition 3.1, Lemma 3.2): boundary actions condition on (MB∪BA)∩before(Xt)(MB ) (X_t), while non-boundary actions require no conditioning. Intuitively, in Example 1 and Figure 2, the Markov boundary selects recent instances of Z and X within the k-step window while instances of W, a collider with no outgoing edges, is excluded from XO^X and thus from any adjustment set. The second stage (Lines 11-15) transfers the proxy-graph adjustment sets to the full horizon. By time-homogeneity, the sets tk\Z_t^k\ exhibit a repeating pattern of relative lags; clipping each to a window of k steps reduces dimensionality from O(H)O(H) to O(k)O(k). The output is a fixed-dimensional sliding window S that specifies adjustment set lags and dimensions for each timestep, used to encode inputs for the imitator. Its correctness is guaranteed by the following theorem (Proof in Appendix B). Theorem 1 (Correctness of Windowed Adjustment). Let HG_H be the causal diagram of an SCM unrolled over horizon H, and let Assumption 1 hold with window size k. Let tkt=0k\Z_t^k\_t=0^k be adjustment sets constructed by Algorithm 1 that satisfy the sequential π-backdoor criterion (Definition 2) for the proxy graph kG_k with horizon k+1k+1. Then the transferred sets tHt=0H−1\Z_t^H\_t=0^H-1 returned by Algorithm 1 satisfy the sequential π-backdoor criterion for (H,,Y)(G_H,X,Y). With windowed adjustment sets tHt=0H−1\Z_t^H\_t=0^H-1 and specification S in hand, the causal encoding tz_t is constructed at each timestep: for each observed variable type V appearing in the adjustment sets, we concatenate its values at lags −1,…,−k\-1,…,-k\ relative to t. This representation comprises the state observation in both Causal SQIL and Causal IQ-Learn, making the full procedure from causal graph analysis to policy optimization accessible and feasible for arbitrarily long horizons. 4 Experiments We evaluate the proposed algorithms on confounded continuous-control locomotion environments derived from OGBench (park2025ogbench). Each environment is defined as an SCM to support the modeling of unobserved confounders. In Confounded AntMaze (H=1000H=1000), an 8-DoF ant navigates a maze under latent wind; the imitator observes a wind-affected compass W in place of orientation O (see Example 1). In Confounded HumanoidMaze (H=2000H=2000), a 21-DoF humanoid navigates under latent seismic tremors; the imitator observes a tremor-affected vibration sensor W in place of the hidden center-of-mass velocity C. In both environments, causal methods exclude W from the adjustment set, while causally unaware methods condition on it and thereby fail under distributional shift. Full environment details, causal graphs, and visualizations are provided in Appendix D. We compare eight algorithms total, four causal (Causal BC (kumor2021sequential), Causal GAIL (ruan2023causal), Causal SQIL (ours), Causal IQ-Learn (ours)) and four causally unaware (BC (ross2011reduction), GAIL (ho2016generative), SQIL (reddy2020sqil), IQ-Learn (garg2021iqlearn)), to isolate the contributions of causal adjustment and algorithmic choice. Expert policies are constructed via offline-to-online RL (BC and TD3 fine-tuning); full implementation details are in Appendix E. 4.1 Results Causal variants use Algorithm 1 to compute their per-timestep observation using windowed causal adjustment. Non-causal variants condition on the full OV^O at each timestep. All hyperparameters remain the same between causal and non-causal variants of the same algorithm. Table 1: Evaluation results on confounded tasks. Normalized [Y]E[Y] linearly shifts the worst-performing algorithm to 0 due to the purely negative reward. Best non-expert result per task in bold. Standard errors are provided; unnormalized data can be found in Appendix F.1. Expert C-BC C-GAIL C-SQIL C-IQ-Learn BC GAIL SQIL IQ-Learn AntMaze-Medium Norm. [Y]E[Y] 271.4±87.5271.4 ± 87.5 250.1±100.9250.1 ± 100.9 239.5±113.6239.5 ± 113.6 275.8±85.7275.8 ± 85.7 257.1±101.3257.1 ± 101.3 0.0±119.20.0 ± 119.2 10.4±112.210.4 ± 112.2 4.7±119.14.7 ± 119.1 2.8±120.82.8 ± 120.8 Success rate (%) 87.6±1.087.6 ± 1.0 77.9±1.377.9 ± 1.3 74.1±1.474.1 ± 1.4 90.7±0.990.7 ± 0.9 84.3±1.284.3 ± 1.2 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 AntMaze-Large Norm. [Y]E[Y] 229.0±122.8229.0 ± 122.8 199.3±121.6199.3 ± 121.6 155.6±122.6155.6 ± 122.6 204.3±125.9204.3 ± 125.9 225.6±127.9225.6 ± 127.9 14.0±150.214.0 ± 150.2 85.8±132.985.8 ± 132.9 0.0±151.50.0 ± 151.5 16.1±153.516.1 ± 153.5 Success rate (%) 55.9±1.655.9 ± 1.6 39.8±1.539.8 ± 1.5 7.3±0.87.3 ± 0.8 45.0±1.645.0 ± 1.6 58.9±1.658.9 ± 1.6 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 HumanoidMaze-Medium Norm. [Y]E[Y] 224.8±218.4224.8 ± 218.4 92.0±172.792.0 ± 172.7 0.5±161.90.5 ± 161.9 192.2±217.2192.2 ± 217.2 158.9±203.8158.9 ± 203.8 69.4±169.169.4 ± 169.1 0.0±162.50.0 ± 162.5 41.5±157.041.5 ± 157.0 50.1±169.750.1 ± 169.7 Success rate (%) 33.8±1.533.8 ± 1.5 10.4±1.010.4 ± 1.0 0.0±0.00.0 ± 0.0 24.7±1.424.7 ± 1.4 19.1±1.219.1 ± 1.2 5.4±0.75.4 ± 0.7 0.0±0.00.0 ± 0.0 0.1±0.10.1 ± 0.1 2.4±0.52.4 ± 0.5 HumanoidMaze-Large Norm. [Y]E[Y] 136.4±212.9136.4 ± 212.9 125.1±227.9125.1 ± 227.9 0.8±210.20.8 ± 210.2 139.3±226.0139.3 ± 226.0 90.0±220.590.0 ± 220.5 93.0±204.693.0 ± 204.6 0.0±208.90.0 ± 208.9 80.7±194.980.7 ± 194.9 70.8±193.370.8 ± 193.3 Success rate (%) 7.0±0.87.0 ± 0.8 8.0±2.78.0 ± 2.7 0.0±0.00.0 ± 0.0 8.0±2.78.0 ± 2.7 3.0±1.73.0 ± 1.7 5.0±2.25.0 ± 2.2 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 0.0±0.00.0 ± 0.0 All non-causal methods fail catastrophically. As seen in Table 1, non-causal algorithms consistently fail across all environments: standard BC, GAIL, SQIL, and IQ-Learn achieve 0%0\% or near-0%0\% success rates with low [Y]E[Y] across all tasks. This confirms that the fundamental failure is not a consequence of any particular learning paradigm but of the decision to condition on all observed variables, and no amount of temporal-difference learning or adversarial training can overcome a fundamentally misspecified conditioning set. The improvement of non-causal algorithms in HumanoidMaze tasks from AntMaze tasks, despite the former being more difficult, can be attributed to the less disruptive confounding effect of seismic tremors than wind fields on the state dynamics. Causal adjustment is necessary but not sufficient for scaling. Although causal adjustment alone leads to significant improvements in most algorithms (Table 1), Causal BC and especially Causal GAIL see substantial performance decreases as the task horizon and dimensionality increases, with Causal GAIL collapsing to about the level of non-causal GAIL by the HumanoidMaze-Medium task. Causal SQIL and Causal IQ-Learn scale more gracefully despite increasing dimensionality and horizon, and at times surpassing the expert’s performance. Figure 4: Evaluation return during training and runtime for GAIL, SQIL, and IQ-Learn (causal and non-causal variants). During training, both variants of each algorithm achieve comparable returns. The gap emerges only at runtime (Table 1) due to unobserved confounders causing distributional changes from where the expert demonstrations were collected, which affects only non-causal methods. Hyperparameters and other training metrics are provided in detail in Appendix E. Confounding cannot be revealed by in-distribution evaluation. Figure 4 reveals that during training, when the imitator operates under the same confounder distribution P()P(U) as the expert, non-causal variants of each algorithm achieve nearly identical evaluation returns to causal variants, and in some cases appear better since conditioning on the spurious proxy W provides additional predictive signal that is useful when under the training P()P(U). At runtime, when P()P(U) shifts, every non-causal method collapses to near-0%0\% success while the causal methods retain significantly more of the performance. Thus, confounding cannot be diagnosed during training. Figure 5: Sensitivity of causal algorithms to k in Algorithm 1 on Confounded AntMaze-Large. Windowed approximation is necessary. Figure 5 demonstrates that a moderate k∈[1,10]k∈[1,10] is ideal for imitation, whereas k=0k=0 (pure Markov) and k=100k=100 see significant drops in most methods. This implies that while the environments require sequential decision-making capabilities, the relevant causal effects on any XtX_t can be captured within a few timesteps; meanwhile, large k bloats the representation. Interestingly, at k=100k=100, Q-learning methods collapse due to bootstrapping instability in high-dimensional state spaces while BC remains robust due to its supervised learning approach. The causality gap dominates the algorithmic gap. In every task, the causal variant of each algorithm outperforms its non-causal counterpart, with the exception of Causal GAIL and non-causal GAIL both achieving 0%0\% success rate in HumanoidMaze tasks (Table 1). Thus, causal reasoning remains the primary determinant of success in confounded environments. The algorithm choice becomes the secondary yet still substantial factor that determines how well a causal method performs. 5 Conclusion We introduce Causal SQIL and Causal IQ-Learn, off-policy CIL algorithms that combine sequential π-backdoor adjustment with soft Q-learning objectives, and windowed causal adjustment that is tractable for long-horizon control. Our experiments demonstrate that causal adjustment is necessary for robust policy learning and that off-policy Q-learning is necessary for scaling. While our approach relies on knowledge of the causal diagram (see Appendix G for a full discussion on limitations), it is algorithm-agnostic. Thus, looking forward, it can be composed with expressive policy classes (e.g. diffusion policies (chi2023diffusion) or action-chunking transformers (zhao2023aloha)), high-dimensional sensory inputs where confounding manifests through pixel-level corruptions (li2025confounding; li2026causal; juliani2026confounding), and any setting in which the expert and imitator operate under different sensor configurations, such as tele-operation, sim-to-real transfer, and multi-agent imitation. As unobserved confounding is the norm in real-world deployment, integrating causal reasoning with scalable policy learning is essential for trustworthy imitation in safety-critical domains. Acknowledgements This research is supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation. References Supplementary Materials The following content was not necessarily subject to peer review. Appendix A Related Work Imitation learning. Imitation learning (IL) trains policies from expert demonstrations without access to a reward signal. Behavioral cloning (BC) reduces IL to supervised learning (pomerleau1989alvinn), but suffers from compounding errors due to covariate shift (ross2011reduction). Interactive methods such as DAgger (ross2011reduction) mitigate this by querying the expert on-policy, though at the cost of requiring an interactive demonstrator. Inverse reinforcement learning (IRL) recovers a reward function rationalizing expert behavior (abbeel2004apprenticeship; ziebart2008maxent); adversarial formulations such as GAIL (ho2016generative) and AIRL (fu2018learning) cast IRL as occupancy-measure matching, avoiding explicit reward modeling but relying on on-policy rollouts and adversarial training that scale poorly to long horizons. More recently, off-policy value-based methods have achieved strong results on continuous-control benchmarks: SQIL (reddy2020sqil) reformulates IL as soft Q-learning with binary rewards, and IQ-Learn (garg2021iqlearn) learns a Q-function whose implicit reward is consistent with expert data. Both build on the SAC framework (haarnoja2018sac) and propagate the expert signal across trajectories via temporal-difference learning, making them substantially more effective than BC or GAIL on long-horizon tasks. Modern IL has also scaled to real-world robotics through expressive policy classes such as diffusion policies (chi2023diffusion) and action-chunking transformers (zhao2023aloha). All of these methods, however, assume that the expert and imitator share the same observation space, i.e., No Unobserved Confounders (NUC). The present work retains the scalability advantages of off-policy soft Q-learning while relaxing NUC via causal adjustment. Formal objectives of prior CIL and IL methods. For reference, we provide the formal objectives of the methods discussed in the main text. Causal BC (kumor2021sequential) directly clones the expert’s conditional policy over the admissible adjustment variables, π^t(xt∣t)=P(Xt∣t) π_t(x_t _t)=P(X_t _t), via supervised learning on expert demonstrations. Causal GAIL (ruan2023causal) extends the framework to inverse reinforcement learning by matching expert and imitator occupancy measures over the adjusted variables: minπ∼maxD[logD(t,xt)]+π[log(1−D(t,xt))]. _π _D\;E\! [ D(z_t,x_t) ]+E_π\! [ (1-D(z_t,x_t)) ]. (5) In non-causal settings, SQIL (reddy2020sqil) assigns a fixed reward of 11 to expert transitions and 0 to policy transitions, and then applies SAC on the combined replay buffer. The SAC critic minimizes the soft Bellman residual y=r+γ(minj=1,2Qθ¯j(s′,a′)−αlogπϕ(a′∣s′)),ℒQ=[(Qθ(s,a)−y)2],y=r+γ ( _j=1,2Q_ θ_j(s ,a )-α _φ(a s ) ), _Q=E [ (Q_θ(s,a)-y )^2 ], (6) with a′∼πϕ(⋅∣s′)a _φ(· s ). The actor maximizes the entropy-regularized objective ℒπ=s[αlogπϕ(a∣s)−minj=1,2Qθj(s,a)],a∼πϕ(⋅∣s).L_π=E_s [α _φ(a s)- _j=1,2Q_ _j(s,a) ], a _φ(· s). (7) IQ-Learn (garg2021iqlearn) learns a Q-function whose induced reward is consistent with expert behavior. The critic enforces the soft Bellman equation on expert data, ℒexpert=(s,a,s′)∼exp[(Qθ(s,a)−γVθ¯(s′))2],L_expert=E_(s,a,s ) _exp [ (Q_θ(s,a)-γ\,V_ θ(s ) )^2 ], (8) where V(s)=loga∼π[expQ(s,a)]V(s)= _a π[ Q(s,a)], and applies a policy-consistency regularizer on policy data, ℒreg=(s,a)∼π[(logπϕ(a∣s)−Qθ(s,a)+Vθ(s))2].L_reg=E_(s,a) π [ ( _φ(a s)-Q_θ(s,a)+V_θ(s) )^2 ]. (9) The combined loss ℒQ=ℒexpert+λℒregL_Q=L_expert+λ\,L_reg trains the critic, while the actor uses the same objective as above. Despite their scalability advantages, these methods assume fully observed (i.e., unconfounded) environments; using them in the presence of latent confounders leads to biased policies. Causal imitation learning. Causal imitation learning (CIL) leverages structural causal knowledge to approximate the expert policy in the presence of unobserved confounding. zhang2020causal introduces the π-backdoor criterion, a complete graphical condition for determining policy imitability from observational data when the expert and imitator have different sensory inputs. kumor2021sequential extend this to sequential decision-making with the sequential π-backdoor criterion and the FindOX algorithm for constructing per-timestep adjustment sets. ruan2023causal develop CIL via inverse reinforcement learning, and ruan2024partial introduce a partial-identification approach that can enable the imitator to surpass expert performance. In practice, these methods have been instantiated as Causal BC and Causal GAIL, both of which assume access to a π-backdoor admissible scope. Despite their theoretical appeal, existing CIL algorithms have remained restricted to low-dimensional, short-horizon domains: Causal BC inherits the covariate-shift limitations of behavioral cloning, Causal GAIL’s reliance on on-policy rollouts and adversarial training leads to sample and stability challenges at scale, and the adjustment sets produced by FindOX grow linearly with the horizon, making exact computation infeasible in long-horizon settings. Our work addresses all three limitations by combining the causal adjustment framework with off-policy soft Q-learning objectives and introducing a windowed approximation that reduces the adjustment set computation to a fixed-size sliding window. Confounding, causal confusion, and spurious correlations in IL. A related line of work studies the problem of spurious correlations in IL without assuming access to a causal graph. dehaan2019causal identify “causal confusion,” showing that conditioning on non-causal features can degrade imitation performance, and propose targeted interventions to select the correct causal model. ortega2021delusions formalize “delusions” in sequence models, demonstrating that treating past actions as observations rather than interventions leads to incorrect inference. Variations of this problem have been studied under different names: as feedback-driven covariate shift (spencer2021feedback), as the copycat problem in BC from observation histories (wen2020copycat), as confounding in driving settings (codevilla2019exploring), and as deconfounding via initial-state interventions (pfrommer2023initial). These works diagnose the problem and propose heuristic or environment-specific solutions. In contrast, the CIL framework this paper builds on provides a complete, non-parametric graphical criterion for determining imitability and identifying the correct adjustment set. Several works address confounded IL through mechanisms other than graphical backdoor adjustment. swamy2022causal apply instrumental variable (IV) regression to handle temporally correlated noise in expert actions, proposing DoubIL and ResiduIL. swamy2022sequence prove that on-policy sequence models can asymptotically recover expert behavior when the expert observes privileged information, while showing that off-policy methods “latch” onto confounded features—a failure mode that directly motivates the causal adjustment layer in our off-policy algorithms. vuorio2024deconfounded train a variational inference model to infer the expert’s latent information and learn a latent-conditional policy. shao2025dmlil propose DML-IL, a unifying framework that reformulates causal IL as a conditional moment restriction problem solved via IV regression, handling both expert-observable and expert-unobservable confounders. zeng2025confounded similarly leverage IVs for confounded sequential IL. These IV and latent-inference approaches typically assume additive or temporally bounded confounding for instrument validity, or require ergodic dynamics for latent inference; they do not require a known causal graph but trade this for stronger distributional assumptions. Our approach instead assumes a known causal graph and applies non-parametric graphical adjustment, which is complementary: it provides exact deconfounding guarantees when the graph is available, and our windowed approximation makes it tractable at scale. Robust imitation learning without causal structure. A complementary line of work improves the robustness of imitation policies to distribution shift without positing a causal model, and it is natural to ask whether such methods could substitute for causal adjustment. These methods are best organized by the shift they target. A first family targets the covariate shift induced by the imitator’s compounding errors: DAgger (ross2011reduction) queries the expert on learner-visited states, DART (laskey2017dart) injects noise into the expert’s controls during collection so that demonstrations cover the learner’s error distribution, and chang2021mitigating handle the offline case by exploiting a supplementary dataset with partial coverage through model-based pessimism. A second family targets covariate shift already present in the demonstrations: distribution-matching methods (kim2022demodice) align the imitator’s stationary distribution with that of the data, while distributionally robust formulations optimize the worst-case imitation loss over an uncertainty set of state distributions around the data: a total-variation ball in DR-BC (panaganti2023distributionally), and, in DrilDICE (seo2024drildice), an f-divergence ball intersected with the set of stationary distributions satisfying the Bellman flow constraint, which handles demonstrations collected from arbitrary non-stationary state distributions. Related robust formulations address noisy expert actions (bashiri2021distributionally), suboptimal or corrupted demonstrations (sasaki2021behavioral; wu2019imitation; xu2022discriminator), and variations in the environment dynamics (chae2022robust); tennenholtz2021covariate intertwine this literature with ours by studying covariate shift of latent confounders themselves. The unifying structure of the covariate-shift family is that each method changes where the supervised imitation loss is evaluated, whether by reweighting states, enlarging coverage, or optimizing a worst case over state distributions, while fixing the observational conditional of the expert’s action given the imitator’s observations. This is explicit in seo2024drildice, whose problem statement assumes that the training and target distributions differ only in their state marginals while sharing the same state-conditional expert action everywhere. Orthogonality of confounding and covariate-shift robustness. Distribution shift in sequential decision-making arises from distinct causes (see bareinboim2016fusion for a broader treatment of the biases underlying generalization): (1) non-overlap, where the imitator visits state-action regions unsupported by the data; (2) confounding bias, where unobserved variables jointly influence observations, actions, and outcomes, so that the observational conditional differs from the interventionally correct policy; and (3) structural change between the training and deployment environments. Covariate-shift-robust IL addresses cause (1) (and dynamics-robust variants a form of (3)) under the assumption that the per-observation conditional is invariant. However, cause (2) violates that invariance. When unobserved confounders shift at deployment, the conditional P(x∣O)P(x ^O) that was correct on the demonstrations becomes the wrong policy, and no reweighting of where that conditional is fit can repair it, because the bias lives inside the conditional rather than in the marginal it is averaged over. This mirrors the situation in offline RL, where conservatism toward out-of-distribution actions cannot correct value estimates biased by confounding (li2025confounding). This is seen in the following instance, in which the covariate-shift machinery is provably inert. Example 2 (Confounded Route Choice). Consider a single-stage SCM with binary variables. A goal lies on side Z∼Bern(1/2)Z (1/2) of a junction, and the binary actuation X selects a route A=X⊕ZA=X Z relative to the goal side: route a=0a=0 is exposed and succeeds only in calm conditions, P(Y=1∣a=0,u)=1−uP(Y=1 a=0,u)=1-u, while route a=1a=1 is sheltered and succeeds with probability 3/43/4 regardless, P(Y=1∣a=1,u)=3/4P(Y=1 a=1,u)=3/4. The wind U∼Bern(p)U (p) is exogenous and unobserved by all agents, with ptr=1/4p_tr=1/4 during demonstration collection. The expert observes its orientation O∼Bern(2/3)O (2/3) in addition to Z; the imitator does not observe O but instead carries a compass W=O⊕UW=O U whose needle is deflected by the wind, so that O→W←UO→ W← U is a collider with no outgoing edges, exactly as in Figure 2. Under ptr=1/4p_tr=1/4 both routes have expected reward 3/43/4 for every (z,o)(z,o), so the expert is optimal over its information set and, being indifferent, takes the route it is facing: X=Z⊕OX=Z O. The induced diagram has edges Z→XZ→ X, Z→YZ→ Y, X→YX→ Y, O→XO→ X, O→WO→ W, U→WU→ W, U→YU→ Y; its latent projection onto Z,W,X,Y\Z,W,X,Y\ has directed edges Z→XZ→ X, Z→YZ→ Y, X→YX→ Y and bidirected edges X↔WX W, W↔YW Y. Applying Definition 2, the set Z\Z\ is π-backdoor admissible; Z,W\Z,W\ is not, since conditioning on the collider W opens X↔W↔YX W Y; and ∅ is not, since X←Z→YX← Z→ Y remains open. The demonstrations (z,w,x)∼Ptr(z,w,x) P_tr have full support (the smallest cell has probability 1/241/24) and are drawn i.i.d. from the expert’s own occupancy. There is thus no non-overlap, no compounding error as the horizon is one, and no mismatch between the dataset and the expert’s state distribution as the shift targeted by seo2024drildice is intentionally absent. Nevertheless, the compass is genuinely informative in-distribution, Ptr(X=z∣z,W=0)=(1/3)(3/4)(1/3)(3/4)+(2/3)(1/4)=35, P_tr(X=z z,W=0)= (1/3)(3/4)(1/3)(3/4)+(2/3)(1/4)= 35, Ptr(X=1−z∣z,W=1)=(2/3)(3/4)(2/3)(3/4)+(1/3)(1/4)=67, P_tr(X=1-z z,W=1)= (2/3)(3/4)(2/3)(3/4)+(1/3)(1/4)= 67, so the loss-minimizing deterministic imitator over (Z,W)(Z,W) follows the compass, π^n(z,w)=z⊕w π_n(z,w)=z w (i.e., a=wa=w), while the adjusted conditional over Z\Z\ is πc(x∣z)=Ptr(x∣z) _c(x z)=P_tr(x z), which takes the sheltered route with probability P(O=1)=2/3P(O=1)=2/3, and its deterministic counterpart π^c π_c always takes it. At deployment the wind picks up, pte=3/4p_te=3/4. The compass-following imitator now takes the exposed route precisely when the wind is blowing (P(a=0∣U=1)=P(W=0∣U=1)=2/3P(a=0 U=1)=P(W=0 U=1)=2/3) and collects zero reward there, whereas the adjusted policies are indifferent to the compass and unaffected by its miscalibration. Table 2 reports the exact interventional values. Two features mirror our main experiments: in-distribution, the confounded policies trail the expert by margins comparable to evaluation noise (0.7210.721 and 0.6880.688 against 0.7500.750), so the bias is nearly invisible during training, as in Figure 4; and the deterministic adjusted policy surpasses the fixed expert at deployment, as causal methods occasionally do in Table 1 (cf. ruan2024partial). By contrast, the stochastic adjusted policy πc _c attains the expert’s interventional value under every deployment wind distribution: the expert’s action satisfies X=Z⊕OX=Z O with O⟂(Z,U)O (Z,U), so the expert and do(πc)do( _c) induce the same joint over (Z,X)(Z,X) with X⟂U∣ZX U Z in both, and since the parents of Y are X,Z,U\X,Z,U\, the two interventional distributions of Y coincide for every P(U)P(U). ■ Table 2: Interventional values [Y∣do(π)]E[Y do(π)] in Example 2 under the training wind (ptr=1/4p_tr=1/4) and the deployment wind (pte=3/4p_te=3/4). “Robust-BC output” is the deterministic policy returned by BC and by every covariate-shift-robust method in the class of Proposition 1. All values are exact; the general closed forms in s=P(U=1)s=P(U=1) are shown for reference. Every entry is reproduced by exhaustive enumeration. Policy Conditions on [Y∣do(π)]E[Y do(π)], any s Train (s=14s= 14) Deploy (s=34s= 34) Expert (X=Z⊕OX=Z O) Z,O\Z,O\ 12+1−s3 12+ 1-s3 3/4=0.7503/4=0.750 7/12≈0.5837/12≈ 0.583 πc(x∣z)=Ptr(x∣z) _c(x z)=P_tr(x z) Z\Z\ 12+1−s3 12+ 1-s3 3/4=0.7503/4=0.750 7/12≈0.5837/12≈ 0.583 π^c π_c (deterministic, sheltered) Z\Z\ 34 34 3/4=0.7503/4=0.750 3/4=0.7503/4=0.750 πn(x∣z,w)=Ptr(x∣z,w) _n(x z,w)=P_tr(x z,w) Z,W\Z,W\ (1−s)173210+s87210(1-s) 173210+s 87210 101/140≈0.721101/140≈ 0.721 31/60≈0.51731/60≈ 0.517 π^n π_n (Robust-BC output: a=wa=w) Z,W\Z,W\ (1−s)56+s14(1-s) 56+s 14 11/16≈0.68811/16≈ 0.688 /≈0.39619/48≈ 0.396 Proposition 1 (State reweighting cannot remove confounding bias). Consider the demonstration distribution PtrP_tr of Example 2; any single-stage instance with full support over the imitator’s observations suffices. For a deterministic policy π over observations (z,w)(z,w), let Cπ(z,w):=x∼Ptr(⋅∣z,w)[ℓ(π(z,w),x)]C_π(z,w):=E_x P_tr(· z,w)[ (π(z,w),x)] denote the expected loss against the sampled expert actions, with ℓ the 0–1 loss. Then π^n(z,w)=argmaxxPtr(x∣z,w) π_n(z,w)= _xP_tr(x z,w) minimizes (z,w)∼d[Cπ(z,w)]E_(z,w) d[C_π(z,w)] simultaneously for every distribution d over observations. Consequently, π^n π_n attains the minimum of every objective formed by nonnegatively aggregating the per-observation losses Cπ(z,w)\C_π(z,w)\: including any supremum or infimum over a family of state weightings, with or without an f-divergence regularizer, minπsupd∈d[Cπ]−αDf(d∥dD), _π\; _d \;E_d\! [C_π ]-α D_f(d\,\|\,d_D), for every uncertainty set Q, every convex generator f, and every α≥0α≥ 0. This class contains behavioral cloning, adversarially weighted BC, DR-BC (panaganti2023distributionally), OptiDICE-BC (lee2021optidice), and DrilDICE (seo2024drildice), which therefore all return the compass-following policy of Example 2 and its deployment value of 19/4819/48. This also holds for the stochastic conditional πn(x∣z,w)=Ptr(x∣z,w) _n(x z,w)=P_tr(x z,w) under the log loss. Proof. Each objective decomposes over observations, d[Cπ]=∑z,wd(z,w)Cπ(z,w)E_d[C_π]= _z,wd(z,w)\,C_π(z,w), and by full support Cπ(z,w)C_π(z,w) is well defined at every (z,w)(z,w) and is minimized pointwise by π^n π_n (respectively, by πn _n under the log loss). A simultaneous pointwise minimizer minimizes every nonnegatively weighted aggregate of the per-observation losses, hence every supremum or infimum of such aggregates over families of weightings, and subtracting a term αDf(d∥dD)α D_f(d\|d_D) that does not depend on π changes neither the inner optimizer’s feasible directions in π nor the outer argmin . For DrilDICE specifically, each policy update minimizes (s,a)∼dD[w∗(s,a)Cπ(s)]E_(s,a) d_D[w^*(s,a)\,C_π(s)] with fixed nonnegative weights; since the cost is constant in a given s, this equals a state-weighted BC loss with weights ω(s)=dD[w∗(s,a)∣s]≥0ω(s)=E_d_D[w^*(s,a) s]≥ 0, and in the single-stage problem the weights are moreover directly state-measurable, as eπ,ν(s,a,s′)=Cπ(s)−ν(s)e_π,ν(s,a,s )=C_π(s)-ν(s) with the terminal value fixed to zero. Hence π^n π_n is returned by the two-step instantiation of DrilDICE and is a fixed point of its alternating scheme. When the optimal weighting places positive mass on every observation, π^n π_n is the unique minimizer, since the argmax in Example 2 is strict at every (z,w)(z,w). ∎ The orthogonality runs in both directions, and neither correction subsumes the other. Causal adjustment does nothing for cause (1): Causal BC in Figure 3 drifts outside the demonstration support on AntMaze-Large despite conditioning on admissible adjustment sets, purely a non-overlap failure. Symmetrically, the covariate-shift constructions of seo2024drildice contain no unobserved confounding, so a causal method handed the unconfounded diagram would reduce to plain BC and inherit its failure there. Note that robustifying against the deployment-time shift in the observable marginals would not rescue the confounded imitator in Example 2: the compass marginal does shift (P(W=1)P(W=1) moves from 7/127/12 to 5/125/12), but the failure is driven by the shifted conditional, which every method in the class intentionally leaves unmodified. Because the two corrections act on different components of the learning problem, either the conditioning set or the distribution over which the loss is evaluated, they compose rather than compete: Causal SQIL and Causal IQ-Learn are themselves such a composition, pairing adjustment for cause (2) with off-policy temporal-difference learning against the compounding-error consequences of cause (1), and composing windowed causal adjustment with DICE-style stationary distribution corrections is a natural direction for future work. Causal reinforcement learning. Causal reasoning has been increasingly integrated into RL to handle confounded offline data and distribution shift. li2025confounding develop confounding-robust deep RL via causal adjustment in the presence of unobserved confounders; li2026causal extend this to offline settings via causal flow Q-learning; and juliani2026confounding address confounders in continuous control through automatic reward shaping. These works operate in the RL setting where a reward signal is available, whereas the present paper addresses the harder IL setting where the reward is entirely latent. Nevertheless, the structural causal formalization and the challenge of off-policy learning under confounders are shared, and our causal adjustment layer is compatible with advances in causal RL. Learned history compression. Since Algorithm 1 ultimately encodes a fixed window of recent observations, a natural question is whether a learned sequence model, such as an RNN or Transformer over t−k:ts_t-k:t, could replace graphical adjustment altogether. The two mechanisms are not interchangeable: Algorithm 1 determines both which variables may be conditioned on and which lags are required, whereas a learned compressor addresses only the latter. A sequence model over the raw history still conditions on every observed variable, including the collider W, and its training objective provides no signal to exclude it: the spurious association is predictive in-distribution (Figure 4), and history-conditioned imitators are prone to latching onto past actions and confounded features (wen2020copycat; swamy2022sequence), so such a model would inherit the biased conditional that defeats the non-causal baselines in Table 1. Furthermore, Theorem 1 and the window ablation (Figure 5) indicate that the sufficient window is small relative to the horizon in our environments, limiting the benefit of expressive history models over the direct encoding of Algorithm 1. Learned compression instead becomes appropriate when Assumption 1 fails and long-range latent dependencies require summarizing history beyond any fixed window (Appendix G); composing the variable selection of Algorithm 1 with learned temporal compression of the selected variables is a noteworthy direction for future work. Appendix B Proof Theorem 2 (Restatement of Theorem 1). Let HG_H be the causal diagram of an SCM unrolled over horizon H, and let Assumption 1 hold with window size k. Let tkt=0k\Z_t^k\_t=0^k be adjustment sets constructed by Algorithm 1 that satisfy the sequential π-backdoor criterion (Definition 2) for the proxy graph kG_k with horizon k+1k+1. Then the transferred sets tHt=0H−1\Z_t^H\_t=0^H-1 returned by Algorithm 1 satisfy the sequential π-backdoor criterion for (H,,Y)(G_H,X,Y). Proof. We show that for every t∈0,…,H−1t∈\0,…,H-1\, one of the two conditions of Definition 2 holds for tHZ_t^H in the full-horizon manipulated graph (H)t′(G_H)_t . We handle t≥kt≥ k first, then t<kt<k. Setup and notation. Throughout the proof we work in the latent projections ~H G_H and ~k G_k onto O∪YV^O∪\Y\ (respectively YkY_k), on which Assumption 1(i) is stated. Since latent projection preserves d-separation among the retained variables and directed (ancestral) relationships between them (verma1990equivalence; tian2002ccomp), both conditions of Definition 2 are invariant under projection, and it suffices to establish them in the projected graphs; the manipulated-graph operations (deleting incoming edges of future actions and re-parenting them on the observed sets jHZ_j^H) involve only retained variables and can equivalently be performed on the projections. To avoid clutter, we write HG_H and kG_k for the projections in the remainder of the proof. In particular, every node of these graphs is observed (or is the reward), pa+pa^+ always denotes the extended parent set in the projection, and a single edge may represent a latent-mediated path of the original SCM, so edges span up to k timesteps rather than one. Let (H)t′(G_H)_t denote the manipulated graph at timestep t: all incoming edges into future actions XjX_j (j>tj>t) are removed and each XjX_j is given parents jHZ_j^H. Let ((H)t′)Xt((G_H)_t )_X_t further delete all outgoing edges from XtX_t. Define Wint:=t−k,…,tWin_t:=\t-k,…,t\. We use VrV_r (non-bold) for an individual variable at timestep r. Step 1: Structure of the manipulated graph. By k-bounded influence (Assumption 1(i)), for every t the extended parent set pa+(t)pa^+(V_t) lies within the window Wint=t−k,…,tWin_t=\t-k,…,t\: all directed parents of tV_t, all variables collider-connected to tV_t, and the parents of those variables, occur at timesteps ≥t−k≥ t-k. In (H)t′(G_H)_t , the original incoming edges to future actions are replaced by edges from jHZ_j^H, which by Algorithm 1 contain only variables within k steps of j; this replacement only deletes edges and redirects action parents to in-window variables; since deleting a bidirected edge can only shrink a C-component (Definition 3) and redirecting adds only in-window parents, no extended parent set grows beyond its width-k bound. Therefore the bound on pa+pa^+ is preserved in (H)t′(G_H)_t . Note also that the bound implies every edge of the (projected, manipulated) graph spans at most k timesteps: for a directed edge B→AB→ A, B∈pa(A)⊆pa+(slice(A))B (A) ^+(V_slice(A)), and for a bidirected edge B↔AB A with B earlier, B is collider-connected to A and hence B∈pa+(slice(A))B ^+(V_slice(A)). Crucially, the window is sealed against open escapes into the past. Suppose a path segment lying behind XtX_t (i.e., before any crossing into the tail defined below) leaves WintWin_t downward, and let A∈WintA _t be a node at which it exits or re-enters, with neighbor B on the segment at a timestep <t−k<t-k. Since directed edges respect the temporal order and every edge spans at most k lags, the boundary edge must be B→AB→ A or B↔AB A; either way it places an arrowhead at A. If the other path-edge at A also carries an arrowhead, A is a collider, and the segment is blocked at A unless A∈An(tH)A (Z_t^H); if it carries a tail, A is a non-collider, and the segment is blocked at A unless A∉tHA _t^H. It is here that the proof uses the specific construction of Lines 1–10 of Algorithm 1 rather than mere proxy admissibility. By the Markov-boundary property of the construction (kumor2021sequential, Definition 3.1 and Lemma 3.2), transferred to WintWin_t by the isomorphism of Step 2, every observed window variable that receives an incoming arrowhead from outside the conditioned corridor and transmits along a tail edge either belongs to tHZ_t^H (and hence blocks the segment as a conditioned non-collider) or is d-separated from XtX_t given tHZ_t^H within the window (so that the segment is blocked before reaching XtX_t). In the collider case, A∈tHA _t^H opens A, but the segment then continues below the window, where all nodes are unconditioned, and must eventually re-enter WintWin_t to reach Y (every edge spans at most k lags, and Y lies above the window); at the re-entry node the same dichotomy applies, and since a finite path can sustain only finitely many below-window excursions, it must ultimately traverse a transmitting (tail-edge) window node, where the Markov-boundary property blocks it. Hence no open path behind XtX_t leaves WintWin_t, and no variable at a timestep ≤t−k−1≤ t-k-1 participates in an open path into WintWin_t. We decompose (H)t′(G_H)_t into two parts: • The local subgraph ℒtL_t: the restriction of (H)t′(G_H)_t to variables at timesteps in WintWin_t. • The tail subgraph tT_t: the restriction of (H)t′(G_H)_t to variables at timesteps in t+1,…,H\t+1,…,H\, including Y. By the pa+pa^+ bound, every directed or bidirected connection between ℒtL_t and tT_t runs from a variable in t−k+1,…,t⊂Wint\t-k+1,…,t\ _t to a variable in t+1,…,t+k⊂t\t+1,…,t+k\ _t. We call the variables in tT_t at timesteps t+1,…,t+k\t+1,…,t+k\ that are collider-connected to, or have a parent in, ℒtL_t the interface variables tI_t. By the sealing property, every path leaving ℒtL_t for tT_t does so through a node in tI_t whether via a directed or bidirected edge; these interface nodes together with their incident edges are themselves contained in pa+pa^+-bounded neighborhoods of the window. In particular, both directed and bidirected boundary crossings are confined to tI_t; the manipulated-graph construction additionally removes incoming edges to future actions Xj∈tX_j _t, so no boundary crossing terminates at a future action. Step 2: Isomorphism between the local subgraph and the proxy graph. Consider the proxy graph kG_k on timesteps 0,…,k\0,…,k\ with terminal reward YkY_k. Define the time-shift ϕt(τ)=τ+(t−k) _t(τ)=τ+(t-k). By time-homogeneity (Assumption 1(i)), ϕt _t induces a graph isomorphism between kG_k restricted to 0,…,k\0,…,k\ and ℒtL_t, preserving all directed and bidirected edges of the projections as well as the identity of state variables, actions, and reward. (Time-homogeneity of ℱ F carries over to the projections, since the latent projection is determined by the diagram.) This extends to manipulated graphs. In the proxy manipulated graph (k)k′(G_k)_k , timestep k is the final step so there are no future actions to modify. In ℒtL_t, the action XtX_t is likewise the last action (all actions at j>tj>t belong to tT_t). The adjustment sets satisfy ϕt(τ)H=ϕt(τk)Z_ _t(τ)^H= _t(Z_τ^k) by the identical relative-lag construction of Algorithm 1. Therefore ℒtL_t with the manipulated-graph modifications is isomorphic to (k)k′(G_k)_k under ϕt _t. Step 3: Relating YkY_k and Y via the interface. In kG_k, the terminal reward YkY_k receives edges from variables at timesteps k−k,…,k=0,…,k\k-k,…,k\=\0,…,k\ (by k-bounded influence). Since Y is a terminal sink node with only incoming edges in both kG_k and HG_H, and its parent structure follows the same time-homogeneous template, the structural role of YkY_k relative to 0,…,k\0,…,k\ in kG_k is identical to that of Y relative to H−k,…,H\H-k,…,H\ in HG_H. Under the isomorphism, these correspond to variables in WintWin_t in HG_H. In (H)t′(G_H)_t , the terminal reward Y at timestep H is not solely adjacent to WintWin_t; it is connected to WintWin_t only through paths that traverse tT_t. However, by Step 1, all connections from ℒtL_t into tT_t pass through the interface tI_t at timesteps t+1,…,t+k\t+1,…,t+k\. This means that tI_t is a cut set between WintWin_t and Y in tT_t: every path from any variable in ℒtL_t to Y in (H)t′(G_H)_t must pass through some node of tI_t. The extended parents of YkY_k in kG_k (variables at timesteps 0,…,k\0,…,k\) are mapped by ϕt _t to variables in WintWin_t that are exactly the extended parents of the interface variables tI_t within ℒtL_t. Thus YkY_k in kG_k and Y in HG_H attach to ℒtL_t through the same set of nodes, with YkY_k directly and Y through tI_t and the tail. By the sealing property of Step 1, every path from a variable in ℒtL_t to Y in (H)t′(G_H)_t passes through tI_t. Step 4: Transferring the sequential π-backdoor conditions. Since tk\Z_t^k\ satisfies the sequential π-backdoor on kG_k, at timestep k either: Case A: Xk∉An(k)k′(Yk)X_k _(G_k)_k (Y_k). This means there is no directed path from XkX_k to YkY_k in (k)k′(G_k)_k . By the isomorphism of Step 2, there is no directed path from XtX_t to any interface variable in tI_t within ℒtL_t. Since every directed path from XtX_t to Y in (H)t′(G_H)_t must pass through tI_t (Step 3), and no directed path from XtX_t reaches tI_t, we conclude Xt∉An(H)t′(Y)X_t _(G_H)_t (Y). Case B: (Xk⟂Yk∣k)((k)k′)Xk(X_k \!\!\! Y_k _k^k)_((G_k)_k )_X_k. We show (Xt⟂Y∣tH)((H)t′)Xt(X_t \!\!\! Y _t^H)_((G_H)_t )_X_t. Let p be any path from XtX_t to Y in ((H)t′)Xt((G_H)_t )_X_t. Since outgoing edges from XtX_t are deleted, p must leave XtX_t via an incoming edge from a parent or a node collider-connected to XtX_t, all of which lie in WintWin_t by the pa+pa^+ bound (Step 1). If any node of p behind XtX_t (i.e., before its first crossing into tT_t) occurred at a timestep <t−k<t-k, then p would exit WintWin_t downward, and by the sealing property of Step 1 it would be blocked by tHZ_t^H; in that case we are done. It therefore suffices to consider paths whose portion behind XtX_t, up to the first crossing into tT_t, is contained in ℒtL_t. Since p connects XtX_t to Y and Y∉ℒtY _t, the path p must at some point cross from ℒtL_t into tT_t. By Step 1, this crossing must go through an interface variable I∈tI _t. Now consider the path p as having two segments: the segment pLp_L from XtX_t to the first interface variable I (contained in ℒt∪IL_t∪\I\), and the segment pTp_T from I onward to Y (contained in tT_t). We show p is blocked by tHZ_t^H by showing pLp_L is blocked. Construct the corresponding path pL′p_L in ((k)k′)Xk((G_k)_k )_X_k: apply ϕt−1 _t^-1 to map pLp_L from ℒtL_t to kG_k. The interface variable I at timestep t+jt+j (for some 1≤j≤k1≤ j≤ k) maps to a variable at timestep k+jk+j in kG_k. However, kG_k only contains timesteps 0,…,k\0,…,k\, so if j≥1j≥ 1, the variable I maps to a timestep beyond kG_k. We handle this as follows. By definition of tI_t, the interface variable I is connected to ℒtL_t through its extended parents in ℒtL_t: either a directed parent, or a node collider-connected to I whose own parents lie in ℒtL_t. By Step 3, YkY_k in the proxy attaches to the in-window nodes through the same extended-parent structure (under ϕt _t). Therefore any path from XkX_k that reaches an extended parent of YkY_k in ((k)k′)Xk((G_k)_k )_X_k corresponds to a path from XtX_t that reaches an extended parent of tI_t in ((H)t′)Xt((G_H)_t )_X_t, and conversely. Concretely, the path pLp_L from XtX_t to I reaches I through some node Vq∈pa+(I)∩ℒtV_q ^+(I) _t. The sub-path from XtX_t to VqV_q lies entirely in ℒtL_t (any step to a timestep <t−k<t-k is covered by the sealing property of Step 1, under which p is already blocked) and maps under ϕt−1 _t^-1 to a path from XkX_k to ϕt−1(Vq) _t^-1(V_q) in ((k)k′)Xk((G_k)_k )_X_k. Because pa+pa^+ is defined as the parents of a C-component, the collider/non-collider status of VqV_q along pLp_L is preserved under ϕt−1 _t^-1: a node entered through a bidirected edge inside a collider-connected chain remains so in the proxy, and a directed parent remains a directed parent. Since VqV_q is an extended parent of an interface variable, ϕt−1(Vq) _t^-1(V_q) is an extended parent of YkY_k in kG_k (Step 3), so the path from XkX_k to ϕt−1(Vq) _t^-1(V_q) extends to YkY_k through the corresponding parent or collider-connecting edge, forming a path q from XkX_k to YkY_k in ((k)k′)Xk((G_k)_k )_X_k. By the d-separation hypothesis, q is blocked by kZ_k^k. The extension from ϕt−1(Vq) _t^-1(V_q) to YkY_k introduces no open collider at ϕt−1(Vq) _t^-1(V_q) that was not already present on pLp_L: if the extension makes ϕt−1(Vq) _t^-1(V_q) a collider, the identical collider configuration appears at VqV_q on the full-graph path p (via the matching edge into the interface chain), so the two paths share collider status at this node and a block of one implies a block of the other. Hence the block of q occurs on the sub-path from XkX_k to ϕt−1(Vq) _t^-1(V_q), at a node of kZ_k^k. Mapping back via ϕt _t, the same block occurs on pLp_L at the corresponding node of tH=ϕt(k)Z_t^H= _t(Z_k^k). Therefore p is blocked by tHZ_t^H. Boundary timesteps (t<kt<k). For t<kt<k, the window is 0,…,t\0,…,t\. The before(Xt)before(X_t) constraint in Algorithm 1 (Line 6) ensures tkZ_t^k contains only variables at timesteps ≤t−1≤ t-1. Since t<kt<k, all such variables satisfy τ≥0>t−kτ≥ 0>t-k, so the clipping in Lines 11–13 is vacuous: tH=tkZ_t^H=Z_t^k. The subgraph of HG_H restricted to 0,…,t\0,…,t\ is identical to the subgraph of kG_k on the same timesteps (a direct subgraph inclusion requiring no time-shift). The manipulated graph (H)t′(G_H)_t restricted to 0,…,t\0,…,t\ matches (k)t′(G_k)_t restricted to 0,…,t\0,…,t\, since the modifications at timesteps >t>t (replacement of future action parents) do not alter the subgraph at 0,…,t\0,…,t\. Therefore the d-separation or non-ancestry condition holding for XtX_t in (k)t′(G_k)_t also holds in (H)t′(G_H)_t . Since for every t∈0,…,H−1t∈\0,…,H-1\ one of the two conditions of Definition 2 holds, tHt=0H−1\Z_t^H\_t=0^H-1 satisfies the sequential π-backdoor criterion for (H,,Y)(G_H,X,Y). ∎ B.1 Necessity of Extended Parents Set Assumption 1(i) bounds the temporal reach of the extended parent set pa+(t)pa^+(V_t) rather than that of the ordinary parent set pa(t)pa(V_t) or, equivalently, the maximal length of an individual edge. This is necessary, as an SCM may satisfy a naive edge-length bound yet violate the windowed criterion. Figure 6: Causal graph for Example 3, shown over the final three timesteps. Every edge spans a single timestep, yet the open path ρ, of XH↔XH−1←XH−2→OH−1↔OH↔YX_H X_H-1← X_H-2→ O_H-1 O_H Y, reaches the non-collider XH−2X_H-2 at lag −2-2, outside the width-11 window WinH=H−1,HWin_H=\H-1,H\. Example 3. Consider an SCM unrolled over horizon H with endogenous variables Ot,Xtt=1H\O_t,X_t\_t=1^H and terminal reward Y (Figure 6), with directed edges Ot→XtO_t→ X_t, Xt→Ot+1X_t→ O_t+1, and Xt→Xt+1X_t→ X_t+1, terminal edge XH→YX_H→ Y, and bidirected edges Ot↔Ot+1O_t O_t+1, Xt↔Xt+1X_t X_t+1, and OH↔YO_H Y from unobserved confounders, with only Y∈LY ^L. Every directed and bidirected edge spans exactly one timestep, satisfying the condition that pa(Vt)pa(V_t) lie within k=1k=1 of Vt,Vt∈V_t,V_t . Now examine action XHX_H at the final step under the resulting window from Algorithm 1, WinH=H−1,HWin_H=\H-1,H\, and consider the path ρ:XH↔XH−1←XH−2→OH−1↔OH↔Y.ρ: X_H X_H-1← X_H-2→ O_H-1 O_H Y. On ρ, the nodes XH−1X_H-1, OH−1O_H-1, OHO_H are colliders that are conditioned and thus opened. To block ρ one must condition on XH−2X_H-2. But XH−2X_H-2 lies at timestep H−2H-2 outside WinHWin_H, so a window of width k=1k=1 omits it and ρ remains open rendering the windowed adjustment set is invalid. Though XH−2X_H-2 is only a single directed edge removed from the window, it is a parent of the C-component OH−2,OH−1,OH,YO_H-2,O_H-1,O_H,Y due to XH−2→OH−1X_H-2→ O_H-1. Hence XH−2∈pa+(H)X_H-2 ^+(V_H), even though no single bidirected edge reaches it from the window. Assumption 1(i), which bounds pa+pa^+, is violated by this SCM at k=1k=1 as it requires every element of pa+(H)pa^+(V_H) including XH−2X_H-2 to occur no earlier than H−kH-k, forcing k≥2k≥ 2. This blocks ρ and deems the windowed adjustment set valid while satisfying Assumption 1. Therefore, by bounding pa+(V)pa^+(V) instead of pa(V)pa(V), collider-connected paths that have causal influence on V beyond the edge length of direct parents are considered. ■ Appendix C Algorithm Pseudocode Algorithms 2 and 3 give the full training procedures for Causal SQIL and Causal IQ-Learn, respectively. Both algorithms share the same causal encoding layer (Algorithm 1) and SAC-style actor update, differing only in how the critic is trained. In Causal SQIL the critic minimizes a standard soft Bellman residual on binary-labeled transitions, whereas in Causal IQ-Learn it minimizes a chi-squared divergence objective on implicit rewards. All other components (entropy tuning, soft target updates, replay buffer management, and the causal adjustment algorithm) are identical. Algorithm 2 Causal SQIL 0: Expert demonstrations expD_exp, causal graph G, window size k, horizon H, discount γ, soft update rate τ, batch size B. 1: Compute windowed adjustment sets tHt=0H−1\Z_t^H\_t=0^H-1 and window specification S via Algorithm 1. 2: Build encoder Encode(,t) Encode(s,t) from S: concatenates values of V∈O∩tHV ^O _t^H at lags 0,−1,…,−k\0,-1,…,-k\ relative to t, zero-padding when t<kt<k. 3: Initialize actor πϕ _φ, twin Q-networks Qθ1,Qθ2Q_ _1,Q_ _2, target networks Qθ¯1←Qθ1Q_ θ_1← Q_ _1, Qθ¯2←Qθ2Q_ θ_2← Q_ _2. 4: Initialize entropy coefficient logα←0 α← 0, target entropy ℋ¯←−|| H←-|A|. 5: ℬexp←∅B_exp← , ℬπ←∅B_π← . 6: for each transition (t,t,t+1,dt)(s_t,x_t,s_t+1,d_t) in expD_exp do 7: ℬexp←ℬexp∪(Encode(t,t),t,r=1,Encode(t+1,t+1),dt)B_exp _exp∪\( Encode(s_t,t),\;x_t,\;r=1,\; Encode(s_t+1,t+1),\;d_t)\ 8: end for 9: for each episode e=1,2,…e=1,2,… do 10: Reset environment, observe 0s_0. 11: for t=0,…,H−1t=0,…,H-1 do 12: t←Encode(t,t)z_t← Encode(s_t,t) 13: Sample t∼πϕ(⋅∣t)x_t _φ(· _t), execute, observe t+1s_t+1, dtd_t. 14: ℬπ←ℬπ∪(t,t,r=0,Encode(t+1,t+1),dt)B_π _π∪\(z_t,\;x_t,\;r=0,\; Encode(s_t+1,t+1),\;d_t)\ 15: end for 16: for each gradient step do 17: Sample (,,r,′,d)i=1B\(z,x,r,z ,d)\_i=1^B with B/2B/2 from ℬexpB_exp and B/2B/2 from ℬπB_π. 18: // Critic update 19: ′∼πϕ(⋅∣′)x _φ(· ) 20: y←r+γ(1−d)(minjQθ¯j(′,′)−αlogπϕ(′∣′))y← r+γ(1-d) ( _jQ_ θ_j(z ,x )-α _φ(x ) ) 21: ℒQ←12∑j=12‖Qθj(,)−y‖2L_Q← 12 _j=1^2 \|Q_ _j(z,x)-y \|^2 22: Update θ1,θ2 _1, _2 by ∇θℒQ _θL_Q. 23: // Actor update 24: ~∼πϕ(⋅∣) x _φ(· ) (reparameterized) 25: ℒπ←[αlogπϕ(~∣)−minjQθj(,~)]L_π [α _φ( x )- _jQ_ _j(z, x) ] 26: Update ϕφ by ∇ϕℒπ _φL_π. 27: // Entropy tuning 28: ℒα←−logα[logπϕ(~∣)+ℋ¯]L_α←- α\;E [ _φ( x )+ H ] 29: Update logα α by ∇ℒα _α, α←exp(logα)α← ( α). 30: // Soft target update 31: θ¯j←τθj+(1−τ)θ¯j θ_j←τ\, _j+(1-τ)\, θ_j for j=1,2j=1,2. 32: end for 33: end for 34: return πϕ _φ. Algorithm 3 Causal IQ-Learn 0: Expert demonstrations expD_exp, causal graph G, window size k, horizon H, discount γ, soft update rate τ, batch size B, number of value samples K. 1: Compute adjustment sets and build encoder Encode as in Algorithm 2, steps 1–2. 2: Initialize actor πϕ _φ, twin Q-networks Qθ1,Qθ2Q_ _1,Q_ _2, target networks Qθ¯1←Qθ1Q_ θ_1← Q_ _1, Qθ¯2←Qθ2Q_ θ_2← Q_ _2. 3: Initialize entropy coefficient logα←0 α← 0, target entropy ℋ¯←−|| H←-|A|. 4: ℬexp←∅B_exp← , ℬπ←∅B_π← . 5: Encode expert demonstrations into ℬexpB_exp as (t,t,t+1,dt)\(z_t,x_t,z_t+1,d_t)\ (no reward labels). 6: for each episode e=1,2,…e=1,2,… do 7: Roll out πϕ _φ in the environment with causal encoding (as in Algorithm 2, steps 9–13). Store transitions in ℬπB_π. 8: for each gradient step do 9: Sample (e,e,′e,de)i=1B/2\(z^e,x^e,z e,d^e)\_i=1^B/2 from ℬexpB_exp and (p,p,′p,dp)i=1B/2\(z^p,x^p,z p,d^p)\_i=1^B/2 from ℬπB_π. 10: Let (,,′,d)(z,x,z ,d) denote the concatenation of expert and policy batches. 11: // Compute soft state value via Monte Carlo 12: ~mm=1K∼πϕ(⋅∣′)\ x_m\_m=1^K _φ(· ) 13: V¯(′)←1K∑m=1K[minjQθ¯j(′,~m)−αlogπϕ(~m∣′)] V(z )← 1K _m=1^K [ _jQ_ θ_j(z , x_m)-α _φ( x_m ) ] 14: // Implicit reward 15: r^j(,)←Qθj(,)−γ(1−d)V¯(′) r_j(z,x)← Q_ _j(z,x)-γ(1-d)\, V(z ) for j=1,2j=1,2 16: // Critic update (chi-squared divergence) 17: ℒQj←−exp[r^j(e,e)]+12all[r^j(,)2]L_Q_j←-E_exp [ r_j(z^e,x^e) ]+ 12\,E_all [ r_j(z,x)^2 ] for j=1,2j=1,2 18: Update θ1,θ2 _1, _2 by ∇θℒQ _θL_Q. 19: // Actor update (identical to Algorithm 2, steps 21–24) 20: ~∼πϕ(⋅∣) x _φ(· ), ℒπ←[αlogπϕ(~∣)−minjQθj(,~)]L_π [α _φ( x )- _jQ_ _j(z, x) ] 21: Update ϕφ by ∇ϕℒπ _φL_π. 22: // Entropy tuning and soft target update (identical to Algorithm 2, steps 26–29) 23: Update logα α, α, and θ¯1,θ¯2 θ_1, θ_2. 24: end for 25: end for 26: return πϕ _φ. For completeness, we reproduce the FindOX algorithm of kumor2021sequential (Algorithm 1 in that paper), which returns the maximal set X⊆OO_X ^O from which sequential π-backdoor admissible sets can be constructed. Given the causal diagram G, action set X, and target Y, FindOX iteratively grows XO_X by checking, for each observed node, whether a valid backdoor adjustment exists that would make that node a non-ancestor of Y in the manipulated graph. A sequential π-backdoor exists for (,,Y)(G,X,Y) if and only if ⊆XX _X (kumor2021sequential, Theorem 3.1). Once XO_X is obtained, the per-action adjustment sets tZ_t are constructed from the Markov boundary of XO_X in ′YG^Y_X (where ′=X∩X =O_X ), intersected with before(t)before(X_t). Our windowed adjustment procedure (Algorithm 1) invokes FindOX on a short-horizon proxy graph kG_k rather than the full-horizon graph HG_H. Algorithm 4 FindOX (kumor2021sequential): Find largest valid XO_X in ancestral graph of Y 0: Causal diagram G, action set X, target Y. 1: function HasValidAdjustment(,X,Oi,XiG,O^X,O_i,X_i) 2: C←C← the C-component of OiO_i in YG^Y 3: C←G_C← the subgraph of YG^Y containing only Pa+(C)Pa^+(C) and intermediate latent variables 4: C←C∖(X∪Oi)O^C← C (O^X∪\O_i\) // Elements of C-component that might be ancestors of Y in i′G _i 5: return (Oi⟂C∣C∩before(Xi))(O_i \!\!\! ^C ^C (X_i)) in CG_C 6: 7: function FindOX(,,YG,X,Y) 8: X← O^X← empty map from elements of OV^O to elements of X 9: repeat 10: for Oi∈O_i ^O of YG^Y (ancestral graph of Y) in reverse temporal order do 11: if |ch+(Oi)|>0|ch^+(O_i)|>0 and ch+(Oi)⊆keys(X)ch^+(O_i) ( O^X) then 12: Xi←X_i← earliest element of X[ch+(Oi)] O^X[ch^+(O_i)] in temporal order 13: if HasValidAdjustment(,keys(X),Oi,XiG,keys( O^X),O_i,X_i) then 14: X[Oi]←Xi O^X[O_i]← X_i 15: else if Oi∈O_i and HasValidAdjustment(,keys(X),Oi,OiG,keys( O^X),O_i,O_i) then 16: X[Oi]←Oi O^X[O_i]← O_i 17: while |X|| O^X| changed in most recent pass 18: return keys(X)keys( O^X) Appendix D Environment Details We describe each confounded environment in detail, including the causal structure, the confounding mechanism, and the observation partition. Table 3 summarizes the key dimensions and setup. Confounded AntMaze. The base task is goal-conditioned navigation in a windy maze using an 8-DoF ant robot as described in Example 1. The imitator does not observe O, or torso orientation; to compensate, a 2D compass reading W is added providing a noisy surrogate for heading. The latent wind field U follows a piecewise-constant gust process and affects both the dynamics and the reward function. The compass is prone to distributional shift between expert and imitator environments due to changing influence from U. Causally unaware methods that condition on W conflate the wind’s influence on the compass with the ant’s true heading, learning policies that turn into walls when the wind changes direction. Causal methods exclude W from their adjustment set and learn to navigate the maze using position, joint angles, and velocities alone, without explicit orientation information. Figure 7: Confounded AntMaze. P is global position, L is torso linear velocity, O is torso orientation, A is joint angles, T is torso angular velocity, J is joint angular velocities, U is the latent wind field, W is the compass, X is joint torques, and Y is the latent terminal reward. Figure 8: Visualization for Confounded AntMaze, demonstrating the effect of latent winds U (blue) on the heading compass W (red) despite the original heading O (green) being completely different. Confounded HumanoidMaze. The base task is goal-conditioned navigation in a maze using a 21-DoF humanoid robot. The imitator does not observe C, the 3D center-of-mass velocity; to compensate, a 2D ground-vibration reading W is added that provides a noisy surrogate for locomotion velocity. The latent seismic tremor U follows a piecewise-constant impulse process and affects both the dynamics (applying external force to the torso) and the reward function. The vibration sensor is prone to distributional shift between expert and imitator environments due to changing influence from U. Causally unaware methods that condition on W conflate the tremor’s influence on the vibration reading with the humanoid’s true velocity, learning policies that stumble or over-correct when the tremor changes direction. Causal methods exclude W from their adjustment set and must learn to navigate the maze using position, joint angles, head height, extremity positions, torso orientation, and joint velocities alone, without explicit velocity information. The confounding effect of the seismic tremor is less disruptive than that of the wind field in AntMaze. Figure 9: Confounded HumanoidMaze. P is global position, A is joint angles, H is head height, E is extremities in torso frame, V is torso vertical, C is center-of-mass gravity, J is joint velocities, U is the latent seismic tremor, W is the vibration sensor, X is joint torques, and Y is the latent reward. Figure 10: Visualization for Confounded HumanoidMaze, demonstrating the effect of latent seismic tremors U (yellow) on the perceived ground vibration sensor W (red) despite the true center-of-mass velocity C (green) pointing in a completely different direction. Environment H |||V| |||X| Latent Confounder Collider AntMaze 1000 35 8 Wind field Compass HumanoidMaze 2000 89 21 Seismic tremors Vibration Table 3: Summary of confounded environments. |||V| denotes the imitator’s observation dimensionality (excluding hidden variables). |||X| is the action dimensionality. Appendix E Hyperparameters and Training Details Table 4 summarizes the hyperparameters for all algorithms. Causal and non-causal variants of each algorithm use identical hyperparameters and network architectures; the only difference is the input representation (causally-adjusted tZ_t vs. full observation tOV^O_t). Where hyperparameters differ between environment families we write A (AntMaze) and ℋH (HumanoidMaze). All tasks used 3 seeds. Note: due to limited computational resources, HumanoidMaze-Large evaluations consisted of 100100 episodes rather than the other tasks’ 10001000 episodes, and non-causal SQIL for HumanoidMaze-Large was trained on only 1M1M timesteps instead of 2M2M; this decision was informed by the fact that every algorithm in every task reached its best checkpoint before the 1M1M timestep mark during training. Table 4: Hyperparameters for all algorithms. Causal and non-causal variants share identical settings. “—” denotes a hyperparameter not applicable to that algorithm. A / ℋH distinguishes AntMaze and HumanoidMaze values where they differ. Hyperparameter BC GAIL SQIL IQ-Learn Network Architecture Hidden dimension 256 256 256 256 Actor residual blocks 4 3 3 3 Actor dropout 0.0 0.05 0.05 0.05 Layer normalization ✓ ✓ ✓ ✓ Activation SiLU SiLU SiLU SiLU Output squashing Tanh Tanh Tanh Tanh Optimization Optimizer Adam Adam Adam Adam Actor learning rate 3×10−43× 10^-4 1×10−41× 10^-4 3×10−43× 10^-4 3×10−43× 10^-4 Critic learning rate — 3×10−43× 10^-4 3×10−43× 10^-4 3×10−43× 10^-4 Batch size 2048 1024 256 256 Discount γ — 0.99 0.99 0.99 Max gradient norm — 0.5 1.0 1.0 BC-Specific Loss function Huber — — — Training epochs A: 100 / ℋH-Med: 100 / ℋH-Large: 200 — — — Early stopping patience A: 15 / ℋH-Med: 30 / ℋH-Large: 15 — — — Validation fraction 0.2 — — — GAIL-Specific (PPO + Discriminator) PPO clip ratio ϵε — 0.2 — — GAE λ — 0.95 — — PPO epochs per round — A: 4 / ℋH: 2 — — PPO entropy coefficient — 10−210^-2 — — Value loss coefficient — 0.5 — — Normalize advantages — ✓ — — Discriminator learning rate — 3×10−43× 10^-4 — — Discriminator dropout — 0.2 — — Discriminator updates per round — 2 — — Discriminator minibatch size — 1024 — — Gradient penalty λGP _GP — 5.0 — — Episodes per round — A: 20 / ℋH: 10 — — Total training rounds — A: 500 / ℋH: 200 — — Disc. LR scheduler — StepLR(100100, 0.50.5) — — SQIL / IQ-Learn (SAC-Based) Soft update rate τ — — 0.005 0.005 Entropy coefficient α — — auto-tuned auto-tuned Entropy learning rate — — 3×10−43× 10^-4 3×10−43× 10^-4 Target entropy ℋ¯ H — — −||-|A| −||-|A| α clamp range — — [e−log1000,e−log10][e^- 1000,\;e^- 10] [e−log1000,e−log10][e^- 1000,\;e^- 10] Replay buffer capacity — — 10610^6 10610^6 Expert buffer ratio — — 0.5 0.5 Random exploration steps — — 5000 5000 Update-to-data ratio — — A: 0.25 / ℋH: 0.5 A: 0.25 / ℋH: 0.5 Total timesteps — — 2×1062× 10^6 2×1062× 10^6 V estimation — — — A: MC (K=16K=16) / ℋH: single-sample Critic LR scheduler — — CosineAnnealing CosineAnnealing Shared Lookback window k A: 10 / ℋH: 2 A: 10 / ℋH: 2 A: 10 / ℋH: 2 A: 10 / ℋH: 2 Max episode steps A: 1000 / ℋH: 2000 A: 1000 / ℋH: 2000 A: 1000 / ℋH: 2000 A: 1000 / ℋH: 2000 Expert construction. Expert policies for the maze tasks are constructed using offline-to-online RL. For each environment, we begin by training a goal-conditioned behavioral cloning (BC) policy on provided demonstrations from the base OGBench dataset. This policy provides an initialization that captures the global structure of the task, but it is not yet ready for the confounders introduced in the modified environment. To obtain an expert that reflects performance under the confounded dynamics, we then fine-tune this BC policy through a period of off-policy actor–critic training using TD3 (fujimoto2018td3). Reward shaping is added optionally during fine-tuning to compensate for the sparse reward signals. The resulting expert policy is capable of operating effectively under the latent disturbances, partial observability, and altered transition dynamics of the confounded environment. Network architecture. All algorithms share a residual MLP backbone for the actor. The actor network maps the causally-adjusted encoding tz_t to actions through a linear projection into hidden dimension 256256, followed by residual blocks, and a final linear output with tanh squashing to the action bounds. Each residual block consists of LayerNorm, SiLU, a linear layer, a second LayerNorm, SiLU, dropout, and a second linear layer, with a skip connection from input to output. For GAIL, SQIL, and IQ-Learn, the actor outputs the mean of a squashed Gaussian with a state-independent learnable log-variance; for BC, the actor outputs a deterministic action. The Q-networks (SQIL and IQ-Learn) use the same residual MLP architecture, taking concatenated (t,t)(z_t,x_t) as input and producing a scalar output. GAIL’s value network (critic) uses a simpler three-layer MLP with ReLU activations, taking tz_t as input. GAIL’s discriminator similarly uses a three-layer MLP with ReLU activations, skip connections between layers, higher dropout (0.20.2), and binary cross-entropy loss to classify (t,t)(z_t,x_t) pairs as expert or policy-generated. SQIL training details. Causal SQIL uses SAC with twin Q-networks and soft target updates (τ=0.005τ=0.005). The entropy coefficient α is automatically tuned toward a target entropy of −||-|X| and clamped to [exp(−log1000),exp(−log10)]≈[0.001,0.1][ (- 1000),\; (- 10)]≈[0.001,0.1]. The replay buffer is split equally: 50%50\% capacity for expert transitions (labeled r=1r=1) and 50%50\% for policy transitions (r=0r=0), with each training batch sampled 50/5050/50 from both halves. The first 5,0005,000 timesteps use random actions for exploration before policy rollouts begin. Training runs for 2×1062× 10^6 environment steps. The update-to-data (UTD) ratio is 0.250.25 for AntMaze tasks (one gradient step per four environment steps) and 0.50.5 for HumanoidMaze tasks. A cosine annealing schedule is applied to the critic learning rate. IQ-Learn training details. Causal IQ-Learn shares the SAC actor update and twin Q-network architecture with Causal SQIL. The key difference is the critic loss: instead of minimizing a soft Bellman residual on binary-labeled transitions, IQ-Learn minimizes the chi-squared divergence between the implicit reward distribution under the expert and the combined (expert ++ policy) data. For AntMaze tasks, the soft state value V(′)V(z ) is estimated via K=16K=16 Monte Carlo samples from the current policy; for HumanoidMaze tasks, a single-sample estimate is used (matching the standard SAC target computation), which avoids the high variance of multi-sample estimates in the higher-dimensional action space. The entropy coefficient α is auto-tuned and clamped to [exp(−log1000),exp(−log10)]≈[0.001,0.1][ (- 1000),\; (- 10)]≈[0.001,0.1] to prevent entropy collapse or explosion. IQ-Learn uses the same UTD ratios as SQIL (0.250.25 for AntMaze, 0.50.5 for HumanoidMaze). GAIL training details. Causal GAIL alternates between on-policy rollouts and discriminator–policy updates. Each round collects 2020 episodes for AntMaze (1010 for HumanoidMaze), up to H steps each, using the current policy, computes advantages via GAE (λ=0.95λ=0.95, γ=0.99γ=0.99), and performs PPO updates with clipping ratio ϵ=0.2ε=0.2, entropy regularization coefficient 10−210^-2, and value loss coefficient 0.50.5. The number of PPO epochs per round is 44 for AntMaze and 22 for HumanoidMaze. The discriminator is updated 22 times per round on minibatches of 10241024 with gradient penalty (λGP=5.0 _GP=5.0). The discriminator learning rate follows a StepLR schedule, halving every 100100 rounds. Training runs for 500500 rounds on AntMaze and 200200 rounds on HumanoidMaze. BC training details. Causal BC performs supervised learning on expert demonstrations using Huber loss, optimized with Adam at learning rate 3×10−43× 10^-4. The actor uses 44 residual blocks with no dropout, trained for up to 100100 epochs (200 for HumanoidMaze-Large) with early stopping on a held-out 20%20\% validation split. The early stopping patience is 1515 for all tasks except HumanoidMaze-Medium, which uses a patience of 3030. Training uses large batches of 20482048 to reduce variance in the gradient estimates. BC is the only algorithm that does not interact with the environment during training; it learns entirely from the static expert dataset. Appendix F Additional Results F.1 Raw Evaluation Data Table 5: Evaluation results on confounded tasks without normalization. Best non-expert result per task in bold. Ant-Med Ant-Large Hum-Med Hum-Large Algorithm [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) Expert −104.0±87.5-104.0 ± 87.5 87.6±1.087.6 ± 1.0 −255.2±122.8-255.2 ± 122.8 55.9±1.655.9 ± 1.6 −522.9±218.4-522.9 ± 218.4 33.8±1.533.8 ± 1.5 −840.6±212.9-840.6 ± 212.9 7.0±0.87.0 ± 0.8 C-BC −125.2±100.9-125.2 ± 100.9 77.9±1.377.9 ± 1.3 −284.9±121.6-284.9 ± 121.6 39.8±1.539.8 ± 1.5 −655.7±172.7-655.7 ± 172.7 10.4±1.010.4 ± 1.0 −858.6±227.9-858.6 ± 227.9 8.0±2.78.0 ± 2.7 C-GAIL −135.8±113.6-135.8 ± 113.6 74.1±1.474.1 ± 1.4 −328.5±122.6-328.5 ± 122.6 7.3±0.87.3 ± 0.8 −747.2±161.9-747.2 ± 161.9 0.0±0.00.0 ± 0.0 −976.3±210.2-976.3 ± 210.2 0.0±0.00.0 ± 0.0 C-SQIL −99.6±85.7-99.6 ± 85.7 90.7±0.990.7 ± 0.9 −279.9±125.9-279.9 ± 125.9 45.0±1.645.0 ± 1.6 −555.5±217.2-555.5 ± 217.2 24.7±1.424.7 ± 1.4 −837.7±226.0-837.7 ± 226.0 8.0±2.78.0 ± 2.7 C-IQL −118.2±101.3-118.2 ± 101.3 84.3±1.1584.3 ± 1.15 −258.6±127.9-258.6 ± 127.9 58.9±1.658.9 ± 1.6 −588.8±203.8-588.8 ± 203.8 19.1±1.219.1 ± 1.2 −887.1±220.5-887.1 ± 220.5 3.0±1.73.0 ± 1.7 BC −375.3±119.2-375.3 ± 119.2 0.0±0.00.0 ± 0.0 −470.2±150.2-470.2 ± 150.2 0.0±0.00.0 ± 0.0 −678.4±169.1-678.4 ± 169.1 5.4±0.75.4 ± 0.7 −884.1±204.6-884.1 ± 204.6 5.0±2.25.0 ± 2.2 GAIL −364.9±112.2-364.9 ± 112.2 0.0±0.00.0 ± 0.0 −398.4±132.9-398.4 ± 132.9 0.0±0.00.0 ± 0.0 −747.7±162.5-747.7 ± 162.5 0.0±0.00.0 ± 0.0 −977.0±208.9-977.0 ± 208.9 0.0±0.00.0 ± 0.0 SQIL −370.7±119.1-370.7 ± 119.1 0.0±0.00.0 ± 0.0 −484.2±151.5-484.2 ± 151.5 0.0±0.00.0 ± 0.0 −706.2±157.0-706.2 ± 157.0 0.1±0.10.1 ± 0.1 −896.3±194.9-896.3 ± 194.9 0.0±0.00.0 ± 0.0 IQL −372.5±120.8-372.5 ± 120.8 0.0±0.00.0 ± 0.0 −468.1±153.5-468.1 ± 153.5 0.0±0.00.0 ± 0.0 −697.6±169.7-697.6 ± 169.7 2.4±0.52.4 ± 0.5 −906.2±193.3-906.2 ± 193.3 0.0±0.00.0 ± 0.0 Remark on expert success rates. We note that the raw expert success rates in Table 5 are substantially lower than those typically reported in standard MuJoCo benchmarks. This is by design: our confounded environments are partially observable even to the expert, as the exogenous disturbances U (represented by bidirected arrows in the causal diagrams) are unobserved by all agents, including the expert. The unpredictable external forces generated by U make consistent task completion inherently difficult regardless of the agent’s sensory access or learning algorithm. In this sense, the environments are designed to reflect realistic deployment conditions in which no agent has full knowledge of the environment dynamics, rather than the fully observed settings common in standard benchmarks. Nevertheless, the expert has strictly more information than the imitator (observing O∪LV^O ^L rather than OV^O alone) and serves as a meaningful upper bound on what can be achieved given the imitator’s observation set. The normalized metrics in Table 1 should therefore be interpreted relative to this upper bound: an algorithm recovering, say, 80%80\% of the expert’s success rate is operating near the frontier of what is achievable given the imitator’s partial observability, not at 80%80\% of a trivially solvable task. Cases in which a causal method exceeds 100%100\% normalized performance (e.g., Causal SQIL on AntMaze-Medium, Causal IQ-Learn on AntMaze-Large) indicate that the imitator’s learned policy is slightly more robust to the stochastic disturbances than the expert policy obtained via offline-to-online RL, likely because the off-policy Q-learning objective implicitly averages over the disturbance distribution encountered during training rather than committing to the point estimates used during expert fine-tuning. F.2 Episode Lengths We report the mean episode length for successfully solved episodes in Table 6. Shorter episodes indicate more efficient navigation. Only algorithms with nonzero success rates are included. Table 6: Mean episode length (steps) for successfully solved episodes (± std). Only algorithms with nonzero success rates are shown. Best result per task in bold. Algorithm AntMaze-Med AntMaze-Large HumanoidMaze-Med HumanoidMaze-Large Expert 301±57301 ± 57 648±99648 ± 99 976±410976 ± 410 1463±3661463 ± 366 C-BC 318±79318 ± 79 683±116683 ± 116 1356±3441356 ± 344 1477±4211477 ± 421 C-GAIL 309±75309 ± 75 709±134709 ± 134 — — C-SQIL ±62299 ± 62 695±119695 ± 119 ±425912 ± 425 ±3161251 ± 316 C-IQL 321±90321 ± 90 ±108636 ± 108 1024±4431024 ± 443 1848±841848 ± 84 BC — — 1364±3201364 ± 320 1663±2371663 ± 237 GAIL — — — — SQIL — — 1800±01800 ± 0 — IQL — — 1085±3961085 ± 396 — F.3 Wall-Clock Runtime of Full vs. Windowed Adjustment To empirically motivate the windowed approximation of Algorithm 1, we measure the wall-clock time required to compute the per-timestep adjustment sets tkZ_t^k under different horizons up to H=2000H=2000. Runtime for H=1000,2000H=1000,2000 reflects full sequential π-backdoor computation for environments such as AntMaze and HumanoidMaze, while the runtime for smaller H (i.e. H=2,10H=2,10) is equivalent to that of Algorithm 1 which uses k=2,10k=2,10 respectively in our experiments due to windowed adjustment. Figure 11: Runtime to compute adjustment sets as a function of horizon H (log–log scale). Algorithm 1 (H=2H=2 for HumanoidMaze, H=10H=10 for AntMaze; Appendix E) completes in a few seconds, while full-horizon adjustment (H=1000,2000H=1000,2000) takes over four hours. The runtime is roughly constant for small H but grows super-linearly once H exceeds a few hundred, increasing from 7.17.1s at H=1H=1 to 14,85414,854s (4.14.1 hours) at H=2000H=2000. By restricting the sequential π-backdoor computation to a fixed-size window (Theorem 1), Algorithm 1 reduces this cost to O(k)O(k) independent of H, making causal adjustment tractable for the long-horizon tasks in this work. F.4 Ablations over Horizon Length and Confounding Strength The main experiments (Table 1) fix the episode horizon at the default (H=1000H=1000 for AntMaze, H=2000H=2000 for HumanoidMaze) and the confounding strength at 100%100\%. To test the effect of these settings in higher granularity, we sweep the horizon H∈500,1000,1500H∈\500,1000,1500\ (Table 7) and the confounding strength ∈15%,50%,85%∈\15\%,50\%,85\%\ (Table 8) on both AntMaze tasks (in which confounding strength is represented by heading compass diversion from the true heading) for all eight algorithms; due to limited computational resources, the HumanoidMaze tasks are excluded from these ablations. Both sweeps utilize the same expert as the main experiments, though for H∈500,1500H∈\500,1500\ expert demonstrations are generated with the appropriate horizon. Note that at 0%0\% confounding strength the compass is an uncorrupted (if noisy) heading sensor, so a non-causal imitator would incur no confounding bias and could even benefit from the additional signal, as seen in the 15%15\%-confounded columns. The H=1000H=1000 and 100%100\%-confounded columns reproduce the results reported in Table 1 and are listed for reference. [Y]E[Y] is normalized in the same manner as in Table 1. Because the reward penalizes every step taken before reaching the goal, it reflects solving speed; thus, an episode solved in a given number of steps yields a larger normalized value under a longer horizon (failed episodes accrue penalties for all H steps), while the anchor changes with the strength setting per configuration. Normalized [Y]E[Y] therefore compares algorithms within a configuration and is generally not comparable across horizons or across strengths. H=500H=500 H=1000H=1000 H=1500H=1500 Algorithm [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) C-BC 90.4±62.490.4 ± 62.4 78.2±1.378.2 ± 1.3 250.1±100.9250.1 ± 100.9 77.9±1.377.9 ± 1.3 406.5±153.1406.5 ± 153.1 81.3±1.281.3 ± 1.2 C-GAIL 76.5±68.976.5 ± 68.9 64.5±1.464.5 ± 1.4 239.5±113.6239.5 ± 113.6 74.1±1.474.1 ± 1.4 400.1±159.4400.1 ± 159.4 80.2±1.380.2 ± 1.3 C-SQIL 98.3±59.398.3 ± 59.3 87.5±1.087.5 ± 1.0 275.8±85.7275.8 ± 85.7 90.7±0.990.7 ± 0.9 431.0±135.8431.0 ± 135.8 87.0±1.187.0 ± 1.1 C-IQL 99.1±59.099.1 ± 59.0 86.3±1.186.3 ± 1.1 257.1±101.3257.1 ± 101.3 84.3±1.184.3 ± 1.1 431.7±134.0431.7 ± 134.0 87.2±1.187.2 ± 1.1 BC 17.1±77.717.1 ± 77.7 0.0±0.00.0 ± 0.0 0.0±119.20.0 ± 119.2 0.0±0.00.0 ± 0.0 82.3±132.882.3 ± 132.8 0.5±0.10.5 ± 0.1 GAIL 17.1±77.317.1 ± 77.3 0.0±0.00.0 ± 0.0 10.4±112.210.4 ± 112.2 0.0±0.00.0 ± 0.0 199.0±170.4199.0 ± 170.4 26.5±1.026.5 ± 1.0 SQIL 6.0±81.96.0 ± 81.9 0.0±0.00.0 ± 0.0 4.7±119.14.7 ± 119.1 0.0±0.00.0 ± 0.0 2.3±144.42.3 ± 144.4 0.0±0.00.0 ± 0.0 IQL 0.0±84.30.0 ± 84.3 0.0±0.00.0 ± 0.0 2.8±120.82.8 ± 120.8 0.0±0.00.0 ± 0.0 0.0±145.20.0 ± 145.2 0.0±0.00.0 ± 0.0 (a) AntMaze-Medium H=500H=500 H=1000H=1000 H=1500H=1500 Algorithm [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) C-BC 61.2±83.961.2 ± 83.9 0.7±0.30.7 ± 0.3 199.3±121.6199.3 ± 121.6 39.8±1.539.8 ± 1.5 271.6±179.6271.6 ± 179.6 25.3±1.425.3 ± 1.4 C-GAIL 49.1±89.449.1 ± 89.4 0.0±0.00.0 ± 0.0 155.6±122.6155.6 ± 122.6 7.3±0.87.3 ± 0.8 257.8±167.3257.8 ± 167.3 16.2±0.916.2 ± 0.9 C-SQIL 63.3±84.263.3 ± 84.2 0.4±0.10.4 ± 0.1 204.3±125.9204.3 ± 125.9 45.0±1.645.0 ± 1.6 330.4±194.4330.4 ± 194.4 46.7±1.646.7 ± 1.6 C-IQL 65.4±82.665.4 ± 82.6 0.1±0.10.1 ± 0.1 225.6±127.9225.6 ± 127.9 58.9±1.658.9 ± 1.6 322.1±181.6322.1 ± 181.6 32.4±1.432.4 ± 1.4 BC 5.6±105.95.6 ± 105.9 0.0±0.00.0 ± 0.0 14.0±150.214.0 ± 150.2 0.0±0.00.0 ± 0.0 8.0±184.68.0 ± 184.6 0.0±0.00.0 ± 0.0 GAIL 15.4±102.115.4 ± 102.1 0.0±0.00.0 ± 0.0 85.8±132.985.8 ± 132.9 0.0±0.00.0 ± 0.0 58.2±174.058.2 ± 174.0 0.0±0.00.0 ± 0.0 SQIL 1.3±107.51.3 ± 107.5 0.0±0.00.0 ± 0.0 0.0±151.50.0 ± 151.5 0.0±0.00.0 ± 0.0 1.1±186.31.1 ± 186.3 0.0±0.00.0 ± 0.0 IQL 0.0±108.10.0 ± 108.1 0.0±0.00.0 ± 0.0 16.1±153.516.1 ± 153.5 0.0±0.00.0 ± 0.0 0.0±186.60.0 ± 186.6 0.0±0.00.0 ± 0.0 (b) AntMaze-Large Table 7: Ablation over horizon length H∈500,1000,1500H∈\500,1000,1500\ on Confounded AntMaze, at the default 100%100\% confounding strength. Best result per column in bold. Normalized [Y]E[Y] is not comparable across horizons (see text). The prevalence of scalable causal algorithms over non-causal and less scalable algorithms is stable across horizons. On AntMaze-Medium, every causal method attains 6464–91%91\% success at all three horizons while every non-causal method remains near 0%0\%, with a single exception discussed below, and on AntMaze-Large the same ordering holds at every H under normalized [Y]E[Y]. At H=500H=500 on AntMaze-Large, success rates saturate near zero for all methods including the causal ones; this is likely due to successful expert traversals requiring 648±99648± 99 steps (Table 6). Normalized [Y]E[Y] is more informative in this case, with causal imitators’ higher values indicating that they progress further toward the goal than non-causal imitators even under an infeasible time budget. The clearest departure from the overall pattern is non-causal GAIL at H=1500H=1500 on Medium (26.5%26.5\%). Two factors plausibly contribute. First, GAIL’s training budget is specified in rounds and episodes (Appendix E), so its total environment interaction scales linearly with H and is largest in this configuration, whereas SQIL and IQ-Learn are capped at 2×1062× 10^6 steps throughout. Second, GAIL is the only on-policy algorithm among the non-causal baselines, and on-policy divergence minimization has been shown to partially recover from errors induced by unobserved contexts where off-policy conditional fitting latches onto them (swamy2022sequence). Even so, this partial recovery remains far below every causal method at the same horizon (80.280.2–87.2%87.2\%). 15%15\% 50%50\% 85%85\% 100%100\% Algorithm [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) C-BC 59.1±99.559.1 ± 99.5 80.6±1.280.6 ± 1.2 246.8±98.8246.8 ± 98.8 81.1±1.281.1 ± 1.2 248.5±99.3248.5 ± 99.3 81.6±1.281.6 ± 1.2 250.1±100.9250.1 ± 100.9 77.9±1.377.9 ± 1.3 C-GAIL 45.5±115.245.5 ± 115.2 76.6±1.376.6 ± 1.3 254.2±102.4254.2 ± 102.4 83.4±1.283.4 ± 1.2 240.0±110.0240.0 ± 110.0 79.5±1.379.5 ± 1.3 239.5±113.6239.5 ± 113.6 74.1±1.474.1 ± 1.4 C-SQIL 73.2±97.973.2 ± 97.9 86.5±1.186.5 ± 1.1 267.2±90.1267.2 ± 90.1 87.8±1.087.8 ± 1.0 256.3±99.4256.3 ± 99.4 82.6±1.282.6 ± 1.2 275.8±85.7275.8 ± 85.7 90.7±0.990.7 ± 0.9 C-IQL 67.5±102.367.5 ± 102.3 82.5±1.282.5 ± 1.2 256.1±102.2256.1 ± 102.2 83.4±1.283.4 ± 1.2 257.8±97.8257.8 ± 97.8 84.6±1.184.6 ± 1.1 257.1±101.3257.1 ± 101.3 84.3±1.184.3 ± 1.1 BC 0.0±108.90.0 ± 108.9 47.7±1.447.7 ± 1.4 136.5±113.7136.5 ± 113.7 30.6±1.430.6 ± 1.4 29.9±110.529.9 ± 110.5 0.0±0.00.0 ± 0.0 0.0±119.20.0 ± 119.2 0.0±0.00.0 ± 0.0 GAIL 61.1±104.361.1 ± 104.3 80.5±1.280.5 ± 1.2 152.9±128.5152.9 ± 128.5 40.0±1.540.0 ± 1.5 150.3±130.9150.3 ± 130.9 38.4±1.438.4 ± 1.4 10.4±112.210.4 ± 112.2 0.0±0.00.0 ± 0.0 SQIL 62.6±102.862.6 ± 102.8 80.2±1.380.2 ± 1.3 0.0±118.30.0 ± 118.3 0.0±0.00.0 ± 0.0 0.0±118.80.0 ± 118.8 0.0±0.00.0 ± 0.0 4.7±119.14.7 ± 119.1 0.0±0.00.0 ± 0.0 IQL 64.8±99.164.8 ± 99.1 80.7±1.280.7 ± 1.2 1.0±118.21.0 ± 118.2 0.0±0.00.0 ± 0.0 1.8±118.11.8 ± 118.1 0.0±0.00.0 ± 0.0 2.8±120.82.8 ± 120.8 0.0±0.00.0 ± 0.0 (a) AntMaze-Medium 15%15\% 50%50\% 85%85\% 100%100\% Algorithm [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) [Y]E[Y] SR (%) C-BC 43.7±128.943.7 ± 128.9 41.5±1.541.5 ± 1.5 189.1±127.3189.1 ± 127.3 39.9±1.539.9 ± 1.5 191.2±128.5191.2 ± 128.5 40.6±1.540.6 ± 1.5 199.3±121.6199.3 ± 121.6 39.8±1.539.8 ± 1.5 C-GAIL 2.9±129.62.9 ± 129.6 15.5±0.815.5 ± 0.8 154.2±138.8154.2 ± 138.8 25.9±1.225.9 ± 1.2 142.5±129.3142.5 ± 129.3 10.1±0.910.1 ± 0.9 155.6±122.6155.6 ± 122.6 7.3±0.87.3 ± 0.8 C-SQIL 57.3±131.057.3 ± 131.0 51.8±1.651.8 ± 1.6 207.8±136.1207.8 ± 136.1 54.6±1.654.6 ± 1.6 201.7±127.8201.7 ± 127.8 49.0±1.649.0 ± 1.6 204.3±125.9204.3 ± 125.9 45.0±1.645.0 ± 1.6 C-IQL 52.0±131.052.0 ± 131.0 46.8±1.646.8 ± 1.6 212.3±131.7212.3 ± 131.7 52.7±1.552.7 ± 1.5 204.1±129.8204.1 ± 129.8 51.4±1.651.4 ± 1.6 225.6±127.9225.6 ± 127.9 58.9±1.658.9 ± 1.6 BC 13.1±122.913.1 ± 122.9 18.4±1.218.4 ± 1.2 79.0±134.379.0 ± 134.3 0.0±0.00.0 ± 0.0 14.9±148.214.9 ± 148.2 0.0±0.00.0 ± 0.0 14.0±150.214.0 ± 150.2 0.0±0.00.0 ± 0.0 GAIL 0.0±127.60.0 ± 127.6 11.9±1.011.9 ± 1.0 120.0±124.0120.0 ± 124.0 6.5±0.46.5 ± 0.4 12.5±148.212.5 ± 148.2 0.0±0.00.0 ± 0.0 85.8±132.985.8 ± 132.9 0.0±0.00.0 ± 0.0 SQIL 23.7±121.323.7 ± 121.3 14.9±1.014.9 ± 1.0 1.7±151.11.7 ± 151.1 0.0±0.00.0 ± 0.0 0.0±152.10.0 ± 152.1 0.0±0.00.0 ± 0.0 0.0±151.50.0 ± 151.5 0.0±0.00.0 ± 0.0 IQL 25.0±121.325.0 ± 121.3 21.2±1.021.2 ± 1.0 0.0±151.80.0 ± 151.8 0.0±0.00.0 ± 0.0 0.0±151.90.0 ± 151.9 0.0±0.00.0 ± 0.0 16.1±153.516.1 ± 153.5 0.0±0.00.0 ± 0.0 (b) AntMaze-Large Table 8: Ablation over confounding strength ∈15%,50%,85%,100%∈\15\%,50\%,85\%,100\%\ on Confounded AntMaze, at the default horizon H=1000H=1000. Best result per column in bold. Normalized [Y]E[Y] is not comparable across strengths (see text). The confounding-strength sweep isolates the mechanism behind the causal methods’ advantage. Because causal variants exclude the compass W from their adjustment sets, and the strength parameter scales only the compass corruption, the learning problem faced by a causal imitator is identical at every strength; the causal rows (e.g., Causal IQ-Learn on Large: 46.846.8–58.9%58.9\%) therefore provide a scale against which the behavior of non-causal methods can be assessed (e.g. the higher variation of Causal GAIL on Large, 7.37.3–25.9%25.9\%, is consistent with its training instability at scale discussed in Section 3). Their smaller normalized [Y]E[Y] values in the 15%15\% rows reflect the improved worst-performing anchor at low strength rather than any degradation of the causal methods, whose success rates remain consistent. Non-causal methods, conversely, degrade monotonically as strength increases, and they collapse in a consistent order: SQIL and IQ-Learn reach 0%0\% success by 50%50\% strength on both tasks, BC follows (0%0\% by 85%85\% on Medium and by 50%50\% on Large), and GAIL persists longest (38.4%38.4\% at 85%85\% strength on Medium; 6.5%6.5\% at 50%50\% on Large). This phenomenon implies how directly each objective commits to the confounded conditional: BC fits P(xt∣tO)P(x_t ^O_t) pointwise; SQIL and IQ-Learn additionally propagate it through bootstrapped value targets across the horizon, amplifying the corrupted association; and GAIL, the only on-policy method, matches occupancy measures under its own rollouts rather than fitting the conditional directly, consistent with the greater robustness of on-policy divergence minimization to unobserved contexts (swamy2022sequence), although GAIL’s larger interaction budget (Appendix E) cannot be fully separated from this effect. At the lowest strength, conditioning on W becomes nearly costless and sometimes net-beneficial: 15%15\% confounding leaves the compass a mostly faithful heading sensor even after the runtime shift, and non-causal GAIL (80.5%80.5\%) exceeds Causal GAIL (76.6%76.6\%) on Medium while non-causal SQIL and IQ-Learn recover to within a few points of their causal counterparts (80.280.2 against 86.5%86.5\% and 80.780.7 against 82.5%82.5\%). This substantiates the observation of Figure 4 that the spurious proxy carries genuine predictive signal, and is consistent with the discussion in Appendix A: the bias resides in the conditional over the imitator’s observations, and its cost is governed by how far that conditional moves at runtime. Excluding W sacrifices this signal for invariance, a trade that is repaid as strength grows. The favorable regime for non-causal methods does not extend to the harder task, however: on Large at 15%15\%, Causal SQIL (51.8%51.8\%) still more than doubles the best non-causal result (21.2%21.2\%), as even mild per-step miscalibration compounds over the longer traversal. Relatedly, non-causal BC is anomalously weak at 15%15\% on Medium (47.7%47.7\% against roughly 80%80\% for the other non-causal methods): BC is the only algorithm that never interacts with the environment during training (Appendix E) and thus has no mechanism to recover once the mildly miscalibrated compass drifts it off the demonstration support, an interaction between confounding bias and compounding error, the two causes of distribution shift discussed in Appendix A. Finally, the low-strength rows provide further evidence for non-causal methods’ comparatively better HumanoidMaze results being an effect of weaker confounding influence. Appendix G Limitations Known causal graph. Our framework assumes that the causal diagram G is specified a priori. In practice, the graph must be elicited from domain expertise or estimated from data. Misspecification of G, such as missing an edge from a confounder to the action, can lead to invalid adjustment sets and biased policies. Integrating causal discovery from observational data, either as a preprocessing step or jointly with policy learning, remains an important open problem. Bounded temporal influence of confounders. The windowed approximation in Algorithm 1 assumes that the influence of any single confounder realization decays within k timesteps. This is justified by the MuJoCo physics of the environments we consider, but environments with long-range latent dependencies (e.g. persistent hidden goals, slowly drifting dynamics, or latent agent intentions in multi-agent settings) would violate this assumption. In such cases, alternative approximation strategies (e.g., hierarchical windows or attention-based aggregation over history) or exact methods on compressed representations would be required. No exploitation of reward structure. Our algorithms treat the reward as entirely latent and make no parametric assumptions about its form. Prior work on partial identification for CIL (ruan2024partial) has shown that incorporating reward priors can tighten bounds on the imitating policy and even enable the imitator to surpass expert performance. Combining Causal IQ-Learn with such reward priors is a natural extension that we leave to future work. Sensitivity to expert optimality. While Causal SQIL and Causal IQ-Learn successfully address the compounding error and credit assignment issues seen in Causal BC and Causal GAIL, they are more sensitive than BC to the quality and optimality of expert demonstrations. Because these off-policy algorithms utilize demonstrations to ground an implicit reward signal or a value function via soft Q-learning objectives, they are more susceptible to noise in high-dimensional state-action spaces where the expert signal may be weak. This dependency is clearly demonstrated in the HumanoidMaze-Large task (Table 1), where the expert itself is sub-optimal, achieving a success rate of only 7.0%7.0\%. While Causal BC maintains a success rate of 8.0%8.0\%, Causal IQ-Learn collapses to 3.0%3.0\% and Causal SQIL matches at 8.0%8.0\% despite both algorithms outperforming BC in other tasks where the expert is more consistent. These results suggest that when the expert is sub-optimal, the off-policy grounding of value functions is more easily corrupted than the supervised cloning objective, indicating that the scalability benefits of our proposed methods are most reliably realized when provided with a high-quality expert signal.