Paper deep dive
Robust Counterfactual Policy Optimisation via Nondeterministic Causal Models
Jessica Lally, Milad Kazemi, Nicola Paoletti, David Watson, Sander Beckers
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/5/2026, 4:38:32 AM
Summary
This paper introduces a framework for robust counterfactual policy optimization in sequential decision-making using Probabilistic Nondeterministic Structural Causal Models (PNSCMs). It addresses the limitation of existing methods that assume deterministic causal models by separating latent confounding from irreducible stochasticity. The authors propose a sensitivity analysis framework to bound the influence of unobserved global confounders and derive robust counterfactual policies. The approach is validated on a Sepsis treatment simulator, demonstrating that the derived policies improve upon observed suboptimal behaviors even when the confounder (diabetes status) is hidden.
Entities (6)
Relation Signals (5)
Diabetes → actsas → hidden global confounder
confidence 95% · diabetes status acts as a hidden global confounder.
PNSCM → handles → irreducible stochasticity
confidence 92% · PNSCMs ... allow for such inherent stochasticity.
PNSCM → separates → latent confounding
confidence 90% · properly separates latent confounding from irreducible stochasticity
Sepsis MDP → usedforvalidation → Robust Counterfactual Policy Optimization
confidence 90% · We validate our approach on the Sepsis MDP
Counterfactual Policy Optimization → uses → Sensitivity Analysis
confidence 88% · identifying robust counterfactual policies under a sensitivity analysis framework.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Counterfactual inference approaches for sequential decision-making typically assume deterministic causal models, where all randomness stems from latent variables. However, Markov Decision Processes (MDPs) are inherently stochastic. We address this by formalising counterfactual policy optimisation under probabilistic nondeterministic causal models, which properly separates latent confounding from irreducible stochasticity, and here propose a first practical optimisation problem for identifying robust counterfactual policies under a sensitivity analysis framework. We validate our approach on a sepsis treatment simulator, where diabetes status acts as a hidden global confounder.
Tags
Links
- Source: https://arxiv.org/abs/2608.02893v1
- Canonical: https://arxiv.org/abs/2608.02893v1
Trouble viewing inline? Open PDF directly →
Full Text
26,795 characters extracted from source content.
Expand or collapse full text
Robust Counterfactual Policy Optimisation via Nondeterministic Causal Models Jessica Lally King’s College London Milad Kazemi Durham University Nicola Paoletti King’s College London David Watson King’s College London Sander Beckers University College London Abstract Counterfactual inference approaches for sequential decision-making typically assume deterministic causal models, where all randomness stems from latent variables. However, Markov Decision Processes (MDPs) are inherently stochastic. We address this by formalising counterfactual policy optimisation under probabilistic nondeterministic causal models, which properly separates latent confounding from irreducible stochasticity, and here propose a first practical optimisation problem for identifying robust counterfactual policies under a sensitivity analysis framework. We validate our approach on a sepsis treatment simulator, where diabetes status acts as a hidden global confounder. 1 Introduction Counterfactual inference in sequential decision-making asks: given an observed sequence of states, actions, and outcomes, what would the outcome have been under a different policy? In safety-critical domains, such as healthcare, rerunning experiments under alternative policies is often infeasible or unethical, making counterfactual reasoning crucial for offline policy evaluation. Existing work applying counterfactual inference to MDPs [Oberst and Sontag, 2019, Tsirtsis et al., 2021, Killian et al., 2022, Tsirtsis and Rodriguez, 2024, Lally et al., 2026] assumes all randomness comes from unobserved latent variables, and therefore models MDPs as deterministic structural causal models (SCMs) [Pearl, 2009]. However, in practice, randomness can arise from fundamentally different sources: at one extreme, all randomness may arise from unobserved latent variables (the deterministic SCM setting); at the other, all randomness may be irreducible (e.g., from random number generators). Recently, probabilistic nondeterministic causal models (PNSCMs) [Beckers, 2026, 2025b] have been developed to allow for such inherent stochasticity. More commonly, however, both sources of randomness coexist: health outcomes usually depend both on unobserved latent factors, and on random differences in applying the treatment, or random individual-level genetic variation. Similarly, in the social sciences, variation in the outcome of policies is both due to latent factors and to individual-level variation that is represented as noise at the aggregate level. In these settings, we propose modelling the environment as a PNSCM where randomness arises from both latent variables and irreducible stochasticity, and bounding the influence of latent variables using a sensitivity-analysis approach similar to Kausik et al. [2024]. In this paper, we formalise counterfactual inference for MDPs under PNSCMs, and propose optimisation procedures for deriving robust counterfactual policies under confounding. We then evaluate these approaches on the Sepsis MDP [Oberst and Sontag, 2019], where diabetes acts as an unobserved global confounder, demonstrating that the derived policies are robust when evaluated on the true, fully-specified environment. 2 Background In this section, we introduce the necessary background on Markov decision processes and causal models. 2.1 Markov Decision Processes Markov decision processes (MDPs) model sequential decision-making under uncertainty, and are defined by a tuple (,,I,,ℛ,γ)(S,A,P_I,P,R,γ) where S is the state space, A is the action space, IP_I is the initial state distribution, :×→[0,1]P:S×A×S→[0,1] is the transition kernel, ℛ:×→ℝR:S×A is a reward function, and γ is a discount factor. The goal is to identify an optimal policy π:→π:S , that optimises the expected total discounted reward. A trajectory τ under policy π is a sequence τ=(s0,a0,s1,a1,…,s|τ|−1,a|τ|−1,s|τ|)τ=(s_0,a_0,s_1,a_1,…,s_|τ|-1,a_|τ|-1,s_|τ|) where s0∼Is_0 _I, at=π(st)a_t=π(s_t), and st+1∼(⋅∣st,at)s_t+1 (· s_t,a_t). Latent Variables and Confounding In many applications, the observed state cannot capture all factors influencing system dynamics, e.g., electronic health records may have missing information on patient demographics or underlying conditions. We consider settings where latent factors U affect transition dynamics, such that the true transitions P(s′∣s,a,u)P(s s,a,u) differ from the observed marginal P(s′∣s,a)P(s s,a). In this initial paper, we focus on global confounders, where the value of U is fixed throughout the trajectory, confounding all transitions [Kausik et al., 2024]. Since U is unobserved, its effect on transitions cannot be identified from observational data alone. Sensitivity analysis addresses this problem by bounding the degree to which U can influence the transitions. Existing work on sensitivity analysis of confounding in RL primarily focuses on off-policy evaluation [Bruns-Smith, 2021, Namkoong et al., 2020, Kausik et al., 2024, Bruns-Smith and Zhou, 2023]: estimating the expected value of a target policy, given a dataset of observed trajectories under an unknown behaviour policy. Here, we adapt this sensitivity framework for counterfactual policy analysis [Oberst and Sontag, 2019, Tsirtsis et al., 2021, Tsirtsis and Rodriguez, 2024, Lally et al., 2026]: given an observed trajectory, what might have happened under a different policy, and what policy would have been optimal? 2.2 Structural Causal Models Structural causal models (SCMs) [Pearl, 2009] provide a framework for counterfactual inference. An SCM =(,,ℱ,P())C=(U,V,F,P(U)) consists of observed variables V, unobserved variables U with joint distribution P()P(U), and structural equations (X=fX())∈ℱ(X=f_X(pa_X)) that determine the values of each X∈X as a function of its direct causes – parents – Xpa_X. Counterfactual inference proceeds by estimating P(∣)P(U ) given an observation =V=v, performing an intervention that modifies the structural equations, and evaluating the values of observed variables. Probabilistic nondeterministic SCMs (PNSCMs) [Beckers, 2025b] replace deterministic structural equations fXf_X with conditional probability distributions X(⋅∣X)P_X(· _X), thereby allowing probabilistic causal mechanisms that are not reducible to deterministic mechanisms over unobserved variables. A fully specified MDP together with a policy is an instance of a PNSCM. A key feature of counterfactual inference with PNSCMs is actualised refinement: given the values of the observed and unobserved variables that produce the actual setting (,)(u,v), actualised refinement replaces each conditional distribution XP_X by X(X,x)P_X^(pa_X,x), which behaves identically to XP_X for all inputs except that it deterministically returns the observed value x when given the observed parents Xpa_X. This expresses the nondeterministic semantics that all we learn from the actual setting is that the actual parents resulted in the actual child, and nothing else (see [Beckers, 2025a, b] for more details and motivation.) 3 Methodology Given an observed trajectory τ=(s0,a0,…,s|τ|)τ=(s_0,a_0,…,s_|τ|), our goal is to identify the optimal policy π that maximises the worst-case counterfactual value V(0,s0)V CF(0,s_0) under uncertainty over the unobserved global confounder U: V(t,s)=maxπminPt∈t,Δ V^CF(t,s)= _π _P CF_t CF, _t (1) s′∼Pt(⋅∣s,π(s))[ℛ(s,π(s))+γ⋅V(t+1,s′)] _s P CF_t(· s,π(s)) [R(s,π(s))+γ· V^CF(t+1,s ) ] where t,ΔP CF, _t is the set of feasible counterfactual transition probabilities at time t, whose constraints are defined below. 3.1 Sensitivity Constraints Similar to the odds-ratio model used in [Bruns-Smith, 2021, Bruns-Smith and Zhou, 2023, Kausik et al., 2024, Bennett et al., 2024], we parametrise the influence of U via a sensitivity parameter Δ≥1 ≥ 1, which constrains how much P(s′∣s,a,u)P(s s,a,u) can differ from the marginal P(s′∣s,a)P(s s,a): 1Δ≤odds(P(S′=s′∣S=s,A=a,U=u))odds(P(S′=s′∣S=s,A=a))≤Δ 1 ≤ odds(P(S =s S=s,A=a,U=u))odds(P(S =s S=s,A=a))≤ where odds(p)=p/(1−p)odds(p)=p/(1-p). When Δ=1 =1, this is equivalent to a fully specified PNSCM (i.e., no latent U). As Δ→∞ →∞, the bounds on P(S′∣S,A,U)P(S S,A,U) approach [0,1][0,1], representing maximum confounding. Δ can be global – i.e., one value – or state-action dependent Δ(s,a) (s,a) [Bennett et al., 2024]. 3.2 Choosing Δ A key challenge in sensitivity analysis is choosing Δ . Most existing work either delegates this to a domain expert, based on their understanding of the environment dynamics (e.g., [Bruns-Smith, 2021, Bennett et al., 2024]), or evaluates over a range from Δ=1 =1 (no confounding) to some large value (strong confounding), assessing how the estimated value function changes. If the target policy consistently outperforms the observed policy, this demonstrates robustness. Without domain knowledge, a more principled approach is to calibrate Δ from the observed state variables [Bruns-Smith and Zhou, 2023] by hiding each variable in turn and computing the resulting odds ratio between the observed and marginalised transitions. Under the assumption that no unobserved variable influences transitions more than the strongest observed variable, this provides a justifiable upper bound on Δ(s,a) (s,a). Formally, for each variable i, let s−is^-i denote the state omitting variable i, and let nin_i denote the number of distinct values variable i can take. The marginalised transition distribution when variable i is hidden is: P(St+1=s′∣St−i=s−i,At=a)=∑j=0ni−1wi(j∣s−i)⋅ P(S_t+1=s S^-i_t=s^-i,A_t=a)= _j=0^n_i-1w_i(j s^-i)· (2) P(s′∣s−i,Si=j,a) P(s s^-i,S^i=j,a) where wi(j∣s−i)=P(Si=j∣s−i)w_i(j s^-i)=P(S^i=j s^-i) is the conditional probability that variable i takes value j given the projected state s−is^-i, estimated from the stationary distribution of the MDP. The sensitivity Δ(s,a) (s,a) can be computed as: Δi(s,a)=maxs′∈P(s′∣s,a)>0max(ri(s,a,s′),ri(s,a,s′)−1) _i(s,a)= _ subarraycs \\ P(s s,a)>0 subarray (r_i(s,a,s ),\ r_i(s,a,s )^-1 ) where ri(s,a,s′)=odds(P(St+1=s′∣s,a))odds(P(St+1=s′∣s−i,a))r_i(s,a,s )= odds (P(S_t+1=s s,a) )odds (P(S_t+1=s s^-i,a) ). The overall sensitivity for each state-action pair is Δ(s,a)=maxiΔi(s,a). (s,a)= _i _i(s,a). Additionally, a scaling factor Γ≥0 ≥ 0 can be applied to calibrate the sensitivity, giving a final sensitivity estimate of Γ⋅Δ(s,a) · (s,a) [McClean et al., 2025]. This provides an interpretable parameter to assess how counterfactual estimates and policies vary under stronger (Γ>1)( >1) or weaker (Γ<1)( <1) confounding than the proxy Δ(s,a) (s,a) suggests. 3.3 Actualised Refinement Given an observed transition (st,at,st+1)(s_t,a_t,s_t+1), if we were to take the same action ata_t in sts_t in some counterfactual world, the next state would be the same st+1s_t+1 as well. The reason is that no matter the unobserved value U=uU=u, it remains constant across worlds. Still, given that we do not know the actual value U=uU=u, the counterfactual transition probability must reflect this posterior uncertainty: with probability P(U∣τ)P(U τ), U=uU=u generated the trajectory and the transition to st+1s_t+1 is certain; otherwise, with probability 1−P(U∣τ)1-P(U τ), the counterfactual probability equals the estimated probability P~(s′∣st,at,u) P(s s_t,a_t,u) as chosen by the optimisation. As a result, the counterfactual transition probability is: PtCF(s′|s,a,u)=P(U=u|τ)⋅[s′=st+1]+(1−P(U=u|τ))⋅P~(s′|s,a,u)if (s,a)=(st,at)P~(s′|s,a,u)otherwiseP^CF_t(s |s,a,u)= cases aligned &P(U=u|τ)·1[s =s_t+1]\\ & +(1-P(U=u|τ))· P(s |s,a,u) aligned\\ (s,a)=(s_t,a_t)\\[4.0pt] P(s |s,a,u) cases (One exception is when Δ=1 =1: this corresponds to a setting with no confounder U, so PtCF(st+1∣st,at)=1P^CF_t(s_t+1 s_t,a_t)=1). Mean Difference δ1¯=VΔ,Γ(0,s0)−VΔ∗(0,s0) _1=V^CF_ , (0,s_0)-V^CF_ ^*(0,s_0)Mean Difference δ2¯=V,πΔ,Γ∗(0,s0)−V,πΔ∗(0,s0) _2=V^CF,π^*_ , _ full(0,s_0)-V^CF,π^*_ *_ full(0,s_0)No scaling (Γ=1 =1) 00.40.40.80.81.21.21.61.622−5-50551010Scale Γ δ1¯ _1 (a) Diabetic Trajectories 00.40.40.80.81.21.21.61.622−10-10−5-50Scale Γ δ¯2 δ_2 (b) Nondiabetic Trajectories Figure 1: Mean difference between counterfactual policy values and the oracle (under true diabetes sensitivity Δ∗ ^*), averaged across 1010 randomly sampled suboptimal trajectories. 3.4 Optimisation Procedures Due to the coupling of P(S′∣S,A,U)P(S S,A,U) between all state-action pairs (s,a)(s,a) and all time steps t, identifying globally optimal solutions is challenging on all but toy examples. In this work, we therefore consider a decoupled approach: we simplify the optimisation problem by removing the time-homogeneity assumption and the coupling between state-action pairs (s,a)(s,a), treating each (t,s,a)(t,s,a) triple independently. For each (t,s)(t,s) we solve: V(t,s)=maxa[R(s,a)+γ⋅minP~(st+1∣st,at,u)uP~(s′∣s,a,u)u,s′ V CF(t,s)= _a [R(s,a)+γ· _ subarrayc\ P(s_t+1 s_t,a_t,u)\_u\\ \ P(s s,a,u)\_u,s subarray ∑uP(U=u|τ)∑s′PtCF(s′|s,a,u)⋅V(t+1,s′)] _uP(U=u|τ) _s P^CF_t(s |s,a,u)· V CF(t+1,s ) ] where the posterior P(U=u∣τ)P(U=u τ) is: P(U=u∣τ)=P(U=u)∏tP~(st+1∣st,at,u)∑u′P(U=u′)∏tP~(st+1∣st,at,u′)P(U=u τ)= P(U=u) _t P(s_t+1 s_t,a_t,u) _u P(U=u ) _t P(s_t+1 s_t,a_t,u ) Since P(U∣τ)P(U τ) is a continuous function of the adversary’s choice of transition probabilities, we compute its feasible range by setting each observed transition P~(st+1∣st,at,u) P(s_t+1 s_t,a_t,u) to its extremes and treat it as an optimisation variable. The adversary therefore jointly optimises over P(U∣τ)P(U τ) and P~(⋅|s,a,u)u\ P(·|s,a,u)\_u at each (t,s,a)(t,s,a) block, subject to: ∑uP(U=u)⋅P~(s′|s,a,u)=P(s′|s,a),∀s′ _uP(U=u)· P(s |s,a,u)=P(s |s,a), ∀ s ∑s′P~(s′|s,a,u)=1,∀u _s P(s |s,a,u)=1, ∀ u 1Δ⋅P(s′|s,a)≤P~(s′|s,a,u)≤Δ⋅P(s′|s,a),∀s′,u 1 · P(s |s,a)≤ P(s |s,a,u)≤ · P(s |s,a), ∀ s ,u Note that at the observed transition (s,a)=(st,at)(s,a)=(s_t,a_t), P(U=u∣τ)P(U=u τ) appears both as a posterior weight and inside PtCFP^CF_t, creating an adversarial tradeoff that makes the optimisation non-convex; we use Gurobi to guarantee a globally optimal solution. The decoupling makes the problem tractable but produces a conservative lower bound on VCF(0,s0)V^CF(0,s_0): the decoupled adversary has strictly more freedom than the true coupled adversary (which must use the same P~(s′∣s,a,u) P(s s,a,u) and P(U∣τ)P(U τ) across all (t,s,a)(t,s,a) blocks), and since greater adversarial freedom can only decrease the value, this guarantees a valid lower bound. Incorporating Time-homogeneity To obtain a tighter bound, we could incorporate time-homogeneity by requiring a single P(s′∣s,a,u)P(s s,a,u) for each (s,a)(s,a) pair across all timesteps. This creates a trilinear coupling between the posterior P(U∣τ)P(U τ), transition probabilities P(s′∣s,a,u)P(s s,a,u), and value function, which can be solved using gradient descent. Since gradient descent finds only local minima, we cannot guarantee we will identify the true pessimistic value function. However, the decoupled and gradient descent solutions will together bound the true pessimistic value function. 4 Sepsis Simulation We demonstrate our approach on a sepsis treatment simulator [Oberst and Sontag, 2019]. Each state consists of four vital signs (heart rate, blood pressure, oxygen concentration, and glucose levels), categorised as low, normal, or high. At each step, three treatments can be toggled on or off. Rewards range from −10-10 (death) to 1010 (discharge) based on the number of out-of-range vital signs. The full transition matrix is P(S′∣S,A,U)P(S S,A,U), where U∈0,1U∈\0,1\ indicates whether the patient is diabetic, which acts as a global latent confounder. In this population, 20%20\% of patients are diabetic. We evaluate our approach on the marginalised transition matrix (where diabetes is hidden), given observed trajectories from a suboptimal policy (which acts optimally with probability 0.50.5, and randomly otherwise). We use the maximum sensitivity of the observed vital signs Δ(s,a) (s,a) (see Section 3.2) as a proxy for the unobserved diabetes sensitivity, and an optional scaling parameter Γ . For each trajectory, we derive the optimal pessimistic counterfactual policy πΔ∗π^*_ that maximises the worst-case counterfactual value function VΔ,Γ(0,s0)V CF_ , (0,s_0), and evaluate its performance on the fully-observed MDP (where the diabetes state is known) to obtain V,πΔ∗(0,s0)V CF,π^*_ _ full(0,s_0). We evaluate our approach along two axes: first, whether πΔ∗π^*_ improves on the observation; and second, how closely our estimated VΔ,Γ(0,s0)V CF_ , (0,s_0) and policies match those obtained under the true diabetes sensitivity Δ∗ ^*. Improvement over Observation Table 1 shows that our approach can identify counterfactual policies that robustly improve on the observed suboptimal behaviour when evaluated on the fully-observed MDP. The estimated values VΔ,Γ(0,s0)V CF_ , (0,s_0) underestimate this improvement, particularly for nondiabetic trajectories as the Δ(s,a) (s,a) proxy overestimates the confounding effect. This is expected, as the marginalised transition matrix already closely reflects nondiabetic dynamics (since P(U=0)=0.8)P(U=0)=0.8). Table 1: Mean difference between counterfactual policy values and observed discounted return G0G_0 (γ=0.9γ=0.9), across 1010 suboptimal diabetic and nondiabetic trajectories. Estimated reports the mean difference VΔ,Γ(0,s0)−G0V CF_ , (0,s_0)-G_0; Actual reports the mean difference V,πΔ∗(0,s0)−G0V CF,π^*_ _ full(0,s_0)-G_0. Diabetic Nondiabetic Estimated Actual Estimated Actual 3.4±10.93.4± 10.9 7.4±10.77.4± 10.7 −0.50±9.1-0.50± 9.1 5.0±7.45.0± 7.4 Comparison to Diabetes Sensitivity Figures 1 and 1 assess how well our proxy Δ(s,a) (s,a) approximates the true diabetes sensitivity Δ∗ ^* (the oracle). Essentially, δ1¯ _1 (the blue curve) reflects how close our estimated counterfactual values are to the oracle, and δ2¯ _2 (the red curve) reflects how close the policies derived from our proxy Δ(s,a) (s,a) perform relative to the oracle in practice. The oracle represents the most accurate estimates achievable by our robust pessimistic approach, if we knew the true sensitivity Δ∗ ^*. Since Δ(s,a)≥1 (s,a)≥ 1, we clamp Γ⋅Δ(s,a) · (s,a) to a minimum of 11 (so Γ=0 =0 corresponds to a global sensitivity Δ=1 =1, i.e., no confounding). At Γ=0 =0, δ1¯>0 _1>0 (and is much greater than 0 for diabetic trajectories), confirming that ignoring confounding leads to inaccurate estimates. For diabetic trajectories, Γ=1.0 =1.0 minimises both δ1¯ _1 and δ2¯ _2, demonstrating that the unscaled proxy reliably estimates the true diabetes sensitivity. For nondiabetic trajectories, δ1¯ _1 is strongly negative at Γ=1 =1, and approaches 0 as we reduce Γ , showing our estimates are conservative. This is expected, as the marginalised transition matrix already closely reflects nondiabetic dynamics, so the true confounding effect for nondiabetics is weaker than the proxy suggests. However, crucially, δ2¯ _2 remains close to 0 across 0<Γ≤10< ≤ 1, suggesting that despite the conservative value estimates, we recover policies close to the oracle. 5 Conclusion In this paper, we introduced a framework for counterfactual policy optimisation under global confounding and demonstrated its robustness on the Sepsis MDP. Future work will explore extending this framework to settings with different types of latent variables (e.g., memoryless confounders, and confounders that persist over a subset of the time steps) and evaluating the gradient descent-based optimisation method to obtain more accurate counterfactual value estimates. References Beckers [2025a] Sander Beckers. Causal counterfactuals reconsidered. arXiv preprint arXiv:2512.12804, 2025a. Beckers [2025b] Sander Beckers. Nondeterministic causal models. In Biwei Huang and Mathias Drton, editors, Proceedings of the Fourth Conference on Causal Learning and Reasoning, volume 275 of Proceedings of Machine Learning Research, pages 1532–1554. PMLR, 07–09 May 2025b. URL https://proceedings.mlr.press/v275/beckers25b.html. Beckers [2026] Sander Beckers. Large language models as nondeterministic causal models. In Proceedings of the 23rd International Conference on Principles of Knowledge Representation and Reasoning, 2026. Bennett et al. [2024] Andrew Bennett, Nathan Kallus, Miruna Oprescu, Wen Sun, and Kaiwen Wang. Efficient and sharp off-policy evaluation in robust markov decision processes. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 112962–113000. Curran Associates, Inc., 2024. 10.52202/079017-3590. URL https://doi.org/10.52202/079017-3590. Bruns-Smith and Zhou [2023] David Bruns-Smith and Angela Zhou. Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders, 2023. URL https://arxiv.org/abs/2302.00662. Bruns-Smith [2021] David A Bruns-Smith. Model-free and model-based policy evaluation when causality is uncertain. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1116–1126. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/bruns-smith21a.html. Kausik et al. [2024] Chinmaya Kausik, Yangyi Lu, Kevin Tan, Maggie Makar, Yixin Wang, and Ambuj Tewari. Offline policy evaluation and optimization under confounding. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1459–1467. PMLR, 02–04 May 2024. URL https://proceedings.mlr.press/v238/kausik24a.html. Killian et al. [2022] Taylor W Killian, Marzyeh Ghassemi, and Shalmali Joshi. Counterfactually guided policy transfer in clinical settings. In Conference on Health, Inference, and Learning, pages 5–31. PMLR, 2022. Lally et al. [2026] Jessica Lally, Milad Kazemi, and Nicola Paoletti. Robust counterfactual inference in markov decision processes. AAMAS ’26, page 1527–1535, Richland, SC, 2026. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400723179. 10.65109/TXUQ4572. URL https://doi.org/10.65109/TXUQ4572. McClean et al. [2025] Alec McClean, Zach Branson, and Edward H. Kennedy. Calibrated sensitivity models, 2025. URL https://arxiv.org/abs/2405.08738. Namkoong et al. [2020] Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18819–18831. Curran Associates, Inc., 2020. URL https://proceedings.neurips.c/paper_files/paper/2020/file/da21bae82c02d1e2b8168d57cd3fbab7-Paper.pdf. Oberst and Sontag [2019] Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881–4890. PMLR, 2019. Pearl [2009] Judea Pearl. Causality. Cambridge University Press, 2nd nd edition, 2009. 10.1017/CBO9780511803161. Tsirtsis and Rodriguez [2024] Stratis Tsirtsis and Manuel Rodriguez. Finding counterfactually optimal action sequences in continuous state spaces. Advances in Neural Information Processing Systems, 36, 2024. Tsirtsis et al. [2021] Stratis Tsirtsis, Abir De, and Manuel Rodriguez. Counterfactual explanations in sequential decision making under uncertainty. Advances in Neural Information Processing Systems, 34:30127–30139, 2021.