Paper deep dive
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving
Mahmoud Selim, Cristina Cipriani, Karl Henrik Johansson
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/11/2026, 3:10:21 AM
Summary
The paper introduces Interaction-Constrained Drive Policy (ICDP), an offline reinforcement learning framework for autonomous driving that addresses 'interaction distribution shift' (IDS). Unlike traditional methods that only constrain the ego vehicle's actions, ICDP explicitly controls the compatibility between the ego trajectory and surrounding agents' behavior using contrastive density-ratio estimation. This approach improves policy performance in interaction-critical scenarios without requiring explicit joint-density modeling or simulator rollouts.
Entities (7)
Relation Signals (6)
ICDP → addresses → Interaction Distribution Shift
confidence 96% · ICDP... explicitly controls interaction-level distribution shift.
ICDP → uses → Contrastive Density-Ratio Estimation
confidence 95% · We recover the latter through contrastive density-ratio estimation... in ICDP.
ICDP → evaluatedon → nuPlan
confidence 93% · Closed-loop evaluations on nuPlan... show that ICDP suppresses high-value yet interaction-unsupported trajectory selections
ICDP → evaluatedon → InterPlan
confidence 93% · Closed-loop evaluations on... Interplan... show that ICDP... improves performance
Offline Reinforcement Learning → suffersfrom → Distribution Shift
confidence 90% · A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data
ICDP → isbasedon → Diffusion Policy
confidence 88% · ICDP combines a trajectory-level diffusion policy with chunk-level value learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy's own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as \emph{interaction distribution shift} (IDS), and introduce \emph{Interaction-Constrained Drive Policy} (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.09763v1
- Canonical: https://arxiv.org/abs/2610.09763v1
Trouble viewing inline? Open PDF directly →
Full Text
69,343 characters extracted from source content.
Expand or collapse full text
Beyond Policy Support: Interaction Constrained Offline Reinforcement Learning for Autonomous Driving Mahmoud Selim Email: textsuperscript1TRATON@se.traton.com Affiliation: KTH Royal Institute of Technology, mase2, kallej@kth.se Cristina Cipriani Email: mahmoud.selim@se.traton.com Karl H. Johansson Email: cristina.cipriani@se.traton.com Affiliation: KTH Royal Institute of Technology, mase2, kallej@kth.se Abstract Offline reinforcement learning enables reward-driven policy improvement from fixed datasets without requiring online exploration, making it particularly attractive in safety-critical domains. A central challenge, however, is distribution shift: policy optimization may favor actions that are weakly supported by the offline data, rendering value estimates unreliable. Existing approaches primarily control this shift in the policy’s own action space. In interactive environments such as autonomous driving, this can be insufficient: a candidate ego trajectory may remain well supported under the marginal behavior distribution while being poorly supported jointly with the surrounding-agent behavior observed in the logged interaction. We refer to this degradation in interaction support as interaction distribution shift (IDS), and introduce Interaction-Constrained Drive Policy (ICDP), an offline reinforcement learning framework that explicitly controls interaction-level distribution shift. Starting from the joint data distribution over ego and surrounding-agent futures, we show that joint-support degradation decomposes exactly into an ego-support component and a residual interaction-support component. We recover the latter through contrastive density-ratio estimation, isolating interaction compatibility without explicit joint-density modeling, surrounding-agent prediction, or rollouts in reactive simulators or learned world models during policy optimization. Closed-loop evaluations on nuPlan, Interplan and real-world truck experiments show that ICDP suppresses high-value yet interaction-unsupported trajectory selections and improves performance in interaction-critical driving scenarios. Project webpage: https://mahmoud-selim.github.io/ICDP/ 1 Introduction Autonomous driving requires motion planners that can produce safe, comfortable, and efficient behavior while continuously interacting with other road users. Classical rule- and optimization-based systems provide explicit structure and constraint handling, but often rely on carefully engineered objectives, interaction models, and scenario-specific design, making it difficult to scale their behavior across the long tail of complex traffic situations (Fan et al., 2018; Hawke et al., 2020; Dauner et al., 2023). Learning-based planning offers a compelling alternative by shifting this complexity from manual design to data. Imitation learning (IL), in particular, has enabled driving policies to acquire realistic behavior from large-scale demonstrations (Bojarski et al., 2016; Chitta et al., 2022; Cheng et al., 2024b), while recent diffusion- and flow-based planners further capture rich multimodal trajectory distributions and generate diverse, high-quality motion plans (Zheng et al., 2025; Huang et al., 2023; Cheng et al., 2024a). Yet greater generative expressiveness does not remove a fundamental limitation of imitation: the policy is still optimized to reproduce the behavior contained in the data, rather than to improve it according to long-horizon driving objectives. Consequently, biases and suboptimal decisions in the demonstrations can be inherited by the learned planner, motivating reward-driven policy optimization beyond imitation (Bansal et al., 2018; Cheng et al., 2024b). Reinforcement learning (RL) provides a natural mechanism beyond imitation by optimizing policies directly for long-horizon driving objectives. Applying RL to autonomous driving, however, is difficult: online methods require extensive closed-loop interaction with sufficiently realistic environments (Kendall et al., 2019; Yang et al., 2026; Zhang et al., 2025), while model-based approaches shift this burden to learned dynamics or world models, whose inaccuracies can propagate into policy optimization (Wang et al., 2025; Yang et al., 2026). Offline RL offers an appealing alternative by learning exclusively from previously collected driving data, avoiding repeated environment interaction while using bootstrapped value learning to propagate reward beyond the demonstrated behavior (Levine et al., 2020; Noguchi and Yamamoto, 2026). This benefit comes with a fundamental challenge: distribution shift. As policy optimization moves toward actions that are weakly represented in the dataset, the critic must extrapolate beyond its training support and can assign unreliable values to out-of-distribution behavior. Existing offline RL methods therefore control policy improvement through conservative value estimation, in-sample learning, or explicit regularization toward the behavior distribution (Kumar et al., 2020; Kostrikov et al., 2021; Peng et al., 2019; Wang et al., 2022; Fang et al., 2025; Huang et al., 2025). Yet these mechanisms largely characterize support in the policy’s own action space. In interactive driving, this is not enough: an ego trajectory can itself be well supported by the data while being poorly supported jointly with the surrounding-agent evolution observed in the same interaction. We refer to this loss of interaction support as interaction distribution shift (IDS). This raises the central question of our work: How can an offline driving policy improve beyond the demonstrations while remaining supported not only as an ego trajectory, but as part of the multi-agent interaction represented in the data? We answer this question with ICDP (Interaction-Constrained Drive Policy), an offline RL framework for reward-driven improvement preserving support at the level of multi-agent interactions. ICDP combines a trajectory-level diffusion policy with chunk-level value learning to evaluate the long-horizon utility of candidate ego trajectories. Its key idea is to go beyond conventional ego-trajectory regularization and explicitly control how policy updates alter the support of the logged ego–agent interaction. We show that the degradation of joint interaction support decomposes exactly into an ego-support term and a residual interaction-support term corresponding to interaction distribution shift. Crucially, the latter can be recovered through contrastive density-ratio estimation, allowing ICDP to assess whether a candidate ego trajectory remains compatible with the surrounding-agent behavior represented in the data without explicitly modeling the high-dimensional joint trajectory density, predicting agent responses, or performing reactive simulator or world-models rollouts during policy optimization. Our contributions are: • We identify and formalize interaction distribution shift (IDS), a source of distribution shift specific to interactive offline RL: an ego trajectory may remain well supported marginally while losing support jointly with the surrounding-agent evolution. • We derive a contrastive density-ratio estimator whose residual score recovers the interaction-support component of IDS without explicit joint-density estimation or prediction of surrounding-agent futures, and integrate this constraint with chunk-level value learning and diffusion-policy optimization in ICDP. • We evaluate ICDP in closed-loop driving scenarios on nuPlan and Interplan, as well as in real-world truck experiments, demonstrating its ability to suppress high-value but interaction-unsupported trajectory selections and improve policy performance in challenging scenarios. 2 Background & Related Works Learning-based and generative motion planning. Imitation learning (IL) has become a dominant paradigm for autonomous-driving planning, progressing from rasterized representations and convolutional architectures Bojarski et al. (2016); Codevilla et al. (2019) to structured vector representations and transformers that better capture scene context and interactions Liang et al. (2020); Casas et al. (2021); Chitta et al. (2022); Hu et al. (2023); Jiang et al. (2023); Cheng et al. (2024b). More recently, diffusion-based generative planners have enabled multimodal trajectory generation and flexible inference-time guidance Zheng et al. (2025); Liao et al. (2025). While these advances increase policy expressiveness, improvement remains largely driven by demonstrated behavior. In contrast, we use a generative trajectory policy as the foundation for reward-driven improvement from offline interaction data. Reinforcement learning for autonomous driving. RL complements imitation-based planning by optimizing driving policies directly for long-horizon reward. Model-free approaches learn driving behavior through closed-loop environment interaction Toromanoff et al. (2020), while model-based RL improves training efficiency by learning environment dynamics and optimizing policies through imagined rollouts Li et al. (2024); Wang et al. (2025). Recent work has further demonstrated the scalability of RL-based planning: CarPlanner combines autoregressive trajectory generation with expert-guided rewards for large-scale policy optimization Zhang et al. (2025), while CaRL scales PPO with simple reward design across CARLA and nuPlan Jaeger et al. (2025). Interaction-aware RL has also incorporated explicit intent inference and multi-agent policy learning to reason about surrounding drivers Wu et al. (2023). Collectively, these methods demonstrate the potential of reward-driven planning, but typically obtain counterfactual feedback through repeated environment interaction or learned dynamics. In contrast, our setting considers policy improvement entirely from fixed driving logs, without simulator interaction or model-generated traffic evolution during optimization. Offline RL and support constraints. Offline RL learns reward-maximizing policies from fixed datasets, but distribution shift can cause value overestimation on poorly supported actions. Existing methods mitigate this through conservative value estimation, behavior or support constraints, and in-sample policy extraction Kumar et al. (2020); Kumar et al. (2019); Fujimoto and Gu (2021); Kostrikov et al. (2021). These ideas have been extended to diffusion policies Wang et al. (2022); Fang et al. (2025); Gao et al. (2025); Selim et al. (2026) and temporally extended actions Li et al. (2026). In autonomous driving, offline RL has been studied through model-based, sequence-modeling, traffic-generation, and behavior-regularized approaches Diehl et al. (2021); Li et al. (2023); Rowe et al. (2024); Noguchi and Yamamoto (2026), including explicit support constraints for lane changing Huang et al. (2025). Offline multi-agent methods similarly address joint-action distribution shift Nguyen et al. (2025), but optimize multiple learning agents. In contrast, ICDP optimizes only the ego policy, treating surrounding traffic as logged environment evolution, and decomposes joint interaction-support degradation into complementary ego-trajectory and conditional interaction-support components. 3 Problem Formulation We formulate autonomous driving planning as a discounted Markov decision process (MDP) ℳ=(,,r,ρ,p,γ)M=(S,A,r,ρ,p,γ), where st∈s_t denotes the current driving scene, at∈a_t is the primitive ego control, r(s,a)r(s,a) is the reward function, ρ(s)ρ(s) the initial state distribution, p(s′|s,a)p(s |s,a) the transition dynamics, and γ∈[0,1)γ∈[0,1) the discount factor. Rather than learning over isolated controls, we operate in a temporally extended action space and define an ego-action chunk τe,t=(at,…,at+H−1)∈H, _e,t=(a_t,…,a_t+H-1) ^H, (1) over a committed horizon H. Accordingly, the policy and critic are defined as πθ(τe,t∣st), _θ( _e,t s_t), and Qϕ(st,τe,t),Q_φ(s_t, _e,t), where the critic estimates the return obtained by executing the complete chunk. The offline dataset is represented as =st,τe,tβ,τa,tβ,Rt(H),st+H,D= \s_t,\, _e,t^β,\, _a,t^β,\,R_t^(H),\,s_t+H \, (2) Where τe,tβ _e,t^β and τa,tβ _a,t^β are the logged ego-action chunk the corresponding surrounding-agent futures, and Rt(H)=∑k=0H−1γkrt+k.R_t^(H)= _k=0^H-1γ^kr_t+k. The critic is trained a the chunk-level target: yt=Rt(H)+γHQϕ¯(st+H,τ^e,t+H),τ^e,t+H∼πθ(⋅∣st+H).y_t=R_t^(H)+γ^HQ_ φ (s_t+H, τ_e,t+H ), τ_e,t+H _θ(· s_t+H). (3) where Qϕ¯Q_ φ is a delayed target critic. Because QϕQ_φ conditions on the complete action sequence that generated the intermediate rewards, this backup avoids the off-policy bias incurred when an H-step return is applied to a single-action critic Li et al. (2026). The surrounding-agent trajectories remain part of the observed environment evolution and are later used to characterize interaction support. A standard behavior-regularized offline RL objective improves the policy with respect to the learned critic while restricting its deviation from the data distribution: maxθ _θ s∼dτ^e∼πθ(⋅∣s)[Qϕ(s,τ^e)] _ subarraycs d_D\\ τ_e _θ(· s) subarray [Q_φ(s, τ_e) ] s.t. s∼d[D(πθ(⋅∣s),β(⋅∣s))]≤ϵe, _s d_D [D\! ( _θ(· s),β(· s) ) ]≤ _e, (4) where d_D is the empirical state distribution and D denotes a suitable divergence or behavior-regularization measure. While this constraint limits distribution shift in the ego-action space, it does not ensure that a candidate ego trajectory remains compatible with the surrounding-agent evolution represented in the same driving interaction. In particular, high ego support, p(τ^e∣s)p_D( τ_e s), does not necessarily imply high joint support, p(τ^e,τaβ∣s)p_D( τ_e, _a^β s). Given the interactive nature of driving, our objective is therefore to maximize long-horizon return while controlling distribution shift in the joint ego–agent interaction space p(τ^e,τaβ∣s)p_D( τ_e, _a^β s). 4 Methodology In this section, we introduce ICDP, an offline RL framework for diffusion-based motion planning that improves long-horizon driving performance while preserving support at the level of multi-agent interactions. ICDP combines chunk-level value learning with interaction-aware policy regularization, explicitly controlling both deviation from the logged ego-trajectory distribution and interaction distribution shift (IDS) induced by policy improvement. We first present the overall actor–critic framework, then derive a tractable characterization of IDS together with a contrastive estimator, and finally incorporate the resulting interaction constraint into diffusion-policy optimization. 4.1 ICDP Overview ICDP improves a diffusion-based driving policy from fixed interaction logs while constraining policy improvement to remain supported by the multi-agent interactions represented in the dataset. As illustrated in Fig. 1, the framework consists of three components: a scene representation shared in structure across the actor and critic, a diffusion actor with an ensemble of chunk-level critics for value-guided policy improvement, and an interaction-support module that evaluates the compatibility of policy-generated ego trajectories with the logged surrounding-agent evolution. The actor proposes candidate ego trajectories, while the critic and interaction-support estimator operate on the corresponding clean trajectories to provide value and support signals for policy optimization. Figure 1: Overview of ICDP. The logged scene is encoded into structured scene features used by the diffusion actor and critic. The actor generates candidate ego trajectories, which are evaluated by a pessimistic chunk-level critic and by the interaction-support module. The latter combines a joint ego–agent classifier with an ego-only classifier; their residual score isolates interaction-specific support and yields the IDS estimate used to constrain policy improvement. Given a logged interaction (s,τeβ,τaβ)∼(s, _e^β, _a^β) , where τeβ _e^β and τaβ _a^β denote the logged ego and surrounding-agent trajectories, respectively, the policy generates a candidate ego-action chunk τ^e∼πθ(⋅∣s). τ_e _θ(· s). (5) The candidate is evaluated by an ensemble of critics Qϕk(s,τe)k=1K\Q_ _k(s, _e)\_k=1^K. To reduce exploitation of uncertain value estimates during policy improvement, we use the lower-confidence-bound estimate QLCB(s,τe)=1K∑k=1KQϕk(s,τe)−κQStdk[Qϕk(s,τe)],Q_LCB(s, _e)= 1K _k=1^KQ_ _k(s, _e)- _QStd_k [Q_ _k(s, _e) ], (6) where κQ≥0 _Q≥ 0 controls the degree of pessimism. Maximizing value alone, however, does not guarantee reliable offline policy improvement. A candidate ego trajectory may receive a high value estimate while inducing an ego–agent interaction that is weakly supported by the logged data. We therefore seek policy improvements that increase long-horizon return while preserving support under the joint distribution of ego and surrounding-agent trajectories. To formalize this requirement, consider replacing the logged ego trajectory τeβ _e^β with a policy-generated candidate τ^e τ_e while retaining the surrounding-agent trajectory τaβ _a^β observed in the same interaction. We define the resulting joint-support degradation as ΔJ(s,τeβ,τ^e,τaβ)=logp(τeβ,τaβ∣s)p(τ^e,τaβ∣s). _J (s, _e^β, τ_e, _a^β )= p_D( _e^β, _a^β s)p_D( τ_e, _a^β s). (7) A positive value of ΔJ _J indicates that the interaction formed by the candidate ego trajectory is less supported by the data than the corresponding logged interaction. This motivates the ideal joint-support-constrained policy improvement problem maxθ _θ (s,τeβ,τaβ)∼τ^e∼πθ(⋅∣s)[QLCB(s,τ^e)] _ subarrayc(s, _e^β, _a^β) \\ τ_e _θ(· s) subarray [Q_LCB(s, τ_e) ] s.t. ΔJ(s,τeβ,τ^e,τaβ)≤ϵJ,a.s., _J (s, _e^β, τ_e, _a^β )≤ _J, .s., (8) where ϵJ≥0 _J≥ 0 specifies the maximum admissible degradation in joint interaction support. Equivalently, the constraint requires p(τ^e,τaβ∣s)≥e−ϵJp(τeβ,τaβ∣s).p_D( τ_e, _a^β s)≥ e^- _Jp_D( _e^β, _a^β s). (9) Directly evaluating this constraint is impractical because it requires density estimation in the high-dimensional joint trajectory space. The key idea of ICDP is to avoid estimating this joint density directly. Instead, we show that the joint-support degradation admits an exact decomposition into a marginal ego-trajectory component and an interaction-specific component. The latter captures interaction distribution shift (IDS), which we estimate contrastively and use to regularize diffusion-policy optimization. 4.2 Interaction Distribution Shift The joint-support constraint in (8) couples two distinct effects: departure of the candidate ego trajectory from the logged behavior distribution, and loss of compatibility between that trajectory and the surrounding-agent evolution observed in the data. Distinguishing these effects is central to ICDP, since conventional behavior regularization acts primarily on the former. Using the conditional factorization p(τe,τa∣s)=p(τe∣s)p(τa∣s,τe)p_D( _e, _a s)=p_D( _e s)p_D( _a s, _e), the joint-support degradation in (7) decomposes exactly as ΔJ=ΔE+ΔI,ΔE=logp(τeβ∣s)p(τ^e∣s),ΔI=logp(τaβ∣s,τeβ)p(τaβ∣s,τ^e). _J= _E+ _I, _E= p_D( _e^β s)p_D( τ_e s), _I= p_D( _a^β s, _e^β)p_D( _a^β s, τ_e). (10) Here, ΔE _E captures the ego marginal-support shift and ΔI _I the additional degradation due to ego–agent interaction. We refer to ΔI _I as interaction distribution shift (IDS). Unlike ΔE _E, which reflects how far the candidate ego trajectory moves from the logged ego distribution, IDS measures whether the surrounding-agent evolution observed in the interaction remains supported when paired with the candidate trajectory. Consequently, a candidate may remain well supported marginally, p(τ^e∣s)p_D( τ_e s), yet incur substantial interaction shift if p(τaβ∣s,τ^e)p_D( _a^β s, τ_e) is small. This decomposition isolates the component of distribution shift that is specific to interaction. Importantly, evaluating IDS does not require predicting how surrounding agents would respond to the candidate ego trajectory; it only requires assessing the support of the resulting ego–agent pairing under the logged interaction distribution. Direct evaluation of (10), however, would require estimating high-dimensional conditional trajectory densities. We next show that the required density ratio can instead be recovered contrastively. 4.3 Contrastive Recovery of Interaction Distribution Shift Our key observation is that this conditional density need not be modeled explicitly. Instead, IDS can be recovered by contrasting two notions of support: support of the complete ego–agent interaction and marginal support of the ego trajectory alone. Concretely, we introduce two contrastive scores. A joint score gJ(s,τe,τa)g_J(s, _e, _a) distinguishes logged ego–agent interactions from mismatched interactions formed using policy-generated ego trajectories, whereas an ego score gE(s,τe)g_E(s, _e) distinguishes logged ego trajectories from the same policy-generated candidates irrespective of the surrounding-agent evolution. Using the same candidate trajectories in both contrastive problems is important: it ensures that the component associated with marginal ego-trajectory shift is shared by both scores and can therefore be eliminated by subtraction. The precise contrastive distributions and training objectives are given in Appendix A. The joint score captures both marginal ego-trajectory shift and ego–agent dependence, while the ego score captures only the former. We therefore define the residual interaction score gI(s,τe,τa)=gJ(s,τe,τa)−gE(s,τe).g_I(s, _e, _a)=g_J(s, _e, _a)-g_E(s, _e). (11) The following result shows that this residual isolates the interaction-specific component of IDS. Theorem 1 (Contrastive recovery of IDS). Suppose the joint and ego contrastive objectives are constructed using the same policy-generated candidate trajectories, with balanced class sampling and common support. Let gJ∗g_J^* and gE∗g_E^* denote their population-optimal logits, and define the residual interaction score as gI∗(s,τe,τa)=gJ∗(s,τe,τa)−gE∗(s,τe).g_I^*(s, _e, _a)=g_J^*(s, _e, _a)-g_E^*(s, _e). (12) Then, for a logged interaction (s,τeβ,τaβ)(s, _e^β, _a^β) and a candidate ego trajectory τ^e τ_e, the difference between their residual interaction scores, evaluated against the same logged surrounding-agent trajectory τaβ _a^β, exactly recovers the interaction distribution shift: ΔI=gI∗(s,τeβ,τaβ)−gI∗(s,τ^e,τaβ). _I=g_I^*(s, _e^β, _a^β)-g_I^*(s, τ_e, _a^β). (13) Theorem 1 is the central mechanism behind ICDP. It shows that IDS is identifiable from the difference of two contrastive scores: the ego score removes the effect of marginal trajectory novelty from the joint score, leaving precisely the change in interaction support induced by the candidate ego trajectory. In practice, replacing the population scores with their learned counterparts yields Δ^I=gI(s,τeβ,τaβ)−gI(s,τ^e,τaβ), _I=g_I(s, _e^β, _a^β)-g_I(s, τ_e, _a^β), (14) which provides a tractable measure of interaction shift for policy optimization. The proof of Theorem 1 is deferred to Appendix A. Beyond its pointwise interpretation, IDS also admits a direct distributional interpretation. For fixed s, τeβ _e^β, and τ^e τ_e, averaging the interaction shift over the surrounding-agent evolution associated with the logged ego trajectory gives the divergence between the corresponding conditional interaction distributions. Corollary 2 (Distributional interpretation of IDS). For fixed s, τeβ _e^β, and τ^e τ_e, τa∼p(⋅∣s,τeβ)[ΔI]=DKL(p(τa∣s,τeβ)∥p(τa∣s,τ^e)).E_ _a p_D(· s, _e^β) [ _I ]=D_KL (p_D( _a s, _e^β)\, \|\,p_D( _a s, τ_e) ). (15) Thus, while ΔI _I measures interaction shift for an individual logged ego–agent realization, its expectation quantifies the distributional change in surrounding-agent behavior associated with replacing the logged ego trajectory by the candidate. This provides an information-theoretic interpretation of the interaction constraint used by ICDP. Algorithm 1 ICDP 0: Offline dataset D 1: Pretrain diffusion policy πθ _θ by behavior cloning 2: Initialize critic ensemble Qϕkk=1K\Q_ _k\_k=1^K and contrastive estimators gJ,gEg_J,g_E 3: repeat 4: Sample (s,τeβ,τaβ,R(H),s′)∼(s, _e^β, _a^β,R^(H),s ) 5: Sample τ^e∼πθ(⋅∣s) τ_e _θ(· s) 6: Update the critic ensemble 7: Update gJg_J and gEg_E 8: Compute Δ^I _I and the objective in equation 17 9: Update πθ _θ using NSPG 10: until convergence 11: return πθ _θ 4.4 Interaction-Constrained Policy Optimization (ICDP) The decomposition in equation 10 allows ICDP to control ego-trajectory shift and interaction distribution shift separately during policy improvement. For a logged interaction (s,τeβ,τaβ)(s, _e^β, _a^β) and candidate trajectory τ^e∼πθ(⋅∣s) τ_e _θ(· s), we define the interaction penalty CI=[Δ^I−ϵI]+,C_I= [ _I- _I ]_+, (16) where ϵI≥0 _I≥ 0 denotes the tolerated interaction shift. Let CEC_E denote the ego-support regularization term, and let ℒBCL_BC denote the behavior-cloning loss on logged ego trajectories. The policy is optimized according to maxθ[QLCB(s,τ^e)−λECE−λICI]−λBCℒBC, _θ\;E [Q_LCB(s, τ_e)- _EC_E- _IC_I ]- _BCL_BC, (17) where λE _E, λI _I, and λBC _BC control ego-support regularization, interaction-support regularization, and behavior-cloning regularization, respectively. The BC term anchors the policy to the demonstrated behavior, while CEC_E and CIC_I explicitly control marginal ego and interaction-level distribution shift. Since the actor is parameterized as a diffusion model, we optimize equation 17 using Noisy Space Policy Gradient (NSPG) (Selim et al., ). Gradients are propagated through the noisy denoising process, while the critic and interaction estimator evaluate the corresponding clean ego trajectory. Algorithm 1 summarizes the complete training procedure. 5 Experimental Results We evaluate ICDP with four goals: (i) to assess its closed-loop driving performance relative to prior learning-based planning methods, (i) to determine whether interaction-constrained offline RL improves over behavior cloning and unconstrained policy optimization under a controlled backbone, and (i) to study sensitivity to interaction complexity and regularization strength. Additional results on InterPlan, implementation details, and extended analyses are provided in the appendix. Baselines. We compare ICDP against representative prior planners evaluated on the same nuPlan benchmarks. PDM-Open (Dauner et al., 2023) is the learned variant of the PDM planning framework; GameFormer (Huang et al., 2023) models interactive prediction and planning using a game-theoretic transformer architecture; PlanTF (Cheng et al., 2024b) is an end-to-end transformer-based motion planner trained from expert demonstrations; PLUTO (Cheng et al., 2024a) augments imitation-based planning with auxiliary contrastive objectives; Diffusion Planner (Zheng et al., 2025) uses a diffusion model for multimodal trajectory generation; and Flow Planner (Tan et al., 2026) adopts flow-based generative modeling for motion planning. When available, we report variants without rule-based trajectory refinement to compare the learned policies directly. Table 1: Closed-loop performance on nuPlan Val14, Test14-Hard, and Test14-Random under non-reactive (NR) and reactive (R) evaluation. The best result in each column is highlighted in blue. Val14 Test14-Hard Test14-Random Planner NR R NR R NR R PDM-Open∗ 53.53 54.24 33.51 35.83 52.81 57.23 GameFormer w/o refine. 13.32 8.69 7.08 6.69 11.36 9.31 PlanTF 84.27 76.95 69.70 61.61 85.62 79.58 PLUTO w/o refine.∗ 88.89 78.11 70.03 59.74 89.90 78.62 Diffusion Planner 89.87 82.80 75.99 69.22 89.19 82.93 Flow Planner 90.43 83.31 76.47 70.42 89.88 82.93 ICDP 91.3 84.3 78.34 72.87 93.33 86.34 5.1 Main results Closed-loop benchmark performance. Table 1 compares ICDP with prior planners on the nuPlan closed-loop benchmark. ICDP achieves the strongest performance on all evaluations. On Test14-Hard, it reaches 78.3478.34 in non-reactive evaluation and 72.8772.87 in reactive evaluation, exceeding the strongest previously reported results of 76.4776.47 and 70.4270.42, respectively. The improvement is also consistent on Test14-Random, where ICDP obtains 93.3393.33 and 86.3486.34 under non-reactive and reactive simulation. Notably, the gains are preserved under reactive evaluation, where surrounding agents respond to the ego vehicle and interaction quality becomes particularly important. Controlled policy improvement. We next isolate the effect of reward-driven policy optimization and interaction-aware regularization using the same policy backbone and training data. Table 2 reports performance across ten evaluation checkpoints spaced by 2k optimization steps. Offline RL improves reactive performance over BC on both Test14-Hard and Test14-Random, but its gains are not consistent across the non-reactive evaluations. Incorporating the interaction constraint yields the strongest performance in five of the six settings. In particular, ICDP improves the reactive score from 70.0770.07 to 72.8772.87 on Test14-Hard and from 83.0283.02 to 86.3486.34 on Test14-Random relative to unconstrained Offline RL. These results indicate that interaction-aware regularization is particularly beneficial when the evaluation explicitly captures the response of surrounding traffic. Table 2: Controlled comparison of BC, unconstrained Offline RL, and ICDP. Results are averaged over ten evaluation checkpoints spaced by 2k optimization steps and reported as mean ± std. Val14 Test14-Hard Test14-Random Method NR R NR R NR R BC 89.8±1.289.8± 1.2 82.4±1.0982.4± 1.09 78.42±0.6978.42± 0.69 68.21±0.9568.21± 0.95 91.98±0.6291.98± 0.62 82.71±0.5382.71± 0.53 Offline RL 89.4±1.7489.4± 1.74 83.4±2.1183.4± 2.11 78.75±0.7578.75± 0.75 70.07±1.0870.07± 1.08 90.74±0.7090.74± 0.70 83.02±1.0883.02± 1.08 ICDP 91.3±1.9191.3± 1.91 84.3±2.0484.3± 2.04 78.34±1.0378.34± 1.03 72.87±2.0172.87± 2.01 93.33±0.8593.33± 0.85 86.34±1.1886.34± 1.18 5.2 Analysis of Interaction-Aware Regularization We analyze how the interaction-aware component of ICDP depends on the strength of ego- and interaction-support regularization and on the amount of surrounding-agent context provided to the support estimator. For compactness, we report the average closed-loop score across Test14-Hard and Test14-Random under both non-reactive and reactive evaluation. Regularization sensitivity. Figure 2(a–b) examines the effect of the ego- and interaction-support regularization strengths. For ego-support regularization, performance improves from λE=0.1 _E=0.1 to 0.30.3, but decreases as the weight is increased further, indicating that moderate regularization provides the best balance between preserving data support and allowing reward-driven policy improvement. A similar pattern is observed for interaction regularization: λI=1 _I=1 performs better than both weaker and stronger regularization among the tested values. We therefore use λE=0.3 _E=0.3 and λI=1 _I=1 in the reported ICDP configuration. Number of surrounding agents. Figure 2(c) studies the amount of multi-agent context used by the interaction-support estimator. Performance improves consistently as the number of surrounding agents increases from 1616 to 4848. Expanding the context further to 6464 agents yields a small decrease in average performance, from 82.4082.40 to 81.9781.97. These results suggest that incorporating a sufficiently broad set of surrounding agents is beneficial, while increasing the interaction context beyond this point provides limited additional benefit. (a) Ego-support weight λE _E (b) Interaction-support weight λI _I (c) Number of surrounding agents Figure 2: Sensitivity of ICDP to (a) ego-support regularization, (b) interaction-support regularization, and (c) the number of surrounding agents. Scores are averaged across Test14-Hard and Test14-Random under non-reactive and reactive evaluation. 5.3 Real-World Deployment Finally, we deploy ICDP on a full-scale autonomous truck to assess its behavior outside simulation. The experiments were conducted on a snow-covered test track and covered two complementary settings: multi-vehicle interactions and standalone driving on curved roads. Interactive driving. We evaluate ICDP in real-world multi-vehicle scenarios involving overtaking and lead-vehicle following under varying traffic configurations. Figure 3(a–b) shows representative overtaking examples together with the corresponding onboard perception view. These experiments provide qualitative evidence that the policy can execute interaction-aware maneuvers under real sensing and vehicle dynamics. Standalone driving. We additionally evaluate the policy on curved road segments without nearby interacting vehicles. Representative examples are shown in Fig. 3(c–d), illustrating nominal trajectory generation and execution under real road geometry, sensing, and vehicle dynamics. Overall, the real-world experiments provide qualitative evidence that ICDP can be deployed across both interactive and nominal driving conditions. (a) (b) (c) (d) Figure 3: Real-world deployment of ICDP on a full-scale autonomous truck. (a–b) Representative multi-agent interaction trial involving an overtaking vehicle, shown from external and onboard-perception viewpoints. (c–d) Standalone driving on curved road segments under real sensing and vehicle dynamics. 6 Conclusion and Future Work We introduced ICDP, an offline RL framework for diffusion-based driving that explicitly controls distribution shift at the level of multi-agent interactions. By decomposing joint-support degradation into ego-trajectory and interaction-specific components, we formalized interaction distribution shift (IDS) and showed that it can be recovered contrastively without explicit conditional density estimation or prediction of surrounding-agent responses. ICDP combines this interaction constraint with chunk-level value learning and diffusion-policy optimization to improve long-horizon driving performance while preserving support under the logged interaction distribution. Closed-loop evaluation on nuPlan shows consistent improvements over behavior cloning and unconstrained offline RL, particularly under reactive evaluation, while real-world deployment demonstrates feasibility on a full-scale autonomous truck. A natural next step is to extend ICDP beyond the offline setting through online reinforcement learning, allowing the policy to refine its behavior from closed-loop interaction while retaining interaction-aware support control. References Bansal et al. (2018) M. Bansal, A. Krizhevsky, and A. Ogale Chauffeurnet: learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079. Cited by: §1. Bojarski et al. (2016) M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. Cited by: §1, §2. Casas et al. (2021) S. Casas, A. Sadat, and R. Urtasun Mp3: a unified model to map, perceive, predict and plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14403–14412. Cited by: §2. Cheng et al. (2024a) J. Cheng, Y. Chen, and Q. Chen Pluto: pushing the limit of imitation learning-based planning for autonomous driving. arXiv preprint arXiv:2404.14327. Cited by: §1, §5. Cheng et al. (2024b) J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu Rethinking imitation-based planners for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 14123–14130. Cited by: §1, §2, §5. Chitta et al. (2022) K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, and A. Geiger Transfuser: imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence 45 (11), p. 12878–12895. Cited by: §1, §2. Codevilla et al. (2019) F. Codevilla, E. Santana, A. M. López, and A. Gaidon Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, p. 9329–9338. Cited by: §2. Dauner et al. (2023) D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta Parting with misconceptions about learning-based vehicle motion planning. In Conference on Robot Learning, p. 1268–1281. Cited by: §1, §5. Diehl et al. (2021) C. Diehl, T. Sievernich, M. Krüger, F. Hoffmann, and T. Bertram Umbrella: uncertainty-aware model-based offline reinforcement learning leveraging planning. arXiv preprint arXiv:2111.11097. Cited by: §2. Fan et al. (2018) H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong Baidu apollo em motion planner. arXiv preprint arXiv:1807.08048. Cited by: §1. Fang et al. (2025) L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing Diffusion actor-critic: formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In International Conference on Learning Representations, Vol. 2025, p. 75743–75763. Cited by: §1, §2. Fujimoto and Gu (2021) S. Fujimoto and S. S. Gu A minimalist approach to offline reinforcement learning. Advances in neural information processing systems 34, p. 20132–20145. Cited by: §2. Gao et al. (2025) C. Gao, C. Wu, M. Cao, C. Xiao, Y. Yu, and Z. Zhang Behavior-regularized diffusion policy optimization for offline reinforcement learning. arXiv preprint arXiv:2502.04778. Cited by: §2. Hawke et al. (2020) J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, et al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), p. 251–257. Cited by: §1. Hu et al. (2023) Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. Planning-oriented autonomous driving. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17853–17862. Cited by: §2. Huang et al. (2025) C. Huang, Y. Wang, Z. Zhang, W. Feng, and D. Huang Lane-changing policy offline reinforcement learning of autonomous vehicles based on bear algorithm with support set constraints. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 239 (9), p. 3784–3797. Cited by: §1, §2. Huang et al. (2023) Z. Huang, H. Liu, and C. Lv Gameformer: game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3903–3913. Cited by: §1, §5. Jaeger et al. (2025) B. Jaeger, D. Dauner, J. Beißwenger, S. Gerstenecker, K. Chitta, and A. Geiger Carl: learning scalable planning policies with simple rewards. arXiv preprint arXiv:2504.17838. Cited by: §2. Jiang et al. (2023) B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang Vad: vectorized scene representation for efficient autonomous driving. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 8306–8316. Cited by: §2. Kendall et al. (2019) A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), p. 8248–8254. Cited by: §1. Kostrikov et al. (2021) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169. Cited by: §1, §2. Kumar et al. (2019) A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems 32. Cited by: §2. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems 33, p. 1179–1191. Cited by: §1, §2. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. Cited by: §1. Li et al. (2024) Q. Li, X. Jia, S. Wang, and J. Yan Think2drive: efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European conference on computer vision, p. 142–158. Cited by: §2. Li et al. (2026) Q. Li, Z. P. Zhou, and S. Levine Reinforcement learning with action chunking. Advances in Neural Information Processing Systems 38, p. 55518–55553. Cited by: §2, §3. Li et al. (2023) Z. Li, F. Nie, Q. Sun, F. Da, and H. Zhao Uncertainty-aware decision transformer for stochastic driving environments. arXiv preprint arXiv:2309.16397. Cited by: §2. Liang et al. (2020) M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun Learning lane graph representations for motion forecasting. In European Conference on Computer Vision, p. 541–556. Cited by: §2. Liao et al. (2025) B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y. Zhang, Q. Zhang, et al. Diffusiondrive: truncated diffusion model for end-to-end autonomous driving. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12037–12047. Cited by: §2. Nguyen et al. (2025) T. Nguyen T. Mai et al. Comadice: offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. In International Conference on Learning Representations, Vol. 2025, p. 29071–29085. Cited by: §2. Noguchi and Yamamoto (2026) C. Noguchi and T. Yamamoto Pseudo-expert regularized offline rl for end-to-end autonomous driving in photorealistic closed-loop environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1096–1105. Cited by: §1, §2. Peng et al. (2019) X. B. Peng, A. Kumar, G. Zhang, and S. Levine Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: §1. Rowe et al. (2024) L. Rowe, R. Girgis, A. Gosselin, B. Carrez, F. Golemo, F. Heide, L. Paull, and C. Pal Ctrl-sim: reactive and controllable driving agents with offline reinforcement learning. arXiv preprint arXiv:2403.19918. Cited by: §2. Selim et al. (2026) M. Selim, C. Cipriani, and K. H. Johansson Noisy-space policy gradient for diffusion policies in offline reinforcement learning. arXiv preprint arXiv:2609.06882. Cited by: §2. [35] M. Selim, C. Cipriani, and K. H. Johansson Noisy-space policy gradient for diffusion policies in offline reinforcement learning. In Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning, Cited by: §4.4. Tan et al. (2026) T. Tan, Y. Zheng, R. Liang, Z. Wang, K. Zheng, J. Zheng, J. Li, X. Zhan, and J. Liu Flow matching-based autonomous driving planning with advanced interactive behavior modeling. Advances in Neural Information Processing Systems 38, p. 38310–38335. Cited by: §5. Toromanoff et al. (2020) M. Toromanoff, E. Wirbel, and F. Moutarde End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7153–7162. Cited by: §2. Wang et al. (2025) H. Wang, X. Ye, F. Tao, C. Pan, A. Mallik, B. Yaman, L. Ren, and J. Zhang Adawm: adaptive world model based planning for autonomous driving. In International Conference on Learning Representations, Vol. 2025, p. 85591–85615. Cited by: §1, §2. Wang et al. (2022) Z. Wang, J. J. Hunt, and M. Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193. Cited by: §1, §2. Wu et al. (2023) X. Wu, R. Chandra, T. Guan, A. S. Bedi, and D. Manocha Iplan: intent-aware planning in heterogeneous traffic via distributed multi-agent reinforcement learning. arXiv preprint arXiv:2306.06236. Cited by: §2. Yang et al. (2026) Z. Yang, X. Jia, Q. Li, X. Yang, M. Yao, and J. Yan Raw2drive: reinforcement learning with aligned world models for end-to-end autonomous driving (in carla v2). Advances in Neural Information Processing Systems 38, p. 134122–134147. Cited by: §1. Zhang et al. (2025) D. Zhang, J. Liang, K. Guo, S. Lu, Q. Wang, R. Xiong, Z. Miao, and Y. Wang Carplanner: consistent auto-regressive trajectory planning for large-scale reinforcement learning in autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17239–17248. Cited by: §1, §2. Zheng et al. (2025) Y. Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501.15564. Cited by: §1, §2, §5. Appendix A Contrastive Estimation of Interaction Distribution Shift This section provides the detailed construction of the contrastive objectives used to estimate interaction distribution shift (IDS), together with the proof of Theorem 1 and the information-theoretic interpretation of ΔI _I. A.1 Contrastive Objectives Let ν(τe∣s)ν( _e s) denote the distribution from which candidate ego trajectories are sampled during policy optimization. The same candidate distribution is used in both contrastive objectives described below. Joint interaction classification. The first contrastive objective distinguishes ego–agent interactions observed in the offline dataset from interactions formed by pairing a policy-generated ego trajectory with the surrounding-agent trajectory observed in the same scene. Its positive distribution is pJ+(s,τe,τa)=d(s)p(τe,τa∣s),p_J^+(s, _e, _a)=d_D(s)\,p_D( _e, _a s), (18) where d(s)d_D(s) denotes the empirical state distribution. The corresponding negative distribution is pJ−(s,τe,τa)=d(s)ν(τe∣s)p(τa∣s).p_J^-(s, _e, _a)=d_D(s)\,ν( _e s)\,p_D( _a s). (19) Thus, the negative construction preserves the scene-conditioned marginals of the ego and surrounding-agent trajectories while removing their dependence under the logged interaction distribution. Operationally, given a logged sample (s,τeβ,τaβ)∼(s, _e^β, _a^β) and a candidate trajectory τ^e∼ν(⋅∣s) τ_e ν(· s), the positive and negative examples are respectively (s,τeβ,τaβ),(s,τ^e,τaβ).(s, _e^β, _a^β), (s, τ_e, _a^β). (20) Let gJ(s,τe,τa)g_J(s, _e, _a) denote the logit of the corresponding binary classifier. Marginal ego classification. The joint classifier is sensitive both to whether the ego trajectory is supported by the logged behavior distribution and to whether it is compatible with the surrounding-agent trajectory. To separate these two effects, we introduce a second contrastive objective defined only over ego trajectories. Its positive and negative distributions are pE+(s,τe) p_E^+(s, _e) =d(s)p(τe∣s), =d_D(s)\,p_D( _e s), (21) pE−(s,τe) p_E^-(s, _e) =d(s)ν(τe∣s). =d_D(s)\,ν( _e s). (22) Let gE(s,τe)g_E(s, _e) denote the corresponding classifier logit. In practice, the negative ego example is the same candidate trajectory τ^e τ_e used to construct the negative joint interaction. Under balanced positive and negative sampling, the two classifiers are trained using the binary cross-entropy objectives ℒJ _J =−pJ+[logσ(gJ)]−pJ−[log(1−σ(gJ))], =-E_p_J^+ [ σ(g_J) ]-E_p_J^- [ (1-σ(g_J)) ], (23) ℒE _E =−pE+[logσ(gE)]−pE−[log(1−σ(gE))], =-E_p_E^+ [ σ(g_E) ]-E_p_E^- [ (1-σ(g_E)) ], (24) where σ(⋅)σ(·) denotes the logistic sigmoid. As defined in the main text, the residual interaction score is gI(s,τe,τa)=gJ(s,τe,τa)−gE(s,τe).g_I(s, _e, _a)=g_J(s, _e, _a)-g_E(s, _e). (25) A.2 Proof of Theorem 1 For balanced binary classification, the population-optimal logit is the log-density ratio between the positive and negative distributions. Applying this result to equation 18–equation 19 gives gJ∗(s,τe,τa) g_J^*(s, _e, _a) =logpJ+(s,τe,τa)pJ−(s,τe,τa) = p_J^+(s, _e, _a)p_J^-(s, _e, _a) =logp(τe,τa∣s)ν(τe∣s)p(τa∣s). = p_D( _e, _a s)ν( _e s)\,p_D( _a s). (26) Similarly, the population-optimal ego classifier satisfies gE∗(s,τe) g_E^*(s, _e) =logpE+(s,τe)pE−(s,τe) = p_E^+(s, _e)p_E^-(s, _e) =logp(τe∣s)ν(τe∣s). = p_D( _e s)ν( _e s). (27) Subtracting equation 27 from equation 26, the candidate-trajectory distribution ν(τe∣s)ν( _e s) cancels: gI∗(s,τe,τa) g_I^*(s, _e, _a) =gJ∗(s,τe,τa)−gE∗(s,τe) =g_J^*(s, _e, _a)-g_E^*(s, _e) =logp(τe,τa∣s)p(τe∣s)p(τa∣s) = p_D( _e, _a s)p_D( _e s)\,p_D( _a s) =logp(τa∣s,τe)p(τa∣s), = p_D( _a s, _e)p_D( _a s), (28) where the final equality follows from p(τe,τa∣s)=p(τe∣s)p(τa∣s,τe).p_D( _e, _a s)=p_D( _e s)\,p_D( _a s, _e). (29) Now consider a logged interaction (s,τeβ,τaβ)(s, _e^β, _a^β) and a candidate ego trajectory τ^e τ_e. Evaluating equation 28 at the logged and candidate ego trajectories while keeping the same surrounding-agent trajectory gives gI∗(s,τeβ,τaβ)−gI∗(s,τ^e,τaβ) g_I^*(s, _e^β, _a^β)-g_I^*(s, τ_e, _a^β) =logp(τaβ∣s,τeβ)p(τaβ∣s)−logp(τaβ∣s,τ^e)p(τaβ∣s) = p_D( _a^β s, _e^β)p_D( _a^β s)- p_D( _a^β s, τ_e)p_D( _a^β s) =logp(τaβ∣s,τeβ)p(τaβ∣s,τ^e) = p_D( _a^β s, _e^β)p_D( _a^β s, τ_e) =ΔI. = _I. (30) The common marginal p(τaβ∣s)p_D( _a^β s) therefore cancels, proving that the difference of residual interaction scores exactly recovers IDS at the population optimum. □ Why the candidate distribution must be shared. The use of the same candidate distribution in both contrastive objectives is essential to the cancellation above. To see this, suppose instead that the joint and ego classifiers used candidate distributions νJ(τe∣s) _J( _e s) and νE(τe∣s) _E( _e s), respectively. Their population-optimal difference would become gJ∗−gE∗=logp(τa∣s,τe)p(τa∣s)+logνE(τe∣s)νJ(τe∣s).g_J^*-g_E^*= p_D( _a s, _e)p_D( _a s)+ _E( _e s) _J( _e s). (31) The second term introduces an additional proposal-dependent contribution. Using the same policy-generated candidates for both objectives removes this term and isolates the interaction-specific density ratio. A.3 KL Interpretation of Interaction Distribution Shift For clarity, define the pointwise interaction shift for an arbitrary surrounding-agent trajectory τa _a as ΔI(s,τeβ,τ^e,τa)=logp(τa∣s,τeβ)p(τa∣s,τ^e). _I(s, _e^β, τ_e, _a)= p_D( _a s, _e^β)p_D( _a s, τ_e). (32) For fixed s, τeβ _e^β, and τ^e τ_e, its expectation under the conditional surrounding-agent distribution associated with the logged ego trajectory is τa∼p(⋅∣s,τeβ)[ΔI(s,τeβ,τ^e,τa)] _ _a p_D(· s, _e^β) [ _I(s, _e^β, τ_e, _a) ] =τa∼p(⋅∣s,τeβ)[logp(τa∣s,τeβ)p(τa∣s,τ^e)] =E_ _a p_D(· s, _e^β) [ p_D( _a s, _e^β)p_D( _a s, τ_e) ] =DKL(p(τa∣s,τeβ)∥p(τa∣s,τ^e)). =D_KL (p_D( _a s, _e^β)\, \|\,p_D( _a s, τ_e) ). (33) Hence, the pointwise IDS measures the change in support assigned to a particular surrounding-agent realization, while its expectation measures the divergence between the full conditional interaction distributions associated with the logged and candidate ego trajectories. □ Appendix B Implementation Details This section provides additional details on the data construction, network architectures, and optimization procedure used for ICDP. B.1 Data and Trajectory Representation We construct chunk-level transitions from the nuPlan trainval logs. Each transition contains the current scene s, a logged ego trajectory τeβ _e^β, the corresponding surrounding-agent futures τaβ _a^β, the discounted return over the action horizon, and the subsequent scene. Ego trajectories contain 3232 future steps sampled at 10Hz10\,Hz, corresponding to a 3.2s3.2\,s planning horizon. The scene history contains 1616 frames spanning the preceding 1.5s1.5\,s. Trajectories are represented in the current ego rear-axle frame, while the subsequent scene is independently expressed in the ego frame at the end of the action chunk. The processed dataset contains 744,832744,832 transitions spanning 6262 recorded scenario types, including 12,10812,108 recovery transitions. We reserve approximately 0.5%0.5\% of the data using a deterministic split, resulting in 741,172741,172 training and 3,6603,660 validation transitions. The resulting scenario distribution is naturally imbalanced and is shown in Fig. 5. The actor and critic scene representation contains agent histories, vectorized map elements, preferred-route information, and static objects. We retain up to 3232 agents and 55 static objects for the actor–critic scene encoder, together with at most 7070 lane and 2525 route polylines. Each polyline is represented by 2020 points within a 100m100\,m map-query radius. Ego actions are represented as (x,y,cosψ,sinψ)(x,y, ψ, ψ) over the prediction horizon. For the interaction-support estimator, the final ICDP configuration uses up to 4848 surrounding-agent trajectories. B.2 Reward Function Each ego trajectory is evaluated over an action chunk of H=32H=32 steps at Δt=0.1 t=0.1 s. The reward combines normalized route progress with multiplicative factors for route compliance, lane alignment, speed, time to collision (TTC), and comfort. Additional penalties are applied for near-critical drivable-area and TTC violations. Let ℱF denote the occurrence of either an ego collision or a drivable-area violation larger than 0.50.5 m at any point in the chunk. The chunk return is R=−5,if ℱ,∑h=1Hγh−1rh,otherwise,γ=0.99,R= cases-5,&if F,\\[2.0pt] _h=1^Hγ^h-1r_h,&otherwise, cases γ=0.99, (34) where an invalidating event also disables bootstrapping. For valid trajectories, the per-step reward takes the form rh=ph∏k∈fhk−charea−chttc,=area,center,direction,speed,ttc,comfort,r_h=p_h _k f_h^k-c_h^area-c_h^ttc, =\area,center,direction,speed,ttc,comfort\, (35) where php_h denotes speed-normalized progress along the preferred route. Backward progress is clipped to zero, while progress above the local speed-limit distance is capped. The multiplicative factors remain in [0.1,1][0.1,1] (except comfort, which is lower bounded by 0.50.5), so poor driving quality reduces the reward without reversing the progress signal. Table 3 summarizes the principal reward parameters. TTC and collision checks are computed against all tracked dynamic and static objects available in the logged scene. The ego- and interaction-support penalties used by ICDP are applied separately during policy optimization and are not part of the reward. Table 3: Principal reward parameters used for training. Parameter Value Action horizon H 3232 steps (3.23.2 s) Sampling interval Δt t 0.10.1 s Discount factor γ 0.990.99 Collision / severe off-road return −5-5 Soft / hard drivable-area threshold 0.30.3 / 0.50.5 m TTC additive-penalty threshold 0.950.95 s TTC factor saturation range 0.50.5–3.03.0 s Speed-limit tolerance 0.50.5 m/s Default speed limit 13.913.9 m/s Lane-direction tolerance 20∘20 Comfort-factor range [0.5,1][0.5,1] B.3 Network Architecture Scene encoding. The actor and critic use the same scene-encoder architecture with separate parameters. Agent histories and lane polylines are processed using modality-specific MLP-Mixer encoders, while static objects are embedded with an MLP. Position and semantic-type embeddings are added before multi-modal fusion through two Transformer layers with hidden dimension 192192 and eight attention heads. The critic encoder is initialized from the actor encoder and subsequently optimized independently. Diffusion actor. The actor predicts the complete clean ego trajectory using an x0x_0 diffusion parameterization. The current ego state and noisy future trajectory are embedded into a trajectory-query token, which cross-attends to the encoded scene through two diffusion-Transformer layers. The decoder is additionally conditioned on the preferred route and diffusion time. Only the ego trajectory is generated; surrounding-agent trajectories are used as observed context rather than prediction targets. Critic ensemble. The critic evaluates clean candidate trajectories. The 3232 trajectory steps are first embedded as time-indexed query tokens and processed by a two-layer trajectory trunk using temporal self-attention and cross-attention to the scene representation. The resulting trajectory representation is combined with pooled scene features and the normalized action sequence. Five independent MLP value heads then predict the chunk-level action value. The pessimistic estimate used for policy improvement is obtained from the ensemble mean and standard deviation as defined in Eq. equation 6. Interaction-support networks. The support estimator reuses pooled features from the delayed critic, with gradients stopped before the support networks. The ego trajectory and scene state are separately projected to 128128-dimensional features. For the joint classifier, surrounding-agent trajectories, their motion relative to the candidate ego trajectory, and trajectory-validity masks are additionally encoded and aggregated using masked mean and max pooling. The ego and joint classifiers use three-layer MLP heads with hidden width 256256. Their outputs are the logits gEg_E and gJg_J; the residual interaction score gI=gJ−gEg_I=g_J-g_E is computed directly and is not parameterized by a third network. B.4 Learning and Optimization Reward and chunk-level critic learning. The chunk reward combines progress along the preferred route with terms capturing collision avoidance, time to collision, drivable-area and route compliance, lane positioning, travel direction, speed, and comfort. Collisions and severe departures from the drivable area invalidate the action chunk, assign a terminal penalty, and disable bootstrapping. We use a per-step discount of γ=0.99γ=0.99. For valid transitions, four candidate trajectories are sampled from the target policy at the subsequent state, and their target-critic lower-confidence-bound estimates are averaged: y=R(H)+γHm1K∑k=1KQLCBtarg(s′,τe,k′),K=4,y=R^(H)+γ^Hm 1K _k=1^KQ_LCB^targ(s ,τ _e,k), K=4, (36) where m denotes the bootstrap-validity mask. All critic heads regress toward the same detached target. Diffusion-policy optimization. The actor uses a variance-preserving diffusion process with a linear noise schedule β(t)=0.1+19.9tβ(t)=0.1+19.9t and t∼(10−3,1)t (10^-3,1). During policy optimization, four candidate trajectories are decoded from each noisy action using five reverse-diffusion steps. Candidate value and support estimates are evaluated on the corresponding clean trajectories and used to construct the noisy-space policy gradient. The diffusion denoising objective remains active throughout training and acts as a behavioral anchor. Reward-driven guidance and interaction-support regularization are enabled after a 5,0005,000-update warm-up. Contrastive support learning. The ego and joint classifiers use the same policy-generated candidate trajectories as negative examples. Logged ego trajectories define the positive class, while logged surrounding-agent futures remain fixed when constructing joint examples. Both classifiers are trained with balanced binary cross-entropy, and delayed copies are used when scoring actor candidates. The final policy configuration uses λE=0.3 _E=0.3 and λI=1 _I=1, as analyzed in Sec. 5.2. Optimization. Actor, critic, and support networks use separate AdamW optimizers with learning rate 3×10−43× 10^-4, zero weight decay, and gradient-norm clipping at 1.01.0. Training uses four distributed workers with 256256 transitions per worker, giving a global batch size of 1,0241,024. Target networks are updated by exponential moving average with coefficient 0.0050.005. Unless otherwise stated, training is run for up to 250,000250,000 updates. Recovery and actor augmentation. To expose the policy to recoverable deviations from the demonstrations, we additionally construct recovery trajectories by perturbing eligible ego states and reconnecting them to the logged future after 2.0s2.0\,s. During actor training, eligible states are further perturbed within bounded lateral and heading offsets and the initial portion of the denoising target is refitted accordingly. Perturbed examples contribute to the diffusion anchor but are excluded from value guidance and contrastive support updates, ensuring that the interaction objectives remain defined with respect to consistent logged scenes and surrounding-agent futures. Table 4: Principal implementation settings used for ICDP. Parameter Value Planning horizon 3232 steps (3.2s3.2\,s) History horizon 1616 frames (1.5s1.5\,s) Actor/critic agent capacity 3232 Interaction-support agent context 4848 Lane / route capacity 7070 / 2525 Map-query radius 100m100\,m Scene hidden dimension 192192 Scene / actor Transformer layers 22 / 22 Critic ensemble size 55 Support feature dimension 128128 Per-step discount γ 0.990.99 Bellman-target samples 44 NSPG candidate samples 44 Reverse diffusion steps 55 Optimizer AdamW Learning rate 3×10−43× 10^-4 Global batch size 1,0241,024 EMA coefficient 0.0050.005 λE _E / λI _I 0.30.3 / 1.01.0 Guidance warm-up 5,0005,000 updates Training budget 250,000250,000 updates B.5 Qualitative Samples and Dataset Composition Policy samples. Figure 4 visualizes trajectory samples from the learned diffusion policy across diverse road geometries and traffic configurations. The examples span Singapore, Boston, and Las Vegas, covering turning, lane-following, and curved-road scenarios. For each scene, ten trajectories are sampled independently from the EMA policy and shown without critic-based ranking, rejection, smoothing, or post-processing. The logged ego trajectory and surrounding-agent futures are included for reference. These examples illustrate the diversity and geometric consistency of the learned trajectory distribution rather than closed-loop performance. (a) Singapore (b) Boston (c) Las Vegas (d) Additional cross-city scenes Figure 4: Qualitative trajectory samples from the learned diffusion policy across diverse nuPlan scenes. Each panel shows ten independently generated ego trajectories together with the logged ego trajectory, preferred route, and surrounding traffic. Samples are shown directly without critic-based selection or post-processing. Dataset composition. The offline dataset spans 6262 recorded nuPlan scenario categories with a long-tailed frequency distribution. Figure 5 shows the more- and less-frequent halves separately for readability. (a) More frequent scenario types (b) Less frequent scenario types Figure 5: Distribution of scenario types in the processed offline dataset. The 6262 recorded categories are divided into more- and less-frequent groups to make the long-tailed distribution legible. The offline dataset covers a broad range of driving situations, with substantial variation in the frequency of individual scenario types. Figure 5 reports the empirical distribution over all 6262 recorded scenario categories. We show the more- and less-frequent halves separately to preserve readability across the long-tailed distribution. Table 5: Closed-loop performance on InterPlan and selected interaction-critical scenario categories. The best result in each column is highlighted in blue. Planner Overall Nudge Around High Traffic Jaywalk PlanTF 47.70 49.40 58.85 33.94 PLUTO w/o refine. 58.47 71.56 67.25 25.48 Diffusion Planner 52.90 60.48 49.71 26.20 Flow Planner 61.82 72.96 67.21 43.57 ICDP 62.7 51.8 70.2 51.1 Appendix C Additional InterPlan Evaluation We additionally evaluate ICDP on InterPlan, which emphasizes interactive and challenging closed-loop driving scenarios. We compare against the same representative planning methods where reported results are available. Table 5 reports the overall score together with three interaction-focused scenario categories.