Paper deep dive
DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Hanyang Chen, Anirudh Satheesh, Longchao Da, Hua Wei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 4:25:13 AM
Summary
The paper introduces DADiff, a diffusion-based framework for online dynamics adaptation in reinforcement learning. It addresses the challenge of transferring policies from a source domain to a target domain with limited interactions by leveraging generative trajectory deviation. DADiff estimates dynamics mismatch using latent states from diffusion models and implements two variants: DADiff-modify, which adjusts rewards based on deviation, and DADiff-select, which filters source data based on deviation. Theoretical analysis bounds performance difference by generative trajectory deviation, and experiments show superior performance over existing methods.
Entities (10)
Relation Signals (8)
DDPM → istypeof → Diffusion Models
confidence 95% · We mainly focus on the denoising diffusion probabilistic model (DDPM) [12] in this paper.
DADiff-select → isvariantof → DADiff
confidence 95% · We further introduce two variants... data selection... DADiff-select
DADiff-modify → isvariantof → DADiff
confidence 95% · We further introduce two variants based on SAC... including reward modification... DADiff-modify
DADiff → uses → Diffusion Models
confidence 95% · DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories
DADiff → solves → Online Dynamics Adaptation
confidence 92% · In this paper, we consider the setting of online dynamics adaptation... we introduce DADiff
Generative Trajectory Deviation → estimates → Dynamics Mismatch
confidence 90% · leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch
DADiff-modify → uses → Generative Trajectory Deviation
confidence 90% · DADiff-modify, which adjusts source-domain rewards with deviation-based penalties
DADiff-select → uses →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2607.16090v1
- Canonical: https://arxiv.org/abs/2607.16090v1
Trouble viewing inline? Open PDF directly →
Full Text
54,663 characters extracted from source content.
Expand or collapse full text
DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning Hanyang Chen1, Anirudh Satheesh2, Longchao Da1 and Hua Wei1 1Hanyang Chen, Longchao Da, and Hua Wei are with Arizona State University, hchen478, longchao, hua.wei@asu.edu2Anirudh Satheesh is with the University of Maryland, College Park, anirudhs@terpmail.umd.edu Abstract Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release. I Introduction Reinforcement learning (RL) has shown strong potential in complex decision-making tasks, but training directly in the real-world environment (target domain) is often restricted by safety, cost, and limited interaction budgets. An alternative strategy is to train policies in a surrogate environment (source domain), such as a simulator, and then transfer them to the target domain. But due to the dynamics mismatch between the source and target domains, directly transferring the policy often leads to performance degradation, which is a critical challenge in the sim-to-real problem [35, 5]. One solution to this transfer problem is known as online dynamics adaptation [33, 17], where policies are trained with abundant source-domain data and only limited interactions in the target domain. In this setting, the state space, action space, and reward function remain consistent across domains, while the transition dynamics differ. Compared with solutions such as domain randomization [23, 20, 4] or simulator calibration [2], online dynamics adaptation does not require access to high-fidelity simulators or prior knowledge of target dynamics, and can therefore be applied in situations where such information is unavailable. Existing online dynamics adaptation methods, including classifier-based approaches [6], value-guided filtering [33], and representation learning [18], capture dynamics discrepancy from different perspectives: classifiers provide coarse distinctions between domains, value-guided methods depend on the modeling of forward predictions, and representation learning relies on assumptions of invariant latent structures across domains. When the domains are complex or stochastic, a key challenge that remains is to develop an approach capable of capturing dynamics discrepancy in a more fine-grained and distributional manner. The generative modeling perspective provides a potential direction. Generative models, such as diffusion models [26, 12], have demonstrated strong capability in representing complex distributions. When state transitions are viewed as a conditional generative process, the mismatch between source and target domains can be interpreted as a discrepancy between their respective generative processes. Specifically, the multi-step sampling procedure in diffusion models and flow matching methods produces several latent states, which construct a generative trajectory, serving as structured signals of source–target dynamics deviation. These latent states allow the discrepancy to be captured not only at the next-state level but also along the entire trajectory. Intuitively, if the source and target domains follow different dynamics, their trajectories will diverge at multiple steps, a phenomenon we term generative trajectory deviation. This notion provides a fine-grained view of dynamics discrepancy by revealing how divergence accumulates along the trajectory, rather than relying solely on local or aggregated comparisons. Our theoretical analysis further connects trajectory deviation to performance guarantees, providing motivation for algorithmic design. Building on this perspective, we introduce DADiff, a diffusion-based framework for online dynamics adaptation. DADiff leverages latent states in diffusion models to measure generative trajectory deviation between source and target domains, and exploits this deviation in two complementary ways: (i) DADiff-modify, which adjusts source-domain rewards with deviation-based penalties, and (i) DADiff-select, which filters source-domain data based on deviation before value function updates. We further discuss the applicability of these variants to different tasks, highlight the advantages of our method compared to prior work, and establish a connection between our analysis and the theoretical guarantee of prior work. Empirical results in environments with various shifts show the superior performance of our method compared to existing algorithms. I Related Works Domain Adaptation in RL Generalizing RL policies to diverse environments is critical for real-world deployment, where transition dynamics [6, 34], state or action spaces [8, 22] may be different. To address domain adaptation, prior work falls under three categories: (i) domain randomization that randomizes transition dynamics to expose agents to many environment configurations [25, 13], (i) meta-learning to few-shot adapt to many environments [21, 32], and (i) expert demonstrations of target environments through imitation learning [24, 7]. However, these approaches are either computationally expensive or require hard-to-obtain demonstrations. With only limited target-domain data, some works perform reward modifications to transition to the target domain by using transition classifiers [6, 9] or reward augmentations [29, 17]. Data selection methods [33, 31] have also been used to filter out part of the source-domain transitions and train policies on both source and target domain data. When the domains are complex or stochastic, a key challenge that remains is to develop an approach capable of capturing the dynamics discrepancy. Our method explores this challenge from a generative modeling perspective by measuring the generative trajectory deviation between the source and target domains. Diffusion Models in RL Diffusion models [26, 12] have been extensively used for generating effective decision-making policies in several domains, such as reinforcement learning [14] and robotics [3]. Specifically, they are widely leveraged to synthesize data for offline RL [15], facilitate planning and action generation in multi-task scenarios [11], and enhance the representational capacity of learned RL policies [30]. In addition, diffusion models have also been extended to the multi-agent settings [36]. In the field of domain adaptation, they are utilized to augment the target-domain data in order to boost the performance of offline RL policies [28]. However, the introduction of synthesizers may lead to extra computational costs, and the quality of synthesized data is hard to guarantee. In contrast, we choose to directly estimate the dynamics discrepancy by multiple latent states from diffusion models instead of generating more synthetic data. I Preliminaries Online Dynamics Adaptation We consider two Markov Decision Processes (MDPs), denoted as ℳsrc=(,,Psrc,r,γ)M_src=(S,A,P_src,r,γ) and ℳtar=(,,Ptar,r,γ)M_tar=(S,A,P_tar,r,γ) for the source domain and target domain, respectively. The state space S, action space A, reward function r:×→ℝr:S×A and discount factor γ∈[0,1]γ∈[0,1] are consistent across both domains, while the transition dynamics PsrcP_src and PtarP_tar differ. The goal of online dynamics adaptation is to learn a policy π that achieves high performance in the target domain ℳtarM_tar, utilizing sufficient data from the source domain and only limited interactions from the target domain. In addition, we specify a domain ℳM and define the probability that a policy π encounters a state s at time step t as Pℳ,tπ(s)P_M,t^π(s). Therefore, the normalized probability that a policy π visits a state-action pair (s,a)(s,a) in the domain ℳM can be represented as ρℳπ(s,a)≔(1−γ)∑t=0∞γtPℳ,tπ(s)π(a|s) _M^π(s,a) (1-γ) _t=0^∞γ^tP_M,t^π(s)π(a|s). The expected return of a policy π in ℳM is defined as ηℳ(π)=(s,a)∼ρℳπ[r(s,a)] _M(π)=E_(s,a) _M^π[r(s,a)]. We assume the reward are bounded by |r(s,a)|≤rmax,∀s∈,a∈|r(s,a)|≤ r_max,∀ s ,a . Diffusion Models Diffusion models [26, 12] are a family of generative models that learn to generate samples from a target distribution. We mainly focus on the denoising diffusion probabilistic model (DDPM) [12] in this paper. DDPM consists of a forward process and a reverse process. The forward process is regarded as a Markov chain that gradually adds noise to data, transforming a clean data point x0x_0 into Gaussian noise, which is formulated as follows, xk=1−βkxk−1+βkϵ,ϵ∼(0,I),x_k= 1- _kx_k-1+ _kε, ε (0,I), (1) where xkx_k is the noisy data at diffusion timestep k, βk _k is the noise schedule, and ϵε is Gaussian noise. To simplify the forward process, we can directly sample the noisy data at diffusion timestep k as follows, xk=α¯kx0+1−α¯kϵ,ϵ∼(0,I),x_k= α_kx_0+ 1- α_kε, ε (0,I), (2) where αk=1−βk _k=1- _k and α¯k=∏i=1kαi α_k= _i=1^k _i. The reverse process learns to denoise the noisy data step by step, which is formulated as follows, xk−1=1αk(xk−βk1−α¯kϵθ(xk,k))+1−α¯k−11−α¯kβkϵ,ϵ∼(0,I), aligned x_k-1&= 1 _k (x_k- _k 1- α_k _θ(x_k,k) )+ 1- α_k-11- α_k _k\,ε, ε (0,I), aligned (3) where ϵθ(xk,k) _θ(x_k,k) is a noise model that estimates the noise from the noisy data point xkx_k. The noisy data points xkk=0K\x_k\_k=0^K form a generative trajectory from the initial noisy data xKx_K to the clean data x0x_0. The training objective of the noise model is formulated as follows, ℒdiff=x0,ϵ,k[‖ϵ−ϵθ(α¯kx0+1−α¯kϵ,k)‖2].L_diff=E_x_0,ε,k [||ε- _θ( α_kx_0+ 1- α_kε,k)||^2 ]. (4) IV Methodology Figure 1: Illustration of DADiff. This figure visualizes the generative trajectories in the source and target domains. The deviation d(s,a,s′)d(s,a,s ) is measured by the discrepancy dkd_k of each latent state sk′s _k in the source and target domain generative trajectories. In this section, we first introduce a theoretical analysis to demonstrate the connection between the dynamics mismatch and the generative trajectory mismatch. Then, we present our diffusion-based method, DADiff, which measures the generative trajectory deviation from the perspective of diffusion models and adapts the learned policy to the target domain. The overview of our method is shown in Figure 1. IV-A Theoretical Analysis Before introducing the theoretical analysis, we first provide the definition of a generative trajectory, which is crucial for the analysis. For clarity, we denote the next state s′s as s0′s _0. Definition IV.1 (Generative trajectory.) Specify a domain ℳM with transition dynamics Pℳ(s0′|s,a)P_M(s _0|s,a). There is a generative trajectory for the next state s0′s _0 consisting of K auxiliary variables sk′k=1K\s _k\_k=1^K, referred to as latent states. These latent states form a Markov chain from the initial latent state sK′s _K to the next state s0′s _0 conditioned on the state–action pair (s,a)(s,a). Remark. The Markov-chain definition enables the transition dynamics to be decomposed into multiple conditional probabilities, i.e., Pℳ(s0′|s,a)=∫Pℳ(sK′|s,a)∏k=1KPℳ(sk−1′|sk′,s,a)ds1:K′P_M(s _0|s,a)= P_M(s _K|s,a) _k=1^KP_M(s _k-1|s _k,s,a)ds _1:K. In this way, the next state s0′s _0 can be viewed as being generated step by step with latent states, forming a generative trajectory. The discrepancy of such generative trajectories across domains provides a natural estimation of the dynamics discrepancy. We construct generative trajectories in both source and target domains, starting from the same initial latent state sK′s _K, and derive Theorem IV.2 to establish the connection between the dynamics mismatch and the generative trajectory mismatch. The detailed proof is provided in Appendix VII-B. Theorem IV.2 (Performance bound controlled by generative trajectory discrepancy.) Denote ℳsrcM_src and ℳtarM_tar as the source and target domains with different dynamics, respectively. The performance difference of any policy π evaluated in ℳsrcM_src and ℳtarM_tar can be bounded as below, ηℳsrc(π)−ηℳtar(π)≤2γrmax(1−γ)2ρsrcπ[Psrc[DKL(Psrc(sK′∣s,a)∥Ptar(sK′∣s,a))]]⏟(a):initiallatentstatedeviation+2γrmax(1−γ)2ρsrcπ[Psrc[∑k=1KDKL(Psrc(sk−1′∣sk′,s,a)∥Ptar(sk−1′∣sk′,s,a))]]⏟(b):latentstatetransitionmismatch. aligned & _M_src(π)- _M_tar(π)≤\\ & 2γ r_max(1-γ)^2 E_ _src^π\! [ E_P_src\! [D_KL\! (P_src(s _K s,a)\,\|\,P_tar(s _K s,a) ) ] ]_(a):\ initial\ latent\ state\ deviation+\\ & 2γ r_max(1-γ)^2 E_ _src^π\! [ E_P_src\! [ _k=1^KD_KL\! (P_src(s _k-1 s _k,s,a)\,\|\,P_tar(s _k-1 s _k,s,a) ) ] ]_(b):\ latent\ state\ transition\ mismatch. aligned (5) Remark. This bound indicates that the performance difference of a policy π between the source and target domains is controlled by the initial latent state deviation term (a) and the latent state transition mismatch term (b). Since the generative trajectories in both the source and target domains share the same initial latent state sK′s _K, term (a) vanishes, leaving term (b) as the sole determinant of the performance difference. In other words, as long as the generative trajectories are similar in the source and target domains, the performance difference is small, and vice versa. We note that PAR [18] can be considered as a special case of Theorem IV.2 when K=1K=1. A discussion on the connection between our analysis and the theoretical guarantee of PAR is provided in Section VI-A. IV-B Domain Adaptation with Diffusion Theorem IV.2 provides a theoretical guarantee linking the performance difference of a policy π to the generative trajectory, thereby motivating a careful design of latent states in the trajectory. In this section, we adopt the formulation of DDPM to better characterize the dynamics discrepancy. We first redeclare the reverse process of DDPM in a reparameterized form to describe the latent state transition in domain ℳM as follows, sk−1′=1αk(sk′−βk1−α¯kϵℳ(sk′,s,a,k))+1−α¯k−11−α¯kβkϵ,ϵ∼(0,I), aligned s _k-1= 1 _k (s _k- _k 1- α_k _M(s _k,s,a,k) )+ 1- α_k-11- α_k _k\,ε, ε (0,I), aligned (6) where ϵℳ(sk′,s,a,k) _M(s _k,s,a,k) is the noise from the latent state sk′s _k in domain ℳM. It indicates that the latent state transition follows a Gaussian distribution, i.e., Pℳ(sk−1′∣sk′,s,a)∼(1αk(sk′−βk1−α¯kϵℳ(sk′,s,a,k)),1−α¯k−11−α¯kβkI). aligned P_M(s _k-1& s _k,s,a)& ( 1 _k (s _k- _k 1- α_k _M(s _k,s,a,k) ), 1- α_k-11- α_k _kI ). aligned (7) According to Theorem IV.2, the performance difference of a policy π across domains is determined by the latent state transition mismatch term (b). Therefore, we can estimate the generative trajectory deviation d(s,a,s′)d(s,a,s ) with the defined distribution of latent state transition in Equation 7 as follows, d(s,a,s′)=∑k=1KDKL(Psrc(sk−1′|sk′,s,a)||Ptar(sk−1′|sk′,s,a))=∑k=1Kβk2(1−α¯k−1)αk‖ϵsrc(sk′,s,a,k)−ϵtar(sk′,s,a,k)‖2. aligned d(s,a,s )&= _k=1^KD_KL(P_src(s _k-1|s _k,s,a)||P_tar(s _k-1|s _k,s,a))\\ &= _k=1^K _k2(1- α_k-1) _k \| _src(s _k,s,a,k)- _tar(s _k,s,a,k) \|^2. aligned (8) We derive this equation by computing the KL divergence between two Gaussian distributions. Notably, as the state transition tuple (s,a,s′)(s,a,s ) comes from the source domain, the noise ϵsrc(sk′,s,a,k) _src(s _k,s,a,k) estimated in the reverse process must be consistent with the noise used in the forward process to generate the latent state sk′s _k, which indicates ϵsrc(sk′,s,a,k)=ϵ _src(s _k,s,a,k)=ε with ϵ∼(0,I)ε (0,I). Besides, we introduce a noise model ϵtarθ(sk′,s,a,k)ε^θ_tar(s _k,s,a,k), trained with target-domain data, to estimate the noise in the target domain. The training objective is formulated as follows, ℒnoise=(s,a,s′)∼tar,ϵ,k[‖ϵ−ϵtarθ(α¯ks0′+1−α¯kϵ,s,a,k)‖2].L_noise=E_(s,a,s ) _tar,ε,k [ \|ε-ε^θ_tar( α_ks _0+ 1- α_kε,s,a,k) \|^2 ]. (9) This objective mirrors the standard DDPM training loss, but conditions on (s,a)(s,a) to capture dynamics in the target domain. For the latent state sk′s _k in Equation 8, there are two ways to obtain it: (i) by iteratively applying the reverse process in Equation 6, and (i) by sampling directly from the forward process of DDPM, i.e., sk′=α¯ks0′+1−α¯kϵs _k= α_ks _0+ 1- α_kε with ϵ∼(0,I)ε (0,I). Specifically, the first way requires sequential sampling across all steps to generate the entire generative trajectory, which is computationally expensive. In contrast, the second way can produce all latent states in parallel, yielding a much more efficient implementation. Therefore, we choose to obtain the latent state sk′s _k via the forward process in our method. Finally, the deviation d(s,a,s′)d(s,a,s ) can be practically estimated as follows, d(s,a,s′)=∑k=1Kβk2(1−α¯k−1)αk‖ϵ−ϵtarθ(α¯ks0′+1−α¯kϵ,s,a,k)‖2,ϵ∼(0,I). aligned d(s,a,s )= _k=1^K _k2(1- α_k-1) _k \|ε-ε^θ_tar( α_ks _0+ 1- α_kε,s,a,k) \|^2,\\ ε (0,I). aligned (10) We further introduce two variants based on SAC [10] to utilize the deviation d(s,a,s′)d(s,a,s ), including reward modification and data selection, since we find that baselines adopting these two techniques exhibit complementary advantages in different tasks, which is shown in Section V-B. We analyze the possible reason for this phenomenon from the reward distribution aspect in Section VI-B. The details of DADiff variants are provided as follows. Reward modification. We refer to this variant as DADiff-modify. It adopts the deviation d(s,a,s′)d(s,a,s ) as a reward penalty to modify the reward function in the source domain, i.e., rmod(s,a,s′)=r(s,a,s′)−λd(s,a,s′),r_mod(s,a,s )=r(s,a,s )-λ d(s,a,s ), (11) where λ is a penalty coefficient to balance the original reward and the penalty. The objective function for training the value function gives, ℒcritic=(s,a,rmod,s′)∼src∪tar[(Qϕ−Qϕ)2],L_critic=E_(s,a,r_mod,s ) _src _tar [(Q_φ-TQ_φ)^2 ], (12) where tarD_tar and srcD_src are the datasets from the target and source domains, respectively, QϕQ_φ is the value function, and T is the Bellman operator. Data selection. We refer to this variant as DADiff-select. We select fixed percentage data with the lowest deviation d(s,a,s′)d(s,a,s ) from a batch of source domain data. The selected data is then used to update the value function. We formulate the objective function of the value function as follows, ℒcritic= _critic= (s,a,r,s′)∼tar[(Qϕ−Qϕ)2]+ _(s,a,r,s ) _tar [(Q_φ-TQ_φ)^2 ]+ (13) (s,a,r,s′)∼src[ω(s,a,s′)(Qϕ−Qϕ)2], _(s,a,r,s ) _src [ω(s,a,s )(Q_φ-TQ_φ)^2 ], where ω(s,a,s′)=(d(s,a,s′)<dξ%)ω(s,a,s )= 1(d(s,a,s )<d_ξ\%), 1 is the indicator function, and dξ%d_ξ\% denotes the lowest ξ-quantile deviation in the batch. For both variants, the objective function of the policy π is formulated as: ℒactor=(s,a,r,s′)∼src∪tar[−mini=1,2Qϕi(s,a)+τlogπ(a|s)],L_actor=E_(s,a,r,s ) _src _tar [- _i=1,2Q_ _i(s,a)+τ π(a|s) ], (14) where τ is the entropy temperature coefficient, and i denotes the value function index. We provide the pseudocode of DADiff in Algorithm 1. 1 Input: Source domain ℳsrcM_src, target domain ℳtarM_tar, and target domain interaction frequency F Initialization: Policy π, value function Qϕii=1,2\Q_ _i\_i=1,2, target value function Qϕi′i=1,2\Q_φ _i\_i=1,2, noise model ϵtarθε^θ_tar, replay buffers src,tar\D_src,D_tar\, penalty coefficient λ, data selection ratio ξ, batch size N 2 3for j=1,2,…j=1,2,… do 4 Collect (ssrc,asrc,rsrc,ssrc′)(s_src,a_src,r_src,s _src) from ℳsrcM_src, store in srcD_src 5 if jmodF=0j F=0 then 6 Collect (star,atar,rtar,star′)(s_tar,a_tar,r_tar,s _tar) from ℳtarM_tar, store in tarD_tar 7 8 end if 9 Sample N transitions from tarD_tar, train model ϵtarθε^θ_tar via Eq. 9 10 Sample N transitions from srcD_src, compute d(ssrc,asrc,ssrc′)d(s_src,a_src,s _src) via Eq. 10 11 if using reward modification then 12 Modify source domain rewards via Eq. 11 13 Update value functions QϕiQ_ _i by minimizing Eq. 12 14 15 end if 16 else if using data selection then 17 Select ξ-quantile data from srcD_src by d(ssrc,asrc,ssrc′)d(s_src,a_src,s _src) 18 Update value functions QϕiQ_ _i by minimizing Eq. 13 19 20 end if 21 Update actor π by minimizing Eq. 14 22 Update target value functions Qϕi′Q_φ _i 23 24 end for Algorithm 1 Domain Adaptation with DADiff V Experiments V-A Experimental Setup We conduct experiments in four environments (ant, hopper, halfcheetah, walker) from Gym MuJoCo [27, 1]. The source domain is set as the original environment, while the target domain is set as the environment with shifts in kinematics, morphology, friction, or gravity. Kinematic shifts restrict joint rotation ranges, morphology shifts reduce limb sizes, friction shifts modify the friction coefficient, and gravity shifts adjust gravitational acceleration. Kinematic and morphology configurations follow PAR [18], while friction and gravity shifts follow ODRL [19] at a level of 0.5. We compare our method with the following baselines: DARC [6], which trains domain classifiers to estimate the dynamics discrepancy and modifies the reward function in the source domain; VGDF [33], which uses a value-guided data filtering method to select data from the source domain; PAR [18], which trains encoders to estimate the representation discrepancy and modifies the reward function in the source domain; SAC-IW, which estimates the dynamics discrepancy as an importance sampling term for value function; SAC-tune, which fine-tunes the policy in the target domain for 10510^5 environmental steps; SAC-tar [10], which is the vanilla SAC trained in the target domain with 10510^5 environmental steps; Oracle [10], which is the vanilla SAC trained in the target domain with 1M environmental steps. We implement all algorithms based on the official code of ODRL [19] and follow the hyperparameters in the original paper. We allow all algorithms to interact with the source domain for 1M environmental steps and the target domain for 10510^5 environmental steps, i.e., the target domain interaction frequency F=10F=10. All algorithms are trained with five random seeds. V-B Adaptation Performance Evaluation Figure 2: Adaptation performance under kinematic, morphology, friction, and gravity shifts (from top to bottom). The solid curves and the shaded regions denote the mean and standard deviation over five random seeds, respectively. DADiff demonstrates superior or highly competitive performance against all baselines in the majority of tasks. Figure 3: GPU memory and runtime comparisons on the halfcheetah (broken back thigh) task. In the GPU memory comparison, DADiff-modify and DADiff-select exhibit slightly higher GPU memory cost compared to PAR and DARC. In the runtime comparison, VGDF requires 3×3× more training time than other methods due to its model-based approach. We conduct experiments on sixteen tasks with diverse shifts to evaluate the adaptation performance of DADiff and baselines. The results are summarized in Figure 2. Overall, DADiff exhibits consistently strong performance, demonstrating superior or competitive performance against all baselines in the majority of tasks. While some existing methods, such as VGDF, PAR, or SAC-tune, occasionally reach competitive results in specific tasks, their performance fluctuates significantly across different tasks. In contrast, DADiff maintains stable and superior adaptation performance across a wide range of shift types. We further discuss the performance of two variants of DADiff, DADiff-modify and DADiff-select, respectively. Reward modification variant. The reward modification variant of our method, DADiff-modify, demonstrates strong and consistent performance across diverse tasks. As shown in Figure 2, it surpasses other reward modification baselines, including PAR, DARC, and SAC-IW, in most tasks and achieves performance comparable to oracle-level methods. On average, DADiff-modify improves by 8.7% across all sixteen tasks, with the largest gain of 42.3% on the halfcheetah (broken back thigh). In addition to its performance advantages, we observe that our method incurs a slight increase in GPU memory usage compared to PAR and DARC due to latent state generation, as shown in Figure 3. This modest increase, however, contributes positively to adaptation performance by enabling better discrepancy estimation, thus representing a favorable trade-off between computational cost and effectiveness. To further explore the performance of DADiff-modify in stochastic environments, we provide an experiment in Section VI-A. Data selection variant. In Figure 2, the data selection variant, DADiff-select, proves to be a highly effective alternative by achieving competitive performance against top baselines in tasks where reward modification methods falter. Specifically, in the halfcheetah (no thighs), hopper (big head), and hopper (friction) tasks, reward modification methods exhibit poor performance. In contrast, DADiff-select achieves results that are highly competitive with the top-performing baseline, VGDF. This indicates that in certain tasks, directly filtering for transitions with low dynamics mismatch is a more effective strategy than modifying rewards. We analyze the possible reason in Section VI-B. Furthermore, while VGDF demonstrates top-tier performance in these tasks, it carries significant trade-offs. Since VGDF is a model-based approach, it takes significantly longer to train by more than 3×3×, as shown in Figure 3. On the other hand, DADiff-select is able to match or exceed the performance of VGDF on such environments while maintaining comparable efficiency to similar model-free baselines. V-C Parameter Study The performance of DADiff is influenced by several key hyperparameters. To better understand their roles, we conducted a series of experiments across different tasks. The results on halfcheetah (broken back thigh) and walker (no right thigh) are presented in Figure 4. Penalty Coefficient λ. λ controls the scale of reward penalty in DADiff-modify. As shown in Figure 4a, we evaluate the performance of DADiff-modify across multiple values of λ. We find that a worse performance is often shown in the setting λ=0λ=0, where no penalty is adopted for rewards. It demonstrates the necessity of reward modification. Meanwhile, the results also indicate that the optimal value of λ is task-dependent, and there could be multiple values that yield good performance for a specific task. For instance, in the halfcheetah (broken back thigh) task, both λ=0.5λ=0.5 and λ=5.0λ=5.0 achieve the best performance. A poorly chosen λ can significantly degrade performance, highlighting the importance of tuning this coefficient. Data Selection Ratio ξ%ξ\%. ξ%ξ\% controls the ratio of source domain data to retain in DADiff-select. As shown in Figure 4b, we evaluate the performance of DADiff-select across multiple values of ξ%ξ\%. Similar to the penalty coefficient, the optimal value of ξ%ξ\% is task-dependent. We find that both too much (ξ%=100%ξ\%=100\%) and too little ((ξ%=0%ξ\%=0\%)) source data can lead to suboptimal performance. As retaining too much source data may introduce transitions with significant dynamics mismatch, while retaining too little may result in insufficient data for effective learning. Diffusion Timesteps K. K controls the number of diffusion timesteps used to measure the discrepancy in both DADiff-modify and DADiff-select. We provide the results of DADiff-modify in Figure 4c. The results shows that performance improves up to K=100K=100. Increasing K further to 200 causes a decline, likely due to the limited capacity of the noise model, which may struggle to accurately estimate noise across too many timesteps. (a) Penalty coefficient λ. (b) Data ratio ξ%ξ\%. (c) Diffusion timesteps K. Figure 4: Parameter study. The solid curves and the shaded regions denote the mean and standard deviation over five random seeds, respectively. VI Discussions VI-A Connection between DADiff and PAR We explore the connection between PAR and our method from a theoretical perspective. The performance bound of our method is controlled by the generative trajectory discrepancy in Theorem IV.2. We consider a special case, where the number of latent states in the trajectory is K=1K=1. Instead of considering latent states in the generative trajectory, we take s1′s _1 as a latent representation and introduce the one-to-one representation mapping assumption in PAR [18], which assumes that there exists a one-to-one mapping for each state-action pair (s,a)(s,a) and its latent representation s1′s _1. In this setting, the state-action pair (s,a)(s,a) in Equation 5 can be all replaced by the corresponding latent representation s1′s _1. Therefore, the performance bound can be rewritten as follows, ηℳsrc(π)−ηℳtar(π)≤2γrmax(1−γ)2ρsrcπ[Psrc[DKL(Psrc(s0′|s1′)||Ptar(s0′|s1′))]]. aligned _M_src(π)&- _M_tar(π)≤\\ & 2γ r_max(1-γ)^2E_ _src^π [ E_P_src [D_KL(P_src(s _0|s _1)||P_tar(s _0|s _1)) ] ]. aligned (15) We further introduce a conclusion proven in PAR [18]: DKL(Psrc(s1′|s0′)||Ptar(s1′|s0′))=DKL(Psrc(s0′|s1′)||Ptar(s0′|s1′))+ℍ(ssrc′)−ℍ(star′). aligned D_KL(P_src(s _1|s _0)||&P_tar(s _1|s _0))=\\ &D_KL(P_src(s _0|s _1)||P_tar(s _0|s _1))+H(s _src)-H(s _tar). aligned (16) Therefore, the performance bound can be rewritten as follows, ηℳsrc(π)−ηℳtar(π)≤2γrmax(1−γ)2ρsrcπ[Psrc[DKL(Psrc(s1′|s0′)||Ptar(s1′|s0′))]]+2γrmax(1−γ)2ρsrcπ[Psrc[ℍ(ssrc′)−ℍ(star′)]]. aligned _M_src(π)-& _M_tar(π)≤\\ & 2γ r_max(1-γ)^2E_ _src^π [ E_P_src [D_KL(P_src(s _1|s _0)||P_tar(s _1|s _0)) ] ]+\\ & 2γ r_max(1-γ)^2E_ _src^π [ E_P_src [H(s _src)-H(s _tar) ] ]. aligned (17) This performance bound is consistent with the performance bound of PAR, which indicates that PAR can be considered as a special case of our method. However, the one-to-one representation mapping assumption may not hold in practice, especially in stochastic environments, which limits the application of PAR. In contrast, our method does not rely on this assumption and can handle more general scenarios. We validate this point in environments with stochastic dynamics. Noises with different standard deviation ς are introduced to the actions to simulate stochastic dynamics, and two tasks with kinematic shifts, hopper (broken joints) and walker (broken right foot), are considered. We evaluate the performance of DADiff-modify and PAR, which is presented in Table I. Notably, our method maintains robust performance even as the standard deviation ς increases, while PAR’s performance degrades significantly. We believe the decrease in PAR’s performance is due to its reliance on one-to-one representation assumptions, which may not hold in stochastic settings. TABLE I: Adaptation performance under stochastic dynamics controlled by the standard deviation parameter ς . Average return and standard deviation over five random seeds are reported. The best results are in bold, and performance change relative to the deterministic setting (ς=0.0 =0.0) is shown in parentheses. Environment ς DADiff-modify PAR hopper (broken joints) 0.00 2582.1± 251.6 2623.1± 105.2 0.01 2591.0± 159.2 (↑ 0.34%) 2398.3± 297.8 (↓ 8.57%) 0.02 2515.9± 101.8 (↓ 2.57%) 2328.7± 302.9 (↓ 11.22%) 0.03 2574.2± 280.6 (↓ 0.31%) 2406.1± 455.7 (↓ 8.27%) walker (broken right foot) 0.00 3390.4± 464.4 2943.3± 546.7 0.01 2879.3± 688.9 (↓ 15.08%) 2373.8± 1072.4 (↓ 19.35%) 0.02 2812.5± 934.6 (↓ 17.05%) 2825.8± 466.6 (↓ 3.99%) 0.03 3176.8± 796.4 (↓ 6.30%) 1613.9± 878.7 (↓ 45.17%) VI-B Reward Distribution Analysis Figure 5: Reward distribution comparison between the source-domain rewards before processing (Original) and after modification or selection (Processed). We further examine the reasons behind the superior performance of DADiff-select, in contrast to the severe failure of DADiff-modify on halfcheetah (no thighs) and hopper (big head) tasks, as illustrated in Figure 2. Specifically, we analyze the reward distributions of source-domain data after modification or selection. The results are presented in Figure 5. We find that DADiff-select generates a higher distribution in the low-reward region compared to DADiff-modify on both tasks. This suggests that the low-reward data may play a crucial role in these tasks, which can effectively guide the policy to avoid undesirable states and actions. VII Conclusion This work explores the problem of online dynamics adaptation in reinforcement learning from a generative modeling perspective. We first theoretically analyze the performance bound of a policy in the source and target domains, which is controlled by the generative trajectory discrepancy. Based on this analysis, we propose a novel method, DADiff, which utilizes diffusion models to measure the dynamics discrepancy and performs either reward modification or data selection to adapt to the target domain. Extensive experiments demonstrate that our method outperforms existing baselines in tasks with various shifts. APPENDIX VII-A Useful Lemmas Lemma VII.1 (Telescoping lemma.) Denote ℳ1=(,,P1,r,γ)M_1=(S,A,P_1,r,γ) and ℳ2=(,,P2,r,γ)M_2=(S,A,P_2,r,γ) as two MDPs with the same state and action spaces but different transition dynamics P1P_1 and P2P_2. The performance difference of a policy π evaluated in ℳ1M_1 and ℳ2M_2 can be expressed as: ηℳ1(π)−ηℳ2(π)=γ1−γρℳ1π(s,a)[s′∼P1[Vℳ2π(s′)]−s′∼P2[Vℳ2π(s′)]] _M_1(π)- _M_2(π)= γ1-γE_ρ^π_M_1(s,a) [E_s P_1[V_M_2^π(s )]-E_s P_2[V_M_2^π(s )] ] Proof. Please see Lemma 4.3 in SLBO [16] for a detailed proof. VII-B Proof of Theorem IV.2 Theorem VII.2 (Performance bound controlled by generative trajectory discrepancy.) Denote ℳsrcM_src and ℳtarM_tar as the source and target domains with different dynamics, respectively. The performance difference of any policy π evaluated in ℳsrcM_src and ℳtarM_tar can be bounded as below, ηℳsrc(π)−ηℳtar(π)≤2γrmax(1−γ)2ρsrcπ[Psrc[DKL(Psrc(sK′∣s,a)∥Ptar(sK′∣s,a))]]⏟(a):initiallatentstatedeviation+2γrmax(1−γ)2ρsrcπ[Psrc[∑k=1KDKL(Psrc(sk−1′∣sk′,s,a)∥Ptar(sk−1′∣sk′,s,a))]]⏟(b):latentstatetransitionmismatch. aligned & _M_src(π)- _M_tar(π)≤\\ & 2γ r_max(1-γ)^2 E_ _src^π\! [ E_P_src\! [D_KL\! (P_src(s _K s,a)\,\|\,P_tar(s _K s,a) ) ] ]_(a):\ initial\ latent\ state\ deviation+\\ & 2γ r_max(1-γ)^2 E_ _src^π\! [ E_P_src\! [ _k=1^KD_KL\! (P_src(s _k-1 s _k,s,a)\,\|\,P_tar(s _k-1 s _k,s,a) ) ] ]_(b):\ latent\ state\ transition\ mismatch. aligned Proof. As the value function Vℳπ(s)V^π_M(s) estimates the expected return of a policy π starting from state s in domain ℳM, and the rewards are bounded, we have |Vℳπ(s)|≤rmax/(1−γ),∀s|V^π_M(s)|≤ r_max/(1-γ),∀ s. By using Lemma VII.1, we have: ηℳsrc _M_src (π)−ηℳtar(π)=γ1−γρsrcπ[Psrc[r(s,a)]−Ptar[r(s,a)]] (π)- _M_tar(π)= γ1-γE_ _src^π\! [E_P_src[r(s,a)]-E_P_tar[r(s,a)] ] =γ1−γρsrcπ[∫s0′Psrc(s0′|s,a)Vtarπ(s0′)−∫s0′Ptar(s0′|s,a)Vtarπ(s0′)s0′] = γ1-γE_ _src^π\! [ _s _0P_src(s _0|s,a)V_tar^π(s _0)- _s _0P_tar(s _0|s,a)V_tar^π(s _0)ds _0 ] ≤γ1−γρsrcπ[∫s0′(Psrc(s0′|s,a)−Ptar(s0′|s,a))|Vtarπ(s0′)|s0′] ≤ γ1-γE_ _src^π\! [ _s _0(P_src(s _0|s,a)-P_tar(s _0|s,a)) |V_tar^π(s _0) |ds _0 ] ≤γrmax(1−γ)2ρsrcπ[∫s0′Psrc(s0′|s,a)−Ptar(s0′|s,a)ds0′] ≤ γ r_max(1-γ)^2E_ _src^π\! [ _s _0P_src(s _0|s,a)-P_tar(s _0|s,a)ds _0 ] =γrmax(1−γ)2ρsrcπ[∫s0:K′Psrc(s0:K′|s,a)−Ptar(s0:K′|s,a)ds0:K′] = γ r_max(1-γ)^2E_ _src^π\! [ _s _0:KP_src(s _0:K|s,a)-P_tar(s _0:K|s,a)ds _0:K ] =2γrmax(1−γ)2ρsrcπ[DTV(Psrc(s0:K′|s,a)||Ptar(s0:K′|s,a))] = 2γ r_max(1-γ)^2E_ _src^π\! [D_TV(P_src(s _0:K|s,a)||P_tar(s _0:K|s,a)) ] ≤2γrmax(1−γ)2ρsrcπ[DKL(Psrc(s0:K′|s,a)∥Ptar(s0:K′|s,a))] ≤ 2γ r_max(1-γ)^2E_ _src^π\! [ D_KL(P_src(s _0:K|s,a)\|P_tar(s _0:K|s,a)) ] (a) =2γrmax(1−γ)2ρsrcπ[Psrc[logPsrc(s0:K′|s,a)Ptar(s0:K′|s,a)]] = 2γ r_max(1-γ)^2E_ _src^π\! [ E_P_src\! [ P_src(s _0:K|s,a)P_tar(s _0:K|s,a) ] ] =2γrmax(1−γ)2ρsrcπ[Psrc[logPsrc(sK′|s,a)Ptar(sK′|s,a)+∑k=1KlogPsrc(sk−1′|sk′,s,a)Ptar(sk−1′|sk′,s,a)]] = 2γ r_max(1-γ)^2E_ _src^π\! [ E_P_src\! [ P_src(s _K|s,a)P_tar(s _K|s,a)+ _k=1^K P_src(s _k-1|s _k,s,a)P_tar(s _k-1|s _k,s,a) ] ] (b) ≤2γrmax(1−γ)2ρsrcπ[Psrc[DKL(Psrc(sK′|s,a)||Ptar(sK′|s,a))]]+ ≤ 2γ r_max(1-γ)^2E_ _src^π\! [ E_P_src\! [D_KL(P_src(s _K|s,a)||P_tar(s _K|s,a)) ] ]+ 2γrmax(1−γ)2ρsrcπ[Psrc[∑k=1KDKL(Psrc(sk−1′|sk′,s,a)||Ptar(sk−1′|sk′,s,a))]] 2γ r_max(1-γ)^2E_ _src^π\! [ E_P_src\! [ _k=1^KD_KL(P_src(s _k-1|s _k,s,a)||P_tar(s _k-1|s _k,s,a)) ] ] (c) where DTV(P||Q)D_TV(P||Q) is the total variation distance between two distributions P and Q, the step (a) holds by Pinsker’s inequality, the step (b) holds by the Markov property, and the step (c) holds by the subadditivity of the square root function. The proof shows that the performance difference can be controlled by the distributional divergence of latent states in generative trajectories. ACKNOWLEDGMENT The work was partially supported by NSF award #2442477, #2550203 and #2536297. The views and conclusions in this paper should not be interpreted as representing any funding agencies. References [1] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba (2016) Openai gym. arXiv preprint arXiv:1606.01540. Cited by: §V-A. [2] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox (2019) Closing the sim-to-real loop: adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), p. 8973–8979. Cited by: §I. [3] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §I. [4] A. Curtis, E. Li, M. Noseworthy, N. Gothoskar, S. Chitta, H. Li, L. P. Kaelbling, and N. E. Carey (2025) Flow-based domain randomization for learning and sequencing robotic skills. In Forty-second International Conference on Machine Learning, Cited by: §I. [5] L. Da, J. Turnau, T. P. Kutralingam, A. Velasquez, P. Shakarian, and H. Wei (2025) A survey of sim-to-real methods in rl: progress, prospects and challenges with foundation models. arXiv preprint arXiv:2502.13187. Cited by: §I. [6] B. Eysenbach, S. Asawa, S. Chaudhari, S. Levine, and R. Salakhutdinov (2020) Off-dynamics reinforcement learning: training for transfer with domain classifiers. arXiv preprint arXiv:2006.13916. Cited by: §I, §I, §V-A. [7] A. Fickinger, S. Cohen, S. Russell, and B. Amos (2021) Cross-domain imitation learning via optimal transport. arXiv preprint arXiv:2110.03684. Cited by: §I. [8] Y. Ge, A. Macaluso, L. E. Li, P. Luo, and X. Wang (2023) Policy adaptation from foundation model feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19059–19069. Cited by: §I. [9] Y. Guo, Y. Wang, Y. Shi, P. Xu, and A. Liu (2024) Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. Advances in Neural Information Processing Systems 37, p. 136326–136360. Cited by: §I. [10] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. Cited by: §IV-B, §V-A. [11] H. He, C. Bai, K. Xu, Z. Yang, W. Zhang, D. Wang, B. Zhao, and X. Li (2023) Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems 36, p. 64896–64917. Cited by: §I. [12] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §I, §I, §I. [13] Y. Jiang, C. Li, W. Dai, J. Zou, and H. Xiong (2023) Variance reduced domain randomization for reinforcement learning with policy gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (2), p. 1031–1048. Cited by: §I. [14] B. Kang, X. Ma, C. Du, T. Pang, and S. Yan (2023) Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 67195–67212. Cited by: §I. [15] C. Lu, P. Ball, Y. W. Teh, and J. Parker-Holder (2023) Synthetic experience replay. Advances in Neural Information Processing Systems 36, p. 46323–46344. Cited by: §I. [16] Y. Luo, H. Xu, Y. Li, Y. Tian, T. Darrell, and T. Ma (2018) Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03858. Cited by: §VII-A. [17] J. Lyu, C. Bai, J. Yang, Z. Lu, and X. Li (2024) Cross-domain policy adaptation by capturing representation mismatch. arXiv preprint arXiv:2405.15369. Cited by: §I, §I. [18] J. Lyu, C. Bai, J. Yang, Z. Lu, and X. Li (2024) Cross-domain policy adaptation by capturing representation mismatch. In Proceedings of the 41st International Conference on Machine Learning, p. 33638–33663. Cited by: §I, §IV-A, §V-A, §V-A, §VI-A, §VI-A. [19] J. Lyu, K. Xu, J. Xu, J. Yang, Z. Zhang, C. Bai, Z. Lu, X. Li, et al. (2024) Odrl: a benchmark for off-dynamics reinforcement learning. Advances in Neural Information Processing Systems 37, p. 59859–59911. Cited by: §V-A, §V-A. [20] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull (2020) Active domain randomization. In Conference on Robot Learning, p. 1162–1176. Cited by: §I. [21] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn (2018) Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv:1803.11347. Cited by: §I. [22] K. Pan, M. Chen, Y. Huang, X. Liu, and P. Hsieh Cross-domain reinforcement learning under distinct state-action spaces via hybrid q functions. Cited by: §I. [23] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel (2018) Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), p. 3803–3810. Cited by: §I. [24] D. S. Raychaudhuri, S. Paul, J. Vanbaar, and A. K. Roy-Chowdhury (2021) Cross-domain imitation from observations. In International conference on machine learning, p. 8902–8912. Cited by: §I. [25] R. B. Slaoui, W. R. Clements, J. N. Foerster, and S. Toth (2019) Robust visual domain randomization for reinforcement learning. arXiv preprint arXiv:1910.10537. Cited by: §I. [26] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, p. 2256–2265. Cited by: §I, §I, §I. [27] E. Todorov, T. Erez, and Y. Tassa (2012) Mujoco: a physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, p. 5026–5033. Cited by: §V-A. [28] L. L. P. Van, M. H. Nguyen, D. Kieu, H. Le, H. T. Tran, and S. Gupta (2025) DmC: nearest neighbor guidance diffusion model for offline cross-domain reinforcement learning. arXiv preprint arXiv:2507.20499. Cited by: §I. [29] L. L. P. Van, H. T. Tran, and S. Gupta (2024) Policy learning for off-dynamics rl with deficient support. arXiv preprint arXiv:2402.10765. Cited by: §I. [30] Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, et al. (2024) Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems 37, p. 54183–54204. Cited by: §I. [31] X. Wen, C. Bai, K. Xu, X. Yu, Y. Zhang, X. Li, and Z. Wang (2024) Contrastive representation for data filtering in cross-domain offline reinforcement learning. arXiv preprint arXiv:2405.06192. Cited by: §I. [32] Z. Wu, Y. Xie, W. Lian, C. Wang, Y. Guo, J. Chen, S. Schaal, and M. Tomizuka (2022) Zero-shot policy transfer with disentangled task representation of meta-reinforcement learning. arXiv preprint arXiv:2210.00350. Cited by: §I. [33] K. Xu, C. Bai, X. Ma, D. Wang, B. Zhao, Z. Wang, X. Li, and W. Li (2023) Cross-domain policy adaptation via value-guided data filtering. Advances in Neural Information Processing Systems 36, p. 73395–73421. Cited by: §I, §I, §I, §V-A. [34] Z. Xue, Q. Cai, S. Liu, D. Zheng, P. Jiang, K. Gai, and B. An (2023) State regularized policy optimization on data with dynamics shift. Advances in neural information processing systems 36, p. 32926–32937. Cited by: §I. [35] W. Zhao, J. P. Queralta, and T. Westerlund (2020) Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), p. 737–744. Cited by: §I. [36] Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang (2024) Madiff: offline multi-agent learning with diffusion models. Advances in Neural Information Processing Systems 37, p. 4177–4206. Cited by: §I.