Paper deep dive
LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL
Songyuan Zhang, Oswin So, Eric Yang Yu, Matthew Cleaveland, Peter Crowley-Dolen, Chuchu Fan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 10/8/2026, 6:37:49 AM
Summary
The paper introduces LASER, a novel offline reinforcement learning algorithm that addresses the out-of-distribution (OOD) action problem by combining latent-space flow policies with entropy regularization. LASER uses latent-space adjoint matching to optimize entropy-regularized policies within a constrained latent space, ensuring actions remain within the support of the behavior policy while avoiding policy collapse and critic exploitation. It achieves state-of-the-art performance on 40 OGBench tasks with fixed hyperparameters.
Entities (10)
Relation Signals (8)
LASER → uses → Entropy Regularization
confidence 95% · LASER, a framework for entropy-regularized support-constrained policy extraction
LASER → uses → Latent-Space Adjoint Matching
confidence 95% · We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching
LASER → evaluatedon → OGBench
confidence 90% · Through comprehensive experiments on 40 challenging OGBench tasks
Entropy Regularization → prevents → Policy Collapse
confidence 90% · discouraging premature collapse of the policy to local minima
LASER → solves → Out-of-Distribution Actions
confidence 90% · mitigate this by learning a behavior-cloning policy... LASER... avoids backpropagation through time... achieves state-of-the-art performance
LASER → outperforms → ReFORM
confidence 85% · outperforms the evaluated baselines, including those with task- and dataset-specific tuning
LASER → outperforms → DSRL
confidence 85% · outperforms the evaluated baselines, including those with task- and dataset-specific tuning
Flow Matching → usedin → Offline Reinforcement Learning
confidence 85% · learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While offline reinforcement learning (RL) enables policy optimization from static datasets without costly online interaction, it remains bottlenecked by the risk of executing out-of-distribution (OOD) actions. Recent approaches mitigate this by learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space. However, naively optimizing the latent policy can easily cause the policy to collapse into a brittle mode or exploit sharp artifacts of the learned critic. In this work, we find that entropy regularization is essential in latent-space RL for addressing these challenges. We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching to achieve entropy-regularized latent-space RL with expressive flow policies while avoiding backpropagation through time. Through comprehensive experiments on 40 challenging OGBench tasks with varying dataset qualities, we show that LASER achieves state-of-the-art performance. Notably, LASER uses fixed method-specific hyperparameters across all tasks and outperforms the evaluated baselines, including those with task- and dataset-specific tuning, which highlights the robust applicability of LASER. Project website: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.08989v1
- Canonical: https://arxiv.org/abs/2610.08989v1
Trouble viewing inline? Open PDF directly →
Full Text
604,103 characters extracted from source content.
Expand or collapse full text
LASER: Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL Songyuan Zhang Affiliation: MIT Email: szhang21@mit.edu Oswin So Affiliation: MIT Email: oswinso@mit.edu Eric Yang Yu Affiliation: MIT Email: eyyu@mit.edu Matthew Cleaveland Affiliation: MIT Lincoln Laboratory Email: chuchu@mit.edu Peter Crowley-Dolen Affiliation: MIT Lincoln Laboratory Email: matthew.cleaveland@l.mit.edu Chuchu Fan Affiliation: MIT Email: Peter.Crowley-Dolen@l.mit.edu Abstract While offline reinforcement learning (RL) enables policy optimization from static datasets without costly online interaction, it remains bottlenecked by the risk of executing out-of-distribution (OOD) actions. Recent approaches mitigate this by learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space. However, naïvely optimizing the latent policy can easily cause the policy to collapse into a brittle mode or exploit sharp artifacts of the learned critic. In this work, we find that entropy regularization is essential in latent-space RL for addressing these challenges. We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching to achieve entropy-regularized latent-space RL with expressive flow policies while avoiding backpropagation through time. Through comprehensive experiments on 40 challenging OGBench tasks with varying dataset qualities, we show that LASER achieves state-of-the-art performance. Notably, LASER uses fixed method-specific hyperparameters across all tasks and outperforms the evaluated baselines, including those with task- and dataset-specific tuning, which highlights the robust applicability of LASER. Project website: https://mit-realm.github.io/laser/. 00footnotetext: DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. 1 Introduction Offline reinforcement learning (RL) aims to learn policies that improve upon the behavior policy using only a static dataset generated by that behavior policy, without further environment interaction [1]. This creates a central tension: improvement requires deviating from the behavior policy while choosing actions where the dataset provides reliable supervision. For a fixed state, actions outside the dataset support may receive spuriously high predicted values, which can be exploited by policy optimization to yield out-of-distribution (OOD) actions that perform poorly when deployed [1]. A major line of work regularizes a statistical distance between the learned and behavior policies to mitigate OOD actions [2, 3, 4, 5, 6]. However, depending on the distance used, these methods can be either too restrictive and produce an overly conservative policy (e.g., KL divergence), or not restrictive enough and still produce OOD actions (e.g., Wasserstein distance, MMD distance) [7]. This has motivated a more direct alternative: enforce support constraints on the policy extraction process to guarantee that the actions produced lie within the support of the action distribution of the behavior policy [8, 7]. In particular, these methods learn a flow-based generative model using flow matching [9, 10, 11] to imitate the behavior policy, resulting in a compact latent space that maps to the behavior policy. Assuming the decoder maps the latent prior’s support into the behavior policy’s support, any latent policy restricted to that support produces in-support actions, while allowing their probabilities to differ from those of the behavior policy. Policy optimization performed on this latent space, either via gradient descent (as in [7]) or SAC [12] (as in [8]), yields a policy that satisfies support constraints by construction. In this work, we argue that support constraints are only part of the story. Once policy optimization is restricted to the support, the central difficulty becomes policy extraction: how should one reliably select high-value actions from the support without collapsing to brittle modes or exploiting sharp artifacts of the learned critic? Even when actions remain in support, the critic may be sharp, irregular, or locally overestimated, and a low-entropy extraction procedure can collapse onto brittle modes. We study entropy regularization as an explicit objective for support-constrained flow policy optimization. While entropy regularization is well studied in online RL for its role in improving exploration [13, 12] and robustness [13, 14], its role in offline RL, and in particular in support-constrained policy extraction, has not been isolated and made explicit. We argue that entropy regularization is also crucial in offline RL, not only in discouraging premature collapse of the policy to local minima, but also in smoothing the policy optimization landscape by regularizing the critic [15]. To this end, we propose Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL (LASER), a framework for entropy-regularized support-constrained policy extraction with latent-space flow policies. LASER builds on the latent support parameterization used in recent flow-based offline RL methods [7, 8], but replaces implicit or architecture-dependent regularization with an explicit entropy-regularized policy improvement objective. The main technical challenge is that computing and optimizing the entropy of a flow policy in latent space is nontrivial. We address this using adjoint matching [16], which enables entropy-regularized optimization of expressive flow policies in latent space while preserving the support constraints induced by the learned latent space. We make the following contributions: 1. We identify entropy regularization as a central ingredient for stable support-constrained policy extraction with flow-matching policies in offline RL. 2. We reinterpret recent support-constrained flow policy methods, including DSRL and ReFORM, through the lens of entropy regularization. 3. We introduce LASER, an algorithm for entropy-regularized offline policy improvement with latent-space flow policies that preserves dataset-support constraints while enabling smoother and more reliable policy extraction. 4. Extensive experiments demonstrate that LASER, using a fixed set of method-specific hyperparameters, achieves the strongest performance profiles among the evaluated flow-policy baselines. 2 Preliminaries 2.1 Offline RL We use Δ() (X) to denote the set of probability distributions over X, and gray notation for placeholder variables. A Markov Decision Process (MDP) is defined by ℳ=(,,r,ρ0,P,γ)M=(S,A,r, _0,P,γ), with state space S, action space A, reward function r(s,a):×→ℝr( [rgb]0.5352,0.5352,0.5352s, [rgb]0.5352,0.5352,0.5352a):S×A , initial state distribution ρ0∈Δ() _0∈ (S), transition dynamics function P(s′|s,a):×→Δ()P( [rgb]0.5352,0.5352,0.5352s | [rgb]0.5352,0.5352,0.5352s, [rgb]0.5352,0.5352,0.5352a):S×A→ (S), and discount factor γ∈[0,1]γ∈[0,1]. Given a dataset of trajectories D generated by some behavior policy πβ(a|s):→Δ() _β( [rgb]0.5352,0.5352,0.5352a| [rgb]0.5352,0.5352,0.5352s):S→ (A), offline RL aims to learn a policy πA(a|s):→Δ() _ A( [rgb]0.5352,0.5352,0.5352a| [rgb]0.5352,0.5352,0.5352s):S→ (A) that maximizes the expected discounted return R(πA)=τ∼ρπA(τ)[∑h=0Hγhr(sh,ah)]R( _ A)=E_τ ρ _ A( [rgb]0.5352,0.5352,0.5352τ) [ _h=0^Hγ^hr(s_h,a_h) ], where ρπA(τ)=ρ0(s0)πA(a0|s0)P(s1|s0,a0)⋯πA(aH|sH)ρ _ A(τ)= _0(s_0) _ A(a_0|s_0)P(s_1|s_0,a_0)·s _ A(a_H|s_H). Note that sampling trajectories during training is not allowed in the offline RL setting. The OOD Problem. A primary challenge in offline RL is handling OOD actions [1]. Standard actor-critic frameworks evaluate a policy πA _ A by estimating its state-action value function (Q-function) Q(s,a):×→ℝQ( [rgb]0.5352,0.5352,0.5352s, [rgb]0.5352,0.5352,0.5352a):S×A , representing the expected discounted return: QπA(s,a)=[∑h=0Hγhr(sh,ah)|s0=s,a0=a,ah∼πA(sh),∀h≥1].Q _ A(s,a)=E [\, _h=0^Hγ^hr(s_h,a_h) |s_0=s,a_0=a,a_h _ A(s_h),∀ h≥ 1 ]. (1) To learn QπAQ _ A, fitted Q-evaluation is typically used to minimize the temporal difference (TD) error: L(ϕ)=(s,a,s′)∼,a′∼πA(s′)[(r(s,a)+γQϕ^πA(s′,a′)−QϕπA(s,a))2],L(φ)=E_(s,a,s ) ,a _ A(s ) [ (r(s,a)+γ Q_ φ _ A(s ,a )-Q_φ _ A(s,a) )^2 ], (2) where Qϕ^πAQ_ φ _ A is a target network often updated via Polyak averaging [17]. However, because the dataset D is fixed, querying the target network with unfamiliar actions a′∼πA(s′)a _ A(s ) can yield unreliable and potentially large Q-values. Consequently, the policy may erroneously optimize toward these OOD actions [1]. To address this issue, the learned policy should satisfy the support constraint [5]: supp(πA(⋅|s))⊆supp(πβ(⋅|s)),∀s, ( _ A(·|s)) ( _β(·|s)), ∀ s, (3) where supp(p)≔x∣p(x)>0 (p) \x p(x)>0\ denotes the support of a distribution p. To achieve this, many prior offline RL algorithms regularize the statistical distance D(πA∥πβ)D( _ A\;\|\; _β) between the learned policy πA _ A and the behavior policy πβ _β, resulting in the following objective for policy updates: minπAs∼,a∼πA(s)[−QϕπA(s,a)+αD(πA∥πβ)], _ _ AE_s ,a _ A(s) [-Q_φ _ A(s,a)+α D( _ A\;\|\; _β) ], (4) where α is a hyperparameter for the regularization weight. However, depending on the statistical distance D, these approaches may fail to prevent OOD actions or may limit the improvement of πA _ A over the behavior policy πβ _β [7]. The weight α also requires task- and dataset-specific tuning [1, 18]. 2.2 Mitigating OOD Actions with Latent Space RL To mitigate the limitations of statistical distance regularization, recent works propose learning policies in a latent space ℒL [19, 8, 7]. Specifically, a decoder μθL → As(z):ℒ×→ _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A [rgb]0.5352,0.5352,0.5352s( [rgb]0.5352,0.5352,0.5352z):L×S is learned that maps samples z∼qLz q_ L drawn from the latent-space prior distribution qL(z)∈Δ(ℒ)q_ L( [rgb]0.5352,0.5352,0.5352z)∈ (L) to the dataset distribution πβ(⋅|s) _β(·|s), i.e., the pushforward of the prior qLq_ L through the decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A approximates the behavior policy πβ(⋅|s) _β(·|s). Note that the decoder is deterministic for a given state and latent input, and the stochasticity of the action-space policy comes from sampling the latent input. Then, instead of learning a policy πA(⋅|s) _ A(·|s) in the action space A, we can learn a latent-space policy πL(z|s):→Δ(ℒ) _ L( [rgb]0.5352,0.5352,0.5352z| [rgb]0.5352,0.5352,0.5352s):S→ (L) which induces a full policy in the action space via the pushforward through the decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A, i.e., πA(⋅|s)≔(μθL → As)♯πL(⋅|s), _ A(·|s) ( _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s)_ _ L(·|s), (5) where (⋅)♯(·)_ is the pushforward operator. This parameterization (5) transforms the difficult action-space support constraint (3) under the complex behavior policy πβ _β into a simple latent-space constraint: supp(πL(⋅|s))⊆supp(qL). ( _ L(·|s)) (q_ L). (6) Since qLq_ L is chosen to have simple support (e.g., a uniform distribution on a d-ball [7]), this constraint is much easier to enforce, thereby decoupling OOD action avoidance from policy optimization. We can thus define the resulting policy optimization problem in the latent space ℒL induced by the decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A. For a Q-function QϕπAQ_φ _ A in the action space (1), this induces a corresponding latent-space Q-function QϕπL(s,z):×ℒ→ℝQ_φ _ L( [rgb]0.5352,0.5352,0.5352s, [rgb]0.5352,0.5352,0.5352z):S×L as the pullback of QϕπAQ_φ _ A through the decoder, i.e., QϕπL(s,z)=QϕπA(s,μθL → A(z,s)).Q_φ _ L(s,z)=Q_φ _ A(s, _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z;s)). (7) As long as the latent-space support constraint (6) is satisfied (e.g., with tanh squashing, clipping or projection), any off-policy optimization algorithm, such as DDPG or TD3 in [19], SAC in [8], or even plain gradient ascent in [7], can then be used to optimize the latent policy πL _ L without needing to worry about OOD actions. Figure 1: Both support constraints and entropy regularization are essential. Without support constraints, the policy generates OOD actions that have unreliable Q-values and thus fail. Without entropy regularization, the Q-function exhibits strange artifacts and the agents get stuck at the wall, resulting in failures on the left side. By combining support constraints and entropy regularization, LASER avoids OOD actions, regularizes the Q-function, and encourages policy exploration, resulting in a successful policy. Figure 2: Q-function and policy are poorly learned without entropy regularization. We plot the learned Q-functions (lighter colors denote higher value) at the state marked by the white circle in Figure 1. Without entropy regularization, the policy becomes nearly deterministic and clusters at a false maximum which causes the agents to move up and get stuck at the wall (Figure 1). Adding noise to the policy in TD learning (target policy smoothing [20]) fixes the Q-function landscape, but the policy remains nearly deterministic and is suboptimal. With entropy regularization, the policy samples spread out more and yield a better regularized Q-function, enabling the agents to move around the wall and reach the target (Figure 1). Learning expressive decoders with flow matching. A key component of latent-space RL is the decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A. A poorly learned decoder may output actions outside the dataset support and defeat the purpose of latent-space RL, or be too simple to capture the behavior policy and limit the performance of the final policy. Recent works use flow matching [9, 10, 11] to learn μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A due to its ability to model complex multimodal behavior distributions [8, 7]. Flow matching learns a time-dependent velocity field vθL → A(z,t,s):ℒ×[0,1]×→ℒv_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( [rgb]0.5352,0.5352,0.5352z, [rgb]0.5352,0.5352,0.5352t; [rgb]0.5352,0.5352,0.5352s):L×[0,1]×S , parameterized by θL → A _ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A, that transports an easy-to-sample latent distribution qL∈Δ(ℒ)q_ L∈ (L) (e.g., standard Gaussian (0,Id)N(0,I^d)) to the pushforward action distribution (μθL → As)♯qL≈πβ(⋅|s)( _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s)_ q_ L≈ _β(·|s) over the time interval t∈[0,1]t∈[0,1], using the flow-matching loss: L → A(θL → A)=(s,a)∼,z∼qL,t∼[0,1][∥vθL → A(xt,t,s)−(a−z)∥2],xt=(1−t)z+ta.L_ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( _ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A)=E_(s,a) ,z q_ L,t [0,1] [ v_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(x_t,t;s)-(a-z) ^2 ], x_t=(1-t)z+ta. (8) The corresponding BC flow map ψθL → A(z,t,s):ℒ×[0,1]×→ℒ _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( [rgb]0.5352,0.5352,0.5352z, [rgb]0.5352,0.5352,0.5352t; [rgb]0.5352,0.5352,0.5352s):L×[0,1]×S for the velocity field vθL → Av_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A can be found as the solution to the ODE with initial condition z at t=0t=0 and velocity field vθL → Av_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A, i.e., ψθL → A(z,t,s)=z+∫0tvθL → A(ψθL → A(z,τ,s),τ,s)τ. _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z,t;s)=z+ _0^tv_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z,τ;s),τ;s) \!dτ. (9) The decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A is then defined as the BC flow map at time t=1t=1, i.e., μθL → A(z,s)≔ψθL → A(z,1,s) _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z;s) _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z,1;s). 3 Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL (LASER) 3.1 Entropy Regularization is Essential for Latent Space RL Given that latent-space RL methods claim to solve the OOD problem, one might be tempted to conclude that directly performing policy optimization in the latent space with support constraints results in a strong policy. However, we find this to be false. We illustrate this in a simple 2D reach-avoid environment (see Appendix E for details). Choosing πL _ L to be a flow policy as in [7], i.e., for a base distribution qBq_ B and velocity field vθB → Lv_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L, and corresponding flow map ψθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L, πL(⋅|s)=(μθB → Ls)♯qB,μθB → Ls(w)≔ψθB → Ls(w,1), _ L(·|s)=( _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s)_ q_ B, _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s(w) _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s(w,1), (10) we find that naïvely training πL _ L to maximize QϕπLQ_φ _ L with reparametrized gradients and enforcing the latent-space support constraint (6) via tanh squashing (Figure 1) yields a policy that causes the agent to become stuck at the wall and fail to reach the target (Figure 1). Moreover, the value function is poorly learned and exhibits artifacts. We explore this further by visualizing the learned Q-function and policy πA _ A at the state right before the agent gets stuck at the wall (Figure 2). Surprisingly, the learned policy is maximizing the learned Q-function and is not stuck in a local optimum despite the policy failing to reach the target. Using target policy smoothing (TPS) [20] to smooth the learned Q-function by adding noise to the target action a′a in the TD learning target (2) fixes the Q-function landscape (Figure 2), but the resulting policy is now suboptimal. This is a problem of the policy having insufficient exploration11 1 Here, exploration refers to sampling diverse in-support actions during policy optimization and critic updates., which can be mitigated by increasing policy entropy [12]. This motivates the use of an explicit entropy regularization term to obtain a policy with higher entropy, which not only helps with policy optimization via exploration, but also obviates the need for TPS in TD learning and yields a better regularized Q-function. Namely, for weight α>0α>0, we consider the following entropy-regularized objective to train the latent-space policy πL _ L: LB → L(θB → L)=−s∼[w∼qB[QϕπL(s,μθB → L(w,s))]+αℋ(μθB → L(⋅,s))].L_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L( _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L)=-E_s [E_w q_ B [Q_φ _ L(s, _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w;s)) ]+ ( _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(·;s)) ]. (11) The entropy term can also be optimized using reparametrized gradients by computing the log probability under the flow map μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L using the continuity equation [9] to obtain ℋ(μθB → L(⋅;s))=w∼qB[−logqB(w)+∫01∇xt⋅vθB → L(xt,t;s)dt],xt=ψθB → L(w,t;s).H( _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(·;s))=E_w q_ B [- q_ B(w)+ _0^1 _x_t· v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(x_t,t;s) \!dt ], x_t= _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w,t;s). (12) We discuss a more efficient method that does not require backpropagation through the flow map in the next subsection. Adding explicit entropy regularization improves both the Q-function and the policy (Figure 2), resulting in a policy that successfully guides the agent around the wall to the target in the simple illustrative environment (Figure 1). Remark 3.1. Examining recent latent-space RL methods, we find that their policy extraction does include an entropy regularization term despite its importance not being made explicit in their papers. This is done either explicitly, as in DSRL [8], which uses a Gaussian latent policy and an SAC-style objective [12], or implicitly as in ReFORM [7], where the geometric reflection term prevents latent collapse (see Section C.1). However, using an expressive flow policy for the latent-space policy and explicitly considering entropy regularization enables our method to empirically Pareto-dominate Gaussian policies (Figure 3) and maximize the Q-function better than other implicit entropy regularization schemes (Section C.1). A comparison of methods with no, implicit, and explicit entropy regularization across 20 tasks in more complex manipulation environments further supports the benefit of explicit regularization (Section F.5). Figure 3: Flow latent policy + adjoint matching Pareto-dominates a squashed Gaussian latent policy. For every entropy and Q-value achieved by the Gaussian latent policy, there exists an operating point achievable by the flow latent policy that has both higher entropy and higher Q-value. Why use a flow policy for πL _ L? While a Gaussian latent-space policy significantly simplifies entropy-regularized optimization, we motivate our use of a flow model with two arguments. (1) A Gaussian πL _ L produces unimodal latents, which restricts the ability of the full policy in action space to capture multimodality [7]. (2) While high-entropy policies facilitate robust Q-function learning and help avoid local minima, they often yield lower Q-values. By being more expressive, flow policies empirically Pareto-dominate Gaussian policies on the performance-entropy tradeoff (Figure 3). 3.2 Efficient Entropy-Regularized Policy Optimization via Adjoint Matching While (11) can be used as-is to train the latent policy πL _ L with reparametrized gradients, computing the gradient requires backpropagating through the flow map μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L (10) and the entropy (12), both of which are solutions to ODEs and require backpropagation through time (BPTT), which is computationally expensive and prone to numerical instability [21, 22, 23]. To circumvent these computational bottlenecks, we adopt an alternative approach. Instead of optimizing the loss directly via BPTT, we derive the optimal latent-space policy πL(⋅|s)⋆=(μθB → Ls)♯⋆qB _ L (·|s)=( _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s) _ q_ B that minimizes the entropy-regularized policy loss (11) in the typical case where the support of the latent-space prior distribution qLq_ L has finite Lebesgue measure (e.g., a d-ball). Lemma 3.2. Fix a state s∈s and parameters of the critic and the decoder. Suppose supp(qL) (q_ L) has finite positive Lebesgue measure, and the Q-function QϕπL(s,⋅)Q _ L_φ(s,·) is measurable and has a finite upper bound on supp(qL) (q_ L). Then, among latent densities supported on supp(qL) (q_ L), the optimal distribution πL∗ _ L^* that minimizes the loss (11) is given by the following Q-tilted distribution: πL(⋅|s)⋆∝qLunif(⋅)exp(QϕπL(s,⋅)/α), _ L (·|s) q_ L^unif(·) (Q _ L_φ(s,·)/α ), (13) where qLunif=(supp(qL))q_ L^unif=U( (q_ L)) is the uniform distribution on the support of qLq_ L. See Appendix B for the proof. Note that this is a statewise, fixed-critic optimum and it does not establish convergence of the joint actor-critic updates. With Lemma 3.2, we can optimize (11) by learning to sample from the optimal distribution πL⋆ _ L , which can be done efficiently without BPTT using adjoint matching [16]. Namely, suppose both πL _ L and qLunifq_ L^unif are flow models with a common base distribution qBq_ B, i.e., (10) holds for πL _ L, and for flow map ψB → Lbase _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base generated by velocity field vB → Lbasev_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base, qLunif=(μB → Lbase)♯qB,μB → Lbase(w)=ψB → Lbase(w,1),q_ L^unif=( _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base)_ q_ B, _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w)= _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w,1), (14) and let pv(wtt∈[0,1])∈Δ(ℬ[0,1])p_v(\ [rgb]0.5352,0.5352,0.5352w_t\_t∈[0,1])∈ (B^[0,1]) denote the trajectory law of the following memoryless stochastic differential equation (SDE) with control v and w0∼qBw_0 q_ B: dwt=(2v(wt,t,s)−wt/t)dt+σtdBt, \!dw_t= (2\,v(w_t,t;s)-w_t/t )dt+ _t \!dB_t, (15) where σt=σ02(1−t)/t _t= _0 2(1-t)/t is the memoryless noise schedule and BtB_t is standard Brownian motion. We fix the state s and suppress it in pvp_v. For the Gaussian base qB=(0,σ02I)q_ B=N(0, _0^2I) used here, the reference SDE with v=vB → Lbasev=v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base shares the time marginals of the reference flow ODE in the exact flow-matching formulation, including the terminal marginal qLunifq_ L^unif. Moreover, consider the tilted distribution over SDE trajectories pv⋆p_v , defined as pv⋆(wtt∈[0,1])∝pvB → Lbase(wtt∈[0,1])exp(QϕπL(s,w1)/α),p_v (\w_t\_t∈[0,1]) p_v_ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(\w_t\_t∈[0,1]) (Q _ L_φ(s,w_1)/α ), (16) where (13) now corresponds to the marginal of pv⋆p_v at t=1t=1. Then, pv⋆p_v can be learned without BPTT by repeatedly optimizing a regression loss using a “special estimate” of v⋆v as the regression target. Specifically, the optimal control v⋆v realizing (16) solves the following stochastic optimal control (SOC) problem with quadratic control cost and terminal cost given by the negative Q-function [16]: minvs∼,wt[∫012σt2‖v(wt,t,s)−vB → Lbase(wt,t)‖2t−QϕπL(s,w1)/α]. _vE_s ,\w_t\ [ _0^1 2 _t^2 \|v(w_t,t;s)-v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w_t,t) \|^2dt-Q_φ _ L(s,w_1)/α ]. (17) Then, given a rollout wtt∈[0,1]\w_t\_t∈[0,1] sampled with control v, we can construct the following estimate v^⋆ v of the optimal control v⋆v : v^⋆(wt,t,s)=vB → Lbase(wt,t)−σt2g~t/2, v (w_t,t;s)=v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w_t,t)- _t^2 g_t/2, (18) where g~t g_t is the “lean” adjoint state computed by integrating the reverse ODE using wtt∈[0,1]\w_t\_t∈[0,1] dg~t=−(∇wt[2vB → Lbase(wt,t)−wt/t])⊤g~tdt,g~1=−∇w1QϕπL(s,w1)/α.d g_t=- ( _w_t [2v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w_t,t)-w_t/t ] ) g_tdt, g_1=- _w_1Q _ L_φ(s,w_1)/α. (19) In implementation, we substitute the target critic Qϕ^πLQ_ φ _ L for QϕπLQ_φ _ L in these policy-improvement expressions (Appendix D). Using the estimate v^⋆ v as the regression target in the following regression loss then results in a unique fixed point at the optimal control v⋆v [16], where we replace the current velocity field v with the trainable velocity field vθB → Lv_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L to be optimized: LAM(θB → L)=s∼,wt[12∫01∥2σt(vθB → L(wt,t,s)−v^⋆(wt,t,s))∥2t].L_AM( _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L)=E_s ,\w_t\ [ 12 _0^1 2 _t (v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_t,t;s)- v (w_t,t;s) ) ^2dt ]. (20) Importantly, optimizing (20) does not require backpropagation through the flow and only requires the solution of an ODE to compute the adjoint state g~t g_t for the regression target v^⋆ v , circumventing the computational and numerical stability issues of BPTT. For a more comprehensive mathematical treatment of adjoint matching, we refer readers to [16]. Figure 4: LASER Algorithm. () We train a flow-matching decoder μθL → A _ _ L 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.73pt to1.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.22 L 2.39 2.22 L 2.39 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A that maps samples from the uniform distribution in the d-ball qL=qLunif=(ℬld)q_ L=q_ L^unif=U(B_l^d) in latent space to the dataset actions in action space. () We then learn a base latent flow μB → Lbase _ B 2.0mu to5.95pt to4.54pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.28 L 8.23 6.28 L 8.23 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.95pt to4.54pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.28 L 8.23 6.28 L 8.23 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.38pt to3.15pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.36 L 4.68 4.36 L 4.68 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base that maps the prior distribution qBq_ B in the base space to the uniform distribution in the latent space qLunifq_ L^unif. () The base latent flow is then fine-tuned via adjoint matching, which results in a latent flow policy μθB → L _ _ B 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.73pt to1.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.22 L 2.39 2.22 L 2.39 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L that generates the final action distribution πA=(μθL → A)♯(μθB → L)♯qB _ A=( _ _ L 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.73pt to1.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.22 L 2.39 2.22 L 2.39 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A)_ ( _ _ B 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.73pt to1.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.22 L 2.39 2.22 L 2.39 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L)_ q_ B. 3.3 Algorithm We now present Latent Space Adjoint Matching for Support-Constrained Entropy-Regularized Offline RL (LASER). LASER combines the aforementioned insights to perform latent-space RL with an expressive flow decoder and efficient entropy-regularized policy optimization via adjoint matching. • Following [7], we choose the latent prior qL=(ℬld)q_ L=U(B_l^d) to be the uniform distribution on the ball ℬldB_l^d of radius l in ℝdR^d, which has finite Lebesgue measure and satisfies the conditions of Lemma 3.2. • Adjoint matching requires qLunifq_ L^unif, the uniform distribution over the support of qLq_ L, to also be a flow model. While an analytical vector field can be derived since qLunif=qLq_ L^unif=q_ L is the uniform distribution over the ball, it involves special functions with no easy closed-form expression (see Section C.2). Instead, we train a flow model via flow matching to match qLunifq_ L^unif with the same base distribution qBq_ B as πL _ L to simplify the implementation. We now summarize LASER (Figure 4), which consists of four components trained jointly: (1. Flow-matching decoder) We train a state-conditioned flow decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A using flow matching to map samples from the uniform distribution on the d-ball qLq_ L to dataset actions, i.e., behavior cloning (Section 2.2). (2. Flow matching for Lq_ L unif) We train a flow model μB → Lbase _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base to match the uniform distribution qLunifq_ L^unif in latent space from qBq_ B to use in adjoint matching. (3. Critic learning) We learn a critic QϕπAQ _ A_φ using the standard TD learning objective (2). (4. Entropy-regularized policy optimization) Finally, we perform entropy-regularized policy optimization in latent space to maximize the latent-space Q-function QϕπLQ _ L_φ (7) using adjoint matching (Section 3.2).22 2 In practice, we project the numerical latent flow output onto ℬldB_l^d before decoding. This support correction approximates the density-based formulation (Appendix D). We provide practical implementation details and a high-level algorithm for LASER in Appendix D. 4 Related Work We discuss the most closely related work in this section and defer a broader discussion of offline RL and generative policies to Appendix A. Distribution shift in offline RL. A central challenge in offline RL is distribution shift: learned value functions can overestimate actions outside the dataset support, causing policy improvement to select out-of-distribution actions with poor true value. Prior methods address this issue by regularizing the policy toward the behavior policy with statistical distances, learning pessimistic value functions, or explicitly restricting policy improvement to supported actions [5, 24, 25, 26, 4]. We follow the support-constrained perspective, using an expressive flow policy to enforce support by construction. Expressive generative policies for offline RL. Diffusion [27, 28] and flow-matching [9, 11, 10] models have recently been used as expressive alternatives to deterministic or Gaussian policies, especially in robotic imitation learning where action distributions are often multimodal [29, 30]. These policies have also been adapted to offline RL, where the goal is to improve beyond the behavior policy using a learned critic. Existing methods differ in how they use the critic: some use critic values to select, reweight, or clone high-value actions [31, 32, 33, 34], while others use critic gradients to directly improve the generated actions [35, 30, 21]. Most closely related to our optimization mechanism, [23] fine-tunes flow policies using critic gradients under a KL-regularized reward objective. Unlike these methods, which rely on statistical regularization to control distribution shift, LASER enforces support constraints through its latent action parameterization. Support-constrained policy extraction with latent-space RL. Recent methods avoid OOD actions by learning a latent action space whose decoder maps into the support of the behavior policy, then performing policy optimization in this latent space [19, 8, 7]. Our method builds on this idea, but revisits the optimization problem that remains once support has been controlled. In particular, we show that entropy regularization is important for preventing policy collapse and stabilizing policy extraction within the supported action set. Entropy regularization. Entropy regularization is a classical tool for smoothing policy optimization, improving robustness, and encouraging exploration [13, 12, 15]. In offline RL, though some works have noted its benefits for exploration during online fine-tuning [35, 36], it receives much less attention, perhaps due to the implicit entropy regularization effect from behavior policy regularization techniques. Our perspective is different: because support constraints are enforced by the policy parameterization, entropy regularization can be used for its standalone role in policy extraction. 5 Experiments We conduct experiments to answer the following research questions. (Q1): How does LASER perform against other offline RL algorithms with flow policies? (Q2): How sensitive is LASER to hyperparameters? (Q3): How does adjoint matching compare with BPTT? (Q4): Does the latent space decouple entropy regularization from behavior regularization? Details on implementation, baselines, environments, and additional results and ablations are in Appendix F. 5.1 Setup Environments. We evaluate our method on the OGBench offline RL benchmark [18]. The main evaluation suite spans four environments, covering both locomotion and manipulation domains, with five tasks per environment. Each task is evaluated on two datasets (clean and noisy), yielding a total of 4×5×2=404× 5× 2=40 evaluation settings. We additionally evaluate harder tasks in Section F.5. Baselines. Since recent studies have shown that flow-based policies outperform traditional alternatives [21, 37, 22, 23], we benchmark LASER against state-of-the-art flow-based methods. These include IFQL [32, 21], which combines a BC flow policy with an IQL critic and best-of-N action selection; FQL [21], which applies Wasserstein distance regularization; and QAM [23], which uses adjoint matching to approximate the Q-value-weighted BC action distribution, alongside its extension QAM-E [23] that learns an additional residual policy. We also compare against latent-space RL methods, specifically DSRL [8] with a Gaussian latent policy, and ReFORM [7] with a reflected flow latent policy. The method-specific hyperparameters of LASER, ReFORM, and IFQL remain fixed across all evaluated tasks, while those of the other baselines vary by environment and dataset (Table 3). All methods share the environment-dependent action chunk lengths and pessimistic backup weights (Table 2). 5.2 Results (Q1): LASER achieves state-of-the-art performance using a single set of method-specific hyperparameters. In Figure 5, we present the performance profiles [38] aggregating the 40 main evaluation settings across both the clean and noisy datasets. The profiles show that LASER outperforms all evaluated baselines in aggregate. It achieves at least 95%95\% of the best mean success rate in 34/4034/40 settings and a mean success rate of 100%100\% in 13/4013/40 settings. A paired statistical comparison supports the aggregate advantage (global p=1.35×10−3p=1.35× 10^-3), with details in Section F.5. Notably, on the noisy datasets, LASER and ReFORM achieve the highest mean success rates, underscoring the advantage of enforcing support constraints, which does not restrict policy improvement. Furthermore, LASER significantly improves upon ReFORM by using explicit entropy regularization and avoiding the static-flow behavior (Sections C.1 and F.5). Figure 5: Performance profile plots for the main evaluation suite with different datasets. For a given success rate τ (x-axis), we plot the fraction of runs with a success rate ≥τ≥τ [38]. Shading shows pointwise 95%95\% bootstrap confidence intervals, computed by resampling training seeds within each task 2000 times. LASER attains the strongest empirical performance profile on both datasets. Figure 6: Ablation studies in cube-double-play-singletask-task2-v0. Moderate values of the inverse temperature 1/α∈[5,20]1/α∈[5,20] yield comparable results. Adjoint matching achieves better and more stable results than BPTT. Using a different qLq_ L yields comparable final success rates in this task, consistent with the proposed separation between entropy regularization and behavior regularization. Lines show mean success rates, and shading spans the minimum and maximum across runs. (Q2): The key hyperparameter of LASER is the inverse temperature. Using a constant set of method-specific hyperparameters for LASER across all tasks demonstrates its robustness to different tasks and datasets. Nevertheless, we analyze the sensitivity of LASER to the inverse temperature 1/α1/α (Figure 6). Values of 1/α∈[5,20]1/α∈[5,20] yield comparable results. At 1/α=31/α=3, performance drops on both datasets, whereas 1/α=301/α=30 degrades performance on clean data but remains competitive on noisy data (Table 6). We find that 1/α=101/α=10 yields stable results for all environments. We also compare LASER with Q-only optimization and TPS in this ablation and find that LASER with 1/α∈[5,20]1/α∈[5,20] outperforms both of them (Table 6). In addition, we analyze sensitivity to σ0 _0 and find that LASER is robust to this hyperparameter (Section F.5). (Q3): Adjoint matching achieves better results and is faster than BPTT using reparametrized gradients. A key advantage of adjoint matching is that it circumvents the need to perform BPTT on (12), which is computationally expensive and numerically unstable. We validate this empirically and observe that adjoint matching yields more stable training and stronger performance compared to BPTT (Figure 6) while training 3.6×3.6× faster (123it/s123\,it/s vs 34it/s34\,it/s). (Q4): The latent space successfully decouples entropy regularization from behavior regularization. To understand the improvements of LASER, we change qLq_ L to a distribution with the same support but a different density, which changes the behavior regularization effect but keeps the same latent-space entropy regularization (see Section F.5). Both runs retain the uniform entropy reference on the same latent ball. Changing the decoder prior can still alter the learned latent-to-action mapping and learning dynamics, while leaving the entropy regularizer unchanged. We observe that changing the decoder prior yields comparable final success rates in this task, consistent with the proposed separation between latent-space entropy regularization and behavior regularization (Figure 6). 6 Conclusion We introduced LASER, an entropy-regularized approach to support-constrained flow policy optimization for offline RL. Our key insight is that support constraints and entropy regularization solve different problems: the former prevent OOD actions, while the latter stabilizes policy extraction within the learned dataset support. By combining a flow-based decoder with latent-space adjoint matching, LASER effectively helps prevent mode collapse, smooths the optimization landscape, and avoids local minima, demonstrating state-of-the-art performance on the OGBench suite with fixed method-specific hyperparameters. Limitations and Future Work. We identify several promising avenues for extension. First, although adjoint matching circumvents the need for BPTT, solving the ODEs needed to sample from the flow model during training remains computationally intensive for deep neural ODEs. This computational cost could be reduced through distillation [22]. Second, by relying on the decoder to enforce support constraints, LASER inherits OOD errors made by the decoder itself. Integrating behavior cloning methods with stricter support constraints, diagnosing when the BC model generates OOD errors, or applying a pre-trained robust BC model could mitigate this. In addition, LASER uses a simple value-function learning method and an actor-critic architecture similar to those in [21]. Both could be improved using ideas from [39, 40, 41, 37]. Broader Impacts. Learning policies from existing datasets may reduce the cost and risks of online data collection in robotics. However, support constraints alone do not guarantee safe behavior, as datasets may contain unsafe actions and learned decoders can make errors. Deployment under distribution shift could therefore cause physical harm. Safety validation and monitoring remain necessary before deploying these policies on physical systems. Acknowledgements This material is based upon work supported by the Under Secretary of War for Research and Engineering under Air Force Contract No. FA8702-15-D-0001 or FA8702-25-D-B002. In addition, Zhang, So, and Fan are supported by the National Science Foundation (NSF) CAREER Award #CCF-2238030. This work was also supported in part by a grant from Amazon. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of War for Research and Engineering. ©2026 Massachusetts Institute of Technology. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work. References [1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. [2] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018. [3] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019. [4] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829–23851. PMLR, 2023. [5] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019. [6] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. [7] Songyuan Zhang, Oswin So, H M Sabbir Ahmad, Eric Yang Yu, Matthew Cleaveland, Mitchell Black, and Chuchu Fan. ReFORM: Reflected flows for on-support offline RL via noise manipulation. In The Fourteenth International Conference on Learning Representations, 2026. [8] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506.15799, 2025. [9] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. [10] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023. [11] Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571, 2022. [12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870. PMLR, 2018. [13] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010. [14] Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022. [15] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, pages 151–160. PMLR, 2019. [16] Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861, 2024. [17] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992. [18] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In International Conference on Learning Representations (ICLR), 2025. [19] Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719–1735. PMLR, 2021. [20] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596. PMLR, 2018. [21] Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning. In International Conference on Machine Learning (ICML), 2025. [22] Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kianté Brantley, and Wen Sun. Scaling offline RL via efficient and expressive shortcut models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [23] Qiyang Li and Sergey Levine. Q-learning with adjoint matching. In The Fourteenth International Conference on Learning Representations, 2026. [24] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020. [25] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2022. [26] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278–31291, 2022. [27] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. [28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. [29] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023. [30] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. [31] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023. [32] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023. [33] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195–67212, 2023. [34] Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR 2025), 2025. [35] Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via Q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945–53968, 2024. [36] Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07986, 2025. [37] Bhavya Agrawalla, Michal Nauman, Khush Agrawal, and Aviral Kumar. floq: Training critics via flow-matching for scaling compute in value-based RL. arXiv preprint arXiv:2509.06863, 2025. [38] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304–29320, 2021. [39] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:40587–40609, 2023. [40] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-learning: Maxent RL without entropy. In International Conference on Learning Representations, 2023. [41] Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In International Conference on Machine Learning, pages 31406–31424. PMLR, 2024. [42] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019. [43] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020. [44] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. [45] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768–7778, 2020. [46] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative Q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. [47] Zhepeng Cen, Zuxin Liu, Zitong Wang, Yihang Yao, Henry Lam, and Ding Zhao. Learning from sparse offline datasets via conservative density estimation. In The Twelfth International Conference on Learning Representations, 2024. [48] Qingjun Wang, Hongtu Zhou, Hang Yu, Junqiao Zhao, Yanping Zhao, Chen Ye, Ziqiao Wang, and Guang Chen. Beyond penalization: Diffusion-based out-of-distribution detection and selective regularization in offline reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. [49] Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:5616–5630, 2023. [50] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023. [51] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677, 2023. [52] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth International Conference on Learning Representations, 2024. [53] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. [54] Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B Schön, and Per Mattsson. Entropy-regularized diffusion policy with Q-ensembles for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:98871–98897, 2024. [55] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 9(4):3116–3123, 2024. [56] Franki Nguimatsia Tiofack, Théotime Le Hellard, Fabian Schramm, Nicolas Perrin-Gilbert, and Justin Carpentier. Guided flow policy: Learning from high-value actions in offline reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. [57] Jongseong Chae, Jongeui Park, Yongjae Shin, Gyeongmin Kim, Seungyul Han, and Youngchul Sung. Flow actor-critic for offline reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026. [58] Xiaoyuan Cheng, Haoyu Wang, Wenxuan Yuan, Ziyan Wang, Zonghao Chen, Li Zeng, and Zhuo Sun. Fisher decorator: Refining flow policy via a local transport map. arXiv preprint arXiv:2604.17919, 2026. [59] He Zhang, Ying Sun, and Hui Xiong. Goldenstart: Q-guided priors and entropy control for distilling flow policies. In The Fourteenth International Conference on Learning Representations, 2026. [60] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021. [61] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. [62] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825–22855. PMLR, 2023. [63] Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator. arXiv preprint arXiv:2505.23458, 2025. [64] Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, and Amy Zhang. Value gradient flow: Behavior-regularized RL without regularization. In The Fourteenth International Conference on Learning Representations, 2026. [65] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC. In International Conference on Machine Learning, pages 8489–8510. PMLR, 2023. [66] Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? Estimating uncertainties for offline RL through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267–18281, 2022. [67] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. [68] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016. [69] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Appendix A Extended Related Work Mitigating distribution shift in offline RL. A central challenge in offline RL is mitigating distribution shift: learned value functions can overestimate actions that are not in the dataset, causing the policy to select these out-of-distribution actions. Classical approaches address this issue by either regularizing the policy to remain close to the behavior policy with statistical distances (e.g., KL divergence [2, 3, 6, 42, 43, 44, 45, 25], MMD distance [5] or Wasserstein distance [6]), or regularizing the value function to be pessimistic on out-of-distribution actions [24, 25, 46, 39, 47, 48]. Another family of methods addresses the OOD problem by penalizing actions that fall outside the support of the behavior policy [5, 26, 4, 49]. However, these methods rely on heuristic sample-based proxies [5] or Gaussian density approximations [26, 4, 49] which can fail to capture expressive multimodal behavior policies. Expressive generative policies for offline RL. Diffusion [27, 28] and flow-matching [9, 11, 10] models have recently been used as expressive alternatives to deterministic or Gaussian policies, especially in robotic imitation learning where action distributions are often multimodal [29, 50, 51, 30]. These policies have also been adapted to offline RL, where the goal is to improve beyond the behavior policy using a learned critic. However, because flow and diffusion policies generate actions through an iterative sampling process, critic-based optimization is more challenging than for standard Gaussian actors. Existing reward-optimization methods for generative policies can be broadly divided by how they use the learned critic: critic-evaluation methods, which use only scalar values Q(s,a)Q(s,a) to select or reweight actions, and critic-gradient methods, which use ∇aQ(s,a) _aQ(s,a) to directly improve generated actions. (i) Critic-evaluation methods include sampling-based methods that imitate the behavior policy and use the critic to sample from the reward-tilted distribution either at test time [31, 32] or during training, when the sampled actions serve as imitation targets [52]. These methods often reliably improve the quality of the extracted policy at greater computational cost and depend on the quality of the behavior policy. Weighted behavioral cloning methods also fall under this category. These methods use the critic to assign per-sample weights in the diffusion or flow-matching loss, selectively cloning high-value actions in the dataset [33, 35, 34], but generally do not optimize the learned policy towards the reward-tilted distribution [23]. (i) Some critic-gradient methods combine a behavior-cloning term with a term that increases the critic-estimated value. These methods require backpropagation through the iterative sampling process [30, 53, 54, 55], which can be computationally expensive and unstable [21, 22, 23]. Other methods learn a distilled policy [21, 56, 57, 58, 59] at the expense of policy expressiveness. Another line of work uses guidance [60, 61] during the iterative sampling process to favor high-value actions [62, 63, 64], but these methods rely on approximations that do not have theoretical guarantees [65, 23]. Most closely related to our optimization mechanism, [23] provides an efficient way to fine-tune flow-based policies using critic gradients and a KL-regularized reward objective via adjoint matching [16]. However, unlike critic-evaluation methods where samples from the learned policy are either directly cloned (sampling-based methods) or selectively cloned (weighted BC methods), these methods rely on statistical distance regularization to avoid the OOD problem (e.g., L2 / Wasserstein distance [21] or KL regularization [23]). A key challenge is that the regularization weight is a hyperparameter that significantly affects the performance and needs to be tuned for each task and dataset [18, 21]. Our method falls under the critic-gradient category as we also use adjoint matching to optimize the policy with critic gradients under KL regularization. However, to address the OOD problem, LASER relies on the latent space to enforce the support constraints by construction instead of using KL regularization toward the behavior policy, as in [23], which can be too restrictive in cases where optimal actions have very low probability under the behavior policy [5]. Support constraints in offline RL. Instead of addressing the OOD problem with regularization, another line of work proposes to address the OOD problem by learning a bounded latent space that maps to the support of the behavior policy, then performing policy extraction in this latent space to restrict the support of the learned policy by construction [19, 8, 7]. Our work follows this support-constrained perspective, but revisits the optimization problem that remains after support has been controlled. Namely, we find that entropy regularization is essential to prevent policy collapse, and leverage adjoint matching [16] to perform entropy-regularized policy extraction in latent space with an expressive flow policy that achieves a better performance-entropy tradeoff than Gaussian policies. Entropy-regularized policy extraction. The role of entropy regularization has been well identified in online RL as a way to improve exploration [13, 12] and policy robustness [13, 14] and to smooth the optimization landscape [15]. In offline RL, while some works note the benefits of entropy regularization for exploration [35, 36], the importance of entropy regularization has been less emphasized, perhaps due to the implicit entropy regularization effect from various behavior policy regularization techniques. In our setting, the learned decoder and latent support constraint handle support control, so no separate behavior-policy regularization term is needed. This makes the role of entropy regularization in policy extraction explicit. Appendix B Proof of Lemma 3.2 For a Lebesgue-measurable set X⊆ℝdX ^d, let λ(X)λ(X) denote its Lebesgue measure. We first prove the following technical lemma to express entropy as a constant minus a KL divergence. Lemma B.1. Fix a state s∈s . For brevity, we omit the state dependence in the policy. Let ℒq≔supp(qL)⊆ℒL_q (q_ L) denote the support of qLq_ L. Suppose ℒqL_q has finite positive Lebesgue measure, i.e., 0<λ(ℒq)<∞0<λ(L_q)<∞, and πL _ L has a density and satisfies the latent-space support constraints (6), i.e., supp(πL)⊆supp(qL). ( _ L) (q_ L). (21) Let qLunifq_ L^unif be the uniform distribution over supp(qL) (q_ L), i.e., qLunif(z)=1/λ(ℒq)z∈ℒq0otherwise.q_ L^unif(z)= cases1/λ(L_q)&z _q\\ 0&otherwise cases. (22) Then, the entropy of πL _ L can be expressed as ℋ(πL)=−DKL(πL∥qLunif)+logλ(ℒq),H( _ L)=-D_KL( _ L\|q_ L^unif)+ λ(L_q), (23) where the second term in (23) is a constant independent of πL _ L. Proof. Expanding the KL divergence, DKL(πL∥qLunif) D_KL( _ L\|q_ L^unif) =∫ℝdπL(z)logπL(z)qLunif(z)z, = _R^d _ L(z) _ L(z)q_ L^unif(z) \!dz, (24) =∫ℒqπL(z)logπL(z)z−∫ℒqqLunif(z)logqLunif(z)z, = _L_q _ L(z) _ L(z) \!dz- _L_qq_ L^unif(z) q_ L^unif(z) \!dz, (25) =−ℋ(πL)−∫ℒqqLunif(z)log(1λ(ℒq))z, =-H( _ L)- _L_qq_ L^unif(z) ( 1λ(L_q) ) \!dz, (26) =−ℋ(πL)+logλ(ℒq). =-H( _ L)+ λ(L_q). (27) Rearranging the above equation yields the desired result. ∎ We are now ready to prove Lemma 3.2. Proof. Fix a state s∈s . For brevity, we omit the state dependence in the policy. We write QπL(z)Q _ L(z) for QϕπL(s,z)Q_φ _ L(s,z), held fixed with respect to the candidate policy. Restating the entropy-regularized optimization problem (11) in terms of the policy πL _ L directly, we wish to solve the following optimization problem over densities supported on ℒqL_q minπL−z∼πL[QπL(z)]−αℋ(πL). gathered _ _ L-E_z _ L [Q _ L(z) ]- ( _ L). gathered (28) By Lemma B.1, the entropy of πL _ L can be expressed as ℋ(πL)=−DKL(πL∥qLunif)+logλ(ℒq).H( _ L)=-D_KL( _ L\|q_ L^unif)+ λ(L_q). (29) Substituting this into (28) and dropping the constant term, we now have a KL-regularized optimization problem minπL−z∼πL[QπL(z)]+αDKL(πL∥qLunif). gathered _ _ L-E_z _ L [Q _ L(z) ]+α D_KL( _ L\|q_ L^unif). gathered (30) Now, define the partition function Z as Z≔∫ℝdqLunif(z)exp(1αQπL(z))z.Z _R^dq_ L^unif(z) ( 1αQ _ L(z) ) \!dz. (31) Since QπLQ _ L is measurable and bounded from above, there exists an m∈ℝm such that QπL(z)≤mQ _ L(z)≤ m for all z∈ℒqz _q. Hence, 0<exp(1αQπL(z))≤exp(1αm)<∞,0< ( 1αQ _ L(z) )≤ ( 1αm )<∞, (32) and thus 0<Z≤exp(1αm)<∞,0<Z≤ ( 1αm )<∞, (33) so πL⋆ _ L , defined as πL⋆(z)≔1ZqLunif(z)exp(1αQπL(z)), _ L (z) 1Zq_ L^unif(z) ( 1αQ _ L(z) ), (34) is well-defined. Since DKL(πL∥qLunif)=+∞D_KL( _ L\|q_ L^unif)=+∞ for any πL ／≪qLunif _ L to0.0pt 5.0pt -5.27776pt$ $ to0.0pt 5.0pt -5.27776pt$ $ to0.0pt 3.98611pt -4.45831pt$ $ to0.0pt 3.40282pt -3.95834pt$ $ q_ L^unif, it is sufficient to consider policies πL _ L that are absolutely continuous with respect to qLunifq_ L^unif. Note that for z∈ℒqz _q, πL⋆(z)qLunif(z)=1Zexp(1αQπL(z)),log(πL⋆(z)qLunif(z))=−logZ+1αQπL(z). _ L (z)q_ L^unif(z)= 1Z ( 1αQ _ L(z) ), ( _ L (z)q_ L^unif(z) )=- Z+ 1αQ _ L(z). (35) For any πL≪qLunif _ L q_ L^unif, we have DKL(πL∥πL)⋆ D_KL( _ L\| _ L ) =∫ℝdπL(z)logπL(z)πL⋆(z)z, = _R^d _ L(z) _ L(z) _ L (z) \!dz, (36) =∫ℝdπL(z)logπL(z)qLunif(z)z−∫ℝdπL(z)logπL⋆(z)qLunif(z)z, = _R^d _ L(z) _ L(z)q_ L^unif(z) \!dz- _R^d _ L(z) _ L (z)q_ L^unif(z) \!dz, (37) =DKL(πL∥qLunif)+logZ−1α∫ℝdπL(z)QπL(z)dz, =D_KL( _ L\|q_ L^unif)+ Z- 1α _R^d _ L(z)Q _ L(z) \!dz, (38) =DKL(πL∥qLunif)+logZ−1αz∼πL[QπL(z)]. =D_KL( _ L\|q_ L^unif)+ Z- 1αE_z _ L[Q _ L(z)]. (39) Thus, the objective can be written as the KL divergence to πL⋆ _ L plus a constant: −z∼πL[QπL(z)−αDKL(πL∥qLunif)]=αDKL(πL∥πL)⋆−αlogZ.-E_z _ L[Q _ L(z)-α D_KL( _ L\|q_ L^unif)]=α D_KL( _ L\| _ L )-α Z. (40) Since DKL(πL∥πL)⋆≥0D_KL( _ L\| _ L )≥ 0 for all πL _ L with equality if and only if πL=πL⋆ _ L= _ L , we conclude that πL⋆ _ L is the unique optimal solution to the optimization problem (28). ∎ Appendix C More Theoretical Results C.1 The Static Flow Issue and Implicit Entropy Regularization in ReFORM To enforce the latent-space support constraint (6), ReFORM [7] introduces a reflection term during the integration of the latent flow policy μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L. The corresponding modified Euler step is defined as: wk+1=w^k+1∈ℬldw^k+1+(1−w^k+1∈ℬld)(w^k+1−⟨vθB → L(wk,kΔt;s)Δt,nk+1⟩nk+1),w_k+1= 1\ w_k+1 _l^d\ w_k+1+(1- 1\ w_k+1 _l^d\) ( w_k+1- v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_k,k t;s) t,n_k+1 n_k+1 ), (41) where w^k+1=wk+vθB → L(wk,kΔt,s)Δt w_k+1=w_k+v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_k,k t;s) t denotes the standard Euler step, N is the total number of integration steps, k∈0,…,N−1k∈\0,…,N-1\, Δt=1N t= 1N, nk+1=w^k+1‖w^k+1‖n_k+1= w_k+1\| w_k+1\|, and ⟨⋅,⋅⟩ ·,· represents the inner product. Integrating this modified step yields the final latent variable z←wNz← w_N. While [7] proves that this modified integration step guarantees adherence to the latent support constraint (6), it introduces a degenerate behavior we term “static flow”. This occurs when the proposed step violates the boundary (w^k+1∉ℬld w_k+1 _l^d) and the flow velocity is strictly codirectional with the outward normal vector (vθB → L⇈nk+1v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L n_k+1). Under these conditions, the reflection term perfectly cancels the velocity step: wk+1 w_k+1 =w^k+1−⟨vθB → L(wk,kΔt,s)Δt,nk+1⟩nk+1 = w_k+1- v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_k,k t;s) t,n_k+1 n_k+1 (42) =(wk+vθB → L(wk,kΔt,s)Δt)−vθB → L(wk,kΔt,s)Δt=wk. = (w_k+v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_k,k t;s) t )-v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_k,k t;s) t=w_k. Consequently, the corrected step is stationary under this alignment condition. Rather than following the flow field vθB → Lv_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L to the boundary of ℬldB_l^d, the pushforward distribution of μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L unnaturally accumulates along the radial trajectory. To illustrate this phenomenon, we train ReFORM on the 2D toy environment introduced in Figure 3. The learned latent distribution and its resulting action distribution are visualized in Figure 7. We empirically observe that the latent mass inappropriately concentrates along two radii, resulting in a suboptimal action distribution. We also conduct additional experiments in cube-double in Section F.5 to measure how often the boundary correction is activated. Interestingly, we find that this static flow artifact unintentionally serves as an implicit entropy regularizer, which partially explains ReFORM’s empirical success. As demonstrated in Figure 7, the premature halting of trajectories prevents the latent distribution from collapsing to a single point, artificially maintaining the spread of the distribution. Figure 7: ReFORM fails to maximize the Q-value because of the “static flow” issue. For a given behavior policy and a ground-truth Q-function, the generated latent-space distribution (μθB → Ls)♯qB( _ _ B 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.45pt to3.22pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.45 L 4.77 4.45 L 4.77 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.42pt to2.25pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.11 L 3.34 3.11 L 3.34 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.73pt to1.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.22 L 2.39 2.22 L 2.39 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s)_ q_ B concentrates along radial directions aligned with the flow, rather than moving toward the boundary of ℬldB_l^d. This makes the flow static, resulting in suboptimal performance compared with LASER. C.2 The Ground-truth Base Velocity Field from the Base Space to the Latent Space As mentioned in Section 3.3, there exists a ground-truth velocity field vB → Lv_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L that maps samples from w∼qB=(0,σ02I)w q_ B=N(0, _0^2I) to z∼(ℬld)z (B_l^d). We show here how we can find such a ground-truth velocity field and why we choose not to use it in our implementation. Since both the base distribution qB=(0,σ02I)q_ B=N(0, _0^2I) and the target distribution (ℬld)U(B_l^d) are spherically symmetric, the optimal transport (OT) mapping between them operates purely along the radial direction. Let rw=‖w‖r_w=\|w\| denote the radius of a sample w from the base distribution, and rz=‖z‖r_z=\|z\| denote the radius in the latent space. The cumulative distribution function (CDF) for the uniform distribution on a d-dimensional ball of radius l is F(r)=(r/l)dF_U(r)=(r/l)^d for r∈[0,l]r∈[0,l]. Meanwhile, the radius of the Gaussian base distribution follows a scaled χ-distribution with d degrees of freedom, with its CDF denoted as Fχ(r)F_χ(r). By matching the quantiles of these radial distributions, we can construct the exact analytical mapping μB → L:ℝd→ℬld _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L:R^d _l^d as follows: μB → L(w)=w‖w‖F−1(Fχ(‖w‖))=wl⋅Fχ(‖w‖)1/d‖w‖. _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w)= w\|w\|F_U^-1(F_χ(\|w\|))=w l· F_χ(\|w\|)^1/d\|w\|. (43) At the origin, we define μB → L(0)=0 _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(0)=0 by continuity. However, although we have derived an exact analytical mapping μB → L _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L, it is nontrivial to derive the corresponding velocity field vB → L(wt,t,s)v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L( [rgb]0.5352,0.5352,0.5352w_t, [rgb]0.5352,0.5352,0.5352t; [rgb]0.5352,0.5352,0.5352s), i.e., the instantaneous velocity of a particle currently at location wtw_t at time t. Under standard optimal transport displacement interpolation, a particle travels in a straight line from its origin w to its destination μB → L(w) _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w) at a constant speed, yielding the trajectory: wt=(1−t)w+tμB → L(w).w_t=(1-t)w+t _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w). (44) While the velocity of this specific particle is trivially v(w)=μB → L(w)−wv(w)= _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w)-w, the velocity field must be expressed independently of the origin w. This requires inverting the trajectory equation to express w purely as a function of the current state wtw_t and time t. Because our mapping μB → L _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L is purely radial, the vectors w and wtw_t share the same direction, meaning w‖w‖=wt‖wt‖ w\|w\|= w_t\|w_t\| for w≠0w≠ 0. Let rw=‖w‖r_w=\|w\| and rt=‖wt‖r_t=\|w_t\|. Taking the norm of the trajectory equation yields the following relationship between the magnitudes: rt=(1−t)rw+t⋅l⋅Fχ(rw)1/d.r_t=(1-t)r_w+t· l· F_χ(r_w)^1/d. (45) To compute the velocity at wtw_t, we must find the inverse function gt(rt)g_t(r_t) that recovers the initial radius rwr_w given rtr_t and t. For each t∈[0,1]t∈[0,1], the radial map is strictly increasing, so it has a unique inverse on its image. For t>0t>0 and wt≠0w_t≠ 0, the exact Eulerian velocity field is: vB → Lbase(wt,t)=wt−wt=(‖wt‖−gt(‖wt‖)t)wt‖wt‖.v^base_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_t,t)= w_t-wt= ( \|w_t\|-g_t(\|w_t\|)t ) w_t\|w_t\|. (46) The values at t=0t=0 and wt=0w_t=0 are defined by continuity. Thus, although gtg_t exists, evaluating it generally requires numerical inversion. For intermediate times 0<t<10<t<1, computing gt(‖wt‖)g_t(\|w_t\|) would add a numerical solve at each velocity evaluation. We instead learn vB → Lbasev_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base via flow matching to avoid these repeated solves. Appendix D Practical Implementation of LASER We provide a high-level summary of LASER in Algorithm 1. Algorithm 1 LASER 1: while not converged do 2: ⊳ Train Critic 3: Sample batch (s,a,r,s′)∼\(s,a,r,s )\ 4: w∼(0,σ02Id)w (0, _0^2I_d) 5: a′=μθL → As′(μθB → Ls′(w))a = _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s ( _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s (w)) 6: Update ϕ [rgb]0.6094,0.3086,0.5898φ to minimize L(ϕ)=[(r(s,a)+γQϕ^πA(s′,a′)−QϕπA(s,a))2]L( [rgb]0.6094,0.3086,0.5898φ)=E[ (r(s,a)+γ Q _ A_ φ(s ,a )-Q _ A_ [rgb]0.6094,0.3086,0.5898φ(s,a) )^2] (2) 7: 8: ⊳ Train vector field vθL → Av_ [rgb]0.6094,0.3086,0.5898 _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A in decoder μθL → A _ [rgb]0.6094,0.3086,0.5898 _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A 9: x0∼qLx^0 q_ L 10: x1←ax^1← a 11: t∼([0,1])t ([0,1]) 12: xt←(1−t)x0+tx1x^t←(1-t)x^0+tx^1 13: Update θL → A [rgb]0.6094,0.3086,0.5898 _ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A to minimize L → A(θL → A)=[∥vθL → A(xt,t,s)−(x1−x0)∥2]L_ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( [rgb]0.6094,0.3086,0.5898 _ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A)=E [ v_ [rgb]0.6094,0.3086,0.5898 _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(x^t,t;s)-(x^1-x^0) ^2 ] (8) 14: 15: ⊳ Train vector field vB → Lbase [rgb]0.6094,0.3086,0.5898v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base to match qLunifq_ L^unif 16: x0∼qBx^0 q_ B 17: x1∼qLunifx^1 q_ L^unif 18: t∼([0,1])t ([0,1]) 19: xt←(1−t)x0+tx1x^t←(1-t)x^0+tx^1 20: Update vB → Lbase [rgb]0.6094,0.3086,0.5898v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base to minimize [∥vB → Lbase(xt,t)−(x1−x0)∥2]E [ [rgb]0.6094,0.3086,0.5898v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(x^t,t)-(x^1-x^0) ^2 ] 21: 22: ⊳ Train Latent-Space Policy using Adjoint Matching 23: Set Δt=1/N t=1/N and ti=iΔt_i=i t for i=0,…,Ni=0,…,N 24: w0∼(0,σ02I)w_0 (0, _0^2I) 25: Forward simulate using (49) to obtain wtii=1N\w_t_i\_i=1^N, held fixed during policy regression. 26: g~1←−1α∇w1Qϕ^πL(s,w1) g_1←- 1α _w_1Q_ φ _ L(s,w_1) 27: Compute g~ti g_t_i for i=N−1,…,0i=N-1,…,0 using (50) 28: Compute detached regression targets v^ti⋆ v _t_i for i=0,…,N−1i=0,…,N-1 using (51) 29: Update vθB → Lv_ [rgb]0.6094,0.3086,0.5898 _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L to minimize LAM(θB → L)=[12N∑i=0N−1∥2σ¯ti(vθB → L(wti,ti,s)−v^ti⋆)∥2]L_AM( [rgb]0.6094,0.3086,0.5898 _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L)=E [ 12N _i=0^N-1 2 σ_t_i (v_ [rgb]0.6094,0.3086,0.5898 _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#9B4F96 @begingroup@fill=#9B4F96 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_t_i,t_i;s)- v _t_i ) ^2 ] (20) 30: return policy πA _ A Flow-matching Decoder Learning. We parameterize the velocity field vθL → Av_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A of the decoder μθL → A _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A with a multilayer perceptron (MLP), and learn it via the flow-matching loss (8). The prior latent-space distribution is chosen to be a bounded uniform distribution over a d-ball with radius l, i.e., qL=(ℬld)q_ L=U(B_l^d). After training, we sample z∼qLz q_ L and use Euler integration to get the action samples: z0←z,z(k+1)Δt=zkΔt+vθL → A(zkΔt,kΔt,s)Δt,z_0← z, z_(k+1) t=z_k t+v_ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z_k t,k t;s) t, (47) where N is the total number of integration steps, Δt=1N t= 1N, k∈0,…,N−1k∈\0,…,N-1\, and the final generated action is a←z1a← z_1. Flow matching for qLunifq_ L^unif. Similarly, the velocity field vB → Lbasev_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base is parameterized by an MLP and learned via the flow-matching loss (8), with prior distribution qB=(0,σ02I)q_ B=N(0, _0^2I) and the desired pushforward qLunif=(ℬld)q_ L^unif=U(B_l^d). Note that vB → Lbasev_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base is state-independent. Critic Learning. The critic is learned via the standard TD learning objective (2) with an ensemble of K critics. In some environments, we found that using a pessimistic target-value backup [66] can improve performance. In that case, the loss function becomes L(ϕj)=(s,a,s′)∼,a′∼πA(⋅|s′)[(Qϕj(s,a)−r−γ(Qϕ^mean(s′,a′)−ρQϕ^std(s′,a′)))2],L( _j)=E_(s,a,s ) ,a _ A(·|s ) [ (Q_ _j(s,a)-r-γ (Q_ φ^mean(s ,a )-ρ Q_ φ^std(s ,a ) ) )^2 ], (48) where ϕj _j denotes the parameters of the jjth critic, Qϕ^mean(s′,a′)≔1K∑kQϕ^k(s′,a′)Q_ φ^mean(s ,a ) 1K _kQ_ φ_k(s ,a ), Qϕ^std(s′,a′)=1K∑k(Qϕ^k(s′,a′)−Qϕ^mean(s′,a′))2Q_ φ^std(s ,a )= 1K _k (Q_ φ_k(s ,a )-Q_ φ^mean(s ,a ) )^2, and ρ is the pessimistic backup weight. Entropy-regularized Policy Optimization via Adjoint Matching. We adapt the adjoint-matching implementation [16], retaining its endpoint regularization and final noiseless reference-flow step. We discretize the forward SDE and backward adjoint ODE on ti=iΔt_i=i t, with Δt=1/N t=1/N and N=10N=10. To avoid singular endpoint coefficients, we replace the denominator tit_i by ti+Δt_i+ t and use the regularized noise schedule σ¯ti=σ02(1−ti+Δt)/(ti+Δt) σ_t_i= _0 2(1-t_i+ t)/(t_i+ t). Starting from w0∼(0,σ02I)w_0 (0, _0^2I), the first N−1N-1 steps use Euler integration. The final step uses the reference flow and then projects its output onto the latent ball, ensuring w1∈ℬldw_1 _l^d: wti+1 w_t_i+1 ←wti+(2vθB → L(wti,ti;s)−wtiti+Δt)Δt+Δtσ¯tini,i=0,…,N−2, ← w_t_i+ (2v_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w_t_i,t_i;s)- w_t_it_i+ t ) t+ t\, σ_t_in_i, i=0,…,N-2, (49) w1 w_1 ←Πℬld(wtN−1+ΔtvB → Lbase(wtN−1,tN−1)), ← _B_l^d\! (w_t_N-1+ t\,v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w_t_N-1,t_N-1) ), where ni∼(0,I)n_i (0,I) and Πℬld _B_l^d is Euclidean projection. We initialize the lean adjoint with g~1=−∇w1Qϕ^πL(s,w1)/α g_1=- _w_1Q_ φ _ L(s,w_1)/α. Writing bi(w)=2vB → Lbase(w,ti+1)−w/ti+1b_i(w)=2v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w,t_i+1)-w/t_i+1, the backward update is g~ti←g~ti+1+ΔtVJP(∇wbi(wti),g~ti+1),i=N−1,…,0, g_t_i← g_t_i+1+ t\,VJP ( _wb_i(w_t_i), g_t_i+1 ), i=N-1,…,0, (50) where VJP(J,g)=J⊤gVJP(J,g)=J g. Regression uses only t0,…,tN−1t_0,…,t_N-1, excluding tN=1t_N=1, with targets v^ti⋆=sg(vB → Lbase(wti,ti)−σ¯ti22g~ti),i=0,…,N−1, v _t_i=sg\! (v_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^base(w_t_i,t_i)- σ_t_i^22 g_t_i ), i=0,…,N-1, (51) where sgsg denotes stop-gradient. The sampled states and regression targets are held fixed when differentiating the loss in Algorithm 1, so gradients pass only through vθB → Lv_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L. To stabilize training, we clip terminal latent critic gradients before initializing g~1 g_1. For each minibatch, we compute the Euclidean norm of each sample’s gradient ∇w1Qϕ^πL(s,w1) _w_1Q_ φ _ L(s,w_1) and rescale gradients whose norms exceed the minibatch’s 90th percentile so that their norms equal that threshold, while preserving their directions. We then divide all gradients by their mean norm, floored at 10−610^-6, before multiplying by −1/α-1/α. This operation leaves environment rewards unchanged and is separate from clipping network-parameter gradients. For critic targets and evaluation, we numerically integrate the latent policy ODE and project its terminal output before decoding. Denoting the numerical ODE map by μ^θB → Ls μ_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s, the deployed latent policy is (Πℬld∘μ^θB → Ls)♯qB( _B_l^d μ_ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L^s)_ q_ B. Projection enforces latent support, but if it places positive probability on the boundary, the resulting law has a singular component and no Lebesgue density. Thus, the entropy identity (12) and the optimality characterization in Lemma 3.2 do not directly apply to this projected law. The projected implementation is a practical approximation to the density-based formulation. Appendix E Details of the Reach-avoid Example in Figure 1 The agent starts in one of two regions at the bottom and must reach the goal at the top. Actions are two-dimensional velocities. If a proposed next state lies inside an obstacle, we set s′=s =s. The “No Entropy Regularization” baseline is constructed by replacing the adjoint matching component in LASER with plain gradient ascent, and the generated latent samples z are squashed inside the ball ℬldB_l^d to satisfy the support constraint. The “No Support Constraint” baseline is constructed with a QAM [23] model with a large inverse temperature τ that drives the action distribution away from the behavior policy distribution. Appendix F Experiments F.1 Environments We conduct experiments on the widely used OGBench offline RL benchmark [18]. Although OGBench is designed for offline goal-conditioned RL, it provides the -singletask flag for standard offline RL. We use 4 environments in the main comparison, including one locomotion environment (antmaze) and three manipulation environments (cube-single, cube-double, and scene), each with 5 tasks. For each task, we consider 2 datasets, yielding a total of 40 tasks. We consider the following tasks with clean datasets, i.e., the demonstrations are randomly generated by an expert policy: • antmaze-large-navigate-singletask-task1, 2, 3, 4, 5-v0 • cube-single-play-singletask-task1, 2, 3, 4, 5-v0 • cube-double-play-singletask-task1, 2, 3, 4, 5-v0 • scene-play-singletask-task1, 2, 3, 4, 5-v0 In addition, we consider the following tasks with noisy datasets, i.e., the demonstrations are generated by a highly noisy policy: • antmaze-large-explore-singletask-task1, 2, 3, 4, 5-v0 • cube-single-noisy-singletask-task1, 2, 3, 4, 5-v0 • cube-double-noisy-singletask-task1, 2, 3, 4, 5-v0 • scene-noisy-singletask-task1, 2, 3, 4, 5-v0 More details of the environments and datasets can be found in [18]. F.2 Baselines In this section, we briefly introduce the details of each baseline. FQL. FQL [21] is an algorithm that uses the statistical distance regularization objective (4), using a squared action-matching penalty that upper-bounds the squared 2-Wasserstein distance to its learned BC policy. It begins by training the decoder μθL → As _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s via the flow-matching loss (8), assuming a standard normal latent distribution qL=(0,I)q_ L=N(0,I). Subsequently, it fine-tunes this decoder by learning a one-step distilled policy μθL → A′(z,s):ℒ×→ _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A( [rgb]0.5352,0.5352,0.5352z, [rgb]0.5352,0.5352,0.5352s):L×S . This policy is optimized to maximize expected return while remaining anchored to the behavior cloning (BC) decoder μθL → As _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s. Specifically, the one-step policy minimizes the following loss: L(θL → A′) L(θ _ L 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A) =s∼D,z∼qL[−QϕμθL → A′(s,μθL → A′(z,s))] =E_s D,z q_ L [-Q _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A_φ(s, _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z,s)) ] (52) +αs∼D,z∼qL[‖μθL → A′(z,s)−μθL → As(z)‖2], + _s D,z q_ L [\| _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(z,s)- _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s(z)\|^2 ], where the first term optimizes task performance, and the second term is an upper-bound surrogate for their squared 2-Wasserstein distance [21]. Although the one-step policy is less expressive than the original flow policy [22], training this policy to maximize the Q-value avoids the BPTT process. However, the Wasserstein distance regularization is not enough to avoid OOD actions [7]. QAM. QAM [23] is an algorithm that uses the statistical distance regularization objective (4), instantiating D(πθ∥πβ)D( _θ\;\|\; _β) as the KL divergence. It begins by training the decoder μθL → As _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s via the flow-matching loss (8), assuming a standard normal latent distribution qL=(0,I)q_ L=N(0,I). Subsequently, it fine-tunes the decoder via adjoint matching to align with the target tilted distribution πβ(⋅|s)exp(τQμθL → A(s,⋅)) _β(·|s) (τ Q _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(s,·)). This procedure is mathematically equivalent to optimizing the following KL-divergence-regularized objective: L(θ)=−s∼[a∼πθ(⋅|s)[QμθL → A(s,a)]−1τDKL(πθ(⋅|s)∥πβ(⋅|s))],L(θ)=-E_s [E_a _θ(·|s) [Q _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A(s,a) ]- 1τD_KL( _θ(·|s)\;\|\; _β(·|s)) ], (53) where πθ(⋅|s)=(μθL → A′s)♯qL _θ(·|s)=( _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s)_ q_ L, and μθL → A′s _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s is the fine-tuned flow decoder. QAM avoids the BPTT process by leveraging the adjoint matching technique, similar to LASER, but the KL divergence regularization can limit the policy improvement [7]. QAM-E. QAM-E [23] extends QAM with an edit policy. After fine-tuning the decoder μθL → A′s _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s, it learns an edit policy πω(a|s,a):×→Δ() _ω( [rgb]0.5352,0.5352,0.5352a| [rgb]0.5352,0.5352,0.5352s, [rgb]0.5352,0.5352,0.5352a):S×A→ (A) that modifies the action generated by πθ(⋅|s)=(μθL → A′s)♯qL _θ(·|s)=( _θ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s)_ q_ L by at most σa _a to further improve performance. In practice, the edit policy πω _ω is parameterized as a squashed Gaussian policy that only outputs values in [−σa,σa][- _a, _a]. Empirically, QAM-E outperforms QAM [23], but it is sensitive to σa _a, and the edit policy distribution may generate OOD actions. IFQL. Presented in [21], IFQL is the flow version of the implicit diffusion Q-learning (IDQL) algorithm [32]. Similarly to the aforementioned algorithms, it first learns the decoder μθL → As _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s via the flow-matching loss (8), assuming qL=(0,I)q_ L=N(0,I). However, instead of learning the policy Q-function with an actor-critic structure with loss (2), it learns an approximately optimal critic via implicit Q-learning (IQL) [25] using the following losses: L(ϕ) L(φ) =(s,a,s′)∼[(Qϕ(s,a)−r−γVξ(s′))2], =E_(s,a,s ) [(Q_φ(s,a)-r-γ V_ξ(s ))^2 ], (54) L(ξ) L(ξ) =(s,a)∼[L2κ(Qϕ^(s,a)−Vξ(s))], =E_(s,a) [L_2^κ(Q_ φ(s,a)-V_ξ(s)) ], (55) where L2κ(u)=|κ−(u<0)|u2L_2^κ(u)=|κ- 1(u<0)|u^2 is the expectile regression loss. We use κ=0.9κ=0.9 for all tasks. The bootstrap term γVξ(s′)γ V_ξ(s ) is masked out for terminal transitions. To extract a policy from the learned Q-function and the decoder, IFQL uses best-of-N sampling with N=32N=32 candidates: i⋆=argmaxi∈1,…,NQ(s,ai),a⋆=ai⋆,a1,…,aN∼i.i.d.(μθL → As)♯qL. gatheredi = _i∈\1,…,N\Q(s,a_i), a =a_i ,\\ a_1,…,a_N i.i.d. ( _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s)_ q_ L. gathered (56) IFQL satisfies the support constraint by construction, but the candidate search is inefficient in high-dimensional spaces with a complex Q-function. DSRL. DSRL [8] is a recently proposed latent-space RL algorithm, which has several key differences compared with LASER. First, the latent distribution of the decoder is a standard Gaussian in DSRL, i.e., qL=(0,I)q_ L=N(0,I). Because the support of qLq_ L is unbounded, steering qLq_ L might generate a distribution that has high density far from the origin, where the samples are not well represented during the training of the decoder. To avoid this problem, DSRL parameterizes the latent policy πL _ L with a squashed Gaussian distribution that only generates samples within the range of [−σz,σz][- _z, _z], where σz _z is a hyperparameter that requires fine-tuning for each task. DSRL follows an SAC-style loss [12] and explicitly regularizes the entropy, which is straightforward for a Gaussian latent policy. However, because of the Gaussian parameterization, DSRL has a suboptimal entropy-performance tradeoff as shown in Figure 3. ReFORM. ReFORM [7] is another recently proposed latent-space RL algorithm, which also uses a flow latent policy. ReFORM learns its BC decoder μθL → As _ _ L 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope A^s in the same way as LASER. However, unlike LASER, ReFORM does not train the latent policy via adjoint matching, but via plain gradient descent. In addition, ReFORM does not have the explicit entropy regularization term, resulting in the following loss: LB → L(θB → L)=−s∼,w∼qB[QπL(s,μθB → L(w,s))].L_ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L( _ B 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to6.14pt to5.03pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 6.96 L 8.5 6.96 L 8.5 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to3.76pt to3.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 4.84 L 5.2 4.84 L 5.2 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L)=-E_s ,w q_ B [Q _ L(s, _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L(w;s)) ]. (57) Directly using gradient descent on this loss cannot guarantee satisfaction of the latent-space support constraint (6), as the flow μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L can drive the samples outside the closed ball ℬldB_l^d. To address this issue, ReFORM modifies the Euler integration step to (41) when integrating the flow μθB → L _ _ B 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to5.29pt to3.61pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 5 L 7.32 5 L 7.32 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to2.68pt to2.5pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 3.46 L 3.71 3.46 L 3.71 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope 2.0mu to1.92pt to1.79pt 0.0pt 0.0pt to0.0pt @begingroup@_scopebegin=1 @begingroup@stroke=#000000 @begingroup@fill=#000000 @setlinewidth @begingroup@stroke-width=0.4pt @inpgf@ignorespaces to0.0pt @discardpath @discardpath@clippedM 0 0 L 0 2.47 L 2.65 2.47 L 2.65 0 Z @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @begingroup@_scopebegin=1 @transformcm1.00.00.01.00.0pt0.0pt @begingroup@transform=matrix(1.0 0.0 0.0 1.0 0 0) @hbox58 @closescope @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @inpgf@ignorespaces @discardpath @closescope @closescope L, and proves that this modified step guarantees satisfaction of the latent-space support constraint. However, as discussed in Section C.1, this modified step can cause “static flow” when a proposal leaves the ball and the velocity aligns with the outward normal. In our toy example, this acts as implicit entropy regularization but yields a suboptimal action distribution. F.3 Hyperparameters We organize the hyperparameters into three distinct categories. Fixed common hyperparameters (Table 1) are shared across all algorithms and remain constant across all tasks. We do not fine-tune these, but instead adopt the standard values established in prior work [21, 23, 7]. Environment-specific common hyperparameters (Table 2) are also shared across algorithms but vary by environment and dataset to optimize baseline performance. Following [23], we perform a minimal sweep over two candidate values for these settings: 1 or 5 for the action chunking length, and 0 or 0.5 for the pessimistic backup weight. Environment-specific unique hyperparameters (Table 3) are specific to individual baselines (namely FQL, DSRL, QAM, and QAM-E) and require per-environment tuning. For these, we rely on the optimal values reported in their respective papers or tune them empirically. One notable exception is the edit scale σa _a for QAM-E. While the original authors sometimes set σa=0 _a=0 [23], making QAM-E identical to QAM, we enforce a minimum value of σa=0.01 _a=0.01 to ensure the algorithms remain distinct. Finally, while LASER introduces two unique hyperparameters, we keep them strictly fixed across all tasks. Specifically, we set 1/α=101/α=10 and choose σ0 _0 such that Prw∼(0,σ02I)(‖w‖≤l)=0.999Pr_w (0, _0^2I)(\|w\|≤ l)=0.999. For LASER and ReFORM, the latent ball radius is l=dl= d, where d is the action or action chunk dimension [7]. Table 1: Fixed common hyperparameters. These hyperparameters are shared by LASER and all the baselines and are fixed in all environments. Hyperparameter Value Learning rate 0.0003 Optimizer Adam [67] Maximum gradient norm 1 (QAM, QAM-E), 10 (others) Target network smoothing coefficient 0.0050.005 Discount factor γ 0.9950.995 MLP dimensions [512,512,512,512][512,512,512,512] Nonlinearity GELU [68] Flow integration steps 1010 Flow time sampling distribution [0,1]U[0,1] Minibatch size 256256 Number of critics in the ensemble 8 Q-function aggregation method mean Actor layer normalization [69] False Critic layer normalization [69] True Table 2: Environment-specific common hyperparameters. These hyperparameters are shared by LASER and all baselines, but vary by environment and dataset. Environment Dataset Action chunk length Pessimistic backup weight ρ antmaze-large-navigate clean 1 0.5 antmaze-large-explore noisy 1 0.5 cube-single-play clean 5 0.5 cube-single-noisy noisy 1 0 cube-double-play clean 5 0.5 cube-double-noisy noisy 1 0 scene-play clean 5 0.5 scene-noisy noisy 5 0.5 puzzle-4x4-play clean 5 0.5 Table 3: Environment- and dataset-specific unique hyperparameters. These hyperparameters are specific to individual algorithms and are tuned by environment and dataset. The method-specific hyperparameters of LASER, ReFORM, and IFQL remain fixed across environments, while their common settings follow Table 2. Environment Dataset FQL (α) DSRL (σz _z) QAM (τ) QAM-E (τ,σaτ, _a) antmaze-large-navigate clean 3.0 1.25 10.0 (1.0, 0.1) antmaze-large-explore noisy 0.3 1.25 10.0 (1.0, 0.1) cube-single-play clean 300 0.5 1.0 (1.0, 0.01) cube-single-noisy noisy 30 0.5 1.0 (3.0, 0.5) cube-double-play clean 300 1.5 1.0 (1.0, 0.01) cube-double-noisy noisy 30 1.5 1.0 (1.0, 0.01) scene-play clean 300 0.75 1.0 (1.0, 0.01) scene-noisy noisy 30 0.75 1.0 (1.0, 0.01) puzzle-4x4-play clean 1.0 1.0 30.0 (0.1, 0.9) F.4 Computational Resources Most of our experiments are run on a 13th Gen Intel(R) Core(TM) i7-13700KF CPU with 64 GB of RAM and an NVIDIA GeForce RTX 4090 GPU. Table 4 reports training throughput. For a fixed iteration budget, LASER requires about 12%12\% more training time than ReFORM and 33%33\% more than QAM, while its throughput is comparable to QAM-E (123123 vs 125it/s125\,it/s). The additional work includes computing lean adjoints and learning a latent policy alongside the decoder. Note that the lean-adjoint solve is needed only during offline training. At deployment, actions are generated by integrating the learned latent flow and then the decoder, without adjoints or critic gradients. Table 4: Training throughput comparison. We report the number of iterations per second (it/s) for LASER and all the baselines. Algorithm FQL QAM QAM-E IFQL DSRL ReFORM LASER it/s 220 163 125 210 165 138 123 F.5 Additional Results Ablation study of entropy regularization. To test the efficacy of entropy regularization beyond the simple environment introduced in Section 3.1, we extend the entropy regularization comparison to cube-single and cube-double environments with both clean and noisy datasets. NoER uses the same latent support constraint, network architectures, critic design, and other training settings as LASER, but directly maximizes Q through the latent flow using reparametrized gradients without entropy regularization. ReFORM instead uses a reflected flow with implicit entropy regularization (Section C.1), while LASER optimizes an explicitly entropy-regularized objective via adjoint matching. Table 5 reports the results for all five tasks in each environment with both clean and noisy datasets. We observe that both implicit (ReFORM) and explicit (LASER) entropy regularization help and significantly improve on NoER, and explicit entropy regularization (LASER) performs better than implicit entropy regularization (ReFORM). Table 5: Entropy regularization ablation on 20 cube-single and cube-double tasks. Success rates (%) are reported as mean ± standard deviation across 3 seeds, with 50 evaluation runs per seed. Bold denotes results at or above 95%95\% of the best mean in each setting. Task Dataset NoER ReFORM LASER cube-single-play-singletask-task1-v0 clean 6969±27± 27 99±2± 2 100±0± 0 cube-single-play-singletask-task2-v0 clean 5959±36± 36 96±3± 3 100±0± 0 cube-single-play-singletask-task3-v0 clean 8787±5± 5 98±3± 3 100±0± 0 cube-single-play-singletask-task4-v0 clean 5151±18± 18 8787±2± 2 99±1± 1 cube-single-play-singletask-task5-v0 clean 7474±14± 14 8686±13± 13 93±5± 5 cube-single-noisy-singletask-task1-v0 noisy 7676±6± 6 100±0± 0 100±0± 0 cube-single-noisy-singletask-task2-v0 noisy 8686±3± 3 100±0± 0 100±0± 0 cube-single-noisy-singletask-task3-v0 noisy 9191±7± 7 100±0± 0 100±0± 0 cube-single-noisy-singletask-task4-v0 noisy 4545±15± 15 100±0± 0 100±0± 0 cube-single-noisy-singletask-task5-v0 noisy 5959±7± 7 100±0± 0 100±0± 0 cube-double-play-singletask-task1-v0 clean 7373±29± 29 9191±7± 7 97±1± 1 cube-double-play-singletask-task2-v0 clean 5555±7± 7 8383±7± 7 89±1± 1 cube-double-play-singletask-task3-v0 clean 6363±12± 12 7373±7± 7 81±7± 7 cube-double-play-singletask-task4-v0 clean 2929±10± 10 2222±4± 4 49±12± 12 cube-double-play-singletask-task5-v0 clean 3636±21± 21 5454±13± 13 70±6± 6 cube-double-noisy-singletask-task1-v0 noisy 6161±15± 15 8080±3± 3 92±2± 2 cube-double-noisy-singletask-task2-v0 noisy 5151±12± 12 2929±14± 14 81±10± 10 cube-double-noisy-singletask-task3-v0 noisy 2727±5± 5 2929±3± 3 61±12± 12 cube-double-noisy-singletask-task4-v0 noisy 5050±3± 3 99±2± 2 65±5± 5 cube-double-noisy-singletask-task5-v0 noisy 2727±6± 6 1111±6± 6 38±13± 13 Inverse temperature and target policy smoothing. We compare Q-only latent optimization (NoER), target policy smoothing (TPS), and LASER with different inverse temperatures on cube-double task 2 using both clean and noisy datasets. TPS [20] augments NoER by adding independent zero-mean Gaussian noise with standard deviation 0.20.2, clipped to [−0.5,0.5][-0.5,0.5], to the target actions in the critic update. The perturbed actions are then clipped to the action bounds. As shown in Table 6, our default 1/α=101/α=10 outperforms both NoER and TPS on both datasets. Strong regularization (1/α=31/α=3) reduces performance on both datasets, whereas weak regularization (1/α=301/α=30) performs poorly on the clean dataset but remains competitive on the noisy dataset. One possible explanation is that noisy demonstrations provide broader action coverage, reducing critic artifacts. Table 6: Comparison of NoER, TPS, and LASER with different inverse temperatures on cube-double task 2. Success rates (%) are reported as mean ± standard deviation across 3 seeds, with 50 evaluation runs per seed. Bold denotes results at or above 95%95\% of the best mean for each dataset. Dataset NoER TPS LASER (1/α1/α) 1/α=31/α=3 1/α=51/α=5 1/α=101/α=10 1/α=201/α=20 1/α=301/α=30 clean 5555±7± 7 4949±24± 24 5050±14± 14 7878±19± 19 89±1± 1 7979±12± 12 3030±42± 42 noisy 5151±12± 12 6161±17± 17 3131±19± 19 7272±15± 15 81±10± 10 83±18± 18 83±19± 19 Uniform-mixture ablation in action space. LASER combines entropy optimization with bounded latent support. To examine a simple alternative in action space, we evaluate QAM+Mix, which broadens QAM’s reference toward a uniform distribution over the action space. With probability η, each behavior-cloning target is replaced by an action sampled uniformly within A. Adjoint matching then uses the reference flow learned from these modified targets. In the ideal KL-regularized formulation, η=0η=0 recovers QAM, while η=1η=1 yields action-space maximum-entropy optimization. Thus, η controls how strongly the reference distribution is broadened toward uniform. On cube-double-play-singletask-task2-v0 with the clean dataset, both tested mixing probabilities (η=0.03η=0.03 and η=0.1η=0.1) reduce QAM’s mean success rate (Table 7). The results show that this naïve approach does not perform well. Table 7: Uniform-mixture ablation in action space on cube-double task 2 with the clean dataset. Success rates (%) are reported as mean ± standard deviation across 3 seeds, with 50 evaluation runs per seed. Bold denotes results at or above 95%95\% of the best mean. QAM QAM+Mix (η=0.03η=0.03) QAM+Mix (η=0.1η=0.1) LASER 8383±11± 11 3838±10± 10 99±6± 6 89±1± 1 Evaluation on harder tasks. We compare LASER with all six baselines on the five puzzle-4x4 tasks using the clean (100M) dataset. Every run uses two million updates, and we evaluate the final checkpoint for each method and seed. The action chunk length is 5, giving a 25-dimensional action chunk, and the pessimistic backup weight is ρ=0.5ρ=0.5. Baseline-specific settings are listed in Table 3. As shown in Table 8, LASER still achieves high performance on these more challenging tasks. Table 8: Results on the five puzzle-4x4 tasks with the clean dataset (100M). Success rates (%) are reported as mean ± standard deviation across 3 seeds, with 50 evaluation runs per seed. Bold denotes results at or above 95%95\% of the best mean for each task. Task FQL QAM QAM-E IFQL DSRL ReFORM LASER 1 55±4± 4 22±2± 2 6363±14± 14 5050±13± 13 6565±24± 24 8686±6± 6 97±3± 3 2 1313±9± 9 00±0± 0 7474±24± 24 22±0± 0 1717±7± 7 66±7± 7 87±2± 2 3 55±7± 7 33±2± 2 1818±12± 12 87±1± 1 6363±41± 41 8181±7± 7 84±7± 7 4 11±1± 1 00±0± 0 3333±15± 15 3232±4± 4 3131±21± 21 3333±25± 25 60±3± 3 5 00±0± 0 11±1± 1 5353±26± 26 22±3± 3 1414±14± 14 66±3± 3 69±18± 18 Boundary-hit diagnostic for ReFORM. We evaluate the final ReFORM checkpoints on all five cube-double tasks with both clean and noisy datasets. At each policy call, we record the fraction of N=10N=10 latent Euler proposals satisfying ‖w^k+1‖>l\| w_k+1\|>l, before applying the reflection correction in (41). We average these rates over policy calls within each episode, then over episodes and seeds. Table 9 reports the results. The high rates (97.4%97.4\%–99.0%99.0\%) show that boundary correction is frequently active, consistent with the proposed static-flow mechanism in Section C.1. Table 9: Boundary-hit rates (%) for ReFORM on cube-double. Results are averaged across 3 seeds, with 50 evaluation episodes per seed and 10 latent flow steps per policy call. Dataset Task 1 Task 2 Task 3 Task 4 Task 5 clean 98.298.2 99.099.0 98.998.9 97.697.6 98.098.0 noisy 97.497.4 97.797.7 97.997.9 97.597.5 97.997.9 Figure 8: Additional ablation studies. LASER demonstrates robustness to the choice of σ0 _0. Furthermore, applying terminal-gradient clipping improves training stability. Reducing the parameter-gradient cap from 10 to 1 yields a comparable stabilizing effect. Lines show mean success rates, and shading spans the minimum and maximum across runs. Alternative latent prior. For the decoder-prior ablation in Figure 6, we use qmix=0.4(ℬld)+0.3(Πℬld)♯(0.5e1,0.12I)+0.3(Πℬld)♯(0.4ed,0.152I),q_ L^mix=0.4\,U(B_l^d)+0.3\,( _B_l^d)_ N(0.5e_1,0.1^2I)+0.3\,( _B_l^d)_ N(0.4e_d,0.15^2I), where eie_i is the iith coordinate vector and Gaussian samples outside the ball are projected onto its boundary. This experiment uses d=25d=25 and l=5l=5. Only the decoder’s training prior changes. The latent policy’s entropy reference remains qLunif=(ℬld)q_ L^unif=U(B_l^d). Ablation study of the standard deviation of the base distribution. We study the sensitivity of LASER w.r.t. the standard deviation σ0 _0 of qBq_ B. We vary σ0 _0 such that the ball ℬldB_l^d contains probability mass p under the Gaussian distribution (0,σ02I)N(0, _0^2I). We report the results in Figure 8. The results show that LASER is not sensitive to σ0 _0. Ablation study of the gradient clipping technique. A known drawback of adjoint matching is its tendency to produce large gradients, which can destabilize the training process [16, 23]. In Figure 8, we compare three variants on cube-double task 2 with clean data: terminal-gradient clipping disabled with a parameter-gradient cap of 10, disabled with a cap of 1, and enabled with a cap of 10 (our default). Both enabling terminal-gradient clipping and reducing the parameter-gradient cap improve stability over the first variant, with comparable performance. Detailed results. The full results for each algorithm on the 40 main evaluation settings are provided in Table 11. Success rates are bolded for algorithms achieving at least 95%95\% of the best performance per task. For every method and seed, we evaluate the final checkpoint. The main-suite training budgets are 5 million updates for antmaze-large-navigate, 8 million for antmaze-large-explore, 1 million for cube-single-noisy, and 2 million for the remaining settings. The training budgets are selected such that all methods converge. Each checkpoint is evaluated over 50 episodes, and we report the mean and standard deviation across 3 seeds. We observe that LASER yields the best overall performance, achieving near-perfect success rates on multiple tasks. Statistical comparison. We pair results across the 40 main task–dataset settings, averaging each method’s success rate over its 3 seeds before comparison. For each baseline j, a one-sided paired t-test evaluates H0,j:Δj≤0H_0,j: _j≤ 0, where Δj _j denotes the expected difference between LASER’s and baseline j’s setting-level mean success rates. To test for improvement over every baseline, we use an intersection–union test of H0:⋃jΔj≤0H_0: _j\ _j≤ 0\ against H1:Δj>0H_1: _j>0 for all j. The global p-value is maxjpj=1.35×10−3 _jp_j=1.35× 10^-3 (Table 10), rejecting H0H_0 at the 0.050.05 level under the paired-test assumptions. This inference treats setting-level differences as independent despite shared environments and datasets, so we interpret it as supporting evidence for aggregate performance on this suite. Table 10: Paired statistical comparisons on the 40 main evaluation settings. Mean gaps are LASER minus each baseline in percentage points. One-sided p-values are computed from unrounded success rates averaged over 3 seeds per setting, with 50 evaluation runs per seed. Baseline FQL QAM QAM-E IFQL DSRL ReFORM Mean gap 20.0220.02 18.7018.70 17.7517.75 47.5047.50 16.3316.33 11.8211.82 p-value ×10−69.08\!×\!10^-6 ×10−43.47\!×\!10^-4 ×10−46.44\!×\!10^-4 ×10−113.22\!×\!10^-11 ×10−31.35\!×\!10^-3 ×10−53.42\!×\!10^-5 Training curves. Figures 9 to 16 show training curves for all algorithms and tasks on both the clean and noisy datasets. Lines show means over 3 seeds, and shading spans the minimum and maximum across seeds. Each point uses 20 evaluation episodes per seed, whereas the reported table results use a separate evaluation of 50 episodes per seed. F.6 Code We provide code on our project website https://mit-realm.github.io/laser/. Table 11: Full results on 40 OGBench tasks. Success rates (%) are reported as mean ± standard deviation across 3 seeds, with 50 evaluation episodes per seed. Bold denotes results at or above 95%95\% of the best mean in each setting, following [18]. We omit the -singletask tags from the task names to save space. Task Dataset FQL QAM QAM-E IFQL DSRL ReFORM LASER antmaze-large-navigate-task1-v0 clean 6464±45± 45 96±3± 3 93±2± 2 7979±13± 13 9090±3± 3 9191±2± 2 95±1± 1 antmaze-large-navigate-task2-v0 clean 3131±43± 43 6060±42± 42 94±3± 3 6565±5± 5 8181±5± 5 7878±2± 2 89±2± 2 antmaze-large-navigate-task3-v0 clean 7171±12± 12 8585±3± 3 97±1± 1 8585±9± 9 94±3± 3 7474±6± 6 92±3± 3 antmaze-large-navigate-task4-v0 clean 3333±41± 41 93±1± 1 91±1± 1 3131±43± 43 7575±14± 14 6969±4± 4 8585±1± 1 antmaze-large-navigate-task5-v0 clean 8888±6± 6 95±1± 1 93±4± 4 8383±2± 2 91±4± 4 8383±1± 1 91±1± 1 antmaze-large-explore-task1-v0 noisy 97±1± 1 00±0± 0 00±0± 0 5050±36± 36 00±0± 0 9191±4± 4 97±2± 2 antmaze-large-explore-task2-v0 noisy 97±1± 1 00±0± 0 00±0± 0 33±2± 2 00±0± 0 8888±9± 9 8787±2± 2 antmaze-large-explore-task3-v0 noisy 100±0± 0 00±0± 0 00±0± 0 95±1± 1 00±0± 0 100±0± 0 98±3± 3 antmaze-large-explore-task4-v0 noisy 89±3± 3 00±0± 0 00±0± 0 3636±13± 13 00±0± 0 8282±9± 9 89±2± 2 antmaze-large-explore-task5-v0 noisy 94±6± 6 00±0± 0 00±0± 0 4747±34± 34 00±0± 0 93±2± 2 92±3± 3 cube-single-play-task1-v0 clean 95±4± 4 9494±3± 3 95±4± 4 7979±2± 2 9595±5± 5 99±2± 2 100±0± 0 cube-single-play-task2-v0 clean 97±1± 1 99±1± 1 99±2± 2 7070±5± 5 9090±7± 7 96±3± 3 100±0± 0 cube-single-play-task3-v0 clean 99±1± 1 100±0± 0 99±1± 1 8282±7± 7 97±4± 4 98±3± 3 100±0± 0 cube-single-play-task4-v0 clean 9090±4± 4 9191±2± 2 8585±4± 4 8282±3± 3 98±2± 2 8787±2± 2 99±1± 1 cube-single-play-task5-v0 clean 8181±2± 2 89±4± 4 8585±4± 4 7575±6± 6 8585±3± 3 8686±13± 13 93±5± 5 cube-single-noisy-task1-v0 noisy 100±0± 0 100±0± 0 100±0± 0 11±1± 1 9393±6± 6 100±0± 0 100±0± 0 cube-single-noisy-task2-v0 noisy 100±0± 0 100±0± 0 99±1± 1 77±6± 6 9292±9± 9 100±0± 0 100±0± 0 cube-single-noisy-task3-v0 noisy 100±0± 0 100±0± 0 100±0± 0 8585±9± 9 5353±25± 25 100±0± 0 100±0± 0 cube-single-noisy-task4-v0 noisy 100±0± 0 100±0± 0 100±0± 0 00±0± 0 9191±4± 4 100±0± 0 100±0± 0 cube-single-noisy-task5-v0 noisy 100±0± 0 100±0± 0 99±2± 2 00±0± 0 9191±5± 5 100±0± 0 100±0± 0 cube-double-play-task1-v0 clean 7878±2± 2 96±2± 2 97±1± 1 1919±2± 2 95±1± 1 9191±7± 7 97±1± 1 cube-double-play-task2-v0 clean 5252±3± 3 8383±11± 11 8383±1± 1 1515±4± 4 6363±4± 4 8383±7± 7 89±1± 1 cube-double-play-task3-v0 clean 4040±6± 6 6969±11± 11 7575±6± 6 99±7± 7 6363±17± 17 7373±7± 7 81±7± 7 cube-double-play-task4-v0 clean 77±3± 3 2121±1± 1 2020±4± 4 11±1± 1 1717±5± 5 2222±4± 4 49±12± 12 cube-double-play-task5-v0 clean 3737±10± 10 81±10± 10 7373±2± 2 88±3± 3 7373±1± 1 5454±13± 13 7070±6± 6 cube-double-noisy-task1-v0 noisy 5353±17± 17 8989±4± 4 97±2± 2 11±1± 1 8787±5± 5 8080±3± 3 92±2± 2 cube-double-noisy-task2-v0 noisy 55±2± 2 7474±3± 3 80±4± 4 00±0± 0 79±11± 11 2929±14± 14 81±10± 10 cube-double-noisy-task3-v0 noisy 55±2± 2 5555±5± 5 5353±6± 6 00±0± 0 60±5± 5 2929±3± 3 61±12± 12 cube-double-noisy-task4-v0 noisy 11±1± 1 4343±13± 13 5151±2± 2 11±1± 1 5858±18± 18 99±2± 2 65±5± 5 cube-double-noisy-task5-v0 noisy 22±3± 3 3535±4± 4 3030±4± 4 00±0± 0 40±11± 11 1111±6± 6 38±13± 13 scene-play-task1-v0 clean 100±0± 0 100±0± 0 100±0± 0 99±1± 1 100±0± 0 100±0± 0 100±0± 0 scene-play-task2-v0 clean 100±0± 0 100±0± 0 99±1± 1 33±2± 2 100±0± 0 100±0± 0 100±0± 0 scene-play-task3-v0 clean 9393±5± 5 9393±5± 5 9494±3± 3 8080±4± 4 99±1± 1 9393±8± 8 100±0± 0 scene-play-task4-v0 clean 6666±13± 13 1616±19± 19 77±4± 4 33±1± 1 100±0± 0 6060±42± 42 5858±29± 29 scene-play-task5-v0 clean 13±9± 9 00±0± 0 00±0± 0 00±0± 0 00±0± 0 00±0± 0 00±0± 0 scene-noisy-task1-v0 noisy 8686±12± 12 100±0± 0 100±0± 0 99±2± 2 100±0± 0 7171±17± 17 100±0± 0 scene-noisy-task2-v0 noisy 7070±8± 8 98±2± 2 98±2± 2 00±0± 0 96±3± 3 97±4± 4 100±0± 0 scene-noisy-task3-v0 noisy 00±0± 0 2525±6± 6 3939±16± 16 4444±4± 4 100±0± 0 4545±27± 27 8080±27± 27 scene-noisy-task4-v0 noisy 00±0± 0 88±11± 11 11±1± 1 11±1± 1 3737±39± 39 00±0± 0 67±31± 31 scene-noisy-task5-v0 noisy 0±0± 0 0±0± 0 0±0± 0 0±0± 0 0±0± 0 0±0± 0 0±0± 0 Figure 9: Training curves for the antmaze-large environment with the clean dataset. Figure 10: Training curves for the antmaze-large environment with the noisy dataset. Figure 11: Training curves for the cube-single environment with the clean dataset. Figure 12: Training curves for the cube-single environment with the noisy dataset. Figure 13: Training curves for the cube-double environment with the clean dataset. Figure 14: Training curves for the cube-double environment with the noisy dataset. Figure 15: Training curves for the scene environment with the clean dataset. Figure 16: Training curves for the scene environment with the noisy dataset.