Paper deep dive
From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
Shenghua Lu, Zixuan Yang, Rongxu Zhao, Bohao Qu, Jifeng Hu, Hechang Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval $\Delta t=1$, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.16984v1
- Canonical: https://arxiv.org/abs/2610.16984v1
Trouble viewing inline? Open PDF directly →
Full Text
59,244 characters extracted from source content.
Expand or collapse full text
From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning Shenghua Lu* Zixuan Yang* Rongxu Zhao Bohao Qu Jifeng Hu Hechang ChenJilin University Abstract Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval Δt=1 t=1, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from T to K≈T/HK≈ T/H and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines. A Preprint **footnotetext: These authors contributed equally to this work†footnotetext: Corresponding authors 1 Introduction Reinforcement learning (RL) has achieved remarkable progress across diverse domains, including robotics(Zhu et al., 2023; Hwangbo et al., 2019), games(Schrittwieser et al., 2020; Perolat et al., 2022), and fundamental science(Jumper et al., 2021; Ren et al., 2018). In many real-world settings such as robotics(Zhu et al., 2023; Ma et al., 2026), healthcare(Yu et al., 2021a), and autonomous driving(Li et al., 2023), however, online interaction is expensive, unsafe, or infeasible. Offline RL addresses this by learning solely from pre-collected datasets(Huang et al., 2024), but faces two persistent challenges: limited policy expressiveness as behavioral distributions grow complex and multimodal(Kang et al., 2023), and the underutilization of suboptimal trajectories that constitute the majority of practical datasets. Generative policies based on diffusion(Wang et al., 2022) and flow matching(Park et al., 2025) improve expressiveness, yet share two structural limitations. First, they generate behaviors through multi-step denoising or ODE integration, incurring high inference costs. Second, they model policies directly in raw action space, treating trajectories of different qualities as homogeneous samples(Fujimoto and Gu, 2021; Kostrikov et al., 2021) and offering no mechanism to lift suboptimal behaviors toward high-value regions. We argue that suboptimal trajectories are not pure noise but starting points for improvement. Drawing on the neural null-space mechanism in motor cortex (Stavisky et al., 2017), we design Gaussian-Induced Decision Sources as a semantic and value null-space origin from which structured flow learning sculpts task-relevant decision representations. Hierarchical behavior theory (Sutton et al., 1999a; Botvinick et al., 2009) motivates our chunk-level latent decision representation, and value correction principles (Sutton and Barto, 2018) inspire the hierarchical chunk-level advantage-guided flow mapping toward high-value regions. We propose MF2B (MeanFlow2Better, MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. A Chunk-Level Latent Decision Encoder compresses trajectory chunks into high-level latent representations, reducing sequence length from T to K≈T/HK≈ T/H. A shared conditional mean velocity field learns across two branches: one transporting Gaussian-Induced Decision Sources toward positive behavioral regions, another transporting suboptimal decisions toward high-value regions. Hierarchical chunk-level advantage estimation guides both branches toward high-return areas. Unlike multi-step diffusion or flow models, MF2B predicts the conditional mean velocity over the unit interval Δt=1 t=1 and completes refinement in a single Euler step. Evaluation on the D4RL Gym-MuJoCo benchmark(Fu et al., 2020) demonstrates an average score of 90.91, surpassing all one-step flow policy baselines. Our main contributions are: • We propose MF2B, which reformulates offline RL as generative decision chunk optimization from suboptimal to high-value in a high-level latent decision space, realized through a dual-branch conditional mean velocity field with hierarchical chunk-level advantage guidance. • We propose a one-step decision refinement mechanism based on conditional mean velocity fields, replacing multi-step ODE integration with a single Euler step over the unit interval Δt=1 t=1. • Evaluations on 9 D4RL Gym-MuJoCo tasks demonstrate an average score of 90.91, establishing state-of-the-art performance among one-step flow policies. 2 Related Work Offline Reinforcement Learning. Reinforcement learning is inherently an online paradigm(Sutton et al., 1999b), and its online learning characteristics present limitations in scenarios where online data collection and learning cannot occur simultaneously, such as autonomous driving. Offline reinforcement learning enables agents to learn from pre-collected fixed datasets without collecting new data during the learning process(Fujimoto et al., 2019; Kumar et al., 2020; Yu et al., 2021b; Kumar et al., 2019; Hu et al., 2022; Hu et al., 2023). Diffusion Policies in Offline RL. Offline RL aims to learn effective policies from fixed datasets without environment interaction (Levine et al., 2020), with early methods addressing distributional shift via conservative objectives such as CQL (Kumar et al., 2020), TD3+BC (Fujimoto and Gu, 2021), and IQL (Kostrikov et al., 2021). However, these approaches often rely on unimodal Gaussian policies, which struggle to model complex, multi-modal action distributions. To address this, recent approaches have adopted diffusion models (Ho et al., 2020; Song et al., 2020) for offline RL. Diffusion Q-learning (DQL) (Wang et al., 2022) replaces Gaussian policies with diffusion models, modeling complex multimodal action distributions through multi-step denoising. IDQL (Hansen-Estruch et al., 2023) and EDP (Kang et al., 2023) improve upon DQL but still share the critical limitation of high sampling costs from multi-step denoising. Flow Matching and Flow Policies. Flow matching is a generative modeling approach that learns time-varying velocity fields through regressing target vector fields, which define probability paths transporting samples from simple prior distributions to target data distributions, characterized by simple training and fast inference(Lipman et al., 2023) (see Section 3 for details). Recently, flow matching has attracted attention in reinforcement learning due to its strong expressiveness and faster inference. FQL (Park et al., 2025) uses flow matching policies to approximate behavior policies and distills flow policies for efficient policy learning and inference. FlowQ (Alles et al., 2025) learns a conditional velocity field for flow policies by modeling energy-guided probability paths as Gaussian trajectories. However, these methods typically rely on simple Gaussian priors and multi-step sampling from flow policies, limiting representation richness and bringing additional computational overhead. To address computational bottlenecks from multi-step generative policies, MeanFlowQL (Wang et al., 2026) reformulates MeanFlow (Geng et al., 2025a) as a residual mapping that directly transforms noise into actions. SSCP (Koirala and Fleming, 2026) uses enhanced flow matching objective functions to predict direct completion vectors from intermediate samples, achieving one-shot action generation. Unlike the above methods, this paper introduces MF2B, achieving paradigm shifts in three dimensions. First, at the representation space level, existing flow policies operate directly in raw action space(Park et al., 2025; Alles et al., 2025), requiring learning highly nonlinear mappings from Gaussian noise to complex action distributions. MF2B compresses trajectory chunks into high-level latent decision representations through an encoder, reformulating flow matching as short-range approximately linear flow mappings within decision space, thereby establishing foundations for one-step generation. Second, at the generation mechanism level, FQL(Park et al., 2025) and FlowQ(Alles et al., 2025) learn marginal velocity fields requiring multi-step ODE integration, while MF2B learns mean velocity fields(Geng et al., 2025a) directly predicting residual directions from starting points to endpoints, achieving one-step high-fidelity generation. Third, at the optimization objective level, existing methods indiscriminately fit dataset behavior frequencies, while MF2B introduces a dual-branch flow architecture: the Source-to-Positive Alignment Mean-Flow Branch establishes baseline mappings from Gaussian-Induced Decision Sources to dataset decisions, and the Quality-Uplift Mean-Flow Branch achieves value-guided flow mapping from suboptimal decisions to high-value decisions through hierarchical chunk-level advantage weighting, with both branches sharing the same conditional mean velocity field parameters. This design enables the model to extract flow mapping patterns leading from suboptimal trajectories to optimal policies, breaking through dataset quality ceilings. To our knowledge, this is the first offline reinforcement learning method combining mean flow matching with value-guided dual-branch flow mapping in high-level latent decision space to exploit suboptimal-to-optimal data capabilities, achieving unity of expressiveness and computational efficiency. 3 Preliminaries 3.1 Offline Reinforcement Learning Reinforcement learning (RL) is typically formalized as a Markov Decision Process (MDP), defined by ℳ=(,,P,R,γ)M=(S,A,P,R,γ). Here, S and A denote the state and action spaces, P(s′∣s,a)P(s s,a) denotes the transition probability of moving from s to s′s given action a, and R(s,a)R(s,a) denotes the reward function. The discount factor γ∈[0,1)γ∈[0,1) governs the trade-off between immediate and future rewards. In RL, the objective is to learn a policy πθ(a∣s) _θ(a s), parameterized by θ, that maximizes the expected discounted return π[∑h=0∞γhR(sh,ah)].E_π [ _h=0^∞γ^hR(s_h,a_h) ]. (1) To support policy learning, the action-value (Q) function is defined: Qπ(s,a)=π[∑h=0∞γhR(sh,ah)∣s0=s,a0=a],Q^π(s,a)=E_π [ _h=0^∞γ^hR(s_h,a_h) s_0=s,a_0=a ], (2) which measures the expected cumulative return starting from state s and action a under policy π. Offline RL is a setting of RL where the agent does not interact with the environment but instead learns from a fixed dataset of transitions =(sh,ah,sh+1,rh)D=\(s_h,a_h,s_h+1,r_h)\. The challenge lies in learning an optimal policy solely from this static dataset that often contains suboptimal behavior, without any further exploration. 3.2 Implicit Q-Learning Implicit Q-Learning (IQL)(Kostrikov et al., 2021) is an offline reinforcement learning method that learns upper-bound estimates of value functions through expectile regression, thereby avoiding out-of-distribution action evaluation. Unlike traditional Q-learning that directly fits Q-value means, IQL’s core innovation lies in using asymmetric expectile loss functions to train the state-value network V(s)V(s). IQL maintains two Q-networks Qω,1(s,a)Q_ω,1(s,a) and Qω,2(s,a)Q_ω,2(s,a), a state-value network Vν(s)V_ν(s), and corresponding target Q-networks. The value network is learned through expectile regression: ℒV=[|τexp−(δ<0)|δ2],L_V=E [ | _exp-I(δ<0) |δ^2 ], (3) where δ=min(Qω¯,1(s,a),Qω¯,2(s,a))−Vν(s)δ= (Q_ ω,1(s,a),Q_ ω,2(s,a))-V_ν(s), and τexp∈(0.5,1) _exp∈(0.5,1) is the expectile parameter (typically 0.7). This asymmetric loss makes Vν(s)V_ν(s) learn as a high expectile of the behavior value distribution rather than a simple average, thereby providing more conservative value estimates. Based on this value baseline, an advantage function can be defined: A(s,a)=min(Qω¯,1(s,a),Qω¯,2(s,a))−Vν(s).A(s,a)= (Q_ ω,1(s,a),Q_ ω,2(s,a))-V_ν(s). (4) This advantage function can identify relatively high-value state-action pairs in the dataset, providing value guidance signals for policy learning without explicit policy improvement steps. 3.3 Flow Matching Flow matching is a generative modeling approach that adopts simple and intuitive training objectives by regressing target vector fields, which define desired probability paths(Lipman et al., 2023). Flow matching learns a time-varying vector field by defining an ordinary differential equation (ODE), whose solution transports samples from a simple prior distribution to the target data distribution. Given a data distribution p(x)p(x), the goal is to learn a time-varying velocity field vθ(xt,t):ℝd×[0,1]→ℝdv_θ(x_t,t):R^d×[0,1] ^d parameterized by θ, where xt∈ℝdx_t ^d denotes a d-dimensional data sample and t∈[0,1]t∈[0,1] denotes time. This velocity field defines a continuous flow that transports samples from a simple prior distribution (e.g., unit Gaussian) at t=0t=0 to the target data distribution p(x)p(x) at t=1t=1. Flow Path Construction and Velocity Field. Flow paths can be constructed as xt=αtx0+βtϵx_t= _tx_0+ _tε (Park et al., 2025; Lipman et al., 2023), where αt _t and βt _t are predefined time-varying functions (common choice: αt=1−t _t=1-t, βt=t _t=t), x0∼p(x)x_0 p(x) is a data sample, and ϵ∼(0,I)ε (0,I) is noise. The corresponding conditional velocity is given by vt(xt∣x0)=dtxt=αt′x0+βt′ϵv_t(x_t x_0)= ddtx_t= _t x_0+ _t ε, where ′ denotes the time derivative with respect to t. Standard flow matching learns the marginal velocity field v(xt,t)≜pt(x0|xt)[vt(xt|x0)]v(x_t,t) _p_t(x_0|x_t)[v_t(x_t|x_0)] via the Conditional Flow Matching (CFM) loss (Lipman et al., 2023), which requires multi-step ODE integration during generation. In contrast, the mean velocity field(Geng et al., 2025b) directly models the average velocity between two timesteps, defined as: u(xt,r,t)≜1t−r∫rtv(xτ,τ)τ,u(x_t,r,t) 1t-r _r^tv(x_τ,τ)dτ, (5) where u denotes the average velocity from time r to time t, and v denotes the marginal velocity. As r→tr→ t, u converges to v. The advantage of the mean velocity field is that it can directly predict residual directions from starting points to endpoints, achieving one-step high-fidelity generation without iterative ODE integration. By learning the mean velocity field uθ(xt,r,t)u_θ(x_t,r,t), samples can be directly generated through a one-step update x1=x0+uθ(x0,0,1)x_1=x_0+u_θ(x_0,0,1) over the unit time interval Δt=1 t=1. Training Objective for Mean Velocity Field. The mean velocity field uθu_θ is optimized through the mean flow identity. Defining the average velocity as u(xt,r,t)=1t−r∫rtv(xτ,τ)τu(x_t,r,t)= 1t-r _r^tv(x_τ,τ)dτ, by taking the time derivative and applying the chain rule, the mean flow identity can be obtained(Geng et al., 2025b): u(xt,r,t)=v(xt,t)−(t−r)(v(xt,t)⋅∂xtu+∂tu),u(x_t,r,t)=v(x_t,t)-(t-r) (v(x_t,t)· _x_tu+ _tu ), (6) where ∂xtu _x_tu and ∂tu _tu denote partial derivatives of u with respect to xtx_t and t respectively. This identity relates the marginal velocity v to the mean velocity u, enabling construction of supervision targets for mean velocity through the known conditional velocity vt(xt∣x1)=x1−ϵv_t(x_t x_1)=x_1-ε. Training minimizes: ℒMF(θ)=0≤r≤t≤1,ϵ∼(0,I),x1∼p(x)[‖uθ(xt,r,t)−u∗(xt,r,t∣x1)‖22],L_MF(θ)=E_ subarrayc0≤ r≤ t≤ 1,\\ ε (0,I),\\ x_1 p(x) subarray [ \|u_θ(x_t,r,t)-u^*(x_t,r,t x_1) \|_2^2 ], (7) where u∗(xt,r,t∣x1)u^*(x_t,r,t x_1) is computed from the conditional velocity through Equation (6). Sampling and Generation with Mean Velocity Field. Given the mean velocity field uθ(xt,r,t)u_θ(x_t,r,t), samples can be directly generated through a one-step update: x1=x0+uθ(x0,0,1),x_1=x_0+u_θ(x_0,0,1), (8) where x0=ϵ∼(0,I)x_0=ε (0,I) is prior noise. The mean velocity field directly predicts residual directions from starting points to endpoints, achieving one-step high-fidelity generation without iterative ODE integration, significantly reducing inference costs. 4 MeanFlow2Better Method MF2B reformulates offline reinforcement learning as a quality improvement problem in high-level latent decision space. Trajectory chunks are compressed into high-level latent decision representations through a Chunk-Level Latent Decision Encoder qϕq_φ, a shared Conditional Latent Decision Mean-Flow Field vθv_θ learns conditional mean velocities across two supervisory branches, and a Hierarchical Action Decoder pψp_ψ recovers continuous action sequences from refined decisions. A Hierarchical Chunk-Level Advantage Critic ηC_η provides value guidance during training only. Throughout this section, we fix t0=0t_0=0 and t1=1t_1=1 as the start and end of the unit flow interval. Algorithm 1 summarizes the complete pipeline. 4.1 Chunk-Level High-Level Latent Decision Representation Given an offline dataset =τii=1ND=\ _i\_i=1^N, each trajectory τi=(s0,a0,r0,…,sT−1,aT−1,rT−1) _i=(s_0,a_0,r_0,…,s_T-1,a_T-1,r_T-1) is segmented into consecutive chunks of length H: τk=(sk,0:H−1,ak,0:H−1), _k=(s_k,0:H-1,\,a_k,0:H-1), (9) yielding K≈T/HK≈ T/H chunks per trajectory. The encoder qϕq_φ maps each chunk to a distribution over high-level latent decisions: qϕ(τk)=[μk,logσk2]∈ℝ2dz,q_φ( _k)=[ _k, _k^2] ^2d_z, (10) and the full decision sequence is Z=[z1,…,zK]∈ℝK×2dzZ=[z_1,…,z_K] ^K× 2d_z. The mean-flow field operates on the concatenated [μ,logσ2][μ, σ^2] representation directly. The decoder samples via reparameterization, zk=μk+exp(12logσk2)⊙ϵkz_k= _k+ ( 12 _k^2) _k, and generates actions as a^k,0:H−1=pψ(sk,0:H−1,zk) a_k,0:H-1=p_ψ(s_k,0:H-1,z_k). This separation of high-level behavioral intention from low-level action execution allows the flow model to perform behavioral improvement in a compact, stable decision space rather than over raw action sequences. 4.2 High-Level Decision Source Construction 4.2.1 Gaussian-Induced Decision Source A Gaussian pseudo-trajectory τG−∼(0,I)τ^-_G (0,I) is drawn with the same temporal shape as real trajectories. Inspired by the neural null-space mechanism (Stavisky et al., 2017), we explicitly set its reward, RTG, chunk-level RTG, and trajectory return to zero, placing Gaussian-Induced Decision Sources at the zero point of the value dimension. After encoding: Zneg=qϕ(τG−).Z_neg=q_φ(τ^-_G). (11) The Source-to-Positive Alignment Mean-Flow Branch transports ZnegZ_neg toward positive behavioral decision regions, establishing task-relevant structure from a value-neutral baseline unconstrained by any specific return interval in the dataset. 4.2.2 Suboptimal-to-High-Value Trajectory Pairing Trajectories are partitioned into a high-value pool ℐtopI_top and a suboptimal pool ℐlowI_low by sorting on complete return Ri=∑tri,tR_i= _tr_i,t. The high-value timestep coverage ratio ρtraj∈(0,1] _traj∈(0,1] controls the timestep budget of ℐtopI_top. Suboptimal and high-value latent decisions are then obtained by encoding trajectories sampled from each pool: Zlow=qϕ(τlow),Ztop=qϕ(τtop).Z_low=q_φ( _low), Z_top=q_φ( _top). (12) The Quality-Uplift Mean-Flow Branch learns to transport ZlowZ_low toward ZtopZ_top, distilling high-value decision structure from existing suboptimal experiences. 4.3 Dual-Branch Conditional Latent Decision Mean Flow Mapping Both branches share the Conditional Latent Decision Mean-Flow Field vθ(Z,t0,t1,c)v_θ(Z,t_0,t_1,c) but differ in source, target, and condition. Return-sorted pairing ensures the model learns mean velocities aligned with return improvement rather than arbitrary intra-distribution variation. Each branch receives chunk-level and global conditions. The local condition for the k-th chunk is the target trajectory’s return-to-go: cchunk(b,k)=∑t=kHT−1rtop,t(b),c_chunk^(b,k)= _t=kH^T-1r_top,t^(b), (13) and the global condition is the return gap: cglobal(b)=Rtop(b)−Rlow(b).c_global^(b)=R_top^(b)-R_low^(b). (14) For the Source-to-Positive Alignment branch, the pair is (τneg,τpos)( _neg, _pos) with Rneg=0R_neg=0, so cglobal=Rpos(b)c_global=R_pos^(b). Both conditions are independently projected and injected into chunk tokens and a global condition token. 4.3.1 Source-to-Positive Alignment Mean-Flow Branch The branch predicts the mean velocity transporting ZnegZ_neg to ZposZ_pos over the unit interval [t0,t1][t_0,t_1]: U^flow(b)=vθ(Zneg(b),t0,t1,cflow(b)),Uflow∗(b)=Zpos(b)−Zneg(b)t1−t0. U_flow^(b)=v_θ\! (Z_neg^(b),\,t_0,\,t_1,\,c_flow^(b) ), U_flow^*(b)= Z_pos^(b)-Z_neg^(b)t_1-t_0. (15) 4.3.2 Quality-Uplift Mean-Flow Branch The branch predicts the mean velocity transporting ZlowZ_low to ZtopZ_top over the unit interval [t0,t1][t_0,t_1]: U^main(b)=vθ(Zlow(b),t0,t1,cmain(b)),Umain∗(b)=Ztop(b)−Zlow(b)t1−t0. U_main^(b)=v_θ\! (Z_low^(b),\,t_0,\,t_1,\,c_main^(b) ), U_main^*(b)= Z_top^(b)-Z_low^(b)t_1-t_0. (16) Refined high-level decisions are obtained by one-step integration over [t0,t1][t_0,t_1]: Zplus(b)=Zlow(b)+(t1−t0)U^main(b),Z_plus^(b)=Z_low^(b)+(t_1-t_0)\, U_main^(b), (17) which are decoded to reconstruct action sequences for the target high-value trajectory. This branch directly instantiates the core hypothesis: learnable improvement directions exist between suboptimal and high-value behaviors in high-level decision space. 4.3.3 High-Level Decision Transformer Architecture The mean-flow field vθv_θ is a Transformer with bidirectional multi-head self-attention over the chunk sequence, taking Z∈ℝK×2dzZ ^K× 2d_z as input tokens. Unlike the causal encoder and decoder, the flow field uses full attention so every chunk token can attend to the complete high-level decision sequence, enabling globally coherent mean velocity prediction. The attention sequence length K≈T/HK≈ T/H compresses the original T-step horizon into K decision units. 4.4 Hierarchical Chunk-Level Latent Decision Value Estimation and Advantage Guidance To avoid indiscriminate fitting to dataset behavior frequency, a hierarchical chunk-level advantage critic provides value guidance. Following IQL (Kostrikov et al., 2021) (see Section 3), the critic operates on chunk-level transitions (sb,k,zb,k,rb,kchunk,sb,k+1)(s_b,k,z_b,k,r_b,k^chunk,s_b,k+1) rather than primitive actions or full trajectories, and is used only during training. The Bellman target is: yb,k=rb,kchunk+γbootVν(sb,k+1),y_b,k=r_b,k^chunk+ _boot\,V_ν(s_b,k+1), (18) with γboot=0.99 _boot=0.99 and target networks maintained via Polyak averaging. The expectile value loss learns Vν(s)V_ν(s) as a high-expectile baseline of the behavioral decision value distribution. Chunk-level advantage is: Ab,k=min(Qω¯,1(sb,k,zb,k),Qω¯,2(sb,k,zb,k))−Vν(sb,k).A_b,k= \! (Q_ ω,1(s_b,k,z_b,k),\,Q_ ω,2(s_b,k,z_b,k) )-V_ν(s_b,k). (19) Advantages are converted to exponential weights: wb,k∝exp(βAb,k).w_b,k (β\,A_b,k). (20) The Quality-Uplift branch uses a weight interpolated between wb,kw_b,k and uniform to prevent extreme chunks from dominating flow supervision while preserving quality preference. The Source-to-Positive branch uses uniform weighting. Low-value chunks remain as source endpoints for the Quality-Uplift branch. The critic adjusts their training contribution rather than discarding them. 4.5 Joint Training Objective The total loss is: ℒ=λactℒact+λmainℒmain+λflowℒflow.L= _actL_act+ _mainL_main+ _flowL_flow. (21) The action reconstruction loss reweights samples by hierarchical chunk-level advantage: ℒact=b,k,t[wb,k‖a^bkt−abkttop‖22].L_act=E_b,k,t\! [w_b,k \| a_bkt-a_bkt^top \|_2^2 ]. (22) To prevent flow supervision targets from drifting jointly with encoder parameters, stop-gradient is applied to both endpoints when constructing mean velocity targets. Let sg(⋅)sg(·) denote the stop-gradient operator: Umain∗=sg(Ztop)−sg(Zlow)t1−t0,Uflow∗=sg(Zpos)−sg(Zneg)t1−t0.U_main^*= sg(Z_top)-sg(Z_low)t_1-t_0, U_flow^*= sg(Z_pos)-sg(Z_neg)t_1-t_0. (23) This ensures ℒmainL_main and ℒflowL_flow update vθv_θ only through the predicted velocity branch and do not backpropagate into the encoder via the target construction path. The encoder receives end-to-end gradients through ℒactL_act and the decoding path from ZplusZ_plus. The flow losses are normalized by the target velocity scale: ℒmain=b,k[wb,kmain‖U^main−Umain∗‖22][‖Umain∗‖22],ℒflow=b,k[‖U^flow−Uflow∗‖22][‖Uflow∗‖22].L_main= E_b,k\! [w_b,k^main \| U_main-U_main^* \|_2^2 ]E\! [ \|U_main^* \|_2^2 ], _flow= E_b,k\! [ \| U_flow-U_flow^* \|_2^2 ]E\! [ \|U_flow^* \|_2^2 ]. (24) Algorithm 1 MF2B Training and Inference 1: Input: Offline dataset D, coverage ratio ρtraj _traj; fix t0=0,t1=1t_0=0,\,t_1=1 2: Initialize: qϕq_φ, vθv_θ, pψp_ψ, Qω,1Q_ω,1, Qω,2Q_ω,2, VνV_ν 3: Sort by return; partition ℐtopI_top and ℐlowI_low 4: while not converged do 5: Sample τlow∼ℐlow _low\! \!I_low, τtop∼ℐtop _top\! \!I_top, τneg∼(0,I) _neg\! \!N(0,I), τpos∼ _pos\! \!D 6: Encode: Zlow,Ztop,Zneg,Zpos←qϕ(⋅)Z_low,Z_top,Z_neg,Z_pos← q_φ(·) 7: Compute Ab,kA_b,k, wb,kw_b,k via hierarchical chunk-level IQL critic 8: U^flow←vθ(Zneg,t0,t1,cflow) U_flow← v_θ(Z_neg,t_0,t_1,c_flow); Uflow∗←(sg(Zpos)−sg(Zneg))/(t1−t0)U^*_flow← (sg(Z_pos)-sg(Z_neg) )/(t_1-t_0) 9: U^main←vθ(Zlow,t0,t1,cmain) U_main← v_θ(Z_low,t_0,t_1,c_main); Umain∗←(sg(Ztop)−sg(Zlow))/(t1−t0)U^*_main← (sg(Z_top)-sg(Z_low) )/(t_1-t_0) 10: Zplus←Zlow+(t1−t0)U^mainZ_plus← Z_low+(t_1-t_0)\, U_main; A^top←pψ(Stop,Zplus) A_top← p_ψ(S_top,Z_plus) 11: Update θ,ϕ,ψθ,φ,ψ: minimize λactℒact+λmainℒmain+λflowℒflow _actL_act+ _mainL_main+ _flowL_flow 12: Update Qω,1,Qω,2,VνQ_ω,1,Q_ω,2,V_ν; Polyak-update target networks 13: end while 14: // Inference: Online Decision Execution 15: Input: Trained qϕq_φ, vθv_θ, pψp_ψ; target return condition c 16: for each episode do 17: Reset environment; initialize observation buffer ℬ←s0B←\s_0\ 18: while episode not done do 19: Construct source chunk τsource←(st−H+1:t,at−H+1:t−1) _source←(s_t-H+1:t,\,a_t-H+1:t-1) from buffer ℬB 20: Zt0←qϕ(τsource)Z_t_0← q_φ( _source) encode to high-level latent decision 21: Zt1←Zt0+(t1−t0)vθ(Zt0,t0,t1,c)Z_t_1← Z_t_0+(t_1-t_0)\,v_θ(Z_t_0,t_0,t_1,c) one-step mean-flow refinement 22: a^0:H−1←pψ(st−H+1:t,Zt1) a_0:H-1← p_ψ(s_t-H+1:t,\,Z_t_1) decode H actions 23: for h=0h=0 to H−1H-1 do 24: Execute a^h a_h in environment; observe st+h+1s_t+h+1, rt+hr_t+h 25: end for 26: Update buffer ℬB with new states st+1:t+Hs_t+1:t+H; t←t+Ht← t+H 27: end while 28: end for 4.6 One-Step High-Level Decision Refinement and Inference Given a source trajectory at test time, inference reduces to: Zt0 Z_t_0 =qϕ(τsource), =q_φ( _source), (25) Zt1 Z_t_1 =Zt0+(t1−t0)vθ(Zt0,t0,t1,c), =Z_t_0+(t_1-t_0)\,v_θ(Z_t_0,\,t_0,\,t_1,\,c), (26) a^0:T−1 a_0:T-1 =pψ(Zt1,s0:T−1). =p_ψ(Z_t_1,\,s_0:T-1). (27) The mean-flow field performs a single forward pass to predict the full mean velocity over [t0,t1][t_0,t_1], so no iterative ODE integration is required. The core of the method is not generating trajectories from scratch, but learning directional transport from existing suboptimal behaviors toward high-value behavioral regions in structured high-level decision space. 5 Experiments 5.1 Environmental Setup We evaluate MF2B on 9 Gym-MuJoCo locomotion tasks from the D4RL benchmark (Fu et al., 2020), covering HalfCheetah, Hopper, and Walker2d under medium, medium-replay, and medium-expert dataset conditions. 5.2 Evaluation Metrics We report D4RL normalized scores: Normalized Score=100×(score−random score)/(expert score−random score)Normalized Score=100×(score-random score)/(expert score-random score), where 0 corresponds to a uniform random policy and 100 to a domain-specific expert. For evaluation, we report the average normalized return over 5 training seeds, with each model evaluated for 50 episodes per task. 5.3 Baseline Models We compare MF2B with representative baselines across three categories: regression-based policies (BC, TD3-BC (Fujimoto and Gu, 2021), IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020), MOReL (Kidambi et al., 2020), ReBRAC (Tarasov et al., 2023), SRPO (Chen et al., 2024), CAC (Zhang et al., 2021), FAMAC, FERAC), diffusion policies (Diffuser (Janner et al., 2022), D (Ajay et al., 2022), EDP (Kang et al., 2023), IDQL (Hansen-Estruch et al., 2023), DQL (Wang et al., 2022), TCD (Hu et al., 2026)), and flow policies (FQL (Park et al., 2025), IFQL, FlowQ (Alles et al., 2025), LaMPO (Liu et al., 2026), SSCP (Koirala and Fleming, 2026), Value Flows (Dong et al., 2026), FAN (Lee et al., 2026)). 6 Experimental Results 6.1 Main Performance Comparison 6.1.1 Gym-MuJoCo Locomotion Task Comprehensive Comparison Table 1 presents the comprehensive comparison of MF2B with existing offline RL methods on 9 Gym-MuJoCo tasks. We categorize baseline models into three classes: regression-based policies, diffusion policies, and one-step flow policies. MF2B achieves an average score of 90.91 and total score of 818.22 on Gym9 tasks, performing particularly well on medium-replay datasets, which contain suboptimal and noisy trajectories posing challenges for standard policy learning. It achieves competitive performance among one-step flow policies. Table 1: D4RL Gym-MuJoCo Benchmark Comprehensive Comparison. Regression-based Policies Diffusion Policies Flow Policies Dataset BC TD3-BC IQL CQL MOReL CAC Diffuser D EDP IDQL DQL TCD FQL FlowQ SSCP MF2B HalfCheetah ME 55.2 90.7 86.7 91.6 53.3 84.3 90.3±0.1 88.9±1.9 95.8±0.1 91.3±0.6 96.8±0.3 92.7±3.4 99.8±0.1 89.6±2.3 98.1±0.9 95.47±0.31 Hopper ME 52.5 98.0 91.5 105.4 108.7 100.4 107.2±0.9 110.4±0.6 110.8±0.4 110.1±0.7 111.1±1.3 112.6±1.0 86.2±1.3 103.5±7.0 110.9±0.9 119.74±1.24 Walker2d ME 107.5 110.1 109.6 108.8 95.6 110.4 107.4±0.1 108.4±0.1 110.4±0.0 110.6±0.0 110.1±0.3 111.3±0.7 100.5±0.1 109.1±0.2 111.1±0.1 117.78±0.95 HalfCheetah M 42.6 48.3 47.4 44.0 42.1 69.1 43.8±0.1 45.3±0.3 50.8±0.0 51.5±0.1 51.1±0.5 47.2±0.7 60.1±0.1 56.8±0.8 52.3±0.5 46.69±0.38 Hopper M 52.9 59.3 66.3 58.5 95.4 80.7 89.5±0.7 98.2±0.1 72.6±0.2 70.1±2.0 90.5±4.6 99.4±0.6 74.5±0.2 88.9±6.4 102.4±0.2 100.93±0.89 Walker2d M 75.3 83.7 78.3 72.5 77.8 83.1 79.4±1.0 79.6±0.9 86.5±0.2 88.1±0.4 87.0±0.9 82.1±1.8 72.7±0.8 84.9±0.4 84.2±0.9 93.20±1.44 HalfCheetah MR 36.6 44.6 44.2 45.5 40.2 58.7 36.0±0.7 42.9±0.1 44.9±0.4 46.5±0.3 47.8±0.3 40.6±1.4 51.1±0.1 50.3±0.9 44.4±1.0 43.81±0.30 Hopper MR 18.1 60.9 94.7 95.0 93.6 99.7 91.8±0.5 99.2±0.2 83.0±1.7 99.4±0.1 101.3±0.6 97.2±2.4 85.4±0.5 99.2±1.4 101.4±0.4 104.36±2.12 Walker2d MR 26.0 81.8 73.9 77.2 49.8 79.5 58.3±1.8 75.6±0.6 87.0±2.6 89.1±2.4 95.5±1.5 88.0±1.9 82.1±1.2 88.5±2.9 85.9±13.4 96.24±1.02 Total (Gym9) 466.7 677.4 692.6 698.5 656.5 765.9 703.7 748.5 741.8 756.7 791.2 771.1 712.3 770.8 788.3 818.22 Average (Gym9) 51.9 75.3 77.0 77.6 72.9 85.1 78.2 83.2 82.4 84.1 87.9 85.7 79.2 85.6 87.9 90.91 From Table 1, MF2B achieves an average score of 90.91 on Gym9, surpassing all flow policies (FQL 79.2, FlowQ 85.6, SSCP 87.9) and most diffusion policies (DQL 87.9, TCD 85.7), demonstrating the efficiency of one-step conditional mean velocity field flow mapping on continuous control tasks. 7 Ablation Studies 7.1 Module Ablation Experiments Module ablations are conducted on 9 Gym-MuJoCo tasks. Each variant removes or replaces exactly one mechanism while keeping all other settings identical to the complete model. Direct High-Level Latent Decision Mapping (MF2B-DLM) Replaces conditional mean velocity transport with a direct mapping Z^top=fDLM(Zlow,cchunk,cglobal) Z_top=f_DLM(Z_low,c_chunk,c_global), testing whether the mean velocity flow formulation is necessary. No Value Guidance (MF2B-NVG) Removes hierarchical chunk-level IQL advantage estimation and reweighting, testing whether value guidance is required to steer high-level latent decisions toward high-value regions. Return-Ordered Uniform Pairing (MF2B-UP) Draws trajectory pairs uniformly from the full dataset while preserving return direction (Rtop≥RlowR_top≥ R_low), removing the explicit high-value and suboptimal trajectory pools. Tests whether directional pool-based pairing provides supervision beyond simple return-ordered pairs. No Source-to-Positive Alignment Mean-Flow Branch (MF2B-NAF) Sets the Source-to-Positive Alignment branch loss weight to zero, removing auxiliary supervision from Gaussian-Induced Decision Sources (ZnegZ_neg) to positive behavioral decisions (ZposZ_pos) while retaining the Quality-Uplift Mean-Flow Branch. Tests whether the structure-forming branch is necessary for establishing task-relevant behavioral structure in the high-level latent decision space. 7.1.1 Module Ablation Experiment Results Table 2 presents results on Gym-MuJoCo9. The complete MF2B (average 90.91) outperforms all variants. Key findings: (1) MF2B-DLM (−-4.47) confirms the necessity of conditional mean velocity transport over direct target mapping. (2) MF2B-NVG (−-4.07) shows chunk-level value guidance is critical for directing latent decisions toward high-value regions. (3) MF2B-UP (−-5.71) demonstrates that explicit directional pool-based pairing provides stronger supervision than uniform return-ordered pairs. (4) MF2B-NAF (−-5.28) validates that the Source-to-Positive Alignment Mean-Flow Branch—analogous to the neural null-space mechanism (Stavisky et al., 2017)—establishes task-relevant topological structures and value gradient baselines that support subsequent quality-uplift transport. Table 2: Module Ablation Experiment Results (Gym-MuJoCo 9 Tasks) Dataset MF2B-DLM MF2B-NVG MF2B-UP MF2B-NAF MF2B HalfCheetah Medium 45.12 ± 0.25 46.09 ± 0.65 45.61 ± 0.38 44.99 ± 0.13 46.69 ± 0.38 HalfCheetah Medium-Replay 43.11 ± 0.42 42.33 ± 0.19 42.97 ± 0.26 42.37 ± 0.12 43.81 ± 0.30 HalfCheetah Medium-Expert 94.01 ± 0.22 94.25 ± 0.17 94.34 ± 0.31 94.74 ± 0.49 95.47 ± 0.31 Hopper Medium 81.57 ± 2.23 90.91 ± 3.83 92.74 ± 7.14 77.46 ± 1.17 100.93 ± 0.89 Hopper Medium-Replay 99.97 ± 3.37 93.10 ± 1.80 83.00 ± 13.69 97.28 ± 2.30 104.36 ± 2.12 Hopper Medium-Expert 117.75 ± 1.39 116.92 ± 0.72 116.71 ± 0.39 115.13 ± 0.51 119.74 ± 1.24 Walker2d Medium 88.37 ± 0.87 88.16 ± 0.43 86.15 ± 0.68 87.10 ± 0.35 93.20 ± 1.44 Walker2d Medium-Replay 89.94 ± 0.32 92.10 ± 0.66 90.76 ± 4.39 91.92 ± 1.42 96.24 ± 1.02 Walker2d Medium-Expert 114.74 ± 1.27 114.26 ± 1.37 111.12 ± 0.60 111.30 ± 0.20 117.78 ± 0.95 Total (Gym9) 774.58 778.12 763.40 767.29 818.22 Average (Gym9) 86.06 86.46 84.82 85.25 90.91 7.1.2 High-Level Latent Decision Space Distribution Analysis To understand the geometric structure of the learned high-level latent decision space, we analyze four types of representations: low-return behavioral latent decisions (ZlowZ_low), high-return behavioral latent decisions (ZtopZ_top), Gaussian source negative endpoints (ZnegZ_neg), and source-to-positive positive endpoints (ZposZ_pos). Note that all analyses operate on the model’s high-level latent representations, not the original action space. Figure 1 shows t-SNE visualizations of the four distributions. ZposZ_pos closely overlaps with ZtopZ_top, confirming that the Source-to-Positive Alignment branch targets align with high-value behavioral regions. ZnegZ_neg forms a clearly separated cluster, demonstrating that Gaussian-Induced Decision Sources provide non-trivial structure-forming training signals rather than copying the behavioral distribution. ZlowZ_low and ZtopZ_top maintain partial neighborhood continuity, supporting the shared high-level latent decision space hypothesis and confirming that the Quality-Uplift branch performs directional improvement within a common representation space rather than cross-domain generation. KDE analyses in Figure 1 quantitatively corroborate these findings across the first three latent dimensions. The KDE plots in Figure 1 reveal three quantitative patterns across the first three latent dimensions. First, ZposZ_pos and ZtopZ_top exhibit nearly identical density curves in all three dimensions—their peaks, scales, and tail shapes closely coincide—confirming that the Source-to-Positive Alignment branch successfully transports Gaussian sources into the high-value behavioral region. Second, ZnegZ_neg deviates significantly from ZtopZ_top in location, scale, and shape, with particularly pronounced differences in dimensions 1 and 3 where ZnegZ_neg extends further in the negative direction with heavier tails, validating that Gaussian-Induced Decision Sources occupy a semantically neutral region distinct from task-relevant behaviors. Third, ZlowZ_low retains substantial density overlap with ZtopZ_top in the major probability mass regions of all three dimensions, confirming that low-return and high-return behaviors share a common high-level latent decision space—the geometric premise that makes directional quality-uplift transport feasible. (a) t-SNE visualization of four types of high-level latent decision representations. ZposZ_pos closely overlaps with ZtopZ_top. ZnegZ_neg forms an independent cluster. ZlowZ_low retains partial neighborhood continuity with ZtopZ_top. (b) KDE plots of the first three latent dimensions. ZposZ_pos closely follows ZtopZ_top. ZnegZ_neg significantly deviates. ZlowZ_low retains density overlap with ZtopZ_top in major regions. Figure 1: High-level latent decision space analysis. Left: t-SNE visualization of ZtopZ_top (green), ZlowZ_low (blue), ZposZ_pos (yellow), and ZnegZ_neg (red). Right: KDE plots of the first three latent dimensions comparing the four distributions. Together, these results establish three conclusions: (1) the Source-to-Positive Alignment branch constructs a non-trivial structure-forming flow from a zero-value baseline to high-value behavioral regions, (2) low-return and high-return behaviors share a common high-level latent decision space, providing the geometric premise for directional quality uplift, and (3) the dual-branch design achieves complementary roles—structure formation and directional improvement—within this shared space. 7.2 Hyperparameter Sensitivity Analysis Sensitivity experiments are conducted on three medium-replay tasks (HalfCheetah, Hopper, Walker2d) to study high-level time scale and trajectory coverage ratio independently of data distribution changes. 7.2.1 Chunk Length and High-Level Context Table 3 compares (H,Kmax)∈(5,200),(10,100),(20,50)(H,K_ )∈\(5,200),(10,100),(20,50)\ at approximately equal total primitive-step coverage. The configuration H=10,Kmax=100H=10,K_ =100 achieves the best balance. Both finer (H=5H=5) and coarser (H=20H=20) granularities slightly reduce performance, indicating that moderate temporal abstraction is most suitable for high-level latent decision flow mapping. Table 3: Results for Different Chunk Lengths and Maximum High-Level Chunk Counts Dataset H=5, KmaxK_ =200 H=10, KmaxK_ =100 H=20, KmaxK_ =50 HalfCheetah MR 43.03 ± 0.35 43.81 ± 0.30 42.95 ± 0.70 Hopper MR 100.03 ± 1.39 104.36 ± 2.12 99.41 ± 2.62 Walker2d MR 91.73 ± 1.23 96.24 ± 1.02 93.28 ± 1.85 Table 4 fixes H=10H=10 and varies Kmax∈25,50,100K_ ∈\25,50,100\. Performance degrades sharply at Kmax=25K_ =25 and Kmax=50K_ =50, confirming that the model requires sufficient high-level context to capture cross-stage task structure. Kmax=100K_ =100 covers the full 1000-step trajectory horizon. Table 4: Results for Different KmaxK_ with Fixed H=10H=10 Dataset KmaxK_ =25 KmaxK_ =50 KmaxK_ =100 HalfCheetah MR 11.74 ± 0.14 22.07 ± 0.22 43.81 ± 0.30 Hopper MR 25.69 ± 0.43 52.02 ± 0.45 104.36 ± 2.12 Walker2d MR 21.13 ± 0.86 42.58 ± 0.29 96.24 ± 1.02 7.2.2 Trajectory Coverage Ratio Table 5 compares ρtraj∈0.1,0.2,0.5,0.8 _traj∈\0.1,0.2,0.5,0.8\. Ratios of 0.1 and 0.5 both achieve performance close to the default ρtraj=0.2 _traj=0.2, showing that MF2B can learn stable suboptimal-to-high-value flow mapping directions from small high-value timestep budgets. However, ρtraj=0.8 _traj=0.8 causes sharp drops on Hopper and Walker2d, as an excessively large high-value pool reduces the directional contrast in suboptimal-to-high-value pairing. Table 5: Results for Different High-Value Timestep Coverage Ratios Dataset ρ=0.1 ρ=0.2 ρ=0.5 ρ=0.8 HalfCheetah MR 42.32 ± 0.36 43.81 ± 0.30 43.00 ± 0.51 42.66 ± 0.30 Hopper MR 101.45 ± 2.33 104.36 ± 2.12 99.49 ± 3.48 66.24 ± 5.75 Walker2d MR 96.77 ± 2.34 96.24 ± 1.02 95.70 ± 2.17 85.92 ± 8.10 8 Conclusion This paper proposes MF2B, a one-step policy optimization method for offline reinforcement learning in high-level latent decision space. MF2B reformulates offline RL as a generative policy optimization problem from suboptimal to optimal, mapping trajectory chunks to high-level decision representations through a Chunk-Level Latent Decision Encoder, and learning a shared conditional mean velocity field in this space. The method completes decision refinement through a dual-branch flow mapping mechanism: the first branch transports Gaussian-Induced Decision Sources toward positive behavioral decision regions, establishing task-relevant decision structures. The second branch transports suboptimal decisions toward high-value decision regions, distilling superior behavioral patterns from existing experiences. Hierarchical chunk-level advantage estimation further guides the flow mapping toward high-return regions. Unlike multi-step denoising diffusion policies and flow matching policies requiring multi-step ODE integration, MF2B directly predicts conditional mean velocity over the unit time interval Δt=1 t=1 and completes decision refinement through one-step Euler integration, requiring neither iterative sampling nor backpropagation through time. On 9 Gym-MuJoCo tasks from the D4RL benchmark, MF2B achieves an average score of 90.91, surpassing FlowQ (85.6) and SSCP (87.9), and significantly outperforming FQL (79.2). The high-level latent decision representation compresses sequence length from original action steps T to K≈T/HK≈ T/H, significantly reducing computational overhead in long-horizon tasks. The suboptimal-to-high-value transport mechanism effectively utilizes low-quality trajectories for policy improvement, making MF2B particularly effective in long-horizon continuous control tasks. Future research can extend MF2B to goal-conditioned RL and multi-task RL scenarios, learning conditional mean velocity fields to support diverse goal-oriented behaviors and further advancing the applicability of offline RL in real-world domains. Nevertheless, the high-level latent decision representations of MF2B are constrained by the diversity of the offline dataset, and excessively narrow behavioral support may limit the quality of the learned decision space. Addressing this through meta-learning initialization, data augmentation, or VLM-grounded latent priors remains an open direction, as does extending the framework to goal-conditioned RL and multi-task RL settings to support diverse goal-oriented behaviors. References Zhu et al. [2023] Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13344–13362, 2023. Hwangbo et al. [2019] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, 2019. Schrittwieser et al. [2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020. Perolat et al. [2022] Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T Connor, Neil Burch, Thomas Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990–996, 2022. Jumper et al. [2021] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021. Ren et al. [2018] Fang Ren, Logan Ward, Travis Williams, Kevin J Laws, Christopher Wolverton, Jason Hattrick-Simpers, and Apurva Mehta. Accelerated discovery of metallic glasses through iteration of machine learning and high-throughput experiments. Science Advances, 4(4):eaaq1566, 2018. Ma et al. [2026] Hao Ma, Zhihan Luo, Tuan V Vo, Kaiwen Sima, and Tze-Yun Leong. Centralized reward agent for knowledge sharing and transfer in multi-task reinforcement learning. Advances in Neural Information Processing Systems, 38:163080–163103, 2026. Yu et al. [2021a] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1–36, 2021a. Li et al. [2023] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3461–3475, 2023. Huang et al. [2024] Lili Huang, Baopu Dong, and Weinan Zhang. Efficient offline reinforcement learning with relaxed conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5260–5272, 2024. Kang et al. [2023] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 67195–67212, 2023. Wang et al. [2022] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022. Park et al. [2025] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Proceedings of International Conference on Machine Learning, 2025. Fujimoto and Gu [2021] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021. Kostrikov et al. [2021] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021. Stavisky et al. [2017] Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, and Krishna V. Shenoy. Motor cortical visuomotor feedback activity is initially isolated from downstream targets in output-null neural state space dimensions. Neuron, 95(1):195–208.e9, 2017. ISSN 0896-6273. doi: 10.1016/j.neuron.2017.05.023. URL https://w.sciencedirect.com/science/article/pii/S0896627317304610. Sutton et al. [1999a] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999a. Botvinick et al. [2009] Matthew M. Botvinick, Yael Niv, and Andrew G. Barto. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113(3):262–280, 2009. ISSN 0010-0277. doi: 10.1016/j.cognition.2008.08.011. Sutton and Barto [2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249. Fu et al. [2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. Sutton et al. [1999b] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999b. Fujimoto et al. [2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052–2062. PMLR, 2019. Kumar et al. [2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179–1191, 2020. Yu et al. [2021b] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34:28954–28967, 2021b. Kumar et al. [2019] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019. Hu et al. [2022] Yihao Hu, Haoran Wang, Xiangyu Zhan, Zhuoran Wang, Michael I Jordan, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2022. Hu et al. [2023] Yihao Hu, Nathan Kallus, and Xiaojie Mao. Stochastic multi-armed bandits in constant space. Journal of Machine Learning Research, 24:1–33, 2023. Levine et al. [2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. Hansen-Estruch et al. [2023] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023. Lipman et al. [2023] Yaron Lipman, Ricky T Q Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In Proceedings of International Conference on Learning Representations, 2023. Alles et al. [2025] Moritz Alles, Nianchen Chen, Patrick van der Smagt, and Botond Cseke. Flowq: Energy-guided flow policies for offline reinforcement learning. arXiv preprint arXiv:2505.14139, 2025. Wang et al. [2026] Zihan Wang, Donglin Li, Yixuan Chen, Yiming Shi, Lei Bai, Tianhe Yu, and Yang Fu. One-step generative policies with q-learning: A reformulation of meanflow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 26751–26759, 2026. Geng et al. [2025a] Zhijie Geng, Minglei Deng, Xiang Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. In Advances in Neural Information Processing Systems, 2025a. Koirala and Fleming [2026] Pratyush Koirala and Cody Fleming. Flow-based single-step completion for efficient and expressive policy learning. In Proceedings of International Conference on Learning Representations, 2026. Geng et al. [2025b] Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13447, 2025b. Kidambi et al. [2020] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 33:21810–21823, 2020. Tarasov et al. [2023] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2023. Chen et al. [2024] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In Proceedings of International Conference on Learning Representations, 2024. Zhang et al. [2021] Shangding Zhang, Yunpeng Jiang, Yu Pan, Weipeng Chen, Bosen Zhang, Mingchuan Zhang, and Jianye Hao. Cautious actor-critic. arXiv preprint arXiv:2107.05217, 2021. Janner et al. [2022] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022. Ajay et al. [2022] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022. Hu et al. [2026] Jifeng Hu, Yanchao Sun, Sili Huang, Siyuan Guo, Hechang Chen, Li Shen, Lichao Sun, Yi Chang, and Dacheng Tao. Instructed diffuser with temporal condition guidance for offline reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(5):5346–5356, 2026. doi: 10.1109/TPAMI.2026.3652193. Liu et al. [2026] Tenglong Liu, Xin Xu, Yixing Lan, Jiacheng Huang, Xinglong Zhang, and Ling-Li Zeng. Latent meanflow policy optimization for offline reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–14, 2026. doi: 10.1109/TPAMI.2026.3720530. Dong et al. [2026] Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach. Value flows. In Proceedings of the 14th International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=2VyNYUVF2k. Lee et al. [2026] Sungyoung Lee, Dohyeong Kim, Eshan Balachandar, Zelal Su Mustafaoglu, and Keshav Pingali. Towards efficient and expressive offline rl via flow-anchored noise-conditioned q-learning, 2026. URL https://arxiv.org/abs/2605.01663.