Paper deep dive
Variance-Averse $n$-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments
Guhyeon Kang, Minhae Kwon
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected $Q$-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse $n$-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (ii) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (iii) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean-variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 40 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.
Tags
Links
- Source: https://arxiv.org/abs/2610.07899v1
- Canonical: https://arxiv.org/abs/2610.07899v1
Trouble viewing inline? Open PDF directly →
Full Text
117,185 characters extracted from source content.
Expand or collapse full text
Variance-Averse n-Step Offline Reinforcement Learning for Sparse Long-Horizon Environments Guhyeon Kang Minhae Kwon †thanks: Corresponding author Affiliation: Department of Electrical and Computer Engineering Affiliation: Sungkyunkwan University, Republic of Korea Affiliation: guhyeon.kang, minhae.kwon@skku.edu Abstract Generative actors are transforming offline reinforcement learning (RL) by enabling expressive policy classes that model complex action distributions. However, this expressiveness also exposes a key challenge in heterogeneous datasets: generative policies can reproduce unreliable action modes whose return distributions exhibit high variance, occasionally yielding high returns by chance but lacking consistency. Consequently, maximizing the expected Q-value alone is insufficient for identifying reliable actions. We propose VAN-Flow (Variance-Averse n-step Flow), a framework that promotes reliable actions in generative offline RL. VAN-Flow combines (i) a categorical distributional critic, (i) a variance-averse expectation operator that smoothly reweights atom probabilities to favor actions with both high returns and low dispersion, and (i) a flow-matching generative actor guided via rejection sampling. Unlike CVaR or mean–variance objectives, the operator redistributes probability mass over the categorical return distribution without hard truncation or auxiliary penalty terms. Across more than 4040 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in long-horizon and high-variance regimes where reliable action selection becomes critical.11 1 Code: https://github.com/9hyeon1225/van 1 Introduction Generative actors, both diffusion-based [1, 2] and flow-matching policies [3, 4] are reshaping offline reinforcement learning (RL) and increasingly serve as the backbone of foundation models for robotics and physical AI. Their strength is representational: they capture multi-modal action distributions that Gaussian policies cannot. However, this very flexibility becomes a liability in offline RL. Offline datasets are typically collected from heterogeneous behavior policies of varying quality [5, 6], so the same state–action pair may yield drastically different outcomes, success in some trajectories and failure in others. A generative actor faithfully fits the dataset distribution and therefore equally represents both reliable and unreliable modes, including those that yield high returns by chance but lack consistency. Without explicit guidance, the actor may converge toward such unreliable modes [7], especially in sparse-reward, long-horizon tasks where successful trajectories are rare and even small amounts of unreliable behavior dominate value estimation. This raises a fundamental design question. How can we guide a generative actor to selectively learn the reliable behaviors in an offline dataset? We argue that reliable behaviors are those with low return variance under the dataset: a state–action pair whose returns are consistent across observed trajectories is more dependable than one that occasionally yields high returns by chance. A natural concern is whether this conflates reliability with mere safety, since a trivial action with consistently low returns is also low-variance. We address this by jointly favoring high return and low variance, so the actor is steered toward modes that are simultaneously rewarding and dependable, not merely safe. Crucially, expected Q-value alone cannot capture reliability: two state–action pairs with identical [Q]E[Q] may differ drastically in dispersion, and a regression-based critic exposes only the mean. Distributional value estimation [8, 9, 10] addresses this gap by modeling the full return distribution. To exploit this structure, we introduce a variance-averse expectation ℰ(⋅)E(·), an aggregation operator over categorical return distributions that smoothly reweights atom probabilities to jointly favor high return and low variance. Rather than defining a new risk-sensitive objective, ℰ(⋅)E(·) is designed for action selection in generative offline RL, where a policy produces multiple candidate actions that must be evaluated using a distributional critic. While risk-sensitive RL has studied measures such as CVaR [10] and entropic risk to encode risk preferences in control problems, our goal is different: improving reliability when selecting among candidate actions generated by a multimodal policy. Unlike tail-truncation approaches such as CVaR, ℰ(⋅)E(·) smoothly redistributes probability mass across the categorical support, favoring actions whose returns are both high and consistent. Figure 1: Success rate on antmaze-large under two dataset variance regimes: (a) navigate (low-variance) and (b) explore (high-variance). In (b), the shaded bar denotes the low-variance reference, and the downward arrow indicates the performance drop. This framework is especially impactful in n-step learning. While n-step returns reduce bootstrapping bias [11, 12], they amplify return variance under heterogeneous data [13, 14, 15], widening the gap between reliable and unreliable behaviors. Figure 1 confirms this: on the high-variance explore dataset, the flow-based method (FQL-n) suffers a markedly larger drop than its Gaussian counterpart (ReBRAC-n) under the same n-step setting, exposing the very vulnerability of generative actors discussed above. In contrast, our reliability-driven n-step training preserves performance with only marginal degradation. We instantiate these ideas in VAN-Flow (Variance-Averse n-step Flow), which integrates: (i) a categorical distributional critic that exposes return variance, (i) the variance-averse expectation ℰ(⋅)E(·) for reliability scoring, and (i) a flow-matching actor guided by ℰ(⋅)E(·) via rejection sampling, which re-ranks candidate actions so that the final selection aligns with the reliable subset of the dataset. Our main contributions are as follows. 1) We identify a fundamental vulnerability of generative actors in offline RL: under heterogeneous datasets, they tend to reproduce unreliable action modes whose return distributions exhibit high variance. We formalize reliable behavior through the dispersion of dataset return distributions, showing that expected Q-values alone are insufficient for guiding generative policies. 2) We propose a novel variance-averse expectation ℰ(⋅)E(·) for categorical return distributions, which smoothly reweights atom probabilities to jointly favor high-return and low-variance actions without hard truncation or auxiliary penalty terms. 3) We develop VAN-Flow, a unified framework combining a flow-matching actor, a categorical distributional critic, and ℰ(⋅)E(·)-guided rejection sampling. 4) Across more than 4040 tasks from D4RL and OGBench, VAN-Flow consistently outperforms strong baselines, with the largest gains in high-variance and long-horizon regimes. Extensive ablations and sensitivity analyses further confirm the effectiveness of each component and highlight the importance of reliability-aware action selection in n-step offline RL. To the best of our knowledge, this is the first work to use variance-averse aggregation over categorical return distributions to select reliable actions for generative actors in offline RL, and to show empirically that this matters for stable n-step learning under heterogeneous data. 2 Preliminaries Markov Decision Process. We consider an MDP ℳ=(,,P,r,γ)M=(S,A,P,r,γ) with state t∈ s_t , action t∈ a_t , transition P, reward r, and discount γ∈[0,1)γ∈[0,1). The objective is to learn a policy π(⋅∣t)π(· s_t) that maximizes the expected return (π)=π[Gt(∞)]=π[∑k=t∞γk−trk]J(π)=E_π[G_t^(∞)]=E_π[ _k=t^∞γ^k-tr_k]. Offline RL with n-step Return. In offline RL, learning relies on a fixed dataset [16] =(t,t,Gt(n),t+n)t=1ND=\( s_t, a_t,G_t^(n), s_t+n)\_t=1^N, where the n-step return Gt(n)=∑k=t+n−1γk−trkG_t^(n)= _k=t^t+n-1γ^k-tr_k approximates Gt(∞)G_t^(∞). Recent work shows that n-step returns improve performance in sparse-reward settings [17, 18]. In the actor–critic setup, the critic QψQ_ψ is trained against the n-step Bellman target, (n)Qψ(t,t)=Gt(n)+γnQψ(t+n,t+nπ), ^(n)Q_ψ( s_t, a_t)=G_t^(n)+γ^nQ_ψ( s_t+n, a_t+n^π), (1) with t+nπ∼πθ(⋅∣t+n) a_t+n^π _θ(· s_t+n), and the actor maximizes t[Qψ(t,tπ)]E_ s_t[Q_ψ( s_t, a_t^π)]. Categorical Value Estimation. A categorical critic Zψ(t,t)Z_ψ( s_t, a_t) models the full return distribution over I fixed atoms zii=1I\z_i\_i=1^I with probabilities pii=1I\p_i\_i=1^I, exposing variance structure that a regression-based critic cannot, a property we exploit in Section 3. The expected return recovers as [Gt(∞)]=∑ipiziE[G_t^(∞)]= _ip_iz_i. The critic is trained via cross-entropy against the n-step distributional Bellman target. z(n)Zψ(t,t)=Φ(∑i=1Ipi(t+n,t+nπ)δGt(n)+γnzi) _z^(n)Z_ψ( s_t, a_t)= ( _i=1^Ip_i( s_t+n, a_t+n^π)\, _G_t^(n)+γ^nz_i ) (2) Here, pi(,)p_i( s, a) is the probability that the critic assigns to atom ziz_i and δy _y is the Dirac measure at y (not to be confused with the aversion coefficient δ introduced in Section 3). In words, each atom ziz_i is shifted to Gt(n)+γnziG_t^(n)+γ^nz_i while keeping its probability mass, and the shifted distribution is projected back onto the fixed support by the projection operator Φ(⋅) (·). We employ the projection method introduced in [8] (see Appendix C). Flow-matching Policy. Flow matching parameterizes a stochastic policy through a state-conditioned velocity field vθπ(τ,t,t(τ))v^π_θ(τ, s_t, x_t(τ)) defining the ODE, dt(τ)dτ=vθπ(τ,t,t(τ)),t(0)=ϵ, d x_t(τ)dτ=v^π_θ(τ, s_t, x_t(τ)), x_t(0)= ε, (3) with ϵ∼(,) ε (0,I). The trajectory linearly interpolates noise and action, t(τ)=(1−τ)ϵ+τt, x_t(τ)=(1-τ) ε+τ a_t, (4) and the action is taken as t=t(1) a_t= x_t(1). The velocity field is trained by regressing onto t−ϵ a_t- ε [3]. 3 Variance-averse Expectation with Categorical Value Estimation Figure 2: Example of variance-averse return with varying δ and return variance. The key contribution of this work is the proposal of a variance-averse expectation that penalizes high-variance return distributions, where variance refers to the dispersion of the return distribution captured by the categorical critic. Such variance would otherwise undermine the effectiveness of the n-step method. Definition 3.1 (Variance-averse Expectation ℰ(Z)E(Z)). Let Z be a categorical return distribution with ℙ(Z=zi)=piP(Z=z_i)=p_i over ordered atoms z1<⋯<zIz_1<·s<z_I. The variance-averse expectation ℰ(Z)E(Z) is defined as follows. ℰ(Z)=∑i=1I(pi(1−(zi))δ∑j=1Ipj(1−(zj))δ)⏟variance-averse probabilityziE(Z)= _i=1^I ( p_i (1-C (z_i ) )^δ _j=1^Ip_j (1-C (z_j ) )^δ )_ [rgb]1,0,1variance-averse probabilityz_i (5) In (5), (zi)=ℙ(Z≤zi)C(z_i)=P(Z≤ z_i) denotes the cumulative distribution function (CDF) evaluated at ziz_i, and δ≥0δ≥ 0 controls the degree of aversion. As δ increases, the variance-averse expectation increasingly penalizes high-variance return distributions. When δ=0δ=0, the operator reduces to the standard expectation, i.e., ℰ(Z)=[Z]E(Z)=E[Z]. An illustrative example of ℰ(Z)E(Z) for different values of δ is presented in Figure 2.22 2 We discuss design considerations including the role of the support boundary and numerical stability in Appendix B.2. While the variance-averse expectation ℰ(Z)E(Z) in Definition 3.1 focuses on the lower-return regions of the distribution, its underlying role is to penalize high-variance return distributions. We formalize this through the convex order, denoted by ≤cx _cx, which provides a distributional notion of dispersion beyond second-order moments. Specifically, for two return distributions Z†Z and Z∘Z with identical expectations, the relation Z†≤cxZ∘Z _cxZ indicates that Z∘Z is more dispersed than Z†Z , meaning that its probability mass is spread further away from the mean. The operator in (5) is a finite-atom discretization of a spectral functional of the quantile function of Z. The following theorem establishes variance aversion exactly for that functional and bounds its gap to the implemented operator. Theorem 3.2 (Variance aversion of ℰE). Let Z be a categorical return distribution as in Definition 3.1, write ¯(zi)=1−(zi) C(z_i)=1-C(z_i) for its survival function with ¯(z0)=1 C(z_0)=1, and define the spectral form ℰsp(Z)=∑i=1I[¯(zi−1)δ+1−¯(zi)δ+1]zi=(δ+1)∫01ΩZ(u)(1−u)δu,E^sp(Z)= _i=1^I [ C(z_i-1)^δ+1- C(z_i)^δ+1 ]\,z_i=(δ+1) _0^1 _Z(u)\,(1-u)^δ\,du, (6) where ΩZ _Z is the quantile function of Z. Then, for any δ>0δ>0: (i) (Convex-order monotonicity) If [Z†]=[Z∘]E[Z ]=E[Z ] and Z†≤cxZ∘Z _cxZ , then ℰsp(Z∘)≤ℰsp(Z†)E^sp(Z ) ^sp(Z ). (i) (Discretization error) The operator ℰE in (5) is the normalized right-endpoint discretization of (6), and with R=zI−z1R=z_I-z_1, |ℰ(Z)−ℰsp(Z)|≤Rεδ(Z),εδ(Z):=1−(δ+1)∑i=1Ipi¯(zi)δ∈[0,1], |E(Z)-E^sp(Z) |\;≤\;R\, _δ(Z), _δ(Z):=1-(δ+1) _i=1^Ip_i\, C(z_i)^δ∈[0,1], (7) where εδ(Z)≤(δ+1)maxipi _δ(Z)≤(δ+1) _ip_i and, for δ≥1δ≥ 1, εδ(Z)≤δ(δ+1)∑ipi2 _δ(Z)≤δ(δ+1) _ip_i^2. Consequently, under the assumptions of (i), ℰ(Z∘)≤ℰ(Z†)+R[εδ(Z†)+εδ(Z∘)].E(Z )\;≤\;E(Z )+R [ _δ(Z )+ _δ(Z ) ]. (8) If Z=Φ(Y)Z= (Y) is the C51 projection of a target Y supported on [Vmin,Vmax][V_ ,V_ ] with a Lipschitz-continuous CDF, then maxipi=O(1/I) _ip_i=O(1/I) and εδ(Z)=O(1/I) _δ(Z)=O(1/I), so the gap vanishes as the number of atoms I increases. Part (i) states that the spectral form prefers the less dispersed of two equal-mean return distributions, and part (i) states that the implemented operator inherits this preference up to a gap governed by how concentrated the critic’s mass is on individual atoms. In particular, whenever the convex-order margin ℰsp(Z†)−ℰsp(Z∘)E^sp(Z )-E^sp(Z ) exceeds the error budget in (8), ℰE induces the same ordering as the spectral form (Corollary B.4). The certificate εδ(Z) _δ(Z) is computable from the critic output. On trained critics the realized gap is small (median 0.37%0.37\% of R), and ℰE and ℰspE^sp select the same rejection-sampling candidate in 96.9%96.9\% of states with Kendall’s τ=0.99τ=0.99 (Appendix B.3). Proofs are provided in Appendices B.1 and B.3. 4 VAN-Flow: Variance-averse n-step RL with Flow-based Policy Algorithm 1 VAN-Flow 1: Input: Dataset D, networks ZψZ_ψ, vθπv^π_θ 2: Hyperparameters: discount γ, return-step n, aversion intensity δ, num of candidates M, flow steps K, flow-matching λ, learning rates αZ,απ _Z, _π, total epochs E 3: for e=1e=1 to E do 4: Sample mini-batch B from D 5: Generate target action t+n∗ a_t+n^* via rejection sampling (10) 6: Compute target z(n)ZψT_z^(n)Z_ψ by (12) 7: Compute critic loss ℒ(ψ)L(ψ) by (11) 8: Compute actor loss ℒ(θ)L(θ) by (13) 9: ψ←ψ−αZ∇ψℒ(ψ)ψ←ψ- _Z _ψL(ψ) 10: θ←θ−απ∇θℒ(θ)θ←θ- _π _θL(θ) 11: end for Building on the variance-averse expectation in Definition 3.1, we propose VAN-Flow, a flow-based offline RL framework that operationalizes reliable-action selection for generative actors and exploits the benefits of n-step returns even under high-variance datasets. The flow-based architecture is motivated by the strong representational capacity of flow matching [19, 20], which helps mitigate OOD action generation in offline RL [21, 3]. Furthermore, we introduce a variance-averse value–guided flow-matching policy that effectively leverages the proposed variance-averse expectation. In this section, we first describe the design of the flow-based actor and then provide implementation details of the proposed method. 4.1 Flow-based Actor Design One distinctive feature of our flow-based actor is the use of rejection sampling based on the variance-averse expectation. In this subsection, we describe the action construction procedure and the corresponding rejection sampling mechanism. Flow-based Action Construction. We design an action generation mechanism that directly exploits the learned velocity field. Rather than explicitly solving the continuous-time ODE in (3), our approach constructs actions via a lightweight discretization of the flow trajectory. Specifically, given the interpolated point t(τ) x_t(τ) defined in (4), the policy evaluates the velocity field vθπ(τ,t,t(τ))v^π_θ(τ, s_t, x_t(τ)) conditioned on the current state t s_t. The action tπ a_t^π is then obtained by advancing τ x_τ along the predicted flow using an Euler update. tπ=τ+vθπ(τ,t,t(τ))K a^π_t= x_τ+ v^π_θ(τ, s_t, x_t(τ))K (9) Rejection Sampling. While a flow-based actor faithfully reproduces dataset actions, it equally reproduces both reliable and unreliable modes, and may further generate OOD actions during early training due to an incompletely learned vector field. To selectively retain reliable, in-distribution candidates, we design the flow-based actor to perform rejection sampling guided by the variance-averse expectation [22, 23]. Specifically, the actor generates M candidate actions tπ=t,1π,…,t,mπ,…,t,MπA_t^π=\ a_t,1^π,…, a_t,m^π,…, a_t,M^π\, and selects the action t∗ a_t^* that maximizes the variance-averse value as follows. t∗=argmaxt,mπ∈tπℰ(Zψ(t,t,mπ)) a_t^*= _ a_t,m^π _t^π [rgb]1,0,1E (Z_ψ ( s_t, a_t,m^π ) ) (10) Such rejection sampling enables the actor to select actions with low return variance from in-distribution samples. 4.2 Variance-averse Q-guided Flow Matching In this subsection, we provide the implementation details of VAN-Flow. The proposed method is implemented within an actor–critic framework. Our framework captures return variance through a categorical critic Zψ(t,t)Z_ψ( s_t, a_t) and employs a flow-based actor πθ(⋅∣t) _θ(· s_t). Categorical Critic Training with n-step Return. The categorical critic is trained with the standard cross-entropy loss between the predicted return distribution and the projected n-step Bellman target. ℒ(ψ)=(t,t,Gt(n),t+n)∼[CE(Zψ(t,t),z(n)Zψ(t,t))] (ψ)=E_( s_t, a_t,G_t^(n), s_t+n) [CE (Z_ψ( s_t, a_t),\,T_z^(n)Z_ψ( s_t, a_t) ) ] (11) z(n)Zψ(t,t)=Φ(∑i=1Ipi(t+n,t+n∗)δGt(n)+γnzi) _z^(n)Z_ψ( s_t, a_t)= ( _i=1^Ip_i( s_t+n, a_t+n^*)\, _G_t^(n)+γ^nz_i ) (12) The notable component is that we use the variance-averse target action t+n∗ a_t+n^* obtained via rejection sampling (10) instead of an actor-sampled action t+nπ a_t+n^π. This distinct target action selection enables valid value propagation aligned with the training objective. Projection details follow C51 [8] (see Appendix C). Flow-based Actor with Q Guidance. To train the actor network πθ(⋅∣t) _θ(· s_t), we define a variance-averse, critic-guided loss function as follows. ℒ(θ)=(t,t)∼,ϵ∼(0,),τ∼Unif[0,1][λ‖vθπ(τ,t,t(τ))−(t−ϵ)‖22−ℰ(Zψ(t,t(τ+Δτ)))⏟Q guidance] (θ)=E_ subarrayc( s_t, a_t) ,\\ ε (0,I),\\ τ [0,1] subarray [λ \|v^π_θ(τ, s_t, x_t(τ))-( a_t- ε) \|_2^2- [rgb]1,0,1E (Z_ψ ( s_t, x_t(τ+ τ) ) )_Q guidance ] (13) In (13), λ∈ℝ+λ ^+ is a constant that controls the magnitude of the flow-matching loss, and Δτ=1/K τ=1/K denotes the flow integration step size, where K is the total number of flow steps. The term t(τ+Δτ) x_t(τ+ τ) denotes a single Euler step from the current flow state t(τ) x_t(τ), defined as follows. t(τ+Δτ)=t(τ)+Δτ⋅vθπ(τ,t,t(τ)) x_t(τ+ τ)= x_t(τ)+ τ· v^π_θ(τ, s_t, x_t(τ)) (14) Here, t(τ+Δτ) x_t(τ+ τ) serves as a single-step action proxy through which the critic gradient is propagated to the velocity field, acting as a computationally efficient surrogate for the fully integrated action. Unlike prior work that employs full ODE integration for the actor [1, 24], we reduce the ODE computation cost by using a single Euler step. Under the actor loss in (13), the flow-based actor is trained to favor actions with low return variance, thereby mitigating training instability caused by high-variance n-step returns. The pseudocode of the proposed method is presented in Algorithm 1. For implementation, we also use a target network, as commonly employed for training stabilization [25, 26, 27], along with the double Q method to mitigate overestimation bias [28, 29, 30]. 5 Simulation Result We conduct extensive experiments to evaluate VAN-Flow on sparse, long-horizon reinforcement learning benchmarks. All results are reported as averages over five random seeds. Experiments are performed on a system equipped with an AMD Ryzen Threadripper PRO 9975WX (CPU) and an NVIDIA RTX PRO 6000 (GPU). 5.1 Simulation Setup Benchmark. To evaluate our method under the core challenges of offline RL in sparse, long-horizon settings, we focus on two benchmark tasks that are widely regarded as difficult due to sparse rewards and long-horizon structure, and that additionally exhibit substantial return-variance changes across dataset variants. A detailed explanation of each benchmark task is presented in Appendix D. • D4RL [31]: We evaluate our method on the AntMaze tasks from D4RL, which remain challenging due to their long-horizon structure and sparse rewards. While many D4RL benchmarks exhibit performance saturation [32, 33], AntMaze continues to expose limitations in long-term credit assignment and value extrapolation. • OGBench [34]: OGBench is a recently introduced benchmark designed to evaluate offline RL under long-horizon control, sparse and deceptive rewards, and limited dataset coverage. We focus on state-based OGBench tasks with noisy and suboptimal datasets, where sensitivity to out-of-distribution actions is pronounced, providing a stringent test for behavior-regularized, value-guided methods. Baseline. We compare our proposed VAN-Flow under the following evaluation criteria. Detailed descriptions of the baseline algorithms are provided in Appendix H.1. • Policy Architecture: We evaluate methods employing Gaussian-based policies (IQL [35], ReBRAC [36], HIQL [37]), which are the conventional approach for representing stochastic policies. In addition, we compare with methods using flow-based policies (FQL [3], BFN [4], FQL-n, BFN-n [4], QC [4]), which are most closely related to our actor design. • Algorithm Design: For an extensive comparison, we evaluate n-step return methods (Retrace(λ) [38], PQL [39], LEQ [17], TD3BC+MS [40], QC [4])33 3 Retrace(λ) and PQL were originally proposed for off-policy online RL. In our experiments, we adapt them to the fixed offline-dataset setting. and distributional critics (PA-RL [41]). We also include D4PG [42], which incorporates both a distributional critic and n-step return, with an added behavior cloning (BC) term [43]. In addition, pure offline RL methods (IQL [35], ReBRAC [36], FQL [3]) are considered as baselines that use 11-step returns. 5.2 Performance Comparison In this subsection, we present an extensive performance comparison of the proposed method.44 4 We use results reported in [34, 32, 4] along with our own experiments. Additional experiments are conducted only for tasks not covered in the original papers. Details are provided in Appendix H.2. Learning curves, the full performance table, and additional comparison results are provided in Appendices I.1, I.2, and I.3, respectively. Furthermore, Off-to-online fine-tuning results are reported in Appendix J. Table 1: Performance comparison on OGBench. Each entry averages five tasks per environment. Best results are in bold. Gaussian-based Flow-based Proposed Task IQL HIQL ReBRAC ReBRAC-n FQL BFN FQL-n BFN-n QC VAN-Flow Standard tasks antmaze-large-navigate 53 91 81 82 79 71 91 81 49 95 antmaze-teleport-navigate 23 42 7 35 26 40 22 41 40 57 scene-play 0 38 11 40 57 85 18 57 84 93 puzzle-3x3-play 0 12 55 10 100 98 98 58 100 100 Long-horizon tasks humanoidmaze-large-navigate 2 49 2 4 4 26 20 46 16 81 antmaze-giant-navigate 4 65 26 16 9 11 54 5 2 68 humanoidmaze-giant-navigate 1 12 0 0 1 0 3 74 12 92 Noisy tasks antmaze-large-explore 15 4 48 33 20 22 1 17 58 84 puzzle-3x3-noisy 33 51 2 10 100 100 33 98 100 97 Table 2: Offline performance on D4RL AntMaze tasks. Scores are normalized following the D4RL protocol. Entries marked “–” indicate results not reported in prior work. Best results are in bold. Pure Offline RL n-step Dist. Dist.+n Proposed Task IQL ReBRAC FQL Retrace(λ) PQL LEQ QC TD3BC+MS PA-RL D4PG VAN-Flow umaze 77.0±5.5± 5.5 97.7±1.5± 1.5 96.0±2.0± 2.0 96.0±3.0± 3.0 94.0±3.0± 3.0 94.4±6.3± 6.3 95.0±1.4± 1.4 – – 87.6±4.1± 4.1 98.0±2.8± 2.8 umaze-diverse 54.2±5.5± 5.5 83.5±7.0± 7.0 89.0±2.0± 2.0 84.0±10.0± 10.0 63.0±6.0± 6.0 71.0±12.3± 12.3 78.0±8.5± 8.5 – – 61.3±9.4± 9.4 95.6±2.6± 2.6 medium-play 65.7±11.7± 11.7 89.5±3.3± 3.3 78.0±7.0± 7.0 79.0±10.0± 10.0 81.0±6.0± 6.0 58.8±33.0± 33.0 72.0±2.4± 2.4 70.0 88.0 89.1±3.0± 3.0 90.7±3.1± 3.1 medium-diverse 73.7±5.4± 5.4 83.5±8.2± 8.2 71.0±13.0± 13.0 53.0±17.0± 17.0 60.0±33.0± 33.0 46.2±23.2± 23.2 64.0±2.8± 2.8 66.0 88.0 81.3±4.1± 4.1 89.3±5.0± 5.0 large-play 42.0±4.5± 4.5 52.2±29.0± 29.0 84.0±7.0± 7.0 12.0±26.0± 26.0 33.0±10.0± 10.0 58.6±9.1± 9.1 80.0±2.8± 2.8 22.0 87.0 71.3±4.1± 4.1 86.5±5.7± 5.7 large-diverse 30.2±3.6± 3.6 64.0±5.4± 5.4 83.0±4.0± 4.0 20.0±3.0± 3.0 44.0±9.0± 9.0 60.2±18.3± 18.3 82.0±5.6± 5.6 28.0 73.0 80.6±6.1± 6.1 88.6±3.0± 3.0 Comparison of Policy Architecture. Table 1 compares different policy architectures across tasks with varying horizon lengths and noise levels. Overall, Gaussian-based policies (IQL, ReBRAC, HIQL) exhibit strong performance on several tasks but show noticeable degradation in long-horizon and noisy environments, reflecting limited expressiveness under complex multi-modal action distributions. Flow-based policies substantially improve performance in these challenging settings, demonstrating the advantage of expressive action modeling; however, naive flow-based methods (FQL, BFN) remain unstable and sensitive to distributional shift. While incorporating n-step returns into flow-based policies (FQL-n, BFN-n) yields more robust performance, these methods still suffer from performance drops in high-variance regimes. In contrast, VAN-Flow consistently achieves strong results across long-horizon and noisy tasks, indicating that combining expressive flow-based policies with variance-aware value guidance is crucial for stable and effective offline RL. In particular, on high-variance datasets such as antmaze-large-explore, flow-based n-step methods (FQL-n, BFN-n) collapse far more severely than their Gaussian counterpart (ReBRAC-n) under the same n-step setting, while VAN-Flow remains robust. Comparison of Algorithmic Design. Table 2 compares representative design choices on D4RL maze tasks. Pure offline RL methods perform competitively on small-scale tasks but degrade as the horizon and dataset diversity increase. Methods using only n-step returns (Retrace(λ), PQL, LEQ, QC, TD3BC+MS) show inconsistent performance across task scales and often drop sharply on large or diverse settings. PA-RL, which uses a distributional critic without n-step returns, is strong on the medium mazes but falls behind on large-diverse. Even D4PG, which combines both components, fails to maintain robust performance as task difficulty increases. In contrast, VAN-Flow consistently achieves strong results from umaze to large-diverse, indicating that explicit return-variance control is crucial for stably leveraging n-step returns in offline RL. 5.3 Ablation Studies We conduct controlled ablations to isolate the effect of each major component while keeping all other components fixed. Specifically, we examine (1) the distributional critic and flow-based policy, (2) the variance-averse expectation, and (3) the Q guidance mechanism. For clarity, we report representative results on one task from each environment. Additional component-wise ablations that further isolate the contributions of variance-averse expectation, the distributional critic, and the flow-based policy across action-dimension regimes are provided in Appendix F. Categorical Critic and Flow-based Actor. Figure 3: Effect of flow-based policies and categorical critics on humanoidmaze-giant-navigate. Figure 3 demonstrates the effect of combining a distributional critic with a flow-matching actor. In this result, our proposed combination (i.e., Flow+Categorical) consistently outperforms methods that rely partially on a regression-based critic or a Gaussian-based actor. Moreover, the consistent superiority of the flow-based actor over the Gaussian-based counterpart highlights its effectiveness in addressing long-horizon and sparse-reward problems. While Flow+Regression also exhibits improved performance, it is consistently outperformed by the VAN-Flow, underscoring the importance of capturing and penalizing return variance via a categorical critic. We further validate the choice of flow matching over diffusion-based parameterizations in Appendix G, where flow matching achieves over 90% success rate with only 3 flow steps while a diffusion-based variant requires 20 steps for comparable performance. Table 3: Ablation comparing return aggregation rules. Task Metric [Z]E[Z] CVaR Entropic Mean–Var ℰ(Z)E(Z) humanoidmaze -large-navigate Var[Z]↓Var[Z]\! 1.001.00 0.950.95 0.970.97 0.940.94 0.930.93 Success↑Success\! 7777 8383 7979 8080 87 antmaze -teleport-navigate Var[Z]↓Var[Z]\! 1.001.00 1.081.08 0.930.93 0.900.90 0.620.62 Success↑Success\! 3838 5454 4444 4646 71 antmaze-large-explore Var[Z]↓Var[Z]\! 1.001.00 0.870.87 0.720.72 0.770.77 0.660.66 Success↑Success\! 8484 22 8686 8585 93 Variance-averse Expectation. We evaluate whether the proposed variance-averse expectation suppresses high-variance return distributions and how it compares to existing risk-sensitive aggregation rules. To this end, we replace the aggregation rule in (10) and (13) with [Z]E[Z], CVaR, entropic risk, mean–variance, and ℰ(Z)E(Z), while keeping all other components unchanged.55 5 For each baseline, the associated hyperparameter (e.g., the confidence level of CVaR or the penalty coefficient β of entropic risk and mean–variance) is selected via a per-task sweep that reports the best-performing value. We report two complementary metrics: Var[Z]Var[Z], which measures the dispersion of the learned return distribution on training samples after convergence and is normalized by its value under [Z]E[Z] within each task, and SuccessSuccess, which measures task performance. As shown in Table 3, ℰ(Z)E(Z) consistently achieves the lowest Var[Z]Var[Z] across all tasks, and this variance reduction directly translates into the highest success rates. These results indicate that suppressing dispersion in return distributions is critical for reliable action selection in sparse, long-horizon offline RL. Existing risk-sensitive rules fall short for structural reasons. CVaR focuses on the lower tail of the distribution, while mean–variance and entropic risk introduce auxiliary penalty terms of the form Q−βVarQ- or Q−βEntropyQ- that require task-specific tuning. In contrast, ℰ(Z)E(Z) acts as a single aggregation operator over the categorical return distribution, redistributing probability mass to favor actions that are both high-return and low-variance without hard truncation or auxiliary penalties. This simple formulation consistently improves both variance and success using a fixed δ=2δ=2 across nearly all tasks. Importantly, the operator is directly compatible with categorical distributional critics, requiring no modification to the underlying distributional RL training procedure. Additional comparisons with other risk measures (CPW, Norm, Wang, Sharpe, Sortino) are provided in Appendix K. Q Guidance. Figure 4: Effect of Q guidance on policy performance across multiple tasks. Bars report mean success rate and time efficiency. We also examine the effect of the Q guidance term in (13) on actor performance. To evaluate its impact, we consider two metrics: (1) the success rate, which measures whether the actor successfully solves the task, and (2) the time efficiency, which assesses how rapidly the task is completed. The time efficiency is computed from the actual episode length T required to reach the goal, normalized as TE=100⋅(1−T/Tmax)TE=100·(1-T/T_ ), where TmaxT_ denotes the maximum episode horizon. Higher values thus indicate faster task completion. Figure 4 clearly demonstrates that the Q-guided actor achieves both high success rates and high time efficiency, consistently outperforming an actor trained without Q guidance. In contrast, while the actor without Q guidance may achieve high success rates in some tasks, its time efficiency is often extremely low, indicating inefficient decision-making in the absence of explicit Q guidance. 5.4 A Close Look at VAN-Flow In this subsection, we provide a sensitivity analysis on the humanoidmaze-giant-navigate task to examine how key algorithmic parameters (e.g., the n-step horizon, the number of rejection samples, and the number of flow steps) affect the performance. The sensitivity analyses across various tasks are presented in Appendix L. Figure 5: Sensitivity analysis with respect to key algorithmic parameters. n-step Horizon. We examine the impact of varying the n-step return horizon with Figure 5(a). For long-horizon tasks, increasing n substantially improves performance by enabling more effective long-term credit assignment. However, excessively large values of n lead to diminishing or even negative returns, reflecting the amplified variance and instability of long-horizon targets. Importantly, the proposed method remains robust across a wide range of n, indicating that variance-aware learning alleviates the sensitivity typically associated with long-horizon returns. Number of Rejection Samples. The effect of the number of candidate actions used in rejection sampling is presented in Figure 5(b). Performance consistently improves as more samples are considered, particularly on challenging tasks where conservative target action selection is critical. Beyond a moderate number of samples, the improvement gradually saturates, with most tasks exhibiting stable performance using fewer than 3232 candidates. This suggests that reliable variance-averse target action selection can be achieved without excessive computational overhead. Number of Flow Steps. Figure 5(c) analyzes sensitivity to the number of flow integration steps for action generation. A single integration step is not enough, but performance is already high with three to five steps and changes only mildly beyond that, so the proposed actor does not depend on precise numerical integration. Overall, these results show that evaluating critic values on flow-induced actions enables stable optimization with flexible integration granularity. 5.5 Computational Complexity Analysis Table 4: Runtime comparison on humanoidmaze-large-navigate. Method Complexity Step (ms) Wall (h) FQL (UKd2)O(UKd^2) 1.1 0.32 BFN (UMKd2)O(UMKd^2) 2.1 0.60 QC (UMKd2)O(UMKd^2) 3.0 0.83 VAN-Flow (UMKd2)O(UMKd^2) 1.9 0.54 VAN-Flow introduces additional computation due to rejection sampling and flow-based action generation. Table 4 compares VAN-Flow with representative generative-policy methods, including FQL, BFN, and QC, in terms of both theoretical complexity and practical runtime. The overall training complexity of VAN-Flow is (UMKd2)O(UMKd^2), where U denotes the number of gradient updates, d the network width (assuming fully-connected layers), M the number of rejection-sampling candidates, and K the number of flow steps. The factors M and K arise from evaluating M candidate actions generated through K flow transformations during policy optimization. Although this matches the asymptotic order of QC, the empirical overhead remains modest: VAN-Flow is faster than QC in both step time and wall-clock time, while remaining comparable to other generative-policy baselines. As discussed in Section 5.4, strong performance is achieved with relatively small values of M and K, which keeps the practical computational cost low. 6 Related Works Offline RL with n-step Returns. n-step returns have recently regained attention in offline RL as an effective tool for mitigating long-horizon bootstrapping bias. PiLaR [44] analyzes unfavorable bias–variance trade-offs of naive n-step returns and introduces compound return estimators. Decoupled Q-chunking [45] eliminates pessimistic bias by decoupling policy and critic horizons, while LEQ [17] stabilizes long-horizon learning using n-step λ-returns with expectile regression. Other routes to long-horizon offline RL augment transitions by temporal distance [46] or build foresight into the agent [47, 48]. Generative Policies for Reinforcement Learning. Recent offline RL methods increasingly adopt expressive generative models to parameterize policies. Diffusion-based approaches, including Diffusion Policy [2] and Diffuser [49], model complex action or trajectory distributions, while Diffusion-QL [1] and IDQL [50] integrate critic-based guidance to move beyond pure imitation. Flow-matching-based policies offer a scalable alternative, as exemplified by Flow Q-Learning [3], Q-chunking [4], and energy-weighted flow matching [20]. Distributional and Risk-sensitive RL. Distributional RL methods such as C51 [8], QR-DQN [9], and IQN [10] model the full return distribution but typically aggregate it via the standard expectation. Risk-sensitive variants based on CVaR [10], entropic risk, or mean–variance objectives modify the aggregation rule, but were designed to encode risk preferences in online RL rather than to suppress unreliable, high-variance targets in offline n-step learning. Distortion-based aggregation in actuarial risk theory [51, 52] reweights continuous distributions through the survival probability. Our variance-averse expectation ℰ(⋅)E(·) instead operates on discrete categorical critics with explicit normalization, reweighting all atoms without hard truncation while penalizing dispersion. 7 Conclusion We investigate offline RL in sparse, long-horizon environments and show that naive horizon reduction via n-step returns fundamentally breaks down on high-variance datasets. To address this issue, we propose VAN-Flow, a variance-averse offline RL framework that combines n-step learning with categorical distributional value estimation. VAN-Flow introduces a variance-averse expectation that downweights unreliable high-variance return targets and couples it with a flow-based actor guided by variance-averse Q estimates and rejection sampling. Across challenging benchmarks, our method consistently outperforms strong baselines. To our knowledge, VAN-Flow is the first framework to select reliable behaviors for generative actors through variance-averse aggregation over categorical return distributions, and our results show that this choice matters for n-step offline RL under heterogeneous data. Acknowledgments This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No. RS-2026-25519475), the National Research Foundation of Korea (NRF) grants (RS-2023-00278812, RS-2025-02214082, RS-2026-25522801) funded by the Korean government (MSIT), and Korea Institute of Police Technology (KIPoT) funded by the Korean National Police Agency & Korea Customs Service (RS-2026-25536784). The computing resources were partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP, AI Computing Support Project for R&D) grant funded by the Korea government(MSIT) (RS-2026-25505492). The authors would like to thank Chanin Eom for his helpful discussions and comments during this research. The authors declare no competing interests. References [1] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2023. [2] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10–11):1684–1704, 2025. [3] Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning. In International Conference on Machine Learning (ICML), 2025. [4] Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [5] Jeyeon Eo, Dongsu Lee, and Minhae Kwon. The impact of dataset on offline reinforcement learning performance in uav-based emergency network recovery tasks. IEEE Communications Letters, 28(5):1058–1061, 2024. [6] Dongsu Lee, Chanin Eom, and Minhae Kwon. AD4RL: Autonomous driving benchmarks for offline reinforcement learning with value-based dataset. In IEEE International Conference on Robotics and Automation, 2024. [7] Chanin Eom, Dongsu Lee, and Minhae Kwon. Selective imitation for efficient online reinforcement learning with pre-collected data. ICT Express, 10(6):1308–1314, 2024. [8] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning (ICML), 2017. [9] Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence (AAAI), 2018. [10] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning (ICML), 2018. [11] Michael J. Kearns and Satinder Singh. Bias-variance error bounds for temporal difference updates. In Conference on Computational Learning Theory (COLT), 2000. [12] Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes RL scalable. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [13] Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In International Conference on Machine Learning (ICML), 2000. [14] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2nd edition, 2018. [15] Tongzheng Ren, Jialian Li, Bo Dai, Simon S. Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021. [16] Dongsu Lee, Chanin Eom, Sungwoo Choi, Sungkwan Kim, and Minhae Kwon. Survey on practical reinforcement learning : from imitation learning to offline reinforcement learning. Journal of Korean Institute of Communications and Information Sciences, 48(11):1405–1417, 2023. [17] Kwanyoung Park and Youngwoon Lee. Model-based offline reinforcement learning with lower expectile Q-learning. In International Conference on Learning Representations (ICLR), 2025. [18] Kaylee Burns, Tianhe Yu, Chelsea Finn, and Karol Hausman. Offline reinforcement learning at multiple frequencies. In Conference on Robot Learning (CoRL), 2022. [19] Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, and Vaneet Aggarwal. Deep generative models for offline policy learning: Tutorial, survey, and perspectives on future directions. Transactions on Machine Learning Research, 2024. [20] Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. In International Conference on Learning Representations (ICLR), 2025. [21] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [22] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations (ICLR), 2023. [23] Christian P. Robert and George Casella. Monte Carlo Statistical Methods. Springer, New York, 2nd edition, 2004. [24] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 9(4):3116–3123, 2024. [25] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016. [26] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019. [27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. [28] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018. [29] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In AAAI Conference on Artificial Intelligence (AAAI), 2016. [30] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018. [31] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. [32] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023. [33] Rafael Rafailov, Kyle Beltran Hatch, Anikait Singh, Aviral Kumar, Laura Smith, Ilya Kostrikov, Philippe Hansen-Estruch, Victor Kolev, Philip J. Ball, Jiajun Wu, Sergey Levine, and Chelsea Finn. D5RL: Diverse datasets for data-driven deep reinforcement learning. Reinforcement Learning Journal, 5:2178–2197, 2024. [34] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In International Conference on Learning Representations (ICLR), 2025. [35] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations (ICLR), 2022. [36] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [37] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline goal-conditioned RL with latent states as actions. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [38] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016. [39] Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting Peng’s Q(λ) for modern reinforcement learning. In International Conference on Machine Learning (ICML), 2021. [40] Zhiyong Peng, Yadong Liu, and Zongtan Zhou. Deadly triad matters for offline reinforcement learning. Knowledge-Based Systems, 284:111341, 2024. [41] Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic RL: Offline RL and online RL fine-tuning of any class and backbone. arXiv preprint arXiv:2412.06685, 2024. [42] Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations (ICLR), 2018. [43] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021. [44] Brett Daley, Martha White, and Marlos C. Machado. Averaging n-step returns reduces variance in reinforcement learning. In International Conference on Machine Learning (ICML), 2024. [45] Qiyang Li, Seohong Park, and Sergey Levine. Decoupled Q-chunking. In International Conference on Learning Representations (ICLR), 2026. [46] Dongsu Lee and Minhae Kwon. Temporal distance-aware transition augmentation for offline model-based reinforcement learning. In International Conference on Machine Learning, 2025. [47] Dongsu Lee and Minhae Kwon. Episodic future thinking mechanism for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [48] Dongsu Lee and Minhae Kwon. Foresighted decisions for inter-vehicle interactions: An offline reinforcement learning approach. In IEEE International Conference on Intelligent Transportation Systems, 2023. [49] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022. [50] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023. [51] Menahem E. Yaari. The dual theory of choice under risk. Econometrica, 55(1):95–115, 1987. [52] Shaun Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin, 26(1):71–92, 1996. [53] Moshe Shaked and J. George Shanthikumar. Stochastic Orders. Springer, New York, 2007. [54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. [55] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016. [56] Guhyeon Kang, Jaehwi Lee, and Minhae Kwon. Simple actors and deep critics for scalable reinforcement learning. In ACM International Conference on Information and Knowledge Management, 2026. [57] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In International Conference on Learning Representations (ICLR), 2024. [58] Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In International Conference on Learning Representations (ICLR), 2024. [59] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020. [60] Dongsu Lee and Minhae Kwon. Scenario-free autonomous driving with multi-task offline-to-online reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 26(9):13317–13330, 2025. [61] Dongsu Lee and Minhae Kwon. Episodic future thinking with offline reinforcement learning for autonomous driving. IEEE Internet of Things Journal, 12(11):17012–17023, 2025. [62] Guhyeon Kang, Jaehwi Lee, Chanin Eom, and Minhae Kwon. Offline reinforcement learning based autonomous driving on complex roads with on-ramp merging and bottleneck. Transactions of the Korean Society of Automotive Engineers, 33(6):465–479, 2025. [63] Sangeun Park, Chanin Eom, and Minhae Kwon. xLSTM-based noise-robust driving characteristic inference network for V2I systems. Transactions of the Korean Society of Automotive Engineers, 33(7):487–502, 2025. [64] Jeyeon Eo, Dongsu Lee, and Minhae Kwon. Offline reinforcement learning based uav training for disaster network recovery. Journal of Korean Institute of Communications and Information Sciences, 49(1):1–11, 2024. [65] Jaehwi Lee, Chanin Eom, Chan Kim, Kyeongsoo Kim, Hyeongdo Lee, Hyunsu Kang, and Minhae Kwon. Ai fire support officer: Military decision support system based on reward adaptive reinforcement learning. Journal of Korean Institute of Communications and Information Sciences, 51(1):209–222, 2026. [66] Jaehwi Lee, Chanin Eom, Kyeongsoo Kim, Hyunsu Kang, and Minhae Kwon. Deep reinforcement learning based weapon-target assignment to support military decision-making. Journal of Korean Institute of Communications and Information Sciences, 50(6):884–895, 2025. Appendix A Notation Tables.A B Theoretical Analysis of the Variance-Averse Expectation.B B.1 Proof of Theorem 3.2(i).B.1 B.2 Design Considerations.B.2 B.3 Proof of Theorem 3.2(i): Discretization Error.B.3 C Projection Operator.C D Benchmarks and Task Descriptions.D D.1 D4RL.D.1 D.2 OGBench.D.2 E Implementation Details of VAN-Flow.E F Component-wise Ablation Study.F G Empirical Comparison of Flow Matching and Diffusion Policies.G H Baselines Details.H H.1 Baselines Description.H.1 H.2 Sources of Reported Results.H.2 I Extensive Results of Performance Comparisons.I I.1 Learning Curve of VAN-Flow.I.1 I.2 Full Performance Table.I.2 I.3 Performance Comparison in Denser Reward Environments.I.3 I.4 D4RL Locomotion Tasks.I.4 J Off-to-Online Fine-Tuning Results.J K Additional Comparisons with Risk-Sensitive Objectives.K L Full Results on Sensitivity Analyses.L L.1 n-step Horizon.L.1 L.2 Number of Rejection Samples.L.2 L.3 Number of Flow Steps.L.3 L.4 Aversion Coefficient δ.L.4 L.5 Categorical Support.L.5 L.6 Number of Atoms.L.6 M Broader Impact and Limitation.M Appendix A Notation Tables Table 5 summarizes the key notations used throughout the paper. Table 5: Notation table. Notation Description ℳ=(,,P,r,γ)M=(S,A,P,r,γ) Markov decision process πθ(⋅∣s) _θ(· s) Flow-based policy (actor) with parameters θ vθπ(τ,s,x)v^π_θ(τ,s,x) Flow-matching velocity field (policy parameterization) Qψ(s,a)Q_ψ(s,a) Scalar critic (used in standard 1-step formulations) Zψ(s,a)Z_ψ(s,a) Categorical return distribution critic used in VAN-Flow Z(s,a)Z(s,a) Return random variable (categorical distribution) for (s,a)(s,a) zii=1I\z_i\_i=1^I Fixed support atoms for categorical distribution pi(s,a)i=1I\p_i(s,a)\_i=1^I Atom probabilities of Zψ(s,a)Z_ψ(s,a), ∑ipi=1 _ip_i=1 D Offline dataset of transitions/segments Gt(n)=∑k=0n−1γkrt+kG_t^(n)= _k=0^n-1γ^kr_t+k n-step truncated return (n)T^(n) n-step Bellman operator (scalar) z(n)T^(n)_z n-step distributional Bellman operator Φ(⋅) (·) Categorical projection operator onto fixed atoms (zi)=ℙ(Z≤zi)C(z_i)=P(Z≤ z_i) CDF of the return random variable Z at ziz_i ¯(zi)=1−(zi) C(z_i)=1-C(z_i) Survival function of Z at ziz_i, with ¯(z0)=1 C(z_0)=1 ΩZ(u) _Z(u) Generalized quantile function of return distribution Z FZ(u)=∫0uΩZ(t)tF_Z(u)= _0^u _Z(t)dt Integrated quantile function ℰ(Z)E(Z) Variance-averse expectation, implemented operator (Def. 3.1) ℰsp(Z)E^sp(Z) Spectral form of ℰE, exact finite-sum representation (6) εδ(Z) _δ(Z) Discretization-gap certificate of Theorem 3.2(i) δ Aversion intensity for ℰ(⋅)E(·) t∗ a_t^* Action selected via variance-averse rejection sampling tπA_t^π Set of M candidate actions sampled from the policy τ∼Unif[0,1]τ [0,1] Flow time variable K Number of flow discretization steps M Number of rejection-sampling candidates λ Weight for flow-matching regression term ≤cx _cx Convex order Appendix B Theoretical Analysis of the Variance-Averse Expectation B.1 Proof of Theorem 3.2(i) Throughout Appendix B, Z is a categorical return distribution with probabilities pi≥0p_i≥ 0, ∑i=1Ipi=1 _i=1^Ip_i=1, on ordered atoms z1<⋯<zIz_1<·s<z_I. We write i:=(zi)=∑j≤ipjC_i:=C(z_i)= _j≤ ip_j for its CDF, with 0:=0C_0:=0, and Si:=¯(zi)=1−iS_i:= C(z_i)=1-C_i for its survival function, so that S0=1S_0=1, SI=0S_I=0, and pi=Si−1−Sip_i=S_i-1-S_i. The generalized quantile function of Z is ΩZ(u)=zi,u∈(i−1,i], _Z(u)=z_i, u∈ (C_i-1,\,C_i ], (15) and wi⋆:=Si−1δ+1−Siδ+1w_i :=S_i-1^δ+1-S_i^δ+1 denotes the weights of the spectral form (6). We first verify that the two expressions in (6) coincide. Lemma B.1 (Exact finite-sum representation). For every δ≥0δ≥ 0, (δ+1)∫01ΩZ(u)(1−u)δu=∑i=1Iwi⋆zi(δ+1) _0^1 _Z(u)(1-u)^δ\,du= _i=1^Iw_i z_i. The weights wi⋆w_i are nonnegative and sum to one, and wi⋆=piw_i =p_i when δ=0δ=0. Proof. By (15), ΩZ _Z is constant on each interval (i−1,i](C_i-1,C_i], hence (δ+1)∫01ΩZ(u)(1−u)δu (δ+1) _0^1 _Z(u)(1-u)^δ\,du =∑i=1Izi∫i−1i(δ+1)(1−u)δu = _i=1^Iz_i _C_i-1^C_i(δ+1)(1-u)^δ\,du =∑i=1Izi[(1−i−1)δ+1−(1−i)δ+1]=∑i=1Iwi⋆zi. = _i=1^Iz_i [(1-C_i-1)^δ+1-(1-C_i)^δ+1 ]= _i=1^Iw_i z_i. (16) Nonnegativity follows from Si−1≥SiS_i-1≥ S_i, the sum telescopes to S0δ+1−SIδ+1=1S_0^δ+1-S_I^δ+1=1, and for δ=0δ=0 we have wi⋆=Si−1−Si=piw_i =S_i-1-S_i=p_i. ∎ Proof of Theorem 3.2(i). By Lemma B.1, ℰsp(Z)=(δ+1)∫01ΩZ(u)(1−u)δuE^sp(Z)=(δ+1) _0^1 _Z(u)(1-u)^δ\,du. Define FZ(u):=∫0uΩZ(t)tF_Z(u):= _0^u _Z(t)\,dt. Integrating by parts, ℰsp(Z) ^sp(Z) =(δ+1)[FZ(u)(1−u)δ]01+(δ+1)δ∫01FZ(u)(1−u)δ−1u =(δ+1) [F_Z(u)(1-u)^δ ]_0^1+(δ+1)δ _0^1F_Z(u)(1-u)^δ-1\,du =(δ+1)δ∫01FZ(u)(1−u)δ−1u, =(δ+1)δ _0^1F_Z(u)(1-u)^δ-1\,du, (17) where the boundary term vanishes because FZ(0)=0F_Z(0)=0, FZF_Z is bounded, and (1−u)δ→0(1-u)^δ→ 0 as u→1u→ 1 for δ>0δ>0. For 0<δ<10<δ<1 the weight (1−u)δ−1(1-u)^δ-1 is integrable on [0,1][0,1], so the right-hand side is well defined. Now let Z†Z and Z∘Z satisfy [Z†]=[Z∘]E[Z ]=E[Z ] and Z†≤cxZ∘Z _cxZ . The quantile characterization of the convex order (see, e.g., [53, Sec. 3.A]) states that, for random variables with equal means, Z†≤cxZ∘Z _cxZ holds if and only if FZ†(u)=∫0uΩZ†(t)t≥∫0uΩZ∘(t)t=FZ∘(u)for all u∈[0,1]. F_Z (u)= _0^u _Z (t)\,dt\;≥\; _0^u _Z (t)\,dt=F_Z (u) all u∈[0,1]. (18) Substituting (18) into (17), ℰsp(Z†)−ℰsp(Z∘)=(δ+1)δ∫01[FZ†(u)−FZ∘(u)](1−u)δ−1u≥ 0, ^sp(Z )-E^sp(Z )=(δ+1)δ _0^1 [F_Z (u)-F_Z (u) ](1-u)^δ-1\,du\;≥\;0, (19) since the bracket is nonnegative for all u∈[0,1]u∈[0,1] and (1−u)δ−1>0(1-u)^δ-1>0 on (0,1)(0,1). Hence ℰsp(Z∘)≤ℰsp(Z†)E^sp(Z ) ^sp(Z ). ∎ The operator is variance averse, but it never prefers a uniformly worse return distribution. The following proposition makes this precise. Proposition B.2 (Monotonicity under first-order stochastic dominance). Let Z1Z^1 and Z2Z^2 be categorical return distributions on the same atoms with Z1(z)≤Z2(z)C_Z^1(z) _Z^2(z) for every z, that is, Z1Z^1 first-order stochastically dominates Z2Z^2. Then ℰsp(Z1)≥ℰsp(Z2)E^sp(Z^1) ^sp(Z^2) for every δ≥0δ≥ 0. Proof. First-order stochastic dominance is equivalent to ΩZ1(u)≥ΩZ2(u) _Z^1(u)≥ _Z^2(u) for all u∈(0,1]u∈(0,1]. By Lemma B.1, ℰsp(Z1)−ℰsp(Z2)=(δ+1)∫01[ΩZ1(u)−ΩZ2(u)](1−u)δu≥0E^sp(Z^1)-E^sp(Z^2)=(δ+1) _0^1 [ _Z^1(u)- _Z^2(u) ](1-u)^δ\,du≥ 0, since both the bracket and the weight (1−u)δ(1-u)^δ are nonnegative. ∎ Consequently, an action whose returns are uniformly higher than those of another action is never rejected in favor of it, whatever its variance. Variance affects the ranking only among actions that are not ordered by first-order stochastic dominance, which is exactly the situation addressed by Theorem 3.2. By Proposition B.3, the implemented operator ℰE inherits this property up to the discretization gap Rεδ(Z)R\, _δ(Z). B.2 Design Considerations This subsection elaborates on two design considerations of the variance-averse expectation ℰ(Z)E(Z) defined in Equation (5). Both are quantified by the discretization analysis of Appendix B.3. Design Choice on the Boundary Atom. By construction (zI)=1C(z_I)=1, and thus the reweighting in (5) assigns zero weight to the largest support atom zIz_I. More generally, any atom above which the critic places no mass receives zero weight, whereas the spectral form (6) assigns such an atom the weight ¯(zi−1)δ+1 C(z_i-1)^δ+1. Proposition B.3 bounds the effect of this truncation (and of the normalization) on the value by Rεδ(Z)R\, _δ(Z), a quantity that is small whenever the probability mass is spread over several atoms. In the standard C51-style parameterization the support [Vmin,Vmax][V_ ,V_ ] is deliberately chosen to bracket the plausible return range, and the projection operator Φ(⋅) (·) distributes learned probability mass across multiple interior atoms, leaving the boundary atom zIz_I with negligible mass in practice. We measure the resulting gap on trained critics in Appendix B.3, where ℰE and ℰspE^sp select the same rejection-sampling candidate in 96.9%96.9\% of states. Numerical Stability. The denominator of (5), given by ∑j=1Ipj(1−(zj))δ=(1−εδ(Z))/(δ+1) _j=1^Ip_j(1-C(z_j))^δ=(1- _δ(Z))/(δ+1), is strictly positive whenever some atom with positive probability has positive survival probability. The categorical critic outputs are parameterized via softmax, ensuring strictly positive atom probabilities, and the C51 projection further spreads mass across multiple atoms. As a safeguard, we additionally include a small numerical stabilizer η=10−8η=10^-8 in the denominator. The denominator becomes small only when the mass concentrates on a single atom (maxipi≈1 _ip_i≈ 1), which is also the regime in which the bound of Theorem 3.2(i) is loose (Remarks B.6 and B.7). Throughout all reported experiments, we observed no numerical instability arising from this term. For critics that output nearly deterministic returns, the spectral form ℰspE^sp is a drop-in alternative that requires no normalization and returns ziz_i exactly for a point mass at ziz_i. B.3 Proof of Theorem 3.2(i): Discretization Error of the Implemented Operator We use the notation of Appendix B.1 and additionally write ai:=piSiδa_i:=p_iS_i^δ for the unnormalized weights of Definition 3.1, wi:=ai/∑jajw_i:=a_i/ _ja_j for its normalized weights (we take the stabilizer η=0η=0 here and treat η>0η>0 in Remark B.7), εδ:=1−(δ+1)∑iai _δ:=1-(δ+1) _ia_i as in (7), ρ:=maxipiρ:= _ip_i, and R:=zI−z1R:=z_I-z_1. In this notation ℰ(Z)=∑iwiziE(Z)= _iw_iz_i and ℰsp(Z)=∑iwi⋆ziE^sp(Z)= _iw_i z_i. Substituting x=1−ux=1-u in Lemma B.1, wi⋆=∫SiSi−1(δ+1)xδx, w_i = _S_i^S_i-1(δ+1)\,x^δ\,dx, (20) so that (δ+1)ai=(δ+1)piSiδ(δ+1)a_i=(δ+1)p_iS_i^δ is the right-endpoint rule (in u) applied to the same integral, and ℰE is this rule followed by normalization, which justifies the description in Theorem 3.2(i). Proposition B.3 (Discretization error). For every δ>0δ>0 the following hold. (a) (δ+1)ai≤wi⋆≤(δ+1)piSi−1δ(δ+1)a_i≤ w_i ≤(δ+1)p_iS_i-1^δ for all i. Consequently, εδ∈[0,1] _δ∈[0,1]. (b) ∑i=1I|wi⋆−wi|≤2εδ _i=1^I|w_i -w_i|≤ 2\, _δ. (c) |ℰsp(Z)−ℰ(Z)|≤Rεδ|E^sp(Z)-E(Z)|≤ R\, _δ. (d) εδ≤(δ+1)ρ _δ≤(δ+1)ρ for every δ>0δ>0, and εδ≤δ(δ+1)∑ipi2 _δ≤δ(δ+1) _ip_i^2 for δ≥1δ≥ 1. Proof. (a) The integrand in (20) is nondecreasing on [Si,Si−1][S_i,S_i-1], an interval of length pip_i, so it lies between (δ+1)Siδ(δ+1)S_i^δ and (δ+1)Si−1δ(δ+1)S_i-1^δ, and integrating gives (a). Summing the lower bound over i and using Lemma B.1 yields (δ+1)∑iai≤∑iwi⋆=1(δ+1) _ia_i≤ _iw_i =1, i.e., εδ∈[0,1] _δ∈[0,1]. (b) Write wi⋆−wi=(wi⋆−(δ+1)ai)+(δ+1)ai(1−11−εδ)w_i -w_i= (w_i -(δ+1)a_i )+(δ+1)a_i (1- 11- _δ ). The first term is nonnegative by (a) and sums to εδ _δ, whereas the second is nonpositive and sums to −εδ- _δ. Hence ∑i|wi⋆−wi|≤εδ+εδ _i|w_i -w_i|≤ _δ+ _δ. (c) Both weight vectors sum to one, so for any constant c, ℰsp(Z)−ℰ(Z)=∑i(wi⋆−wi)(zi−c)E^sp(Z)-E(Z)= _i(w_i -w_i)(z_i-c). Taking c=(z1+zI)/2c=(z_1+z_I)/2 gives |zi−c|≤R/2|z_i-c|≤ R/2, and (b) yields the claim. (d) By (a) and (20), εδ=∑i∫SiSi−1(δ+1)(xδ−Siδ)x _δ= _i _S_i^S_i-1(δ+1) (x^δ-S_i^δ )\,dx. For the first bound, monotonicity of x↦xδx x^δ gives xδ−Siδ≤Si−1δ−Siδx^δ-S_i^δ≤ S_i-1^δ-S_i^δ on [Si,Si−1][S_i,S_i-1], so εδ≤(δ+1)∑ipi(Si−1δ−Siδ) _δ≤(δ+1) _ip_i (S_i-1^δ-S_i^δ ) ≤(δ+1)ρ∑i(Si−1δ−Siδ) ≤(δ+1)\,ρ _i (S_i-1^δ-S_i^δ ) =(δ+1)ρ(S0δ−SIδ)=(δ+1)ρ. =(δ+1)\,ρ\, (S_0^δ-S_I^δ )=(δ+1)\,ρ. (21) For the second bound, if δ≥1δ≥ 1 the mean value theorem gives xδ−Siδ≤δxδ−1(x−Si)≤δpix^δ-S_i^δ≤δ x^δ-1(x-S_i)≤δ p_i on [Si,Si−1][S_i,S_i-1] (using x≤1x≤ 1), so εδ≤(δ+1)δ∑ipi2 _δ≤(δ+1)δ _ip_i^2. ∎ Proposition B.3(c)–(d) is the bound stated in Theorem 3.2(i). The remaining claims of part (i) are the two corollaries below. Corollary B.4 (Ordering transfer). Let [Z†]=[Z∘]E[Z ]=E[Z ] and Z†≤cxZ∘Z _cxZ . Then ℰ(Z∘)≤ℰ(Z†)+R[εδ(Z†)+εδ(Z∘)], (Z ) (Z )+R [ _δ(Z )+ _δ(Z ) ], (22) which is (8). In particular, if the convex-order margin satisfies ℰsp(Z†)−ℰsp(Z∘)>R[εδ(Z†)+εδ(Z∘)]E^sp(Z )-E^sp(Z )>R[ _δ(Z )+ _δ(Z )], then ℰ(Z†)>ℰ(Z∘)E(Z )>E(Z ). That is, the implemented operator induces the same ordering as the spectral form whenever the margin exceeds the certified error budget. Proof. By Theorem 3.2(i), ℰsp(Z∘)≤ℰsp(Z†)E^sp(Z ) ^sp(Z ). Applying Proposition B.3(c) to both distributions, ℰ(Z∘)≤ℰsp(Z∘)+Rεδ(Z∘)≤ℰsp(Z†)+Rεδ(Z∘)≤ℰ(Z†)+R[εδ(Z†)+εδ(Z∘)]E(Z ) ^sp(Z )+R _δ(Z ) ^sp(Z )+R _δ(Z ) (Z )+R[ _δ(Z )+ _δ(Z )]. The second claim follows from the triangle inequality in the same way. ∎ Corollary B.5 (Vanishing gap under the C51 projection). Let Y be a target return supported on [Vmin,Vmax][V_ ,V_ ] whose CDF is L-Lipschitz, and let Z=Φ(Y)Z= (Y) be its projection (29) onto I atoms with spacing Δ=(Vmax−Vmin)/(I−1) =(V_ -V_ )/(I-1). Then ρ≤2LΔ=O(1/I)ρ≤ 2L =O(1/I) and |ℰsp(Z)−ℰ(Z)|≤Rεδ(Z)≤2(δ+1)LRΔ=O(1/I). |E^sp(Z)-E(Z)|≤ R\, _δ(Z)≤ 2(δ+1)LR\, =O(1/I). (23) Proof. The projection moves the mass at y only to the two atoms adjacent to y, so atom ziz_i receives mass only from y:|y−zi|<Δ\y:|y-z_i|< \ and pi≤FY(zi+Δ)−FY(zi−Δ)≤2LΔp_i≤ F_Y(z_i+ )-F_Y(z_i- )≤ 2L (no clipping occurs because Y is supported on [Vmin,Vmax][V_ ,V_ ]). Apply Proposition B.3(c)–(d). ∎ Remark B.6 (When the bound is loose). The certificate Rεδ(Z)R\, _δ(Z) is small whenever the critic spreads its mass over several atoms (ρ≪1ρ 1) and is vacuous for a nearly deterministic return (ρ≈1ρ≈ 1): for a point mass at ziz_i the spectral form returns exactly ziz_i (wi⋆=1w_i =1), whereas Definition 3.1 assigns that atom the weight piSiδ=0p_iS_i^δ=0, so that ℰE is determined by the (vanishingly small) mass below ziz_i. The Lipschitz assumption of Corollary B.5 excludes this regime. The guarantee therefore covers return distributions that are genuinely spread—the regime produced by n-step bootstrapped targets under the C51 projection with I=101I=101 atoms, as verified below—and does not cover deterministic returns. Remark B.7 (Numerical stabilizer). With the stabilizer η>0η>0 of Appendix B.2, the implemented weights sum to ∑jaj/(∑jaj+η)<1 _ja_j/( _ja_j+η)<1, which adds at most maxi|zi|⋅η/(∑jaj+η) _i|z_i|·η/( _ja_j+η) to the error in Proposition B.3(c). This term is negligible unless ∑jaj=(1−εδ)/(δ+1)≲η _ja_j=(1- _δ)/(δ+1) η, i.e., unless εδ≈1 _δ≈ 1, which is exactly the regime of Remark B.6. In that regime ℰspE^sp, which needs no normalization, is the preferable drop-in replacement. Empirical size of the gap on trained critics. To confirm that the regime of Corollary B.5, rather than that of Remark B.6, is the one encountered in practice, we measured the gap on critics trained with the default configuration on antmaze-teleport-navigate, using 256 randomly selected dataset states and M=32M=32 candidate actions per state. For every candidate action drawn from the flow actor we evaluated ℰE and ℰspE^sp. Table 6 reports the results: the realized gap is small (median 0.37%0.37\% of the value range R), both operators select the same candidate in 96.9%96.9\% of states, and their candidate rankings are strongly correlated (Kendall’s τ=0.99τ=0.99), indicating that the discretization gap is negligible at the atom resolution used throughout the paper. Table 6: Discretization gap between the implemented operator ℰE and its spectral form ℰspE^sp on trained critics. Measurement Value Median realized gap |ℰ−ℰsp||E-E^sp| (fraction of the value range R) 0.37%0.37\% States in which ℰE and ℰspE^sp select the same candidate 96.9%96.9\% Kendall’s τ between the two candidate rankings 0.990.99 Appendix C Projection Operator This appendix details the categorical projection operator Φ(⋅) (·) used in (2) and (12), following the standard C51-style projection introduced in [8]. Fixed support. We fix I atoms zii=1I\z_i\_i=1^I uniformly spanning an interval [Vmin,Vmax][V_ ,V_ ]: zi=Vmin+(i−1)Δ,Δ=Vmax−VminI−1.z_i=V_ +(i-1) , = V_ -V_ I-1. (24) The categorical critic outputs a probability vector p(s,a)∈ΔIp(s,a)∈ ^I over these atoms. Target distribution and clamping. Let the (unprojected) target distribution be a random variable Y obtained by shifting the next-state return distribution by the n-step return: Y=Gt(n)+γnZψ(st+n,at+n∗).Y\;=\;G_t^(n)+γ^nZ_ψ(s_t+n,a_t+n^*). (25) Because Y may fall outside [Vmin,Vmax][V_ ,V_ ], we clamp support values using y¯=clip(y,Vmin,Vmax). y=clip(y,V_ ,V_ ). (26) Projection Φ : mass transfer to neighboring atoms. Suppose Y is represented categorically as probabilities qjj=1I\q_j\_j=1^I on some values yjj=1I\y_j\_j=1^I (e.g., yj=Gt(n)+γnzjy_j=G_t^(n)+γ^nz_j when the next distribution is supported on zj\z_j\). The projection Φ maps this distribution onto zi\z_i\ by distributing each mass qjq_j to the two nearest atoms: bj b_j =y¯j−VminΔ+1, = y_j-V_ +1, (27) ℓj _j =⌊bj⌋,uj=⌈bj⌉, = b_j , u_j= b_j , (28) and define the projected probabilities mim_i by mi=∑j=1Iqj[uj=i(bj−ℓj)+ℓj=i(uj−bj)],m_i\;=\; _j=1^Iq_j [1\u_j=i\(b_j- _j)+1\ _j=i\(u_j-b_j) ], (29) with the convention that when ℓj=uj _j=u_j, all mass is assigned to that atom. Then Φ(Y) (Y) is the categorical distribution supported on zi\z_i\ with probabilities mii=1I\m_i\_i=1^I. Appendix D Benchmarks and Task Descriptions This section summarizes the benchmark environments used in our experiments, with emphasis on properties that are directly relevant to the problem studied in this paper: sparse rewards, long-horizon credit assignment, and dataset-induced return variance. Rather than providing a comprehensive description of each benchmark, we highlight how different task and dataset variants give rise to distinct variance regimes for n-step return learning in offline RL. D.1 D4RL antMaze Figure 6: D4RL We evaluate on the AntMaze tasks from D4RL, a canonical benchmark for offline RL under sparse rewards and long planning horizons. AntMaze has become a standard testbed for studying long-horizon credit assignment and robustness to dataset-induced distributional challenges. In these environments, rewards are extremely sparse and are typically provided only when the agent reaches the goal region, making effective long-horizon value propagation essential. Horizon and variance characteristics. AntMaze provides a controlled setting in which both the effective planning horizon and dataset-induced return variance can be systematically varied. • Maze scales: umaze, medium, and large progressively increase the episode horizon and the depth of credit assignment. • Dataset variants: play datasets are collected using relatively stationary behavior policies, resulting in lower return variance, whereas diverse datasets are generated from heterogeneous behavior policies and induce substantially higher return variance. As the maze scale and dataset heterogeneity increase, n-step return targets aggregate rewards over increasingly diverse trajectories, thereby amplifying return variance. Consequently, AntMaze serves as a particularly suitable benchmark for evaluating the stability of long-horizon and multi-step learning methods in offline RL. D.2 OGBench Figure 7: OGBench OGBench is a benchmark suite designed to evaluate offline RL in environments characterized by long horizons, sparse or deceptive rewards, and limited dataset coverage. In contrast to D4RL AntMaze, OGBench offers a broader collection of tasks with diverse dynamics, action spaces, and dataset construction processes. Relevance to return variance. A distinguishing feature of OGBench is that different dataset variants within the same environment exhibit substantial differences in return dispersion. This property enables controlled comparisons between low-variance and high-variance offline datasets under identical task dynamics. Task families used. We focus on state-based OGBench tasks that are particularly sensitive to long-horizon value estimation and return variance: • AntMaze: antmaze-large-navigate, antmaze-teleport-navigate, antmaze-large-explore, and antmaze-giant-navigate, which require long-horizon planning and are highly sensitive to dataset coverage. • HumanoidMaze: humanoidmaze-large-navigate, humanoidmaze-giant-navigate, which combine extremely long horizons with high-dimensional action spaces. • Scene and Puzzle: scene-play, puzzle-3x3-play, and puzzle-3x3-noisy, which exhibit strong multi-modality and heightened sensitivity to out-of-distribution actions. Low-variance vs. high-variance datasets. Following the main text, we categorize dataset variants according to their induced return variance. • navigate and play datasets are treated as relatively low-variance datasets, collected using more stationary behavior policies. • explore and noisy datasets are treated as high-variance datasets, where non-stationary or noisy behavior policies induce large dispersion in return targets. These high-variance datasets pose a significant challenge for naïve n-step return methods, as increasing the return horizon directly amplifies target variance. Consequently, OGBench serves as a particularly suitable testbed for evaluating variance-aware learning strategies in offline RL. Appendix E Implementation Details of VAN-Flow We provide the hyperparameter details of the proposed method in Table 7. Table 7: Hyperparameters for VAN-Flow. Hyperparameter Value Optimization Learning rate (critic) αZ _Z 3×10−43× 10^-4 Learning rate (actor) απ _π 3×10−43× 10^-4 Optimizer Adam [54] Gradient steps 10610^6 (OGBench), 5×1055× 10^5 (D4RL) Minibatch size 256 Network Architecture MLP dimensions [512, 512, 512, 512] Nonlinearity GELU [55] Target network smoothing coefficient ρ 0.005 Discounting and Returns Discount factor γ 0.995 (default) 0.999 (antmaze-giant, antmaze-teleport, humanoidmaze) n-step return n 4 (default) 2 (antmaze-teleport) 3 (antmaze-large-explore, scene-play) 8 (antmaze-giant) Variance-Averse Expectation Aversion intensity δ 2 (default) 7 (antmaze-teleport) Distributional Critic VminV_ -200 (default), -1000 (antmaze-giant, antmaze-teleport, humanoidmaze) VmaxV_ 0 Critic ensemble size 2 Critic aggregation method mean Categorical atoms 101 Flow-based Actor Flow steps K 10 Number of candidate actions M 8 Flow-matching regression term weight λ 3 (default) 10 (antmaze-giant, humanoidmaze) 300 (antmaze-teleport) Reward Preprocessing. We rescale all rewards to −1,0\-1,0\ prior to training, so that the discounted return remains within the categorical critic’s support [Vmin,0][V_ ,0]. The lower bound is set as Vmin≈−1/(1−γ)V_ ≈-1/(1-γ), which corresponds to the worst-case return when the goal is never reached. This transformation adds a state-independent constant to all Q-values and therefore does not affect the optimal policy. Appendix F Component-wise Ablation Study This appendix presents a component-wise ablation that isolates the individual and joint contributions of the three core components of VAN-Flow: the variance-averse expectation (VE), the categorical distributional critic (Dist), and the flow-based actor (Flow). Setup. All variants are trained with n-step returns under identical hyperparameters, differing only in which components are enabled. We evaluate across two action-dimension regimes: low-dimensional locomotion (antmaze-teleport and antmaze-giant, both 8-dim) and a high-dimensional task (humanoidmaze-large, 21-dim). Table 8: Component-wise ablation across action-dimension regimes. All variants use n-step returns. Scores denote success rate (%), averaged over five random seeds. Variant Dist VE Flow antmaze-teleport (8d) antmaze-giant (8d) humanoidmaze-large (21d) VAN-Flow (Ours) ✓ ✓ ✓ ±71\!±\!7 ±58\!±\!4 ±87\!±\!6 w/o VE ✓ ✗ ✓ ±738\!±\!7 ±636\!±\!6 ±477\!±\!4 w/o Flow ✓ ✓ ✗ ±742\!±\!7 ±1338\!±\!13 ±00\!±\!0 w/o VE & Flow ✓ ✗ ✗ ±936\!±\!9 ±532\!±\!5 ±00\!±\!0 Discussion. On low-dimensional tasks (antmaze-teleport, antmaze-giant), removing VE from VAN-Flow causes a substantial performance drop (71→3871→ 38 on teleport; 58→3658→ 36 on giant), while removing Flow leads to a smaller decrease (71→4271→ 42; 58→3858→ 38). This indicates that, under n-step returns, actor expressiveness alone provides little benefit unless return variance is properly controlled. VE downweights unreliable high-variance return targets, while Flow provides the expressiveness required to exploit the resulting value estimates. The high-dimensional task humanoidmaze-large (21d) makes this synergy even clearer. Removing Flow causes complete failure (87→087→ 0), consistent with Figure 3, since a Gaussian actor cannot represent the multi-modal action distribution required by the task. Removing VE instead is far less catastrophic (87→7787→ 77), but still leaves a meaningful gap. These results jointly demonstrate that VE and Flow play complementary roles: Flow supplies the action expressiveness, while VE ensures the value estimates guiding the actor are reliable. How much capacity the actor and the critic each need is studied separately in [56]. Appendix G Empirical Comparison of Flow Matching and Diffusion Policies This appendix provides the empirical justification for adopting flow matching as the policy backbone of VAN-Flow, rather than the more widely used diffusion-based parameterization (e.g., DDPM [1]). Motivation. Diffusion-based actors [1, 2] typically require many denoising steps to produce high-quality actions, substantially increasing both training and inference cost. Flow matching instead parameterizes the policy via a deterministic ODE that enables high-quality samples with far fewer integration steps [3], which is particularly attractive in actor–critic offline RL where the actor is repeatedly evaluated during both training and inference. Experimental Setup. To validate this design choice within our framework, we instantiate a diffusion-based variant of our method, denoted VAN-DDPM, by replacing the flow-matching actor in VAN-Flow with a DDPM-style denoising actor while keeping all other components (categorical critic, variance-averse expectation, rejection sampling, n-step return) unchanged. We evaluate both variants on humanoidmaze-large-navigate over five random seeds, varying the number of denoising/flow steps. We report the success rate as performance and the cumulative inference time per action generation in milliseconds. Table 9: Comparison between flow matching (VAN-Flow) and diffusion (VAN-DDPM) policies on humanoidmaze-large-navigate. Performance denotes success rate (%); Time denotes cumulative inference time per action (ms). Bold entries indicate the first step count achieving >>90% success rate. VAN-DDPM VAN-Flow Steps Perf. Time (ms) Perf. Time (ms) 1 0±00 0.68887pt $± 0$ 2.0 0±00 0.68887pt $± 0$ 1.6 3 0±00 0.68887pt $± 0$ 3.8 ±396 0.68887pt $± 3$ 3.3 5 0±00 0.68887pt $± 0$ 5.7 96±296 0.68887pt $± 2$ 5.0 10 57±1757 0.68887pt $± 17$ 8.4 87±687 0.68887pt $± 6$ 7.2 20 ±494 0.68887pt $± 4$ 11.9 76±676 0.68887pt $± 6$ 10.2 Discussion. As shown in Table 9, VAN-Flow achieves over 90% success rate with only 33 flow steps (3.3ms), whereas VAN-DDPM requires 2020 denoising steps (11.9ms) to reach a comparable level, corresponding to roughly a 3.6×3.6× reduction in inference cost. We further observe that VAN-DDPM exhibits a much steeper sensitivity to the number of denoising steps: it produces near-zero success rates for ≤5≤ 5 steps, indicating that diffusion-based actors require sufficiently many denoising iterations to generate in-distribution actions. In contrast, flow matching attains strong performance even at very low step counts. Overall, these results justify our use of flow matching as the policy backbone, balancing high sample quality and low computational cost. Appendix H Baselines Details H.1 Baselines Description We briefly summarize the baselines used in our comparisons. All methods are run with publicly available implementations when possible, or re-implemented following the original papers, using the same dataset splits and evaluation protocols as VAN-Flow. Pure offline RL (1-step) • IQL [35]: An offline actor–critic method that employs expectile regression for value learning and advantage-weighted updates for policy improvement. • ReBRAC [36]: A behavior-regularized actor–critic method designed to improve robustness against out-of-distribution (OOD) actions. • HIQL [37]: A hierarchical extension of IQL tailored for long-horizon decision-making problems. Flow-based policies • FQL [3]: Flow Q-Learning, an offline actor–critic method that employs a flow-matching actor guided by a learned critic. • BFN / QC [4]: Flow-based policy families with value-guided sampling. We adopt the hyperparameter settings recommended by the authors. • FQL-n, BFN-n: Variants of the original algorithms that incorporate an n-step temporal-difference (TD) target. n-step / horizon reduction methods • LEQ [17]: A long-horizon stabilization method that combines n-step λ-returns with expectile regression for improved value estimation stability. • QC [4]: An action-chunking framework based on flow-based policy families with value-guided sampling. We adopt the hyperparameter settings recommended by the authors. Distributional critics • PA-RL [41]: A policy-agnostic actor–critic algorithm that leverages a distributional critic to optimize actions, which are then distilled into the policy via supervised learning. • D4PG [42]: A distributional actor–critic baseline that combines multi-step returns with a deterministic policy gradient. We adapt it to the offline setting when necessary. H.2 Sources of Reported Results In the D4RL benchmark, we use a combination of previously reported results and results obtained from our own experiments. Specifically, for PA-RL, FQL, and LEQ, we report the performance values as presented in the original papers that proposed each algorithm. For IQL and ReBRAC, we use the results reported in the benchmark paper CORL [32]. In contrast, the performance of QC and D4PG is obtained from experiments conducted by us. In particular, since D4PG is an online reinforcement learning algorithm, we implement it with an additional behavior cloning (BC) term for our experiments. For the OGBench benchmark, we use the reported performance of HIQL from the original OGBench paper as the benchmark baseline [34]. For FQL, we use the results reported in the paper that proposed FQL as well as those reported in Reinforcement Learning with Action Chunking [4], which also introduces QC. Additionally, from the same paper, we use the reported performance of other baselines on the scene-play and puzzle-3x3 tasks. For all remaining tasks in OGBench, we conduct and report our own experimental results. Appendix I Extensive Results of Performance Comparisons I.1 Learning Curve of VAN-Flow In this subsection, we present the learning curves of VAN-Flow across representative benchmarks. Each curve shows the evaluation success rate as a function of training steps, averaged over five random seeds, and the shaded band around each curve indicates one standard deviation. Each run consists of an offline training phase followed by online fine-tuning. On OGBench, the agent is trained offline for the first 11M steps and then fine-tuned online for the remaining 11M steps. On D4RL, the switch occurs at 0.50.5M steps. The vertical dashed line marks this switch, and the darker and lighter backgrounds indicate the offline and online phases, respectively. The corresponding offline-to-online results are summarized in Appendix J. The learning curves for antmaze and humanoidmaze from OGBench are shown in Figure 8 and Figure 9, respectively. Additionally, the learning curves for puzzle and scene are presented in Figure 10 and Figure 11. Figure 12 shows the learning curves for the antmaze tasks in D4RL. Figure 8: Learning curve on the antmaze task in OGBench Figure 9: Learning curve on the humanoidmaze task in OGBench Figure 10: Learning curve on the puzzle task in OGBench Figure 11: Learning curve on the scene task in OGBench Figure 12: Learning curve on the antmaze task in D4RL I.2 Full Performance Table Table 10 provides the full performance table for all evaluated tasks and all compared baselines. We report normalized scores under the corresponding benchmark protocol. Table 10: Task-wise performance comparison in OGBench (singletask setting). Env Task Gaussian-based Flow-based Proposed IQL ReBRAC ReBRAC-n HIQL FQL BFN FQL-n BFN-n QC VAN-Flow antmaze-large-navigate task1 48±9± 9 91±10± 10 64.0±25.8± 25.8 93.0±3.0± 3.0 80±8± 8 93.0±1.4± 1.4 94.2±2.8± 2.8 83.1±1.4± 1.4 49.2±9.9± 9.9 94.6±4.2± 4.2 task2 42±6± 6 88±4± 4 78.3±5.9± 5.9 78.0±9.0± 9.0 57±10± 10 90.2±2.8± 2.8 80.0±3.4± 3.4 88.0±11.3± 11.3 43.4±2.4± 2.4 89.4±7.0± 7.0 task3 72±7± 7 51±18± 18 92.8±2.3± 2.3 96.0±2.0± 2.0 93±3± 3 89.1±7.0± 7.0 97.5±1.4± 1.4 74.5±5.6± 5.6 71.8±2.8± 2.8 98.0±2.0± 2.0 task4 51±9± 9 84±7± 7 90.0±2.8± 2.8 94.0±2.0± 2.0 80±4± 4 9.0±1.4± 1.4 89.1±7.0± 7.0 73.1±1.4± 1.4 22.0±2.8± 2.8 95.4±3.1± 3.1 task5 54±22± 22 90±2± 2 90.8±2.7± 2.7 94.0±3.0± 3.0 83±4± 4 72.0±5.6± 5.6 94.7±5.6± 5.6 85.1±1.4± 1.4 61.6±1.4± 1.4 98.0±2.0± 2.0 antmaze-teleport-navigate task1 17.3±1.5± 1.5 0.0±0.0± 0.0 14.0±12.6± 12.6 37.0±5.0± 5.0 39.0±9.8± 9.8 48.0±2.8± 2.8 5.0±1.4± 1.4 23.2±1.4± 1.4 37.0±1.2± 1.2 70.8±7.5± 7.5 task2 27.0±4.2± 4.2 0.0±0.0± 0.0 38.4±21.2± 21.2 66.0±8.0± 8.0 0.0±0.0± 0.0 40.2±5.6± 5.6 0.0±0.0± 0.0 74.4±14.1± 14.1 56.2±8.4± 8.4 91.2±3.0± 3.0 task3 32.0±5.6± 5.6 27.0±1.4± 1.4 52.4±8.6± 8.6 37.0±5.0± 5.0 47.2±7.0± 7.0 29.6±7.0± 7.0 77.1±4.2± 4.2 41.0±9.9± 9.9 37.0±3.5± 3.5 54.0±14.1± 14.1 task4 26.0±2.8± 2.8 10.5±5.6± 5.6 44.0±4.0± 4.0 30.0±2.0± 2.0 45.2±9.8± 9.8 37.5±3.5± 3.5 0.0±0.0± 0.0 36.2±2.8± 2.8 36.6±7.0± 7.0 48.0±5.5± 5.5 task5 12.0±5.6± 5.6 0±0± 0 24.8±14.6± 14.6 41.0±8.0± 8.0 0±0± 0 44.0±2.8± 2.8 27.5±1.4± 1.4 29.2±1.4± 1.4 31.0±4.2± 4.2 22.0±8.0± 8.0 scene-play-v0 task1 1.0 4.0 75.3±32.2± 32.2 40.0±4.0± 4.0 69.0 99.0 22.0 64.0 100 98.8±1.8± 1.8 task2 0.0 51.0 76.7±23.0± 23.0 40.4±9.0± 9.0 51.0 99.0 4.0 15.0 98 92.4±4.5± 4.5 task3 0.0 0.0 45.3±22.3± 22.3 36.0±5.0± 5.0 68.0 93.0 1.0 34.0 90 87.6±7.9± 7.9 task4 0.0 0.0 4.0±2.5± 2.5 55.0±5.0± 5.0 75.0 86.0 50.0 91.0 93.0 95.2±4.6± 4.6 task5 0.0 11.0 0.0±0.0± 0.0 20.0±5.0± 5.0 25.0 38.0 14.0 71.0 43.0 88.8±4.6± 4.6 puzzle-3x3-play task1 0.0 96.0 48.8±5.4± 5.4 29.0±4.0± 4.0 99 98 100.0 88 100.0 100.0±0.0± 0.0 task2 0.0 70.0 2.8±1.8± 1.8 11.0±3.0± 3.0 99 98 100.0 72 100.0 100.0±0.0± 0.0 task3 0.0 27.0 0.0±0.0± 0.0 7.0±3.0± 3.0 100.0 95 100.0 56 100.0 100.0±0.0± 0.0 task4 0.0 29.0 0.4±0.9± 0.9 5.0±1.0± 1.0 100.0 98 100.0 37 100.0 100.0±0.0± 0.0 task5 0.0 54.0 0.0±0.0± 0.0 8.0±3.0± 3.0 100.0 97 98 47 100.0 100.0±0.0± 0.0 Long-horizon tasks antmaze-giant-navigate task1 0±0± 0 27±22± 22 0.0±0.0± 0.0 47.0±10.0± 10.0 4±5± 5 0.0±0.0± 0.0 11.2±1.6± 1.6 0.0±0.0± 0.0 0.0±0.0± 0.0 58.0±4.0± 4.0 task2 1±1± 1 16±17± 17 81.6±3.0± 3.0 74.0±5.0± 5.0 9±7± 7 0.0±0.0± 0.0 93.6±7.0± 7.0 16.1±8.4± 8.4 4.0±2.8± 2.8 86.6±7.0± 7.0 task3 0±0± 0 34±22± 22 0.0±0.0± 0.0 55.0±7.0± 7.0 0±1± 1 0.0±0.0± 0.0 44.0±8.4± 8.4 0.0±0.0± 0.0 0.0±0.0± 0.0 56.4±14.2± 14.2 task4 0±0± 0 5±12± 12 0.0±0.0± 0.0 69.0±5.0± 5.0 14±23± 23 0.0±0.0± 0.0 57.5±12.7± 12.7 0.0±0.0± 0.0 0.0±0.0± 0.0 87.2±7.5± 7.5 task5 19±7± 7 49±22± 22 0.0±0.0± 0.0 82.0±4.0± 4.0 16±28± 28 55.2±4.7± 4.7 63.6±1.4± 1.4 7.2±4.2± 4.2 6.2±2.8± 2.8 56.8±4.1± 4.1 humanoidmaze-large-navigate task1 3±1± 1 2±1± 1 0.8±1.1± 1.1 67.0±4.0± 4.0 7±6± 6 44.2±5.6± 5.6 25.3±1.4± 1.4 45.2±2.4± 2.4 13.0±1.4± 1.4 87.4±5.8± 5.8 task2 0±0± 0 0±0± 0 0.0±0.0± 0.0 2.0±3.0± 3.0 0±0± 0 7.2±1.4± 1.4 0.0±0.0± 0.0 11.5±1.4± 1.4 0.0±0.0± 0.0 55.4±8.0± 8.0 task3 7±3± 3 2±1± 1 18.5±8.5± 8.5 88.0±3.0± 3.0 7±6± 6 26.3±2.8± 2.8 41.0±9.8± 9.8 65.2±9.8± 9.8 30.0±2.8± 2.8 96.0±2.0± 2.0 task4 1±0± 0 1±1± 1 0.0±0.0± 0.0 42.0±11.0± 11.0 2±3± 3 3.1±1.4± 1.4 24.5±2.8± 2.8 45.1±2.8± 2.8 15.0±1.4± 1.4 84.6±1.1± 1.1 task5 1±1± 1 2±2± 2 0.0±0.0± 0.0 47.0±10.0± 10.0 1±3± 3 52.3±11.1± 11.1 10.0±2.8± 2.8 53.5±9.9± 9.9 23.2±2.8± 2.8 82.6±3.0± 3.0 humanoidmaze-giant-navigate task1 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 13.0±7.0± 7.0 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 54.6±2.8± 2.8 10.2±2.7± 2.7 85.4±5.0± 5.0 task2 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 35.0±11.0± 11.0 1.2±1.4± 1.4 0.0±0.0± 0.0 7.6±1.4± 1.4 87.1±5.6± 5.6 3.7±1.5± 1.5 97.4±1.1± 1.1 task3 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 11.0±4.0± 4.0 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 70.1±2.8± 2.8 3.0±1.2± 1.2 88.0±2.0± 2.0 task4 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 2.0±2.0± 2.0 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 73.5±7.0± 7.0 0.0±0.0± 0.0 96.6±3.0± 3.0 task5 3.6±3.5± 3.5 0.0±0.0± 0.0 0.0±0.0± 0.0 12.0±4.0± 4.0 0.0±0.0± 0.0 0.0±0.0± 0.0 9.0±7.0± 7.0 86.1±2.8± 2.8 44.2±8.4± 8.4 97.2±1.1± 1.1 Noisy tasks antmaze-large-explore task1 0.0±0.0± 0.0 90.0±5.6± 5.6 24.8±10.4± 10.4 1.0±3.0± 3.0 2.0±1.4± 1.4 17.0±24.0± 24.0 0.0±0.0± 0.0 5.2±1.4± 1.4 79.2±4.2± 4.2 93.6±3.0± 3.0 task2 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 2.0±2.8± 2.8 0.0±0.0± 0.0 0.0±0.0± 0.0 7.0±4.2± 4.2 52.0±3.4± 3.4 task3 67.5±1.4± 1.4 93.4±1.4± 1.4 95.6±2.2± 2.2 18.0±24.0± 24.0 87.6±4.2± 4.2 89.3±1.4± 1.4 4.5±3.5± 3.5 77.2±4.2± 4.2 90.2±1.4± 1.4 96.0±4.0± 4.0 task4 2.0±1.0± 1.0 0.5±0.7± 0.7 46.8±13.2± 13.2 0.0±0.0± 0.0 10.2±2.8± 2.8 0.0±0.0± 0.0 0.0±0.0± 0.0 0.0±0.0± 0.0 64.2±2.8± 2.8 86.6±3.7± 3.7 task5 6.0±2.8± 2.8 58.5±14.5± 14.5 1.6±0.9± 0.9 0.0±0.0± 0.0 0.0±0.0± 0.0 1.0±1.4± 1.4 1.0±1.4± 1.4 1.0±1.4± 1.4 48.6±7.0± 7.0 92.2±3.0± 3.0 puzzle-3x3-noisy task1 34.5±1.5± 1.5 9.0±1.4± 1.4 18.8±39.8± 39.8 67.0±10.0± 10.0 100.0±0.0± 0.0 100.0±0.0± 0.0 44.1±2.2± 2.2 100.0±0.0± 0.0 100.0±0.0± 0.0 96.6±3.0± 3.0 task2 53.4±3.5± 3.5 2.3±2.8± 2.8 19.6±41.6± 41.6 54.0±11.0± 11.0 100.0±0.0± 0.0 100.0±0.0± 0.0 6.0±8.4± 8.4 100.0±0.0± 0.0 100.0±0.0± 0.0 98.6±4.1± 4.1 task3 19.0±1.4± 1.4 0.0±0.0± 0.0 0.0±0.0± 0.0 43.0±12.0± 12.0 100.0±0.0± 0.0 100.0±0.0± 0.0 84.0±2.8± 2.8 99.0±1.4± 1.4 100.0±0.0± 0.0 95.3±3.0± 3.0 task4 20.3±0.7± 0.7 1.5±1.4± 1.4 0.0±0.0± 0.0 41.0±13.0± 13.0 100.0±0.0± 0.0 100.0±0.0± 0.0 18.0±8.4± 8.4 93.0±4.2± 4.2 100.0±0.0± 0.0 96.6±3.0± 3.0 task5 32.1±13.8± 13.8 0.0±0.0± 0.0 0.0±0.0± 0.0 47.0±15.0± 15.0 100.0±0.0± 0.0 100.0±0.0± 0.0 13.0±4.2± 4.2 100.0±0.0± 0.0 100.0±0.0± 0.0 97.4±1.8± 1.8 I.3 Performance Comparison in Denser Reward Environments We additionally evaluate VAN-Flow on the D4RL Adroit manipulation suite to compare against a wide range of generative policy-based offline RL methods, including diffusion-based approaches (IDQL [50], SRPO [57], CAC [58]) and flow-based policies (FAWAC, FBRAC, FQL, IFQL). For flow-based baselines, we consider flow-policy instantiations of existing offline RL algorithms (AWAC [59], ReBRAC [36], IDQL) [50] as implemented in Flow Q-Learning [3]. As shown in Table 11, VAN-Flow achieves competitive or superior performance across multiple Adroit tasks, particularly on cloned and expert datasets. Notably, VAN-Flow outperforms flow-based baselines such as FQL and IFQL on most tasks, indicating that the observed gains are not merely due to the use of a generative policy, but arise from variance-aware n-step value learning. Furthermore, VAN-Flow remains competitive with strong diffusion-based methods, demonstrating that variance-averse n-step learning integrates naturally with flow-based policies without sacrificing the advantages typically associated with diffusion models. Table 11: Performance comparison in D4RL Adroit Task Gaussian-based Diffusion Policies Flow-based Proposed BC IQL ReBRAC IDQL SRPO CAC FAWAC FBRAC IFQL FQL Ours pen-human-v1 71 78 103 76 69 64 67 77 71 53 82.3±3.5± 3.5 pen-cloned-v1 52 83 103 64 61 56 62 67 80 74 107.5±14.9± 14.9 pen-expert-v1 110 128 152 140 134 103 118 119 139 142 146.6±5.5± 5.5 door-human-v1 2 3 0 6 3 5 2 4 7 0 2.7±1.3± 1.3 door-cloned-v1 0 3 0 0 0 1 0 0 2 2 3.0±2.8± 2.8 door-expert-v1 105 107 106 105 105 98 103 104 104 104 104.0±0.6± 0.6 hammer-human-v1 3 2 0 2 1 2 2 2 3 1 3.5±0.7± 0.7 hammer-cloned-v1 1 2 5 2 2 1 1 2 2 11 3.0±2.1± 2.1 hammer-expert-v1 127 129 134 125 127 92 118 119 117 125 131±1.4± 1.4 relocate-human-v1 0 0 0 0 0 0 0 0 0 0 0.2±0.0± 0.0 relocate-cloned-v1 0 0 2 0 0 0 0 1 0 0 0.3±0.0± 0.0 relocate-expert-v1 108 106 108 107 106 93 105 105 104 107 108±0.8± 0.8 I.4 D4RL Locomotion Tasks The main experiments focus on sparse-reward, long-horizon tasks because that is where variance amplification under n-step returns is most severe. To check that the method is not tied to goal-conditioned tasks, we also ran VAN-Flow on the dense-reward D4RL locomotion suite with the medium and medium-replay datasets. Neither dataset contains expert demonstrations. We keep the default δ=2δ=2 and use γ=0.99γ=0.99, I=101I=101, and the minimum over the two critics, with n=1n=1 for HalfCheetah and n=3n=3 for Hopper and Walker2d, and λ=0.3λ=0.3, 33, and 1010 for the three environments respectively. Results are averaged over five seeds, and baseline numbers are taken from the corresponding papers. Table 12: Normalized returns on D4RL locomotion tasks. Best results are in bold, and results within one standard deviation of the best are also bolded. Task IQL ReBRAC Diffusion-QL Diffuser VAN-Flow halfcheetah-medium 47.4 65.6±1.065.6 0.68887pt $± 1.0$ 51.1±0.5± 0.5 42.8±0.3± 0.3 63.9±2.1± 2.1 halfcheetah-medium-replay 44.2 51.0±0.8± 0.8 47.8±0.3± 0.3 37.7±0.5± 0.5 51.4±0.351.4 0.68887pt $± 0.3$ hopper-medium 66.3 102.0±1.0102.0 0.68887pt $± 1.0$ 90.5±4.6± 4.6 74.3±1.4± 1.4 95.3±2.5± 2.5 hopper-medium-replay 94.7 98.1±5.3± 5.3 101.3±0.6101.3 0.68887pt $± 0.6$ 93.6±0.4± 0.4 100.3±1.2± 1.2 walker2d-medium 78.3 82.5±3.6± 3.6 87.0±0.9± 0.9 79.6±0.6± 0.6 88.2±1.888.2 0.68887pt $± 1.8$ walker2d-medium-replay 73.9 77.3±7.9± 7.9 95.5±1.595.5 0.68887pt $± 1.5$ 70.6±1.6± 1.6 81.0±4.1± 4.1 Mean 67.5 79.4 78.9 66.4 80.080.0 VAN-Flow is competitive with the strongest baselines here (mean 80.0 against 79.4 for ReBRAC and 78.9 for Diffusion-QL). It matches or outperforms ReBRAC on all three medium-replay datasets, which mix behavior of very different quality, while the gap is small on the more homogeneous medium datasets. This is consistent with what the method is designed for. When several behavior modes with different returns coexist, return variance separates reliable from unreliable actions, and when the dataset comes from a single behavior policy, variance adds little beyond the expected return and the operator behaves close to neutral. Appendix J Off-to-Online Fine-Tuning Results This appendix presents off-to-online fine-tuning results for the off-to-online evaluation. We compare VAN-Flow against QC [4] across long-horizon, noisy, and D4RL AntMaze tasks. Scores in gray indicate performance before online fine-tuning (the offline scores of Tables 1 and 2), and bold scores denote performance after fine-tuning. Table 13: Offline-to-online fine-tuning performance across diverse long-horizon and noisy control tasks. Scores are reported before online fine-tuning in gray and after fine-tuning in bold. The offline scores are those reported in Tables 1 and 2. Task VAN-Flow QC Task VAN-Flow QC antmaze-teleport-navigate 57 → 96 40 → 46 scene-play 93 → 100 84 → 99 antmaze-large-navigate 95 → 99 49 → 98 puzzle-3x3-play 100 → 100 100 → 100 Long-horizon tasks Noisy tasks humanoidmaze-large-navigate 81 → 98 16 → 16 antmaze-large-explore 84 → 98 58 → 96 humanoidmaze-giant-navigate 92 → 98 12 → 67 puzzle-3x3-noisy 97 → 100 100 → 100 antmaze-giant-navigate 68 → 98 2 → 69 D4RL AntMaze tasks antmaze-umaze 98.0 → 96 95.0 → 98 antmaze-medium-diverse 89.3 → 94 64.0 → 95 antmaze-umaze-diverse 95.6 → 98 78.0 → 92 antmaze-large-play 86.5 → 95 80.0 → 94 antmaze-medium-play 90.7 → 98 72.0 → 98 antmaze-large-diverse 88.6 → 98 82.0 → 93 Discussion. Off-to-online evaluation tests whether offline-trained policies serve as strong initializations for online fine-tuning [60]. We perform online post-training from offline agents and compare with QC [4], a state-of-the-art horizon-reduction method, to evaluate practical adaptability. As shown in Table 13, VAN-Flow adapts stably online and improves with more interaction on every task that is not already saturated offline. Notably, it matches or slightly surpasses QC in several settings, indicating that variance-averse learning provides high-quality initializations that transfer smoothly from offline to online RL. Appendix K Additional Comparisons with Risk-Sensitive Objectives We additionally compare the proposed variance-averse expectation with representative risk-sensitive objectives. To this end, we replace ℰ(Z)E(Z) in (10) and (13) with alternative aggregation rules, while keeping all other components unchanged. We consider [Z]E[Z], CVaR, CPW, Norm, Wang, entropic risk, Sharpe ratio, Sortino ratio, and mean–variance. The comparisons are conducted on antmaze-teleport-navigate and antmaze-large-explore, which represent two challenging regimes for variance-aware learning. As shown in Table 14, VE consistently outperforms these alternatives across both tasks. Tail-focused objectives such as CVaR can become overly conservative, while scalar penalty-based objectives such as mean–variance or entropic risk do not fully exploit the categorical return structure. By contrast, ℰ(Z)E(Z) applies a smooth CDF-based reweighting over the full return distribution, which better aligns with our goal of suppressing unreliable high-variance targets without discarding distributional information. Table 14: Comparison with representative risk-sensitive objectives. Scores denote success rates averaged over five random seeds. Task [] E[Z] CVaR CPW Norm Wang Entropic Sharpe Sortino Mean–Var. VE teleport ±738\!±\!7 ±1654\!±\!16 ±537\!±\!5 ±936\!±\!9 ±1040\!±\!10 ±1644\!±\!16 ±1125\!±\!11 ±830\!±\!8 ±846\!±\!8 ±71\!±\!7 explore ±284\!±\!2 ±12\!±\!1 ±487\!±\!4 ±884\!±\!8 ±286\!±\!2 ±486\!±\!4 ±2764\!±\!27 ±870\!±\!8 ±485\!±\!4 ±93\!±\!3 Appendix L Full Results on Sensitivity Analyses We provide extended sensitivity results for the key hyperparameters n, M, and K. All plots report mean ± standard deviation over random seeds. L.1 n-step Horizon Figure 13: Additional sensitivity results on the number of n. We sweep n over a task-dependent grid (e.g., n∈1,4,8,16n∈\1,4,8,16\). For long-horizon tasks, increasing n initially improves performance by reducing bootstrapping bias, but overly large n can cause instability due to amplified return variance. VAN-Flow remains robust across a wide range of n, indicating that variance-aware aggregation mitigates the typical sensitivity of long-horizon targets. L.2 Number of Rejection Samples Figure 14: Additional sensitivity results on the number of rejection samples We vary the number of rejection candidates M. Performance generally improves with M and quickly saturates, so strong performance is achieved without excessive sampling. L.3 Number of Flow Steps Figure 15: Additional sensitivity results on the number of flow steps. We vary the number of flow discretization steps K (e.g., K∈1,3,5,10,20K∈\1,3,5,10,20\). A single step is not enough on any task. The maze tasks are already close to their best with three to five steps, whereas scene-play and puzzle-3x3-play need about ten steps, and going beyond that brings little. On humanoidmaze-large, Table 9 even shows a mild decline for large K. Precise ODE integration is therefore not required, and K=10K=10 is a safe default across tasks. L.4 Aversion Coefficient δ Table 15 sweeps δ on the first task of three environments. Too little aversion leaves the variance unsuppressed, and too much aversion makes the policy overly conservative on some tasks, most visibly on antmaze-large-explore, where the success rate collapses for δ≥5δ≥ 5. The default δ=2δ=2 is at or near the best value on all three tasks and avoids this collapse, which is why we use it as the default. The one exception is antmaze-teleport-navigate, where the stochastic teleporters put irreducible variance into multi-step targets and a stronger aversion (δ=7δ=7, Table 7) works better. The trend matches Theorem 3.2: the spectral form interpolates between the plain expectation at δ=0δ=0 and worst-case selection as δ→∞δ→∞. Table 15: Success rate for different values of the aversion coefficient δ (task1 of each environment, five seeds). Task δ=0δ=0 11 22 (default) 33 44 55 1010 antmaze-giant-navigate 44.0±11.8± 11.8 53.6±6.9± 6.9 58.0±4.058.0 0.68887pt $± 4.0$ 30.0±3.3± 3.3 29.5±6.7± 6.7 31.6±4.1± 4.1 30.4±5.9± 5.9 antmaze-large-explore 82.0±8.6± 8.6 89.5±3.4± 3.4 93.6±3.093.6 0.68887pt $± 3.0$ 90.8±4.5± 4.5 65.2±33.0± 33.0 0.0±0.0± 0.0 0.0±0.0± 0.0 humanoidmaze-large-navigate 78.0±3.8± 3.8 82.0±7.7± 7.7 87.4±5.887.4 0.68887pt $± 5.8$ 83.2±3.2± 3.2 81.6±6.1± 6.1 82.6±2.7± 2.7 84.8±2.7± 2.7 L.5 Categorical Support The support is set by the rule [Vmin,Vmax]=[rmin/(1−γ),rmax/(1−γ)][V_ ,V_ ]=[r_ /(1-γ),\,r_ /(1-γ)] rather than tuned per task. Table 16 varies VminV_ with the number of atoms fixed. Underestimating the support clips the Bellman targets and breaks training, whereas overestimating it only lowers the resolution and costs little. The practical guideline is simply to make sure the support covers the return range, and to err on the wide side. Table 16: Success rate for different lower bounds VminV_ of the categorical support (Vmax=0V_ =0, I=101I=101, five seeds). Task −100-100 −170-170 −200-200 (rule) −230-230 −300-300 antmaze-large-navigate 0.0±0.0± 0.0 94.4±3.7± 3.7 94.6±4.2± 4.2 93.2±4.7± 4.7 96.0±1.4± 1.4 puzzle-3x3-noisy 0.0±0.0± 0.0 81.1±22.5± 22.5 98.6±2.8± 2.8 99.6±0.8± 0.8 94.2±11.0± 11.0 L.6 Number of Atoms Table 17 varies the number of atoms I over roughly a tenfold range and reports the training wall-clock time next to each result. Performance is stable once the resolution is sufficient. The only exception is scene-play with I=51I=51, where the coarse grid under-resolves the return distribution, and doubling the atom count restores the performance. Wall-clock time barely changes across the range, so the default I=101I=101 provides enough resolution at no meaningful cost. Table 17: Success rate and training wall-clock time (hours) for different numbers of atoms I (five seeds). Task I=51I=51 I=101I=101 (default) I=201I=201 I=501I=501 antmaze-large-navigate 91.6±4.6± 4.6 (1.28h) 94.6±4.2± 4.2 (1.26h) 92.4±3.9± 3.9 (1.23h) 95.6±2.3± 2.3 (1.31h) humanoidmaze-giant-navigate 79.0±9.5± 9.5 (2.28h) 85.4±5.0± 5.0 (2.32h) 83.5±6.5± 6.5 (2.24h) 79.5±9.2± 9.2 (2.26h) puzzle-3x3-noisy 98.4±2.3± 2.3 (1.22h) 98.6±2.8± 2.8 (1.19h) 98.8±1.6± 1.6 (1.19h) 97.2±3.0± 3.0 (1.22h) scene-play 20.5±18.6± 18.6 (1.55h) 92.4±4.5± 4.5 (1.57h) 86.8±5.9± 5.9 (1.58h) 86.8±4.3± 4.3 (1.57h) Appendix M Broader Impact and Limitation Broader Impact This work improves the reliability of offline reinforcement learning in sparse and long-horizon environments by reducing sensitivity to high-variance multi-step targets. Such improvements may benefit applications where online exploration is expensive or unsafe, including robotics, autonomous navigation [61, 62, 60, 63], UAV control [64], and decision-support systems trained from logged data [65, 66]. At the same time, stronger offline RL methods could enable automated decision-making systems with greater real-world impact. If trained on biased or incomplete datasets, such systems may inherit or amplify undesirable behaviors present in the data. Careful dataset curation, evaluation, and deployment safeguards therefore remain important when applying offline RL methods in safety-critical settings. Limitation The proposed approach introduces modest additional computation due to the use of rejection sampling and distributional aggregation. However, these operations are fully parallelizable and align well with modern accelerator hardware, making the practical overhead small in typical training settings. Another limitation is that the method relies on categorical distributional critics to represent return distributions. While such critics are widely used in distributional RL, extending the proposed aggregation operator to other distributional representations (e.g., quantile-based critics) remains an interesting direction for future work. Finally, our reliability hypothesis targets return variance that comes from heterogeneous behavior in the dataset. Most of the benchmarks we use have deterministic dynamics, so this is the dominant source of variance there. In environments with strongly stochastic dynamics, part of the return variance is irreducible, and penalizing it indiscriminately can make the policy more conservative than necessary even for well-supported actions. A state-dependent δ that relaxes the aversion where the remaining variance is unavoidable, or a δ that is annealed during online fine-tuning, would be natural extensions. Both fit the framework without modification, but estimating such a coefficient reliably needs additional uncertainty machinery, and we leave this to future work.