Paper deep dive
Posterior-Averaged Bellman Flow Matching for Distributional Critics
Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Distributional critics model the full return distribution, providing richer signals for exploration and risk-sensitive control. Recent work has shown that flow-based critics can model these distributions flexibly and provide effective supervision across flow time for temporal-difference learning. However, unlike standard flow matching with fixed data endpoints, a flow critic is trained on Bellman targets recursively generated by a learned target critic under the current policy. Existing objectives therefore either use noisy single-endpoint supervision or rely on the learned flow itself for additional guidance. Our key observation is that an intermediate bridge state may be compatible with several Bellman endpoints, so its velocity should average their directions under the corresponding posterior. We propose Posterior-Averaged Bellman Flow Matching (PA-BFM), which samples an endpoint pool, infers the closed-form posterior over the latent anchor, and regresses to the posterior-mean velocity. Uniform anchor selection makes this target an exact finite-pool Rao--Blackwellization of the single-endpoint target: for every \(K\), it preserves the population target, cannot increase conditional variance, and its conditional variance scales as \(O(K^{-1})\). The endpoint pool supports conservative updates, while the twin return flows provide critic-disagreement and unsupported-mass diagnostics. Controlled studies show that PA-BFM reduces fixed-query target mean-squared error by \(79.7\%\) and target-induced gradient variance by up to \(98.0\%\), while improving return-distribution fidelity. Across 50 OGBench tasks and four offline-to-online scenarios, PA-BFM achieves higher aggregate performance than Value Flows.
Tags
Links
- Source: https://arxiv.org/abs/2610.17788v1
- Canonical: https://arxiv.org/abs/2610.17788v1
Trouble viewing inline? Open PDF directly →
Full Text
114,763 characters extracted from source content.
Expand or collapse full text
Posterior-Averaged Bellman Flow Matching for Distributional Critics Zeqiao Li1, Yijing Wang1, Haoyu Wang1,2, Zheng Li1, Zhiqiang Zuo1 1Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China 2Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China lizeqiao, yjwang, why2014, zhengl, zqzuo@tju.edu.cn Abstract Distributional critics model the full return distribution, providing richer signals for exploration and risk-sensitive control. Recent work has shown that flow-based critics can model these distributions flexibly and provide effective supervision across flow time for temporal-difference learning. However, unlike standard flow matching with fixed data endpoints, a flow critic is trained on Bellman targets recursively generated by a learned target critic under the current policy. Existing objectives therefore either use noisy single-endpoint supervision or rely on the learned flow itself for additional guidance. Our key observation is that an intermediate bridge state may be compatible with several Bellman endpoints, so its velocity should average their directions under the corresponding posterior. We propose Posterior-Averaged Bellman Flow Matching (PA-BFM), which samples an endpoint pool, infers the closed-form posterior over the latent anchor, and regresses to the posterior-mean velocity. Uniform anchor selection makes this target an exact finite-pool Rao–Blackwellization of the single-endpoint target: for every K, it preserves the population target, cannot increase conditional variance, and its conditional variance scales as O(K−1)O(K^-1). The endpoint pool supports conservative updates, while the twin return flows provide critic-disagreement and unsupported-mass diagnostics. Controlled studies show that PA-BFM reduces fixed-query target mean-squared error by 79.7%79.7\% and target-induced gradient variance by up to 98.0%98.0\%, while improving return-distribution fidelity. Across 50 OGBench tasks and four offline-to-online scenarios, PA-BFM achieves higher aggregate performance than Value Flows. 1 Introduction In actor–critic reinforcement learning (RL), policy improvement depends on the critic used to evaluate actions and construct Bellman targets. Most methods retain only the expected return, whereas distributional RL preserves information about stochastic outcomes, multimodality, and risk (Bellemare et al., 2017; Dabney et al., 2018a; Dabney et al., 2018b; Duan et al., 2022). Recent flow-based critics represent this distribution through a continuous transport from simple noise to return samples (Dong et al., 2026; Xu et al., 2026; Ding et al., 2026; Zhong et al., 2025; Chen et al., 2025). Flow-time supervision further provides dense signals for changing temporal-difference targets (Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023; Liu et al., 2023). Existing flow critics use two complementary forms of Bellman supervision. Endpoint matching is anchored to a sampled Bellman return but uses one continuation, whereas pathwise matching provides denser supervision through another learned successor field. Value Flows combines these objectives with target networks and confidence weighting, while Path-Coupled Bellman Flows (PCBF) introduces source-consistent geometry and a learned-field control variate (Dong et al., 2026; Xu et al., 2026). Related bridge, distillation, and architectural corrections retain the same tension between an anchored but noisy target and guidance from a learned flow (Ding et al., 2026; Chen et al., 2025; Agrawalla et al., 2025). We recast this tension as a conditional estimation problem. At a fixed intermediate bridge state, multiple Bellman endpoints can be compatible with the same observation. Each compatible endpoint implies a candidate velocity, and the population bridge velocity is the posterior average of these endpoint directions. The uncertainty is therefore not whether a Bellman endpoint law exists, but which compatible endpoint explains the observed intermediate state. Our goal is to estimate this posterior velocity directly from Bellman endpoint samples. This estimator is particularly natural in distributional RL: conditioned on a replay transition, the target policy and return flow directly sample the conditional Bellman endpoint law. For the one-dimensional linear Gaussian critic bridge, candidate likelihoods are closed form, so the finite endpoint pool induces a tractable posterior over the latent anchor without endpoint retrieval or an auxiliary proposal distribution. In contrast, posterior augmentation in generative modeling often requires retrieval or approximate candidate construction, and policy-side reverse matching may require a separate proposal (Stoica et al., 2026; Yang et al., 2026; Li et al., 2026). Based on this observation, we propose Posterior-Averaged Bellman Flow Matching (PA-BFM). For each transition, PA-BFM samples a finite endpoint pool, selects one endpoint uniformly to construct the bridge state, and evaluates every candidate using its Gaussian bridge likelihood. The posterior-weighted direction is exactly the conditional expectation of the single-endpoint Bellman conditional flow matching (BCFM) target. Hence every finite pool preserves the population bridge velocity and cannot increase conditional target variance. We further show that its conditional variance scales as O(K−1)O(K^-1) and derive implications for gradient variability, terminal transport error, and Bellman tracking. The endpoint posterior supports conservative reweighting, while the twin return flows provide disagreement and unsupported-mass diagnostics (Kuznetsov et al., 2020; Kumar et al., 2019; Kumar et al., 2020; Ma et al., 2021; Fujimoto et al., 2018). The main experiments use the undistorted target and retain matched flow-critic policy extraction. Our main contributions are: • Posterior-averaged Bellman supervision. We formulate source-consistent endpoint matching as posterior velocity estimation and estimate it from K Bellman endpoints using closed-form bridge posteriors, without regressing to a learned successor field. • Exact finite-pool guarantees. Uniform anchor selection makes PA-BFM an exact Rao–Blackwellization of BCFM. We prove finite-K unbiasedness and variance dominance, characterize K−1K^-1 conditional-variance scaling with an ESS-based bound, and propagate velocity error to terminal Wasserstein and Bellman tracking errors. • Extensions and empirical validation. The endpoint pool supports conservative posterior tilting, while the twin return flows provide critic-disagreement and unsupported-mass diagnostics. Controlled studies, 50 OGBench tasks, and four offline-to-online scenarios show gains in estimation, return-law fidelity, and downstream performance. 2 Preliminaries We review distributional returns, one-dimensional flow matching, and the source-consistent Bellman bridge used by our critic. 2.1 MDPs and Distributional Returns We consider a discounted Markov decision process ℳ=(,,P,ℛ,ρ0,γ)M=(S,A,P,R, _0,γ), where S and A are the state and action spaces, P is the transition kernel, ℛR is the reward function, ρ0 _0 is the initial-state distribution, and γ∈(0,1)γ∈(0,1). In offline reinforcement learning (RL), the learner observes a fixed dataset =(s,a,r,s′,d)D=\(s,a,r,s ,d)\, where d∈0,1d∈\0,1\ indicates termination and r is the realized reward. We use h for environment time and t∈[0,1]t∈[0,1] for flow time. For a policy π, the return random variable is Zπ(s,a)=∑h=0∞γhℛ(Sh,Ah),S0=s,A0=a,Z^π(s,a)= _h=0^∞γ^hR(S_h,A_h), S_0=s, A_0=a, (1) where Sh+1∼P(⋅∣Sh,Ah)S_h+1 P(· S_h,A_h) and Ah∼π(⋅∣Sh)A_h π(· S_h) for h≥1h≥ 1. Its expectation is the action value Qπ(s,a)=[Zπ(s,a)]Q^π(s,a)=E[Z^π(s,a)], whereas distributional RL models the law of Zπ(s,a)Z^π(s,a) itself (Bellemare et al., 2017; Dabney et al., 2018a; Dabney et al., 2018b). For a state–action conditional return variable Z, the distributional Bellman operator is (πZ)(s,a) (T^πZ)(s,a) =dr+γ(1−d)Z(S′,A′), d=r+γ(1-d)Z(S ,A ), (2) S′ S ∼P(⋅∣s,a),A′∼π(⋅∣S′). P(· s,a), A π(· S ). For a nonterminal transition with reward r, its density satisfies pπZ(z∣s,a)=1γS′,A′[pZ(z−rγ|S′,A′)].p_T^πZ(z s,a)= 1γE_S ,A [p_Z\! ( z-rγ |S ,A ) ]. (3) Terminal transitions place all mass at z=rz=r. The operator is a γ-contraction under Wasserstein metrics and has ZπZ^π as its unique fixed point (Bellemare et al., 2017). 2.2 Flow Matching Flow matching (FM) learns a time-dependent vector field that transports a standard Gaussian source to a target distribution p1(⋅∣c)p_1(· c). Let ϵ∼(0,1)ε (0,1), and let p0p_0 denote its density. The flow solves dtϕθ(ϵ∣t,c)=vθ(ϕθ(ϵ∣t,c),t,c),ϕθ(ϵ∣0,c)=ϵ. ddt _θ(ε t,c)=v_θ\! ( _θ(ε t,c),t;c ), _θ(ε 0,c)=ε. (4) Conditional flow matching (CFM) trains this field using tractable paths (Lipman et al., 2023; Albergo and Vanden-Eijnden, 2023; Liu et al., 2023). For independent ϵ∼(0,1)ε (0,1), y∼p1(⋅∣c)y p_1(· c), and t∼Unif([0,1))t ([0,1)), define Xt=(1−t)ϵ+ty,ut(Xt,y)=y−ϵ=y−Xt1−t.X_t=(1-t)ε+ty, u_t(X_t,y)=y-ε= y-X_t1-t. (5) The CFM objective is ℒCFM(θ)=[‖vθ(Xt,t,c)−ut(Xt,y)‖22].L_CFM(θ)=E\! [ \|v_θ(X_t,t;c)-u_t(X_t,y) \|_2^2 ]. (6) Its population minimizer is v⋆(x,t;c)=[ut(x,Y)∣Xt=x,c]v (x,t;c)=E[u_t(x,Y) X_t=x,c]: the conditional average of all path directions compatible with the observed intermediate point. For our critic, c=(s,a)c=(s,a), and Zθ(s,a)=dϕθ(ϵ∣1,s,a)Z_θ(s,a) d= _θ(ε 1,s,a). 2.3 Bellman Endpoint Laws and Source-Consistent Bridges Fix (s,a)(s,a) and let τ=(r,s′,d)τ=(r,s ,d) denote one transition outcome. Let Y∼μτY _τ denote the abstract conditional Bellman endpoint law. A sample from this law is generated by a target-flow return Zθ¯(s′,a′,ϵ′)Z_ θ(s ,a ;ε ): Y Y =r+γ(1−d)Zθ¯(s′,A′,ϵ′), =r+γ(1-d)Z_ θ(s ,A ;ε ), (7) μτ _τ =Law(Y∣τ),A′∼π(⋅∣s′),ϵ′∼(0,1). =Law(Y τ), A π(· s ), ε (0,1). In our implementation, μτ _τ is the law of the shared-noise mean of the two target critics; the explicit construction is given in Appendix C.1. An endpoint pool averages the successor-action and target-flow randomness within one transition; replay sampling performs the outer average over τ|s,aτ s,a. The Bellman endpoint law does not uniquely determine an intermediate path. For a nonterminal transition (d=0d=0), applying the Bellman affine map to each slice of a successor flow gives Xtsl=r+γXt′X_t^sl=r+γ X_t . If Xt′X_t has density pt′(⋅∣s′,A′)p_t (· s ,A ) and velocity vt′v_t , then, with ξ=(x−r)/γξ=(x-r)/γ, qtsl(x) q_t^sl(x) =1γA′[pt′(ξ∣s′,A′)], = 1γE_A [p_t (ξ s ,A )], (8) vtsl(x) v_t^sl(x) =γA′[pt′(ξ∣s′,A′)vt′(ξ∣s′,A′)]A′[pt′(ξ∣s′,A′)]. =γ E_A [p_t (ξ s ,A )v_t (ξ s ,A )]E_A [p_t (ξ s ,A )]. (9) The factor γ follows from X˙tsl=γX˙t′ X_t^sl=γ X_t . Although this path reaches the correct Bellman endpoint, its source is (r,γ2)N(r,γ^2), rather than the common critic source (0,1)N(0,1). For d=1d=1, the endpoint law is the point mass at Y=rY=r. We therefore use the source-consistent bridge Xt X_t =(1−t)ϵ+tY,ϵ⟂Y, =(1-t)ε+tY, ε Y, (10) ϵ ε ∼(0,1),Y∼μτ. (0,1), Y _τ. If the successor uses the linear path Xt′=(1−t)ϵ+tX1′X_t =(1-t)ε+tX_1 with the same source noise, then Xt=Xtsl+(1−t)[(1−γ)ϵ−r].X_t=X_t^sl+(1-t) [(1-γ)ε-r ]. (11) Thus the two paths share the endpoint at t=1t=1, but not their intermediate marginals. Using the standard Gaussian density p0p_0, define ℓt(x∣y)=11−tp0(x−ty1−t),ut(x,y)=y−x1−t. _t(x y)= 11-tp_0\! ( x-ty1-t ), u_t(x,y)= y-x1-t. (12) The transition-conditional bridge velocity is vτ⋆(x,t)=[ut(x,Y)|Xt=x,τ].v_τ (x,t)=E\! [u_t(x,Y) |X_t=x,τ ]. (13) The full state–action velocity is recovered by the outer transition mixture, v⋆(x,t∣s,a)=[vτ⋆(x,t)|Xt=x,s,a].v (x,t s,a)=E\! [v_τ (x,t) |X_t=x,s,a ]. (14) Hence the endpoint pool estimates the velocity conditional on one transition, while replay sampling accounts for environment-transition randomness. 3 Method We use PA-BFM for the complete algorithm and PA-BCFM for its posterior-averaged critic-training objective. 3.1 Bellman Objectives and Posterior Velocity Bellman target noise is recursively reused in later temporal-difference updates. Unlike standard FM with fixed data endpoints, a flow critic draws from a learned Bellman law under the current policy, so error can re-enter later targets. We seek an endpoint-anchored estimator that averages continuation uncertainty already present in this law. For a sampled transition τ=(r,s′,d)τ=(r,s ,d), draw a Bellman endpoint Y Y =r+γ(1−d)Zθ¯(s′,A′,ϵ′), =r+γ(1-d)Z_ θ(s ,A ;ε ), (15) A′∼π(⋅∣s′), A π(· s ), ϵ′∼(0,1), ε (0,1), and construct the source-consistent bridge Xt=(1−t)ϵ+tYX_t=(1-t)ε+tY, whose supervision direction is ut(Xt,Y)=(Y−Xt)/(1−t)=Y−ϵu_t(X_t,Y)=(Y-X_t)/(1-t)=Y-ε. Bellman conditional flow matching (BCFM) minimizes ℒBCFM(θ)=[‖vθ(Xt,t,s,a)−ut(Xt,Y)‖22].L_BCFM(θ)=E\! [ \|v_θ(X_t,t;s,a)-u_t(X_t,Y) \|_2^2 ]. (16) Conditioned on Xt=xX_t=x, its population target is vτ⋆(x,t)=[ut(x,Y)|Xt=x,τ].v_τ (x,t)=E\! [u_t(x,Y) |X_t=x,τ ]. (17) BCFM is conditionally unbiased, but Xt=xX_t=x does not identify which compatible endpoint was sampled. Using one continuation therefore introduces avoidable label variance. This is a latent-identity problem: after observing the bridge point, the missing quantity is which compatible continuation produced it. Value Flows additionally uses distributional conditional flow matching (DCFM) to impose pathwise consistency (Dong et al., 2026). With Xt′=ϕθ¯(ϵ′∣t,s′,A′)X_t = _ θ(ε t,s ,A ), its practical objective is ℒDCFMVF(θ) _DCFM^VF(θ) =[‖ΔtVF‖22], =E\! [\| _t^VF\|_2^2 ], (18) ΔtVF _t^VF =vθ(r+γXt′,t,s,a)−vθ¯(Xt′,t,s′,A′). =v_θ(r+γ X_t ,t;s,a)-v_ θ(X_t ,t;s ,A ). DCFM provides dense learned-field supervision, but it is not an estimator of the source-consistent posterior target in Eq. (17). The corresponding slice-wise affine geometry would instead require vθ(r+γx′,t,s,a)≈γvθ¯(x′,t,s′,A′),v_θ(r+γ x ,t;s,a)≈γ v_ θ(x ,t;s ,A ), (19) and follows the source-mismatched path in Eq. (9). Because both sides of Eq. (18) are learned, degenerate shared fields may also satisfy its residual. DCFM is therefore dense but self-referential, whereas BCFM is anchored but single-sample. Bridge corrections and critic-specific architectures can strengthen this supervision, but preserve the distinction between sampled endpoint anchors and guidance from a learned field (Ding et al., 2026; Chen et al., 2025; Agrawalla et al., 2025). PCBF instead corrects the single-endpoint target with a learned-field control variate and a tunable coefficient (Xu et al., 2026). Thus BCFM samples one continuation, PCBF corrects it with a learned field, and PA-BFM conditions the same target on a finite endpoint pool. This comparison is estimator-level: PA-BFM changes how the Bellman direction is formed while leaving the target policy, return-flow parameterization, and policy-extraction protocol unchanged. 3.2 Posterior-Averaged Bellman Flow Matching PA-BCFM implements the conditional averaging suggested above in four steps: it samples a Bellman endpoint pool, selects one latent anchor to form the bridge point, infers the anchor posterior, and averages the corresponding BCFM directions. Unlike generic posterior augmentation, no retrieval model is needed: the target return flow already supplies candidates from the relevant transition-conditional law, and the linear Gaussian bridge supplies their likelihoods analytically. The pool exposes uncertainty about compatible continuations without adding another learned correction field. For one sampled transition, PA-BCFM draws =(Y1,…,YK),Yj∼i.i.d.μτ,j=1,…,K.Y=(Y_1,…,Y_K), Y_j i.i.d. _τ, j=1,…,K. (20) Each endpoint is generated by the target policy and target return flow as in Eq. (15). In practice, μτ _τ is instantiated by the shared-noise mean of the twin target critics; this does not change the estimator below. All candidates share the sampled transition but retain independent successor-action and return-noise draws, so increasing K targets continuation uncertainty rather than replay-transition uncertainty. It then chooses one candidate uniformly to construct the observed bridge point, I I ∼Unif1,…,K,ϵ∼(0,1), \1,…,K\, ε (0,1), (21) xt x_t =(1−t)ϵ+tYI. =(1-t)ε+tY_I. The index I is the unobserved identity of the endpoint that generated xtx_t. Sampling the anchor from the same pool gives a finite candidate model in which the posterior responsibility of every endpoint can be evaluated exactly. Given the bridge point, a candidate YjY_j explains xtx_t through the source value (xt−tYj)/(1−t)(x_t-tY_j)/(1-t). Bayes’ rule therefore gives wj(xt,t) w_j(x_t,t) =ℓt(xt∣Yj)∑ℓ=1Kℓt(xt∣Yℓ) = _t(x_t Y_j) _ =1^K _t(x_t Y_ ) (22) =Pr(I=j|Xt=xt,,τ,t). = \! (I=j |X_t=x_t,Y,τ,t ). Thus wjw_j is the exact posterior responsibility of candidate YjY_j, not a heuristic similarity score. Uniform averaging ignores this compatibility by assigning every candidate the same responsibility. Early in the bridge, many endpoints can explain xtx_t and the weights are broad; near the endpoint, compatibility becomes sharper. The endpoint-pool effective sample size below measures this time-dependent concentration. Finally, define the candidate directions and posterior target by uj u_j =ut(xt,Yj)=Yj−xt1−t, =u_t(x_t,Y_j)= Y_j-x_t1-t, (23) u^K(xt,t) u_K(x_t,t) =∑j=1Kwj(xt,t)uj. = _j=1^Kw_j(x_t,t)u_j. PA-BCFM trains the twin online critics toward this shared target: ℒPA-BCFM _PA -BCFM =12∑c=12[ = 12 _c=1^2E\! [ (24) ∥vθc(xt,t;s,a)−sg[u^K(xt,t)]∥22], \|v_ _c(x_t,t;s,a)-sg[ u_K(x_t,t)] \|_2^2 ], where sgsg denotes stop-gradient. Replay sampling supplies the outer transition mixture in Eq. (14). Thus PA-BCFM modifies only critic supervision. The policy proposal, action evaluation, and target-network update remain those of the matched flow-critic implementation. The key point is that posterior averaging changes only the target noise, not the target mean. The following result holds for the practical finite pool. Theorem 1 (Finite-pool Rao–Blackwellization). Assume that the bridge normalizer is positive and the endpoint direction has a finite conditional second moment. When K=1K=1, u^1=uI u_1=u_I, so PA-BCFM exactly recovers BCFM. For every K≥1K≥ 1, u^K u_K =[uI|Xt=xt,,τ,t], =E\! [u_I |X_t=x_t,Y,τ,t ], (25) [u^K∣Xt=xt,τ,t] [ u_K X_t=x_t,τ,t] =vτ⋆(xt,t). =v_τ (x_t,t). Moreover, Var(u^K∣Xt,τ,t)≤Var(uI∣Xt,τ,t). ( u_K X_t,τ,t)≤ (u_I X_t,τ,t). (26) Thus every finite pool preserves the population target and cannot increase conditional target variance. Appendix B.2 gives the corresponding weighted total-variance equality. PA-BCFM removes posterior dispersion among compatible endpoint directions while leaving the conditional mean unchanged. The pool supplies additional information by conditioning away the sampled continuation identity, not by changing the Bellman endpoint law. Equivalently, for any prediction v, [(v−u^K)2∣Xt,τ,t]=(v−vτ⋆)2+Var(u^K∣Xt,τ,t),E\! [(v- u_K)^2 X_t,τ,t ]=(v-v_τ )^2+ ( u_K X_t,τ,t), (27) so every finite pool has the same population regression minimizer. Under the moment conditions in Appendix B.3, the remaining large-pool variance obeys Var(u^K∣x,t,τ)=στ2(x,t)K+o(K−1), ( u_K x,t,τ)= _τ^2(x,t)K+o(K^-1), (28) where στ2(x,t)=Y∼μτ[ℓt(x∣Y)2(ut(x,Y)−vτ⋆(x,t))2](Y∼μτ[ℓt(x∣Y)])2. _τ^2(x,t)= E_Y _τ\! [ _t(x Y)^2 (u_t(x,Y)-v_τ (x,t) )^2 ] (E_Y _τ[ _t(x Y)] )^2. (29) The effective sample size ESS()=1∑j=1Kwj2 (w)= 1 _j=1^Kw_j^2 (30) measures posterior concentration. Appendix B.3 shows that the normalized ESS converges to a transition-dependent concentration factor and controls the constant in the asymptotic variance bound. Larger pools help most when several candidates retain nonnegligible posterior mass; a larger nominal K cannot compensate for a fully concentrated posterior. For the squared loss, if g1g_1 is the BCFM gradient and gKg_K the PA-BCFM gradient at the same bridge point, then gK g_K =[g1∣Xt=xt,,τ,t], =E[g_1 X_t=x_t,Y,τ,t], (31) Cov(gK∣Xt,τ,t) (g_K X_t,τ,t) ⪯Cov(g1∣Xt,τ,t). (g_1 X_t,τ,t). Thus the same Rao–Blackwell ordering applies directly to target-induced gradient variability. Corollary 1 (Training and transport error). Under the smoothness, Polyak–Łojasiewicz, and flow-Lipschitz assumptions in Appendix B.4, the pool-induced contribution to the SGD error floor is O((BmbK)−1)O((B_mbK)^-1), and its contribution to terminal W1W_1 error is O((BmbK)−1/2)O((B_mbK)^-1/2). For a fixed policy, Bellman contraction amplifies the one-step error by at most 1/(1−γ)1/(1-γ). The complete constants and the changing-policy tracking recursion are given in Appendix B.4. 3.3 Conservative Learning and Critic Diagnostics Before each critic update, PA-BCFM has formed the weighted endpoint pool (Yj,wj)j=1K\(Y_j,w_j)\_j=1^K. This pool supports conservative reweighting and a training-time disagreement signal. Separate state–action diagnostics are computed directly from the twin return flows. Let g:ℝ→[0,∞)g:R→[0,∞) favor the returns retained by a desired distortion. The reweighted posterior target is wjg w_j^g =wjg(Yj)∑ℓ=1Kwℓg(Yℓ), = w_jg(Y_j) _ =1^Kw_ g(Y_ ), (32) u^Kg u_K^g =∑j=1Kwjguj. = _j=1^Kw_j^gu_j. (33) The exact finite-pool Rao–Blackwell identity applies to the undistorted target g≡1g≡ 1, but generally not to a distorted target g≠1g≠ 1. A nonincreasing g downweights high returns and yields a pessimistically distorted target; hard upper-tail truncation and the soft tilt gη(y)=exp(−ηy)g_η(y)= (-η y) are two examples (Kuznetsov et al., 2020; Kumar et al., 2019; Kumar et al., 2020; Ma et al., 2021). If the population distortion is LgL_g-Lipschitz in WpW_p, its Bellman operator is γLgγ L_g-Lipschitz and is contractive when γLg<1γ L_g<1. Appendix B.5 gives the population definitions and proofs. For c∈1,2c∈\1,2\, the target critics share successor actions and return noises: Yj(c) Y_j^(c) =r+γ(1−d)ϕθ¯c(ϵj′∣1,s′,Aj′), =r+γ(1-d) _ θ_c(ε _j 1,s ,A _j), (34) Yj Y_j =12(Yj(1)+Yj(2)), = 12 (Y_j^(1)+Y_j^(2) ), Dpool(τ,xt,t,) D_pool(τ,x_t,t,Y) =∑j=1Kwj0 = _j=1^Kw_j^0 (Yj(1)−Yj(2))2. (Y_j^(1)-Y_j^(2) )^2. (35) Here Aj′∼π(⋅∣s′)A _j π(· s ) and ϵj′∼(0,1)ε _j (0,1), and wj0w_j^0 denotes the undistorted posterior weight obtained with g≡1g≡ 1. Thus the statistic is defined before any adaptive reweighting. For task e, let Δe=q.95,e−q.05,e _e=q_.95^D,e-q_.05^D,e be the fixed offline return-to-go range. The stop-gradient adaptive heuristic is η(τ,xt,t) η(τ,x_t,t) =η0sg[clip(DpoolΔe,0,1)], = _0sg\! [clip\! ( D_pool _e,0,1 ) ], (36) gτ,xt,t(y) g_τ,x_t,t(y) =exp[−η(τ,xt,t)yΔe]. = \! [-η(τ,x_t,t) y _e ]. For a fixed distortion g, Appendix B.5 establishes consistency and operator-level contraction conditions. The disagreement-adaptive schedule in Eq. (36) is a stop-gradient practical extension and is not covered by the fixed-distortion contraction result; its scope is clarified in Appendix B.6. For a queried action, shared return noises isolate disagreement between the twin target return flows: Zm(c)(s,a) Z_m^(c)(s,a) =ϕθ¯c(ξm∣1,s,a),c∈1,2, = _ θ_c( _m 1,s,a), c∈\1,2\, (37) ξm _m ∼(0,1),m=1,…,Mdiag, (0,1), m=1,…,M_diag, Dtwin(s,a) D_twin(s,a) =1Mdiag∑m=1Mdiag(Zm(1)(s,a)−Zm(2)(s,a))2. = 1M_diag _m=1^M_diag (Z_m^(1)(s,a)-Z_m^(2)(s,a) )^2. For each critic, flow inversion gives F^c(z∣s,a) F_c(z s,a) =Φ(ϕθ¯c−1(z∣1,s,a)), = \! ( _ θ_c^-1(z 1,s,a) ), (38) F^ens(z∣s,a) F_ens(z s,a) =12∑c=12F^c(z∣s,a), = 12 _c=1^2 F_c(z s,a), Pout(s,a) P_out(s,a) =1−F^ens(qhi∣s,a). =1- F_ens(q_hi^D s,a). (39) PoutP_out is the twin-ensemble mass above an empirical upper-tail threshold fixed before diagnostic evaluation. Unlike these state–action diagnostics, DpoolD_pool is conditioned on a training transition and bridge point. 3.4 Distributional Policy Improvement PA-BFM changes only the critic objective: a behavior flow proposes N actions, each evaluated with MactM_act shared return noises: ai a_i ∼πβ(⋅∣s),i=1,…,N, _β(· s), i=1,…,N, (40) ϵim _im ∼(0,1),m=1,…,Mact, (0,1), m=1,…,M_act, zim(c) z_im^(c) =ϕθc(ϵim∣1,s,ai),c∈1,2. = _ _c( _im 1,s,a_i), c∈\1,2\. The main experiments use a Monte Carlo estimate of mean return, μ^i μ_i =12Mact∑c=12∑m=1Mactzim(c), = 12M_act _c=1^2 _m=1^M_actz_im^(c), (41) a⋆ a =argmax1≤i≤Nμ^i. = *arg\,max_1≤ i≤ N μ_i. Appendix C.3 gives the shared candidate-selection and policy-training procedure. PA-BFM introduces no method-specific actor-side objective; matched flow-critic methods share the same policy-side procedure. Gaussian Policies Flow Policies Distributional Critics Domain IQL ReBRAC IFQL FQL IQN Value Flows PCBF PA-BFM cube-double-play (5 tasks) 6±26± 2 12±312± 3 14±514± 5 29±629± 6 42±842± 8 69±469± 4 71±571± 5 ±79± 4 cube-triple-play (5 tasks) 1±11± 1 0±00± 0 0±00± 0 4±24± 2 6±06± 0 ±14± 3 4±14± 1 ±14± 2 puzzle-3x3-play (5 tasks) 9±39± 3 22±222± 2 19±119± 1 30±430± 4 15±115± 1 87±1387± 13 53±353± 3 ±93± 3 puzzle-4x4-play (5 tasks) 7±27± 2 14±314± 3 25±825± 8 17±517± 5 27±427± 4 27±427± 4 30±430± 4 ±52± 6 scene-play (5 tasks) 28±328± 3 41±741± 7 30±430± 4 56±256± 2 40±140± 1 59±459± 4 54±454± 4 ±78± 2 visual-antmaze-medium-navigate (5 tasks) 84±584± 5 87±487± 4 87±287± 2 38±438± 4 74±474± 4 75±1075± 10 75±275± 2 ±91± 3 visual-antmaze-teleport-navigate (5 tasks) 6±46± 4 4±14± 1 10±410± 4 5±25± 2 4±24± 2 13±413± 4 14±414± 4 ±22± 5 visual-cube-double-play (5 tasks) 11±611± 6 1±11± 1 2±22± 2 6±16± 1 1±01± 0 13±213± 2 3±13± 1 ±17± 5 visual-puzzle-3x3-play (5 tasks) 2±32± 3 20±120± 1 21±021± 0 20±120± 1 19±119± 1 ±23± 2 18±118± 1 20±120± 1 visual-scene-play (5 tasks) 26±526± 5 28±528± 5 21±221± 2 41±441± 4 41±641± 6 43±743± 7 30±330± 3 ±50± 8 Table 1: Offline evaluation on OGBench. Each row averages five tasks; state-based and image-based results use eight and four seeds. Uncertainty is the arithmetic mean of the five task-level standard deviations. Matched flow critics share policy-side components while retaining their original critic supervision. Bold denotes the highest displayed mean in each domain. Figure 1: Controlled posterior-target evaluation. (a) Return-law W1W_1; (b) conditional target MSE; (c) target-induced gradient trace-covariance ratio; and (d) normalized ESS()/K (w)/K at K=8K=8. Sol./D-MC and VF denote Solitaire Dice/Discrete-MC and Value Flows; solid/dashed in (c) follow the same environment order, and dotted in (d) is 1/K1/K. Error bars are 95% confidence intervals; (b)–(c) use exact bridge-posterior anchors at fixed (τ,t,xt)(τ,t,x_t). 4 Experiments We separate estimator-level evaluation from downstream control. (Q1) Controlled processes test posterior-target error, gradient variability, and return-law fidelity. (Q2) Offline and offline-to-online benchmarks test downstream policy performance. (Q3) Ablations measure posterior weighting, pool size, and cost. (Q4) Diagnostics test reliability and pessimistic trade-offs. The controlled suite comprises a Bernoulli-return process, Solitaire Dice, and a discrete Monte Carlo chain. Downstream evaluation covers 50 OGBench tasks and four offline-to-online scenarios (Park et al., 2025a). Baselines span scalar, flow-policy, and distributional critic families; RLPD appears only offline-to-online (Kostrikov et al., 2022; Tarasov et al., 2023; Park et al., 2025b; Bellemare et al., 2017; Dabney et al., 2018a; Ma et al., 2021; Dong et al., 2026; Xu et al., 2026; Ball et al., 2023). Methods and evaluation. We use a common JAX evaluation pipeline. BCFM, Value Flows, PCBF, and PA-BFM follow a matched code path and share policy-side and optimization components while retaining their critic objectives and flow-time supervision, isolating posterior averaging within this group. Hyperparameters are tuned once per domain under a common budget; state and image tasks use eight and four seeds. K=8K=8 except in controlled and ablation studies. Full implementation, tuning, and evaluation details appear in Appendices D–E. 4.1 Controlled Posterior-Target Estimation The controlled suite separates conditional target estimation from trained-critic fidelity. Bernoulli fixes the configuration, while Solitaire Dice and Discrete-MC are used for evaluation. For each fixed bridge query (τ,t,xt)(τ,t,x_t), every replicate independently resamples the anchor from the exact bridge posterior and draws the remaining K−1K-1 endpoints from μτ _τ. We compare BCFM, uniform aggregation, and PA-BCFM for K∈1,2,4,8,16K∈\1,2,4,8,16\; the learned-flow study includes Value Flows and PCBF with the fixed coefficient λcv=0.3 _cv=0.3 (Dong et al., 2026; Xu et al., 2026). Figure 1(b–d) reports fixed-query estimates. Signed-bias 95% CIs contain zero for all K. Target MSE falls to 1.022[0.97,1.08]1.022\,[0.97,1.08] at K=8K=8 and 0.570[0.54,0.61]0.570\,[0.54,0.61] at K=16K=16, reductions of 79.7%79.7\% and 88.7%88.7\% relative to K=1K=1. At K=8K=8, the target-induced gradient trace-covariance ratios are 0.081[0.073,0.090]0.081\,[0.073,0.090] on Solitaire Dice and 0.020[0.017,0.023]0.020\,[0.017,0.023] on Discrete-MC. Normalized ESS()/K (w)/K falls from 0.9990.999 at t=0.01t=0.01 to 0.2950.295 at t=0.99t=0.99, indicating broad early averaging and sharper late selection. Figure 1(a) reports seed-level return-law fidelity: PA-BCFM has the lowest mean W1W_1 on both processes and overall, 3.538[3.41,3.67]3.538\,[3.41,3.67]; its paired difference from PCBF is −0.085[−0.156,−0.014]-0.085\,[-0.156,-0.014]. Figure 2: Offline-to-online evaluation. Mean success and one-standard-deviation bands over eight seeds on four OGBench scenarios. Gray marks one million offline optimizer updates, followed by one million online environment steps. Figure 3: Estimator components and pool-size sensitivity. Top: BCFM (K=1K=1), uniform and PA-BCFM (K=8K=8), and Value Flows. Bottom: pool-size sensitivity on two state-based tasks. Figure 4: Critic diagnostics and adaptive pessimism on puzzle-4x4-play-task4. Left: Error, Twin, and Out denote absolute prediction error, twin disagreement, and unsupported mass. Right: None, Fixed, Adapt., and VF denote undistorted, fixed, adaptive, and Value Flows variants. 4.2 Offline and Online Performance Table 1 summarizes offline performance. Fixed policy-side components isolate critic learning among matched flow critics. PA-BFM has the unique highest displayed mean on eight of ten domains and ties for the highest mean on one more. Relative to Value Flows, its state-based average rises from 51.251.2 to 63.263.2, its image-based average from 33.433.4 to 40.040.0, and the ten-domain average from 42.342.3 to 51.651.6. The task-level record is 3535 wins, 77 ties, and 88 losses across both observation modalities. Full task-level results appear in Appendix Tables 7–8 of the supplement. After one million offline optimizer updates and one million online environment steps, with an unchanged PA-BFM critic objective, Figure 2 shows mean aggregate success of 67.567.5 at online entry and a final mean of 94.7594.75, versus 62.2562.25 and 86.2586.25 for Value Flows and 53.2553.25 and 93.593.5 for FQL. 4.3 Estimator and Pool-Size Sensitivity Figure 3 isolates posterior weighting under matched K=8K=8 endpoints and training; only PA-BCFM weights endpoints by bridge compatibility. It improves final success by 19.519.5 and 21.221.2 points over uniform aggregation on cube-double task4 and puzzle-4x4 task2, and by 1414 and 2424 points over Value Flows. The pool-size sweep shows diminishing returns beyond K=8K=8: increasing K from 11 to 88 raises final success from 13.813.8 to 4444 and from 1717 to 5151, whereas K=16K=16 gives 43.443.4 and 50.150.1. On the representative state and visual configurations, the reported 10610^6-update costs at K=8K=8 are 4.04.0 and 11.811.8 hours, respectively, below the corresponding Value Flows times of 4.44.4 and 12.512.5 hours. This performance–cost saturation motivates the default K=8K=8. 4.4 Critic Diagnostics and Adaptive Pessimism On puzzle-4x4-play-task4 at the final offline checkpoint, twin-critic disagreement and unsupported mass track absolute prediction error under action shift, with Spearman correlations ρtwin=0.943 _twin=0.943 and ρout=0.886 _out=0.886. Fixed tilting reduces positive overestimation but eventually degrades success; the adaptive rule reduces it from 0.3800.380 to 0.1650.165 while raising final success from 0.4900.490 to 0.5150.515. Appendix D.4 provides the evaluation protocol and complete results for all evaluated fixed strengths. 5 Conclusion We introduced PA-BFM, whose default target is an exact finite-pool Rao–Blackwellization of the single-endpoint BCFM direction under the exact latent-anchor posterior. It preserves the population velocity for every K and cannot increase conditional target variance. We derive K−1K^-1-scaling variance and, under stated assumptions, optimization, terminal Wasserstein, and Bellman tracking bounds. The endpoint pool enables conservative reweighting, while the twin return flows provide critic-disagreement and unsupported-mass diagnostics. Controlled and OGBench experiments show improved estimation, return-law fidelity, and aggregate offline and offline-to-online performance over Value Flows. References Agrawalla et al. (2025) B. Agrawalla, M. Nauman, K. Agrawal, and A. Kumar Floq: training critics via flow-matching for scaling compute in value-based rl. External Links: 2509.06863 Cited by: §1, §3.1. Albergo and Vanden-Eijnden (2023) M. S. Albergo and E. Vanden-Eijnden Stochastic interpolants: a unifying framework for flows and diffusions. External Links: 2303.08797 Cited by: §1, §2.2. Ball et al. (2023) P. J. Ball, L. Smith, I. Kostrikov, and S. Levine Efficient online reinforcement learning with offline data. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 1577–1594. Cited by: §E.2, §4. Bellemare et al. (2017) M. G. Bellemare, W. Dabney, and R. Munos A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, p. 449–458. Cited by: §A.2, §E.2, §1, §2.1, §2.1, §4. Chen et al. (2025) D. Chen, Y. Liu, Z. Zhou, C. Qu, and Y. Qi Unleashing flow policies with distributional critics. External Links: 2509.23087 Cited by: §1, §1, §3.1. Dabney et al. (2018a) W. Dabney, G. Ostrovski, D. Silver, and R. Munos Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, p. 1096–1105. Cited by: §E.2, §1, §2.1, §4. Dabney et al. (2018b) W. Dabney, M. Rowland, M. G. Bellemare, and R. Munos Distributional reinforcement learning with quantile regression. In Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence, Vol. 32. Cited by: §1, §2.1. Ding et al. (2026) S. Ding, Y. Zhou, K. Hu, M. Pan, S. Zhong, Y. Fu, J. Wang, and Y. Shi Distributional reinforcement learning with diffusion bridge critics. External Links: 2602.05783 Cited by: §1, §1, §3.1. Dong et al. (2026) P. Dong, C. Zheng, C. Finn, D. Sadigh, and B. Eysenbach Value flows. In International Conference on Learning Representations, Cited by: §E.1, §E.2, §E.4, Appendix E, §1, §1, §3.1, §4.1, §4. Duan et al. (2022) J. Duan, Y. Guan, S. E. Li, Y. Ren, Q. Sun, and B. Cheng Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems 33 (11), p. 6584–6598. External Links: Document Cited by: §1. Fujimoto et al. (2018) S. Fujimoto, H. van Hoof, and D. Meger Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, p. 1587–1596. Cited by: §1. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations, Cited by: Table 12. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, Cited by: §E.2, §4. Kumar et al. (2019) A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §1, §3.3. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 1179–1191. Cited by: §1, §3.3. Kuznetsov et al. (2020) A. Kuznetsov, P. Shvechikov, A. Grishin, and D. Vetrov Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of the 37th International Conference on Machine Learning, p. 5556–5566. Cited by: §1, §3.3. Li et al. (2026) Z. Li, S. Tang, and N. Azizan Reverse flow matching: a unified framework for online reinforcement learning with diffusion and flow policies. In Proceedings of the 43rd International Conference on Machine Learning, Cited by: §1. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In International Conference on Learning Representations, Cited by: §1, §2.2. Liu et al. (2023) X. Liu, C. Gong, and Q. Liu Flow straight and fast: learning to generate and transfer data with rectified flow. External Links: 2209.03003 Cited by: §1, §2.2. Ma et al. (2021) Y. J. Ma, D. Jayaraman, and O. Bastani Conservative offline distributional reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 34. Cited by: §E.2, §1, §3.3, §4. Park et al. (2025a) S. Park, K. Frans, B. Eysenbach, and S. Levine OGBench: benchmarking offline goal-conditioned rl. In International Conference on Learning Representations, Cited by: §E.1, §4. Park et al. (2025b) S. Park, Q. Li, and S. Levine Flow q-learning. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 48104–48127. Cited by: §E.2, Table 12, Appendix E, §4. Stoica et al. (2026) G. Stoica, S. Paul, M. Wallingford, V. Ramanujan, A. Nori, W. Han, A. Farhadi, R. Krishna, and J. Hoffman Posterior-augmented flow matching. External Links: 2605.00825 Cited by: §1. Tarasov et al. (2023) D. Tarasov, V. Kurenkov, A. Nikulin, and S. Kolesnikov Revisiting the minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §E.2, §4. Xu et al. (2026) B. Xu, Q. Zou, S. Yang, and H. Yan Path-coupled bellman flows for distributional reinforcement learning. In Proceedings of the 43rd International Conference on Machine Learning, Cited by: §B.4, §E.2, §1, §1, §3.1, §4.1, §4. Yang et al. (2026) D. Yang, Y. Zhang, X. Yu, L. Hou, X. Tao, P. Wan, X. Qi, and R. Liao Stable velocity: a variance perspective on flow matching. External Links: 2602.05435 Cited by: §1. Zhong et al. (2025) S. Zhong, S. Ding, H. Diao, X. Wang, K. C. Teh, and B. Peng FlowCritic: bridging value estimation with flow matching in reinforcement learning. External Links: 2510.22686 Cited by: §1. Appendix A Preliminaries A.1 Core Notation and Bellman Endpoint Law Fix a state–action pair and transition outcome τ=(r,s′,d)τ=(r,s ,d), and let μτ _τ be the conditional Bellman endpoint law. The estimator draws a pool whose twin-critic candidates are shared-noise ensemble means: =(Y1,…,YK),Yj∼i.i.d.μτ, =(Y_1,…,Y_K), Y_j i.i.d. _τ, (42) Yj(c) Y_j^(c) =r+γ(1−d)ϕθ¯c(ϵj′∣1,s′,Aj′),c∈1,2, =r+γ(1-d) _ θ_c(ε _j 1,s ,A _j), c∈\1,2\, Yj Y_j =12(Yj(1)+Yj(2)). = 12 (Y_j^(1)+Y_j^(2) ). (43) Here Aj′∼π(⋅∣s′)A _j π(· s ), ϵj′∼(0,1)ε _j (0,1), and j=1,…,Kj=1,…,K. Candidates are independent across j, both target critics share (Aj′,ϵj′)(A _j,ε _j), and μτ=Law(Yj∣τ) _τ=Law(Y_j τ). Symbol Meaning Symbol Meaning τ=(r,s′,d)τ=(r,s ,d) transition outcome for fixed (s,a)(s,a) μτ _τ conditional Bellman endpoint law Y, Y endpoint and pool (Y1,…,YK)(Y_1,…,Y_K) I uniformly sampled anchor index Xt,xtX_t,x_t bridge random variable and realization ϵ,p0ε,p_0 ϵ∼(0,1)ε (0,1) and its density p0p_0 ℓt(x∣y) _t(x y) bridge likelihood ut(x,y)u_t(x,y) endpoint-conditioned bridge direction uju_j candidate direction ut(xt,Yj)u_t(x_t,Y_j) wjw_j posterior anchor responsibility u^K u_K posterior-mean training target vθv_θ learned velocity field vτ⋆v_τ transition-conditional population velocity ESS() (w) 1/∑jwj21/ _jw_j^2 K ensemble-mean endpoint candidates in Y c∈1,2c∈\1,2\ critic index j endpoint-candidate index, 1≤j≤K1≤ j≤ K m action-evaluation sample index, 1≤m≤Mact1≤ m≤ M_act Table 2: Core notation. Uppercase variables denote random quantities and lowercase variables their realizations. Implementation- and proof-specific symbols are defined in the subsections where they are used. A.2 Distributional Bellman Operators Let R and D denote reward and termination random variables. For a fixed policy π, the return satisfies Zπ(s,a) Z^π(s,a) =R+γ(1−D)Zπ(S′,A′), d=R+γ(1-D)Z^π(S ,A ), (44) A′ A ∼π(⋅∣S′). π(· S ). Under standard finite-moment assumptions, the induced operator is a γ-contraction in maximal Wasserstein distance (Bellemare et al., 2017). For d=1d=1, the conditional endpoint law is the point mass at r. For an approximate iteration Zn+1=^nZnZ_n+1= T_nZ_n, we write εK,n=W¯1(^nZn,πZn) _K,n= W_1( T_nZ_n,T^πZ_n); Appendix B.4 propagates this error. A.3 One-Dimensional Return Flows A conditional return flow solves dtϕθ(ϵ∣t,s,a) ddt _θ(ε t,s,a) =vθ(ϕθ(ϵ∣t,s,a),t,s,a), =v_θ\! ( _θ(ε t,s,a),t,s,a ), (45) ϕθ(ϵ∣0,s,a) _θ(ε 0,s,a) =ϵ. =ε. In one dimension, uniqueness of the ODE makes the terminal map monotone and invertible on its image, enabling the inverse-flow CDF used in Appendix B.6. Appendix B Theoretical Analysis This appendix proves the exact finite-pool Rao–Blackwell identity and its variance, optimization, transport, and Bellman-tracking implications. Each estimator analysis fixes the transition, target policy, and target critic, and uses t∈[0,1−δ]t∈[0,1-δ] for δ>0δ>0. B.1 Path Geometry and Population Bridge Target Fix (s,a)(s,a), let (R,S′,D)(R,S ,D) be the transition variables with realization τ=(r,s′,d)τ=(r,s ,d), and set X0=ϵ∼(0,1)X_0=ε (0,1). The endpoint law μτ _τ is defined in Eq. (7). The results condition on this law and hold for any fixed endpoint law satisfying the stated moments. Slice-wise Bellman geometry. For a nonterminal transition (d=0d=0), let the successor path be Xt′=(1−t)X0+tX1′X_t =(1-t)X_0+tX_1 with density pt(⋅∣S′,A′)p_t(· S ,A ) and velocity vt(⋅∣S′,A′)v_t(· S ,A ), define the slice-wise affine Bellman path Xtsl=r+γXt′X_t^sl=r+γ X_t . Conditional on R=rR=r, the change of variables z=r+γξz=r+γξ gives the mixed density and velocity qtslice(z)=1γS′,A′[pt(ξ∣S′,A′)],ξ=z−rγ, q_t^slice(z)= 1γE_S ,A [p_t(ξ S ,A ) ], ξ= z-rγ, (46) utslice(z)=γ[pt(ξ∣S′,A′)vt(ξ∣S′,A′)][pt(ξ∣S′,A′)]. u_t^slice(z)=γ E\! [p_t(ξ S ,A )v_t(ξ S ,A ) ]E\! [p_t(ξ S ,A ) ]. (47) The second identity follows because each branch has X˙tsl=γX˙t′ X_t^sl=γ X_t , so its numerator is the mixed probability flux. At t=0t=0, however, X0sl=r+γX0∼(r,γ2)X_0^sl=r+γ X_0 (r,γ^2), so the slice-wise path does not satisfy the common Gaussian source used by the critic. For a terminal transition, Y=rY=r and μτ=δr _τ= _r. Source-consistent endpoint bridge. The bridge used by BCFM and PA-BCFM instead draws Y=r+γ(1−d)X1′Y=r+γ(1-d)X_1 and sets Xt X_t =(1−t)X0+tY, =(1-t)X_0+tY, X0 X_0 ⟂Y, Y, (48) X0 X_0 ∼(0,1), (0,1), Y Y ∼μτ. _τ. Using the same source noise in the nonterminal successor path gives Xt X_t =(1−t)X0+t(R+γX1′) =(1-t)X_0+t(R+γ X_1 ) (49) =R+γXt′+(1−t)[(1−γ)X0−R], =R+γ X_t +(1-t) [(1-γ)X_0-R ], (50) which proves Eq. (11). The residual vanishes at t=1t=1, but not generally at intermediate times; thus the two paths share the Bellman endpoint without sharing their intermediate marginals or velocity fields. Independence of X0X_0 and Y∼μτY _τ gives p(Xt=x,Y∈dy∣τ)=ℓt(x∣y)μτ(dy).p(X_t=x,Y∈ dy τ)= _t(x y) _τ(dy). (51) Bayes’ rule therefore yields ντ,x(dy) _τ,x(dy) =ℓt(x∣y)μτ(dy)Bτ(x,t), = _t(x y) _τ(dy)B_τ(x,t), (52) vτ⋆(x,t) v_τ (x,t) =∫ut(x,y)ντ,x(dy). = u_t(x,y) _τ,x(dy). Lemma 1 (Population target of BCFM). Fix (τ,x,t)(τ,x,t) with t<1t<1. For the source-consistent bridge in Eq. (48), the conditional BCFM risk ℛτ(u;x,t)=[(u−(Y−X0))2|Xt=x,τ]R_τ(u;x,t)=E [ (u-(Y-X_0) )^2 |X_t=x,τ ] (53) has the unique L2L_2 minimizer u⋆=vτ⋆(x,t)=[Y−x1−t|Xt=x,τ].u =v_τ (x,t)=E [ Y-x1-t |X_t=x,τ ]. (54) Proof. Since Xt=(1−t)X0+tYX_t=(1-t)X_0+tY, conditioning on Xt=xX_t=x gives Y−X0=Y−x1−t=ut(x,Y).Y-X_0= Y-x1-t=u_t(x,Y). (55) Let mτ(x,t)=[ut(x,Y)∣Xt=x,τ]m_τ(x,t)=E[u_t(x,Y) X_t=x,τ]. The conditional squared risk decomposes as ℛτ(u,x,t) _τ(u;x,t) =[(u−mτ+mτ−ut(x,Y))2|Xt=x,τ] =E [ (u-m_τ+m_τ-u_t(x,Y) )^2 |X_t=x,τ ] (56) =(u−mτ(x,t))2 = (u-m_τ(x,t) )^2 (57) +Var[ut(x,Y)|Xt=x,τ]. + [u_t(x,Y) |X_t=x,τ ]. (58) The second term is independent of u, which proves the result. ∎ Transition mixture. Averaging vτ⋆(x,t)v_τ (x,t) under the posterior transition weight proportional to Bτ(x,t)p(τ∣s,a)B_τ(x,t)p(τ s,a) recovers the source-consistent population bridge velocity. Endpoint pooling therefore estimates the transition-conditional velocity, while replay sampling performs the outer Bellman mixture. Degeneracy of practical DCFM. Remark 1. The practical DCFM residual admits degenerate shared fields. For example, vθ(z,t,s,a)=vθ¯(z,t,s,a)=c(t)v_θ(z,t;s,a)=v_ θ(z,t;s,a)=c(t) makes the residual vanish independently of the reward and transition. Thus the residual alone does not identify the Bellman transform; this observation does not assert that every optimization trajectory converges to a constant field. B.2 Finite-Pool Rao–Blackwellization Fix a transition outcome τ and a flow time t<1t<1. Draw =(Y1,…,YK),Yj∼i.i.d.μτ, =(Y_1,…,Y_K), Y_j i.i.d. _τ, (59) I I ∼Unif1,…,K,Xt=(1−t)ϵ+tYI. \1,…,K\, X_t=(1-t)ε+tY_I. where I is independent of the endpoint values and ϵ∼(0,1)ε (0,1). For a realized bridge point x, define uj(x,t) u_j(x,t) =Yj−x1−t,ωj(x,t)=ℓt(x∣Yj), = Y_j-x1-t, _j(x,t)= _t(x Y_j), (60) wj(x,t) w_j(x,t) =ωj(x,t)∑q=1Kωq(x,t). = _j(x,t) _q=1^K _q(x,t). The single-endpoint BCFM target is UI=uI(Xt,t)=YI−ϵU_I=u_I(X_t,t)=Y_I-ε, whereas PA-BCFM uses u^K(Xt,t)=∑j=1Kwj(Xt,t)uj(Xt,t). u_K(X_t,t)= _j=1^Kw_j(X_t,t)u_j(X_t,t). (61) Posterior anchor index. Conditioned on (Xt=x,,τ,t)(X_t=x,Y,τ,t), the density of x under the hypothesis I=jI=j is ωj(x,t) _j(x,t). Since the prior over I is uniform, Bayes’ rule gives Pr(I=j∣Xt=x,,τ,t)=wj(x,t). (I=j X_t=x,Y,τ,t)=w_j(x,t). (62) Consequently, u^K=[UI|Xt,,τ,t]. u_K=E\! [U_I |X_t,Y,τ,t ]. (63) Taking the conditional expectation once more and using Lemma 1 yields [u^K∣Xt,τ,t] [ u_K X_t,τ,t] =[UI∣Xt,τ,t] =E[U_I X_t,τ,t] (64) =vτ⋆(Xt,t),for every K≥1. =v_τ (X_t,t), every K≥ 1. At K=1K=1, u^1=UI=YI−ϵ u_1=U_I=Y_I-ε, proving exact reduction to BCFM. The outer transition mixture also gives [u^K∣Xt,s,a,t]=v⋆(Xt,t∣s,a)E[ u_K X_t,s,a,t]=v (X_t,t s,a). Hence, for any square-integrable prediction u(Xt,s,a,t)u(X_t,s,a,t), [(u−u^K)2] [(u- u_K)^2 ] =[(u−v⋆)2]+[(u^K−v⋆)2]. =E [(u-v )^2 ]+E [( u_K-v )^2 ]. (65) The second term does not depend on u, so every finite pool preserves the population minimizer. Exact finite-pool variance reduction. Applying conditional total variance to Eq. (63) gives Var(UI∣Xt,τ,t) (U_I X_t,τ,t) =Var(u^K∣Xt,τ,t) = ( u_K X_t,τ,t) +[∑j=1Kwj(uj−u^K)2|Xt,τ,t]. +E\! [ _j=1^Kw_j(u_j- u_K)^2 |X_t,τ,t ]. (66) Therefore Var(u^K∣Xt,τ,t)≤Var(UI∣Xt,τ,t) ( u_K X_t,τ,t)≤ (U_I X_t,τ,t) (67) for every finite K. The reduction is strict whenever the posterior over anchor indices assigns positive mass to endpoints with different velocities. Relation to leave-one-out SNIS. For asymptotic analysis, condition on the selected anchor and relabel it as Y0Y_0. Then Y0∼ντ,xY_0 _τ,x, while Y1,…,YK−1Y_1,…,Y_K-1 are i.i.d. from μτ _τ. Define u~K−1=∑j=1K−1ωjuj∑j=1K−1ωj. u_K-1= _j=1^K-1 _ju_j _j=1^K-1 _j. (68) Direct algebra yields u^K−u~K−1 u_K- u_K-1 =α0,K(u0−u~K−1), = _0,K (u_0- u_K-1 ), (69) α0,K _0,K =ω0∑j=0K−1ωj. = _0 _j=0^K-1 _j. Under a positive normalizer and the stated moment conditions, α0,K=Op(K−1) _0,K=O_p(K^-1), so u^K−u~K−1=Op(K−1). u_K- u_K-1=O_p(K^-1). (70) The practical estimator is therefore consistent and has the same first-order asymptotic distribution as standard SNIS. In finite samples, however, Eq. (64) is the sharper statement: the anchor correction exactly cancels the ratio bias in expectation under the uniform anchor-sampling protocol. B.3 Asymptotic Variance, ESS, and Gradient Covariance Ratio asymptotics. Fix (x,t,τ)(x,t,τ), and write ω(Y)=ℓt(x∣Y)ω(Y)= _t(x Y), u(Y)=ut(x,Y)u(Y)=u_t(x,Y), Bτ(x,t)=μτ[ω(Y)]B_τ(x,t)=E_ _τ[ω(Y)], and vτ⋆=[ω(Y)u(Y)]/Bτ(x,t)v_τ =E[ω(Y)u(Y)]/B_τ(x,t). The multivariate central limit theorem for (ω(Y)u(Y),ω(Y))(ω(Y)u(Y),ω(Y)), followed by the delta method, gives influence variable ψ(Y)=ω(Y)(u(Y)−vτ⋆)Bτ(x,t)ψ(Y)= ω(Y)(u(Y)-v_τ )B_τ(x,t) (71) and asymptotic variance στ2(x,t)=μτ[ω(Y)2(u(Y)−vτ⋆)2](μτ[ω(Y)])2. _τ^2(x,t)= E_ _τ [ω(Y)^2(u(Y)-v_τ )^2 ] (E_ _τ[ω(Y)] )^2. (72) Because the anchor correction in Eq. (70) is op(K−1/2)o_p(K^-1/2), Slutsky’s theorem gives K(u^K−vτ⋆) K( u_K-v_τ ) ⇒(0,στ2(x,t)), (0, _τ^2(x,t)), (73) Var(u^K∣x,t,τ) ( u_K x,t,τ) =στ2(x,t)K+o(K−1). = _τ^2(x,t)K+o(K^-1). Effective sample size. The realized effective sample size is ESS()=(∑jωj)2∑jωj2=1∑jwj2. (w)= ( _j _j)^2 _j _j^2= 1 _jw_j^2. (74) The anchor contributes only one term, so the strong law applied to the nonanchor endpoints yields ESS()K→a.s.χτ(x,t)=(μτ[ω(Y)])2μτ[ω(Y)2]. (w)K a.s. _τ(x,t)= (E_ _τ[ω(Y)] )^2E_ _τ[ω(Y)^2]. (75) If |u(Y)−vτ⋆|≤Cu,τ(x,t)|u(Y)-v_τ |≤ C_u,τ(x,t), then στ2(x,t) _τ^2(x,t) ≤Cu,τ(x,t)2χτ(x,t), ≤ C_u,τ(x,t)^2 _τ(x,t), (76) Var(u^K∣x,t,τ) ( u_K x,t,τ) ≤Cu,τ(x,t)2Kχτ(x,t)+o(K−1). ≤ C_u,τ(x,t)^2K _τ(x,t)+o(K^-1). Thus ESS measures posterior-weight concentration and controls an asymptotic variance bound; it is not an unconditional exact variance identity. Gradient Rao–Blackwellization. Fix (Xt,τ,t,θ)(X_t,τ,t,θ), and let Jθ=∇θvθ(Xt,t,s,a)J_θ= _θv_θ(X_t,t;s,a). The single-endpoint BCFM gradient and PA-BCFM gradient are g1=2Jθ(vθ−UI),gK=2Jθ(vθ−u^K).g_1=2J_θ (v_θ-U_I ), g_K=2J_θ (v_θ- u_K ). (77) Equation (63) implies gK g_K =[g1∣Xt,,τ,t,θ], =E[g_1 X_t,Y,τ,t,θ], (78) Cov(gK∣Xt,τ,t,θ) (g_K X_t,τ,t,θ) ⪯Cov(g1∣Xt,τ,t,θ). (g_1 X_t,τ,t,θ). Moreover, Cov(gK∣x,t,τ,θ)=4στ2(x,t)KJθJθ⊤+o(K−1). (g_K x,t,τ,θ)= 4 _τ^2(x,t)KJ_θJ_θ +o(K^-1). (79) This result requires no conditional independence assumption among candidate gradients. B.4 Optimization, Transport, and Bellman Tracking Relation to PCBF. PCBF adds a learned successor-field control variate with coefficient λcv _cv, so its population field differs from the source-consistent target by approximation and cross-conditioning terms (Xu et al., 2026). PA-BCFM instead preserves the exact population target for every finite K; its pool-dependent error is variance rather than target bias. Optimization effect of the endpoint pool. Define the population field risk ℱ(θ) (θ) =12[|vθ(Xt,t,s,a)−v⋆(Xt,t,s,a)|2], = 12E\! [|v_θ(X_t,t;s,a)-v (X_t,t;s,a)|^2 ], (80) ℱ⋆ _ =infθℱ(θ). = _θF(θ). Exact finite-pool unbiasedness implies that the PA-BCFM stochastic gradient is unbiased for ∇ℱ . Under uniform integrability of the conditional variance bound in Appendix B.3, total variance yields constants σbase2 _base^2 and CpoolC_pool such that a minibatch of size BmbB_mb satisfies [‖g^K,Bmb−∇ℱ(θ)‖2]≤1Bmb(σbase2+CpoolK).E\! [\| g_K,B_mb- (θ)\|^2 ]≤ 1B_mb ( _base^2+ C_poolK ). (81) The first term collects transition, state–action, and flow-time randomness; the second is the endpoint-pool contribution. Assume ℱF is LℱL_F-smooth and satisfies the Polyak–Łojasiewicz condition with constant μPL _PL, using 12‖∇ℱ(θ)‖22≥μPL(ℱ(θ)−ℱ⋆) 12\| (θ)\|_2^2≥ _PL(F(θ)-F_ ). For θn+1=θn−ηoptg^K,Bmb _n+1= _n- _opt g_K,B_mb with ηopt≤1/Lℱ _opt≤ 1/L_F, smoothness and unbiasedness give [ℱ(θn+1)−ℱ⋆∣θn] [F( _n+1)-F_ _n] ≤(1−μPLηopt)(ℱ(θn)−ℱ⋆) ≤(1- _PL _opt)(F( _n)-F_ ) +Lℱηopt22Bmb(σbase2+CpoolK). + L_F _opt^22B_mb ( _base^2+ C_poolK ). (82) Unrolling the recursion proves [ℱ(θT)−ℱ⋆] [F( _T)-F_ ] ≤(1−μPLηopt)T(ℱ(θ0)−ℱ⋆) ≤(1- _PL _opt)^T (F( _0)-F_ ) +Lℱηopt2μPLBmb(σbase2+CpoolK), + L_F _opt2 _PLB_mb ( _base^2+ C_poolK ), (83) which gives the optimization statement in Corollary 1. Velocity-to-endpoint stability. Couple the ideal and learned ODEs by their common source and let et=X^t−Xte_t= X_t-X_t. If the learned field is LvL_v-Lipschitz in its state argument, then Grönwall’s inequality gives |e1|≤∫01eLv(1−t)|vθT(Xt,t)−v⋆(Xt,t)|t.|e_1|≤ _0^1e^L_v(1-t)|v_ _T(X_t,t)-v (X_t,t)|dt. (84) Let ZθT,1exactZ_ _T,1^exact denote the terminal law obtained by exact integration of the learned ODE and ZθT,1(L)Z_ _T,1^(L) the law produced by the L-step numerical solver. We write δode=[W1(ZθT,1(L),ZθT,1exact)] _ode=E\! [W_1\! (Z_ _T,1^(L),Z_ _T,1^exact ) ] (85) for the expected numerical-integration contribution. Cauchy–Schwarz, the common-source coupling, Jensen’s inequality, and the triangle inequality imply W1(ZθT,1(L),Z1⋆) _1(Z_ _T,1^(L),Z_1 ) ≤CLv2ℱ(θT)+δode, ≤ C_L_v 2EF( _T)+ _ode, (86) CLv2 C_L_v^2 =∫01e2Lv(1−t)dt. = _0^1e^2L_v(1-t)dt. Substituting Eq. (83) into Eq. (86) proves the transport statement in Corollary 1. Thus the endpoint-pool contribution is O(K−1)O(K^-1) in the optimization floor and O(K−1/2)O(K^-1/2) in the terminal Wasserstein bound. Bellman tracking. Let εK,n _K,n denote the one-step terminal-distribution error, including Eq. (86), numerical integration error, and any target-law mismatch. Distributional Bellman contraction gives, for fixed policy, dn+1≤γdn+εK,n,lim supndn≤lim supnεK,n1−γ.d_n+1≤γ d_n+ _K,n, _nd_n≤ _n _K,n1-γ. (87) For a changing policy, one additional triangle inequality yields dn+1≤γdn+εK,n+Δnπ,d_n+1≤γ d_n+ _K,n+ _n^π, (88) where Δnπ _n^π measures the change between consecutive policy return laws. This is a tracking result, not a global actor–critic convergence claim. B.5 Conservative Endpoint Reweighting Let g:ℝ→[0,∞)g:R→[0,∞) be a fixed distortion with 0<∫g(y)μτ(y)<∞0< g(y) _τ(dy)<∞. The distorted endpoint law is μτ,g(dy)=g(y)μτ(dy)∫g(y~)μτ(y~). _τ,g(dy)= g(y) _τ(dy) g( y) _τ(d y). (89) After observing Xt=xX_t=x, its bridge posterior is proportional to ℓg(x∣y)μτ(dy) _g(x y) _τ(dy), where ℓg(x∣y)=ℓt(x∣y)g(y). _g(x y)= _t(x y)g(y). (90) For g≡1g≡ 1, the weights coincide with the posterior anchor probabilities in Eq. (62), giving the exact finite-pool Rao–Blackwell identity. For nonconstant g, the weights intentionally define a different target and are not the posterior of the uniformly selected anchor. The practical distorted weights and target are wjg=ℓt(x∣Yj)g(Yj)∑ℓ=1Kℓt(x∣Yℓ)g(Yℓ),u^Kg=∑j=1Kwjgut(x,Yj).w_j^g= _t(x Y_j)g(Y_j) _ =1^K _t(x Y_ )g(Y_ ), u_K^g= _j=1^Kw_j^gu_t(x,Y_j). (91) Proposition 1 (Consistency of the distorted endpoint estimator). Under the moment and positive-normalizer assumptions of Theorem 1, condition on the selected anchor and relabel it as Y0Y_0. With unnormalized weight ℓg _g, u^Kg−u~K−1g u_K^g- u_K-1^g =α0,Kg(u0−u~K−1g), = _0,K^g (u_0- u_K-1^g ), (92) α0,Kg _0,K^g =ℓg(x∣Y0)∑j=0K−1ℓg(x∣Yj). = _g(x Y_0) _j=0^K-1 _g(x Y_j). Consequently, u^Kg−u~K−1g=Op(K−1) u_K^g- u_K-1^g=O_p(K^-1), u^Kg→vτ,g⋆(x,t)=μτ[ℓg(x∣Y)ut(x,Y)]μτ[ℓg(x∣Y)], u_K^g pv_τ,g (x,t)= E_ _τ[ _g(x Y)u_t(x,Y)]E_ _τ[ _g(x Y)], (93) and the practical estimator has the same first-order SNIS variance as the leave-one-out ratio. For nonconstant g, its finite-pool ratio is generally biased by O(K−1)O(K^-1); the exact unbiasedness in Eq. (64) is specific to the undistorted objective. Proof. Equation (92) follows from the same ratio identity as Eq. (69). The nonanchor samples remain independent draws from μτ _τ, so the strong law, ratio expansion, and central limit theorem apply to ℓg _g. The anchor contributes one additional term and changes the leave-one-out ratio by only Op(K−1)O_p(K^-1). Unlike the case g≡1g≡ 1, no tower-property cancellation makes this distorted ratio exactly unbiased at finite K. ∎ Hard truncation. For a population truncation level ρ∈[0,1)ρ∈[0,1), let q1−ρq_1-ρ be the (1−ρ)(1-ρ)-quantile of μτ _τ and define gρhard(y)=y≤q1−ρg_ρ^hard(y)=1\y≤ q_1-ρ\. Replacing the population quantile by the empirical pool quantile is consistent under continuity of the endpoint CDF at the truncation point. The soft tilt gη(y)=exp(−ηy)g_η(y)= (-η y) is covered directly by Proposition 1. Proposition 2 (Distorted Bellman contraction and pessimistic order). Let gD_g be the population distortion and suppose Wp(gμ,gν)≤LgWp(μ,ν).W_p(D_gμ,D_gν)≤ L_gW_p(μ,ν). (94) Then W¯p(gπZ1,gπZ2)≤γLgW¯p(Z1,Z2). W_p(T_g^πZ_1,T_g^πZ_2)≤γ L_g W_p(Z_1,Z_2). (95) Hence γLg<1γ L_g<1 gives a unique fixed point. If g is nonincreasing, then gμ⪯FSDμD_gμ _FSDμ. If the Bellman operator is monotone and both distorted and undistorted iterations converge, their fixed points satisfy Zgπ⪯FSDZπZ_g^π _FSDZ^π. Proof. The Wasserstein bound follows from Bellman affine scaling and the assumed Lipschitz property of gD_g. For nonincreasing g, the standard covariance inequality implies first-order stochastic dominance. Bellman monotonicity then preserves this order along both convergent iterations. ∎ B.6 Adaptive Distortion and Reliability Diagnostics Scope of the adaptive schedule. Proposition 2 applies to a fixed population distortion g. The practical adaptive rule instead uses DpoolD_pool, which depends on the training transition, bridge point, and sampled endpoint pool, and is treated with stop-gradient within each critic update. Freezing this realized coefficient for differentiation does not define a common distorted Bellman operator across endpoint laws or training iterations. We therefore use the adaptive schedule only as a practical extension and make no operator-level contraction claim for it. This training-time quantity is distinct from the state–action diagnostic Dtwin(s,a)D_twin(s,a). Inverse-flow support probability. For a one-dimensional ODE with unique solutions, each target terminal map ϵ↦ϕθ¯c(ϵ∣1,s,a)ε _ θ_c(ε 1,s,a) is order preserving. For c∈1,2c∈\1,2\, define F^c(z∣s,a) F_c(z s,a) =Φ(ϕθ¯c−1(z∣1,s,a)), = \! ( _ θ_c^-1(z 1,s,a) ), (96) F^ens(z∣s,a) F_ens(z s,a) =12∑c=12F^c(z∣s,a). = 12 _c=1^2 F_c(z s,a). For the fixed empirical upper-return threshold qhiq_hi^D, the reported unsupported mass is Pout(s,a)=1−F^ens(qhi∣s,a).P_out(s,a)=1- F_ens(q_hi^D s,a). (97) This is a model-implied tail probability rather than an OOD certificate. Statistical and dynamical conditions. All finite-pool, asymptotic, optimization, and ODE conclusions hold under the assumptions stated in their respective propositions; none imply global actor–critic convergence. Appendix C Practical Algorithm This appendix gives the full critic update, the optional endpoint-pool extensions, and the policy-improvement procedure used in the experiments. C.1 PA-BCFM Critic Update Endpoint construction. For each transition (s,a,r,s′,d)(s,a,r,s ,d) in a minibatch of size BmbB_mb, write τ=(r,s′,d)τ=(r,s ,d) and draw Aj′∼π(⋅∣s′),ϵj′∼(0,1),j=1,…,K,A _j π(· s ), ε _j (0,1), j=1,…,K, (98) and generate common Bellman endpoints from the ensemble-mean target return flow, Yj(c) Y_j^(c) =r+γ(1−d)ϕθ¯c(ϵj′∣1,s′,Aj′), =r+γ(1-d) _ θ_c(ε _j 1,s ,A _j), (99) Yj Y_j =12(Yj(1)+Yj(2)),c∈1,2. = 12 (Y_j^(1)+Y_j^(2) ), c∈\1,2\. The two target critics share successor actions and return-noise samples. Their endpoint outputs are averaged into the common endpoint YjY_j, and the resulting PA-BCFM target is used for both critic losses. Both target flows are evaluated vectorially on the K shared samples and reuse the successor-state encoding. We sample I∼Unif1,…,KI \1,…,K\, independently of endpoint values, then draw ϵ∼(0,1)ε (0,1) and t∼[0,1−ϵt]t [0,1- _t]. The bridge point is xt=(1−t)ϵ+tYI.x_t=(1-t)ε+tY_I. (100) Posterior target. For a distortion g, define the log weight Λj=−(xt−tYj)22(1−t)2+logg(Yj). _j=- (x_t-tY_j)^22(1-t)^2+ g(Y_j). (101) The default PA-BCFM objective uses g≡1g≡ 1. We normalize with log-sum-exp, wj=exp(Λj−logsumexpℓ=1KΛℓ),w_j= \! ( _j-logsumexp_ =1^K _ ), (102) and construct the candidate directions and posterior target uj=Yj−xt1−t,u^K=∑j=1Kwjuj.u_j= Y_j-x_t1-t, u_K= _j=1^Kw_ju_j. (103) The endpoints, weights, and posterior target are stop-gradient quantities. Critic and target updates. Each member of the twin critic minimizes ℒc=[‖vθc(xt,t,s,a)−sg[u^K]‖22],c∈1,2.L_c=E [ \|v_ _c(x_t,t;s,a)-sg[ u_K] \|_2^2 ], c∈\1,2\. (104) The two online critics are parameterized and optimized separately despite sharing this target; their individual target flows are retained for the training-time pool signal and direct state–action diagnostics in Appendix C.2. After the gradient update, the corresponding target parameters are updated by Polyak averaging, θ¯c←τtgtθc+(1−τtgt)θ¯c. θ_c← _tgt _c+(1- _tgt) θ_c. (105) Gradients do not pass through the target flow, posterior construction, diagnostics, or target update. Algorithm 1 Posterior-Averaged Bellman Flow Matching (PA-BFM) 0: replay buffer D, twin critics and targets, policy π, behavior proposal πβ _β, endpoint count K, optional endpoint distortion g, policy-training schedule P 1: repeat 2: Sample BmbB_mb transitions (s,a,r,s′,d)(s,a,r,s ,d) from D 3: Draw K successor actions and target-flow noises per transition 4: Evaluate both targets and set Yj=(Yj(1)+Yj(2))/2Y_j=(Y_j^(1)+Y_j^(2))/2 5: Sample I uniformly and construct xt=(1−t)ϵ+tYIx_t=(1-t)ε+tY_I 6: Form stable posterior logits, weights, and posterior target u^K u_K 7: Update both critics using Eq. (104) 8: Polyak-update both target critics 9: if online interaction is active then 10: Collect transitions and append them to D 11: end if 12: if one-step policy training is enabled by P then 13: Update the behavior-regularized one-step policy with Eq. (114) 14: end if 15: until the training budget is exhausted 16: return twin return critics, behavior proposal, and optional one-step policy C.2 Conservative Reweighting and Diagnostics Fixed endpoint distortions. The main PA-BCFM experiments use the undistorted target g≡1g≡ 1. The reported conservative variant uses the fixed soft tilt gη(Yj)=exp(−ηYj),η≥0.g_η(Y_j)= (-η Y_j), η≥ 0. (106) Hard upper-tail truncation is implemented by zeroing candidates above the empirical pool quantile; it is not used in the reported experiments. The exact finite-pool Rao–Blackwell identity applies to g≡1g≡ 1; nonconstant g deliberately defines a distorted target. Training-time pool disagreement. To separate critic disagreement from candidate-sampling noise, both target critics are evaluated on the same successor actions and return noises: Yj(c)=r+γ(1−d)ϕθ¯c(ϵj′∣1,s′,Aj′),c∈1,2.Y_j^(c)=r+γ(1-d) _ θ_c(ε _j 1,s ,A _j), c∈\1,2\. (107) Let Yj=(Yj(1)+Yj(2))/2Y_j=(Y_j^(1)+Y_j^(2))/2, and compute the undistorted posterior weights wj0w_j^0 from YjY_j. The disagreement score is Dpool(τ,xt,t,)=∑j=1Kwj0(Yj(1)−Yj(2))2.D_pool(τ,x_t,t,Y)= _j=1^Kw_j^0 (Y_j^(1)-Y_j^(2) )^2. (108) Adaptive distortion. For the reported diagnostic experiment, return-valued quantities are scaled by the task-specific Δe=q.95,e−q.05,e _e=q_.95^D,e-q_.05^D,e return-to-go range, and normalized disagreement is clipped to [0,1][0,1]. Training-time pool disagreement modulates the soft tilt as η(τ,xt,t) η(τ,x_t,t) =η0sg[clip(Dpool(τ,xt,t,)Δe,0,1)], = _0sg\! [clip\! ( D_pool(τ,x_t,t,Y) _e,0,1 ) ], (109) gτ,xt,t(Yj) g_τ,x_t,t(Y_j) =exp[−η(τ,xt,t)YjΔe]. = \! [-η(τ,x_t,t) Y_j _e ]. The stop-gradient makes the diagnostic a target-construction signal rather than a second gradient path through the critic. This sample-dependent rule is not covered by the fixed-distortion contraction result. State–action reliability diagnostics. At a queried (s,a)(s,a), shared Gaussian samples isolate disagreement between the twin target return flows: Zm(c)(s,a) Z_m^(c)(s,a) =ϕθ¯c(ξm∣1,s,a),c∈1,2, = _ θ_c( _m 1,s,a), c∈\1,2\, (110) ξm _m ∼(0,1),m=1,…,Mdiag, (0,1), m=1,…,M_diag, Dtwin(s,a) D_twin(s,a) =1Mdiag∑m=1Mdiag(Zm(1)(s,a)−Zm(2)(s,a))2. = 1M_diag _m=1^M_diag (Z_m^(1)(s,a)-Z_m^(2)(s,a) )^2. For each one-dimensional return flow, inversion gives the critic CDF; the reported unsupported mass uses their ensemble average: F^c(b∣s,a) F_c(b s,a) =Φ(ϕθ¯c−1(b∣1,s,a)), = \! ( _ θ_c^-1(b 1,s,a) ), (111) F^ens(b∣s,a) F_ens(b s,a) =12∑c=12F^c(b∣s,a), = 12 _c=1^2 F_c(b s,a), Pout(s,a) P_out(s,a) =1−F^ens(qhi∣s,a). =1- F_ens(q_hi^D s,a). The threshold qhiq_hi^D is a dataset-derived upper-return threshold selected once before diagnostic evaluation and then held fixed. C.3 Policy Extraction and Shared Policy Training All main experiments rank candidate actions by predicted mean return. Where policy training is enabled by P, actions are distilled into a one-step policy using the shared behavior-regularized objective below. This policy-side procedure is matched across PA-BFM, Value Flows, and PCBF. Distributional action evaluation. For a state s, the behavior flow proposes N actions ai∼πβ(⋅∣s)a_i _β(· s). Each action is evaluated with MactM_act shared return noises under the twin return flows: zim(c) z_im^(c) =ϕθc(ϵim∣1,s,ai), = _ _c( _im 1,s,a_i), (112) ϵim _im ∼(0,1),m=1,…,Mact,c∈1,2. (0,1), subarraycm=1,…,M_act,\\ c∈\1,2\ subarray. The mean score and selected action are μ^i μ_i =12Mact∑c=12∑m=1Mactzim(c), = 12M_act _c=1^2 _m=1^M_actz_im^(c), (113) a⋆ a =argmax1≤i≤Nμ^i. = *arg\,max_1≤ i≤ N μ_i. With the default Mact=1M_act=1 in Table 12, μ^i μ_i is a single-sample Monte Carlo estimate of the critic mean. Shared one-step policy update. When policy training is enabled by P, a one-step policy πψ _ψ, with parameters ψ, is trained against the frozen critic with behavior regularization: aψ a_ψ =πψ(s,ζ), = _ψ(s,ζ), (114) ℒpolicy(ψ) _policy(ψ) =s∼ζ∼(0,da)[−μ^(s,aψ) =E_ subarraycs \\ ζ (0,I_d_a) subarray [- μ(s,a_ψ) +αBC‖aψ−πβ(s,ζ)‖22]. + _BC \|a_ψ- _β(s,ζ) \|_2^2 ]. Here μ^(s,a) μ(s,a) is the twin-flow sample mean above evaluated at action a, ζ is action-generation noise, daI_d_a is the identity covariance in action dimension, and αBC≥0 _BC≥ 0 controls behavior regularization. Critic parameters are fixed during this actor update. C.4 Implementation and Complexity Numerical implementation. We restrict flow time to t≤1−ϵt≤ 1- _t, compute posterior weights with log-sum-exp, and reuse the successor-state representation across all K endpoints. Target-flow solves are vectorized and run without gradient storage. The same ODE solver and precision policy are used across estimator variants. Complexity. With TflowT_flow target-flow solver steps, endpoint construction costs O(BmbKTflow)O(B_mbKT_flow) network work and O(BmbK)O(B_mbK) posterior arithmetic and storage. Policy extraction requires O(NMactTflow)O(NM_actT_flow) return-flow evaluations per state. End-to-end wall time is reported separately from these asymptotic counts. Appendix D Additional Experiments D.1 Controlled Estimator Studies Controlled processes and exact references. We use a one-state Bernoulli-return process to fix the controlled-study configuration and evaluate the frozen configuration on Solitaire Dice and a discrete Monte Carlo chain. All three processes admit exact return laws, so the finite-pool posterior target can be evaluated without a learned target critic. Table 3 summarizes the process definitions. Process Configuration Bernoulli return One-state continuing MRP; R∼Bernoulli(0.5)R (0.5); γ=0.5γ=0.5; exact return law Z∼Unif(0,2)Z (0,2). Solitaire Dice A fair six-sided die is repeatedly rolled. Rolls 22–66 produce reward 11 and continue; roll 11 produces reward 00 and terminates; γ=0.9γ=0.9. Discrete-MC Twenty-state nearest-neighbor chain with absorbing terminal states 00 and 1919. States 11–1818 use the fixed cosine–exponential nearest-neighbor transition weights of the controlled-process configuration. Transitions to nonterminal states produce reward 11, transitions into a terminal state produce reward 00, and γ=0.95γ=0.95. Table 3: Controlled-process settings. The Bernoulli process is used to fix the controlled-study configuration; Solitaire Dice and Discrete-MC are used for the reported evaluation. For a controlled conditional endpoint law μ, the exact posterior Bellman velocity at bridge point x and flow time t is v⋆(x,t)=Y∼μ[ℓt(x∣Y)ut(x,Y)]Y∼μ[ℓt(x∣Y)],v (x,t)= E_Y μ [ _t(x Y)u_t(x,Y) ]E_Y μ [ _t(x Y) ], (115) where ℓt(x∣Y) _t(x Y) is the Gaussian bridge likelihood. The expectations are evaluated from the exact endpoint laws. For each reported process, the oracle study uses Nq=4096N_q=4096 bridge queries and Nrep=512N_rep=512 endpoint-pool replicates per query. For query q, we sample and fix (τq,tq,xq)( _q,t_q,x_q) from the exact bridge marginal and compute the oracle velocity vq⋆v_q . In every replicate, the relabeled anchor is resampled from the exact bridge posterior Yanc,q(r)∼ντq,xq(dy)∝ℓtq(xq∣y)μτq(dy).Y_anc,q^(r) _ _q,x_q(dy) _t_q(x_q y)\, _ _q(dy). (116) The remaining K−1K-1 endpoints are sampled independently from μτq _ _q. This is the conditional law of the uniform-anchor estimator given the fixed bridge query, so the K=1K=1 target remains random and provides a nonzero BCFM reference. Uniform aggregation and PA-BCFM use matched candidate pools within each comparison. Process-level results give equal weight to Solitaire Dice and Discrete-MC. Confidence intervals use a process-stratified query-cluster bootstrap, retaining all replicates of each resampled query together. Fixed-query conditional target estimation. We evaluate the anchor-in-pool estimator with K∈1,2,4,8,16K∈\1,2,4,8,16\. For query q and endpoint-pool replicate r, let u^K,q(r) u_K,q^(r) denote the finite-pool target. We report the conditional mean-squared error MSEK=1NqNrep∑q=1Nq∑r=1Nrep(u^K,q(r)−vq⋆)2.MSE_K= 1N_qN_rep _q=1^N_q _r=1^N_rep ( u_K,q^(r)-v _q )^2. (117) K Signed bias [95% CI] Cond. variance MSE [95% CI] 1 0.004[−0.006,0.014]0.004\,[-0.006,0.014] 5.0407 5.0447[4.86,5.23]5.0447\,[4.86,5.23] 2 −0.003[−0.011,0.005]-0.003\,[-0.011,0.005] 3.0301 3.0331[2.91,3.16]3.0331\,[2.91,3.16] 4 0.002[−0.005,0.009]0.002\,[-0.005,0.009] 1.7790 1.7810[1.70,1.86]1.7810\,[1.70,1.86] 8 −0.001[−0.006,0.004]-0.001\,[-0.006,0.004] 1.0209 1.0219[0.97,1.08]1.0219\,[0.97,1.08] 16 0.001[−0.003,0.005]0.001\,[-0.003,0.005] 0.5693 0.5703[0.54,0.61]0.5703\,[0.54,0.61] Table 4: Fixed-query conditional target estimation. Bias is the mean signed error; variance is the mean within-query conditional variance. Brackets give 95% process-stratified query-cluster bootstrap confidence intervals. Lower MSE is better. Process Estimator K=1K=1 K=2K=2 K=4K=4 K=8K=8 K=16K=16 Solitaire Uniform 1.000[1.000,1.000] subarrayc1.000\\ [1.000,1.000] subarray .499[.470,.529] subarrayc.499\\ [.470,.529] subarray .227[.208,.247] subarrayc.227\\ [.208,.247] subarray .114[.102,.126] subarrayc.114\\ [.102,.126] subarray .058[.051,.065] subarrayc.058\\ [.051,.065] subarray Solitaire PA-BCFM 1.000[1.000,1.000] subarrayc1.000\\ [1.000,1.000] subarray .438[.409,.468] subarrayc.438\\ [.409,.468] subarray .179[.164,.195] subarrayc.179\\ [.164,.195] subarray .081[.073,.090] subarrayc.081\\ [.073,.090] subarray .038[.034,.043] subarrayc.038\\ [.034,.043] subarray Discrete-MC Uniform 1.000[1.000,1.000] subarrayc1.000\\ [1.000,1.000] subarray .460[.432,.489] subarrayc.460\\ [.432,.489] subarray .195[.179,.212] subarrayc.195\\ [.179,.212] subarray .098[.089,.108] subarrayc.098\\ [.089,.108] subarray .052[.046,.058] subarrayc.052\\ [.046,.058] subarray Discrete-MC PA-BCFM 1.000[1.000,1.000] subarrayc1.000\\ [1.000,1.000] subarray .349[.322,.377] subarrayc.349\\ [.322,.377] subarray .090[.080,.100] subarrayc.090\\ [.080,.100] subarray .020[.017,.023] subarrayc.020\\ [.017,.023] subarray .010[.0088,.0122] subarrayc.010\\ [.0088,.0122] subarray Table 5: Target-induced gradient trace-covariance ratio. Values are normalized by the corresponding random K=1K=1 BCFM reference; brackets give 95% query-cluster bootstrap confidence intervals. Lower is better. Gradient variability and posterior concentration. For the gradient study, critic parameters and the bridge query (τ,t,xt)(τ,t,x_t) are fixed, while every replicate independently resamples the posterior anchor and the remaining endpoint pool using the same conditional protocol as above. Hence the K=1K=1 target and gradient remain random. For gradient replicate grg_r, we compute the trace covariance GVar=tr(Cov^[g])=1Rg∑r=1Rg‖gr−g¯‖22.GVar=tr ( Cov[g] )= 1R_g _r=1^R_g \|g_r- g \|_2^2. (118) For each query, we compute the trace covariance across Rg=256R_g=256 replicates. The reported process-level ratio divides the aggregate query-level covariance trace at pool size K by the corresponding K=1K=1 aggregate. These ratios measure fixed-query, target-induced gradient variability rather than the unconditional variance of full SGD updates. Confidence intervals use the same query-cluster bootstrap. For PA-BCFM, posterior concentration is summarized by the normalized effective sample size ESS()K=1K∑j=1Kwj2. (w)K= 1K _j=1^Kw_j^2. (119) ESS is used as a within-estimator measure of posterior concentration; cross-estimator variance is assessed directly by Table 5. Learned return-law fidelity. Finally, we train BCFM, uniform endpoint aggregation, Value Flows, PCBF, and PA-BCFM using the same critic architecture, optimizer, transition batches, and 50,00050,000-update budget. Each configuration is trained with eight seeds, and return-law fidelity is evaluated using 50,00050,000 return samples per evaluated state–action pair. Solitaire Dice is evaluated before the first roll. For Discrete-MC, the process-level fidelity value aggregates the configured statewise evaluation grid. Method Solitaire Discrete-MC Avg. BCFM (K=1K=1) 6.644[6.31,6.98] subarrayc6.644\\ [6.31,6.98] subarray 1.454[1.36,1.55] subarrayc1.454\\ [1.36,1.55] subarray 4.049[3.86,4.24] subarrayc4.049\\ [3.86,4.24] subarray Uniform (K=8K=8) 6.252[5.92,6.58] subarrayc6.252\\ [5.92,6.58] subarray 3.630[3.38,3.88] subarrayc3.630\\ [3.38,3.88] subarray 4.941[4.73,5.15] subarrayc4.941\\ [4.73,5.15] subarray Value Flows 6.650[6.25,7.05] subarrayc6.650\\ [6.25,7.05] subarray 4.202[3.91,4.49] subarrayc4.202\\ [3.91,4.49] subarray 5.426[5.18,5.67] subarrayc5.426\\ [5.18,5.67] subarray PCBF (λcv=0.3 _cv=0.3) 6.020[5.76,6.28] subarrayc6.020\\ [5.76,6.28] subarray 1.227[1.15,1.30] subarrayc1.227\\ [1.15,1.30] subarray 3.623[3.48,3.77] subarrayc3.623\\ [3.48,3.77] subarray PA-BCFM (K=8K=8) 5.891[5.65,6.13] subarrayc5.891\\ [5.65,6.13] subarray 1.185[1.12,1.25] subarrayc1.185\\ [1.12,1.25] subarray 3.538[3.41,3.67] subarrayc3.538\\ [3.41,3.67] subarray Table 6: Learned return-law fidelity. Entries are seed-level means with 95% confidence intervals. Avg. is the equal-weight average over Solitaire Dice and Discrete-MC. Lower 11-Wasserstein distance is better. D.2 Full Offline OGBench Results Tables 7 and 8 report the complete task-level results underlying Table 1. Seed counts and the domain-level tuning-task convention follow Appendix E.3. Gaussian Policies Flow Policies Distributional Critics Task BC IQL ReBRAC IFQL FQL C51 IQN CODAC Value Flows PCBF PA-BFM cube-double-play task1 8±38± 3 27±527± 5 45±645± 6 35±935± 9 61±961± 9 9±09± 0 70±1470± 14 80±1180± 11 97±197± 1 92±392± 3 ±99± 1 task2* 0±00± 0 1±11± 1 7±37± 3 9±59± 5 36±636± 6 0±00± 0 24±924± 9 63±463± 4 76±776± 7 74±774± 7 ±89± 2 task3 0±00± 0 0±00± 0 4±14± 1 8±58± 5 22±522± 5 0±00± 0 25±625± 6 66±966± 9 73±473± 4 81±881± 8 ±91± 2 task4 0±00± 0 0±00± 0 1±11± 1 1±11± 1 5±25± 2 0±00± 0 10±110± 1 13±213± 2 30±530± 5 22±522± 5 ±44± 7 task5 0±00± 0 4±34± 3 4±24± 2 17±617± 6 19±1019± 10 0±00± 0 81±881± 8 82±482± 4 69±569± 5 ±84± 3 70±670± 6 cube-triple-play task1* 1±11± 1 4±44± 4 1±21± 2 2±22± 2 20±620± 6 1±01± 0 29±229± 2 9±59± 5 59±1259± 12 18±418± 4 ±60± 8 task2 0±00± 0 0±00± 0 0±00± 0 0±00± 0 ±1± 2 0±00± 0 0±00± 0 ±1± 0 0±00± 0 0±10± 1 0±00± 0 task3 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 1±01± 0 0±00± 0 ±7± 3 1±11± 1 ±7± 2 task4 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 task5 0±00± 0 1±11± 1 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 ±2± 1 1±11± 1 1±11± 1 puzzle-3x3-play task1 5±25± 2 33±633± 6 97±497± 4 94±394± 3 90±490± 4 5±05± 0 71±371± 3 78±878± 8 99±099± 0 73±273± 2 ±100± 0 task2 1±11± 1 4±34± 3 1±11± 1 1±21± 2 16±516± 5 0±00± 0 2±22± 2 5±25± 2 98±298± 2 23±523± 5 ±100± 0 task3 1±11± 1 3±23± 2 3±13± 1 0±00± 0 10±310± 3 0±00± 0 0±00± 0 4±34± 3 ±97± 1 83±183± 1 86±286± 2 task4* 1±11± 1 2±12± 1 2±12± 1 0±00± 0 16±516± 5 0±00± 0 0±00± 0 5±55± 5 84±2484± 24 42±542± 5 ±94± 4 task5 1±01± 0 3±23± 2 5±35± 3 0±00± 0 16±316± 3 0±00± 0 0±00± 0 6±56± 5 58±3958± 39 43±443± 4 ±87± 8 puzzle-4x4-play task1 1±11± 1 12±212± 2 26±426± 4 49±949± 9 34±834± 8 0±00± 0 41±241± 2 37±3237± 32 36±336± 3 38±638± 6 ±73± 8 task2 0±00± 0 7±47± 4 12±412± 4 4±44± 4 16±516± 5 0±00± 0 12±412± 4 10±1010± 10 27±527± 5 23±523± 5 ±51± 2 task3 0±00± 0 9±39± 3 15±315± 3 50±1450± 14 18±518± 5 0±00± 0 45±745± 7 33±2933± 29 30±430± 4 40±440± 4 ±68± 5 task4* 0±00± 0 5±25± 2 10±310± 3 21±1121± 11 11±311± 3 0±00± 0 23±223± 2 12±1012± 10 28±528± 5 28±428± 4 ±49± 10 task5 0±00± 0 4±14± 1 7±37± 3 2±22± 2 7±37± 3 0±00± 0 16±616± 6 10±810± 8 13±213± 2 ±23± 3 20±620± 6 scene-play task1 19±619± 6 94±394± 3 95±295± 2 98±398± 3 ±100± 0 17±317± 3 ±100± 0 99±099± 0 99±099± 0 ±100± 0 ±100± 0 task2* 1±11± 1 12±312± 3 50±1350± 13 0±00± 0 76±976± 9 2±12± 1 1±01± 0 85±485± 4 97±197± 1 57±1357± 13 ±100± 0 task3 1±11± 1 32±732± 7 55±1655± 16 54±1954± 19 98±198± 1 0±10± 1 94±294± 2 90±390± 3 94±294± 2 98±298± 2 ±100± 0 task4 2±22± 2 0±10± 1 3±33± 3 0±00± 0 5±15± 1 2±12± 1 3±13± 1 0±00± 0 7±177± 17 12±312± 3 ±92± 8 task5 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 ±2± 1 0±00± 0 Table 7: Full offline evaluation on state-based OGBench tasks. Results cover 25 tasks and eight training seeds. An asterisk marks the domain-level tuning task. Bold denotes the highest displayed task mean; ties at the highest mean are all bold, while fully tied rows are left unbolded. Gaussian Policies Flow Policies Distributional Critics Task IQL ReBRAC IFQL FQL IQN Value Flows PCBF PA-BFM visual-antmaze-medium-navigate task1* 78±978± 9 54±1554± 15 ±81± 3 32±332± 3 62±762± 7 77±477± 4 65±365± 3 78±778± 7 task2 90±390± 3 ±96± 1 87±187± 1 60±260± 2 88±288± 2 75±575± 5 73±173± 1 92±292± 2 task3 80±680± 6 97±197± 1 92±192± 1 35±835± 8 64±564± 5 81±781± 7 86±486± 4 ±98± 2 task4 89±489± 4 93±293± 2 84±384± 3 35±235± 2 71±671± 6 71±671± 6 78±378± 3 ±96± 3 task5 84±284± 2 ±97± 1 89±289± 2 29±729± 7 84±184± 1 70±2670± 26 71±171± 1 93±193± 1 visual-antmaze-teleport-navigate task1* 5±25± 2 2±02± 0 7±47± 4 2±12± 1 2±12± 1 10±410± 4 8±28± 2 ±21± 9 task2 10±210± 2 10±310± 3 13±313± 3 6±16± 1 7±37± 3 17±517± 5 19±419± 4 ±33± 8 task3 7±77± 7 4±14± 1 8±98± 9 9±49± 4 6±46± 4 16±316± 3 ±18± 4 16±216± 2 task4 4±64± 6 4±04± 0 18±218± 2 9±19± 1 4±24± 2 16±516± 5 18±518± 5 ±32± 3 task5 2±12± 1 2±12± 1 4±24± 2 1±11± 1 2±12± 1 8±28± 2 8±38± 3 ±10± 2 visual-cube-double-play task1* 34±2334± 23 4±44± 4 8±68± 6 23±423± 4 4±14± 1 ±35± 2 10±310± 3 30±1030± 10 task2 3±13± 1 0±00± 0 0±00± 0 0±00± 0 0±00± 0 ±4± 2 0±00± 0 0±00± 0 task3 7±47± 4 2±22± 2 1±11± 1 4±24± 2 0±00± 0 ±11± 2 0±00± 0 10±410± 4 task4 2±12± 1 0±00± 0 0±00± 0 0±00± 0 0±00± 0 2±12± 1 0±00± 0 ±10± 4 task5 11±211± 2 0±00± 0 2±12± 1 4±14± 1 1±11± 1 13±313± 3 3±13± 1 ±33± 9 visual-puzzle-3x3-play task1* 7±157± 15 88±488± 4 ±100± 0 94±194± 1 84±184± 1 93±593± 5 81±181± 1 90±290± 2 task2 0±00± 0 ±12± 1 0±00± 0 0±00± 0 6±26± 2 ±12± 1 0±00± 0 0±00± 0 task3 0±00± 0 1±11± 1 2±12± 1 0±00± 0 1±01± 0 3±13± 1 2±12± 1 ±5± 1 task4 1±11± 1 0±10± 1 1±01± 0 5±45± 4 3±03± 0 ±6± 2 5±35± 3 4±14± 1 task5 0±00± 0 0±00± 0 0±00± 0 1±01± 0 1±01± 0 ±2± 0 1±01± 0 ±2± 1 visual-scene-play task1* 97±297± 2 98±498± 4 86±1086± 10 98±398± 3 95±295± 2 99±099± 0 40±440± 4 ±100± 1 task2 21±1621± 16 30±1530± 15 0±00± 0 ±86± 8 79±1579± 15 40±2740± 27 62±862± 8 42±1442± 14 task3 12±912± 9 10±710± 7 19±219± 2 22±622± 6 31±1431± 14 66±166± 1 35±135± 1 ±68± 12 task4 1±01± 0 0±00± 0 0±00± 0 1±11± 1 0±00± 0 10±610± 6 12±412± 4 ±38± 11 task5 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 0±00± 0 Table 8: Full offline evaluation on 25 image-based OGBench tasks over four seeds. An asterisk marks the tuning task; bold denotes the highest displayed task mean, with fully tied rows left unbolded. D.3 Offline-to-Online Results Table 9 reports scenario-level performance at online entry and at the final online checkpoint. Method Antmaze Cube-D Cube-T Scene Offline checkpoint at 1000k1000k steps IQN 55±555±5 29±429±4 25±525±5 0±00±0 IFQL 11±311±3 12±312±3 2±12±1 0±00±0 FQL ±87 7 40±1140±11 4±14±1 ±82 11 RLPD 0±00±0 0±00±0 0±00±0 0±00±0 IQL 41±541±5 0±00±0 3±23±2 15±415±4 Value Flows 72±472±4 65±765±7 ±29 8 ±83 4 PA-BFM ±89 4 ±78 4 24±224±2 ±79 3 Online final at 2000k2000k steps IQN 91±191±1 42±742±7 14±614±6 0±00±0 IFQL 3±23±2 41±241±2 7±57±5 60±1460±14 FQL ±99 1 ±92 3 ±83 12 ±100 1 RLPD 3±23±2 0±00±0 0±00±0 ±100 0 IQL 45±545±5 0±00±0 0±00±0 10±310±3 Value Flows ±96 1 79±679±6 70±770±7 ±100 0 PA-BFM ±99 1 ±95 2 ±85 4 ±100 0 Table 9: Offline-to-online evaluation on four OGBench scenarios. The two panels report success at online entry and after one million online environment steps. Entries are means ± standard deviations over eight seeds; bold denotes means within 95%95\% of the highest scenario mean. D.4 Critic Diagnostics and Adaptive Pessimism We evaluate critic reliability at the final 10610^6-update checkpoint on puzzle-4x4-play-task4 using eight seeds. This prespecified domain-level tuning task is not selected using the diagnostic results. For each of Nq=128N_q=128 held-out state–action pairs, we construct aδ a_δ =clip((1−δ)a+δarand,−1,1), =clip\! ((1-δ)a_D+δ a_rand,-1,1 ), (120) δ δ ∈0,.2,.4,.6,.8,1. ∈\0,.2,.4,.6,.8,1\. where aranda_rand is sampled uniformly from the normalized action space and shared across methods. Absolute prediction error compares the twin-critic mean with a Monte Carlo return averaged over Nref=64N_ref=64 rollouts after forcing aδa_δ as the first action and then following the checkpoint’s evaluation policy. The assessed critic is not used in this reference. We compute DtwinD_twin from Mdiag=64M_diag=64 shared return-noise samples and define PoutP_out as predicted mass above the 95th percentile of offline-dataset return-to-go. Prediction error, Dtwin D_twin, and positive overestimation are normalized by the difference between the 95th and 5th return-to-go percentiles; unsupported mass is already dimensionless. Metrics are averaged over queries and seeds. Across the six action-shift levels, both diagnostics track absolute return-prediction error. The descriptive Spearman correlations are ρtwin=0.943 _twin=0.943 for twin-critic disagreement and ρout=0.886 _out=0.886 for unsupported mass. For conservative training, we compare the undistorted target, fixed soft tilts η∈0.1,0.3,0.6η∈\0.1,0.3,0.6\, the adaptive rule in Eq. (36), and Value Flows. The adaptive rule uses DpoolD_pool computed from undistorted posterior weights, clips normalized disagreement to [0,1][0,1], and caps the tilt at η0=0.6 _0=0.6. It is a stop-gradient practical heuristic and is not directly covered by the fixed-distortion contraction analysis in Appendix B.5. Table 10 reports the complete fixed-strength sweep. Stronger fixed tilting reduces positive overestimation but eventually degrades policy performance. The adaptive variant reduces normalized positive overestimation from 0.3800.380 to 0.1650.165 while increasing final success from 0.4900.490 to 0.5150.515. Variant Positive overestimation ↓ Final success ↑ No distortion 0.380±0.0330.380± 0.033 0.490±0.1000.490± 0.100 Fixed η=0.1η=0.1 0.300±0.0270.300± 0.027 0.505±0.0900.505± 0.090 Fixed η=0.3η=0.3 0.220±0.0240.220± 0.024 0.455±0.1100.455± 0.110 Fixed η=0.6η=0.6 0.105±0.0190.105± 0.019 0.365±0.1300.365± 0.130 Adaptive 0.165±0.0230.165± 0.023 0.515±0.0900.515± 0.090 Value Flows 0.335±0.0410.335± 0.041 0.280±0.0500.280± 0.050 Table 10: Conservatism–performance trade-off on puzzle-4x4-play-task4. The undistorted PA-BFM and Value Flows scores correspond to the task-level results in the main benchmark table. Lower positive overestimation and higher final success are better. D.5 Runtime Analysis Component and pool-size ablations are shown in Figure 3 of the main paper. The corresponding final-score differences are reported in the main text; here we focus on computational cost. Computational cost. Table 11 reports wall-clock hours for 10610^6-update budgets on one representative state-based configuration and one representative visual configuration. These are configuration-level values rather than averages over all benchmark domains. All methods use one NVIDIA GeForce RTX 4090 with 24 GB of memory, batch size 256256, and the same implementation, compilation, and data-loading pipeline. The same end-to-end timing boundary is applied to every method within a column; endpoint candidates within an update are evaluated in parallel and reuse the successor-state representation. Method Target work State Visual BCFM (K=1K=1) 2 target solves 3.6 10.5 Uniform (K=8K=8) 16 target solves 3.8 11.0 PA-BCFM (K=4K=4) 8 solves + wt. 3.8 11.0 PA-BCFM (K=8K=8) 16 solves + wt. 4.0 11.8 PA-BCFM (K=16K=16) 32 solves + wt. 4.5 13.2 Value Flows path + confidence 4.4 12.5 PCBF (λcv=0.3 _cv=0.3) endpoint + CV 4.0 11.5 Table 11: Wall-clock training cost. State and visual columns report configuration-level hours for 10610^6-update budgets on one NVIDIA GeForce RTX 4090 (24 GB) under the two representative timing settings. Target work counts both target critics, so K ensemble-mean endpoints require 2K2K target-flow solves. The default K=8K=8 configuration remains below the measured Value Flows runtime in both settings; K=16K=16 adds cost with little gain in Figure 3, supporting K=8K=8 as the default. Appendix E Experimental Details All methods are implemented and evaluated in a common JAX codebase adapted from FQL and Value Flows (Park et al., 2025b; Dong et al., 2026). Methods share OGBench wrappers, dataset revisions, preprocessing, budgets, and evaluation code. BCFM, Value Flows, PCBF, and PA-BFM additionally share the actor, behavior proposal, critic backbone, target update, ODE solver, schedule, and mean-based action extraction. They differ in critic supervision: endpoint matching, weighted consistency, a path-coupled control variate, and endpoint-pool posterior averaging, respectively. Isolation claims apply only to this group. E.1 Environments Controlled environments. The one-state Bernoulli-return process is a development process used for configuration selection. Solitaire Dice and the discrete Monte Carlo chain are the controlled evaluation processes. The complete process definitions are given in Table 3. All three processes admit exact return laws, so the reference return distributions and posterior Bellman velocities are computed from the exact endpoint laws; no Monte Carlo reference pool is used for the controlled results. For Discrete-MC, let n=20n=20, =0,…,n−1X=\0,…,n-1\, and define the unnormalized state weight ωi=exp[n−14πcos(4π(i−1)n−1)]. _i= \! [ n-14π \! ( 4π(i-1)n-1 ) ]. (121) For each nonterminal state i∈1,…,n−2i∈\1,…,n-2\, the transition kernel is P(i,i±1) P(i,i ±1) =12ωi±1ωi+ωi±1, = 12 _i± 1 _i+ _i± 1, (122) P(i,i) P(i,i) =1−P(i,i−1)−P(i,i+1), =1-P(i,i-1)-P(i,i+1), with P(0,0)=P(n−1,n−1)=1P(0,0)=P(n-1,n-1)=1. Episodes initialize from X0∼Unif1,…,n−2X_0 \1,…,n-2\. A transition receives reward Rh=Xh+1∉0,n−1R_h=1\X_h+1∉\0,n-1\\, and γ=0.95γ=0.95. Offline OGBench. We evaluate the 50 single-task state and image environments listed in Tables 7–8 using the standard OGBench protocol (Dong et al., 2026; Park et al., 2025a). State-based tasks use native observations; image-based tasks use three stacked RGB frames, a small IMPALA encoder, and random-crop augmentation. Offline-to-online OGBench. We additionally evaluate offline-to-online fine-tuning on antmaze-large, cube-double, cube-triple, and scene. Each method is first trained for one million offline optimizer updates and is then fine-tuned for one million online environment steps while newly collected transitions are appended to the replay buffer. Each method retains its own critic objective throughout both phases. E.2 Methods for Comparison The offline tables compare Gaussian-policy scalar critics (BC, IQL, and ReBRAC), flow-policy scalar critics (IFQL and FQL), distributional critics (C51, IQN, and CODAC), and flow-based distributional critics (Value Flows, PCBF, and PA-BFM) (Kostrikov et al., 2022; Tarasov et al., 2023; Park et al., 2025b; Bellemare et al., 2017; Dabney et al., 2018a; Ma et al., 2021; Dong et al., 2026; Xu et al., 2026). PCBF also appears in the controlled and runtime comparisons but not in offline-to-online evaluation. The latter additionally includes RLPD under the common interaction budget and checkpoint schedule (Ball et al., 2023). PA-BFM, Value Flows, and PCBF share the policy-side implementation in the matched comparison and differ in critic supervision. E.3 Evaluation Protocol Offline evaluation. State-based and image-based tasks use eight and four training seeds, respectively. Each evaluation checkpoint contains 50 episodes, and the main tables report the final checkpoint. For each domain, the reported mean is the arithmetic mean of the five task means, and the displayed uncertainty is the arithmetic mean of the corresponding five task-level standard deviations. The task marked with an asterisk is used for domain-level hyperparameter selection under a common search budget for the matched methods. The selected configuration is fixed across the remaining tasks in that domain, while the tuning task remains in the five-task aggregate for comparability with prior work. Task-level win/tie/loss counts compare final means and are descriptive rather than significance tests. Offline-to-online evaluation. Online interaction begins from the one-million-optimizer-update offline checkpoint. Methods are evaluated every 100k environment steps through 2000k total steps. Each method uses eight independent training seeds. At each checkpoint, the learning curves show the mean success rate over training seeds, and shaded bands show one standard deviation across training seeds. Training seeds and evaluation environment seeds are disjoint, and evaluation seeds are shared across methods within each scenario. Controlled evaluation. The controlled posterior-target and gradient-covariance protocols are specified with their corresponding metrics in Appendix D.1. Paired estimator comparisons reuse the same candidate endpoint pools whenever the estimators share an endpoint budget. Controlled target-MSE and gradient-covariance intervals use a process-stratified query-cluster bootstrap, with all replicates of a selected query retained together. Learned-fidelity intervals use training seeds as the resampling unit, and paired method differences use matched seed indices. All reported intervals are 95% confidence intervals. E.4 Architectures and Hyperparameters The network architectures and optimization settings of the matched flow-critic methods follow Value Flows (Dong et al., 2026). Table 12 summarizes the settings shared by PA-BFM and the corresponding flow-critic baselines. Entries involving flow integration or candidate-action selection apply only to methods containing those components. Hyperparameter Value Optimizer Adam (Kingma and Ba, 2015) Batch size 256256 Critic learning rate 3×10−43× 10^-4 Critic hidden-layer sizes (512,512,512,512)(512,512,512,512) Activation function GELU Actor layer normalization Yes Critic/value layer normalization Yes Return-flow Euler steps 1010 Return samples per candidate action MactM_act 11 Candidate actions N 1616 Target-network update coefficient τtgt _tgt 5×10−35× 10^-3 Number of critics 22 Image encoder Small IMPALA (Park et al., 2025b) Image augmentation Random cropping Image frame stack 33 Numeric precision float32 Table 12: Common hyperparameters for PA-BFM and matched flow-critic baselines. Entries apply where the corresponding component is present. PA-BFM-specific hyperparameters. The default PA-BCFM critic uses K=8K=8 Bellman endpoints and no pessimistic distortion, i.e., g≡1g≡ 1. Bridge time is sampled as t∼Unif(0,1−10−5)t (0,1-10^-5), and posterior logits are normalized by log-sum-exp. All downstream experiments use this setting; K∈1,2,4,8,16K∈\1,2,4,8,16\ is varied only in the controlled estimator and pool-sensitivity studies. Baseline hyperparameter selection. IQL selects its advantage-weighting inverse temperature from 0.3,1,3,10\0.3,1,3,10\. ReBRAC selects its actor and critic regularization coefficients from 0.003,0.01,0.03,0.1,0.3,1\0.003,0.01,0.03,0.1,0.3,1\ and 0,0.001,0.01,0.1\0,0.001,0.01,0.1\, respectively. IFQL selects the number of proposed actions from 32,64,128\32,64,128\, and FQL uses its original regularization grid. C51 uses 51 or 101 atoms. IQN and CODAC select the quantile-loss parameter from 0.7,0.8,0.9,0.95\0.7,0.8,0.9,0.95\, with CODAC penalty 0.10.1. PCBF uses λcv=0.3 _cv=0.3, selected on the Bernoulli development process and frozen for Solitaire Dice and Discrete-MC. Each selected setting is fixed for the remaining tasks in its group. E.5 Runtime and Reproducibility All experiments use JAX on one NVIDIA GeForce RTX 4090 (24 GB), with batch size 256 and common implementation, compilation, data-loading, and evaluation boundaries. Table 11 reports end-to-end wall-clock hours for one million updates on representative state and visual configurations; these are configuration-level measurements, not domain averages. Performance comparisons use matched optimizer-update counts. We record the software and hardware versions, OGBench and dataset revisions, random seeds, training commands, configuration overrides, checkpoint-selection rules, and evaluation commands. Tables and figures use the same plotting and aggregation pipeline across methods. Domain/scenario Evaluation γ cube-double Off.+O2O .995 cube-triple Off.+O2O .995 puzzle-3x3 Offline .99 puzzle-4x4 Offline .99 scene Off.+O2O .99 visual-antmaze- medium Offline .99 visual-antmaze- teleport Offline .99 visual-cube-double Offline .995 visual-puzzle-3x3 Offline .99 visual-scene Offline .99 antmaze-large O2O only .99 Table 13: Domain/scenario settings. PA-BFM uses an undistorted K=8K=8 critic, and all main experiments use mean-based candidate scoring.