Paper deep dive
DOHF: Online Diffusion Fine-tuning with Doob's $h$-transform Guidance
Zhengyi Guo, Jiayuan Sheng, Wenpin Tang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/29/2026, 1:14:12 PM
Summary
The paper introduces Diffusion Online h-guidance Fine-tuning (DOHF), a method that leverages Doob's h-transform to fine-tune diffusion models using reward signals. DOHF estimates the normalized local correction (gradient of the log h-function) under the evolving rollout policy and distills it into the generative model. This approach allows for online training with black-box or non-differentiable rewards without additional network evaluations, improving alignment in scenarios with rare desirable outcomes.
Entities (6)
Relation Signals (5)
DOHF → supports → black-box rewards
confidence 95% · our framework accommodates black-box and non-differentiable rewards without additional network evaluations.
DOHF → uses → Doob's h-transform
confidence 95% · DOHF turns Doob's h-transform into a practical online training algorithm.
DOHF → comparedto → DiffusionNFT
confidence 90% · We further establish a precise connection to DiffusionNFT... showing when its population-optimal velocity coincides with the exact hh-transformed update.
DOHF → evaluatedon → SD3.5-Medium
confidence 90% · We evaluate DOHF on... SD3.5-Medium text-to-image fine-tuning.
DOHF → improves → generative learning
confidence 90% · Our work demonstrates how adapting probabilistic conditioning... can improve generative learning across statistical sampling and visual generation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online $h$-guidance Fine-tuning (DOHF), which turns Doob's $h$-transform into a practical online training algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction $\nabla\log h$ under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified $h$-transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative distillation can improve generative learning across statistical sampling and visual generation.
Tags
Links
- Source: https://arxiv.org/abs/2609.31882v1
- Canonical: https://arxiv.org/abs/2609.31882v1
Trouble viewing inline? Open PDF directly →
Full Text
64,011 characters extracted from source content.
Expand or collapse full text
DOHF: Online Diffusion Fine-tuning with Doob’s h-transform Guidance Zhengyi Guo Jiayuan Sheng Wenpin Tang Abstract Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online h-guidance Fine-tuning (DOHF), which turns Doob’s h-transform into a practical online training algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction ∇logh∇ h under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified h-transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative distillation can improve generative learning across statistical sampling and visual generation. 1 Introduction Doob’s h-transform provides a classical probabilistic mechanism for conditioning a Markov process on a terminal event or reweighting its terminal distribution (Doob, 1957). Given a terminal weight w(X0)w(X_0), the associated function h(xt,t)=[w(X0)∣Xt=xt]h(x_t,t)=E\! [w(X_0) X_t=x_t ] translates terminal information into a local correction of the intermediate dynamics through ∇logh∇ h. This perspective has recently become increasingly relevant to diffusion models. DEFT (Denker et al., 2024) formulates conditional generation through a generalized h-transform and learns an auxiliary network. Related work studies diffusion guidance under hard constraints (Guo et al., 2026b). More recently, DOIT (Zhu et al., 2026) uses Doob’s h-transform to construct a training-free adaptation procedure. From these viewpoints, a central question is how to efficiently estimate local correction for fine-tuning throughout the generative trajectory. The h-transform gives an explicit answer. For a current rollout policy poldp^old and a general reward weight w, the desired terminal distribution is pft(x0)∝w(x0)pold(x0),p^ft(x_0) w(x_0)p^old(x_0), while its intermediate score differs from that of the current model by exactly ∇xtlogh(xt,t) _x_t h(x_t,t). Existing h-transform approaches use this correction for inference-time guidance (Kim et al., 2025; Zhu et al., 2026), typically relative to a fixed reference model. This can become restrictive when the high-reward region is rare under the original proposal: the trajectories used to estimate the correction may provide little information precisely where adaptation is most needed. This motivates an online procedure in which the h-function itself is defined and estimated under the evolving rollout policy, and the resulting local correction is progressively distilled into the generative model. In this work, we propose Diffusion Online h-guidance Fine-tuning (DOHF). At each iteration, samples from the current rollout policy are assigned an optimality weight w(X0)w(X_0), from which we estimate the corresponding ∇logh∇ h and distill the resulting correction directly into the generative model. This online procedure progressively shifts the rollout distribution toward more informative regions, while our simulation-based estimator remains applicable to black-box and non-differentiable rewards. The same formulation naturally accommodates continuous rewards, hard or soft constraints, and relative optimality objectives. We further establish a precise connection to DiffusionNFT (Zheng et al., 2026), showing when its population-optimal velocity coincides with the exact h-transformed update. We evaluate DOHF on rare-mode adaptation, constrained contingency-table generation, and SD3.5-Medium text-to-image fine-tuning. Across these settings, DOHF consistently benefits from jointly adapting the rollout distribution and the h-correction, leading to faster adaptation and strong reward alignment when high-reward regions are poorly represented by the original model. In conclusion, The key of DOHF is not the use of Doob’s h-transform itself, but turning its normalized local correction into an online, self-improving training target: the correction is re-estimated under the evolving rollout policy and distilled into the model that generates the next round of data. Our main contributions are: • Optimality as an unified framework. We formulate reward-based fine-tuning through a single weight w:→[0,1]w:X→[0,1] interpreted as the probability of an optimality event O=1\O=1\. Exponential tilting, hard constraints w=r≥zw=1\r≥ z\, their smooth relaxations, and relative optimality objectives all become instances of one object, and each induces the exact target pft∝wpoldp^ft w\,p^old. In contrast to formulations built on scalar advantages or on a positive/negative split of a rollout group, this retains the full conditioning structure, and in particular accommodates indicator-type hard constraints. • Online h-guidance distillation. We turn the terminal weight into a local velocity correction −t1−t∇xlogh- t1-t _x h and regress it directly into the generative model, re-estimating h under the evolving rollout policy at every iteration. Our simulation-based estimator reuses the velocity already computed at the current step, so leads to no additional network evaluations, and w need not be differentiable. Estimating h and improving the rollout distribution are two sides of one update: each iteration re-collects trajectories under the tilted policy, so the correction is estimated where adaptation is most needed. • An explicit h yields the guidance strength others must tune. We provide a population-level characterization of DiffusionNFT, showing that its update is an h-attenuated version of the exact h-transform and becomes exact only under the state-dependent choice β=2h(xt,t,c)β=2h(x_t,t,c). Since DOHF estimates h explicitly, it realizes this state-dependent strength, whereas methods that do not must leave β a constant — which attenuates the update exactly in the low-h regions where high-reward mass is rare. We verify this prediction and the framework as a whole on rare-mode adaptation, constrained contingency-table generation, and SD3.5-M text-to-image fine-tuning, where our formulation is additionally compatible with classifier-free guidance, a design not available to forward-matching baselines. Related Work. Existing h-transform methods provide foundations for our work. DEFT (Denker et al., 2024), hard-constraint guidance (Guo et al., 2026b), and Doob’s Matching (Chang et al., 2026) learn conditioning corrections, while DOIT (Zhu et al., 2026) estimates them through simulation. SVDD (Li et al., 2024), twisted SMC (Wu et al., 2024), and Feynman-Kac steering (Singhal et al., 2025) offer related inference-time approaches. Tilt Matching (Potaptchik et al., 2025), Iterative Tilting (Pachebat et al., 2025), and Iterative Importance Fine-tuning (Denker et al., 2026) connects tilted distributions with iterative learning. In parallel, a rapidly growing literature studies reward-based fine-tuning of diffusion models (Uehara et al., 2024; Winata et al., 2025). Policy-based approaches formulate the reverse generation process as a policy and optimize it through RL, including DDPO (Black et al., 2024), FlowGRPO (Liu et al., 2025), DanceGRPO (Xue et al., 2025), DPOK (Fan et al., 2023), and MixGRPO (Li et al., 2026). Sampler stochasticity and the resulting training-inference gap are studied in (Sheng et al., 2025). ReFL (Xu et al., 2023) and DRaFT (Clark et al., 2024) directly differentiate reward signals, while DiffusionDPO (Wallace et al., 2023) learns from preference pairs. Adjoint Matching (Domingo-Enrich et al., 2025; Guo et al., 2026a) and Score as Action (Zhao et al., 2024; Zhao et al., 2025) cast fine-tuning as stochastic optimal control. Gradient-based construction (Zhou et al., 2026) and flow-map alignment (Holderrieth et al., 2026; Potaptchik et al., 2026) provide alternatives when reward derivatives are available. Regression-based approaches include Advantage Weighted Matching (Xue et al., 2026) and FlowAWR (Fu et al., 2026), while online reward-weighted flow matching (Fan et al., 2025) directly reweights newly generated samples. Reward Score Matching (Lee et al., 2026), a path-space perspective (Xu et al., 2026), and reward-based velocity matching (Choi et al., 2026) clarify the shared structure of these updates. 2 Preliminaries Throughout, let c denote an optional conditioning context (e.g. a text prompt, a market covariate, or c=∅c= if unconditional). We suppress c from the notation below whenever it plays no special role. A pre-trained diffusion/flow model generates X0∼pbase(x0∣c).X_0 p^base(x_0 c). Moreover, we generate samples online, so we define the policy for rolling out is pold(x0∣c)p^old(x_0 c). 2.1 A Unified Optimality Weight We define optimality through a single weight function w:→[0,1],pold[w(X0)]>0.w:X→[0,1], _p^old[w(X_0)]>0. We introduce an optimality variable Levine (2018); Zheng et al. (2026) defining whether the generated X0X_0 is optimal or not: O:=U≤w(X0),O:=1_\U≤ w(X_0)\, where U∼Unif(0,1)U (0,1) is an auxiliary uniform random variable independent of X0X_0. Intuitively, X0X_0 with higher weight w(X0)w(X_0) is more likely to be optimal. Then ℙ(O=1∣X0=x0)=w(x0)P(O=1 X_0=x_0)=w(x_0), and conditioning the base model on O=1\O=1\ gives the fine-tuned target pft(x0∣c):=pold(x0∣O=1,c)=w(x0)ℙ(O=1∣c)pold(x0∣c)∝w(x0)pold(x0∣c).p^ft(x_0 c):=p^old(x_0 O=1,c)= w(x_0)P(O=1 c)\,p^old(x_0 c)\; \;w(x_0)\,p^old(x_0 c). Every choice of w∈[0,1]w∈[0,1] therefore corresponds to a genuine conditioning event. Note also that w and CwCw (C>0C>0) induce the same pftp^ft, so any normalization on w has no effect. Examples. (i) Exponential tilting Domingo-Enrich et al. (2025): w(x0)=exp(r(x0,c)−rmax(c)τ)w(x_0)= \! ( r(x_0,c)-r_ (c)τ ), temperature τ>0τ>0. (i) Hard threshold Guo et al. (2026b): w(x0)=r(x0,c)≥zw(x_0)=1_\r(x_0,c)≥ z\, giving pft∝pold 1r≥zp^ft p^old\,1_\r≥ z\. (i) Sigmoid relaxation: w(x0)=σ(r(x0,c)−zβ)w(x_0)=σ\! ( r(x_0,c)-zβ ), σ(u)=(1+e−u)−1σ(u)=(1+e^-u)^-1. Note that it recovers the hard threshold when β→0+β→ 0^+. (iv) Loss/statistic-driven weights Kim et al. (2025): w(x0)=ψ(ℓ(x0))w(x_0)=ψ( (x_0)) for a nondecreasing ψ, e.g. extreme summary statistics of x0x_0 or loss of data x0x_0 defined by upstream models. 2.2 The h-Function Define h-function as the probability of optimality given tuple (xt,t,c)(x_t,t,c) as h(xt,t,c):=ℙ(O=1∣Xt=xt,c)=[w(X0)∣Xt=xt,c].h(x_t,t,c):=P(O=1 X_t=x_t,c)=E [w(X_0) X_t=x_t,c ]. In particular h(x0,0,c)=w(x0)h(x_0,0,c)=w(x_0). By Bayes’ Rule, the fine-tuned marginal at time t is ptft(xt∣c)=ptold(xt∣O=1,c)=1Z(c)ptold(xt∣c)h(xt,t,c),p_t^ft(x_t c)=p_t^old(x_t O=1,c)= 1Z(c)\,p_t^old(x_t c)\,h(x_t,t,c), (2.1) where Z(c)Z(c) does not depend on xtx_t. Hence ∇xtlogptft(xt∣c)=∇xtlogptold(xt∣c)+∇xtlogh(xt,t,c), _x_t p_t^ft(x_t c)= _x_t p_t^old(x_t c)+ _x_t h(x_t,t,c), (2.2) so the score correction for fine-tuning is ∇xtlogh(xt,t,c)=∇xtlog[w(X0)∣Xt=xt,c] _x_t h(x_t,t,c)= _x_t [w(X_0) X_t=x_t,c ]. 2.3 Flow Matching Consider Gaussian flow matching Xt=(1−t)X0+tX1,X1∼(0,I),t∈[0,1],X_t=(1-t)X_0+tX_1, X_1 (0,I), t∈[0,1], Let vold(x,t,c)v^old(x,t,c) denote the velocity field corresponding to old policy. For this Gaussian interpolation, the velocity and score are related (see Appendix A for derivation) through v(x,t,c)=−x+t∇xlogpt(x∣c)1−t.v(x,t,c)=- x+t _x p_t(x c)1-t. By Equation (2.2), the corresponding fine-tuned velocity provides a learning target: vft(x,t,c)=vold(x,t,c)−t1−t∇xlogh(x,t,c).v^ft(x,t,c)=v^old(x,t,c)- t1-t _x h(x,t,c). (2.3) 3 Diffusion Online h-guidance Fine-tuning For Flow Matching, the backward process can be reformulated as a SDE Huang et al. (2021); Tang and Zhao (2025): dXt=[(1+λt2)vold(Xt,t,c)+λt21−tXt]dt+λt2t1−tdB¯t,dX_t= [(1+ _t^2)v^old(X_t,t,c)+ _t^21-tX_t ]dt+ _t 2t1-t\,d B_t, (3.1) where λt _t is a parameter that we can choose, and when λt=0 _t=0 it reverts to the original ODE form. Consider discrete reverse times 0=t0<t1<⋯<tL=10=t_0<t_1<·s<t_L=1, and let ϕℓold(xtℓ−1∣xtℓ,c) _ ^old(x_t_ -1 x_t_ ,c) denote the backward Gaussian transition kernel, with mean μℓ(xtℓ) _ (x_t_ ) and variance σℓ2I _ ^2I. In order to do the regression (2.3), we need ∇xlogh(x,t,c) _x h(x,t,c). Fortunately, we could use Monte Carlo Method to estimate it during the online fine-tuning process. 3.1 Simulation-Based Estimator of ∇xlogh(x,t,c) _x h(x,t,c) Zhu et al. (2026) Given Xtℓ=xtℓX_t_ =x_t_ , simulate M backward trajectories xtℓ−1(m),…,x0(m)m=1M\x_t_ -1^(m),…,x_0^(m)\_m=1^M using ϕoldφ^old. Define Wm:=w(x0(m)),Gm:=∇xtℓlogϕℓold(xtℓ−1(m)∣xtℓ,c)=1σℓ2[∇xtℓμℓ(xtℓ)]⊤(xtℓ−1(m)−μℓ(xtℓ)).W_m:=w (x_0^(m) ), G_m:= _x_t_ _ ^old (x_t_ -1^(m) x_t_ ,c )= 1 _ ^2 [ _x_t_ _ (x_t_ ) ] (x_t_ -1^(m)- _ (x_t_ ) ). The plug-in estimator is ∇logh^A=∑m=1MWmGm(∑m=1MWm)∨τℓ, ∇ h_A= _m=1^MW_mG_m ( _m=1^MW_m ) _ , (3.2) with small truncation level τℓ>0 _ >0 for numerical stability. To alleviate the cost by M rollouts of length ℓ , we use one-step-lookahead surrogate x^0(m)=xtℓ−1(m)−tℓ−1vold(xtℓ,tℓ,c),xtℓ−1(m)∼(μℓ(xtℓ),σℓ2I), x_0^(m)=x_t_ -1^(m)-t_ -1\,v^old(x_t_ ,t_ ,c), x_t_ -1^(m) ( _ (x_t_ ), _ ^2I ), in place of x0(m)x_0^(m), at zero extra NFE (reusing vold(xtℓ,tℓ,c)v^old(x_t_ ,t_ ,c) already computed). Moreover, it does not require w to be differentiable, which is more general for various terminal rewards mentioned in Section 2.1. If w is differentiable, we also provide an alternative estimator using finite difference method, which is provided in Appendix B. 3.2 Training and Sampling We instantiate the training A for Text-to-Image fine-tuning, and for generalized task, prompt could be omitted. In terms of sample rolling-out, we use a slow-mix strategy: starts from on-policy and smooth it with exponential moving average. Note that we index time so that t=1 is pure noise and generation runs from tL=1t_L=1 to t0=0t_0=0. For sampling process, especially for Text-to-Image task, we could also implement the similar way used in Classifier-free Guidance Ho and Salimans (2022), as we set some proportion of the prompt to be empty in training process, in order to improve its unconditional generation. For other tasks, just use fine-tuned vθv_θ to sample. For generalization purpose, we provide Algorithm B used in T2I task. Algorithm A Diffusion Online h-guidance Fine-tuning (DOHF) 1: Pretrained vbasev^base, prompt set C, reward model r, schedule ti\t_i\, slow-mix schedule ηk _k 2: Initialize model weight vθ←vbasev_θ← v^base, and sample-generation policy vϕ←vθv_φ← v_θ 3: for k=1,2,…k=1,2,… do 4: for prompt c in mini-batch do 5: Sample tℓt_ uniformly; roll out vϕ(⋅,⋅,c)v_φ(·,·,c) to obtain xtℓx_t_ 6: Compute ∇logh^(xtℓ,tℓ,c) ∇ h(x_t_ ,t_ ,c) ⊳ via 3.2 or B.1 7: vtarget←(vϕ(xtℓ,tℓ,c)−tℓ1−tℓ∇logh^)v^target \! (v_φ(x_t_ ,t_ ,c)- t_ 1-t_ \, ∇ h ) 8: ℒ←ℒ+‖vθ(xtℓ,tℓ,c)−vtarget‖22L +\|v_θ(x_t_ ,t_ ,c)-v^target\|_2^2 9: end for 10: Update θ by one step on ∇θℒ _θL 11: vϕ←ηkvϕ+(1−ηk)vθv_φ← _kv_φ+(1- _k)v_θ ⊳ slow-mix sample-generation policy 12: end forreturn vθv_θ Algorithm B Sampling with Classifier-Free Guidance 1: Trained vθv_θ, prompt c, guidance scale γ≥0γ\!≥\!0, schedule tii=0N\t_i\_i=0^N 2: x←z∼(0,I)x← z (0,I) 3: for i=0,1,…,N−1i=0,1,…,N-1 do 4: vcond←vθ(x,ti,c)v_cond← v_θ(x,t_i,c), vuncond←vθ(x,ti,∅)v_uncond← v_θ(x,t_i, ) 5: v←vuncond+γ(vcond−vuncond)v← v_uncond+γ\,(v_cond-v_uncond) 6: x←x+v(ti+1−ti)x← x+v\,(t_i+1-t_i) 7: end forreturn x 3.3 Population-Level Relation to DiffusionNFT DiffusionNFT (Zheng et al., 2026) is the closest online fine-tuning method to DOHF. Although its positive–negative construction does not explicitly estimate an h-transform, its population optimum admits an exact characterization in terms of the same h-function. Let h(xt,t,c)=pold[w(X0)∣Xt=xt,c].h(x_t,t,c)=E_p^old [w(X_0) X_t=x_t,c ]. DiffusionNFT parameterizes its positive and negative branches as vθ+=(1−β)vold+βvθ,vθ−=(1+β)vold−βvθ,v_θ^+=(1-β)v^old+β v_θ, v_θ^-=(1+β)v^old-β v_θ, where β>0β>0 is a fixed guidance coefficient. Proposition 3.1. At the population optimum of the double-branch DiffusionNFT objective, the learned velocity satisfies vθ∗(xt,t,c)=vold(xt,t,c)−2h(xt,t,c)βt1−t∇xtlogh(xt,t,c).v_θ^*(x_t,t,c)=v^old(x_t,t,c)- 2h(x_t,t,c)β t1-t _x_t h(x_t,t,c). Consequently, its population optimum coincides with the exact h-transformed velocity vft=vold−t1−t∇xtloghv^ft=v^old- t1-t _x_t h if and only if β=2h(xt,t,c).β=2h(x_t,t,c). A complete derivation is provided in Appendix C. This result reveals a key distinction between the two methods. With a fixed β, DiffusionNFT effectively follows the unnormalized direction ∇h=h∇logh∇ h=h∇ h; hence, its update is attenuated in low-h regions where optimal samples are rare. DOHF instead directly estimates the normalized correction ∇logh∇ h under the evolving rollout policy and distills it into the model. It therefore realizes the exact h-transformed update without requiring a state-dependent guidance hyperparameter. 4 Experiment 4.1 Gaussian Mixture To isolate the effect of online iteration from confounders, we conduct a controlled study on a 2D synthetic problem. The base distribution is an imbalanced two-mode Gaussian mixture pdata(x)= 0.01⋅(x,μ0,Σ)+ 0.99⋅(x,μ1,Σ),p_data(x)\;=\;0.01·N\! (x;\, _0,\, )\;+\;0.99·N\! (x;\, _1,\, ), with μ0=(−1,−1)⊤ _0=(-1,-1) , μ1=(1,1)⊤ _1=(1,1) , and Σ=0.16 =0.16\,I. A rectified-flow velocity network (a small MLP) is first pretrained on samples from pdatap_data. The reward is the soft posterior probability of belonging to the minority mode: r(x)=ℙ(cluster=0|x)=0.01⋅(x,μ0,Σ)0.01⋅(x,μ0,Σ)+ 0.99⋅(x,μ1,Σ)∈(0,1).r(x)\;=\;P(cluster=0\,|\,x)\;=\; 0.01·N(x;\, _0,\, )0.01·N\! (x;\, _0,\, )\;+\;0.99·N\! (x;\, _1,\, )\;∈\;(0,1). This turns the alignment task into an extreme mode-seeking problem: the fine-tuned model should transport probability mass from the majority mode μ1 _1 into the minority mode μ0 _0. We compare two reward-tilting strategies, both starting from the same pretrained base: • Off-policy h-guidance Guo et al. (2026b). A separate value network hϕ(xt,t)≈[r(X0)|Xt=xt]h_φ(x_t,t) [r(X_0)\,|\,X_t=x_t] is regressed on rollouts drawn exclusively from the frozen base, then applied as guidance at inference via v^=vbase−t1−t∇loghϕ v=v_base- t1-t∇ h_φ. • DOHF(ours). The same h-transform correction is instead distilled into the velocity network itself via online data collection. Figure 1: Visualization of on/off-policy h-transform guidance. Method Iterations Minority-mode fraction Mean ± std True data – 000.96% (0.98,0.98)±(0.44,0.44)(0.98,0.98)±(0.44,0.44) Base pretrained – 001.12% (0.99,0.98)±(0.42,0.41)(0.99,0.98)±(0.42,0.41) Off-policy h-guidance 2000 0014.88% (0.29,0.25)±(0.73,0.71)(0.29,0.25)±(0.73,0.71) DOHF(ours) 0200 100.00% (−0.82,−0.88)±(0.38,0.38)(-0.82,-0.88)±(0.38,0.38) Table 1: Fraction of samples falling in the minority cluster after fine-tuning. Table 1 summarizes the outcome. Off-policy guidance takes 20002000 update steps and still lifts the minority-mode fraction to some extent (1.12%→14.88%1.12\%→ 14.88\%): most of its training rollouts still land in the majority mode. In contrast, online h-distillation reaches 100%100\% minority-mode concentration after just 200200 steps — one order of magnitude fewer updates for a much better outcome: better preserves the target distribution with closer mean (−0.82,−0.88)(-0.82,-0.88) and standard deviation (0.38,0.38)(0.38,0.38). Because each iteration re-collects trajectories under tilted policy, the reward-relevant regions of state space are explored progressively as the model itself improves. For visualization, see Figure 1. 4.2 Contingency Table We consider contingency-table generation as an application of endogenous conditional sampling under the framework of Guo et al. (2026b). Classical references include Diaconis and Efron (1985); Diaconis and Sturmfels (1998b), with connections to Burnside processes and double cosets discussed in (Diaconis and Howes, 2025, Section 3). Such tables also arise in financial-network and systemic-risk applications Albanesi and Vamossy (2019); Greenwood et al. (2022). Objective. Let X=(xij)∈ℤ+m×nX=(x_ij) _+^m× n be contingency table with prescribed row and column sums ∑jxij=ri,∑ixij=cj. _jx_ij=r_i, _ix_ij=c_j. (4.1) In practice, these marginals may be noisy or partially observed, so we treat them as soft structural targets. Given a pretrained flow model vbasev^base, define 2=∑i(∑jxij−ri)2+∑j(∑ixij−cj)2.S_2= _i ( _jx_ij-r_i )^2+ _j ( _ix_ij-c_j )^2. Exact feasibility corresponds to 2=0S_2=0. Since exact matches are extremely rare for large tables, directly using the indicator reward leads to high-variance h-guidance estimates, usually there is no such table in a batch satisfying 2=0S_2=0. We therefore use the smooth reward w(x)=exp(−β2),w(x)= (- _2), where larger β more closely approximates the hard constraint. Birthday–deathday contingency table. We evaluate on the classical 12×1212× 12 birthday–deathday table of 82 descendants of Queen Victoria Diaconis and Sturmfels (1998a). The target margins are r=(6,5,5,12,12,3,10,7,3,7,9,3),c=(13,4,7,10,8,4,5,3,4,9,7,8),r=(6,5,5,12,12,3,10,7,3,7,9,3), c=(13,4,7,10,8,4,5,3,4,9,7,8), with total count 8282, the original table is provided as Table 4. We pre-train a flow model that generates nonnegative integer-valued 12×1212× 12 tables with total count 8282; implementation details are given in Appendix D.2. Starting from the same pretrained model, we compare frozen off-policy h-guidance with DOHF using the reward exp(−β2) (- _2), where we take β=0.02β=0.02. Method Updates Avg. Margin Error ↓ Max Margin Error ↓ W2W_2 vs. DG sampler Diaconis and Gangolli (1995) ↓ Uniform – 0.802 1.975 13.48 Off-policy 800 0.379 1.005 11.00 DOHF 300 0.210 0.531 9.84 Table 2: Comparison of margin matching and distributional similarity. The margin errors are standardized by the corresponding generated marginal standard deviations. Among 500,000 uniformly sampled pre-training tables, none exactly satisfies all target margins, illustrating the sparsity of the target event and the necessity of using soft targets. As shown in Table 2, uniform sampling (as how we generate pre-training data) yields an average standardized margin error of 0.802 and a maximum error of 1.975. After only 300 updates, DOHF reduces these errors to 0.210 and 0.531, respectively, outperforming the off-policy baseline, which reaches 0.379 and 1.005 after 800 updates. The Diaconis–Gangolli (DG) sampler Diaconis and Gangolli (1995) is a Markov Chain based method, generating approximately uniform contingency tables with fixed margins through repeated local 2×22× 2 integer-preserving moves, which requires initial burn-in (in our experiment 200,000 steps are used). Although we are generating tables with perturbed margins, it is still worthwhile to check the difference between generated distributions. It is shown that DOHF also achieves the smallest Wasserstein distance to the DG sampler baseline among the three methods. The advantage comes from adapting the rollout distribution toward the target region, which provides increasingly informative samples for subsequent h-transform updates. For this experiment, however, we set vϕ=vbasev_φ=v^base at each iteration rather than using the EMA update in Algorithm A, since the latter empirically caused collapse to nearly identical tables. This modification preserves rollout diversity while retaining the benefit of online h-distillation. 4.3 Image Fine-tuning Our flow-based Text-to-Image model fine-tuning task is based on SD3.5-Medium Esser et al. (2024) at 256×256256× 256 resolution. We demonstrate the potential of our method trained with the sum of three reward models: HPSv2.1 Wu et al. (2023), PickScore Kirstain et al. (2023) and ClipScore Radford et al. (2021). We will evaluate our method and other baselines on both in-domain and out-of-domain metrics: ImageReward Xu et al. (2023), Aesthetic Score Schuhmann et al. (2022), GenEval Ghosh et al. (2023) and pair-wise LPIPS Zhang et al. (2018). Note that we used average pair-wise LPIPS per prompt to measure image diversity. We aim to show that rolling out online is not only more efficient compared to off-policy fashion, but also provide a higher reward, combined with better semantic alignment. For Qualitative comparison, refer to Figure 2. SD3.5M w/ CFG DOHF w/o CFG DOHF w/ CFG FlowGRPO “Twin obese archbishops in gold space-suits, Vatican launchpad, Hasselblad style.” “Realistic Gothic cathedral in a stormy night.” “A man wearing a shirt that says “dogs”.” Figure 2: Qualitative comparison across different guidance and fine-tuning methods. The prompts are taken from Pick-a-Pic Kirstain et al. (2023). The training and testing prompt set is the ones used in Pick-a-Pic Kirstain et al. (2023). We finetuned with LoRA(α=32α=32, r=32r=32). Each iteration consists of 2828 prompts, and generates M=8M=8 images per prompt via one-step-lookahead surrogate discussed in Section 3.1. For both training and testing roll-outs, 40 time steps are applied. Additional details are provided in Appendix E. As shown in Table 3, our DOHF methodology works very well even without classifier-free-guidance (CFG) Ho and Salimans (2022) using only 2k iterations, surpassing FlowGRPO Liu et al. (2025) on most of the metrics using 9k iterations. Moreover, it performs slightly better than DiffusionNFT Zheng et al. (2026) overall in 2k iterations. To better explain their relationship, detailed discussion are provided in Appendix C. However, our online h-guidance has CFG design which is not available in DiffusionNFT. During training, we set 10%10\% of the prompts to be empty, in order to enforce model to have better unconditional generation, thus vθ(⋅,⋅,c)−vθ(⋅,⋅,∅)v_θ(·,·,c)-v_θ(·,·, ) acts like guidance. During inference, velocity field is vθ(⋅,⋅,c)+γ(vθ(⋅,⋅,c)−vθ(⋅,⋅,∅))v_θ(·,·,c)+γ (v_θ(·,·,c)-v_θ(·,·, ) ), where γ=4γ=4 is the default guidance scale. It is shown in Table 3 that with guidance, all the quality scores increase to some extent, at little cost of diversity. It is worth noting that GenEval, the out-of-domain rule-based reward measuring how accurately generated images satisfy compositional prompts, increase significantly from 44.27%44.27\% to 62.52%62.52\%, showing its potential in semantic alignment. Model #Iter GenEval PickScore ClipScore HPSv2.1 Aesthetic ImgRwd LPIPS SD3.5-M w/o CFG – 4.49% 0.179 0.178 0.150 4.513 -1.809 0.771 SD3.5-M w/ CFG – 41.35% 0.203 0.266 0.248 5.428 0.250 0.674 SD3.5-L w/ CFG – 25.00% 0.193 0.243 0.208 4.919 -0.452 0.733 DOHF w/o CFG 2k 44.27% 0.216 0.264 0.327 6.327 1.115 0.538 DOHF w/ CFG 2k 62.52% 0.218 0.277 0.336 6.240 1.329 0.521 Off-policy Guidance 2k 19.52% 0.184 0.201 0.177 4.739 -1.320 0.746 8k 38.96% 0.197 0.240 0.249 5.422 0.190 0.655 DiffusionNFT 2k 49.39% 0.214 0.261 0.305 6.151 1.118 0.525 FlowGRPO 2k 35.89% 0.205 0.252 0.272 5.924 0.636 0.602 9k 45.66% 0.211 0.259 0.302 6.162 1.020 0.588 Table 3: Evaluation Results under resolution 256×256256× 256. Bold: best; Underline: second best. 5 Conclusion We introduced DOHF, an online diffusion fine-tuning framework based on Doob’s h-transform. Our method turns reward-based adaptation into a local score or velocity correction that can be estimated online and directly distilled into the generative model. This perspective provides a unified treatment of continuous rewards, hard and soft constraints, and relative optimality objectives, while also clarifying the connection to existing methods such as DiffusionNFT. Across rare-mode adaptation, constrained contingency table generation, and text-to-image fine-tuning, DOHF consistently demonstrates the benefit of adapting the rollout distribution together with the model, particularly when reward-relevant regions are difficult to sample via base model. These results suggest that online h-transform fine-tuning offers a promising alternative to both inference-time guidance and trajectory-based reinforcement learning. More broadly, the framework provides a flexible interface between probabilistic conditioning and generative-model optimization, and may be useful for a wide range of problems involving rare events, structured constraints, black-box rewards, and distributional adaptation. References [1] S. Albanesi and D. F. Vamossy (2019) Predicting consumer default: a deep learning approach. Working Paper Technical Report 26165, Working Paper Series, National Bureau of Economic Research. External Links: Document, Link Cited by: §4.2. [2] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2024) Training diffusion models with reinforcement learning. In International Conference on Learning Representations, External Links: Link Cited by: §1. [3] J. Chang, C. Duan, Y. Jiao, Y. Xu, and J. Z. Yang (2026) Inference-time alignment for diffusion models via variationally stable doob’s matching. External Links: 2601.06514, Link Cited by: §1. [4] J. Choi, W. Guo, Y. Zhu, A. Vahdat, M. Tao, J. Berner, and Y. Chen (2026) Scaling reinforcement learning for diffusion models via velocity matching. External Links: 2608.23664, Link Cited by: §1. [5] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet (2024) Directly fine-tuning diffusion models on differentiable rewards. External Links: 2309.17400, Link Cited by: §1. [6] A. Denker, S. Padhy, F. Vargas, and J. Hertrich (2026) Iterative importance fine-tuning of diffusion models. External Links: 2502.04468, Link Cited by: §1. [7] A. Denker, F. Vargas, S. Padhy, K. Didi, S. Mathis, V. Dutordoir, R. Barbano, E. Mathieu, U. J. Komorowska, and P. Lio (2024) DEFT: efficient fine-tuning of diffusion models by learning the generalised h-transform. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §1, §1. [8] P. Diaconis and B. Efron (1985) Testing for independence in a two-way table: new interpretations of the chi-square statistic. The Annals of Statistics 13 (3), p. 845–874. External Links: Document, Link Cited by: §4.2. [9] P. Diaconis and A. Gangolli (1995) Rectangular arrays with fixed margins. In Discrete Probability and Algorithms, IMA Volumes in Mathematics and Its Applications, Vol. 72, p. 15–41. Cited by: §4.2, Table 2. [10] P. Diaconis and M. Howes (2025) Random sampling of contingency tables and partitions: two practical examples of the burnside process. Statistics and Computing 35 (6). External Links: ISSN 1573-1375, Link, Document Cited by: §4.2. [11] P. Diaconis and B. Sturmfels (1998) Algebraic algorithms for sampling from conditional distributions. The Annals of Statistics 26 (1), p. 363–397. Cited by: §D.1, §4.2. [12] P. Diaconis and B. Sturmfels (1998) Algebraic algorithms for sampling from conditional distributions. The Annals of Statistics 26 (1), p. 363–397. External Links: Document, Link Cited by: §4.2. [13] C. Domingo-Enrich, M. Drozdzal, B. Karrer, and R. T. Q. Chen (2025) Adjoint matching: fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. External Links: 2409.08861, Link Cited by: §1, §2.1. [14] J. L. Doob (1957) Conditional brownian motion and the boundary limits of harmonic functions. Bulletin de la Société Mathématique de France 85, p. 431–458. External Links: Document Cited by: §1. [15] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach (2024) Scaling rectified flow transformers for high-resolution image synthesis. External Links: 2403.03206, Link Cited by: §4.3. [16] J. Fan, S. Shen, C. Cheng, Y. Chen, C. Liang, and G. Liu (2025) Online reward-weighted fine-tuning of flow matching with wasserstein regularization. External Links: 2502.06061, Link Cited by: §1. [17] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee (2023) DPOK: reinforcement learning for fine-tuning text-to-image diffusion models. External Links: 2305.16381, Link Cited by: §1. [18] Z. Fu, R. He, W. Shang, X. Ma, L. Wang, C. Liu, and S. Fu (2026) FlowAWR: online adaptive flow reinforcement via advantage-weighted rectification. External Links: 2606.30376, Link Cited by: §1. [19] D. Ghosh, H. Hajishirzi, and L. Schmidt (2023) GenEval: an object-focused framework for evaluating text-to-image alignment. External Links: 2310.11513, Link Cited by: §4.3. [20] R. Greenwood, S. G. Hanson, A. Shleifer, and J. A. Sørensen (2022) Predictable financial crises. The Journal of Finance 77 (2), p. 863–921. External Links: Document Cited by: §4.2. [21] Z. Guo, J. Sheng, D. D. Yao, and W. Tang (2026) Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline. External Links: 2605.06583, Link Cited by: §1. [22] Z. Guo, W. Tang, and R. Xu (2026) Conditional diffusion guidance under hard constraint: a stochastic analysis approach. arXiv preprint arXiv:2602.05533. External Links: Link Cited by: §1, §1, §2.1, 1st item, §4.2. [23] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. External Links: 2207.12598, Link Cited by: §3.2, §4.3. [24] P. Holderrieth, D. Chen, L. Eyring, I. Shah, G. Anantharaman, Y. He, Z. Akata, T. Jaakkola, N. M. Boffi, and M. Simchowitz (2026) Diamond maps: efficient reward alignment via stochastic flow maps. External Links: 2602.05993, Link Cited by: §1. [25] C. Huang, J. H. Lim, and A. Courville (2021) A variational perspective on diffusion-based generative models and score matching. In Advances in Neural Information Processing Systems, Vol. 34. External Links: Link Cited by: §3. [26] H. Kim, T. Lee, H. Kim, and G. D. Veciana (2025) Generating informative samples for risk-averse fine-tuning of downstream tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix B, §1, §2.1. [27] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy (2023) Pick-a-pic: an open dataset of user preferences for text-to-image generation. External Links: 2305.01569, Link Cited by: Figure 2, Figure 2, §4.3, §4.3. [28] J. Lee, J. Chang, J. Kim, and J. C. Ye (2026) Reward score matching: unifying reward-based fine-tuning for flow and diffusion models. External Links: 2604.17415, Link Cited by: §1. [29] S. Levine (2018) Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv preprint arXiv:1805.00909. External Links: Link Cited by: §2.1. [30] J. Li, Y. Cui, T. Huang, W. Kong, Y. Cheng, C. Zeng, Y. Ma, C. Fan, M. Yang, Z. Zhong, and L. Bo (2026) MixGRPO: unlocking flow-based grpo efficiency with mixed ode-sde. External Links: 2507.21802, Link Cited by: §1. [31] X. Li, Y. Zhao, C. Wang, G. Scalia, G. Eraslan, S. Nair, T. Biancalani, S. Ji, A. Regev, S. Levine, and M. Uehara (2024) Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. External Links: 2408.08252, Link Cited by: §1. [32] J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025) Flow-grpo: training flow matching models via online rl. External Links: 2505.05470, Link Cited by: §1, §4.3. [33] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2025) DPM-solver++. Machine Intelligence Research 22 (4), p. 730–751. External Links: ISSN 2731-5398, Link, Document Cited by: Appendix E. [34] J. Pachebat, G. Conforti, A. Durmus, and Y. Janati (2025) Iterative tilting for diffusion fine-tuning. External Links: 2512.03234, Link Cited by: §1. [35] P. Potaptchik, C. Lee, and M. S. Albergo (2025) Tilt matching for scalable sampling and fine-tuning. External Links: 2512.21829, Link Cited by: §1. [36] P. Potaptchik, A. Saravanan, A. Mammadov, A. Prat, M. S. Albergo, and Y. W. Teh (2026) Meta flow maps enable scalable reward alignment. External Links: 2601.14430, Link Cited by: §1. [37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. External Links: 2103.00020, Link Cited by: §4.3. [38] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev (2022) LAION-5b: an open large-scale dataset for training next generation image-text models. External Links: 2210.08402, Link Cited by: §4.3. [39] J. Sheng, H. Zhao, H. Chen, D. D. Yao, and W. Tang (2025) Understanding sampler stochasticity in training diffusion models for RLHF. arXiv preprint arXiv:2510.10767. Cited by: §1. [40] R. Singhal, Z. Horvitz, R. Teehan, M. Ren, Z. Yu, K. McKeown, and R. Ranganath (2025) A general framework for inference-time scaling and steering of diffusion models. External Links: 2501.06848, Link Cited by: §1. [41] W. Tang and H. Zhao (2025) Score-based diffusion models via stochastic differential equations. Statistic Surveys 19, p. 28–64. Cited by: §3. [42] M. Uehara, Y. Zhao, T. Biancalani, and S. Levine (2024) Understanding reinforcement learning-based fine-tuning of diffusion models: a tutorial and review. arXiv preprint arXiv:2407.13734. Cited by: §1. [43] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik (2023) Diffusion model alignment using direct preference optimization. External Links: 2311.12908, Link Cited by: §1. [44] G. I. Winata, H. Zhao, A. Das, W. Tang, D. D. Yao, S. Zhang, and S. Sahu (2025) Preference tuning with human feedback on language, speech, and vision tasks: a survey. J. Artificial Intelligence Res. 82, p. 2595–2661. Cited by: §1. [45] L. Wu, B. L. Trippe, C. A. Naesseth, D. M. Blei, and J. P. Cunningham (2024) Practical and asymptotically exact conditional sampling in diffusion models. External Links: 2306.17775, Link Cited by: §1. [46] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. External Links: 2306.09341, Link Cited by: §4.3. [47] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong (2023) ImageReward: learning and evaluating human preferences for text-to-image generation. External Links: 2304.05977, Link Cited by: §1, §4.3. [48] Y. Xu, Y. Zhang, S. Luo, L. Wang, and D. He (2026) Designing reinforcement learning for diffusion models: a unified path-space view. External Links: 2608.14430, Link Cited by: §1. [49] S. Xue, C. Ge, S. Zhang, Y. Li, and Z. Ma (2026) Advantage weighted matching: aligning rl with pretraining in diffusion models. External Links: 2509.25050, Link Cited by: §1. [50] Z. Xue, J. Wu, Y. Gao, F. Kong, L. Zhu, M. Chen, Z. Liu, W. Liu, Q. Guo, W. Huang, and P. Luo (2025) DanceGRPO: unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818. Cited by: §1. [51] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. External Links: 1801.03924, Link Cited by: §4.3. [52] H. Zhao, H. Chen, J. Zhang, D. D. Yao, and W. Tang (2024) Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning. arXiv preprint arXiv:2409.08400. Cited by: §1. [53] H. Zhao, H. Chen, J. Zhang, D. D. Yao, and W. Tang (2025) Scores as Actions: fine tuning diffusion generative models by continuous-time reinforcement learning. In ICML, Cited by: §1. [54] K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu (2026) DiffusionNFT: online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117. External Links: Link Cited by: Appendix C, Appendix E, §1, §2.1, §3.3, §4.3. [55] W. Zhou, X. Zhu, L. Kong, B. Chen, L. Zhang, Y. Liang, X. Hou, Y. Tian, X. Sun, Y. Wang, L. Li, S. Wu, L. Qu, F. Li, W. Liu, J. McAuley, and T. Chua (2026) On-policy self-distillation in diffusion models. External Links: 2608.24646, Link Cited by: §1. [56] Q. Zhu, Z. Ye, H. Liu, Z. Wang, and M. Chen (2026) Training-free adaptation of diffusion models via doob’s h-transform. External Links: 2602.16198, Link Cited by: §1, §1, §1, §3.1. Appendix A Relation between score and velocity Consider the rectified-flow interpolation Xt=(1−t)X0+tϵ,ϵ∼(0,I).X_t=(1-t)X_0+tε, ε (0,I). The corresponding velocity field is v(x,t)=[ϵ−X0∣Xt=x].v(x,t)=E\! [ε-X_0 X_t=x ]. Conditioned on X0X_0, we have Xt|X0∼((1−t)X0,t2I),X_t X_0 \! ((1-t)X_0,t^2I ), and hence ∇xlogpt(x∣X0)=−x−(1−t)X0t2=−ϵt. _x p_t(x X_0)=- x-(1-t)X_0t^2=- εt. Using the score marginalization identity, ∇xlogpt(x)=[∇xlogpt(x∣X0)∣Xt=x], _x p_t(x)=E\! [ _x p_t(x X_0) X_t=x ], we obtain ∇xlogpt(x)=−1t[ϵ∣Xt=x]. _x p_t(x)=- 1tE\! [ε X_t=x ]. Therefore, [ϵ∣Xt=x]=x+(1−t)[ϵ−X0∣Xt=x]=x+(1−t)v(x,t).E\! [ε X_t=x ]=x+(1-t)E\! [ε-X_0 X_t=x ]=x+(1-t)v(x,t). Substituting this into the previous expression gives ∇xlogpt(x)=−x+(1−t)v(x,t)t. _x p_t(x)=- x+(1-t)v(x,t)t. Therefore, we have v(x,t)=−x+t∇xlogpt(x)1−t.v(x,t)=- x+t _x p_t(x)1-t. Appendix B Alternative Estimator of ∇xlogh _x h [26]: Approximate h(x,t,c)≈w(x^0(x,t,c))h(x,t,c)≈ w( x_0(x,t,c)) via the Tweedie point estimate x^0(x,t,c)=x−tvold(x,t,c), x_0(x,t,c)=x-t\,v^old(x,t,c), so that ∇xlogh(x,t,c)≈[∂x^0(x,t,c)]⊤u _x h(x,t,c)≈ [ _x x_0(x,t,c) ] u, with u:=∇x^0logw(x^0(x,t,c))u:= _ x_0 w( x_0(x,t,c)). Moreover, we should avoid the Jacobian ∂x^0 _x x_0 via a finite-difference step of size ϵ>0ε>0: ∇logh^B=11−t[u+t2ϵ(∇xlogptold(x+ϵu∣c)−∇xlogptold(x∣c))], ∇ h_B= 11-t [u+ t^2ε ( _x p_t^old(x+ε u c)- _x p_t^old(x c) ) ], (B.1) where ∇xlogptold(x∣c)=−x+(1−t)vold(x,t,c)t _x p_t^old(x c)=- x+(1-t)v^old(x,t,c)t. This method doesn’t require M rollouts in Estimator 3.2, only 2 evaluations of voldv^old plus one backward pass through w. However, it requires w to be differentiable. Appendix C Relation to Diffusion Negative-aware Fine-Tuning (NFT) DiffusionNFT[54] arrives at a guidance mechanism built from the optimality variable r(x0,c)=p(O=1∣x0,c)r(x_0,c)=p(O=1 x_0,c), which is exactly our weight w(x0)w(x_0). We show that its guidance direction, and both its single-branch and double-branch training objectives, are special cases of the h-transform framework developed above. C.1 Distribution split coefficient α(xt)α(x_t) is equvalent to h(xt,t,c)h(x_t,t,c) DiffusionNFT defines π+(x0∣c):=πold(x0∣O=1,c)π^+(x_0 c):=π^old(x_0 O=1,c) and π−(x0∣c):=πold(x0∣O=0,c)π^-(x_0 c):=π^old(x_0 O=0,c). It can be shown that the posterior distribution of old policy can be seperated as π0|told(x0∣xt,c)=α(xt)π0|t+(x0∣xt,c)+[1−α(xt)]π0|t−(x0∣xt,c),π^old_0 t(x_0 x_t,c)=α(x_t)π^+_0 t(x_0 x_t,c)+[1-α(x_t)]π^-_0 t(x_0 x_t,c), where α(xt):=πt+(xt∣c)πtold(xt∣c)⋅πold(x0∣c)[w(x0)].α(x_t):= _t^+(x_t c) _t^old(x_t c)·E_π^old(x_0 c)[w(x_0)]. Also, vold(xt,t,c)=α(xt)v+(xt,t,c)+[1−α(xt)]v−(xt,t,c).v^old(x_t,t,c)=α(x_t)v^+(x_t,t,c)+[1-α(x_t)]v^-(x_t,t,c). (C.1) Since π+(x0∣c)=pold(x0∣O=1,c)=pft(x0∣c)π^+(x_0 c)=p^old(x_0 O=1,c)=p^ft(x_0 c) by definition, we have πt+=ptftπ^+_t=p^ft_t, and from equation 2.1, ptft(xt∣c)ptold(xt∣c)=h(xt,t,c)Z(c). p_t^ft(x_t c)p_t^old(x_t c)= h(x_t,t,c)Z(c). Since πold(x0∣c)[w(x0)]=Z(c)E_π^old(x_0 c)[w(x_0)]=Z(c), substituting gives α(xt)=h(xt,t,c)Z(c)⋅Z(c)=h(xt,t,c).α(x_t)= h(x_t,t,c)Z(c)· Z(c)=h(x_t,t,c). As noted above, πt+≡ptftπ^+_t≡ p_t^ft as distributions, hence their posteriors over x0x_0 given xtx_t coincide, and since the velocity field is a linear functional of the posterior mean, v+(xt,t,c)=vft(xt,t,c)=vbase(xt,t,c)−t1−t∇xlogh(xt,t,c).v^+(x_t,t,c)=v^ft(x_t,t,c)=v^base(x_t,t,c)- t1-t _x h(x_t,t,c). C.2 The Guidance Direction Δ and v−v^- as Anti-Guidance According to equation C.1, vold,v+,v−v^old,v^+,v^- are on the same line. DiffuionNFT defines velocity improvement direction Δ as: Δ:=α(xt)[v+−vold]=[1−α(xt)][vold−v−]. :=α(x_t)\,[v^+-v^old]=[1-α(x_t)]\,[v^old-v^-]. Using α=hα=h and v+−vold=vft−vold=−t1−t∇xloghv^+-v^old=v^ft-v^old=- t1-t _x h, Δ=h⋅(−t1−t∇xlogh)=−t1−th∇xlogh=−t1−t∇xh, =h· (- t1-t _x h )=- t1-t\,h\, _x h=- t1-t _xh, using h∇logh=∇h∇ h=∇ h. That is, Δ is the (unnormalized) gradient ∇xh _xh, rescaled by −t1−t- t1-t. For v−v^-: by the same construction with w−(x0):=1−w(x0)w^-(x_0):=1-w(x_0), π−=pold(⋅∣O=0,c)π^-=p^old(· O=0,c) has h−(xt,t,c)=1−h(xt,t,c)h^-(x_t,t,c)=1-h(x_t,t,c), and v−(xt,c,t)=vold(xt,t,c)−t1−t∇xlog(1−h(xt,t,c)).v^-(x_t,c,t)=v^old(x_t,t,c)- t1-t _x (1-h(x_t,t,c) ). Substituting into the first form of Δ , with ∇xlog(1−h)=−∇xh/(1−h) _x (1-h)=- _xh/(1-h), −[1−h][vold−v−]=−[1−h]⋅t1−t⋅∇xh1−h=−t1−t∇xh,-[1-h] [v^old-v^- ]=-[1-h]· t1-t· _xh1-h=- t1-t _xh, confirming consistency. Thus v−v^- is precisely the h-transform velocity for the complementary event O=0\O=0\: it is not an independent construction but an “anti-guidance” field pointing away from the optimal region. C.3 Double-Branch Optimal Velocity DiffusionNFT trains vθv_θ (it acts as an anchor for improvement guidance while training, and used for inference after training) via ℒ(θ)=c,πold(x0∣c),t[w(x0)‖vθ+−v‖22+(1−w(x0))‖vθ−v‖22],L(θ)=E_c,π^old(x_0 c),t [w(x_0)\,\|v_θ^+-v\|_2^2+(1-w(x_0))\,\|v_θ^--v\|_2^2 ], where vθ+:=(1−β)vold+βvθ,vθ−:=(1+β)vold−βvθ.v_θ^+:=(1-β)v^old+β v_θ, v_θ^-:=(1+β)v^old-β v_θ. (C.2) The ’good velocity’ is parametrized as a linear combination between anchor vθv_θ and voldv^old, and ’bad velocity’ is pushing voldv^old to the opposite direction from the ’good’ one. By definition pft(x0∣c)=w(x0)Z(c)pold(x0∣c)p^ft(x_0 c)= w(x_0)Z(c)p^old(x_0 c), i.e. w(x0)pold(x0∣c)=Z(c)pft(x0∣c).w(x_0)\,p^old(x_0 c)=Z(c)\,p^ft(x_0 c). Multiplying both sides by the forward transition kernel ϕ(xt∣x0)φ(x_t x_0), which is shared by poldp^old and pftp^ft, and applying Bayes’ rule p(x0∣c)ϕ(xt∣x0)=pt(xt∣c)p0|t(x0∣xt,c)p(x_0 c)φ(x_t x_0)=p_t(x_t c)\,p_0|t(x_0 x_t,c) on each side: w(x0)ptold(xt∣c)π0|told(x0∣xt,c)=Z(c)ptft(xt∣c)π0|t+(x0∣xt,c).w(x_0)\,p_t^old(x_t c)\,π^old_0|t(x_0 x_t,c)=Z(c)\,p_t^ft(x_t c)\,π^+_0|t(x_0 x_t,c). Dividing both sides by ptold(xt∣c)p_t^old(x_t c) and using ptft(xt∣c)ptold(xt∣c)=h(xt,t,c)Z(c) p_t^ft(x_t c)p_t^old(x_t c)= h(x_t,t,c)Z(c) in Equation 2.1 w(x0)π0|told(x0∣xt,c)=Z(c)⋅h(xt,t,c)Z(c)⋅π0|t+(x0∣xt,c)=h(xt,t,c)π0|t+(x0∣xt,c).w(x_0)\,π^old_0|t(x_0 x_t,c)=Z(c)· h(x_t,t,c)Z(c)·π^+_0|t(x_0 x_t,c)=h(x_t,t,c)\,π^+_0|t(x_0 x_t,c). (C.3) The same argument with w replaced by 1−w1-w (equivalently h replaced by 1−h1-h, π+π^+ by π−π^-) gives (1−w(x0))π0|told=(1−h(xt,t,c))π0|t−(1-w(x_0))\,π^old_0|t=(1-h(x_t,t,c))\,π^-_0|t. Reduction to a weighted regression at fixed xtx_t. In NFT, loss function is defined as ℒ(θ,c)=πold(x0,xt∣c)[w(x0)‖vθ+(xt,t,c)−v‖22+(1−w(x0))‖vθ−(xt,t,c)−v‖22]L(θ,c)=E_π^old(x_0,x_t c)[w(x_0)\|v_θ^+(x_t,t,c)-v\|_2^2+(1-w(x_0))\|v_θ^-(x_t,t,c)-v\|_2^2] Since the joint density πold(x0,xt∣c)π^old(x_0,x_t c) factorizes as πtold(xt∣c)π0|told(x0∣xt,c) _t^old(x_t c)\,π^old_0|t(x_0 x_t,c), the inner expectation of the positive term at fixed (xt,t,c)(x_t,t,c) is ∫w(x0)π0|told(x0∣xt,c)‖vθ+−v‖2dx0=h(xt,t,c)∫π0|t+(x0∣xt,c)‖vθ+−v‖2dx0=hπ0|t+[‖vθ+−v‖2], w(x_0)\,π^old_0|t(x_0 x_t,c)\,\|v_θ^+-v\|^2\,dx_0=h(x_t,t,c) π^+_0|t(x_0 x_t,c)\,\|v_θ^+-v\|^2\,dx_0=h\,E_π^+_0|t [\|v_θ^+-v\|^2 ], using the identity C.3 and vθ+v_θ^+ being independent of x0x_0. Since vθ+v_θ^+ is a constant w.r.t. x0x_0, π0|t+[‖vθ+−v‖2]=‖vθ+−π0|t+[v]‖2+Varπ0|t+(v)⏟independent of θ=‖vθ+−v+‖2+C+,E_π^+_0|t [\|v_θ^+-v\|^2 ]= \|v_θ^+-E_π^+_0|t[v] \|^2+ Var_π^+_0|t(v)_independent of θ=\|v_θ^+-v^+\|^2+C_+, where π0|t+[v]=v+(xt,c,t)E_π^+_0|t[v]=v^+(x_t,c,t) by definition of the velocity field. The negative term reduces identically to (1−h)[‖vθ−v−‖2+C−](1-h) [\|v_θ^--v^-\|^2+C_- ]. Dropping the θ-independent constants, the loss at fixed xt,t,cx_t,t,c is equivalent to minimizing h‖vθ+−v+‖22+(1−h)‖vθ−v−‖22.h\,\|v_θ^+-v^+\|_2^2+(1-h)\,\|v_θ^--v^-\|_2^2. Let Y:=vθ−voldY:=v_θ-v^old. Let D:=v+−voldD:=v^+-v^old and D−:=v−voldD^-:=v^--v^old. Substituting the parameterization, vθ+−v+=(1−β)vold+βvθ−(vold+D)=β(vθ−vold)−D=β(Y−Dβ),v_θ^+-v^+=(1-β)v^old+β v_θ-(v^old+D)=β(v_θ-v^old)-D=β (Y- Dβ ), vθ−v−=(1+β)vold−βvθ−(vold+D−)=−β(vθ−vold)−D−=−β(Y+D−β).v_θ^--v^-=(1+β)v^old-β v_θ-(v^old+D^-)=-β(v_θ-v^old)-D^-=-β (Y+ D^-β ). Hence ‖vθ+−v+‖2=β2‖Y−D/β‖2\|v_θ^+-v^+\|^2=β^2\|Y-D/β\|^2 and ‖vθ−v−‖2=β2‖Y+D−/β‖2\|v_θ^--v^-\|^2=β^2\|Y+D^-/β\|^2; factoring out β2>0β^2>0 (which does not affect the minimizer) gives the weighted least-squares problem f(Y)=h‖Y−Dβ‖22+(1−h)‖Y+D−β‖22.f(Y)=h \|Y- Dβ \|_2^2+(1-h) \|Y+ D^-β \|_2^2. Solving for the minimizer. f is a weighted sum of squared distances to two points, D/βD/β (weight h) and −D−/β-D^-/β (weight 1−h1-h); its minimizer is the weighted average Y∗=h⋅Dβ+(1−h)⋅(−D−β)=hD−(1−h)D−β.Y^*=h· Dβ+(1-h)· (- D^-β )= hD-(1-h)D^-β. Since vold=hv++(1−h)v−v^old=h\,v^++(1-h)\,v^-, subtracting voldv^old from both sides gives 0=hD+(1−h)D−0=hD+(1-h)D^-, i.e. hD=−(1−h)D−hD=-(1-h)D^-. Substituting, Y∗=hD−(1−h)D−β=hD+hDβ=2hDβY^*= hD-(1-h)D^-β= hD+hDβ= 2hDβ ⟹vθ∗=vold−2h(xt,t,c)β⋅t1−t∇xlogh(xt,t,c). v_θ^*=v^old- 2h(x_t,t,c)β· t1-t _x h(x_t,t,c). For fixed scalar β, this is a state-dependent damped version of vftv^ft: the guidance strength scales with the local optimality probability h, vanishing where h→0h→ 0. Therefore our algorithm is choosing β in the Definition C.2 to be 2h(xt,t,c)2h(x_t,t,c). This is reasonable because when you have (xt,t∣c)(x_t,t c) is more likely to be optimal, that is h(xt,t,c)>0.5h(x_t,t,c)>0.5, we are more confident to push fine-tuned sampling strategy vθv_θ to be farther on the current improvement direction. On the other hand if h(xt,t,c)<0.5h(x_t,t,c)<0.5, we are not willing to trust vθv_θ that much and keep closer to voldv^old. However, in DiffusionNFT, they don’t estimate h-function explicitly, so choosing β as a function depending on (xt,t,c)(x_t,t,c) is not viable. They have to leave β as a hyper-parameter (in their implementation, β=1β=1 is default value). Our algorithm provided more flexibility on this aspect. C.4 Single-Branch Optimal Velocity Training on the positive branch alone, ℒ+(θ)=c,t,πold(x0∣c)[w‖vθ+−v‖22]L^+(θ)=E_c,t,π^old(x_0 c) [w\,\|v_θ^+-v\|_2^2 ], and using wπ0|told=hπ0|t+w\,π^old_0|t=h\,π^+_0|t, ℒ+(θ)=c,t,xt[h(xt,t,c)⋅x0∼π0|t+‖vθ+−v‖22].L^+(θ)=E_c,t,x_t [h(x_t,t,c)·E_x_0 π^+_0|t\|v_θ^+-v\|_2^2 ]. Pointwise in xtx_t, the L2L^2-minimizer is the conditional mean regardless of the (positive) weight h: vθ+∗=π0|t+[v]=v+=vft⟹vθ∗=vbase+Dβ=vbase−1β⋅t1−t∇xlogh,v_θ^+*=E_π^+_0|t[v]=v^+=v^ft v_θ^*=v^base+ Dβ=v^base- 1β· t1-t _x h, which is exact, with no h-dependent damping, for any fixed β. C.5 Unifying the Three via h Comparing the two population optima: double-branch: vθ∗=vbase−2hβ⋅t1−t∇logh,single-branch: vθ∗=vbase−1β⋅t1−t∇logh.double-branch: v_θ^*=v^base- 2hβ· t1-t∇ h, -branch: v_θ^*=v^base- 1β· t1-t∇ h. Setting β(xt,t,c)=2h(xt,t,c)β(x_t,t,c)=2h(x_t,t,c) in the double-branch objective exactly cancels the damping factor, giving vθ∗=vftv_θ^*=v^ft; equivalently, this coincides with the single-branch optimum at β=1β=1. Thus double-branch,β=2h⟺single-branch,β=1⟺vθ∗=vft=vbase−t1−t∇xlogh.double-branch,\ β=2h -branch,\ β=1 v_θ^*=v^ft=v^base- t1-t _x h. Appendix D Discussion on Contingency Table D.1 Birthday-Deathday Data The following Table 4 is taken from the classical contingency-table example of [11]. Birth \ Death Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Jan 1 0 0 0 1 2 0 0 1 0 1 0 6 Feb 1 0 0 1 0 0 0 0 0 1 0 2 5 Mar 1 0 0 0 2 1 0 0 0 0 0 1 5 Apr 3 0 2 0 0 0 1 0 1 3 1 1 12 May 2 1 1 1 1 1 1 1 1 1 1 0 12 Jun 2 0 0 0 1 0 0 0 0 0 0 0 3 Jul 2 0 2 1 0 0 0 0 1 1 1 2 10 Aug 0 0 0 3 0 0 1 0 0 1 0 2 7 Sep 0 0 0 1 1 0 0 0 0 0 1 0 3 Oct 1 1 0 2 0 0 1 0 0 1 1 0 7 Nov 0 1 1 1 2 0 0 2 0 1 1 0 9 Dec 0 1 1 0 0 0 1 0 0 0 0 0 3 Total 13 4 7 10 8 4 5 3 4 9 7 8 82 Table 4: Birthday and death month of 82 descendants of Queen Victoria. D.2 Pretraining the Contingency-Table Flow Model To obtain the pretrained generative model used in the contingency-table experiment, we first construct a training distribution over 12×1212× 12 nonnegative tables with a fixed total count N=82N=82. For each training sample, we randomly generate an integer table X∈ℤ+12×12X _+^12× 12 satisfying ∑i,jXij=N. _i,jX_ij=N. We then normalize the table by its total count, Pij=XijN,∑i,jPij=1,P_ij= X_ijN, _i,jP_ij=1, so that each table can be viewed as a probability vector on the 144144-dimensional simplex. A continuous flow model is pretrained on these normalized probability tables. At inference time, the generated probability table P P is converted back to an integer contingency table with total count exactly N. We first compute Yij=NP^ij,X~ij=⌊Yij⌋.Y_ij=N P_ij, X_ij= Y_ij . Since rounding down may leave a residual R=N−∑i,jX~ij,R=N- _i,j X_ij, we assign one additional count to the R entries having the largest fractional parts Yij−⌊Yij⌋.Y_ij- Y_ij . This largest-remainder rounding procedure guarantees ∑i,jX~ij=N _i,j X_ij=N while keeping the resulting integer table as close as possible to the continuous model output under this rounding scheme. If the total number is not hard-constrained to be the total number, we could let Yij=NP^ij,X~ij=round(Yij).Y_ij=N P_ij, X_ij=round($Y_ij$). Appendix E Additional Details on Text-to-Image Task ODE Integration. Roll-out time steps are 4040 for both training and inference. Note that we only solve the flow ODE step-by-step till ∇logh∇ h, then use one step estimation no matter using Estimator 3.2 or B.1. However, the solver used in training roll-outs is Euler, and during inference is DPM-Solver++++[33], and CFG scale for online h-guidance w/ CFG is 44 by default, for SD3.5-M is 3.53.5 by default (both of them are taken due to the relatively best performance among different metrics). Online Data Collection. Slow mixed policy is used to generate online samples. Follow the default design used in DiffusionNFT[54], ηk=min(0.001⋅k,0.5) _k= (0.001· k,0.5), which means the data collection starts from on-policy, and gradually converges to 0.5⋅vold+0.5⋅vθ0.5· v^old+0.5· v_θ after 500 iterations. On the other hand, for the off policy guidance baseline, the roll-out policy and regression target are based on the base model vbasev^base. Other Configurations. In our T2I implementation, 7 Nvidia L40S is used. For each iteration per GPU, 4 prompts are randomly picked, and 12 initial noise is generated for each prompt. We do not use all time steps in training because two denominators in Flow-equivalent SDE 3.1 will approach to zero when t→1t→ 1, therefore we set tmin=0.05t_ =0.05 and tmax=0.9t_ =0.9. Moreover, stochasticity parameter λt=1 _t=1 in SDE 3.1. Learning Rate is 3e−53e-5, Gradient Clip is set to be 1.01.0, and slight LoRA regularization 1e−51e-5 is applied. Appendix F Additional Qualitative Comparison In this section, we will provide more images showing the aesthetic quality and semantic alignment provided by our algorithm. For baselines, SD3.5-Medium, DiffusionNFT, FlowGRPO and off-policy h-guidance are compared in images. For aesthetic quality, refer to Figure 3. For semantic alignment, refer to Figure 4. Prompt 1 Prompt 2 Prompt 3 Prompt 4 SD3.5M (w/o CFG) SD3.5M (w/ CFG) DiffusionNFT FlowGRPO Off-policy h-guidance DOHF (w/o CFG) DOHF (w/ CFG) Figure 3: Qualitative comparison across different methods and prompts. The prompts are: (Prompt 1) “Pixel art of a space station launching with two suns and a comet, featuring a retro space helmet, minigun, and a bright explosion in a dream-like atmosphere with CGA shading.”; (Prompt 2) “The devil using a heat press.”; (Prompt 3) “Ink drawing of a Shaolin monk in fighting pose.”; (Prompt 4) “A side view of a fox, with a flat design, Artwork of t-shirt graphic.” Prompt 1 Prompt 2 Prompt 3 Prompt 4 SD3.5M (w/o CFG) SD3.5M (w/ CFG) DiffusionNFT FlowGRPO Off-policy h-guidance DOHF (w/o CFG) DOHF (w/ CFG) Figure 4: Qualitative comparison across different methods and prompts. The prompts are: (Prompt 1, color) “A young woman with red hairs and green eyes”; (Prompt 2, spatial) “A penguin astronaut, in a penguin spacesuit, standing next to the apollo landing”; (Prompt 3, text) “A pikachu holding a sign that says ”I’m cute!”; (Prompt 4, text) “Gandalf holding a sign with a text ”you shall not pass”.