Paper deep dive
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Yansen Han, Shengyi Liao, Peng Sun, Deyuan Liu, Yuanxing Zhang, Pengfei Wan, Tao Lin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/10/2026, 3:33:26 AM
Summary
The paper introduces FlowCPO, an offline preference alignment method for flow models that utilizes a unified divergence-based framework. It addresses the gap in offline forward-KL objectives by deriving a nonnegative contrastive flow matching surrogate that uses both preferred and dispreferred samples without online rollouts. Theoretical analysis shows FlowCPO is bounded below, unlike simplified FlowDPO. Empirical results demonstrate superior GenEval and OCR scores compared to baselines like FlowDPO and RFT in in-domain settings.
Entities (14)
Relation Signals (11)
FlowCPO → isa → Offline Preference Alignment Method
confidence 100% · introduce FlowCPO, an offline forward-KL objective
FlowCPO → achieveshigherscoreon → GenEval
confidence 95% · FlowCPO achieves higher mean GenEval... scores
FlowCPO → achieveshigherscoreon → OCR
confidence 95% · FlowCPO achieves higher mean... OCR scores
FlowCPO → developedby → Tao Lin
confidence 95% · Tao Lin∗Equal contribution†Corresponding author
FlowCPO → developedby → Yansen Han
confidence 95% · Yansen Han Affiliation: Westlake University
FlowGRPO → isa → Online Preference Alignment Method
confidence 95% · online reinforcement learning frameworks like FlowGRPO
FlowDPO → isa → Offline Preference Alignment Method
confidence 95% · offline preference optimization methods such as FlowDPO
FlowCPO → uses → Forward-KL Objective
confidence 95% · FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples
FlowCPO → isboundedbelowby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.
Tags
Links
- Source: https://arxiv.org/abs/2609.09905v1
- Canonical: https://arxiv.org/abs/2609.09905v1
Trouble viewing inline? Open PDF directly →
Full Text
142,422 characters extracted from source content.
Expand or collapse full text
_set:Ne tcboxmath _set:Ne FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models Yansen Han Affiliation: Westlake University Affiliation: Zhejiang University Shengyi Liao Affiliation: Kling Team, Kuaishou Technology Peng Sun Affiliation: Westlake University Affiliation: Zhejiang University Deyuan Liu Affiliation: Westlake University Yuanxing Zhang Affiliation: Kling Team, Kuaishou Technology Pengfei Wan Affiliation: Kling Team, Kuaishou Technology Tao Lin∗Equal contribution†Corresponding author Affiliation: Westlake University Abstract Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics. 1 Introduction Recent advancements in flow matching and diffusion-based generative modeling [18, 30, 1, 40, 35, 20] have facilitated the development of practical preference alignment algorithms for continuous generative spaces [32, 55, 33, 51, 3]. These efforts have yielded diverse alignment paradigms, ranging from online reinforcement learning frameworks like FlowGRPO [32], DiffusionNFT [55], AWM [51] and RAM [3] to offline preference optimization methods such as FlowDPO [33]. In contrast to alignment in discrete autoregressive models, these continuous-time methods operate directly on sampling trajectories and velocity fields rather than token-level probabilities. While empirical results are promising, the relations among these methods remain difficult to compare. Table 1: Objective-level view of flow-based preference alignment. The two axes separate the sampling regime from the direction of the idealized KL objective. Divergence Online Offline Reverse-KL FlowGRPO FlowDPO Forward-KL DiffusionNFT FlowCPO (ours) Existing flow-based alignment methods can be organized along two axes (Tab. 1): the sampling regime (online versus offline) and the direction of the idealized KL objective. Under this view, FlowGRPO [32] and FlowDPO [33] represent online and offline instances of the reverse-KL branch, whereas DiffusionNFT [55] provides a forward-process objective but refreshes its training samples online. The unresolved problem is therefore the offline forward-KL objective using both preferred and dispreferred target samples: a forward-KL expectation can use fixed target samples, but its log-likelihood is intractable for flow matching. An offline method needs a justified bridge from this distribution-level objective to velocity-field regression on fixed preferred and dispreferred data. We address this gap by establishing a unified divergence-based framework for flow-based preference alignment and deriving Flow Contrastive Preference Optimization (FlowCPO). Starting from forward-KL terms over preferred and dispreferred target distributions, we use a flow matching bound for linear interpolation to obtain a contrastive regression surrogate. The resulting loss can be estimated from a fixed preference dataset and therefore requires no sampling from the model during training. We also connect FlowCPO to simplified FlowDPO (App. B.5). Simplified FlowDPO subtracts the regression error on dispreferred samples and can be unbounded below, whereas FlowCPO adds two nonnegative regression errors. With equal branch weights (λ=1λ=1), our loss decomposition identifies corrections determined by the difference between the current model and its exponential moving average (EMA). Empirically, we evaluate our approach under a strictly offline protocol. In the in-domain setting, where preference pairs are generated by the reference model, FlowCPO attains higher mean GenEval and OCR scores than RFT and FlowDPO while remaining strong on general-preference metrics. In the out-of-domain setting, the result is more mixed: FlowCPO gives the best GenEval score but RFT is stronger on several reward metrics. The overall offline optimization pipeline is illustrated in Fig. 1. Our contributions are threefold: • We provide a unified divergence-based framework that separates the online/offline sampling regime from the direction of the idealized KL objective, clarifying the assumptions behind connections among existing flow-based alignment methods. • We derive FlowCPO, a nonnegative contrastive surrogate for offline forward-KL alignment. Under stated regularity conditions for linear interpolation, its regression objective bounds the branch-wise forward-KL objective and is estimable from fixed preference data. We also derive its loss decomposition relative to simplified FlowDPO. • We evaluate FlowCPO with two types of offline data, showing higher GenEval and OCR scores than FlowDPO in the in-domain setting and mixed results in the out-of-domain setting. Figure 1: FlowCPO uses a fixed preference dataset to train two coupled flow branches. A frozen generator supplies preferred/dispreferred pairs =(c,0w,0l)D=\(c,x_0^w,x_0^l)\; the positive branch matches preferred samples, while a mirrored branch matches dispreferred samples. Training uses only noised versions of these fixed endpoints and requires no online rollout. 2 Related Work Online alignment with policy-generated samples. Online methods improve a generator using samples drawn from its current or recent policy. ReFL [50] uses reward feedback, DDPO [4] and DPOK [10] estimate trajectory policy gradients, and DRaFT [6] and AlignProp [37] differentiate through the sampler. FlowGRPO [32] optimizes stochasticized flow trajectories, with subsequent refinements to temporal allocation and trajectory reuse [16, 27, 28, 8]. SPO [29] generates candidates at each denoising step and selects preference pairs with a step-aware model. DiffusionNFT [55], AWM [51], and RAM [3] use regression-style objectives on noised on-policy samples. Offline alignment with fixed preference data. Fixed-data methods differ in how they use preferred and dispreferred samples. D3PO [52] extends pairwise learning to diffusion trajectories; DiffusionDPO [45] uses reference-relative denoising errors, with temporal weighting in Dense Reward [53] and a flow-model extension in FlowDPO [33]. DMPO [26] develops a reverse-KL objective, whereas MaPO [21] learns preference margins without a reference model. RFT [49, 5] fits only preferred samples. FlowCPO instead uses both sides of fixed preference pairs in a coupled, two-branch forward-KL objective, with a flow-matching upper-bound surrogate under stated conditions. Divergence-based alignment in language models. LLM alignment also treats divergence and utility as design choices. f-DPG [13] generalizes distribution matching, while f-DPO [46] varies the divergence constraint for preference learning. GPO [44] relates DPO [38], IPO [2], and SLiC [54] through convex pairwise losses. f-PO [14] unifies divergence-minimization formulations including DPO and EXO [22]. 3 Preliminaries We collect notation and background needed for our divergence-based view of flow alignment. Specifically, Sec. 3.1 recalls diffusion models and flow matching under linear interpolation. Sec. 3.2 summarizes RLHF and DPO in the discrete policy formulation, which we then lift to continuous-time models in Sec. 3.3. Finally, Sec. 3.4 reviews forward and reverse KL divergences. 3.1 Diffusion Models and Flow Matching We first briefly review the two continuous-time generative paradigms used throughout the paper. Both start from a simple latent variable, typically Gaussian noise ϵ∼(0,I)ε (0,I), and define intermediate states tx_t for t∈[0,1]t∈[0,1] that connect data and noise. In diffusion models [18, 40], one specifies a forward noising process q(t∣0)q(x_t _0) and trains a network ϵθ(t,c,t) _θ(x_t,c,t), or equivalently a score model, to predict the noise added to the clean sample 0x_0. Generation then approximately reverses this process from noise to data through a reverse-time SDE or the associated probability flow ODE. Flow matching [30, 1] instead directly learns the vector field of a prescribed probability path. In this paper we use the linear interpolation t=(1−t)⋅0+t⋅ϵ,ϵ∼(0,I),x_t=(1-t)·x_0+t·ε, ε (0,I)\,, (1) so that 0x_0 is the clean sample at t=0t=0 and 1=ϵx_1=ε is the noise sample at t=1t=1. The corresponding conditional target velocity is ut(t∣0)=dtdt=ϵ−0.u_t(x_t _0)= dx_tdt=ε-x_0\,. (2) Flow matching trains a velocity field vθ(t,c,t)v_θ(x_t,c,t) to regress to this target, typically by minimizing ℒFM(θ)=c,0,t,ϵ[‖vθ(t,c,t)−ut(t∣0)‖22].L_FM(θ)=E_c,x_0,t,ε [\|v_θ(x_t,c,t)-u_t(x_t _0)\|_2^2 ]\,. (3) Although diffusion and flow matching use different parameterizations, they are closely connected: under suitable probability paths, the denoising model, score function, and velocity field describe the same underlying transport from noise to data [24, 20, 43, 31]. 3.2 RLHF and Direct Preference Optimization (DPO) Reinforcement Learning from Human Feedback (RLHF) [36] typically aligns a generative policy πθ(0|c) _θ(x_0|c) by maximizing expected reward while regularizing deviation from a reference policy, where 0x_0 is the generated content and c is the condition. We use r(0,c)r(x_0,c) to denote the reward function, 0wx_0^w and 0lx_0^l to denote the preferred and dispreferred contents, respectively. DPO [38, 42, 44] bypasses the explicit reward modeling step by directly optimizing the policy using a closed-form solution to the KL-constrained reward maximization problem. Given a dataset =(c,0w,0l)D=\(c,x_0^w,x_0^l)\ of preferred (0wx_0^w) and dispreferred (0lx_0^l) contents, DPO minimizes the following negative log-likelihood loss: ℒDPO(θ)=−(c,0w,0l)∼[logσ(βlogπθ(0w|c)πref(0w|c)−βlogπθ(0l|c)πref(0l|c))],L_DPO(θ)=-E_(c,x_0^w,x_0^l) [ σ (β _θ(x^w_0|c) _ref(x^w_0|c)-β _θ(x_0^l|c) _ref(x_0^l|c) ) ]\,, (4) where πref _ref is the frozen reference policy and β is a temperature parameter controlling the strength of the KL constraint. 3.3 Preference Optimization in Continuous-Time Models Applying equation 4 directly to the diffusion and flow-matching models introduced above is non-trivial. Calculating the exact log-likelihood logπθ(0|c) _θ(x_0|c) requires solving the probability flow ODE, which is computationally prohibitive during training [55]. DiffusionDPO [45] addresses this intractability by optimizing the Evidence Lower Bound (ELBO) as a proxy for the likelihood. The loss function is reformulated using the denoising error at timestep t: ℒDiff-DPO(θ) _Diff-DPO(θ) =−c,0w,0l,t,ϵ[logσ(βρt( =-E_c,x^w_0,x^l_0,t,ε [ σ (β _t ( (5) (‖ϵθ(tl)−ϵ‖2−‖ϵref(tl)−ϵ‖2)⏟Advantage of Loser−(‖ϵθ(tw)−ϵ‖2−‖ϵref(tw)−ϵ‖2)⏟Advantage of Winner))], (\| _θ(x^l_t)-ε\|^2-\| _ref(x^l_t)-ε\|^2 )_Advantage of Loser- (\| _θ(x^w_t)-ε\|^2-\| _ref(x^w_t)-ε\|^2 )_Advantage of Winner ) ) ]\,, where tx_t is the linear interpolation of 0x_0 and noise ϵε, and ρt _t is a weighting function. FlowDPO [33] extends this paradigm to flow matching. By leveraging the connection between the score function and the vector field, FlowDPO replaces the noise prediction error with the flow matching loss, enabling efficient gradient-based preference optimization directly on the velocity field without ODE integration. 3.4 Forward and Reverse KL Divergences Let π(0∣c)π(x_0 c) denote a target distribution and πθ(0∣c) _θ(x_0 c) denote the trainable model distribution. Later in the paper, π will be instantiated by the preferred and dispreferred target distributions π+π^+ and π−π^-, while the model side of the objective will be represented by branch-specific distributions qθ+q_θ^+ and qθ−q_θ^- rather than by a single πθ _θ. The two KL directions are Reverse-KL: KL(πθ∥π)=0∼πθ(⋅∣c)[logπθ(0∣c)π(0∣c)] -KL: D_KL( _θ\|π)=E_x_0 _θ(· c) [ _θ(x_0 c)π(x_0 c) ] (6) Forward-KL: KL(π∥πθ)=0∼π(⋅∣c)[logπ(0∣c)πθ(0∣c)] -KL: D_KL(π\| _θ)=\ E_x_0 π(· c) [ π(x_0 c) _θ(x_0 c) ] The key difference is the sampling distribution inside the expectation. In the reverse-KL term KL(πθ∥π)D_KL( _θ\|π), errors are weighted by the current model πθ _θ, so target regions that are rarely visited by πθ _θ contribute little. In contrast, the forward-KL term KL(π∥πθ)D_KL(π\| _θ) is weighted by the target distribution π. If πθ(0∣c) _θ(x_0 c) is too small in regions where π(0∣c)π(x_0 c) is large, the penalty becomes severe, and finiteness requires supp(π)⊆supp(πθ)supp(π) ( _θ). 4 Methodology This section develops FlowCPO from a unified divergence-based formulation of preference alignment. We first show that RLHF admits a contrastive reverse-KL interpretation, then derive its offline forward-KL counterpart and reduce the resulting objective to a practical flow matching loss. 4.1 Probabilistic Reward Formulation Let c denote the conditioning context and 0x_0 the generated content. Following control-as-inference [25], we introduce a binary optimality variable o∈0,1o∈\0,1\, where o=1o=1 means 0x_0 is preferred and o=0o=0 means 0x_0 is dispreferred. This gives a normalized probabilistic view of any unbounded reward r(0,c)r(x_0,c): p(o=1∣0,c) p(o=1 _0,c) =exp(ω⋅r(0,c))exp(ω⋅r(0,c))+exp(−ω⋅r(0,c))=σ(2ω⋅r(0,c)) = (ω· r(x_0,c)) (ω· r(x_0,c))+ (-ω· r(x_0,c))=σ(2ω· r(x_0,c)) (7) p(o=0∣0,c) p(o=0 _0,c) =exp(−ω⋅r(0,c))exp(ω⋅r(0,c))+exp(−ω⋅r(0,c))=σ(−2ω⋅r(0,c)) = (-ω· r(x_0,c)) (ω· r(x_0,c))+ (-ω· r(x_0,c))=σ(-2ω· r(x_0,c)) Here ω>0ω>0 is an inverse temperature and σ(x)=11+exp(−x)σ(x)= 11+ (-x) is the sigmoid function. Conditioning the reference policy πref(0∣c) _ref(x_0 c) on the two optimality events, o=1o=1 and o=0o=0, yields preferred and dispreferred target distributions: π+(0∣c) π^+(x_0 c) :=p(0∣o=1,c)=πref(0∣c)p(o=1∣0,c)pπref(o=1∣c) :=p(x_0 o=1,c)= _ref(x_0 c)\,p(o=1 _0,c)p_ _ref(o=1 c) (8) π−(0∣c) π^-(x_0 c) :=p(0∣o=0,c)=πref(0∣c)p(o=0∣0,c)pπref(o=0∣c) :=p(x_0 o=0,c)= _ref(x_0 c)\,p(o=0 _0,c)p_ _ref(o=0 c) We use π+π^+ and π−π^- as the positive and negative target distributions throughout the paper. In practice, offline preference construction may only produce empirical approximations to these posteriors. We defer this distinction to App. B.1. By inverting equation 8 and substituting into equation 7, the reward r(0,c)r(x_0,c) becomes r(0,c)=12ωlogp(o=1∣0,c)p(o=0∣0,c)=12ω[logπ+(0∣c)π−(0∣c)+logpπref(o=1∣c)pπref(o=0∣c)].r(x_0,c)= 12ω p(o=1 _0,c)p(o=0 _0,c)= 12ω [ π^+(x_0 c)π^-(x_0 c)+ p_ _ref(o=1 c)p_ _ref(o=0 c) ]\,. (9) Thus, up to a context-only constant, reward is a contrastive log-density ratio: high reward corresponds to regions where π+π^+ dominates π−π^-. 4.2 Re-formalizing Reinforcement Learning from Human Feedback We begin with the unregularized expected-reward term c,0∼πθ(⋅∣c)[r(0,c)]E_c,x_0 _θ(· c)[r(x_0,c)] that appears inside RLHF objectives. Using equation 9, we can rewrite this term directly in divergence form in Thm. 4.1 (see App. B.2 for the proof): Theorem 4.1 (Expected reward as contrastive reverse KL). Up to the positive scale factor 12ω 12ω and an additive term depending only on c, maximizing expected reward is equivalent to minimizing c[KL(πθ∥π+)−KL(πθ∥π−)].E_c [D_KL( _θ\|π^+)-D_KL( _θ\|π^-) ]\,. (10) Equation 10 shows that expected reward induces a contrastive reverse-KL term: the policy is attracted to π+π^+ and repelled from π−π^- under its own sampling distribution. Reference-policy KL penalties used in practical RLHF methods remain additional terms and are not absorbed by this identity. Due to the high computational cost of online sampling, we focus on the offline regime in this work. To derive an offline counterpart, we reverse the direction of the KL divergence in equation 10 and assume fixed sampling distributions: minθℒOffline=c[KL(π+∥πθ+)+λ⋅KL(π−∥πθ−)],λ≥0. _θL_Offline=E_c [D_KL(π^+\| _θ^+)+λ·D_KL(π^-\| _θ^-) ]\,, λ≥ 0\,. (11) Here πθ+ _θ^+ and πθ− _θ^- are branch-specific model distributions. For FlowCPO itself, we focus on the attractive regime λ≥0λ≥ 0. In Sec. 5.3, we additionally instantiate λ<0λ<0 under the generalized framework to analyze repulsive baselines. To avoid evaluating flow-model likelihoods directly, we parameterize the two branches through symmetric mixed velocity fields: μθ=(1−β)vold+βvθ,νθ=(1+β)vold−βvθ. _θ=(1-β)\,v_old+β\,v_θ\,, _θ=(1+β)\,v_old-β\,v_θ\,. Here voldv_old is an EMA copy of the trainable field, and β>0β>0 controls the symmetric displacement of the two branches from that reference. The implicit distributions induced from the common Gaussian endpoint by μθ _θ and νθ _θ are denoted by πθ+ _θ^+ and πθ− _θ^-. The idealized score-field interpretation of this interpolation is given in App. B.3. The remaining bridge is from cross-entropy to regression. Each forward-KL term equals a target entropy plus an expected model negative log-likelihood; under the uniform regularity conditions in App. A, the latter is bounded by conditional flow matching objective plus a constant independent of θ. Applying this argument separately to the two branches yields Thm. 4.2. Theorem 4.2 (A flow-matching upper bound for offline forward KL). Under the regularity conditions in App. A, we have: ℒOffline(θ)≤C _Offline(θ)≤ C +c,0+∼π+(⋅∣c)t,ϵ[‖μθ(t+,t,c)−ut(t+∣0+)‖22] +E_ subarraycc,\,x_0^+ π^+(· c)\\ t,\,ε subarray [ \| _θ(x_t^+,t,c)-u_t(x_t^+ _0^+) \|_2^2 ] (12) +λ⋅c,0−∼π−(⋅∣c)t,ϵ[∥νθ(t−,t,c)−ut(t−∣0−)∥22], +λ·E_ subarraycc,\,x_0^- π^-(· c)\\ t,\,ε subarray [ \| _θ(x_t^-,t,c)-u_t(x_t^- _0^-) \|_2^2 ]\,, where C is independent of θ. The proof of Thm. 4.2 is deferred to App. B.4. This result justifies a tractable upper-bound surrogate. In practice, we replace the idealized source distributions π+,π−π^+,π^- with empirical preferred and dispreferred datasets, +D^+ and −D^-, and minimize the following contrastive flow matching loss: minθℒFlowCPO (θ)= _θL_FlowCPO (θ)= (c,0w)∼+t,ϵ[‖μθ(tw,t,c)−ut(tw∣0w)‖22] _ subarrayc(c,x_0^w) ^+\\ t,ε subarray [ \| _θ(x_t^w,t,c)-u_t(x_t^w _0^w) \|_2^2 ] (13) +λ⋅(c,0l)∼−t,ϵ[∥νθ(tl,t,c)−ut(tl∣0l)∥22]. +\,λ·\,E_ subarrayc(c,x_0^l) ^-\\ t,ε subarray [ \| _θ(x_t^l,t,c)-u_t(x_t^l _0^l) \|_2^2 ]. Pseudo-code is given in Alg. 1. For λ≥0λ≥ 0, equation 13 is bounded below by zero, whereas simplified FlowDPO’s signed objective can be unbounded below. App. B.5 provides the comparison. 4.3 Unified View and Connections to Prior Work Tab. 2 summarizes connections between equation 14 and prior works, and App. C elaborates the assumptions behind each mapping. FlowCPO is derived from the forward-KL objective in equation 11, which is a special case of equation 14. The generalized objective organizes flow-based preference-alignment methods (FlowDPO, FlowGRPO, DiffusionNFT) and regression baselines (SFT, RFT): minθℒGeneral=c[α⋅f(π+∥qθ+)⏟positive alignment+γ⋅f(π−∥qθ−)⏟negative regularization] _θL_General=E_c [α· D_f(π^+\|q_θ^+)_positive alignment\,+\,γ· D_f(π^-\|q_θ^-)_negative regularization ] (14) Here α>0α>0, and γ∈ℝγ . In the forward-KL branch we use f(π±∥q)=KL(π±∥q)D_f(π^±\|q)=D_KL(π^±\|q), and in the reverse-KL branch we use f(π±∥q)=KL(q∥π±)D_f(π^±\|q)=D_KL(q\|π^±). When γ≥0γ≥ 0, the negative regularization matches a negative distribution, and it acts as repulsive regularization when γ<0γ<0. Table 2: Comparison of flow-based alignment methods and related special cases under equation 14. Rows are organized by optimization regime and divergence choice. BT indicates Bradley–Terry preference model, and Ref. KL indicates whether the method uses reference-model KL regularization. Method Regime Divergence Optimized Process α γ qθ+q_θ^+ qθ−q_θ^- BT Ref. KL FlowGRPO [32] Online Reverse KL Reverse Process 11 −1-1 πθ _θ πθ _θ × ✓ DiffusionNFT [55] Online Forward KL Forward Process p(o=1|c)p(o=1|c) p(o=0|c)p(o=0|c) πθ+ _θ^+ πθ− _θ^- × × AWM [51] Online Reverse KL Forward Process 11 −1-1 πθ _θ πθ _θ × × RAM [3] Online Reverse KL Forward Process 11 −1-1 πθ _θ πθ _θ × ✓ SFT Offline Forward KL - p(o=1|c)p(o=1|c) p(o=0|c)p(o=0|c) πθ _θ πθ _θ × × RFT [49, 5] Offline Forward KL - 11 00 πθ _θ −- × × FlowDPO [33] Offline Implicit reverse KL - 11 −1-1 πθ _θ πθ _θ ✓ ✓ FlowCPO Offline Forward KL - 1 λ + _θ^+ − _θ^- × × × 5 Experiments We study FlowCPO along three axes: offline alignment with in-domain and out-of-domain preference data, the effect of negative regularization, and sensitivity to the core hyperparameters (β,λ,η)(β,λ,η). The main text focuses on the central quantitative and qualitative results, while the appendix collects the training pseudo-code, per-domain tables, and extended ablations. 5.1 Experimental Setup Table 3: Quantitative comparison under in-domain and out-of-domain (OOD) offline training regimes on SD3.5-M. We report results with and without Classifier-Free Guidance (CFG). When CFG is enabled, we report separate rows for guidance scales 1.0,3.0,4.5\1.0,3.0,4.5\. Under the in-domain setting, fine-tuned methods are trained separately for each target domain (Concept, Typography, and General Preference). Under the OOD setting, a single model is trained on the OOD offline dataset and evaluated across all domains and metrics. The two blocks therefore correspond to different evaluation protocols and should be compared primarily within, rather than across, regimes. The main table reports means only and rounds all entries to 2-3 decimal places; the corresponding mean ± standard deviation statistics over 5 independent runs for fine-tuned SD3.5-M variants are reported in the App. D.2, whereas pretrained baselines are single evaluations. Within each training regime, the best result is highlighted in bold and the second-best result is underlined. Concept Typography General Preference Model CFG GenEval ↑ OCR ↑ PickScore ↑ CLIPSc. ↑ HPSv2.1 ↑ Aes. ↑ ImgRwd ↑ UniRwd ↑ Pretrained model baselines SD-XL −- 0.55 0.14 22.42 0.287 0.280 5.60 0.76 2.93 SD3.5-L −- 0.71 0.68 22.91 0.289 0.288 5.50 0.96 3.25 FLUX.1-Dev −- 0.66 0.59 22.84 0.295 0.274 5.71 0.96 3.27 Reference model: SD3.5-M Base Model 1.01.0 0.24 0.12 20.51 0.237 0.204 5.13 −-0.58 2.02 3.03.0 0.59 0.47 22.28 0.287 0.284 5.38 0.71 2.96 4.54.5 0.63 0.59 22.34 0.285 0.279 5.36 0.85 3.03 In-domain offline fine-tuning on SD3.5-M + RFT [49, 5] 1.01.0 0.590.59 0.350.35 21.9121.91 0.2790.279 0.2760.276 5.355.35 0.570.57 2.662.66 3.03.0 0.740.74 0.700.70 22.6022.60 0.2960.296 0.3040.304 5.415.41 1.061.06 3.143.14 4.54.5 0.750.75 0.720.72 22.5722.57 0.2970.297 0.3050.305 5.425.42 1.111.11 3.153.15 + FlowDPO [33] 1.01.0 0.590.59 0.510.51 20.7220.72 0.2400.240 0.2210.221 5.225.22 −0.46-0.46 2.112.11 3.03.0 0.810.81 0.740.74 22.7622.76 0.2970.297 0.3010.301 5.565.56 1.041.04 3.103.10 4.54.5 0.810.81 0.750.75 22.8922.89 0.3010.301 0.3110.311 5.565.56 1.151.15 3.183.18 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.760.76 0.830.83 22.4822.48 0.2800.280 0.2900.290 5.645.64 0.900.90 2.932.93 3.03.0 0.840.84 0.870.87 22.9422.94 0.2990.299 0.3110.311 5.545.54 1.251.25 3.303.30 4.54.5 0.820.82 0.860.86 22.6722.67 0.3000.300 0.3030.303 5.465.46 1.221.22 3.283.28 Out-of-domain (OOD) offline fine-tuning on SD3.5-M + RFT [49, 5] 1.01.0 0.440.44 0.170.17 21.5921.59 0.2680.268 0.2610.261 5.495.49 0.310.31 2.562.56 3.03.0 0.670.67 0.530.53 22.6422.64 0.2970.297 0.3030.303 5.485.48 1.051.05 3.183.18 4.54.5 0.680.68 0.600.60 22.6922.69 0.2990.299 0.3070.307 5.475.47 1.121.12 3.233.23 + FlowDPO [33] 1.01.0 0.230.23 0.120.12 20.8320.83 0.2430.243 0.2260.226 5.725.72 −0.39-0.39 2.272.27 3.03.0 0.610.61 0.480.48 22.5422.54 0.2920.292 0.2920.292 5.525.52 0.910.91 3.103.10 4.54.5 0.650.65 0.540.54 22.6222.62 0.2950.295 0.2990.299 5.495.49 0.990.99 3.173.17 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.570.57 0.240.24 22.0222.02 0.2810.281 0.2750.275 5.355.35 0.630.63 2.852.85 3.03.0 0.690.69 0.490.49 22.3222.32 0.2960.296 0.2880.288 5.375.37 1.001.00 3.153.15 4.54.5 0.680.68 0.490.49 22.1222.12 0.2940.294 0.2840.284 5.315.31 0.950.95 3.113.11 + FlowCPO (β=1β=1, Ours) 1.01.0 0.470.47 0.240.24 21.8121.81 0.2760.276 0.2700.270 5.355.35 0.520.52 2.772.77 3.03.0 0.700.70 0.560.56 22.4622.46 0.2970.297 0.2950.295 5.415.41 1.021.02 3.233.23 4.54.5 0.700.70 0.590.59 22.3622.36 0.2940.294 0.2940.294 5.385.38 1.021.02 3.223.22 Data Construction and Implementation. We evaluate FlowCPO on Stable Diffusion 3.5 Medium (SD3.5-M) [9] in a strictly offline setting. Our offline preference data come from two sources. In the in-domain regime, we build winner/loser pairs from a frozen copy of the reference model using prompts drawn from the target benchmark sources. In the out-of-domain (OOD) regime, we use the public Open Image Preferences v1 dataset, whose images are generated by other open models. All models are trained under the same offline protocol, and we defer the exact data-generation pipeline, dataset sizes, reward filtering, and training hyperparameters to App. D.1. In the in-domain setting, each target capability uses its own specialist model, so the Tab. 3 summarizes task-specific fine-tuning results rather than a single universal model. Baselines. We compare against the two published approaches that are most directly comparable under the same fixed-data protocol: • RFT (Rejection Sampling Fine-Tuning) [49, 5]: A positive-only offline baseline that can be viewed as a forward-KL special case, learning exclusively from preferred data and ignoring dispreferred data. • FlowDPO [33]: A flow-based DPO baseline that uses preferred/dispreferred pairs through a reference-relative likelihood-ratio surrogate. Figure 2: Representative qualitative comparison of FlowCPO and the baselines across three preference optimization tasks. Evaluation Metrics. We evaluate two kinds of behavior. For targeted capabilities, we use task-specific benchmarks: GenEval [12] for multi-concept compositional generation and an OCR-based metric for visual text rendering. For general preference alignment, we run inference on the DrawBench prompt set and report PickScore [23], CLIP Score [17], HPS v2.1 [48], Aesthetics [39], ImageReward [50], and UnifiedReward [47]. Because the in-domain general-preference subset is filtered with PickScore, CLIP Score, and HPS v2.1, we treat these as optimization-aligned metrics and use Aesthetics, ImageReward, and UnifiedReward as a cleaner check of transfer beyond the filtering pipeline. 5.2 RQ1: Main Results with In-Domain and Out-of-Domain Data To evaluate method-level performance, we study three capability groups: semantic alignment, typographic generation, and general preference. We summarize the main quantitative results in Tab. 3. The two regimes answer different questions, so we read them separately: the in-domain block evaluates in-domain offline fine-tuning, whereas the out-of-domain (OOD) block evaluates a single model trained once on an OOD offline dataset. • In the in-domain setting, FlowCPO has higher mean GenEval and OCR scores than RFT and FlowDPO. Tab. 3 aggregates separate task-specific fine-tunes for the three capability groups; FlowCPO attains the best GenEval mean (0.84 at CFG 3.0) and the best OCR mean (0.87 at CFG 3.0), exceeding FlowDPO by 0.03 and 0.12, respectively. On general preference, the picture is narrower but still favorable: FlowCPO is best or tied-best on PickScore, HPS v2.1, ImageReward, and UnifiedReward, while CLIP Score is effectively tied with FlowDPO. Since PickScore, CLIP Score, and HPS v2.1 also appear in the in-domain filtering pipeline, we put more weight on Aesthetics, ImageReward, and UnifiedReward, together with the qualitative comparison in Fig. 2 and the extended examples in App. D.2, when judging transfer beyond the optimization-aligned signals. • In the out-of-domain setting, the advantage is metric-dependent. FlowCPO gives the best GenEval score and remains competitive on OCR and UnifiedReward, whereas RFT performs better on several reward-model metrics. Thus, retaining both sides of each preference pair does not guarantee improvement when the fixed data are generated by models other than the reference model. This result motivates the regularization analysis in RQ2 and identifies out-of-domain data as an empirical boundary of the method. Table 4: Analysis of different regularization mechanisms on SD3.5-M under the unified divergence-based framework of equation 14, with all models trained only on GenEval. We consider three representative paradigms: positive-only regularization, which uses only preferred samples; repulsive negative regularization, which explicitly pushes the model away from dispreferred regions; and attractive negative regularization, which incorporates negative samples through distributional matching. For attractive negative regularization, we report two instantiations with different mixing coefficients β. Results are reported with and without Classifier-Free Guidance (CFG). Metrics are grouped into Concept, Typography, and General Preference for consistency with the main results table and to assess cross-metric generalization beyond the training signal. Best results are highlighted in bold and second-best results are underlined. λ CFG Concept Typography General Preference GenEval ↑ OCR ↑ PickScore ↑ CLIPSc. ↑ HPSv2.1 ↑ Aes. ↑ ImgRwd ↑ UniRwd ↑ Positive-only regularization (RFT-like): μθ=vθ,νθ=0 _θ=v_θ,\ _θ=0 00 1.01.0 0.59 0.15 21.67 0.27 0.268 5.32 0.43 2.63 00 3.03.0 0.72 0.51 22.5 0.295 0.301 5.41 1.03 3.13 00 4.54.5 0.75 0.56 22.5 0.296 0.304 5.41 1.08 3.16 Repulsive negative regularization (DPO-like): μθ=νθ=vθ _θ= _θ=v_θ −0.1-0.1 1.01.0 0.64 0.15 21.76 0.277 0.270 5.30 0.53 2.66 −0.1-0.1 3.03.0 0.77 0.50 22.48 0.297 0.300 5.37 1.07 3.17 −0.1-0.1 4.54.5 0.76 0.58 22.47 0.297 0.304 5.40 1.10 3.17 −1-1 1.01.0 0.25 0.08 19.46 0.201 0.154 4.22 -1.66 1.40 −1-1 3.03.0 0.70 0.48 21.39 0.270 0.229 4.70 -0.28 2.51 −1-1 4.54.5 0.71 0.55 21.50 0.276 0.240 4.86 -0.01 2.65 −10-10 – Diverged Diverged Diverged Diverged Diverged Diverged Diverged Diverged Attractive negative regularization (CPO-like): μθ=vθ,νθ=2vold−vθ _θ=v_θ,\ _θ=2v_old-v_θ 0.10.1 1.01.0 0.64 0.17 21.72 0.278 0.266 5.24 0.50 2.71 0.10.1 3.03.0 0.77 0.54 22.47 0.297 0.301 5.35 1.06 3.19 0.10.1 4.54.5 0.77 0.59 22.48 0.298 0.304 5.38 1.14 3.23 11 1.01.0 0.78 0.25 21.74 0.280 0.258 5.23 0.57 2.74 11 3.03.0 0.78 0.51 21.69 0.291 0.266 5.09 0.76 2.96 11 4.54.5 0.67 0.55 21.20 0.285 0.245 4.89 0.43 2.75 1010 – Diverged Diverged Diverged Diverged Diverged Diverged Diverged Diverged Attractive negative regularization (CPO-like): μθ=0.5vold+0.5vθ,νθ=1.5vold−0.5vθ _θ=0.5v_old+0.5v_θ,\ _θ=1.5v_old-0.5v_θ 0.10.1 1.01.0 0.63 0.17 21.48 0.275 0.259 5.22 0.33 2.55 0.10.1 3.03.0 0.78 0.51 22.41 0.298 0.297 5.33 1.04 3.14 0.10.1 4.54.5 0.77 0.58 22.43 0.299 0.299 5.35 1.10 3.19 11 1.01.0 0.76 0.26 21.84 0.280 0.268 5.26 0.64 2.79 11 3.03.0 0.84 0.53 22.20 0.296 0.288 5.27 1.01 3.10 11 4.54.5 0.83 0.55 21.94 0.294 0.282 5.16 0.95 3.04 1010 – Diverged Diverged Diverged Diverged Diverged Diverged Diverged Diverged 5.3 RQ2: Effectiveness of Different Negative Regularization We compare positive-only training with repulsive (λ<0λ<0) and attractive (λ>0λ>0) negative regularization under the unified framework. Tab. 4 favors moderate attractive matching: β=0.5,λ=1β=0.5,λ=1 achieves the best GenEval score (0.84), while β=1,λ=0.1β=1,λ=0.1 gives the best OCR, ImageReward, and UnifiedReward. Repulsion with λ=−1λ=-1 sharply degrades all metrics, and both signs diverge at |λ|=10|λ|=10. Since training optimizes only GenEval, the other gains measure cross-metric transfer. 5.4 RQ3: Sensitivity of FlowCPO to Its Core Hyperparameters Fig. 9 favors moderate interpolation (β∈[0.5,1.0]β∈[0.5,1.0]), balanced negative weight (λ=1λ=1), and slow EMA (η=0.99η=0.99). Extreme weights destabilize training, while small weights weaken the negative contribution. We therefore use β=0.5β=0.5, λ=1λ=1, and η=0.99η=0.99 as the default settings. Complete sweeps appear in App. D.3.2. 6 Conclusion and Limitations We presented a divergence-based framework that separates sampling regime from divergence direction and derived FlowCPO as its offline forward-KL instance. Coupled preferred and dispreferred branches enable regression on fixed preference pairs without online rollouts. Our loss decomposition also connects FlowCPO to simplified FlowDPO and shows how the difference between the current model and its EMA enters the objective. The nonnegative loss avoids simplified FlowDPO’s potentially unbounded negative regression term. Experiments show higher mean in-domain GenEval and OCR scores than RFT and FlowDPO, with mixed out-of-domain gains. The ablations favor moderate attractive matching of dispreferred samples, while excessively large weights can destabilize training. The forward-KL bound assumes linear interpolation and uniform regularity. Loss nonnegativity alone does not guarantee stable training. We evaluate fine-tuning on SD3.5-M, so performance on other backbones remains to be established. Extending the bound to other paths and improving robustness across different data sources remain open challenges. References [1] M. S. Albergo and E. Vanden-Eijnden (2022) Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571. Cited by: §1, §3.1. [2] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello (2024) A general theoretical paradigm to understand learning from human preferences. In International conference on artificial intelligence and statistics, p. 4447–4455. Cited by: §2. [3] A. Bergmeister, S. Jegelka, N. Nüsken, C. Domingo-Enrich, and J. Pidstrigach (2026) Reinforce adjoint matching: scaling rl post-training of diffusion and flow-matching models. arXiv preprint arXiv:2605.10759. Cited by: §C.4, §1, §2, Table 2. [4] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2024) Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, Cited by: §2. [5] H. Chen, K. Zheng, Q. Zhang, G. Cui, Y. Cui, H. Ye, T. Lin, M. Liu, J. Zhu, and H. Wang (2025) Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505.18116v1. Cited by: Table 5, Table 6, Table 7, Table 8, §2, Table 2, 1st item, Table 3, Table 3. [6] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet (2024) Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, Cited by: §2. [7] Data Is Better Together (2024) Open image preferences v1 results. Note: https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-resultsHugging Face dataset, accessed March 26, 2026 Cited by: §D.1. [8] Z. Ding and W. Ye (2025) TreeGRPO: tree-advantage grpo for online rl post-training of diffusion models. arXiv preprint arXiv:2512.08153. Cited by: §2. [9] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §5.1. [10] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee (2023) DPOK: reinforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems, Vol. 36, p. 79858–79885. Cited by: §2. [11] D. Friedman and A. B. Dieng (2023) The vendi score: a diversity evaluation metric for machine learning. Transactions on Machine Learning Research. External Links: Link Cited by: §D.4. [12] D. Ghosh, H. Hajishirzi, and L. Schmidt (2023) Geneval: an object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems 36, p. 52132–52152. Cited by: §5.1. [13] D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman (2023) Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, p. 11546–11583. External Links: Link Cited by: §2. [14] J. Han, M. Jiang, Y. Song, S. Ermon, and M. Xu (2025) f-PO: generalizing preference optimization with f-divergence minimization. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 258, p. 1144–1152. External Links: Link Cited by: §2. [15] Y. Han, H. Sun, and T. Lin (2026) When can conditional flow matching replace pointwise negative log-likelihood?. arXiv preprint arXiv:2608.28010. Cited by: §A.1, Theorem A.3, Appendix A. [16] X. He, S. Fu, Y. Zhao, W. Li, J. Yang, D. Yin, F. Rao, and B. Zhang (2025) Tempflow-grpo: when timing matters for grpo in flow models. arXiv preprint arXiv:2508.04324. Cited by: §2. [17] J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi (2021) Clipscore: a reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528. Cited by: §5.1. [18] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1, §3.1. [19] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §B.3. [20] P. Holderrieth and E. Erives (2025) An introduction to flow matching and diffusion models. arXiv preprint arXiv:2506.02070. Cited by: §B.3, §1, §3.1. [21] J. Hong, S. Paul, N. Lee, K. Rasul, J. Thorne, and J. Jeong (2026) Margin-aware preference optimization for aligning diffusion models without reference. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 4744–4752. External Links: Document Cited by: §2. [22] H. Ji, C. Lu, Y. Niu, P. Ke, H. Wang, J. Zhu, J. Tang, and M. Huang (2024) Towards efficient exact optimization of language model alignment. arXiv preprint arXiv:2402.00856. Cited by: §2. [23] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy (2023) Pick-a-pic: an open dataset of user preferences for text-to-image generation. Advances in neural information processing systems 36, p. 36652–36663. Cited by: §5.1. [24] C. Lai, Y. Song, D. Kim, Y. Mitsufuji, and S. Ermon (2025) The principles of diffusion models. arXiv preprint arXiv:2510.21890. Cited by: §3.1. [25] S. Levine (2018) Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv preprint arXiv:1805.00909. Cited by: §4.1. [26] B. Li, M. Xu, J. Han, M. Dang, and S. Ermon (2025) Divergence minimization preference optimization for diffusion model alignment. arXiv preprint arXiv:2507.07510. Cited by: §2. [27] J. Li, Y. Cui, T. Huang, Y. Ma, C. Fan, M. Yang, and Z. Zhong (2025) Mixgrpo: unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802v1. Cited by: §2. [28] Y. Li, Y. Wang, Y. Zhu, Z. Zhao, M. Lu, Q. She, and S. Zhang (2025) Branchgrpo: stable and efficient grpo with structured branching in diffusion models. arXiv preprint arXiv:2509.06040. Cited by: §2. [29] Z. Liang, Y. Yuan, S. Gu, B. Chen, T. Hang, M. Cheng, J. Li, and L. Zheng (2025) Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13199–13208. Cited by: §2. [30] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §1, §3.1. [31] Y. Lipman, M. Havasi, P. Holderrieth, N. Shaul, M. Le, B. Karrer, R. T. Chen, D. Lopez-Paz, H. Ben-Hamu, and I. Gat (2024) Flow matching guide and code. arXiv preprint arXiv:2412.06264. Cited by: §3.1. [32] J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025) Flow-grpo: training flow matching models via online rl. arXiv preprint arXiv:2505.05470. Cited by: §C.1, §1, §1, §2, Table 2. [33] J. Liu, G. Liu, J. Liang, Z. Yuan, X. Liu, M. Zheng, X. Wu, Q. Wang, M. Xia, X. Wang, et al. (2025) Improving video generation with human feedback. arXiv preprint arXiv:2501.13918. Cited by: §C.7, Table 5, Table 6, Table 7, Table 8, §1, §1, §2, §3.3, Table 2, 2nd item, Table 3, Table 3. [34] C. Lu, K. Zheng, F. Bao, J. Chen, C. Li, and J. Zhu (2022) Maximum likelihood training for score-based diffusion ODEs by high-order denoising score matching. arXiv preprint arXiv:2206.08265. Cited by: Appendix A. [35] C. Luo (2022) Understanding diffusion models: a unified perspective. arXiv preprint arXiv:2208.11970. Cited by: §B.3, §1. [36] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §3.2. [37] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki (2023) Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739v2. Note: Withdrawn in 2024; subsumed by arXiv:2407.08737 Cited by: §2. [38] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2, §3.2. [39] C. Schuhmann (2022) LAION-aesthetics. Note: LAION blog External Links: Link Cited by: §5.1. [40] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, p. 2256–2265. Cited by: §1, §3.1. [41] Y. Song, C. Durkan, I. Murray, and S. Ermon (2021) Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems 34, p. 1415–1428. Cited by: Appendix A. [42] H. Sun, S. Liao, Y. Han, Y. Bai, Y. Gao, C. Fu, W. Shen, F. Wan, M. Yan, J. Zhang, et al. (2025) Solopo: unlocking long-context capabilities in llms via short-to-long preference optimization. arXiv preprint arXiv:2505.11166. Cited by: §3.2. [43] P. Sun, Y. Jiang, and T. Lin (2025) Unified continuous generative models. arXiv preprint arXiv:2505.07447. Cited by: §3.1. [44] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Á. Pires, and B. Piot (2024) Generalized preference optimization: a unified approach to offline alignment. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 47725–47742. External Links: Link Cited by: §2, §3.2. [45] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik (2024) Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8228–8238. Cited by: §2, §3.3. [46] C. Wang, Y. Jiang, C. Yang, H. Liu, and Y. Chen (2024) Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2. [47] Y. Wang, Y. Zang, H. Li, C. Jin, and J. Wang (2025) Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.05236. Cited by: §5.1. [48] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: §5.1. [49] W. Xiong, J. Yao, Y. Xu, B. Pang, L. Wang, D. Sahoo, J. Li, N. Jiang, T. Zhang, C. Xiong, et al. (2025) A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343. Cited by: Table 5, Table 6, Table 7, Table 8, §2, Table 2, 1st item, Table 3, Table 3. [50] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong (2023) Imagereward: learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36, p. 15903–15935. Cited by: §2, §5.1. [51] S. Xue, C. Ge, S. Zhang, Y. Li, and Z. Ma (2025) Advantage weighted matching: aligning rl with pretraining in diffusion models. arXiv preprint arXiv:2509.25050. Cited by: §C.3, §1, §2, Table 2. [52] K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, W. Shen, X. Zhu, and X. Li (2024) Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8941–8951. Cited by: §2. [53] S. Yang, T. Chen, and M. Zhou (2024) A dense reward view on aligning text-to-image diffusion with preference. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235, p. 55998–56032. Cited by: §2. [54] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu (2023) Slic-hf: sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. Cited by: §2. [55] K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu (2025) Diffusionnft: online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117. Cited by: §C.2, §1, §1, §2, §3.3, Table 2. Contents 1 Introduction 2 Related Work 3 Preliminaries 3.1 Diffusion Models and Flow Matching 3.2 RLHF and Direct Preference Optimization (DPO) 3.3 Preference Optimization in Continuous-Time Models 3.4 Forward and Reverse KL Divergences 4 Methodology 4.1 Probabilistic Reward Formulation 4.2 Re-formalizing Reinforcement Learning from Human Feedback 4.3 Unified View and Connections to Prior Work 5 Experiments 5.1 Experimental Setup 5.2 RQ1: Main Results with In-Domain and Out-of-Domain Data 5.3 RQ2: Effectiveness of Different Negative Regularization 5.4 RQ3: Sensitivity of FlowCPO to Its Core Hyperparameters 6 Conclusion and Limitations References A Theoretical Relation between Log-Likelihood and Flow Matching Loss A.1 Pointwise Relation between Log-Likelihood and Flow Matching Loss A.2 Expectation-level Relation between Log-Likelihood and Flow Matching Loss B Theoretical Proofs and Results B.1 Idealized Posteriors and Empirical Source Distributions B.2 Proof of Theorem B.3 An Idealized Interpretation of Interpolated Vector Fields B.4 Proof of Theorem B.5 Relation between FlowCPO and Simplified FlowDPO C Connections to Existing Methods Under the Unified Divergence Framework C.1 Connection to FlowGRPO C.2 Connection to DiffusionNFT C.3 Connection to AWM C.4 Connection to RAM C.5 Connection to SFT C.6 Connection to RFT C.7 Connection to FlowDPO D Experimental Details D.1 Experimental Details of Sec. D.2 Experimental Results on Optimization Targets D.3 Additional Analyses Beyond Main Results D.3.1 Understanding the Role of Negative Regularization D.3.2 Hyperparameter Sensitivity and Ablation D.4 Per-Prompt Diversity Evaluation E Broader Impacts and Release Considerations Appendix A Theoretical Relation between Log-Likelihood and Flow Matching Loss In this section, we discuss the relationship between log-likelihood and flow matching loss. The theoretical results mainly follow from [41, 15, 34]. A.1 Pointwise Relation between Log-Likelihood and Flow Matching Loss The theoretical results in this subsection are from [15]. Fix a sample x0∈ℝdx_0 ^d, t=(1−t)x0+t1∼qtx0=((1−t)x0,t2Id),1∼(0,Id).x_t=(1-t)x_0+tx_1 q_t^x_0=N((1-t)x_0,t^2I_d), _1 (0,I_d). Let utx0()=−x0tu_t^x_0(x)= x-x_0t be the velocity field corresponding to qtx0q_t^x_0. We compare this conditional path (qtx0,utx0)(q_t^x_0,u_t^x_0) to the model path (ptθ,vθ)(p_t^θ,v_θ) through the velocity and score gaps: Δvtθ(,x0):=vθ(,t)−utx0(),Δstθ(,x0):=∇logqtx0()−∇logptθ() v_t^θ(x;x_0):=v_θ(x,t)-u_t^x_0(x), s_t^θ(x;x_0):=∇ q_t^x_0(x)-∇ p_t^θ(x) we also define the following terms: ℓε(θ,x0):=ε∼qεx0[−logpεθ(ε)],w[ε,1](θ,x0):=∫ε1w(t)t∼qtx0[‖Δvtθ(t,x0)‖2]t _ (θ;x_0):=E_x_ q_ ^x_0 [- p_ ^θ(x_ ) ], _w^[ ,1](θ;x_0):= _ ^1w(t)\,E_x_t q_t^x_0 [\| v_t^θ(x_t;x_0)\|^2 ]\,dt Assumption A.1 (Pathwise regularity). For the fixed x0x_0, the densities qtx0q_t^x_0 and ptθp_t^θ are strictly positive and C1C^1 in x for t∈(0,1)t∈(0,1). Their continuity equations hold in strong form, t↦KL(qtx0∥ptθ)t (q_t^x_0\|p_t^θ) is differentiable, and the integrations by parts used below have no boundary terms. Assumption A.2 (Endpoint regularity). The map fθ(x,t):=−logptθ(x)f_θ(x,t):=- p_t^θ(x) is jointly continuous at (x0,0)(x_0,0). Moreover, for some ε0>0 _0>0, C>0C>0, and m≥1m≥ 1, |fθ(x,t)|≤C(1+‖x‖m),x∈ℝd,t∈[0,ε0].|f_θ(x,t)|≤ C(1+\|x\|^m), x ^d, t∈[0, _0]. Theorem A.3 (Pointwise NLL is a CFM term plus residuals [15]). Suppose Assump. A.1 and Assump. A.2 hold, and suppose that p1θ=q1x0=(0,Id)p_1^θ=q_1^x_0=N(0,I_d). Then, for every positive weight w and every ε∈(0,1) ∈(0,1), ℓε(θ,x0) _ (θ;x_0) =H(qεx0)+∫ε1qtx0[⟨Δvtθ(t,x0),Δstθ(t,x0)⟩]t, =H(q_ ^x_0)+ _ ^1E_q_t^x_0 [ v_t^θ(x_t;x_0), s_t^θ(x_t;x_0) ]\,dt, (15) −logp0θ(x0) - p_0^θ(x_0) =H(qεx0)+w[ε,1](θ,x0)+ε,w(θ,x0)+ℬε(θ,x0), =H(q_ ^x_0)+J_w^[ ,1](θ;x_0)+G_ ,w(θ;x_0)+B_ (θ;x_0), (16) where ε,w(θ,x0):=∫ε1qtx0[⟨Δvtθ(t,x0),Δstθ(t,x0)−w(t)Δvtθ(t,x0)⟩]t, _ ,w(θ;x_0):= _ ^1E_q_t^x_0 [ v_t^θ(x_t;x_0), s_t^θ(x_t;x_0)-w(t) v_t^θ(x_t;x_0) ]\,dt, H(qεx0) H(q_ ^x_0) =d2log(2πeε2),ℬε(θ,x0):=−logp0θ(x0)−ℓε(θ,x0),limε↓0ℬε(θ,x0)=0. = d2 (2π e\, ^2), _ (θ;x_0):=- p_0^θ(x_0)- _ (θ;x_0), _ 0B_ (θ;x_0)=0. A.2 Expectation-level Relation between Log-Likelihood and Flow Matching Loss Throughout this subsection, let t∼Unif(0,1),0∼pdata,1∼(0,Id),t (0,1), _0 p_data, _1 (0,I_d), where 0x_0 and 1x_1 are independent, and let t=(1−t)0+t1∼qt.x_t=(1-t)x_0+tx_1 q_t. The marginal velocity field of (qt)t∈[0,1](q_t)_t∈[0,1] is ut(x):=[1−0∣t=x].u_t(x):=E[x_1-x_0 _t=x]. Lemma A.4 (Optimal solution of Flow Matching Loss). Let us consider the following optimization problem: minθt,0,1[‖vθ((1−t)0+t1,t)−(1−0)‖22] _θE_t,x_0,x_1 [\|v_θ((1-t)x_0+tx_1,t)-(x_1-x_0)\|_2^2 ] (17) Then, for every t∈(0,1)t∈(0,1), the optimal solution is given by v⋆(x,t)=0,1[1−0∣t=x]v (x,t)=E_x_0,x_1 [x_1-x_0 _t=x ] (18) where t=(1−t)0+t1x_t=(1-t)x_0+tx_1. Proof. t,0,1[‖vθ((1−t)0+t1,t)−(1−0)‖22] _t,x_0,x_1 [\|v_θ((1-t)x_0+tx_1,t)-(x_1-x_0)\|_2^2 ] =t,0,1[‖vθ(t,t)−v⋆(t,t)+v⋆(t,t)−(1−0)‖22] =E_t,x_0,x_1 [\|v_θ(x_t,t)-v (x_t,t)+v (x_t,t)-(x_1-x_0)\|_2^2 ] =t,t[0,1[‖vθ(t,t)−v⋆(t,t)+v⋆(t,t)−(1−0)‖22∣t]] =E_t,x_t[E_x_0,x_1 [\|v_θ(x_t,t)-v (x_t,t)+v (x_t,t)-(x_1-x_0)\|_2^2 _t ]] (by tower property of expectation) =t,t[0,1[∥vθ(t,t)−v⋆(t,t)∥22+∥v⋆(t,t)−(1−0)∥22 =E_t,x_t[E_x_0,x_1[\|v_θ(x_t,t)-v (x_t,t)\|^2_2+\|v (x_t,t)-(x_1-x_0)\|_2^2 +2⟨vθ(t,t)−v⋆(t,t),v⋆(t,t)−(1−0)⟩∣t]] +2 v_θ(x_t,t)-v (x_t,t),v (x_t,t)-(x_1-x_0) _t]] Now, we prove that t,t[0,1[⟨vθ(t,t)−v⋆(t,t),v⋆(t,t)−(1−0)⟩∣t]]=0E_t,x_t[E_x_0,x_1 [ v_θ(x_t,t)-v (x_t,t),v (x_t,t)-(x_1-x_0) _t ]]=0. t,t[0,1[⟨vθ(t,t)−v⋆(t,t),v⋆(t,t)−(1−0)⟩∣t]] _t,x_t[E_x_0,x_1 [ v_θ(x_t,t)-v (x_t,t),v (x_t,t)-(x_1-x_0) _t ]] =t,t[⟨vθ(t,t)−v⋆(t,t),v⋆(t,t)−0,1[(1−0)∣t]⟩] =E_t,x_t[ v_θ(x_t,t)-v (x_t,t),v (x_t,t)-E_x_0,x_1 [(x_1-x_0) _t ] ] (by the linearity of expectation) =t,t[⟨vθ(t,t)−v⋆(t,t),v⋆(t,t)−v⋆(t,t)⟩](by v⋆(t,t)=0,1[(1−0)∣t]) =E_t,x_t[ v_θ(x_t,t)-v (x_t,t),v (x_t,t)-v (x_t,t) ] (by $v (x_t,t)=E_x_0,x_1 [(x_1-x_0) _t ]$) =0 =0 Therefore, t,0,1[‖vθ((1−t)0+t1,t)−(1−0)‖22] _t,x_0,x_1 [\|v_θ((1-t)x_0+tx_1,t)-(x_1-x_0)\|_2^2 ] =t,0,1[‖vθ(t,t)−v⋆(t,t)‖22+‖v⋆(t,t)−(1−0)‖22] =E_t,x_0,x_1[\|v_θ(x_t,t)-v (x_t,t)\|^2_2+\|v (x_t,t)-(x_1-x_0)\|_2^2] Clearly, the optimal solution is attained when vθ(x,t)=v⋆(x,t)v_θ(x,t)=v (x,t). ∎ Lemma A.5. Under the optimal solution v⋆(x,t)v (x,t), we have: v⋆(x,t)=−11−tx−t1−t∇logqt(x)v (x,t)=- 11-tx- t1-t∇ q_t(x) (19) where t=(1−t)0+t1∼qt(x)x_t=(1-t)x_0+tx_1 q_t(x). Proof. Let qtx0(x)=qt(x∣x0)q^x_0_t(x)=q_t(x x_0). Because t=(1−t)0+t1x_t=(1-t)x_0+tx_1 and 1∼(0,I)x_1 (0,I), then qtx0(t)=((1−t)x0,t2I)q_t^x_0(x_t)=N((1-t)x_0,t^2I) (20) and we have: ∇logqtx0(t) ∇ q_t^x_0(x_t) =−1t1 =- 1tx_1 (21) Therefore, ∇logqt(t) ∇ q_t(x_t) =[∇logqtx0(t)∣t]=−1t[1∣t] =E[∇ q_t^x_0(x_t) _t]=- 1tE[x_1 _t] (22) Then, for the optimal solution v⋆(x,t)v (x,t), we have: v⋆(x,t) v (x,t) =[1−0∣t=x] =E[x_1-x_0 _t=x] (23) =[1−t−t11−t∣t=x] =E[x_1- x_t-tx_11-t _t=x] (24) =−11−tx+11−t[1∣t=x] =- 11-tx+ 11-tE[x_1 _t=x] (25) =−11−tx−t1−t∇logqt(x) =- 11-tx- t1-t∇ q_t(x) (26) ∎ Theorem A.6. Suppose Assump. A.1 and Assump. A.2 hold and let pdata(0)p_data(x_0) be the data distribution. Then, 0∼pdata(0)[−logp0θ(0)] _x_0 p_data(x_0)[- p_0^θ(x_0)] ≤H(pdata)+[‖vθ(t,t)−(1−0)‖22]⋅[‖∇logqt(t)−∇logptθ(t)‖22] ≤ H(p_data)+ E [\|v_θ(x_t,t)-(x_1-x_0)\|^2_2 ]·E [\|∇ q_t(x_t)-∇ p^θ_t(x_t)\|^2_2 ] In the unrestricted, well-specified population setting, the marginal target field v⋆(t,t)=[1−0∣t]v (x_t,t)=E[x_1-x_0 _t] minimizes the flow matching loss and induces the data endpoint distribution, which also minimizes cross-entropy. Proof. By Cauchy-Schwarz inequality, we have: (t,0,t[⟨vθ(t,t)−(1−0),∇logqt(t)−∇logptθ(t)⟩])2 (E_t,x_0,x_t[ v_θ(x_t,t)-(x_1-x_0),∇ q_t(x_t)-∇ p^θ_t(x_t) ])^2 (27) ≤t,0,t[‖vθ(t,t)−(1−0)‖22]⋅t,0,t[‖∇logqt(t)−∇logptθ(t)‖22] _t,x_0,x_t[\|v_θ(x_t,t)-(x_1-x_0)\|^2_2]·E_t,x_0,x_t[\|∇ q_t(x_t)-∇ p^θ_t(x_t)\|^2_2] (28) Then, by Thm. A.3, we have: 0∼pdata(0)[−logp0θ(0)] _x_0 p_data(x_0)[- p_0^θ(x_0)] (29) =H(pdata)+[⟨vθ(t,t)−(1−0),∇logqt(t)−∇logptθ(t)⟩] =H(p_data)+E [ v_θ(x_t,t)-(x_1-x_0),∇ q_t(x_t)-∇ p^θ_t(x_t) ] (30) ≤H(pdata)+([‖vθ(t,t)−(1−0)‖22][‖∇logqt(t)−∇logptθ(t)‖22])1/2 ≤ H(p_data)+(E [\|v_θ(x_t,t)-(x_1-x_0)\|^2_2 ]E [\|∇ q_t(x_t)-∇ p^θ_t(x_t)\|^2_2 ])^1/2 (31) By Lem. A.4, the unrestricted population minimizer is v⋆(t,t)=[1−0∣t]v (x_t,t)=E[x_1-x_0 _t]. This field generates the prescribed marginal path qtq_t, so ptθ⋆=qtp_t^θ =q_t in the well-specified case and the score-gap term vanishes. Consequently the induced endpoint distribution equals pdatap_data, which minimizes cross-entropy. This population statement does not imply equality of the two objectives, nor identical minimizers in a restricted parameter class. ∎ Theorem A.7 (Flow matching upper bound under uniform regularity). Suppose Assump. A.1 and Assump. A.2 hold, and let p1θ=(0,Id)p_1^θ=N(0,I_d). Assume M2:=pdata[‖0‖2]<∞M_2:=E_p_data[\|x_0\|^2]<∞ and that there exist constants C0,C1,C2<∞C_0,C_1,C_2<∞, independent of θ, such that, for every x∈ℝdx ^d and t∈[0,1]t∈[0,1], ∥vθ(0,t)∥≤C0,∥∇vθ(x,t)∥≤C1,∥∇⊤∇⋅vθ(x,t)∥≤C2.\|v_θ(0,t)\|≤ C_0, \|∇ v_θ(x,t)\|≤ C_1, \|∇ ∇· v_θ(x,t)\|≤ C_2. (32) Assume additionally that the data-path score is uniformly square-integrable, Cq:=supt∈(0,1]qt[‖∇logqt(t)‖2]<∞.C_q:= _t∈(0,1]E_q_t[\|∇ q_t(x_t)\|^2]<∞. (33) Then, there exists a constant Cs<∞C_s<∞, independent of θ, such that: 0∼pdata[−logp0θ(0)]≤[‖vθ(t,t)−(1−0)‖2]+H(pdata)+Cs4.E_x_0 p_data[- p_0^θ(x_0)] [\|v_θ(x_t,t)-(x_1-x_0)\|^2 ]+H(p_data)+ C_s4. (34) In the unrestricted, well-specified population setting, the marginal target field minimizes both sides. Proof. Let stθ(x)=∇logptθ(x)s^θ_t(x)=∇ p^θ_t(x), then we consider the continuity equation: ∂tptθ(x)+∇⋅(ptθ(x)vθ(x,t)) _tp^θ_t(x)+∇·(p^θ_t(x)v_θ(x,t)) =0 =0 (35) ∂tptθ(x)+∇ptθ(x)⋅vθ(x,t)+ptθ(x)∇⋅vθ(x,t) _tp^θ_t(x)+∇ p^θ_t(x)· v_θ(x,t)+p^θ_t(x)∇· v_θ(x,t) =0 =0 (36) ∂tlogptθ(x)+vθ(x,t)⋅∇logptθ(x)+∇⋅vθ(x,t) _t p^θ_t(x)+v_θ(x,t)·∇ p^θ_t(x)+∇· v_θ(x,t) =0 =0 (37) Taking ∇ on both sides, we get: ∂t∇logptθ(x)+∇vθ(x,t)⋅∇logptθ(x)+vθ(x,t)∇⊤∇logptθ(x)+∇⊤∇⋅vθ(x,t) _t∇ p^θ_t(x)+∇ v_θ(x,t)·∇ p^θ_t(x)+v_θ(x,t)∇ ∇ p^θ_t(x)+∇ ∇· v_θ(x,t) =0 =0 (38) ∂tstθ(x)+∇vθ(x,t)⋅stθ(x)+vθ(x,t)∇⊤stθ(x)+∇⊤∇⋅vθ(x,t) _ts^θ_t(x)+∇ v_θ(x,t)· s^θ_t(x)+v_θ(x,t)∇ s^θ_t(x)+∇ ∇· v_θ(x,t) =0 =0 (39) Let XtX_t denote the trajectory, then we have: dtstθ(Xt)+∇vθ(Xt,t)⋅stθ(Xt)+∇⊤∇⋅vθ(Xt,t)=0 ddts^θ_t(X_t)+∇ v_θ(X_t,t)· s^θ_t(X_t)+∇ ∇· v_θ(X_t,t)=0 (40) dtstθ(Xt)=−∇vθ(Xt,t)⋅stθ(Xt)−∇⊤∇⋅vθ(Xt,t) ddts^θ_t(X_t)=-∇ v_θ(X_t,t)· s^θ_t(X_t)-∇ ∇· v_θ(X_t,t) (41) Therefore, we need to control ‖∇vθ‖\|∇ v_θ\| and ∥∇⊤∇⋅vθ∥\|∇ ∇· v_θ\|. We assume vθv_θ is smooth and constrained in a compact set, then we can have: ‖∇vθ‖≤C1 \|∇ v_θ\|≤ C_1 (42) ∥∇⊤∇⋅vθ∥≤C2 \|∇ ∇· v_θ\|≤ C_2 (43) We further assume that ‖vθ(0,t)‖≤C0\|v_θ(0,t)\|≤ C_0 uniformly over the parameter set. Then, we define g(λ)=vθ(λx,t)g(λ)=v_θ(λ x,t), λ∈[0,1]λ∈[0,1], and we have: g(1)−g(0) g(1)-g(0) =∫01dλg(λ)λ = _0^1 ddλg(λ)dλ (44) This is actually: vθ(x,t)−vθ(0,t)=∫01∇vθ(λx,t)xλ v_θ(x,t)-v_θ(0,t)= _0^1∇ v_θ(λ x,t)xdλ (45) Therefore, we have: ‖vθ(x,t)‖ \|v_θ(x,t)\| =‖vθ(0,t)+∫01∇vθ(λx,t)xλ‖ =\|v_θ(0,t)+ _0^1∇ v_θ(λ x,t)xdλ\| (46) ≤‖vθ(0,t)‖+‖∫01∇vθ(λx,t)xλ‖ ≤\|v_θ(0,t)\|+\| _0^1∇ v_θ(λ x,t)xdλ\| (47) ≤C0+∫01‖∇vθ(λx,t)‖‖x‖λ ≤ C_0+ _0^1\|∇ v_θ(λ x,t)\|\|x\|dλ (48) ≤C0+C1‖x‖ ≤ C_0+C_1\|x\| (49) Let XrX_r be the characteristic starting from Xt=xX_t=x, namely, Xr=x+∫trvθ(Xs,s)s,r∈[t,1].X_r=x+ _t^rv_θ(X_s,s)\,ds, r∈[t,1]. (50) Using the linear-growth bound above, we obtain ‖Xr‖ \|X_r\| ≤‖x‖+∫tr‖vθ(Xs,s)‖s ≤\|x\|+ _t^r\|v_θ(X_s,s)\|\,ds (51) ≤‖x‖+∫tr(C0+C1‖Xs‖)s. ≤\|x\|+ _t^r(C_0+C_1\|X_s\|)\,ds. (52) Therefore, Grönwall’s inequality gives ‖Xr‖≤eC1(r−t)‖x‖+C0C1(eC1(r−t)−1),r∈[t,1],\|X_r\|≤ e^C_1(r-t)\|x\|+ C_0C_1 (e^C_1(r-t)-1 ), r∈[t,1], (53) Since p1θ=(0,I)p_1^θ=N(0,I), we have s1θ(x)=−xs_1^θ(x)=-x. Then, by equation 41, we have: dtstθ(Xt) ddts^θ_t(X_t) =−∇vθ(Xt,t)⋅stθ(Xt)−∇⊤∇⋅vθ(Xt,t) =-∇ v_θ(X_t,t)· s^θ_t(X_t)-∇ ∇· v_θ(X_t,t) (54) ∫t1dstθ(Xt) ^1_tds^θ_t(X_t) =∫t1(−∇vθ(Xt,t)⋅stθ(Xt)−∇⊤∇⋅vθ(Xt,t))dt = ^1_t(-∇ v_θ(X_t,t)· s^θ_t(X_t)-∇ ∇· v_θ(X_t,t))dt (55) s1θ(X1)−stθ(Xt) s^θ_1(X_1)-s^θ_t(X_t) =∫t1(−∇vθ(Xt,t)⋅stθ(Xt)−∇⊤∇⋅vθ(Xt,t))dt = ^1_t(-∇ v_θ(X_t,t)· s^θ_t(X_t)-∇ ∇· v_θ(X_t,t))dt (56) ‖stθ(Xt)‖ \|s_t^θ(X_t)\| ≤‖s1θ(X1)‖+∫t1(C1‖srθ(Xr)‖+C2)r ≤\|s_1^θ(X_1)\|+ _t^1 (C_1\|s_r^θ(X_r)\|+C_2 )\,dr (57) =‖X1‖+∫t1(C1‖srθ(Xr)‖+C2)r. =\|X_1\|+ _t^1 (C_1\|s_r^θ(X_r)\|+C_2 )\,dr. (58) Applying Grönwall’s inequality once more and using equation 53 gives ‖stθ(x)‖ \|s_t^θ(x)\| ≤eC1(1−t)‖X1‖+C2C1(eC1(1−t)−1) ≤ e^C_1(1-t)\|X_1\|+ C_2C_1 (e^C_1(1-t)-1 ) (59) ≤K0+K1‖x‖, ≤ K_0+K_1\|x\|, (60) where K0,K1<∞K_0,K_1<∞ depend only on C0,C1,C2C_0,C_1,C_2 and are independent of θ. If M2:=pdata[‖0‖2]<∞M_2:=E_p_data[\|x_0\|^2]<∞, then [‖t‖2] [\|x_t\|^2] =[‖(1−t)0+t1‖2] =E[\|(1-t)x_0+tx_1\|^2] (61) =[(1−t)2‖0‖2+t2‖1‖2+2t(1−t)⟨0,1⟩] =E[(1-t)^2\|x_0\|^2+t^2\|x_1\|^2+2t(1-t) _0,x_1 ] (62) =(1−t)2[‖0‖2]+t2[‖1‖2]+2t(1−t)[⟨0,1⟩] =(1-t)^2E[\|x_0\|^2]+t^2E[\|x_1\|^2]+2t(1-t)E[ _0,x_1 ] (63) =(1−t)2[∥0∥2]+t2[∥1∥2]+2t(1−t)⟨[0],[1]⟩](by 0⟂1) =(1-t)^2E[\|x_0\|^2]+t^2E[\|x_1\|^2]+2t(1-t) [x_0],E[x_1] ] (by $ x_0 _1$) (64) =(1−t)2∥0∥2+t2∥1∥2(by [1]=0) =(1-t)^2E\|x_0\|^2+t^2E\|x_1\|^2 (by $ E[x_1]=0$) (65) =(1−t)2M2+t2d(by 1∼(0,Id)) =(1-t)^2M_2+t^2d (by $ x_1 (0,I_d)$) (66) ≤M2+d(by t∈[0,1]) ≤ M_2+d (by $t∈[0,1]$) (67) and consequently supθ,t∈[0,1]qt[‖∇logptθ(t)‖2]≤2K02+2K12(M2+d)=:Cp<∞. _θ,t∈[0,1]E_q_t[\|∇ p_t^θ(x_t)\|^2]≤ 2K_0^2+2K_1^2(M_2+d)=:C_p<∞. (68) It remains to control the score of the data path. By the identity in Lem. A.5, ∇logqt(t)=[−1t|t].∇ q_t(x_t)=E [- x_1t\, |\,x_t ]. (69) Hence, on every truncated interval t∈[ε,1]t∈[ ,1], qt[‖∇logqt(t)‖2] _q_t[\|∇ q_t(x_t)\|^2] =qt[‖−1t[1∣t]‖2] =E_q_t[\|- 1tE[x_1 _t]\|^2] (70) =1t2qt[‖[1∣t]‖2] = 1t^2E_q_t[\|E[x_1 _t]\|^2] (71) ≤1t2qt[[‖1‖2∣t]] ≤ 1t^2E_q_t[E[\|x_1\|^2 _t]] (72) =1t2[∥1∥2](by tower property) = 1t^2E[\|x_1\|^2] (by tower property) (73) ≤dε2. ≤ d ^2. (74) For the full interval, we use the endpoint integrability condition in the theorem, Cq:=supt∈(0,1]qt[‖∇logqt(t)‖2]<∞.C_q:= _t∈(0,1]E_q_t[\|∇ q_t(x_t)\|^2]<∞. (75) Combining the last two bounds yields [‖∇logqt(t)−∇logptθ(t)‖2]≤2(Cq+Cp)=:Cs<∞,E [\|∇ q_t(x_t)-∇ p_t^θ(x_t)\|^2 ]≤ 2(C_q+C_p)=:C_s<∞, (76) uniformly over θ. Substituting this estimate into Thm. A.6, and then using Young’s inequality, gives, for every κ>0κ>0, 0∼pdata[−logp0θ(0)] _x_0 p_data[- p_0^θ(x_0)] ≤H(pdata)+Cs[‖vθ(t,t)−(1−0)‖2] ≤ H(p_data)+ C_s\,E[\|v_θ(x_t,t)-(x_1-x_0)\|^2] (77) ≤κ[‖vθ(t,t)−(1−0)‖2]+H(pdata)+Cs4κ. ≤κ\,E[\|v_θ(x_t,t)-(x_1-x_0)\|^2]+H(p_data)+ C_s4κ. (78) Taking κ=1κ=1 proves the bound. The population-minimizer statement follows from Thm. A.6 under the stated well-specified interpretation. ∎ Appendix B Theoretical Proofs and Results B.1 Idealized Posteriors and Empirical Source Distributions The main text defines π+π^+ and π−π^- as posterior target distributions induced by the binary optimality variable. In offline preference optimization, however, the data available for training are usually fixed empirical source distributions π^+ π^+ and π^− π^- constructed by a separate pipeline, such as best-of-N winner/loser selection from a frozen generator. The formal identities in the main text only require that the source distributions be fixed during optimization. The practical modeling question is therefore whether the offline construction is a faithful enough surrogate for the idealized targets. In our experiments, the data construction pipeline is frozen, so the resulting preferred and dispreferred distributions remain stationary throughout training. B.2 Proof of Theorem 4.1 Proof. For each context c, equation 9 gives r(0,c)=12ω[logπ+(0∣c)−logπ−(0∣c)+logpπref(o=1∣c)pπref(o=0∣c)].r(x_0,c)= 12ω [ π^+(x_0 c)- π^-(x_0 c)+ p_ _ref(o=1 c)p_ _ref(o=0 c) ]. Taking expectation under πθ(⋅∣c) _θ(· c) yields 0∼πθ(⋅∣c)[r(0,c)]= _x_0 _θ(· c)[r(x_0,c)]= 12ω0∼πθ(⋅∣c)[logπ+(0∣c)−logπ−(0∣c)] 12ωE_x_0 _θ(· c) [ π^+(x_0 c)- π^-(x_0 c) ] +12ωlogpπref(o=1∣c)pπref(o=0∣c). + 12ω p_ _ref(o=1 c)p_ _ref(o=0 c). The second term depends only on c, hence is irrelevant for optimization over θ. For the first term, πθ[logπ+] _ _θ[ π^+] =−KL(πθ∥π+)−ℋ(πθ), =-D_KL( _θ\|π^+)-H( _θ), πθ[logπ−] _ _θ[ π^-] =−KL(πθ∥π−)−ℋ(πθ), =-D_KL( _θ\|π^-)-H( _θ), where ℋ(πθ)H( _θ) is the conditional differential entropy of πθ(⋅∣c) _θ(· c). Subtracting the two identities cancels the entropy term: πθ[logπ+−logπ−]=−KL(πθ∥π+)+KL(πθ∥π−).E_ _θ[ π^+- π^-]=-D_KL( _θ\|π^+)+D_KL( _θ\|π^-). Taking expectation over c and dropping the positive constant 12ω 12ω proves the claim. ∎ B.3 An Idealized Interpretation of Interpolated Vector Fields Proposition 1 of [20] gives the following relation when a velocity field and a score describe the same Gaussian marginal path (see also Lem. A.5): ut()=−t1−t∇logpt()−11−t.u_t(x)=- t1-t∇ p_t(x)- 11-tx. (79) For an idealized interpretation, suppose that both vθv_θ and voldv_old satisfy this compatibility condition: vθ(t,t) v_θ(x_t,t) =−t1−t∇logptθ(t)−11−t, =- t1-t∇ p^θ_t(x_t)- 11-tx_t, (80) vold(t,t) v_old(x_t,t) =−t1−t∇logptold(t)−11−t. =- t1-t∇ p^old_t(x_t)- 11-tx_t. (81) Then the positive interpolation satisfies v+(t,t) v^+(x_t,t) =vold(t,t)+β(vθ(t,t)−vold(t,t)) =v_old(x_t,t)+β(v_θ(x_t,t)-v_old(x_t,t)) (82) =−t1−t∇log[ptold(t)(ptθ(t)ptold(t))β]−11−t. =- t1-t∇ [p^old_t(x_t) ( p^θ_t(x_t)p^old_t(x_t) )^β ]- 11-tx_t. (83) Similarly, v−(t,t) v^-(x_t,t) =vold(t,t)−β(vθ(t,t)−vold(t,t)) =v_old(x_t,t)-β(v_θ(x_t,t)-v_old(x_t,t)) (84) =−t1−t∇log[ptold(t)(ptold(t)ptθ(t))β]−11−t. =- t1-t∇ [p^old_t(x_t) ( p^old_t(x_t)p^θ_t(x_t) )^β ]- 11-tx_t. (85) Thus, at each fixed t, v+v^+ and v−v^- admit a score-field interpretation analogous to the CFG [35, 19] interpolation pt(t,∅)(pt(t,c)pt(t,∅))βp_t(x_t, ) ( p_t(x_t,c)p_t(x_t, ) )^β over condition space. B.4 Proof of Theorem 4.2 Proof of Thm. 4.2. Expanding each KL term in equation 11 into cross-entropy plus entropy gives ℒOffline(θ)= _Offline(θ)= c[0+∼π+(⋅∣c)[−logπθ+(0+∣c)]+λ0−∼π−(⋅∣c)[−logπθ−(0−∣c)]]+Cent, _c [E_x_0^+ π^+(· c)[- _θ^+(x_0^+ c)]+ _x_0^- π^-(· c)[- _θ^-(x_0^- c)] ]+C_ent, where Cent=−c[ℋ(π+(⋅∣c))+λℋ(π−(⋅∣c))]C_ent=-E_c[H(π^+(· c))+ (π^-(· c))] is independent of θ. From Thm. A.7, we have: 0∼pdata[−logp0θ(0)]≤[‖vθ(t,t)−(1−0)‖2]+C.E_x_0 p_data[- p_0^θ(x_0)] [\|v_θ(x_t,t)-(x_1-x_0)\|^2 ]+C. (86) Therefore, we have: 0+∼π+(⋅∣c)[−logπθ+(0+∣c)] _x_0^+ π^+(· c)[- _θ^+(x_0^+ c)] ≤C+(c)+0+∼π+(⋅∣c)t,ϵ[‖μθ(t+,t,c)−ut(t+∣0+)‖22], ≤ C_+(c)+E_ subarraycx_0^+ π^+(· c)\\ t,\,ε subarray[\| _θ(x_t^+,t,c)-u_t(x_t^+ _0^+)\|_2^2], 0−∼π−(⋅∣c)[−logπθ−(0−∣c)] _x_0^- π^-(· c)[- _θ^-(x_0^- c)] ≤C−(c)+0−∼π−(⋅∣c)t,ϵ[‖νθ(t−,t,c)−ut(t−∣0−)‖22], ≤ C_-(c)+E_ subarraycx_0^- π^-(· c)\\ t,\,ε subarray[\| _θ(x_t^-,t,c)-u_t(x_t^- _0^-)\|_2^2], Then, ℒOffline(θ) _Offline(θ) ≤c,0+∼π+(⋅∣c)t,ϵ[‖μθ(t+,t,c)−ut(t+∣0+)‖22] _ subarraycc,x_0^+ π^+(· c)\\ t,\,ε subarray[\| _θ(x_t^+,t,c)-u_t(x_t^+ _0^+)\|_2^2] +λ⋅c,0−∼π−(⋅∣c)t,ϵ[∥νθ(t−,t,c)−ut(t−∣0−)∥22]+Constant. +λ·E_ subarraycc,x_0^- π^-(· c)\\ t,\,ε subarray[\| _θ(x_t^-,t,c)-u_t(x_t^- _0^-)\|_2^2]+Constant. This establishes an upper-bound surrogate for equation 11. If the two branch fields can simultaneously realize their population targets, then both branch endpoint distributions match their corresponding sources. ∎ B.5 Relation between FlowCPO and Simplified FlowDPO We isolate the contrastive core of FlowDPO by omitting its sigmoid weighting and frozen-reference terms. Using the preferred/dispreferred notation from equation 13, the simplified objective is minθℒFlowDPOsimple(θ)=[‖vθ(tw,t,c)−(ϵ−0w)‖22−‖vθ(tl,t,c)−(ϵ−0l)‖22]. _θL_FlowDPO^simple(θ)=E [\|v_θ(x_t^w,t,c)-(ε-x_0^w)\|_2^2-\|v_θ(x_t^l,t,c)-(ε-x_0^l)\|_2^2 ]. (87) It captures the per-pair contrastive gradient direction, without retaining the full FlowDPO objective. Below, E averages over the same fixed offline pairs and sampled t,ϵt,ε as in equation 13. We set λ=1λ=1 to match the unweighted preference comparison above. Theorem B.1 (Relation between FlowCPO and Simplified FlowDPO). Let β>0β>0, λ=1λ=1, and hold voldv_old fixed during each gradient step. Assuming the displayed expectations are finite, the FlowCPO loss satisfies ℒFlowCPO (θ)= _FlowCPO (θ)= βℒFlowDPOsimple(θ) _FlowDPO^simple(θ) (88) +β(β−1)[‖vθ(tw,t,c)−vold(tw,t,c)‖22] +β(β-1)E [\|v_θ(x_t^w,t,c)-v_old(x_t^w,t,c)\|_2^2 ] +β(β+1)[‖vθ(tl,t,c)−vold(tl,t,c)‖22]+C, +β(β+1)E [\|v_θ(x_t^l,t,c)-v_old(x_t^l,t,c)\|_2^2 ]+C, where C is independent of θ for the current detached voldv_old. Proof. The preferred branch uses twx_t^w and target ut(tw∣0w)=ϵ−0wu_t(x_t^w _0^w)=ε-x_0^w, while the dispreferred branch uses tlx_t^l and target ut(tl∣0l)=ϵ−0lu_t(x_t^l _0^l)=ε-x_0^l. We expand each branch separately. Preferred branch. Substituting μθ=(1−β)vold+βvθ _θ=(1-β)v_old+β v_θ and collecting terms gives μθ(tw,t,c)−ut(tw∣0w) _θ(x_t^w,t,c)-u_t(x_t^w _0^w) =vold(tw,t,c)−ut(tw∣0w)+β[vθ(tw,t,c)−vold(tw,t,c)]. =v_old(x_t^w,t,c)-u_t(x_t^w _0^w)+β [v_θ(x_t^w,t,c)-v_old(x_t^w,t,c) ]. Expanding the squared norm yields ‖μθ(tw,t,c)−ut(tw∣0w)‖22 \| _θ(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2 =‖vold(tw,t,c)−ut(tw∣0w)‖22 =\|v_old(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2 +β2‖vθ(tw,t,c)−vold(tw,t,c)‖22 +β^2\|v_θ(x_t^w,t,c)-v_old(x_t^w,t,c)\|_2^2 +2β⟨vold(tw,t,c)−ut(tw∣0w),vθ(tw,t,c)−vold(tw,t,c)⟩. +2β v_old(x_t^w,t,c)-u_t(x_t^w _0^w),v_θ(x_t^w,t,c)-v_old(x_t^w,t,c) . The residual of the current model is the sum of the EMA residual and vθ(tw,t,c)−vold(tw,t,c)v_θ(x_t^w,t,c)-v_old(x_t^w,t,c). Expanding the squared norm of this sum and solving for the cross term gives 2⟨vold(tw,t,c)−ut(tw∣0w),vθ(tw,t,c)−vold(tw,t,c)⟩ 2 v_old(x_t^w,t,c)-u_t(x_t^w _0^w),v_θ(x_t^w,t,c)-v_old(x_t^w,t,c) =‖vθ(tw,t,c)−ut(tw∣0w)‖22 =\|v_θ(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2 −‖vold(tw,t,c)−ut(tw∣0w)‖22−‖vθ(tw,t,c)−vold(tw,t,c)‖22. -\|v_old(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2-\|v_θ(x_t^w,t,c)-v_old(x_t^w,t,c)\|_2^2. Substituting this identity into the expansion gives coefficients β, β2−β=β(β−1)β^2-β=β(β-1), and 1−β1-β for the three squared errors, respectively: ‖μθ(tw,t,c)−ut(tw∣0w)‖22 \| _θ(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2 =β‖vθ(tw,t,c)−ut(tw∣0w)‖22 =β\|v_θ(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2 +β(β−1)‖vθ(tw,t,c)−vold(tw,t,c)‖22 +β(β-1)\|v_θ(x_t^w,t,c)-v_old(x_t^w,t,c)\|_2^2 +(1−β)‖vold(tw,t,c)−ut(tw∣0w)‖22. +(1-β)\|v_old(x_t^w,t,c)-u_t(x_t^w _0^w)\|_2^2. Dispreferred branch. Substituting νθ=(1+β)vold−βvθ _θ=(1+β)v_old-β v_θ now gives νθ(tl,t,c)−ut(tl∣0l) _θ(x_t^l,t,c)-u_t(x_t^l _0^l) =vold(tl,t,c)−ut(tl∣0l)−β[vθ(tl,t,c)−vold(tl,t,c)]. =v_old(x_t^l,t,c)-u_t(x_t^l _0^l)-β [v_θ(x_t^l,t,c)-v_old(x_t^l,t,c) ]. The minus sign changes the sign of the cross term, while the coefficient of the squared distance remains β2β^2: ‖νθ(tl,t,c)−ut(tl∣0l)‖22 \| _θ(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2 =‖vold(tl,t,c)−ut(tl∣0l)‖22 =\|v_old(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2 +β2‖vθ(tl,t,c)−vold(tl,t,c)‖22 +β^2\|v_θ(x_t^l,t,c)-v_old(x_t^l,t,c)\|_2^2 −2β⟨vold(tl,t,c)−ut(tl∣0l),vθ(tl,t,c)−vold(tl,t,c)⟩. -2β v_old(x_t^l,t,c)-u_t(x_t^l _0^l),v_θ(x_t^l,t,c)-v_old(x_t^l,t,c) . Evaluating the same norm identity on the dispreferred input gives 2⟨vold(tl,t,c)−ut(tl∣0l),vθ(tl,t,c)−vold(tl,t,c)⟩ 2 v_old(x_t^l,t,c)-u_t(x_t^l _0^l),v_θ(x_t^l,t,c)-v_old(x_t^l,t,c) =‖vθ(tl,t,c)−ut(tl∣0l)‖22 =\|v_θ(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2 −‖vold(tl,t,c)−ut(tl∣0l)‖22−‖vθ(tl,t,c)−vold(tl,t,c)‖22. -\|v_old(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2-\|v_θ(x_t^l,t,c)-v_old(x_t^l,t,c)\|_2^2. Substituting this identity gives coefficients −β-β, β2+β=β(β+1)β^2+β=β(β+1), and 1+β1+β: ‖νθ(tl,t,c)−ut(tl∣0l)‖22 \| _θ(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2 =−β‖vθ(tl,t,c)−ut(tl∣0l)‖22 =-β\|v_θ(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2 +β(β+1)‖vθ(tl,t,c)−vold(tl,t,c)‖22 +β(β+1)\|v_θ(x_t^l,t,c)-v_old(x_t^l,t,c)\|_2^2 +(1+β)‖vold(tl,t,c)−ut(tl∣0l)‖22. +(1+β)\|v_old(x_t^l,t,c)-u_t(x_t^l _0^l)\|_2^2. Take expectations over the preferred and dispreferred samples, respectively, and add. The first terms give βℒFlowDPOsimple _FlowDPO^simple, the second terms give the two corrections in equation 88, and the last terms form C. The latter use their respective sample targets and are independent of θ because the data and voldv_old are fixed during differentiation. ∎ Interpretation. The theorem expresses FlowCPO as β times simplified FlowDPO, plus two corrections determined by the distance between vθv_θ and voldv_old and a term C that is constant during each gradient step. At β=1β=1, only the quadratic constraint on dispreferred inputs remains, with coefficient 22. For 0<β<10<β<1, including our default β=0.5β=0.5, the preferred correction has a negative coefficient, so the two corrections should not both be described as positive regularizers. The ablation analysis of β is in Fig. 9. Why use EMA for interpolation? We use EMA so that the interpolation reference follows the model being trained. Thm. B.1 shows that the two correction terms depend on vθ−voldv_θ-v_old at the corresponding preferred or dispreferred input. These corrections depend on the squared prediction differences, so a reference that falls far behind can substantially change the objective. If we use the frozen vrefv_ref for interpolation, the reference stays at initialization while vθv_θ changes during fine-tuning. EMA updates the parameters of voldv_old from recent training iterates, with the aim of keeping this gap small. Thus, vrefv_ref provides the fixed in-domain offline data, while voldv_old follows training and is used to form μθ _θ and νθ _θ. Advantages of FlowCPO over FlowDPO. For λ=1λ=1, the two losses can be compared directly: ℒFlowDPOsimple(θ) _FlowDPO^simple(θ) =[‖vθ(tw,t,c)−(ϵ−0w)‖22−‖vθ(tl,t,c)−(ϵ−0l)‖22], =E [\|v_θ(x_t^w,t,c)-(ε-x_0^w)\|_2^2-\|v_θ(x_t^l,t,c)-(ε-x_0^l)\|_2^2 ], (89) ℒFlowCPO (θ) _FlowCPO (θ) =[‖μθ(tw,t,c)−(ϵ−0w)‖22+‖νθ(tl,t,c)−(ϵ−0l)‖22]. =E [\| _θ(x_t^w,t,c)-(ε-x_0^w)\|_2^2+\| _θ(x_t^l,t,c)-(ε-x_0^l)\|_2^2 ]. The minus sign allows simplified FlowDPO to decrease without bound if the rejected error grows while the preferred error stays bounded. FlowCPO adds two nonnegative errors, so its loss is bounded below by zero. This removes one potential source of instability, although a lower bound alone does not guarantee stable training. Appendix C Connections to Existing Methods Under the Unified Divergence Framework Sec. 4.3 and Tab. 2 summarize the connections under the unified divergence-based framework. C.1 Connection to FlowGRPO FlowGRPO [32] is an online RL algorithm for flow matching models. Ignoring the KL term added in practice for stabilization, its core objective is maxθc,0∼πθ(⋅∣c)[r(0,c)] _θE_c,\,x_0 _θ(· c)[r(x_0,c)] By Thm. 4.1, this is exactly equivalent to minθc[KL(πθ∥π+)−KL(πθ∥π−)] _θE_c [D_KL( _θ\|π^+)-D_KL( _θ\|π^-) ] Hence FlowGRPO belongs to the reverse-KL branch of equation 14 with qθ+=qθ−=πθ,α=1,γ=−1,q_θ^+=q_θ^-= _θ, α=1, γ=-1, plus the usual stabilization term KL(πθ∥πref)D_KL( _θ\| _ref) used in practice. C.2 Connection to DiffusionNFT DiffusionNFT [55] optimizes a supervised loss of the form ℒNFT(θ) _NFT(θ) =c,0∼πold(⋅∣c)t,ϵ[p(o=1∣0,c)∥μθ(t,t,c)−ut(t∣0)∥22 =E_ subarraycc,\,x_0 _old(· c)\\ t,\,ε subarray [p(o=1 _0,c)\,\| _θ(x_t,t,c)-u_t(x_t _0)\|_2^2 +p(o=0∣0,c)∥νθ(t,t,c)−ut(t∣0)∥22]. +p(o=0 _0,c)\,\| _θ(x_t,t,c)-u_t(x_t _0)\|_2^2 ]. (90) Similar to equation 8, we can have: πold(0∣c)p(o=1∣0,c)=π+(0∣c)pπold(o=1∣c) _old(x_0 c)\,p(o=1 _0,c)=π^+(x_0 c)\,p_ _old(o=1 c) πold(0∣c)p(o=0∣0,c)=π−(0∣c)pπold(o=0∣c) _old(x_0 c)\,p(o=0 _0,c)=π^-(x_0 c)\,p_ _old(o=0 c) Therefore, with formula equation 34 in Thm. A.7, equation 90 can be rewritten as ℒNFT(θ)= _NFT(θ)= c[pπold(o=1∣c)0∼π+(⋅∣c)t,ϵ‖μθ(t,t,c)−ut‖22] _c [p_ _old(o=1 c)\,E_ subarraycx_0 π^+(· c)\\ t,\,ε subarray\| _θ(x_t,t,c)-u_t\|_2^2 ] +c[pπold(o=0∣c)0∼π−(⋅∣c)t,ϵ‖νθ(t,t,c)−ut‖22] +E_c [p_ _old(o=0 c)\,E_ subarraycx_0 π^-(· c)\\ t,\,ε subarray\| _θ(x_t,t,c)-u_t\|_2^2 ] ≥c[pπold(o=1∣c)π+[−logπθ+(0∣c)] _c [p_ _old(o=1 c)\,E_π^+ [- π^+_θ(x_0 c) ] +pπold(o=0∣c)π−[−logπθ−(0∣c)]]+C +p_ _old(o=0 c)\,E_π^- [- π^-_θ(x_0 c) ] ]+C =c[pπold(o=1∣c)DKL(π+∥πθ+)+pπold(o=0∣c)DKL(π−∥πθ−)]+C =E_c[p_ _old(o=1 c)D_KL(π^+\|π^+_θ)+p_ _old(o=0 c)D_KL(π^-\|π^-_θ)]+C where C is a constant independent of θ. The both sides achieve the same optimal solution when μθ⋆(t+,t,c)=[1−0+∣t+,t,c] _θ (x_t^+,t,c)=E[x_1-x_0^+ _t^+,t,c] and νθ⋆(t−,t,c)=[1−0−∣t−,t,c] _θ (x_t^-,t,c)=E[x_1-x_0^- _t^-,t,c]. Therefore, DiffusionNFT corresponds to the forward-KL branch of equation 14 with qθ+=πθ+,qθ−=πθ−,α=pπold(o=1∣c),γ=pπold(o=0∣c).q_θ^+=π^+_θ, q_θ^-=π^-_θ, α=p_ _old(o=1 c), γ=p_ _old(o=0 c). C.3 Connection to AWM AWM [51] aims to optimize the following objective: maxθc,0∼πθ(⋅∣c)[r(0,c)] _θE_c,\,x_0 _θ(· c)[r(x_0,c)] (91) From equation 16, we have: −logπθ(x0∣c) - _θ(x_0 c) =H(qεx0)+t,ϵ[w(t)‖vθ(t,t,c)−(1−x0)‖2]+ε,w(θ,x0)+ℬε(θ,x0) =H(q_ ^x_0)+E_ subarrayct,\,ε subarray[w(t)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2 ]+G_ ,w(θ;x_0)+B_ (θ;x_0) (92) We set θ¯ θ as the stop gradient of θ, we have: logπθ(0∣c)πθ¯(0∣c) _θ(x_0 c) _ θ(x_0 c) =−t,ϵ[w(t)‖vθ(t,t,c)−(1−0)‖2] =-E_t,ε[w(t)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2] +t,ϵ[w(t)‖vθ¯(t,t,c)−(1−0)‖2] +E_t,ε[w(t)\|v_ θ(x_t,t,c)-(x_1-x_0)\|^2] :=ΔℒFM := _FM Therefore, the objective equation 91 can be rewritten as maxθc,0∼πθ(⋅∣c)[r(0,c)] _θE_c,\,x_0 _θ(· c)[r(x_0,c)] =c,0∼πθ¯(⋅∣c)[πθ(0∣c)πθ¯(0∣c)r(0,c)] =E_c,\,x_0 _ θ(· c) [ _θ(x_0 c) _ θ(x_0 c)r(x_0,c) ] =c,0∼πθ¯(⋅∣c)[exp(ΔℒFM)r(0,c)] =E_c,\,x_0 _ θ(· c) [ ( _FM)r(x_0,c) ] Therefore, AWM is optimized over forward process, while its derivation of qθ+,qθ−,α,γq^+_θ,q^-_θ,α,γ is the same as FlowGRPO. Thus, qθ+=qθ−=πθ,α=1,γ=−1,q_θ^+=q_θ^-= _θ, α=1, γ=-1, Because ΔℒFM=0 _FM=0 due to θ¯ θ is the stop gradient of θ, we know exp(x)=1+x (x)=1+x when x is small. Therefore, we can simplify the objective as maxθc,0∼πθ(⋅∣c)[r(0,c)] _θE_c,\,x_0 _θ(· c)[r(x_0,c)] =c,0∼πθ¯(⋅∣c)[exp(ΔℒFM)r(0,c)] =E_c,\,x_0 _ θ(· c) [ ( _FM)r(x_0,c) ] =c,0∼πθ¯(⋅∣c)[(1+ΔℒFM)r(0,c)] =E_c,\,x_0 _ θ(· c) [(1+ _FM)r(x_0,c) ] =c,0∼πθ¯(⋅∣c)[r(0,c)+ΔℒFMr(0,c)] =E_c,\,x_0 _ θ(· c) [r(x_0,c)+ _FMr(x_0,c) ] =−c,0∼πθ¯(⋅∣c)[r(0,c)t,ϵ[w(t)∥vθ(t,t,c)−(1−0)∥2]]+C =-E_c,\,x_0 _ θ(· c) [r(x_0,c)E_t,ε[w(t)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2 ] ]+C =−c,0∼πθ¯(⋅∣c),t,ϵ[w(t)r(0,c)∥vθ(t,t,c)−(1−0)∥2]+C =-E_c,x_0 _ θ(· c),t,ε [w(t)r(x_0,c)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2 ]+C C.4 Connection to RAM First, let’s remind the objective of RAM [3]: ℒRAM(θ)=c,t[‖vθ(t,t,c)−sg(vref(t,t,c)+r(0,c)((1−0)−vθ(t,t,c)))‖2] _RAM(θ)=E_c,t[\|v_θ(x_t,t,c)-sg(v_ref(x_t,t,c)+r(x_0,c)((x_1-x_0)-v_θ(x_t,t,c)))\|^2] where sgsg is the stop gradient operator. Let us define a new objective: ℒ2(θ)=c,t[r(0,c)‖vθ(t,t,c)−(1−0)‖2]+c,t[‖vθ(t,t,c)−vref(t,t,c)‖2] _2(θ)=E_c,t[r(x_0,c)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2]+E_c,t[\|v_θ(x_t,t,c)-v_ref(x_t,t,c)\|^2] It is easy to check that ∇θℒ2(θ)=∇θℒRAM(θ) _θL_2(θ)= _θL_RAM(θ). Therefore, RAM can be considered as AWM objective plus a reference term. Thus, it is also optimized over forward process, while its derivation of qθ+,qθ−,α,γq^+_θ,q^-_θ,α,γ is the same as FlowGRPO. Thus, qθ+=qθ−=πθ,α=1,γ=−1,q_θ^+=q_θ^-= _θ, α=1, γ=-1, C.5 Connection to SFT The reference distribution decomposes as a mixture of the positive and negative posteriors. Lemma C.1. For every context c, πref(0∣c)=pπref(o=1∣c)π+(0∣c)+pπref(o=0∣c)π−(0∣c). _ref(x_0 c)=p_ _ref(o=1 c)\,π^+(x_0 c)+p_ _ref(o=0 c)\,π^-(x_0 c). Proof. Multiplying the definitions in equation 8 by the corresponding normalizing constants and summing the two identities gives pπref(o=1∣c)π+(0∣c)+pπref(o=0∣c)π−(0∣c) p_ _ref(o=1 c)\,π^+(x_0 c)+p_ _ref(o=0 c)\,π^-(x_0 c) =πref(0∣c)[p(o=1∣0,c)+p(o=0∣0,c)] = _ref(x_0 c)\,[p(o=1 _0,c)+p(o=0 _0,c)] =πref(0∣c). = _ref(x_0 c). ∎ Standard SFT maximizes the log-likelihood of samples from πref _ref. By Thm. A.7, we know that when 0∼πref(⋅∣c)[∥vθ(t,t)−(1−0)∥2]E_x_0 _ref(· c) [\|v_θ(x_t,t)-(x_1-x_0)\|^2 ] achieves the minimum, 0∼πref(⋅∣c)[−logπθ(0∣c)]E_x_0 _ref(· c)[- _θ(x_0 c)] also achieves the minimum. Therefore, we have: By Lem. C.1, =argminθ0∼πref(⋅∣c)[−logπθ(0∣c)] = _θE_x_0 _ref(· c)[- _θ(x_0 c)] =∫πref(0∣c)[−logπθ(0∣c)]d0 = _ref(x_0 c)[- _θ(x_0 c)]dx_0 =∫(pπref(o=1∣c)π+(0∣c)+pπref(o=0∣c)π−(0∣c))[−logπθ(0∣c)]d0 = (p_ _ref(o=1 c)\,π^+(x_0 c)+p_ _ref(o=0 c)\,π^-(x_0 c))[- _θ(x_0 c)]dx_0 =pπref(o=1∣c)KL(π+∥πθ)+pπref(o=0∣c)KL(π−∥πθ) =p_ _ref(o=1 c)\,D_KL(π^+\| _θ)+p_ _ref(o=0 c)\,D_KL(π^-\| _θ) Thus SFT corresponds to the forward-KL branch of equation 14 with qθ+=qθ−=πθ,α=pπref(o=1∣c),γ=pπref(o=0∣c).q_θ^+=q_θ^-= _θ, α=p_ _ref(o=1 c), γ=p_ _ref(o=0 c). C.6 Connection to RFT RFT keeps only preferred samples. By Thm. A.7, we know that when 0∼πref(⋅∣c)[∥vθ(t,t)−(1−0)∥2]E_x_0 _ref(· c) [\|v_θ(x_t,t)-(x_1-x_0)\|^2 ] achieves the minimum, 0∼πref(⋅∣c)[−logπθ(0∣c)]E_x_0 _ref(· c)[- _θ(x_0 c)] also achieves the minimum. Therefore, we have: =argminθ0∼π+(⋅∣c)[−logπθ(0∣c)] = _θE_x_0 π^+(· c)[- _θ(x_0 c)] =argminθKL(π+∥πθ) = _θD_KL(π^+\| _θ) Therefore RFT corresponds to the forward-KL branch of equation 14 with qθ+=πθ,α=1,γ=0.q_θ^+= _θ, α=1, γ=0. C.7 Connection to FlowDPO The connection to FlowDPO [33] is heuristic rather than formal. Starting from the KL-regularized RLHF objective, DPO uses the reward parameterization r(0,c)=ηlogπθ(0∣c)πref(0∣c)+ηlogZ(c),r(x_0,c)=η _θ(x_0 c) _ref(x_0 c)+η Z(c), (93) which, when substituted into the Bradley–Terry preference likelihood, yields the usual DPO objective ℒDPO(θ)=c,0w,0l[−logσ(ηlogπθ(0w∣c)πref(0w∣c)−ηlogπθ(0l∣c)πref(0l∣c))]L_DPO(θ)=E_c,x_0^w,x_0^l [- σ (η _θ(x_0^w c) _ref(x_0^w c)-η _θ(x_0^l c) _ref(x_0^l c) ) ] From equation 16, we have: −logπθ(x0∣c) - _θ(x_0 c) =H(qεx0)+t,ϵ[w(t)‖vθ(t,t,c)−(1−x0)‖2]+ε,w(θ,x0)+ℬε(θ,x0) =H(q_ ^x_0)+E_ subarrayct,\,ε subarray[w(t)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2 ]+G_ ,w(θ;x_0)+B_ (θ;x_0) Therefore, we have: logπθ(0∣c)πref(0∣c)=ΔℒFM(0)+Δε,w(0)+Δℬε(0) _θ(x_0 c) _ref(x_0 c)= _FM(x_0)+ _ ,w(x_0)+ _ (x_0) (94) where ΔℒFM(0) _FM(x_0) :=t,ϵ[−w(t)‖vθ(t,t,c)−(1−0)‖2+w(t)‖vref(t,t,c)−(1−0)‖2] :=E_ subarrayct,\,ε subarray[-w(t)\|v_θ(x_t,t,c)-(x_1-x_0)\|^2+w(t)\|v_ref(x_t,t,c)-(x_1-x_0)\|^2 ] Δε,w(0) _ ,w(x_0) :=−ε,w(θ,0)+ε,w(ref,0) :=-G_ ,w(θ;x_0)+G_ ,w(ref;x_0) Δℬε(0) _ (x_0) :=−ℬε(θ,0)+ℬε(ref,0) :=-B_ (θ;x_0)+B_ (ref;x_0) Therefore, we can simplify the DPO objective in the following: ℒDPO(θ) _DPO(θ) =c,0w,0l[−logσ(ηΔℒFM(0w)+Δε,w(0w)+Δℬε(0w) =E_c,x_0^w,x_0^l [- σ (η\ _FM(x^w_0)+ _ ,w(x^w_0)+ _ (x^w_0) −ΔℒFM(0l)−Δε,w(0l)−Δℬε(0l)) - _FM(x^l_0)- _ ,w(x^l_0)- _ (x^l_0)\ ) =c,0w,0l[−logσ(η(ΔℒFM(0w)−ΔℒFM(0l)) =E_c,x_0^w,x_0^l [- σ (η( _FM(x^w_0)- _FM(x^l_0)) +η(Δε,w(0w)+Δℬε(0w)−Δε,w(0l)−Δℬε(0l)))] +η( _ ,w(x^w_0)+ _ (x^w_0)- _ ,w(x^l_0)- _ (x^l_0))\ ) ] because −logσ- σ is convex and then by Jensen’s inequality ≤c,0w,0l[−12logσ(2η(ΔℒFM(0w)−ΔℒFM(0l))) _c,x_0^w,x_0^l [- 12 σ (2η( _FM(x^w_0)- _FM(x^l_0)) ) −12logσ(2η(Δε,w(0w)+Δℬε(0w)−Δε,w(0l)−Δℬε(0l)))] - 12 σ (2η( _ ,w(x^w_0)+ _ (x^w_0)- _ ,w(x^l_0)- _ (x^l_0)) ) ] ≤12ℒFlowDPO(θ) ≤ 12L_FlowDPO(θ) +c,0w,0l[−12logσ(2η(Δε,w(0w)+Δℬε(0w)−Δε,w(0l)−Δℬε(0l)))] +E_c,x_0^w,x_0^l [- 12 σ (2η( _ ,w(x^w_0)+ _ (x^w_0)- _ ,w(x^l_0)- _ (x^l_0)) ) ] where δθ(0):=‖vθ(t,t,c)−(1−0)‖2−‖vref(t,t,c)−(1−0)‖2. _θ(x_0):=\|v_θ(x_t,t,c)-(x_1-x_0)\|^2-\|v_ref(x_t,t,c)-(x_1-x_0)\|^2. ℒFlowDPO(θ)=c,0w,0l,t,ϵ[−logσ(2ηw(t)[δθ(0l)−δθ(0w)])]. _FlowDPO(θ)=E_c,x_0^w,x_0^l,t,ε [- σ (2η w(t)[ _θ(x_0^l)- _θ(x_0^w)] ) ]. Therefore, the FlowDPO objective plus a residual term is an upper bound of the DPO objective. Appendix D Experimental Details D.1 Experimental Details of Sec. 5.2 Algorithm 1 Flow Contrastive Preference Optimization (FlowCPO) 1: Pretrained vrefv_ref, dataset =(c,0w,0l)D=\(c,x^w_0,x^l_0)\, flow interpolation coefficient β, negative regularization weight λ, EMA coefficient η, learning rate ι . 2: Init: vθ←vrefv_θ← v_ref, vold←vrefv_old← v_ref. 3: for each training iteration do 4: Sample batch (c,0w,0l)∼(c,x_0^w,x_0^l) , t∼(0,1)t (0,1), ϵ∼(0,I)ε (0,I). 5: Forward: tw,l=(1−t)⋅0w,l+t⋅ϵx_t^\w,l\=(1-t)·x_0^\w,l\+t·ε; Target utw,l=ϵ−0w,lu_t^\w,l\=ε-x_0^\w,l\. 6: Flow Mixing: 7: μθ=(1−β)vold+βvθ,νθ=(1+β)vold−βvθ _θ=(1-β)v_old+β v_θ, 9.24994pt _θ=(1+β)v_old-β v_θ. 8: Loss Update: 9: ℒ=‖μθ(tw,c,t)−ut(tw|0w)‖22+λ‖νθ(tl,c,t)−ut(tl|0l)‖22L=\| _θ(x_t^w,c,t)-u_t(x_t^w|x_0^w)\|_2^2+λ\| _θ(x_t^l,c,t)-u_t(x_t^l|x_0^l)\|_2^2 10: θ←θ−ι⋅∇θℒθ←θ- · _θL. 11: vold←EMAη(vθ,vold)v_old← EMA_η(v_θ,v_old). 12: end for 13: return vθv_θ In-Domain Offline Preference Data. For the in-domain offline training setting, we build a static preference dataset from a frozen copy of the pretrained reference model Stable Diffusion 3.5 Medium (SD3.5-M). We sample prompts from the three target sources: the GenEval prompt set, the OCR prompt set, and the general-preference prompt set. The resulting in-domain prompt pools contain 50,000 training prompts for GenEval, 19,653 training prompts for OCR, and 25,432 training prompts for general preference. Their held-out evaluation splits contain 2212, 1018, and 2048 prompts, respectively. For each prompt, we generate 16 candidate images with the frozen reference model and convert them into a preferred/dispreferred pair using a fixed offline pipeline: the highest-scoring sample becomes the preferred and the lowest-scoring sample becomes the dispreferred. The GenEval and OCR training data use single-metric filtering aligned with their target benchmark. The general-preference training data uses an equal-weight multi-reward score with PickScore:HPS v2.1:CLIP Score = 1:1:1 during preferred/dispreferred construction. Because the generator parameters stay fixed during data construction, the resulting training distribution remains stationary throughout optimization. These preferred/dispreferred pairs should be read as empirical source distributions induced by the fixed best-of-16 pipeline, rather than as exact samples from the idealized posteriors π+π^+ and π−π^- introduced in the theory. All images used for data construction and evaluation are generated at a resolution of 512×512 with 40 inference steps using the ODE sampler. Unless otherwise noted, we use a guidance scale of 4.5 during dataset construction. We leave negative prompts blank. The VAE and text encoder are frozen, and training prompts do not overlap with validation prompts or evaluation benchmarks. The complete reported experiment suite used approximately 288 aggregate A100 GPU-hours. Out-of-Domain Offline Preference Data. For the out-of-domain offline training setting, we use Open Image Preferences v1 Results from the Data Is Better Together collection on Hugging Face [7]. This dataset contains roughly 10K text-to-image preference pairs with community annotations, and the images are generated by open models including FLUX.1-Dev and SD3.5-Large. We use it to evaluate the out-of-domain regime because the training samples are not generated by the reference model used in our fine-tuning experiments. Training Configuration. We fine-tune SD3.5-M with LoRA on 8×A100 GPUs in fp16 precision. Unless otherwise specified, the LoRA hyperparameters are r=32r=32 and α=64α=64. We optimize with AdamW using a learning rate of 3e-4, cosine decay, and weight decay of 1e-4. The global batch size is 16. The EMA decay η is 0.99, the default value of λ is 1, and reported checkpoints are selected with the corresponding validation score for each training target. Checkpoint Selection. All fine-tuning runs are monitored for 2K training steps. For the in-domain specialists, we report the checkpoint with the best task-aligned validation score within that 2K-step budget: the GenEval model is selected by GenEval validation score, the OCR model by OCR validation score, and the general-preference model by the equal-weight composite validation score over PickScore, HPS v2.1, and CLIP Score. For the OOD setting, where one model is shared across all downstream evaluations, we select the checkpoint with the highest validation PickScore within the same 2K-step budget. We apply the same protocol to FlowCPO and the offline baselines in the corresponding setting. D.2 Experimental Results on Optimization Targets We report results for three capabilities: semantic alignment, typographic generation, and general preference alignment. Tab. 5, Tab. 6, and Tab. 7 correspond to in-domain fine-tuning with domain-specific reward filtering, whereas Tab. 8 reports the out-of-domain setting. The first three tables therefore summarize separate in-domain specialist models, while the OOD table evaluates single models trained once on out-of-domain data. Unless otherwise noted, each fine-tuned SD3.5-M entry is the mean over five independent training and evaluation runs with different random seeds, and the uncertainty is the sample standard deviation of the five run-level metric values. Pretrained baselines are shown as single evaluations for reference. Tab. 5 reports quantitative results on GenEval, with representative qualitative comparisons in Fig. 3. Tab. 6 summarizes typographic generation performance measured by OCR accuracy, with examples in Fig. 4. Tab. 7 presents the in-domain results for general preference alignment; because training data in that setting are filtered with PickScore, CLIP Score, and HPS v2.1, the remaining metrics are the more informative check of transfer beyond the filtering pipeline. Tab. 8 reports the out-of-domain setting, with qualitative examples in Fig. 6, Fig. 7, and Fig. 8. Unless otherwise noted, we use FlowDPO temperature 100 and FlowCPO β=0.5β=0.5 as the default settings. Table 5: Quantitative comparison of fine-tuning SD3.5-M using different methods under the in-domain offline training setting. The fine-tuning data are filtered solely by the GenEval reward function. We evaluate each model with several Classifier-Free Guidance (CFG) scales. The broad set of auxiliary metrics is included to monitor whether optimizing only for GenEval degrades other capabilities. Fine-tuned SD3.5-M variants are reported as mean ± standard deviation over 5 runs; pretrained baselines are listed as single evaluations. Within the SD3.5-M group, the best results are highlighted in bold, and the second best are underlined. Model CFG GenEval OCR PickScore ClipScore HPSv2.1 Aesthetic ImgRwd UniRwd Pretrained Model Baselines SD-XL −- 0.55 0.14 22.42 0.287 0.280 5.60 0.76 2.93 SD3.5-L −- 0.71 0.68 22.91 0.289 0.288 5.50 0.96 3.25 FLUX.1-Dev −- 0.66 0.59 22.84 0.295 0.274 5.71 0.96 3.27 SD3.5-M Fine-Tuning (filtered by GenEval) Base Model 1.01.0 0.24 0.12 20.51 0.237 0.204 5.13 −-0.58 2.02 3.03.0 0.59 0.47 22.28 0.287 0.284 5.38 0.71 2.96 4.54.5 0.63 0.59 22.34 0.285 0.279 5.36 0.85 3.03 + RFT [49, 5] 1.01.0 0.5883±0.01000.5883± 0.0100 0.1421±0.00330.1421± 0.0033 21.6937±0.012521.6937± 0.0125 0.2735±0.00090.2735± 0.0009 0.2686±0.00050.2686± 0.0005 5.3183±0.01555.3183± 0.0155 0.4381±0.01790.4381± 0.0179 2.6175±0.01352.6175± 0.0135 3.03.0 0.7423±0.00970.7423± 0.0097 0.5003±0.00700.5003± 0.0070 22.4661±0.021222.4661± 0.0212 0.2941±0.00060.2941± 0.0006 0.3010±0.00060.3010± 0.0006 5.3998±0.00925.3998± 0.0092 1.0200±0.01721.0200± 0.0172 3.1360±0.01453.1360± 0.0145 4.54.5 0.7489±0.00450.7489± 0.0045 0.5562±0.00980.5562± 0.0098 22.4864±0.009922.4864± 0.0099 0.2966±0.00050.2966± 0.0005 0.3031±0.00020.3031± 0.0002 5.4070±0.00505.4070± 0.0050 1.0808±0.00741.0808± 0.0074 3.1615±0.00803.1615± 0.0080 + FlowDPO [33] 1.01.0 0.5892±0.00460.5892± 0.0046 0.2327±0.00530.2327± 0.0053 21.4067±0.032421.4067± 0.0324 0.2696±0.00100.2696± 0.0010 0.2562±0.00070.2562± 0.0007 5.2817±0.00585.2817± 0.0058 0.4560±0.02250.4560± 0.0225 2.5870±0.02052.5870± 0.0205 3.03.0 0.8118±0.00440.8118± 0.0044 0.4534±0.00750.4534± 0.0075 22.3095±0.018222.3095± 0.0182 0.2970±0.00060.2970± 0.0006 0.2938±0.00090.2938± 0.0009 5.3570±0.01125.3570± 0.0112 1.0811±0.00941.0811± 0.0094 3.1615±0.02003.1615± 0.0200 4.54.5 0.8107±0.00460.8107± 0.0046 0.5067±0.00740.5067± 0.0074 22.3091±0.013822.3091± 0.0138 0.2979±0.00050.2979± 0.0005 0.2963±0.00030.2963± 0.0003 5.3479±0.00525.3479± 0.0052 1.1256±0.00781.1256± 0.0078 3.1680±0.00603.1680± 0.0060 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.7596±0.01060.7596± 0.0106 0.2535±0.00980.2535± 0.0098 21.8045±0.015721.8045± 0.0157 0.2804±0.00010.2804± 0.0001 0.2673±0.00080.2673± 0.0008 5.2487±0.01395.2487± 0.0139 0.6231±0.01600.6231± 0.0160 2.8040±0.01252.8040± 0.0125 3.03.0 0.8415±0.00360.8415± 0.0036 0.5203±0.00550.5203± 0.0055 22.1952±0.018422.1952± 0.0184 0.2957±0.00050.2957± 0.0005 0.2885±0.00070.2885± 0.0007 5.2662±0.00855.2662± 0.0085 0.9999±0.00760.9999± 0.0076 3.1170±0.00853.1170± 0.0085 4.54.5 0.8161±0.00520.8161± 0.0052 0.5702±0.00820.5702± 0.0082 21.9438±0.008621.9438± 0.0086 0.2933±0.00050.2933± 0.0005 0.2807±0.00040.2807± 0.0004 5.1610±0.00625.1610± 0.0062 0.9184±0.00940.9184± 0.0094 3.0245±0.01503.0245± 0.0150 Table 6: Quantitative comparison of fine-tuning SD3.5-M using different methods (RFT, FlowDPO, FlowCPO) under the in-domain offline training setting. The fine-tuning data are filtered solely by the OCR reward metric. We evaluate the models with several Classifier-Free Guidance (CFG) scales. The broad set of auxiliary metrics is included to monitor whether optimizing only for OCR degrades other generation capabilities. Fine-tuned SD3.5-M variants are reported as mean ± standard deviation over 5 runs; pretrained baselines are listed as single evaluations. Within the SD3.5-M fine-tuning group, the best results are highlighted in bold, and the second best are underlined. Model CFG GenEval OCR PickScore ClipScore HPSv2.1 Aesthetic ImgRwd UniRwd Pretrained Model Baselines SD-XL −- 0.55 0.14 22.42 0.287 0.280 5.60 0.76 2.93 SD3.5-L −- 0.71 0.68 22.91 0.289 0.288 5.50 0.96 3.25 FLUX.1-Dev −- 0.66 0.59 22.84 0.295 0.274 5.71 0.96 3.27 SD3.5-M Fine-Tuning (filtered by OCR) Base Model 1.01.0 0.24 0.12 20.51 0.237 0.204 5.13 −-0.58 2.02 3.03.0 0.59 0.47 22.28 0.287 0.284 5.38 0.71 2.96 4.54.5 0.63 0.59 22.34 0.285 0.279 5.36 0.85 3.03 + RFT [49, 5] 1.01.0 0.5127±0.01080.5127± 0.0108 0.3530±0.00570.3530± 0.0057 21.8831±0.019221.8831± 0.0192 0.2784±0.00070.2784± 0.0007 0.2741±0.00050.2741± 0.0005 5.3589±0.00965.3589± 0.0096 0.5248±0.01910.5248± 0.0191 2.7170±0.01602.7170± 0.0160 3.03.0 0.6577±0.01060.6577± 0.0106 0.6981±0.01050.6981± 0.0105 22.5177±0.019422.5177± 0.0194 0.2964±0.00080.2964± 0.0008 0.3008±0.00030.3008± 0.0003 5.3954±0.00435.3954± 0.0043 1.0491±0.00721.0491± 0.0072 3.1630±0.02003.1630± 0.0200 4.54.5 0.6698±0.01010.6698± 0.0101 0.7202±0.00830.7202± 0.0083 22.5098±0.015722.5098± 0.0157 0.2979±0.00090.2979± 0.0009 0.3023±0.00050.3023± 0.0005 5.3927±0.00685.3927± 0.0068 1.0816±0.01001.0816± 0.0100 3.1605±0.01753.1605± 0.0175 + FlowDPO [33] 1.01.0 0.3140±0.00510.3140± 0.0051 0.5136±0.00500.5136± 0.0050 20.9171±0.009020.9171± 0.0090 0.2565±0.00080.2565± 0.0008 0.2219±0.00060.2219± 0.0006 5.1246±0.01275.1246± 0.0127 −0.3015±0.0273-0.3015± 0.0273 2.2580±0.01752.2580± 0.0175 3.03.0 0.5848±0.00980.5848± 0.0098 0.7389±0.00960.7389± 0.0096 22.3210±0.010422.3210± 0.0104 0.2927±0.00040.2927± 0.0004 0.2873±0.00050.2873± 0.0005 5.3620±0.00255.3620± 0.0025 0.8559±0.01430.8559± 0.0143 3.0315±0.02203.0315± 0.0220 4.54.5 0.6149±0.01530.6149± 0.0153 0.7476±0.00410.7476± 0.0041 22.4249±0.014422.4249± 0.0144 0.2960±0.00060.2960± 0.0006 0.2943±0.00050.2943± 0.0005 5.3795±0.00645.3795± 0.0064 0.9493±0.01820.9493± 0.0182 3.0955±0.01953.0955± 0.0195 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.4023±0.00620.4023± 0.0062 0.8349±0.00330.8349± 0.0033 21.2178±0.018521.2178± 0.0185 0.2689±0.00060.2689± 0.0006 0.2430±0.00110.2430± 0.0011 5.1770±0.01025.1770± 0.0102 0.0952±0.01820.0952± 0.0182 2.4955±0.01852.4955± 0.0185 3.03.0 0.5962±0.01130.5962± 0.0113 0.8737±0.00340.8737± 0.0034 22.3139±0.009022.3139± 0.0090 0.2930±0.00020.2930± 0.0002 0.2943±0.00050.2943± 0.0005 5.3506±0.00795.3506± 0.0079 0.9560±0.01150.9560± 0.0115 3.0760±0.00853.0760± 0.0085 4.54.5 0.6124±0.01020.6124± 0.0102 0.8588±0.00780.8588± 0.0078 22.3656±0.015422.3656± 0.0154 0.2953±0.00080.2953± 0.0008 0.2982±0.00050.2982± 0.0005 5.3591±0.00485.3591± 0.0048 1.0438±0.00481.0438± 0.0048 3.1160±0.00353.1160± 0.0035 Table 7: Quantitative comparison of fine-tuning SD3.5-M using different methods (RFT, FlowDPO, and FlowCPO) under the in-domain offline training setting. The fine-tuning data are filtered by a combination of multiple metrics (PickScore, CLIP Score, and HPSv2.1), which are highlighted in gray. We evaluate the models with several Classifier-Free Guidance (CFG) scales. Because the shaded metrics are directly coupled to the filtering pipeline, the remaining metrics are especially useful for assessing transfer beyond the optimization-aligned rewards. Fine-tuned SD3.5-M variants are reported as mean ± standard deviation over 5 runs; pretrained baselines are listed as single evaluations. Within the SD3.5-M fine-tuning group, the best results are highlighted in bold, and the second best are underlined. Model CFG GenEval OCR PickScore ClipScore HPSv2.1 Aesthetic ImgRwd UniRwd Pretrained Model Baselines SD-XL −- 0.55 0.14 22.42 0.287 0.280 5.60 0.76 2.93 SD3.5-L −- 0.71 0.68 22.91 0.289 0.288 5.50 0.96 3.25 FLUX.1-Dev −- 0.66 0.59 22.84 0.295 0.274 5.71 0.96 3.27 SD3.5-M Fine-Tuning Base Model 1.01.0 0.24 0.12 20.51 0.237 0.204 5.13 −-0.58 2.02 3.03.0 0.59 0.47 22.28 0.287 0.284 5.38 0.71 2.96 4.54.5 0.63 0.59 22.34 0.285 0.279 5.36 0.85 3.03 + RFT [49, 5] 1.01.0 0.5312±0.01110.5312± 0.0111 0.2351±0.00450.2351± 0.0045 21.9056±0.012421.9056± 0.0124 0.2793±0.00020.2793± 0.0002 0.2759±0.00080.2759± 0.0008 5.3507±0.01115.3507± 0.0111 0.5694±0.02130.5694± 0.0213 2.6590±0.00652.6590± 0.0065 3.03.0 0.6894±0.00860.6894± 0.0086 0.5722±0.00450.5722± 0.0045 22.5975±0.006922.5975± 0.0069 0.2963±0.00070.2963± 0.0007 0.3041±0.00050.3041± 0.0005 5.4139±0.00935.4139± 0.0093 1.0616±0.00961.0616± 0.0096 3.1370±0.01003.1370± 0.0100 4.54.5 0.6974±0.00560.6974± 0.0056 0.6180±0.00970.6180± 0.0097 22.5664±0.010422.5664± 0.0104 0.2972±0.00090.2972± 0.0009 0.3054±0.00050.3054± 0.0005 5.4207±0.00615.4207± 0.0061 1.1068±0.00631.1068± 0.0063 3.1520±0.00203.1520± 0.0020 + FlowDPO [33] 1.01.0 0.0363±0.01170.0363± 0.0117 0.2132±0.00610.2132± 0.0061 20.7158±0.017920.7158± 0.0179 0.2399±0.00060.2399± 0.0006 0.2205±0.00100.2205± 0.0010 5.2202±0.01135.2202± 0.0113 −0.4610±0.0222-0.4610± 0.0222 2.1075±0.01202.1075± 0.0120 3.03.0 0.3903±0.01000.3903± 0.0100 0.5549±0.00720.5549± 0.0072 22.7637±0.020922.7637± 0.0209 0.2974±0.00080.2974± 0.0008 0.3014±0.00070.3014± 0.0007 5.5590±0.00895.5590± 0.0089 1.0403±0.00891.0403± 0.0089 3.0915±0.01953.0915± 0.0195 4.54.5 0.4610±0.01370.4610± 0.0137 0.5791±0.00710.5791± 0.0071 22.8912±0.019222.8912± 0.0192 0.3010±0.00080.3010± 0.0008 0.3107±0.00060.3107± 0.0006 5.5562±0.01075.5562± 0.0107 1.1543±0.00721.1543± 0.0072 3.1840±0.01903.1840± 0.0190 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.5325±0.00350.5325± 0.0035 0.3426±0.00530.3426± 0.0053 22.4805±0.015522.4805± 0.0155 0.2796±0.00070.2796± 0.0007 0.2900±0.00030.2900± 0.0003 5.6392±0.01005.6392± 0.0100 0.9014±0.01390.9014± 0.0139 2.9300±0.02002.9300± 0.0200 3.03.0 0.6487±0.00880.6487± 0.0088 0.5888±0.00960.5888± 0.0096 22.9432±0.016422.9432± 0.0164 0.2994±0.00050.2994± 0.0005 0.3114±0.00040.3114± 0.0004 5.5395±0.00655.5395± 0.0065 1.2480±0.00711.2480± 0.0071 3.3020±0.01103.3020± 0.0110 4.54.5 0.6526±0.00790.6526± 0.0079 0.5762±0.01360.5762± 0.0136 22.6725±0.008822.6725± 0.0088 0.2997±0.00080.2997± 0.0008 0.3030±0.00040.3030± 0.0004 5.4569±0.00455.4569± 0.0045 1.2174±0.01161.2174± 0.0116 3.2775±0.00953.2775± 0.0095 Table 8: Quantitative comparison of fine-tuning SD3.5-M using different methods (RFT, FlowDPO, and FlowCPO) under the out-of-domain offline training setting. The fine-tuning data come from the open-source Open Image Preferences v1 Results dataset and were generated by models other than the SD3.5-M reference policy. We evaluate the models with several Classifier-Free Guidance (CFG) scales. Fine-tuned SD3.5-M variants are reported as mean ± standard deviation over 5 runs; pretrained baselines are listed as single evaluations. Within the SD3.5-M fine-tuning group, the best results are highlighted in bold, and the second best are underlined. Model CFG GenEval OCR PickScore ClipScore HPSv2.1 Aesthetic ImgRwd UniRwd Pretrained Model Baselines SD-XL −- 0.55 0.14 22.42 0.287 0.280 5.60 0.76 2.93 SD3.5-L −- 0.71 0.68 22.91 0.289 0.288 5.50 0.96 3.25 FLUX.1-Dev −- 0.66 0.59 22.84 0.295 0.274 5.71 0.96 3.27 SD3.5-M Fine-Tuning Base Model 1.01.0 0.24 0.12 20.51 0.237 0.204 5.13 −-0.58 2.02 3.03.0 0.59 0.47 22.28 0.287 0.284 5.38 0.71 2.96 4.54.5 0.63 0.59 22.34 0.285 0.279 5.36 0.85 3.03 + RFT [49, 5] 1.01.0 0.4398±0.00860.4398± 0.0086 0.1723±0.00610.1723± 0.0061 21.5913±0.019421.5913± 0.0194 0.2678±0.00160.2678± 0.0016 0.2611±0.00100.2611± 0.0010 5.4923±0.01295.4923± 0.0129 0.3110±0.02160.3110± 0.0216 2.5551±0.01382.5551± 0.0138 3.03.0 0.6650±0.00760.6650± 0.0076 0.5310±0.00780.5310± 0.0078 22.6427±0.010822.6427± 0.0108 0.2974±0.00080.2974± 0.0008 0.3032±0.00020.3032± 0.0002 5.4780±0.00825.4780± 0.0082 1.0514±0.01341.0514± 0.0134 3.1766±0.02023.1766± 0.0202 4.54.5 0.6845±0.00780.6845± 0.0078 0.5968±0.00590.5968± 0.0059 22.6854±0.006322.6854± 0.0063 0.2991±0.00070.2991± 0.0007 0.3074±0.00030.3074± 0.0003 5.4733±0.00745.4733± 0.0074 1.1228±0.01331.1228± 0.0133 3.2299±0.00843.2299± 0.0084 + FlowDPO [33] 1.01.0 0.2290±0.00840.2290± 0.0084 0.1158±0.00540.1158± 0.0054 20.8273±0.015520.8273± 0.0155 0.2425±0.00110.2425± 0.0011 0.2264±0.00060.2264± 0.0006 5.7158±0.01475.7158± 0.0147 −0.3851±0.0285-0.3851± 0.0285 2.2693±0.01982.2693± 0.0198 3.03.0 0.6065±0.01410.6065± 0.0141 0.4789±0.00530.4789± 0.0053 22.5399±0.021222.5399± 0.0212 0.2919±0.00070.2919± 0.0007 0.2917±0.00080.2917± 0.0008 5.5203±0.00325.5203± 0.0032 0.9063±0.01090.9063± 0.0109 3.0972±0.01703.0972± 0.0170 4.54.5 0.6513±0.01320.6513± 0.0132 0.5406±0.00710.5406± 0.0071 22.6201±0.016522.6201± 0.0165 0.2946±0.00030.2946± 0.0003 0.2993±0.00030.2993± 0.0003 5.4927±0.01085.4927± 0.0108 0.9934±0.01640.9934± 0.0164 3.1694±0.01173.1694± 0.0117 + FlowCPO (β=1β=1, Ours) 1.01.0 0.4670±0.01100.4670± 0.0110 0.2351±0.00270.2351± 0.0027 21.8088±0.019821.8088± 0.0198 0.2762±0.00110.2762± 0.0011 0.2704±0.00030.2704± 0.0003 5.3546±0.01075.3546± 0.0107 0.5230±0.01400.5230± 0.0140 2.7723±0.01522.7723± 0.0152 3.03.0 0.6958±0.00500.6958± 0.0050 0.5561±0.00850.5561± 0.0085 22.4609±0.018822.4609± 0.0188 0.2966±0.00030.2966± 0.0003 0.2952±0.00040.2952± 0.0004 5.4140±0.00295.4140± 0.0029 1.0166±0.00751.0166± 0.0075 3.2263±0.01003.2263± 0.0100 4.54.5 0.6995±0.00600.6995± 0.0060 0.5912±0.00690.5912± 0.0069 22.3570±0.011122.3570± 0.0111 0.2942±0.00070.2942± 0.0007 0.2942±0.00020.2942± 0.0002 5.3810±0.00565.3810± 0.0056 1.0159±0.00891.0159± 0.0089 3.2173±0.01843.2173± 0.0184 + FlowCPO (β=0.5β=0.5, Ours) 1.01.0 0.5706±0.01240.5706± 0.0124 0.2360±0.00470.2360± 0.0047 22.0231±0.014622.0231± 0.0146 0.2813±0.00060.2813± 0.0006 0.2752±0.00050.2752± 0.0005 5.3451±0.01065.3451± 0.0106 0.6294±0.00640.6294± 0.0064 2.8523±0.02002.8523± 0.0200 3.03.0 0.6859±0.00990.6859± 0.0099 0.4919±0.00710.4919± 0.0071 22.3229±0.006422.3229± 0.0064 0.2959±0.00100.2959± 0.0010 0.2882±0.00030.2882± 0.0003 5.3718±0.00745.3718± 0.0074 0.9967±0.00620.9967± 0.0062 3.1464±0.00893.1464± 0.0089 4.54.5 0.6815±0.00400.6815± 0.0040 0.4912±0.00540.4912± 0.0054 22.1181±0.012822.1181± 0.0128 0.2939±0.00090.2939± 0.0009 0.2842±0.00050.2842± 0.0005 5.3073±0.00695.3073± 0.0069 0.9452±0.01080.9452± 0.0108 3.1068±0.01173.1068± 0.0117 Figure 3: Qualitative comparison on the GenEval benchmark. All samples are generated by models trained with in-domain preference data. Figure 4: Qualitative comparison on the OCR benchmark. All samples are generated by models trained with in-domain preference data. Figure 5: Qualitative comparison on the DrawBench benchmark. All samples are generated by models trained with in-domain preference data. Figure 6: Qualitative comparison on the GenEval benchmark. All samples are generated by models trained with out-of-domain preference data. Figure 7: Qualitative comparison on the OCR benchmark. All samples are generated by models trained with out-of-domain preference data. Figure 8: Qualitative comparison on the DrawBench benchmark. All samples are generated by models trained with out-of-domain preference data. D.3 Additional Analyses Beyond Main Results D.3.1 Understanding the Role of Negative Regularization Tab. 4 shows a clear pattern within the GenEval-only preference training setup: how negative samples enter the objective matters at least as much as whether they enter at all. Positive-only regularization provides a stable baseline and remains competitive across all three metric groups, especially under stronger CFG. Repulsive negative regularization is less reliable. Mild repulsion (λ=−0.1λ=-0.1) stays close to positive-only training, but stronger repulsion (λ=−1λ=-1) hurts most metrics, and overly aggressive repulsion (λ=−10λ=-10) leads to numerical instability and collapsed training. This is consistent with the view that explicitly pushing the model away from dispreferred samples can over-amplify contrastive signals and destabilize optimization. At moderate weights, attractive negative regularization gives the best trade-off in this ablation between alignment strength and training stability. The configuration with β=0.5β=0.5 and λ=1λ=1 achieves the highest GenEval score (0.84), whereas repulsion with λ=−1λ=-1 degrades all reported metrics. The configuration with β=1β=1 and λ=0.1λ=0.1 also attains the best OCR (0.59), ImgReward (1.14), and UniReward (3.23). Because all models in this table are trained with GenEval-only supervision, these gains on typography and general-preference metrics should be interpreted as cross-metric transfer rather than direct optimization. Both signs diverge at magnitude 10, so the evidence supports moderate attractive matching rather than a blanket stability claim. D.3.2 Hyperparameter Sensitivity and Ablation To study the hyperparameter sensitivity of the proposed objective function (equation 13), we conduct ablations on the GenEval benchmark and track performance across training steps. (a) Effect of β (b) Effect of λ (c) Effect of η Figure 9: Hyperparameter sensitivity within the GenEval-only setup. We monitor the GenEval score during training to analyze the impact of (a) the flow interpolation coefficient β, (b) the negative loss weight λ, and (c) the EMA parameter η. The figure is intended to show relative stability trends for the tested values. Flow Interpolation Coefficient (β). β controls how far the target flows move away from the reference prior. As shown in Fig. 9(a), moderate values (β∈[0.5,1.0]β∈[0.5,1.0]) perform best in this sweep on GenEval. In particular, β=0.5β=0.5 achieves the highest peak score. Negative Regularization Weight (λ). Balancing positive alignment and negative regularization is important. Fig. 9(b) shows that a balanced choice (λ=1.0λ=1.0) yields the most robust long-term performance among the tested values on GenEval. Reference Prior Stability (η). FlowCPO relies on a stable reference prior inside the flow-matching objective. Fig. 9(c) shows that updating the reference model voldv_old with a high-EMA decay rate (η=0.99η=0.99) yields the most stable behavior in this sweep. D.4 Per-Prompt Diversity Evaluation The primary empirical claim in RQ2 is that FlowCPO provides stable offline preference optimization with a forward-KL objective in the evaluated regime, not that it universally produces more diverse samples than FlowDPO. To evaluate sample diversity directly, we compute the Vendi Score [11] separately over eight samples generated for each prompt and report the mean per-prompt score. We evaluate two classifier-free guidance (CFG) scales, 1.0 and 4.5. For each target column in Tab. 9, the quality score and Vendi Score are obtained from the corresponding specialist: GenEval-only fine-tuning for GenEval, OCR-only fine-tuning for OCR, and multi-reward fine-tuning for PickScore, CLIPScore, HPSv2.1. The main entry in each cell is the quality score, while the parenthesized entry is the Vendi Score. Higher is better for both. Table 9: Target quality and per-prompt diversity on SD3.5-M. Vendi Score is computed from eight samples per prompt and reported in parentheses. Each metric column evaluates the specialist fine-tuned for that target. Bold independently marks the highest quality score and the highest Vendi Score in each column. Method CFG GenEval ↑ (Vendi ↑ ) OCR ↑ (Vendi ↑ ) PickScore ↑ (Vendi ↑ ) SD3.5-M 1.0 0.24 (2.1573) 0.12 (2.3073) 20.51 (2.2824) 4.5 0.63 (1.6540) 0.59 (1.7778) 22.34 (1.7742) RFT 1.0 0.59 (1.8210) 0.35 (2.2417) 21.91 (1.9740) 4.5 0.75 (1.4547) 0.72 (1.4984) 22.57 (1.5544) FlowDPO 1.0 0.59 (1.8926) 0.51 (2.5117) 20.72 (1.8671) 4.5 0.81 (1.5594) 0.75 (1.6310) 22.89 (1.5966) FlowCPO (β=0.5β=0.5) 1.0 0.76 (1.7385) 0.83 (2.0578) 22.48 (1.8936) 4.5 0.82 (1.4773) 0.86 (1.4955) 22.69 (1.5251) Results. The results show a consistent quality–diversity trade-off: stronger CFG improves target quality while reducing per-prompt diversity. Across all comparisons in Tab. 9, increasing CFG from 1.0 to 4.5 raises the target quality score and lowers the corresponding Vendi Score. At CFG 1.0, FlowCPO achieves the highest quality among the compared methods for all three targets (0.76 GenEval, 0.83 OCR, and 22.48 PickScore), while retaining higher diversity than its own CFG-4.5 outputs. In particular, lowering CFG from 4.5 to 1.0 increases the Vendi Score of FlowCPO by 0.2612, 0.5623, and 0.3685 on the three targets, respectively, with corresponding quality decreases of 0.06, 0.03, and 0.21. However, FlowCPO does not maximize raw Vendi Score: the pretrained model is most diverse in the GenEval and PickScore columns. Appendix E Broader Impacts and Release Considerations Potential positive impacts. By turning preference alignment into a strictly offline optimization problem, our method can reduce the need for repeated online rollouts during alignment and make experimentation on continuous generative models more compute-efficient. Better alignment on structure-sensitive tasks such as compositional generation and text rendering can also improve the controllability and practical usefulness of text-to-image systems in benign applications such as design ideation, educational content creation, and accessibility-related graphics. Potential negative impacts. The same improvement in preference alignment can increase the capability of image generators to produce more convincing synthetic content, including misleading text-in-image content or other deceptive media. In addition, the method can inherit biases from offline preference data and from the automatic reward signals used to construct winner/loser pairs, and the out-of-domain results in Sec. 5 together with the limitations discussed in Sec. 6 indicate that behavior may degrade when the offline data distribution differs from the reference model. Release considerations. We do not release new model checkpoints, a new dataset, or a code package with this preprint; the paper describes the training objective and experimental protocol only. The experiments rely on existing pretrained models, public benchmarks, and a public OOD preference dataset under their original access conditions, and any future release of code or checkpoints should preserve the upstream licenses, terms of use, and usage restrictions of those assets.