Paper deep dive
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
Yingqing Guo, Hui Yuan, Zijian He, Mengdi Wang, Zheng Ding
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/8/2026, 3:27:18 AM
Summary
The paper introduces LC-GRPO, a framework for flow-based generative models that bridges the training-inference gap in Group Relative Policy Optimization (GRPO). It replaces standard Stochastic Differential Equation (SDE) rollouts with a hybrid approach: an inference-aligned ODE Euler step followed by a Langevin correction. This method preserves sample quality while enabling stochastic exploration for reinforcement learning, theoretically reducing Wasserstein error compared to standard discretizations. Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate improved reward optimization and reduced discrepancy between training rollouts and test-time inference.
Entities (13)
Relation Signals (11)
LC-GRPO → narrows → training-inference gap
confidence 95% · substantially narrows the gap between stochastic training rollouts and deterministic test-time ODE inference
LC-GRPO → uses → Langevin correction
confidence 95% · We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction.
LC-GRPO → evaluatedon → FLUX.1 [Dev]
confidence 90% · Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization
LC-GRPO → evaluatedon → SD3.5-Medium
confidence 90% · Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization
LC-GRPO → evaluatedon → HunyuanVideo
confidence 90% · Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization
LC-GRPO → improves → reward optimization
confidence 90% · LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks
Flow-GRPO → outperformedby → LC-GRPO
confidence 90% · LC-GRPO consistently improves the optimized rewards over Flow-GRPO
DanceGRPO → outperformedby → LC-GRPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE), whereas online reinforcement learning requires stochastic rollouts for policy exploration and optimization. Existing GRPO methods for flow models therefore replace the inference-time ODE with a stochastic differential equation (SDE) during training. Although the ODE and SDE share the same marginal distributions in continuous time, their finite-step discretizations can differ substantially. In particular, SDE rollouts often become blurry as the exploration noise increases, creating a mismatch between the samples used for reinforcement learning and those generated by the test-time ODE sampler. We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction. Each rollout transition first takes an inference-aligned ODE Euler step and then applies a stochastic Langevin correction targeting the marginal distribution at the resulting timestep. The required score is recovered directly from the flow velocity, requiring no additional score model, while the resulting transition remains an isotropic Gaussian with a tractable likelihood for policy optimization. We theoretically show that, under suitable conditions, one Langevin correction step reduces the Wasserstein error of an imperfect ODE Euler step. At a matched randomness level, we further show that the proposed transition can be more accurate than the standard Euler--Maruyama discretization of the reverse SDE. Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks, preserves generation quality, and substantially narrows the gap between stochastic training rollouts and deterministic test-time ODE inference.
Tags
Links
- Source: https://arxiv.org/abs/2608.05600v1
- Canonical: https://arxiv.org/abs/2608.05600v1
Trouble viewing inline? Open PDF directly →
Full Text
71,490 characters extracted from source content.
Expand or collapse full text
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction Yingqing Guo Hui Yuan Zijian He Mengdi Wang Zheng Ding Abstract Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE), whereas online reinforcement learning requires stochastic rollouts for policy exploration and optimization. Existing GRPO methods for flow models therefore replace the inference-time ODE with a stochastic differential equation (SDE) during training. Although the ODE and SDE share the same marginal distributions in continuous time, their finite-step discretizations can differ substantially. In particular, SDE rollouts often become blurry as the exploration noise increases, creating a mismatch between the samples used for reinforcement learning and those generated by the test-time ODE sampler. We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction. Each rollout transition first takes an inference-aligned ODE Euler step and then applies a stochastic Langevin correction targeting the marginal distribution at the resulting timestep. The required score is recovered directly from the flow velocity, requiring no additional score model, while the resulting transition remains an isotropic Gaussian with a tractable likelihood for policy optimization. We theoretically show that, under suitable conditions, one Langevin correction step reduces the Wasserstein error of an imperfect ODE Euler step. At a matched randomness level, we further show that the proposed transition can be more accurate than the standard Euler–Maruyama discretization of the reverse SDE. Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks, preserves generation quality, and substantially narrows the gap between stochastic training rollouts and deterministic test-time ODE inference. 1 Introduction Flow matching and rectified-flow models have emerged as a powerful foundation for visual generation, supporting high-quality text-to-image and text-to-video synthesis through deterministic ordinary differential equation (ODE) sampling (Liu et al., 2022; Lipman et al., 2022; Esser et al., 2024; Kong et al., 2024). Compared with stochastic diffusion sampling, the near-straight transport paths learned by these models enable accurate generation with relatively few sampling steps, making them especially attractive for large-scale visual models. As their generative capabilities improve, post-training with external feedback has become increasingly important for adapting them to objectives that are difficult to capture through supervised learning alone, including human preference, visual quality, compositional correctness, and accurate text rendering. Online reinforcement learning methods such as DDPO and, more recently, Group Relative Policy Optimization (GRPO), provide a direct way to optimize such non-differentiable rewards from generated samples (Black et al., 2024; Liu et al., 2026a; Xue et al., 2025). This issue is not unique to visual generation: training-inference gap has also emerged in LLM RL post-training, where separate inference and training stacks can introduce numerical and algorithmic inconsistencies, causing the rollout policy used for data collection to differ from the policy being optimized(Qi et al., 2025; Wasti et al., ; Yao et al., 2025; Liang et al., 2026). Figure 1: Performance of LC-GRPO. Top: Langevin correction narrows the gap between stochastic RL rollouts and ODE inference at test time, as shown at step 0, enabling LC-GRPO to achieve the smallest training–evaluation gap. Bottom: Evaluation results on DrawBench (Saharia et al., 2022). Applying online reinforcement learning to flow models, however, introduces a fundamental mismatch between training and inference. At test time, flow models are typically sampled by solving a deterministic reverse-time ODE. Reinforcement learning instead requires a stochastic policy that can explore multiple actions from the same state and whose transition likelihood can be evaluated. Existing flow-based GRPO methods therefore replace the inference ODE with a theoretically equivalent reverse-time stochastic differential equation (SDE) during rollout generation. In continuous time, the ODE and SDE share the same marginal distributions. In practice, however, both processes must be discretized using a small number of sampling steps, and their finite-step behavior can differ substantially. The Euler–Maruyama discretization used for SDE rollouts accumulates error from both the discretized drift and the injected Brownian noise, whereas test-time ODE sampling follows a cleaner deterministic trajectory. We address this tension with LC-GRPO, a flow-based GRPO framework based on Langevin correction. Rather than directly simulating the reverse SDE, each rollout transition first follows the same ODE Euler update used by the inference sampler. We then apply a single Langevin step targeting the marginal distribution at the resulting timestamp. The required score is obtained directly from the flow velocity through Tweedie’s formula, so the correction requires no additional score model. This predictor–corrector construction separates transport from exploration: the ODE step preserves the inference-time sampling trajectory, while the Langevin step introduces stochasticity and corrects local sampling error around the target marginal. Importantly, the resulting transition remains an isotropic Gaussian with a tractable density, allowing it to be used directly as the stochastic policy in the GRPO objective. The Langevin step size naturally controls the amount of policy exploration. We provide theoretical justification for this design. Under strong log-concavity and score smoothness, we show that whenever an ODE Euler step has nonzero error, an appropriately chosen Langevin correction strictly decreases its Wasserstein distance to the target marginal. We further compare Langevin-corrected sampling with the standard reverse-SDE Euler–Maruyama step at a matched randomness level. Under mild regularity conditions and an explicit comparison condition, the two methods have errors of the same asymptotic order, but the Langevin-corrected step has a strictly smaller leading-order Wasserstein error. These results formalize the intuition that stochastic exploration need not come at the cost of the sampling accuracy achieved by the inference ODE. We evaluate LC-GRPO on both image and video generation using SD3.5-Medium(Esser et al., 2024), FLUX.1-Dev(Labs, 2024), and HunyuanVideo(Kong et al., 2024). The experiments cover visual text rendering, human-preference alignment, multi-reward optimization, and video-quality optimization. Across these settings, LC-GRPO consistently improves the optimized rewards over Flow-GRPO, DanceGRPO, and coefficients-preserving sampling, while maintaining a favorable reward–quality trade-off on auxiliary image and video metrics. It also produces substantially clearer stochastic rollouts and the smallest discrepancy between training-rollout and ODE-evaluation rewards. These gains are obtained under a rollout-computation budget that matches or favors the baselines. To summarize, our main contributions are as follows: 1. We identify finite-step SDE discretization as a central source of the training–inference gap in flow-based GRPO and show its adverse effect on rollout quality and reward alignment. 2. We introduce LC-GRPO, which combines inference-aligned ODE transport with a distribution-preserving Langevin correction to provide accurate, stochastic, and likelihood-tractable reinforcement-learning rollouts. 3. We establish theoretical guarantees for the proposed correction and demonstrate consistent improvements across text-to-image and text-to-video reinforcement-learning tasks. 2 Related Work We first review alignment methods for diffusion and flow models, including post-training with external feedback and inference-time guidance, and then focus on closing the training–inference gap in flow-based GRPO and LLM RL post-training. RL for Diffusion and Flow Models. Aligning diffusion and flow models during post-training generally follows one of three strategies. (1) Direct reward backpropagation, which fine-tunes the model by backpropagating through a differentiable reward function (Clark et al., 2024; Prabhudesai et al., 2023). (2) Reward-augmented regression losses, which combine external feedback with a diffusion regression objective. These include Reward-Weighted Regression (RWR) (Lee et al., 2023; Dong et al., 2023); Direct Preference Optimization (DPO), which leverages paired preference data with binary human-feedback rewards (Rafailov et al., 2023; Wallace et al., 2024; Yang et al., 2024); and DiffusionNFT (Zheng et al., 2025), which generalizes DPO by incorporating negative samples with dense, continuous rewards. (3) PPO-style policy-gradient methods, ranging from early approaches such as DPOK (Fan et al., 2023) and DDPO (Black et al., 2024) to recent GRPO extensions for diffusion and flow models (Liu et al., 2026a; Xue et al., 2025; Deng et al., 2026; Savani et al., 2026; Ding & Ye, 2025). Beyond model post-training, complementary inference-time alignment methods guide diffusion and flow sampling using time-dependent classifiers or gradients derived from downstream objectives (Song et al., 2020b; Dhariwal & Nichol, 2021; Chung et al., 2022; Yuan et al., 2023; Guo et al., 2024; 2026). Training–Inference Gap in Flow-Based GRPO. Recent work explores reducing the mismatch between stochastic training rollouts and deterministic inference samplers in flow-based GRPO. MixGRPO (Li et al., 2025) integrates SDE and ODE sampling within the rollout to improve training efficiency. CPS (Wang & Yu, 2025) instead adopts DDIM sampling to improve sample quality. SAGE-GRPO (Zheng et al., 2026) proposes a concise formulation of the noise standard deviation that better aligns the sampling process with the data manifold for video generation. Training–Inference Mismatch in LLM RL Post-Training. Practical LLM RL systems often use separate inference engines for rollout generation and separate frameworks for training, which can introduce a mismatch. One line of work addresses numerical inconsistencies (Qi et al., 2025; Qiu et al., 2026; Wasti et al., ); for example, Qi et al. (2025) switch from BF16 to FP16 during RL fine-tuning. A complementary line of work takes an algorithmic approach by incorporating sampler-side information into the training update (Yao et al., 2025; Liu et al., 2025; Liang et al., 2026). Notably, Yao et al. (2025) use the training-to-sampler probability ratio as a clipped correction weight. 3 Preliminaries Flow Matching. Let 0∼X0 x_0 X_0 denote a sample from the data distribution and 1∼X1 x_1 X_1 denote a noise sample from the prior distribution, such as a standard Gaussian. Rectified Flow (Liu et al., 2022; Lipman et al., 2022) constructs a time-dependent interpolation between 0 x_0 and 1 x_1: t=(1−t)0+t1,t∈[0,1]. x_t=(1-t) x_0+t x_1, t∈[0,1]. Under this convention, t=0t=0 corresponds to data and t=1t=1 to noise. Generation is therefore performed by solving the reverse-time ODE from the prior distribution back to the data distribution: dt=tdt,d x_t= v_tdt, (1) with 1∼X1 x_1 X_1 and t:1→0t:1→ 0. The velocity field t v_t transports samples from the prior distribution to the data distribution. In practice, it is approximated by a transformer θ(t,t) v_θ( x_t,t), trained with the flow matching regression objective: minθt,0,1[‖θ(t,t)−(1−0)‖22]. _θE_t, x_0, x_1 [ \| v_θ( x_t,t)-( x_1- x_0) \|_2^2 ]. The learned velocity field is then used to solve the reverse-time ODE (Eq. 1) for generation. SDE Sampling as RL Rollout. The inference process of flow matching can be formulated as a Markov decision process (MDP) (Black et al., 2024). At denoising step t, the state is defined as t:=(,t,t) s_t:=( c,t, x_t), the action is the next denoised sample t:=t−1 a_t:= x_t-1 predicted by the model, and the policy is given by π(t∣t):=pθ(t−1∣t,)π( a_t s_t):=p_θ( x_t-1 x_t, c). However, ODE-based sampling is deterministic and induces no action-level stochasticity, which blocks the stochastic exploration required by reinforcement learning. To resolve this mismatch, Flow-GRPO (Liu et al., 2026a) and DanceGRPO (Xue et al., 2025) replace the reverse-time ODE with an equivalent SDE: dt=[t(t,t)+σt22t(t+(1−t)t(t,t))]dt+σtdt,d x_t= [ v_t( x_t,t)+ _t^22t ( x_t+(1-t) v_t( x_t,t) ) ]dt+ _td w_t, (2) which adds noise at each denoising step, enabling trajectories to serve as stochastic RL rollouts. GRPO on Flow Models. Given a prompt c, the flow model pθp_θ generates G independent denoising trajectories (ti)t=0Ti=1G\( x^i_t)_t=0^T\_i=1^G, each ending at sample 0i x^i_0. The advantage of the i-th sample is computed by normalizing its final reward R(0i,)R( x^i_0, c) against the rewards of other samples in the same group: A^ti=R(0i,)−mean(R(0j,)j=1G)std(R(0j,)j=1G). A^i_t= R( x^i_0, c)-mean(\R( x^j_0, c)\_j=1^G)std(\R( x^j_0, c)\_j=1^G). (3) The policy is trained by maximizing the GRPO objective: (θ)=∼,ii=1G∼πθold(⋅∣)f(r,A^,θ,ε,β),J(θ)=E_ c ,\ x^i\_i=1^G _ _old(· c)f(r, A,θ, ,β), (4) where f(r,A^,θ,ε,β) f(r, A,θ, ,β) =1G∑i=1G1T∑t=1T(min(rti(θ)A^ti,clip(rti(θ),1−ε,1+ε)A^ti)−βDKL(πθ||πref)), = 1G _i=1^G 1T _t=1^T ( (r^i_t(θ) A^i_t,\ clip (r^i_t(θ),1- ,1+ ) A^i_t )-β D_KL( _θ|| _ref) ), rti(θ) r^i_t(θ) =pθ(t−1i∣ti,)pθold(t−1i∣ti,). = p_θ( x^i_t-1 x^i_t, c)p_ _old( x^i_t-1 x^i_t, c). In flow models, an action is one reverse-time sampling step from ti x^i_t to t−1i x^i_t-1, obtained by discretizing Eq. 2 (see Eq. 6). The resulting transition probability pθ(t−1i∣ti,)p_θ( x^i_t-1 x^i_t, c) is an isotropic Gaussian, whose density can be computed in closed form. 4 Method In this section, we first revisit the training–inference gap in standard GRPO for flow models, where SDE sampling is used to generate RL rollouts. We then propose LC-GRPO, which introduces an additional Langevin correction step to reduce the gap between stochastic RL rollouts and deterministic ODE inference. We validate this approach both theoretically and empirically. Training–Inference Gap from SDE Rollouts. During inference, although the ODE (Eq. 1) and SDE (Eq. 2) reach the same marginal distribution at each timestep in continuous time (Kloeden et al., 2012), their finite-step numerical samplers can differ. ODE Euler step is usually taken for current advanced image and video generation (Esser et al., 2024; Wu et al., 2025; Wan et al., 2025): t−Δt=t−t(t,t)Δt. x_t- t= x_t- v_t( x_t,t) t. (5) Euler–Maruyama discretization is commonly used for SDE rollouts in GRPO for flow models: t−Δt=t−[t(t,t)+σt22t(t+(1−t)t(t,t))]Δt+σtΔtξ, x_t- t= x_t- [ v_t( x_t,t)+ _t^22t ( x_t+(1-t) v_t( x_t,t) ) ] t+ _t t\,ξ, (6) where ξ∼(0,Id)ξ (0,I_d). For the diffusion coefficient, Flow-GRPO sets σt=ηt/(1−t) _t=η t/(1-t). Here, η controls the noise level, and hence the amount of stochasticity and exploration during RL training. Both methods show that larger η values often accelerate GRPO training and lead to higher rewards. As shown in Fig. 2, SDE-based sampling tends to produce lower-quality samples than ODE-based sampling, yielding blurrier images; this degradation becomes more severe as the noise level increases. Similar behavior has been observed in prior work (Lipman et al., 2022; Song et al., 2020b). This discrepancy creates a training–inference gap in RL for flow models: SDE rollouts are worse than inference-time ODE samples, making their rewards a less accurate and less effective learning signal. Figure 2: Sampling Method Comparison. As the noise level222The noise level refers to η in the SDE (Eq. 6), where σt=ηt/(1−t) _t=η t/(1-t) following Flow-GRPO, and in the Langevin step (Eq. 7), where we set 2ϵt=η(t−Δt) 2 _t=η(t- t). For the same η, the sum of the variance of the Gaussian noise over all timesteps is larger for Langevin than for the SDE.increases, SDE-based sampling degrades sample quality, whereas ODE sampling with Langevin correction preserves sample clarity. Langevin Step: Distribution-Preserving Stochastic Correction. Motivated by the training–inference gap, we aim to design a sampler that narrows this mismatch. SDE-based sampling accumulates error from both drift discretization and Brownian increments. To avoid introducing additional error through direct SDE simulation, we first take an ODE Euler step from t x_t to obtain an intermediate proposal ′ x at time t−Δt- t. We then seek a stochastic update that satisfies two requirements: (i) it injects stochasticity while preserving the target distribution at time t−Δt- t; and (i) it improves sample quality by mitigating local numerical and discretization errors. We adopt a simple yet effective strategy: after each ODE step, we apply one Langevin dynamics step initialized at ′ x , using the score estimate at time t−Δt- t: t−Δt=′+ϵt(′,t−Δt)+2ϵtξ, x_t- t= x + _t\, s( x ,t- t)+ 2 _t\,ξ, (7) where ϵt _t is the step size. The score function is obtained via Tweedie’s formula (Efron, 2011) as (,t)=−(+(1−t)(,t))/t s( x,t)=- ( x+(1-t) v( x,t) )/t. Following Song et al. (2020b), we set ϵt∝(‖/‖)2 _t ( \| z \|/ \| s \| )^2. For stability, we use its expectation, which gives 2ϵt=η(t−Δt) 2 _t=η(t- t), where η controls the noise level. This correction has the desired properties. First, Langevin dynamics driven by the score at time t−Δt- t leaves the marginal pt−Δtp_t- t invariant, thereby introducing randomness while preserving that marginal. Second, it serves as a corrector step: prior work on score-based generative modeling has shown that such stochastic correction can improve sampling quality (Song et al., 2020b). As Fig 2 shows, sampling with Langevin correction preserves sample quality even under large noise levels. Algorithm 1 Langevin Correction Sampling 0: flow model θ v_θ, Langevin step size ϵt _t, time schedule tii=1N\t_i\_i=1^N with t1=1,tN=0t_1=1,\;t_N=0 1: 1∼(,Id) x_1 ( 0,I_d) 2: for i=1,…,N−1i=1,…,N-1 do 3: ′←i−(ti−ti+1)θ(i,ti) x ← x_i-(t_i-t_i+1) v_θ( x_i,t_i) ⊳ Euler step 4: if i<N−1i<N-1 then 5: ti+1(′)←−′+(1−ti+1)θ(′,ti+1)ti+1 s_t_i+1( x )←- x +(1-t_i+1) v_θ( x ,t_i+1)t_i+1 ⊳ Score estimation 6: ∼(,Id) z ( 0,I_d) 7: i+1←′+ϵtiti+1(′)+2ϵti x_i+1← x + _t_i s_t_i+1( x )+ 2 _t_i\, z ⊳ Langevin correction 8: else 9: i+1←′ x_i+1← x ⊳ Final ODE step 10: end if 11: end for 12: return N x_N As for the MDP we formulate for GRPO, the transition kernel pθ(t−Δt∣t,)p_θ( x_t- t x_t, c) remains an isotropic Gaussian (θ(t),2ϵtId)N( m_θ( x_t),2 _tI_d), whose mean θ(t)=′+ϵtθ(′,t−Δt) m_θ( x_t)= x + _t s_θ( x ,t- t) is the deterministic part of the Langevin update in Alg. 1, where ′=t−Δtθ(t,t) x = x_t- t\, v_θ( x_t,t) is the Euler step. Thus, the Langevin step size ϵt _t also controls the extent of exploration of the action policy. The following theorem justifies the correction step in Alg. 1: whenever the Euler step is imperfect, one Langevin step provably moves the sample closer to the target marginal. The full statement and proof are deferred to Appx. A and Appx. A.1, respectively. Theorem 1 (Informal). Suppose the target pt−Δtp_t- t is α-strongly log-concave with L-Lipschitz score. Let ode x_ode be the result of one backward Euler step to time t−Δt- t (Eq. 5), and let ε:=W2(law(ode),pt−Δt)>0 :=W_2 (law( x_ode),\,p_t- t )>0 be its error. If the Langevin step size satisfies ϵt≤α/L2 _t≤α/L^2 and ϵt≲α2ε2/(L2d) _t α^2 ^2/(L^2d), then one Langevin correction step strictly reduces the error: W2(law(lc),pt−Δt)<W2(law(ode),pt−Δt),W_2 (law( x_lc),\,p_t- t )\;<\;W_2 (law( x_ode),\,p_t- t ), where lc x_lc is the Langevin-corrected sample (Eq. 7) and W2W_2 is the Wasserstein distance. In flow-based GRPO, increasing the noise level, and thus the extent of policy exploration, typically improves reward optimization (Liu et al., 2026a) but degrades the sample quality of rollouts. Preserving quality at a fixed randomness level is therefore important. Below we show that when our Langevin-corrected sampler is matched to the Euler–Maruyama SDE at the same randomness level, that is, comparing Eq. 6 and Eq. 7 and setting the Langevin step size so that 2ϵt=σtΔt 2 _t= _t t, i.e., ϵt=σt2Δt/2 _t= _t^2 t/2, Langevin correction sampling is provably more accurate than the SDE. The formal statement and its proof are given in Appx. A and Appx. A.2, respectively. Theorem 2 (Informal). Under mild regularity (Assumption 2) and with the step size matched to the noise level, ϵt=σt2Δt/2 _t= _t^2 t/2, both the corrector lc x_lc and the reverse-SDE step sde x_sde (Eq. 6) have leading-order Wasserstein error of order Δt2 t^2 (Eq. 12). If the comparison condition Eq. 13 holds, the Langevin correction sample step is strictly more accurate: W2(law(lc),pt−Δt)<W2(law(sde),pt−Δt),W_2 (law( x_lc),\,p_t- t )\;<\;W_2 (law( x_sde),\,p_t- t ), for all sufficiently small Δt t. Beyond the above theoretical result, Fig. 2 provides empirical verification. Note that at the same noise level η, the sum of the variance of the Gaussian noise over all timesteps is actually larger for Langevin than for the SDE, implying at least as much exploration; nevertheless, our Langevin-corrected sampling produces substantially clearer samples than standard SDE-based sampling. 5 Experiments In this section, we empirically evaluate LC-GRPO on text-to-image and text-to-video generation. 5.1 Experimental Setup Image Generation. We experiment with two flow-based text-to-image models, SD3.5-Medium (Esser et al., 2024) and FLUX.1-Dev (Labs, 2024), both at 512×512512× 512 resolution. These models cover two guidance regimes: SD3.5 uses classifier-free guidance (CFG), while FLUX.1-Dev is guidance-distilled and uses no CFG at inference. We consider three reward-optimization settings: 1. Visual text rendering (OCR). This is a verifiable reward that measures how accurately the model renders a target string. We follow Flow-GRPO (Liu et al., 2026a) for both the reward-assignment strategy and the training and evaluation datasets. 2. Human preference alignment. We use HPS-v2.1 (Wu et al., 2023) as the reward model, a preference model trained on human comparisons to predict which generated image humans prefer for a given prompt. We use the training and evaluation prompts from Pick-a-Pic (Kirstain et al., 2023). 3. Multi-reward optimization. To preserve text-image alignment while improving human preference, we jointly optimize CLIP score (Hessel et al., 2021) and HPS-v2.1. We use the same training and evaluation prompts as in the single-reward HPS-v2.1 setting. Video Generation. For text-to-video generation, we experiment with HunyuanVideo (Kong et al., 2024). Following Xue et al. (2025), we use VideoAlign (Liu et al., 2026b) as the reward model and optimize its visual aesthetic quality component. We also report its motion quality and text–video alignment components for evaluation. As in DanceGRPO, training prompts are curated from VidProM (Wang & Yang, 2024); the evaluation dataset is from Zheng et al. (2026). Quality Evaluation Metrics. When optimizing a target reward, we monitor quality metrics to detect reward-induced degradation or reward hacking. For images, we report two quality metrics, Aesthetic Score and CLIP score, and three human-preference metrics, ImageReward (Xu et al., 2023), HPS-v2.1, and PickScore (Kirstain et al., 2023). All image metrics are computed on DrawBench (Saharia et al., 2022). For videos, we report VBench (Huang et al., 2024) as quality metrics. Table 1: Evaluation Results on Image Generation. Task metric columns denote the reward(s) optimized in each setting. Bold is the best result. LC-GRPO consistently outperforms all baselines. Model Task Metric Quality Metric OCR HPS-v2.1 CLIP Aesthetic CLIP ImgRwd HPS-v2.1 PickScore !35 SD3.5-M SD3.5-M 0.569 0.295 0.288 5.38 0.283 0.83 0.279 22.35 (a) Visual Text Rendering: OCR Flow-GRPO 0.914 — — 5.31 0.288 0.94 0.281 22.43 CPS 0.935 — — 5.20 0.287 0.75 0.265 22.16 LC-GRPO 0.960 — — 5.33 0.291 1.00 0.280 22.46 (b) Human Preference Alignment: HPS-v2.1 Flow-GRPO — 0.381 — 6.30 0.267 1.41 0.357 22.89 CPS — 0.371 — 6.12 0.270 1.38 0.344 22.62 LC-GRPO — 0.393 — 6.11 0.280 1.44 0.367 23.03 (c) Multi-Reward: HPS-v2.1 ++ CLIP Score Flow-GRPO — 0.351 0.296 5.97 0.289 1.28 0.327 22.76 CPS — 0.345 0.291 5.73 0.285 1.31 0.323 22.68 LC-GRPO — 0.356 0.302 5.73 0.297 1.33 0.332 22.93 !35 FLUX.1-Dev FLUX.1-Dev — 0.303 0.276 5.68 0.274 0.92 0.293 22.74 (a) Human Preference Alignment: HPS-v2.1 Flow-GRPO — 0.378 — 6.07 0.265 1.36 0.352 22.99 CPS — 0.373 — 6.03 0.266 1.34 0.350 23.04 LC-GRPO — 0.384 — 6.01 0.270 1.36 0.356 23.28 (b) Multi-Reward: HPS-v2.1 ++ CLIP Score Flow-GRPO — 0.371 0.279 5.94 0.273 1.37 0.347 23.18 CPS — 0.370 0.284 5.90 0.281 1.44 0.350 23.28 LC-GRPO — 0.376 0.285 5.88 0.284 1.43 0.352 23.40 Baselines. We compare LC-GRPO against online reinforcement learning baselines based on GRPO. These methods share the same GRPO objective and differ primarily in how rollouts are sampled. The first baseline uses the standard SDE sampler in Eq. 6: Flow-GRPO (Liu et al., 2026a) for text-to-image generation and DanceGRPO (Xue et al., 2025) for text-to-video generation. The second baseline, CPS (Wang & Yu, 2025), instead samples rollouts with DDIM (Song et al., 2020a). Fair Comparison on Computation. Since the compared methods mainly differ in rollout sampling, we match or favor the baselines in rollout computation. Each Langevin-corrected sampler step requires two NFEs; therefore, for each baseline, we run both the same number of rollout steps and twice that number, and report the better result. For SD3.5-M, FLUX.1-Dev, and HunyuanVideo, LC-GRPO uses 10, 6, and 8 rollout steps during training, while the baselines use 10/20, 6/12, and 16 steps, respectively. For evaluation, all methods use the same sampler: 40-step ODE Euler for SD3.5-M, 28 steps for FLUX.1-Dev, and 50 steps for HunyuanVideo. Thus, the proposed method does not use more computation than the baselines. Other hyperparameters are kept nearly identical, with details in Appendix B. 5.2 Main Results As shown in Tables 5.1 and 2, LC-GRPO consistently outperforms the baselines on both image and video generation tasks while preserving sample quality across auxiliary metrics. Qualitative results are shown in Fig. 5.1, with additional examples provided in Appendix C. Figure 3: Qualitative Comparison. Left: SD3.5-M-based models trained with OCR reward; Right: FLUX.1-Dev-based models trained with HPS-v2.1 and CLIP score. Prompts are taken from OCR and DrawBench, respectively. Gap between Training and Evaluation. We examine the gap between rewards on RL training rollouts and those obtained during evaluation inference. Figure 1 compares training rewards, computed on rollouts, with evaluation rewards, computed with ODE Euler sampling on the evaluation dataset throughout training. Flow-GRPO tends to produce blurry, lower-quality rollouts, so its training rewards are often lower than those of cleaner evaluation samples. In contrast, CPS improves rollout quality but leads to a large train–evaluation gap, with high training rewards dropping during inference. LC-GRPO also improves rollout quality while maintaining a smaller train–test gap. Table 2: Evaluation Results on Video Generation. HunyuanVideo is trained with the Visual Quality reward from VideoAlign. Bold indicates the best result. Model VideoAlign VBench Visual Quality Motion Quality Text Alignment Total Score Quality Score Semantic Score Dynamic Degree Aesthetic Quality HunyuanVideo -0.368 - 0.351 1.247 78.74 80.94 69.96 52.8 61.28 DanceGRPO -0.205 -0.279 1.312 78.85 81.18 69.51 12.5 64.03 CPS -0.276 -0.178 1.048 76.86 79.70 65.49 19.4 59.02 LC-GRPO 0.063 -0.223 1.151 79.10 81.92 67.83 45.8 64.70 5.3 Ablation Studies In this section, we analyze the design choices of the Langevin correction step: (a) Gradients through Langevin Correction. Since the Langevin correction step is not used at test time, we study whether gradients should be propagated through it during training. As shown in Fig. 4, backpropagating through the Langevin step leads to better performance and more stable training. We therefore keep this gradient path in all experiments. (b) Classifier-Free Guidance. Classifier-free guidance (CFG) is commonly used to improve sample quality, raising the question of whether it should also be applied in the Langevin correction step. However, the main role of Langevin correction is to introduce stochasticity and define a sampling distribution, rather than directly steering samples toward higher visual quality. We therefore ablate this design choice. As shown in Fig. 4(b), applying Langevin correction without CFG achieves better performance on SD3.5-M while also reducing computation. (a) FLUX.1-Dev: gradients in Langevin step (b) SD3.5-M: gradients and CFG in Langevin step Figure 4: Effect of design choices in the Langevin correction step. 6 Conclusion In flow-based reinforcement learning, the need for stochasticity in the action policy often motivates the use of SDE discretization during training rollouts, which in turn creates a mismatch with the ODE-based inference procedure. To mitigate this issue, we propose LC-GRPO, which combines inference-aligned ODE transport with a distribution-preserving Langevin correction to produce rollouts that are accurate, stochastic, and likelihood tractable for reinforcement learning. Provably, this sampling method is more accurate than SDE sampling while preserving the same level of noise, thereby maintaining the extent of exploration in RL. Experiments on text-to-image and text-to-video alignment benchmarks show that our method consistently improves rollout quality and reward alignment, validating its effectiveness in practice. References Ambrosio et al. (2005) Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer, 2005. Black et al. (2024) Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, volume 2024, p. 4965–4987, 2024. Chung et al. (2022) Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687, 2022. Clark et al. (2024) Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations, volume 2024, p. 4793–4822, 2024. Deng et al. (2026) Haoyou Deng, Keyu Yan, Chaojie Mao, Xiang Wang, Yu Liu, Changxin Gao, and Nong Sang. Densegrpo: From sparse to dense reward for flow matching model alignment. arXiv preprint arXiv:2601.20218, 2026. Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. Ding & Ye (2025) Zheng Ding and Weirui Ye. Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models. arXiv preprint arXiv:2512.08153, 2025. Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. Efron (2011) Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011. Esser et al. (2024) Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024. Fan et al. (2023) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858–79885, 2023. Guo et al. (2024) Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective. arXiv preprint arXiv:2404.14743, 2024. Guo et al. (2026) Yingqing Guo, Yukang Yang, Hui Yuan, and Mengdi Wang. Training-free guidance beyond differentiability: Scalable path steering with tree search in diffusion and flow models. Advances in Neural Information Processing Systems, 38:73343–73384, 2026. Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528, 2021. Huang et al. (2024) Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. Kirstain et al. (2023) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023. Kloeden et al. (2012) Peter Eris Kloeden, Eckhard Platen, and Henri Schurz. Numerical solution of SDE through computer experiments. Springer Science & Business Media, 2012. Kong et al. (2024) Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. Labs (2024) Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024. Lee et al. (2023) Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. Li et al. (2025) Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802, 2025. Liang et al. (2026) Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, et al. The mirage of optimizing training policies: Monotonic inference policies as the real objective for llm reinforcement learning. arXiv preprint arXiv:2606.29526, 2026. Lipman et al. (2022) Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. Liu et al. (2025) Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying rl collapse from the training-inference mismatch. Notion Blog, 2025. Liu et al. (2026a) Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. Advances in neural information processing systems, 38:40783–40818, 2026a. Liu et al. (2026b) Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, et al. Improving video generation with human feedback. Advances in Neural Information Processing Systems, 38:82155–82192, 2026b. Liu et al. (2022) Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. Otto (2001) Felix Otto. The geometry of dissipative evolution equations: the porous medium equation. 2001. Prabhudesai et al. (2023) Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. 2023. Qi et al. (2025) Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via fp16. arXiv preprint arXiv:2510.26788, 2025. Qiu et al. (2026) Zhaopeng Qiu, Shuang Yu, Jingqi Zhang, Shuai Zhang, Xue Huang, Jingyi Yang, and Junjie Lai. Fp8-rl: A practical and stable low-precision stack for llm reinforcement learning. arXiv preprint arXiv:2601.18150, 2026. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022. Savani et al. (2026) Yash Savani, Branislav Kveton, Yuchen Liu, Yilin Wang, Jing Shi, Subhojyoti Mukherjee, Nikos Vlassis, and Krishna Kumar Singh. Stepwise credit assignment for grpo on flow-matching models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 42007–42017, 2026. Song et al. (2020a) Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a. Song et al. (2020b) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b. Wallace et al. (2024) Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8228–8238, 2024. Wan et al. (2025) Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314, 2025. Wang & Yu (2025) Feng Wang and Zihao Yu. Coefficients-preserving sampling for reinforcement learning with flow matching. arXiv preprint arXiv:2509.05952, 2025. Wang & Yang (2024) Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models. Advances in Neural Information Processing Systems, 37:65618–65642, 2024. (41) Bram Wasti, Wentao Ye, Teja Rao, Michael Goin, et al. No more train-inference mismatch: Bitwise consistent on-policy reinforcement learning with vllm and torchtitan. 2025. URl: https://blog. vllm. ai/2025/11/10/bitwise-consistent-train-inference. html. Wu et al. (2025) Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02324, 2025. Wu et al. (2023) Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 2096–2105, 2023. Xu et al. (2023) Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023. Xue et al. (2025) Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818, 2025. Yang et al. (2024) Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8941–8951, 2024. Yao et al. (2025) Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, august 2025. URL https://fengyao. notion. site/off-policy-rl, 2025. Yuan et al. (2023) Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems, 36:60599–60635, 2023. Zheng et al. (2025) Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117, 2025. Zheng et al. (2026) Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, et al. Manifold-aware exploration for reinforcement learning in video generation. arXiv preprint arXiv:2603.21872, 2026. Appendix A Theoretical Analysis We focus on a single denoising step of the flow model, running from time t to r:=t−hr:=t-h with step size h>0h>0. The probability flow ODE is du=u(u)du,d x_u= v_u ( x_u )\,du, (8) with marginals pup_u for u∈[r,t]u∈[r,t]. Under this assumption, we show that a single Langevin step provably improves upon the Euler step. This result is stated formally as Theorem 3, with the proof given in Sec. A.1. Assumption 1. Let f:=logprf:= p_r. Then −LId⪯∇2f⪯−αId-LI_d ∇^2f -α I_d for some 0<α≤L0<α≤ L; that is, f is α-strongly concave and ∇f∇ f is L-Lipschitz. Theorem 3 (Langevin Step Improves the Euler Step). Let Assumption 1 hold. Starting from ∼qt x q_t at time t, take one backward Euler step of the probability flow ODE Eq. 8 with step size h>0h>0, ode=−ht(), x_ode= x-h\, v_t( x), and denote the resulting error by ε:=W2(law(ode),pr) :=W_2 (law( x_ode),\,p_r ), assumed positive. Then follow with one Langevin correction (LC) step targeting prp_r, lc=ode+ϵ∇logpr(ode)+2ϵξ,ξ∼(0,Id),ξ⟂ode. x_lc= x_ode+ε\,∇ p_r( x_ode)+ 2ε\,ξ, ξ (0,I_d), ξ x_ode. If the step size satisfies ϵ≤αL2andϵ<9100⋅α2ε2L2d,ε\;≤\; αL^2 ε\;<\; 9100· α^2 ^2L^2d, (9) then the Langevin step strictly improves upon the Euler step: W2(law(lc),pr)<W2(law(ode),pr).W_2 (law( x_lc),\,p_r )\;<\;W_2 (law( x_ode),\,p_r ). Theorem 3 shows that the corrector improves on the deterministic Euler step. We now compare it with the standard stochastic alternative: one Euler–Maruyama step of the reverse SDE. Denote u:=∇logpu s_u:=∇ p_u for the score. Given ∼pt x p_t and ξ∼(0,Id)ξ (0,I_d), the SDE step at noise level σ>0σ>0 is sde=−ht()+σ2h2t()+σhξ, x_sde= x-h\, v_t( x)+ σ^2h2\, s_t( x)+σ h\,ξ, (10) while the Langevin correction step is exactly that of Theorem 3. We further define vector fields at time t: ˙:=∂t+∇tt,˙:=∂t+∇tt,:=∇(12‖t‖2+Δlogpt), v:= _t v_t+∇ v_t\, v_t, s:= _t s_t+∇ s_t\, v_t, G:=∇ ( 12 \| s_t \|^2+ p_t ), (11) Here, ˙ v and ˙ s are the rates of change of the velocity and score along the flow, while G is the direction of the bias introduced by a single Langevin step. Under the following regularity assumption, we can compute the exact leading-order W2W_2 error of both the Langevin correction and the SDE step, and show that the Langevin correction step is more accurate under an explicit condition (Theorem 4); see Sec. A.2 for the proof. Assumption 2. Let ,logp∈C2,4([r,t]×ℝd) v, p∈ C^2,4 ([r,t]×R^d ), with all derivatives of polynomial growth, and ptp_t has finite moments of all orders. Theorem 4 (Comparison with the SDE Step). Let Assumption 2 hold and ϵ≤Λhε≤ h for a fixed constant Λ>0 >0. With ˙,˙, v, s, G as defined in Eq. 11, we have W2(law(lc),pr) W_2 (law( x_lc),\,p_r ) =h22‖˙−(ϵh)2‖pt+o(h2), = h^22\, \| v- ( εh )^2 G \|_p_t+o(h^2), (12) W2(law(sde),pr) W_2 (law( x_sde),\,p_r ) =h22‖˙−σ2˙−σ44‖pt+o(h2). = h^22\, \| v-σ^2 s- σ^44 G \|_p_t+o(h^2). In particular, at ϵ=σ2h/2ε=σ^2h/2 matching the noise level, let :=˙−σ44 E:= v- σ^44 G, if the following holds: ⟨,˙⟩pt<σ22∥˙∥pt2, E, s _p_t< σ^22 \| s \|_p_t^2, (13) Langevin correction step is strictly more accurate: W2(law(lc),pr)<W2(law(sde),pr),W_2 (law( x_lc),\,p_r )\;\ <\;W_2 (law( x_sde),\,p_r ), for all sufficiently small h. A.1 Proof of Theorem 3 Proof. Applying Lemma 1 at time r with initial law q=qrq=q_r gives the upper bound: W2(law(lc),pr)≤(1−αϵ2)ε+53Ldϵ3/2.W_2 (law( x_lc),\,p_r )\;≤\; (1- αε2 )\, + 53\,L d\,ε^3/2. The second condition in Eq. 9 is equivalent to ϵ<3αε10Ld ε< 3α 10\,L d, hence 53Ldϵ3/2<53Ld⋅3αε10Ld⋅ϵ=αϵ2ε, 53\,L d\,ε^3/2\;<\; 53\,L d· 3α 10\,L d·ε\;=\; αε2\, , so the discretization error is strictly dominated by the contraction gain, and W2(law(lc),pr)<(1−αϵ2)ε+αϵ2ε=ε.W_2 (law( x_lc),\,p_r )\;<\; (1- αε2 )\, + αε2\, \;=\; . Therefore, we complete the proof. ∎ Lemma 1 (Langevin Corrector). Let Assumption 1 hold and given ∼q z q, langevin correction step: +=+ϵ∇logpr()+2ϵξ,ξ∼(0,Id) z^+= z+ε∇ p_r( z)+ 2ε\,ξ, ξ (0,I_d) (14) with ϵ≤α/L2ε\;≤\;α/L^2. Then, W2(law(+),pr)≤(1−αϵ2)W2(q,pr)+53Ldϵ3/2.W_2 (law( z^+),\,p_r )\;≤\; (1- αε2 )\,W_2(q,\,p_r)\;+\; 53\,L d\;ε^3/2. (15) Proof. Denote f:=logprf:= p_r, so that the corrector step reads +=+ϵ∇f()+2ϵξ z^+= z+ε∇ f( z)+ 2ε\,ξ. Let (,0)( z, w_0) be an optimal W2W_2 coupling of (q,pr)(q,p_r), and let (Bτ)τ≥0(B_τ)_τ≥ 0 be a standard Brownian motion independent of (,0)( z, w_0). Since ∇f∇ f is L-Lipschitz by Assumption 1, the Langevin dynamics dτ=∇f(τ)dτ+2dBτ,τ∈[0,ϵ],0∼pr,d w_τ=∇ f( w_τ)\,dτ+ 2\,dB_τ, τ∈[0,ε], w_0 p_r, admits a unique strong solution with pr∝efp_r e^f as its stationary distribution, so ϵ∼pr w_ε p_r. Let ξ:=Bϵ/ϵ∼(0,Id)ξ:=B_ε/ ε (0,I_d), which is independent of z. Therefore, we have (+,ϵ)( z^+, w_ε) a coupling of law(+)law( z^+) and prp_r, and thus W2(law(+),pr)≤‖+−ϵ‖L2.W_2 (law( z^+),\,p_r )≤ \| z^+- w_ε \|_L^2. We first decompose the difference +−ϵ z^+- w_ε. Writing ϵ=0+∫0ϵ∇f(τ)dτ+2Bϵ w_ε= w_0+ _0^ε∇ f( w_τ)\,dτ+ 2\,B_ε and subtracting it from the corrector step, the Brownian terms cancel and +−ϵ=(−0)+ϵ(∇f()−∇f(0))⏟=:A+∫0ϵ(∇f(0)−∇f(τ))dτ⏟=:B, z^+- w_ε= ( z- w_0 )+ε (∇ f( z)-∇ f( w_0) )_=:\,A+ _0^ε (∇ f( w_0)-∇ f( w_τ) )\,dτ_=:\,B, where A compares the two drifts at the initial points, and B is the local discretization error. Next, we show that A contracts. Expanding the square and using ⟨∇f()−∇f(0),−0⟩≤−α‖−0‖2 ∇ f( z)-∇ f( w_0),\, z- w_0 ≤-α \| z- w_0 \|^2 together with ‖∇f()−∇f(0)‖≤L‖−0‖ \|∇ f( z)-∇ f( w_0) \|≤ L \| z- w_0 \|, where the strong log-concavity and the Lipschitz property of ∇f∇ f are from Assumption 1, we have ‖A‖2≤(1−2αϵ+L2ϵ2)‖−0‖2≤(1−αϵ)‖−0‖2, \|A \|^2≤ (1-2αε+L^2ε^2 ) \| z- w_0 \|^2≤(1-αε) \| z- w_0 \|^2, where the last inequality uses L2ϵ≤αL^2ε≤α; note also αϵ≤α2/L2≤1αε≤α^2/L^2≤ 1, so the right-hand side is nonnegative. Since 1−x≤1−x/2 1-x≤ 1-x/2 for x∈[0,1]x∈[0,1] and the coupling (,0)( z, w_0) is optimal, it holds ‖A‖L2≤(1−αϵ2)W2(q,pr). \|A \|_L^2≤ (1- αε2 )\,W_2(q,\,p_r). Then, we bound the local error B. Since Langevin dynamics is stationary, r∼pr w_r p_r for every r. Using (a+b)2≤2a2+2b2(a+b)^2≤ 2a^2+2b^2, the Cauchy–Schwarz inequality on the drift integral, Lemma 3, and ‖2Bτ‖2=2τdE\, \| 2\,B_τ \|^2=2τ d, we obtain ‖τ−0‖2≤2τ∫0τ‖∇f(r)‖2dr+4τd≤2Ldτ2+4dτ.E\, \| w_τ- w_0 \|^2≤ 2τ _0^τE\, \|∇ f( w_r) \|^2\,dr+4τ d≤ 2Ld\,τ^2+4d\,τ. By the Cauchy–Schwarz inequality on the time integral and the Lipschitz property of ∇f∇ f, ‖B‖2≤ϵ∫0ϵ‖∇f(0)−∇f(τ)‖2dτ≤ϵL2∫0ϵ‖τ−0‖2dτ≤ϵL2(2Ldϵ33+2dϵ2).E\, \|B \|^2≤ε _0^εE\, \|∇ f( w_0)-∇ f( w_τ) \|^2\,dτ≤ε L^2 _0^εE\, \| w_τ- w_0 \|^2\,dτ≤ε L^2 ( 2Ld\,ε^33+2d\,ε^2 ). Since ϵ≤α/L2≤1/Lε≤α/L^2≤ 1/L, we have Lϵ≤1Lε≤ 1 and hence ‖B‖2≤83L2dϵ3,‖B‖L2≤83Ldϵ3/2≤53Ldϵ3/2.E\, \|B \|^2≤ 83\,L^2d\,ε^3, \|B \|_L^2≤ 83\,L d\,ε^3/2≤ 53\,L d\,ε^3/2. Finally, combining the above bounds yields W2(law(+),pr)≤‖A‖L2+‖B‖L2≤(1−αϵ2)W2(q,pr)+53Ldϵ3/2.∎W_2 (law( z^+),\,p_r )≤ \|A \|_L^2+ \|B \|_L^2≤ (1- αε2 )\,W_2(q,\,p_r)+ 53\,L d\;ε^3/2. A.2 Proof of Theorem 4 Proof. For a law q and a smooth test function f, define δ(f;q):=∫fdq−∫fdprδ(f;q):= f\,dq- f\,dp_r. For brevity, we write δlc(f):=δ(f;law(lc)) _lc(f):=δ(f;law( x_lc)) and δsde(f):=δ(f;law(sde)) _sde(f):=δ(f;law( x_sde)). We first expand the Langevin correction step weakly. Let fϵ():=ξf(+ϵr()+2ϵξ)f_ε( y):=E_ξ\,f ( y+ε\, s_r( y)+ 2ε\,ξ ) denote the expected value of f after one corrector step from y, so that f(lc)=ptfϵ(ode)E\,f( x_lc)=E_p_tf_ε( x_ode) and δlc(f)=ptfϵ(ode)−prfϵ⏟predictor error+prfϵ−prf⏟corrector bias. _lc(f)= E_p_tf_ε( x_ode)-E_p_rf_ε_predictor error\;+\; E_p_rf_ε-E_p_rf_corrector bias. (16) For the predictor error, let Φ:=Φt→r := _t→ r denote the exact flow map, which satisfies Φ#pt=pr \#p_t=p_r. Since the trajectory u↦(u)u x(u) has acceleration exactly ˙ v, a Taylor expansion gives Φ()=−ht()+h22˙()+O(h3) ( x)= x-h\, v_t( x)+ h^22\, v( x)+O(h^3). Hence ode−Φ()=−h22˙()+O(h3) x_ode- ( x)=- h^22\, v( x)+O(h^3), and for any test function ψ, ptψ(ode)−prψ=pt[ψ(ode)−ψ(Φ())]=−h22pt⟨˙,∇ψ⟩+O(h3).E_p_t\,ψ( x_ode)-E_p_r\,ψ=E_p_t [ψ( x_ode)-ψ( ( x)) ]=- h^22\,E_p_t v,\,∇ψ +O(h^3). We apply this with ψ=fϵψ=f_ε; Taylor-expanding in ϵε gives ∇fϵ=∇f+O(ϵ)∇ f_ε=∇ f+O(ε), and since ϵ≤Λhε≤ h, the substitution costs only O(h3)O(h^3). The corrector bias is that of one Langevin step started from its own target prp_r. Write :=r s:= s_r and let Lg:=⟨,∇g⟩+ΔgLg:= s,∇ g + g be the Langevin generator; Due to integration by parts, and recalling =∇logpr s=∇ p_r, we get invariance of prp_r, which means prLg=0E_p_rLg=0 for all g. Taylor expanding f in the increment ϵ+2ϵξε\, s+ 2ε\,ξ and taking Gaussian moments, prfϵ−prf=ϵpr[Lf]+ϵ2pr[12⊤∇2f+⟨,∇Δf⟩+12Δ2f]+O(ϵ3).E_p_rf_ε-E_p_rf=ε\,E_p_r [Lf ]+ε^2\,E_p_r [ 12 s ∇^2f\, s+ s,∇ f + 12 ^2f ]+O(ε^3). The order-ϵε term vanishes by invariance. For the order-ϵ2ε^2 term, call the bracket B. By invariance with g=Lfg=Lf, yielding pr[12L2f]=0E_p_r[ 12L^2f]=0, we can subtract it for free. Expanding 12L2f 12L^2f and cancelling the common terms leaves only B−12L2f=−⟨∇,∇2f⟩−12⟨(⋅∇)+Δ,∇f⟩,B- 12L^2f=- ∇ s,\,∇^2f - 12 ( s\!·\!∇) s+ s,\,∇ f , i.e. B differs from 12L2f 12L^2f by the Hessian term ⟨∇,∇2f⟩ ∇ s,∇^2f plus a first-order piece. One integration by parts, using ∇pr=pr∇ p_r=p_r\, s, removes the Hessian: pr⟨∇,∇2f⟩=−pr⟨(⋅∇)+Δ,∇f⟩,E_p_r ∇ s,∇^2f =-\,E_p_r ( s\!·\!∇) s+ s,\,∇ f , so the two first-order pieces combine into 12pr⟨(⋅∇)+Δ,∇f⟩ 12\,E_p_r ( s\!·\!∇) s+ s,∇ f . Because =∇logpr s=∇ p_r is a gradient, (⋅∇)=∇(12‖2)( s\!·\!∇) s=∇( 12\| s\|^2) and Δ=∇Δlogpr s=∇ p_r, so this field is itself a gradient. Hence prfϵ−prf=ϵ22pr⟨∇(12‖2+Δlogpr),∇f⟩+O(ϵ3)=ϵ22pt⟨,∇f⟩+O(h3),E_p_rf_ε-E_p_rf= ε^22\,E_p_r ∇ ( 12 \| s \|^2+ p_r ),\,∇ f +O(ε^3)= ε^22\,E_p_t G,\,∇ f +O(h^3), where moving from time r to time t costs O(ϵ2h)=O(h3)O(ε^2h)=O(h^3). Combining the two parts, δlc(f)=−h22pt⟨˙−(ϵh)2,∇f⟩+O(h3). _lc(f)=- h^22\,E_p_t v- ( εh )^2 G,\;∇ f +O(h^3). (17) Next, we show the SDE step adds exactly one term. Let ~ x be the Langevin step run with ϵ=σ2h/2ε=σ^2h/2 and the same noise ξ. Then 2ϵ=σh 2ε=σ h, so the two schemes share their noise and differ only in where the score is evaluated: sde−~=σ2h2[t()−r(ode)]=σ2h22˙()+O(h3), x_sde- x= σ^2h2\, [ s_t( x)- s_r( x_ode) ]= σ^2h^22\, s( x)+O(h^3), (18) by first-order Taylor expansion of (u,)↦u()(u, y) s_u( y) at (t,)(t, x) in the direction (−h,−ht()) (-h,\,-h\, v_t( x) ), which produces exactly ˙ s. This difference Eq. 18 is a function of x alone, so expanding f between the two points, the cross terms with the noise vanish in expectation (ξ is centered and independent of x), and δsde(f)=δlc(f)|ϵ=σ2h/2+σ2h22pt⟨˙,∇f⟩=−h22pt⟨˙−σ2˙−σ44,∇f⟩+O(h3). _sde(f)= _lc(f) |_ε=σ^2h/2+ σ^2h^22\,E_p_t s,∇ f =- h^22\,E_p_t v-σ^2 s- σ^44 G,\;∇ f +O(h^3). (19) Finally, applying Lemma 4 to Eq. 17 with =˙−(ϵh)2 A= v-( εh)^2 G, and to Eq. 19 with =˙−σ2˙−σ44 A= v-σ^2 s- σ^44 G. In both cases, A is a gradient field: t v_t and t=∇logpt s_t=∇ p_t are gradients (hence so are ˙ v and ˙ s), and G is a gradient by construction. This yields Eq. 12. For the matched step ϵ=σ2h/2ε=σ^2h/2 we have (ϵ/h)2=σ4/4(ε/h)^2=σ^4/4, so by Eq. 12 the two leading coefficients are h22‖p h^22 \| E \|_p and h22‖−σ2˙‖p h^22 \| E-σ^2 s \|_p with :=˙−σ44 E:= v- σ^44 G. Both norms are nonnegative, so comparing them is equivalent to comparing their squares; expanding, ‖p2<‖−σ2˙‖p2=‖p2−2σ2⟨,˙⟩p+σ4‖˙‖p2⟺⟨,˙⟩p<σ22∥˙∥p2. \| E \|_p^2\;<\; \| E-σ^2 s \|_p^2= \| E \|_p^2-2σ^2 E,\, s _p+σ^4 \| s \|_p^2 E,\, s _p\;<\; σ^22\, \| s \|_p^2. Hence the assumed condition makes the leading coefficient strictly of Langevin step smaller, and since both errors enter at order h2h^2, W2(law(lc),pr)<W2(law(sde),pr)W_2 (law( x_lc),\,p_r )<W_2 (law( x_sde),\,p_r ) for all small h. ∎ A.3 Auxiliary Lemmas Assumption 3. For all u∈[r,t]u∈[r,t], the velocity u v_u is L-Lipschitz. Assumption 4. For all u∈[r,t]u∈[r,t] and ∈ℝd x ^d, ‖∂u()+∇u()u()‖≤K \| _u v_u( x)+∇ v_u( x)\, v_u( x) \|≤ K; that is, trajectories of Eq. 8 have acceleration bounded by K. Lemma 2 (One-Step Euler Error for ODEs). Under Assumptions 3–4, let ∼qt x q_t and define the backward Euler step ^=−ht() x= x-h v_t ( x ), it holds W2(law(^),pt−h)≤eLhW2(qt,pt)+K2h2.W_2 (law( x),\,p_t-h )\;≤\;e^Lh\,W_2(q_t,p_t)+ K2h^2. (20) Proof. Let Φ:=Φt→t−h := _t→ t-h denote the exact flow map of du=u(u)dud x_u= v_u ( x_u )du from time t to time t−ht-h. We first show that Φ is eLhe^Lh-Lipschitz. Let u x_u and ~u x_u be two trajectories of the ODE, and define g(u):=‖u−~u‖2g(u):= \| x_u- x_u \|^2. Then, g′(u) g (u) =2⟨u−~u,u(u)−u(~u)⟩≥−2‖u−~u‖⋅‖u(u)−u(~u)‖ =2 x_u- x_u,\, v_u( x_u)- v_u( x_u) ≥-2 \| x_u- x_u \|· \| v_u( x_u)- v_u( x_u) \| ≥−2L‖u−~u‖2=−2Lg(u), ≥-2L \| x_u- x_u \|^2=-2Lg(u), where Cauchy–Schwarz inequality and the L-Lipschitz of velocity u v_u from Assumption 3. Hence du(e2Lug(u))≥0 ddu (e^2Lug(u) )≥ 0. Therefore, we have e2L(t−h)g(t−h)≤e2Ltg(t)e^2L(t-h)g(t-h)≤ e^2Ltg(t). Equivalently, ∥Φ(x_t)-Φ(~x_t)∥ ≤e^Lh∥x_t-~x_t∥. Next, we compare the exact flow with the Euler step map. By Assumption 4, the map u↦u(u)u v_u( x_u) is C1C^1 with derivative ∂u(u)+∇u(u)u(u) _u v_u( x_u)+∇ v_u( x_u)\, v_u( x_u), bounded by K. Hence ‖u(u)−t(t)‖≤K(t−u) \| v_u( x_u)- v_t( x_t) \|≤ K(t-u) for u∈[t−h,t]u∈[t-h,t], and since Φ()=t−∫t−htu(u)du ( x)= x_t- _t-h^t v_u( x_u)\,du, ‖Φ()−^‖=‖∫t−ht(u(u)−t(t))du‖≤∫t−htK(t−u)du=K2h2. \| ( x)- x \|= \| _t-h^t ( v_u( x_u)- v_t( x_t) )\,du \|≤ _t-h^tK(t-u)\,du= K2h^2. Let ∼pt y p_t, thus we have Φ()∼pt−h ( y) p_t-h. By the triangle inequality for Wasserstein distance, it holds W2(law(^),pt−h)≤‖^−Φ()‖L2≤‖^−Φ()‖L2+‖Φ()−Φ()‖L2≤K2h2+eLhW2(qt,pt).W_2(law( x),p_t-h)≤ \| x- ( y) \|_L^2≤ \| x- ( x) \|_L^2+ \| ( x)- ( y) \|_L^2≤ K2h^2+e^Lh\,W_2(q_t,p_t). ∎ Lemma 3 (Stationary Score Moment). Under Assumption 1, it holds pr[‖∇f‖2]≤Ld.E_p_r [ \|∇ f \|^2 ]\;≤\;Ld. Proof. Since ∇pr=pr∇f∇ p_r=p_r\,∇ f, integration by parts gives ∫‖∇f‖2pr=∫⟨∇f,∇pr⟩=−∫(Δf)pr=pr[tr(−∇2f)]≤Ld, \|∇ f \|^2\,p_r= ∇ f,\,∇ p_r =- ( f)\,p_r=E_p_r [tr(-∇^2f) ]≤ Ld, where the last inequality uses −∇2f⪯LId-∇^2f LI_d from Assumption 1, and the boundary terms vanish because prp_r has sub-Gaussian tails under strong log-concavity while ∇f∇ f grows at most linearly. ∎ Lemma 4. Let p,(μh)h>0∈2(ℝd)p,( _h)_h>0 _2(R^d), and suppose μh→p _h→ p in W2W_2 as h→0h→ 0. Suppose =∇ϕ∈L2(p) A=∇φ∈ L^2(p) is a gradient field satisfying ∫fμh−∫fp=−h22∫⟨,∇f⟩px+o(h2), f\,d _h- f\,dp=- h^22 A,∇ f \,p\,dx+o(h^2), for all smooth test functions f. Then W2(μh,p)=h22‖L2(p)+o(h2)W_2( _h,p)= h^22\,\| A\|_L^2(p)+o(h^2). Proof. Write ∥⋅∥:=∥⋅∥L2(p)\|·\|:=\|·\|_L^2(p) and T:=Id−h22T:=Id- h^22 A. Upper bound. Coupling p with T#pT_\#p through x↦T(x)x T(x), W2(μh,p)≤W2(μh,T#p)+W2(T#p,p)≤o(h2)+h22‖W_2( _h,p)≤ W_2( _h,T_\#p)+W_2(T_\#p,p)≤ o(h^2)+ h^22\| A\|, where the first term is o(h2)o(h^2) because μh _h and T#pT_\#p share the same weak expansion, and the second uses ∫‖x−T(x)‖2p=h44‖2 \|x-T(x)\|^2p= h^44\| A\|^2. Lower bound. Testing the hypothesis with f=−ϕf=-φ gives ∫(−ϕ)μh−∫(−ϕ)p=h22∫‖2px+o(h2)=h22‖2+o(h2). (-φ)\,d _h- (-φ)\,dp= h^22 \| A\|^2\,p\,dx+o(h^2)= h^22\| A\|^2+o(h^2). On the other hand, integrating ∇ϕ∇φ along the W2W_2-geodesic from p to μh _h and applying the first-order (Otto) calculus on (2,W2)(P_2,W_2) (Otto, 2001; Ambrosio et al., 2005), ∫(−ϕ)μh−∫(−ϕ)p≤W2(μh,p)(‖+o(1)), (-φ)\,d _h- (-φ)\,dp≤ W_2( _h,p)\, (\| A\|+o(1) ), where the o(1)o(1) holds because μh→p _h→ p in W2W_2, so the geodesic contracts to p. Combining the two displays, h22‖2+o(h2)≤W2(μh,p)(‖+o(1)), h^22\| A\|^2+o(h^2)≤ W_2( _h,p)\, (\| A\|+o(1) ), whence W2(μh,p)≥h22‖+o(h2)W_2( _h,p)≥ h^22\| A\|+o(h^2). Together with the upper bound, this proves the claim. ∎ Appendix B Experimental Setup Details Except for the noise level η and KL coefficient β, all other hyperparameters are kept the same across methods. The image-generation configuration largely follows Flow-GRPO. For SD3.5-M-based image generation, we use 48 prompts per epoch, a group size of 24 generations per prompt, LoRA with rank r=32r=32 and α=64α=64, a learning rate of 3×10−43× 10^-4, and a CFG scale of 4.5. For multi-reward optimization, each reward is normalized separately before being combined. We use weights of 0.4 for HPS-v2.1 and 0.6 for CLIP score. The noise level is set to η=0.7η=0.7 for Flow-GRPO, η=0.9η=0.9 for CPS, and η=0.8η=0.8 for LC-GRPO in the OCR setting and η=0.7η=0.7 in all other human-preference settings. The KL coefficient β is set to 0 for all human-preference reward experiments. For verifiable rewards such as OCR, we use β>0β>0 to preserve image quality: β=0.04β=0.04 for Flow-GRPO, β=10−4β=10^-4 for CPS, and β=4×10−3β=4× 10^-3 for LC-GRPO. For FLUX.1-Dev-based image generation, we use the same prompt and group sizes as above, set β=0β=0, and use LoRA with rank r=64r=64 and α=128α=128 and a learning rate of 3×10−43× 10^-4. For multi-reward optimization, rewards are normalized separately and combined with weights 0.7 for HPS-v2.1 and 0.3 for CLIP score. We set noise level η=0.9η=0.9 for Flow-GRPO and CPS; for LC-GRPO, η=0.8η=0.8 for HPS and η=0.9η=0.9 for the combined HPS–CLIP setting. For video generation, we use 16 prompts per epoch, a group size of 8 generations per prompt, a resolution of 480×480480× 480, 53 frames at 8 fps, and a learning rate of 10−510^-5. The noise level is set to η=0.25η=0.25 for DanceGRPO, η=0.8η=0.8 for CPS, and η=0.6η=0.6 for LC-GRPO. Appendix C Additional Results Figure 5: Qualitative comparison based on HunyuanVideo trained with the visual quality metric in VideoAlign. Figure 6: Additional qualitative comparison based on SD3.5-M trained with OCR reward. Figure 7: Additional qualitative comparison based on FLUX.1-Dev trained with HPS-v2.1 reward, testing on DrawBench.