Paper deep dive
RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation
Yuhan Li, Fangao Zeng, Sicong Kang, Mengfei Xu, Hao Zhou, Wei Li, Pipei Huang, Bingbing Ni
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We instead take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide a natural source of distillation supervision rather than a disposable byproduct of sampling. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage RL-distillation co-training framework that attaches a decoupled student to an arbitrary RL teacher. The student learns segment-wise from the teacher's evolving rollout trajectories while leaving the original teacher optimization unchanged. To prevent uniform imitation from preserving undesirable low-reward behaviors, we further introduce Advantage-Modulated Distillation (AMD), which transforms rollout advantages into signed weights over a base distillation loss. AMD strengthens supervision from preferred trajectories and mildly repels the student from low-reward ones. The resulting framework is lightweight and plug-and-play, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment show that REST enables few-step CFG-free inference that matches or surpasses its 40-step RL teacher, with an overall additional training cost below 25% over pure RL. REST improves DrawBench PickScore over RTDMD by 0.82 while requiring only one-fifth of the training iterations.
Tags
Links
- Source: https://arxiv.org/abs/2608.09226v1
- Canonical: https://arxiv.org/abs/2608.09226v1
Trouble viewing inline? Open PDF directly →
Full Text
65,771 characters extracted from source content.
Expand or collapse full text
RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation Yuhan Li1,∗ Fangao Zeng2,∗,† Sicong Kang2,∗ Mengfei Xu2 Hao Zhou2 Wenxiang Shang2 Wei Li2 Pipei Huang2,‡ Bingbing Ni1,‡ ∗Equal Contribution †Project Leader ‡Joint Corresponding Authors 1Shanghai Jiao Tong University 2Taobao & Tmall group of Alibaba Abstract Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We instead take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide a natural source of distillation supervision rather than a disposable byproduct of sampling. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage RL-distillation co-training framework that attaches a decoupled student to an arbitrary RL teacher. The student learns segment-wise from the teacher’s evolving rollout trajectories while leaving the original teacher optimization unchanged. To prevent uniform imitation from preserving undesirable low-reward behaviors, we further introduce Advantage-Modulated Distillation (AMD), which transforms rollout advantages into signed weights over a base distillation loss. AMD strengthens supervision from preferred trajectories and mildly repels the student from low-reward ones. The resulting framework is lightweight and plug-and-play, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment show that REST enables few-step CFG-free inference that matches or surpasses its 40-step RL teacher, with an overall additional training cost below 25% over pure RL. REST improves DrawBench PickScore over RTDMD by 0.82 while requiring only one-fifth of the training iterations. 1 Introduction Recent progress in diffusion and flow-matching models (Black-forest-labs, 2024; Wu et al., 2025) has substantially improved the visual quality of text-to-image generation. In real practice, however, two post-training procedures are often required before such models become truly useful (Jiang et al., 2025; Fan et al., 2026): few-step sampling and classifier-free guidance (CFG) (Ho and Salimans, 2022) distillation (Luo et al., 2023; Yin et al., 2024b) for efficient inference, and reinforcement-learning-based alignment for human preference (Shao et al., 2024). These two procedures are traditionally executed sequentially, which is cumbersome and often suboptimal: distillation may wash out the reward gains obtained from RL, while subsequent RL can disturb a previously distilled few-step generator and degrade its structure or low-step inference stability (Jiang et al., 2025). A natural question is whether reward alignment and few-step distillation can be performed in a unified post-training stage. As in Tab. 1, recent works such as DMDR (Jiang et al., 2025) and RTDMD (Huang et al., 2026) make progress in this direction by introducing a warm-up distribution-matching distillation (Yin et al., 2024b) before the RL process, followed by RL+DMD parallel optimization. The multi-stage system often involves balancing a number of stage-specific hyperparameters as well as using more data and training iterations. They also suffer from complicated pipelines compared to RL algorithms, such as an extra fake model, extra rollouts, and alternating training. More importantly, the frozen RL-agnostic teacher distribution used by the distillation objective may pull against the continuously evolving reward-optimized policy, creating an inherent tension between imitation and reward improvement. Figure 1: REST achieves few-step, reward-aligned generation through a simple and efficient training process as in the table above. We also present 8-step CFG-free samples across the PickScore, OCR, and GenEval datasets in the figure below. While RL-trained models have demonstrated preliminary few-step generation capabilities, our approach delivers higher-quality outputs with superior preference alignment. In this work, we take an RL-native perspective: distillation should be built directly into RL training, without a separate pre-distillation stage, external data, or auxiliary models such as fake-score networks. Instead, it should reuse the rollouts, rewards, and intermediate states already produced by RL and exploit RL’s own capacity for step reduction. We observe that diffusion RL already possesses an implicit capacity for step reduction, because the policy is directly optimized through terminal samples generated by a finite-step rollout (Fig. 1). Following this principle, we attach a decoupled student branch to an arbitrary diffusion RL algorithm. The teacher retains its original reward-optimization objective, while the student performs segment-wise imitation along the same reward-scored rollout trajectory without affecting teacher optimization. This turns standard RL sampling into native distillation supervision and improves few-step CFG-free inference. Nevertheless, uniform imitation may still preserve undesirable behaviors from low-reward rollouts, such as structural ambiguity and fine-grained artifacts. To address this limitation, inspired by Xue et al. (2025a), we further reuse the rollout reward in student distillation through Advantage-Modulated Distillation (AMD). AMD applies an affine transformation to the reward advantage of each teacher rollout, yielding a signed modulation coefficient for the base distillation loss. This mechanism induces a contrastive effect analogous to CFG (Ho and Salimans, 2022) and NFT (Chen et al., 2026): high-reward trajectories exert stronger attractive supervision toward desirable teacher behaviors, whereas sufficiently low-reward trajectories provide a mild repulsive gradient that discourages the student from reproducing undesirable ones. Compared with uniform distillation, this signed reward modulation sharpens the few-step student and improves visual clarity. Because AMD only reweights the underlying per-sample objective without changing its target, it also serves as a general reward-aware wrapper for different distillation losses. In summary, we present REST (Reward-Enhanced Scored-Trajectory Distillation), a unified RL-distillation co-training framework for efficient and reward-aligned diffusion generation. To the best of our knowledge, it is the first single-stage framework for RL-distillation collaboration. It has three key advantages. First, it delivers high-quality few-step generation, matching or even surpassing the full-step inference RL-trained teacher: the general AMD reward modulation compensates for the quality loss introduced by pure distillation. Second, it is plug-and-play: the decoupled teacher–student design and the generic AMD formulation make the method independent of a specific teacher RL algorithm or a particular distillation loss, which is present in Sec. 4.4. Third, it is simple and efficient: the framework avoids warm-up scheduling and delicate coordination among multiple competing objectives; it also requires no extra image rollouts, no separate distillation dataset, and no fake-score model training as in DMDR and RTDMD. Our contributions are summarized as follows: • We propose REST, a decoupled teacher–student co-training framework that attaches a lightweight student branch to an arbitrary RL teacher branch. The student reuses the teacher’s existing rollout trajectory and reward scores, introducing little extra sampling cost while avoiding interference with teacher RL optimization. • We introduce AMD, a general reward-aware modulation mechanism for arbitrary distillation losses. We show that the scale-and-shift transformation of advantages naturally introduces negative samples and a CFG-like contrastive effect, which improves the quality of the distilled few-step student. • We conduct extensive experiments showing that, with only 26.1% additional training overhead over the base RL pipeline, our method enables CFG-free few-step inference that approaches the quality of the full-inference RL teacher. Compared with a naive two-stage RL-then-distill pipeline or existing unified RL-distillation methods, it requires only 1/5 training iterations and obtains substantially better alignment performance. 2 Related Work Reinforcement learning for diffusion alignment. Reinforcement learning has become a major paradigm for aligning diffusion and flow-matching models with human preferences. Policy-gradient methods such as Flow-GRPO (Liu et al., 2025), DanceGRPO (Xue et al., 2025b), MixGRPO (Li et al., 2025), AWM (Xue et al., 2025a) and DiffusionNFT (Zheng et al., 2025) optimize terminal rewards over denoising trajectories, while preference-based methods (Rafailov et al., 2023; Wallace et al., 2024; Liang et al., 2025) avoid explicit reward modeling through pairwise supervision. Direct-gradient approaches such as ReFL (Xu et al., 2023), DRaFT (Clark et al., 2024), DRTune (Wu et al., 2024), and LeapAlign (Liang et al., 2026) propagate reward gradients through differentiable sampling paths, which require differentiable rewards. Despite their success in improving reward-aligned sampling, these methods largely overlook the potential of integrating distillation into RL, and thus the possibility of achieving reward-aligned few-step generation remains underexplored. Few-step diffusion distillation. Few-step distillation aims to compress multi-step diffusion samplers into generators requiring only one or a few inference steps. Early trajectory-distillation methods directly regress the student toward the outputs of the teacher’s ODE trajectory: Knowledge Distillation (Luhman and Luhman, 2021) matches the teacher’s full sampling result in a single step, and Progressive Distillation (Salimans and Ho, 2022) iteratively halves the number of sampling steps by training the student to match two teacher steps at a time. While consistency-based methods, including Consistency Models (Song et al., 2023), Latent Consistency Models (Luo et al., 2023), and Phased Consistency Models (Wang et al., 2024), enforce trajectory-level consistency, score-based methods such as SwiftBrush (Nguyen and Tran, 2024) and Score Implicit Matching (Luo et al., 2024) pursue fast generation through score distillation. Distribution Matching Distillation (DMD) (Yin et al., 2024b) and DMD2 (Yin et al., 2024a) improve few-step quality by matching student and teacher distributions with auxiliary fake-score or discriminator-like networks. However, these distillation objectives are usually reward-agnostic and often require a frozen teacher, a separate distillation stage, or additional auxiliary models. REST is orthogonal to the choice of base distillation loss: it modulates common distillation losses with reward-derived advantages, making distillation preference-aware without redesigning the underlying distillation algorithm. Unified RL-distillation training. Sequentially applying RL alignment and few-step distillation is cumbersome and can cause objective interference: distillation may wash out RL gains, while later RL can destabilize a previously distilled few-step generator. Recent works such as DMDR (Jiang et al., 2025) and RTDMD (Huang et al., 2026) integrate DMD-style distillation with RL, but their fake-score-centered designs involve multi-network training, warm-up, and staged optimization, and their frozen reference distributions may lag behind the continuously evolving reward-optimized policy. REST instead uses a decoupled dual-branch design: the teacher continues its original RL optimization, while the student simultaneously learns a few-step policy from the same reward-scored teacher rollouts through REST. This yields a unified RL-distillation co-training framework that requires no extra image rollouts, no separate distillation dataset, and no fake-score model training. Figure 2: Overview of REST. The decoupled student reuses reward-scored teacher rollouts, while AMD modulates trajectory distillation without altering teacher RL optimization. 3 Method 3.1 Preliminary: Forward-Process Diffusion RL Our teacher branch can in principle be optimized by any diffusion RL algorithm, but it pairs especially well with forward-process RL algorithms. The representative instances include DiffusionNFT (Zheng et al., 2025), AWM (Xue et al., 2025a) and RAM (Bergmeister et al., 2026), which all cast reward alignment as a reward-weighted velocity regression on the forward noising process xt=(1−t)x0+tϵx_t=(1-t)x_0+tε, where x0x_0 is the rollout sample and ϵ∼(0,I)ε (0,I). DiffusionNFT converts a scalar reward into an optimality probability r∈[0,1]r∈[0,1] and implicitly parameterizes a positive and a negative policy around the old velocity voldv old, vϕ+=(1−β)vold+βvϕ(xt),vϕ−=(1+β)vold−βvϕ(xt),v_φ^+=(1-β)v old+β v_φ(x_t), v_φ^-=(1+β)v old-β v_φ(x_t), (1) and trains the policy vϕv_φ with ℒNFT(ϕ)=[r‖vϕ+(xt)−vgt‖2+(1−r)‖vϕ−(xt)−vgt‖2],L_ NFT(φ)=E [r \|v_φ^+(x_t)-v^gt \|^2+(1-r) \|v_φ^-(x_t)-v^gt \|^2 ], (2) where vgt=ϵ−x0v^gt=ε-x_0 is the forward flow-matching (Lipman et al., 2022) target. RAM instead directly regresses vϕv_φ toward a reward-shifted version of the reference velocity, ℒRAM(ϕ)=t[‖vϕ(xt)−sg(vbase(xt)+r(x0)(vgt−vϕ(xt)))‖2],L_ RAM(φ)=E_t [ \|v_φ(x_t)-sg\! (v base(x_t)+r(x_0) (v^gt-v_φ(x_t) ) ) \|^2 ], (3) where sg(⋅)sg(·) is the stop-gradient operator and vbase(⋅)v base(·) means frozen base model. In our framework, the teacher branch is optimized by any forward-process algorithm, e.g., Eq. 2 or Eq. 3, producing an evolving teacher policy vϕv_φ together with reward-scored rollouts that the student branch reuses. 3.2 Decoupled Teacher–Student Co-training Given the teacher policy vϕv_φ as in 3.1, REST attaches a decoupled student branch vθv_θ that learns a few-step, CFG-free generator from the teacher’s rollouts, without altering the teacher optimization, as shown in Fig. 2. For each prompt c, the teacher branch samples with its ODE solver (Song et al., 2020) on an M-step schedule (e.g., M=20M=20) with classifier-free guidance (CFG), producing a trajectory =xt0,xt1,…,xtM,1=t0>t1>⋯>tM=0,T= \x_t_0,x_t_1,…,x_t_M \, 1=t_0>t_1>·s>t_M=0, (4) and a terminal reward r=R(x0,c)r=R(x_0,c). The student, in contrast, is designed to run with only K≪MK M steps (e.g., K=8K=8) at inference time. Rather than sampling a separate trajectory for the student, we select a K-step subset of the teacher’s schedule and re-index it as an imitation trajectory sub=xtq0,xtq1,…,xtqK⊂,tq0,…,tqK⊂t0,…,tM,T_ sub= \x_t_q0,x_t_q1,…,x_t_qK \ , \t_q0,…,t_qK\⊂\t_0,…,t_M\, (5) so that the student can be trained directly on the teacher’s existing rollout, with no additional sampling required. For each student step k, the piecewise trajectory velocity of the corresponding segment is directly computed from the teacher’s rollout, vkgt=xtq(k+1)−xtqktq(k+1)−tqk.v gt_k= x_t_q(k+1)-x_t_qkt_q(k+1)-t_qk. (6) In this decoupled co-training framework, the student model θ is trained to imitate this piecewise trajectory, as detailed in Sec. 3.3, thereby inheriting the teacher’s generation quality while enabling few-step CFG-free sampling. This dual-branch design brings two benefits. First, it is a one-stage, end-to-end training framework: the student is trained by reusing the teacher’s existing rollout trajectory, making training more efficient than a two-stage pipeline. Second, because the teacher policy vϕv_φ continues to evolve under RL training, avoiding a frozen teacher that could hold back the student branch’s RL progress. 3.3 Advantage-Weighted Regression for Teacher Tracking Advantage-weighted regression (Peters and Schaal, 2007; Peng et al., 2019; Kostrikov et al., 2021), i.e. AWR, recasts policy improvement as a reward-weighted supervised regression problem: it treats each observed state-action pair as a demonstration and re-weights its log-likelihood under the current policy by the advantage of that action, so that high-advantage actions are imitated more strongly while low- or negative-advantage actions are discouraged. We adopt this view to let the student track the teacher’s reward-scored rollout: each teacher trajectory segment is treated as a state-action demonstration, whose imitation strength is modulated by the reward of the rollout it comes from. For prompt condition c, the state-action pair sk=(xtqk,tqk,c)s_k= (x_t_qk,t_qk,c ) and the demonstrated action ak=vkgta_k=v gt_k given by the teacher’s segment velocity (Eq. 6). We interpret the distribution of student output as the mean of an implicit, fixed-variance Gaussian policy following Liu et al. (2025) over teacher segment velocities, πθ(ak∣sk)=(ak;vθ(sk),σ2I), _θ(a_k s_k)=N\! (a_k;\,v_θ(s_k),σ^2I ), (7) under which the negative log-likelihood of the demonstrated action is, up to an additive constant independent of θ, −logπθ(ak∣sk)=12σ2‖ak−vθ(sk)‖2+const.- _θ(a_k s_k)= 12σ^2\|a_k-v_θ(s_k)\|^2+const. (8) Dropping the constant, this yields the plain per-step imitation loss ℓbase(θ;k)=‖vθ(sk)−ak‖2∝−logπθ(ak∣sk), _ base(θ;k)= \|v_θ(s_k)-a_k \|^2\; \;- _θ(a_k s_k), (9) which treats every teacher trajectory segment as equally reliable supervision. AWR improves a policy by imitating demonstrated actions with weights given by their advantages, maxθ(s,a)∼[A(s,a)logπθ(a∣s)]. _θ\;E_(s,a) [A(s,a) _θ(a s) ]. (10) Using Eq. 8, this objective induces an advantage-weighted regression loss over teacher trajectory segments, where the log-likelihood term is replaced by the base imitation loss ℓbase _ base. where D is the reward-scored teacher trajectory dataset formed by all segments (sk,ak)(s_k,a_k). For each reward source i, its advantage A(i)(sk,ak)∈[−1,1]A^(i)(s_k,a_k)∈[-1,1] is the clipped group-normalized advantage of the rollout associated with prompt c, and is shared by all segments from the same rollout, as in Flow-GRPO (Liu et al., 2025) and AWM (Xue et al., 2025a). We further fuse different rewards on the normalized advantages level, Amix=∑i=1NRαiA(i)∑i=1NRαi,αi≥0,A_ mix= _i=1^N_R _iA^(i) _i=1^N_R _i, _i≥ 0, (11) where NRN_R is the number of reward sources and αi _i controls their contribution. We then introduce a global scale λ=1λ=1 and a positive shift b=0.5b=0.5 to obtain a signed modulation coefficient, leading to our Advantage-Modulated Distillation (AMD) objective for teacher trajectory tracking: ℒAMD(θ)=(sk,ak)∼[λ(Amix+b)ℓbase(θ;k)].L_ AMD(θ)=E_(s_k,a_k) [λ (A_ mix+b )\, _ base(θ;k) ]. (12) In practice, we further regularize the student against an EMA copy of itself, denoted by vθemav_ _ ema, using a fixed-variance Gaussian KL surrogate ℒKL-EMA(θ)=k[‖vθ(sk)−vθema(sk)‖2],L_ KL -EMA(θ)=E_k [ \|v_θ(s_k)-v_ _ema(s_k) \|^2 ], (13) which stabilizes optimization and reduces the training jitter caused by the continuously changing teacher policy. The final student objective is therefore ℒstudent(θ)=ℒAMD(θ)+βℒKL-EMA(θ),L_ student(θ)=L_ AMD(θ)+ _ KL -EMA(θ), (14) where β controls the strength of the EMA-student regularization. 3.4 Understanding Advantage-Modulated Distillation Decomposing the AMD objective. Equation 12 can be decomposed into two additive terms, ℒAMD(θ)=λb(sk,ak)∼[ℓbase(θ;k)]⏟imitation prior+λ(sk,ak)∼[Amixℓbase(θ;k)]⏟reward-driven correction.L_ AMD(θ)= λ b\,E_(s_k,a_k) \! [ _ base(θ;k) ]_imitation prior+ λ\,E_(s_k,a_k) \! [A_ mix\, _ base(θ;k) ]_reward-driven correction. (15) The first term is a constant-weighted imitation prior. A positive shift b is crucial in our setting: the initial student branch does not yet possess reliable image-generation capability, and therefore needs a stable positive imitation signal to bootstrap its few-step generator. From another perspective, even trajectories with relatively low advantages still contain useful teacher dynamics for the student, especially at early training stages, as shown in Fig. 6. The second term is the reward-driven correction inherited from reward-weighted supervised regression: it biases the student toward teacher segments with high combined reward advantages, allowing the few-step student to concentrate on the best parts of the teacher’s reward-scored rollouts and potentially surpass the average behavior of the RL-trained full-step teacher. This reward-aware distillation effect is unavailable to conventional trajectory distillation methods (Salimans and Ho, 2022) that imitate teacher rollouts uniformly. AMD as a generic reward-aware distillation wrapper. Under the AWR view above, the base per-step imitation loss ℓbase _ base is instantiated as the segment-velocity MSE in Eq. 9. This MSE could be understood as the simplest trajectory distillation objective (Salimans and Ho, 2022) rather than a required design choice. Since AMD only modulates the weight in front of ℓbase _ base, the same mechanism can wrap more advanced distillation losses, such as PCM-style phase consistency losses (Wang et al., 2024), DMD-style distribution-matching losses (Yin et al., 2024a), or other trajectory-level objectives. Therefore, AMD can serve as a generic reward-aware wrapper over diffusion distillation objectives, making different student distillation algorithms benefit from the reward-scored teacher rollouts without changing the teacher’s RL training procedure. 4 Experiments 4.1 Experimental Setup Benchmarks and rewards. Following the experimental protocol of RAM and Flow-GRPO (Liu et al., 2025), we consider three representative text-to-image reward objectives: compositional correctness (GenEval), visual text rendering (OCR), and human preference alignment (PickScore). GenEval (Ghosh et al., 2023) measures whether generated images satisfy object, attribute, counting, and spatial-relation constraints. OCR evaluates visual text rendering through an edit-distance reward that checks whether the target text specified in the prompt appears legibly in the image. PickScore (Kirstain et al., 2023) is a learned human-preference model trained from large-scale pairwise image comparisons. For PickScore alignment, we use PickScore as the training reward. For OCR and GenEval, however, we use a multi-reward setting: OCR+PickScore and GenEval+PickScore, respectively. This design is motivated by an empirical failure mode of prior single-reward optimization (Fig. 5): although OCR or GenEval scores can become high, the generated images may collapse into reward-hacking artifacts, such as overly large text covering the whole image or simplified layouts that discard most visual details. This collapse is reflected by DrawBench quality metrics becoming even lower than the base model SD3.5M. Adding PickScore as an auxiliary reward makes the optimization more conservative and better preserves general image quality while still improving the target task reward. Training protocol and AMD settings. We use Stable Diffusion 3.5 Medium (SD3.5M) (Esser et al., 2024) as the backbone and train one LoRA (Hu et al., 2022) (r=32r=32, α=64α=64) per reward setting with bf16 precision, learning rate 3×10−43× 10^-4, 48 prompts per group, and 24 samples per prompt. PickScore models are trained for 500 steps, while GenEval and OCR models are trained for 300 steps. The teacher follows RAM’s forward-process RL setup: 20-step training solver, 40-step evaluation solver, and CFG-enabled sampling. The student is jointly trained with AMD and evaluated using the EMA student (decay 0.90.9) with 8 CFG-free steps. For AMD, each reward source uses group-relative advantages normalized within same-prompt samples to [−1,1][-1,1]; OCR/GenEval task advantages are fused with PickScore advantages via Eq. 11. We set the AMD scale to λ=1λ=1, shift to b=0.5b=0.5, and KL-EMA coefficient to 0.20.2. Training rewards are evaluated on held-out benchmark prompts, while generic image quality is evaluated on DrawBench (Saharia et al., 2022). Generic image-quality evaluation. Optimizing a certain reward can degrade generic image quality, a failure mode often referred to as reward hacking. Following DiffusionNFT (Zheng et al., 2025), we evaluate each trained model on DrawBench (Saharia et al., 2022) prompts that are disjoint from the reward-training and reward-test prompts. We report five quality and preference metrics: Aesthetic (Schuhmann and Beaumont, 2022) and DeQA (You et al., 2025) for perceptual quality, and ImageReward (Xu et al., 2023), HPSv2 (Wu et al., 2023), and PickScore (Kirstain et al., 2023) for human preference. Figure 3: Qualitative comparison on DrawBench. REST achieves high-quality outputs with superior preference alignment under few-step inference, while matching full-step inference RL teacher. 4.2 Main Results Table 1: SD3.5M post-training results on three tasks (GenEval, OCR, PickScore). Gray rows use a single task reward; † denotes joint training with the task reward and PickScore. Higher is better. Model NFE Training Reward Image Quality Metrics GenEval OCR PickScore Aesthetic DeQA ImgRwd HPSv2 PickScore SD3.5M 40 0.64 0.63 21.78 5.39 4.08 0.85 0.28 22.40 SD3.5M 8 0.28 0.12 19.50 5.15 2.38 -0.96 0.18 20.39 Compositional Image Generation Flow-GRPO 40 !150.95 !15 !15 !155.25 !154.01 !151.03 !150.27 !1522.37 AWM 40 !150.83 !15 !15 !155.14 !153.75 !150.67 !150.24 !1522.04 DiffusionNFT 40 !150.95 !15 !15 !154.98 !154.10 !150.30 !150.24 !1521.59 RAM 40 !150.97 !15 !15 !155.38 !154.09 !151.19 !150.29 !1522.52 RAM† 40 0.95 5.42 4.12 1.12 0.28 22.76 RAM† 8 0.65 5.46 2.27 -0.19 0.20 21.36 Ours† 8 0.94 5.67 4.15 1.05 0.30 22.79 Visual Text Rendering Flow-GRPO 40 !15 !150.92 !15 !155.32 !154.06 !150.95 !150.28 !1522.44 AWM 40 !15 !150.97 !15 !155.01 !152.83 !15-0.85 !150.18 !1520.56 DiffusionNFT 40 !15 !150.96 !15 !154.87 !153.01 !15-0.97 !150.18 !1520.26 RAM 40 !15 !150.97 !15 !155.23 !153.90 !150.44 !150.26 !1521.83 RAM† 40 0.96 5.58 4.16 1.07 0.30 22.81 RAM† 8 0.68 5.59 2.60 -0.21 0.22 21.45 Ours† 8 0.96 5.65 4.16 1.00 0.30 22.83 Human Preference Alignment Flow-GRPO 40 23.31 5.92 4.22 1.28 0.32 23.53 AWM 40 23.39 6.31 4.10 1.27 0.31 23.76 DiffusionNFT 40 23.29 6.16 4.13 1.23 0.31 23.65 RAM 40 23.70 6.17 4.21 1.35 0.32 23.95 RAM 8 23.30 6.42 3.82 1.11 0.31 23.55 Ours 8 23.73 6.30 4.15 1.35 0.33 23.96 The full-step RL teacher provides the reward-aligned trajectory distribution that REST reuses for student training, and the REST student is designed to retain the gains of the 40-step RAM teacher while reducing inference to a few CFG-free steps. Tab. 1 summarizes the reward and image-quality results. For GenEval and OCR, the gray-shaded rows report single-reward baselines as diagnostic references: as discussed above, these settings can achieve high task rewards while severely degrading generic image quality. We therefore focus on the rows marked by † , which use the multi-reward setting with the task reward and PickScore. As shown in Tab. 1, REST achieves performance comparable to the 40-step RAM teacher on both training rewards and out-of-domain DrawBench metrics, while using only a few CFG-free inference steps. It also obviously surpasses full-step RL baselines such as Flow-GRPO (Liu et al., 2025), AWM (Xue et al., 2025a) and DiffusionNFT (Zheng et al., 2025), and substantially improves over the naive 8-step RAM (Bergmeister et al., 2026). We note that 8-step RAM can occasionally obtain a high Aesthetic score, but this is often associated with fragmented structures and noisy details that hack the metric rather than reflect better perceptual quality. Overall, by continuously exposing the student to the evolving reward-optimized teacher rollouts and using AMD to select and amplify high-value trajectory segments, REST enables the student to go beyond plain trajectory compression. Qualitative comparisons in Figs. 3, 4, and 5 further show that REST preserves generic image quality, human preference, and visual text rendering under few-step CFG-free sampling. Figure 4: Qualitative comparison under PickScore alignment on the Pick-a-Pic testset. Figure 5: Qualitative comparison on visual text rendering. Single OCR reward leads to collapse by overly large text covering the whole image. OCR+PickScore optimization prevents the mode collapse, while REST achieves the highest image quality. Table 2: Few-step DrawBench comparison with RTDMD. RTDMD training iterations include 1,500 warm-up and 1,000 parallel training iterations; † denotes officially released weights and ‡ denotes our reproduced version using official code on PickScore reward and Pick-a-Pic dataset. Methods NFE Train Iters Aesthetic ImgRwd HPSv2 PickScore CLIPScore RTDMD† 4 1500+1000 6.09 1.26 0.326 23.30 0.930 RTDMD† 8 1500+1000 6.27 1.20 0.330 23.14 0.916 RTDMD‡ 4 1500+1000 6.18 1.27 0.310 23.71 0.885 RTDMD‡ 8 1500+1000 6.07 1.21 0.310 23.49 0.886 REST (full) 8 500 6.30 1.35 0.333 23.96 0.949 Compared with Few-step RL Methods. DMDR and RTDMD (Jiang et al., 2025; Huang et al., 2026) combine RL and DMD-style distillation through a DMD warm-up stage followed by parallel GRPO-DMD training. However, they use larger training datasets (Schuhmann et al., 2021) and longer training of roughly 2,500–3,000 steps. More importantly, neither work directly compares its distilled generator against the corresponding full-step RL model trained under the same setting, leaving unclear how much of the RL policy’s performance is actually preserved after distillation. REST instead targets the quality of full-step RL models under the popular GenEval, OCR, PickScore, and DrawBench protocols in diffusion RL fields, while requiring far fewer training steps. For an additional comparison, we evaluate the official RTDMD weights and reproduced RTDMD using official code on the fair rewards and prompts, at the training-independent DrawBench benchmark. As shown in Tab. 2, REST achieves stronger aesthetic and preference scores over both RTDMD versions despite using substantially fewer training steps. Figure 6: The shift b controls the positive imitation prior in the AMD weight. A moderate shift preserves the necessary imitation prior while turning clearly low-quality trajectories into repulsive supervision, thereby amplifying the effect/style from RL signals. 4.3 Ablation Study Table 3: Progressive ablation of REST under PickScore training. All methods use 8-step CFG-free sampling on DrawBench dataset. Variant Teacher RL Dual Branch AMD Aesthetic DeQA ImgRwd HPSv2 PickScore Base model – – – 5.15 2.38 -0.96 0.18 20.39 RL Yes – – 6.42 3.82 1.11 0.31 23.55 RL then Distill Yes – – 6.22 4.14 1.32 0.32 23.83 RL & Distill Yes Yes – 6.20 4.09 1.32 0.32 23.74 REST (full) Yes Yes Yes 6.30 4.15 1.35 0.33 23.96 Tab. 3 studies the contribution of each component under PickScore training. We start from direct 8-step CFG-free inference with the base model, then add standard RL (RAM), a sequential RL-then-distill two-stage pipeline, simultaneous dual-branch RL with distillation, and finally the full REST model with AMD. The results lead to three observations. First, full-step standard RL itself improves 8-step CFG-free sampling, showing that reward optimization can partially compensate for the low-step generation gap. Second, simultaneous dual-branch RL-distillation in REST retains most of the gains of the RL-then-distill pipeline while reducing training cost by only 1/4 training iterations, since the student reuses the teacher rollouts instead of requiring a separate distillation stage. Third, AMD further strengthens reward preference and improves perceptual quality, indicating that reward-aware trajectory weighting is more effective than uniform imitation. Table 4: Ablation of the AMD shift b and EMA regularization under PickScore training, evaluated with DrawBench metrics. Variant Shift b EMA Aesthetic DeQA ImgRwd HPSv2 PickScore No shift 0 Yes 6.03 3.92 0.09 0.25 21.48 Larger shift 1.0 Yes 6.25 4.19 1.33 0.32 23.76 W/o EMA 0.5 No 6.32 4.11 1.34 0.31 23.91 REST (full) 0.5 Yes 6.30 4.15 1.35 0.33 23.96 Tab. 4 further isolates the stabilizing components of the full AMD objective in the same PickScore setting. The shift b controls the positive imitation prior in the AMD weight. Setting b=0b=0 removes this baseline imitation pressure, making the objective dominated by aggressive negative modulation and thus potentially destabilizing early student training; a larger shift, in contrast, makes the objective closer to conservative imitation. As shown in Fig. 6, a moderate shift preserves the necessary imitation prior while turning clearly low-quality trajectories into repulsive supervision, thereby amplifying the effect of RL signals. We interpret this moderate negative modulation as a CFG-like contrastive force that is beneficial for improving generation quality. We also remove the EMA term to evaluate its stabilizing role: without this anchor, the student exhibits much stronger performance oscillation rather than a stable improvement, confirming that EMA regularization helps absorb the jitter caused by the continuously evolving teacher policy. Figure 7: REST with different distillation objectives. AMD performs consistently with segment-velocity and PCM-style losses. 4.4 Analysis Training efficiency. Since REST reuses the teacher’s existing rollout trajectories and reward evaluations, the student branch incurs no additional sampling cost over the base RL pipeline—the most expensive stage in RL. Its only extra computation is the student forward/backward pass and EMA regularization. Specifically, in REST training, sampling rollout with reward computation accounts for 40.7% of the wall-clock time, teacher training for 33.3%, and student training for only 26.1%. Taking into account overheads such as model checkpointing and online validation, the overall additional computational cost introduced by REST is below 25% over pure RL. Compared with sequential RL-then-distill training and RTDMD, both of which require substantially more training steps, REST also exhibits a clear advantage in convergence speed, as shown in Fig. 1. Generality across distillation losses. AMD only modulates the weight of a base distillation objective and is therefore not tied to the per-step imitation loss (segment-velocity MSE). We further integrate AMD with Phased-Consistency-Model-style phase consistency. Specifically, within each student-defined phase, REST trains every adjacent pair of teacher states to predict the same phase endpoint (more expensive than per-step imitation loss): the online student predicts from the higher-noise state, while the EMA student provides a stop-gradient target from the lower-noise state, and the two predictions are matched with a pseudo-Huber loss. The reward-derived AMD weight is then applied to this phase-local consistency loss. As shown in Fig. 7, the PCM variant surpasses 8-step RAM in our experiments, and gets similar performance with segment-velocity MSE, implying REST’s compatibility with other distillation losses. Detailed procedures of the per-step imitation loss and PCM+AMD variants are provided in Algs. 1 and 2. 5 Limitations and future work. Although we validate AMD with both per-step imitation MSE and PCM-style consistency (Wang et al., 2024), its integration with DMD2 (Yin et al., 2024a) remains unexplored. A promising direction is to combine RL and DMD2 within a single REST stage to enable high-quality 4-step CFG-free inference. A second limitation is that our experiments instantiate the teacher branch only with RAM (Bergmeister et al., 2026). Nevertheless, REST explicitly decouples teacher-side RL from student-side distillation: the student consumes reward-scored ODE trajectories without depending on the teacher’s specific policy-optimization objective. Appendix A provides theoretical support for extending REST to DiffusionNFT (Zheng et al., 2025) by showing that representative ODE-based diffusion RL algorithms, including RAM and DiffusionNFT, share an anchored, reward-shifted velocity-regression structure despite differences in their anchor choices and hyperparameters. This structural commonality suggests that REST can serve as a general co-training framework for a broader class of ODE-based diffusion RL teachers. Empirically validating REST with DiffusionNFT and other policy-optimization algorithms remains important future work. 6 Conclusion In this paper, we present REST, a unified RL-distillation co-training framework for efficient and reward-aligned diffusion generation. Our core insight is that reward-optimized diffusion rollouts already contain rich few-step distillation signals, but existing RL methods mainly optimize from terminal samples and leave the intermediate teacher trajectory underused. We operationalize this insight through a decoupled teacher–student design: the teacher branch continues standard RL post-training, while the student branch simultaneously learns a CFG-free few-step generator from the same reward-scored rollout trajectory. To make this distillation preference-aware, we introduce Advantage-Modulated Distillation (AMD), a general reward-aware wrapper that modulates arbitrary base distillation losses with advantage-derived weights, strengthening high-reward trajectory segments while suppressing or repelling low-reward ones. Experiments on SD3.5M across GenEval, OCR, and PickScore settings demonstrate that REST preserves the reward gains of a full-step RL teacher while substantially reducing inference cost, and the ablation and analysis studies further validate the importance of decoupled co-training, advantage modulation, KL-EMA stabilization, and the generality of REST across teacher RL algorithms and distillation losses. References A. Bergmeister, S. Jegelka, N. Nüsken, C. Domingo-Enrich, and J. Pidstrigach (2026) Reinforce adjoint matching: scaling rl post-training of diffusion and flow-matching models. arXiv preprint arXiv:2605.10759. Cited by: §3.1, §4.2, §5. Black-forest-labs (2024) FLUX.1-dev. Note: https://github.com/black-forest-labs/flux Cited by: §1. H. Chen, K. Zheng, Q. Zhang, G. Cui, Y. Cui, H. Ye, T. Lin, M. Liu, J. Zhu, and H. Wang (2026) NFT: bridging supervised learning and reinforcement learning in math reasoning. In International Conference on Learning Representations, Vol. 2026, p. 124025–124042. Cited by: §1. K. Clark, P. Vicol, K. Swersky, and D. Fleet (2024) Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations, Vol. 2024, p. 4793–4822. Cited by: §2. P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, and R. Rombach (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 12606–12633. Cited by: §4.1. L. Fan, P. Sun, T. Wen, S. Lu, and C. Song (2026) Rdm: re-conceptualizing distribution matching as a reward for diffusion distillation. arXiv preprint arXiv:2603.28460. Cited by: §1. D. Ghosh, H. Hajishirzi, and L. Schmidt (2023) GenEval: an object-focused framework for evaluating text-to-image alignment. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §4.1. J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §1, §1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR. Cited by: §4.1. Y. Huang, X. Zhou, R. Wang, C. Zhang, J. Zhang, and T. Pang (2026) Reinforcing few-step generators via reward-tilted distribution matching. arXiv preprint arXiv:2605.26108. Cited by: §1, §2, §4.2. D. Jiang, D. Liu, Z. Wang, Q. Wu, L. Li, H. Li, X. Jin, D. Liu, Z. Li, B. Zhang, et al. (2025) Distribution matching distillation meets reinforcement learning. arXiv preprint arXiv:2511.13649. Cited by: §1, §1, §2, §4.2. Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy (2023) Pick-a-pic: an open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §4.1, §4.1. I. Kostrikov, A. Nair, and S. Levine (2021) Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169. Cited by: §3.3. J. Li, Y. Cui, T. Huang, Y. Ma, C. Fan, Y. Cheng, M. Yang, Z. Zhong, and L. Bo (2025) Mixgrpo: unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802. Cited by: §2. Z. Liang, T. Yang, J. Wu, C. Feng, and L. Zheng (2026) LeapAlign: post-training flow matching models at any generation step by building two-step trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 23238–23248. Cited by: §2. Z. Liang, Y. Yuan, S. Gu, B. Chen, T. Hang, M. Cheng, J. Li, and L. Zheng (2025) Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13199–13208. Cited by: §2. Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §3.1. J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025) Flow-grpo: training flow matching models via online rl. arXiv preprint arXiv:2505.05470. Cited by: §2, §3.3, §3.3, §4.1, §4.2. E. Luhman and T. Luhman (2021) Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388. Cited by: §2. S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao (2023) Latent consistency models: synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. Cited by: §1, §2. W. Luo, Z. Huang, Z. Geng, J. Z. Kolter, and G. Qi (2024) One-step diffusion distillation through score implicit matching. Advances in Neural Information Processing Systems 37, p. 115377–115408. Cited by: §2. T. H. Nguyen and A. Tran (2024) Swiftbrush: one-step text-to-image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7807–7816. Cited by: §2. X. B. Peng, A. Kumar, G. Zhang, and S. Levine (2019) Advantage-weighted regression: simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177. Cited by: §3.3. J. Peters and S. Schaal (2007) Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, p. 745–750. Cited by: §3.3. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2. C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, S. K. S. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi (2022) Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, Vol. 35, p. 36479–36494. Cited by: §4.1, §4.1. T. Salimans and J. Ho (2022) Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. Cited by: §2, §3.4, §3.4. C. Schuhmann and R. Beaumont (2022) LAION-aesthetics. Note: laion.ai Cited by: §4.1. C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki (2021) LAION-400m: open dataset of clip-filtered 400 million image-text pairs. arXiv:2111.02114. Cited by: §4.2. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1. J. Song, C. Meng, and S. Ermon (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §3.2. Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In Proceedings of the 40th International Conference on Machine Learning, p. 32211–32252. Cited by: §2. B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik (2024) Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8228–8238. Cited by: §2. F. Wang, Z. Huang, A. W. Bergman, D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, et al. (2024) Phased consistency models. Advances in neural information processing systems 37, p. 83951–84009. Cited by: §2, §3.4, §5. C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S. Yin, S. Bai, X. Xu, Y. Chen, Y. Chen, Z. Tang, Z. Zhang, Z. Wang, A. Yang, B. Yu, C. Cheng, D. Liu, D. Li, H. Zhang, H. Meng, H. Wei, J. Ni, K. Chen, K. Cao, L. Peng, L. Qu, M. Wu, P. Wang, S. Yu, T. Wen, W. Feng, X. Xu, Y. Wang, Y. Zhang, Y. Zhu, Y. Wu, Y. Cai, and Z. Liu (2025) Qwen-image technical report. Cited by: §1. X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: §4.1. X. Wu, Y. Hao, M. Zhang, K. Sun, Z. Huang, G. Song, Y. Liu, and H. Li (2024) Deep reward supervisions for tuning text-to-image diffusion models. In European Conference on Computer Vision, p. 108–124. Cited by: §2. J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong (2023) Imagereward: learning and evaluating human preferences for text-to-image generation. NeurIPS. Cited by: §2, §4.1. S. Xue, C. Ge, S. Zhang, Y. Li, and Z. Ma (2025a) Advantage weighted matching: aligning rl with pretraining in diffusion models. arXiv preprint arXiv:2509.25050. Cited by: §1, §2, §3.1, §3.3, §4.2. Z. Xue, J. Wu, Y. Gao, F. Kong, L. Zhu, M. Chen, Z. Liu, W. Liu, Q. Guo, W. Huang, et al. (2025b) Dancegrpo: unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818. Cited by: §2. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and W. T. Freeman (2024a) Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems 37, p. 47455–47487. Cited by: §2, §3.4, §5. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park (2024b) One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6613–6623. Cited by: §1, §1, §2. Z. You, X. Cai, J. Gu, T. Xue, and C. Dong (2025) Teaching large language models to regress accurate image quality scores using score distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14483–14494. Cited by: §4.1. K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu (2025) Diffusionnft: online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.16117. Cited by: §2, §3.1, §4.1, §4.2, §5. Appendix A Relationship between DiffusionNFT and RAM Although our main experiments instantiate the teacher branch only with RAM, REST explicitly decouples teacher-side reinforcement learning from student-side distillation: the student consumes reward-scored ODE trajectories without depending on the teacher’s specific policy-optimization objective. This section provides theoretical support for extending REST to a broader class of ODE-based diffusion RL teachers by comparing RAM and DiffusionNFT. We show that these representative methods share an anchored, reward-shifted velocity-regression structure despite differences in their anchor choices and hyperparameters. Rather than claiming that the two algorithms are fully equivalent, this structural commonality identifies a shared trajectory interface that enables REST to accommodate different teacher optimizers without changing its student-side distillation mechanism. To make this connection explicit, we compare their objectives on the forward noising process. Let vgt=ϵ−x0v gt=ε-x_0 denote the flow-matching target, voldv old the lagged policy used for sampling, and vϕv_φ the trainable policy. DiffusionNFT constructs two implicit policies symmetric around voldv old, vϕ+=(1−β)vold+βvϕ,vϕ−=(1+β)vold−βvϕ,v_φ^+=(1-β)v old+β v_φ, v_φ^-=(1+β)v old-β v_φ, (16) and optimizes the reward-weighted flow-matching objective ℒNFT=r‖vϕ+−vgt‖2+(1−r)‖vϕ−vgt‖2,L_ NFT=r \|v_φ^+-v gt \|^2+(1-r) \|v_φ^--v gt \|^2, (17) where r∈[0,1]r∈[0,1] is obtained from the clipped group-relative advantage. For clarity, we first omit the time-dependent and detached self-normalization factors used in the implementation; their effect is discussed below. Equivalent target of DiffusionNFT. Define Δ=vold−vgt,δϕ=vϕ−vold,A=2r−1∈[−1,1]. =v old-v gt, _φ=v_φ-v old, A=2r-1∈[-1,1]. (18) Equation 16 gives vϕ+−vgt=Δ+βδϕv_φ^+-v gt= +β _φ and vϕ−vgt=Δ−βδϕv_φ^--v gt= -β _φ. Substituting them into Eq. 17 yields ℒNFT _ NFT =r‖Δ+βδϕ‖2+(1−r)‖Δ−βδϕ‖2 =r \| +β _φ \|^2+(1-r) \| -β _φ \|^2 (19) =‖Δ‖2+2βA⟨Δ,δϕ⟩+β2‖δϕ‖2 = \| \|^2+2β A , _φ +β^2 \| _φ \|^2 (20) =β2‖δϕ+AβΔ‖2+(1−A2)‖Δ‖2. =β^2 \| _φ+ Aβ \|^2+(1-A^2) \| \|^2. (21) The last term is independent of vϕv_φ. Therefore, Eq. 17 is gradient-equivalent to ℒ~NFT=β2‖vϕ−vtargetNFT‖2,vtargetNFT=vold+Aβ(vgt−vold). L_ NFT=β^2 \|v_φ-v_ target NFT \|^2, v_ target NFT=v old+ Aβ (v gt-v old ). (22) Thus, DiffusionNFT can be interpreted as regression from the old policy toward a reward-dependent target. Positive advantages move the policy toward vgtv gt, while negative advantages extrapolate it away from vgtv gt through the same anchor voldv old. Comparison with RAM. RAM directly constructs the stop-gradient target vtargetRAM=vbase+ηARAM(vgt−vold),ℒRAM=‖vϕ−sg(vtargetRAM)‖2,v_ target RAM=v base+η A_ RAM (v gt-v old ), _ RAM= \|v_φ-sg (v_ target RAM ) \|^2, (23) where vbasev base is the frozen pretrained velocity, ARAMA_ RAM is the normalized advantage, and η is its scale. Equations 22 and 23 expose their shared template, vtarget=vanchor+γ(A)(vgt−vold).v_ target=v_ anchor+γ(A) (v gt-v old ). (24) Both methods therefore perform forward-process velocity regression along the same reward-controlled direction vgt−voldv gt-v old. DiffusionNFT uses vanchor=voldv_ anchor=v old and γ(A)=A/βγ(A)=A/β, whereas RAM uses vanchor=vbasev_ anchor=v base and γ(A)=ηARAMγ(A)=η A_ RAM. In the practically relevant regime where the lagged policy remains close to the pretrained model, vold≈vbasev old≈ v base, and the reward scales satisfy A/β≈ηARAMA/β≈η A_ RAM, their regression targets approximately coincide. Full DiffusionNFT objective and limitations of the equivalence. The implementation of DiffusionNFT uses detached residual normalizers w+w_+ and w−w_- and an additional time weight. With the definitions above, its gradient has the form ∇vϕℒNFT∝t[(rw+−1−rw−)Δ+β(rw++1−rw−)δϕ]. _v_φL_ NFT t [ ( rw_+- 1-rw_- ) +β ( rw_++ 1-rw_- ) _φ ]. (25) Because w+w_+ and w−w_- are stop-gradient quantities, this remains an anchored regression update: the first term supplies the reward-dependent direction and the second pulls the trainable policy toward voldv old. When w+=w−w_+=w_-, Eq. 25 reduces exactly, up to a positive scalar, to the gradient of Eq. 22. When the two normalizers are merely close, this equivalence is approximate; when they differ substantially, they adaptively rescale the effective advantage and trust-region strength. The relationship above establishes structural similarity rather than complete equivalence. DiffusionNFT parameterizes symmetric positive and negative policies, applies time-dependent self-normalization, and anchors its policy-improvement target at voldv old. RAM instead uses a single closed-form stop-gradient target anchored at the frozen vbasev base, with a separately chosen advantage scale. Nevertheless, both optimize the same forward-process flow-matching direction through reward-shifted velocity regression. This shared structure explains why REST, although instantiated with RAM in our main experiments, can in principle use DiffusionNFT as its teacher optimizer without changing the student-side trajectory distillation mechanism. Appendix B Detailed Training Algorithms of REST We provide the complete training procedures for the two REST variants evaluated in this work. They share the same reward-optimized teacher branch, reuse the same teacher trajectories and terminal rewards, and differ only in the student-side base distillation objective wrapped by AMD. Let ϕφ, ϕ¯ φ, θ, and θ¯ θ denote the online teacher, EMA teacher, online student, and EMA student, respectively. Let M and K be the numbers of teacher and student steps, with K≪MK M. Algorithm 1 REST with Segment-Velocity Distillation and AMD 1:Online/EMA teacher vϕ,vϕ¯v_φ,v_ φ; online/EMA student vθ,vθ¯v_θ,v_ θ; rewards Rmm=1NR\R_m\_m=1^N_R. 2:Teacher steps M; student boundaries 0=q0<⋯<qK=M0=q_0<·s<q_K=M; AMD parameters λ,b,αmλ,b,\ _m\; EMA coefficient βema _ ema. 3:while not converged 4:Phase 1: Teacher Rollout and Multi-Reward Advantage 5:Sample prompt c and a full CFG-enabled trajectory with vϕ¯v_ φ: 6: =xt0,…,xtMT=\x_t_0,…,x_t_M\, where xtj+1=xtj+(tj+1−tj)vϕ¯CFG(xtj,tj,c)x_t_j+1=x_t_j+(t_j+1-t_j)v_ φ CFG(x_t_j,t_j,c). 7:Decode xtMx_t_M and compute Rm(xtM,c)R_m(x_t_M,c) for every reward source m. 8:Normalize and clip each reward within the same-prompt group to obtain A(m)∈[−1,1]A^(m)∈[-1,1]. 9:Fuse advantages: Amix=∑mαmA(m)/∑mαmA_ mix= _m _mA^(m)/ _m _m. 10:Phase 2: Decoupled Teacher RL Update 11:Update ϕφ with the original teacher RL objective using xtMx_t_M and AmixA_ mix. 12:Do not propagate student gradients into ϕφ or the stored trajectory T. 13:Phase 3: Segment-Velocity Student Distillation 14:for k=0,…,K−1k=0,…,K-1 do 15: Construct sk=(xtqk,tqk,c)s_k=(x_t_q_k,t_q_k,c) and segment target 16: ak=(xtqk+1−xtqk)/(tqk+1−tqk)a_k=(x_t_q_k+1-x_t_q_k)/(t_q_k+1-t_q_k). 17: Compute wAMD=λ(Amix+b)w_ AMD=λ(A_ mix+b). 18: Compute ℓseg=‖vθ(sk)−sg(ak)‖2 _ seg=\|v_θ(s_k)-sg(a_k)\|^2. 19: Compute ℓema=‖vθ(sk)−sg(vθ¯(sk))‖2 _ ema=\|v_θ(s_k)-sg(v_ θ(s_k))\|^2. 20: Accumulate ℒstudentseg+=wAMDℓseg+βemaℓemaL_ student seg +=w_ AMD _ seg+ _ ema _ ema. 21:end for 22:Phase 4: Independent Optimization and EMA Update 23:Update only θ by minimizing ℒstudentsegL_ student seg. 24:ϕ¯←ρTϕ¯+(1−ρT)ϕ φ← _T φ+(1- _T)φ; θ¯←ρSθ¯+(1−ρS)θ θ← _S θ+(1- _S)θ. 25:end while 26:EMA student vθ¯v_ θ for K-step CFG-free inference. Preliminary: PCM-style phase consistency. The default REST variant above represents each student step by a single segment velocity between two selected teacher states. The PCM variant instead divides the fine-grained M-step teacher trajectory into K student phases and enforces consistency within each phase. The indices qkq_k and qk+1q_k+1 denote the teacher-grid columns at the beginning and end of student phase k, respectively. Within this phase, j∈qk,…,qk+1−1j∈\q_k,…,q_k+1-1\ indexes an adjacent pair of teacher states. Since the denoising schedule satisfies tj>tj+1t_j>t_j+1, xtjx_t_j is the higher-noise state and xtj+1x_t_j+1 is the immediately following lower-noise state. All adjacent pairs in phase k share the same endpoint time tqk+1t_q_k+1. Starting from the higher-noise state, the online student predicts the phase-end latent as x^k,jon=xtj+(tqk+1−tj)vθ(xtj,tj,c) x on_k,j=x_t_j+(t_q_k+1-t_j)v_θ(x_t_j,t_j,c). Here, the superscript “on” denotes the online, trainable student, and the hat indicates that this is a predicted endpoint latent rather than an observed teacher state. From the lower-noise state, the EMA student analogously produces x^k,jema x ema_k,j, where “ema” denotes the lagged student and the prediction is treated as a stop-gradient target. In these expressions, vθ(⋅,t,c)v_θ(·,t,c) is the conditional velocity predicted at time t for prompt c. PCM trains the two endpoint predictions to agree through the pseudo-Huber distance ℓpcmk,j _ pcm^k,j. Intuitively, regardless of which nearby state within a phase is used as the starting point, the student should predict the same phase endpoint. REST then applies the rollout-level AMD weight wAMD=λ(Amix+b)w_ AMD=λ(A_ mix+b) to this consistency loss, strengthening phase-consistency learning on preferred trajectories and weakening or reversing it on low-reward trajectories. Algorithm 2 REST with PCM-Style Phase Consistency and AMD 1:Online/EMA teacher vϕ,vϕ¯v_φ,v_ φ; online/EMA student vθ,vθ¯v_θ,v_ θ; rewards Rmm=1NR\R_m\_m=1^N_R. 2:Teacher steps M; phase boundaries 0=q0<⋯<qK=M0=q_0<·s<q_K=M; AMD parameters λ,b,αmλ,b,\ _m\; EMA coefficient βema _ ema; pseudo-Huber constant chuberc_ huber. 3:while not converged 4:Phase 1: Teacher Rollout and Multi-Reward Advantage 5:Sample prompt c and a full CFG-enabled trajectory =xt0,…,xtMT=\x_t_0,…,x_t_M\ with vϕ¯v_ φ. 6:Decode xtMx_t_M, compute all rewards, and obtain the fused advantage 7: Amix=∑mαmA(m)/∑mαmA_ mix= _m _mA^(m)/ _m _m after per-reward normalization and clipping. 8:Phase 2: Decoupled Teacher RL Update 9:Update ϕφ with its original RL objective; detach T from the teacher graph. 10:Phase 3: PCM+AMD Student Distillation 11:for k=0,…,K−1k=0,…,K-1 do 12: for j=qk,…,qk+1−1j=q_k,…,q_k+1-1 do 13: Set the common phase endpoint to tqk+1t_q_k+1. 14: Predict the endpoint from the higher-noise state xtjx_t_j with the online student: 15: x^k,jon=xtj+(tqk+1−tj)vθ(xtj,tj,c) x on_k,j=x_t_j+(t_q_k+1-t_j)v_θ(x_t_j,t_j,c). 16: Predict the same endpoint from the lower-noise state xtj+1x_t_j+1 with the EMA student: 17: x^k,jema=sg[xtj+1+(tqk+1−tj+1)vθ¯(xtj+1,tj+1,c)] x ema_k,j=sg[x_t_j+1+(t_q_k+1-t_j+1)v_ θ(x_t_j+1,t_j+1,c)]. 18: Compute ℓpcmk,j=‖x^k,jon−x^k,jema‖22+chuber2−chuber _ pcm^k,j= \| x on_k,j- x ema_k,j\|_2^2+c_ huber^2-c_ huber. 19: Compute wAMD=λ(Amix+b)w_ AMD=λ(A_ mix+b). 20: Compute ℓemak,j=‖vθ(xtj,tj,c)−sg(vθ¯(xtj,tj,c))‖2 _ ema^k,j=\|v_θ(x_t_j,t_j,c)-sg(v_ θ(x_t_j,t_j,c))\|^2. 21: Accumulate ℒstudentpcm+=wAMDℓpcmk,j+βemaℓemak,jL_ student pcm +=w_ AMD _ pcm^k,j+ _ ema _ ema^k,j. 22: end for 23:end for 24:Phase 4: Independent Optimization and EMA Update 25:Update only θ by minimizing ℒstudentpcmL_ student pcm over all phase-local pairs. 26:ϕ¯←ρTϕ¯+(1−ρT)ϕ φ← _T φ+(1- _T)φ; θ¯←ρSθ¯+(1−ρS)θ θ← _S θ+(1- _S)θ. 27:end while 28:EMA student vθ¯v_ θ for K-step CFG-free inference. Shared rollout and decoupled optimization. Both variants incur no additional image sampling or reward-model evaluation beyond the teacher RL pipeline. The trajectory and terminal rewards are collected once by the teacher and reused by both optimizers. Although the teacher is updated before the student in each training iteration, the student loss is evaluated on the stored pre-update trajectory and never backpropagates into the teacher. REST therefore preserves the teacher optimizer exactly and changes only how the student consumes its reward-scored rollout. Difference between the two student objectives. The default variant uses one finite-difference velocity target for each student phase. It directly teaches the student to traverse the complete phase in one step and requires K student targets per trajectory. PCM+AMD instead enumerates every adjacent teacher pair inside each phase. The online and EMA students start from different noise levels but are constrained to predict the same phase endpoint. If the selected student boundaries cover the complete M-step teacher trajectory, the PCM variant uses ∑k(qk+1−qk)=M _k(q_k+1-q_k)=M consistency pairs rather than K segment targets. This explains its higher forward/backward cost. Role of AMD and EMA. In both variants, AMD multiplies the per-sample base distillation loss; it does not modify the segment velocity or PCM endpoint target. Consequently, positive Amix+bA_ mix+b strengthens imitation or consistency on preferred trajectories, whereas a negative coefficient reverses the corresponding gradient and discourages low-reward trajectories. The shift b supplies a positive imitation prior during early training. The EMA student plays two related but distinct roles: in the default variant it is an explicit stability regularizer, while in PCM+AMD it additionally provides the stop-gradient phase-consistency target. The online student is the only student branch updated by backpropagation in either case. Additional qualitative results for visual text rendering and compositional generation are shown in Figs. 8 and 9, respectively. Figure 8: Additional qualitative results on visual text rendering. Figure 9: Additional qualitative results on compositional generation (GenEval task).