Paper deep dive
GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scales
Zhenxing Zhang, Jiayan Teng, Wenxu Wu, Zhuoyi Yang, Jiazheng Xu, Wendi Zheng, Jie Tang, Dan Guo, Meng Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) has demonstrated two important capabilities in language models: compressing large teachers into smaller students and merging expert models into a single model. Existing diffusion OPD, however, mostly focus on the latter, with teachers and students sharing the same backbone and scale. We investigate large-to-small diffusion opd from large teachers to a small student and find that the standard recipe fails. To find the underlying cause, we propose Fixed-State KL, an effective and fair way to measure the distribution gap between student and teacher during OPD training for diffusion models. We are the first to clarify why large-to-small OPD is challenging for diffusion models: a smaller student struggles to perfectly match the distribution of a larger teacher, while classifier-free guidance can accumulate and amplify the distributional discrepancies between the student's conditional and unconditional branches and those of the teacher. To solve this problem, we propose GFD-OPD, a simple yet effective method that reduces the student-teacher gap while avoiding the error amplification of the CFG composition. Across numerous experiments, GFD outperforms previous baselines in both training efficiency and final performance, achieving state-of-the-art results on all benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2609.39692v1
- Canonical: https://arxiv.org/abs/2609.39692v1
Trouble viewing inline? Open PDF directly ā
Full Text
47,460 characters extracted from source content.
Expand or collapse full text
GFD-OPD: Guidance-Folded On-Policy Distillation of Diffusion Models Across Scales Zhenxing Zhang, Jiayan Teng, Wenxu Wu, Zhuoyi Yang, Jiazheng Xu, Affiliation: Hefei University of Technology Tsinghua University Zhipu AI Wendi Zheng, Jie Tang, Dan Guo, Meng Wang Affiliation: Abstract On-policy distillation (OPD) has demonstrated two important capabilities in language models: compressing large teachers into smaller students and merging expert models into a single model. Existing diffusion OPD, however, mostly focus on the latter, with teachers and students sharing the same backbone and scale. We investigate large-to-small diffusion opd from large teachers to a small student and find that the standard recipe fails. To find the underlying cause, we propose Fixed-State KL, an effective and fair way to measure the distribution gap between student and teacher during OPD training for diffusion models. We are the first to clarify why large-to-small OPD is challenging for diffusion models: a smaller student struggles to perfectly match the distribution of a larger teacher, while classifier-free guidance can accumulate and amplify the distributional discrepancies between the studentās conditional and unconditional branches and those of the teacher. To solve this problem, we propose GFD-OPD, a simple yet effective method that reduces the studentāteacher gap while avoiding the error amplification of the CFG composition. Across numerous experiments, GFD outperforms previous baselines in both training efficiency and final performance, achieving state-of-the-art results on all benchmarks. The code and checkpoints for this study are available at here. 11footnotetext: Equal contribution.22footnotetext: Work done during internship at Zhipu AI.33footnotetext: Technical lead.44footnotetext: Corresponding author: eric.mengwang@gmail.com Figure 1: GFD-OPD distills SD3.5-Large into SD3.5-Medium. (a) Score versus training compute (GPU hours) on OCR, PickScore, and GenEval: GFD-OPD overtakes all baselines within 1010 GPU-hours. (b) GFD-OPD ranks first on all seven evaluation dimensions, covering both task rewards and image quality. 1 Introduction On-policy distillation (OPD) has recently emerged as a powerful post-training recipe for large language models. Like reinforcement learning (RL), OPD trains the student on trajectories sampled from its own policy and therefore optimizes the studentās own distribution; unlike RL, which has to work with sparse and delayed rewards, OPD lets a teacher provide dense supervision at every step of the student-generated trajectory. This fine-grained feedback largely removes the credit-assignment difficulty of RL, so OPD enjoys a more stable optimization and a much better sample efficiency at a comparable final quality. In language models OPD is primarily applied in two paradigms. The first is strong-to-weak distillation, where the capability of a large teacher is transferred to a small student that is cheap to deploy (Lu and Lab, 2025). The second is capability merging, where several specialist models derived from the same base model are merged into a single student without conflicts (Zeng et al., 2026). Diffusion models have begun to adopt OPD as a post-training method as well (Li et al., 2026b; Fang et al., 2026), but the existing diffusion work mostly focus on the second paradigm. The first paradigm, distilling a large diffusion model into a small one, remains unexplored. It is also important: a compact model that inherits capabilities of a larger teacher can achieve near-teacher quality at inference while retaining a low cost close to that of the smaller model. Our goal is to investigate effective approaches for distilling the capabilities of large diffusion models into smaller ones. Although DiffusionOPD performs effectively when merging models with the same architecture, we find directly extending it to large-to-small distillation leads to severe degradation. As shown in Fig. 4, generated images exhibit artifacts, such as corrupted regions, loss of fine-grained textures and so on. To understand the source of this degradation, we propose Fixed-State KL for measuring the distributional discrepancy between the student and teacher. Specifically, we compute KL divergence on states sampled from normal teacher trajectories and noise-corrupted trajectories constructed from real images, providing a fair and well-behaved evaluation distribution for all methods. Under this evaluation metric, we find that large-to-small DiffusionOPD produces a substantially larger distributional discrepancy between the student and teacher than its same-architecture counterpart. Furthermore, we quantitatively characterize modelsā final sampling result through spectral energy analysis of the final latent representations and high-pass texture analysis of the decoded images, which quantify the deviation from the teacher at both coarse-grained (e.g., global tone) and fine-grained (e.g., local texture) levels. Through extensive analysis, we indicate that a smaller-capacity student consistently struggles to perfectly match the distribution of a larger teacher. Besides, classifier-free guidance (CFG), used during diffusion sampling, can accumulate and amplify the distributional discrepancies between the studentās conditional and unconditional branches and those of the teacher, causing a larger deviation in the final generation trajectory. Motivated by these observations, we propose a simple yet effective method, Guidance Folding Distillation (GFD). GFD reduces the distributional discrepancy between the student and teacher while avoiding the error amplification introduced by CFG composition, enabling effective large-to-small OPD for diffusion models. Our contributions are as follows. ā¢ We propose Fixed-State KL, an effective and fair way to measure the distribution gap between student and teacher during OPD training for diffusion models. ā¢ Through multifaceted analysis, we are the first to clarify why large-to-small OPD is challenging for diffusion models. To address these challenges, we propose GFD, a simple and effective method that reduces the studentāteacher gap while avoiding the error amplification of the CFG composition. ā¢ Through multiple experiments, combined with quantitative and qualitative analysis, we show the effectiveness of GFD. GFD removes the artifacts in cross-scale DiffusionOPD and outperforms previous baselines in both training efficiency and final performance. 2 method 2.1 Preliminary: On-Policy Distillation in Language Models and Diffusion Models On-policy distillation (OPD) trains the student on trajectories sampled from its own policy, while the teacher provides dense supervision on every visited state. In language models, OPD minimizes the reverse KL between the student policy Ļ _ Īø and the frozen teacher ĻāĻ along student-generated sequences: āOPDLLM()=ā¼Ļ[āt=1||KL(Ļ(ā ā£<t)ā„Ļā(ā ā£<t))].L_OPD^LLM( Īø)=E_ y _ Īø [ _t=1^| y|KL ( _ Īø(Ā· y_<t)\,\|\,Ļ (Ā· y_<t) ) ]. (1) DiffusionOPD transfers the same idea to diffusion and flow-matching samplers. At each denoising step j, the student and teacher induce Gaussian transition kernels pSā(xtj+1ā£xtj)p_S(x_t_j+1 x_t_j) and pTā(xtj+1ā£xtj)p_T(x_t_j+1 x_t_j) whose covariance Ļj2ā _j^2 I is fixed by the shared sampler, so the per-step reverse KL has the closed form āĪ¼Sā(xtj,)āĪ¼Tā(xtj)ā22/(2āĻj2)\| _S(x_t_j; Īø)- _T(x_t_j)\|_2^2/(2 _j^2). Since the transition mean is affine in the velocity field with sampler-fixed coefficients, and sampling deploys the CFG-guided velocity, DiffusionOPD matches the CFG-guided output to the teacherās: āOPDdiff()=x0:Nā¼pS,[āj=0Nā1Ļjā„vSg(xtj,tj,)āvTg(xtj,tj,)ā„22],L_OPD^diff( Īø)=E_x_0:N p_S, Īø [ _j=0^N-1 _j \|v^g_S(x_t_j,t_j, c)-v^g_T(x_t_j,t_j, c) \|_2^2 ], (2) where vāg=vāu+wā”(vācāvāu)v^g_ =v^u_ +w\,(v^c_ -v^u_ ), with āāS,T \!ā\!\S,T\ indexing the student and the teacher, is the guided velocity composed from the conditional and unconditional branches vāc,vāuv^c_ ,v^u_ at guidance scale w, and Ļj=Ī·j2/(2āĻj2) _j= _j^2/(2 _j^2) is fixed by the sampler, with Ī·j _j the coefficient of the velocity in the transition mean and Ļj2 _j^2 the per-step variance; Ī·j=(1+Ļtj2ā(1ātj)/(2ātj))āĪātj _j= (1+ _t_j^2(1-t_j)/(2t_j) ) t_j and Ļj2=Ļtj2āĪātj _j^2= _t_j^2 t_j. In both SDE and ODE settings, DiffusionOPD is thus a pathwise, closed-form velocity-matching problem along the studentās rollout. 2.2 Fixed-state KL Evaluation We first directly apply the standard DiffusionOPD objective to the large-to-small OPD setting. However, after training, from Fig. 4 we can see that the resulting images exhibit obvious visual artifacts: they contain corrupted textures and are generally darker. To further quantify the degradation in generation quality, we conduct a spectral energy analysis on the final latent representations and a high-pass texture analysis on the decoded images. As shown in Fig. 6, DiffusionOPDLāM deviates substantially from the teacher across all evaluated aspects. To investigate the underlying cause, we need to measure the distributional discrepancy between the student and teacher. A natural choice is to use the objective in OPD training: the reverse KL divergence between the student and teacher distributions, computed along the trajectory rolled out by the student. However, although this formulation is well suited as a training objective, we argue that it is not an appropriate metric to evaluate the distributional discrepancy between the student and the teacher after training. There are two main reasons. First, different training methods generally produce different student policies. Computing the KL divergence on each studentās own trajectories therefore evaluates the models on different sets of states, confounding the effect of distributional mismatch with differences in state visitation. It makes the resulting metrics difficult to compare in a fair and controlled manner. Second, the student model may deviate from the normal generation trajectory during training and enter abnormal states. As pointed out by Xin et al. (2026), the teacher is no longer guaranteed to provide reliable predictions in such states, since the queried states may lie far outside the region on which the teacherās behavior is meaningful. Consequently, the KL divergence computed on these abnormal states may no longer faithfully reflect the discrepancy between a well-behaved student and the teacher. Figure 2: Illustration of the KL evaluation. Table 1: KL divergence between each student and its corresponding teacher. PstuP_stu and PteaP_tea denote sampling trajectories generated by the student and teacher models, respectively, while PrealP_real denotes trajectories constructed by adding noise to real images. KL (PteaP_tea) KL (PrealP_real) KL (PstuP_stu) branch KL (PteaP_tea) Student guided guided guided cond uncond DiffusionOPDMāM 0.041 0.022 0.039 0.017 0.023 DiffusionOPDLāM 0.068 0.046 0.057 0.037 0.039 PDMLāM 0.063 0.046 0.052 0.023 0.022 split-KLLāM 0.074 0.049 0.065 0.021 0.020 GFDLāM (w=1w\!=\!1) 0.050 0.030 0.043 ā ā GFDLāM (w=2w\!=\!2) 0.053 0.032 0.053 ā ā Motivated by these considerations, we propose Fixed-State KL, which is performed on a fixed and normal set of states shared by all student models. Specifically, we use states from normal teacher trajectories, together with noise-corrupted trajectories constructed from real images, as common evaluation inputs for computing the KL divergence between the student and teacher. By decoupling the evaluation states from the studentās own rollout distribution, this protocol enables a more controlled and comparable assessment of how closely different students match the teacher distribution. Furthermore, as shown in Tab. 2, the KL divergence evaluated on student rollout trajectories exhibits a markedly different trend from that computed on teacher rollout trajectories and noise-corrupted trajectories constructed from real images. The former fails to reflect the true trend of studentāteacher distributional discrepancy. It may provide a misleading evaluation signal. 2.3 From Cross-scale DiffusionOPD to GFD Based on the KL evaluation above, we can figure out why the performance of DiffusionOPD in LāML\!ā\!M setting degrade severely. As shown in Table 2, the guided-output KL, conditional KL, and unconditional KL of DiffusionOPDLāM are all higher than those of DiffusionOPDMāM. āguided outputā refers to the CFG-combined output used during sampling. Accordingly, the final evaluation should focus on the distributional KL divergence between the studentās guided output and the teacherās guided output. Figure 3: GFD-OPD overview. Left (motivation): the deployed error of a CFG-composed student decomposes as Ī“g=Ī“u+Ļād _g= _u+Ļ\,d ā same-size distillation survives through error cancellation, large-to-small does not, and guidance folding removes the composition so that nothing is left for Ļ to amplify. Right (method): the process of GFD-OPD. Attempting branch-wise supervision Recall that DiffusionOPD objective directly matches the studentās guided prediction vSgv^g_S to the teacherās guided prediction vTgv^g_T. It lacks separately constraining of the conditional and unconditional branches. Such an objective may be sufficient when the student has comparable model capacity to the teacher, as in the MāM\!ā\!M setting. However, in a large-to-small setting, the smaller student has limited approximation capacity, which may make the conditional and unconditional branches more prone to drifting away from their teacher counterparts. Such branch-wise drift may in turn lead to a larger discrepancy in the final guided output. This observation motivates us to explicitly supervise the conditional and unconditional branches separately, thereby preventing them from drifting. A natural choice is to adopt the following training objective: ā=āvScāvTcā22+āvSuāvTuā22,L= \|v_S^c-v_T^c \|_2^2+ \|v_S^u-v_T^u \|_2^2, (3) PDM (Li et al., 2026a) also propose a similar objective as ā=āvScāvTcā22+Ī±āāvSgāvTgā22,L= \|v_S^c-v_T^c \|_2^2+Ī± \|v_S^g-v_T^g \|_2^2, (4) However, their motivation is fundamentally different from ours: we use this objective to prevent the conditional and unconditional branches from drifting in the large-to-small setting but theirs are not. After training with separate supervision on the two branches, from Table 2, we observe that the KL divergences of both the conditional and unconditional branches are indeed reduced. However, the KL divergence of the final guided output does not decrease accordingly. Consistent with this observation, the generated images in Fig. 4 still exhibit noticeable quality degradation. As shown in Fig. 6, the corresponding quantitative metrics also remain substantially deviated from those of the teacher. Why the branch error induce but the guided field does not? Figure 4: Shared cross-architecture artifacts under a SD3.5-L teacher. Rows: the SD3.5-M base, three prior OPD recipes (DiffusionOPD, PDM, splitKL), GFD and large teacher. To investigate the cause of the increased guided-output KL, we decompose the error of the guided prediction as Ī“g=Ī“u+wād,Ī“^g=Ī“^u+wd, (5) where Ī“g=vSgāvTg,Ī“u=vSuāvTu,Ī“c=vScāvTc,Ī“^g=v^g_S-v^g_T,Ī“^u=v^u_S-v^u_T,Ī“^c=v^c_S-v^c_T, and d=Ī“cāĪ“u.d=Ī“^c-Ī“^u. Table 6 reveals that, when trained to directly match the guided output, DiffusionOPDMāM is able to exploit the cancellation between the two error components, Ī“uĪ“^u and wādwd, resulting in a relatively small final guided error Ī“gĪ“^g. In contrast, for DiffusionOPDLāM, the limited approximation capacity of the smaller student weakens such error cancellation. For PDM and SplitKL, separately supervising the conditional and unconditional branches largely removes this cancellation effect, causing the errors to be amplified rather than canceled after guidance composition. We provide a more detailed analysis in Section B.1. This observation further suggests that, in the large-to-small setting, discrepancies in the conditional and unconditional branches can be systematically amplified by the guidance operation, ultimately leading to a larger shift in the final generative distribution. Guidance Folding for Large-to-Small Distillation This motivates a different design: directly absorb the teacherās guided policy into the studentās conditional branch, eliminating the need to recover the target distribution through the composition of two imperfectly matched branches. We propose Guidance Folding Distillation (GFD). Rather than jointly optimizing the studentās conditional and unconditional distributions, it converts the original two-branch composition problem into a single-branch velocity matching problem. Formally, GFD minimize the discrepancy between the studentās conditional distribution and the teacherās guided distribution: minĪøxtā¼dS[DKL(ĻSc(ā ā£xt,c)ā„ĻTg(ā ā£xt,c))]. _ĪøE_x_t d^S [D_KL (Ļ^c_S(Ā· x_t,c)\,\|\,Ļ^g_T(Ā· x_t,c) ) ]. (6) Applying the per-step derivation of Eq. 2, the GFD objective can be written explicitly as āGFD()=x0:Nā¼pS,[āj=0Nā1Ļjā„vSc(xtj,tj,;)āvTg(xtj,tj;w)ā„22].L_GFD( Īø)=E_x_0:N p_S, Īø [ _j=0^N-1 _j \|v_S^c(x_t_j,t_j, c; Īø)-v_T^g(x_t_j,t_j;w) \|_2^2 ]. (7) Table 2 shows that GFD achieves substantially lower KL divergence than the other methods. As illustrated in Fig. 4 and Fig. 6, GFD also restores normal generation quality, and the generated images are closer to those of the teacher in both coarse-grained and fine-grained metrics. But a small, systematic gap remains. Fig. 6 shows that GFD (w=1w\!=\!1) has slightly weaker spectral energy than the teacherās guided output. And Appendix B.2 shows that the student learns the direction of the teacherās field but shrinks its magnitude toward its own prior. At inference this can be patched with a slightly larger guidance scale, with w=2w\!=\!2 the student distribution moves closer to the teacherās. GFD does not constrain the unconditional branch during training. In the low-noise regime, the unconditional branch co-evolves with the conditional branch and exhibits similar shifts, resulting in the CFG only containing a small increment and avoiding excessive guidance. Although the trajectory-level KL increases slightly at w=2w\!=\!2, we attribute this to the limited sensitivity of this metric to differences at such a small scale. 2.4 Reward Extrapolation for Large-to-Small Distillation Yang et al. (2026) show that OPD in language models is a KL-regularized RL objective in which the teacher enters through a dense reward, and that scaling this reward by a factor Ī»>1Ī»>1 moves the distillation target away from the reference model and beyond the teacher. We adopt this idea to further improve the student: by extrapolating the target in the direction away from the studentās prior, the trained field is brought closer to the teacher without relying on an inference-time guidance increment. In language models, the ExOPD objective is equivalent to standard OPD with the teacher replaced by the target Ļtgtā(Ļā)Ī»ā(Ļref)1āĪ» _tgt (Ļ )^Ī»( _ref)^1-Ī», where Ļref _ref is a frozen reference policy. We apply the same replacement to the per-step transition kernels of diffusion OPD: ptgtāpTĪ»āpR1āĪ»p_tgt p_T^Ī»\,p_R^1-Ī», where pRp_R is the kernel induced by the reference velocity vRv_R. Since both kernels are Gaussians with the same sampler-fixed variance, ptgtp_tgt remains a valid Gaussian kernel with that variance for every Ī», and only its mean moves, corresponding to the extrapolated velocity (Appendix A.1): vtgt=Ī»āvT+(1āĪ»)āvR=vT+(Ī»ā1)ā(vTāvR).v_tgt=Ī»\,v_T+(1-Ī»)\,v_R=v_T+(Ī»-1)(v_T-v_R). (8) 3 Experiments 3.1 Experimental Setup Task rewards Image quality Model GenEval OCR Pick Score Aes- thetic Image Reward Unified Reward HPSv3 SD3.5-M (base) 0.675 0.554 0.843 5.373 0.907 3.201 9.020 SD3.5-L (base) 0.720 0.716 0.854 5.527 1.025 3.289 9.853 Per-task Large teachers GenEval-Teacher 0.949 0.801 0.834 5.463 1.144 3.416 9.812 OCR-Teacher 0.700 0.969 0.850 5.405 1.057 3.298 8.982 PickScore-Teacher 0.781 0.688 0.936 6.245 1.300 3.535 11.29 Per-task Medium teachers GenEval-Teacher 0.917 0.558 0.845 5.336 1.074 3.332 9.024 OCR-Teacher 0.599 0.948 0.824 5.129 0.749 3.071 6.345 PickScore-Teacher 0.696 0.605 0.924 6.188 1.283 3.460 11.55 DiffusionOPDMāM 0.908 0.944 0.917 6.067 1.275 3.461 11.07 PDMMāM 0.902 0.940 0.913 6.124 1.261 3.455 10.97 DiffusionOPDLāM 0.943 0.962 0.901 6.222 1.009 3.365 8.844 PDMLāM 0.944 0.955 0.906 6.211 1.065 3.396 9.360 split-KLLāM 0.939 0.959 0.912 6.211 1.103 3.417 10.08 GFDLāM(Ours) 0.963 0.973 0.930 6.262 1.302 3.542 11.27 Table 2: SD3.5 merged multi-objective OPD results. Table 3: SD3.5 single-objective OPD results. Task rewards Image quality Model GenEval OCR Pick Score HPSv3 SD3.5-Medium (base) 0.675 0.554 0.843 9.020 Teachers (SD3.5-L) 0.949 0.969 0.936 ā DiffusionOPDLāM GenEval 0.948 ā ā 6.844 OCR ā 0.967 ā 6.097 PickScore ā ā 0.904 8.377 GFD LāM(Ours) GenEval 0.963 ā ā 9.128 OCR ā 0.973 ā 8.755 PickScore ā ā 0.932 11.24 Table 4: FLUX.2 merged OPD results. Task rewards Image quality Model LongText GenEval Pick Score HPSv3 FLUX.2-4B (base) 0.624 0.769 0.823 9.214 Teacher (FLUX.2-9B) 0.891 0.820 0.839 9.629 PDMLāM 0.635 0.815 0.829 8.904 DiffusionOPDLāM 0.635 0.830 0.832 9.180 GFDLāM(Ours) 0.692 0.880 0.839 10.20 Teacher (FLUX.2-32B) 0.979 0.893 0.862 11.29 PDMLāM 0.763 0.887 0.849 10.17 DiffusionOPDLāM 0.828 0.885 0.856 10.69 GFDLāM(Ours) 0.857 0.908 0.859 10.47 Tasks and evaluation. We follow the three-reward suite of Flow-GRPO (Liu et al., 2026) and reuse its official prompt splits: GenEval (Ghosh et al., 2023) for compositional alignment, OCR accuracy for visual text rendering, and PickScore (Kirstain et al., 2023) optimized on Pick-a-Pic prompts and evaluated on DrawBench (Saharia et al., 2022). Task rewards are the primary metrics; following Flow-GRPO, we also report Aesthetic, ImageReward (Xu et al., 2023), UnifiedReward, and HPSv3 on DrawBench as image-quality probes. All methods are evaluated at 1024Ć10241024Ć 1024 with the same prompts, seeds, and sampler, each at its own deployment guidance scale (w=4.5w=4.5 for CFG-composition students, w=2w=2 for GFD-OPD; Ā§2). Baselines. (a) The pretrained SD3.5-Medium and SD3.5-Large base models (Esser et al., 2024); (b) direct Flow-GRPO on both, which also serve as our teachers; (c) DiffusionOPD, PDM, and the split-KL variant of Ā§2, trained with the same teachers, student initialization, and prompts as ours. Implementation details. All OPD methods share the same on-policy setup: following Fast Flow-GRPO, rollouts use a 10-step reverse-SDE sampler with a 2-step training window. Training uses full-parameter fine-tuning at 1024Ć10241024Ć 1024 with learning rate 5Ć10ā65Ć 10^-6 and batch size 48 on 8 H100 GPUs; each single-task GFD-OPD run finishes in about 10 GPU-hours, versus 15 for DiffusionOPD and 160 for direct Flow-GRPO (Fig. 1). GFD-OPD composes the folded target at w=2w=2 with extrapolation Ī»=1.25Ī»=1.25. 3.2 Main Results Large-to-small distillation on SD3.5 and FLUX.2. Table 4 reports the single-objective SD3.5 setting (Largeā ). GFD-OPD outperforms DiffusionOPD on all three rewards, surpasses the teacher on GenEval (0.9630.963 vs. 0.9490.949) and OCR (0.9730.973 vs. 0.9690.969), and is the only student whose HPSv3 stays at or above the untrained baseāDiffusionOPD falls clearly below it, the fidelity loss behind the artifacts of Ā§2. In the merged multi-objective setting (Table 4), the baselines trade one side for the other: with the medium teacher they preserve image quality but are capped by its rewards; with the large teacher their rewards rise but quality drops (haze, desaturation, soft glyphs; Fig. 4). GFD-OPD attains the best task rewards (0.9630.963 GenEval, 0.9730.973 OCR, 0.9300.930 PickScore) and the best image quality at the same time; per Ā§3.3, folding alone roughly matches the teacher and the extrapolated target accounts for the rest. The same picture holds on FLUX.2 (Table 4; 9B and 32B teachers into 4B, OCR replaced by our long-text dataset): GFD-OPD leads on all task rewards, again exceeds the teacher on GenEval, and widens its LongText margin at 32B (qualitative examples in Fig. 8, appendix). Figure 5: Qualitative comparison at Ć10241024\!Ć\!1024. Columns, left to right: SD3.5-M base; the medium and large teachers; DiffusionOPDMāM; and GFDLāM. Qualitative comparison. Fig. 5 presents a qualitative comparison. Since the large-teacher composition students already suffer from the artifacts shown in Fig. 4, we compare with DiffusionOPD distilled from the medium teacher, together with the base model and the two RL teachers. These medium-teacher students are free of visible artifacts, but their prompt adherence and fine-grained rendering are weaker: the crop-circle text in āHumans LOLā loses glyph structure, and the orange scene misses part of the required objects. Among all compared methods, GFD-OPD is the closest to the large teacher in both layout and fine detail. Besides, Fig. 8 shows qualitative examples for the FLUX.2 experiments of Table 4. 3.3 Ablations Component ablation Table 5 adds the two components on top of standard OPD one at a time. Standard OPD improves the base on all three rewards but stays well below the teacher. Guidance folding contributes most of the gain, removing the artifacts of and bringing the student roughly to the teacher level; the extrapolated target adds a further consistent gain and pushes GenEval and OCR past the teacher. This supports the attribution: folding closes the gap to the teacher, and extrapolation provides the margin beyond it. Table 5: Ablation: standard OPD ++ guidance folding ++ extrapolated-mean target (full GFD-OPD), on the SD3.5-Medium student at 1024Ć10241024Ć 1024; HPSv3 probes image quality. Method GenEval OCR PickScore HPSv3 SD3.5-Medium (base) 0.675 0.554 0.843 9.020 Standard OPD 0.884 0.908 0.880 9.175 ++ Guidance folding 0.952 0.954 0.923 11.19 ++ Extrapolated mean (GFD-OPD) 0.963 0.973 0.930 11.27 The impact of CFG scale In Fig. 6, we evaluate different models under varying CFG scales w, measuring the discrepancy between their final sampling results and those of their respective teacher models. DiffusionOPDMāM approaches the origin at w=4.5w=4.5, indicating a close match to its teacher. In contrast, for DiffusionOPD, PDM, and Split-KL under the LāML\!ā\!M setting, no choice of w brings the results close to the origin. Their outputs remain substantially different from those of the teacher. GFD achieves the closest match to the teacher at w=2w=2. Figure 6: Statistics of the studentās final output as the guidance scale w varies. Each axis is the log ratio of a statistic of the studentās samples to the teacherās, so the teacher sits at the origin (ā ). (a) Final latent: low-frequency energy (global tone) vs. high-frequency energy (fine texture). (b) Decoded image: tonal amplitude (RMS contrast Ć saturation) vs. fine-texture amplitude (3px high-pass MAE). The definition of all metrics are as detailed in Appendix B.3. 4 Related Work On-Policy Distillation. OPD (Agarwal et al., 2024; Gu et al., 2024; Lu and Lab, 2025) post-trains language models by aligning the student with the teacher on student-generated trajectories, and is used both to compress large teachers into small students and to merge domain specialists into a single model. Yang et al. (2026) further interpret OPD as dense KL-constrained RL, we adopt this view in Ā§2.4. On-Policy Distillation for Diffusion Models. A recent line of work ports OPD to diffusion and flow-matching models (Ma et al., 2026; Fang et al., 2026; Wei et al., 2026). DiffusionOPD (Li et al., 2026b) derives the closed-form per-step objective and merges task-specific teachers into a unified student; PDM (Li et al., 2026a) supervises the conditional and unconditional CFG branches separately. These methods target same-family merging, while the large-to-small regime we study remains largely unexplored. 5 Conclusion We studied on-policy distillation of diffusion models in the large-to-small regime and showed that the existing recipe does not transfer. To diagnose this we propose Fixed-State KL, which fairly makes methods comparable. The analysis attributes the failure to the CFG composition: a small student cannot fit the teacherās guided field exactly, the composition amplifies the error of its guidance increment by the guidance scale. GFD-OPD directly absorb the teacherās guided policy into the studentās conditional branch. It reduces the studentāteacher gap while avoiding the error amplification of the CFG composition. We hope this study can encourage further investigation into how the distributional discrepancy between teacher and student affect OPD of diffusion models. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246ā21263. Cited by: Ā§4. Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. MĆ¼ller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: Ā§3.1. Fang et al. (2026) Z. Fang, W. Huang, Y. Zeng, Y. Zhao, S. Chen, K. Feng, Y. Lin, L. Chen, Z. Chen, S. Cao, et al. Flow-opd: on-policy distillation for flow matching models. arXiv preprint arXiv:2605.08063. Cited by: Ā§1, Ā§4. Ghosh et al. (2023) D. Ghosh, H. Hajishirzi, and L. Schmidt Geneval: an object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems 36, p. 52132ā52152. Cited by: Ā§3.1. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang Minillm: knowledge distillation of large language models. In International Conference on Learning Representations, Vol. 2024, p. 32694ā32717. Cited by: Ā§4. Kirstain et al. (2023) Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy Pick-a-pic: an open dataset of user preferences for text-to-image generation. Advances in neural information processing systems 36, p. 36652ā36663. Cited by: Ā§3.1. Li et al. (2026a) B. Li, H. Wang, H. Xiong, F. Wu, J. Yu, Y. Shi, J. Liu, and R. Huang Rethinking classifier-free guidance in on-policy diffusion distillation. arXiv preprint arXiv:2607.24731. Cited by: Ā§2.3, Ā§4. Li et al. (2026b) Q. Li, J. Yu, K. Jiang, Y. Wei, Z. Xing, P. Li, R. Chu, S. Zhang, Y. Liu, and Z. Wu DiffusionOPD: a unified perspective of on-policy distillation in diffusion models. arXiv preprint arXiv:2605.15055. Cited by: Ā§1, Ā§4. Liu et al. (2026) J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang Flow-grpo: training flow matching models via online rl. Advances in neural information processing systems 38, p. 40783ā40818. Cited by: Ā§3.1. Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: Ā§1, Ā§4. Ma et al. (2026) W. Ma, J. Wei, L. Zhao, H. Zhang, B. Xiao, L. Li, Q. Yang, B. Gao, Y. Wang, R. Li, et al. Mopd: multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406. Cited by: Ā§4. Saharia et al. (2022) C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems 35, p. 36479ā36494. Cited by: Ā§3.1. Wei et al. (2026) Q. Wei, G. Li, X. Tu, Y. Wang, X. Zhang, J. Ren, X. Liu, and L. Zhang STEP-opd: rethinking output targets and internal dynamics in on-policy distillation for diffusion models. arXiv preprint arXiv:2608.04887. Cited by: Ā§4. Xin et al. (2026) H. Xin, A. Zhao, Y. Sun, J. Li, X. Shen, and H. Xiong Escaping the kl agreement trap in on-policy distillation. arXiv preprint arXiv:2606.09471. Cited by: Ā§2.2. Xu et al. (2023) J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong Imagereward: learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36, p. 15903ā15935. Cited by: Ā§3.1. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: Ā§A.1, Ā§2.4, Ā§4. Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: Ā§1. Appendix A Derivation A.1 Reward extrapolation and the extrapolated target Reward extrapolation in language models. Yang et al. (2026) generalize the OPD objective (Eq. 1) by introducing a frozen reference policy Ļref _ref and a reward weight Ī»: J()=ā¼Ļ(ā |)[Ī»logĻāā(|)Ļrefā(|)]āKL(Ļ(ā |)ā„Ļref(ā |)).J( Īø)=E_ y _ Īø(Ā·| x) [Ī» Ļ ( y| x) _ref( y| x) ]-KL ( _ Īø(Ā·| x)\,\|\, _ref(Ā·| x) ). (9) Standard OPD is the case Ī»=1Ī»=1 (the reference cancels). Define Ļtgtā(|)=(Ļā)Ī»ā(Ļref)1āĪ»/Zā”() _tgt( y| x)=(Ļ )^Ī»( _ref)^1-Ī»/Z( x) with Zā”()=ā(Ļā)Ī»ā(Ļref)1āĪ»Z( x)= _ y(Ļ )^Ī»( _ref)^1-Ī». Expanding the KL term, Jā”() J( Īø) =Ļā[Ī»ālogā”ĻāāĪ»ālogā”Ļrefālogā”Ļ+logā”Ļref] =E_ _ Īø [Ī» Ļ -Ī» _ref- _ Īø+ _ref ] =Ļ[Ī»logĻā+(1āĪ»)logĻrefālogĻ]=āKL(Ļā„Ļtgt)+logZ(). =E_ _ Īø [Ī» Ļ +(1-Ī») _ref- _ Īø ]=-\,KL ( _ Īø\,\|\, _tgt )+ Z( x). (10) Since Zā”()Z( x) does not depend on Īø, maximizing J is the same as minimizing the reverse KL to Ļtgt _tgt; that is, ExOPD is standard OPD with the teacher replaced by Ļtgt _tgt, where logā”Ļtgt=logā”Ļā+(Ī»ā1)ā(logā”Ļāālogā”Ļref)+const. _tgt= Ļ +(Ī»-1) ( Ļ - _ref )+const. (11) For 0<Ī»<10<Ī»<1 the target interpolates between reference and teacher; for Ī»>1Ī»>1 it extrapolates along the referenceā direction. Diffusion. At denoising step j, the teacher and reference kernels are pT=ā”(T,Ļj2ā)p_T=N( Ī¼_T, _j^2 I) and pR=ā”(R,Ļj2ā)p_R=N( Ī¼_R, _j^2 I) with the same sampler-fixed variance (Ā§2.1). Applying the replacement above to the kernels gives ptgtāpTĪ»āpR1āĪ»p_tgt p_T^Ī»p_R^1-Ī». Lemma 1. Let p1=ā”(1,Ļ2ā)p_1=N( Ī¼_1,Ļ^2 I) and p2=ā”(2,Ļ2ā)p_2=N( Ī¼_2,Ļ^2 I). For every Ī»āāĪ» , the normalized product pāp1Ī»āp21āĪ»p p_1^Ī»p_2^1-Ī» equals ā”(Ī»ā1+(1āĪ»)ā2,Ļ2ā)N (Ī» Ī¼_1+(1-Ī») Ī¼_2,\ Ļ^2 I ). Proof. logā”pā”()ā=āĪ»2āĻ2āāā1ā22ā1āĪ»2āĻ2āāā2ā22 p( x) c=- Ī»2Ļ^2\| x- Ī¼_1\|_2^2- 1-Ī»2Ļ^2\| x- Ī¼_2\|_2^2. The coefficient of ā22\| x\|_2^2 is āĪ»+(1āĪ»)2āĻ2=ā12āĻ2- Ī»+(1-Ī»)2Ļ^2=- 12Ļ^2, independent of Ī», so the covariance is Ļ2āĻ^2 I; the linear term is 1Ļ2āāØ,Ī»ā1+(1āĪ»)ā2ā© 1Ļ^2 x,Ī» Ī¼_1+(1-Ī») Ī¼_2 , and completing the square gives the mean. ā By Lemma 1, ptgt=ā”(Ī»āT+(1āĪ»)āR,Ļj2ā)p_tgt=N (Ī» Ī¼_T+(1-Ī») Ī¼_R,\ _j^2 I ): extrapolation changes the mean only, and ptgtp_tgt is a valid transition kernel for every Ī», including Ī»>1Ī»>1. Because the transition mean is affine in the velocity with sampler-fixed coefficients, =Ī±jātj+Ī²jāv Ī¼= _j x_t_j+ _jv, the target mean corresponds to vtgt=Ī»āvT+(1āĪ»)āvRv_tgt=Ī» v_T+(1-Ī»)v_R (Eq. 8). The reverse KL between the student kernel and ptgtp_tgt is a KL between Gaussians with equal covariance, āSātgtā22/(2āĻj2)\| Ī¼_S- Ī¼_tgt\|_2^2/(2 _j^2), which in velocity space is the weighted matching loss of Eq. 2 with the target velocity vtgtv_tgt in place of vTgv^g_T. Appendix B Detailed Analysis B.1 Analysis of error cancellation The two error components. Given the guided output vg=vu+wā”(vcāvu),v_g=v_u+w(v_c-v_u), the guided prediction error can be written as Ī“g=Ī“u+wād,Ī“^g=Ī“^u+wd, where d=Ī“cāĪ“u.d= _c- _u. Alternatively, d can be expressed as d=ĪSāĪT,d= _S- _T, where ĪS=vScāvSu,ĪT=vTcāvTu. _S=v_S^c-v_S^u, _T=v_T^c-v_T^u. These two error components have distinct meanings. Specifically, Ī“uĪ“^u represents the prompt-independent denoising prediction error, while d captures the error in the prompt-induced increment. Table 6: Decomposition g=u+wā Ī“^g= Ī“^u+w d of the guided error at w=4.5w\!=\!4.5. Cross term=2āāØu,4.5āā©/ā2=2 Ī“^u,4.5 d /\| r\|^2. Student āuā\| Ī“^u\| ā\| d\| ā\| r\| cross term DiffusionOPDMāM (reference) 0.092 0.032 0.106 ā1.56 -1.56 DiffusionOPDLāM 0.090 0.043 0.177 ā0.48-0.48 PDMLāM 0.053 0.041 0.178 ā0.18-0.18 split-KLLāM 0.051 0.043 0.185 ā0.18-0.18 Cancellation in MāMMā M and its loss in LāMLā M. The guided objective only constrains the sum of the two components, Ī“u+wādĪ“^u+wd. Therefore, any pair of errors satisfying d=āĪ“u/wd=-Ī“^u/w produces the same guided prediction error and lies in the null space of the objective. Consequently, minimizing this objective does not necessarily force both error components to vanish simultaneously. Instead, it allows the student to learn a negative correlation between d and Ī“uĪ“^u, enabling one component to compensate for the other. DiffusionOPDMāMDiffusionOPD_Mā M nearly reaches such a configuration, where the two error components Ī“uĪ“^u and d substantially cancel each other. For DiffusionOPDLāMDiffusionOPD_Lā M, besides a similar shrinkage effect, the error in d contains additional components that are orthogonal to Ī“uĪ“^u, resulting in weaker error cancellation. PDM and Split-KL explicitly supervise the two branches separately. Although this reduces the magnitude of the unconditional error component Ī“uĪ“^u, it also makes the two error components nearly orthogonal to each other. As a result, the cancellation effect is largely removed, and the final guided error remains substantial. B.2 Shrinkage of the trained field along the teacherās guided field Tab. 7 quantifies the residual shrinkage of the guidance-folded student and how the inference-time guidance increment compensates it. At every step we project the studentās deployed field onto the teacherās guided field, g=āØSā(w),Tcfgā©/āTcfgā2g= v_S(w), v_T^cfg /\| v_T^cfg\|^2, so that g=1g\!=\!1 means the student carries the teacherās full amplitude along the teacherās direction. Results in the table is averaged over 500 prompts and over the steps of each window. Table 7: Projection g of the studentās deployed field onto the teacherās guided field. Student Probe 0ā9 10ā19 20ā29 30ā37 0ā37 (total) GFD (w=1w\!=\!1) PteaP_tea 0.963 0.984 0.991 0.983 0.980 PrealP_real 0.943 0.971 0.987 0.984 0.968 GFD (w=2w\!=\!2) PteaP_tea 1.030 0.991 0.993 0.984 1.000 PrealP_real 1.023 0.978 0.989 0.985 0.997 At w=1w\!=\!1 the trained field of GFD is short along Tcfg v_T^cfg in every window, most so at the high-noise steps 00ā99. The student has thus learned the direction of the teacherās guidance but shrinks its magnitude toward its own prior. Raising the guidance scale to w=2w\!=\!2 compensates the shrinkage, and the average over steps 00ā3737 moves from 0.970.97ā0.980.98 to 1.001.00 on both probe sets. Besides, in the low-noise regime, the unconditional branch co-evolves with the conditional branch and exhibits similar shifts, leading to only a marginal increase of the CFG increment and avoiding excessive guidance. B.3 Metrics of Fig. 6 All four quantities compare a student sample with the teacher sample generated from the same prompt and the same initial noise, over N=500N\!=\!500 prompts; each axis of Fig. 6 is a log ratio to the teacher, so the teacher sits at the origin. Latent frequency energies (panel a). Let zāā64Ć64Ć64z ^64Ć 64Ć 64 be the final latent in the modelās packed token layout (6464 channels on a 64Ć6464Ć 64 grid) and z^c=DCT2ā(zc) z_c=DCT_2(z_c) the orthonormal 2-D DCT-I of channel c. With u,vā0,163,ā¦,1u,vā\0, 163,ā¦,1\ the normalized frequency indices and Ļā”(u,v)=u2+v2/2ā[0,1]Ļ(u,v)= u^2+v^2/ 2ā[0,1] the radial frequency, the energy of a band B is EB(z)=1643ācā(u,v)āBz^c(u,v)2,Blow=Ļ<0.25,Bhigh=Ļā„0.75.E_B(z)= 164^3 _c _(u,v)ā B z_c(u,v)^2, B_low=\Ļ<0.25\, B_high=\Ļā„ 0.75\. (12) The two coordinates of panel (a) are logā”(1NāānEBā(znS)/EBā(znT)) ( 1N _nE_B(z^S_n)/E_B(z^T_n) ) for B=BlowB=B_low (horizontal) and B=BhighB=B_high (vertical), where znSz^S_n and znTz^T_n are the studentās and the teacherās final latents for prompt n. Image tonal and fine-texture amplitude (panel b). The decoded 102421024^2 image I is area-downsampled to 5122512^2 and converted to luminance Y=0.2126āR+0.7152āG+0.0722āBY=0.2126R+0.7152G+0.0722B. Tonal amplitude combines the RMS contrast and the mean saturation, cā”(I)=stdā”(Y)meanā”(Y),sā”(I)=meanpmaxkā”Ikā(p)āminkā”Ikā(p)maxkā”Ikā(p),atoneā(I)=cā”(I)āsā(I),c(I)= std(Y)mean(Y), s(I)= *mean_p _kI_k(p)- _kI_k(p) _kI_k(p), a_tone(I)= c(I)\,s(I), (13) with k ranging over the RGB channels and p over pixels. Fine-texture amplitude is the mean magnitude of the 3Ć33Ć 3 high-pass residual of the luminance, atexā(I)=meanpāĪ©|Yā”(p)ābox3Ć3āYā(p)|,a_tex(I)= *mean_pā |Y(p)-box_3Ć 3Y(p) |, (14) where box3Ć3box_3Ć 3 is the 3Ć33Ć 3 mean filter with reflect padding and Ī© excludes a 1616-pixel border. The two coordinates of panel (b) are the mean log ratios 1Nāānlogā”(atoneā(InS)/atoneā(InT)) 1N _n (a_tone(I^S_n)/a_tone(I^T_n) ) (horizontal) and 1Nāānlogā”(atexā(InS)/atexā(InT)) 1N _n (a_tex(I^S_n)/a_tex(I^T_n) ) (vertical). Appendix C More Ablations C.1 GFD with unconditonal branch supervision Supervising the unconditional branch (GFD++uncond). GFD only constrains the conditional branch and leaves the unconditional branch unconstrained. A natural variant, GFD++uncond, keeps the folded target for the conditional branch and additionally matches the studentās unconditional branch to the teacherās: ā()=x0:Nā¼pS,[āj=0Nā1Ļj(āvScāvTgā22+āvSuāvTuā22)].L( Īø)=E_x_0:N p_S, Īø [ _j=0^N-1 _j ( \|v_S^c-v_T^g \|_2^2+ \|v_S^u-v_T^u \|_2^2 ) ]. (15) Same starting point. At w=1w\!=\!1 the two students deploy fields trained toward the same target and are indistinguishable: their KL divergence values are very similar, and their final sampling results sit at the same point in Fig. 7. Wrong increment direction. What changes is the content of the CFG increment vScāvSuv^c_S-v^u_S. In GFD the unconditional branch co-evolves with the conditional branch and exhibits similar shifts, resulting in the CFG only containing a small increment and avoiding excessive guidance. Raising w mainly adds a native-scale texture correction. However, GFD++uncond matches the unconditional branch to vTuv^u_T, removing the co-evolving. Its CFG increment is consequently doubled in magnitude. As demonstrated in Tab. 8 and Fig. 7, this causes a larger KL divergence and drives the final sampled results further away from the teacher distribution. Table 8: KL divergence to the teacher for GFD and GFD++uncond, in the format of Tab. 2. Student KL (PteaP_tea) KL (PrealP_real) GFDLāM (w=1w\!=\!1) 0.050 0.030 GFDLāM (w=2w\!=\!2) 0.053 0.032 GFD+uncondLāM (w=1w\!=\!1) 0.048 0.029 GFD+uncondLāM (w=2w\!=\!2) 0.060 0.040 Figure 7: Statistics of the final output of GFD and GFD++uncond as the guidance scale w varies, in the coordinates of Fig. 6 (log ratio to the teacher, teacher at the origin). Appendix D Additional qualitative results on FLUX.2 Figure 8: Qualitative comparison on FLUX.2 at Ć10241024\!Ć\!1024, complementing Table 4. Columns, left to right: the FLUX.2-4B base model; the FLUX.2-9B teacher and the GFD-OPD student distilled from it into 4B; the FLUX.2-32B teacher and the GFD-OPD student distilled from it into 4B. The top four rows are GenEval prompts; the bottom four are LongText prompts probing long-form visual text rendering.