Paper deep dive
Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models
Kuluhan Binici, Cihan Acar, Shivam Aggarwal, Siying Liu, Tulika Mitra
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-image diffusion models often use a fixed number of denoising steps, balancing time costs and image quality. However, the optimal number of steps depends on the complexity of the input text prompt. We propose an adaptive diffusion controller that dynamically adjusts the number of steps to generate high-quality images efficiently, without additional model training. By leveraging a mixture of step schedules with varying step sizes and evaluating the error term discrepancy at each timestep, our method transitions between schedules to optimize performance. Experiments on COCO and DiffusionDB show that our approach reduces inference time while maintaining visual fidelity, offering a more efficient alternative for text-to-image diffusion models.
Tags
Links
- Source: https://arxiv.org/abs/2609.16572v1
- Canonical: https://arxiv.org/abs/2609.16572v1
Trouble viewing inline? Open PDF directly →
Full Text
30,054 characters extracted from source content.
Expand or collapse full text
Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models Kuluhan Binici Cihan Acar Shivam Aggarwal Siying Liu Tulika Mitra Abstract Text-to-image diffusion models often use a fixed number of denoising steps, balancing time costs and image quality. However, the optimal number of steps depends on the complexity of the input text prompt. We propose an adaptive diffusion controller that dynamically adjusts the number of steps to generate high-quality images efficiently, without additional model training. By leveraging a mixture of step schedules with varying step sizes and evaluating the error term discrepancy at each timestep, our method transitions between schedules to optimize performance. Experiments on COCO and DiffusionDB show that our approach reduces inference time while maintaining visual fidelity, offering a more efficient alternative for text-to-image diffusion models. Index Terms: Diffusion models, efficient AI †address: 1SAP, 2National University of Singapore, 3Institute for Infocomm Research (I2R), A*STAR, Singapore 1 Introduction Diffusion models have showcased an impressive ability to create high-diversity and high-quality images from textual descriptions [16, 18]. One of the primary challenges with diffusion models is the high computational time incurred during the inference process. Knowledge distillation and neural architecture search techniques have been implemented to decrease the required number of time steps in the generation process [14, 8]. However, these require supplementary training of models. In addition, these methods typically use a fixed number of diffusion steps without considering the specific needs or complexity of the task. This one-size-fits-all approach to the denoising schedule overlooks the nuanced dynamics of the diffusion process, which can lead to either redundant computation or potentially immature results. As illustrated in Figure 1, the optimal number of steps is affected by the initialization of the noisy image, i.e. the random seed, and the text prompt. Notably, the dependence of the optimal time step schedule for the specific generation task has also been acknowledged by previous works [9]. Figure 1: Effect of text prompt and seed combinations on diffusion steps for generating visually sufficient images. Step counts shown are examples; optimal steps may vary. In this paper, we address this inefficiency by proposing an adaptive controller designed to allocate a minimal number of steps required to achieve visually satisfactory results for each text prompt. This allows better visual quality retention while improving time efficiency, rather than simply reducing the fixed schedule for all images. We named our approach as Adaptive Diffusion Step Controller (ADSC). ADSC does not require additional model training and functions by monitoring the convergence of the generation process during runtime to adjust the schedule accordingly. We postulate that the cosine distance between the conditional and unconditional update directions can serve as a heuristic for estimating convergence. When the cosine distance is large, indicating that the content described by the text prompt has not been generated in the denoised image yet, our scheduler opts for taking smaller step sizes to refine the generative process with finer granularity. Conversely, when the distance narrows, we infer that such content information is already contained in the image; thus taking larger step sizes are employed, expediting the overall process without compromising the image quality. Our observations reveal a discernible pattern where the cosine distance between update directions is initially substantial, reflecting the high entropy in the early stages that the diffusion process has not converged and the denoised image has not aligned with the text prompt. This distance typically diminishes as the process unfolds, permitting the scheduler to incrementally increase the step size. To validate the efficacy of our proposed method, we conducted extensive experiments on two well-established datasets: COCO [12] and DiffusionDB [23]. For evaluation metric, we utilize a variant of the CLIP score [4]; namely CLIP-I [20], and also the DINO metric [1, 20]. These metrics mainly compare the feature-space projections of the generated and reference images. The experimental results reveal that our adaptive approach can effectively reduce the average number of timesteps required to generate images from a given sequence of text prompts, in some cases by more than 50%, all while maintaining the visual quality. 2 Related Work To direct the generative process towards predetermined attributes or categories, external classifiers were incorporated to diffusion models but this requires the training and integration of distinct classifier models. Classifier-free diffusion guidance [6] effectively tackles this by embedding the guidance mechanism within the diffusion model itself, eliminating the need for an external classifier. Subsequent studies have enhanced the image generation process more precisely by guiding the internal representations within the diffusion models to control attributes like shape, location, and appearance of objects [3] and also by incorporating additional inputs such segmentation maps, keypoints and bounding boxes to introduce spatial conditioning controls [11, 2]. Due to the iterative nature of the denoising process, diffusion models face a significant issue in that they require a considerable amount of computational time during the inference process, as multiple forward passes are needed for each image generation. To tackle this challenge, efficient solvers such as DDIM [21], PNDM [13] and LMS [7] has been introduced to accelerate the sampling process. In addition, architectural compression [8], quantization [10], and knowledge distillation techniques have been implemented to decrease the required number of time steps in the generation process [14, 15]. Recently, Denoising Diffusion Step-aware Models (DDSMs) [25] have been developed, utilizing neural networks of varying sizes tailored to the specific demands of different stages of the generation process. The main drawback of these techniques is the need for supplementary training of models specifically for distillation, as well as the requirement to undertake searches for the most effective model architectures [9]. Furthermore, these methods employ a fixed number of diffusion steps, overlooking the need to adapt to the distinct challenges posed by different prompts. 3 Adaptive Diffusion Step Controller Our Adaptive Diffusion Step Controller (ADSC) encompasses two key technical components. First, we establish a diffusion step schedule control space within which our ADSC operates. This space is comprised of various schedules with distinct step lengths, enabling the controller to determine an optimal configuration by intelligently blending these schedules. Importantly, the step schedule control space allows each generation task to be completed in a varying number of steps, depending on the requirements of the generation task. Secondly, we introduce a heuristic criterion that guides the ADSC controller in crafting schedules tailored to each text prompt. This criterion strives to estimate the convergence of the diffusion process by monitoring the cosine distances between the conditional and unconditional noise estimations of the diffusion model. 3.1 Heterogenous schedule control space Figure 2: The denoising process begins with the longest schedule (e.g., 40 steps in the example) and transitions to shorter schedules (20 followed by 10) upon detecting convergence. The control space consists of K schedules, each with a varying number of steps, denoted as Sk=t1k,t2k,…tNkkS^k=\t^k_1,t^k_2,...t^k_N^k\, where |Sk|=Nk|S^k|=N^k represents the number of steps in schedule k∈[1,K]k∈[1,K]. The difference in time units between the steps in each schedule is distinct, resulting in diverse step sizes across the schedules. The controller initiates the execution by following the schedule with the most number of steps. It transitions to the corresponding step in the subsequent schedule whenever convergence is flagged. Each schedule SkS^k has fewer steps and consequently larger step sizes compared to the previous one : Nk> N^k> Nk+1 N^k+1 (1) ti+1k+1−tik+1>tj+1k−tjk>0; t^k+1_i+1-t^k+1_i>t^k_j+1-t^k_j>0; ∀i∈[1,Nk+1),∀j∈[1,Nk) ∀ i∈[1,N^k+1),∀ j∈[1,N^k) (2) This is motivated by the presumption that as the diffusion process converges and a major portion of the content information has been generated, the estimation error incurred by taking larger steps could have less impact on the final image quality.Moreover, the control space can be represented by C=S1,S2,…,SKC=\S^1,S^2,...,S^K\. When the transition criterion is met during the execution of the tikt^k_i timestep of a certain schedule SkS^k, the controller transitions to the subsequent schedule and resumes diffusion starting from the timestep tjk+1t^k+1_j that is closest to tikt^k_i, i.e. j=argminj′|tj′k+1−tik|j= _j |t^k+1_j -t^k_i| (3) Finally, the set of all possible mixtures of schedules that can be generated by the controller through its exploration on the control space, can be formulated as: M=⋃k=1Ktstk,…,tendk;t1k≤tst≤tend≤tNkkM= _k=1^K\t^k_st,…,t^k_end\;\ t^k_1≤ t_st≤ t_end≤ t^k_N^k (4) where tstkt^k_st and tendkt^k_end denote the starting and ending timesteps, respectively, of the selected segment from schedule k. The overall process is explained in pseudo-code format in Alg. 1. Algorithm 1 Control Process of the ADSC Controller 1: Input: Control space C=S1,S2,…,SKC=\S^1,S^2,...,S^K\ 2: Output: Mixed step schedule m 3: m←∅,k←1,i←1m← , 8.19447ptk← 1, 8.19447pti← 1 ⊳ Start with the longest schedule 4: while tikt^k_i < tNkkt^k_N^k do 5: if convergence detected then 6: k←k+1k← k+1 ⊳ Transition to the next schedule 7: i←argminj′|tj′k−tik−1|i← _j |t^k_j -t^k-1_i| 8: else 9: m←m∪tikm← m∪\t^k_i\ ⊳ Add the timestep to the mixed schedule 10: i←i+1i← i+1 11: end if 12: end while 3.2 Convergence criterion In this work we consider text-to-image generation using classifier-free guidance [6]. Classifier-Free Guidance generates both conditional and unconditional noise estimates (denoted by ϵc _c and ϵu _u) at each denoising step, steering model outputs to match specific conditions. The conditional estimate targets the given condition (e.g., a text description), while the unconditional estimate is produced without any guiding condition. The noisy image xtix_t_i is updated as xti+1=f(xti,ϵu+γ(ϵc−ϵu)⏟estimated noise (ϵ))x_t_i+1=f(x_t_i, _u+γ( _c- _u)_estimated noise (ε)) (5) where f is the function used to update the noisy image, and ϵε is the total noise estimate. γ is a coefficient called “guidance scale”, that adjusts the contribution of ϵc _c and ϵu _u while updating the image. To determine the convergence of the diffusion process, we track the cosine distance between the unconditional noise estimate and the total estimated noise ϵε, which is influenced by the conditional estimate ϵc _c. The cosine distance at each time step t is denoted as d(t)cos(ϵu,ϵ)=dcos(t)d^(t)cos( _u,ε)=d_cos(t). As the diffusion progresses, dcos(t)d_cos(t) exhibits a decline that resembles an inverse exponential function, as exemplified in Figure 3 for a sample drawn from DiffusionDB dataset. Figure 3: Plot of cosine distance between total and unconditional noise estimates across diffusion steps, along with first and second derivative curves, for a sample from the DiffusionDB dataset. The X-axis represents the diffusion steps. The elbow point on this curve is the point after which dcos(t)d_cos(t) does not change significantly, indicating that the denoised image has sufficiently converged to the conditional, i.e. the text prompt. At this stage, the quality of the denoised image becomes less sensitive to estimation errors resulting from large step sizes, which allows us to safely transition to the next schedule in the control space. The cosine distance curve Dcos(t)=dcos(t1),dcos(t2),…,dcos(tT)D_cos(t)=\d_cos(t_1),d_cos(t_2),…,d_cos(t_T)\ is subject to fluctuations, and approaches that make use of the derivative values to locate the elbow can erroneously detect local minima instead. To address this issue, we first smoothen the curve by applying exponential averaging. Let d~cos(ti) d_cos(t_i) denote the smoothed cosine distance at timestep tit_i, which can be computed using exponential averaging: d~cos(ti)=αdcos(ti)+(1−α)d~cos(ti−1) d_cos(t_i)=α d_cos(t_i)+(1-α) d_cos(t_i-1) (6) where α is the smoothing factor, which determines the degree of smoothing applied to the curve. Next, we examine the first and second derivatives of the smoothed cosine distance curve, denoted by d~cos(ti)′ d_cos(t_i) and d~cos(ti)′ d_cos(t_i) respectively: d~cos(ti)′ d_cos(t_i) =d~cos(ti)−d~cos(ti−1) = d_cos(t_i)- d_cos(t_i-1) (7) d~cos(ti)′ d_cos(t_i) =d~cos(ti+1)+d~cos(ti−1)−2×d~cos(ti) = d_cos(t_i+1)+ d_cos(t_i-1)-2× d_cos(t_i) (8) If the absolute values of both derivatives fall below predefined thresholds, we signal that the elbow point has been reached. Finally, we define the convergence criterion in terms of the first and second derivatives: =1,if |d~cos(ti)′|<δ1 and |d~cos(ti)′|<δ2,0,otherwise. C= cases1,&if | d_ (t_i) |< _1 and | d_ (t_i) |< _2,\\ 0,&otherwise. cases (9) As expressed in Eq. 9 the convergence criterion is set to 1 (indicating convergence) if the absolute values of both the first derivative, and the second derivative, fall below their respective threshold values, δ1 _1 and δ2 _2. 4 Experiments We integrate ADSC into the Stable Diffusion pipeline [19] and compare the quality of the generated images with those produced using a fixed number of step size. Our experiments involve utilizing PNDM [13] and DDIM [21] schedulers, while attaching our ADSC to these schedulers to control the generation process based on text prompts. In our experiments we chose the number of schedules in our control space, K, as 4, with each schedule containing half the number of steps as the previous one. This way we ensure one of every two time steps to be coinciding among consecutive schedules, allowing flexible transitions. For the guidance scale γ, we use the default value used in Stable Diffusion, that is 7.5. We evaluate the performance of our approach on two text-to-image generation datasets, namely DiffusionDB [23] and COCO [12]. To correct spelling errors, we pre-process DiffusionDB following the procedure outlined in Wu et al. [24]. For evaulation metrics, we use the CLIP-T variant of the CLIP score [4, 17], that measures the alignment between the text prompt describing the image and the generated image itself. Later, to also assess the visual quality, we used the CLIP-I [4, 20] and DINO [1] scores which compares the CLIP and DINO image embeddings of the generated samples with the reference images. Here we assume that images produced with many diffusion steps - mostly 50 steps - are near-optimal and use them as reference points. 4.1 COCO results For our evaluation on the COCO dataset, we first processed the prompts with our ADSC attached to the PNDM [13] and DDIM [21] schedulers within the Stable Diffusion v1-5 pipeline. In contrast to these fixed-step schedulers, our method tailors the number of steps specifically for each unique image sampling task. Therefore, to ensure a fair comparison with fixed-step schedulers, we determine their step counts based on the minimum, maximum, and average number of steps utilized by our adaptive controller. These values are displayed along with the visualization of the step distributions in Figure 4. Results indicate that the distribution of steps for both schedulers exhibits similarities to the Gaussian distribution. Moreover, the distribution of steps differs for each schedule, and among the two, DDIM appears to require a smaller number of steps when controlled by our ADSC. (a) DDIM (b) PNDM Figure 4: Distribution of steps upon controlling DDIM and PNDM schedulers using our ADSC in COCO dataset. The ADSC-controlled DDIM resulted in 18.8918.89 steps taken on average per prompt while PNDM led to 24.4824.48 steps on average. These step values are rounded and used for DDIM and PNDM schedulers as baselines. After establishing the number of steps for our baselines, we proceed to compare our method using CLIP-I, CLIP-T, and DINO scores. For calculating reference-based scores, namely CLIP-I and DINO, we employ images generated by fixed 50-step schedules as 50 is the default number of step used in Stable Diffusion pipeline [22]. As presented in Table 1, the results demonstrate that, on average, ADSC attains enhanced visual quality retention for the same number of reverse diffusion steps. When comparing the ADSC-controlled DDIM scheduler to its 19-step fixed counterpart, we observe a 3.3%3.3\% (95.7095.70 vs 92.4092.40) increase in CLIP-I score and an 8.30%8.30\% (92.4292.42 vs 84.1284.12) improvement in DINO score, with an average step count of 18.8918.89 steps. Likewise, the ADSC-controlled PNDM scheduler outperforms its fixed-step counterpart, with a CLIP-I score improvement of 1.50%1.50\% (94.1094.10 vs 92.6092.60) and a DINO score increase of 4.28%4.28\% (89.2489.24 vs 84.9684.96). Notably, the ADSC-controlled DDIM also surpasses the fixed 31-step schedule. This can be attributed to the heterogeneous schedules offered by our adaptive controller, which are not directly comparable to homogeneous schedules containing the same number of steps. In the case of CLIP-T, the scores remain relatively consistent across various baselines and our method, primarily because it evaluates text-image alignment. DDIM PNDM Strategy ADSC - ours* (18.89 steps) fixed (12 steps) fixed* (19 steps) fixed (31 steps) ADSC - ours* (24.48 steps) fixed (16 steps) fixed* (24 steps) fixed (34 steps) CLIP-I (↑ ) 95.70 90.00 92.40 95.20 94.10 91.30 92.60 94.50 CLIP-T (↑ ) 31.40 31.40 31.40 31.40 31.40 31.30 31.40 31.30 DINO (↑ ) 92.42 78.78 84.12 90.72 89.24 81.06 84.96 89.20 Avg wallclock time (sec/img)(↓ ) 0.79 0.55 0.78 1.23 1.06 0.71 1.06 1.40 Table 1: Comparison between ADSC and baseline fixed-step schedulers using various metrics on the COCO dataset. (*) highlights our method and the main baselines. DDIM PNDM Figure 5: Text prompts from COCO and images generated by our ADSC, and schedulers fixed number of steps. To ensure a comprehensive assessment, we also measure the average wall-clock times (latencies) per generated image. The results reveal that the latency of ADSC is nearly identical to that of fixed schedulers, with only minor differences up to two decimal places. For instance, the ADSC-controlled DDIM scheduler took an average of 0.79 seconds for 18.8918.89 steps, which is equal to the runtime of the 19-step DDIM schedule. Lastly, we provide example images generated by our ADSC and fixed baselines in Figure 5. The first and last two rows are obtained by DDIM and PNDM schedulers, respectively. An initial observation reveals that, in general, ADSC resulted in higher-quality images within a smaller number of timesteps compared to using a fixed step scheduler. Moreover, the prompts given in the first and last rows required more steps to generate images of sufficient quality compared to others. This can be attributed to the fact that the objects described in these prompts inherently contain intricate details, such as the keyboard and home screen of a laptop or street names written on a sign. Conversely, cats, cars, and tables are relatively more plain objects. The commonality of the objects and their presence in the training dataset of the diffusion model can be another contributing factor. 4.2 DiffusionDB results We followed the same procedures described in Section 4.1 to evaluate our method on the DiffusionDB dataset. The distribution of steps exhibits differences compared to COCO experiments, as seen in Figure 6. These differences are also reflected in the avg, min, and max statistics. (a) DDIM (b) PNDM Figure 6: Distribution of steps upon controlling DDIM and PNDM schedulers using our ADSC in DiffusionDB dataset. Overall, both adaptive and fixed scheduling approaches saw an improvement in CLIP-T scores beyond those observed in the COCO experiments. Upon coupling ADSC with DDIM, we observed that the improvement margins of CLIP-I and DINO remain significant, which are 3.2%3.2\% (95.1095.10 vs. 91.9091.90) and 6.4%6.4\% (89.3689.36 vs. 82.9682.96) respectively. As for the PNDM scheduler, prompt-adaptive control did not lead to any noticeable improvement in CLIP-I and caused the DINO score to drop by 1.59%1.59\%. This suggests that ADSC is more compatible with DDIM and more consistently improves performance when coupled with it. However, it is also worth noting that ADSC did not degrade PNDM performance significantly either. Therefore, considering the improvements seen in benchmarks like COCO, it remains beneficial to employ ADSC to enhance PNDM as well. DDIM PNDM Strategy ADSC - ours* (18.06 steps) fixed (12 steps) fixed* (18 steps) fixed (33 steps) ADSC - ours* (24.75 steps) fixed (16 steps) fixed* (25 steps) fixed (35 steps) CLIP-I (↑ ) 95.10 89.40 91.90 95.20 93.70 89.90 93.60 95.70 CLIP-T (↑ ) 33.70 33.70 33.90 34.00 33.80 33.30 33.70 33.80 DINO (↑ ) 89.36 77.01 82.96 90.90 85.50 79.02 87.09 87.21 Avg wallclock time (sec/img)(↓ ) 0.76 0.55 0.75 1.32 1.06 0.71 1.06 1.42 Table 2: Comparison between ADSC and baseline fixed-step schedulers using various metrics on the DifusionDB dataset. (*) highlights our method and the main baselines. DDIM PNDM Figure 7: Text prompts from DiffusionDB and images generated by our ADSC, and schedulers with fixed numbers of steps. Figure 7 contains images generated based on DiffusionDB prompts. The first two and last two rows are generated using DDIM and PNDM, respectively. For all four prompts, ADSC produced high-quality images, while several problems can be noticed in the images generated by fixed schedules. The first one is the misalignment with the text prompt, which can be observed in the last two rows: The 16 and 24-step generated images from the prompt "A programmer in a cyberpunk…" do not contain distinctive features of a programmer, and the 16, 24-step generated images from the prompt "A young man with black…" miss the color information of facial features described in the prompt as they are in grayscale. Although the 34-step generated image contains blue eyes, it is inconsistent with the grayscale theme. The second problem is the visual defects which can be seen in the first, second, and third rows. Finally, we expand our evaluation by comparing our method against fixed step schedules from two other schedulers, DDPM [5] and LMS [7]. We employ our ADSC combined with the DDIM scheduler for adaptive scheduling, as it yielded the best results. Table 3 presents the findings. Dataset DiffusionDB (50 → 18 steps) COCO (50 → 19 steps) Metric CLIP-I CLIP-T DINO CLIP-I CLIP-T DINO DDPM [5] 86.7 33.5 69.68 88.5 31.3 74.8 PNDM [13] 90.8 33.5 80.69 91.7 31.3 82.3 DDIM [21] 91.9 33.9 83.0 92.4 31.4 84.1 LMS [7] 94.1 33.7 88.2 94.8 31.3 90.5 DDIM + ADSC (ours) 95.1 33.7 89.4 95.7 31.4 92.4 Table 3: Comparison with various schedulers 5 Conclusion We introduced ADSC to address the inefficiency of fixed denoising schedules in text-to-image diffusion generative models. Our method dynamically adjusts the number of denoising steps required to generate high-quality images based on the specific characteristics of each text prompt, thereby optimizing computational resources and time efficiency without compromising significant visual quality. Experiments have demonstrated the effectiveness of our approach in reducing the average number of diffusion time steps, while still preserving the visual fidelity. Importantly, our method does not necessitate additional model training. One limitation is the need for manual tuning of hyper-parameters, such as gradient thresholds, which can be time-consuming process. Future work can explore adaptive thresholding and learned convergence criteria using reinforcement learning methods. References [1] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In ICCV, p. 9650–9660. Cited by: §1, §4. [2] M. Chen, I. Laina, and A. Vedaldi (2024) Training-free layout control with cross-attention guidance. In WACV, p. 5343–5353. Cited by: §2. [3] D. Epstein, A. Jabri, B. Poole, A. Efros, and A. Holynski (2024) Diffusion self-guidance for controllable image generation. NeurIPS 36. Cited by: §2. [4] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi (2021) Clipscore: a reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718. Cited by: §1, §4. [5] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. NeurIPS 33, p. 6840–6851. Cited by: §4.2, Table 3. [6] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §2, §3.2. [7] T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. NeurIPS 35, p. 26565–26577. Cited by: §2, §4.2, Table 3. [8] B. Kim, H. Song, T. Castells, and S. Choi (2023) On architectural compression of text-to-image diffusion models. arXiv preprint arXiv:2305.15798. Cited by: §1, §2. [9] L. Li, H. Li, X. Zheng, J. Wu, X. Xiao, R. Wang, M. Zheng, X. Pan, F. Chao, and R. Ji (2023) Autodiffusion: training-free optimization of time steps and architectures for automated diffusion model acceleration. In ICCV, p. 7105–7114. Cited by: §1, §2. [10] X. Li, Y. Liu, L. Lian, H. Yang, Z. Dong, D. Kang, S. Zhang, and K. Keutzer (2023) Q-diffusion: quantizing diffusion models. In ICCV, p. 17535–17545. Cited by: §2. [11] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee (2023) Gligen: open-set grounded text-to-image generation. In ICCV, p. 22511–22521. Cited by: §2. [12] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In ECCV, p. 740–755. Cited by: §1, §4. [13] L. Liu, Y. Ren, Z. Lin, and Z. Zhao (2022) Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778. Cited by: §2, §4.1, Table 3, §4. [14] E. Luhman and T. Luhman (2021) Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388. Cited by: §1, §2. [15] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans (2023) On distillation of guided diffusion models. In CVPR, p. 14297–14306. Cited by: §2. [16] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen (2021) Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Cited by: §1. [17] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In ICML, p. 8748–8763. Cited by: §4. [18] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen (2022) Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1 (2), p. 3. Cited by: §1. [19] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In CVPR, p. 10684–10695. Cited by: §4. [20] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman (2023) Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In ICCV, p. 22500–22510. Cited by: §1, §4. [21] J. Song, C. Meng, and S. Ermon (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §2, §4.1, Table 3, §4. [22] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf Diffusers: state-of-the-art diffusion models. Note: https://github.com/huggingface/diffusers Cited by: §4.1. [23] Z. J. Wang, E. Montoya, D. Munechika, H. Yang, B. Hoover, and D. H. Chau (2022) Diffusiondb: a large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896. Cited by: §1, §4. [24] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: §4. [25] S. Yang, Y. Chen, L. Wang, S. Liu, and Y. Chen (2023) Denoising diffusion step-aware models. arXiv preprint arXiv:2310.03337. Cited by: §2.