Paper deep dive
Controllable Diverse Sampling through Pullback Directions
Sylvain Dehayem Kenfouo, Aaron Zweig, Mingxuan Zhang, Elham Azizi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/14/2026, 9:45:47 AM
Summary
The paper introduces a training-free method for enhancing diversity in conditional diffusion models by leveraging local pullback geometry. It identifies directions of maximal output sensitivity in the conditioning-embedding space and steers sampling trajectories along these directions to generate distinct images while maintaining prompt alignment. The method is evaluated on MS-COCO and BrushNet datasets, showing improved diversity metrics without requiring fine-tuning or significant computational overhead.
Entities (10)
Relation Signals (9)
Pullback Directions → isappliedto → Conditional Diffusion Models
confidence 98% · We propose a training-free method... for conditional diffusion models
Pullback Directions → isevaluatedon → MS-COCO
confidence 97% · Experiments on MS-COCO... show the efficacy of the method
Pullback Directions → improves → Diversity
confidence 96% · Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity
Pullback Directions → isevaluatedon → BrushNet
confidence 96% · Experiments on... BrushNet show the efficacy of the method
Pullback Directions → uses → Jacobian
confidence 95% · we require only a constant number of Jacobian evaluations... to identify directions of maximal output sensitivity.
Pullback Directions → maintains → Prompt Alignment
confidence 94% · encourage distinct generations while maintaining consistency with the conditioning signal.
Pullback Directions → outperforms → DDIM
confidence 90% · pullback-guided sampling consistently improves diversity over clean DDIM
Pullback Directions → outperforms → CADS
confidence 89% · pullback-guided sampling consistently improves diversity over... CADS
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conditional diffusion models can generate high-quality images that align closely with a given prompt. However, samples generated from different random seeds may remain visually similar, limiting creative exploration. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment. We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-embedding space to the image space to identify directions of maximal output sensitivity. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal. Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on prompt-conditioned image generation and inpainting.
Tags
Links
- Source: https://arxiv.org/abs/2610.14294v1
- Canonical: https://arxiv.org/abs/2610.14294v1
Trouble viewing inline? Open PDF directly →
Full Text
41,508 characters extracted from source content.
Expand or collapse full text
1 *2,3 3,4 3,4,5,6,7 1École Polytechnique, Institut Polytechnique de Paris 2New York Genome Center 3Irving Institute for Cancer Dynamics, Columbia University 4Department of Systems Biology, Columbia University 5Department of Computer Science, Columbia University 6Department of Biomedical Engineering, Columbia University 7Data Science Institute, Columbia University Controllable Diverse Sampling through Pullback Directions These authors contributed equally. Sylvain Dehayem Kenfouo Aaron Zweig Mingxuan Zhang Elham Azizi Abstract Conditional diffusion models can generate high-quality images that align closely with a given prompt. However, samples generated from different random seeds may remain visually similar, limiting creative exploration. Existing approaches can increase variation, but often require additional optimization or sampling overhead, while stronger perturbations may introduce distortions that degrade image quality or prompt alignment. We propose a training-free method that leverages the local pullback geometry induced by the mapping from the conditioning-embedding space to the image space to identify directions of maximal output sensitivity. By steering different sampling trajectories along these directions, we encourage distinct generations while maintaining consistency with the conditioning signal. Experiments on MS-COCO and BrushNet show the efficacy of the method in improving diversity on prompt-conditioned image generation and inpainting. 1 Introduction Recent advances in diffusion models (Ho et al., 2020; Song and Ermon, 2019; Song et al., 2020a; Song et al., 2020b; Rombach et al., 2022), have enabled text-to-image systems to generate visually compelling images that closely follow natural-language prompts. However, image quality and conditional alignment do not fully characterize a generative model. For a fixed condition, the model should also represent diverse valid outputs. In practice, trajectories may concentrate around the same dominant modes of the conditional distribution, resulting in semantically redundant outputs. This limited coverage increases the cost of obtaining a diverse set of samples. Inverse problems like image inpainting(Ju et al., 2024) further constrain diversity, as only a small masked region is generated. Many methods consider postprocessing steps that require many more gradient calculations, typically fine-tuning either the score model or the initial noise distribution to optimize a measure of diversity. Alternatively, inference-time methods based on sequential Monte Carlo can introduce diversity by drawing many samples and iteratively resampling to enforce a final distribution with high diversity. Both these methods suffer from substantially increased time complexity, either during fine-tuning or during inference. In this work, we therefore seek to answer: How can a pretrained diffusion model generate high-quality, diverse samples without fine tuning or sampling prohibitively many particles? We propose an efficient guidance method to encourage diverse samples. Namely, for a given prompt and possible inpainting mask, we require only a constant number of Jacobian evaluations (as few as one) from which we can generate an arbitrary number of diverse samples that are conditioned by the text and/or mask. We demonstrate improved performance on diversity metrics over comparable baselines operating in the regime of no fine-tuning and cheap inference. 2 Preliminaries Diffusion Models. We consider Denoising Diffusion Probabilistic Models (DDPM), which define a forward process that gradually corrupts samples from the data distribution with Gaussian noise (Ho et al., 2020). Given a clean image x0∼pdatax_0 p_data, the forward Markov chain is defined as q(x1:T∣x0)=∏t=1Tq(xt∣xt−1),q(xt∣xt−1)=(xt;αtxt−1,(1−αt)I),q(x_1:T x_0)= _t=1^Tq(x_t x_t-1), q(x_t x_t-1)=N (x_t; _t\,x_t-1,(1- _t)I ), (1) where αt=1−βt _t=1- _t and βtt=1T\ _t\_t=1^T denotes a predefined noise schedule. We let α¯t=∏s=1tαs α_t= _s=1^t _s. For a sufficiently long diffusion process, xTx_T approaches an isotropic Gaussian distribution. Given a network ϵθ _θ trained to predict the noise used to generate xtx_t conditioned on the condition c, the reverse process is modeled by pθ(xt−1∣xt,c)=(xt−1,1αt(xt−βt1−α¯tϵθ(xt,t,c)),σt2I).p_θ(x_t-1 x_t,c)=N (x_t-1; 1 _t (x_t- _t 1- α_t _θ(x_t,t,c) ), _t^2I ). (2) The predicted noise can then be used to estimate the original clean image at any timestep. Using Tweedie’s formula, the denoised estimate is x^0|t(xt,t,c):=[x0∣xt,c]=1α¯t(xt−1−α¯tϵθ(xt,t,c)). x_0 t(x_t,t,c):=E[x_0 x_t,c]= 1 α_t (x_t- 1- α_t\, _θ(x_t,t,c) ). (3) It therefore defines a differentiable mapping from the current noisy state to an estimate of the final clean image, which will later be used to construct the pullback geometry. Inpainting as an inverse problem. In image restoration problems, only a partial or corrupted measurement y of image x0x_0 is available. For noiseless image inpainting, let M∈0,1dM∈\0,1\^d denote the mask of the observed region, then y=M⊙x0y=M x_0. Text guidance controls the semantic content through c, whereas the mask constrains the generated sample to agree with the visible region y. 3 Conditional Pullback Metric In this section, we introduce a differential geometric framework for identifying directions in the conditioning space of diffusion models. In Euclidean spaces with a smooth map f:ℝn→ℝmf:R^n ^m, the pullback metric at x∈ℝnx ^n evaluated at v∈ℝnv ^n is defined as ‖Jf(x)v‖\|J_f(x)v\|. We therefore seek directions in the input space that induce proportionally larger shifts in the output space via the top right singular vectors of Jf(x)J_f(x). We consider the Jacobian of the noise predictor with respect to condition: Jc(xt,t,c)=∂ϵθ(xt,t,c)∂c,J_c(x_t,t,c)= ∂ _θ(x_t,t,c)∂ c, (4) In the case of image inpainting, we can chain the linear masking process with Tweedie’s formula to define a Jacobian from the context to the masked portion of the image. Let Medit=1−M_edit=1-M denote the bit-flipped mask of M, then we can define the linear operator WMW_M acting on image space by WMv=Medit⊙v,W_Mv=M_edit v, The differential of the masked clean estimate is ∂(WMx^0|t(xt,t,c))∂c=−1−α¯tα¯tWMJc(xt,t,c). ∂ (W_M x_0 t(x_t,t,c) )∂ c=- 1- α_t α_tW_MJ_c(x_t,t,c). (5) Using the conditional pullback metric, we can infer a subspace that locally produces large variability in the latent space. Explicitly, we consider span of the top K right singular vectors of Jacobian J~c=WMJc J_c=W_MJ_c. Since explicitly constructing the condition Jacobian is computationally intractable, we use a sketching method to obtain these directions using only Jacobian-vector products dominant singular directions Halko et al. (2011), for more details see Section B.1. 4 Pullback-informed prompt perturbation With the top singular vectors, we can identify directions in the prompt embedding space along which small perturbations induce large changes in the denoising prediction and, consequently, in the predicted clean latent. We use these directions to diversify the reverse trajectories while preserving the semantic content of the original condition. Initialization and trajectory diversification. We first sample a batch of B independent initial latent variables, T=zT(i)i=1BZ_T=\z_T^(i)\_i=1^B with zT(i)∼(0,I),z_T^(i) (0,I), and use the same latent batch for all methods considered in our experiments. Given the base prompt c0c_0, a pullback basis is then estimated from an anchor trajectory at the reference timestep tbt_b. The dominant eigenvectors of the condition-space metric Gtb=Jc0,tb⊤Jc0,tbG_t_b=J_c_0,t_b J_c_0,t_b are partitioned into mutually disjoint subspaces, one for each particle. Each particle is assigned a persistent direction obtained by projecting a fixed Gaussian field onto its corresponding subspace. The perturbations are mutually orthogonal and encourage different local variations of the conditional model. For particle i, the perturbed condition is therefore written as ci(t)=c0+ρi⋆γtΔci,c_i(t)=c_0+ _i _t c_i, where Δci c_i denotes the prompt-relative pullback direction defined in Equation 7, γt _t is the time-dependent annealing coefficient, and ρi⋆ _i controls the particle-specific perturbation strength. Scheduled perturbation. Let S>ES>E denote the perturbation window in reverse-diffusion timestep coordinates. We define the annealing schedule as a hard-sigmoid raised to a power p>0p>0, namely γ(t,S,E)=max(0,min(1,t−ES−E))p.γ(t;S,E)= (0, (1, t-ES-E ) )^p. The perturbation is fully active at the beginning of the reverse process, progressively annealed between S and E, and completely removed afterward. For particle i, the condition used at timestep t is ci,t=c0+ρi⋆γ(t,S,E,p)Δcic_i,t=c_0+ _i γ(t;S,E,p)\, c_i where Δci c_i is its assigned pullback direction. Adaptive pullback refresh. The pullback basis computed at the reference timestep tbt_b only captures the local geometry around the current state. Since this geometry evolves along the reverse trajectory, we update the basis at P intermediate timesteps: trref≈S+rP+1(E−S),r=1,…,P.t_r^ref≈ S+ rP+1(E-S), r=1,…,P. At each refresh timestep, we recompute the dominant eigenspace of the local pullback metric using the current latent of the anchor particle, and reproject each persistent Gaussian field ηi _i onto its refreshed particle-specific subspace. The previous and refreshed directions are interpolated over a short transition window to avoid abrupt changes in the conditioning trajectory 8. Particle-specific scale selection. The sensitivity of the condition to response map varies along the diffusion landscape. Consequently, a single perturbation scale may be too weak for some particles and too strong for others. We therefore select a particle’s specific scale ρi∗ _i^* from a finite candidate set ℛ=ρ1,…,ρMR=\ _1,…, _M\ by using the selection method in B.3. 5 Experiments & Results For text-to-image generation, we use captions from the MSCOCO 2017 validation (Lin et al., 2014). For image inpainting, we use the BrushBench validation set introduced with BrushNet (Ju et al., 2024). We generate using the Stable Diffusion family of models, comparing our pullback method against baselines of DDIM, CADS (Sadat et al., 2024), and TPSO (Meng et al., 2025) with standard metrics of prompt alignment and diversity. Full details are given in Section C. Tables 1 and 2 summarize the results for image generation and text-guided inpainting, respectively. Across both tasks, pullback-guided sampling consistently improves diversity over clean DDIM while maintaining a high level of alignment with the text condition. The results are particularly good for image inpainting, where the editable region constrains the possible variations and therefore makes the pullback metric a more significant initial signal. Figure 1: Qualitative comparison on a BrushBench example with BrushNet-SDXL. Each row contains masked samples (see Figure 2 in the Appendix). Clean DDIM, CADS, and TPSO introduce local changes, whereas adaptive pullback produces more pronounced variations. Table 1: Generation results on MSCOCO 2017 validation captions with five generated images per caption. Stable Diffusion 1.5 and XL results use all 500 captions. Entries report mean ± standard deviation across captions. The best mean within each backbone is shown in bold. Alignment Diversity Method CLIP ↑ MSS ↓ Vendi ↑ DINO Mean ↓ Stable Diffusion 1.5 Clean DDIM 31.3957±2.473031.3957± 2.4730 0.3359±0.05190.3359± 0.0519 4.6831±0.19714.6831± 0.1971 0.5945±0.16380.5945± 0.1638 CADS 31.2537±2.425331.2537± 2.4253 0.3066±0.04330.3066± 0.0433 4.7833±0.14264.7833± 0.1426 0.5469±0.16280.5469± 0.1628 TPSO 30.9592±2.294130.9592± 2.2941 0.3048±0.04300.3048± 0.0430 4.7838±0.14644.7838± 0.1464 0.5143±0.17290.5143± 0.1729 Pullback (ours) 31.3153±2.467131.3153± 2.4671 0.3194±0.05010.3194± 0.0501 4.7377±0.18334.7377± 0.1833 0.5663±0.16610.5663± 0.1661 Stable Diffusion XL Clean DDIM 32.0012±2.727132.0012± 2.7271 0.3827±0.05830.3827± 0.0583 4.4979±0.25374.4979± 0.2537 0.6899±0.15710.6899± 0.1571 CADS 31.7740±2.582231.7740± 2.5822 0.3485±0.05320.3485± 0.0532 4.6320±0.20894.6320± 0.2089 0.6374±0.16670.6374± 0.1667 TPSO 31.7017±2.553531.7017± 2.5535 0.3517±0.05460.3517± 0.0546 4.6141±0.21924.6141± 0.2192 0.6263±0.17280.6263± 0.1728 Pullback (ours) 31.4248±2.551131.4248± 2.5511 0.3196±0.04700.3196± 0.0470 4.7309±0.16364.7309± 0.1636 0.5639±0.17850.5639± 0.1785 Table 2: BrushBench inpainting results for Stable Diffusion 1.5 and Stable Diffusion XL, with five images per example. Stable Diffusion 1.5 and XL results use all 500 examples. Entries report mean ± standard deviation across examples. The best result within each backbone is shown in bold. Alignment Diversity Method CLIP ↑ DINO ↓ MSS ↓ Vendi ↑ Stable Diffusion 1.5 Clean DDIM 32.0003±2.738332.0003± 2.7383 0.8930±0.10940.8930± 0.1094 0.9643±0.03230.9643± 0.0323 1.1746±0.14411.1746± 0.1441 CADS 31.9284±2.727831.9284± 2.7278 0.8736±0.12150.8736± 0.1215 0.9594±0.03670.9594± 0.0367 1.1973±0.16421.1973± 0.1642 TPSO 31.9825±2.707731.9825± 2.7077 0.8807±0.11740.8807± 0.1174 0.9621±0.03360.9621± 0.0336 1.1844±0.14901.1844± 0.1490 Pullback (ours) 31.8796±2.805531.8796± 2.8055 0.8504±0.13360.8504± 0.1336 0.9498±0.04140.9498± 0.0414 1.2380±0.18171.2380± 0.1817 Stable Diffusion XL Clean DDIM 32.2147±2.741332.2147± 2.7413 0.9077±0.10010.9077± 0.1001 0.9701±0.02850.9701± 0.0285 1.1488±0.12731.1488± 0.1273 CADS 32.0407±2.727332.0407± 2.7273 0.8920±0.11020.8920± 0.1102 0.9656±0.03080.9656± 0.0308 1.1692±0.13731.1692± 0.1373 TPSO 32.1414±2.728532.1414± 2.7285 0.9014±0.10430.9014± 0.1043 0.9679±0.03060.9679± 0.0306 1.1585±0.13721.1585± 0.1372 Pullback (ours) 32.1108±2.727432.1108± 2.7274 0.8788±0.12460.8788± 0.1246 0.9604±0.03430.9604± 0.0343 1.1918±0.15191.1918± 0.1519 6 Conclusion We introduced a training-free method for diversity conditional diffusion models through pullback-informed prompt perturbations. Our method identifies locally expressive directions in the conditioning space that induce the largest variation in the image space, by estimating the dominant directions of the pullback operator. These results demonstrate that the local geometry of the denoiser with respect to condition provides useful directions for diversifying generation without fine-tuning. Future work will evaluate the effectiveness of pullback directions on other constrained generation tasks. References Astolfi et al. (2024) P. Astolfi, M. Careil, M. Hall, O. Mañas, M. Muckley, J. Verbeek, A. R. Soriano, and M. Drozdzal Consistency-diversity-realism pareto fronts of conditional image generative models. arXiv preprint arXiv:2406.10429. Cited by: §A.1. Ban et al. (2025) Y. Ban, R. Wang, T. Zhou, B. Gong, C. Hsieh, and M. Cheng The crystal ball hypothesis in diffusion models: anticipating object positions from initial noise. In International Conference on Learning Representations, Vol. 2025, p. 34384–34408. Cited by: §A.1. Corso et al. (2024) G. Corso, Y. Xu, V. De Bortoli, R. Barzilay, and T. Jaakkola Particle guidance: non-iid diverse sampling with diffusion models. In International Conference on Learning Representations, Vol. 2024, p. 22480–22507. Cited by: §A.3. De Santi et al. (2025) R. De Santi, M. Vlastelica, Y. Hsieh, Z. Shen, N. He, and A. Krause Provable maximum entropy manifold exploration via diffusion models. arXiv preprint arXiv:2506.15385. Cited by: §A.2. Domingo i Enrich et al. (2025) C. Domingo i Enrich, M. Drozdzal, B. Karrer, and R. T. Chen Adjoint matching: fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In International Conference on Learning Representations, Vol. 2025, p. 53791–53846. Cited by: §A.2. Eyring et al. (2024) L. Eyring, S. Karthik, K. Roth, A. Dosovitskiy, and Z. Akata Reno: enhancing one-step text-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems 37, p. 125487–125519. Cited by: §A.3. Friedman and Dieng (2022) D. Friedman and A. B. Dieng The vendi score: a diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410. Cited by: §C.1. Halko et al. (2011) N. Halko, P. Martinsson, and J. A. Tropp Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM review 53 (2), p. 217–288. Cited by: §B.1, §3. Harrington et al. (2025) A. Harrington, A. Koepke, S. Karthik, T. Darrell, and A. A. Efros It’s never too late: noise optimization for collapse recovery in trained diffusion models. arXiv preprint arXiv:2601.00090. Cited by: §A.3. Hessel et al. (2021) J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi Clipscore: a reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 7514–7528. Cited by: §C.1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1, §2. Ho and Salimans (2022) J. Ho and T. Salimans Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. Cited by: §A.1. Ju et al. (2024) X. Ju, X. Liu, X. Wang, Y. Bian, Y. Shan, and Q. Xu Brushnet: a plug-and-play image inpainting model with decomposed dual-branch diffusion. In European Conference on Computer Vision, p. 150–168. Cited by: §A.1, Appendix C, Appendix C, §1, §5. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: Appendix C, §5. Lugmayr et al. (2022) A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool Repaint: inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11461–11471. Cited by: §A.1. Meng et al. (2025) D. Meng, C. Jin, Z. Gao, Y. Li, I. Patras, and G. Tzimiropoulos Training-free generation of diverse and high-fidelity images via prompt semantic space optimization. arXiv preprint arXiv:2511.19811. Cited by: §A.3, 3rd item, §5. Nichol et al. (2021) A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Cited by: §A.1. Oquab et al. (2023) M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §C.1. Park et al. (2023) Y. Park, M. Kwon, J. Choi, J. Jo, and Y. Uh Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems 36, p. 24129–24142. Cited by: §D.1, §D.1. Rombach et al. (2022) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10684–10695. Cited by: §A.1, §1. Sadat et al. (2024) S. Sadat, J. Buhmann, D. Bradley, O. Hilliges, and R. Weber CADS: unleashing the diversity of diffusion models through condition-annealed sampling. In International Conference on Learning Representations, Vol. 2024, p. 23723–23755. Cited by: §A.3, 2nd item, §C.1, §5. Saharia et al. (2021) C. Saharia, W. Chan, H. Chang, C. A. Lee, J. Ho, T. Salimans, D. J. Fleet, and M. Norouzi Palette: image-to-image diffusion models. arXiv preprint arXiv:2111.05826. Cited by: §A.1. Song et al. (2020a) J. Song, C. Meng, and S. Ermon Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §1. Song and Ermon (2019) Y. Song and S. Ermon Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems 32. Cited by: §1. Song et al. (2020b) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §1. Zhang et al. (2023) L. Zhang, A. Rao, and M. Agrawala Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3836–3847. Cited by: §A.1. Zheng et al. (2023) G. Zheng, X. Zhou, X. Li, Z. Qi, Y. Shan, and X. Li Layoutdiffusion: controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 22490–22499. Cited by: §A.1. Appendix A Related works A.1 Image Diffusion Models Conditional image generation Conditional image generation aims to synthesize realistic images that satisfy user-specified controls. These conditions may take the form of textual descriptions, reference images, class labels, segmentation maps, depth maps, or spatial layouts. Diffusion models have become a dominant framework for this task because of their strong image quality and flexibility in incorporating heterogeneous conditions. Text-conditioned latent diffusion models introduce semantic information through cross-attention, enabling high-resolution text-to-image synthesis (Rombach et al., 2022), while classifier-free guidance strengthens alignment by combining conditional and unconditional denoising predictions (Ho and Salimans, 2022). Other approaches introduce image-to-image conditioning (Saharia et al., 2021) or additional spatial controls such as edges, poses, depth maps, and layouts (Zheng et al., 2023; Zhang et al., 2023). These developments have substantially improved the controllability and fidelity of conditional image synthesis. Image inpainting Image inpainting consists in reconstructing a missing region of an image while preserving the visible content and maintaining semantic and visual coherence across the edited region. Diffusion-based approaches address inpainting either by training models that explicitly receive the masked image and mask as conditioning inputs or by adapting the reverse process of a pretrained generative model (Ju et al., 2024; Lugmayr et al., 2022; Nichol et al., 2021). These approaches aim to generate plausible content within the masked region while preserving the observed pixels and maintaining semantic and visual coherence with the unmasked region. Diversity image generation Although diffusion models generate samples from random initial noise, stochasticity alone does not guarantee meaningful diversity under a fixed condition. Different initializations may still converge toward semantically similar outputs, particularly when strong guidance, textual constraints, or spatial observations restrict the reverse process. Recent analyses therefore emphasize a trade-off between realism, condition consistency, and distributional coverage (Astolfi et al., 2024), while other work shows that the initial noise can already strongly determine the spatial organization of the final image (Ban et al., 2025). Consequently, producing multiple distinct yet equally valid completions remains challenging, especially for constrained tasks such as text-guided inpainting. Existing methods address this limitation through additional training(finetuning diversity methods), condition perturbations, or interactions between sampling trajectories. A.2 Fine-tuning diversity methods Fine-tuning methods enable further training towards a target reward or distributional loss. For a scalar reward, Adjoint Matching (Domingo i Enrich et al., 2025) backpropagates through the entire dynamics to improve the reward while staying close to the original reverse process. This has been generalized as a tool to promote diversity in the generated distribution in De Santi et al. (2025). Notably, these methods require expensive sampling through the diffusion model and are therefore no longer simulation-free. A.3 Inference-Time Diversity Methods Inference-time diversity methods aim to improve the coverage of the conditional distribution without retraining the underlying diffusion model. Existing approaches differ primarily in the component of the sampling process on which they intervene. Condition-space methods modify the conditioning signal supplied to the denoising network. CADS (Sadat et al., 2024) perturbs the condition embedding with Gaussian noise whose magnitude is progressively annealed throughout sampling, thereby increasing variation during the early stages of generation. TPSO (Meng et al., 2025) instead explores underutilized directions of the token-embedding space through optimized prompt perturbations, while constraining the resulting samples to preserve semantic consistency. Some family directly modifies the source of stochasticity by optimizing the initial diffusion noise (Harrington et al., 2025; Eyring et al., 2024) , an others instead construct a joint, non-i.i.d. distribution sample (Corso et al., 2024). Particle Guidance introduces pairwise repulsive interactions during denoising to prevent particles from converging toward the same modes (Corso et al., 2024). Appendix B Pullback-Informed Prompt Perturbation Let C0+∈ℝL×dcC_0^+ ^L× d_c denote the positive token condition, where dcd_c is the token-embedding dimension. After vectorization, the condition belongs to ℝDR^D, with D=LdcD=Ld_c. B.1 Matrix-free estimation of expressive directions We first follow the clean CFG-DDIM trajectory of one reference particle to a basis-estimation timestep tbt_b, obtaining ztbrefz_t_b^ref. At the fixed state (ztbref,tb,C0+)(z_t_b^ref,t_b,C_0^+), we seek directions in the conditioning space that induce the largest local variation in the denoiser output. For a unit direction v, this variation can be measured by 1ε2‖fb(C0++εv)−fb(C0+)‖22,‖v‖2=1, 1 ^2 \|f_b(C_0^++ v)-f_b(C_0^+) \|_2^2, \|v\|_2=1, where fb(C)=ϵθ(ztbref,tb,C).f_b(C)= _θ(z_t_b^ref,t_b,C). Using the first-order approximation fb(C0++εv)−fb(C0+)≃εJbvf_b(C_0^++ v)-f_b(C_0^+) J_bv, the problem of finding the K most expressive orthonormal directions becomes V⋆=argmaxV⊤V=IK‖JbV‖F2=argmaxV⊤V=IKtr(V⊤Jb⊤JbV).V = _V V=I_K\|J_bV\|_F^2= _V V=I_Ktr\! (V J_b J_bV ). (6) Therefore, by the Rayleigh–Ritz theorem, the columns of V⋆V are the K dominant eigenvectors of Gb=Jb⊤Jb,Jb=∂fb∂C|C=C0+.G_b=J_b J_b, J_b= . ∂ f_b∂ C |_C=C_0^+. Equivalently, they are the dominant right singular vectors of JbJ_b. We estimate these directions without explicitly constructing JbJ_b. Jbv≈fb(C0++εv)−fb(C0+−εv)2ε,Gbv=Jb⊤(Jbv),J_bv≈ f_b(C_0^++ v)-f_b(C_0^+- v)2 , G_bv=J_b (J_bv), where the second operation is evaluated with a vector–Jacobian product. To approximate the K dominant eigenvectors, we use randomized block subspace iteration followed by Rayleigh–Ritz extraction (Halko et al., 2011). We sample Ω∈ℝD×K ^D× K with Gaussian entries and set Q0=QR(Ω),Qr+1=QR(GbQr),r=0,…,R−1.Q_0=QR( ), Q_r+1=QR(G_bQ_r), r=0,…,R-1. We then construct B=12(QR⊤GbQR+(QR⊤GbQR)⊤),B= 12 (Q_R G_bQ_R+(Q_R G_bQ_R) ), diagonalize B=UΛU⊤B=U U , sort the eigenvalues in decreasing order, and lift the resulting vectors back to condition space: V=QRU=[v1,…,vK].V=Q_RU=[v_1,…,v_K]. B.2 Fixed disjoint perturbation directions The basis indices are divided into balanced, mutually disjoint sets ℐ1,…,ℐNI_1,…,I_N. For each particle, a Gaussian field ξi∼(0,ID) _i (0,I_D) is sampled once and kept fixed. Its projection onto the assigned pullback subspace is d¯i=∑k∈ℐi⟨ξi,vk⟩vk,di=d¯i‖d¯i‖2. d_i= _k _i _i,v_k v_k, d_i= d_i\| d_i\|_2. To express the perturbation relative to the magnitude of the original condition, each real token is scaled by its clean embedding norm: Δci=‖C0+‖2di, c_i=||C_0^+||_2d_i, (7) where ‖C0+‖2||C_0^+||_2 denote the norms of the token embeddings. Adaptive pullback refresh For the refresh, the pullback basis and the corresponding directions are recomputed at intermediate denoising steps. Let dioldd_i^old and dinewd_i^new denote the directions before and after recomputation. To avoid an abrupt change, we interpolate between them over L=2L=2 steps: di(ℓ)=(1−βℓ)diold+βℓdinew‖(1−βℓ)diold+βℓdinew‖2,βℓ=ℓ2,ℓ∈1,2.d_i^( )= (1- _ )d_i^old+ _ d_i^new \|(1- _ )d_i^old+ _ d_i^new \|_2, _ = 2, ∈\1,2\. (8) . B.3 Particle-specific magnitude selection We select a particle’s specific magnitude from a finite candidate set ℛ=ρ1,…,ρMR=\ _1,…, _M\ at a specific timestep tpt_p before the final sampling run. For each particle i and candidate magnitude ρm _m, we compute the predicted clean image x^i,m x_i,m at tpt_p. We additionally use ρ0=0 _0=0 to obtain the unperturbed reference x^i,0 x_i,0. Semantic diversity between two candidates is measured using the cosine distance between their DINO representations: dDINO(x,x′)=1−cos(ϕDINO(x),ϕDINO(x′)),d_DINO(x,x )=1- ( _DINO(x), _DINO(x ) ), (9) where ϕDINO _DINO is a frozen DINO image encoder. Prompt alignment is measured by sCLIP(x,C)=cos(ϕCLIPI(x),ϕCLIPT(C)),s_CLIP(x,C)= ( _CLIP^I(x), _CLIP^T(C) ), (10) where ϕCLIPI _CLIP^I and ϕCLIPT _CLIP^T are the frozen CLIP image and text encoders, respectively. The particle-specific magnitudes are jointly selected by maximizing the average pairwise DINO distance while limiting the loss of prompt alignment relative to the unperturbed prediction: (m1⋆,…,mB⋆)∈argmaxm1,…,mBmi∈1,…,M (m_1 ,…,m_B )∈ _ subarraycm_1,…,m_B\\ m_i∈\1,…,M\ subarray 2B(B−1)∑i<jdDINO(x^i,mi,x^j,mj) 2B(B-1) _i<jd_DINO ( x_i,m_i, x_j,m_j ) (11) subject to to sCLIP(x^i,mi,C)≥sCLIP(x^i,0,C)−δCLIP,i=1,…,B. s_CLIP ( x_i,m_i,C )≥ s_CLIP ( x_i,0,C )- _CLIP, i=1,…,B. The selected magnitudes are ρi⋆=ρmi⋆ _i = _m_i . B.4 Scheduled reverse sampling After selecting ρi⋆ _i , sampling is restarted from the original initial latents. The perturbation envelope is α(t)=1,t≥TS,(t−TETS−TE)p,TE<t<TS,0,t≤TE.α(t)= cases1,&t≥ T_S,\\[5.69054pt] ( t-T_ET_S-T_E )^p,&T_E<t<T_S,\\[8.53581pt] 0,&t≤ T_E. cases At each reverse step, the particle-specific positive condition is Ci,t+=C0++α(t)ρi⋆ΔCi,tCi,t−=C0−.C_i,t^+=C_0^++α(t) _i _Ci,t C_i,t^-=C_0^-. The resulting pair is used in the standard CFG prediction, followed by one DDIM update. Appendix C Experimental Details Tasks and datasets For complete text-to-image generation, we use captions from the MSCOCO 2017 validation (Lin et al., 2014). For image inpainting, we use the BrushBench validation set introduced with BrushNet (Ju et al., 2024). Each inpainting example contains a source image, a binary edit mask, and a text condition describing the expected content. We evaluate 500 conditions from each benchmark and generate five images per condition, resulting in 2,500 generated images for every method on each task. Configuration Complete generation is performed with Stable Diffusion 1.5. The inpainting evaluation uses the corresponding Stable Diffusion 1.5 and XL BrushNet backbone (Ju et al., 2024). We use 50 DDIM sampling steps, classifier-free guidance with scale 7.57.5, and η=0η=0. We follow the publicly available default configurations of the corresponding models. For every condition, all compared methods receive exactly the same initial latent particles. All experiments use fixed random seeds to ensure fair and model-consistent comparisons. Compared methods We compare the following sampling procedures: • Clean DDIM: standard DDIM sampling without condition perturbation; • CADS: condition annealing with isotropic Gaussian perturbations (Sadat et al., 2024); • TPSO: token-prompt embedding space optimization applied during sampling (Meng et al., 2025); • Pullback: our method, which perturbs the condition along dominant pullback directions and uses particle-specific perturbation magnitudes. C.1 Evaluation Metrics The evaluation measures three complementary properties: semantic alignment, perceptual diversity, and set-level diversity. Text–image alignment. Text–image consistency is evaluated using the CLIP Score (Hessel et al., 2021), defined as the cosine similarity between CLIP embeddings of the generated image and its corresponding prompt. Higher values indicate stronger semantic alignment with the prompt. Semantic diversity. We extract DINOv2 image representations (Oquab et al., 2023) and compute the mean pairwise cosine similarity within each set of samples: Lower mean similarity indicates greater overall semantic diversity. Set-level diversity. We report the Mean Similarity Score (MSS) and Vendi Score (Friedman and Dieng, 2022) for per-prompt (intra-set) diversity, both computed using SSCD features following (Sadat et al., 2024). Lower MSS and higher Vendi indicates greater within-condition diversity. Figure 2: Illustration of the text-guided inpainting task. Given a text prompt, the model regenerates the masked region while preserving the unmasked region. The prompts are “a small bird perched on top of a rock” (top) and “a young girl holding a husky dog on a black background” (bottom). Prompt. “a red car with tinted windows driving through the desert, photorealistic, 8k” Figure 3: Token-dependent cross-attention at DDIM timestep t=499t=499. Each visualization averages 28 U-Net cross-attention maps over attention heads and layers. Warmer regions receive greater relative attention for the corresponding token. Appendix D Further Experimental Details D.1 Feature-space and conditioning-space pullback directions Park et al. Park et al. (2023) use the pullback metric to identify locally sensitive directions in the noisy latent space of a diffusion model. Their construction relies on the observation that the intermediate representations of the denoising model encode semantic and spatial information about the generated image. Let ft(⋅,c):t→ℋtf_t(·;c):X_t _t (12) denote the mapping from the noisy latent xtx_t to a selected intermediate U-Net feature representation ht=ft(xt,c)h_t=f_t(x_t;c) under condition c. Its Jacobian and induced pullback metric are Jx=∂ft(xt,c)∂xt,Gx=Jx⊤Jx.J_x= ∂ f_t(x_t;c)∂ x_t, G_x=J_x J_x. (13) The dominant eigenvectors vk\v_k\ of GxG_x, identify input directions that produce large local variations in the feature representation. For direction vkv_k and strength s, we construct the norm-matched latent perturbation δxt,k(s)=s∥xt∥2∥vk∥2vk.δ x_t,k(s)=s x_t _2 v_k _2v_k. (14) Following the x-space guidance formulation of Park et al. Park et al. (2023), the perturbed latent is x~t,k(s)=xt+γ[ϵθ(xt+δxt,k(s),t,c)−ϵθ(xt,t,c)], x_t,k(s)=x_t+γ [ _θ (x_t+δ x_t,k(s),t,c )- _θ(x_t,t,c) ], (15) where γ controls the magnitude of the noise-prediction response. We compare this construction with the conditioning-space pullback. For a fixed latent xtx_t, we consider the condition-response mapping gt(C)=ϵθ(xt,t,C),g_t(C)= _θ(x_t,t,C), (16) with Jacobian and pullback metric JC=∂ϵθ(xt,t,C)∂C,GC=JC⊤JC.J_C= ∂ _θ(x_t,t,C)∂ C, G_C=J_C J_C. (17) The dominant eigenvectors dk\d_k\ of GCG_C identify condition directions that induce large changes in the predicted noise. For both constructions, we compute a rank-44 approximation of the pullback metric at the same latent and retain the four dominant directions. We keep others parameters same for both. For the latent-feature pullback, we evaluate s∈0.005,0.015,0.02,0.025,0.03,0.04,0.05,s∈\0.005,0.015,0.02,0.025,0.03,0.04,0.05\, (18) with γ=5γ=5. For the conditioning pullback, we evaluate s∈0.1,0.25,0.5,1.0,1.5,2.0,2.5.s∈\0.1,0.25,0.5,1.0,1.5,2.0,2.5\. (19) The numerical values of s are not directly comparable across the two methods because the perturbations operate in different spaces with different sensitivity scales. We therefore select a specific range of values that clearly reveals the progressive effect of the perturbation on the generated image. The experiment compares how the output evolves within each method as the perturbation strength increases. Qualitative observations. The latent-feature pullback primarily produces localized variations and does not induce substantial semantic diversity. Increasing the perturbation strength does not necessarily amplify these variations in a meaningful way; instead, it progressively introduces high-contrast contours, geometric distortions, and other off-manifold artifacts. In contrast, the conditioning-space pullback produces broader semantic transformations that affect a larger portion of the image. These transformations remain direction-dependent across the strength sweep, with different pullback directions controlling distinct semantic attributes. Figure 4: Effect of the perturbation strength along latent-feature pullback directions. Rows correspond to the four dominant directions v1,…,v4v_1,…,v_4 of the pullback metric Gx=Jx⊤JxG_x=J_x J_x, where JxJ_x is the Jacobian of the selected U-Net feature representation with respect to the noisy latent xtx_t. Columns show the clean sample followed by increasing strengths. Moderate strengths produce predominantly localized, whereas larger strengths introduce visible distortions. Figure 5: Effect of the perturbation strength along conditioning-space pullback directions. Rows correspond to the four dominant directions of the condition pullback metric GC=JC⊤JCG_C=J_C J_C, where JC=∂ϵθ(xt,t,C)/∂CJ_C=∂ _θ(x_t,t,C)/∂ C. Columns show the clean sample followed by increasing strengths. The different directions produce distinct global semantic transformations, while large strengths can reduce image quality and fidelity. Figure 6: Qualitative comparison between latent-feature and conditioning-space pullback directions computed from the same anchor. The top row shows the clean sample and the four dominant latent-feature directions at s=0.025s=0.025, using an x-space guidance scale of γ=5γ=5. The bottom row shows the clean sample and the four dominant conditioning directions at s=1s=1. In this example, the latent-feature directions primarily affect localized appearance, pose, and nearby image structure, whereas the conditioning-space directions induce broader semantic changes. All results use the same prompt, initial noise, and reverse-process parameters.