Paper deep dive
Scale Space Diffusion
Soumik Mukhopadhyay, Prateksha Udhayanan, Abhinav Shrivastava
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/13/2026, 12:55:18 AM
Summary
Scale Space Diffusion (SSD) is a novel generative framework that integrates scale-space theory into the diffusion process by replacing standard additive noise with generalized linear degradations, such as downsampling. By formalizing the information hierarchy between diffusion timesteps and Gaussian pyramid scales, the authors introduce Flexi-UNet, an architecture capable of resolution-preserving and resolution-increasing denoising, which improves computational efficiency and scaling behavior compared to traditional pixel-space diffusion models.
Entities (5)
Relation Signals (3)
Scale Space Diffusion → evaluatedon → CelebA
confidence 95% · We evaluate our framework on CelebA and ImageNet
Scale Space Diffusion → generalizes → DDPM
confidence 95% · Denoising diffusion probabilistic models (DDPM) emerge as a special case of SSD
Scale Space Diffusion → utilizes → Flexi-UNet
confidence 95% · To support Scale Space Diffusion, we introduce Flexi-UNet
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion models degrade images through noise, and reversing this process reveals an information hierarchy across timesteps. Scale-space theory exhibits a similar hierarchy via low-pass filtering. We formalize this connection and show that highly noisy diffusion states contain no more information than small, downsampled images - raising the question of why they must be processed at full resolution. To address this, we fuse scale spaces into the diffusion process by formulating a family of diffusion models with generalized linear degradations and practical implementations. Using downsampling as the degradation yields our proposed Scale Space Diffusion. To support Scale Space Diffusion, we introduce Flexi-UNet, a UNet variant that performs resolution-preserving and resolution-increasing denoising using only the necessary parts of the network. We evaluate our framework on CelebA and ImageNet and analyze its scaling behavior across resolutions and network depths. Our project website ( this https URL ) is available publicly.
Tags
Links
- Source: https://arxiv.org/abs/2603.08709v1
- Canonical: https://arxiv.org/abs/2603.08709v1
Trouble viewing inline? Open PDF directly →
Full Text
77,500 characters extracted from source content.
Expand or collapse full text
Scale Space Diffusion Soumik Mukhopadhyay ∗ soumik@umd.edu Prateksha Udhayanan ∗ pudhayan@umd.edu Abhinav Shrivastava abhinav2@umd.edu University of Maryland, College Park Abstract Diffusion models degrade images through noise, and revers- ing this process reveals an information hierarchy across timesteps. Scale-space theory exhibits a similar hierarchy via low-pass filtering. We formalize this connection and show that highly noisy diffusion states contain no more in- formation than small, downsampled images - raising the question of why they must be processed at full resolution. To address this, we fuse scale spaces into the diffusion process by formulating a family of diffusion models with generalized linear degradations and practical implementations. Using downsampling as the degradation yields our proposed Scale Space Diffusion. To support Scale Space Diffusion, we intro- duce Flexi-UNet, a UNet variant that performs resolution- preserving and resolution-increasing denoising using only the necessary parts of the network. We evaluate our frame- work on CelebA and ImageNet and analyze its scaling be- havior across resolutions and network depths. Our project website is available publicly. 1. Introduction Diffusion models [16,36] are a class of generative mod- els that achieve image synthesis by reversing an iterative noising process. It has been observed that states at differ- ent stages of the diffusion process encode different types of information [28]. As shown along the y-axis of Fig. 1(a), increasing diffusion noise progressively removes fine facial details while retaining only coarse structure. Eventually, with sufficient noising, even this structural information is lost. This illustrates that diffusion timesteps form an intrinsic information hierarchy. A similar property underlies scale space theory [24], a fundamental subfield of computer vision. Scale spaces also represent image signals in an information-hierarchical man- ner through successive low-pass filtering. Along the x-axis of Fig. 1(a), we see the loss of details as the resolution de- creases in a Gaussian pyramid, mirroring the information * Equal contribution. Scale Space (Gaussian Pyramid) Diffusion Noising Scale Space Diffusion Resize + noise (a) 2 4 CelebA-64x64 (1M iterations) 5 10 FID CelebA-128x128 (300k iterations) 406080 Train time (hrs) 5 10 CelebA-256x256 (300k iterations) DDPMOurs (multi-scale) (b) Figure 1. (a) Our proposed Scale Space Diffusion fuses scale spaces into diffusion models. (b) We show trends in image generation performance versus time for our proposed Flexi-UNet for CelebA- 64, CelebA-128, and CelebA-256. Multiple point on the same plot represent our models with different number levels (i.e., number of intermediate resolutions). We see immense gains in efficiency with resolution scaling while having reasonable performance. dissipation in the diffusion process. The main distinction lies in the mechanism of information degradation: diffusion uses iterative noising, whereas scale spaces use progressive blurring or downsampling. We investigate this relationship between diffusion and scale spaces formally through a preliminary mathematical modeling of information in both processes. This reveals striking parallels in their information content, suggesting a fundamental connection between the two. Intuitively, one may ask why completely noisy images should be processed at high resolution when they contain information equivalent to that of a tiny image. These parallels indicate that the two axes in Fig. 1(a) correspond to different but compatible ways of information degradation. In this work, we revisit pixel diffusion to achieve a uni- fication of scale spaces and the diffusion process. Previ- ous attempts at this either operate only at the highest res- olution [3,18], making them computationally inefficient, or rely on simplistic covariance assumptions [1] that may not hold in practice, or perform noisy scale shifting using high-frequency [2] or decorrelation noise [6,19,20], which 1 arXiv:2603.08709v1 [cs.CV] 9 Mar 2026 remain inference-time approximations. Unlike pyramidal flow-matching approaches that approximate scale changes only during inference, our formulation integrates scale tran- sitions directly into the diffusion process. In contrast, we first develop a mathematical theory for diffusion processes under generalized linear degradations, yielding a family of diffusion processes. We further illustrate how these can be implemented in modern deep-learning frameworks. Next, using image resizing as the linear degradation, we realize a fusion of scale spaces with diffusion. We term this process Scale Space Diffusion (SSD). Denoising diffusion probabilis- tic models (DDPM) [16] emerge as a special case of SSD, corresponding to the trivial case of resizing to the same size, i.e., the identity operator. These generalized degradations naturally induce non-isotropic posteriors, which we handle through an implicit sampling procedure. To realize the general version of Scale Space Diffusion, we require a neural network architecture capable of reversing the downsizing degradation, i.e., it must be able to upsample a noisy state. A na ̈ ıve approach could use a UNet [33] di- rectly, but this would require even small-scale images to pass through the full network, leading to unnecessary computa- tional cost. To address this, we propose a novel convolutional neural network (ConvNet) architecture that augments the standard UNet to use only the relevant levels of the network. It supports both resolution-preserving diffusion steps and next-resolution upscaling at all stages of a Gaussian pyramid. We denote this architecture as Flexi-UNet. We analyze our framework and architectures on uncon- ditional image generation using commonly used datasets of CelebA [25] and ImageNet [8]. To study the scaling prop- erties of our method, we conduct experiments at multiple resolutions of CelebA dataset as shown in Fig. 1 (b). We observe that our models are faster during both training and inference while achieving reasonable FID scores. The key contributions of this work are: 1. We uncover and analyze the relationship between the states of diffusion models and the levels of scale spaces. 2. We build the mathematical foundation for a family of generalized linear diffusion processes, and techniques to implement them in modern deep-learning frameworks. With resizing as the choice for the linear degradation, we realize the fusion of diffusion and scale spaces, which we term Scale Space Diffusion. 3. To enable Scale Space Diffusion, we introduce a novel architecture Flexi-UNet capable of handling both resolution-changing as well as resolution-preserving re- verse diffusion across multiple resolutions. 2. Related Work Diffusion Models. Diffusion Models have become the de facto standard for image generation in recent times. Early works such as DDPM [16] achieved high-quality image gen- eration without adversarial training, but relied on simulating a Markov chain with a large number of steps for sampling. DDIM [35] accelerated the sampling process, while meth- ods such as LDM [32] performed denoising in a compact latent space rather than directly in the pixel-space. Recently, DiTs have become popular, replacing traditional UNet based backbones with transformer architecture [31]. Motivated by the goal of scaling diffusion models for high-resolution image generation while maintaining architectural simplicity and high-frequency image details, we propose an end-to- end Scale Space Diffusion model that performs denoising directly in the pixel domain. Scale-Space Theory. Scale-space theory [24] is a fundamen- tal concept in computer vision, that provides a framework for multi-scale image representation and analysis. It has been widely used in visual understanding tasks [5,27]. The underlying idea of representation at multiple scales has been smartly used in the context of generative models to progres- sively generate images at increasing resolutions. In GAN- based approaches, Progressive GAN [21] has shown excel- lent results in generating high-resolution images by learning to generate at increasing resolutions during the training pro- cess. In some other works such as LAPGAN [9], multiple GANs, one for each scale, are used to upscale the image by producing a residual, similar to a Laplacian pyramid. Several works in the space of diffusion models have also drawn inspiration from scale-space theory. Cascaded dif- fusion model [17] consists of a series of diffusion models that generate images of increasing resolutions, where the base model produces a low-resolution image and subsequent super-resolution models refine it using the upsampled ver- sion of the low-resolution image as a condition. Matryoshka Diffusion [12] model proposes a diffusion process that de- noises inputs at multiple resolutions jointly. However, none of these approaches directly incorporate scale-space theory in the diffusion process because the noise component of the noisy intermediate state leads to correlated noise pixels at an upsampled state. Some works solve this by adding additional noise at the higher resolution. Relay Diffusion [37] imagines a low-resolution generation as a high-resolution image with block noise and trains a model to denoise it at higher resolution with a weighted combi- nation of block noise and high-resolution noise. Laplacian Diffusion Models [2] train separate models for different res- olutions and add a Laplacian residual of high-resolution noise during resolution transitions. However, simply adding high-resolution noise does not fully resolve the distribution mismatch between noisy states at different resolutions. Pyra- midal Flow Matching [20] addresses this issue by adding decorrelated noise while also rolling the diffusion process back to a noisier timestep. PixelFlow [6] and Region Adap- tive Latent Sampling [19] build on this idea. Bottleneck- Sampling [38] as opposed to increasing scales introduces a bottleneck scale for better generation, while Decomposed 2 Flow Matching [13] predicts Laplacian residuals of clean im- ages. UDPM [1] tries to add blurring and subsampling into the diffusion process, assumes isotropic posterior covariance to simplify their reverse diffusion derivation, which may not hold, given that the blurring kernels usually overlap in most implementations of resizing. We show through Scale Space Diffusion that end-to-end training of a single diffusion model capable of handling multiple resolutions, with a generalized mathematical formulation for resolution transitions, helps to achieve faster generation, while preserving high-quality. 3. Scale Spaces vis- ` a-vis Diffusion Timesteps In this section, we outline the motivation behind our ap- proach, which originates from a simple but compelling intu- ition. Consider the intermediate states of a diffusion model (Fig. 2(a), bottom) and the scales of a Gaussian pyramid (Fig. 2(b), bottom). If one squints and focuses on the third image from the left along thet-axis, the overall structure of the face begins to emerge, which is remarkably similar to the information present in the images corresponding to smaller spatial scales along ther-axis of the Gaussian pyramid. As we move rightward along either axis (i.e., decreasingtor increasingr), it becomes evident how finer details are added progressively. This observation suggests a striking correspondence in the information hierarchy between diffusion timesteps and scale-space resolutions (or scales). Our goal is to quantify this correspondence. To do so, we first review the standard diffusion process, and then formalize our intuition by mathe- matically characterizing the amount of information present across diffusion states. 3.1. Preliminary: Standard Diffusion Process In standard denoising diffusion probabilistic models (DDPM) [16], the forward diffusion process is modeled as a Markov chain that progressively noises a signal by adding Gaussian noise. Forx 0 ∼ q(x 0 ), whereq(x 0 )is the data distribution, the process is defined as: x t = √ α t x t−1 + √ 1− α t ε, ε∼N (0, I) (1) whereβ t T t=1 is the variance schedule (withα t := 1− β t ). This expression, when applied iteratively overt, leads to an alternative definition that expresses the noisy state as a linear combination of the signal x 0 and the noise ε: x t = √ ̄α t x 0 + √ 1− ̄α t ε, ε∼N (0, I) (2) where ̄α t := Q t i=0 α i .Diffusion models aim to re- verse this process by approximating the posterior distribu- tionq(x t−1 |x 0 ,x t )using a neural network (with param- etersθ) that predicts the noiseεin Eq. 2. This model, ε θ (x t ,t), is trained using a simplified loss functionL simple = E x 0 ,t,ε ∥ε θ (x t ,t)− ε∥ 2 2 . The model can also be parameter- ized to predict x 0 instead of ε. Figure 2. Information Analysis. (a) Amount of information present in a diffusion state as diffusion steptchanges. (b) Amount of information present in images at various resolutions (scales). 3.2. Information Degradation in Diffusion and Scale Spaces Diffusion States. In this section, we formally model the information degradation over the diffusion process. Eq. 2 has two terms – a signal term and a noise term. One way to model the amount of information present inx t is to compute the percentage of pixels for which the noise term dominates the signal term, i.e.,| √ 1− ̄α t ε| > | √ ̄α t x 0 |. In other words, we are looking for the probability that |ε|is greater thans(t)|x 0 |, wheres(t) = √ ̄α t √ 1− ̄α t is the square root of the signal-to-noise coefficient ratio. We have P (|ε| > s(t)|x 0 |) = (1− Φ(s(t)|x 0 |)) + Φ(−s(t)|x 0 |) = 2 Φ(−s(t)|x 0 |), whereΦis the CDF of the standard nor- mal distribution, and the second equality follows from symmetry of the standard normal distribution.Hence, P (|ε|≤ s(t)|x 0 |) = 1− 2 Φ(−s(t)|x 0 |). Now to obtain the expected fraction of signal-dominated pixels, a proxy for information, we average this probability over the data distri- butionq(x 0 ). For simplicity, let us assumex 0 ∼U (−1, 1). Then the variation of information over timesteptcan be written as: Info(t) = E x 0 ∼U (−1,1) [1− 2Φ(−s(t)|x 0 |)] = 1− 2 Z 1 −1 p U (−1,1) (x)Φ(−s(t)|x|)dx = 1− 2 Z 1 −1 1 2 Φ(−s(t)|x|)dx = 1− Z 1 −1 Φ(−s(t)|x|)dx = 1− 2 Z 1 0 Φ(−s(t)x)dx, where we use the fact that the uniform distribution has den- sityp U = 1 2 over[−1, 1], and for the final equality we split the integral aboutx = 0. Using this simplification,Info(t) can be numerically computed as a function oft, as shown in Fig. 2(a). Scale Spaces. Similar to the approximation of information across diffusion steps, here we want to approximate the information as a function of image resolution (i.e., scale). A simple way to model this is to assume: Info∝ Area. 3 Let us consider a normalized resolutionr ∈ [0, 1], where0 represents no pixels and1represents the highest resolution. Under this assumption, the information can be written as: Info(r) = r 2 . This implies, for example, that if the spatial dimensions of an image are halved, then the information becomes one quarter, which may not be strictly true due to redundancy in pixel space. However, the monotonic trend should still hold. This trend is visualized in Fig. 2(b). Notice how there is a similarity in the trends of informa- tion degradation astincreases versus asrdecreases. This analysis quantifies our main intuition regarding the similar- ity in the information trends across diffusion steps and scale spaces. Given this insight, we aim to leverage this intuition to construct a framework that realizes scale spaces within the current formulation of diffusion models. In our initial attempts to incorporate scale spaces into diffusion models, we tried to frame this problem as jumping across the same timesteps of independent diffusion processes at varying scales. However, this led to an accumulation of errors during the iterative inference procedure, resulting in suboptimal outputs. Methods such as Pyramidal Flow Match- ing [6,19,20] address this issue by adding decorrelation noise when transitioning across scales and then backtracking in time so that an appropriate noise level is selected. This strategy helps mitigate the error accumulation. Nonetheless, it does not actually resolve the underlying issue – the diffu- sion process itself is not mathematically modeled to handle scale changes. In this work, we aim to fill this gap. 4. Scale Space Diffusion (SSD) In this section, we introduce a new family of diffusion pro- cesses that use a generalized linear degradation operation for degrading the signal, in addition to the standard addi- tive Gaussian noise. We then show how this formulation can be implemented in deep learning frameworks such as PyTorch [30] for any choice of a linear degradation that is available as a function call. In our case, we choose a downsizing operator as our linear degradation. Next, we present our training and sampling pipelines. Finally, we introduce our architecture that can handle scale-preservation and scale-changing transitions at multiple resolutions. 4.1. Generalized Linear Diffusion Process 4.1.1. Extension to Linear Degradation We now replace the scalar coefficient ofx t−1 in Eq. 1, i.e., √ α t , with a more generic linear operatorM t . For exam- ple, blurring or downsampling can serve as such a linear operator. Let us assume a Gaussian distribution for this up- dated formulation for the transition distributionq(x t |x t−1 ) asx t = M t x t−1 + η t ,η t ∼N (0, Σ t|t−1 ). Here, we do not assume Σ t|t−1 to be isotropic. Now, repeatedly sampling the next state using the tran- sition distribution, we want to derive an equation analo- gous to Eq. 2, which provides usx t givenx 0 . It is clear that this will also be a Gaussian distributionq(x t |x 0 ) = N (μ t , Σ t ) . The only constraint we want to enforce is isotropy, i.e.,Σ t = σ 2 t I. For the coefficient ofx 0 , in- stead of √ ̄α t = √ α t √ α t−1 ... √ α 1 in Eq. 2, we get M 1:t = M t M t−1 ...M 1 , i.e.,μ t = M 1:t x 0 . Hence, q(x t |x 0 ) =N (M 1:t x 0 ,σ 2 t I), which can be expressed as: x t = M 1:t x 0 + σ t ε, ε∼N (0, I)(3) Using Theorem 1, similar to blurring diffusion [18], the transition distribution q(x t |x t−1 ) is given by: x t = M t x t−1 + η t , η t ∼N (0, Σ t|t−1 ), where Σ t|t−1 = Σ t − M t Σ t−1 M T t . (4) For the isotropic marginalsΣ t = σ 2 t IandΣ t−1 = σ 2 t−1 I , we obtainΣ t|t−1 = σ 2 t I− σ 2 t−1 M t M T t . For positive semi- definite feasibility we requireσ 2 t I⪰ σ 2 t−1 M t M T t , i.e.,σ 2 t ≥ σ 2 t−1 λ max (M t M T t ). As shown in Theorem 2, the reverse diffusion step, i.e., the posterior distributionq(x t−1 |x t ,x 0 ), conditioned addi- tionally on x 0 , is also a normal distribution: q(x t−1 |x t ,x 0 ) =N (μ t→t−1 , Σ μ t→t−1 ), where Σ t→t−1 = (Σ −1 t−1 + M T t Σ −1 t|t−1 M t ) −1 , and μ t→t−1 = Σ t→t−1 (Σ −1 t−1 μ t−1 + M T t Σ −1 t|t−1 x t ) (5) Using the Woodbury matrix identity and isotropic covari- ance assumption, this simplifies to (Theorem 3): Σ t→t−1 = σ 2 t−1 I− σ 4 t−1 σ 2 t M T t M t μ t→t−1 = μ t−1 + σ 2 t−1 σ 2 t M T t (x t − M t μ t−1 ) (6) Please refer to Table 1 for the comparison of our Generalized Linear Diffusion Process framework against DDPM and Blurring Diffusion (BD). DDPM as a special case of SSD. WhenM t = √ ̄α t Iand σ t = √ 1− ̄α t , the forward, marginal, and posterior distri- butions of SSD collapse to those of the DDPM model. 4.1.2. Implementation Details Choice ofM t . We derived the above framework so that we can introduce scale spaces from Gaussian pyramids into the diffusion process. AlthoughM t may be any arbitrary linear operator, for our purposes we select it to be a resize operator, which effectively blurs and downsamples the image, and then multiplies it bya t = √ ̄α t , as shown in Algo. 1. Note that this changes the dimensionality of the signal, in contrast 4 Table 1. Comparison between the formulations of the forward, marginal, and posterior distributions of DDPM and Blurring Diffusion (BD) against our Scale Space Diffusion. For Blurring Diffusion, we use ‘a’ instead ofαused in their paper, to not confuse it with theαin DDPM. Note that BD applies a change of variableu t = V T x t , whereV T is the Discrete Cosine Transform, before performing diffusion, i.e., diffusion in frequency space. BD and DDPM have equivalent formulations whena t = √ ̄α t andσ t = √ 1− ̄α t . While the formulations share structural similarities, Scale Space Diffusion extends the framework to support general linear degradations (e.g., downscaling), which are not handled by DDPM or BD. We highlight analogous terms with consistent background colors for easier correspondences across different formulations. Legend: Forward mean,varMarginalmean,varPosteriormean,var DistributionsDDPM [16]Blurring Diffusion [18]Scale Space Diffusion Forward q(x t |x t−1 ) x t = √ α t x t−1 + √ 1− α t ε, ε∼N (0, I) u t =a t|t−1 u t−1 +σ t|t−1 ε, ε∼N (0, I) wherea t|t−1 = a t a t−1 , σ t|t−1 =σ 2 t −a 2 t|t−1 σ 2 t−1 x t =μ t|t−1 +η t|t−1 ,η t|t−1 ∼N (0,Σ t|t−1 ) whereμ t|t−1 =M t x t−1 =M 1:t (M 1:t−1 ) −1 x t−1 , Σ t|t−1 =Σ t −M t Σ t−1 M T t Marginal q(x t |x 0 ) x t = √ ̄α t x 0 + √ 1− ̄α t ε, ε∼N (0, I) u t =a t u 0 +σ t ε, ε∼N (0, I) x t =μ t +η t ,η t ∼N (0,Σ t ) whereμ t =M 1:t x 0 ,Σ t =σ 2 t I Posterior q(x t−1 |x t ,x 0 ) x t−1 = ̃μ t−1 + ̃ β t−1 ε, ε∼N (0, I) where ̃ β t−1 = 1− ̄α t−1 1− ̄α t β t = 1 1− ̄α t−1 + α t 1− α t −1 , ̃μ t−1 = √ α t (1− ̄α t−1 ) 1− ̄α t x t + ̄α t−1 β t 1− ̄α t x 0 = ̃ β t−1 √ α t 1− α t x t + ̄α t−1 1− ̄α t−1 x 0 u t−1 =μ t→t−1 +σ t→t−1 ε, ε∼N (0, I) whereσ 2 t→t−1 = 1 σ 2 t−1 + a 2 t|t−1 σ 2 t|t−1 ! −1 , μ t→t−1 =σ 2 t→t−1 a t|t−1 σ 2 t|t−1 x t + a t−1 σ 2 t−1 x 0 ! x t−1 =μ t→t−1 +η t→t−1 ,η t→t−1 ∼N (0,Σ t→t−1 ) where Σ t→t−1 = (Σ −1 t−1 +M T t Σ −1 t|t−1 M t ) −1 = σ 2 t−1 I− σ 4 t−1 σ 2 t M T t M t , μ t→t−1 =Σ t→t−1 (M T t Σ −1 t|t−1 x t + Σ −1 t−1 μ t−1 ) = μ t−1 + σ 2 t−1 σ 2 t M T t (x t − M t μ t−1 ) with previous diffusion formulations. However, since we make no assumptions about dimensionality, our framework remains valid regardless. Furthermore, with this choice of M t , we also define a resolution scheduler(t)that maps diffusion timestep (t) to the corresponding resolution, such that the resolution monotonically decreases astincreases (Fig. 5). Refer to the supplementary Sec. 8.2.1 for another degradation example. Calculating the Transpose.Since operators like image resizing are implicit, we may not have the matrix form available, making it non-trivial to ap- ply the transposeM T t .To address this, we use a vector-Jacobian product of the function callM t (·), i.e.,M T v = torch.autograd.grad(M t (x), x, gradoutputs=v)[0], which, for linear operators, does not depend onx, as shown in Algo. 2. This computes the derivative of the inner product⟨v,M t x⟩with respect tox, i.e.,∇ x ⟨v,M t x⟩ = M T t v. Sampling from a Non-Isotropic Gaussian Distribution. A neat trick to sample from a non-isotropic Gaussian dis- tribution with covariance matrixΣis to first sample a standard Gaussian noiseε ∼ N (0, I), and then multi- ply with the square root of the covariance matrix, so that Σ 1 2 ε∼N (0, Σ 1 2 I(Σ 1 2 ) T ) =N (0, Σ). In our case, we need to sample noise fromΣ t→t−1 from Eq. 6, which depends on implicit operatorsM t (·)andM T t (·). Thus, we need a way to applyΣ t→t−1 (·)implicitly to a standard Gaussian noise ε. For this purpose, we use the Lanczos algorithm [11,23], which numerically computesA(x)given an implicit sym- metric linear operatorA(·)and vectorx. When the Lanczos algorithm is applied with a square root spectral function over the eigenvalues, we can obtainA 1 2 x. In our case, this gives η t→t−1 = Σ 1 2 t→t−1 ε∼N (0, Σ t→t−1 ) as shown in Algo. 3. Algorithm 1 Implicit Linear Operator # M resizes and attenuates signal x def M(x, a_t, a_t_minus1, size_out): return (a_t / a_t_minus1) * F.interpolate( x, size=size_out, mode="bilinear", align_corners=False, antialias=True) Algorithm 2 Implicit Linear Operator’s Transpose # M_T applies the transpose of M on v def M_T(M, v, a_t, a_t_minus1, M_input_shape): size_out = v.shape()[-2:] with torch.enable_grad(): x = torch.zeros(M_input_shape, requires_grad=True) out = M(x, a_t, a_t_minus1, size_out) # calculate MˆTv = d<v,Mx>/dx (g,) = torch.autograd.grad(out, x, grad_outputs=v, retain_graph=False) return g Algorithm 3 Sampling Non-Isotropic Gaussian Noise # Sample noise from posterior covariance Sigma_t-->t-1 def sample_non_isotropic_noise(M, M_T, sigma_t, sigma_tminus1, x): rho = (sigma_tminus1 ** 2) / (sigma_t ** 2) # Define matvec operator A A = lambda v: v - rho * M_T(M(v)) # Lanczos approximation of Aˆ1/2v y = lanczos(A, x, f=lambda l: l.sqrt()) return sigma_tminus1 * y 4.2. Training and Sampling To reverse the diffusion process using Eq. 6, our model must predictμ t−1 = M 1:t−1 x 0 , which, with our choice ofM t , reduces to a scaled version of an imagex 0 at res- olutionr(t − 1). To train such a model, using our Gen- eralized Linear Diffusion Process, we need to first adapt L simple . When predictingx 0 , the loss becomesL simple = E x 0 ,t,ε [s 2 (t)∥x θ 0 (x t ,t)− x 0 ∥ 2 2 ] , wheres 2 (t)is the signal to noise ratio, as shown in [34]. In Min-SNR-γ[14] instead 5 Algorithm 4 Train def train_iter(x, t, a_t_minus1, model, opt): opt.zero_grad() t_minus1 = (t-1).clamp(min=0) # clean image at res r(t-1) = M_1:t-1(x) / a_t-1 x_start_t_minus1 = cummulative_M[t_minus1](x)/a_t_minus1 # Using Eq.3 x_t = diffuse(x, t) pred_x_start_t_minus1 = model(x_t, t) # Using Eq.7 loss = min_snr_5(t) * ((pred_x_start_t_minus1 - x_start_t_minus1) ** 2) loss.backward() opt.step() return loss Algorithm 5 Sampling # get x_t-1 given x_t def sample_iter(x_t, t, model): pred_x_start_t_minus1 = model(x_t, t) mu_t_minus1 = a_t_minus1 * pred_x_start_t_minus1 # Using Eq.6 posterior_noise = calculate_posterior_noise(t) posterior_mean = calculate_posterior_mean(x_t, mu_t_minus1, t) x_t_minus1 = posterior_mean + posterior_noise return x_t_minus1 of thes 2 (t)weighting, they usemin(s 2 (t),γ), withγ = 5, which improves the performance ofx 0 parameterization sig- nificantly. Following this, our loss function evaluates to: L =E x 0 ,t,ε h min(s 2 (t),γ) x r(t−1) 0,θ (x t ,t)− 1 a t−1 M 1:t−1 x 0 2 2 i (7) where we predict an unscaledμ t−1 using a neural network x r(t−1) 0,θ (Algo. 4). Note that the input resolutionr(t)ofx t may be smaller than the resolution of the output atr(t−1)as seen in Fig. 3 (left). In standard diffusion training, timesteps are simply sampled uniformly for each batch. However, this is non-trivial in our setting because the(r(t),r(t − 1))pairs may not match. To solve this, we first uniformly sample a singlet, and ifr(t) = r(t− 1), then uniformly sample the batch size number oft i ’s that haver(t i ) = r(t). Otherwise, ifr(t) ̸= r(t− 1), then we fill the entire batch with the samet, so there is no size mismatch. Since not all t’s change resolution, many of theM t ’s can be replaced by scalar multiplication with (a t /a t−1 ) = √ α t . For sampling (Algo. 5), we start from a random Gaus- sian noise at the lowest resolutionr(T ). Our modelx r(t−1) 0,θ predicts a clean image at the next resolutionr(t− 1), using which we can calculateμ t−1 and denoise using the poste- rior distribution (Eq. 6). This also involves sampling from Σ t→t−1 , which may not be isotropic, and hence we use Algo. 3 to sample noise from this distribution. Eq. 6 is equivalent to DDPM sampling whenr(t) = r(t− 1), so the non-isotropic noise sampling can be replaced with normal torch.randn() calls for resolution-preserving steps. 4.3. Architecture We adapt the UNet architecture from Ablated Diffusion Model (ADM) [10] to design our proposed model Flexible- UNet (Flexi-UNet), which supports multi-resolution inputs and outputs to fully realize the scale-space formulation. Be- cause Scale Space Diffusion embeds a resizing operator in the forward diffusion process, the spatial resolution ofx t varies across timesteps, and the reverse model must therefore operate on variable-sized noisy states and sometimes predict a higher-resolution output at the next scale (Fig. 3). A standard diffusion model, such as ADM [10], is trained to operate at a single fixed resolution throughout all timesteps, and even multiresolution UNet variants only process multiple scales within a fixed-resolution diffusion process. In contrast, SSD requires an architecture that na- tively handles different input resolutions across timesteps. To address this, we explore two architectural designs. Full UNet (Single Path). The base UNet architecture in- herently supports variable-size inputs and outputs, and in principle can operate on any spatial resolutionR×Ras long as the kernel sizes, strides, padding, and pooling operations produce valid feature maps at every layer. However, this de- sign has two key limitations for Scale Space Diffusion. First, it requires the input and output resolutions to be equal. In our setting, certain timesteps involve a resolution transition, which would require the model to output at a higher resolu- tion. To handle this, the input must be manually upsampled before entering the UNet whenever such a transition occurs. Second, the depth of the UNet determines how many distinct spatial scales it can represent. For a UNet with Ldownsampling blocks, the smallest internal resolution is R 2 L−1 , which fixes the total number of scales toL. This number is typically small and does not grow with the input resolution. For example, the ADM architecture uses 4 fea- ture map resolutions for64×64, 5 for128×128, and 6 for 256×256, meaning that across all these models the number of downsampling stages remains fixed at 4. Thus, even at higher resolutions, the network cannot represent more than a handful of scales, limiting the usefulness of a scale-space formulation where many more levels naturally exist. Flexi-UNet. These limitations motivate our proposed archi- tecture, Flexi-UNet, where different subsets of UNet lay- ers are dynamically activated based on the input resolution. High-resolution inputs traverse the full UNet, while lower- resolution inputs are routed only through the deeper layers, effectively bypassing the early and late blocks. Since each block expects a specific channel dimensionality, we insert 1×1conv layers to map the input features to the appropriate channel size while preserving spatial resolution. For denoising steps that do not involve a resolution change, the active pathway through the UNet remains sym- metric, using the same number of downsampling and upsam- pling blocks. When a resolution increase is required, the 6 Active Layers Inactive Layers Input, Noisy Image, at Diffusion step and scale Predicted, Clean Image at scale from Resolution change Diffusion step Same resolution Diffusion steps (See Eq. (3)) Forward Sampling: 0000 Input Predicted Skip connections (See Eq. (6)) Reverse Sampling: TrainingInference Architecture Details Figure 3. Overview. Left: During trainingx t ’s at resolutionr(t)are sampled using Eq. 3, and our model is trained to predict clean image x r(t−1) 0,θ using the loss as in Eq. 7. Our Flexi-UNet is able to process both resolution-preserving and resolution-changing steps at multiple resolution using only parts of the network. Right-top: During sampling, Eq. 6 is used to progressively denoise and upsample to generate images. Right-bottom: Our Flexi-UNet has additional 1×1 Conv layers to take inputs at any UNet encoder block and get outputs form any decoder blocks. For resolution changing, the skip connections are fed with zero-filled tensors. pathway becomes asymmetric: the model uses one additional upsampling block compared to the number of downsampling blocks encountered. In these cases, the skip connections that would normally come from the bypassed encoder blocks are replaced with zero tensors (Fig. 3). This design allows the model to share parameters across resolutions while support- ing valid diffusion dynamics during resolution transitions. 5. Experiments Datasets. We perform experiments and analyze the perfor- mance of Scale Space Diffusion on the CelebA dataset[25] and the ImageNet dataset [8]. The CelebA dataset con- sists of around 200K training images, while the ImageNet dataset contains around 1.3 million images from 1000 differ- ent classes. We use JPEG images for these datasets. We con- duct experiments at64×64resolution for both CelebA and ImageNet. We additionally show experiments on128×128 and256×256for CelebA dataset. The CelebA experiments helps us understand our method’s scalability with increasing resolutions, while ImageNet helps in evaluating the model’s ability to learn complex and diverse distributions. Unconditional Image Generation. We analyze and eval- uate Scale Space Diffusion on unconditional image genera- tion, as it allows us to clearly study how scale-space theory integrates with the diffusion process. Implementation Details. We use the ADM [10] repository as our base codebase and build our baselines (DDPM [16], Blurring Diffusion [18]), as well as Scale Space Diffusion on top of it. For DDPM, we consider two standard parametriza- tions as our baselines, theε-prediction, and thex 0 -prediction formulation. We train the baseline model with Min-SNR-γ weighting for thex 0 -parametrization to ensure an accurate 푟=64 푟=32 푟=16 푟=8 풓=ퟐퟓퟔ 푟=64 푟=64 푟=256 풓=ퟏퟐퟖ 푟=128 푟=32 푟=16 푟=8 Figure 4. Visual samples. Top: ImageNet-64 unconditional gen- eration. For the top-most sample we also show model prediction at various scales (8, 16, 32, 64) during SSD. Bottom: CelebA-256 unconditional generation. For the top-most sample we also show model predictions at various scales (8, 16, 32, 64, 128, 256). ScheduleFIDTime (hrs) equal9.6412.88 ConvexDecay211.0311.71 ConvexDecay0.54.8713.81 SigmoidLikeDecay37.0813.06 TanhLikeDecay38.0912.50 Figure 5. Resolution Schedules. Mapping diffusion timestepstto resolutionracross 4 scales. Both discrete and continuous variants are shown. The right shows FIDs at 500k iterations (batch size 8). comparison to Scale Space Diffusion. We implement Blur- ring Diffusion using the pseudo-code provided in their paper. For the diffusion process, we follow the linear noise schedule proposed in DDPM [16] and use the standard set- ting of 1000 timesteps. For training, we use AdamW [22,26] optimizer with a fixed learning rate. We conducted all our 7 Table 2. Main Results. Unconditional image generation results on CelebA dataset over multiple resolutions. Training time is specified in hours. Average GFlops per iteration. Effective batch size is 128 for resolutions 64 and 128, 64 for resolution 256. Here BD refers to Blurring Diffusion [18] and all SSD models use our Flexi- UNet architecture. Method CelebA-64 (1M iters)CelebA-128 (300K iters)CelebA-256 (300K iters) FIDTimeGFlopsFIDTimeGFlopsFIDTimeGFlops DDPM-ε2.2270.3060.054.1650.50132.305.5287.31497.03 DDPM-x 0 2.9870.71–3.5050.33–5.4787.33– BD2.0671.79–3.67–4.7688.08– SSD (2L)2.1462.6350.61– SSD (3L)3.6156.1344.276.5331.7187.387.7959.00317.36 SSD (4L)4.2852.3838.48–10.5251.70272.98 SSD (5L)–10.4725.4166.72–237.70 SSD (6L)–13.5042.88209.69 Table 3.ImageNet- 64 Results. Uncondi- tional image generation results on ImageNet-64 dataset. MethodFID DDPM-ε12.82 DDPM-x 0 13.07 Blurring Diffusion15.34 SSD (2L)13.08 SSD (4L)17.89 Figure 6. Temporal Scaling. Training time of proposed SSD with our Flexi- UNet across multiple resolutions. 64128256 Resolution 20 40 60 80 Training time (hrs) Training time across scales and resolutions DDPM Ours (multi-scale) experiments on NVIDIA H100 and NVIDIA RTX A4000 GPUs. We maintained consistent combinations of learning rate and batch-size across dataset and resolutions. For64×64 and128×128, we used an effective batch size of 128, and trained the models either on a single H100, or on4A4000 GPUs, with a per-GPU batch-size of 32. For256×256, we used an effective batch size of 64 due to memory constraints and trained them on2H100s with a per-GPU batch size of 32. Our learning rate was set to1×10 −4 for the64×64and 128×128models, and5×10 −5 for256×256model, follow- ing linear learning rate scaling. Evaluation. We evaluate our models using the exponential moving average (EMA) weights with a decay rate of 0.9999. We assess the quality of generated images by computing FID [15] scores on 50k samples w.r.t. the training set. We further compare Scale Space Diffusion with the DDPM base- line model in terms of training time, and FLOPs (Floating Point Operations) per forward pass. In addition to FLOPs, we report the sampling latency as the total time to generate a single image. All speed and compute metrics are measured on a single NVIDIA GH200 node. Main results. Our results are presented in Table 2 and 3. We train the baseline models and Scale Space Diffusion model for 1 million iterations for CelebA-64 and 300k iterations for CelebA-128 and CelebA-256. We report the total train- ing time, average GFLOPs per iteration, and the FID value. We notice that increasing the number of levels significantly decreases training time and GFLOPs. SSD (6L) at 256 res- olution takes less than half the time as the baseline DDPM. Table 3 shows that SSD, trained for 1 million iterations, Table 4. Architecture Ablation. FID (at 500K iterations on CelebA-64) and Inference time (in secs/generation, 1000 steps, batch size=1, on 1×A4000) of network architecture variants at 2 and 4 levels. Method FIDInference time res. 64res.64res. 256 Full Unet, 2L2.3316.1943.07 Flexi-UNet, 2L2.2615.3838.99 Full Unet, 4L4.9016.2834.74 Flexi-UNet, 4L4.8713.4331.08 achieves comparable performance to baselines even on the harder ImageNet-64 benchmark. Figure 6 shows that training time for SSD scales well with increasing resolution. Please refer to the supplementary Sec. 8.4 for more comparisons. Qualitative results. We present qualitative results of our method on ImageNet-64 with SSD (4L) and CelebA-256 with SSD (6L) in Fig. 4. We also show multiple intermediate predictions of the model in SSD. Individual effectiveness of our mathematical formulation vs architecture. In the supplementary Sec. 8.2, we first show that our Generalized Linear Diffusion Process works, albeit suboptimally, even without Flexi-UNet, and also with an alternate degradation. Approximating anisotropic gaussian noise with isotropic leads to saturation artifacts, showing the need for our anisotropic sampling. Secondly, we show that Flexi-UNet is effective on its own for other formulations of iterative multi-resolution pixel-space generation, albeit suboptimal. We do this by applying it to approximate multi- resolution diffusion as in PyramidalFlow [6, 19, 20]. Resolution Schedule.r(t)specifies the spatial resolution as a function of diffusion timestep. We present 5 different reso- lution schedules in Figure 5 (left) at 4 levels(64, 32, 16, 8) (refer to the supplementary Sec. 8.3 for details) and note their effects for a CelebA-64 model in Fig. 5 (right). We observe that schedules that spend the least number of timesteps at the higher resolutions train the fastest, but also yield the worst FID (i.e. ConvexDecay2). In contrast, the model trained with ConvexDecay0.5, which spends the most steps at the highest resolution, achieves the best FID, but requires the longest training time. We use this for all our experiments. Full UNet vs Flexi-UNet. In Table 4, we observe that Flexi- UNet has slightly better FID for both 2L and 4L, while being faster than the Full UNet. Hence, we use Flexi-UNet. Sampling. Use of Lanczos has negligible overhead. Further, SSD does not suffer from performance drop, like DDPM, on sampling steps reduction. (Refer supplementary Sec. 8.5.) Conclusion. We showed that diffusion models and scale spaces share an information hierarchy, and we quantified this connection mathematically. Observing that highly noised diffusion states contain only low-resolution information, we introduced a generalized family of diffusion models that embeds scale-space structure into the forward process, yield- ing Scale Space Diffusion. To realize this in practice, we proposed the Flexi-UNet architecture and demonstrated its effectiveness on unconditional image generation. 8 Acknowledgment. This research is based upon work sup- ported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily repre- senting the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and dis- tribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. 9 Scale Space Diffusion Supplementary Material Figure 7. Animation of the predicted clean imagex r(t−1) 0,θ over the generation process for gradual downsizing degradation operator in SSD framework. (Best viewed in Adobe Reader). Contents 6. Clarifications1 7. Future Works1 8. Additional Material2 8.1. Hyperparameters . . . . . . . . . . . . . . .2 8.2. Parts of our Approach . . . . . . . . . . . .2 8.3. Resolution Schedules . . . . . . . . . . . . .4 8.4. More Comparisons . . . . . . . . . . . . . .5 8.5. Quantitative Results . . . . . . . . . . . . .5 8.6. Qualitative Results . . . . . . . . . . . . . .6 9. Mathematical Derivations20 9.1. Forward Transition . . . . . . . . . . . . . .20 9.2. Posterior Distribution . . . . . . . . . . . . .20 9.3. Posterior Under Isotropic Marginals . . . . .21 6. Clarifications We add some clarifications for our main paper here. 1. All variable names used in Algo. 1, 2, and 3 have their usual meanings. For example,at,atminus1de- note the signal coefficientsa t ,a t−1 respectively, while sigmat,sigmatminus1denote the noise schedule σ t ,σ t−1 respectively.sizeoutdenotes the (height, width) of the output of M operator. 7. Future Works The focus of this work has been to analyze the connection be- tween diffusion models and scale space theory, while propos- ing to merge them using Scale Space Diffusion with Flexi- UNet. We do not use any advanced techniques to tune our framework or architectures for the most optimal performance. Instead, we use the standard hyperparameters from the base Figure 8. Animation of the noisy intermediate statex t over the generation process for the gradual downsizing degradation operator in SSD framework. (Best viewed in Adobe Reader). codebase to keep the choices simple and the number of ex- periments under check given the expense of each training. The use of advanced techniques is out of scope for this work given the conference length manuscript. However, there are multiple future exploration directions which have high potential for improvement in performance. For example, adapting newer diffusion samplers instead of using DDPM-style samplers can improve both performance and inference speeds. Similarly, progressive curriculum learning for different layers or resolutions, as done by works with multi-resolution trainings [12,21], should also yield improvement in training optimization. Why not use a Transformer-based architecture? The two most popularly used architectures in diffusion are – convolu- tional UNet [33] based ADM [10], and vision transformer (ViT) based DiT [31]. Another popular architecture is U- ViT [4], that combines the skip connections from UNet with a ViT architecture. One thing to note is that, DiT was de- signed for latent spaces and hence did not take into consid- eration the blowing up of the quadratic complexity of the attention mechanism when applied in the pixel space [6]. U- ViT acknowledges this issue, and explicitly works in a latent space for higher resolutions. Newer works like HDiT [7] try to mitigate this issue using neighborhood attention instead of global attention is all layers. But such non-trivial design decisions in the architecture can develop into confounding factors. Since our goal is to understand how scale-spaces can be integrated into diffusion models, for simplicity we stick to the standard ADM base architecture, a widely used pixel and latent diffusion architecture [32]. Nonetheless, for future work, a similar integration of scale-space theory should also be explored with transformer based architectures. 1 HyperparameterCelebA-64CelebA-128CelebA-256ImageNet-64 Noise ScheduleLinearLinearLinearLinear Denoising Steps1000100010001000 OptimizerAdamWAdamWAdamWAdamW Batch Size12812864128 Learning Rate0.00010.00010.000050.0001 Number of Iterations1 million300k300k1 million Table 5. Hyperparameters for all datasets. Implementation ChoiceDDPM-εDDPM-x 0 Blurring DiffusionSSD Reverse Process Variancefixed-largefixed-largefixed-smallfixed-small LossL simple L simple + Min-SNR-5L simple L simple + Min-SNR-5 Table 6. Additional implementation details. Table 7. Inference time. By default we use DDPM sampling, but we also show 25 † steps DDIM [35] speeds. Method#StepsSpeedup (Inference Time)FID DDPM-x 0 10001.00×2.98 2504.18×14.00 25 † 38.87×4.70 DDPM-ε 10001.05×2.22 2504.18×11.02 25 † 38.06×3.76 SSD(Flexi-UNet, 2L) 10001.18×2.14 2504.80×2.87 SSD(Flexi-UNet, 4L) 10001.58×4.28 2505.91×4.90 Table 8. Inference time (in secs) per gen at 64 res (1000 steps, bs=1, 1 A4000): Lanczos sampling vs. torch.randn call. MethodSSD (2L) SSD (4L) w/ Lanczos15.3813.43 w/o Lanczos15.3513.40 8. Additional Material 8.1. Hyperparameters The set of hyperparameters that we use for each dataset is summarized in Table 5. We also note additional experimental details in Table 6. 8.2. Parts of our Approach Our approach consists of two parts. The first part is the Scale Space Diffusion mathematical formulation and the second part is the Flexi-UNet architecture. In the main paper, we have presented the combination of both parts as our complete approach. But here we also want to show that each part is effective on its own. So, in Section 8.2.1, we explore whether the mathematics behind SSD can be applied without a modified architecture, while in Section 8.2.2, we check if Flexi-UNet can be used without our mathematical framework, summarized in Table 9. 8.2.1. Validity of SSD One way to verify whether SSD framework works without using a modified architecture is to assume that the actual states of the diffusion model are at a certain resolution, but when passing through the model, we resize them to the model Table 9. Parts of our approach, and validity of each part. SSDFlexi-UNet Section 8.2.1✓✗ Section 8.2.2✗✓ Main paper✓ Table 10. Results of only SSD (w/o Flexi-UNet) on CelebA-32. (Here we resize the inputs to the model input resolution.) MethodFID DDPM-ε2.85 SSD (w/o Flexi-UNet, 5L)5.55 SSD (w/o Flexi-UNet, gradual downsizing)4.10 input size. Similarly, the model outputs are resized to the required output resolution before applying losses. We test this with CelebA-32 dataset just to check the correctness of SSD. For this, we use a DDPM reimplementation (not ours) optimized for resolution 32 images [39], since ADM’s codebase does not support that resolution, and a smaller resolution is faster to verify on. We train these models for 300 epochs and use 5 steps of resolutions (2, 4, 8, 16, 32). We note their FIDs in Table 10. Alternative Degradation. All the degradations used in this work till now have been 2×downsampling. However, given the general nature of the theory, it is not limited to just this choice. Here we test using a gradual downsizing instead of 2×downsizing steps. In this degradation, whenever the resolution changes, it does so by only 1 pixel at a time. We try going from2→ 32. We report its FID in Table 10. We show some static visual results in Fig. 9. We show some interesting animated visualizations (view in Adobe Reader) in Fig. 7, and Fig. 8. Effect of Isotropic Approximation. Another thing we wanted to test was whether we could approximate the non- isotropic Gaussian noise sampling (Algo. 3) with isotropic Gaussian noise. For testing purposes, during the generation procedure (of the gradual downsizing degradation case), in the resolution changing steps, we first use Algo. 3 to sample non-isotropic noise, and then find the mean and variance over the height and width dimensions of this noise tensor. Instead of using the sampled non-isotropic noise for the stochasticity in Eq. 6, we instead use an isotropic noise sampled using torch.randn()with the calculated mean and variance. As seen in Fig. 10, this leads to the colors becoming flat and saturated, despite having facial structures. This shows that the assumption of isotropic covariance for the reverse process may not actually be valid, as assumed in [1]. And we need to sample from non-isotropic Gaussians depending upon the linear operator. 2 (a) Generated Samples (b) Predicted clean imagesx r(t−1) 0,θ (c) Noisy statesx t Figure 9. Visual results of SSD with gradual downsizing degradation (1 pixel downsizing instead of 2× downsizing) Figure 10. Effect of using isotropic noise instead of non-isotropic noise in the reverse diffusion process of SSD. 8.2.2. Effectiveness of Flexi-UNet In this section, we demonstrate that Flexi-UNet can natu- rally accommodate different formulations of the diffusion process to support multi-resolution inputs and outputs. To do so, we build upon previous works that introduce cor- rective noise when an upsampling operation is performed in the diffusion process [6,19,20]. We implement these ideas within Flexi-UNet to both validate the flexibility of our architecture and quantify the computational benefits ob- tained from operating across resolutions. A key challenge addressed in these works is the distribution mismatch that arises when a noisy latent is upsampled. Prior works [19] show that applying a2×nearest-neighbor upsampling step produces a block-structured covariance that deviates from the forward diffusion trajectory. Their solution injects struc- tured noise and identifies an adjusted timestep that realigns the upsampled latent with the original process. While this motivates our analysis, our setting is different from this in two ways: a) we operate entirely in pixel-space rather than latent space, and b) we consider multiple (more than one) upsampling stages throughout the denoising process. With these conditions in mind, our setup is as follows: Letx r t be a valid DDPM forward state at a timestep t for resolution r: x r t ∼N √ ̄α t x r 0 , (1− ̄α t )I , Let x R t = Upsample(x r t ), x R t ∼N √ ̄α t Ux r 0 , (1− ̄α t )U ⊤ where U is the Upsampling matrix. Letx R s be a valid DDPM forward state at some other timestep s for resolution R: x R s ∼N √ ̄α s x R 0 , (1− ̄α s )I , The upsampled statex R t has covariance proportional to U ⊤ , which differs from the isotropic Gaussian noise as- sumed by the DDPM forward process at resolutionR. To correct this mismatch, we add corrective noise and roll back to a previous timestep. Let the corrected sample be ̃x R t = ax R t + bz, z ∼N 0, I − cUU ⊤ . Then the distribution of ̃x R t is ̃x R t ∼N a √ ̄α t Ux r 0 , a 2 (1− ̄α t )U ⊤ + b 2 I − cUU ⊤ . We make an approximation to match the mean and co- variance of ̃x R t to x R s a 2 ̄α t = ̄α s b 2 = 1− ̄α s a 2 (1− ̄α t ) = b 2 c (8) 3 Table 11. Results of Flexi-UNet (w/o SSD) on CelebA-64. Com- puted at 500k iterations. Inference time is computed as the average time (in minutes) to generate a batch of samples (256 samples). MethodFIDInference Time Flexi-UNet (w/o SSD, Equal, 2L)2.4415.52 Flexi-UNet (w/o SSD, Equal, 4L)5.7913.32 Flexi-UNet (w/ SSD, ConvexDecay0.5, 2L)2.2614.98 Flexi-UNet (w/ SSD, ConvexDecay0.5, 4L)4.8711.20 Solving the three equations mentioned in Equation 8, gives us c = a 2 (1− ̄α t ) b 2 = ̄α s (1− ̄α t ) ̄α t (1− ̄α s ) (9) We first obtain the value of ̄α s that satisfies Equation 9 for a given choice ofc, and then obtain the correspond- ing timesteps. We sweep through values ofcin range 0 ≤ c ≤ 0.25(as mentioned in [19]) to produce different values ofs. We compute all such candidate values ofsand pick the bestsempirically. For each value ofc, we gener- ate the corrected samples ̃x R t and the corresponding DDPM forward samplesx R s using 2048 training images. We then compute the Jensen–Shannon divergence between these dis- tributions to obtain the final backtracking indexsas the one that produces the minimum JS divergence. This experiment serves as our validation of our proposed method Flexi-UNet. During training, we follow a specific resolution schedule, so that for each timestept, the model receives a statex r(t) t . To support distribution correction, we additionally include timesteps, ̃x R t to the training samples. During inference, the denoising process follows the stan- dard reverse diffusion trajectory, with the following change: whenever the process reaches a timestep that has an upsam- pling step, the model rolls back to a slightly earlier timestep and continues denoising from that point at the higher reso- lution. This experiment illustrates the computational advan- tages of operating at multiple resolutions, using an architec- ture like Flexi-UNet, as a lot of the early denoising occurs at lower spatial resolutions. However, this setup requires rollback around each upsampling point, creating overlap- ping steps in the reverse process. While this model provides computational savings, there is an additional overhead of denoising for additional timesteps. In Table 11, we show the FID values obtained for this experiment after training the model for 500k iterations. We compare the performance of Flexi-UNet trained with SSD to Flexi-UNet trained without SSD. We observe that Flexi- UNet with SSD has better FID values, while also being faster at inference. 8.3. Resolution Schedules Here, we will define the functions that we used for the res- olution schedules. We define what the resolution of the image should be given the diffusion timestept, using a func- tionr(t). As shown in Fig. 5, we use a discrete version of the resolution schedule, but it is based on a continuous function. Suppose for the discrete version we use a list of resolutions[r min , 2r min ,..., 2 n−2 r min , 2 n−1 r min ]wherer min is the smallest resolution andnis the number of resolutions. For the continuous version, let’s first define normalized time τ = t/(T − 1), whereTdenotes the number of diffusion states. Then the normalized time to resolution schedule is defined as: r cont (τ ) = r min · 2 (n−1)f (τ ) wheref (τ )is the exponential schedule function that works as the multiplier to the exponent of2. For example, when f (τ ) = 0, thenr cont (τ ) = r min , while whenf (τ ) = 1, then r cont (τ ) = r max = 2 n−1 r min . For the discrete version, we want to similarly sample from R = [r min , 2r min ,..., 2 n−2 r min , 2 n−1 r min = r max ], using the same schedule but over these discrete values. So, here we instead index the schedule functioni(τ )that gives the index to select from R given τ . r(τ ) = R[i(τ )] Similar tof, wheni(τ ) = 0, we haver(τ ) = r min , and wheni(τ ) = 1,r(τ ) = r max . Now we can introduce our schedules. 8.3.1. Equal This is the easiest linear schedule. • Continuous: f (τ ) = 1− τ • Discrete: i(τ ) = n− 1−⌊nτ⌋ 8.3.2. ConvexDecay γ With aγ > 0parameter, this function can simulate a convex or concave function depending on this parameter. • Continuous: f (τ ) = 1− (1− τ ) γ • Discrete: i(τ ) = n− 1−⌊nf (τ )⌋ Forγ > 1, it shows slow decay first, then faster, while for γ < 1, fast decay first, then slower. 8.3.3. TanhLikeDecay γ Here we wanted a function that looks liketanh(·)function, which is steep at the highest and the lowest timesteps but is flat in the middle. This essentially spends more time in the middle resolutions. We approximate this using a polynomial. First, we define a polynomial over a variableu ∈ [−0.5, 0.5] as follows: x(u,γ) = sign(u)|u| γ + 0.5 p(x) =−2x 3 + 3x 2 − 0.5 The polynomialp(x)is monotonically increasing in the range of[−0.5, 0.5]forx∈ [0, 1], whilex(u)is a function that looks like thetanh(·)function but is centered around 0.5. Essentially,p(x(u))looks like thetanhshape and is 4 centered around the origin, but has varying range dependent onγ. We want this function to be equal to 1 atu = 0.5and -1 atu =−0.5. To achieve that, we normalize this function: ˆp(u,γ) = p(x(u,γ)) p(x(0.5,γ)) Finally, to shift this function from[−0.5, 0.5]→ [−1, 1]to [0, 1]→ [0, 1], we apply the following transformation: tanhlike(u,γ) = 0.5· ˆp(x(u− 0.5,γ)) + 0.5 Figure 11. Visualization of tanhlike(·) for different γ’s. Now, based on this definition, we can define the schedule. • Continuous: f (τ ) = 1− tanhlike(1− τ,γ) • Discrete: i(τ ) =⌊nf (τ )⌋ 8.3.4. SigmoidLikeDecay γ Here we want a simoid-like curve, i.e., steep in the middle while flatter at the beginning and the end. We can such a curve by invertingˆp(·). Following similar stretching and normalization, we can define another function that goes from [−0.5, 0.5]→ [−1, 1] as: ˆ h(u) = (0.5· ˆp) −1 (u,γ) (0.5· ˆp) −1 (−0.5,γ) . Using the same shifting to transform[−0.5, 0.5]→ [−1, 1] to [0, 1]→ [0, 1], we have: sigmoid like(u,γ) = 0.5· ˆ h(x(u− 0.5,γ)) + 0.5 Finally, we define the schedule. • Continuous: f (τ ) = 1− sigmoidlike(1− τ,γ) • Discrete: i(τ ) =⌊nf (τ )⌋ 8.4. More Comparisons Upsampling Diffusion Probabilistic Models (UDPM) [1]. The mathematical formulation and implementation of Table 12. FID comparison of SSD and super-resolved DDPM (64 res.) using an OpenImages pretrained 4× LDM. MethodFID SSD (3L, res. 256)7.79 low-res diffusion (res. 64) + super-res (4×) 7.91 Table 13. Inference time per batch:SSD and LDMs (1000 steps, bs=32, A4000). MethodInference time (secs) SSD (6L, res. 256)495 LDM (res. 256)515 Table 14. Comparison with UDPM at 64 resolution: FID, training (1 H100), and inference speed (1 A4000, bs=256) MethodInferenceInferenceTraining TimeFID stepsTime / batch(250K iters)(250K iters) (in secs)(in hours) DDPM-ε10001018.0717.5752.36 SSD (2L)1000898.7115.6582.68 SSD (4L)1000672.0913.0954.1 UDPM31.8830.587.51 UDMP (w/o Adv. & Perceptual loss)31.8431.6398.61 Figure 12. UDPM generations w/ (left) and w/o (right) adversarial and perceptual losses. SSD can be viewed as a generalization of UDPM. How- ever, UDPM should be considered as a GAN instead of diffusion, as their performance degrades without perceptual and adversarial losses (Table 14) and generations are washed out (Fig. 12, right). Nonetheless, even without extra losses, SSD outperforms UDPM in FID and training time. Further- more, UDPM has not been tested at resolutions higher than 64. Latent Diffusion Models (LDM) [32]. LDMs operate in latent space with different architectures and rely on a compute-intensive pipeline, including two-stage VAE train- ing on large-scale datasets such as OpenImages, making fair comparison difficult. Nonetheless, Table 13 shows SSD (6L) is faster than LDM. Moreover, SSD can be applied in la- tent space as a multi-resolution interpolation degradation, enabling more efficient Scale Space LDMs. Low-res diffusion + super-res. Another baseline could be using a low-resolution generation and applying a super- resolution model over it. Table 12 shows that even with a pretrained LDM super-res model trained for 3×more iter- ations, and on a large dataset, SSD has better performance. Adding multiple stages normally leads to distribution shifts as well as more inference steps coming from different stages. PixelFlow [6], DFM [13]. These are flow-based DiT models with differential equation solver-based sampling, and hence, are hard to compare fairly against. Nonetheless, in a fair setting in section 8.2.2, we recreate a multi-res pixel diffu- sion similar to PixelFlow, and show that using Flexi-UNet formulation outperforms it (Table 11). 8.5. Quantitative Results Number of Inference Steps. In Table 7, we compare in- ference speed across different samplers and denoising steps. We report DDPM sampling with the default 1000 steps, a reduced 250 step process, and DDIM with 25 steps. For our 5 method, we report results with 1000 and 250 DDPM steps, since SSD is formulated in the DDPM setting. We observe that reducing the number of diffusion steps leads to a much larger performance degradation for DDPM-εand DDPM-x 0 than for our approach. This aligns with prior observations that DDPM-εmodels trained with theL simple loss (with fixed sigmas) deteriorate substantially when the number of sam- pling steps is reduced [29], which is reflected in our results as well. We note that SSD degrades far less when reducing the sampling steps to 250, while also providing substantial infer- ence speedups. However, to ensure a fair comparison against baselines, we report all final quantitative results in the paper using the standard 1000-step setting. The speedup column in Table 7 reports the speedup obtained in generating a batch of 256 samples relative to the time taken by DDPM-x 0 . Lanczos sampling overhead. Table 8 shows that the over- head of using Lanczos instead of torch.randn call is negligi- ble, since it is applied only in the resolution-changing steps (1×in SSD (2L), 2×in SSD (3L)). Refer to Table 7 for comparison against DDPM. 8.6. Qualitative Results We show qualitative results of SSD on every setting that we have trained and noted in Tables 2, and 3. For every setting, we show the progression of noisy statesx t and pre- dicted clean imagesx r(t−1) 0,θ during generation, and a grid of generated images. The results start on the next page. 6 Figure 13. Progression of noisy states x t during generation using SSD (Flexi-UNet, 3L) on CelebA-256. Figure 14. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 3L) on CelebA-256. 7 Figure 15. Generated Samples using SSD (Flexi-UNet, 3L) on CelebA-256. Figure 16. Progression of noisy states x t during generation using SSD (Flexi-UNet, 4L) on CelebA-256. 8 Figure 17. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 4L) on CelebA-256. Figure 18. Generated Samples using SSD (Flexi-UNet, 4L) on CelebA-256. 9 Figure 19. Progression of noisy states x t during generation using SSD (Flexi-UNet, 6L) on CelebA-256. Figure 20. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 6L) on CelebA-256. 10 Figure 21. Generated Samples using SSD (Flexi-UNet, 6L) on CelebA-256. Figure 22. Progression of noisy states x t during generation using SSD (Flexi-UNet, 3L) on CelebA-128. 11 Figure 23. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 3L) on CelebA-128. Figure 24. Generated Samples using SSD (Flexi-UNet, 3L) on CelebA-128. Figure 25. Progression of noisy states x t during generation using SSD (Flexi-UNet, 5L) on CelebA-128. 12 Figure 26. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 5L) on CelebA-128. Figure 27. Generated Samples using SSD (Flexi-UNet, 5L) on CelebA-128. Figure 28. Progression of noisy states x t during generation using SSD (Flexi-UNet, 2L) on CelebA-64. 13 Figure 29. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 2L) on CelebA-64. Figure 30. Generated Samples using SSD (Flexi-UNet, 2L) on CelebA-64. Figure 31. Progression of noisy states x t during generation using SSD (Flexi-UNet, 3L) on CelebA-64. Figure 32. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 3L) on CelebA-64. 14 Figure 33. Generated Samples using SSD (Flexi-UNet, 3L) on CelebA-64. Figure 34. Progression of noisy states x t during generation using SSD (Flexi-UNet, 4L) on CelebA-64. Figure 35. Progression of predicted clean images x r(t−1) 0,θ during generation using SSD (Flexi-UNet, 4L) on CelebA-64. 15 Figure 36. Generated Samples using SSD (Flexi-UNet, 4L) on CelebA-64. Figure 37. Progression of noisy statesx t during generation using SSD (Flexi-UNet, 2L) on ImageNet-64. Here we show the progression of 3 samples; each pair of rows corresponds to a single sample. 16 Figure 38. Progression of predicted clean imagesx r(t−1) 0,θ during generation using SSD (Flexi-UNet, 2L) on ImageNet-64. Here we show the progression of 3 samples; each pair of rows corresponds to a single sample. Figure 39. Generated Samples using SSD (Flexi-UNet, 2L) on ImageNet-64. 17 Figure 40. Progression of noisy statesx t during generation using SSD (Flexi-UNet, 4L) on ImageNet-64. Here we show the progression of 3 samples; each pair of rows corresponds to a single sample. Figure 41. Progression of predicted clean imagesx r(t−1) 0,θ during generation using SSD (Flexi-UNet, 4L) on ImageNet-64. Here we show the progression of 3 samples; each pair of rows corresponds to a single sample. 18 Figure 42. Generated Samples using SSD (Flexi-UNet, 4L) on ImageNet-64. 19 9. Mathematical Derivations In this section, we provide derivations for various mathematical results provided in the main paper. 9.1. Forward Transition Theorem 1 (Forward Transition). Let a generalized linear diffusion process be defined by x t = M t x t−1 + η t , η t ∼N (0, Σ t|t−1 ),(10) and suppose the marginal distribution satisfies q(x t | x 0 ) =N (μ t , Σ t ).(11) Then the transition mean and covariance are given by μ t = M 1:t x 0 (12) Σ t|t−1 = Σ t − M t Σ t−1 M T t .(13) Proof. The mean part is true by design of the cumulative linear operator. Here we derive Σ t|t−1 . x t = M t x t−1 + η t , η t ∼N (0, Σ t|t−1 ) = M t (M 1:t−1 x 0 + ε t−1 ) + η t , ε t−1 ∼N (0, Σ t−1 ) = M 1:t x 0 + M t ε t−1 + η t . Hence, Cov(x t | x 0 ) = Cov(M t ε t−1 + η t | x 0 ) = Cov(M t ε t−1 | x 0 ) + Cov(η t ) (independence) = M t Σ t−1 M T t + Σ t|t−1 . Σ t = M t Σ t−1 M T t + Σ t|t−1 , =⇒ Σ t|t−1 = Σ t − M t Σ t−1 M T t . 9.2. Posterior Distribution Theorem 2 (Posterior Distribution). Consider the linear generalized linear diffusion process x t = M t x t−1 + η t , η t ∼N (0, Σ t|t−1 ),(14) with marginals q(x t−1 | x 0 ) =N (μ t−1 , Σ t−1 ),(15) q(x t | x 0 ) =N (μ t , Σ t ).(16) Then the posterior distribution q(x t−1 | x t ,x 0 )(17) is Gaussian: q(x t−1 | x t ,x 0 ) =N (μ t→t−1 , Σ t→t−1 ),(18) with Σ t→t−1 = (Σ −1 t−1 + M T t Σ −1 t|t−1 M t ) −1 ,(19) μ t→t−1 = Σ t→t−1 Σ −1 t−1 μ t−1 + M T t Σ −1 t|t−1 x t .(20) 20 Proof. q(x t−1 | x t ,x 0 ) = q(x t | x t−1 ,x 0 )q(x t−1 | x 0 ) q(x t | x 0 ) = q(x t | x t−1 )q(x t−1 | x 0 ) q(x t | x 0 ) (a) ∝ exp − h (x t − M t x t−1 ) ⊤ Σ −1 t|t−1 (x t − M t x t−1 ) i − h (x t−1 − μ t−1 ) ⊤ Σ −1 t−1 (x t−1 − μ t−1 ) i (b) + h (x t − μ t ) ⊤ Σ −1 t (x t − μ t ) i ! = exp − h x T t Σ −1 t|t−1 x t − (x T t Σ −1 t|t−1 M t x t−1 + (x T t Σ −1 t|t−1 M t x t−1 ) T ) + x T t−1 M T t Σ −1 t|t−1 M t x t−1 i − h x T t−1 Σ −1 t−1 x t−1 − (μ T t−1 Σ −1 t−1 x t−1 + (μ T t−1 Σ −1 t−1 x t−1 ) T ) + μ T t−1 Σ t−1 μ t−1 i + C 1 (x 0 ,x t ) ! = exp − x T t−1 (Σ −1 t−1 + M T t Σ −1 t|t−1 M t )x t−1 + h (x T t Σ −1 t|t−1 M t + μ T t−1 Σ −1 t−1 )x t−1 + ((x T t Σ −1 t|t−1 M t + μ T t−1 Σ −1 t−1 )x t−1 ) T i + C 2 (x 0 ,x t ) ! In Eq. a, we first use Bayes’ rule, and then use the Markov chain assumption. In Eq. b, we then substitute the marginal (Eq. 3) and forward transition (Eq. 4) distributions. Then we start collecting the terms quadratic (red) and linear (blue) inx t−1 . From the quadratic and linear terms, we can complete the square and hence extract the mean and variance of the posterior normal distribution: Σ t→t−1 = (Σ −1 t−1 + M T t Σ −1 t|t−1 M t ) −1 μ t→t−1 = Σ t→t−1 (x T t Σ −1 t|t−1 M t + μ T t−1 Σ −1 t−1 ) T = Σ t→t−1 (M T t Σ −1 t|t−1 x t + Σ −1 t−1 μ t−1 ). The last step comes from the fact that for a symmetric matrixA,(A −1 ) T = A −1 , and covariance matrices are symmetric. 9.3. Posterior Under Isotropic Marginals Theorem 3 (Closed-Form Posterior Under Isotropic Marginals). Assume isotropic marginals Σ t = σ 2 t I,Σ t−1 = σ 2 t−1 I.(21) Then the posterior covariance simplifies to Σ t→t−1 = σ 2 t−1 I− σ 4 t−1 σ 2 t M T t M t ,(22) and the posterior mean simplifies to μ t→t−1 = μ t−1 + σ 2 t−1 σ 2 t M T t (x t − M t μ t−1 ).(23) 21 Proof. Σ t→t−1 = (Σ −1 t−1 + M T t Σ −1 t|t−1 M t ) −1 = Σ t−1 − Σ t−1 M T t (Σ t|t−1 + M t Σ t−1 M T t ) −1 M t Σ t−1 (c) = Σ t−1 − Σ t−1 M T t Σ −1 t M t Σ t−1 (d) = σ 2 t−1 I− σ 4 t−1 σ 2 t M T t M t (e) We start from Eq. 5 derived in the previous section. In Eq. c, we used the Woodbury matrix identity(A + UCV ) −1 = A −1 − A −1 U (C −1 + V A −1 U ) −1 V A −1 withA = Σ −1 t−1 ,U = M T t ,C = Σ −1 t|t−1 ,V = M t . In Eq. d, we substitute the value of Σ t|t−1 from Eq. 4. Finally, in Eq. e we substitute the isotropic values for Σ t and Σ t−1 . μ t→t−1 = Σ t→t−1 (Σ −1 t−1 μ t−1 + M T t Σ −1 t|t−1 x t ) = Σ t→t−1 Σ −1 t−1 μ t−1 + Σ t→t−1 M T t Σ −1 t|t−1 x t = (Σ t−1 − Σ t−1 M T t Σ −1 t M t Σ t−1 )Σ −1 t−1 μ t−1 + Σ t→t−1 M T t Σ −1 t|t−1 x t (f) = (I− Σ t−1 M T t Σ −1 t M t )μ t−1 + Σ t→t−1 M T t Σ −1 t|t−1 x t = (I− Σ t−1 M T t Σ −1 t M t )μ t−1 + (Σ −1 t−1 + M T t Σ t|t−1 M t ) −1 M T t Σ −1 t|t−1 x t (g) = (I− Σ t−1 M T t Σ −1 t M t )μ t−1 + Σ −1 t−1 M T t (Σ t|t−1 + M t Σ t−1 M T t ) −1 x t (h) = (I− Σ t−1 M T t Σ −1 t M t )μ t−1 + Σ −1 t−1 M T t Σ −1 t x t (i) = μ t−1 + Σ t−1 M T t Σ −1 t (x t − M t μ t−1 ) = μ t−1 + σ 2 t−1 σ 2 t M T t (x t − M t μ t−1 )(j) Starting fromμ t→t−1 in Eq. 5, in Eq. f we substituteΣ t→t−1 from Eq. d, and then in Eq g we substituteΣ t→t−1 from Eq. 5. In Eq. h, we use a corollary of Woodbury identity(A + UCV ) −1 UC = A −1 U (C −1 + V A −1 U ) −1 , with the same substitution as described above. In Eq. i, we substitute the value ofΣ t|t−1 from Eq. 4, and finally, we substitute the isotropic values for Σ t and Σ t−1 . 22 References [1]Shady Abu-Hussein and Raja Giryes. Udpm: Upsampling dif- fusion probabilistic models. arXiv preprint arXiv:2305.16269, 2023. 1, 3, 2, 5 [2]Yuval Atzmon, Maciej Bala, Yogesh Balaji, Tiffany Cai, Yin Cui, Jiaojiao Fan, Yunhao Ge, Siddharth Gururani, Jacob Huffman, Ronald Isaac, et al. Edify image: High-quality image generation with pixel space laplacian diffusion models. arXiv preprint arXiv:2411.07126, 2024. 1, 2 [3]Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie S Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. NeurIPS, 2023. 1 [4]Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 22669–22679, 2023. 1 [5] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelli- gence, (6):679–698, 2009. 2 [6] Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo. Pixelflow: Pixel-space generative models with flow. arXiv preprint arXiv:2504.07963, 2025. 1, 2, 4, 8, 3, 5 [7] Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Ship- pole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Forty-first Interna- tional Conference on Machine Learning, 2024. 1 [8]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 7 [9]Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adver- sarial networks. NeurIPS, 2015. 2 [10]Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021. 6, 7, 1 [11]Gene H Golub and G ́ erard Meurant. Matrices, moments and quadrature with applications. Princeton University Press, 2009. 5 [12] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In ICLR, 2024. 2, 1 [13] Moayed Haji-Ali, Willi Menapace, Ivan Skorokhodov, Arpit Sahni, Sergey Tulyakov, Vicente Ordonez, and Aliaksandr Siarohin. Improving progressive generation with decompos- able flow matching. arXiv preprint arXiv:2506.19839, 2025. 3, 5 [14]Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffu- sion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7441–7451, 2023. 5 [15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 8 [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020. 1, 2, 3, 5, 7 [17]Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 2022. 2 [18]Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. ICLR, 2023. 1, 4, 5, 7, 8 [19]Wongi Jeong, Kyungryeol Lee, Hoigi Seo, and Se Young Chun. Upsample what matters: Region-adaptive latent sam- pling for accelerated diffusion transformers. arXiv preprint arXiv:2507.08422, 2025. 1, 2, 4, 8, 3 [20]Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.05954, 2024. 1, 2, 4, 8, 3 [21]Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018. 2, 1 [22]Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 7 [23]Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of research of the National Bureau of Standards, 45(4):255–282, 1950. 5 [24]Tony Lindeberg. Scale-space theory: A basic tool for analyz- ing structures at different scales. Journal of applied statistics, 21(1-2):225–270, 1994. 1, 2 [25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. 2, 7 [26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 7 [27]David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, pages 1150–1157. Ieee, 1999. 2 [28] Soumik Mukhopadhyay, Matthew Gwilliam, Yosuke Yam- aguchi, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Tianyi Zhou, Jun Ohya, and Abhinav Shri- vastava. Do text-free diffusion models learn discriminative visual representations? In European Conference on Computer Vision, pages 253–272. Springer, 2024. 1 [29]Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021. 6 [30]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Ad- vances in neural information processing systems, 32, 2019. 4 23 [31]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205, 2023. 2, 1 [32]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ̈ orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 1, 5 [33]Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmenta- tion. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 234–241. Springer, 2015. 2, 1 [34]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.arXiv preprint arXiv:2202.00512, 2022. 5 [35]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2 [36]Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021. 1 [37]Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unify- ing diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.03350, 2023. 2 [38] Ye Tian, Xin Xia, Yuxi Ren, Shanchuan Lin, Xing Wang, Xue- feng Xiao, Yunhai Tong, Ling Yang, and Bin Cui. Training- free diffusion acceleration with bottleneck sampling. arXiv preprint arXiv:2503.18940, 2025. 2 [39]XIANG Weilai. FutureXiang/Diffusion, 2025. original-date: 2022-10-18T11:42:46Z. 2 24