Paper deep dive
Latent Generative Models with Tunable Complexity for Compressed Sensing and other Inverse Problems
Sean Gunn, Jorio Cocola, Oliver De Candido, Vaggos Chatziafratis, Paul Hand
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:32:48 AM
Summary
The paper introduces tunable-complexity generative priors for inverse problems, allowing users to adjust latent dimensionality at inference time. By leveraging nested dropout in VAEs, normalizing flows, and latent diffusion models, the authors demonstrate that intermediate latent complexities often yield superior reconstruction quality compared to fixed-complexity baselines across tasks like compressed sensing, inpainting, and denoising.
Entities (5)
Relation Signals (3)
Tunable Generative Priors → solves → Inverse Problems
confidence 98% · The results in this paper demonstrate the benefits of using generative priors with a tunable complexity for inverse problems both empirically and theoretically.
Nested Dropout → enablestunabilityin → Latent Diffusion Models
confidence 95% · We propose a new training algorithm for latent diffusion that utilizes nested dropout... enabling a single model to be tuned as a prior for downstream inverse problems.
Variational Autoencoders → usedasbackbonefor → Latent Diffusion Models
confidence 95% · The VAE consists of an encoder... and a decoder... Once the autoencoder is trained, a denoising network is trained in latent space.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative models have emerged as powerful priors for solving inverse problems. These models typically represent a class of natural signals using a single fixed complexity or dimensionality. This can be limiting: depending on the problem, a fixed complexity may result in high representation error if too small, or overfitting to noise if too large. We develop tunable-complexity priors for diffusion models, normalizing flows, and variational autoencoders, leveraging nested dropout. Across tasks including compressed sensing, inpainting, denoising, and phase retrieval, we show empirically that tunable priors consistently achieve lower reconstruction errors than fixed-complexity baselines. In the linear denoising setting, we provide a theoretical analysis that explicitly characterizes how the optimal tuning parameter depends on noise and model structure. This work demonstrates the potential of tunable-complexity generative priors and motivates both the development of supporting theory and their application across a wide range of inverse problems.
Tags
Links
- Source: https://arxiv.org/abs/2603.07357v2
- Canonical: https://arxiv.org/abs/2603.07357v2
Trouble viewing inline? Open PDF directly →
Full Text
77,386 characters extracted from source content.
Expand or collapse full text
Latent Generative Models with Tunable Complexity for Compressed Sensing and other Inverse Problems Sean Gunn Northeastern University gunn.s@northeastern.edu Jorio Cocola Harvard University jcocola@seas.harvard.edu Oliver De Candido Technical University of Munich oliver.de-candido@tum.de Vaggos Chatziafratis UC Santa Cruz vaggos@ucsc.edu Paul Hand Northeastern University p.hand@northeastern.edu Abstract Generative models have emerged as powerful priors for solving inverse problems. These models typically represent a class of natural signals using a single fixed complexity or dimensionality. This can be limiting: depending on the problem, a fixed complexity may result in high representation error if too small, or overfitting to noise if too large. We develop tunable-complexity priors for diffusion models, normalizing flows, and variational autoencoders, leveraging nested dropout. Across tasks including compressed sensing, inpainting, denoising, and phase retrieval, we show empirically that tunable priors consistently achieve lower reconstruction errors than fixed-complexity baselines. In the linear denoising setting, we provide a theoretical analysis that explicitly characterizes how the optimal tuning parameter depends on noise and model structure. This work demonstrates the potential of tunable-complexity generative priors and motivates both the development of supporting theory and their application across a wide range of inverse problems. 1 Introduction Inverse problems aim to reconstruct an unknown signal, potentially corrupted by noise, from a set of measurements given by a forward model, which may or may not be known in advance. Such a formulation applies to various image-processing applications, including compressive sensing, denoising, and super-resolution. In practice, inverse problems are ill-posed and thus they require prior information about the signal to yield a successful recovery [40]. Deep generative models have been demonstrated to be powerful signal priors when used to solve inverse problems [4, 15, 2, 8, 7, 43]. The process of using a deep generative prior typically has two phases: training and inversion. In the training phase, a generative neural network is trained on a dataset representative of the natural signal class intended for inversion. In the second phase, the model parameters are fixed from training, and an algorithm is deployed to estimate the signal of interest for a given forward operator. This approach has the benefit that the prior can be learned in isolation from the inverse problems being solved. Thus, the approach can apply to a variety of inverse problems, which is in contrast to other neural network-based approaches such as end-to-end training. In recent years, there has been a dominant framework for using generative models as priors for inverse problems. Within this framework, generative priors have a fixed complexity that is set during training. For example, existing priors include Generative Adversarial Networks (GANs), which typically have a latent space of fixed low dimensionality [4, 15, 32, 6, 14]; Normalizing Flow models, which have a latent space of fixed high-dimensionality equal to that of the images [2, 48, 1, 27]; and Score-Based Models, which maintain information about the probability density over the fixed high-dimensional space of all images [43, 31, 17, 10, 5]. Observation Low Complexity Prior GL(z)G_L(z) Medium Complexity Prior GM(z)G_M(z) High Complexity Prior GH(z)G_H(z) z z z PSNR 11.43 dB PSNR 25.85 dB PSNR 34.59 dB PSNR 28.48 dB Figure 1: Medium-complexity priors can outperform both low- and high-complexity alternatives for image reconstruction. We trained three separate generative models with low, medium, and high latent dimensionality. The size of the boxes representing z depicts the latent dimensionality of each model. We test the models on a random pixel inpainting problem. The medium-complexity prior yields the reconstruction with the highest Peak Signal-to-Noise Ratio (PSNR). We step outside the dominant framework and demonstrate the benefits of solving inverse problems using generative priors whose complexity can be selected by the user at inference time, after training. Such generative priors simultaneously maintain representations of varying complexities of the natural signal class, and we refer to them as generative priors with a tunable complexity. In this paper, we demonstrate that tunable generative priors can lead to significantly improved reconstruction errors when model complexity is appropriately tuned for a specific inverse problem. We focus on latent generative models whose complexity is governed by the latent dimension k. By training the model to preserve meaningful representations across different values of k, we enable practitioners to tune its complexity to the inverse problem at hand. We illustrate this point in Figure˜1 for a random pixel inpainting problem. The medium-complexity prior yields a higher quality image reconstruction than the low- and high-complexity prior, both qualitatively and quantitatively. To motivate our approach, we first consider injective flows [37]. We train a family of models, each with a distinct latent dimensionality k and no parameter sharing, and apply them to compressed sensing with random measurements. As shown in Figure˜2, reconstruction error follows an upside-down U-shaped curve: with fewer measurements, models of intermediate complexity outperform both higher- and lower-complexity ones. Moreover, the optimal complexity depends on the number of available measurements. While this experiment requires training separate models for each k, a naïve and computationally expensive strategy, it illustrates the importance of tunability. This motivates the more efficient algorithms we develop next, which scale to practical settings such as production-size images. Building on this motivation, we will show how tunability can be achieved efficiently in three major classes of generative models: variational autoencoders (VAEs) [23], normalizing flows (NFs) [11], and latent diffusion models (LDMs) [36]. In each case, we design a single family of models with parameter sharing and provide an efficient training procedure. For LDMs, we introduce a new algorithm based on nested dropout [35], described in Section˜3.1. For NFs, we adopt an existing ordering method [3] Appendix˜B. For VAEs, we extend the adversarial objective of [13] with nested dropout regularization term. Across all three settings, we empirically demonstrate that tunable generative priors consistently achieve lower reconstruction errors than fixed-complexity baselines over a range of inverse problems, undersampling ratios, and noise levels (Figure˜4). As an initial theoretical exploration into the effect of model tunability, we study denoising in the case of an invertible linear generative model and the best lower-dimensional linear models that approximate it. We rigorously establish a theory for how to select the tuning parameter in the case of a linear generative model. In this setting, we provide explicit expressions for the reconstruction error as a function of the modeled dimensionality in the cases of maximum likelihood estimation and maximum a posteriori estimation. This analytical expression permits a direct expression for optimal signal complexity in this setting, revealing the theoretical benefit for tunability. The results in this paper demonstrate the benefits of using generative priors with a tunable complexity for inverse problems both empirically and theoretically. Empirically, we show that tunable complexity consistently improves reconstruction quality across multiple generative model architectures, inverse problem types, and inversion algorithms. Additionally, for linear models, we provide a theoretical analysis of denoising. This work motivates research on how to bring tunability into generative models. Figure 2: Intermediate latent dimensionalities yield the best reconstruction at low measurement ratios. We train separate injective flow models for each latent dimensionality k, ranging from 1616 to 456456, on MNIST images of size n=32×32=1024n=32× 32=1024 pixels. No parameter sharing is used across models. Each panel shows reconstruction performance for a different number of measurements m: for m<n=1024m<n=1024, the forward operator is an m×nm× n random Gaussian matrix (compressed sensing), while m=1024m=1024 corresponds to the identity operator (no compression). At small measurement ratios m/nm/n, intermediate latent dimensions (150≤k≤300150≤ k≤ 300) yield the lowest reconstruction error, while the optimal k shifts as the number of measurements increases (error bars indicate ±1± 1 standard deviation). Our main contributions are as follows: • We observe a potentially surprising phenomenon in the use of latent generative priors for solving inverse problems. We train a single generative model to simultaneously represent the natural signal class across multiple latent dimensionalities k. Across a variety of architectures, inverse problems, and inversion algorithms, we empirically find that some intermediate latent dimensionality yields an improved reconstruction error. • We propose a new training algorithm for latent diffusion that utilizes nested dropout and a convex combination of the original and truncated latent objective. This yields a tunable latent diffusion model that learns hierarchical representations across latent dimensionalities, enabling a single model to be tuned as a prior for downstream inverse problems. • We provide a theoretical analysis of tunability in the context of denoising with linear invertible generative models. Under this model, we derive an explicit expression for the reconstruction error of MLE and MAP estimators as a function of the model complexity. 2 Background and Related Work The background will focus on diffusion models due to their relevance in recent research trends. For additional details on normalizing flows and variational autoencoders, we refer the reader to the appropriate references. (a) CelebA (b) MS COCO Figure 3: Nested dropout training produces tunable latent diffusion models that maintain generation quality across latent dimensionalities. FID score is plotted as a function of latent dimensionality k for models trained with different values of the dropout distribution parameter pkp_k. The vanilla LDM baseline (dotted line) operates only at full dimensionality. As k increases, the tunable models approach baseline performance while retaining the ability to generate from lower-dimensional representations. Results are evaluated on 50k training images from each dataset. FID scores computed with [34]. 2.1 Diffusion Models The standard diffusion framework gradually corrupts data 0∼pdata∈ℝn x_0\! \!p_ data\ x ^n into noise T∼(0,n) x_T\! \!N(0, I_n) via a forward noising process, and learns a parameterization of the reverse process to recover the data distribution [16, 44, 9]. We adopt latent diffusion models (LDMs) [36], which improve computational efficiency by performing diffusion in a learned latent space: a variational autoencoder (VAE) [23] with encoder ℰ:ℝn→ℝkE:R^n\!→\!R^k and decoder :ℝk→ℝnD:R^k\!→\!R^n (typically k≪nk\! \!n) yields 0=ℰ(0) z_0=E( x_0) and 0≈(0) x_0\!≈\!D( z_0) for all 0∼pdata x_0\! \!p_ data. The encoder maps pdatap_ data to to the laten distribution pZ:=ℰ#pdatap_Z:=E_\#p_ data, and the diffusion process then operates on the latent variable t z_t. With schedule βtt=1T\ _t\_t=1^T, αt:=1−βt _t:=1- _t, and α¯t:=∏j=1tαj α_t:= _j=1^t _j, the forward marginal in latent space is t=α¯t0+1−α¯tϵ,ϵ∼(0,k), z_t= α_t\, z_0+ 1- α_t\, ε, ε N(0, I_k), (forward marginal) which allows sampling t z_t at any timestep t directly from 0 z_0. The reverse process is parameterized by a neural network: t−1=1αt(t−1−αt1−α¯tϵθ(t,t))+σtϵt′,ϵt′∼(0,k), z_t-1= 1 _t\! ( z_t- 1- _t 1- α_t\, _θ( z_t,t) )+ _t\, ε _t, ε _t N(0, I_k), (reverse) where σt∈[0,σtDDPM] _t∈[0,\ σ^DDPM_t] controls sampling stochasticity and σtDDPM=1−α¯t−11−α¯t(1−αt) σ^DDPM_t= 1- α_t-11- α_t\,(1- _t). Setting σt=0 _t=0 yields the deterministic denoising diffusion implicit model (DDIM) sampler [42]. A denoising network ϵθ _θ (typically a U-Net) is trained to predict the noise in the latent space: ℒLDM=t∼(1,…,T),0,ϵ∼(0,)‖ϵ−ϵθ(α¯t0+1−α¯tϵ,t)‖22, _LDM=E_\,t (\1,…,T\),\ z_0,\ ε N(0, I) \|\, ε- _θ\! ( α_t\, z_0+ 1- α_t\, ε,\,t ) \|_2^2, (1) where t∼(1,…,T)t (\1,…,T\) denotes that the timestep is sampled uniformly from all diffusion steps. To enable latent diffusion, the autoencoder must be pretrained; its objective combines reconstruction, regularization, and adversarial terms. The primary term is the reconstruction loss, ℒreconL_recon, typically a combination of ℓ1 _1 and perceptual losses, which measures the discrepancy between the original high-dimensional signal x and its reconstruction (ℰ())D(E( x)). A regularization penalty, ℒregL_reg, is enforced via Kullback–Leibler (KL) divergence with respect to a reference distribution, typically the standard Gaussian. Finally, an adversarial loss, ℒadvL_adv, is introduced by training a discriminator C to distinguish between real images x and reconstructed samples (ℰ())D(E( x)) ℒAutoencoder=ℒrec(,(ℰ()))+λregℒreg(ℰ())+λadvℒadv(,,ℰ()). L_Autoencoder=L_rec ( x,\,D(E( x)) )+ _reg\,L_reg (E( x) )+ _adv\,L_adv\! (C,\, x,\,E( x) ). (2) After this stage, diffusion is performed entirely on t z_t. 2.2 Inverse Problems with Generative Priors We consider the general linear inverse problem of recovering an unknown signal ∈ℝn x ^n from noisy measurements ∈ℝm y ^m, =()+,∼(0,σ2m), y=A( x)+ η, η N(0,σ^2 I_m), (3) where :ℝn→ℝmA:R^n ^m [40] is the forward operator (e.g., a linear projection, convolutional blur, or other transformation), and η is additive noise drawn from a Gaussian distribution. Generative models can be employed as priors for inverse problems in two main ways: supervised, where the forward operator is known during training and models are trained on paired data (xi,yi)\(x_i,y_i)\, or unsupervised, where the forward operator is unknown [40, 25]. For inverse problems, we modify the reverse dynamics in Equation˜reverse to account for measurements =()+ y=A( x)+ η. Since we operate entirely in latent space (=() x=D( z)), posterior sampling is guided by the conditional score, which decomposes as ∇tlogpt(t∣)=∇tlogpt(t)+∇tlogpt(∣t). _ z_t p_t( z_t y)= _ z_t p_t( z_t)+ _ z_t p_t( y z_t). (4) The first term is provided by the pretrained diffusion prior through the denoising network ϵθ _θ, while the second term enforces data-consistency with the forward operator A under the measurement model. Following the taxonomy of interleaving methods highlighted by Wang et al. [46], these solvers alternate unconditional reverse diffusion steps (e.g., DDIM) with data-consistency corrections that either explicitly approximate the measurement likelihood or enforce feasibility. One line of methods explicitly approximates the measurement-likelihood term via a projection or gradient update [17, 21, 5, 47, 38]. A canonical formulation is latent diffusion posterior sampling [39]: ∇tlogp(∣t)≈∇tlogp(∣0=([z0∣zt]))=1σ2∇t‖−(0)‖22, _ z_t p( y z_t)\;≈\; _ z_t p( y x_0=D(E [z_0 z_t ]))= 1σ^2\, _ z_t\,\| y-D( z_0)\|_2^2, (5) where the posterior mean is given by Tweedie’s formula [12]. Another class of methods does not explicitly compute the measurement likelihood, but instead approximates the feasible set of solutions ∣=()\ x y=A( x)\ [21, 20, 41]. In particular Song et al. [41] employs a hard data-consistency term that enforces ()∈∣=()D( z)∈\\, x y=A( x)\,\, which is approximated in practice by gradient-descent. 3 Methods This section describes the training of a tunable latent diffusion model (Section˜3.1) and its application as a prior for inverse problems (Section˜3.2). We begin by training a variational autoencoder (VAE) as the backbone of the latent diffusion model. The VAE consists of an encoder ℰ:ℝn→ℝkE:R^n ^k and a decoder :ℝk→ℝnD:R^k ^n, such that for samples 0∼pdata(0) x_0 p_ data( x_0) we have 0≈(ℰ(0)) x_0≈D(E( x_0)). Once the autoencoder is trained, a denoising network ϵθ(t,t) ε_θ( z_t,t) is trained in latent space to predict the Gaussian noise at each diffusion step t. Finally, diffusion is performed in latent space using the denoising network ϵθ ε_θ. 3.1 Training a Tunable Latent Diffusion Model We aim to train a latent generative model that can represent the natural signal class across multiple latent dimensionalities. To achieve this, we leverage nested dropout [35], which imposes an ordered structure on the latent variables by always preserving a prefix of coordinates. Formally, let k∼pkk p_k be drawn from 1,…,d\1,…,d\. In our experiments, we use a truncated geometric distribution with success parameter p, but other distribution over 1,…,d\1,…,d\ can be adopted depending on the application. The truncation operator is defined as ↓k=[1,2,…,k,0,…,0] z_ k=[ z_1, z_2,…, z_k,0,…,0] with k≤dk≤ d. Given =ℰ() z=E( x), reconstruction is performed from ↓k z_ k, which encourages earlier coordinates to carry more information about the signal class. Latent diffusion is trained in two stages. First, the autoencoder is trained to reconstruct ∈ℝn x ^n robustly across a range of latent dimensions k. Building on Section˜2.1, the VAE backbone is trained with reconstruction, regularization, and adversarial terms, which we extend here with a nested dropout objective. This follows the standard VAE training objective with an added nested dropout term: ℒVAE=minE,Dmax[ _VAE= _E,\,D\ _C\ [ ℒrec(,(ℰ()))+λregℒreg(ℰ()) \ L_rec ( x,\,D(E( x)) )+ _reg\,L_reg (E( x) ) +λadvℒadv(,,ℰ())+λdropℒdrop(,(ℰ()↓k))], \ + _adv\,L_adv\! (C,\, x,\,E( x) )+ _drop\,L_drop ( x,\,D(E( x)_ k) ) ], (6) where :ℝn→(0,1)C:R^n→(0,1) is a discriminator. We adopt a perceptual loss [49] for ℒdropL_drop. In the second stage, the diffusion model is trained in latent space with a loss that interpolates between the standard diffusion objective and its truncated-latent variant. For λ∈[0,1]λ∈[0,1], we define ℒLDM L_LDM =ℰ(),ϵ∼(0,1),t∼(1,…,T)[(1−λ)‖ϵ−ϵθ(t,t)‖22+λ‖ϵ−ϵθ((t)↓k,t)‖22]. =E_E( x), ε (0,1),t (\1,…,T\) [(1-λ) \| ε- ε_θ( z_t,t) \|_2^2+λ \| ε- ε_θ(( z_t)_ k,t) \|_2^2 ]. (7) The first term is the standard latent diffusion objective, while the second applies it to the truncated latent (t)↓k( z_t)_ k, encouraging effective denoising even from a reduced representation. Here, t z_t is sampled from the forward DDPM process (Equation˜forward marginal), and k is drawn from the same distribution pkp_k used in the VAE objective, ensuring consistency. The result is a hierarchically organized latent space: the first few coordinates capture the most essential signal structure, with each additional coordinate contributing finer detail as the model’s representational capacity grows with k. This effect is illustrated in Figures˜3(a) and 3(b), which show how FID varies with latent dimensionality under different hyperparameters. Lastly, for details about training please go to Appendix A.2. 3.2 Inverse Problems with Tunable Priors A broad family of methods applies a data-consistency step via projection, gradient, or small optimization after each reverse update to move the prior iterate toward the feasible set ∣()=\ x A( x)= y\. In latent space, representative examples include Latent Posterior Diffusion (LDPS) [39], Posterior Sampling with Latent Diffusion (PSLD) [39], ReSample [41], and our formulation in Algorithm˜2. The goal of Algorithm 2 is not to establish state-of-the-art performance, but rather to provide a simple and broadly effective algorithm that applies naturally across a variety of inverse problems, including nonlinear forward operators. This design choice is motivated by the fact that incorporating tunability into more specialized methods, such as PSLD or ReSample, requires additional care due to their respective architectural constraints. The experiments in Figure˜4 employ Algorithm˜2, whereas tunable versions of LDPS and PSLD would fall under the general template of Algorithm˜1. Algorithm 1 General Template for Tunable Diffusion Priors 1:Input: y, A, ℰE, D, ϵθ ε_θ, steps T, tunable parameter k 2:Output: (^0)D( z_0) 3:T∼(0,I) z_T (0,I) 4:for t=T−1t=T-1 to 0 do 5: ^←ϵθ(t,t) s← ε_θ( z_t,t) 6: ^0←1α¯t(t+1−α¯t^) z_0← 1 α_t( z_t+ 1- α_t s) 7: t−1′← z _t-1← DDIM/DDPM reverse with 8: ^0, z_0, s 9: t−1← z_t-1← project/gradient update with 10: ^0 z_0 and t−1′ z _t-1 to get closer to 11: ∣(())=\ z A(D( z))= y\ 12: t−1←(t−1)↓k z_t-1←( z_t-1)_ k 13:end for 14:return (^0)D( z_0) Algorithm 2 Tunable Posterior Sampling (Concrete Instantiation) 1:Input: y, A, ℰE, D, ϵθ ε_θ, steps T, variances σt\ _t\, tunable parameter k 2:Output: (^0)D( z_0) 3:T∼(0,I) z_T (0,I) 4:for t=T−1t=T-1 to 0 do 5: ^←ϵθ(t,t) s← ε_θ( z_t,t) 6: ^0←1α¯t(t+1−α¯t^) z_0← 1 α_t( z_t+ 1- α_t s) 7: ϵ∼(0,I) ε (0,I) 8: 9: t−1′←αt(1−α¯t−1)1−α¯tt+α¯t−1βt1−α¯t^0 z _t-1← _t(1- α_t-1)1- α_t z_t+ α_t-1 _t1- α_t z_0 10: +σtϵ+\; _t ε 11: Initialize z at ^0 z_0 12: t−1←argmin‖−((z))‖22 z_t-1← _ z\|\, y-A(D(z))\,\|_2^2 13: + 12σt2‖−t−1′‖22 12 _t^2\|\, z- z _t-1\,\|_2^2 14: t−1←(t−1)↓k z_t-1←( z_t-1)_ k 15:end for 16:return (^0)D( z_0) Our approach follows a generic template for tunable diffusion priors. Starting from Gaussian noise in the latent space, the algorithm iteratively denoises the latent variable using the learned noise prediction network. Each reverse step is then corrected by a data-consistency operation that is consistent with measurements =(()) y=A(D( z)). At each reverse iteration, a truncation operator is applied to the latent variable to control the representation capacity at a chosen dimensionality k. This template unifies a broad family of inversion methods: the prior supplies a generative update, the data-consistency step enforces measurements, and the tunable operator governs the complexity of the representation. Algorithm 2 provides a concrete instantiation of this template, combining a DDPM/DDIM reverse update with a quadratic data-consistency optimization. 4 Experiments Figure 4: Tunable priors outperform fixed-complexity baselines across multiple inverse problems. Reconstruction performance (LPIPS, lower is better) is shown as a function of latent dimensionality k for compressed sensing, denoising, phase retrieval, and inpainting on the CelebA dataset. The tunable LDM prior (blue) is compared against a fixed-complexity baseline operating at full dimensionality (orange). For all four tasks, intermediate values of k yield lower reconstruction error than both the low-complexity and high-complexity extremes, demonstrating the benefit of tuning model complexity to the inverse problem at hand. CS PHASE Inpaint (random) Deblur (Gauss) Method PSNR↑ LPIPS↓ PSNR↑ LPIPS↓ PSNR↑ LPIPS↓ PSNR↑ LPIPS↓ DPS [5] 25.65 ± 2.67 0.159 ± 0.067 22.15 ± 3.84 0.267 ± 0.061 23.18 ± 2.11 0.146 ± 0.042 27.90 ± 1.53 0.093 ± 0.019 Tunable LDM Prior (Ours) 25.49 ± 1.70 0.179 ± 0.045 25.24 ± 1.71 0.168 ± 0.046 25.30 ± 1.76 0.122 ± 0.028 29.52 ± 1.11 0.085 ± 0.024 LDPS / PSLD [39] 23.30 ± 1.24 0.248 ± 0.063 24.48 ± 1.35 0.169 ± 0.038 25.85 ± 2.18 0.136 ± 0.036 27.33 ± 1.23 0.165 ± 0.037 Tunable LDPS / PSLD (Ours) 24.69 ± 1.37 0.164 ± 0.036 21.95 ± 1.31 0.256 ± 0.065 26.75 ± 1.71 0.095 ± 0.028 28.50 ± 1.05 0.115 ± 0.031 NF [2] 21.31 ± 1.94 0.508 ± 0.079 20.67 ± 2.32 0.554 ± 0.073 20.07 ± 2.42 0.376 ± 0.091 21.43 ± 2.11 0.438 ± 0.034 Tunable NF (Ours) 27.16 ± 1.56 0.238 ± 0.058 26.11 ± 2.07 0.246 ± 0.076 23.24 ± 2.43 0.167 ± 0.057 24.85 ± 2.21 0.215 ± 0.043 Table 1: Quantitative results on CelebA-HQ across inverse problems. Values reported as mean ± std. Our experiments underscore several key findings. First, we demonstrate that a single generative model can be trained to operate across a wide range of latent dimensionalities, enabling the same model to be meaningfully tuned at inference time rather than retrained for each complexity level (Figures˜3(a) and 3(b)). Second, in the denoising setting, we empirically validate the theoretical findings of Section 5, which identify an intermediate optimal latent complexity in the presence of additive noise (Figure˜18). Third, we show that the observed non-monotonic dependence on latent dimensionality generalizes across multiple inverse problems, datasets, and generative architectures, including latent diffusion models, variational autoencoders, and normalizing flows (Figures˜4, 18 and 7). Finally, we demonstrate that incorporating tunability complements recent advances in inversion algorithms, improving or matching strong contemporary baselines [5, 39] (Table˜1). The code is available here Tunable Complexity LGM for Inverse Problems. Datasets and Metrics. We evaluate on four datasets spanning different resolutions: CelebA [28], CelebA-HQ [18], and MS COCO [26] at 64×64×364× 64× 3, and FFHQ [19] at 256×256×3256× 256× 3. For generative quality, we report the Fréchet Inception Distance (FID ↓ ) computed on 50k samples. For inverse problems, we measure reconstruction fidelity using Peak Signal-to-Noise Ratio (PSNR ↑ ) and perceptual quality using LPIPS (↓ ). Unless otherwise specified, reconstruction results are reported on a test set of 100 images. For the experiments reported in Tables˜1 and 2, the latent dimensionality k was selected using a held-out validation set of 12 images (see A.2 for details). Baselines. We compare against three classes of generative priors. DPS [5] is a pixel-space diffusion method that applies data-consistency guidance directly in image space; we use a diffusion model trained on CelebA that achieves an FID of 1.27 [33]. LDPS/PSLD [39] performs posterior sampling in the latent space of a pretrained autoencoder. NF [2] uses normalizing flows with MAP estimation for inversion. For each latent-space baseline (LDPS/PSLD and NF), we also evaluate a tunable variant using our approach (Algorithm˜2). Figure 5: Qualitative results across four inverse problems on FFHQ. Columns: ground truth, measurement (⊤A y for CS/phase retrieval; degraded input for SR/deblurring), baseline, and tunable prior. CS and phase retrieval (10% measurements) use Algorithm˜2; 4×4× SR and Gaussian deblurring use PSLD [39]. Insets show an enlarged view of the highlighted yellow boxes Implementation Details. We build upon the Diffusers package [45] for both the denoising U-Net and VAE architectures. All VAE models employ continuous latent spaces without quantized codebooks. For each dataset, the VAE was trained on the same data used for subsequent diffusion model training, with a held-out validation set used to evaluate LPIPS for model selection. For the LDM, we employ a U-Net in half-precision, following the architectural specifications of [36], and optimize using AdamW [29]. During training, we maintained a held-out set to evaluate FID, selecting the model with the lowest FID score for inverse problem applications. For additional implementation details, please refer to Appendix Section˜A.2. Task Method PSNR↑ LPIPS↓ Gaussian Blur PSLD 23.78 ± 1.49 0.385 ± 0.066 + Tunable 24.07 ± 1.56 0.357 ± 0.061 Super Resolution PSLD 25.02 ± 2.54 0.356 ± 0.068 + Tunable 25.43 ± 1.71 0.323 ± 0.061 Phase Retrieval LDM Prior 24.58 ± 1.31 0.363 ± 0.032 + Tunable 28.21 ± 1.11 0.261 ± 0.044 Table 2: Quantitative results on FFHQ across inverse problems. Values reported as mean ± std.. Inverse Problem Details For Figure˜4, we evaluated four forward operators. All experiments were done on 64×64×364× 64× 3. Compressed sensing used a random Gaussian operator ∈ℝm×n A ^m× n with i.i.d. (0,1/m)N(0,1/m) entries, where m=1228m=1228 (approximately 10%10\% of n=12288n=12288). Denoising was modeled as additive Gaussian noise ∼(0,σ2n) η (0,σ^2 I_n) with σ=0.25σ=0.25. Inpainting employed a binary mask M with approximately 80%80\% missing pixels. Phase retrieval used phaseless Gaussian measurements || A x , sharing the same A as compressed sensing, with a measurement ratio m/n=0.15m/n=0.15 (15%15\%). All inverse problems were solved using Algorithm 2. In Table˜1, we compare our approach to a contemporary state-of-the-art (SOTA) pixel-based diffusion prior [5] and show that our framework can also enhance other SOTA priors, such as PSLD [39], by leveraging tunability to improve reconstruction quality. In addition, we introduce a tunable generative prior based on normalizing flows; details are provided in Appendix˜B. This model is trained following the objective of [3] and adopts the MAP estimator of [2]. Methods designated as tunable fall under our general formulation (Algorithm˜1), and our goal is to enable a broad class of latent diffusion–based inversion algorithms to achieve improved results. For the measurement operators, we consider: compressed sensing with a measurement ratio m/n=7.5%m/n=7.5\%; phase retrieval with 10%10\% measurements; inpainting with 80%80\% of pixels missing at random; and Gaussian deblurring with kernel size 55 and standard deviation 33. Lastly, for DPS we use a SOTA diffusion model trained on CelebA that achieves an FID of 1.27 on the training set [33]. Furthermore, as reported in Table˜2, Gaussian blur is implemented using a kernel of size 11 with a standard deviation of 3, while super-resolution is performed via bicubic downsampling. To better reflect realistic measurement conditions, additive Gaussian noise with a standard deviation of σ=0.05σ=0.05 is applied. For the phase retrieval setting, the forward operator consists of a Rademacher sign flip followed by circular convolution with a randomly generated Gaussian filter, after which the signal is subsampled to obtain m measurements. For these particular experiments we used a measurement ratio m/n=10%m/n=10\% 5 Theory for Denoising with Linear Generative Model The goal of this section is to provide preliminary theoretical conclusions that show tunability can lead to improved reconstruction errors relative to corresponding nontunable models. Further, this section aims to provide justification for how to select the tuning parameter for a generative model with tunable complexity. We present these insights for the problem of denoising under additive Gaussian noise. We show that the effect of tunability can be proven even in the case of linear generative models, and thus we restrict our attention to those. We consider a family of linear generative models given as follows. Fix an invertible ∈ℝn×n G ^n× n, and let =T G= U V^T be a singular value decomposition with =diag(s1,…,sn)∈ℝn×n =diag(s_1,…,s_n) ^n× n. For any k≤nk≤ n, let =T∈ℝn×n G_k= U _k V^T ^n× n, where =diag(s1,…,sk,0,…,0)∈ℝn×n. _k=diag(s_1,…,s_k,0,…,0) ^n× n. Note that =n G= G_n. Each k G_k induces a probability distribution over ℝnR^n by =k, where ∼(0,n). x= G_k z, where z (0, I_n). We will refer to this distribution as pkp_ G_k and observe that pk=(0,kkT)p_ G_k=N(0, G_k G_k^T). Note that k is a parameter that governs the complexity of the modeled signal class. We consider the following denoising problem. Let =0+ y= x_0+ η, where 0∼p x_0 p_ G_n and ∼(0,σ2n) η (0,σ^2 I_n). Our goal is to recover 0 x_0 given y and n G_n. For a given k, we consider a maximum a posteriori (MAP) estimate of 0 x_0 under the signal prior pkp_ G_k. This results in the following optimization problem over the latent space: ^γ(k): z_γ(k): -2.5pt =argmink∈ℝk12‖−(k0)‖2+γ2‖k‖2, = *arg\,min_ z^k ^k 12 \| y- G_k pmatrix z^k\\ 0 pmatrix \|^2+ γ2\| z^k\|^2, (8) where (k0)∈ℝn pmatrix z^k\\ 0 pmatrix ^n is the vector k z^k padded with zeroes, the estimated signal is ^γ(k)=k(^γ(k)0) x_γ(k)= G_k pmatrix z_γ(k)\\ 0 pmatrix, and γ is a parameter governing how strongly the prior is enforced. The case of γ=0γ=0 corresponds to a Maximum Likelihood Estimate (MLE) formulation, and the case of γ=σ2γ=σ^2 corresponds to true MAP. It is common in the literature to consider γ as a hyperparameter, and thus we study the behavior Equation˜8 for all γ≥0γ≥ 0. The following theorem provides an exact expression for the mean square error of the estimate above. Theorem 5.1. Suppose we have a family k=1…n\ G_k\_k=1… n of generative models as given above, and let p=(0,T)p_ G_k=N(0, G_k G_k^T), and let Gn∈ℝn×nG_n ^n× n have singular values s1≥s2≥⋯≥sn>0s_1≥ s_2≥·s≥ s_n>0. Let 0∼pn x_0 p_ G_n and ∼(0,σ2n) η (0,σ^2 I_n). Then the estimator given by equation 8 yields 0,[‖^γ(k)−0‖2]=∑i=1ksi2(si2σ2+γ2)(si2+γ)2+∑j=k+1nsj2. _ x_0,\, η [\,\| x_γ(k)- x_0\|^2\, ]= _i=1^k s_i^2\, (s_i^2σ^2+γ^2 )(s_i^2+γ)^2+ _j=k+1^ns_j^2. The exact expression for reconstruction error in Theorem 5.1 makes it possible to analytically find the optimal value of the tunable complexity parameter k, as established in the following corollary. Corollary 5.2. Under the assumptions of Theorem 5.1, if γ≤σ2/2γ≤σ^2/2, then the parameter k that leads lowest reconstruction error of the target signal is given by: argmink∈[n]0,‖^γ(k)−0‖2=maxk∣sk≥σ2−2γ. *arg\,min_k∈[n]E_ x_0, η\| x_γ(k)- x_0\|^2= \k s_k≥ σ^2-2γ \. This corollary shows that for small enough hyperparameters γ, and in the presence of sufficient noise, then the optimal complexity parameter for minimizing reconstruction error can be less than the full signal dimensionality. In the case of MLE, an intermediate signal complexity is optimal if the variance of the measurement noise is larger than the smallest singular value of the linear generator G. Additionally, we observe theoretically that if the noise level increases, then the optimal value of the tunable complexity decreases. The proofs to Theorem 5.1 and Corollary 5.2 are provided in Appendix. 6 Conclusion Prior work on solving inverse problems with generative priors has primarily focused on fixed models of fixed complexity, with most efforts directed at improving inversion algorithms (for example, alternative posterior sampling methods such as DPS, PSLD, or ReSample). In this paper, we introduce tunability of complexity as a complementary and orthogonal axis of improvement. Rather than replacing algorithmic developments, tunability adds a new degree of freedom that practitioners can incorporate into their preferred inversion methods with minimal additional training overhead. We demonstrate consistent improvements across multiple dimensions: major generative model classes (VAEs, injective and normalizing flows, and latent diffusion models), inverse problem types (compressed sensing, inpainting, denoising, deblurring, and phase retrieval), and across various inversion algorithms and measurement conditions. Across all settings, intermediate complexity consistently outperforms both lower and higher complexity extremes. These empirical successes are reinforced by theoretical results showing that, in the denoising setting, the optimal complexity depends on the noise level, providing theoretical confirmation of the benefits of tunability. Finally, we introduce the first nested dropout training algorithm for latent diffusion models, enabling adaptive complexity in state-of-the-art generative models. This not only validates the broader framework but also constitutes a methodological advance for the diffusion community, with potential impact independent of inverse problems. We have focused primarily on latent generative models, but an important direction for future work is to extend tunability to broader model families, including those that operate directly in pixel space. It is also worth exploring more flexible forms of tunability: while our approach has relied on “hard” restrictions on latent dimensions, softer or continuous forms of complexity control may provide additional benefits. Another open challenge is scaling tunability to very large, high-resolution datasets while keeping computational cost manageable. Our experiments have mostly leveraged previously proposed inversion and sampling algorithms. A natural next step is to design inversion methods that are specifically tailored to tunable models, including strategies for selecting the tuning parameter automatically at inference time based on measurement conditions or noise statistics. Finally, on the theoretical side, our current analysis is restricted to linear denoising. Extending this theory to nonlinear generative models or other inverse problems, would be highly valuable. References [1] L. Ardizzone, J. Kruse, C. Rother, and U. Köthe (2019) Analyzing inverse problems with invertible neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §1. [2] M. Asim, M. Daniels, O. Leong, A. Ahmed, and P. Hand (2020) Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, p. 399–409. Cited by: Appendix B, Appendix B, §1, §1, §4, §4, Table 1. [3] A. Bekasov and I. Murray (2020) Ordering dimensions with nested dropout normalizing flows. External Links: 2006.08777 Cited by: Appendix B, §1, §4. [4] A. Bora, A. Jalal, E. Price, and A. G. Dimakis (2017) Compressed sensing using generative models. In International Conference on Machine Learning, p. 537–546. Cited by: §A.1, Appendix B, §1, §1. [5] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye (2023) Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §A.2, §1, §2.2, §4, §4, Table 1, §4. [6] J. Cocola, P. Hand, and V. Voroninski (2020) Nonasymptotic guarantees for spiked matrix recovery with generative priors. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 15185–15197. External Links: Link Cited by: §1. [7] G. Daras, Y. Dagan, A. Dimakis, and C. Daskalakis (2022-17–23 Jul) Score-guided intermediate level optimization: fast Langevin mixing for inverse problems. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 4722–4753. External Links: Link Cited by: §1. [8] G. Daras, J. Dean, A. Jalal, and A. Dimakis (2021) Intermediate layer optimization for inverse problems using deep generative models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 2421–2432. External Links: Link Cited by: §1. [9] P. Dhariwal and A. Nichol (2021) Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, p. 8780–8794. Cited by: §2.1. [10] Z. Dou and Y. Song (2024) Diffusion posterior sampling for linear inverse problem solving: a filtering perspective. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1. [11] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios (2019) Neural spline flows. Advances in neural information processing systems 32. Cited by: Appendix B, §1. [12] B. Efron (2011) Tweedie’s formula and selection bias. Journal of the American Statistical Association 106 (496), p. 1602–1614. Note: PMID: 22505788 External Links: Document, Link, https://doi.org/10.1198/jasa.2011.tm11181 Cited by: §2.2. [13] P. Esser, R. Rombach, and B. Ommer (2021-06) Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12873–12883. Cited by: §1. [14] P. Hand and B. Joshi (2019) Global guarantees for blind demodulation with generative priors. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . External Links: Link Cited by: §1. [15] P. Hand, O. Leong, and V. Voroninski (2018) Phase retrieval under a generative prior. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, p. . External Links: Link Cited by: §1, §1. [16] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 6840–6851. External Links: Link Cited by: §2.1. [17] A. Jalal, S. Karmalkar, A. Dimakis, and E. Price (2021-18–24 Jul) Instance-optimal compressed sensing via posterior sampling. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, p. 4709–4720. External Links: Link Cited by: §1, §2.2. [18] T. Karras, S. Laine, and T. Aila (2019) A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4401–4410. Cited by: §4. [19] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila (2020) Analyzing and improving the image quality of stylegan. External Links: 1912.04958, Link Cited by: §4. [20] B. Kawar, M. Elad, S. Ermon, and J. Song (2022) Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, Cited by: §2.2. [21] B. Kawar, G. Vaksman, and M. Elad (2021) SNIPS: solving noisy inverse problems stochastically. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: §2.2, §2.2. [22] D. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA. Cited by: Appendix B. [23] D. P. Kingma and M. Welling (2014) Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, External Links: http://arxiv.org/abs/1312.6114v10 Cited by: §1, §2.1. [24] D. P. Kingma and P. Dhariwal (2018) Glow: generative flow with invertible 1x1 convolutions. Advances in neural information processing systems 31. Cited by: Appendix B. [25] X. Li, Y. Ren, X. Jin, C. Lan, X. Wang, W. Zeng, X. Wang, and Z. Chen (2025) Diffusion models for image restoration and enhancement: a comprehensive survey. External Links: 2308.09388, Link Cited by: §2.2. [26] T. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár (2015) Microsoft coco: common objects in context. External Links: 1405.0312, Link Cited by: §4. [27] T. Liu, T. Yang, Q. Zhang, and Q. Lei (2023) Optimization for amortized inverse problems. In International Conference on Machine Learning, p. 22289–22319. Cited by: §1. [28] Z. Liu, P. Luo, X. Wang, and X. Tang (2015-12) Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), Cited by: §4. [29] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §A.2, §4. [30] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2022) DPM-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §A.2. [31] X. Meng and Y. Kabashima (2023) Quantized compressed sensing with score-based generative models. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1. [32] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin (2020) PULSE: self-supervised photo upsampling via latent space exploration of generative models. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, p. 2434–2442. External Links: Link, Document Cited by: §1. [33] M. Ning, E. Sangineto, A. Porrello, S. Calderara, and R. Cucchiara (2023) Input perturbation reduces exposure bias in diffusion models. External Links: 2301.11706, Link Cited by: §4, §4. [34] G. Parmar, R. Zhang, and J. Zhu (2022-06) On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11410–11420. Cited by: Appendix A, Figure 3, Figure 3. [35] O. Rippel, M. Gelbart, and R. Adams (2014) Learning ordered representations with nested dropout. In International Conference on Machine Learning, p. 1746–1754. Cited by: §1, §3.1. [36] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022-06) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10684–10695. Cited by: §1, §2.1, §4. [37] B. L. Ross and J. C. Cresswell (2021) Tractable density estimation on learned manifolds with conformal embedding flows. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: §1. [38] L. Rout, Y. Chen, A. Kumar, C. Caramanis, S. Shakkottai, and W. Chu (2024) Beyond first-order tweedie: solving inverse problems using latent diffusion. In CVPR, p. 9472–9481. External Links: Link Cited by: §2.2. [39] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai (2023) Solving linear inverse problems provably via posterior sampling with latent diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §A.2, §2.2, §3.2, Figure 5, Figure 5, §4, §4, Table 1, §4. [40] J. Scarlett, R. Heckel, M. R. D. Rodrigues, P. Hand, and Y. C. Eldar (2022) Theoretical perspectives on deep learning methods in inverse problems. IEEE Journal on Selected Areas in Information Theory 3 (3), p. 433–453. External Links: Document Cited by: §1, §2.2. [41] B. Song, S. M. Kwon, Z. Zhang, X. Hu, Q. Qu, and L. Shen (2024) Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.2, §3.2. [42] J. Song, C. Meng, and S. Ermon (2021) Denoising diffusion implicit models. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. [43] Y. Song, L. Shen, L. Xing, and S. Ermon (2022) Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations, External Links: Link Cited by: §1, §1. [44] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. [45] P. von Platen, S. Xu, P. C. Anaya, T. Wolf, and H. Schmid (2022) Diffusers: state-of-the-art diffusion models. In Proceedings of the 2022 Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, External Links: Link Cited by: §A.1, §A.2, §4. [46] H. Wang, X. Zhang, T. Li, Y. Wan, T. Chen, and J. Sun (2024) DMPlug: a plug-in method for solving inverse problems with diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2.2. [47] Y. Wang, J. Yu, and J. Zhang (2023) Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.2. [48] J. Whang, Q. Lei, and A. Dimakis (2021) Solving inverse problems with a flow-based noise model. In International Conference on Machine Learning, p. 11146–11157. Cited by: Appendix B, Appendix B, §1. [49] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. External Links: 1801.03924, Link Cited by: §3.1. Appendix A Appendix The code is publicly available here Tunable Complexity LGM for Inverse Problems. All fid scores are computed with [34]. A.1 VAE Training/Inversion Our VAE follows the AutoencoderKL architecture from von Platen et al. [45]. For CelebA, we use a latent dimensionality of 16×16×1616× 16× 16, resulting in a model with approximately 44 million parameters. For MS COCO, we use a latent space of 24×16×1624× 16× 16. Both models were trained as described in the main text, with the hyperparameter λ fixed at 0.10.1 and a geometric series success parameter of 10−310^-3. To incorporate perceptual similarity, we employed a VGG-based perceptual loss. We note that further hyperparameter tuning may yield additional improvements. For the inversion algorithm, we use gradient descent on the latent vector, following the implementation in the GitHub repository of Bora et al. [4]. (a) CelebA dataset (b) MS COCO dataset Figure 6: Reconstruction FID as a function of latent dimensionality k. Figure 7: Performance of a generative prior with tunable complexity (Tunable VAE Prior) compared to its fixed-complexity counterpart (VAE) on compressed sensing and denoising tasks on the CelebA dataset. The tunable prior achieves improved reconstructions over a range of latent dimensionalities k, as measured by LPIPS, relative to the fixed-complexity baseline. A.2 LDM Training/Inversion Following von Platen et al. [45], we trained a UNet2DModel with approximately 200 million parameters using the AdamW optimizer with a learning rate of 1×10−41× 10^-4 and a batch size of 128. We employed the DDPMScheduler provided in the diffusers package. The FID scores reported below were computed using the DPMSolverMultistepScheduler [30] with 100 sampling steps. During training, we withheld a validation set and evaluated the FID score every 20,000 iterations, retaining the three checkpoints with the lowest FID scores. For inverse problems, we selected the checkpoint with the best FID score. The FID score was computed using 50,000 training examples, while the test score was evaluated on 10,000 unseen images from the PyTorch implementation of CelebA. This procedure was followed for CIFAR-10, MS COCO, and FFHQ as well. (a) Training set CelebA (b) Test set CelebA Figure 8: FID Score as a function of latent dimensionality k. For the inversion algorithms DPS [5] and PSLD/LPDS [39], we used the official codebases but adapted them to the Diffusers framework, which already provides existing pipelines. All hyperparameters were selected by grid search over the configuration files provided on GitHub, and we report results using the best-performing settings. These search procedures will be released alongside our official code. In our experiments, DDIM sampling consistently yielded stronger reconstruction quality at intermediate latent dimensionalities compared to stochastic samplers, though a full theoretical understanding of this effect remains an open question. For Algorithm˜2, the best-performing setup used the Adam optimizer [29] with the DDIM reverse equation, 3 optimization steps per update, 500 reverse steps, and variance set to 0.1. Importantly, we found that the algorithm requires a small amount of variance in the reverse diffusion step, as setting σt=0 _t=0, i.e., fully deterministic DDIM sampling, leads to unstable or degraded reconstructions. This approach is particularly effective in our setting due to the relatively large latent space compared to the image space. The overall objective was to implement a straightforward inversion algorithm that can be readily applied across a wide range of problems. Tunable Priors on Various Datasets Each problem was solved with a batch size of 12 for 5 random samples from the test set that the models were not trained on. The blue shaded represents 1 standard deviation. Figure 9: Performance of a generative prior with tunable complexity (Tunable LDM Prior) and its fixed-complexity counterpart (Baseline) for compressed sensing and inpainting on MS COCO. The tunable prior demonstrates a range of parameters k that yield better LPIPS scores than the baseline. Figure 10: Performance of a generative prior with tunable complexity (Tunable LDM Prior) and its fixed-complexity counterpart (Baseline) for compressed sensing and inpainting on CIFAR-10. The tunable prior demonstrates a range of parameters k that yield better LPIPS scores than the baseline. How does the number of measurements affect where the optimal latent dimensionality for reconstruction is? Figure 11: Compressed sensing performance as a function of latent dimensionality k for different measurement regimes. Figure 12: Denoising performance as a function of latent dimensionality k for different measurement regimes. How the optimal k was selected for Table˜1. We select the latent dimensionality k using a held-out validation set. Specifically, the value of k achieving the lowest mean LPIPS score on the validation set is used for the results reported in Table˜1. The same procedure is applied for the experiments in Table˜2. Figure 13: Validation performance used to select the latent dimensionality k. Figure 14: Generated samples from the latent diffusion model with different latent dimensionalities. Each row shows images produced when only the first k of the 40964096 latent dimensions are kept during sampling (k=4096k=4096: 100%, k=3600k=3600: 88%, k=3000k=3000: 73%, k=2500k=2500: 61%, k=2000k=2000: 49%, k=1500k=1500: 37%.) Figure 15: Qualitative results based on Table 2. Red indicates the lowest LPIPS Score. FID Scores for FFHQ Model We evaluate the FID scores as a function of latent dimensionality k (12×64×6412× 64× 64) for the FFHQ training set of size 10k. The results are shown in Figure 16. Figure 16: FID as a function of latent dimensionality K for the FFHQ training set of size 10k. Appendix B Normalizing Flow Inverse Problems We assume the practitioner has trained a tunable normalizing flow with the following objective provided by [3]. Our formulation is for latent-variable generative priors. For simplicity assume the measurements have the following form: the forward operator =∈Rm×nA= A∈ R^m× n and the additive noise term is i.i.d ∼(0,σ2m) η (0,σ^2 I_m), as in the compressed sensing case. Thus the measurements are generated by =+ y=A x+ η, where ∈ℝn x ^n is an unknown signal. Consider a fixed invertible neural network G:ℝn→ℝnG:R^n ^n. Define for each k∈[n]=1,…,nk∈[n]=\1,…,n\ the map Gk:ℝk G_k ^k ⟶ℝn ^n 8.5359pt z z ⟼G((0)). G ( pmatrix z\\ 0 pmatrix ). We note that k is the parameter that governs the complexity of the prior, and (0)∈ℝn pmatrix z\\ 0 pmatrix ^n is obtained by latent representation ∈ℝk z ^k. Then the signal representation is given by =G((T,0)) x=G(( z^T,0)) where ∈ℝn x ^n is as above. We are interested in recovering a signal ∈ℝn x ^n given a set of noisy measurements ∈ℝm y ^m. Our prior has a valid density over the entire signal space when x∈Range(Gk)x (G_k), therefore a natural attempt to solve the given inverse problems is by a maximum a posteriori estimation: x^MAP(k) x_MAP(k) :=argmin∈Range(Gk)−logp(|) := *arg\,min_ x (G_k)\;- p( x| y) =argmin∈Range(Gk)−logp(|)−logpGk() = *arg\,min_ x (G_k)\;- p( y| x)- p_G_k( x) =argminx∈Range(Gk)12‖−‖2−σ2logpGk() = *arg\,min_x (G_k)\; 12\| y- A x\|^2-σ^2 p_G_k( x) where pGkp_G_k is the density function on x induced by GkG_k, ∥⋅∥\|·\| is the Euclidean norm, and σ is given by the model noise. Similar to previous work of [2, 48], we optimized over the latent space given by: ^MAP(k) z_MAP(k) :=argmin∈ℝk12‖−G((0))‖2−σ2logpGk(G((0))) := *arg\,min_ z ^k\; 12 \| y- AG ( pmatrix z\\ 0 pmatrix ) \|^2-σ^2 p_G_k (G ( pmatrix z\\ 0 pmatrix ) ) The proposed optimization problem above is solved via gradient descent with Adam optimizer [22] and is initialized at z=0z=0. Depending on the specific inverse problem at hand the tunable parameter k can be selected to change the dimensionality of the model, so the first k elements of the vector z will be optimized over and the rest will be set to zero. Our formulation relies on a density function pGkp_G_k, and works by [2, 48] have shown that a smoothing parameter on a density function improves performance when solving the optimization problem above. We empirically observe the same phenomena, and as a result, we replace σ2σ^2 with a hyperparameter γ. For more information about choosing hyperparameters and how to compute pGkp_G_k, please refer to the Appendix. Therefore, instead, we optimized over a modified MAP estimate given by ^MAP(k) z_MAP(k) :=argmin∈ℝk12‖−G((0))‖2−γlogpGk(G((0))). := *arg\,min_ z ^k\; 12 \| y- AG ( pmatrix z\\ 0 pmatrix ) \|^2-γ p_G_k (G ( pmatrix z\\ 0 pmatrix ) ). For the VAE we follow the same formulation as [4], which is similar to the one above, but instead of having a tractable density, we approximate with ℓ2 _2 penalty in the latent space. Figure 17: Results on inpainting and denoising across various measurement regimes for Normalizing Flow. The tuning parameter should be chosen based on the complexities of the given inverse problem. The red bounding box indicates which model achieved the lowest reconstruction error with respect to SSIM. Figure 18: Performance of a generative prior with tunable complexity (Tunable NF Prior) and its fixed-complexity counterpart (Normalizing Flow) on compressed sensing, inpainting, phase retrieval, and denoising tasks on the CelebA dataset. The tunable prior exhibits a range of latent dimensionalities k for which it yields improved reconstructions, as measured by the perceptual metric LPIPS, compared to the fixed-complexity baseline. Generative Priors with Tunable Complexity on Various Architectures In these experiments we want to demonstrate that generative priors with tunable complexities are not tied to one specific Normalizing Flow architecture. Here we train three architectures: Rational Quadratic Spline Flow [11], GLOW [24], and architecture inspired by glow called Glow-ADD which uses an additive coupling layer instead of an affine. Each architecture was trained Figure 19: Results showing for different Normalizing Flow architectures. All architectures demonstrate they benefit from a prior with a tunable complexity. Figure 20: Performance of generative prior with a tunable complexity (Tunable Prior) and its fixed complexity counterpart (Normalizing Flow) for compressed sensing, inpainting, deblurring, and denoising on STL dataset. Tunable prior demonstrates a range of tunable parameters k that lead to a better estimate of the target signal measured in PSNR than its baseline. Figure 21: Performance of generative prior with a tunable complexity (Tunable Prior) and its fixed complexity counterpart (Normalizing Flow) for compressed sensing, inpainting, deblurring, and denoising on STL dataset. Tunable prior demonstrates a range of tunable parameters k, leading to a better estimate of the target signal measured in SSIM than its baseline. Appendix C Proofs Proof of Theorem˜5.1. Without loss of generality, we take GkG_k to be a diagonal matrix. This follows because of invariance of Euclidean norms with respect to orthogonal transformations and because of the rotational invariance of the random variable z_0. Let kP_k and k⟂P_k be the orthogonal projectors onto span(e1,…,ek)span(\e_1,…,e_k\) and its orthogonal complement, respectively. For a diagonal matrix Σ=diag(s1,s2,⋯,sn)∈ℝn×n =diag(s_1,s_2,·s,s_n) ^n× n, let k∈ℝk×k S_k ^k× k and k⟂∈ℝn−k×n−k S_k ^n-k× n-k be such that =(k⟂). = pmatrix S_k& 0\\ 0& S_k pmatrix. We want to find the expectation 0,η‖^γ(k)−0‖2 _ x_0,η\| x_γ(k)- x_0\|^2 =0,‖k(^γ(k)−0)‖2+0‖k⟂(0)‖2. =E_ x_0, η\|P_k( x_γ(k)- x_0)\|^2+E_ x_0\|P_k ( x_0)\|^2. (9) For the equation above we will compute the first term on the right hand side first, then do the same for the second term on the right hand side. The first term on the right hand side breaks into sum of two terms which can readily computed, while the second term on the right hand side follows from a lemma. Notice k(^γ(k))=k^γ(k)∈ℝkP_k( x_γ(k))= S_k z_γ(k) ^k and the solution of the optimization problem equation 8 is ^γ(k): z_γ(k): =(kkT+γk)−1kTk(y) =( S_k S_k^T+γ I_k)^-1 S_k^TP_k(y) =(kkT+γk)−1kT(kk(z0)+k()). =( S_k S_k^T+γ I_k)^-1 S_k^T( S_kP_k(z_0)+P_k( η)). Thus, 0,∥ _ x_0, η\| k(^γ(k)−0)∥2=,∥k^γ(k)−k()∥2 _k( x_γ(k)- x_0)\|^2=E_ z_0, η\| S_k z_γ(k)- S_kP_k( z_0)\|^2 =,‖k(kkT+γk)−1kTk(y)−kk()‖2 =E_ z_0, η\| S_k( S_k S_k^T+γ I_k)^-1 S_k^TP_k(y)- S_kP_k( z_0)\|^2 =,∥k(kkT+γk)−1kT(kk(z0)+k()) =E_ z_0, η\| S_k( S_k S_k^T+γ I_k)^-1 S_k^T( S_kP_k(z_0)+P_k( η)) −k()∥2 - S_kP_k( z_0)\|^2 =,∥(k(kkT+γk)−1k−k)k() =E_ z_0, η\|( S_k( S_k S_k^T+γ I_k)^-1 S_k- S_k)P_k( z_0) +k(kkT+γk)−1kTk()∥2 + S_k( S_k S_k^T+γ I_k)^-1 S_k^TP_k( η)\|^2 (10) The last term above Equation˜10 can be decomposed ‖(k(kkT+γk)−1k−k)k(0)‖2 _ z_0 \| ( S_k( S_k S_k^T+γ I_k)^-1 S_k- S_k )P_k( z_0) \|^2 (11) +,⟨(k(kkT+γk)−1k−k)k(),k(kkT+γk)−1kTk()⟩ +E_ z_0, η ( S_k( S_k S_k^T+γ I_k)^-1 S_k- S_k )P_k( z_0), S_k( S_k S_k^T+γ I_k)^-1 S_k^TP_k( η) (12) +‖k(kkT+γk)−1kTk()‖2 +E_ η \| S_k( S_k S_k^T+γ I_k)^-1 S_k^TP_k( η) \|^2 (13) For the term above, the expectation of inner product term (Equation˜12) is zero because both random variables have an expectation of zero. Furthermore, apply Lemma˜C.1 to Equation˜11, yielding _ z_0 ‖(k(kkT+γk)−1kTk−k)k(0)‖2 \|( S_k( S_k S_k^T+γ I_k)^-1 S_k^T S_k- S_k)P_k( z_0)\|^2 =‖(kkkT+γk)−1kTk−k‖F2 =\|( S_k S_k S_k^T+γ I_k)^-1 S_k^T S_k- S_k\|_F^2 =∑i=1k|si3si2+γ−si|2 = _i=1^k | s_i^3s_i^2+γ-s_i |^2 =∑i=1ksi2γ2(si2+γ)2. = _i=1^k s_i^2γ^2(s_i^2+γ)^2. (14) Then apply Lemma˜C.2 to Equation˜13 _ η ‖k(kkT+γk)−1kTk()‖2 \| S_k( S_k S_k^T+γ I_k)^-1 S_k^TP_k( η)\|^2 =Tr(k(kkT+γk)−1kTσ2kk(kkT+γk)−1kT) =Tr( S_k( S_k S_k^T+γ I_k)^-1 S_k^Tσ^2 I_k S_k( S_k S_k^T+γ I_k)^-1 S_k^T) =∑i=1ksi4σ2(si2+γ)2. = _i=1^k s_i^4σ^2(s_i^2+γ)^2. (15) This concludes the calculation of the first term on the right hand side of Equation˜9. Now we apply Lemma˜C.1 to the second term on the right hand side 0‖k⟂(0)‖2=z0‖k⟂k⟂(z0)‖2=‖k⟂‖F2=∑j=k+1nsj2. _ x_0\|P_k ( x_0)\|^2=E_z_0\| S_k P_k (z_0)\|^2=\| S_k \|_F^2= _j=k+1^ns_j^2. (16) Lastly, combine Eq.16,14,15 and simplify 0,‖^γ(k)−0‖2=∑i=1ksi2(si2σ2+γ2)(si2+γ)2+∑j=k+1nsj2.E_ x_0, η\| x_γ(k)- x_0\|^2= _i=1^k s_i^2(s_i^2σ^2+γ^2)(s_i^2+γ)^2+ _j=k+1^ns_j^2. ∎ Proof of Corollary 5.2. For notational convenience, let E(k)=0,‖^γ(k)−0‖2E(k)=E_ x_0, η\| x_γ(k)- x_0\|^2. Since [n][n] is a finite non-empty set, argminkE(k) *arg\,min_k\,E(k) exists, and so we consider the set C of candidate minimizers, C=k∣E(k−1)≥E(k).C=\k E(k-1)≥ E(k)\. Notice E(k−1)−E(k)≥0E(k-1)-E(k)≥ 0 for each k∈Ck∈ C, and moreover that this difference is ∑i=1k−1si2(si2σ2+γ2)(si2+γ)2+∑j=knsj2−∑i=1ksi2(si2σ2+γ2)(si2+γ)2 _i=1^k-1 s_i^2(s_i^2σ^2+γ^2)(s_i^2+γ)^2+ _j=k^ns_j^2- _i=1^k s_i^2(s_i^2σ^2+γ^2)(s_i^2+γ)^2 −∑j=k+1nsj2≥0 - _j=k+1^ns_j^2≥ 0 ⇔ sk−sk2(sk2σ2+γ2)(sk2+γ)2≥0 s_k- s_k^2(s_k^2σ^2+γ^2)(s_k^2+γ)^2≥ 0 ⇔ sk≥σ2−2γ, s_k≥ σ^2-2γ, which reveals C=k∣sk≥σ2−2γ.C=\k s_k≥ σ^2-2γ\. From our assumptions, σ2−2γ≥0σ^2-2γ≥ 0. Note now that for any i,j∈Ci,j∈ C with i<ji<j, we have that sj≥sk≥si≥σ2−2γfor all i<k≤j, s_j≥ s_k≥ s_i≥ σ^2-2γ all $i<k≤ j$, hence k∈Ck∈ C. To prove the corollary is to prove that argminkE(k)=maxC *arg\,min_kE(k)= C, so it suffices to show E is decreasing on C. Since si2≥σ2−2γs_i^2≥σ^2-2γ for all i∈Ci∈ C, it is clear that si4+si2(2γ−σ2)≥si4−si2⋅si2=0 s_i^4+s_i^2(2γ-σ^2)≥ s_i^4-s_i^2· s_i^2=0 is non-negative. From this, we deduce that si2σ2+γ2≤(si2+γ)2s_i^2σ^2+γ^2≤(s_i^2+γ)^2, hence Δ(i):=1−si2σ2+γ2(si2+γ)2≥0 (i):=1- s_i^2σ^2+γ^2(s_i^2+γ)^2≥ 0 is also non-negative. Let P,Q∈CP,Q∈ C be such that P≤QP≤ Q. We’l show E(P)≥E(Q)E(P)≥ E(Q) by showing the difference E(P)−E(Q)E(P)-E(Q) is non-negative. It is easily computed by E(P)−E(Q)=∑i=P+1Qsi2Δ(i). E(P)-E(Q)= _i=P+1^Qs_i^2 (i). Previous work shows that Δ(i) (i) is non-negative for all i in the range P<i≤QP<i≤ Q since all such i are elements of C, thus concluding the proof. ∎ Lemma C.1. Let ∼(0,n)∈ℝn x (0, I_n) ^n then for any matrix ∈ℝm×n M ^m× n, we have ‖‖2=‖F2. _ x\| M x\|^2=\| M\|_F^2. Proof of Lemma C.1. Compute that ‖‖2 _ x\| M x\|^2 =⟨,⟩ =E_ x M x, M x =⟨T,T⟩ =E_ x M^T M, x x^T =⟨T,n⟩ = M^T M, I_n =‖F2. =\| M\|_F^2. ∎ Lemma C.2. Let ∼(0,)∈ℝn x (0, ) ^n then for any matrix ∈ℝm×n M ^m× n, we have ‖‖2=Tr(T). _ x\| M x\|^2=Tr( M M^T). Proof of Lemma C.2. Let = Y= M x, then ∼(0,T) Y (0, M M^T). Thus, ‖‖2=Y‖Y‖2=[T]=∑ini2=∑i=1nVar(i) _ x\| M x\|^2=E_Y\|Y\|^2=E[ Y^T Y]= _i^n Y_i^2= _i=1^nVar( Y_i) =Tr(T). =Tr( M M^T). (17) ∎