Paper deep dive
Imaginative Generative AI: Crossing the Entropy Wall into Worlds Beyond Imitation
Hossein Goli, Farzan Farnia, Amin Gohari
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative AI models are primarily designed to imitate the data distribution, an objective that neither corrects diversity lost by a learned generator nor defines how generation should extend beyond the diversity of the data itself. We introduce Imaginative Generative AI (IGA), a framework that makes diversity part of the target-distribution design problem: among distributions close to a reference, IGA selects one whose spectral diversity reaches a prescribed level. Diversity is measured by the von Neumann entropy of the generated distribution's kernel covariance operator in a fixed representation space, providing a reference-free representation-guided measure of how broadly probability mass occupies embedding directions. The spectral entropy of the population data distribution defines an Entropy Wall. Below the wall, IGA performs diversity repair, recovering variation that a learned generator has lost while remaining within the diversity level of the data. Beyond the wall, the data distribution itself becomes infeasible, and IGA deliberately departs from it to produce distributions with greater representation-relative spectral diversity, an operational notion of imaginative generation. These regimes form a single regularization path from imitation to imagination and define an i.i.d. target distribution at each prescribed diversity level. We develop the theory of this entropy-constrained projection and show that, under a KL anchor to a pretrained generator, the optimum satisfies a self-consistent exponential-tilt relation. This characterization leads to IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDPM and DDIM samplers. Experiments on synthetic and vision benchmarks demonstrate diversity repair below the Entropy Wall and controlled spectral extrapolation beyond it.
Tags
Links
- Source: https://arxiv.org/abs/2608.09385v1
- Canonical: https://arxiv.org/abs/2608.09385v1
Trouble viewing inline? Open PDF directly →
Full Text
199,985 characters extracted from source content.
Expand or collapse full text
Imaginative Generative AI : Crossing the Entropy Wall into Worlds Beyond Imitation Hossein Goli , Amin Gohari , Farzan Farnia Department of Computer Science and Engineering, The Chinese University of Hong Kong, hosseingoli@cse.cuhk.edu.hkDepartment of Information Engineering, The Chinese University of Hong Kong, agohari@ie.cuhk.edu.hkDepartment of Computer Science and Engineering, The Chinese University of Hong Kong, farnia@cse.cuhk.edu.hk Abstract Generative AI models are primarily designed to imitate the data distribution, an objective that neither corrects diversity lost by a learned generator nor defines how generation should extend beyond the diversity of the data itself. We introduce Imaginative Generative AI (IGA), a framework that makes diversity part of the target-distribution design problem: among distributions close to a reference, IGA selects one whose spectral diversity reaches a prescribed level. Diversity is measured by the von Neumann entropy of the generated distribution’s kernel covariance operator in a fixed representation space, providing a reference-free representation-guided measure of how broadly probability mass occupies embedding directions. The spectral entropy of the population data distribution defines an Entropy Wall. Below the wall, IGA performs diversity repair, recovering variation that a learned generator has lost while remaining within the diversity level of the data. Beyond the wall, the data distribution itself becomes infeasible, and IGA deliberately departs from it to produce distributions with greater representation-relative spectral diversity, an operational notion of imaginative generation. These regimes form a single regularization path from imitation to imagination and define an i.i.d. target distribution at each prescribed diversity level. We develop the theory of this entropy-constrained projection and show that, under a KL anchor to a pretrained generator, the optimum satisfies a self-consistent exponential-tilt relation. This characterization leads to IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDPM and DDIM samplers. Experiments on synthetic and vision benchmarks demonstrate diversity repair below the Entropy Wall and controlled spectral extrapolation beyond it. 1 Introduction “Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world.” — Albert Einstein The typical goal of a generative model is to reproduce the underlying distribution of its training data. For example, a successful image generator is expected to produce realistic images with approximately the same content and variation as the images on which it was trained. This principle underlies the standard paradigms of generative modeling in the literature, including generative adversarial networks (GANs) [13], variational autoencoders (VAEs) [23], and score-based and diffusion models [46, 15, 47]. Although these frameworks differ significantly in architecture and training, their population-level goal can be summarized in the following distributional discrepancy minimization: minQ∈(Q;Pdata), _Q _G\;D (Q;P_data ), (1) where PdataP_data is the underlying data distribution, Q is the distribution produced by the generator over the set of feasible models P_G, and (⋅;⋅)D(·;·) measures the discrepancy (or divergence) between the two distributions. We refer to this prevailing view of generative modeling as distributional imitation. Imitation is a natural statistical objective, but it also places a ceiling on what the generator is asked to do. Even an ideal solution of (1) is asked to match PdataP_data, not to produce a distribution that is systematically more diverse or generate novel and creative content. Moreover, practical generators may not reach even the diversity of their training distribution. The recent study [10] by Farnia, et al. has found that generated samples can exhibit lower spectral diversity than real data when diversity is measured using reference-free diversity measures of the Vendi score [11] and Rényi kernel entropy [19]. This raises a fundamental question: How should the target of generative modeling be regularized when diversity and novelty, in addition to fidelity, are something we want to control? To address this question, we propose Imaginative Generative AI (IGA), a framework that makes diversity part of the target-distribution design problem. Instead of asking only for the distribution closest to a reference distribution, IGA asks for the closest distribution whose diversity score is at least above a given prescribed level. Let PrefP_ref denote the reference distribution, which could be the empirical distribution P^n P_n of training data or the distribution PθP_θ of a pretrained generator. Then, IGA solves the following regularized discrepancy minimization problem: minQ∈ _Q _G (Q;Pref) \! (Q;P_ref ) (2) subject to H(Q)≥ρ, H(Q)≥ρ, where H(Q)H(Q) measures the spectral entropy (interpreted as diversity) of distribution Q and ρ is the desired diversity level. Under the duality conditions developed in our theoretical analysis, the constrained problem at level ρ can equivalently be stated using a Lagrangian penalty at a corresponding multiplier λ≥0λ≥ 0: minQ∈(Q;Pref)−λH(Q) _Q _G\;D\! (Q;P_ref )-λ\,H(Q) (3) Note that the two terms have complementary roles: The discrepancy term keeps generated samples close to the reference distribution, while the entropy term rewards the spectral diversity in the Vendi score. Setting λ=0λ=0 recovers standard reference matching; increasing λ gives more weight to the spectral diversity term. We measure diversity using the von Neumann entropy (VNE) of the normalized kernel covariance operator induced by Q in a fixed embedding space. Intuitively, VNE is low when generated samples concentrate along a few embedding directions and high when they spread across many directions; in the empirical setting, it is the logarithm of the Vendi score [11, 19, 34]. This measure is reference-free but representation-dependent: it evaluates the diversity of Q without requiring a comparison distribution, while the chosen embedding specifies which variations are meaningful. IGA thus controls spectral diversity relative to a given embedding. The data distribution itself provides a natural reference level for this diversity. To characterize this wall, we define the underlying distribution’s entropy as ρ⋆:=H(Pdata). _ :=H (P_data ). (4) We call ρ⋆ _ the Entropy Wall. It is the spectral diversity of the data distribution in the chosen representation. The notion of entropy wall separates two different regimes in the IGA generative modeling approach: (Regime I) Below the Entropy Wall: Diversity Repair. The below-the-wall regime concerns IGA when we choose entropy lower-bound ρ to satisfy ρ≤ρ⋆ρ≤ _ . In this regime, the required diversity level is no greater than the diversity already present in the underlying data distribution. Therefore, in this regime, IGA can then be viewed as repairing a diversity deficit in a learned generator: it encourages the generator to recover variation that was present in the data but weakened or lost during training the generative model. Note that, as empirically demonstrated by Farnia et al. in recent work [10], the standard generative models commonly suffer from a diversity bias, and the spectral entropy of their generated data cannot match that of the underlying distribution generating their training samples. In brief, the goal in this regime remains faithful modeling of the data, with an explicit mechanism for counteracting spectral diversity shortfall as shown in [10]. (Regime I) Beyond the Entropy Wall: Imaginative Generation. This regime of applying IGA is when we select the projection lower-bound to satisfy the strict inequality: ρ>ρ⋆ρ> _ . Especially, we highlight that in this regime, the data distribution itself no longer satisfies the diversity constraint, and thus the IGA solution in (2) must intentionally differ from PdataP_data, regardless of whether it is anchored directly to the data or to a pretrained model. The discrepancy term prevents this solution from moving arbitrarily far from the chosen reference PrefP_ref, while the entropy constraint pushes it to occupy a broader set of embedding directions. We call this regime imaginative because the target has greater spectral diversity than the data distribution that defines the wall. Figure 1: From Imitation to Imagination. Imitation aims to return a distribution Q close to PdataP_data; IGA returns the closest Q with H(Q)≥ρH(Q)≥ρ, which meets the data distribution below and at the entropy wall ρ⋆=H(Pdata) _ =H(P_data) and leaves it beyond. Images are Stable Diffusion XL at matched prompts and seeds; the right column sweeps one multiplier from the low-entropy projection to strong extrapolation. Figure 2: From imitation to imagination with SDXL. As the IGA multiplier λ increases, SDXL produces progressively stronger structural and compositional variations while preserving the underlying concept. Here, we use the term imaginative to describe generation whose spectral diversity exceeds that of the data in a specified representation space, e.g. CLIP or DINO embedding spaces for images. This representation-relative definition makes imagination operational while allowing the embedding to encode domain-relevant semantics. We evaluate the resulting variation using both the guiding representation and independent measures of sample quality and diversity. The two regimes form a single regularization path. As the target ρ increases, IGA moves from ordinary imitation to diversity repair and then, after crossing the entropy wall ρ⋆=H(Pdata) _ =H(P_ data) (corresponding to the real data distribution), to deliberate imagination. Therefore, our IGA framework defines diversity enhancement as a property of a single target distribution, enabling i.i.d. generation at a prescribed spectral-diversity level, including levels beyond the entropy of the data. We emphasize that this differs from methods that induce diversity through repulsive or sample-dependent interactions over the course of generating multiple samples, whose outputs form a coupled and generally non-i.i.d. batch [41, 7, 18]. IGA therefore provides an i.i.d. distributional alternative to interaction-based diversity promotion, while complementing work that evaluates diversity after generation [11, 19]. Next, we demonstrate that this distribution-level formulation further leads to a practical method for diversity-improved sampling from pretrained score-based and diffusion models. When the reference is a pretrained model PθP_θ and the discrepancy measure is the KL-divergence KL(Q∥Pθ)KL(Q\|P_θ), the analysis in the main body shows that the optimal target takes the form Q⋆(dx)∝Pθ(dx)⋅exp(λGQ⋆(x)),Q (dx)\, \,P_θ(dx)· (λ\,G_Q (x) ), (5) where GQ⋆(x)G_Q (x) measures how placing probability mass near x changes the VNE spectral entropy term. Because this energy depends on Q⋆Q , the relation is self-consistent. It reweights the base model to increase the spectral diversity of the generated population while remaining close to the original distribution as much as possible. We specifically show that the application of this formulation to score-based and generative models can be performed by our proposed IGA Guidance. IGA Guidance is an inference-time approximation for score-based, DDPM, and DDIM samplers that requires no retraining. Our numerical experiments show promising results of the IGA guidance for large-scale diffusion models. For example, Figure 2 shows the application of IGA-Guidance to the large-scale SD-XL model and how increasing the parameter λ leads to visually more diverse and imaginative image outputs for the input prompt ”A Skyscraper”. The summary of our contributions are as follows: ∙ We formulate IGA, an entropy-constrained projection framework enabling i.i.d. generation at prescribed VNE diversity levels. ∙ We introduce the entropy wall, separating diversity repair from controlled extrapolation beyond the data’s spectral diversity. ∙ We characterize the IGA regularization path and derive a self-consistent exponential tilt for the KL-anchored optimum. ∙ We develop IGA guidance for inference-time steering of pretrained score-based, DDPM, and DDIM samplers without retraining. imagined worlds (>⋆ρ> _ )spectral diversity H⋆ _ =λ=0 (anchor)=2.16H=2.16 =λ=5=2.64H=2.64 =λ=20=3.36H=3.36 =λ=30=3.65H=3.65entropy wall⋆=() _ =H(P_data)Crossing the Entropy Wall into Worlds Beyond Imitation Figure 3: A multiverse of diversity levels. Matched-seed PixArt-Σ samples at increasing λ in the IGA framework (λ=0λ=0 represents the original regularization-free PixArt-Σ ), arranged by measured spectral entropy H. Prompt: “A skyscraper for a humid coastal city” The entropy wall ρ⋆=H(Pdata) _ =H(P_data), i.e., the entropy of the real data distribution, separates data-consistent imitated models from higher-diversity imagined distributions. 2 Preliminaries In this section, we introduce the representation-level and distributional quantities used throughout the paper. We associate every distribution with a normalized kernel covariance matrix whose spectrum defines our notion of diversity and introduce a differentiable smoothed surrogate of the resulting entropy together with its per-sample energy. Extended conventions and an elementary Gibbs-tilt identity are deferred to Appendix C. Throughout this work, PdataP_data denotes the population data distribution and P^N=1N∑i=1NδXi P_N= 1N _i=1^N _X_i the empirical measure of N i.i.d. samples from it; the upper-case letter N is reserved for generic empirical sample counts (evaluation batches, generated minibatches), and the lower-case letter n for the size of the training set, whose empirical measure we denote with P^n P_n. 2.1 Representation Space and Kernel Covariance Matrix Consider a measurable representation map ϕ:→ℝdφ:X ^d satisfying ‖ϕ(x)‖2=1 φ(x) _2=1 for every x∈x , which induces the normalized kernel k(x,x′)=ϕ(x)⊤ϕ(x′)k(x,x )=φ(x) φ(x ). For a probability distribution Q on X, we associate with the representation its kernel covariance matrix ΣQ:=X∼Q[ϕ(X)ϕ(X)⊤]∈ℝd×d. _Q:=E_X Q [φ(X)φ(X) ] ^d× d. The unit-norm normalization makes ΣQ _Q a density matrix, i.e., a positive semidefinite matrix with unit trace, whose spectrum records how the representation of Q distributes its mass across orthogonal feature directions. The covariance spectrum carries the notion of diversity developed next. The same spectrum is accessible from pairwise similarities. Given samples x1,…,xNx_1,…,x_N with Gram matrix K=[k(xi,xj)]i,j=1NK=[k(x_i,x_j)]_i,j=1^N, the empirical kernel covariance Σ^N:=1N∑i=1Nϕ(xi)ϕ(xi)⊤ _N:= 1N _i=1^Nφ(x_i)φ(x_i) shares its nonzero eigenvalues with 1NK 1NK. Consequently, every spectral quantity introduced below can be computed from the normalized Gram matrix without explicitly forming the feature vectors. 2.2 Spectral Entropy Measures and Diversity Scores Following the discussion in [4, 11, 19], we use the following definition for the von Neumann entropy of a distribution Q as H0(Q):=−Tr(ΣQlogΣQ),H_0(Q):=-Tr ( _Q _Q ), (6) which is the Shannon entropy of the covariance spectrum. We note that the exponential of the above quantity Vendi(Q):=exp(H0(Q))Vendi(Q):= (H_0(Q)) is the Vendi score [11]. This can be interpreted as an effective number of occupied feature directions. For example, a spectrum uniform over r orthogonal directions yields H0(Q)=logrH_0(Q)= r and Vendi(Q)=rVendi(Q)=r. This notion of diversity is inherently reference-free, yet it depends on the representation model to embed the data: evaluating H0(Q)H_0(Q) requires no comparison distribution, while the fixed choice of ϕφ, or equivalently k, determines which variations count as distinct. Moreover, since S↦−Tr(SlogS)S -Tr(S S) is concave on density matrices and Q↦ΣQ _Q is affine, H0H_0 is concave in Q (Lemma 2). At rank-deficient covariance matrices, however, H0H_0 need not be differentiable. The guidance analysis of Section 5 requires a well-defined first variation, so we move the covariance uniformly away from the boundary of the density-matrix cone. For ε∈(0,1) ∈(0,1), we define the smoothed covariance and entropy SQε:=(1−ε)ΣQ+εIdd,Hε(Q):=−Tr(SQεlogSQε).S_Q :=(1- ) _Q+ \, I_dd, H_ (Q):=-Tr (S_Q S_Q ). (7) The spectral floor SQε⪰εdIdS_Q dI_d places every eigenvalue in [εd, 1−ε(1−1d)][ d,\,1- (1- 1d)] and guarantees that HεH_ is differentiable throughout the feasible covariance set. The derivative of HεH_ acts on individual samples through one quantity that recurs at every stage of the paper. We define the entropy energy GQε(x):=(1−ε)ϕ(x)⊤(−logSQε)ϕ(x),G _Q(x):=(1- )\,φ(x) (- S_Q )φ(x), (8) which is the first variation of HεH_ at Q (Lemma 1, Section 5). Since −logSQε- S_Q has large eigenvalues precisely where SQεS_Q has small ones, GQε(x)G _Q(x) is large when ϕ(x)φ(x) aligns with feature directions that Q underrepresents, and the spectral floor gives the uniform bound 0≤GQε(x)≤(1−ε)log(d/ε)0≤ G _Q(x)≤(1- ) (d/ ). The energy reappears as the payoff of the spectral adversary at training time (Section 3) and as the exponent of the guidance tilt at sampling time (Section 5). The two entropy functionals play distinct roles in our analysis: Section 4 uses H0H_0 to define population spectral diversity, whereas Section 5 uses HεH_ to derive the guidance potential. Lemma 3 quantifies their uniform proximity as ε becomes small. 3 The IGA Framework: From Imitative to Imaginative Generative Modeling Distributional imitation, which is mathematically formulated in (1), requires a generative model to match a reference distribution. In the IGA framework which we formulate in this work, we intentionally augment this objective by requiring the generated distribution to attain a prescribed level of spectral diversity, and the resulting formulation separates two questions: (i) what distribution should be targeted, and (i) how should that target be realized by a generative model? In what follows, we first present the distribution-level objective function and the regularized form in IGA. We then derive a min-max reformulation of the entropy reward in the IGA optimization, and then develop the framework’s application to sampling-time and training-time settings. 3.1 Formulating IGA via Constraining and Penalizing the Spectral Entropy Consider a reference distribution PrefP_ref, which in our applications is either the empirical data distribution P^n P_n for the observed training data or the underlying distribution PθP_θ of the (already trained) generative model. We also consider the spectral entropy functional H. Given a divergence measure (Q;Pref)D(Q;P_ref), IGA selects the most faithful distribution whose spectral diversity reaches a target level ρ∈ℝρ : minimizeQ∈ Q minimize (Q;Pref) (Q;P_ref) (Cρ) subject to H(Q)≥ρ. H(Q)≥ρ. The divergence objective function anchors the solution to PrefP_ref, ensuring the distribution solution remains as close as possible to the reference distribution, while the constraint specifies the desired spectral diversity level. In next section, we review and extend the discussion from [10] to interpret ρ relative to the entropy of the data, distinguishing diversity repair from spectral extrapolation. We note that due to the convex structure of the above optimization, the application of standard convex duality shows that (Cρ) is equivalent to the Lagrangian formulation for a corresponding Lagrangian multiplier parameter λ≥0λ≥ 0: Qλ=argminQ∈Fλ(Q):=(Q;Pref)−λH(Q)Q_λ= Q \! \;F_λ(Q):=D(Q;P_ref)-λ H(Q) (Pλ) Remark 1. For every constrained solution satisfying the standard regularity conditions, there is a multiplier λ≥0λ≥ 0 for which the same distribution solves (Pλ). Conversely, each QλQ_λ solves (Cρ) at its attained diversity level. The precise duality and attainment statements are given in Appendix D. A Min-Max Formulation of the IGA Optimization. The spectral entropy term in (Pλ) is a nonlinear function of the covariance spectrum, yet it admits a precise convex dual formulation due to its concavity. For the smoothed entropy, the application of the Gibbs variational principle for the matrix-based entropy converts the smoothed IGA problem into a two-player game. Proposition 1 (Spectral Min-Max Formulation of IGA Optimization). Consider smoothed spectral entropy with parameter ε∈(0,1) ∈(0,1). Define the spectral adversary class as ε:=Θ∈ℝd×d:Θ=Θ⊤,Tr(Θ)=0,‖Θ‖op≤log(d/ε).T_ := \ ^d× d:\ = ,\ Tr( )=0,\ _op≤ (d/ ) \. Then, the following equivalences hold: (i) Dual representation of the negative entropy. For every distribution Q on X, we have −Hε(Q)=maxΘ∈ε−(1−ε)X∼Q[ϕ(X)⊤Θϕ(X)]−logTr(e−Θ),-H_ (Q)= _ _ \-(1- )\,E_X Q [φ(X) \,φ(X) ]- (e^- ) \, (9) and the maximum is attained at the unique traceless matrix Θ⋆(Q):=−logSQε+1dTr(logSQε)Id∈ε. (Q):=- S_Q + 1dTr ( S_Q )I_d _ . (i) Min-Max form of (Pλ). For every λ≥0λ≥ 0, distribution PrefP_ref, and every Q∈Q , we have (Q;Pref)−λHε(Q)=maxΘ∈ελ(Q,Θ),D(Q;P_ref)-λ H_ (Q)= _ _ \;A_λ(Q, ), (10) where λ(Q,Θ):=(Q;Pref)−λ(1−ε)X∼Q[ϕ(X)⊤Θϕ(X)]−λlogTr(e−Θ)A_λ(Q, ):=D(Q;P_ref)-λ(1- )\,E_X Q [φ(X) \,φ(X) ]-λ (e^- ). Therefore, the above results show that (Pλ) can be rewritten as the following two-player min-max problem (game): minQ∈maxΘ∈ελ(Q,Θ) _Q _ _ A_λ(Q, ) (i) Min-Max and Max-Min Equivalence. If P is a convex set and (⋅;Pref)D(·;P_ref) is a convex and lower semicontinuous function, the order of minimization and maximization in this game may be interchanged: minQ∈maxΘ∈ελ(Q,Θ)=maxΘ∈εminQ∈λ(Q,Θ) _Q \, _ _ \,A_λ(Q, )\>=\> _ _ \, _Q \,A_λ(Q, ) Proof. We defer the proof to the Appendix. ∎ We note that the best response in part (i) is the centered log-spectrum of the smoothed covariance, and its per-sample payoff coincides, up to an additive constant, with the entropy energy: the spectral adversary pays the generator exactly λGQελ G _Q of (8), rewarding samples along directions that Q underrepresents (Lemma 7). Therefore, the training-time adversarial payoff and the sampling-time guidance field are induced by the same first-order quantity. The proof of Proposition 1, given in Appendix D.1, relies on Klein’s matrix relative-entropy inequality and the Gibbs variational principle for matrix entropy, both of which are proved there in full. We refer to Remark 8 for further structural implications of (10), including the linearization of the entropy reward and the role of smoothing in compactifying the adversary class. 3.2 IGA for Sampling From a Pretrained Model: the Special case of KL-divergence This subsection presents the application of IGA for sampling from an available (supposedly pretrained) model. At sampling time, the pretrained model is held fixed as the reference distribution in the framework. We specifically consider the reference PθP_θ as the underlying distribution of the pretrained generator, and the Lagrangian IGA optimization problem becomes: Q⋆=argminQ∈D(Q,Pθ)−λHε(Q).Q = Q \! \; \D(Q,P_θ)-λ H_ (Q) \. (11) Here, our goal is to sample from the optimal distribution Q⋆Q . We recall that (11) separates the desired target distribution from the algorithm used to sample it: the Bregman divergence determines which departures from the base generator are costly, while the HεH_ Lagrangian penalty rewards higher spectral diversity. In our analysis, we specifically focus on the geometry resulting from choosing the discrepancy measure to be the KL-divergence. In this specific case, the optimality conditions yield an explicit density-ratio characterization of the optimal solution. Note that, in the case of KL-divergence, the sampling-based IGA aims to minimize the following objective function: F(Q):=KL(Q∥Pθ)−λHε(Q)F(Q):=KL(Q\|P_θ)-λ H_ (Q) (12) over Q∈Q , where the extended-value convention for the KL term sets F(Q)=+∞F(Q)=+∞ off the set Q≪Pθ\Q P_θ\. Proposition 2 (Sampling-time target as a self-consistent exponential tilt). Let λ≥0λ≥ 0 and ε∈(0,1) ∈(0,1). If F attains a finite minimum over Q∈:Q≪Pθ\Q :Q P_θ\, then the minimizer Q⋆Q is unique, Q⋆Q and PθP_θ are mutually absolutely continuous, and, with the total reward defined by RQ(x):=λGQε(x),R_Q(x):=λ\,G _Q(x), (13) the density ratio is the exponential tilt dQ⋆dPθ(x)=exp(RQ⋆(x))X∼Pθ[exp(RQ⋆(X))] dQ dP_θ(x)= (R_Q (x) )E_X P_θ [ (R_Q (X) ) ] (14) Proof. We defer the proof to the Appendix. ∎ Corollary 1 (Score-function relation under the exponential tilt). Under the assumptions of Proposition 2, suppose that PθP_θ and Q⋆Q admit differentiable densities pθp_θ and q⋆q , respectively. Then their score functions satisfy ∇logq⋆(x)=∇logpθ(x)+λ∇GQ⋆ε(x).∇ q (x)=∇ p_θ(x)+λ\,∇ G _Q (x). (15) Proof. The result follows directly by taking the logarithm of (14) and differentiating with respect to x, noting that the log-normalizing constant is independent of x. ∎ The target is an exponential reweighting of the pretrained law. We highlight that the reweighting is self-consistent rather than externally prescribed, since the total reward RQ⋆R_Q depends on the covariance of the unknown target itself. The multiplier λ sets the strength of the reweighting, and the uniform bound on the energy noted after (8) limits how strongly any single sample can be up- or down-weighted, while the KL term confines the redistribution of mass to the support of the base law. We note that Proposition 2 characterizes the target distribution over clean outputs and does not yet provide a sampler; also, the finite-minimum hypothesis remains to be verified. Section 5 addresses both points: under mild topological conditions the minimizer exists (Theorem 1), the tilt propagates exactly through the forward noising process to an explicit time-dependent guidance field (Theorem 2), and the field is approximated with denoised predictions at a quantified endpoint error, all without changing the pretrained score network. Unlike guidance by a fixed sample-wise reward, the tilt depends on the target law itself, through its covariance; practical sampling therefore estimates this distribution-level quantity, for example from a pilot batch. We keep the sampling target Q⋆Q notationally distinct from the training-time optimum QλtrainQ_λ^train of Section 3.3, realized by changing generator parameters. 3.3 IGA for Training Generative Models with Entropy-Regularized Objective For the training-time application of the imaginative generative modeling in IGA, we change the original divergence minimization in standard generative modeling and include the additional Lagrangian term in the objective function −λH(Q)-λ H(Q) to promote higher spectral entropy in the trained model. Mathematically, we choose the reference distribution to be the empirical distribution P^n P_n of n training samples x1,…,xnx_1,…,x_n (i.e., P^n=1n∑i=1nδxi P_n= 1n _i=1^n _x_i). Then, the optimization problem for training-time IGA will be computing the optimal solution to the spectral entropy-regularized divergence minimization problem: Qλtrain=argminQ∈gen(Q;P^n)−λH(Q).Q_λ^train\,=\, Q _gen \! \; \D(Q; P_n)-λ H(Q) \. (16) The first term specifies how divergence to the training data distribution is measured, while the second is a distribution-level regularizer: it acts jointly on generated examples and rewards coverage of feature directions that would otherwise be underrepresented. IGA therefore only augments the model’s distributional discrepancy objective and can be interpreted as a spectral entropy regularization in the divergence minimization task of training the generative model. Specifically, in the following, we focus on and apply the IGA training framework to the adversarial training of generative adversarial networks (GANs). Further discussion on application of training-time IGA to other generative modeling frameworks is deferred to the Appendix. Adversarial training and GANs. The min-max format appearing in Proposition 1 composes smoothly with objective functions that are formed in the adversarial-learning formulations of generative modeling. We note that the standard GAN [13, 30, 1] objectives measure the discrepancy through a critic (discriminator) class cD_c and real-valued link functions u,v:ℝ→ℝu,v:R : (Q;P^n)=maxD∈cX∼P^n[u(D(X))]−X∼Q[v(D(X))].D(Q; P_n)= _D _c \E_X P_n [u (D(X) ) ]-E_X Q [v (D(X) ) ] \. (17) As notable examples, Wasserstein GANs take a 11-Lipschitz critic class with u=v=idu=v=id being the identity map [1]; f-GANs choose v=f∗v=f^* for the convex conjugate of the convex f function underlying the target f-divergence, recovering the original GAN objective as a special case for JS-divergence [13, 30]. Substituting (17) and the entropy dual (9) into (16) gives an exact reformulation in which the entropy reward joins the discriminator inside a single adversary. Proposition 3 (IGA-GAN formulation as min-max optimization). Let (⋅;P^n)D(·; P_n) admit the representation (17), and let λ≥0λ≥ 0, ε∈(0,1) ∈(0,1). Then, for every class Q of distributions, we have the following minQ∈(Q;P^n)−λHε(Q) _Q \D(Q; P_n)-λ H_ (Q) \ =minQ∈max(D,Θ)∈c×εX∼P^n[u(D(X))]−X∼Q[v(D(X))] \>=\> _Q \; _(D, ) _c×T_ \; \E_X P_n [u (D(X) ) ]-E_X Q [v (D(X) ) ] (18) −λ(1−ε)X∼Q[ϕ(X)⊤Θϕ(X)]−λlogTr(e−Θ), -λ(1- )\,E_X Q [φ(X) \,φ(X) ]-λ (e^- ) \, Note that the above has a single maximization over the joint adversary (D,Θ)(D, ). For every fixed Q the joint maximization decouples across the two components, and the Θ -component is attained at the best response Θ⋆(Q) (Q) of Proposition 1. Proof. We defer the proof to the Appendix. ∎ The proof, given in Appendix D.2, relies on a structural observation: the critic and the spectral adversary enter through suprema over independent variables, and such suprema combine additively. Hence, the identity holds pointwise in Q, and neither convexity of genP_gen nor a minimax interchange is used; the interchange remains reserved for the ambient class (Remark 3, Appendix C). Algorithmically, (18) adds one adversary to standard GAN training, and this additional adversary is computationally inexpensive: while the critic D is trained by gradient steps, the spectral player requires no training at all, since its best response is the centered log-spectrum Θ⋆(Q) (Q) and can be computed from an eigendecomposition of the minibatch covariance at O(d3)O(d^3) cost per refresh. The two adversaries play complementary roles. The critic enforces the fidelity of individual samples by comparing generated examples against data, whereas Θ acts on the generated distribution as a whole and pays the generator the spectral novelty reward λGQε(x)λ G _Q(x), up to a sample-independent constant, for occupying directions that the current generated law neglects (Lemma 7). We also highlight that freezing Θ at its best response is not a heuristic: the gradients of the generator parameters through the frozen payoff are exactly the gradients of λHε(Qϑ)λ H_ (Q_ ) (Proposition 7, Appendix D.2). This envelope-type identity removes the need to differentiate through the eigendecomposition. 4 The Entropy Wall: Spectral Entropy of Real Data as the Boundary between Imitation and Imagination (a) CelebA-HQ (b) ImageNet Figure 4: The Entropy walls on CelebA-HQ and ImageNet. Real and generated entropy estimates use matched sample sizes. Both base models remain below the data wall; increasing IGA λIGA λ closes the deficit and eventually crosses into the imagination regime. As we discussed in the introduction, the entropy wall is the spectral diversity of the data itself, measured in the chosen representation. We note that the concept of the entropy wall is implied by the discussion in [10], in which the authors reveal the spectral entropy gap between the standard generative models and their target underlying data distributions. In this section, we formalize the concept and propose the term ”Entropy Wall” to highlight the spectral entropy level of the underlying real data distribution. Particularly, we highlight that the definition of entropy wall separates two qualitatively different uses of the IGA regularization framework: as long as the required spectral diversity level in IGA stays at or below what the data exhibits, increasing diversity can be read as repairing a deficiency of the learned generator; once the request exceeds it, the data distribution is no longer feasible for (Cρ), and pushing further is deliberate extrapolation beyond the data. Like every diversity statement in this paper, the wall’s location depends on the fixed pair (ϕ,k)(φ,k) and is therefore representation-relative. Definition 1 (Entropy Wall). For spectral entropy function H, the entropy wall is ρ⋆,H:=H(Pdata) _ ,H:=H(P_data) Based on the above definition, a distribution Q is below, on, or beyond the entropy wall according as H(Q)H(Q) is <<, ==, or >ρ⋆,H> _ ,H. Reading the regularization path of Theorem 4 through the wall gives it the statistical interpretation promised in the introduction. Note that we use the notation QλQ_λ for the optimal solution to the problem with Lagrangian coefficient λ. Below the wall: diversity repair. When H(Qλ)≤H(Pdata)H(Q_λ)≤ H(P_data), increasing λ moves QλQ_λ toward higher diversity, and this movement is provably safe for a base-anchored, correctly oriented Bregman objective: every below-wall point on the path is no farther from PdataP_data, in the anchoring divergence, than the base model is (Theorem 6, Appendix E). This is the precise sense in which the sub-wall path performs repair, counteracting the spectral contraction reported in modern generators [10]. It does not identify QλQ_λ with PdataP_data, nor does it guarantee that every induced semantic change recovers a genuine data mode. Beyond the wall: spectral extrapolation. When H(Qλ)>H(Pdata)H(Q_λ)>H(P_data), the same monotone increase in λ means something different: QλQ_λ is no longer estimating PdataP_data but performing representation-relative extrapolation, spreading its mass across directions of the representation more broadly than the data does. The transition is a change of statistical interpretation, not a geometric barrier: the wall can be crossed at arbitrarily small discrepancy whenever a higher-entropy direction exists in P and the discrepancy is continuous along the mixture path toward it (Proposition 9, Appendix E). Beyond the wall we claim no improved estimation of the data distribution; the regime is evaluated as controlled, representation-relative extrapolation. 5 IGA Guidance for Score-based and Diffusion Models Figure 5: Fashion design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla samples cluster around beige and gold eveningwear with familiar gown and tailored shapes. IGA adds bright color blocking, asymmetric cuts, mixed materials, and large sculptural or feathered elements. Figure 6: Architectural design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mainly produces straight glass towers with similar overall forms. IGA introduces curved shells, open frames, split tops, stacked blocks, and larger changes in color and proportion, while preserving a clear full-building view. Figure 7: Stylized underwater scenes with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mostly depicts coral reefs and schools of fish. IGA expands the scene content to divers, large creatures, vehicles, built structures, and cave-like spaces, while keeping the gouache rendering style. Figure 8: Fantasy throne design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mostly returns ornate high-backed chairs with similar carved frames. IGA introduces spiked metal forms, curved black shells, moss-covered structures, and larger changes in the seat and back, while keeping the throne centered and fully visible. Figure 9: IGA λ-sweep across SDXL and PixArt-Σ . Top: SDXL with λ∈0,4,8λ∈\0,4,8\. Bottom: PixArt-Σ with λ∈0,10,20λ∈\0,10,20\. Within each model, corresponding positions use the same initial noise seed. As λ increases, both models move from familiar dress shapes toward stronger asymmetry, larger volumes and accessories, and wider material and color choices, while staying consistent with the prompt. Here, we focus on diffusion models and apply sampling-time IGA to pre-trained score-based and diffusion models. The developments in this section largely build on the score-function characterization established in Corollary 1. We first establish existence and uniqueness of the tilted target Q⋆Q , then propagate the tilt through the forward noising process, and finally derive retraining-free approximations for score-based, DDPM, and DDIM samplers. 5.1 The target distribution and exact guidance Assumption 1 (Sampling-time setup). Given the sample space X, the representation map ϕφ is continuous, hence bounded by the unit-norm normalization. PθP_θ is the original distribution of the pretrained generator. Also, as stated in previous sections, we suppose the hyperparameters satisfy λ≥0λ≥ 0, ε∈(0,1) ∈(0,1). Under Assumption 1, we minimize the functional F of (12) over P. The KL anchor makes the problem tractable for two reasons: the effective domain Q≪Pθ\Q P_θ\ is convex, and the first variation of KL(⋅∥Pθ)KL(·\|P_θ) along a density perturbation is simply the log-density ratio. The main difficulty is that HεH_ is highly nonlinear in ΣQ _Q: although ΣQ _Q is affine in Q, the matrix logarithm couples all eigenvalues. The following lemma resolves this difficulty and justifies the role assigned to the entropy energy at its definition (8); it is the analytical core of Proposition 2 and of the existence theorem below. Lemma 1 (First variation of the smoothed entropy). Let ϕφ be bounded and measurable with ‖ϕ(x)‖2=1 φ(x) _2=1, and let GQεG _Q be the entropy energy (8). Then for every distribution Q and every finite signed measure ν with ν()=0ν(X)=0 such that Q+tνQ+tν is a probability measure for all sufficiently small t>0t>0, dtHε(Q+tν)|t=0+=∫GQε(x)ν(x). ddt\,H_ (Q+tν) |_t=0^+= G _Q(x)\,dν(x). The lemma therefore identifies the total reward RQR_Q of (13) as the first variation of the reward part λHε(Q)λ H_ (Q) of the objective (proof and further discussion in Appendix F). Theorem 1. Under Assumption 1, the following hold for the functional F of (12). (i) Existence and uniqueness. F has a unique minimizer Q⋆Q , and Q⋆Q is mutually absolutely continuous with PθP_θ. (i) Self-consistent exponential tilt. Q⋆Q satisfies the tilt characterization (14). (i) Boundedness. RQ⋆R_Q is uniformly bounded; consequently dQ⋆/dPθdQ /dP_θ is bounded above and below by positive constants, and Q⋆Q has the same PθP_θ-essential support as PθP_θ. Proof. We defer the proof to the Appendix. ∎ (14) is exactly the tilt announced in (5), with GQ⋆=GQ⋆εG_Q =G _Q . The theorem verifies the finite-minimum hypothesis of Proposition 2 rather than assuming it: the proof, given in Appendix F, establishes existence by the tightness of KL sublevel sets on a Polish space together with weak lower semicontinuity, and then inherits uniqueness, mutual absolute continuity, and the tilt formula from that proposition. What the theorem adds is that the target is well defined without any attainment hypothesis and that the reward is uniformly bounded, so the tilt redistributes mass within the PθP_θ-essential support and creates none (Remark 12). The tilt (14) concerns the clean distribution over x0x_0, but diffusion samplers generate x0x_0 as the endpoint of a denoising process that starts from noise at time T, so guidance must be injected at every noise level t. Then, we let w(x0)=exp(RQ⋆(x0))w(x_0)= (R_Q (x_0)) and Z=Pθw(X0)Z=E_P_θw(X_0), so that Q⋆(dx0)=1Zw(x0)Pθ(dx0)Q (dx_0)= 1Zw(x_0)P_θ(dx_0), and let Kt(dxt|x0)K_t(dx_t\,|\,x_0) denote the forward noising kernel with time-t marginals pt,qt⋆p_t,q_t under Pθ,Q⋆P_θ,Q . Theorem 2 (Derivation of the exact guidance field). Let ht(xt)=Pθ[w(X0)∣Xt=xt]h_t(x_t)=E_P_θ[w(X_0) X_t=x_t]. Then dqt⋆dpt=1Zht dq_t dp_t= 1Zh_t, and if pt,qt⋆p_t,q_t admit positive differentiable densities, then we have ∇logqt⋆(xt)=∇logpt(xt)+ut(xt),ut(xt):=∇xtloght(xt).∇ q_t (x_t)=∇ p_t(x_t)+u_t(x_t), u_t(x_t):= _x_t h_t(x_t). (19) Note that the function hth_t averages the clean-sample reward w over all origins that the base posterior regards as plausible for xtx_t. We emphasize that hth_t is not the clean tilt evaluated at a denoised point estimate, and this distinction is what makes the identity exact: applying the tilt before the noising process does not commute with applying it afterward. Adding ut=∇loghtu_t=∇ h_t to the base score yields a reverse process whose marginals match qt⋆q_t at every noise level (Theorem 7, Appendix F); for continuous-time samplers, the guided score can be used directly in the reverse SDE or the probability-flow ODE. One caveat accompanies this exactness. The exact reverse process must be initialized at qT⋆q_T , which is not directly samplable, whereas practical samplers initialize from pTp_T (typically Gaussian noise). The two distributions coincide only when hTh_T is constant, i.e. when the terminal noise level has erased all reward information. The resulting mismatch enters the end-to-end bound of Theorem 8 as the initialization term KL(pT∥qT⋆)KL(p_T\|q_T ), and its magnitude is quantified in Remark 13. 5.2 Practical diffusion guidance There exist three approximation items that separate the discussed theoretical framework from an implementable sampler. In the following, we make each one explicit and discuss how to address it. Plug-in guidance fields. The exact field ut=∇loghtu_t=∇ h_t requires the gradient of a conditional log-moment-generating function under the base posterior Pθ(dx0∣xt)P_θ(dx_0 x_t), which is generally intractable. What is available at every noise level is a denoiser x^0(xt,t)≈[X0∣Xt=xt] x_0(x_t,t) [X_0 X_t=x_t], and the plug-in approximation substitutes this point estimate for the posterior average. Two variants differ in how the reward is turned into a vector field: the chain-rule variant differentiates xt↦RQ⋆(x^0(xt,t))x_t R_Q ( x_0(x_t,t)) through the denoiser Jacobian Jx^0J_ x_0, while the cheaper direct-injection variant reuses the clean-space gradient as a direction in noisy-sample space: u~tchain(xt)=ωtJx^0(xt,t)⊤∇xRQ⋆(x^0(xt,t)),u~tdir(xt)=ωt∇xRQ⋆(x^0(xt,t)), u_t^chain(x_t)= _tJ_ x_0(x_t,t) _xR_Q ( x_0(x_t,t) ), u_t^dir(x_t)= _t _xR_Q ( x_0(x_t,t) ), with guidance scale ωt≥0 _t≥ 0. Both are heuristics without a general error bound: RQ⋆R_Q is nonlinear, and neither conditional expectation nor differentiation commutes with a point-mass substitution (Definition 2, Remark 14). We always report which variant is used. Estimating the self-referential reward. The reward RQ⋆R_Q depends on the unknown covariance SQ⋆εS_Q , so a practical sampler replaces it by an estimate, and how the estimate is maintained determines the statistical status of the outputs. If the covariance is frozen, i.e., computed once from a pilot batch and used to define a single estimated potential R R for all subsequent trajectories, the draws are conditionally i.i.d. from the frozen-potential law. If instead the covariance is recomputed on the fly from the batch being generated, each particle’s drift depends on the others, and the outputs form an exchangeable but non-i.i.d. interacting particle system (Remark 16). Frozen-potential estimation is therefore the setting in which IGA guidance can be described as sampling from a well-defined target distribution. Discrete sampler updates. Once a guidance field u~t u_t is chosen, it is converted to a correction on the noise prediction. In the ε -prediction parameterization, with the standard noise–score convention ∇logpt(xt)=−εθ(xt,t)/1−α¯t∇ p_t(x_t)=- _θ(x_t,t)/ 1- α_t [47], guiding the score by +u~t+ u_t corresponds to εθIGA(xt,t)=εθ(xt,t)−1−α¯tu~t(xt). _θ^IGA(x_t,t)= _θ(x_t,t)- 1- α_t\, u_t(x_t). (20) Substituting (20) into the DDPM posterior mean [15] μθ=1αt(xt−βt1−α¯tεθ) _θ= 1 _t (x_t- _t 1- α_t _θ ) and the DDIM update [45] gives μθIGA(xt,t)=μθ(xt,t)+βtαtu~t(xt),xt−1=α¯t−1x^0IGA+1−α¯t−1−σt2εθIGA+σtz, gathered _θ^IGA(x_t,t)= _θ(x_t,t)+ _t _t\, u_t(x_t),\\ x_t-1= α_t-1\, x_0^IGA+ 1- α_t-1- _t^2\, _θ^IGA+ _tz, gathered (21) with x^0IGA=(xt−1−α¯tεθIGA)/α¯t x_0^IGA=(x_t- 1- α_t\, _θ^IGA)/ α_t and σt=0 _t=0 for deterministic DDIM; the 1−α¯t 1- α_t in (20) and the βt/αt _t/ _t in (21) cancel algebraically, so the DDPM mean correction is exactly +(βt/αt)u~t+( _t/ _t) u_t. A model trained with v- or x0x_0-prediction is first converted to an equivalent εθ _θ in the standard way (e.g. εθ=α¯tvθ+1−α¯txt _θ= α_t\,v_θ+ 1- α_t\,x_t for v-prediction). These discrete updates are implementations inspired by Theorem 2, and they do not exactly sample Q⋆Q even when u~t=ut u_t=u_t, because the reverse kernels are discretized (Remark 15). The end-to-end guarantee is given by Theorem 8 in Appendix F. This theorem bounds KL(Q^∥Q⋆)KL( Q\|Q ) and TV(Q^,Q⋆)TV( Q,Q ) for the deployed continuous-time process in terms of the initialization mismatch, the score error, and the guidance error; a separate discretization term is required for the implemented sampler. 6 Numerical Evaluation (a) Population (b) Fitted DDPM (c) At the entropy wall (d) Beyond the wall Population Fitted DDPM Entropy wall Beyond wall Rare-region samples (e) Distributional repair (f) Rare-region coverage Figure 10: Repair and extrapolation on a nonlinear manifold. The fitted DDPM underrepresents the spiral endpoints. IGA restores population-level entropy and rare-region coverage near the wall and increases endpoint exploration beyond it. Panels (e,f) trace the covariance discrepancy and rare-region coverage; thin curves denote individual seeds and thick curves their mean. Throughout our numerical study, we aim to empirically address the following questions: 1. Do widely used pretrained generative models show a diversity deficit and existence of the entropy wall? 2. Does our proposed framework address this deficit, and are there values of λ that reach and go beyond the entropy wall? If so, as predicted by Theorem 6, is there an initial repair region in which the pretrained model moves closer to the data distribution it was meant to imitate? 3. Does crossing the wall produce structured, novel, and imaginative variations across different pretrained generative models, including text-conditional models? We test our theory and hypothesis on CelebA-HQ and ImageNet, and then ask what the resulting variation looks like in a large text-conditioned model. 6.1 Experimental Protocol Our experiments cover sampling-time IGA on controlled synthetic distributions and real-world image benchmarks, a training-time study on MNIST, and qualitative text-conditioned generation with SDXL. Because these settings use different models and evaluation criteria, we state only the shared experimental conventions here and introduce the setting-specific configurations in the corresponding subsections. Figure 11: IGA fills underrepresented inter-mode regions more coherently than entropy-matched noise. Top: base and IGA-guided DDIM samples as λ increases. Bottom: Gaussian-convolved samples with σ selected to match the entropy of the corresponding IGA distribution. IGA connects the gaps between modes while preserving the original modal structure; Gaussian convolution broadens each mode isotropically. Figure 12: The entropy wall separates repair from extrapolation in a controlled mixture. (A) Population, base, calibrated-wall, and beyond-wall distributions. (B) IGA repairs the component imbalance at the wall and produces more uniform weights beyond it. (C) Population KL first decreases and then increases as the path crosses the entropy wall. Entropy representation and evaluation. Across all sampling-time experiments, we set the spectral floor of (7) to ε=10−3 =10^-3, for which Lemma 3 bounds the gap between HεH_ and H0H_0 by 0.0150.015 nats, several times smaller than the smallest entropy difference we report. IGA guidance uses the smoothed entropy HεH_ , while, unless stated otherwise, we report the unsmoothed von Neumann entropy H0H_0 and its exponential, exp(H0) (H_0), corresponding to the Vendi score [11]. In all sampling-time experiments, we fix the coefficient of the IGA score correction to its theoretically prescribed value of one and vary only λ, so each point along the reported path corresponds to a different target Qλ⋆Q_λ , rather than to a different guidance strength. For the real-image benchmarks, we use the CLS-token embeddings of DINOv2 ViT-B/14 [31] and approximate an RBF kernel on these embeddings using 1024 random Fourier features [37], with the RBF bandwidth selected by the median heuristic. Because empirical spectral entropy depends on the number of samples, we compare real and generated distributions using matched sample sizes when locating the empirical entropy wall. In DINOv2 feature space, we report Fréchet distance, kernel distance, and recall. As an evaluation independent of the guidance representation, we additionally report FID and KID in Inception-v3 feature space [14, 5, 25]. Synthetic and training-time experiments use the problem-specific metrics introduced in their respective subsections. Pilot estimation. In all sampling-time experiments, we use the chain-rule plug-in guidance field. The covariance entering the IGA potential is estimated from an independent pilot batch of 2048 samples drawn from the unguided base model and then frozen during subsequent sampling. Thus, conditioned on the frozen pilot estimate, individual sampling trajectories are independent. 6.2 Numerical Application of IGA in Post-hoc Sampling-time Mode 6.2.1 Synthetic Experiments with Known Groundtruth Model We first study three controlled settings in which the population distribution is known. These experiments allow us to evaluate whether the IGA path approaches the population below the entropy wall and departs from it beyond the wall. The three settings provide complementary evidence: a nonlinear manifold illustrates rare-region repair, an entropy-matched control distinguishes IGA from isotropic noise for increasing entropy, and a finite mixture exposes the redistribution of probability mass across modes. Rare-region repair on a nonlinear manifold. Figure 10 considers a DDPM [15] trained on a one-dimensional population embedded in ℝ128R^128. The fitted model captures the dominant central portion of the manifold but substantially underrepresents its endpoints, leading to lower entropy and reduced rare-region coverage. Here, rare regions are defined as the portions outside the central 70%70\% of the normalized manifold coordinate. Increasing λ initially corrects this contraction. Near the entropy wall, IGA recovers both the population entropy and the missing endpoint mass. Panels 10(e) and 10(f) show the corresponding transition: the covariance discrepancy decreases as the path approaches the wall, while rare-region coverage increases. Beyond the wall, coverage continues to grow, but the discrepancy to the population turns upward. Thus, the same path first repairs variation lost by the fitted model and then promotes exploration beyond the population level. Structured coverage versus entropy-matched noise. The spiral experiment shows that IGA directs probability toward underrepresented regions. To determine whether this behavior could be reproduced by simply adding noise, Figure 11 compares IGA with an entropy-matched Gaussian-convolution baseline applied to a multimodal DDIM model [45]. For each IGA setting, the convolution scale σ is selected by bisection so that Qσ=PDDIM∗(0,σ2I)Q_σ=P_DDIM*N(0,σ^2I) attains the same representation-space von Neumann entropy. Despite matching entropy, the two methods distribute their additional mass differently. Gaussian convolution broadens every mode approximately isotropically, producing increasingly diffuse clouds around the original modal centers. IGA instead selectively fills the underrepresented regions between neighboring modes. As λ increases, these inter-mode regions form a coherent ring while the original modes remain visible. The entropy increase produced by IGA therefore reflects structure-aware redistribution rather than an undirected increase in noise. Population-level confirmation in a controlled mixture. Figure 12 provides a complementary view using an eight-component mixture with known population weights. The base distribution overweights its most frequent components and underrepresents the remaining modes, resulting in lower entropy than the population. Increasing λ initially corrects this imbalance: at the calibrated wall, the guided distribution approximately recovers both the population entropy and its component probabilities. Beyond the wall, the component probabilities become more uniform than those of the population. Panel C of Figure 12 makes the change in regime explicit. Along the below-wall portion of the path, the population KL decreases as IGA repairs the component imbalance. After the wall is crossed, entropy continues to increase while KL turns upward. The path therefore first approaches the population through diversity repair and subsequently departs from it through deliberate extrapolation. Together, these controlled experiments show that IGA restores underrepresented population structure below the entropy wall and enters an extrapolative regime beyond it. They further show that the increase in diversity arises from selective redistribution toward underrepresented regions rather than isotropic perturbation. We next examine whether the same progression appears in pretrained diffusion models on real-world image benchmarks. 6.2.2 Real-World Image Distribution Benchmarks Having established the repair-to-extrapolation transition in controlled settings, we next ask whether the same progression appears in pretrained diffusion models on real-world image distributions. We evaluate sampling-time IGA on unconditional CelebA-HQ and class-conditional ImageNet generation. Benchmark settings. On CelebA-HQ [22], we guide the pretrained google/ddpm-ema-celebahq-256 DDPM [15] at 256×256256× 256, using deterministic DDIM sampling [45] for 100100 steps. On ImageNet, we use the standard ImageNet-100 subset introduced by Tian et al. [48], consisting of their fixed 100-class subset of ILSVRC-2012 [40]. We guide the class-conditional facebook/DiT-XL-2-256 model [35] for 5050 DDIM steps, using classifier-free guidance [16] at scale 2.02.0. (a) Qualitative transition across the entropy wall. (b) FID–entropy phase portrait. (c) Spectral occupancy. Figure 13: IGA across the ImageNet entropy wall. (a) Matched samples along the IGA target path as λ increases. (b) FID initially decreases as entropy approaches the empirical wall and turns beyond it. (c) Cumulative spectral occupancy relative to the data, across feature directions ordered from dominant to rare. The coefficient on the IGA score correction is fixed to one, as prescribed by the sampling-time construction. We vary only the entropy multiplier λ. Each point along the reported path therefore corresponds to a different entropy-regularized target, rather than to a stronger or weaker application of the same guidance field. The DDIM sampler implements the corresponding unit-scale plug-in correction at each denoising step. Diversity deficit and wall crossing. Because empirical spectral entropy depends on sample size, Figure 4 compares real and generated distributions using matched numbers of samples. On both datasets, the base model remains below the corresponding empirical data wall throughout the evaluated sample-size range. Increasing λ progressively closes this deficit, reaches the wall at an intermediate point, and crosses it for larger values. The tested path therefore spans three interpretable regimes: a diversity-deficient base model, below-wall repair, and beyond-wall imagination. The ImageNet path across the wall. Figure 13 summarizes the progression on ImageNet. The matched samples in Figure 13(a) show the transition from the base model through below-wall repair and into beyond-wall extrapolation. The FID–entropy phase portrait in Figure 13(b) shows the corresponding distributional trend: FID initially decreases as the entropy deficit is repaired and turns after the target approaches and crosses the empirical wall. To examine how the additional entropy is obtained, we define the cumulative spectral-occupancy ratio Tλ(r)=log∑i=rdvi⊤Sλvi∑i=rdvi⊤Sdatavi,T_λ(r)= _i=r^dv_i S_λv_i _i=r^dv_i S_datav_i, where SλS_λ and SdataS_data denote the generated and data covariance matrices in DINOv2 feature space, respectively, and the data-covariance eigenvectors viv_i are ordered from dominant to rare. Negative values indicate an occupancy deficit relative to the data, whereas positive values indicate excess occupancy. Figure 13(c) shows that increasing λ progressively closes the deficit across underrepresented directions and produces excess occupancy after the wall is crossed. IGA therefore gains entropy by allocating more probability to directions that the base generator covers insufficiently. (a) CelebA-HQ: FID. (b) CelebA-HQ: KID. (c) ImageNet: FID. (d) ImageNet: KID. Figure 14: Independent Inception-v3 evaluation along the IGA target path. FID and KID are measured in Inception-v3 feature space, independently of the DINOv2 representation used to define spectral diversity and the empirical entropy wall. On both CelebA-HQ and ImageNet, the initial below-wall portion of the path improves distributional agreement with the data while diversity increases. At larger values of λ, the behavior becomes metric- and dataset-dependent, with distributional distances eventually flattening or turning as the target enters the extrapolative regime. Corresponding DINOv2-space FD and KD curves are reported in Appendix H. Table 1: Comparison with diversity-guidance methods. Distributional distances, coverage, and spectral diversity on CelebA-HQ and ImageNet. Shaded rows trace the IGA target path as λ increases. Bold indicates the lowest distributional distance or highest recall within each dataset block. Inception-v3 DINOv2 Coverage & diversity Method Vendi KID ×103↓× 10^3 FID ↓ KD ×102↓× 10^2 FD ↓ Recall ↑ H0H_0 CelebA-HQ Base 12.5 30.98 32.80 12.331 158.7 0.611 2.528 SPARKE [18] 23.0 50.314 54.77 30.294 792.5 0.479 3.134 , λ=0.5λ=0.5 13.4 29.53 31.79 12.018 152.5 0.627 2.597 , λ=1λ=1 14.3 28.74 31.02 11.876 151.0 0.649 2.657 , λ=2λ=2 15.7 27.50 30.08 12.049 156.4 0.685 2.755 , λ=3λ=3 17.2 27.06 29.66 12.616 169.8 0.713 2.843 , λ=4λ=4 18.6 26.71 29.86 13.272 192.1 0.732 2.923 ImageNet Base 7.7 3.107 9.00 4.595 118.3 0.598 2.040 CADS [41] 13.1 1.260 6.05 4.518 123.3 0.645 2.571 SPARKE [18] 7.9 2.912 7.89 4.593 115.2 0.470 2.061 , λ=0.36λ=0.36 8.5 2.317 7.51 4.572 114.3 0.636 2.141 , λ=1.2λ=1.2 10.5 0.753 5.92 4.550 123.0 0.696 2.349 , λ=2λ=2 12.2 0.617 6.37 4.546 143.5 0.727 2.498 , λ=3λ=3 13.9 1.446 8.13 4.548 170.7 0.760 2.633 Repair below the wall. Figure 14 shows a consistent initial repair regime on both benchmarks. At the beginning of the below-wall path, spectral diversity and recall increase while the reported feature-space distances decrease relative to the base model. IGA therefore recovers variation missing from the pretrained generator while improving its agreement with the data under both DINOv2 and Inception-v3 representations. This behavior is consistent with the repair result of Theorem 6. As the target approaches the empirical wall, entropy and recall continue to increase, while the different distances attain their minima at nearby but nonidentical values of λ. This is expected: Inception-v3 and DINOv2 encode different properties of image distributions and need not identify the same target as closest to the data. Imagination beyond the wall. Once λ moves the target beyond the wall, entropy and recall continue to increase, but the feature-space distances no longer decrease uniformly. Their turning points depend on the dataset, metric, and evaluation representation. This should not be interpreted as a direct measurement of declining perceptual image quality. FID, KID, FD, and KD measure distributional departure from the data in particular feature spaces. Beyond the wall, their increase instead indicates that the generated distribution is moving farther from the data reference while occupying additional feature directions. The two benchmarks therefore exhibit the same overall progression: IGA first repairs a measurable diversity deficit and then enters a different statistical regime after crossing the wall. The central result is not a single optimal value of λ, but an interpretable target path whose meaning changes from distributional repair to deliberate spectral extrapolation. Reference diversity-guidance methods. Table 1 includes CADS [41] and SPARKE [18] as reference points rather than like-for-like baselines. CADS perturbs the conditioning signal and is therefore reported only on class-conditional ImageNet, while the evaluated SPARKE configuration uses joint batch guidance and produces coupled samples. Neither method defines its operating point relative to the data entropy or distinguishes below-wall repair from beyond-wall extrapolation. In contrast, IGA traces a wall-calibrated family of target distributions and, once its potential is estimated and frozen, applies the same guidance independently to each sampling trajectory. The table therefore provides numerical context under common evaluation metrics rather than a comparison of identical objectives or guarantees. Across both benchmarks, the empirical picture is consistent. The pretrained model begins below the entropy wall; below-wall values of λ repair part of this deficit while increasing coverage and reducing distributional distances; and larger values cross the wall, where additional coverage is accompanied by a representation-dependent departure from the data distribution. IGA therefore exposes an interpretable target path from diversity repair to controlled imagination. 6.3 Numerical Application of Training-Time IGA The main empirical focus of this paper is sampling-time IGA, which can be applied to a pretrained generator without retraining. Nevertheless, the same distribution-level regularization principle also extends naturally to model training. Section 3.3 formulates training-time IGA as (16), where the model is trained to balance fidelity to the empirical data distribution with the spectral entropy of its generated distribution. The corresponding MNIST results are reported in Appendix H.2. We evaluate this training-time realization using a GAN on MNIST. Following Proposition 3, the entropy reward is added to the adversarial objective through the joint-adversary formulation in (18). For each generated minibatch, we compute the spectral adversary at its closed-form best response and hold it fixed during the generator update. By Proposition 7, this frozen payoff gives the exact gradient of the minibatch entropy at the refresh point. Implementation details. We use a convolutional GAN with a 64-dimensional latent and batch size 128, trained for 20 epochs with Adam using learning rate 2×10−42× 10^-4 and (β1,β2)=(0.5,0.999)( _1, _2)=(0.5,0.999) for both generator and discriminator. We use the non-saturating logistic generator objective and one discriminator update per generator update. The IGA representation is the unit-normalized 64-dimensional embedding of a frozen MNIST classifier, while an architecturally distinct frozen classifier with a 96-dimensional embedding is used for independent evaluation. We set ε=0.05 =0.05 and report means and standard errors over five random seeds. The baseline GAN exhibits a noticeable imbalance in generated digit frequencies despite being trained on the nearly balanced MNIST distribution. Figure 20 in Appendix H.2 shows that moderate IGA regularization redistributes probability mass toward digit classes underrepresented by the baseline generator. At the best intermediate settings, the total variation distance between the generated and empirical class distributions decreases by 37.1%37.1\%, while the Fréchet distance measured in the feature space of a separate evaluator network decreases by 40.1%40.1\%. The improvement in both metrics indicates that the effect is not limited to the class-frequency statistic. Their nonmonotone dependence on the IGA multiplier also illustrates the tradeoff between the GAN fidelity objective and the distribution-level entropy reward. 6.4 IGA Application to Prompt-Conditioned Generative Models We finally test sampling-time IGA on two text-to-image models: Stable Diffusion XL (SDXL) [36] and PixArt-Σ [6]. We use stable-diffusion-xl-base-1.0 at 768×768768× 768 and PixArt-Sigma-XL-2-1024-MS at 1024×10241024× 1024, with deterministic DDIM sampling for 5050 steps. The IGA score correction is fixed at unit scale, and only λ is varied. Figure 9 shows the resulting path, using λ∈0,4,8λ∈\0,4,8\ for SDXL and λ∈0,10,20λ∈\0,10,20\ for PixArt-Σ . Within each model, the initial noise seeds are matched across the sweep. Increasing λ leads to larger changes in garment shape, volume, material, and color while remaining consistent with the prompt. Figures 5–8 compare vanilla and IGA SDXL across fashion, architecture, underwater painting, and throne design. The base samples tend to stay near familiar forms, whereas IGA produces sculptural garments, curved and stacked towers, underwater scenes with divers and vehicles, and more varied throne structures. Across these examples, the main changes are in shape, structure, and scene composition rather than only color or texture. Additional qualitative results for PixArt-Σ are provided in Appendix B. 7 Conclusion and Discussion Generative modeling is typically formulated as distributional imitation, i.e., the ultimate goal is to generate fresh samples from the underlying distribution of real training samples. However, as shown in [10], such an approach can empirically lead to a model that generates high-quality samples while remaining systematically less diverse than the target real distribution. Our work introduces Imaginative Generative AI (IGA), a distribution-level framework that incorporates spectral entropy as an explicit and controllable diversity component of the generative modeling objective. The real data distribution’s spectral entropy establishes an Entropy Wall in the application of IGA: below this wall, IGA entropy regularization repairs diversity lost during training while remaining compatible with the diversity of the data; beyond the wall, the generated distribution intentionally attains greater representation-relative spectral diversity than the real data. We note that the IGA regularization principle can be applied to both the training of a generative model and post-hoc sampling from a pretrained model. Therefore, IGA provides a general framework for diversity regularization and imaginative generation. Beginning with improving the imitation regime, the approach first counteracts spectral-diversity deficits and encourages the recovery of variation underrepresented by the learned generator. Upon reaching the Entropy Wall, additional regularization transitions into a controlled extrapolative regime, balancing increased representation-relative diversity with closeness to the reference distribution. During sampling, once the IGA guidance potential is fixed, the resulting target enables independent and identically distributed generation, eliminating the need for an interacting batch. Our numerical results support the application of IGA for both diversity repair and imaginative data generation. Pretrained diffusion models demonstrate a measurable entropy deficit compared with matched real-data samples; moderate IGA guidance addresses this deficit, enhancing diversity and, in several cases, distributional fidelity. Stronger guidance crosses the Entropy Wall and generates structured variation beyond the data reference level, including qualitatively novel and imaginative changes in large text-to-image models. These findings indicate that diversity enhancement does not need to be treated as an architecture-specific heuristic or as an uncontrolled deviation from quality. Instead, IGA offers a general regularization framework for systematically transitioning from imitation, through diversity repair, to controlled imaginative extrapolation. The notion of imagination in IGA is intentionally representation-relative: exceeding the Entropy Wall means exceeding the spectral diversity of the data in a specified embedding space, rather than satisfying a representation-independent notion of creativity or novelty. Consequently, the choice of representation, the reference distribution, and the fidelity discrepancy remain important modeling decisions. Subject to these choices, the Entropy Wall provides an explicit and measurable boundary between improving imitation and deliberately moving beyond it, making the transition from imitation to imagination mathematically well-defined and controllable. References [1] M. Arjovsky, S. Chintala, and L. Bottou (2017) Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 70, p. 214–223. Cited by: §3.3, §3.3. [2] S. Arora, A. Risteski, and Y. Zhang (2018) Do GANs learn the distribution? some theory and empirics. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. [3] R. Askari Hemmat, M. Hall, A. Sun, C. Ross, M. Drozdzal, and A. Romero-Soriano (2025) Improving geo-diversity of generated images with contextualized Vendi score guidance. In Computer Vision – ECCV 2024, Lecture Notes in Computer Science, Vol. 15145, p. 213–229. External Links: Document Cited by: Appendix A. [4] F. Bach (2023) Information theory with kernel methods. IEEE Transactions on Information Theory 69 (2), p. 752–775. Cited by: §2.2. [5] M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton (2018) Demystifying MMD GANs. In International Conference on Learning Representations, External Links: Link Cited by: §6.1. [6] J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li (2024) PixArt-Σ : weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision (ECCV), p. 74–91. External Links: Document Cited by: §6.4. [7] G. Corso, Y. Xu, V. De Bortoli, R. Barzilay, and T. S. Jaakkola (2024) Particle guidance: non-I.I.D. diverse sampling with diffusion models. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §1. [8] M. Dombrowski, W. Zhang, S. Cechnicka, H. Reynaud, and B. Kainz (2025) Image generation diversity issues and how to tame them. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3029–3039. External Links: Document Cited by: Appendix A, Appendix A. [9] A. Elgammal, B. Liu, M. Elhoseiny, and M. Mazzone (2017) CAN: creative adversarial networks, generating “art” by learning about styles and deviating from style norms. In Proceedings of the Eighth International Conference on Computational Creativity, p. 96–103. Cited by: Appendix A. [10] F. Farnia, M. Jalali, and A. Ospanov (2026) Exposing diversity bias in deep generative models: statistical origins and correction of diversity error. arXiv preprint arXiv:2602.14682. External Links: 2602.14682 Cited by: Appendix A, Appendix C, §1, §1, §3.1, §4, §4, §7. [11] D. Friedman and A. B. Dieng (2023) The Vendi score: a diversity evaluation metric for machine learning. Transactions on Machine Learning Research. External Links: Link Cited by: Appendix A, §1, §1, §1, §2.2, §2.2, §6.1. [12] S. Ge, V. Goswami, C. L. Zitnick, and D. Parikh (2021) Creative sketch generation. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. [13] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Advances in Neural Information Processing Systems, Vol. 27. Cited by: §1, §3.3, §3.3. [14] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §6.1. [15] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Vol. 33, p. 6840–6851. Cited by: §D.2, §1, §5.2, §6.2.1, §6.2.2. [16] J. Ho and T. Salimans (2022) Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. External Links: 2207.12598 Cited by: §6.2.2. [17] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2020) The curious case of neural text degeneration. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. [18] M. Jalali, H. Lei, A. Gohari, and F. Farnia (2025) SPARKE: scalable prompt-aware diversity and novelty guidance in diffusion models via RKE score. In Advances in Neural Information Processing Systems, Vol. 38, p. 119943–119980. Cited by: Appendix A, §1, §6.2.2, Table 1, Table 1. [19] M. Jalali, C. T. Li, and F. Farnia (2023) An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems, Vol. 36, p. 9931–9943. External Links: Document Cited by: Appendix A, Appendix A, §1, §1, §1, §2.2. [20] M. Jalali, A. Ospanov, A. Gohari, and F. Farnia (2026) Conditional Vendi score: prompt-aware diversity evaluation for text-guided generative AI models. In The 29th International Conference on Artificial Intelligence and Statistics, External Links: Link Cited by: Appendix A. [21] L. Jiang, Y. Chai, M. Li, M. Liu, R. Fok, N. Dziri, Y. Tsvetkov, M. Sap, and Y. Choi (2025) Artificial hivemind: the open-ended homogeneity of language models (and beyond). In Advances in Neural Information Processing Systems, Vol. 38. Note: Datasets and Benchmarks Track Cited by: Appendix A. [22] T. Karras, T. Aila, S. Laine, and J. Lehtinen (2018) Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, External Links: Link Cited by: §6.2.2. [23] D. P. Kingma and M. Welling (2014) Auto-encoding variational Bayes. In International Conference on Learning Representations, Cited by: §D.2, §1. [24] M. Kirchhof, J. Thornton, L. Béthune, P. Ablin, E. Ndiaye, and M. Cuturi (2025) Shielded diffusion: generating novel and diverse images using sparse repellency. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 30911–30942. Cited by: Appendix A. [25] T. Kynkänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila (2019) Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §6.1. [26] J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan (2016) A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 110–119. External Links: Document Cited by: Appendix A. [27] J. Lu, R. Teehan, and M. Ren (2025) ProCreate, don’t reproduce! propulsive energy diffusion for creative generation. In Computer Vision – ECCV 2024, Lecture Notes in Computer Science, Vol. 15118, p. 397–414. External Links: Document Cited by: Appendix A. [28] Z. Miao, J. Wang, Z. Wang, Z. Yang, L. Wang, Q. Qiu, and Z. Liu (2024) Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10844–10853. Cited by: Appendix A. [29] Q. Nguyen and A. B. Dieng (2024) Quality-weighted Vendi scores and their application to diverse experimental design. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 37667–37682. External Links: Link Cited by: Appendix A. [30] S. Nowozin, B. Cseke, and R. Tomioka (2016) f-GAN: training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, Vol. 29, p. 271–279. Cited by: §3.3, §3.3. [31] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P. Huang, S. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski (2023) DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. External Links: 2304.07193 Cited by: §6.1. [32] A. Ospanov and F. Farnia (2025) Do Vendi scores converge with finite samples? truncated Vendi score for finite-sample convergence guarantees. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, Proceedings of Machine Learning Research, Vol. 286, p. 3272–3299. Cited by: Appendix A. [33] A. Ospanov, M. Jalali, and F. Farnia (2025) Scendi score: prompt-aware diversity evaluation via schur complement of CLIP embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 16927–16937. Cited by: Appendix A. [34] A. Ospanov, J. Zhang, M. Jalali, X. Cao, A. Bogdanov, and F. Farnia (2024) Towards a scalable reference-free evaluation of generative models. In Advances in Neural Information Processing Systems, Vol. 37, p. 120892–120927. External Links: Document Cited by: Appendix A, §1. [35] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4195–4205. Cited by: §6.2.2. [36] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2024) SDXL: improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, External Links: Link Cited by: §6.4. [37] A. Rahimi and B. Recht (2007) Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, Vol. 20, p. 1177–1184. Cited by: §6.1. [38] D. J. Rezende, S. Mohamed, and D. Wierstra (2014) Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 32, p. 1278–1286. Cited by: §D.2. [39] E. Richardson, K. Goldberg, Y. Alaluf, and D. Cohen-Or (2024) ConceptLab: creative concept generation using VLM-guided diffusion prior constraints. ACM Transactions on Graphics 43 (3), p. 1–14. External Links: Document Cited by: Appendix A. [40] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei (2015) ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115 (3), p. 211–252. External Links: Document Cited by: §6.2.2. [41] S. Sadat, J. Buhmann, D. Bradley, O. Hilliges, and R. M. Weber (2024) CADS: unleashing the diversity of diffusion models through condition-annealed sampling. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §1, §6.2.2, Table 1. [42] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen (2016) Improved techniques for training GANs. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: Appendix A. [43] V. Sehwag, C. Hazirbas, A. Gordo, F. Ozgenel, and C. Canton Ferrer (2022) Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11492–11501. Cited by: Appendix A. [44] G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein (2023) Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6048–6058. Cited by: Appendix A. [45] J. Song, C. Meng, and S. Ermon (2021) Denoising diffusion implicit models. In International Conference on Learning Representations, External Links: Link Cited by: §5.2, §6.2.1, §6.2.2. [46] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, Vol. 32, p. 11895–11907. Cited by: §1. [47] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, External Links: Link Cited by: §1, §5.2. [48] Y. Tian, D. Krishnan, and P. Isola (2020) Contrastive multiview coding. In European Conference on Computer Vision, Cited by: §6.2.2. [49] A. Yadav, A. Garg, Ta Duc Huy, and L. Liu (2026) STRIDE: training-free diversity guidance via PCA-directed feature perturbation in single-step diffusion models. arXiv preprint arXiv:2605.11494. External Links: 2605.11494 Cited by: Appendix A. [50] J. Zhang, C. T. Li, and F. Farnia (2024) An interpretable evaluation of entropy-based novelty of generative models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 59148–59172. Cited by: Appendix A. Appendix A Related Work Diversity in generative models. Diversity loss is a persistent problem across generative modeling. In generative adversarial networks (GANs), mode collapse and limited effective support motivated minibatch discrimination and support-size diagnostics [42, 2]. Language models likewise tend toward generic, repetitive, or homogeneous outputs, motivating diversity-aware objectives and decoding strategies [26, 17, 21]. More recently, Rényi Kernel Entropy (RKE) evaluations have shown that modern generators can produce high-quality samples while still missing modes [19]. Related deficits have been documented in image diffusion models, together with a systematic gap between real and generated diversity for which finite-sample entropy underestimation is one statistical source [8, 10]. These findings motivate IGA’s premise: diversity should be specified as a property of the target distribution rather than left as a by-product of distribution fitting. Measuring novelty and diversity. Reference-free measures assess variation within a distribution through similarities among its samples. The Vendi Score uses the von Neumann entropy of a normalized kernel matrix [11], while RKE provides a tractable order-two counterpart with mode-count interpretations [19]. Kernel-based Entropic Novelty compares which modes are more strongly expressed than in a reference distribution [50]. Conditional Vendi and Scendi extend diversity evaluation to prompt-conditioned generators [20, 33], while scalable and truncated variants address computational cost and finite-sample estimation [34, 32]. A complementary line folds sample quality directly into the diversity score itself, yielding quality-weighted Vendi scores [29]. IGA moves this spectral perspective from post-hoc evaluation into the generative objective itself. Population-data entropy then defines an entropy wall that separates recovery of lost diversity from deliberate extrapolation beyond the data. Promoting novelty and diversity. Existing interventions typically specialize either training or generation. At training time, diversity has been promoted through reinforcement learning with explicit diversity rewards [28] and through diversity-aware diffusion modules [8]. At inference, CADS anneals noise in the conditioning signal [41]; c-VSG and SPARKE guide generation using contextualized Vendi and conditional RKE, respectively [3, 18]; and STRIDE perturbs intermediate features in distilled one- and few-step generators [49]. Particle Guidance instead evolves an interacting set under a pairwise diversity potential and is explicitly non-i.i.d. [7], while SPELL repels trajectories from protected, concurrent, or previously generated images [24]. IGA instead provides a single distribution-level regularizer with both training- and sampling-time realizations. It does not define diversity through the active batch or generation history: during sampling, its potential is estimated beforehand, held fixed, and applied independently to each initialized trajectory, yielding i.i.d. samples from the approximated IGA target. The entropy wall additionally identifies whether regularization repairs diversity lost during learning or intentionally moves beyond the population data. IGA thus unifies diversity control across end-to-end learning and post-hoc sampling while providing a principled transition from imitation to extrapolation. Creative generation. Creative generation has been pursued through deviation from learned styles, novel composition, rare-region sampling, and repulsion from exemplars. Creative Adversarial Networks depart from established artistic styles [9]; DoodlerGAN recombines object parts into unseen sketches [12]; and ConceptLab searches for new category members [39]. For diffusion models, evidence of training-data replication sharpens the distinction between creativity and reproduction [44]. Low-density sampling explores rare regions of the learned distribution [43], whereas ProCreate pushes generations away from reference images [27]. IGA instead gives creativity a distributional interpretation: crossing the entropy wall produces a target whose representation-relative spectral diversity exceeds that of the population data, while the fidelity term controls departure from the reference. The same definition governs both training and sampling. Appendix B Additional Qualitative Results We provide additional qualitative comparisons on PixArt-Σ . Unless stated otherwise, IGA uses λ=20λ=20. Across prompts, the vanilla model often concentrates on a narrow set of familiar forms or compositions, while IGA produces broader structural and semantic variation. This is particularly visible in the underwater example, where vanilla PixArt repeatedly generates very similar coral-reef scenes, whereas IGA explores substantially different subjects and layouts while retaining the requested rendering style. Figure 15: Fantasy throne design with PixArt-Σ . Vanilla PixArt (left) and IGA with λ=20λ=20 (right). IGA produces broader variation in silhouette and structure. Figure 16: Underwater scenes with PixArt-Σ . Vanilla PixArt (left) largely collapses to the same coral-reef scene across seeds. IGA with λ=40λ=40 (right) produces substantially broader variation in subjects and composition while preserving the gouache style. Figure 17: Architectural design with PixArt-Σ . Vanilla PixArt (left) and IGA with λ=20λ=20 (right). IGA introduces stronger geometric and structural variation while retaining the skyscraper concept. Figure 18: Fashion design with PixArt-Σ . Vanilla PixArt (left) and IGA with λ=20λ=20 (right). IGA produces broader variation in garment silhouette, volume, material, and color, while remaining consistent with the wearable haute-couture prompt. Appendix C Extended Preliminaries and Conventions We begin by stating two conventions that are used throughout the paper. We would like to further clarify that some of the theoretical lemmas and basic statements are also discussed in [10]. Remark 2. For any discrepancy D we write (Q;Pref)D(Q;P_ref) with the optimized distribution Q first and the reference PrefP_ref second. For symmetric discrepancies this is cosmetic. However, for KL and general Bregman divergences the ordering cannot be generally swapped. In our notation, the KL anchor is always KL(Q∥Pref)KL(Q\|P_ref), and a Bregman anchor is always DΦ(Q,Pref)D_ (Q,P_ref). A data-first object such as KL(P^n∥Q)KL( P_n\|Q) or DΦ(P^n,Q)D_ ( P_n,Q) is a different problem and does not inherit the guarantees below (see also Remark 10 and, for the maximum-likelihood setting where the data-first orientation is forced, Proposition 8). Remark 3. P is assumed to be the convex ambient class of probability measures on X, used for the convex-analytic theory (concavity of entropy, convex duality, Bregman projection). genP_gen is the possibly nonconvex class realizable by a fixed architecture, used for training. Convexity, strong duality, and Pythagorean statements are proved on P and never silently transferred to genP_gen; a trained or guided model reaches the ambient optimum only in an approximation-theoretic sense. Assumption 2 (Finite-dimensional embedding). Unless stated otherwise, ϕ:→ℝdφ:X ^d with ‖ϕ(x)‖2=1 φ(x) _2=1 is measurable, and ΣQ∈ℝd×d _Q ^d× d with ΣQ⪰0 _Q 0 and Tr(ΣQ)=1Tr( _Q)=1. When weak continuity of Q↦ΣQ _Q is invoked, we further assume X is Polish and ϕφ is bounded and continuous. Remark 4 (Use of entropy symbol H). When a result holds for either entropy functional we let H denote a fixed but arbitrary choice from H0,Hε\H_0,H_ \ (as in (Cρ), (Pλ), Theorem 4, and Definition 1); H is never used to mix the two within a single statement. When differentiability of the entropy is needed we specialize to the smoothed HεH_ (Section 5), and when defining the population wall in its main statistical interpretation we use the unsmoothed H0H_0 (Section 4). Lemma 3 controls the gap between the two. Lemma 2 (Concavity of H0H_0 and HεH_ in Q). If Q↦ΣQ _Q is affine, then Q↦H0(Q)Q H_0(Q) and Q↦Hε(Q)Q H_ (Q) are concave. Proof. Note that S↦−Tr(SlogS)S -Tr(S S) is a concave functional of density matrices (unit-trace PSD matrices), and a concave function composed with an affine map is concave. This proves concavity of H0H_0. For HεH_ , the map Q↦SQε=(1−ε)ΣQ+ε1dIdQ S_Q =(1- ) _Q+ 1dI_d is also affine in Q, and therefore the same argument applies. ∎ Lemma 3 (Bounding Smoothing Gap of Spectral Entropy). Let d≥2d≥ 2, let Σ be a d×d× d density matrix, and set S=(1−ε)Σ+ε1dIdS=(1- ) + 1dI_d for ε∈[0,1] ∈[0,1]. Writing τ=ε(1−1d)τ= (1- 1d) and h2(t)=−tlogt−(1−t)log(1−t)h_2(t)=-t t-(1-t) (1-t), |(S)−(Σ)|≤τlog(d−1)+h2(τ). | H(S)- H( ) |≤τ (d-1)+h_2(τ). Since Hε(Q)=(SQε)H_ (Q)= H(S_Q ) and H0(Q)=(ΣQ)H_0(Q)= H( _Q), this implies the following for every Q: |Hε(Q)−H0(Q)|≤τlog(d−1)+h2(τ). |H_ (Q)-H_0(Q) |≤τ (d-1)+h_2(τ). In particular, |ρ⋆,Hε−ρ⋆|| _ ,H_ - _ | admits the same bound. The bound is achieved when Σ is rank one: then 12‖S−Σ‖1=τ 12 S- _1=τ exactly, S has eigenvalues 1−τ,τd−1,…,τd−11-τ,\ τd-1,…, τd-1, and |(S)−(Σ)|=τlog(d−1)+h2(τ) | H(S)- H( ) |=τ (d-1)+h_2(τ). Proof. First, we bound the trace distance between S and Σ . Since S−Σ=ε(1dId−Σ)S- = ( 1dI_d- ), we have 12‖S−Σ‖1=ε⋅12‖1dId−Σ‖1 12 S- _1= · 12 1dI_d- _1. If Σ has eigenvalues pip_i, then 12‖1dId−Σ‖1=12∑i|pi−1d| 12 1dI_d- _1= 12 _i|p_i- 1d|, which over the probability simplex is maximized at a vertex p=ejp=e_j, giving 1−1d1- 1d. Hence 12‖S−Σ‖1≤ε(1−1d)=τ. 12 S- _1≤ (1- 1d)=τ. Next, we apply the Fannes–Audenaert inequality, showing that for d×d× d density matrices A,BA,B with 12‖A−B‖1≤t≤1−1d 12 A-B _1≤ t≤ 1- 1d, |(A)−(B)|≤tlog(d−1)+h2(t).| H(A)- H(B)|≤ t (d-1)+h_2(t). The right-hand side is non-decreasing in t on [0,1−1d][0,1- 1d]: its derivative log(d−1)+log1−t (d-1)+ 1-tt is non-negative there, vanishing only at t=1−1dt=1- 1d. Applying the inequality at t=12‖S−Σ‖1≤τ≤1−1dt= 12 S- _1≤τ≤ 1- 1d (here d≥2d≥ 2 ensures log(d−1)≥0 (d-1)≥ 0) gives the claim with A=SA=S and B=ΣB= . Finally, the consequence for Hε,H0H_ ,H_0 follows by the stated identities, and the wall bound follows by taking Q=PdataQ=P_data. For rank-one Σ=vv⊤ =v , the eigenvalues of 1dId−Σ 1dI_d- are 1d−1 1d-1 (once) and 1d 1d (with multiplicity d−1d-1), so 12‖S−Σ‖1=τ 12 S- _1=τ exactly; the spectrum of S is then (1−τ,τd−1,…,τd−1) (1-τ,\ τd-1,…, τd-1 ), whence (Σ)=0 H( )=0 and (S)=τlog(d−1)+h2(τ) H(S)=τ (d-1)+h_2(τ), so equality holds. ∎ The last ingredient is an elementary identity for exponential tilts. For a fixed bounded reward G, it identifies the minimizer of the KL-anchored linear objective in closed form; the sampling-time tilt of Theorem 1 is its self-consistent analogue, in which G is the entropy energy evaluated at the optimum itself. Lemma 4 (Elementary Gibbs identity). Let P be a probability law and G measurable with ZG=P[eG]<∞Z_G=E_P[e^G]<∞; define PG(dx)=ZG−1eG(x)P(dx)P^G(dx)=Z_G^-1e^G(x)P(dx). For every Q≪PQ P, KL(Q∥P)−Q[G]=KL(Q∥PG)−logZG,KL(Q\|P)-E_Q[G]=KL(Q\|P^G)- Z_G, (22) so PGP^G is the unique minimizer over Q≪P\Q P\ of the left-hand side whenever it is finite. Proof. On Q≪P\Q P\ we have log(dQ/dPG)=log(dQ/dP)−G+logZG (dQ/dP^G)= (dQ/dP)-G+ Z_G. Integrating against Q gives KL(Q∥PG)=KL(Q∥P)−Q[G]+logZG,KL(Q\|P^G)=KL(Q\|P)-E_Q[G]+ Z_G, which is (22). The left-hand side equals KL(Q∥PG)−logZGKL(Q\|P^G)- Z_G, minimized (over Q≪PQ P, equivalently Q≪PGQ P^G since the two are equivalent) uniquely at Q=PGQ=P^G, where KL=0KL=0. ∎ Appendix D The Constrained–Penalized Correspondence, the Spectral Game, and Training This appendix proves the results of Section 3. We first state and prove the constrained–penalized correspondence invoked in Section 3.1, together with the monotone regularization path; Appendix D.1 then proves the spectral min–max representation, and Appendix D.2 the training-time instantiations of Section 3.3. Theorem 3 (Constrained–penalized correspondence). Suppose that: (i) P is a nonempty compact convex subset of a locally convex Hausdorff space of finite signed measures; (i) Q↦(Q;Pref)Q (Q;P_ref) is proper, convex, and lower semicontinuous on P; (i) Q↦ΣQ _Q is affine and continuous, so that H∈H0,HεH∈\H_0,H_ \ is concave (Lemma 2) and upper semicontinuous; (iv) (Slater condition) there exists Q¯∈ Q with (Q¯;Pref)<∞D( Q;P_ref)<∞ and H(Q¯)>ρH( Q)>ρ. Then the following hold. (a) Attainment. The feasible set ∩H≥ρP∩\H≥ρ\ is nonempty and compact, and both the constrained minimum in (Cρ) and the inner minimum defining FλF_λ are attained. (b) Strong duality. minQ∈H(Q)≥ρ(Q;Pref)=maxλ≥0minQ∈(Q;Pref)−λH(Q)+λρ. _ subarraycQ \\ H(Q)≥ρ subarrayD(Q;P_ref)\;=\; _λ≥ 0\; \ _Q \D(Q;P_ref)-λ H(Q) \+λρ \. (23) (c) Optimal multiplier. There exists λ⋆≥0λ ≥ 0 such that every solution Q⋆Q of (Cρ) minimizes Fλ⋆F_λ and satisfies the complementary-slackness identity λ⋆(ρ−H(Q⋆))=0λ (ρ-H(Q ) )=0. Theorem 3 justifies replacing (Cρ) by (Pλ) at the specific multiplier λ⋆λ dual to ρ; it does not claim that every λ≥0λ≥ 0 corresponds to a user-chosen target level. The proof proceeds through the value function of the constrained problem, after recording an unconditional min–max identity (Proposition 4). Proposition 4 (Exact primal min–max identity). For any feasible set Q and arbitrary functionals J,HJ,H, infQ∈H(Q)≥ρJ(Q)=infQ∈supλ≥0J(Q)+λ(ρ−H(Q)). _ subarraycQ \\ H(Q)≥ρ subarrayJ(Q)= _Q _λ≥ 0 \J(Q)+λ(ρ-H(Q)) \. Proof. For fixed Q, supλ≥0J(Q)+λ(ρ−H(Q)) _λ≥ 0\J(Q)+λ(ρ-H(Q))\ equals J(Q)J(Q) if H(Q)≥ρH(Q)≥ρ (the coefficient of λ is nonpositive, so the supremum is at λ=0λ=0) and +∞+∞ if H(Q)<ρH(Q)<ρ (the coefficient is positive, so the expression diverges as λ→∞λ→∞). Taking the infimum over Q∈Q retains only feasible Q and reproduces the constrained value. ∎ Proof of Theorem 3. By hypotheses (i)–(i), J(Q)=(Q;Pref)J(Q)=D(Q;P_ref) is proper, convex, l.s.c. on the convex set P, and H∈H0,HεH∈\H_0,H_ \ is concave (Lemma 2) and u.s.c., so the feasible set ∩H≥ρP∩\H≥ρ\ is convex and closed. First, we verify feasibility and attainment. By Slater (iv) the feasible set contains Q¯ Q, hence is nonempty; it is a closed subset of the compact P (i), hence compact. A l.s.c. function attains its minimum on a nonempty compact set, so the constrained minimum in (Cρ) is attained; likewise, for each λ≥0λ≥ 0, Q↦J(Q)−λH(Q)Q J(Q)-λ H(Q) is l.s.c. on the compact P and attains its minimum, so FλF_λ has a minimizer and the displayed min ’s in (23) are justified. The remainder of the proof runs through the value function of the constrained problem, v(r)=infQ∈J(Q):H(Q)≥r,v(r)= _Q \J(Q):H(Q)≥ r \, with v(r)=+∞v(r)=+∞ if no feasible Q exists; by the previous paragraph, v(ρ)v(ρ) is finite and attained. Note also that J, being l.s.c. on the compact P, is bounded below on P, so v(r)≥infJ>−∞v(r)≥ _PJ>-∞ for every r. Next, we establish the two structural properties of v. The value function is nondecreasing: if r1≤r2r_1≤ r_2 then H≥r2⊆H≥r1\H≥ r_2\ \H≥ r_1\, so the infimum over the smaller set is at least as large, i.e. v(r1)≤v(r2)v(r_1)≤ v(r_2). The value function is also convex. To see this, fix r1,r2∈ℝr_1,r_2 , θ∈[0,1]θ∈[0,1], and η>0η>0, and choose feasible QiQ_i (that is, H(Qi)≥riH(Q_i)≥ r_i) with J(Qi)≤v(ri)+ηJ(Q_i)≤ v(r_i)+η. The mixture Qθ=θQ1+(1−θ)Q2∈Q_θ=θ Q_1+(1-θ)Q_2 then satisfies H(Qθ)≥θr1+(1−θ)r2H(Q_θ)≥θ r_1+(1-θ)r_2 by concavity of H, and, by convexity of J, J(Qθ)≤θJ(Q1)+(1−θ)J(Q2)≤θv(r1)+(1−θ)v(r2)+η.J(Q_θ)≤θ J(Q_1)+(1-θ)J(Q_2)≤θ v(r_1)+(1-θ)v(r_2)+η. Hence v(θr1+(1−θ)r2)≤θv(r1)+(1−θ)v(r2)+ηv(θ r_1+(1-θ)r_2)≤θ v(r_1)+(1-θ)v(r_2)+η, and letting η↓0η 0 gives convexity. Then, we show that v is subdifferentiable at the target level ρ. By the Slater condition there is Q¯∈ Q with J(Q¯)<∞J( Q)<∞ and H(Q¯)>ρH( Q)>ρ; hence v(r)≤J(Q¯)<∞v(r)≤ J( Q)<∞ for all r≤H(Q¯)r≤ H( Q). Combined with the lower bound above, v is finite on (−∞,H(Q¯)](-∞,H( Q)], an interval whose interior contains ρ. A finite convex function on an open interval is subdifferentiable at every interior point; pick λ⋆∈∂v(ρ)λ ∈∂ v(ρ). Since v is nondecreasing, λ⋆≥0λ ≥ 0. With the multiplier λ⋆λ in hand, we can prove strong duality. The subgradient inequality gives, for every Q∈Q , J(Q)≥v(H(Q))≥v(ρ)+λ⋆(H(Q)−ρ),J(Q)≥ v(H(Q))≥ v(ρ)+λ (H(Q)-ρ ), hence J(Q)−λ⋆H(Q)≥v(ρ)−λ⋆ρJ(Q)-λ H(Q)≥ v(ρ)-λ ρ; taking the infimum over Q∈Q yields infQ∈J−λ⋆H+λ⋆ρ≥v(ρ) _Q \J-λ H\+λ ρ≥ v(ρ). Conversely, weak duality holds: for any λ≥0λ≥ 0 and any feasible Q (that is, H(Q)≥ρH(Q)≥ρ), J(Q)≥J(Q)−λ(H(Q)−ρ)≥infQ′∈J−λH+λρ,J(Q)≥ J(Q)-λ (H(Q)-ρ )≥ _Q \J-λ H\+λρ, and taking the infimum over feasible Q gives v(ρ)≥supλ≥0infQ′J−λH+λρv(ρ)≥ _λ≥ 0\ _Q \J-λ H\+λρ\. The two inequalities together yield (23), with the outer supremum attained at λ⋆λ . Finally, we establish complementary slackness. Let Q⋆Q solve (Cρ). Feasibility gives H(Q⋆)≥ρH(Q )≥ρ, and by strong duality J(Q⋆)=v(ρ)=infQJ−λ⋆H+λ⋆ρ.J(Q )=v(ρ)= _Q\J-λ H\+λ ρ. On the one hand, J(Q⋆)−λ⋆H(Q⋆)≥infQJ−λ⋆H=v(ρ)−λ⋆ρJ(Q )-λ H(Q )≥ _Q\J-λ H\=v(ρ)-λ ρ. On the other hand, feasibility and λ⋆≥0λ ≥ 0 give J(Q⋆)−λ⋆H(Q⋆)≤J(Q⋆)−λ⋆ρ=v(ρ)−λ⋆ρJ(Q )-λ H(Q )≤ J(Q )-λ ρ=v(ρ)-λ ρ. The two bounds match, forcing λ⋆(H(Q⋆)−ρ)=0λ (H(Q )-ρ )=0 and J(Q⋆)−λ⋆H(Q⋆)=infQJ−λ⋆HJ(Q )-λ H(Q )= _Q\J-λ H\, i.e. Q⋆Q minimizes Fλ⋆F_λ . ∎ Remark 5 (Which hypotheses do what). The hypotheses of Theorem 3 play three separable roles. Duality. The value-function argument shows that strong duality and the existence of an optimal multiplier λ⋆∈∂v(ρ)λ ∈∂ v(ρ) require only convexity of P, convex l.s.c. J, concave H, the Slater condition (iv), and the value function v being proper and finite near ρ, i.e. v(ρ)>−∞v(ρ)>-∞ (equivalently, J bounded below on the feasible set), in addition to v(ρ)<∞v(ρ)<∞ from Slater. Convexity and Slater alone do not guarantee v(ρ)>−∞v(ρ)>-∞: if J is unbounded below on P then v≡−∞v≡-∞, no finite subgradient exists, and the duality statement is vacuous. In the compact setting of Theorem 3 this cannot happen, because an l.s.c. J on the compact P is bounded below; the properness caveat matters only in the noncompact variant. Attainment. The compactness in (i), with l.s.c. J and u.s.c. H, is otherwise used only to guarantee attainment of the constrained minimum, of the inner minima defining FλF_λ, and hence of the displayed min/max / in (23). Noncompact classes. In settings where Q≪Pθ\Q P_θ\ is convex but not compact (Section 5), existence and attainment are instead obtained by the direct method of Theorem 1, where J=KL(⋅∥Pθ)≥0J=KL(·\|P_θ)≥ 0 is automatically bounded below. Proposition 5 (Saddle representation). Under the hypotheses of Theorem 3, with Q⋆Q a solution of (Cρ) and λ⋆λ the optimal multiplier, (Q⋆,λ⋆)(Q ,λ ) is a saddle point of ℒ(Q,λ)=J(Q)+λ(ρ−H(Q))L(Q,λ)=J(Q)+λ(ρ-H(Q)) on ×[0,∞)P×[0,∞): ℒ(Q⋆,λ)≤ℒ(Q⋆,λ⋆)≤ℒ(Q,λ⋆)∀Q∈,λ≥0,L(Q ,λ) (Q ,λ ) (Q,λ ) ∀ Q ,\ λ≥ 0, and consequently infQ∈supλ≥0ℒ(Q,λ)=supλ≥0infQ∈ℒ(Q,λ) _Q _λ≥ 0L(Q,λ)= _λ≥ 0 _Q L(Q,λ). Proof. Since Q⋆Q minimizes Fλ⋆=J−λ⋆HF_λ =J-λ H over P (Theorem 3) and ℒ(⋅,λ⋆)=Fλ⋆(⋅)+λ⋆ρL(·,λ )=F_λ (·)+λ ρ, the right inequality ℒ(Q⋆,λ⋆)≤ℒ(Q,λ⋆)L(Q ,λ ) (Q,λ ) holds for all Q∈Q . For the left inequality, ℒ(Q⋆,λ)=J(Q⋆)+λ(ρ−H(Q⋆))L(Q ,λ)=J(Q )+λ(ρ-H(Q )) is nonincreasing in λ≥0λ≥ 0 because ρ−H(Q⋆)≤0ρ-H(Q )≤ 0; together with λ⋆(ρ−H(Q⋆))=0λ (ρ-H(Q ))=0 this gives ℒ(Q⋆,λ)≤ℒ(Q⋆,λ⋆)L(Q ,λ) (Q ,λ ) for all λ≥0λ≥ 0. The equality of the two mixed extrema is the standard consequence of a saddle point. ∎ Even without convexity, the multiplier acts as a monotone control on global minimizers; the following statement, summarized in Section 3.1, applies both to the ambient problem and to a nonconvex generator family. Theorem 4 (Monotone regularization path). Let ∈,genQ∈\P,P_gen\, write J(Q)=(Q;Pref)J(Q)=D(Q;P_ref), and suppose that a minimizer Qλ∈argminQ∈Fλ(Q)Q_λ \,min_Q F_λ(Q) exists for every λ≥0λ≥ 0. Then, for any 0≤λ1<λ20≤ _1< _2 and any choices of minimizers Qλ1Q_ _1 and Qλ2Q_ _2, H(Qλ2)≥H(Qλ1),J(Qλ2)≥J(Qλ1).H(Q_ _2)≥ H(Q_ _1), J(Q_ _2)≥ J(Q_ _1). Proof of Theorem 4. Let 0≤λ1<λ20≤ _1< _2 and let Qλ1,Qλ2Q_ _1,Q_ _2 be any minimizers of Fλ1,Fλ2F_ _1,F_ _2 over Q. Abbreviate Hi=H(Qλi)H_i=H(Q_ _i), Ji=J(Qλi)J_i=J(Q_ _i). Optimality of Qλ1Q_ _1 at λ1 _1 and of Qλ2Q_ _2 at λ2 _2 gives J1−λ1H1≤J2−λ1H2,J2−λ2H2≤J1−λ2H1.J_1- _1H_1≤ J_2- _1H_2, J_2- _2H_2≤ J_1- _2H_1. Adding these two inequalities cancels J1,J2J_1,J_2 and yields (λ2−λ1)(H2−H1)≥0( _2- _1)(H_2-H_1)≥ 0, so H2≥H1H_2≥ H_1. Substituting into the first inequality, rearranged as J1−J2≤λ1(H1−H2)≤0J_1-J_2≤ _1(H_1-H_2)≤ 0, gives J2≥J1J_2≥ J_1. No convexity, differentiability, uniqueness, or path continuity is used; the argument is valid for any selection of minimizers, so the ordering holds even when minimizers are nonunique. ∎ Proposition 6 (Every penalized optimizer is a constrained optimizer at its attained level). Under the hypotheses of Theorem 4, set ρλ:=H(Qλ) _λ:=H(Q_λ). Then Qλ∈argminQ∈J(Q):H(Q)≥ρλQ_λ∈ Q \! \> \J(Q):H(Q)≥ _λ \. Proof. Take any Q∈Q with H(Q)≥ρλH(Q)≥ _λ. Penalized optimality gives J(Qλ)−λH(Qλ)≤J(Q)−λH(Q)J(Q_λ)-λ H(Q_λ)≤ J(Q)-λ H(Q), so J(Qλ)≤J(Q)+λ(H(Qλ)−H(Q))=J(Q)−λ(H(Q)−ρλ)≤J(Q),J(Q_λ)≤ J(Q)+λ(H(Q_λ)-H(Q))=J(Q)-λ(H(Q)- _λ)≤ J(Q), using λ≥0λ≥ 0 and H(Q)≥ρλH(Q)≥ _λ. ∎ Remark 6 (Constrained and penalized problems are not interchangeable in general). Outside the convex setting the two problems need not share solutions for a prescribed λ or ρ. Proposition 6 matches each QλQ_λ to its own attained level ρλ _λ, while Theorem 3 recovers a prescribed level ρ only under its convexity and Slater hypotheses. On a nonconvex genP_gen the saddle representation (Proposition 5) can fail with a positive duality gap; we therefore do not claim per-λ constrained–penalized equivalence for genP_gen. D.1 The spectral min–max representation: proofs This appendix proves Proposition 1 and the best-response identity quoted in Sections 3.1 and 3.3. The main tool is the Gibbs variational principle for matrix entropy, which we derive from Klein’s inequality; both results are proved in full. Lemma 5 (Klein’s inequality for matrix relative entropy). Let S,TS,T be d×d× d density matrices with T≻0T 0. Then Tr(SlogS−SlogT)≥Tr(S)−Tr(T)=0,Tr (S S-S T ) (S)-Tr(T)=0, with equality if and only if S=TS=T. Proof. Write spectral decompositions S=∑iαiuiui⊤S= _i _iu_iu_i and T=∑jβjvjvj⊤T= _j _jv_jv_j with orthonormal bases (ui),(vj)(u_i),(v_j), eigenvalues αi≥0 _i≥ 0, βj>0 _j>0, and set cij:=(ui⊤vj)2c_ij:=(u_i v_j)^2. The matrix (cij)(c_ij) is doubly stochastic: ∑jcij=‖ui‖2=1 _jc_ij= u_i ^2=1 and ∑icij=‖vj‖2=1 _ic_ij= v_j ^2=1, since each basis is orthonormal. Expanding the traces in these bases, Tr(SlogS)=∑iαilogαi=∑i,jcijαilogαi,Tr(SlogT)=∑i,jcijαilogβj,Tr(S S)= _i _i _i= _i,jc_ij\, _i _i, (S T)= _i,jc_ij\, _i _j, using ∑jcij=1 _jc_ij=1 for the first identity and ui⊤(logT)ui=∑jcijlogβju_i ( T)u_i= _jc_ij _j for the second. The scalar inequality xlogx−xlogy≥x−yx x-x y≥ x-y, valid for x≥0x≥ 0, y>0y>0 (with 0log0=00 0=0; it is the tangent-line inequality for the convex function x↦xlogx x x at y), holds with equality if and only if x=yx=y: for x>0x>0 this is strict convexity, and at x=0x=0 the inequality reads 0≥−y0≥-y, strict since y>0y>0. Applying it termwise, Tr(SlogS−SlogT)=∑i,jcij(αilogαi−αilogβj)≥∑i,jcij(αi−βj)=∑iαi−∑jβj=0,Tr (S S-S T )= _i,jc_ij ( _i _i- _i _j )≥ _i,jc_ij( _i- _j)= _i _i- _j _j=0, where the last step again uses double stochasticity. If equality holds, then every pair (i,j)(i,j) with cij>0c_ij>0 satisfies αi=βj _i= _j. Fix j and expand vj=∑i(ui⊤vj)uiv_j= _i(u_i v_j)u_i; then Svj=∑iαi(ui⊤vj)ui=∑iβj(ui⊤vj)ui=βjvj,Sv_j= _i _i(u_i v_j)\,u_i= _i _j(u_i v_j)\,u_i= _jv_j, since every index i contributing a nonzero coefficient has cij>0c_ij>0, hence αi=βj _i= _j. Thus S acts as βj _j on each vjv_j, so S=∑jβjvjvj⊤=TS= _j _jv_jv_j =T. Conversely S=TS=T gives equality trivially. ∎ Lemma 6 (Gibbs variational principle for matrix entropy). For every d×d× d density matrix S, supΘ=Θ⊤−Tr(SΘ)−logTr(e−Θ)=−(S), _ = \-Tr(S )- (e^- ) \=- H(S), where the supremum runs over all symmetric d×d× d matrices. The objective is invariant under Θ↦Θ+cId +cI_d for c∈ℝc . If S≻0S 0, the supremum is attained exactly at the family Θ=−logS+cId =- S+cI_d, c∈ℝc ; if S is singular, the supremum is not attained, but is approached along Θδ=−log(S+δId) _δ=- (S+δ I_d) as δ↓0δ 0, whose objective value is ∑iαilog(αi+δ)−log(1+δd) _i _i ( _i+δ)- (1+δ d) in terms of the eigenvalues αi _i of S. Proof. First, the invariance: replacing Θ by Θ+cId +cI_d changes −Tr(SΘ)-Tr(S ) by −cTr(S)=−c-c\,Tr(S)=-c and changes −logTr(e−Θ−cI)=−log(e−cTr(e−Θ))- (e^- -cI)=- (e^-cTr(e^- )) by +c+c, so the objective is unchanged. Next, the upper bound. For symmetric Θ , let RΘ:=e−Θ/Tr(e−Θ)R_ :=e^- /Tr(e^- ), a positive-definite density matrix with logRΘ=−Θ−logTr(e−Θ)Id R_ =- - (e^- )\,I_d. Klein’s inequality (Lemma 5) with T=RΘT=R_ gives Tr(SlogS)−Tr(SlogRΘ)≥0Tr(S S)-Tr(S R_ )≥ 0, which expands to 0≤Tr(SlogS)+Tr(SΘ)+logTr(e−Θ),0 (S S)+Tr(S )+ (e^- ), i.e. −Tr(SΘ)−logTr(e−Θ)≤−(S)-Tr(S )- (e^- )≤- H(S), with equality if and only if RΘ=SR_ =S. Then, attainment. If S≻0S 0, the equation RΘ=SR_ =S has the solutions Θ=−logS+cId =- S+cI_d, c∈ℝc , and no others: RΘ=SR_ =S forces −Θ=logS+logTr(e−Θ)Id- = S+ (e^- )I_d. If S is singular, no symmetric Θ satisfies RΘ=SR_ =S, since RΘ≻0R_ 0 always; hence the supremum is not attained. Finally, evaluating the objective at Θδ=−log(S+δId) _δ=- (S+δ I_d) gives −Tr(SΘδ)−logTr(e−Θδ) -Tr(S _δ)- (e^- _δ) =Tr(Slog(S+δId))−logTr(S+δId) =Tr (S (S+δ I_d) )- (S+δ I_d) =∑iαilog(αi+δ)−log(1+δd), = _i _i ( _i+δ)- (1+δ d), which converges to ∑iαilogαi=−(S) _i _i _i=- H(S) as δ↓0δ 0 (the terms with αi=0 _i=0 contribute 0⋅logδ=00· δ=0). Hence the supremum equals −(S)- H(S) in all cases. ∎ Remark 7 (Smoothed versus unsmoothed dual, and concavity as a byproduct). Lemma 6 explains why the min–max form is stated for the smoothed entropy. For the unsmoothed H0H_0, the covariance ΣQ _Q can be singular, in which case the supremum is not attained, and the near-maximizers Θδ=−log(ΣQ+δId) _δ=- ( _Q+δ I_d) have operator norm growing like log(1/δ) (1/δ): no compact adversary class captures the supremum uniformly over all Q. The spectral floor SQε⪰(ε/d)IdS_Q ( /d)I_d removes both obstructions: it confines the best response to the compact class εT_ of Proposition 1 and guarantees attainment. The lemma also yields an independent proof of fact (a) of the organization paragraph: it displays −(S)- H(S) as a supremum of affine functions of S, hence convex, so H is concave, which also yields Lemma 2. Remark 8 (Further readings of the min–max form). Two structural readings of (10) complement the closed-form best response discussed in Section 3.1. Linearization: for fixed Θ , the inner objective depends on Q only through the expectation of the per-sample payoff ϕ(x)⊤Θϕ(x)φ(x) \,φ(x), so the distribution-level reward −λHε(Q)-λ H_ (Q) becomes an ordinary expected loss at the cost of one d×d× d symmetric adversarial variable; Section 3.3 exploits this directly, where the spectral player joins the discriminator as a second adversary that admits a closed-form best response. Why smoothing: for the unsmoothed entropy the supremum runs over an unbounded matrix class and is not attained at rank-deficient covariances (Remark 7), whereas the spectral floor ε1dId 1dI_d confines the adversary to the compact class εT_ and guarantees attainment; the interchange in part (i) then follows from Sion’s minimax theorem, whose compactness requirement is satisfied by the Θ -side alone. Proof of Proposition 1. First, part (i). Since SQε⪰(ε/d)Id≻0S_Q ( /d)I_d 0, Lemma 6 gives −Hε(Q)=−(SQε)=supΘ=Θ⊤−Tr(SQεΘ)−logTr(e−Θ),-H_ (Q)=- H(S_Q )= _ = \-Tr(S_Q )- (e^- ) \, attained exactly at the family −logSQε+cId- S_Q +cI_d. Imposing Tr(Θ)=0Tr( )=0 pins the constant at c=1dTr(logSQε)c= 1dTr( S_Q ), which is the matrix Θ⋆(Q) (Q) of the statement; it is the unique traceless maximizer, since attainment forces membership in the family. For the operator-norm bound, the eigenvalues of SQεS_Q lie in [ε/d, 1−ε(1−1d)]⊆[ε/d,1][ /d,\,1- (1- 1d)] [ /d,1], so the eigenvalues ℓ1,…,ℓd _1,…, _d of −logSQε- S_Q lie in [0,log(d/ε)][0, (d/ )]; centering replaces ℓi _i by ℓi−ℓ¯ _i- with ℓ¯=1d∑jℓj∈[0,log(d/ε)] = 1d _j _j∈[0, (d/ )], so each centered eigenvalue satisfies |ℓi−ℓ¯|≤maxjℓj−minjℓj≤log(d/ε)| _i- |≤ _j _j- _j _j≤ (d/ ). Hence Θ⋆(Q)∈ε (Q) _ , and restricting the supremum to εT_ preserves both the value and the attainment, upgrading sup to max . It remains to pass to the per-sample form: for traceless Θ , Tr(SQεΘ)=(1−ε)Tr(ΣQΘ)+εdTr(Θ)=(1−ε)X∼Q[ϕ(X)⊤Θϕ(X)],Tr(S_Q )=(1- )Tr( _Q )+ dTr( )=(1- )\,E_X Q [φ(X) \,φ(X) ], using Tr(ΣQΘ)=Q[Tr(ϕϕ⊤Θ)]=Q[ϕ⊤Θϕ]Tr( _Q )=E_Q[Tr(φ )]=E_Q[φ φ]. This is (9). Next, part (i). Multiplying (9) by λ≥0λ≥ 0 preserves the maximum (for λ=0λ=0 both sides of the resulting identity vanish identically on εT_ , since −λHε(Q)=0-λ H_ (Q)=0 and the Θ -dependent terms carry the factor λ), and adding (Q;Pref)D(Q;P_ref), which does not depend on Θ , gives the pointwise identity (10). Since the two sides agree as functions of Q, the problem (Pλ) of minimizing the left side over P is the two-player game of minimizing the right side, as claimed. Finally, part (i). εT_ is convex and compact. For fixed Θ , Q↦λ(Q,Θ)Q _λ(Q, ) is convex and l.s.c.: the expectation term is affine in Q (and weakly continuous when the continuity clause of Assumption 2 is in force, ϕφ being bounded and continuous), and (⋅;Pref)D(·;P_ref) is convex l.s.c. by hypothesis. For fixed Q, Θ↦λ(Q,Θ) _λ(Q, ) is concave and continuous: the expectation term is linear in Θ , and −λlogTr(e−Θ)-λ (e^- ) is concave, since Lemma 6 exhibits Θ↦−logTr(e−Θ) - (e^- ) as an infimum over S of affine functions of Θ (namely −logTr(e−Θ)=infSTr(SΘ)−(S)- (e^- )= _S\Tr(S )- H(S)\, the dual reading of the same variational identity). Sion’s minimax theorem requires compactness of only one side, here the Θ -side εT_ , so no compactness of P is needed, and the interchange holds as stated. ∎ Lemma 7 (Best response and the entropy energy). For every distribution Q and every x∈x , λ(1−ε)ϕ(x)⊤Θ⋆(Q)ϕ(x)=λGQε(x)−cQ,λ(1- )\,φ(x) (Q)\,φ(x)=λ\,G _Q(x)-c_Q, where cQ:=λ(1−ε)dTr(−logSQε)∈[0,λ(1−ε)log(d/ε)]c_Q:= λ(1- )dTr (- S_Q )∈ [0,\ λ(1- ) (d/ ) ] is a constant independent of x. In particular, the per-sample payoff of the best-responding spectral adversary equals the entropy energy (8), up to an additive constant independent of x. Proof. By definition Θ⋆(Q)=−logSQε+1dTr(logSQε)Id (Q)=- S_Q + 1dTr( S_Q )I_d, so λ(1−ε)ϕ(x)⊤Θ⋆(Q)ϕ(x) λ(1- )\,φ(x) (Q)\,φ(x) =λ(1−ε)ϕ(x)⊤(−logSQε)ϕ(x) =λ(1- )\,φ(x) (- S_Q )φ(x) +λ(1−ε)dTr(logSQε)‖ϕ(x)‖22. + λ(1- )dTr ( S_Q ) φ(x) _2^2. The first term is λGQε(x)λ\,G _Q(x) by (8), and since ‖ϕ(x)‖2=1 φ(x) _2=1 the second term is the constant −cQ-c_Q. The range of cQc_Q follows because the eigenvalues of −logSQε- S_Q lie in [0,log(d/ε)][0, (d/ )], so their average lies in the same interval. ∎ D.2 Training-time instantiations: adversarial and maximum-likelihood models This appendix proves the results invoked in Section 3.3: the joint-adversary identity for adversarially trained generators (Proposition 3), the exactness of gradients computed through the frozen spectral adversary (Proposition 7), and the likelihood–KL–ELBO relations underlying the maximum-likelihood instantiation (Proposition 8). Proof of Proposition 3. Fix Q∈Q . By the critic representation (17) and the entropy dual (9), (Q;P^n)−λHε(Q)=supD∈cAQ(D)+maxΘ∈εBQ(Θ),D(Q; P_n)-λ H_ (Q)= _D _cA_Q(D)+ _ _ B_Q( ), where AQ(D)=P^n[u(D)]−Q[v(D)]A_Q(D)=E_ P_n[u(D)]-E_Q[v(D)] and BQ(Θ)=−λ(1−ε)Q[ϕ⊤Θϕ]−λlogTr(e−Θ)B_Q( )=-λ(1- )E_Q[φ φ]-λ (e^- ); here the passage from (9) to −λHε(Q)=maxΘBQ(Θ)-λ H_ (Q)= _ B_Q( ) is part (i) of the proof of Proposition 1 (multiplication by λ≥0λ≥ 0, with the degenerate case λ=0λ=0 giving BQ≡0=−λHε(Q)B_Q≡ 0=-λ H_ (Q)). Since D and Θ range over independent sets and the two objectives share no variable, the suprema add: supD∈cAQ(D)+maxΘ∈εBQ(Θ)=supD∈cmaxΘ∈εAQ(D)+BQ(Θ), _D _cA_Q(D)+ _ _ B_Q( )= _D _c\, _ _ \, \A_Q(D)+B_Q( ) \, and the right-hand side is the inner expression of (18). The identity therefore holds pointwise in Q, and taking the infimum over an arbitrary class Q, convex or not, preserves it. Attainment of the Θ -maximum at Θ⋆(Q) (Q) is part (i) of Proposition 1. No convexity of Q was used and no minimax interchange was performed. ∎ Proposition 7 (Frozen spectral adversary yields exact entropy gradients). Let Z∼PZZ P_Z on a latent space Z, let gϑ:→g_ :Z be measurable for each ϑ∈ℝp ^p and differentiable in ϑ at PZP_Z-a.e. z, let ϕφ be differentiable on an open set containing the relevant ranges with ‖ϕ‖2≡1 φ _2≡ 1, and let QϑQ_ denote the law of gϑ(Z)g_ (Z). Suppose there are a neighborhood U of ϑ0 _0 and L∈L1(PZ)L∈ L^1(P_Z) such that ‖∇ϑ[ϕ(gϑ(z))ϕ(gϑ(z))⊤]‖≤L(z)for all ϑ∈U and PZ-a.e. z. \| _ [φ(g_ (z))φ(g_ (z)) ] \|≤ L(z) all ∈ U and P_Z-a.e.\ z. Then ϑ↦Hε(Qϑ) H_ (Q_ ) is differentiable at ϑ0 _0 and ∇ϑHε(Qϑ)|ϑ0=(1−ε)Z[∇ϑϕ(gϑ(Z))⊤Θ⋆(Qϑ0)ϕ(gϑ(Z))|ϑ0], _ \,H_ (Q_ ) |_ _0=(1- )\,E_Z [ _ \,φ (g_ (Z) ) \, (Q_ _0)\,φ (g_ (Z) ) |_ _0 ], i.e. the exact gradient of the entropy coincides with the gradient of the expected per-sample payoff in which the spectral adversary is frozen at its best response Θ⋆(Qϑ0) (Q_ _0). Proof. Write S(ϑ):=SQϑε=(1−ε)Z[ϕ(gϑ(Z))ϕ(gϑ(Z))⊤]+ε1dIdS( ):=S _Q_ =(1- )\,E_Z [φ(g_ (Z))φ(g_ (Z)) ]+ 1dI_d. First, the domination hypothesis justifies differentiation under the expectation: ϑ↦S(ϑ) S( ) is differentiable at ϑ0 _0 with ∂ϑkS(ϑ0)=(1−ε)Z[∂ϑk(ϕ(gϑ(Z))ϕ(gϑ(Z))⊤)|ϑ0] _ _kS( _0)=(1- )\,E_Z [ _ _k (φ(g_ (Z))φ(g_ (Z)) ) |_ _0 ]. Next, as in the proof of Lemma 1, the matrix entropy H is Fréchet differentiable at every positive-definite matrix with D(S)[B]=−Tr(B(logS+Id))D H(S)[B]=-Tr (B( S+I_d) ) for symmetric B, and this applies at S(ϑ0)⪰(ε/d)Id≻0S( _0) ( /d)I_d 0. By the chain rule, ∂ϑkHε(Qϑ)|ϑ0=−Tr(∂ϑkS(ϑ0)(logS(ϑ0)+Id)). _ _kH_ (Q_ ) |_ _0=-Tr ( _ _kS( _0)\, ( S( _0)+I_d ) ). Then, the trace term drops out: since ‖ϕ‖22≡1 φ _2^2≡ 1, we have Tr(∂ϑkS(ϑ0))=(1−ε)∂ϑkZ[‖ϕ(gϑ(Z))‖22]=∂ϑk(1−ε)=0Tr ( _ _kS( _0) )=(1- )\, _ _kE_Z [ φ(g_ (Z)) _2^2 ]= _ _k(1- )=0, so the IdI_d contribution vanishes and ∂ϑkHε(Qϑ)|ϑ0=Tr(∂ϑkS(ϑ0)(−logS(ϑ0)))=(1−ε)Z[∂ϑkϕ⊤(−logS(ϑ0))ϕ|ϑ0], _ _kH_ (Q_ ) |_ _0=Tr ( _ _kS( _0)\, (- S( _0) ) )=(1- )\,E_Z [ _ _k\,φ (- S( _0) )φ |_ _0 ], where the matrix −logS(ϑ0)- S( _0) is held fixed under the derivative. Finally, replacing −logS(ϑ0)- S( _0) by Θ⋆(Qϑ0)=−logS(ϑ0)+1dTr(logS(ϑ0))Id (Q_ _0)=- S( _0)+ 1dTr( S( _0))I_d changes the per-sample payoff by a multiple of ‖ϕ‖22≡1 φ _2^2≡ 1, whose ϑ -gradient is zero; hence the displayed identity. In the language of the min–max game (10), this is an envelope (Danskin-type) statement: the inner maximum is attained at the unique Θ⋆(Qϑ0) (Q_ _0), and differentiating the value equals differentiating at the frozen maximizer. The direct computation above proves the identity without invoking any general envelope theorem. ∎ Maximum-likelihood training and VAEs. Deep maximum-likelihood models fit an explicit density qϑq_ by minimizing the empirical negative log-likelihood, which is the per-sample empirical proxy for the data-first divergence: up to an additive constant independent of ϑ , Pdata[−logqϑ(X)]=KL(Pdata∥Qϑ)+constE_P_data[- q_ (X)]=KL(P_data\|Q_ )+const (Proposition 8 below). This orientation is forced at training time: the model-first quantity KL(Qϑ∥P^n)KL(Q_ \| P_n) is typically infinite for a continuously supported model against an atomic empirical reference, whereas the likelihood is finite and estimable sample by sample. The IGA-regularized maximum-likelihood objective is minϑX∼P^n[−logqϑ(X)]−λHε(Qϑ), _ \;E_X P_n [- q_ (X) ]-λ\,H_ (Q_ ), (24) and when the likelihood is intractable, as in variational autoencoders, the negative evidence lower bound takes its place [23, 38]: minϑ,ηX∼P^n[−ELBO(X;ϑ,η)]−λHε(Qϑ). _ ,η\;E_X P_n [-ELBO(X; ,η) ]-λ\,H_ (Q_ ). (25) Proposition 8 records the exact relation between the two: the negative ELBO exceeds the negative log-likelihood by the encoder-posterior gap KL(rη(⋅∣x)∥pϑ(⋅∣x))≥0KL(r_η(· x)\,\|\,p_ (· x))≥ 0, a λ-independent quantity, so (25) is (24) plus a nonnegative gap that only the encoder parameters η tighten; the same reading applies to diffusion models trained through variational bounds [15]. We highlight two features of this combination. First, the entropy regularizer is likelihood-free: evaluating Hε(Qϑ)H_ (Q_ ) requires only samples from the decoder, never density values, so it applies to any latent-variable model whose sampler is differentiable, alongside a fidelity term that does require likelihoods. Second, the orientation caveat of Remark 10 applies: the repair guarantee of Theorem 6 is proved for base-anchored objectives and does not transfer to this data-first geometry, while the monotone path of Theorem 4, which is orientation- and convexity-agnostic, continues to describe the global minimizers of (24) and (25) as λ grows. When the reference is instead a smooth law, such as a pretrained teacher in fine-tuning rather than P^n P_n, the model-first KL anchor becomes directly usable. Proposition 8 (Likelihood, data-first KL, and the ELBO). Let ν be a σ-finite measure on X and let each model law QϑQ_ have ν-density qϑq_ . (i) Likelihood is data-first KL. Suppose Pdata≪νP_data ν with density p0p_0 and Pdata|logp0(X)|<∞E_P_data| p_0(X)|<∞. Then, for every ϑ , Pdata[−logqϑ(X)]=KL(Pdata∥Qϑ)+hν(Pdata),E_P_data [- q_ (X) ]=KL(P_data\|Q_ )+h_ν(P_data), where hν(Pdata):=−Pdata[logp0(X)]h_ν(P_data):=-E_P_data [ p_0(X) ] is finite and independent of ϑ , and the two sides are finite or +∞+∞ together. (i) Structure of the data-first divergence. For fixed P, the map Q↦KL(P∥Q)Q (P\|Q) is convex on P, and if X is Polish it is weakly lower semicontinuous. Consequently Theorem 4 applies to the objectives (24) and (25) whenever global minimizers exist, while Theorem 6, proved for the base-anchored orientation, does not transfer (Remark 10). (i) ELBO gap. Let qϑ(x)=∫pϑ(x∣z)pZ(dz)q_ (x)= p_ (x z)\,p_Z(dz) be a latent-variable model and rη(⋅∣x)r_η(· x) an encoder with rη(⋅∣x)≪pϑ(⋅∣x)r_η(· x) p_ (· x), where pϑ(⋅∣x)p_ (· x) is the model posterior. Then, for every x with qϑ(x)∈(0,∞)q_ (x)∈(0,∞), −ELBO(x;ϑ,η)=−logqϑ(x)+KL(rη(⋅∣x)∥pϑ(⋅∣x))≥−logqϑ(x),-ELBO(x; ,η)=- q_ (x)+KL (r_η(· x)\, \|\,p_ (· x) )\ ≥\ - q_ (x), with equality if and only if the encoder matches the model posterior at x. Proof. First, part (i). Decompose −logqϑ=log(p0/qϑ)−logp0- q_ = (p_0/q_ )- p_0 on the set p0>0\p_0>0\, which carries full PdataP_data-mass. The second term integrates to hν(Pdata)h_ν(P_data), finite by hypothesis. For the first term, set r:=qϑ/p0r:=q_ /p_0 on p0>0\p_0>0\; the positive part of logr r is PdataP_data-integrable, since logr≤r−1 r≤ r-1 gives Pdata[(logr)+]≤Pdata[r]=∫p0>0qϑν≤1E_P_data[( r)_+] _P_data[r]= _\p_0>0\q_ \,dν≤ 1, so Pdata[−logr]=Pdata[log(p0/qϑ)]E_P_data[- r]=E_P_data[ (p_0/q_ )] is well defined in (−∞,+∞](-∞,+∞]. If Pdata≪QϑP_data Q_ , then dPdata/dQϑ=p0/qϑdP_data/dQ_ =p_0/q_ holds PdataP_data-a.s. and Pdata[log(p0/qϑ)]=KL(Pdata∥Qϑ)E_P_data[ (p_0/q_ )]=KL(P_data\|Q_ ) by definition. If Pdata≪̸QϑP_data Q_ , pick A with Qϑ(A)=0<Pdata(A)Q_ (A)=0<P_data(A); then qϑ=0q_ =0 ν-a.e. on A, so log(p0/qϑ)=+∞ (p_0/q_ )=+∞ on a set of positive PdataP_data-measure and, the negative part being integrable, Pdata[log(p0/qϑ)]=+∞=KL(Pdata∥Qϑ)E_P_data[ (p_0/q_ )]=+∞=KL(P_data\|Q_ ) under the extended-value convention. In both cases the displayed identity holds, with both sides finite or +∞+∞ together since hν(Pdata)h_ν(P_data) is finite. Next, part (i). For convexity, fix Q0,Q1Q_0,Q_1 and θ∈(0,1)θ∈(0,1), and let ν′ν be a σ-finite measure dominating P, Q0Q_0, and Q1Q_1 (for instance P+Q0+Q1P+Q_0+Q_1), with densities p,q0,q1p,q_0,q_1; the mixture Qθ=(1−θ)Q0+θQ1Q_θ=(1-θ)Q_0+θ Q_1 has density qθ=(1−θ)q0+θq1q_θ=(1-θ)q_0+θ q_1. For fixed x with p(x)>0p(x)>0, the map q↦p(x)log(p(x)/q)q p(x) (p(x)/q) is convex in q>0q>0 (as −log- is convex), and extends convexly to q≥0q≥ 0 with value +∞+∞ at q=0q=0; composing with the affine θ↦qθ(x)θ q_θ(x) and integrating dν′dν preserves convexity, giving KL(P∥Qθ)≤(1−θ)KL(P∥Q0)+θKL(P∥Q1)KL(P\|Q_θ)≤(1-θ)KL(P\|Q_0)+ (P\|Q_1). For lower semicontinuity, we invoke the Donsker–Varadhan variational formula, a standard fact: for probability measures on a Polish space, KL(P∥Q)=supf∈Cb()P[f]−logQ[ef].KL(P\|Q)= _f∈ C_b(X) \E_P[f]- _Q[e^f] \. For each fixed f∈Cb()f∈ C_b(X), the map Q↦P[f]−logQ[ef]Q _P[f]- _Q[e^f] is weakly continuous: efe^f is bounded continuous, so Q↦Q[ef]Q _Q[e^f] is weakly continuous with values in the compact interval [e−‖f‖∞,e‖f‖∞]⊂(0,∞)[e^- f _∞,e f _∞]⊂(0,∞), on which log is continuous. A supremum of weakly continuous functions is weakly lower semicontinuous, which proves the claim. The consequences for Theorems 4 and 6 are as stated: the former uses only the existence of global minimizers and is agnostic to orientation and convexity, while the latter’s three-point argument differentiates the Bregman divergence in its first argument and is unavailable in the data-first orientation. Finally, part (i). Write the ELBO with encoder rηr_η: ELBO(x;ϑ,η)=Z∼rη(⋅∣x)[logpϑ(x∣Z)+logpZ(Z)−logrη(Z∣x)],ELBO(x; ,η)=E_Z r_η(· x) [ p_ (x Z)+ p_Z(Z)- r_η(Z x) ], with logpZ p_Z understood as the density of the prior with respect to the latent reference measure. By Bayes’ rule, pϑ(z∣x)=pϑ(x∣z)pZ(z)/qϑ(x)p_ (z x)=p_ (x z)\,p_Z(z)/q_ (x) for qϑ(x)∈(0,∞)q_ (x)∈(0,∞), so logpϑ(x∣z)+logpZ(z)=logpϑ(z∣x)+logqϑ(x) p_ (x z)+ p_Z(z)= p_ (z x)+ q_ (x), and substituting, ELBO(x;ϑ,η) (x; ,η) =logqϑ(x)−Z∼rη(⋅∣x)[logrη(Z∣x)pϑ(Z∣x)] = q_ (x)-E_Z r_η(· x) [ r_η(Z x)p_ (Z x) ] =logqϑ(x)−KL(rη(⋅∣x)∥pϑ(⋅∣x)). = q_ (x)-KL (r_η(· x)\, \|\,p_ (· x) ). Negating gives the display; nonnegativity of KL gives the inequality, with equality if and only if rη(⋅∣x)=pϑ(⋅∣x)r_η(· x)=p_ (· x). ∎ Appendix E Proofs for Section 4 This appendix proves the results of Section 4: the bias and consistency of the empirical entropy wall in Theorem 5, the sub-wall repair guarantee, and the wall-crossing statement (Proposition 9). Theorem 5 (Bias and consistency of the empirical wall). Let X1,X2,…X_1,X_2,… be i.i.d. from PdataP_data, let ϕ:→ℝdφ:X ^d satisfy ‖ϕ(x)‖2=1 φ(x) _2=1, and let P^N=1N∑i=1NδXi P_N= 1N _i=1^N _X_i. Then, for either H∈H0,HεH∈\H_0,H_ \ and every N≥1N≥ 1: (i) downward bias: H(P^N)≤H(Pdata)E\,H( P_N)≤ H(P_data); (i) monotonicity in the sample size: H(P^N+1)≥H(P^N)E\,H( P_N+1) \,H( P_N); (i) consistency: H(P^N)⟶H(Pdata)E\,H( P_N) H(P_data) as N→∞N→∞. Proof of Theorem 5. We prove the three parts in turn. First, for part (i), recall from Lemma 2 that H∈H0,HεH∈\H_0,H_ \ is concave in its distribution argument through the affine map Q↦ΣQ _Q. The empirical covariance is unbiased: ΣP^N=1N∑i=1N[ϕ(Xi)ϕ(Xi)⊤]=ΣPdata.E\, _ P_N= 1N _i=1^NE [φ(X_i)φ(X_i) ]= _P_data. Jensen’s inequality for the concave map Σ↦H H then gives H(P^N)≤H(ΣP^N)=H(Pdata).E\,H( P_N)≤ H (E\, _ P_N )=H(P_data). Next, for part (i), we use a leave-one-out averaging identity. Fix N+1N+1 samples and, for j=1,…,N+1j=1,…,N+1, let P^N(−j)=1N∑i≠jδXi P_N^(-j)= 1N _i≠ j _X_i. Each index appears in exactly N of the N+1N+1 leave-one-out measures, so P^N+1=1N+1∑j=1N+1P^N(−j),equivalentlyΣP^N+1=1N+1∑j=1N+1ΣP^N(−j). P_N+1= 1N+1 _j=1^N+1 P_N^(-j), _ P_N+1= 1N+1 _j=1^N+1 _ P_N^(-j). Concavity of H gives, pathwise, H(P^N+1)≥1N+1∑j=1N+1H(P^N(−j)).H( P_N+1)≥ 1N+1 _j=1^N+1H ( P_N^(-j) ). Each P^N(−j) P_N^(-j) has the same distribution as P^N P_N, so taking expectations yields H(P^N+1)≥H(P^N)EH( P_N+1) ( P_N). Finally, for part (i), note that since ‖ϕ(x)‖2=1 φ(x) _2=1, the summands ϕ(Xi)ϕ(Xi)⊤φ(X_i)φ(X_i) are i.i.d. bounded random matrices with mean ΣPdata _P_data, so by the (matrix) strong law of large numbers ΣP^N→ΣPdata _ P_N→ _P_data almost surely. Both H0H_0 and HεH_ are continuous functions of the covariance matrix on the compact set of density matrices and bounded in [0,logd][0, d], so H(P^N)→H(Pdata)H( P_N)→ H(P_data) a.s.; dominated convergence then gives H(P^N)→H(Pdata)EH( P_N)→ H(P_data). ∎ Remark 9 (Interpretation and experimental consequence). Theorem 5 upgrades the informal assumption of downward diversity bias to a theorem for the empirical law; it does not by itself establish that a trained generator Q0Q_0 satisfies H0(Q0)<ρ⋆H_0(Q_0)< _ , which remains a separate empirical claim, consistent with reported spectral deficits in modern generators. Because ρ^⋆=H0(P^N) ρ_ =H_0( P_N) is downward biased, and a finite generated batch inherits the same downward bias when estimating H0(Qλ)H_0(Q_λ), wall-crossing plots should use matched sample sizes, repeated subsampling, or a bias-aware estimator. Theorem 6 (Population discrepancy improves up to the wall). Let Φ be Fréchet differentiable and strictly convex on an open convex set containing P, with Bregman divergence DΦ(P,Q)=Φ(P)−Φ(Q)−⟨∇Φ(Q),P−Q⟩D_ (P,Q)= (P)- (Q)- ∇ (Q),P-Q , and let Qλ∈argminQ∈DΦ(Q,Q0)−λH(Q)Q_λ∈ Q \! \>\D_ (Q,Q_0)-λ H(Q)\ with H concave and P convex. If Pdata∈P_data and H(Qλ)≤H(Pdata)H(Q_λ)≤ H(P_data), then DΦ(Pdata,Qλ)+DΦ(Qλ,Q0)≤DΦ(Pdata,Q0),D_ (P_data,Q_λ)+D_ (Q_λ,Q_0)≤ D_ (P_data,Q_0), and hence in particular DΦ(Pdata,Qλ)≤DΦ(Pdata,Q0)D_ (P_data,Q_λ)≤ D_ (P_data,Q_0). Proof. By Proposition 6 applied to J(Q)=DΦ(Q,Q0)J(Q)=D_ (Q,Q_0) over the convex P, Qλ∈argminDΦ(Q,Q0):Q∈,H(Q)≥ρλQ_λ \,min\D_ (Q,Q_0):Q ,\ H(Q)≥ _λ\ with ρλ=H(Qλ) _λ=H(Q_λ). The feasible set =∩H≥ρλC=P∩\H≥ _λ\ is convex (H concave). The below-wall hypothesis H(Pdata)≥ρλH(P_data)≥ _λ and Pdata∈P_data give Pdata∈P_data . Because Q↦DΦ(Q,Q0)Q D_ (Q,Q_0) is convex and differentiable in its first argument (with ∇QDΦ(Q,Q0)=∇Φ(Q)−∇Φ(Q0) _QD_ (Q,Q_0)=∇ (Q)-∇ (Q_0)) and C is convex, first-order optimality of the minimizer QλQ_λ over C gives, for every feasible P, ⟨∇Φ(Qλ)−∇Φ(Q0),P−Qλ⟩≥0. ∇ (Q_λ)-∇ (Q_0),P-Q_λ ≥ 0. Setting P=Pdata∈P=P_data and using the three-point Bregman identity DΦ(Pdata,Q0) D_ (P_data,Q_0) =DΦ(Pdata,Qλ)+DΦ(Qλ,Q0) =D_ (P_data,Q_λ)+D_ (Q_λ,Q_0) +⟨∇Φ(Qλ)−∇Φ(Q0),Pdata−Qλ⟩, + ∇ (Q_λ)-∇ (Q_0),P_data-Q_λ , together with nonnegativity of the inner-product term, yields DΦ(Pdata,Q0)≥DΦ(Pdata,Qλ)+DΦ(Qλ,Q0)D_ (P_data,Q_0)≥ D_ (P_data,Q_λ)+D_ (Q_λ,Q_0). Nonnegativity of DΦ(Qλ,Q0)D_ (Q_λ,Q_0) gives the second inequality. ∎ Remark 10 (What the repair theorem does not cover). Theorem 6 concerns the base-anchored orientation DΦ(Q,Q0)D_ (Q,Q_0) (optimized law first) and does not transfer to a data-first objective DΦ(P^n,Q)D_ ( P_n,Q): the gradient of a Bregman divergence in its second argument involves the Hessian of Φ and is not ∇Φ(Q)−∇Φ(P^n)∇ (Q)-∇ ( P_n), so the three-point identity no longer collapses the cross term. In particular, the maximum-likelihood instantiation of Section 3.3, whose fidelity is the data-first KL by Proposition 8(i), inherits the monotone path of Theorem 4 but not the repair guarantee. Two clean options remain: (i) the base-anchored objective DΦ(Q,Q0)−λH(Q)D_ (Q,Q_0)-λ H(Q), especially with Q0=PθQ_0=P_θ. (i) a symmetric Hilbertian discrepancy such as squared MMD, for which the orientation is immaterial. Proposition 9 (Local wall crossing). Let P be convex, Pdata∈P_data , and suppose there exists R∈R with H0(R)>H0(Pdata)H_0(R)>H_0(P_data). Put Qt=(1−t)Pdata+tRQ_t=(1-t)P_data+tR for t∈(0,1]t∈(0,1]. If (Qt;Pdata)→0D(Q_t;P_data)→ 0 as t↓0t 0, then for every δ>0δ>0 there exists Q∈Q with (Q;Pdata)≤δD(Q;P_data)≤δ and H0(Q)>H0(Pdata)H_0(Q)>H_0(P_data). Proof. By convexity of P, Qt∈Q_t . By concavity of H0H_0 (Lemma 2), H0(Qt)≥(1−t)H0(Pdata)+tH0(R)H_0(Q_t)≥(1-t)H_0(P_data)+tH_0(R); since H0(R)>H0(Pdata)H_0(R)>H_0(P_data), the right-hand side equals H0(Pdata)+t(H0(R)−H0(Pdata))>H0(Pdata)H_0(P_data)+t(H_0(R)-H_0(P_data))>H_0(P_data) for all t∈(0,1]t∈(0,1], so H0(Qt)>H0(Pdata)H_0(Q_t)>H_0(P_data) (plain concavity suffices; no strictness is used). By hypothesis choose tδ>0t_δ>0 with (Qtδ;Pdata)≤δD(Q_t_δ;P_data)≤δ; then Q=QtδQ=Q_t_δ satisfies both requirements. ∎ Appendix F Proofs for Section 3.2 and Section 5 This appendix proves the sampling-time results in the following order. We first record a tightness lemma for KL sublevel sets, which drives the existence argument in Theorem 1, and compute the first variation of the smoothed entropy (Lemma 1). We then record the formal stationarity condition behind the density-ratio representations (Remark 11), prove the tilt characterization (Proposition 2), and derive Theorem 1 by combining the existence argument with that proposition. The remainder of the appendix treats the propagation of the tilt through the noising process and the endpoint guarantees. Lemma 8 (Relative-entropy sublevel tightness). Let P be a probability measure on a Polish space and C≥0C≥ 0. The sublevel set Q:KL(Q∥P)≤C\Q:KL(Q\|P)≤ C\ is tight. Concretely, for any measurable A with 0<P(A)<10<P(A)<1 and any Q with KL(Q∥P)≤CKL(Q\|P)≤ C, Q(A)≤C+1log(1/P(A)).Q(A)\;≤\; C+1 (1/P(A)). (26) Proof. By the data-processing inequality applied to the binary partition A,Ac\A,A^c\, KL(Q∥P)≥d2(Q(A)∥P(A)),d2(q∥p)=qlogqp+(1−q)log1−q1−p,KL(Q\|P)≥ d_2 (Q(A)\,\|\,P(A) ), d_2(q\|p)=q qp+(1-q) 1-q1-p, where d2d_2 is the binary KL, with the usual conventions and d2(q∥p)=+∞d_2(q\|p)=+∞ if p∈0,1p∈\0,1\ while q∉0,1q∉\0,1\. Write p=P(A)∈(0,1)p=P(A)∈(0,1) and q=Q(A)q=Q(A). Since 1−p≤11-p≤ 1 gives log1−q1−p≥log(1−q) 1-q1-p≥ (1-q), and tlogt≥−e−1t t≥-e^-1 on [0,1][0,1], the terms qlogq q and (1−q)log1−q1−p(1-q) 1-q1-p together contribute at least −2/e≥−1-2/e≥-1; hence d2(q∥p)=qlog(1/p)+qlogq+(1−q)log1−q1−p≥qlog(1/p)−1;d_2(q\|p)=q (1/p)+q q+(1-q) 1-q1-p≥ q (1/p)-1; hence Q(A)log(1/P(A))≤KL(Q∥P)+1≤C+1Q(A) (1/P(A)) (Q\|P)+1≤ C+1, which is (26). For tightness, fix η>0η>0. Since every probability measure on a Polish space is tight, there is a compact K with P(Kc)P(K^c) as small as desired; taking P(Kc)P(K^c) small enough that (C+1)/log(1/P(Kc))≤η(C+1)/ (1/P(K^c))≤η yields supKL(Q∥P)≤CQ(Kc)≤η _KL(Q\|P)≤ CQ(K^c)≤η. ∎ Proof of Lemma 1. Write Aν:=∫ϕ(x)ϕ(x)⊤ν(x)A_ν:= φ(x)φ(x) dν(x), a symmetric matrix (finite since ϕφ is bounded and ν is finite). Since Q↦ΣQ _Q is affine, ΣQ+tν=ΣQ+tAν _Q+tν= _Q+tA_ν and hence SQ+tνε=SQε+t(1−ε)AνS_Q+tν =S_Q +t(1- )A_ν for all t for which Q+tνQ+tν is a probability measure. The matrix entropy (S)=−Tr(SlogS) H(S)=-Tr(S S) is Fréchet differentiable at every positive definite S, with derivative D(S)[B]=−Tr(B(logS+Id))D H(S)[B]=-Tr (B( S+I_d) ) for symmetric B; this applies at S=SQεS=S_Q because SQε⪰(ε/d)Id≻0S_Q ( /d)I_d 0, and the perturbed matrices SQε+t(1−ε)AνS_Q +t(1- )A_ν remain in a compact neighborhood of positive definite matrices for small t. By the chain rule along the affine path, dtHε(Q+tν)|t=0+ ddt\,H_ (Q+tν) |_t=0^+ =−(1−ε)Tr(Aν(logSQε+Id)) =-(1- )\,Tr (A_ν( S_Q +I_d) ) =(1−ε)∫ϕ(x)⊤(−logSQε)ϕ(x)ν(x) =(1- )\! \!φ(x) (- S_Q )φ(x)\,dν(x) −(1−ε)∫‖ϕ(x)‖22ν(x). -(1- )\! \! φ(x) _2^2\,dν(x). Since ‖ϕ(x)‖22=1 φ(x) _2^2=1, the last integral equals ν()=0ν(X)=0, and the first term is ∫GQεν G _Q\,dν by definition (8). ∎ We note that the normalization ‖ϕ‖2=1 φ _2=1 makes the trace term of the derivative drop out, so GQεG _Q represents the first variation of the entropy functional: the first variation of HεH_ at Q, along any admissible mass-preserving perturbation, integrates GQεG _Q against the perturbation. Remark 11 (Bregman stationarity). For a general Bregman anchor, a formal first-order condition explains how the geometry of Φ converts the entropy first variation into a displacement of the law: an interior optimizer Q⋆Q of (11) satisfies ∇Φ(Q⋆)−∇Φ(Pθ)=λgQ⋆+c,∇ (Q )-∇ (P_θ)=λ\,g_Q +c, where gQ=GQεg_Q=G _Q is the first variation of HεH_ (Lemma 1) and c is the scalar multiplier of the unit-mass constraint. We do not rely on this identity: in the KL geometry, the derivation below obtains the density ratio directly from the first-variation computation, with no interiority hypothesis. We now prove the tilt characterization of Section 3.2. Proof of Proposition 2. Throughout, write θ=Q∈:Q≪PθP_θ=\Q :Q P_θ\, a convex set, let F be extended by +∞+∞ off θP_θ, and recall from the statement that F attains a finite minimum on θP_θ; let Q⋆Q be any minimizer and q⋆=dQ⋆/dPθq =dQ /dP_θ. We first prove uniqueness, then mutual absolute continuity, then derive the tilt from the first-variation computation, and finally bound the density ratio. Uniqueness. On its finite domain the KL term is strictly convex in Q, and −λHε-λ H_ is convex by Lemma 2. Hence F is strictly convex where finite, and its minimizer is unique. Mutual absolute continuity. We have Q⋆≪PθQ P_θ with KL(Q⋆∥Pθ)<∞KL(Q \|P_θ)<∞ by finiteness of the minimum; it remains to prove q⋆>0q >0 PθP_θ-a.s. Suppose instead that q⋆=0q =0 on a measurable set A with Pθ(A)>0P_θ(A)>0. Let R=Pθ(⋅∣A)R=P_θ(· A) and Qt=(1−t)Q⋆+tR∈θQ_t=(1-t)Q +tR _θ for t∈(0,1)t∈(0,1). Exactly as in the corresponding computation for the KL term (splitting the integral over A, where the density of QtQ_t is t 1A/Pθ(A)t\,1_A/P_θ(A), and AcA^c, where it is scaled by 1−t1-t), KL(Qt∥Pθ)−KL(Q⋆∥Pθ)=tlogt+O(t).KL(Q_t\|P_θ)-KL(Q \|P_θ)=t t+O(t). The remaining entropy term of F changes by only O(t)O(t). Indeed, SQtε−SQ⋆ε=t(1−ε)(ΣR−ΣQ⋆)S_Q_t -S_Q =t(1- )( _R- _Q ) has norm O(t)O(t) (with constants depending only on ‖ϕ‖2=1 φ _2=1), and S↦−Tr(SlogS)S -Tr(S S) is Lipschitz on the compact spectral range [ε/d,1][ /d,1], its derivative −(logS+I)-( S+I) being bounded in operator norm by log(d/ε)+1 (d/ )+1 there. Therefore F(Qt)−F(Q⋆)=tlogt+O(t)<0for small t>0,F(Q_t)-F(Q )=t t+O(t)<0 small t>0, since tlogt→0−t t→ 0^- dominates O(t)O(t); this contradicts optimality. Thus q⋆>0q >0 PθP_θ-a.s. and Q⋆∼PθQ P_θ. First variation and the tilt. For bounded measurable h with Q⋆h=0E_Q h=0, set dQt=(1+th)dQ⋆dQ_t=(1+th)\,dQ , a valid probability law for |t|≤1/(1+‖h‖∞)|t|≤ 1/(1+ h _∞), and let ν=hdQ⋆ν=h\,dQ , a finite signed measure with ν()=0ν(X)=0. By Lemma 1 and the identity gQ=GQεg_Q=G _Q, the entropy term has derivative dtHε(Qt)|t=0=∫gQ⋆hQ⋆ ddtH_ (Q_t)|_t=0= g_Q \,h\,dQ . The KL term has derivative dtKL(Qt∥Pθ)|t=0=∫(logq⋆+1)hQ⋆=∫logq⋆hdQ⋆, ddtKL(Q_t\|P_θ) |_t=0= ( q +1)\,h\,dQ = q \,h\,dQ , where the +1+1 term vanishes since Q⋆h=0E_Q h=0; differentiation under the integral is justified by dominated convergence, as h is bounded and ∫q⋆|logq⋆|Pθ<∞ q | q |\,dP_θ<∞ from KL(Q⋆∥Pθ)<∞KL(Q \|P_θ)<∞ together with the uniform bound tlogt≥−e−1t t≥-e^-1. First-order optimality dtF(Qt)|t=0=0 ddtF(Q_t)|_t=0=0 for all such h therefore gives ∫(logq⋆−λgQ⋆)hQ⋆=0 ( q -λ g_Q )\,h\,dQ =0 for all bounded h with Q⋆h=0E_Q h=0, so the integrand in parentheses is Q⋆Q -a.s. (and, by mutual absolute continuity, PθP_θ-a.s.) equal to a constant. Exponentiating and normalizing, with the constant absorbed into the normalizer, yields (14). Boundedness of the ratio. The eigenvalues of SQεS_Q lie in [ε/d,1][ /d,1], so ‖−logSQε‖op≤log(d/ε) - S_Q _op≤ (d/ ) and 0≤λgQ≤λ(1−ε)log(d/ε)0≤λ g_Q≤λ(1- ) (d/ ) uniformly over Q and x. The exponent in (14) is therefore uniformly bounded, so the normalizer lies in (0,∞)(0,∞) and the density ratio is bounded above and below by positive constants. ∎ Proof of Theorem 1. We first establish existence, then obtain the remaining claims from Proposition 2. Existence. Work on θ=Q:Q≪PθP_θ=\Q:Q P_θ\, a convex set, and extend F by +∞+∞ off it. The infimum is finite: F(Pθ)=−λHε(Pθ)∈[−λlogd,0]F(P_θ)=-λ H_ (P_θ)∈[-λ d,0], while F≥−λlogd>−∞F≥-λ d>-∞ termwise. Let (Qn)(Q_n) be a minimizing sequence. Since 0≤Hε(Q)≤logd0≤ H_ (Q)≤ d, boundedness of F(Qn)F(Q_n) implies supnKL(Qn∥Pθ)=:C<∞ _nKL(Q_n\|P_θ)=:C<∞. By Lemma 8, the KL sublevel set Q:KL(Q∥Pθ)≤C\Q:KL(Q\|P_θ)≤ C\ is tight, so (Qn)(Q_n) is tight. By Prokhorov’s theorem a subsequence converges weakly to some Q⋆Q . Since ϕφ is bounded and continuous, Q↦ΣQ,SQεQ _Q,S_Q are weakly continuous; because SQε⪰(ε/d)Id≻0S_Q ( /d)I_d 0 uniformly, S↦−Tr(SlogS)S -Tr(S S) is continuous on the relevant compact spectral range, so Q↦Hε(Q)Q H_ (Q) is weakly continuous. The map Q↦KL(Q∥Pθ)Q (Q\|P_θ) is weakly l.s.c.: by the Donsker–Varadhan formula, KL(Q∥Pθ)=supf∈Cb()Q[f]−logPθ[ef]KL(Q\|P_θ)= _f∈ C_b(X)\E_Q[f]- _P_θ[e^f]\ is a supremum of weakly continuous functions of Q, exactly as in the proof of Proposition 8(i) with the roles of the two arguments exchanged. Therefore F(Q⋆)≤lim infnF(Qn)F(Q )≤ _nF(Q_n), so Q⋆Q attains the infimum; in particular KL(Q⋆∥Pθ)<∞KL(Q \|P_θ)<∞, so Q⋆≪PθQ P_θ. Specialization. By the existence step, F attains a finite minimum on θP_θ, so Proposition 2 applies: the minimizer Q⋆Q is unique and mutually absolutely continuous with PθP_θ, which proves part (i), and the tilt (14) holds with the total reward RQ⋆R_Q of (13), proving part (i). Boundedness. The spectral floor gives 0≤λGQ⋆ε≤λ(1−ε)log(d/ε)0≤λ\,G _Q ≤λ(1- ) (d/ ). Thus RQ⋆R_Q is uniformly bounded, so Z=Pθexp(RQ⋆)∈(0,∞)Z=E_P_θ (R_Q )∈(0,∞) and dQ⋆/dPθdQ /dP_θ is bounded above and below by positive constants; hence Q⋆Q has the same PθP_θ-essential support as PθP_θ, proving part (i). ∎ Remark 12 (Scope of the tilt characterization). The characterization (14) is a fixed point: RQ⋆R_Q depends on Q⋆Q through SQ⋆εS_Q (contrast Lemma 4, where the reward is fixed and the tilt is explicit). The KL anchor only reweights within supp(Pθ)supp(P_θ) and creates no mass where Pθ=0P_θ=0. Strict convexity proves uniqueness of the target law but does not imply that any particular fixed-point iteration is contractive; convergence of a numerical solver requires a separate argument. Proof of Theorem 2. For measurable A, qt⋆(A)=∫Kt(A∣x0)Q⋆(dx0)=Z−1∫Kt(A∣x0)w(x0)Pθ(dx0).q_t (A)= K_t(A x_0)\,Q (dx_0)=Z^-1\! K_t(A x_0)\,w(x_0)\,P_θ(dx_0). Disintegrate the base joint law of (X0,Xt)(X_0,X_t) as Kt(dxt∣x0)Pθ(dx0)=Pθ(dx0∣xt)pt(dxt)K_t(dx_t x_0)P_θ(dx_0)=P_θ(dx_0 x_t)\,p_t(dx_t). By Fubini’s theorem (applicable since w is bounded, by Theorem 1), qt⋆(A)=Z−1∫A(∫w(x0)Pθ(dx0∣xt))pt(dxt)=Z−1∫Aht(xt)pt(dxt),q_t (A)=Z^-1\! _A\! ( w(x_0)P_θ(dx_0 x_t) )p_t(dx_t)=Z^-1\! _Ah_t(x_t)\,p_t(dx_t), with ht(xt)=Pθ[w(X0)∣Xt=xt]h_t(x_t)=E_P_θ[w(X_0) X_t=x_t]. Since A was arbitrary, dqt⋆/dpt=ht/Zdq_t /dp_t=h_t/Z. If both marginals have positive differentiable densities, then logqt⋆=logpt+loght−logZ q_t = p_t+ h_t- Z; as Z is constant in xtx_t, ∇logqt⋆=∇logpt+∇loght∇ q_t =∇ p_t+∇ h_t. ∎ Theorem 7 (Exact reverse process). Suppose the forward SDE dXt=f(Xt,t)dt+g(t)dWtdX_t=f(X_t,t)\,dt+g(t)\,dW_t (for t increasing from 0 to T) admits strictly positive differentiable marginal densities and satisfies the standard regularity conditions for time reversal and the probability-flow construction. We use the standard reverse-time convention in which the displayed equations are integrated with decreasing t from T to 0. Then the reverse-time SDE, initialized at qT⋆q_T , dXt=[f(Xt,t)−g(t)2(∇logpt(Xt)+ut(Xt))]dt+g(t)dW¯t,dX_t= [f(X_t,t)-g(t)^2 (∇ p_t(X_t)+u_t(X_t) ) ]\,dt+g(t)\,d W_t, has time-zero law exactly Q⋆Q , where W¯ W is a reverse-time Brownian motion. The probability-flow ODE X˙t=f(Xt,t)−12g(t)2(∇logpt(Xt)+ut(Xt)), X_t=f(X_t,t)- 12\,g(t)^2 (∇ p_t(X_t)+u_t(X_t) ), likewise integrated with decreasing t, has the same one-time marginals. Equivalently, under the forward reparameterization τ=T−tτ=T-t and Yτ=XT−τY_τ=X_T-τ, both dynamics run with increasing τ and their drifts are the negatives of the displayed drifts evaluated at t=T−τt=T-τ (the diffusion term is unchanged). Proof. Anderson’s time-reversal theorem, under the stated regularity, gives the reverse-time SDE (integrated with decreasing t) for the process with marginals qt⋆q_t as having drift f−g2∇logqt⋆f-g^2∇ q_t and initial law qT⋆q_T . Substituting ∇logqt⋆=∇logpt+ut∇ q_t =∇ p_t+u_t from Theorem 2 yields the stated drift, and the marginals are qt⋆q_t for all t, in particular Q⋆Q at t=0t=0. The probability-flow ODE X˙t=f−12g2∇logqt⋆ X_t=f- 12g^2∇ q_t is the deterministic process with identical one-time marginals under the qT⋆q_T initialization. The τ=T−tτ=T-t statement follows from the chain rule dτYτ=−dtXt|t=T−τ ddτY_τ=- ddtX_t|_t=T-τ, which flips the sign of every drift while preserving the (sign-indifferent) diffusion coefficient. If instead one initializes at pT≠qT⋆p_T≠ q_T , the time-zero law is not Q⋆Q ; the discrepancy is quantified in Theorem 8. ∎ Remark 13 (Initialization). Practical samplers initialize from pTp_T, not qT⋆q_T . These coincide only when hTh_T is constant. Because dqT⋆/dpT=hT/Zdq_T /dp_T=h_T/Z, the mismatch is KL(pT∥qT⋆)=pTlogdpTdqT⋆=logZ−pTloghT(XT)KL(p_T\|q_T )=E_p_T dp_Tdq_T = Z-E_p_T h_T(X_T), which enters any rigorous comparison between a deployed sampler and Q⋆Q (Theorem 8). Theorem 7 is exact only with the qT⋆q_T initialization. Definition 2 (Plug-in fields). Assume ⊆ℝDX ^D and that ϕφ and the denoiser are differentiable. With guidance scale ωt≥0 _t≥ 0, u~tchain(xt)=ωtaJx^0(xt,t)⊤∇xRQ⋆(x^0(xt,t)), u_t^chain(x_t)= _taJ_ x_0(x_t,t) _xR_Q ( x_0(x_t,t)), where Jx^0J_ x_0 is the Jacobian of the denoiser. The direct-injection variant, applicable when clean and noisy states share dimension, is u~tdir(xt)=ωt∇xRQ⋆(x^0(xt,t)), u_t^dir(x_t)= _t _xR_Q ( x_0(x_t,t)), which uses a chosen state-space direction rather than the derivative of the composite map xt↦RQ⋆(x^0(xt,t))x_t R_Q ( x_0(x_t,t)). Remark 14 (Plug-in is uncontrolled). No general equality or one-sided bound relates utu_t and u~t u_t: RQ⋆R_Q is nonlinear, and neither conditional expectation nor differentiation commutes with a point-mass substitution. For the reward RQ⋆=λGQ⋆εR_Q =λ G _Q of (13), ∇xRQ⋆(x)=Jϕ(x)⊤[2λ(1−ε)(−logSQ⋆ε)ϕ(x)], _xR_Q (x)=J_φ(x) [2λ(1- )(- S_Q )φ(x) ], where JϕJ_φ is the Jacobian of ϕφ (the factor 22 comes from differentiating the quadratic form ϕ⊤Mϕφ Mφ with symmetric M=−logSQ⋆εM=- S_Q , which is the x-gradient of λGQ⋆ελ G _Q ). Remark 15 (Discrete updates are approximations). The DDPM/DDIM updates (20)–(21) are algebraically consistent with the corrected noise prediction under the noise–score convention ∇logpt=−εθ/1−α¯t∇ p_t=- _θ/ 1- α_t, and are discrete implementations inspired by Theorem 2. They do not exactly sample Q⋆Q even if u~t=ut u_t=u_t; DDIM adds a further ODE discretization and path-selection approximation. A discretization term must therefore be added to the continuous-time bound of Theorem 8. Theorem 8 (Endpoint KL and TV bounds). Let the exact reverse process have initial law qT⋆q_T , base score st=∇logpts_t=∇ p_t, and exact guidance ut=∇loghtu_t=∇ h_t; let the deployed process have initial law πT _T, learned score s^t s_t, and approximate guidance u~t u_t. Put et=s^t−st,δt=u~t−ut.e_t= s_t-s_t, _t= u_t-u_t. Assume that: (i) both continuous-time processes share the diffusion coefficient g(t)Ig(t)I with g(t)>0g(t)>0 on (0,T)(0,T) (nondegeneracy on the open interval); any endpoint degeneracy g(0)=0g(0)=0 or g(T)=0g(T)=0 is handled by truncating to [η,T−η][η,T-η], applying the bound there, and letting η↓0η 0, assuming the resulting integral converges; (i) the absolute-continuity and Novikov conditions for Girsanov’s theorem hold on each such subinterval. Then, for the orientation KL(Q^∥Q⋆)KL( Q\|Q ) between the clean endpoint laws Q⋆,Q^Q , Q (deployed law first), KL(Q^∥Q⋆)≤KL(πT∥qT⋆)+12∫0Tg(t)2ℙ^[‖et(Xt)+δt(Xt)‖22]t,KL( Q\|Q ) ( _T\|q_T )+ 12 _0^Tg(t)^2\,E_ P [ e_t(X_t)+ _t(X_t) _2^2 ]\,dt, and, by Pinsker’s inequality (again for the orientation KL(Q^∥Q⋆)KL( Q\|Q )), TV(Q^,Q⋆)≤[12KL(πT∥qT⋆)+14∫0Tg(t)2ℙ^[‖et(Xt)+δt(Xt)‖22]t]1/2.TV( Q,Q )≤ [ 12KL( _T\|q_T )+ 14 _0^Tg(t)^2\,E_ P [ e_t(X_t)+ _t(X_t) _2^2 ]\,dt ]^1/2. Proof. Let ℙ⋆P be the path law of the exact reverse process (initial law qT⋆q_T , drift bt⋆=f−g2(st+ut)b_t =f-g^2(s_t+u_t)) and ℙ P the path law of the deployed process (initial law πT _T, drift b^t=f−g2(s^t+u~t) b_t=f-g^2( s_t+ u_t)), both with diffusion coefficient g(t)Ig(t)I. The drift difference is b^t−bt⋆=−g(t)2(et+δt) b_t-b_t =-g(t)^2(e_t+ _t). First, we decompose the path-space relative entropy in the direction KL(ℙ^∥ℙ⋆)KL( P\|P ) (deployed first) by the chain rule over the initial time T, KL(ℙ^∥ℙ⋆)=KL(πT∥qT⋆)+πTKL(ℙ^(⋅∣XT)∥ℙ⋆(⋅∣XT)).KL( P\|P )=KL( _T\|q_T )+E_ _TKL ( P(· X_T)\,\|\,P (· X_T) ). Next, we evaluate the conditional term. Conditionally on XTX_T, the two processes share the diffusion coefficient g(t)Ig(t)I, which is nondegenerate on (0,T)(0,T) (or on each [η,T−η][η,T-η], with η↓0η 0 afterwards), and differ only in drift, so on that interval the change of measure is absolutely continuous and Girsanov’s theorem applies, giving KL(ℙ^(⋅∣XT)∥ℙ⋆(⋅∣XT)) ( P(· X_T)\,\|\,P (· X_T) ) =12ℙ^[∫0T‖g(t)−1(b^t−bt⋆)‖2t|XT] = 12\,E_ P [ _0^T \|g(t)^-1( b_t-b_t ) \|^2\,dt\ |\ X_T ] =12ℙ^[∫0Tg(t)2‖et+δt‖2t|XT], = 12\,E_ P [ _0^Tg(t)^2 e_t+ _t ^2\,dt\ |\ X_T ], using g(t)−1(b^t−bt⋆)=−g(t)(et+δt)g(t)^-1( b_t-b_t )=-g(t)(e_t+ _t). Averaging over XT∼πTX_T _T, we arrive at KL(ℙ^∥ℙ⋆)=KL(πT∥qT⋆)+12∫0Tg(t)2ℙ^‖et+δt‖2t.KL( P\|P )=KL( _T\|q_T )+ 12 _0^Tg(t)^2\,E_ P e_t+ _t ^2\,dt. Finally, the clean endpoint laws Q^,Q⋆ Q,Q are measurable images (the time-0 coordinate) of the path laws, so the data-processing inequality gives KL(Q^∥Q⋆)≤KL(ℙ^∥ℙ⋆)KL( Q\|Q ) ( P\|P ), which is the stated KL bound. Pinsker’s inequality TV(μ,ν)≤12KL(μ∥ν)TV(μ,ν)≤ 12KL(μ\|ν) applied to Q^,Q⋆ Q,Q gives the TV bound. ∎ The exact-guidance bound is recovered only when πT=qT⋆ _T=q_T and et≡0e_t≡ 0; a separate discretization term is still needed for the implemented DDPM/DDIM sampler (Remark 15). Remark 16 (i.i.d. sampling holds only for fixed guidance). At the population level Q⋆Q is a single law, so independent exact samplers with a fixed potential RQ⋆R_Q produce i.i.d. draws from Q⋆Q ; this distinguishes IGA from methods that define diversity only through a coupled batch objective. However, RQ⋆R_Q depends on the unknown Q⋆Q through SQ⋆εS_Q . If a practical algorithm recomputes covariance or entropy gradients from the same batch being generated, each particle’s drift depends on the others: the outputs are exchangeable but not independent. An i.i.d. guarantee requires one of the following: frozen-potential sampling, in which the potential is estimated in a separate stage, frozen, and used to run independent trajectories; independent-pilot estimation, in which the potential is estimated on an independent pilot sample; or a mean-field analysis, invoking a propagation-of-chaos argument when the potential is updated from the active batch. Absent these, finite-batch IGA guidance should be described as an interacting particle system. Appendix G Training-Time IGA for Diffusion Models: Proofs and Discussion Diffusion models are trained through variational bounds, which places them in the maximum-likelihood family of Appendix D.2. Attaching −λHε(Qϑ)-λ H_ (Q_ ) directly to the denoising objective requires samples from QϑQ_ , and hence full reverse rollouts inside the training loop. The framework offers a rollout-free alternative: perform the IGA correction on the data before fitting the denoiser. To this end, we apply Proposition 2 with the reference distribution P^n P_n; note that the proposition depends on PθP_θ only through its role as the reference measure, and the finite-minimum hypothesis holds automatically since the feasible set is the simplex over the training atoms. This application yields unique weights qi⋆=wi∑j=1nwj,wi=exp(λGQλ⋆ε(xi)),q_i = w_i _j=1^nw_j, w_i= (λ\,G _Q _λ(x_i) ), (27) which form a self-consistent softmax over the training set and define Qλ⋆=∑i=1nqi⋆δxiQ _λ= _i=1^nq_i _x_i. The weights can be computed as a finite-dimensional convex–concave saddle problem through the spectral dual of Proposition 1, with the spectral adversary and the reweighting playing the detection and response roles described after that proposition. The outcome can be viewed as a distributionally robust reweighting of the dataset, although not a worst-case-loss one (Remark 17, Appendix G). The following proposition shows that training on the reweighted data is justified exactly rather than heuristically: Proposition 10 (IGA training as divergence minimization toward reweighted data). Let PrefP_ref be a probability measure, let λ≥0λ≥ 0, ε∈(0,1) ∈(0,1), and let F be the objective (12) with PrefP_ref in place of PθP_θ. Suppose F attains a finite minimum over Q∈:Q≪Pref\Q :Q P_ref\, at Qλ⋆Q _λ. Then for every Q with F(Q)<∞F(Q)<∞, F(Q)−F(Qλ⋆)=KL(Q∥Qλ⋆)+λB−Hε(Q,Qλ⋆),F(Q)-F(Q _λ)=KL(Q\,\|\,Q _λ)+λ\,B_-H_ (Q,Q _λ), (28) where B−Hε(Q,Q′):=Hε(Q′)−Hε(Q)+∫GQ′εd(Q−Q′)≥0B_-H_ (Q,Q ):=H_ (Q )-H_ (Q)+ G _Q \,d(Q-Q )≥ 0 is the Bregman divergence of the convex functional −Hε-H_ . Every term on the right-hand side of (28) is a divergence between Q and the IGA-reweighted reference, and both terms vanish exactly at Q=Qλ⋆Q=Q _λ. Therefore, over any generator class, minimizing the IGA objective is equivalent to matching the reweighted law. For a diffusion model, this equivalence justifies weighted denoising score matching, i.e., the standard training loss with clean samples drawn according to the weights q⋆q in place of uniform weights, which coincides with diffusion training under the data law Qλ⋆Q _λ. When the variational bound is tight and the generator class is expressive, the minimizers of the weighted bound attain the IGA optimum. In general, the weighted bound controls the data-first divergence KL(Qλ⋆∥Qϑ)KL(Q _λ\|Q_ ), whereas the IGA excess (28) is the model-first sum; this is the standard mass-covering versus mode-seeking asymmetry, stated here in an exact form (Corollary 3, Appendix G, which also records the empirical-versus-population role of the reference). The following proves the results of the diffusion-training paragraph of Section 3.3: the finite-sample IGA reweighting of the data (Corollary 2), the exact decomposition of Proposition 10, its consequence for weighted denoising training (Corollary 3), and the relation to distributionally robust optimization (Remark 17). Corollary 2 (Finite-sample IGA reweighting). Let x1,…,xnx_1,…,x_n be the training samples and P^n=1n∑i=1nδxi P_n= 1n _i=1^n _x_i. For every λ≥0λ≥ 0 and ε∈(0,1) ∈(0,1), the objective of (12) with reference P^n P_n attains a finite minimum over Q∈:Q≪P^n\Q :Q P_n\, and its unique minimizer Qλ⋆=∑i=1nqi⋆δxiQ _λ= _i=1^nq_i _x_i has strictly positive weights given by the self-consistent softmax (27). Proof. The map q↦Qq=∑iqiδxiq Q_q= _iq_i _x_i identifies Q∈:Q≪P^n\Q :Q P_n\ with the simplex Δn=q∈ℝn:q≥0,∑iqi=1 _n=\q ^n:q≥ 0,\ _iq_i=1\; if some training points coincide, the identification merges the corresponding atoms and the argument below is unchanged. On Δn _n each term of the objective is finite and continuous: KL(Qq∥P^n)=∑iqilog(nqi)KL(Q_q\| P_n)= _iq_i (nq_i), continuous with the convention 0log0=00 0=0; and Hε(Qq)H_ (Q_q), the composition of the matrix entropy H, continuous on density matrices, with the affine map q↦(1−ε)∑iqiϕ(xi)ϕ(xi)⊤+εdIdq (1- ) _iq_iφ(x_i)φ(x_i) + dI_d. A continuous function on the compact set Δn _n attains its minimum, which is finite, so the hypothesis of Proposition 2 holds with P^n P_n in the role of the reference. That proposition gives uniqueness, mutual absolute continuity (equivalently, qi⋆>0q_i >0 for every i), and the tilt (14), which on atoms reads nqi⋆=wi/Znq_i =w_i/Z with Z=1n∑jwjZ= 1n _jw_j and with wiw_i as in (27); cancelling the factors of n gives (27). ∎ Two roles of the reference. The corollary solves the IGA problem anchored at the empirical training distribution exactly; the resulting Qλ⋆Q _λ is supported on the training set and is the implicit data law of the weighted training scheme below. When the generated law QϑQ_ of a continuously supported model is compared against a reference, KL(Qϑ∥P^n)=+∞KL(Q_ \| P_n)=+∞, so the decomposition of Proposition 10 is applied with a population reference, such as PdataP_data or a smooth teacher law in fine-tuning, for which F(Qϑ)F(Q_ ) is finite for absolutely continuous models. The empirical weights (27) are then the plug-in counterpart of the population tilt: the exponent is a fixed continuous function of x once the covariance SQλ⋆εS _Q _λ is given, and the empirical fixed point estimates exactly this covariance. We do not pursue a finite-sample analysis of this plug-in step here; Theorem 5 describes the behavior of the underlying moment estimates. Proof of Proposition 10. Write Q⋆=Qλ⋆Q =Q _λ and let T(x):=λGQ⋆ε(x)T(x):=λ\,G _Q (x) be the total reward (13), with PrefP_ref in the role of the reference, so that dQ⋆/dPref=eT/ZdQ /dP_ref=e^T/Z with Z=Pref[eT(X)]Z=E_P_ref[e^T(X)] by (14). Since the eigenvalues of SQεS _Q lie in [ε/d,1][ /d,1], the exponent is uniformly bounded: 0≤T≤λ(1−ε)log(d/ε)0≤ T≤λ(1- ) (d/ ). Fix Q with F(Q)<∞F(Q)<∞. Because 0≤Hε≤logd0≤ H_ ≤ d, finiteness of F(Q)F(Q) is equivalent to KL(Q∥Pref)<∞KL(Q\|P_ref)<∞, and in particular Q≪PrefQ P_ref. Step 1: chain rule for the KL term. The ratio dQ⋆/dPref=eT/ZdQ /dP_ref=e^T/Z is bounded above and below by positive constants, so Q≪PrefQ P_ref if and only if Q≪Q⋆Q Q , and PrefP_ref-almost surely logdQdPref=logdQdQ⋆+T−logZ. dQdP_ref= dQdQ +T- Z. The negative part of log(dQ/dPref) (dQ/dP_ref) is Q-integrable (as always for a log-density ratio: t(logt)−≤e−1t( t)_-≤ e^-1 for t≥0t≥ 0), and T−logZT- Z is bounded, hence Q-integrable; therefore all three expectations below are well defined in (−∞,+∞](-∞,+∞] and additivity holds: KL(Q∥Pref)=KL(Q∥Q⋆)+Q[T]−logZ,KL(Q\|P_ref)=KL(Q\|Q )+E_Q[T]- Z, (29) with KL(Q∥Q⋆)<∞KL(Q\|Q )<∞ exactly when KL(Q∥Pref)<∞KL(Q\|P_ref)<∞. Step 2: the entropy Bregman term is nonnegative. Let ν=Q−Q⋆ν=Q-Q , a finite signed measure with ν()=0ν(X)=0. For t∈[0,1]t∈[0,1], Q⋆+tν=(1−t)Q⋆+tQ∈Q +tν=(1-t)Q +tQ , and h(t):=Hε(Q⋆+tν)h(t):=H_ (Q +tν) is concave on [0,1][0,1] (Lemma 2, through the affine map Q↦ΣQ _Q) with right derivative h′(0+)=∫GQ⋆ενh (0^+)= G _Q \,dν (Lemma 1). Concavity places h(1)h(1) below the tangent at 0: Hε(Q)≤Hε(Q⋆)+∫GQ⋆εd(Q−Q⋆),H_ (Q)≤ H_ (Q )+ G _Q \,d(Q-Q ), which is exactly B−Hε(Q,Q⋆)≥0B_-H_ (Q,Q )≥ 0, and by the definition of B−HεB_-H_ , −λHε(Q)=−λHε(Q⋆)−λ∫GQ⋆εd(Q−Q⋆)+λB−Hε(Q,Q⋆).-λ H_ (Q)=-λ H_ (Q )-λ\! \!G _Q \,d(Q-Q )+λ\,B_-H_ (Q,Q ). (30) Step 3: assembly. Summing (29) and (30), and substituting Q[T]=λQ[GQ⋆ε]E_Q[T]= _Q[G _Q ], F(Q)=KL(Q∥Q⋆)+λB−Hε(Q,Q⋆)+C,F(Q)=KL(Q\|Q )+λ\,B_-H_ (Q,Q )+C, where the Q[GQ⋆ε]E_Q[G _Q ] terms cancel against the integral in (30), leaving λQ⋆[GQ⋆ε] _Q [G _Q ], and C=λQ⋆[GQ⋆ε]−logZ−λHε(Q⋆).C= _Q [G _Q ]- Z-λ H_ (Q ). Finally, applying (29) at Q=Q⋆Q=Q gives KL(Q⋆∥Pref)=λQ⋆[GQ⋆ε]−logZ,KL(Q \|P_ref)= _Q [G _Q ]- Z, so C=KL(Q⋆∥Pref)−λHε(Q⋆)=F(Q⋆)C=KL(Q \|P_ref)-λ H_ (Q )=F(Q ), which is (28). ∎ Corollary 3 (Weighted diffusion training). Let q⋆q be the weights of Corollary 2 and let ℓ(x;ϑ) (x; ) be any per-example diffusion training loss (a denoising-score-matching loss or a negative variational bound). Then: (i) The weighted objective ∑i=1nqi⋆ℓ(xi;ϑ)=X∼Qλ⋆[ℓ(X;ϑ)] _i=1^nq_i (x_i; )=E_X Q _λ[ (X; )] coincides with the corresponding standard training objective with data law Qλ⋆Q _λ; no other component of the training pipeline changes. (i) Let PrefP_ref and Qλ⋆Q _λ be as in Proposition 10. A law Q with F(Q)<∞F(Q)<∞ attains minQ′≪PrefF _Q P_refF if and only if Q=Qλ⋆Q=Q _λ; and for any sequence (ϑk)( _k) with KL(Qϑk∥Qλ⋆)→0KL(Q_ _k\|Q _λ)→ 0, F(Qϑk)→minQ′≪PrefF(Q′)F(Q_ _k)→ _Q P_refF(Q ). Proof. Part (i) is the definition of expectation under a finitely supported law. For part (i), the “only if” direction: if F(Q)=F(Qλ⋆)<∞F(Q)=F(Q _λ)<∞, then by (28) the two nonnegative terms vanish, in particular KL(Q∥Qλ⋆)=0KL(Q\|Q _λ)=0, so Q=Qλ⋆Q=Q _λ; the converse is trivial. For the convergence claim, write Qk=QϑkQ_k=Q_ _k and δk=supA|Qk(A)−Qλ⋆(A)| _k= _A|Q_k(A)-Q _λ(A)|, so that |∫fd(Qk−Qλ⋆)|≤2‖f‖∞δk | f\,d(Q_k-Q _λ) |≤ 2 f _∞ _k for bounded measurable f, and δk≤KL(Qk∥Qλ⋆)/2→0 _k≤ KL(Q_k\|Q _λ)/2→ 0 by Pinsker’s inequality. By (28) it suffices that each right-hand term vanishes along the sequence. The KL term does by hypothesis. For the Bregman term, each entry of ΣQk−ΣQλ⋆ _Q_k- _Q _λ is ∫ϕaϕbd(Qk−Qλ⋆) _a _b\,d(Q_k-Q _λ) with |ϕaϕb|≤1| _a _b|≤ 1, so ΣQk→ΣQλ⋆ _Q_k→ _Q _λ, hence SQkε→SQλ⋆εS _Q_k→ S _Q _λ within the compact set of density matrices with spectrum in [ε/d,1][ /d,1], on which H is continuous, giving Hε(Qk)→Hε(Qλ⋆)H_ (Q_k)→ H_ (Q _λ); and |∫GQλ⋆εd(Qk−Qλ⋆)|≤2(1−ε)log(d/ε)δk→0 | G _Q _λ\,d(Q_k-Q _λ) |≤ 2(1- ) (d/ )\, _k→ 0. Hence B−Hε(Qk,Qλ⋆)→0B_-H_ (Q_k,Q _λ)→ 0, completing the proof. ∎ Remark 17 (Saddle computation of the weights, and the relation to DRO). (a) Computation. By Proposition 1, on Δn _n the weights of Corollary 2 solve the finite-dimensional saddle problem minq∈ΔnmaxΘ∈ε∑iqilog(nqi) _q∈ _n\; _ _ \; \ _iq_i (nq_i) −λ(1−ε)∑iqiϕ(xi)⊤Θϕ(xi)−λlogTr(e−Θ), -λ(1- ) _iq_i\,φ(x_i) \,φ(x_i)-λ (e^- ) \, whose objective is convex and continuous in q on the compact Δn _n and concave and continuous in Θ on the compact εT_ ; by Sion’s minimax theorem the order of optimization may be interchanged, and both optima are attained. Alternating best responses are natural: at fixed q the inner maximum is the closed form Θ⋆(Qq) (Q_q) of Proposition 1, while at fixed Θ the outer minimization is, by Lemma 4 with reward G(x)=λ(1−ε)ϕ(x)⊤Θϕ(x)G(x)=λ(1- )φ(x) \,φ(x), the explicit softmax qi∝exp(λ(1−ε)ϕ(xi)⊤Θϕ(xi))q_i (λ(1- )φ(x_i) \,φ(x_i) ). (b) Not worst-case-loss DRO. It is instructive to contrast (27) with KL-penalized distributionally robust training of the denoiser, minϑmaxQ≪P^nQ[ℓ(X;ϑ)]−ηKL(Q∥P^n),η>0. _ \; _Q P_n\; \E_Q[ (X; )]-η\,KL(Q\| P_n) \, η>0. There, by Lemma 4, the inner maximizer reweights the data by the loss, qi∝exp(ℓ(xi;ϑ)/η)q_i ( (x_i; )/η): the adversarial reweighting tracks loss hardness and changes with ϑ at every step. The IGA reweighting (27) is ϑ -independent and tilts by the entropy energy: it up-weights points along spectral directions the data underpopulates, whether or not the current model finds them hard. Comparing the two Gibbs exponents, the schemes produce the same weights only when ℓ(⋅;ϑ) (·; ) is, on the training set, an affine function of the IGA exponent, which is a nongeneric coincidence. Replacing the denoiser’s training distribution by a worst-case-loss adversary therefore optimizes robustness, not spectral diversity, and is not equivalent to IGA training; the rigorous route to training-time IGA for diffusion models is the reweighting-and-refit composition of Corollaries 2 and 3. Appendix H Additional Numerical Results (a) CelebA-HQ: Fréchet distance (FD). (b) CelebA-HQ: kernel distance (KD). (c) ImageNet: Fréchet distance (FD). (d) ImageNet: kernel distance (KD). Figure 19: DINOv2-space distributional distances along the IGA target path. Fréchet distance (FD) and kernel distance (KD) are evaluated in the DINOv2 representation used for the spectral analysis. The initial portion of the IGA path reduces the diversity deficit while improving or maintaining distributional agreement with the data. As λ increases further, the distance minima occur at metric- and dataset-dependent operating points, consistent with the transition from below-wall diversity repair to beyond-wall spectral extrapolation. This appendix collects supplementary numerical results for the sampling-time and training-time experiments. Section H.1 reports Dinov2-space distributional distances for the CelebA-HQ and ImageNet experiments, complementing the independent Inception-v3 evaluation in the main text. Section H.2 reports the MNIST GAN results for the training-time realization of IGA discussed in Section 6.3. H.1 DINOv2-Space Distributional Distances The main text reports FID and KID in Inception-v3 feature space as an evaluation independent of the DINOv2 representation used to define the spectral entropy and entropy wall. Here we provide the corresponding DINOv2-space distributional distances. These results complement the independent Inception-v3 evaluation and make explicit the representation dependence of the precise fidelity optimum along the IGA regularization path. (a) (b) (c) Figure 20: Training-time IGA reduces class imbalance in an MNIST GAN. (a) Generated digit frequencies move toward the empirical MNIST distribution as the IGA multiplier increases. (b,c) Moderate regularization reduces both class-distribution total variation and independent evaluator-feature Fréchet distance. Results are averaged over five seeds; error bars denote standard error. H.2 Training-Time IGA on MNIST The training-time experiment of Section 6.3 evaluates whether the same spectral-entropy regularizer used for sampling-time IGA can be incorporated directly into adversarial training. Figure 20 reports the resulting class-frequency and distributional-fidelity measurements across the IGA path. Experimental details. We train a convolutional GAN on MNIST for 20 epochs. The generator maps a 64-dimensional standard-normal latent through a fully connected projection and two transposed-convolution stages to a 28×2828× 28 image, while the discriminator uses two strided convolutional layers followed by a linear output. We use batch size 128 and Adam with learning rate 2×10−42× 10^-4 and (β1,β2)=(0.5,0.999)( _1, _2)=(0.5,0.999) for both networks, with the non-saturating logistic generator objective and one discriminator update per generator update. We evaluate λ∈0,0.5,1,1.5,2,4λ∈\0,0.5,1,1.5,2,4\ over five random seeds. The fixed IGA representation is the unit-normalized 64-dimensional embedding of a separately trained MNIST classifier. At each generator update, the spectral adversary is recomputed from the current generated minibatch at its closed-form best response and then held fixed during the generator update, as in Proposition 7. We set ε=0.05 =0.05. For independent evaluation, we use a second frozen classifier with a different architecture and a 96-dimensional embedding. For each run, 10,000 generated samples are used to compute class frequencies, total variation distance to the empirical MNIST test-set class distribution, and feature Fréchet distance between generated and real test samples in the independent evaluator space. As shown in Figure 20(a), increasing λ initially redistributes generated mass away from overrepresented classes and toward classes that are underrepresented by the baseline GAN. This redistribution is reflected quantitatively in panels (b) and (c): both class-distribution total variation and independent evaluator-feature Fréchet distance improve substantially at intermediate values of λ. Their nonmonotone behavior at larger λ illustrates the tradeoff between the GAN fidelity objective and the distribution-level entropy reward.