Paper deep dive
Imaginative Generative AI: Crossing the Entropy Wall into Worlds Beyond Imitation
Farzan Farnia, Hossein Goli, Amin Gohari
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/13/2026, 4:16:57 AM
Summary
The paper introduces Imaginative Generative AI (IGA), a framework that enhances the diversity of generated samples by constraining the spectral entropy of the output distribution. It defines an 'Entropy Wall' based on the data distribution's spectral entropy, distinguishing between 'diversity repair' (below the wall) and 'imaginative generation' (beyond the wall). The authors derive a self-consistent exponential-tilt relation for the optimal target distribution under KL divergence, leading to 'IGA Guidance,' an inference-time method for score-based and diffusion models like DDPM and DDIM that requires no retraining.
Entities (10)
Relation Signals (8)
Imaginative Generative AI → defines → Entropy Wall
confidence 95% · The spectral entropy of the population data distribution defines an Entropy Wall.
Imaginative Generative AI → uses → von Neumann entropy
confidence 95% · Diversity is measured by the von Neumann entropy of the generated distribution's kernel covariance operator
Imaginative Generative AI → optimizesusing → KL Divergence
confidence 93% · under a KL anchor to a pretrained generator, the optimum satisfies a self-consistent exponential-tilt relation.
Entropy Wall → separates → Diversity Repair
confidence 92% · Below the wall, IGA performs diversity repair... Beyond the wall... imaginative generation.
Entropy Wall → separates → Imaginative Generation
confidence 92% · Below the wall, IGA performs diversity repair... Beyond the wall... imaginative generation.
IGA Guidance → appliedto → DDPM
confidence 90% · IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDPM
IGA Guidance → appliedto → DDIM
confidence 90% · IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDIM
Vendi score → relatedto →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative AI models are primarily designed to imitate the data distribution, an objective that neither corrects diversity lost by a learned generator nor defines how generation should extend beyond the diversity of the data itself. We introduce Imaginative Generative AI (IGA), a framework that makes diversity part of the target-distribution design problem: among distributions close to a reference, IGA selects one whose spectral diversity reaches a prescribed level. Diversity is measured by the von Neumann entropy of the generated distribution's kernel covariance operator in a fixed representation space, providing a reference-free representation-guided measure of how broadly probability mass occupies embedding directions. The spectral entropy of the population data distribution defines an Entropy Wall. Below the wall, IGA performs diversity repair, recovering variation that a learned generator has lost while remaining within the diversity level of the data. Beyond the wall, the data distribution itself becomes infeasible, and IGA deliberately departs from it to produce distributions with greater representation-relative spectral diversity, an operational notion of imaginative generation. These regimes form a single regularization path from imitation to imagination and define an i.i.d. target distribution at each prescribed diversity level. We develop the theory of this entropy-constrained projection and show that, under a KL anchor to a pretrained generator, the optimum satisfies a self-consistent exponential-tilt relation. This characterization leads to IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDPM and DDIM samplers. Experiments on synthetic and vision benchmarks demonstrate diversity repair below the Entropy Wall and controlled spectral extrapolation beyond it.
Tags
Links
- Source: https://arxiv.org/abs/2608.09385v2
- Canonical: https://arxiv.org/abs/2608.09385v2
Trouble viewing inline? Open PDF directly →
Full Text
157,257 characters extracted from source content.
Expand or collapse full text
Imaginative Generative AI : Crossing the Entropy Wall into Worlds Beyond Imitation Farzan Farnia ∗ , Hossein Goli ∗ , Amin Gohari † , Abstract Generative AI models are primarily designed to imitate the data distribution, an objective that neither corrects diversity lost by a learned generator nor defines how generation should extend beyond the diversity of the data itself. We introduceImaginativeGenerativeAI (IGA), a framework that makes diversity part of the target-distribution design problem: among distributions close to a reference, IGA selects one whose spectral diversity reaches a prescribed level. Diversity is measured by the von Neumann entropy of the generated distribution’s kernel covariance operator in a fixed representation space, providing a reference-free representation-guided measure of how broadly probability mass occupies embedding directions. The spectral entropy of the population data distribution defines an Entropy Wall. Below the wall, IGA performs diversity repair, recovering variation that a learned generator has lost while remaining within the diversity level of the data. Beyond the wall, the data distribution itself becomes infeasible, and IGA deliberately departs from it to produce distributions with greater representation- relative spectral diversity, an operational notion of imaginative generation. These regimes form a single regularization path from imitation to imagination and define an i.i.d. target distribution at each prescribed diversity level. We develop the theory of this entropy-constrained projection and show that, under a KL anchor to a pretrained generator, the optimum satisfies a self-consistent exponential-tilt relation. This characterization leads to IGA Guidance, a retraining-free inference-time method for score-based and diffusion models, including DDPM and DDIM samplers. Experiments on synthetic and vision benchmarks demonstrate diversity repair below the Entropy Wall and controlled spectral extrapolation beyond it. 1 Introduction “Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world.” — Albert Einstein The typical goal of a generative model is to reproduce the underlying distribution of its training data. For example, a successful image generator is expected to produce realistic images with approximately the same content and variation as the images on which it was trained. This principle underlies the standard paradigms of generative modeling in the literature, including generative adversarial networks (GANs) [1], variational autoencoders (VAEs) [2], and score-based and diffusion models [3–5]. Although these frameworks differ significantly in architecture and training, their population-level goal can be summarized in the following distributional discrepancy minimization: min Q∈P G D Q;P data ,(1) whereP data is the underlying data distribution,Qis the distribution produced by the generator over the set of feasible modelsP G , andD(·;·) measures the discrepancy (or divergence) between the two distributions. We refer to this prevailing view of generative modeling as distributional imitation. ∗ Contributed Equally and Listed in Alphabetical Order. Department of Computer Science and Engineering, The Chinese University of Hong Kong,farnia,hosseingoli@cse.cuhk.edu.hk. † Department of Information Engineering, The Chinese University of Hong Kong, agohari@ie.cuhk.edu.hk 1 arXiv:2608.09385v2 [cs.LG] 11 Aug 2026 Imitation is a natural statistical objective, but it also places a ceiling on what the generator is asked to do. Even an ideal solution of(1)is asked to matchP data , not to produce a distribution that is systematically more diverse or generate novel and creative content. Moreover, practical generators may not reach even the diversity of their training distribution. The recent study [6] by Farnia, et al. has found that generated samples can exhibit lower spectral diversity than real data when diversity is measured using reference-free diversity measures of the Vendi score [7] and R ́enyi kernel entropy [8]. This raises a fundamental question: How should the target of generative modeling be regularized when diversity and novelty, in addition to fidelity, are something we want to control? To address this question, we proposeImaginativeGenerativeAI (IGA), a framework that makes diversity part of the target-distribution design problem. Instead of asking only for the distribution closest to a reference distribution, IGA asks for the closest distribution whose diversity score is at least above a given prescribed level. LetP ref denote the reference distribution, which could be the empirical distribution b P n of training data or the distributionP θ of a pretrained generator. Then, IGA solves the following regularized discrepancy minimization problem: min Q∈P G D(Q;P ref ) subject to H(Q)≥ ρ, (2) whereH(Q) measures the spectral entropy (interpreted as diversity) of distributionQandρis the desired diversity level. Under the duality conditions developed in our theoretical analysis, the constrained problem at level ρ can equivalently be stated using a Lagrangian penalty at a corresponding multiplier λ≥ 0: min Q∈P G D(Q;P ref )− λH(Q)(3) Note that the two terms have complementary roles: The discrepancy term keeps generated samples close to the reference distribution, while the entropy term rewards the spectral diversity in the Vendi score. Setting λ = 0 recovers standard reference matching; increasing λ gives more weight to the spectral diversity term. We measure diversity using the von Neumann entropy (VNE) of the normalized kernel covariance operator induced byQin a fixed embedding space. Intuitively, VNE is low when generated samples concentrate along a few embedding directions and high when they spread across many directions; in the empirical setting, it is the logarithm of the Vendi score [7–9]. This measure is reference-free but representation-dependent: it evaluates the diversity ofQwithout requiring a comparison distribution, while the chosen embedding specifies which variations are meaningful. IGA thus controls spectral diversity relative to a given embedding. The data distribution itself provides a natural reference level for this diversity. To characterize this wall, we define the underlying distribution’s entropy as ρ ⋆ := H P data .(4) We callρ ⋆ the Entropy Wall. It is the spectral diversity of the data distribution in the chosen representation. The notion of entropy wall separates two different regimes in the IGA generative modeling approach: (Regime I) Below the Entropy Wall: Diversity Repair. The below-the-wall regime concerns IGA when we choose entropy lower-bound ρ to satisfy ρ≤ ρ ⋆ . In this regime, the required diversity level is no greater than the diversity already present in the underlying data distribution. Therefore, in this regime, IGA can then be viewed as repairing a diversity deficit in a learned generator: it encourages the generator to recover variation that was present in the data but weakened or lost during training the generative model. Note that, as empirically demonstrated by Farnia et al. in recent work [6], the standard generative models commonly suffer from a diversity bias, and the spectral entropy of their generated data cannot match that of the underlying distribution generating their training samples. In brief, the goal in this regime remains faithful modeling of the data, with an explicit mechanism for counteracting spectral diversity shortfall as shown in [6]. 2 Imitation Imagination An Overview of Imaginative GenAI Low entropy Projection Strong Imagination Figure 1: From Imitation to Imagination. Imitation aims to return a distributionQclose toP data ; IGA returns the closestQwithH(Q)≥ ρ, which meets the data distribution below and at the entropy wall ρ ⋆ =H(P data ) and leaves it beyond. Images are Stable Diffusion XL at matched prompts and seeds; the right column sweeps one multiplier from the low-entropy projection to strong extrapolation. (Regime I) Beyond the Entropy Wall: Imaginative Generation. This regime of applying IGA is when we select the projection lower-bound to satisfy the strict inequality: ρ > ρ ⋆ . Especially, we highlight that in this regime, the data distribution itself no longer satisfies the diversity constraint, and thus the IGA solution in(2)must intentionally differ fromP data , regardless of whether it is anchored directly to the data or to a pretrained model. The discrepancy term prevents this solution from moving arbitrarily far from the chosen referenceP ref , while the entropy constraint pushes it to occupy a broader set of embedding directions. We call this regime imaginative because the target has greater spectral diversity than the data distribution that defines the wall. Here, we use the term imaginative to describe generation whose spectral diversity exceeds that of the data in a specified representation space, e.g. CLIP or DINO embedding spaces for images. This representation-relative definition makes imagination operational while allowing the embedding to encode domain-relevant semantics. We evaluate the resulting variation using both the guiding representation and independent measures of sample quality and diversity. The two regimes form a single regularization path. As the targetρincreases, IGA moves from ordinary imitation to diversity repair and then, after crossing the entropy wallρ ⋆ =H(P data ) (corresponding to the real data distribution), to deliberate imagination. Therefore, our IGA framework defines diversity enhancement as a property of a single target distribution, enabling i.i.d. generation at a prescribed spectral-diversity level, including levels beyond the entropy of the data. We emphasize that this differs from methods that induce diversity through repulsive or sample-dependent interactions over the course of generating multiple samples, whose outputs form a coupled and generally non-i.i.d. batch [10–12]. IGA therefore provides an i.i.d. distributional alternative to interaction-based diversity promotion, while complementing work that evaluates 3 Increasing Imaginative Strength ( ) Figure 2: From imitation to imagination with SDXL. As the IGA multiplierλincreases, SDXL produces progressively stronger structural and compositional variations while preserving the underlying concept. diversity after generation [7, 8]. Next, we demonstrate that this distribution-level formulation further leads to a practical method for diversity-improved sampling from pretrained score-based and diffusion models. When the reference is a pretrained modelP θ and the discrepancy measure is the KL-divergenceKL(Q∥P θ ), the analysis in the main body shows that the optimal target takes the form Q ⋆ (dx) ∝ P θ (dx)· exp λG Q ⋆ (x) ,(5) whereG Q ⋆ (x) measures how placing probability mass nearxchanges the VNE spectral entropy term. Because this energy depends onQ ⋆ , the relation is self-consistent. It reweights the base model to increase the spectral diversity of the generated population while remaining close to the original distribution as much as possible. We specifically show that the application of this formulation to score-based and generative models can be performed by our proposed IGA Guidance. IGA Guidance is an inference-time approximation for score-based, DDPM, and DDIM samplers that requires no retraining. Our numerical experiments show promising results of the IGA guidance for large-scale diffusion models. For example, Figure 2 shows the application of IGA-Guidance to the large-scale SD-XL model and how increasing the parameterλleads to visually more diverse and imaginative image outputs for the input prompt ”A Skyscraper”. The summary of our contributions are as follows: •We formulate IGA, an entropy-constrained projection framework enabling i.i.d. generation at prescribed VNE diversity levels. •We introduce the entropy wall, separating diversity repair from controlled extrapolation beyond the data’s spectral diversity. • We characterize the IGA regularization path and derive a self-consistent exponential tilt for the KL- anchored optimum. •We develop IGA guidance for inference-time steering of pretrained score-based, DDPM, and DDIM samplers without retraining. 4 imagined worlds (ρ > ρ ⋆ ) spectral diversity H ρ ⋆ λ = 0 (anchor) H = 2.16 λ = 5 H = 2.64 λ = 20 H = 3.36 λ = 30 H = 3.65 entropy wallρ ⋆ = H(P data ) Crossing the Entropy Wall into Worlds Beyond Imitation Figure 3: A multiverse of diversity levels. Matched-seed PixArt-Σ samples at increasingλin the IGA framework (λ= 0 represents the original regularization-free PixArt-Σ), arranged by measured spectral entropy H. Prompt: “A skyscraper for a humid coastal city” The entropy wallρ ⋆ =H(P data ), i.e., the entropy of the real data distribution, separates data-consistent imitated models from higher-diversity imagined distributions. 5 2 Preliminaries In this section, we introduce the representation-level and distributional quantities used throughout the paper. We associate every distribution with a normalized kernel covariance matrix whose spectrum defines our notion of diversity and introduce a differentiable smoothed surrogate of the resulting entropy together with its per-sample energy. Extended conventions and an elementary Gibbs-tilt identity are deferred to Appendix C. Throughout this work,P data denotes the population data distribution and b P N = 1 N P N i=1 δ X i the empirical measure ofNi.i.d. samples from it; the upper-case letterNis reserved for generic empirical sample counts (evaluation batches, generated minibatches), and the lower-case letternfor the size of the training set, whose empirical measure we denote with b P n . 2.1 Representation Space and Kernel Covariance Matrix Consider a measurable representation mapφ:X → R d satisfying∥φ(x)∥ 2 = 1 for everyx∈X, which induces the normalized kernelk(x,x ′ ) =φ(x) ⊤ φ(x ′ ). For a probability distributionQonX, we associate with the representation its kernel covariance matrix Σ Q := E X∼Q φ(X)φ(X) ⊤ ∈ R d×d . The unit-norm normalization makes Σ Q a density matrix, i.e., a positive semidefinite matrix with unit trace, whose spectrum records how the representation ofQdistributes its mass across orthogonal feature directions. The covariance spectrum carries the notion of diversity developed next. The same spectrum is accessible from pairwise similarities. Given samplesx 1 ,...,x N with Gram matrix K = [k(x i ,x j )] N i,j=1 , the empirical kernel covariance b Σ N := 1 N N X i=1 φ(x i )φ(x i ) ⊤ shares its nonzero eigenvalues with 1 N K . Consequently, every spectral quantity introduced below can be computed from the normalized Gram matrix without explicitly forming the feature vectors. 2.2 Spectral Entropy Measures and Diversity Scores Following the discussion in [13,7,8], we use the following definition for the von Neumann entropy of a distribution Q as H 0 (Q) :=− Tr Σ Q log Σ Q ,(6) which is the Shannon entropy of the covariance spectrum. We note that the exponential of the above quantity Vendi(Q) :=exp(H 0 (Q)) is the Vendi score [7]. This can be interpreted as an effective number of occupied feature directions. For example, a spectrum uniform overrorthogonal directions yieldsH 0 (Q) =logrand Vendi(Q) = r. This notion of diversity is inherently reference-free, yet it depends on the representation model to embed the data: evaluatingH 0 (Q) requires no comparison distribution, while the fixed choice ofφ, or equivalently k, determines which variations count as distinct. Moreover, sinceS 7→− Tr(S logS) is concave on density matrices andQ 7→Σ Q is affine,H 0 is concave inQ(Lemma 2). At rank-deficient covariance matrices, however, H 0 need not be differentiable. The guidance analysis of Section 5 requires a well-defined first variation, so we move the covariance uniformly away from the boundary of the density-matrix cone. Forε ∈(0,1), we define the smoothed covariance and entropy S ε Q := (1− ε)Σ Q + ε I d d , H ε (Q) :=− Tr S ε Q logS ε Q .(7) 6 The spectral floorS ε Q ⪰ ε d I d places every eigenvalue in [ ε d , 1−ε(1− 1 d )] and guarantees thatH ε is differentiable throughout the feasible covariance set. The derivative ofH ε acts on individual samples through one quantity that recurs at every stage of the paper. We define the entropy energy G ε Q (x) := (1− ε)φ(x) ⊤ − logS ε Q φ(x),(8) which is the first variation ofH ε atQ(Lemma 1, Section 5). Since− logS ε Q has large eigenvalues precisely whereS ε Q has small ones,G ε Q (x) is large whenφ(x) aligns with feature directions thatQunderrepresents, and the spectral floor gives the uniform bound 0≤ G ε Q (x)≤(1− ε)log(d/ε). The energy reappears as the payoff of the spectral adversary at training time (Section 3) and as the exponent of the guidance tilt at sampling time (Section 5). The two entropy functionals play distinct roles in our analysis: Section 4 usesH 0 to define population spectral diversity, whereas Section 5 usesH ε to derive the guidance potential. Lemma 3 quantifies their uniform proximity as ε becomes small. 3The IGA Framework: From Imitative to Imaginative Generative Modeling Distributional imitation, which is mathematically formulated in(1), requires a generative model to match a reference distribution. In the IGA framework which we formulate in this work, we intentionally augment this objective by requiring the generated distribution to attain a prescribed level of spectral diversity, and the resulting formulation separates two questions: (i) what distribution should be targeted, and (i) how should that target be realized by a generative model? In what follows, we first present the distribution-level objective function and the regularized form in IGA. We then derive a min-max reformulation of the entropy reward in the IGA optimization, and then develop the framework’s application to sampling-time and training-time settings. 3.1 Formulating IGA via Constraining and Penalizing the Spectral Entropy Consider a reference distributionP ref , which in our applications is either the empirical data distribution b P n for the observed training data or the underlying distributionP θ of the (already trained) generative model. We also consider the spectral entropy functionalH. Given a divergence measureD(Q;P ref ), IGA selects the most faithful distribution whose spectral diversity reaches a target level ρ∈ R: minimize Q∈P D(Q;P ref ) subject to H(Q)≥ ρ. (C ρ ) The divergence objective function anchors the solution toP ref , ensuring the distribution solution remains as close as possible to the reference distribution, while the constraint specifies the desired spectral diversity level. In next section, we review and extend the discussion from [6] to interpretρrelative to the entropy of the data, distinguishing diversity repair from spectral extrapolation. We note that due to the convex structure of the above optimization, the application of standard convex duality shows that(C ρ )is equivalent to the Lagrangian formulation for a corresponding Lagrangian multiplier parameter λ≥ 0: Q λ = argmin Q∈P F λ (Q) :=D(Q;P ref )− λH(Q)(P λ ) 7 Remark 1. For every constrained solution satisfying the standard regularity conditions, there is a multiplier λ≥0 for which the same distribution solves(P λ ). Conversely, eachQ λ solves(C ρ )at its attained diversity level. The precise duality and attainment statements are given in Appendix D. A Min-Max Formulation of the IGA Optimization. The spectral entropy term in(P λ )is a nonlinear function of the covariance spectrum, yet it admits a precise convex dual formulation due to its concavity. For the smoothed entropy, the application of the Gibbs variational principle for the matrix-based entropy converts the smoothed IGA problem into a two-player game. Proposition 1 (Spectral Min-Max Formulation of IGA Optimization). Consider smoothed spectral entropy with parameter ε∈ (0, 1). Define the spectral adversary class as T ε := Θ∈ R d×d : Θ = Θ ⊤ , Tr(Θ) = 0, ∥Θ∥ op ≤ log(d/ε) . Then, the following equivalences hold: (i) Dual representation of the negative entropy. For every distribution Q on X , we have −H ε (Q) = max Θ∈T ε n −(1− ε) E X∼Q φ(X) ⊤ Θφ(X) − log Tr e −Θ o ,(9) and the maximum is attained at the unique traceless matrix Θ ⋆ (Q) :=− logS ε Q + 1 d Tr logS ε Q I d ∈ T ε . (i) Min-Max form of (P λ ). For every λ≥ 0, distribution P ref , and every Q∈P , we have D(Q;P ref )− λH ε (Q) = max Θ∈T ε A λ (Q, Θ),(10) where A λ (Q, Θ) :=D(Q;P ref )− λ(1− ε) E X∼Q φ(X) ⊤ Θφ(X) − λ log Tr e −Θ . Therefore, the above results show that(P λ )can be rewritten as the following two-player min-max problem (game): min Q∈P max Θ∈T ε A λ (Q, Θ) (i) Min-Max and Max-Min Equivalence. IfPis a convex set andD(·;P ref ) is a convex and lower semicontinuous function, the order of minimization and maximization in this game may be interchanged: min Q∈P max Θ∈T ε A λ (Q, Θ) = max Θ∈T ε min Q∈P A λ (Q, Θ) Proof. We defer the proof to the Appendix. We note that the best response in part (i) is the centered log-spectrum of the smoothed covariance, and its per-sample payoff coincides, up to an additive constant, with the entropy energy: the spectral adversary pays the generator exactlyλG ε Q of(8), rewarding samples along directions thatQunderrepresents (Lemma 7). Therefore, the training-time adversarial payoff and the sampling-time guidance field are induced by the same first-order quantity. The proof of Proposition 1, given in Appendix D.1, relies on Klein’s matrix relative-entropy inequality and the Gibbs variational principle for matrix entropy, both of which are proved there in full. We refer to Remark 8 for further structural implications of(10), including the linearization of the entropy reward and the role of smoothing in compactifying the adversary class. 8 3.2IGA for Sampling From a Pretrained Model: the Special case of KL- divergence This subsection presents the application of IGA for sampling from an available (supposedly pretrained) model. At sampling time, the pretrained model is held fixed as the reference distribution in the framework. We specifically consider the referenceP θ as the underlying distribution of the pretrained generator, and the Lagrangian IGA optimization problem becomes: Q ⋆ = argmin Q∈P n D(Q,P θ )− λH ε (Q) o .(11) Here, our goal is to sample from the optimal distribution Q ⋆ . We recall that(11)separates the desired target distribution from the algorithm used to sample it: the Bregman divergence determines which departures from the base generator are costly, while theH ε Lagrangian penalty rewards higher spectral diversity. In our analysis, we specifically focus on the geometry resulting from choosing the discrepancy measure to be the KL-divergence. In this specific case, the optimality conditions yield an explicit density-ratio characterization of the optimal solution. Note that, in the case of KL-divergence, the sampling-based IGA aims to minimize the following objective function: F (Q) := KL(Q∥P θ )− λH ε (Q)(12) over Q∈P, where the extended-value convention for the KL term sets F (Q) = +∞ off the set Q≪ P θ . Proposition 2 (Sampling-time target as a self-consistent exponential tilt). Letλ≥0 andε∈(0,1). IfF attains a finite minimum overQ∈P:Q≪ P θ , then the minimizerQ ⋆ is unique,Q ⋆ andP θ are mutually absolutely continuous, and, with the total reward defined by R Q (x) := λG ε Q (x),(13) the density ratio is the exponential tilt dQ ⋆ dP θ (x) = exp R Q ⋆ (x) E X∼P θ exp R Q ⋆ (X) (14) Proof. We defer the proof to the Appendix. Corollary 1 (Score-function relation under the exponential tilt). Under the assumptions of Proposition 2, suppose thatP θ andQ ⋆ admit differentiable densitiesp θ andq ⋆ , respectively. Then their score functions satisfy ∇ logq ⋆ (x) =∇ logp θ (x) + λ∇G ε Q ⋆ (x).(15) Proof. The result follows directly by taking the logarithm of(14)and differentiating with respect tox, noting that the log-normalizing constant is independent of x. The target is an exponential reweighting of the pretrained law. We highlight that the reweighting is self-consistent rather than externally prescribed, since the total rewardR Q ⋆ depends on the covariance of the unknown target itself. The multiplier λ sets the strength of the reweighting, and the uniform bound on the energy noted after(8)limits how strongly any single sample can be up- or down-weighted, while the KL term confines the redistribution of mass to the support of the base law. We note that Proposition 2 characterizes the target distribution over clean outputs and does not yet provide a sampler; also, the finite-minimum hypothesis remains to be verified. Section 5 addresses both points: under mild topological conditions the minimizer exists (Theorem 1), the tilt propagates exactly 9 through the forward noising process to an explicit time-dependent guidance field (Theorem 2), and the field is approximated with denoised predictions at a quantified endpoint error, all without changing the pretrained score network. Unlike guidance by a fixed sample-wise reward, the tilt depends on the target law itself, through its covariance; practical sampling therefore estimates this distribution-level quantity, for example from a pilot batch. We keep the sampling targetQ ⋆ notationally distinct from the training-time optimum Q train λ of Section 3.3, realized by changing generator parameters. 3.3 IGA for Training Generative Models with Entropy-Regularized Objective For the training-time application of the imaginative generative modeling in IGA, we change the original divergence minimization in standard generative modeling and include the additional Lagrangian term in the objective function −λH(Q) to promote higher spectral entropy in the trained model. Mathematically, we choose the reference distribution to be the empirical distribution b P n ofntraining samplesx 1 ,...,x n (i.e., b P n = 1 n P n i=1 δ x i ). Then, the optimization problem for training-time IGA will be computing the optimal solution to the spectral entropy-regularized divergence minimization problem: Q train λ = argmin Q∈P gen n D(Q; b P n )− λH(Q) o .(16) The first term specifies how divergence to the training data distribution is measured, while the second is a distribution-level regularizer: it acts jointly on generated examples and rewards coverage of feature directions that would otherwise be underrepresented. IGA therefore only augments the model’s distributional discrepancy objective and can be interpreted as a spectral entropy regularization in the divergence minimization task of training the generative model. Specifically, in the following, we focus on and apply the IGA training framework to the adversarial training of generative adversarial networks (GANs). Further discussion on application of training-time IGA to other generative modeling frameworks is deferred to the Appendix. Adversarial training and GANs. The min-max format appearing in Proposition 1 composes smoothly with objective functions that are formed in the adversarial-learning formulations of generative modeling. We note that the standard GAN [1,14,15] objectives measure the discrepancy through a critic (discriminator) class D c and real-valued link functions u,v : R→ R: D(Q; b P n ) = max D∈D c n E X∼ b P n u D(X) − E X∼Q v D(X) o .(17) As notable examples, Wasserstein GANs take a 1-Lipschitz critic class withu=v=idbeing the identity map [15];f-GANs choosev=f ∗ for the convex conjugate of the convexffunction underlying the target f-divergence, recovering the original GAN objective as a special case for JS-divergence [1,14]. Substituting (17)and the entropy dual(9)into(16)gives an exact reformulation in which the entropy reward joins the discriminator inside a single adversary. Proposition 3 (IGA-GAN formulation as min-max optimization). LetD(·; b P n ) admit the representation (17), and let λ≥ 0, ε∈ (0, 1). Then, for every class Q of distributions, we have the following min Q∈Q D(Q; b P n )− λH ε (Q) = min Q∈Q max (D,Θ)∈D c ×T ε n E X∼ b P n u D(X) − E X∼Q v D(X) − λ(1− ε) E X∼Q φ(X) ⊤ Θφ(X) − λ log Tr e −Θ o , (18) Note that the above has a single maximization over the joint adversary (D,Θ). For every fixedQthe joint maximization decouples across the two components, and the Θ-component is attained at the best response Θ ⋆ (Q) of Proposition 1. 10 Proof. We defer the proof to the Appendix. The proof, given in Appendix D.2, relies on a structural observation: the critic and the spectral adversary enter through suprema over independent variables, and such suprema combine additively. Hence, the identity holds pointwise inQ, and neither convexity ofP gen nor a minimax interchange is used; the interchange remains reserved for the ambient class (Remark 3, Appendix C). Algorithmically,(18)adds one adversary to standard GAN training, and this additional adversary is computationally inexpensive: while the critic Dis trained by gradient steps, the spectral player requires no training at all, since its best response is the centered log-spectrum Θ ⋆ (Q) and can be computed from an eigendecomposition of the minibatch covariance at O(d 3 ) cost per refresh. The two adversaries play complementary roles. The critic enforces the fidelity of individual samples by comparing generated examples against data, whereas Θ acts on the generated distribution as a whole and pays the generator the spectral novelty rewardλG ε Q (x), up to a sample-independent constant, for occupying directions that the current generated law neglects (Lemma 7). We also highlight that freezing Θ at its best response is not a heuristic: the gradients of the generator parameters through the frozen payoff are exactly the gradients ofλH ε (Q θ ) (Proposition 7, Appendix D.2). This envelope-type identity removes the need to differentiate through the eigendecomposition. 4The Entropy Wall: Spectral Entropy of Real Data as the Bound- ary between Imitation and Imagination 250500100050001000030000 Number of samples 8 10 12 14 16 18 Exponential entropy Entropy wall Base Model Guided below wall Guided beyond wall =0.5 =1 =2 =3 =4 Beyond wall · extrapolation Below wall · diversity repair (a) CelebA-HQ 25005000100002500050000 Number of samples 4 6 8 10 12 14 16 Exponential entropy Entropy wall Base Model Guided below wall Guided beyond wall =0.36 =1.2 =3 Beyond wall · extrapolation Below wall · diversity repair (b) ImageNet Figure 4: The Entropy walls on CelebA-HQ and ImageNet. Real and generated entropy estimates use matched sample sizes. Both base models remain below the data wall; increasingIGA λcloses the deficit and eventually crosses into the imagination regime. As we discussed in the introduction, the entropy wall is the spectral diversity of the data itself, measured in the chosen representation. We note that the concept of the entropy wall is implied by the discussion in [6], in which the authors reveal the spectral entropy gap between the standard generative models and their target underlying data distributions. In this section, we formalize the concept and propose the term ”Entropy Wall” to highlight the spectral entropy level of the underlying real data distribution. Particularly, we highlight that the definition of entropy wall separates two qualitatively different uses of the IGA regularization framework: as long as the required spectral diversity level in IGA stays at or below what the data exhibits, increasing diversity can be read as repairing a deficiency of the learned generator; once 11 the request exceeds it, the data distribution is no longer feasible for(C ρ ), and pushing further is deliberate extrapolation beyond the data. Like every diversity statement in this paper, the wall’s location depends on the fixed pair (φ,k) and is therefore representation-relative. Definition 1 (Entropy Wall). For spectral entropy function H, the entropy wall is ρ ⋆,H := H(P data ) Based on the above definition, a distributionQis below, on, or beyond the entropy wall according asH(Q) is <, =, or > ρ ⋆,H . Reading the regularization path of Theorem 4 through the wall gives it the statistical interpretation promised in the introduction. Note that we use the notationQ λ for the optimal solution to the problem with Lagrangian coefficient λ. Below the wall: diversity repair. WhenH(Q λ )≤ H(P data ), increasingλmovesQ λ toward higher diversity, and this movement is provably safe for a base-anchored, correctly oriented Bregman objective: every below-wall point on the path is no farther fromP data , in the anchoring divergence, than the base model is (Theorem 6, Appendix E). This is the precise sense in which the sub-wall path performs repair, counteracting the spectral contraction reported in modern generators [6]. It does not identifyQ λ withP data , nor does it guarantee that every induced semantic change recovers a genuine data mode. Beyond the wall: spectral extrapolation. WhenH(Q λ )> H(P data ), the same monotone increase inλmeans something different:Q λ is no longer estimatingP data but performing representation-relative extrapolation, spreading its mass across directions of the representation more broadly than the data does. The transition is a change of statistical interpretation, not a geometric barrier: the wall can be crossed at arbitrarily small discrepancy whenever a higher-entropy direction exists inPand the discrepancy is continuous along the mixture path toward it (Proposition 9, Appendix E). Beyond the wall we claim no improved estimation of the data distribution; the regime is evaluated as controlled, representation-relative extrapolation. 5 IGA Guidance for Score-based and Diffusion Models Here, we focus on diffusion models and apply sampling-time IGA to pre-trained score-based and diffusion models. The developments in this section largely build on the score-function characterization established in Corollary 1. We first establish existence and uniqueness of the tilted targetQ ⋆ , then propagate the tilt through the forward noising process, and finally derive retraining-free approximations for score-based, DDPM, and DDIM samplers. 5.1 The target distribution and exact guidance Assumption 1 (Sampling-time setup). Given the sample spaceX, the representation mapφis continuous, hence bounded by the unit-norm normalization.P θ is the original distribution of the pretrained generator. Also, as stated in previous sections, we suppose the hyperparameters satisfy λ≥ 0, ε∈ (0, 1). Under Assumption 1, we minimize the functionalFof(12)overP. The KL anchor makes the problem tractable for two reasons: the effective domainQ≪ P θ is convex, and the first variation ofKL(·∥P θ ) along a density perturbation is simply the log-density ratio. The main difficulty is thatH ε is highly nonlinear in Σ Q : although Σ Q is affine inQ, the matrix logarithm couples all eigenvalues. The following lemma resolves this difficulty and justifies the role assigned to the entropy energy at its definition(8); it is the analytical core of Proposition 2 and of the existence theorem below. 12 A wearable haute couture outfit on a mannequin, neutral studio background Vanilla SDXL IGA SDXL Figure 5: Fashion design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla samples cluster around beige and gold eveningwear with familiar gown and tailored shapes. IGA adds bright color blocking, asymmetric cuts, mixed materials, and large sculptural or feathered elements. 13 Vanilla SDXLIGA SDXL A skyscraper for a humid coastal city, full building visible in an urban skyline, realistic architectural visualization Figure 6: Architectural design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mainly produces straight glass towers with similar overall forms. IGA introduces curved shells, open frames, split tops, stacked blocks, and larger changes in color and proportion, while preserving a clear full-building view. 14 A gouache painting of an underwater scene Vanilla SDXLIGA SDXL Figure 7: Stylized underwater scenes with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mostly depicts coral reefs and schools of fish. IGA expands the scene content to divers, large creatures, vehicles, built structures, and cave-like spaces, while keeping the gouache rendering style. 15 Vanilla SDXL IGA SDXL A throne for a fantasy film, full object visible, neutral studio background, production design render Figure 8: Fantasy throne design with vanilla and IGA SDXL. Left: vanilla SDXL. Right: IGA SDXL. Corresponding cells use the same initial noise seed. Vanilla SDXL mostly returns ornate high-backed chairs with similar carved frames. IGA introduces spiked metal forms, curved black shells, moss-covered structures, and larger changes in the seat and back, while keeping the throne centered and fully visible. 16 A wearable haute couture outfit on a mannequin, neutral studio background Figure 9: IGAλ-sweep across SDXL and PixArt-Σ. Top: SDXL withλ∈0,4,8. Bottom: PixArt-Σ withλ ∈ 0,10,20. Within each model, corresponding positions use the same initial noise seed. Asλ increases, both models move from familiar dress shapes toward stronger asymmetry, larger volumes and accessories, and wider material and color choices, while staying consistent with the prompt. Lemma 1 (First variation of the smoothed entropy). Letφbe bounded and measurable with∥φ(x)∥ 2 = 1, and letG ε Q be the entropy energy(8). Then for every distributionQand every finite signed measureνwith ν(X ) = 0 such that Q + tν is a probability measure for all sufficiently small t > 0, d dt H ε (Q + tν) t=0 + = Z G ε Q (x)dν(x). The lemma therefore identifies the total rewardR Q of(13)as the first variation of the reward partλH ε (Q) of the objective (proof and further discussion in Appendix F). Theorem 1. Under Assumption 1, the following hold for the functional F of (12). (i)Existence and uniqueness.Fhas a unique minimizerQ ⋆ , andQ ⋆ is mutually absolutely continuous with P θ . (i) Self-consistent exponential tilt. Q ⋆ satisfies the tilt characterization (14). 17 (i)Boundedness.R Q ⋆ is uniformly bounded; consequentlydQ ⋆ /dP θ is bounded above and below by positive constants, and Q ⋆ has the same P θ -essential support as P θ . Proof. We defer the proof to the Appendix. (14)is exactly the tilt announced in(5), withG Q ⋆ =G ε Q ⋆ . The theorem verifies the finite-minimum hypothesis of Proposition 2 rather than assuming it: the proof, given in Appendix F, establishes existence by the tightness of KL sublevel sets on a Polish space together with weak lower semicontinuity, and then inherits uniqueness, mutual absolute continuity, and the tilt formula from that proposition. What the theorem adds is that the target is well defined without any attainment hypothesis and that the reward is uniformly bounded, so the tilt redistributes mass within the P θ -essential support and creates none (Remark 12). The tilt(14)concerns the clean distribution overx 0 , but diffusion samplers generatex 0 as the endpoint of a denoising process that starts from noise at timeT, so guidance must be injected at every noise levelt. Then, we letw(x 0 ) =exp(R Q ⋆ (x 0 )) andZ=E P θ w(X 0 ), so thatQ ⋆ (dx 0 ) = 1 Z w(x 0 )P θ (dx 0 ), and letK t (dx t |x 0 ) denote the forward noising kernel with time-t marginals p t ,q ⋆ t under P θ ,Q ⋆ . Theorem 2 (Derivation of the exact guidance field). Leth t (x t ) =E P θ [w(X 0 )| X t =x t ]. Then dq ⋆ t dp t = 1 Z h t , and if p t ,q ⋆ t admit positive differentiable densities, then we have ∇ logq ⋆ t (x t ) =∇ logp t (x t ) + u t (x t ), u t (x t ) :=∇ x t logh t (x t ).(19) Note that the functionh t averages the clean-sample rewardwover all origins that the base posterior regards as plausible forx t . We emphasize thath t is not the clean tilt evaluated at a denoised point estimate, and this distinction is what makes the identity exact: applying the tilt before the noising process does not commute with applying it afterward. Addingu t =∇ logh t to the base score yields a reverse process whose marginals matchq ⋆ t at every noise level (Theorem 7, Appendix F); for continuous-time samplers, the guided score can be used directly in the reverse SDE or the probability-flow ODE. One caveat accompanies this exactness. The exact reverse process must be initialized atq ⋆ T , which is not directly samplable, whereas practical samplers initialize fromp T (typically Gaussian noise). The two distributions coincide only whenh T is constant, i.e. when the terminal noise level has erased all reward information. The resulting mismatch enters the end-to-end bound of Theorem 8 as the initialization term KL(p T ∥q ⋆ T ), and its magnitude is quantified in Remark 13. 5.2 Practical diffusion guidance There exist three approximation items that separate the discussed theoretical framework from an implementable sampler. In the following, we make each one explicit and discuss how to address it. Plug-in guidance fields. The exact fieldu t =∇ logh t requires the gradient of a conditional log-moment- generating function under the base posteriorP θ (dx 0 | x t ), which is generally intractable. What is available at every noise level is a denoiserbx 0 (x t ,t)≈ E[X 0 | X t =x t ], and the plug-in approximation substitutes this point estimate for the posterior average. Two variants differ in how the reward is turned into a vector field: the chain-rule variant differentiatesx t 7→ R Q ⋆ (bx 0 (x t ,t)) through the denoiser JacobianJ bx 0 , while the cheaper direct-injection variant reuses the clean-space gradient as a direction in noisy-sample space: eu chain t (x t ) = ω t J bx 0 (x t ,t) ⊤ ∇ x R Q ⋆ bx 0 (x t ,t) , eu dir t (x t ) = ω t ∇ x R Q ⋆ bx 0 (x t ,t) , with guidance scaleω t ≥0. Both are heuristics without a general error bound:R Q ⋆ is nonlinear, and neither conditional expectation nor differentiation commutes with a point-mass substitution (Definition 2, Remark 14). We always report which variant is used. 18 Estimating the self-referential reward. The rewardR Q ⋆ depends on the unknown covarianceS ε Q ⋆ , so a practical sampler replaces it by an estimate, and how the estimate is maintained determines the statistical status of the outputs. If the covariance is frozen, i.e., computed once from a pilot batch and used to define a single estimated potential b Rfor all subsequent trajectories, the draws are conditionally i.i.d. from the frozen-potential law. If instead the covariance is recomputed on the fly from the batch being generated, each particle’s drift depends on the others, and the outputs form an exchangeable but non-i.i.d. interacting particle system (Remark 16). Frozen-potential estimation is therefore the setting in which IGA guidance can be described as sampling from a well-defined target distribution. Discrete sampler updates. Once a guidance fieldeu t is chosen, it is converted to a correction on the noise prediction. In theε-prediction parameterization, with the standard noise–score convention∇ logp t (x t ) = −ε θ (x t ,t)/ √ 1− ̄α t [5], guiding the score by +eu t corresponds to ε IGA θ (x t ,t) = ε θ (x t ,t)− √ 1− ̄α t eu t (x t ).(20) Substituting (20) into the DDPM posterior mean [4] μ θ = 1 √ α t x t − β t √ 1− ̄α t ε θ and the DDIM update [16] gives μ IGA θ (x t ,t) = μ θ (x t ,t) + β t √ α t eu t (x t ), x t−1 = √ ̄α t−1 bx IGA 0 + q 1− ̄α t−1 − σ 2 t ε IGA θ + σ t z, (21) withbx IGA 0 = (x t − √ 1− ̄α t ε IGA θ )/ √ ̄α t andσ t = 0 for deterministic DDIM; the √ 1− ̄α t in(20)and theβ t / √ α t in(21)cancel algebraically, so the DDPM mean correction is exactly +(β t / √ α t )eu t . A model trained withv- orx 0 -prediction is first converted to an equivalentε θ in the standard way (e.g.ε θ = √ ̄α t v θ + √ 1− ̄α t x t for v-prediction). These discrete updates are implementations inspired by Theorem 2, and they do not exactly sampleQ ⋆ even wheneu t =u t , because the reverse kernels are discretized (Remark 15). The end-to-end guarantee is given by Theorem 8 in Appendix F. This theorem boundsKL( b Q∥Q ⋆ ) andTV( b Q,Q ⋆ ) for the deployed continuous-time process in terms of the initialization mismatch, the score error, and the guidance error; a separate discretization term is required for the implemented sampler. 6 Numerical Evaluation Throughout our numerical study, we aim to empirically address the following questions: 1. Do widely used pretrained generative models show a diversity deficit and existence of the entropy wall? 2. Does our proposed framework address this deficit, and are there values ofλthat reach and go beyond the entropy wall? If so, as predicted by Theorem 6, is there an initial repair region in which the pretrained model moves closer to the data distribution it was meant to imitate? 3. Does crossing the wall produce structured, novel, and imaginative variations across different pretrained generative models, including text-conditional models? We test our theory and hypothesis on CelebA-HQ and ImageNet, and then ask what the resulting variation looks like in a large text-conditioned model. 19 (a) Population −0.50.00.5 embedding coordinate z 1 −0.5 0.0 0.5 embedding coordinate z 2 H ρ (Q) =3.11 Rare coverage: 4.2% ε (b) Fitted DDPM −0.50.00.5 embedding coordinate z 1 −0.5 0.0 0.5 embedding coordinate z 2 H ρ (Q) =2.87 Rare coverage: 0.6% ε (c) At the entropy wall −0.50.00.5 embedding coordinate z 1 −0.5 0.0 0.5 embedding coordinate z 2 H ρ (Q) =3.11 Rare coverage: 4.6% ε (d) Beyond the wall −0.50.00.5 embedding coordinate z 1 −0.5 0.0 0.5 embedding coordinate z 2 H ρ (Q) =3.28 Rare coverage: 17.5% ε PopulationFitted DDPMEntropy wallBeyond wallRare-region samples (e) Distributional repair 2.93.03.13.23.3 von Neumann entropy H ρ (Q) 0.000 0.025 0.050 0.075 0.100 0.125 ‖ C Q − C data ‖ F entropy wall ε (f ) Rare-region coverage 2.93.03.13.23.3 von Neumann entropy H ρ (Q) 0 5 10 15 20 rare-region coverage (%) entropy wall ε Figure 10: Repair and extrapolation on a nonlinear manifold. The fitted DDPM underrepresents the spiral endpoints. IGA restores population-level entropy and rare-region coverage near the wall and increases endpoint exploration beyond it. Panels (e,f) trace the covariance discrepancy and rare-region coverage; thin curves denote individual seeds and thick curves their mean. 6.1 Experimental Protocol Our experiments cover sampling-time IGA on controlled synthetic distributions and real-world image benchmarks, a training-time study on MNIST, and qualitative text-conditioned generation with SDXL. Because these settings use different models and evaluation criteria, we state only the shared experimental conventions here and introduce the setting-specific configurations in the corresponding subsections. Entropy representation and evaluation. Across all sampling-time experiments, we set the spectral floor of(7)toε= 10 −3 , for which Lemma 3 bounds the gap betweenH ε andH 0 by 0.015 nats, several times smaller than the smallest entropy difference we report. IGA guidance uses the smoothed entropyH ε , while, unless stated otherwise, we report the unsmoothed von Neumann entropyH 0 and its exponential,exp(H 0 ), corresponding to the Vendi score [7]. In all sampling-time experiments, we fix the coefficient of the IGA score correction to its theoretically prescribed value of one and vary onlyλ, so each point along the reported path corresponds to a different target Q ⋆ λ , rather than to a different guidance strength. For the real-image benchmarks, we use the CLS-token embeddings of DINOv2 ViT-B/14 [17] and approximate 20 −3 0 3 x 2 (a) λ=0 H=1.80 (b) λ=0.03 H=1.86 (c) λ=0.15 H=2.00 (d) λ=0.4 H=2.06 −303 x 1 −3 0 3 x 2 matched σ=0.00 −303 x 1 matched σ=0.28 −303 x 1 matched σ=0.57 −303 x 1 matched σ=0.75 IGA guidance Gaussian convolution Figure 11: IGA fills underrepresented inter-mode regions more coherently than entropy-matched noise. Top: base and IGA-guided DDIM samples asλincreases. Bottom: Gaussian-convolved samples with σselected to match the entropy of the corresponding IGA distribution. IGA connects the gaps between modes while preserving the original modal structure; Gaussian convolution broadens each mode isotropically. 505 PC1 7.5 5.0 2.5 0.0 2.5 5.0 7.5 PC2 dots: samples ring area: component probability A Population target H = 1.737 505 PC1 Base model H = 1.592 505 PC1 At calibrated wall, = 0.4 H = 1.743 505 PC1 Beyond wall, = 3 H = 1.906 12345678 nearest mixture component, ranked by base probability 0.0 0.1 0.2 0.3 0.4 0.5 component probability B Guidance repairs component imbalance base model population target guided, calibrated wall (= 0.4) guided, beyond (= 3) 0.10.00.1 entropy gap to population, H(Q)H(P data ) 0.00 0.01 0.02 0.03 0.04 fidelity cost, KL( Q P data ) Base At wall Beyond Population C Entropy-wall phase portrait Population target At wall Base model Beyond wall Figure 12: The entropy wall separates repair from extrapolation in a controlled mixture. (A) Population, base, calibrated-wall, and beyond-wall distributions. (B) IGA repairs the component imbalance at the wall and produces more uniform weights beyond it. (C) Population KL first decreases and then increases as the path crosses the entropy wall. 21 an RBF kernel on these embeddings using 1024 random Fourier features [18], with the RBF bandwidth selected by the median heuristic. Because empirical spectral entropy depends on the number of samples, we compare real and generated distributions using matched sample sizes when locating the empirical entropy wall. In DINOv2 feature space, we report Fr ́echet distance, kernel distance, and recall. As an evaluation independent of the guidance representation, we additionally report FID and KID in Inception-v3 feature space [19–21]. Synthetic and training-time experiments use the problem-specific metrics introduced in their respective subsections. Pilot estimation. In all sampling-time experiments, we use the chain-rule plug-in guidance field. The covariance entering the IGA potential is estimated from an independent pilot batch of 2048 samples drawn from the unguided base model and then frozen during subsequent sampling. Thus, conditioned on the frozen pilot estimate, individual sampling trajectories are independent. 6.2 Numerical Application of IGA in Post-hoc Sampling-time Mode 6.2.1 Synthetic Experiments with Known Groundtruth Model We first study three controlled settings in which the population distribution is known. These experiments allow us to evaluate whether the IGA path approaches the population below the entropy wall and departs from it beyond the wall. The three settings provide complementary evidence: a nonlinear manifold illustrates rare-region repair, an entropy-matched control distinguishes IGA from isotropic noise for increasing entropy, and a finite mixture exposes the redistribution of probability mass across modes. Rare-region repair on a nonlinear manifold. Figure 10 considers a DDPM [4] trained on a one- dimensional population embedded inR 128 . The fitted model captures the dominant central portion of the manifold but substantially underrepresents its endpoints, leading to lower entropy and reduced rare-region coverage. Here, rare regions are defined as the portions outside the central 70% of the normalized manifold coordinate. Increasingλinitially corrects this contraction. Near the entropy wall, IGA recovers both the population entropy and the missing endpoint mass. Panels 10e and 10f show the corresponding transition: the covariance discrepancy decreases as the path approaches the wall, while rare-region coverage increases. Beyond the wall, coverage continues to grow, but the discrepancy to the population turns upward. Thus, the same path first repairs variation lost by the fitted model and then promotes exploration beyond the population level. Structured coverage versus entropy-matched noise. The spiral experiment shows that IGA directs probability toward underrepresented regions. To determine whether this behavior could be reproduced by simply adding noise, Figure 11 compares IGA with an entropy-matched Gaussian-convolution baseline applied to a multimodal DDIM model [16]. For each IGA setting, the convolution scaleσis selected by bisection so that Q σ = P DDIM ∗N (0,σ 2 I) attains the same representation-space von Neumann entropy. Despite matching entropy, the two methods distribute their additional mass differently. Gaussian convolution broadens every mode approximately isotropically, producing increasingly diffuse clouds around the original modal centers. IGA instead selectively fills the underrepresented regions between neighboring modes. Asλ increases, these inter-mode regions form a coherent ring while the original modes remain visible. The entropy increase produced by IGA therefore reflects structure-aware redistribution rather than an undirected increase in noise. Population-level confirmation in a controlled mixture. Figure 12 provides a complementary view using an eight-component mixture with known population weights. The base distribution overweights its 22 most frequent components and underrepresents the remaining modes, resulting in lower entropy than the population. Increasingλinitially corrects this imbalance: at the calibrated wall, the guided distribution approximately recovers both the population entropy and its component probabilities. Beyond the wall, the component probabilities become more uniform than those of the population. Panel C of Figure 12 makes the change in regime explicit. Along the below-wall portion of the path, the population KL decreases as IGA repairs the component imbalance. After the wall is crossed, entropy continues to increase while KL turns upward. The path therefore first approaches the population through diversity repair and subsequently departs from it through deliberate extrapolation. Together, these controlled experiments show that IGA restores underrepresented population structure below the entropy wall and enters an extrapolative regime beyond it. They further show that the increase in diversity arises from selective redistribution toward underrepresented regions rather than isotropic perturbation. We next examine whether the same progression appears in pretrained diffusion models on real-world image benchmarks. 6.2.2 Real-World Image Distribution Benchmarks Having established the repair-to-extrapolation transition in controlled settings, we next ask whether the same progression appears in pretrained diffusion models on real-world image distributions. We evaluate sampling-time IGA on unconditional CelebA-HQ and class-conditional ImageNet generation. Benchmark settings. On CelebA-HQ [22], we guide the pretrainedgoogle/ddpm-ema-celebahq-256 DDPM [4] at 256×256, using deterministic DDIM sampling [16] for 100 steps. On ImageNet, we use the standard ImageNet-100 subset introduced by Tian et al. [23], consisting of their fixed 100-class subset of ILSVRC-2012 [24]. We guide the class-conditionalfacebook/DiT-XL-2-256model [25] for 50 DDIM steps, using classifier-free guidance [26] at scale 2.0. The coefficient on the IGA score correction is fixed to one, as prescribed by the sampling-time construction. We vary only the entropy multiplierλ. Each point along the reported path therefore corresponds to a different entropy-regularized target, rather than to a stronger or weaker application of the same guidance field. The DDIM sampler implements the corresponding unit-scale plug-in correction at each denoising step. Diversity deficit and wall crossing. Because empirical spectral entropy depends on sample size, Figure 4 compares real and generated distributions using matched numbers of samples. On both datasets, the base model remains below the corresponding empirical data wall throughout the evaluated sample-size range. Increasingλprogressively closes this deficit, reaches the wall at an intermediate point, and crosses it for larger values. The tested path therefore spans three interpretable regimes: a diversity-deficient base model, below-wall repair, and beyond-wall imagination. The ImageNet path across the wall. Figure 13 summarizes the progression on ImageNet. The matched samples in Figure 13a show the transition from the base model through below-wall repair and into beyond- wall extrapolation. The FID–entropy phase portrait in Figure 13b shows the corresponding distributional trend: FID initially decreases as the entropy deficit is repaired and turns after the target approaches and crosses the empirical wall. To examine how the additional entropy is obtained, we define the cumulative spectral-occupancy ratio T λ (r) = log P d i=r v ⊤ i S λ v i P d i=r v ⊤ i S data v i , whereS λ andS data denote the generated and data covariance matrices in DINOv2 feature space, respectively, and the data-covariance eigenvectorsv i are ordered from dominant to rare. Negative values indicate an occupancy deficit relative to the data, whereas positive values indicate excess occupancy. Figure 13c shows 23 Base modelBelow the wallAt the wallBeyond the wall (a) Qualitative transition across the entropy wall. 8101214 Exponential von Neumann entropy 6.0 6.5 7.0 7.5 8.0 8.5 9.0 FID diversity repairspectral extrapolation λ=0 λ=0.36 λ=2 λ=3 (b) FID–entropy phase portrait. 32128256384512 ImageNet feature directions 0 0.36 1.2 2 3 λ −0.31 deficit 0.00 match +0.31 excess Spectral occupancy (c) Spectral occupancy. Figure 13: IGA across the ImageNet entropy wall. (a) Matched samples along the IGA target path as λincreases. (b) FID initially decreases as entropy approaches the empirical wall and turns beyond it. (c) Cumulative spectral occupancy relative to the data, across feature directions ordered from dominant to rare. that increasingλprogressively closes the deficit across underrepresented directions and produces excess occupancy after the wall is crossed. IGA therefore gains entropy by allocating more probability to directions that the base generator covers insufficiently. Repair below the wall. Figure 14 shows a consistent initial repair regime on both benchmarks. At the beginning of the below-wall path, spectral diversity and recall increase while the reported feature-space distances decrease relative to the base model. IGA therefore recovers variation missing from the pretrained generator while improving its agreement with the data under both DINOv2 and Inception-v3 representations. This behavior is consistent with the repair result of Theorem 6. As the target approaches the empirical wall, entropy and recall continue to increase, while the different distances attain their minima at nearby but nonidentical values ofλ. This is expected: Inception-v3 and DINOv2 encode different properties of image distributions and need not identify the same target as closest to the data. Imagination beyond the wall. Onceλmoves the target beyond the wall, entropy and recall continue to increase, but the feature-space distances no longer decrease uniformly. Their turning points depend on the dataset, metric, and evaluation representation. This should not be interpreted as a direct measurement of declining perceptual image quality. FID, KID, FD, and KD measure distributional departure from the data in particular feature spaces. Beyond the wall, their increase instead indicates that the generated distribution is moving farther from the data reference while occupying additional feature directions. 24 131415161718 Exponential von Neumann entropy 30.0 30.5 31.0 31.5 32.0 32.5 FID Inception-v3 Base Model Guided (a) CelebA-HQ: FID. 131415161718 Exponential von Neumann entropy 27 28 29 30 31 KID Inception-v3 × 10 3 Base Model Guided (b) CelebA-HQ: KID. 891011121314 Exponential von Neumann entropy 6.0 6.5 7.0 7.5 8.0 8.5 9.0 FID Inception-v3 Base Model Guided (c) ImageNet: FID. 891011121314 Exponential von Neumann entropy 0.5 1.0 1.5 2.0 2.5 3.0 KID Inception-v3 × 10 3 Base Model Guided (d) ImageNet: KID. Figure 14: Independent Inception-v3 evaluation along the IGA target path. FID and KID are measured in Inception-v3 feature space, independently of the DINOv2 representation used to define spectral diversity and the empirical entropy wall. On both CelebA-HQ and ImageNet, the initial below-wall portion of the path improves distributional agreement with the data while diversity increases. At larger values ofλ, the behavior becomes metric- and dataset-dependent, with distributional distances eventually flattening or turning as the target enters the extrapolative regime. Corresponding DINOv2-space FD and KD curves are reported in Appendix H. The two benchmarks therefore exhibit the same overall progression: IGA first repairs a measurable diversity deficit and then enters a different statistical regime after crossing the wall. The central result is not a single optimal value ofλ, but an interpretable target path whose meaning changes from distributional repair to deliberate spectral extrapolation. Reference diversity-guidance methods. Table 1 includes CADS [10] and SPARKE [12] as reference points rather than like-for-like baselines. CADS perturbs the conditioning signal and is therefore reported only on class-conditional ImageNet, while the evaluated SPARKE configuration uses joint batch guidance and produces coupled samples. Neither method defines its operating point relative to the data entropy or distinguishes below-wall repair from beyond-wall extrapolation. In contrast, IGA traces a wall-calibrated family of target distributions and, once its potential is estimated and frozen, applies the same guidance 25 Table 1: Comparison with diversity-guidance methods. Distributional distances, coverage, and spectral diversity on CelebA-HQ and ImageNet. Shaded rows trace the IGA target path asλincreases. Bold indicates the lowest distributional distance or highest recall within each dataset block. Inception-v3DINOv2Coverage & diversity MethodVendi KID×10 3 ↓ FID↓ KD×10 2 ↓FD↓Recall↑H 0 CelebA-HQ Base12.530.9832.8012.331158.70.6112.528 SPARKE [12]23.050.31454.7730.294792.50.4793.134 IGA, λ = 0.513.429.5331.7912.018152.50.6272.597 IGA, λ = 114.328.7431.0211.876151.00.6492.657 IGA, λ = 215.727.5030.0812.049156.40.6852.755 IGA, λ = 317.227.0629.6612.616169.80.7132.843 IGA, λ = 418.626.7129.8613.272192.10.7322.923 ImageNet Base7.73.1079.004.595118.30.5982.040 CADS [10]13.11.2606.054.518123.30.6452.571 SPARKE [12]7.92.9127.894.593115.20.4702.061 IGA, λ = 0.368.52.3177.514.572114.30.6362.141 IGA, λ = 1.210.50.7535.924.550123.00.6962.349 IGA, λ = 212.20.6176.374.546143.50.7272.498 IGA, λ = 313.91.4468.134.548170.70.7602.633 independently to each sampling trajectory. The table therefore provides numerical context under common evaluation metrics rather than a comparison of identical objectives or guarantees. Across both benchmarks, the empirical picture is consistent. The pretrained model begins below the entropy wall; below-wall values ofλrepair part of this deficit while increasing coverage and reducing distributional distances; and larger values cross the wall, where additional coverage is accompanied by a representation- dependent departure from the data distribution. IGA therefore exposes an interpretable target path from diversity repair to controlled imagination. 6.3 Numerical Application of Training-Time IGA The main empirical focus of this paper is sampling-time IGA, which can be applied to a pretrained generator without retraining. Nevertheless, the same distribution-level regularization principle also extends naturally to model training. Section 3.3 formulates training-time IGA as(16), where the model is trained to balance fidelity to the empirical data distribution with the spectral entropy of its generated distribution. The corresponding MNIST results are reported in Appendix H.2. We evaluate this training-time realization using a GAN on MNIST. Following Proposition 3, the entropy reward is added to the adversarial objective through the joint-adversary formulation in(18). For each generated minibatch, we compute the spectral adversary at its closed-form best response and hold it fixed during the generator update. By Proposition 7, this frozen payoff gives the exact gradient of the minibatch entropy at the refresh point. Implementation details. We use a convolutional GAN with a 64-dimensional latent and batch size 128, trained for 20 epochs with Adam using learning rate 2×10 −4 and (β 1 ,β 2 ) = (0.5,0.999) for both generator and discriminator. We use the non-saturating logistic generator objective and one discriminator update per generator update. The IGA representation is the unit-normalized 64-dimensional embedding of a frozen MNIST classifier, while an architecturally distinct frozen classifier with a 96-dimensional embedding is used for independent evaluation. We set ε = 0.05 and report means and standard errors over five random seeds. 26 The baseline GAN exhibits a noticeable imbalance in generated digit frequencies despite being trained on the nearly balanced MNIST distribution. Figure 20 in Appendix H.2 shows that moderate IGA regularization redistributes probability mass toward digit classes underrepresented by the baseline generator. At the best intermediate settings, the total variation distance between the generated and empirical class distributions decreases by 37.1%, while the Fr ́echet distance measured in the feature space of a separate evaluator network decreases by 40.1%. The improvement in both metrics indicates that the effect is not limited to the class- frequency statistic. Their nonmonotone dependence on the IGA multiplier also illustrates the tradeoff between the GAN fidelity objective and the distribution-level entropy reward. 6.4 IGA Application to Prompt-Conditioned Generative Models We finally test sampling-time IGA on two text-to-image models: Stable Diffusion XL (SDXL) [27] and PixArt-Σ [28]. We usestable-diffusion-xl-base-1.0at 768×768 andPixArt-Sigma-XL-2-1024-MSat 1024×1024, with deterministic DDIM sampling for 50 steps. The IGA score correction is fixed at unit scale, and onlyλis varied. Figure 9 shows the resulting path, usingλ∈0,4,8for SDXL andλ∈0,10,20for PixArt-Σ. Within each model, the initial noise seeds are matched across the sweep. Increasingλleads to larger changes in garment shape, volume, material, and color while remaining consistent with the prompt. Figures 5–8 compare vanilla and IGA SDXL across fashion, architecture, underwater painting, and throne design. The base samples tend to stay near familiar forms, whereas IGA produces sculptural garments, curved and stacked towers, underwater scenes with divers and vehicles, and more varied throne structures. Across these examples, the main changes are in shape, structure, and scene composition rather than only color or texture. Additional qualitative results for PixArt-Σ are provided in Appendix B. 7 Conclusion and Discussion Generative modeling is typically formulated as distributional imitation, i.e., the ultimate goal is to generate fresh samples from the underlying distribution of real training samples. However, as shown in [6], such an approach can empirically lead to a model that generates high-quality samples while remaining systematically less diverse than the target real distribution. Our work introduces Imaginative Generative AI (IGA), a distribution-level framework that incorporates spectral entropy as an explicit and controllable diversity component of the generative modeling objective. The real data distribution’s spectral entropy establishes an Entropy Wall in the application of IGA: below this wall, IGA entropy regularization repairs diversity lost during training while remaining compatible with the diversity of the data; beyond the wall, the generated distribution intentionally attains greater representation-relative spectral diversity than the real data. We note that the IGA regularization principle can be applied to both the training of a generative model and post-hoc sampling from a pretrained model. Therefore, IGA provides a general framework for diversity regularization and imaginative generation. Beginning with improving the imitation regime, the approach first counteracts spectral-diversity deficits and encourages the recovery of variation underrepresented by the learned generator. Upon reaching the Entropy Wall, additional regularization transitions into a controlled extrapolative regime, balancing increased representation-relative diversity with closeness to the reference distribution. During sampling, once the IGA guidance potential is fixed, the resulting target enables independent and identically distributed generation, eliminating the need for an interacting batch. Our numerical results support the application of IGA for both diversity repair and imaginative data generation. Pretrained diffusion models demonstrate a measurable entropy deficit compared with matched real-data samples; moderate IGA guidance addresses this deficit, enhancing diversity and, in several cases, distributional fidelity. Stronger guidance crosses the Entropy Wall and generates structured variation beyond the data reference level, including qualitatively novel and imaginative changes in large text-to-image models. These findings indicate that diversity enhancement does not need to be treated as an architecture- specific heuristic or as an uncontrolled deviation from quality. Instead, IGA offers a general regularization 27 framework for systematically transitioning from imitation, through diversity repair, to controlled imaginative extrapolation. The notion of imagination in IGA is intentionally representation-relative: exceeding the Entropy Wall means exceeding the spectral diversity of the data in a specified embedding space, rather than satisfying a representation-independent notion of creativity or novelty. Consequently, the choice of representation, the reference distribution, and the fidelity discrepancy remain important modeling decisions. Subject to these choices, the Entropy Wall provides an explicit and measurable boundary between improving imitation and deliberately moving beyond it, making the transition from imitation to imagination mathematically well-defined and controllable. References [1]Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014. [2]Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014. [3]Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, volume 32, pages 11895–11907, 2019. [4]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851, 2020. [5]Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. [6]Farzan Farnia, Mohammad Jalali, and Azim Ospanov. Exposing diversity bias in deep generative models: Statistical origins and correction of diversity error. arXiv preprint arXiv:2602.14682, 2026. [7]Dan Friedman and Adji Bousso Dieng. The Vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023. [8] Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. An information-theoretic evaluation of generative models in learning multi-modal distributions. In Advances in Neural Information Processing Systems, volume 36, pages 9931–9943, 2023. [9]Azim Ospanov, Jingwei Zhang, Mohammad Jalali, Xuenan Cao, Andrej Bogdanov, and Farzan Farnia. Towards a scalable reference-free evaluation of generative models. In Advances in Neural Information Processing Systems, volume 37, pages 120892–120927, 2024. [10]Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through condition-annealed sampling. In International Conference on Learning Representations, 2024. [11]Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi S. Jaakkola. Particle guidance: non-I.I.D. diverse sampling with diffusion models. In International Conference on Learning Representations, 2024. [12]Mohammad Jalali, Haoyu Lei, Amin Gohari, and Farzan Farnia. SPARKE: Scalable prompt-aware diversity and novelty guidance in diffusion models via RKE score. In Advances in Neural Information Processing Systems, volume 38, pages 119943–119980, 2025. 28 [13]Francis Bach. Information theory with kernel methods. IEEE Transactions on Information Theory, 69(2):752–775, 2023. [14]Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, volume 29, pages 271–279, 2016. [15]Martin Arjovsky, Soumith Chintala, and L ́eon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR, 2017. [16]Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. [17]Maxime Oquab, Timoth ́e Darcet, Th ́eo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herv ́e J ́egou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. [18]Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, volume 20, pages 1177–1184, 2007. [19]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017. [20]Miko laj Bi ́nkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. [21]Tuomas Kynk ̈a ̈anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32, 2019. [22]Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. [23]Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, 2020. [24]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015. [25]William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. [26]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. [27]Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ̈uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024. [28]Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision (ECCV), pages 74–91, 2024. 29 [29]Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, volume 29, 2016. [30]Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018. [31]Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119. Association for Computational Linguistics, 2016. [32]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. [33] Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi. Artificial hivemind: The open-ended homogeneity of language models (and beyond). In Advances in Neural Information Processing Systems, volume 38, 2025. Datasets and Benchmarks Track. [34]Mischa Dombrowski, Weitong Zhang, Sarah Cechnicka, Hadrien Reynaud, and Bernhard Kainz. Image generation diversity issues and how to tame them. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3029–3039, 2025. [35]Jingwei Zhang, Cheuk Ting Li, and Farzan Farnia. An interpretable evaluation of entropy-based novelty of generative models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 59148–59172. PMLR, 2024. [36]Mohammad Jalali, Azim Ospanov, Amin Gohari, and Farzan Farnia. Conditional Vendi score: Prompt- aware diversity evaluation for text-guided generative AI models. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. [37]Azim Ospanov, Mohammad Jalali, and Farzan Farnia. Scendi score: Prompt-aware diversity evaluation via schur complement of CLIP embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16927–16937, 2025. [38]Azim Ospanov and Farzan Farnia. Do Vendi scores converge with finite samples? truncated Vendi score for finite-sample convergence guarantees. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, volume 286 of Proceedings of Machine Learning Research, pages 3272–3299. PMLR, 2025. [39]Quan Nguyen and Adji Bousso Dieng. Quality-weighted Vendi scores and their application to diverse experimental design. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 37667–37682. PMLR, 2024. [40] Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10844–10853, 2024. [41]Reyhane Askari Hemmat, Melissa Hall, Alicia Sun, Candace Ross, Michal Drozdzal, and Adriana Romero-Soriano. Improving geo-diversity of generated images with contextualized Vendi score guidance. In Computer Vision – ECCV 2024, volume 15145 of Lecture Notes in Computer Science, pages 213–229. Springer, 2025. [42]Ankit Yadav, Arpit Garg, Ta Duc Huy, and Lingqiao Liu. STRIDE: Training-free diversity guidance via PCA-directed feature perturbation in single-step diffusion models. arXiv preprint arXiv:2605.11494, 2026. 30 [43]Michael Kirchhof, James Thornton, Louis B ́ethune, Pierre Ablin, Eugene Ndiaye, and Marco Cuturi. Shielded diffusion: Generating novel and diverse images using sparse repellency. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30911–30942. PMLR, 2025. [44] Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. CAN: Creative adversarial networks, generating “art” by learning about styles and deviating from style norms. In Proceedings of the Eighth International Conference on Computational Creativity, pages 96–103, 2017. [45] Songwei Ge, Vedanuj Goswami, C. Lawrence Zitnick, and Devi Parikh. Creative sketch generation. In International Conference on Learning Representations, 2021. [46] Elad Richardson, Kfir Goldberg, Yuval Alaluf, and Daniel Cohen-Or. ConceptLab: Creative concept generation using VLM-guided diffusion prior constraints. ACM Transactions on Graphics, 43(3):1–14, 2024. [47]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048–6058, 2023. [48] Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton Ferrer. Generating high fidelity data from low-density regions using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11492–11501, 2022. [49]Jack Lu, Ryan Teehan, and Mengye Ren. ProCreate, don’t reproduce! propulsive energy diffusion for creative generation. In Computer Vision – ECCV 2024, volume 15118 of Lecture Notes in Computer Science, pages 397–414. Springer, 2025. [50]Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278–1286. PMLR, 2014. 31 A Related Work Diversity in generative models. Diversity loss is a persistent problem across generative modeling. In generative adversarial networks (GANs), mode collapse and limited effective support motivated minibatch discrimination and support-size diagnostics [29,30]. Language models likewise tend toward generic, repetitive, or homogeneous outputs, motivating diversity-aware objectives and decoding strategies [31–33]. More recently, R ́enyi Kernel Entropy (RKE) evaluations have shown that modern generators can produce high-quality samples while still missing modes [8]. Related deficits have been documented in image diffusion models, together with a systematic gap between real and generated diversity for which finite-sample entropy underestimation is one statistical source [34,6]. These findings motivate IGA’s premise: diversity should be specified as a property of the target distribution rather than left as a by-product of distribution fitting. Measuring novelty and diversity. Reference-free measures assess variation within a distribution through similarities among its samples. The Vendi Score uses the von Neumann entropy of a normalized kernel matrix [7], while RKE provides a tractable order-two counterpart with mode-count interpretations [8]. Kernel- based Entropic Novelty compares which modes are more strongly expressed than in a reference distribution [35]. Conditional Vendi and Scendi extend diversity evaluation to prompt-conditioned generators [36,37], while scalable and truncated variants address computational cost and finite-sample estimation [9,38]. A complementary line folds sample quality directly into the diversity score itself, yielding quality-weighted Vendi scores [39]. IGA moves this spectral perspective from post-hoc evaluation into the generative objective itself. Population-data entropy then defines an entropy wall that separates recovery of lost diversity from deliberate extrapolation beyond the data. Promoting novelty and diversity. Existing interventions typically specialize either training or generation. At training time, diversity has been promoted through reinforcement learning with explicit diversity rewards [40] and through diversity-aware diffusion modules [34]. At inference, CADS anneals noise in the conditioning signal [10]; c-VSG and SPARKE guide generation using contextualized Vendi and conditional RKE, respectively [41, 12]; and STRIDE perturbs intermediate features in distilled one- and few-step generators [42]. Particle Guidance instead evolves an interacting set under a pairwise diversity potential and is explicitly non-i.i.d. [11], while SPELL repels trajectories from protected, concurrent, or previously generated images [43]. IGA instead provides a single distribution-level regularizer with both training- and sampling-time realizations. It does not define diversity through the active batch or generation history: during sampling, its potential is estimated beforehand, held fixed, and applied independently to each initialized trajectory, yielding i.i.d. samples from the approximated IGA target. The entropy wall additionally identifies whether regularization repairs diversity lost during learning or intentionally moves beyond the population data. IGA thus unifies diversity control across end-to-end learning and post-hoc sampling while providing a principled transition from imitation to extrapolation. Creative generation. Creative generation has been pursued through deviation from learned styles, novel composition, rare-region sampling, and repulsion from exemplars. Creative Adversarial Networks depart from established artistic styles [44]; DoodlerGAN recombines object parts into unseen sketches [45]; and ConceptLab searches for new category members [46]. For diffusion models, evidence of training-data replication sharpens the distinction between creativity and reproduction [47]. Low-density sampling explores rare regions of the learned distribution [48], whereas ProCreate pushes generations away from reference images [49]. IGA instead gives creativity a distributional interpretation: crossing the entropy wall produces a target whose representation-relative spectral diversity exceeds that of the population data, while the fidelity term controls departure from the reference. The same definition governs both training and sampling. 32 B Additional Qualitative Results We provide additional qualitative comparisons on PixArt-Σ. Unless stated otherwise, IGA usesλ= 20. Across prompts, the vanilla model often concentrates on a narrow set of familiar forms or compositions, while IGA produces broader structural and semantic variation. This is particularly visible in the underwater example, where vanilla PixArt repeatedly generates very similar coral-reef scenes, whereas IGA explores substantially different subjects and layouts while retaining the requested rendering style. A throne for a fantasy film, full object visible, neutral studio background, production design render Vanilla PixArtIGA PixArt Figure 15: Fantasy throne design with PixArt-Σ. Vanilla PixArt (left) and IGA withλ= 20 (right). IGA produces broader variation in silhouette and structure. 33 A gouache painting of an underwater scene Vanilla PixArtIGA PixArt Figure 16: Underwater scenes with PixArt-Σ. Vanilla PixArt (left) largely collapses to the same coral-reef scene across seeds. IGA withλ= 40 (right) produces substantially broader variation in subjects and composition while preserving the gouache style. 34 A skyscraper for a humid coastal city, full building visible in an urban skyline, realistic architectural visualization Vanilla PixArtIGA PixArt Figure 17: Architectural design with PixArt-Σ. Vanilla PixArt (left) and IGA withλ= 20 (right). IGA introduces stronger geometric and structural variation while retaining the skyscraper concept. 35 Vanilla PixArtIGA PixArt A wearable haute couture outfit on a mannequin, neutral studio background Figure 18: Fashion design with PixArt-Σ. Vanilla PixArt (left) and IGA withλ= 20 (right). IGA produces broader variation in garment silhouette, volume, material, and color, while remaining consistent with the wearable haute-couture prompt. 36 C Extended Preliminaries and Conventions We begin by stating two conventions that are used throughout the paper. We would like to further clarify that some of the theoretical lemmas and basic statements are also discussed in [6]. Remark 2. For any discrepancyDwe writeD(Q;P ref ) with the optimized distributionQfirst and the referenceP ref second. For symmetric discrepancies this is cosmetic. However, for KL and general Bregman divergences the ordering cannot be generally swapped. In our notation, the KL anchor is alwaysKL(Q∥P ref ), and a Bregman anchor is alwaysD Φ (Q,P ref ). A data-first object such as KL( b P n ∥Q) or D Φ ( b P n ,Q) is a different problem and does not inherit the guarantees below (see also Remark 10 and, for the maximum-likelihood setting where the data-first orientation is forced, Proposition 8). Remark 3.Pis assumed to be the convex ambient class of probability measures onX, used for the convex- analytic theory (concavity of entropy, convex duality, Bregman projection).P gen is the possibly nonconvex class realizable by a fixed architecture, used for training. Convexity, strong duality, and Pythagorean statements are proved onPand never silently transferred toP gen ; a trained or guided model reaches the ambient optimum only in an approximation-theoretic sense. Assumption 2 (Finite-dimensional embedding). Unless stated otherwise,φ:X → R d with∥φ(x)∥ 2 = 1 is measurable, and Σ Q ∈ R d×d with Σ Q ⪰ 0 andTr(Σ Q ) = 1. When weak continuity ofQ7→Σ Q is invoked, we further assume X is Polish and φ is bounded and continuous. Remark 4 (Use of entropy symbolH). When a result holds for either entropy functional we letHdenote a fixed but arbitrary choice fromH 0 ,H ε (as in(C ρ ),(P λ ), Theorem 4, and Definition 1);His never used to mix the two within a single statement. When differentiability of the entropy is needed we specialize to the smoothedH ε (Section 5), and when defining the population wall in its main statistical interpretation we use the unsmoothed H 0 (Section 4). Lemma 3 controls the gap between the two. Lemma 2 (Concavity ofH 0 andH ε inQ). IfQ 7→Σ Q is affine, thenQ 7→ H 0 (Q) andQ 7→ H ε (Q) are concave. Proof. Note thatS 7→− Tr(S logS) is a concave functional of density matrices (unit-trace PSD matrices), and a concave function composed with an affine map is concave. This proves concavity of H 0 . ForH ε , the mapQ 7→ S ε Q = (1− ε)Σ Q +ε 1 d I d is also affine inQ, and therefore the same argument applies. Lemma 3 (Bounding Smoothing Gap of Spectral Entropy). Letd≥2, let Σ be ad× ddensity matrix, and set S = (1− ε)Σ + ε 1 d I d for ε∈ [0, 1]. Writing τ = ε(1− 1 d ) and h 2 (t) =−t logt− (1− t) log(1− t), H(S)−H(Σ) ≤ τ log(d− 1) + h 2 (τ ). Since H ε (Q) =H(S ε Q ) and H 0 (Q) =H(Σ Q ), this implies the following for every Q: H ε (Q)− H 0 (Q) ≤ τ log(d− 1) + h 2 (τ ). In particular,|ρ ⋆,H ε − ρ ⋆ |admits the same bound. The bound is achieved when Σ is rank one: then 1 2 ∥S− Σ∥ 1 = τ exactly, S has eigenvalues 1− τ, τ d−1 ,..., τ d−1 , and H(S)−H(Σ) = τ log(d− 1) + h 2 (τ ). Proof. First, we bound the trace distance betweenSand Σ. SinceS−Σ =ε( 1 d I d −Σ), we have 1 2 ∥S− Σ∥ 1 = ε· 1 2 1 d I d − Σ 1 . If Σ has eigenvaluesp i , then 1 2 1 d I d − Σ 1 = 1 2 P i |p i − 1 d |, which over the probability simplex is maximized at a vertex p = e j , giving 1− 1 d . Hence 1 2 ∥S− Σ∥ 1 ≤ ε(1− 1 d ) = τ. 37 Next, we apply the Fannes–Audenaert inequality, showing that ford × ddensity matricesA,Bwith 1 2 ∥A− B∥ 1 ≤ t≤ 1− 1 d , |H(A)−H(B)|≤ t log(d− 1) + h 2 (t). The right-hand side is non-decreasing inton [0,1− 1 d ]: its derivativelog(d−1) +log 1−t t is non-negative there, vanishing only att= 1− 1 d . Applying the inequality att= 1 2 ∥S− Σ∥ 1 ≤ τ ≤ 1− 1 d (hered≥2 ensures log(d−1)≥0) gives the claim withA=SandB= Σ. Finally, the consequence forH ε ,H 0 follows by the stated identities, and the wall bound follows by takingQ=P data . For rank-one Σ =v ⊤ , the eigenvalues of 1 d I d −Σ are 1 d −1 (once) and 1 d (with multiplicityd−1), so 1 2 ∥S− Σ∥ 1 =τexactly; the spectrum ofSis then 1− τ, τ d−1 ,..., τ d−1 , whenceH(Σ) = 0 andH(S) = τ log(d− 1) + h 2 (τ ), so equality holds. The last ingredient is an elementary identity for exponential tilts. For a fixed bounded rewardG, it identifies the minimizer of the KL-anchored linear objective in closed form; the sampling-time tilt of Theorem 1 is its self-consistent analogue, in which G is the entropy energy evaluated at the optimum itself. Lemma 4 (Elementary Gibbs identity). LetPbe a probability law andGmeasurable withZ G =E P [e G ]<∞; define P G (dx) = Z −1 G e G(x) P (dx). For every Q≪ P , KL(Q∥P )− E Q [G] = KL(Q∥P G )− logZ G ,(22) so P G is the unique minimizer over Q≪ P of the left-hand side whenever it is finite. Proof. On Q≪ P we have log(dQ/dP G ) = log(dQ/dP )− G + logZ G . Integrating against Q gives KL(Q∥P G ) = KL(Q∥P )− E Q [G] + logZ G , which is(22). The left-hand side equalsKL(Q∥P G )− logZ G , minimized (overQ≪ P, equivalentlyQ≪ P G since the two are equivalent) uniquely at Q = P G , where KL = 0. DThe Constrained–Penalized Correspondence, the Spectral Game, and Training This appendix proves the results of Section 3. We first state and prove the constrained–penalized correspon- dence invoked in Section 3.1, together with the monotone regularization path; Appendix D.1 then proves the spectral min–max representation, and Appendix D.2 the training-time instantiations of Section 3.3. Theorem 3 (Constrained–penalized correspondence). Suppose that: (i) P is a nonempty compact convex subset of a locally convex Hausdorff space of finite signed measures; (i) Q7→D(Q;P ref ) is proper, convex, and lower semicontinuous on P ; (i) Q7→Σ Q is affine and continuous, so thatH ∈H 0 ,H ε is concave (Lemma 2) and upper semicontinuous; (iv) (Slater condition) there exists ̄ Q∈P with D( ̄ Q;P ref ) <∞ and H( ̄ Q) > ρ. Then the following hold. (a)Attainment. The feasible setP ∩H ≥ ρis nonempty and compact, and both the constrained minimum in (C ρ ) and the inner minimum defining F λ are attained. (b) Strong duality. min Q∈P H(Q)≥ρ D(Q;P ref ) = max λ≥0 n min Q∈P D(Q;P ref )− λH(Q) + λρ o .(23) 38 (c)Optimal multiplier. There existsλ ⋆ ≥0 such that every solutionQ ⋆ of(C ρ )minimizesF λ ⋆ and satisfies the complementary-slackness identity λ ⋆ ρ− H(Q ⋆ ) = 0. Theorem 3 justifies replacing(C ρ )by(P λ )at the specific multiplierλ ⋆ dual toρ; it does not claim that everyλ≥0 corresponds to a user-chosen target level. The proof proceeds through the value function of the constrained problem, after recording an unconditional min–max identity (Proposition 4). Proposition 4 (Exact primal min–max identity). For any feasible set Q and arbitrary functionals J,H , inf Q∈Q H(Q)≥ρ J (Q) = inf Q∈Q sup λ≥0 J (Q) + λ(ρ− H(Q)) . Proof.For fixedQ,sup λ≥0 J(Q) +λ(ρ−H(Q))equalsJ(Q) ifH(Q)≥ ρ(the coefficient ofλis nonpositive, so the supremum is atλ= 0) and +∞ifH(Q)< ρ(the coefficient is positive, so the expression diverges as λ→∞). Taking the infimum overQ∈Qretains only feasibleQand reproduces the constrained value. Proof of Theorem 3.By hypotheses (i)–(i),J(Q) =D(Q;P ref ) is proper, convex, l.s.c. on the convex setP, and H ∈H 0 ,H ε is concave (Lemma 2) and u.s.c., so the feasible set P ∩H ≥ ρ is convex and closed. First, we verify feasibility and attainment. By Slater (iv) the feasible set contains ̄ Q, hence is nonempty; it is a closed subset of the compactP(i), hence compact. A l.s.c. function attains its minimum on a nonempty compact set, so the constrained minimum in(C ρ )is attained; likewise, for eachλ≥0,Q7→ J(Q)−λH(Q) is l.s.c. on the compactPand attains its minimum, soF λ has a minimizer and the displayedmin’s in(23)are justified. The remainder of the proof runs through the value function of the constrained problem, v(r) = inf Q∈P J (Q) : H(Q)≥ r , withv(r) = +∞if no feasibleQexists; by the previous paragraph,v(ρ) is finite and attained. Note also that J , being l.s.c. on the compact P, is bounded below on P, so v(r)≥ inf P J >−∞ for every r. Next, we establish the two structural properties ofv. The value function is nondecreasing: ifr 1 ≤ r 2 thenH ≥ r 2 ⊆ H ≥ r 1 , so the infimum over the smaller set is at least as large, i.e.v(r 1 )≤ v(r 2 ). The value function is also convex. To see this, fixr 1 ,r 2 ∈ R,θ ∈[0,1], andη >0, and choose feasible Q i (that is,H(Q i )≥ r i ) withJ(Q i )≤ v(r i ) +η. The mixtureQ θ =θQ 1 + (1− θ)Q 2 ∈ Pthen satisfies H(Q θ )≥ θr 1 + (1− θ)r 2 by concavity of H, and, by convexity of J , J (Q θ )≤ θJ (Q 1 ) + (1− θ)J (Q 2 )≤ θv(r 1 ) + (1− θ)v(r 2 ) + η. Hence v(θr 1 + (1− θ)r 2 )≤ θv(r 1 ) + (1− θ)v(r 2 ) + η, and letting η ↓ 0 gives convexity. Then, we show thatvis subdifferentiable at the target levelρ. By the Slater condition there is ̄ Q∈P withJ( ̄ Q)<∞andH( ̄ Q)> ρ; hencev(r)≤ J( ̄ Q)<∞for allr ≤ H( ̄ Q). Combined with the lower bound above,vis finite on (−∞,H( ̄ Q )], an interval whose interior containsρ. A finite convex function on an open interval is subdifferentiable at every interior point; pick λ ⋆ ∈ ∂v(ρ). Since v is nondecreasing, λ ⋆ ≥ 0. With the multiplierλ ⋆ in hand, we can prove strong duality. The subgradient inequality gives, for every Q∈P, J (Q)≥ v(H(Q))≥ v(ρ) + λ ⋆ H(Q)− ρ , henceJ(Q)− λ ⋆ H(Q)≥ v(ρ)− λ ⋆ ρ; taking the infimum overQ∈Pyieldsinf Q∈P J − λ ⋆ H+λ ⋆ ρ≥ v(ρ). Conversely, weak duality holds: for any λ≥ 0 and any feasible Q (that is, H(Q)≥ ρ), J (Q)≥ J (Q)− λ H(Q)− ρ ≥ inf Q ′ ∈P J − λH + λρ, 39 and taking the infimum over feasibleQgivesv(ρ)≥ sup λ≥0 inf Q ′ J − λH +λρ. The two inequalities together yield (23), with the outer supremum attained at λ ⋆ . Finally, we establish complementary slackness. LetQ ⋆ solve(C ρ ). Feasibility givesH(Q ⋆ )≥ ρ, and by strong duality J (Q ⋆ ) = v(ρ) = inf Q J − λ ⋆ H + λ ⋆ ρ. On the one hand,J(Q ⋆ )−λ ⋆ H(Q ⋆ )≥ inf Q J−λ ⋆ H=v(ρ)−λ ⋆ ρ. On the other hand, feasibility andλ ⋆ ≥0 giveJ(Q ⋆ )−λ ⋆ H(Q ⋆ )≤ J(Q ⋆ )−λ ⋆ ρ=v(ρ)−λ ⋆ ρ. The two bounds match, forcingλ ⋆ H (Q ⋆ )−ρ = 0 and J (Q ⋆ )− λ ⋆ H(Q ⋆ ) = inf Q J − λ ⋆ H, i.e. Q ⋆ minimizes F λ ⋆ . Remark 5 (Which hypotheses do what). The hypotheses of Theorem 3 play three separable roles. Duality. The value-function argument shows that strong duality and the existence of an optimal multiplierλ ⋆ ∈ ∂v(ρ) require only convexity ofP, convex l.s.c.J, concaveH, the Slater condition (iv), and the value functionv being proper and finite nearρ, i.e.v(ρ)>−∞(equivalently,Jbounded below on the feasible set), in addition tov(ρ)<∞from Slater. Convexity and Slater alone do not guaranteev(ρ)>−∞: ifJis unbounded below onPthenv ≡−∞, no finite subgradient exists, and the duality statement is vacuous. In the compact setting of Theorem 3 this cannot happen, because an l.s.c.Jon the compactPis bounded below; the properness caveat matters only in the noncompact variant. Attainment. The compactness in (i), with l.s.c.Jand u.s.c.H, is otherwise used only to guarantee attainment of the constrained minimum, of the inner minima definingF λ , and hence of the displayedmin/ maxin(23). Noncompact classes. In settings whereQ≪ P θ is convex but not compact (Section 5), existence and attainment are instead obtained by the direct method of Theorem 1, where J = KL(·∥P θ )≥ 0 is automatically bounded below. Proposition 5 (Saddle representation). Under the hypotheses of Theorem 3, withQ ⋆ a solution of(C ρ ) and λ ⋆ the optimal multiplier, (Q ⋆ ,λ ⋆ ) is a saddle point of L(Q,λ) = J (Q) + λ(ρ− H(Q)) on P × [0,∞): L(Q ⋆ ,λ)≤L(Q ⋆ ,λ ⋆ )≤L(Q,λ ⋆ ) ∀Q∈P, λ≥ 0, and consequently inf Q∈P sup λ≥0 L(Q,λ) = sup λ≥0 inf Q∈P L(Q,λ). Proof. SinceQ ⋆ minimizesF λ ⋆ =J − λ ⋆ HoverP(Theorem 3) andL(·,λ ⋆ ) =F λ ⋆ (·) +λ ⋆ ρ, the right inequalityL(Q ⋆ ,λ ⋆ )≤L(Q,λ ⋆ ) holds for allQ∈P. For the left inequality,L(Q ⋆ ,λ) =J(Q ⋆ ) +λ(ρ−H(Q ⋆ )) is nonincreasing inλ≥0 becauseρ− H(Q ⋆ )≤0; together withλ ⋆ (ρ− H(Q ⋆ )) = 0 this givesL(Q ⋆ ,λ)≤ L(Q ⋆ ,λ ⋆ ) for allλ ≥0. The equality of the two mixed extrema is the standard consequence of a saddle point. Even without convexity, the multiplier acts as a monotone control on global minimizers; the following statement, summarized in Section 3.1, applies both to the ambient problem and to a nonconvex generator family. Theorem 4 (Monotone regularization path). LetQ∈P,P gen , writeJ(Q) =D(Q;P ref ), and suppose that a minimizerQ λ ∈ arg min Q∈Q F λ (Q) exists for everyλ≥0. Then, for any 0≤ λ 1 < λ 2 and any choices of minimizers Q λ 1 and Q λ 2 , H(Q λ 2 )≥ H(Q λ 1 ), J (Q λ 2 )≥ J (Q λ 1 ). Proof of Theorem 4. Let 0≤ λ 1 < λ 2 and letQ λ 1 ,Q λ 2 be any minimizers ofF λ 1 ,F λ 2 overQ. Abbreviate H i = H(Q λ i ), J i = J (Q λ i ). Optimality of Q λ 1 at λ 1 and of Q λ 2 at λ 2 gives J 1 − λ 1 H 1 ≤ J 2 − λ 1 H 2 , J 2 − λ 2 H 2 ≤ J 1 − λ 2 H 1 . Adding these two inequalities cancelsJ 1 ,J 2 and yields (λ 2 −λ 1 )(H 2 −H 1 )≥0, soH 2 ≥ H 1 . Substituting into the first inequality, rearranged asJ 1 − J 2 ≤ λ 1 (H 1 − H 2 )≤0, givesJ 2 ≥ J 1 . No convexity, differentiability, uniqueness, or path continuity is used; the argument is valid for any selection of minimizers, so the ordering holds even when minimizers are nonunique. 40 Proposition 6 (Every penalized optimizer is a constrained optimizer at its attained level). Under the hypotheses of Theorem 4, set ρ λ := H(Q λ ). Then Q λ ∈ argmin Q∈Q J (Q) : H(Q)≥ ρ λ . Proof. Take anyQ∈QwithH(Q)≥ ρ λ . Penalized optimality givesJ(Q λ )− λH(Q λ )≤ J(Q)− λH(Q), so J (Q λ )≤ J (Q) + λ(H(Q λ )− H(Q)) = J (Q)− λ(H(Q)− ρ λ )≤ J (Q), using λ≥ 0 and H(Q)≥ ρ λ . Remark 6 (Constrained and penalized problems are not interchangeable in general). Outside the convex setting the two problems need not share solutions for a prescribedλorρ. Proposition 6 matches eachQ λ to its own attained levelρ λ , while Theorem 3 recovers a prescribed levelρonly under its convexity and Slater hypotheses. On a nonconvexP gen the saddle representation (Proposition 5) can fail with a positive duality gap; we therefore do not claim per-λ constrained–penalized equivalence for P gen . D.1 The spectral min–max representation: proofs This appendix proves Proposition 1 and the best-response identity quoted in Sections 3.1 and 3.3. The main tool is the Gibbs variational principle for matrix entropy, which we derive from Klein’s inequality; both results are proved in full. Lemma 5 (Klein’s inequality for matrix relative entropy). LetS,Tbed× ddensity matrices withT ≻0. Then Tr S logS− S logT ≥ Tr(S)− Tr(T ) = 0, with equality if and only if S = T . Proof.Write spectral decompositionsS= P i α i u i u ⊤ i andT= P j β j v j v ⊤ j with orthonormal bases (u i ),(v j ), eigenvaluesα i ≥0,β j >0, and setc ij := (u ⊤ i v j ) 2 . The matrix (c ij ) is doubly stochastic: P j c ij =∥u i ∥ 2 = 1 and P i c ij =∥v j ∥ 2 = 1, since each basis is orthonormal. Expanding the traces in these bases, Tr(S logS) = X i α i logα i = X i,j c ij α i logα i ,Tr(S logT ) = X i,j c ij α i logβ j , using P j c ij = 1 for the first identity andu ⊤ i (logT)u i = P j c ij logβ j for the second. The scalar inequality x logx− x logy ≥ x− y, valid forx ≥0,y >0 (with 0log0 = 0; it is the tangent-line inequality for the convex functionx7→ x logxaty), holds with equality if and only ifx=y: forx >0 this is strict convexity, and at x = 0 the inequality reads 0≥−y, strict since y > 0. Applying it termwise, Tr S logS− S logT = X i,j c ij α i logα i − α i logβ j ≥ X i,j c ij (α i − β j ) = X i α i − X j β j = 0, where the last step again uses double stochasticity. If equality holds, then every pair (i,j) withc ij >0 satisfies α i = β j . Fix j and expand v j = P i (u ⊤ i v j )u i ; then Sv j = X i α i (u ⊤ i v j )u i = X i β j (u ⊤ i v j )u i = β j v j , since every indexicontributing a nonzero coefficient hasc ij >0, henceα i =β j . ThusSacts asβ j on each v j , so S = P j β j v j v ⊤ j = T . Conversely S = T gives equality trivially. 41 Lemma 6 (Gibbs variational principle for matrix entropy). For every d× d density matrix S, sup Θ=Θ ⊤ − Tr(SΘ)− log Tr(e −Θ ) =−H(S), where the supremum runs over all symmetricd× dmatrices. The objective is invariant under Θ7→Θ +cI d forc∈ R. IfS ≻0, the supremum is attained exactly at the family Θ =− logS+cI d ,c∈ R; ifSis singular, the supremum is not attained, but is approached along Θ δ =− log(S+δI d ) asδ ↓0, whose objective value is P i α i log(α i + δ)− log(1 + δd) in terms of the eigenvalues α i of S. Proof.First, the invariance: replacing Θ by Θ +cI d changes− Tr(SΘ) by−c Tr(S) =−cand changes − log Tr(e −Θ−cI ) =− log(e −c Tr(e −Θ )) by +c, so the objective is unchanged. Next, the upper bound. For symmetric Θ, letR Θ :=e −Θ / Tr(e −Θ ), a positive-definite density matrix with logR Θ =−Θ−log Tr(e −Θ )I d . Klein’s inequality (Lemma 5) withT=R Θ givesTr(S logS)−Tr(S logR Θ )≥0, which expands to 0≤ Tr(S logS) + Tr(SΘ) + log Tr(e −Θ ), i.e. − Tr(SΘ)− log Tr(e −Θ )≤−H(S), with equality if and only if R Θ = S. Then, attainment. IfS ≻0, the equationR Θ =Shas the solutions Θ =− logS+cI d ,c ∈ R, and no others:R Θ =Sforces−Θ =logS+log Tr(e −Θ )I d . IfSis singular, no symmetric Θ satisfiesR Θ =S, since R Θ ≻0 always; hence the supremum is not attained. Finally, evaluating the objective at Θ δ =− log(S+δI d ) gives − Tr(SΘ δ )− log Tr(e −Θ δ ) = Tr S log(S + δI d ) − log Tr(S + δI d ) = X i α i log(α i + δ)− log(1 + δd), which converges to P i α i logα i =−H(S) asδ ↓0 (the terms withα i = 0 contribute 0· logδ= 0). Hence the supremum equals −H(S) in all cases. Remark 7 (Smoothed versus unsmoothed dual, and concavity as a byproduct). Lemma 6 explains why the min–max form is stated for the smoothed entropy. For the unsmoothedH 0 , the covariance Σ Q can be singular, in which case the supremum is not attained, and the near-maximizers Θ δ =− log(Σ Q +δI d ) have operator norm growing likelog(1/δ): no compact adversary class captures the supremum uniformly over all Q. The spectral floorS ε Q ⪰(ε/d)I d removes both obstructions: it confines the best response to the compact classT ε of Proposition 1 and guarantees attainment. The lemma also yields an independent proof of fact (a) of the organization paragraph: it displays−H(S) as a supremum of affine functions ofS, hence convex, soH is concave, which also yields Lemma 2. Remark 8 (Further readings of the min–max form). Two structural readings of(10)complement the closed-form best response discussed in Section 3.1. Linearization: for fixed Θ, the inner objective depends onQ only through the expectation of the per-sample payoffφ(x) ⊤ Θφ(x), so the distribution-level reward−λH ε (Q) becomes an ordinary expected loss at the cost of oned×dsymmetric adversarial variable; Section 3.3 exploits this directly, where the spectral player joins the discriminator as a second adversary that admits a closed-form best response. Why smoothing: for the unsmoothed entropy the supremum runs over an unbounded matrix class and is not attained at rank-deficient covariances (Remark 7), whereas the spectral floorε 1 d I d confines the adversary to the compact classT ε and guarantees attainment; the interchange in part (i) then follows from Sion’s minimax theorem, whose compactness requirement is satisfied by the Θ-side alone. Proof of Proposition 1. First, part (i). Since S ε Q ⪰ (ε/d)I d ≻ 0, Lemma 6 gives −H ε (Q) =−H(S ε Q ) = sup Θ=Θ ⊤ − Tr(S ε Q Θ)− log Tr(e −Θ ) , 42 attained exactly at the family− logS ε Q +cI d . ImposingTr(Θ) = 0 pins the constant atc= 1 d Tr (logS ε Q ), which is the matrix Θ ⋆ (Q) of the statement; it is the unique traceless maximizer, since attainment forces membership in the family. For the operator-norm bound, the eigenvalues ofS ε Q lie in [ε/d,1− ε(1− 1 d )]⊆ [ε/d,1], so the eigenvaluesℓ 1 ,...,ℓ d of− logS ε Q lie in [0, log(d/ε)]; centering replacesℓ i byℓ i − ̄ ℓwith ̄ ℓ= 1 d P j ℓ j ∈ [0, log(d/ε)], so each centered eigenvalue satisfies|ℓ i − ̄ ℓ|≤ max j ℓ j − min j ℓ j ≤ log(d/ε). Hence Θ ⋆ (Q)∈ T ε , and restricting the supremum toT ε preserves both the value and the attainment, upgradingsup to max. It remains to pass to the per-sample form: for traceless Θ, Tr(S ε Q Θ) = (1− ε) Tr(Σ Q Θ) + ε d Tr(Θ) = (1− ε) E X∼Q φ(X) ⊤ Θφ(X) , using Tr(Σ Q Θ) = E Q [Tr(φ ⊤ Θ)] = E Q [φ ⊤ Θφ]. This is (9). Next, part (i). Multiplying(9)byλ≥0 preserves the maximum (forλ= 0 both sides of the resulting identity vanish identically onT ε , since−λH ε (Q) = 0 and the Θ-dependent terms carry the factorλ), and addingD(Q;P ref ), which does not depend on Θ, gives the pointwise identity(10). Since the two sides agree as functions ofQ, the problem(P λ )of minimizing the left side overPis the two-player game of minimizing the right side, as claimed. Finally, part (i).T ε is convex and compact. For fixed Θ,Q 7→ A λ (Q,Θ) is convex and l.s.c.: the expectation term is affine inQ(and weakly continuous when the continuity clause of Assumption 2 is in force,φbeing bounded and continuous), andD(·;P ref ) is convex l.s.c. by hypothesis. For fixedQ, Θ7→ A λ (Q,Θ) is concave and continuous: the expectation term is linear in Θ, and−λ log Tr(e −Θ ) is concave, since Lemma 6 exhibits Θ7→− log Tr(e −Θ ) as an infimum overSof affine functions of Θ (namely − log Tr(e −Θ ) =inf S Tr(SΘ)−H(S), the dual reading of the same variational identity). Sion’s minimax theorem requires compactness of only one side, here the Θ-sideT ε , so no compactness ofPis needed, and the interchange holds as stated. Lemma 7 (Best response and the entropy energy). For every distribution Q and every x∈X , λ(1− ε)φ(x) ⊤ Θ ⋆ (Q)φ(x) = λG ε Q (x)− c Q , wherec Q := λ(1−ε) d Tr − logS ε Q ∈ 0, λ(1− ε)log(d/ε) is a constant independent ofx. In particular, the per-sample payoff of the best-responding spectral adversary equals the entropy energy(8), up to an additive constant independent of x. Proof. By definition Θ ⋆ (Q) =− logS ε Q + 1 d Tr(logS ε Q )I d , so λ(1− ε)φ(x) ⊤ Θ ⋆ (Q)φ(x) = λ(1− ε)φ(x) ⊤ − logS ε Q φ(x) + λ(1− ε) d Tr logS ε Q ∥φ(x)∥ 2 2 . The first term isλG ε Q (x) by(8), and since∥φ(x)∥ 2 = 1 the second term is the constant−c Q . The range ofc Q follows because the eigenvalues of − logS ε Q lie in [0, log(d/ε)], so their average lies in the same interval. D.2 Training-time instantiations: adversarial and maximum-likelihood models This appendix proves the results invoked in Section 3.3: the joint-adversary identity for adversarially trained generators (Proposition 3), the exactness of gradients computed through the frozen spectral adversary (Proposition 7), and the likelihood–KL–ELBO relations underlying the maximum-likelihood instantiation (Proposition 8). Proof of Proposition 3. Fix Q∈Q. By the critic representation (17) and the entropy dual (9), D(Q; b P n )− λH ε (Q) = sup D∈D c A Q (D) + max Θ∈T ε B Q (Θ), 43 whereA Q (D) =E b P n [u(D)]− E Q [v(D)] andB Q (Θ) =−λ(1− ε)E Q [φ ⊤ Θφ]− λ log Tr(e −Θ ); here the passage from(9)to−λH ε (Q) =max Θ B Q (Θ) is part (i) of the proof of Proposition 1 (multiplication byλ≥0, with the degenerate caseλ= 0 givingB Q ≡0 =−λH ε (Q)). SinceDand Θ range over independent sets and the two objectives share no variable, the suprema add: sup D∈D c A Q (D) + max Θ∈T ε B Q (Θ) = sup D∈D c max Θ∈T ε A Q (D) + B Q (Θ) , and the right-hand side is the inner expression of(18). The identity therefore holds pointwise inQ, and taking the infimum over an arbitrary classQ, convex or not, preserves it. Attainment of the Θ-maximum at Θ ⋆ (Q) is part (i) of Proposition 1. No convexity of Q was used and no minimax interchange was performed. Proposition 7 (Frozen spectral adversary yields exact entropy gradients). LetZ ∼ P Z on a latent spaceZ, let g θ :Z →X be measurable for each θ∈ R p and differentiable in θ at P Z -a.e. z, let φ be differentiable on an open set containing the relevant ranges with∥φ∥ 2 ≡1, and letQ θ denote the law ofg θ (Z). Suppose there are a neighborhood U of θ 0 and L∈ L 1 (P Z ) such that ∇ θ φ(g θ (z))φ(g θ (z)) ⊤ ≤ L(z)for all θ∈ U and P Z -a.e. z. Then θ7→ H ε (Q θ ) is differentiable at θ 0 and ∇ θ H ε (Q θ ) θ 0 = (1− ε) E Z h ∇ θ φ g θ (Z) ⊤ Θ ⋆ (Q θ 0 )φ g θ (Z) θ 0 i , i.e. the exact gradient of the entropy coincides with the gradient of the expected per-sample payoff in which the spectral adversary is frozen at its best response Θ ⋆ (Q θ 0 ). Proof.WriteS(θ) :=S ε Q θ = (1− ε)E Z φ(g θ (Z))φ(g θ (Z)) ⊤ +ε 1 d I d . First, the domination hypothesis justifies differentiation under the expectation:θ 7→ S(θ) is differentiable atθ 0 with∂ θ k S(θ 0 ) = (1− ε)E Z ∂ θ k φ(g θ (Z))φ(g θ (Z)) ⊤ θ 0 . Next, as in the proof of Lemma 1, the matrix entropyH is Fr ́echet differentiable at every positive-definite matrix withDH(S)[B] =− Tr B(logS+I d ) for symmetricB, and this applies at S(θ 0 )⪰ (ε/d)I d ≻ 0. By the chain rule, ∂ θ k H ε (Q θ ) θ 0 =− Tr ∂ θ k S(θ 0 ) logS(θ 0 ) + I d . Then, the trace term drops out: since∥φ∥ 2 2 ≡1, we haveTr ∂ θ k S(θ 0 ) = (1− ε)∂ θ k E Z ∥φ(g θ (Z))∥ 2 2 = ∂ θ k (1− ε) = 0, so the I d contribution vanishes and ∂ θ k H ε (Q θ ) θ 0 = Tr ∂ θ k S(θ 0 ) − logS(θ 0 ) = (1− ε) E Z h ∂ θ k φ ⊤ − logS(θ 0 ) φ θ 0 i , where the matrix− logS(θ 0 ) is held fixed under the derivative. Finally, replacing− logS(θ 0 ) by Θ ⋆ (Q θ 0 ) = − logS(θ 0 ) + 1 d Tr(logS(θ 0 ))I d changes the per-sample payoff by a multiple of∥φ∥ 2 2 ≡1, whoseθ-gradient is zero; hence the displayed identity. In the language of the min–max game(10), this is an envelope (Danskin- type) statement: the inner maximum is attained at the unique Θ ⋆ (Q θ 0 ), and differentiating the value equals differentiating at the frozen maximizer. The direct computation above proves the identity without invoking any general envelope theorem. Maximum-likelihood training and VAEs. Deep maximum-likelihood models fit an explicit densityq θ by minimizing the empirical negative log-likelihood, which is the per-sample empirical proxy for the data- first divergence: up to an additive constant independent ofθ,E P data [− logq θ (X)] =KL(P data ∥Q θ ) +const (Proposition 8 below). This orientation is forced at training time: the model-first quantityKL(Q θ ∥ b P n ) is 44 typically infinite for a continuously supported model against an atomic empirical reference, whereas the likelihood is finite and estimable sample by sample. The IGA-regularized maximum-likelihood objective is min θ E X∼ b P n − logq θ (X) − λH ε (Q θ ),(24) and when the likelihood is intractable, as in variational autoencoders, the negative evidence lower bound takes its place [2, 50]: min θ,η E X∼ b P n −ELBO(X;θ,η) − λH ε (Q θ ).(25) Proposition 8 records the exact relation between the two: the negative ELBO exceeds the negative log- likelihood by the encoder-posterior gapKL(r η (·| x)∥p θ (·| x))≥0, aλ-independent quantity, so(25)is(24) plus a nonnegative gap that only the encoder parametersηtighten; the same reading applies to diffusion models trained through variational bounds [4]. We highlight two features of this combination. First, the entropy regularizer is likelihood-free: evaluating H ε (Q θ ) requires only samples from the decoder, never density values, so it applies to any latent-variable model whose sampler is differentiable, alongside a fidelity term that does require likelihoods. Second, the orientation caveat of Remark 10 applies: the repair guarantee of Theorem 6 is proved for base-anchored objectives and does not transfer to this data-first geometry, while the monotone path of Theorem 4, which is orientation- and convexity-agnostic, continues to describe the global minimizers of(24)and(25)asλgrows. When the reference is instead a smooth law, such as a pretrained teacher in fine-tuning rather than b P n , the model-first KL anchor becomes directly usable. Proposition 8 (Likelihood, data-first KL, and the ELBO). Letνbe aσ-finite measure onXand let each model law Q θ have ν-density q θ . (i)Likelihood is data-first KL. SupposeP data ≪ νwith densityp 0 andE P data | logp 0 (X)| < ∞. Then, for every θ, E P data − logq θ (X) = KL(P data ∥Q θ ) + h ν (P data ), whereh ν (P data ) :=−E P data logp 0 (X) is finite and independent ofθ, and the two sides are finite or +∞ together. (i) Structure of the data-first divergence. For fixedP, the mapQ7→ KL(P∥Q) is convex onP, and ifXis Polish it is weakly lower semicontinuous. Consequently Theorem 4 applies to the objectives(24)and(25) whenever global minimizers exist, while Theorem 6, proved for the base-anchored orientation, does not transfer (Remark 10). (i)ELBO gap. Letq θ (x) = R p θ (x | z)p Z (dz) be a latent-variable model andr η (· | x) an encoder with r η (·| x)≪ p θ (·| x), where p θ (·| x) is the model posterior. Then, for every x with q θ (x)∈ (0,∞), −ELBO(x;θ,η) =− logq θ (x) + KL r η (·| x) p θ (·| x) ≥ − logq θ (x), with equality if and only if the encoder matches the model posterior at x. Proof. First, part (i). Decompose− logq θ =log(p 0 /q θ )− logp 0 on the setp 0 >0, which carries full P data -mass. The second term integrates toh ν (P data ), finite by hypothesis. For the first term, setr:=q θ /p 0 on p 0 >0; the positive part oflogrisP data -integrable, sincelogr ≤ r−1 givesE P data [(logr) + ]≤ E P data [r] = R p 0 >0 q θ dν ≤ 1, soE P data [− logr] =E P data [log(p 0 /q θ )] is well defined in (−∞,+∞]. IfP data ≪ Q θ , then dP data /dQ θ =p 0 /q θ holdsP data -a.s. andE P data [log(p 0 /q θ )] =KL(P data ∥Q θ ) by definition. IfP data ̸≪ Q θ , pickAwithQ θ (A) = 0< P data (A); thenq θ = 0ν-a.e. onA, solog(p 0 /q θ ) = +∞on a set of positive P data -measure and, the negative part being integrable,E P data [log(p 0 /q θ )] = +∞=KL(P data ∥Q θ ) under the extended-value convention. In both cases the displayed identity holds, with both sides finite or +∞together since h ν (P data ) is finite. 45 Next, part (i). For convexity, fixQ 0 ,Q 1 andθ ∈(0,1), and letν ′ be aσ-finite measure dominating P,Q 0 , andQ 1 (for instanceP+Q 0 +Q 1 ), with densitiesp,q 0 ,q 1 ; the mixtureQ θ = (1− θ)Q 0 +θQ 1 has densityq θ = (1− θ)q 0 +θq 1 . For fixedxwithp(x)>0, the mapq 7→ p(x)log(p(x)/q) is convex inq >0 (as− logis convex), and extends convexly toq ≥0 with value +∞atq= 0; composing with the affine θ 7→ q θ (x) and integratingdν ′ preserves convexity, givingKL(P∥Q θ )≤(1− θ)KL(P∥Q 0 ) +θKL(P∥Q 1 ). For lower semicontinuity, we invoke the Donsker–Varadhan variational formula, a standard fact: for probability measures on a Polish space, KL(P∥Q) =sup f∈C b (X ) E P [f ]− log E Q [e f ] . For each fixedf ∈ C b (X), the mapQ7→ E P [f]− log E Q [e f ] is weakly continuous:e f is bounded continuous, soQ7→ E Q [e f ] is weakly continuous with values in the compact interval [e −∥f∥ ∞ ,e ∥f∥ ∞ ]⊂(0,∞), on which logis continuous. A supremum of weakly continuous functions is weakly lower semicontinuous, which proves the claim. The consequences for Theorems 4 and 6 are as stated: the former uses only the existence of global minimizers and is agnostic to orientation and convexity, while the latter’s three-point argument differentiates the Bregman divergence in its first argument and is unavailable in the data-first orientation. Finally, part (i). Write the ELBO with encoder r η : ELBO(x;θ,η) = E Z∼r η (·|x) logp θ (x| Z) + logp Z (Z)− logr η (Z | x) , withlogp Z understood as the density of the prior with respect to the latent reference measure. By Bayes’ rule, p θ (z | x) =p θ (x | z)p Z (z)/q θ (x) forq θ (x)∈(0,∞), sologp θ (x | z) +logp Z (z) =logp θ (z | x) +logq θ (x), and substituting, ELBO(x;θ,η) = logq θ (x)− E Z∼r η (·|x) h log r η (Z | x) p θ (Z | x) i = logq θ (x)− KL r η (·| x) p θ (·| x) . Negating gives the display; nonnegativity of KL gives the inequality, with equality if and only ifr η (·| x) = p θ (·| x). E Proofs for Section 4 This appendix proves the results of Section 4: the bias and consistency of the empirical entropy wall in Theorem 5, the sub-wall repair guarantee, and the wall-crossing statement (Proposition 9). Theorem 5 (Bias and consistency of the empirical wall). LetX 1 ,X 2 ,...be i.i.d. fromP data , letφ:X → R d satisfy ∥φ(x)∥ 2 = 1, and let b P N = 1 N P N i=1 δ X i . Then, for either H ∈H 0 ,H ε and every N ≥ 1: (i) downward bias: EH( b P N )≤ H(P data ); (i) monotonicity in the sample size: EH( b P N +1 )≥ EH( b P N ); (i) consistency: EH( b P N )−→ H(P data ) as N →∞. Proof of Theorem 5. We prove the three parts in turn. First, for part (i), recall from Lemma 2 thatH ∈H 0 ,H ε is concave in its distribution argument through the affine map Q7→ Σ Q . The empirical covariance is unbiased: E Σ b P N = 1 N N X i=1 E φ(X i )φ(X i ) ⊤ = Σ P data . 46 Jensen’s inequality for the concave map Σ7→ H then gives EH( b P N )≤ H E Σ b P N = H(P data ). Next, for part (i), we use a leave-one-out averaging identity. FixN+ 1 samples and, forj= 1,...,N+ 1, let b P (−j) N = 1 N P i̸=j δ X i . Each index appears in exactly N of the N + 1 leave-one-out measures, so b P N +1 = 1 N + 1 N +1 X j=1 b P (−j) N ,equivalentlyΣ b P N+1 = 1 N + 1 N +1 X j=1 Σ b P (−j) N . Concavity of H gives, pathwise, H( b P N +1 )≥ 1 N + 1 N +1 X j=1 H b P (−j) N . Each b P (−j) N has the same distribution as b P N , so taking expectations yields EH( b P N +1 )≥ EH( b P N ). Finally, for part (i), note that since∥φ(x)∥ 2 = 1, the summandsφ(X i )φ(X i ) ⊤ are i.i.d. bounded random matrices with mean Σ P data , so by the (matrix) strong law of large numbers Σ b P N → Σ P data almost surely. Both H 0 andH ε are continuous functions of the covariance matrix on the compact set of density matrices and bounded in [0, logd], soH( b P N )→ H(P data ) a.s.; dominated convergence then givesEH( b P N )→ H(P data ). Remark 9 (Interpretation and experimental consequence). Theorem 5 upgrades the informal assumption of downward diversity bias to a theorem for the empirical law; it does not by itself establish that a trained generatorQ 0 satisfiesH 0 (Q 0 )< ρ ⋆ , which remains a separate empirical claim, consistent with reported spectral deficits in modern generators. Becausebρ ⋆ =H 0 ( b P N ) is downward biased, and a finite generated batch inherits the same downward bias when estimatingH 0 (Q λ ), wall-crossing plots should use matched sample sizes, repeated subsampling, or a bias-aware estimator. Theorem 6 (Population discrepancy improves up to the wall). Let Φ be Fr ́echet differentiable and strictly con- vex on an open convex set containingP, with Bregman divergenceD Φ (P,Q) = Φ(P)−Φ(Q)−⟨∇Φ(Q),P − Q⟩, and letQ λ ∈ argmin Q∈P D Φ (Q,Q 0 )− λH(Q)withHconcave andPconvex. IfP data ∈ PandH(Q λ )≤ H(P data ), then D Φ (P data ,Q λ ) + D Φ (Q λ ,Q 0 )≤ D Φ (P data ,Q 0 ), and hence in particular D Φ (P data ,Q λ )≤ D Φ (P data ,Q 0 ). Proof.By Proposition 6 applied toJ(Q) =D Φ (Q,Q 0 ) over the convexP,Q λ ∈ arg minD Φ (Q,Q 0 ) :Q∈ P, H(Q)≥ ρ λ withρ λ =H(Q λ ). The feasible setC=P∩H ≥ ρ λ is convex (Hconcave). The below-wall hypothesis H(P data )≥ ρ λ and P data ∈P give P data ∈C. BecauseQ 7→ D Φ (Q,Q 0 ) is convex and differentiable in its first argument (with∇ Q D Φ (Q,Q 0 ) = ∇Φ(Q)−∇Φ(Q 0 )) andCis convex, first-order optimality of the minimizerQ λ overCgives, for every feasible P , ⟨∇Φ(Q λ )−∇Φ(Q 0 ),P − Q λ ⟩≥ 0. Setting P = P data ∈C and using the three-point Bregman identity D Φ (P data ,Q 0 ) = D Φ (P data ,Q λ ) + D Φ (Q λ ,Q 0 ) +⟨∇Φ(Q λ )−∇Φ(Q 0 ),P data − Q λ ⟩, together with nonnegativity of the inner-product term, yieldsD Φ (P data ,Q 0 )≥ D Φ (P data ,Q λ ) +D Φ (Q λ ,Q 0 ). Nonnegativity of D Φ (Q λ ,Q 0 ) gives the second inequality. 47 Remark 10 (What the repair theorem does not cover). Theorem 6 concerns the base-anchored orientation D Φ (Q,Q 0 ) (optimized law first) and does not transfer to a data-first objectiveD Φ ( b P n ,Q): the gradient of a Bregman divergence in its second argument involves the Hessian of Φ and is not∇Φ(Q)−∇Φ( b P n ), so the three-point identity no longer collapses the cross term. In particular, the maximum-likelihood instantiation of Section 3.3, whose fidelity is the data-first KL by Proposition 8(i), inherits the monotone path of Theorem 4 but not the repair guarantee. Two clean options remain: (i) the base-anchored objectiveD Φ (Q,Q 0 )− λH(Q), especially withQ 0 =P θ . (i) a symmetric Hilbertian discrepancy such as squared MMD, for which the orientation is immaterial. Proposition 9 (Local wall crossing). LetPbe convex,P data ∈ P, and suppose there existsR ∈ Pwith H 0 (R)> H 0 (P data ). PutQ t = (1− t)P data +tRfort∈(0,1]. IfD(Q t ;P data )→0 ast↓0, then for every δ > 0 there exists Q∈P with D(Q;P data )≤ δ and H 0 (Q) > H 0 (P data ). Proof.By convexity ofP,Q t ∈ P. By concavity ofH 0 (Lemma 2),H 0 (Q t )≥(1− t)H 0 (P data ) +tH 0 (R); sinceH 0 (R)> H 0 (P data ), the right-hand side equalsH 0 (P data ) +t(H 0 (R)− H 0 (P data ))> H 0 (P data ) for all t∈(0,1], soH 0 (Q t )> H 0 (P data ) (plain concavity suffices; no strictness is used). By hypothesis chooset δ >0 with D(Q t δ ;P data )≤ δ; then Q = Q t δ satisfies both requirements. F Proofs for Section 3.2 and Section 5 This appendix proves the sampling-time results in the following order. We first record a tightness lemma for KL sublevel sets, which drives the existence argument in Theorem 1, and compute the first variation of the smoothed entropy (Lemma 1). We then record the formal stationarity condition behind the density- ratio representations (Remark 11), prove the tilt characterization (Proposition 2), and derive Theorem 1 by combining the existence argument with that proposition. The remainder of the appendix treats the propagation of the tilt through the noising process and the endpoint guarantees. Lemma 8 (Relative-entropy sublevel tightness). LetPbe a probability measure on a Polish space andC ≥0. The sublevel setQ:KL(Q∥P)≤ Cis tight. Concretely, for any measurableAwith 0< P(A)<1 and any Q with KL(Q∥P )≤ C, Q(A) ≤ C + 1 log(1/P (A)) .(26) Proof. By the data-processing inequality applied to the binary partition A,A c , KL(Q∥P )≥ d 2 Q(A)∥P (A) , d 2 (q∥p) = q log q p + (1− q) log 1−q 1−p , whered 2 is the binary KL, with the usual conventions andd 2 (q∥p) = +∞ifp∈0,1whileq /∈0,1. Write p=P(A)∈(0,1) andq=Q(A). Since 1− p≤1 giveslog 1−q 1−p ≥ log(1− q), andt logt≥−e −1 on [0,1], the terms q logq and (1− q) log 1−q 1−p together contribute at least −2/e≥−1; hence d 2 (q∥p) = q log(1/p) + q logq + (1− q) log 1−q 1−p ≥ q log(1/p)− 1; hence Q(A) log(1/P (A))≤ KL(Q∥P ) + 1≤ C + 1, which is (26). For tightness, fixη >0. Since every probability measure on a Polish space is tight, there is a compact KwithP(K c ) as small as desired; takingP(K c ) small enough that (C+ 1)/ log(1/P(K c ))≤ ηyields sup KL(Q∥P )≤C Q(K c )≤ η. Proof of Lemma 1.WriteA ν := R φ (x)φ(x) ⊤ dν(x), a symmetric matrix (finite sinceφis bounded andνis finite). SinceQ7→Σ Q is affine, Σ Q+tν = Σ Q +tA ν and henceS ε Q+tν =S ε Q +t(1− ε)A ν for alltfor which 48 Q+tνis a probability measure. The matrix entropyH(S) =− Tr(S logS) is Fr ́echet differentiable at every positive definiteS, with derivativeDH(S)[B] =− Tr B(logS+I d ) for symmetricB; this applies atS=S ε Q becauseS ε Q ⪰(ε/d)I d ≻0, and the perturbed matricesS ε Q +t(1− ε)A ν remain in a compact neighborhood of positive definite matrices for small t. By the chain rule along the affine path, d dt H ε (Q + tν) t=0 + =−(1− ε) Tr A ν (logS ε Q + I d ) = (1− ε) Z φ(x) ⊤ − logS ε Q φ(x)dν(x) − (1− ε) Z ∥φ(x)∥ 2 2 dν(x). Since ∥φ(x)∥ 2 2 = 1, the last integral equals ν(X ) = 0, and the first term is R G ε Q dν by definition (8). We note that the normalization∥φ∥ 2 = 1 makes the trace term of the derivative drop out, soG ε Q represents the first variation of the entropy functional: the first variation ofH ε atQ, along any admissible mass-preserving perturbation, integrates G ε Q against the perturbation. Remark 11 (Bregman stationarity). For a general Bregman anchor, a formal first-order condition explains how the geometry of Φ converts the entropy first variation into a displacement of the law: an interior optimizer Q ⋆ of (11) satisfies ∇Φ(Q ⋆ )−∇Φ(P θ ) = λg Q ⋆ + c, whereg Q =G ε Q is the first variation ofH ε (Lemma 1) andcis the scalar multiplier of the unit-mass constraint. We do not rely on this identity: in the KL geometry, the derivation below obtains the density ratio directly from the first-variation computation, with no interiority hypothesis. We now prove the tilt characterization of Section 3.2. Proof of Proposition 2.Throughout, writeP θ =Q∈P:Q≪ P θ , a convex set, letFbe extended by +∞ offP θ , and recall from the statement thatFattains a finite minimum onP θ ; letQ ⋆ be any minimizer and q ⋆ =dQ ⋆ /dP θ . We first prove uniqueness, then mutual absolute continuity, then derive the tilt from the first-variation computation, and finally bound the density ratio. Uniqueness. On its finite domain the KL term is strictly convex inQ, and−λH ε is convex by Lemma 2. Hence F is strictly convex where finite, and its minimizer is unique. Mutual absolute continuity. We haveQ ⋆ ≪ P θ withKL(Q ⋆ ∥P θ )< ∞by finiteness of the minimum; it remains to proveq ⋆ >0P θ -a.s. Suppose instead thatq ⋆ = 0 on a measurable setAwithP θ (A)>0. Let R=P θ (·| A) andQ t = (1− t)Q ⋆ +tR∈P θ fort∈(0,1). Exactly as in the corresponding computation for the KL term (splitting the integral overA, where the density ofQ t ist1 A /P θ (A), andA c , where it is scaled by 1− t), KL(Q t ∥P θ )− KL(Q ⋆ ∥P θ ) = t logt + O(t). The remaining entropy term ofFchanges by onlyO(t). Indeed,S ε Q t − S ε Q ⋆ =t(1− ε)(Σ R −Σ Q ⋆ ) has norm O(t) (with constants depending only on∥φ∥ 2 = 1), andS 7→ − Tr(S logS) is Lipschitz on the compact spectral range [ε/d,1], its derivative−(logS+I) being bounded in operator norm bylog(d/ε) + 1 there. Therefore F (Q t )− F (Q ⋆ ) = t logt + O(t) < 0for small t > 0, since t logt→ 0 − dominates O(t); this contradicts optimality. Thus q ⋆ > 0 P θ -a.s. and Q ⋆ ∼ P θ . First variation and the tilt. For bounded measurablehwithE Q ⋆ h= 0, setdQ t = (1 +th)dQ ⋆ , a valid probability law for|t|≤1/(1 +∥h∥ ∞ ), and letν=hdQ ⋆ , a finite signed measure withν(X) = 0. By Lemma 1 49 and the identityg Q =G ε Q , the entropy term has derivative d dt H ε (Q t )| t=0 = R g Q ⋆ hdQ ⋆ . The KL term has derivative d dt KL(Q t ∥P θ ) t=0 = Z (logq ⋆ + 1)hdQ ⋆ = Z logq ⋆ hdQ ⋆ , where the +1 term vanishes sinceE Q ⋆ h= 0; differentiation under the integral is justified by dominated convergence, ashis bounded and R q ⋆ | logq ⋆ |dP θ < ∞fromKL(Q ⋆ ∥P θ )< ∞together with the uniform bound t logt≥−e −1 . First-order optimality d dt F (Q t )| t=0 = 0 for all such h therefore gives Z logq ⋆ − λg Q ⋆ hdQ ⋆ = 0 for all boundedhwithE Q ⋆ h= 0, so the integrand in parentheses isQ ⋆ -a.s. (and, by mutual absolute continuity,P θ -a.s.) equal to a constant. Exponentiating and normalizing, with the constant absorbed into the normalizer, yields (14). Boundedness of the ratio. The eigenvalues ofS ε Q lie in [ε/d,1], so − logS ε Q op ≤ log(d/ε) and 0≤ λg Q ≤ λ(1− ε)log(d/ε) uniformly overQandx. The exponent in(14)is therefore uniformly bounded, so the normalizer lies in (0,∞) and the density ratio is bounded above and below by positive constants. Proof of Theorem 1. We first establish existence, then obtain the remaining claims from Proposition 2. Existence. Work onP θ =Q:Q≪ P θ , a convex set, and extendFby +∞off it. The infimum is finite: F(P θ ) =−λH ε (P θ )∈[−λ logd,0], whileF ≥−λ logd >−∞termwise. Let (Q n ) be a minimizing sequence. Since 0≤ H ε (Q)≤ logd, boundedness ofF(Q n ) impliessup n KL(Q n ∥P θ ) =:C < ∞. By Lemma 8, the KL sublevel setQ:KL(Q∥P θ )≤ Cis tight, so (Q n ) is tight. By Prokhorov’s theorem a subsequence converges weakly to someQ ⋆ . Sinceφis bounded and continuous,Q 7→Σ Q ,S ε Q are weakly continuous; becauseS ε Q ⪰(ε/d)I d ≻0 uniformly,S 7→− Tr(S logS) is continuous on the relevant compact spectral range, soQ7→ H ε (Q) is weakly continuous. The mapQ7→ KL(Q∥P θ ) is weakly l.s.c.: by the Donsker–Varadhan formula,KL(Q∥P θ ) =sup f∈C b (X ) E Q [f]− log E P θ [e f ]is a supremum of weakly continuous functions of Q, exactly as in the proof of Proposition 8(i) with the roles of the two arguments exchanged. Therefore F (Q ⋆ )≤ lim inf n F (Q n ), so Q ⋆ attains the infimum; in particular KL(Q ⋆ ∥P θ ) <∞, so Q ⋆ ≪ P θ . Specialization. By the existence step,Fattains a finite minimum onP θ , so Proposition 2 applies: the minimizerQ ⋆ is unique and mutually absolutely continuous withP θ , which proves part (i), and the tilt(14) holds with the total reward R Q ⋆ of (13), proving part (i). Boundedness. The spectral floor gives 0≤ λG ε Q ⋆ ≤ λ(1− ε)log(d/ε). ThusR Q ⋆ is uniformly bounded, so Z=E P θ exp(R Q ⋆ )∈(0,∞) anddQ ⋆ /dP θ is bounded above and below by positive constants; henceQ ⋆ has the same P θ -essential support as P θ , proving part (i). Remark 12 (Scope of the tilt characterization). The characterization(14)is a fixed point:R Q ⋆ depends on Q ⋆ throughS ε Q ⋆ (contrast Lemma 4, where the reward is fixed and the tilt is explicit). The KL anchor only reweights withinsupp(P θ ) and creates no mass whereP θ = 0. Strict convexity proves uniqueness of the target law but does not imply that any particular fixed-point iteration is contractive; convergence of a numerical solver requires a separate argument. Proof of Theorem 2. For measurable A, q ⋆ t (A) = Z K t (A| x 0 )Q ⋆ (dx 0 ) = Z −1 Z K t (A| x 0 )w(x 0 )P θ (dx 0 ). Disintegrate the base joint law of (X 0 ,X t ) asK t (dx t | x 0 )P θ (dx 0 ) =P θ (dx 0 | x t )p t (dx t ). By Fubini’s theorem (applicable since w is bounded, by Theorem 1), q ⋆ t (A) = Z −1 Z A Z w(x 0 )P θ (dx 0 | x t ) p t (dx t ) = Z −1 Z A h t (x t )p t (dx t ), 50 withh t (x t ) =E P θ [w(X 0 )| X t =x t ]. SinceAwas arbitrary,dq ⋆ t /dp t =h t /Z. If both marginals have positive differentiable densities, thenlogq ⋆ t =logp t +logh t − logZ; asZis constant inx t ,∇ logq ⋆ t = ∇ logp t +∇ logh t . Theorem 7 (Exact reverse process). Suppose the forward SDEdX t =f(X t ,t)dt+g(t)dW t (fortincreasing from 0 toT) admits strictly positive differentiable marginal densities and satisfies the standard regularity conditions for time reversal and the probability-flow construction. We use the standard reverse-time convention in which the displayed equations are integrated with decreasingtfromTto 0. Then the reverse-time SDE, initialized at q ⋆ T , dX t = f (X t ,t)− g(t) 2 ∇ logp t (X t ) + u t (X t ) dt + g(t)dW t , has time-zero law exactly Q ⋆ , whereW is a reverse-time Brownian motion. The probability-flow ODE ̇ X t = f (X t ,t)− 1 2 g(t) 2 ∇ logp t (X t ) + u t (X t ) , likewise integrated with decreasingt, has the same one-time marginals. Equivalently, under the forward reparameterizationτ=T − tandY τ =X T−τ , both dynamics run with increasingτand their drifts are the negatives of the displayed drifts evaluated at t = T − τ (the diffusion term is unchanged). Proof. Anderson’s time-reversal theorem, under the stated regularity, gives the reverse-time SDE (integrated with decreasingt) for the process with marginalsq ⋆ t as having driftf − g 2 ∇ logq ⋆ t and initial lawq ⋆ T . Substituting∇ logq ⋆ t =∇ logp t +u t from Theorem 2 yields the stated drift, and the marginals areq ⋆ t for all t, in particularQ ⋆ att= 0. The probability-flow ODE ̇ X t =f − 1 2 g 2 ∇ logq ⋆ t is the deterministic process with identical one-time marginals under theq ⋆ T initialization. Theτ=T −tstatement follows from the chain rule d dτ Y τ =− d dt X t | t=T−τ , which flips the sign of every drift while preserving the (sign-indifferent) diffusion coefficient. If instead one initializes atp T ̸=q ⋆ T , the time-zero law is notQ ⋆ ; the discrepancy is quantified in Theorem 8. Remark 13 (Initialization). Practical samplers initialize fromp T , notq ⋆ T . These coincide only whenh T is constant. Becausedq ⋆ T /dp T =h T /Z, the mismatch isKL(p T ∥q ⋆ T ) =E p T log dp T dq ⋆ T =logZ − E p T logh T (X T ), which enters any rigorous comparison between a deployed sampler andQ ⋆ (Theorem 8). Theorem 7 is exact only with the q ⋆ T initialization. Definition 2 (Plug-in fields). AssumeX ⊆ R D and thatφand the denoiser are differentiable. With guidance scale ω t ≥ 0, eu chain t (x t ) = ω t a J bx 0 (x t ,t) ⊤ ∇ x R Q ⋆ (bx 0 (x t ,t)), whereJ bx 0 is the Jacobian of the denoiser. The direct-injection variant, applicable when clean and noisy states share dimension, is eu dir t (x t ) = ω t ∇ x R Q ⋆ (bx 0 (x t ,t)), which uses a chosen state-space direction rather than the derivative of the composite mapx t 7→ R Q ⋆ (bx 0 (x t ,t)). Remark 14 (Plug-in is uncontrolled). No general equality or one-sided bound relatesu t andeu t :R Q ⋆ is nonlinear, and neither conditional expectation nor differentiation commutes with a point-mass substitution. For the reward R Q ⋆ = λG ε Q ⋆ of (13), ∇ x R Q ⋆ (x) = J φ (x) ⊤ 2λ(1− ε)(− logS ε Q ⋆ )φ(x) , whereJ φ is the Jacobian ofφ(the factor 2 comes from differentiating the quadratic formφ ⊤ Mφwith symmetric M =− logS ε Q ⋆ , which is the x-gradient of λG ε Q ⋆ ). Remark 15 (Discrete updates are approximations). The DDPM/DDIM updates(20)–(21)are algebraically consistent with the corrected noise prediction under the noise–score convention∇ logp t =−ε θ / √ 1− ̄α t , and are discrete implementations inspired by Theorem 2. They do not exactly sampleQ ⋆ even ifeu t =u t ; DDIM adds a further ODE discretization and path-selection approximation. A discretization term must therefore be added to the continuous-time bound of Theorem 8. 51 Theorem 8 (Endpoint KL and TV bounds). Let the exact reverse process have initial lawq ⋆ T , base score s t =∇ logp t , and exact guidanceu t =∇ logh t ; let the deployed process have initial lawπ T , learned scorebs t , and approximate guidance eu t . Put e t = bs t − s t , δ t = eu t − u t . Assume that: (i) both continuous-time processes share the diffusion coefficientg(t)Iwithg(t)>0 on (0,T) (nondegeneracy on the open interval); any endpoint degeneracyg(0) = 0 org(T) = 0 is handled by truncating to [η,T − η], applying the bound there, and letting η ↓ 0, assuming the resulting integral converges; (i) the absolute-continuity and Novikov conditions for Girsanov’s theorem hold on each such subinterval. Then, for the orientation KL( b Q∥Q ⋆ ) between the clean endpoint laws Q ⋆ , b Q (deployed law first), KL( b Q∥Q ⋆ )≤ KL(π T ∥q ⋆ T ) + 1 2 Z T 0 g(t) 2 E b P ∥e t (X t ) + δ t (X t )∥ 2 2 dt, and, by Pinsker’s inequality (again for the orientation KL( b Q∥Q ⋆ )), TV( b Q,Q ⋆ )≤ h 1 2 KL(π T ∥q ⋆ T ) + 1 4 Z T 0 g(t) 2 E b P ∥e t (X t ) + δ t (X t )∥ 2 2 dt i 1/2 . Proof. LetP ⋆ be the path law of the exact reverse process (initial lawq ⋆ T , driftb ⋆ t =f − g 2 (s t +u t )) and b P the path law of the deployed process (initial lawπ T , drift b b t =f−g 2 (bs t +eu t )), both with diffusion coefficient g(t)I. The drift difference is b b t − b ⋆ t =−g(t) 2 (e t +δ t ). First, we decompose the path-space relative entropy in the direction KL( b P∥P ⋆ ) (deployed first) by the chain rule over the initial time T , KL( b P∥P ⋆ ) = KL(π T ∥q ⋆ T ) + E π T KL b P(·| X T )∥ P ⋆ (·| X T ) . Next, we evaluate the conditional term. Conditionally onX T , the two processes share the diffusion coefficient g(t)I, which is nondegenerate on (0,T) (or on each [η,T − η], withη ↓0 afterwards), and differ only in drift, so on that interval the change of measure is absolutely continuous and Girsanov’s theorem applies, giving KL b P(·| X T )∥ P ⋆ (·| X T ) = 1 2 E b P h Z T 0 g(t) −1 ( b b t − b ⋆ t ) 2 dt X T i = 1 2 E b P h Z T 0 g(t) 2 ∥e t + δ t ∥ 2 dt X T i , using g(t) −1 ( b b t − b ⋆ t ) =−g(t)(e t + δ t ). Averaging over X T ∼ π T , we arrive at KL( b P∥P ⋆ ) = KL(π T ∥q ⋆ T ) + 1 2 Z T 0 g(t) 2 E b P ∥e t + δ t ∥ 2 dt. Finally, the clean endpoint laws b Q,Q ⋆ are measurable images (the time-0 coordinate) of the path laws, so the data-processing inequality givesKL( b Q∥Q ⋆ )≤ KL( b P∥P ⋆ ), which is the stated KL bound. Pinsker’s inequality TV(μ,ν)≤ q 1 2 KL(μ∥ν) applied to b Q,Q ⋆ gives the TV bound. The exact-guidance bound is recovered only whenπ T =q ⋆ T ande t ≡0; a separate discretization term is still needed for the implemented DDPM/DDIM sampler (Remark 15). 52 Remark 16 (i.i.d. sampling holds only for fixed guidance). At the population levelQ ⋆ is a single law, so independent exact samplers with a fixed potentialR Q ⋆ produce i.i.d. draws fromQ ⋆ ; this distinguishes IGA from methods that define diversity only through a coupled batch objective. However,R Q ⋆ depends on the unknownQ ⋆ throughS ε Q ⋆ . If a practical algorithm recomputes covariance or entropy gradients from the same batch being generated, each particle’s drift depends on the others: the outputs are exchangeable but not independent. An i.i.d. guarantee requires one of the following: frozen-potential sampling, in which the potential is estimated in a separate stage, frozen, and used to run independent trajectories; independent-pilot estimation, in which the potential is estimated on an independent pilot sample; or a mean-field analysis, invoking a propagation-of-chaos argument when the potential is updated from the active batch. Absent these, finite-batch IGA guidance should be described as an interacting particle system. G Training-Time IGA for Diffusion Models: Proofs and Discussion Diffusion models are trained through variational bounds, which places them in the maximum-likelihood family of Appendix D.2. Attaching−λH ε (Q θ ) directly to the denoising objective requires samples fromQ θ , and hence full reverse rollouts inside the training loop. The framework offers a rollout-free alternative: perform the IGA correction on the data before fitting the denoiser. To this end, we apply Proposition 2 with the reference distribution b P n ; note that the proposition depends onP θ only through its role as the reference measure, and the finite-minimum hypothesis holds automatically since the feasible set is the simplex over the training atoms. This application yields unique weights q ⋆ i = w i P n j=1 w j , w i = exp λG ε Q ⋆ λ (x i ) ,(27) which form a self-consistent softmax over the training set and defineQ ⋆ λ = P n i=1 q ⋆ i δ x i . The weights can be computed as a finite-dimensional convex–concave saddle problem through the spectral dual of Proposition 1, with the spectral adversary and the reweighting playing the detection and response roles described after that proposition. The outcome can be viewed as a distributionally robust reweighting of the dataset, although not a worst-case-loss one (Remark 17, Appendix G). The following proposition shows that training on the reweighted data is justified exactly rather than heuristically: Proposition 10 (IGA training as divergence minimization toward reweighted data). LetP ref be a probability measure, letλ≥0,ε∈(0,1), and letFbe the objective(12)withP ref in place ofP θ . SupposeFattains a finite minimum over Q∈P : Q≪ P ref , at Q ⋆ λ . Then for every Q with F (Q) <∞, F (Q)− F (Q ⋆ λ ) = KL(Q∥Q ⋆ λ ) + λB −H ε (Q,Q ⋆ λ ),(28) whereB −H ε (Q,Q ′ ) :=H ε (Q ′ )− H ε (Q) + R G ε Q ′ d(Q− Q ′ )≥0 is the Bregman divergence of the convex functional −H ε . Every term on the right-hand side of(28)is a divergence betweenQand the IGA-reweighted reference, and both terms vanish exactly atQ=Q ⋆ λ . Therefore, over any generator class, minimizing the IGA objective is equivalent to matching the reweighted law. For a diffusion model, this equivalence justifies weighted denoising score matching, i.e., the standard training loss with clean samples drawn according to the weights q ⋆ in place of uniform weights, which coincides with diffusion training under the data lawQ ⋆ λ . When the variational bound is tight and the generator class is expressive, the minimizers of the weighted bound attain the IGA optimum. In general, the weighted bound controls the data-first divergenceKL(Q ⋆ λ ∥Q θ ), whereas the IGA excess(28)is the model-first sum; this is the standard mass-covering versus mode-seeking asymmetry, stated here in an exact form (Corollary 3, Appendix G, which also records the empirical-versus-population role of the reference). The following proves the results of the diffusion-training paragraph of Section 3.3: the finite-sample IGA reweighting of the data (Corollary 2), the exact decomposition of Proposition 10, its consequence for weighted denoising training (Corollary 3), and the relation to distributionally robust optimization (Remark 17). 53 Corollary 2 (Finite-sample IGA reweighting). Letx 1 ,...,x n be the training samples and b P n = 1 n P n i=1 δ x i . For everyλ ≥0 andε ∈(0,1), the objective of(12)with reference b P n attains a finite minimum over Q ∈ P:Q ≪ b P n , and its unique minimizerQ ⋆ λ = P n i=1 q ⋆ i δ x i has strictly positive weights given by the self-consistent softmax (27). Proof.The mapq 7→ Q q = P i q i δ x i identifiesQ ∈ P:Q ≪ b P n with the simplex ∆ n =q ∈ R n :q ≥ 0, P i q i = 1; if some training points coincide, the identification merges the corresponding atoms and the argument below is unchanged. On ∆ n each term of the objective is finite and continuous:KL(Q q ∥ b P n ) = P i q i log(nq i ), continuous with the convention 0log0 = 0; andH ε (Q q ), the composition of the matrix entropy H, continuous on density matrices, with the affine mapq 7→(1− ε) P i q i φ (x i )φ(x i ) ⊤ + ε d I d . A continuous function on the compact set ∆ n attains its minimum, which is finite, so the hypothesis of Proposition 2 holds with b P n in the role of the reference. That proposition gives uniqueness, mutual absolute continuity (equivalently,q ⋆ i >0 for everyi), and the tilt(14), which on atoms readsnq ⋆ i =w i /ZwithZ= 1 n P j w j and with w i as in (27); cancelling the factors of n gives (27). Two roles of the reference. The corollary solves the IGA problem anchored at the empirical training distribution exactly; the resultingQ ⋆ λ is supported on the training set and is the implicit data law of the weighted training scheme below. When the generated lawQ θ of a continuously supported model is compared against a reference,KL(Q θ ∥ b P n ) = +∞, so the decomposition of Proposition 10 is applied with a population reference, such asP data or a smooth teacher law in fine-tuning, for whichF(Q θ ) is finite for absolutely continuous models. The empirical weights(27)are then the plug-in counterpart of the population tilt: the exponent is a fixed continuous function ofxonce the covarianceS ε Q ⋆ λ is given, and the empirical fixed point estimates exactly this covariance. We do not pursue a finite-sample analysis of this plug-in step here; Theorem 5 describes the behavior of the underlying moment estimates. Proof of Proposition 10. Write Q ⋆ = Q ⋆ λ and let T (x) := λG ε Q ⋆ (x) be the total reward(13), withP ref in the role of the reference, so thatdQ ⋆ /dP ref =e T /ZwithZ= E P ref [e T (X) ] by(14). Since the eigenvalues ofS ε Q lie in [ε/d,1], the exponent is uniformly bounded: 0≤ T ≤ λ (1− ε)log(d/ε). FixQwithF(Q)<∞. Because 0≤ H ε ≤ logd, finiteness ofF(Q) is equivalent to KL(Q∥P ref ) <∞, and in particular Q≪ P ref . Step 1: chain rule for the KL term. The ratiodQ ⋆ /dP ref =e T /Zis bounded above and below by positive constants, so Q≪ P ref if and only if Q≪ Q ⋆ , and P ref -almost surely log dQ dP ref = log dQ dQ ⋆ + T − logZ. The negative part oflog(dQ/dP ref ) isQ-integrable (as always for a log-density ratio:t(logt) − ≤ e −1 for t≥0), andT − logZis bounded, henceQ-integrable; therefore all three expectations below are well defined in (−∞, +∞] and additivity holds: KL(Q∥P ref ) = KL(Q∥Q ⋆ ) + E Q [T ]− logZ,(29) with KL(Q∥Q ⋆ ) <∞ exactly when KL(Q∥P ref ) <∞. Step 2: the entropy Bregman term is nonnegative. Letν=Q−Q ⋆ , a finite signed measure withν(X) = 0. Fort∈[0,1],Q ⋆ +tν= (1− t)Q ⋆ +tQ∈P, andh(t) :=H ε (Q ⋆ +tν) is concave on [0,1] (Lemma 2, through the affine mapQ7→Σ Q ) with right derivativeh ′ (0 + ) = R G ε Q ⋆ dν(Lemma 1). Concavity placesh(1) below the tangent at 0: H ε (Q)≤ H ε (Q ⋆ ) + Z G ε Q ⋆ d(Q− Q ⋆ ), 54 which is exactly B −H ε (Q,Q ⋆ )≥ 0, and by the definition of B −H ε , −λH ε (Q) =−λH ε (Q ⋆ )− λ Z G ε Q ⋆ d(Q− Q ⋆ ) + λB −H ε (Q,Q ⋆ ).(30) Step 3: assembly. Summing (29) and (30), and substituting E Q [T ] = λE Q [G ε Q ⋆ ], F (Q) = KL(Q∥Q ⋆ ) + λB −H ε (Q,Q ⋆ ) + C, where the E Q [G ε Q ⋆ ] terms cancel against the integral in (30), leaving λE Q ⋆ [G ε Q ⋆ ], and C = λE Q ⋆ [G ε Q ⋆ ]− logZ− λH ε (Q ⋆ ). Finally, applying (29) at Q = Q ⋆ gives KL(Q ⋆ ∥P ref ) = λE Q ⋆ [G ε Q ⋆ ]− logZ, so C = KL(Q ⋆ ∥P ref )− λH ε (Q ⋆ ) = F (Q ⋆ ), which is (28). Corollary 3 (Weighted diffusion training). Letq ⋆ be the weights of Corollary 2 and letℓ(x;θ) be any per-example diffusion training loss (a denoising-score-matching loss or a negative variational bound). Then: (i)The weighted objective P n i=1 q ⋆ i ℓ (x i ;θ) =E X∼Q ⋆ λ [ℓ(X;θ)] coincides with the corresponding standard training objective with data law Q ⋆ λ ; no other component of the training pipeline changes. (i) LetP ref andQ ⋆ λ be as in Proposition 10. A lawQwithF(Q)<∞attainsmin Q ′ ≪P ref Fif and only if Q = Q ⋆ λ ; and for any sequence (θ k ) with KL(Q θ k ∥Q ⋆ λ )→ 0, F (Q θ k )→ min Q ′ ≪P ref F (Q ′ ). Proof. Part (i) is the definition of expectation under a finitely supported law. For part (i), the “only if” direction: ifF(Q) =F(Q ⋆ λ )<∞, then by(28)the two nonnegative terms vanish, in particularKL(Q∥Q ⋆ λ ) = 0, soQ=Q ⋆ λ ; the converse is trivial. For the convergence claim, writeQ k =Q θ k andδ k =sup A |Q k (A)−Q ⋆ λ (A)|, so that R f d (Q k −Q ⋆ λ ) ≤2∥f∥ ∞ δ k for bounded measurablef, andδ k ≤ p KL(Q k ∥Q ⋆ λ )/2→ 0 by Pinsker’s inequality. By(28)it suffices that each right-hand term vanishes along the sequence. The KL term does by hypothesis. For the Bregman term, each entry of Σ Q k −Σ Q ⋆ λ is R φ a φ b d(Q k − Q ⋆ λ ) with|φ a φ b |≤1, so Σ Q k →Σ Q ⋆ λ , henceS ε Q k → S ε Q ⋆ λ within the compact set of density matrices with spectrum in [ε/d,1], on whichH is continuous, givingH ε (Q k )→ H ε (Q ⋆ λ ); and R G ε Q ⋆ λ d (Q k −Q ⋆ λ ) ≤2(1−ε)log(d/ε)δ k →0. Hence B −H ε (Q k ,Q ⋆ λ )→ 0, completing the proof. Remark 17 (Saddle computation of the weights, and the relation to DRO). (a) Computation. By Proposi- tion 1, on ∆ n the weights of Corollary 2 solve the finite-dimensional saddle problem min q∈∆ n max Θ∈T ε n X i q i log(nq i )− λ(1− ε) X i q i φ(x i ) ⊤ Θφ(x i )− λ log Tr e −Θ o , whose objective is convex and continuous inqon the compact ∆ n and concave and continuous in Θ on the compactT ε ; by Sion’s minimax theorem the order of optimization may be interchanged, and both optima are attained. Alternating best responses are natural: at fixedqthe inner maximum is the closed form Θ ⋆ (Q q ) of Proposition 1, while at fixed Θ the outer minimization is, by Lemma 4 with rewardG(x) =λ(1−ε)φ(x) ⊤ Θφ(x), the explicit softmax q i ∝ exp λ(1− ε)φ(x i ) ⊤ Θφ(x i ) . (b) Not worst-case-loss DRO. It is instructive to contrast(27)with KL-penalized distributionally robust training of the denoiser, min θ max Q≪ b P n E Q [ℓ(X;θ)]− η KL(Q∥ b P n ) , η > 0. 55 131415161718 Exponential von Neumann entropy 150 160 170 180 190 FD DINOv2 Base Model Guided (a) CelebA-HQ: Fr ́echet distance (FD). 131415161718 Exponential von Neumann entropy 0.120 0.122 0.124 0.126 0.128 0.130 0.132 KDD DINOv2 Base Model Guided (b) CelebA-HQ: kernel distance (KD). 891011121314 Exponential von Neumann entropy 120 130 140 150 160 170 FD DINOv2 Base Model Guided (c) ImageNet: Fr ́echet distance (FD). 891011121314 Exponential von Neumann entropy 0.0455 0.0456 0.0457 0.0458 0.0459 KDD DINOv2 Base Model Guided (d) ImageNet: kernel distance (KD). Figure 19: DINOv2-space distributional distances along the IGA target path. Fr ́echet distance (FD) and kernel distance (KD) are evaluated in the DINOv2 representation used for the spectral analysis. The initial portion of the IGA path reduces the diversity deficit while improving or maintaining distributional agreement with the data. Asλincreases further, the distance minima occur at metric- and dataset-dependent operating points, consistent with the transition from below-wall diversity repair to beyond-wall spectral extrapolation. There, by Lemma 4, the inner maximizer reweights the data by the loss,q i ∝ exp(ℓ(x i ;θ)/η): the adversarial reweighting tracks loss hardness and changes withθat every step. The IGA reweighting(27)isθ-independent and tilts by the entropy energy: it up-weights points along spectral directions the data underpopulates, whether or not the current model finds them hard. Comparing the two Gibbs exponents, the schemes produce the same weights only whenℓ(·;θ) is, on the training set, an affine function of the IGA exponent, which is a nongeneric coincidence. Replacing the denoiser’s training distribution by a worst-case-loss adversary therefore optimizes robustness, not spectral diversity, and is not equivalent to IGA training; the rigorous route to training-time IGA for diffusion models is the reweighting-and-refit composition of Corollaries 2 and 3. 56 0123456789 Digit class 0% 2% 5% 8% 10% 12% 15% Sample frequency Baseline (λ= 0) IGA λ= 0.5 IGA λ= 1 IGA λ= 1.5 IGA λ= 2 IGA λ= 4 MNIST reference (a) 0.00.51.01.52.04.0 IGA 0.09 0.10 0.11 0.12 0.13 0.14 Total Variation Distance (b) 0.00.51.01.52.04.0 IGA 0.016 0.018 0.020 0.022 0.024 0.026 Feature Fréchet distance (c) Figure 20: Training-time IGA reduces class imbalance in an MNIST GAN. (a) Generated digit frequencies move toward the empirical MNIST distribution as the IGA multiplier increases. (b,c) Moderate regularization reduces both class-distribution total variation and independent evaluator-feature Fr ́echet distance. Results are averaged over five seeds; error bars denote standard error. H Additional Numerical Results This appendix collects supplementary numerical results for the sampling-time and training-time experiments. Section H.1 reports Dinov2-space distributional distances for the CelebA-HQ and ImageNet experiments, complementing the independent Inception-v3 evaluation in the main text. Section H.2 reports the MNIST GAN results for the training-time realization of IGA discussed in Section 6.3. H.1 DINOv2-Space Distributional Distances The main text reports FID and KID in Inception-v3 feature space as an evaluation independent of the DINOv2 representation used to define the spectral entropy and entropy wall. Here we provide the corresponding DINOv2-space distributional distances. These results complement the independent Inception-v3 evaluation and make explicit the representation dependence of the precise fidelity optimum along the IGA regularization path. 57 H.2 Training-Time IGA on MNIST The training-time experiment of Section 6.3 evaluates whether the same spectral-entropy regularizer used for sampling-time IGA can be incorporated directly into adversarial training. Figure 20 reports the resulting class-frequency and distributional-fidelity measurements across the IGA path. Experimental details. We train a convolutional GAN on MNIST for 20 epochs. The generator maps a 64-dimensional standard-normal latent through a fully connected projection and two transposed-convolution stages to a 28×28 image, while the discriminator uses two strided convolutional layers followed by a linear output. We use batch size 128 and Adam with learning rate 2×10 −4 and (β 1 ,β 2 ) = (0.5,0.999) for both networks, with the non-saturating logistic generator objective and one discriminator update per generator update. We evaluate λ∈0, 0.5, 1, 1.5, 2, 4 over five random seeds. The fixed IGA representation is the unit-normalized 64-dimensional embedding of a separately trained MNIST classifier. At each generator update, the spectral adversary is recomputed from the current generated minibatch at its closed-form best response and then held fixed during the generator update, as in Proposition 7. We setε= 0.05. For independent evaluation, we use a second frozen classifier with a different architecture and a 96-dimensional embedding. For each run, 10,000 generated samples are used to compute class frequencies, total variation distance to the empirical MNIST test-set class distribution, and feature Fr ́echet distance between generated and real test samples in the independent evaluator space. As shown in Figure 20(a), increasingλinitially redistributes generated mass away from overrepresented classes and toward classes that are underrepresented by the baseline GAN. This redistribution is reflected quantitatively in panels (b) and (c): both class-distribution total variation and independent evaluator-feature Fr ́echet distance improve substantially at intermediate values ofλ. Their nonmonotone behavior at largerλ illustrates the tradeoff between the GAN fidelity objective and the distribution-level entropy reward. 58