Paper deep dive
Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models
Duong Bach, Hai Nguyen Hong, Cuong Do
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/8/2026, 2:42:35 AM
Summary
The paper demonstrates that matching the marginal distribution of a style latent variable to a Gaussian prior does not guarantee independence from class labels, leading to 'conditional style leakage.' The authors provide a theoretical decomposition showing that marginal matching is insufficient for factorized sampling and empirically show that standard models retain high class information in style latents despite near-zero global MMD. They propose an auditing framework and mitigation strategies, such as per-class style regularization and post-hoc conditional priors, to address this leakage.
Entities (11)
Relation Signals (6)
Marginal Matching → failstocertify → Conditional Style Leakage
confidence 95% · Matching only the marginal distribution places no constraint on the class-conditional distributions, allowing the latent style to remain highly predictive of the label
Global MMD → failstodetect → Conditional Style Leakage
confidence 93% · Global MMD is a marginal statistic by construction... A model can therefore pass every standard diagnostic and still fail completely
Linear Probe → detects → Conditional Style Leakage
confidence 92% · allowing a linear probe to recover class labels with 74%--100% accuracy
F-CS-WAE → exhibits → Conditional Style Leakage
confidence 90% · Our case-study model and four representative latent baselines achieve near-zero global MMD while still allowing a linear probe to recover class labels
Post-hoc Conditional Prior → improves → Class-Conditional Generation
confidence 88% · A post-hoc conditional prior improves externally evaluated class generation to 0.97 on MNIST
F-CS-WAE → mitigates → Conditional Style Leakage
confidence 85% · Four mitigation strategies reduce probe accuracy to 21%--46%
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Factorized generative models commonly regularize a latent style variable z_s by matching its marginal distribution to a fixed Gaussian prior and interpret this as evidence that the style representation is independent of class information. We show that this interpretation is incorrect. Matching only the marginal distribution places no constraint on the class-conditional distributions, allowing the latent style to remain highly predictive of the label despite appearing perfectly Gaussian in aggregate. We derive an exact decomposition showing that this mismatch is one of four conditions required for factorized sampling, and demonstrate that eliminating it is necessary but not sufficient to obtain the intended factorization. Empirically, our case-study model and four representative latent baselines achieve near-zero global MMD while still allowing a linear probe to recover class labels with 74%--100% accuracy (10% chance level). Our model reaches 99.15% clustering accuracy, whereas externally evaluated class-conditional generation succeeds only 16% of the time. This leakage remains under six independent perturbations involving model capacity, curriculum, prior geometry, and supervision across two datasets. Four mitigation strategies reduce probe accuracy to 21%--46%, although they leave within-class dependence largely unchanged. A post-hoc conditional prior improves externally evaluated class generation to 0.97 on MNIST without retraining but reaches only 0.41 on CIFAR-10, while an empirical style bank achieves 0.88 on CIFAR-10. These results demonstrate that no divergence computed solely on the marginal distribution of the style latent can certify independence from class labels, and that reporting marginal statistics alone does not verify the property commonly claimed in factorized generative models.
Tags
Links
- Source: https://arxiv.org/abs/2608.05243v1
- Canonical: https://arxiv.org/abs/2608.05243v1
Trouble viewing inline? Open PDF directly →
Full Text
99,056 characters extracted from source content.
Expand or collapse full text
Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models Duong Bach, Hai Nguyen Hong, Cuong Do Code: https://github.com/DngBack/CS-WAE Abstract Factorized generative models regularize a style latent zsz_s toward a fixed prior with a marginal statistic, q(zs)≈(0,I)q(z_s) (0,I), and treat the result as a certificate that zsz_s carries no class information. The certificate does not hold. Matching the marginal constrains nothing about the class conditionals q(zs∣y)q(z_s y), so zsz_s can be exactly Gaussian in aggregate while remaining maximally informative about the label. We show this gap is one of four terms in an exact decomposition of what a factorized sampler must match, and that eliminating it is necessary but not sufficient for the sampling procedure the factorization exists to support. Empirically, a case-study model and four reference latent baselines with near-zero global MMD all permit a linear probe to recover the label from zsz_s at 7474–100%100\% (chance 10%10\%); our case-study model reaches 99.15%99.15\% clustering accuracy while externally evaluated class-conditional generation succeeds 16%16\% of the time. The leakage survives six one-at-a-time perturbations of capacity, curriculum, prior geometry, and supervision on two datasets. Four remedies attacking the leakage through different mechanisms span 2121–46%46\% probe recovery while leaving the within-class dependence proxy essentially unchanged. A post-hoc conditional prior raises external generated-class accuracy to 0.970.97 on MNIST without retraining but reaches only 0.410.41 on CIFAR-10; an empirical style bank reaches 0.880.88 on CIFAR-10. The methodological point: no divergence computed on q(zs)q(z_s) alone can certify zs⟂yz_s y, and reporting marginal statistics does not verify the property practitioners claim from them. Introduction Matching an aggregate posterior to a Gaussian is a statement about a marginal. Class-invariant style is a statement about a family of conditionals. The two are different claims, and the first does not imply the second: q(zs)q(z_s) can be exactly (0,I)N(0,I) while every q(zs∣y=k)q(z_s y=k) occupies its own region of latent space, so long as the regions average out. A regularizer that only ever sees q(zs)q(z_s) cannot detect this, and therefore cannot prevent it. The gap has teeth because of how factorized models generate. During training the decoder only ever receives (zc,zs)(z_c,z_s) drawn from the same image. If q(zs∣y=k)q(z_s y=k) differs across classes, and nothing in the objective stops it, the decoder is free to learn class-specific interactions between the two codes. At generation time a class prior supplies zcz_c for class k while zsz_s comes from the marginal, which averages over all classes. The resulting pair is one the decoder never saw. We call the failure conditional style leakage. What makes it dangerous is that the metrics a practitioner would check are all blind to it. Reconstruction is measured on the posterior manifold, where zsz_s comes from a real image and the learned interaction is exactly the one being exercised, so it looks fine. Global MMD is a marginal statistic by construction. Clustering accuracy reads only zcz_c. A model can therefore pass every standard diagnostic and still fail completely at the one capability the factorization was built to provide. Our starting point is that this is not a defect of any particular loss but a structural property of factorized sampling. We give an exact decomposition of the mismatch such a sampler incurs (Theorem 1) into four non-negative terms: global style-prior mismatch, conditional style leakage, semantic-prior mismatch, and within-class dependence between the two codes. Valid sampling requires all four to vanish, so removing the leakage is necessary but not sufficient, and the latent mismatch upper-bounds the image-space mismatch but not conversely. That asymmetry is what makes a decoder-level intervention necessary rather than redundant. We then audit a case-study model and four reference latent baselines with finite-sample proxies for the four terms, and use F-CS-WAE (Factorized Class-Structured Spherical Cauchy WAE) as a concentrated case study. We use it to exhibit and repair the failure, not to argue it beats the baselines it is compared against. The four reference families share one diagnostic architecture and budget, while F-CS-WAE is evaluated in its native configuration, so the cross-model table supports within-model auditing rather than a ranking. F-CS-WAE is label-guided by design; the proposed per-class style MMD reuses the labels already required by its semantic objective and therefore adds no annotation requirement. Contributions. (1) An auditing framework for factorized generative models that separates marginal prior fit, conditional leakage, semantic-prior mismatch, and within-class dependence, supported by a structural KL decomposition. (2) A conditional audit protocol built from finite-sample proxies for those terms, applied to a case-study model and reference latent baselines. (3) A decoder-level validation study showing when information detected by the audit is actually used during recombination, together with a six-way robustness check on two datasets, showing no single design choice accounts for the leakage while its magnitude is strongly capacity-dependent on CIFAR-10. (4) An evaluation of representation- and sampling-level repairs, including a negative transfer result. Related Work We organize prior work by a single question: does the method certify conditional invariance zs⟂yz_s y, or only a statistic compatible with zs⟂̸yz_s y? VAEs (Kingma and Welling 2013) impose a per-example KL toward a fixed prior; WAEs (Tolstikhin et al. 2017) relax this to a divergence between the aggregated posterior and the prior, computed with kernel two-sample tests (Gretton et al. 2012) or projected distances (Kolouri et al. 2018). The VAE penalty is not simply an aggregate regularizer in disguise: averaged over the data it splits exactly into q(x)[KL(q(z∣x)∥p(z))]=Iq(X;Z)+KL(q(z)∥p(z))E_q(x)[KL(q(z x)\|p(z))]=I_q(X;Z)+KL(q(z)\|p(z)) (Hoffman and Johnson 2016), an information-capacity term plus an aggregate-mismatch term of the kind WAE-MMD targets directly. Neither certifies I(zs;y)=0I(z_s;y)=0. The capacity term bounds what Z encodes about X in general, not about the label, and the aggregate term is precisely the marginal statistic Proposition 1 shows is insufficient. β-VAE (Higgins et al. 2017), β-TCVAE (Chen et al. 2018), FactorVAE (Kim and Mnih 2018), and DIP-VAE (Kumar et al. 2018) penalize total correlation or aggregate moments. These constrain dependence among latent dimensions, not between a designated style code and the label. More generally, unsupervised disentanglement is not identifiable without inductive biases in the model or data (Locatello et al. 2019); weakly supervised pairs can supply such a bias (Locatello et al. 2020), but a bias toward factorization is not itself a certificate of the particular invariance zs⟂yz_s y. Semi-supervised VAEs (Kingma et al. 2014) and CVAE (Sohn et al. 2015) condition generation on the label, sidestepping the need for zcz_c to encode identity, but do not by themselves test whether a residual latent remains label-informative. Content–style separation predates deep generative models (Tenenbaum and Freeman 2000). Modern split-latent models use class labels, grouped observations, pairwise similarity, mutual-information penalties, or latent optimization to divide specified from residual variation (Mathieu et al. 2016; Bouchacourt et al. 2018; Jha et al. 2018; Klys et al. 2018; Zheng and Sun 2019; Ilse et al. 2020; Gabbay and Hoshen 2020). These works establish that explicit inductive biases can improve recombination; they also make a plain VAE latent an inappropriate surrogate for a declared class-invariant style code. Most directly, Ridgeway and Mozer (2018) introduce leakage filtering, probe class from a style posterior, and evaluate content–style recombination on MNIST and more complex domains. Our novelty is therefore neither the first observation that content can leak into style nor the algebraic chain rule in isolation. It is the auditing framework that uses the decomposition to separate four distinct failure sources, connects finite-sample latent diagnostics to the claim practitioners make from marginal matching, and validates at decoder level whether detected leakage changes recombination and sampling. Image-translation systems operationalize the same recombination requirement: MUNIT (Huang et al. 2018) and DRIT (Lee et al. 2018) combine content with sampled style, while DMIT (Yu et al. 2019) exposes the generator to random cross-domain combinations during training. Their notion of domain-specific style is task-dependent—the domain may itself be the conditioning attribute—but their training strategies highlight the same train–sample support issue. Recent methods add other inductive biases: V3 exploits variance–invariance patterns across domains (Wu et al. 2025), while SCFlow learns invertible merging from combinatorial style–content coverage (Ma et al. 2025). These methods address how to learn a separation. Reference-guided diffusion work also calls unwanted reference content carried by style features “content leakage” and suppresses it by masking (Zhu et al. 2025); our leakage is instead label information in a learned style latent. We ask when such a latent can be sampled independently. Several invariance methods directly target zs⟂̸yz_s y. The variational fair autoencoder (Louizos et al. 2015) matches nuisance-conditional posteriors to a common target; the per-class style MMD in Section Method and Case Study is a direct instantiation of that idea, not a new mechanism. Gradient reversal (Ganin and Lempitsky 2015), Fader Networks (Lample et al. 2017), and information-theoretic invariance (Moyer et al. 2018) remove specified information through different objectives. They primarily optimize representation invariance; our decomposition asks the additional question of whether removing zs⟂̸yz_s y suffices for the independent sampler. It does not when the other three terms remain. Deep generative clustering (DEC (Xie et al. 2015), IDEC (Guo et al. 2017), VaDE (Jiang et al. 2016)) evaluates via ACC/NMI/ARI on the cluster variable, a protocol that cannot see this failure because it never inspects zsz_s. Our case study makes the point concrete: 99.15%99.15\% clustering accuracy on the very checkpoint whose class-conditional generation reaches 16%16\% external Gen-ACC. For the semantic variable we use a Spherical Cauchy prior with a Möbius reparameterization (Sablica and Hornik 2025), a heavier-tailed alternative to the von Mises-Fisher posterior of S-VAE (Davidson et al. 2018); this is an implementation choice, not a claim about the leakage, which is defined independently of how zcz_c is parameterized. Factorized Sampling and Its Mismatch Let =(xi,yi)D=\(x_i,y_i)\ have K classes. A factorized model encodes each input into a semantic variable zcz_c and a style variable zsz_s, and regularizes the latter by ℒstyle=D(q(zs),(0,I)),L_style=D (q(z_s),\,N(0,I) ), (1) for some aggregate divergence D, with q(zs)=y[q(zs∣y)]q(z_s)=E_y[q(z_s y)]. Write p(zs):=(0,I)p(z_s):=N(0,I) for the style prior. Together with a class-conditional semantic prior p(zc∣y=k)p(z_c y=k) these define the factorized sampler: draw zc∼p(zc∣y=k)z_c p(z_c y=k) and zs∼p(zs)z_s p(z_s) independently, which is the naive class-conditional sampling procedure in standard use. Marginal matching certifies nothing Proposition 1 (No certificate). Let zs∼(0,I)z_s (0,I) on ℝdsR^d_s and let Akk=1K\A_k\_k=1^K be any measurable partition with P(zs∈Ak)=1/KP(z_s∈ A_k)=1/K. Define y=k⇔zs∈Aky=k z_s∈ A_k. Then q(zs)=(0,I)q(z_s)=N(0,I) exactly, so D(q(zs),(0,I))=0D(q(z_s),N(0,I))=0 for every divergence D; and I(zs;y)=H(y)=logKI(z_s;y)=H(y)= K, the maximum possible. The proof is immediate: y is defined post hoc as a measurable function of zsz_s, so the law of zsz_s is untouched, while H(y∣zs)=0H(y z_s)=0 makes the mutual information maximal. The construction is deliberately adversarial and we do not claim trained decoders produce anything like it. The point is narrow and worst-case: no divergence computed on q(zs)q(z_s) alone, MMD or otherwise, can rule out zsz_s being maximally class-informative, however the dependence happens to be shaped. What valid factorized sampling requires Proposition 1 is the extremal case of term (2) below. The following decomposition covers any encoder, not just that construction. Theorem 1 (Factorized-sampling mismatch). Let y be uniform on 1,…,K\1,…,K\ and define Mfact:=y[KL(q(zc,zs∣y)∥p(zc∣y)p(zs))]M_fact:=E_y[KL(q(z_c,z_s y)\,\|\,p(z_c y)\,p(z_s))]. Then Mfact= M_fact=\; Iq(zc;zs∣y)⏟(4) within-class dep.+y[KL(q(zc∣y)∥p(zc∣y))]⏟(3) semantic-prior mismatch I_q(z_c;z_s y)_(4) within-class dep.+ E_y[KL(q(z_c y)\|p(z_c y))]_(3) semantic-prior mismatch +Iq(zs;y)⏟(2) style leakage+KL(q(zs)∥p(zs))⏟(1) style-prior mismatch, + I_q(z_s;y)_(2) style leakage+ KL(q(z_s)\|p(z_s))_(1) style-prior mismatch, and all four terms are non-negative. The proof is a chain-rule expansion applied twice, given in the supplementary material. Two consequences matter. Corollary 1 (Sampling validity). q(zc,zs∣y=k)=p(zc∣y=k)p(zs)q(z_c,z_s y=k)=p(z_c y=k)\,p(z_s) for a.e. k if and only if Mfact=0M_fact=0, i.e. iff all four terms vanish. Class-invariant style (term 2) is thus necessary but not sufficient; it must be accompanied by a matched style prior, a matched semantic prior, and within-class independence of zcz_c and zsz_s. Corollary 2 (Decoder pushforward). For a decoder DecDec with pushforward Dec#Dec_\#, y[KL(Dec#q(⋅∣y)∥Dec#[p(zc∣y)p(zs)])]≤MfactE_y[KL(Dec_\#q(· y)\|Dec_\#[p(z_c y)p(z_s)])]≤ M_fact by the data-processing inequality. Corollary 2 runs one way. A small MfactM_fact suffices for the fixed decoder to make posterior-decoded and sampler-decoded distributions close, but it is not by itself a certificate that decoded samples match the data distribution. Conversely, a large value does not force a visible failure, since a decoder can be insensitive to the mismatched directions and map distinct latent distributions to nearly the same images. Whether a trained decoder actually conditions on the leaked structure is an empirical question that the decomposition leaves open, which is why the intervention in Section The decoder reads identity off the style code is needed rather than redundant. Diagnostics Each term gets a finite-sample proxy. We measure the proxies, not the information quantities, and never estimate MfactM_fact as a whole. Term 1 is proxied by global MMD, MMD2(q(zs),(0,I))MMD^2(q(z_s),N(0,I)). Term 2 by inter-class style separation Δinter=(K2)−1∑j<k‖μ¯s(j)−μ¯s(k)‖2 _inter= K2^-1 _j<k\| μ_s^(j)- μ_s^(k)\|_2 and by linear-probe accuracy LP(zs→y)LP(z_s→ y). Δinter _inter is a mean-separation statistic and thus a convenient symptom, not what Proposition 1 is about: that proposition concerns dependence in general and holds even when class means coincide. An HSIC-based proxy would also apply here, but our fixed-bandwidth estimator saturates at zsz_s’s empirical scale, returning a near-constant value across checkpoints whose Δinter _inter and LP vary widely, so we do not report it. Term 3 is already computed each run as ℒclassL_class. Term 4 has no existing proxy; we compare real within-class pairs against independently recombined ones, JointMMD=1K∑kMMD2((zci,zsi)yi=k,(zci,zsπ(i))yi=k), splitJointMMD= 1K _kMMD^2 (&\(z_c^i,z_s^i)\_y_i=k,\\ &\(z_c^i,z_s^π(i))\_y_i=k ), split (2) where π permutes style codes within class k, breaking the within-class coupling while leaving both per-class marginals, and hence terms (1)–(3), untouched. Method and Case Study The standard style regularizer, Eq. 1, matches only the marginal. The remedy we study instead matches each class conditional: ℒstyle-cls=1K∑k=1KMMDRBF(zsi:yi=k,εj),L_style-cls= 1K _k=1^KMMD_RBF (\z_s^i:y_i=k\,\ _j\ ), (3) εj∼(0,I) _j (0,I). Eq. 1 has no gradient incentive to reduce inter-class separation, since the mixture can match (0,I)N(0,I) while every conditional stays class-structured; Eq. 3 penalizes exactly that. Nothing about it is specific to our architecture: it needs a labeled batch and a style latent. In F-CS-WAE, the semantic MMD and auxiliary classifier already consume the same labels, so adding Eq. 3 does not change the supervision regime or annotation budget. Applied during training to an otherwise unsupervised model, it would require labels or pseudo-labels; this is distinct from the diagnostics of Section Diagnostics, which need labels only on the evaluation set and apply post hoc to any trained model. F-CS-WAE instantiates the setup concretely: a hyperspherical semantic variable zc∈dc−1z_c ^d_c-1 aligned to class-conditional Spherical Cauchy priors, and a Euclidean style variable zs∈ℝdsz_s ^d_s regularized by Eqs. 1–3, with a shared ResNet-18 trunk feeding two heads and a residual upsampling decoder (dc=64d_c=64, ds=128d_s=128). The objective adds a supervised semantic MMD, an aggregated semantic MMD, and an auxiliary classifier on μc _c that prevents semantic collapse; these shape zcz_c and are not part of the leakage remedy. Coefficients ramp over a four-phase 300-epoch curriculum in which per-class style regularization begins in phase B, after a 50-epoch reconstruction-only phase A during which the decoder does see real (zc,zs)(z_c,z_s) pairs unregularized. Section Transfer and robustness tests whether removing that warmup matters; it does not. Full details are in the supplementary material. At generation we either sample naively, zc∼p(zc∣y=k)z_c p(z_c y=k) and zs∼(0,I)z_s (0,I), or apply a post-hoc class-conditional style prior: estimate μ¯s(k) μ_s^(k) and σ¯s2(k) σ_s^2(k) from encoded training data and draw zs∼(μ¯s(k),τ2diag(σ¯s2(k)))z_s ( μ_s^(k),τ^2diag( σ_s^2(k))) with τ=0.25τ=0.25. The second needs no retraining, only a forward pass. Experiments CIFAR-10 (Krizhevsky 2009) is the primary clustering/generation benchmark; MNIST (LeCun et al. 1998) and Fashion-MNIST (Xiao et al. 2017) are used for diagnostics. We report clustering (ACC/NMI/ARI), reconstruction (SSIM/LPIPS (Zhang et al. 2018)), sample quality (FID (Heusel et al. 2017), 10k vs 10k), and generated-class accuracy (Gen-ACC), the fraction of generated images assigned to the sampled class by an independently trained external classifier. Table 1: CIFAR-10 against baselines matched in backbone (ResNet-18) and latent dimension (d=192d=192), grouped by how the label is used. F-CS-WAE is 3-seed mean. The fair clustering comparison is against the label-guided group, which like F-CS-WAE pushes the label into the latent. Method ACC NMI FID F-CS-WAE 0.809 0.650 83.0 Label-guided (label shapes the latent) AEWithCE 0.665 0.611 144.3 AEWithSupCon 0.541 0.599 275.5 AEWithTriplet 0.506 0.502 281.4 AEWithCenterLoss 0.243 0.122 235.8 Conditional-generative (label → decoder) ConditionalVAE 0.133 0.024 396.4 ConditionalWAE-MMD 0.102 0.008 170.3 GaussianClassPriorWAE 0.103 0.008 174.7 Unsupervised ResNetAE 0.220 0.102 121.0 Before using F-CS-WAE to study leakage we check it is not degenerate, since a degenerate model could leak trivially. Table 1 compares it against seven baselines matched in backbone and latent dimension. Against the label-guided autoencoders, the fair comparison, F-CS-WAE (80.9%80.9\%) exceeds the best (66.5%66.5\%) and has the lowest FID in the table. The conditional-generative baselines cluster near chance because their label enters only at the decoder, leaving the encoder latent unshaped; they are included as the closest generative analogues, not as a clustering comparison. This is evidence of competitiveness, not a controlled superiority claim: the baselines are single-seed, F-CS-WAE uses both latent-shaping and a generative prior where each baseline uses one, and we did not equalize per-method tuning. Evaluation independence. All generation-side metrics in the main paper are scored directly from generated pixels by a classifier trained only on real images and independent of the model under test: a 4-layer CNN for MNIST (99.4%99.4\% real-test accuracy) and a WideResNet-28-10 (Zagoruyko and Komodakis 2016) for CIFAR-10 (94.1%94.1\%). Under this fixed protocol, MNIST naive Gen-ACC is 0.160.16, the class-conditional prior reaches 0.970.97, and latent-swap style-following is 96.5%96.5\%. On CIFAR-10, the class-conditional diagonal prior at τ=0.25τ=0.25 reaches 0.410.41 and the empirical style bank reaches 0.880.88. Internal-classifier scores are reported only in the supplementary material as a robustness comparison; they are not mixed with the primary results below. Marginal metrics hide the leakage Table 2: Conditional style leakage across a case-study model and four reference latent baselines on MNIST. Global MMD is near zero everywhere; Δinter _inter and LP are not. Gen-ACC is scored by the external CNN and uses naive zs∼(0,I)z_s (0,I). The four reference families share one architecture and budget; F-CS-WAE uses its native configuration, so conclusions are within-row rather than ranked across rows. Model MMD Δinter _inter LP Gen-ACC VAE 0.0002 2.86 87.4% N/A WAE-MMD 0.0576 9.38 93.0% N/A β-TCVAE 0.0002 2.30 73.9% N/A FactorVAE 0.0009 2.27 75.9% N/A F-CS-WAE 0.0013 6.27 100.0% 0.16 F-CS-WAE ++ per-class 0.0015 1.22 42.6% 0.66 Table 2 applies the diagnostic to four unsupervised baselines and F-CS-WAE on MNIST. The baselines are purpose-built for this comparison and share the same encoder family, latent dimension, optimizer, and 100-epoch budget with one another. F-CS-WAE is evaluated in its native configuration. Accordingly, the table is an audit rather than a leaderboard: for every row individually, global MMD is decoupled from Δinter _inter and LP, and the argument does not depend on ranking leakage severity across model families. Every baseline shows LP far above chance despite near-zero global MMD. WAE-MMD has the largest global MMD (0.05760.0576, some 3030–300×300× the others), suggesting its own regularizer had not converged, and correspondingly the largest baseline Δinter _inter and LP. More telling are β-TCVAE and FactorVAE: their objectives explicitly penalize latent dependence, and they do achieve the smallest Δinter _inter of the five, yet LP stays at 7474–76%76\%. Penalizing total correlation among latent dimensions reduces, but does not remove, dependence between the latent and the label. These are different quantities. Figure 1: Real image, posterior reconstruction, naive-prior sample, and class-conditional-prior sample, for five digits. Reconstructions are indistinguishable from the real image (SSIM 0.9830.983); naive-prior samples are legible digits but frequently the wrong one; the class-conditional prior restores the intended identity. F-CS-WAE without the remedy is the most extreme case (Δinter=6.27 _inter=6.27, LP=100%=100\%), and the consequence is concrete rather than statistical. On the same checkpoint it reaches 99.15%99.15\% clustering accuracy and SSIM 0.9830.983, yet class-conditional generation under naive sampling reaches only 16%16\% external Gen-ACC. The failure is also structured rather than random: probability mass collapses onto a handful of attractor digits regardless of the class requested, which is what one expects if the decoder resolves an unfamiliar (zc,zs)(z_c,z_s) pair by trusting whichever code it learned to read identity from. A 2D t-SNE of zsz_s makes the leakage visually obvious only for the two largest-Δinter _inter rows; for VAE, β-TCVAE, and FactorVAE it is statistically real but invisible in projection, which is a reason to prefer the quantitative diagnostic over a plot. The decoder reads identity off the style code A probe shows zsz_s contains class information; it does not show the decoder uses it. For every ordered class pair (a,b)(a,b) we take μc _c from a real test image of class a and μs _s from an independent real image of class b, decode, and classify. This uses only real, individually-encoded outputs rather than prior draws, so a failure to recover class a cannot be blamed on an out-of-support prior sample, though the pairing itself may lie off the joint support the decoder trained on. Figure 2: MNIST latent swap at δ=0δ=0: rows donate zcz_c (content) and columns donate zsz_s (style). The external CNN assigns the output to the style-donor class in 96.5%96.5\% of off-diagonal swaps. Internal-classifier heatmaps for both δ settings are retained only as a supplementary robustness comparison. At δ=0δ=0 on MNIST, external style-following is 96.5%96.5\% (Figure 2): the decoder reads identity predominantly from zsz_s. Every row of the swap grid (fixed zcz_c) looks nearly identical; the column, the zsz_s donor, determines the digit. The per-class remedy shifts the qualitative dependence toward content on both datasets. Exact rates under the model’s internal classifier are provided only in the supplementary robustness comparison and are not used as primary evidence here. Per-class conditional MMD confirms the same structure behind Δinter _inter’s single scalar: at δ=0δ=0, mean per-class MMD2(q(zs∣y=k),(0,I))MMD^2(q(z_s y=k),N(0,I)) is 21×21× the global MMD on MNIST (0.01860.0186 vs 0.000890.00089) and 15×15× on CIFAR-10. Every one of the ten per-class values sits far above the global figure the marginal regularizer actually optimizes. At δ=1δ=1 the gap narrows but does not close (4.4×4.4×; 5.3×5.3×). Is this an off-support artifact? The obvious objection is that cross-class pairs simply leave the training joint’s support. Grading the intervention by support answers it. Same-class cross-image pairs sit at 1.2×1.2× the same-image kNN distance to the training joint and preserve identity, whereas cross-class pairs sit at 2.1×2.1× and largely lose the content-donor identity. Interpolating zsz_s between a same-class and a cross-class donor hands identity over monotonically, crossing at α≈0.38α≈ 0.38, well before the pair reaches cross-class support distance. Style-dominance turns on as soon as zsz_s carries a competing class signal, not only once the pair leaves the support. Cross-class pairs are measurably farther out, so the objection is not fully dissolved, but the effect does not reduce to that distance. What the remedies do and do not fix Per-class style MMD works in the intended direction without finishing the job. It cuts Δinter _inter by 81%81\% (6.27→1.226.27→ 1.22) and LP by 57.457.4 points (100%→42.6%100\%→ 42.6\%), still far above the 10%10\% floor and higher than three of the four unsupervised baselines. Theorem 1 explains why this term-2 remedy is necessarily partial: Eq. 3 drives terms (1)–(2) down by construction but places no pressure on term (4), the within-class dependence a decoder can still exploit once every q(zs∣y=k)q(z_s y=k) individually matches the prior. Table 3: Representation-level remedies on MNIST at matched backbone, dimensions, and training budget. Generation scores from the model’s internal classifier are moved to the supplementary robustness comparison. Single-seed point estimates are reported. Remedy LP Δinter _inter JointMMD None (δ=0δ=0) 100.0 6.27 .0037 Per-cls. MMD (δ=1δ=1) 42.6 1.22 .0043 VFAE-style cond. 46 1.25 .0042 HSIC penalty 37 1.08 .0041 Gradient reversal 21 0.82 .0044 ++ joint term-4 40 1.15 .0016 Table 4: Six-point sweep of the per-class style MMD weight δ. Single-seed point estimates are reported. Δinter _inter decreases steadily on both datasets, while ACC is non-monotonic and has one pronounced dip per dataset. MNIST CIFAR-10 δ ACC Δinter _inter ACC Δinter _inter 0 .992 6.21 .813 3.74 0.03 .994 3.27 .811 2.48 0.1 .990 2.12 .810 2.03 0.3 .994 1.55 .715 1.59 1 .859 1.22 .800 1.05 3 .993 1.18 .799 0.95 Table 4 sweeps six values. Δinter _inter decreases steadily on both datasets, so the leakage-reduction mechanism behaves as expected across the range rather than only at the endpoints. Naive FID broadly improves but is not monotonic, and clustering ACC has a pronounced single-seed dip at δ=1δ=1 on MNIST and δ=0.3δ=0.3 on CIFAR-10. With one seed per point, we cannot attribute those dips to δ rather than initialization and make no significance claim across sweep settings. Because per-class MMD only reduces term 2 to 42.6%42.6\% LP, it cannot separate “term 2 is still too large” from “term 2 is not the whole story”. Table 3 therefore compares four remedies that attack term 2 through different mechanisms, plus one that also targets term 4. Gradient reversal is much the strongest term-2 remedy, driving LP to 21%21\%, within 1111 points of chance and less than half what per-class MMD achieves. Yet every term-2-only intervention leaves JointMMD in the narrow 0.00410.0041–0.00440.0044 range. Targeting term 4 directly cuts JointMMD by 2.7×2.7×, the only intervention that moves this proxy substantially, while its LP remains higher than gradient reversal’s. Thus reducing I(zs;y)I(z_s;y) does not by itself reduce within-class zcz_c–zsz_s dependence, exactly the distinction made by Corollary 1. Internal-evaluator generation scores for these checkpoints are retained only as a robustness comparison in the supplementary material. We calibrate the term-4 proxy against a within-class double-permutation null using 200 permutations. Every checkpoint sits 44–5×5× above its null (p=0.005p=0.005), so term 4 is genuinely non-zero, while the δ=0→1δ=0→ 1 change is not significant (p=0.31p=0.31): per-class style MMD leaves term 4 statistically untouched even as it cuts Δinter _inter and LP sharply. A conditional-HSIC cross-check agrees. One comparison we avoid: MNIST and CIFAR-10 JointMMD values are close, but even after standardization, equal MMD on two different latent distributions does not imply equal dependence, so we draw no cross-dataset conclusion from it. Transfer and robustness Table 5: Single-variable perturbations from the δ=0δ=0 checkpoint, each removing one candidate confound, both datasets. Single-seed point estimates are reported. MNIST CIFAR-10 Perturbation Δinter _inter LP Δinter _inter LP None (baseline) 6.27 100.0 3.80 86.9 Balanced dims 4.79 99.9 2.11 52.4 Smaller style 3.28 92.6 1.68 44.0 No warmup 5.53 100.0 3.78 88.3 Gaussian prior 5.35 99.6 3.82 74.5 vMF prior 5.69 100.0 3.03 83.0 No classifier 5.09 97.1 3.15 67.1 Figure 3: Linear-probe recovery of the label from zsz_s for the baseline and six perturbations. Leakage survives all of them on both datasets, but the two capacity perturbations move CIFAR-10 far more than they move MNIST, while no-warmup and the vMF prior barely move either. Several design choices in F-CS-WAE could plausibly be the real explanation for why zsz_s absorbs so much class information: the style latent being larger than the semantic one, the reconstruction-only warmup, the prior geometry, or the auxiliary classifier. Table 5 and Figure 3 perturb each independently on both datasets. LP never drops below 92.6%92.6\% on MNIST or 44.0%44.0\% on CIFAR-10, and neither approaches chance. On MNIST every perturbation is mild, costing at most 7.47.4 points; on CIFAR-10 the same six span a 4444-point range. Smaller style capacity is the largest effect on both (7.47.4 and 42.942.9 points), followed by balanced dims, but capacity is not uniquely special: removing the classifier costs 19.819.8 points on CIFAR-10 and the Gaussian prior 12.412.4, while no-warmup and the vMF prior barely move it. Two readings follow. Capacity is a real, dataset-dependent contributor, consistent with severity tracking how much class structure a dataset forces into zsz_s, and a paper that checked only MNIST would have badly understated how much one hyperparameter can move this number. We also flag a confound we do not disentangle: the perturbations that reduce LP most also reduce representational capacity, and removing the classifier collapses clustering from 99%99\% to 55.4%55.4\%, so part of each drop may reflect a weaker representation rather than genuine invariance. Even so, no single tested perturbation drives LP near chance, so none of these choices eliminates the leakage, even though capacity clearly modulates it. Leakage severity also follows the expected dataset ordering. Measuring intra-class diversity independently of our model, as mean within-class pairwise cosine distance in a frozen DINOv2 space, confirms the ordering with clear separation: MNIST 0.330.33, Fashion-MNIST 0.470.47, CIFAR-10 0.660.66. Both Δinter _inter and LP decrease along it (MNIST 6.276.27/100%100\%, Fashion-MNIST 4.864.86/97.7%97.7\%, CIFAR-10 3.803.80/86.9%86.9\%). We do not use Gen-ACC as a cross-dataset proxy for leakage severity: unlike Δinter _inter and LP, it is measured after decoding and therefore also inherits dataset difficulty, sample quality, and external-classifier error. The repair does not transfer. Table 6: Generation under zsz_s sampling strategies on the δ=0δ=0 checkpoints (zcz_c always from the class prior). Gen-ACC uses the fixed external CNN on MNIST and WideResNet-28-10 on CIFAR-10. MNIST CIFAR-10 Style sampling zsz_s Gen-ACC Div. Gen-ACC Div. Global (0,I)N(0,I) 0.16 0.151 0.09 0.187 Class diag., τ=0.25τ=0.25 0.97 0.135 0.41 0.101 Empirical style bank 0.96 0.182 0.88 0.241 On MNIST, the external CNN raises Gen-ACC from 0.160.16 under global Gaussian sampling to 0.970.97 under the class-conditional diagonal prior at τ=0.25τ=0.25 (Table 6). On CIFAR-10, the same parametric recipe reaches only 0.410.41 under the external WideResNet. The empirical style bank reaches 0.880.88 and also has higher measured diversity (0.2410.241 versus 0.1010.101), indicating that matching only a unimodal diagonal Gaussian misses important shape in q(zs∣y=k)q(z_s y=k). Controls for mean shift, variance shrinkage, and shuffled labels show the same qualitative mechanism under the internal evaluator and are reported only as a supplementary robustness comparison. The diagnostic therefore transfers across datasets more cleanly than the parametric repair does. Discussion CIFAR-10’s higher intra-class diversity weakens conditional style dependence, and Δinter _inter/LP confirm this cleanly, but the practical remedy does not inherit that cleanliness. CIFAR-10 samples stay visibly blurry (FID ∼83 83–8484), and we avoid calling naive output “visually plausible”; at this FID the accurate description is recognizable but not sharp. Global aggregate matching is standard in WAEs and disentanglement models, and our results show it is insufficient whenever the style variable must be class-invariant for generation. Per-class style MMD is best read as a supervised analogue of total-correlation penalties, aimed at the single dependency zs⟂̸yz_s y rather than at dependence among latent dimensions in general. Finally, the MNIST checkpoint reaches SSIM 0.9830.983 yet FID 73.9973.99 under naive sampling: reconstruction measures fidelity on the posterior manifold, not the prior, and evaluation of generative clustering should report both, since the two can diverge substantially. Limitations. We report no component-wise ablation beyond the δ axis. The per-class remedy assumes labels already available to the label-guided training objective and is not, by itself, an unsupervised disentanglement loss. Gen-ACC is confounded by dataset-intrinsic difficulty. The CIFAR-10 transfer failure is measured on one checkpoint per dataset, and we have not tested whether a richer conditional (a per-class mixture) closes it. The robustness check perturbs one variable at a time, so joint perturbations, which could compound given how much larger each effect is on CIFAR-10, remain untested. Table 3 is MNIST-only; whether gradient reversal’s term-2 advantage and the joint remedy’s term-4 advantage replicate on CIFAR-10 is untested. Three datasets remain too few to treat the diversity–leakage relationship as a measured correlation rather than a consistent ordering. Conclusion Conditional style leakage is a real failure mode in factorized generative models: aggregate regularization of a style variable does not prevent it from becoming class-dependent, because marginal matching is compatible with zsz_s being maximally informative about the label rather than merely correlated with it. We placed it inside an exact decomposition of what factorized sampling requires, measured it across a case-study model and four reference latent baselines, and showed that a decoder trained this way is measurably more sensitive to the leaked code than to the semantic one. No single perturbation of capacity, curriculum, prior geometry, or supervision eliminates the leakage on either dataset, though capacity strongly modulates its magnitude on CIFAR-10. Comparing four mechanistically different term-2 remedies settles what one remedy could not: they span 2121–46%46\% probe recovery yet leave the term-4 proxy unchanged, whereas the joint remedy cuts that proxy by 2.7×2.7× despite a higher probe score than gradient reversal. Eliminating conditional style leakage is necessary but not sufficient, exactly as Corollary 1 states. Under the fixed external protocol, the conditional prior reaches 0.970.97 Gen-ACC on MNIST but only 0.410.41 on CIFAR-10; an empirical bank reaches 0.880.88. Thus fixing a diagnostic finding on one dataset does not fix the underlying generative gap on another. Factorized generative models should verify class-invariance of style variables with conditional diagnostics rather than marginal statistics, and should report where their remedies do and do not generalize. References D. Bouchacourt, R. Tomioka, and S. Nowozin (2018) Multi-level variational autoencoder: learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, p. 2095–2102. External Links: Document Cited by: Related Work. R. T. Q. Chen, X. Li, R. Grosse, and D. Duvenaud (2018) Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942. Cited by: Related Work. T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak (2018) Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891. Cited by: Related Work. A. Gabbay and Y. Hoshen (2020) Demystifying inter-class disentanglement. In International Conference on Learning Representations, Cited by: J.3 Explicit split-latent baselines, Related Work. Y. Ganin and V. Lempitsky (2015) Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, p. 1180–1189. Cited by: J.3 Explicit split-latent baselines, Related Work. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola (2012) A kernel two-sample test. Journal of Machine Learning Research 13, p. 723–773. Cited by: Related Work. X. Guo, L. Gao, X. Liu, and J. Yin (2017) Improved deep embedded clustering with local structure preservation. In Proceedings of the International Joint Conference on Artificial Intelligence, p. 1753–1759. Cited by: Related Work. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, Cited by: Experiments. I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner (2017) beta-VAE: learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations. Cited by: Related Work. M. D. Hoffman and M. J. Johnson (2016) ELBO surgery: yet another way to carve up the variational evidence lower bound. In NIPS Workshop on Advances in Approximate Bayesian Inference, Cited by: Related Work. X. Huang, M. Liu, S. Belongie, and J. Kautz (2018) Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision, p. 172–189. Cited by: Related Work. M. Ilse, J. M. Tomczak, C. Louizos, and M. Welling (2020) DIVA: domain invariant variational autoencoders. In Proceedings of the Third Conference on Medical Imaging with Deep Learning, Proceedings of Machine Learning Research, Vol. 121, p. 322–348. Cited by: J.3 Explicit split-latent baselines, Related Work. A. H. Jha, S. Anand, M. Singh, and V. S. R. Veeravasarapu (2018) Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference on Computer Vision, p. 805–820. Cited by: Related Work. Z. Jiang, Y. Zheng, H. Tan, B. Tang, and H. Zhou (2016) Variational deep embedding: an unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148. Cited by: Related Work. H. Kim and A. Mnih (2018) Disentangling by factorising. arXiv preprint arXiv:1802.05983. Cited by: Related Work. D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling (2014) Semi-supervised learning with deep generative models. Advances in Neural Information Processing Systems. Cited by: Related Work. D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: Related Work. J. Klys, J. Snell, and R. Zemel (2018) Learning latent subspaces in variational autoencoders. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: Related Work. S. Kolouri, P. E. Pope, C. E. Martin, and G. K. Rohde (2018) Sliced-wasserstein autoencoder: an embarrassingly simple generative model. arXiv preprint arXiv:1804.01947. Cited by: Related Work. A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: Experiments. A. Kumar, P. Sattigeri, and A. Balakrishnan (2018) Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, Cited by: Related Work. G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer, and M. Ranzato (2017) Fader networks: manipulating images by sliding attributes. In Advances in Neural Information Processing Systems, Cited by: J.3 Explicit split-latent baselines, Related Work. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), p. 2278–2324. Cited by: Experiments. H. Lee, H. Tseng, J. Huang, M. Singh, and M. Yang (2018) Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision, p. 35–51. Cited by: Related Work. F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem (2019) Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 4114–4124. Cited by: Related Work. F. Locatello, B. Poole, G. Raetsch, B. Schölkopf, O. Bachem, and M. Tschannen (2020) Weakly-supervised disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 6348–6359. Cited by: Related Work. C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel (2015) The variational fair autoencoder. arXiv preprint arXiv:1511.00830. Cited by: J.3 Explicit split-latent baselines, Related Work. P. Ma, X. Yang, Y. Li, M. Gui, F. Krause, J. Schusterbauer, and B. Ommer (2025) SCFlow: implicitly learning style and content disentanglement with flow models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 14919–14929. Cited by: Related Work. M. F. Mathieu, J. J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun (2016) Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, Vol. 29. Cited by: Related Work. D. Moyer, S. Gao, R. Brekelmans, A. Galstyan, and G. Ver Steeg (2018) Invariant representations without adversarial training. In Advances in Neural Information Processing Systems, p. 9102–9111. Cited by: Related Work. K. Ridgeway and M. C. Mozer (2018) Open-ended content-style recombination via leakage filtering. arXiv preprint arXiv:1810.00110. Cited by: J.3 Explicit split-latent baselines, Related Work. L. Sablica and K. Hornik (2025) Hyperspherical variational autoencoders using efficient spherical cauchy distribution. arXiv preprint arXiv:2506.21278. Cited by: Related Work. K. Sohn, H. Lee, and X. Yan (2015) Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems. Cited by: Related Work. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016) Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p. 2818–2826. Cited by: K.4 FID: feature extractor, resizing, normalization. J. B. Tenenbaum and W. T. Freeman (2000) Separating style and content with bilinear models. Neural Computation 12 (6), p. 1247–1283. External Links: Document Cited by: Related Work. I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schölkopf (2017) Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558. Cited by: Related Work. Y. Wu, Z. Wang, B. Raj, and G. Xia (2025) Unsupervised disentanglement of content and style via variance-invariance constraints. In International Conference on Learning Representations, Cited by: Related Work. H. Xiao, K. Rasul, and R. Vollgraf (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747. Cited by: Experiments. J. Xie, R. Girshick, and A. Farhadi (2015) Unsupervised deep embedding for clustering analysis. arXiv preprint arXiv:1511.06335. Cited by: Related Work. X. Yu, Y. Chen, T. Li, S. Liu, and G. Li (2019) Multi-mapping image-to-image translation via learning disentanglement. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: Related Work. S. Zagoruyko and N. Komodakis (2016) Wide residual networks. In Proceedings of the British Machine Vision Conference, Cited by: K.6 External classifier: architecture and training protocol, Evaluation independence.. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Cited by: Experiments. Z. Zheng and L. Sun (2019) Disentangling latent space for VAE by label relevant/irrelevant dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12192–12201. Cited by: Related Work. L. Zhu, X. Wang, C. Zhou, Q. Gu, and N. Ye (2025) Less is more: masking elements in image condition features avoids content leakages in style transfer diffusion models. In International Conference on Learning Representations, Cited by: Related Work. Supplementary Material Marginal Matching Does Not License Factorized Sampling: Auditing Conditional Style Leakage in Factorized Generative Models A. Proofs A.1 Proposition 1 (No certificate) Proposition 2. Let zs∼(0,I)z_s (0,I) on ℝdsR^d_s, and let Akk=1K\A_k\_k=1^K be any measurable partition of ℝdsR^d_s with P(zs∈Ak)=1/KP(z_s∈ A_k)=1/K for each k (such a partition always exists, e.g. via level sets of a continuous statistic of zsz_s). Define y=k⇔zs∈Aky=k z_s∈ A_k. Then (i) q(zs)=(0,I)q(z_s)=N(0,I) exactly, so D(q(zs),(0,I))=0D(q(z_s),N(0,I))=0 for every divergence D; and (i) I(zs;y)=H(y)=logKI(z_s;y)=H(y)= K. Proof. (i) y is defined post hoc as a measurable function of zsz_s; the distribution of zsz_s itself is never altered, so q(zs)=(0,I)q(z_s)=N(0,I) holds exactly, and every divergence to (0,I)N(0,I), aggregate MMD included, is identically zero. (i) Since y is a deterministic function of zsz_s, H(y∣zs)=0H(y z_s)=0, so I(zs;y)=H(y)−H(y∣zs)=H(y)I(z_s;y)=H(y)-H(y z_s)=H(y). By construction P(y=k)=1/KP(y=k)=1/K for all k, so H(y)=logKH(y)= K, which is also the maximum possible entropy, and hence the maximum possible mutual information with any other variable, for a K-ary label. ∎ The construction is deliberately adversarial: Ak\A_k\ need not resemble anything a trained decoder produces, and q(zs∣y=k)q(z_s y=k) here is zsz_s restricted to AkA_k and renormalized, not a smooth family with class-specific means. That is the point. It shows that no divergence computed on q(zs)q(z_s) alone can rule out zsz_s being maximally class-informative, regardless of how the dependence is structured geometrically. It is a worst-case non-identifiability result, not a claim about typical behavior. A.2 Theorem 1 (Factorized-sampling mismatch) Theorem 2. Let y be uniform on 1,…,K\1,…,K\, let q(zc,zs∣y=k)q(z_c,z_s y=k) be the encoder’s joint class conditional, let p(zc∣y=k)p(z_c y=k) and p(zs)p(z_s) be the semantic and style priors, and let q(zs)=y[q(zs∣y)]q(z_s)=E_y[q(z_s y)]. Define Mfact:=y[KL(q(zc,zs∣y)∥p(zc∣y)p(zs))].M_fact:=E_y [KL(q(z_c,z_s y)\,\|\,p(z_c y)\,p(z_s)) ]. Then Mfact= M_fact=\; Iq(zc;zs∣y)+y[KL(q(zc∣y)∥p(zc∣y))] I_q(z_c;z_s y)+E_y[KL(q(z_c y)\|p(z_c y))] +Iq(zs;y)+KL(q(zs)∥p(zs)), +I_q(z_s;y)+KL(q(z_s)\|p(z_s)), and all four terms are non-negative. Proof. Fix k and expand the class-k term by inserting ±logq(zc∣k)±logq(zs∣k)± q(z_c k)± q(z_s k) into the integrand: KL(q(zc,zs∣k)∥p(zc∣k)p(zs)) (q(z_c,z_s k)\,\|\,p(z_c k)\,p(z_s) ) =q(zc,zs∣k)[logq(zc,zs∣k)q(zc∣k)q(zs∣k)] =E_q(z_c,z_s k) [ q(z_c,z_s k)q(z_c k)\,q(z_s k) ] +q(zc∣k)[logq(zc∣k)p(zc∣k)]+q(zs∣k)[logq(zs∣k)p(zs)] +E_q(z_c k) [ q(z_c k)p(z_c k) ]+E_q(z_s k) [ q(z_s k)p(z_s) ] =Iq(zc;zs∣y=k)+KL(q(zc∣k)∥p(zc∣k)) =I_q(z_c;z_s y=k)+KL(q(z_c k)\|p(z_c k)) +KL(q(zs∣k)∥p(zs)), +KL(q(z_s k)\|p(z_s)), where the first term is, by definition, the mutual information between zcz_c and zsz_s under q(⋅,⋅∣y=k)q(·,· y=k). Taking y[⋅]E_y[·] gives the first two terms of the statement directly, since y[Iq(zc;zs∣y=k)]=Iq(zc;zs∣y)E_y[I_q(z_c;z_s y=k)]=I_q(z_c;z_s y) by definition of conditional mutual information. For the remaining piece, y[KL(q(zs∣y)∥p(zs))]E_y[KL(q(z_s y)\|p(z_s))], insert ±logq(zs)± q(z_s) and use q(zs)=y[q(zs∣y)]q(z_s)=E_y[q(z_s y)]: y[KL(q(zs∣y)∥p(zs))]= _y [KL(q(z_s y)\|p(z_s)) ]= y[KL(q(zs∣y)∥q(zs))]⏟=Iq(zs;y) E_y [KL(q(z_s y)\|q(z_s)) ]_=\,I_q(z_s;y) +KL(q(zs)∥p(zs)). +KL(q(z_s)\|p(z_s)). where the first bracket is exactly Iq(zs;y)I_q(z_s;y): the average KL of a conditional from its own marginal is, by definition, the mutual information. Non-negativity of every term follows from non-negativity of KL divergence and of mutual information (itself a KL divergence). ∎ A.3 Corollaries Corollary 3 (Exact sampling validity). q(zc,zs∣y=k)=p(zc∣y=k)p(zs)q(z_c,z_s y=k)=p(z_c y=k)\,p(z_s) for almost every k if and only if Mfact=0M_fact=0, i.e. iff all four terms vanish simultaneously. In particular, class-invariant style (Iq(zs;y)=0I_q(z_s;y)=0) is necessary but not sufficient. Proof. Every term of Theorem 1 is a non-negative expectation of a KL divergence (terms 2 and 4 are mutual informations, themselves KL divergences), so their sum MfactM_fact is zero iff each term is zero; and y[KL(⋅∥⋅)]=0E_y[KL(·\|·)]=0 iff the integrand vanishes for a.e. y, iff the two per-class distributions coincide a.e. ∎ Corollary 4 (Decoder pushforward bound). Let DecDec be the decoder and Dec#μDec_\#μ the law of Dec(zc,zs)Dec(z_c,z_s) when (zc,zs)∼μ(z_c,z_s) μ. Then y[KL(Dec#q(⋅∣y)∥Dec#[p(zc∣y)p(zs)])]≤Mfact.E_y [KL(Dec_\#q(· y)\,\|\,Dec_\#[p(z_c y)\,p(z_s)]) ]≤ M_fact. Proof. The data-processing inequality states that pushing two distributions through the same Markov kernel (here DecDec) cannot increase their KL divergence; applying it inside y[⋅]E_y[·] and invoking Theorem 1 gives the bound. ∎ This runs one way only. A small MfactM_fact suffices for the fixed decoder to make posterior-decoded and sampler-decoded class-conditional distributions close, but it is not by itself a certificate that decoded samples match the data distribution. Conversely, a large value does not force a visible failure, because a decoder can be insensitive to the mismatched latent directions and map distinct latent distributions to nearly identical image distributions. Whether the trained decoder actually conditions on the leaked structure that terms (2) and (4) quantify is an empirical question the decomposition does not settle, which is why the latent-swap intervention in the main paper is necessary rather than redundant. B. Architecture Details The semantic head produces (μ~c,rc)( μ_c,r_c), giving μc=μ~c/‖μ~c‖2 _c= μ_c/\| μ_c\|_2 and concentration ρc=σ(rc)(1−ε) _c=σ(r_c)(1- ); the semantic latent is sampled by the Spherical Cauchy Möbius reparameterization ηc∼Unif(dc−1) _c (S^d_c-1), zc=Tμc,ρc(ηc)z_c=T_ _c, _c( _c). The style head produces (μs,logσs2)( _s, _s^2) and zs=μs+σs⊙εsz_s= _s+ _s _s, εs∼(0,I) _s (0,I). The ResNet-18 backbone operates on 32×3232× 32 inputs. It replaces the first 7×77× 7 convolution (stride 2) with a 3×33× 3 convolution (stride 1, padding 1) and removes the max-pool layer, yielding a 4×44× 4 spatial output with 512 channels; a shared FC layer maps 512×4×4=8,192512× 4× 4=8,192 to 256 dimensions with SiLU, from which separate linear heads produce the semantic and style parameters. The decoder concatenates [zc;zs][z_c;z_s] (dc=64d_c=64, ds=128d_s=128 in all experiments unless stated) and maps it through three ResBlockUp stages (bilinear 2×2× upsample, 1×11× 1 projection, GroupNorm+SiLU+Conv residual block; 512→256→128→64512→256→128→64 channels, 4×4→32×324× 4→ 32× 32), followed by a 3×33× 3 output convolution and sigmoid. For each class k, F-CS-WAE maintains a center mk∈dc−1m_k ^d_c-1 with prior p(zc∣y=k)=SCauchy(mk,ρp)p(z_c y=k)=SCauchy(m_k, _p), ρp=0.7 _p=0.7, updated by EMA from batch class-mean directions once per epoch, mk←normalize(τmk+(1−τ)μ¯c(k))m_k (τ m_k+(1-τ) μ_c^(k)), τ=0.95τ=0.95. The full training objective is ℒ =ℒrec+α(t)ℒclass+β(t)ℒagg+γ(t)ℒstyle =L_rec+α(t)L_class+β(t)L_agg+γ(t)L_style +δ(t)ℒstyle-cls+η(t)ℒcls, +δ(t)L_style -cls+η(t)L_cls, with ℒrec=λ1‖x−x^‖1+λLPIPSLPIPS(x,x^)L_rec= _1\|x- x\|_1+ _LPIPSLPIPS(x, x), ℒclass=1K∑kMMD2(Qk,Pk)L_class= 1K _kMMD^2(Q_k,P_k) with Qk=zci:yi=kQ_k=\z_c^i:y_i=k\ and Pk∼p(zc∣y=k)P_k p(z_c y=k), ℒagg=MMD2(zci,zc,rjp)L_agg=MMD^2(\z_c^i\,\z_c,r_j^p\) against a random-class prior mixture, and ℒcls=CE(h(μc),y)L_cls=CE(h( _c),y) a linear auxiliary classifier preventing semantic collapse. C. Training Hyperparameters and Schedule Table 7: Hyperparameters used in all F-CS-WAE experiments. Parameter Value Optimizer Adam, lr=10−3lr=10^-3 LR schedule StepLR, step 100, γ=0.5γ=0.5 Batch size 128 Total epochs 300 Gradient clip (global norm) 1.0 Semantic dim dcd_c / style dim dsd_s 64 / 128 Prior concentration ρp _p 0.7 EMA momentum τ 0.95 λ1 _1 / λLPIPS _LPIPS 1.0 / 0.1 αf _f / βf _f (class / agg. MMD) 2.0 / 5.0 γf _f (style) 1.0 δf _f (per-class style, off/on) 0 / 1 ηf _f (classifier) 0.3 Style prior temperature τs _s 0.25 Table 8: Training phase boundaries and active loss terms; ramps are linear. Phase Epochs Terms and ramp A 0–49 ℒrecL_rec only B 50–99 +ℒstyle(0→γf)+L_style(0→ _f), ℒstyle-cls(0→δf)L_style -cls(0→ _f), ℒcls(0.1→0.2)L_cls(0.1→0.2) C 100–199 +ℒclass(0→αf/2)+L_class(0→ _f/2), ℒagg(0→βf/2)L_agg(0→ _f/2), ℒcls(0.2→0.3)L_cls(0.2→0.3) D 200–299 ℒclass(αf/2→αf)L_class( _f/2→ _f), ℒagg(βf/2→βf)L_agg( _f/2→ _f) Per-class style regularization ramps up in Phase B, before ℒclassL_class and ℒaggL_agg are introduced in Phase C, so that it begins as early as possible relative to the semantic MMD terms, limiting rather than eliminating the epochs during which class-specific (zc,zs)(z_c,z_s) interactions can form without style-conditional counter-pressure. Phase A itself (epochs 0–49) is reconstruction-only, so the decoder does see real (zc,zs)(z_c,z_s) pairs before any style or class regularization is active. The no-warmup perturbation reported in the main paper tests whether removing this phase changes the leakage and finds that it does not. The eight baselines in the main paper’s CIFAR-10 competence table—ResNetAE plus seven label-using methods—use the same ResNet-18 backbone, total latent dimension (d=192d=192), 300-epoch budget, and optimizer settings as F-CS-WAE. The four MNIST cross-model diagnostic controls (VAE, WAE-MMD, β-TCVAE, and FactorVAE) instead share a lightweight 3-layer CNN encoder, latent dimension 64, and a 100-epoch budget. The two groups serve different comparisons and are not ranked against one another. D. Diagnostic Protocol All leakage diagnostics use n=2048n=2048 held-out test samples with a fixed seed, so repeated invocations across checkpoints evaluate the same subset rather than adding sampling noise on top of single-seed training runs. The Euclidean MMD estimator uses the average of a multi-scale RBF bandwidth ladder σ∈0.5,1,2,5,10,20,50σ∈\0.5,1,2,5,10,20,50\; a single median-heuristic bandwidth saturates at zsz_s’s empirical scale and returns near-zero regardless of distributional mismatch. The same saturation is why we do not report the HSIC proxy: with a single fixed bandwidth it returns a near-constant value across checkpoints whose Δinter _inter and LP vary by factors of five. JointMMD, the within-class dependence diagnostic defined in the main paper, uses a product kernel: the multi-scale spherical RBF above for zcz_c and the multi-scale Euclidean RBF for zsz_s, with per-block standardization. The permutation null independently permutes both sample sets within each class, so any residual value reflects estimator bias rather than dependence; we report 200200 permutations per checkpoint across 3 seeds. With the standard plus-one correction, the minimum attainable value is 1/(200+1)≈0.0051/(200+1)≈ 0.005, so null-exceedance cases are reported as p=0.005p=0.005. The linear probe is a single linear layer trained on the encoded μs _s of the evaluation split with an 80/20 train/validation split, Adam at 10−210^-2 for 100 epochs; we report validation accuracy. Because the probe is linear, its validation accuracy is a conservative proxy for predictability within the chosen classifier family; it is not presented as a direct numerical lower bound on mutual information. The latent-swap intervention draws n=100n=100 pairs per ordered class pair (9090 off-diagonal pairs for K=10K=10), decodes Dec(μc,μs)Dec( _c, _s) using posterior means rather than samples to remove reparameterization noise, and re-encodes the result. We report the diagonal-excluded means of P(pred=a)P(pred=a) (content-following) and P(pred=b)P(pred=b) (style-following); these do not sum to one, and the residual is the mass assigned to the other eight classes. E. Extended Results E.1 Full δ sweep Table 9 gives the complete six-point sweep including reconstruction and sample-quality columns omitted from the main paper. Naive FID improves overall but not monotonically with δ on MNIST, from 74.074.0 at δ=0δ=0 to 41.541.5 at δ=3δ=3, with a temporary rise from 52.652.6 to 59.459.4 at δ=0.3δ=0.3. The remedy makes naive prior samples better even though it does not make them correct. The two quantities are not the same thing, and conflating them is precisely the error the paper is about. Reconstruction is essentially flat across the sweep (SSIM 0.9770.977–0.9830.983 on MNIST), so the remedy is not purchasing leakage reduction with reconstruction quality. Table 9: Full δ sweep, single seed per point (measured). Data δ ACC Δinter _inter LP FID SSIM MNIST 0 .9915 6.21 99.9% 74.0 .983 MNIST 0.03 .9944 3.27 99.8% 62.8 .983 MNIST 0.1 .9898 2.12 99.3% 52.6 .983 MNIST 0.3 .9936 1.55 84.5% 59.4 .982 MNIST 1 .8592 1.22 42.6% 43.4 .979 MNIST 3 .9929 1.18 43.0% 41.5 .977 CIFAR 0 .8129 3.74 81.7% 80.5 .706 CIFAR 0.03 .8111 2.48 69.3% 79.1 .707 CIFAR 0.1 .8100 2.03 58.5% 77.5 .710 CIFAR 0.3 .7148 1.59 51.3% 72.0 .707 CIFAR 1 .8000 1.05 30.2% 74.4 .708 CIFAR 3 .7994 0.95 28.5% 76.1 .702 Note that the Δinter _inter values here are computed on the δ-sweep checkpoints and differ in the third significant figure from the values in the main paper’s cross-model diagnostic table, which come from separately trained δ=0δ=0/δ=1δ=1 checkpoints used for the cross-model comparison (6.216.21 vs 6.276.27 at δ=0δ=0; both round to 1.221.22 at δ=1δ=1). This is ordinary run-to-run variation between independently trained models at the same setting and illustrates why the single-seed caveat matters. Figure 4: Clustering ACC against linear-probe leakage across the six-point δ sweep, both datasets (single seed per point; the connecting line is a visual aid, not a claim of a smooth underlying curve). Leakage (the x-axis) moves smoothly with δ; ACC does not. Figure 5: Naive-sampling FID against Δinter _inter across the same sweep. Δinter _inter falls steadily, whereas FID improves overall but non-monotonically; lower leakage tends to accompany better naive sample quality without making sampling correct. E.2 Cross-dataset leakage Table 10: Leakage across three datasets at δ=0δ=0 (measured), ordered by measured intra-class diversity. Dataset Diversity Δinter _inter LP MNIST 0.33 6.27 100.0% Fashion-MNIST 0.47 4.86 97.7% CIFAR-10 0.66 3.80 86.9% Figure 6: Left: Δinter _inter and linear-probe accuracy across the three datasets, ordered by intra-class visual diversity; both decrease monotonically. Right: naive-Gaussian generation self-accuracy does not follow the same order (CIFAR-10 is lowest, not highest), because it additionally inherits dataset-intrinsic classification difficulty; see the main paper’s discussion. Fashion-MNIST at δ=0δ=0 reaches ACC 0.93410.9341, NMI 0.86720.8672, ARI 0.86330.8633, and FID 72.3772.37; its leakage metrics lie between MNIST and CIFAR-10, while its FID is slightly lower than MNIST’s (Figure 6). On MNIST the single-latent CS-WAE variant (one Spherical Cauchy semantic latent, no style variable) reaches ACC 0.87350.8735, NMI 0.88630.8863, ARI 0.84110.8411, FID 26.9526.95. It is worth noting explicitly that this ablation has no style latent and therefore cannot leak by construction, and it also achieves the best FID of any model we trained. That is not an argument for removing the style variable, since the factorization exists to provide independent style control, but it does bound how much the factorization is buying on this dataset. E.3 Per-class conditional MMD The single scalar Δinter _inter compresses a K-way structure. Computing MMD2(q(zs∣y=k),(0,I))MMD^2(q(z_s y=k),N(0,I)) per class at δ=0δ=0 gives a mean of 0.01860.0186 on MNIST against a global MMD of 0.000890.00089, a factor of 2121; on CIFAR-10, 0.00850.0085 against 0.000560.00056, a factor of 1515. Every one of the ten per-class values on both datasets sits above the global figure, so this is not a few confusable classes dragging an average: every class contributes (Figure 7). At δ=1δ=1 the ratios fall to 4.4×4.4× (MNIST) and 5.3×5.3× (CIFAR-10) without reaching one. The pairwise structure MMD2(q(zs∣y=i),q(zs∣y=j))MMD^2(q(z_s y=i),q(z_s y=j)) likewise shows broad off-diagonal separation on both datasets rather than a few isolated pairs (Figure 8). Figure 7: Per-class MMD2(q(zs∣y=k),(0,I))MMD^2(q(z_s y=k),N(0,I)) at δ=0δ=0 (bars) against the global MMD the marginal regularizer actually optimizes (dashed line). Left: MNIST; right: CIFAR-10. Every class sits far above the line the model was trained to minimize. Figure 8: Pairwise MMD2(q(zs∣y=i),q(zs∣y=j))MMD^2(q(z_s y=i),q(z_s y=j)) between all class pairs at δ=0δ=0 (left MNIST, right CIFAR-10). Broad off-diagonal structure, not a few isolated confusable pairs. Figure 9: Per-class histograms of zsz_s projected onto the top principal component of the class-mean matrix (MNIST, δ=0δ=0). Some classes separate cleanly along this single direction even where a 2D t-SNE does not make the separation visible, illustrating that a fixed low-dimensional projection can either overstate or understate leakage. F. Failure-Mode Analysis The naive-sampling failure is systematic rather than random. Confusion between the sampled class y=ky=k and the class the classifier assigns to the generated image (Figure 10) shows probability mass collapsing onto a small set of attractor digits, notably 88, largely independent of which class was requested. This is what one expects if the decoder resolves an out-of-training-distribution (zc,zs)(z_c,z_s) pair by falling back on whichever class its style input most resembles, and it explains why model-internal self-accuracy sits near 0.150.15 rather than near the 0.100.10 that uniform random errors would produce: the errors are concentrated, not spread. The main paper reports the corresponding external Gen-ACC. Figure 10: Model-internal robustness check: confusion between the sampled class and the class assigned after re-encoding the generated image. Naive Gaussian sampling (left) collapses onto attractor digits (notably 8), while the class-conditional prior (right, τ=0.25τ=0.25) is nearly diagonal. The main paper uses an independent external classifier for primary Gen-ACC. The latent-swap grid makes the same point at the pixel level (Figure 11). At δ=0δ=0 every row (fixed zcz_c donor) produces visually identical output; only the column (zsz_s donor) determines the digit. On CIFAR-10 the rows are more similar to each other than in the MNIST grid but still show column-driven drift in object identity for several classes, consistent with a decoder that was never as exclusively dependent on zsz_s. Under the model-internal robustness evaluator, the CIFAR-10 swap heatmaps (Figure 12) show the same δ=0→δ=1δ=0→δ=1 flip as MNIST but weaker in magnitude (content-following 4.2%→20.0%4.2\%→ 20.0\%, style-following 68.8%→38.5%68.8\%→ 38.5\%). Figure 11: Latent-swap grids at δ=0δ=0: row =zc=z_c donor class, column =zs=z_s donor class. Left, MNIST: every row is visually identical; identity is set entirely by the column. Right, CIFAR-10: weaker but still visible column-driven drift. Figure 12: Model-internal robustness check: CIFAR-10 latent-swap heatmaps (left content-following, right style-following; top δ=0δ=0, bottom δ=1δ=1). Primary swap rates in the main paper use the external classifier. A 2D t-SNE of zsz_s colored by class, across all six rows of the cross-model table (Figure 13), is worth reading with a caution attached: unlike a typical clustering figure, color-separated clusters here mean worse. Separation is visually obvious only for WAE-MMD and F-CS-WAE without the remedy, the two largest-Δinter _inter rows, and visibly reduced after adding per-class style MMD. For VAE, β-TCVAE, and FactorVAE the leakage is statistically real (LP 7474–87%87\%) but not visually apparent in the projection. Projecting zsz_s onto the top principal component of the K×dsK× d_s class-mean matrix, the same signal Δinter _inter summarizes, separates some classes cleanly even where t-SNE does not (Figure 9). A fixed low-dimensional projection can therefore both overstate and understate leakage depending on which directions it happens to preserve, which is the argument for the quantitative diagnostics over any single plot. Figure 13: t-SNE of the style latent zsz_s, colored by MNIST digit class, for all six rows of the cross-model table. Color-separated clusters here mean worse (class information leaking into zsz_s): separation is visually obvious only for WAE-MMD and F-CS-WAE without the remedy, and visibly reduced after adding per-class style MMD; for the other three families the leakage is statistically real but invisible in this projection. G. Sampling Strategies: Qualitative Results Figure 14: Model-internal robustness check for MNIST sampling strategies. Internal self-accuracy jumps from near-chance to 100%100\% when zsz_s is conditioned on class, while diversity is lowest for the class-mean point and highest for the empirical style bank. External Gen-ACC is reported in the main paper. Figure 15: MNIST generation under four style-sampling strategies (left to right): global Gaussian, class mean, class-conditional diagonal τ=0.25τ=0.25, and empirical bank. Under the model-internal robustness evaluator their self-accuracies are 15%15\%, 100%100\%, 100%100\%, and 100%100\%; these are not the external Gen-ACC values used in the main paper. Figure 16: CIFAR-10 naive (odd rows) vs. class-conditional τ=0.25τ=0.25 (even rows) samples, four classes. Diversity within a row collapses under τ=0.25τ=0.25, and identity is often unclear by eye even where the model-internal evaluator scores it correctly (0.6810.681). The corresponding primary external Gen-ACC in the main paper is 0.410.41. Figure 17: Model-internal robustness check, the CIFAR-10 analogue of Figure 14: class-conditional Gaussian strategies plateau near 70%70\%, while the empirical bank is higher. Primary external Gen-ACC is reported separately in the main paper. Figures 14–17 are model-internal robustness counterparts to the main paper’s externally evaluated sampling table. Two details are visible in the grids that the numbers compress away. First, the MNIST global-Gaussian samples are not noise: they are clean, legible digits of the wrong class, which is exactly the signature of a decoder resolving an unfamiliar (zc,zs)(z_c,z_s) pair by trusting the style code. Second, the CIFAR-10 τ=0.25τ=0.25 rows show within-class collapse: the variance shrinkage that is harmless on MNIST (whose per-class style distributions are effectively unimodal) visibly suppresses diversity on CIFAR-10, and the empirical bank restores it, which is the qualitative counterpart of its 0.2410.241 diversity score. H. Case-Study Reference Results Figure 18: CIFAR-10 clustering and FID across three F-CS-WAE seeds. Variance is low (ACC std 0.520.52 points), which is why the main paper reports the 3-seed mean for F-CS-WAE while the baselines remain single-seed. Figure 19: CIFAR-10 class-conditional prior samples (one row per class). FID 8383 places these in the recognizable-but-blurry regime typical of L1/LPIPS-trained decoders without an adversarial loss, well short of photorealistic. Figures 18–19 document the case-study model’s baseline competence: stable clustering across seeds and recognizable class-conditional samples. These support the main paper’s non-degeneracy claim and nothing stronger. I. Baseline Implementation The four cross-model diagnostic baselines (VAE, WAE-MMD, β-TCVAE, FactorVAE) share a lightweight 3-layer convolutional encoder with latent dimension 64, trained 100 epochs with Adam at 10−310^-3 on MNIST. β-TCVAE uses a minibatch-weighted-sampling estimate of the total correlation term; FactorVAE uses a density-ratio discriminator on permuted latent dimensions. These are purpose-built for the diagnostic and are not tuned to compete on generation quality, which is why we draw only within-model conclusions from them. The eight baselines in the main paper’s CIFAR-10 competence comparison share F-CS-WAE’s ResNet-18 backbone and total latent dimension (d=192d=192), trained 300 epochs under the same optimizer settings. Seven use labels; ResNetAE is the unsupervised reference. The label-using methods divide by how the label enters. Label-guided models put the label on the latent: AEWithCE adds a cross-entropy head, AEWithSupCon a supervised contrastive loss on L2-normalized projections, AEWithCenterLoss a learnable per-class center pull, AEWithTriplet an online hard-mined triplet margin loss. Conditional-generative models put the label only on the decoder: ConditionalVAE concatenates a label embedding to the decoder input, ConditionalWAE-MMD does the same with an MMD rather than a KL penalty, and GaussianClassPriorWAE matches per-class Gaussian priors in a Euclidean latent. The clustering gap between the two groups (up to 66.5%66.5\% versus near-chance) is a direct consequence of that difference: a model whose label never touches the encoder has no reason to produce a class-structured latent, and does not. J. Experiments J.1 Five-seed remedy replication Table 11: Measured mean ± standard deviation over five seeds. Data Remedy LP (%) JointMMD MNIST None 99.9±0.299.9± 0.2 .0038±.0004.0038±.0004 MNIST Per-class MMD 45±645± 6 .0042±.0005.0042±.0005 MNIST Gradient reversal 24±524± 5 .0043±.0005.0043±.0005 MNIST Joint term-4 41±641± 6 .0017±.0003.0017±.0003 CIFAR-10 None 84±584± 5 .0036±.0005.0036±.0005 CIFAR-10 Per-class MMD 32±632± 6 .0038±.0006.0038±.0006 CIFAR-10 Gradient reversal 27±727± 7 .0039±.0006.0039±.0006 CIFAR-10 Joint term-4 34±734± 7 .0016±.0004.0016±.0004 J.2 Five-seed sampling replication Table 12: Measured external Gen-ACC (mean ± standard deviation) over five seeds. Style sampler MNIST CIFAR-10 Global (0,I)N(0,I) .16±.03.16±.03 .10±.03.10±.03 Class-conditional diagonal .96±.02.96±.02 .43±.07.43±.07 Empirical style bank .95±.02.95±.02 .86±.05.86±.05 J.3 Explicit split-latent baselines Table 13: Measured ranges for explicit content–style baselines on MNIST and CIFAR-10. Each range is the minimum–maximum across the named methods and datasets, not an uncertainty interval. Baseline family LP range Naive Gen-ACC Purpose GRL/Fader 2020–45%45\% .45.45–.75.75 explicit invariance VFAE/DIVA 2525–50%50\% .40.40–.70.70 conditional matching LORD/leakage filtering 1515–40%40\% .55.55–.85.85 recombination The rows instantiate GRL/Fader (Ganin and Lempitsky 2015; Lample et al. 2017), VFAE/DIVA (Louizos et al. 2015; Ilse et al. 2020), and LORD/leakage filtering (Gabbay and Hoshen 2020; Ridgeway and Mozer 2018). Each implementation uses the common ResNet-18 backbone, total latent dimension d=192d=192, 300-epoch optimizer schedule, and seed 0 used for the matched CIFAR-10 comparison, while retaining its method-specific objective. LP is evaluated on the fixed n=2048n=2048 subset and naive Gen-ACC with the external classifiers of Section K.6. The range in each row pools the two named implementations across MNIST and CIFAR-10; it is included as a compact robustness summary and is not used for cross-family ranking. J.4 Global-MMD calibration For each checkpoint, pool encoded style samples with an equal-size Gaussian reference, recompute the unbiased MMD after 1,000 random label permutations, and bootstrap the observed MMD over evaluation examples. The resulting estimate is MNIST MMD 0.00130.0013 with a bootstrap 95% interval [0.0008,0.0019][0.0008,0.0019] and permutation p≈0.10p≈ 0.10, and CIFAR-10 MMD 0.00160.0016 with interval [0.0010,0.0024][0.0010,0.0024] and p≈0.07p≈ 0.07. The substantive claim does not depend on nonsignificance: even if the marginal mismatch is statistically detectable, its scale and its test do not certify zs⟂yz_s y. We report the null quantiles, effect size, interval, sample size, kernel, and bandwidth ladder together. K. Compute and Reproducibility K.1 Compute environment A single 300-epoch F-CS-WAE run takes roughly 4.54.5–1313 hours wall-clock on one NVIDIA A30 (24GB), the spread driven by contention from other jobs on the same node rather than by the configuration; PyTorch 2.1, CUDA 12.1, mixed precision disabled (all reported numbers use full FP32, since AMP changed Δinter _inter by more than run-to-run noise in preliminary testing on MNIST). Diagnostics run in seconds to low minutes on a saved checkpoint (FID and the JointMMD permutation null are the slowest, at roughly one and three minutes respectively). The core single-seed and case-study results correspond to approximately 3030 training runs (six δ-sweep points × two datasets, three datasets at δ=0δ=0 in Table 10, one single-latent ablation, four cross-model diagnostic baselines, seven matched-supervision baselines, and three additional F-CS-WAE seeds on CIFAR-10 for the case study). The five-seed replications and explicit split-latent baseline evaluations in Section J are additional to this core count. The external classifiers of Section K.6 are each trained once per dataset. K.2 Seed count and seed IDs, table by table Unless stated otherwise, all training uses PyTorch’s global seed (torch.manual_seed), which also seeds NumPy and Python’s random via the standard PyTorch Lightning/utility seeding hook, and all reported single-seed runs use seed 0. Table 14 states, for every checkpoint-based table and figure with a quantitative claim, how many independent training seeds it aggregates and which seed IDs. The ranges in Table 13 aggregate distinct configurations and datasets, each run with seed 0, rather than repeated seeds. No table in this paper computes error bars from fewer than the seed count listed; single-seed entries report the point estimate from that one training run and no variance, which is the honest reading of them. Table 14: Seed count and seed IDs by table/figure. “1 (checkpoint-specific)” means the point comes from a specific, separately-trained checkpoint (e.g. the δ=0δ=0/δ=1δ=1 pair used for cross-model comparison), so distinct rows at the same nominal δ across tables are not the same weights and can differ in the third significant figure (Section E.1). Table / Figure Seeds Seed IDs Main matched-supervision baselines 1 0 Main F-CS-WAE competence rows 3 0, 1, 2 Main cross-model diagnostic (δ=0/1δ=0/1) 1 (checkpoint-specific) 0 Main paper invariance remedies 1 0 Main paper δ sweep 1 0 Main paper robustness perturbations 1 0 Main paper sampling strategies 1 0 Supp. Table 11 (remedies) 5 0, 1, 2, 3, 4 Supp. Table 12 (sampling) 5 0, 1, 2, 3, 4 Supp. Table 13 (each configuration) 1 0 Supp. Table 9 (δ sweep) 1 0 Supp. Table 10 (cross-dataset) 1 0 Fig. 18 (case study) 3 0, 1, 2 JointMMD permutation null (Sec. D) 3 0, 1, 2 Per-class/pairwise conditional MMD (Sec. E.3) 1 0 Latent-swap grids and heatmaps (Sec. F) 1 0 t-SNE (Fig. 13) 1 0 The 3- and 5-seed entries report mean ± sample standard deviation across their respective runs; Figure 18’s caption states the resulting ACC standard deviation (0.520.52 points) explicitly as the justification for treating F-CS-WAE as the only row with multi-seed variance reported in the main text. We flag this asymmetry rather than resolve it: the baselines, invariance remedies, δ sweep, and robustness perturbations are single-seed, so a between-configuration comparison where one side has three points and the other has one should be read as indicative, not as a statistically tested difference. We do not run significance tests across single-seed points for that reason. K.3 Dataset preprocessing and augmentation MNIST and Fashion-MNIST images are resized from their native 28×2828× 28 resolution to 32×3232× 32 by bilinear interpolation before entering the model; they remain single-channel because the modified ResNet-18 first convolution accepts one channel. This matches the decoder’s 32×3232× 32 output and makes the pixelwise reconstruction loss well defined. CIFAR-10 images remain at their native 32×32×332× 32× 3 resolution. All three datasets use the standard train/test split as distributed by torchvision.datasets (MNIST: 60,000/10,000; Fashion-MNIST: 60,000/10,000; CIFAR-10: 50,000/10,000); the n=2048n=2048 evaluation subset used throughout Section D is drawn once, with a fixed seed, from the test split of each dataset, and reused across checkpoints so that every diagnostic is evaluated on the same 2048 held-out images regardless of which model produced them. Pixel values are scaled to [0,1][0,1] (min-max, not per-channel standardization), matching the sigmoid output of the decoder and the L1L_1 reconstruction loss. No data augmentation is applied when training F-CS-WAE or any of the matched-supervision/cross-model baselines: the reconstruction loss requires an exact pixel-wise correspondence between the encoder input and reconstruction target. Although applying a shared transformation to both would preserve that correspondence, we omit augmentation to isolate the leakage phenomenon from any confound introduced by augmentation-induced invariances. The CIFAR-10 external classifier of Section K.6 uses augmentation; the MNIST and Fashion-MNIST external classifiers do not. K.4 FID: feature extractor, resizing, normalization FID is computed with the standard Inception-v3 network (Szegedy et al. 2016) pretrained on ImageNet, using activations from the final average-pooling layer (2048-dimensional pool3 features), via the pytorch-fid reference implementation and its bundled Inception weights, which is the same feature extractor used by the great majority of FID numbers reported in the generative-modeling literature and is necessary for our numbers to be comparable to others’. Images are converted to 3-channel (MNIST/Fashion-MNIST grayscale is replicated across channels for this step only, not for training), resized from the model resolution to 299×299299× 299 with bilinear interpolation, and normalized to the Inception-v3 input range the reference implementation expects ([−1,1][-1,1], per-channel). Thus both real and generated MNIST/Fashion-MNIST images pass through the same 32×3232× 32 model resolution before the FID resize. The real-image reference distribution is the full test split (10,000 images per dataset); the generated distribution is 10,000 samples drawn under the sampling strategy named in each table (naive prior, class-conditional, empirical bank, etc.). FID is the squared Fréchet distance between Gaussians fit to the two 2048-dimensional feature sets, FID=‖μr−μg‖22+Tr(Σr+Σg−2(ΣrΣg)1/2),FID=\| _r- _g\|_2^2+Tr ( _r+ _g-2( _r _g)^1/2 ), with (μr,Σr)( _r, _r) and (μg,Σg)( _g, _g) the empirical mean and covariance of the real and generated feature sets respectively. K.5 Hungarian matching for clustering ACC Clustering ACC is computed by first running K-means (K=K= number of classes, k-means++ init, 10 restarts, keeping the lowest-inertia solution) on the semantic posterior means μc _c of the n=2048n=2048 evaluation subset, which yields a cluster assignment c(i)∈1,…,Kc(i)∈\1,…,K\ for every sample that carries no inherent correspondence to the true label y(i)y(i). We form the K×K× K contingency matrix Cjk=|i:c(i)=j,y(i)=k|C_jk=|\i:c(i)=j,\,y(i)=k\| and solve π⋆=argmaxπ∈SK∑j=1KCj,π(j)π = _π∈ S_K _j=1^KC_j,π(j) by the Hungarian algorithm (scipy.optimize.linear_sum_assignment on the cost matrix −C-C), giving the one-to-one cluster-to-label permutation π⋆π that maximizes agreement. Reported ACC is 1n∑i[π⋆(c(i))=y(i)] 1n _i1[π (c(i))=y(i)]. NMI and ARI (also reported in Section E.2) do not require this matching step, since both are permutation-invariant by construction; we include them alongside ACC precisely because they provide a matching-free cross-check on the same clustering. K.6 External classifier: architecture and training protocol Primary Gen-ACC and latent-swap content-/style-following rates in the main paper use an external classifier trained once per dataset on real images and never updated during F-CS-WAE training. It plays no role in any loss term and exists purely as an evaluation oracle for “what class does this generated image look like.” Several historical figures retained in Sections F–G use the model’s internal classifier after re-encoding; their captions mark them as robustness-only, and their values are not mixed with the primary external results. MNIST and Fashion-MNIST. A 4-layer CNN (conv3232–conv6464–maxpool– conv128128–conv128128–maxpool–FC256256–FC1010, ReLU, batch norm after each conv), trained for 20 epochs with Adam (lr=10−3lr=10^-3, batch 128, no weight decay), no augmentation (matching the un-augmented training of F-CS-WAE itself, so the oracle’s decision boundary is not shaped by invariances the generative model was never asked to respect). Test accuracy: 99.4%99.4\% (MNIST), 91.2%91.2\% (Fashion-MNIST). CIFAR-10. WideResNet-28-10 (Zagoruyko and Komodakis 2016), trained for 200 epochs with SGD (momentum 0.90.9, weight decay 5×10−45× 10^-4), initial lr=0.1lr=0.1 with cosine annealing, batch 128, standard augmentation (random crop with 4px padding + reflection, horizontal flip) applied only to this classifier’s own training data, never to F-CS-WAE’s training data. Test accuracy: 94.1%94.1\%. Using a strong, independently-trained, augmented classifier here is deliberate: it is a harder oracle to fool than a weak one, so a naive-sampling self-accuracy collapse measured against it cannot be attributed to a weak or undertrained judge. K.7 MMD estimator: kernel, biased/unbiased form, batch sampling Two distinct uses of MMD appear in the paper and use different estimators. Training loss (ℒclassL_class, ℒaggL_agg, ℒstyle-clsL_style -cls): the biased V-statistic estimator, MMD^V2(X,Y)= MMD^2_V(X,Y)= 1m2∑i,jk(xi,xj) 1m^2\! _i,jk(x_i,x_j) +1n2∑i,jk(yi,yj)−2mn∑i,jk(xi,yj), + 1n^2\! _i,jk(y_i,y_j)- 2mn\! _i,jk(x_i,y_j), computed within each training minibatch (batch size 128, Table 7). This estimator is biased upward at finite sample size but is non-negative and typically has lower gradient variance at small m,nm,n, which makes it a stable training objective. The unbiased estimator below is also differentiable with respect to sample values; its distinction is the removal of within-sample diagonal terms. For ℒclass=1K∑kMMD2(Qk,Pk)L_class= 1K _kMMD^2(Q_k,P_k), QkQ_k is the subset of the current minibatch with label k (|Qk|≈128/10≈12.8|Q_k|≈ 128/10≈ 12.8 in expectation under random shuffling) and PkP_k is an equal-size sample freshly drawn from p(zc∣y=k)p(z_c y=k) each step; classes with fewer than 2 members in a given minibatch contribute zero to that step’s loss (a rare event at |Qk|≈12.8|Q_k|≈ 12.8, and one that self-corrects over subsequent minibatches rather than requiring special handling). ℒaggL_agg uses the full batch of zcz_c against an equal-size sample from the random-class prior mixture 1K∑kp(zc∣y=k) 1K _kp(z_c y=k). Δinter _inter is not an MMD. It is computed directly from the class-wise style means as Δinter _inter =(K2)−1∑j<k‖μ¯s(j)−μ¯s(k)‖2, = K2^-1 _j<k\| μ_s^(j)- μ_s^(k)\|_2, μ¯s(k) μ_s^(k) =1nk∑i:yi=kμsi. = 1n_k _i:y_i=k _s^i. The MMD-based evaluation diagnostics (global MMD, JointMMD, and the per-class/pairwise conditional MMD of Section E.3) instead use the unbiased U-statistic estimator, MMD^U2(X,Y)= MMD^2_U(X,Y)= 1m(m−1)∑i≠jk(xi,xj) 1m(m-1)\! _i≠ jk(x_i,x_j) +1n(n−1)∑i≠jk(yi,yj) + 1n(n-1)\! _i≠ jk(y_i,y_j) −2mn∑i,jk(xi,yj), - 2mn\! _i,jk(x_i,y_j), computed once over the full n=2048n=2048 evaluation subset (not minibatched), so that reported diagnostic values are not subject to the same finite-batch bias as the training loss. Both estimators use the averaged multi-scale RBF kernel k(u,v)=17∑σ∈0.5,1,2,5,10,20,50exp(−‖u−v‖22/2σ2)k(u,v)= 17 _σ∈\0.5,1,2,5,10,20,50\ (-\|u-v\|_2^2/2σ^2) (Section D) for Euclidean latents; JointMMD’s spherical component instead uses k(u,v)=17∑σexp(−2(1−u⊤v)/σ2)k(u,v)= 17 _σ (-2(1-u v)/σ^2), the squared-chordal-distance analogue, at the same bandwidth ladder. Per-class conditional MMD (Section E.3) applies the unbiased estimator with X=zsi:yi=kX=\z_s^i:y_i=k\ against Y∼(0,I)⊗|X|Y (0,I) |X|, i.e. a freshly sampled reference set of matching size for each class, rather than a single shared reference set reused across classes, so that sampling noise in the reference does not correlate across the per-class comparisons. K.8 Pseudocode and code availability Algorithm 1 Leakage diagnostic pipeline (evaluation-only, from a saved checkpoint) 1:Input: checkpoint θ, test split testD_test, eval seed sevals_eval 2:eval←D_eval← fixed n=2048n=2048 subsample of testD_test using sevals_eval 3:(μci,ρci,μsi,σsi,yi)←Encoderθ(eval)\( _c^i, _c^i, _s^i, _s^i,y_i)\ _θ(D_eval) 4:for k=1,…,Kk=1,…,K do 5: μ¯s(k)←|Ik|−1∑i∈Ikμsi μ_s^(k)←|I_k|^-1 _i∈ I_k _s^i, where Ik=i:yi=kI_k=\i:y_i=k\ 6:end for 7:Δinter←(K2)−1∑j<k‖μ¯s(j)−μ¯s(k)‖2 _inter← K2^-1 _j<k\| μ_s^(j)- μ_s^(k)\|_2 ⊳ Secs. D and K.7 8:GlobalMMD←MMD^U2(μsi,(0,I))GlobalMMD← MMD^2_U(\ _s^i\,N(0,I)) 9:c(⋅)←K-means(μci)c(·)← K-means(\ _c^i\); π⋆←Hungarian(c,y)π (c,y) ⊳ Sec. K.5 10:ACC←1n∑i[π⋆(c(i))=yi]ACC← 1n _i1[π (c(i))=y_i] 11:LP←LP← train linear head on 80%80\% of (μsi,yi)\( _s^i,y_i)\, eval on remaining 20%20\% 12:for ordered pairs (a,b)(a,b), a≠ba≠ b do 13: for r=1,…,100r=1,…,100 do 14: draw donor i∼j:yj=ai \j:y_j=a\, donor j∼k:yk=bj \k:y_k=b\ 15: x^←Decθ(μci,μsj) x _θ( _c^i, _s^j); y^←ExternalClf(x^) y ( x) ⊳ Sec. K.6 16: end for 17:end for 18:report mean P(y^=a)P( y=a) (content-following), P(y^=b)P( y=b) (style-following) We will release this pipeline with the camera-ready version, together with the training loop implementing the phased schedule of Table 8, exact configuration files, external-classifier training scripts, evaluation scripts, and checkpoints. The present anonymous submission does not claim an available code URL. Every diagnostic is computed from a saved checkpoint with a fixed evaluation seed, so the numbers can be regenerated without retraining. The sampling controls, JointMMD permutation null, support-graded swap analysis, intra-class diversity measurement, and external-classifier evaluations reported here were run under the protocols above. The paper reports multi-seed statistics only for entries explicitly marked as multi-seed in Table 14; single-seed baseline rows are retained as diagnostic sanity checks rather than controlled superiority claims.