Paper deep dive
Historical Consensus: Preventing Posterior Collapse via Iterative Selection of Gaussian Mixture Priors
Zegu Zhang, Jian Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/22/2026, 6:19:25 AM
Summary
The paper introduces 'Historical Consensus Training', a novel framework to prevent posterior collapse in Variational Autoencoders (VAEs) by leveraging the multiplicity of Gaussian Mixture Model (GMM) clusterings. By iteratively training the VAE to satisfy multiple distinct clustering constraints, the model develops a 'historical barrier' in parameter space that excludes collapsed solutions, ensuring non-collapsed representations even when trained with a single objective.
Entities (5)
Relation Signals (3)
Historical Consensus Training → utilizes → Gaussian Mixture Model
confidence 99% · leveraging the multiplicity of Gaussian mixture model (GMM) clusterings
Historical Consensus Training → prevents → Posterior Collapse
confidence 98% · We introduce Historical Consensus Training, a novel framework that prevents posterior collapse
Historical Barrier → excludes → Posterior Collapse
confidence 95% · We prove that this barrier excludes the collapsed solution
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Variational autoencoders (VAEs) frequently suffer from posterior collapse, where latent variables become uninformative and the approximate posterior degenerates to the prior. Recent work has characterized this phenomenon as a phase transition governed by the spectral properties of the data covariance matrix. In this paper, we propose a fundamentally different approach: instead of avoiding collapse through architectural constraints or hyperparameter tuning, we eliminate the possibility of collapse altogether by leveraging the multiplicity of Gaussian mixture model (GMM) clusterings. We introduce Historical Consensus Training, an iterative selection procedure that progressively refines a set of candidate GMM priors through alternating optimization and selection. The key insight is that models trained to satisfy multiple distinct clustering constraints develop a historical barrier -- a region in parameter space that remains stable even when subsequently trained with a single objective. We prove that this barrier excludes the collapsed solution, and demonstrate through extensive experiments on synthetic and real-world datasets that our method achieves non-collapsed representations regardless of decoder variance or regularization strength. Our approach requires no explicit stability conditions (e.g., $\sigma^{\prime 2} < \lambda_{\max}$) and works with arbitrary neural architectures. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.10935v2
- Canonical: https://arxiv.org/abs/2603.10935v2
Trouble viewing inline? Open PDF directly →
Full Text
39,942 characters extracted from source content.
Expand or collapse full text
Historical Consensus: Preventing Posterior Collapse via Iterative Selection of Gaussian Mixture Priors Zegu Zhang ( zeguzhang@outlook.com) Jian Zhang (tsegoochang2000@gmail.com) Abstract Variational autoencoders (VAEs) frequently suffer from posterior collapse, where latent variables become uninformative and the approximate posterior degenerates to the prior. Recent work has characterized this phenomenon as a phase transition governed by the spectral properties of the data covariance matrix. In this paper, we propose a fundamentally different approach: instead of avoiding collapse through architectural constraints or hyperparameter tuning, we eliminate the possibility of collapse altogether by leveraging the multiplicity of Gaussian mixture model (GMM) clusterings. We introduce Historical Consensus Training, an iterative selection procedure that progressively refines a set of candidate GMM priors through alternating optimization and selection. The key insight is that models trained to satisfy multiple distinct clustering constraints develop a historical barrier—a region in parameter space that remains stable even when subsequently trained with a single objective. We prove that this barrier excludes the collapsed solution, and demonstrate through extensive experiments on synthetic and real-world datasets that our method achieves non-collapsed representations regardless of decoder variance or regularization strength. Our approach requires no explicit stability conditions (e.g., σ′2<λmaxσ 2< _ ) and works with arbitrary neural architectures. The code is available at https://github.com/tsegoochang/historical-consensus-vae. 1 Introduction Variational autoencoders (VAEs) (Kingma and Welling, 2013) are a cornerstone of deep generative modeling, combining variational inference with neural networks to learn latent representations of high-dimensional data. Despite their widespread success, VAEs are plagued by posterior collapse (Bowman et al., 2015; Lucas et al., 2019), a phenomenon where the approximate posterior qϕ(z|x)q_φ(z|x) becomes indistinguishable from the prior p(z)p(z), rendering the latent variables uninformative. Recent theoretical advances have revealed that posterior collapse is not merely an optimization artifact but a phase transition governed by the interplay between data structure and model hyperparameters (Li et al., 2026). For deep Gaussian VAEs, collapse occurs when the decoder variance σ′2σ 2 exceeds the largest eigenvalue λmax _ of the data covariance matrix. This insight has led to practical guidelines: to avoid collapse, one must ensure σ′2<λmaxσ 2< _ . However, this condition is inherently restrictive—it imposes a hard constraint on model architecture and hyperparameters. Moreover, it addresses collapse by avoiding the unstable region rather than eliminating the possibility of collapse itself. In parallel, a separate line of research has explored combining VAEs with Gaussian mixture models (GMMs) to enhance representational power (Liu et al., 2023; Dilokthanakul et al., 2016). These models treat the latent space as a mixture of Gaussians, enabling clustering and generation within a unified framework. However, they do not directly address the posterior collapse problem. Our key observation is that GMM clustering of the same dataset yields multiple distinct solutions due to random initialization and the non-convexity of the EM algorithm. These solutions represent different but equally valid ways to partition the data. We argue that this multiplicity, often viewed as a nuisance, is actually a valuable resource for preventing posterior collapse. Main idea. We propose to train a VAE by iteratively selecting among multiple GMM clustering results. Starting with a diverse set of candidate priors, we alternate between: • Training the VAE to satisfy all current clustering constraints (via a conditioned loss), • Evaluating the model’s performance on each constraint, • Retaining only the best-performing half of the candidates. This process continues until only two candidates remain, followed by refinement training to an extremely low loss threshold (<10−5<10^-5), and finally training with a single candidate. Why it works. Models trained through this procedure develop what we call a historical barrier: the parameter trajectory is constrained by the requirement to satisfy all previously selected clustering constraints. The collapsed solution, which would incur high loss on these historical constraints, lies outside the feasible region. Remarkably, even when finally trained with a single clustering objective, the model remains non-collapsed, held in place by the memory of its training history. Contributions. 1. We introduce Historical Consensus Training, a novel framework that prevents posterior collapse by leveraging the multiplicity of GMM clusterings. 2. We prove the existence of a historical barrier that separates non-collapsed solutions from collapsed ones, and show that models trained with our method reside in the non-collapsed region. 3. We demonstrate empirically that our method achieves non-collapsed representations without any explicit stability conditions (e.g., σ′2<λmaxσ 2< _ ), across multiple datasets and architectures. 4. We provide theoretical and experimental evidence that even when reduced to a single clustering objective, models retain their non-collapsed state due to historical inertia. 2 Related Work 2.1 Posterior Collapse in VAEs Posterior collapse has been extensively studied since the introduction of VAEs (Bowman et al., 2015). Early explanations focused on the KL divergence term in the ELBO, which encourages the posterior to match the prior (Higgins et al., 2017) . This led to heuristic solutions such as KL annealing (Huang et al., 2018) and β-VAE (Higgins et al., 2017). More recent work has provided rigorous theoretical analyses. Lucas et al. (2019) studied linear VAEs and identified the role of the log marginal likelihood. Dai et al. (2020) examined local optima. Crucially, Li et al. (2026) characterized posterior collapse as a phase transition, deriving the condition σ′2>λmaxσ 2> _ for collapse onset. Ichikawa and Hukushima (2025) extended this analysis using the replica method, revealing dataset-size dependence. Despite these advances, all existing approaches treat collapse as something to be avoided through careful design. Our work is the first to eliminate collapse by leveraging solution multiplicity. 2.2 Gaussian Mixture VAEs Combining VAEs with mixture priors has a rich history. The GMVAE (Dilokthanakul et al., 2016) introduces a categorical variable to model cluster assignments. Liu et al. (2023) proposed Cloud-VAE with concept embeddings. Liu et al. (2024) introduced dimension-weighting for disentanglement. These works focus on enhancing latent representations, not on preventing collapse. Our work differs in that we use multiple GMM clusterings of the same data as training constraints, rather than as a fixed prior. 2.3 Multi-Task and Continual Learning Our iterative selection procedure bears resemblance to multi-task learning (Caruana, 1997) and curriculum learning (Bengio et al., 2009). The key difference is that our tasks (clustering constraints) are generated from the same data, ensuring compatibility. The concept of historical barrier is related to elastic weight consolidation (Kirkpatrick et al., 2017) in continual learning, where past tasks constrain future optimization. 3 Preliminaries 3.1 Variational Autoencoders A VAE models data ∈ℝNx ^N using latent variables ∈ℝnz ^n with prior p()=(0,I)p(z)=N(0,I). The encoder qϕ(|)q_φ(z|x) approximates the true posterior, and the decoder pθ(|)p_θ(x|z) generates reconstructions. Training maximizes the evidence lower bound (ELBO): ELBO()=qϕ(|)[logpθ(|)]−KL(qϕ(|)∥p()).ELBO(x)=E_q_φ(z|x)[ p_θ(x|z)]-KL(q_φ(z|x)\|p(z)). (1) For Gaussian VAEs, we have: pθ(|) p_θ(x|z) =(;fθ(),σ′2I), =N(x;f_θ(z),σ 2I), (2) qϕ(|) q_φ(z|x) =(;μϕ(),diag(σϕ2())). =N(z; _φ(x),diag( _φ^2(x))). (3) 3.2 Posterior Collapse as Phase Transition Li et al. (2026) showed that posterior collapse corresponds to the stability of the trivial solution qϕ(|)=p()q_φ(z|x)=p(z), pθ(|)=pdata()p_θ(x|z)=p_data(x). By analyzing perturbations around this solution, they derived the collapse condition: σ′2>λmax,σ 2> _ , (4) where λmax _ is the largest eigenvalue of the data covariance matrix Σ=pdata[(x−⟨x⟩)(x−⟨x⟩)T] =E_p_data[(x- x )(x- x )^T]. 3.3 Gaussian Mixture Models A GMM with K components represents a distribution as: p(x)=∑k=1Kπk(x;μk,Σk),p(x)= _k=1^K _kN(x; _k, _k), (5) with ∑kπk=1 _k _k=1, πk≥0 _k≥ 0. Parameters are typically estimated via the EM algorithm, which is sensitive to initialization and yields multiple local optima. 3.4 GMM-Conditioned VAE Training Given a GMM clustering result =πk,μk,Σkk=1KC=\ _k, _k, _k\_k=1^K, we can define a conditioned loss that encourages the VAE’s reconstructions to be consistent with this clustering: ℒ(θ,ϕ)=pdata(x)[z∼qϕ[logpθ(x|z)]]−KL(qϕ∥p)+λ⋅pdata[mink‖x^−μk‖Σk−12],L_C(θ,φ)=E_p_data(x) [E_z q_φ[ p_θ(x|z)] ]-KL(q_φ\|p)+λ·E_p_data [ _k\| x- _k\|^2_ _k^-1 ], (6) where x x is the reconstruction and ∥⋅∥Σk−1\|·\|_ _k^-1 is the Mahalanobis distance. For simplicity, we use the Euclidean distance to the nearest component mean in this work. 4 Method: Historical Consensus Training 4.1 Motivation: Multiplicity as a Resource Given a dataset =xii=1MD=\x_i\_i=1^M, running EM with different initializations yields a set of clustering results 1,2,…,R\C_1,C_2,…,C_R\. These results differ in their component parameters but all achieve comparable likelihood. We treat this multiplicity as a feature, not a bug. If we train a VAE to satisfy multiple distinct clustering constraints simultaneously, the model must develop a representation flexible enough to accommodate all of them. This flexibility inherently precludes posterior collapse, as the collapsed solution cannot satisfy diverse constraints. 4.2 The Selection Pipeline Our method proceeds in three stages, illustrated in Figure 1. Figure 1: Overview of Historical Consensus Training. (1) Run EM multiple times to obtain diverse clusterings. (2) Iteratively train the VAE with all current clusterings and retain the best half. (3) Refine with the final two clusterings to ultra-low loss. (4) Train with a single clustering to verify non-collapse. 4.2.1 Stage 1: Power-of-Two Selection Let R0=2kR_0=2^k be the initial number of clustering results. For t=0,1,…,T−1t=0,1,…,T-1: 1. Train the VAE for E epochs, cycling through all clustering results in RtR_t. For each epoch, we minimize: ℒtotal=ℒVAE+β⋅ℒ(x,x^),L_total=L_VAE+β·L_C(x, x), (7) where ℒL_C is the clustering consistency loss. 2. After training, evaluate the model’s performance on each clustering result ∈RtC∈ R_t: ℓ=1||∑x∈mink‖x^(x)−μk‖2. _C= 1|D| _x _k\| x(x)- _k^C\|^2. (8) 3. Retain the |Rt|/2|R_t|/2 clustering results with the smallest ℓ _C. Set Rt+1R_t+1 to this subset. Continue until |Rt|=2|R_t|=2. 4.2.2 Stage 2: Consensus Refinement With the final two clustering results a,bC_a,C_b, continue training until: max(ℓa,ℓb)<ϵ, ( _C_a, _C_b)<ε, (9) where ϵε is a very small threshold (e.g., 10−510^-5). This ensures the model satisfies both constraints to high precision. 4.2.3 Stage 3: Final Single-Cluster Training As a stress test, we continue training with only aC_a (or bC_b) for an additional EfinalE_final epochs. We monitor the KL divergence to detect any signs of collapse. Algorithm 1 Historical Consensus Training 0: Dataset D, number of initial clusterings R=2kR=2^k, epochs per cycle E, refinement threshold ϵε 0: Trained VAE model θ,ϕθ,φ Run EM R times with different seeds to get 1,…,R\C_1,…,C_R\ Initialize VAE parameters θ,ϕθ,φ S←1,…,RS←\C_1,…,C_R\ while |S|>2|S|>2 do for epoch=1epoch=1 to E do for ∈SC∈ S do Sample batch x∼x Compute VAE loss ℒVAEL_VAE and clustering loss ℒL_C Update θ,ϕθ,φ using ∇(ℒVAE+βℒ)∇(L_VAE+ _C) end for end for Compute ℓ _C for each ∈SC∈ S Keep only the |S|/2|S|/2 clusterings with smallest ℓ _C end while Let a,bC_a,C_b be the remaining clusterings while max(ℓa,ℓb)>ϵ ( _C_a, _C_b)>ε do Train as above, alternating between aC_a and bC_b end while (Optional) Train with only aC_a for EfinalE_final epochs return θ,ϕθ,φ 4.3 Theoretical Analysis: The Historical Barrier Definition 4.1 (Historical Loss). For a model with parameters Θ , the historical loss with respect to a set of clustering results S is: ℒS(Θ)=max∈Sℓ(Θ).L_S( )= _C∈ S _C( ). (10) Definition 4.2 (Feasible Region). The feasible region after stage t is: ℱt=Θ:ℒSt(Θ)≤ϵt,F_t=\ :L_S_t( )≤ _t\, (11) where StS_t is the set of retained clusterings and ϵt _t is the maximum loss achieved during training. Lemma 4.3 (Nested Feasible Regions). The feasible regions are nested: ℱT⊂ℱT−1⊂⋯⊂ℱ0F_T _T-1⊂·s _0. Proof. Each stage retains only clusterings with the smallest losses, and training ensures ℒSt(Θt)≤ϵtL_S_t( _t)≤ _t. Since St+1⊂StS_t+1⊂ S_t, any Θ satisfying ℒSt+1(Θ)≤ϵt+1L_S_t+1( )≤ _t+1 also satisfies ℒSt(Θ)≤max(ϵt,ϵt+1)L_S_t( )≤ ( _t, _t+1), but the training process ensures ϵt+1≤ϵt _t+1≤ _t, hence ℱt+1⊂ℱtF_t+1 _t. ∎ Theorem 4.4 (Exclusion of Collapsed Solutions). Let Θcollapse _collapse be any collapsed solution (i.e., qϕ(z|x)=p(z)q_φ(z|x)=p(z)). Then there exists a constant δ>0δ>0 such that for any clustering result C, ℓ(Θcollapse)≥δ _C( _collapse)≥δ. If ϵT<δ _T<δ, then Θcollapse∉ℱT _collapse _T. Proof. At a collapsed solution, reconstructions are independent of z and equal to pdata[x]E_p_data[x] (for optimal decoder) or some constant. The distance to any non-trivial cluster mean μk _k^C is bounded below by the minimum separation between cluster means and the data mean, which is positive for any reasonable clustering. Taking δ as the minimum over all ∈S0C∈ S_0 of this distance gives the result. ∎ Corollary 4.5 (Historical Inertia). If Θ∗∈ℱT ^* _T, then even when trained subsequently with a single clustering ∉STC∉ S_T, gradient descent cannot reach Θcollapse _collapse without passing through regions where ℒST(Θ)>ϵTL_S_T( )> _T. This corollary explains our empirical finding: models refined on multiple constraints retain a “memory” that prevents collapse even when the constraints are removed. 4.4 Why Power-of-Two? The choice of R0=2kR_0=2^k and halving each round is deliberate: • It creates a clear hierarchy of constraints, from diverse to focused. • The exponential decay ensures rapid convergence in number of rounds (log2R0 _2R_0 rounds total). • It guarantees we end with exactly 2 clusterings, enabling pairwise refinement. 5 Experiments 5.1 Experimental Setup 5.1.1 Datasets We evaluate on: • Synthetic GMM: 50,000 samples from an 8-component GMM in 32 dimensions, with component variances controlled to be below a threshold. • MNIST: 60,000 grayscale images, resized to 14×1414× 14 (196 dimensions). • Fashion-MNIST: 70,000 grayscale images, resized to 14×1414× 14. • CIFAR-10: 60,000 color images, converted to grayscale and resized to 8×88× 8 (64 dimensions) following Li et al. (2026). 5.1.2 Baselines We compare against: • Vanilla VAE (Kingma and Welling, 2013) • β-VAE (Higgins et al., 2017) with β∈1,2,4,8β∈\1,2,4,8\ • KL Annealing (Huang et al., 2018) • EM-type VAE (Li et al., 2023) 5.1.3 Architecture For fair comparison, all methods use the same encoder/decoder architecture per dataset: • Synthetic/MNIST/Fashion: MLP with hidden layers [256,128] for encoder, [128,256] for decoder. • CIFAR-10: Convolutional encoder (two Conv+ReLU layers) and transposed convolutional decoder. Latent dimension n=8n=8 for all experiments. 5.1.4 Evaluation Metrics • KL divergence: Average DKL(q(z|x)∥p(z))D_KL(q(z|x)\|p(z)) over test set. Values near zero indicate collapse. • Active units: Number of latent dimensions with [σj2(x)]<0.99E[ _j^2(x)]<0.99 (Burda et al., 2015). • Reconstruction error: Mean squared error between input and reconstruction. • FID (for CIFAR-10): Frechet Inception Distance (Heusel et al., 2017). 5.2 Quantitative Results We evaluate our method on three datasets under the violating condition σ′2=2λmaxσ 2=2 _ . Table 1 shows that our method achieves DKL>2.0D_KL>2.0 on MNIST and Fashion-MNIST, while vanilla VAE collapses completely (DKL<0.01D_KL<0.01). Active Units Analysis. Despite the high KL values, we observe that the number of active latent units remains limited. On MNIST, only 2 out of 48 dimensions are active (variance >0.005>0.005), with the remaining dimensions having near-zero variance (∼10−5 10^-5). Similarly, Fashion-MNIST has 3 active units and CIFAR-10 has 3 active units. This indicates that while our method successfully prevents complete collapse, the model tends to concentrate information into a small subset of dimensions. Improving the distribution of information across latent dimensions remains an important direction for future work. 5.3 Verification of Historical Barrier Figure 2: Verification of historical barrier. Left: Loss on discarded clusterings remains low throughout training, indicating memory. Right: Distance to collapsed solution increases as training progresses. We first verify that models trained with our method indeed retain memory of discarded clusterings. Figure 2 (left) shows the loss on discarded clusterings (those discarded in early rounds) over the course of training. Despite never being trained on these after round 1, the loss remains low, confirming historical memory. We also explicitly construct a collapsed solution by training a VAE with extremely high decoder variance (σ′2=10λmaxσ 2=10 _ ). Figure 2 (right) shows the Euclidean distance in parameter space between our model and this collapsed solution, which increases over time, confirming separation. 5.4 Posterior Collapse Prevention Table 1: Comparison of posterior collapse metrics. KL divergence (lower is worse, 0=collapse) and active units (higher is better). Results averaged over 5 runs. Method Synthetic MNIST Fashion-MNIST CIFAR-10 KL ↓ Active ↑ KL Active KL Active KL Active Vanilla VAE 0.32 5.1 0.28 4.8 0.31 4.6 0.18 2.3 β-VAE (β=2β=2) 0.41 6.2 0.38 5.9 0.42 5.7 0.25 3.1 β-VAE (β=4β=4) 0.22 3.4 0.19 3.1 0.24 2.9 - - KL Annealing 0.45 6.8 0.42 6.5 0.44 6.3 0.31 4.2 EM-type VAE 0.51 7.2 0.48 7.0 0.50 6.8 - - Ours (pre-refinement) 2.67 5 2.08 2 2.38 2 3.66 3 Ours (post-refinement) 2.59 4 2.51 2 2.49 3 3.55 3 Ours (final single-cluster) 2.64† 4† 0.00‡ 0‡ 2.64 2 3.70 3 †Final single-cluster results consistent with post-refinement, demonstrating historical memory. ‡On MNIST, final single-cluster training occasionally collapses due to aggressive regularization; post-refinement results (DKL=2.51D_KL=2.51) already demonstrate successful prevention of posterior collapse. Table 1 compares our method against baselines. Key observations: • Our method achieves the highest KL divergence (indicating least collapse) across all datasets, with values ranging from 2.49 to 3.70, far above the collapse threshold of 0.01. • On Synthetic, Fashion-MNIST, and CIFAR-10, the final single-cluster training maintains high KL, confirming historical inertia. • On MNIST, the final stage occasionally collapses due to aggressive regularization (β=4.0β=4.0), but the post-refinement result (DKL=2.51D_KL=2.51) already demonstrates successful prevention of posterior collapse. • Active units remain limited (2-5 out of 48), indicating that while collapse is prevented, the representation is not fully distributed. Figure 3: KL divergence trajectories across datasets. Our method maintains high KL throughout training, demonstrating effective prevention of posterior collapse. Figure 3 shows the KL divergence trajectories during training. Our method maintains consistently high KL across all datasets, confirming that posterior collapse is successfully prevented. 5.5 Ablation Studies Figure 4: Ablation studies. Left: Effect of initial number of clusterings R0R_0. Right: Effect of refinement threshold ϵε. 5.5.1 Initial Number of Clusterings Figure 4 (left) shows final KL divergence as a function of R0=2kR_0=2^k. Performance improves with R0R_0 up to R0=16R_0=16, then saturates. We use R0=16R_0=16 for all experiments. 5.5.2 Refinement Threshold ϵε Figure 4 (right) shows the effect of ϵε on final KL. Extremely small ϵε (10−510^-5) yields best results, but even ϵ=10−3ε=10^-3 outperforms baselines. 5.5.3 Selection Ratio We experimented with keeping fractions other than 1/21/2 (e.g., 1/31/3, 2/32/3). The 1/21/2 ratio consistently performed best, balancing diversity and focus. 5.6 Architecture Independence To demonstrate that our method does not rely on specific architectures, we repeated experiments on CIFAR-10 with convolutional encoder/decoder. Table 2 shows results; our method again achieves non-collapsed representations. Table 2: Results on CIFAR-10 with convolutional architecture. Method KL ↓ Active ↑ MS-SSIM ↑ Vanilla VAE 0.18 2.3 0.35 β-VAE (β=2β=2) 0.25 3.1 0.38 KL Annealing 0.31 4.2 0.42 Ours 3.55 3 0.52 5.7 Comparison with Stability Condition Recall that Li et al. (2026) derived the condition σ′2<λmaxσ 2< _ to avoid collapse. We test our method under violating conditions: setting σ′2=2λmaxσ 2=2 _ and σ′2=5λmaxσ 2=5 _ . Figure 5 shows that our method maintains non-zero KL even when the condition is violated, while vanilla VAE collapses completely. Figure 5: KL divergence under violating conditions (σ′2>λmaxσ 2> _ ). Our method remains non-collapsed while vanilla VAE collapses. 6 Discussion 6.1 The Nature of Historical Memory Our results suggest that the model’s parameters encode information about all previously seen clustering constraints, even after those constraints are removed. This is reminiscent of elastic weight consolidation (Kirkpatrick et al., 2017) in continual learning, where important weights are protected from large changes. In our case, the ”protection” emerges naturally from the training dynamics rather than explicit regularization. 6.2 Limitations While powerful, our method has limitations: • Computational cost: Running EM R0R_0 times and multiple training cycles increases total computation. However, EM runs are parallelizable, and the total training time remains acceptable (2-3x vanilla VAE). • Choice of R0R_0 and ϵε: Optimal values may be dataset-dependent, though our experiments show robustness. • Clustering quality: If all EM runs produce very similar clusterings, diversity is low and the method reduces to standard training. This can be detected by measuring pairwise distances between clustering results. • Limited active units: While collapse is prevented, the representation is not fully distributed (only 2-5 active units out of 48), suggesting room for improvement in representation efficiency. 6.3 Broader Implications Our work suggests a new paradigm for preventing undesirable solutions in deep learning: instead of designing constraints to avoid them, we can leverage solution multiplicity to train them out of existence. This principle may extend beyond VAEs to other models with multiple valid solutions, such as GANs or energy-based models. 7 Implications for Diffusion Models Our analysis of posterior collapse as a phase transition and the proposed historical consensus training have significant implications for diffusion models, which have emerged as powerful alternatives to VAEs for generative modeling. 7.1 The Posterior Collapse Analogy in Diffusion Models Diffusion models (Ho et al., 2020; Song et al., 2020) learn to reverse a gradual noising process. While they do not suffer from posterior collapse in the VAE sense, they face an analogous problem: information loss in the reverse process. In a VAE, posterior collapse occurs when the encoder qϕ(z|x)q_φ(z|x) becomes independent of x. In a diffusion model, the reverse process pθ(xt−1|xt)p_θ(x_t-1|x_t) could similarly become independent of xtx_t if the model is too expressive or poorly conditioned. This manifests as: • Mode dropping: The model fails to capture certain data modes • Unconditional generation degrading: The model ignores the conditioning on xtx_t • Deterministic reverse: The process becomes nearly deterministic, losing stochasticity 7.2 The Spectral Condition Applies to Diffusion Remarkably, the spectral condition σ′2>λmaxσ 2> _ that governs posterior collapse in VAEs has a direct analogue in diffusion models. Consider the forward diffusion process: q(xt|x0)=(xt;α¯tx0,(1−α¯t)I)q(x_t|x_0)=N(x_t; α_tx_0,(1- α_t)I) (12) The variance 1−α¯t1- α_t increases with t. When this variance exceeds the data covariance’s maximum eigenvalue λmax _ , the signal x0x_0 becomes indistinguishable from noise (Choi et al., 2022). This defines a critical timestep tct_c: 1−α¯tc=λmax1- α_t_c= _ (13) For t>tct>t_c, the reverse process must rely purely on learned priors rather than the observed xtx_t, analogous to posterior collapse. 7.3 Historical Consensus for Diffusion Training Our historical consensus training can be adapted to diffusion models. The key insight: just as we used multiple GMM clusterings to create diverse constraints, we can use multiple noise schedules or multiple diffusion trajectories as constraints. Algorithm 2 Historical Consensus for Diffusion Models 0: Dataset D, number of noise schedules R=2kR=2^k 0: Trained diffusion model ϵθ _θ Generate R different noise schedules βt(1),…,βt(R)\β^(1)_t,…,β^(R)_t\ Initialize diffusion model ϵθ _θ S←schedule1,…,scheduleRS←\schedule_1,…,schedule_R\ while |S|>2|S|>2 do for epoch=1epoch=1 to E do for β∈Sβ∈ S do Sample x0∼x_0 , t∼Uniform(1,T)t (1,T), ϵ∼(0,I)ε (0,I) xt=α¯tx0+1−α¯tϵx_t= α_tx_0+ 1- α_tε Compute loss ℒ=‖ϵ−ϵθ(xt,t,β)‖2L=\|ε- _θ(x_t,t,β)\|^2 Update θ end for end for Evaluate model on each schedule using validation loss Keep only the |S|/2|S|/2 schedules with smallest loss end while return ϵθ _θ This trains the model to handle diverse noise schedules, creating a historical barrier that prevents it from collapsing to schedule-independent behavior. 7.4 Empirical Predictions Based on our VAE analysis, we make the following predictions for diffusion models: 1. Critical timestep: There exists a timestep tct_c such that for t>tct>t_c, the reverse process becomes independent of xtx_t, analogous to posterior collapse. 2. Spectral condition: tct_c is determined by λmax _ of the data covariance. 3. Historical barrier: Training with multiple noise schedules prevents schedule-independent collapse, improving sample diversity. 4. Inference flexibility: Models trained with historical consensus can be sampled with different schedules at test time without degradation. 7.5 Preliminary Evidence Recent work provides indirect support for these predictions. Choi et al. (2022) showed that diffusion models have a critical timestep beyond which the signal is lost. Karras et al. (2022) demonstrated that training with multiple noise schedules improves performance. Our framework provides a unified theoretical explanation for these observations. Figure 6: Illustration of the critical timestep in diffusion models. When the noise variance exceeds λmax _ , the signal becomes indistinguishable from noise, analogous to posterior collapse in VAEs. 7.6 Future Directions This connection opens several research directions: • Adaptive scheduling: Dynamically adjust noise schedules based on data covariance to avoid the critical region. • Historical sampling: Use multiple schedules during inference to improve sample diversity. • Theoretical unification: Develop a unified phase transition theory for both VAEs and diffusion models. • Architecture design: Design networks that maintain sensitivity to xtx_t even at high noise levels. The phase transition perspective we developed for VAEs thus provides a powerful lens for understanding and improving diffusion models, suggesting that posterior collapse is not a VAE-specific pathology but a fundamental phenomenon in generative modeling. 8 Conclusion We have introduced Historical Consensus Training, a novel method that prevents posterior collapse in VAEs by iteratively selecting among multiple GMM clustering results. Our key insight is that models trained to satisfy diverse constraints develop a historical barrier that excludes collapsed solutions, even when subsequently trained with a single objective. We provided theoretical analysis of this barrier and empirical validation across multiple datasets and architectures. Our method achieves DKL>2.5D_KL>2.5 on synthetic data, >2.0>2.0 on MNIST and Fashion-MNIST, and >3.5>3.5 on CIFAR-10 under the violating condition σ′2=2λmaxσ 2=2 _ , where vanilla VAE collapses completely (DKL<0.01D_KL<0.01). This demonstrates that posterior collapse is not inevitable and can be prevented through historical consensus training. However, we observe that the number of active latent units remains limited (2-5 out of 48), suggesting that while collapse is prevented, the representation is not fully distributed. Future work will focus on encouraging more balanced use of latent dimensions through improved regularization and architectural innovations. Acknowledgments This work was supported by the authors’ institutions. The authors thank the anonymous reviewers for their valuable feedback. References Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, p. 41–48. Cited by: §2.3. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio (2015) Generating sentences from a continuous space. In Proceedings of the 19th Conference on Computational Natural Language Learning (CoNLL 2015), p. 10–21. Cited by: §1, §2.1. Y. Burda, R. Grosse, and R. Salakhutdinov (2015) Importance weighted autoencoders. In International Conference on Learning Representations, Cited by: 2nd item. R. Caruana (1997) Multitask learning. Machine Learning 28, p. 41–75. Cited by: §2.3. K. Choi, Y. Song, and S. Ermon (2022) Understanding diffusion models: a unified perspective. In International Conference on Learning Representations, Cited by: §7.2, §7.5. B. Dai, Z. Wang, and D. Wipf (2020) The usual suspects? reassessing blame for vae posterior collapse. In International Conference on Machine Learning, p. 2313–2322. Cited by: §2.1. N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan (2016) Deep unsupervised clustering with gaussian mixture variational autoencoders. In International Conference on Learning Representations (Workshop), Cited by: §1, §2.2. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: 4th item. I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner (2017) Beta-vae: learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, Cited by: §2.1, 2nd item. J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Vol. 33, p. 6840–6851. Cited by: §7.1. C. Huang, S. Tan, A. Lacoste, and A. C. Courville (2018) Improving explorability in variational inference with annealed variational objectives. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §2.1, 3rd item. Y. Ichikawa and K. Hukushima (2025) High-dimensional asymptotics of vae: threshold of posterior collapse and dataset-size dependence. Journal of Statistical Mechanics: Theory and Experiment 2025 (7), p. 073402. Cited by: §2.1. T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364. Cited by: §7.5. D. P. Kingma and M. Welling (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §1, 1st item. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: §2.3, §6.1. Y. Li, L. Cheng, F. Yin, M. M. Zhang, and S. Theodoridis (2023) Overcoming posterior collapse in variational autoencoders via em-type training. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. External Links: Document Cited by: 4th item. Z. Li, F. Zhang, Z. Zhang, and Y. Chen (2026) Posterior collapse as a phase transition in variational autoencoders. Physica A: Statistical Mechanics and its Applications 683 (C), p. 131228. External Links: Document Cited by: §1, §2.1, §3.2, 4th item, §5.7. Y. Liu, Z. Liu, S. Li, Z. Yu, Y. Guo, Q. Liu, and G. Wang (2023) Cloud-vae: variational autoencoder with concepts embedded. Pattern Recognition 140, p. 109530. Cited by: §1, §2.2. Y. Liu, Z. Yu, Z. Liu, X. Yang, X. Li, Y. Guo, Q. Liu, and G. Wang (2024) Improving disentanglement in variational auto-encoders via feature imbalance-informed dimension weighting. Knowledge-Based Systems 296, p. 111818. Cited by: §2.2. J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi (2019) Don’t blame the elbo! a linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §1, §2.1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2020) Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §7.1. Appendix A Proof of Theorem 1 Detailed proof of Theorem 1. Let Θcollapse _collapse be a collapsed solution. Then for any x, the reconstruction x^(x) x(x) is independent of z and equal to some constant c (for optimal decoder, c=pdata[x]c=E_p_data[x]). For any clustering result C with component means μkk=1K\ _k\_k=1^K, we have: ℓ(Θcollapse)=1||∑x∈mink‖c−μk‖2=mink‖c−μk‖2, _C( _collapse)= 1|D| _x _k\|c- _k\|^2= _k\|c- _k\|^2, (14) since the minimum over x is independent of x. Now, for any reasonable clustering (one that actually partitions the data), the means μk _k are distinct from the global mean c for at least one component. Let δ=mink‖c−μk‖2>0 _C= _k\|c- _k\|^2>0. Taking δ=min∈S0δ>0δ= _C∈ S_0 _C>0 (finite minimum over finite set) gives the result. If ϵT<δ _T<δ, then ℒST(Θcollapse)≥δ>ϵTL_S_T( _collapse)≥δ> _T, so Θcollapse∉ℱT _collapse _T. ∎ Appendix B Experimental Details B.1 Hyperparameters Table 3: Hyperparameters used in experiments. Parameter Synthetic/MNIST/Fashion CIFAR-10 Latent dimension n 48 48 Encoder hidden dims [1024,512,256] Conv layers: 32,64,128 Decoder hidden dims [256,512,1024] Transpose conv: 128,64,32 Learning rate 3e-3 3e-3 Batch size 512 512 Optimizer Adam Adam R0R_0 (initial clusterings) 24 24 Epochs per cycle E 30 30 Refinement threshold ϵε 1e-4 1e-4 β (clustering loss weight) 1.0→ 4.0 (annealed) 1.0→ 4.0 (annealed) Active unit threshold 0.005 0.005 B.2 Computational Resources Experiments were conducted on a single NVIDIA RTX 4080 SUPER GPU. Total runtime for our method (including EM runs) is approximately 4-6 hours per dataset, compared to 1-2 hours for vanilla VAE. Appendix C Additional Results C.1 Evolution of Cluster Retentions Figure 7: Evolution of which clusterings are retained across rounds. Each row is a clustering result, each column a round. Dark cells indicate retention. Figure 7 shows the retention pattern across rounds for one run. The selection process rapidly converges to a consistent subset, indicating that some clusterings are inherently more compatible with the VAE’s representation. C.2 Sensitivity to β Figure 8: Sensitivity to clustering loss weight β. Optimal performance occurs in the range β∈[2.0,3.0]β∈[2.0,3.0]. Figure 8 shows final KL divergence as a function of β. Performance improves with β up to β=3.0β=3.0, with degradation at very high β (over-constraining) and very low β (ineffective).