Paper deep dive
Heterogeneous Decentralized Diffusion Models
Zhiying Jiang, Raihan Seraj, Marcos Villagra, Bidhan Roy
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:15:39 AM
Summary
The paper introduces a Heterogeneous Decentralized Diffusion Model (HDDM) framework that enables training generative models in isolation across heterogeneous objectives (DDPM and Flow Matching). By utilizing a deterministic schedule-aware conversion to a common velocity space, the framework eliminates the need for synchronization and retraining. It incorporates PixArt-alpha's AdaLN-Single architecture and a checkpoint conversion strategy, achieving a 16x reduction in compute and 14x reduction in data requirements compared to prior decentralized diffusion models, while improving FID and intra-prompt diversity.
Entities (6)
Relation Signals (4)
Heterogeneous Decentralized Diffusion Models → utilizes → AdaLN-Single
confidence 98% · Our approach combines three contributions... (3) PixArt-alpha's efficient AdaLN-Single architecture
Heterogeneous Decentralized Diffusion Models → evaluatedon → LAION-Aesthetics
confidence 97% · Experiments on LAION-Aesthetics show that, relative to the training scale reported for prior DDM work
Heterogeneous Decentralized Diffusion Models → supports → DDPM
confidence 95% · a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching)
Heterogeneous Decentralized Diffusion Models → supports → Flow Matching
confidence 95% · a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Training frontier-scale diffusion models often requires substantial computational resources concentrated in tightly coupled clusters, limiting participation to well-resourced institutions. While Decentralized Diffusion Models (DDM) enable training multiple experts in isolation, existing approaches require 1176 GPU-days and homogeneous training objectives across all experts. We present an efficient framework that reduces resource requirements while supporting heterogeneous training objectives. Our approach combines three contributions: (1) a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching), unified at inference time via a deterministic schedule-aware conversion into a common velocity space without retraining; (2) pretrained checkpoint conversion from ImageNet-DDPM to Flow Matching objectives, accelerating convergence and enabling initialization without objective-specific pretraining; and (3) PixArt-alpha's efficient AdaLN-Single architecture, reducing parameters while maintaining quality. Experiments on LAION-Aesthetics show that, relative to the training scale reported for prior DDM work, our approach reduces compute from 1176 to 72 GPU-days (16x) and data from 158M to 11M (14x). Under aligned inference settings, our heterogeneous 2DDPM:6FM configuration achieves better FID (11.88 vs. 12.45) and higher intra-prompt diversity (LPIPS 0.631 vs. 0.617) than the homogeneous 8FM baseline. By eliminating synchronization requirements and enabling mixed DDPM/FM objectives, our framework lowers infrastructure requirements for decentralized generative model training.
Tags
Links
- Source: https://arxiv.org/abs/2603.06741v1
- Canonical: https://arxiv.org/abs/2603.06741v1
Trouble viewing inline? Open PDF directly →
Full Text
80,780 characters extracted from source content.
Expand or collapse full text
Heterogeneous Decentralized Diffusion Models Zhiying Jiang, Raihan Seraj, Marcos Villagra and Bidhan Roy Bagel Labs Training frontier-scale diffusion models often requires substantial computational resources concentrated in tightly-coupled clusters, limiting participation to well-resourced institutions. While Decentralized Diffusion Models (DDM) enable training multiple experts in isolation, existing approaches require 1176 GPU-days and homogeneous training objectives across all experts. We present an efficient framework that dramatically reduces resource requirements while supporting heterogeneous training objectives. Our approach combines three key contributions: (1) a heterogeneous decentralized training paradigm that allows experts to use different objectives (DDPM and Flow Matching), unified at inference time via a deterministic schedule-aware conversion into a common velocity space without any retraining; (2) pretrained checkpoint conversion from ImageNet-DDPM to Flow Matching objectives, accelerating convergence and enabling initialization without objective-specific pretraining; and (3) PixArt-훼’s efficient AdaLN-Single architecture, reducing parameters while maintaining quality. Experiments on LAION- Aesthetics show that, relative to the training scale reported for prior DDM work, our approach reduces the compute from 1176 to 72 GPU-days (16×) and data from 158M to 11M (14×). Under aligned inference settings, our heterogeneous 2DDPM:6FM configuration achieves better FID (11.88 vs. 12.45) and higher intra-prompt diversity (LPIPS 0.631 vs. 0.617) than the homogeneous 8FM baseline. By eliminating synchronization requirements and enabling mixed DDPM/FM objectives, our framework makes decentralized generative model training accessible to contributors with single GPUs requiring only 20–48GB VRAM. 1. Introduction Training frontier-scale diffusion models [2,26,28,29] often requires hundreds of GPU-days on tightly- coupled clusters [24], concentrating capability within well-resourced institutions. This infrastructure barrier can limit broader participation in foundational model development. Recent work on decentralized diffusion models (DDM) [22] offers a promising direction by demon- strating that multiple expert models can be trained in complete isolation on disjoint data partitions and later combined for high-quality generation. However, this framework assumes homogeneous training objectives across all experts, requiring coordination that may be impractical in truly decentralized settings where contributors operate independently with different resources, preferences, and technical constraints. Moreover, the computational requirements remain prohibitive, with the original DDM requiring 1176 A100-days for training on 158M images to achieve competitive results [22]. We present a heterogeneous decentralized diffusion framework that embraces the diversity inherent in distributed AI development. Our key insight is that different diffusion objectives, DDPM’s epsilon- prediction [10] and Flow Matching’s velocity-prediction [19,20], induce complementary specialization patterns. By deliberately training experts with different objectives in complete isolation, we achieve greater generation diversity than homogeneous alternatives while maintaining semantic coherence. To address the computational barrier, we introduce an efficient checkpoint conversion strategy that leverages pretrained ImageNet diffusion models [26]. We demonstrate that visual features learned under DDPM objectives [10] transfer effectively to Flow Matching formulations [19,20] through deterministic mathematical conversions, enabling faster convergence without requiring objective- Corresponding author(s): gin, raihan, marcos, bidhan@bagel.com arXiv:2603.06741v1 [cs.LG] 6 Mar 2026 Heterogeneous Decentralized Diffusion Models A white canvas tote bag hanging on a potted plant A small, cute teddy bear wearing a white dress and a flower in its hair A reminiscent of the Hudson River School style painting of a large house with a beautiful garden A dark silver car model with MILLER MOTORCARS in the background A delicious stack of pancakes topped with fresh berries and nuts A northern lights dancing over snowy mountains A butterfly landing on blooming flowers A wooden table with various food items, including a jar of honey, a bottle of wine, and a few apples Figure 1|Text-to-Image Generation with Heterogeneous Decentralized Diffusion. Our framework combines multiple expert models trained with different objectives (DDPM and Flow Matching) in complete isolation to generate high-quality, diverse images from text prompts. All samples are generated at 256×256 resolution using 8 heterogeneous experts trained on LAION-Aesthetics with only 72 A100-days of compute. specific pretraining. Combined with architectural optimizations from PixArt-훼[1], specifically AdaLN- Single conditioning [26] that reduces parameters by 30% while maintaining quality, our approach achieves competitive generation results with dramatically reduced resource requirements. Contributions. We make three primary contributions that advance decentralized generative model training: •Heterogeneous Decentralized Training: We extend the DDM framework [22] to support mixed diffusion objectives, specifically DDPM [10] and Flow Matching [19,20], across experts trained in complete isolation. By leveraging the well-known reparameterization equivalence between휖- and velocity-prediction [13,30], we apply a schedule-aware deterministic conversion at inference time without any retraining, enabling seamless integration of experts trained with different objectives. •Efficient Architecture with Checkpoint Initialization: We adopt PixArt-훼’s AdaLN-Single condi- tioning [1] for each expert that achieves 30% parameter reduction while maintaining quality. We further demonstrate that pretrained ImageNet-DDPM checkpoints [26] can be effectively initialized for flow matching training [19] through architectural component transfer and objective-specific layer reinitialization, accelerating convergence by 1.2×. •Scalable Decentralized Training: Through the combination of heterogeneous objectives, architec- tural efficiency, and pretrained initialization, and relative to the training scale reported for prior DDM work [22], we reduce compute from 1176 to 72 GPU-days (16×) and data from 158M to 11M images (14×), while each expert requires only 20-48GB VRAM for single-GPU deployment 2 Heterogeneous Decentralized Diffusion Models without specialized interconnects. Our experimental evaluation on LAION-Aesthetics [31] demonstrates that decentralized training matches or exceeds monolithic approaches. Using 8 DiT-B/2 [26] experts trained in complete isolation, we achieve 23.7% FID improvement [8] over centralized baselines with intelligent expert selection. Under matched inference settings with 8 DiT-XL/2 experts, heterogeneous experts (2DDPM:6FM) improve both FID (11.88 vs. 12.45) and intra-prompt diversity (LPIPS [40] 0.631 vs. 0.617) rela- tive to homogeneous experts (8FM). By eliminating synchronization requirements, our framework broadens the range of resources and training objectives that can participate in decentralized model development. 2. Method We present Decentralized Diffusion Models with heterogeneous objectives that enable fully indepen- dent training of expert models without any gradient, parameter, or activation synchronization. Our framework builds upon decentralized flow matching theory [22], in which independently trained experts are combined via a learned router at inference time. We extend this foundation to support mixed diffusion objectives — experts may train with either DDPM or Flow Matching — by introducing a deterministic conversion that maps all expert predictions into a velocity space for unified sampling. To make decentralized training practical at scale, we further adopt an efficient architecture and a checkpoint initialization strategy that together reduce both parameter count and convergence time. 2.1. Decentralized Flow Matching Following the decentralized Flow Matching formulation of McAllister et al.[22], we decompose the velocity field across퐾expert models trained on disjoint data partitions. The key theoretical foundation is that the marginal flow can be expressed as a weighted combination of conditional flows: 푢 푡 (푥 푡 )= 퐾 ∑︁ 푘=1 푝 푡 (푘|푥 푡 )· 푢 (푘) 푡 (푥 푡 ),(1) where푢 (푘) 푡 (푥 푡 )is the velocity predicted by expert푘trained only on cluster푆 푘 , and푝 푡 (푘|푥 푡 )represents the posterior probability from a router network. We partition the datasetDinto퐾semantic clusters푆 1 ,푆 2 , . . . ,푆 퐾 using DINOv2 [25] features, extracting 1024-dimensional representations and applying hierarchical k-means clustering. This produces semantically coherent partitions (e.g., portraits, landscapes, architecture) that enable mean- ingful expert specialization. Each expert휃 푘 then trains exclusively on its assigned cluster푆 푘 without any communication with other experts, optimizing its assigned diffusion objective independently. The router network 휙 learns to predict cluster assignments from noisy inputs: 푝 휙 (푘|푥 푡 , 푡)= softmax(Router 휙 (푥 푡 , 푡)) 푘 ,(2) trained with cross-entropy loss against ground-truth cluster assignments. At inference, the router dynamically selects and combines experts based on the noisy input and timestep. 2.2. Heterogeneous Decentralized Diffusion We extend the decentralized framework to support heterogeneous training objectives across experts. Specifically, we train 푛 experts with Flow Matching loss and 푚 experts with DDPM loss [10]. 3 Heterogeneous Decentralized Diffusion Models Input 푥 푡 , 푡, 푐 Router 푝 휙 (푘 | 푥 푡 , 푡) Expert 1 DDPM 휖 휃 1 (푥 푡 , 푡, 푐) Expert 2 FM 푣 휃 2 (푥 푡 , 푡, 푐) Expert 퐾 FM 푣 휃 퐾 (푥 푡 , 푡, 푐) Schedule-aware conversion to 푣 Convert to푣: 푣 (1) = 푑훼 푡 푑푡 ˆ푥 0 + 푑휎 푡 푑푡 휖 휃 1 whereˆ푥 0 = 푥 푡 −휎 푡 휖 휃 1 훼 푡 Convert to푣: 푣 (2) = 푣 휃 2 (푥 푡 , 푡) Convert to푣: 푣 (퐾) = 푣 휃 퐾 (푥 푡 , 푡) Fuse: 푢 푡 (푥 푡 )= Í 퐾 푘=1 푝 푡 (푘|푥 푡 )· 푣 (푘) ODE Step 푥 푡−Δ = 푥 푡 −Δ·푢 푡 (푥 푡 ) Figure 2|Inference Pipeline for Heterogeneous Expert Fusion. Given noisy input(푥 푡 , 푡, 푐), the router predicts cluster probabilities푝 휙 (푘|푥 푡 , 푡)to weight expert contributions. DDPM experts output epsilon predictions while Flow Matching experts output velocity predictions. Schedule-aware conversion functions deterministically unify all predictions into a common velocity space푣 (푘) without retraining, enabling router-weighted fusion 푢 푡 (푥 푡 )= Í 퐾 푘=1 푝 푡 (푘|푥 푡 )· 푣 (푘) for ODE-based sampling. At inference, we unify predictions from heterogeneous experts through deterministic conversion functions. DDPM experts output epsilon predictions휖 휃 푘 (푥 푡 , 푡), which we convert to velocity predic- tions via schedule-aware pure algebraic operations. Flow Matching experts directly output velocity predictions푣 휃 푘 (푥 푡 , 푡). The router network then combines these unified predictions through sampling strategies. This design enables seamless ensemble of mixed objectives without requiring retraining or objective-specific sampling procedures. Details can be seen from Figure 2. 2.3. Heterogeneous Objectives and Conversion A central design choice of our framework is training experts with different diffusion objectives to exploit their complementary strengths. We assign experts to either휖-prediction or velocity-prediction objectives: DDPM Experts predict the noise 휖 added during the forward process: L (푘) DDPM = 피 푥 0 ∈푆 푘 ,휖,푡 ∥휖 휃 푘 (훼 푡 푥 0 + 휎 푡 휖, 푡)− 휖∥ 2 ,(3) where훼 푡 , 휎 푡 follow a cosine schedule for stable training. The forward process corrupts the clean data 푥 0 by progressively adding Gaussian noise휖 ∼ N(0, 퐼)according to the noise schedule, producing noisy observations푥 푡 = 훼 푡 푥 0 +휎 푡 휖at timestep푡. The model휖 휃 푘 is trained to predict the noise component 휖 from 푥 푡 , which can then be used to estimate the clean signal by inverting the linear forward map. Flow Matching Experts directly predict velocity fields: L (푘) FM = 피 푥 0 ∈푆 푘 ,휖,푡 ∥푣 휃 푘 (푥 푡 , 푡)−(휖− 푥 0 )∥ 2 ,(4) where푥 푡 =(1− 푡)푥 0 + 푡휖represents the linear interpolation between clean data푥 0 and Gaussian noise 휖∼N(0, 퐼). Following the rectified flow framework [20], we parameterize the probability path with 푡 ∈ [0,1], where푡=0 corresponds to the data distribution and푡=1 to the noise distribution. The model푣 휃 푘 learns to predict the velocity field푣(푥 푡 , 푡)= 푑푥 푡 푑푡 , which for our linear interpolation yields the target velocity 휖− 푥 0 . At inference, we unify predictions through schedule-aware deterministic conversion. Starting from the DDPM forward process푥 푡 = 훼 푡 푥 0 + 휎 푡 휖, we recover an estimate of the clean sample by inverting 4 Heterogeneous Decentralized Diffusion Models the linear map: ˆ푥 0 = 푥 푡 − 휎 푡 휖 휃 푘 (푥 푡 , 푡) 훼 푡 .(5) For any choice of schedule functions훼 푡 , 휎 푡 , substitutingˆ푥 0 back defines a deterministic path through the model’s current estimates: ̃푥 푡 (ˆ푥 0 , 휖 휃 )= 훼 푡 ˆ푥 0 + 휎 푡 휖 휃 (푥 푡 , 푡).(6) Differentiating with respect to푡while treatingˆ푥 0 and휖 휃 as fixed at their current-timestep values gives the velocity along this path: 푣(푥 푡 , 푡) ≡ 푑 ̃푥 푡 푑푡 = 푑훼 푡 푑푡 ˆ푥 0 + 푑휎 푡 푑푡 휖 휃 (푥 푡 , 푡).(7) For the linear interpolation schedule훼 푡 =1− 푡,휎 푡 = 푡, we have 푑훼 푡 푑푡 =−1, 푑휎 푡 푑푡 =1, so Eq.(7)simplifies to 푣(푥 푡 , 푡)= 휖 휃 (푥 푡 , 푡)− ˆ푥 0 .(8) This is the data-to-noise velocity matching the FM target푣= 휖− 푥 0 ; during sampling we integrate from푡=1 to푡=0 via푥 푡−Δ푡 = 푥 푡 −푣·Δ푡. To ensure numerical stability, we clamp predictedˆ푥 0 to[−20,20] for VAE latents, use훼 safe = max(훼 푡 ,0.01)in Eq.(5), and apply adaptive velocity scaling that dampens converted predictions at elevated noise levels where schedule derivatives become large (details in Section 8). 2.4. Implicit Timestep Weighting Across Objectives To analyze why mixed objectives can be complementary, we compare the effective timestep weighting induced by휖-prediction and velocity prediction under the same variance-preserving (VP) perturbation family푥 푡 = 훼 푡 푥 0 + 휎 푡 휖with훼 2 푡 + 휎 2 푡 = 1 (following the parameterization analysis of Kingma et al. [13]). Although our FM experts in Eq.(4)use linear interpolation, this calculation isolates the objective-induced weighting effect; we show in the Remark below that the conclusion holds for linear interpolation as well. Notation. In this subsection,푣= 훼 푡 휖− 휎 푡 푥 0 denotes the diffusion푣-parameterization of Salimans and Ho [30], not the ODE velocity field 푣(푥 푡 , 푡) used elsewhere in the paper for sampling. Proposition 1. LetL 휖 (푡)andL 푣 (푡)denote per-timestep MSE losses for휖-prediction and velocity prediction, respectively. Writing both losses in terms of clean-sample estimation error yields L 휖 (푡)= 피 h 푤 휖 (푡)∥ˆ푥 (휖) 0 − 푥 0 ∥ 2 2 i , 푤 휖 (푡)= 훼 2 푡 휎 2 푡 ,(9) L 푣 (푡)= 피 h 푤 푣 (푡)∥ˆ푥 (푣) 0 − 푥 0 ∥ 2 2 i , 푤 푣 (푡)= 1 휎 2 푡 .(10) Hence 푤 푣 (푡) 푤 휖 (푡) = 1 훼 2 푡 .(11) Proof. For 휖-prediction, inverting the forward process gives ˆ푥 (휖) 0 =(푥 푡 − 휎 푡 휖 휃 )/훼 푡 (Eq. (5)), so 휖 휃 − 휖= 훼 푡 휎 푡 (푥 0 − ˆ푥 (휖) 0 )=⇒ ∥휖 휃 − 휖∥ 2 2 = 훼 2 푡 휎 2 푡 ∥ˆ푥 (휖) 0 − 푥 0 ∥ 2 2 ,(12) 5 Heterogeneous Decentralized Diffusion Models which proves Eq.(9). For velocity prediction, the target is푣= 훼 푡 휖−휎 푡 푥 0 [30]. Under the VP constraint 훼 2 푡 + 휎 2 푡 =1, the clean sample is recovered viaˆ푥 (푣) 0 = 훼 푡 푥 푡 − 휎 푡 푣 휃 , since훼 푡 푥 푡 − 휎 푡 푣=(훼 2 푡 + 휎 2 푡 )푥 0 = 푥 0 . Then 푣 휃 − 푣= 푥 0 − ˆ푥 (푣) 0 휎 푡 =⇒ ∥푣 휃 − 푣∥ 2 2 = 1 휎 2 푡 ∥ˆ푥 (푣) 0 − 푥 0 ∥ 2 2 ,(13) which proves Eq. (10). Dividing gives Eq. (11).□ Remark. Since훼 푡 ≤1, the ratio푤 푣 /푤 휖 =1/훼 2 푡 ≥ 1, with equality only at푡=0 and diverging as훼 푡 →0 (high noise). Velocity-prediction experts therefore receive relatively stronger gradients at high-noise timesteps, creating natural complementary specialization with휖-prediction experts that are relatively upweighted at low noise. Moreover, this ratio depends only on훼 푡 , not on the specific schedule: under linear interpolation (훼 푡 =1− 푡,휎 푡 = 푡) one obtains푤 푣 /푤 휖 =1/(1− 푡) 2 , recovering the same 1/훼 2 푡 structure. Thus the complementary weighting applies directly to our FM experts, not only to the VP family analyzed above. 2.5. Efficient Expert Architecture Each expert employs a Diffusion Transformer (DiT) [26] adapted with PixArt-훼optimizations [1]. The model processes 32×32×4 VAE latents [28] using 2×2 patch embedding to create 256-token sequences. AdaLN-Single Conditioning. This module [1] computes all layer-wise adaptive modulation param- eters through a single global computation rather than per-block MLPs. Given timestep embedding 휏(푡) ∈ ℝ 푑 , the global modulation is computed as: c= MLP global (휏(푡)) ∈ ℝ 6퐿푑 ,(14) which we reshape into a tensor C∈ ℝ 퐿×6×푑 .(15) We denote by C 푏 ∈ ℝ 6×푑 the slice corresponding to block푏. The six modulation vectors for block푏are then [훾 msa 푏 , 훽 msa 푏 , 훼 msa 푏 , 훾 mlp 푏 , 훽 mlp 푏 , 훼 mlp 푏 ]= C 푏 + E 푏 ,(16) where E 푏 is a learned per-block embedding. This reduces parameters by 30% (891M to 605M for text-conditioned DiT-XL/2) while maintaining quality. Transformer Block Architecture. Each block implements adaptive layer normalization with gated residual connections: h 1 = h+ 훼 msa 푏 · MSA(LN(h)⊙(1+ 훾 msa 푏 )+ 훽 msa 푏 ),(17) h 2 = h 1 + CrossAttn(LN(h 1 ), e text ),(18) h ′ = h 2 + 훼 mlp 푏 · FFN(LN(h 2 )⊙(1+ 훾 mlp 푏 )+ 훽 mlp 푏 ),(19) whereLNdenotes layer normalization without learnable affine parameters, MSA is multi-head self- attention, and 훼 푏 parameters act as learnable gates controlling each sub-layer’s contribution. For classifier-free guidance during inference, we randomly drop conditioning with probability푝 cfg =0.1 during training, using learned null embeddings for unconditional generation. Initialization Strategy. Following Chen et al.[1], we initialize the final linear projection in each modulation path to zero, ensuring initial forward passes approximate an identity function. Cross- attention output projections are similarly zero-initialized to stabilize early training when incorporating 6 Heterogeneous Decentralized Diffusion Models text conditioning. The per-block embeddings E 푏 are initialized withN(0,1/ √ 푑)to maintain gradient scale. 2.6. Efficient Checkpoint Conversion for Experts A critical challenge in scaling DDM is computational cost. We address this by converting pretrained ImageNet DiT checkpoints [26] to Flow Matching for accelerated convergence. This leverages the insight that low-level visual features learned under DDPM objectives remain valuable for alternative formulations like Flow Matching, despite different training targets. Our conversion methodology transfers all core architectural components while reinitializing objective- specific layers: 휃 (푙) expert = 휃 (푙) DiT if 푙 ∈ patch_embed, pos_embed, blocks N(0, 0.02) if 푙 ∈ final_layer, text_proj ∅if 푙= class_embed (20) Patch embeddings and transformer blocks are fully transferred to preserve spatial and temporal dynamics. Final projection is reinitialized for different prediction targets (휖vs. velocity푣), text projection is newly initialized, and class embeddings are removed. A key technical consideration is timestep compatibility between objectives. DiT models expect discrete timesteps푡 ∈ 0,1, . . . ,999while Flow Matching uses continuous푡 ∈ [0,1]. Rather than modifying pretrained weights, we implement runtime conversion that preserves the learned timestep embedding MLP: 푡 DiT = ( round(999푡) if 푡 ∈ [0, 1] (Flow Matching experts), 푡if 푡 ∈ 0, . . . , 999 (DDPM experts). (21) Hereround(·)denotes rounding to the nearest integer in[0,999], which we use as the index into DiT’s learned timestep embedding table. This approach maintains temporal reasoning capabilities acquired during pretraining while adapting seamlessly to different noise schedules used by heterogeneous objectives. 3. Experiments We evaluate our decentralized diffusion framework with heterogeneity through comprehensive experiments on LAION-Aesthetics [15]. The experiments show (1) the efficiency of decentralized training compared to monolithic training; (2) the effectiveness of using checkpoints with objective conversion; and (3) that heterogeneous objectives improve FID under aligned inference settings while providing stronger diversity in conversion studies. 3.1. Experimental Setup We train on a subset of 11M LAION-Aesthetics images. To improve text-image alignment, we employ the LLaVA vision-language model to enhance caption quality for a high-quality 3.9M text-image pairs, generating more detailed and descriptive captions that better capture visual content and semantic relationships. Each expert trains on semantically clustered data partitions obtained via DINOv2 features. We evaluate generation quality using FID-50K on a held-out 50K test set following [22]. Unless otherwise noted, homogeneous and heterogeneous comparisons use matched inference settings (CFG= 7.5, 50 steps) on this same holdout split. 7 Heterogeneous Decentralized Diffusion Models (a) From Scratch(b) Pretrained Initialization Figure 3|Impact of Pretrained Checkpoint Conversion. Comparison of generated samples after 75K optimization steps using the text prompt “a red car in front of the tree”. Concretely, we: (i) pre-encode images into a latent space with a pretrained VAE [12] (“sd-vae-ft- mse”) [35] to reduce compute, following Rombach et al.[28]; (i) compute semantic features with DINOv2 [25] and cluster the data; (i) train퐾diffusion experts; (iv) train a small transformer router independently; and (v) perform different expert selection strategies at sampling. We train our models at two scales: DiT-B/2 (where B denotes Base model size with 768 hidden dimensions, and /2 indicates patch size 2 for latent tokenization) with 129M parameters per expert and DiT-XL/2 with 605M parameters per expert. We employ DiT-XL/2 with PixArt-훼’s efficient AdaLN-Single conditioning layers for each expert. Our standard configuration uses퐾=8 experts with experts 0 and 3 trained with DDPM objectives (chosen for clusters containing high-fidelity subjects like cars and flowers where DDPM’s epsilon-prediction excels at detail preservation) and the others trained with Flow Matching, each operating in complete isolation without gradient, parameter, or activation synchronization. Experts require only 20-48GB VRAM each, enabling deployment on consumer GPUs or fragmented cloud resources without specialized interconnects. The complete absence of synchronization eliminates network bottlenecks that constrain traditional distributed training. During inference, we support three expert selection modes: Top-1: Routes to single best expert, minimizing computation; Top-K: Weighted ensemble of K highest-probability experts; Full: Uses all experts weighted by router probabilities. 3.2. Monolithic versus DDM We compare our decentralized multi-expert approach against a monolithic baseline on LAION-Art (3.9M images). The monolithic model trains a single DiT-B/2 on the entire dataset, while our approach distributes training across 8 independent experts on semantic clusters (<500K images per expert). To ensure fair comparison, we match the aggregate computational budget following [22]: the monolithic batch size of 256 becomes a per-expert batch size of 32, ensuring equivalent total FLOPs. Both train 8 Heterogeneous Decentralized Diffusion Models Inference StrategyFID-50K↓ Monolithic (single model)29.64 Top-130.60 Top-222.60 Full Ensemble (all experts)47.89 Improvement vs. Monolithic7.04 Table 1|FID-50K (lower is better) comparing monolithic training and decentralized multi-expert training with different inference strategies on LAION-Art. All models use the DiT-B/2 architecture (129M parameters per expert). MethodDataCompute FID@50K↓ DDM [22] † 158M 1176 A100 days 5.5–10.5 Ours (Homo)11M 72 A100 days12.45 Ours (Hetero)11M 72 A100 days11.88 Reduction vs DDM ∼14× ∼16×– Table 2|Baseline Comparison. Resource comparison against DDM. Both of our rows use the same aligned inference settings (CFG=7.5, 50 steps). † McAllister et al. (2025) does not report a single FID on LAION-Aesthetic; the range 5.5–10.5 is estimated from results at varying training FLOPs in their paper. from scratch without pretrained checkpoints. For this evaluation, all experts are trained with Flow Matching objectives. At inference, we evaluate different expert combination strategies. Top-K uses a learned router to select the 퐾 most confident experts per step, combining predictions via weighted averaging. Full Ensemble combines all 8 experts with router-weighted contributions, incurring higher computational overhead. Results show Top-2 achieves FID 22.60, outperforming the monolithic baseline by 7.04 points (23.7% improvement), demonstrating that strategic expert selection leverages specialized knowledge more effectively. Full Ensemble underperforms (FID 47.89), suggesting that indiscriminate combination introduces prediction conflicts. Selective expert activation proves crucial for superior performance. 3.2.1. Resource Efficiency Table 2 places our results in the context of the training scale reported for prior DDM work [22]. Relative to that reported scale, our approach reduces compute from 1176 to 72 A100-days (16×) and data from 158M to 11M images (14×). Note that the DDM FID range of 5.5–10.5 is achieved at substantially larger training scale and data; our numbers are therefore not directly comparable in absolute FID terms, but illustrate that competitive generation quality is attainable at a fraction of the resources. Under our homogeneous baseline (8FM), we achieve 12.45 FID. Introducing heterogeneous objectives (2DDPM:6FM) further improves FID to 11.88, demonstrating that objective diversity provides an additional quality gain at no extra training cost. 3.2.2. Impact of Pretrained Checkpoint Initialization We validate checkpoint conversion effectiveness by comparing models at 75K optimization steps. Figure 3 shows that models initialized with converted ImageNet checkpoints generate substantially higher-quality samples than those trained from scratch. Validation loss confirms 1.2×convergence 9 Heterogeneous Decentralized Diffusion Models Sampling MethodLPIPS↑ FID↓CLIP↑ Native DDPM0.787 27.04 0.316 ±0.030 FM0.752 20.23 0.324 ±0.034 DDPM→FM0.761 25.61 0.319 ±0.032 Combined (same schedule) 0.782 32.67 0.312 ±0.035 Combined (diff. schedules) 0.777 33.29 0.312 ±0.034 Table 3|Sampling Quality Comparison. DDPM→FM conversion improves over native DDPM and enables mixed-objective sampling. Combined experts achieve higher diversity (LPIPS) than single FM, though at a FID cost, reflecting the quality–diversity trade-off of heterogeneous fusion. acceleration, demonstrating effective transfer of visual priors across objectives. Pretrained models produce sharper details and better semantic alignment, while scratch-trained models exhibit artifacts and lower fidelity. 3.3. DDPM→FM without Training We investigate converting DDPM experts to FM objectives without retraining, enabling flexible expert combination at inference time for heterogeneous DDM deployment. 3.3.1. Conversion Experiment Configuration We use two inference settings in Section 3. Conversion-focused analyses in this subsection use CFG [9] scale 6 with 75 sampling steps on 5,000 held-out samples. For direct homogeneous-versus- heterogeneous comparison, Table 4 reports aligned settings (CFG=7.5, 50 steps) on the held-out 50K split. Within this conversion setting, we evaluate five sampling configurations using experts trained on the same data cluster to isolate objective conversion effects from data distribution differences. Both DDPM and FM experts use the DiT-XL/2 architecture with identical hyperparameters. For combined experts, a deterministic router switches between experts at a native-time threshold푡=0.5, allocating high-noise timesteps (푡 > 0.5) to FM experts and low-noise timesteps to DDPM experts. 3.3.2. Results and Analysis Table 3 presents our quantitative evaluation. In addition to FID, we include CLIP [27] for text-image alignment and mean pairwise LPIPS [40] across generated samples as a diversity metric. Because LPIPS measures perceptual distance between image pairs, higher mean pairwise LPIPS indicates greater output diversity (LPIPS↑). Three key findings emerge from our analysis: (1) Effective inference-time alignment: The DDPM→FM conversion improves generation quality compared to native DDPM (FID 25.61 vs. 27.04) and preserves semantic coherence (CLIP score 0.319 vs. 0.316), enabling interoperability with FM experts. Native FM remains the strongest single-expert baseline (FID 20.23), indicating that the conversion is most valuable as a compatibility mechanism rather than a lossless objective replacement. (2) Enhanced diversity through combination: Combined expert sampling achieves higher output diversity (mean pairwise LPIPS), approaching native DDPM levels (0.787) while surpassing a single FM expert (0.752). This demonstrates that heterogeneous objectives create complementary generation patterns, producing more varied outputs than a single-objective expert. (3) Schedule impact on combination: Interestingly, using the same cosine schedule for both 10 Heterogeneous Decentralized Diffusion Models Figure 4|Impact of Router Threshold on Generation Quality. Different thresholds affect quality- diversity trade-offs. ModelCFG Steps FID-50K↓ Homogeneous (8FM)7.5 5012.45 Heterogeneous (1DDPM:7FM) 6.0 7519.75 Heterogeneous (2DDPM:6FM) 6.0 7515.09 Heterogeneous (2DDPM:6FM) 7.5 5011.88 Table 4|Homogeneous vs Heterogeneous Comparison. The first and last rows are directly comparable (same CFG and steps). objectives yields marginally better results than different schedules (FID 32.67 vs. 33.29), suggesting that schedule alignment facilitates smoother expert transitions. However, both combinations exhibit similar diversity gains, indicating that objective heterogeneity drives the primary benefits rather than schedule diversity. The increased FID for combined methods compared to single experts reflects the challenge of seamless expert switching during sampling. However, this trade-off is acceptable given the substantial diversity gains and the practical benefits of heterogeneous training, where experts can leverage different computational resources and training strategies while maintaining compatible inference. 3.3.3. Routing Threshold Analysis We examine how the routing threshold푡affects the transition point between DDPM and FM experts. We evaluate thresholds on 1,000 held-out samples using converted DDPM and native FM experts under the same cosine schedule. The threshold parameter푡determines the transition point between experts during sampling: for timesteps푡 ′ ≤ 푡, the DDPM expert is used, while for푡 ′ > 푡, the FM expert is used. Figure 4 reveals a clear quality-diversity trade-off as measured by FID and LPIPS. Threshold 0.2 achieves optimal FID (38.28) with FM-dominated denoising but lower LPIPS diversity. Threshold 0.5 produces the highest LPIPS diversity with balanced workload but elevated FID. The results reveal that threshold selection should be application-dependent: lower values (0.2-0.3) favor quality, while mid-range values (0.4-0.5) favor diversity. 3.4. Homogeneous versus Heterogeneous 3.4.1. Quantitative Comparison under Aligned Settings To isolate the effect of objective heterogeneity, we evaluate homogeneous and heterogeneous 8-expert models under matched inference settings on the same held-out 50K split. Under aligned settings (CFG= 7.5, 50 steps), the heterogeneous 2DDPM:6FM model achieves 11.88 FID and outperforms homogeneous 8FM (12.45). Under the conversion setting (CFG=6, 75 steps), increasing DDPM experts from 1 to 2 improves FID from 19.75 to 15.09. 11 Heterogeneous Decentralized Diffusion Models Homogeneous (FM only) Heterogeneous (FM+DDPM) Figure 5|Qualitative comparison: Homogeneous vs. Heterogeneous models. Images generated from identical prompts and random seeds. Homogeneous models (left, trained with Flow Matching only) often appear smoother in texture. Heterogeneous models (right, combining FM and DDPM experts) often preserve sharper local details and richer texture variation. To assess diversity under aligned settings, we measure intra-prompt LPIPS [40] by generating 10 images per prompt for 100 held-out prompts and computing mean pairwise LPIPS within each prompt’s outputs. Heterogeneous experts achieve 0.631 (±0.078) vs. homogeneous 0.617 (±0.074), confirming that objective heterogeneity produces more varied outputs for identical prompts. 3.4.2. Qualitative Comparison Figure 5 presents a direct visual comparison between 8-expert homogeneous (Flow Matching only) and heterogeneous (FM+DDPM) models on identical prompts with matching random seeds. Upon close visual inspection, homogeneous FM-only models occasionally produce softer textures. In contrast, heterogeneous models combining FM and DDPM experts often preserve sharper local structure and richer texture statistics in the same prompt/seed pairs. These visual trends are consistent with Table 4, where heterogeneous objectives improve FID under matched sampling settings. 12 Heterogeneous Decentralized Diffusion Models 4. Related Work Diffusion and Flow Matching. DDPM [10] establishes epsilon-prediction for iterative denoising, extended to latent space by LDMs [28] and scaled via DiT [26]. Variational Diffusion Models (VDM) [13] provide a unified variational framework showing that the diffusion ELBO depends only on the signal-to-noise ratio schedule, and that different prediction targets (휖,푣,푥 0 ) correspond to different implicit loss weightings over timesteps — an analysis we leverage in Section 2.4 to explain how heterogeneous objectives produce complementary specialization. Flow Matching [16, 19,20] learns velocity fields for continuous transport, enabling straighter trajectories and few-step sampling. Recent work bridges these frameworks: Diff2Flow [32] enables finetuning flow models from DDPM checkpoints via timestep rescaling and interpolant alignment. Unlike these single-model conversion methods, we enable heterogeneous training where multiple experts use different objectives in isolation, then ensemble their predictions through training-free conversion — allowing decentralized contributors to train experts with mixed objectives, while yielding complementary specialization in our experiments. Efficient Training and Inference. Training efficiency for monolithic diffusion models has advanced through improved noise schedules and Min-SNR weighting [4,5], architectural optimizations in LDMs [28] and DiT [26], and parameter-efficient conditioning like PixArt-훼’s AdaLN-Single [1]. Haxholli & Lorenzi [6] accelerate training by partitioning score learning across independent networks for different time sub-intervals, enabling parallelization beyond standard data parallelism. Inference acceleration focuses on distillation [23,30], consistency models [7,34], fast sampling methods [11, 39], and distributed parallel inference such as DistriFusion [18], which splits denoising across multiple GPUs via displaced patch parallelism with asynchronous communication. These approaches optimize centralized training or sampling of single large models. In contrast, we achieve efficiency through decentralized training plus the parameter-efficient architecture: multiple smaller experts trained independently on clustered data achieve better computational and data efficiency compared to prior decentralized work [22], while heterogeneous objectives induce complementary specialization that improves quality beyond homogeneous training. Decentralized Training. Decentralized and federated learning enable training across distributed nodes without centralized coordination, with foundational algorithms providing convergence guar- antees [14,21] and practical systems emphasizing communication efficiency [3,36–38,41]. For generative modeling, Decentralized Diffusion Models (DDM) [22] train isolated experts on clustered data and ensemble them via a router, eliminating high-bandwidth interconnect requirements but requiring homogeneous training objectives across all experts. We extend this framework in two ways: (i) enabling heterogeneous objectives, so that experts may train with different formulations (DDPM or Flow Matching) without coordination, encouraging complementary specialization; and (i) supporting pretrained checkpoint conversion, which allows existing checkpoints trained under one objective to be converted and incorporated as experts under another, achieving a 16×compute reduction through efficient architecture reuse. 5. Conclusion Relative to the training scale reported for prior DDM work, our decentralized diffusion training setup reduces compute from 1176 to 72 GPU-days (16×) and data from 158M to 11M images (14×), while achieving 11.88 FID under our aligned inference setting. Our framework supports heterogeneous objectives — through AdaLN-Single, pretrained checkpoint conversion, and training-free inference unification — enabling independent contributors to train experts with either DDPM or Flow Matching objectives and seamlessly combine them at inference time without coordination. Under aligned 13 Heterogeneous Decentralized Diffusion Models inference settings, heterogeneous experts improve both FID (11.88 vs. 12.45) and intra-prompt diversity (LPIPS 0.631 vs. 0.617) over homogeneous experts. Each expert requires only 20-48GB VRAM, enabling deployment on single GPUs without specialized interconnects or synchronization overhead. Taken together, these results suggest a practical path toward more decentralized generative model training by reducing some of the infrastructure barriers imposed by tightly coupled centralized training. Limitations and Future Work. We identify several limitations of the current framework that suggest promising directions for future research. (i) Optimal objective ratios. Our experiments evaluate only a narrow set of DDPM-to-FM ratios (e.g., 2:6). The ideal allocation likely depends on the data distribution, expert count, and downstream requirements. Developing a principled and potentially adaptive strategy for selecting objective ratios remains an important open problem. (i) Robust inference-time conversion. Our schedule-aware algebraic conversion enables training-free unification of heterogeneous experts but relies on hand-tuned numerical safeguards (clamping, safe denominators, adaptive scaling at high noise levels). A more robust and universal conversion mechanism that generalizes across arbitrary noise schedules and objective families without manual stabilization would strengthen practical applicability. (i) Beyond two objective families. This work considers only휖-prediction and velocity-prediction. Ex- tending to additional parameterizations, such as푥 0 -prediction,푣-prediction under variance-preserving schedules [30], or consistency objectives [34] could further diversify expert specialization and po- tentially improve ensemble quality, though conversion and routing mechanisms would need to be generalized accordingly. (iv) Dynamic expert participation. The current framework assumes a fixed set of experts; adding or removing experts requires retraining the router. Investigating incremental router updates, zero-shot expert integration, or self-describing expert metadata would move toward the plug-and-play paradigm needed for a truly open, community-driven ecosystem. (v) Extension to other modalities. Our evaluation is limited to text-to-image generation, yet the principles of heterogeneous decentralized training are modality-agnostic. Video, 3D, and audio diffusion models, where computational demands are even greater, stand to benefit substantially from decentralized training and represent a natural next step. References [1] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-훼: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2024. [2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021. [3]Akash Dhasade, Anne-Marie Kermarrec, Rafael Pires, Rishi Sharma, and Milos Vujasinovic. Decentralized learning made easy with decentralizepy. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pages 34–41, 2023. [4] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7441–7451, 2023. 14 Heterogeneous Decentralized Diffusion Models [5]Tiankai Hang, Shuyang Gu, Jianmin Bao, Fangyun Wei, Dong Chen, Xin Geng, and Baining Guo. Improved noise schedule for diffusion training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4796–4806, 2025. [6]Etrit Haxholli and Marco Lorenzi. Faster training of diffusion models and improved density estimation via parallel score matching. In Advances in Neural Information Processing Systems, 2023. [7]Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024. [8]Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017. [9]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. [10]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851, 2020. [11]Hongxu Jiang, Muhammad Imran, Teng Zhang, Yuyin Zhou, Muxuan Liang, Kuang Gong, and Wei Shao. Fast-ddpm: Fast denoising diffusion probabilistic models for medical image-to-image generation. IEEE Journal of Biomedical and Health Informatics, 2025. [12]Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014), 2014. [13] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, pages 21696–21707, 2021. [14]Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International conference on machine learning, pages 3478–3487. PMLR, 2019. [15] LAION-AI. Laion-aesthetics v2: Aesthetic-filtered subset of laion-5b.https://laion.ai/blog/ laion-aesthetics/, 2022. Subset of LAION-5B filtered for high predicted aesthetic scores. [16]Sangyun Lee, Zinan Lin, and Giulia Fanti. Improving the training of rectified flows. Advances in neural information processing systems, 37:63082–63109, 2024. [17]Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2021. [18] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7183–7193, 2024. [19] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations (ICLR), 2023. [20]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2023. [21] Yucheng Lu and Christopher De Sa. Optimal complexity in decentralized training. In International conference on machine learning, pages 7111–7123. PMLR, 2021. [22]David McAllister, Matthew Tancik, Jiaming Song, and Angjoo Kanazawa. Decentralized diffusion models. arXiv preprint arXiv:2501.05450, 2025. [23]Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14297–14306, 2023. [24]Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, pages 8162–8171. PMLR, 2021. [25]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. [26] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023. [27]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from 15 Heterogeneous Decentralized Diffusion Models natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. [28]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. [29]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022. [30]Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. [31]Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022. [32]Johannes Schusterbauer, Ming Gui, Frank Fundel, and Björn Ommer. Diff2flow: Training flow matching models via diffusion model alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28347–28357, 2025. [33]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. [34]Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 32211–32252. PMLR, 2023. [35]StabilityAI. sd-vae-ft-mse (ft-mse version, fine-tuned kl-f8 vae decoder). HuggingFace Model Repository, 2022. https://huggingface.co/stabilityai/sd-vae-ft-mse-original. [36] Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, N Ahmad Aziz, et al. Swarm learning for decentralized and confidential clinical machine learning. Nature, 594 (7862):265–270, 2021. [37]Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication-efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022. [38]Anqi Zhang, Ping Zhao, Wenke Lu, and Guanglin Zhang. Decentralized federated learning towards communication efficiency, robustness, and personalization. ACM Transactions on Sensor Networks, 21(3): 1–20, 2025. [39]Kexun Zhang, Xianjun Yang, William Yang Wang, and Lei Li. Redi: efficient learning-free diffusion inference via trajectory retrieval. In International Conference on Machine Learning, pages 41770–41785. PMLR, 2023. [40]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. [41]Shenglong Zhou, Kaidi Xu, and Geoffrey Ye Li. Communication-efficient decentralized federated learning via one-bit compressive sensing. In 2024 IEEE 99th Vehicular Technology Conference (VTC2024-Spring), pages 1–5. IEEE, 2024. 16 Heterogeneous Decentralized Diffusion Models LAION Dataset D DINOv2 Feature Extraction Hierarchical Clustering Cluster 푆 1 Cluster 푆 2 Cluster 푆 8 · Expert 1 DDPM 휖 휃 1 Expert 2 FM 푣 휃 2 Expert K FM 푣 휃 퐾 Router 휙 Cross-entropy loss Router 휙 Cross-entropy loss × No gradient sync × No param sharing × No activation passing Figure 6|Training Pipeline for Decentralized Heterogeneous Experts. LAION datasetDis partitioned into퐾semantic clusters푆 1 ,푆 2 , . . . ,푆 퐾 using DINOv2 feature extraction and hierarchical clustering. Each expert trains independently on its assigned cluster with heterogeneous objectives: DDPM experts predict noise휖 휃 푘 (푥 푡 , 푡)while Flow Matching experts predict velocity푣 휃 푘 (푥 푡 , 푡). The router network휙trains on all data to predict cluster assignments via cross-entropy loss. Crucially, there is zero gradient synchronization, parameter sharing, or activation passing between experts during training. 6. Training Details 6.1. Data Preprocessing and Clustering Dataset. We train on the LAION-Aesthetics subset. For DiT-B/2, we utilize LAION-Art whose aesthetic score is≥8, containing around 3.9M image-text pairs. For DiT-XL/2, we filter LAION-Aesthetic for aesthetic score≥4.5 and resolution≥256×256. Images are center-cropped to square aspect ratio and resized to 256× 256 pixels before VAE encoding. Feature Extraction. We extract semantic features using the pretrained DINOv2-ViT-L/14 model [25], which outputs 1024-dimensional embeddings for each image. Features are computed from the [CLS] token of the final layer without finetuning. Hierarchical Clustering. We apply hierarchical k-means clustering with퐾=8 clusters using cosine distance as the similarity metric. The clustering is performed in two stages: first partitioning into 1024 fine-grained groups using standard k-means, then grouping them into 8 coarse clusters. This produces semantically coherent partitions (e.g., portraits, landscapes, architecture, abstract art, animals). Each image in the dataset is then assigned to its nearest cluster based on DINOv2 features. Latent Encoding. All images are encoded using the pretrained VAE encoder from Stable Diffusion [28], producing 32×32×4 latent representations with scaling factor 0.18215. We pre-calculated the encoded latents and save them to disk to avoid redundant encoding during training. 6.2. Expert Training Architecture. Each expert uses the DiT-XL/2 architecture with 28 transformer blocks, hidden di- mension 1152, 16 attention heads. Text conditioning uses frozen CLIP-ViT-L/14 text encoder (768- dimensional embeddings, maximum 77 tokens). Objective Assignment. In the experiment of homogeneous experts, we assign Flow Matching 1 Heterogeneous Decentralized Diffusion Models objectives to all퐾=8 experts. For heterogeneous training, we assign 2 experts to DDPM (휖-prediction) and 6 experts to Flow Matching (velocity prediction). We specifically assign DDPM experts to cluster 0 and cluster 3 as they contain high-fidelity subjects like cars and flowers. DDPM experts use cosine noise schedule [24]. Flow Matching experts use linear interpolation schedule푥 푡 =(1− 푡)푥 0 + 푡휖with 푡 ∈ [0, 1]. Initialization. We initialize all experts from the same pretrained ImageNet DiT-XL/2 checkpoint trained with DDPM objective [26]. Following our conversion procedure (Section 2.6), we transfer patch embeddings, positional embeddings, and all transformer blocks. We reinitialize the final projection layer withN(0,0.02)and add new text cross-attention projection layers with zero initialization. For Flow Matching experts, we implement runtime timestep conversion푡 DiT = round(999푡)to maintain compatibility with pretrained timestep embeddings. Optimization. Each expert trains independently with no communication: • Optimizer: AdamW with 훽 1 = 0.9, 훽 2 = 0.999, 휖= 10 −8 • Learning rate: 1× 10 −4 • Weight decay: 0.0 (following Chen et al. [1]) • Warmup: Linear warmup over first 5,000 steps • Batch size: 128 per expert • Training steps: 500,000 steps per expert • Gradient clipping: Max norm 1.0 • Mixed precision: We use mixed precision training (FP16) with gradient scaling and TF32 acceler- ation on NVIDIA A40 GPUs. Exponential Moving Average. We maintain EMA weights with decay휇=0.9999 for generation, updating after each training step: 휃 EMA ← 휇휃 EMA +(1− 휇)휃. Classifier-Free Guidance. During training, we randomly drop text conditioning with probability 푝 cfg =0.1, replacing text embeddings with learned null embeddings e ∅ ∈ ℝ 77×768 . This enables classifier-free guidance [9] during inference. Numerical Stability. For DDPM-to-velocity conversion at inference, we clamp predicted clean latents ˆ푥 0 to[−20,20]and apply adaptive scaling푠(푡)= min(1.0,15.0/(1+푒 10(푡−0.85) ))for timesteps푡 >0.85 to prevent instability at high noise levels. 6.3. Router Training Architecture. The router uses DiT-B/2 architecture with 12 transformer blocks, hidden dimension 768, 12 attention heads, and 129M parameters. Unlike experts, the router is trained from scratch without text conditioning, processing only noisy latents 푥 푡 and timestep 푡. Training Data. The router trains on the full LAION dataset (all clusters combined) with ground-truth cluster assignments from the clustering stage serving as labels. Each training sample(푥 0 , 푘)consists of a clean latent and its cluster ID 푘 ∈ 1, . . . , 퐾. Optimization. Router training hyperparameters: • Optimizer: AdamW with 훽 1 = 0.9, 훽 2 = 0.999 2 Heterogeneous Decentralized Diffusion Models • Learning rate: 5× 10 −5 with cosine annealing to 5× 10 −7 • Weight decay: 1× 10 −2 • Warmup: None (0 steps) • Batch size: 64 per GPU with gradient accumulation of 4 (effective batch: 256 per GPU) • Training: 25 epochs • Loss: Cross-entropyL router =−log 푝 휙 (푘|푥 푡 , 푡) Timestep Sampling. During router training, we sample timesteps푡uniformly from the appropriate range for each objective:푡 ∼U(0,999)when preparing labels for DDPM experts, and푡 ∼U(0,1)for Flow Matching experts. This ensures the router learns to handle timesteps from both parameteriza- tions. 6.4. Computational Resources Hardware. All experiments use NVIDIA A40 48GB GPUs, 1 GPU per expert. Router training uses 1 24GB GPU. Training Time. With batch size 128 and 500K steps, each expert requires approximately 9 GPU- days (72 GPU-days total for 8 experts when trained sequentially, or 9 days wall-clock time if fully parallelized). 7. Additional Qualitative Analysis In this section, we provide extensive qualitative results to demonstrate the capabilities of our het- erogeneous decentralized diffusion framework. All images are generated at 256×256 resolution with 75 Euler sampling steps and CFG scale 6. Importantly, all text prompts used for generation are either from a held-out test set or synthetically generated by large language models, ensuring they were unseen during training. The samples showcase the diversity and quality achieved by our system trained with mixed DDPM and Flow Matching objectives across 8 specialized experts. 7.1. Diverse Generation Examples Figures 7–10 present a wide variety of generated samples demonstrating our framework’s versatility across different semantic categories, styles, and subjects. The samples shown here are generated from three representative configurations: (1) eight Flow Matching experts, (2) one DDPM expert plus seven Flow Matching experts, and (3) two DDPM experts plus six Flow Matching experts. All configurations demonstrate consistent high-quality generation, validating that our framework maintains visual fidelity across different objective mixtures while achieving significant computational efficiency gains. 7.2. Heterogeneous vs. Homogeneous Objectives To validate the effectiveness of mixing different diffusion objectives, we compare our heterogeneous approach (combining DDPM and Flow Matching experts) against a homogeneous baseline where all experts use the same Flow Matching objective. Figures 11 and 12 show side-by-side comparisons on identical text prompts across diverse semantic categories. Our heterogeneous framework achieves comparable or superior visual quality in these examples. The results demonstrate that mixing objectives does not compromise generation fidelity and may even improve diversity and detail coherence by leveraging complementary strengths of different formulations. 3 Heterogeneous Decentralized Diffusion Models A peacock displaying its colorful feathers Delicious strawberry cake with white frosting A sailboat on a misty lake at sunriseSliced bread with jam A cute cat sitting on a cushion Sunset over a calm ocean Trees are surrounded by a dry grass field, giving the scene a somewhat barren appearance Close-up of a chocolate cupcake with a swirl of frosting on top A mountainous landscape with a large mountain covered in snow and the surrounding area is filled with trees. The scene is painted in a vibrant color palette, with the mountain and trees appearing in shades of blue, green, and yellow. A lighthouse on a rocky cliff during a storm A small wooden cabin with a shingled roof, surrounded by a garden of flowers. A miniature park scene with a variety of people and animals painted mainly in green and brown with a combination of a painting and a photograph A beautiful beach scene with a rocky coastline and a lush green hillside. Woman wearing a black dress and a red scarf A close-up of a purple flower A delicious-looking dish, possibly a quiche, served in a black pan. Figure 7|Diverse Generation Examples (Set 1). Representative samples demonstrating our frame- work’s ability to generate high-quality, diverse images across multiple semantic categories. Images are produced by our heterogeneous ensemble combining DDPM and Flow Matching objectives. 4 Heterogeneous Decentralized Diffusion Models Painting depicting a woman with long, flowing hair and a beautifully adorned headpiece, wearing a blue dress, is surrounded by various decorative elements. Wooden cabin in snowy forest A bustling city street scene with a group of people walking down the sidewalk. A beautiful landscape with a lush green field and a mountain in the background, painted in vibrant blue color. A tranquil Japanese garden with koi pond and bamboo A large white boat traveling through the water A kitchen with a red and silver theme. The kitchen is equipped with a red oven, a red refrigerator, and a red countertop. SUV in desert landscape A painting showing people gathered around a tree, with some sitting and others standing, warm and inviting mood The outfit consists of a pink top and blue shorts, both adorned with a floral pattern Mountain lake with perfect reflection A woman wearing a gold and blue headdress, possibly a Pharaoh’s headdress, and a brown dress An Underwater coral reef with tropical fish A small, modern-looking house with a large deck and a glass roof A steaming cup of coffee on a wooden table The image showcases a large buffet table filled with a variety of food items, including fruits, desserts, and beverages. Figure 8|Diverse Generation Examples (Set 2). Additional samples showcasing consistent generation quality across various prompts and content types. 5 Heterogeneous Decentralized Diffusion Models A rustic kitchen with wooden walls and a dirt floor. A snowy owl perched on a frost-covered branch A serene scene of a grassy field with a hill in the background, filled with a variety of flowers, impressionistic art style A serene scene of a tree-lined pathway overlooking the ocean, with predominantly green, blue, and white color, creating a calm and peaceful atmosphere Wolf standing in the darkness. A beautifully decorated Christmas tree in a living room and the tree is adorned with numerous red ornaments A rainbow appearing after rainfall over hills A cozy and colorful living space, featuring a small kitchen and a living area. A small red house with white trim, surrounded by greenery. The house has a white door and a window, and it is adorned with a variety of potted plants, flowers, and vases. A delicious dessert in a glass bowl, consisting of layers of cake and strawberries; the cake is white and creamy, while the strawberries are red and fresh A picturesque scene of a tent pitched on a rocky hillside, overlooking a serene lake A serene scene of a mountain lake surrounded by lush greenery, where the lake is filled with rocks A bird showing off its beautiful feather Realistic portrait of a young woman with soft, natural lighting, wearing a detailed red and white textured neckline, reminiscent of Renaissance or Baroque portraiture A desert oasis with palm trees A cozy, old-fashioned kitchen with a wooden table and chairs where the room is painted in a light green color and the table is set with bowls and spoons Figure 9|Diverse Generation Examples (Set 3). Further examples demonstrating the robustness and versatility of our heterogeneous framework. 6 Heterogeneous Decentralized Diffusion Models A small yellow house situated on a hillside, overlooking a body of water Sandwich with macaroni A small wooden cabin with a shingled roof, situated in a forest. The cabin is surrounded by trees, creating a serene and peaceful atmosphere. The cabin has a porch with a bench A serene landscape with a river flowing through a valley, surrounded by trees and mountains. The river is the main subject, with its calm waters reflecting the natural beauty of the scene. A watercolor style painting illustrating a charming cottage with a garden setting, surrounded by a variety of flowers, including pink and purple ones. A cozy library with tall bookshelves and warm lighting A blue sports car parked in a dark parking lot A cafe terrace in Paris at evening A lively scene of numerous hot air balloons floating in the sky and are spread across the entire sky and a crowd of people gathered on the ground, watching the spectacle and enjoying the event An impressionistic painting describing a bustling city scene with a large crowd of people walking. The cityscape features a mix of architectural styles, including a prominent cathedral and a castle-like structure The impressionistic painting depicts a serene scene of a forest with a mix of trees and bushes. The trees are predominantly orange, creating a warm and vibrant atmosphere. A painting depicting serene scene of a small stream flowing through a forest, with natural and peaceful mood A rare blue poppy blooming among rocks A brown teddy bear wearing a brown bow tie a white teepee tent with a lacey canopy, set up in a grassy area, adorned with a white flower, adding a touch of elegance to the scene An off-road rally car splashing through muddy terrain Figure 10|Diverse Generation Examples (Set 4). Additional samples showcasing the framework’s ability to generate high-quality images across various prompts and configurations. 7 Heterogeneous Decentralized Diffusion Models Heterogeneous (DDPM + FM)Homogeneous (FM only) A frog resting on a lily pad.A close-up of a tiger.A frog resting on a lily pad.A close-up of a tiger. A cute cat sitting on a cushionAn underwater coral reefA cute cat sitting on a cushion an underwater coral reef with tropical fish A patch of daisies growing in a field. A violet surrounded by moss in a forest. A patch of daisies growing in a field. A violet surrounded by moss in a forest. Figure 11|Heterogeneous vs. Homogeneous Comparison (Part 1). Direct side-by-side compari- son showing that our heterogeneous approach (columns 1-2) maintains or improves visual quality compared to homogeneous baseline (columns 3-4) across diverse categories including animals and nature. 8 Heterogeneous Decentralized Diffusion Models Heterogeneous (DDPM + FM)Homogeneous (FM only) A rusty abandoned car overgrown with vines in a forest A chrome-plated Harley Davidson motorcycle parked on the road A rusty abandoned car overgrown with vines in a forest A chrome-plated Harley Davidson motorcycle parked on the road A motocross bike performing a backflip trick in mid-air A white bowl filled with a delightful assortment of cookies. The cookies are decorated with white icing and are topped with a green mint leaf. A motocross bike performing a backflip trick in mid-air A white bowl filled with a delightful assortment of cookies. The cookies are decorated with white icing and are topped with a green mint leaf. Figure 12|Heterogeneous vs. Homogeneous Comparison (Part 2). Continued comparison across vehicles and food categories, demonstrating consistent quality maintenance across different semantic domains. 9 Heterogeneous Decentralized Diffusion Models 7.3. Effects of Expert Selection and Router Thresholds When combining heterogeneous experts in a 2-expert configuration (1 converted DDPM + 1 Flow Matching), both the order of expert application and the router confidence threshold significantly impact generation quality. Figure 13 presents a comparison on identical prompts (sunset scenes) under a unified schedule, varying: (1) expert ordering (DDPM→FM vs. FM→DDPM), and (2) router confidence threshold 휏∈ 0.3, 0.5, 0.7. The results reveal a striking asymmetry between the two orderings. The FM→DDPM configuration (bottom row) produces cleaner, more coherent images with smooth gradients and well-defined struc- tures across all threshold values. In contrast, DDPM→FM ordering (top row) exhibits visible quality degradation, particularly at higher thresholds (휏=0.7), where blocky artifacts and oversaturation appear in the sky regions. At lower thresholds (휏=0.3), DDPM→FM recovers somewhat by allowing earlier transition to the native FM expert, though still showing less refinement than the FM→DDPM counterpart. This asymmetry exposes a critical limitation in epsilon-to-velocity conversion at different noise levels. When DDPM operates first (handling high noise levels where훼 푡 →0), the conversion formula ˆ푥 0 =(푥 푡 − 휎 푡 휖 휃 )/훼 푡 becomes numerically unstable. Our clamping and scaling safeguards (Section 8) introduce systematic biases that manifest as blocky artifacts and color distortions. Critically, these errors occur early in the reverse diffusion process and become “baked into" the emerging image structure, which the subsequent low-noise FM expert cannot fully correct. Conversely, when native FM handles the high-noise phase first, it establishes a clean structural foundation without conversion artifacts. The converted DDPM expert then refines this foundation at low noise levels (훼 푡 ≈1), where the conversion is numerically stable and introduces minimal bias. The threshold parameter휏controls when the router switches between experts during the denoising trajectory. Lower thresholds (휏=0.3) favor earlier transitions, allowing the second expert to contribute more to the generation. Higher thresholds (휏=0.7) maintain the first expert’s influence longer. For DDPM→FM, lower thresholds mitigate conversion artifacts by transitioning to native FM earlier, explaining the quality improvement at휏=0.3. For FM→DDPM, the ordering is already optimal, so threshold variation has minimal impact on overall quality. These findings provide strong empirical evidence that DDPM-to-velocity conversion should be restricted to low-noise regimes (푡 <0.5) for this simple conversion method. High-noise generation should be handled by native Flow Matching experts or unconverted DDPM experts operating in their original parameterization. 7.4. Effects of Noise Schedules The choice of noise schedule fundamentally shapes how diffusion models learn to denoise at different noise levels, affecting both training dynamics and generation quality. To investigate the impact of schedule heterogeneity in our framework, we conducted controlled experiments using a 2-expert configuration (1 DDPM expert + 1 Flow Matching expert, both trained on the same data cluster). Figure 14 compares two training strategies: (1) Different schedules, DDPM with cosine schedule and Flow Matching with linear interpolation, versus (2) Same schedule, where both experts are constrained to train with the cosine schedule. The qualitative examples suggest that allowing each expert to train with its preferred schedule can improve some visual attributes. Images from the different-schedules configuration (left column) exhibit better visual coherence, sharper details, and more natural color gradients compared to the same-schedule baseline (right column). Vehicle samples demonstrate improved material rendering 10 Heterogeneous Decentralized Diffusion Models DDPM→ FM (Converted DDPM first, then FM) Threshold 휏= 0.3Threshold 휏= 0.5Threshold 휏= 0.7 FM→ DDPM (FM first, then Converted DDPM) Threshold 휏= 0.3Threshold 휏= 0.5Threshold 휏= 0.7 Prompt: Sunset over a calm ocean Figure 13|Expert Ordering and Router Threshold Effects. Comparison of 2-expert heterogeneous configurations showing the impact of expert ordering (DDPM→FM vs. FM→DDPM) and router threshold values. All experiments use the same unified schedule. FM→DDPM ordering produces more stable, coherent results, while DDPM→FM shows higher sensitivity to threshold selection. and structural clarity when experts use their native schedules. This performance difference stems from fundamental differences in how each objective interacts with its noise schedule. Our FM experts use the linear-path formulation푥 푡 =(1− 푡)푥 0 + 푡휖[20], for which the target velocity푣 푡 = 휖− 푥 0 is simple and time-independent. When forced to train with a cosine schedule instead, the FM objective must learn a more complex, time-varying velocity field that accounts for the nonlinear signal-to-noise ratio progression. This additional complexity can hinder optimization and lead to suboptimal learned representations, particularly at intermediate noise levels where the cosine schedule’s curvature is most pronounced. (More broadly, Flow Matching [19] supports a general family of Gaussian probability paths, including diffusion-style paths; our discussion here concerns the specific linear-path variant used in our experiments.) Conversely, DDPM with cosine scheduling benefits from a carefully calibrated noise distribution that allocates more training emphasis to perceptually important noise levels. The cosine schedule’s design, maintaining higher signal-to-noise ratios for longer before rapid transition to pure noise, aligns well with DDPM’s epsilon-prediction objective, enabling more stable gradient flow during training. This natural alignment between objective and schedule cannot be fully recovered when forcing disparate 11 Heterogeneous Decentralized Diffusion Models objectives to share a common schedule. Our findings suggest that schedule-aware training can improve some qualitative visual traits in this 2-expert setting, even though the same-schedule setting achieves slightly better FID in the main- text comparison. The improved quality from schedule-aware training may justify the additional engineering complexity of supporting multiple schedules during both training and inference. Heterogeneous Schedules (Ours)Unified Schedule A red and white sports car, likely a Ford Mustang, parked in a garage. A large white boat floating on a body of water A red and white sports car, likely a Ford Mustang, parked in a garage. A large white boat floating on a body of water A large luxury yacht floating on a body of water, possibly the ocean. A black and silver Mustang sports car, adorned with a striped design, parked on a driveway a large luxury yacht floating on a body of water, possibly the ocean. A black and silver Mustang sports car, adorned with a striped design, parked on a driveway A blue sports car with a number 40 on the side, parked on a dirt road A bright orange sports car parked in front of a garage A blue sports car with a number 40 on the side, parked on a dirt road A bright orange sports car parked in front of a garage Figure 14|Noise Schedule Comparison. Controlled comparison using 2-expert configuration with different training strategies. Columns 1-2: Different schedules — DDPM expert trained with cosine schedule and Flow Matching expert trained with linear interpolation schedule (their natural configurations). Columns 3-4: Same schedule baseline — both DDPM and Flow Matching experts trained with cosine schedule. The different-schedules approach shows qualitatively sharper details in these examples, although the same-schedule setting achieves slightly better FID in Table 3. 7.5. Effects of Sampling Strategies Figure 15 compares three key strategies: (1) Top-1 selection, which queries only the single highest- confidence expert at each timestep, (2) Top-퐾sampling, which averages predictions from the퐾most confident experts, and (3) Full ensemble, which performs weighted averaging across all K=8 experts. 12 Heterogeneous Decentralized Diffusion Models While the full ensemble strategy theoretically implements the complete router posterior푝(푘|푥 푡 , 푡)and should minimize prediction variance, our empirical results reveal a more nuanced picture. Interestingly, the Top-2 strategy achieves the best FID score, outperforming both Top-1 and the full K=8 ensemble. This non-monotonic relationship between ensemble size and generation quality suggests that expert prediction diversity introduces both complementary information and conflicting guidance that must be carefully balanced. The Top-1 strategy produces perceptually sharp samples by committing to a single expert’s denoising trajectory at each timestep, maintaining strong sample coherence but sacrificing the benefits of multi-expert collaboration. As shown in Figure 15, Top-1 samples exhibit clear details and consistent styles, though they may miss refinements that alternative experts could provide. The Top-2 strategy strikes an optimal balance: it leverages complementary information from two specialized experts while avoiding the over-smoothing that occurs when averaging many potentially conflicting velocity predictions. This finding aligns with mixture-of-experts literature showing that small ensembles (K=2-3) often outperform larger combinations due to reduced prediction interference [17, 33]. The full K=8 ensemble, while theoretically sound, suffers from averaging artifacts when combining predictions from experts with divergent specializations. When DDPM and Flow Matching experts disagree on fine-grained details, their weighted average can blur sharp features or introduce color inconsistencies, as visible in the full ensemble samples. This suggests that choosing the most relevant subset rather than averaging all available predictions may be more effective for heterogeneous diffusion ensembles. From a computational perspective, Top-퐾strategies also offer a significant speedup over the full ensemble. 8. DDPM to Flow Matching Conversion: Implementation Details The conversion of DDPM expert outputs to Flow Matching velocity predictions is critical for het- erogeneous ensemble inference. While theoretically straightforward, the practical implementation requires careful handling of numerical instabilities, schedule-dependent derivatives, and multi-expert coordination. This section provides comprehensive details on our conversion methodology. 8.1. Theoretical Foundation 8.1.1. General Conversion Formula For any noise schedule parameterized by훼 푡 and휎 푡 , the conversion from epsilon prediction휖 휃 (푥 푡 , 푡)to velocity prediction푣(푥 푡 , 푡)follows from the time derivative of the forward process. Given the forward diffusion: 푥 푡 = 훼 푡 푥 0 + 휎 푡 휖,(22) we first recover the clean sample estimate by inverting the forward process: ˆ푥 0 = 푥 푡 − 휎 푡 휖 휃 (푥 푡 , 푡) 훼 푡 .(23) Treatingˆ푥 0 and휖 휃 as fixed at their current-timestep values, the velocity field is computed as the time derivative of the deterministic path ̃푥 푡 = 훼 푡 ˆ푥 0 + 휎 푡 휖 휃 : 푣(푥 푡 , 푡)= 푑훼 푡 푑푡 ˆ푥 0 + 푑휎 푡 푑푡 휖 휃 (푥 푡 , 푡).(24) This is the data-to-noise (forward) velocity. During sampling we integrate from푡=1 to푡=0, updating 푥 푡−Δ푡 = 푥 푡 − 푣·Δ푡. 13 Heterogeneous Decentralized Diffusion Models Top-1Top-2Full Ensemble (K=8) An Underwater coral reef with tropical fish An autumn forest path with fallen leaves Northern lights dancing over snowy mountains A field of sunflowers under blue sky Figure 15|Sampling Strategy Comparison. Visual comparison of expert selection strategies on identical prompts. Top-1 commits to a single expert per timestep, producing sharp but potentially limited samples. Top-2 achieves optimal FID by balancing complementary information with prediction coherence. Full ensemble (K=8) can over-smooth details when averaging divergent expert predictions. All strategies use 75 Euler steps with CFG scale 6. 14 Heterogeneous Decentralized Diffusion Models 8.1.2. Schedule-Specific Formulations For the linear interpolation schedule used in standard Flow Matching (훼 푡 =1−푡,휎 푡 = 푡), the derivatives simplify to 푑훼 푡 푑푡 =−1 and 푑휎 푡 푑푡 = 1, yielding: 푣(푥 푡 , 푡)= 휖 휃 (푥 푡 , 푡)− ˆ푥 0 .(25) This matches the Flow Matching objective for the forward (data-to-noise) path. For the cosine schedule commonly used in DDPM training: 훼 푡 = cos 휋푡 2 , 휎 푡 = sin 휋푡 2 ,(26) 푑훼 푡 푑푡 =− 휋 2 sin 휋푡 2 , 푑휎 푡 푑푡 = 휋 2 cos 휋푡 2 ,(27) resulting in a more complex velocity computation that varies significantly with timestep. 8.2. Numerical Stability Challenges 8.2.1. Division by Small 훼 푡 The primary numerical challenge arises when훼 푡 →0 at high noise levels (푡 →1), causing the clean sample recoveryˆ푥 0 = 푥 푡 −휎 푡 휖 휃 훼 푡 to become unstable. For cosine schedules,훼 푡 approaches zero rapidly near 푡= 1, amplifying any prediction errors in 휖 휃 . 8.2.2. Large Schedule Derivatives The derivatives 푑훼 푡 푑푡 and 푑휎 푡 푑푡 can become large, particularly for cosine schedules. At푡 ≈0, we have | 푑휎 푡 푑푡 |= 휋 2 | cos( 휋푡 2 )| ≈ 휋 2 , while at푡 ≈1, we have| 푑훼 푡 푑푡 |= 휋 2 | sin( 휋푡 2 )| ≈ 휋 2 . These large derivatives can amplify velocity magnitudes and cause integration instability during sampling. 8.2.3. Accumulation of Conversion Errors In multi-expert ensembles where some experts use DDPM objectives, conversion errors accumulate across the sampling trajectory. Small biases in individual conversions compound through the iterative sampling process, potentially causing divergence or color shifts in generated images. 8.3. Implementation Solutions 8.3.1. Adaptive Clamping We implement data-type-aware clamping to prevent ˆ푥 0 from reaching unrealistic values: ˆ푥 clamp 0 = clamp(ˆ푥 0 ,−푟, 푟), 푟= ( 20.0 for VAE latents 5.0 for pixel space (28) This range is empirically determined based on the typical distribution of clean samples in each representation space. VAE latents require a wider range due to their unbounded nature, while pixel- space values are typically normalized to either [−1, 1] or [0, 1] depending on the implementation. 15 Heterogeneous Decentralized Diffusion Models 8.3.2. Safe Division with Minimum Threshold To handle small 훼 푡 values, we implement safe division: 훼 safe = max(훼 푡 , 0.01),(29) ensuring numerical stability while minimizing bias. This threshold is chosen to balance stability against accuracy, as values below 0.01 occur only at extreme noise levels where exact recovery is inherently difficult. 8.3.3. Schedule Derivative Computation For accurate velocity conversion, we compute finite-difference derivatives of the schedule coefficients: 푑훼 푡 푑푡 ≈ 훼 푡+ℎ − 훼 푡−ℎ 2ℎ , 푑휎 푡 푑푡 ≈ 휎 푡+ℎ − 휎 푡−ℎ 2ℎ ,(30) whereℎ=10 −4 is the derivative epsilon. These derivatives are essential for computing the correct velocity under non-linear schedules. 8.3.4. Schedule-Aware Velocity Scaling For cosine schedules, we apply adaptive dampening based on the noise level to control velocity magnitudes: 푣 scaled = 푠(푡)· 푣(푥 푡 , 푡), 푠(푡)= 0.88 if 푡 > 0.85 0.93 if 0.6 < 푡 ≤ 0.85 0.96 if 푡 ≤ 0.6 (31) 16