Paper deep dive
KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training
Ramchand Kumaresan
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 1:38:18 AM
Summary
The paper introduces KALAVAI, a protocol for post-hoc cooperative LLM training where independently trained domain specialists are fused using a lightweight Mixture-of-Experts (MoE) router. The authors demonstrate that fusion gain is linearly predictable based on specialist divergence from a shared base checkpoint. The protocol enables asynchronous, privacy-preserving training without communication, achieving consistent performance improvements across various model scales (410M to 6.9B parameters) and cross-lingual settings.
Entities (4)
Relation Signals (3)
KALAVAI → requires → Shared initialisation
confidence 100% · Shared initialisation is necessary: checkpoint mismatch degrades routing.
Fusion gain → scaleswith → Specialist divergence
confidence 98% · Fusion gain scales linearly with specialist divergence: gain = 0.82 x divergence - 2.72
KALAVAI → uses → MoE
confidence 95% · submit checkpoints for lightweight MoE routing
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable: gain = 0.82 x divergence - 2.72 (R^2 = 0.856, n=6, 3-26% divergence). This enables practitioners to estimate cooperative value before committing compute. Below ~3.3% divergence, gains approach this http URL the KALAVAI protocol, contributors fine-tune copies of a shared checkpoint independently, then submit for lightweight MoE routing (500 steps). Gains are consistent: +7.72% at 410M (+/-0.02%, 3 seeds), +7.49% at 1B (+/-0.01%, 3 seeds), +6.53% at 6.9B, each over the best specialist. The router matches domain-oracle routing within <10^{-5} nats. Cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76%, with Yoruba perplexity falling 41.9 to 7.7. A 20-contributor federation achieves +16.71% (+/-0.07pp, 3 seeds).Three requirements bound the protocol. Shared initialisation is necessary: checkpoint mismatch degrades routing. Frozen layers are optional below ~10,000 steps and beneficial beyond. Learned routing is essential: uniform averaging degrades by -1.2% vs. best specialist, while any trained router achieves oracle-optimal assignment.
Tags
Links
- Source: https://arxiv.org/abs/2603.22755v1
- Canonical: https://arxiv.org/abs/2603.22755v1
Trouble viewing inline? Open PDF directly →
Full Text
102,269 characters extracted from source content.
Expand or collapse full text
KALAVAI: Predicting When Independent Specialist Fusion Works A Quantitative Model for Post-Hoc Cooperative LLM Training Ramchand Kumaresan Murai Labs Abstract Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist—and the gain is predictable before training begins: gain≈0.82×divergence−2.72gain≈ 0.82×divergence-2.72 (R2=0.856R^2=0.856, n=6n=6, spanning 3–26% divergence). A cooperative whose specialists diverge 15% from the shared base checkpoint yields approximately +7.5%+7.5\% improvement over the best individual specialist. Below ≈ 3.3% divergence the formula predicts near-zero gain; practitioners can verify specialist divergence before committing training resources. In the Kalavai 111KALAVAI (kalaVAI) is the ISO 15919 romanisation of the Tamil word for “fusion” or “mixing”. protocol, contributors each fine-tune a copy of a shared base checkpoint on their own data without communication, then submit checkpoints for lightweight MoE routing (500 gradient steps on mixed data). Gains are consistent across scales: +7.72%+7.72\% at 410M (± 0.02%, 3 seeds), +7.49%+7.49\% at 1B (± 0.01%, 3 seeds), and +6.53%+6.53\% at 6.9B, each over the best individual specialist. The fused model achieves oracle-optimal routing: the learned router matches the domain oracle with gap <10−5<10^-5 nats. Phase 2 extends the protocol to high-divergence settings: cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76%+21.76\%, with Yoruba perplexity falling 41.9→7.741.9→ 7.7. A 20-contributor federation (10 languages + 10 domains) achieves +16.71%+16.71\% (± 0.07p, 3 seeds) vs. best specialist. Three structural requirements bound the protocol. Shared initialisation is necessary: specialists from mismatched checkpoints degrade routing clarity. Frozen layers are optional below approximately 10,000 specialist training steps—peak gain reaches +17.7%+17.7\% at 5,000 steps without freezing—and become beneficial at longer training horizons. Learned routing is essential: uniform equal-weight averaging degrades by −1.2%-1.2\% versus the best specialist, while a trained linear or MLP router achieves oracle-optimal assignment; routing architecture has minimal effect once gates are trained. 1 Introduction Training competitive language models requires centralised compute at a scale most researchers and institutions cannot access. A single 70B-parameter training run requires hundreds of A100 GPUs operating synchronously for weeks. This creates a structural barrier: the organisations that can train frontier models are those that can pay for frontier compute, and the rest must fine-tune what is already available. The core insight. A different path exists. If multiple contributors each train a specialist copy of a shared base checkpoint on their own domain, and if those specialists are subsequently fused via a learned router, the resulting model captures complementary knowledge that no single contributor could build alone. The training step requires no communication: contributors work independently, asynchronously, on their own hardware, using their own data. The only coordination is the shared starting point. The fused model requires all N specialists to run at inference, increasing inference compute by a factor of N relative to any individual specialist. This is a training-time democratisation: inference cost scales with N specialists, but so does the knowledge that no single contributor could acquire alone. This observation is not entirely new—the branch-train-mix (BTX) paradigm (Sukhbaatar et al., 2024) demonstrates that MoE fusion of independently trained models is feasible. What is missing from the literature is an empirical characterisation of the conditions under which independent specialist fusion succeeds or fails: when shared initialisation alone is sufficient, when frozen layers become necessary, and what routing architecture drives improvement. The protocol. Kalavai222Code and all experiment scripts: https://github.com/mechramc/Kalavai operationalises cooperative LLM training as a four-step protocol: (1) a coordinator distributes a shared base checkpoint; (2) each contributor fine-tunes independently on their domain for a fixed number of steps; (3) contributors submit their checkpoints; (4) a lightweight router is trained on a small mixed-domain dataset and used for inference. Contributors never share gradients, intermediate activations, or data. The only shared artefact is the initial checkpoint. Key results. We demonstrate: 1. Predictive divergence–gain relationship. Fusion gain scales linearly with specialist divergence: gain=−2.72+0.82×divergencegain=-2.72+0.82×divergence (R2=0.856R^2=0.856, slope 95% CI [0.35, 1.28]), validated across six conditions from Qwen-1.5B (3.16% divergence, +1.06% gain) to cross-lingual (25.65%, +21.76%). The linear fit is substantially better than log-linear (R2=0.662R^2=0.662). The cross-lingual condition exceeds the prediction by +3.6p, consistent with base-model incompetence on those languages producing outsized gains. Practitioners can estimate cooperative value from divergence alone before committing training resources. 2. Oracle-optimal routing. The learned MoE router matches the domain-level oracle (optimal static assignment of each domain to its best specialist) with a gap of <10−5<10^-5 nats at 410M and 6.9B—effectively zero. The cooperative achieves specialist-level quality on every domain simultaneously, something monolithic training cannot: the monolithic model underperforms the MoE on code (−4.34%-4.34\%) and science (−3.12%-3.12\%), despite seeing all domain data during training. 3. Consistent improvement at scale. Post-hoc MoE fusion beats the best individual specialist by +7.70% at 410M (3-seed mean, ± 0.02%), +7.49% at 1B (3 seeds, ± 0.01%), and +6.53% at 6.9B (± 0.024%, 3 seeds) on equal-weight per-domain evaluation.333All results use the per-domain equal-weight protocol implemented in kalavai_eval_utils.py; see Appendix R. Routing is near-deterministic (>>99.9%) at all scales. Improvement is stable across Pythia-410M training maturities from 3.5% to 100% of training (+7.0%–+8.8%). 4. Training duration crossover. Without frozen layers, fusion improvement peaks at 5,000 specialist training steps (+17.7%) then degrades to +14.7% at 20,000 steps. With four frozen layers, improvement degrades more slowly (+17.0% at 20,000 steps). The crossover occurs at approximately 10,000 steps, above which frozen layers are recommended. Even 50 training steps produce +4.0% gain—meaningful improvement achievable in under two minutes on a consumer GPU. 5. Routing must be learned. Uniform equal-weight averaging (no routing training) degrades by −1.19%-1.19\% vs. best specialist; a trained linear router achieves +7.70%+7.70\%. Hard routing (argmax of learned gates) matches soft routing (+7.72%+7.72\% both)—specialist participation drives improvement, not weighting precision. The learned router achieves oracle-optimal domain assignment: gap <10−5<10^-5 nats at 410M and 6.9B. 6. Per-domain advantage over monolithic. On aggregate equal-weight loss, MoE and monolithic achieve near-parity (+0.47%+0.47\% at 410M). The distinction is per-domain: the MoE matches the best specialist on every domain simultaneously, while the monolithic model sacrifices code (−4.34%-4.34\% vs. MoE) and science (−3.12%-3.12\%) to improve on fiction. The cooperative advantage is simultaneous per-domain quality, not lower aggregate loss. 7. Cross-lingual cooperative training works. Tamil/Yoruba/Welsh/Code specialists fused on Pythia-410M: Yoruba perplexity 41.9→ 7.7 (5.4×), Welsh 102.7→ 22.1 (4.6×). Contributors speaking different languages can collectively build a model none could train alone. Contributions. • A predictive gain model. Fusion improvement scales linearly with specialist divergence (gain=0.82×divergence−2.72gain=0.82×divergence-2.72, R2=0.856R^2=0.856, six conditions, 3–26% divergence). Practitioners can estimate cooperative value from divergence alone, before committing training resources. • A mechanistic account. Shared initialisation preserves representational compatibility; specialists exhibit catastrophic forgetting on out-of-domain tokens; learned MoE routing recovers full-domain coverage. These three facts jointly explain why the protocol succeeds—and why routing must be trained rather than uniform. • Operational guidelines. Frozen layers are optional below 10,000 training steps and recommended beyond. Shared initialisation is the essential coordination requirement: checkpoint mismatch degrades routing clarity even when absolute performance loss appears modest. • Capacity controls. A Pythia-1.4B model (3.5× parameters, centralised training) achieves +10.87%+10.87\% vs. best specialist—exceeding the cooperative gain (+7.72%+7.72\%). The cooperative advantage is privacy-preserving distributed training, not architectural performance supremacy. • High-divergence validation. Phase 2 extends the protocol to private professional domains (+10.17%+10.17\%, medical/legal/patent) and cross-lingual settings (+21.76%+21.76\%, Tamil/Yoruba/Welsh/Code; Yoruba perplexity 41.9→7.741.9→ 7.7). Gain scales most steeply where the base model is least competent. • The Kalavai protocol. A four-step cooperative training workflow with zero communication during training. Code, experiment scripts, and all result artefacts are released publicly. 2 Related Work Branch-Train-Mix (BTX). The closest prior work is BTX (Sukhbaatar et al., 2024), which demonstrates that models branched from a shared checkpoint, independently trained, and mixed via MoE routing form a better model than any individual branch. Building on this foundation, our empirical contribution addresses four questions that remain open in the literature: when shared initialisation alone is sufficient versus when frozen layers become necessary (Section 4.4); whether improvement persists against a compute-matched monolithic baseline (Section 4.3); when routing must be trained versus when uniform routing suffices (Section 4.5); and whether improvement is explained by increased parameter count (Section 4.6). BTX demonstrates fusion at comparable model scales with substantially longer specialist training budgets; our focus is the systematic conditions analysis rather than a head-to-head performance comparison. MoErging and PHATGOOSE. The MoErging survey (Yadav et al., 2024) taxonomises approaches for recycling and routing among independently trained experts. PHATGOOSE (Muqeeth et al., 2024) achieves +11% zero-shot generalisation improvement via learned routing among fine-tuned models, compared to Kalavai’s +7.72%+7.72\% vs. best-specialist or +16.3%+16.3\% vs. base. The contribution is the conditions analysis (training-duration crossover, routing learning requirement, oracle saturation), not magnitude. Kalavai adds a monolithic baseline, training duration analysis, and explicit capacity controls not present in PHATGOOSE. Pari thesis. Pari (2025) provides a theoretical analysis using Centered Kernel Alignment (CKA) of why weight averaging of independently trained models fails: divergent representations produce destructive interference when merged by linear interpolation. Kalavai provides the empirical complement, demonstrating that MoE routing avoids this interference (+7.72%+7.72\% vs. best specialist versus weight averaging’s −3.4%-3.4\% vs. best specialist). Weight interpolation methods. Model soups (Wortsman et al., 2022), TIES-Merging (Yadav et al., 2023), and DARE (Yu et al., 2024) combine fine-tuned models via weight interpolation. These methods require specialised merging procedures and typically produce smaller gains than routing-based approaches. In our experiments, simple weight averaging achieves −-3.4% vs. best specialist (equal-weight evaluation) versus +7.72% for learned MoE routing. Multilingual cooperative training. Extending specialist fusion to low-resource languages has not been studied in prior work. Section 4.9 demonstrates that contributors speaking different languages can collectively build a model that achieves specialist-level perplexity on all languages simultaneously (Yoruba: 41.9→ 7.7, Welsh: 102.7→ 22.1), even when each specialist trains only on their own language data. Federated learning. Federated approaches (McMahan et al., 2017) distribute training with periodic gradient synchronisation. Kalavai requires zero communication during training; contributors are never synchronised until the fusion step, making the protocol fully asynchronous. FuseLLM. FuseLLM (Wan et al., 2024) fuses LLMs via knowledge distillation into a modified single model; Kalavai preserves all specialist parameters intact. Sparse Upcycling. Komatsuzaki et al. (2023) initialise MoE models from dense checkpoints and continue training jointly; Kalavai trains specialists independently with no shared computation. STAR and related concurrent work. Qin et al. (2025) demonstrate modular composition over frozen foundations for multimodal learning; Kalavai provides the language modelling instantiation with analysis of the crossover point where freezing transitions from optional to required. 3 Method The Kalavai protocol consists of four phases. Phase 1: Shared initialisation. A coordinator selects a publicly available base checkpoint θ0 _0 and distributes it to all contributors. All specialists begin from identical weights. This shared initialisation is the core structural guarantee that enables post-hoc fusion: specialists diverge in representation space, but their representational geometry remains compatible because they begin from the same point. Hash verification ensures all contributors use exactly the same checkpoint. Phase 2: Optional freezing. Optionally, the first K transformer layers are frozen during specialist training. Frozen layers guarantee that lower-level representations remain shared across specialists, providing a structural anchor that is robust to extended training. Our experiments show freezing is unnecessary at short training horizons (≤ 10,000 steps) but becomes beneficial beyond approximately 10,000 steps (Section 4.4). Phase 3: Independent specialist training. Each contributor trains their copy of θ0 _0 on a single knowledge domain using standard next-token prediction loss. Training is fully independent: contributors never share data, gradients, or activations. Any training infrastructure, hardware, or optimiser the contributor prefers may be used, provided the architecture and freeze configuration match the coordinator’s specification. We use full fine-tuning of unfrozen layers rather than low-rank adaptation (LoRA); LoRA specialists exhibit negative divergence from base (specialists become worse than the base model even on their target domain at r=64r=64, or near-zero divergence at r=8r=8), placing all LoRA conditions below the divergence floor (≈ 3.3%) where the empirical gain formula predicts near-zero or negative returns (Appendix C, Table 13). Formally, contributor i trains specialist θi _i by minimising: ℒi=−x∼i[∑tlogpθi(xt∣x<t)]L_i=-E_x _i [ _t p_ _i(x_t x_<t) ] where iD_i is contributor i’s domain-specific dataset and θi _i shares the first K frozen layers with all other specialists. Phase 4: Post-hoc MoE fusion. After all specialists submit their checkpoints, a lightweight router is trained on a small mixed-domain dataset (500 gradient steps in our experiments). The router is a single linear layer mapping the model’s hidden state at position t to a distribution over experts: gt=softmax(Wr⋅ht),Wr∈ℝN×dg_t=softmax (W_r· h_t ), W_r ^N× d where hth_t is the mean-pooled final hidden state averaged across all specialists’ forward passes (see Appendix K), N is the number of specialists, and d is the hidden dimension. At inference, all N specialists process each token in parallel and the fused output is a weighted combination of their logit vectors: l~t=∑i=1Ngt(i)⋅lθi,t,pfused(xt∣x<t)=softmax(l~t) l_t= _i=1^Ng_t^(i)· l_ _i,t, p_fused(x_t x_<t)=softmax\! ( l_t ) where lθi,t∈ℝ||l_ _i,t ^|V| is the logit vector from specialist i at position t. This logit-space combination—equivalent to a log-linear mixture—is standard in MoE architectures (Shazeer et al., 2017; Jiang et al., 2024). An alternative probability-space formulation ∑igt(i)⋅softmax(lθi,t) _ig_t^(i)·softmax(l_ _i,t) would produce a proper mixture distribution; we use logit-space combination throughout all experiments. Design decisions—LoRA vs. full fine-tuning, softmax vs. argmax, linear vs. MLP router—are discussed in Appendix C; none meaningfully affects the core results. 4 Experiments 4.1 Experimental Setup Models. We run experiments at three scales: Pythia-410M (24 layers, hidden size 1024), Pythia-1B (16 layers, hidden size 2048), and Pythia-6.9B (32 layers, hidden size 4096) (Biderman et al., 2023). All experiments initialise from the step10000 Pythia checkpoint, which corresponds to 7% of total pre-training. We use Pythia because it releases checkpoints at multiple training stages, enabling the maturity sweep analysis. Domains. Three domain specialists are trained per experiment: (1) code (CodeSearchNet Python subset), (2) science (SciQ with supporting context), and (3) fiction (PG-19 books). For each domain, 90% of samples are used for specialist training, 10% are held out and never seen during training or router training. Training configuration. All 410M and 1B experiments: 2,000 specialist training steps (effective batch size 8, sequence length 512), 500 router training steps. 6.9B experiments: 1,000 specialist training steps, 500 router steps. Freeze depth K=4K=4 for 410M (4/24 = 17%), K=4K=4 for 1B (4/16 = 25%), K=6K=6 for 6.9B (6/32 = 19%). Optimiser: AdamW, lr=2×10−5lr=2× 10^-5, weight decay 0.1, linear warmup over 10% of steps. Evaluation metric. All improvement percentages are computed as: Δ(%)=ℒbaseline−ℒmethodℒbaseline×100 (\%)= L_baseline-L_methodL_baseline× 100 where ℒL is average cross-entropy loss on the held-out mixed-domain evaluation set (equal weighting across three domains). Lower loss is better; positive Δ indicates improvement over the baseline. The baseline for the main result is the best individual specialist on mixed evaluation; the baseline for the monolithic comparison is the monolithic model.444A 14% reduction in cross-entropy loss (e.g., 2.248 → 1.930) corresponds to approximately 37% reduction in perplexity (e.g., e2.248→e1.930e^2.248→ e^1.930, from ≈ 9.5 to ≈ 6.9). We report loss-based percentages throughout for consistency; perplexity values can be recovered via exp(ℒ) (L). Seeds. All main results are reported across 3 random seeds (42, 137, 2026). Evaluation protocol. All results use the per-domain equal-weight protocol: separate evaluation for each domain at consistent batch size bs=4, equal-weight average across domains (base EW loss 2.651 at 410M). See Appendix R for implementation details and the bugs this protocol corrects. 4.2 Core Results Table 1: Main results across three model scales. Per-domain equal-weight evaluation: each domain evaluated separately at bs=4; equal-weight average =(ℒcode+ℒsci+ℒfiction)/3=(L_code+L_sci+L_fiction)/3 (Appendix R). 410M results: 3-seed means (seeds 42/137/2026). 6.9B: seed 42. Absolute base loss values differ from appendix ablation tables, which use a mixed-domain protocol; improvement percentages are internally consistent within each section. Scale Method EW Loss vs. Best Spec. vs. Base Seeds Std Pythia-410M Base model 2.651 — — — — Best specialist 2.404 — +9.3% 3 ± 0.02% Weight averaging 2.486 −-3.4% +6.2% 3 ± 0.00% Monolithic baseline 2.229 — +16.0% 3 ± 0.00% Kalavai (MoE) 2.218 +7.72% +16.3% 3 ± 0.02% Pythia-1B Base model 2.474 — — — — Best specialist 2.259 — +8.7% 1 — Monolithic baseline 2.097 — +15.3% 1 — Kalavai (MoE) 2.090 +7.49% +15.5% 3 ± 0.01% Pythia-6.9B Base model 2.320 — — — — Best specialist 2.266 — +2.3% 1 — Kalavai (MoE) 2.118 +6.53% +8.7% 3 ± 0.024% Table 1 presents the main results under per-domain equal-weight evaluation (separate per-domain eval at bs=4, equal-weight average). Kalavai consistently outperforms the best individual specialist at all tested scales. The 410M improvement of +7.72% is robust: variance across three random seeds is near-zero (± 0.02%), and results are stable across Pythia training maturities (see Appendix I). The 6.9B improvement of +6.53% (± 0.024%, 3 seeds) is smaller than at 410M/1B, but the mechanism is the same. As Table 2 shows, fusion gain scales directly with specialist divergence from base. At 410M and 1B, specialists diverge 10–25% from base per domain (mean ∼ 15.5%), producing ∼ +7.5% fusion gain. At 6.9B, specialists diverge 7–10% per domain (mean 8.73%, Table 2)—approximately half the divergence at smaller scales—and the fusion gain is proportionally smaller (+6.53%). Importantly, the conversion efficiency (gain per unit divergence) is 0.75× at 6.9B versus 0.49× at 410M/1B: larger models convert divergence into fusion gain more efficiently, not less. The reduced gain is entirely explained by reduced divergence, not by scale-dependent degradation of the protocol. Routing is near-deterministic at all scales (>>99.9% per-domain gate weight). Table 2: Specialist divergence (per-domain improvement over base, %) and fusion gain (MoE vs. best individual specialist, equal-weight), seed 42. Per-domain separate eval at consistent batch size (bs=4). † Qwen-1.5B uses code + fiction only (2 specialists). Fusion gain conversion rate = gain/divergence. All Pythia routing >>99.9% per-domain deterministic. Model Code div. Sci. div. Fiction div. Mean div. Fusion gain Conv. rate Pythia-410M 9.97% 11.60% 25.37% 15.65% +7.72% 0.49× Pythia-1B 11.05% 9.39% 25.41% 15.28% +7.49% 0.49× Pythia-6.9B 9.28% 10.16% 6.76% 8.73% +6.53% 0.75× Qwen-1.5B† 1.76% — 4.56% 3.16% +1.06% 0.34× Evaluation protocol. All reported improvements use the per-domain equal-weight protocol: each domain evaluated separately at bs=4, gains averaged equally across domains. Full specification and implementation details are in Appendix R. 4.3 Comparison to Equal-Compute Monolithic Training Figure 1: Kalavai core results. (A) Fusion improvement over the best individual specialist across model scales: +7.72%+7.72\% at 410M, +7.49%+7.49\% at 1B, +6.53%+6.53\% at 6.9B (per-domain equal-weight eval). Gains are proportional to specialist divergence; conversion rate 0.49× at 410M/1B, 0.75× at 6.9B. (B) Training duration crossover: freeze=0 peaks at 5k steps (+17.7%+17.7\%) then degrades to +14.7%+14.7\% at 20k steps; freeze=4 degrades more slowly (+17.0%+17.0\% at 20k); crossover at ≈ 10k steps. (C) Router architecture: uniform routing (no training) achieves −1.2%-1.2\% vs. best specialist; trained linear or MLP routers achieve +7.7%+7.7\%; architecture is irrelevant, learning is not. (D) Kalavai vs. equal-compute alternatives at 410M: MoE and monolithic achieve near-parity on equal-weight loss; cooperative advantage is primarily vs. best individual specialist (+7.72%). All results seed 42 or means over 3 seeds where noted. A natural question is whether centralised training on equivalent total compute would perform equally well. We test this directly at both 410M and 1B scales. A single model is fine-tuned from the same base checkpoint for 6,000 steps (equal to three specialists × 2,000 steps) on a mixed dataset containing equal proportions of code, science, and fiction data. The monolithic baseline achieves strong results: 410M monolithic EW loss 2.229 (+16.0% vs. base) vs. Kalavai MoE EW loss 2.218 (+0.47% over monolithic, Table 4). At 1B, monolithic EW loss 2.097 vs. Kalavai 2.090 (+0.34% over monolithic). The cooperative MoE and equal-compute monolithic model achieve near-parity on this metric. The main headline result of Kalavai is the advantage over the best individual specialist: +7.72% at 410M (specialist EW loss 2.404 vs. MoE 2.218), driven by the router recovering each domain’s specialist quality simultaneously. The decomposition shows: (1) Specialisation advantage: each specialist achieves ∼ 9% over base on equal-weight eval while the monolithic model achieves ∼ 16%; the monolithic model is actually stronger than any individual specialist on the mixed metric because it trains on all domains. (2) Fusion advantage: the MoE then routes each token to the appropriate specialist, achieving the best per-domain performance across all domains at once—equivalent to selecting the diagonal of the specialist cross-domain loss matrix (Figure 5). Per-domain breakdown. The +0.47% equal-weight advantage understates the cooperative benefit. Table 3 shows the per-domain decomposition at 410M. Table 3: Per-domain held-out loss at Pythia-410M (seed 42, per-domain equal-weight evaluation). Bold entries are the best value in each column. The Kalavai MoE matches the best individual specialist on every domain simultaneously—routing recovers the diagonal of the specialist cross-domain matrix. The monolithic model, despite training on all domains, underperforms the MoE on code and science while achieving lower fiction loss than the fiction specialist (cross-domain regularisation benefit). Method Code ↓ Science ↓ Fiction ↓ EW Avg ↓ Base model 2.0872 2.8920 2.9739 2.6510 Code specialist 1.8791 2.9085 2.9768 2.5881 Science specialist 2.1738 2.5565 3.0613 2.5972 Fiction specialist 2.1018 2.8904 2.2194 2.4039 Monolithic (6k) 1.9644 2.6389 2.0832 2.2288 Weight averaging 2.0142 2.7323 2.7106 2.4857 Kalavai MoE 1.8791 2.5565 2.2194 2.2183 The Kalavai MoE achieves oracle-optimal routing: code loss 1.8791 (matching code specialist exactly), science loss 2.5565 (matching science specialist exactly), fiction loss 2.2194 (matching fiction specialist exactly). The domain-level oracle—the optimal static assignment of each domain to its best specialist—achieves EW loss 2.2183, a gap of 3×10−63× 10^-6 nats from the actual MoE. The router is effectively routing-saturated at the domain level. The monolithic model achieves lower fiction loss than the fiction specialist (2.0832 vs. 2.2194), a benefit of cross-domain regularisation. However, it underperforms the MoE on code (−4.34%-4.34\%: 1.9644 vs. 1.8791) and science (−3.12%-3.12\%: 2.6389 vs. 2.5565). The aggregate effect favours the cooperative: MoE EW 2.2183 vs. monolithic 2.2288 (+0.47%). The structural advantage of Kalavai over monolithic training is per-domain specialist quality simultaneously: the cooperative achieves the best available quality on every domain without any contributor sharing data—something centralised training cannot do despite having access to all data. Table 4: Equal-compute comparison at Pythia-410M and Pythia-1B. Monolithic trained for 6,000 steps on mixed data; Kalavai uses three specialists × 2,000 steps. Equal-weight per-domain average (per-domain separate eval, bs=4). Seed 42. The monolithic model achieves strong equal-weight loss because it trains on all domains; Kalavai’s advantage is per-domain specialist quality for each domain simultaneously. Scale Method EW Loss vs. Base vs. Monolithic 410M Base model 2.651 — — Best specialist 2.404 +9.3% — Monolithic (6k steps) 2.229 +16.0% — Kalavai MoE 2.218 +16.3% +0.47% 1B Base model 2.474 — — Best specialist 2.259 +8.7% — Monolithic (6k steps) 2.097 +15.3% — Kalavai MoE 2.090 +15.5% +0.34% 4.4 Training Duration and the Role of Frozen Layers A key design question is whether frozen layers are necessary. The freeze depth sweep (Appendix E) shows only 1.89 percentage points of variation across freeze depths from 0 to 50% of layers, suggesting freezing is largely optional at 2,000-step training horizons. However, this masks a training duration dependence. Table 5 (and Figure 6 in Appendix D) show fusion improvement as a function of specialist training duration. Without frozen layers, improvement peaks at 5,000 steps (+17.7%) and degrades to +14.7% at 20,000 steps. With four frozen layers, improvement degrades more slowly and becomes the better configuration above approximately 10,000 training steps (+17.0% at 20,000 steps vs. +14.7% without freezing). The crossover occurs at approximately 10,000 training steps: above this threshold, frozen layers prevent over-specialisation that degrades post-hoc fusion. Minimum viable specialist training. Fusion improvement is non-trivial even at extreme compute constraint: 50 specialist training steps produces +4.0% gain (freeze=0) and 100 steps produces +5.1%—meaningful improvements achievable in under two minutes on a consumer GPU. This directly supports the cooperative accessibility argument: compute-constrained contributors can participate with minimal investment, provided they train for at least a few dozen steps to achieve some representational divergence from the shared initialisation. Table 5: Fusion improvement vs. base model as a function of training duration, with and without frozen layers. Pythia-410M, seed 42, per-domain equal-weight evaluation (base EW loss = 2.6510). Bold entries indicate the better configuration at each step count; † marks the crossover point. Steps Freeze=0 (% vs. base) Freeze=4 (% vs. base) Leader 50 +4.0% +3.5% Freeze=0 100 +5.1% +4.5% Freeze=0 500 +9.2% +8.3% Freeze=0 1,000 +12.1% +10.8% Freeze=0 2,000 +15.5% +14.2% Freeze=0 5,000 +17.7% +17.5% Freeze=0 10,000† +16.7% +17.7% Freeze=4 20,000 +14.7% +17.0% Freeze=4 Practical guideline: For specialist training up to approximately 10,000 steps, frozen layers are largely optional (freeze=0 peaks at 5,000 steps with +17.7%). For training horizons beyond 10,000 steps, freezing the first K layers is recommended to prevent over-specialisation (+2.4p advantage for freeze=4 at 20,000 steps). The optimal K is not sensitive: the freeze depth sweep shows only 1.89p variation across K∈0,2,4,6,8,12K∈\0,2,4,6,8,12\ at 2,000 steps (Appendix E). Note that the 1B freeze depth (K=4K=4, 25% of layers) was not independently optimised; we use the 410M-derived value and rely on the freeze depth sweep’s low sensitivity (1.89p spread) as justification that this choice is unlikely to materially affect results. The 6.9B experiment uses K=6K=6 at only 1,000 specialist training steps—a regime where the 410M analysis shows freezing is largely optional and the crossover threshold (∼ 10,000 steps) is still in the future. The 6.9B result is therefore insensitive to freeze depth at these step counts, consistent with the 410M finding that freeze sensitivity is low below 10,000 training steps (Appendix E). A freeze depth sweep at 6.9B is included in the step-budget sweep (Appendix A). 4.5 Routing Must Be Learned Routing architecture matters only insofar as the router is trained. Uniform routing (equal 1/N1/N weight to each specialist, no training) degrades performance by −1.19%-1.19\% relative to the best individual specialist—below-zero because equal-weight mixing averages each specialist’s cross-domain degradation with no domain assignment. A trained linear router achieves +7.70%+7.70\%; a 2-layer MLP achieves +7.72%+7.72\%—effectively identical. Routing precision does not matter beyond the threshold of learning a domain assignment. The fused model achieves oracle-optimal routing. Domain-level oracle assignment—routing each domain’s evaluation set to its own specialist—achieves equal-weight loss 2.218319 at 410M. The trained linear router achieves 2.218316: a gap of 3×10−63× 10^-6 nats. Hard routing (argmax of learned gates, all experts forward-pass) achieves +7.72%+7.72\%—identical to soft routing—confirming that specialist participation, not weighting precision, drives improvement. Full results in Appendix H. 4.6 Capacity Controls: The Cooperative Advantage Is Distributed Training The fused model has 3×3× the unfrozen parameters of any individual specialist. A Pythia-1.4B model (3.5×3.5× total parameters) trained 6,000 steps on mixed data achieves +10.87%+10.87\% vs. best specialist—exceeding the cooperative gain (+7.72%+7.72\%), but requiring centralised access to all training data. A multi-head baseline (same parameter count as MoE, hard-routing each token to one specialist via learned gates) achieves +7.72%+7.72\%—identical to soft MoE—confirming that the cooperative improvement requires neither additional parameters nor a specific routing mechanism, only that specialists are trained independently. Full comparison in Appendix H. 4.7 The Role of Shared Initialisation A structural claim of Kalavai is that all specialists must begin from the same checkpoint. We test this empirically by training specialists from checkpoints at different training stages and measuring the effect on fusion quality (Pythia-410M, 2,000 specialist training steps, 3 domains, seed 42 for small-gap and 3 seeds for control/large-gap). Three conditions: (1) Control: all specialists from step 10,000 (identical initialisation); (2) Large gap: specialists from step 5,000 / 10,000 / 20,000 respectively (spanning 2×2× training progress); (3) Small gap: step 8,000 / 10,000 / 12,000 (± 20% around anchor). Table 6: Shared initialisation ablation at Pythia-410M. Note: this experiment uses mixed-domain held-out evaluation, not the per-domain equal-weight protocol used elsewhere in the paper. All values are internally consistent; the meaningful comparison is the relative degradation across conditions, not the absolute loss values. “MoE loss” is absolute mixed-domain held-out loss; lower is better. “Best Spec. Loss” is the best individual specialist’s mixed-domain loss for that condition. “vs. Base” uses base mixed-domain loss 2.248. “vs. Spec” is improvement over best individual specialist—this metric is misleading under mismatch because mismatched specialists are also worse (best spec loss degrades from 2.089 to 2.157 under large gap); the absolute MoE loss is the appropriate comparison. Condition Init Revisions Best Spec. Loss MoE Loss vs. Base Seeds Control (matched) 10k / 10k / 10k 2.089 2.015 +10.4% 3 Small gap 8k / 10k / 12k 2.122 2.034 +9.5% 1 Large gap 5k / 10k / 20k 2.157 2.036 +9.4% 3 The absolute MoE quality degrades by 0.93p (10.37% → 9.44%) under the large-gap condition—a modest but consistent degradation. However, the routing behaviour degrades more substantially: under the large-gap condition, the code expert receives 11% weight on fiction inputs versus near-zero under control. The router can no longer reliably distinguish specialist roles. We note that the “improvement vs. best specialist” metric is misleading here: it appears higher under mismatch (large gap: +5.6% vs. control: +3.6%) because the individual specialists are also worse under mismatch. The absolute MoE loss is the appropriate comparison. Interpretation. Shared initialisation is not strictly required for fusion to produce positive improvement over base—mismatched specialists still produce a positive fused model. However, shared initialisation is important for routing stability: mismatched checkpoints produce routing confusion that may become more pronounced at scale or with larger checkpoint gaps. Shared initialisation is the essential coordination requirement of the Kalavai protocol and remains our recommendation for all deployments. 4.8 Heterogeneous Cooperative: Robustness to Realistic Contributor Variation A real cooperative will not have identical training conditions across contributors. We test the protocol’s robustness to three practical sources of variation (Pythia-410M, seed 42): • Control: all three specialists trained identically (bs=2, lr=2×10−52× 10^-5, 2,000 steps). • diff_batch: one specialist trained at bs=4 instead of bs=2. • diff_lr: one specialist trained at lr=4×10−54× 10^-5 instead of 2×10−52× 10^-5. • diff_steps: one specialist trained for 1,000 steps instead of 2,000. Table 7: Heterogeneous cooperative results (Pythia-410M, per-domain equal-weight evaluation). “Δ vs. control” is the difference in fusion gain from the identical-conditions baseline. Condition MoE EW Loss vs. Spec vs. Base Δ vs. control Control (identical) 2.2185 +7.72% +16.33% — diff_batch 2.2401 +7.74% +15.49% +0.01p diff_lr 2.1576 +7.73% +18.63% +0.01p diff_steps 2.2001 +7.33% +17.00% −-0.39p The maximum spread across all heterogeneous conditions is 0.41p—well within the noise floor of the experiment. The protocol is robust to realistic variation in batch size, learning rate, and training budget. The only meaningful degradation is a 0.39p reduction when one specialist trains for half the steps; even then, the fusion gain remains +7.33% vs. best specialist. This validates the cooperative premise: contributors do not need to coordinate hyperparameters, only the shared checkpoint and architecture. 4.9 High-Divergence Domains: Phase 2 Experiments Phase 1 establishes that the protocol works at English domain scales (code/science/fiction, divergence ∼ 8–16%). Phase 2 tests whether the divergence-proportional gain relationship extends to settings where KALAVAI is most practically valuable: domains invisible to the base model and languages not in its training corpus. 4.9.1 Private-Domain Fusion (Experiment 2) Three highly specialised domains are selected: medical (PubMed article abstracts, ccdv/pubmed-summarization), legal (European legislation, lex_glue/eurlex), and patent (patent descriptions, big_patent/a). Pythia-410M step10000 with K=0K=0 frozen layers (Section 4.4 shows freeze=0 is optimal below 10k steps), 2,000 specialist training steps, 3 seeds. Table 8: Phase 2 Experiment 2: Private-domain fusion results (Pythia-410M, per-domain equal-weight evaluation). Per-seed mean divergence: 18.52%, 18.51%, 18.51%. Routing: medical 99.98%, legal 99.77%–100%, patent 91.65%–98.75%. Monolithic baseline: 6,000 steps on shuffled medical+legal+patent mix. Scale Method EW Loss vs. Best Spec. vs. Monolithic Seeds Std Pythia-410M Base model 2.954 — — — — Best specialist 2.694 — — 3 — Monolithic (6k) 2.462 — — 1 — Weight avg. 2.709 — — — — Kalavai (MoE) 2.418 +10.17% +1.78% 3 ± 0.15p The legal specialist diverges 34.16% from base—the largest per-domain divergence in any Phase 1 or Phase 2 experiment. Medical diverges 12.71%, patent 8.68%, yielding a mean of 18.52%. The mean 0.55× conversion rate (18.52% divergence → +10.17% gain) is higher than the English domain rate (0.49×), consistent with the pattern that high-divergence settings convert more efficiently. All three seeds achieve GO verdict (divergence >>15% AND gain >>7%), with seed variance ± 0.15p confirming robustness. The +1.78% improvement over the monolithic baseline (which has access to all domains during training) demonstrates that zero-data-sharing cooperative training remains competitive with centralised mixed-domain training even in professional domain settings. 4.9.2 Cross-Lingual Fusion (Experiment 1) Four specialists are trained on Pythia-410M step10000 (K=0K=0, 2,000 steps): Tamil (Wikipedia ta, 208k chunks), Yoruba (Wikipedia yo, 13.7k chunks), Welsh (Wikipedia cy, 37.5k chunks), and code (CodeSearchNet Python). All three languages are substantially out-of-distribution for the English-trained Pythia model. Table 9: Phase 2 Experiment 1: Cross-lingual fusion perplexity (Pythia-410M, seeds 137/2026—perfect routing). Seed 42 had router collapse on Yoruba (gate 99.84% Tamil); reported separately. Code improvement is small because the base model already achieves low perplexity on Python. Domain Divergence Base PPL Specialist PPL MoE PPL Tamil 23.3% 4.2 3.0 3.0 Yoruba 45.5% 41.9 7.7 7.7 Welsh 33.2% 102.7 22.1 22.1 Code 0.4% 8.2 8.1 8.1 Figure 2: Cross-lingual fusion results: base model perplexity vs. specialist vs. Kalavai MoE (Pythia-410M, seeds 137/2026). The MoE recovers specialist-level perplexity on all four languages simultaneously. Yoruba improvement 5.4× (PPL 41.9→ 7.7); Welsh 4.6× (102.7→ 22.1). Improvement annotations show the base→ ratio. Seeds 137 and 2026 achieved perfect routing; seed 42 had router collapse (see text). Seeds 137 and 2026 achieve near-identical results (+21.76% / +21.75% vs. best specialist, ± 0.005p) with perfect routing: each language routes to its specialist at >>99.98% gate weight. The extreme Yoruba improvement (PPL 41.9→ 7.7, 5.4×) reflects that Pythia’s English-dominated training produces very poor Yoruba representations; the specialist corrects this, and the router successfully assigns Yoruba inputs to the Yoruba specialist. Router collapse at seed 42. Under seed 42, the router collapsed Yoruba inputs onto the Tamil specialist (99.84% gate weight). Both Tamil and Yoruba are tokenizer-OOD byte-fallback scripts; at this random seed, their hidden-state representations were insufficiently differentiated for the router to separate them. Yoruba MoE PPL remained 41.5 (near base) rather than 7.7, reducing the gain for seed 42 to +6.14%. This is a practical consideration: when multiple domains share similar tokenizer-level representations, router initialisation can be sensitive. 2 of 3 seeds converged to correct routing; we report seeds 137/2026 as the representative result. Mean fusion gain (all 3 seeds): +16.55% ± 9.02p; mean divergence 25.65%; final verdict GO. Excluding seed 42 router collapse: +21.76% ± 0.005p. 4.9.3 20-Contributor Federation (Experiment 3) This is the largest cooperative tested: 20 specialists (10 languages + 10 domains) on Pythia-1B, scaling from the 3–4 specialists of Phase 1. Each specialist trains 2,000 steps on its own data; the linear router then trains 1,000 steps on mixed data (physical bs = 4, gradient accumulation = 5, effective bs = 20, lr = 0.0002).555The default router uses mean-pooled specialist hidden states as input. A base-model hidden-state router variant—using only the frozen base model’s hidden states, with no specialist-specific signal—achieves +16.67% vs. best specialist (MoE EW loss 2.3143), compared to +16.79% for the specialist-hidden-state router (2.3108). The 0.12p difference is within noise. Routing distributions are nearly identical across both variants, including the medical–chemistry confusion pattern (Section 4.9.3). This confirms that routing quality is determined by the representational geometry of the shared initialisation, not by specialist-specific hidden states. The fused model achieves +16.71% vs. best specialist and +17.09% vs. base (best-specialist EW loss 2.7771; MoE EW loss 2.3130; mean divergence 15.68%, 3 seeds, ± 0.07p). Table 10: Per-domain MoE gain vs. base (%) for the 20-contributor federation (Pythia-1B, 3-seed mean across seeds 42/137/2026). Gain = (Lbase−LMoE)/Lbase×100(L_base-L_MoE)/L_base× 100. † Undertrained domains: fewer than 500 training chunks (dialogue: 184, instructions: 283), flagged as warnings during data loading. All other domains show positive gains; language specialists (mean +23.8%) outperform domain specialists (mean +14.8% excluding † ) due to higher base-model perplexity on non-English text. Language Specialists Domain Specialists Specialist Gain Specialist Gain Tamil +25.29% Code +2.70% Yoruba +58.37% Medical +14.41% Welsh +37.21% Legal +36.48% Spanish +5.20% Patent +8.37% Hindi +18.04% Math +22.47% Swahili +38.26% Finance +12.54% Vietnamese +12.34% Chemistry +13.12% Arabic +11.24% Fiction +8.34% Indonesian +12.49% Dialogue −-23.84%† Thai +19.15% Instructions −-12.54%† Mean +23.76% Mean (all) +8.21% Router robustness note. A base-model hidden-state router variant (seed 42) achieves +16.67% vs. best specialist (EW loss 2.3143) with near-identical per-domain routing, confirming router input choice does not affect fusion quality at this scale. Router distribution. The linear router achieves near-perfect specialisation: 17 of 20 domains assign >>98% gate weight to the correct specialist, and all 10 language specialists route >>99.4% correctly. Two patterns merit attention. Medical–chemistry routing. Medical routes 60.2% to itself and 38.4% to chemistry; chemistry routes 52.7% to medical and 46.6% to itself. These two domains share scientific vocabulary and document structure (abstracts, methods, results), making their hidden-state representations similar. This is a genuine domain-similarity finding, not a protocol failure: the router correctly identifies their overlap, and both domains show positive MoE gains (+14.3% and +13.1% respectively). Data-scarce degradation. Dialogue (−-23.84% vs. base) and instructions (−-12.54%) are the only two degraded domains. Crucially, routing is correct for both (dialogue: 97.1% own gate; instructions: 88.9% own gate)—the specialists themselves are undertrained, not misrouted. With only 184 and 283 training chunks respectively, these specialists could not diverge meaningfully from base, so routing to them incurs overhead without benefit. This is consistent with the divergence-gain framework: insufficient training data produces insufficient divergence, which produces no gain. The protocol requires minimally sufficient specialist training data to function. Excluding these two data-scarce specialists, mean gain across the remaining 18 domains is +19.8%. Scale validation. The +16.71% gain (3-seed mean) at 15.68% mean divergence is +6.57p above the linear regression prediction (+10.11%) from Section 4.10, consistent with the pattern that heterogeneous multi-domain cooperatives—mixing language and domain specialists with widely varying divergence levels—outperform the English-domain regression baseline. The routing saturation result holds at scale: a simple linear router on 20 specialists converges to near-optimal domain assignment, confirming that router architecture does not matter once shared initialisation is in place. Replication note. Experiment 3 was replicated across three random seeds (42, 137, 2026). The 3-seed mean is +16.71% ± 0.07p vs. best specialist (+17.09% ± 0.07p vs. base), with mean divergence 15.68% ± 0.04p. Variance is tighter than Experiment 2 (± 0.15p) and comparable to Phase 1 (± 0.01–0.02p at 410M/1B). Per-domain routing distributions are near-identical across seeds, with all 10 language specialists routing >>99% correctly in every seed. The result is confirmed stable. 4.10 Divergence–Gain Relationship Across six experimental conditions—Qwen-1.5B (mean div. 3.16%), Pythia-6.9B (8.73%), Pythia-1B (15.28%), Pythia-410M (15.65%), Exp 2 private-domain (18.52%), and Exp 1 cross-lingual (25.65%)—fusion gain scales monotonically with specialist divergence (Figure 3). Exp 3 (20-contributor, 15.68%, +16.71%, 3-seed mean) serves as an out-of-sample validation point (Table 11). Linear regression fit. We fit OLS regression to the six data points. The linear model (gain=−2.72+0.82×divergencegain=-2.72+0.82×divergence) achieves R2=0.856R^2=0.856 (slope 95% CI [0.35, 1.28], n=6n=6, t-distribution). A log-linear fit achieves only R2=0.662R^2=0.662—the relationship is closer to linear than sublinear across the 3–26% divergence range. The regression line and 95% prediction band are shown in Figure 3. Figure 3: Fusion gain vs. mean specialist divergence (%) with OLS regression line and 95% prediction band. Linear fit: gain=−2.72+0.82×divgain=-2.72+0.82×div (R2=0.856R^2=0.856, n=6n=6 in-sample conditions). English-domain conditions (Qwen, Pythia-6.9B/1B/410M) cluster near the line; Exp 2 (private, purple) and Exp 1 (cross-lingual, red) both lie above the English-domain prediction, consistent with base-model incompetence on target domains producing outsized gains. The cross-lingual condition is the largest in-sample outlier (+3.6p). Annotations show gain/divergence conversion rate per condition. Note: Exp 3 (20-contributor, div = 15.68%, gain = +16.71%, 3-seed mean) is an out-of-sample validation point lying +6.57p above the regression line (Table 11); it is not shown in this figure as the regression was fit before Exp 3 results were available. Table 11: Summary of divergence–gain relationship across all Phase 1 and Phase 2 experiments. Predicted gain from linear fit =−2.72+0.82×div=-2.72+0.82×div; residual = actual−-predicted. Predicted values are computed from full-precision OLS coefficients (slope =0.8170=0.8170, intercept =−2.724=-2.724); the displayed formula rounds to two decimal places, so applying it directly may differ by ≤ 0.1p. The regression was fit on the six in-sample conditions (rows 1–6); Exp 3 (row 7) is an out-of-sample validation point. †Qwen predicted gain (≈−0.1%≈-0.1\%, effectively zero) is at the divergence floor (≈ 3.3%); the cooperative produced modest positive gain in practice (+1.06%). ‡In a 20-specialist cooperative no single specialist achieves strong equal-weight performance across all 20 domains; the best specialist (Arabic, EW 2.7771) is only 0.46% above base (2.7898), making gain-vs-spec ≈ gain-vs-base (+17.17%). The nominal conversion rate >1>1 reflects this near-equal baseline, not an anomalous efficiency. Condition Mean Div. Gain vs Spec Conv. rate Pred. gain Residual Qwen-1.5B (Ph.1) 3.16% +1.06% 0.34× ≈ 0%† — Pythia-6.9B (Ph.1) 8.73% +6.53% 0.75× +4.41% +2.12p Pythia-1B (Ph.1) 15.28% +7.49% 0.49× +9.81% −-2.32p Pythia-410M (Ph.1) 15.65% +7.72% 0.49× +10.11% −-2.39p Exp 2: private (Ph.2) 18.52% +10.17% 0.55× +12.43% −-2.26p Exp 1: cross-lingual (Ph.2) 25.65% +21.76% 0.85× +18.18% +3.58p Exp 3: 20-contrib (Ph.2, OOS) 15.68% +16.71% 1.07ׇ +10.14% +6.57p The residual pattern is informative. The four English-domain conditions (Pythia 410M/1B/6.9B and Private) all lie within ± 2.4p of the line, forming a coherent cluster. The 6.9B point sits +2.12p above the line—consistent with larger models converting divergence more efficiently. The cross-lingual condition is the largest in-sample outlier (+3.58p), explained by base-model near-incompetence on Yoruba and Welsh: when the base model achieves near-random perplexity on a domain, the specialist corrects this from a high baseline and the router routes with near-perfect confidence, leaving no gain on the table. The out-of-sample Exp 3 point (+6.57p residual, 3-seed mean) lies further above the line, consistent with its heterogeneous mix of high-divergence language specialists (Yoruba +58%, Welsh +37%, Swahili +38%) pulling the cooperative gain above what the English-domain regression would predict. The practical implication is that the formula gain≈0.82×divergencegain≈ 0.82×divergence provides a reliable pre-training estimate for English-domain and professional-domain cooperatives. Cross-lingual settings with low-resource languages will likely exceed this prediction. The formula also sets a divergence floor: below ≈ 3.3% mean divergence, the linear prediction approaches zero, indicating the cooperative is unlikely to produce positive gains over individual specialists. Base-model competence as a secondary predictor. Specialist divergence captures how much specialists move from the base model; a complementary factor is how competent the base model already is on the target domain. Across the six experimental conditions, the log of the mean base-model perplexity on each domain’s evaluation data correlates with the conversion efficiency (gain / divergence) at r=+0.560r=+0.560 (Pearson, n=6n=6), compared with r=+0.614r=+0.614 for divergence alone. On the six-point sample this is suggestive rather than definitive, but the pattern is mechanistically plausible: when the base model achieves near-random perplexity on a domain (Yoruba PPL ≈42≈ 42, Welsh PPL ≈103≈ 103), the specialist must correct the base from a high-loss floor, the router routes with near-certainty, and essentially all specialist gain is preserved. When the base is already competent (English code PPL ≈7≈ 7), specialist gains are smaller in absolute terms and the cooperative receives less incremental value. This suggests a two-factor heuristic—measure both specialist divergence and base-model competence before committing to a cooperative—though validation on more conditions is needed before treating the secondary predictor as quantitatively reliable. Figure 4 shows the relationship across all conditions. Figure 4: Base-model perplexity as a secondary predictor of cooperative fusion efficiency. Left: Conversion efficiency (gain / divergence) versus mean base-model perplexity per condition. Centre: Same with log-scaled perplexity axis (Pearson r=+0.560r=+0.560). Right: Divergence versus gain coloured by base-model PPL quartile. Cross-lingual conditions (high base PPL) convert divergence most efficiently; English-domain conditions (low base PPL) sit near the baseline conversion rate. Dashed lines are OLS fits; n=6n=6 conditions. 5 Analysis Routing quality does not depend on router architecture. Under per-domain equal-weight evaluation, a uniform router achieves −1.19%-1.19\% vs. best specialist (worse than the best individual specialist: uniform mixing averages cross-domain degradation); a trained linear router achieves +7.70%+7.70\%; a 2-layer MLP achieves +7.72%+7.72\%. The ordering—uniform << learned routing; linear ≈ MLP—confirms that the gap is entirely explained by whether routing is trained, not by the function class. Full results in Appendix G. The strongest evidence for this claim is oracle saturation. A domain-level routing oracle—the optimal static assignment of each evaluation domain to whichever specialist achieves the lowest loss on that domain—achieves EW loss 2.218319 at 410M. The learned linear router achieves 2.218316—a gap of 3×10−63× 10^-6 nats, or 0.0002%0.0002\% of the MoE loss. At 6.9B, the oracle gap is <10−5<10^-5 nats (also effectively zero). At 1B, the oracle gap is 0.059 nats (+2.73% headroom), reflecting a modest routing suboptimality at 1B. At 410M and 6.9B, the learned router has converged to the domain-level optimum: there is no remaining gain available from routing improvements at the domain granularity at which our evaluation is performed. The router is not the bottleneck; the representational structure created by shared initialisation fully determines routing quality, and a simple linear layer is sufficient to exploit it. This claim extends to 20-specialist scale. In the Exp 3 federation, a base-model hidden-state router—which receives only the frozen base model’s representations as input, with no access to specialist hidden states—achieves +16.67% vs. best specialist, compared to +16.79% for the specialist-hidden-state router (a 0.12p difference). Routing distributions are nearly identical: all 10 language specialists route >>99.3% correctly under both variants, and the medical–chemistry confusion pattern (60/40 split) persists across both, confirming it reflects genuine domain-level semantic overlap rather than a router limitation. The base model’s representational geometry, established at shared initialisation, fully determines which specialist should handle each token—specialist-specific signal during routing is redundant. Improvement is robust across training maturities at 410M. Fusion improvement is consistent at 410M across Pythia checkpoints from step 5,000 to step 143,000 (+7.0%–+8.8%); the mechanism does not depend on the base model being under-trained. At 1B, improvement drops markedly at the fully-trained checkpoint (+0.40% at step 143,000 vs. +8.75% at step 5,000), consistent with the divergence–gain relationship: fully-trained base models produce less specialist divergence. (Appendix I). Improvement scales monotonically with specialist count. Under five-domain equal-weight evaluation, adding specialists shows clear monotonic improvement: 2 specialists (+1.76%+1.76\%, code and fiction only), 3 specialists (+4.39%+4.39\%), 4 specialists (+11.39%+11.39\%, adding math), and 5 specialists (+12.95%+12.95\%, all five domains). Each new specialist improves its own domain without degrading others; the monotonic increase reflects domain coverage expansion (Appendix J). Token-level routing confirms mid-sequence switching. On hybrid-domain prompts, the router produces 2.2 expert switches per prompt on average, assigning domain-appropriate weights within a single sentence—confirming the router operates at token granularity, not document level (Appendix N). Representational divergence confirms specialisation. Figure 5 shows the cross-domain evaluation loss matrix at step 2,000. The pronounced diagonal structure confirms that each specialist has learned domain-specific representations: each specialist achieves its lowest loss on its own domain and highest loss on the furthest domain. The code specialist evaluates at 1.879 on code data and 2.909 on science data—a gap of 0.032 above base on science, confirming out-of-domain degradation. The off-diagonal pattern directly motivates MoE fusion: a router that dispatches each token to the appropriate diagonal entry recovers all specialist gains. Figure 5: Cross-domain evaluation loss matrix at Pythia-410M, step 2,000 (freeze=4, seed=42). Rows are specialists; columns are evaluation domains. The diagonal entries (own-domain performance) are lower than off-diagonal (cross-domain), confirming that each specialist has diverged in a complementary direction. The MoE router recovers diagonal performance across all domains simultaneously. Color scale: green indicates lower loss (better performance), red indicates higher loss. Downstream benchmarks. At 1B scale, MoE leads on HellaSwag (35.0% vs. 34.4% for the base) and best individual specialist (34.2–34.4%). Monolithic training produces the worst HellaSwag score (33.4%). Overall average accuracy: base 50.6%, MoE 49.6%, monolithic 49.3%. Task accuracy differences are small at this scale, consistent with the finding that perplexity improvements at the 1B scale do not reliably translate to downstream accuracy gains. At 6.9B, MoE achieves average accuracy 52.2% versus base 51.6%. Full benchmark tables in Appendix L. Caveat: All benchmarks use 500 examples per task; at this sample size, differences of 1–2 percentage points are within statistical noise. Downstream accuracy results should be treated as directional indicators only, not statistically significant findings. 6 Discussion and Limitations What the 6.9B result means. Fusion gain is proportional to specialist divergence (Table 2). At 410M and 1B, specialists diverge 10–25% from base per domain (mean ∼ 15.5%), producing ∼ +7.5% fusion gain. At 6.9B on the same domains, specialists diverge 7–10% per domain (mean 8.73%)—approximately half the divergence at smaller scales—and fusion gain is +6.53% (± 0.024%, 3 seeds). The conversion rate (gain per unit divergence) is actually higher at 6.9B (0.75×) than at 410M/1B (0.49×): larger models convert divergence into fusion gain more efficiently. The reduced gain at 6.9B is due entirely to reduced specialist divergence, not to any scale-dependent degradation of the protocol. Routing is near-deterministic (>>99.9%) at 6.9B as at all scales. Kalavai gains will be largest precisely where they are most needed: low-resource languages, specialised technical domains, and early-stage models where contributors’ data fills genuine gaps in the base model’s competence, maximising specialist divergence. KALAVAI gains scale with divergence; wherever contributors’ data fills genuine gaps, specialists diverge more and fusion gains more. What the Qwen result means. Qwen-1.5B achieves +1.06% (± 0.01%, 3 seeds) with perfectly deterministic routing (100% per-domain gate weight). This is not a failure case: the gain is small because divergence is small (code 1.76%, fiction 4.56%, mean 3.16%, Table 2), consistent with the divergence-proportional gain relationship at 0.34× conversion rate. Routing succeeds at all divergence levels tested, including Qwen’s 3.16% mean divergence. The simpler narrative: small divergence → small gain; large divergence → large gain. Inference cost. The Kalavai fused model runs all N specialists in parallel at inference, increasing compute by a factor of N for the unfrozen layers. For N=3N=3 with 17% frozen layers, the effective inference overhead is approximately 2.5×2.5× (frozen layers run once; unfrozen layers run 3×3×). Measured benchmarks on an NVIDIA RTX 5090 confirm this: dense MoE latency is 2.86×2.86× base at 410M and 3.35×3.35× base at 1B (Table 25 in Appendix O). The observed hard-switching behaviour (>>99.7% weight on one expert) suggests a potential sparse inference optimisation: run frozen layers once, route, then run only the top-1 expert’s unfrozen layers. We test this directly. At 410M, top-1 routing agreement between full-parallel and single-expert forward passes is 100%—frozen-layer hidden states fully determine which expert would be selected. At 1B, agreement drops to 10%, meaning routing decisions change for 90% of tokens when other specialists’ hidden states are absent. Critically, even with 100% routing agreement at 410M, sparse top-1 inference collapses quality: sparse evaluation loss is 3.106 versus 2.568 for dense MoE (21% degradation relative to dense MoE, worse than the base model at 2.692). At 1B, sparse loss is 2.412 versus 2.382 dense (1.3% degradation), but routing agreement is only 10%. In both cases, sparse inference is not equivalent to dense inference. We hypothesise two factors explain the quality collapse despite correct routing. First, the router input in dense mode is the mean-pooled hidden state averaged across all specialists’ forward passes (see footnote 3); in sparse mode, this becomes a single specialist’s hidden state—a different representation that alters the conditioning context for the gate computation. Second, even near-deterministic routing (>>99.7% weight on one expert in dense mode) preserves a residual ensemble contribution from all specialists that is lost under strict top-1 selection. Both factors mean that running only one specialist’s unfrozen layers discards complementary signal regardless of how accurately the top specialist is identified. The memory footprint remains N×N×, since all specialist weights must be loaded regardless of routing sparsity. We leave efficient sparse inference implementation to future work. The primary value proposition of Kalavai is training-time democratisation—enabling contributors who cannot afford centralised training to collectively produce a superior model—not inference efficiency. Applications. The zero-communication-during-training property enables cooperative training scenarios that are infeasible with synchronous methods: multi-hospital medical language models where patient data cannot leave the facility; multi-jurisdictional legal AI where training data is subject to national regulations; low-resource language coverage where each language community trains a specialist on their language. Phase 2 experiments provide direct empirical evidence for the first and third scenarios: Experiment 2 demonstrates +10.17% on medical/legal/patent domains with no data sharing; Experiment 1 demonstrates that Yoruba Wikipedia contributors can collectively achieve 5.4× perplexity improvement (41.9→ 7.7) with zero Tamil, Welsh, or code data exposure. Scope and limitations. • Inference cost. The fused model runs all N specialists in parallel at inference, at approximately N×N× the compute of a single specialist. • Architecture generality. Primary results use Pythia (GPT-NeoX). The Qwen result (+1.06%) provides a second architecture data point confirming the mechanism generalises beyond GPT-NeoX. • Downstream task accuracy. Perplexity gains are consistent; benchmark gains are modest (<<1p at 1B scale) and should be treated as directional. • Real cooperative deployment. All experiments are simulated cooperatives on single machines. Deployed cooperative training—including data heterogeneity, checkpoint verification, contributor reliability, and freeze specification communication—remains open engineering work. • Frontier-scale behaviour. Experiments reach 6.9B; results are positive at all tested scales, but behaviour beyond 70B is untested. • Domain generality. Phase 2 covers medical, legal, patent, and ten non-English languages. Data-scarce domains (<<300 training chunks) degrade, as the protocol requires minimally sufficient specialist training data. Cross-modal settings (image, audio) are untested. • Gain magnitude on already-competent domains. Gains are proportional to specialist divergence; where the base model is already competent, specialists diverge less and gains are correspondingly smaller (+7.72% at 410M on English domains). 7 Conclusion We have demonstrated that independently trained domain specialists, initialised from a shared checkpoint and fused via a lightweight MoE router, consistently outperform the best individual specialist and equal-compute monolithic training. The mechanism is not routing sophistication—a linear router is optimal—but the combination of specialised representations from domain-specific training with joint inference that aggregates those representations at each token position. The training duration crossover finding provides a practical guideline for cooperative training: frozen layers are optional insurance at training horizons below 10,000 steps and recommended beyond 10,000 steps. The oracle routing saturation result—learned soft routing matches domain-level oracle and hard routing (all <10−5<10^-5 nats from oracle at 410M and 6.9B)—confirms that routing must be trained rather than uniform, but once trained, precision is irrelevant and specialist participation, not weighting scheme, drives improvement. Phase 2 extends the protocol beyond English domains: private professional domains (medical/legal/patent) achieve +10.17% gain at 18.52% mean divergence; cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves +21.76% at 25.65% divergence, with Yoruba perplexity falling 5.4× (41.9→ 7.7) and Welsh 4.6× (102.7→ 22.1). Gain scales with divergence at a conversion rate that improves in high-divergence settings (0.85× cross-lingual vs. 0.49× English domains). The protocol is most valuable precisely where it is most needed. Together, these findings validate the core premise of Kalavai: contributors speaking different languages, working with data they cannot share, and training on hardware they own can collectively produce a model that none of them could build alone. The shared initialisation constraint is the primary coordination requirement: our ablation (Section 4.7) shows that mismatched checkpoints degrade routing clarity, making it the essential protocol constraint that contributors must honour. Broader impact. Kalavai lowers the compute barrier for training competitive language models. Any group that can collectively afford the inference compute of N models can produce a model that matches a single model with N-times the training budget. The protocol is most impactful for under-resourced language communities (Phase 2 Exp 1: Yoruba PPL 41.9→ 7.7 with no English data sharing) and organisations with data privacy constraints (Phase 2 Exp 2: medical/legal/patent +10.17% with zero data sharing). We release all code, experiment scripts, and result artefacts at https://github.com/mechramc/Kalavai. References S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal (2023) Pythia: a suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §4.1. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, and G. Lengyel (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §3. A. Komatsuzaki, J. Puigcerver, J. Lee-Thorp, C. R. Ruiz, B. Mustafa, J. Ainslie, Y. Tay, M. Dehghani, and X. Zhai (2023) Sparse upcycling: training mixture-of-experts from dense checkpoints. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), External Links: Link Cited by: §2. M. Muqeeth, H. Liu, Y. Liu, and C. Raffel (2024) Learning to route among specialized experts for zero-shot generalization. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. J. N. K. Pari (2025) Post-hoc merging of independently trained models. Master’s Thesis, Massachusetts Institute of Technology. Cited by: §2. J. Qin, J. Huang, L. Qiao, and L. Ma (2025) STAR: STacked AutoRegressive scheme for unified multimodal learning. arXiv preprint arXiv:2512.13752. Cited by: §2. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), Cited by: §3. S. Sukhbaatar, J. Weston, and A. Szlam (2024) Branch-train-mix: mixing expert LLMs into a mixture-of-experts LLM. In Conference on Language Modeling (COLM), External Links: Link Cited by: §1, §2. F. Wan, X. Huang, D. Cai, X. Quan, W. Bi, and S. Shi (2024) FuseLLM: knowledge fusion of large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carlin, S. Kornblith, and L. Schmidt (2022) Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2. P. Yadav, C. Raffel, et al. (2024) A survey on model MoErging: recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.07057. Cited by: §2. P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal (2023) TIES-merging: resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), External Links: Link Cited by: §2. L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li (2024) Language models are super mario: absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning (ICML), External Links: Link Cited by: §2. Appendix A Complete Experiment Inventory Table 12 lists all experiments conducted for this paper with their configurations and key outcomes. Table 12: Complete experiment inventory. All experiments are committed to the repository with result JSONs. “Seeds” column indicates number of random seeds; std ≈ 0.00 for all multi-seed runs unless otherwise noted. Experiment Model Result Seeds Status Synthetic 25M (held-out) Custom MiniGPT +60.7% ± 0.7% 3 Done Pythia-410M 3-domain Pythia-410M +7.72% ± 0.02% 3 Done Pythia-1B 3-domain Pythia-1B +7.49% ± 0.01% 3 Done Pythia-6.9B 3-domain Pythia-6.9B +6.53% ± 0.024% 3 Done Qwen-1.5B code+fiction Qwen-1.5B +1.06% ± 0.01% 3 Done Router ablation Pythia-410M Linear=+7.70%+7.70\%, 2-layer=+7.72%+7.72\%, Uniform=−1.19%-1.19\% vs. spec 1 Done Freeze depth sweep (0–12) Pythia-410M +7.92% to +6.03%, 1.89p spread 1+3 Done Maturity sweep 410M (6 ckpts) Pythia-410M +7.03% to +8.81% mixed Done Maturity sweep 1B (4 ckpts) Pythia-1B +0.40% (step143k) to +8.75% (step5k) 1 Done Maturity sweep 6.9B (2 ckpts) Pythia-6.9B +2.43% (step10k), +2.26% (step143k) 1 Done 5-domain scaling (2–5 spec.) Pythia-410M +1.76%+1.76\% (2 spec) to +12.95%+12.95\% (5 spec) vs. spec, 5-domain EW eval 3 Done Monolithic baseline Pythia-410M Mono ++7.28% vs. spec; MoE ++7.72% vs. spec; MoE ++0.47% over mono 3 Done Training duration crossover Pythia-410M Crossover at ≈ 10,000 steps (50-step floor: +4.0%) 1 Done Routing ablation (oracle, uniform) Pythia-410M Oracle dispatch +7.72%+7.72\% = MoE; uniform −1.19%-1.19\% vs. spec 1 Done Hard routing verification Pythia-410M Hard +7.72%+7.72\% = soft; gap <10−5<10^-5 nats from oracle 1 Done Wider model capacity control Pythia-1.4B +10.87%+10.87\% vs. spec; MoE +7.72%+7.72\%; wider exceeds MoE but requires centralised data 1 Done Hard routing verification Pythia-410M Hard +20.27% vs Soft +20.24% (vs base) 1 Done Hybrid routing analysis Pythia-410M 11 switches across 5 prompts 1 Done Downstream benchmarks 1B Pythia-1B MoE leads HellaSwag; near-parity avg 1 Done Downstream benchmarks 6.9B Pythia-6.9B MoE 52.2% vs base 51.6% avg 1 Done Shared init ablation (3 cond.) Pythia-410M Ctrl +10.4%, large-gap +9.4% (abs); router confusion 11% 3/3/1 Done Inference routing agreement Pythia-410M/1B 410M 100%, 1B 10% sparse agreement 1 Done 1B monolithic baseline Pythia-1B Mono ++15.3% vs. base; MoE ++0.34% over mono 3 Done Results integrity audit All 322/322 checks passed, 0 issues — Done Phase 2 experiments (high-divergence domains) Exp 2: Private-domain (410M) Pythia-410M +10.17% ± 0.15p 3 Done Exp 1: Cross-lingual (410M) Pythia-410M +21.76% (seeds 137/2026) 2 GO / 1 PIVOT Done Exp 3: 20-contributor (1B) Pythia-1B +16.71% ± 0.07p vs. spec (mean div. 15.68%) 3 GO Done (seeds 42/137/2026) 6.9B step+freeze sweep Pythia-6.9B Best: k=4, 2k steps, +2.73% ± 0.007p 3 Done Appendix B Synthetic 25M Proof-of-Concept To validate the mechanism on a fully controlled setting, we ran the cooperative protocol on a custom 25M-parameter GPT-style model (6 layers, hidden size 256) trained from scratch on synthetic domain data. Three specialists were trained independently for 5,000 steps each. The fused model achieves ++60.7% ± 0.7% over the best individual specialist on held-out evaluation (3 seeds). The larger improvement compared to Pythia experiments is expected: the synthetic model starts from random initialisation (greater diversity between specialists) and the synthetic domains are maximally distinct. This experiment confirms the mechanism functions end-to-end before any Pythia-scale computation. Appendix C Design Decisions • Why not LoRA? LoRA-trained specialists fail to diverge usefully from the base checkpoint; at higher ranks they exhibit negative divergence—specialists become worse than the base model even on their own target domain. Table 13 shows the ablation at Pythia-410M, seed 42. Table 13: LoRA ablation at Pythia-410M (seed 42, 2,000 training steps). “Mean div.” is the equal-weight average of each specialist’s improvement over base on its assigned domain. Negative divergence means the specialist is worse than the base model. Full fine-tuning (bottom row) is the main Kalavai result. Per-domain equal-weight evaluation. Method LR Mean div. MoE vs. spec MoE vs. base LoRA r=8r=8 2e-4 −-1.48% ++0.32% ++0.65% LoRA r=64r=64 2e-4 −-20.31% −-13.85% −-14.92% LoRA r=64r=64 5e-4 −-29.25% −-15.22% −-19.97% Full FT (freeze=4) 2e-4 ++15.65% ++7.72% ++16.3% At r=8r=8, LoRA adapters produce near-zero divergence (−-1.48% mean, below the divergence floor of ≈ 3.3% at which the empirical formula predicts zero gain), yielding +0.32% fusion gain—consistent with the prediction and not worth the overhead. At r=64r=64, specialists become markedly worse than the base model on their own domain (code: −-37.3%; science: −-29.0%), pushing mean divergence to −-20.3% and causing the fused model to underperform base by 14–20%. Higher learning rate (5×10−45× 10^-4) worsens this: mean divergence falls to −-29.3%, gain −-15.2%. The mechanism is over-fitting: LoRA at this rank modifies enough capacity to harm generalisation without producing the stable representational divergence that full fine-tuning achieves. Both failure modes are consistent with the divergence–gain framework: insufficient divergence produces insufficient gain; negative divergence produces negative gain. Full fine-tuning of unfrozen layers is required for Kalavai to work. • Why softmax over argmax? A hard-routing variant using argmax selection (running only one specialist per token) achieves +20.27% over base; soft routing achieves +20.24%—a 0.03p difference that is not practically meaningful. We use softmax as the default. Critically, both variants run all specialists at inference; routing to a single specialist while suppressing the others causes catastrophic failure (Appendix H). • Why a linear router? A 2-layer MLP router achieves +7.72%+7.72\% versus +7.70%+7.70\% for a linear router—an immaterial difference. Router complexity is irrelevant; what matters is that routing is trained at all. Uniform routing (no training) achieves −1.19%-1.19\% vs. best specialist. Appendix D Training Duration Crossover Figure Figure 6: Fusion improvement vs. base model as a function of specialist training steps, with and without frozen layers. Pythia-410M, seed 42. See Table 5 in Section 4.4 for exact values. Appendix E Freeze Depth Sweep Table 14: Freeze depth sweep at Pythia-410M, 2,000 specialist training steps. Seed 42 single-run for depths 4–12; three seeds for depths 0, 2. “% Frozen” refers to fraction of total transformer layers frozen. Per-domain equal-weight evaluation (bs=4). Freeze Layers % Frozen MoE Loss Improvement (seed 42) Std (3 seeds) 0 0% 2.199 +7.92% ± 0.012% 2 8% 2.207 +7.86% ± 0.015% 4 17% 2.218 +7.72% — 6 25% 2.241 +7.49% — 8 33% 2.269 +7.17% — 12 50% 2.346 +6.03% — The total spread across all tested freeze depths is 1.89 percentage points (7.92% to 6.03%). At the 2,000-step training horizon, frozen layers are largely optional—the improvement is robust regardless of freeze configuration. Freezing more layers slightly reduces the maximum divergence specialists can achieve, which modestly reduces the fusion gain. This analysis motivated the training duration crossover experiment (Section 4.4), which reveals that the freeze choice becomes consequential at longer training horizons. Appendix F Equal-Compute Monolithic Comparison Figure 7: Per-domain equal-weight held-out loss for Base model, Monolithic baseline, and Kalavai MoE at Pythia-410M scale. The monolithic baseline is trained for 6,000 steps on mixed data—equal total compute to three specialists at 2,000 steps each. MoE EW loss 2.218 vs. base 2.651 and monolithic 2.229 (Table 4). The decomposition of the monolithic gap is discussed in Section 4.3. Briefly: specialisation contributes ≈ 0.4p (best specialist vs. monolithic) and routing contributes the remaining ≈ 7.1p (fused model vs. best specialist). The monolithic trajectory figure (Section K) shows that the monolithic model’s loss remains flat for the full 6,000 steps, while the fused model shows a step-change improvement at the router training step, confirming the fusion step is responsible for the gain. Appendix G Router Architecture Ablation Table 15: Router architecture ablation at Pythia-410M (freeze=4, seed=42, 2,000 training steps), per-domain equal-weight evaluation. Best specialist EW loss: 2.404 (baseline). Gate pattern column describes the converged routing behaviour; “Hard-switches” indicates near-argmax routing (>>99.7% weight on dominant expert). Router EW Loss vs. Best Spec. Gate Pattern Uniform (1/N1/N, no training) 2.432 −1.19%-1.19\% Equal Simple linear (trained) 2.219 +7.70%+7.70\% Hard-switches 2-layer MLP (trained) 2.218 +7.72%+7.72\% Hard-switches Both trained routers converge to near-deterministic routing: the code domain is assigned 99.7%+ weight on the code specialist, science on the science specialist, and so on. The uniform averaging result (−1.19%-1.19\%) shows that shared initialisation without routing training is worse than the best individual specialist: equal-weight mixing averages each specialist’s cross-domain degradation with no domain compensation. The +8.9+8.9p gap between uniform and learned routing reflects the router’s ability to suppress out-of-domain specialists per token, recovering the domain-specific quality of each specialist. Figure 8 shows the learned gate weight distributions for all three domains. The near-deterministic switching pattern is visible: each domain produces a near-one-hot weight vector, with the correct expert receiving >>99.7% of the weight. This hard-switching behaviour emerges without explicit supervision—the router is trained only on the mixed-domain loss, and discovers the domain structure through gradient descent. Figure 8: Learned gate weight distributions for all three domain evaluation sets (Pythia-410M, freeze=4, seed=42). Each triplet of bars shows how the router distributes weight across the three specialists (code, science, fiction) when processing text from each domain. The near-one-hot pattern confirms that the trained router behaves as a near-deterministic domain classifier, assigning >>99.7% weight to the correct specialist. Appendix H Dispatch Failure and Capacity Controls Single-Specialist Dispatch Table 16: Routing strategies at Pythia-410M (freeze=4, seed=42), per-domain equal-weight evaluation (base EW loss 2.651, best specialist 2.404). All configurations use the same three specialist models. Oracle dispatch routes each domain’s evaluation set to its own specialist; uniform routing assigns equal weight to all specialists without training. Method Specialists Run Routing EW Loss vs. Best Spec. Base model — — 2.651 — Best specialist (fiction) 1 of 3 Fixed 2.404 — Uniform routing (no training) 3 of 3 Equal 1/31/3 2.432 −1.19%-1.19\% MoE soft routing (trained) 3 of 3 Softmax weights 2.218 +7.72%+7.72\% MoE hard routing (argmax) 3 of 3 Argmax (all run) 2.218 +7.72%+7.72\% Oracle dispatch (per domain) 1 of 3 Domain oracle 2.218 +7.72%+7.72\% Capacity Controls Table 17: Capacity control comparison, per-domain equal-weight evaluation (base EW loss 2.651, best specialist 2.404). All methods use Pythia-410M base checkpoint (step10000). Wider model = Pythia-1.4B trained 6,000 steps on all domain data. Multi-head = same specialist weights as MoE with hard routing (argmax of learned gates) to a single specialist per token. Method Params (unfrozen) EW Loss vs. Best Spec. Monolithic Pythia-410M (6,000 steps) 1× 2.229 +7.28%+7.28\% Kalavai MoE (3 specialists, 2,000 steps each) 3× unfrozen 2.218 +7.72%+7.72\% Multi-head baseline (same params as MoE) 3× unfrozen 2.218 +7.72%+7.72\% Wider single model (Pythia-1.4B, 6,000 steps) 3.5× total 2.143 +10.87%+10.87\% Appendix I Maturity Sweeps Table 18: Maturity sweep results at Pythia-410M. % Training indicates fraction of total Pythia pre-training steps. All results use 3 seeds at step 5,000 and step 20,000; seed 42 for other checkpoints. Improvement vs. best individual specialist, per-domain equal-weight evaluation (bs=4). Checkpoint % Training Base Loss MoE Loss Improvement step5000 3.5% 2.855 2.324 +8.81% step10000 7.0% 2.651 2.218 +7.72% step20000 14.0% 2.496 2.122 +7.03% step50000 35.0% 2.270 2.015 +7.25% step100000 70.0% 2.159 1.955 +7.06% step143000 100.0% 2.157 1.960 +7.51% Table 19: Maturity sweep results at Pythia-1B (seed 42 all checkpoints). Improvement vs. best individual specialist, per-domain equal-weight evaluation (bs=4). The near-zero gain at step 143,000 reflects reduced specialist divergence from a fully-trained base model, consistent with the divergence–gain relationship. Checkpoint % Training Base Loss MoE Loss Improvement step5000 3.5% 2.703 2.194 +8.75% step20000 14.0% 2.279 2.007 +6.68% step50000 35.0% 2.109 1.928 +6.40% step143000 100.0% 1.992 1.960 +0.40% Table 20: Maturity results at Pythia-6.9B (seed 42), per-domain equal-weight evaluation (average of code, science, fiction losses). Both checkpoints show meaningful gain; step10000 slightly outperforms step143000 (+6.53% vs. +5.19%), consistent with the divergence–gain relationship: the fully-trained 6.9B base diverges less, yielding slightly lower fusion gain. Checkpoint % Training Base Loss (EW) MoE Loss (EW) Improvement step10000 7.0% 2.320 2.118 +6.53% step143000 100.0% 1.900 1.758 +5.19% The 410M maturity sweep shows consistent improvement from +7.03% to +8.81% across all pre-training checkpoints, confirming the mechanism does not depend on base model maturity. At 1B, improvement is strong at early checkpoints (+8.75% at step 5,000) but drops markedly to +0.40% at the fully-trained checkpoint (step 143,000). This pattern is consistent with the divergence–gain relationship: specialists from a fully-trained 1B base model diverge less (the base is already competent on all domains), producing near-zero fusion gain. The 6.9B maturity table (Table 20) shows +6.53% at step10000 and +5.19% at step143000. Figure 9: Maturity sweep results for Pythia-410M, Pythia-1B, and Qwen-1.5B across training checkpoints. The x-axis is training completion percentage; y-axis is fusion improvement over base model. Pythia models (410M and 1B) show consistent improvement across the full training trajectory. Qwen-1.5B at full training shows +1.06%+1.06\%. The +1.06%+1.06\% reflects small specialist divergence (3.16%), not a routing failure. Appendix J 5-Domain Specialist Scaling Table 21: Specialist count scaling at Pythia-410M, five-domain equal-weight evaluation (code, science, fiction, math, multilingual). Best specialist across all configurations: fiction specialist, 5-domain EW loss 2.583. Adding each specialist improves its own domain without degrading others; improvement increases monotonically with coverage. All results: 3 seeds. Specialists Domains Improvement Std Note 2 Code, Fiction +1.76%+1.76\% ± 0.007% 3/5 domains uncovered 3 Code, Science, Fiction +4.39%+4.39\% ± 0.024% 2/5 domains uncovered 4 Code, Science, Fiction, Math (GSM8K) +11.39%+11.39\% ± 0.022% 1/5 domain uncovered 5 + Multilingual (Spanish Wikipedia) +12.95%+12.95\% ± 0.022% All 5 domains covered Improvement scales monotonically with specialist count: each additional specialist improves its own domain without degrading the others. The +6.96+6.96p jump from 3 to 4 specialists reflects the addition of a math specialist covering a domain where the base model has high loss (ℒmathbase=2.611L_math^base=2.611). The 5-specialist result (+12.95%+12.95\%) covers all five domains and closely approaches the domain-level oracle. The 2-specialist result (+1.76%+1.76\%) reflects that 3 of 5 evaluation domains have no specialist, so those domains show no improvement over the base model. Appendix K Training Dynamics This appendix documents the within-training behaviour of domain specialists, demonstrating the three properties that make post-hoc fusion work: (i) monotonic improvement on the specialist’s own domain, (i) monotonic degradation on out-of-domain data, and (i) growing fusion benefit as specialists diverge. Within-domain improvement and cross-domain degradation. Figure 10 shows the held-out evaluation loss for each specialist on each domain throughout training at Pythia-410M. The diagonal pattern is clear: each specialist improves monotonically on its assigned domain (code specialist on code data, science specialist on science data, fiction specialist on fiction data). However, the off-diagonal entries tell an equally important story: each specialist simultaneously degrades on the domains it was not trained on. By step 2,000, the code specialist evaluates at 2.908 on science data, worse than the base model’s 2.892; the science specialist evaluates at 3.061 on fiction, worse than base (2.974). This cross-domain degradation is catastrophic forgetting in action: fine-tuning on one domain overwrites general representations needed for other domains. This cross-domain degradation explains why routing must be learned (Section 4.5). A uniform router that assigns equal weight to all specialists averages these degraded cross-domain losses with no domain compensation, producing a result worse than the best individual specialist. A trained router recovers full-domain coverage by assigning near-zero weight to out-of-domain specialists for each input token. Growing fusion benefit. Figure 12 shows how the fusion benefit (MoE improvement over best individual specialist) evolves over specialist training steps. Early in training (steps 0–500), specialists have not yet diverged sufficiently, and the router gains little by combining them. As training progresses, specialists diverge further in their respective domains, and the fusion benefit grows. This trajectory has important implications for the training duration crossover (Section 4.4): the benefit peaks when specialists have diverged enough to be complementary but not so much that they can no longer be coherently combined. Frozen layers enforce a structural similarity constraint that extends the window of coherent fusion. Cross-domain evaluation at training checkpoint. Figure 11 presents the full cross-domain evaluation matrix at step 2,000. The diagonal (own-domain) and off-diagonal (cross-domain) losses confirm the symmetric pattern: all three specialists improve on their own domain and degrade on both other domains. The 6×3 matrix of (specialist, eval domain) pairs provides the quantitative basis for the routing strategy: a router that learns to assign each token to its domain-appropriate specialist will recover from the cross-domain losses by never sending a token to an out-of-domain specialist. Figure 10: Per-domain held-out evaluation loss for each specialist over training steps (Pythia-410M, freeze=4, seed=42). Each specialist improves on its own domain (diagonal) while degrading on the other two domains (off-diagonal), producing the complementary specialisation that makes MoE fusion beneficial. Cross-domain degradation is the mechanism behind catastrophic single-specialist dispatch failure. Figure 11: Cross-domain evaluation matrix at Pythia-410M step 2,000 (freeze=4, seed=42). Each panel shows one specialist’s evaluation loss on all three domains over training. Dashed horizontal lines mark the base model’s loss on each domain. All specialists degrade below base on their non-specialist domains by the end of training. Figure 12: Fusion benefit (MoE improvement over base model, %) as a function of specialist training steps at Pythia-410M. Benefit grows up to approximately 5,000 steps (+17.7%, freeze=0), then plateaus or degrades. The crossover between freeze=0 (optimal at ≤ 10,000 steps) and freeze=4 (better beyond 10,000 steps) is shown in Section 4.4. Per-domain equal-weight evaluation. Appendix L Downstream Benchmark Results Table 22: Downstream benchmark accuracy (%) at Pythia-1B (step10000 base, freeze=4, seed=42, 500 examples per benchmark). Random chance: HellaSwag 25%, ARC-Easy 25%, LAMBADA 0%, SciQ 25%, WinoGrande 50%. Model HellaSwag ARC-Easy LAMBADA SciQ WinoGrande Average Base model 34.4 40.4 60.2 68.4 49.6 50.6 Code specialist 34.2 39.4 57.4 65.8 50.0 49.4 Sci. specialist 34.2 41.0 56.4 65.8 48.2 49.1 Fict. specialist 34.4 39.8 58.6 66.8 48.2 49.6 Weight average 34.6 39.0 57.8 67.8 48.6 49.6 Monolithic 33.4 38.4 58.2 67.0 49.4 49.3 Kalavai MoE 35.0 40.0 59.0 64.8 49.4 49.6 Table 23: Downstream benchmark accuracy (%) at Pythia-6.9B (step10000 base, freeze=6, seed=42, 500 examples per benchmark). Due to compute constraints at 6.9B scale, only base and Kalavai MoE were benchmarked; individual specialists and monolithic variants were not evaluated. Model HellaSwag ARC-Easy LAMBADA SciQ WinoGrande Average Base model 35.4 43.6 61.2 66.8 51.0 51.6 Kalavai MoE 35.6 45.2 62.8 67.8 49.4 52.2 At 1B scale, the MoE model leads on HellaSwag (35.0% vs. 34.4% base), the benchmark most sensitive to language modelling quality. Monolithic training produces the worst average accuracy (49.3%), below even individual specialists (49.1–49.6%), suggesting mixed-domain gradient interference degrades general reasoning as well as language modelling. At 6.9B, the MoE leads on four of five benchmarks, with an average improvement of +0.56p over base. Downstream improvements are modest at these scales; we expect larger differentiation at 13B and above. Appendix M Qwen-1.5B Result Experiments with Qwen-1.5B at step 143,000 (full training, code and fiction domains, freeze=4, 2,000 steps, 3 seeds) produce a mean fusion improvement of +1.06% ± 0.01% vs. best individual specialist (per-domain equal-weight evaluation). Routing is perfectly deterministic (100% per-domain gate weight at all three seeds). The per-domain equal-weight protocol yields +1.06%; a mixed concatenated eval underrepresents fiction—the domain where Qwen’s MoE has its largest advantage—and would produce a misleading negative result. The gain of +1.06% is small, consistent with small specialist divergence (code 1.76%, fiction 4.56%, mean 3.16%). Applying the empirical conversion rate (0.34× for Qwen, Table 2), a 3.16% mean divergence predicts ≈ 1.1% fusion gain—exactly what is observed. Routing succeeds at all tested divergence levels. The Qwen result represents the low-divergence end of the gain relationship rather than a failure case. The Pythia-410M maturity sweep (step 143,000, ∼ 7% improvement) confirms this is consistent behaviour across model families. Appendix N Hybrid Routing Visualisation Table 24 shows token-level gate weights for five hybrid-domain prompts. The router switches experts mid-sequence on all five prompts, with 11 total switches across the prompt set (2.2 per prompt on average). Table 24: Token-level gate weights (softmax over 3 experts: code, science, fiction) for hybrid-domain prompts. Pythia-410M, freeze=4, seed=42. Dominant weight (>>0.5) shown in bold. “—” indicates transition token. Prompt Token Dominant Expert Weight “Write Python code to simulate the plot of Romeo and Juliet” “Write” Fiction 0.787 “Python” Fiction 0.821 “simulate” Fiction 0.929 “plot” Code 0.540 “Juliet” Fiction 1.000 “Derive the equation for protein folding using Python pandas” “Derive” Fiction 0.703 “protein” Science 0.959 “folding” Fiction 0.962 “Python” Code 0.585 “pandas” Code 0.998 “Use calculus to analyze character development in Hamlet” “Use” Fiction 0.852 “analyze” Science 0.920 “character” Science 0.794 “Ham” Fiction 1.000 The routing patterns confirm that the router operates at token granularity, not document level. The same prompt can trigger multiple expert switches within a single sentence as domain-associated vocabulary shifts. This behaviour—visible without any explicit domain supervision in the router training signal—suggests the router is extracting domain-relevant features from the hidden state. Figure 13: Gate weight heatmap for the prompt “Derive the equation for protein folding using Python pandas” (Pythia-410M, freeze=4, seed=42). Each column is a token; each row is an expert (code, science, fiction). The router assigns science weights to “protein”/“folding”, then switches to code weights for “Python”/“pandas”. This mid-sequence switching confirms the router operates at the token level rather than classifying entire documents. Appendix O Inference Benchmark We measured end-to-end inference latency, peak VRAM, and throughput for all Kalavai configurations at 410M and 1B scale on an NVIDIA GeForce RTX 5090, sequence length 512, 10 measured runs after 3 warmup runs. Table 25: Inference benchmark results. Latency and VRAM are per-forward-pass. “Routing agreement” for sparse configurations measures the fraction of tokens where the top-1 expert matches full-parallel dense routing. “—” indicates not applicable. All results: single seed 42. Scale Config Params (M) VRAM (GB) Latency (ms) Rel. Lat. Routing Agr. 410M Base model 405 1.15 17.9 1.00× — Single specialist 405 1.97 17.3 0.97× — Monolithic 405 1.97 17.4 0.97× — Kalavai dense (3×) 1216 3.46 51.2 2.86× — Kalavai sparse top-1 1216 2.67 16.8 0.94× 100% 1B Base model 1012 2.39 16.1 1.00× — Kalavai dense (3×) 3035 8.59 53.7 3.35× — Kalavai sparse top-1 3035 8.34 18.5 1.15× 10% Kalavai sparse top-2 3035 6.32 31.3 1.95× — The sparse top-1 configuration at 410M achieves 100% routing agreement but collapses evaluation quality (loss 3.106 vs. 2.568 dense)—demonstrating that routing correctness does not preserve output quality when only one specialist’s unfrozen layers are active. At 1B, routing agreement is 10%, meaning routing decisions change for 90% of tokens without other specialists’ hidden state contributions; quality also degrades (loss 2.412 vs. 2.382 dense). Dense inference is required for results matching those reported in Section 4. Appendix P Results Integrity Audit A systematic integrity audit was run across all committed result files using kalavai_results_audit.py. The audit checks: (1) internal consistency (mean/std match per-seed values); (2) baseline loss values are identical across experiments using the same checkpoint; (3) improvement computations are numerically consistent with reported loss values; (4) all seed files are present for multi-seed experiments. Outcome: 322/322 checks passed, 0 issues detected. Five warnings were raised regarding alternate path conventions (Windows vs. Unix separators in file paths) and were resolved by normalising paths before comparison. Appendix Q Phase 2 Detailed Results Q.1 Experiment 2: Private-Domain Fusion Table 26: Experiment 2 per-seed results. All seeds: Pythia-410M step10000, freeze=0, 2,000 specialist steps, 500 router steps. Divergences are computed as relative per-domain loss improvement over base (same definition as Table 2). Seed Med. div. Legal div. Patent div. Mean div. Gain vs spec Verdict 42 12.71% 34.16% 8.68% 18.52% +10.23% GO 137 12.71% 34.15% 8.67% 18.51% +10.27% GO 2026 12.71% 34.15% 8.68% 18.51% +10.00% GO Mean 12.71% 34.15% 8.68% 18.51% +10.17% GO Std — — — — ± 0.15p — Routing distributions (seed 42): medical 99.98% to medical specialist; legal 99.77% to legal; patent 97.53% to patent. Seeds 137/2026 show tighter routing (legal 100%, patent 98.75%/91.65%). The patent specialist receives slightly more off-expert weight (2.27–6.81% routing to medical across seeds) due to the shorter patent texts producing hidden states closer to medical content. Q.2 Experiment 1: Cross-Lingual Fusion Table 27: Experiment 1 per-seed results. Pythia-410M step10000, freeze=0, 2,000 specialist steps. Wikipedia fallback used for Tamil/Yoruba/Welsh (c100 uses legacy loading scripts blocked at datasets≥ 3.0; Wikipedia provides equivalent or better content). Seed Tamil div. Yoruba div. Welsh div. Mean div. Gain vs spec Verdict 42 23.28% 45.54% 33.69% 25.74% +6.14% PIVOT (router collapse) 137 23.26% 45.52% 33.19% 25.60% +21.76% GO 2026 23.35% 45.50% 33.21% 25.62% +21.75% GO Mean (all) — — — 25.65% +16.55% GO Clean seeds — — — 25.61% +21.76% GO (± 0.005p) Code domain divergence is negligible (0.43–0.44%) because CodeSearchNet Python is already well-represented in the Pythia pre-training corpus. Code routing remains correct (96.45–98.63% to code specialist) despite the low divergence. Appendix R Evaluation Correction Methodology During development, an initial evaluation protocol produced +14.2% at Pythia-410M. Code review identified two inconsistencies; the corrected protocol yields +7.72%. This appendix documents the bugs and the fix for reproducibility. Bug A: Asymmetric batch sizes. The original evaluation used batch size 2 for the MoE model and batch size 4 for all baselines (specialist and base). PackedChunkDataset packing means different batch sizes evaluate different token subsequences. Since the MoE was evaluated on different data than its baselines, the comparison was not valid. Fix: all models evaluated at batch size 4 (bs=4 across all conditions). Bug B: Concatenated mixed evaluation. The original evaluation concatenated code, science, and fiction chunks into a single mixed dataset and computed one aggregate loss. Due to chunk ordering, fiction chunks were systematically under-represented in the MoE evaluation pass. Since the MoE had its largest advantage on fiction (the domain with highest specialist divergence, 25.4%), the mixed-batch eval underweighted the domain where MoE gained most vs. the domain where it gained least. Fix: evaluate each domain separately at consistent batch size, then compute equal-weight average: 13(ℒcode+ℒsci+ℒfiction) 13(L_code+L_sci+L_fiction). Additional 6.9B fix. The 6.9B result was stabilised by seeded shuffling of the evaluation dataset (original: mean +2.72%, std ± 8.17% across seeds; corrected: +6.53% ± 0.024% over 3 seeds, computed from stored per-domain losses without re-running specialists). The high variance in the original 6.9B result was caused by non-deterministic chunk ordering producing different effective evaluation sets per seed. Corrected infrastructure. The corrected evaluation protocol is implemented in experiments/kalavai_eval_utils.py (eval_all_domains, eval_loss_domain). All Phase 2 experiments import this module rather than implementing inline evaluation, preventing recurrence. The corrected evaluation was run on all committed result files; original result files are preserved in experiments/results/ with suffixes indicating evaluation method.