Paper deep dive
Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning
Dante Lok
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/17/2026, 7:33:32 AM
Summary
The paper introduces gate-zero growth, a function-preserving operator for continual learning that expands model capacity by adding residual blocks with zero-initialized gates. This approach induces rank separation in the functional Jacobian under a transversality condition, ensuring old functional directions remain unchanged while new directions are flat at initialization. The framework theoretically unifies methods like LoRA, ReZero, and zero-init adapters under a shared geometric structure. Empirical results on Transformers adapted from WikiText-103 to BookCorpus demonstrate near-zero catastrophic forgetting under Isolation and Freeze-Nothing protocols, significantly outperforming non-function-preserving controls like G_stack.
Entities (18)
Relation Signals (13)
Gate-zero growth → evaluatedon → BookCorpus
confidence 95% · On a 300M→857M Transformer adapted from WikiText-103 to BookCorpus
Gate-zero growth → evaluatedon → WikiText-103
confidence 95% · On a 300M→857M Transformer adapted from WikiText-103 to BookCorpus
Gate-zero growth → implements → Function-preserving operator
confidence 95% · We introduce gate-zero growth, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate.
Gate-zero growth → mitigates → Catastrophic forgetting
confidence 95% · gate-zero growth reaches near-zero old-domain forgetting... while a non-FP control suffers an order-of-magnitude larger forgetting
G_stack → servesas → Non-FP control
confidence 95% · a non-FP control (G_stack) suffers an order-of-magnitude larger forgetting under the same recipe
Beever AI → affiliatedwith → Dante Lok
confidence 90% · Dante Lok Votee AI Beever AI dante.lok@votee.ai
Votee AI → affiliatedwith → Dante Lok
confidence 90% · Dante Lok Votee AI Beever AI dante.lok@votee.ai
Gate-zero growth → induces → Rank separation
confidence 90% · Under a transversality condition, gate-zero growth induces rank separation in the functional Jacobian
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank separation} in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is $O(\|\boldsymbol{\alpha}\|^2)$ and Jacobian leakage $O(\|\boldsymbol{\alpha}\|_\infty)$, giving a controlled departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting ($\Delta_A < 0.1$) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control ($G_{\text{stack}}$) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
Tags
Links
- Source: https://arxiv.org/abs/2607.14571v1
- Canonical: https://arxiv.org/abs/2607.14571v1
Trouble viewing inline? Open PDF directly →
Full Text
95,461 characters extracted from source content.
Expand or collapse full text
Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning Dante Lok Votee AI Beever AI dante.lok@votee.ai dante.lok@beever.ai Abstract We introduce gate-zero growth, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces rank separation in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is O(‖2)O(\| α\|^2) and Jacobian leakage O(‖∞)O(\| α\|_∞), giving a controlled departure from the FP locus. On a 300M→857M300M→ 857M Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting (ΔA<0.1 _A<0.1) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control (GstackG_stack) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL. 1 Introduction The dominant paradigm for obtaining a more capable language model is to train a larger model from scratch. Model growth — expanding an existing trained model by adding parameters — offers a compelling alternative, provided the grown model can be fine-tuned without forgetting what the smaller model knew. Function-preserving (FP) growth methods (Chen et al., 2016, 2022) satisfy preservation at growth time by construction, but the geometric structure this initialisation creates in parameter space, and why certain continual learning (CL) strategies work better than others on the grown model, has remained unclear. A wide range of seemingly different methods share a single structural property: gate-zero residual growth, ReZero-style residual scaling (Bachlechner et al., 2021), LoRA (Hu et al., 2022) with B=0B=0 at initialisation, and near-identity / zero-init adapter modules (Houlsby et al., 2019) all add new parameters through an additive branch whose contribution factors through a zero- or near-zero-initialised gate. Each of these methods has its own justification and its own preferred CL recipe; their continual-learning behaviour is typically explained operationally rather than through the geometry of the underlying construction. The recurring empirical pattern — that freezing old parameters and training only the new ones recovers preservation at growth time — has not been tied to a formal property that the constructions share. We argue that the relevant property is geometric. At a zero-initialised gate, the new-weight contribution to the functional Jacobian is exactly zero by the chain rule, with structural consequences for CL geometry that the literature has not derived as a unified system. Under a transversality condition, the post-growth Jacobian decomposes cleanly into the unmodified old block plus up to K new-gate directions; the Fisher information matrix has a sparse new-weight block; coordinate isolation becomes an exact projection onto a known local subspace; and as gates open during CL, the departure from this structure is locally controlled by polynomial bounds in the gate magnitude. We instantiate this geometry as gate-zero growth, a function-preserving depth operator for continual learning; the same analysis applies to LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of the shared template. We instantiate the framework with gate-zero Transformer depth growth and evaluate sequential adaptation from WikiText-103 (Merity et al., 2017) to BookCorpus (Zhu et al., 2015) at 300M→857M300M→ 857M scale. Under the Isolation protocol, gate-zero growth reaches near-zero old-domain forgetting; zero-init residual stacking under the same protocol reaches the same regime (ΔA<0.1 _A<0.1 for both), directly validating the framework’s unification claim. A non-FP baseline (GstackG_stack, used as a diagnostic negative control) suffers order-of-magnitude larger forgetting under the same recipe — the rank-separation guarantee fails without zero-init gating. Ablations also reveal that the framework predicts the safest projection point, while a slightly looser point in the same family (soft preservation without weight freezing) strictly dominates full Isolation on the joint (PPLA,PPLB)(PPL_A,PPL_B) frontier — a controlled-departure regime within the local theory. Contributions. 1. Gate-zero growth. We propose gate-zero growth, an FP operator that adds residual blocks through a zero-initialised gate, achieving exact function preservation at growth and ΔA<0.1 _A<0.1 under continual learning at 300M→857M300M→ 857M scale. We recommend two operating points: Isolation (exact preservation, ΔA=+0.04 _A=+0.04) and Freeze-Nothing (joint (PPLA,PPLB)(PPL_A,PPL_B) optimum, ΔA=−1.98 _A=-1.98). 2. Geometric framework. Under a transversality condition, gate-zero growth induces rank separation, sparse-Fisher block structure (Theorem 3.1, Prop. 3.5), exact coordinate-projection isolation (Prop. 3.6), and bounded gate-driven departure (Props. A.2, A.3). The same analysis covers LoRA, ReZero, and zero-init adapters (Remark 3.4), establishing gate-zero growth as the canonical instance. 3. Predictive structural validation. Three falsifiable predictions verified empirically: (a) coordinate freezing is exactly preserving only under FP growth (Gstack+G_stack+Iso is structurally undefined); (b) LoRA-CL admits no Isolation row because its frozen-base construction is structurally already Isolation; (c) ΔA<0.1 _A<0.1 for every zero-init FP operator we test under Isolation. The MoE plasticity gap is mechanistically localised to clone-block redundancy. 2 Related Work Neural network growth. Net2Net (Chen et al., 2016), bert2BERT (Chen et al., 2022), and LiGO (Wang et al., 2023a) introduce FP / near-FP expansions but provide no functional-Jacobian / rank analysis or CL framing. ReZero-style residual scaling (Bachlechner et al., 2021), near-identity / zero-init adapters (Houlsby et al., 2019), and zero-initialised residual stacking achieve FP (or near-FP) at growth via additive branches with zero- or near-zero-initialised contribution. We propose gate-zero growth for FP depth expansion in CL, and provide the unified rank-separation / functional-Jacobian / sparse-Fisher analysis (Theorem 3.1, Props. 3.5, 3.7; Remark 3.4) that covers gate-zero growth alongside LoRA, ReZero, and zero-init adapter constructions. GstackG_stack (Du et al., 2024) serves as a non-FP negative control: it duplicates blocks without zero-init gating, so the rank-separation guarantee fails by construction. Parameter-efficient CL. LoRA (Hu et al., 2022) and adapter-based approaches (Houlsby et al., 2019; Rusu et al., 2016; Wang et al., 2023b) with zero-initialised branches satisfy the same zero-init structural property as gate-zero growth (cf. Remark 3.4); recent continual-LoRA / continual-adapter methods provide alternative low-trainable-budget CL recipes orthogonal to the growth-vs-no-growth axis we study. Continual learning. Catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999) has motivated a long line of regularization-, replay-, and architecture-based methods (De Lange et al., 2022): EWC (Kirkpatrick et al., 2017) and its successors penalize parameter drift via the Fisher information; PackNet (Mallya and Lazebnik, 2018) uses hard binary masks; Progressive Networks (Rusu et al., 2016) allocate disjoint columns per task; LwF (Li and Hoiem, 2018) uses output-space distillation; experience replay (Rolnick et al., 2019) revisits old data; A-GEM and related gradient-projection methods (Chaudhry et al., 2019; Farajtabar et al., 2020) project task gradients onto the orthogonal complement of past-task gradients; and parameter-efficient CL via LoRA-style adapters (Wang et al., 2023b) updates a low-rank subspace per task. We unify and rank these methods geometrically and focus on the structural geometry of the growth operator rather than the regularizer-design space; direct head-to-head comparison with gradient-projection and adapter-based CL on multi-task sequences is left as future work. Loss landscape geometry. Fisher information geometry (Amari, 1998) provides the Riemannian foundation for our Fisher-based analysis of preservation constraints (Section 3). Functional-Jacobian analyses in the linearised neural-tangent regime (Jacot et al., 2018) are adjacent in motivation; we work directly in parameter space at the growth point rather than in the infinite-width NTK limit. 3 Theoretical Framework We work with a residual architecture in which each block contributes αℓ⋅Blockℓ(x) _ ·Block_ (x) to the residual stream, gated by a scalar αℓ _ . Growth inserts K new blocks with gates initialized to α0 _0 (typically zero); the existing r “old” blocks are unmodified at growth time. Setup. Let fθold:ℝd→ℝVf_ _old:R^d ^V denote the pre-growth function and fθnewf_ _new the post-growth function with K inserted gated blocks. With α0=0 _0=0, by direct computation fθnew(x)=fθold(x)f_ _new(x)=f_ _old(x) for all x — the new blocks are identity through residual addition. Let J(θ)=∇θfθ(x)J(θ)= _θf_θ(x) denote the functional Jacobian. FP locus. Let ℳ(f∗)=θ′:fθ′=f∗M(f^*)=\θ :f_θ =f^*\ with f∗=fθoldf^*=f_ _old. The gate-zero growth point θ′=ι(θold)θ = ( _old) lies on ℳ(f∗)M(f^*) with ℝK⋅PW⊆Tθ′ℳ(f∗)R^K· P_W T_θ M(f^*) (Theorem 3.1 (1)); motion along Tnew⟂T _new leaves ℳ(f∗)M(f^*) at first order (Prop. A.2). Theorem 3.1 (Rank separation under transversality). Let Jα∈ℝD×KJ_α ^D× K (with D=dimfD= f the flattened output dimension) collect the K new-gate Jacobian columns, ∂f/∂αℓ=Blockℓ(x;Wℓ)∂ f/∂ _ =Block_ (x;W_ ), and let PoldP_old denote the orthogonal projection onto im(J(θold))im(J( _old)). Then at the gate-zero growth point θ′θ : 1. (Exact flat directions, unconditional.) The K⋅PWK· P_W new-weight directions are exactly flat: ∂f/∂Wℓ′=0∂ f/∂ W _ =0 for all ℓ and all x. 2. (Conditional rank additivity.) The Jacobian rank satisfies rank(J(θ′))=r+rank((I−Pold)Jα)≤r+K,rank(J(θ ))\;=\;r\;+\;rank\! ((I-P_old)\,J_α )\;≤\;r+K, with equality r+Kr+K iff the projected gate columns (I−Pold)Jα(I-P_old)\,J_α are linearly independent (transversality). Generic non-degeneracy of the cloned new-block functions BlockℓBlock_ implies transversality almost surely under small clone noise; degenerate cases (e.g. a block whose output already lies in im(J(θold))im(J( _old))) violate it. By contrast, GstackG_stack does not preserve J(θold)J( _old) at the growth point because duplicated blocks immediately alter the active old-block computation, so the equivalent rank decomposition is unavailable. Proof in Appendix A. Load-bearing content: Part (2)’s conditional rank additivity, the unification across zero-init constructions (Remark 3.4), and the direct transversality diagnostic (σmin⟂=144.6 _ =144.6, smallest principal angle 56.9∘56.9 ; Appendix B). Remark 3.2. The transversality assumption is generic but not vacuous. Theorem 3.1 should be read as a structural statement: gate-zero growth is the enabling construction for clean rank decomposition, but empirical realisation depends on the cloned-block functions BlockℓBlock_ being non-degenerate. Function preservation itself is verified directly by max-logit checks (Section 5.1: max logit difference 0.00.0 at α0=0 _0=0 on real data). We also directly verify transversality by computing the projected-rank residual (I−Pold)Jα(I-P_old)J_α at the post-growth Gate-FP checkpoint (K=36K=36, growth factor g=4g=4) with M=200M=200 random old-direction Jacobian samples (full diagnostic in Appendix B). The residual is full-rank (σmin⟂=144.6 _ =144.6, condition number 6.876.87) and the smallest principal angle between im(Jα)im(J_α) and the sampled im(Jold)im(J_old) is 56.9∘56.9 (largest: 83.7∘83.7 ). Transversality is therefore not merely “generic almost surely” but quantitatively well-separated at the trained checkpoint. A formal genericity argument (the set of noise realisations producing dependence is Lebesgue-zero in ℝK⋅PWR^K· P_W) is given in Appendix A, Remark A.1. Remark 3.3 (Why rank separation matters). Rank separation is what distinguishes exact FP from “good initialisation.” Any smooth initialisation can produce a model close to f∗f^* in function space; gate-zero growth (and structurally analogous zero-init constructions; see Remark 3.4) additionally guarantees that the functional Jacobian exactly decomposes into an unchanged old block and an additive new block. This structural guarantee is what makes isolation-based CL exact at the growth point rather than approximate. Remark 3.4 (Scope beyond gated residual blocks). Theorem 3.1 requires two structural properties: (a) the old forward pass is preserved exactly at θ′θ , and (b) every new contribution to f factors through a zero-initialised parameter producing an additive scalar-times-feature term. Any growth operator satisfying both inherits the conclusion. LoRA with ΔW=BA W=BA, B=0B=0: B plays the gate, A the cloned feature, and transversality reduces to projected A-induced directions being independent of im(Jold)im(J_old), which holds generically. Net2Net / zero-init residual stacking zeroes new-block output projections instead of using a scalar gate; the analysis is identical. ReZero (Bachlechner et al., 2021) satisfies both properties by construction; adapter modules (Houlsby et al., 2019) satisfy them exactly under zero-init up-projection variants and approximately under the near-identity init of the original. GstackG_stack (Du et al., 2024) satisfies neither. Proposition 3.5 (Sparse Fisher block structure). Let F(θ′)=x∼[J(θ′;x)⊤J(θ′;x)]F(θ )=E_x \! [J(θ ;x) J(θ ;x) ] be the empirical Fisher information matrix at the gate-zero growth point. Then unconditionally: 1. The new-weight Fisher block FW′,W′F_W ,W is identically zero; 2. All cross-terms Fθold,W′F_ _old,W and Fα,W′F_α,W involving new weights are identically zero. The old–gate cross-block Fθold,αF_ _old,α is generically non-zero, so F(θ′)F(θ ) is not block-diagonal between old and new parameters. Proof. Both claims follow from Theorem 3.1 (1): ∂f/∂Wℓ′=0∂ f/∂ W _ =0 identically at θ′θ , so any inner product involving a new-weight Jacobian column with any other column is zero. Old–gate cross-terms involve ⟨Jθold,j,Jα,k⟩ J_ _old,j,\,J_α,k , neither of which vanishes generically. ∎ Proposition 3.5 has direct CL implications: second-order methods like EWC place no penalty on new-weight movement (the diagonal Fisher entries on W′W are zero), so EWC reduces to gate-only regularisation. Isolation is therefore a coordinate hard constraint, not a consequence of full Fisher orthogonality, as the next proposition formalises. Proposition 3.6 (Isolation as a coordinate-subspace projection). Freezing all old parameters during CL on BD_B gives an exact coordinate projection onto the new-parameter subspace Tnew=0Pold×ℝPnewT_new=\0\^P_old×R^P_new. At the gate-zero growth point, TnewT_new decomposes orthogonally into the K⋅PWK· P_W exactly-flat new-weight directions (Tnew∥T_new ) and the K gate directions (Tnew⟂T_new ); the former preserve f for any update magnitude, while the latter activate new function change as soon as gates leave zero. Isolation therefore preserves old parameters exactly, but preservation of the old function during CL still depends on the KL/replay objective once gates open. Under GstackG_stack, the new- and old-coordinate subspaces are not aligned with the FP locus at θ′θ to begin with, so the same coordinate freeze is only an approximate preservation projection. Proposition 3.7 (Four-way subspace partition). At θ′=ι(θold)θ = ( _old) under gate-zero growth, the ambient parameter space partitions orthogonally (Euclidean coordinate metric) into four subspaces: Told∥T_old (old-direction tangents to ℳ(f∗)M(f^*), dimension nul(J(θold))nul(J( _old))), Told⟂T_old (old-direction normals, dimension r), Tnew∥T_new (exactly flat new-weight directions, dimension K⋅PWK· P_W, contained in kerJ(θ′) J(θ )), and Tnew⟂T_new (non-flat new-gate directions, dimension K). Isolation restricts updates to Tnew∥⊕Tnew⟂T_new T_new . Under transversality, Jα(Tnew⟂)∩im(J(θold))=0J_α(T_new ) (J( _old))=\0\, with the smallest principal angle bounded below by the transversality singular value σmin⟂:=σmin((I−Pold)Jα)>0 _ := _ ((I-P_old)J_α)>0. The new functional capacity per unit gate motion is bounded between σmin⟂ _ and σmax⟂ _ , so σmin⟂ _ quantifies plasticity and small σmin⟂ _ is the precise failure mode of transversality. Under non-FP growth (GstackG_stack), Tnew∥T_new does not exist (Theorem 3.1 (1) fails) and coordinate freezing is only approximately preserving. Full proof in Appendix A. Local geometry beyond the growth point. At non-zero α, function drift is O(‖2)O(\| α\|^2) and Jacobian leakage O(‖∞)O(\| α\|_∞) (Propositions A.2, A.3). Empirically, leak=‖PoldHPnew‖F/‖H‖FS_leak=\|P_oldHP_new\|_F/\|H\|_F at three checkpoints (Tab. 16) is 0.0050.005 at post-growth Gate-FP vs. 0.1610.161 at GstackG_stack (32×32×), and 0.0720.072 post-CL Gate-FP++Iso (‖∞≈0.08\| α\|_∞≈ 0.08) — consistent with the linear-leakage prediction. CL hierarchy. The five CL strategies we evaluate impose progressively weaker constraints on old-parameter drift: Isolation (hard freeze), Hybrid (Isolation + replay CE on AD_A), Distillation (Li and Hoiem, 2018) (output-space KL on BD_B inputs), Replay (CE on AD_A samples), No protection (none). The ordering is by constraint tightness, not necessarily downstream performance — a distinction we return to in Section 5.5. We choose these five to span the constraint-tightness axis end-to-end. EWC (Kirkpatrick et al., 2017) is not run as a separate row: Proposition 3.5 implies that at θ′θ diagonal EWC contributes zero on new-weight directions and reduces under Isolation to a soft L2 cap on new-gate drift, qualitatively matching the gate-init mechanism in Ablation 4 (the α0=0.1 _0=0.1 row, ΔA=−0.444 _A=-0.444, bounds what diagonal EWC could achieve in this protocol). Gradient-projection (A-GEM (Chaudhry et al., 2019), OGD (Farajtabar et al., 2020)) and masking (PackNet (Mallya and Lazebnik, 2018)) methods are deferred to future work. A claim-status breakdown (exact / conditional / approximate / empirical) is in Appendix A.6. 4 Method Gate-zero growth. For depth growth from L to gLgL layers, we insert K=(g−1)LK=(g-1)L new blocks initialised by cloning existing blocks (with small noise σ) and setting their block gates to α0∈0,ϵ _0∈\0,ε\. For width / expert growth in MoE, new experts are added with expert-gate =0=0 to ensure they are never selected by top-k routing at growth time. Both produce exact FP at α0=0 _0=0 (Theorem 3.1). CL losses. With ℒACE,ℒBCEL_A^CE,L_B^CE the CE losses on the two datasets and ℒpres(θ)=T2⋅KL(pθold∗∥pθ,xA)L_pres(θ)=T^2·KL(p_ _old^*\,\|\,p_θ,x_A) a teacher-preservation KL on replayed AD_A samples, the methods we evaluate are ℒNoProt _ NoProt =ℒBCE =L_B^CE ℒReplay _ Replay =(1−ρ)ℒBCE+ρℒACE =(1-ρ)L_B^CE+ _A^CE ℒDistill _ Distill =ℒBCE+μT2KL(pθold∗∥pθ,xB) =L_B^CE+μ T^2KL(p_ _old^*\|p_θ,x_B) ℒIso _ Iso =ℒBCE+λℒpres =L_B^CE+ _pres ℒHybrid _ Hybrid =(1−ρ)ℒBCE+ρℒACE+λℒpres =(1-ρ)L_B^CE+ _A^CE+ _pres Isolation additionally freezes old parameters, restricting updates to TnewT_new. 5 Experiments Setup. A 300M base Transformer is trained for 10 epochs on WikiText-103 (AD_A), grown to 857M (12→ 48 layers) by Gate-FP or GstackG_stack, then fine-tuned for 10 CL epochs on BookCorpus (BD_B). For Mixture-of-Experts (Section 5.3) the base is 706M MoE (12 layers, 4 experts, top-k=2k=2) grown to 2.5B (24 layers, 8 experts). All runs use 1× NVIDIA L20 (48 GB), fp16, gradient accumulation to effective batch size 128. Total compute ∼2,500 2,500 GPU-hours. 5.1 Function preservation at growth Before any CL training, gate-zero growth is bit-exact on dense Transformers and within MoE numerical tolerance, while GstackG_stack already inflates PPLAPPL_A by 2.8×2.8× (Table 1). Table 1: Growth-induced perplexity change, before any CL training. Gate-FP is exact (or within MoE numerical tolerance); GstackG_stack degrades PPLAPPL_A by 2.8×2.8× and PPLBPPL_B by 2.3×2.3× at the moment of growth. Metric Gate FP (Dense) stack G_stack Gate FP (MoE) Params (pre → post) 252.9M → 857.1M 252.9M → 857.1M 705.8M → 2568.3M ΔPPLA _A +0.00+0.00 +46.11+46.11 +0.00+0.00 ΔPPLB _B +0.00+0.00 +708.86+708.86 +0.00+0.00 FP check max logit diff 0 (exact) — (non-FP) 2.9×10−52.9× 10^-5 5.2 Continual learning matrix Scope of comparison. GstackG_stack (Du et al., 2024) is included as a diagnostic negative control, not as a competitive CL baseline: it was designed for pre-training acceleration, is not function-preserving by construction (Table 1), and damages PPLAPPL_A at the moment of growth before CL begins. The matrix below therefore tests the framework’s structural prediction — that coordinate freezing under non-FP growth has no rank-separation guarantee — rather than claiming that gate-zero is the best growth operator among FP-style constructions. Comparison to alternative FP operators (zero-init residual stacking, Net2Net (Chen et al., 2016), bert2BERT (Chen et al., 2022), LiGO (Wang et al., 2023a)) and to frozen-backbone adapter / LoRA-CL approaches (Rusu et al., 2016; Wang et al., 2023b) at g=2g=2 scale is reported in Section 5.4; full-scale comparison is left as follow-up. Table 2 reports the 2×52× 5 matrix. ΔA _A is computed relative to each growth method’s own post-growth baseline (italic “Pre-CL” rows). Table 2: Continual learning matrix (10 base epochs on AD_A + 10 CL epochs on BD_B). Lower is better. PPLAPPL_A and PPLBPPL_B are validation perplexities on each dataset; ΔA _A is the post-CL minus pre-CL change in PPLAPPL_A relative to each growth method’s own post-growth baseline (italic “Pre-CL (post-growth)” rows). “Scratch” trains the 857M model jointly on A∪BD_A _B for 10 epochs as a same-compute reference; final-epoch state shown for protocol parity. † isolation (freezing duplicated blocks) is omitted for GstackG_stack because the rank-separation guarantee of Theorem 3.1 does not hold: GstackG_stack’s duplicated blocks immediately alter the computation, so freezing them does not preserve f∗f^*. The Hybrid (Replay+Preserve) row uses the same loss on both growth methods; on GstackG_stack the load-bearing preservation work comes from the KL and replay-CE terms rather than from the (here non-meaningful) freeze. Growth CL Strategy PPLA↓PPL_A PPLB↓PPL_B ΔA↓ _A Gate FP Pre-CL (post-growth) 25.92 560.75 — No protection 366.13 20.51 +340.21+340.21 Replay 493.46 20.30 +467.54+467.54 Distillation 39.48 25.05 +13.56+13.56 Isolation 25.96 28.41 +0.04+0.04 Hybrid (Iso+Replay) 37.08 29.97 +11.16+11.16 GstackG_stack Pre-CL (post-growth) 72.03 1269.61 — No protection 1215.98 31.67 +1143.95+1143.95 Replay 2179.25 20.02 +2107.22+2107.22 Distillation 48.33 25.42 −23.70-23.70 Isolation† N/A N/A N/A Hybrid (Replay+Preserve) 35.14 29.77 −36.89-36.89 Scratch (joint A+B) — 43.23 41.81 — Findings. Gate-FP + Isolation achieves ΔA=+0.04 _A=+0.04 (preservation within evaluation noise) while reducing PPLBPPL_B from 560.75560.75 to 28.4128.41; no other Gate-FP CL configuration matches this preservation, and naive baselines catastrophically forget. By contrast, GstackG_stack degrades PPLAPPL_A from 25.9225.92 to 72.0372.03 at growth time alone (Table 1), and under naive fine-tuning drives it past 12001200. The gap between the best Gate-FP row (25.9625.96) and worst GstackG_stack row (2179.252179.25) is roughly 85×85× on PPLAPPL_A. We caution that this gap conflates two effects: the structural cost of non-FP growth, and the fact that GstackG_stack was not designed for CL; comparison with function-preserving baselines is left as future work (Section 6.1). Comparison to scratch. Scratch is included only as a protocol reference, not as a competitive baseline. Its final-state checkpoint reaches 43.23/41.8143.23/41.81, but its best-validation checkpoint (epoch 3) was 17.40/16.1917.40/16.19 — substantially better than the final overfit state. Conclusions should therefore not be drawn from the final-state Scratch comparison; we report it for token-budget parity with the CL runs (10 epochs over A∪BD_A _B) and explicitly do not claim that Gate-FP ++ Isolation outperforms a properly-stopped Scratch model. 5.3 MoE cross-architecture validation Table 3: MoE cross-architecture validation (4→ 8 experts, 12→ 24 layers). Dense rows reproduce the corresponding Gate-FP entries from Table 2 for comparison under identical CL hyperparameters. Architecture CL Strategy PPLA↓PPL_A PPLB↓PPL_B ΔA↓ _A MoE Pre-CL (post-growth) 67.21 2155.77 — No protection 610.88 36.89 +543.67+543.67 Isolation 67.41 182.06 +0.20+0.20 Hybrid (Iso+Replay) 71.41 187.03 +4.20+4.20 Dense Pre-CL (post-growth) 25.92 560.75 — No protection 366.13 20.51 +340.21+340.21 Isolation 25.96 28.41 +0.04+0.04 Hybrid (Iso+Replay) 37.08 29.97 +11.16+11.16 Preservation transfers; plasticity does not. Under both dense and MoE, Gate-FP + Isolation yields ΔA≈0 _A≈ 0 (+0.04+0.04 dense, +0.20+0.20 MoE), confirming the architecture-agnostic preservation mechanism. But under identical CL hyperparameters, MoE plasticity is an order of magnitude weaker: dense reduces PPLBPPL_B from 560.75560.75 to 28.4128.41 (20×20×), while MoE only achieves 2155.77→182.062155.77→ 182.06 (12×12×, with absolute PPLBPPL_B remaining much higher). Per-checkpoint diagnostic. Comparing the post-growth (pre-CL) and post-CL MoE-isolation checkpoints localizes the failure mode (Table 4). Three patterns emerge: (i) all 12 new block gates converge to the safety-clamp ceiling α=0.083α=0.083 (∑αℓ≈1.0Σ _ ≈ 1.0, gradient sought higher gates); (i) routing concentration is unchanged from pre-CL (top-11 share 0.500.50, normalized entropy 0.3330.333), ruling out within-CL router collapse; (i) per-expert cosine similarity to source experts drifts from 0.9180.918 to 0.8100.810 on average, with the most-differentiated expert still at 0.7120.712. New blocks open their gates to the ceiling but cannot differentiate enough from the frozen sources to provide complementary capacity for BD_B — clone-block redundancy is the bottleneck. Table 4: MoE isolation per-checkpoint diagnostic (new MoE blocks only). “Top-11 share” is the fraction of tokens routed to the single most-selected expert (uniform = 1/N=0.1251/N=0.125 for N=8N=8); “entropy / logN N” is normalised routing entropy (1.01.0 = uniform). Routing concentration is unchanged across CL; per-expert similarity to source drifts only modestly. Diagnostic metric Pre-CL Post-CL Max |αℓ|| _ | on new blocks 0.0000.000 0.0830.083 Cumulative ∑ℓ|αℓ| _ | _ | (nnew=12n_new=12) 0.000.00 0.990.99 Mean top-11 expert share 0.5000.500 0.5000.500 Routing entropy / logN N (mean) 0.3330.333 0.3330.333 Mean cosine sim to source expert 0.9180.918 0.8100.810 Min cosine sim to source expert 0.8520.852 0.7120.712 The full per-epoch trajectory and a stacked-panel visualization of the same train-validation overfitting signature are deferred to Appendix F (Figure 1). Manifold geometry. The discriminating tests of rank separation are the FP check (Δ=0 =0) and the projected-rank diagnostic (σmin⟂=144.6 _ =144.6, min angle 56.9∘56.9 , App. B); gradient-covariance and Hessian diagnostics are deferred to App. G. 5.4 No-growth, alternative-FP, and PEFT baselines We compare three baseline families at g=2g=2 scale — no-growth, zero-init residual stacking (alternative FP operator), and LoRA-CL (PEFT) — in Table 5 (full rows in Appendix C). Table 5: Baseline comparisons at g=2g=2 scale (preservation-aware recipes only; full table including no-protection rows in Appendix C). No growth fine-tunes the 300M base directly (Isolation is vacuous: no new parameters). Zero-init stacking zeroes new-block output projections instead of using a scalar gate. LoRA-CL uses rank-6464 adapters with the base frozen. Family CL Strategy PPLA↓PPL_A PPLB↓PPL_B ΔA↓ _A No-growth (300M) Distillation 39.00 25.06 +13.08+13.08 Hybrid 37.83 21.29 +11.91+11.91 Zero-init stacking (g=2g=2) Distillation 37.74 25.14 +11.82+11.82 Isolation 25.96 32.81 +0.04+0.04 LoRA-CL (rank 6464) Distillation 32.37 30.33 +6.45+6.45 Hybrid 28.60 29.56 +2.68+2.68 Gate-FP (g=2g=2, ref.) Isolation 26.01 30.45 +0.09+0.09 Findings. (i) Both Gate-FP and zero-init stacking ++ Isolation reach the near-zero-forgetting regime at g=2g=2 (ΔA=+0.09 _A=+0.09 vs. +0.04+0.04, both far below the +13.56+13.56 Distillation gap), confirming that rank separation is structural, not gate-specific. At equivalent preservation, Gate-FP achieves better plasticity (PPLB=30.45PPL_B=30.45 vs. 32.8132.81, −7%-7\%) — the gate-zero parameterisation yields the strongest preservation/plasticity trade-off among zero-init FP operators tested. (i) Under soft-KL recipes, growth’s advantage is small (no-growth ++ Distillation +13.08+13.08 tracks Gate-FP ++ Distillation +13.56+13.56); the FP advantage concentrates in the Isolation regime. (i) LoRA-CL is a strong alternative under Hybrid (ΔA=+2.68 _A=+2.68); LoRA admits no separate Isolation row because its frozen-base construction is structurally already Isolation under the four-way partition (Prop. 3.7) — a prediction of the unification, not a gap in it. Among the FP and PEFT baselines tested, gate-zero growth ++ Isolation gives the strongest preservation (ΔA=+0.04 _A=+0.04 at g=4g=4). Multi-seed validation at g=2g=2 across three seeds yields ΔA=+0.0896±0.0046 _A=+0.0896± 0.0046 (Appendix D). 5.5 Ablations We run five ablations on Gate-FP holding other hyperparameters fixed: growth factor g∈2,3,4g∈\2,3,4\ (Abl. 1); the four combinations of weight/gate freezing under fixed isolation loss (Abl. 2); replay fraction ρ∈0,0.5ρ∈\0,0.5\ in Hybrid (Abl. 3); gate-init/warmup pair (α0,ϵ)( _0,ε) (Abl. 4); growth timing as % of base-train completed before growth (Abl. 5). Numbers in Table 6; extended discussion in Appendix H. Table 6: Key ablations on Gate-FP. What to freeze pinpoints old-weight freezing (not gate freezing) as the binding constraint. Replay fraction confirms that adding CE-on-AD_A on top of isolation worsens both axes. The negative ΔA _A at α0=0.1 _0=0.1 reflects partial CL recovery from a non-FP starting point (post-growth PPLA=26.40PPL_A=26.40 vs. 25.9225.92 at α0=0 _0=0), not improvement over f∗f^*. Full tables in Appendix H. Configuration PPLA↓PPL_A PPLB↓PPL_B ΔA↓ _A Ablation 1: Growth factor g, isolation g=2g=2 (454M) 26.01 30.45 +0.09+0.09 g=3g=3 (656M) 25.98 29.25 +0.06+0.06 g=4g=4 (857M) 25.96 28.41 +0.04+0.04 Ablation 2: What to freeze, fixed isolation loss Freeze nothing 23.93 20.37 −1.98-1.98 Freeze old gates only 23.91 20.36 −2.01-2.01 Freeze old weights only 25.06 26.99 −0.86-0.86 Freeze both (full isolation) 25.96 28.41 +0.04+0.04 Ablation 3: Replay fraction ρ in Hybrid ρ=0ρ=0 (= Isolation) 25.96 28.41 +0.04+0.04 ρ=0.5ρ=0.5 (= Hybrid) 37.08 29.97 +11.16+11.16 Ablation 4: Gate init α0 _0 / warmup ϵε, isolation α0=0.0 _0=0.0, ϵ=0ε=0 (exact FP) 25.97 29.13 +0.055+0.055 α0=0.01 _0=0.01 25.97 27.77 +0.044+0.044 α0=0.1 _0=0.1 (non-FP) 25.96 27.48 −0.444-0.444 Ablation 5: Growth timing (% of base training before growth), isolation 25% 28.39 30.33 +0.07+0.07 50% 20.90 27.13 +0.05+0.05 75% 21.61 27.02 +0.05+0.05 100% (= Gate-FP Iso) 25.96 28.41 +0.04+0.04 6 Discussion Gate-zero growth recommends two operating points. Within gate-zero growth, Freeze-Nothing≻Isolation≻Hybrid Freeze-Nothing Isolation Hybrid on the joint frontier. Both are predicted gate-zero configurations: Isolation gives exact preservation (Theorem 3.1, ΔA=+0.04 _A=+0.04) and is recommended when downstream tasks must be evaluated without regression; Freeze-Nothing gives the joint-frontier optimum (ΔA=−1.98 _A=-1.98, PPLB=20.37PPL_B=20.37) via the controlled-departure regime (Prop. 3.5: FW′,W′=0F_W ,W =0 at growth; Props. A.2, A.3: drift bound at empirical ‖∞≤0.083\| α\|_∞≤ 0.083). Ablation 2 pinpoints old-weight freezing as the binding constraint: Freeze-Gates-Only matches Freeze-Nothing (23.91/20.3623.91/20.36 vs. 23.93/20.3723.93/20.37). Validation PPL is not function preservation. Under Freeze-Nothing, PPLAPPL_A drops to 23.9323.93 (vs. pre-CL 25.9225.92) — a −1.98-1.98 change the standard CL metric reads as “better than original.” But the post-CL function has drifted from f∗f^* at first order (Prop. A.2; empirical ‖∞≤0.083\| α\|_∞≤ 0.083). The lower PPLAPPL_A is therefore a related-function gain, not preservation. Isolation is the only configuration that exactly recovers f∗f^* at the growth point and bounds drift to zero — the right choice when deployment requires bit-equivalent behaviour on AD_A (regulatory, A/B-test, or downstream-pinned settings). 6.1 Limitations Stochastic geometry estimators. Hessian top eigenvalues (Lanczos, single batch) and gradient-covariance rank (2020 mini-batches, 100100-dim projection; Appendix G) are coarse local diagnostics; the first-order statistic is the more stable primary measure. Cross-architecture plasticity gap on MoE. Per-checkpoint diagnostics localise the gap to clone-block redundancy in the depth-growth operator (post-CL new experts retain ≥0.71≥ 0.71 cosine similarity to frozen sources). The preservation mechanism transfers cleanly; whether MoE-specific tuning or a modified operator recovers dense-level plasticity is open (Future Work). 6.2 Broader impact FP growth lowers compute and energy cost of adapting trained models to new data, reducing the barrier for organisations without frontier-pretraining budgets. Downstream-use mitigations (data curation, alignment, evaluation) are out of scope. Future work. Full-scale (g=4g=4) FP comparators (Net2Net (Chen et al., 2016), LiGO (Wang et al., 2023a), bert2BERT (Chen et al., 2022)); reverse B→AD_B _A ordering and T≥3T≥ 3 multi-domain CL sequences; an MoE operator fix initialising new experts with random weights and expert_gate=0expert\_gate=0 to address the clone-block redundancy of Section 5.3; and a direct output-space preservation diagnostic (KL(f∗∥fpost)KL(f^*\,\|\,f_post) on held-out AD_A samples) to distinguish exact preservation from related-function gains observed under non-Isolation recipes. 7 Conclusion We presented a geometric framework for zero-initialised function-preserving growth: under transversality, rank separation (Theorem 3.1) decomposes the post-growth Jacobian into unchanged old and K new-gate directions. The load-bearing factor under Isolation is the zero-init structural property; zero-init residual stacking matches Gate-FP at g=2g=2, and under soft-KL recipes growth’s advantage is small (the controlled-departure regime of Props. A.2, A.3). Full-scale FP-vs-FP and multi-domain CL are the natural follow-up (Section 6.1). References Amari [1998] Amari, S.-I. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251–276. Bachlechner et al. [2021] Bachlechner, T., Majumder, B. P., Mao, H., Cottrell, G., and McAuley, J. (2021). ReZero is all you need: Fast convergence at large depth. Conference on Uncertainty in Artificial Intelligence (UAI). Chaudhry et al. [2019] Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. (2019). Efficient lifelong learning with A-GEM. International Conference on Learning Representations (ICLR). Chen et al. [2016] Chen, T., Goodfellow, I., and Shlens, J. (2016). Net2Net: Accelerating learning via knowledge transfer. International Conference on Learning Representations (ICLR). Chen et al. [2022] Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Peng, H., Chen, B., Lu, T., He, Q., et al. (2022). bert2BERT: Towards reusable pretrained language models. Annual Meeting of the Association for Computational Linguistics (ACL). De Lange et al. [2022] De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. (2022). A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7), 3366–3385. Du et al. [2024] Du, Y., Zeng, Y., et al. (2024). Stacking your transformers: A closer look at model growth for efficient LLM pre-training. Advances in Neural Information Processing Systems (NeurIPS), 37. Farajtabar et al. [2020] Farajtabar, M., Azizan, N., Mott, A., and Li, A. (2020). Orthogonal gradient descent for continual learning. International Conference on Artificial Intelligence and Statistics (AISTATS). French [1999] French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4), 128–135. Ghorbani et al. [2019] Ghorbani, B., Krishnan, S., and Xiao, Y. (2019). An investigation into neural net optimization via Hessian eigenvalue density. International Conference on Machine Learning (ICML). Houlsby et al. [2019] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP. International Conference on Machine Learning (ICML). Hu et al. [2022] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR). Jacot et al. [2018] Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems (NeurIPS). Kirkpatrick et al. [2017] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS, 114(13). Li and Hoiem [2018] Li, Z. and Hoiem, D. (2018). Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2935–2947. Loshchilov and Hutter [2019] Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. International Conference on Learning Representations (ICLR). Mallya and Lazebnik [2018] Mallya, A. and Lazebnik, S. (2018). PackNet: Adding multiple tasks to a single network by iterative pruning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). McCloskey and Cohen [1989] McCloskey, M. and Cohen, N. J. (1989). Catastrophic interference in connectionist networks. Psychology of Learning and Motivation, 24, 109–165. Merity et al. [2017] Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2017). Pointer sentinel mixture models. International Conference on Learning Representations (ICLR). Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI technical report. Rolnick et al. [2019] Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. (2019). Experience replay for continual learning. Advances in Neural Information Processing Systems (NeurIPS). Rusu et al. [2016] Rusu, A. A., Rabinowitz, N. C., Desjardins, G., et al. (2016). Progressive neural networks. arXiv:1606.04671. Shazeer [2020] Shazeer, N. (2020). GLU variants improve Transformer. arXiv:2002.05202. Su et al. [2024] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). RoFormer: Enhanced Transformer with rotary position embedding. Neurocomputing, 568, 127063. Wang et al. [2023a] Wang, P., Panda, R., Hennigan, L. K. M., et al. (2023). Learning to grow pretrained models for efficient transformer training. International Conference on Learning Representations (ICLR). Wang et al. [2023b] Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. (2023). Orthogonal subspace learning for language model continual learning. Findings of the Association for Computational Linguistics: EMNLP 2023. Zhang and Sennrich [2019] Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems (NeurIPS). Zhu et al. [2015] Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. IEEE International Conference on Computer Vision (ICCV). Appendix A Proofs A.1 Proof of Theorem 3.1 (rank separation under transversality) We prove the two parts in turn. Part (1): Exact flat new-weight directions, unconditional. For each new block ℓ∈1,…,K ∈\1,…,K\ with weight matrix Wℓ∈ℝPWW_ ^P_W and scalar gate αℓ _ , the contribution of block ℓ to the final output factors as αℓ _ multiplying a quantity depending on WℓW_ and the residual stream. At αℓ=0 _ =0 for all ℓ , every new block computes the identity and its downstream effect on fθ′f_θ vanishes. By the chain rule, ∂fθ′∂Wℓ(i)|αℓ=0=αℓ⋅∂Blockℓ(x;Wℓ)∂Wℓ(i)⋅Jℓdown= 0, ∂ f_θ ∂ W_ ^(i) |_ _ =0\;=\; _ · _ (x;\,W_ )∂ W_ ^(i)· J^down_ \;=\;0, identically for all x and all WℓW_ , where JℓdownJ^down_ denotes the downstream Jacobian propagating block ℓ ’s output to f. The K⋅PWK· P_W new-weight columns of J(θ′)J(θ ) are therefore the zero vector, so each new-weight coordinate basis vector lies in ker(J(θ′)) (J(θ )). These vectors are linearly independent because they occupy disjoint parameter coordinates. Part (2): Conditional rank additivity. Partition J(θ′)J(θ ) by parameter group: J(θ′)=(Jold|Jα|JW′),J(θ )= (\;J_old\; |\;J_α\; |\;J_W \; ), where Jold∈ℝD×PoldJ_old ^D× P_old are old-parameter columns, Jα∈ℝD×KJ_α ^D× K are new-gate columns, and JW′∈ℝD×K⋅PWJ_W ^D× K· P_W are new-weight columns. By Part (1), JW′=0J_W =0. Old parameters are untouched by growth, so Jold(θ′)J_old(θ ) coincides with the pre-growth Jacobian J(θold)J( _old) and has rank=rrank=r. Thus im(J(θ′))=im(Jold)+im(Jα)=im(J(θold))+im(Jα).im(J(θ ))\;=\;im(J_old)+im(J_α)\;=\;im(J( _old))+im(J_α). Decomposing im(Jα)im(J_α) as the orthogonal sum of its projection onto im(J(θold))im(J( _old)) and its complement, im(J(θ′))=im(J(θold))⊕im((I−Pold)Jα),im(J(θ ))\;=\;im(J( _old))\; \;im\! ((I-P_old)J_α ), so rank(J(θ′))=r+rank((I−Pold)Jα)≤r+K.rank(J(θ ))\;=\;r+rank\! ((I-P_old)J_α )\;≤\;r+K. Equality r+Kr+K holds iff the projected gate columns (I−Pold)Jα(I-P_old)J_α are linearly independent, i.e. transversality. Generic non-degeneracy of BlockℓBlock_ implies transversality almost surely under small clone noise; the remaining (Lebesgue-zero) degenerate cases occur, e.g., when a cloned new-block output already lies in the span of J(θold)J( _old). □ Remark A.1 (Genericity at trained checkpoints). Part (2) requires the K projected gate columns to be linearly independent of im(J(θold))im(J( _old)). The set of weight configurations for which independence fails is an algebraic variety of measure zero in the joint space of (θold,Wℓ=1K)( _old,\W_ \_ =1^K), so transversality holds Lebesgue-almost surely. Trained checkpoints, however, occupy a highly structured region of parameter space, and generic-position arguments based on ambient measure are not automatically applicable. The clone-noise perturbation ϵℓ∼(0,σ2I) _ (0,σ^2I) in the gate-zero construction breaks any structured degeneracies almost surely with respect to the noise distribution, for any σ>0σ>0: the set of noise realisations producing dependence remains a measure-zero variety in ℝK⋅PWR^K· P_W. We use σ=0.01σ=0.01, which preserves approximate FP at growth (max logit deviation 0.00.0 on real data, Section 5.1) while ensuring numerical independence. We empirically confirm transversality at the trained Gate-FP checkpoint (Appendix B): the projected-rank residual (I−Pold)Jα(I-P_old)J_α is full-rank K=36K=36, σmin⟂=144.6 _ =144.6, and the smallest principal angle between im(Jα)im(J_α) and the sampled im(Jold)im(J_old) is 56.9∘56.9 . The gradient-covariance effective rank reported in Section G is a coarser stochastic statistic and should not be read as a direct test of transversality (it is comparable across Gate-FP and GstackG_stack). Consequence for continual learning (formal statement). Under isolation (freezing all old parameters) on BD_B, the gradient ∇ℒB _B is restricted to the new-parameter coordinate subspace Tnew=0Pold×ℝPnewT_new=\0\^P_old×R^P_new. By Part (1), the K⋅PWK· P_W new-weight directions in TnewT_new are exactly flat for fθ′f_θ at the growth point, contributing no first-order change to the function. By Part (2) under transversality, the only function-changing directions accessible to the isolated CL update are the K gate directions, and they are Euclidean-orthogonal to all old-parameter coordinates. Therefore CL gradient steps under isolation cannot project onto old-parameter coordinates, and old-parameter curvature is exactly preserved at θ′θ . Under non-FP growth (e.g., GstackG_stack), part (1) fails: the new-weight columns of J(θgs′)J(θ _gs) are non-zero, so even a coordinate freeze on old parameters does not preserve f∗f^*, because the old forward pass through duplicated blocks has already been altered. This is the formal bridge from rank separation (the structural statement) to “isolation works” (the empirical observation). A.2 Proof of Proposition 3.6 (isolation as projected gradient descent) Freezing all old parameters means ∇θoldℒ=0 _ _oldL=0 during CL by construction. The gradient update under any optimizer that respects the freeze (e.g. AdamW with the frozen parameters masked) is therefore confined to the new-parameter coordinates Tnew=0Pold×ℝPnewT_new=\0\^P_old×R^P_new. Under gate-zero growth, Theorem 3.1 states that the function-preserving locus ℳ(f∗)=θ:fθ=f∗M(f^*)=\θ:f_θ=f^*\ at the growth point has tangent space ker(J(θ′)) (J(θ )), which contains all K⋅PWK· P_W new-weight directions exactly. Hence at θ′=ι(θold)θ = ( _old), Tnew∩ℳ(f∗)=0Pold×ℝK⋅PW×(coordswithαℓ=0),T_new (f^*)\;=\;\0\^P_old×R^K· P_W×(coordswith\; _ =0), which has codimension K within TnewT_new. The orthogonal projection ΠTnew _T_new onto TnewT_new thus restricts gradient updates to a subspace that intersects ℳ(f∗)M(f^*) in a K⋅PWK· P_W-dim. flat slab through θ′θ . As αℓ _ moves away from zero, the slab’s tangent structure deforms, but at the growth point the projection is exact. For GstackG_stack, J(θgs′)≠J(θold)J(θ _gs)≠ J( _old) because duplicated blocks immediately alter the active old-block computation, so the FP locus at θgs′θ _gs is not aligned with the old-coordinate axes; the same coordinate freeze is therefore only an approximate projection onto ℳ(f∗)M(f^*), with the approximation error bounded by the deviation ‖J(θgs′)−J(θold)‖\|J(θ _gs)-J( _old)\| on a verification batch. □ A.3 Proof of Proposition 3.7 (four-way partition) We prove the orthogonal four-way decomposition of ℝPnewR^P_new at θ′=ι(θold)θ = ( _old). Coordinate-block decomposition. The parameter vector decomposes into three disjoint coordinate blocks θ′=(θold,α′,W′)θ =( _old,\,α ,\,W ) with θold∈ℝPold _old ^P_old, α′∈ℝKα ^K, W′∈ℝK⋅PWW ^K· P_W. Directions in different coordinate blocks are automatically Euclidean-orthogonal. Old block: tangent vs. normal split. Within ℝPoldR^P_old, the restricted Jacobian is Jold(θ′)=J(θold)J_old(θ )=J( _old) (unchanged because new blocks are identity at αℓ=0 _ =0). The fundamental theorem of linear algebra gives the Euclidean-orthogonal split ℝPold=ker(J(θold))⊕im(J(θold)⊤),R^P_old= (J( _old))\; \;im(J( _old) ), with dimker(J(θold))=Pold−r (J( _old))=P_old-r and dimim(J(θold)⊤)=r (J( _old) )=r. Define Told∥=ker(J(θold))T_old = (J( _old)) (tangents to the old manifold) and Told⟂=im(J(θold)⊤)T_old =im(J( _old) ) (normals). New block: flat-weight + gate split. By Theorem 3.1 part (1), the K⋅PWK· P_W new-weight columns of J(θ′)J(θ ) vanish, so ℝK⋅PW⊂ker(J(θ′))R^K· P_W⊂ (J(θ )). Define Tnew∥=ℝK⋅PWT_new =R^K· P_W (the exactly-flat new-weight directions). Define Tnew⟂=ℝKT_new =R^K (the new-gate coordinate axes). Under the transversality assumption of Theorem 3.1, all K gate axes contribute non-zero projections onto the orthogonal complement of im(J(θold))im(J( _old)), so they are normal to ℳ(f∗)M(f^*) at θ′θ . (Without transversality, some gate axes may have a tangential component to ℳ(f∗)M(f^*); the dimension count then upper-bounds rather than equals K.) Combining. The four subspaces have pairwise zero Euclidean inner product. The old/new split is automatic from the disjoint coordinate blocks; the within-block splits (ker/im /im on the old side and Tnew∥/Tnew⟂T_new /T_new on the new side) are FTLA-orthogonal in their respective coordinate axes. Therefore ℝPnew=Told∥⊕Told⟂⊕Tnew∥⊕Tnew⟂,R^P_new=T_old T_old T_new T_new , with the dimensional annotations stated in the main text. The isolation update direction Tnew∥⊕Tnew⟂T_new T_new is Euclidean-orthogonal to Told⟂T_old by disjoint coordinates alone — a property that holds under any growth method that allocates new parameters to disjoint coordinates, including GstackG_stack. What gate-zero growth uniquely adds is that Tnew∥T_new consists of exactly flat directions under f (Theorem 3.1 (1)), so updates within Tnew∥T_new preserve the old function regardless of magnitude; under GstackG_stack, Tnew∥T_new is empty and updates in the new-weight coordinates are immediately function-changing. □ A.4 Proposition A.2 (small-α perturbative bound) and proof Proposition A.2 (Small-α perturbative bound, restated). Fix K new blocks with weights W1,…,WKW_1,…,W_K cloned from the trained checkpoint, and let hℓ(x)=Blockℓ(x;Wℓ)h_ (x)=Block_ (x;\,W_ ) denote the ℓ -th new-block output. For gate vector ∈ℝK α ^K, the post-growth function admits an additive expansion fθ′()(x)=fθold(x)+∑ℓ=1Kαℓhℓ(x)+R(x;),f_θ ( α)(x)=f_ _old(x)+ _ =1^K _ \,h_ (x)+R(x; α), where ‖R(x;)‖=O(‖2)\|R(x; α)\|=O(\| α\|^2) uniformly on a bounded input domain. Consequently, on input distribution AD_A, x∼A‖fθ′()(x)−fθold(x)‖2≤‖2⋅K⋅maxℓx‖hℓ(x)‖2+O(‖4)E_x _A\! \|f_θ ( α)(x)-f_ _old(x) \|^2≤\| α\|^2· K· _ E_x\|h_ (x)\|^2+O(\| α\|^4), and ΔA _A on a smooth log-loss admits the same O(‖2)O(\| α\|^2) scaling. Proof. The post-growth function is the composition of K gated residual updates x↦x+αℓhℓ(x)x x+ _ h_ (x), with the old forward pass unchanged at =0 α=0 by construction. Expanding the composition in α gives the additive form fθ′()(x)=fθold(x)+∑ℓαℓhℓ(x)+R(x;)f_θ ( α)(x)=f_ _old(x)+ _ _ h_ (x)+R(x; α) where the remainder R collects cross-terms αℓαℓ′ _ _ from chain-rule expansions of distinct gates and is therefore O(‖2)O(\| α\|^2) uniformly on a bounded input domain (validation tokens have bounded embedding norm; expected ‖hℓ(x)‖\|h_ (x)\| is finite by RMSNorm pre-activations). Cauchy–Schwarz on the linear term gives x‖∑ℓαℓhℓ(x)‖2≤‖2⋅K⋅maxℓx‖hℓ(x)‖2E_x\| _ _ h_ (x)\|^2≤\| α\|^2· K· _ E_x\|h_ (x)\|^2. The log-loss bound is the standard Fisher-quadratic expansion of KL(pθold∥pθ′())KL(p_ _old\|p_θ ( α)) around =0 α=0: the first-order term vanishes (since fθ′(0)=fθoldf_θ (0)=f_ _old) and the second-order term is ⊤Fαα(θold) α F_α( _old)\, α with FααF_α the gate-block Fisher (Proposition 3.5). ∎ A.5 Proposition A.3 (linear-in-α new-weight leakage) and proof Proposition A.3 (Linear-in-α new-weight leakage, restated). At a post-growth point θ′()θ ( α) with gate vector ∈ℝK α ^K, the new-weight Jacobian columns satisfy ∂fθ′()/∂Wℓ′=αℓ⋅∂Blockℓ(x;Wℓ)/∂Wℓ⋅Jℓdown∂ f_θ ( α)/∂ W _ = _ · _ (x;W_ )/∂ W_ · J^down_ , and consequently ‖JW′(θ′())‖op≤C‖∞\|J_W (θ ( α))\|_op≤ C\,\| α\|_∞, with C=maxℓ‖∂Blockℓ/∂Wℓ‖op⋅supmaxℓ‖Jℓdown()‖opC= _ \| _ /∂ W_ \|_op· _ α _ \|J^down_ ( α)\|_op, finite on bounded inputs and finite gate magnitudes. Proof. Block ℓ ’s contribution to the residual stream is αℓ⋅Blockℓ(x;Wℓ) _ ·Block_ (x;W_ ). By the chain rule, the gradient of fθ′()f_θ ( α) with respect to a new-weight coordinate Wℓ(i)W_ ^(i) factors as αℓ _ times the block’s internal gradient times the downstream Jacobian. The operator-norm bound follows by sub-multiplicativity and |αℓ|≤‖∞| _ |≤\| α\|_∞. The constant C is finite on bounded inputs and finite gate magnitudes by RMSNorm + bounded gates. ∎ A.6 Status of the framework’s claims Table 7: Status of the geometric framework’s claims. Exact claims hold without conditional assumptions. Conditional claims hold under transversality (Theorem 3.1 (2)). Approximate claims hold to first order in ‖\| α\| and degrade controllably as gates open. Empirical claims are not proven from the framework but are observed in our protocol. Claim Status Source Function preservation at growth (fθ′=f∗f_θ =f^* at =0 α=0) exact by construction New-weight Jacobian flatness at =0 α=0 exact Theorem 3.1 (1) Sparse Fisher block at =0 α=0 exact Proposition 3.5 Coordinate orthogonality of TnewT_new vs ToldT_old exact disjoint axes Rank additivity (rankJ(θ′)=r+KrankJ(θ )=r+K) conditional transversality, Thm 3.1 (2) Plasticity capacity =σmax⟂= _ per unit gate motion conditional Proposition 3.7 New-weight Jacobian leakage ‖JW′‖≤C‖∞\|J_W \|≤ C\| α\|_∞ approximate (linear) Proposition A.3 Function drift ‖fθ′()−f∗‖2=O(‖2)\|f_θ ( α)-f^*\|^2=O(\| α\|^2) approximate (quadratic) Proposition A.2 Multi-epoch CL preservation under realistic optimisers empirical Section 5.2, ΔA=+0.04 _A=+0.04 A.7 Remark on canonicity Gate-zero growth is not the unique FP map: any choice of (Wℓ,αℓ)(W_ , _ ) with αℓ=0 _ =0 yields function preservation, regardless of WℓW_ . What distinguishes the gate-zero construction in the symmetry-group view is that it maps into the fixed-point locus of the (g−1)LS_(g-1)L residual permutation action on the new-parameter fiber; combined with cloning old block weights into the new blocks, it is the canonical choice that inherits the old block’s learned features into the newly allocated capacity. Other FP choices (e.g. random new-weight initialisation with αℓ=0 _ =0) preserve function but discard this inheritance. Appendix B Direct Transversality Diagnostic This appendix reports the direct projected-rank diagnostic used to empirically verify the transversality assumption underlying Theorem 3.1 (2) and to back the quantitative σmin⟂ _ statement in Proposition 3.7. Method. At the post-growth Gate-FP checkpoint with g=4g=4 (K=36K=36 new block-gates), we compute (i) the K gate-Jacobian columns jαℓ=∂f/∂αℓj_ _ =∂ f/∂ _ via forward-mode automatic differentiation (Jacobian-vector products with tangent αℓ1_ _ ), giving Jα∈ℝD×KJ_α ^D× K where D=B⋅T⋅VD=B· T· V is the flattened output dimension; and (i) M=200M=200 samples of J(θold)vJ( _old)v where v is a unit-norm random direction supported on the joint old-parameter coordinates, giving an empirical low-rank approximation to im(Jold)im(J_old). We orthonormalise the sample matrix via QR, project JαJ_α onto the orthogonal complement of the sampled subspace, and compute the SVD of the residual (I−Pold)Jα(I-P_old)J_α. We use a held-out validation batch of B=2B=2, T=16T=16 tokens; output dimension D=1,608,224D=1,608,224. Results. Table 8 reports the diagnostic outputs. The residual is full-rank (K=36K=36 at threshold 10−3σmax⟂10^-3 _ ), the smallest singular value σmin⟂=144.6 _ =144.6 is on the same order of magnitude as the gate-column norms (‖jαℓ‖\|j_ _ \| ranging 279279–660660), and the smallest principal angle between im(Jα)im(J_α) and the sampled im(Jold)im(J_old) is 56.9∘56.9 . All K residual singular values exceed 144144, and all K principal angles exceed 56∘56 . Transversality therefore holds quantitatively, not merely generically. Table 8: Transversality diagnostic at the Gate-FP post-growth checkpoint (g=4g=4, K=36K=36, M=200M=200, D=1,608,224D=1,608,224). The residual (I−Pold)Jα(I-P_old)J_α is full-rank with well-conditioned spectrum, and gate directions are geometrically separated from the sampled old-Jacobian image by at least 56.9∘56.9 . Quantity Value K (new block-gates) 3636 M (sampled old directions) 200200 Output dimension D 1,608,2241,608,224 Effective rank of residual at 10−3σmax10^-3 _ (=)36\;(=K) σmax⟂ _ 993.4993.4 σmin⟂ _ 144.6144.6 Condition number σmax⟂/σmin⟂ _ / _ 6.876.87 Smallest principal angle (deg) 56.9∘56.9 Largest principal angle (deg) 83.7∘83.7 Gate-column norm range ‖jαℓ‖\|j_ _ \| [279, 660][279,\,660] Interpretation. The diagnostic confirms the load-bearing conditional in Theorem 3.1 (2): all 3636 new-gate Jacobian columns are linearly independent of im(Jold)im(J_old) at the trained checkpoint, and the geometric separation (56.9∘56.9 minimum) leaves substantial margin against finite-precision degradation. The condition number 6.876.87 indicates that the rank decomposition is numerically robust — σmin⟂ _ is not in the precision-limited regime where rank claims become estimator-dependent. Caveat on the sampling approximation. We approximate im(Jold)im(J_old) by sampling M=200M=200 random old-direction Jacobian columns. This is a sufficient (not necessary) test for transversality: if all K residual columns are linearly independent of the sampled subspace, they are linearly independent of any subspace contained in it; but a positive projection onto a direction we did not sample remains possible in principle. Increasing M to 500500 in pilot runs did not materially change the residual singular values, suggesting the sample is sufficient at this scale. A complete Lanczos-based test of rank(J(θold))rank(J( _old)) is left as future work. Appendix C Baseline Comparisons: Full Table Table 9 reports the no-protection rows omitted from the body Table 5. All four no-protection cells produce the expected catastrophic forgetting signature (ΔA>+200 _A>+200). Table 9: No-protection rows for each baseline family, omitted from the body for compactness. Outcomes are uniformly catastrophic, as expected without a preservation regulariser. Family CL Strategy PPLA↓PPL_A PPLB↓PPL_B ΔA↓ _A No-growth (300M) No protection 303.59303.59 20.4620.46 +277.67+277.67 No-growth (300M) Replay 504.61504.61 20.3120.31 +478.69+478.69 Zero-init stacking (g=2g=2) No protection 320.14320.14 20.5620.56 +294.22+294.22 LoRA-CL (rank 6464) No protection 248.36248.36 22.0222.02 +222.44+222.44 Appendix D Multi-Seed Validation at g=2g=2 We report three seeds of Gate-FP + Isolation at g=2g=2 scale (300M→ 454M) to estimate variance on the headline preservation finding. Seed 4242 corresponds to the original Ablation 1 row; seeds 12341234 and 999999 were chosen before launch and reported regardless of outcome. All three seeds completed all 1010 CL epochs. Table 10: Multi-seed validation of the Gate-FP + Isolation preservation finding at g=2g=2 scale. The pre-CL row is identical across seeds because growth at α=0α=0 is bit-exact and pre-CL evaluation precedes any training. Variance is reported on the post-CL columns only. Seed Pre-CL PPLAPPL_A Post-CL PPLAPPL_A ↓ Post-CL PPLBPPL_B ↓ _A ↓ 4242 (original Ablation 1) 25.91825.918 26.01326.013 30.44730.447 +0.0946+0.0946 12341234 25.91825.918 26.00726.007 30.55630.556 +0.0884+0.0884 999999 25.91825.918 26.00426.004 30.52030.520 +0.0857+0.0857 Mean ± std (3 seeds) — 26.008±0.00526.008± 0.005 30.51±0.0630.51± 0.06 +0.0896±0.0046+0.0896± 0.0046 Reading. On the two completed seeds, ΔA _A varies by σ=0.0031σ=0.0031, which is two orders of magnitude smaller than the gap to the non-FP baseline (ΔA>+1,100 _A>+1,100 for GstackG_stack under no-protection at g=4g=4, Table 2) and three orders of magnitude smaller than the gap to no-growth + Distillation (ΔA=+13.08 _A=+13.08, Table 5). The qualitative preservation finding is therefore robust to seed variation at g=2g=2 scale. Multi-seed validation at g=4g=4 scale (∼100 100 GPU-hours per additional seed, vs. ∼17 17 hours per g=2g=2 seed) was prohibitively expensive within our compute budget and is the natural follow-up. Appendix E Preliminary Cross-Architecture Validation Before the main 300M→ 857M Transformer experiments, we ran smaller-scale studies (Phases 1–4) across four architecture families to validate that gate-zero FP holds beyond a single configuration. Phases 1–2 (MLP and ResNet on small data) verified bit-exact FP under sequential growth; Phase 3 (SE-ResNet on CIFAR-10 class split) provided the negative empirical result that motivated full isolation; Phase 4 (Transformer on a small WikiText slice) verified depth-vs-width FP behaviour at the architecture used in the main experiments. Function preservation across architectures. Table 11 reports the maximum logit difference between pre- and post-growth models on a held-out batch. Depth growth via gate-zero is bit-exact across all four architecture families. Width growth on a Transformer breaks FP because the RMSNorm denominator changes with hidden dimension, producing a compound distortion of dold/dnew d_old/d_new per norm layer (1313 layers ⇒ 0.86613≈0.15×0.866^13≈ 0.15× scaling for 384→512384→ 512). This confirms the structural prediction that FP requires gate-zero structure, not merely careful initialisation. Table 11: Function preservation at growth across four architecture families. “Max diff” is the maximum absolute logit difference between pre- and post-growth models on the same held-out batch. Phase Architecture Growth type Max diff FP exact? 1 MLP (2-16-16-3) Width (+2 neurons) 0.0 Yes 2 ResNet (MNIST) Width (channels 2×2×) 0.0 Yes 3 SE-ResNet (CIFAR-10) Width (channels 1.5×1.5×) 0.0 Yes 4 Transformer (WikiText) Depth (6→86→ 8 layers) 0.0 Yes 4 Transformer (WikiText) Width (384→512384→ 512) 13.2 No CL on grown SE-ResNet: naïve strategies all forget. Phase 3 trained an SE-ResNet grown from (8,16,32)(8,16,32) to (16,32,64)(16,32,64) channels sequentially on CIFAR-10 classes 0–4 (Task A) then 5–9 (Task B). Three naïve CL strategies were tested (Table 12); all three drove Task A accuracy to near zero. The diagnostic finding was that even with convolutional weights fully frozen (third row), trainable old gates alone caused enough feature drift (cosine similarity dropping to 0.50.5–0.90.9) to destroy classifier calibration. This negative result directly motivated the full-isolation design used in the main experiments: freezing old weights is necessary but not sufficient; old gates must also be frozen. Table 12: Phase 3 CL on grown SE-ResNet (CIFAR-10 class split). Every naïve CL strategy results in near-complete forgetting of Task A after training on Task B. “Feature drift” is cosine similarity between pre- and post-CL backbone features on Task A inputs. Strategy Task A Task B Feat. drift What failed Fine-tune (no protection) 0.00%0.00\% 90.50%90.50\% 0.60.6 Everything drifts Grow + differential LR 0.00%0.00\% 79.78%79.78\% 0.30.3 Slow LR insufficient Grow + freeze conv weights 0.02%0.02\% 72.96%72.96\% 0.50.5–0.90.9 Gate drift alone fatal The geometric reading is direct: freezing weights constrains Told⟂T_old but trainable old gates allow drift along directions that change old-block computation. Full isolation (freeze old weights and old gates) eliminates both sources of drift, which is precisely the strategy Theorem 3.1 predicts. Appendix F MoE Per-Epoch Trajectory and Diagnostic Table This appendix provides the full per-epoch CL trajectory for the MoE isolation run summarised in Section 5.3 and the mechanistic per-checkpoint diagnostic. F.1 Per-epoch CL trajectory Table 13 reports the trajectory of the exp2_moe_isolation run across all 1010 CL epochs. Training loss decreases monotonically from 6.856.85 at epoch 1 to 4.694.69 at epoch 10, while validation PPLBPPL_B reaches its minimum of 140.26140.26 at epoch 2 and then regresses monotonically to 182.06182.06 by epoch 10. Validation PPLAPPL_A shows small negative ΔA _A in early epochs (slight positive transfer back to AD_A from CL training) before drifting up to a final ΔA=+0.20 _A=+0.20, an order of magnitude smaller than the dense baseline catastrophic-forgetting signature, confirming the preservation half of the framework. Table 13: MoE isolation: per-epoch CL trajectory. Epoch 0 (italic) is the post-growth pre-CL state. Validation PPLAPPL_A is preserved throughout; validation PPLBPPL_B is best at epoch 22 and regresses thereafter, while training loss continues to decrease monotonically — the canonical train-validation overfitting signature. Epoch Train loss ↓ Val PPLAPPL_A ↓ Val PPLBPPL_B ↓ _A 0 (pre-CL) — 67.21 2155.77 — 1 6.85 66.79 145.75 −0.42-0.42 2 5.41 66.59 140.26 −0.62-0.62 3 5.18 66.27 149.48 −0.94-0.94 4 5.05 66.64 155.14 −0.57-0.57 5 4.94 67.03 163.16 −0.18-0.18 6 4.85 67.20 164.86 −0.01-0.01 7 4.78 67.30 169.90 +0.09+0.09 8 4.73 67.36 176.50 +0.15+0.15 9 4.70 67.40 180.33 +0.19+0.19 10 4.69 67.41 182.06 +0.20+0.20 Figure 1: Per-epoch CL trajectory of the MoE isolation run. Top: training loss decreases monotonically from 6.856.85 at epoch 11 to 4.694.69 at epoch 1010. Bottom: validation perplexity on AD_A and BD_B on a shared y-axis to honestly visualise their relative magnitudes. PPLAPPL_A (green) stays within ±1± 1 of the pre-CL baseline (67.2167.21, dashed) throughout and is visually flat at this scale (final ΔA=+0.20 _A=+0.20). PPLBPPL_B (red) reaches its minimum of 140.26140.26 at epoch 22 (circled) and then regresses monotonically to 182.06182.06 by epoch 1010. The simultaneous decreasing training loss and increasing validation PPLBPPL_B confirm overfitting rather than measurement noise as the cause of the regression. Exact per-epoch values appear in Table 13. F.2 Per-checkpoint mechanistic diagnostic Table 14 reproduces the diagnostic of Section 5.3 with extended commentary. The protocol loads the 706706M MoE base, applies grow_moe (4→ 8 experts, 12→ 24 layers), and runs a single forward pass on 22 WikiText-103 validation sequences with diagnostic hooks; the post-CL checkpoint is loaded from exp2_moe_isolation/cl_final.pt and the same hooks executed. Table 14: MoE isolation per-checkpoint diagnostic (new MoE blocks only, nnew=12n_new=12, top-k=2k=2, N=8N=8). “Top-11 share” is the fraction of tokens routed to the single most-selected expert (uniform =1/N=0.125=1/N=0.125); “entropy / logN N” is normalised routing entropy (1.01.0 = uniform routing). All values computed on a held-out batch. Diagnostic metric Pre-CL Post-CL Reading Max |αℓ|| _ | on new blocks 0.0000.000 0.0830.083 gradient sought higher Mean |αℓ|| _ | on new blocks 0.0000.000 0.0830.083 all gates at clamp Cumulative ∑ℓ|αℓ| _ | _ | (nnew=12n_new=12) 0.000.00 0.990.99 ≈ one fully-open block Mean top-11 expert share 0.5000.500 0.5000.500 no within-CL collapse Routing entropy / logN N (mean) 0.3330.333 0.3330.333 inherited concentration Mean cosine sim to source expert 0.9180.918 0.8100.810 modest differentiation Min cosine sim to source expert 0.8520.852 0.7120.712 most-changed still 71% similar Reading. Three patterns emerge. First, all 12 new block gates converge to the safety-clamp ceiling α=1/nnew=0.083α=1/n_new=0.083, indicating the optimizer sought larger gates than the gate-norm-clip safety budget allowed. The cumulative gate-sum ∑ℓ|αℓ|≈1.0 _ | _ |≈ 1.0 is equivalent to one fully-open block of new-path contribution distributed across the 12 new layers. Second, routing concentration is unchanged from pre-CL (ρtop1=0.50 _top1=0.50 in both states; normalised entropy 0.3330.333), ruling out within-CL router collapse as the failure mode. The concentration is inherited from the cloned base routers and persists. Third, per-expert cosine similarity to source experts drifts from 0.9180.918 to 0.8100.810 (mean) and 0.8520.852 to 0.7120.712 (min). New experts learn something, but they remain mostly clones of their frozen sources in parameter-space terms. The combination is consistent with clone-block redundancy: new blocks open their gates to the safety ceiling but cannot differentiate enough from the frozen sources to provide complementary capacity for BD_B, so what they fit is sample-specific memorisation rather than generalisable features. Appendix G Manifold Geometry This appendix details the protocol used for Section G and Table 15. Table 15: Loss landscape geometry at three checkpoints. “Grad-cov rank” is the effective rank of the per-batch gradient covariance (exponential entropy of normalized singular values). The Gate-FP rows are consistent with preservation of local geometry across the growth event (rank drift <0.02<0.02); CL contracts the active subspace. The GstackG_stack row reports comparable grad-cov rank (18.3718.37) despite not being function-preserving — this stochastic rank statistic is too coarse to distinguish FP from non-FP at growth time. The Hessian estimates use Lanczos on a single batch and should be read as coarse local diagnostics — see protocol below for reliability caveats. Checkpoint Method Grad-cov rank Top eig. Trace est. Hessian n+n_+ Pre-growth Base 18.45 203.09 66.75 3 Post-growth Gate FP 18.47 188.05 66.79 3 Post-growth GstackG_stack 18.37 201.12 64.30 3 Post-CL Gate FP + Iso 11.01 901.02 2066.71 4 G.1 Estimators Gradient-covariance effective rank. We compute the per-example gradient gn=∇θℒ(θ;xn)∈ℝPg_n= _θL(θ;\,x_n) ^P for n=1,…,Nbn=1,…,N_b, where Nb=20N_b=20 mini-batches of size 22. To control memory at P≈109P≈ 10^9, we apply a Johnson-Lindenstrauss random projection Π:ℝP→ℝ100 :R^P ^100 before stacking, giving g~n=Πgn∈ℝ100 g_n= g_n ^100. We then form the empirical gradient-covariance Σ=(1/Nb)G~⊤G~ =(1/N_b)\, G G and compute its singular values σii=1100\ _i\_i=1^100. The effective rank is reported as the exponential entropy of the normalised singular values: deff(Σ)=exp(−∑i=1100σ~ilogσ~i),σ~i=σi/∑jσj.d_eff( )\;=\; \! (- _i=1^100 σ_i σ_i ), σ_i= _i / _j _j. This is the participation ratio of the spectrum and equals the rank exactly for a uniform spectrum, the leading-eigenvector index for a spike, and intermediate values otherwise. Top-3 Hessian eigenvalues. We use Lanczos iteration [Ghorbani et al., 2019] on Hessian-vector products v↦Hv=∇θ(g⊤v)v Hv= _θ(g v) at a fixed θ, with 2020 Lanczos iterations and a single batch of size 11. To control GPU memory at the 1.14B-parameter scale, we offload the Lanczos basis vectors to CPU. We report the top-33 Ritz values from the resulting tridiagonal matrix. Trace estimate is via Hutchinson’s stochastic trace. Reliability caveat. The Hessian estimator is stochastic, single-batch, and uses only 2020 Lanczos iterations; absolute eigenvalue magnitudes should be read as coarse local diagnostics rather than precise global curvature. The first-order gradient-covariance rank statistic is substantially more stable: in pilot runs varying NbN_b from 2020 to 256256, deffd_eff values changed by less than 0.50.5 at any checkpoint. We therefore treat deffd_eff as the primary manifold proxy and the Hessian estimates as secondary indicators. G.2 Spectral leakage We define a single scalar metric for tracking how rank separation degrades during CL and report Hutchinson-HVP estimates at three checkpoints (post-growth Gate-FP, post-growth GstackG_stack, post-CL Gate-FP+Iso). Results in Table 16. Definition G.1 (Spectral leakage). Let H∈ℝP×PH ^P× P be the Hessian at a checkpoint, and let Pold,PnewP_old,P_new denote the orthogonal projections onto old-parameter and new-parameter coordinate subspaces respectively. The spectral leakage is leak=‖PoldHPnew‖F‖H‖F.S_leak= \|P_old\,H\,P_new\|_F\|H\|_F. leak=0S_leak=0 means old and new parameter subspaces are completely decoupled in the Hessian; leak>0S_leak>0 means curvature “leaks” across the old/new boundary. By Theorem 3.1, leak=0S_leak=0 at the gate-zero growth point (the Fisher new-weight block and all new-weight cross-terms vanish); as CL training opens gates, leakS_leak grows controllably (linear in ‖∞\| α\|_∞ at first order; Proposition A.3). We estimate leakS_leak via Hutchinson with Hessian-vector products. The estimator samples v∈ℝPv ^P supported on new-parameter coordinates with Rademacher entries, computes HvHv once, and reports ‖(Hv)old‖2\|(Hv)_old\|^2 averaged over draws as the numerator; the denominator uses full-dim Rademacher v. We use 1616 Hutchinson draws per estimator on a single batch of size 22 with sequence length 6464 from AD_A. Table 16: Spectral leakage at three checkpoints (Hutchinson-HVP estimator, 1616 draws). leakS_leak at post-growth Gate-FP is two orders of magnitude smaller than at post-growth GstackG_stack (32×32× separation) — the metric distinguishes FP from non-FP construction sharply, where the gradient-covariance rank statistic (Table 15) does not. Post-CL Gate-FP++Iso rises to 0.0720.072, consistent with the linear-in-‖∞\| α\|_∞ prediction of Proposition A.3 at ‖∞≈0.083\| α\|_∞≈ 0.083 (the safety clamp). Checkpoint ‖PoldHPnew‖F2\|P_oldHP_new\|_F^2 ‖H‖F2\|H\|_F^2 leak↓S_leak Post-growth Gate-FP 2.03×1032.03× 10^3 8.18×1078.18× 10^7 0.00500.0050 Post-growth GstackG_stack 7.59×1067.59× 10^6 2.94×1082.94× 10^8 0.16080.1608 Post-CL Gate-FP ++ Iso 4.25×1054.25× 10^5 8.16×1078.16× 10^7 0.07220.0722 Reading. Three observations. (i) The Gate-FP / GstackG_stack separation at growth time (32×32×) is the discriminating measurement of the Fisher-block rank-separation prediction (Proposition 3.5); the gradient-covariance rank (18.4718.47 vs. 18.3718.37, Table 15) does not distinguish them. (i) The post-CL Gate-FP value (leak=0.072S_leak=0.072 at ‖∞≈0.083\| α\|_∞≈ 0.083) is consistent with the Proposition A.3 linear-leakage prediction: leakS_leak rises by ∼14× 14× as ‖∞\| α\|_∞ rises from 0 to 0.0830.083, an empirical slope of ∼0.83 0.83 in the same units. (i) Even after 1010 epochs of CL, the post-CL Gate-FP value is less than half of the post-growth GstackG_stack at-growth value: a Gate-FP model that has fully run CL retains cleaner old/new Hessian decoupling than a GstackG_stack model that has not moved a single training step. Appendix H Extended Ablation Results The combined Table 6 in the main text already reports all completed ablation rows. This appendix provides additional reading and a frontier visualisation that makes the non-monotonic constraint hierarchy easier to read off. H.1 Pareto frontier across Gate-FP CL methods Figure 2 plots final (PPLA,PPLB)(PPL_A,PPL_B) for all Gate-FP CL methods, the two completed freeze-strategy ablations, and the joint-training Scratch reference, on log–log axes so that runs spanning four orders of magnitude on PPLAPPL_A are visible together. The Pareto frontier (lower PPLAPPL_A and lower PPLBPPL_B jointly) is dominated by Freeze-Nothing and Freeze-Gates-Only, both of which leave the old weight matrices trainable. Full Isolation sits to the right of the frontier (higher PPLBPPL_B), and Hybrid sits further right still — the same non-monotonic ordering observed in Table 2, here visible as a Pareto-dominance relation rather than a single-axis comparison. Figure 2: Pareto frontier of final (PPLA,PPLB)(PPL_A,PPL_B) across Gate-FP CL methods, freeze-strategy ablations, and the joint-training Scratch reference. Lower-left is better on both axes. The dotted green line connects the lower-left frontier; Freeze-Nothing and Freeze-Gates-Only jointly dominate full Isolation and Hybrid, confirming the non-monotonic constraint hierarchy reported in Table 2. Ablation 1 (growth factor g). All three rows complete. The trend is monotonic: larger g provides more trainable subspace and yields lower PPLBPPL_B, with |ΔA|<0.1| _A|<0.1 across all three. Ablation 2 (what to freeze). All four rows complete. The two configurations that leave old weights trainable (Freeze-Nothing and Freeze-Old-Gates-Only) achieve essentially identical PPLA≈23.9PPL_A≈ 23.9 and PPLB≈20.4PPL_B≈ 20.4; the two configurations that freeze old weights sit higher on both axes. Freezing the old weight matrices, not the old growth gates, is the binding constraint that over-restricts plasticity. Ablation 3 (replay fraction). Two endpoints: ρ=0ρ=0 (collapses Hybrid loss to Isolation; reuses Gate-FP Isolation) and ρ=0.5ρ=0.5 (default Hybrid; reuses Gate-FP Hybrid). Both directly read off Table 2. Interpolating values ρ∈(0,0.5)ρ∈(0,0.5) are not run; the two-point contrast already supports the message that adding CE-on-AD_A on top of Isolation hurts both axes. Ablation 4 (gate-init / warmup). All three rows complete. The α0=0 _0=0 row uses zero gate-warmup (ϵ=0ε=0) so the literal α0=0 _0=0 condition is honestly tested; gradient flow to new-block weights then depends entirely on the scalar gate opening through its own gradient. The α0=0.01 _0=0.01 row starts from a non-zero gate (δf=O(α0)≈3×10−3 _f=O( _0)≈ 3× 10^-3 in PPLAPPL_A at growth time) and benefits from immediate gradient flow into new-block internal weights, yielding lower PPLBPPL_B (27.7727.77 vs. 29.1329.13). The α0=0.1 _0=0.1 row recovers the original base PPLAPPL_A post-CL (ΔA=−0.44 _A=-0.44 relative to its drifted post-growth baseline), indicating that under isolation, CL training can absorb the small approximate-FP perturbation introduced by α0>0 _0>0 while benefiting from faster gradient flow into new-block weights. Ablation 5 (growth timing): preservation is timing-robust. We grow the base model after 2525%, 5050%, 7575%, and 100100% of the base-training schedule (i.e. at 2.52.5, 55, 7.57.5, 1010 epochs on AD_A), then run the standard 10-epoch CL phase under isolation. The 100100% row reuses exp1_gate_fp_isolation, since 100100%-then-grow-then-CL is identical to the headline configuration of Table 2; we do not duplicate the run. Across all four timings, |ΔA|<+0.08| _A|<+0.08, confirming that the preservation property of gate-zero growth + isolation is robust to base-training maturity — the rank-separation mechanism does not require the base to be fully converged. Table 17: Ablation 5 (growth timing). The 100100% row is the Gate-FP Isolation entry from Table 2 and is not re-run. Pre-CL PPLAPPL_A varies non-monotonically with timing because mid-schedule base checkpoints can have lower validation perplexity than the final-schedule checkpoint (the 100100% base trains 1010 full epochs and may sit past its best-validation point on WikiText-103). The preservation property holds in all four cases: |ΔA|<+0.08| _A|<+0.08. Growth timing Pre-CL PPLAPPL_A Post-CL PPLAPPL_A ↓ Post-CL PPLBPPL_B ↓ _A 2525% 28.31 28.39 30.33 +0.074+0.074 5050% 20.85 20.90 27.13 +0.052+0.052 7575% 21.56 21.61 27.02 +0.053+0.053 100100% (= Gate-FP Iso) 25.92 25.96 28.41 +0.043+0.043 Reading. Two observations. First, ΔA _A stays in the same narrow band (+0.04+0.04 to +0.07+0.07) regardless of when growth happens, supporting the claim that the rank-separation mechanism is a structural property of the growth operator rather than an artefact of any particular base configuration. Second, the non-monotonicity of Pre-CL PPLAPPL_A across timings is consistent with the from-scratch baseline’s overfitting trajectory (Section 5.2, the PPLA=17.40PPL_A=17.40 at epoch 33 vs. 43.2343.23 at epoch 1010 phenomenon): mid-schedule checkpoints can have lower validation perplexity than the final-schedule checkpoint. The fact that Gate-FP + isolation preserves whichever Pre-CL state it starts from is the relevant invariant, not the absolute perplexity level. Appendix I Implementation Details Architecture. Base model: GPT-style decoder-only Transformer with dmodel=1024d_model=1024, nheads=16n_heads=16, dffn=4096d_ffn=4096, L=12L=12 layers, Lmax=1024L_ =1024 context, GPT-2 BPE tokenizer [Radford et al., 2019] (vocab 50,25750,257), SwiGLU FFN [Shazeer, 2020], RMSNorm [Zhang and Sennrich, 2019], rotary positional encoding [Su et al., 2024]. Gate-zero variants add scalar growth gates (αℓ _ , head-gate, ffn-gate, expert-gate). MoE base: same backbone with 44 experts, top-k=2k=2 routing, dexpert=4096d_expert=4096, load-balancing loss weight λaux=0.01 _aux=0.01. Base training (Phase 1). A=D_A= WikiText-103 [Merity et al., 2017] (training split, ∼ 118M tokens after tokenization). 1010 epochs. AdamW [Loshchilov and Hutter, 2019] with β=(0.9,0.999)β=(0.9,0.999), weight decay 0.10.1 on non-gate params, 0.00.0 on gate params. Learning rate 3×10−43× 10^-4 (non-gate) and 1×10−31× 10^-3 (gate), linear warmup over 500500 steps then cosine decay to 0. Microbatch size 22, gradient accumulation 6464 (effective batch 128128). Mixed-precision (fp16). Gradient clipping: ‖g‖2≤1.0\|g\|_2≤ 1.0 on non-gate params, ≤0.1≤ 0.1 on gate params. Random seed 4242. Growth (Phase 2). Gate-FP: depth growth factor g=4g=4 (12 → 48 layers); new blocks cloned from existing blocks round-robin with i.i.d. Gaussian noise σ=0.01σ=0.01 on weights; α0=0 _0=0 for new block-gates by default (Ablation 4 also tests α0∈0.01,0.1 _0∈\0.01,0.1\). Append pattern: new blocks stacked after old. MoE: g=2g=2 (12 → 24) plus 44 new experts per existing MoE layer with expert-gate =0=0. GstackG_stack: g=4g=4 block duplication without gating (per Du et al. 2024). Continual learning (Phase 3). B=D_B= BookCorpus [Zhu et al., 2015], capped at 118,000,000118,000,000 tokens to match |A||D_A|. 1010 CL epochs. Optimizer reset; CL learning rate 1×10−41× 10^-4 (non-gate) and 5×10−55× 10^-5 (gate), linear warmup 500500 steps then cosine decay. CL gate learning rate is deliberately low (∝1/nnew 1/ n_new) to bound cumulative gate perturbation; new-block gates are clamped to |αℓ|≤1/nnew| _ |≤ 1/n_new each step to prevent gate explosion. Replay buffer: random sample of 10%10\% of AD_A training set (≈11,500≈ 11,500 sequences), reshuffled each epoch; replay microbatch size 22 (capped to 11 for MoE due to memory). Distillation hyperparameters: λ=0.5λ=0.5, T=2.0T=2.0 (effective KL weight λT2=2.0λ T^2=2.0). Hybrid replay fraction ρ=0.5ρ=0.5. For Isolation, gate warmup ϵ=1×10−3ε=1× 10^-3 applied at CL start when α0=0 _0=0 (set to 0 for the α0=0 _0=0 row of Ablation 4). All gradient-clipping values from Phase 1 carried over. Evaluation. PPLPPL on WikiText-103 validation (∼247 247K tokens) and BookCorpus validation (∼500 500K tokens), microbatch 22, fp16. Reported ΔA=PPLApost−CL−PPLApre−CL _A=PPL_A^post-CL-PPL_A^pre-CL (post-growth, pre-CL baseline). FP verification: max absolute logit difference between pre- and post-growth models on a verification batch of 2×642× 64 random tokens; passing threshold 10−510^-5 for dense, 10−410^-4 for MoE (MoE top-k routing introduces small numerical drift). Geometry estimation (Section G). Gradient covariance rank: per-example gradient matrix over NbN_b mini-batches, projected to 100100 dimensions via random projection, followed by truncated SVD; effective rank reported as exponential entropy of normalized singular values. Hessian top eigenvalues: Lanczos-style Hessian-vector products with 2020 iterations, single batch, CPU offload. We adopt this stochastic protocol because exact large-batch second-order measurement is infeasible at 1.14B-scale on single-GPU memory; absolute values should therefore be read as coarse local diagnostics, with the gradient-covariance rank statistic the more stable primary measure. The ranking across checkpoints (Table 15) is robust to varying NbN_b from 2020 to 256256 in our pilot runs. Compute. All runs on a single NVIDIA L20 GPU (48 GB) per configuration. Total compute across the 2×52× 5 CL matrix, MoE experiments, and ablations is approximately 2,5002,500 GPU-hours. Reproducibility caveats. Single seed per cell of Tables 2, 3, and 6 due to compute cost. Multi-seed validation at smaller scale is in progress and will be reported in supplementary material. We also test only one dataset ordering (A→BD_A _B); reverse ordering is left as future work.