Paper deep dive
Federated Lightweight Fine-Tuning
Radhakrishna Achanta, Will Reed
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/22/2026, 2:24:26 AM
Summary
The paper introduces FLITE (Federated Low-rank Iterative Training Engine), a method for federated fine-tuning that drastically reduces communication bandwidth by transmitting small trainable latent vectors instead of full model weights. By using a delta low-rank mapping network over a shared pretrained base, FLITE maintains the exact averaging identity of FedAvg while reducing per-round payload to ~5 KB (an 8718x reduction) with negligible accuracy loss on CIFAR-100.
Entities (8)
Relation Signals (6)
FLITE → evaluatedon → CIFAR-100
confidence 95% · On CIFAR-100 with ResNet-18+GroupNorm, our method... reaches 74.67%
FLITE → uses → Mapping Network
confidence 95% · Mapping networks generate a network's weights from a small trainable latent... We turn this into a practical low-bandwidth federated channel
FLITE → usesmodel → ResNet-18
confidence 95% · On CIFAR-100 with ResNet-18+GroupNorm
FLITE → outperforms → FedAvg
confidence 90% · FLITE communicates 1,280 floats... an 8718x reduction... and reaches 74.67%, within ~0.5 pp of full-weight FedAvg
FLITE → outperforms → PowerSGD
confidence 90% · the method sits one to two orders of magnitude below PowerSGD and top-k on the bandwidth-accuracy Pareto
Mapping Network → enables → Delta Formulation
confidence 85% · delta formulation $ heta = heta^{ ext{pre}} + U V^{ op} z$ that learns an additive correction around a shared centrally-pretrained base
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor. We take a different lever. Mapping networks generate a network's weights from a small trainable latent through a frozen affine projection; because the map is shared and affine, averaging latents is exactly averaging the generated weights. We turn this into a practical low-bandwidth federated channel with two changes: a low-rank, seed-regenerable factorisation of the projection (cutting generator memory from ~80 GB to ~10 MB), and a delta formulation $\theta = \theta^{\mathrm{pre}} + U V^{\top} z$ that learns an additive correction around a shared centrally-pretrained base -- federated fine-tuning, which is what makes the method work at scale. A frozen orthogonal classifier head further removes the head from the payload while improving accuracy. On CIFAR-100 with ResNet-18+GroupNorm, our method (FLITE, Federated Low-rank Iterative Training Engine) communicates 1,280 floats (~5 KB) per client per round -- an 8718x reduction -- and reaches 74.67%, within ~0.5 pp of full-weight FedAvg. The averaging identity holds to floating-point precision ($6 \times 10^{-8}$); the method sits one to two orders of magnitude below PowerSGD and top-k on the bandwidth-accuracy Pareto; it matches or exceeds full-weight FedAvg under strong non-IID skew. int4 latents reach 648 bytes per round at unchanged accuracy, whereas int4 full-weight FedAvg collapses to chance.
Tags
Links
- Source: https://arxiv.org/abs/2607.18343v1
- Canonical: https://arxiv.org/abs/2607.18343v1
Trouble viewing inline? Open PDF directly →
Full Text
51,709 characters extracted from source content.
Expand or collapse full text
11institutetext: Cisco Systems Inc., USA 11email: rachanta,wilreed@cisco.com Federated Lightweight Fine-Tuning Radhakrishna Achanta Will Reed Abstract Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor. We take a different lever. Mapping networks generate a network’s weights from a small trainable latent through a frozen affine projection; because the map is shared and affine, averaging latents is exactly averaging the generated weights. We turn this into a practical low-bandwidth federated channel with two changes: a low-rank, seed-regenerable factorisation of the projection (cutting generator memory from ∼ 8080 GB to ∼ 1010 MB), and a delta formulation θ=θpre+UV⊤zθ=θ^pre+UV z that learns an additive correction around a shared centrally-pretrained base — federated fine-tuning, which is what makes the method work at scale. A frozen orthogonal classifier head further removes the head from the payload while improving accuracy. On CIFAR-100 with ResNet-18+GroupNorm, our method (FLITE, Federated Low-rank Iterative Training Engine) communicates 1,2801,280 floats (≈5≈ 5 KB) per client per round — an 8718×8718× reduction — and reaches 74.67%74.67\%, within ≈0.5≈ 0.5 p of full-weight FedAvg. The averaging identity holds to floating-point precision (6×10−86×10^-8); the method sits one to two orders of magnitude below PowerSGD and top-k on the bandwidth–accuracy Pareto; it matches or exceeds full-weight FedAvg under strong non-IID skew. int4 latents reach 648648 bytes per round at unchanged accuracy, whereas int4 full-weight FedAvg collapses to chance. 1 Introduction Federated and distributed training let many workers improve a shared model without pooling their data [21, 9], but they pay a steep communication cost. The dominant paradigm exchanges model state directly: FedAvg [21] transmits the model itself, and pseudo-gradient schemes such as DiLoCo [5] transmit a full-size weight delta acting as an outer-loop update. Both have per-round bandwidth Θ(|W|) (|W|), the number of model parameters, so for a modern network the message — tens of megabytes for even a small ResNet — dominates the cost of federation. A structural assumption underlies these methods: bandwidth is reduced along the frequency axis — communicating less often — while the size of each message is treated as fixed. Even dedicated gradient-compression methods (low-rank projection [29], top-k sparsification [1], quantization [2]) only shrink the fixed message by a constant factor, still transmitting an object of size proportional to |W||W|, and lowering synchronisation frequency trades amortised cost for client drift, which is most damaging precisely when data are heterogeneous. This paper takes the orthogonal size axis: we keep communication frequent but make each message a tiny, model-independent latent, while preserving the exact FedAvg averaging semantics in weight space. Mapping networks: a different lever. A recent and very different idea, mapping networks, trains a small latent vector z that, through a fixed (frozen) projection, generates the weights of a much larger network. The latent is a compact carrier of model state. Crucially, when the generator is affine and shared, averaging latents is exactly averaging the generated weights — so a federated client could communicate only the small latent while the server aggregation remains ordinary FedAvg. The idea is attractive but, as proposed, has two blockers: (i) the projection matrix is itself huge (on the order of tens of gigabytes for a ResNet-18), and (i) the latent cannot drive a large network when trained from scratch (it collapses to near-chance accuracy). Our approach. FLITE (Federated Low-rank Iterative Training Engine) turns the mapping-network latent into a practical low-bandwidth communication channel by addressing both blockers and re-targeting the method to where it is strong. First, we replace the dense projection with a low-rank, seed-regenerable factorisation, cutting generator memory from ≈80≈ 80 GB to ≈2≈ 2 GB (or ≈10≈ 10 MB if regenerated from a seed) and the regeneration compute correspondingly. Second, rather than asking the latent to encode a whole network, we use it to encode an additive delta around a shared, centrally-pretrained frozen base — federated fine-tuning rather than from-scratch training. This keeps the exact averaging identity, sidesteps the from-scratch failure, and exploits the fact that the correction a well-trained model needs is empirically low-dimensional. Finally, we freeze an orthogonally initialised classifier head, which removes it from the communicated payload and, as an ablation shows, improves accuracy. Results. On CIFAR-100 with ResNet-18+GroupNorm, our latent communicates 1,2801,280 floats (≈5≈ 5 KB) per client per round — an 8718×8718× reduction — and reaches 74.67%74.67\%, within ≈0.5≈ 0.5 p of full-weight FedAvg and statistically at the centralized baseline. The dimension traces a controllable bandwidth–accuracy frontier; the method sits one to two orders of magnitude below PowerSGD and top-k on the Pareto; it matches or exceeds full-weight FedAvg under strong non-IID skew and is stable across client counts. The frozen orthogonal head adds +0.54+0.54 p at zero communication. Being low-dimensional, the latent is highly quantization-robust: int4 latents reach 648648 bytes per round at unchanged accuracy, while int4 full-weight FedAvg collapses to chance. Contributions. • A delta low-rank mapping θ=θpre+UV⊤zθ=θ^pre+UV z that preserves the exact FedAvg averaging identity (verified to 6×10−86×10^-8) while reducing generator memory by ∼ 40×40× (further to ∼ 1010 MB via seed regeneration). • A low-bandwidth federated fine-tuning protocol that transmits a ∼ 55 KB latent per round at parity accuracy, with a favourable bandwidth–accuracy Pareto against PowerSGD and top-k, and improved robustness under heterogeneity and small client counts. • The frozen orthogonal classifier as a free, composable design choice that removes the head from the payload and improves accuracy; int4 quantization-robustness of the latent channel at 648648 bytes per round, where matched-bit full-weight FedAvg collapses. • Negative results showing that mapping networks as a stand-alone training method collapse at ResNet-18 scale regardless of dimension, rank, or warm start, identifying federated fine-tuning as the regime in which the idea does pay off (§5.7). The remainder of the paper presents the method (§3), the setup (§4), results (§5), and concluding remarks on scope and limitations (§6). 2 Related Work Federated averaging and its variants. FedAvg [21] established the dominant paradigm: clients train locally and the server averages their models. Subsequent work improves robustness to heterogeneity and client drift — FedProx [16] adds a proximal term, SCAFFOLD [10] uses control variates, Matched Averaging [30] aligns neurons before averaging, and adaptive server optimisation [25] stabilises aggregation. All of these communicate an object of size Θ(|W|) (|W|) each round; they change how weights are aggregated, not the size of the message. Our method is complementary: it preserves the FedAvg averaging rule exactly but shrinks the message to a latent. We further note a setting distinction: FedAvg, FedProx, and DiLoCo are typically run as from-scratch federated training, with the server initialising θ0 _0 and the federation itself producing the trained model. Our scheme operates in the federated fine-tuning regime, in which every client — ours and the FedAvg baseline alike — starts from the same shared pretrained checkpoint. All bandwidth and accuracy comparisons in §5 use this matched setting, so the contrast is between two fine-tuning protocols at different bandwidths, not between a fine-tuning method and a from-scratch one. Low-frequency and pseudo-gradient methods. Local SGD [27, 12] and DiLoCo [5] reduce communication by synchronising less often, performing many local steps between merges. This lowers amortised cost but transmits a full-size update at each merge and risks client drift, particularly under heterogeneity. Our latent is small enough that frequent synchronisation is affordable, which we show converts into a robustness advantage at a fixed byte budget (§5.5). Communication compression. A large literature compresses the transmitted gradient/update: low-rank projection (PowerSGD [29]), magnitude sparsification (top-k [1], deep gradient compression [20]), and quantization (QSGD [2], signSGD [3]), often with error feedback to preserve convergence [11]. These methods reduce the payload by a constant factor but the compressed object is still derived from, and scales with, |W||W|. We compare directly against PowerSGD and top-k and show our latent is one to two orders of magnitude cheaper at matched accuracy (§5.3); the two approaches are moreover composable (a latent can itself be quantized). Subspace training, low-rank adaptation, and mapping networks. Training in a low-dimensional random subspace is known to suffice for many objectives, an observation formalised by intrinsic-dimension studies [14] and exploited for parameter-efficient fine-tuning by LoRA [8]. Mapping networks [26] take this further, generating a network’s full weights from a low-dimensional latent through a frozen generator whose core is an affine projection. We adopt the affine special case of the mapping-network generator and differ in three ways: (i) we factor the projection into a low-rank, seed-regenerable form (the original materialises a dense projection), removing its prohibitive memory and compute; (i) we operate in delta mode on a shared pretrained base, which is what makes the method work at scale (from-scratch mapping fails, §5.7); and (i) we use it as a federated communication channel, exploiting the exact averaging identity that the affine map admits. Unlike LoRA, our factors are frozen and shared so that averaging latents equals averaging weights — the property federated aggregation requires. Optimization-centric decentralized FL and function-space methods. A parallel line improves decentralized FL through optimisation, topology, and personalisation while remaining in parameter space [32, 13, 28, 18, 4]; their messages still scale with |W||W| and our latent channel is complementary. A distinct line collaborates in function space, exchanging predictions on a shared probe set (FedMD [15], FedDF [19]). These decouple the message from |W||W| and accommodate model heterogeneity, but optimise a different (distillation) objective and are not directly comparable to exact parameter-space averaging. Our channel is weight averaging, carried out exactly in a low-dimensional latent. Frozen classifiers and normalization. Fixing the classifier head is known to be largely harmless [6] and can be beneficial with well-conditioned geometries such as regular-polytope/ETF classifiers [24] motivated by neural collapse [23]; FedBABU [22] freezes the head during federation. We use a frozen orthogonal head as a composable component that removes the classifier from the payload and improves accuracy (§5.6). BatchNorm statistics are not naturally averageable across heterogeneous clients, motivating GroupNorm [31] or local BatchNorm [17] in federated settings [7]; we adopt GroupNorm and accept absolute accuracy below BatchNorm SOTA. 3 Method We first recall the mapping-network generator and the property that makes it attractive for federated averaging (§3.1). We then identify the two obstacles that prevent it from being used directly at the scale of a modern network (§3.2), and present our delta low-rank parameterisation that removes them (§3.3). Finally we describe an optional seed-regenerated projection (§3.4), the frozen orthogonal classifier (§3.5), and the full federated protocol (§3.6). 3.1 Mapping networks and the averaging identity Mapping networks [26] generate the weights of a target layer from a small trainable latent vector z through a fixed (frozen) generator. The original proposal augments this map with modulation and a nonlinearity; we study its affine special case, which is the piece that admits an exact averaging identity. For a layer with P parameters and z∈ℝdz ^d, d≪Pd P, θ=b+Wmz,θ\;=\;b\;+\;W_m\,z, (1) where b∈ℝPb ^P is a fixed base vector and Wm∈ℝP×dW_m ^P× d is a fixed projection; only z is trained. In the original formulation WmW_m is a dense random matrix, materialised in full. Because the map z↦θz θ is affine and b,Wmb,W_m are shared across all clients, averaging latents is identical to averaging the generated weights: 1K∑k=1K(b+Wmzk)=b+Wm(1K∑k=1Kzk). 1K _k=1^K (b+W_m\,z_k )\;=\;b+W_m\! ( 1K _k=1^Kz_k ). (2) This is exactly the FedAvg update, carried out in the d-dimensional latent space instead of the P-dimensional weight space. A client therefore needs to transmit only z (d floats) rather than θ (P floats), and the server aggregation is unchanged. Dropping the modulation term and any post-map nonlinearity is thus a deliberate design choice: it preserves Eq. (2) by construction. 3.2 Two obstacles at scale Memory and compute. In the original mapping network the projection WmW_m is a dense matrix of shape P×dP× d per layer. For a ResNet-18 (P≈11.2P≈ 11.2M aggregated over layers) even a modest d makes WmW_m enormous: materialising the dense projections requires on the order of 8080 GB in fp32, far exceeding a single accelerator, and every weight regeneration is a dense P×dP× d matrix–vector product. The latent is small, but the dense generator that expands it is not. From-scratch training. When b is a random initialisation and the latent must encode the entire network, optimisation collapses at ResNet-18 scale: training the latent from a Kaiming-initialised base reaches only ≈2.5%≈ 2.5\% test accuracy on CIFAR-100 (§5). The latent does not have the capacity to drive a large network from scratch. 3.3 Delta low-rank mapping We address both obstacles with a single change of parameterisation. Rather than generating each layer’s weights from scratch with a dense projection, we (i) generate an additive delta on top of a frozen, centrally-pretrained backbone, and (i) replace the dense WmW_m with a low-rank factorisation Wm=UV⊤W_m=UV of rank r≪dr d. The low-rank factorisation is not part of the original mapping network — which materialises WmW_m densely — but is introduced here specifically to cut both the storage of the generator and, by consequence, the cost of every weight-regeneration matrix–vector product (§3.2); it is also what makes the small-d regime that drives the per-round payload down practical. For each mapped layer l, θl=θlpre⏟frozen base+UlVl⊤zl,Ul∈ℝPl×r,Vl∈ℝdl×r, _l\;=\; θ^pre_l_frozen base\;+\;U_l\,V_l \,z_l, U_l ^P_l× r,\;V_l ^d_l× r, (3) where θlpreθ^pre_l is the (flattened) pretrained weight of layer l, Ul,VlU_l,V_l are frozen factors with orthonormal columns, and only the per-layer latent zl∈ℝdlz_l ^d_l is trained. The effective projection is Wm(l)=UlVl⊤W_m^(l)=U_lV_l of rank at most r. We initialise the latent at zero (zl=0z_l=0), so the network at the start of mapping is exactly the pretrained model; the latent only learns refinements. The averaging identity still holds exactly. Equation (3) is affine in zlz_l with a shared frozen offset θlpreθ^pre_l and shared frozen factors Ul,VlU_l,V_l. The same algebra as Eq. (2) applies per layer, and the pretrained offset cancels in the mean: 1K∑k(θlpre+UlVl⊤zl,k)=θlpre+UlVl⊤(1K∑kzl,k). 1K _k (θ^pre_l+U_lV_l z_l,k )=θ^pre_l+U_lV_l \! ( 1K _kz_l,k ). (4) We verify this numerically end-to-end: the maximum discrepancy between averaging latents and averaging the generated weights is 5.96×10−85.96× 10^-8, i.e. floating-point round-off. Memory and compute. The factored projection costs r(Pl+dl)r\,(P_l+d_l) parameters per layer instead of PldlP_l\,d_l, and the regeneration becomes two small matrix–vector products (Vl⊤zlV_l z_l then Ul(⋅)U_l(·)) costing (r(Pl+dl))O (r(P_l+d_l) ) rather than (Pldl)O(P_ld_l) flops. Aggregated over ResNet-18 at r=32r=32 the storage is ≈2≈ 2 GB rather than ≈80≈ 80 GB — a reduction of more than an order of magnitude that brings the generator onto a single device. Crucially, the low rank does not cost accuracy in delta mode: varying r∈8,32,128,256r∈\8,32,128,256\ changes test accuracy by <0.2<0.2 p (§5), because the delta a well-pretrained model needs is itself low-dimensional. Why delta mode resolves the from-scratch failure. The latent no longer has to represent the whole network, only the small correction around a strong base. This both makes optimisation tractable and explains why a tiny latent suffices: with dl=64d_l=64, r=32r=32 the entire ResNet-18 is steered by 1,2801,280 trainable floats (20 mapped layers × 64×\,64), an 8718×8718× reduction relative to the 11.211.2M parameters, while recovering the pretrained accuracy (§5). 3.4 Seed-regenerated projection (optional) The factors Ul,VlU_l,V_l are deterministic functions of a shared random seed (Gaussian draws orthonormalised by QR): a single 6464-bit seed regenerates them identically on every client, preserving Eq. (4) exactly. This trades recompute for a further memory reduction, from ≈2≈ 2 GB of stored factors to ≈10≈ 10 MB of peak working memory. 3.5 Frozen orthogonal classifier We initialise the final classifier (the fully-connected head) with an orthogonal weight matrix and freeze it. This removes the classifier from the set of mapped/communicated parameters entirely, and — as an ablation confirms (§5) — is accuracy-positive rather than merely neutral: on CIFAR-100 the frozen orthogonal head reaches 73.53%73.53\% versus 72.99%72.99\% for a trainable head and 72.95%72.95\% for a frozen Kaiming head. An orthonormal, fixed set of class prototypes gives a well-conditioned target geometry for the backbone to align to, while contributing zero communication. 3.6 Federated protocol The complete scheme is a drop-in replacement for FedAvg in which the per-round payload is the latent rather than the model. The server broadcasts the frozen pretrained base θpreθ^pre and the seed once; thereafter each round transmits only z (Algorithm 1). With K clients and D=∑ldlD= _ld_l latents, one round transfers 2KD⋅42KD· 4 bytes: ≈5≈ 5 KB per client per direction for ResNet-18 with D=1,280D=1,280, against ≈45≈ 45 MB for the full 11.211.2M-parameter model. Algorithm 1 FLITE federated round with optional latent quantization. All clients cache θpreθ^pre and factors Ul,Vl\U_l,V_l\ (or the shared seed that regenerates them, §3.4); factors live on the mapped weight tensors only. Non-mapped parameters (FC head, biases, normalization) remain frozen. The generator θl(zk)=θlpre+UlVl⊤zk,l _l(z_k)=θ^pre_l+U_lV_l z_k,l is re-evaluated on every forward/backward pass; initialise z¯←0 z← 0. Aggregation is the unweighted mean; sample-weighted pk∝nkp_k n_k preserves the identity too. 1:Server broadcasts z¯ z to clients k∈k . 2:for each client k∈k in parallel do 3: Set zk←z¯z_k← z. 4: Train zkz_k for E local epochs on client data kD_k. 5: (Optional) Quantize zkz_k to q-bit integers (uniform-affine, per-tensor scale). 6: Upload zkz_k (4D4D bytes at fp32, qD/8qD/8 at q-bit; D is the total latent size). 7:end for 8:Server takes the unweighted mean of the (dequantized) uploads. 4 Experimental Setup Models and datasets. Our primary setting is ResNet-18 with GroupNorm (≈11.2≈ 11.2M parameters) on CIFAR-100. We additionally evaluate ResNet-18+GN on CIFAR-10 and, to test scaling in model and label space, ResNet-34 with GroupNorm (≈21.3≈ 21.3M parameters) on TinyImageNet (200 classes). To probe a non-vision modality we additionally run a compact TinyGPT (66 layers, 384384-dim embeddings, ≈11.5≈ 11.5M trainable parameters) on WikiText-2 language modelling. We use GroupNorm rather than BatchNorm throughout because BatchNorm running statistics are not naturally averageable across clients in federated learning, whereas GroupNorm carries no such buffers [31, 7]. We note up front that GroupNorm places the absolute accuracies somewhat below BatchNorm SOTA; our claims concern relative parity at greatly reduced bandwidth, not absolute state of the art. Centrally-pretrained base. All methods share a single centrally pretrained backbone, trained with SGD (momentum 0.90.9, learning rate 0.050.05 with a 33-epoch warmup and cosine decay, weight decay 5×10−45× 10^-4, batch size 128128) for 100100 epochs, with the orthogonal classifier head frozen from initialisation (§3.5). The base reaches 94.4%94.4\% on CIFAR-10, 74.6%74.6\% on CIFAR-100, and 62.5%62.5\% on TinyImageNet. The TinyGPT base reaches validation perplexity 66.166.1 on WikiText-2. This base is broadcast once at the start of federation; thereafter only latents (ours) or the corresponding per-method payload (baselines) are communicated. Federated configuration. We simulate K clients (K∈4,8,16,32K∈\4,8,16,32\, default K=8K=8) over R communication rounds with E local epochs per round. Client data is partitioned either IID (uniform random) or non-IID via a Dirichlet split with concentration α∈0.5,0.1α∈\0.5,0.1\ (smaller α = more heterogeneous). Each round, clients start from the global state, train locally, and upload their payload for averaging (Algorithm 1). The averaging identity is re-checked at every run. Methods compared. On a single bandwidth–accuracy axis we compare: • Latent (ours): delta low-rank mapping, default latent dimension d=64d=64 per layer and rank r=32r=32 on CIFAR-100 (D=1,280D=1,280 latents total), with a d/rd/r sweep for the Pareto curve. Trained with Adam (lr 10−210^-2, weight decay 10−410^-4, gradient clip 1.01.0) on the latents only. • Full-weight FedAvg [21]: the standard baseline, communicating all trainable weights. • PowerSGD [29]: low-rank gradient compression with rank ∈2,8∈\2,8\ and error feedback. • Top-k [1]: magnitude sparsification of the update at densities ∈1%,10%∈\1\%,10\%\ with error feedback. Except for the head-to-head study in §5.2, all baselines share the same frozen pretrained base and frozen orthogonal head, so differences reflect the communication scheme alone; the head-to-head study additionally reports FedAvg with a trainable Kaiming head to isolate the head’s contribution. Communication accounting. We report cumulative bytes per client, counting both upload and broadcast directions (fp32 = 44 bytes). For top-k we count transmitted values and their integer indices; for PowerSGD we count both transmitted factors. The per-round latent payload is D⋅4D· 4 bytes (≈5≈ 5 KB for D=1,280D=1,280); the per-round full-weight payload is ≈45≈ 45 MB.111We report bytes, not wall-clock time: the experiments are a faithful federated simulation with exact byte accounting, but we make no latency or system-throughput claims. Ablations and protocol. The frozen-classifier ablation (§3.5) compares a trainable head, a frozen Kaiming-initialised head, and our frozen orthogonal head, each over 33 seeds. Unless stated otherwise, all federated results are reported over 33 seeds (mean ± standard deviation). Pretraining and mapping both checkpoint per epoch and are resumable, and every run logs the latent-averaging identity error as a correctness check. 5 Results We organise the results around the three claims of §1: massive bandwidth reduction at parity accuracy (§5.1–5.3), robustness under heterogeneity and client count (§5.4–5.5), and the supporting design choices (§5.6–5.7). All federated numbers are over 33 seeds (mean ± std) unless noted. Additional plots (fixed-byte budget, K-sweep, quantization Pareto, WikiText-2 language modelling) and the full appendix appear in Appendix 0.A–0.G. 5.1 Federated fidelity at extreme compression Table 1 reports the headline federated result on CIFAR-100 (ResNet-18+GN, K=8K=8, IID). Communicating only the latent — 1,2801,280 floats (≈5≈ 5 KB) per client per round, an 8718×8718× reduction relative to the 11.211.2M-parameter model — our method reaches 74.67±0.01%74.67± 0.01\%, within ≈0.5≈ 0.5 p of full-weight FedAvg (75.16±0.15%75.16± 0.15\%) and statistically indistinguishable from the centrally pretrained ceiling (74.6%74.6\%). The latent carries essentially all of the federated signal at a per-round payload four orders of magnitude smaller than the model. Table 1: Federated CIFAR-100 (ResNet-18+GN, K=8K=8, IID, 33 seeds). Our latent exchange matches full-weight FedAvg within seed noise at 8718×8718× fewer transmitted parameters per round. Method Test acc. (%) Payload/round/client Compression Full-weight FedAvg 75.16±0.1575.16± 0.15 ≈45≈ 45 MB 1×1× Latent (ours, d=64d=64) 74.67±0.0174.67± 0.01 ≈5≈ 5 KB ×8718× Centralized base (ceiling) 74.674.6 — — 5.2 Head-to-head: ours vs textbook FedAvg Table 2 and Fig. 1 summarise the central comparison across both datasets and partitions: our full scheme (frozen orthogonal head ++ latent) against textbook FedAvg (trainable Kaiming head, full-weight averaging; the only experiment where the FedAvg baseline uses a trainable Kaiming head — elsewhere the head is matched to isolate the communication channel). Under IID data our method is within seed noise of FedAvg (−0.25-0.25 p on CIFAR-10, −0.23-0.23 p on CIFAR-100) at 540540–8700×8700× lower per-round bandwidth. Under strong heterogeneity (α=0.1α=0.1) the comparison reverses: we are +0.39+0.39 p ahead on CIFAR-10 and +0.97+0.97 p ahead on CIFAR-100. Table 2: Ours (frozen orthogonal FC head + latent communication) vs. textbook FedAvg (trainable Kaiming FC, full-weight averaging). Best test accuracy (%) (mean over 3 seeds); Δ= = ours −- FedAvg (positive favours ours). Dataset / partition Latent (ours) Textbook FedAvg Δ CIFAR-10, IID 94.23± 0.02 94.47± 0.14 -0.25 CIFAR-10, α=0.1α=0.1 94.10± 0.04 93.71± 0.16 +0.39 CIFAR-100, IID 74.74± 0.01 74.97± 0.15 -0.23 CIFAR-100, α=0.1α=0.1 74.72± 0.05 73.75± 0.18 +0.97 Figure 1: Our scheme (frozen-orthogonal head ++ latent) vs textbook FedAvg (trainable head, full-weight). Parity under IID, a clear advantage under non-IID (α=0.1α=0.1), at a fraction of the bandwidth. 5.3 Bandwidth–accuracy Pareto Figure 2 plots test accuracy against cumulative communication. On CIFAR-100 our latent points occupy the bottom-left of the plot — the entire trajectory lives below ≈2≈ 2 MB cumulative, while full-weight FedAvg requires 10310^3–10410^4 MB to reach comparable accuracy. The latent dimension d traces a controllable frontier (Table 3): from d=64d=64 (8718×8718×, 74.67%74.67\%) up to d=1024d=1024 (545×545×, 74.66%74.66\%), accuracy is essentially flat, confirming that the correction a pretrained model needs is genuinely small. On CIFAR-10 the same picture holds against the stronger compression baselines: PowerSGD and top-k reduce full-weight cost by 4040–140×140×, but our latent is a further 11–22 orders of magnitude cheaper at matched accuracy. Figure 2: Bandwidth–accuracy Pareto. Left: CIFAR-100, latent (ours) vs full-weight FedAvg. Right: CIFAR-10, vs full-weight, PowerSGD (r∈2,8r∈\2,8\) and top-k (1%,10%\1\%,10\%\). Our latent occupies the low-bandwidth frontier. Table 3: Latent-dimension sweep, federated CIFAR-100 (K=8K=8, IID). Accuracy is flat across two orders of magnitude of per-round payload. Config Latents D Payload/round Compression Test acc. (%) Latent d=64d=64 1,2801,280 ≈5≈ 5 KB 8718×8718× 74.67±0.0174.67± 0.01 Latent d=256d=256 5,1205,120 ≈20≈ 20 KB 2180×2180× 74.6674.66 Latent d=1024d=1024 20,48020,480 ≈82≈ 82 KB 545×545× 74.6674.66 Full-weight FedAvg 11.211.2M ≈45≈ 45 MB 1×1× 75.16±0.1575.16± 0.15 5.4 Robustness to non-IID data Figure 3 compares our method with full-weight FedAvg on CIFAR-10 across IID and Dirichlet α∈0.5,0.1α∈\0.5,0.1\ splits. The gap to full-weight, already small under IID (−0.2-0.2 p), vanishes at α=0.5α=0.5 and reverses under the strongest skew: at α=0.1α=0.1 our latent reaches 94.07±0.07%94.07± 0.07\% versus full-weight 93.79±0.18%93.79± 0.18\% (+0.28+0.28 p). Because the latent moves in a low-dimensional shared subspace anchored at the pretrained base, client updates are constrained and average more stably than full-weight updates — exactly the regime where full-weight FedAvg suffers most from client drift. Figure 3: Non-IID robustness on CIFAR-10. Our latent matches full-weight FedAvg at α=0.5α=0.5 and exceeds it under strong heterogeneity (α=0.1α=0.1), at ≈545×≈ 545× lower per-round bandwidth. 5.5 Fixed-byte budget and client-count sensitivity Two further axes support the same picture (details in Appendix 0.A): under a fixed 200200 MB per-client budget on CIFAR-10 (α=0.5α=0.5), our method completes ∼ 150150 latent rounds and reaches 94.21%94.21\% while full-weight FedAvg can afford one round and attains 93.48%93.48\% (+0.73+0.73 p for ours). Sweeping K∈4,8,16,32K∈\4,8,16,32\ on CIFAR-100, our latent accuracy is essentially flat in K, whereas full-weight under α=0.5α=0.5 degrades to 74.2%74.2\% at K=4K=4 before recovering at large K — a further symptom of drift that our low-dimensional channel avoids. 5.6 Design ablations Frozen orthogonal classifier. On CIFAR-100 (3 seeds), freezing a Kaiming-initialised head is accuracy-neutral (72.95%72.95\% vs trainable 72.99%72.99\%), but freezing an orthogonal head improves accuracy to 73.53%73.53\% (+0.54+0.54 p) while removing it from the payload. A fixed, well-conditioned set of class prototypes gives the backbone a clean target geometry to align to at zero communication cost. Rank irrelevance under delta mode. Consistent with §3.3, the projection rank r has little effect on accuracy in delta mode: varying r∈8,32,128,256r∈\8,32,128,256\ at d=256d=256 on CIFAR-100 moves test accuracy by <0.2<0.2 p. Rank therefore trades only generator memory and compute, not accuracy — justifying the small r=32r=32 used throughout. 5.7 Negative result: mapping networks do not scale as stand-alone training The original mapping-network proposal targets stand-alone training: a tiny trainable latent, generated through a frozen random projection, replaces direct weight optimisation. We find this does not extend beyond small architectures. At ResNet-18 scale on CIFAR-100, training the latent from a randomly initialised base collapses to ≈2.5%≈ 2.5\% test accuracy, against 74.7%74.7\% that direct SGD reaches and that our delta-mode latent recovers. The collapse is robust: it persists across latent dimensions d∈1024,4096,16384d∈\1024,4096,16384\, projection ranks r∈8,32,128,256r∈\8,32,128,256\, and warm-start variants including PCANet-initialised bases, short SGD warm-ups, and least-squares latent initialisation from a converged teacher. The latent has ample capacity to encode a correction around a strong base, but not to drive a large network from scratch. This is the negative result that shapes our framing: rather than competing with direct training, mapping networks are useful precisely as a low-bandwidth communication channel between models that have already been trained. 5.8 Scaling to TinyImageNet To test whether the trade-off holds at larger scale, we federate a ResNet-34+GN backbone (≈21.3≈ 21.3M parameters, 200200 classes) on TinyImageNet with d=1024d=1024 (K=8K=8, R=20R=20, 22 local epochs). Under IID partitioning, the latent channel matches full-weight FedAvg within 0.20.2 p (62.14±0.08%62.14± 0.08\% vs 62.34±0.11%62.34± 0.11\%) while transmitting ≈144≈ 144 KB per client per round vs ≈81≈ 81 MB (≈577×≈ 577× less). Under non-IID data (α=0.5α=0.5), the latent path is more accurate (62.22±0.10%62.22± 0.10\% vs 60.79±0.19%60.79± 0.19\%, +1.43+1.43 p), echoing the CIFAR head-to-head trend at a larger scale (Table 4). Table 4: TinyImageNet (ResNet-34+GN, d=1024d=1024, K=8K=8, R=20R=20): latent vs full-weight FedAvg. Best test accuracy (%) (mean over 3 seeds); Δ= = latent −- full-weight. Per-round payload: latent 144 KB vs full-weight 81.2 MB. Partition Latent (ours) Full-weight FedAvg Δ IID 62.14± 0.08 62.34± 0.11 -0.20 non-IID (α=0.5α=0.5) 62.22± 0.10 60.79± 0.19 +1.42 5.9 Integer-quantized latents The latent is a short vector, so it composes directly with standard payload compression. We apply uniform-affine integer quantization to the uploaded latent and compare against quantizing the full-weight update at matched bit-widths (Table 5; see Fig. 7 in the appendix for the accuracy-vs-precision plot). On CIFAR-100, quantizing our latent to int8 or int4 leaves accuracy essentially unchanged (74.73±0.02%74.73± 0.02\% at int4 vs 74.72±0.03%74.72± 0.03\% at fp32, 33 seeds), shrinking the payload to 648648 bytes per client per round — a 68,943×68,943× reduction relative to the fp32 full-weight model. By contrast, textbook full-weight FedAvg tolerates int8 but collapses at int4 (to 1.11±0.02%1.11± 0.02\%, i.e. chance), because 44-bit quantization of 11.211.2M weights injects far more error than averaging can absorb. Quantization and our latent channel are complementary and together push the per-round payload below 11 KB at full accuracy. Table 5: CIFAR-100 (d=64d=64) matched-bit-width communication ladder. Payload is per client per round; best test accuracy (%) (mean over 3 seeds). Our latent scheme is unaffected by 8/4-bit quantization, while textbook full-weight FedAvg collapses at int4 († ). Method Bits Payload/client Best acc. (%) n Latent (ours) fp32 5.0 KB 74.72± 0.03 3 Latent (ours) int8 1.3 KB 74.74± 0.05 3 Latent (ours) int4 648.0 B 74.73± 0.02 3 Full-weight FedAvg fp32 42.6 MB 75.12± 0.09 3 Full-weight FedAvg int8 10.7 MB 74.80± 0.17 3 Full-weight FedAvg int4 5.3 MB 1.11± 0.02 † 3 6 Conclusion Why it helps under heterogeneity. Latent updates live in a low-dimensional subspace anchored at the shared base, so client updates are implicitly constrained and average more stably than full-weight updates. The gap to full-weight FedAvg thus closes and then reverses as data become more non-IID (§5.4); because each round is cheap, the method can also synchronise often within a fixed byte budget, directly attacking drift (§5.5). The empirical flatness of accuracy in both d (Table 3) and r (§5.6) is consistent with adapting a well-pretrained network being intrinsically low-dimensional [14]: the latent encodes only a correction around a strong base — which is also why the from-scratch variant fails (§5.7). Composability. The frozen orthogonal head, seed regeneration, and latent are modular: any can be dropped without breaking the others, and the latent itself composes with standard payload compression — integer quantization pushes the payload to 648648 B/round at parity, where matched-bit full-weight FedAvg collapses (§5.9). The shared centrally-pretrained base fits current federated practice, in that many practical deployments adapt off public checkpoints rather than train from scratch. Limitations. (i) GroupNorm places absolute accuracy below BatchNorm SOTA; our claims are relative parity at massive compression, not absolute state of the art. (i) The method is federated fine-tuning: it requires a shared centrally-pretrained base and a one-time broadcast of that base and the seed. (i) Our evaluation is a federated simulation with exact byte accounting; we make no wall-clock claims. A preliminary autoregressive language-modelling extension is in Appendix 0.B. Code. The code will be released publicly upon acceptance of the paper. References [1] A. F. Aji and K. Heafield (2017) Sparse communication for distributed gradient descent. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 440–445. Cited by: §1, §2, 4th item. [2] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic (2017) QSGD: communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems 30 (NeurIPS), Cited by: §1, §2. [3] J. Bernstein, Y. Wang, K. Azizzadenesheli, and A. Anandkumar (2018) signSGD: compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning (ICML), Cited by: §2. [4] S. Choudhary, S. A. Aketi, and K. Roy (2026) Achieving global flatness in decentralized learning with heterogeneous data. Transactions on Machine Learning Research. Note: Accepted by TMLR Cited by: §2. [5] A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen (2023) DiLoCo: distributed low-communication training of language models. arXiv preprint arXiv:2311.08105. Cited by: §1, §2. [6] E. Hoffer, I. Hubara, and D. Soudry (2018) Fix your classifier: the marginal value of training the last weight layer. In International Conference on Learning Representations, Cited by: §2. [7] K. Hsieh, A. Phanishayee, O. Mutlu, and P. B. Gibbons (2020) The non-IID data quagmire of decentralized machine learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), Cited by: §2, §4. [8] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §2. [9] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. (2019) Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977. Cited by: §1. [10] S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh (2019) SCAFFOLD: stochastic controlled averaging for on-device federated learning. arXiv preprint arXiv:1910.06378. Cited by: §2. [11] S. P. Karimireddy, Q. Rebjock, S. U. Stich, and M. Jaggi (2019) Error feedback fixes SignSGD and other gradient compression schemes. In Proceedings of the 36th International Conference on Machine Learning (ICML), Cited by: §2. [12] A. Khaled, K. Mishchenko, and P. Richtárik (2020) Tighter theory for local SGD on identical and heterogeneous data. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, Vol. 108, p. 4519–4529. Cited by: §2. [13] S. Kharrat, M. Canini, and S. Horváth (2025) DPFL: decentralized personalized federated learning. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 258, p. 5086–5094. Cited by: §2. [14] C. Li, H. Farkhoor, R. Liu, and J. Yosinski (2018) Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, Cited by: §2, §6. [15] D. Li and J. Wang (2019) FedMD: heterogeneous federated learning via model distillation. arXiv preprint arXiv:1910.03581. Note: NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality Cited by: §2. [16] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2018) Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127. Cited by: §2. [17] X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou (2021) FedBN: federated learning on non-IID features via local batch normalization. In International Conference on Learning Representations, Cited by: §2. [18] I. Lin, O. Yagan, and C. Joe-Wong (2025) FedSPD: a soft-clustering approach for personalized decentralized federated learning. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, Proceedings of Machine Learning Research, Vol. 286, p. 2618–2641. Cited by: §2. [19] T. Lin, L. Kong, S. U. Stich, and M. Jaggi (2020) Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242. Cited by: §2. [20] Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally (2018) Deep gradient compression: reducing the communication bandwidth for distributed training. In International Conference on Learning Representations, Cited by: §2. [21] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research, Vol. 54, p. 1273–1282. Cited by: §1, §2, 2nd item. [22] J. Oh, S. Kim, and S. Yun (2022) FedBABU: toward enhanced representation for federated image classification. In International Conference on Learning Representations, Cited by: §2. [23] V. Papyan, X. Y. Han, and D. L. Donoho (2020) Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences 117 (40), p. 24652–24663. Cited by: §2. [24] F. Pernici, M. Bruni, C. Baecchi, and A. Del Bimbo (2021) Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §2. [25] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan (2020) Adaptive federated optimization. arXiv preprint arXiv:2003.00295. Cited by: §2. [26] L. Sen and S. Mukherjee (2026) Mapping networks: generating network weights from low-dimensional latents. arXiv preprint arXiv:2602.19134. Cited by: §2, §3.1. [27] S. U. Stich (2019) Local SGD converges fast and communicates little. In International Conference on Learning Representations, Cited by: §2. [28] G. Thompson, K. Yue, C. Wong, and H. Dai (2025) NTK-dfl: enhancing decentralized federated learning in heterogeneous settings via neural tangent kernel. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 59470–59491. Cited by: §2. [29] T. Vogels, S. P. Karimireddy, and M. Jaggi (2019) PowerSGD: practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems 32 (NeurIPS), Cited by: §1, §2, 3rd item. [30] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni (2020) Federated learning with matched averaging. In International Conference on Learning Representations, Cited by: §2. [31] Y. Wu and K. He (2018) Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), Cited by: §2, §4. [32] S. Zehtabi, D. Han, R. Parasnis, S. Hosseinalipour, and C. Brinton (2025) Decentralized sporadic federated learning: a unified algorithmic framework with convergence guarantees. In International Conference on Learning Representations, Note: Spotlight Cited by: §2. Appendix 0.A Additional experimental figures This appendix collects extra figures referenced by the main results (§5) but not shown there for length reasons. 0.A.1 Fixed-byte budget Under a fixed 200200 MB per-client budget on CIFAR-10 (α=0.5α=0.5), our latent completes ∼ 150150 rounds and reaches 94.21%94.21\% while full-weight FedAvg can afford one round and attains 93.48%93.48\% (+0.73+0.73 p for ours, Fig. 4). This is the fixed-budget view of §5.5. Figure 4: Fixed 200200 MB/client budget (CIFAR-10, α=0.5α=0.5): many cheap latent rounds vs one expensive full-weight round. 0.A.2 Client-count sweep Figure 5 sweeps K∈4,8,16,32K∈\4,8,16,32\ on CIFAR-100. Latent accuracy is essentially flat in K (and the per-round compression ratio is K-invariant by construction). Full-weight FedAvg under α=0.5α=0.5 degrades at small K (down to 74.2%74.2\% at K=4K=4) and only recovers as K grows, while our method is stable across the range. This complements the client-count summary in §5.5. Figure 5: Scaling in client count K (CIFAR-100). Latent accuracy is stable in K; full-weight is more sensitive at small K. 0.A.3 Frozen orthogonal head figure Figure 6 visualises the classifier-head ablation cited in §5.6. The frozen orthogonal head improves accuracy over both a trainable and a frozen Kaiming head, at zero communication cost. Figure 6: Classifier-head ablation on CIFAR-100 (frozen orthogonal vs. trainable vs. frozen Kaiming). 0.A.4 Quantization precision plot Figure 7 shows accuracy vs. communicated precision on CIFAR-100 (d=64d=64), the plot complement of Table 5 in §5.9. Our latent is flat across fp32/int8/int4; full-weight FedAvg collapses at int4. Figure 7: Latent quantization (CIFAR-100, d=64d=64): accuracy vs communicated precision. 0.A.5 Quantization–bandwidth Pareto Combining the latent channel with integer quantization pushes the per-round payload below 11 KB at parity accuracy (Fig. 8), extending the Pareto discussion of §5.3. Figure 8: Latent-quantization bandwidth–accuracy points: int4 latent reaches 648648 B/round at parity accuracy. Appendix 0.B Language modelling on WikiText-2 Beyond vision, we apply FLITE to federated language-model fine-tuning on WikiText-2 with the TinyGPT backbone (d=1024d=1024, K=8K=8, R=20R=20, 11 local epoch, IID split; 33 seeds). Table 6 and Fig. 9 report validation perplexity (lower is better). The headline result here is communication: our latent channel transmits ≈144≈ 144 KB per client per round versus ≈44≈ 44 MB for full-weight FedAvg — a ≈313×≈ 313× reduction — while holding validation perplexity essentially at the shared pretrained base (66.0966.09 throughout federation). By contrast, full-weight FedAvg can briefly improve perplexity in early rounds (61.71±0.0361.71± 0.03 best) but then diverges as heterogeneous local updates are averaged (81.06±0.0981.06± 0.09 final). We therefore treat WikiText-2 primarily as evidence that the kilobyte-scale payload transfers to autoregressive LMs; accuracy parity on this task is left to future tuning of local steps and learning rates. Table 6: WikiText-2 (TinyGPT, d=1024d=1024, K=8K=8, R=20R=20, IID). Per-round payload and validation perplexity (mean over 3 seeds). Latent payload is 313×313× smaller. Lower perplexity is better. Method Payload/client Best val PPL Final val PPL Latent (ours) 144 KB 66.09± 0.00 66.10± 0.00 Full-weight FedAvg 44.0 MB 61.71± 0.03 81.06± 0.09 Figure 9: WikiText-2 (TinyGPT): validation perplexity vs. communication round. Shaded bands are ±1± 1 std over 33 seeds. Latent federation is stable at the pretrained base; full-weight FedAvg improves briefly then drifts. Appendix 0.C Hyperparameters and training recipes Centralized pretraining (shared base). SGD with momentum 0.90.9, initial learning rate 0.050.05, 33-epoch linear warmup then cosine decay to zero, weight decay 5×10−45× 10^-4, batch size 128128, 100100 epochs, standard crop+flip augmentation. GroupNorm with 3232 groups. The classifier head is orthogonally initialised and frozen from the first step. Per-epoch checkpointing enables resume. Latent (mapping) training. Adam, learning rate 10−210^-2, weight decay 10−410^-4, gradient clipping at 1.01.0, latents initialised at zero (so the model starts exactly at the pretrained base). Default d=64d=64, r=32r=32 on CIFAR-100. Backbone, projection factors U,VU,V, and the classifier head are all frozen; only the per-layer latents zlz_l are trained. Federated configuration. K clients (K∈4,8,16,32K∈\4,8,16,32\, default 88), E local epochs per round, IID or Dirichlet(α) partition with α∈0.5,0.1α∈\0.5,0.1\. Server averaging is the unweighted mean of client latents (ours) or client payloads (baselines). Baselines: full-weight FedAvg; PowerSGD rank ∈2,8∈\2,8\ with error feedback; top-k density ∈1%,10%∈\1\%,10\%\ with error feedback. Except for the head-to-head study in the main paper, all methods share the same frozen pretrained base and the same frozen orthogonal head. Appendix 0.D Generator memory analysis The original mapping-network generator stores a dense projection WmW_m of shape Pl×dlP_l× d_l for each mapped layer. Aggregated over ResNet-18 (∑lPl≈11.2 _lP_l≈ 11.2M), even a modest per-layer dld_l makes the dense projections require on the order of 8080 GB in fp32, exceeding a single accelerator. Our low-rank factorisation Wm=UV⊤W_m=UV stores r(Pl+dl)r\,(P_l+d_l) values per layer instead of PldlP_l\,d_l; at r=32r=32 this is ≈2≈ 2 GB. Seed regeneration stores neither factor — a single 6464-bit seed deterministically regenerates Ul,VlU_l,V_l (Gaussian draw + QR) identically on every client — so only ≈10≈ 10 MB of peak working memory is needed during regeneration. Figure 10 compares these storage options. The exact averaging identity is preserved in all variants because Ul,VlU_l,V_l are identical across clients (same seed ⇒ same factors). Figure 10: Generator storage for ResNet-18: dense (≈80≈ 80 GB) vs low-rank (≈2≈ 2 GB) vs seed-regenerated (≈10≈ 10 MB working memory). Appendix 0.E From-scratch failure at scale Training the latent from a randomly (Kaiming) initialised base, with no pretrained offset, collapses at ResNet-18 scale: on CIFAR-100 it reaches only ≈2.5%≈ 2.5\% test accuracy, against 74.7%74.7\% in delta mode under otherwise identical settings. The latent has the capacity to encode a correction around a strong base, but not to drive a large network from scratch; this motivates the delta-on-pretrained framing and our positioning of the method as low-bandwidth federated fine-tuning. Appendix 0.F Rank sweep (rank irrelevance under delta mode) At fixed d=256d=256 on CIFAR-100, varying the projection rank r∈8,32,128,256r∈\8,32,128,256\ changes test accuracy by <0.2<0.2 p. The rank therefore trades only generator memory and regeneration compute, not accuracy, consistent with the delta a well-pretrained model needs being itself low-dimensional. We use r=32r=32 throughout. Appendix 0.G Reproducibility Every run logs the latent-averaging identity error (maximum discrepancy between averaging latents and averaging the generated weights), which is at the level of floating-point round-off (≤6×10−8≤ 6× 10^-8) in all experiments. Pretraining and mapping checkpoint per epoch and are resumable. Communication is accounted in bytes (both directions, fp32), counting transmitted values and indices for top-k and both factors for PowerSGD. Code and configuration files will be released.