Paper deep dive
Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs
Dario Fenoglio, Gabriele Dominici, Martin Gjoreski, Marc Langheinrich
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0$\times$. In federated classification, DSL reaches 96.83% on MNIST with 8.9$\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.
Tags
Links
- Source: https://arxiv.org/abs/2610.05378v1
- Canonical: https://arxiv.org/abs/2610.05378v1
Trouble viewing inline? Open PDF directly →
Full Text
71,197 characters extracted from source content.
Expand or collapse full text
Collaborative, Open, and DECentralized training of Foundation Models (CODEC-FM) Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs Dario Fenoglio Affiliation: Università della Svizzera italiana (USI), Lugano, Switzerland Affiliation: MIT Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA*Corresponding author: dario.fenoglio@usi.ch Gabriele Dominici Affiliation: Università della Svizzera italiana (USI), Lugano, Switzerland Martin Gjoreski Affiliation: Università della Svizzera italiana (USI), Lugano, Switzerland Marc Langheinrich Affiliation: Università della Svizzera italiana (USI), Lugano, Switzerland Abstract Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56%94.56\% vs. 87.76%87.76\%) and a larger GSM8K gain over the base model (22.022.0 vs. 0.60.6 points), while reducing upload by 30.630.6–49.0×49.0×. In federated classification, DSL reaches 96.83%96.83\% on MNIST with 8.9×8.9× less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms. Figure 1: Distributed Subliminal Learning. Participants adapt a common base model on local data and probe it with task-unrelated random carriers. Instead of model updates or adapters, participants transmit only carrier outputs, illustrated here with random-number completions. The coordinator pools these messages and distills a shared model, combining local specializations in function space. The same principle applies to sampled text completions, probabilities, or logits, and can be used either once for model composition or iteratively in federated learning. 1 Introduction Modern machine learning increasingly relies on data, computation, and expertise. Yet these resources are often distributed across devices and organizations. Collaborative learning addresses this fragmentation by enabling participants to harness distributed resources toward a shared model without requiring the underlying data to be centralized [1, 2]. With foundation models, this paradigm extends naturally to independently adapted models: participants can fine-tune a common pretrained model on their own data or expertise and subsequently combine the resulting specializations [3, 4, 5]. Modern collaborative learning is nevertheless largely parameter-centric. Federated clients repeatedly upload model deltas, while organizations that fine-tune a common foundation model exchange adapters or checkpoints and combine them with parameter averaging or model-merging rules [1, 2, 3, 4, 5]. This design has two immediate costs. Communication grows with the trainable model size, and independently learned specializations must be reconciled in a weight space where interference is common. Moreover, gradients and model updates expose a well-studied privacy leakage surface [6, 7, 8, 9]. Distillation can move aggregation to function space, but existing federated variants typically rely on task-related public data, class-level summaries, or a learned generator [10, 11, 12, 13]. Subliminal learning suggests a different channel. Cloud et al. [14] showed that a model can transfer a learned trait through outputs calculated on inputs that are semantically unrelated to that trait, particularly when teacher and student share a base model. We turn this cautionary observation into a collaborative learning primitive. Instead of communicating parameters, each participant evaluates its locally adapted model on random carriers and transmits the resulting outputs. A coordinator pools these messages and trains a shared model on them. For example, three LoRA-adapted language models can communicate distinct preferences through random completions, allowing a single shared model to combine the participants’ specializations (Figure 1). We call this primitive Distributed Subliminal Learning (DSL). DSL shifts knowledge sharing from parameter exchange to function-space communication through random carriers: participants communicate how their locally adapted models behave on task-unrelated inputs rather than transmitting the adaptations themselves. In this work, we study two DSL instantiations corresponding to two common collaborative workflows. In one-shot foundation-model composition, DSL communicates carrier completions, which can be substantially smaller than adapters, and combines them through distillation, avoiding direct weight-space merging where independently learned specializations can interfere. In iterative federated learning (FL), the same mechanism yields a proxy-free protocol for discriminative models, where clients repeatedly transmit carrier logits on task-unrelated inputs (e.g., images, tabular data, or latent embeddings). By avoiding update transmission, DSL removes the direct gradient- and update-leakage surface exploited by many existing attacks [6, 7, 8, 9]. While this does not constitute a privacy guarantee, communication is mediated through task-unrelated model behavior rather than direct exposure of the local adaptation. Our contributions are empirical and methodological: • We introduce DSL, a collaborative learning primitive for knowledge sharing, and instantiate it in one-shot LLM composition and iterative FL. Participants communicate only random carrier outputs, and a coordinator trains a shared model with a modality-appropriate distillation objective. • In LLM composition, DSL combines independently learned participant specializations into a single shared model: it retains three controlled preferences better than the LoRA-averaging baseline (94.56%94.56\% vs. 87.76%87.76\%) and transfers substantially more mathematical competence (22.022.0 vs. 0.60.6 points over the base model), while reducing participant upload by 30.630.6–49.0×49.0×. • In FL classification, DSL exposes a controllable accuracy–communication frontier: it nearly matches FedAvg on MNIST with 8.9×8.9× less uplink and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. A local KL-curvature analysis and carrier ablations support the interpretation that success depends on the probed model interface and carrier coverage. 2 Distributed Subliminal Learning 2.1 Blueprint: communication through random carriers Let =1,…,KK=\1,…,K\ denote the set of participants. Participant k∈k holds private data kD_k and adapts a shared model θ into a local model θk=LocalTrain(θ,k) _k= LocalTrain(θ,D_k). Classical collaboration communicates θk−θ _k-θ or an adapter to the coordinator. DSL instead changes the object through which local knowledge is communicated. Participant k draws MkM_k carriers xk,i∼qx_k,i q, where q is independent of kD_k, evaluates its local model, and sends ℳk=(xk,i,sθk(xk,i))i=1Mk.M_k=\(x_k,i,s_ _k(x_k,i))\_i=1^M_k. (1) The signal sθk(x)s_ _k(x) denotes the model output communicated through the carrier channel; depending on the instantiation, it can be a logit vector, a probability distribution, or a sampled text completion. Carriers may be transmitted explicitly or reproduced from shared pseudorandom seeds, so the carrier inputs need not be transmitted. The coordinator pools the messages and optimizes θ+=argmin∑k=1KϑpkMk∑i=1Mkd(fϑ(xk,i),sθk(xk,i)),θ^+= _ _k=1^K p_kM_k _i=1^M_kd\! (f_ (x_k,i),s_ _k(x_k,i) ), (2) where pkp_k are normalized participant weights and d is the appropriate distillation loss. Our experiments primarily use private-carrier pooling, in which participants sample carriers independently and the coordinator distills their pooled carrier outputs into the shared model by minimizing Eq. (2) over the union of their messages. Shared-carrier variants, in which participants evaluate a common carrier set and their outputs are aggregated at the coordinator, and full pseudocode appear in Appendix A. Carriers need not resemble the task distribution. Their role is to probe functional directions in which the local models differ from the common model. Consequently, “random” does not mean arbitrary: the carrier distribution must elicit outputs that distinguish the relevant local behaviors. We test this dependence by varying carrier distribution, budget, and the probed model interface in the experiments. 2.2 LLM composition from random completions We first instantiate DSL for one-shot LLM composition. Each participant starts from the same pretrained language model and applies a local LoRA adaptation [15], yielding a specialized local model. The carrier prompts are deliberately unrelated to the local specialization. For example, a participant may receive the carrier prompt “Continue the generation of this sequence of random numbers: 87, 56, 12, …”. The participant evaluates this prompt with its locally adapted model and records the resulting carrier completion. Only this completion is transmitted to the coordinator. The carrier prompt can instead be reconstructed from the agreed pseudorandom procedure; the LoRA adapter, gradients, and private training examples remain local. Repeating this procedure over independently sampled carriers gives participant k a synthetic message k=(xk,i,yk,i)i=1Mk,xk,i∼q,yk,i∼pθk(⋅∣xk,i),S_k=\(x_k,i,y_k,i)\_i=1^M_k, x_k,i q, y_k,i p_ _k(· x_k,i), (3) where q generates task-unrelated prompts and pθk(⋅∣x)p_ _k(· x) is the local model’s conditional output distribution, from which the communicated signal sθk(x)s_ _k(x) is sampled. The coordinator pools =⋃kS= _kS_k, initializes a shared model from the common base model, and performs supervised fine-tuning: ℒSFT(ϑ)=−∑k=1KpkMk∑i=1Mk∑j=1|yk,i|logpϑ(yk,i,j∣xk,i,yk,i,<j).L_SFT( )=- _k=1^K p_kM_k _i=1^M_k _j=1^|y_k,i| p_ \! (y_k,i,j x_k,i,y_k,i,<j ). (4) The completions need not explicitly reveal the participant’s specialization or contain examples from its task. Under a shared initialization, local adaptation can change the conditional output distribution even on neutral probes; a collection of such completions therefore acts as a function-space signature of the local specialization. Distillation combines these signatures in the shared model, while communication scales with the number and length of completions rather than the adapter size. This is the operational difference from task arithmetic or adapter averaging: the coordinator merges behavior, not parameters. 2.3 Federated learning from random logits We next instantiate DSL as an iterative FL protocol for discriminative models, which we call Federated Subliminal Learning (FSL). At round t, each client locally adapts the current global model θt _t into θt,k _t,k, samples random carriers, and uploads the corresponding logit vectors zt,k(x)=sθt,k(x)z_t,k(x)=s_ _t,k(x). Starting from θt _t, the server minimizes a temperature-scaled KL loss over the pooled client messages, ℒt(ϑ)=∑k,ipkMkτ2KL(σ(zt,k(xk,i)/τ)∥σ(fϑ(xk,i)/τ)).L_t( )= _k,i p_kM_kτ^2KL\! (σ(z_t,k(x_k,i)/τ)\, \|\,σ(f_ (x_k,i)/τ) ). (5) The carriers are task-agnostic random inputs or random intermediate representations; no public examples or learned generator are used. Local interpretation. Appendix A.3 shows that, near the current global model, distilling clients’ carrier logits can recover the weighted average client update along the parameter directions identified by the carrier outputs. This is characterized by a Fisher-weighted carrier Gram matrix: for population private-carrier pooling, the quadratic Taylor model recovers the weighted update up to its null space, while for shared carriers with averaged logits the equivalence is exact within the linearized logit model. The finite-carrier result applies to the empirical quadratic model, so the analysis provides a local identifiability interpretation rather than a convergence guarantee for the nonlinear SGD protocol. 3 Experiments We evaluate DSL across the two collaborative workflows introduced above: one-shot LLM composition through random completions and iterative FL through random-logit exchange. Appendix B provides implementation details, while Appendix C reports the complete federated ablations. 3.1 LLM specialization without adapter exchange Common protocol and baselines. The controlled preference setting uses Qwen2.5-7B-Instruct and the GSM8K setting uses Qwen2.5-1.5B-Instruct; within each setting, all participants and the coordinator share the same base model. Participants independently fine-tune LoRA adapters and communicate only task-unrelated random-number completions; the coordinator fine-tunes a fresh copy of the base model on their pooled messages. We compare against equal-weight arithmetic averaging of the LoRA adapters, denoted LoRA Avg. [16]. Upload is measured as UTF-8 completion bytes for DSL and serialized adapter size for LoRA Avg. 3.1.1 Controlled preference composition Setup. We first isolate the communication mechanism in a controlled synthetic setting inspired by subliminal learning [14]. Three participants are independently adapted to state that their favorite animal, food, or tree is respectively cat, pasta, or maple, and each communicates 40,000 random-number completions. The goal is to produce one model retaining all three independently injected preferences. We report preference retention as the increase over the base model in the fraction of completions containing the target preference word, averaged over 100 completions for each of 30 held-out prompts per category. Table 1 summarizes the results. DSL combines independently injected traits with substantially less interference. DSL reaches 94.56%94.56\% average retention, only 2.982.98 points below the local model, whereas LoRA Avg. loses 9.789.78 points. The gain is largest for pasta (+9.73+9.73 points) and maple (+11.37+11.37 points). The communicated completions occupy only 1.581.58–2.522.52 MiB per participant, corresponding to 30.630.6–48.8×48.8× less upload than the adapters. This controlled result establishes that neutral completions can carry multiple local behaviors without requiring those behaviors to appear explicitly in the carrier prompts. Table 1: Controlled preference composition with Qwen2.5-7B-Instruct. Higher retention is better. DSL upload is measured per participant; reduction is relative to the LoRA payload. Preference Local LoRA Avg. DSL Upload (MiB) Reduction Cat 98.23 95.60 94.90 2.52 30.6×30.6× Pasta 96.07 85.27 95.00 1.85 41.6×41.6× Maple 98.33 82.40 93.77 1.58 48.8×48.8× Average 97.54 87.76 94.56 – – 3.1.2 GSM8K composition Setup. We next consider task competence learned from local task data rather than injected preferences. Ten participants each fine-tune a LoRA adapter on 4,000 examples sampled from the GSM8K training set and communicate 12,000 task-unrelated random-number completions. We evaluate the models on 500 held-out GSM8K examples using exact-answer accuracy (Table 2). Table 2: GSM8K composition with Qwen2.5-1.5B-Instruct. Δ reports the accuracy gain over the common base model; upload is measured per participant. Model Acc. (%\%) Δ Base (p) Upload (MiB) Reduction Base model 10.2 – – – Local models (avg.) 58.9 +48.7 – – LoRA Avg. 10.8 +0.6 35.27 1.0×1.0× DSL 32.2 +22.0 0.72 49.0×49.0× DSL transfers task competence where parameter averaging largely fails. The local models reach 58.9%58.9\% accuracy, confirming that substantial mathematical competence is learned locally. LoRA Avg., however, reaches only 10.8%10.8\%, just 0.60.6 points above the base model. In contrast, DSL reaches 32.2%32.2\%, improving over the base by 22.022.0 points and over LoRA Avg. by 21.421.4 points. This transfer requires only 0.720.72 MiB per participant instead of 35.2735.27 MiB of adapter parameters, roughly a 49×49× reduction in upload. This shows that the random-carrier channel can communicate competence learned from local task supervision in addition to injected behavioral traits. 3.2 Federated classification Setup and protocols. We evaluate FSL on MNIST with an MLP, CIFAR-10 with ResNet9, and Tiny ImageNet with a frozen ViT-small backbone and trainable downstream head, using 20 clients and three seeds. The main comparison in Table 3 uses private-carrier pooling: clients upload only logits on pseudo-random carriers that the server can regenerate. We use M=2,048M=2,048, 4,0964,096, and 1,0241,024 carriers per client for MNIST, CIFAR-10, and Tiny ImageNet, respectively, with budgets chosen from the carrier sweeps in Appendix C to balance accuracy and communication for each model interface. Appendix B provides full protocol details, while Appendix C additionally compares private, shared, and weighted shared carriers; uniform, blurred, and Fourier distributions; and multiple budgets. We compare against parameter-sharing baselines FedAvg and scaffold [1, 17], and distillation-based alternatives. FedMD exchanges logits on public data [18]; FedDF uploads client models for distillation on unlabeled or generated proxy data [19]; and FedGen and dfrd combine model communication with generators [13, 20]. In contrast, FSL communicates only outputs on task-agnostic random carriers, requiring neither model/update uploads, public or proxy data, nor a generator. FSL traces a favorable accuracy–communication frontier, trading accuracy for substantial uplink savings. On MNIST, FSL reaches 96.83%96.83\%, only 0.510.51 points below FedAvg, with 8.9×8.9× less uplink. On CIFAR-10, it reduces uplink by 136.3×136.3× at a 9.949.94-point accuracy cost; increasing the carrier budget M and refining the carrier design raises accuracy to 63.02±1.15%63.02± 1.15\% while retaining a 7.08×7.08× reduction relative to FedAvg (Table 4). On Tiny ImageNet, FSL reaches 63.77%63.77\%, 4.354.35 points below FedAvg, with 7.5×7.5× less uplink. These results support FSL as a tunable communication primitive rather than a fixed drop-in replacement for parameter aggregation. Figure 2: MNIST test accuracy versus total uplink. Increasing the carrier budget moves FSL along an explicit accuracy–communication frontier under the same accounting as the baselines. Carrier budget and design control the information exposed to FSL. Figure 2 shows that larger carrier sets progressively close the MNIST accuracy gap while increasing uplink. Broader ablations (Appendix C, Secs. C.2–C.5) show that carrier suitability depends on the probed model interface rather than task semantics alone: independent uniform carriers are strongest for dense or representation-space interfaces, whereas blurred or Fourier-structured carriers are more effective when probing convolutional networks in image space. Thus, the carriers need not carry task semantics, but they must excite the trainable interface in directions that distinguish the local models. Appendix C also reports reduced-data checks, output-entropy diagnostics, and additional protocol ablations. Table 3: Test accuracy and total uplink across datasets, reported as mean ± standard deviation. MNIST CIFAR-10 Tiny ImageNet Method Acc. (%) ↑ Uplink (GiB) ↓ Acc. (%) ↑ Uplink (GiB) ↓ Acc. (%) ↑ Uplink (GiB) ↓ FedAvg 97.34±0.3297.34± 0.32 2.37±0.4732.37± 0.473 66.25±0.6766.25± 0.67 14.99±1.4714.99± 1.47 68.12±0.0568.12± 0.05 0.726±0.0480.726± 0.048 scaffold 94.62±0.2294.62± 0.22 8.07±1.128.07± 1.12 56.00±2.6056.00± 2.60 85.72±18.4085.72± 18.40 65.35±0.7365.35± 0.73 19.22±2.6619.22± 2.66 dfrd 97.24±0.2797.24± 0.27 2.18±0.4852.18± 0.485 66.46±0.6166.46± 0.61 11.94±0.28311.94± 0.283 68.13±0.1868.13± 0.18 0.685±0.1890.685± 0.189 FedDF 96.65±0.1396.65± 0.13 1.61±0.2301.61± 0.230 66.11±0.6066.11± 0.60 10.14±1.5810.14± 1.58 66.92±0.2666.92± 0.26 0.489±0.1690.489± 0.169 FedGen 97.26±0.2497.26± 0.24 2.15±0.4082.15± 0.408 66.39±1.1666.39± 1.16 9.32±3.229.32± 3.22 61.63±0.5261.63± 0.52 0.182±0.0240.182± 0.024 FedMD 92.43±0.2392.43± 0.23 0.030±0.0030.030± 0.003 40.25±0.6940.25± 0.69 0.003±0.0010.003± 0.001 45.68±0.0745.68± 0.07 0.153±0.0810.153± 0.081 FSL 96.83±0.2796.83± 0.27 0.266±0.0480.266± 0.048 56.31±0.5456.31± 0.54 0.110±0.0310.110± 0.031 63.77±0.1563.77± 0.15 0.097±0.0230.097± 0.023 4 Related Work Model merging and subliminal transfer. Model merging combines independently adapted models in parameter space. Model soups and Fisher merging average model weights [3, 21], while task arithmetic operates on fine-tuning deltas and methods such as ties and dare mitigate interference or redundancy [4, 5, 22]. Recent work also studies composition directly at the LoRA-adapter level [16]. DSL instead composes local specializations through generated behavior, avoiding parameter-space merging and making communication depend on the carrier payload rather than adapter size. Its mechanism is inspired by subliminal learning [14], which showed that behavioral traits can transfer through semantically unrelated generated data. DSL reverses this original safety framing: hidden transfer through unrelated data becomes an explicit channel for collaborative knowledge sharing. Federated optimization and communication efficiency. FedAvg and its variants aggregate client parameter updates, with methods such as FedProx and scaffold addressing statistical heterogeneity and client drift [1, 23, 17]. Communication-efficient FL further reduces this cost through structured or sketched updates, update sharding and compression, quantization, and less frequent aggregation [24, 25, 26]. These approaches reduce the size or frequency of parameter exchange but remain update-centric. DSL instead changes the communicated object itself, replacing model updates with outputs on task-unrelated carriers. Federated and data-free distillation. Knowledge distillation transfers model behavior through output distributions rather than parameters [10]. In FL, FedMD exchanges predictions on shared public data [18], while FedDF distills uploaded client models using unlabeled or generated proxy data [19]. FedGen and dfrd instead learn server-side generators for data-free distillation while retaining model communication [13, 20]. More broadly, zero-shot and data-free distillation synthesize surrogate inputs from model predictions or internal statistics [27, 28]. In contrast, FSL combines output-only communication with a fixed, task-agnostic carrier distribution, requiring neither model/update uploads, task-related public or proxy data, nor a learned or optimized generator. 5 Discussion Limitations. Our experiments and analysis assume related models sharing a common initialization; extending carrier-based communication to independently pretrained or heterogeneous models remains open. The local analysis characterizes recovery only on the identifiable update subspace and does not establish global equivalence for arbitrary nonlinear adaptations. The choice of carrier distribution and budget is problem dependent, and DSL adds coordinator-side distillation. Avoiding direct gradient transmission removes a known leakage channel but provides no privacy guarantee, as carrier outputs and distillation may still reveal local information [29]. Finally, our evaluation covers one LLM family and mostly IID, full-participation federations. Broader merging baselines [16, 30], non-IID data [31, 32], adversarial participants [33, 34] and decentralized coordination remain to be studied [35]. Benefits and extensibility. DSL exposes a direct accuracy–communication knob: the communicated payload is controlled by the carrier budget and output size rather than the number of locally adapted parameters, making it particularly attractive when coordinator compute is abundant but participant uplink is constrained. The same primitive supports different carrier interfaces and output modalities and is not tied to a particular collaboration pipeline or local adaptation method, making it applicable in principle to one-shot composition as well as iterative or distributed learning settings. Moreover, function-space communication need not exclude parameter-space methods: when a merged or averaged model is available, it could initialize subsequent carrier-based distillation, enabling hybrid forms of collaboration. 6 Conclusion Collaborative learning typically exchanges model updates and reconciles local knowledge in parameter space. We introduced Distributed Subliminal Learning (DSL), a collaborative learning primitive that replaces parameter exchange with function-space communication through task-unrelated random carriers and distills the resulting carrier outputs into a shared model. In LLM composition, DSL preserves controlled preferences better than LoRA averaging and transfers substantially more mathematical competence, while reducing upload by 30.630.6–49.0×49.0×; in federated classification, it yields a tunable accuracy–communication frontier, nearly matching FedAvg on MNIST while substantially reducing uplink across datasets. Together, these results suggest that model parameters need not be the default currency of collaboration. Useful knowledge can instead be shared through behavior on suitably chosen, task-unrelated carriers. Acknowledgments and Disclosure of Funding This research was funded by the Swiss National Science Foundation, the European Union’s Horizon Europe programme, and the Slovenian Research and Innovation Agency through the projects SmartCHANGE (No. 101080965) and XAI-PAC (No. Z00P2_216405). References [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, p. 1273–1282. External Links: ISSN 2640-3498 Cited by: §1, §1, §3.2, §4. [2] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, et al. (2021) Advances and Open Problems in Federated Learning. Foundations and Trends® in Machine Learning 14 (1–2), p. 1–210. External Links: ISSN 1935-8237, 1935-8245, Document Cited by: §1, §1. [3] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt (2022) Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 162, p. 23965–23998. External Links: Link Cited by: §1, §1, §4. [4] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi (2023) Editing models with task arithmetic. External Links: 2212.04089, Document Cited by: §1, §1, §4. [5] P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal (2023) TIES-Merging: resolving interference when merging models. External Links: 2306.01708, Document Cited by: §1, §1, §4. [6] L. Zhu, Z. Liu, and S. Han (2019) Deep Leakage from Gradients. In Advances in Neural Information Processing Systems, Vol. 32. Cited by: §1, §1. [7] B. Zhao, K. R. Mopuri, and H. Bilen (2020) iDLG: Improved Deep Leakage from Gradients. arXiv. External Links: 2001.02610, Document Cited by: §1, §1. [8] H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov (2021) See Through Gradients: Image Batch Recovery via GradInversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16337–16346. Cited by: §1, §1. [9] D. Fenoglio, M. Gjoreski, E. Gabrielli, and M. Langheinrich (2026) Security and Privacy in Federated Learning: A Survey and Unified Taxonomy of Threat Models, Attacks, and Defenses. SSRN Scholarly Paper, Social Science Research Network, Rochester, NY. External Links: 6771005, Document Cited by: §1, §1. [10] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531, Document Cited by: §1, §4. [11] E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S. Kim (2023) Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data. arXiv. Note: arXiv:1811.11479 [cs, stat]Comment: presented at the 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2nd Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2), Montr\’eal, CanadaComment: presented at the 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2nd Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2), Montr\’eal, Canada External Links: Link Cited by: §1. [12] C. He, M. Annavaram, and S. Avestimehr (2020) Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge. In Advances in Neural Information Processing Systems, Vol. 33, p. 14068–14080. External Links: Link Cited by: §1. [13] Z. Zhu, J. Hong, and J. Zhou (2021) Data-Free Knowledge Distillation for Heterogeneous Federated Learning. In Proceedings of the 38th International Conference on Machine Learning, p. 12878–12889. External Links: ISSN 2640-3498 Cited by: §1, §3.2, §4. [14] A. Cloud, M. Le, J. Chua, J. Betley, A. Sztyber-Betley, S. Mindermann, J. Hilton, S. Marks, and O. Evans (2026) Language models transmit behavioural traits through hidden signals in data. Nature 652 (8110), p. 615–621. External Links: ISSN 1476-4687, Document Cited by: §1, §3.1.1, §4. [15] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Document Cited by: §2.2. [16] A. Prabhakar, Y. Li, K. Narasimhan, S. Kakade, E. Malach, and S. Jelassi (2025) LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert, K. Darwish, and A. Agarwal (Eds.), Abu Dhabi, UAE, p. 644–655. Cited by: §3.1, §4, §5. [17] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh (2020) SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In Proceedings of the 37th International Conference on Machine Learning, p. 5132–5143. External Links: ISSN 2640-3498 Cited by: §3.2, §4. [18] D. Li and J. Wang (2019) FedMD: Heterogenous Federated Learning via Model Distillation. arXiv. Note: NeurIPS 2019 Workshop on Federated Learning for Data Privacy and Confidentiality External Links: 1910.03581, Document Cited by: §3.2, §4. [19] T. Lin, L. Kong, S. U. Stich, and M. Jaggi (2020) Ensemble Distillation for Robust Model Fusion in Federated Learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 2351–2363. Cited by: §3.2, §4. [20] K. Luo, S. Wang, Y. Fu, X. Li, Y. Lan, and M. Gao (2023) DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning. In Advances in Neural Information Processing Systems, Vol. 36, p. 17854–17866. External Links: Document Cited by: §3.2, §4. [21] M. S. Matena and C. Raffel (2022) Merging Models with Fisher-Weighted Averaging. Advances in Neural Information Processing Systems 35, p. 17703–17716. External Links: Document Cited by: §4. [22] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li (2024) Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. In Proceedings of the 41st International Conference on Machine Learning, p. 57755–57775. Cited by: §4. [23] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2020) Federated Optimization in Heterogeneous Networks. Proceedings of Machine Learning and Systems 2, p. 429–450. Cited by: §4. [24] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon (2017) Federated Learning: Strategies for Improving Communication Efficiency. arXiv. External Links: 1610.05492, Document Cited by: §4. [25] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani (2020) FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, p. 2021–2031. External Links: ISSN 2640-3498 Cited by: §4. [26] D. Fenoglio, P. Polverino, J. Quizi, M. Gjoreski, A. Dhasade, and M. Langheinrich (2026) ERIS: enhancing privacy and scalability in federated learning via federated shard aggregation. Note: Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026) External Links: 2602.08617 Cited by: §4. [27] G. K. Nayak, K. R. Mopuri, V. Shaj, V. B. Radhakrishnan, and A. Chakraborty (2019) Zero-Shot Knowledge Distillation in Deep Networks. In Proceedings of the 36th International Conference on Machine Learning, p. 4743–4751. External Links: ISSN 2640-3498 Cited by: §4. [28] H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz (2020) Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8712–8721. External Links: ISSN 2575-7075, Document Cited by: §4. [29] M. Jagielski, M. Nasr, K. Lee, C. Choquette-Choo, N. Carlini, and F. Tramèr (2023) Students parrot their teachers: membership inference on model distillation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, p. 44382–44397. Cited by: §5. [30] G. Stoica, P. Ramesh, B. Ecsedi, L. Choshen, and J. Hoffman (2024) Model merging with SVD to tie the Knots. In The Thirteenth International Conference on Learning Representations, Cited by: §5. [31] D. Fenoglio, M. Li, P. Barbiero, N. Lane, M. Langheinrich, and M. Gjoreski (2025) FLUX: Efficient Descriptor-Driven Clustered Federated Learning under Arbitrary Distribution Shifts. In Advances in Neural Information Processing Systems, Vol. 38, Main Conference, p. 71229–71292. External Links: Document Cited by: §5. [32] D. Jhunjhunwala, P. Sharma, A. Nagarkatti, and G. Joshi (2022) Fedvarp: Tackling the variance due to partial client participation in federated learning. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, p. 906–916. External Links: ISSN 2640-3498 Cited by: §5. [33] Y. Tang, Z. Wu, B. Gao, T. Wen, Y. Wang, and S. Sun (2024) Logits Poisoning Attack in Federated Distillation. In Knowledge Science, Engineering and Management: 17th International Conference, KSEM 2024, Birmingham, UK, August 16–18, 2024, Proceedings, Part I, Berlin, Heidelberg, p. 286–298. External Links: Document, ISBN 978-981-97-5497-7 Cited by: §5. [34] D. Fenoglio, G. Dominici, P. Barbiero, A. Tonda, M. Gjoreski, and M. Langheinrich (2024) Federated behavioural planes: explaining the evolution of client behaviour in federated learning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Vol. 37, Red Hook, NY, USA, p. 112777–112813. External Links: ISBN 979-8-3313-1438-5 Cited by: §5. [35] S. Kalra, J. Wen, J. C. Cresswell, M. Volkovs, and H. R. Tizhoosh (2023) Decentralized federated learning through proxy model sharing. Nature Communications 14 (1), p. 2899. External Links: ISSN 2041-1723, Document Cited by: §5. Appendix A Protocol and Analysis A.1 Generic protocol Algorithm 1 Distributed Subliminal Learning with private-carrier pooling Input: Initial model θ0 _0, private datasets kk=1K\D_k\_k=1^K, carrier distribution q, carrier budgets Mkk=1K\M_k\_k=1^K, rounds T, participant weights pkp_k 1 for t=0,…,T−1t=0,…,T-1 do 2 Select participants tS_t and broadcast θt _t; 3 foreach k∈tk _t in parallel do 4 θt,k←LocalTrain(θt,k) _t,k← LocalTrain( _t,D_k); 5 Sample t,k=xt,k,ii=1Mk∼i.i.d.qX_t,k=\x_t,k,i\_i=1^M_k i.i.d. q; 6 Form ℳt,k=(xt,k,i,sθt,k(xt,k,i))i=1MkM_t,k=\(x_t,k,i,s_ _t,k(x_t,k,i))\_i=1^M_k; 7 Transmit ℳt,kM_t,k; do not transmit θt,k _t,k or θt,k−θt _t,k- _t; 8 Pool messages and set θt+1←Distill(θt,∪k∈tℳt,k,pk) _t+1← Distill( _t, _k _tM_t,k,\p_k\); Output: Shared model θT _T For one-shot LLM composition, T=1T=1: participants adapt the same pretrained model, produce random completions, and the coordinator supervised-fine-tunes a shared model on their pooled messages. For iterative FSL, each round alternates client-side local training and server-side carrier distillation. A.2 Carrier aggregation variants Private-carrier pooling. Each participant samples its own carrier set and the coordinator minimizes the participant-weighted objective in Eq. (2). Equal carrier counts and uniform example weighting implement uniform participant weights. Data-size weighting can be implemented explicitly through pkp_k or by sampling carrier counts proportionally. Shared-carrier averaging. All participants evaluate the same carrier set xii=1M\x_i\_i=1^M. For vector-valued outputs, the coordinator forms s¯(xi)=∑kpksθk(xi) s(x_i)= _kp_ks_ _k(x_i) and distills against the averaged target. In the FL setting, this is the closest function-space analogue of FedAvg. Communication object. Carriers may be transmitted explicitly or regenerated from an agreed pseudorandom procedure. In the LLM experiments, carrier prompts are reproducible and upload counts only the UTF-8 bytes of the communicated completions. In FSL, pseudo-random carriers are regenerated by the server and uplink counts the transmitted logits. A.3 Local interpretation of the KL objective Let zθ(x)∈ℝcz_θ(x) ^c denote the logit vector and fix the common model θ0 _0 broadcast at the start of a round. Write δk=θk−θ0 _k= _k- _0, δ¯=∑kpkδk δ= _kp_k _k, and J(x)=∇θzθ0(x)J(x)= _θz_ _0(x). For temperature τ, define π0(x)=σ(zθ0(x)/τ),C0(x)=Diag(π0(x))−π0(x)π0(x)⊤, _0(x)=σ\! (z_ _0(x)/τ ), C_0(x)=Diag( _0(x))- _0(x) _0(x) , (6) and the Fisher-weighted carrier Gram matrix Hq=x∼q[J(x)⊤C0(x)J(x)].H_q=E_x q\! [J(x) C_0(x)J(x) ]. (7) The matrix C0(x)C_0(x) is the Hessian of the log-sum-exp function at the base natural parameter. In particular, it removes directions that add the same scalar to every logit and are therefore invisible to softmax. Theorem A.1 (Quadratic local equivalence). Assume pk≥0p_k≥ 0, ∑kpk=1 _kp_k=1, and HqH_q is finite. Suppose the logits are twice continuously differentiable near θ0 _0 and their local Taylor remainder is dominated by a q-integrable envelope. For perturbations satisfying max(‖δ‖2,maxk‖δk‖2)≤ρ (\|δ\|_2, _k\| _k\|_2)≤ρ, the population private-carrier objective in Eq. (5) has the expansion ℒprivate(θ0+δ)=12∑kpk(δ−δk)⊤Hq(δ−δk)+o(ρ2).L_private( _0+δ)= 12 _kp_k(δ- _k) H_q(δ- _k)+o(ρ^2). (8) The minimizers of the quadratic term are exactly δ¯+ker(Hq) δ+ (H_q). Equivalently, its identifiable component is the weighted client update: Projrange(Hq)(δ⋆)=Projrange(Hq)(δ¯).Proj_range(H_q)(δ )=Proj_range(H_q)( δ). (9) If Hq≻0H_q 0 on the considered trainable parameterization, the quadratic minimizer is unique and equals δ¯ δ. For shared carriers with weighted logit target z¯(x)=∑kpkzθk(x) z(x)= _kp_kz_ _k(x), the same minimizer set is exact under the linearized logit model zθ0+δ(x)=zθ0(x)+J(x)δz_ _0+δ(x)=z_ _0(x)+J(x)δ. Proof. At fixed x, let A(η)=log∑jeηjA(η)= _je _j. The KL divergence between two softmax distributions is the Bregman divergence generated by A, whose Hessian at the base natural parameter zθ0(x)/τz_ _0(x)/τ is C0(x)C_0(x). Therefore, τ2KL(σ(zθk(x)/τ)∥σ(zθ0+δ(x)/τ))=12(δ−δk)⊤J(x)⊤C0(x)J(x)(δ−δk)+o(ρ2).τ^2KL\! (σ(z_ _k(x)/τ)\, \|\,σ(z_ _0+δ(x)/τ) )= 12(δ- _k) J(x) C_0(x)J(x)(δ- _k)+o(ρ^2). Taking expectations and summing over clients gives Eq. (8). Completing the square yields ∑kpk(δ−δk)⊤Hq(δ−δk)=(δ−δ¯)⊤Hq(δ−δ¯)+∑kpk(δk−δ¯)⊤Hq(δk−δ¯), _kp_k(δ- _k) H_q(δ- _k)=(δ- δ) H_q(δ- δ)+ _kp_k( _k- δ) H_q( _k- δ), so the quadratic minimizers are δ¯+ker(Hq) δ+ (H_q). For shared carriers, the linearized weighted logit target is zθ0(x)+J(x)δ¯z_ _0(x)+J(x) δ. The KL loss is zero at δ=δ¯δ= δ, and it is also zero only when J(x)(δ−δ¯)J(x)(δ- δ) is a common-logit shift for q-almost every x. This condition is equivalent to δ−δ¯∈ker(Hq)δ- δ∈ (H_q). ∎ The private-carrier statement is deliberately local: the exact population minimizer of a nonlinear KL mixture need not equal the parameter average away from the common initialization. The theorem identifies the second-order mechanism relevant near that initialization; it is not an exact equivalence claim for the implemented finite optimization. A.4 Finite private carriers in the quadratic model Let U∈ℝP×rU ^P× r have orthonormal columns spanning range(Hq)range(H_q), let Gq=U⊤HqU≻0G_q=U H_qU 0, and write ak=U⊤δka_k=U _k. Because the per-carrier curvature matrices are positive semidefinite, every direction in ker(Hq) (H_q) is also annihilated q-almost surely by J(x)⊤C0(x)J(x)J(x) C_0(x)J(x); the empirical quadratic loss therefore depends only on the coordinates in U. For client k, define G^k=1Mk∑i=1MkU⊤J(xk,i)⊤C0(xk,i)J(xk,i)U,Ek=G^k−Gq. G_k= 1M_k _i=1^M_kU J(x_k,i) C_0(x_k,i)J(x_k,i)U, E_k= G_k-G_q. The identifiable component of the empirical quadratic minimizer of 12∑kpk(a−ak)⊤G^k(a−ak) 12 _kp_k(a-a_k) G_k(a-a_k) satisfies a^=(∑kpkG^k)−1(∑kpkG^kak), a= ( _kp_k G_k )^-1 ( _kp_k G_ka_k ), (10) whenever the matrix is invertible. Proposition A.2 (Deterministic finite-carrier perturbation bound). Let μ=λmin(Gq)>0μ= _ (G_q)>0. If ‖Ek‖2≤ε<μ\|E_k\|_2≤ <μ for every client, then ‖a^−a¯‖2≤εμ−ε∑kpk‖ak−a¯‖2,a¯=∑kpkak.\| a- a\|_2≤ μ- _kp_k\|a_k- a\|_2, a= _kp_ka_k. (11) Proof. Set A=∑kpkG^kA= _kp_k G_k. Since ‖A−Gq‖2≤ε\|A-G_q\|_2≤ , Weyl’s inequality gives λmin(A)≥μ−ε _ (A)≥μ- and ‖A−1‖2≤(μ−ε)−1\|A^-1\|_2≤(μ- )^-1. Moreover, a^−a¯=A−1∑kpkEk(ak−a¯). a- a=A^-1 _kp_kE_k(a_k- a). The triangle inequality yields Eq. (11). ∎ If the carriers are i.i.d. from the same q and ‖U⊤J(x)⊤C0(x)J(x)U‖2≤B\|U J(x) C_0(x)J(x)U\|_2≤ B almost surely, matrix concentration and a union bound give maxk‖Ek‖2=O(Blog(rK/η)/Mmin) _k\|E_k\|_2=O\! (B (rK/η)/M_ ) with probability at least 1−η1-η, up to the usual lower-order Mmin−1M_ ^-1 term. Thus the proposition controls carrier-sampling error in the empirical quadratic model. It does not control the KL Taylor remainder, server optimization error, or nonlinear round-to-round dynamics. The bound worsens when carrier coverage is poorly conditioned (small μ) or client updates are widely dispersed; the present experiments vary carrier distribution and budget, qualitatively probing the former. Appendix B Experimental Details B.1 LLM composition details Common configuration. The controlled preference experiment uses unsloth/Qwen2.5-7B -Instruct as its common base model, while the GSM8K experiment uses unsloth/Qwen2.5-1.5B-Instruct. Participants independently fine-tune rank-8 LoRA adapters with α=8α=8 on the attention and MLP projection modules (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj). Local adaptation uses Unsloth SFT with a linear learning-rate schedule, 5 warmup steps, and gradient clipping at 1.0. For DSL composition, the coordinator initializes a fresh copy of the common base model and supervised-fine-tunes it on the shuffled union of participant completions for 12 epochs, with learning rate 10−410^-4, batch size 16, and gradient accumulation 4. Random-number carrier prompts are generated from randomized templates containing 3–8 integers sampled from 100 to 999 and ask the model to generate up to 10 additional numbers with at most three digits. Carrier completions are sampled with temperature 1.0 and a 2048-token generation cap; other decoding settings use the vLLM 0.10.0 defaults. The LoRA Avg. baseline is obtained by equal-weight arithmetic averaging of the participant LoRA tensors. Controlled preference composition. Three participants are respectively adapted on 5,000 synthetic examples expressing the cat, pasta, and maple preferences. Each then generates 40,000 random-number continuations. For evaluation, we use 30 held-out prompts per category and sample 100 completions for each prompt. Preference retention is the mean target-string rate after subtracting the corresponding base-model rate. GSM8K composition. Ten participants independently sample 4,000 examples from the GSM8K training split, without replacement within each client. Each participant fine-tunes its LoRA adapter for one epoch with learning rate 10−410^-4, batch size 16, gradient accumulation 4, and maximum sequence length 1,024. Each then generates 12,000 task-unrelated random-number completions for DSL composition. Models are evaluated on 500 held-out GSM8K test examples using greedy decoding. We report exact-answer accuracy after extracting and normalizing the final predicted and reference answers. B.2 Federated protocol summary Figure 3 illustrates the round-based FSL protocol used throughout these experiments. All federated experiments use 20 clients with full participation, at most 200 communication rounds, and validation-based early stopping. Results are reported as mean ± standard deviation over three seeds. Unless otherwise stated, client data are IID random shards, with 20% of each local shard reserved for validation. Both local client training and server distillation use Adam. Clients train for 5 local epochs with learning rate 5×10−45× 10^-4, while the server distills for up to 5 epochs with learning rate 3×10−43× 10^-4. Batch sizes are 1,024 for MNIST, 256 for CIFAR-10, and 64 for Tiny ImageNet, and the KL objective uses τ=1τ=1. Round-level early stopping uses a minimum improvement of 10−410^-4 with patience 6 for MNIST and CIFAR-10 and patience 10 for Tiny ImageNet. The MNIST MLP has two 512-unit ReLU hidden layers. Tiny ImageNet uses a pretrained timm vit_small_patch16_224 frozen feature extractor with a trainable two-layer 512-unit MLP head. The main comparison in Table 3 uses private-carrier pooling. MNIST–MLP uses M=2,048M=2,048 independent uniform pixel carriers per client; CIFAR-10–ResNet9 uses M=4,096M=4,096 spatially blurred random-image carriers; and Tiny ImageNet uses M=1,024M=1,024 independent uniform representation-space carriers at the trainable interface of a frozen ViT-small backbone. Clients communicate only the resulting logits, while pseudo-random carriers are reproducible at the server. The controlled MNIST and CIFAR-10 input-space experiments use three ghost outputs by default: additional logit dimensions with no associated task class or direct supervision, whose carrier logits are nevertheless communicated to increase the dimensionality of the behavioral signal. Tiny ImageNet representation-space experiments use none. Server distillation uses the complete communicated output vector, including all ghost logits; no auxiliary ghost heads are used. We study their effect separately in Appendix C.7. Figure 3: Federated Subliminal Learning. In each communication round, clients train locally, probe their models with task-unrelated carriers, and upload only the resulting carrier outputs. The server pools these outputs and distills the next global model, without receiving model updates or task-related proxy data. B.3 Communication accounting For parameter-sharing federated baselines, total uplink is the cumulative number of serialized client-update bytes transmitted before early stopping. For FSL, it is the cumulative carrier-output upload over the same period. Because unsuccessful configurations may terminate early, communication is interpreted jointly with accuracy. For LLM composition, DSL upload is the UTF-8 size of each participant’s transmitted completion corpus; the random carrier prompts are reproducible and are therefore not included in the transmitted payload. LoRA Avg. upload is the serialized LoRA adapter size, and reduction factors are computed relative to this payload. Appendix C Additional Federated Random-Carrier Experiments This appendix reports the complete carrier-distribution, aggregation-protocol, carrier-budget, and communication results underlying Sec. 3.2. The controlled input-space experiments use 20 clients, a maximum of 200 communication rounds, early stopping, and three ghost outputs (Appendix C.7) unless otherwise specified. The additional Tiny ImageNet experiments instead use private-carrier pooling with no ghost outputs and inject carriers directly at the representation interface of a frozen ViT-small backbone. C.1 Carrier protocols and interpretation of the carrier budget For completeness, we distinguish the three protocols used in the ablations. Private-carrier pooling. Each participating client k independently samples t,k∼qX_t,k q and communicates the resulting carrier–logit pairs. The server concatenates the client-specific sets and distills on their union. Consequently, a nominal per-client carrier budget M can expose the server to substantially more than M distinct probes across the federation. For example, with 20 clients, M=2,048M=2,048 corresponds to up to approximately 4141k independently generated carrier instances when all clients participate. Shared-carrier averaging. All participating clients evaluate the same carrier set tX_t. For every carrier x∈tx _t, the server forms z¯t(x)=1|t|∑k∈tzt,k(x), z_t(x)= 1|S_t| _k _tz_t,k(x), (12) and distills the global model from the resulting common carrier–target set. Weighted shared-carrier averaging. The weighted variant instead uses z¯t(x)=∑k∈twkzt,k(x),∑k∈twk=1, z_t(x)= _k _tw_kz_t,k(x), _k _tw_k=1, (13) where wkw_k denotes the normalized client aggregation weight. Unlike private pooling, both shared protocols force all client information through the same M-element carrier basis. This difference is especially important at small M. Carrier distributions. Uniform carriers sample each input or representation coordinate independently over the value range observed in the training data. Blurred carriers are obtained by applying Gaussian blur (σ=1.2σ=1.2) to Gaussian noise, standardizing it per sample/channel, rescaling it using the training-data mean and standard deviation, and clamping it to the observed range. Fourier carriers instead sample complex Gaussian Fourier coefficients with amplitude scaled as 1/fα1/f^α with α=1α=1, remove the DC component, inverse-transform and standardize the result, and apply the same mean–standard-deviation rescaling and clamping. Communication under early stopping. The reported uplink corresponds to the total synthetic-message communication accumulated before termination. Because training uses early stopping, low communication is not automatically a positive result: configurations that fail to learn can stop after very few rounds and therefore transmit substantially less data. Accordingly, the tables below should be interpreted jointly in terms of accuracy and communication rather than communication alone. C.2 CIFAR-10 with ResNet9 Table 4 reports the complete CIFAR-10 sweep. At the smallest budget, blurred carriers are substantially more sample-efficient than the alternatives. Increasing the shared carrier basis substantially improves both blurred and Fourier carriers, whereas uniform pixel noise remains ineffective even at the largest tested budget. Table 4: Complete CIFAR-10–ResNet9 carrier sweep. Accuracy and total synthetic uplink are reported as mean ± standard deviation. “Shared-w.” denotes weighted shared-carrier averaging. M Protocol Carrier Accuracy (%) ↑ Total uplink ↓ 4,096 Private Uniform 41.60±15.4241.60± 15.42 144.90±112.31144.90± 112.31 MiB 4,096 Private Blurred 55.44±0.7655.44± 0.76 96.60±13.0696.60± 13.06 MiB 4,096 Private Fourier 39.38±12.6339.38± 12.63 120.52±145.48120.52± 145.48 MiB 4,096 Shared Uniform 14.58±0.5214.58± 0.52 27.08±10.2227.08± 10.22 MiB 4,096 Shared Blurred 51.17±8.6451.17± 8.64 193.65±83.49193.65± 83.49 MiB 4,096 Shared Fourier 35.87±2.1435.87± 2.14 31.15±15.3831.15± 15.38 MiB 4,096 Shared-w. Uniform 11.67±0.9411.67± 0.94 5.42±2.355.42± 2.35 MiB 4,096 Shared-w. Blurred 56.27±2.0956.27± 2.09 219.38±53.12219.38± 53.12 MiB 4,096 Shared-w. Fourier 44.98±10.1044.98± 10.10 125.94±105.86125.94± 105.86 MiB 32,768 Shared Uniform 21.88±0.1821.88± 0.18 86.67±18.7686.67± 18.76 MiB 32,768 Shared Blurred 60.06±1.7560.06± 1.75 920.83±276.41920.83± 276.41 MiB 32,768 Shared Fourier 55.76±4.2855.76± 4.28 736.67±411.52736.67± 411.52 MiB 32,768 Shared-w. Uniform 19.41±1.0219.41± 1.02 65.00±0.0065.00± 0.00 MiB 32,768 Shared-w. Blurred 62.27±0.5762.27± 0.57 1.401.40 GiB ±245.37± 245.37 MiB 32,768 Shared-w. Fourier 61.16±0.4661.16± 0.46 1.351.35 GiB ±473.58± 473.58 MiB 65,536 Shared-w. Uniform 18.03±0.7718.03± 0.77 130.00±0.00130.00± 0.00 MiB 65,536 Shared-w. Blurred 63.02±1.1563.02± 1.15 2.122.12 GiB ±603.95± 603.95 MiB 65,536 Shared-w. Fourier 62.06±1.1562.06± 1.15 2.202.20 GiB ±695.02± 695.02 MiB The shared-carrier results make the interaction between carrier geometry and coverage especially clear. With blurred carriers, increasing the weighted shared basis from 4,0964,096 to 32,76832,768 and 65,53665,536 raises accuracy from 56.27%56.27\% to 62.27%62.27\% and 63.02%63.02\%. Fourier carriers exhibit an even larger relative improvement, from 44.98%44.98\% to 61.16%61.16\% and 62.06%62.06\%. Uniform carriers remain below 20%20\% at the two largest budgets. Thus, increasing M can compensate for an initially under-sampled but useful carrier family, as with Fourier, but does not necessarily compensate for a carrier distribution that poorly probes the function exposed to the carrier channel. C.3 MNIST with an MLP Table 5 reports the corresponding MNIST MLP experiments. Unlike the convolutional models probed directly in image space, the MLP consistently favors uniform pixelwise carriers. Private pooling is already effective with 2,0482,048 carriers per client, while the shared protocols progressively approach the same performance as the common carrier basis grows. Table 5: Complete MNIST–MLP carrier sweep. Accuracy and total synthetic uplink are reported as mean ± standard deviation. “Shared-w.” denotes weighted shared-carrier averaging. M Protocol Carrier Accuracy (%) ↑ Total uplink ↓ 2,048 Private Uniform 96.28±1.2196.28± 1.21 175.10±58.39175.10± 58.39 MiB 2,048 Private Blurred 94.98±0.0694.98± 0.06 153.02±8.46153.02± 8.46 MiB 2,048 Private Fourier 93.78±0.2493.78± 0.24 109.69±16.63109.69± 16.63 MiB 2,048 Shared Uniform 92.70±0.3492.70± 0.34 192.97±41.23192.97± 41.23 MiB 2,048 Shared Blurred 88.34±0.8188.34± 0.81 77.19±19.5977.19± 19.59 MiB 2,048 Shared Fourier 82.94±4.1682.94± 4.16 52.81±35.4252.81± 35.42 MiB 2,048 Shared-w. Uniform 92.39±0.1592.39± 0.15 146.25±16.12146.25± 16.12 MiB 2,048 Shared-w. Blurred 87.77±1.4187.77± 1.41 65.68±18.2165.68± 18.21 MiB 2,048 Shared-w. Fourier 84.96±1.2384.96± 1.23 53.49±13.0653.49± 13.06 MiB 16,384 Shared Uniform 95.58±0.1895.58± 0.18 1.391.39 GiB ±170.69± 170.69 MiB 16,384 Shared Blurred 93.79±0.1593.79± 0.15 1.201.20 GiB ±124.11± 124.11 MiB 16,384 Shared Fourier 93.42±0.1393.42± 0.13 1.161.16 GiB ±197.98± 197.98 MiB 16,384 Shared-w. Uniform 95.69±0.1495.69± 0.14 1.421.42 GiB ±119.78± 119.78 MiB 16,384 Shared-w. Blurred 92.58±0.5792.58± 0.57 677.08±315.24677.08± 315.24 MiB 16,384 Shared-w. Fourier 92.68±0.6092.68± 0.60 812.50±197.02812.50± 197.02 MiB 32,768 Shared-w. Uniform 96.72±0.2196.72± 0.21 2.672.67 GiB ±424.99± 424.99 MiB 32,768 Shared-w. Blurred 93.86±0.2093.86± 0.20 1.871.87 GiB ±148.93± 148.93 MiB 32,768 Shared-w. Fourier 93.48±0.8293.48± 0.82 1.931.93 GiB ±1.14± 1.14 GiB The MLP results differ systematically from the image-space convolutional experiments. Uniform carriers are strongest for every tested shared-weighted budget and improve from 92.39%92.39\% at 2,0482,048 carriers to 95.69%95.69\% at 16,38416,384 and 96.72%96.72\% at 32,76832,768. Blurred and Fourier carriers also improve with M, but remain consistently below the uniform configuration. This pattern is compatible with independent pixelwise probes providing broad directions for a dense input layer, while spatial smoothing removes variation that the MLP could otherwise exploit. C.4 MNIST with LeNet5 To disentangle dataset effects from architectural effects, we repeat the MNIST experiment with LeNet5. Table 6 shows that replacing the MLP by a convolutional model qualitatively changes the preferred carrier family. Table 6: Complete MNIST–LeNet5 carrier sweep. Accuracy and total synthetic uplink are reported as mean ± standard deviation. “Shared-w.” denotes weighted shared-carrier averaging. M Protocol Carrier Accuracy (%) ↑ Total uplink ↓ 1,024 Private Uniform 75.58±33.8075.58± 33.80 116.12±104.89116.12± 104.89 MiB 1,024 Private Blurred 95.09±0.8795.09± 0.87 196.35±28.04196.35± 28.04 MiB 1,024 Private Fourier 94.98±0.5194.98± 0.51 179.09±6.12179.09± 6.12 MiB 1,024 Shared Uniform 13.95±4.1513.95± 4.15 3.05±1.023.05± 1.02 MiB 1,024 Shared Blurred 13.25±1.8913.25± 1.89 3.393.39 MiB ±600.44± 600.44 KiB 1,024 Shared Fourier 36.20±40.9236.20± 40.92 89.04±148.9489.04± 148.94 MiB 1,024 Shared-w. Uniform 14.05±4.1314.05± 4.13 3.05±1.023.05± 1.02 MiB 1,024 Shared-w. Blurred 13.25±1.9213.25± 1.92 3.393.39 MiB ±600.44± 600.44 KiB 1,024 Shared-w. Fourier 37.41±43.0437.41± 43.04 96.48±161.8496.48± 161.84 MiB 4,096 Shared Uniform 13.50±3.1913.50± 3.19 10.82±2.3410.82± 2.34 MiB 4,096 Shared Blurred 92.02±1.3192.02± 1.31 796.83±162.86796.83± 162.86 MiB 4,096 Shared Fourier 92.56±1.3492.56± 1.34 1.091.09 GiB ±355.42± 355.42 MiB 4,096 Shared-w. Uniform 39.06±47.4539.06± 47.45 346.67±582.86346.67± 582.86 MiB 4,096 Shared-w. Blurred 91.14±2.2091.14± 2.20 758.33±225.54758.33± 225.54 MiB 4,096 Shared-w. Fourier 88.78±2.2288.78± 2.22 617.50±251.65617.50± 251.65 MiB 8,192 Shared-w. Uniform 52.75±40.1852.75± 40.18 796.25±999.77796.25± 999.77 MiB 8,192 Shared-w. Blurred 93.71±0.3693.71± 0.36 1.701.70 GiB ±223.15± 223.15 MiB 8,192 Shared-w. Fourier 92.73±1.9692.73± 1.96 1.401.40 GiB ±676.09± 676.09 MiB The private-carrier result is particularly informative because the dataset is unchanged relative to the MLP experiment. With only 1,0241,024 private carriers per client, blurred and Fourier carriers both reach approximately 95%95\% accuracy, while uniform carriers are much less reliable. The shared-carrier experiments additionally show that the common carrier basis must be sufficiently large: blurred and Fourier configurations fail at 1,0241,024 shared carriers but recover above 88%88\%–92%92\% at 4,0964,096 and above 92%92\% at 8,1928,192. Uniform carriers remain substantially less reliable even as M grows. Taken together with the MNIST MLP results, this experiment provides evidence that carrier suitability depends on the architecture exposed to the carrier channel rather than being determined by the dataset alone. C.5 Representation-Space Carriers on Tiny ImageNet The preceding experiments inject carriers in the original image space. We additionally consider a complementary setting on Tiny ImageNet in which the ViT-small representation extractor is frozen and random carriers are injected directly at the embedding interface of the trainable downstream layers. This experiment uses private-carrier pooling, 20 clients, a maximum of 200 communication rounds, early stopping, and no ghost outputs. This setting provides an additional test of whether carrier suitability is determined by the original input modality or by the function actually probed by the carrier channel. Since the frozen transformer is bypassed during carrier communication, the random representations directly probe the trainable downstream mapping. Table 7: Private representation-space carrier sweep on Tiny ImageNet using 50k training samples and a frozen ViT-small feature extractor. Accuracy and total synthetic uplink are reported as mean ± standard deviation. The M=1,024M=1,024 uniform configuration is the Tiny ImageNet FSL setting reported in Table 3. M Protocol Carrier Accuracy (%) ↑ Total uplink ↓ 64 Private Uniform 60.54±0.5860.54± 0.58 33.53±2.0333.53± 2.03 MiB 128 Private Uniform 61.84±0.3561.84± 0.35 44.27±17.5144.27± 17.51 MiB 256 Private Uniform 62.74±0.3262.74± 0.32 74.22±14.0874.22± 14.08 MiB 512 Private Blurred 60.12±0.1960.12± 0.19 101.56±0.00101.56± 0.00 MiB 512 Private Uniform 63.18±0.3263.18± 0.32 83.33±16.2683.33± 16.26 MiB 1,024 Private Blurred 60.46±0.3360.46± 0.33 151.04±47.74151.04± 47.74 MiB 1,024 Private Uniform 63.77±0.1563.77± 0.15 98.96±23.8798.96± 23.87 MiB 2,048 Private Blurred 61.29±0.1861.29± 0.18 197.92±36.08197.92± 36.08 MiB 2,048 Private Uniform 63.13±0.4063.13± 0.40 239.58±36.08239.58± 36.08 MiB 4,096 Private Blurred 61.17±0.1961.17± 0.19 312.50±0.00312.50± 0.00 MiB 4,096 Private Uniform 63.10±0.3663.10± 0.36 375.00±125.00375.00± 125.00 MiB Uniform representation-space carriers consistently outperform blurred carriers at every matched carrier budget. At M=512M=512, 1,0241,024, 2,0482,048, and 4,0964,096, the respective uniform accuracies are 63.18%63.18\%, 63.77%63.77\%, 63.13%63.13\%, and 63.10%63.10\%, compared with 60.12%60.12\%, 60.46%60.46\%, 61.29%61.29\%, and 61.17%61.17\% for blurred carriers. The uniform budget sweep additionally exhibits a clear saturation effect. Performance improves from 60.54%60.54\% with only 64 carriers to 62.74%62.74\% with 256 and 63.77%63.77\% with 1,024, after which increasing the carrier set does not improve accuracy and substantially increases communication. We therefore use the M=1,024M=1,024 configuration in the cross-dataset comparison in Table 3. This result complements the raw-image experiments in the main paper. Although Tiny ImageNet is itself an image dataset, the carrier channel here does not probe the frozen visual representation extractor. Instead, the random representations directly probe the trainable downstream layers. The preference for independent uniform carriers is therefore consistent with the broader hypothesis that carrier geometry should match the function and representation interface through which knowledge is communicated. Robustness to the available training data. We additionally repeat the matched uniform-versus-blurred comparison after reducing the Tiny ImageNet training pool from 50k to 30k samples. Table 8 reports the resulting accuracies. We focus on M≥512M≥ 512, for which both carrier distributions were evaluated. Table 8: Tiny ImageNet representation-space carrier comparison with a reduced 30k-sample training pool. Results are mean ± standard deviation. M Uniform (%) Blurred (%) 512 61.40±0.4561.40± 0.45 57.45±0.2557.45± 0.25 1,024 62.04±0.2862.04± 0.28 58.15±0.1058.15± 0.10 2,048 61.67±0.3661.67± 0.36 58.28±0.2058.28± 0.20 4,096 61.34±0.3661.34± 0.36 58.69±0.1958.69± 0.19 Reducing the training pool lowers absolute accuracy, as expected, but preserves the carrier ordering at every matched budget. Uniform carriers outperform blurred carriers by approximately 2.72.7–4.04.0 percentage points across the sweep. The consistency of this effect under a different amount of training data provides additional evidence that the observed preference is associated with the representation-space carrier interface rather than with a particular training-set size. C.6 Complete teacher-output entropy diagnostics Table 9 reports the complete output-entropy measurements available on CIFAR-10. Besides the private 4,096-carrier configuration, we evaluate larger weighted shared-carrier settings. For teacher probability vector pip_i on carrier xix_i and p¯=M−1∑i=1Mpi p=M^-1 _i=1^Mp_i, we use Hquery=1MlogC∑i=1MH(pi),Hmarg=H(p¯)logC,H_query= 1M C _i=1^MH(p_i), H_marg= H( p) C, where C is the number of classes. The first quantity measures average target softness; the second measures aggregate coverage of the teacher’s output classes. Table 9: Teacher-output diagnostics on CIFAR-10. HqueryH_query is the normalized mean per-carrier class entropy and HmargH_marg is the normalized entropy of the mean class distribution. Results are mean ± standard deviation. Model Protocol M Carrier HqueryH_query (%) HmargH_marg (%) ResNet9 Private 4,096 Uniform 31.11±4.7931.11± 4.79 75.95±1.9175.95± 1.91 ResNet9 Private 4,096 Blurred 43.31±2.4743.31± 2.47 84.33±2.2084.33± 2.20 ResNet9 Private 4,096 Fourier 38.65±0.0438.65± 0.04 79.68±4.4879.68± 4.48 ResNet9 Shared-w. 32,768 Uniform 41.42±4.4641.42± 4.46 53.49±5.3253.49± 5.32 ResNet9 Shared-w. 32,768 Blurred 49.40±0.9949.40± 0.99 91.98±1.3391.98± 1.33 ResNet9 Shared-w. 32,768 Fourier 42.82±4.3442.82± 4.34 88.05±1.8688.05± 1.86 MLP Private 4,096 Uniform 75.62±1.3175.62± 1.31 95.42±0.4395.42± 0.43 MLP Private 4,096 Blurred 54.76±8.3354.76± 8.33 90.70±1.4290.70± 1.42 MLP Private 4,096 Fourier 48.87±2.9148.87± 2.91 92.89±0.9592.89± 0.95 MLP Shared-w. 16,384 Uniform 89.61±0.8389.61± 0.83 99.41±0.0599.41± 0.05 MLP Shared-w. 16,384 Blurred 66.99±3.7766.99± 3.77 97.24±0.1797.24± 0.17 MLP Shared-w. 16,384 Fourier 61.26±7.6461.26± 7.64 96.97±0.4596.97± 0.45 The larger shared-carrier experiment strengthens the ResNet9 observation. At M=32,768M=32,768, uniform carriers exhibit substantially lower marginal output entropy (0.5350.535) than blurred (0.9200.920) or Fourier (0.8810.881), matching the large performance gap among these configurations. For the MLP, uniform carriers instead achieve the highest marginal entropy in both the private and shared-weighted settings. We emphasize that these statistics should be interpreted as diagnostics rather than sufficient conditions for successful transfer. In particular, high per-query entropy may simply indicate uncertain teacher outputs. Marginal entropy is more directly related to the extent to which the carrier set elicits outputs across the available class space, but it does not by itself measure whether those outputs encode all functional directions required for distillation. C.7 Ablation on the number of ghost outputs Ghost outputs are additional logit dimensions with no associated task class and no direct supervision. Although they do not affect the task prediction, their values on random carriers are included in the communicated output vector and can therefore provide additional dimensions through which local model behavior is expressed. The controlled MNIST and CIFAR-10 input-space experiments use three ghost outputs by default. Figure 4 reports the corresponding sensitivity analysis on MNIST. Increasing the number of ghost outputs generally improves the attainable accuracy, but also increases the amount of synthetic information communicated. Since this mechanism is orthogonal to the carrier-distribution and carrier-interface questions studied in Sec. 3.2, we fix it to three for the corresponding input-space experiments. The Tiny ImageNet representation-space experiments use no ghost outputs and are reported separately in Sec. C.5. Figure 4: Accuracy–communication trade-off as the number of ghost outputs is varied on MNIST. Marker size is proportional to the number of ghosts. Error bars report the standard error.