Paper deep dive
See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents
Siyi Chen, Xiaoyan Zhang, Meng Wu, Jonathan Tremblay, Valts Blukis, Stan Birchfield, Rene Vidal, Alvaro Velasquez, Sijia Liu, Qing Qu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/9/2026, 5:20:27 AM
Summary
This paper addresses the inefficiency and information loss of text-based communication in multi-agent systems by proposing dense latent communication via heterogeneous KV-cache alignment. Through compressed-sensing analysis, the authors reveal a duality where context-aware transfer relies on sparse reasoning signals, while context-unaware transfer requires dense knowledge preservation. They introduce a two-phase training framework with lightweight cross-model cache transformation, demonstrating superior performance and 2-3x lower compute compared to text communication across Qwen3 models.
Entities (10)
Relation Signals (10)
Qwen3-4B → usedas → Sender/Receiver agents
confidence 99% · Across all six directions of {Qwen3-4B, 8B, 14B} and six in-domain and out-of-domain benchmarks
Dense alignment → enables → Heterogeneous KV-cache communication
confidence 98% · we propose dense alignment for heterogeneous KV-cache communication via a lightweight cross-model cache transformation
Context-aware transfer → drivenby → Sparse reasoning signals
confidence 95% · context-aware transfer is driven by sparse reasoning signals
Text communication → hasdrawback → Lossy and expensive decode/re-encode
confidence 95% · paying a lossy and expensive decode and re-encode cost.
Proposed Method → outperforms → Prior heterogeneous baselines
confidence 95% · our method outperforms prior heterogeneous baselines
Proposed Method → reducescomputevs → Text communication
confidence 95% · matches or exceeds text communication in context-aware settings at roughly 2 to 3 times lower compute
Context-unaware transfer → requires → Dense contextual knowledge preservation
confidence 95% · context-unaware transfer, where the receiver sees no input, requires dense contextual knowledge preservation.
Heterogeneous agents → requires →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-agent systems communicate mostly through text, paying a lossy and expensive decode and re-encode cost. KV-cache communication is a promising alternative, yet most prior work is homogeneous, using duplicate copies of the same model, and avoids the central challenge of cross-model latent alignment; existing heterogeneous methods are also restrictive, typically assuming shared input and using transferred caches mainly for steering. We study a more fundamental question: can heterogeneous agents be aligned well enough to perform real "mind reading" and transfer both what one agent sees and how it thinks? Our information-structure analysis reveals a duality: context-aware transfer is driven by sparse reasoning signals, while context-unaware transfer, where the receiver sees no input, requires dense contextual knowledge preservation. Motivated by this, we propose dense alignment for heterogeneous KV-cache communication via a lightweight cross-model cache transformation and two-phase training: reconstruction followed by generation. Across all six directions of {Qwen3-4B, 8B, 14B} and six in-domain and out-of-domain benchmarks, our method outperforms prior heterogeneous baselines, matches or exceeds text communication in context-aware settings at roughly 2 to 3 times lower compute, and remains effective in context-unaware transfer where prior methods collapse.
Tags
Links
- Source: https://arxiv.org/abs/2606.13594v1
- Canonical: https://arxiv.org/abs/2606.13594v1
Trouble viewing inline? Open PDF directly →
Full Text
72,583 characters extracted from source content.
Expand or collapse full text
See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents Siyi Chen 1,2 , Xiaoyan Zhang 1 , Meng Wu 1 , Jonathan Tremblay 2 , Valts Blukis 2 , Stan Birchfield 2 , Rene Vidal 3 , Alvaro Velasquez 4 , Sijia Liu 5 , Qing Qu 1 1 University of Michigan· 2 NVIDIA· 3 University of Pennsylvania· 4 University of Colorado Boulder· 5 Michigan State University Abstract Multi-agent systems communicate mostly through text, paying a lossy and expensive decode and re-encode cost. KV-cache communication is a promising alternative, yet most prior work is homogeneous, using duplicate copies of the same model, and avoids the central challenge of cross-model latent alignment; existing heterogeneous methods are also restrictive, typically assuming shared input and using transferred caches mainly for steering. We study a more fundamental question: can heterogeneous agents be aligned well enough to perform real “mind reading” and transfer both what one agent sees and how it thinks? Our information-structure analysis reveals a duality: context-aware transfer is driven by sparse reasoning signals, while context-unaware transfer, where the receiver sees no input, requires dense contextual knowledge preservation. Motivated by this, we propose dense alignment for heterogeneous KV-cache communication via a lightweight cross-model cache transformation and two-phase training: reconstruction followed by generation. Across all six directions of Qwen3-4B, 8B, 14B and six in-domain and out-of-domain benchmarks, our method outperforms prior heterogeneous baselines, matches or exceeds text communication in context-aware settings at roughly 2 to 3×lower compute, and remains effective in context-unaware transfer where prior methods collapse. Keywords: Multi-agent, Latent Communication, Heterogeneous, Alignment Date: June 12, 2026 Correspondence: siyich@umich.edu Resources: Project page TFLOPs per Example↓ Accuracy (%) ↑ Context-AwareContext-Unaware Accuracy-Computation Trade-off Settings Methods Text-to-Text Cache-to-Cache Ours Context-Aware Context-Unaware Figure 1: See what I see, know what I think. We study real latent mind reading across heteroge- neous agents, in which one agent can read both what another agent sees and what it thinks. Guided by our latent communication information structure analysis, we learn dense alignment between agents and evaluate context-aware and context-unaware settings. Dense alignment is accurate and efficient in both regimes, surpassing sparse-steering heterogeneous baselines (cache-to-cache) while using less compute than text communication. 1 arXiv:2606.13594v1 [cs.MA] 11 Jun 2026 Contents 1 Introduction3 2 Background and Problem Setup5 2.1 Latent MAS Communication via KV-Cache . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Latent Communication: Homogeneous vs. Heterogeneous Multi-Agent Systems . . 6 2.3 Context-Aware vs. Context-Unaware Latent Communication . . . . . . . . . . . . . . 6 3 The Information Bottleneck: Sparse Reasoning vs. Dense Knowledge7 3.1 Compressed-Sensing Analysis of Information Bottlenecks . . . . . . . . . . . . . . . . 7 3.2 Sparse Reasoning and Dense Knowledge in KV-Cache Communication . . . . . . . . 8 4 Design of Dense Latent Communication10 4.1 Two-Phase Training for Dense and Actionable Alignment . . . . . . . . . . . . . . . . 10 4.2 Architecture Design: Heterogeneous Dense Cache Alignment . . . . . . . . . . . . . 11 5 Experiments13 5.1 Context-aware Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.2 Context-unaware Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.3 Latent Space Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 6 Conclusion15 A Phase-I Trace Construction19 B Efficiency Analysis and Per-Side Breakdown19 B.1 Measurement recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.2 Per-side breakdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.3 Structural observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 C Compressed-Sensing Analysis Across Regimes23 C.1 Self-communication setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.2 Stage 1: CS head ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.3 Stage 2: 퐾-sweep on full test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C.4 Random-filter baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.5 Recovery-limit caveat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.6 Context-Aware and Context-Unaware Results . . . . . . . . . . . . . . . . . . . . . . . 24 C.6.1 Context-aware regime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.6.2 Context-unaware regime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C.6.3 Takeaway for model design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2 1 Introduction LLM-based multi-agent systems (MAS) increasingly rely on specialized agents, such as planners, retrievers, executors, and verifiers, to solve problems beyond the reach of a single model [Tra+25]. Frameworks like AutoGen [Wu+24] and MetaGPT [Hon+24] operationalize this promise through role assignment and workflow-based collaboration. Yet even as recent work improves coordination [Che+25; Wan+25; Zha+25a; Wan+26; Zha+25b; Zha+26], modern MAS still communicates predominantly through text. This text bottleneck is flexible and interpretable, but it forces a decode and re-encode cycle at every handoff, introducing information loss, substantial generation overhead, and limited access to rich latent intermediate representations [Zhe+26; Du+25; Zou+25]. This bottleneck motivates latent communication as a more efficient and interpretable alternative to text-based message passing [Yu+26]. Instead of exchanging decoded natural language, agents directly share internal representations. By transmitting embedding-level signals [Pha+24], hidden state trajectories [Du+25; RL25; Tan+25; Fei+25; Yan+26], or key-value (KV) caches [Shi+25; Fu+25; Jin+26; LAD26; Liu+24], latent methods reduce decoding overhead, enable computation reuse, and expose richer context for collaboration [Shi+25; LAD26; Zou+25; Zhe+26]. Among these representations, KV cache communication has emerged as a particularly compelling approach. By directly transmitting the Key (퐾) and Value (푉) tensors from transformer attention layers, this method serves as an instant, pre-computed memory injection for the receiving agent. Rather than parsing a lossy text summary, the receiver seamlessly integrates the sender’s dense, sequence- level contextual state. KV caches are especially well-suited for latent communication because they directly participate in attention during decoding and support selective transmission, compression, projection, and fusion, preserving task-relevant context while minimizing communication costs [Shi+25; Fu+25; LAD26]. Existing work, however, leaves two fundamental challenges underexplored. First, latent com- munication is largely restricted to homogeneous agents, i.e., replicas of the same model whose latent representations are naturally aligned. In contrast to natural language, which provides a shared symbolic interface, transferring KV caches across heterogeneous agents is non-trivial due to differences in layer depth, head structure, channel geometry, and positional encoding [Liu+24; Fu+25; RL25]. Recent efforts address heterogeneous communication via signal fusion [Fu+25] when agents observe the same input. In contrast, we directly learn KV-cache alignment and impose no such constraint. Second, these existing works typically evaluate the channel under a context-aware regime, where the receiver retains access to the original question context [Fei+25; Pha+24; Zhe+26]. In this setting, the transmitted latent message merely steers reasoning over information the receiver already possesses, allowing the signal to be partial or lossy. It remains unclear whether the latent channel can carry the input itself densely enough across heterogeneous architectures. This leads to our fundamental question: Can heterogeneous agents be aligned well enough for real latent “mind reading,” to transfer both what one agent sees and how it thinks? In contrast to existing works, our method supports direct information transfer in context-unaware settings, where the receiver solves the task solely from the sender’s transmitted latents. This demonstrates the potential of dense alignment between heterogeneous agents for knowledge transfer, allowing the receiver to reuse the sender’s latent representations instead of re-encoding the original context, thereby improving computational efficiency. 3 SenderAgent ReceiverAgent ReasoningOnly Communication Sparse reasoning signals partially are transmitted HeterogeneousKV Cache Alinement Sparse Alignment Dense Alignment Full-information Communication Bothsparse reasoning and dense contextare transmitted PartialReasoning High Efficiency MissingContext RelativelyEfficient RichContext RobustReasoning LowInfoDensity HighInfoDensity LowBandwidth High Bandwidth Figure 2: Sparse vs. dense heterogeneous alignment. Prior sparse methods partially align reasoning (mainly in context-aware transfer) and do not preserve dense context. Our dense alignment maps sender caches into receiver-compatible caches to support both robust reasoning and dense context transfer across context-aware and context-unaware regimes. Contribution of this work. We revisit the information requirement of latent communication through compressed-sensing analysis over KV caches (Section 3) in two regimes, context-aware (receiver sees the input) and context-unaware (receiver sees no input), and find a clear duality: context-aware transfer is sparse in reasoning signal, while context-unaware transfer requires dense contextual knowledge preservation. Motivated by this, we propose dense alignment for heteroge- neous latent communication with a lightweight cross-model KV-cache adapter, positional disentan- glement, fine-grained per-head transformation and selection, and two-stage training (reconstruction then generation). In context-aware settings, our dense alignment surpasses existing sparse-steering heterogeneous baselines while running at 2–3×lower compute than text communication, and it remains effective in the harder context-unaware setting where existing baselines fail. Our core contributions are summarized as follows: • Compressed-sensing analysis and context-unaware communication: We provide compressed- sensing analysis and a stricter context-unaware protocol, showing that latent communication is sparse in reasoning signal but dense in knowledge transfer. • Dense alignment across heterogeneous models: We introduce a dense alignment framework that enables direct KV-cache transfer across heterogeneous models while preserving both reasoning and contextual information. • Efficient and information-preserving transfer: In context-aware communication, dense align- ment surpasses existing sparse-steering heterogeneous baselines while also being more compute- efficient than text communication; it further enables robust transfer in the challenging context- unaware regime. Organization. The remainder of this paper is organized as follows. Section 2 introduces back- ground on latent MAS communication, including KV-cache representations and the distinction between homogeneous and heterogeneous agents. Section 3 presents the compressed-sensing anal- ysis, which reveals the sparse-versus-dense nature of information transfer across context-aware and context-unaware regimes. Section 4 describes the dense alignment framework for heterogeneous 4 latent communication composed of the cross-model KV-cache adapter, positional disentanglement, per-head transformations with gating, and the two-stage reconstruction-then-generation training strategy. Section 5 evaluates our method across in-domain and out-of-domain benchmarks under both communication regimes. Finally, Section 6 concludes the paper and discusses future directions. 2 Background and Problem Setup In this section, we introduce the basic setups of latent MAS communication and distinct communi- cation regimes. 2.1 Latent MAS Communication via KV-Cache Large Language Models (LLMs) are increasingly deployed in multi-agent systems (MAS), where multiple specialized agents collaborate to solve complex tasks that exceed the capabilities of a single model [Tra+25]. Formally, letℳ =풜 1 ,풜 2 , ... ,풜 푛 denote a MAS consisting of푛distinct agents, where풜 푘 represents the푘-th specialized agent. Such systems typically comprise role- specific entities, such as planners, retrievers, executors, and verifiers, that dynamically exchange information and coordinate actions. For simplicity, let us consider a representative two-agent setting to study fundamental communication challenges. Existing MAS typically rely on text-based communication [Wu+24; Che+25; Wan+25; Zha+25a; Wan+26]: the sender decodes its internal states into discrete text tokens, which are then transmitted and re-encoded by the receiver. While natural and model-agnostic, this protocol incurs substantial overhead from autoregressive decoding and redundant receiver-side computation. As illustrated in Figure 3, latent MAS communication instead transmits intermediate latent states, such as KV caches [Zou+25; Fu+25], bypassing explicit text generation and reducing communication overhead. Given an input sequence푿 =(풙 1 , ... , 풙 푁 )of푁tokens, a transformer processes the sequence through퐿layers. At each layer푙, token representations are projected into query, key, and value states. Specifically, for each attention headℎ ∈ 1, ... , 퐻, the input sequence is mapped to its corresponding head matrices: 푸 (푙,ℎ) = 푿 (푙−1) 푾 (푙,ℎ) 푄 , 푲 (푙,ℎ) = 푿 (푙−1) 푾 (푙,ℎ) 퐾 , 푽 (푙,ℎ) = 푿 (푙−1) 푾 (푙,ℎ) 푉 , where푿 (푙−1) ∈R 푁×푑 model represents the hidden states from the previous(푙 −1)-th layer, and 푾 푄 , 푾 퐾 , 푾 푉 ∈R 푑 model ×푑 head are the head specific projection weights. The scaled dot product at- tention mechanism then computes a weighted context matrix by evaluating token dependencies across a softmax-normalized similarity matrix: Attention(푸 (푙,ℎ) , 푲 (푙,ℎ) , 푽 (푙,ℎ) ) = softmax 푸 (푙,ℎ) (푲 (푙,ℎ) ) ⊤ √ 푑 head 푽 (푙,ℎ) . In autoregressive generation, computing the attention matrix requires context from all preceding tokens. To avoid the prohibitive푂(푁 2 )recomputation of past key and value vectors at each decoding step, these matrices are preserved within a key-value (KV) cache. During the prefill stage, the sender and receiver construct their respective KV caches by storing the key and value tensors produced at every transformer layer and attention head: 풞 푆 (푿) =(푲 (푙,ℎ) 푆 (푿), 푽 (푙,ℎ) 푆 (푿)) 푙,ℎ , 풞 푅 (푿) =(푲 (푙,ℎ) 푅 (푿), 푽 (푙,ℎ) 푅 (푿)) 푙,ℎ ,(1) These caches serve as compact summaries of the previously processed context and eliminate the need to recompute historical attention states during autoregressive decoding. 5 Sender ! " Text Tokens Decoding Receiver ! # Re-encoding Text Communication Latent Communication KVCache No Token Decoding No Re-encoding ! ! Cache Transformation $ % (') ) $ * (') Sender ! " Receiver ! # Figure 3: Text communication decodes and re-encodes mes- sages between agents, whereas latent communication trans- fers KV caches directly for computation reuse. KV caches provide a natural in- terface for latent communication. As shown in Figure 3, the sender풜 푆 can directly transmit its precomputed KV cache tensors풞 푆 (푿)to the re- ceiver풜 푅 . Rather than exchanging information through decoded text, the receiver can directly consume the sender’s internal representations, forming the basis for latent commu- nication between language models. The key challenge is latent space alignment, where different instances of the same model naturally share an aligned latent space, but this is not the case for heterogeneous models. 2.2 Latent Communication: Homogeneous vs. Heterogeneous Multi-Agent Systems As shown in recent works [Du+25; Tan+25; Shi+25; Jin+26; Zou+25], the latent communication framework described in Section 2.1 is natively supported in homogeneous MAS, where all agents inℳshare identical model architectures, layer configurations, and hidden dimensions. Given this architectural symmetry, the key and value tensors generated by the sender풜 푆 are mapped directly to the attention blocks of the receiver풜 푅 . This dimensional alignment enables zero-shot state sharing via direct tensor copy operations, completely bypassing the need for cross-model projection, feature alignment, or latent space transformations. However, this direct transfer paradigm fails when applied to heterogeneous MAS, where individual agents inℳdiffer significantly in their internal architectures, capabilities, information access, or foundational designs. Unlike natural language, which acts as a universal symbolic interface, latent representations and key-value (KV) caches are strictly tied to a model’s internal coordinate space and optimization landscape. Consequently, these representations are severely misaligned across heterogeneous architectures, rendering direct injection functionally incoherent without explicit alignment [RL25; Fu+25]. Existing efforts have begun to address latent-space mismatch across heterogeneous agents, but important limitations remain. One line of work learns projection modules in the relatively simple text-embedding space [Du+25; Yan+26], rather than aligning the denser and more expressive KV-cache representations. Another line uses cache-fusion networks [Fu+25], but assumes that agents process the same input, limiting their flexibility in general communication settings. Finally, several methods rely on aggressive latent compression or sparsification [Fu+25; Shi+25] to simplify alignment. While sparse latents can be sufficient for transmitting high-level reasoning signals in context-aware settings, they may discard the dense contextual information that KV caches are natu- rally suited to carry. Moreover, such sparsification can introduce inefficient communication patterns, where fixed-size latent channels carry little useful information after many components are pruned or suppressed. In contrast, we propose a lightweight and efficient dense-alignment framework that enables heterogeneous agents to exchange KV-cache representations while preserving both reasoning-relevant and context-rich information. 2.3 Context-Aware vs. Context-Unaware Latent Communication To characterize the information dynamics within latent multi-agent channels, we formalize two distinct communication regimes based on the informational availability at the receiver side. Crucially, 6 while prior literature on latent communication has almost exclusively operated within the context- aware paradigm, this work highlights and investigates the significant yet under-explored context- unaware setting. •Context-Aware Communication: The receiver agent풜 푅 has access to the original input context푿 and uses it together with the transferred, aligned cache e 풞 푅 (푿)for generation:푃(풚| 푿, e 풞_푅(푿)). • Context-Unaware Communication: The receiver agent풜 푅 has no access to the source context푿 and must generate solely from the transferred latent representations: 푃(풚| e 풞 푅 (푿)). The distinction between these two regimes changes the role of the latent channel. In context- aware communication, the cache serves primarily as a reasoning signal: the receiver can still consult and re-process the original context푿. In context-unaware communication, the cache must instead serve as a self-contained information carrier: it is the receiver’s only access to source-side information. It must therefore transmit both contextual evidence and reasoning state, enabling the receiver to see what the sender sees and know what the sender thinks. This makes context-unaware communication both scientifically and practically important. Scientifically, it tests whether dense latent alignment can preserve task-critical knowledge across heterogeneous models. Practically, it reduces redundant context re-encoding, enables efficient model switching via transferred KV states, and supports deployments where source inputs cannot be shared. Overall, context-unaware communication exposes a stronger requirement for latent MAS: the transferred cache must faithfully carry the dense information on its own. 3 The Information Bottleneck: Sparse Reasoning vs. Dense Knowledge To understand the intrinsic information structure of latent communication signals, we conduct a sys- tematic post-hoc compressed-sensing analysis of KV caches in a homogeneous self-communication setting. By using architecturally identical sender and receiver models, we eliminate cross-model misalignment as a confounding factor and isolate the information carried by the transmitted la- tent representations. This analysis identifies the minimal subset of KV components needed for accurate reasoning communication in context-aware settings, while also informing lightweight, information-preserving alignment designs for heterogeneous agents. We evaluate two communication regimes. In the context-aware regime, the receiver has access to the original input, allowing effective performance with only a small, sparse subset of the KV cache. In the more challenging context-unaware regime, the receiver has no access to the input and must rely entirely on the transmitted latent cache, requiring a much denser fraction of the KV cache to preserve task-critical information. Together, these findings reveal a fundamental duality: latent communication is sparse in reasoning but dense in knowledge, motivating the heterogeneous dense alignment framework we developed in Section 4. 3.1 Compressed-Sensing Analysis of Information Bottlenecks To quantify the contribution of individual components in the sender’s KV cache and identify the minimal information required for effective latent communication, we employ a post-hoc compressed- sensing (CS) framework. We consider a homogeneous self-communication setup,풜 푆 →풜 푅 , where both the sender풜 푆 and receiver풜 푅 use the same Qwen3-4B model. The sender transformer contains 퐻attention heads across퐿layers. Due to Grouped-Query Attention (GQA), each layer contains퐺 Key-Value (KV) groups, with each KV group shared by 퐶 query heads. 7 1.Head Groups in Sender 퐺attention head groups in each layer Layerℓ Layer퐿 1 Layer1 2G ... 12G ... ... ... ... 12G ... Total Heads:퐻=퐿×퐺×퐶 2.Compressed Measurements Randomly mask out some heads and then evaluate task performance (avg. Acc) Mask1 횽 !,: ... Performance 푦 ! Mask2 횽 $,: ... 푦 " MaskM 횽 %,: 푦 # Ablate(zero out) Keep Model: 풚 , = 풚−푦 ! =횽휶+흐 ... 횽∈0,1 !×# : Measurement Matrix ... ... 휶∈ℝ # : Head Importance 3.Sparse Recovery (LASSO) Solve head importance scores 휶with Lasso 휶 ( =푎푟푔푚푖푛 1 2 풚 2 −횽휶 $ $ +휆휶 % Importance score 훼 # & Head index (sorted by 훼# & ) Then aggregate the importance score per head group 4.SelectTop-K Head Groups Choose the top-K head groupswith the highest importance scores Top-Khead groups indices 푖 " , 푖 # , 푖 $ ... 푖 % ... Layer ℓ Layer 퐿 Layer 1 ... ... ... ... ... ... Selected(remain) Not Selected 퐾<퐿×퐺 Irrelevant Heads 퐶heads per group 푦 & ∈ℝ: Baseline Performance Figure 4: Compressed-sensing head selection: random ablation masks estimate sender-head impor- tance, which is aggregated to KV-group scores and used to keep top-퐾groups for communication. We first sample푁data points from a benchmark and evaluate the full-cache latent commu- nication performance, denoted by푦 0 , measured as mean accuracy over these examples. We then generate푀random binary retention masks, represented by a matrix횽∈0,1 푀×퐻 , where each row corresponds to one ablation experiment. The entryΦ 푖푗 =1 indicates that the푗-th sender head is retained in the푖-th experiment, whileΦ 푖푗 =0 indicates that it is ablated. For each mask, we evaluate the downstream task on the same푁examples and record the resulting performance, producing an observation vector풚∈R 푀 . We center these observations by subtracting the full-cache baseline: ̃풚 = 풚−푦 0 . We assume that each head contributes independently to the final communication quality and denote these unknown contributions by the coefficient vector휶∈R 퐻 . Recovering휶from the observed performances can then be formulated as the following Lasso regression problem: ˆ 휶 = arg min 휶 1 2푀 ∥ ̃풚−횽휶∥ 2 2 + 휆∥휶∥ 1 ,(2) where larger-magnitude positive coefficients indicate higher task relevance. Finally, we aggregate the per-head coefficients within each KV group to obtain the score for the푔-th KV group in layer푙: score (푙,푔) KV = Õ ℎ∈group(푔) ˆ 훼 푙,ℎ .(3) This procedure isolates the importance of KV components in a setting free from cross-model misalignment, providing a clear measurement of the information bottleneck. We then use the resulting CS-derived scores to perform KV-group pruning sweeps under both context-aware and context-unaware regimes. For comparison, we also evaluate random pruning, showing that the CS-derived ranking more effectively identifies the KV components most critical for task performance. 3.2 Sparse Reasoning and Dense Knowledge in KV-Cache Communication Our post-hoc CS analysis across multiple benchmarks reveals a fundamental contrast between context-aware and context-unaware latent communication, reflecting the underlying information structure of the communicated KV-cache signals. 8 Figure 5: Sparse reasoning signal vs. dense context signal (Qwen3-4B self-communication). Accuracy is plotted against the number of KV groups kept (퐾of 288). Solid blue: context-aware CS filtering, where the receiver still sees the input;퐾 =0 is the single-agent receiver baseline. Open blue squares: random KV-group selection. Solid red: context-unaware CS filtering, where the receiver relies only on transmitted KV caches. Context-aware communication reaches near-ceiling accuracy with few KV groups, suggesting a sparse reasoning signal. In contrast, context-unaware communication requires dense context transfer, staying near chance until퐾 >150 and approaching the ceiling only at 퐾 = 250 (87% of the cache). As shown in Figure 5, in the context-aware setting, the receiver has access to the original input context, and retaining only a small fraction of the KV cache suffices to achieve near-full performance. This indicates that the channel primarily conveys a sparse reasoning signal, as the receiver’s local context already provides core task information. Despite the small absolute information volume required, the redundancy is highly structured: CS-derived rankings significantly outperform uniform random pruning at low densities, highlighting that correctly identifying the functional sub-networks of attention heads is critical. In practice, only the statistically reliable, high-importance KV groups recovered by CS are considered, as lower-ranked groups collapse to the noise floor and carry no meaningful contribution. These findings suggest that context-aware evaluation provides a relatively weak test of latent communication: success depends on isolating and prioritizing the sparse subset of heads carrying reasoning signals, rather than uniformly transmitting the cache. In contrast, the context-unaware regime removes the receiver’s access to the input, forcing it to rely entirely on the transmitted latent representations. Here, the information bottleneck shifts dramatically: performance remains at chance or zero for most levels of dense compression, exhibiting a sharp phase transition as more KV groups are retained. Unlike the sparse reasoning case, successful communication in this setting requires a dense fraction of the cache to preserve both contextual knowledge and reasoning structure. Because this regime directly tests whether the latent channel faithfully conveys task-critical information without external context, it imposes stricter constraints on alignment fidelity and motivates architectural mechanisms to preserve dense knowledge while remaining computationally efficient. These observations directly inform the design of our heterogeneous latent communication framework, which addresses the dual demands of sparse reasoning and dense knowledge: •Two-Stage Training for Density Preservation: Phase I enforces reconstruction of the KV cache to preserve dense contextual and reasoning information, while Phase I optimizes downstream generation to make the dense signal actionable for the receiver. • Per-Head Transformations with Learnable Gating: Each query head is transformed individually with a learnable gate, allowing the model to recover the structural importance of heads identified 9 by CS analysis and dynamically prioritize sparse reasoning signals. • Positional Disentanglement: Rotary positional embeddings are explicitly stripped prior to transformation and restored afterward, ensuring that positional information does not interfere with content-aligned latent transfer. 4 Design of Dense Latent Communication The analysis in Section 3 reveals two complementary aspects of the information structure in latent communication. First, reasoning information is sparse: only a small set of KV groups can provide the high-level guidance needed to steer a receiver that already has access to the source context. Second, knowledge is dense: when the transferred cache must stand in for the source context itself, task- critical information is distributed across a much larger fraction of the cache. These two properties motivate a communication interface that is both density-preserving and structurally selective. As illustrated in Figure 6, the proposed interface transforms the sender-side KV cache into a receiver- compatible cache through structured cache operations, including position disentanglement, layer alignment, head transformation, and information selection. Rather than designing a homogeneous compression mechanism and later extending it across models, we directly learn a heterogeneous dense-alignment interface that maps a sender cache into a receiver-compatible cache. Concretely, for a sender agent풜 푆 and a receiver agent풜 푅 , we learn a cache transformation 풯 휽 that maps the sender KV cache풞 푆 (푿), defined in Section 2.1, into a receiver-compatible cache e 풞 푅 (푿) =풯 휽 (풞 푆 (푿)) . The transformation is trained and parameterized so that the same interface can serve both regimes: in context-aware communication, e 풞 푅 (푿)provides a structured reasoning signal alongside the receiver’s access to푿; in context-unaware communication, it must act as a dense surrogate for the missing source context. 4.1 Two-Phase Training for Dense and Actionable Alignment A heterogeneous adapter trained only through the receiver’s final generation loss is undercon- strained: many transformed caches may lead to the correct answer on a training example while failing to preserve the source information in a reusable receiver-native form. We therefore separate training into two phases. Phase I learns dense latent alignment by reconstructing the receiver’s own cache. Phase I then tunes the aligned cache for downstream generation. Phase I: receiver-cache reconstruction. For paired inputs, we run both agents on the same source context 푿 and obtain the sender and receiver caches 풞 푚 (푿) = n 푲 (푙,푔) 푚 (푿), 푽 (푙,푔) 푚 (푿) o 푙,푔 , 푚 ∈푆, 푅,(4) where푙indexes transformer layers and푔indexes KV groups, consistent with the GQA notation in Section 3.1. The adapter produces e 풞 푅 (푿) =풯 휽 (풞 푆 (푿))and is optimized to reconstruct the receiver’s own cache: ℒ rec = Õ 푙,푔 e 푲 (푙,푔) 푅 − 푲 (푙,푔) 푅 2 2 + e 푽 (푙,푔) 푅 − 푽 (푙,푔) 푅 2 2 .(5) This phase teaches the adapter to express sender information in the receiver’s latent language. It is especially important for context-unaware communication: if풜 푅 cannot access푿, then the transformed cache must serve as a dense surrogate for what the receiver would otherwise have encoded from 푿 itself. 10 1 " # 23 ... Sender& ' Layer 1 1 " ( 23 ... Layer 2 1 " ( 23 ... Layer ) ( 1 " ( 23 ... ... SenderKVCache * ( (,) Layer1 Layer2 Layer) ( Head1Head2Head" ( . ( (/,1) 2 ( (/,1) ... ... ... ... Layer 1 1 " # 23 ... Layer 2 Layer ) # ... SenderKVCache 3 * # (,) Layer1 Layer2 Layer) # Head1Head2Head" # ... ... ... ... Receiver& 4 1 " # 23 ... Cache Transformation 5 6 Position Disentanglement LayerAlignment Head Transformation Information Selection 7 * ( Input Prompt ... Generated Answer ... . ( (/,8) 2 ( (/,8) . ( (/,9 : ) 2 ( (/,9 : ) . # (/,1) 2 # (/,1) . # (/,8) 2 # (/,8) . # (/,9 ; ) 2 # (/,9 ; ) Figure 6: Overview of dense alignment: transform sender KV caches into receiver-compatible caches with positional disentanglement, structured per-head transformation, and two-phase training. Phase I: generation-oriented communication. After reconstruction pretraining, we optimize the transformed cache for downstream generation under both communication regimes. For each training example, the receiver-side context푿 푅 is drawn from a mixture of context-aware and context-unaware prompts, 푿 푅 = ( 푿, context-aware, ∅, context-unaware, (6) and the same adapter풯 휽 is updated across both cases: ℒ gen =− Õ 푡 log 푝 풜 푅 푦 푡 | 푦 <푡 , e 풞 푅 (푿), 푿 푅 ,(7) This joint Phase-I training is important because the two regimes stress different aspects of the same latent channel: context-aware examples teach the cache to act as a structured reasoning signal alongside the receiver’s prompt, while context-unaware examples force it to preserve enough dense knowledge to replace the missing source context. This phase makes the aligned cache actionable: the receiver must not only host a cache that resembles its own internal states, but also use that cache to generate the correct output. Together, the two phases implement the main lesson from Section 3: dense reconstruction preserves task knowledge, while mixed-regime generation tuning calibrates one shared transformation for both operating regimes. 4.2 Architecture Design: Heterogeneous Dense Cache Alignment The training objective above defines what풯 휽 should achieve. We now describe how풯 휽 is parame- terized to respect the structure of heterogeneous transformer caches. In a heterogeneous MAS,풞 푆 and풞 푅 may differ in depth, hidden dimension, KV-group organization, and positional convention. Direct cache injection is therefore ill-defined: even if the tensor shapes can be forced to match, the sender cache is expressed in the wrong coordinate system for the receiver. Position-disentangled cache transformation. Rotary positional embeddings entangle content with model-specific phase rotations. Since dense communication should align the information stored in the cache rather than copy the sender’s positional convention,풯 휽 first maps caches into a position-disentangled space: b 풞 푚 = RemoveRoPE 푚 ( 풞 푚 ) , 푚 ∈푆, 푅.(8) 11 The cross-model transformation is applied to b 풞 푆 to produce a receiver-side content cache풞 ′ 푅 . The final communicated cache restores the receiver’s positional convention: e 풞 푅 = AddRoPE 푅 풞 ′ 푅 .(9) This makes position handling an architectural component of the interface: the adapter aligns content in a shared position-disentangled space, then returns a cache that can be consumed by the receiver’s attention blocks. Layer alignment across different depths. Let퐿 푆 and퐿 푅 denote the sender and receiver depths. For each receiver layer푙, we pair it with a sender layer through a monotonic depth-preserving map 푎(푙) = round 푙(퐿 푆 − 1) 퐿 푅 − 1 , 푙 = 0, ... , 퐿 푅 − 1.(10) This mapping aligns early, middle, and late representations while avoiding a free routing problem over all layer pairs. The design reflects a useful inductive bias: although two models may have different depths, their computations still progress from local/token-level features toward more task-level abstractions. KV-group transformation with structured gates. After layer alignment, each receiver KV group is produced from the corresponding sender-side content cache: 푲 ′(푙,푔) 푅 , 푽 ′(푙,푔) 푅 = 훾 (푙,푔) · 푇 (푙,푔) 퐾,휽 b 푲 (푎(푙),휋 푙 (푔)) 푆 ,푇 (푙,푔) 푉,휽 b 푽 (푎(푙),휋 푙 (푔)) 푆 ,(11) where휋 푙 (푔)maps receiver KV group푔to a sender KV group, and훾 (푙,푔) ∈[0,1]is a learnable gate. When the two models have the same KV-group layout,휋 푙 reduces to the identity map. In our final architecture,푇 (푙,푔) 퐾,휽 and푇 (푙,푔) 푉,휽 are separate per-KV-group head-dimension MLPs applied token-wise to key and value vectors: 푇 (푙,푔) ★,휽 (풛) = 푾 (푙,푔) ★,2 휎 푾 (푙,푔) ★,1 풛+ 풃 (푙,푔) ★,1 + 풃 (푙,푔) ★,2 ,(12) where★∈퐾,푉,휎(·)is GELU,풛 ∈R 푑 푆 is a sender key or value vector,푾 (푙,푔) ★,1 ∈R 16푑 푆 ×푑 푆 expands the head dimension by a factor of 16, and푾 (푙,푔) ★,2 ∈R 푑 푅 ×16푑 푆 projects into the receiver head dimension 푑 푅 . Separate MLP parameters are used for keys, values, and each routed receiver-layer/KV-group pair. The KV-group-wise parameterization is motivated by the structure revealed in Section 3. Sparse reasoning signals are not uniformly distributed across the cache; they concentrate in particular heads or KV groups. At the same time, context-unaware transfer cannot collapse the channel to only a few sparse groups because dense contextual knowledge is distributed broadly. The gate therefore should not be viewed as a homogeneous compression trick. It is a structured reliability weight inside a dense heterogeneous channel: it lets the model emphasize high-utility reasoning subspaces without abandoning the dense information needed when the receiver cannot see the source context. 12 PairMethod In-domainOOD TFLOPs↓ GSM8K MATH-500 ARC-C MMLU-Redux MedQA OpenBookQA 4B→ 8B Receiver-only 81.1049.2091.0072.1053.0091.2019.85 T2T88.1076.0091.7480.7567.4090.4037.73 C2C77.8644.2086.0975.8756.8785.604.50 Ours92.9582.0093.6978.5267.2491.2012.56 4B→ 14B Receiver-only 83.7046.4092.6071.8064.7091.8031.16 T2T92.3477.8092.0082.8771.1792.0056.24 C2C82.3444.2092.4372.7663.0087.006.64 Ours93.8686.0094.2078.5771.9693.6021.54 8B→ 4B Receiver-only 82.4044.2089.2065.0047.7088.009.18 T2T89.3963.2090.9679.3966.4687.0033.91 C2C72.4837.4086.7870.1955.0777.203.43 Ours91.8183.4093.0077.8666.3089.607.95 8B→ 14B Receiver-only 83.7046.4092.6071.8064.7091.8031.16 T2T93.9375.0091.7484.1372.6692.8067.08 C2C82.2643.2092.4373.8164.6587.007.42 Ours94.0985.0094.3777.3870.4693.4021.79 14B→ 4B Receiver-only 82.4044.2089.2065.0047.7088.009.18 T2T90.6060.2089.8380.6169.9185.0043.48 C2C70.5836.6085.8369.7353.4276.405.01 Ours91.1382.6091.8977.6663.0088.8010.18 14B→ 8B Receiver-only 81.1049.2091.0072.1053.0091.2019.85 T2T91.5873.0092.3583.4372.9090.2055.50 C2C76.3543.0089.3975.9962.6985.607.83 Ours92.9581.4093.7778.0470.3892.6015.64 Table 1: Multi-task context-aware heterogeneous communication results on in-domain and out- of-domain benchmarks. FLOPs are average total inference cost per example in TFLOPs, including sender, receiver, and adapter computation when applicable (see Appendix B.2). 5 Experiments Training Data. All cache transformation models are trained on a mixture of GSM8K [Cob+21], MATH [Hen+21] (algebra subset), and ARC-Challenge [Cla+18]. Phase I requires only paired sender and receiver cache states on the same source context. For Phase I, we construct receiver self-guided reasoning traces, where the receiver model generates step-by-step solutions on the training split using the ground-truth answer as guidance. Consequently, for a pair풜 푆 →풜 푅 , the supervision target follows the reasoning style of the receiver rather than that of the sender or an external teacher. Additional details are provided in Appendix A. Experimental Setup. We evaluate our learned cache transformation across all six directions of the Qwen3-4B, 8B, 14B [Yan+25a] pair set. Evaluation is performed on three in-domain tasks (GSM8K, MATH-500, ARC-C) and three held-out MCQ benchmarks (MMLU-Redux [Gem+25], MedQA [Yan+25b], OpenBookQA [Mih+18]). We compare three baselines against our method: Receiver-only (single-agent with the receiver model), T2T (text-based communication), and C2C [Fu+25] (learned cache transformation via steering). Both regimes from Section 3 are evaluated: context-aware, where the receiver also sees the question, and context-unaware, where the receiver only sees the transferred signal. Inference TFLOPs are reported under a 2-parameters-per-token estimator (Appendix B.1). 13 PairMethod In-domainOOD TFLOPs↓ GSM8K MATH-500 ARC-C MMLU-Redux MedQA OpenBookQA 4B→ 8B T2T-context-unaware51.6374.4019.4821.0018.8524.0037.86 C2C-context-unaware1.903.0022.5221.918.9627.0014.94 Ours-context-unaware 91.4378.8091.3874.5961.8288.809.42 4B→ 14B T2T-context-unaware56.7975.2023.3923.0627.6527.2060.83 C2C-context-unaware0.000.0010.7010.2612.8016.4020.05 Ours-context-unaware 82.2670.6086.8662.8653.2682.4017.15 8B→ 4B T2T-context-unaware27.9869.4023.7425.1822.9423.2032.75 C2C-context-unaware0.382.4010.358.207.2310.0014.76 Ours-context-unaware 91.3681.6093.6077.0664.5790.006.56 8B→ 14B T2T-context-unaware30.8670.4022.9622.8027.8127.4072.06 C2C-context-unaware0.000.007.397.146.367.609.84 Ours-context-unaware 81.5865.0088.4854.9257.7483.8015.85 14B→ 4B T2T-context-unaware18.5766.6020.6124.5221.1320.2041.95 C2C-context-unaware0.680.404.966.110.636.6017.93 Ours-context-unaware 82.4964.0088.4867.1559.5485.408.90 14B→ 8B T2T-context-unaware18.8066.4022.8721.4125.2226.2053.80 C2C-context-unaware0.000.202.785.345.182.2024.77 Ours-context-unaware 84.1568.4088.9967.8454.2088.0013.28 Table 2: Multi-task context-unaware heterogeneous communication results on in-domain and out-of-domain benchmarks. Only the sender observes the original input; the receiver relies solely on the communicated signal. T2T-context-unaware transmits a natural-language message, while C2C-context-unaware and Ours-context-unaware transmit latent KV caches. 5.1 Context-aware Results Table 1 reports context-aware results, where the receiver still has the question in its prompt and the channel acts as an auxiliary signal. Our learned cache transformation matches or exceeds T2T on every in-domain task across all six pairs (GSM8K+0.16 to+4.85 p; MATH-500+6.00 to+20.40 p; ARC-C+1.34 to+3.94 p), and is competitive on the held-out OOD benchmarks (within∼5 p of T2T on MMLU-Redux/MedQA, and at-or-above T2T on OpenBookQA in 5 of 6 pairs). C2C underperforms our method on all tasks, confirming the advantage of our dense alignment over sparse reasoning-signal steering in the same context-aware setting. The efficiency story is just as strong. Our channel runs 2–3×cheaper than T2T in TFLOPs (e.g., 4B→14B: 21.5 vs 56.2; 8B→14B: 21.8 vs 67.1), and is cheaper than even the bare Receiver-only baseline in 5 of 6 directions, because (i) the sender does no autoregressive reasoning (ℓ 푆 dec =0) and (i) the receiver does not re-encode a long natural-language sender message but instead attends to a compact transferred cache. So the channel is simultaneously more accurate and cheaper than the natural-text alternative. 5.2 Context-unaware Results Table 2 reports the context-unaware regime: only the sender observes the question, and the receiver must produce the answer from the transferred signal alone. This is the strict test of channel information density. Without our dense transformation, context-unaware communication essentially fails. T2T-context-unaware drops the receiver to 19–57% on GSM8K and to MCQ-chance on ARC- C/MedQA/OpenBookQA, because the sender’s free-form text was generated for a question-aware reader and does not preserve task content end-to-end. C2C-context-unaware performs even worse: accuracy collapses to 0–2% on every reasoning task, indicating that steering-based KV transfer fails to preserve or align the contextual information needed by the receiver. 14 Figure 7: PCA of KV cache latents: transformed sender caches directly overlap receiver-native manifolds, indicating dense geometric alignment rather than sparse shortcuts used for steering. In stark contrast, our channel sustains accuracy within 0–10 p of its own context-aware num- bers across all six pairs and all six benchmarks (GSM8K 81–91, MATH-500 64–82, ARC-C 87–94, MMLU-Redux 55–77, MedQA 53–65, OpenBookQA 82–90). The TFLOPs are even lower than the corresponding context-aware row, since the receiver’s prefill shrinks further with no question text to encode. The 8B→4Bpair illustrates the headline efficiency win: Ours-context-unaware reaches 91.4/81.6/93.6 on the in-domain triple at 6.6 TFLOPs – cheaper than the bare 4B receiver (9.2 TFLOPs) and within 1–2 p of T2T context-aware at 5×fewer FLOPs. Dense knowledge transfer through the learned latent channel is therefore both feasible and substantially more efficient than the natural-text alternative. 5.3 Latent Space Visualization To check that our quantitative results reflect a genuine geometric alignment rather than a brittle decoder-fooling shortcut, we project per-token value vectors of the receiver’s cache space onto their first two principal components (Figure 7) for an early layer (L7) and a late layer (L34). On identical inputs, the transformed sender cache lies on the same manifold as the receiver’s native cache at both depths, while the untransformed sender cache (not shown for clarity) occupies a disjoint region. The learned transformation푇 휃 thus maps sender activations into the receiver’s geometry rather than producing a representation that only the trained receiver can decode. 6 Conclusion We studied heterogeneous latent communication through a simple question aligned with our title: can one agent transfer both what it sees and how it thinks to another agent. Our compressed-sensing analysis shows a duality: latent channels are sparse in reasoning for context-aware communication, but dense in knowledge for context-unaware communication where the receiver sees no input. This clarifies why prior evaluations in mostly context-aware settings are insufficient. Motivated by this structure, we proposed dense alignment with three components: per-head transformation and gating, position-disentangled alignment, and two-phase reconstruction-then-generation training. Across all six directions of Qwen3-4B, 8B, 14B, our method surpasses sparse-steering heteroge- neous baselines, matches or exceeds text communication in context-aware settings at 2–3×lower 15 compute, and remains accurate in context-unaware settings where prior baselines collapse. Limitations and future work. Our method currently needs one training pass per sender-receiver pair; scaling to open-set pairings and shared transforms across many senders is a key next step. The context-unaware regime also raises privacy questions, since transferring task content without the prompt changes what can be inferred from the channel. References [Bai+26] Anish Bair et al. “Identifying Functional Heads of Large Language Models with Compressed Sensing”. In: arXiv preprint arXiv:2603.03335 (2026). [Che+25]Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, and Maosong Sun. “Optima: Optimizing effectiveness and efficiency for llm-based multi-agent system”. In: Findings of the Association for Computational Linguistics: ACL 2025. 2025, p. 11534– 11557. [Cla+18]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. “Think you have solved question answering? try arc, the ai2 reasoning challenge”. In: arXiv preprint arXiv:1803.05457 (2018). [Cob+21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. “Train- ing verifiers to solve math word problems”. In: arXiv preprint arXiv:2110.14168 (2021). [Du+25]Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyong Zhu, Yu Cheng, Bo Zheng, Wei Chen, and Haochao Ying. “Enabling agents to communicate entirely in latent space”. In: arXiv preprint arXiv:2511.09149 (2025). [Fei+25]Jacob Fein-Ashley, Dhruv Parikh, Rajgopal Kannan, and Viktor Prasanna. “Mixture of thoughts: Learning to aggregate what experts think, not just what they say”. In: arXiv preprint arXiv:2509.21164 (2025). [Fu+25] Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. “Cache-to-cache: Direct semantic communication between large language models”. In: arXiv preprint arXiv:2510.03215 (2025). [Gem+25]Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. “Are we done with mmlu?” In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025, p. 5069–5096. [Hen+21]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. “Measuring mathematical problem solving with the math dataset”. In: arXiv preprint arXiv:2103.03874 (2021). [Hon+24]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Steven Yau, Zijuan Lin, Liyang Zhou, et al. “MetaGPT: Meta programming for a multi-agent collaborative framework”. In: International Conference on Learning Representations. Vol. 2024. 2024, p. 23247–23275. [Jin+26]Haibo Jin, Kuang Peng, Ye Yu, Xiaopeng Yuan, and Haohan Wang. “Agent Primi- tives: Reusable Latent Building Blocks for Multi-Agent Systems”. In: arXiv preprint arXiv:2602.03695 (2026). 16 [LAD26]Yiping Li, Zhiyu An, and Wan Du. “When Less Latent Leads to Better Relay: Information- Preserving Compression for Latent Multi-Agent LLM Collaboration”. In: arXiv preprint arXiv:2604.13349 (2026). [Liu+24]Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, et al. “DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving”. In: arXiv preprint arXiv:2411.02820 (2024). [Mih+18]Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. “Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”. In: EMNLP. 2018. [Pha+24] Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan Plummer, Zhaoran Wang, and Hongxia Yang. “Let models speak ciphers: Multiagent debate through embeddings”. In: International Conference on Learning Representations. Vol. 2024. 2024, p. 51899–51928. [RL25]Vignav Ramesh and Kenneth Li. “Communicating activations between language model agents”. In: arXiv preprint arXiv:2501.14082 (2025). [Shi+25]Xiangyu Shi, Marco Chiesa, Gerald Q Maguire Jr, and Dejan Kostic. “KVComm: Enabling Efficient LLM Communication through Selective KV Sharing”. In: arXiv preprint arXiv:2510.03346 (2025). [Tan+25]Yichen Tang, Weihang Su, Yujia Zhou, Yiqun Liu, Min Zhang, Shaoping Ma, and Qingyao Ai. “Augmenting Multi-Agent Communication with State Delta Trajectory”. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025, p. 10230–10251. [Tra+25]Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. “Multi-agent collaboration mechanisms: A survey of llms”. In: arXiv preprint arXiv:2501.06322 (2025). [Wan+25] Zhexuan Wang, Yutong Wang, Xuebo Liu, Liang Ding, Miao Zhang, Jie Liu, and Min Zhang. “Agentdropout: Dynamic agent elimination for token-efficient and high- performance llm-based multi-agent collaboration”. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, p. 24013–24035. [Wan+26] Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, et al. “Rema: Learning to meta-think for llms with multi-agent reinforcement learning”. In: Advances in Neural Information Processing Systems 38 (2026), p. 126621–126667. [Wu+24] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. “Autogen: Enabling next-gen LLM applications via multi-agent conversations”. In: First conference on language modeling. 2024. [Yan+25a] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. “Qwen3 technical report”. In: arXiv preprint arXiv:2505.09388 (2025). 17 [Yan+25b]Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. “Llm-medqa: Enhancing medical question answering through case studies in large language models”. In: 2025 International Joint Conference on Neural Networks (IJCNN). IEEE. 2025, p. 1–8. [Yan+26]Xiyuan Yang, Jiaru Zou, Rui Pan, Ruizhong Qiu, Pan Lu, Shizhe Diao, Jindong Jiang, Hanghang Tong, Tong Zhang, Markus J Buehler, et al. “Recursive multi-agent systems”. In: arXiv preprint arXiv:2604.25917 (2026). [Yu+26]Ye Yu, Heming Liu, Haibo Jin, Xiaopeng Yuan, Peng Kuang, and Haohan Wang. “Learn- ing to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems”. In: arXiv preprint arXiv:2604.21794 (2026). [Zha+25a] Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Yu, and Tianlong Chen. “Cut the crap: An economical commu- nication pipeline for llm-based multi-agent systems”. In: International Conference on Learning Representations. Vol. 2025. 2025, p. 75389–75428. [Zha+25b]Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. “Aflow: Automating agentic workflow generation”. In: International Conference on Learning Representations. Vol. 2025. 2025, p. 34040–34077. [Zha+26]Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, and James Zou. “Sirius: Self-improving multi-agent systems via bootstrapped reasoning”. In: Advances in Neural Information Processing Systems 38 (2026), p. 124475–124504. [Zhe+26] Yujia Zheng, Zhuokai Zhao, Zijian Li, Yaqi Xie, Mingze Gao, Lizhu Zhang, and Kun Zhang. “Thought communication in multiagent collaboration”. In: Advances in Neural Information Processing Systems 38 (2026), p. 123389–123418. [Zou+25]Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, et al. “Latent collaboration in multi-agent systems”. In: arXiv preprint arXiv:2511.20639 (2025). 18 A Phase-I Trace Construction Phase I trains the cache transformation with a supervised generation loss, so each source context must be paired with a receiver-side target output. We construct these targets as receiver-self guided traces: for each sender–receiver pair풜 푆 →풜 푅 , the trace generator is the receiver model풜 푅 itself. This choice avoids teaching the receiver to imitate a different model’s wording or reasoning style after cache transfer; the adapter instead learns to produce receiver-native latent states that decode into outputs the receiver is already well suited to generate. Trace-generation tasks. The Phase-I trace pool contains three in-domain training tasks: GSM8K, MATH-algebra, and ARC-Challenge. These are the same tasks used for multitask Phase-I training in the main experiments. MATH-500, MMLU-Redux, MedQA, and OpenBookQA are not used for Phase-I trace generation; they are held for evaluation, with MATH-500 replacing MATH-algebra as the reported mathematical reasoning benchmark. Guided trace generation. For each training example(푿, 풚), we prompt the receiver model with the source question and the ground-truth answer, then ask it to produce a step-by-step solution that arrives at that answer. The resulting JSONL record contains the question, the gold answer, the generated solution trace, and a trace-mode tag. For multiple-choice tasks such as ARC-Challenge, the trace is relabeled to end with the canonical answer formatAnswer: X, matching the receiver- side evaluation prompt. This relabeling prevents the training target from using a format that the evaluator later rejects or scores inconsistently. Pair-specific receiver traces. Because the trace generator is the receiver, different cross-model directions use different trace files. For example, 14B→4B uses Qwen3-4B-self traces, while 4B→14B uses Qwen3-14B-self traces. These traces are generated on the training split with guided decoding and then matched back to training examples by question text before Phase-I optimization. Mixed-regime receiver prompts. During Phase I, we train the same adapter on both context- aware and context-unaware receiver prompts. We set the receiver prompt to be context-aware with probability 0.5 and context-unaware with probability 0.5. Equivalently, the receiver-side context variable in Section 4 is sampled as푿 푅 = 푿half of the time and푿 푅 =∅half of the time. The target trace is unchanged across these two cases; only the receiver’s direct access to the source context changes. This forces the transferred cache to support both roles identified in Section 3: a sparse reasoning signal when the receiver has the context, and a dense knowledge carrier when it does not. In all main experiments, Phase I starts from the Phase-I reconstruction checkpoint and runs for 2000 optimization steps with cross-entropy on the receiver-self trace tokens. B Efficiency Analysis and Per-Side Breakdown We measure system-level compute efficiency under each method’s canonical inference recipe and break the FLOPs column of Tables 1 and 2 into its four constituent per-side token counts: sender prefill ℓ 푆 pre , sender decode ℓ 푆 dec , receiver prefill ℓ 푅 pre , and receiver decode ℓ 푅 dec . 19 B.1 Measurement recipe Sample composition. For each (pair, method, mode) cell we sample 17 examples from each of six benchmarks (GSM8K, MATH-500, ARC-Challenge, MMLU-Redux, OpenBookQA, MedQA) using a fixed seed, pooled to≈102 samples per cell. This is small enough that we report only the mean per token-count field; per-task and p50/p95 breakdowns are released as a CSV companion to this paper. Inference recipe. All methods are evaluated under a shared decoding setup with fixed sampling parameters and the same token budget, using single-sample inference to avoid batch-padding artifacts in latency and token counts. Receiver-only and T2T use their standard text-generation configuration, and Ours uses the same reasoning style as in training. Each method is evaluated in its strongest canonical configuration for a fair comparison. FLOPs estimate. We use the standard “2 parameters per token” rule of thumb for the forward pass: FLOPs ≈ 2푁 푆 (ℓ 푆 pre +ℓ 푆 dec ) + 2푁 푅 (ℓ 푅 pre +ℓ 푅 dec ), where푁 푆 , 푁 푅 are the (non-embedding) parameter counts of the sender and receiver models. The attention term’s푂(퐿 kv )contribution is excluded; at our reasoning lengths it is below 5% of the linear-projection total and including it does not change any ranking in the headline tables. Token-count instrumentation. Token counts are exact, not estimated from rendered text. We record generated token IDs directly and compute per-sample prefill/decode lengths from model outputs. Bandwidth disclosure. The KV-cache payload for Ours is approximately 20–30 MB per sample, whereas an equivalent text message is on the order of hundreds of bytes. This is a network- bandwidth axis, not a compute axis, so it is excluded from the headline TFLOPs comparison. B.2 Per-side breakdown Table 3 reports the four per-side token counts plus the unweighted totalAll tokfor every (pair, method, mode) cell. The structural mechanism behind each method’s compute profile is then visible. Both Ours and C2C eliminate the sender’s autoregressive reasoning (ℓ 푆 dec =0); the receiver prefill is small for both (ℓ 푅 pre =220 for Ours, 117 for C2C in w/ctx mode) – substantially below T2T’s ℓ 푅 pre ∼1,100–1,300 tokens of re-encoded sender text. The remaining gap between Ours and C2C is in the receiver decode length: Ours produces explicit step-by-step reasoning before the answer (ℓ 푅 dec ≈400–500 in w/ctx mode), whereas C2C emits a more terse answer (ℓ 푅 dec ≈80–170). This trades fewer FLOPs for substantially lower accuracy on most benchmarks (see Tables 1 and ??). B.3 Structural observations Three observations follow from Table 3. (i) Both Ours and C2C eliminate sender decoding. T2T spends∼900–1,080 sender-decode tokens producing a∼2,048-token sender CoT. Ours instead encodes the question into a sender KV cache without autoregressive sender decoding (ℓ 푆 dec =0); C2C similarly transfers a cache without sender generation. 20 (i) Receiver prefill is small for both cache-based methods,∼6×smaller than T2T. Ours has ℓ 푅 pre = 220 (w/ctx) / 63 (context-unaware); C2C has 117 / 14. Both stand in sharp contrast to T2T’s ℓ 푅 pre ∼ 1,100–1,300 tokens of re-encoded sender message: the receiver no longer needs to ingest the sender’s natural-language reasoning through its embedding pipeline. (i) The remaining FLOPs gap between Ours and C2C is in receiver decode, not in the commu- nication mechanism. In w/ctx mode Ours produces explicit step-by-step reasoning before the answer (ℓ 푅 dec ∼400–500); C2C emits a more terse output (ℓ 푅 dec ∼80–170). The lower decode length translates to a∼2–3×smaller TFLOPs total for C2C w/ctx – but at the cost of substantially lower accuracy on most benchmarks (Tables 1 and 2). In context-unaware mode C2C’s receiver decode swings wildly across pairs (ℓ 푅 dec ∈[270,1,817]); Ours keeps a stable decode profile across all six pairs (ℓ 푅 dec ≈ 400), and Ours-context-unaware is the cheapest cell in 5 of 6 pairs. B.4 Notes Receiver-only in 14B→4B. In the one direction with a 4B receiver, the bare Receiver-only baseline (4B alone,∼994 receiver-decode tokens) costs 9.18 TFLOPs. Adding our 14B sender prefill (∼184 tokens) makes Ours marginally more expensive at 10.18 TFLOPs in w/ctx mode, although Ours- context-unaware drops back to 8.90. In every other direction Ours is cheaper than Receiver-only. Accuracy / FLOPs Pareto. C2C achieves the lowest TFLOPs by emitting short answers, but its accuracy on most benchmarks trails Receiver-only and Ours by a wide margin (e.g., 14B→4B GSM8K: C2C 70.58% vs. Ours 91.13%; 4B→8B MATH-500: C2C 44.20% vs. Ours 82.00%). Ours sits between C2C and T2T on the FLOPs axis while matching or beating T2T on accuracy in nearly every cell, so the FLOPs-vs-accuracy Pareto frontier of Table 1 is dominated by Ours: it strictly beats T2T (lower FLOPs and higher accuracy on most cells) and strictly beats C2C on accuracy at moderate extra FLOPs cost. 21 PairMethodModeSpre Sdec Rpre Rdec All tok TFLOPs 4B→ 8B Receiver-onlyw/ctx00 154 1,0871,24119.85 T2Tw/ctx180 1,033 1,255 4972,96537.73 T2T-context-unaware context-unaware 180 1,077 1,180 5582,99537.86 C2Cw/ctx1170 117 1063404.50 C2C-context-unaware context-unaware 117014 86199314.94 Oursw/ctx1840 220 47387712.56 Ours-context-unaware context-unaware 184063 4336819.42 4B→ 14B Receiver-onlyw/ctx00 154 9591,11331.16 T2Tw/ctx180 1,020 1,242 4232,86656.24 T2T-context-unaware context-unaware 180 1,052 1,155 6663,05360.83 C2Cw/ctx1170 117873216.64 C2C-context-unaware context-unaware 117014 66880020.05 Oursw/ctx1840 220 49790121.54 Ours-context-unaware context-unaware 184063 49774417.15 8B→ 4B Receiver-onlyw/ctx00 154 9941,1479.18 T2Tw/ctx180 1,074 1,296 4332,98433.91 T2T-context-unaware context-unaware 180 1,037 1,139 5202,87732.75 C2Cw/ctx1170 117773113.43 C2C-context-unaware context-unaware 117014 1,5971,72814.76 Oursw/ctx1840 220 4058107.95 Ours-context-unaware context-unaware 184063 3886366.56 8B→ 14B Receiver-onlyw/ctx00 154 9591,11331.16 T2Tw/ctx180 1,066 1,288 3962,93067.08 T2T-context-unaware context-unaware 180 1,073 1,175 6823,11172.06 C2Cw/ctx1170 117813157.42 C2C-context-unaware context-unaware 117014 2704029.84 Oursw/ctx1840 220 45385721.79 Ours-context-unaware context-unaware 184063 39864515.85 14B→ 4B Receiver-onlyw/ctx00 154 9941,1479.18 T2Tw/ctx180 923 1,145 4302,67743.48 T2T-context-unaware context-unaware 180 886 988 5252,57941.95 C2Cw/ctx1170 117993335.01 C2C-context-unaware context-unaware 117014 1,8171,94817.93 Oursw/ctx1840 220 40781110.18 Ours-context-unaware context-unaware 184063 4056528.90 14B→ 8B Receiver-onlyw/ctx00 154 1,0871,24119.85 T2Tw/ctx180 890 1,112 4832,66655.50 T2T-context-unaware context-unaware 180 887 989 5052,56253.80 C2Cw/ctx1170 117 1674027.83 C2C-context-unaware context-unaware 117014 1,3291,46124.77 Oursw/ctx1840 220 43583915.64 Ours-context-unaware context-unaware 184063 44469213.28 Table 3: Per-side efficiency breakdown. Mean per-sample token counts on the sender (Spre / Sdec = prefill / decode) and receiver (Rpre / Rdec) sides, pooled across the six benchmarks of Tables 1 and 2.All tok=ℓ 푆 pre +ℓ 푆 dec +ℓ 푅 pre +ℓ 푅 dec (every token that touches either model). TFLOPs computed via the formula in Appendix B.1. Bold marks the lowest TFLOPs cell within each pair (across all methods + modes). 22 C Compressed-Sensing Analysis Across Regimes Section 3 uses post-hoc compressed sensing (CS) [Bai+26] to identify which sender heads carry the channel’s task-relevant signal and then evaluate accuracy as a function of how many of those heads are kept. This section documents the setup and results behind Figure 5. C.1 Self-communication setup Sender and receiver are the same Qwen3-4B model (homogeneous, identity KV pass-through): the sender encodes the question into a KV cache with zero sender decoding, and the receiver consumes that cache directly. Qwen3-4B has 36 layers×32 query heads=1152 query heads, organized into 36×8=288 KV groups under GQA (each KV head shared across 4 query heads). All ablations target sender-side transmitted KV components, while receiver attention is unchanged. From query-head importance to KV-group importance. The CS measurements (Stage 1 below) perturb individual query heads, so the Lasso problem is naturally posed at the 1152-query-head granularity. The transmitted KV cache itself, however, has only 288 KV groups. To go from one to the other, we sum the 4 query-head Lasso coefficients within each GQA group: score (푙,ℎ KV ) KV = Õ ℎ 푄 ∈ group(ℎ KV ) ˆ 푥 푙,ℎ 푄 , which gives a per-(layer, KV-head) importance score; Stage 2 keeps the top-퐾 of these. C.2 Stage 1: CS head ranking We take푀 =200 binary ablation masksΦ ∈ 0,1 푀×푁 over the푁 =1152 query heads, with each row stratified to zero exactly 5% of heads (≈58 heads per mask). For each mask푚, we run the full sender→receiver pipeline on 100 evaluation samples and record the resulting metric푦 푚 (accuracy on GSM8K / MATH-algebra / ARC-Challenge), then center it by the unablated baseline, ̃ 푦 푚 = 푦 푚 − 푦 baseline . We then solve a Lasso regression for the per-head importance vectorx∈R 푁 : ˆx = arg min x∈R 푁 1 2푀 ̃y− Φx 2 2 + 훼∥x∥ 1 ,훼 = 10 −4 , (an intercept is also fit but omitted from the display), and rank heads by− ˆ 푥 푖 (most-negative ˆ 푥 푖 indicates highest importance, since masking it most degrades accuracy). C.3 Stage 2: 퐾-sweep on full test Using the per-head ranking from Stage 1, we evaluate accuracy on the full test split when only the top-퐾KV groups (out of 288) are retained in the transmitted cache and the rest are zeroed. We sweep 퐾 ∈10, 20, 50, 100, 150, 288 under two regimes: • Context-aware: the receiver receives the standard prompt including the question text, on top of the (filtered) sender KV. This is the regime evaluated by most prior work [Du+25; Shi+25; Fu+25]. •Context-unaware: the receiver’s prompt omits the question, so the filtered sender KV is the only task signal it sees. The two regimes share the same head ranking and the same퐾levels; only receiver access to the input differs. 23 C.4 Random-filter baseline For the random-filter markers in Figure 5, we replace the CS-Lasso ranking with a uniform random selection of퐾KV groups, repeat with 3 seeds, and report the mean. The CS-vs-random gap measures how much the head ranking is doing beyond uniform sparsity. C.5 Recovery-limit caveat The recovery limit applies to the underlying Lasso problem, not to the aggregated KV-group ranking. With푀 =200 measurements over푁 =1152 query heads, standard CS recovery bounds give푠 max ≈ 푀/log 2 (푁/푠)≈70–80 informative coefficients; query-head coefficients past Lasso rank ∼80 are essentially zero. After GQA aggregation to 288 KV groups, this means at most∼70 KV groups inherit a meaningful ranking, and the remaining∼220 KV groups have aggregated scores that are tiebreaks over zeros and an arbitrary layer-imbalanced order. Consequently, at퐾 =200 (i.e., keeping all but∼90 KV groups – well into the noise-floor region) CS and random schemes converge and a small inversion appears. We therefore restrict the headline figure to퐾 ≤150 on the CS-Lasso side, where the ranking is statistically reliable. C.6 Context-Aware and Context-Unaware Results C.6.1 Context-aware regime In the context-aware regime (solid blue curve in Figure 5), retaining only퐾 =10 KV groups of 288 already matches the full-KV ceiling on all three tasks: GSM8K [Cob+21] 0.883 (vs. 0.917 at 퐾=288), ARC-Challenge [Cla+18] 0.873 (vs. 0.880), and MATH-algebra [Hen+21] 0.699 (vs. 0.698). Compared to the single-agent receiver at퐾 =0, the channel’s actual lift over the receiver’s own reasoning is at most∼10 p (MATH-algebra) and under 2 p (ARC-Challenge). The receiver’s own input already supplies the task content; the channel only needs to carry a small reasoning signal on top. The redundancy is, however, structured rather than uniform. CS-Lasso head selection meaningfully outperforms random selection at small퐾: at퐾=50, CS reaches 0.876/0.862/0.577 on the three tasks vs. 0.605/0.583/0.484 for random (+27/+28/+9 p gaps). On MATH-algebra at 퐾=20, random selection (0.559) actually drops below the receiver-only baseline (0.597) – unguided pruning actively hurts even when the receiver has the prompt. The absolute information needed is small, but which bits are kept matters substantially. C.6.2 Context-unaware regime When receiver access to the prompt is removed (solid red curve), the same compression behaves very differently: the channel becomes the only task signal, and the profile shifts from a plateau to a sharp transition zone. Across all three tasks, accuracy stays near chance for퐾≤150 (≤52% of the cache kept), rises sharply between퐾=150 and퐾=200 (e.g., ARC-Challenge 0.27→0.79, GSM8K 0.00→0.28), and approaches the channel ceiling by 퐾=250 (87% kept; 0.834/0.904/0.686, close to 퐾=288 values 0.880/0.901/0.694). Below the transition, chance-level and zero-level points reflect failure modes rather than meaningful performance gradations. This is the core sparse-vs-dense contrast. Compared against??, where퐾=10 (∼3.5% of the cache) was already sufficient under context-aware evaluation, the context-unaware regime requires roughly퐾=250 – a∼25×swing in how much of the channel must be preserved depending on whether the receiver has the input. Crucially, the channel does not need to carry all of the cache to function under context-unaware evaluation (퐾=250≈ 퐾=288); it needs an information-preserving fraction of it, which is much larger than the small reasoning steer that suffices when the input is already with the receiver. 24 C.6.3 Takeaway for model design Context-aware evaluation is a weak test of latent channels because the receiver can compensate with its own input. By contrast, context-unaware evaluation is a strict test of information transfer and alignment fidelity, since the channel is the only task signal. This sparse-vs-dense dichotomy directly motivates our heterogeneous design (Section 4): two-phase training preserves dense information and makes it decoder-actionable, while per-head transformation and gating capture structured sparse reasoning signals. For the dense regime, two-phase training pairs (i) Phase I cache reconstruction, which forces the transformation to preserve receiver-equivalent information density, with (i) Phase I generation training, which makes that dense signal decoder-actionable. For the sparse regime, we parameterize푇 휃 at per-head granularity with a lightweight learnable gate per head, letting the model recover from data the same head-importance structure CS-Lasso recovers post-hoc. Position is treated separately via RoPE strip/restore, since the redundancy lives in content heads, not positional structure. 25