Paper deep dive
Capacity and Redundancy Trade-offs in Multi-Task Learning
Asif Khan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/21/2026, 3:32:29 AM
Summary
This paper introduces a Capacity-Redundancy (CR) identity for multi-task learning (MTL), decomposing predictive information into joint information, label redundancy (Total Correlation), and residual coupling (interference). It proves that negative transfer arises from limited shared capacity and weak task redundancy. The authors derive a clustering-gap decomposition to determine when clustered sharing outperforms global sharing and establish a gradient-TC bridge in Gaussian models, justifying gradient cosine similarity as a proxy for redundancy. Empirical results on clustered LoRA show significant performance gains by reducing residual coupling.
Entities (8)
Relation Signals (8)
Total Correlation → measures → Label Redundancy
confidence 96% · label redundancy defined via total correlation (TC)
Capacity-Redundancy Identity → decomposes → Predictive Information
confidence 95% · decomposes the sum of per-task predictive informations into joint predictive information... and a residual coupling term
Negative Transfer → causedby → Limited Shared Capacity
confidence 94% · negative transfer... can also be viewed as a consequence of limited shared capacity and weak task redundancy
Residual Coupling → quantifies → Interference
confidence 94% · residual coupling term that quantifies interference left unresolved by the shared representation
Gradient Cosine Similarity → servesasproxyfor → Redundancy Ordering
confidence 93% · formally justifies gradient cosine similarity as a proxy for redundancy ordering
Clustered LoRA → reduces → Residual Coupling
confidence 92% · clustered LoRA substantially reduces Δ^
Clustering-Gap Decomposition → providesconditionfor → Clustered Sharing Outperformance
confidence 91% · clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing
Clustered LoRA → outperforms → Random Partitions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity--Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (ii) a gradient--TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling $\Delta$ from validation residual correlations, showing that clustered LoRA substantially reduces $\widehat{\Delta}$, outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals.
Tags
Links
- Source: https://arxiv.org/abs/2607.16554v1
- Canonical: https://arxiv.org/abs/2607.16554v1
Trouble viewing inline? Open PDF directly →
Full Text
76,102 characters extracted from source content.
Expand or collapse full text
Capacity and Redundancy Trade-offs in Multi-Task Learning Asif Khan Harvard Medical School, Boston, USA Abstract In multi-task learning (MTL) negative transfer is often considered as an optimization artifact, but it can also be viewed as a consequence of limited shared capacity and weak task redundancy. We investigate this effect through a Capacity–Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information that includes label redundancy defined via total correlation (TC), and a residual coupling term that quantifies interference left unresolved by the shared representation. Additionally, we show two key results: (i) a clustering-gap decomposition that gives a necessary and sufficient condition for clustered sharing to outperform global sharing, and (i) a gradient–TC bridge in a Gaussian multi-task model that formally justifies gradient cosine similarity as a proxy for redundancy ordering. Empirically, we estimate the residual coupling Δ from validation residual correlations, showing that clustered LoRA substantially reduces Δ , outperforms size-matched random partitions, and results in statistically significant gains with multi-seed confidence intervals. 1 Introduction A single machine learning model trained to optimize multiple prediction tasks simultaneously is useful for improving efficiency and generalization performance [caruana1997multitask, ruder2017overview]. In a typical setting, MTL is implemented through a shared encoder with task-specific output heads, which allows information to be transferred across tasks while retaining task specialization. Such a training approach has been successfully applied in several areas including computer vision (CV) [zhang2014facial, misra2016cross], natural language processing (NLP) [collobert2008unified], and speech recognition [deng2013new]. However, despite these successes, there is a key challenge of negative transfer (when training on additional tasks degrades rather than improving performance) [wang2019characterizing]. Recent developments in pre-trained large language models (LLMs) have renewed interest in MTL in the context of parameter-efficient fine-tuning (PEFT). Methods such as adapters [houlsby2019parameter], prefix-tuning [li2021prefix, ding2023parameter], and low-rank adaptation (LoRA) [hu2022lora] make it possible to adapt massive models to new tasks with only a small number of additional parameters. A design choice here is to decide when multiple tasks should share the same lightweight module encouraging cross-task transfer, or should each task be assigned its own module to avoid interference? While previous work have explored both ends as well as various hybrids [he2021towards, karimi2021compacter, mahabadi2021parameter], our focus is on a theoretical framework that can be used to decide when to share and when to specialize model parameters. Negative transfer occurs when unrelated tasks share a low-rank subspace, whose gradient signals may conflict and cause degradation of performance across all tasks [yu2020gradient, chen2018gradnorm]. Giving each task its own private features can avoid this interference but can reduce efficiency. Hybrid approaches such as task clustering, mixture-of-adapters, or routing via task embeddings [gururangan2021demix, mudrakarta2018k], attempt to balance these trade-offs but lack a unifying theoretical justification. In this paper, we treat a shared encoder in MTL as a finite-capacity channel, given the budget I(X;Zs)≤CsI(X;Z_s)≤ C_s on the shared latent ZsZ_s. We then prove a CR inequality that shows the total predictive information a single shared latent can provide across tasks is bounded by its capacity plus the label redundancy. Thus, correlations let the same bits be reused across tasks, while weakly related tasks force competition for capacity and make negative transfer unavoidable. We extend the bound to shared–private representations with per-task budgets and derive conditional variants together with a Bayes-error lower bound that formalize when performance trade-offs cannot be avoided. Finally, we model LoRA as a capacity-constrained channel and show that the CR perspective explains how similarity-based sharing helps when task redundancy is high, whereas low-redundancy task sets require growing effective adapter capacity (rank) or private routes to prevent interference. In summary, we develop operational limits of sharing features and use it to prescribe rules for explicit capacity budgets. Our key contributions are: (i) We isolate the slack term Δ=TC(Y(1:T)∣Zs) =TC(Y^(1:T) Z_s) and interpret it as interference left after conditioning on the shared representation. (i) We derive an exact decomposition of the gain of clustered sharing over global sharing into interference reduction minus redundancy loss (Theorem˜5.3), which provides a necessary and sufficient condition for when clustering should help. (i) In a Gaussian multi-task model, we prove that gradient cosine similarity preserves the sign and (under matched noise) the ordering of label correlations, and therefore induces the same clusters as total-correlation-based redundancy (Theorem˜4.4). (iv) We show the CR bound is achieved with equality when capacity spans the task subspace and quantify the exact gap under rank constraints via TC(Y(1:T)∣Zs)TC(Y^(1:T) Z_s). These results make the intuitive capacity-versus-redundancy tradeoff quantitative: Theorem˜5.3 gives an if-and-only-if sharing rule, the Gaussian specialization identifies when the CR bound is tight, and the LoRA rank bound connects adapter capacity to the effective task subspace dimension. 2 Related Work Multi-task learning. Early work on MTL utilize a shared-backbone as well as task-specific heads to show consistent gain in performance when tasks are related [caruana1997multitask]. collobert2008unified propose a unified architecture jointly training several NLP sequence tasks. In CV, multi-task feature sharing has been used to exploit structural relatedness between tasks. For example, zhang2014facial showed improvement in facial landmark detection when jointly trained with auxiliary tasks such as head pose estimation, gender, and smile classification, since these signals share low-level visual features. Similarly, misra2016cross introduced cross-stitch networks, which explicitly learns linear combination of feature maps across tasks, thus allowing the model to adaptively decide how much representation should be shared versus kept task-specific. Beyond deep nets, a large body of regularization approaches formalizes parameter sharing via matrix norms or task relationships [evgeniou2004regularized, ando2005framework, argyriou2008convex]. We refer readers to ruder2017overview for a comprehensive survey of MTL. Parameter-efficient fine-tuning and adapters. Adapter layers are used as a light-weight alternative to full fine-tuning by inserting small bottleneck modules into frozen backbones [houlsby2019parameter]. Subsequent PEFT methods include prefix/prompt tuning [li2021prefix]. ding2023parameter, and he2021towards provide a much more detailed overviews of PEFT for LLMs. Multi-task PEFT variants allocate a shared adapter or per-task adapters, or combine both through routing, hypernetworks, or mixture strategies [mahabadi2021parameter, karimi2021compacter, mudrakarta2018k, gururangan2021demix]. While empirical heuristics guide which modules to share, principled selection criteria remains limited. Empirical analyses of task relatedness and negative transfer. standley2020tasks investigate which tasks benefit from being trained together and which cause conflicts, while zamir2018taskonomy chart the transferability between tasks through a large-scale task graph. Gradient-based analyses of task conflict and mitigation are widely used in practice [yu2020gradient, chen2018gradnorm, chai2023getmtl]. GradNorm [chen2018gradnorm], PCGrad [yu2020gradient] treat interference as a gradient conflict to be resolved step-by-step. This works well in practice but it offers no guarantee that interference can be resolved for a given capacity. Broader studies characterize and attempt to avoid negative transfer [wang2019characterizing]. These methods typically operate at the level of optimization dynamics rather than at the level of distributional limits. Theoretical generalization and representation sharing. baxter2000model formalized task families and bias learning; evgeniou2004regularized analyzed kernelized regularization for MTL; and maurer2016benefit show the benefits of learning a shared representation with task-averaged Rademacher complexity bounds. Several other works explicitly partition representations or allocate small task-specific heads to mitigate conflicts [newell2019featurepartition, wang2020smalltowers]. Some other related approaches analyze multi-task trade-offs through multi-objective optimization and controllable Pareto frontiers to show explicit preference control during inference or training [lin2020controllable, momma2022pareto]. maurer2016benefit propose a bound on generalization error and show that sharing helps but their bound loosens as tasks become dissimilar without explaining where the capacity goes. In contrast, our focus is on a distribution-level converse on the achievable predictive information through a shared latent which is independent of any estimator. Our work is complementary to adaptive-transfer and selective-sharing mechanism using clustered task structure [kang2011learning, jacob2008clustered]. denevi2022conditional, chua2021fine, and tian2025similar analyze representation or meta-learning gains through estimator-dependent rates and robustness to similar but nonidentical representations, and hanneke2022no show that multitask benefit cannot be guaranteed without structural assumptions. CR differs by giving a distribution-level converse on predictive information through a fixed-capacity latent, with the structural condition expressed as capacity versus redundancy. Multi-task information bottleneck. Multi-task variational information bottleneck (IB) I(Z;Y)−βI(Z;X)I(Z;Y)-β I(Z;X) formulations optimize multi-task IB objectives (∑tI(Z;Y(t)) _tI(Z;Y^(t))) directly [qian2020mtvib] which offers algorithmic objectives but not converses [tishby2000information, alemi2016deep]. Classical results used in our analysis include the independence bound on entropy [cover1999elements] and its conditional form and the data processing inequality [cover1999elements], total correlation [watanabe1960information] as a redundancy measure, and subset entropy inequalities [sun1975linear, madiman2010information]. Our results characterize when sharing is fundamentally limited and when structured sharing (clustering/private capacity) is provably beneficial, independent of a particular training algorithm. Figure 1: Capacity-Redundancy Tradeoff in Multi-Task Low-Rank Adaptation. (a) shows task entropies H(Y1)H(Y_1), H(Y2)H(Y_2) with overlap TC(Y)=I(Y1;Y2)TC(Y)=I(Y_1;Y_2) representing redundancy. The shared bottleneck ZsZ_s (gray) has capacity CsC_s. Blue arrows indicate task-specific information I(Zs;Yt)I(Z_s;Y_t). (b) rank-r matrices (A,B)(A,B) add to frozen weights W, where rank r≪dr d limits capacity CsC_s. Sharing (A,B)(A,B) across tasks creates the shared bottleneck. (c) estimate redundancy via gradient similarity (step 1), cluster tasks by redundancy (step 2), then allocate shared LoRA modules to redundant clusters and private modules to independent tasks (step 3), optimizing the CR bound. 3 Capacity–Redundancy Inequality Let T be the number of tasks with labels Y(1),…,Y(T)Y^(1),…,Y^(T) and input features X. A shared representation Zs=f(X)Z_s=f(X) is used across all tasks. Throughout this section, we assume the standard representation setting in which Y(1:T)−X−ZsY^(1:T)\;-\;X\;-\;Z_s forms a Markov chain (equivalently, I(Zs;Y(1:T)∣X)=0I(Z_s;Y^(1:T) X)=0), and we impose the shared capacity budget I(Zs;X)≤Cs.I(Z_s;X)\;≤\;C_s. We use total correlation as a measure of label redundancy, TC(Y(1:T))=∑t=1TH(Y(t))−H(Y(1:T))≥ 0.TC(Y^(1:T))\;=\; _t=1^TH\! (Y^(t) )\;-\;H\! (Y^(1:T) )\;≥\;0. TC(Y(1:T))=0TC(Y^(1:T))=0 iff the labels are mutually independent. The sum of per-task information carried by a shared bottleneck is always bracketed by the joint information plus redundancy. The main idea requires only the chain rule and data processing, so we focus on implication of bound and for proofs/preliminaries we refer readers to Appendix S1. Lemma 3.1 (Chain-rule sandwich). For random variables (Z,Y(1),…,Y(T))(Z,Y^(1),…,Y^(T)), I(Z;Y(1:T))≤∑t=1TI(Z;Y(t))≤ I\! (Z;Y^(1:T) )\;≤\; _t=1^TI\! (Z;Y^(t) )\;≤\; I(Z;Y(1:T))+TC(Y(1:T)). I\! (Z;Y^(1:T) )+TC\! (Y^(1:T) ). (1) Next, we combine the sandwich with the Markov chain and the capacity budget to define CR inequality. Theorem 3.2 (CR inequality). Under the Markov chain and a fixed capacity budget the following bound holds, ∑t=1TI(Zs;Y(t))≤Cs+TC(Y(1:T)). _t=1^TI\! (Z_s;Y^(t) )\;≤\;C_s+TC\! (Y^(1:T) ). (2) and the canonical sandwich Lemma 3.1 holds when Z=ZsZ=Z_s, I(Zs;Y(1:T))≤∑t=1TI(Zs;Y(t))≤ I\! (Z_s;Y^(1:T) )\;≤\; _t=1^TI\! (Z_s;Y^(t) )\;≤\; I(Zs;Y(1:T))+TC(Y(1:T)). I\! (Z_s;Y^(1:T) )+TC\! (Y^(1:T) ). (3) The upper bound in Equation˜2 is induced by an exact identity, ∑t=1TI(Zs;Y(t))=I(Zs;Y(1:T))+ _t=1^TI\! (Z_s;Y^(t) )=I\! (Z_s;Y^(1:T) )+ (TC(Y(1:T))−TC(Y(1:T)∣Zs))⏟dependence explained by Zs. (TC(Y^(1:T))-TC(Y^(1:T) Z_s) )_dependence explained by Z_s. which is equivalent to ∑t=1TI(Zs;Y(t))=I(Zs;Y(1:T))+TC(Y(1:T))−Δ _t=1^TI\! (Z_s;Y^(t) )=I\! (Z_s;Y^(1:T) )+TC(Y^(1:T))- , where Δ=TC(Y(1:T)∣Zs)≥0 =TC(Y^(1:T) Z_s)≥ 0. Here, Δ is the residual multi-task coupling that remains after observing ZsZ_s, Δ=0 =0 iff the labels become conditionally independent given ZsZ_s. When Δ is large, the right inequality in Equation˜3 can be loose, meaning the shared representation has not factorized the task joint distribution, and that additional shared or task-specific capacity (Section˜5) is required. The tightness of Equation˜2 depends on conditional independence of tasks given ZsZ_s i.e. Δ=TC(Y(1:T)∣Zs)≈0 =TC(Y^(1:T) Z_s)≈ 0, and capacity saturation I(Zs;Y(1:T))≈I(Zs;X)≈CsI(Z_s;Y^(1:T))≈ I(Z_s;X)≈ C_s. In a linear–Gaussian setting dependence is explained by ZsZ_s spanning the label-relevant subspace so that cross-task coupling disappears after conditioning, while saturation corresponds to operating at the channel capacity of the bottleneck. 4 Consequences and Impossibility Results The CR inequality (Theorem˜3.2) has a sharp consequence: without redundancy, adding tasks to a fixed-capacity bottleneck must degrade per-task performance as the number of tasks grows unless capacity scales. It further implies that the total per-task information extractable from a shared bottleneck cannot exceed shared capacity plus label redundancy ∑t=1TI(Zs;Y(t))≤Cs+TC(Y(1:T)). _t=1^TI(Z_s;Y^(t))\;≤\;C_s+TC(Y^(1:T)). Corollary 4.1 (Average per-task information). Under the assumptions of Theorem˜3.2, 1T∑t=1TI(Zs;Y(t))≤CsT+TC(Y(1:T))T. 1T _t=1^TI(Z_s;Y^(t))\;≤\; C_sT+ TC(Y^(1:T))T. (4) In particular, there exists at least one task t⋆t such that I(Zs;Y(t⋆))≤Cs+TC(Y(1:T))T.I(Z_s;Y^(t ))\;≤\; C_s+TC(Y^(1:T))T. When tasks are weakly dependent, TC(Y(1:T))TC(Y^(1:T)) does not grow with T, and Equation˜4 forces the average per-task information to decay like O(1/T)O(1/T) for fixed CsC_s. Corollary 4.2 (Linear capacity is necessary for maintaining per-task signal). Let TC(Y(1:T))≤τTC(Y^(1:T))≤τ for all T, now for a uniform per-task information level I(Zs;Y(t))≥bI(Z_s;Y^(t))≥ b for all t∈[T]t∈[T], then necessarily Cs≥Tb−τ.C_s\;≥\;Tb-τ. In particular, for independent tasks (TC=0TC=0), maintaining I(Zs;Y(t))≥bI(Z_s;Y^(t))≥ b for all t forces Cs≥TbC_s≥ Tb. Corollary 4.2 tells us when negative transfer is unavoidable. If new tasks are added without increasing shared capacity then some tasks must lose predictive signal through the shared bottleneck, unless the new tasks are redundant with existing ones. Log-loss lower bound. For each task t, let qt(⋅∣Zs)q_t(· Z_s) denote any predictor and consider the log-loss ℓt=−logqt(Y(t)∣Zs) _t=- q_t(Y^(t) Z_s). Then by the Gibbs inequality, [ℓt]≥H(Y(t)∣Zs)=H(Y(t))−I(Zs;Y(t)).E[ _t]\;≥\;H(Y^(t) Z_s)\;=\;H(Y^(t))-I(Z_s;Y^(t)). (5) Theorem 4.3 (Total log-loss lower bound with residual coupling). Under the assumptions of Theorem˜3.2, for any collection of per-task predictors qtt=1T\q_t\_t=1^T, ∑t=1T[−logqt(Y(t)∣Zs)]≥∑t=1TH(Y(t)∣Zs)= _t=1^TE[- q_t(Y^(t) Z_s)]\;≥\; _t=1^TH(Y^(t) Z_s)\;=\; H(Y(1:T)∣Zs)+Δ, H(Y^(1:T) Z_s)\;+\; , (6) where Δ=TC(Y(1:T)∣Zs)≥0 =TC(Y^(1:T) Z_s)≥ 0. Moreover, using I(Zs;Y(1:T))≤I(Zs;X)≤CsI(Z_s;Y^(1:T))≤ I(Z_s;X)≤ C_s, ∑t=1T[−logqt(Y(t)∣Zs)]≥H(Y(1:T))−Cs+Δ. _t=1^TE[- q_t(Y^(t) Z_s)]\;≥\;H(Y^(1:T))-C_s+ . (7) The lower bound in Equation˜6 implies that Δ>0 >0 forces extra aggregate log-loss for any set of decoupled per-task predictors qt(⋅∣Zs)\q_t(· Z_s)\, unless the shared features ZsZ_s makes tasks conditionally independent. This is why private features (Section˜5) are useful as they can reduce Δ by explaining away residual coupling that a shared representation leaves behind. Appendix S1.1 gives a Fano-style error lower bound as a corollary, while constants can be loose it still provides a clean impossibility statement directly in terms of misclassification. When should tasks share? If tasks are redundant (large TC(Y(1:T))TC(Y^(1:T)) or large explained dependence TC(Y(1:T))−TC(Y(1:T)∣Zs)TC(Y^(1:T))-TC(Y^(1:T) Z_s)), then sharing features is favorable. If tasks are heterogeneous (small redundancy) and T grows then Corollary˜4.2 implies that any globally shared bottleneck must either increase capacity roughly linearly in T or accept degraded per-task signal. Using this observation, we suggest clustered sharing and private capacity allocation as an approach to task heterogeneity. Gradient cosine similarity as a redundancy proxy. Gradient cosine similarity is widely used to measure task relatedness, but does it track the redundancy that CR predicts. We justify gradient cosine similarity as a proxy for redundancy ordering in a Gaussian model. Theorem 4.4 (Gradient–TC Bridge (Gaussian multi-task model)). Let X∼(0,Σ)X (0, ) and Y(t)=wt⊤X+εtY^(t)=w_t X+ _t with independent εt∼(0,σt2) _t (0, _t^2). We then define whitened task directions w~t=Σ1/2wt w_t= ^1/2w_t and the label correlation ρts=wt⊤Σws(wt⊤Σwt+σt2)(ws⊤Σws+σs2). _ts= w_t w_s (w_t w_t+ _t^2)(w_s w_s+ _s^2). The natural-gradient, equivalently whitened-gradient, cosine similarity at initialization is Gtsnat=w~t⊤w~s‖w~t‖‖w~s‖=wt⊤Σws(wt⊤Σwt)(ws⊤Σws).G^nat_ts= w_t w_s\| w_t\|\,\| w_s\|= w_t w_s (w_t w_t)(w_s w_s). Then sign(ρts)=sign(Gtsnat)sign( _ts)=sign(G^nat_ts) for all t≠st≠ s, and under matched noise levels and equal signal powers wt⊤Σwtw_t w_t across tasks, the ordering of |ρts|| _ts| across pairs is the same as the ordering of |Gtsnat||G^nat_ts|. For the raw-gradient cosine in unwhitened coordinates, Gtsraw=wt⊤Σ2ws‖Σwt‖‖Σws‖,G^raw_ts= w_t ^2w_s\| w_t\|\,\| w_s\|, the exact equivalence need not hold under anisotropic Σ ; it coincides with the whitened-gradient form when inputs are whitened or isotropic, with the discrepancy controlled by the conditioning of Σ . Since for Gaussian labels TC(Y(1:T))=−12logdetRTC(Y^(1:T))=- 12 R where R is the correlation matrix, any clustering rule that depends only on the ranking of pairwise similarities results in the same partition when applied to GnatG^nat as when applied to R. 5 Shared–Private Extension In practice, multi-task models often augment shared features with task-private features that can absorb heterogeneity and mitigate interference. A shared-only bound ignores private capacity that most practical architectures already use. We therefore add per task-budgets and derive a corresponding shared–private CR bound. Let Zs=fs(X)Z_s=f_s(X) denote shared features and for each task t∈[T]t∈[T], let Zt=ft(X)Z_t=f_t(X) denote task-private features. The task-specific representation is Z(t):=(Zs,Zt).Z^(t):=(Z_s,Z_t). We assume the (test-time) Markov property Y(1:T)−X−(Zs,Z1,…,ZT),Y^(1:T)\;-\;X\;-\;(Z_s,Z_1,…,Z_T), and impose separated information budgets, I(Zs;X)≤Cs,I(Zt;X∣Zs)≤Ct∀t∈[T]. I(Z_s;X)≤ C_s, I(Z_t;X Z_s)≤ C_t ∀ t∈[T]. (8) where I(Zt;X∣Zs)I(Z_t;X Z_s) is a conditional budget that measures incremental private capacity beyond what the shared bottleneck already covers. This results in an additive upper bound on the total information extracted from X by the collection (Zs,Z1,…,ZT)(Z_s,Z_1,…,Z_T), even when the private feature maps are statistically dependent through X. Theorem 5.1 (Shared–private capacity–redundancy bound). Under Markov chain and Equation˜8, ∑t=1TI(Z(t);Y(t))≤Cs+∑t=1TCt+TC(Y(1:T)). _t=1^TI\! (Z^(t);Y^(t) )\;≤\;C_s+ _t=1^TC_t\;+\;TC\! (Y^(1:T) ). (9) Theorem˜5.1 mirrors the shared-only CR inequality, with the shared capacity CsC_s replaced by a total budget Cs+∑tCtC_s+ _tC_t. This shows that private features can reduce the residual task coupling left unexplained by a shared bottleneck. Shared-only slack Δs _s measures how task labels remain coupled after conditioning on the shared representation. When Δs _s is large, the shared representation has not factorized the joint task structure. In such a setting, global sharing is prone to interference unless CsC_s grows. We now use shared–private features to define the post-augmentation residual coupling Δsp=TC(Y(1:T)∣Zall) _sp=TC(Y^(1:T) Z_all), where Zall=(Zs,Z1,…,ZT)Z_all=(Z_s,Z_1,…,Z_T). Since conditioning cannot increase entropy, TC(⋅∣⋅)TC(· ·) is monotone in the conditioning set, implying Δsp≤Δs. _sp\;≤\; _s. Thus private features can only decrease residual coupling, and doing so is what makes per-task heads easier to fit without forcing the shared bottleneck to encode heterogeneous, task-specific details. The bound in Equation˜9 is tight when the total representation saturates the information budget I(Zall;X)≈Cs+∑tCtI(Z_all;X)≈ C_s+ _tC_t, and the residual dependence is largely removed, Δsp≈0 _sp≈ 0 (labels become approximately conditionally independent given ZallZ_all). In contrast, if Δs _s remains large at fixed CsC_s, then increasing private budgets Ct\C_t\ is the only way within this framework to reduce Δsp _sp and avoid the shared-only obstruction. The shared–private inequality in Equation˜9 suggests to share capacity within groups of redundant tasks, and use private capacity to address residual coupling. Later in experiments we validate this by estimating task similarity and comparing global sharing, clustered sharing, and private adapters under matched total rank budgets. 5.1 Clustering as structured partial sharing MTL approaches already cluster tasks and share within clusters. The question is whether this provably helps and by how much? The basic CR bound alone cannot certify when clustering improves over global sharing. We now provide a sharp gain decomposition. Definition 5.2 (Inter-cluster total correlation). Let =S1,…,SKP=\S_1,…,S_K\ be a partition of [T][T] into disjoint clusters. We define TCbetween()=TC(Y(1:T))−∑k=1KTC(Y(Sk)),TC_between(P)\;=\;TC\! (Y^(1:T) )\;-\; _k=1^KTC\! (Y^(S_k) ), as the redundancy that crosses cluster boundaries. Theorem 5.3 (Clustering gap decomposition). Assume the test-time Markov property Y(1:T)−X−Zs,ZS1,…,ZSKY^(1:T)-X-\Z_s,Z_S_1,…,Z_S_K\. Let Zs⋆Z_s be a capacity-optimal globally shared encoder under budget I(Zs;X)≤CI(Z_s;X)≤ C. Let ZSk⋆k=1K\Z_S_k \_k=1^K be capacity-optimal per-cluster encoders with budgets I(ZSk;X)≤CkI(Z_S_k;X)≤ C_k and ∑kCk=C _kC_k=C. We define residual couplings as Δsh=TC(Y(1:T)∣Zs⋆),Δk=TC(Y(Sk)∣ZSk⋆). _sh=TC\! (Y^(1:T) Z_s ), _k=TC\! (Y^(S_k) Z_S_k ). Then the sum-rate gain of clustering over global sharing decomposes as, ∑t=1TI(ZSk(t)⋆;Y(t))−∑t=1TI(Zs⋆;Y(t))= _t=1^TI\! (Z_S_k(t) ;Y^(t) )- _t=1^TI\! (Z_s ;Y^(t) )\;=\; (Δsh−∑k=1KΔk)⏟interference reduction ≥0−TCbetween()⏟redundancy loss ≥0. ( _sh- _k=1^K _k )_interference reduction ≥ 0\;-\; TC_between(P)_redundancy loss ≥ 0. Thus clustered sharing is strictly beneficial if and only if the interference reduction exceeds the redundancy loss. We can use Theorem˜5.3 to choose partitions that keep strongly redundant tasks together (small TCbetweenTC_between) while separating antagonistic tasks to reduce residual coupling (large Δsh−∑kΔk _sh- _k _k). fifty2021efficiently search over task groupings combinatorially, Theorem˜5.3 tells us why a particular grouping wins. Here, the interference-reduction term Δsh−∑kΔk _sh- _k _k measures how much residual cross-task coupling is removed when weakly related tasks no longer compete for one shared representation. The redundancy-loss term TCbetween()TC_between(P) measures useful signal that crosses cluster boundaries and is no longer shared by a single global representation. Thus clustering is most beneficial when the task graph has clear communities with high intra-cluster redundancy and low inter-cluster redundancy. 6 Subsets and Side Information The global CR bound treats all T tasks as one block, which is wasteful when tasks cluster. In such cases, aggregate CR bound over all tasks may be too coarse to reflect finer-grained task relationships. Subset bounds exploit this structure. Subset bounds. Let S⊆[T]S [T] be any subset of tasks and denote YS=Y(t):t∈SY^S=\Y^(t):t∈ S\. Then the subset total correlation is defined as TC(YS)=∑t∈SH(Y(t))−H(YS).TC(Y^S)\;=\; _t∈ SH(Y^(t))\;-\;H(Y^S). Applying Lemma 3.1 and Theorem˜3.2 to the subcollection results in an immediate tightening. Corollary 6.1 (Subset CR inequality). Under the assumptions of Theorem˜3.2, for any S⊆[T]S [T], ∑t∈SI(Zs;Y(t))≤Cs+TC(YS). _t∈ SI(Z_s;Y^(t))\;≤\;C_s+TC(Y^S). (10) Moreover, the canonical sandwich holds on S, I(Zs;YS)≤∑t∈SI(Zs;Y(t))≤I(Zs;YS)+TC(YS).I(Z_s;Y^S)\;≤\; _t∈ SI(Z_s;Y^(t))\;≤\;I(Z_s;Y^S)+TC(Y^S). (11) The subset bound is usually tighter than Equation˜2 because redundancy is not uniform, tasks often form clusters with high intra-cluster dependence but low inter-cluster dependence. This refinement suggests that shared capacity should be allocated relative to TC(YS)TC(Y^S) within each cluster, with respect to the global quantity TC(Y(1:T))TC(Y^(1:T)). Side information and conditional CR. Let W denote side information available when constructing or analyzing the representation. The conditional total correlation is defined as TC(Y(1:T)∣W)=∑t=1TH(Y(t)∣W)−H(Y(1:T)∣W)≥ 0.TC(Y^(1:T) W)\;=\; _t=1^TH(Y^(t) W)\;-\;H(Y^(1:T) W)\;≥\;0. Theorem 6.2 (Conditional CR inequality). Under a conditional Markov property Y(1:T)−X−Zsgiven WY^(1:T)\;-\;X\;-\;Z_s W, and a shared capacity budget I(Zs;X∣W)≤Cs(W)I(Z_s;X W)≤ C_s(W), ∑t=1TI(Zs;Y(t)∣W)≤Cs(W)+TC(Y(1:T)∣W). _t=1^TI(Z_s;Y^(t) W)\;≤\;C_s(W)+TC(Y^(1:T) W). (12) More generally, for any subset S⊆[T]S [T], ∑t∈SI(Zs;Y(t)∣W)≤Cs(W)+TC(YS∣W). _t∈ SI(Z_s;Y^(t) W)\;≤\;C_s(W)+TC(Y^S W). (13) Residual coupling with side information. The exact decomposition also holds conditionally ∑t=1TI(Zs;Y(t)∣W)=I(Zs;Y(1:T)∣W)+(TC(Y(1:T)∣W)−TC(Y(1:T)∣Zs,W)) _t=1^TI(Z_s;Y^(t) W)=I(Z_s;Y^(1:T) W)+ (TC(Y^(1:T) W)-TC(Y^(1:T) Z_s,W) ) and the conditional slack Δ(W)=TC(Y(1:T)∣Zs,W)≥0 (W)=TC(Y^(1:T) Z_s,W)≥ 0 again measures residual task coupling after observing ZsZ_s, now at a fixed value of side information W. During training, the representation parameters depend on labels, so the unconditional Markov chain Y(1:T)−X−ZsY^(1:T)-X-Z_s need not hold. However, at evaluation time weights are fixed making it possible to use conditional Markov property, DPI and CR arguments at test time. This is crucial as it separates learning dynamics from the representational limit that are the main focus of this work. If W includes task identity or context, then TC(Y(1:T)∣W)TC(Y^(1:T) W) can be much smaller than TC(Y(1:T))TC(Y^(1:T)), since conditioning can decouple labels across tasks. In such a setting, using conditional adapters or MoE routing increase effective performance by reducing conditional redundancy terms or allowing Cs(W)C_s(W) to vary with W. The conditional CR bound Equation˜12 serves as a unified lens for these designs. It also covers task-dependent input distributions by taking W to include task identity or context. Implications for clustered sharing. Subset and conditional refinements together suggest to first estimate task similarity (as a proxy for dependence), cluster tasks into subsets Sk\S_k\, and allocate shared capacity per cluster. Formally, applying Equation˜10 within each cluster gives ∑t∈SkI(Zs(k);Y(t))≤Cs(k)+TC(YSk) _t∈ S_kI(Z_s^(k);Y^(t))\;≤\;C_s^(k)+TC(Y^S_k) so at fixed total budget ∑kCs(k) _kC_s^(k), clustered sharing targets the local redundancy structure rather than paying for global heterogeneity. We refer readers to Appendix Appendix˜S2 for a Gaussian model where both shared capacity and redundancy can be computed in closed form. 7 LoRA Fine-tuning Next, we introduce a capacity proxy to link adapter rank to an effective shared budget CsC_s that results in a testable predictions about when globally shared low-rank updates saturate and when clustered/private updates are necessary. LoRA as a low-rank information channel. Consider a frozen backbone with a hidden representation ϕ(X)∈ℝdφ(X) ^d. A LoRA update replaces a linear map W∈ℝo×dW ^o× d by W↦W+ΔW,ΔW=BA,A∈ℝr×d,B∈ℝo×r,W\; \;W+ W, W=BA, A ^r× d,\;B ^o× r, (14) where r≪mino,dr \o,d\ is the LoRA rank. We view LoRA branch as a feature channel that compresses the input into a low-dimensional signal Zs=Aϕ(X)∈ℝr,Z_s\;=\;A\,φ(X) ^r, which is then mixed into the output via B. In a multi task setup, this implies a structural constraint that tasks sharing the same LoRA factors share the same bottleneck ZsZ_s. Thus, the shared representation capacity is controlled by the rank r and the scale of A relative to the feature distribution ϕ(X)φ(X). This is consistent with the intrinsic-dimensionality perspective [aghajanyan-etal-2021-intrinsic]. CR predictions for LoRA design. Let us consider T tasks fine-tuned by sharing the same LoRA factors (A,B)(A,B) (and thus the same bottleneck ZsZ_s). Applying Theorem˜3.2 with shared budget Cs≈I(Zs;ϕ(X))C_s≈ I(Z_s;φ(X)) the capacity can be upper bound via Equation˜S3.32. If tasks are heterogeneous so that TC(Y(1:T))TC(Y^(1:T)) is small, then CR prediction implies that the sum of task-relevant information is controlled primarily by I(Zs;ϕ(X))I(Z_s;φ(X)), which grows slowly with r. Consequently, performance under a single globally shared LoRA should saturate quickly as rank increases, and adding tasks without increasing rank induces interference (Section˜4). r=4 r=8 r=16 Shared LoRA 0.763 0.773 0.755 Private LoRA 0.796 0.795 0.797 Clustered LoRA 0.812 0.817 0.806 Table 1: Mean validation accuracy on SST2/MRPC/RTE for budget-valid total ranks. Clustered sharing consistently outperforms global sharing under matched rank budgets. To obtain a per-task information level I(Zs;Y(t))≳bI(Z_s;Y^(t)) b across many weakly dependent tasks, Corollary 4.2 tells us that the total shared information I(Zs;ϕ(X))I(Z_s;φ(X)) must scale approximately like TbTb (modulo redundancy). Under our rank-controlled bound, LoRA rank must grow roughly linearly in the number of distinct task directions or clusters. If tasks form clusters Sk\S_k\ with high within-cluster dependence but low across-cluster dependence, then we can use the subset CR bound (Corollary˜6.1), sharing LoRA within each cluster and allocating rank per cluster, ∑t∈SkI(Zs(k);Y(t))≤Cs(k)+TC(YSk),∑kCs(k)≤Ctot. _t∈ S_kI(Z_s^(k);Y^(t))\;≤\;C_s^(k)+TC(Y^S_k), _kC_s^(k)≤ C_tot. Estimating residual coupling Δ from validation residuals. We use validation data (xi,yi(1:T))i=1n\(x_i,y_i^(1:T))\_i=1^n to compute per-task residuals et(i)=yt(i)−y^t(i)e_t^(i)=y_t^(i)- y_t^(i). Let R^e R_e be the sample correlation matrix of (e1,…,eT)(e_1,…,e_T). We approximate Δ^=−12logdet(R^e+ϵI)≈TC(Y(1:T)∣Zs), \;=\;- 12 ( R_e+ε I)\;≈\;TC\! (Y^(1:T) Z_s ), and report it across different settings. In NLP experiments we use ϵ=10−5ε=10^-5; the synthetic sensitivity sweep below shows this value is in the near-unbiased regime for our validation sample sizes, while larger ϵε persistently underestimates the conditional TC. 8 Empirical Validation 8.1 Validation on synthetic data. CR inequality for Gaussian case. We first verify the CR inequality in a linear–Gaussian multitask model where all terms are available in closed form, and show that the bound is tight once the representation spans the task subspace. In particular, when the encoder row-space contains the span of task directions, the residual coupling term vanishes, while under rank constraints the CR slack equals the conditional total correlation TC(Y1:T∣Z)TC(Y^1:T Z). Full derivations and controlled sweeps across redundancy, encoder alignment, and rank are provided in Appendix S7. Validation for Δ . We decouple estimator behavior from representation learning using a synthetic Gaussian multitask model where the target conditional TC quantity is known in closed form. We then estimate Δ from N i.i.d. samples and study both absolute error and signed bias across ridge parameters ε . Figure 2 shows that small ε in a near-unbiased estimates as N increases, while larger ε induces a persistent negative bias that does not vanish with sample size. At the largest sample size (N=16384N=16384), Table 2 quantifies this effect ε∈10−6,10−5 ∈\10^-6,10^-5\ achieves small error, whereas ε≥10−4 ≥ 10^-4 results in an underestimation. This shows that the ε acts as a regularization term for a log-determinant inverse problem, and using small ε in downstream evaluations. This sensitivity analysis directly motivates the choice ϵ=10−5ε=10^-5 used for the LoRA residual-coupling estimates. Figure 2: Mean absolute error |Δ^−TCexact|| -TC_exact| vs sample size N on the left. Signed bias Δ^−TCexact -TC_exact vs N on the right. Small ε converges toward the ground truth as N increases, while larger ε results in a persistent negative bias. Table 2: Gaussian setting: error of Δ at the largest sample size (N=16384N=16384, 10 seeds). ε mean |Δ^−TC|| -TC| std mean signed std 1e-06 0.0123 0.0089 -0.0037 0.0153 1e-05 0.0201 0.0138 -0.0191 0.0152 1e-04 0.1528 0.0148 -0.1528 0.0148 3e-04 0.3660 0.0143 -0.3660 0.0143 1e-03 0.7807 0.0138 -0.7807 0.0138 8.2 CR aligned PEFT. We now evaluate applicability of CR in a real setting by training multi-task PEFT with a frozen bert-base-uncased encoder and LoRA adapters applied to attention query and value projections. GoEmotions Dataset. We use GoEmotions [demszky-etal-2020-goemotions] dataset which consists of shared inputs and multiple binary labels. We treat each label as a task sharing the same input text and select twelve most frequent GoEmotions labels (fixed across runs) and evaluate each task using AUROC on the validation split and report the macro-average AUROC across the chosen tasks. We analyze two finetuning setting under a fixed LoRA rank budget rtot∈4,8,12,16,24,32r_tot∈\4,8,12,16,24,32\: using one shared adapter across all tasks, and cluster tasks into K clusters based on similarity and use K adapters. For clustered routing, the clustering is done once using gradient-similarity probes (details in Appendix S6). We also track Δ as a diagnostic of residual dependence. Figure 3 shows that similarity-based routing substantially improves performance under small budgets. From rtot=8r_tot=8 onward, clustered consistently outperforms shared in average AUCROC (Table 3). Moreover, the clustered routing results in a systematically lower Δ at matched rank budgets (Figure 4) which shows routing reduces residual interference in the learned predictor. Figure 3: Macro-average AUROC on the validation split as a function of total LoRA rank budget rtotr_tot. Clustered routing consistently improves over naive sharing from rtot=8r_tot=8 onward. Shaded regions denote CI. Figure 4: Δ computed from validation residuals vs total rank budget. Clustered routing results in a systematically smaller Δ than sharing at matched budgets (rank ≥8≥ 8), showing reduced residual dependence under similarity-guided finetuning. Table 3: Average AUROC (mean ± CI95) vs total rank budget. Rank Shared Clustered 4 0.744 ± 0.021 – 8 0.708 ± 0.030 0.821 ± 0.011 12 0.738 ± 0.022 0.805 ± 0.012 16 0.734 ± 0.023 0.804 ± 0.012 24 0.596 ± 0.091 0.809 ± 0.017 32 0.682 ± 0.085 0.813 ± 0.011 GLUE8 benchmark. We evaluate on eight GLUE tasks (SST-2, MRPC, RTE, QNLI, QQP, MNLI, CoLA, STS-B) using task metrics and report the average score across tasks.(Appendix S6). Figure 5 shows that clustered and private adapters provide modest but consistent gains over naive sharing for budgets where routing is feasible (rank ≥8≥ 8 in our setup). Unlike GoEmotions, the improvements on GLUE are smaller in magnitude, which can be attributed to heterogeneous task formats and limited sharing opportunities once syntax and sentiment tasks are isolated by similarity. This asymmetry is consistent with the CR prediction: GoEmotions labels are subcategories of a shared emotion taxonomy with higher cross-label redundancy, whereas GLUE8 combines heterogeneous task formats with weaker cross-task structure, so clustering has less redundant signal to reuse. Nevertheless, the results show similarity-guided finetuning improves transfer under a parameter budget. Figure 5: Shared is evaluated at all budgets, clustered/private are feasible at rank ≥8≥ 8 due to the number of adapters/tasks. We observe consistent improvement over naive sharing for the feasible budgets. Table 4: GLUE8: average score vs total rank budget (mean ± CI95) using official task metrics. Rank Regime n Score mean Score ci95 2 shared 10 0.712 0.010 2 clustered 0 – – 2 private 0 – – 4 shared 10 0.711 0.010 4 clustered 0 – – 4 private 0 – – 8 shared 10 0.712 0.009 8 clustered 10 0.723 0.003 8 private 10 0.720 0.003 16 shared 10 0.709 0.009 16 clustered 10 0.721 0.003 16 private 10 0.724 0.005 32 shared 10 0.712 0.010 32 clustered 10 0.722 0.003 32 private 10 0.727 0.003 64 shared 7 0.715 0.011 64 clustered 7 0.720 0.005 64 private 6 0.730 0.005 Overall, our empirical results across GoEmotions and GLUE8 demonstrate that routing based on gradient similarity improves performance under fixed adapter capacity. So when tasks compete for limited shared capacity a naive sharing can amplify interference while specialization restores effective capacity aligned with our CR perspective. On GoEmotions, clustered routing reduces Δ alongside improving AUROC (Figure 4). The Gaussian experiment clarifies how Δ acts like a regularization, increasing N reduces variance, while ε controls bias (Figure 2). Thus, Δ can be used as a compact diagnostic of residual dependence in multi-task settings. 9 Conclusion We introduced the CR inequality, a distribution-level converse that upper-bounds the total predictive information extractable from a shared latent by its information capacity plus the redundancy among tasks. This result explains negative transfer as an unavoidable consequence of low redundancy and finite capacity, and provides a principled approach for the design of shared–private architectures, including LoRA-based fine-tuning schemes. Furthermore, we demonstrate a similarity-guided routing approach in PEFT improves performance on multitask under strict capacity budgets. Overall, our results suggest that under limited adapter capacity sharing parameters based on gradient based task similarity provides a practical basis for budget-aware specialization. Because TC measures redundancy, not synergy, CR does not capture complementary information across tasks. Extensions via partial information decomposition or multivariate interaction measures (e.g., O-information [rosas2019quantifying]) are possible future directions to go beyond redundancy. Moreover, extending the analysis to dynamic training settings may explain how optimization implicitly allocates capacity across tasks. References Supplementary Material Appendix S1 Notation and preliminaries Symbol Meaning X input variable Y1:T=(Y1,…,YT)Y_1:T=(Y_1,…,Y_T) task labels Z learned representation (shared or composite) ZsZ_s shared representation ZtZ_t private representation for task t W parameterized weight matrix H(⋅)H(·) Shannon entropy I(A;B∣C)I(A;B C) conditional mutual information TC(Y1:T∣U)TC(Y_1:T U) conditional total correlation Cs,CtC_s,C_t capacity budgets (information or proxy budget) Total correlation. For label sequence Y1:TY_1:T, the TC and conditional TC terms are defined as, TC(Y1:T)=∑t=1TH(Yt)−H(Y1:T),TC(Y1:T∣U)=∑t=1TH(Yt∣U)−H(Y1:T∣U).TC(Y_1:T)= _t=1^TH(Y_t)-H(Y_1:T), (Y_1:T U)= _t=1^TH(Y_t U)-H(Y_1:T U). (S1.15) where TC(Y1:T∣U)≥0TC(Y_1:T U)≥ 0 with equality iff Y1,…,YTY_1,…,Y_T are conditionally independent given U. Identity underlying CR. For any representation Z and side information W, ∑t=1TI(Z;Yt∣W)=I(Z;Y1:T∣W)+TC(Y1:T∣W)−TC(Y1:T∣Z,W). _t=1^TI(Z;Y_t W)=I(Z;Y_1:T W)+TC(Y_1:T W)-TC(Y_1:T Z,W). (S1.16) Corollary S1.1 (Per-task error lower bound via Fano). Assume each Y(t)Y^(t) takes values in a finite alphabet of size |t|≥2|Y_t|≥ 2 and define the Bayes error given ZsZ_s, Pe,t⋆=infy^t(Zs)ℙ[y^t(Zs)≠Y(t)].P_e,t = _ y_t(Z_s)P[ y_t(Z_s)≠ Y^(t)]. Then Pe,t⋆≥H(Y(t)∣Zs)−1log|t|=H(Y(t))−I(Zs;Y(t))−1log|t|.P_e,t \;≥\; H(Y^(t) Z_s)-1 |Y_t|\;=\; H(Y^(t))-I(Z_s;Y^(t))-1 |Y_t|. (S1.17) There exists a task t⋆t with Pe,t⋆≳H(Y(t⋆))−Cs+TC(Y(1:T))Tlog|t⋆|.P_e,t \; \; H(Y^(t ))- C_s+TC(Y^(1:T))T |Y_t |. (S1.18) Appendix S2 Gaussian Specialization and Tightness A linear–Gaussian model offers an interpretable setting where we can evaluate the CR bounds exactly. Additionally, it shows exact conditions such as span alignment, orthogonality, and residual coupling under which these bounds become tight. Linear–Gaussian setup Let X∼(0,ΣX)∈ℝdX (0, _X) ^d, we assume tasks are generated as noisy linear measurements, Y(t)=ut⊤X+εt,εt∼(0,σt2),Y^(t)\;=\;u_t X+ _t, _t (0, _t^2), (S2.19) where ε1:Tindependent of X _1:T\ independent of X, we write Y(1:T)=U⊤X+εY^(1:T)=U X+ where U=[u1,…,uT]∈ℝd×TU=[u_1,…,u_T] ^d× T and ε∼(0,Σε) (0, _ ) with Σε=diag(σ12,…,σT2) _ =diag( _1^2,…, _T^2). A shared representation is produced by a linear map with an additive Gaussian noise: Zs=AX+η,η∼(0,σz2Im),η⟂(X,ε),Z_s\;=\;AX+η, η (0, _z^2I_m), η (X, ), (S2.20) where A∈ℝm×dA ^m× d is a design matrix, and σz2 _z^2 controls the bottleneck noise. S2.1 Closed-form capacity and redundancy For the Gaussian channel (S2.20), I(Zs;X)=12logdet(Im+1σz2AΣXA⊤).I(Z_s;X)= 12 \! (I_m+ 1 _z^2A _XA ). (S2.21) The above expression makes the capacity dependence on the spectrum of AΣX1/2A _X^1/2 explicit. At fixed output dimension m, capacity is governed by the nonzero singular values and saturates when AΣXA⊤A _XA becomes low-rank or small-norm. Label redundancy. Since Y(1:T)Y^(1:T) is jointly Gaussian with covariance ΣY=U⊤ΣXU+Σε, _Y=U _XU+ _ , the total correlation reduces to a log-determinant ratio, TC(Y(1:T))=12log∏t=1TΣY,ttdet(ΣY).TC(Y^(1:T))= 12 _t=1^T _Y,t ( _Y). (S2.22) TC(Y(1:T))TC(Y^(1:T)) is large when task outputs are strongly correlated (e.g., when ut\u_t\ have large overlap in the ΣX _X-geometry), and vanishes when ΣY _Y is diagonal (independent tasks). CR bound is tight in a Gaussian specialization under capacity saturation I(Zs;Y(1:T))≈I(Zs;X)I(Z_s;Y^(1:T))≈ I(Z_s;X), and explained dependence Δ=TC(Y(1:T)∣Zs)≈0 =TC(Y^(1:T) Z_s)≈ 0 (labels become nearly conditionally independent given ZsZ_s). When sharing is optimal. In the Gaussian setting I(Zs;Y(1:T))I(Z_s;Y^(1:T)) depends on how well the row space of A captures the subspace spanned by ut\u_t\. Let the task subspace be :=spanu1,…,uT⊆ℝdU:=span\u_1,…,u_T\ ^d in the ΣX _X-geometry. If the row space of A contains U (equivalently, AXAX preserves all task-relevant directions), then ZsZ_s is a sufficient statistic for U⊤XU X up to noise, and the joint information I(Zs;Y(1:T))I(Z_s;Y^(1:T)) approaches I(Zs;X)I(Z_s;X) as σz2→0 _z^2→ 0 under a fixed power constraint on A. Thus, conditioning on ZsZ_s removes most of the cross-task dependence induced by the shared latent X, so Δ≈0 ≈ 0 when the representation is informative enough about U⊤XU X. This results in a near-tightness of CR, ∑t=1TI(Zs;Y(t))≈I(Zs;X)+TC(Y(1:T)). _t=1^TI(Z_s;Y^(t))≈ I(Z_s;X)+TC(Y^(1:T)). (S2.23) When sharing must fail at fixed rank. Suppose task directions are nearly orthogonal in the ΣX _X inner product, so that U⊤ΣXU _XU is close to diagonal. Then TC(Y(1:T))TC(Y^(1:T)) is small and there is little redundancy to reuse across tasks. Additionally, the representation rank is limited then ZsZ_s cannot preserve all task directions simultaneously. In this scenario, I(Zs;Y(t))I(Z_s;Y^(t)) must be small for many tasks and the sum information is bounded by capacity alone, ∑t=1TI(Zs;Y(t))≲I(Zs;X)≤Cs, _t=1^TI(Z_s;Y^(t))\; \;I(Z_s;X)\;≤\;C_s, (S2.24) when TC(Y(1:T))≈0when TC(Y^(1:T))≈ 0. This shows negative transfer under global sharing adding heterogeneous tasks forces interference unless the shared capacity scales with the number of distinct task directions. Proposition S2.1 (Near-tightness under span capture). Assume ΣX=Id _X=I_d and σt2=σ2 _t^2=σ^2 for all t. Let =spanu1,…,uTU=span\u_1,…,u_T\ have dimension k and suppose A has rank at least k and row space containing U. Under a fixed power constraint tr(AA⊤)≤Ptr(A )≤ P, there exists a choice of A and σz2 _z^2 such that ∑t=1TI(Zs;Y(t))≥I(Zs;X)+TC(Y(1:T))−ϵ. _t=1^TI(Z_s;Y^(t))\;≥\;I(Z_s;X)+TC(Y^(1:T))-ε. (S2.25) Appendix S3 Full proofs S3.1 CR inequality and subset region Theorem S3.1 (CR bound). Assume the Markov chain Y1:T−X−ZsY_1:T-X-Z_s holds given W and the fixed capacity constraint I(X;Zs∣W)≤CsI(X;Z_s W)≤ C_s. Then ∑t=1TI(Zs;Yt∣W)≤Cs+TC(Y1:T∣W). _t=1^TI(Z_s;Y_t W)≤ C_s+TC(Y_1:T W). (S3.26) For any subset S⊆[T]S [T] we can write, ∑t∈SI(Zs;Yt∣W)≤Cs+TC(YS∣W). _t∈ SI(Z_s;Y_t W)≤ C_s+TC(Y_S W). (S3.27) Proof. Using CR identity with Z=ZsZ=Z_s and non-negativity of TC, ∑t=1TI(Zs;Yt∣W)≤I(Zs;Y1:T∣W)+TC(Y1:T∣W). _t=1^TI(Z_s;Y_t W)≤ I(Z_s;Y_1:T W)+TC(Y_1:T W). By data-processing under Y1:T−X−ZsY_1:T-X-Z_s given W, I(Zs;Y1:T∣W)≤I(Zs;X∣W)≤CsI(Z_s;Y_1:T W)≤ I(Z_s;X W)≤ C_s. The subset statement repeats the same argument for YSY_S. ∎ S3.2 Shared–private extension Theorem S3.2 (Shared–private CR bound). Assume Y1:T−X−(Zs,Z1:T)Y_1:T-X-(Z_s,Z_1:T) is a Markov chain given W and I(X;Zs∣W)≤CsI(X;Z_s W)≤ C_s, I(X;Zt∣W)≤CtI(X;Z_t W)≤ C_t for each t. Then, ∑t=1TI((Zs,Zt);Yt∣W)≤Cs+∑t=1TCt+TC(Y1:T∣W). _t=1^TI((Z_s,Z_t);Y_t W)≤ C_s+ _t=1^TC_t+TC(Y_1:T W). (S3.28) Proof. For each task t, I((Zs,Zt);Yt∣W)≤I((Zs,Z1:T);Yt∣W)I((Z_s,Z_t);Y_t W)≤ I((Z_s,Z_1:T);Y_t W). Summing over t and applying CR identity with Z=(Zs,Z1:T)Z=(Z_s,Z_1:T) gives, ∑t=1TI((Zs,Z1:T);Yt∣W)≤I((Zs,Z1:T);Y1:T∣W)+TC(Y1:T∣W). _t=1^TI((Z_s,Z_1:T);Y_t W)≤ I((Z_s,Z_1:T);Y_1:T W)+TC(Y_1:T W). Using data-processing, I((Zs,Z1:T);Y1:T∣W)≤I((Zs,Z1:T);X∣W)I((Z_s,Z_1:T);Y_1:T W)≤ I((Z_s,Z_1:T);X W) with chain rule and non-negativity, I((Zs,Z1:T);X∣W)≤I(Zs;X∣W)+∑t=1TI(Zt;X∣W)≤Cs+∑t=1TCt.I((Z_s,Z_1:T);X W)≤ I(Z_s;X W)+ _t=1^TI(Z_t;X W)≤ C_s+ _t=1^TC_t. ∎ S3.3 LoRA capacity bound (Gaussian-noise channel) Theorem S3.3 (Linear-Gaussian channel upper bound). Let U be any random vector with finite covariance ΣU=Cov(U∣W) _U=Cov(U W). Let Z=AU+ξZ=AU+ξ with ξ∼(0,σ2I)ξ (0,σ^2I) independent of U given W. Then I(U;Z∣W)≤12logdet(I+σ−2AΣUA⊤).I(U;Z W)≤ 12 \! (I+σ^-2A _UA ). (S3.29) If additionally rank(A)≤rrank(A)≤ r and tr(AΣUA⊤)≤κtr(A _UA )≤κ, then I(U;Z∣W)≤r2log(1+κrσ2).I(U;Z W)≤ r2 \! (1+ κrσ^2 ). (S3.30) Proof. Conditioning on W, we can write ΣZ=AΣUA⊤+σ2I _Z=A _UA +σ^2I. Since Z|UZ|U is a Gaussian noise, h(Z∣U)=h(ξ)h(Z U)=h(ξ). By maximum-entropy h(Z)≤12log((2πe)mdet(ΣZ))h(Z)≤ 12 ((2π e)^m ( _Z)). Thus I(U;Z)=h(Z)−h(Z∣U)≤12logdet(ΣZ)det(σ2I)I(U;Z)=h(Z)-h(Z U)≤ 12 ( _Z) (σ^2I). For the rank/trace bound, apply AM–GM on nonzero eigenvalues of M=σ−2AΣUA⊤⪰0M=σ^-2A _UA 0: logdet(I+M)≤rlog(1+tr(M)/r) (I+M)≤ r (1+tr(M)/r). ∎ A Gaussian surrogate for LoRA capacity. Consider a Gaussian/noisy-channel surrogate at inference time, Zs=Aϕ(X)+η,η∼(0,σ2Ir),Z_s\;=\;A\,φ(X)+η, η (0,σ^2I_r), (S3.31) where η is a noise used to make mutual information finite for deterministic encoders. Let Σϕ:=Cov(ϕ(X)) _φ:=Cov(φ(X)). Then, by the Gaussian channel formula, I(Zs;ϕ(X))=12logdet(Ir+1σ2AΣϕA⊤).I(Z_s;φ(X))= 12 \! (I_r+ 1σ^2A\, _φ\,A ). (S3.32) where ϕ(X)φ(X) is a deterministic function of X given frozen weights. Now using DPI I(Zs;X)≥I(Zs;ϕ(X))I(Z_s;X)\;≥\;I(Z_s;φ(X)) and I(Zs;Y(1:T))≤I(Zs;X),I(Z_s;Y^(1:T))≤ I(Z_s;X),, which implies (S3.32) is a conservative, architecture-linked proxy for the shared budget. Rank- and power-controlled upper bound. Assume a power constraint tr(AA⊤)≤κtr(A )≤κ and denote by λ1(Σϕ)≥⋯≥λd(Σϕ) _1( _φ)≥…≥ _d( _φ) the eigenvalues of Σϕ _φ. By eigenvalue majorization and log-det bounds, I(Zs;ϕ(X))≤r2log(1+κrσ2λ1(Σϕ))≤12∑i=1rlog(1+κσ2λi(Σϕ)).I(Z_s;φ(X))\;≤\; r2 \! (1+ κr\,σ^2\, _1( _φ) )\;≤\; 12 _i=1^r \! (1+ κσ^2\, _i( _φ) ). This shows that effective shared capacity grows at most linearly with rank r, and it saturates if Σϕ _φ is low-rank or if κ is small. Appendix S4 Proof of Theorem 5.3 Proof of Theorem 5.3. Let =S1,…,SKP=\S_1,…,S_K\ of [T][T] be a fixed partition. For any globally shared representation ZsZ_s and any collection of per-cluster representations ZSkk=1K\Z_S_k\_k=1^K, we apply CR identity once with (Z,Y(1:T))=(Zs,Y(1:T))(Z,Y^(1:T))=(Z_s,Y^(1:T)) and once for each cluster with (Z,Y(1:|Sk|))=(ZSk,Y(Sk))(Z,Y^(1:|S_k|))=(Z_S_k,Y^(S_k)), ∑t=1TI(Zs;Y(t)) _t=1^TI(Z_s;Y^(t)) =I(Zs;Y(1:T))+TC(Y(1:T))−TC(Y(1:T)∣Zs), =I(Z_s;Y^(1:T))+TC(Y^(1:T))-TC(Y^(1:T) Z_s), (S4.33) ∑t∈SkI(ZSk;Y(t)) _t∈ S_kI(Z_S_k;Y^(t)) =I(ZSk;Y(Sk))+TC(Y(Sk))−TC(Y(Sk)∣ZSk). =I(Z_S_k;Y^(S_k))+TC(Y^(S_k))-TC(Y^(S_k) Z_S_k). (S4.34) Summing (S4.34) over k=1,…,Kk=1,…,K gives, ∑t=1TI(ZSk(t);Y(t))=∑k=1KI(ZSk;Y(Sk))+∑k=1KTC(Y(Sk))−∑k=1KTC(Y(Sk)∣ZSk). _t=1^TI(Z_S_k(t);Y^(t))= _k=1^KI(Z_S_k;Y^(S_k))+ _k=1^KTC(Y^(S_k))- _k=1^KTC(Y^(S_k) Z_S_k). (S4.35) Subtracting (S4.33) from (S4.35) yields the general gain identity ∑t=1TI(ZSk(t);Y(t))−∑t=1TI(Zs;Y(t)) _t=1^TI(Z_S_k(t);Y^(t))- _t=1^TI(Z_s;Y^(t)) =(TC(Y(1:T)∣Zs)−∑k=1KTC(Y(Sk)∣ZSk))⏟difference in residual couplings = (TC(Y^(1:T) Z_s)- _k=1^KTC(Y^(S_k) Z_S_k) )_difference in residual couplings +(∑k=1KTC(Y(Sk))−TC(Y(1:T)))⏟redundancy term + ( _k=1^KTC(Y^(S_k))-TC(Y^(1:T)) )_redundancy term +(∑k=1KI(ZSk;Y(Sk))−I(Zs;Y(1:T)))⏟joint-predictive term. + ( _k=1^KI(Z_S_k;Y^(S_k))-I(Z_s;Y^(1:T)) )_joint-predictive term. (S4.36) Now note that by Definition 5.2, ∑k=1KTC(Y(Sk))−TC(Y(1:T))=−TCbetween(). _k=1^KTC(Y^(S_k))-TC(Y^(1:T))=-\,TC_between(P). Also define Δsh:=TC(Y(1:T)∣Zs) _sh:=TC(Y^(1:T) Z_s) and Δk:=TC(Y(Sk)∣ZSk) _k:=TC(Y^(S_k) Z_S_k). Plugging these into (S4) gives ∑t=1TI(ZSk(t);Y(t))−∑t=1TI(Zs;Y(t))=(Δsh−∑k=1KΔk)−TCbetween()+(∑k=1KI(ZSk;Y(Sk))−I(Zs;Y(1:T))). _t=1^TI(Z_S_k(t);Y^(t))- _t=1^TI(Z_s;Y^(t))= ( _sh- _k=1^K _k )\;-\;TC_between(P)\;+\; ( _k=1^KI(Z_S_k;Y^(S_k))-I(Z_s;Y^(1:T)) ). (S4.37) Theorem˜5.3 is stated for Zs⋆Z_s and ZSk⋆\Z_S_k \ that are capacity optimal under budgets ∑kCk=C _kC_k=C. In the capacity constrained case the tightness of bound is when the joint-predictive term saturates its information budget, I(Zs⋆;Y(1:T))=CandI(ZSk⋆;Y(Sk))=Ck∀k,I(Z_s ;Y^(1:T))=C I(Z_S_k ;Y^(S_k))=C_k\;\;∀ k, (S4.38) which holds in the Gaussian linear channel model when the encoder family can represent the optimal channel for each budget, and more generally whenever the optimum lies on the active constraint boundary. Under (S4.38), ∑k=1KI(ZSk⋆;Y(Sk))−I(Zs⋆;Y(1:T))=∑k=1KCk−C=0. _k=1^KI(Z_S_k ;Y^(S_k))-I(Z_s ;Y^(1:T))= _k=1^KC_k-C=0. Therefore, (S4.37) reduces exactly to ∑t=1TI(ZSk(t)⋆;Y(t))−∑t=1TI(Zs⋆;Y(t))=(Δsh−∑k=1KΔk)−TCbetween(), _t=1^TI(Z_S_k(t) ;Y^(t))- _t=1^TI(Z_s ;Y^(t))= ( _sh- _k=1^K _k )\;-\;TC_between(P), . ∎ The clustered setting is strictly better than global sharing iff the right-hand side is strictly positive Δsh−∑k=1KΔk>TCbetween(). _sh- _k=1^K _k\;>\;TC_between(P). Without the saturation condition the exact gain identity consists of an additional joint-predictive difference term. In our empirical setup fixed-rank LoRA budget is typically budget active so that Equation˜S4.38 is a good approximation, and the measured effect is dominated by the residual coupling and redundancy terms. Appendix S5 Proof of Theorem 4.4 Proof of Theorem 4.4. Using the squared-loss for a shared linear probe θ∈ℝdθ ^d, ℒt(θ)=12[(θ⊤X−Y(t))2].L_t(θ)= 12\,E [(θ X-Y^(t))^2 ]. Taking the derivative w.r.t θ, ∇θℒt(θ)=[(θ⊤X−Y(t))X]=Σθ−[Y(t)X]. _θL_t(θ)=E [(θ X-Y^(t))X ]= θ-E[Y^(t)X]. Under the model Y(t)=wt⊤X+εtY^(t)=w_t X+ _t with εt⟂X _t X and [εt]=0E[ _t]=0, [Y(t)X]=[(wt⊤X)X]=Σwt.E[Y^(t)X]=E[(w_t X)X]= w_t. Hence, at initialization θ0=0 _0=0, gt=∇θℒt(θ0)=−Σwt.g_t\;=\; _θL_t( _0)=- w_t. Therefore the raw population gradient cosine similarity is Gtsraw=cos(gt,gs)=gt⊤gs‖gt‖‖gs‖=wt⊤Σ2ws‖Σwt‖‖Σws‖,G^raw_ts= (g_t,g_s)= g_t g_s\|g_t\|\,\|g_s\|= w_t ^2w_s\| w_t\|\,\| w_s\|, which establishes the raw-gradient expression. Let X~=Σ−1/2X X= ^-1/2X, so X~∼(0,I) X (0,I), and let us define w~t=Σ1/2wt w_t= ^1/2w_t. Then Y(t)=w~t⊤X~+εtY^(t)= w_t X+ _t and Cov(Y(t),Y(s))=w~t⊤w~s.Cov(Y^(t),Y^(s))= w_t w_s. Moreover, the natural gradient, equivalently the gradient at θ0=0 _0=0 in the whitened coordinates, equals −w~t- w_t, so the whitened-gradient cosine is Gtsnat=w~t⊤w~s‖w~t‖‖w~s‖.G^nat_ts= w_t w_s\| w_t\|\,\| w_s\|. Since the denominators in the label correlation ρts=Cov(Y(t),Y(s))Var(Y(t))Var(Y(s)) _ts= Cov(Y^(t),Y^(s)) Var(Y^(t))\,Var(Y^(s)) are strictly positive, sign(ρts)=sign(Cov(Y(t),Y(s)))=sign(w~t⊤w~s)=sign(Gtsnat)sign( _ts)=sign(Cov(Y^(t),Y^(s)))=sign( w_t w_s)=sign(G^nat_ts). Ordering under matched noise and normalized signal. Assume matched noise σt2=σ2 _t^2=σ^2 for all t and equal signal power w~t⊤w~t=α2 w_t w_t=α^2 for all t (equivalently wt⊤Σwtw_t w_t constant across tasks). Then Var(Y(t))=α2+σ2for all t,Var(Y^(t))=α^2+σ^2 all t, so the correlation simplifies to ρts=w~t⊤w~sα2+σ2. _ts= w_t w_sα^2+σ^2. Thus, for any two pairs (t,s)(t,s) and (t′,s′)(t ,s ), |ρts|>|ρt′s′|⟺|w~t⊤w~s|>|w~t′⊤w~s′|.| _ts|>| _t s | | w_t w_s|>| w_t w_s |. Under the same equal-norm condition, the cosine similarity satisfies |Gtsnat|=|w~t⊤w~s|‖w~t‖‖w~s‖=|w~t⊤w~s|α2.|G^nat_ts|= | w_t w_s|\| w_t\|\,\| w_s\|= | w_t w_s|α^2. Hence |ρts|| _ts| and |Gtsnat||G^nat_ts| induce the same ordering over pairs, proving (i). (If task signal powers differ, GtsnatG^nat_ts still tracks the numerator but correlations are additionally rescaled by the marginal variances; the equal-power assumption isolates the redundancy structure.) Total correlation of Gaussian labels and clustering equivalence. Let Y=(Y(1),…,Y(T))Y=(Y^(1),…,Y^(T)) be jointly Gaussian with covariance ΣY _Y and diagonal marginal variances D=diag(ΣY,11,…,ΣY,TT)D=diag( _Y,11,…, _Y,T). Then ΣY=DRD _Y=DRD, where R is the correlation matrix. For a multivariate Gaussian, TC(Y)=∑t=1TH(Y(t))−H(Y)=12log∏t=1TΣY,ttdet(ΣY).TC(Y)= _t=1^TH(Y^(t))-H(Y)= 12 _t=1^T _Y,t ( _Y). Using det(ΣY)=det(D)2det(R)=(∏tΣY,tt)det(R) ( _Y)= (D)^2 (R)= ( _t _Y,t ) (R) yields TC(Y)=−12logdet(R).TC(Y)=- 12 (R). Thus R fully determines TC(Y)TC(Y). Under conditions in (i) GtsnatG^nat_ts is a strictly monotone rescaling of ρts _ts (and hence of RtsR_ts) on [0,1][0,1]. Similarly for absolute similarities when using |⋅||·|. Therefore any clustering rule whose decisions depend only on the ordering of pairwise similarities produces the same partition when applied to GnatG^nat as when applied to R. The raw-gradient cosine GrawG^raw coincides with this quantity after whitening or when Σ is isotropic; otherwise the difference is governed by the anisotropy of Σ . This proves (i). ∎ Appendix S6 Experimental setup and implementation details S6.1 Datasets and evaluation metrics GoEmotions. We select the twelve most frequent labels in the training split and treat each as a binary task. For evaluation we compute AUROC per label on the validation split and report the macro-average across all tasks. We also report per-label AUC in Section˜S7.2. GLUE8. We evaluate on SST-2, MRPC, RTE, QNLI, QQP, MNLI (matched), CoLA, and STS-B. We report standard metrics, accuracy for SST-2/RTE/QNLI/MNLI, (Acc+F1)/2(Acc+F1)/2 for MRPC/QQP and Matthews correlation coefficient (MCC) for CoLA, and (ρPearson+ρSpearman)/2( _Pearson+ _Spearman)/2 for STS-B. We report the average of task scores as the GLUE8 score. S6.2 Models and parameter-efficient fine-tuning All experiments use bert-base-uncased as the frozen encoder. We attach LoRA adapters to the attention query/value projections and only LoRA parameters and task heads are trainable. For LoRA we use α=16α=16 with a dropout probability of 0.050.05. We control parameterization through a total rank budget rtotr_tot. In the shared setting, the single adapter has rank rtotr_tot. In clustered routing with K adapters, we allocate ranks across adapters such that ∑k=1Krk=rtot _k=1^Kr_k=r_tot with rk≥1r_k≥ 1. For GLUE8 we use proportional rank allocation (larger clusters receive more rank), while GoEmotions uses fixed-sum allocation consistent with the split runner. Model training. We train with AdamW, FP16 enabled, linear warmup (6%6\% of total steps), and gradient clipping (norm 1.01.0). We use batch size of 1616 and steps-per-epoch 14001400 for 33 epochs. We use weight decay on task heads but set it to 0 for LoRA weights. This is done to avoid decoupled weight decay shrinking inactive adapters in multi-adapter setting. S6.3 Similarity estimation and clustering We first partition tasks into K clusters and use one adapter per cluster with ranks rk\r_k\ such that ∑krk=R _kr_k=R and rk≥1r_k≥ 1. We estimate task similarity without training full adapters by attaching a probe LoRA adapter of rank 11, denoted aprobea_probe, and computing per-task gradient vectors, gt:=1B∑b=1B∇θprobeℓt(θprobe;batch b),g~t:=gt‖gt‖2.g_t:= 1B _b=1^B _ _probe\, _t( _probe;batch b), g_t:= g_t\|g_t\|_2. where for similarity we use cosine sij=⟨g~i,g~j⟩s_ij= g_i, g_j . We use agglomerative clustering on the distance matrix Dij=1−sijD_ij=1-s_ij with average linkage and fixed K. For GoEmotions we use probe rank 1, 5 similarity seeds, and 20 gradient-accumulation steps per task, and for GLUE8 we use 30 steps per task. We warm-start task heads for 200 steps before computing gradients to stabilize the probe. Given the mean similarity matrix, we compute an average-linkage agglomerative clustering and select K using a gap criterion constrained to K∈[2,6]K∈[2,6]. S6.4 Clustering-method sensitivity To test sensitivity to the clustering design, we sweep the number of clusters K∈2,3,4,5,6K∈\2,3,4,5,6\ and compare average-linkage, complete-linkage, and spectral clustering at ranks 88 and 1212. Complete linkage gives the best AUC at the strongest setting (K=6K=6, r=8r=8), while average linkage tracks it closely and obtains the lowest residual-coupling value at K=6K=6, r=12r=12. Across algorithms, Δ generally decreases as K grows, consistent with the prediction that more specialized routes reduce residual conflict. Table S6.5: Clustering sensitivity on GoEmotions: mean ± std over five random seeds. Algorithm K Rank avg_auc Δ avg 2 8 0.6950 ± 0.0602 0.2331 ± 0.0286 avg 2 12 0.7011 ± 0.0729 0.2305 ± 0.0298 avg 3 8 0.7149 ± 0.0368 0.2229 ± 0.0145 avg 3 12 0.7021 ± 0.0559 0.2088 ± 0.0124 avg 4 8 0.8004 ± 0.0087 0.2341 ± 0.0089 avg 4 12 0.7817 ± 0.0174 0.2155 ± 0.0102 avg 5 8 0.8065 ± 0.0182 0.2290 ± 0.0093 avg 5 12 0.8081 ± 0.0228 0.2065 ± 0.0086 avg 6 8 0.8133 ± 0.0177 0.2074 ± 0.0091 avg 6 12 0.8101 ± 0.0208 0.2013 ± 0.0095 complete 2 8 0.7005 ± 0.0542 0.2340 ± 0.0271 complete 2 12 0.7092 ± 0.0575 0.2298 ± 0.0260 complete 3 8 0.8057 ± 0.0172 0.2437 ± 0.0102 complete 3 12 0.7945 ± 0.0180 0.2295 ± 0.0099 complete 4 8 0.7900 ± 0.0187 0.2322 ± 0.0103 complete 4 12 0.7882 ± 0.0141 0.2095 ± 0.0094 complete 5 8 0.8144 ± 0.0144 0.2272 ± 0.0095 complete 5 12 0.8067 ± 0.0208 0.2087 ± 0.0090 complete 6 8 0.8392 ± 0.0091 0.2159 ± 0.0083 complete 6 12 0.8355 ± 0.0117 0.2028 ± 0.0089 spectral 2 8 0.7598 ± 0.0296 0.2688 ± 0.0181 spectral 2 12 0.7632 ± 0.0271 0.2632 ± 0.0167 spectral 3 8 0.7799 ± 0.0186 0.2528 ± 0.0124 spectral 3 12 0.7546 ± 0.0474 0.2400 ± 0.0146 spectral 4 8 0.7824 ± 0.0139 0.2173 ± 0.0098 spectral 4 12 0.7899 ± 0.0076 0.2296 ± 0.0093 spectral 5 8 0.8124 ± 0.0119 0.2040 ± 0.0086 spectral 5 12 0.8057 ± 0.0094 0.2144 ± 0.0089 spectral 6 8 0.8033 ± 0.0083 0.2154 ± 0.0084 spectral 6 12 0.8008 ± 0.0127 0.2062 ± 0.0088 S6.5 Additional results Appendix S7 Linear–Gaussian verification of the CR inequality We verify the CR inequality in a controlled linear–Gaussian setting where all quantities are computable in closed form. We sample X∼(0,Id)X (0,I_d) (d=20d=20) and define T task labels Y(t)=at⊤X+εt,εt∼(0,σε2),Y^(t)=a_t X+ _t, _t (0, _ ^2), (S7.39) with independent εtt=1T\ _t\_t=1^T and σε=0.1 _ =0.1. Let A∈ℝT×dA ^T× d stack at⊤a_t as rows. By varying the alignment of at\a_t\ we sweep label redundancy. If ai≈aja_i≈ a_j then labels share signal and TC(Y1:T)TC(Y^1:T) increases, while near-orthogonality yields TC(Y1:T)≈0TC(Y^1:T)≈ 0. Representations are produced by a rank-r linear encoder Z=W⊤X+η,η∼(0,ση2Ir),Z=W X+η, η (0, _η^2I_r), (S7.40) with ση=0.5 _η=0.5. If row(W)⊇row(A)row(W) (A), then AXAX is a linear function of Z, i.e. there exists M such that AX=MZAX=MZ, and hence Y=AX+ε=MZ+ε.Y=AX+ =MZ+ . (S7.41) Because ε has independent coordinates, the tasks are conditionally independent given Z, implying TC(Y1:T∣Z)=0TC(Y^1:T Z)=0, so the CR inequality becomes tight. When the representation is rank-limited, the residual coupling term is exactly the conditional total correlation TC(Y1:T∣Z)TC(Y^1:T Z), which quantifies the remaining dependence among tasks after conditioning on the representation. We compute TC(Y1:T)TC(Y^1:T) and TC(Y1:T∣Z)TC(Y^1:T Z) from Gaussian covariance identities (log-determinant forms), and compute I(Z;Y(t))I(Z;Y^(t)) via conditional-variance formulas. Figure S7.6 reports sweeps across redundancy level, comparing optimal vs random vs misaligned encoders, number of tasks at low redundancy to illustrate negative transfer in average per-task information, and rank r to show saturation once the encoder spans the task subspace. Figure S7.6: Left: CR holds across encoder types, with tightness for the optimal encoder and slack for random or misaligned ones. Middle: Capacity sweep shows aggregate information tracks Cs+TC(Y)C_s+TC(Y), saturating at low rank. Right: Negative transfer demonstrated, as average per-task information declines when the number of tasks increases under fixed capacity and low redundancy. S7.1 GoEmotions: clustered vs random partitions (available baseline) Figure S7.7: GoEmotions (rank=8): clustered vs random partitions (available baseline). Distribution of macro-average ROC-AUC over 45 random partitions (seed 41) compared to similarity-based clustering. We report this as a sanity check rather than a universal significance claim because random baselines are only available at rank 8 in the split-full runs. Table S7.6: Clustered vs random-clustered (empirical p-value and effect size). Rank Clustered_mean Random_mean p_emp Cohen_d N_clustered N_random 8.0000 0.821 0.800 0.0889 1.499 5 45 S7.2 Per-emotion gains at rank 12 Figure S7.8: GoEmotions per-label effect of clustering. Difference in AUROC per emotion (clustered minus shared). Improvement in performance is on several harder labels (e.g., amusement, curiosity), while some labels change minimally. Error bars are reported in Table S7.7. Table S7.7: Per-emotion AUROC for shared vs clustered (mean ± CI95) and their difference. Task Shared_mean Shared_ci95 Clustered_mean Clustered_ci95 Delta_mean amusement 0.564 0.093 0.809 0.021 0.245 curiosity 0.675 0.094 0.886 0.017 0.211 approval 0.494 0.047 0.620 0.045 0.126 anger 0.753 0.074 0.839 0.033 0.086 annoyance 0.727 0.064 0.785 0.043 0.058 love 0.890 0.026 0.944 0.011 0.053 admiration 0.848 0.011 0.900 0.024 0.052 disapproval 0.704 0.047 0.742 0.040 0.038 neutral 0.761 0.012 0.784 0.009 0.023 gratitude 0.982 0.006 0.984 0.004 0.002 joy 0.800 0.008 0.784 0.055 -0.016 optimism 0.653 0.041 0.585 0.136 -0.068 S7.3 GoEmotions baselines We include additional baselines from the earlier harness using PCGrad, GradNorm, single-task oracle. Here, the single-task baseline is an oracle upper bound and is not compute-matched. Table S7.8: Additional baselines across available ranks (mean ± CI95). Rank Regime N Avg_auc_mean Avg_auc_ci95 4 shared 5 0.725 0.032 4 clustered 0 – – 4 private 0 – – 4 single_task 0 – – 4 pcgrad_shared 5 0.903 0.003 4 gradnorm_shared 5 0.894 0.001 4 random_clustered 0 – – 8 shared 5 0.702 0.081 8 clustered 0 – – 8 private 0 – – 8 single_task 0 – – 8 pcgrad_shared 5 0.904 0.003 8 gradnorm_shared 5 0.901 0.002 8 random_clustered 0 – – 12 shared 5 0.738 0.012 12 clustered 5 0.827 0.005 12 private 5 0.830 0.003 12 single_task 5 0.929 0.000 12 pcgrad_shared 0 – – 12 gradnorm_shared 0 – – 12 random_clustered 55 0.824 0.003 S7.4 Similarity heatmaps Figure S7.9: Gradient similarity heatmap for GoEmotions. Mean cosine similarity of probe gradients across five similarity seeds. The resulting clusters group semantically related labels. Figure S7.10: Gradient similarity heatmap for GLUE Mean cosine similarity of probe gradients across 5 similarity seeds. The clustering isolates sentiment (SST-2) and acceptability (CoLA) while grouping entailment/paraphrase-style tasks (e.g., MRPC–RTE and QNLI–MNLI). We make a note that clustered/private settings require rtot≥Kr_tot≥ K (and private additionally requires rtot≥Tr_tot≥ T).For GLUE8 and GoEmotions, K=6K=6 in our clustering, hence routing is only feasible for rank budgets ≥8≥ 8 in the plotted ranges.