Paper deep dive
Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections
Zhaoyi Liu, Haichuan Zhang, Ang Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/26/2026, 2:20:34 AM
Summary
The paper introduces Spectral-Sphere-Constrained Hyper-Connections (sHC), a novel method for residual feature mixing in deep learning. It addresses the limitations of existing Manifold-Constrained Hyper-Connections (mHC), specifically identity degeneration, expressivity bottlenecks due to non-negativity constraints, and parameterization inefficiencies. By shifting the constraint from the Birkhoff polytope to a spectral norm sphere, sHC enables subtractive feature interactions and eliminates the need for unstable Sinkhorn iterations or factorial-scaling parameterizations.
Entities (5)
Relation Signals (3)
sHC → improvesupon → mHC
confidence 98% · sHC overcomes the identity degeneration, expressivity bottleneck, and parameterization inefficiencies in existing manifold-constrained hyper-connections (mHC)
mHC → uses → Birkhoff polytope
confidence 95% · mHC xie2025mhc constrains H_l^res to lie in the Birkhoff polytope B_n
sHC → uses → spectral norm sphere
confidence 95% · we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Hyper-Connections (HC) generalize residual connections into multiple streams, employing residual matrices for cross-stream feature mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variant restrict these matrices to the Birkhoff polytope (doubly stochastic matrices) via Sinkhorn iterations or permutation-based parameterizations. We reveal three limitations of this polytope constraint: (1) identity degeneration, where learned matrices collapse around the identity and diminish cross-stream interactions, (2) an expressivity bottleneck, as the non-negativity constraint prevents subtractive feature disentanglement, and (3) parameterization inefficiencies, manifesting as unstable Sinkhorn iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere, sHC allows negative entries, unlocking subtractive interactions for selective feature diversification. This shift eliminates unstable Sinkhorn projections and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.
Tags
Links
- Source: https://arxiv.org/abs/2603.20896v1
- Canonical: https://arxiv.org/abs/2603.20896v1
Trouble viewing inline? Open PDF directly →
Full Text
59,435 characters extracted from source content.
Expand or collapse full text
Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections Zhaoyi Liu1 Haichuan Zhang2 Ang Li1 1University of Maryland, College Park 2University of Utah 1zhaoyil, angliece@umd.edu 2hc.zhang@utah.edu Abstract Hyper-Connections (HC) generalize residual connections into multiple streams, employing residual matrices for cross-stream feature mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variant restrict these matrices to the Birkhoff polytope (doubly stochastic matrices) via Sinkhorn iterations or permutation-based parameterizations. We reveal three limitations of this polytope constraint: (1) identity degeneration, where learned matrices collapse around the identity and diminish cross-stream interactions, (2) an expressivity bottleneck, as the non-negativity constraint prevents subtractive feature disentanglement, and (3) parameterization inefficiencies, manifesting as unstable Sinkhorn iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). By geometrically shifting the feasible set from a rigid polytope to a spectral norm sphere, sHC allows negative entries, unlocking subtractive interactions for selective feature diversification. This shift eliminates unstable Sinkhorn projections and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability. Figure 1: sHC overcomes the identity degeneration, expressivity bottleneck, and parameterization inefficiencies in existing manifold-constrained hyper-connections (mHC xie2025mhc ) and its permutation-based variant (mHC-lite yang2026mhc ). Left: Learned residual matrices (4 streams). Residual matrices of mHC and mHC-lite degenerate into the identity mapping, whereas sHC leverages diverse signed entries for subtractive mixing. Middle: Language Modeling Performance. Perplexity is presented relative to the standard residual connection baseline. sHC yields observable perplexity reductions across all five corpora. Right: Parameterization overhead. As the number of streams increases, sHC eliminates the factorial explosion of auxiliary parameters inherent to mHC-lite. 1 Introduction Residual connections he2016deep have been a cornerstone of deep learning for over a decade, stabilizing gradient propagation through identity mappings and becoming a standard component of modern deep networks, including large language models liu2024deepseek ; touvron2023llama ; brown2020language . Hyper-Connections (HC) zhu2024hyper have recently extended this traditional single-stream residual connection into parallel residual streams by using a dynamic residual matrix at each layer to mix the features across the streams, increasing the model’s topological complexity and capacity. However, unconstrained residual matrices compromise the identity preservation property intrinsic to the residual connection, which causes training instability xie2025mhc . To address this, DeepSeek’s Manifold-Constrained Hyper-Connections (mHC) xie2025mhc proposes constraining residual matrices to be doubly stochastic to preserve the identity mapping property. Doubly stochastic matrices belong to the Birkhoff polytope, characterized by non-negative entries and unit row and column sums. This structure theoretically bounds the spectral norm by 11 to mitigate gradient explosion, while preserving the mean component of the residual streams xie2025mhc . To enforce this constraint, mHC projects residual matrices onto the Birkhoff polytope via Sinkhorn–Knopp (SK) iteration sinkhorn1967concerning , which iteratively normalizes rows and columns to approximate constraints. However, SK yields only an unstable approximate projection onto the polytope. As reported in yang2026mhc , the resulting constraint violations accumulate across depth, potentially undermining stability. A recent variant, mHC-lite yang2026mhc , guarantees exact doubly stochasticity by parameterizing residual matrices as a convex combination of permutation matrices. This parametrization introduces factorial growth in auxiliary parameters, leading to prohibitive complexity and limited scalability. Beyond these parameterization inefficiencies, we identify two intrinsic limitations of the Birkhoff polytope constraint adopted in these methods. First, it is prone to identity degeneration, where the learned residual matrices in each layer concentrate around the identity, practically abandoning the intended cross-stream interactions. Second, the non-negativity constraint imposes a structural expressivity bottleneck: residual streams are restricted to convex combinations, precluding subtractive interactions and limiting the model’s ability to suppress noise or disentangle features. (Detailed analysis is in § 4.) To overcome these limitations, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). Instead of confining the residual matrices to Birkhoff polytope, we geometrically shift the feasible set to a spectral norm sphere. As shown in Figure 1, this shift yields three key advantages: ❶ By permitting negative matrix entries, sHC explicitly unlocks the model’s capacity for subtractive feature interactions and selective feature diversification. ❷ sHC produces non-degenerate residual matrices that support expressive feature interactions, improving the model performance. ❸ Restricted on a sphere instead of a faceted polytope, we eliminate factorial parameterization overhead and ensure stable spectral-norm control without relying on Sinkhorn iterations. In summary, our contributions are: • We introduce Spectral-Sphere-Constrained Hyper-Connections (sHC), which reformulates hyper-connection constraints within a spectral-norm sphere, mitigating identity degeneration, alleviating the expressivity bottleneck, and eliminating factorial parameterization overhead. • Experiments demonstrate that sHC improves model capability over existing hyper-connection methods while preserving the structural constraints across depth. • The expressivity, stability, and scalability of sHC provide a new and viable design direction for residual connections in deep learning architectures. 2 Related Work Residual connections he2016deep stabilize deep network training by introducing identity skip connections. Expanding on this, Hyper-Connections (HC) zhu2024hyper introduce parallel residual streams mixed by dynamic matrices to enhance model capacity. Concurrently, Frac-Connections zhu2025frac explore fragmenting streams into chunks as an alternative topology to reduce the memory access costs of parallel residual streams. Manifold-Constrained Hyper-Connections (mHC) xie2025mhc and its variant yang2026mhc directly target the training instability inherent in HC, where unconstrained feature mixing compromises identity preservation. By confining the residual matrices to the doubly stochastic space, they restore stability. However, this rigid constraint introduces parameterization inefficiency and expressivity bottlenecks. In this work, we align with the hyper-connection paradigm, focusing on resolving the expressivity and efficiency bottlenecks of full-stream mixing rather than exploring fractional topologies. 3 Preliminary Hyper-Connections (HC). Despite residual connections’ he2016deep widespread success, the single-stream design restricts signal flow to a single pathway, potentially limiting the model’s capacity. To enrich the expressivity of the model, Hyper-Connections zhu2024hyper extend the single-stream paradigm to n parallel residual streams. Let Xl=(l,1⊤,l,2⊤,…,l,n⊤)⊤∈ℝn×CX_l=( x_l,1 , x_l,2 ,..., x_l,n ) ^n× C represent the expanded features of n streams at the l-th layer. HC introduces a dynamic mixing mechanism: Xl+1=ℋlresXl+(ℋlpost)⊤ℱ(ℋlpreXl,l)X_l+1=H_l^resX_l+(H_l^post) F(H_l^preX_l,W_l) (1) Here, ℱ(⋅,l)F(·,W_l) denotes the learnable layer transformation with parameter lW_l in the branch (e.g., Attention or MLP block). ℋlres∈ℝn×nH_l^res ^n× n represents a learnable residual matrix that mixes features within the residual streams. Similarly, ℋlpre∈ℝ1×nH_l^pre ^1× n aggregates features from the (n×C)(n× C)-dim stream into a (1×C)(1× C)-dim layer branch input, and conversely, ℋlpost∈ℝ1×nH_l^post ^1× n maps the layer branch output back onto the streams. However, without constraints on these residual matrices, HC may suffer from severe training instability xie2025mhc . Manifold-Constrained Hyper-Connections (mHC). To ensure training stability, mHC xie2025mhc constrains ℋlresH_l^res to lie in the Birkhoff polytope ℬnB_n, i.e., the set of doubly stochastic matrices. This polytope is a subset of the affine subspace n=ℋ∈ℝn×n∣ℋn=n, 1n⊤ℋ=n⊤A_n=\H ^n× n 1_n=1_n,\ 1_n H=1_n \, which is obtained by adding the element-wise non-negativity constraint to nA_n: ℬn=ℋ∈n∣ℋ≥0.B_n=\H _n ≥ 0\. (2) The affine constraint ensures that the uniform vector n1_n remains invariant, thereby conserving the mean component across residual streams. Imposing the additional non-negativity condition bounds the spectral norm to 1 (i.e., ‖ℋlres‖2=1\|H_l^res\|_2=1), which prevents signal amplification and stabilizes deep propagation. As such, mHC can be fully formulated as: l′=RMSNorm(l) x_l =RMSNorm( x_l) (3) ℋlpre=σ(αlpre⋅(l′Wlpre)+lpre) _l^pre=σ ( _l^pre·( x_l W_l^pre)+ b_l^pre ) ℋlpost=2σ(αlpost⋅(l′Wlpost)+lpost) _l^post=2σ ( _l^post·( x_l W_l^post)+ b_l^post ) ℋlres=SK(αlres⋅mat(l′Wlres)+lres) _l^res=SK ( _l^res·mat( x_l W_l^res)+ b_l^res ) where l∈ℝ1×nC x_l ^1× nC is the vector flattened from the expanded input XlX_l. Wlpre,Wlpost∈ℝnC×nW_l^pre,W_l^post ^nC× n and Wlres∈ℝnC×n2W_l^res ^nC× n^2 are linear projections for dynamic mappings. The terms lpre,lpost∈ℝ1×n b_l^pre, b_l^post ^1× n and lres∈ℝn×n b_l^res ^n× n are learnable biases. αlpre,αlpost,αlres _l^pre, _l^post, _l^res are scalar gating factors. RMSNorm(⋅)RMSNorm(·) refers to the RMSNorm zhang2019root . σ(⋅)σ(·) denotes the Sigmoid function. mat(⋅)mat(·) reshapes a matrix from ℝ1×n2R^1× n^2 to ℝn×nR^n× n while SK(⋅)SK(·) denotes the Sinkhorn–Knopp iteration which iteratively alternates row-wise and column-wise normalization to enforce approximate doubly stochasticity. However, finite SK iterations cannot guarantee exact doubly stochasticity. The resulting approximation errors can accumulate across layers, potentially undermining the stability of deep networks. Permutation-based Parameterization. Instead of approximate SK projection, mHC-lite yang2026mhc employs the Birkhoff-von Neumann theorem birkhoff1946three ; von1950certain to parameterize ℋlresH_l^res as convex combinations of permutation matrices to achieve exact doubly stochasticity: l=softmax(αlres⋅(l′Wlres)+lres) a_l=softmax ( _l^res·( x _lW_l^res)+ b_l^res ) (4) ℋlres=∑i=1n!(l)iPi _l^res= _i=1^n!( a_l)_iP_i Here, the coefficients l a_l are predicted from the normed flattened input vector l′ x _l. Pi∈Pii=1n!⊂[0,1]n×nP_i∈\P_i\_i=1^n!⊂[0,1]^n× n is the set of all permutation matrices. While mHC-lite guarantees ℋlresH_l^res to be exactly doubly stochastic, the linear projection weight Wlres∈ℝnC×n!W_l^res ^nC× n! and bias lres∈ℝ1×n! b_l^res ^1× n! scale factorially with the number of streams n, introducing a heavy parameterization overhead. 4 Observation Figure 2: Dynamics of ℋlresH_l^res for mHC and mHC-lite during training. Left: the row-wise maximum entries of ℋlresH_l^res. The solid lines represent the median, and the shaded regions show the 10th to 90th percentiles. Right: the proportion of ℋlresH_l^res where all row maximums are on the diagonal. Statistics are computed across all layers of the model at each training step. To evaluate the practical expressivity of these Birkhoff polytope based hyper-connections (mHC and mHC-lite), we analyze their training dynamics of residual matrix ℋlresH_l^res and pairwise cosine similarity among residual streams after being mixed by ℋlresH_l^res across all layers of the model at each training step. Experiments are conducted on a 12-layer, 0.12B-parameter nanoGPT model karpathy2022nanogpt . 4.1 Identity Degeneration As shown in Figure 2, for both mHC and mHC-lite, the median row-wise maximum of the learned residual matrices ℋlresH_l^res remains tightly concentrated around 1 throughout training. The 10th–90th percentile bands (the shaded regions in the figure) exhibit minimal dispersion, with the widest range only spanning 0.85–0.99 for mHC and 0.95-1 for mHC-lite. Moreover, the fraction of matrices whose row maxima lie strictly on the diagonal stays consistently high (above 96% for mHC-lite and 94% for mHC). Since ℋlresH_l^res is doubly stochastic (non-negative with unit row and column sums), the concentration of row-wise maxima on the diagonal with values approaching 1 implies that the learned matrices degenerate around the identity, up to small off-diagonal tails. This indicates that the model abandons active cross-stream interactions at individual layers, relying instead on these small tails for a slow and passive feature diffusion accumulated over depths. This degeneration is also reflected in the pairwise similarity within the residual streams. As illustrated in Figure 3, the similarity trajectories of mHC closely follow those of the identity mapping baseline (where ℋlresH_l^res is fixed as an identity matrix while keeping all other settings identical to mHC), with only a slight increase in similarity. This marginal shift, relative to the overall similarity dynamics driven by model learning itself, indicates that feature evolution is predominantly governed by the nonlinear branches rather than the hyper-connections. (We observe the same phenomenon in mHC-lite. See Appendix B for details.) This degeneration may explain the resulting gradient stability of mHC and mHC-lite. However, prioritizing this form of apparent stability runs counter to the original design motivation of hyper-connections, which aim to enhance the model’s topological expressivity through active cross-stream feature interactions xie2025mhc ; zhu2024hyper . Figure 3: Mean pairwise cosine similarity among residual streams after being mixed by ℋlresH_l^res. The left shows the baseline with identity mapping (where ℋlresH_l^res is fixed as an identity matrix while keeping all other settings identical to mHC). The right shows mHC. Each colored line tracks a layer depth. 4.2 Expressivity Bottleneck As shown in Figure 3, for both the identity baseline and mHC, pair-wise stream similarity rapidly decreases and stabilizes. This aligns with studies revealing a natural, structural drive toward feature independence in internal representation learning dong2021attention ; valeriani2023geometry ; skean2025layer . In contrast, mHC consistently exhibits a small but systematic elevation in similarity across layers. We pinpoint that this effect is structural rather than incidental. Since ℋlresH_l^res is doubly stochastic, it enforces non-negative convex mixing across residual streams. Even minimal off-diagonal entries thus induce averaging, producing a persistent upward bias in inter-stream similarity, echoing recent theoretical finding liu2026homogeneity on spectral collapse in such doubly stochastic networks. We thus hypothesize that the polytope constraint introduces an intrinsic expressivity bottleneck: as observed in Figure 3 and discussed above, residual streams in internal layers naturally decorrelate during training. However, ℋlresH_l^res is restricted to convex mixing and therefore can only average features across streams. Lacking the ability to form signed interactions, mHC and mHC-lite provide no mechanism to actively enhance representation diversification, limiting its topological expressivity. 5 Methodology Motivated by these observations, we propose Spectral-Sphere-Constrained Hyper-Connections (sHC). Instead of confining the residual matrix ℋlresH_l^res in the Birkhoff polytope ℬnB_n, we reformulate it into a spectral norm sphere restricted to the affine subspace nA_n. 5.1 Affine-Constrained Spectral Sphere We define an affine-constrained spectral norm sphere n=ℋ∈n∣‖ℋ‖2=1S_n=\H _n \|H\|_2=1\. Constraining the residual matrix ℋlresH_l^res within nS_n guarantees three critical properties: 1. Mean Preservation. nS_n resides in nA_n, where the affine constraint ensures that the uniform vector n1_n remains invariant, thereby conserving the mean component across residual streams. 2. Spectral Stability. Any ℋlres∈nH_l^res _n satisfies ‖ℋlres‖2=1\|H_l^res\|_2=1. Moreover, nS_n is closed under multiplication (proof in Appendix C), preventing signal amplification and gradient explosion in deep networks. 3. Enhanced Expressivity. Since all doubly stochastic matrices possess a spectral norm of 1, the Birkhoff polytope ℬnB_n is contained within nS_n. By dropping the non-negativity constraint, nS_n allows negative entries, enabling subtractive interactions such as selective noise suppression and feature diversification, which are prohibited in mHC and mHC-lite. Parameterization Equivalence via Spectral Decoupling. Directly parameterizing ℋlres∈nH_l^res _n with a strict unit spectral norm is challenging. To address this, we note that the affine subspace nA_n is a translation of the zero-marginal subspace n=ℋ∈ℝn×n∣ℋn=n,n⊤ℋ=n⊤Z_n=\H ^n× n 1_n=0_n,1_n H=0_n \ by the uniform matrix J=1nnn⊤J= 1n1_n1_n (Appendix D). Thus, any target residual matrix admits a unique decomposition ℋlres=J+ℋldispH_l^res=J+H_l^disp, where ℋldisp∈nH_l^disp _n. Then consider any input vector ∈ℝn x ^n, which can be uniquely decomposed as =∥+⟂,∥∈spann,⟂∈n⟂. x= x_ + x_ , x_ \1_n\,\ x_ 1_n . The operators J and ℋldispH_l^disp act orthogonally on these components: J⟂=n,ℋldisp∥=n.J x_ =0_n,H_l^disp x_ =0_n. This decouples their spectral contributions and leads to Proposition 1 (proof deferred to Appendix E). Proposition 1 (Spectral Decoupling). Let J=1nnn⊤J= 1n1_n1_n . For any displacement matrix ℋldisp∈nH_l^disp _n, the spectral norm of the corresponding residual matrix ℋlres=J+ℋldispH_l^res=J+H_l^disp satisfies: ‖ℋlres‖2=max(‖J‖2,‖ℋldisp‖2)\|H_l^res\|_2= (\|J\|_2,\|H_l^disp\|_2 ) (5) Since ‖J‖2=1\|J\|_2=1, Proposition 1 establishes that enforcing ‖ℋlres‖2=1\|H_l^res\|_2=1 in the affine subspace nA_n is equivalent to bounding ‖ℋldisp‖2≤1\|H_l^disp\|_2≤ 1 in the zero-marginal subspace nZ_n. 5.2 Spectral-Sphere-Constrained Hyper-Connections Figure 4: Overview of Spectral-Sphere-Constrained Hyper-Connections (sHC). The right orange plane depicts the zero-marginal subspace nZ_n, where the blue disk centered at the origin O represents the bounded spectral region ‖ℋldisp‖2≤1\|H_l^disp\|_2≤ 1. The SVD parameterization generates the displacement matrix ℋldispH_l^disp (red point) within this region. The left blue plane illustrates the target affine space nA_n, containing the Birkhoff polytope ℬnB_n (inner orange polygon), which is enclosed by the affine-constrained spectral norm sphere nS_n (black circle centered at the uniform matrix J). The affine shift +J+J maps the origin O to J and the displacement ℋldispH_l^disp to the final residual matrix ℋlresH_l^res. As established in Proposition 1, enforcing ℋlres∈nH_l^res _n reduces to constructing a displacement matrix ℋldisp∈nH_l^disp _n satisfying ‖ℋldisp‖2≤1\|H_l^disp\|_2≤ 1. Since ℋldispH_l^disp lies in the zero-marginal subspace nZ_n, its rank is at most n−1n-1. We therefore parameterize it via the compact singular value decomposition (SVD): ℋldisp=UlΣlVl⊤H_l^disp=U_l _lV_l (6) where Ul,Vl∈ℝn×(n−1)U_l,V_l ^n×(n-1) satisfy Ul⊤Ul=Vl⊤Vl=In−1U_l U_l=V_l V_l=I_n-1, and Σl∈ℝ(n−1)×(n−1) _l ^(n-1)×(n-1) is a diagonal matrix containing singular values. To satisfy the zero-sum constraint (i.e. ℋldisp∈nH_l^disp _n), we constrain the singular vectors to lie in n⟂1_n via factorizing: Ul=UUlcore,Vl=UVlcoreU_l=U_ZU_l^core, V_l=U_ZV_l^core (7) where U∈ℝn×(n−1)U_Z ^n×(n-1) denotes the truncated Helmert matrix, which forms an orthonormal basis of subspace n⟂1_n , and Ulcore,Vlcore∈ℝ(n−1)×(n−1)U_l^core,V_l^core ^(n-1)×(n-1) are orthogonal matrices. This factorization provides a complete parameterization of the target zero-marginal subspace nZ_n with exact norm preservation (proof in Appendix F). The problem therefore reduces to generating the orthogonal matrices Ulcore,VlcoreU_l^core,V_l^core and bounding Σl _l. Given the normalized residual input l′∈ℝ1×nC x _l ^1× nC at l-th layer, we dynamically generate these three components: Ulcore=Cayley(skew(γlUtanh(τlU(l′WlU)+lU))) U_l^core=Cayley (skew ( _l^U ( _l^U( x_l W_l^U)+ b_l^U) ) ) (8) Vlcore=Cayley(skew(γlVtanh(τlV(l′WlV)+lV))) V_l^core=Cayley (skew ( _l^V ( _l^V( x_l W_l^V)+ b_l^V) ) ) Σl=diag(tanh(τlS(l′WlS)+lS)) _l=diag ( ( _l^S( x_l W_l^S)+ b_l^S ) ) Here, WlU,WlV∈ℝnC×kW_l^U,W_l^V ^nC× k (with k=12(n−1)(n−2)k= 12(n-1)(n-2)) and WlS∈ℝnC×(n−1)W_l^S ^nC×(n-1) are learnable projection weights, while lU,lV∈ℝ1×k b_l^U, b_l^V ^1× k and lS∈ℝ1×(n−1) b_l^S ^1×(n-1) are learnable biases. The parameters τlU,τlV,τlS _l^U, _l^V, _l^S are trainable scalar factors, and γlU,γlV _l^U, _l^V act as learnable rotation magnitude gates. The operator skew(⋅)skew(·) constructs an (n−1)×(n−1)(n-1)×(n-1) skew-symmetric matrix by populating its strictly upper-triangular entries with the k activated outputs and completing the lower triangle via anti-symmetry. Cayley(⋅)Cayley(·) is the Cayley transform which maps the constructed skew-symmetric matrices into orthogonal matrices UlcoreU_l^core and VlcoreV_l^core. diag(⋅)diag(·) maps the output n−1n-1 singular values onto the main diagonal, where the tanh(⋅) (·) bounds the singular values |(Σl)i,i|≤1|( _l)_i,i|≤ 1 for i=1…n−1i=1… n-1, thus ensuring ‖ℋldisp‖2≤1\|H_l^disp\|_2≤ 1. Substituting the parametrized matrices and adding the translation J yields the final residual matrix: ℋlres=J+(UUlcore)Σl(UVlcore)⊤H_l^res=J+(U_ZU_l^core) _l(U_ZV_l^core) (9) Finally, except for the above residual matrix ℋlresH_l^res, our sHC keeps other structures of mHC unchanged. Figure 4 summarizes our sHC. 6 Experiment Models and Datasets. We evaluate sHC in language models and measure its impact on model performance and training efficiency across model scales and datasets. Due to computational constraints and following mHC-lite yang2026mhc , we build on the nanoGPT framework karpathy2022nanogpt and consider two model sizes: M (12 layers, 0.12B parameters), and L (24 layers, 0.36B parameters). Models are trained on FineWeb-Edu penedo2024fineweb and OpenWebText Gokaslan2019OpenWeb . We scale the training tokens proportionally to the model size, allocating a nearly 10×10× token budget that yields approximately 1.3B tokens for the M model and 3.6B for the L model. This ensures comparable training regimes across model sizes and avoids under-training larger models. Baselines. We evaluate the effectiveness of different residual connection paradigms including standard single-stream Residual Connections (RC he2016deep ), unconstrained Hyper-Connections (HC zhu2024hyper ), Manifold-Constrained Hyper-Connections using Sinkhorn iterations (mHC xie2025mhc ), and its permutation-based variant (mHC-lite yang2026mhc ). Aligned with their settings, the number of residual streams for all hyper-connections including our sHC is set to n=4n=4. Initialization. Following the original papers of HC/mHC/mHC-lite, we adopt their initialization schemes so that each variant reduces to a standard residual connection (identity mapping) at initialization. sHC is also set as identity mapping at initialization. Evaluation Metrics. To assess training convergence, we report the final training and validation losses. Furthermore, to evaluate generalization and mitigate the bias of relying solely on the in-domain pre-training corpus, we compute the zero-shot perplexity of the trained models across five diverse out-of-distribution corpora: C4 raffel2020exploring , Dolma V1.5 soldaini2024dolma , Falcon RefinedWeb penedo2023refinedweb , RedPajama weber2024redpajama , and Wikitext-103 merity2016pointer via the Paloma suite paloma within the lm-eval harness eval-harness . Other hyperparameters and detailed initialization are provided in Appendix G. Table 1: Loss of trained models at different scales under different residual connection paradigms. We report training and validation loss at the end of training, with training loss computed as a 200-iteration moving average to mitigate fluctuations. Dataset OpenWebText FineWeb-Edu Model Scale M L M L Train Val Train Val Train Val Train Val RC 3.347 3.328 3.052 3.066 3.313 3.325 3.058 3.048 HC 3.282 3.264 3.111 3.132 3.276 3.288 3.112 3.111 mHC 3.268 3.250 3.000 3.023 3.241 3.255 3.017 3.009 mHC-lite 3.271 3.252 3.001 3.023 3.241 3.254 3.013 3.006 sHC (Ours) 3.225 3.239 2.998 3.012 3.230 3.233 2.993 3.003 6.1 Performance To evaluate different residual connection paradigms, we compare their training convergence (Table 1) and zero-shot generalization capability (Table 2). As shown, unconstrained HC disrupts identity-preserving property of residual connection, even underperforming the standard RC baseline at some cases. Although Birkhoff polytope constrained methods (mHC and mHC-lite) reliably improve upon RC, their rigid polytope constraint limits efficacy. In contrast, our sHC shows performance improvement in both training metrics and the generalization corpora. In particular, on the L scale, sHC achieves observable reductions in perplexity over baselines. This shows that our spectral norm sphere constraint achieves considerable expressivity. We further analyze the expressivity of sHC, which can be found in Appendix A. Table 2: Zero-shot perplexity (PPL) on out-of-distribution corpora with trained models on FineWeb-Edu. Lower values indicate better performance. Model Scale M L Benchmark C4 Dolma Falcon RedPajama Wikitext C4 Dolma Falcon RedPajama Wikitext RC 144.8 397.0 186.5 3152.8 92.1 104.4 260.7 132.5 1725.9 62.0 HC 138.8 367.4 178.6 2890.2 88.1 113.6 264.3 144.6 1517.8 70.4 mHC 132.0 351.8 168.8 2757.8 82.9 98.9 228.4 125.3 1288.4 58.8 mHC-lite 131.7 344.0 168.5 2458.3 81.6 98.3 223.0 124.9 1216.4 58.0 sHC (Ours) 129.0 358.8 165.4 2785.6 79.0 97.2 218.8 123.3 1126.3 57.0 6.2 Stability Analysis Figure 5: Gradient norm dynamics during training for the L model on OpenWebText. The unconstrained HC exhibits exploding gradients (light orange), clamped at 5.0 for visualization. Other residual connection paradigms show stable gradient trajectories. Training Stability. Figure 5 shows the gradient norms during training. The unconstrained HC is unstable, and its gradient fluctuates severely. The constrained hyper-connections (sHC, mHC, and mHC-lite) stabilize the optimization. Notably, mHC and mHC-lite maintain minimal gradient norms from the outset. We attribute this persistently flat profile to their noticeable identity degradation, which likely hinders active feature mixing. In contrast, sHC exhibits an initial rise in gradient norm before converging to a stable level. We attribute this early increase to sHC actively exploring non-trivial feature interactions between residual streams, thereby achieving optimization stability without passively defaulting to the identity mapping. Hyper-Connection Stability. We evaluate the layer-wise and propagation stability of residual matrices for mHC, mHC-lite and sHC. Results are shown in Figure 6. As illustrated in the Left panel, mHC fails to strictly satisfy the doubly stochastic constraint despite 20 Sinkhorn iterations. The column sums of its layer-wise residual matrices ℋlresH_l^res deviate from 1.0, with a peak reaching 1.4. These column-sum deviations accumulate across layers as illustrated in the Middle panel: the composite mapping ∏l=023ℋ23−lres _l=0^23H_23-l^res across 24 layers for mHC exhibits a pronounced shift of its column sums away from 1.0, with accumulated outliers spiking to 1.6. In contrast, both sHC and mHC-lite maintain unit column sums for both the layer-wise residual matrices and the global composite mapping, thereby preserving the mean component of residual streams without signal diminishment or amplification. Furthermore, the Right panel tracks the spectral norm of the cumulative composite mapping ∏l=0L−1ℋL−lres _l=0^L-1H_L-l^res across the first L layers. It reveals that mHC fails to bound this cumulative norm, which increases steadily as network depth grows. Conversely, sHC and mHC-lite constrain the cumulative spectral norm around 1.0, effectively preventing depth-induced gradient amplification. However, although mHC-lite achieves strict stability, as shown in § 6.1, its performance remains limited. It also incurs a factorial parameterization overhead, making scalability to a larger number of residual streams impractical, as we discuss in the following. Figure 6: Empirical validation of the stability of constrained residual matrices for mHC, mHC-lite, and sHC. Statistics are computed from the trained L model with 4 residual streams on the 1024 samples from the validation set. Left: Column-sum distribution of layer-wise residual matrices ℋlresH_l^res for each layer. Middle: Column-sum distribution of the full-model composite mapping ∏l=023ℋ23−lres _l=0^23H_23-l^res. Right: Spectral norm of the cumulative composite mapping ‖∏l=0L−1ℋL−lres‖2\| _l=0^L-1H_L-l^res\|_2 across the first L layers. Lines denote means across samples while shaded regions indicate standard deviations. 6.3 Efficiency and Scalability Figure 7: Training throughput (tokens/s) and parameterization overhead across varying numbers of streams n. Left: Training throughput evaluated on an 8×A6000 setup with a fixed global batch size. The data point for mHC-lite at n=8n=8 is omitted, as its factorial growth in parameterization exceeds the available GPU memory budget and results in an Out-Of-Memory failure. Right: Parameterization overhead (params) of the hyper-connections, excluding base model weights. We evaluate the end-to-end training throughput (tokens/s) and parameterization overhead (params) of mHC, mHC-lite, and sHC based on the M model (0.12B). All experiments are conducted on an 8×A6000 GPU setup with a fixed sequence length of 1024 tokens and a constant global batch size (micro-batch 8, gradient accumulation 16). As shown in Figure 7, sHC and mHC scale stably with compact parameterizations. For sHC, the projection weights (WlU,WlV,WlSW_l^U,W_l^V,W_l^S) have a combined parameter complexity corresponding to ℝnC×(n−1)2R^nC×(n-1)^2. Similar to mHC (which utilizes Wlres∈ℝnC×n2W_l^res ^nC× n^2), they grow polynomially with the number of streams n ((n3)O(n^3)), in contrast to mHC-lite whose weights Wlres∈ℝnC×n!W_l^res ^nC× n! grow factorially ((n⋅n!)O(n· n!)). This difference in scaling explains why sHC and mHC maintain stable throughput, whereas mHC-lite throughput degrades rapidly beyond n=6n=6. At n=8n=8, the auxiliary parameters of mHC-lite reach 6.07B, approximately 51 times the size of the base model, exceeding our GPU memory and resulting in an Out-of-Memory failure. The factorial growth is intrinsic to the mHC-lite parameterization and occurs regardless of base model size. By avoiding this factorial design, sHC alleviates the scalability limitations inherent to mHC-lite. 7 Conclusion In this paper, we introduce Spectral-Sphere-Constrained Hyper-Connections (sHC) to eliminate the parameterization overhead, and alleviate identity degeneration, and expressivity bottleneck issues inherent to prior manifold-constraint hyper-connection methods. By executing a geometric shift from a rigid polytope to a affine-constrained spectral norm sphere, sHC overcomes these limitations. We hope that the expressivity, stability, and scalability of sHC potentially illuminate new pathways for extending residual designs toward the evolution of next-generation foundational architectures. References (1) Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Liang Zhao, et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025. (2) Yongyi Yang and Jianyang Gao. mhc-lite: You don’t need 20 sinkhorn-knopp iterations. arXiv preprint arXiv:2601.05732, 2026. (3) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. (4) Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. (5) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. (6) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. (7) Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. arXiv preprint arXiv:2409.19606, 2024. (8) Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343–348, 1967. (9) Defa Zhu, Hongzhi Huang, Jundong Zhou, Zihao Huang, Yutao Zeng, Banggu Wu, Qiyang Min, and Xun Zhou. Frac-connections: Fractional extension of hyper-connections. arXiv preprint arXiv:2503.14125, 2025. (10) Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019. (11) Garrett Birkhoff. Three observations on linear algebra. Univ. Nac. Tacuman, Rev. Ser. A, 5:147–151, 1946. (12) John Von Neumann. A certain zero-sum two-person game equivalent to the optimal assignment problem1. Contributions to the Theory of Games, (24):5, 1950. (13) Andrej Karpathy. nanogpt. https://github.com/karpathy/nanoGPT, 2022. GitHub repository. (14) Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning, pages 2793–2803. PMLR, 2021. (15) Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36:51234–51252, 2023. (16) Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025. (17) Yizhi Liu. The homogeneity trap: Spectral collapse in doubly-stochastic deep networks. arXiv preprint arXiv:2601.02080, 2026. (18) Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. (19) Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019. (20) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. (21) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, 2024. (22) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023. (23) Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:116462–116492, 2024. (24) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. (25) Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneneh Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. technical report, 2023. (26) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Appendix A Expressivity Analysis Figure 8: Distribution of layer-wise residual matrix entries across mHC, mHC-lite, and our proposed sHC. Figure 9: Visualization of layer-wise residual matrices and composite mapping. This figure displays single-layer residual matrices ℋlresH_l^res at depths (l∈1,6,18,23l∈\1,6,18,23\) and their end-to-end composite mappings (∏l=023ℋ23−lres _l=0^23H_23-l^res) for mHC (top row), mHC-lite (middle row), and our proposed sHC (bottom row). Each matrix is computed by averaging over all tokens within a sequence. The labels annotated along the y-axis and x-axis indicate the row sum and the column sum, respectively. Residual Matrix Dynamics. We analyze the entry distributions (Figure 8) and heatmap visualizations (Figure 9) of the learned layer-wise residual matrices ℋlresH_l^res, evaluated on 1024 samples from the trained large model. For both mHC and mHC-lite, matrix entries sharply concentrate around 0.00 and 1.00, producing sparse and near-identity layer-wise residual matrices. These observations are consistent with the identity degeneration discussed in § 4. In contrast, sHC exhibits a continuous entry distribution with a noticeable portion of values in the negative region, indicating that it leverages the extended value space to enable subtractive feature interactions. Importantly, this increased expressivity does not compromise stability. As shown in Figure 9, sHC maintains exact row and column sums of 1.00 across all depths, whereas mHC accumulates normalization errors, resulting in composite column sums ranging from 0.82 to 1.24. Figure 10: Mean pairwise cosine similarity among residual streams after being mixed by ℋlresH_l^res. The left shows the baseline with identity mapping (where ℋlresH_l^res is fixed as an identity matrix while keeping all other settings identical to sHC). The right shows our sHC. Each colored line tracks a layer depth. Expressivity of sHC. We track the mean pairwise cosine similarity among residual streams throughout the training process to examine the impact of sHC on feature evolution. As Figure 10 illustrates, compared to the identity mapping baseline (where the residual matrix in each layer is fixed as an identity matrix), the introduction of sHC drives inter-stream similarity to lower converging values across layers. This distinct decorrelation phenomenon provides empirical support for our hypothesis regarding the expressivity bottleneck of mHC and mHC-lite formulated in § 4. Specifically, while the doubly stochastic constraint in mHC and mHC-lite forcibly averages features and induces a persistent upward bias in similarity, sHC allows residual streams to decorrelate by leveraging its extended signed interactions. By structurally accommodating these negative interactions, sHC promotes the diversification of representations that is essential to mitigate representation collapse. Appendix B Observation: Stream Similarity Figure 11: Mean pairwise cosine similarity among residual streams after being mixed by ℋlresH_l^res. The left shows the baseline with identity mapping (where ℋlresH_l^res is fixed as an identity matrix while keeping all other settings identical to mHC-lite). The right shows mHC-lite. Each colored line tracks a layer depth. Appendix C Proof of Closure Proof. For any ℋ1,ℋ2∈nH_1,H_2 _n, the affine constraints are preserved under multiplication: (ℋ1ℋ2)n (H_1H_2)1_n =ℋ1(ℋ2n)=ℋ1n=n, =H_1(H_21_n)=H_11_n=1_n, (10) n⊤(ℋ1ℋ2) 1_n (H_1H_2) =(n⊤ℋ1)ℋ2=n⊤ℋ2=n⊤. =(1_n H_1)H_2=1_n H_2=1_n . Hence ℋ1ℋ2∈nH_1H_2 _n. For the spectral norm, submultiplicativity gives ‖ℋ1ℋ2‖2≤‖ℋ1‖2‖ℋ2‖2=1.\|H_1H_2\|_2≤\|H_1\|_2\|H_2\|_2=1. (11) On the other hand, since ℋ1ℋ2n=nH_1H_21_n=1_n, we have ‖ℋ1ℋ2‖2=sup≠0‖ℋ1ℋ2‖2‖2≥‖ℋ1ℋ2n‖2‖n‖2=1.\|H_1H_2\|_2= _ x≠ 0 \|H_1H_2 x\|_2\| x\|_2≥ \|H_1H_21_n\|_2\|1_n\|_2=1. (12) Therefore 1≤‖ℋ1ℋ2‖2≤1,1≤\|H_1H_2\|_2≤ 1, (13) which implies ‖ℋ1ℋ2‖2=1\|H_1H_2\|_2=1 and hence ℋ1ℋ2∈nH_1H_2 _n. ∎ Appendix D Proof of Affine Translation Proof. Let J=1nnn⊤J= 1n1_n1_n , which satisfies Jn=n,n⊤J=n⊤.J1_n=1_n, 1_n J=1_n . (14) For any matrix ℋH, ℋ∈n⟺(ℋ−J)n=, 1n⊤(ℋ−J)=⊤⟺ℋ−J∈n.H _n\; \;(H-J)1_n=0,\;1_n (H-J)=0 \; \;H-J _n. (15) Therefore, n=J+n.A_n=J+Z_n. (16) ∎ Appendix E Proof of Proposition 1 Proof. Let ∈ℝn x ^n, which can be orthogonally decomposed =∥+⟂ x= x_ + x_ , where ∥∈spann x_ \1_n\ and ⟂∈n⟂ x_ 1_n . Thus, ‖22=‖∥‖22+‖⟂‖22\| x\|_2^2=\| x_ \|_2^2+\| x_ \|_2^2. Since J=1nnn⊤J= 1n1_n1_n and ℋldisp∈nH_l^disp _n, we have: J⟂=n,ℋldisp∥=n,andn⊤ℋldisp⟂=0.J x_ =0_n, _l^disp x_ =0_n, 1_n H_l^disp x_ =0. (17) Applying ℋlres=J+ℋldispH_l^res=J+H_l^disp to x, the action decouples: ℋlres=(J+ℋldisp)(∥+⟂)=J∥+ℋldisp⟂.H_l^res x=(J+H_l^disp)( x_ + x_ )=J x_ +H_l^disp x_ . (18) Since J∥∈spannJ x_ \1_n\ and ℋldisp⟂∈n⟂H_l^disp x_ 1_n , the output terms are mutually orthogonal. Consequently: ‖ℋlres‖22 \|H_l^res x\|_2^2 =‖J∥‖22+‖ℋldisp⟂‖22 =\|J x_ \|_2^2+\|H_l^disp x_ \|_2^2 (19) ≤‖J‖22‖∥‖22+‖ℋldisp‖22‖⟂‖22 ≤\|J\|_2^2\| x_ \|_2^2+\|H_l^disp\|_2^2\| x_ \|_2^2 ≤max(‖J‖22,‖ℋldisp‖22)(‖∥‖22+‖⟂‖22) ≤ (\|J\|_2^2,\|H_l^disp\|_2^2 ) (\| x_ \|_2^2+\| x_ \|_2^2 ) =max(‖J‖22,‖ℋldisp‖22)‖22. = (\|J\|_2^2,\|H_l^disp\|_2^2 )\| x\|_2^2. Dividing by ‖22\| x\|_2^2 and taking the supremum over ≠n x 0_n explicitly establishes the upper bound: ‖ℋlres‖2=sup≠‖ℋlres‖2‖2≤max(‖J‖2,‖ℋldisp‖2).\|H_l^res\|_2= _ x 0 \|H_l^res x\|_2\| x\|_2≤ (\|J\|_2,\|H_l^disp\|_2 ). (20) To establish the lower bound, we evaluate the quotient over specific subspaces. For any non-zero vector ∈spann u \1_n\, we have ℋldisp=nH_l^disp u=0_n, which implies ℋlres=J.H_l^res u=J u. Therefore, ‖ℋlres‖2=sup≠‖ℋlres‖2‖2≥sup∈spann,≠‖J‖2‖2=‖J‖2.\|H_l^res\|_2= _ x 0 \|H_l^res x\|_2\| x\|_2≥ _ u \1_n\,\, u 0 \|J u\|_2\| u\|_2=\|J\|_2. (21) Similarly, for any non-zero vector ∈n⟂ v 1_n , we have J=J v=0, which implies ℋlres=ℋldisp.H_l^res v=H_l^disp v. Thus, ‖ℋlres‖2=sup≠‖ℋlres‖2‖2≥sup∈n⟂,≠‖ℋldisp‖2‖2=‖ℋldisp‖2.\|H_l^res\|_2= _ x 0 \|H_l^res x\|_2\| x\|_2≥ _ v 1_n ,\, v 0 \|H_l^disp v\|_2\| v\|_2=\|H_l^disp\|_2. (22) Combining these conditions yields ‖ℋlres‖2≥max(‖J‖2,‖ℋldisp‖2)\|H_l^res\|_2≥ (\|J\|_2,\|H_l^disp\|_2 ). Since both the upper and lower bounds hold, exact equality is established: ‖ℋlres‖2=max(‖J‖2,‖ℋldisp‖2).\|H_l^res\|_2= (\|J\|_2,\|H_l^disp\|_2 ). (23) ∎ Appendix F Proof of Completeness and Spectral Preservation Proof. We prove that the parameterization ℋldisp=(UUlcore)Σl(UVlcore)⊤H_l^disp=(U_ZU_l^core)\, _l\,(U_ZV_l^core) (24) covers the subspace nZ_n and preserves the spectral norm. 1. Completeness. For any ℋldisp∈nH_l^disp _n with rank r≤n−1r≤ n-1, consider its strict compact SVD: ℋldisp=U~lΣ~lV~l⊤,H_l^disp= U_l\, _l\, V_l , (25) where U~l,V~l∈ℝn×r U_l, V_l ^n× r satisfy U~l⊤U~l=V~l⊤V~l=Ir U_l U_l= V_l V_l=I_r, and Σ~l∈ℝr×r _l ^r× r is a diagonal matrix containing only non-zero singular values. From ℋldispn=nH_l^disp1_n=0_n and n⊤ℋldisp=n⊤1_n H_l^disp=0_n , we obtain U~lΣ~l(V~l⊤n)=n,(n⊤U~l)Σ~lV~l⊤=n⊤. U_l\, _l\,( V_l 1_n)=0_n, (1_n U_l)\, _l\, V_l =0_n . (26) Since U~l U_l and V~l V_l have full column rank and Σ~l _l is invertible, this implies V~l⊤n=r,n⊤U~l=r⊤, V_l 1_n=0_r, 1_n U_l=0_r , (27) hence the column spaces of U~l U_l and V~l V_l lie in n⟂1_n . To align with the parameterization size n−1n-1, since n⟂1_n is an (n−1)(n-1)-dimensional subspace, we can find (n−1−r)(n-1-r) orthonormal vectors in n⟂1_n to expand U~l U_l and V~l V_l into Ul,Vl∈ℝn×(n−1)U_l,V_l ^n×(n-1), such that col(Ul),col(Vl)⊂n⟂col(U_l),col(V_l) 1_n and Ul⊤Ul=Vl⊤Vl=In−1U_l U_l=V_l V_l=I_n-1. By padding Σ~l _l with zeros to form an (n−1)×(n−1)(n-1)×(n-1) diagonal matrix Σl _l, we have equivalently: ℋldisp=UlΣlVl⊤.H_l^disp=U_l\, _l\,V_l . (28) Since col(Ul),col(Vl)⊂n⟂col(U_l),col(V_l) 1_n and U_Z is an orthonormal basis of n⟂1_n , we have Ul=UU⊤Ul,Vl=UU⊤Vl.U_l=U_Z\,U_Z U_l, V_l=U_Z\,U_Z V_l. (29) Define Ulcore=U⊤UlU_l^core=U_Z U_l and Vlcore=U⊤VlV_l^core=U_Z V_l. Then (Ulcore)⊤Ulcore (U_l^core) U_l^core =Ul⊤(UU⊤Ul)=Ul⊤Ul=In−1, =U_l (U_ZU_Z U_l)=U_l U_l=I_n-1, (30) (Vlcore)⊤Vlcore (V_l^core) V_l^core =Vl⊤(UU⊤Vl)=Vl⊤Vl=In−1. =V_l (U_ZU_Z V_l)=V_l V_l=I_n-1. Therefore any ℋldisp∈nH_l^disp _n can be written as ℋldisp=(UUlcore)Σl(UVlcore)⊤.H_l^disp=(U_ZU_l^core) _l(U_ZV_l^core) . (31) 2. Spectral norm preservation. Given the parameterization Ul=UUlcoreU_l=U_ZU_l^core and Vl=UVlcoreV_l=U_ZV_l^core with orthogonal core matrices, we have: Ul⊤Ul=(Ulcore)⊤(U⊤U)Ulcore=In−1,U_l U_l=(U_l^core) (U_Z U_Z)U_l^core=I_n-1, (32) Vl⊤Vl=(Vlcore)⊤(U⊤U)Vlcore=In−1,V_l V_l=(V_l^core) (U_Z U_Z)V_l^core=I_n-1, (33) Thus both UlU_l and VlV_l have exact orthonormal columns. Then the spectral norm satisfies: ‖ℋldisp‖2=‖UlΣlVl⊤‖2=‖Σl‖2=maxi|(Σl)i,i|.\|H_l^disp\|_2=\|U_l _lV_l \|_2=\| _l\|_2= _i|( _l)_i,i|. (34) Hence bounding Σl _l directly controls the spectral norm of ℋldispH_l^disp. ∎ Appendix G Experiment Setup Initialization. In all cases, WlresW_l^res, WlpreW_l^pre, and WlpostW_l^post are initialized to zero; αlres _l^res, αlpre _l^pre, and αlpost _l^post are set to 0.01; and lpre b_l^pre and lpost b_l^post are initialized to −1-1 except for a single entry set to 11. The residual branch differs across variants. In HC, lres b_l^res is initialized to I. In mHC, off-diagonal entries are set to −8-8 and diagonal entries to 0, yielding an identity-like matrix after Sinkhorn normalization. For mHC-lite, lres b_l^res is set to −8-8 for all entries except the entry corresponding to the identity matrix, which is set to 0, so that after the softmax operation the weights concentrate on the identity matrix. For sHC initialization, projection weights WlU,WlV,WlSW_l^U,W_l^V,W_l^S are zero-initialized. We set the biases lU,lV b_l^U, b_l^V to zero, so that Ulcore,VlcoreU_l^core,V_l^core reduce to identity after the Cayley transform. Simultaneously, we initialize lS b_l^S to 4, saturating the tanh activation to yield an identity matrix for Σl _l. Finally, the rotation magnitude gates γlU,γlV _l^U, _l^V are fixed at 1, and the scaling factors τlU,τlV,τlS _l^U, _l^V, _l^S are set to 0.01. Training Setup. Our implementation builds upon nanoGPT [13], with all unspecified hyperparameters left at their default values. Models are trained from scratch using the AdamW optimizer with a cosine learning rate schedule and linear warmup. We employ mixed-precision training (bfloat16) and apply gradient clipping throughout training. All experiments are conducted on 8 NVIDIA A6000 GPUs using PyTorch DistributedDataParallel (DDP) with the NCCL backend. We fix the global batch size across all methods. The hyperparameters of training are summarized in Table 3. Table 3: Training hyperparameters. Setting M L Micro-batch size per GPU 8 8 Gradient accumulation steps 16 48 Sequence length 1024 1024 Training iterations 10000 10000 LR decay iterations 10000 10000 Warmup iterations 200 200 Weight decay 0.1 0.1 β1 _1 0.9 0.9 β2 _2 0.95 0.95 Gradient clipping norm 1.0 1.0 Initial learning rate 6×10−46× 10^-4 3×10−43× 10^-4 Minimum learning rate 6×10−56× 10^-5 3×10−53× 10^-5 Model Configurations. We evaluated two model scales: Medium (M) and Large (L). Their architecture-specific configurations are listed in Table 4. Table 4: Architecture configurations for the M, and L models. Configuration M L Number of layers 12 24 Number of attention heads 12 16 Hidden dimension 768 1024