Paper deep dive
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce \textbf{DepthBench}, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ($d_{\text{model}}/n_{\text{layer}}$) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.
Tags
Links
- Source: https://arxiv.org/abs/2609.32534v1
- Canonical: https://arxiv.org/abs/2609.32534v1
Trouble viewing inline? Open PDF directly →
Full Text
85,735 characters extracted from source content.
Expand or collapse full text
DEPTHBENCH: Measuring How Residual Connections Enable More Computational Depth Keyu Wang 1,2,3∗ , Yangyi Huang 4∗ , Jiale Kang 4 , David Gonz ́ alez-Mart ́ ınez 1,2,3 , Weiyang Liu 4 , Shiwei Liu 1,2,3 1 ELLIS Institute T ̈ubingen, 2 Max Planck Institute for Intelligent Systems, 3 T ̈ubingen AI Center, 4 The Chinese University of Hong Kong ∗ Equal contribution. Correspondence to: keyu.wang@tuebingen.mpg.de, sliu@tue.ellis.eu Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (e.g., LayerNorm Scaling) or residual connections (e.g., mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce DepthBench, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width–depth aspect ratio (d model /n layer ) from shallow–wide to deep–narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation. Code: DepthBenchCheckpoints: HuggingFace 305070 400M 800M Model Size HC or Full AttnRes sweet zone Pre-LN sweet zone Aspect Ratio (d model /n layer ) deeper – narrowershallower – wider GPT-2 Large GPT-2 Medium Qwen3-0.6B Qwen3.5-0.8B MobileLLM-350M SmolLM2-360M MobileLLM-R1 360M 9.119.028.042.756.076.0 2.68 2.70 2.72 2.74 2.76 2.78 Validation Loss ↓ Aspect Ratio (d model /n layer ) deeper – narrowershallower – wider (b) Model Aspect Ratios in the Wild (a) DEPTHBENCH Pre-LNFull AttnResHCBlock AttnRes mHCSandwich-LNLNS Figure 1. Left: Validation loss across aspect ratios under an approximately fixed 400M parameter budget and shared pre-training recipe. Right:Model sizes and aspect ratios of representative industry-released models with Pre-LN architecture, overlaid with shaded regions illustrating the contrasting shape preferences observed in DepthBench. 1 arXiv:2609.32534v1 [cs.CV] 26 Sep 2026 1. Introduction Scaling depth is a fundamental way to increase the computational capacity of Transformers, as more layers allow representations to undergo a longer sequence of nonlinear transformations (Levine et al., 2020; Sanford et al., 2024). However, current LLMs suffer from the curse of depth (Sun et al., 2026), where purely increasing architectural depth does not necessarily yield a proportional increase in model capacity. In conventional Pre-LN Transformers, residual connections make deep networks easier to optimize (Xiong et al., 2020), but they do not ensure that every layer contributes useful computation (Sun et al., 2026). As models become deeper, later-layer updates can become increasingly weak, redundant, or similar to those of preceding layers (Li et al., 2024; Sun et al., 2026; Team et al., 2026b; Liu et al., 2026). Consequently, additional layers may therefore increase nominal depth without comparable gains in effective computation or model quality. Recent work addresses this problem by modifying normalization or residual propagation. For instance, LayerNorm Scaling (LNS) (Sun et al., 2026) and KEEL (Chen & Wei, 2026) refine normalization, whereas AttnRes (Team et al., 2026b), HC (Zhu et al., 2025), and mHC (Xie et al., 2025) redesign the residual pathway. These methods report improved optimization and model quality at large scales, and related designs are being adopted in frontier LLMs such as Kimi K3 (Team et al., 2026a) and DeepSeek V4 (Xu et al., 2026). However, it remains unclear whether they truly translate increased architectural depth into effective computational depth 1 . Existing methods are typically evaluated with different architectures, training recipes, and system configurations. Their gains therefore cannot be cleanly attributed to improved optimization, better access to information across depth, preservation of diverse features, or differences in capacity and computational overhead. We therefore still lack a unified and comprehensive understanding of which architectural designs can reliably translate more architectural depth into useful computational depth. This motivates our primary research question: Which architectural designs make additional depth computational ly effective? Answering this question determines when depth can serve as a reliable scaling axis rather than merely increasing the number of layers. To address this question, we develop DepthBench to study when depth pays off under a fixed parameter budget. Varying the aspect ratio (d model /n layer ) reallocates parameters between width and depth, enabling a direct comparison between shallow-wide and deep-narrow architectures with a broad range of aspect ratios for 10 representative residual designs while holding both the parameter count and training recipe fixed. We sweep multiple learning rates for every aspect ratio and report each configuration at its best-performing learning rate, ensuring that differences reflect the architecture rather than a suboptimal learning rate. We then rank model performance across configurations and conduct controlled layer-level analyses to determine whether each approach enables effective aspect ratio scaling by making better use of the greater depth. Our results reveal several insights about how residual connections affect computational depth: •Conventional residual connections do not benefit consistently from deeper architectures at iso- parameter (Section 3). Pre-LN and its normalization/residual-scaling variants do not benefit from deeper, narrower architectures. Their optima remain at relatively large aspect ratios 42.7–76.0 (e.g.,d model = 1120 and n layer = 20), obscuring aspect ratio as a useful scaling axis. • New residual designs unleash the power of deeper models even with extremely small aspect ratios (Section 3). HC and Full AttnRes continue to reduce pre-training loss as models become deeper-narrower under iso-parameter scaling, even at an extreme aspect ratio of 9.1 withd model = 640 andn layer = 70. Effective residual design thus turns aspect ratio into a practical scaling dimension. •Alternative residual mechanisms fundamentally change how computation evolves across depth 1 Effective computational depth is the amount of sequential computation that a model can exploit through successive layers, as measured by the incremental benefit of additional layers or iterations under a controlled compute budget. 2 (Section 4.1 and Section 4.2). Full AttnRes and HC preserve heterogeneous, layer-specific transformations across layers, unlike the increasingly homogeneous deep-layer representations in Pre-LN. • The benefits of depth scaling extend beyond pre-training loss (Section 3 and Section 4.1). The gains of HC and Full AttnRes transfer to domain-specific evaluations. Layer-wise diagnostics further indicate that these methods make more effective use of the additional computational depth. •No free lunch: deep models introduce a systems-level efficiency trade-off (Section 4.4). Although deeper architectures can improve modeling performance, they increase compute and memory overhead while reducing hardware utilization. Realizing their benefits at scale therefore requires better infrastructure and kernel optimization. 2. Preliminaries & Setup 2.1. Modern Deep Transformer Architectures Pre-LN Transformers with Residual Connections. Modern LLMs predominantly adopt the Pre-LN Transformer architectures (Xiong et al., 2020), where layer normalization is applied before each transformation branch and the resulting update is added to the residual stream (Baevski & Auli, 2018; Dai et al., 2019). Formally, a Pre-LN sublayer updates the hidden state as: h ℓ = h ℓ−1 + F ℓ (LN(h ℓ−1 ))(1) whereF ℓ denotes the attention or feed-forward transformation at layerℓ, andLNis typically instantiated as RMSNorm (Zhang & Sennrich, 2019). Compared with Post-Layer Normalization (Post-LN) (Ba et al., 2016), Pre-LN substantially improves training stability at large depth by preserving a direct residual pathway for gradient propagation (Xiong et al., 2020; Li et al., 2024). Pre-LN Issues and Improvements. While Pre-LN largely resolves the trainability issue of deep Transformers (Xiong et al., 2020), it does not necessarily guarantee effective compuational depth: a model can be made very deep without fully utilizing its later layers (Li et al., 2024; Sun et al., 2026). This limitation has been widely described as the Curse of Depth (Sun et al., 2026) and Pre-LN dilution (Team et al., 2026b), where the contribution of deeper layers progressively weakens. A growing line of architectures therefore modifies normalization and residual connections, seeking better depth utilization (Chen & Wei, 2026; Sun et al., 2026; Team et al., 2026b; Li et al., 2026). To provide a thorough evaluation, we categorize the architectures considered in this work into four groups, summarized in Table 1: • Baseline. We employ standard Pre-LN (Xiong et al., 2020) as our primary baseline. •Normalization Variants. This category primarily modifies layer normalization. We include Sandwich-LN (also called Peri-LN) (Ding et al., 2021; Kim et al., 2025), which introduces normalization before and after transformation branch; LNS (Sun et al., 2026), which adjusts the scale of LayerNorm outputs in Pre-LN; DeepNorm (Wang et al., 2024) and KEEL (Chen & Wei, 2026), which improve upon the Post-LN formulation to enable more stable gradient propagation in deep Transformers. •Multi-stream Residuals. These architectures extend the conventional single residual stream into multiple streams. We include HC (Zhu et al., 2025) and mHC (Xie et al., 2025), which maintain and mix multiple residual streams to provide more flexible cross-layer information propagation. • Cross-layer Access. Finally, this category enables each layer to directly access representations from earlier depths, rather than receiving information solely through recursive propagation from the immediately preceding layer. We include AttnRes (Team et al., 2026b), which aggregates preceding hidden states through attention-based residual connections, and MoDA (Zhu et al., 2026), which dynamically combines KV cache representations from earlier layers. 3 Table 1. Overview of residual update mechanisms for the architectures (Team et al., 2026b). Architectures differ primarily in how layerℓreceives and aggregates information from preceding layers. Weight: whether the mixing coefficients are fixed or input-dependent (dynamic). Source: which earlier representations layer ℓ can access. MethodUpdate ruleWeightSource Baseline Pre-LN (Xiong et al., 2020)h ℓ = h ℓ−1 + F ℓ (LN(h ℓ−1 ))Fixed h ℓ−1 LayerNorm variants Sandwich-LN (Ding et al., 2021) h ℓ = h ℓ−1 + LN out (F ℓ (LN in (h ℓ−1 )))Fixed h ℓ−1 LNS (Sun et al., 2026)h ℓ = h ℓ−1 + F ℓ 1 √ ℓ LN(h ℓ−1 ) Fixed h ℓ−1 DeepNorm (Wang et al., 2024)h ℓ = LN(αh ℓ−1 + F ℓ (h ℓ−1 ))Fixed h ℓ−1 KEEL (Chen & Wei, 2026)h ℓ = LN(αh ℓ−1 + F ℓ (LN(h ℓ−1 )))Fixed h ℓ−1 Multi-stream residuals HC (Zhu et al., 2025)H ℓ = H ℓ−1 A ℓ + F ℓ (LN(H ℓ−1 α ℓ ))β ⊤ ℓ Dynamic m residual streams mHC (Xie et al., 2025) H ℓ = H ℓ−1 Sinkhorn( e A ℓ ) + F ℓ (LN(H ℓ−1 σ(eα ℓ ))) 2σ( e β ⊤ ℓ ) Dynamic m residual streams Cross-layer access AttnRes (Team et al., 2026b) Full: h ℓ ∝ P ℓ−1 i=0 φ(w ℓ ,v i )v i , v 0 = h 1 , v i≥1 = f i (h i )Dynamic [h 1 ,...,h ℓ−1 ] Block: h ℓ ∝ P n−1 i=0 φ(w ℓ ,v i )v i + φ(w ℓ ,v j n )v j n Dynamic [v 0 ,...,v n−1 ,v j n ] MoDA (Pre-LN) (Zhu et al., 2026) h ℓ = h ℓ−1 + MoDA ℓ (LN(h ℓ−1 );C <ℓ )Dynamic h ℓ−1,≤t ;h 0,t ,...,h ℓ−2,t Table 2. Overview of the experimental settings in DepthBench. SettingMatched Budget # Shapes Aspect Ratios Purpose Main benchmark≈400M total79.1–76.0Comprehensive architecture comparison Iso-backbone control ≈300M backbone66.5–78.0Control for backbone-size variation Multi-scale suite200M–500M total3 ≈28, ≈43, ≈76Test consistency across model scales Validation at scale ≈1.6B total327.9, 43.2, 73.1 Validate trends at larger scale 2.2. DepthBench Design Controlled Width–Depth Aspect Ratio Scaling. Simply increasing depth at fixed width also increases model size and compute, confounding the effect of depth with generic scale. DepthBench instead treats depth as a capacity al location choice, varyingd model /n layer from shallow–wide to deep–narrow configurations under an approximately fixed parameter budget. Architectural Backbone. To isolate the effects of norm- and residual- design, we instantiate all variants on a shared LLaMA-like backbone. For sub-1B models, the backbone uses multi-head self-attention (MHA) (Vaswani et al., 2017) with 16 heads, rotary positional embeddings (RoPE) (Su et al., 2024), RMSNorm (Zhang & Sennrich, 2019) withε= 10 −6 , SwiGLU feed-forward layers (Shazeer, 2020), the GPT-NeoX tokenizer (Black et al., 2022) with a vocabulary size of 50280, and untied input and output embeddings. For the 1.6B models, we follow the Qwen3-1.7B attention configuration (Yang et al., 2025), replacing MHA with grouped-query attention (GQA) (Ainslie et al., 2023) using 16 query heads and 8 key-value heads. All other architectural choices remain unchanged. Architecture-specific implementation details are provided in Appendix A. Model Configurations. We organize DepthBench into four complementary experimental suites, summarized in Table 2. Our main benchmark operates at the 400M total size scale and compares 10 architectures across seven model shapes, spanning aspect ratios from 76.0 (d model = 1216,n layer = 16) to 9.1 (d model = 640,n layer = 70). We additionally conduct an iso-backbone control, in which the Transformer backbone size, excluding the input embeddings and LM head, is held fixed at 300M parameters. This removes a confound of fixed-total-parameter comparisons: narrower models allocate fewer parameters to the embedding and LM head, and thus a larger fraction of the total budget to the Transformer backbone (Porian et al., 2024). To assess robustness across scale, we evaluate three representative aspect ratios at200M, 300M, 400M, 500Mparameters. Finally, we test whether the same trends persist at the 1.6B scale under three shapes. 400M model configurations are shown in Table 3 and full model configurations for the other three experimental suites are provided in Appendix B. For each depthL=n layer , we choose hidden dimensiond=d model such that the relevant parameter budget is 4 Table 3. 400M model configurations across different width–depth aspect ratios. LayersHiddenIntermediateHeadsHead Dim.Aspect RatioBackbone SizeTotal SizeTotal Diff. 1612163248167676.00284M407M+0.28% 2011202992167056.00301M414M+2.14% 2410242736166442.67302M405M0.00% 289602560166034.29310M406M+0.21% 328962400165628.00309M399M −1.49% 428002144165019.05324M404M −0.31% 70640171216409.14345M409M+0.94% Table 4. Optimal learning rate across architectures. Pre-LNSandwich-LNLNSDeepNormKEELMoDAFull AttnResBlock AttnResHCmHC 2× 10 −3 2× 10 −3 1× 10 −2 1× 10 −3 2× 10 −3 2× 10 −3 2× 10 −3 2× 10 −3 2× 10 −3 2× 10 −3 approximately matched. For sub-1B models, we set the intermediate dimension tod f =ceil 16 (8d/3),whereceil 16 denotes rounding up to the next multiple of 16. For the 1.6B models, we used f = 3d. Since total parameters scale approximately asN ∝12Ld 2 + 2V dwhereVis vocabulary size, increasing depth under a fixed total or backbone parameter budget necessarily requires reducing width. This construction yields a controlled spectrum from shallow–wide to deep–narrow shapes, enabling us to isolate how architectures trade off capacity between width and depth. 2.3. Pre-training Settings We use OLMo-core 2 to pre-train all models from scratch on FineWeb-Edu (Penedo et al., 2024), with a token budget of 20 tokens per parameter following the Chinchilla scaling law (Hoffmann et al., 2022) (especially, 8B tokens for 400M models and 32B for 1.6B models), and reserve a disjoint held-out split for evaluation. We use a sequence length of 2048 and a global batch size of 512 sequences, corresponding to approximately 1M tokens per optimization step. Optimization uses AdamW (Loshchilov & Hutter, 2017) withβ 1 = 0.9,β 2 = 0.95,ε= 10 −8 , weight decay 0.1, and gradient clipping at 1.0. We follow the default OLMo-core initialization with normally distributed weights and standard deviation 0.02. The learning rate follows cosine decay with a 10% linear warmup and decays to 10% of its peak value (Loshchilov & Hutter, 2016). For 400M models, we perform architecture-specific learning rate sweeps over5×10 −4 ,1×10 −3 ,2×10 −3 ,5×10 −3 , except for LNS, for which we use2×10 −3 , 5×10 −3 , 1×10 −2 , 2×10 −2 . The optimal learning rate is consistent across width–depth shapes within each architecture and reported in Table 4. We provide the detailed learning rate sweep results in Appendix C. We reuse these learning rates for the iso-backbone and 200M–500M multi-scale experiments, and use 5×10 −4 for 1.6B models following common practice (Sun et al., 2026; Li et al., 2024). All other pre-training settings are held fixed across architectural variants and aspect ratios. 3. Main Results Figure 2 shows validation loss on the 400M benchmark across a wide range of architectures and aspect ratios. Figure 3 shows validation loss across aspect ratios of Pre-LN, Full AttnRes and HC in 200M–500M and 1.6B suites. To complement the validation loss, we further evaluate teacher-forced negative log-likelihood (NLL) on coding, STEM, and math tasks in Figure 4, following the NLL-based pre-training evaluation protocol in (Team, 2026; Chen et al., 2026). We observe a clear pattern: whether increasing depth helps strongly depends on the residual connections. Pre-LN and normalization variants are largely insensitive or even unfavorable to increasingly deep–narrow shape. As shown in Figure 2 (a), for Pre-LN, validation loss monotonically increases from 2.759 atL= 16 to 2 https://github.com/allenai/OLMo-core 5 28.0 (32, 896) 34.3 (28, 960) 42.7 (24, 1024) 56.0 (20, 1120) 76.0 (16, 1216) Aspect ratio d/L; (L, d) 2.70 2.72 2.74 2.76 2.78 2.80 Validation loss ↓ log scale (a) Architecture comparison 9.1 (70, 640) 19.1 (42, 800) 28.0 (32, 896) 42.7 (24, 1024) 56.0 (20, 1120) 76.0 (16, 1216) 2.68 2.72 2.76 2.80 Validation loss ↓ log scale (b) Fixed total size: 400M 6.48 (84, 544) 354M 8.7 (70, 608) 373M 18.3 (42, 768) 375M 28 (32, 896) 399M 42.7 (24, 1024) 405M 78 (16, 1248) 425M Aspect ratio d/L; (L, d); total size 2.70 2.74 2.78 2.82 Validation loss ↓ log scale (c) Fixed backbone size: 300M Pre-LN Full AttnRes Sandwich-LN Block AttnRes LNS HC DeepNorm mHC KEEL MoDA Figure 2. Left: 400M main results of validation loss across 10 architectures. Right: Validation loss across a wider range of shapes, including extreme small aspect ratios, under fixed total size (400M, top) and fixed backbone size (300M, bottom) for Pre-LN, HC and Full AttnRes. 2.782 atL= 32, showing that reallocating parameters from width to depth hurts performance. Sandwich-LN, LNS, DeepNorm, KEEL and MoDA also show either weak or non-monotonic trends, with their optima occurring at intermediate or shallower shapes. HC and Full AttnRes scale favorably with depth and even surprisingly continue to improve at extremely deep shapes. In contrast, both HC and Full AttnRes consistently improve as models become deeper and narrower. Full AttnRes steadily improves from 2.751 atL= 16 to 2.718 atL= 32, while HC improves from 2.729 atL= 16 to 2.699 atL= 32. This trend is the opposite of Pre-LN, suggesting that these new residual designs strongly prefer deep shapes. We further extend the scaling range, up to 70 layers. As shown in Figure 2 (b-c), the favorable trend persists in both fixing total size and backbone size. Notably, with the backbone size fixed, deep–narrow models improve even as the total size decreases. This suggests that their depth-scaling gains arise from the width–depth allocation itself rather than simply from increased backbone size. Overall, HC and Full AttnRes are exceptionally well suited to depth scaling, with performance continuing to improve as models are pushed toward increasingly deep–narrow, even extreme, shapes. The favorable width-depth scaling behavior of HC and Full AttnRes is less evident in their derived variants, Block AttnRes and mHC. As shown in Figure 2 (a), Block AttnRes remains competitive but performs best at an intermediate shape, reaching 2.715 atL= 20 before degrading to 2.730 atL= 32. Similarly, mHC achieves strong overall performance and its best performance at L = 24 but shows no consistent gain with increasing depth. A wide range of scales coupled with domain-specific evaluation shows the consistent trend. As shown in Figure 3, across 200M–500M models, Full AttnRes consistently benefits from deeper–narrower shapes, and this trend persists at 1.6B, suggesting that its favorable depth scaling extends beyond the 400M regime. HC shows a similar trend at smaller scales, but exhibits less stable behavior as scale increases: the 500M run at the largest aspect ratio encounters gradient explosion, while the 1.6B results do not show the same clear improvement with depth. The reason is that HC is more sensitive to optimization hyperparameters and may require finer learning-rate tuning across scales, consistent with the motivation of mHC to improve the large-scale optimization stability of unconstrained HC (Zhu et al., 2025; Xie et al., 2025). As shown in Figure 4, deeper HC and Full AttnRes models generally achieve lower NLL across coding, STEM, and math evaluations. The trend is particularly clear for STEM and math, where both architectures consistently improve as capacity is shifted toward depth. In contrast, Pre-LN and most other variants show weaker or 6 200300400500 2.70 2.80 2.90 3.00 Validation loss ↓ Pre-LN 3.006 3.047 2.694 2.700 200300400500 Full AttnRes 2.950 2.963 2.651 2.678 200300400500 diverged HC 2.937 2.970 2.639 2.648 Aspect ratio d/L: Model size (M) (a) 200–500M Multi-scale Models ≈ 28≈ 43 ≈ 76 27.943.273.1 Aspect Ratio 2.39 2.40 2.41 2.42 2.43 2.44 2.45 Validation loss ↓ log scale (b) 1.6B Models Pre-LNFull AttnResHC Figure 3. Validation loss for Pre-LN, Full AttnRes, and HC under three aspect ratios. Left:200M, 300M, 400M, 500M models. Right: 1.6B models. 28.0 (32, 896) 34.3 (28, 960) 42.7 (24, 1024) 56.0 (20, 1120) 76.0 (16, 1216) Aspect ratio d/L; (L, d) 3.45 3.55 3.65 3.75 3.85 3.95 NLL loss ↓ log scale (a) Coding (MBPP, HumanEval) 28.0 (32, 896) 34.3 (28, 960) 42.7 (24, 1024) 56.0 (20, 1120) 76.0 (16, 1216) Aspect ratio d/L; (L, d) 2.36 2.38 2.40 2.42 2.44 2.46 2.48 NLL loss ↓ log scale (b) STEM (SciQ, GPQA) 28.0 (32, 896) 34.3 (28, 960) 42.7 (24, 1024) 56.0 (20, 1120) 76.0 (16, 1216) Aspect ratio d/L; (L, d) 2.85 2.90 2.95 3.00 3.05 3.10 3.15 3.20 NLL loss ↓ log scale (c) Math (GSM8K, MATH-500) Pre-LN Full AttnRes Sandwich-LN Block AttnRes LNS HC DeepNorm mHC KEEL MoDA Figure 4. NLL results of 400M models on Coding, STEM and Math categories. We report negative log-likelihood across model aspect ratios and architectures. Each panel shows the average NLL over two representative tasks: MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021) for coding (left), SciQ (Welbl et al., 2017) and GPQA (Rein et al., 2023) for STEM (middle), and GSM8K (Cobbe et al., 2021) and MATH-500 (Lightman et al., 2024) for math (right). non-monotonic trends. These results suggest that the favorable width-depth scaling translates to domain-specific predictive capability rather than only lower held-out pre-training loss. Overall, our results identify HC and Full AttnRes as the two architectures with the clearest favorable depth-scaling behavior. We next study how these architectures utilize their layers. 4. Analysis 4.1. Evaluating Depth Utilization Across Architectures Prior work has shown that deep layers in Pre-LN Transformers can become increasingly ineffective (Sun et al., 2026; Yang et al., 2026). In particular, representations produced by neighboring deep layers become increasingly similar (Sun et al., 2026; Liu et al., 2026), suggesting that many layers only make small refinements to the residual stream (Csord ́as et al., 2025). We therefore ask whether the favorable depth scaling is accompanied by more effective utilization across their layers. Representation diversity across depth. We first examine the angular distance between representations at different depths, following prior work (Sun et al., 2026; Li et al., 2024). For representationsh ℓ andh ℓ+n after the 7 051015202530 0 5 10 15 20 Pre-LN 051015202530 0 5 10 15 20 Sandwich-LN 051015202530 0 5 10 15 20 LNS 051015202530 0 5 10 15 20 KEEL 051015202530 0 5 10 15 20 MoDA 051015202530 0 5 10 15 20 HC 051015202530 0 5 10 15 20 mHC 051015202530 0 5 10 15 20 Full AttnRes 051015202530 0 5 10 15 20 Block AttnRes 0.0 0.1 0.2 0.3 0.4 0.5 Angular distance Layer Index ℓ Subsequent n th Layer Figure 5. Angular distance from the initial layerℓ(x-axis) and its subsequentn th layer (y-axis). Angular distance ranges in [0, 1] and yellow indicates smaller distances and dark blue indicates larger distances. Here, we use a unified learning rate of 2×10 −3 for all models (only LNS does not use the optimal learning rate), because the learning rate significantly affects the learned representations, as shown in Appendix C. We exclude DeepNorm because it explodes at a learning rate of 2× 10 −3 . outputs 3 from layers ℓ and ℓ + n, respectively, the angular distance is defined as: d(h ℓ ,h ℓ+n ) = 1 π arccos h ℓ · h ℓ+n ∥h ℓ ∥ 2 ∥h ℓ+n ∥ 2 .(2) A smaller angular distance indicates more similar representations and therefore less representational change. As shown in Figure 5, The hidden representations of Pre-LN, Sandwich-LN, LNS, DeepNorm and KEEL become broadly similar across depth, generally similar with patterns in (Sun et al., 2026; Li et al., 2024). In contrast, HC- and AttnRes- architectures exhibit not only larger angular distances, but more importantly, a markedly non-smooth structure across depth. Their representations continue to change in a layer-specific and non-uniform manner, rather than gradually converging toward small refinements of the same underlying representation. Block AttnRes exhibits a distinct block-structured pattern: representations remain relatively smooth within each block, but change sharply across block boundaries. mHC shows a more moderate non-smooth pattern, but remains clearly distinct from the progressively smoothed representations like Pre-LN. The key distinction is therefore not merely the magnitude of representational change, but its structure across 3 The representation here corresponds toh ℓ orH ℓ in Table 1. For HC and mHC, the outputs from the four streams are concatenated together; for AttnRes, we use the output after each depth mixing. 8 Pre-LN Sandwich-LN LNS DeepNorm KEEL MoDA Full AttnRes Block AttnRes HC mHC 0% 25% 50% 75% 100% Fraction of layer pairs L16L24L32 08162431 Affected layer ℓ 0 8 16 24 31 Skipped layer s Pre-LN 08162431 Affected layer ℓ LNS 08162431 Affected layer ℓ Full AttnRes 08162431 Affected layer ℓ HC 0.0 0.5 1.0 Pre-LN Sandwich-LN LNS DeepNorm KEEL MoDA Full AttnRes Block AttnRes HC mHC 0% 25% 50% 75% 100% Fraction of layer pairs L16L24L32 08162431 Layer ℓ 2 0 8 16 24 31 Layer ℓ 1 Pre-LN 08162431 Layer ℓ 2 LNS 08162431 Layer ℓ 2 Full AttnRes 08162431 Layer ℓ 2 HC 0.0 0.5 1.0 (a) Causal score > 0.45 Layer pairs in the last 3/4 of the network (b) Causal score · L32 (c) Permutation score > 0.15 Layer pairs in the last 3/4 of the network (d) Permutation score · L32 Figure 6. (a,c) Fraction of layer pairs in the last 3/4 of the network exceeding the causal-score (>0.45) and permutation- score (>0.15) thresholds, respectively. (b,d) Pairwise causal and permutation score maps forL= 32 models. Higher causal scores indicate stronger cross-layer dependence, while higher permutation scores indicate lower layer interchangeability. depth: HC, mHC and AttnRes preserve heterogeneous, layer-specific transformations, whereas other architectures progressively collapses toward smoother and increasingly similar deep-layer representations. Layer perturbations across depth. We further study whether each layer is functionally important by adopting causal scores and permutation scores from (Muhtar et al., 2026; Csord ́as et al., 2025) to quantify how strongly individual layers affect subsequent computation and how interchangeable different layers are. For a skipped layer s and a subsequent layer ℓ > s, the causal score is defined as: C(s,ℓ) = (h ℓ+1 − h ℓ )− ( ̄ h ℓ+1 − ̄ h ℓ ) 2 ∥h ℓ+1 − h ℓ ∥ 2 ,(3) wherehdenotes the original hidden states and ̄ hdenotes the hidden states obtained after skipping layers. A largerC(s,ℓ) means that removing layersmore strongly changes the update performed by a subsequent layerℓ, indicating stronger causal dependence across depth. We additionally measure how sensitive the model is to exchanging the order of two layers. For layersℓ 1 andℓ 2 , the permutation score is defined as: P (ℓ 1 ,ℓ 2 ) = L(M )−L(M swap(ℓ 1 ,ℓ 2 ) ) L(M ) ,(4) whereL(M) denotes the language modeling loss of the original model andL(M swap(ℓ 1 ,ℓ 2 ) ) denotes the loss of the model after swapping layersℓ 1 andℓ 2 . A larger permutation score indicates that exchanging the two layers causes a larger performance degradation, suggesting that they perform more specialized and order-dependent computations. While we use the pairwise scores above, our aggregation differs from that of (Muhtar et al., 2026). We find that the first few layers are typically important across almost all architectures, which can dominate a global layer effectiveness statistic and obscure differences in how later layers are utilized. Since our focus is specifically on effective deep computation, we therefore restrict the analysis to the last three quarters of the network and measure the fraction of pairwise scores that exceed a fixed threshold. Specifically, let S late =(s,ℓ) : s≥⌈L/4⌉, ℓ > s(5) 9 denote the causal pairs whose skipped layer lies in the last three quarters of the network. We summarize causal utilization as: R causal = 1 |S late | X (s,ℓ)∈S late 1 [C(s,ℓ) > τ c ], τ c = 0.45.(6) Similarly, for layer pairs P late =(ℓ 1 ,ℓ 2 ) : ℓ 1 ≥⌈L/4⌉, ℓ 2 > ℓ 1 ,(7) we define: R perm = 1 |P late | X (ℓ 1 ,ℓ 2 )∈P late 1 [P (ℓ 1 ,ℓ 2 ) > τ p ], τ p = 0.15.(8) These statistics directly measure how frequently later layers exhibit substantial causal influence or non- interchangeable computation, while avoiding the universally strong effects of the earliest layers. As shown in Figure 6, Pre-LN and its normalization variants, as well as MoDA, exhibit very few strong causal interactions in its deeper layers. For theL= 32 model, only about 1% of Pre-LN’s late-layer causal scores exceed 0.45, compared with roughly 16% for Full AttnRes and 7% for HC. The separation is even clearer under layer permutation: only about 8% of Pre-LN permutation scores exceed 0.15, compared with approximately 46% for Full AttnRes and 35% for HC. Although focusing on the last three quarters could potentially benefit Post-LN-based methods such as DeepNorm and KEEL, these methods still do not exhibit larger causal or permutation scores in our analysis. mHC likewise shows a marked drop in permutation sensitivity, consistent with its weaker width-depth scaling. The corresponding pairwise heatmaps show the same pattern, with substantially stronger cross-layer dependence and order sensitivity in Full AttnRes and HC. Complete visualizations are provided in Appendix E. Taken together, Pre-LN, most normalization-based variants and MoDA become broadly smooth across depth, with weak causal influence and limited sensitivity to layer permutation, indicating that many late layers contribute only small and partly substitutable refinements. In contrast, HC and Full AttnRes maintain non-smooth, layer-specific representational changes together with substantially stronger causal dependence and permutation sensitivity, suggesting that their additional layers remain functionally consequential. 4.2. Characterizing Depth Processing in Full AttnRes and HC Section 4.1 shows that AttnRes and HC retain stronger cross-layer dependence and order sensitivity than Pre-LN and its normalization variants. We next examine whether these architectures realize more effective depth through different computational mechanism. Explicit paths across depth. Figure 7 traces how earlier sublayer outputs contribute to later computations. Pre-LN accumulates all updates in a single residual stream, givingw i→ℓ ≡1. Full AttnRes instead forms a normalized, non-negative mixture over the embedding and all preceding sublayer outputs, providing direct access to earlier computations. HC realizes cross-depth access recursively, with the contribution of branch outputf i to branchℓgiven byw i→ℓ =β ⊤ i A i+1 ·A ℓ−1 α ℓ .Its learned paths are strongest locally but remain substantial over longer ranges. Thus, Full AttnRes directly retrieves stored earlier outputs, whereas HC propagates and recombines them through interacting residual streams. Functional consequences. We first prune individual layers and measure the resulting change in ARC-Easy accuracy (Figure 8 (a)). Pre-LN and HC show similar patterns: both are dominated by the first layer, while pruning most later layers causes only modest degradation. Full AttnRes is sensitive across a broader range, with strong dependence on both its first and final layers. We further apply LogitLens (Csord ́as et al., 2025) to track how predictions evolve across depth. For Pre-LN, we decode the current residual stream; for Full AttnRes, the mixture produced by the next layer’s depth-mixing query; and for HC 4 , the sum of its four residual streams. For probe stateh ℓ , we computep (ℓ) probe =softmax(W LM Norm(h ℓ )) 4 We implement HC following the original HC paper (Zhu et al., 2025), where the final output is an sum of the four residual streams. Later HC/mHC implementations, such as DeepSeek V4 (Xu et al., 2026), may instead use a learnable final mixer. 10 08162431 Writing sublayer i (block) 0 8 16 24 31 Reading sublayer ℓ (block) w i → ℓ ≡ 1 (a) Pre-LN 08162431 Writing sublayer i (block) (b) Full AttnRes 08162431 Writing sublayer i (block) (c) HC 0.00 0.05 0.10 0.15 0.20 Softmax weight w i → ℓ (token mean) −0.5 0.0 0.5 1.0 2.0 Path weight w i → ℓ (token mean) Figure 7. Explicit residual-path weightsw i→ℓ . Each cell shows the mixing weight of sublayer outputf i to the pre-norm input of a later sublayerℓ, averaged over validation calibration tokens. (a) Pre-LN:w i→ℓ ≡1. (b) Full AttnRes:w i→ℓ is the per-token softmax weight onf i . (c) HC:w i→ℓ =β ⊤ i A i+1 ·A ℓ−1 α ℓ , combining write mappings, residual mappings, and read mappings. −0.2 −0.1 0.0 Pre-LN (a) Performance drop of layer pruning −0.2 −0.1 0.0 Full AttnRes 08162431 Pruned layer ℓ −0.2 −0.1 0.0 HC 18162432 Completed layers ℓ 0 2 4 6 8 10 D KL ( p final ‖ p ( ℓ ) probe ) (b) LogitLens Pre-LN Full AttnRes HC Figure 8. Left: Performance drop after removing a single layer. Right: The KL divergence between layerwise and final prediction distributions. and reportD KL (p final ∥p (ℓ) probe ). As shown in Figure 8 (b), Pre-LN approaches the final prediction monotonically, consistent with progressive refinement of a single residual stream. HC follows a similarly smooth trajectory, but its earlier representations remain farther from the final prediction. Full AttnRes is markedly less monotonic, consistent with continued retrieval and late integration of stored sub-layer output sources. Together, these results suggest that Full AttnRes and HC need not make every layer individually indispensable. Rather, their learned cross-depth routing preserves differentiated access to earlier computations, maintaining stronger cross-layer dependence and order sensitivity instead of reducing later layers to largely interchangeable refinements. 4.3. Potential Hidden Cost for mHC and Block AttnRes Figure 2 shows that the scaling behavior of Full AttnRes and HC is less evident in their derived variants, Block AttnRes and mHC. We investigate the underlying causes of this discrepancy. Block AttnRes. Block AttnRes restricts the sources for depth-mixing to a slower growing schedule. Layer outputs are accumulated into a running sum s ℓ , and only at a block boundary, everyb/2 layers for block sizeb, is the running sum appended to the setB ℓ of saved block outputs that later layers may attend over and then reset. At every layer the depth-mixing softmax is taken over s ℓ ∪B ℓ , so the mixed input is ̃ h ℓ ∝ α ℓ s ℓ + X v∈B ℓ β ℓ,v v, α ℓ + P v β ℓ,v = 1,(9) 11 1481216 Layer 0.0 0.5 1.0 Depth-mixing weight block size 4 · boundary every 2 layers L16 16121824 Layer block size 6 · boundary every 3 layers L24 18162432 Layer block size 8 · boundary every 4 layers L32 Depth-mixing weight on last layer outputDepth-mixing weight on previous block outputsBlock boundary Figure 9. Block AttnRes depth-mixing mechanism atL= 16, 24, 32. At each layer, the share of the depth-mixing weight in Eq. 9 placed on the last layer output (α ℓ , green) versus on previous block outputs ( P v β ℓ,v , grey); dashed lines mark block boundaries, where the set of saved block outputs is updated. 28.0 (32,896) 34.3 (28,960) 42.7 (24,1024) 56.0 (20,1120) 76.0 (16,1216) Aspect ratio d/L 1 2 3 4 Effective rank Identity Uniform mixing (a) Across aspect ratios HCmHC 08162432 Blocks traversed 0.0 0.5 1.0 Direction similarity (b) Direction overlap, L32 1234 Singular-value index 0.0 0.5 1.0 Relative strength σ j / σ 1 (c) Singular spectrum, L32 Figure 10. Residual transport in HC and mHC. (a) Effective rank across aspect ratios, with shapes (L,d) below each tick. Effective rank is the exponential of the entropy of the normalized singular values. (b) Mean absolute cosine similarity between propagated stream directions inL= 32 models. (c) Singular values of their full residual products, normalized by the largest. Statistics are averaged over sampled held-out FineWeb-Edu data; bands and error bars show 95% document-bootstrap intervals. whereα ℓ is the weight on the last layer output (the running sum) and theβ ℓ,v are the weights on previous block outputs. We follow original AttnRes (Team et al., 2026b) and Kimi K3 (Team et al., 2026a) settings to fix the number of boundaries at 8, and the block size grows with depth (b= 4,5,6,7,8 forL= 16,20,24,28,32). Figure 9 shows how this schedule shapes the computation, by plotting α ℓ for the pre-trained models. Before the first block output has been saved, the depth-mixing softmax assigns essentially zero weight to the running sum: each layer’s own computation is dropped from the mixed input, and the network effectively keeps re-reading the embedding. Only once saved block outputs exist doesα ℓ rise. The length of this initial dead segment scales with depth (2, 5, and 8 layers atL= 16, 24, and 32), so a quarter of theL= 32 network can not benefit from stacking layers. This potentially explains why Block AttnRes stops benefiting from added depth. mHC. mHC retains HC’s multi-stream structure but constrains its read/write weights to be non-negative and uses Sinkhorn–Knopp normalization to make residual mappings approximately doubly stochastic (Xie et al., 2025). These constraints aim to stabilize propagation across depth. However, as shown in Figure 2, mHC performs best at L = 24 and does not retain HC’s gains in deeper, narrower models. To examine this difference, we compose all attention and FFN residual maps along each forward pass. As shown in Figure 10(a), the resulting products have consistently lower effective rank in mHC: 1.44–1.65, compared with 2.53–2.81 for HC. AtL= 32, mHC’s propagated stream directions become nearly collinear, with much weaker non-leading singular values, as shown in Figure 10(b–c). The doubly stochastic constraint suggests a mechanism for this concentration. Exact doubly stochastic maps 12 76.042.728.09.1 aspect ratio d/L 3.0 3.2 3.4 3.6 3.8 4.0 4.2 4.4 TFLOPs (a) Prefill FLOPs / sequence & val. loss 76.042.728.09.1 aspect ratio d/L 0.3 0.4 0.5 0.6 0.7 GiB identical across architectures (b) KV cache / sequence 76.042.728.09.1 aspect ratio d/L 6 8 10 12 GiB (c) Peak memory (batch 8) 76.042.728.09.1 aspect ratio d/L 2 4 6 8 10 12 A100 GPU-hours (d) GPU-hours / 1B prefill tokens (batch 8) 2.68 2.70 2.72 2.74 2.76 2.78 Validation loss ↓ Pre-LNHCFull AttnRes Figure 11. Measured prefill cost including FLOPs, KV cache memory, Peak GPU memory and GPU hours at sequence length = 4096, model size at about 400M, BF16, A100-SXM4-40GB. The dashed line in (a) corresponds to validation loss. share the uniform mixing matrix as a fixed point: U = 1 m 11 ⊤ , A ℓ U = UA ℓ = U.(10) When repeated mixing contracts the remaining stream directions, residual products approachU, making earlier contributions less distinguishable across streams even as new branch outputs enter. Different sufficiently mixing orders of fixed residual maps then approach the same average, consistent with mHC’s weaker permutation effects as shown in Figure 6. Together, these observations suggest a trade-off: stabilizing residual transport can reduce the diversity of long-range routes through which additional layers reuse earlier computations. 4.4. No Free Lunch: The Accuracy–Efficiency Trade-off of Depth Scaling Depth scaling is not free even under a fixed parameter budget. SinceN ≈12Ld 2 + 2V d, while the dominant projection cost proportional toLd 2 remains approximately constant, quantities proportional toLdincrease when scaling depth. For example, during prefill with sequence lengthT, attention prefill cost scales asO(T 2 Ld) and becomes increasingly important in the long-context regimeT ≫ d, while KV cache memory similarly scales as O(TLd). We provide the FLOPs counting analysis in Appendix D. Moreover, the practical latency overhead can exceed the increase suggested by FLOPs alone, as deeper models incur lower hardware utilization. Figure 11 reports prefill cost at sequence lengthT= 4096 for our 400M models in BF16 on A100-SXM4-40GB GPUs under specific setting. We measure forward FLOPs and KV-cache size per sequence, as well as peak memory and GPU-hours for a batch size of 8. A first observation is that depth itself introduces a non-negligible systems cost even at a fixed parameter scale. As the aspect ratio decreases from 76.0 to 9.1, prefill FLOPs increase from roughly 3.0 to 4.4 TFLOPs per sequence, while the KV cache increases by more than 2×. The latter is identical across architectures, confirming that this component is intrinsic to the width–depth reallocation rather than to a particular residual design. More importantly, the practical cost of depth grows substantially fast. Peak memory and GPU-hours increase sharply toward the deepest configurations. Full AttnRes has the largest peak-memory footprint, consistent with its explicit cross-layer source access, whereas HC shows the highest measured GPU-hour cost despite having a much smaller difference in nominal FLOPs. Notably, those algorithmic and empirical forward costs do not directly reflect training wall clock. Deeper models incur more sequential execution and smaller matrix multiplications, so their practical cost depends strongly on hardware, kernels, parallelism, and memory behavior. When fixing parameter counts, depth scaling therefore introduces a systems-level efficiency trade-off rather than a universally fixed compute penalty. HC and Full AttnRes obtain better modeling performance from additional depth, but realizing these gains at scale likely requires architecture–system co-design. In particular, specialized kernels and infrastructure-level aspecs such as pipeline scheduling, communication overlap among others might shape the practical frontier. To facilitate further exploration of these systems-level trade-offs, we release our codebase together with checkpoints spanning the wide range of model aspect ratios. 13 5. Related Work Scaling Laws and Model Aspect Ratios. Scaling laws have primarily characterized language modeling perfor- mance as a function of model size, data, and training compute (Kaplan et al., 2020; Hoffmann et al., 2022). Within the regimes considered by early studies, architectural choices such as network width and depth were found to have comparatively limited effects once model size was controlled (Kaplan et al., 2020). However, subsequent work has shown that how capacity is allocated across architectural dimensions can matter. Levine et al. (2020) characterizes distinct depth-efficient and depth-inefficient regimes in self-attention. Tay et al. (2021) and Petty et al. (2024) show that model shape can substantially affect downstream fine-tuning performance, implying that deep models potentially have better generalization. Relatedly, MobileLLM (Liu et al., 2024) demonstrates the practical effectiveness of deep-and-thin architectures in the sub-billion-parameter regime. More recently, Gemstones (McLeish et al., 2025) systematically studies scaling across diverse model shapes and shows that scaling-law prescriptions can be sensitive to width–depth choices and experimental design. These studies, however, largely examine model shape on fixed conventional model architectures. In contrast, we study width–depth allocation across different normalization and residual architectures and show that whether reallocating capacity from width to depth is beneficial depends strongly on residual connection design. Deep Transformer Training and the Curse of Depth. The original Post-LN Transformer (Ba et al., 2016) can exhibit unstable gradients at initialization, while Pre-LN substantially improves optimization stability by providing better-behaved gradient propagation (Xiong et al., 2020). Subsequent work further improves the trainability of deep Transformers through normalization, residual scaling, and initialization. For example, DeepNorm (Wang et al., 2024) combines residual scaling with depth-aware initialization to stably train Post-LN Transformers with up to thousands of layers, while Sandwich-LN (Ding et al., 2021; Kim et al., 2025) improves activation-variance control and gradient stability through putting normalization both before and after the transformation branch. More recent work has shifted attention from merely training deep models to whether their additional layers are effectively utilized. Mix-LN (Li et al., 2024) shows that Pre-LN can produce increasingly weak gradients in deeper layers and combines Pre- and Post-LN to improve their contribution. Similarly, the Curse of Depth (Sun et al., 2026) attributes ineffective deep layers in Pre-LN Transformers to depth-dependent residual-stream variance growth and introduces LayerNorm Scaling to strengthen deep-layer updates. Recent approaches further revisit the Pre-/Post-Norm trade-off: KEEL (Chen & Wei, 2026) stabilizes very deep Post-LN Transformers through highway-style residual connections, SiameseNorm (Li et al., 2026) decouples Pre- and Post-Norm behavior using two coupled streams, and KiteNorm (Trochelmann et al., 2026) regularizes hidden-state variance to stabilize Post-LN training. These studies establish that trainability and depth utilization are distinct: a Transformer can be optimized successfully at large architectural depth while still deriving limited benefit from its later layers. Our work takes the next step by asking whether improved deep-layer utilization is sufficient to make depth a favorable scaling axis under a fixed parameter budget. Residual Connections. Residual connections are a fundamental mechanism for enabling information and gradient propagation through deep networks. Highway Networks (Srivastava et al., 2015) introduced gated shortcuts to regulate information flow across layers, while ResNet (He et al., 2016) established identity residual connections as a simple and effective means of optimizing substantially deeper networks. Beyond local identity shortcuts, DenseNet (Huang et al., 2017) showed the benefit of providing layers with direct access to earlier representations, motivating richer forms of cross-depth information reuse. Recent work has generalized this idea in Transformers by making residual propagation increasingly learnable. DenseFormer (Pagliardini et al., 2024) mixes current and preceding representations through learned depth-weighted averaging, while LAuReL (Menghani et al., 2024) augments the canonical residual layer with learnable residual mappings. Hyper-Connections (HC) (Zhu et al., 2025) expand the single residual stream into multiple streams with learned read, write, and mixing operations. mHC (Xie et al., 2025) further constrains these mappings to restore the identity-preserving properties required for stable large-scale training. Complementarily, AttnRes (Team et al., 2026b) replace fixed residual accumulation with input-dependent attention over preceding layer outputs, enabling selective access to computations across depth, and Block AttnRes trades this fine-grained access for a more efficient block-wise variant. Related approaches such as MoDA (Zhu et al., 2026) and Depth-Attention 14 (Zeng et al., 2026) provide direct cross-layer access through depth-wise key–value representations. While these methods demonstrate that richer residual pathways can improve optimization and model quality, they are typically evaluated under different model shapes and training settings. We instead study them under controlled width–depth aspect ratio scaling, asking whether their residual topology actually converts additional architectural depth into effective computation. 6. Conclusion In this work, we introduce DepthBench, a controlled benchmark for studying when architectural depth translates into effective computational depth in Transformers. By systematically varying width–depth aspect ratios under approximately fixed parameter and training budgets, we show that the benefit of allocating more capacity to depth is strongly architecture-dependent. In particular, HC and Full AttnRes consistently benefit from deeper and narrower model shapes, while Pre-LN and most normalization-based variants show much weaker or even unfavorable depth scaling. Our layer-wise analyses further suggest that these gains are associated with more heterogeneous, consequential, and order-sensitive computation across depth, rather than simply improved trainability. We also show the hidden cost of mHC and Block AttnRes. Finally, these gains come with additional efficiency costs, highlighting an important accuracy–efficiency trade-off. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis for Transformer models. References Ainslie, J., Lee-Thorp, J., De Jong, M., Zemlyanskiy, Y., Lebr ́on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 4895–4901, 2023. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. Gpt-neox-20b: An open-source autoregressive language model, 2022. URL https://arxiv.org/abs/2204.06745. Chen, C. and Wei, L. Post-layernorm is back: Stable, expressive, and deep. arXiv preprint arXiv:2601.19895, 2026. Chen, H., Zhang, H., Li, X., Dong, Y., Shen, K., and Zhu, J. Nexus: Same pretraining loss, better downstream generalization via common minima. arXiv preprint arXiv:2604.09258, 2026. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Csord ́as, R., Manning, C. D., and Potts, C. Do language models use their depth efficiently? In Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., and Chen, N. (eds.), Advances in Neural Information Processing Systems, volume 38, Main Conference, p. 160313–160362. Curran Associates, Inc., 2025. doi: 10.52202/085713-5357. URLhttps://proceedings.neurips.c/paper_files/paper/2025/file/ eae3af0f5868f0a2eceb74208966d55b-Paper-Conference.pdf. 15 Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 2978–2988, 2019. Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34: 19822–19835, 2021. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778, 2016. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), p. 2261–2269. Ieee, 2017. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Kim, J., Lee, B., Park, C., Oh, Y., Kim, B., Yoo, T., Shin, S., Han, D., Shin, J., and Yoo, K. M. Peri-ln: Revisiting normalization layer in the transformer architecture. arXiv preprint arXiv:2502.02732, 2025. Levine, Y., Wies, N., Sharir, O., Bata, H., and Shashua, A. Limits to depth efficiencies of self-attention. Advances in Neural Information Processing Systems, 33:22640–22651, 2020. Li, P., Yin, L., and Liu, S. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795, 2024. Li, T., Han, D., Cao, Z., Huang, H., Zhou, M., Chen, M., Zhao, E., Jiang, X., Jiang, G., and Huang, G. Siamesenorm: Breaking the barrier to reconciling pre/post-norm. arXiv preprint arXiv:2602.08064, 2026. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, p. 39578–39601, 2024. Liu, Y., Kangaslahti, S., Liu, Z., and Gore, J. Inverse depth scaling from most layers being similar. arXiv preprint arXiv:2602.05970, 2026. Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905, 2024. Loshchilov, I. and Hutter, F.Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. McLeish, S., Kirchenbauer, J., Miller, D., Singh, S., Bhatele, A., Goldblum, M., Panda, A., and Gold- stein, T.Gemstones: A model suite for multi-faceted scaling laws.In Belgrave, D., Zhang, C., Lin, H., Pascanu, R., Koniusz, P., Ghassemi, M., and Chen, N. (eds.), Advances in Neural Infor- mation Processing Systems, volume 38, Main Conference, p. 123419–123454. Curran Associates, Inc., 2025. doi: 10.52202/085713-4114. URLhttps://proceedings.neurips.c/paper_files/paper/2025/file/ b2b781badeeb49896c4b324c466ec442-Paper-Conference.pdf. Menghani, G., Kumar, R., and Kumar, S. Laurel: Learned augmented residual layer.arXiv preprint arXiv:2411.07501, 2024. 16 Muhtar, D., Song, X., Pokutta, S., Zimmer, M., Pelleriti, N., Hofmann, T., and Liu, S. When does sparsity mitigate the curse of depth in llms. arXiv preprint arXiv:2603.15389, 2026. Pagliardini, M., Mohtashami, A., Fleuret, F., and Jaggi, M. Denseformer: Enhancing information flow in transformers via depth weighted averaging. Advances in neural information processing systems, 37:136479– 136508, 2024. Penedo, G., Kydl ́ıˇcek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Von Werra, L., and Wolf, T.The fineweb datasets: Decanting the web for the finest text data at scale.In Glober- son, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, p. 30811–30849. Curran Associates, Inc., 2024. doi: 10.52202/079017-0970.URLhttps://proceedings.neurips.c/paper_files/paper/2024/file/ 370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf. Petty, J., van Steenkiste, S., Dasgupta, I., Sha, F., Garrette, D., and Linzen, T. The impact of depth on compositional generalization in transformer language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7239–7252, 2024. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535–100570, 2024. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. Sanford, C., Hsu, D., and Telgarsky, M. Transformers, parallel computation, and logarithmic depth. arXiv preprint arXiv:2402.09268, 2024. Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. Srivastava, R. K., Greff, K., and Schmidhuber, J. Highway networks. arXiv preprint arXiv:1505.00387, 2015. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. Sun, W., Song, X., Li, P., Yin, L., Zheng, Y., and Liu, S. The curse of depth in large language models, 2026. URL https://arxiv.org/abs/2502.05795. Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686, 2021. Team, K., Bai, T., Bai, Y., Bao, Y., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., Charles, Y., et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026a. Team, K., Chen, G., Zhang, Y., Su, J., Xu, W., Pan, S., Wang, Y., Wang, Y., Chen, G., Yin, B., et al. Attention residuals. arXiv preprint arXiv:2603.15031, 2026b. Team, T. M. A. Mai-thinking-1: Building a hill-climbing machine. Technical report, Microsoft AI, 2026. URL https://microsoft.ai/pdf/mai-thinking-1.pdf. Trochelmann, L. A., Movahedi, S., Liu, S., and Orvieto, A. Kitenorm: Variance regularisation for stable and scalable post-ln transformers. In High-dimensional Learning Dynamics 2026, 2026. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017. Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intel ligence, 46(10):6761–6774, 2024. 17 Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, p. 94–106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4413. URL https://aclanthology.org/W17-4413/. Xie, Z., Wei, Y., Cao, H., Zhao, C., Deng, C., Li, J., Dai, D., Gao, H., Chang, J., Yu, K., et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In International conference on machine learning, p. 10524–10533. PMLR, 2020. Xu, A., Lin, B., Xue, B., Wang, B., Xu, B., Wu, B., Zhang, B., Lin, C., Dong, C., Ling, C., et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang, W., Ma, Y., Conzelmann, A., Zheng, X., Mahoney, M. W., Rusch, T. K., and Liu, S. Alphaq: Calibration-free bit allocation for mixture-of-experts quantization. arXiv preprint arXiv:2606.04980, 2026. Zeng, B., Hao, Y., Wang, Z., Song, S., Li, H., Song, F., Liu, Y., He, Z., Wang, X., and Lin, Z. Depth-attention: Cross-layer value mixing for language models. arXiv preprint arXiv:2606.05014, 2026. Zhang, B. and Sennrich, R. Root mean square layer normalization. Advances in neural information processing systems, 32, 2019. Zhu, D., Huang, H., Huang, Z., Zeng, Y., Mao, Y., Wu, B., Min, Q., and Zhou, X. Hyper-connections. In International Conference on Learning Representations, volume 2025, p. 97183–97219, 2025. Zhu, L., Fang, Y., Liao, B., Wang, S., Cheng, T., Huang, Z., Chen, C., Wei, L., Zeng, Y., Wang, Y., et al. Mixture-of-depths attention. arXiv preprint arXiv:2603.15619, 2026. 18 A. Architecture-specific Settings All variants share the LLaMA-like backbone described in Section 2.1. Here we record every architecture-specific choice that deviates from it. Table 6 summarises the resulting parameter counts. Pre-LN (Xiong et al., 2020). The reference blockh ℓ =h ℓ−1 +F ℓ (RMSNorm(h ℓ−1 )) with one pre-normalisation per sub-layer and no additional parameters. Sandwich-LN (Ding et al., 2021). Each sub-layer output is normalised a second time before the residual addition, h ℓ =h ℓ−1 +RMSNorm out F ℓ (RMSNorm in (h ℓ−1 )) , giving four RMSNorms per block. The output norms are initialised to the identity (unit scale). This adds 2d parameters per block. LayerNorm Scaling (LNS) (Sun et al., 2026). The normalised input of both sub-layers in blockℓ(1-indexed) is multiplied by 1/ √ ℓbefore entering attention and the feed-forward network; the same factor is used for both sub-layers of a block, so it ranges from 1 in the first block to 1/ √ Lin the last (e.g., 0.20 atL=24, 0.18 atL=32). LNS introduces no parameters but requires a 5×larger optimal learning rate (1×10 −2 ) than the other variants. DeepNorm (Wang et al., 2024). Based on Post-LN (Ba et al., 2016), DeepNorm up-scales the residual connection before performing layer normalization. h ℓ = LN αh ℓ−1 + F ℓ (h ℓ−1 ) , α = (2L) 1/4 , withαranging from 2.38 atL=16 to 2.83 atL=32. Following the original recipe (Wang et al., 2024), the block normalisation is a mean-centring LayerNorm (without bias) rather than RMSNorm, while the head normalisation remains RMSNorm. Initialisation follows DeepNet (Wang et al., 2024): the value and output projections of attention and all three feed-forward matrices are scaled byβ= (8L) −1/4 (std 0.02,β ≈0.0054 atL=24); query, key, embedding, and head weights keep std 0.02. DeepNorm is trained with optimal learning rate 1× 10 −3 . KEEL (Chen & Wei, 2026). KEEL combines pre- and post-normalisation with a large residual gain, h ℓ = RMSNorm post αh ℓ−1 + F ℓ (RMSNorm pre (h ℓ−1 )) , α = total number of sub-layers, whereαcounts attention and feed-forward sub-layers separately. The first block is treated specially: its attention sub-layer is a plain Pre-LN update (no post-norm, no residual up-scaling), and its feed-forward sub-layer is both pre-normarlised and post-normalised without residual up-scaling. KEEL addsdparameters in the first block and 2d in every other block. AttnRes (Full) (Team et al., 2026b). Every sub-layer replaces the residual sum by a softmax-weighted mixture over the outputs of all preceding sub-layers and the embedding, i.e. the sources are individual sub-layer outputs rather than accumulated hidden states. The attention sub-layer of blockℓmixes 2ℓ+1 sources, the feed-forward sub-layer 2ℓ+2, and the output head 2L+1 (49 atL=24). Each sub-layer owns a learned query vector w∈R d and a per-query RMSNorm gain applied to the keys, and scores are w ⊤ rmsnorm(v i ) with softmax temperature 1 (no 1/ √ dscaling). The mixture is RMS-normalised with the consuming sub-layer’s pre-norm weights inside the fused kernel, so the standard pre-normalisation is folded into the mixing step, and the Language modeling (LM) head likewise consumes a mixture normalised with the LM head norm. All query vectors, including the LM head query, are zero-initialised, so training starts from uniform averaging over sources. The first block’s attention reads the embedding directly and has no query. Query vectors and key gains are subject to the global weight decay. AttnRes adds 2d parameters per sub-layer plus 2d for the head. AttnRes (Block) (Team et al., 2026b). Block AttnRes uses the same block, kernel, initialisation, and parameter count as Full AttnRes and differs only in the depth mixing schedule. Sub-layer outputs are accumulated into a running sum like Pre-LN. At everyb-th sub-layer (block sizeb= (2L)/8: 4,5,6,7,8 sub-layers for L= 16,20,24,28,32) the running sum is frozen as a new source and reset. Each mixture is taken over the frozen block sums and the current running sum, so the network is always partitioned into 2L/b= 8 5 blocks and the head mixes the embedding with eight block outputs. For evenbthe boundaries fall on attention sub-layers only; for odd b (L=20, 28) they alternate between attention and feed-forward sub-layers. 5 We follow the same recipe in AttnRes report (Team et al., 2026b) and Kimi K3 report (Team et al., 2026a) for choosing block size, i.e., fixing the number of blocks at approximately 8. 19 Hyper-Connections (HC) (Zhu et al., 2025). We usen= 4 residual streams, with an HC connector replacing the residual addition around each pre-normalised attention and FFN branch. Routing combines learned static coefficients with input-dependenttanhcorrections projected from each affine-free RMS-normalised stream. Routing parameters use BF16. Dynamic projections are zero-initialised, with two learned scalar gains initialised to 0.01. The initial residual map is the identity. Zero-indexed sub-layerkreads only streamk mod nand writes its output to all streams, with unit read and write weights. Attention output and FFN down-projections are additionally scaled by 1/ √ n= 1/2 at initialisation. Static routing weights receive no weight decay, while dynamic projections use 0.1. Embeddings are replicated across streams, which are summed before the final RMSNorm and LM head. Manifold-Constrained Hyper-Connections (mHC) (Xie et al., 2025). We use Liger Kernel 0.8.0 6 withn= 4 streams and 20 Sinkhorn–Knopp iterations. Each attention and FFN connector replaces the standard residual addition around the pre-normalised branch. A linear projection of the affine-free RMS-normalised, concatenated streams produces input-dependent routing logits: read weights use sigmoid, write weights use twice sigmoid, and residual maps use Sinkhorn normalisation, withouttanh. RMS and Sinkhorn epsilons are 10 −6 . The read-weight epsilon is zero. Projection weights and branch computation use BF16, while routing biases, learned gains, and coefficient normalisation use FP32. We initialise the dynamic projection Φ to zero and its three scalar gains to 0.01. At zero-indexed sub-layerk, read biases are +8 for streamk mod nand−8 otherwise; write biases are zero; residual biases are 0 on the diagonal and−8 off-diagonal (the 0/-8 GAP-8 init mentioned in Figure 12). This gives approximately one-hot reads, unit writes, and near-identity residual mixing, overriding Liger’s default initialisation. Attention output and FFN down-projection weights are scaled by 1/ √ n= 1/2 after the base initialisation with std 0.02. Biases and scalar gains receive no weight decay, while Φ uses 0.1. Embeddings are replicated across streams, which are averaged before the final RMSNorm and LM head. Mixture-of-Depth Attention (MoDA) (Zhu et al., 2026). We use Pre-LN MoDA with the authors’v17Triton kernel, patched for our GPU types and head dimensions. RMSNorm precedes both attention and the FFN. Attention uses one softmax over causal sequence KV and earlier-layer, same-token depth KV, with scale 1/ √ d head . Attention KV projections are reused; FFN branches add KV projections, except in the final block where no later layer reads them. The baseline’s RoPE and additive residuals are retained. Although the original study favours Post-LN (Zhu et al., 2026), it optimises poorly under our shared recipe. As shown in Table 5, atL= 24 and learning rate 2×10 −3 , Post-LN MoDA ends at validation CE 3.842 versus 2.764 for Pre-LN MoDA. Reducing the learning rate to 1×10 −3 gives the best tested Post-LN MoDA result, 2.912, still above Pre-LN MoDA. We therefore retain Pre-LN MoDA for the shape comparison. Table 5. Validation loss for MoDA normalization variants (L = 24 model). Peak LR5× 10 −3 2× 10 −3 1× 10 −3 5× 10 −4 2× 10 −4 Pre-LN MoDA2.7752.7642.7732.864– Post-LN MoDA4.8933.8422.9122.9333.443 Table 6. Total parameters and overhead relative to Pre-LN at L = 24 (d model =1024). ArchitectureParameters∆ vs. Pre-LN Pre-LN / LNS / DeepNorm405.41M— Sandwich-LN405.46M+0.049M KEEL405.45M+0.048M AttnRes (Full / Block)405.51M+0.098M HC (n=4)405.70M+0.296M mHC (n=4)410.13M+4.72M MoDA453.64M+48.2M 6 https://github.com/linkedin/Liger-Kernel/tree/v0.8.0 20 B. Complete Model Configurations Tables 3, 7, 8 and 9 summarize the complete model configurations used in our experiments. For the 400M benchmark, we use (L,d) = (24,1024) as the reference configuration. This is a convenient and representative Transformer shape:d= 1024 is a standard hidden dimension, gives a head dimension of 64 with 16 attention heads, and results in a model of approximately 405M parameters under our backbone. We construct the remaining shapes around this reference by trading width for depth, extending from the shallow–wide (16,1216) configuration to the substantially deeper (70,640) configuration. The reference shape also serves as the anchor for our iso-backbone study, making the two experimental settings directly comparable. For the 1.6B experiments, we choose (L,d) = (28, 2048) as the reference shape, matching the same Qwen3-1.7B (Yang et al., 2025) shape to define our larger-scale backbone. In particular, it usesd f = 6144, 16 query heads, 8 key–value heads, and head dimension 128. We then construct two deeper and narrower variants, (40,1728) and (54,1504), while keeping the total model size close to 1.6B. This gives three representative aspect ratios, 73.1, 43.2, and 27.9, covering a comparable shallow-to-deep range to the main 400M study. The multi-scale 200M–500M suite is designed to preserve approximately the same three width–depth regimes across model scales. We use aspect ratios near 28, 43, and 76, corresponding to the deep, intermediate, and shallow reference points in the 400M sweep. For example, at 400M these are exactly (32,896), (24,1024), and (16,1216). At other parameter scales, we adjustLanddto reproduce these aspect ratios as closely as possible while keeping the total parameter count matched within each scale. This construction allows us to test whether the preferred width–depth allocation changes with model size rather than with a particular absolute choice of width or depth. Table 7. Iso-backbone 300M model configurations Layers Hidden Intermediate Heads Head Dim. Aspect Ratio Backbone Size Backbone Diff. Total Size 1612483328167878.00299M−1.11%425M 2410242736166442.67302M0.00%405M 328962400165628.00309M+2.26%399M 427682048164818.29297M−1.69%375M 70608163216388.69312M+3.15%373M 84544145616346.48299M−0.99%354M Table 8. Model configurations for the depth scaling ladder across 200M–500M scales. LayersHiddenIntermediateHeadsHead Dim.Aspect RatioBackbone SizeTotal SizeTotal Diff. 200M – 4B training tokens 246721792164228.00130M198M −3.42% 187682048164842.67127M205M0.00% 128962400165674.67116M206M+0.69% 300M – 6B training tokens 288002144165028.57216M296M+1.09% 218962400165642.67203M293M0.00% 1410562816166675.43187M294M+0.18% 400M – 8B training tokens 328962400165628.00309M399M −1.49% 2410242736166442.67302M405M0.00% 1612163248167676.00284M406M+0.28% 500M – 10B training tokens 349922656166229.18403M502M −0.42% 2611202992167043.08392M504M0.00% 1713443584168479.06368M504M −0.16% 21 Table 9. 1.6B model configurations across different width–depth aspect ratios. LayersHiddenIntermediateQ/KV HeadsHead Dim.Aspect RatioBackbone SizeTotal SizeTotal Diff. 282048614416 / 812873.141.409B1.615B0.00% 401728518416 / 810843.201.433B1.607B −0.51% 541504451216 / 89427.851.466B1.617B+0.11% 4.30 4.36 (a) Pre-LN 1e-32e-35e-3 Learning rate 2.76 2.79 2.82 5e-31e-22e-2 Learning rate 2.736 2.744 2.752 2.760 (b) LNS 1e-32e-35e-3 Learning rate 2.76 2.78 2.80 2.82 (c) Sandwich-LN 7.56 7.64 (d) DeepNorm 5e-41e-32e-3 Learning rate 2.80 2.86 2.92 1e-32e-35e-3 Learning rate 2.75 2.80 2.85 2.90 (e) KEEL 1e-32e-35e-3 Learning rate 2.72 2.74 2.76 2.78 2.80 2.82 (f) MoDA 4.00 4.20 (g) HC 1e-32e-35e-3 Learning rate 2.70 2.72 2.74 1e-32e-35e-3 Learning rate 2.72 2.76 2.80 (h) mHC 3.00 3.20 3.40 (i) Full AttnRes 1e-32e-35e-3 Learning rate 2.72 2.74 2.76 Validation loss ↓ L16L20L24L28L32Best measured LR Figure 12. Learn rate sweep over the architectures. We choose the same optimal learning rate for Block AttnRes as for Full AttnRes. For mHC, we used Legacy near-uniform init for the learning rate sweep, which differs very slightly from the results reported in the main text using 0/-8 gap-8 init. C. Learning Rates & Their Impact Figure 12 presents the learning-rate sweeps for the architectures evaluated in DepthBench across different model shapes. These results are used to select the architecture-specific learning rates adopted in our main results. Figure 13 shows the analyses in Section 4.1 can be affected by the choice of learning rate. While 2×10 −3 is optimal for most architectures, LNS is substantially more sensitive and achieves its best performance at a much larger learning rate of 1× 10 −2 , which correspondingly affects its layer-wise analysis. 22 06121824 Layer index ℓ 1 6 12 18 24 Subsequent n -th layer 06121824 Layer index ℓ 06121824 Layer index ℓ 06121824 Layer index ℓ 06121824 Layer index ℓ 1 6 12 18 24 Subsequent n -th layer 06121824 Layer index ℓ 06121824 Layer index ℓ 06121824 Layer index ℓ 0.00 0.25 0.50 Angular distance Pre-LN LNS η= 5 × 10 −4 η= 1 × 10 −3 η= 2 × 10 −3 η= 5 × 10 −3 η= 1 × 10 −2 Angular distance vs. learning rate · L24 Figure 13. The analyses in Section 4.1 can be affected by the choice of learning rate. D. FLOP Accounting We report forward pass FLOPs using the convention that one multiply–add counts as two FLOPs. We count the dominant matrix multiplications and vector dot products, but omit secondary operations like element-wise functions or RMSNorm. We follow the notation used throughout the paper:Tis the sequence length,L=n layer is the number of Transformer layers,d=d model is the hidden dimension,d f is the FFN intermediate dimension,Vis the vocabulary size, andmis the number of residual streams for HC and mHC. We usem= 4 in all experiments. Pre-LN Transformer. For the baseline Transformer, we count the four attention projectionsQ,K,V,O, the three SwiGLU projections, the causal attention matrix products, and the LM head. The resulting cost is F Pre-LN = L 8Td 2 + 6Tdd f + 2T 2 d + F head , where F head = ( 2TdV, for logits at all positions, 2dV,for final position logits. The 2T 2 dterm accounts for bothQK ⊤ andPVunder causal attention. Each product would cost 2T 2 dif evaluated densely, but we follow the convention that a causal kernel evaluates approximately half of theT 2 query–key pairs and that the two products together cost approximately 2T 2 d. A dense implementation that evaluates the full attention matrix would instead require approximately 4T 2 d. HC. HC replaces the residual addition around each attention and FFN sublayer with a hyper-connection, giving 2Lconnectors. We count the dynamic routing projections and the read, residual mixing, and write operations, while omitting normalization and elementwise operations. The additional cost is ∆F HC = 2L [2Tmd(m + 1) + 2Tmd + 2Tm(m + 1)d + 2Tmd]. For m = 4, ∆F HC = 192TdL. mHC. mHC likewise introduces one connector around each attention and FFN sub-layer. Its dominant additional cost is the projection of the flattenedmd-dimensional residual state to them 2 + 2mrouting logits, followed by the read,m×mresidual-stream mixing, and write operations. RMSNorm, sigmoid evaluations, and Sinkhorn–Knopp 23 normalization (which is done entirely on-chip over very small matrices) are omitted under our FLOP convention. Thus, ∆F mHC = 2L 2Tmd(m 2 + 2m) + 2Tmd + 2Tm 2 d + 2Tmd . For m = 4, ∆F mHC = 480TdL. The higher FLOP cost of mHC relative to HC is therefore dominated by itsmd→ m 2 + 2mrouting projection rather than by the Sinkhorn iterations. Full AttnRes. Full AttnRes scores each preceding residual source using the query–source dot productw ⊤ v i and forms a weighted sum of the source vectors. For each source, these two operations cost 2TdFLOPs each, giving 4TdFLOPs per source. Source normalization and the softmax over depth are omitted. The first attention sub-layer reads the embedding directly and requires no depth-mixing operation. The remaining sub-layers and the LM-head readout mix 2, 3,..., 2L + 1 sources, respectively. Hence, ∆F Full = 4Td 2L+1 X ℓ=2 ℓ = 4TdL(2L + 3), and we further approximate for simplicity ∆F Full ≈ 8TdL 2 . Block AttnRes. Block AttnRes uses the same depth-mixing operation but accumulates sub-layer outputs within blocks. Following our experimental setup, the 2Lattention and FFN sub-layers are divided into eight blocks (more blocks are possible, but here we use 8 following their recommendations) with block size b = 2L 8 . Indexing the sub-layers byk= 0,...,2L−1, the first attention sub-layer (k= 0) reads the embedding directly and requires no depth-mixing operation. Fork= 1,...,2L−1, the mixture contains the embedding, the current running block sum, and one additional frozen source for every completed block, giving 2 + k b sources. The LM head mixes the embedding with the eight completed block outputs, giving nine sources. The additional cost is therefore ∆F Block = 4Td " 2L−1 X k=1 2 + k b + 9 # , b = 2L 8 . E.Complete Results of Causal Score, Permutation Score and Angular Distance This section reports the complete layer-pair heatmaps in Section 4.1 for all 10 architectures atL= 16, 24, and 32. Causal score: entry (s,ℓ) measures the effect of skipping layer s on the update of a later layer ℓ. Permutation score: entry (ℓ 1 ,ℓ 2 ) measures the effect of swapping layersℓ 1 < ℓ 2 ; blue means the two layers are interchangeable, red that their order matters. Angular distance: entry (ℓ,n) is the distance between the residual state after layerℓand the statenlayers later (ℓ = 0 is the embedding). 24 0481215 Affected layer ℓ 0 4 8 12 15 Skipped layer s Pre-LN 0481215 Affected layer ℓ Sandwich-LN 0481215 Affected layer ℓ LNS 0481215 Affected layer ℓ DeepNorm 0481215 Affected layer ℓ KEEL 0481215 Affected layer ℓ 0 4 8 12 15 Skipped layer s HC 0481215 Affected layer ℓ mHC 0481215 Affected layer ℓ Full AttnRes 0481215 Affected layer ℓ Block AttnRes 0481215 Affected layer ℓ MoDA 0.0 0.5 1.0 Causal score Causal score · L16 06121823 Affected layer ℓ 0 6 12 18 23 Skipped layer s Pre-LN 06121823 Affected layer ℓ Sandwich-LN 06121823 Affected layer ℓ LNS 06121823 Affected layer ℓ DeepNorm 06121823 Affected layer ℓ KEEL 06121823 Affected layer ℓ 0 6 12 18 23 Skipped layer s HC 06121823 Affected layer ℓ mHC 06121823 Affected layer ℓ Full AttnRes 06121823 Affected layer ℓ Block AttnRes 06121823 Affected layer ℓ MoDA 0.0 0.5 1.0 Causal score Causal score · L24 08162431 Affected layer ℓ 0 8 16 24 31 Skipped layer s Pre-LN 08162431 Affected layer ℓ Sandwich-LN 08162431 Affected layer ℓ LNS 08162431 Affected layer ℓ DeepNorm 08162431 Affected layer ℓ KEEL 08162431 Affected layer ℓ 0 8 16 24 31 Skipped layer s HC 08162431 Affected layer ℓ mHC 08162431 Affected layer ℓ Full AttnRes 08162431 Affected layer ℓ Block AttnRes 08162431 Affected layer ℓ MoDA 0.0 0.5 1.0 Causal score Causal score · L32 Figure 14. Causal score for all architectures at L = 16, 24, and 32 (top to bottom). 25 0481215 Layer ℓ 2 0 4 8 12 15 Layer ℓ 1 Pre-LN 0481215 Layer ℓ 2 Sandwich-LN 0481215 Layer ℓ 2 LNS 0481215 Layer ℓ 2 DeepNorm 0481215 Layer ℓ 2 KEEL 0481215 Layer ℓ 2 0 4 8 12 15 Layer ℓ 1 HC 0481215 Layer ℓ 2 mHC 0481215 Layer ℓ 2 Full AttnRes 0481215 Layer ℓ 2 Block AttnRes 0481215 Layer ℓ 2 MoDA 0.0 0.5 1.0 Permutation score Permutation score · L16 06121823 Layer ℓ 2 0 6 12 18 23 Layer ℓ 1 Pre-LN 06121823 Layer ℓ 2 Sandwich-LN 06121823 Layer ℓ 2 LNS 06121823 Layer ℓ 2 DeepNorm 06121823 Layer ℓ 2 KEEL 06121823 Layer ℓ 2 0 6 12 18 23 Layer ℓ 1 HC 06121823 Layer ℓ 2 mHC 06121823 Layer ℓ 2 Full AttnRes 06121823 Layer ℓ 2 Block AttnRes 06121823 Layer ℓ 2 MoDA 0.0 0.5 1.0 Permutation score Permutation score · L24 08162431 Layer ℓ 2 0 8 16 24 31 Layer ℓ 1 Pre-LN 08162431 Layer ℓ 2 Sandwich-LN 08162431 Layer ℓ 2 LNS 08162431 Layer ℓ 2 DeepNorm 08162431 Layer ℓ 2 KEEL 08162431 Layer ℓ 2 0 8 16 24 31 Layer ℓ 1 HC 08162431 Layer ℓ 2 mHC 08162431 Layer ℓ 2 Full AttnRes 08162431 Layer ℓ 2 Block AttnRes 08162431 Layer ℓ 2 MoDA 0.0 0.5 1.0 Permutation score Permutation score · L32 Figure 15. Permutation score for all architectures at L = 16, 24, and 32 (top to bottom). 26 0481216 Layer index ℓ 1 4 8 12 16 Subsequent n -th layer Pre-LN 0481216 Layer index ℓ Sandwich-LN 0481216 Layer index ℓ LNS 0481216 Layer index ℓ DeepNorm 0481216 Layer index ℓ KEEL 0481216 Layer index ℓ 1 4 8 12 16 Subsequent n -th layer HC 0481216 Layer index ℓ mHC 0481216 Layer index ℓ Full AttnRes 0481216 Layer index ℓ Block AttnRes 0481216 Layer index ℓ MoDA 0.00 0.25 0.50 Angular distance Angular distance · L16 06121824 Layer index ℓ 1 6 12 18 24 Subsequent n -th layer Pre-LN 06121824 Layer index ℓ Sandwich-LN 06121824 Layer index ℓ LNS 06121824 Layer index ℓ DeepNorm 06121824 Layer index ℓ KEEL 06121824 Layer index ℓ 1 6 12 18 24 Subsequent n -th layer HC 06121824 Layer index ℓ mHC 06121824 Layer index ℓ Full AttnRes 06121824 Layer index ℓ Block AttnRes 06121824 Layer index ℓ MoDA 0.00 0.25 0.50 Angular distance Angular distance · L24 08162432 Layer index ℓ 1 8 16 24 32 Subsequent n -th layer Pre-LN 08162432 Layer index ℓ Sandwich-LN 08162432 Layer index ℓ LNS 08162432 Layer index ℓ DeepNorm 08162432 Layer index ℓ KEEL 08162432 Layer index ℓ 1 8 16 24 32 Subsequent n -th layer HC 08162432 Layer index ℓ mHC 08162432 Layer index ℓ Full AttnRes 08162432 Layer index ℓ Block AttnRes 08162432 Layer index ℓ MoDA 0.00 0.25 0.50 Angular distance Angular distance · L32 Figure 16. Angular distance for all architectures at L = 16, 24, and 32 (top to bottom). 27