Paper deep dive
Sphere Retraction Normalizations
Jie Zhang, Cheng-Fang Su, Yi-Jui Huang, Min-Te Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/5/2026, 3:58:33 AM
Summary
The paper introduces Sphere Retraction Normalizations (SpheretNorms) to stabilize deep neural network training by confining hidden states to a hypersphere. It generalizes Geodesic Normalization (GeoNorm) by replacing the Riemannian exponential map with a family of retraction maps, specifically metric projection and Cayley retractions. These are unified into a one-parameter family, p-SpheretNorm, which allows for tunable rotation angle saturation. The proposed methods, particularly Proj-SpheretNorm, outperform existing lightweight deep connection schemes like Pre-LN and GeoNorm on nanoGPT models.
Entities (10)
Relation Signals (8)
GeoNorm → uses → Exponential Map
confidence 95% · GeoNorm recasts them on a Riemannian manifold... applying the resulting update through the Riemannian exponential map.
Proj-SpheretNorm → uses → Metric Projection Retraction
confidence 95% · Instantiating it with the metric projection retraction... yields Proj-SpheretNorm
Cay-SpheretNorm → uses → Cayley Retraction
confidence 95% · Instantiating it with the ... Cayley retraction yields ... Cay-SpheretNorm
GeoNorm → constrains → Hypersphere
confidence 90% · Every hidden state thus keeps a constant l2-norm, confining the residual stream to a hypersphere.
p-SpheretNorm → generalizes → Cay-SpheretNorm
confidence 90% · Both prove to be members of a one-parameter family of angular retractions, p-SpheretNorm... while the identity map and GeoNorm arise only as limits at either end.
p-SpheretNorm → generalizes → Proj-SpheretNorm
confidence 90% · Both prove to be members of a one-parameter family of angular retractions, p-SpheretNorm... The two methods above are recovered exactly at p = 1
p-SpheretNorm → generalizes → GeoNorm
confidence 85% · while the identity map and GeoNorm arise only as limits at either end.
Proj-SpheretNorm → outperforms →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant $\ell_{2}$-norm, confining the residual stream to a hypersphere. The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework. Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions, $p$-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at $p = 1$ and $p = 2$, while the identity map and GeoNorm arise only as limits at either end. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite $p$, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum.
Tags
Links
- Source: https://arxiv.org/abs/2608.02668v1
- Canonical: https://arxiv.org/abs/2608.02668v1
Trouble viewing inline? Open PDF directly →
Full Text
123,340 characters extracted from source content.
Expand or collapse full text
1]Department of Computer Science and Information Engineering, National Central University, Taiwan 2]Department of Applied Mathematics, National Yang Ming Chiao Tung University, Taiwan 3]Department of Mathematics, National Central University, Taiwan [*]Equal contribution Sphere Retraction Normalizations Jie Zhang Cheng-Fang Su Yi-Jui Huang Min-Te Sun [ [ [ hazdzz@g.ncu.edu.tw scf1204@nycu.edu.tw (August 2, 2026) Abstract Residual connections are the de facto mechanism for training deep neural networks stably. Geodesic Normalization (GeoNorm) recasts them on a Riemannian manifold, orthogonalizing each layer output against the current hidden state and applying the resulting update through the Riemannian exponential map. Every hidden state thus keeps a constant ℓ2 _2-norm, confining the residual stream to a hypersphere. The exponential map, however, is only one member of a broad family of retraction maps. We show that on the hypersphere this entire family collapses to a single scalar design choice. What distinguishes one retraction from another is only how the magnitude of an update is converted into a rotation angle within the plane spanned by the hidden state and the update. This view places Euclidean residual connections and GeoNorm in one framework. Instantiating it with the metric projection retraction and the Cayley retraction yields Proj-SpheretNorm and Cay-SpheretNorm, which are exactly norm-preserving yet require only algebraic operations. Both prove to be members of a one-parameter family of angular retractions, p-SpheretNorm, whose rotation angle saturates rather than growing without bound. The two methods above are recovered exactly at p=1p=1 and p=2p=2, while the identity map and GeoNorm arise only as limits at either end. On nanoGPT, all three methods outperform existing lightweight deep connection schemes, and the best validation loss is attained at finite p, indicating that the exponential map is not the preferred retraction for spherical residual streams but merely one end of a spectrum. †preprint: Preprint [Code] ://github.com/hazdzz/deep_connection [Correspondence]Jie Zhang (), Correspondence: Cheng-Fang Su () “Geometry is not true, it is advantageous.” — Henri Poincaré 1 Introduction Training neural networks in a deep and stable manner remains an open problem. Prior research (Li et al., 2025; Kim et al., 2025; Chen and Wei, 2026) primarily tackles this challenge by combining skip connections (He et al., 2016b, a), normalization methods (Ba et al., 2016; Zhang and Sennrich, 2019), and gating mechanisms (Hochreiter and Schmidhuber, 1997; Cho et al., 2014). We refer to this class of methodologies collectively as deep connections. In the Transformer architecture (Vaswani et al., 2017), Post-LayerNorm (Post-LN) (Vaswani et al., 2017) and Pre-LayerNorm (Pre-LN) (Wang et al., 2019; Xiong et al., 2020) have emerged as the two most common choices for the placement of deep connections. However, both Pre-LN and Post-LN exhibit inherent limitations. Transformers with Pre-LN suffer from the curse of depth (Sun et al., 2025), where the contribution of deeper layers progressively diminishes. Moreover, the unbounded residual stream in Pre-LN allows massive activations to accumulate across layers, which has been linked to more pronounced attention sinks — a phenomenon in which attention scores concentrate excessively on the first token (Xiao et al., 2024; Sun et al., 2024; Gu et al., 2025). By contrast, Transformers with Post-LN, on the other hand, suffer from large gradients near the output layer at initialization (Xiong et al., 2020), which destabilizes training and necessitates a carefully tuned learning-rate warm-up schedule that slows down optimization and adds hyperparameter tuning. (a) The whole process of SpheretNorm. ℱl(⋅)F_l(·) represents a (non-)linear mapping, the GS process stands for the Gram–Schmidt process, and ℛl(⋅)R_l(·) denotes the retraction map. (b) Illustration of the geometric meaning of SpheretNorm. The retraction map ℛl(⋅)R_l(·) is defined on the hypersphere of radius ∥l∥2 h_l _2. Figure 1: Overview of SpheretNorm (including GeoNorm, Proj-SpheretNorm, Cay-SpheretNorm, and p-SpheretNorm). (a) The overall computational pipeline, and (b) its underlying geometric interpretation. A recent line of work (Li et al., 2025; Kim et al., 2025; Chen and Wei, 2026) seeks to reconcile the Post-LN and Pre-LN paradigms within Euclidean space. Zheng et al. (2026) pursue a similar goal but take a distinct route: their Geodesic Normalization (GeoNorm) accomplishes the residual connection on a Riemannian manifold. Let l h_l denote the l-th hidden state, and let ℱl(⋅)F_l(·) denote the corresponding (non-)linear mapping. Specifically, GeoNorm first orthogonalizes ℱl(l)F_l( h_l) against l h_l via a Gram-Schmidt process to obtain an intermediate tensor l z_l, and then applies an exponential map at l h_l along l z_l to produce l+1 h_l+1. By keeping the ℓ2 _2-norm of every hidden state constant across layers, GeoNorm in effect confines the hidden states to a hypersphere, which stabilizes forward propagation. While GeoNorm offers a compelling geometric perspective, its update is tied to one specific choice: the exponential map. From the viewpoint of Riemannian geometry, however, the exponential map is merely one member of a broad family of retraction maps (Absil et al., 2007; Absil and Malick, 2012). Thus, a natural question arises: which retraction should a deep connection use on the hypersphere, and how does this choice shape the behavior of a deep neural network? Rather than proposing another way to unify Post-LN and Pre-LN into a single normalization mechanism, we revisit retraction maps on the hypersphere and propose three Sphere Retraction Normalizations (SpheretNorms): Proj-SpheretNorm, Cay-SpheretNorm, and -SpheretNorm. We first unify residual connection and GeoNorm within a single framework, under which GeoNorm can be interpreted as a Riemannian residual connection on the hypersphere. Building upon this perspective, we further exploit the metric projection retraction and the Cayley retraction on the hypersphere to derive Proj-SpheretNorm and Cay-SpheretNorm, respectively. We then observe that both Proj-SpheretNorm and Cay-SpheretNorm can be subsumed into a unified formulation based on the exponential map with different effective angles. We refer to this generalized formulation as p-SpheretNorm with a positive parameter p, which recovers Proj-SpheretNorm at p=1p=1 and Cay-SpheretNorm at p=2p=2. Moreover, as p→0+p→ 0^+, p-SpheretNorm reduces to the identity mapping, while as p→+∞p→+∞ with αl∈(0,1] _l∈(0,1], it recovers GeoNorm. Through a single tunable parameter, p-SpheretNorm provides a principled and flexible family of normalization layers that bridges existing methods. We evaluate three types of SpheretNorm using nanoGPT (Karpathy, 2022) as the backbone on both pretraining and downstream tasks. Proj-SpheretNorm achieves the best results, attaining the lowest pretraining and validation losses on the medium- and large-size nanoGPT models. Results on the downstream tasks also show that Proj-SpheretNorm achieves high accuracy. Taken together, these results indicate that Proj-SpheretNorm is the strongest of the three SpheretNorm variants across both pretraining and downstream evaluations. Contributions. In conclusion, our key contributions are summarized as follows: 1. We unify residual connections and GeoNorm as retraction-based updates, showing that GeoNorm amounts to a Riemannian residual connection driven by the exponential map on the hypersphere. 2. We instantiate two alternative retractions on the hypersphere, the metric projection retraction and the Cayley retraction, which yield Proj-SpheretNorm and Cay-SpheretNorm. 3. We generalize both instances into p-SpheretNorm, a single-parameter family that reduces to the identity mapping as p→0+p→ 0^+, recovers Proj-SpheretNorm at p=1p=1 and Cay-SpheretNorm at p=2p=2, and recovers GeoNorm as p→+∞p→+∞. 4. On nanoGPT at up to 36 layers and 6.556.55 B tokens, SpheretNorm attains the lowest pre-training and validation losses at 24 and 36 layers and the highest average zero-shot accuracy on both OpenWebText and FineWeb-Edu, outperforming Pre-LN, Pre-DyT, Peri-LN, Keel, and GeoNorm, while converging in every setting in which Pre-LN or Keel diverges. 2 Related Work Normalization Methods. LayerNorm (Ba et al., 2016) has served as the foundation of Transformers (Vaswani et al., 2017) since its inception. Later, RMSNorm (Zhang and Sennrich, 2019), a simplified variant of LayerNorm that drops the re-centering operation, became the de facto standard normalization. A closely related and contemporaneous normalization method, ScaleNorm (Nguyen and Salazar, 2019), differs mainly in that it normalizes by the ℓ2 _2-norm rather than the root mean square, and replaces the per-dimension gain vector with a single tied scalar, making it more parameter-efficient. Normalization-Free Normalization Functions. A noteworthy line of work replaces LayerNorm with a zero-centered, bounded, center-sensitive, monotonic element-wise non-linearity. Zhu et al. (2025) propose the Dynamic Tanh (DyT) function, Stollenwerk (2026) introduce the Dynamic Inverse Square Root Unit (DyISRU) function, and Chen et al. (2026) develop the Dynamic erf (Derf) function. By eliminating the reduction along the hidden dimension, these activation functions remove the synchronization bottleneck of LayerNorm and can be fused with the preceding linear layer, substantially reducing overhead while preserving the squashing behavior of LayerNorm. Skip Connection, and its Variants. ResNet (He et al., 2016a, b) ushered in the era of very deep neural networks. Later, Bachlechner et al. (2021) proposed ReZero, a residual connection with a learnable step size αl _l initialized at zero. Touvron et al. (2021) further enhanced ReZero by using a learnable vector instead of a scalar. Recently, Zheng et al. (2026) proposed GeoNorm, a Riemannian residual connection that preserves the ℓ2 _2-norm of each hidden state throughout forward propagation. Inspired by the property of GeoNorm that maintains a constant ℓ2 _2-norm across hidden layers, we revisit retraction maps on the hypersphere, and propose SpheretNorms. Post-LN, Pre-LN, and their Variants. With the advent of Transformers, applying a residual connection followed by LayerNorm, known as Post-LN, became standard practice. However, Transformers using Post-LN typically rely on a carefully tuned learning-rate warm-up schedule. In contrast, Pre-LN applies LayerNorm before each sub-layer and adds the residual connection afterward. While more stable, Transformers with Pre-LN suffer from the curse of depth (Sun et al., 2025), where deeper layers contribute far less to learning and representation than earlier ones, effectively widening the network rather than deepening it. LayerNorm Scaling (LNS) (Sun et al., 2025) attenuates the normalized hidden state by a depth-dependent factor before each sub-layer, mitigating the curse of depth. In addition to the conventional Pre-LN and Post-LN designs, Peri-LN (Kim et al., 2025) and Keel (Chen and Wei, 2026) apply LayerNorm twice to reconcile the two paradigms. Collectively, these works explore different placements and scalings of LayerNorm within the residual stream to stabilize and deepen Transformers, yet they all operate within the Euclidean geometry of the hidden representations. 3 Background and Preliminary Let lh_l denote the l-th hidden state, with 1h_1 being the input feature (or the output of the embedding layer, for Transformers), and let ℱl(⋅)F_l(·) denote the l-th (non-)linear mapping in Euclidean space, such as multi-head self-attention (Vaswani et al., 2017) or a multi-layer perceptron. Unless otherwise specified, for notational convenience, all bias terms are omitted throughout this work. Moreover, we assume l∈ℝdh_l ^d and l≠h_l 0. For l=1, 2,…,L−1l=1,\,2,\,…,\,L-1, we take two instances of deep connection methods for illustrating. Residual Connection (RC). To facilitate the training of deep CNNs, He et al. (2016a, b) propose ResNet, whose key innovation is the residual connection. It can be formulated as l+1=l+ℱl(l).h_l+1=h_l+F_l(h_l). (1) Geodesic Normalization (GeoNorm). The exponential map in Riemannian geometry is defined as exp()=cos(∥2∥2)+∥2sin(∥2∥2)∥2, _h(v)= ( v _2 h _2)h+ h _2 ( v _2 h _2) v v _2, (2) with ⊤=0h v=0. Building upon this exponential map, Zheng et al. (2026) propose GeoNorm, which is defined as l=(−ll⊤∥l∥2)ℱl(l),βl=αl∥l∥2∥l∥2,l+1=cos(βl)l+∥l∥2sin(βl)l∥l∥2. splitz_l&= (I- h_lh_l h_l ^2 )F_l(h_l), _l= _l z_l _2 h_l _2,\\ h_l+1&= ( _l)h_l+ h_l _2 ( _l) z_l z_l _2. split (3) Here, αl∈(0,1] _l∈(0,1] is a learnable parameter 111The range of αl _l is not explicitly specified in (Zheng et al., 2026). Given that αl _l acts as the step size, we deduce its range from this property.. 4 Proposed Method 4.1 Sphere Retraction Normalizations In this subsection, we begin by casting the standard residual connection and GeoNorm into a common framework that exposes their shared computational structure and distinguishes their geometric components. We single out the retraction map for redesign and consider alternatives to the exponential map. Instantiating the framework with the metric projection and Cayley retractions yields Proj-SpheretNorm and Cay-SpheretNorm, respectively. We then introduce a unified one-parameter construction, termed p-SpheretNorm, which recovers these two methods as exact special cases. 4.1.1 Retraction Maps on the Hypersphere In a deep neural network, the l-th block maps l−1h_l-1 to lh_l, for l=1,…,Ll=1,…,L. Given a manifold ℳM, the tangent-space projection projTℳproj_T_hM, a step size αl _l, and a retraction ℛR_h, a deep connection can be written abstractly as l=ℛl−1[αl⋅projTl−1ℳ(ℱl(l−1))].h_l=R_h_l-1\! [ _l·proj_T_h_l-1M (F_l(h_l-1) ) ]. (4) Within this framework, RC and GeoNorm share the same computational skeleton and differ only in how the individual components are instantiated. In particular, GeoNorm corresponds to choosing the hypersphere as the manifold and the exponential map as the retraction, which makes it a Riemannian residual connection. Table 1 contrasts the two methods component by component. Table 1: Residual Connection vs. Geodesic Normalization. RC GeoNorm ℳM ℝdR^d rd−1=∈ℝd:∥2=rS^d-1_r=\h ^d: _2=r\ TℳT_hM ℝdR^d ∈ℝd:⊤=0\v ^d:h v=0\ projTℳ()proj_T_hM(u) u (−⊤∥22) (I- hh _2^2 )u αl _l 11 αl∈(0,1] _l∈(0,1] ℛ()R_h(v) +h+v exp() _h(v) Table 1 shows that GeoNorm chooses the retraction map to be the exponential map on the hypersphere. The exponential map, however, is merely one instance of a broader class of retractions (Absil et al., 2007; Absil and Malick, 2012). We replace the exponential map with other retractions keeps the framework intact while enriching the design space. We recall the definition of a retraction below. Definition 1 (Retraction (Absil et al., 2007)). A retraction on a manifold ℳM is a smooth mapping ℛ:Tℳ→ℳ,(,)↦ℛ()R:TM ,\ (h,v) _h(v), such that for every ∈ℳh and every ∈Tℳv∈ T_hM, ℛ()=,dtℛ(t)|t=0=R_h(0)=h,\ \!d \!dt\,R_h(tv) |_t=0=v. Sphere Metric Projection Retraction Normalization. We first choose the metric projection retraction on the hypersphere (Absil et al., 2007), also known as the gnomonic retraction on the hypersphere (Absil and Malick, 2012), as the foundation for proposing a novel normalization layer, which we name the Sphere Metric Projection Retraction Normalization (Proj-SpheretNorm). The definition of the metric projection retraction on the hypersphere is described as blows. Definition 2 (Metric Projection Retraction (Absil et al., 2007)). Let ℳ=rn−1=∈ℝn:‖2=rM=S_r^n-1=\h ^n:\|h\|_2=r\ be the hypersphere of radius r>0r>0, endowed with the Riemannian metric induced from the Euclidean inner product. For each ∈ℳh , the tangent space is Tℳ=∈ℝn:⟨,⟩=0T_hM=\v ^n:\, ,v =0\, and the projective retraction is defined by ℛProj()=r⋅+∥+∥2,∈Tℳ.R^Proj_h(v)=r· h+v h+v _2, ∈ T_hM. (5) Then ℛProjR^Proj is a second-order retraction on ℳM; that is, for every ∈ℳh and every ∈Tℳv∈ T_hM, the curve c(t):=ℛ(t)c(t):=R_h(tv) satisfies c(0)=,c′(0)=,(c′(0))=c(0)=h,\ c (0)=v,\ P_h (c (0) )=0, where :ℝn→TℳP_h:R^n→ T_hM denotes the orthogonal projection onto the tangent space. Based on Eq. (4) and Eq. (5), we have l=(−ll⊤∥l∥2)ℱl(l),l+1=∥l∥2⋅l+αll∥l+αll∥2. splitz_l&= (I- h_lh_l h_l ^2 )F_l(h_l),\\ h_l+1&= _l _2· h_l+ _lz_l h_l+ _lz_l _2. split (6) Sphere Cayley Retraction Normalization. We also consider the Cayley retraction on the hypersphere (Absil et al., 2007), also known as the stereographic retraction on the hypersphere (Absil and Malick, 2012), to propose Sphere Cayley Retraction Normalization (Cay-SpheretNorm). For a skew-symmetric matrix ∈(d)W∈ so(d), where (d)=∈ℝd×d:⊤=− so(d)= \ W ^d× d: W =- W \, the Cayley transform is defined by ()=(−12)−1(+12)C(W)= (I- 12W )^-1 (I+ 12W ). On the hypersphere, a tangent vector determines a unique skew-symmetric generator supported on the two-dimensional subspace spanned by the current point and the tangent vector. This yields the following closed-form expression. Proposition 3. Let ∈rd−1h _r^d-1 and ∈Trd−1v∈ T_hS_r^d-1. There exists a unique matrix =(,)∈(d)W=W(h,v)∈ so(d) satisfying =,range()⊆span,Wh=v,\ range(W) \h,v\, and it is given by (,)=⊤−⊤r2.W(h,v)= vh -hv r^2. (7) Moreover, define ρ=‖2rρ= \|v\|_2r. The corresponding Cayley retraction admits the closed form ℛCay()=((,))=4−ρ24+ρ2+44+ρ2.R_h^Cay(v)=C\! (W(h,v) )h= 4-ρ^24+ρ^2h+ 44+ρ^2v. (8) The proof is provided in Appendix D.1. Let ρ=βlρ= _l, based on Eq. (4) and Eq. (8), the whole process of Cay-SpheretNorm is defined as l=(−ll⊤∥l∥2)ℱl(l),l+1=4−βl24+βl2l+4αl4+βl2l. splitz_l&= (I- h_lh_l h_l ^2 )F_l(h_l),\\ h_l+1&= 4- _l^24+ _l^2h_l+ 4 _l4+ _l^2z_l. split (9) Sphere p-Angular Retraction Normalization. The metric projection and Cayley retractions share the same planar-rotation structure: they update the hidden state within the two-dimensional plane spanned by the current state and the projected tangent direction, differing only in how the normalized step magnitude is converted into a rotation angle. Motivated by this common structure, we introduce a one-parameter family of hyper-spherical retraction normalizations, termed p-SpheretNorm. The parameter p>0p>0 controls the saturation of the finite-step rotation angle while preserving the same local behavior near the origin. Definition 4 (p-SpheretNorm). Let parameter p∈(0,+∞)p∈(0,+∞), and the rotation angle θl=parctan(βlp) _l=p ( _lp ), then the p-SpheretNorm is defined as l=(−ll⊤∥l∥2)ℱl(l),βl=αl∥l∥2∥l∥2,θl=parctan(βlp),l+1=cos(θl)l+∥l∥2sin(θl)l∥l∥2. splitz_l&= (I- h_lh_l h_l ^2 )F_l(h_l),\\ _l&= _l z_l _2 h_l _2, _l=p ( _lp ),\\ h_l+1&= ( _l)h_l+ h_l _2 ( _l) z_l z_l _2. split (10) Proposition 5. For every fixed p>0p>0, the p-angular map underlying p-SpheretNorm defines a smooth second-order retraction on the hypersphere. Moreover, the following identities and limits hold: 1. When p=1p=1, p-SpheretNorm reduces exactly to Proj-SpheretNorm. 2. When p=2p=2, p-SpheretNorm reduces exactly to Cay-SpheretNorm. 3. As p→+∞p→+∞, the p-angular update converges to the exponential-map update used by GeoNorm. 4. As p→0+p→ 0^+, the p-angular update converges point-wise to the identity mapping. To make the local geometry explicit, under the local boundedness conditions stated in Appendix D.4, for every fixed p>0p>0, as αl→0 _l→ 0, l+1=l+αll−αl2‖l‖222rl2l+(αl3).h_l+1=h_l+ _lz_l- _l^2\|z_l\|_2^22r_l^2h_l+O( _l^3). The quadratic term is normal to the hypersphere at lh_l; hence its tangent projection vanishes, which is precisely the second-order retraction condition. In particular, since l=lℱl(l)z_l=P_lF_l(h_l), we obtain l+1=l+αllℱl(l)+(αl2).h_l+1=h_l+ _lP_lF_l(h_l)+O( _l^2). Thus, all members of the p-angular family share the same first-order tangent dynamics and the same second-order normal correction, while the dependence on p first appears at cubic order. The complete proof, including smoothness at l=z_l=0, is provided in Appendix D.4. 4.2 Implementation Details For the p-SpheretNorm, the rotation angle θl _l in each block is always bounded in (−pπ2,pπ2)(- pπ2,\, pπ2). Zheng et al. (2026) suggest that the rotation angle in each block should be restricted to (0,π2)(0,\, π2), and that restricting it to (0,π4](0,\, π4] produces better results than restricting it to (0,π2)(0,\, π2). Our experiments also support this view. We find that the rotation angle in each block should be restricted to (0,π4](0,\, π4] for Cay-SpheretNorm and for p-SpheretNorm with p>1p>1; otherwise, training becomes unstable and may even suffer from exploding gradients. Therefore, for p-SpheretNorm, we choose p=0.5p=0.5, in which case the rotation angle in each block is kept in (−π4,π4)(- π4,\, π4). For all three types of SpheretNorm, we further adopt two additional approaches that ensure training stability with fewer loss spikes. First, we adopt a decay method similar to the one in (Zheng et al., 2026): we let αl=λl⋅Softplus(al) _l= _l·Softplus(a_l), where λl∈(0, 1) _l∈(0,\,1) is a decay factor and ala_l is a learnable parameter. We find that setting λl=1/l _l=1/ l, initializing al=ln(e−1)a_l= (e 1.0mu-1), and constraining Softplus(al)∈(0,1]Softplus(a_l)∈(0,1] can ensure training stability with fewer loss spikes. Second, we apply a modified ScaleNorm before the first (non-)linear mapping, i.e., 1←Softplus(γ)⋅1∥1∥2,h_1 (γ)· h_1 h_1 _2, (11) where γ∈ℝγ is a learnable parameter that is initialized as γ=dγ= d when d≥20 d≥ 20, and as γ=ln(ed−1)γ= (e 1.0mu d-1) otherwise. We use the torch.clamp function to ensure that Softplus(γ)∈[1,d]Softplus(γ)∈[1, d], which we find can accelerate model convergence while stabilizing training. 4.3 Analysis Our analysis of SpheretNorms focuses on four parts. First, we show that p-angular retraction is a second-order retraction. Second, we analyze the Jacobian to establish training stability. Third, we prove that the modified ScaleNorm in the first layer preserves the hypersphere. Fourth, we discuss the trade-off in choosing different decay factors. For analysis, we define rl=‖l‖2,l=I−ll⊤rl2,l=lℱl(l),sl=‖l‖2r_l=\|h_l\|_2,\ P_l=I- h_lh_l r_l^2,\ z_l=P_lF_l(h_l),\ s_l=\|z_l\|_2. p-SpheretNorm based on the second order retraction. Let ηl=slrl,βl=αlηl,θl=parctan(βlp) _l= s_lr_l,\ _l= _l _l,\ _l=p ( _lp ). For every fixed p>0p>0, the p-angular retraction is a second-order retraction. Under the local boundedness conditions stated in Appendix D.4, it also satisfies, as αl→0 _l→ 0, l+1=l+αllℱl(l)+(αl2).h_l+1=h_l+ _lP_lF_l(h_l)+O( _l^2). The proof is provided in Appendix D.4. Thus, all members of the p-angular family share the same first-order tangent dynamics and differ only through their higher-order finite-step angular responses. Jacobian conditioning from signal propagation. Exact norm preservation controls the activation scale, but does not by itself control perturbation or gradient propagation. Define Al=∂ℱl(l)∂l,cl=l⊤ℱl(l)rl2,l=l−clA_l= _l(h_l) _l,\ c_l= h_l F_l(h_l)r_l^2,\ B_l=A_l-c_lI, and let l=∂l+1∂lJ_l= _l+1 _l. For sl>0s_l>0, define Ql=cos(θl)+sin(θl)rlsl(ll⊤−ll⊤)Q_l= ( _l )I+ ( _l )r_ls_l (z_lh_l -h_lz_l ). Proposition 6 (Layer-wise and depth-wise Jacobian bounds). For sl>0s_l>0, the Jacobian of the l-th SpheretNorm block admits the exact decomposition l=l+lJ_l=Q_l+E_l, where ‖l‖2=1,σmin(l)≥|cos(θl)|\|Q_l\|_2=1,\ _ (Q_l)≥ | ( _l ) |. Define Δl=|αl|(2‖l‖2+ηl) _l=| _l| (2\|B_l\|_2+ _l ), ql=|cos(θl)|q_l= | ( _l ) |, and ml=max0,ql−Δlm_l= \0,q_l- _l\. Then ‖l‖2≤Δl\|E_l\|_2≤ _l, and, σmax(l)≤1+Δl,σmin(l)≥ml. _ (J_l)≤ 1+ _l, _ (J_l)≥ m_l. Consequently, for L=∂L∂1=LL−1⋯1G_L= _L _1=J_LJ_L-1·sJ_1, one has σmax(L)≤exp(∑l=1LΔl),σmin(L)≥∏l=1Lml. _ (G_L)≤ ( _l=1^L _l ), _ (G_L)≥ _l=1^Lm_l. The full Jacobian calculation and the smooth sl=0s_l=0 extension are provided in Appendix D.7. The angle θl _l controls the geometric component lQ_l, whereas |αl|‖l‖2| _l|\|B_l\|_2 and |βl|| _l| control the block-induced perturbation. Hence, a cap |θl|≤θmax<π2| _l|≤ _ < π2 keeps lQ_l away from singularity but does not by itself control the full Jacobian; a counterexample is given in Appendix D.8. Moreover, if ml≥m∗>0m_l≥ m_*>0, then ∑l=1LΔl=(1),∑l=1L|θl|2=(1) _l=1^L _l=O(1), _l=1^L | _l |^2=O(1) is a sufficient condition for depth-independent conditioning; see Appendix D.5. First-layer modified ScaleNorm preserves hypersphere. Proposition 7 (Fixed-radius spherical dynamics). Recall Eq. (11). For l=1, 2,…,Ll=1,\,2,\,…,\,L, suppose that l+1h_l+1 is generated from lh_l by a SpheretNorm update. Then ∥l∥2=Softplus(γ) h_l _2=Softplus(γ). Consequently, ∥L∥2≤Softplus(γ) h_L _2 (γ). The proof is provided in Appendix D.3. Thus, the first modified ScaleNorm selects the hypersphere Softplus(γ)d−1=∈ℝd:‖2=Softplus(γ)S^d-1_Softplus(γ)=\h ^d:\|h\|_2=Softplus(γ)\, while all subsequent SpheretNorm blocks preserve this hypersphere exactly. For Proj-SpheretNorm, this relation can be made more explicit. By Proposition 7, ∥l∥2=Softplus(γ) h_l _2=Softplus(γ), and therefore l+1 _l+1 =Softplus(γ)⋅l+αll‖l+αll‖2 =Softplus(γ)· h_l+ _lz_l\|h_l+ _lz_l\|_2 =ΠSoftplus(γ)(l+αllℱl(l)), = _Softplus(γ) (h_l+ _lP_lF_l(h_l) ), where ΠSoftplus(γ)()=Softplus(γ)‖2 _Softplus(γ)(u)=Softplus(γ) u\|u\|_2 is the radial projection onto Softplus(γ)d−1S^d-1_Softplus(γ). Training stability trade-off from different decay factors. The prescribed factor λl _l remains fixed throughout training. Since 0<al≤10<a_l≤ 1, we have 0<αl=λlal≤λl0< _l= _la_l≤ _l. Let κl=2‖l‖2+ηl _l=2\|B_l\|_2+ _l, L=∑l=1LΔl=∑l=1LλlalκlS_L= _l=1^L _l= _l=1^L _la_l _l, and L=∑l=1L|θl|2A_L= _l=1^L | _l |^2. Proposition 8 (Growth under depth schedules). Suppose that there exist constants K,H>0K,\,H>0, independent of l and L, such that κl≤K _l≤ K, alηl≤H\ a_l _l≤ H. Then L≤K∑l=1Lλl,L≤H2∑l=1Lλl2.S_L≤ K _l=1^L _l, _L≤ H^2 _l=1^L _l^2. Consequently, λlLL1(L)(L)1/l(L)(logL)l(logL)(1) array[]c|c _l&S_L&A_L\\ 1&O(L)&O(L)\\ 1/ l&O( L)&O( L)\\ l&O( L)&O(1) array The proof is provided in Appendix D.6. Thus, the decay factor λl=1/l _l=1/l gives the more conservative global worst-case bounds. However, Proposition 8 only provides global upper bounds. A finer dyadic analysis is given in Appendix D.9. At any fixed optimization iteration, under the bounded small-step and non-degeneracy assumptions stated there, the accumulated squared angular motion over each dyadic interval ImI_m has the same asymptotic order as ∑l∈Imλl2 _l∈ I_m _l^2. Hence, the square-root schedule assigns an approximately constant squared angular budget to each logarithmic depth scale, whereas the the decay factor λl=1/l _l=1/l assigns a budget of order m−1m^-1. The empirically preferred the decay factor λl=1/l _l=1/ l should be interpreted as a finite-depth compromise: it provides a tighter Jacobian envelope than the non-decaying schedule while preserving stronger transformations in deeper layers than the decay factor λl=1/l _l=1/l. This mechanism is consistent with the empirical advantage of the decay factor λl=1/l _l=1/ l, but does not imply uniformly better worst-case conditioning or a guaranteed lower loss. Table 2: Loss of trained models. We report training and validation loss at the end of training. To mitigate stochastic fluctuations, training loss is computed as a moving average over the last 200 iterations. Dataset FineWeb-Edu OpenWebText Model Scale S M L S M L Train Val Train Val Train Val Train Val Train Val Train Val Pre-LN 3.0573.057 3.0713.071 2.9752.975 2.9762.976 2.8422.842 2.8662.866 3.0623.062 3.0613.061 4.6974.697 4.7164.716 2.8582.858 2.8792.879 Pre-DyT 3.6263.626 3.6193.619 3.0253.025 3.0193.019 2.8452.845 2.8702.870 3.5393.539 3.5563.556 3.0123.012 3.0353.035 2.8432.843 2.8662.866 Peri-LN 3.0913.091 3.0883.088 2.9522.952 2.9492.949 2.8172.817 2.8442.844 3.0713.071 3.0893.089 2.9502.950 2.9732.973 2.8282.828 2.8502.850 Keel 7.6457.645 7.6417.641 3.0063.006 3.0013.001 2.9412.941 2.9592.959 3.4063.406 3.4223.422 2.9972.997 3.0183.018 2.9452.945 2.9642.964 GeoNorm 3.1763.176 3.1853.185 3.0963.096 3.0883.088 2.9402.940 2.9582.958 3.2093.209 3.2063.206 3.0183.018 3.0393.039 3.0013.001 3.0173.017 0.50.5-SpheretNorm 3.089¯ 3.089 3.086¯ 3.086 2.8312.831 2.8282.828 2.809¯ 2.809 2.833¯ 2.833 3.0673.067 3.0853.085 2.932¯ 2.932 2.955¯ 2.955 2.826¯ 2.826 2.847¯ 2.847 11-SpheretNorm 3.0983.098 3.1093.109 2.8252.825 2.8222.822 2.8082.808 2.8322.832 3.0643.064 3.0823.082 2.9312.931 2.9542.954 2.7952.795 2.8222.822 22-SpheretNorm 3.1193.119 3.1143.114 2.826¯ 2.826 2.824¯ 2.824 2.8092.809 2.8342.834 3.063¯ 3.063 3.081¯ 3.081 2.9312.931 2.955¯ 2.955 2.8412.841 2.8612.861 Table 3: Perplexity of trained models. We report validation perplexity at the end of training. Dataset FineWeb-Edu OpenWebText Scale S M L S M L Pre-LN 21.5621.56 19.6119.61 17.5717.57 21.3521.35 111.70111.70 17.8017.80 Pre-DyT 37.2937.29 20.4820.48 17.6417.64 35.0335.03 20.7920.79 17.5617.56 Peri-LN 21.9421.94 19.0919.09 17.1817.18 21.9621.96 19.5519.55 17.2917.29 Keel 20812081 20.1120.11 19.2719.27 30.6330.63 20.4620.46 19.3719.37 GeoNorm 24.1624.16 21.9321.93 19.2719.27 24.6724.67 20.8820.88 20.4220.42 SpheretNorm (ours) 0.50.5 21.88¯ 21.88 16.9116.91 16.99¯ 16.99 21.8621.86 19.20¯ 19.20 17.24¯ 17.24 11 22.4022.40 16.8116.81 16.9816.98 21.8121.81 19.1819.18 16.8116.81 22 22.5222.52 16.84¯ 16.84 17.0117.01 21.79¯ 21.79 19.20¯ 19.20 17.4817.48 5 Experiments Experimental Settings. To evaluate SpheretNorms against other deep connection methods, including Pre-LN (Wang et al., 2019; Xiong et al., 2020), Pre-DyT (Zhu et al., 2025), Peri-LN (Kim et al., 2025), Keel (Chen and Wei, 2026), and GeoNorm (Zheng et al., 2026), we implement all baselines within the nanoGPT framework (Karpathy, 2022). For pre-training, we adopt two datasets: OpenWebText (Gokaslan and Cohen, 2019) and FineWeb-Edu (Penedo et al., 2024). All models are optimized with the AdamW optimizer (Loshchilov and Hutter, 2019) under a cosine annealing learning rate schedule with a linear warm-up. We employ bfloat16 mixed-precision training together with gradient clipping. We conducted all experiments on a single node equipped with two 56-core Intel Xeon Platinum 8480+ CPUs, 2 TB of RAM, and eight NVIDIA H100 SXM GPUs, each with 80 GB of memory. The code is implemented in PyTorch (Paszke et al., 2019) using its Distributed Data Parallel (DDP) with the NCCL backend. We train each model for 50,000 iterations (about 6.556.55 B tokens in total). We fix the random seed to 1337 in all experiments. Detailed hyperparameter configurations are reported in Appendix B. 5.1 Pre-Training Language Models To assess whether SpheretNorm attains loss improvements comparable to existing residual connection designs, we compare the final training and validation losses in Table 2 and the corresponding validation perplexities in Table 3. At the small scale (12 layers, ∼0.12 \!0.12 B parameters), Pre-LN attains the lowest training and validation loss on both FineWeb-Edu and OpenWebText; the best SpheretNorm variant stays within 0.020.02 nats of it (21.8821.88 vs. 21.5621.56 validation perplexity on FineWeb-Edu, 21.7921.79 vs. 21.3521.35 on OpenWebText) and is on par with Peri-LN. This ordering changes as depth increases. At the medium scale (24 layers, ∼0.35 \!0.35 B), 11-SpheretNorm attains the lowest training and validation loss on both datasets, improving validation perplexity over the strongest baseline, Peri-LN, from 19.0919.09 to 16.8116.81 on FineWeb-Edu and from 19.5519.55 to 19.1819.18 on OpenWebText. The same holds at the large scale (36 layers, ∼0.77 \!0.77 B), where Proj-SpheretNorm reaches 16.9816.98 and 16.8116.81 validation perplexity on the two datasets, ahead of Peri-LN at 17.1817.18 and 17.2917.29. 22-SpheretNorm and 0.50.5-SpheretNorm track 11-SpheretNorm closely in every setting, with all three variants falling within 0.040.04 nats of one another, indicating that the benefit is not sensitive to the particular choice of p. The methods also differ in stability: training destabilizes for Pre-LN on OpenWebText at the medium scale (validation perplexity 111.70111.70) and for Keel on FineWeb-Edu at the small scale (2081.502081.50), whereas all three SpheretNorm variants converge in all six settings. We show the training and validation losses in FineWeb-Edu in Figure 2. 5.2 Downstream Evaluations We evaluate zero-shot performance using the lm-evaluation-harness library (Gao et al., 2021) on eleven benchmarks: ARC-Challenge, ARC-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), SocialIQa (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021), LAMBADA (Paperno et al., 2016), and WikiText-2 (Merity et al., 2017). Tables 4 and 5 report the large-size models trained for 6.55 B tokens on FineWeb-Edu and OpenWebText, respectively. SpheretNorms attain the best score on 11 of the 12 metrics on FineWeb-Edu and 10 of 12 on OpenWebText; the exceptions are SocialIQa on both corpora and HellaSwag on OpenWebText, where the leading baseline is ahead by at most 0.520.52 points. 11-SpheretNorm is the strongest variant overall, though the three stay within about one point of each other on most tasks. The gains are clearest in perplexity, the metric least affected by the noise of zero-shot evaluation at this scale: on FineWeb-Edu every variant improves over every baseline, with 11-SpheretNorm reaching 46.7446.74 on LAMBADA against 59.5559.55 for the best baseline (21.5%21.5\% relative) and 33.7033.70 on WikiText-2; on OpenWebText it reaches 31.1331.13 and 30.7930.79. Two baselines warrant comment. Keel, a Post-LN design targeting gradient flow at extreme depth, brings no benefit in our shallow, short-budget regime and is among the weakest configurations. GeoNorm is the closest comparison, sharing our view of the hidden state as a point on a sphere and of each sublayer output as an update direction, but taking the exact geodesic step via the exponential map where SpheretNorm uses a retraction. The retraction is the better choice here: our best variant improves over GeoNorm on all twelve metrics on both corpora, by margins far larger than those separating the other methods. Since several benchmarks remain near chance at this scale (e.g., WinoGrande, SocialIQa), our conclusion rests on the perplexities and on the consistency across the two corpora rather than on individual sub-point differences. Table 4: Zero-shot evaluation results for large-size models trained on FineWeb-Edu for 6.55 B tokens and evaluated with lm-evaluation-harness. acc means accuracy, acc_n means length-normalized accuracy, and ppl means perplexity. nanoGPT w/ ARC-C ARC-E BoolQ HellaSwag LMB. OBQA PIQA SciQ SocialIQa WinoGrande LMB. Wiki. acc_n ↑ acc_n ↑ acc ↑ acc_n ↑ acc ↑ acc_n ↑ acc_n ↑ acc ↑ acc ↑ acc ↑ ppl ↓ ppl ↓ Pre-LN 26.5426.54 51.8551.85 55.8155.81 36.8136.81 29.1129.11 33.4033.40 65.1865.18 81.2081.20 37.5637.56 52.4952.49 66.6966.69 36.2336.23 Pre-DyT 27.3927.39 51.1451.14 60.3460.34 36.6436.64 29.5229.52 31.4031.40 65.0265.02 82.50¯ 82.50 38.2838.28 50.7550.75 59.5559.55 35.5235.52 Peri-LN 26.6226.62 50.9350.93 53.2753.27 38.09¯ 38.09 29.0129.01 31.2031.20 64.8564.85 82.6082.60 38.13¯ 38.13 52.3352.33 67.0967.09 34.7734.77 Keel 26.4526.45 49.2449.24 61.5961.59 34.3934.39 26.9726.97 30.6030.60 64.2564.25 80.9080.90 38.0838.08 48.6248.62 81.0681.06 39.7339.73 GeoNorm 27.8227.82 48.0248.02 62.05¯ 62.05 34.3534.35 27.9827.98 30.2030.20 63.9363.93 80.5080.50 36.8036.80 51.7851.78 80.6480.64 39.8839.88 0.50.5-SpheretNorm 27.90¯ 27.90 50.9350.93 60.8760.87 37.7737.77 30.8430.84 33.80¯ 33.80 66.27¯ 66.27 81.6081.60 37.6937.69 52.8852.88 50.00¯ 50.00 33.74¯ 33.74 11-SpheretNorm 28.6728.67 52.7852.78 62.7062.70 38.2138.21 32.9332.93 32.6032.60 66.0566.05 82.1082.10 38.13¯ 38.13 51.3651.36 46.7446.74 33.7033.70 22-SpheretNorm 27.90¯ 27.90 52.53¯ 52.53 61.3461.34 38.0738.07 31.19¯ 31.19 34.4034.40 66.7066.70 82.6082.60 37.2637.26 52.72¯ 52.72 53.4553.45 33.8533.85 Table 5: Zero-shot evaluation results for large-size models trained on OpenWebText for 6.55 B tokens and evaluated with lm-evaluation-harness. acc means accuracy, acc_n means length-normalized accuracy, and ppl means perplexity. nanoGPT w/ ARC-C ARC-E BoolQ HellaSwag LMB. OBQA PIQA SciQ SocialIQa WinoGrande LMB. Wiki. acc_n ↑ acc_n ↑ acc ↑ acc_n ↑ acc ↑ acc_n ↑ acc_n ↑ acc ↑ acc ↑ acc ↑ ppl ↓ ppl ↓ Pre-LN 23.3823.38 40.4940.49 59.5459.54 32.7332.73 32.1932.19 26.2026.20 62.8962.89 69.9069.90 38.3338.33 51.5451.54 39.2439.24 33.1933.19 Pre-DyT 25.60¯ 25.60 40.8740.87 54.3154.31 32.8932.89 33.9233.92 26.2026.20 62.1362.13 71.6071.60 38.54¯ 38.54 51.5451.54 33.4033.40 34.1134.11 Peri-LN 25.4325.43 42.0042.00 59.1759.17 33.9533.95 33.7333.73 25.8025.80 63.8763.87 73.2073.20 38.6438.64 50.0450.04 31.4831.48 32.9032.90 Keel 23.2123.21 38.8038.80 57.6157.61 31.1731.17 30.7030.70 27.4027.40 61.9761.97 71.9071.90 38.1338.13 51.7051.70 44.4344.43 38.6438.64 GeoNorm 24.8724.87 37.8437.84 60.55¯ 60.55 29.9729.97 30.0030.00 26.2026.20 61.9261.92 69.3069.30 37.6737.67 51.1451.14 51.9151.91 42.2442.24 0.50.5-SpheretNorm 25.1525.15 42.21¯ 42.21 59.8859.88 33.43¯ 33.43 34.60¯ 34.60 27.4027.40 63.4463.44 73.2073.20 37.9237.92 52.9652.96 31.38¯ 31.38 31.02¯ 31.02 11-SpheretNorm 26.4626.46 42.2642.26 61.0461.04 32.9032.90 34.9934.99 27.80¯ 27.80 63.98¯ 63.98 73.6073.60 37.3637.36 52.85¯ 52.85 31.1331.13 30.7930.79 22-SpheretNorm 25.3225.32 42.0542.05 58.7458.74 32.8632.86 34.3934.39 28.6028.60 64.2064.20 73.40¯ 73.40 38.4838.48 51.1451.14 31.5031.50 31.5531.55 6 Limitations While the proposed SpheretNorms are designed to stabilize the training of a broad range of deep neural networks, two limitations remain. First, the proposed methods are not directly applicable to certain specialized architectures, such as Equivariant Neural Networks (Lim and Nelson, 2022), for which the structural constraints are not naturally compatible with our formulation. Since SpheretNorms rely on ℓ2 _2-norm and the associated projection, they commute with a group representation only when that representation preserves the norm, and they generally break the equivariance condition for non-orthogonal representations such as the GL(n)GL(n) and Lorentz groups. Second, owing to limited research resources, we have not evaluated SpheretNorms on larger-scale deep Transformer models. We leave the extension to such architectures and the verification at larger scales to future work. (a) Training loss. (b) Validation loss. Figure 2: Training and validation losses in the FineWeb-Edu. 7 Conclusion and Future Work In this work, we revisit retraction maps on the hypersphere and introduce three types of SpheretNorms: Proj-SpheretNorm, Cay-SpheretNorm, and p-SpheretNorm. We analyze the three SpheretNorms from both theoretical and practical perspectives. We implement them within the nanoGPT framework and evaluate them on pre-training and downstream tasks. The experimental results show that all three SpheretNorms achieve lower loss and higher accuracy than the baseline methods. At a deeper level, designing deep connection mechanisms is fundamentally a mathematical problem at the intersection of graph theory, optimization, and dynamical systems, raising two intertwined questions: (i) which connection topologies enable non-trivial information propagation across depth while keeping the singular values of the end-to-end Jacobian bounded by constants independent of the depth L; and (i) among such topologies, is there a unifying principle that identifies the optimal trade-off between information propagation and Jacobian conditioning? We leave these questions open and regard them as promising directions for future work. References Absil and Malick (2012) P.-A. Absil and Jérôme Malick. Projection-like Retractions on Matrix Manifolds. SIAM Journal on Optimization, 22(1):135–158, 2012. Absil et al. (2007) P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2007. ISBN 0691132984. Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization, 2016. Bachlechner et al. (2021) Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. ReZero is all you need: fast convergence at large depth. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352–1361. PMLR, 07 2021. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. Thirty-Fourth AAAI Conference on Artificial Intelligence, 34(05):7432–7439, 04 2020. Chen and Wei (2026) Chen Chen and Lai Wei. Post-LayerNorm Is Back: Stable, Expressive, and Deep, 2026. Chen et al. (2026) Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, and Zhuang Liu. Stronger Normalization-Free Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27418–27428, 06 2026. Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder – Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734. Association for Computational Linguistics, 10 2014. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. Association for Computational Linguistics, 06 2019. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018. Gao et al. (2021) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The Language Model Evaluation Harness, 2021. https://github.com/EleutherAI/lm-evaluation-harness. Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. OpenWebText Corpus, 2019. https://Skylion007.github.io/OpenWebTextCorpus. Gu et al. (2025) Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When Attention Sink Emerges in Language Models: An Empirical View. In The Thirteenth International Conference on Learning Representations, 2025. He et al. (2016a) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 630–645. Springer International Publishing, 2016a. He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016b. Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, 11 1997. Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448–456. PMLR, 07 2015. Karpathy (2022) Andrej Karpathy. nanoGPT, 2022. https://github.com/karpathy/nanoGPT. Kim et al. (2025) Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-LN: Revisiting Normalization Layer in the Transformer Architecture. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 30400–30436. PMLR, 07 2025. Li et al. (2025) Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN. In The Thirteenth International Conference on Learning Representations, 2025. Lim and Nelson (2022) Lek-Heng Lim and Bradley J. Nelson. What is an equivariant neural network?, 2022. Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In The Seventh International Conference on Learning Representations, 2019. Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In International Conference on Learning Representations, 2017. Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. Association for Computational Linguistics, 10 2018. Nguyen and Salazar (2019) Toan Q. Nguyen and Julian Salazar. Transformers without Tears: Improving the Normalization of Self-Attention. In Proceedings of the 16th International Conference on Spoken Language Translation. Association for Computational Linguistics, 11 2019. Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534. Association for Computational Linguistics, 08 2016. Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 8024–8035. Curran Associates, Inc., 2019. Penedo et al. (2024) Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, volume 37, pages 30811–30849. Curran Associates, Inc., 2024. Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 08 2021. Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense Reasoning about Social Interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473. Association for Computational Linguistics, 11 2019. Stollenwerk (2026) Felix Stollenwerk. On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 674–681. Association for Computational Linguistics, 03 2026. Sun et al. (2024) Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive Activations in Large Language Models. In First Conference on Language Modeling, 2024. Sun et al. (2025) Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The Curse of Depth in Large Language Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going Deeper With Image Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 32–42, 10 2021. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017. Wang et al. (2019) Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning Deep Transformer Models for Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822. Association for Computational Linguistics, 07 2019. Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing Multiple Choice Science Questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106. Association for Computational Linguistics, 09 2017. Xiao et al. (2024) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations, 2024. Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On Layer Normalization in the Transformer Architecture. In Hal Daumé I and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524–10533. PMLR, 07 2020. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. Association for Computational Linguistics, 07 2019. Zhang and Sennrich (2019) Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 12360–12371. Curran Associates, Inc., 2019. Zheng et al. (2026) Chuanyang Zheng, Jiankai Sun, Yihang Gao, Chi Wang, Yuehao Wang, Jing Xiong, Liliang Ren, Bo Peng, Qingmei Wang, Xiaoran Shang, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, and Xiaodong Liu. GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization, 2026. Zhu et al. (2025) Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without Normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14901–14911, 06 2025. Appendix A Normalization as Hypersphere Projection Throughout this appendix we work with idealized normalization maps: we omit the numerical stabilizer ε in the denominators, and we set all additive bias terms to zero, since a translation does not affect the projection interpretation. For r>0r>0, let rd−1:=∈ℝd:∥2=r,S^d-1_r:= \x ^d:\, _2=r \, and, for ∈ℝd∖x ^d \ 0\, define the radial projection onto rd−1S^d-1_r by Πr():=r⋅∥2. _r(x):=r· x _2. For every nonzero x, the point Πr() _r(x) is also the unique Euclidean metric projection of x onto rd−1S^d-1_r. More generally, for a linear subspace ⊆ℝdV ^d we write r():=∈:∥2=r,S_r(V):= \y :\, _2=r \, which is a (dim−1)( -1)-dimensional sphere isometric to rdim−1S -1_r. LayerNorm. Inspired by BatchNorm (Ioffe and Szegedy, 2015), Ba et al. (2016) proposed LayerNorm, which normalizes each input vector along its feature dimension. Let =(1,…,1)⊤∈ℝd1=(1,…,1) ^d and define μ=1d⊤,=−1d⊤.μ= 1d1 x, =I- 1d11 . Since =⊤=2P=P =P^2 and ker=span =span\1\, the matrix P is the orthogonal projection onto the mean-zero hyperplane ℋ=∈ℝd:⊤=0,dimℋ=d−1,H= \y ^d:1 y=0 \, =d-1, and the centered input satisfies −μ=x- 1=Px. Consequently, the coordinate-wise standard deviation of x satisfies σ=1d∑i=1d(xi−μ)2=∥2d,σ= 1d _i=1^d(x_i-μ)^2= _2 d, so that σ>0σ>0 if and only if ≠Px≠ 0, i.e., ∉spanx \1\; we restrict attention to this case, on which LayerNorm is well defined. Let ∈ℝd γ ^d denote the learnable gain and let =diag()D_ γ=diag( γ). LayerNorm can be written as LayerNorm()=⊙−μσ=(d⋅∥2)=(Πd()). splitLayerNorm(x)&= γ x- 1σ\\ &=D_ γ ( d· Px _2 )\\ &=D_ γ ( _ d(Px) ). split (12) Equation (12) shows that, before the coordinate-wise gain is applied, LayerNorm performs two successive projections: the orthogonal projection of x onto ℋH, followed by the radial projection of the centered vector onto d(ℋ)S_ d(H), a (d−2)(d-2)-dimensional sphere of radius d d contained in ℋH. The image of the complete map is therefore (d(ℋ))=:∈d(ℋ),D_ γ (S_ d(H) )= \D_ γy:\,y _ d(H) \, which lies in the hyperplane ℋD_ γH and is in general an ellipsoid, not necessarily axis-aligned, rather than a sphere. Assume d≥3d≥ 3 and γi≠0 _i≠ 0 for every i. The image is a round sphere if and only if D_ γ acts conformally on ℋH, i.e., 2=c2PD_ γ^2P=c^2P for some c>0c>0. Testing this identity on the pairs i−je_i-e_j and i−ke_i-e_k with i,j,ki,j,k distinct, which is possible because d≥3d≥ 3, yields γi2=c2 _i^2=c^2 for every i. Hence the image is a sphere precisely when |γ1|=|γ2|=⋯=|γd| _1 = _2 =·s= _d . Since γ is typically initialized at 1, at initialization LayerNorm reduces to ↦Πd()x _ d(Px) and all outputs share the same norm d d. Once training drives γ away from 1, the outputs leave d(ℋ)S_ d(H); they remain on some sphere only in the special case in which all |γi| _i coincide, and are otherwise confined to the ellipsoid (d(ℋ))D_ γ (S_ d(H) ). RMSNorm. RMSNorm (Zhang and Sennrich, 2019) dispenses with the mean-centering step of LayerNorm and normalizes an input solely by its root mean square. For ∈ℝd∖x ^d \ 0\, define the root mean square (RMS) operation as RMS()=1d∑i=1dxi2=∥2d.RMS(x)= 1d _i=1^dx_i^2= _2 d. Let ∈ℝd γ ^d denote the learnable coordinate-wise gain and let =diag()D_ γ=diag( γ). RMSNorm is then given by RMSNorm()=⊙RMS()=(d⋅∥2)=Πd(). splitRMSNorm(x)&= γ xRMS(x)\\ &=D_ γ ( d· x _2 )\\ &=D_ γ _ d(x). split (13) The normalization stage of RMSNorm, prior to the coordinate-wise gain, is therefore precisely the radial projection onto d−1S^d-1_ d. The complete map, however, is in general no longer spherical: its image is (d−1)D_ γ\! (S^d-1_ d ), which, when γi≠0 _i≠ 0 for every i, is an axis-aligned ellipsoid whose semi-axes have lengths d|γi| d\, _i ; if some γi _i vanishes, the image degenerates to a lower-dimensional set. For d≥2d≥ 2, the spherical geometry is retained precisely when all entries of γ share a common absolute value c≠0c≠ 0, in which case the image is the rescaled sphere cd−1S^d-1_c d. Since γ is typically initialized at 1, RMSNorm coincides with the projection onto d−1S^d-1_ d at initialization; once training updates γ, its outputs generally drift off the hypersphere. ScaleNorm. Among commonly used normalization methods, ScaleNorm (Nguyen and Salazar, 2019) is the one most directly related to hypersphere projection. For ∈ℝd∖x ^d \ 0\ it is defined by ScaleNorm()=g∥2,ScaleNorm(x)=g\, x _2, (14) where g∈ℝg is a learnable scalar parameter. If g>0g>0, then ScaleNorm()=Πg(),ScaleNorm(x)= _g(x), so ScaleNorm maps every nonzero input onto the hypersphere gd−1S^d-1_g. More generally, when g is unconstrained, ∥ScaleNorm()∥2=|g|. (x) _2= g . Thus, for g<0g<0 the spherical interpretation still holds: the map is the composition of the radial projection Π|g| _ g onto |g|d−1S^d-1_ g with the antipodal map ↦−x -x, and only the degenerate case g=0g=0 collapses the image to the origin. If an explicitly positive radius parameter is needed, one may use a positive reparameterization such as replacing g with Softplus(g)Softplus(g), although this is not required for the geometric interpretation. Connecting ScaleNorm with Proj-SpheretNorm. The relationship between ScaleNorm and Proj-SpheretNorm can be made precise through their common radial-projection structure. Let ∈ℝd∖u ^d \ 0\ be the input to a generic Proj-SpheretNorm update, let ℱF denote the associated sub-layer map, and let α∈ℝα be the residual coefficient. Set r=∥2,=−⊤r2,=ℱ().r= _2, _u=I- uu r^2, =P_uF(u). Here P_u is the orthogonal projection onto ⟂u , which is the tangent space of rd−1S^d-1_r at u; in particular ⊤=0u z=0 by construction. The Proj-SpheretNorm update can therefore be written as +=r+α∥+α∥2=Πr(+α).u^+=r\, u+ + _2= _r (u+ ). (15) Moreover, orthogonality gives ∥+α∥22=r2+α2∥22>0, + _2^2=r^2+α^2 _2^2>0, so the radial projection is well defined whenever ≠u≠ 0. Equation (15) also gives ∥+∥2=∥2 ^+ _2= _2. For l=1, 2,…,L−1l=1,\,2,\,…,\,L-1, the (l+1)(l+1)-th hidden state is given by l+1=SpheretNorm(l,ℱl(l)).h_l+1=SpheretNorm\! (h_l,\,F_l(h_l) ). When Proj-SpheretNorm is used, the definition in Eq. (6) gives l+1=Π∥l∥2(l+αll),l=lℱl(l),h_l+1= _ _l _2 (h_l+ _lz_l ), _l=P_h_lF_l(h_l), so that ∥l+1∥2=∥l∥2 _l+1 _2= _l _2 for l=1,2,…,L−1l=1,2,…,L-1. Consequently, the residual stream preserves the hidden-state norm across layers, i.e., ∥1∥2=∥2∥2=⋯=∥L∥2 _1 _2= _2 _2=·s= _L _2. At the beginning of the residual stream considered in our analysis, we take 1h_1 to be the hidden state produced by the modified ScaleNorm, so that, by construction, ∥1∥2=Softplus(γ), _1 _2=Softplus(γ), which is strictly positive for every γ∈ℝγ . We do not introduce notation for the representations preceding 1h_1, and any operations applied after Lh_L lie outside the scope of this analysis. Equation (15) indicates that ScaleNorm and Proj-SpheretNorm share the same radial-projection operation but use it for different purposes. ScaleNorm fixes the radius of the initial hidden state 1h_1, whereas each Proj-SpheretNorm sub-layer forms a tangent residual update and projects the resulting point back onto the sphere determined by the radius of its own input state. Under the above fixed-radius dynamics, every hidden state along the residual stream lies on the same hypersphere Softplus(γ)d−1S^d-1_Softplus(γ). In summary, LayerNorm, RMSNorm, and ScaleNorm are all related to hypersphere projection, but in different ways. The normalization stages of LayerNorm and RMSNorm, taken before the gain, are spherical maps: LayerNorm first projects onto the zero-mean hyperplane and then performs a radial projection within that hyperplane, while RMSNorm performs a radial projection in the ambient space. Their complete maps, however, generally include a coordinate-wise gain and therefore need not have spherical images. ScaleNorm differs in that it uses a single scalar gain. Its complete map therefore has a spherical image of radius |g| g , and coincides exactly with the radial projection Πg _g when g>0g>0. This positive-radius case provides the closest traditional counterpart to Proj-SpheretNorm. Specifically, Proj-SpheretNorm replaces the globally learned radius g with the radius ∥l∥2 _l _2 carried by the current hidden state, and applies the radial projection to the point obtained after the tangent update. It can thus be viewed as a state-dependent, retraction-based variant of the radial projection used in ScaleNorm, designed to preserve the hidden-state norm exactly across the entire depth. Appendix B Hyperparameters The shared hyperparameters used across all experiments are summarized in Table 6. The architecture hyperparameters for the small and medium model sizes are listed in Table 7. The training and validation losses in OpenWebText is illustrated in Figure 3. Table 6: Shared hyperparameters. The effective batch size is held fixed across all runs, independent of the number of GPUs. Name Value optimizer AdamW β1 _1 0.90.9 β2 _2 0.950.95 weight decay 0.10.1 gradient clip 1.01.0 dropout 0.00.0 bias enable # of iterations 50,00050,000 # of learning-rate decay iterations 50,00050,000 # of warm-up iterations 2,0002,000 block size (sequence length) 1,0241,024 batch size (per GPU) 1616 effective (global) batch size 128128 sequences tokens per update 131,072131,072 total training tokens ∼6.55 \!6.55 B precision bfloat16 torch.compile disable vocabulary size 50,30450,304 Table 7: Architecture hyperparameters. Parameter counts include the tied token embedding and are reported for the Pre-LN baseline. All variants have identical parameter counts. Name S M L # of layers 1212 2424 3636 # of heads 1212 1616 2020 # of params. ∼0.12 \!0.12 B ∼0.35 \!0.35 B ∼0.77 \!0.77 B hidden dim. 768768 10241024 12801280 lr 6×10−46× 10^-4 3×10−43× 10^-4 2.5×10−42.5× 10^-4 min. lr 6×10−56× 10^-5 3×10−53× 10^-5 2.5×10−52.5× 10^-5 (a) Training loss. (b) Validation loss. Figure 3: Training and validation losses in the OpenWebText. Appendix C Pseudocode for SpheretNorms In this section, we provide PyTorch-style pseudocode for our SpheretNorms: Proj-SpheretNorm, Cay-SpheretNorm, and p-SpheretNorm. List of codeblocks 1 PyTorch-style pseudocode for GeoNorm. ⬇ 1def geonorm(h_in, Fh, decay_method, layer_idx, num_layers, angle_upper_bound, alpha): 2 h_in_norm_sq = (h_in * h_in).sum(dim=-1, keepdim=True).clamp(min=1e-12) 3 h_in_norm = h_in_norm_sq.sqrt() 4 h_in_trs_dot_Fh = (h_in * Fh).sum(dim=-1, keepdim=True) 5 6 z = Fh - (h_in_trs_dot_Fh / h_in_norm_sq) * h_in 7 z_norm_sq = (z * z).sum(dim=-1, keepdim=True).clamp(min=1e-12) 8 z_norm = z_norm_sq.sqrt() 9 z_hs = z / z_norm 10 11 if decay_method == 'harmonic': 12 alpha = torch.clamp(F.softplus(alpha), max=1.0) / layer_idx 13 elif decay_method == 'linear': 14 alpha = torch.clamp(F.softplus(alpha), max=1.0) * (num_layers - layer_idx) / num_layers 15 elif decay_method == 'sqrt': 16 alpha = torch.clamp(F.softplus(alpha), max=1.0) / math.sqrt(layer_idx) 17 18 beta = (alpha * (z_norm / h_in_norm)).clamp(max=angle_upper_bound) 19 h_out = torch.cos(beta) * h_in + h_in_norm * torch.sin(beta) * z_hs 20 21 return h_out List of codeblocks 2 PyTorch-style pseudocode for Proj-SpheretNorm. ⬇ 1def proj_spheretnorm(h_in, Fh, layer_idx, alpha): 2 h_in_norm_sq = (h_in * h_in).sum(dim=-1, keepdim=True).clamp(min=1e-12) 3 h_in_norm = h_in_norm_sq.sqrt() 4 h_in_trs_dot_Fh = (h_in * Fh).sum(dim=-1, keepdim=True) 5 6 z = Fh - (h_in_trs_dot_Fh / h_in_norm_sq) * h_in 7 8 alpha = torch.clamp(F.softplus(alpha), max=1.0) / math.sqrt(self.layer_idx) 9 10 h_plus = h_in + alpha * z 11 h_plus_norm_sq = (h_plus * h_plus).sum(dim=-1, keepdim=True).clamp(min=1e-12) 12 h_plus_norm = h_plus_norm_sq.sqrt() 13 h_out = h_in_norm * (h_plus / h_plus_norm) 14 15 return h_out List of codeblocks 3 PyTorch-style pseudocode for Cay-SpheretNorm. ⬇ 1def cay_spheretnorm(h_in, Fh, layer_idx, angle_upper_bound, alpha): 2 h_in_norm_sq = (h_in * h_in).sum(dim=-1, keepdim=True).clamp(min=1e-12) 3 h_in_norm = h_in_norm_sq.sqrt() 4 h_in_trs_dot_Fh = (h_in * Fh).sum(dim=-1, keepdim=True) 5 6 z = Fh - (h_in_trs_dot_Fh / h_in_norm_sq) * h_in 7 z_norm_sq = (z * z).sum(dim=-1, keepdim=True).clamp(min=1e-12) 8 z_norm = z_norm_sq.sqrt() 9 10 alpha = torch.clamp(F.softplus(alpha), max=1.0) / math.sqrt(layer_idx) 11 12 beta_max = math.tan(angle_upper_bound * 0.5) * 2.0 13 gamma = torch.minimum(alpha, beta_max * (h_in_norm / z_norm)) 14 beta = gamma * (z_norm / h_in_norm) 15 beta_sq = beta * beta 16 h_out = ((4 - beta_sq) * h_in + 4 * gamma * z) / (4 + beta_sq) 17 18 return h_out List of codeblocks 4 PyTorch-style pseudocode for p-SpheretNorm. ⬇ 1def p_spheretnorm(h_in, Fh, layer_idx, angle_upper_bound, alpha, p): 2 h_in_norm_sq = (h_in * h_in).sum(dim=-1, keepdim=True).clamp(min=1e-12) 3 h_in_norm = h_in_norm_sq.sqrt() 4 h_in_trs_dot_Fh = (h_in * Fh).sum(dim=-1, keepdim=True) 5 6 z = Fh - (h_in_trs_dot_Fh / h_in_norm_sq) * h_in 7 z_norm_sq = (z * z).sum(dim=-1, keepdim=True).clamp(min=1e-12) 8 z_norm = z_norm_sq.sqrt() 9 z_hs = z / z_norm 10 11 alpha = torch.clamp(F.softplus(alpha), max=1.0) / math.sqrt(self.layer_idx) 12 13 beta = alpha * (z_norm / h_in_norm) 14 theta = p * torch.arctan(beta / p) 15 if p > 1.0: 16 theta = torch.clamp(theta, max=angle_upper_bound) 17 18 h_out = torch.cos(theta) * h_in + (h_in_norm * torch.sin(theta)) * z_hs 19 20 return h_out Appendix D Technical Proofs for Section 4 D.1 Proof of Proposition 3 Let =span,U=span\h,v\. We first prove uniqueness. Suppose that W∈(d)W∈ so(d) satisfies =,range()⊆.Wh=v, (W) . For every ∈⟂x and ∈y , skew-symmetry gives ⟨,⟩=−⟨,⟩. ,y =- ,Wy . Since ∈Wy and ∈⟂x , it follows that ⟨,⟩=0 ,y =0 for every ∈y . On the other hand, ∈Wx by the range condition. Therefore, =,∈⟂.Wx= 0, . Thus, W is completely determined by its restriction to U. If =v= 0, then =spanU=span\h\ and =Wh= 0. Hence =W= 0, which agrees with the claimed formula. Suppose now that ≠v≠ 0, and define the orthonormal vectors 1=r,2=‖2.e_1= hr, _2= v\|v\|_2. Since =Wh=v, we have 1=‖2r2.We_1= \|v\|_2re_2. Skew-symmetry then uniquely determines the action on 2e_2: 2=−‖2r1We_2=- \|v\|_2re_1. Together with =W= 0 on ⟂U , this proves that at most one such skew-symmetric matrix exists. Now define ^=⊤−⊤r2. W= vh -hv r^2. Then ^⊤=− W =- W, so ^∈(d) W∈ so(d). Since ⊤=0h v=0 and ‖22=r2\|h\|_2^2=r^2, ^=⊤−⊤r2=. Wh= vh h-hv hr^2=v. Moreover, range(^)⊆span,range( W) \h,v\. Therefore, by uniqueness, =^=⊤−⊤r2.W= W= vh -hv r^2. We next derive the closed form of the Cayley update. Define ρ=‖2rρ= \|v\|_2r, we obtain =⊤−⊤r2=−ρ2.Wv= vh v-hv vr^2=-ρ^2h. The matrix −12I- 12W is invertible. Indeed, if (−12)=0 (I- 12W )x=0, then =12x= 12Wx. Taking the inner product with x and using ⊤=x Wx= 0 gives ‖22=0\|x\|_2^2=0, and hence =x= 0. Set a=4−ρ24+ρ2a= 4-ρ^24+ρ^2 and b=44+ρ2b= 44+ρ^2. Using =Wh=v and =−ρ2Wv=-ρ^2h, we have (−12)(a+b) (I- 12W )(ah+bv) =a+b−a2+bρ22 =ah+bv- a2v+ bρ^22h =(a+bρ22)+(b−a2). = (a+ bρ^22 )h+ (b- a2 )v. The definitions of a and b give a+bρ22=1a+ bρ^22=1 and b−a2=12b- a2= 12. Therefore, (−12)(a+b)=+12=(+12). (I- 12W )(ah+bv)=h+ 12v= (I+ 12W )h. Since −12I- 12W is invertible, (W) (W)h =(−12)−1(+12) = (I- 12W )^-1 (I+ 12W )h =a+b =ah+bv =4−ρ24+ρ2+44+ρ2. = 4-ρ^24+ρ^2h+ 44+ρ^2v. This proves the claimed closed-form expression. D.2 Proof of Proposition 5 For p=1p=1, we have θl=arctan(βl) _l= ( _l), and hence cos(θl)=11+βl2,sin(θl)=βl1+βl2. \! ( _l )= 1 1+ _l^2, \! ( _l )= _l 1+ _l^2. Since βl=αlsl/rl _l= _ls_l/r_l and l⊤l=0h_l z_l=0, it follows that l+1(1)=l+αll1+βl2=rll+αll‖l+αll‖2.h_l+1^(1)= h_l+ _lz_l 1+ _l^2=r_l h_l+ _lz_l\|h_l+ _lz_l\|_2. This is precisely the Proj-SpheretNorm update. For p=2p=2, the double-angle identities give cos(2arctanβl2)=4−βl24+βl2,sin(2arctanβl2)=4βl4+βl2. \! (2 _l2 )= 4- _l^24+ _l^2,\, \! (2 _l2 )= 4 _l4+ _l^2. Substituting βl=αlsl/rl _l= _ls_l/r_l into Definition 4 yields l+1(2)=4−βl24+βl2l+4αl4+βl2l,h_l+1^(2)= 4- _l^24+ _l^2h_l+ 4 _l4+ _l^2z_l, which is the Cay-SpheretNorm update. For every fixed βl _l, limp→+∞parctan(βlp)=βl. _p→+∞p ( _lp )= _l. The continuity of the sine and cosine functions therefore gives the exponential-map limit. Finally, |parctan(βlp)|≤πp2→0as p→0+. |p ( _lp ) |≤ π p2→ 0 p→ 0^+. Thus cos(θl)→1 ( _l)→ 1 and sin(θl)→0 ( _l)→ 0, proving convergence to the identity mapping. D.3 Proof of Proposition 7 Consistent with the L retained hidden states 1,…,Lh_1,…,h_L, we index the individual SpheretNorm updates by l=1,…,L−1l=1,…,L-1. Let l+1=SpheretNorm(l,ℱl(l)),h_l+1=SpheretNorm (h_l,F_l(h_l) ), where the step-size and retraction parameters are suppressed from the notation. Define rl=‖l‖2r_l=\|h_l\|_2, l=−ll⊤rl2P_l=I- h_lh_l r_l^2, and l=lℱl(l)z_l=P_lF_l(h_l). Then l⊤l=0h_l z_l=0. Suppose first that l≠0z_l≠ 0. For an appropriate intrinsic angle θl _l, every SpheretNorm variant considered in this work admits the representation l+1=cos(θl)l+rlsin(θl)l‖l‖2.h_l+1= ( _l)h_l+r_l ( _l) z_l\|z_l\|_2. Since l⊤l=0h_l z_l=0, the two terms on the right-hand side are orthogonal, and therefore ‖l+1‖22 \|h_l+1\|_2^2 =cos2(θl)‖l‖22+rl2sin2(θl)‖l‖l‖2‖22=rl2. = ^2( _l)\|h_l\|_2^2+r_l^2 ^2( _l) \| z_l\|z_l\|_2 \|_2^2=r_l^2. Hence, ‖l+1‖2=‖l‖2\|h_l+1\|_2=\|h_l\|_2. When l=0z_l=0, the continuously extended update satisfies l+1=lh_l+1=h_l, and the same identity holds. Therefore, by induction, ‖l‖2=‖1‖2,l=1,…,L.\|h_l\|_2=\|h_1\|_2, l=1,…,L. Since 1h_1 is the output of the modified ScaleNorm operation, ‖1‖2=Softplus(γ).\|h_1\|_2=Softplus(γ). Consequently, ‖l‖2=Softplus(γ),l=1,…,L.\|h_l\|_2=Softplus(γ), l=1,…,L. In particular, ‖L‖2=Softplus(γ),\|h_L\|_2=Softplus(γ), which is stronger than, and therefore implies, the stated upper bound ‖L‖2≤Softplus(γ).\|h_L\|_2 (γ). D.4 Second-Order Retraction and Local Consistency We prove the smoothness, second-order retraction, and local-consistency properties used in Section 1.3. Fix p>0p>0, let ∈rd−1h _r^d-1, ∈Trd−1v∈ T_hS_r^d-1, and define ρ=‖2rρ= \|v\|_2r, θp(ρ)=parctan(ρp) _p(ρ)=p ( ρp ). For ρ>0ρ>0, define ap(ρ)=cos(θp(ρ)),bp(ρ)=sin(θp(ρ))ρ,a_p(ρ)= ( _p(ρ) ), b_p(ρ)= ( _p(ρ) )ρ, and set bp(0)=1b_p(0)=1. The p-angular map can then be written as ℛ(p)()=ap(ρ)+bp(ρ).R_h^(p)(v)=a_p(ρ)h+b_p(ρ)v. Smoothness at the zero section. The apparent singularity in bp(ρ)b_p(ρ) at ρ=0ρ=0 is removable. To see this, consider the real-analytic odd function θ~p(t)=parctan(tp),t∈ℝ. θ_p(t)=p ( tp ), t . Define Ap(t)=cos(θ~p(t))A_p(t)= ( θ_p(t) ) and Bp(t)=sin(θ~p(t))t,t≠0,1,t=0.B_p(t)= cases ( θ_p(t) )t,&t≠ 0,\\[5.16663pt] 1,&t=0. cases. Since θ~p θ_p is odd, both ApA_p and BpB_p are even real-analytic functions in a neighborhood of the origin. Hence, there exist real-analytic functions A^p A_p and B^p B_p, defined near zero, such that Ap(t)=A^p(t2),Bp(t)=B^p(t2).A_p(t)= A_p(t^2), B_p(t)= B_p(t^2). Consequently, ℛ(p)()=A^p(‖22r2)+B^p(‖22r2).R_h^(p)(v)= A_p ( \|v\|_2^2r^2 )h+ B_p ( \|v\|_2^2r^2 )v. This representation is smooth at =v=0. Since r>0r>0 is fixed on rd−1S_r^d-1, it also shows that (,)→ℛ(p)()(h,v) _h^(p)(v) is smooth on the tangent bundle Trd−1TS_r^d-1 near the zero section. Away from =v=0, smoothness follows directly from the defining formula. Sphere-valued property. Since ⊤=0h v=0, ‖2=r\|h\|_2=r, and ‖2=rρ\|v\|_2=rρ, we obtain, for ≠v≠ 0, ‖ℛ(p)()‖22 \|R_h^(p)(v) \|_2^2 =ap(ρ)2‖22+bp(ρ)2‖22 =a_p(ρ)^2\|h\|_2^2+b_p(ρ)^2\|v\|_2^2 =r2cos2(θp(ρ))+r2sin2(θp(ρ))=r2. =r^2 ^2 ( _p(ρ) )+r^2 ^2 ( _p(ρ) )=r^2. The same conclusion is immediate at =v=0. Therefore, ℛ(p)()∈rd−1R_h^(p)(v) _r^d-1. Retraction and second-order properties. As ρ→0ρ→ 0, θp(ρ)=ρ−ρ33p2+(ρ5). _p(ρ)=ρ- ρ^33p^2+O(ρ^5). It follows that ap(ρ)=1−ρ22+(124+13p2)ρ4+(ρ6),a_p(ρ)=1- ρ^22+ ( 124+ 13p^2 )ρ^4+O(ρ^6), and bp(ρ)=1−(16+13p2)ρ2+(ρ4).b_p(ρ)=1- ( 16+ 13p^2 )ρ^2+O(ρ^4). Equivalently, as s→0s→ 0, it follows that A^p(s)=1−s2+(124+13p2)s2+(s3), A_p(s)=1- s2+ ( 124+ 13p^2 )s^2+O(s^3), and B^p(s)=1−(16+13p2)s+(s2). B_p(s)=1- ( 16+ 13p^2 )s+O(s^2). Fix ∈Trd−1v∈ T_hS_r^d-1, set ρ0=‖2r _0= \|v\|_2r, and consider the curve (t)=ℛ(p)(t)c(t)=R_h^(p)(tv). Using the even-function representation above, we have (t)=A^p(t2ρ02)+tB^p(t2ρ02).c(t)= A_p(t^2 _0^2)h+t B_p(t^2 _0^2)v. Therefore, (t)=+t−t2‖222r2+(t3).c(t)=h+tv- t^2\|v\|_2^22r^2h+O(t^3). Hence, (0)=c(0)=h, ′(0)=c (0)=v and ′(0)=−‖22r2c (0)=- \|v\|_2^2r^2h. The first two identities verify the defining conditions of a retraction. Moreover, since Nrd−1=span,N_hS_r^d-1=span\h\, the acceleration ′(0)c (0) is normal to the hypersphere. Equivalently, ′(0)=,=−⊤r2.P_hc (0)= 0, _h=I- hh r^2. Hence, the retraction curve has no tangential acceleration at the origin, which is precisely the second-order retraction condition. Therefore, ℛ(p)R^(p) is a second-order retraction on rd−1S_r^d-1 for every fixed p>0p>0. Local expansion of the SpheretNorm update. For the l-th SpheretNorm update, let rl=‖l‖2r_l=\|h_l\|_2, l=−ll⊤rl2P_l=I- h_lh_l r_l^2, l=lℱl(l)z_l=P_lF_l(h_l), and sl=‖l‖2s_l=\|z_l\|_2. Applying the preceding expansion to =αllv= _lz_l gives l+1= _l+1= l+αll−αl2‖l‖222rl2l _l+ _lz_l- _l^2\|z_l\|_2^22r_l^2h_l −(16+13p2)αl3‖l‖22rl2l+(αl4). - ( 16+ 13p^2 ) _l^3\|z_l\|_2^2r_l^2z_l+O( _l^4). Here, the expansion is taken as αl→0 _l→ 0, with lh_l and lz_l fixed. More generally, the remainder is uniform over any local regime in which rlr_l is bounded away from zero and ηl=‖l‖2rl _l= \|z_l\|_2r_l remains bounded. The linear term αll _lz_l is tangent to the hypersphere, while the quadratic term −αl2‖l‖222rl2l- _l^2\|z_l\|_2^22r_l^2h_l is normal to the hypersphere at lh_l. Moreover, the parameter p first appears in the cubic tangent correction −(16+13p2)αl3‖l‖22rl2l.- ( 16+ 13p^2 ) _l^3\|z_l\|_2^2r_l^2z_l. Thus, all members of the p-angular family have the same first-order tangent dynamics and the same second-order normal correction, while their finite-step dependence on p first appears at cubic order. Finally, since l=lℱl(l)z_l=P_lF_l(h_l), we obtain the first-order consistency relation l+1=l+αllℱl(l)+(αl2).h_l+1=h_l+ _lP_lF_l(h_l)+O( _l^2). D.5 Depth-Wise Conditioning Criterion We first formulate the conditioning criterion for a generic ordered sequence of individual SpheretNorm updates. Let M denote the number of such updates, and let j=∂j+1∂j,j=1,…,M.J_j= _j+1 _j, j=1,…,M. For each j, define qj=|cos(θj)|,sj>0,1,sj=0,q_j= cases| ( _j)|,&s_j>0,\\ 1,&s_j=0, cases and mj=max0,qj−Δjm_j= \0,q_j- _j\. Assume that there exists a constant m∗>0m_*>0, independent of j and M, such that mj≥m∗m_j≥ m_*, j=1,…,Mj=1,…,M. Since mj≤1m_j≤ 1, we have −logmj=∫mj1dt≤1−mjm∗.- m_j= _m_j^1 dtt≤ 1-m_jm_*. Because mj>0m_j>0, the maximum in its definition is inactive, and hence mj=qj−Δjm_j=q_j- _j. Therefore, 1−mj=1−qj+Δj.1-m_j=1-q_j+ _j. For every real θ, 1−|cosθ|≤1−cosθ≤θ221-| θ|≤ 1- θ≤ θ^22. Consequently, 1−mj≤|θj|22+Δj1-m_j≤ | _j|^22+ _j. It follows that −log(∏j=1Mmj) - ( _j=1^Mm_j ) =−∑j=1Mlogmj =- _j=1^M m_j ≤1m∗[12∑j=1M|θj|2+∑j=1MΔj]. ≤ 1m_* [ 12 _j=1^M| _j|^2+ _j=1^M _j ]. Hence, ∏j=1Mmj≥exp−1m∗[12∑j=1M|θj|2+∑j=1MΔj]. _j=1^Mm_j≥ \- 1m_* [ 12 _j=1^M| _j|^2+ _j=1^M _j ] \. Let M=MM−1⋯1G_M=J_MJ_M-1·sJ_1. Combining the preceding estimate with the layerwise bounds of Proposition 6 gives σmax(M)≤exp(∑j=1MΔj) _ (G_M)≤ ( _j=1^M _j ) and σmin(M)≥exp−1m∗[12∑j=1M|θj|2+∑j=1MΔj]. _ (G_M)≥ \- 1m_* [ 12 _j=1^M| _j|^2+ _j=1^M _j ] \. Therefore, if ∑j=1MΔj=(1) _j=1^M _j=O(1) and ∑j=1M|θj|2=(1) _j=1^M| _j|^2=O(1), then both σmax(M) _ (G_M) and σmin(M)−1 _ (G_M)^-1 remain bounded independently of the number M of individual SpheretNorm updates. For the residual stream 1,2,…,Lh_1,h_2,…,h_L considered in Appendix A and Appendix D.3, there are M=L−1M=L-1 individual SpheretNorm updates. More precisely, setting j=j,j=∂j+1∂j,j=1,…,L−1,x_j=h_j, _j= _j+1 _j, j=1,…,L-1, gives M=L−1=∂L∂1=L−1L−2⋯1.G_M=G_L-1= _L _1=J_L-1J_L-2·sJ_1. Therefore, the preceding depth-wise conditioning criterion applies to the actual residual stream with ∑j=1L−1Δj=(1),∑j=1L−1|θj|2=(1). _j=1^L-1 _j=O(1), _j=1^L-1| _j|^2=O(1). Proposition 6 uses L as the generic number of individual SpheretNorm updates. When the result is applied to the retained-state indexing 1,…,Lh_1,…,h_L, used in Appendix A and Appendix D.3, the corresponding update count is M=L−1M=L-1. D.6 Proof of the Depth-Schedule Growth Proposition The three schedules compared in Proposition 8 are λl=1 _l=1, λl=l−1/2 _l=l^-1/2, and λl=l−1 _l=l^-1. In particular, the third schedule is the harmonic schedule λl=l−1 _l=l^-1, for which ∑l=1Lλl=(logL),∑l=1Lλl2=(1). _l=1^L _l=O( L), _l=1^L _l^2=O(1). By definition, SL=∑l=1LλlalκlS_L= _l=1^L _la_l _l. Since 0<al≤10<a_l≤ 1 and κl≤K _l≤ K, it follows that SL≤K∑l=1LλlS_L≤ K _l=1^L _l. Furthermore, since |parctan(βp)|≤|β|,β∈ℝ, |p ( βp ) |≤|β|, β , we have |θl|≤|βl|=λlalηl | _l |≤| _l|= _la_l _l. Using alηl≤Ha_l _l≤ H, we obtain AL=∑l=1L|θl|2≤∑l=1Lλl2(alηl)2≤H2∑l=1Lλl2.A_L= _l=1^L | _l |^2≤ _l=1^L _l^2(a_l _l)^2≤ H^2 _l=1^L _l^2. For the constant schedule λl=1 _l=1, we have ∑l=1Lλl=L _l=1^L _l=L and ∑l=1Lλl2=L _l=1^L _l^2=L. For the layerwise square-root schedule λl=l−1/2 _l=l^-1/2, ∑l=1Lλl=∑l=1L1l≤2L=(L), _l=1^L _l= _l=1^L 1 l≤ 2 L=O( L), while ∑l=1Lλl2=∑l=1L1l≤1+logL=(logL). _l=1^L _l^2= _l=1^L 1l≤ 1+ L=O( L). For the harmonic schedule λl=l−1 _l=l^-1, ∑l=1Lλl _l=1^L _l =∑l=1L1l=(logL) and = _l=1^L 1l=O( L)\, and ∑l=1Lλl2 _l=1^L _l^2 =∑l=1L1l2≤π26=(1). = _l=1^L 1l^2≤ π^26=O(1). The stated growth rates follow. Application to the retained-state indexing. Proposition 8 uses L as the generic number of individual SpheretNorm updates. For the retained-state sequence 1,2,…,Lh_1,h_2,…,h_L, the corresponding number of updates is M=L−1M=L-1. For l=1,…,Ml=1,…,M, let αl=λlal _l= _la_l and Δl=λlalκl _l= _la_l _l, where κl=2‖l‖2+ηl _l=2\|B_l\|_2+ _l. Suppose that κl≤K,alηl≤H,l=1,…,M. _l≤ K, a_l _l≤ H, l=1,…,M. Define the accumulated sensitivity envelope and squared angular budget by SMres=∑l=1MΔl,AMres=∑l=1M|θl|2.S_M^res= _l=1^M _l, A_M^res= _l=1^M| _l|^2. Then the preceding estimates give SMres≤K∑l=1Mλl,AMres≤H2∑l=1Mλl2.S_M^res≤ K _l=1^M _l, A_M^res≤ H^2 _l=1^M _l^2. Since M=L−1M=L-1, these estimates are equivalently λlSL−1resAL−1res1(L)(L)1/l(L)(logL)1/l(logL)(1). array[]c|c _l&S_L-1^res&A_L-1^res\\ 1&O(L)&O(L)\\ 1/ l&O( L)&O( L)\\ 1/l&O( L)&O(1). array Thus, specializing the generic update count in Proposition 8 changes only the endpoint of the sums and does not alter any of the stated asymptotic growth rates. D.7 Full Jacobian and Layerwise Estimates Indexing convention. Consistent with the retained-state indexing used in Appendix A and Appendix D.3, we consider the sequence 1,2,…,Lh_1,h_2,…,h_L, where, for k=1,…,L−1k=1,…,L-1, k+1=SpheretNorm(k,ℱk(k)).h_k+1=SpheretNorm (h_k,F_k(h_k) ). There are therefore N=L−1N=L-1 individual SpheretNorm updates between 1h_1 and Lh_L. For notational convenience in the Jacobian calculation, set k=kx_k=h_k for k=1,…,N+1k=1,…,N+1. Then 1=1,N+1=L,x_1=h_1, _N+1=h_L, and the k-th individual SpheretNorm update is written as k⟼k+1,k=1,…,N.x_k _k+1, k=1,…,N. Proposition 6 uses L as a generic count of individual SpheretNorm updates. To avoid conflict with the retained-state count L used here, we denote that generic update count by N. For the retained-state sequence 1,…,Lh_1,…,h_L, one has N=L−1N=L-1. Differential of a single SpheretNorm update. For each k=1,…,Nk=1,…,N, let ℱkF_k denote the mapping associated with the k-th SpheretNorm update, and let αk _k denote its step size. We assume that ℱkF_k is continuously differentiable in a neighborhood of kx_k. Throughout this subsection, all derivatives are taken with respect to kx_k, while p and αk _k are treated as constants. Define rk=‖k‖2r_k=\|x_k\|_2, k=−kk⊤rk2P_k=I- x_kx_k r_k^2, ck=k⊤ℱk(k)rk2c_k= x_k F_k(x_k)r_k^2, k=ℱk(k)−ckk=kℱk(k)z_k=F_k(x_k)-c_kx_k=P_kF_k(x_k), sk=‖k‖2s_k=\|z_k\|_2, and ηk=skrk,βk=αkηk,θk=parctan(βkp). _k= s_kr_k, _k= _k _k, _k=p ( _kp ). Let k=Dℱk(k)=∂ℱk(k)∂kA_k=DF_k(x_k)= _k(x_k) _k, k=k−ckB_k=A_k-c_kI, and define k=∂k+1∂kJ_k= _k+1 _k. For a perturbation ∈ℝd ξ ^d, differentiation of k=ℱk(k)−ckkz_k=F_k(x_k)-c_kx_k gives Dk[]=k−Dck[]k.Dz_k[ ξ]=B_k ξ-Dc_k[ ξ]x_k. Since kk=P_kx_k= 0, we immediately obtain kDk[]=kkP_kDz_k[ ξ]=P_kB_k ξ. We next compute the derivative of ckc_k. Direct differentiation gives Dck[]=⊤ℱk(k)+k⊤krk2−2k⊤ℱk(k)rk4k⊤.Dc_k[ ξ]= ξ F_k(x_k)+x_k A_k ξr_k^2-2 x_k F_k(x_k)r_k^4x_k ξ. Using ℱk(k)=k+ckkF_k(x_k)=z_k+c_kx_k and k=k+ckA_k=B_k+c_kI, the terms containing ckk⊤c_kx_k ξ cancel, yielding Dck[]=k⊤+k⊤krk2.Dc_k[ ξ]= z_k ξ+x_k B_k ξr_k^2. Suppose first that sk>0s_k>0. Since sk=‖k‖2s_k=\|z_k\|_2 and rk=‖k‖2r_k=\|x_k\|_2, we have ∇ksk=k⊤ksk _x_ks_k= B_k z_ks_k and ∇krk=krk _x_kr_k= x_kr_k. Therefore, ∇kβk=αk(k⊤krksk−skkrk3). _x_k _k= _k ( B_k z_kr_ks_k- s_kx_kr_k^3 ). Moreover, dθkdβk=p2p2+βk2 d _kd _k= p^2p^2+ _k^2, and hence ∇kθk=αkp2p2+βk2(k⊤krksk−skkrk3). _x_k _k= _kp^2p^2+ _k^2 ( B_k z_kr_ks_k- s_kx_kr_k^3 ). For sk>0s_k>0, the k-th SpheretNorm update is k+1=cos(θk)k+rksin(θk)kskx_k+1= ( _k)x_k+r_k ( _k) z_ks_k. Differentiating this expression and substituting the formula for Dck[]Dc_k[ ξ] yields k= _k= cos(θk)+sin(θk)rksk(kk⊤−kk⊤) ( _k)I+ ( _k)r_ks_k (z_kx_k -x_kz_k ) +rksin(θk)sk(k−kk⊤sk2)k + r_k ( _k)s_k (P_k- z_kz_k s_k^2 )B_k +(rkcos(θk)skk−sin(θk)k)(∇kθk)⊤. + ( r_k ( _k)s_kz_k- ( _k)x_k ) ( _x_k _k ) . Define k=cos(θk)+sin(θk)rksk(kk⊤−kk⊤)Q_k= ( _k)I+ ( _k)r_ks_k (z_kx_k -x_kz_k ). Then k=k+kJ_k=Q_k+E_k, where k= _k= rksin(θk)sk(k−kk⊤sk2)k r_k ( _k)s_k (P_k- z_kz_k s_k^2 )B_k +(rkcos(θk)skk−sin(θk)k)(∇kθk)⊤. + ( r_k ( _k)s_kz_k- ( _k)x_k ) ( _x_k _k ) . Estimate of the perturbation term. Since k−kk⊤sk2P_k- z_kz_k s_k^2 is the orthogonal projection onto the subspace of Tkrkd−1T_x_kS_r_k^d-1 orthogonal to kz_k, its operator norm is at most one. Therefore, ‖rksin(θk)sk(k−kk⊤sk2)k‖2≤rk|sin(θk)|sk‖k‖2. \| r_k ( _k)s_k (P_k- z_kz_k s_k^2 )B_k \|_2≤ r_k| ( _k)|s_k\|B_k\|_2. Since |βk|=|αk|skrk| _k|=| _k| s_kr_k, we obtain ‖rksin(θk)sk(k−kk⊤sk2)k‖2≤|αk||sin(θk)||βk|‖k‖2. \| r_k ( _k)s_k (P_k- z_kz_k s_k^2 )B_k \|_2≤| _k| | ( _k)|| _k|\|B_k\|_2. For the second term in kE_k, the orthogonality k⊤k=0x_k z_k=0 gives ‖rkcos(θk)skk−sin(θk)k‖22 \| r_k ( _k)s_kz_k- ( _k)x_k \|_2^2 =rk2cos2(θk)+rk2sin2(θk)=rk2. =r_k^2 ^2( _k)+r_k^2 ^2( _k)=r_k^2. Hence, ‖rkcos(θk)skk−sin(θk)k‖2=rk. \| r_k ( _k)s_kz_k- ( _k)x_k \|_2=r_k. Furthermore, ‖∇kθk‖2≤p2p2+βk2|αk|‖k‖2+|βk|rk. \| _x_k _k \|_2≤ p^2p^2+ _k^2 | _k|\|B_k\|_2+| _k|r_k. It follows that ‖k‖2≤δkexact\|E_k\|_2≤ _k^exact, where δkexact=|αk||sin(θk)||βk|‖k‖2+p2p2+βk2(|αk|‖k‖2+|βk|). _k^exact=| _k| | ( _k)|| _k|\|B_k\|_2+ p^2p^2+ _k^2 (| _k|\|B_k\|_2+| _k| ). The quotient |sin(θk)||βk| | ( _k)|| _k| is understood through its continuous extension at βk=0 _k=0. Since |sin(θk)|≤|θk|≤|βk|| ( _k)|≤| _k|≤| _k|, p2p2+βk2≤1 p^2p^2+ _k^2≤ 1, and |βk|=|αk|ηk| _k|=| _k| _k, we obtain δkexact≤|αk|(2‖k‖2+ηk). _k^exact≤| _k| (2\|B_k\|_2+ _k ). Define Δk=|αk|(2‖k‖2+ηk) _k=| _k| (2\|B_k\|_2+ _k ). Then ‖k‖2≤Δk\|E_k\|_2≤ _k. The zero-tangent case. We now consider sk=0s_k=0. At the point under consideration, k=z_k= 0, βk=0 _k=0. Direct differentiation of βk=αk‖k‖2rk _k= _k \|z_k\|_2r_k is not valid at k=z_k= 0, because the Euclidean norm is not differentiable at the origin. We therefore use the even-function representation established in Appendix D.4. Let ap(β)=cos(parctan(βp))a_p(β)= (p ( βp ) ) and bp(β)=sin(parctan(β/p))β,β≠0,1,β=0.b_p(β)= cases (p (β/p) )β,&β≠ 0,\\[5.16663pt] 1,&β=0. cases There exist real-analytic functions A^p A_p and B^p B_p, defined near zero, such that ap(β)=A^p(β2),bp(β)=B^p(β2),a_p(β)= A_p(β^2), b_p(β)= B_p(β^2), with A^p(0)=B^p(0)=1 A_p(0)= B_p(0)=1. For ≠x 0, define k()=(−⊤‖22)ℱk(),ωk()=αk2‖k()‖22‖22.z_k(x)= (I- xx \|x\|_2^2 )F_k(x), _k(x)= _k^2 \|z_k(x)\|_2^2\|x\|_2^2. The layer map can be written locally as k()=A^p(ωk())+αkB^p(ωk())k().T_k(x)= A_p ( _k(x) )x+ _k B_p ( _k(x) )z_k(x). For a perturbation ∈ℝd ξ ^d, Dωk(k)[]=2αk2rk2k⊤Dk[]−2αk2sk2rk4k⊤.D _k(x_k)[ ξ]= 2 _k^2r_k^2z_k Dz_k[ ξ]- 2 _k^2s_k^2r_k^4x_k ξ. When sk=0s_k=0, we have ωk(k)=0 _k(x_k)=0 and Dωk(k)[]=0D _k(x_k)[ ξ]=0. Consequently, Dk(k)[]=+αkDk[].DT_k(x_k)[ ξ]= ξ+ _kDz_k[ ξ]. Since k=z_k=0, one has ℱk(k)=ckkF_k(x_k)=c_kx_k. The derivative of ckc_k therefore reduces to Dck[]=k⊤krk2.Dc_k[ ξ]= x_k B_k ξr_k^2. Hence, Dk[] Dz_k[ ξ] =k−Dck[]k =B_k ξ-Dc_k[ ξ]x_k =k−kk⊤rk2k=kk. =B_k ξ- x_kx_k r_k^2B_k ξ=P_kB_k ξ. It follows that k=+αkkkJ_k=I+ _kP_kB_k when sk=0s_k=0. Define δk(0)=|αk|‖kk‖2 _k^(0)=| _k|\|P_kB_k\|_2. Weyl’s singular-value perturbation inequality gives σmax(k)≤1+δk(0) _ (J_k)≤ 1+ _k^(0) and σmin(k)≥max0,1−δk(0). _ (J_k)≥ \0,1- _k^(0)\. Moreover, δk(0)≤|αk|‖k‖2≤Δk _k^(0)≤| _k|\|B_k\|_2≤ _k. Singular-value bounds. For sk>0s_k>0, define 1,k=krke_1,k= x_kr_k, 2,k=kske_2,k= z_ks_k. On span1,k,2,kspan\e_1,k,e_2,k\, the matrix of kQ_k, with respect to this orthonormal basis, is (cos(θk)−sin(θk)sin(θk)cos(θk)). pmatrix ( _k)&- ( _k)\\ ( _k)& ( _k) pmatrix. Thus, kQ_k acts as an isometry on this two-dimensional subspace. On its orthogonal complement, kQ_k acts as multiplication by cos(θk) ( _k). Consequently, we obtain that ‖k‖2=1\|Q_k\|_2=1 and σmin(k)≥|cos(θk)| _ (Q_k)≥| ( _k)|. For sk>0s_k>0, Weyl’s inequality therefore gives σmax(k)≤1+Δk _ (J_k)≤ 1+ _k and σmin(k)≥max0,|cos(θk)|−Δk. _ (J_k)≥ \0,| ( _k)|- _k\. Define qk=|cos(θk)|,sk>0,1,sk=0,q_k= cases| ( _k)|,&s_k>0,\\ 1,&s_k=0, cases and mk=max0,qk−Δkm_k= \0,q_k- _k\. Combining the cases sk>0s_k>0 and sk=0s_k=0, we obtain σmax(k)≤1+Δk,σmin(k)≥mk,k=1,…,N. _ (J_k)≤ 1+ _k, _ (J_k)≥ m_k, k=1,…,N. Depth-wise Jacobian. The end-to-end Jacobian over the retained residual stream is =∂L∂1=∂N+1∂1.G= _L _1= _N+1 _1. Since N=L−1N=L-1, the chain rule gives =NN−1⋯1=L−1L−2⋯1G=J_NJ_N-1·sJ_1=J_L-1J_L-2·sJ_1. By submultiplicativity, σmax() _ (G) ≤∏k=1Nσmax(k)≤∏k=1N(1+Δk) ≤ _k=1^N _ (J_k)≤ _k=1^N(1+ _k) ≤exp(∑k=1NΔk). ≤ ( _k=1^N _k ). Equivalently, σmax()≤exp(∑k=1L−1Δk) _ (G)≤ ( _k=1^L-1 _k ). Similarly, σmin()≥∏k=1Nσmin(k)≥∏k=1Nmk=∏k=1L−1mk. _ (G)≥ _k=1^N _ (J_k)≥ _k=1^Nm_k= _k=1^L-1m_k. A consequence of exact norm preservation. Since ‖k+1‖22=‖k‖22\|x_k+1\|_2^2=\|x_k\|_2^2 for every admissible kx_k, differentiation with respect to kx_k yields k⊤k+1=kJ_k x_k+1=x_k. Therefore, ‖k‖2=‖k⊤‖2≥‖k⊤k+1‖2‖k+1‖2=‖k‖2‖k+1‖2=1.\|J_k\|_2=\|J_k \|_2≥ \|J_k x_k+1\|_2\|x_k+1\|_2= \|x_k\|_2\|x_k+1\|_2=1. Thus, an individual SpheretNorm update Jacobian cannot be a strict contraction in every ambient direction, although it may still strongly contract particular tangential directions. D.8 Angle Control Alone Is Insufficient The intrinsic rotation angle controls the geometric component of the update Jacobian, but it does not control the differential sensitivity of the underlying mapping. The following example shows that even zero angular displacement provides no uniform upper bound on the update-Jacobian norm and does not guarantee nonsingularity. Proposition 9. For every R>1R>1 and every α>0α>0, there exist a linear mapping ℱ:ℝ2→ℝ2F:R^2 ^2 and a hidden state ∈1h ^1 such that the corresponding SpheretNorm update satisfies β=0β=0 and θ(p)=0θ^(p)=0, while its update Jacobian satisfies ‖2=R\|J\|_2=R. Proof. Let =1=(10)h=e_1= pmatrix1\\ 0 pmatrix and define ℱ()=F(x)=Ax and =(μ00μ+K)A= pmatrixμ&0\\ 0&μ+K pmatrix, where μ,K∈ℝμ,K . When =1h=e_1, we have ℱ()=μ1F(h)= _1 and c=⊤ℱ()=μc=h F(h)=μ. Therefore, =−c=(000K).B=A-cI= pmatrix0&0\\ 0&K pmatrix. Moreover, =−⊤=(0001)P=I-hh = pmatrix0&0\\ 0&1 pmatrix, and hence =ℱ()=z=PF(h)= 0. Therefore, β=0β=0 and θ(p)=0θ^(p)=0 for every p>0p>0. Since =z= 0, the zero-tangent formula established in Appendix D.7 gives =+α=(1001+αK).J=I+ = pmatrix1&0\\ 0&1+α K pmatrix. Choosing K=R−1αK= R-1α gives =(100R)J= pmatrix1&0\\ 0&R pmatrix, and therefore ‖2=R\|J\|_2=R. ∎ The same construction also shows that zero angular displacement does not imply nonsingularity. Indeed, choosing K=−1αK=- 1α gives =(1000)J= pmatrix1&0\\ 0&0 pmatrix, so that σmin()=0 _ (J)=0. Therefore, angular control must be supplemented by control of the update-sensitivity term |α|‖2|α|\|B\|_2. D.9 Dyadic Angular Budget of the Depth Schedules The global estimates in Proposition 8 compare the accumulated stability envelopes over the entire sequence of SpheretNorm updates. We now examine how the available angular motion is distributed across different depth scales. Under the retained-state indexing 1,2,…,Lh_1,h_2,…,h_L, the individual SpheretNorm updates are indexed by l=1,…,L−1l=1,…,L-1. For an integer m≥1m≥ 1 satisfying 2m≤L2m≤ L, define the dyadic update interval Im=m,m+1,…,2m−1I_m=\m,m+1,…,2m-1\. The condition 2m≤L2m≤ L ensures that Im⊆1,…,L−1I_m \1,…,L-1\. Proposition 10. Fix an optimization iteration τ and let p>0p>0. Suppose that there exist constants 0<H−≤H+<∞0<H_-≤ H_+<∞ and ε>0 >0 such that, for every l∈Iml∈ I_m, H−≤al(τ)ηl(τ)≤H+ and |βl(τ)|≤ε.H_-≤ a_l^(τ) _l^(τ)≤ H_+ and | _l^(τ) |≤ . Then there exist constants C−,C+>0C_-,C_+>0, independent of m and L, such that C−∑l∈Imλl2≤∑l∈Im|θl,τ(p)|2≤C+∑l∈Imλl2.C_- _l∈ I_m _l^2≤ _l∈ I_m | _l,τ^(p) |^2≤ C_+ _l∈ I_m _l^2. Consequently, ∑l∈Im|θl,τ(p)|2=Θ(1) _l∈ I_m | _l,τ^(p) |^2= (1) for the layerwise square-root schedule λl=l−1/2 _l=l^-1/2, while ∑l∈Im|θl,τ(p)|2=Θ(1m) _l∈ I_m | _l,τ^(p) |^2= ( 1m) for the harmonic schedule λl=l−1 _l=l^-1. Proof. Define ϕp(t)=parctan(t/p)t,t>0,1,t=0. _p(t)= cases p (t/p)t,&t>0,\\[8.53581pt] 1,&t=0. cases The function ϕp _p is continuous and strictly positive on the compact interval [0,ε][0, ]. Hence there exists a constant cp,ε=min0≤t≤εϕp(t)>0c_p, = _0≤ t≤ _p(t)>0 such that cp,εt≤parctan(tp)≤t,0≤t≤ε.c_p, t≤ p ( tp )≤ t, 0≤ t≤ . Applying this estimate with t=|βl(τ)|t=| _l^(τ)| gives cp,ε|βl(τ)|≤|θl,τ(p)|≤|βl(τ)|c_p, | _l^(τ) |≤ | _l,τ^(p) |≤ | _l^(τ) |. Since |βl(τ)|=λlal(τ)ηl(τ) | _l^(τ) |= _la_l^(τ) _l^(τ), the assumptions imply cp,εH−λl≤|θl,τ(p)|≤H+λlc_p, H_- _l≤ | _l,τ^(p) |≤ H_+ _l. Squaring and summing over ImI_m yields cp,ε2H−2∑l∈Imλl2≤∑l∈Im|θl,τ(p)|2≤H+2∑l∈Imλl2.c_p, ^2H_-^2 _l∈ I_m _l^2≤ _l∈ I_m | _l,τ^(p) |^2≤ H_+^2 _l∈ I_m _l^2. Thus, the first assertion holds with C−=cp,ε2H−2C_-=c_p, ^2H_-^2 and C+=H+2C_+=H_+^2. For the layerwise square-root schedule λl=l−1/2 _l=l^-1/2, we have ∑l=m2m−1λl2=∑l=m2m−11l. _l=m^2m-1 _l^2= _l=m^2m-1 1l. Since m≤l≤2m−1m≤ l≤ 2m-1 for every l∈Iml∈ I_m, we have 12m−1≤1l≤1m 12m-1≤ 1l≤ 1m. Since ImI_m contains exactly m indices, it follows that m2m−1≤∑l=m2m−11l≤1 m2m-1≤ _l=m^2m-1 1l≤ 1. Hence, ∑l=m2m−1λl2=Θ(1). _l=m^2m-1 _l^2= (1). For the harmonic schedule λl=l−1 _l=l^-1, we have ∑l=m2m−1λl2=∑l=m2m−11l2 _l=m^2m-1 _l^2= _l=m^2m-1 1l^2. Using again m≤l≤2m−1m≤ l≤ 2m-1, we obtain 1(2m−1)2≤1l2≤1m2 1(2m-1)^2≤ 1l^2≤ 1m^2. Therefore, m(2m−1)2≤∑l=m2m−11l2≤1m. m(2m-1)^2≤ _l=m^2m-1 1l^2≤ 1m. Since m(2m−1)2≥14m m(2m-1)^2≥ 14m, we conclude that 14m≤∑l=m2m−1λl2≤1m. 14m≤ _l=m^2m-1 _l^2≤ 1m. Hence, ∑l=m2m−1λl2=Θ(1m). _l=m^2m-1 _l^2= ( 1m ). The stated conclusions follow. ∎ Proposition 10 distinguishes the distribution of squared angular motion from the global worst-case stability envelope. The harmonic schedule gives the smaller global accumulated-sensitivity bound, but allocates progressively less squared angular motion to deeper dyadic update intervals. By contrast, the layerwise square-root schedule retains an approximately constant squared angular budget across logarithmic scales of update depth, allowing the transformations associated with deeper SpheretNorm updates to remain nontrivial.