Paper deep dive
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 5:49:05 AM
Summary
The paper introduces Mean Root Square Normalization (MRSNorm), a novel normalization technique that pairs channels into 2D phasors to compute localized L2 magnitudes before aggregating via a global L1 average. This approach inverts traditional scaling paradigms, preserving conformal invariance and phase information while halving the number of learnable parameters. MRSNorm provides built-in gradient homogeneity and stability, preventing numerical explosion and gradient starvation under extreme hyperparameter settings, as demonstrated on ResNet with CIFAR-100.
Entities (8)
Relation Signals (7)
MRSNorm → reducesparametersby → 50%
confidence 95% · By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters
MRSNorm → usesmechanism → Phasor
confidence 95% · By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm
MRSNorm → ensuresproperty → Gradient Homogeneity
confidence 92% · We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper... to ensure Gradient Homogeneity.
MRSNorm → appliedto → CIFAR-100
confidence 90% · Empirical evaluations on a ResNet with CIFAR-100
ResNet → evaluatedwith → MRSNorm
confidence 90% · Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability
MRSNorm → improvesstabilityover → RMSNorm
confidence 90% · MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories where standard normalizations suffer from gradient divergence.
MRSNorm → improvesstabilityover → LayerNorm
confidence 90% · MRSNorm maintains a stable gradient flow... while LN and RMS suffer from catastrophic optimization collapse.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.
Tags
Links
- Source: https://arxiv.org/abs/2607.17822v1
- Canonical: https://arxiv.org/abs/2607.17822v1
Trouble viewing inline? Open PDF directly →
Full Text
37,262 characters extracted from source content.
Expand or collapse full text
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation Sungwoo Goo College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea Hwi-yeol Yun College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea Sangkeun Jung Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea Abstract While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars (∑x2Σ x^2) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized L2L_2 magnitudes (Root Square) before aggregating them via a global L1L_1 average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C. 1 Introduction Normalization layers are the indispensable backbone of modern deep learning, essential for mitigating internal covariate shift and stabilizing the training of exceedingly deep networks. While Root Mean Square Normalization (RMSNorm) [11] has been widely adopted in Large Language Models (LLMs) for its computational efficiency over Layer Normalization (LN) [1], it has a critical structural vulnerability: the quadratic accumulation of variances (∑x2Σ x^2). In mixed-precision environments, a single magnitude outlier can disproportionately dominate the denominator, triggering catastrophic numerical explosions and inducing gradient starvation for remaining channels. To address these limitations, we propose Mean Root Square Normalization (MRSNorm). MRSNorm mathematically inverts the order of operations. By pairing adjacent channels into 2D2D phasors, MRSNorm first computes localized L2L_2 magnitudes (Root Square) and subsequently aggregates them via a global L1L_1 average (Mean). MRSNorm is also parameter-efficient by sharing a single affine weight (γ) across 2D2D phasor components, it halves the total learnable parameters, eliminating the redundant degrees of freedom that distort phase. Crucially, as derived in Section 5.1, the Pythagorean identity (cos2θ+sin2θ≡1 ^2θ+ ^2θ≡ 1) ensures that the local gradient update magnitude for each phasor is equalized. This structurally guaranteed Gradient Homogeneity functions as a built-in gradient clipper, preventing starvation without distorting the directional optimization trajectory. 2 Related Works The quest for stable, efficient deep learning models has driven the continuous evolution of normalization techniques and normalization-free architectures. Our proposed MRSNorm stands at the intersection of geometric phase-coding and robust gradient stabilization, addressing the limitations of prior methodologies. 2.1 Evolution of Normalization Layers Layer Normalization (LN) [1] has been the cornerstone of Transformer architectures, ensuring stable forward dynamics by normalizing activations across the channel dimension with mean-centering and variance scaling. However, the computational overhead of calculating the mean led to the development of RMSNorm [11]. RMSNorm demonstrated that the mean-centering operation is largely dispensable for success, effectively accelerating training throughput. Consequently, RMSNorm has become the standard for modern Large Language Models (LLMs). Despite its computational advantages, RMSNorm introduces a structural vulnerability: the unbounded quadratic accumulation of channel magnitudes (∑x2Σ x^2). This mechanism makes the denominator highly susceptible to activation outliers, resulting in frequent numerical explosions (NaNs) and gradient starvation for low-magnitude channels [8]. MRSNorm resolves this by inverting the operation sequence to a linear mean of localized L2L_2 phasors, eliminating the risk of catastrophic variance explosion while retaining the zero-centering computational efficiency of RMSNorm. 2.2 Phase and Complex-Valued Representations The encoding of information using phase and complex numbers has a history in signal processing and deep learning. Complex-valued Neural Networks [9] attempted to leverage phase information by substituting real numbers with complex arithmetic. However, without stringent geometric constraints on the amplitude manifold, these networks effectively defaulted to 2-channel real-valued operations, failing to mitigate the vanishing and exploding gradient problems. A notable attempt to preserve directional information was the Squashing function introduced in Capsule Networks [6], which maps the vector length to a range between 0 and 1 while preserving its orientation. However, as we theoretically analyzed in Section LABEL:sec:amp_block, instance-wise amplitude normalization (e.g., Squashing or direct unit-norm projection) inherently suffers from a Radial Gradient Blockade, where the Jacobian becomes singular in the direction of the vector’s magnitude, thereby stifling learning signals in deep architectures. More recently, RoPE [7] successfully introduced phase-domain rotations to encode relative positional information without altering the vector norm. but this approach is limited to positional encoding and does not address the broader issues of gradient starvation and numerical instability in deep networks. Unlike individual-based squashing or projection methods, MRSNorm employs an ensemble-level homeostatic scaling, preserving the essential radial gradient pathways while enforcing an gradient homogeneity across network depths. 2.3 Gradient Clipping and Optimization Stability Gradient clipping [5] remains a ubiquitous heuristic to prevent gradient explosion in deep and recurrent architectures. However, traditional magnitude-based clipping blindly truncates the gradient vectors, distorting the critical directional and angular information essential for accurate geometric optimization [2]. Alternatively, adaptive gradient scaling methods (e.g., AdamW [4]) attempt to smooth the optimization trajectory but frequently suffer from a generalization gap compared to Stochastic Gradient Descent (SGD) [10], as they tend to warp the update directions of multidimensional features anisotropically. MRSNorm introduces a fundamentally different paradigm to address this optimization instability. Rather than relying on heuristic, we mathematically enforce Gradient Homogeneity at the microscopic phasor level. As we formally derive in Section 5.1, our 2D2D phasor grouping guarantees that the local gradient update magnitude of each individual phasor is unconditionally equalized, rendering the gradient scale invariant to local coordinate magnitudes. This structural guarantee ensures the gradient flow across all channels, preventing gradient starvation and paving the way for smooth convergence into flat, robust minima under deeply-layered conditions. 3 Methods Given an input feature vector ∈ℝNh ^N (where N is even), we restructure the feature space into P=N/2P=N/2 pairs of 2D phasors. Let i=[xi,yi]T∈ℝ2z_i=[x_i,y_i]^T ^2 denote the i-th phasor pair for i∈1,…,Pi∈\1,…,P\. MRSNorm deliberately inverts the traditional scaling paradigm. Rather than calculating a global quadratic mean of individual scalars, we first compute the localized L2L_2 magnitude (Root Square) for each phasor, followed by a global L1L_1 average (Mean) across all P pairs: SMRS=1P∑i=1P‖i‖2+ϵ=1P∑i=1Pxi2+yi2+ϵS_MRS= 1P _i=1^P\|z_i\|_2+ε= 1P _i=1^P x_i^2+y_i^2+ε (1) where ϵε is a small constant added for numerical stability. The normalized phasor outputs ^i=[x^i,y^i]T z_i=[ x_i, y_i]^T are then obtained via isotropic vector scaling: ^i=γiiSMRS z_i= _i z_iS_MRS (2) where γi∈ℝ _i is a affine parameter applied uniformly to the i-th phasor pair. By enforcing a unified scale factor (SMRS_MRS) and sharing the affine weight (γi _i) across the 2D2D plane, MRSNorm strictly preserves the exact phase angle (θi _i) of the original phasor: θ(^i)=arctan(y^ix^i)=arctan(yixi)=θiθ( z_i)= ( y_i x_i )= ( y_ix_i )= _i (3) This design choice not only maintains conformal invariance but also intrinsically halves the total number of affine parameters (N/2N/2) compared to standard normalizations. Consequently, MRSNorm eradicates the harmful degrees of freedom that traditionally distort geometric phase information, proving that unconstrained spatial scaling is a redundant source of instability. 4 Experiments Figure 1: Empirical evaluation of training stability under extreme hyperparameter regimes. We compare MRSNorm (MRS, blue) against standard LayerNorm (LN, orange) and RMSNorm (RMS, green) across varying learning rates (lr∈0.1,0.3,0.4lr∈\0.1,0.3,0.4\) and batch sizes (bs∈128,64,32bs∈\128,64,32\). Line styles represent batch sizes (solid: 128128, dashed: 6464, dotted: 3232), and shaded regions denote variance. While LN and RMS suffer from catastrophic optimization collapse at higher learning rates (lr≥0.3lr≥ 0.3), MRSNorm maintains a stable gradient flow, successfully converging for a batch size of 128128 at lr=0.3lr=0.3. Notably, under the most extreme setting (lr=0.4lr=0.4), MRSNorm is the only method that prevents immediate divergence. Robustness Under Extreme Optimization Regimes To empirically validate the theoretical stability of the phasor manifold introduced by MRSNorm, we designed a series of stress-test benchmarks. The primary objective is to evaluate the resilience of normalization methods against severe optimization noise, exacerbated by high learning rates and small batch sizes. Experimental Setup. We trained a ResNet model on the CIFAR-100 dataset [3], varying the learning rate lr∈0.1,0.3,0.4lr∈\0.1,0.3,0.4\ and batch size bs∈128,64,32bs∈\128,64,32\. We employed the SGD optimizer with a momentum of 0.90.9 and weight decay of 5e−45e^-4. We utilized 10 warmup epochs and applied cosine annealing for the learning rate schedule. Each configuration was repeated across three random seeds (42, 43, 44) to ensure statistical significance, with the mean and variance of the training accuracy recorded. We compared our proposed MRSNorm against two ubiquitous baseline normalizations: LayerNorm (LN) and RMSNorm (RMS). All models were trained from scratch for 100 epochs. Results in Standard Regimes (lr=0.1lr=0.1). As illustrated in the top panel of Figure 1, under a standard learning rate (lr=0.1lr=0.1), all normalization methods converge effectively at larger batch sizes (bs∈64,128bs∈\64,128\). However, when the batch size is reduced to bs=32bs=32 (dotted lines)—thereby introducing significant stochastic noise into the gradient estimates—the structural vulnerability of 1D scalar normalization methods becomes apparent. RMSNorm exhibits severe instability with high variance (indicated by the green shaded area) and fails to optimize effectively. In contrast, MRSNorm maintains a smooth and stable training trajectory, outperforming both LN and RMS. Catastrophic Collapse of Baselines in Extreme Regimes (lr≥0.3lr≥ 0.3). The fundamental geometric superiority of MRSNorm is decisively demonstrated when optimization conditions are pushed to extreme limits. As illustrated in Figure 1 (lr=0.3lr=0.3), both LayerNorm and RMSNorm suffer from catastrophic optimization collapse. Remarkably, MRSNorm remains robust under these severe hyperparameter regimes. It successfully optimizes with a batch size of 128128, and even though the training curves exhibit oscillations at smaller batch sizes (bs=64bs=64 and bs=32bs=32), several seeds still manage to converge. In contrast, the baseline normalization methods fail entirely, with their training accuracy immediately flattening near zero. Even at the extreme learning rate of lr=0.4lr=0.4—four times the standard baseline setting—MRSNorm sustains initial gradient flow. Although configurations with smaller batch sizes eventually diverge before 30 epochs, MRSNorm demonstrates significant endurance. Conversely, the other normalization methods collapse instantaneously, failing to survive beyond the first few epochs. 5 Discussion 5.1 The Paradox of Constraint: Analytical Contrast of RMSNorm and MRSNorm Gradients To mathematically unveil why MRSNorm sustains identical representational capacity and optimization stability despite having exactly halved affine parameters, we conduct a comparative analysis of the backpropagation dynamics of standard RMSNorm and our proposed MRSNorm. Under standard RMSNorm, which operates on independent 1D scalar channels xi∈ℝx_i , the gradient of the loss ℒL with respect to the input xix_i is formulated as: ∂ℒ∂xi=1SRMS[∇oiγi−^(1N∑k=1Nγk∇okx^k)] ∂ x_i= 1S_RMS [∇ o_i _i- x_i ( 1N _k=1^N _k∇ o_k x_k ) ] (4) where N=2PN=2P is the number of independent channels. In contrast, under the 2D2D phasor-based MRSNorm, the gradient vector of the loss with respect to the input phasor i=[xi,yi]T∈ℝ2z_i=[x_i,y_i]^T ^2 is expressed as: ∂ℒ∂i=1SMRS[∇iγi−i(1P∑k=1Pγk(∇k⋅^k))] _i= 1S_MRS [ _i _i-u_i ( 1P _k=1^P _k ( _k· z_k ) ) ] (5) where i=[cosθi,sinθi]Tu_i=[ _i, _i]^T is the unit direction vector of the i-th phasor, and ⋅· represents the standard Euclidean dot product in ℝ2R^2. The comparative contrast between Equation 4 and Equation 5 reveals three numerical-analytical insights: First, we identify a Global-Local Paradox in gradient scale propagation. In standard RMSNorm, the scalar gradient update is scaled by the raw, normalized coordinate magnitude x_i. If xix_i acts as an unbounded outlier, its gradient scales proportionally, causing it to monopolize the backpropagated learning signal and starve the remaining channels. Conversely, if xix_i falls near the origin, its gradient collapses, leading to local representation underflow. Let Ω=1P∑k=1Pγk(∇k⋅^k)∈ℝ = 1P _k=1^P _k ( _k· z_k ) denote the global gradient projection scalar, which represents the ensemble-averaged error projection onto the phasor manifold. In MRSNorm, the unstable and outlier-prone multiplier x^i x_i of RMSNorm is replaced by the unit direction vector iu_i. Because ‖i‖2=cos2θi+sin2θi≡1\|u_i\|_2= ^2 _i+ ^2 _i≡ 1 unconditionally, the L2L_2 norm of the local gradient interference vector is mathematically guaranteed to be uniform and invariant to the phase angle θi _i: ‖ΩSMRSi‖2=|Ω|SMRS \| S_MRSu_i \|_2= | |S_MRS (6) This local isotropic normalization ensures that every single 2D2D phasor, regardless of its raw activation scale or angular orientation, receives an identical magnitude of gradient updates. This Gradient Homogeneity proactively prevents both local gradient starvation and scale explosion. Second, MRSNorm guarantees Conformal Gradient Collinearity. Because the subtraction term in Equation 5 is multiplied directly by the unit vector iu_i, the scale adjustment is applied strictly parallel to the input phasor iz_i itself: i⋅Scale∥iu_i·Scale _i (7) Subtracting a collinear vector from a 2D2D phasor strictly alters its radial magnitude (amplitude) while leaving its phase angle (θi _i) mathematically decoupled. Standard RMSNorm, by scaling coordinates independently (γx,i≠γy,i _x,i≠ _y,i), deforms the circular aspect ratio into an ellipse, corrupting the phase. Third, this geometric purity explains why MRSNorm matches the performance of RMSNorm despite having halved affine parameters. The independent scaling of x and y in standard norms is not an asset, but a harmful redundancy that merely serves to inject phase-domain noise. By eliminating this redundant parameterization and restricting the update to a conformal, phase-preserving manifold, MRSNorm simplifies the optimization landscape, allowing the model to converge into flatter, more robust minima with half the parameter footprint. 5.2 Mathematical Foundations: Why 2D2D Phasor Pairing is a Structural Necessity Our architecture relies on two mathematical pillars: why phase-stable normalization is impossible in 1D1D, and why strict amplitude normalization fails via radial gradient blockade. First, we prove that stable periodic gradient propagation is structurally impossible in 1D1D real analysis. Let f:ℝ→ℝf:R be a continuously differentiable, periodic activation function with a constant gradient norm |f′(x)|=c>0|f (x)|=c>0. By the Intermediate Value Theorem, a continuous derivative with a constant absolute value must be either c or −c-c. Integrating this yields a linear function (f(x)=±cx+df(x)=± cx+d), which is inherently non-periodic. Thus, no 1D1D scalar channel can simultaneously exhibit periodicity and gradient stability. MRSNorm overcomes this bottleneck by ascending to the 2D2D phasor space ℝ2R^2. By pairing channels, we exploit the Pythagorean identity ‖i‖2=cos2θi+sin2θi≡1\|u_i\|_2= ^2 _i+ ^2 _i≡ 1, allowing individual components to behave as smooth, periodic functions while maintaining a strictly constant update norm. Second, we address why we must avoid strict instance-wise projection onto the unit circle. Let ^i=i/‖i‖2 z_i=z_i/\|z_i\|_2. The Jacobian i=∂^i∂i=1‖i‖2[−iiT]J_i= ∂ z_i _i= 1\|z_i\|_2[I-u_iu_i^T] contains a rank-1 projection matrix i=−iiTP_i=I-u_iu_i^T. Since ii=P_iu_i=0, any backpropagating error signal along the radial direction is identically annihilated (∇^iTii=0∇ z_i^TJ_iu_i=0). This Radial Gradient Blockade renders deep training impossible by severing the optimization path for amplitude parameters. MRSNorm resolves this via ensemble-based scaling. By normalizing with the global L1L_1 average magnitude SMRS_MRS rather than the instance-specific ‖i‖2\|z_i\|_2, the Jacobian takes the form: ∂^i∂i=1SMRS[−1Pi^iT] ∂ z_i _i= 1S_MRS [I- 1Pu_i z_i^T ] (8) The projection term is scaled by 1/P1/P, leaving the radial gradient unblocked and stable as P increases. Consequently, 2D2D phasor pairing is not merely a design choice, but the minimal viable dimension to preserve both phase periodicity and radial gradient flow, providing the theoretically sound foundation. 5.3 Revisiting Attention Mechanisms: Geometric Resolution of Logit Imbalance via MRSNorm Standard scaled dot-product attention fundamentally relies on the dot product between queries (q) and keys (k). When specific tokens disproportionately expand their magnitudes, they dominate the attention logits regardless of semantic direction, driving the softmax function into a near one-hot state (softmax collapse) and causing severe gradient vanishing. By integrating an affine-free MRSNorm immediately after the linear projections (QK-MRSNorm), we introduce a soft geometric constraint that alleviates this issue. MRSNorm binds the pairwise channels into a phasor manifold. To understand its geometric implication, consider a query vector decomposed into d/2d/2 phasor blocks, where each 2D block iq_i has a local magnitude rq,i=‖i‖2r_q,i=\|q_i\|_2 and a phase. MRSNorm normalizes these blocks by the global Mean Root Square energy SqS_q. Thus, the attention score between a query and a key evaluates to: Score=∑i=1d/2^i⋅^i=∑i=1d/2(rq,i⋅rk,iSq⋅Sk)cos(Δθi)Score= _i=1^d/2 q_i· k_i= _i=1^d/2 ( r_q,i· r_k,iS_q· S_k ) ( _i) (9) This mathematical equivalence reveals an architectural property: QK-MRSNorm implicitly transforms the standard dot product into an Energy-Weighted Cosine Similarity Operator. Specifically, the term cos(Δθi) ( _i) captures the pure semantic phase alignment, while the magnitude ratio (rq,irk,i)/(SqSk)(r_q,ir_k,i)/(S_qS_k) acts as an auto-gating mechanism. The network learns to dynamically allocate magnitude to highly informative phasor blocks, amplifying their vote in the final score while suppressing irrelevant features. Furthermore, the algebraic derivation demonstrating how MRSNorm disentangles the standard dot product into an energy-weighted cosine similarity is provided in Appendix A. A formal mathematical proof establishing the strict numerical upper bound of this operator—and detailing its localized geometric superiority over QK-RMSNorm via the semantic phase veto—is presented in Appendix B. Together, these appendices theoretically validate both the enhanced representational capacity and the guaranteed numerical stability of the proposed QK-MRSNorm framework. 6 Conclusion While this study conclusively demonstrates the theoretical and empirical robustness of MRSNorm, our evaluation primarily focuses on rigorous stress-testing of optimization dynamics under extreme hyperparameters (e.g., high learning rates and small batch sizes). These micro-benchmarks are essential for proving the fundamental stability of the phasor manifold and the preservation of Jacobian isometry. However, scaling this architectural modification to billion-parameter LLMs inherently requires substantial computational clusters. Consequently, an empirical analysis of MRSNorm’s impact on pre-training dynamics at a massive scale remains outside the scope of this work. The introduction of MRSNorm opens promising avenues for future research, most notably in redesigning attention routing mechanisms. As discussed in Section 5.3, applying an affine-free MRSNorm directly to the queries and keys (QK-MRSNorm) mathematically transforms the standard scaled dot-product into an Energy-Weighted Cosine Similarity Operator. We theoretically demonstrated that this formulation intrinsically resolves logit imbalance and acts as an auto-gating mechanism via localized magnitude variations (rq,irk,ir_q,ir_k,i). We strongly invite the community to empirically explore this QK-MRSNorm architecture in large-scale Transformer variants and state-space models. Validating whether this geometric soft constraint can prevent softmax collapse in long-context attention will be a critical next step. We believe the geometric stability provided by MRSNorm will serve as a foundational building block for the next generation of robust neural architectures. References [1] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §1, §2.1. [2] X. Chen, S. Z. Wu, and M. Hong (2020) Understanding gradient clipping in private sgd: a geometric perspective. Advances in neural information processing systems 33, p. 13773–13782. Cited by: §2.3. [3] A. Krizhevsky, G. Hinton, et al. (2009) Learning multiple layers of features from tiny images. Cited by: §4. [4] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §2.3. [5] R. Pascanu, T. Mikolov, and Y. Bengio (2013) On the difficulty of training recurrent neural networks. In International conference on machine learning, p. 1310–1318. Cited by: §2.3. [6] S. Sabour, N. Frosst, and G. E. Hinton (2017) Dynamic routing between capsules. Advances in neural information processing systems 30. Cited by: §2.2. [7] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §2.2. [8] M. Sun, X. Chen, J. Z. Kolter, and Z. Liu (2024) Massive activations in large language models. arXiv preprint arXiv:2402.17762. Cited by: §2.1. [9] C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, J. F. Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and C. J. Pal (2017) Deep complex networks. arXiv preprint arXiv:1705.09792. Cited by: §2.2. [10] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht (2017) The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems 30. Cited by: §2.3. [11] B. Zhang and R. Sennrich (2019) Root mean square layer normalization. Advances in neural information processing systems 32. Cited by: §1, §2.1. Appendix A Appendix: Mathematical Derivation of Energy-Weighted Cosine Similarity via MRSNorm In Section 5.3, we proposed that applying an affine-free MRSNorm to queries and keys (QK-MRSNorm) inherently transforms the standard dot-product attention into an energy-weighted cosine similarity operator. This appendix provides the rigorous step-by-step mathematical derivation of this equivalence and details its optimization advantages over strict L2L_2 normalization. A.1 Step 1: Phasor Decomposition and Polar Representation Let ,∈ℝdq,k ^d be a query and a key vector, respectively, where the hidden dimension d is an even number. MRSNorm processes channels in pairwise blocks. We can decompose the query vector into d/2d/2 two-dimensional blocks, i∈ℝ2q_i ^2 for i∈1,2,…,d/2i∈\1,2,…,d/2\. Each 2D block can be uniquely represented in polar coordinates. Let rq,i=‖i‖2r_q,i=\|q_i\|_2 be the local magnitude (energy) of the block, and θq,i _q,i be its phase angle. Thus, i=[rq,icosθq,irq,isinθq,i],i=[rk,icosθk,irk,isinθk,i]q_i= bmatrixr_q,i _q,i\\ r_q,i _q,i bmatrix, _i= bmatrixr_k,i _k,i\\ r_k,i _k,i bmatrix (10) A.2 Step 2: Global RMS Normalization (Soft Constraint) MRSNorm calculates the global Root Mean Square (RMS) energy across all blocks. For the query vector, the scaling factor SqS_q is defined as: Sq=1d/2∑i=1d/2‖i‖22+ϵ=1d/2∑i=1d/2rq,i2+ϵS_q= 1d/2 _i=1^d/2\|q_i\|_2^2+ε= 1d/2 _i=1^d/2r_q,i^2+ε (11) The normalized vector q consists of blocks ^i=iSq q_i= q_iS_q. Notice that each block is scaled by the global energy SqS_q, not its local energy rq,ir_q,i. This is the fundamental distinction between MRSNorm’s soft constraint and strict L2L_2 projection. A.3 Step 3: Dot Product Expansion and Equivalence The attention score before applying softmax is the dot product of the normalized vectors q and k: Score=^⋅^=∑i=1d/2^i⋅^i=∑i=1d/2(iSq)⋅(iSk)=1SqSk∑i=1d/2i⋅iScore= q· k= _i=1^d/2 q_i· k_i= _i=1^d/2 ( q_iS_q )· ( k_iS_k )= 1S_qS_k _i=1^d/2q_i·k_i (12) Expanding the dot product of the individual 2D blocks using their polar representations: i⋅i _i·k_i =(rq,icosθq,i)(rk,icosθk,i)+(rq,isinθq,i)(rk,isinθk,i) =(r_q,i _q,i)(r_k,i _k,i)+(r_q,i _q,i)(r_k,i _k,i) =rq,irk,i(cosθq,icosθk,i+sinθq,isinθk,i) =r_q,ir_k,i ( _q,i _k,i+ _q,i _k,i ) =rq,irk,icos(θq,i−θk,i) =r_q,ir_k,i ( _q,i- _k,i) (13) Let Δθi=θq,i−θk,i _i= _q,i- _k,i be the phase difference (semantic alignment) between the query and key for the i-th micro-head. Substituting Eq. A.3 back into Eq. 12, we arrive at the exact formulation presented in the main text: Score=∑i=1d/2(rq,iSqrk,iSk)⏟Energy Weight Wi⋅cos(Δθi)⏟Phase AlignmentScore= _i=1^d/2 ( r_q,iS_q r_k,iS_k )_Energy Weight W_i· ( _i)_Phase Alignment (14) A.4 Step 4: Computational Equivalence and Zero-Overhead Implementation It is crucial to emphasize that Eq. 14 represents a mathematical interpretation of the geometric manifold, not a required computational graph. In practice, the forward pass never explicitly computes polar coordinates, local magnitudes (rq,ir_q,i), or trigonometric functions (cosΔθi _i). Because the formulations are strictly mathematically equivalent, one only needs to apply the standard linear dot product operation (^⋅ q· k) over the MRSNorm-processed vectors. The geometric constraints embedded in the MRSNorm manifold naturally guarantee that this simple inner product behaves exactly as the complex, auto-gated ensemble of micro-heads derived in Eq. 14. Consequently, QK-MRSNorm endows the attention mechanism with robust representation power and dynamic confidence routing with zero additional computational overhead or latency. Appendix B Mathematical Proof of Logit Stability in QK-MRSNorm In Section 5.3, we discussed how QK-MRSNorm geometrically resolves logit imbalance. Here, we formally prove its strict upper bound and demonstrate its theoretical superiority over RMSNorm, highlighting the critical role of the continuous phase in neutralizing magnitude spikes. Limitations of QK-RMSNorm: Unilateral Magnitude Dominance. Recently, QK-RMSNorm has been adopted to mitigate logit explosion by applying 1D scalar normalization. However, RMSNorm is fundamentally limited by its lack of geometric dimensionality. Because a 1D scalar possesses only a binary sign (±1± 1), the interaction between a query and key (q^ik^i q_i k_i) is entirely dictated by their normalized magnitudes. If a single channel monopolizes the overall variance and spikes, this magnitude surge unilaterally forces a massive localized inner product. Without a mechanism to decouple magnitude from semantic direction, an anomalously large scalar spike bypasses all semantic filters, inevitably monopolizing the global attention score. Local Stability via Semantic Phase Veto. In contrast, QK-MRSNorm binds pairwise channels into a continuous phasor manifold, transforming how local features interact. The local inner product PiP_i between a single query and key phasor block evaluates to: Pi=^i⋅^i=(rq,i⋅rk,iSq⋅Sk)cos(Δθi)P_i= q_i· k_i= ( r_q,i· r_k,iS_q· S_k ) ( _i) (15) This geometric decoupling introduces a profound architectural safeguard: the Semantic Phase Veto. Unlike 1D scalars that blindly multiply their magnitudes, the magnitude term in a phasor is merely a potential energy. Even if the magnitude weight (rq,irk,i)/(SqSk)(r_q,ir_k,i)/(S_qS_k) reaches its structural extreme, this surge is completely neutralized if the semantic directions are orthogonal (cos(Δθi)→0 ( _i)→ 0). In other words, a magnitude spike cannot brute-force its way into the attention score; it is strictly gated by the continuous semantic alignment of the features. The localized contribution is therefore bounded not just structurally, but semantically: |Pi|≤rq,iSqrk,iSk≤d2|P_i|≤ r_q,iS_q r_k,iS_k≤ d2 (16) Global Bounding via Cauchy-Schwarz Inequality. Guarded by this semantic phase veto at the micro-level, the global attention score—the sum of all localized phasor interactions—is mathematically guaranteed to remain exceptionally stable. Applying the Cauchy-Schwarz inequality to the sum of the magnitudes yields: |Score|=|∑i=1d/2Pi|≤∑i=1d/2rq,i⋅rk,iSq⋅Sk≤1Sq⋅Sk∑i=1d/2rq,i2∑i=1d/2rk,i2|Score|= | _i=1^d/2P_i |≤ _i=1^d/2 r_q,i· r_k,iS_q· S_k≤ 1S_q· S_k _i=1^d/2r_q,i^2 _i=1^d/2r_k,i^2 (17) By the definition of the global RMS energy, ∑rq,i2=d2Sq2Σ r_q,i^2= d2S_q^2 and ∑rk,i2=d2Sk2Σ r_k,i^2= d2S_k^2. Substituting these terms simplifies the absolute upper bound to: |Score|≤1Sq⋅Sk(d2Sq⋅d2Sk)=d2|Score|≤ 1S_q· S_k ( d2S_q· d2S_k )= d2 (18) This rigorous derivation demonstrates that QK-MRSNorm establishes absolute geometric stability. While the Cauchy-Schwarz inequality guarantees a structural maximum logit of d/2d/2, the true architectural breakthrough lies in the phase veto. Regardless of how extremely an individual feature expands, it cannot hijack the attention mechanism without strict semantic alignment. This dual protection—semantic gating at the local level and absolute bounding at the global level—structurally alleviates the possibility of magnitude-driven softmax collapse. Appendix C Implementation of GroupMRSNorm ⬇ import torch import torch.n as n from typing import Union, Tuple, List import math class GroupMRSNorm(n.Module): def __init__( self, num_groups: int, num_channels: int, channel_dim: int, eps: float = 1e-6 ): super().__init__() if num_channels % 2 != 0: raise ValueError( f"channel (num_channels) must be even for 2D Phasor pairs." ) num_bundles = num_channels // 2 if num_bundles % num_groups != 0: raise ValueError( f"number of hidden bundles (num_bundles) must be divisible by the specified number of groups (num_groups)." ) self.num_groups = num_groups self.num_channels = num_channels self.num_bundles = num_bundles self.channel_dim = channel_dim self.eps = eps self.weight = n.Parameter(torch.empty(num_bundles)) self.reset_parameters() def reset_parameters(self) -> None: n.init.ones_(self.weight) def forward(self, x: torch.Tensor) -> torch.Tensor: orig_dtype = x.dtype orig_shape = x.shape x_fp32 = x.to(torch.float32) c_dim = ( self.channel_dim if self.channel_dim >= 0 else x_fp32.dim() + self.channel_dim ) p_dim = c_dim + 1 paired_shape = ( list(orig_shape)[:c_dim] + [self.num_bundles, 2] + list(orig_shape)[c_dim + 1 :] ) x_paired = x_fp32.view(*paired_shape) mag = torch.norm(x_paired, p=2, dim=p_dim, keepdim=True) C_per_G = self.num_bundles // self.num_groups grouped_mag_shape = ( list(mag.shape)[:c_dim] + [self.num_groups, C_per_G] + list(mag.shape)[c_dim + 1 :] ) mag_g = mag.view(*grouped_mag_shape) reduce_dims = tuple(range(c_dim + 1, mag_g.dim())) l1_mean_g = torch.mean(mag_g, dim=reduce_dims, keepdim=True) + self.eps x_grouped_shape = ( list(x_paired.shape)[:c_dim] + [self.num_groups, C_per_G] + list(x_paired.shape)[c_dim + 1 :] ) x_g = x_paired.view(*x_grouped_shape) x_normed_g = x_g / l1_mean_g x_normed = x_normed_g.view(*orig_shape) weight_expanded = torch.repeat_interleave(self.weight, repeats=2, dim=0) view_shape = [1] * x.dim() view_shape[c_dim] = self.num_channels weight_view = weight_expanded.view(*view_shape) out = x_normed * weight_view return out.to(orig_dtype)