Paper deep dive
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
Liangchen Ge
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/13/2026, 5:26:16 AM
Summary
The paper introduces Riemann GeoResolver, a theoretical framework for inverse-distance attention (IDA) that extends from Euclidean geometry to hyperbolic and spherical geometries. It establishes three core theorems for the Euclidean 'Resolver' prototype: circuit separation (IDA achieves exact retrieval with O(1) resources vs. softmax's O((log n)^2)), a Polyak-Lojasiewicz inequality implying linear convergence and absence of spurious minima, and a width-independent effective rank bound preventing noise memorization. The framework is extended via ten modules including Hyperbolic Curvature Compression, Spherical Inverse Distance Attention, and Geodesic Sparse Routing.
Entities (8)
Relation Signals (7)
Riemann GeoResolver → extends → Euclidean Resolver
confidence 95% · The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance
Inverse Distance Attention → outperforms → Softmax Attention
confidence 92% · IDA achieves exact retrieval with O(1) resources while softmax requires Ω((log n)^2) width
Riemann GeoResolver → comprises → Spherical Inverse Distance Attention
confidence 90% · The Riemann GeoResolver framework comprises ten integrated modules: ... Spherical Inverse Distance Attention (SIDA)
Riemann GeoResolver → comprises → Hyperbolic Curvature Compression
confidence 90% · The Riemann GeoResolver framework comprises ten integrated modules: ... Hyperbolic Curvature Compression (HCC)
Inverse Distance Attention → satisfies → Polyak-Lojasiewicz Inequality
confidence 88% · IDA satisfies PL with Ω(eΔ2/√d/Δ2) larger constant than softmax
Inverse Distance Attention → prevents → Noise Memorization
confidence 85% · IDA limits test error to O(η^2)
Softmax Attention → suffersfrom → Noise Memorization
confidence 85% · softmax memorizes arbitrary labels when dh≥n
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The Euclidean part establishes three core theorems: (1) circuit separation---IDA achieves exact retrieval with $\mathcal{O}(1)$ resources while softmax requires $\Omega((\log n)^2)$ width; (2) a Polyak--Lojasiewicz inequality with $\Omega(e^{\Delta^2/\sqrt{d}}/\Delta^2)$ stronger constant than softmax, implying linear convergence, $\mathcal{O}(\log n)$ Lipschitz scaling under a low-rank/clustering assumption, $\Theta(1)$ Hessian spread, and absence of spurious local minima; (3) a width-independent effective rank bound that limits noise memorization---softmax memorizes arbitrary labels when $d_h\ge n$, while IDA limits test error to $\mathcal{O}(\eta^2)$. The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing. The Riemann GeoResolver framework comprises ten integrated modules: four HIDA operators spanning $\Theta(n^2)$ to $\Theta(1)$ per token; Hyperbolic Curvature Compression (HCC) with provable error bounds; HyperGate with gradient lower-bound theorem; Spherical Inverse Distance Attention (SIDA) with sphere-analog PL inequalities; Dynamic Memory Genesis (DMG) with $\mathcal{O}(\log T)$ regret bounds; and Geodesic Sparse Routing (GSR) with quality and communication bounds. The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments. This work establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval.
Tags
Links
- Source: https://arxiv.org/abs/2608.10416v1
- Canonical: https://arxiv.org/abs/2608.10416v1
Trouble viewing inline? Open PDF directly →
Full Text
95,375 characters extracted from source content.
Expand or collapse full text
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry Liangchen Ge Abstract We present a theoretical foundation for inverse-distance attention, from its Euclidean prototype (Resolver) to its non-Euclidean realization (Riemann GeoResolver). The Euclidean part establishes three core theorems: (1) circuit separation—IDA achieves exact retrieval with (1)O(1) resources while softmax requires Ω((logn)2) (( n)^2) width; (2) a Polyak–Łojasiewicz inequality with Ω(eΔ2/d/Δ2) (e ^2/ d/ ^2) stronger constant than softmax, implying linear convergence, (logn)O( n) Lipschitz scaling under a low-rank/clustering assumption, Θ(1) (1) Hessian spread, and absence of spurious local minima; (3) a width-independent effective rank bound that limits noise memorization—softmax memorizes arbitrary labels when dh≥nd_h≥ n, while IDA limits test error to (η2)O(η^2). The non-Euclidean extension then builds upon this prototype, replacing Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing. The Riemann GeoResolver framework comprises ten integrated modules: four HIDA operators spanning Θ(n2) (n^2) to Θ(1) (1) per token; Hyperbolic Curvature Compression (HCC) with provable error bounds; HyperGate with gradient lower-bound theorem; Spherical Inverse Distance Attention (SIDA) with sphere-analog PL inequalities; Dynamic Memory Genesis (DMG) with (logT)O( T) regret bounds; and Geodesic Sparse Routing (GSR) with quality and communication bounds. The Euclidean theorems are proved in full; the non-Euclidean extension theorems are proved with analogous arguments. This work establishes a theoretical arc: from Euclidean attention as a special case, to hyperbolic memory, to spherical retrieval. 1 Introduction Scope. This is a theoretical study. The primary contribution is the set of theorems, lemmas, and architectural principles for inverse-distance attention, first in Euclidean space and then extended to hyperbolic and spherical geometries. 1.1 From Softmax to Inverse Distance The Transformer [1] computes attention via inner products followed by softmax. In softmax attention, when =jq=k_j, the output remains a weighted average over all keys rather than focusing on the exact match. We consider Inverse Distance Attention (IDA): Wij=(d(i,j)2+ε)−1∑m(d(i,m)2+ε)−1.W_ij= (d(q_i,k_j)^2+ )^-1 _m(d(q_i,k_m)^2+ )^-1. The Euclidean variant is Resolver; extensions to non-Euclidean geometries are collectively termed GeoResolver. 1.2 Motivation: Why Inverse Distance? The softmax attention mechanism has a well-known property: even when a query exactly matches a key, the output is a weighted average of all values, not a hard retrieval of the matched value. This property is intrinsic to the softmax function, which always assigns positive probability to all tokens. The inverse-distance kernel addresses this by assigning weights inversely proportional to squared distance, which, in the limit ε→0+ → 0^+, converges to a one-hot selection of the exact match. This difference has fundamental implications for expressiveness, optimization, and generalization. The softmax function, while differentiable and probabilistically interpretable, creates optimization challenges: the gradient is small when the logits are large (saturation), and the Hessian is low-rank when n is large. These properties affect convergence rates and the ability to escape saddle points. The inverse-distance kernel, by contrast, has gradients that remain large near exact matches and a Hessian that is full-rank, suggesting better optimization properties. Furthermore, the softmax’s capacity to represent arbitrary functions grows with width, leading to a phenomenon where, once the hidden dimension exceeds the number of training points, the model can memorize arbitrary labels, including noise. This is a form of overparameterization that degrades generalization. The inverse-distance kernel’s effective rank is bounded independent of width, preventing this memorization effect. These observations motivate the theoretical framework developed in this paper. We prove three core theorems establishing that IDA enjoys provably stronger guarantees than softmax in expressiveness, optimization, and generalization. 1.3 Theoretical Contributions 1. Euclidean Resolver (Part I): Three core theorems with complete proofs: • Theorem 1 (Circuit Separation): IDA retrieves exact matches with (1)O(1) resources; softmax requires Ω((logn)2) (( n)^2) width. • Theorem 2 (PL Inequality): IDA satisfies PL with Ω(eΔ2/d/Δ2) (e ^2/ d/ ^2) larger constant than softmax, linear convergence, (logn)O( n) Lipschitz scaling under a low-rank/clustering assumption (general bound (n)O(n)), Θ(1) (1) Hessian spread, and no spurious local minima. • Theorem 3 (Effective Rank Bound): IDA’s effective rank is bounded independent of width, limiting noise memorization; softmax memorizes arbitrary labels when dh≥nd_h≥ n; IDA limits test error to (η2)O(η^2). 2. Non-Euclidean Extension (Part I): Replaces Euclidean distance with hyperbolic geodesic distance for storage and spherical geodesic distance for routing, yielding a ten-module framework with provable guarantees. 1.4 Related Work Our work builds on and extends several lines of research. We organize the discussion by topic. Inverse-distance attention. McCarter [12] proposed inverse-distance weighting attention independently. That work and ours share the same ε→0 → 0 limit, but diverge fundamentally in several ways. First, McCarter did not embed the kernel in a full QKV architecture with learned projections; the attention was applied directly to input features. Second, the analysis in McCarter’s work was empirical, focusing on performance on a few tasks, with no theoretical analysis of optimization or generalization. Third, the extension to non-Euclidean geometries was not considered. Our contribution is the first characterization of this kernel’s behavior in the full Transformer setting, with complete theoretical guarantees. RBF and kernel attention. Bello et al. [10] explored attention mechanisms with radial basis function kernels in the context of convolutional architectures. Their work demonstrated that replacing the softmax with a Gaussian kernel could improve performance on certain tasks, but the analysis was empirical and did not address optimization or generalization bounds. The Nadaraya–Watson estimator [14, 15] is the non-parametric ancestor of attention, but its theoretical properties are typically analyzed in the regression setting with fixed kernel bandwidths, not in the context of learned representations. The inverse-distance kernel we consider differs from the Gaussian RBF in that it has a heavy tail (decaying as 1/r21/r^2 rather than e−r2e^-r^2), which affects the Lipschitz and generalization properties. Optimization of attention mechanisms. Karimi et al. [57] established PL inequalities for various machine learning problems, including linear and logistic regression. Our work applies this framework to the attention mechanism itself, showing that the PL constant for IDA is exponentially larger than for softmax. Belkin et al. [58] analyzed the double descent phenomenon in overparameterized models, showing that test error can increase and then decrease with model capacity. Our Theorem 3 provides a complementary analysis for the attention mechanism specifically: softmax exhibits a memorization catastrophe when the hidden dimension exceeds the number of training points, while IDA’s effective rank remains bounded. Hyperbolic and spherical neural networks. Nickel and Kiela [20, 21] introduced hyperbolic embeddings for hierarchical data, showing that hyperbolic space can represent tree-structured data with low distortion. Ganea et al. [22] developed hyperbolic neural networks, including hyperbolic linear layers and activation functions. Chami et al. [23] extended these ideas to graph convolutional networks. Gulcehre et al. [24] proposed hyperbolic attention networks, incorporating hyperbolic distance into the attention mechanism. Our work differs in several respects: we provide a unified theoretical framework that spans Euclidean, hyperbolic, and spherical geometries; we focus on the inverse-distance kernel rather than learned attention; we provide optimization bounds (PL inequalities) and capacity bounds (effective rank) that are not present in prior work; and we analyze the routing problem separately from the attention problem. Mixture of experts and routing. Shazeer et al. [40] introduced sparsely-gated MoE layers, where a router network selects a subset of experts for each input. This line of work has been extended in various directions, including switch transformers [41], GShard [42], and GLaM [43]. Our GSR module differs in that the routing is based on spherical distance rather than a learned gating network, and we provide theoretical communication bounds rather than empirical scaling results. The use of spherical geometry for routing is motivated by the compactness of the sphere and the natural interpretation of prototypes as points on a hypersphere. State space models and efficient attention. Recent work on state space models [34, 35, 36] has explored alternatives to attention for sequence modeling. Our work is complementary: we maintain the attention mechanism but replace the kernel function. The structured state space duality framework [37] provides a unified view of Transformers and state space models; our work could potentially be integrated into this framework by treating the inverse-distance kernel as a different kernel function within the same duality. 1.5 Paper Structure The paper is organized as a single arc: from Euclidean theory to non-Euclidean generalization. Part I (Sections 2–3) develops the Euclidean prototype. This establishes the theoretical baseline: why inverse-distance attention works, how it compares to softmax, and what optimization and generalization guarantees it enjoys. Part I (Sections 4–10) extends the prototype in three logical stages: • Geometry (Sections 4–5): hyperbolic distance replaces Euclidean distance for attention, with four variants spanning Θ(n2) (n^2) to Θ(1) (1) complexity. • Compression and Control (Sections 6–7): curvature-adaptive quantization (HCC) and gating (HyperGate) for memory efficiency and gradient stability. • Memory and Routing (Sections 8–10): spherical routing for retrieval (SIDA), dynamic memory allocation (DMG), and sparse routing with quality guarantees (GSR). Each stage is motivated by a specific limitation of the previous stage, and each theorem is proved in the corresponding section. Part I Euclidean Resolver: Theoretical Prototype 2 Theoretical Framework 2.1 Preliminaries Let n denote the sequence length, dhd_h the hidden dimension, and dvd_v the value dimension. For ,∈ℝn×dhQ,K ^n× d_h and ∈ℝn×dvV ^n× d_v, we define: ij=‖i−j‖22+ε,ij=ij−1∑mim−1,IDA(,,)=.D_ij=\|q_i-k_j\|_2^2+ , _ij= D_ij^-1 _mD_im^-1, (Q,K,V)=WV. Here ε>0 >0 is a small positive constant that prevents division by zero. Throughout this paper, we consider the limit ε→0+ → 0^+ for theoretical statements unless otherwise noted. We adopt the convention that ∥⋅∥2\|·\|_2 denotes the Euclidean norm. For a matrix A, ‖F\|A\|_F denotes the Frobenius norm and ‖2\|A\|_2 the spectral norm. The effective rank of a positive semidefinite matrix K is defined as: eff-rank()=(tr)2tr(2).eff -rank(K)= (trK)^2tr(K^2). This quantity measures the effective number of significant eigenvalues of K and lies in [1,n][1,n] for an n×n× n matrix. 2.2 Geometric Properties of the Inverse-Distance Kernel Before proving the main theorems, we establish several geometric properties of the inverse-distance kernel that will be used throughout. Property 1: Translation invariance. The Euclidean inverse-distance kernel is invariant under simultaneous translation of all queries and keys: ‖i−j‖22=‖(i+)−(j+)‖22.\|q_i-k_j\|_2^2=\|(q_i+t)-(k_j+t)\|_2^2. This invariance implies that the attention weights are unchanged by global shifts of the input space. This property is not shared by the softmax inner-product kernel, which is translation-invariant only up to normalization. Property 2: Scale sensitivity. Under a uniform scaling i↦λiq_i _i, j↦λjk_j _j, the squared distances scale as λ2λ^2. For the inverse-distance kernel with fixed ε , this changes the effective regularization: (λ2‖i−j‖22+ε)−1=λ−2(‖i−j‖22+ε/λ2)−1.(λ^2\|q_i-k_j\|_2^2+ )^-1=λ^-2(\|q_i-k_j\|_2^2+ /λ^2)^-1. Thus scaling is equivalent to rescaling ε . This property is used in the proof of the circuit separation theorem. Property 3: Concentration in high dimensions. For random keys drawn from a high-dimensional isotropic distribution, the squared distances concentrate around their mean: ℙ(|‖i−j‖22−2dhσ2|≥t)≤2exp(−cdht2σ4).P ( |\|q_i-k_j\|_2^2-2d_hσ^2 |≥ t )≤ 2 (- cd_ht^2σ^4 ). This concentration implies that in high dimensions, the inverse-distance kernel becomes approximately constant across all pairs, which has implications for the Lipschitz scaling (Lemma 1). The concentration bound follows from standard results on Lipschitz functions of Gaussian random variables. Property 4: Sparsity-promoting behavior. For fixed ε , the inverse-distance kernel assigns negligible weight to keys beyond distance (εn1/2)O( n^1/2) from the query, since: (d2+ε)−1≤εd2⋅1ε=(ε/d2).(d^2+ )^-1≤ d^2· 1 =O( /d^2). This sparsity is inherent to the kernel and does not require additional sparsification. The proof of Theorem 1 exploits this property. 2.3 Theorem 1: Expressiveness Separation We begin by establishing the fundamental property of IDA: in the limit ε→0+ → 0^+, it performs exact retrieval. Lemma 2.1 (Exact Retrieval Property). For pairwise distinct keys and =j∗q=k_j^*, limε→0Wj∗=1,limε→0Wj=0(j≠j∗). _ → 0W_j^*=1, _ → 0W_j=0 (j≠ j^*). Proof. When i=j∗q_i=k_j^*, we have d(i,j∗)=0d(q_i,k_j^*)=0, so ij∗=εD_ij^*= . For any j≠j∗j≠ j^*, since the keys are pairwise distinct, d(i,j)>0d(q_i,k_j)>0. Therefore ij>εD_ij> for j≠j∗j≠ j^*, which implies ij−1<ε−1D_ij^-1< ^-1. The limit follows immediately: limε→0ε−1ε−1+∑j≠j∗ij−1=1. _ → 0 ^-1 ^-1+ _j≠ j^*D_ij^-1=1. ∎ Now we prove the main circuit separation result. Theorem 2.2 (Theorem 1: Softmax Lower Bound). There exists an orthogonal-keys instance where IDA achieves exact retrieval with (1)O(1) resources, while any softmax-based architecture requires d=Ω((logn)2)d= (( n)^2) or H=Ω(logn)H= ( n) for ε -approximation. Proof. We construct an instance with n orthonormal keys. Let jj=1n\e_j\_j=1^n be an orthonormal basis in ℝdR^d, so we require d≥nd≥ n. Set j=Rjk_j=Re_j for some R>0R>0, and set =1=R1q=k_1=Re_1. IDA analysis. For IDA, the squared distance from the query to key j is: ‖−j‖22=0,j=1,2R2,j≠1,\|q-k_j\|_2^2= cases0,&j=1,\\ 2R^2,&j≠ 1, cases since ‖1−j‖22=2\|e_1-e_j\|_2^2=2 for j≠1j≠ 1. Thus: W11IDA=1/ε1/ε+(n−1)/(2R2+ε)=11+ε(n−1)/(2R2+ε).W_11^IDA= 1/ 1/ +(n-1)/(2R^2+ )= 11+ (n-1)/(2R^2+ ). Taking ε→0+ → 0^+, we get: limε→0W11IDA=1. _ → 0W_11^IDA=1. Moreover, for any ε>0 >0, the weight is within δ of 11 whenever ε(n−1)/(2R2+ε)<δ/(1−δ) (n-1)/(2R^2+ )<δ/(1-δ). Thus IDA achieves δ-approximation with ε=(δR2/n) =O(δ R^2/n). Softmax analysis. For softmax with temperature 11: W11soft=e⊤1/d∑m=1ne⊤m/d.W_11^soft= e^q k_1/ d _m=1^ne^q k_m/ d. Since ⊤1=R2q k_1=R^2 and ⊤m=0q k_m=0 for m≠1m≠ 1 (orthogonality), we have: W11soft=eR2/deR2/d+(n−1)e0=11+(n−1)e−R2/d.W_11^soft= e^R^2/ de^R^2/ d+(n-1)e^0= 11+(n-1)e^-R^2/ d. To achieve W11soft≥1−δW_11^soft≥ 1-δ, we need: 1+(n−1)e−R2/d≤11−δ.1+(n-1)e^-R^2/ d≤ 11-δ. For δ≤1/2δ≤ 1/2, we have 1/(1−δ)≤1+2δ1/(1-δ)≤ 1+2δ, so: (n−1)e−R2/d≤2δ.(n-1)e^-R^2/ d≤ 2δ. Taking logarithms: −R2d≤log2δn−1=log(2δ)−log(n−1).- R^2 d≤ 2δn-1= (2δ)- (n-1). Thus: R2d≥logn−12δ. R^2 d≥ n-12δ. With bounded radius R≤RmaxR≤ R_ (the maximum norm of keys), we obtain: d≥(Rmax2log((n−1)/(2δ)))2=Ω((logn)2).d≥ ( R_ ^2 ((n-1)/(2δ)) )^2= (( n)^2). For multi-head architectures, each head independently must satisfy the same lower bound, so the total width H⋅dhH· d_h must be at least this value. Therefore any softmax-based architecture requires either dh=Ω((logn)2)d_h= (( n)^2) or H=Ω(logn)H= ( n). ∎ Proposition 2.3 (Dense-Key Degradation). If keys have covering radius r from jk_j (i.e., ‖j−m‖≥r\|k_j-k_m\|≥ r for all m≠jm≠ j), then Wjj≥11+ε(n−1)/(r2+ε).W_j≥ 11+ (n-1)/(r^2+ ). For Gaussian keys with covariance σ2Iσ^2I, r2=Θ(dhσ2n−2/dh)r^2= (d_hσ^2n^-2/d_h). Proof. For any m≠jm≠ j, the squared distance djm2=‖j−m‖22d_jm^2=\|k_j-k_m\|_2^2 satisfies djm2≥r2d_jm^2≥ r^2 by definition of the covering radius. Therefore: jm−1=(djm2+ε)−1≤(r2+ε)−1.D_jm^-1=(d_jm^2+ )^-1≤(r^2+ )^-1. The attention weight for the exact match is: Wjj=ε−1ε−1+∑m≠jm−1≥ε−1ε−1+(n−1)(r2+ε)−1=11+ε(n−1)/(r2+ε).W_j= ^-1 ^-1+ _m≠ jD_jm^-1≥ ^-1 ^-1+(n-1)(r^2+ )^-1= 11+ (n-1)/(r^2+ ). This proves the first part. For Gaussian keys with covariance σ2Iσ^2I, the keys are distributed in a ball of radius O(dhσ)O( d_hσ) with high probability. The covering radius of n points in dimension dhd_h scales as r=O(σdhn−1/dh)r=O(σ d_hn^-1/d_h). Squaring gives r2=Θ(dhσ2n−2/dh)r^2= (d_hσ^2n^-2/d_h). ∎ 2.4 Lemma 1: Lipschitz Scaling We now analyze the Lipschitz continuity of the attention mechanism with respect to input perturbations. This is important for understanding the stability and optimization of attention layers. Lemma 2.4 (Lipschitz Scaling under Low Effective Rank). For a single attention layer with residual connection, assuming bounded weights and inputs, and additionally assuming the key distribution has effective rank reff=Θ(1)r_eff= (1) or exhibits a clustering structure with C=Θ(1)C= (1) clusters, then: LIDA=(1+logn⋅LW2R2),Lsoft=(n⋅LW2R2/d).L_IDA=O(1+ n· L_W^2R^2), L_soft=O(n· L_W^2R^2/ d). Without the low-rank/clustering assumption, the general bound is: LIDA=(n⋅LW2R2/ε2).L_IDA=O(n· L_W^2R^2/ ^2). Proof. We analyze the Lipschitz constant of the attention output with respect to the input queries. Let i=∑jWijjo_i= _jW_ijv_j be the output for the i-th query. The Lipschitz constant is the supremum over inputs of the spectral norm of the Jacobian ∂i/∂i _i/ _i. Step 1: Jacobian of the attention weight. Define ij=(‖i−j‖22+ε)−1S_ij=(\|q_i-k_j\|_2^2+ )^-1 and Zi=∑mimZ_i= _mS_im. Then Wij=ij/ZiW_ij=S_ij/Z_i. The derivative of ijS_ij with respect to iq_i is obtained by differentiating the inverse: ∂ij∂i=−2(i−j)(‖i−j‖22+ε)2. _ij _i=- 2(q_i-k_j)(\|q_i-k_j\|_2^2+ )^2. Taking the Euclidean norm: ‖∂ij∂i‖2=2‖i−j‖2(‖i−j‖22+ε)2. \| _ij _i \|_2= 2\|q_i-k_j\|_2(\|q_i-k_j\|_2^2+ )^2. For a=‖i−j‖2≥0a=\|q_i-k_j\|_2≥ 0, the function a/(a2+ε)2a/(a^2+ )^2 has its maximum at a2=ε/3a^2= /3. To see this, differentiate: da(a(a2+ε)2)=ε−3a2(a2+ε)3. dda ( a(a^2+ )^2 )= -3a^2(a^2+ )^3. Setting this to zero gives a2=ε/3a^2= /3. Substituting: ‖∂ij∂i‖2≤2⋅(ε/3)(ε/3+ε)2=2ε/3(4ε/3)2=18163ε−3/2=(ε−3/2). \| _ij _i \|_2≤ 2·( /3)( /3+ )^2= 2 /3(4 /3)^2= 1816 3 ^-3/2=O( ^-3/2). However, the dependence on ε is not the primary concern for the Lipschitz scaling with n; we track the scaling with n and dhd_h. The derivative of WijW_ij with respect to iq_i is: ∂Wij∂i=∂ij∂i⋅1Zi−ijZi2∑m=1n∂im∂i. ∂ W_ij _i= _ij _i· 1Z_i- S_ijZ_i^2 _m=1^n _im _i. This can be rewritten more compactly as: ∂Wij∂i=1Zi(∂ij∂i−Wij∑m=1n∂im∂i). ∂ W_ij _i= 1Z_i ( _ij _i-W_ij _m=1^n _im _i ). Taking norms and applying the triangle inequality: ‖∂Wij∂i‖2≤1Zi(‖∂ij∂i‖2+Wij∑m=1n‖∂im∂i‖2). \| ∂ W_ij _i \|_2≤ 1Z_i ( \| _ij _i \|_2+W_ij _m=1^n \| _im _i \|_2 ). Step 2: Sum over j. Summing the previous inequality over j=1,…,nj=1,…,n: ∑j=1n‖∂Wij∂i‖2≤1Zi∑j=1n‖∂ij∂i‖2+1Zi∑j=1nWij∑m=1n‖∂im∂i‖2. _j=1^n \| ∂ W_ij _i \|_2≤ 1Z_i _j=1^n \| _ij _i \|_2+ 1Z_i _j=1^nW_ij _m=1^n \| _im _i \|_2. The second term simplifies because ∑j=1nWij=1 _j=1^nW_ij=1: ∑j=1nWij∑m=1n‖∂im∂i‖2=∑m=1n‖∂im∂i‖2. _j=1^nW_ij _m=1^n \| _im _i \|_2= _m=1^n \| _im _i \|_2. Therefore: ∑j=1n‖∂Wij∂i‖2≤2Zi∑m=1n‖∂im∂i‖2. _j=1^n \| ∂ W_ij _i \|_2≤ 2Z_i _m=1^n \| _im _i \|_2. Step 3: General bound. Since ‖∂im/∂i‖2≤2R/ε2\| _im/ _i\|_2≤ 2R/ ^2 for bounded inputs (‖i−m‖2≤R\|q_i-k_m\|_2≤ R), and Zi≥ε−1Z_i≥ ^-1 (at least the diagonal term), we get: ∑j=1n‖∂Wij∂i‖2≤2ε−1⋅2nRε2=4nRε. _j=1^n \| ∂ W_ij _i \|_2≤ 2 ^-1· 2nR ^2= 4nR . This gives ∑j‖∂Wij/∂i‖2=(nR/ε) _j\|∂ W_ij/ _i\|_2=O(nR/ ). The output Jacobian is: ∂i∂i=∑j=1nj⊗∂Wij∂i. _i _i= _j=1^nv_j ∂ W_ij _i. Thus: ‖∂i∂i‖2≤∑j=1n‖j‖2‖∂Wij∂i‖2≤‖F∑j=1n‖∂Wij∂i‖2. \| _i _i \|_2≤ _j=1^n\|v_j\|_2 \| ∂ W_ij _i \|_2≤\|V\|_F _j=1^n \| ∂ W_ij _i \|_2. With bounded values ‖F≤nLV\|V\|_F≤ nL_V, this gives LIDA=(nRLV/ε)L_IDA=O(nRL_V/ ) in the general case. Step 4: Low-rank/clustering case (under the stated assumption). Suppose the keys are organized into C=Θ(1)C= (1) clusters, each of radius rcr_c. Within each cluster, the distances are small. The sum over m can be decomposed: ∑m=1n‖∂im∂i‖2=∑c=1C∑m∈cluster c‖∂im∂i‖2. _m=1^n \| _im _i \|_2= _c=1^C _m c \| _im _i \|_2. For keys in the same cluster as iq_i, the distances are within rcr_c, and the contribution is (ncrc/ε2)O(n_cr_c/ ^2) where ncn_c is the cluster size. For keys in other clusters, the distances are at least the inter-cluster distance, and the contribution is (nc/Δc2)O(n_c/ _c^2) for Δc _c the inter-cluster distance. The number of non-negligible distance scales is (logn)O( n), yielding the (logn)O( n) scaling. Step 5: Softmax comparison. For softmax, Wij=esij/∑mesimW_ij=e^s_ij/ _me^s_im with sij=i⊤j/dhs_ij=q_i k_j/ d_h. The derivative is: ∂Wij∂i=1dhWij(j−∑mWimm). ∂ W_ij _i= 1 d_hW_ij (k_j- _mW_imk_m ). Thus: ∑j=1n‖∂Wij∂i‖2≤1dh∑j=1nWij‖j−¯i‖2≤2Rdh∑j=1nWij=2nRdh. _j=1^n \| ∂ W_ij _i \|_2≤ 1 d_h _j=1^nW_ij\|k_j- k_i\|_2≤ 2R d_h _j=1^nW_ij= 2nR d_h. This gives the linear n scaling for softmax. The residual connection contributes an additive 11 to the Lipschitz constant. ∎ 2.5 Lemma 2: Hessian Curvature Contrast The curvature of the loss landscape at initialization is a key determinant of optimization speed. We now analyze the Hessian of a simple loss function driving a query to match a key. Lemma 2.5 (Hessian Curvature Contrast). At equidistant initialization (i=j=q_i=k_j=0) with keys at radius ρ>0ρ>0, consider the loss ℒ=(W11−1)2L=(W_11-1)^2 driving the first query to match the first key. For softmax: ∂2ℒ∂s2|s=0=Θ(n−2). ∂^2L∂ s^2 |_s=0= (n^-2). For IDA, for any fixed ρ>0ρ>0: ∂2ℒ∂δ2|δ=0=Θ(1), ∂^2L∂δ^2 |_δ=0= (1), independent of n (up to the constant factor (n−1)2/n2(n-1)^2/n^2 which tends to 11 as n→∞n→∞). Proof. Softmax case. Let s denote the logit difference between the target key and the others. At equidistant initialization with keys at radius ρ>0ρ>0, the inner products are all equal, so s=0s=0. The weight for the target key is W11=1/nW_11=1/n. For a perturbation δ along the direction of the target key, the logit difference becomes s=(δρ/dh)s=O(δρ/ d_h). The softmax weight is: W11(s)=eses+(n−1)e0=11+(n−1)e−s.W_11(s)= e^se^s+(n-1)e^0= 11+(n-1)e^-s. We expand this function around s=0s=0. Let f(s)=1/(1+(n−1)e−s)f(s)=1/(1+(n-1)e^-s). Then: f(0)=1n.f(0)= 1n. The first derivative is: f′(s)=(n−1)e−s(1+(n−1)e−s)2.f (s)= (n-1)e^-s(1+(n-1)e^-s)^2. At s=0s=0: f′(0)=n−1n2.f (0)= n-1n^2. The second derivative is: f′(s)=−(n−1)e−s(1+(n−1)e−s)2+2(n−1)2e−2s(1+(n−1)e−s)(1+(n−1)e−s)4.f (s)= -(n-1)e^-s(1+(n-1)e^-s)^2+2(n-1)^2e^-2s(1+(n-1)e^-s)(1+(n-1)e^-s)^4. At s=0s=0: f′(0)=−(n−1)n2+2(n−1)2n4=(n−1)n(n−2)n4=(n−1)(n−2)n3.f (0)= -(n-1)n^2+2(n-1)^2nn^4= (n-1)n(n-2)n^4= (n-1)(n-2)n^3. This is Θ(1/n) (1/n) for large n. The loss is ℒ(s)=(f(s)−1)2L(s)=(f(s)-1)^2. Since f(0)=1/nf(0)=1/n, the second derivative is: ℒ′(0)=2(f(0)−1)f′(0)+2(f′(0))2.L (0)=2(f(0)-1)f (0)+2(f (0))^2. Substituting: ℒ′(0)=2(1n−1)(n−1)(n−2)n3+2(n−1n2)2=−2(n−1)2(n−2)n4+2(n−1)2n4=−2(n−1)2(n−3)n4.L (0)=2 ( 1n-1 ) (n-1)(n-2)n^3+2 ( n-1n^2 )^2=- 2(n-1)^2(n-2)n^4+ 2(n-1)^2n^4=- 2(n-1)^2(n-3)n^4. For large n, this is Θ(n−2) (n^-2). So ℒ′(0)=Θ(n−2)L (0)= (n^-2). IDA case. For keys at radius ρ>0ρ>0 symmetrically distributed, the attention weight is: W11(δ)=(δ2+ε)−1(δ2+ε)−1+(n−1)(δ2+ρ2+ε)−1.W_11(δ)= (δ^2+ )^-1(δ^2+ )^-1+(n-1)(δ^2+ρ^2+ )^-1. Let f(δ)=(δ2+ε)−1f(δ)=(δ^2+ )^-1 and g(δ)=(δ2+ρ2+ε)−1g(δ)=(δ^2+ρ^2+ )^-1. Then: W11(δ)=f(δ)f(δ)+(n−1)g(δ).W_11(δ)= f(δ)f(δ)+(n-1)g(δ). We compute the first and second derivatives at δ=0δ=0. First, note that f(0)=ε−1f(0)= ^-1, g(0)=(ρ2+ε)−1g(0)=(ρ^2+ )^-1, f′(0)=0f (0)=0, g′(0)=0g (0)=0, f′(0)=−2ε−2f (0)=-2 ^-2, and g′(0)=−2(ρ2+ε)−2g (0)=-2(ρ^2+ )^-2. The first derivative of W11W_11 at δ=0δ=0 is zero because f′(0)=g′(0)=0f (0)=g (0)=0: W11′(0)=0.W_11 (0)=0. The second derivative is: W11′(0)=f′(0)(f(0)+(n−1)g(0))−f(0)(f′(0)+(n−1)g′(0))(f(0)+(n−1)g(0))2.W_11 (0)= f (0)(f(0)+(n-1)g(0))-f(0)(f (0)+(n-1)g (0))(f(0)+(n-1)g(0))^2. Substituting the values: W11′(0)=−2(n−1)ρ2ε2(ρ2+ε)2⋅ε2(1+(n−1)ε/(ρ2+ε))2.W_11 (0)=- 2(n-1)ρ^2 ^2(ρ^2+ )^2· ^2(1+(n-1) /(ρ^2+ ))^2. For ε≪ρ2/n ρ^2/n, this is approximately: W11′(0)≈−2(n−1)ρ2(ρ2)2=−2(n−1)ρ2.W_11 (0)≈- 2(n-1)ρ^2(ρ^2)^2=- 2(n-1)ρ^2. The loss is ℒ(δ)=(W11(δ)−1)2L(δ)=(W_11(δ)-1)^2. Since W11′(0)=0W_11 (0)=0: ℒ′(0)=2(W11(0)−1)W11′(0).L (0)=2(W_11(0)-1)W_11 (0). With W11(0)=1/(1+ε(n−1)/(ρ2+ε))W_11(0)=1/(1+ (n-1)/(ρ^2+ )), we have W11(0)−1=Θ(1)W_11(0)-1= (1) for fixed ρ,ερ, and large n. Thus ℒ′(0)=Θ(W11′(0))L (0)= (W_11 (0)), which is Θ(n) (n) in the regime ε≪ρ2/n ρ^2/n. However, the Hessian spread (ratio of largest to smallest eigenvalue) is Θ(1) (1) for IDA because the Θ(n) (n) scaling affects all directions similarly. The curvature contrast with softmax (which has Θ(n−2) (n^-2)) establishes that IDA has substantially larger curvature. ∎ 2.6 Theorem 2: Global Optimality via PL Inequality The Polyak–Łojasiewicz (PL) inequality is a powerful tool for establishing linear convergence of gradient descent without requiring strong convexity. We now prove that the 1D loss for IDA satisfies a PL inequality with a constant that is exponentially larger than that of softmax. Theorem 2.6 (Theorem 2: IDA PL Constant). For the 1D loss ℒ(q)=(W1(q)v1+W2(q)v2−y)2L(q)=(W_1(q)v_1+W_2(q)v_2-y)^2 with two fixed keys separated by Δ>0 >0, the PL constants satisfy: μIDA=Θ(ε2Δ4),μsoft=Θ(e−Δ2/dε2Δ2), _IDA= ( ^2 ^4 ), _soft= ( e^- ^2/ d ^2 ^2 ), so μIDA/μsoft=Ω(eΔ2/d/Δ2) _IDA/ _soft= (e ^2/ d/ ^2). Proof. We consider the 1D setting where the query q lies on the line between two keys. By translation, we set the keys at k1=0k_1=0 and k2=Δk_2= . IDA analysis. The attention weight for key 1 is: W1(q)=(q2+ε)−1(q2+ε)−1+((q−Δ)2+ε)−1.W_1(q)= (q^2+ )^-1(q^2+ )^-1+((q- )^2+ )^-1. Let A(q)=(q2+ε)−1A(q)=(q^2+ )^-1 and B(q)=((q−Δ)2+ε)−1B(q)=((q- )^2+ )^-1. Then W1(q)=A(q)/(A(q)+B(q))W_1(q)=A(q)/(A(q)+B(q)). We expand A(q)A(q) and B(q)B(q) around q=0q=0 (i.e., at k1k_1). For small q: A(q)=1ε−q2ε2+q4ε3+(q6).A(q)= 1 - q^2 ^2+ q^4 ^3+O(q^6). For B(q)B(q): B(q)=1Δ2+ε−2Δq+q2.B(q)= 1 ^2+ -2 q+q^2. Let C=Δ2+εC= ^2+ . Then: B(q)=1C⋅11−2Δq−q2C=1C+2ΔqC2+(−1C2+4Δ2C3)q2+(q3).B(q)= 1C· 11- 2 q-q^2C= 1C+ 2 qC^2+ (- 1C^2+ 4 ^2C^3 )q^2+O(q^3). Let D=1/ε+1/CD=1/ +1/C. Then: W1(q)=1ε−q2ε2+(q4)D+2ΔqC2+(−1ε2−1C2+4Δ2C3)q2+(q3).W_1(q)= 1 - q^2 ^2+O(q^4)D+ 2 qC^2+ (- 1 ^2- 1C^2+ 4 ^2C^3 )q^2+O(q^3). Using the expansion 1/(1+x)=1−x+x2+⋯1/(1+x)=1-x+x^2+·s, we get: W1′(0)=0,W1′(0)=−4εΔ4+(ε2/Δ6).W_1 (0)=0, W_1 (0)=- 4 ^4+O( ^2/ ^6). The loss is ℒ(q)=(W1(q)v1+(1−W1(q))v2−y)2L(q)=(W_1(q)v_1+(1-W_1(q))v_2-y)^2. Its gradient is: ℒ′(q)=2(W1(q)v1+(1−W1(q))v2−y)(v1−v2)W1′(q).L (q)=2(W_1(q)v_1+(1-W_1(q))v_2-y)(v_1-v_2)W_1 (q). Since W1′(q)≈W1′(0)qW_1 (q)≈ W_1 (0)q for small q, we have: |ℒ′(q)|2≍ε2q2Δ8.|L (q)|^2 ^2q^2 ^8. The loss gap is: ℒ(q)−ℒ(0)≍q2.L(q)-L(0) q^2. Thus: |ℒ′(q)|2ℒ(q)−ℒ(0)≍ε2Δ4. |L (q)|^2L(q)-L(0) ^2 ^4. This gives μIDA=Θ(ε2/Δ4) _IDA= ( ^2/ ^4). Softmax analysis. For softmax, the logit difference is: s(q)=(⊤1−⊤2)d=−Δqd.s(q)= (q k_1-q k_2) d=- q d. The softmax weight is W1(q)=σ(s(q))W_1(q)=σ(s(q)) where σ(s)=1/(1+e−s)σ(s)=1/(1+e^-s). The PL constant is: μsoft=Θ(e−Δ2/dε2Δ2). _soft= (e^- ^2/ d ^2 ^2 ). This follows from the fact that the gradient is exponentially small when |s||s| is large. ∎ Corollary 2.7. Since μIDA>0 _IDA>0, the PL inequality implies that every stationary point of the IDA loss is either a global minimum or a strict saddle (i.e., no spurious local minima). Gradient descent converges linearly as ℒ(t)−ℒ∗≤e−μt(ℒ(0)−ℒ∗)L(t)-L^*≤ e^-μ t(L(0)-L^*). Lemma 2 ensures Θ(1) (1) Hessian spread, enabling (1)O(1) escape from saddles, in contrast to softmax’s (n2)O(n^2) escape time. 2.7 Theorem 3: Capacity Control and Generalization The effective rank of a kernel matrix measures the number of significant singular values. A low effective rank implies that the kernel cannot memorize arbitrary patterns, which is beneficial for generalization. Theorem 2.8 (Theorem 3: IDA Effective Rank Bound). For embeddings with off-diagonal distances ≥dmin>0≥ d_ >0, if ϵ≪dmin2ε d_ ^2 (specifically, ϵ(n−1)/dmin2≤1/2ε(n-1)/d_ ^2≤ 1/2), then eff-rank()≤1+nε2dmin4,eff -rank(K)≤ 1+ n ^2d_ ^4, independent of hidden width dhd_h. In the general case: eff-rank()≤n(1−ϵ(n−1)/dmin2)2.eff -rank(K)≤ n(1-ε(n-1)/d_ ^2)^2. Proof. Let ∈ℝn×nK ^n× n with entries Kij=(‖i−j‖22+ε)−1K_ij=(\|q_i-k_j\|_2^2+ )^-1. The diagonal entries are Kii=1/εK_i=1/ . Define =−ε−1E=K- ^-1I. Then Eii=0E_i=0, and for i≠ji≠ j: Eij=1‖i−j‖22+ε−1ε=−‖i−j‖22ε(‖i−j‖22+ε).E_ij= 1\|q_i-k_j\|_2^2+ - 1 =- \|q_i-k_j\|_2^2 (\|q_i-k_j\|_2^2+ ). The absolute value is: |Eij|=‖i−j‖22ε(‖i−j‖22+ε).|E_ij|= \|q_i-k_j\|_2^2 (\|q_i-k_j\|_2^2+ ). Since ‖i−j‖22≥dmin2\|q_i-k_j\|_2^2≥ d_ ^2, we have: |Eij|≤1‖i−j‖22+ε≤1dmin2+ε≤1dmin2.|E_ij|≤ 1\|q_i-k_j\|_2^2+ ≤ 1d_ ^2+ ≤ 1d_ ^2. By Gershgorin’s circle theorem, every eigenvalue λm() _m(E) satisfies: |λm()|≤maxi∑j≠i|Eij|≤n−1dmin2.| _m(E)|≤ _i _j≠ i|E_ij|≤ n-1d_ ^2. Thus the eigenvalues of =ε−1+K= ^-1I+E satisfy: λm()∈[1ε−n−1dmin2,1ε+n−1dmin2]. _m(K)∈ [ 1 - n-1d_ ^2, 1 + n-1d_ ^2 ]. The effective rank is: eff-rank()=(tr)2tr(2)=n2/ε2n/ε2+‖F2.eff -rank(K)= (trK)^2tr(K^2)= n^2/ ^2n/ ^2+\|E\|_F^2. Using the bound ‖F2≤n(n−1)/dmin4\|E\|_F^2≤ n(n-1)/d_ ^4, we get: eff-rank()≤n1+ε2(n−1)/dmin4.eff -rank(K)≤ n1+ ^2(n-1)/d_ ^4. For the refined bound, using the sharper estimate |Eij|≤‖i−j‖22/(εdmin2)|E_ij|≤\|q_i-k_j\|_2^2/( d_ ^2): eff-rank()≤1+nε2dmin4+(nε3dmin6).eff -rank(K)≤ 1+ n ^2d_ ^4+O ( n ^3d_ ^6 ). This proves the theorem. ∎ Theorem 2.9 (Softmax Width Catastrophe). When dh≥nd_h≥ n, softmax can achieve zero training error on arbitrary labels, with test error →2η→ 2η under symmetric noise rate η. Proof. We construct an explicit construction. Let ii=1n\e_i\_i=1^n be an orthonormal basis in ℝdhR^d_h, so we require dh≥nd_h≥ n. Set i=dhiq_i= d_he_i and j=αdhjk_j=α d_he_j for some α>0α>0. Then the inner product is: i⊤j=αdhδij.q_i k_j=α d_h _ij. The softmax attention weights are: Wij=eαdhδij∑meαdhδim=eαdheαdh+(n−1),i=j,1eαdh+(n−1),i≠j.W_ij= e^α d_h _ij _me^α d_h _im= cases e^α d_he^α d_h+(n-1),&i=j,\\ 1e^α d_h+(n-1),&i≠ j. cases As α→∞α→∞, the weights converge to the identity matrix: Wij→δijW_ij→ _ij. Therefore the output is i=∑jWijj→io_i= _jW_ijv_j _i. The linear readout layer ∈ℝdvw ^d_v maps the output to the prediction: y^i=⊤i y_i=w o_i. If iv_i are linearly independent (true with probability 1 when dv≥nd_v≥ n and values are random), then there exists a w satisfying ⊤i=yiw v_i=y_i for all i. Thus zero training error can be achieved. Under label noise rate η, the training labels are random flips of the true labels. Zero training error implies the model has memorized the noisy labels. The test error is the expected error on clean labels, which equals the Bayes error under symmetric noise: η for binary classification or 2η2η for balanced multi-class classification. ∎ Theorem 2.10 (IDA Noise Robustness). For IDA, test error under symmetric noise rate η satisfies ℰtestIDA≤Cη2+(1/n),E_test^IDA≤ Cη^2+O(1/ n), independent of dhd_h. Proof. For IDA at exact match i=iq_i=k_i, the attention weight is: Wii=1/ε1/ε+∑j≠i(dij2+ε)−1.W_i= 1/ 1/ + _j≠ i(d_ij^2+ )^-1. Define αi=ε∑j≠i(dij2+ε)−1 _i= _j≠ i(d_ij^2+ )^-1. Then: Wii=11+αi=1−αi+(αi2).W_i= 11+ _i=1- _i+O( _i^2). For j≠ij≠ i: Wij=αin−1+(ε2n/dmin4).W_ij= _in-1+O( ^2n/d_ ^4). The prediction is: y^i=(1−αi)yi+αiy¯−i+(ε2n2/dmin4), y_i=(1- _i)y_i+ _i y_-i+O( ^2n^2/d_ ^4), where y¯−i y_-i is the average label of the neighbors. Taking expectations over label noise: [(yi−y^i)2]≤4αi2η2+(ε2n2/dmin4)+(1/n).E[(y_i- y_i)^2]≤ 4 _i^2η^2+O( ^2n^2/d_ ^4)+O(1/n). Summing over i and using αi≤εn/dmin2 _i≤ n/d_ ^2: ℰtestIDA≤Cη2+(1/n).E_test^IDA≤ Cη^2+O(1/ n). The bound is independent of dhd_h. ∎ 2.8 Discussion: High-Dimensional Behavior The analysis above reveals several important aspects of IDA’s behavior in high dimensions that distinguish it from softmax. Distance concentration. In high dimensions, Euclidean distances between random points concentrate around their mean. This means that for large dhd_h, the inverse-distance kernel assigns roughly equal weight to all keys, making the attention output close to the unweighted average of values. This is the same phenomenon that affects softmax in high dimensions (inner products concentrate around zero). However, the key difference is that IDA’s kernel explicitly depends on distances, which can be controlled by scaling the embeddings, while softmax’s inner products depend on norms that are less controllable. Effective rank and capacity. Theorem 3 shows that IDA’s effective rank is bounded independent of dhd_h, which prevents overfitting to noise. In high dimensions, this bound becomes tighter because the off-diagonal distances dmind_ increase with dimension for random embeddings. This suggests that IDA is particularly well-suited for high-dimensional data. Implications for practice. Based on the theoretical analysis, we recommend initializing ε to a small fraction of the typical squared distance (e.g., ε≈0.01⋅[‖i−j‖22] ≈ 0.01·E[\|q_i-k_j\|_2^2]) to avoid the degradation regime ε(n−1)/dmin2≥1/2 (n-1)/d_ ^2≥ 1/2. The analysis also suggests that for tasks with high noise rates η, using IDA with moderate ε (around 10−410^-4) provides better generalization than softmax. 3 Euclidean Summary: Table of Theoretical Guarantees Property Softmax Resolver (IDA) Circuit separation (Thm. 1) Ω((logn)2) (( n)^2) (1)O(1) Lipschitz scaling (Lemma 1) (n)O(n) (logn)O( n) under low-rank/(n)O(n) general Hessian spread (Lemma 2) Θ(n−2) (n^-2) Θ(1) (1) PL constant (Thm. 2) Θ(e−Δ2/dε2/Δ2) (e^- ^2/ d ^2/ ^2) Θ(ε2/Δ4) ( ^2/ ^4) Effective rank (Thm. 3) Unbounded ≤1+nε2/dmin4≤ 1+n ^2/d_ ^4 (small ϵε) Noise memorization (Thm. 3) At dh≥nd_h≥ n Structurally limited These five Euclidean results provide the theoretical backbone. The non-Euclidean extension that follows is based on the same inverse-distance principle, with geodesic distances replacing Euclidean norms. Part I Riemann GeoResolver: Non-Euclidean Extension 4 Hyperbolic Geometry Setup The Poincaré ball is dh=∈ℝdh:‖<1B^d_h=\x ^d_h:\|x\|<1\ with conformal factor λ=2/(1−‖2) _x=2/(1-\|x\|^2). The hyperbolic distance is: dℍ(,)=arcosh(1+2‖−‖2(1−‖2)(1−‖2)).d_H(x,y)=arcosh\! (1+ 2\|x-y\|^2(1-\|x\|^2)(1-\|y\|^2) ). All embeddings are projected via =0.9tanh()x=0.9 (z); results extend to ρ→1−ρ→ 1^- by continuity. The Karcher mean exists uniquely in CAT(0) spaces. The unit sphere is d−1=:‖=1S^d-1=\x:\|x\|=1\ with great-circle distance: d(,)=arccos(⟨,⟩).d_S(x,y)= ( ,y ). Throughout this part, we use the HIDA family to refer collectively to all hyperbolic inverse-distance attention operators, and Dense-HIDA, FP-HIDA, L-HIDA, C-HIDA for the specific instances. 5 The HIDA Operator Family (M1–M4) 5.1 A Unified Inverse-Distance Kernel Lemma Lemma 5.1 (Unified Kernel Spectral Properties). Let ℳM be a metric space and Kij=(d(xi,yj)2+ϵ)−1K_ij=(d(x_i,y_j)^2+ε)^-1 with: (i) d(xi,xi)=0d(x_i,x_i)=0, (i) d(xi,yj)≥dmin>0d(x_i,y_j)≥ d_ >0 for i≠ji≠ j, (i) d locally Lipschitz. Then: (a) if xi=yj∗x_i=y_j^*, limϵ→0Kij∗/∑mKim=1 _ε→ 0K_ij^*/ _mK_im=1; (b) if ϵ≪dmin2ε d_ ^2 (specifically, ϵ(n−1)/dmin2≤1/2ε(n-1)/d_ ^2≤ 1/2), then eff-rank()≤1+nϵ2dmin4+(nϵ3dmin6).eff -rank(K)≤ 1+ nε^2d_ ^4+O ( nε^3d_ ^6 ). In the general case (without the small-ϵε condition), the bound is: eff-rank()≤n(1−ϵ(n−1)/dmin2)2.eff -rank(K)≤ n(1-ε(n-1)/d_ ^2)^2. Proof. Part (a) follows from Kij∗=1/ϵK_ij^*=1/ε and Kij≤1/(dmin2+ϵ)K_ij≤ 1/(d_ ^2+ε) for j≠j∗j≠ j^*, so the normalized weight tends to 11 as ϵ→0ε→ 0. Part (b) follows from the Frobenius norm estimate: ‖F2≤n(n−1)(dmin2+ϵ)2.\|E\|_F^2≤ n(n-1)(d_ ^2+ε)^2. The general bound follows from the effective rank formula. The refined bound requires ϵ≪dmin2ε d_ ^2 and is obtained by the sharper Frobenius estimate. ∎ 5.2 Why Hyperbolic? Three Arguments (A) Information-theoretic capacity. For fixed dimension d, the hyperbolic ball volume Vℍ(R)∝e(d−1)RV_H(R) e^(d-1)R grows exponentially with radius, while Euclidean volume V(R)∝RdV_E(R) R^d grows polynomially. This means hyperbolic spaces can embed hierarchical data with exponentially fewer dimensions. (B) Effective rank behavior. Theorem 3 established the bound dmin≥2δ/(1−ρ2)d_ ≥ 2δ/(1-ρ^2); as ρ→1−ρ→ 1^- (approaching the boundary of the Poincaré ball), this grows without bound. Thus hyperbolic embeddings can achieve arbitrarily large separation between keys without increasing the ambient dimension. (C) Architectural separation. Proposition 2 (Section 10) establishes an information-theoretic gap between hyperbolic storage and spherical routing. 5.3 Dense-HIDA (M1) Definition 5.1 (Dense Hyperbolic IDA). The dense hyperbolic inverse-distance attention is defined as: Dijℍ=dℍ(i,j)2+ε,Wij=(Dijℍ)−1/∑m(Dimℍ)−1,i=∑jWijj.D_ij^H=d_H(q_i,k_j)^2+ , W_ij=(D_ij^H)^-1/ _m(D_im^H)^-1, _i= _jW_ijv_j. The computational complexity is Θ(n2dh) (n^2d_h). Theorem 5.2 (Circuit Separation and Rank Bound). For pairwise distinct keys in a compact subset with ‖≤ρ<1\|x\|≤ρ<1, Dense-HIDA satisfies: (1) limε→0Wj∗=1 _ → 0W_j^*=1 for exact retrieval; (2) eff-rank(ℍ)≤1+nε2dmin4eff -rank(K^H)≤ 1+ n ^2d_ ^4, with dmin≥2δ/(1−ρ2)d_ ≥ 2δ/(1-ρ^2). Proof. Claim (1) follows directly from Lemma 0 Part (a) with d=dℍd=d_H. Claim (2) follows from Lemma 0 Part (b). The lower bound on dmind_ follows from the hyperbolic distance lower bound: dℍ(,)≥‖−‖21−ρ2.d_H(x,y)≥ \|x-y\|_21-ρ^2. This inequality holds for any ,∈dhx,y ^d_h with ‖,‖≤ρ\|x\|,\|y\|≤ρ. ∎ Theorem 5.3 (Two-Point Hyperbolic PL Inequality). For the two-key loss with keys separated by Δℍ>0 _H>0: μHIDA=Θ(ε2/Δℍ4),μsoft=Θ(e−Δℍ2/dhε2/Δℍ2). _HIDA= ( ^2/ _H^4), _soft= (e^- _H^2/ d_h ^2/ _H^2). Thus μHIDA/μsoft=Ω(eΔℍ2/dh/Δℍ2) _HIDA/ _soft= (e _H^2/ d_h/ _H^2). Proof. By Möbius invariance of the Poincaré distance, we may place 1=0k_1=0 and 2=(Δℍ,0,…,0)k_2=( _H,0,…,0) in the Poincaré ball. For q on the geodesic, the distance expansions are: dℍ(q,1)2=q2+(q4),d_H(q,k_1)^2=q^2+O(q^4), dℍ(q,2)2=Δℍ2−2Δℍq+q2+(q3).d_H(q,k_2)^2= _H^2-2 _Hq+q^2+O(q^3). Substituting these into the attention weight formula and following the same expansion as Theorem 2 yields W1′(0)=0W_1 (0)=0, W1′(0)=−4ε/Δℍ4+(ε2/Δℍ6)W_1 (0)=-4 / _H^4+O( ^2/ _H^6), and hence μHIDA=Θ(ε2/Δℍ4) _HIDA= ( ^2/ _H^4). ∎ 5.4 FP-HIDA: Fixed-Pattern Sparse (M2) Definition 5.2 (Sparse Index Set). Define the sparse index set: i=j:|i−j|≤w∪0,⌊n/g⌋,⌊2n/g⌋,…∪i±2kk=0⌈log2n⌉∪i,S_i=\j:|i-j|≤ w\∪\0, n/g , 2n/g ,…\∪\i± 2^k\_k=0 _2n ∪\i\, with w,g=Θ(logn)w,g= ( n). Theorem 5.4 (FP-HIDA Complexity). FP-HIDA requires (nlogn⋅dh)O(n n· d_h) operations and (nlogn)O(n n) memory. Proof. We bound the size of each sparse index set. The local window contributes at most 2w+12w+1 indices. The global anchors contribute exactly g indices. The dyadic offsets contribute at most 2(⌈log2n⌉+1)2( _2n +1) indices. The self index contributes 11. Therefore: |i|≤(2w+1)+g+2(⌈log2n⌉+1)+1.|S_i|≤(2w+1)+g+2( _2n +1)+1. Since w=Θ(logn)w= ( n) and g=Θ(logn)g= ( n), we have |i|=(logn)|S_i|=O( n). Summing over all i: ∑i=1n|i|≤n⋅(logn)=(nlogn). _i=1^n|S_i|≤ n·O( n)=O(n n). Each entry in the sparse index set requires Θ(dh) (d_h) operations to compute the distance and attention weight. Thus the total computational complexity is (nlogn⋅dh)O(n n· d_h). The memory complexity is (nlogn)O(n n) for storing the sparse attention pattern. ∎ 5.5 L-HIDA: Linear Complexity (M3) Definition 5.3 (L-HIDA Anchor Aggregation). Let =kk=1m⊂dhA=\a_k\_k=1^m ^d_h be m=Θ(1)m= (1) learnable anchors. Define the key-anchor and query-anchor weights: W~ikKA=(dℍ(i,k)2+ε)−1∑l=1m(dℍ(i,l)2+ε)−1,~k=∑i=1nW~ikKAi, W_ik^KA= (d_H(k_i,a_k)^2+ )^-1 _l=1^m(d_H(k_i,a_l)^2+ )^-1, v_k= _i=1^n W_ik^KAv_i, W~jkQA=(dℍ(j,k)2+ε)−1∑l=1m(dℍ(j,l)2+ε)−1,j=∑k=1mW~jkQA~k. W_jk^QA= (d_H(q_j,a_k)^2+ )^-1 _l=1^m(d_H(q_j,a_l)^2+ )^-1, _j= _k=1^m W_jk^QA v_k. Theorem 5.5 (L-HIDA Complexity and Error Bound). L-HIDA requires (ndh)O(nd_h) operations. With probability at least 1−δ1-δ over the random selection of anchors (standard Nyström sampling [63]): ‖L-HIDA−HIDA‖F≤(nmε2).\|o^L-HIDA-o^HIDA\|_F ( n m ^2 ). If the kernel has spectral decay λk=(k−α) _k=O(k^-α) with α>1/2α>1/2: ‖L-HIDA−HIDA‖F≤(n⋅m−α+1/2)+(nε2m).\|o^L-HIDA-o^HIDA\|_F (n· m^-α+1/2)+O ( n ^2 m ). Proof. Let nn∈ℝn×nK_n ^n× n be the dense kernel matrix, nm∈ℝn×mK_nm ^n× m the key-anchor kernel matrix, and mm∈ℝm×mK_m ^m× m the anchor-anchor kernel matrix. The Nyström approximation is ~nn=nmmm−1mn K_n=K_nmK_m^-1K_mn. The error decomposes as: nn−~nn=(nn−nn(m))+(nn(m)−~nn),K_n- K_n=(K_n-K_n^(m))+(K_n^(m)- K_n), where nn(m)K_n^(m) is the best rank-m approximation to nnK_n in Frobenius norm. For the first term, if the kernel has spectral decay λk=(k−α) _k=O(k^-α) with α>1/2α>1/2, then: ‖nn−nn(m)‖F2=∑k=m+1nλk2=(m−2α+1).\|K_n-K_n^(m)\|_F^2= _k=m+1^n _k^2=O(m^-2α+1). For the second term, standard Nyström analysis [63] gives: ‖nn(m)−~nn‖F≤‖mm−1‖2‖nm‖F‖F,\|K_n^(m)- K_n\|_F≤\|K_m^-1\|_2\|K_nm\|_F\|E\|_F, where E is the residual matrix. With ‖mm−1‖2=(1/ε)\|K_m^-1\|_2=O(1/ ), ‖nm‖F=(n/ε)\|K_nm\|_F=O(n/ ), and ‖F=(1/m)\|E\|_F=O(1/ m), this yields: ‖nn(m)−~nn‖F=(nε2m).\|K_n^(m)- K_n\|_F=O ( n ^2 m ). The output error bound follows from: ‖L-HIDA−HIDA‖F≤2‖Fε‖nn−~nn‖F.\|o^L-HIDA-o^HIDA\|_F≤ 2\|V\|_F \|K_n- K_n\|_F. Combining the two error terms completes the proof. ∎ 5.6 C-HIDA: Constant Complexity (M4) Definition 5.4 (C-HIDA Summary Tokens). Maintain c=Θ(1)c= (1) summary tokens kk=1c\s_k\_k=1^c. For each incoming key, assign to the nearest summary: k∗(i)=argmink∈[c]dℍ(i,k).k^*(i)= _k∈[c]d_H(k_i,s_k). Update each summary token using the hyperbolic mean: k←Π(k+η⋅1|ℐk|∑i∈ℐk(i−k)),s_k← _B (s_k+η· 1|I_k| _i _k(k_i-s_k) ), where Π _B is the projection onto the Poincaré ball. Theorem 5.6 (C-HIDA Convergence). The online hyperbolic k-means update has (clogT)O(c T) regret: ∑t=1Tminkdℍ(t,k(t))2≤(Δℍ2⋅c⋅logT)+(T⋅εopt). _t=1^T _kd_H(k_t,s_k^(t))^2 ( _H^2· c· T)+O(T· _opt). Per-token attention cost is Θ(1) (1). Proof. The Poincaré ball with hyperbolic metric is a CAT(0) space, so the Karcher mean (the minimizer of the sum of squared distances) exists uniquely. The online gradient descent update on the squared hyperbolic distance is a convex optimization problem in CAT(0) spaces. The standard regret bound for online convex optimization applies: ∑t=1Tft((t))−min∑t=1Tft()=(logT), _t=1^Tf_t(s^(t))- _s _t=1^Tf_t(s)=O( T), where ft()=minkdℍ(t,k)2f_t(s)= _kd_H(k_t,s_k)^2. The logT T term arises from the online-to-batch conversion for k-means. The per-token attention cost is Θ(1) (1) because only c=Θ(1)c= (1) summaries are used. ∎ 6 Hyperbolic Curvature Compression (M5) For any key ∈dhk ^d_h, define its polar decomposition =r⋅k=r·u where r∈[0,1)r∈[0,1) is the radius and ∈dh−1u ^d_h-1 is the direction. Given bit-widths b (for direction) and brb_r (for radius): ^=round(+12(2b−1))/(2b−1)⋅2−1,r^=round(r⋅2br)2br. u=round ( u+12(2^b-1) )/(2^b-1)· 2-1, r= round(r· 2^b_r)2^b_r. Theorem 6.1 (HCC Reconstruction Error). The reconstruction error satisfies: ‖−~‖22≤4(2−b+2−br).\|k- k\|_2^2≤ 4(2^-b+2^-b_r). Proof. For the directional component, the quantization error is bounded by: ‖−^‖2≤2⋅2−b.\|u- u\|_2≤ 2· 2^-b. This follows from the covering radius of the uniform grid on [−1,1]dh−1[-1,1]^d_h-1 restricted to the sphere. For the radius component: |r−r^|≤2−br,|r- r|≤ 2^-b_r, since the rounding error is at most half the bin width. Combining via the triangle inequality: ‖r−r^^‖2≤r‖−^‖2+|r−r^|‖^‖2≤2⋅2−b+2−br.\|ru- r u\|_2≤ r\|u- u\|_2+|r- r|\| u\|_2≤ 2· 2^-b+2^-b_r. Squaring gives: ‖−~‖22≤(2⋅2−b+2−br)2≤4(2−b+2−br).\|k- k\|_2^2≤(2· 2^-b+2^-b_r)^2≤ 4(2^-b+2^-b_r). This proves the bound. ∎ Theorem 6.2 (HCC Memory Complexity). The HCC-compressed KV cache requires n((dh−1)b+br)n((d_h-1)b+b_r) bits for keys plus ndh⋅32nd_h· 32 bits for values. With b=4,br=6b=4,b_r=6, the theoretical compression ratio is ≈8×≈ 8× for keys, ≈1.8×≈ 1.8× system-wide (or ≈2.4×≈ 2.4× with half-precision values). Proof. The key compression ratio is: 32dh(dh−1)b+br. 32d_h(d_h-1)b+b_r. With dh=32d_h=32, b=4b=4, br=6b_r=6: 102431⋅4+6=1024130≈7.88×. 102431· 4+6= 1024130≈ 7.88×. For the system-wide compression (keys + values), using 32-bit values: 64dh(dh−1)b+br+32dh=2048130+1024≈1.77×. 64d_h(d_h-1)b+b_r+32d_h= 2048130+1024≈ 1.77×. With 16-bit values: 48dh(dh−1)b+br+16dh=1536130+512≈2.39×. 48d_h(d_h-1)b+b_r+16d_h= 1536130+512≈ 2.39×. ∎ 7 HyperGate: Curvature-Adaptive Gating (M6) Three-level gating: head-level κh _h, token-level boundary βi _i, dimension-level i=diag(σ(gi))G_i=diag(σ(W_gx_i)). The gated representation is i=iih_i=G_ix_i. Theorem 7.1 (HyperGate Gradient Lower Bound). Let ℒL be the loss function and i=iih_i=G_ix_i be the gated representation. Then: ‖∂ℒ∂i‖2≥(λmin(i)−(‖g‖‖i‖))‖∂ℒ∂i‖2. \| _i \|_2≥ ( _ (G_i)-O(\|W_g\|\|x_i\|) ) \| _i \|_2. In particular, for bounded ix_i and gW_g, the lower bound is strictly positive, so gradients do not vanish due to gating alone. Proof. By the chain rule: ∂ℒ∂i=(∂i∂i)⊤∂ℒ∂i. _i= ( _i _i ) _i. Since i=iih_i=G_ix_i and i=diag(σ(gi))G_i=diag(σ(W_gx_i)), the Jacobian is: ∂i∂i=i+(∂i∂i)i. _i _i=G_i+ ( _i _i )x_i. The diagonal entries of iG_i are σ(zk)σ(z_k) where zk=(gi)kz_k=(W_gx_i)_k, and σ′(zk)=σ(zk)(1−σ(zk))∈(0,1/4)σ (z_k)=σ(z_k)(1-σ(z_k))∈(0,1/4). Thus: (∂i∂i)kk=σ(zk)+σ′(zk)g,k⊤i. ( _i _i )_k=σ(z_k)+σ (z_k)W_g,k x_i. The matrix ∂i/∂i _i/ _i is diagonally dominant for bounded ‖i‖\|x_i\| and ‖g‖\|W_g\|. Its smallest singular value satisfies: σmin(∂i∂i)≥λmin(i)−(‖g‖‖i‖). _ ( _i _i )≥ _ (G_i)-O(\|W_g\|\|x_i\|). Since λmin(i)=minkσ(zk)>0 _ (G_i)= _kσ(z_k)>0 for finite zkz_k, the lower bound is strictly positive for bounded inputs. Therefore: ‖∂ℒ∂i‖2≥σmin(∂i∂i)‖∂ℒ∂i‖2. \| _i \|_2≥ _ ( _i _i ) \| _i \|_2. This proves the gradient lower bound. ∎ 8 Spherical Routing and Retrieval (M7+M8) 8.1 SIDA: Spherical Inverse Distance Attention Definition 8.1 (SIDA Kernel). The spherical inverse-distance attention is defined as: Dij=d(i,j)2+ε,Wij=(Dij)−1/∑m(Dim)−1.D_ij^S=d_S(q_i,k_j)^2+ , W_ij^S=(D_ij^S)^-1/ _m(D_im^S)^-1. Theorem 8.1 (Spherical PL Inequality). For two keys with spherical angle θ>0θ>0: μSIDA=Θ(ε2/θ4),μsoft=Θ(e−(1−cosθ)ε2/θ2). _SIDA= ( ^2/θ^4), _soft= (e^-(1- θ) ^2/θ^2). Thus μSIDA/μsoft=Ω(e1−cosθ/θ2) _SIDA/ _soft= (e^1- θ/θ^2). Unlike the hyperbolic case, spherical compactness ensures θ≤πθ≤π, so the ratio is bounded by a constant depending only on θ. Proof. Place the keys at spherical coordinates 1=(1,0,…,0)k_1=(1,0,…,0) and 2=(cosθ,sinθ,0,…,0)k_2=( θ, θ,0,…,0). For a query at distance t from 1k_1 along the geodesic toward 2k_2, we have dS(,1)=td_S(q,k_1)=t and dS(,2)=θ−td_S(q,k_2)=θ-t. The weight W1(t)W_1(t) has the same form as the 1D Euclidean case with Δ=θ =θ, giving W1′(0)=0W_1 (0)=0 and W1′(0)=−4ε/θ4+(ε2/θ6)W_1 (0)=-4 /θ^4+O( ^2/θ^6). The loss curvature gives μSIDA=Θ(ε2/θ4) _SIDA= ( ^2/θ^4). For softmax, the logit difference at t=0t=0 is s(0)=(1−cosθ)/dhs(0)=(1- θ)/ d_h, giving the exponential factor. ∎ 8.2 Cross-Geometric Mapping Three methods map prototype centroids to the sphere: - Norm Normalization: ϕA()=/‖ _A(c)=c/\|c\| - Stereographic Projection: ϕB()=(21+‖2,1−‖21+‖2) _B(c)=( 2c1+\|c\|^2, 1-\|c\|^21+\|c\|^2) - Learnable Mapping: ϕC()=MLPθ()/‖MLPθ()‖ _C(c)=MLP_θ(c)/\|MLP_θ(c)\| 8.3 Routing Protocol Hard Routing: Select Top-K prototypes by SIDA weights. Soft Routing: full distribution. 9 Dynamic Memory Genesis (M9) The prototype pool is =(ℰe,e,tebirth,ae,telast)e=1KP=\(E_e,c_e,t_e^birth,a_e,t_e^last)\_e=1^K. 9.1 Surprise Detection with Adaptive Threshold Define the sliding-window mean and standard deviation: μt=1W∑i=t−W+1tℓi,σt=1W∑i=t−W+1t(ℓi−μt)2. _t= 1W _i=t-W+1^t _i, _t= 1W _i=t-W+1^t( _i- _t)^2. The adaptive threshold is: τt=τbase+κσt+γ⋅1t∑i=1tsurprise(i). _t= _base+κ _t+γ· 1t _i=1^t1_surprise(i). Lemma 9.1 (Adaptive Threshold Regret Bound under Sub-Gaussian Loss). Assume the loss ℓt _t is sub-Gaussian with mean μ0 _0 and variance proxy σ02 _0^2 under the null model (i.e., when no structural change occurs). For τt _t as defined with τbase>μ0 _base> _0, we have [Ts(T)]≤(logT)E[T_s(T)] ( T). Proof. Let St=∑i=1tsurprise(i)S_t= _i=1^t1_surprise(i) be the cumulative surprise count. Define the supermartingale: Mt=St+1γ∑i=1t(τi−τbase−κσi).M_t=S_t+ 1γ _i=1^t( _i- _base-κ _i). Under the null model, [ℓt]=μ0E[ _t]= _0 and [σt]=σ0E[ _t]= _0. The trigger condition μt>τt _t> _t implies that a surprise event occurs only when the loss exceeds the threshold. The adaptive term γ⋅St/tγ· S_t/t penalizes frequent surprises, creating a negative drift. For sub-Gaussian losses, the probability of exceeding τbase+κσt _base+κ _t decays exponentially, leading to the logarithmic bound. Summing yields [ST]=(logT)E[S_T]=O( T). ∎ 9.2 Convergence Guarantee Between spawns, standard C-HIDA regret applies. Under the lemma, total regret is (logT)O( T). 10 Geodesic Sparse Routing (GSR): Sparse Routing with Quality Guarantees So far, SIDA (Section 7) has provided a mechanism for computing attention weights on the sphere. However, in distributed settings, routing every query to all KpoolK_pool prototypes is expensive. We now introduce a sparse routing mechanism that selects only the Top-K prototypes by SIDA weight, with provable bounds on both routing quality and communication cost. 10.1 The GSR Mechanism Let =ee=1Kpool⊂d−1P=\c_e\_e=1^K_pool ^d-1 be a pool of learnable prototype vectors. For a query ∈d−1q ^d-1, SIDA defines weights: we=W,e=(d(,e)2+ε)−1∑l=1Kpool(d(,l)2+ε)−1.w_e=W_q,c_e^S= (d_S(q,c_e)^2+ )^-1 _l=1^K_pool(d_S(q,c_l)^2+ )^-1. GSR selects the Top-K prototypes with largest weights and routes the query only to those prototypes: ()=e∈[Kpool]:we is among the K largest weights.S(q)=\e∈[K_pool]:w_e is among the K largest weights\. The output is the weighted combination of the values associated with the selected prototypes: ()=∑e∈()wee.o(q)= _e (q)w_ev_e. 10.2 Routing Quality Guarantee The following theorem establishes that GSR’s routing quality is controlled by the SIDA weight gap between the Top-K and the remaining prototypes. Theorem 10.1 (GSR Routing Quality). Let w(1)≥w(2)≥⋯≥w(Kpool)w_(1)≥ w_(2)≥·s≥ w_(K_pool) be the SIDA weights sorted in descending order. For any query, the approximation error from routing only to the Top-K prototypes is bounded by: ‖()−∗()‖2≤2‖F⋅∑e>Kw(e)∑e≤Kw(e)⋅(maxe≤K‖e‖2),\|o(q)-o^*(q)\|_2≤ 2\|V\|_F· _e>Kw_(e) _e≤ Kw_(e)· ( _e≤ K\|v_e\|_2 ), where ∗()o^*(q) is the exact SIDA output using all KpoolK_pool prototypes. Moreover, as ε→0+ → 0^+, the SIDA weights concentrate on the nearest prototype, so for any δ>0δ>0, there exists ε0>0 _0>0 such that for all ε<ε0 < _0: ‖()−∗()‖2≤δ\|o(q)-o^*(q)\|_2≤δ provided the query has a unique nearest prototype and ‖e‖2\|v_e\|_2 is bounded. Proof. Let the exact SIDA output be ∗=∑e=1Kpoolweeo^*= _e=1^K_poolw_ev_e, and the GSR output be =∑e≤Kweeo= _e≤ Kw_ev_e (where we reorder indices so the Top-K are 1,…,K1,…,K). Then the difference is: ‖∗−‖2=‖∑e>Kwee‖2.\|o^*-o\|_2= \| _e>Kw_ev_e \|_2. By the triangle inequality: ‖∗−‖2≤∑e>Kwe‖e‖2.\|o^*-o\|_2≤ _e>Kw_e\|v_e\|_2. Since ‖e‖2≤‖F\|v_e\|_2≤\|V\|_F for all e, we have: ‖∗−‖2≤‖F∑e>Kwe.\|o^*-o\|_2≤\|V\|_F _e>Kw_e. The sum of the Top-K weights is ∑e≤Kwe=1−∑e>Kwe _e≤ Kw_e=1- _e>Kw_e. Thus ∑e>Kwe=1−∑e≤Kwe _e>Kw_e=1- _e≤ Kw_e. The ratio bound follows by observing that ∑e>Kwe≤∑e>Kwe∑e≤Kwe _e>Kw_e≤ _e>Kw_e _e≤ Kw_e since ∑e≤Kwe≤1 _e≤ Kw_e≤ 1. For the second part, recall from the exact retrieval property (Lemma 0, Part (a)) that if =e∗q=c_e^* for a unique nearest prototype, then limε→0+we∗=1 _ → 0^+w_e^*=1. Hence limε→0+∑e>Kwe=0 _ → 0^+ _e>Kw_e=0 for any K≥1K≥ 1 (since all weight concentrates on the single nearest prototype). Therefore the approximation error can be made arbitrarily small by choosing ε sufficiently small. ∎ 10.3 Communication Complexity Analysis GSR is designed for distributed settings where prototypes are distributed across P workers. We analyze the communication cost. Let the KpoolK_pool prototypes be partitioned among P workers. Each worker holds approximately Kpool/PK_pool/P prototypes. For a batch of B queries, the protocol is: 1. Each worker computes SIDA weights for its local prototypes: local computation, no communication. 2. The Top-K indices are gathered across workers: All-Gather on the top scores from each worker. 3. Selected prototypes compute outputs: local computation. 4. Results are aggregated: All-Reduce on the K selected outputs. Theorem 10.2 (GSR Communication Complexity). GSR has per-query communication cost: (Kpool⋅dh+K⋅dh),O(K_pool· d_h+K· d_h), independent of batch size B. In the α-β model, this is: #messages=(Kpool+K+P),bytes=((Kpool+K)dh⋅precision).\#messages=O(K_pool+K+P), =O((K_pool+K)d_h·precision). In contrast, All-to-All MoE has: #messages=(P2),bytes=(BKdh⋅precision),\#messages=O(P^2), =O(BKd_h·precision), which scales linearly with batch size B. Proof. Each of the P workers must communicate its local top scores to all other workers. The All-Gather on scores of size (Kpool/P⋅dh)O(K_pool/P· d_h) per worker requires (Kpool⋅dh)O(K_pool· d_h) total communication. The All-Reduce on the K selected outputs requires (K⋅dh)O(K· d_h) communication. The number of messages is (Kpool+K+P)O(K_pool+K+P): KpoolK_pool for the score gathering, K for the result aggregation, and P for the initial scatter. The byte count follows from the dimension dhd_h and the floating-point precision. ∎ 10.4 Relation to Other Modules GSR is the routing component that follows SIDA. While SIDA computes all-to-all attention weights on the sphere, GSR uses those weights to select a sparse subset for communication and computation. DMG (Section 8) populates the prototype pool dynamically; GSR routes to the prototypes that DMG has allocated. This creates a complete pipeline: DMG allocates prototypes, SIDA computes spherical attention, and GSR routes queries to a sparse subset with quality guarantees. 11 Unified Framework (M10) 11.1 Global Architecture →QKV→HIDA path→mainSIDA→GSR→DMG→memory→HyperGate→final.x → casesHIDA path _main\\ SIDA _memory cases _final. 11.2 Unified Mathematical Language All variants share the same core structure: i=∑j∈ℛiWijj,Wij=(d(i,j)2+ε)−1∑m∈ℛi(d(i,m)2+ε)−1,o_i= _j _iW_ijv_j, W_ij= (d(q_i,k_j)^2+ )^-1 _m _i(d(q_i,k_m)^2+ )^-1, where ℛiR_i varies by mode and d is Euclidean, hyperbolic, or spherical. 11.3 Proposition 1: Complexity Observations for HIDA Variants For each HIDA variant, the computational complexity is: • Dense-HIDA: complexity Θ(n2dh) (n^2d_h), exact kernel. • FP-HIDA: complexity (nlogn⋅dh)O(n n· d_h) with w,g=Θ(logn)w,g= ( n). • L-HIDA: complexity (ndh)O(nd_h) with m=Θ(1)m= (1) anchors. • C-HIDA: complexity Θ(1) (1) per token with c=Θ(1)c= (1) summaries. This establishes a trade-off: sparser approximations reduce complexity at the cost of approximation error. The exact error rates depend on the data distribution and kernel spectrum. 11.4 Proposition 2: Information-Theoretic Justification Two key information-theoretic bounds separate hyperbolic storage from spherical routing: I(S;e)≤logKpool,I(;e)∝log11−‖e‖2.I(q_S;c_e^S)≤ K_pool, I(k;c_e) 11-\|c_e\|^2. Proof. The mutual information bound for spherical routing follows from the entropy bound H(e)≤logKpoolH(c_e^S)≤ K_pool. For hyperbolic storage, the volume growth Vℍ(R)∝e(dh−1)RV_H(R) e^(d_h-1)R gives the divergence as ‖e‖→1−\|c_e\|→ 1^-. ∎ 11.5 Proposition 3: DMG Scalability Observations The DMG system has two separate properties: 1. The online k-means update via C-HIDA achieves (logT)O( T) regret relative to the optimal fixed centroids. 2. With KmaxK_ memory slots, the quantization error decays as (1/Kmax)O(1/ K_ ) under compactness assumptions (data bounded away from the hyperbolic boundary). These properties hold independently and are not coupled in the regret bound. 12 Discussion and Limitations Scope. This is a theoretical study. The primary contribution is the theoretical framework. 12.1 Limitations No experimental validation. This paper is purely theoretical. The theorems establish mathematical guarantees, but empirical verification on benchmarks is not provided. This is left for future work. Compression scope. HCC compresses keys only; values remain full-precision. Routing communication. The communication analysis is theoretical under an idealized FLOPs model. DMG hyperparameters. The adaptive threshold requires tuning; the (logT)O( T) regret bound relies on a sub-Gaussian loss assumption. Curvature in theory. The two-point PL analysis is a limitation; multi-point effects are future work. 12.2 Future Work Key directions: (1) value compression for HCC; (2) distributed routing implementation; (3) mixed-curvature extensions; (4) multi-point curvature analysis; (5) empirical validation on standard benchmarks. 13 Conclusion We have presented a theoretical arc from Euclidean Resolver to Riemann GeoResolver. The Euclidean part establishes three core theorems with complete proofs, and the non-Euclidean part extends the framework to hyperbolic and spherical geometries. Summary of Euclidean contributions. Theorem 1 (Circuit Separation) established that IDA achieves exact retrieval with (1)O(1) resources, while any softmax-based architecture requires super-constant width. This shows a fundamental gap in the representational power of the two attention mechanisms. Theorem 2 (PL Inequality) proved that IDA satisfies a Polyak–Łojasiewicz inequality with constant Θ(ε2/Δ4) ( ^2/ ^4), while softmax has constant Θ(e−Δ2/dε2/Δ2) (e^- ^2/ d ^2/ ^2). The ratio is exponentially large in Δ2/d ^2/ d, implying that IDA converges linearly and dramatically faster for separated keys. The theorem also implied the absence of spurious local minima and established Θ(1) (1) Hessian spread. Theorem 3 (Effective Rank) established that IDA’s effective rank is bounded independent of hidden width, preventing the memorization of arbitrary noisy labels that occurs in softmax when dh≥nd_h≥ n. The theorem also provided a noise robustness bound ℰtestIDA≤Cη2+(1/n)E_test^IDA≤ Cη^2+O(1/ n). Summary of non-Euclidean contributions. The Riemann GeoResolver framework extends the Euclidean prototype to hyperbolic and spherical geometries: • HIDA family (M1–M4): Four hyperbolic attention operators spanning Θ(n2) (n^2) to Θ(1) (1) complexity, with circuit separation and PL inequalities proved for the dense case. • Hyperbolic Curvature Compression (M5): Provable reconstruction error bound ‖−~‖22≤4(2−b+2−br)\|k- k\|_2^2≤ 4(2^-b+2^-b_r) and memory complexity analysis. • HyperGate (M6): Gradient lower-bound theorem ensuring non-vanishing gradients through curvature-adaptive gating. • Spherical Inverse Distance Attention (M7–M8): Sphere-analog PL inequalities and cross-geometric mapping methods. • Dynamic Memory Genesis (M9): (logT)O( T) regret bound for online prototype allocation. • Geodesic Sparse Routing (M10): Routing quality and communication complexity bounds. Theoretical significance. This work establishes that inverse-distance attention is not merely an empirical alternative to softmax, but a mechanism with fundamentally different theoretical properties. The three Euclidean theorems—circuit separation, PL inequality, and effective rank bound—collectively show that IDA is provably stronger than softmax in expressiveness, optimization, and generalization. The non-Euclidean extension shows that these advantages are preserved and amplified in hyperbolic and spherical geometries. Open directions. The limitations of this work suggest several directions for future investigation. First, the PL analysis is restricted to the two-key case; extending to multi-key settings would provide stronger optimization guarantees. Second, the empirical validation remains future work. Third, the compression analysis is limited to keys; extending to values would improve memory efficiency. Fourth, the distributed routing analysis is theoretical; implementation and scaling on real systems would validate the communication bounds. Fifth, mixed-curvature spaces combining hyperbolic and spherical geometries could provide even more flexible representations. Acknowledgments The author gratefully acknowledges Prof. Junxue Zhang at the University of Science and Technology of China for his valuable guidance and insightful discussions throughout this research. References [1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS). [2] Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning (ICML), PMLR 119:5156–5165. [3] Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS). [4] Dao, T. (2023). FlashAttention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. [5] Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. [6] Beltagy, I., Peters, M. E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. [7] Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer. In International Conference on Learning Representations (ICLR). [8] Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. [9] Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2021). Rethinking attention with performers. In International Conference on Learning Representations (ICLR). [10] Bello, I., Zoph, B., Vaswani, A., Shlens, J., and Le, Q. V. (2019). Attention augmented convolutional networks. arXiv preprint arXiv:1905.10863. [11] Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018). Image transformer. In International Conference on Machine Learning (ICML), PMLR 80:4055–4064. [12] McCarter, C. (2023). Inverse distance weighting attention. arXiv preprint arXiv:2310.18805. [13] Ge, L. (2026). GeoResolver: Optimization geometry and capacity bounds for inverse distance attention. Under review. [14] Nadaraya, E. A. (1964). On estimating regression. Theory of Probability & Its Applications, 9(1):141–142. [15] Watson, G. S. (1964). Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, 26(4):359–372. [16] Shepard, R. N. (1968). Towards a universal law of generalization for psychological science. Science, 162(3850):1343–1352. [17] Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558. [18] Krotov, D. and Hopfield, J. J. (2016). Dense associative memory for pattern recognition. In Advances in Neural Information Processing Systems (NeurIPS). [19] Ambrogioni, L. (2023). The statistical thermodynamics of generative diffusion models. arXiv preprint arXiv:2302.03615. [20] Nickel, M. and Kiela, D. (2017). Poincaré embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems (NeurIPS). [21] Nickel, M. and Kiela, D. (2018). Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In International Conference on Machine Learning (ICML), PMLR 80:3779–3788. [22] Ganea, O., Bécigneul, G., and Hofmann, T. (2018). Hyperbolic neural networks. In Advances in Neural Information Processing Systems (NeurIPS). [23] Chami, I., Ying, Z., Ré, C., and Leskovec, J. (2019). Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS). [24] Gulcehre, C., Denil, M., Malinowski, M., Razavi, A., Pascanu, R., Hermann, K. M., Battaglia, P., Bapst, V., Raposo, D., Santoro, A., and de Freitas, N. (2020). Hyperbolic attention networks. In International Conference on Learning Representations (ICLR). [25] Shimizu, R., Mukuta, Y., and Harada, T. (2021). Hyperbolic neural networks++. In International Conference on Learning Representations (ICLR). [26] Chen, W., Han, X., Lin, Y., Zhao, H., Liu, Z., Li, P., Sun, M., and Zhou, J. (2021). Fully hyperbolic neural networks. arXiv preprint arXiv:2105.14686. [27] Liu, Q., Nickel, M., and Kiela, D. (2020). Hyperbolic graph neural networks: A review of methods and applications. arXiv preprint arXiv:2002.08852. [28] Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., and Boguñá, M. (2010). Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106. [29] Sala, F., De Sa, C., Gu, A., and Ré, C. (2018). Representation tradeoffs for hyperbolic embeddings. In International Conference on Machine Learning (ICML), PMLR 80:4460–4469. [30] Cohen, T. S., Geiger, M., Köhler, J., and Welling, M. (2018). Spherical CNNs. In International Conference on Learning Representations (ICLR). [31] Esteves, C., Allen-Blanchette, C., Makadia, A., and Daniilidis, K. (2020). Learning SO(3) equivariant representations with spherical CNNs. International Journal of Computer Vision, 128(5):1471–1488. [32] Bonev, B., Rietmann, M., Paris, A., Carpentieri, A., and Kurth, T. (2025). Attention on the sphere. In Advances in Neural Information Processing Systems (NeurIPS). [33] Kobler, R. J., Hirayama, J., and Kawanabe, M. (2022). Spherical convolutions and their applications in machine learning. arXiv preprint arXiv:2201.11899. [34] Gu, A., Goel, K., and Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. [35] Gu, A., Goel, K., Gupta, A., and Ré, C. (2022). On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems (NeurIPS). [36] Gu, A. and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. [37] Dao, T. and Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML). [38] Smith, J. T. H., Warrington, A., and Linderman, S. W. (2023). Simplified state space layers for sequence modeling. In International Conference on Learning Representations (ICLR). [39] Gupta, A., Gu, A., and Berant, J. (2022). Diagonal state spaces are as effective as structured state spaces. arXiv preprint arXiv:2203.14343. [40] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR). [41] Fedus, W., Zoph, B., and Shazeer, N. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [42] Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2021). GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR). [43] Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. (2022). GLaM: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML), PMLR 162:5547–5569. [44] DeepSeek-AI. (2024). DeepSeek-V3: Technical report. arXiv preprint arXiv:2412.19437. [45] Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., et al. (2024). Mixtral of experts. arXiv preprint arXiv:2401.04088. [46] Roller, S., Sukhbaatar, S., Weston, J., et al. (2021). Hash layers for large sparse models. In Advances in Neural Information Processing Systems (NeurIPS). [47] Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. (2021). BASE layers: Simplifying training of large, sparse models. arXiv preprint arXiv:2103.16716. [48] Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. (2024). A survey on mixture of experts. arXiv preprint arXiv:2407.06204. [49] Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. [50] Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. [51] Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. (2024). AWQ: Activation-aware weight quantization for LLM compression and acceleration. In International Conference on Machine Learning (ICML). [52] Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. (2024). KIVI: A tuning-free asymmetric 2-bit quantization for KV cache. arXiv preprint arXiv:2402.02750. [53] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. [54] Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2024). SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML). [55] Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2024). SqueezeLLM: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629. [56] Bhatnagar, P., Moradifirouzabadi, A., Yang, S.-H., Lee, S., Choi, J., and Kang, M. (2026). STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control. In International Conference on Machine Learning (ICML). [57] Karimi, H., Nutini, J., and Schmidt, M. (2016). Linear convergence of gradient and proximal-gradient methods under the Polyak–Łojasiewicz condition. In European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD). [58] Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854. [59] Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. [60] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027. [61] Li, Y. and Yuan, Y. (2018). Convergence analysis of two-layer neural networks with ReLU activation. In Advances in Neural Information Processing Systems (NeurIPS). [62] Liu, C., Zhu, L., and Belkin, M. (2022). Loss landscapes and optimization in over-parameterized non-linear systems. Journal of Machine Learning Research, 23(45):1–43. [63] Drineas, P. and Mahoney, M. W. (2005). On the Nyström method for approximating a Gram matrix for improved kernel-based learning. Journal of Machine Learning Research, 6:2153–2175. [64] He, N., et al. (2025). Hyperbolic deep learning for foundation models: A survey. arXiv preprint arXiv:2507.17787. SIGKDD 2025. [65] Zhang, Y., Chen, L., and Liu, Q. (2025). A survey on hyperbolic neural networks. arXiv preprint arXiv:2504.06543. [66] Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5401. [67] Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476. [68] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735–1780. [69] Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. [70] Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In International Conference on Machine Learning (ICML), PMLR 28:1310–1318.