Paper deep dive

RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding

Fang Li

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 41

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/12/2026, 2:16:46 AM

Summary

The paper introduces Riemannian Language Models (RiLM), a parameter-efficient language modeling framework that replaces the standard output matrix with geodesic decoding on Riemannian manifolds. By using a shared MLP composition map to update states along geodesics and deriving next-token probabilities from squared geodesic distances to vocabulary embeddings, RiLM eliminates the large output layer. The authors propose two variants: Flat RiLM (Euclidean space) and HypRiLM (Hyperbolic Poincaré ball). HypRiLM significantly outperforms Flat RiLM and traditional baselines (LSTM, Transformer, SSM) on WikiText-2, while Flat RiLM performs better on Penn Treebank. The study also identifies and resolves 'boundary collapse' in hyperbolic recurrence using Möbius stabilization.

Entities (12)

RiLM · model · 100%Flat RiLM · model · 95%HypRiLM · model · 95%Penn Treebank · dataset · 95%WikiText-2 · dataset · 95%Geodesic Decoding · method · 90%LSTM · baseline-model · 90%Möbius Stabilization · technique · 90%Poincaré ball · mathematical-space · 90%SSM · baseline-model · 90%Transformer · baseline-model · 90%Boundary Collapse · phenomenon · 85%

Relation Signals (6)

HypRiLM → operateson → Poincaré ball

confidence 95% · We instantiate the framework on flat R^d (Flat RiLM) and the Poincare ball H^d (HypRiLM)

RiLM → uses → Geodesic Decoding

confidence 95% · next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings

HypRiLM → outperforms → Flat RiLM

confidence 90% · HypRiLM reaches 54.2 +/- 0.2 validation perplexity versus 87.6 +/- 0.6 for Flat RiLM

HypRiLM → outperforms → SSM

confidence 90% · HypRiLM leads by roughly 2x over the strongest tied recurrent baseline (SSM, 113.0 +/- 3.8)

Möbius Stabilization → resolves → Boundary Collapse

confidence 90% · show how Mobius stabilization restores trainability

Flat RiLM → outperforms → HypRiLM

confidence 85% · Flat RiLM (40.9 +/- 0.6) beats HypRiLM (69.8 +/- 0.5) at identical ~289k parameters [on Penn Treebank]

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transformer at embedding width d = 128 still spends roughly one third of its capacity on the output matrix W_out in R^(d x |V|). We propose Riemannian Language Models (RiLM), which remove that layer entirely: context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings. The same embedding map serves input and output -- decoding is geometry. We instantiate the framework on flat R^d (Flat RiLM) and the Poincare ball H^d (HypRiLM) with a shared MLP composition map phi (~290k parameters, d = 128, |V| = 2000). Across five seeds on WikiText-2, HypRiLM reaches 54.2 +/- 0.2 validation perplexity versus 87.6 +/- 0.6 for Flat RiLM; tied and matched LSTM, Transformer, and SSM controls remain at 113-147 PPL on WT-2 -- HypRiLM leads by roughly 2x over the strongest tied recurrent baseline (SSM, 113.0 +/- 3.8). Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V|, while hyperbolic curvature helps selectively. We also characterize boundary collapse in naive hyperbolic recurrence and show how Mobius stabilization restores trainability. Claims are scoped to controlled small-model comparisons, not full-vocabulary state of the art.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,180 characters extracted from source content.

Expand or collapse full text

RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding Fang Li Affiliation: Department of Computer Science Oklahoma Christian University Edmond, USA Email: fang.li@oc.edu Abstract Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transformer at embedding width d=128d=128 still spends roughly one third of its capacity on the output matrix Wout∈ℝd×|V|W_out ^d×|V|. We propose Riemannian Language Models (RiLM), which remove that layer entirely: context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings. The same embedding map serves input and output—decoding is geometry. We instantiate the framework on flat ℝdR^d (Flat RiLM) and the Poincaré ball ℍdH^d (HypRiLM) with a shared MLP composition map φ (∼290 290k parameters, d=128d=128, |V|=2000|V|=2000). Across five seeds on WikiText-2, HypRiLM reaches 54.2±0.254.2± 0.2 validation perplexity versus 87.6±0.687.6± 0.6 for Flat RiLM; tied and matched LSTM, Transformer, and SSM controls remain at 113113–147147 PPL on WT-2—HypRiLM leads by roughly 2×2× over the strongest tied recurrent baseline (SSM, 113.0±3.8113.0± 3.8). Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V||V|, while hyperbolic curvature helps selectively. We also characterize boundary collapse in naive hyperbolic recurrence and show how Möbius stabilization restores trainability. Claims are scoped to controlled small-model comparisons, not full-vocabulary state of the art. Index Terms: language modeling, Riemannian geometry, hyperbolic neural networks, parameter efficiency I Introduction Neural language models map token sequences to distributions over the next word. The dominant recipe—token vectors in ℝdR^d, state updates via LSTM gating [1] or self-attention [2], and an affine readout Wouth+bW_outh+b—scales to billions of parameters and underpins systems from machine translation to code generation [3, 4]. That recipe is the right default when compute and data are abundant. It is a poor default when the model must fit on device, adapt quickly to a narrow domain, or serve as a reproducible scientific instrument: below one million parameters, Wout∈ℝd×|V|W_out ^d×|V| can consume one third of the budget at d=128d=128 and |V|=2000|V|=2000, matching the embedding table. Shrinking d to compensate weakens the recurrent or attention core; enlarging |V||V| without enlarging the model makes the output layer dominate entirely. Tied embeddings [5] halve redundant rows but still decode via a linear functional detached from how states move through representation space. Language also carries hierarchical structure—syntax, lexical relations, topical organization—that Euclidean space embeds with distortion that grows in depth [6]. Hyperbolic space offers exponentially growing volume and has improved static embeddings and feed-forward layers [7], yet it is rarely used to govern how context is composed step by step. We ask whether manifold geometry can replace not only word vectors but the decoder itself. We introduce Riemannian Language Models (RiLM). The state hth_t lives on a Riemannian manifold (M,g)(M,g). A single map φ , shared at every timestep, reads the current state and incoming token and proposes a tangent direction; the state then moves along the corresponding geodesic. Decoding needs no WoutW_out: word w is likely when its embedding ewe_w lies close to hth_t in geodesic distance, p(w∣ht)∝exp(−dM(ht,ew)2/τ)p(w h_t) (-d_M(h_t,e_w)^2/τ). Context becomes a trajectory h0→h1→⋯→hTh_0\!→\!h_1\!→\!·s\!→\!h_T; classification and representation are coupled because the model must arrange vocabulary points so that plausible continuations lie near that trajectory. We study two instantiations: Flat RiLM on M=ℝdM=R^d (the Euclidean special case of our earlier unpublished prototype) and HypRiLM on the Poincaré ball M=ℍcdM=H^d_c. Hyperbolic recurrence posed an unexpected obstacle—naive exponential-map updates drive states to the ball boundary and flatten all logits—which we analyze and resolve with Möbius translation (Section V). Empirically, we report multi-seed WikiText-2 and Penn Treebank results against untied, tied, and matched-parameter LSTM, Transformer, and selective state-space baselines. HypRiLM improves ∼38% 38\% relative to Flat RiLM on web text and trails Flat RiLM on newswire PTB; on both corpora, each RiLM variant beats matched and tied LSTM, Transformer, and SSM controls by wide margins (Tables IV and VI). We do not target billion-parameter pretraining; the contribution is a geometric language-modeling template, a trainability analysis for hyperbolic recurrence, and evidence that decoding geometry matters when parameters are scarce. Contributions. This paper makes four claims, each backed by controlled experiments at ∼290 290k parameters. First, geodesic decoding replaces WoutW_out with distance to vocabulary points on M, coupling representation and classification without a separate readout layer. Second, the same template instantiates flat and hyperbolic sequential models, enabling a direct test of whether curvature helps composition when φ is held fixed. Third, we identify boundary collapse in naive hyperbolic recurrence—a failure mode that drives perplexity to the uniform |V||V| baseline—and show that Möbius-based updates restore stable training. Fourth, under explicit fairness protocols (tied and matched baselines, including a modern SSM control), both RiLM variants outperform LSTM, Transformer, and SSM decoders on WikiText-2 and Penn Treebank by margins that tied embeddings alone do not explain. Scope and organization. Our primary evaluation uses the 2000 most frequent words on WikiText-2 and PTB so that WoutW_out accounts for a transparent ∼256 256k parameters at d=128d=128; we do not claim numerically comparable perplexity to full-vocabulary leaderboard models. Section I defines RiLM; Section IV presents multi-seed results, vocabulary scaling, ablations, and geometry analysis; Section V interprets boundary collapse, when curvature helps, and limitations. I Related Work Recurrent and attention decoders. Modern LMs stack depth, width, and data until perplexity on WikiText-2 drops into the twenties—but only at 33M–257M parameters with heavy regularization [8, 4, 3]. At sub-million scale the bottleneck shifts: a two-layer LSTM or Transformer must either carry a full WoutW_out or tie embeddings [5]. Tying removes redundant rows but preserves the form of decoding as a linear functional on hidden states in ℝdR^d. RiLM instead decodes by geometry on M: the hidden state and vocabulary live in the same space, and logits are derived from distances. We compare against both untied models (to show the cost of WoutW_out) and tied/matched models (to isolate the effect of decoding geometry). Hyperbolic representation learning. Hyperbolic space embeds trees and hierarchies with low distortion [6]; hyperbolic neural networks provide Möbius operations and Riemannian optimization tools [7]. Most prior work applies hyperbolicity to static objects—word vectors, graph nodes, or single feed-forward layers. RiLM asks whether curvature should govern the trajectory of context: each token applies the same φ and moves the state along a geodesic, and every prediction step reuses the same distance-based readout. The sequential, recurrent use of hyperbolic geometry is therefore qualitatively different from Poincaré word embeddings with a Euclidean softmax on top. Efficiency and modern recurrent baselines. Parameter-efficient fine-tuning (e.g., LoRA [9]) reduces adaptation cost but leaves the pretrained decoder intact. Continuous-time models [10] and volume-preserving flows [11] study dynamics but do not remove vocabulary-sized output layers in small LMs. Selective state-space models [12] offer linear-time recurrence with input-dependent gating and represent the strongest modern alternative to LSTM/Transformer backbones at similar width. We implement a lightweight diagonal SSM as an additional baseline; RiLM competes on decoding geometry rather than on replacing φ with a different recurrent operator. I Method RiLM treats a sentence as motion on a manifold: each token nudges the state along a learned geodesic, and the next-word distribution is read off from geometry rather than a separate classifier. Figure 1 summarizes one timestep. The design deliberately mirrors tied embeddings—input and output share parameters—but replaces the linear map WouthW_outh with negative squared distance −dM(h,ew)2-d_M(h,e_w)^2, so the model must organize vocabulary points on M in a way that supports both ingestion and prediction. I-A Geometric Background A Riemannian manifold (M,g)(M,g) equips each tangent space TxMT_xM with an inner product gxg_x varying smoothly in x. The exponential map expx:TxM→Mexp_x T_xM→ M sends tangent vector v along the geodesic starting at x in direction v; the logarithmic map logx _x inverts it locally. Geodesic distance dM(x,y)d_M(x,y) is the length of the shortest path on M. On ℝdR^d, expx(v)=x+vexp_x(v)=x+v and dM(x,y)=‖x−y‖2d_M(x,y)=\|x-y\|_2. The Poincaré ball ℍcd=x∈ℝd:‖x‖<1/cH^d_c=\x ^d:\|x\|<1/ c\ models hyperbolic space with curvature c>0c>0. The conformal factor λxc=2/(1−c‖x‖2) _x^c=2/(1-c\|x\|^2) diverges at the boundary, yielding exponential volume growth Vol⁡(Bℍ(r))≈ecrVol(B_H(r))≈ e c\,r. Möbius addition [7], x⊕cy=(1+2c⟨x,y⟩+c‖y‖2)x+(1−c‖x‖2)y1+2c⟨x,y⟩+c2‖x‖2‖y‖2,x _cy= (1+2c x,y +c\|y\|^2)x+(1-c\|x\|^2)y1+2c x,y +c^2\|x\|^2\|y\|^2, (1) is closed on the open ball. Hyperbolic distance is dℍ(x,y)=(2/c)arctanh(c∥(−x)⊕cy∥)d_H(x,y)=(2/ c)\,arctanh( c\,\|(-x) _cy\|). I-B Recurrence and Geodesic Decoding Given vocabulary V, each word w has embedding ew∈Me_w∈ M and the contextual state ht∈Mh_t∈ M summarizes prefix w1:tw_1:t. A shared composition map φ:ℝd×ℝd→ℝd ^d×R^d ^d reads tangent coordinates at the origin and proposes an update direction: ht+1=expht(φ⁡(log0⁡(ht),log0⁡(ewt))).h_t+1=exp_h_t\! ( ( _0(h_t),\, _0(e_w_t)) ). (2) Decoding with temperature τ>0τ>0 defines a softmax over negative squared distances, p⁡(w∣ht)=exp(−dM(ht,ew)2/τ)∑w′exp(−dM(ht,ew′)2/τ).p(w h_t)= (-d_M(h_t,e_w)^2/τ) _w (-d_M(h_t,e_w )^2/τ). (3) which is analogous to metric learning: likely words are those whose embeddings lie in a small geodesic ball around hth_t. The softmax can be viewed as a kernel classifier with k(x,y)=exp(−dM(x,y)2/τ)k(x,y)= (-d_M(x,y)^2/τ); on ℝdR^d this resembles a Gaussian RBF over embedding space, while on ℍdH^d distances grow faster with separation, sharpening the effective decision boundary near the state trajectory. Unlike a free linear head WouthW_outh, the classifier weights are identified with vocabulary geometry—the model cannot assign high probability to a word without placing ewe_w near regions of M that hth_t actually visits. On ℝdR^d, identifying tangent vectors with ambient coordinates (log0⁡(x)=x _0(x)=x, exph(v)=h+vexp_h(v)=h+v), Eq. (2) reduces to additive composition ht+1=ht+φ⁡(ht,ewt)h_t+1=h_t+ (h_t,e_w_t)—the Euclidean special case of our unpublished flat prototype (anonymous supplementary code). The sequence starts from h0=exp0(θroot)=θrooth_0=exp_0( _root)= _root with learned θroot∈ℝd _root ^d. ht∈Mh_t∈ Mewte_w_tφ stepht+1h_t+1state update−dM(ht,ew)2-d_M(h_t,e_w)^2softmaxgeodesic decode Fig. 1: RiLM at timestep t: shared φ , geodesic state update, and distance-based decoding without WoutW_out. I-C Hyperbolic RiLM and Möbius Recurrence Eq. (2) is the natural hyperbolic analogue of additive recurrence, but implementing exphtexp_h_t directly on ℍdH^d fails in practice. As ‖ht‖→1/c\|h_t\|→ 1/ c, the conformal factor λht=2/(1−c‖ht‖2) _h_t=2/(1-c\|h_t\|^2) diverges: small tangent steps at hth_t correspond to enormous boundary motion, states stick to the ball boundary, and dℍ(ht,ew)d_H(h_t,e_w) becomes nearly constant for all w. We observed ‖ht‖≈0.999\|h_t\|≈ 0.999 within three steps and validation perplexity equal to |V||V| (Section V). Our fix computes increments where the metric is well-behaved—at the origin—and transports them with Möbius addition: δt=exp0(s⋅φ(log0ht,log0ewt)),ht+1=ht⊕cδt, _t=exp_0\! (s· ( _0h_t, _0e_w_t) ), h_t+1=h_t _c _t, (4) with learnable step scale s initialized near 1/d1/ d. After training on WikiText-2, ‖ht‖\|h_t\| on held-out prefixes typically lies in [0.29,0.71][0.29,0.71]—well inside the ball (Figure 3). Embeddings and θroot _root are projected to ‖x‖≤0.999/c\|x\|≤ 0.999/ c; projecting hth_t after every step stalled optimization at ∼130 130 PPL, so we project only static parameters. I-D Composition Function and Training We use a single MLP for φ in all main experiments: φ⁡(vh,ve)=tanh⁡(W⁡[vh;ve]+b) (v_h,v_e)= (W[v_h;v_e]+b) with W∈ℝd×2dW ^d× 2d (∼33 33k parameters at d=128d=128), shared across timesteps. Sharing φ is central to the RiLM hypothesis: composition rules should not proliferate with sequence length the way layer-specific attention maps do. The model minimizes masked token negative log-likelihood, ℒ=−1∑tmt∑tmtlogp(wt+1∣ht),L=- 1 _tm_t _tm_t p(w_t+1 h_t), (5) with padding mask mt∈0,1m_t∈\0,1\. Long compositions are trained with truncated BPTT (k=8k=8): states detach every k steps, limiting gradient depth through repeated φ while matching the effective update frequency of windowed baseline training. At k=8k=8 and context 64, each token still participates in eight compositional steps before gradients truncate—comparable to the receptive field of a shallow LSTM unrolled over the same window. Under this parameter budget, most predictive signal appears local: extending context to 256 with spline φ regresses to 65.665.6 PPL (Table VII), so main experiments keep context 64 and MLP φ . Flat RiLM uses Adam at η=10−3η=10^-3; HypRiLM uses η=3×10−3η=3× 10^-3, curvature c=1.0c=1.0, embedding initialization ⁡(0,(0.3/d)2)N(0,(0.3/ d)^2), and gradient clipping at norm 1.01.0. All models train for 10 epochs with batch size 128 and context length 64 (Table I). Volume-preserving word flows Φw(h)=exph(φ⁡(log0⁡h,log0⁡ew)) _w(h)=exp_h( ( _0h, _0e_w)) are possible in principle [11]; our MLP φ does not satisfy this, and spline variants (Table VII) move perplexity only slightly—suggesting that gains come from decoding geometry and manifold choice rather than from entropy-preserving dynamics alone. I-E Parameter Budget and Baseline Fairness At d=128d=128 and |V|=2000|V|=2000, geodesic decoding eliminates the ∼256 256k parameters that LSTM and Transformer decoders allocate to WoutW_out (Table I). RiLM reinvests that capacity in full-width embeddings and φ . Fair comparison therefore requires two baseline regimes. Tied models at d=128d=128 share input/output embeddings like RiLM but retain flat LSTM, attention, or SSM dynamics. Matched models equalize total parameters at ∼290 290k by shrinking hidden width to d′≈82d ≈ 82–9797, which weakens their recurrent core. Tied rows are the stronger control on decoding; matched rows isolate whether RiLM wins only because it keeps d=128d=128. TABLE I: Parameter allocation (d=128d=128, |V|=2000|V|=2000). Component RiLM LSTM Transformer Embeddings 256k 256k 256k Output head WoutW_out 0 256k 256k Composition core 33k 264k 265k Total 289k 776k 777k TABLE I: Shared hyperparameters. Setting Value Embedding d / vocab. |V||V| 128 / 2000 (most frequent) Context length / TBPTT k 64 / 8 Batch size / epochs 128 / 10 Decoding temperature τ 1.0 Flat / LSTM / TX learning rate 10−310^-3 HypRiLM learning rate 3×10−33× 10^-3 SSM learning rate 3×10−43× 10^-4 LSTM / Transformer 2 layers; TX: 4 heads, FFN 256 WT-2 seeds / PTB seeds 42,…,46\42,…,46\ / 42,43,44\42,43,44\ IV Experiments We designed experiments to answer three questions. Q1: Does geodesic decoding improve language modeling when decoder capacity is fairly counted against LSTM, Transformer, and SSM baselines? Q2: Holding φ fixed, does hyperbolic curvature improve over flat space? Q3: Do relative conclusions survive a tenfold increase in |V||V|? We report validation perplexity exp⁡(ℒ) (L) on WikiText-2 (WT-2; full train/validation splits, 2.1M training tokens) and the Zaremba Penn Treebank (PTB; 887k training tokens). Primary tables use the 2000 most frequent words on both corpora. This protocol fixes |V||V| so that WoutW_out costs ∼256 256k parameters at d=128d=128, making the trade-off in Table I explicit. Absolute perplexities are not comparable to full-vocabulary leaderboard entries (Table VIII); our claims concern controlled comparisons at matched scale. Baselines and training protocol. We compare against two-layer LSTM, two-layer causal Transformer (four heads, FFN width 256), and a selective diagonal SSM inspired by Mamba [12]—each in untied, tied [5], and matched (∼290 290k) regimes. SSM uses Adam at η=3×10−4η=3× 10^-4 and reports the best validation checkpoint within 10 epochs (it overfits after ∼4 4 epochs on WT-2); Flat RiLM, LSTM, and Transformer use η=10−3η=10^-3; HypRiLM uses 3×10−33× 10^-3. All non-SSM models use the final epoch checkpoint. WikiText-2 aggregates mean ± std over five seeds 42,…,46\42,…,46\; PTB over three seeds 42,43,44\42,43,44\. Effect sizes on WT-2 are large: the HypRiLM–Flat gap (∼33 33 PPL) exceeds 100×100× the HypRiLM standard deviation (0.20.2), and HypRiLM–SSM-tied gaps (∼59 59 PPL) exceed 15×15× the SSM-tied std (3.83.8). Reported means ± std over seeds therefore suffice; formal significance tests add little beyond these intervals. All training uses one GPU; reproduction scripts and unit tests for hyperbolic primitives are in anonymous supplementary material. IV-A WikiText-2 at Controlled Vocabulary Table I reports headline WT-2 results. HypRiLM reaches 54.2±0.254.2± 0.2 perplexity at ∼289 289k parameters—the best result by a wide margin, with standard deviation far smaller than the gaps to baselines. Flat RiLM attains 87.6±0.687.6± 0.6 with the same φ and parameter count, isolating curvature: hyperbolic space accounts for a ∼38% 38\% relative improvement on this corpus. Untied LSTM (149.9±2.7149.9± 2.7) and Transformer (137.4±5.1137.4± 5.1) use ∼776 776k parameters yet remain ∼84 84–9696 PPL worse, indicating that adding WoutW_out capacity does not close the gap when the compositional core stays flat and Euclidean. TABLE I: WikiText-2 validation perplexity (five seeds, 2k vocab). Model PPL ↓ Params HypRiLM (MLP φ ) 54.2 ± 0.2 289k Flat RiLM (MLP φ ) 87.6 ± 0.6 289k LSTM, untied 149.9 ± 2.7 776k Transformer, untied 137.4 ± 5.1 777k TABLE IV: Fair WikiText-2 comparison (five seeds 42,…,46\42,…,46\; all models). Primary controls: tied and matched rows. Model PPL ↓ Params HypRiLM 54.2 ± 0.2 289k Flat RiLM 87.6 ± 0.6 289k LSTM-tied (d=128d=128) 117.9 ± 1.4 520k LSTM-matched 125.2 ± 6.8 273k Transformer-tied (d=128d=128) 147.0 ± 1.0 521k Transformer-matched 142.8 ± 1.1 283k SSM-tied (d=128d=128) 113.0 ± 3.8 374k SSM-matched 118.1 ± 0.7 271k Fair comparisons in Table IV address whether RiLM wins only by tying or by parameter accounting. Against tied LSTM (117.9±1.4117.9± 1.4), Transformer-tied (147.0±1.0147.0± 1.0), and SSM-tied (113.0±3.8113.0± 3.8)—all at d=128d=128 with shared input/output weights—HypRiLM still leads by 59–93 PPL while using fewer parameters than the LSTM and Transformer rows. The SSM-tied row is particularly informative: it is a modern recurrent decoder with selective gating, yet remains ∼2× 2× worse than HypRiLM (113.0113.0 vs. 54.254.2). Against matched baselines at ∼290 290k total, hidden widths shrink to d′≈82d ≈ 82–9797; HypRiLM wins by 64–89 PPL while retaining d=128d=128. No fair regime reverses the ranking. On a small held-out subset (10k training / 500 validation tokens; seed 42), model ordering matches Table I (HypRiLM 76.4, Flat RiLM 103.0, LSTM 189.0, Transformer 148.5 PPL). Ablating the Möbius fix—using exphtexp_h_t directly in Eq. (2)—drives perplexity to the uniform baseline |V|=2000|V|=2000 (Table IX), confirming that trainable hyperbolic dynamics are prerequisite to the headline results. IV-B Vocabulary Scaling to |V|=10,000|V|=10,000 When |V||V| grows tenfold, embedding tables dominate parameter count (∼1.3 1.3M for RiLM at d=128d=128) and the original ∼290 290k matched baselines are no longer meaningful—shrinking d to match budget would yield d′<40d <40. We therefore compare RiLM variants and tied baselines at full width with batch size 32 (Table V). Absolute perplexity rises and is not comparable to Table I; the question is whether ranking persists. Both RiLM variants remain far below SSM-tied (708.3±9.0708.3± 9.0): Flat RiLM at 341.8±0.5341.8± 0.5, HypRiLM at 345.8±16.0345.8± 16.0. Flat RiLM is marginally best on mean; HypRiLM is competitive but seed 44 is an outlier (368.4368.4 PPL), inflating variance. We interpret this as evidence that (i) geodesic decoding remains advantageous at larger |V||V|, but (i) hyperbolic curvature is not uniformly superior—on this scale, Euclidean geometry is slightly more stable. Tied LSTM and Transformer runs were unstable under the shared 10-epoch schedule (PPL >1400>1400), highlighting optimization fragility of flat decoders at large |V||V| under matched training budgets; SSM-tied is the reliable recurrent control here. IV-C Penn Treebank and Cross-Corpus Behavior PTB reverses the within-family WT-2 ordering: Flat RiLM (40.9±0.640.9± 0.6) beats HypRiLM (69.8±0.569.8± 0.5) at identical ∼289 289k parameters (Table VI). Against fair baselines, both variants remain far below tied LSTM (214.3±11.3214.3± 11.3), Transformer-tied (182.3±1.2182.3± 1.2), and SSM-tied (108.9±23.7108.9± 23.7), as well as matched LSTM (161.6±1.6161.6± 1.6), Transformer (171.4±4.1171.4± 4.1), and SSM (149.3±27.3149.3± 27.3) controls. Geodesic decoding therefore transfers across corpora even when negative curvature is not the right inductive bias. PTB is smaller and stylistically homogeneous (newswire); WT-2 is noisier web text with broader topical structure. We report both corpora so readers can judge when to prefer Flat vs. HypRiLM rather than treating hyperbolicity as a universal upgrade. IV-D Ablations and Literature Context Table VII disentangles architectural choices on WT-2 (seed 42). Replacing MLP φ with monotone spline compose nodes (8 knots) improves Flat RiLM by ∼4.4 4.4 PPL but changes HypRiLM by only ∼0.1 0.1 PPL—smaller shifts than the ∼34 34 PPL gap between hyperbolic and flat MLP rows. Reducing curvature to c=0.1c=0.1 degrades HypRiLM from 54.054.0 to 71.171.1 PPL, confirming that the ball’s negative curvature is doing work, not merely adding parameters. Extending context to 256 with spline φ yields 65.665.6 PPL, so we retain context 64 and MLP φ in all headline runs. Table VIII places our seed-42 numbers beside published full-vocabulary models for context only. Transformer-XL, GPT-2, and AWD-LSTM operate at 33M–257M parameters with extensive regularization; HypRiLM uses 2k words and ∼289 289k parameters. The numeric proximity of HypRiLM (54.054.0) to AWD-LSTM (60.760.7) is not a controlled comparison and should not be read as competitive SOTA; it illustrates how controlled-vocabulary perplexity can look deceptively strong relative to leaderboard entries. TABLE V: WikiText-2 at 10k vocabulary (d=128d=128, three seeds). Model PPL ↓ Params HypRiLM 345.8 ± 16.0 ∼1.31 1.31M Flat RiLM 341.8 ± 0.5 ∼1.31 1.31M SSM-tied (d=128d=128) 708.3 ± 9.0 ∼1.38 1.38M LSTM-tied (d=128d=128)† 2021 ± 135 ∼1.54 1.54M Transformer-tied (d=128d=128)† 1513 ± 116 ∼1.52 1.52M †Unstable under identical training at |V|=10|V|=10k. TABLE VI: Penn Treebank validation perplexity (three seeds, 2k vocab; tied and matched baselines). Model PPL ↓ Params Flat RiLM 40.9 ± 0.6 289k HypRiLM 69.8 ± 0.5 289k LSTM-tied (d=128d=128) 214.3 ± 11.3 520k LSTM-matched 161.6 ± 1.6 273k Transformer-tied (d=128d=128) 182.3 ± 1.2 521k Transformer-matched 171.4 ± 4.1 283k SSM-tied (d=128d=128) 108.9 ± 23.7 374k SSM-matched 149.3 ± 27.3 271k TABLE VII: WikiText-2 ablations (seed 42). Variant Manifold PPL ↓ MLP φ (default) ℍdH^d, c=1.0c=1.0 54.0 MLP φ (default) ℝdR^d 88.1 Spline 1D (8 knots) ℍdH^d 53.9 Spline 1D (8 knots) ℝdR^d 83.7 MLP, c=0.1c=0.1 ℍdH^d 71.1 Spline φ , context 256 ℍdH^d 65.6 TABLE VIII: WikiText-2 perplexity in the literature (token-level; not comparable to our 2k-vocab setup). Model PPL ↓ Params Transformer-XL [4] 23.1 257M GPT-2 small [3] 29.4 117M AWD-LSTM [8] 60.7 33M HypRiLM (ours, seed 42) 54.0 289k Flat RiLM (ours, seed 42) 88.1 289k TABLE IX: Boundary collapse vs. Möbius fix (seed 42). Naïve exphtexp_h_t: 10k train / 500 val subset. Möbius: ‖ht‖\|h_t\| after training; PPL on full WT-2 validation. Recurrence ‖ht‖\|h_t\| after 3 steps PPL expht(φ)exp_h_t( ) (naïve) ≈0.999≈0.999 2000 Möbius Eq. (4) 0.290.29–0.710.71 ∼54 54 IV-E Learned Geometry and Training Dynamics Quantitative perplexity alone does not show whether geodesic decoding organizes representation space. We therefore inspect trained HypRiLM and Flat RiLM models (WT-2, seed 42, 10 epochs) on a held-out validation prefix. Training stability. Figure 2 tracks validation perplexity when the hyperbolic recurrence uses naive exphtexp_h_t updates versus the Möbius fix (Eq. (4)). Naive training remains near ∼150 150 PPL across 10 epochs; Möbius HypRiLM improves monotonically to ∼54 54—the same regime as Table I. The forward-pass collapse in Table IX (‖ht‖≈0.999\|h_t\|≈0.999, PPL =|V|=|V|) explains why naive training fails to descend; Möbius transport is required for usable hyperbolic dynamics. Trajectory and ball geometry. Figure 3 visualizes a 20-token prefix. Panels (a–b) project vocabulary embeddings (gray) and the state trajectory h0→⋯→hTh_0\!→\!·s\!→\!h_T (colored path; green/red: start/end) via PCA; the dashed circle in (a) is the Poincaré boundary (‖x‖=1\|x\|=1 at c=1c=1). Along this prefix, HypRiLM satisfies ‖ht‖∈[0.29,0.71]\|h_t\|∈[0.29,0.71] (panel c)—inside the ball and consistent with the post-training range in Table IX. Flat RiLM follows a comparable low-dimensional path in ℝdR^d. Distance-based decoding in practice. After the prefix “along with ⋯·s city of”, both models rank was and related function words among the highest-probability continuations (Table X). Neither model was cherry-picked on this example; it is the first long validation prefix returned by our data loader. Fig. 2: Validation perplexity per epoch on WikiText-2 (seed 42): naive exphtexp_h_t recurrence vs. Möbius HypRiLM. Fig. 3: Learned geometry on a WT-2 validation prefix (seed 42). (a)–(b) PCA projections with Poincaré boundary (dashed) in (a); (c) ‖ht‖\|h_t\| along the HypRiLM trajectory. TABLE X: Top-5 next-token candidates by geodesic distance after prefix “along with ⋯·s city of” (seed 42; logits are −dM2/τ-d_M^2/τ). Rank HypRiLM / Flat RiLM 1 <unk> / <unk> 2 was / was 3 is / and 4 had / the 5 the / in V Discussion V-A Boundary Collapse as a Trainability Phenomenon Our first HypRiLM implementation applied expht(φ(⋅))exp_h_t( (·)) directly in Eq. (2). The hyperbolic exponential map can be written expht(v)=ht⊕cexp0(tanh(cλht∥v∥/2)v/(c∥v∥))exp_h_t(v)=h_t _cexp_0( ( c\, _h_t\,\|v\|/2)\,v/( c\,\|v\|)). As ‖ht‖→1/c\|h_t\|→ 1/ c, the conformal factor λht=2/(1−c‖ht‖2) _h_t=2/(1-c\|h_t\|^2) grows without bound, tanh saturates, and updated states remain at the boundary. Within three recurrence steps we observed ‖ht‖≈0.999\|h_t\|≈ 0.999; geodesic distances to all vocabulary embeddings became nearly equal; logits flattened; and perplexity reached |V|=2000|V|=2000—exactly the uniform-guess baseline (Table IX). This boundary collapse is not a small optimization issue: it renders naive hyperbolic recurrence untrainable for language modeling at modest depth. Equation (4) sidesteps the pathology by computing tangent increments at the origin—where λ0 _0 is moderate—and translating via ⊕c _c, which maps interior points to interior points. After training, ‖ht‖\|h_t\| stays in [0.29,0.71][0.29,0.71] on held-out prefixes and perplexity falls to ∼54 54. Hard projection of hth_t after every step was counterproductive (∼130 130 PPL), likely because it interrupts gradient flow along geodesics; we project only embeddings and θroot _root. We regard stable Möbius recurrence as a necessary engineering contribution alongside geodesic decoding: without it, the hyperbolic instantiation of RiLM does not function. V-B Why Geodesic Decoding Helps at Small Scale When |V||V| and d are moderate, WoutW_out can consume roughly one third of a standard untied LM’s parameters (Table I). Matched baselines respond by shrinking hidden width, which weakens LSTM gates or attention heads. RiLM instead keeps d=128d=128 and forces the vocabulary embedding table to serve double duty: tokens are inputs and the targets of distance-based classification. The model cannot learn arbitrary linear functionals of hth_t independent of where words live on M; plausible continuations must lie near the contextual trajectory. This coupling may explain why tied LSTM/Transformer models—which remove WoutW_out but decode linearly in ℝdR^d—still trail by tens of perplexity points. Swapping ℝdR^d for ℍdH^d at fixed φ yields a large WT-2 gain but not a PTB gain. We do not claim a formal hierarchy theorem for web text; rather, WT-2’s broader topical and lexical structure may benefit from the exponential separation hyperbolic space affords, whereas compact PTB sentences are adequately served by flat geometry. Table XI summarizes when each instantiation is preferable in our experiments. TABLE XI: Practical guide: which RiLM variant to try first (under our ∼290 290k, 2k-vocab protocol). Setting Recommendation Noisy web text (WT-2) HypRiLM (large margin over flat and baselines) Compact newswire (PTB) Flat RiLM (beats HypRiLM at same φ ) |V||V| ≳ 10k, fixed training budget Flat RiLM (more stable; similar mean PPL) Need fastest inference Flat RiLM (∼0.15 0.15 vs. ∼0.52 0.52 ms/token) Practitioners should treat manifold choice as a validation-set decision, not a default. V-C Relation to Tied Embeddings Tied LSTM/Transformer models set Wout=E⊤W_out=E where E∈ℝ|V|×dE ^|V|× d is the embedding table, so logits are EhtEh_t—a bilinear form in ℝdR^d. RiLM instead uses −‖ht−ew‖2-\|h_t-e_w\|^2 on ℝdR^d (or −dℍ(ht,ew)2-d_H(h_t,e_w)^2 on ℍdH^d). Expanding the Euclidean case, −‖h−ew‖2=−‖h‖2+2⟨h,ew⟩−‖ew‖2-\|h-e_w\|^2=-\|h\|^2+2 h,e_w -\|e_w\|^2. The term −‖h‖2-\|h\|^2 is shared across all words at a given timestep and cancels in the softmax; RiLM logits therefore differ from tied linear decoding by the word-dependent norms ‖ew‖2\|e_w\|^2 and by whether inner products or distances define similarity. Geodesic decoding is not equivalent to tying alone: it imposes a metric structure on how hidden states relate to all vocabulary points simultaneously. Our tied baselines isolate this distinction—they remove WoutW_out but keep linear scoring—and still trail HypRiLM by 59+ PPL on WT-2. V-D Efficiency and Limitations Per generated token, RiLM requires O⁡(d)O(d) work for φ and O⁡(|V|⋅d)O(|V|· d) for distance evaluation—the same order as softmax over |V||V|. The benefit is parametric: eliminating WoutW_out at small |V||V|. Measured on WikiText-2 (seed 42, autoregressive ms/token), Flat RiLM averages 0.15 ms/token, HypRiLM 0.52 ms/token, LSTM 0.44 ms/token, and Transformer 0.40 ms/token. Hyperbolic operations add ∼3.5× 3.5× overhead over Flat RiLM but remain in the same ballpark as the recurrent baselines; Flat RiLM is the fastest variant in this benchmark. Our study is intentionally bounded. Results use <2<2M parameters in the largest 10k-vocab setting, context 64, and TBPTT k=8k=8. The 2k vocabulary is a controlled instrument, not a claim about full WikiText-2 benchmarking. Our SSM baseline is a lightweight diagonal model, not a tuned Mamba implementation; SSM uses best-validation checkpoint selection while other models use the final epoch—a potential advantage for SSM that our headline gaps absorb. Matched baselines equalize parameter count but not representation width. Reported ms/token timings use autoregressive generation with model-specific prefix lengths (see supplementary material) and are indicative, not hardware-independent benchmarks. Product-manifold variants improved WT-2 by only ∼3 3 PPL at higher complexity in exploratory runs. All models are trained from scratch without distillation. Scaling outlook. At full vocabulary (|V|≈33|V|≈33k), WoutW_out dominates even billion-parameter transformers—but RiLM’s O⁡(|V|⋅d)O(|V|· d) decoding cost remains, and geodesic distance over the full table may need approximation (hierarchical softmax, sampled negatives, or product-manifold factorization of V). Our 10k study is a first stress test: RiLM variants stay ∼2× 2× better than SSM-tied but flat geometry becomes competitive with hyperbolic. We expect the decoding idea to matter most where |V||V| is moderate and parameters scarce; whether curvature helps at scale is an open empirical question we do not resolve here. Extending RiLM to full vocabulary and wider embeddings is the natural next step for community-scale evaluation. VI Conclusion RiLM reframes small language modeling as geometry: context is a path on a manifold, and decoding reads distances to vocabulary points rather than applying a separate output matrix. With a shared MLP φ and ∼290 290k parameters, HypRiLM leads on WikiText-2 against fair LSTM, Transformer, and SSM baselines; Flat RiLM is preferable on Penn Treebank; both beat tied and matched controls on both corpora. Stable Möbius recurrence proved essential—without it, hyperbolic RiLM does not train. Within the controlled regime we study, geodesic decoding is a concrete way to couple representation and prediction when parameters are scarce. It is not a drop-in replacement for billion-parameter pretraining, but it offers a principled alternative to WoutW_out whose benefits persist under tied and matched comparisons and partially under larger |V||V|. We hope the framework, the boundary-collapse analysis, and the multi-seed protocol provide a reproducible baseline for future work on geometric language models. References [1] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, 1997. [2] A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017. [3] A. Radford et al., “Language models are unsupervised multitask learners,” OpenAI Tech. Rep., 2019. [4] Z. Dai et al., “Transformer-XL: Attentive language models beyond a fixed-length context,” in Proc. ACL, 2019. [5] O. Press and L. Wolf, “Using the output embedding to improve language models,” in Proc. EACL, 2017. [6] M. Nickel and D. Kiela, “Poincaré embeddings for learning hierarchical representations,” in Proc. NeurIPS, 2017. [7] O. Ganea, G. Bécigneul, and T. Hofmann, “Hyperbolic neural networks,” in Proc. NeurIPS, 2018. [8] S. Merity, N. S. Keskar, and R. Socher, “Regularizing and optimizing LSTM language models,” in Proc. ICLR, 2018. [9] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. [10] R. T. Q. Chen et al., “Neural ordinary differential equations,” in Proc. NeurIPS, 2018. [11] D. J. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in Proc. ICML, 2015. [12] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv:2312.00752, 2023.

Instant research discovery