Paper deep dive

Riemannian Geometry for Pre-trained Language Model Embeddings

Szczepan Konior, Alexandre Quemy, Przemysław Klocek, Bartłomiej Sobieski, Grégoire Cattan

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 57

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 7/13/2026, 1:52:06 AM

Summary

The paper investigates whether sentence-level classification signal resides in the Riemannian geometry of pre-trained language model embeddings. It introduces Riemannian Mean Pooling (RMP), a method that extracts per-token pullback metrics from an encoder's Jacobian and aggregates them using the Fréchet mean on the SPD manifold. RMP consistently outperforms Euclidean mean pooling on CoLA, CREAK, and RTE, while correctly performing at chance on the debiased FEVER-Symmetric benchmark. Ablation studies reveal that the geometric aggregation mechanism itself, rather than the learned encoder, is the primary driver of performance gains.

Entities (8)

CoLA · dataset · 99%CREAK · dataset · 99%FEVER-Symmetric · dataset · 98%Riemannian Mean Pooling · method · 98%Euclidean mean pooling · baseline-method · 97%Fréchet mean · mathematical-operation · 97%SPD manifold · mathematical-space · 96%Intrinsic Green's Learning · framework · 95%

Relation Signals (6)

Fréchet mean → aggregateson → SPD manifold

confidence 96% · aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold

Riemannian Mean Pooling → performsatchanceon → FEVER-Symmetric

confidence 96% · on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance

Riemannian Mean Pooling → outperforms → Euclidean mean pooling

confidence 95% · Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling

Intrinsic Green's Learning → extracts → Pullback metric

confidence 94% · extracting per-token pullback metrics from a learned encoder’s analytical Jacobian

Random encoder → combinedwith → Fréchet mean

confidence 93% · a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets

Tangent space classifier → classifies → SPD matrices

confidence 92% · projected to the tangent space, and classified

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder's analytical Jacobian and aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP). Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

55,713 characters extracted from source content.

Expand or collapse full text

Riemannian Geometry for Pre-trained Language Model Embeddings Szczepan Konior1 Alexandre Quemy2 Przemysław Klocek1 Bartłomiej Sobieski3,4 Grégoire Cattan1 1IBM Automation and AI, Krakow, Poland 2Hother, Krakow, Poland 3University of Warsaw 4Centre for Credible AI, Warsaw University of Technology name.surname@ibm.com alexandre@hother.io sobieski.bartlomiej.jan@gmail.com Abstract Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder’s analytical Jacobian and aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP). Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets. Riemannian Geometry for Pre-trained Language Model Embeddings Szczepan Konior1 Alexandre Quemy2 Przemysław Klocek1 Bartłomiej Sobieski3,4 Grégoire Cattan1 1IBM Automation and AI, Krakow, Poland 2Hother, Krakow, Poland 3University of Warsaw 4Centre for Credible AI, Warsaw University of Technology name.surname@ibm.com alexandre@hother.io sobieski.bartlomiej.jan@gmail.com 1 Introduction Pre-trained language models achieve remarkable success in natural language understanding, yet the geometric structure of their internal representations remains an open challenge. Most analyses treat token embeddings as points in flat Euclidean space, but three lines of evidence motivate departing from that assumption: hierarchical structure in language admits lower-distortion embeddings in negatively curved spaces (Nickel and Kiela, 2017, 2018; Sala et al., 2018; Park et al., 2025) and is reflected in non-standard inner products on the embedding space (Park et al., 2024; Marks and Tegmark, 2024); contextualized representations in BERT, ELMo, and GPT-2 are strongly anisotropic, occupying narrow cones not invariant under Euclidean transformation (Ethayarajh, 2019; Kataiwa et al., 2025); and token embedding spaces may fail to form smooth manifolds globally, even if local-manifold-like structure still supports geometric analysis at the neighbourhood level (Robinson et al., 2025). This work investigates whether sentence-level classification signal lives in the Riemannian geometry of token embeddings. We address three questions: (1) Do embeddings exhibit local geometric structure accessible via pullback metrics? (2) Does this geometric structure carry signal beyond what flat Euclidean aggregation extracts? (3) Which component of a geometric aggregation pipeline is responsible for any observed gain: the encoder, the metric, or the aggregation? We evaluate our approach, Riemannian Mean Pooling (RMP), on four diverse NLP tasks: fact verification (FEVER-Symmetric), textual entailment (RTE), grammatical acceptability (CoLA), and commonsense reasoning (CREAK). FEVER-Symmetric is included specifically as a negative control: it was constructed to remove lexical and annotation artifacts (Schuster et al., 2019), and we expect any method relying solely on claim-surface geometry to perform at chance. Any differentiable encoder yields a pullback metric through its Jacobian, but the metric’s geometric interpretability depends on the training objective. We use Intrinsic Green’s Learning (IGL) (Quemy, 2026)111Reference implementation: https://github.com/hotherio/intrinsic-green-learning.: a closed-form kernel readout replaces the learned decoder, constraining the encoder to coordinates aligned with the smooth structure of the data manifold rather than with arbitrary nonlinear features. Our ablations (Section 5) test this choice by comparing IGL against random and untrained encoders. Contributions • We instantiate a pipeline that extracts per-token pullback metrics from a learned encoder’s Jacobian, aggregates them on the SPD manifold via the Fréchet mean, and classifies in the tangent space at the population Riemannian mean. • Across three signal-bearing datasets (RTE, CoLA, CREAK), Riemannian aggregation consistently outperforms Euclidean mean pooling under a controlled comparison on identical embeddings; on FEVER-Symmetric, included as a negative control, the method correctly stays at chance. • A randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling, localising the source of the geometric signal to the aggregation rather than to learned manifold structure. The trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets. 2 Related work Riemannian Methods in Neuroscience Riemannian geometry is an ubiquitous machine learning technique in brain-computer interfaces (BCIs) and signal processing Congedo et al. (2017). Barachant et al. (2012) introduced the use of symmetric positive definite (SPD) matrices as a descriptor for electroencephalography time series, demonstrating that geometric methods can capture the intrinsic structure of signals better than Euclidean approaches. The riemannian minimum-distance-to-means (MDM) and tangent space (TS) classifiers have become ubiquitous tools for neurosciences application, still unbeaten even by the most recent deep learning algorithms Chevallier et al. (2024), while remaining simple, deterministic, robust to noise, computationally efficient, and prone to transfer learning Andreev et al. (2025). Beyond the BCI setting, SPD-manifold deep learning has matured: differentiable Fréchet-mean layers (Lou et al., 2020), Riemannian batch normalisation for SPD networks (Brooks et al., 2019), and Riemannian extensions of logistic regression (Chen et al., 2024) now provide standard components for end-to-end SPD pipelines, with recent surveys covering the area (Ju et al., 2024). Geometric Approaches in NLP In modern NLP architectures, tokens are represented as vectors in a high-dimensional latent space. As tokens propagate through the network layers, their representations evolve inside the Residual Stream, the model’s dynamic internal state. This evolution is not purely linear or syntactic, but can be described as a structured trajectory whose geometry changes depending on the semantic context Manson (2025). Concrete instances of this structure have been demonstrated empirically: linear directions in LLM representations encode space, time, and other conceptual attributes (Gurnee and Tegmark, 2024), and architectural variants explicitly equip attention with non-Euclidean geometry, either via mixed-curvature transformers (Cho et al., 2023) or via parallel-transport-based attention on manifolds (Ji, 2025). Recent work has also proposed that the latent space itself can be modelled as a curved spacetime whose curvature reflects linguistic structure (Di Sipio et al., 2025). Current approaches estimate local tangent spaces from k-nearest-neighbor (kNN) patches and derive curvature information from variations between neighboring tangent directions (e.g., Gao and Liang (2011); Li (2017)). However, these methods often suffer from numerical instability and sensitivity to sampling density, with metric estimates degrading as the local sample size shrinks (Brand, 2003; Singer, 2006). A different style of approach first projects the data to a lower-dimensional space (for example via UMAP (McInnes et al., 2018)) and computes metrics in the projection, but UMAP preserves local topology rather than metric structure: distances after UMAP are unsuitable as approximations of distances on the original manifold, so subsequent geometric computations operate on a non-isometric image of the data. Recent work has also questioned whether token embeddings form a single connected manifold at all (Robinson et al., 2025); the question we ask here is whether aggregated token geometry nevertheless carries class signal at the sentence level, regardless of whether global manifold structure holds. Intrinsic Dimension and Manifold Learning Sobieski et al. (2026) showed that local intrinsic dimension (LID) reveals hallucinations in diffusion models by identifying instabilities on model-induced manifolds. Their work treats hallucinations as geometric instabilities, proposing Intrinsic Quenching to correct these anomalies. Similarly, Damirchi et al. (2026) introduced "Truth as a Trajectory," analysing LLM reasoning through layer-wise geometric metrics rather than static activations. By examining how representations evolve across layers, they uncover geometric invariants distinguishing valid reasoning from spurious behaviour. Beyond these layer-wise trajectory analyses, a body of recent work uses intrinsic dimension as a probe of how LLMs process information across layers: representations expand in early layers and compress in later layers (Valeriani et al., 2023), with characteristic per-layer profiles that include a high-dimensional abstraction phase in the middle layers (Cheng et al., 2025). ID has also been used to compare learning paradigms such as supervised fine-tuning and in-context learning (Janapati and Ji, 2025). Our work complements these perspectives. Sentence Representation Learning Standard sentence-representation approaches such as mean pooling of token embeddings and the BERT [CLS] token implicitly assume flat Euclidean geometry. We replace arithmetic averaging with an SPD-manifold Fréchet aggregation. Recent work has also highlighted the entanglement of pooled sentence representations (Tehenan et al., 2025), motivating geometric alternatives. 3 Material & Method 3.1 Datasets We evaluate on four binary-classification benchmarks. FEVER-Symmetric (Schuster et al., 2019) contains 316 fact-verification claims, debiased by Schuster et al. against the annotation artifacts of the original FEVER (Thorne et al., 2018) by pairing each claim with its negation against shared evidence; we include it as a negative control, since methods relying solely on claim-surface features cannot exceed chance by construction. RTE (Wang et al., 2018): 2,490 premise–hypothesis pairs for textual entailment. CoLA (Warstadt et al., 2018): 3,000 sentences labelled for grammatical acceptability. CREAK (Onoe et al., 2021): 3,000 commonsense statements requiring world knowledge to verify. 3.2 Baseline Method: Linear Probe We use Linear Probe (Alain and Bengio, 2017) as the headline baseline, referring to this baseline as the Linear Probe throughout and reserving the term Euclidean mean pooling for the underlying aggregation operation when contrasting it with the Riemannian alternative. The baseline operates entirely on internal states of the pre-trained model, isolating the contribution of geometric aggregation from any additional trainable parameters or external information sources. Token embeddings and pooling. We extract token-level embeddings from the ninth hidden layer of BERT-base-uncased (Devlin et al., 2019), yielding 768-dimensional representations per token. The layer choice is motivated by prior work showing that upper layers encode more abstract semantic features while lower layers capture syntactic information (Rogers et al., 2020). For single-sentence datasets (FEVER, CoLA, CREAK) we encode the input directly; for the textual-entailment task (RTE) we concatenate premise and hypothesis with separator tokens ([CLS] [premise] [SEP] [hypothesis] [SEP]). Token embeddings are aggregated via arithmetic mean pooling: sentence=1n∑i=1ni∈ℝ768,v_sentence= 1n _i=1^nv_i ^768, (1) with 1,…,n\v_1,…,v_n\ the token embeddings for a sentence with n tokens. Classification. Pooled sentence representations are normalised and classified with a logistic regression using scikit-learn’s (Pedregosa et al., 2011) default hyperparameters (no regularisation, LBFGS solver, max-iter 1000). The same classification head is used for the Riemannian pipeline downstream of the Fréchet aggregation (Section 3.5), so performance differences attribute cleanly to the aggregation step. 3.3 Intrinsic Green’s Learning Inverse-PDE objective. IGL (Quemy, 2026) is a framework for learning low-dimensional manifold representations through an inverse-PDE formulation. A small MLP encoder Ψθ:ℝD→ℝdmax _θ:R^D ^d_ maps each input ∈ℝDv ^D to a latent code =Ψθ() ξ= _θ(v). In the encoder’s coordinates, the target function (here, the input itself) is expressed as the solution of a linear PDE Lu=s,Lu=s, (2) where L is a fixed operator (a Laplacian in our setup whose Green’s function is approximated by a Gaussian kernel) and s is a learned source term. Closed-form kernel readout. The PDE is solved in closed form via a Green’s-function kernel: at any latent point ξ, the reconstruction is a linear combination of kernel evaluations at K learned anchor points kk=1K\ ζ_k\_k=1^K, u^()=∑k=1KwkG(,k), u( ξ)= _k=1^Kw_k\,G( ξ, ζ_k), (3) where G is the Gaussian kernel. The weights w are not learnable parameters: at every minibatch they are recomputed by ridge regression on the reconstruction targets (variable projection). Encoder-only optimisation. IGL has no symmetric decoder. Because w is closed-form, gradient descent operates only on the encoder parameters θ, and the encoder must find coordinates in which the input admits a low-rank kernel decomposition: coordinates aligned with the smooth structure of the data manifold rather than with arbitrary nonlinear features a learned decoder could compensate for. As a byproduct, because Ψθ _θ is a differentiable MLP, its Jacobian J()=∂Ψθ/∂J(v)=∂ _θ/ is analytically computable, giving direct access to the pullback metric at every input point. Under the configuration used in this work (MLP encoder, dmax=64d_ =64, K=128K=128 anchors, 4 Gaussian scales per factor), training converges to stable reconstruction. Full architectural details are deferred to Appendix A. The full pipeline (training top, inference bottom) is shown schematically in Figure 1. A representative validation-loss curve is shown in Appendix D. TokenembeddingsEncoder Ψθ _θLatentcode zzProductGreen’s kernelVariableprojectionReconstructedembeddingsreconstruction lossIGL pipelinePer-tokenJacobianPullbackmetricSentence desc.(Fréchet mean)RiemannianwhiteningTangent-spaceclassifierPredictedclass Figure 1: Overview of the pipeline. Top row (training): the encoder Ψθ _θ is trained to reconstruct token embeddings (early stopping with patience=50, δ=2×10−5δ=2× 10^-5), mapping them to a low-dimensional latent code from which a product Green’s kernel with variable projection recovers the input; the reconstruction loss flows back to the encoder (dashed arrow). Bottom row (inference): once trained, the encoder’s Jacobian induces a pullback metric at each token; the per-token metrics are aggregated on the SPD manifold via the Fréchet mean to a sentence-level descriptor, preprocessed with Riemannian whitening, projected to the tangent space, and classified. The classification (inference) path carries no gradients and uses the encoder frozen post-training; only the reconstruction (training) path is differentiable. The full architecture with formulas is given in Figure 3 of Appendix A. 3.4 Pullback metric extraction After training the IGL encoder Ψθ:ℝ768→ℝdmax _θ:R^768 ^d_ on token-level BERT embeddings ∈ℝ768v ^768 (Section 3.3), we extract a local Riemannian metric at each token via the pullback metric. The intuition is that Ψθ _θ maps the ambient embedding space to a lower-dimensional latent space; the pullback metric measures distances in the input space as they appear after this mapping, capturing how the encoder locally stretches or compresses different directions of the embedding space. From Jacobian to metric. For an input ∈ℝ768v ^768, the Jacobian of the encoder is J()=∂Ψθ∂∈ℝdmax×768,J(v)= ∂ _θ ^d_ × 768, (4) computed in closed form by automatic differentiation through Ψθ _θ. Pulling back the ambient Euclidean metric I768I_768 through the encoder yields the co-metric on the latent manifold: g−1()=J()J()⊤∈ℝdmax×dmax.g^-1(v)=J(v)\,J(v) ^d_ × d_ . (5) By construction g−1()g^-1(v) is symmetric and positive semi-definite. To obtain a metric tensor that is strictly positive-definite (required for the Riemannian operations in Section 3.5), we add a small regulariser before inversion: g()=(g−1()+ϵIdmax)−1,ϵ=10−6,g(v)= (g^-1(v)+ε I_d_ )^-1, ε=10^-6, (6) and a further additive regularisation λIdmaxλ I_d_ with λ=10−2λ=10^-2 to guarantee numerical stability across the full range of input tokens. The resulting g()∈++dmaxg(v) _++^d_ is the per-token SPD matrix used as input to the sentence-level aggregation. Why pullback rather than direct estimation. The two most direct alternatives—kNN-based metric estimation and projection-then-metric approaches such as UMAP—suffer from sensitivity to local sampling density and non-isometric distortion, respectively (Section 2). The pullback through a trained encoder avoids both issues: the metric is computed analytically from J rather than estimated from neighbours, and its position dependence is a smooth function of the original 768-dimensional input rather than of a non-isometric projection of it. Pulling back an ambient metric through an encoder’s Jacobian follows Hauser and Ray (2017) and Arvanitidis et al. (2018); recent extensions analyse discretisation of continuous inputs (Brandon et al., 2025) and score-based extraction of data-manifold geometry (Diepeveen et al., 2024). Independence from the training loss. Because the pullback metric depends only on the encoder’s Jacobian rather than on its training loss, the geometric pipeline downstream of g()g(v) is well-defined for any differentiable encoder. We exploit this in Section 5 to isolate the contribution of the encoder from the contribution of the geometric aggregation. 3.5 Sentence aggregation and classification Token-to-Sentence Aggregation Given a sentence with n tokens producing per-token SPD matrices g(1),…,g(n)∈++dmax\g(v_1),…,g(v_n)\ _++^d_ (Section 3.4), we aggregate them into a single sentence-level SPD descriptor using the Fréchet mean (Fletcher et al., 2004) on the SPD manifold: Gsentence=arg⁡minG∈++dmax∑i=1ndRiem2(G,g(i)),G_sentence= G _++^d_ _i=1^nd_Riem^2(G,g(v_i)), (7) with dRiemd_Riem the affine-invariant Riemannian distance (Förstner et al., 2000): dRiem(G1,G2)=‖log⁡(G1−1/2G2G1−1/2)‖F.d_Riem(G_1,G_2)= \| (G_1^-1/2G_2G_1^-1/2) \|_F. (8) GsentenceG_sentence is the geometric counterpart of Euclidean mean pooling: a Fréchet-mean point on the SPD manifold rather than an arithmetic mean. Classification The sentence-level SPD matrices are classified in three steps. First, we apply Riemannian whitening, which rescales each matrix to unit determinant while preserving its affine-invariant geometric properties, removing scale differences across samples. Second, we project each whitened matrix to the tangent space at the population Riemannian mean of the training set; the tangent space at a point on the SPD manifold is a flat Euclidean space, so standard linear classifiers apply. Third, we fit a logistic regression on the vectorised tangent-space representations. We use pyRiemann’s (Barachant et al., 2026) TSClassifier with default hyperparameters, which implements the projection-plus-linear-classifier pipeline. 3.6 Experimental configuration All experiments employed 5-fold stratified cross-validation to ensure balanced class distributions across folds, using two seeds. We report the mean and standard deviation across folds to capture performance variability.222The headline comparison (Table 1) and the ablation experiment (Table 2, appendix) are independent runs with identical protocol but different random initialisations; small numerical differences for the shared rows are within the reported cross-seed variance. Because the classification pipeline reuses established BCI components (e.g., Fréchet mean of SPD matrices, Riemannian whitening, and tangent-space classifier) we adopt the evaluation and statistical protocol of the Mother of all BCI Benchmarks (MOABB) (Aristimunha et al., 2023). Statistical significance was computed independently for each dataset using significance thresholds of α<0.05α<0.05 and α<0.01α<0.01. Effect sizes are reported using the standardised mean difference (SMD), and the aggregated meta-effect across datasets is also provided. 4 Experimental results Table 1 reports two baselines, Linear Probe (Euclidean mean pooling) and CLS Token Aggregation, alongside Riemannian Mean Pooling on the three signal-bearing datasets (CoLA, CREAK, RTE) and the FEVER-Symmetric negative control, with means and standard deviations across the multi-seed cross-validation protocol. Table 1: Multi-seed meta-analysis results (mean ± std across 10 evaluations). †FEVER is included as a negative control (Section 3); chance-level performance is the expected and correct behaviour, so no method is bolded for this dataset. Dataset Method Accuracy F1-Score AUC FEVER† Linear Probe 0.407 ± 0.044 0.429 ± 0.035 0.424 ± 0.028 CLS Token Aggregation 0.374 ± 0.049 0.397 ± 0.050 0.402 ± 0.050 Riemannian mean pooling 0.486 ± 0.038 0.488 ± 0.057 0.508 ± 0.047 CoLA Linear Probe 0.705 ± 0.022 0.706 ± 0.025 0.772 ± 0.025 CLS Token Aggregation 0.661 ± 0.013 0.662 ± 0.015 0.711 ± 0.012 Riemannian mean pooling 0.723 ± 0.015 0.730 ± 0.018 0.799 ± 0.018 CREAK Linear Probe 0.606 ± 0.018 0.606 ± 0.020 0.648 ± 0.018 CLS Token Aggregation 0.594 ± 0.024 0.594 ± 0.025 0.631 ± 0.023 Riemannian mean pooling 0.656 ± 0.015 0.654 ± 0.019 0.708 ± 0.019 RTE Linear Probe 0.560 ± 0.014 0.555 ± 0.019 0.591 ± 0.024 CLS Token Aggregation 0.574 ± 0.027 0.581 ± 0.027 0.611 ± 0.027 Riemannian mean pooling 0.603 ± 0.012 0.604 ± 0.014 0.638 ± 0.023 Figure 2 presents forest plot comparing the distribution of AUC scores across all 10 evaluations for each dataset. Figure 2: Statistical comparison between Linear Probe and Riemannian mean pooling across random seeds. Box plots show the distribution of AUC scores obtained from 10 evaluations (2 seeds × 5 folds). Corresponding p-values are displayed on the right side of each plot. One (respectively, two and three) red star(s) indicate that the p-value is below the significance threshold of α<0.05α<0.05 (respectively, α<0.01α<0.01, α<0.001α<0.001). Effect sizes are reported using SMD. Following common interpretation guidelines, absolute SMD values around 0.2, 0.5, and 0.8 correspond to small, medium, and large effects, respectively. FEVER-Symmetric is included as a negative control (Section 3): a chance-level result is the expected and correct behaviour, confirming that the pipeline does not exploit lexical artifacts. Per-dataset AUC, Accuracy, and F1 bar plots together with the CLS-vs-RMP forest plot (Figures 10, 11, 12, 13) are deferred to Appendix C; Figure 2 and Table 1 carry the headline comparison in the body. The multi-seed meta-analysis shows Riemannian Mean Pooling outperforming both baselines (Linear Probe and CLS Token Aggregation) on CoLA, CREAK, and RTE, the three datasets with classification signal accessible from claim surface. CLS Token Aggregation underperforms Linear Probe on three of four datasets (FEVER, CoLA, CREAK), with RTE the exception (0.611 vs 0.591 AUC). This replicates the observation of Reimers and Gurevych (2019) that vanilla CLS embeddings underperform mean pooling on transfer tasks without task-specific fine-tuning, and provides a second controlled Euclidean baseline against which to measure the Riemannian gain. On FEVER-Symmetric, both baselines and Riemannian Mean Pooling stay at chance, the behaviour expected from a negative control. Although the Riemannian mean pooling point estimate of 0.5080.508 AUC nominally exceeds 0.50.5, it falls within the 95 % confidence interval [0.443, 0.557][0.443,\,0.557] derived from permutation testing (Müller-Putz et al., 2008; Maris and Oostenveld, 2007), confirming that the result is indistinguishable from chance. 5 Decomposing the geometric gain 5.1 Ablation Components To localise the source of the geometric gain, we ablate the pipeline along two axes: (i) whether the encoder is trained or randomly initialised, and (i) whether the encoder is the IGL MLP architecture or a simpler random linear projection. Encoder-Only projects tokens through the trained IGL encoder and applies Euclidean mean pooling in latent space (no metric extraction), testing whether the encoder alone matters. Frozen Random IGL replaces the trained encoder with a randomly initialised MLP of identical architecture (256-d hidden, SiLU) and runs the full geometric pipeline, testing whether encoder training matters. Random Projection + SiLU uses a frozen random linear projection ℝ768→ℝ64R^768 ^64 followed by SiLU, then the full geometric pipeline, testing whether the depth and nonlinearity of the IGL MLP matter even without training. 5.2 Experimental Results The full ablation table (Table 2) and per-comparison forest/bar plots are deferred to Appendix B; the qualitative findings are summarised below. 5.3 Findings Three patterns emerge from Table 2. First, on CoLA and RTE, the gap between the trained IGL encoder and a randomly initialised one is within one standard deviation in AUC (CoLA: 0.778 vs 0.762; RTE: 0.635 vs 0.616), indicating that the geometric aggregation, rather than encoder training, carries most of the gain. Second, on CREAK the trained encoder gains about 3.6 AUC points over the random one (0.703 vs 0.667), and CREAK is also the most knowledge-heavy of the three signal-bearing datasets, suggesting that the encoder contributes specifically where labels depend on world knowledge rather than on local linguistic structure. Third, the simpler Random Projection + SiLU underperforms the random IGL encoder on every signal-bearing dataset, indicating that the depth and nonlinearity of the MLP matter even without training. The FEVER-Symmetric rows are reported for completeness but, by the negative-control framing of Section 3, are not informative about the source of geometric signal. Taken together, the ablations support the central claim: a randomly initialised encoder combined with geometric aggregation already beats Euclidean pooling, which is stronger evidence that the signal lives in the geometry than if only a carefully trained encoder could extract it. The encoder then does additional work on harder, composition-heavy tasks. 6 Discussion 6.1 Hypothesis validation Geometric advantage and what it tests. Riemannian aggregation of token-level pullback metrics outperforms Euclidean mean pooling on the three signal-bearing datasets and stays at chance on the FEVER-Symmetric negative control (Table 1; Figure 2). The gain is largest on CREAK (ΔAUC≈0.060 ≈ 0.060) and smallest on CoLA (ΔAUC≈0.027 ≈ 0.027), tracking how non-linearly separable class structure is in the raw embedding. The ablations localise most of this gain to the geometric aggregation rather than to learned manifold structure: random encoders combined with Fréchet aggregation already beat Euclidean pooling on CoLA and RTE, while the trained encoder contributes specifically on CREAK (+3.6 AUC points), the most knowledge-heavy of the three (Table 2, with per-comparison forest plots in Appendix B). Rather than concluding that BERT embeddings lie on a globally coherent manifold, the evidence supports locally manifold-like structure (Robinson et al., 2025): aggregated geometric features yield classification gains even where global manifold structure fails. Role of the negative control The inclusion of FEVER-Symmetric (Schuster et al., 2019) serves a methodological purpose distinct from the other three datasets. Where CoLA, CREAK, and RTE probe whether geometric structure encodes specific linguistic properties, FEVER-Symmetric probes whether the method exploits residual signal in the underlying representations. The ablation pattern on this dataset is informative. The Encoder-Only ablation reaches 0.553±0.0370.553± 0.037 AUC on FEVER-Symmetric, above chance, while the full Riemannian pipeline and both random-encoder ablations remain at chance (Table 2). FEVER-Symmetric was constructed by Schuster et al. to remove annotation-driven lexical and stylistic artifacts in the original FEVER, specifically by pairing each claim with its negation against shared evidence. This construction targets surface features arising from the annotation process; it does not target all model-accessible content features. BERT representations encode substantial world knowledge through pre-training, and the trained IGL encoder exposes some of this knowledge in its learned coordinates, sufficient for Euclidean classification to detect a small signal that Schuster’s debiasing does not remove. The full Riemannian pipeline does not access this residual signal. The pullback metric depends on the encoder’s Jacobian rather than on the value of its coordinates: it captures how the encoder locally bends the embedding space rather than which features it preserves. World-knowledge content is feature content, and the geometric pipeline filters it by construction. This is supported by the random-encoder ablations: both Frozen Random IGL (0.443 AUC) and Random Projection + SiLU (0.499 AUC) stay at or below chance on FEVER-Symmetric, because random encoders do not learn to expose specific content features. The below-chance reading on Frozen Random IGL is consistent with sample-specific bias on the 316-sample FEVER-Symmetric set rather than with a learned content signal. The pattern (trained encoder + Euclidean aggregation as the only above-chance method, geometric methods at or below chance regardless of encoder training) supports rather than weakens the negative control: the geometric pipeline behaves correctly even when residual signal is present in the underlying representations and accessible to simpler methods. What does the pullback metric encode? The random-encoder ablations clarify what the pipeline is actually testing. The pullback metric g()=(JJ⊤+ϵI)−1g(v)=(J +ε I)^-1 captures the local anisotropy of the encoder map: how a unit input perturbation at v is stretched or compressed across latent dimensions. For a random MLP with SiLU nonlinearity, J()J(v) varies smoothly with v because of the nonlinearity, even though the encoder weights themselves are random. Consequently g()g(v) encodes the position of v in input space through the local Jacobian structure of the random transformation, not through any property the encoder learned. Fréchet aggregation on the SPD manifold then summarises this local-geometry signal into a sentence-level descriptor. The geometric-aggregation gain over Euclidean pooling thus reflects that token positions in BERT embeddings carry classification signal in their local nonlinear-transformation structure, which Euclidean averaging discards. 6.2 Limitations and challenges Computational cost. On top of one-time IGL training, each sentence requires a per-token Jacobian and an SPD Fréchet-mean iteration, an order of magnitude more arithmetic than Euclidean pooling, though still negligible against LLM inference or gradient-based fine-tuning. Exact wall-clock and FLOP figures are reserved for a scaling study. Hyperparameter and architectural scope. Our encoder-width and anchor-count sweep was conducted only on FEVER-Symmetric, so sensitivity on the signal-bearing datasets and to the metric regulariser λ is unverified. The encoder objective itself was held fixed at IGL; alternative objectives that also yield a Jacobian (vanilla autoencoders, VAEs, normalising flows, contrastive encoders) were not swept, although the random-encoder ablation already suggests the specific objective is largely transparent on most tasks. Scope of the controlled comparison. The pipeline was evaluated against Euclidean and CLS-token baselines on a single encoder (BERT-base) at a single layer (layer 9) and a fixed projection dimensionality. This is a deliberate methodological choice: the comparison isolates the contribution of geometric aggregation on identical embeddings, which is why fine-tuned BERT and Sentence-BERT (Reimers and Gurevych, 2019) are excluded rather than missing. Absolute classification performance is not the question this paper investigates; whether geometric aggregation extracts signal that flat-Euclidean pooling discards is. 6.3 Future work Architecture and layer coverage. Extending the evaluation to larger and structurally different encoders (RoBERTa, DistilBERT, decoder-only LLMs) and across layers would clarify whether the geometric-aggregation gain is general or specific to BERT-base layer 9, and whether the layer-wise intrinsic-dimension trajectory (Valeriani et al., 2023; Cheng et al., 2025) interacts with where the pullback metric becomes informative. Richer SPD descriptors. The Fréchet mean is the simplest aggregator on the SPD manifold; weighted Fréchet means, trajectory-based descriptors in the spirit of layer-wise geometric probes (Damirchi et al., 2026; Mir, 2025), and explicit curvature features are natural next steps, with per-token pullback eigenvalues offering a route to relate the geometric signal to interpretable linguistic phenomena. 7 Conclusion The SPD Fréchet mean of per-token pullback metrics beats Euclidean mean pooling on CoLA, CREAK and RTE under a controlled comparison on identical embeddings; on FEVER-Symmetric, included as a negative control, the method correctly stays at chance, confirming it does not exploit the lexical and annotation artifacts the dataset was constructed to remove. Random-encoder ablations localise most of the gain to the geometric aggregation rather than to learned manifold structure, with the trained encoder contributing additional signal specifically on CREAK, the most knowledge-heavy of the three datasets. Together these results support the framing of embeddings as locally manifold-like rather than globally coherent (Robinson et al., 2025): aggregated geometric features yield classification gains even where global manifold structure may fail. References Alain and Bengio (2017) Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR) Workshop Track. Andreev et al. (2025) Anton Andreev, Gregoire Cattan, and Marco Congedo. 2025. The Riemannian Means Field Classifier for EEG-Based BCI Data. Sensors, 25(7):2305. Aristimunha et al. (2023) Bruno Aristimunha, Igor Carrara, Pierre Guetschel, Sara Sedlar, Pedro Rodrigues, Jan Sosulski, Divyesh Narayanan, Erik Bjareholt, Barthélémy Quentin, Robin Tibor Schirrmeister, Emmanuel Kalunga, Ludovic Darmet, Cattan Gregoire, Ali Abdul Hussain, Ramiro Gatti, Vladislav Goncharenko, Jordy Thielen, Thomas Moreau, Yannick Roy, and 3 others. 2023. Mother of all BCI benchmarks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. Arvanitidis et al. (2018) Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. 2018. Latent space oddity: On the curvature of deep generative models. In International Conference on Learning Representations (ICLR). Barachant et al. (2026) Alexandre Barachant, Quentin Barthélemy, Jean-Rémi King, Alexandre Gramfort, Sylvain Chevallier, Pedro L. C. Rodrigues, Emanuele Olivetti, Vladislav Goncharenko, Gabriel Wagner vom Berg, Ghiles Reguig, Arthur Lebeurrier, Erik Bjäreholt, Maria Sayu Yamamoto, Pierre Clisson, Marie-Constance Corsi, Igor Carrara, Apolline Mellot, Bruna Junqueira Lopes, Brent Gaisford, and 4 others. 2026. pyriemann. Barachant et al. (2012) Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. 2012. Multiclass brain-computer interface classification by Riemannian geometry. IEEE transactions on bio-medical engineering, 59(4):920–928. Brand (2003) Matthew Brand. 2003. Charting a manifold. In Advances in Neural Information Processing Systems (NeurIPS). Brandon et al. (2025) Julian Brandon, Angus Chadwick, and Arthur Pellegrino. 2025. Emergent Riemannian geometry over learning discrete computations on continuous manifolds. arXiv preprint arXiv:2512.00196. Brooks et al. (2019) Daniel Brooks, Olivier Schwander, Frédéric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. 2019. Riemannian batch normalization for SPD neural networks. In Advances in Neural Information Processing Systems (NeurIPS). Chen et al. (2024) Ziheng Chen, Yue Song, Rui Wang, Xiaojun Wu, and Nicu Sebe. 2024. RMLR: Extending multinomial logistic regression into general geometries. In Advances in Neural Information Processing Systems (NeurIPS). Cheng et al. (2025) Emily Cheng, Diego Doimo, Corentin Kervadec, Iuri Macocco, Jade Yu, Alessandro Laio, and Marco Baroni. 2025. Emergence of a high-dimensional abstraction phase in language transformers. In International Conference on Learning Representations (ICLR). Chevallier et al. (2024) Sylvain Chevallier, Igor Carrara, Bruno Aristimunha, Pierre Guetschel, Sara Sedlar, Bruna Lopes, Sebastien Velut, Salim Khazem, and Thomas Moreau. 2024. The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark. arXiv preprint. ArXiv:2404.15319. Cho et al. (2023) Sungjun Cho, Seunghyuk Cho, Sungwoo Park, Hankook Lee, Honglak Lee, and Moontae Lee. 2023. Curve your attention: Mixed-curvature transformers for graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence. Congedo et al. (2017) Marco Congedo, Alexandre Barachant, and Rajendra Bhatia. 2017. Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review. Brain-Computer Interfaces, 4(3):155–174. Damirchi et al. (2026) Hamed Damirchi, Ignacio Meza De la Jara, Ehsan Abbasnejad, Afshar Shamsi, Zhen Zhang, and Javen Shi. 2026. Truth as a trajectory: What internal representations reveal about large language model reasoning. Preprint, arXiv:2603.01326. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT). Di Sipio et al. (2025) Riccardo Di Sipio, Jairo Diaz-Rodriguez, and Luis Serrano. 2025. The curved spacetime of transformer architectures. arXiv preprint arXiv:2511.03060. Diepeveen et al. (2024) Willem Diepeveen, Georgios Batzolis, Zakhar Shumaylov, and Carola-Bibiane Schönlieb. 2024. Score-based pullback Riemannian geometry: Extracting the data manifold geometry using anisotropic flows. arXiv preprint arXiv:2410.01950. Ethayarajh (2019) Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Fletcher et al. (2004) P Fletcher, Conglin Lu, Stephen Pizer, and Sarang Joshi. 2004. Principal geodesic analysis for the study of nonlinear statistics of shape. Medical Imaging, IEEE Transactions on, 23:995 – 1005. Förstner et al. (2000) Wolfgang Förstner, Boudewijn Moonen, and Carl Gauss. 2000. A metric for covariance matrices. Gao and Liang (2011) Xiaofang Gao and Jiye Liang. 2011. The dynamical neighborhood selection based on the sampling density and manifold curvature for isometric data embedding. Pattern Recognition Letters, 32(2):202–209. Gurnee and Tegmark (2024) Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In International Conference on Learning Representations (ICLR). Hauser and Ray (2017) Michael Hauser and Asok Ray. 2017. Principles of Riemannian geometry in neural networks. In Advances in Neural Information Processing Systems (NeurIPS). Janapati and Ji (2025) Saahith Janapati and Yangfeng Ji. 2025. A comparative study of learning paradigms in large language models via intrinsic dimension. In Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025), pages 59–86, Albuquerque, NM. Association for Computational Linguistics. Ji (2025) Zhongping Ji. 2025. RiemannFormer: A framework for attention in curved spaces. arXiv preprint arXiv:2506.07405. Ju et al. (2024) Ce Ju, Reinmar Kobler, Antoine Collas, Motoaki Kawanabe, Cuntai Guan, and Bertrand Thirion. 2024. SPD matrix learning for neuroimaging analysis: Perspectives, methods, and challenges. arXiv preprint arXiv:2504.18882. Kataiwa et al. (2025) Takuya Kataiwa, Cho Hakaze, and Tetsushi Ohki. 2025. Measuring intrinsic dimension of token embeddings. arXiv preprint arXiv:2503.02142. Kusupati et al. (2022) Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Advances in Neural Information Processing Systems (NeurIPS). Li (2017) Yangyang Li. 2017. Curvature-aware Manifold Learning. arXiv preprint. ArXiv:1706.07167 [cs.LG]. Lou et al. (2020) Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. 2020. Differentiating through the Fréchet mean. In Proceedings of the 37th International Conference on Machine Learning (ICML). Manson (2025) Rob Manson. 2025. Curved inference: Concern-sensitive geometry in large language model residual streams. Preprint, arXiv:2507.21107. Maris and Oostenveld (2007) Eric Maris and Robert Oostenveld. 2007. Nonparametric statistical testing of EEG- and MEG-data. Journal of Neuroscience Methods, 164(1):177–190. Marks and Tegmark (2024) Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM). McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426. Mir (2025) Amir Hameed Mir. 2025. The geometry of truth: Layer-wise semantic dynamics for hallucination detection. arXiv preprint arXiv:2510.04933. Müller-Putz et al. (2008) Gernot Müller-Putz, Reinhold Scherer, Clemens Brunner, Robert Leeb, and Gert Pfurtscheller. 2008. Better than Random? A closer look on BCI results. International Journal of Bioelektromagnetism, 10:52–55. Nickel and Kiela (2017) Maximilian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems (NeurIPS). Nickel and Kiela (2018) Maximilian Nickel and Douwe Kiela. 2018. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In Proceedings of the 35th International Conference on Machine Learning (ICML). Onoe et al. (2021) Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. Creak: A dataset for commonsense reasoning over entity knowledge. OpenReview. Park et al. (2025) Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025. The geometry of categorical and hierarchical concepts in large language models. In International Conference on Learning Representations (ICLR). Park et al. (2024) Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning (ICML). Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. Quemy (2026) Alexandre Quemy. 2026. Intrinsic Green’s Learning: Supervised Learning on Manifolds via Inverse PDE. In AI & PDE Workshop at ICLR. Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Robinson et al. (2025) Michael Robinson, Sourya Dey, and Tony Chiang. 2025. Token embeddings violate the manifold hypothesis. In Advances in Neural Information Processing Systems (NeurIPS). Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics (TACL), 8:842–866. Sala et al. (2018) Frederic Sala, Christopher De Sa, Albert Gu, and Christopher Ré. 2018. Representation tradeoffs for hyperbolic embeddings. In Proceedings of the 35th International Conference on Machine Learning (ICML). Schuster et al. (2019) Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. Singer (2006) A. Singer. 2006. From graph to manifold laplacian: The convergence rate. Applied and Computational Harmonic Analysis, 21(1):128–134. Special Issue: Diffusion Maps and Wavelets. Sobieski et al. (2026) Bartlomiej Sobieski, Matthew Tivnan, Dawid Płudowski, Michał Jan Włodarczyk, Pengfei Jin, Przemyslaw Biecek, and Quanzheng Li. 2026. Local intrinsic dimension unveils hallucinations in diffusion models. Preprint, arXiv:2605.05026. Tehenan et al. (2025) Matthieu Tehenan, Vikram Natarajan, Jonathan Michala, Milton Lin, and Juri Opitz. 2025. Mechanistic decomposition of sentence representations. arXiv preprint arXiv:2506.04373. Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT. Valeriani et al. (2023) Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. 2023. The geometry of hidden representations of large transformer models. In Advances in Neural Information Processing Systems (NeurIPS). Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. Appendix A IGL architecture details This appendix collects the architectural details of the IGL configuration used in Section 3.3. Figure 3: Detailed architecture of token-level IGL for sentence classification via Riemannian mean pooling. Right path (reconstruction): Token embeddings i∈ℝ768v_i ^768 are encoded to latent space ∈ℝdmaxz ^d_ via encoder Ψθ _θ, then mapped through Product Green’s kernel G(ξ,ζ)G(ξ,ζ) to design matrix Φ . Variable projection (VP) solves for optimal weights w∗=arg⁡minw⁡‖Φw−(y−b)‖2+λ‖w‖2w^*= _w\| w-(y-b)\|^2+λ\|w\|^2 to reconstruct input ^=Φw∗+b v= w^*+b. Left path (classification): The trained encoder’s Jacobian J=∂Ψθ∂J= ∂ _θ induces pullback metrics g(i)∈++dmaxg(v_i) _++^d_max at each token. These are aggregated via Fréchet mean to sentence-level SPD matrix GsentenceG_sentence, preprocessed with Riemannian whitening, projected to tangent space, vectorised, and classified with logistic regression. Solid arrows: forward flow. Dashed red arrow: reconstruction-loss gradient ∇θ‖^−‖2 _θ\| v-v\|^2 flowing back to encoder Ψθ _θ. The encoder is trained jointly with VP solve until reconstruction converges (early stopping with patience=50, δ=2×10−5δ=2× 10^-5), then frozen for metric extraction and classification. Encoder and Matryoshka Truncation A small MLP encoder Ψθ:ℝD→ℝdmax _θ:R^D ^d_ maps input to latent codes ξn=Ψθ(vn) _n= _θ(v_n) with dmax=64d_ =64. During training, a random truncation k∼Uniform1,…,dmaxk \1,…,d_ \ is sampled and trailing coordinates zeroed out, following Matryoshka representation learning (Kusupati et al., 2022). This ensures earlier dimensions capture the most informative features, with effective dimension dmaxd_max determined post-training from the dimension-loss curve. Product Green’s Kernel The reconstruction target is the input vnv_n itself. We model the reconstruction by a tensor-factorised product Green’s kernel: G(ξ,ζ)≈∑k=1Kγk∏j=1dmaxGk,j(ξj,ζj),K=nanchors=128G(ξ,ζ)≈ _k=1^K _k _j=1^d_ G_k,j( _j, _j),\\ K=n_anchors=128 56.9055pt (9) Each factor Gk,jG_k,j is a one-dimensional kernel evaluated at the latent coordinate ξjξ^j and an anchor ζkjζ^j_k, with K anchors placed adaptively. We use Gaussian kernel factors: each factor is itself a convex mixture of nscales=4n_scales=4 Gaussian basis functions at learnable bandwidths σs,j _s,j, Gk,j(ξj,ζjk)=∑s=14αsexp⁡(−(ξj−ζjk)22σs,j2),a=softmax(⋅)G_k,j( _j, _j^k)= _s=1^4 _s (- ( _j- _j^k)^22 _s,j^2 ),\\ a=softmax(·) 56.9055pt (10) Variable-Projection Reconstruction Each minibatch solves bias-centered ridge regression for weights: w∗=arg⁡minw⁡‖Φw−(v−b)‖2+λ‖w‖2,λ=10−3w^*= _w\| w-(v-b)\|^2+λ\|w\|^2, λ=10^-3 (11) The design matrix Φ costs O(NKd)O(NKd) to compute, while the closed-form solve is O(NK2+K3)O(NK^2+K^3) and independent of d. Weights w∗w^* are recomputed per minibatch (not stored), so gradient descent operates only on encoder parameters θ. Appendix B Ablation figures The full ablation-results table and per-comparison forest plots and bar plots for the three ablation contrasts (Section 5) are collected here for space. Table 2: Ablation study results (mean ± std across 10 evaluations). Bold indicates best per dataset. F1 is omitted from this table (tracks Accuracy closely on binary tasks); the full Accuracy/F1/AUC table is in Table 1. †FEVER is included as a negative control (Section 3); rows are reported for completeness but are not informative about the source of geometric signal, so no method is bolded. Dataset Method Accuracy AUC FEVER† Linear Probe 0.407 ± 0.044 0.424 ± 0.028 CLS Token Aggregation 0.374 ± 0.049 0.402 ± 0.050 Riemannian Mean Pooling 0.460 ± 0.056 0.508 ± 0.051 Encoder-Only 0.519 ± 0.029 0.553 ± 0.037 Frozen Random IGL 0.405 ± 0.045 0.443 ± 0.047 Random Projection + SiLU 0.491 ± 0.049 0.499 ± 0.052 CoLA Linear Probe 0.712 ± 0.018 0.729 ± 0.022 CLS Token Aggregation 0.661 ± 0.013 0.711 ± 0.012 Riemannian Mean Pooling 0.758 ± 0.009 0.778 ± 0.022 Encoder-Only 0.754 ± 0.010 0.754 ± 0.022 Frozen Random IGL 0.760 ± 0.011 0.762 ± 0.022 Random Projection + SiLU 0.731 ± 0.012 0.699 ± 0.037 CREAK Linear Probe 0.597 ± 0.016 0.632 ± 0.015 CLS Token Aggregation 0.594 ± 0.024 0.631 ± 0.023 Riemannian Mean Pooling 0.647 ± 0.012 0.703 ± 0.013 Encoder-Only 0.625 ± 0.020 0.675 ± 0.018 Frozen Random IGL 0.618 ± 0.016 0.667 ± 0.019 Random Projection + SiLU 0.609 ± 0.016 0.639 ± 0.018 RTE Linear Probe 0.560 ± 0.014 0.591 ± 0.024 CLS Token Aggregation 0.574 ± 0.027 0.611 ± 0.027 Riemannian Mean Pooling 0.596 ± 0.017 0.635 ± 0.022 Encoder-Only 0.585 ± 0.019 0.621 ± 0.023 Frozen Random IGL 0.578 ± 0.023 0.616 ± 0.025 Random Projection + SiLU 0.550 ± 0.021 0.575 ± 0.029 Figure 4: Riemannian Mean Pooling vs Encoder-Only: forest plot. Metric: AUC. Figure 5: AUC bar comparison: Riemannian Mean Pooling vs Encoder-Only. Figure 6: Riemannian Mean Pooling vs Frozen Random IGL: forest plot. Metric: AUC. Figure 7: AUC bar comparison: Riemannian Mean Pooling vs Frozen Random IGL. Figure 8: Riemannian Mean Pooling vs Random Projection + SiLU: forest plot. Metric: AUC. Figure 9: AUC bar comparison: Riemannian Mean Pooling vs Random Projection + SiLU. Appendix C Additional results figures The per-dataset AUC, Accuracy, and F1 bar plots together with the CLS-vs-RMP forest plot are deferred here; the body keeps the headline Linear-Probe-vs-RMP forest plot (Figure 2) and Table 1 as the main visualisations. Figure 10: Multi-seed AUC comparison across datasets. Figure 11: Multi-seed accuracy comparison across datasets. Error bars represent standard deviation across 10 evaluations. Figure 12: Multi-seed F1-score comparison across datasets. Figure 13: Statistical comparison between CLS Token Aggregation and Riemannian Mean Pooling across random seeds. Box plots show the distribution of AUC scores from 10 evaluations (2 seeds × 5 folds). Effect sizes are reported using SMD. FEVER-Symmetric is included as a negative control (Section 3); a chance-level result is the expected and correct behaviour for both methods. Appendix D Reconstruction convergence Figure 14: Training (blue) and validation (red) reconstruction MSE for the IGL encoder on fold 1 of FEVER-Symmetric, over ∼ 310 epochs. The validation curve decreases monotonically from ∼ 0.17 to ∼ 0.14. This is representative of all folds and datasets.

Instant research discovery