Paper deep dive
Rethinking the Harmonic Loss via Non-Euclidean Distance Layers
Maxwell Miller-Golub, Collin Coil, Kamil Faber, Marcin Pietron, Panpan Zheng, Pasquale Minervini, Roberto Corizzo
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/13/2026, 1:08:29 AM
Summary
The paper introduces a generalized framework for harmonic loss in deep learning by replacing the standard Euclidean distance with a variety of non-Euclidean metrics (e.g., Manhattan, Cosine, Mahalanobis, Bray-Curtis). This approach aims to improve model interpretability, training stability, and computational efficiency while mitigating issues like grokking and unbounded weight growth found in cross-entropy loss. The authors evaluate these distance-tailored losses across vision and language models, demonstrating that metrics like cosine distance offer superior trade-offs in performance and sustainability.
Entities (5)
Relation Signals (3)
Harmonic Loss → replaces → Cross-Entropy
confidence 95% · Harmonic loss replaces the conventional inner-product logits and softmax normalization with a distance-based formulation
Cosine Distance → improves → Gradient Stability
confidence 90% · On language modeling tasks, cosine-based harmonic losses improve gradient and learning stability
Non-Euclidean Harmonic Loss → mitigates → Grokking
confidence 90% · Models trained with harmonic loss were also shown to require less data to generalize and to mitigate grokking
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cross-entropy loss has long been the standard choice for training deep neural networks, yet it suffers from interpretability limitations, unbounded weight growth, and inefficiencies that can contribute to costly training dynamics. The harmonic loss is a distance-based alternative grounded in Euclidean geometry that improves interpretability and mitigates phenomena such as grokking, or delayed generalization on the test set. However, the study of harmonic loss remains narrow: only Euclidean distance is explored, and no systematic evaluation of computational efficiency or sustainability was conducted. We extend harmonic loss by systematically investigating a broad spectrum of distance metrics as replacements for the Euclidean distance. We comprehensively evaluate distance-tailored harmonic losses on both vision backbones and large language models. Our analysis is framed around a three-way evaluation of model performance, interpretability, and sustainability. On vision tasks, cosine distances provide the most favorable trade-off, consistently improving accuracy while lowering carbon emissions, whereas Bray-Curtis and Mahalanobis further enhance interpretability at varying efficiency costs. On language models, cosine-based harmonic losses improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross-entropy and Euclidean heads. Our code is available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.10225v2
- Canonical: https://arxiv.org/abs/2603.10225v2
Trouble viewing inline? Open PDF directly →
Full Text
187,234 characters extracted from source content.
Expand or collapse full text
Rethinking the Harmonic Loss via Non-Euclidean Distance Layers Maxwell Miller-Golub American University Washington, DC, USA m9628a@american.edu Collin Coil University of Vermont Burlington, USA Kamil Faber AGH University of Science and Technology Krakow, Poland Marcin Pietron AGH University of Science and Technology Krakow, Poland Panpan Zheng Xinjiang University Ürümqi, China Pasquale Minervini University of Edinburgh Edinburgh, United Kingdom Roberto Corizzo American University Washington, DC, USA rcorizzo@american.edu Abstract Cross-entropy loss has long been the standard choice for training deep neural networks, yet it suffers from interpretability limitations, unbounded weight growth, and inefficiencies that can contribute to costly training dynamics. The harmonic loss is a distance-based alternative grounded in Euclidean geometry that improves inter- pretability and mitigates phenomena such as grokking, or delayed generalization on the test set. However, the study of harmonic loss remains narrow: only the Euclidean distance is explored, and no systematic evaluation of computational efficiency or sustainability was conducted. We extend harmonic loss by systematically investi- gating a broad spectrum of distance metrics as replacements for the Euclidean distance. We comprehensively evaluate distance-tailored harmonic losses on both vision backbones and large language mod- els. Our analysis is framed around a three-way evaluation of model performance, interpretability, and sustainability. On vision tasks, cosine distances provide the most favorable trade-off, consistently improving accuracy while lowering carbon emissions, whereas Bray-Curtis and Mahalanobis further enhance interpretability at varying efficiency costs. On language modeling tasks, cosine-based harmonic losses improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross- entropy and Euclidean harmonic loss. Our code is available at: https: //anonymous.4open.science/r/rethinking-harmonic-loss-5BAB/. CCS Concepts • Computing methodologies→Machine learning; Machine learning; Artificial intelligence; Artificial intelligence;• Theory of computation→ Design and analysis of algorithms. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X Keywords harmonic loss, distance metrics, neural networks, loss functions, deep learning, interpretability, green AI, computational efficiency, cross-entropy, cosine distance ACM Reference Format: Maxwell Miller-Golub, Collin Coil, Kamil Faber, Marcin Pietron, Panpan Zheng, Pasquale Minervini, and Roberto Corizzo. 2018. Rethinking the Harmonic Loss via Non-Euclidean Distance Layers. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 45 pages. https://doi. org/X.X 1 Introduction Cross-entropy is the de facto loss function for classification tasks. However, it has shortcomings in terms of model interpretability and training dynamics. Cross-entropy training provides no inher- ent meaning to the learned weight vectors (they serve as abstract parameters rather than intuitive prototypes) and can drive those weights to grow without bound in pursuit of confident predic- tions [3]. This unbounded weight growth can lead to phenomena such as grokking: a delayed generalization in which the model closes the train–test performance gap only after extensive over- training [45]. Moreover, in high-stakes applications where trans- parency is critical (e.g., healthcare or finance), the opaque nature of cross-entropy–trained models poses challenges for trust and error diagnosis. These issues motivate the exploration of alternative loss functions that may yield more interpretable, efficient, and robust model behavior. Recently, the harmonic loss was proposed as an alternative train- ing objective to address some of these concerns [3]. The harmonic loss replaces the conventional inner-product logits and softmax nor- malization with a distance-based formulation: model predictions are derived from the distances between the sample’s representa- tion and class prototype vectors (learned weight vectors for each class). Intuitively, this means that a model is trained to move each sample toward its correct class center in the feature space rather than simply increasing a classification score. This approach endows the learning process with two key properties: i) scale invariance arXiv:2603.10225v2 [cs.LG] 12 Mar 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. – distance comparisons do not depend on vector norm, and i) fi- nite convergence point – training aims for a distance of zero to the correct prototype. As a result, each class weight converges to an anchor point that can be interpreted as the center of that class’s feature distribution. Empirically, Baek et al. [3]demonstrated that harmonic loss can close the train–test gap faster and yield more interpretable representations than cross-entropy. For example, the learned weight vectors in a harmonic-loss model directly reflect class prototypes, making them semantically meaningful. Models trained with harmonic loss have been shown to require less data to generalize and to mitigate grokking, while achieving competitive or better accuracy on vision and language benchmarks. These findings suggest that distance-based loss functions are a promising direction for improving both performance and transparency in deep learning. However, research on harmonic loss has been limited in scope so far. Baek et al. [3]focused exclusively on Euclidean distance as the metric for their loss function and did not examine the broader impacts on computational efficiency or energy consumption. On the other hand, distance-based metrics have been explored in other contexts and problems. Notably, Coil et al. [12]investigated a wide range of distance measures for a problem of change point detection in concept-drift scenarios for anomaly detection. Their study found that the choice of distance metric can drastically affect both the accuracy and efficiency of detecting distribution shifts. For instance, replacing a costly metric (e.g., Wasserstein) with simpler alterna- tives yielded comparable detection performance at substantially lower computational cost. This evidence that “metric matters” in learning algorithms raises a natural question: might other distance measures offer advantages over Euclidean in a harmonic loss setting? To date, no work has evaluated harmonic loss using distance met- rics beyond Euclidean, nor has it benchmarked their impacts across different domains. In this paper, we present the first comprehensive study of custom distance-based loss functions in deep learning classification, extend- ing the harmonic loss framework to a variety of distance measures across multiple problem domains. We experiment with a rich set of non-Euclidean distance metrics, including Manhattan, Euclidean, Chebyshev, Minkowski, and cosine distance, as well as specialized metrics such as Hamming, Canberra, Bray-Curtis, and Mahalanobis. These metrics are integrated as drop-in replacements for Euclidean distance in the harmonic loss formulation. We evaluate the resulting non-Euclidean harmonic losses on two heterogeneous task families: image classification (MLP, ResNet, PVT) and language modeling with transformer-based LLMs (GPT- 2, BERT, and others). This diversity enables us to assess whether certain distance-based losses consistently outperform cross-entropy and Euclidean harmonic loss on metrics of effectiveness, efficiency, and explainability. Specifically, we pursue the following research questions: RQ1 (Model Performance): Do non-Euclidean harmonic loss functions offer higher accuracy or faster convergence compared to cross-entropy and Euclidean harmonic loss? RQ2 (Interpretability): Do models trained with non-Euclidean harmonic losses exhibit more interpretable representations than those trained with cross-entropy? RQ3 (Efficiency & Sustainability): If a custom non-Euclidean har- monic loss outperforms cross-entropy on a given downstream task, does it do so without incurring a higher computational cost? We track training time, resource utilization, and energy consumption to assess the Green AI perspective [51]. By addressing these questions, our aim is to explore a three-way trade-off between accuracy, interpretability, and sustainability in the training process of deep learning models. Previous work has typically optimized one or two of these aspects in isolation: for instance, improving accuracy at the cost of enormous compute, known as “Red AI” [51], or simplifying models for interpretability while losing accuracy. In contrast, we seek solutions that improve predictive performance while also reducing energy consumption and yielding more transparent models. Contributions. This paper introduces distance-tailored har- monic losses and provides an extensive empirical and analytical evaluation of their merits. To our knowledge, this is the first work to: i) extend the harmonic loss beyond Euclidean distance, and benchmark a wide spectrum of metrics on both vision and NLP tasks; i) assess the carbon footprint and resource usage of dif- ferent loss functions in a controlled setting; and i) investigate interpretability outcomes of distance-based losses. We also offer preliminary theoretical insights into how different distance metrics influence the geometry of the learned model (e.g., relating퐿 1 losses to median-based class centers vs.퐿 2 to mean-based centers), which could inform the selection of an optimal loss for a given objective. 2 Harmonic loss Harmonic loss replaces the conventional inner-product logits and softmax normalization with a distance-based formulation: model predictions are derived from the distances between the sample’s representation and class prototype vectors (the learned weight vec- tors for each class). Intuitively, this means a model is trained to bring each sample closer to its correct class center in the feature space, rather than simply increasing a classification score. From Baek et al. [3], given the training set퐷=(푥 푖 ,푦 푖 ) 푛 푖=1 with 푦 푖 ∈ 1, ...,퐾and class prototypes푊= w 푘 퐾 푘=1 with w 푘 ∈ R 푑 , the harmonic logit is the퐿 2 distance between the prototype w 푘 and the instance representation h∈ R 푑 , i.e.,푑 푘 =∥h−w 푘 ∥ 2 . Then, the harmonic probabilities are given by: 푝 푊 (푦 푘 | 푥)= 푑 −푛 푘 Í 퐾 푗=1 푑 −푛 푗 ,(1) where the harmonic exponent푛is a hyperparameter that controls the heavy-tailedness of the probability distribution. The Harmonic loss is then given by: L(푊)=− ∑︁ (푥,푦)∈퐷 log푝 푊 (푦 | 푥).(2) This approach endows the learning process with two key properties: i) ratio invariance: for 1-homogeneous distances (e.g., Euclidean, Manhattan), the class probabilities푝 푊 (푦 | 푥)are invariant to uni- form rescaling of both representations and prototypes, since factors of푐 −푛 cancel in the ratio (Theorem A.2); this contrasts with cross- entropy, where scaling logits changes the softmax distribution; 1 1 Individual scale invariance—invariance to scaling h alone—holds only for angular distances such as cosine; for Euclidean distance,∥훼h−w∥ 2 ≠ ∥h−w∥ 2 in general. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY and i) finite convergence point: optimization seeks a distance of zero to the correct prototype. As a result, each class weight converges to an anchor point that can be interpreted as the center of that class’s feature distribution. Empirically, Baek et al. [3]demonstrated that harmonic loss can close the train–test gap faster and yield more interpretable rep- resentations than cross-entropy. For example, the learned weight vectors in a harmonic-loss model directly reflect class prototypes, making them semantically meaningful. Models trained with har- monic loss were also shown to require less data to generalize and to mitigate grokking, all while achieving competitive or better ac- curacy on both vision and language benchmarks. These findings suggest that distance-based loss functions are a promising direction for improving performance and transparency in deep learning. 3 Non-Euclidean Harmonic Losses Our framework introduces non-Euclidean harmonic losses as a gen- eralization of the harmonic loss, and as a replacement for conven- tional cross-entropy training. The idea is that, in Equation (1), the Euclidean distance푑 푘 =∥h−w 푘 ∥ 2 is replaced by a non-Euclidean distance. 3.1 Class Prototypes, Distances, and non-Euclidean Harmonic Loss functions Each class푘 ∈ 1, . . .,퐾is associated with a prototype vector w∈ R 푑 . Given an instance representation h, we compute its distance to all prototypes via a chosen metric 푑(·,·). Prototypes are learned parameters, just like the weight matrix in linear classification. Thus, prototype learning is no more compu- tationally expensive than learning a final linear layer. We extend the Euclidean formulation of harmonic loss [3] with the following distances: Euclidean. Baseline Euclidean distance between feature and prototype: 푑 Euclidean (h, w)=∥h− w∥ 2 . Manhattan (퐿 1 ). The퐿 1 norm emphasizes absolute differences, providing robustness to outliers since large deviations contribute linearly rather than quadratically to the total distance [20, 34, 62]. 푑 Manhattan (h, w)=∥h− w∥ 1 . Furthermore, computing퐿 1 distance requires only additions and subtractions without squaring or square-root operations, making it computationally efficient. Recent work on AdderNets demonstrates that replacing multiplication-based convolutions with퐿 1 -based operations can reduce energy consumption while maintaining com- petitive accuracy [9]. Chebyshev (퐿 ∞ ). 푑 Chebyshev (h, w)=∥h− w∥ ∞ . Captures the maximum coordinate deviation, offering a highly interpretable measure of the most discriminative feature dimension. Its simplicity makes it computationally efficient. Minkowski (퐿 푝 ). Generalizes both퐿 1 and퐿 2 , with a tunable exponent푝that controls the trade-off between robustness and sensitivity [1, 25]: 푑 Minkowski (h, w;푝)=∥h− w∥ 푝 . Lower values of푝(closer to 1) down-weight large individual coor- dinate differences, yielding behavior more robust to outliers, while higher values (toward 2 or above) increase sensitivity to larger de- viations. This flexibility allows the metric to be tailored to dataset characteristics, treating푝as a hyperparameter to optimize accuracy. Cosine. Ignores vector magnitudes and instead measures angu- lar similarity, making it particularly effective for high-dimensional embeddings [13, 33, 48, 53, 57]. 푑 cosine (h, w)= 1− h ⊤ w ∥h∥ 2 ∥w∥ 2 . Because the output is bounded, cosine similarity decreases acti- vation variance compared to unbounded dot products, which can improve generalization [39]. The computational overhead is mini- mal, only requiring an additional normalization step beyond the standard dot product. Hamming. Counts mismatches directly, providing a highly in- terpretable signal: each unit of distance corresponds to exactly one differing feature dimension. 푑 Hamming (h, w)= 1 푑 푑 ∑︁ 푖=1 1 ℎ 푖 ≠푤 푖 . Although inherently discrete, Hamming distance can be integrated into neural network training via continuous relaxations such as the Gumbel-Softmax trick [27,40], which provide differentiable approximations to binary sampling. Binary embeddings offer sub- stantial efficiency gains – comparing binary codes via Hamming distance can be significantly faster and more memory-efficient than comparing full-precision vectors. Canberra. Normalizes each coordinate difference by the sum of the magnitudes, yielding heightened sensitivity when both values are near zero [35]. 푑 Canberra (h, w)= 푑 ∑︁ 푖=1 |ℎ 푖 −푤 푖 | |ℎ 푖 |+|푤 푖 |+ 휀 . Since each term is bounded in[0,1], the metric emphasizes propor- tional rather than absolute differences. This property can benefit fine-grained recognition tasks in which small variations in low- magnitude features carry important discriminative information. Bray-Curtis. A normalized variant of the퐿 1 distance that mea- sures proportional differences across all features [8, 18, 52]. 푑 Bray-Curtis (h, w)= Í 푑 푖=1 |ℎ 푖 −푤 푖 | Í 푑 푖=1 (|ℎ 푖 |+|푤 푖 |)+ 휀 . Widely used in ecology for comparing species abundance profiles, Bray–Curtis produces values in[0,1]that are easy to interpret: 0 indicates identical composition, 1 indicates no overlap. Because it does not require estimating a covariance matrix, it offers a com- putationally lighter alternative to Mahalanobis distance while still accounting for relative differences. Mahalanobis. Incorporates feature correlations by weighting distances according to the inverse covariance matrixΣ −1 , effectively measuring distances in a whitened feature space [21, 36, 42, 43]. 푑 Mahalanobis (h, w;Σ)= √︁ (h− w) ⊤ Σ −1 (h− w). This can improve discrimination when features have different scales or are correlated; for example, Lee et al. [36]achieve state-of-the-art out-of-distribution detection in deep networks using Mahalanobis Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. distance on CNN features. The computational trade-off is that esti- mating and inverting the covariance matrix incurs overhead, par- ticularly in high dimensions. However, the resulting distance is scale-invariant and unitless, facilitating the interpretability of class boundaries as ellipsoids in feature space. In our framework, we generalize the harmonic loss by replacing the Euclidean distance in Equation (1) with one of the distances defined above. This substitution is applied only at the final classifica- tion layer, replacing the standard softmax cross-entropy head. The feature extraction backbone remains unchanged; we do not intro- duce additional normalization layers or architectural modifications beyond the classification head. Compared to standard cross-entropy, distance-based harmonic losses offer several potential benefits: the scale-invariant formula- tion avoids unbounded logit growth, the geometric interpretation provides transparency (class decisions correspond to proximity to prototype vectors), and certain distance choices (e.g.,퐿 1 , cosine) can reduce computational cost. A formal treatment of our distance–based probabilistic layer is provided in Section A. There, we generalize the harmonic-loss analysis to broad distance families and prove: i) scale invariance and the existence of finite minimizers under 1-homogeneous dis- tances (Theorem A.2), and i) a margin-style PAC–Bayes generaliza- tion bound whose finiteness follows from the finite–norm solution (Theorem A.4). These results clarify when geometry choices are well-posed and why the resulting classifiers admit standard gener- alization guarantees. 4 Experiments 4.1 Training and Evaluation Datasets. We evaluate on five vision benchmarks (MNIST, CIFAR- 10, CIFAR-100, MarathiSignLanguage, TinyImageNet) and one lan- guage corpus (OpenWebText). Vision. We consider a simple MLP with two hidden layers (512, 256, ReLU), a simple CNN (two 3×3 conv blocks with[32,64]chan- nels and 2×2 max-pooling, then a 128-dim FC), ResNet-50 (standard [3,4,6,3]bottleneck stages; for small inputs we remove the initial max-pool and use a 3×3 stride-1 stem), and PVTv2-B0 (four hierar- chical stages with overlapping patch embeddings; output pooled to a 256-dim vector). Language. We study three Transformer families: GPT-style (decoder-only causal LM), BERT (encoder-only masked LM with 15% masking), and Qwen2-style decoders. Optimization. Unless noted, models are trained from scratch with Adam/AdamW-style optimizers (weight decay,(훽 1 ,훽 2 )as con- figured), cosine learning-rate decay with linear warmup, mixed precision (FP16/BF16 when available), and gradient accumulation. We apply gradient clipping, dataset-specific schedulers, and early stopping with dataset-specific patience and a minimum improve- ment threshold (Δ min ). For fairness, all harmonic layers and the baseline share the same backbone, batch size, scheduler, and data or- der. Additional details about optimization are reported in Appendix C.1. Model Performance. For vision tasks, we report average Accu- racy and F1. For language task, we report the following metrics: Perplexity (Train / Val). Given a sequence of targets푦 푡 푇 푡=1 and model probabilities푝 휃 (푦 푡 | context), where휃denotes the model parameters, the average negative log–likelihood is: L NLL =− 1 푇 푇 ∑︁ 푡=1 log푝 휃 푦 푡 | context , and the corresponding perplexity isPPL= exp L NLL .Lower per- plexity indicates better next-token prediction. 2 Gradient Stability (GS). To quantify the smoothness of opti- mization, we measure the variance of the퐿 2 -norm of the gradient across consecutive training steps. LetL 푡 denote the training loss at step 푡 ; then: GS= 1− Var ∥∇ 휃 L 푡 ∥ 2 Var ∥∇ 휃 L 푡 ∥ 2 CE , whereVar(·)is computed over a fixed evaluation window (e.g., 500 steps) and the denominator corresponds to the variability under cross–entropy (CE). This metric is motivated by gradient variance analyses [16,37], which demonstrate that gradient variance dy- namics correlate with training stability and convergence behavior. Thus,GS=0 indicates equal smoothness as CE,GS>0 indicates reduced gradient variance (smoother training), andGS<0 reflects more unstable gradient dynamics (higher variance than CE). This metric is anchored in standard variance-of-gradient analyses used in optimizing large-scale LLMs. Effective Rank. This metric is inspired by analyses of dimen- sional collapse [4,30], that track the covariance spectrum of learned representations to detect embedding collapse and maintain repre- sentation quality in neural networks. LetΣ ℎ = Cov(h 푡 ) with eigen- values휆 푘 푑 푘=1 . Define the normalized eigenvalues푝 푘 = 휆 푘 / Í 푗 휆 푗 . The effective rank is: ER(Σ ℎ )= exp − 푑 ∑︁ 푘=1 푝 푘 ln푝 푘 ! . This is the exponential of the Shannon entropy of the eigenvalue distribution [19] [49]. Metric values are in the range[1,푑], where a value of 1 means total collapse (one dimension dominates), and value of푑means perfectly uniform spread. Higher values indicate that the learned weight matrices retain higher intrinsic dimension- ality, suggesting the distance metric discourages low-rank collapse in the parameter space and is consistent with more distributed use of the embedding capacity. Interpretability. We probe whether learned prototypes/weights act as class centers and whether features become more structured by computing PCA explained variance on the penultimate features: i) PC2 EV (variance explained by the top two PCs), and i) PCA@90% (dimensions required to reach 90% variance). Lower PCA@90% and higher PC2 EV indicate more concentrated, low-dimensional structure. For language, we report PCA5:Δvariance explained by the top 5 principal components of final hidden states (causal LM: 2 For visualization in the radar plots, we invert perplexity (and all metrics where lower values indicate better performance) and then normalize to the range[0,10]relative to the harmonic Euclidean baseline. The effect is that the greater the coverage on the plot, the better the relative performance compared to Euclidean. Importantly, the absolute numeric values on the radial axis do not have a direct “good/bad” interpretation in perplexity space; they are meaningful only as normalized, experiment–specific comparisons against the Euclidean harmonic. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY last token; MLM: masked positions); higher values implies more concentrated, low-dimensional structure. Sustainability. We perform training with CodeCarbon to log duration, energy, and CO 2 emissions. Emissions are reported per run and differentially vs. the cross-entropy baseline (grams CO 2 ; nega- tive means greener-than-baseline). We aggregate by (dataset, back- bone, distance) and also report cumulative figures across seeds. For language, we also report Speed (−Δ time_to_90_percent): higher values denotes fewer steps to reach 90% of final performance. To isolate the effect of the loss geometry, we only swap the clas- sifier head (linear vs. distance-based) while keeping: backbone weights initialization scheme, data preprocessing/augmentation, optimizer and LR schedule, batch size, number of epochs, early- stopping rule, and randomness controls (seeds). For ResNet-50/PVT we use identical augmentation; for LLMs we use the same context length퐿, optimizer, and schedule across heads. We run multiple seeds and report means. Exact architectures and preprocessing pipelines are detailed in Appendix C.1. Full hyperparameter grids (including head-specific parametersΘ, e.g.,푝for Minkowski or covariance settings for Mahalanobis) are provided in Appendix D. This unified protocol lets us systematically test how replacing the Euclidean harmonic head with alternative distances impacts: i) fi- nal model performance, i) representation structure and prototype semantics, and i) measured energy and carbon footprint. 4.2 Vision: Radar Plots Figure 1 summarizes the behavior of distance-based harmonic losses across all vision settings, including a high–resolution sign language dataset (Marathi Sign) and TinyImageNet in addition to CIFAR- 100. Additional results on MNIST and CIFAR10 are provided in Appendix F. Together, these radar plots expose how the choice of distance in the harmonic loss shapes performance, representation geometry, and sustainability. RQ1: Model Performance (F1, Accuracy). Across datasets and backbones, cosine–based harmonic losses remain the most reli- able all–round performers. On CIFAR-100, cosine (stable/unstable) typically attains the highest or near–highest accuracy and F1 on CNN and ResNet50, and is consistently among the top curves on PVT. On the more realistic, higher–resolution Marathi Sign and TinyImageNet, the same pattern largely persists: cosine (stable) and Bray–Curtis (normalized) frequently improve or match Euclidean and cross–entropy on CNN, ResNet50, and PVT, while also appear- ing in the top group on MLP. TinyImageNet is the most challenging setting: cross–entropy remains a strong baseline, but cosine heads still achieve competitive accuracy on ResNet50 and PVT, demon- strating that the benefits of distance–tailored heads extend beyond small benchmarks. Other non–Euclidean distances (Bray-Curtis variants, Manhattan, Minkowski) can occasionally match or exceed cosine in specific architecture–dataset combinations. RQ2: Interpretability (PC2 EV, PCA 90%). Non–Euclidean distances reshape the final embedding geometry in a systematic, dataset–agnostic way. Across Marathi Sign, TinyImageNet, and CIFAR-100, Bray–Curtis (standard/normalized) and Chebyshev (stan- dard) repeatedly yield the largest PC2 explained variance and the lowest dimensionality required to reach 90% EV, indicating com- pact, prototype–aligned feature spaces with sharper class clusters than those produced by Euclidean harmonic loss or cross–entropy. Cosine harmonic loss generally provides substantial EV gains over Euclidean, while retaining top accuracy, offering a favorable ac- curacy–interpretability balance on both convolutional backbones and PVT. Mahalanobis variants often achieve extreme variance concentration (very high EV) and pronounced cluster separation, but this representation clarity sometimes co–occurs with less sta- ble optimization on the hardest datasets. Overall, the same geo- metric trends observed on earlier small benchmarks persist when moving to higher resolutions and deeper models: non–Euclidean harmonic losses, especially Bray-Curtis and Chebyshev, produce more structured, low–dimensional embeddings than Euclidean or cross–entropy heads. RQ3: Sustainability (Duration/Epoch/GFLOPs, Emissions). Distance choice also affects efficiency, but in a controlled way. Across all datasets, cosine harmonic loss is typically neutral to favor- able in emissions relative to Euclidean and cross–entropy: normal- ized Duration/Epoch/GFLOPs and gCO 2 eq remain comparable, and in several ResNet50 and PVT runs, cosine achieves slightly lower emissions due to a faster approach to high accuracy. Bray-Curtis losses incur modest overhead while delivering strong interpretabil- ity gains, whereas Mahalanobis distances are the most costly, reflect- ing their covariance–related computation and sometimes slower convergence on complex data. Even on high–resolution Marathi Sign and TinyImageNet, the harmonic head accounts for only a small fraction of total FLOPs; thus, differences in Duration/Epoch are smaller than differences in accuracy or EV, yet cumulative emis- sions still meaningfully separate distances. Across all vision workloads, three regularities emerge: i) cosine harmonic loss is the best all–around choice, offering consistently strong accuracy/F1, clear geometric structure relative to Euclidean, and neutral–to–lower emissions from MLPs up to ResNet50/PVT on Marathi Sign and TinyImageNet; i) Bray–Curtis and Chebyshev are the most interpretability–forward options, reliably increasing vari- ance concentration and reducing PCA 90% dimensionality, with ac- curacy effects that are positive but more configuration–dependent; i) Mahalanobis emphasizes representation clarity at a higher sus- tainability cost. Taken together, the radar plots show that the geom- etry of the harmonic loss, especially non–Euclidean choices, has a consistent, architecturally robust effect on performance, structure, and sustainability across both small and large vision benchmarks. 4.3 Language: Radar Plots Figure 2 summarizes the effect of distance-tailored harmonic losses on BERT, GPT, and Qwen-style decoders across the three perspec- tives. Scores are normalized so that larger areas indicate more desirable behavior. RQ1: Model Performance (Perplexity, Health, Stability). Across architectures, cosine–based harmonic losses remain the most reliable choices on performance–oriented axes. For BERT, cosine heads achieve low train and validation perplexity while improv- ing Gradient Stability and preserving high effective rank relative to both cross–entropy and Euclidean harmonic loss. For GPT, co- sine and Minkowski (푝=2) again provide steady training dynamics with competitive perplexity, whereas the cross–entropy baseline ex- hibits higher variability and weaker stability. On Qwen, none of the Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. approaches provides a satisfactory trade-off. Euclidean harmonic loss offers the strongest effective rank and gradient stability, while Minkowski provides the best perplexity. In summary, in two out of three cases, replacing the linear classifier with distance–based harmonic heads reduces gradient volatility and collapse symptoms while maintaining or improving perplexity. RQ2: Interpretability (PCA Structure). Non–Euclidean dis- tances consistently concentrate token representations into more structured latent spaces. In BERT and GPT, cosine and Minkowski enlarge the PCA Structure wedge (higher variance explained by a small number of components), indicating more organized, proto- type–aligned embeddings than those produced by cross–entropy or Euclidean harmonic loss. Qwen shows a similar pattern: dis- tance–based heads achieve clearer low–dimensional structure even when Euclidean is slightly stronger on stability. As in the vision experiments, geometries that emphasize angles (cosine) or퐿 푝 struc- ture (Minkowski) tend to yield hidden states that are easier to summarize with a few principal components. RQ3: Sustainability (Emissions). Results confirm that dis- tance–based harmonic heads introduce little computational over- head and can be greener than cross–entropy in practice. In all three models, the cross–entropy baseline occupies the largest emissions wedge, while cosine and Minkowski are neutral–to–favorable, of- ten matching or improving on Euclidean harmonic loss. Extremely sharp cosine temperatures may reduce emissions slightly but at the cost of stability and perplexity; moderate settings avoid this trade–off. Because the classifier head is lightweight compared to the Transformer backbone, these sustainability gains primarily arise from smoother optimization and faster convergence rather than per–step FLOPs. In summary, cosine–based harmonic losses are the most robust all–around choice for LLMs, jointly improving perplexity, stability, and representation structure with neutral or reduced emissions. Minkowski (푝=2) provides a strong alternative when cosine hy- perparameters are poorly tuned, while Euclidean remains a solid reference but is rarely dominant over non–Euclidean geometries. Additional results showcasing optimization dynamics for all mod- els, including a larger GPT2 (2B) model, are reported in Appendix H. 5 Related Work Loss functions for classification. The majority of classification models are trained with cross-entropy loss due to its empirical ef- fectiveness and probabilistic interpretation. However, it only cares about separating classes, not about how the representations are sep- arated, often yielding features that are separable but not necessarily interpretable. Over the years, alternative loss functions have been proposed to address these limitations. Metric learning losses, such as contrastive and triplet loss, train models to preserve distances between examples, but require sampling strategies that add training complexity. Boudiaf et al. [7]propose a unifying mutual information framework connecting cross-entropy to standard pairwise losses, showing that cross-entropy implicitly bounds pairwise distance ob- jectives. These insights motivate a deeper theoretical understanding of distance-based training. Regularization-based approaches such as center loss [61] explicitly encourage compact intra-class clusters and large inter-class separation. These works foreshadow the idea that directly leveraging distances to class prototypes can improve representation quality. Angular margin losses, such as AMC-Loss in Choi et al. [11], introduce geometric constraints on angular separations to enhance interpretability via hyperspherical metrics. Orthogonal Projection Loss (OPL) introduced by Ranasinghe et al. [47] encourages inter-class orthogonality and intra-class cohesion without sampling overhead. Several studies have assessed how loss functions affect neural network performance. Miller et al. [41]in- troduce Class Anchor Clustering (CAC) loss that encourages tight class clusters centered on anchored prototypes, enhancing distance- based open-set classification performance. This approach aligns with the prototype-centered philosophy underlying harmonic loss. Cho et al. [10]analyzed how eight loss functions impact neural network accuracy and convergence speed, finding that additive- margin softmax loss resulted in the fastest convergence and highest performance on multiple datasets. Janocha and Czarnecki[28]as- sessed 12 loss functions for classification, finding that choice of loss function impacted learning speed and testing accuracy. Gonzalez and Miikkulainen[22]used genetic programming to develop Baikal loss, which not only led to networks achieving higher accuracy than networks trained with cross-entropy loss, but also faster train- ing and higher performance in low-data settings. These studies demonstrate a large focus on the impact of loss function on neural networks performance. Our work builds on the discussion of the importance of loss function choice by drilling deeper on harmonic loss, examining how distance metric choice impacts the effective- ness of neural networks. Our focus is not on comparing harmonic loss with other loss functions, which was done by Baek et al. [3], but rather to shed light on the performance of a generalized harmonic loss. Efficiency and Green AI. Green AI is an emerging initiative that calls for efficiency and energy usage to be treated as first- class evaluation criteria [51]. Many works on green AI focus on model compression [44,46], comparing multiple models [56] or fine- tuning strategies [59], or hyperparameter optimization for carbon emission reduction [58]. While prior works on new loss functions rarely report sustainability metrics, we incorporate carbon footprint analysis into our evaluation due to claims that models trained with harmonic loss are more data efficient and have less grokking [3]. Interpretability in neural networks. Neural networks are complex and not inherently interpretable, but substantial effort has been devoted to improving interpretability [64]. The push for inter- pretable by design models argues that transparency should be built into model architectures and losses rather than added post-hoc [50]. Harmonic loss aligns with this vision by structurally linking model weights to class prototypes. The study by [6] discusses how inter- nal model components reveal human-understandable circuits and features in LLMs. Techniques such as activation patching, sparse autoencoders, transcoders, and crosscoders enable structural inter- pretations of model behavior. Parallel to our interpretability focus, Wen et al. [60]introduced a framework combining interpretable models with deep networks for time-series tasks, preserving un- derstandable reasoning where possible; though not loss-centric, it reflects the growing emphasis on transparency in deep learning research. Some work has focused on using loss functions specif- ically to improve model interpretability. Liu et al. [38]combine Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY MarathiSign MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Bray−Curtis (Norm.) Canberra (Weighted) Chebyshev (Std.) Euclidean TinyImageNet MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Bray−Curtis (Norm.) Euclidean Hamming (Soft) Manhattan CIFAR100 MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Canberra (Std.) Cosine (Stable) Cosine (Unst.) Euclidean MarathiSign CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Cosine (Stable) Cosine (Unst.) Euclidean Mahalanobis (Chol.) TinyImageNet CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Canberra (Robust) Canberra (Std.) Euclidean Mahalanobis (Std.) CIFAR100 CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Canberra (Robust) Euclidean Mahalanobis (Chol.) Manhattan MarathiSign ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Chebyshev (Std.) Cosine (Stable) Cosine (Unst.) Euclidean TinyImageNet ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Cosine (Stable) Cosine (Unst.) Euclidean Manhattan Minkowski (p=1.5) CIFAR100 ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Cosine (Stable) Cosine (Unst.) Euclidean Manhattan MarathiSign PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Chebyshev (Std.) Euclidean Hamming (Soft) Manhattan TinyImageNet PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Bray−Curtis (Norm.) Cosine (Stable) Cosine (Unst.) Euclidean CIFAR100 PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Std.) Canberra (Robust) Canberra (Std.) Chebyshev (Std.) Euclidean Figure 1: Vision: Radar plots: 1) Model Performance (F1, Accuracy); 2) Interpretability (PC2 EV, PCA 90%), and 3) Sustainability (Duration/Epoch/GFLOPs, Emissions). Plots feature Baseline (Cross-Entropy), Euclidean harmonic, and the four top-performing non-Euclidean harmonic losses. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. BERT−0.1B 0 2.5 5 7.5 10 Emissions PCA Structure Val. Perplexity Train Perplexity Gradient Stability Effective Rank Baseline (C.E.) Cosine (Temp. 0.1) Cosine (Simple) Minkowski (p=2.0) Cosine (Temp. 0.5) Cosine (Temp. 1.0) Euclidean GPT−0.1B 0 2.5 5 7.5 10 Emissions PCA Structure Val. Perplexity Train Perplexity Gradient Stability Effective Rank Baseline (C.E.) Minkowski (p=2.0) Cosine (Temp. 0.1) Cosine (Simple) Cosine (Temp. 1.0) Cosine (Temp. 0.5) Euclidean QWEN2−0.5B 0 2.5 5 7.5 10 Emissions PCA Structure Val. Perplexity Train Perplexity Gradient Stability Effective Rank Minkowski (p=2.0) Baseline (C.E.) Euclidean Figure 2: Language: Radar plots: 1) Model Performance (Perplexity, Effective Rank, Gradient Stability); 2) Interpretability (PCA5 EV), and 3) Sustainability (Emissions). Plots feature Baseline (CE), Euclidean harmonic, and the top-performing non-Euclidean harmonic losses. sparse coding constraints with cross-entropy to produce concise, interpretable word-level attributions. Dong et al. [14]introduced interpretative loss to improve the interpretability of learned features during video captioning tasks. Within classification tasks. Zhang et al. [63]designed a loss function to improve CNN filter inter- pretability. Methods such as the one proposed by Hagos et al. [23] augment standard losses with distance-based penalties that align model attributions with user-provided annotations, strengthening interpretability. Distance metrics in learning algorithms. Beyond supervised classification, the choice of distance measure is known to be crucial. Coil et al. [12]compared twelve distance metrics in anomaly detec- tion for concept drift; their results highlighted that performance depends heavily on the chosen metric and that efficient alternatives can sometimes match the performance of more costly distances. A variety of other works have shown the importance of distance metric choice. Amaya-Tejera et al. [2]used a kernel for SVMs that supported a variety of kernels, finding that the choice of distance metric impacted performance. Kalra et al. [32]and Hu et al. [25] both found that distance metric choice impacted performance of k- nearest neighbors algorithms on a variety of datasets. These result highlights the importance of systematically exploring metrics in different contexts. To our knowledge, our paper is the first to bring this perspective into loss functions. 6 Conclusion This work examined distance–based harmonic losses as drop–in re- placements for cross–entropy across image classification (MNIST, CIFAR-10, CIFAR-100, Marathi Sign Language, TinyImageNet) with four vision backbones (MLP, CNN, ResNet50, PVT) and LLM pre- training (GPT, BERT, Qwen, GPT-2B), leveraging a broad family of distances (cosine, Euclidean, Bray–Curtis, Mahalanobis, Minkowski, Chebyshev, Canberra, etc.) and comparing them against strong mod- ern baselines (Focal Loss, Label Smoothing, Center Loss, Confidence Penalty, ArcFace). What we learned: i) Geometry matters for optimization. Across vision and language tasks, Cosine consistently delivers smoother training dynamics, higher or competitive final perfor- mance, and reduced grokking–like behavior in toy modulo–addition experiments. Euclidean remains a solid reference; Bray–Curtis is often competitive but architecture–sensitive; Mahalanobis exhibits the largest variance—sometimes yielding very sharp, well–separated clusters, but with less stable plateaus in more difficult scenarios (larger datasets and model backbones). Loss–convergence curves for both vision and LLMs show that all investigated distances (includ- ing cosine and Mahalanobis) exhibit smooth optimization without problematic instabilities. i) Sustainability depends jointly on distance and archi- tecture. On vision tasks, several non–Euclidean harmonic losses are carbon–negative per step relative to cross–entropy for CN- N/ResNet50 (largest gains occur on deeper CNNs), mixed on MLP, and closer to neutral on PVT and TinyImageNet, where back- bone FLOPs dominate. For LLM pretraining, the classifier head is lightweight, so differences arise primarily via convergence: the cross–entropy baseline typically incurs the largest cumulative emis- sions, while cosine and Minkowski heads are neutral–to–favorable. Our FLOPs–normalized analysis and extended emissions study show that the best non–Euclidean harmonic losses lie on or near the sustainability–accuracy Pareto frontier. i) Interpretability can be quantified. PCA–based probes (variance concentration and PCA@90%) and geometric visualiza- tions of prototype neighborhoods provide reproducible evidence that distance–tailored heads yield more structured representations. Bray–Curtis and Chebyshev consistently increase variance con- centration and reduce intrinsic dimensionality, while Mahalanobis emphasizes representation clarity at a higher computational cost. These trends hold for image features and for token representations in LLMs (last–token and masked–token states) and are supported by statistical tests (Wilcoxon) and confidence intervals across seeds. Language. Cosine–based harmonic losses markedly improve gradient/learning stability, perplexity, and representation structure for GPT, BERT, Qwen, and GPT-2B, while keeping emissions on par with or below cross–entropy and Euclidean harmonic loss. Mahalanobis remains less attractive for large–scale pretraining due to covariance overheads and sensitivity to ill–conditioned statistics. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Vision. For accuracy–focused workloads across MNIST, CIFAR, Marathi Sign, and TinyImageNet, cosine (stable) is the preferred all–round choice; Bray–Curtis is a strong secondary option; Maha- lanobis should be used when its inductive bias (sharp, anisotropic clusters) is explicitly desired. For sustainability on CNN/ResNet50, several non–Euclidean distances reduce per–step CO 2 ; on PVT and LLMs, the lightest geometries (cosine/Euclidean) should be favored, or cross–entropy retained unless a distance–based head reduces steps to target enough to offset higher per–step cost. In summary, our framework including a plug–and–play har- monic distance layer, a catalogue of non-Euclidean distances, and a three–axis evaluation protocol (performance, interpretability, sus- tainability) can be effectively exploited in future work: practitioners can choose distances according to their priorities, and researchers can extend our study to new geometries, learning settings, and do- main–specific constraints. In this sense, distance–based harmonic losses provide a principled, empirically validated toolbox for re- thinking the geometry of classification layers in both vision and language models. References [1]Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. 2001. On the Surprising Behavior of Distance Metrics in High Dimensional Spaces. In ICDT (Lecture Notes in Computer Science, Vol. 1973). Springer, 420–434. [2]Nazhir Amaya-Tejera, Margarita Gamarra, Jorge I Vélez, and Eduardo Zurek. 2024. A distance-based kernel for classification via Support Vector Machines. Frontiers in Artificial Intelligence 7 (2024), 1287875. [3]David D. Baek, Ziming Liu, Riya Tyagi, and Max Tegmark. 2025. Harmonic Loss Trains Interpretable AI Models. arXiv preprint arXiv:2502.01628 (2025). doi:10.48550/arXiv.2502.01628 [4]Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: Variance-Invariance- Covariance Regularization for Self-Supervised Learning. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=xm6YD62D1Ub [5]Magnus Bengtsson. 2025. Compressing Large Language Models with PCA With- out Performance Loss. arXiv:2508.04307 https://arxiv.org/abs/2508.04307 [6] Leonard Bereska and Stratis Gavves. 2024. Mechanistic Interpretability for AI Safety - A Review. Trans. Mach. Learn. Res. 2024 (2024). [7]Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, and Ismail Ben Ayed. 2020. A Unifying Mutual Information View of Metric Learning: Cross-Entropy vs. Pairwise Losses. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 548–564. doi:10.1007/978-3- 030-58539-6_33 [8]Anne Chao, Robin L Chazdon, Robert K Colwell, and Tsung-Jen Shen. 2010. An additive decomposition formula for the Bray–Curtis dissimilarity and their ecological meaning. Ecological Modelling 221, 9 (2010), 1275–1283. [9] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. 2020. AdderNet: Do We Really Need Multiplications in Deep Learning?. In CVPR. Computer Vision Foundation / IEEE, 1465–1474. [10] Kwantae Cho, Jong-hyuk Roh, Youngsam Kim, and Sangrae Cho. 2019. A perfor- mance comparison of loss functions. In 2019 International Conference on Informa- tion and Communication Technology Convergence (ICTC). IEEE, 1146–1151. [11]Hongjun Choi, Anirudh Som, and Pavan K. Turaga. 2020. AMC-Loss: Angular Margin Contrastive Loss for Improved Explainability in Image Classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020. Computer Vision Foundation / IEEE, 3659–3666. doi:10.1109/CVPRW50498.2020.00427 [12]Collin Coil, Kamil Faber, Bartlomiej Sniezynski, and Roberto Corizzo. 2025. Distance-based change point detection for novelty detection in concept-agnostic continual anomaly detection. Journal of Intelligent Information Systems (2025), 1–39. [13]Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. ArcFace: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4690–4699. [14] Yinpeng Dong, Hang Su, Jun Zhu, and Bo Zhang. 2017. Improving interpretability of deep neural networks with semantic information. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4306–4314. [15] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy Models of Superposition. Trans- former Circuits Thread (2022). https://transformer-circuits.pub/2022/toy_model/ index.html [16] Fartash Faghri, David Duvenaud, David J. Fleet, and Jimmy Ba. 2020.A Study of Gradient Variance in Deep Learning. CoRR abs/2007.04532 (2020). arXiv:2007.04532 https://arxiv.org/abs/2007.04532 [17]FAR AI. 2023. Uncovering Latent Human Wellbeing in LLM Embeddings. https: //far.ai/news/uncovering-latent-human-wellbeing-in-llm-embeddings. Shows first principal component of GPT-3 embeddings correlates with ethics/well-being labels. [18]Carolina Fuschi, Davide Delfino, Thomas Klammsteiner, Franco Biasioli, Gino Fernandez, Sabina Angeli, and Raffaella Causin. 2025. Microbiome data: tell me which metrics and I will tell you which communities. Scientific Reports 15, 1 (2025), 1–14. [19]Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. 2023. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. In ICML (Proceedings of Machine Learning Re- search, Vol. 202). PMLR, 10929–10974. [20]Arie Giloni and Manfred Padberg. 2003. The finite sample breakdown point of ℓ 1 -regression. In SIAM Journal on Optimization, Vol. 14. SIAM, 608–620. [21]María José Gómez-Silva, Arturo de la Escalera, and José María Armingol. 2021. Back-propagation of the Mahalanobis distance through a deep triplet learning model for person re-identification. Integrated Computer-Aided Engineering 28, 3 (2021), 277–288. [22] Santiago Gonzalez and Risto Miikkulainen. 2020. Improved training speed, accuracy, and data utilization through loss function optimization. In 2020 IEEE congress on evolutionary computation (CEC). IEEE, 1–8. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. [23]Misgina Tsighe Hagos, Niamh Belton, Kathleen M. Curran, and Brian Mac Namee. 2023. Distance-Aware Explanation Based Learning for Interpretable Neural Networks. In 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 279–286. [24]Yihan He, Yuan Cao, Hong-Yu Chen, Dennis Wu, Jianqing Fan, and Han Liu. 2024. Can Transformers Perform PCA?https://openreview.net/forum?id= mjDNVksC5G ICLR 2025 Conference Withdrawn Submission. [25]Li-Yu Hu, Min-Wei Huang, Shih-Wen Ke, and Chih-Fong Tsai. 2016. The dis- tance function effect on k-nearest neighbor classification for medical datasets. SpringerPlus 5, 1 (2016), 1304. [26]Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse Autoencoders Find Highly Interpretable Features in Lan- guage Models. In International Conference on Learning Representations (ICLR), Poster. https://openreview.net/forum?id=F76bwRSLeK [27]Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. In ICLR (Poster). OpenReview.net. [28]Katarzyna Janocha and Wojciech Marian Czarnecki. 2017. On loss functions for deep neural networks in classification. arXiv preprint arXiv:1702.05659 (2017). [29]Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2025. Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?. In Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics, Abu Dhabi, UAE, 558–573. https://aclanthology.org/ 2025.coling-main.37/ [30] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. 2022. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. In The Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=YevsQ05DEN7 [31] Ole Jorgensen. 2023. Understanding and Controlling the Activations of Language Models. Ph. D. Dissertation. Imperial College London. https://ojorgensen.github. io/assets/pdfs/Imperial_Dissertation.pdf [32]Vandana Kalra, Indu Kashyap, and Harmeet Kaur. 2022. Effect of distance mea- sures on K-nearest neighbour classifier. In 2022 Second International Conference on Computer Science, Engineering and Applications (ICCSEA). IEEE, 1–7. [33]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769–6781. [34]S. L. Keeling and K. Kunisch. 2016. Robustℓ 1 approaches to computing the geomet- ric median and principal and independent components. Journal of Mathematical Imaging and Vision 56, 2 (2016), 286–300. [35]Godfrey N. Lance and William T. Williams. 1967. A General Theory of Clas- sificatory Sorting Strategies: 1. Hierarchical Systems. Comput. J. 9, 4 (1967), 373–380. [36] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems 31 (2018). [37] Yunwen Lei and Yiming Ying. 2020. Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 5809–5819. http://proceedings.mlr.press/v119/lei20c.html [38] Junhong Liu, Yijie Lin, Liang Jiang, Jia Liu, Zujie Wen, and Xi Peng. 2022. Improve Interpretability of Neural Networks via Sparse Contrastive Coding. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 460–470. doi:10.18653/ V1/2022.FINDINGS-EMNLP.32 [39]Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, and Qiang Yang. 2018. Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks. In ICANN (1) (Lecture Notes in Computer Science, Vol. 11139). Springer, 382–391. [40]Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The Concrete Distri- bution: A Continuous Relaxation of Discrete Random Variables. In ICLR (Poster). OpenReview.net. [41]David Miller, Garrett Stewart, and Fernando de la Torre. 2021. CAC: Class Anchor Clustering for Open-Set Recognition. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34. 16326–16339. [42] Ibrahim Omara, Xue-juan Wu, Huan Zhang, Yingying Du, and Wangmeng Zuo. 2021. A novel approach for ear recognition: learning Mahalanobis distance features from deep CNNs. Machine Vision and Applications 32, 2 (2021), 1–17. [43] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. 2018. Max-Mahalanobis linear discriminant analysis networks. In International Conference on Machine Learning. PMLR, 4016–4025. [44]Eileen Paula, Jayesh Soni, Himanshu Upadhyay, and Leonel Lagos. 2025. Com- parative analysis of model compression techniques for achieving carbon efficient AI. Scientific Reports 15, 1 (2025), 23461. [45]Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. CoRR abs/2201.02177 (2022). [46] Kazi Rafat, Sadia Islam, Abdullah Al Mahfug, Md Ismail Hossain, Fuad Rahman, Sifat Momen, Shafin Rahman, and Nabeel Mohammed. 2023. Mitigating carbon footprint for knowledge distillation based deep learning model compression. Plos one 18, 5 (2023), e0285668. [47]Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman H. Khan, and Fahad Shahbaz Khan. 2021. Orthogonal Projection Loss. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 12313–12323. doi:10.1109/ICCV48922.2021.01211 [48]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3982–3992. [49]Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference. IEEE, 606–610. [50]Cynthia Rudin. 2019. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence 1, 5 (2019), 206–215. doi:10.1038/s42256-019-0048-x [51]Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green AI. Commun. ACM 63, 12 (2020), 54–63. doi:10.1145/3381831 [52]Ling Song, Peter Langfelder, and Steve Horvath. 2020. Systematic comparisons for composition profiles, taxonomic levels, and machine learning methods for microbiome-based disease prediction. Frontiers in Molecular Biosciences 7 (2020), 618573. [53]Yantao Sun, Yuxin Chen, Xiaogang Wang, and Xiaoou Tang. 2016. Learning discriminative CNN features and similarity metrics for image retrieval. In 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, 1013–1018. [54] Arthur Templeton et al.2023.Sparse Autoencoders Find Highly Inter- pretable Directions in Language Models. https://w.alignmentforum.org/ posts/Qryk6FqjtZk9FHHJR/sparse-autoencoders-find-highly-interpretable- directions-in. [55]Alex Turntrout. 2023. Steering GPT-2-XL by Adding an Activation Vector. https: //turntrout.com/gpt2-steering-vectors. [56]Anil Verma, Sumit Kumar Singh, Rupesh Kumar Sah, Rajiv Misra, and TN Singh. 2024. Performance Comparison of Deep Learning Models for CO2 Prediction: Analyzing Carbon Footprint with Advanced Trackers. In 2024 IEEE International Conference on Big Data (BigData). IEEE, 4429–4437. [57]Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018. CosFace: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5265–5274. [58] Irene Wang, Newsha Ardalani, Mostafa Elhoushi, Daniel Jiang, Samuel Hsia, Ekin Sumbul, Divya Mahajan, Carole-Jean Wu, and Bilge Acun. 2025. CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization. arXiv preprint arXiv:2505.01386 (2025). doi:10.48550/arXiv.2505.01386 Journal reference: NeurIPS 2025. [59] Xiaorong Wang, Clara Na, Emma Strubell, Sorelle Friedler, and Sasha Luccioni. 2023. Energy and carbon considerations of fine-tuning BERT. arXiv preprint arXiv:2311.10267 (2023). [60]Yunshi Wen, Tengfei Ma, Ronny Luss, Debarun Bhattacharjya, Achille Fokoue, and Anak Agung Julius. 2025. Shedding Light on Time Series Classification using Interpretability Gated Networks. In ICLR. OpenReview.net. [61]Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A Discriminative Feature Learning Approach for Deep Face Recognition. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 499–515. doi:10.1007/ 978-3-319-46478-7_31 [62]Jinfeng Ye, Tao Li, Tao Xiong, and Ravi Janardan. 2012. A pure퐿 1 -norm principal component analysis. Computational Statistics & Data Analysis 56, 12 (2012), 4474–4486. [63]Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018. Interpretable convo- lutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8827–8836. [64]Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. 2021. A survey on neural network interpretability. IEEE transactions on emerging topics in computational intelligence 5, 5 (2021), 726–742. Appendix A Theoretical Properties of Distance-based Probabilistic Layers Setup. Let(푥 푖 ,푦 푖 ) 푛 푖=1 be the training set with푦 푖 ∈ 1, ...,퐾. Each class has a prototype푤 푘 ∈R 푑 and a nonnegative distance푑(푥,푤)≥0. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Given a decreasing link 휅 : R + →R + we define 푝 푘 (푥 푖 )= 휅 푑(푥 푖 ,푤 푘 ) Í 퐾 푗=1 휅 푑(푥,푤 푗 ) , L(푤 푘 )= − 푛 ∑︁ 푖=1 log푝 푦 푖 (푥 푖 ). harmonic휅(푟)= 푟 −휔 with휔>0; while distances include Eu- clidean/Mahalanobis,퐿 푝 , Bregman divergences, and cosine/angle on the sphere. A.1 Scale invariance and finite minimizers We begin by generalizing the finite-minimizer result of the har- monic loss (cf. Thm. 1, Sec. G in Baek et al. [3]). Definition A.1 (Metric separability and homogeneity). A dataset is metric-separable if for each푖there exists푤 푘 s.t.푑(푥 푖 ,푤 푦 푖 )< min 푗≠푦 푖 푑(푥 푖 ,푤 푗 ). A distance푑is 1-homogeneous if푑(푐푥,푐푤)= |푐|푑(푥,푤) for all 푐> 0. Theorem A.2 (Finite minimizer and scale invariance for harmonic link). Assume푑is 1-homogeneous and the training set is metric-separable. For휅(푟)= 푟 −휔 , the empirical lossLis invariant to the joint rescaling(푥,푤) ↦→ (푐푥,푐푤)and attains a global minimum at finite푤 푘 . In particular, increasing∥푤 푘 ∥further does not reduce L. Proof.Following the proof of Sec. G Thm. 1 in Baek et al. [3], the probabilities remain unchanged under uniform scaling for any 1-homogeneous distance푑. For the probabilities, if we replace푥 푖 by 푐푥 푖 and푤 푗 with푐푤 푗 , then푑(푐푥 푖 ,푐푤 푗 )= 푐푑(푥 푖 ,푤 푗 ), so the scaling factors cancel when using a harmonic link휅. Therefore, once the correct classification is achieved, no further reduction in loss is ob- tained by increasing∥푤 푘 ∥, and the loss achieves a global minimum at a finite푤 푘 . □ A.2 Margin-style generalization (PAC-Bayes view) Sec. G gives a PAC-Bayes margin bound that is finite because the harmonic solution has finite norm (Thm. 2) in Baek et al. [3]. Definition A.3 (Distance margin). Given prototypes푊=푤 푘 , define훾(푊)= min 푖 min 푗≠푦 푖 푑(푥 푖 ,푤 푗 ) − 푑(푥 푖 ,푤 푦 푖 ) . That is, 훾(푊)measures the gap between the nearest incorrect prototype and the correct prototype;훾(푊)>0 implies all training points are strictly closer to their true class prototype. Theorem A.4 (Generalization with metric margin). Assume all푥 푖 lie in a ball of radius푅(in the native norm of푑or its inducing space). Let∥푊∥ ★ denote a capacity measure compatible with푑. With probability at least 1− 훿, the generalization error of the classifier satisfies Pr (푥,푦) ℎ 푊 (푥)≠ 푦 = O 푅∥푊∥ ★ 훾(푊) √ 푛 + √︂ log(1/훿) 푛 ! , whereℎ 푊 (푥)= arg max 푘 푝 푘 (푥)denotes the predicted class and푛is the number of training samples. For the harmonic link,∥푊∥ ★ is finite by Thm. A.2, yielding a finite bound (cf. Sec. G Thm. 2) in Baek et al. [3]. Proof.Mirroring the proof for Sec. G Thm. 2 in Baek et al. [3], applying the standard PAC-Bayes margin bounds, one obtains that with at least probability 1−훿 , Pr (푥,푦) ℎ 푊 (푥)≠ 푦 = O 푅∥푊∥ ★ 훾(푊) √ 푛 + √︂ log(1/훿) 푛 ! . Since∥푊∥ ★ is finite by Theorem A.2, the bound is finite. □ B Integration into Deep Learning Pipelines TheDistLayerabstraction highlights that distance-based harmonic loss functions are highly modular and can be seamlessly integrated into existing deep learning pipelines. Theforwardmethod requires only three operations: i) computing pairwise distances between sample embeddings and class prototype weights, i) clamping values for numerical stability, and i) applying a softmin vialog_softmax to obtain normalized class probabilities. This makes the substitution of Euclidean distance with alternative metrics essentially a one-line change in the distance registry, with no modifications required in the broader training loop. Several design choices make the implementation robust. First, all distance functions are implemented in a vectorized form, ensur- ing GPU efficiency and avoiding explicit loops. Second, numerical safeguards (e.g.,휀-offsets, clamping before roots and divisions, regu- larization of covariance matrices) prevent instability across diverse datasets and architectures. Third, the registry-based design allows new distance functions to be added without disrupting the existing workflow, reinforcing the flexibility of harmonic loss as a general framework. From a methodological perspective, this implementation high- lights one of the key contributions of this work: the ease of re- placing cross-entropy with distance-based harmonic loss. Unlike cross-entropy, which relies on unbounded logit growth, the har- monic formulation treats classification as a problem of minimizing distances to interpretable class prototypes. The plug-and-play na- ture of theDistLayerdemonstrates that alternative geometries (e.g., cosine, Mahalanobis, Bray–Curtis) can be explored at negligi- ble engineering cost, paving the way for systematic evaluation of accuracy, sustainability, and interpretability across diverse tasks. C Model Architectures C.1 Vision We detail the architectures of the vision models used in our ex- periments – including a simple MLP, a small CNN, ResNet-50, and PVTv2-B0 – specifying their layers and neuron counts for reproducibility. All models were implemented in PyTorch, and for distance-based variants, the final fully-connected layer is replaced by a specialized distance layer as noted below. MLP: Input Layer: Accepts the flattened image input (e.g., 28× 28=784 features for MNIST, 32×32×3=3072 for CIFAR). Hidden Layer 1: Fully-connected layer with 512 neurons, followed by ReLU. Hidden Layer 2: Fully-connected layer with 256 neurons, followed by ReLU. Output Layer: Linear mapping from 256 units to the number of classes (10 for MNIST/CIFAR-10, 100 for CIFAR- 100). In_DISTvariants, this layer is replaced with a distance-based classification head (e.g. Euclidean, cosine) that computes distances Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. between the embedding and class prototypes, outputting negative distances as logits. CNN: Conv Layer 1: 2D convolution, 32 filters, kernel size 3×3, padding 1, followed by ReLU, then 2×2 max pooling. Conv Layer 2: 2D convolution, 64 filters, kernel size 3×3, padding 1, followed by ReLU, then 2×2 max pooling. Fully Connected Layer: Flattened output fed into a 128-unit linear layer with ReLU. Output Layer: Linear layer mapping the 128-D representation to the number of classes. In_DISTvariants, this is replaced by a distance metric layer. ResNet-50: Stem: Standard 7×7 convolution with 64 filters and stride 2, batch norm, ReLU, then 3×3 max pooling. For CIFAR/M- NIST, we use a 3×3 conv with stride 1 and remove max pooling. Stage 1: 3 bottleneck blocks, output 256 channels. Stage 2: 4 bot- tleneck blocks, output 512 channels. Stage 3: 6 bottleneck blocks, output 1024 channels. Stage 4: 3 bottleneck blocks, output 2048 channels. Global Pooling and Output: Global average pooling yields a 2048-D vector. In the baseline, a linear FC layer maps to logits. In_DISTvariants, the FC is replaced by a distance layer (e.g. cosine similarity) that outputs similarity-based logits. Pyramid Vision Transformer (PVTv2-B0): Stage 1: Over- lapping patch embedding with a 7×7 conv (stride 4), output 32 channels, followed by 2 Transformer encoder layers (1 attention head). Stage 2: 3×3 conv (stride 2), output 64 channels, followed by 2 encoder layers (2 heads). Stage 3: 3×3 conv (stride 2), output 160 channels, followed by 2 encoder layers (5 heads). Stage 4: 3×3 conv (stride 2), output 256 channels, followed by 2 encoder layers (8 heads). Global Pooling and Output: Global average pooling yields a 256-D vector. A linear classifier maps to the number of classes in the baseline, while in_DISTvariants this is replaced with a distance layer producing log-similarity or negative distance scores. Preprocessing Pipelines: MNIST: For MLP/CNN, grayscale input normalized to mean 0.5, std 0.5. For ResNet, normalization uses dataset statistics (mean 0.1307, std 0.3081). For PVT, grayscale converted to 3 channels, resized to 32 (PVT), normalized to mean/std 0.5. CIFAR-10: Normalization with mean (0.4914, 0.4822, 0.4465) and std (0.2023, 0.1994, 0.2010). ResNet uses data augmentation (random flips, crops, small rotations). CIFAR-100: Normalization with mean (0.5071, 0.4867, 0.4408) and std (0.2675, 0.2565, 0.2761). Stronger augmentation (random flips, crops, rotations, color jitter). PVT models use 32× 32 resized inputs with normalization. C.2 LLMs This section documents the LLM configurations used in our ex- periments for reproducibility. We report data preprocessing, archi- tectural details for GPT, BERT, and Qwen2-style models, how distance-based heads are integrated in place of the standard linear classifier, and the training/evaluation/emissions-logging pipeline. All models are implemented in PyTorch and trained with mixed precision when available. Data and Preprocessing Corpus and Storage. We pre-process a text corpus into contiguous token ID arrays and store them as memory-mapped files: • train.bin andval.bin:np.memmaparrays of typeuint16 containing token IDs. • meta.pkl: contains metadata includingvocab_size(used to configure model embeddings). Let푉denote the discovered vocabulary size frommeta.pkl(fall- back푉=50304 if not found). Batching. For a givenblock_size 퐿, batches are sampled by picking random starting indices and slicing 퐿 tokens: X = data[i : i+L], Y = data[i+1 : i+1+L](causal LM) All batching is performed on-device with pinned memory. We de- note batch_size by 퐵. Masking for MLM (BERT). For BERT runs, we construct masked language modeling (MLM) batches with the standard 15% corrup- tion: •Select≈15% token positions per sequence to form mask indicesM. • For each푖 ∈M: with 80% probability replace푥 푖 with[MASK] (id≤103 or capped by푉−1), with 10% replace by a random token in [0,푉), with 10% keep 푥 푖 unchanged. • Labels use the original token at masked positions and−100 (ignore index) elsewhere. This yieldsinput_ids,attention_mask(all-ones here), andlabels containing ground-truth only at masked positions. Architectures Across models below, the principal hyperparam- eters are: layers (푛 ℓ ), heads (푛 ℎ ), embedding dim (푑), context length (퐿=block_size), vocab size (푉). Unless otherwise specified, po- sitional encodings follow each model’s default (e.g., learned or rotary). GPT2 (Causal LM). Backbone. A standard decoder-only Trans- former with 푛 ℓ blocks. Each block has: •Multi-Head Causal Self-Attention with푛 ℎ heads, hidden size 푑 , and causal mask. •Position-wise MLP of width typically≈4푑with nonlinearity (e.g., GELU). •Pre/post LayerNorm and residual connections as in GPT- style decoders. Token Embeddings. Learnable token and (implicit) position em- beddings of sizes푉 ×푑and퐿×푑(or rotary embeddings if enabled). Projection Head (baseline). A linear layer푊 lm ∈ R 푑×푉 produc- ing logits over the vocabulary at each position. Distance Head (_DIST). The linear projection is replaced by a distance-based layer that treats the vocabulary columns as prototypes푤 푣 ∈ R 푑 푉 푣=1 . Given a hidden stateℎ 푡 ∈ R 푑 , the head returns per-token logits 푧 푡,푣 = −퐷(ℎ 푡 ,푤 푣 ;Θ)(orlog푆(ℎ 푡 ,푤 푣 )for similarity-type layers), where퐷(·,·;Θ)is one of the distances defined in the main text (Euclidean, cosine, Manhattan, Minkowski, Canberra, Bray–Curtis, Chebyshev, Mahalanobis, Hamming). This integrates seamlessly with the causal LM objective (next-token prediction via softmax over푉 ). BERT (Masked LM). Backbone. An encoder-only Transformer with 푛 ℓ layers, each with: • Multi-Head Self-Attention (bidirectional) with 푛 ℎ heads. • Position-wise MLP, LayerNorm, residual connections. Embeddings. Token embeddings푉 × 푑, segment/type embed- dings (size 2), and positional embeddings of length퐿. Head (base- line). The standard MLM classifier projects푑→푉(optionally via Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY an intermediate nonlinearity tied to the embedding matrix). Dis- tance Head (_DIST). We replace the MLM classifier with the same prototype-based distance layer used for GPT, but applied only at masked positions. For each masked token representationℎ 푖 , logits are푧 푖,푣 = −퐷(ℎ 푖 ,푤 푣 ;Θ)(or log-similarity), and cross-entropy is computed against the ground-truth token at 푖. Qwen2-style Decoder (Causal LM). Backbone. A decoder-only Trans- former similar to GPT, with model-specific details: • Rotary Position Embeddings (RoPE) with 휃 (e.g., 휃= 10 6 ). • RMSNorm with 휖 (e.g., 10 −6 ) in place of LayerNorm. • Grouped key/value heads:num_key_value_headsmay be < 푛 ℎ . • Intermediate MLP width (intermediate_size) configurable. Vocabulary. By default, we use Qwen’s native vocabulary (vocab_size=151,936); alternatively, one can adapt to the dataset vocab. Head (baseline vs. _DIST). As with GPT, the final projec- tion is either a linear layer to푉or a distance-based head over푉 prototype vectors. Distance-Based Output Layer For all three families (GPT, BERT/MLM, Qwen2), the baseline푑→푉 classifier is replaced in _DIST runs by a distance head: 푧 푣 (ℎ)= −∥ℎ− 푤 푣 ∥ 2 (Euclidean) −∥ℎ− 푤 푣 ∥ 1 (Manhattan) −∥ℎ− 푤 푣 ∥ 푝 (Minkowski, 푝 specified) − 1− ℎ ⊤ 푤 푣 ∥ℎ∥ 2 ∥푤 푣 ∥ 2 (Cosine) −퐷 Canberra (ℎ,푤 푣 ) or − 퐷 Bray–Curtis (ℎ,푤 푣 )(variants as defined) −∥ℎ− 푤 푣 ∥ ∞ (Chebyshev) − √︁ (ℎ− 푤 푣 ) ⊤ Σ −1 (ℎ− 푤 푣 )(Mahalanobis, variants) −퐷 Hamming (ℎ,푤 푣 )(soft/Gumbel/hard) where푤 푣 are learned prototype vectors (analogous to classifier weights). We adopt the numerically robust implementations given in the main text (e.g., small 휀, clamping, optional normalization of ℎand/or푤 푣 where appropriate). For cosine, we may outputlog- similarities for stability. Loss is standard cross-entropy over the푉 logits per position (causal) or per masked position (MLM). Training Setup and Optimization Device and Precision. We usebfloat16/float16/float32(con- figurable) with automatic mixed precision: torch.autocast(device_type=’cuda’, dtype=ptdtype). Training can run in single-GPU or DDP (torch.distributed) multi-GPU mode. In DDP,LOCAL_RANKselects the device, and gra- dients are synchronized across ranks. Initialization and Checkpointing. Models are initialized from scratch using the specified architecture config (layers, heads, width, 퐿,푉). For GPT-only runs we optionally supportinit_from=’gpt2*’, and for BERT we supportinit_from=’bert*’(when provided), with appropriate overrides. Checkpoints store model/optimizer state, iter_num, best_val_loss, and the configuration. Optimizer and LR Schedule. We use the model’s configure_optimizershelper to instantiate an Adam/AdamW- style optimizer with weight decay and(훽 1 ,훽 2 ). Learning rate fol- lows cosine decay with warmup: lr(푡)= lr max · 푡/warmup푡< warmup, lr min + 1 2 1+ cos 휋(푡−warmup) 푇−warmup lr max − lr min 푡 ≤ 푇, where푇 is lr_decay_iters. We apply gradient accumulation (gradient_accumulation_steps), optional gradient clipping (grad_clip), and AMP scaling (GradScaler). Objectives. • GPT/Qwen2 (causal LM): next-token cross-entropy over 푉 at each position. • BERT (MLM): cross-entropy computed only at masked po- sitions; non-masked labels set to−100 (ignored). Accuracy reporting: we compute token-level accuracy for monitor- ing (on next-token for causal LM, on masked tokens for MLM). Evaluation and Early Signals. At fixedeval_interval, we runestimate_loss()overeval_itersbatches on train/val splits (model ineval()), then resume training. Best validation loss check- points are saved; optional compile (torch.compile) can be enabled. Sustainability Tracking We integrate CodeCarbon to measure energy and emissions. At each evaluation interval: (1)Stop the tracker and record interval-level metrics: emissions (kg CO 2 ), duration, estimated CPU/GPU/RAM power and energy. (2)Log cumulative emissions and training metrics (loss, lr) to W&B (if enabled). (3)Restart the tracker for the next interval to avoid long-running file locks and to attribute emissions to training phases cleanly. At the end of training, we stop the tracker one final time and persist all accumulated records to a CSV (emissions_*.csv) alongside model checkpoints. Key Configurations (Reproducibility) The following knobs are saved in run configs/checkpoints and should be reported along- side results: (푛 ℓ , 푛 ℎ , 푑, 퐿, 푉),distance head type and parameters (Θ),batch size 퐵, precision, optimizer & betas, lr schedule (warmup, 푇, lr max , lr min ), grad accumulation, grad clip, DDP world size. When using_DISTvariants, we additionally report which dis- tance (Euclidean, cosine, Manhattan, Minkowski(푝), Canberra, Bray– Curtis, Chebyshev, Mahalanobis, Hamming), any normalization/s- caling flags, and regularization choices (e.g., Mahalanobis covari- ance learning/regularization). In all models, the sole architectural change introduced by har- monic loss is confined to the output head: a drop-in replacement of the linear classifier with a distance-based prototype head over the vocabulary. This isolates the effect of the loss geometry while keeping the Transformer backbone (and training recipe) unchanged, enabling controlled comparisons across distances in terms of accu- racy, interpretability (e.g., PCA-based analyses), and sustainability (emissions and runtime). Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. D Hyperparameter Configurations The hyperparameter settings in Tables 1–8 were chosen to bal- ance comparability, training stability, and sustainability. Below we highlight several important considerations. D.1 Language Models (OpenWebText) Table 1 specifies the core training parameters for GPT, BERT, and Qwen on OpenWebText. The main goal was to maintain a fair com- parison across models of varying scale by using effective batch sizes of similar order (76–128). This ensures that any differences observed in performance or emissions are attributable to the loss formulation, not simply to batch scaling. The use of AdamW with default훽 values (0.9, 0.999) follows current best practices for stability. Table 2 details architecture-specific modifications. BERT includes type embeddings and a masked language modeling (MLM) setup, while GPT and Qwen use causal language modeling (CLM). Qwen, being substantially larger, incorporates more advanced design ele- ments such as grouped query attention (GQA) and rotary position embeddings (RoPE). Table 3 summarizes these differences: GPT and Qwen follow causal objectives, while BERT relies on bidirectional context, which may affect the degree to which distance-based losses interact with their representations. D.2 Vision Models Tables 4–7 provide the vision settings across datasets. As shown in Table 4, optimizer and learning-rate schedules are backbone- specific: Adam for MLPs and CNNs, AdamW for transformers (PVT), and SGD with momentum for ResNet50. This reflects both conven- tion and empirical stability in preliminary experiments. Batch size selection (Table 5) reflects hardware utilization on H100 GPUs. Notably, lightweight backbones (e.g., CNNs) leverage very large batches (up to 8192 for MNIST), while transformer-based PVT is limited to much smaller batches (128–256) to fit memory constraints. These design choices affect emissions profiles: large-batch training can reduce wall-clock time but at the cost of GPU memory overhead. Learning-rate schedulers differ across models. For example, PVT employs cosine annealing, which smooths convergence and inter- acts well with distance-based loss formulations. ResNet50 relies on multi-step decay, ensuring stability across the long 200-epoch training horizon on CIFAR-100. Distance Layer Parameters. Table 8 summarizes the shared hyperparameters across all distance functions. The exponent푛is fixed to 1.0 and휀=10 −4 provides numerical stability. Importantly, distances are not scaled post hoc, ensuring that differences in results are directly attributable to the geometric properties of the chosen distance (Euclidean, Manhattan, Mahalanobis, etc.), rather than to auxiliary tuning. D.3 Discussion Language models. GPT and BERT use comparable depth/width with learned positional encodings, while Qwen is larger, adopts RoPE, and GQA. Effective batch sizes (via gradient accumulation) normalize throughput across models for fair comparison on Open- WebText. Vision models. Optimizer and scheduler choices follow com- mon practice: Adam/AdamW for MLP/CNN/PVT, SGD with mo- mentum for ResNet50; deeper/longer CIFAR-100 runs employ stepped or cosine schedules. Early-stopping patience scales with dataset difficulty. DistLayer defaults. A unified setting (푛=1.0,휀=10 −4 , no scaling) ensures distance variants differ only in geometry, not in auxiliary hyperparameters. These settings match the configuration used in our main experiments and figures. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 1: Core configuration for GPT, BERT, Qwen, and GPT-2B on OpenWebText. ConfigurationGPTBERTQwenGPT-2B 푛 layer 12122448 푛 head 12121420 푛 embd 7687688961600 Vocab size503045030415193650304 Dropout0.10.10.00.1 BiasTrueTrueTrueTrue Batch size163863 Grad. accum. steps821021 Effective batch size128766063 Learning rate2e-41e-41e-41e-4 Warmup iters500100010001000 Weight decay0.010.010.010.01 Grad clip1.01.01.01.0 Min LR2e-61e-61e-61e-6 Decay LRTrueTrueTrueTrue LR decay iters10000100001000010000 Max iters10000100001000010000 DatasetOpenWebTextOpenWebTextOpenWebTextOpenWebText dtypebfloat16bfloat16bfloat16bfloat16 OptimizerAdamWAdamWAdamWAdamW 훽 1 ,훽 2 0.9, 0.9990.9, 0.9990.9, 0.9990.9, 0.999 Eval interval100010001000500 Eval iters10010010025 Log interval50505025 Scale attn by inverse layer idxFalseFalseFalseFalse Table 2: Architecture-specific settings for GPT, BERT, Qwen, and GPT-2B. ConfigurationGPT BERT Qwen GPT-2B Block size / Seq length10245121024512 Type vocab size–2– Pad token id–0– MLM probability–0.15– Intermediate size–4864– # key–value heads–2– RMSNorm 휖–1e-6– RoPE 휃–1,000,000.0– E Statistical significance against Euclidean harmonic loss. E.1 Wilcoxon Signed-rank Tests To quantify whether non–Euclidean harmonic losses differ sys- tematically from the Euclidean reference, we ran paired Wilcoxon signed–rank tests over all dataset–backbone combinations (푁=16 pairs per distance). The resulting median score improvements and 푝–values are reported in Tables 9–11. Table 3: Key differences summary (task and position encod- ing). AspectGPTBERTQwen GPT-2B Model size (approx.) ∼124M ∼110M ∼494M ∼2B AttentionCausal Bidirectional Causal (GQA) Causal Training taskCLMMLMCLMCLM Position encodingLearnedLearnedRoPELearned CLM = Causal Language Modeling; MLM = Masked Language Modeling; GQA = Grouped Query Attention. On model performance (Table 9), the non–Euclidean distances do not achieve a statistically significant positive median improve- ment over the Euclidean harmonic loss. Several metrics (e.g., Ma- halanobis (Std.), Bray–Curtis (Std.), Canberra variants, Manhat- tan, Minkowski, Hamming) show significant negative medians (푝<0.05), indicating that when a difference is present it tends to favor the Euclidean reference in raw accuracy. This is consis- tent with our main results, where non–Euclidean geometries target interpretability and sustainability rather than headline accuracy gains. For interpretability (Table 10), we observe the opposite pattern. Distances such as Bray–Curtis (Norm.), Canberra (Robust/Std.), Chebyshev (Std.), Manhattan, and both cosine variants exhibit sta- tistically significant shifts in the number of principal components Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 4: Core training configuration by backbone and dataset. ConfigurationMLP CNN PVT ResNet50 LR (MNIST)3e-43e-41e-30.1 LR (CIFAR-10)3e-43e-41e-30.1 LR (CIFAR-100)3e-43e-45e-40.1 LR (MarathiSign)1.5e-41.5e-45e-40.05 LR (TinyImageNet)3e-43e-45e-40.1 Epochs (MNIST)404080100 Epochs (CIFAR-10)404080100 Epochs (CIFAR-100)150150150200 Epochs (MarathiSign)505010075 Epochs (TinyImageNet)100100200150 OptimizerAdamAdamAdamWSGD Weight decay000.011e-4 Momentum–0.9 Table 5: Batch size configuration on H100 GPU. Model MNIST CIFAR-10 CIFAR-100 MarathiSign TinyImageNet MLP204810241024128256 CNN81924096512512512 PVT25651225664128 ResNet50512512256128256 needed to explain 90% of the variance. The median differences are large in magnitude (e.g.,+12.8 for Bray–Curtis (Norm.),+9.8 for Canberra (Robust)), confirming that switching away from Euclidean induces a consistent and substantial change in representation ge- ometry across datasets and backbones. For sustainability (Table 11), four distances reach푝<0.05: Ma- halanobis (Std.) and Bray–Curtis (Std.) with positive medians, and Canberra (Weighted) and Mahalanobis (Chol.) with negative me- dians. This suggests that the carbon footprint differences between Euclidean and most non–Euclidean harmonic losses are modest and model–dependent: some geometries slightly increase emissions, others slightly decrease them, but strong systematic effects are rare once we fix backbone, data, and training budget. Overall, these nonparametric tests support our main claims: non– Euclidean harmonic losses do not uniformly dominate Euclidean in accuracy, but several of them induce statistically significant changes in representation structure, with only mild and mixed effects on emissions. E.2 Accuracy with Confidence Intervals. To complement the aggregate tables and Wilcoxon tests, Figure 3 reports accuracy curves for the top-performing losses on each dataset/backbone pair. For every setting we re-train each candidate with three random seeds and plot the mean trajectory together with a shaded 95% confidence interval (푛=3). Across datasets and architectures, two consistent patterns emerge. First, the ranking suggested by our radar plots and summary tables is preserved under multi-seed training: distance-based harmonic losses that previously appeared as strong contenders (e.g., cosine, Bray–Curtis, Minkowski) continue to track at least as well as, and often above, the cross-entropy and Euclidean baselines throughout training. In several regimes (notably CIFAR-10/CIFAR-100 with ResNet50 and MNIST with ResNet50/PVT), the confidence bands of the leading non-Euclidean harmonic loss lie systematically above those of the baselines in the later epochs, indicating that the final accuracy gains are not artifacts of seed choice but persist under sampling noise. Second, the width of the confidence intervals is often compa- rable or smaller for harmonic losses than for standard baselines. On datasets where optimization is more fragile (e.g., CIFAR-100 with PVT), cross-entropy and some regularized baselines (Focal, Center Loss) display visibly wider bands and occasional late-epoch fluctuations, whereas harmonic distances yield smoother trajecto- ries with tighter intervals, echoing our gradient stability findings. Importantly, we do not observe any case where a harmonic loss that outperforms Euclidean in the aggregate tables suffers a reversal when confidence intervals are taken into account. Overall, these multi-seed curves provide statistical depth to our vision experiments: performance improvements for non-Euclidean harmonic losses are accompanied by tight, stable confidence bands, supporting the claim that their advantages over Euclidean and cross-entropy are robust rather than due to random initialization. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 6: Learning-rate schedulers by backbone and dataset. ModelMNISTCIFAR-10CIFAR-100MarathiSignTinyImageNet MLPNoneNoneStepLR (50, 0.5)ReduceLR*StepLR (50, 0.5) CNNNoneNoneStepLR (50, 0.5)ReduceLR*StepLR (50, 0.5) PVTCosineAnn. (80)CosineAnn. (80)CosineAnn. (150)CosineAnn. (100)CosineAnn. (200) ResNet50StepLR (30, 0.1)StepLR (30, 0.1)MultiStep**StepLR (25, 0.1)MultiStep*** *ReduceLROnPlateau (mode=max, factor=0.5, patience=5, min_lr=1e-6) **MultiStepLR (milestones=[60,100,140],훾 =0.2) ***MultiStepLR (milestones=[80,120],훾 =0.2) Table 7: Dataset metadata and early-stopping settings (vision). ParameterMNIST CIFAR-10 CIFAR-100 MarathiSign TinyImageNet Num classes101010043200 Early stopping patience1515251015 Min improvement (%)0.010.010.010.010.01 Native image size28×28×1 32×32×332×32×3varies64×64×3 Processed size (MLP/CNN/PVT) 28×28×1 32×32×332×32×332×32×3224×224×3 Processed size (ResNet50)28×28×1 32×32×332×32×3224×224×3224×224×3 Table 8: Distance-layer shared parameters (all backbones). ParameterValue 푛1.0 휀1e-4 Scale distancesFalse Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 9: Wilcoxon signed–rank test comparing each non–Euclidean harmonic loss against the Euclidean harmonic baseline on average final test accuracy. Median score improvement is the median paired difference (non–Euclidean minus Euclidean) across푁dataset–backbone combinations. Positive values indicate that the non–Euclidean distance attains higher accuracy; the last column marks tests with 푝< 0.05. ComparisonN Pairs Median Impr. 푝-value 푝 adj Sig. (푝< 0.05) Bray–Curtis (Norm.)160.53170.170601.0000 Cosine (Stable)160.15980.453391.0000 Cosine (Unst.)160.09650.552081.0000 Mahalanobis (Chol.)160.04000.587171.0000 Bray–Curtis (Abs.)16-1.19670.103351.0000 Mahalanobis (Diag.)16-1.46330.083231.0000 Chebyshev (Std.)16-1.7232< 0.0010.0736Yes Minkowski (푝=3.0)16-1.81830.005670.6234Yes Canberra (Weighted)16-2.81670.002950.3565Yes Manhattan16-4.94670.002490.3083Yes Minkowski (푝=1.5)16-5.81170.001760.2232Yes Hamming (Soft)16-6.31830.003480.4110Yes Canberra (Robust)16-16.5883< 0.0010.0736Yes Canberra (Std.)16-18.3767< 0.0010.0736Yes Chebyshev (Smooth)16-21.3817< 0.0010.0736Yes Bray–Curtis (Std.)16-34.4379< 0.0010.0975Yes Mahalanobis (Std.)16-64.8082< 0.0010.0736Yes Table 10: Wilcoxon signed–rank test comparing each non– Euclidean harmonic loss against the Euclidean harmonic baseline on average intrinsic dimension (number of PCs required to reach 90% EV). Median score improvement is again the median paired difference (non–Euclidean minus Euclidean); here more negative values correspond to fewer required components. ComparisonN Pairs Median Impr. 푝-value 푝 adj Sig. (푝< 0.05) Bray–Curtis (Norm.)1612.8333< 0.001 0.1142Yes Canberra (Robust)169.8333 0.00162 0.2086Yes Canberra (Std.)168.1667< 0.001 0.0988Yes Chebyshev (Std.)167.7083 0.01864 1.0000Yes Manhattan167.6667 0.00412 0.4764Yes Cosine (Unst.)165.8333 0.01043 1.0000Yes Canberra (Weighted)165.2083 0.26768 1.0000 Cosine (Stable)164.3333 0.01127 1.0000Yes Mahalanobis (Std.)164.0417 0.12716 1.0000 Bray–Curtis (Std.)162.1667 0.34869 1.0000 Bray–Curtis (Abs.)161.0000 0.66019 1.0000 Hamming (Soft)160.7083 0.77730 1.0000 Minkowski (푝=1.5)16-0.0000 0.80665 1.0000 Minkowski (푝=3.0)16-0.0000 0.75554 1.0000 Mahalanobis (Diag.)16-0.0000 0.30656 1.0000 Chebyshev (Smooth)16-0.2083 0.77638 1.0000 Mahalanobis (Chol.)16-0.3333 0.85062 1.0000 Table 11: Wilcoxon signed–rank test comparing each non– Euclidean harmonic loss against the Euclidean harmonic baseline on average emissions (gCO 2 eq). Median score im- provement is the median paired difference (non–Euclidean minus Euclidean); negative values indicate lower emissions than the Euclidean reference. ComparisonN Pairs Median Impr. 푝-value 푝 adj Sig. (푝< 0.05) Mahalanobis (Std.)161.8037< 0.001 0.114Yes Bray–Curtis (Std.)161.1880 0.01620 1.000Yes Cosine (Unst.)160.2341 0.05249 1.000 Canberra (Std.)160.0351 0.77611 1.000 Chebyshev (Smooth)160.0208 0.73679 1.000 Canberra (Robust)16-0.0206 0.73679 1.000 Cosine (Stable)16-0.0239 0.97937 1.000 Minkowski (푝=1.5)16-0.0492 0.36552 1.000 Hamming (Soft)16-0.0909 0.26625 1.000 Bray–Curtis (Abs.)16-0.1066 0.20520 1.000 Chebyshev (Std.)16-0.1421 0.14056 1.000 Bray–Curtis (Norm.)16-0.1625 0.11477 1.000 Manhattan16-0.1976 0.12716 1.000 Minkowski (푝=3.0)16-0.4158 0.05249 1.000 Mahalanobis (Diag.)16-0.4587 0.05249 1.000 Canberra (Weighted)16-0.7687 0.03188 1.000Yes Mahalanobis (Chol.)16-0.9431 0.00411 0.476Yes Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY 30 40 50 010203040 Epoch Test Accuracy ArcFace (m=0.5, s=30) Baseline (C.E.) Center Loss (L=0.5) Center Loss (L=1) Center Loss (L=2) Euclidean Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR10 − MLP − Test Accuracy 10 15 20 25 01020304050 Epoch Test Accuracy Baseline (C.E.) Conf. Penalty (B=0.1) Conf. Penalty (B=0.2) Euclidean Label Smoothing (E=0.1) Label Smoothing (E=0.2) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR100 − MLP − Test Accuracy 80 90 010203040 Epoch Test Accuracy ArcFace (m=0.5, s=30) ArcFace (m=0.5, s=64) ArcFace Easy (m=0.5, s=30) Baseline (C.E.) Euclidean Label Smoothing (E=0.2) Shaded regions show 95% confidence intervals (n=3 seeds) MNIST − MLP − Test Accuracy 20 30 40 50 60 010203040 Epoch Test Accuracy ArcFace (m=0.3, s=30) ArcFace Easy (m=0.5, s=30) Baseline (C.E.) Bray−Curtis (Norm.) Euclidean Minkowski (p=3.0) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR10 − CNN − Test Accuracy 10 20 30 40 01020304050 Epoch Test Accuracy Baseline (C.E.) Conf. Penalty (B=0.1) Conf. Penalty (B=0.2) Euclidean Label Smoothing (E=0.1) Label Smoothing (E=0.2) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR100 − CNN − Test Accuracy 0 25 50 75 100 010203040 Epoch Test Accuracy ArcFace (m=0.3, s=30) ArcFace (m=0.5, s=30) ArcFace (m=0.7, s=30) ArcFace Easy (m=0.5, s=30) Baseline (C.E.) Euclidean Shaded regions show 95% confidence intervals (n=3 seeds) MNIST − CNN − Test Accuracy 0 50 100 150 0255075100 Epoch Test Accuracy ArcFace (m=0.7, s=30) Baseline (C.E.) Bray−Curtis (Norm.) Cosine (Stable) Cosine (Unst.) Euclidean Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR10 − ResNet50 − Test Accuracy 0 50 100 04080120 Epoch Test Accuracy Baseline (C.E.) Cosine (Stable) Cosine (Unst.) Euclidean Focal (G=2, A=0.25) Focal (G=2, A=0.5) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR100 − ResNet50 − Test Accuracy 0 100 01020304050 Epoch Test Accuracy ArcFace (m=0.3, s=30) ArcFace Easy (m=0.5, s=30) Baseline (C.E.) Bray−Curtis (Norm.) Cosine (Stable) Euclidean Shaded regions show 95% confidence intervals (n=3 seeds) MNIST − ResNet50 − Test Accuracy 20 40 60 0102030405060 Epoch Test Accuracy ArcFace (m=0.7, s=30) Baseline (C.E.) Conf. Penalty (B=0.1) Conf. Penalty (B=0.2) Euclidean Focal (G=2, A=0.25) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR10 − PVT − Test Accuracy 10 20 30 40 50 0204060 Epoch Test Accuracy Baseline (C.E.) Conf. Penalty (B=0.1) Euclidean Focal (G=2, A=0.25) Focal (G=2, A=0.5) Focal (G=3, A=0.25) Shaded regions show 95% confidence intervals (n=3 seeds) CIFAR100 − PVT − Test Accuracy 90 95 100 0204060 Epoch Test Accuracy ArcFace (m=0.3, s=30) Baseline (C.E.) Euclidean Focal (G=2, A=0.25) Focal (G=2, A=0.5) Focal (G=3, A=0.25) Shaded regions show 95% confidence intervals (n=3 seeds) MNIST − PVT − Test Accuracy Figure 3: Vision: Accuracy curves with Confidence Intervals. Shaded regions show 95% confidence intervals (n = 3 seeds) Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. F Vision: Radar Plots: Additional datasets (MNIST, CIFAR10) Figure 4 reports results for distance–based harmonic losses on MNIST and CIFAR10 across all four backbones (MLP, CNN, ResNet50, PVT). In summary, the MNIST and CIFAR-10 radar plots confirm what has been observed on other datasets: even on smaller bench- marks, non–Euclidean harmonic losses (particularly cosine, Bray- Curtis, and Chebyshev) can enhance representation structure and, on harder datasets, improve accuracy, all while maintaining com- parable or better sustainability than Euclidean harmonic loss and the cross–entropy baseline. F.1 MNIST RQ1: Model Performance (F1, Accuracy). Across all MNIST backbones, accuracy and F1 are saturated for several distances, but cosine–based harmonic losses (stable/unstable) and Bray–Curtis (normalized) remain among the most reliable high–performers. On MLP and CNN, these distances match or slightly exceed both Eu- clidean harmonic loss and the cross–entropy baseline. On ResNet50 and PVT, where capacity is ample, almost all distances reach nearly perfect accuracy, confirming that changing the distance in the har- monic head does not harm performance. RQ2: Interpretability (PC2 EV, PCA 90%). Even on this simple dataset, non–Euclidean distances already reshape the embedding geometry. Bray–Curtis (normalized) and Chebyshev (standard) pro- duce noticeably higher PC2 explained variance and reduce the num- ber of components needed to reach 90% EV, indicating compact, prototype–aligned clusters. Cosine harmonic losses also improve EV relative to Euclidean while maintaining top accuracy. Maha- lanobis and Minkowski variants (on ResNet50) further concentrate variance, but their interpretability advantage is less pronounced on MNIST because the task is almost linearly separable. RQ3: Sustainability (Duration/Epoch/GFLOPs, Emissions). For MNIST, the harmonic head constitutes a tiny fraction of the over- all compute, so all distances exhibit similar Duration/Epoch/GFLOPs and emissions. Cosine and Bray–Curtis are essentially neutral relative to Euclidean and cross–entropy; small differences arise mainly from minor variations in convergence speed rather than per–step cost. The key takeaway from MNIST is therefore that non–Euclidean harmonic losses can improve representation struc- ture without sacrificing accuracy or sustainability. F.2 CIFAR10 RQ1: Model Performance (F1, Accuracy). On CIFAR-10, cosine harmonic losses become clearly advantageous. For MLP and CNN, cosine (stable/unstable) and Bray–Curtis (normalized) consistently occupy the highest or near–highest F1 and accuracy, outperform- ing Euclidean harmonic loss and the cross–entropy baseline. On ResNet50 and PVT, cosine again delivers strong accuracy while re- maining competitive with the best non–Euclidean alternatives (e.g., Minkowski푝=3.0). Overall, cosine is the most robust choice across architectures once the task requires nontrivial feature extraction. RQ2: Interpretability (PC2 EV, PCA 90%). CIFAR-10 further highlights the interpretability benefits of non–Euclidean geome- try. Bray–Curtis and Chebyshev systematically increase PC2 EV and reduce PCA 90% dimensionality on MLP, CNN, and ResNet50, yielding sharper, more compact embeddings than Euclidean or cross–entropy. Cosine harmonic losses also improve EV over Eu- clidean, providing a favorable accuracy/interpretability compro- mise. On PVT, Canberra–weighted and Bray–Curtis variants simi- larly enhance variance concentration while preserving strong per- formance, reinforcing the observation that prototype–friendly dis- tances induce more structured feature spaces. RQ3: Sustainability (Duration/Epoch/GFLOPs, Emissions). On CIFAR-10, sustainability trends mirror those seen on larger datasets. Cosine harmonic loss is typically neutral to slightly favor- able in Duration/Epoch/GFLOPs and emissions relative to Euclidean and cross–entropy, especially on CNN and ResNet50 where con- vergence is faster. Bray–Curtis and Canberra variants introduce modest overhead, reflecting their more complex computations, but remain within the same qualitative efficiency regime. In all cases, the harmonic head is lightweight compared to the backbone, so the main sustainability differences arise from reduced steps–to–high accuracy rather than large per–step cost. G Results with Alternative Loss Functions To contextualize our distance–based harmonic losses, we bench- mark against four widely used alternatives that are often motivated by calibration, robustness in low–data regimes, or representational compactness. Below we summarize each loss, its objective, and why it is relevant along our three axes: effectiveness, sustainability, and interpretability. Unless otherwise noted, these baselines are applied with a conventional linear head and softmax; they can also be evaluated on distance–parameterized logits (e.g.,−dist(h,w 푐 ) ) to ensure architectural parity. Focal Loss (calibration, anti–grokking, class imbalance). Focal Loss reweights examples by their difficulty: L focal (z,푦)=−훼(1− 푝 푦 ) 훾 log푝 푦 , 푝 푦 = 푒 푧 푦 Í 푗 푒 푧 푗 , with focusing parameter훾 ≥0 and class weight훼 ∈ (0,1]. By down–weighting well–classified (overconfident) samples, it yields smoother gradient signals, often improving calibration and mit- igating late–stage overfitting behaviors akin to grokking. In our grids we consider훾∈2,3and훼∈0.25,0.5. Sustainability: mod- est compute overhead (same forward/backward shape as CE), but potentially fewer effective updates on easy samples; net carbon effect is typically neutral to slightly higher than CE, depending on convergence behavior. Label Smoothing (reduced overconfidence, low–data stability). La- bel Smoothing replaces the one–hot target with ̃ y=(1− 휀) e 푦 + 휀 퐾 1, L LS (z,푦)=− 퐾 ∑︁ 푐=1 ̃ 푦 푐 log푝 푐 , where휀 ∈ [0,1)controls smoothing (we use휀 ∈ 0.1,0.2). The softened targets reduce overconfidence and improve generalization in scarce–label settings; they also stabilize optimization by shrink- ing logit magnitudes. Interpretability: mild regularization can yield more isotropic features; sustainability: training cost matches CE, with potential reductions in steps–to–target when overconfidence previously harmed convergence. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Center Loss (prototype compactness, cluster interpretability). Cen- ter Loss explicitly penalizes the distance to a class prototype: L center (h,푦)= 1 2 h− c 푦 2 2 , L=L CE + 휆L center , with learnable class centersc 푘 and trade–off휆>0 (we test휆∈ 0.5,1,2). This encourages intra–class compactness and inter–class separability—properties that make feature clusters and decision prototypes easier to inspect. Sustainability: small extra memory and updates for c 푘 ; the overhead is minor relative to the backbone but measurable in long runs. Confidence Penalty (entropy regularization, anti–grokking). Confi- dence Penalty adds a negative–entropy term to discourage over–peaked posteriors: L CP (z,푦)=L CE (z,푦) − 훽H(p), H(p)=− 퐾 ∑︁ 푐=1 푝 푐 log푝 푐 , with훽>0 (we use훽∈0.1,0.2). By explicitly rewarding higher pre- dictive entropy when appropriate, it reduces brittle overconfidence and can temper delayed generalization (grokking–like) effects. Sus- tainability: essentially identical compute to CE; any carbon changes stem from altered convergence trajectories rather than per–step cost. ArcFace (angular margins, maximized class separation). ArcFace [13] introduces an additive angular margin that enlarges the decision boundary between classes on the unit hypersphere. Given nor- malized features h and normalized class weights w 푐 , the cosine similaritycos휃 푐 = ⟨h,w 푐 ⟩is modified for the target class푦by adding a fixed angular margin푚: cos(휃 푦 +푚)= cos휃 푦 cos푚− sin휃 푦 sin푚. ArcFace replaces the final linear classifier with a scaled angular softmax: L arcface (h,푦)=− log exp 푠 cos(휃 푦 +푚) exp 푠 cos(휃 푦 +푚) + Í 푐≠푦 exp ( 푠 cos휃 푐 ) , where푠is a feature-scale parameter (typically푠=30–64). By ma- nipulating angles rather than norms, ArcFace enforces tighter class clustering and larger inter-class separation, and is widely regarded as a strong margin-based baseline in metric learning. This makes it a particularly relevant comparator for harmonic losses: both ap- proaches normalize features to a hypersphere and control geometry around class prototypes, but ArcFace explicitly pushes angular mar- gins, whereas harmonic losses adjust the entire distance landscape. Sustainability: ArcFace is lightweight (same complexity as cosine classifiers), but the angular margin can slightly increase optimiza- tion stiffness, occasionally raising per-step compute or slowing convergence. In our experiments we consider푚∈0.3,0.5and 푠∈30, 64. Each baseline addresses a failure mode that harmonic losses also target but via different inductive biases: Focal/Label Smoothing em- phasize calibration and data efficiency; Center Loss operationalizes prototype compactness; Confidence Penalty discourages pathologi- cal overconfidence. This makes them natural comparators for our distance–based formulation, which subsumes prototype reasoning in its very parameterization and, as we show, can simultaneously improve accuracy, reduce emissions, and enhance interpretability. G.1 Vision: Fine-grained results A cross–backbone inspection in Tables 12–15 shows that the addi- tional baselines (Focal, Label Smoothing, Confidence Penalty, Cen- ter Loss, ArcFace) are competitive on accuracy, but do not displace the non-Euclidean harmonic losses as the most balanced options. On CIFAR-10 and CIFAR-100 with ResNet-50, cosine-based harmonic heads remain among the strongest configurations: they deliver the largest accuracy and F1 gains over cross-entropy (up to ≈11% relative on CIFAR-10 and 5% on CIFAR-100), while simultane- ously reducing emissions by 10–30% and sharply concentrating the representation (PC90% dropping from 50 to 5–8 components). Focal and ArcFace variants sometimes match top accuracy, but typically exhibit weaker EV/PC90% improvements or higher emissions, so they do not dominate the multi-criteria trade-off. For PVT backbones, where capacity and input resolution are higher, the picture is more nuanced. On CIFAR-10 PVT, calibration- oriented losses (Label Smoothing, Focal) offer slight accuracy im- provements and sizable emissions reductions, yet Euclidean har- monic achieves the most compact geometry (PC90% from 17 to 3) at only a small performance cost. On CIFAR-100 PVT, focal and confidence-penalty losses are best in accuracy, but Euclidean har- monic again produces the most concentrated feature spaces (PC90% 4 vs. 50), highlighting an interpretability advantage even when it is not the accuracy winner. On the high-resolution Marathi Sign dataset, nearly all meth- ods saturate accuracy (≥0.999), so the comparison is driven by structure and sustainability. Here, cosine-based harmonic losses for ResNet-50 (and to a lesser extent for PVT) achieve substan- tial EV/PC90% gains. e.g., Cosine (Unst.) reduces PC90% from 15.5 to 6.5 while slightly lowering emissions, demonstrating that our non-Euclidean harmonic heads remain competitive even in settings where strong baselines like ArcFace and Center Loss are present. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 12: Results for CIFAR100 PVT (top-8 losses) and % changes w.r.t. Baseline (CE). MethodAccF1gCO 2 eqEVPC90% Baseline0.39940.39703.670.097350.0 Focal (훾=2,훼=0.25)0.4017 (0.59%)0.3996 (0.65%)4.9609 (-35.35%) 0.1257 (29.22%) 50.0 (0%) Focal (훾=3,훼=0.25)0.4015 (0.53%)0.3999 (0.72%)4.9567 (-35.24%) 0.1364 (40.21%) 50.0 (0%) Focal (훾=2,훼=0.5)0.4010 (0.40%)0.3997 (0.66%)3.9182 (-6.90%) 0.1271 (30.64%) 50.0 (0%) Conf. Penalty (훽=0.1)0.3999 (0.13%)0.3977 (0.17%)5.2574 (-43.44%) 0.0963 (-0.96%) 50.0 (0%) Label Smoothing (휀=0.1) 0.3894 (-2.50%) 0.3888 (-2.08%) 2.6626 (27.36%) 0.0818 (-15.92%) 50.0 (0%) Conf. Penalty (훽=0.2)0.3859 (-3.38%) 0.3834 (-3.45%) 2.1718 (40.74%) 0.0749 (-23.00%) 50.0 (0%) Label Smoothing (휀=0.2) 0.3847 (-3.67%) 0.3851 (-3.00%) 2.2097 (39.71%) 0.0742 (-23.74%) 50.0 (0%) ArcFace (푚=0.5,푠=30)0.3728 (-6.65%) 0.3772 (-5.00%) 2.6171 (28.60%) 0.1259 (29.41%) 50.0 (0%) Euclidean0.2864 (-28.29%) 0.2945 (-25.83%) 6.4329 (-75.51%) 0.8414 (765.15%) 4.0 (92.00%) Table 13: Results for CIFAR100 ResNet50 (top-8 losses) and % changes w.r.t. Baseline (CE). MethodAccF1gCO 2 eqEVPC90% Baseline0.70060.699389.640.106950.0 Cosine (Stable)0.7381 (5.35%) 0.7384 (5.59%) 79.2831 (11.55%) 0.5915 (453.51%) 8.0 (84.00%) Focal (훾=2,훼=0.25)0.7349 (4.90%) 0.7342 (5.00%) 72.1795 (19.48%) 0.1468 (37.32%) 50.0 (0%) Focal (훾=2,훼=0.5)0.7341 (4.79%) 0.7332 (4.85%) 78.7234 (12.18%) 0.1224 (14.52%) 50.0 (0%) Cosine (Unst.)0.7340 (4.77%) 0.7349 (5.09%) 72.5413 (19.07%) 0.5891 (451.21%) 8.0 (84.00%) Focal (훾=3,훼=0.25)0.7311 (4.36%) 0.7308 (4.51%) 81.5875 (8.98%) 0.1554 (45.41%) 50.0 (0%) Label Smoothing (휀=0.1) 0.7261 (3.64%) 0.7248 (3.65%) 79.8223 (10.95%) 0.1469 (37.44%) 50.0 (0%) Label Smoothing (휀=0.2) 0.7221 (3.08%) 0.7206 (3.04%) 81.1883 (9.43%) 0.1524 (42.59%) 50.0 (0%) ArcFace (푚=0.7,푠=30)0.7166 (2.29%) 0.7150 (2.25%) 70.5590 (21.29%) 0.6059 (466.93%) 48.33 (3.33%) Euclidean0.7047 (0.59%) 0.7055 (0.89%) 87.7280 (2.13%) 0.4301 (302.49%) 33.67 (32.67%) Table 14: Results for MarathiSign PVT (top-8 losses) and % changes w.r.t. Baseline (CE). MethodAccF1gCO 2 eqEVPC90% Baseline0.99650.99642.850.213524.75 ArcFace (푚=0.7,푠=30)0.9997 (0.33%) 0.9997 (0.33%) 4.2108 (-47.88%) 0.1132 (-47.00%) 36.0 (-45.45%) Focal (훾=3,훼=0.25)0.9997 (0.32%) 0.9996 (0.32%) 2.7240 (4.34%)0.2749 (28.74%) 19.33 (21.89%) Center Loss (휆=1)0.9995 (0.31%) 0.9995 (0.31%) 5.1746 (-81.72%) 0.1824 (-14.56%) 29.67 (-19.87%) Conf. Penalty (훽=0.2)0.9995 (0.30%) 0.9994 (0.30%) 3.6912 (-29.63%) 0.1504 (-29.58%) 33.67 (-36.03%) Focal (훾=2,훼=0.25)0.9993 (0.29%) 0.9993 (0.28%) 2.5648 (9.93%)0.2853 (33.63%) 19.33 (21.89%) ArcFace (푚=0.3,푠=30)0.9992 (0.28%) 0.9992 (0.28%) 7.5158 (-163.94%) 0.1756 (-17.76%) 28.0 (-13.13%) Label Smoothing (휀=0.2) 0.9992 (0.28%) 0.9991 (0.27%) 2.6977 (5.26%)0.1190 (-44.26%) 35.0 (-41.41%) Cosine (Unst.)0.9991 (0.27%) 0.9991 (0.26%) 7.3447 (-157.93%) 0.5552 (160.04%) 7.0 (71.72%) Euclidean0.9994 (0.30%) 0.9994 (0.30%) 4.3621 (-53.19%) 0.5035 (135.83%) 14.25 (42.42%) Table 15: Results for MarathiSign ResNet50 (top-8 losses) and % changes w.r.t. Baseline (CE). MethodAccF1gCO 2 eqEVPC90% Baseline0.99980.999835.410.450715.5 Conf. Penalty (훽=0.1)0.9999 (0.01%) 0.9999 (0.01%) 29.6596 (16.24%) 0.4393 (-2.55%) 17.5 (-12.90%) Cosine (Stable)0.9999 (0.01%) 0.9999 (0.01%) 54.8139 (-54.80%) 0.7321 (62.42%) 5.5 (64.52%) Conf. Penalty (훽=0.2)0.9998 (0.00%) 0.9998 (0.00%) 30.3697 (14.23%) 0.3288 (-27.05%) 33.0 (-112.90%) Focal (훾=2,훼=0.25)0.9998 (0.00%) 0.9998 (0.00%) 24.95 (29.54%)0.4016 (-10.89%) 17.0 (-9.68%) Focal (훾=3,훼=0.25)0.9998 (0.00%) 0.9998 (0.00%) 28.1206 (20.58%) 0.4286 (-4.91%) 16.5 (-6.45%) Cosine (Unst.)0.9997 (-0.01%) 0.9997 (-0.01%) 33.8287 (4.46%)0.6427 (42.58%) 6.5 (58.06%) Label Smoothing (휀=0.1) 0.9997 (-0.01%) 0.9997 (-0.01%) 33.5266 (5.31%)0.1758 (-60.99%) 36.0 (-132.26%) Label Smoothing (휀=0.2) 0.9997 (-0.01%) 0.9997 (-0.01%) 38.7093 (-9.32%) 0.1641 (-63.60%) 36.0 (-132.26%) Euclidean0.9984 (-0.14%) 0.9983 (-0.15%) 50.2050 (-41.79%) 0.2787 (-38.18%) 50.0 (-222.58%) Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY G.2 Vision: Radar Plots with Additional Losses (MNIST, CIFAR10, CIFAR100) Figure 5 presents an expanded multi-criteria analysis across MNIST, CIFAR-10, and CIFAR-100 using MLP, CNN, ResNet50, and PVT backbones. This comparison tests whether the advantages previ- ously attributed to harmonic losses persist when measured against widely adopted alternatives for regularization, interpretability, and robustness. RQ1: Model Performance (F1, Test Accuracy). Across datasets and architectures, the harmonic losses – particularly the cosine- and Bray–Curtis–based variants – remain the strongest overall per- formers. While Focal Loss and Label Smoothing occasionally nar- row the gap on more complex datasets such as CIFAR-100, they do not consistently surpass harmonic losses across backbones. Cosine- based harmonic loss maintains higher accuracy and smoother con- vergence, especially for CNN and ResNet50, showing greater ro- bustness to data imbalance and optimization noise than either Focal or Confidence Penalty Loss. Even when Center Loss improves class compactness, it rarely translates into superior end-task accuracy, reinforcing that distance-based formulations bring more balanced generalization benefits. RQ2: Interpretability (PC2 EV, PCA 90%). The advantage of harmonic losses extends beyond performance: non-Euclidean har- monics, especially Bray–Curtis and Chebyshev, consistently yield the most structured latent geometries. They capture more variance with fewer principal components and align features more distinctly around class prototypes. Although Center Loss achieves compara- ble compactness in isolated cases, its representations tend to be less stable across architectures. Label Smoothing and Confidence Penalty slightly improve feature spread, but their effects remain shallow compared to the systematic geometric alignment achieved by harmonic formulations. This supports the notion that explicit metric-based geometry is a stronger driver of interpretability than indirect regularization. RQ3: Sustainability (Duration/Epoch, Emissions). When considering efficiency, harmonic losses continue to hold their edge. They achieve competitive or lower CO 2 emissions than both Eu- clidean and cross-entropy baselines. Among the new baselines, only Label Smoothing approaches similar energy efficiency, while Fo- cal Loss incurs additional computational cost due to its per-sample weighting. Despite this, none of the conventional alternatives out- perform the best-performing harmonic distances on a joint accu- racy–emission axis, confirming that the added geometric structure of harmonic loss does not come at a sustainability penalty. Three key findings emerge: i) Cosine- and Bray–Curtis–based harmonic losses remain the most consistently effective across accuracy, interpretability, and sustainability; i) Conventional reg- ularized losses such as Focal or Label Smoothing can mitigate spe- cific failure modes (imbalance, overconfidence) but do not achieve the same balance across criteria; i) The geometric grounding of harmonic losses continues to provide superior inductive structure, yielding smoother optimization, clearer feature organization, and greener training. Overall, these results reaffirm the general dom- inance and stability of harmonic loss formulations, even against strong baselines optimized for robustness and interpretability. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. MNIST MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Chebyshev (Std.) Cosine (Stable) Cosine (Unst.) Euclidean CIFAR10 MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Canberra (Robust) Chebyshev (Std.) Cosine (Stable) Euclidean MNIST CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Canberra (Robust) Cosine (Stable) Cosine (Unst.) Euclidean CIFAR10 CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Chebyshev (Std.) Cosine (Stable) Cosine (Unst.) Euclidean MNIST ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Norm.) Chebyshev (Std.) Euclidean Mahalanobis (Diag.) Minkowski (p=3.0) CIFAR10 ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Chebyshev (Std.) Cosine (Stable) Cosine (Unst.) Euclidean Minkowski (p=3.0) MNIST PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Canberra (Weighted) Cosine (Stable) Cosine (Unst.) Euclidean Mahalanobis (Chol.) CIFAR10 PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Baseline (C.E.) Bray−Curtis (Abs.) Bray−Curtis (Std.) Euclidean Hamming (Soft) Minkowski (p=3.0) Figure 4: Vision: Radar plots: 1) Model Performance (F1, Accuracy); 2) Interpretability (PC2 EV, PCA 90%), and 3) Sustainability (Duration/Epoch/GFLOPs, Emissions). Plots feature Baseline (Cross-Entropy), Euclidean harmonic, and the four top-performing non-Euclidean harmonic losses. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY MNIST MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Cosine (Stable) Chebyshev (Std.) Cosine (Unst.) Baseline (C.E.) Bray−Curtis (Norm.) CIFAR10 MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Baseline (C.E.) Euclidean Bray−Curtis (Norm.) Canberra (Robust) Cosine (Stable) Chebyshev (Std.) CIFAR100 MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) Baseline (C.E.) Conf. Penalty (B=0.1) Conf. Penalty (B=0.2) Label Smoothing (E=0.1) MNIST CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) Center Loss (L=2) Focal (G=2, A=0.5) Center Loss (L=1) Focal (G=2, A=0.25) CIFAR10 CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) Focal (G=3, A=0.25) Conf. Penalty (B=0.2) Focal (G=2, A=0.25) Baseline (C.E.) CIFAR100 CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) Focal (G=2, A=0.25) Baseline (C.E.) Label Smoothing (E=0.1) Conf. Penalty (B=0.2) MNIST ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Chebyshev (Std.) Bray−Curtis (Norm.) Focal (G=3, A=0.25) Minkowski (p=3.0) Mahalanobis (Diag.) CIFAR10 ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Minkowski (p=3.0) Chebyshev (Std.) Cosine (Unst.) Cosine (Stable) Bray−Curtis (Norm.) CIFAR100 ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Cosine (Unst.) Cosine (Stable) Manhattan Bray−Curtis (Abs.) Hamming (Soft) MNIST PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Canberra (Weighted) Cosine (Unst.) Cosine (Stable) Mahalanobis (Chol.) Baseline (C.E.) CIFAR10 PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Std.) Hamming (Soft) Bray−Curtis (Abs.) Minkowski (p=3.0) Cosine (Unst.) CIFAR100 PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch PC2 EV PCA 90% Euclidean Bray−Curtis (Std.) Canberra (Std.) Canberra (Robust) Chebyshev (Std.) Minkowski (p=1.5) Figure 5: Vision: Radar plots – MNIST, CIFAR10, CIFAR100: 1) Model Performance (F1, Accuracy); 2) Interpretability (PC2 EV, PCA 90%), and 3) Sustainability (Duration/Epoch, Emissions). Plots feature Baseline (Cross-Entropy), Euclidean harmonic, and the four top-performing losses. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. MarathiSign MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Baseline (C.E.) Bray−Curtis (Norm.) ArcFace (m=0.7, s=30) ArcFace (m=0.5, s=64) Conf. Penalty (B=0.1) TinyImageNet MLP 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) ArcFace (m=0.5, s=64) ArcFace (m=0.5, s=30) Center Loss (L=1) Center Loss (L=0.5) MarathiSign CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Baseline (C.E.) Bray−Curtis (Norm.) Focal (G=3, A=0.25) Conf. Penalty (B=0.2) Focal (G=2, A=0.25) TinyImageNet CNN 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Bray−Curtis (Norm.) Focal (G=2, A=0.25) Conf. Penalty (B=0.2) ArcFace (m=0.5, s=64) ArcFace (m=0.7, s=30) MarathiSign ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Chebyshev (Std.) Bray−Curtis (Norm.) Cosine (Stable) Cosine (Unst.) Minkowski (p=1.5) TinyImageNet ResNet50 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Cosine (Unst.) Cosine (Stable) Minkowski (p=1.5) Manhattan Chebyshev (Std.) MarathiSign PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Chebyshev (Std.) Bray−Curtis (Abs.) Manhattan Hamming (Soft) Canberra (Weighted) TinyImageNet PVT 0 2.5 5 7.5 10 Test Accuracy F1 Score Emissions Duration/Epoch/GFLOP PC2 EV PCA 90% Euclidean Cosine (Stable) Cosine (Unst.) Bray−Curtis (Norm.) ArcFace (m=0.5, s=64) ArcFace (m=0.5, s=30) Figure 6: Vision: Radar plots – Marathi Sign Language, TinyImageNet: 1) Model Performance (F1, Accuracy); 2) Interpretability (PC2 EV, PCA 90%), and 3) Sustainability (Duration/Epoch, Emissions). Plots feature Baseline (Cross-Entropy), Euclidean harmonic, and the four top-performing losses. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY G.3 Vision: Radar Plots with Additional Losses (Marathi Sign Language, TinyImageNet) Figure 6 extends the radar analysis to higher–resolution bench- marks (Marathi Sign Language, TinyImageNet) and augments the comparison set with strong loss baselines such as Focal Loss, Arc- Face, Center Loss, and Confidence Penalty. RQ1: Model Performance (Accuracy, F1). On Marathi Sign, the added losses make MLP and CNN particularly competitive: ArcFace and Focal occasionally obtain strong accuracy/F1, yet non–Euclidean harmonic losses remain among the best methods. For MLP, Bray–Curtis (Normalized) tracks or exceeds both cross entropy and ArcFace while preserving a smooth performance pro- file. For CNN, Bray–Curtis (Normalized) and cosine (stable/un- stable) consistently occupy the top accuracy/F1 slices; Focal and Confidence Penalty are competitive but never clearly dominate. On deeper backbones, the picture is even clearer: for ResNet50 and PVT on Marathi Sign, all top–performing methods are harmonic losses, indicating that distance–based harmonic heads outperform alternative losses outright in this regime. On TinyImageNet, a harder and more fine–grained benchmark, a similar pattern emerges. For MLP, ArcFace and Center Loss join Bray–Curtis (Normalized) and Euclidean in the top–performing losses, but Bray–Curtis remains competitive in accuracy while pro- viding different geometric and sustainability properties. For CNN, the strongest Focal and ArcFace variants reach high F1, yet Bray– Curtis (Normalized) again sits near the performance frontier. On ResNet50, the top–performing losses are entirely harmonic (cosine, Minkowski, Chebyshev, Euclidean), and on PVT TinyImageNet the leaders are dominated by cosine and Bray–Curtis, with ArcFace ap- pearing only as an alternative angular baseline. Overall, even in the presence of sophisticated angular–margin and confidence–shaping losses, non–Euclidean harmonic heads remain on or very near the performance Pareto frontier. RQ2: Interpretability (PC2 EV, PCA 90%). The higher–resolution datasets accentuate differences in representation geometry. On Marathi Sign, harmonic distances such as Bray–Curtis (Normal- ized/Absolute), Chebyshev (Standard), and Canberra/Ham- ming for PVT yield the strongest PCA structure: they maximize PC2 explained variance and minimize the number of components required to reach 90% EV, indicating compact, prototype–aligned embeddings. ArcFace and Focal improve angular separation but generally do not achieve the same variance concentration as the best harmonic distances. TinyImageNet confirms this trend. On MLP and CNN, Bray– Curtis and Chebyshev produce markedly higher PC2 EV and lower PCA 90% dimensionality than Euclidean and most additional base- lines, including Center Loss and Focal. For ResNet50 and PVT, co- sine and Bray–Curtis continue to enlarge the PCA wedges relative to Euclidean, whereas ArcFace’s contribution is mainly on perfor- mance rather than on variance concentration. Thus, across both Marathi Sign and TinyImageNet, the most interpretable geometries are consistently induced by non–Euclidean harmonic losses rather than by the newly added baselines. RQ3: Sustainability (Duration/Epoch/GFLOPs, Emissions). The sustainability axes show that richer loss design does not neces- sarily translate into greener training. On Marathi Sign MLP/CNN, several harmonic distances (e.g., Bray–Curtis (Normalized), Cheby- shev) attain equal or lower normalized Duration/Epoch/GFLOPs and emissions than cross–entropy and the added baselines; Focal and ArcFace occasionally incur slightly higher emissions due to their sharper gradients and additional computations. For ResNet50 and PVT, where backbone FLOPs dominate, all harmonic variants remain sustainability–competitive. On TinyImageNet, the pattern persists. ArcFace and Focal may match harmonic losses in accuracy, but they usually do so with simi- lar or higher emissions. Cosine and Bray–Curtis heads on ResNet50 and PVT often achieve comparable or better emissions than Eu- clidean, while still improving representation structure. Center Loss introduces modest overhead but does not surpass harmonic dis- tances in overall sustainability. Across Marathi Sign and TinyImageNet, adding strong base- lines such as ArcFace, Focal, Center Loss, and Confidence Penalty does not displace the non–Euclidean harmonic losses from the top tier. Whenever these baselines are competitive in accuracy, har- monic distances typically offer superior interpretability and compa- rable or lower emissions. This reinforces our central claim that dis- tance–tailored harmonic heads provide a robust, geometry–aware alternative to contemporary loss designs, remaining competitive or superior across performance, structure, and sustainability, even on challenging high–resolution vision benchmarks. G.4 Vision: Aggregated Emissions Figure 7 reports the cumulative CO 2 emissions for all vision experi- ments (MNIST, CIFAR-10/100, Marathi Sign, and TinyImageNet), ex- pressed as the difference in grams of CO 2 relative to the cross–entropy baseline (total CE emissions=650.49 gCO 2 eq over 680 runs). All methods lie within a band of roughly±8% of this baseline, showing that changing the distance in the harmonic head affects emissions in a controlled rather than catastrophic. Harmonic losses remain competitive or greener. The most sus- tainable region of the plot is dominated by non–Euclidean har- monic losses. In particular, Bray–Curtis (Normalized), Bray– Curtis (Absolute), Canberra (Weighted), and Mahalanobis (Cholesky) consistently achieve lower cumulative emissions than cross–entropy, even after adding the more demanding Marathi Sign and TinyImageNet settings. Cosine variants (Cosine (Stable) and Cosine (Unstable)) and Euclidean harmonic loss sit very close to the baseline, indicating that distance–based heads introduce essen- tially no sustainability penalty while still improving accuracy and representation structure. Behavior of additional baselines. Among the newly added con- ventional baselines, Label Smoothing and Confidence Penalty occupy the middle of the spectrum: their emissions are compara- ble to, but generally not better than, those of the best harmonic distances. In contrast, more aggressive objectives such as Focal Loss and large–margin Center Loss variants tend to cluster on the higher–emission side, reflecting the extra computation and slower convergence induced by power–scaled gradients and auxiliary cen- ter updates. ArcFace configurations behave similarly: moderate settings can be near baseline, but high margin/scale choices in- crease emissions relative to the most efficient harmonic distances. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Across all four datasets and backbones, the qualitative picture is stable. Non–Euclidean harmonic losses provide some of the greenest options, often achieving lower or baseline–level emissions while simultaneously improving accuracy and interpretability. The main exception is Mahalanobis (Standard), which remains the least sustainable configuration due to its covariance estimation cost—consistent with our earlier observation that Mahalanobis em- phasizes representation clarity at a computational price. Overall, the expanded analysis confirms that distance choice in the har- monic head materially affects the carbon footprint of training, and that carefully chosen non–Euclidean geometries (e.g., Bray–Curtis, Canberra, cosine) offer a favorable trade-off between performance, interpretability, and sustainability compared to both Euclidean har- monic loss and modern regularized baselines. 0 500 1000 Mahalanobis (Std.) Center Loss (L=2) Bray−Curtis (Std.) Canberra (Std.) ArcFace (m=0.5, s=64)ArcFace (m=0.3, s=30) Conf. Penalty (B=0.1)Conf. Penalty (B=0.2) ArcFace Easy (m=0.5, s=30) Chebyshev (Smooth) Baseline (C.E.) Canberra (Robust) Focal (G=3, A=0.25)Focal (G=2, A=0.25) Focal (G=2, A=0.5) Center Loss (L=0.5) Label Smoothing (E=0.1) Hamming (Soft) ArcFace (m=0.5, s=30) Manhattan Chebyshev (Std.) Center Loss (L=1) Minkowski (p=1.5)Minkowski (p=3.0) ArcFace (m=0.7, s=30) Cosine (Unst.) Cosine (Stable) Euclidean Label Smoothing (E=0.2) Mahalanobis (Diag.) Canberra (Weighted) Bray−Curtis (Absolute) Bray−Curtis (Norm.) Mahalanobis (Chol.) Distance Method Total Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −50 −25 0 25 50 Total Baseline = 650.49 gCO2eq | Total Experiments: 680 Vision Experiments: Aggregated Emissions by Distance Method Figure 7: Vision: Emissions Averaged Across Seeds and Ag- gregated Over all 12 Model Backbones. Figure 7 reports cumulative emission differences (gCO 2 eq) for each custom loss function across all 12 model/dataset combinations. (Total Baseline= 181.2 gCO 2 eq). Lower-than-baseline emissions: Mahalanobis (Standard) shows the largest positive delta, indicating consistently lower emissions; Bray–Curtis (Standard) and Cosine (Unstable) also sit on the positive side, with Canberra (Standard) and Cosine (Stable) slightly above zero. Euclidean and Manhattan are close to baseline. Of the new baseline loss functions introduced, Confidence Penalty performs on par with Cosine (Unstable) and the most efficiently compared to its counterparts. Almost all new losses are more efficient than Cross Entropy Loss, with varying degrees of success. Other distances are characterized by higher emissions, as shown by the red cluster. Results reinforce that non-Euclidean harmonic losses can be more sustainable than their Euclidean counterpart, and that the choice of distance materially affects the carbon footprint of model training. H Convergence analysis Vision: Figure 8 reports the training and validation loss trajectories for PVT and ResNet50 across all datasets and all non-Euclidean harmonic losses. A key concern is whether distances that introduce nontrivial geometric structure such as cosine and Mahalanobis lead to unstable optimization or distorted convergence landscapes. Empirically, we observe no such issues. Across MNIST, CIFAR-10, CIFAR-100, and Marathi Sign, all non– Euclidean harmonic losses exhibit smooth, monotonic decrease in the training objective and stable validation trends, with no os- cillation, divergence, or gradient explosion. Even distances with stronger geometric bias (e.g., Mahalanobis, Chebyshev, Bray-Curtis) converge at rates comparable to or faster than Euclidean harmonic loss. Cosine variants in particular show the fastest early descent, followed by steady tightening of the validation curves, consistent with their angularly flatter basins. Notably, none of the distances introduce optimization barriers, despite their differing curvature properties. Mahalanobis maintains stable descent even though anisotropic curvature could, in principle, yield direction-dependent gradients. Likewise, Canberra, Hamming, Manhattan, and Minkowski losses converge smoothly, indicating that the harmonic formulation effectively normalizes distance ge- ometry into a well-conditioned optimization surface. Overall, the loss curves demonstrate that the harmonic link func- tion absorbs geometric variability and translates heterogeneous distance metrics into similarly well-behaved training dynamics. This provides experimental evidence that alternative geometries do not impair convergence nor destabilize class separation boundaries. Language: Figure 9 reports training/validation loss, training accuracy, and (for GPT–2B) training and validation perplexity for cross–entropy, Euclidean harmonic, and Minkowski (푝=2) heads across BERT–0.1B, GPT–0.1B, GPT–2B, and QWEN2–0.5B. Across all architectures, the distance–based harmonic losses exhibit smooth optimization dynamics: losses decrease monotonically with no os- cillatory or unstable regimes, and accuracy curves increase steadily towards a plateau. For BERT–0.1B, Euclidean and Minkowski harmonic losses re- duce both training and validation loss more quickly than cross– entropy and converge to a lower plateau, while achieving higher final training accuracy. GPT–0.1B shows a similar pattern: all three heads converge, but the harmonic variants reach a given accuracy earlier and with gently sloping curves, indicating stable gradients. For GPT–2B and QWEN2–0.5B, the three heads track each other closely in both loss and accuracy, confirming that the change of geometry does not impede convergence even at larger scale. The validation loss curves mirror the training behaviour: no divergence or late–stage degradation is observed for any harmonic configura- tion. The GPT–2B perplexity panel further corroborates this picture. Training and validation perplexity decrease rapidly and stabilize to comparable levels for all heads; the harmonic variants sometimes achieve slightly faster early reductions, but do not introduce patho- logical behaviour. Overall, these results show that replacing the linear classifier with a distance–based harmonic head preserves, and in some cases marginally improves, the convergence proper- ties of standard cross–entropy while enabling the geometric and interpretability benefits discussed in the main paper. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY 01020304050607080 Epoch 0.0 0.5 1.0 1.5 2.0 Loss PVT: CIFAR10 020406080100120 Epoch 1 2 3 4 PVT: CIFAR100 01020304050607080 Epoch 0.0 0.5 1.0 1.5 2.0 Loss PVT: MNIST 010203040506070 Epoch 0 1 2 3 PVT: MarathiSign 0255075100125150175 Epoch 1 2 3 4 5 Loss PVT: TinyImageNet 020406080100 Epoch 0.0 0.5 1.0 1.5 2.0 ResNet50: CIFAR10 0255075100125150175200 Epoch 1 2 3 4 Loss ResNet50: CIFAR100 020406080 Epoch 0.0 0.5 1.0 1.5 2.0 ResNet50: MNIST 010203040506070 Epoch 0 1 2 3 4 Loss ResNet50: MarathiSign 020406080100120140 Epoch 1 2 3 4 5 ResNet50: TinyImageNet Bray-Curtis (norm) Canberra (weighted) Chebyshev (smooth) Cosine (stable) Euclidean Hamming (soft) Mahalanobis (diag.) Manhattan Minkowski (p=1.5) train_loss test_loss Figure 8: Loss convergence behavior with PVT and ResNet50: Training and Validation loss across all datasets with different non-Euclidean harmonic losses. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. GPT−2BQWEN2−0.5B BERT−0.1BGPT−0.1B 0250050007500100000250050007500 0250050007500100000250050007500 6 8 10 6 8 10 12 3 5 7 9 3 5 7 9 11 Training Step Training Loss Distance Metric Baseline (C.E.)EuclideanMinkowski (p=2.0) Training Loss GPT−2BQWEN2−0.5B BERT−0.1BGPT−0.1B 0250050007500100000250050007500 0250050007500100000250050007500 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.4 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 Training Step Training Accuracy Distance Metric Baseline (C.E.)EuclideanMinkowski (p=2.0) Training Accuracy GPT−2BQWEN2−0.5B BERT−0.1BGPT−0.1B 0250050007500100000250050007500 0250050007500100000250050007500 6 8 10 6 8 10 12 3 5 7 9 6 8 10 Training Step Validation Loss Distance Metric Baseline (C.E.)EuclideanMinkowski (p=2.0) Validation Loss 0 200 400 600 800 25005000750010000 Training Step Perplexity Distance Metric Baseline (C.E.) Euclidean Minkowski (p=2.0) Metric Type Training Validation Training and Validation Perplexity − GPT−2B Model Figure 9: Loss convergence behavior with language models (BERT-0.1B, GPT-0.1B, QWEN2-0.5B, GPT-2B). Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY I Grokking analysis: Modulo Addition Figure 10 summarizes the behavior of standard MLPs and H–MLPs trained on the synthetic modulo–addition task, a setting known to exhibit pronounced grokking effects under cross–entropy. The top two rows illustrate training curves and corresponding 2D em- beddings for baseline cross–entropy models (first two columns) and Euclidean harmonic loss (third and fourth columns). The re- maining rows extend this comparison to alternative non–Euclidean harmonic losses. Cross–entropy exhibits clear grokking. For both the standard MLP and its lightly regularized variant, cross–entropy produces the char- acteristic grokking pattern: training accuracy rapidly converges while test accuracy improves only after a long delay. This decou- pling is consistent with prior observations in algorithmic tasks, where cross–entropy tends to overfit memorization pathways be- fore discovering the true modular arithmetic structure. The PCA plots confirm this: the learned embeddings under cross–entropy have diffuse, irregular geometry, and the first two principal compo- nents explain only a small fraction of the variance (EV≈ 20–30%). Euclidean harmonic loss eliminates grokking and induces a per- fect geometric structure. In contrast, the Euclidean harmonic model reaches high train and test accuracy simultaneously. No grokking delay is observed. The PCA projection reveals a striking property: the latent representation forms a perfect 2D circle, and the first two principal components explain nearly all variance (EV≈100%). This matches theoretical expectations for harmonic distance–based clas- sification on cyclic group structure: the model learns an isometric embedding ofZ 푛 into the plane, validating the geometric alignment induced by harmonic objectives. Other distance–based harmonic losses replicate the circle struc- ture with similarly fast generalization. The bottom rows show that this desirable behavior is not unique to Euclidean distance. Cosine, stable cosine, Manhattan (1–norm), several Canberra variants, Ham- ming losses, Minkowski푝=3, Chebyshev, and others all produce the same qualitative outcome: •Immediate or near–immediate generalization, with no grokking phase. • Highly structured 2D embeddings, often forming a near– perfect circle. • Explained variance approaching 100%, indicating strong alignment to a low–dimensional manifold reflecting the al- gebraic symmetry of the task. Some distances (e.g., Hamming and Chebyshev) produce slightly rotated or warped circles, but the essential geometric structure and variance concentration remain intact. This demonstrates that harmonic losses robustly recover the underlying modular arithmetic structure regardless of the distance family. Harmonic losses reduce grokking compared to cross–entropy. Across all non–Euclidean distances tested, harmonic losses exhibit two consistent advantages over cross–entropy: (1) Reduced grokking or complete elimination of delayed generalization. Training and test accuracy rise together, in- dicating that the model discovers the algorithmic rule rather than memorizing individual cases. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. (2)Improved interpretability via stable geometric struc- ture. The emergence of a low–dimensional circular manifold with EV close to 1.0 serves as a quantitative and visual cer- tificate of representation clarity. These results reinforce the core claims of the paper: harmonic losses promote structured, prototype–aligned representations and smoother, more reliable optimization dynamics, even on tasks where cross–entropy typically groks. The fact that many distances achieve EV≈100% highlights that the benefits of harmonic classifi- cation do not depend on Euclidean geometry alone, but arise from the broader class of distance–based harmonic objectives. J Computational complexity: FLOPS Table 16 reports the approximate floating–point operations per for- ward pass for each backbone–dataset. On 32×32 inputs (MNIST re- sized, CIFAR-10/100, MarathiSign), the FLOP hierarchy is consistent: MLP is cheapest (<0.004 GFLOPs), CNN roughly 3×more expensive (≈0.012 GFLOPs), PVT adds another∼3×(≈0.038 GFLOPs), and ResNet50 is about 2×PVT (≈0.08 GFLOPs). Moving to high–resolution inputs (224×224 for TinyImageNet and our high–resolution MarathiSign runs) increases cost by two orders of magnitude: PVT rises to ∼1.9 GFLOPs and ResNet50 to∼4.1 GFLOPs per forward pass. These numbers highlight that i) sustainability differences across architec- tures are dominated by backbone FLOPs, while i) swapping Eu- clidean harmonic loss for alternative distances or baselines changes only the final classifier head, adding an푂(퐶푑)cost that is negligible compared to the convolutional / transformer body. Consequently, the per–step FLOP budget is effectively distance–invariant, and our sustainability comparisons across losses can be interpreted as dif- ferences in optimization dynamics (steps-to-target, stability) rather than raw arithmetic cost. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY 01000 Epoch 0.0 0.5 1.0 Accuracy Standard MLP weight decay=0 train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 26% 01000 Epoch 0.0 0.5 1.0 Standard MLP weight decay=0.5 train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 58% 01000 Epoch 0.0 0.5 1.0 Standard Transformer train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 46% 01000 Epoch 0.0 0.5 1.0 H-MLP (Euclidean) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 100% 01000 Epoch 0.0 0.5 1.0 Accuracy H-MLP (Manhattan) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 85% 01000 Epoch 0.0 0.5 1.0 H-MLP (Cosine) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 100% 01000 Epoch 0.0 0.5 1.0 H-MLP (Cosine (Stable)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 100% 01000 Epoch 0.0 0.5 1.0 H-MLP (Minkowski) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 51% 01000 Epoch 0.0 0.5 1.0 Accuracy H-MLP (Hamming (Hard)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 32% 01000 Epoch 0.0 0.5 1.0 H-MLP (Hamming (Soft)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 81% 01000 Epoch 0.0 0.5 1.0 H-MLP (Hamming (Gumbel)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 62% 01000 Epoch 0.0 0.5 1.0 H-MLP (Chebyshev) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 96% 01000 Epoch 0.0 0.5 1.0 Accuracy H-MLP (Chebyshev (Smooth)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 75% 01000 Epoch 0.0 0.5 1.0 H-MLP (Canberra (Regular)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 29% 01000 Epoch 0.0 0.5 1.0 H-MLP (Canberra (Weighted)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 89% 01000 Epoch 0.0 0.5 1.0 H-MLP (Canberra (Robust)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 27% 01000 Epoch 0.0 0.5 1.0 Accuracy H-MLP (Bray-Curtis (Regular)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 46% 01000 Epoch 0.0 0.5 1.0 H-MLP (Bray-Curtis (Absolute)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 93% 01000 Epoch 0.0 0.5 1.0 H-MLP (Bray-Curtis (Normalized)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 30% 01000 Epoch 0.0 0.5 1.0 H-MLP (Mahalanobis (Regular)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 34% 01000 Epoch 0.0 0.5 1.0 Accuracy H-MLP (Mahalanobis (Cholesky)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 72% 01000 Epoch 0.0 0.5 1.0 H-MLP (Mahalanobis (Diagonal)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 40% 01000 Epoch 0.0 0.5 1.0 H-Trans (Euclidean) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 32% 01000 Epoch 0.0 0.5 1.0 H-Trans (Manhattan) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 42% 01000 Epoch 0.0 0.5 1.0 Accuracy H-Trans (Cosine) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 35% 01000 Epoch 0.0 0.5 1.0 H-Trans (Cosine (Stable)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 33% 01000 Epoch 0.0 0.5 1.0 H-Trans (Minkowski) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 64% 01000 Epoch 0.0 0.5 1.0 H-Trans (Hamming (Hard)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 37% 01000 Epoch 0.0 0.5 1.0 Accuracy H-Trans (Hamming (Soft)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 51% 01000 Epoch 0.0 0.5 1.0 H-Trans (Hamming (Gumbel)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 32% 01000 Epoch 0.0 0.5 1.0 H-Trans (Chebyshev) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 52% 01000 Epoch 0.0 0.5 1.0 H-Trans (Chebyshev (Smooth)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 29% 01000 Epoch 0.0 0.5 1.0 Accuracy H-Trans (Canberra (Regular)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 29% 01000 Epoch 0.0 0.5 1.0 H-Trans (Canberra (Weighted)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 40% 01000 Epoch 0.0 0.5 1.0 H-Trans (Canberra (Robust)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 33% 01000 Epoch 0.0 0.5 1.0 H-Trans (Bray-Curtis (Regular)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 41% 01000 Epoch 0.0 0.5 1.0 Accuracy H-Trans (Bray-Curtis (Absolute)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 48% 01000 Epoch 0.0 0.5 1.0 H-Trans (Bray-Curtis (Normalized)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 32% 01000 Epoch 0.0 0.5 1.0 H-Trans (Mahalanobis (Regular)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 43% 01000 Epoch 0.0 0.5 1.0 H-Trans (Mahalanobis (Cholesky)) train test 3210123 PC0 2 0 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 34% 01000 Epoch 0.0 0.5 1.0 Accuracy H-Trans (Mahalanobis (Diagonal)) train test 3210123 PC0 2 0 2 PC1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 EV: 32% Figure 10: Results on standard MLP trained for modular addition. The harmonic model trained for modular addition generalizes quickly without grokking. Moreover, the embedding forms a perfect 2D circle. EV in the plot represents the explained variance by the first two principal components of the embedding. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 16: Per-sample FLOPs, GFLOPs, and parameter counts for each backbone and dataset. Model DatasetIn Ch. H W #ClsFLOPs Params GFLOPs CNNCIFAR103323210123070725450980.0123 CNNCIFAR10033232100123301125567080.0123 CNNMarathiSign3323243123155205493550.0123 CNNMNIST128281085200644216420.0085 CNNTinyImageNet3 224 224200602966144257354320.6029 MLPCIFAR103323210341376017072740.0034 MLPCIFAR10033232100345984017304040.0034 MLPMarathiSign3323243343065617157550.0034 MLPMNIST128281010708485358180.0010 MLPTinyImageNet3 224 224200309536256 1547690960.3095 PVTCIFAR10332321038268630127465600.0382 PVTCIFAR1003323210038314710127696000.0383 PVTMarathiSign332324338285526127550080.0382 PVTMNIST332321038268630127465600.0382 PVTTinyImageNet3 224 224200 1899590400127952001.8995 ResNet50 CIFAR10332321079618429234724800.0796 ResNet50 CIFAR1003323210079987069236568000.0799 ResNet50 MarathiSign3 224 22443 4096080128235400644.0960 ResNet50 MNIST128281060007460234724800.0600 ResNet50 TinyImageNet3 224 224200 4096723200238616004.0967 Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY K Geometric Insights To better illustrate how different harmonic distances shape the embedding geometry, we visualize the last–layer representations of ResNet50 on MNIST (see Figure 11) and CIFAR10 (see Figure 12) using 2D PCA, with class prototypes overlaid as markers. For the Euclidean harmonic head, the class clusters are roughly spherical and separated by (approximately) straight boundaries in the pro- jection: decision regions are controlled mainly by radial distance to each prototype, yielding isotropic attraction basins around each center. Under Cosine harmonic loss, the picture changes markedly. Fea- tures and prototypes concentrate on (or very near to) a common hypersphere, so the PCA plot shows clusters arranged along a cir- cle. Classes are separated primarily by their angle rather than their norm, and decision boundaries correspond to angular bisectors between prototypes. This matches our geometric claim that cosine harmonic removes radial curvature and constrains optimization to an angular manifold: as training proceeds, points slide along the sphere towards their prototype, producing wide, smooth basins and stable gradient norms. By contrast, Mahalanobis harmonic loss induces anisotropic curvature. After whitening byΣ −1/2 , the decision boundaries are linear, but in the original feature space they correspond to ellip- soidal contours. In the PCA plots this appears as elongated clusters and distorted attraction basins around prototypes, with some di- rections exhibiting much tighter concentration than others. When the empirical covariance is well–conditioned this yields very sharp, well–separated clusters (high variance concentration), but when eigenvalues are highly unbalanced the same anisotropy can make optimization more sensitive to particular directions. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. (a)(b) (c)(d) Figure 11: Geometric effect of distance–based harmonic losses on ResNet50 embeddings (MNIST). From top to bottom: Baseline (a), Euclidean harmonic loss (b), cosine harmonic loss (c), and Mahalanobis harmonic loss (d). Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY (a)(b) (c)(d) Figure 12: Geometric effect of distance–based harmonic losses on ResNet50 embeddings (CIFAR10). From top to bottom: Baseline (a), Euclidean harmonic loss (b), cosine harmonic loss (c), and Mahalanobis harmonic loss (d). Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. L Additional results L.1 Vision: Tables The empirical evaluation of non-Euclidean harmonic losses across MNIST, CIFAR-10, and CIFAR-100 with MLP, CNN, and ResNet50 backbones reveals several consistent patterns. Model Performance. Cosine distance emerges as the most reli- able performer across architectures and datasets. In both stable and unstable variants, cosine harmonic loss consistently improves test accuracy and F1 relative to Euclidean, with gains most pronounced in deeper models (CNNs and ResNets) and in medium-complexity datasets such as CIFAR-10. Bray–Curtis offers modest gains in cer- tain contexts but is less consistent, while Mahalanobis can improve accuracy on simple datasets (e.g., MNIST) but often lags behind cosine in more challenging regimes. Euclidean harmonic loss, while better than cross-entropy in terms of stability, is consistently out- performed by cosine-based alternatives. Interpretability. Distances strongly reshape the geometry of the learned representations. Cosine and Bray–Curtis often yield large improvements in explained variance (EV), indicating more compact feature spaces aligned with class prototypes. Mahalanobis produces the most dramatic gains in EV, frequently approaching full variance explanation, but this comes at the cost of stability and efficiency. Prototype coverage (PC90%) tends to shrink under cosine and Mahalanobis, highlighting sharper clustering effects: models assign fewer prototypes to cover 90% of variance, mak- ing the representation space more interpretable but less evenly distributed. Sustainability. Sustainability outcomes mirror performance trends. Cosine distances typically reduce carbon emissions relative to Euclidean, in some cases by up to 40%, making them both effective and energy-efficient. Bray–Curtis shows mixed results, with occa- sional emission savings but less consistent behavior. Mahalanobis tends to incur higher emissions, reflecting the computational over- head of covariance estimation and matrix operations. Shallow ar- chitectures (MLPs) show less differentiation across distances in emissions, while deeper backbones amplify both the benefits (co- sine) and costs (Mahalanobis). Trade-offs. Taken together, the results confirm that distance choice is not neutral in harmonic loss. Cosine provides the most favorable balance across performance, interpretability, and sustain- ability, representing the strongest general-purpose alternative to Euclidean. Bray–Curtis occupies a middle ground, offering inter- pretability benefits without always delivering accuracy or efficiency gains. Mahalanobis maximizes interpretability at a clear sustain- ability cost, making it attractive primarily when prototype clarity outweighs computational expense. Euclidean serves as a stable but suboptimal baseline. Conclusion. This systematic study establishes that non-Euclidean harmonic losses provide a flexible and effective design space. In particular, cosine distance offers a compelling replacement for cross- entropy and Euclidean harmonic loss in vision tasks, consistently improving accuracy, interpretability, and sustainability. These find- ings position distance-tailored harmonic losses as a promising av- enue for advancing deep learning models that are not only accurate but also more transparent and energy-conscious. L.2 Vision: Sustainability L.2.1 MNIST. Figure 13 summarizes the carbon deltas (gCO 2 eq relative to cross-entropy) when swapping the training objective for harmonic-loss variants on MNIST across four backbones. MLP. Most distances reduce per–step emissions vs. cross-entropy (green bars), with the largest savings from heavier geometry that replaces the softmax/cross-entropy path (e.g., Mahalanobis/stan- dardized, Chebyshev). Euclidean and Bray–Curtis yield modest savings; only a few variants show small positive overheads. Given MNIST’s simplicity and the near-saturation accuracies, these reduc- tions likely translate into net greener runs because steps-to-target are comparable. CNN. A broad set of distances are carbon-negative vs. base- line. Again, standardized Mahalanobis/Chebyshev rank among the lowest-emission options; Bray–Curtis and Euclidean remain con- sistently frugal. Variants that introduce extra normalization or tem- perature schedules can erode part of the gain but rarely flip the sign. ResNet50. The deepest convolutional model shows the largest per–step savings: many distances deliver substantial negative deltas relative to cross-entropy, suggesting that replacing the softmax loss with metric-based objectives amortizes well at this scale. Only a handful of choices (e.g., certain Chebyshev/Canberra parameteriza- tions) incur small positive overheads. PVT (vision transformer). In contrast to the CNN family, most distances increase per–step emissions over the baseline. The trans- former’s attention and normalization stack appears less amenable to the heavier distance computations; only a couple of standard- ized/normalized variants produce small savings. On PVT, greener training favors the lightest geometries or retaining cross-entropy. Takeaways. i) On MNIST, distance-based harmonic losses are often carbon-favorable for MLP/CNN/ResNet50, with the biggest gains on the deepest CNN; i) these gains are not universal—PVT tends to pay a premium; i) because test accuracy curves on MNIST converge similarly across losses, the per–step savings for CNN/Res- Net50 likely convert into lower end-to-end energy. Practically, we recommend Euclidean/Bray–Curtis/standardized Mahalanobis for convolutional backbones, and cautious use (or kernel-fused, mixed- precision implementations) of heavier distances on transformer- style models. Reporting both per–step emissions and energy-to- target accuracy remains essential for fair sustainability claims. L.2.2 CIFAR-10. Figure 14 reports carbon deltas in gCO 2 eq relative to cross-entropy when training with harmonic-loss distances on CIFAR-10. MLP. Most distances are carbon–negative versus baseline, yield- ing small–to–moderate per–step savings. A few choices incur mild overheads (rightmost bars), indicating that added normalization or temperature scheduling can offset the gains on shallow networks. CNN. The pattern strengthens: a broad set of distances reduce per–step emissions relative to cross-entropy. Only a handful of vari- ants sit near zero or slightly positive, suggesting that, for convolu- tional encoders on CIFAR-10, metric-based objectives are generally more frugal per step. ResNet50. Savings are uniform and largest: all distances fall below the baseline, with substantial negative deltas. This indi- cates that replacing the softmax loss amortizes particularly well Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 17: Results for CIFAR100 CNN. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.37950.37951.180.45929549.3333 Bray-Curtis (Norm.)0.3229 (-14.91%)0.3182 (-16.16%)0.8132 (30.94%)0.9094 (98%)2.6667 (94.59%) Mahalanobis (Chol.)0.2927 (-22.86%)0.2921 (-23.04%)0.727 (38.25%)0.341 (-25.75%)50 (-1.35%) Cosine (Unst.)0.2602 (-31.44%)0.2667 (-29.73%)2.1156 (-79.68%)0.5306 (15.52%)45 (8.78%) Cosine (Stable)0.2501 (-34.09%)0.2516 (-33.71%)1.4263 (-21.14%)0.5216 (13.57%)45 (8.78%) Euclidean0.2413 (-36.4%)0.2431 (-35.95%)1.2866 (-9.28%)0.4362 (-5.02%)50 (-1.35%) Table 18: Results for CIFAR100 MLP. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.26170.25820.850.28520350.0 Bray-Curtis (Norm.)0.226 (-13.64%)0.2191 (-15.15%)0.8938 (-5.35%)0.9843 (245.11%)1 (98%) Mahalanobis (Chol.)0.1833 (-29.96%)0.1811 (-29.85%)1.0889 (-28.34%)0.0354 (-87.57%)50 (-0%) Bray-Curtis (Abs.)0.1444 (-44.81%)0.1392 (-46.07%)2.1255 (-150.51%)0.5317 (86.43%)47.6667 (4.67%) Cosine (Unst.)0.1237 (-52.74%)0.1186 (-54.06%)0.5064 (40.31%)0.3799 (33.21%)40.6667 (18.67%) Euclidean0.119 (-54.53%)0.1222 (-52.69%)0.589 (30.58%)0.2437 (-14.55%)50 (-0%) Table 19: Results for CIFAR100 ResNet50. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.69830.696987.770.10721650.0 Cosine (Stable)0.7357 (5.35%)0.736 (5.61%)72.9745 (16.85%)0.5979 (457.66%)8 (84%) Cosine (Unst.)0.7323 (4.87%)0.7332 (5.21%)71.7592 (18.24%)0.5857 (446.27%)8 (84%) Bray-Curtis (Norm.)0.655 (-6.19%)0.6513 (-6.54%)106.4049 (-21.24%)0.7131 (565.08%)6 (88%) Mahalanobis (Chol.)0.6274 (-10.15%)0.6239 (-10.47%)138.9317 (-58.3%)0.7353 (585.81%)17.5 (65%) Euclidean0.7055 (1.03%)0.7062 (1.33%)97.432 (-11.01%)0.5679 (429.66%)25.5 (49%) Table 20: Results for CIFAR10 CNN. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.62780.62691.120.6880819.0 Mahalanobis (Chol.)0.6644 (5.82%)0.6642 (5.95%)1.1139 (0.68%)0.4752 (-30.93%)50 (-455.56%) Bray-Curtis (Norm.)0.6597 (5.08%)0.6551 (4.5%)1.1489 (-2.45%)0.8913 (29.54%)4.3333 (51.85%) Minkowski (p=3.0)0.6589 (4.95%)0.6593 (5.17%)1.1598 (-3.42%)0.5425 (-21.15%)50 (-455.56%) Cosine (Stable)0.6584 (4.87%)0.6566 (4.74%)1.1663 (-3.99%)0.647 (-5.97%)18.6667 (-107.41%) Euclidean0.6495 (3.45%)0.6476 (3.31%)1.1228 (-0.12%)0.6582 (-4.34%)14.3333 (-59.26%) Table 21: Results for CIFAR10 MLP. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.53970.53850.530.34650447.0 Bray-Curtis (Norm.)0.5224 (-3.21%)0.5201 (-3.41%)0.5264 (0.81%)0.967 (179.07%)1 (97.87%) Mahalanobis (Chol.)0.5087 (-5.75%)0.5088 (-5.51%)0.458 (13.7%)0.0522 (-84.94%)50 (-6.38%) Bray-Curtis (Abs.)0.4934 (-8.59%)0.4924 (-8.55%)0.6313 (-18.96%)0.2434 (-29.76%)50 (-6.38%) Bray-Curtis (Std.)0.4931 (-8.64%)0.4935 (-8.35%)0.6435 (-21.25%)0.2906 (-16.14%)50 (-6.38%) Euclidean0.4871 (-9.74%)0.4852 (-9.9%)0.4303 (18.92%)0.4303 (24.19%)42.3333 (9.93%) Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. Table 22: Results for CIFAR10 ResNet50. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.8430.843148.650.25721150.0 Cosine (Stable)0.9262 (9.87%)0.9262 (9.86%)40.6776 (16.39%)0.7559 (193.9%)5 (90%) Cosine (Unst.)0.9234 (9.54%)0.9234 (9.53%)29.3968 (39.58%)0.761 (195.86%)5 (90%) Bray-Curtis (Norm.)0.9193 (9.05%)0.9192 (9.02%)45.6222 (6.23%)0.7883 (206.49%)5 (90%) Chebyshev (Std.)0.905 (7.36%)0.905 (7.34%)48.5505 (0.21%)0.9995 (288.59%)1 (98%) Euclidean0.9185 (8.96%)0.9185 (8.94%)45.8759 (5.71%)0.683 (165.56%)25.5 (49%) Table 23: Results for MNIST CNN. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.97820.97821.190.58563310.6667 Bray-Curtis (Norm.)0.9889 (1.09%)0.9888 (1.09%)1.1348 (4.42%)0.7225 (23.38%)13.6667 (-28.12%) Mahalanobis (Chol.)0.9879 (1%)0.9879 (0.99%)1.0639 (10.39%)0.4673 (-20.2%)36.3333 (-240.63%) Minkowski (p=3.0)0.9877 (0.97%)0.9876 (0.96%)1.1154 (6.06%)0.4195 (-28.37%)49.3333 (-362.5%) Hamming (Soft)0.9833 (0.52%)0.9832 (0.51%)1.1815 (0.49%)0.3089 (-47.26%)50 (-368.75%) Euclidean0.9831 (0.5%)0.9831 (0.5%)1.1543 (2.78%)0.4413 (-24.65%)20.3333 (-90.62%) Table 24: Results for MNIST MLP. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.9760.97580.550.56572310.3333 Cosine (Unst.)0.978 (0.2%)0.9778 (0.2%)0.5264 (3.58%)0.382 (-32.48%)10 (3.23%) Mahalanobis (Chol.)0.9774 (0.14%)0.9771 (0.14%)0.5611 (-2.78%)0.092 (-83.74%)50 (-383.87%) Cosine (Stable)0.9766 (0.06%)0.9764 (0.06%)0.5266 (3.54%)0.4033 (-28.71%)9.3333 (9.68%) Chebyshev (Std.)0.9756 (-0.04%)0.9754 (-0.04%)0.5881 (-7.73%)0.7865 (39.03%)5.6667 (45.16%) Euclidean0.9799 (0.4%)0.9798 (0.41%)0.5221 (4.35%)0.358 (-36.72%)9 (12.9%) Table 25: Results for MNIST ResNet50. Parentheses: % changes w.r.t. Baseline (Cross-Entropy). MethodAccF1gCO 2 eqEVPC90% Baseline0.99090.990929.360.42035350.0 Bray-Curtis (Norm.)0.9962 (0.52%)0.9961 (0.53%)25.2889 (13.86%)0.8453 (101.09%)4 (92%) Cosine (Unst.)0.996 (0.51%)0.996 (0.52%)26.1851 (10.8%)0.6888 (63.87%)6 (88%) Cosine (Stable)0.9953 (0.44%)0.9953 (0.45%)26.4064 (10.05%)0.6974 (65.91%)6 (88%) Mahalanobis (Chol.)0.9938 (0.29%)0.9938 (0.3%)31.9246 (-8.75%)0.9966 (137.09%)1 (98%) Euclidean0.9934 (0.25%)0.9934 (0.25%)24.457 (16.69%)0.9998 (137.84%)1 (98%) at depth/width, likely due to better kernel utilization and reduced softmax/backprop overhead relative to the total compute. PVT (vision transformer). Most distances are again carbon–negative, though the spread is narrower than ResNet50 and a couple of vari- ants hover around parity or slightly positive. Transformers benefit, but less dramatically than deep CNNs. Takeaways. i) On CIFAR-10, distance-based harmonic losses are typically greener per step for CNN/ResNet50/PVT, with the strongest effect on ResNet50; i) MLP shows mixed but mostly fa- vorable outcomes; i) because our accuracy-vs-epoch curves on CIFAR-10 show similar or faster convergence for several distances, these per–step gains are likely to translate into lower end-to-end energy for deep backbones. Practically, we recommend adopting the more frugal distances for convolutional and transformer mod- els and pairing per–step reports with energy-to-target-accuracy to substantiate sustainability claims. L.2.3 CIFAR-100. Figure 15 shows the carbon delta (gCO 2 eq vs. cross-entropy) when training with harmonic-loss distances on CIFAR-100. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY −0.3 −0.2 −0.1 0.0 Mahalanobis (Std.) Chebyshev (Smooth) Bray−Curtis (Norm.) Euclidean Cosine (Unst.) Cosine (Stable) Bray−Curtis (Absolute) Minkowski (p=3.0)Minkowski (p=1.5) Bray−Curtis (Std.) Baseline (C.E.) Mahalanobis (Chol.) Manhattan Hamming (Soft) Mahalanobis (Diag.) Canberra (Std.) Canberra (Robust) Chebyshev (Std.) Canberra (Weighted) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 0.55 gCO2eq MNIST − MLP −0.4 −0.2 0.0 Mahalanobis (Std.) Mahalanobis (Chol.)Mahalanobis (Diag.) Chebyshev (Smooth) Canberra (Robust) Canberra (Weighted) Bray−Curtis (Std.) Minkowski (p=1.5) Cosine (Stable) Cosine (Unst.) Minkowski (p=3.0) Bray−Curtis (Absolute) Bray−Curtis (Norm.) Canberra (Std.) Chebyshev (Std.) Euclidean Hamming (Soft) Baseline (C.E.) Manhattan Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 1.19 gCO2eq MNIST − CNN −20 −10 0 10 20 Mahalanobis (Std.) Manhattan Bray−Curtis (Std.) Canberra (Weighted) Bray−Curtis (Absolute) Cosine (Unst.) Hamming (Soft) Minkowski (p=3.0) Mahalanobis (Diag.) Cosine (Stable) Minkowski (p=1.5) Bray−Curtis (Norm.) Chebyshev (Std.) Baseline (C.E.) Euclidean Mahalanobis (Chol.) Chebyshev (Smooth) Canberra (Std.) Canberra (Robust) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 27.57 gCO2eq MNIST − ResNet50 −1 0 1 2 3 4 Chebyshev (Smooth) Mahalanobis (Std.) Bray−Curtis (Std.) Cosine (Unst.) Baseline (C.E.) Cosine (Stable) Euclidean Minkowski (p=1.5) Canberra (Weighted) Mahalanobis (Diag.)Mahalanobis (Chol.) Bray−Curtis (Norm.) Hamming (Soft) Bray−Curtis (Absolute) Chebyshev (Std.) Manhattan Minkowski (p=3.0) Canberra (Robust) Canberra (Std.) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 2.59 gCO2eq MNIST − PVT Figure 13: Carbon emission differences for MNIST across four model backbones (MLP, CNN, ResNet50, PVT) when replacing cross-entropy with harmonic loss variants. Bars show the emission difference in grams of CO 2 eq relative to the baseline (cross-entropy). Values above zero indicate higher emissions than baseline, while negative values indicate greener, more sustainable outcomes. MLP. Savings are modest and geometry–dependent. Light/stan- dardized variants (e.g., cosine, Euclidean, some Minkowski/Can- berra settings) are carbon–negative, while heavier norms and co- variance–based Mahalanobis parameterizations flip to positive over- heads. On shallow models, extra normalization steps can outweigh gains. CNN. A broad swath of distances are carbon–negative relative to the 1.18 gCO 2 eq baseline; several Mahalanobis and Bray–Curtis settings deliver the largest per–step reductions. A few choices (e.g., certain cosine/Canberra/Minkowski configurations) hover near par- ity or slightly positive, indicating mild architecture sensitivity. ResNet50. The deepest convolutional model exhibits a mixed but wide spread: many distances achieve substantial savings (left cluster of dark-green bars), yet others incur clear premiums (right cluster). Thus, distance choice materially changes footprint at scale. Notably, cosine variants are among the frugal options here, whereas some Chebyshev/Minkowski/Bray–Curtis (absolute) settings are costlier. PVT (vision transformer). Most distances are carbon–positive vs. the 3.67 gCO 2 eq baseline, with only a couple of standardized/s- moothed variants slightly negative. As on MNIST/CIFAR-10, the attention/normalization stack appears less amenable to heavier metric computations. Takeaways. i) On CIFAR-100, harmonic distances can be greener per step for CNNs and selectively for ResNet50, but PVT generally pays a premium; i) cosine tends to be frugal on deeper CNNs (and competitive on MLP), aligning with its strong accuracy dynamics, whereas several Mahalanobis/Minkowski/Chebyshev configura- tions increase emissions unless they deliver clear quality gains; i) because CIFAR-100 accuracy converges differently across distances, claims of sustainability should couple per–step deltas with energy- to-target-accuracy/perplexity. Practically, prefer cosine/Euclidean/s- tandardized Bray–Curtis (and selected Mahalanobis settings that are both stable and frugal) for CNN/ResNet50, and use kernel fusion + mixed precision if heavier geometries are needed on transformer backbones. Insights across datasets: A clear trend emerges across datasets: transformer models (PVT) often incur higher emissions with Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. distance-based harmonic losses, particularly on CIFAR-100 (see Figure 15), whereas convolutional and residual networks (CNN, ResNet50) frequently yield greener outcomes (see results in Figures 13 – 15). The sustainability benefit is especially pronounced when distances incorporate robustness (Hamming-gumbel, Canberra- robust) or covariance awareness (Mahalanobis-diagonal). Simpler datasets like MNIST show limited differences, while CIFAR-10 and CIFAR-100 highlight the greater impact of distance choice on carbon footprint. Cross-architecture insights: MLPs present a limited sustain- ability differences; emissions remain close to baseline across all dis- tances. With CNNs, multiple distances (Hamming-gumbel, Mahalanobis- diagonal, Canberra-weighted) consistently reduce emissions, show- ing CNNs benefit most from harmonic loss efficiency. In PVT, harmonic losses generally increase emissions, especially on CIFAR- 100, highlighting potential overhead in attention-based models. ResNet50 demonstrates an effective integration with several dis- tances (Hamming, Canberra, Bray–Curtis), which achieve signif- icant reductions in emissions over baseline, indicating that deep CNNs can combine effectiveness with sustainability. Overall, the sustainability analysis shows that harmonic losses can improve or degrade carbon efficiency depending on the back- bone and dataset. The choice of distance measure therefore plays a critical role not only in accuracy but also in environmental impact, reinforcing the need for holistic evaluation across the accuracy– sustainability–interpretability triangle. L.3 Language: Sustainability Figure 16 reports per–1k-step carbon differences (gCO 2 eq) when replacing cross-entropy with distance-based harmonic losses for BERT, GPT, and QWEN. Positive bars indicate higher emissions than the cross-entropy baseline (annotated atop each subplot). Overall. Across all three backbones, distance-based losses tend to increase per–1k-step emissions relative to cross-entropy. The magnitude of overhead correlates with the computational com- plexity of the distance: lightweight cosine variants add the least overhead, while Mahalanobis and Minkowski incur the most. BERT. Cosine (simple or temperature-scaled) yields small over- heads (low single-digit gCO 2 eq over a 7.87 gCO 2 eq baseline), sug- gesting that the extra normalization and dot-product operations have modest cost. Euclidean and Bray–Curtis sit mid-pack, whereas Mahalanobis (Cholesky/standard/diagonal) and Minkowski (푝>2) are consistently more carbon intensive per 1k steps. GPT. All distances increase emissions over the 60.36 gCO 2 eq baseline, with a clearer spread: cosine remains the most frugal among alternatives; Euclidean and Manhattan are mid-range; Ma- halanobis (any parameterization) and Minkowski/L2 are the heavi- est. This indicates that the per-step FLOPs and memory traffic of covariance-related computations (and higher-order norms) become more pronounced at GPT scale. QWEN. For this larger model (baseline 75.29 gCO 2 eq), the meth- ods we evaluated (Minkowski/L2 and Euclidean) both raise per–1k- step emissions, with Minkowski/L2 showing a substantial increase. Although the set of distances is smaller here, the pattern mirrors GPT: heavier metrics cost more per step as model width/depth grows. Implications. i) If Green AI considerations are primary, cosine- based harmonic losses are the most promising drop-in replacements, especially on encoder-style models (BERT). i) Mahalanobis and Minkowski should be justified by clear accuracy or stability gains, as they carry the largest per-step carbon premiums. i) Reported values are per–1k-step; end-to-end footprint also depends on steps- to-target-quality. Thus, a distance that reduces time-to-accuracy could still yield net carbon savings even with higher per-step cost. Summary. Distance choice in harmonic loss is not carbon- neutral: cosine variants introduce minimal overhead; Euclidean/Bray– Curtis are moderate; Mahalanobis/Minkowski are expensive. Any claimed performance gains from richer geometries should be weighed against these systematic energy costs, preferably via energy-normalized quality metrics (e.g., accuracy per kWh). L.4 Language: Interpretability Mechanistic and representation-level interpretability of large lan- guage models (LLMs) increasingly leverages the hypothesis that internal activations admit approximately linear structure: many features behave like directions in an activation space, and linear op- erations can steer or probe them [15,26,55]. Within this paradigm, Principal Component Analysis (PCA) is a simple, well-understood lens for: i) summarizing dominant sources of variance in activations; i) stabilizing analyses by denoising; and (i) producing human- auditable axes that can be inspected, correlated with concepts, and tracked over time. Given a layerℓwith residual-stream activations퐻 ℓ ∈ R 푁×푑 col- lected across푁tokens (or prompts), PCA factorizes퐻 ℓ via SVD to yield orthogonal directions푢 푘 푑 푘=1 ordered by explained variance. In practice this supports: (1)Concept probing and visualization. Projections onto top PCs often align with semantically meaningful contrasts; e.g., the first PC of GPT-style embeddings correlated with human well-being judgments in zero-shot tests [17], and per-layer PCA can reconstruct or predict response modes in GPT-2 [31]. (2)Diagnosing and localizing phenomena. Layer-wise or head-wise PCA reveals where variance concentrates, help- ing localize depth at which concepts emerge or consolidate (complementary to linear probing) [29]. Tracking subspace distance across checkpoints detects representational drift during fine-tuning or domain shift. (3)Sanity checks and baselines. With growing interest in sparse autoencoders (SAEs) for monosemantic features [26], PCA serves as a transparent baseline decomposition: if SAEs meaningfully improve sparsity/faithfulness over PCA while matching reconstruction, that strengthens the interpretabil- ity claim [54]. PCA is most compelling under: a) approximately linear feature superposition and b) high signal-to-noise in dominant directions. Toy and empirical studies argue that Transformers often encode many features as directions (superposition) [15], and even simple linear additions to activations can steer model behavior [55]. PCA then becomes an appropriate first-pass tool to: • extract high-variance axes that frequently correlate with coherent features or tasks, Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY •reduce dimensionality before causal tests (e.g., ablate/project- out a PC and re-evaluate behavior), • build compact surrogates (e.g., PCA embeddings for down- stream analyses or compression) [5, 24]. Under widely observed linear-structure assumptions in Trans- former activations, PCA offers an interpretable, testable starting point: it surfaces dominant directions, supports hypothesis genera- tion, and provides quantitative targets for more advanced decom- positions. −0.2 −0.1 0.0 0.1 0.2 Canberra (Weighted) Mahalanobis (Std.) Canberra (Robust) Manhattan Euclidean Mahalanobis (Chol.) Cosine (Stable) Minkowski (p=1.5) Bray−Curtis (Norm.) Baseline (C.E.) Cosine (Unst.) Canberra (Std.) Chebyshev (Std.) Minkowski (p=3.0) Bray−Curtis (Absolute) Bray−Curtis (Std.) Hamming (Soft) Mahalanobis (Diag.) Chebyshev (Smooth) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 0.53 gCO2eq CIFAR10 − MLP −0.6 −0.4 −0.2 0.0 0.2 Mahalanobis (Std.) Hamming (Soft) Canberra (Std.) Bray−Curtis (Std.) Bray−Curtis (Absolute) Mahalanobis (Diag.) Chebyshev (Std.) Canberra (Robust) Canberra (Weighted) Chebyshev (Smooth) Mahalanobis (Chol.) Baseline (C.E.) Euclidean Minkowski (p=1.5) Cosine (Unst.) Bray−Curtis (Norm.) Minkowski (p=3.0) Cosine (Stable) Manhattan Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 1.12 gCO2eq CIFAR10 − CNN −40 −30 −20 −10 0 Mahalanobis (Std.) Canberra (Std.) Bray−Curtis (Std.) Cosine (Unst.) Canberra (Robust) Euclidean Cosine (Stable) Canberra (Weighted) Bray−Curtis (Norm.) Chebyshev (Std.) Manhattan Mahalanobis (Diag.)Mahalanobis (Chol.) Hamming (Soft) Baseline (C.E.) Chebyshev (Smooth) Bray−Curtis (Absolute) Minkowski (p=1.5)Minkowski (p=3.0) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 48.62 gCO2eq CIFAR10 − ResNet50 −3 −2 −1 0 Mahalanobis (Std.) Chebyshev (Std.) Bray−Curtis (Std.) Chebyshev (Smooth) Minkowski (p=3.0) Hamming (Soft) Cosine (Unst.) Bray−Curtis (Absolute) Euclidean Minkowski (p=1.5) Canberra (Robust) Bray−Curtis (Norm.) Mahalanobis (Diag.) Cosine (Stable) Manhattan Baseline (C.E.) Mahalanobis (Chol.) Canberra (Std.) Canberra (Weighted) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 3.71 gCO2eq CIFAR10 − PVT Figure 14: Carbon emission differences for CIFAR10 across four model backbones (MLP, CNN, ResNet50, PVT) when replacing cross-entropy with harmonic loss variants. Bars show the emission difference in grams of CO 2 eq relative to the baseline (cross-entropy). Values above zero indicate higher emissions than baseline, while negative values indi- cate greener, more sustainable outcomes. Conference acronym ’X, June 03–05, 2018, Woodstock, NYTrovato et al. 0.0 0.5 1.0 Cosine (Unst.) Cosine (Stable) Minkowski (p=1.5) Canberra (Std.) Manhattan Euclidean Hamming (Soft) Canberra (Robust) Chebyshev (Std.) Baseline (C.E.) Bray−Curtis (Norm.) Mahalanobis (Std.) Bray−Curtis (Std.) Mahalanobis (Chol.) Chebyshev (Smooth) Minkowski (p=3.0) Mahalanobis (Diag.) Bray−Curtis (Absolute) Canberra (Weighted) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 0.85 gCO2eq CIFAR100 − MLP −0.5 0.0 0.5 1.0 Bray−Curtis (Absolute) Minkowski (p=3.0) Mahalanobis (Diag.) Chebyshev (Smooth) Mahalanobis (Chol.) Mahalanobis (Std.) Chebyshev (Std.) Hamming (Soft) Bray−Curtis (Norm.) Manhattan Bray−Curtis (Std.) Canberra (Weighted) Baseline (C.E.) Canberra (Robust) Euclidean Cosine (Stable) Minkowski (p=1.5) Canberra (Std.) Cosine (Unst.) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 1.18 gCO2eq CIFAR100 − CNN −60 −30 0 30 Mahalanobis (Std.) Bray−Curtis (Std.) Canberra (Robust) Canberra (Std.) Cosine (Unst.) Cosine (Stable) Euclidean Baseline (C.E.) Manhattan Bray−Curtis (Norm.) Hamming (Soft) Bray−Curtis (Absolute) Mahalanobis (Diag.) Minkowski (p=3.0)Minkowski (p=1.5) Chebyshev (Std.) Chebyshev (Smooth) Canberra (Weighted) Mahalanobis (Chol.) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 89.64 gCO2eq CIFAR100 − ResNet50 −2 0 2 4 Chebyshev (Smooth) Bray−Curtis (Std.) Canberra (Std.) Mahalanobis (Std.) Canberra (Robust) Baseline (C.E.) Chebyshev (Std.) Minkowski (p=1.5) Cosine (Stable) Cosine (Unst.) Manhattan Mahalanobis (Diag.) Bray−Curtis (Absolute) Euclidean Bray−Curtis (Norm.) Minkowski (p=3.0) Canberra (Weighted) Mahalanobis (Chol.) Hamming (Soft) Distance Method Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 3.67 gCO2eq CIFAR100 − PVT Figure 15: Carbon emission differences for CIFAR100 across four model backbones (MLP, CNN, ResNet50, PVT) when replacing cross-entropy with harmonic loss variants. Bars show the emission difference in grams of CO 2 eq relative to the baseline (cross-entropy). Values above zero indicate higher emissions than baseline, while negative values indicate greener, more sustainable outcomes. Rethinking the Harmonic Loss via Non-Euclidean Distance LayersConference acronym ’X, June 03–05, 2018, Woodstock, NY 0 5 10 15 20 Baseline Cosine (Simple) Cosine (Temp Scale 0.5)Cosine (Temp Scale 0.1) Manhattan Euclidean Cosine (Temp Scale 1.0) Mahalanobis (Cholesky) Mahalanobis (Diagonal) Mahalanobis (Standard) Minkowski (L2) Distance Method (ordered by emissions) Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 7.87 gCO2eq Average Emissions (gCO2eq) per 1000 steps for BERT Model 0 10 20 30 40 Baseline Cosine (Simple) Cosine (Temp Scale 1.0)Cosine (Temp Scale 0.5)Cosine (Temp Scale 0.1) Mahalanobis (Cholesky) Mahalanobis (Standard) Minkowski (L2) Manhattan Euclidean Mahalanobis (Diagonal) Distance Method (ordered by emissions) Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 60.36 gCO2eq Average Emissions (gCO2eq) per 1000 steps for GPT Model 0 10 20 30 Baseline Minkowski (L2) Euclidean Distance Method (ordered by emissions) Emissions Difference from Baseline (gCO2eq) Difference (gCO2eq) −10 −5 0 5 10 Baseline = 75.29 gCO2eq Average Emissions (gCO2eq) per 1000 steps for QWEN Model Figure 16: Carbon emission differences for LLM pretraining on OpenWebText (BERT, GPT2, QWEN) when replacing cross- entropy with harmonic loss variants. Bars show the emission difference in grams of CO 2 eq relative to the baseline (cross-entropy). Values above zero indicate higher emissions than baseline, while negative values indicate greener, more sustainable outcomes.