Paper deep dive
The Geometry of Language Model Logits: Generic Structure and Learned Departures
Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation \[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.
Tags
Links
- Source: https://arxiv.org/abs/2610.18520v1
- Canonical: https://arxiv.org/abs/2610.18520v1
Trouble viewing inline? Open PDF directly →
Full Text
55,634 characters extracted from source content.
Expand or collapse full text
PriGM The Geometry of Language Model Logits: Generic Structure and Learned DeparturesThanks: Equal Contribution; Correspondence: narashim@stanford.edu Swathi Shree Narashiman Affiliation: Stanford University Aravind Ramana Venkatramanan Affiliation: Indian Institute of Technology Madras Nambi Seshadri Affiliation: University of California San Diego (UCSD) Abstract Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation z(r)≈μ−αlogr.z_(r)≈μ-α r. across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work such as Basu et al. (2021). While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope α: trained language models exhibit α≈1.4α≈ 1.4–1.71.7, sitting well above the initialization baseline (α≈0.2α≈ 0.2) . Moreover, α correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects. 1 Introduction At every prediction step, an autoregressive language model assigns a score to every token in its vocabulary, which governs how probability mass is distributed over possible continuations. Sampling is the last step of generation; the logits are where the real decision is already made. While substantial work has studied the resulting token probabilities, generated text, and hidden representations, the structure of this vocabulary-wide logit vector remains comparatively unexplored. One natural way to expose this structure is to ignore token identities and examine the logits in rank order. This perspective is also directly relevant to common decoding procedures, such as top-k and top-p sampling, which operate on the vocabulary head by selecting or truncating the highest-ranked logits. This leads to a simple question: Does the ordered logit vector possess a simple geometric structure? We empirically find a remarkably regular structure in the vocabulary head: ordered logits follow an approximately log-linear relation across prediction contexts, architectures, and model scales, and this structure emerges during pretraining. This raises the question of how much of this structure is attributable to language models and how much arises generically from order statistics. Controlled order-statistic baselines show that ranking any finite collection of random variables induces its own structure, and light-tailed distributions can produce approximately log-linear ordered spectra. We therefore focus on the spectral slope α, examining how it differs from generic baselines, how it varies across models, and what information it carries about the model’s predictive state. Our contributions are: (i) We characterize a consistent log-linear geometry in the ordered vocabulary-head logits of autoregressive language models across contexts, architectures, scales, and training. (i) We distinguish generic order-statistic geometry from learned structure using controlled distributional baselines, identifying the spectral slope α as the key empirical distinction between the regimes. (i) We show that the spectral distribution can be efficiently reconstructed from the vocabulary head and that α correlates with pre-hoc predictive uncertainty (entropy) and post-hoc prediction quality, measured by NLL, ground-truth token probability ptruep_true, and ground-truth token rank. 2 Preliminaries Autoregressive prediction. At each prediction step, a decoder-only Transformer (Vaswani et al., 2017) maps the preceding context to a hidden state hk∈ℝdh_k ^d and produces a vocabulary-sized logit vector zk=Wuhk+b∈ℝVz_k=W_uh_k+b ^V. The autoregressive factorization and conditional next-token distribution are P(t1:m)=∏k=1mP(tk∣t1:k−1)P(tk=v∣t1:k−1)=exp(zk,v)∑v′exp(zk,v′).P(t_1:m)= _k=1^mP(t_k t_1:k-1) P(t_k=v t_1:k-1)= (z_k,v) _v (z_k,v ). Here, Wu∈ℝV×dW_u ^V× d is the unembedding matrix and b∈ℝVb ^V is the output bias. Inference-time sampling may further rescale zkz_k by a temperature T before softmax (Ackley et al., 1985). Rank-space representation. Because token identities are not relevant to the structural question we aim to answer, we study the vocabulary head in rank space by sorting the logits in descending order: z(1)≥z(2)≥⋯≥z(V),z(r):=r-th largest entry of zk.z_(1)≥ z_(2)≥·s≥ z_(V), z_(r):=$r$-th largest entry of z_k. For a chosen head size K≤VK≤ V, we analyze the ordered spectrum z(r)r=1K\z_(r)\_r=1^K. Relation to Zipfian decoding. Our analysis is closely related to the Zipfian assumption underlying Mirostat (Basu et al., 2021), but addresses a more fundamental structural question. While Mirostat exploits a power-law next-token distribution for adaptive decoding, we characterize the ordered logit spectrum before softmax, ask whether its structure is generic or learned, and show that its two-parameter law can approximately reconstruct the full next-token distribution from a small exact vocabulary head. A detailed comparison is provided in Appendix A. 3 Results 3.1 Log-Linear Logit Geometry Figure 1: Ordered logit geometry. (a) Ordered logit spectrum for a representative prediction context from Llama-3.2-1B, with logits sorted in descending order and plotted against logr r. The red line shows the least-squares fit of z(r)=μ−αlogrz_(r)=μ-α r over the top K=2000K=2000 vocabulary ranks. (b) Collapse of normalized ordered logit spectra across 1,000 randomly sampled prediction contexts from ten books for Llama-3.2-1B. Each spectrum is normalized independently as (z−μ)/σ(z-μ)/σ, and the black curve shows the corpus mean. (c) Mean divergence between the true next-token distribution and its spectral reconstruction using K exact logits and a two-parameter log-linear approximation of the remaining ranks, averaged over four language models. Experimental details and additional results are provided in Appendix B For a prediction context, let z(1)≥⋯≥z(V)z_(1)≥·s≥ z_(V) denote the vocabulary logits sorted in descending order. We empirically observe that the vocabulary head is well approximated by z(r)≈μ−αlogr,r=1,…,K,z_(r)≈μ-α r, r=1,…,K, where μ is a context-dependent intercept and α>0α>0 is the spectral slope. Across the models and contexts studied, this relation provides an excellent fit over the vocabulary head, with R2>0.99R^2>0.99 for a broad range of K. Figure 2: Rank-cutoff robustness of the learned spectral regime. Spectral slope α as a function of rank cutoff K for representative models from four model families. Each language model is compared with Gaussian and Exponential order-statistic baselines matched to its vocabulary size. Error bars denote standard deviations across the 10 books for language models and across synthetic samples for the null distributions. The corresponding dependence of goodness of fit R2R^2 on K is reported in Figure 5 (Appendix C). Figure 1(a) shows that the log-linear structure persists across substantially different prediction contexts, rather than being tied to a particular sequence. Normalization reveals a strong cross-context spectral collapse (Figure 1(b)), suggesting that much of the vocabulary-wide variation is captured by a low-dimensional rank-space structure. Robustness across K, models, and corpora is reported in Appendix C.3. 3.2 Generic vs. Learned Structure The near-perfect fit of ordered logits does not by itself imply language-model-specific structure. We therefore apply the same fitting procedure to controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions. Figure 2 shows that this separation is not specific to the choice of rank cutoff. Across a broad range of K, language models exhibit stable slopes, while the order-statistic baselines display distribution-dependent behavior. Uniform samples fit substantially worse, with average R2≈0.75R^2≈ 0.75 across vocabulary sizes and α≈0.001α≈ 0.001–0.0200.020, while Gaussian and Exponential samples achieve R2≈0.99R^2≈ 0.99 and R2≈0.97R^2≈ 0.97, respectively. Lognormal samples achieve R2≈0.92R^2≈ 0.92 but produce much larger slopes, α≈4.67α≈ 4.67–10.610.6. Thus, although log-linear structure can arise generically from order statistics, its fit and associated slope depend substantially on the underlying distribution. Figure 3: Evolution of ordered logit geometry during pretraining. Mean normalized ordered-logit spectra for Pythia-410M (solid) and Pythia-1B (dashed) across representative training checkpoints. Shaded regions denote the interquartile range across prediction contexts. What shapes the spectral slope α ? The cross-entropy objective provides a partial answer. At the population optimum, zi(h)=logq(i∣h)+c(h)z_i(h)= q(i h)+c(h), making the log-linear logit law equivalent to an approximately Zipfian conditional distribution q(⋅∣h)q(· h) - the same regularity estimated online by Mirostat (Basu et al., 2021). This equivalence does not itself explain why language-model distributions should be Zipfian: it holds exactly only at the unconstrained population optimum. Rather, we view the empirical tightness and stability of α across architectures and scales as the phenomenon requiring explanation. Further equivalences and consequences are given in Appendix G. Evolution during pretraining. Publicly available Pythia checkpoints show that the spectral regime develops during training: α increases from approximately 0.20.2 at initialization, while normalized spectra progressively approach the final-model shape, with most changes occurring early and the structure clearly established by step 512 (Figure 3). Training therefore systematically shapes the spectral parameters within an ordered-logit structure. Additional analyses of the training dynamics are provided in Appendix D. 3.3 Spectral Reconstruction of Next-Token Distributions The log-linear structure also provides a compact approximation of the next-token distribution. Given only the top-K ordered logits, we retain their exact values and fit the spectral parameters (μ,α)(μ,α) by least squares. The remaining vocabulary logits are then reconstructed according to z^(r)=μ^−α^logr z_(r)= μ- α r, yielding an approximate distribution after softmax. Distribution reconstruction. Despite reconstructing the majority of vocabulary logits using only the two fitted spectral parameters, the resulting distributions remain close to the exact softmax distribution, with divergence consistently decreasing as the number of retained exact logits K increases Figure 1(c). At K=500K=500 (≤1%≤ 1\% of the vocabulary), the reconstruction achieves a mean Jensen–Shannon divergence of only 0.0080.008 across four model families. Thus, a small exact head together with a two-parameter spectral description of the remaining tail approximately recovers the full rank-space next token distribution(see Appendix E.) Relation to predictive state. The spectral slope α also varies systematically with the model’s predictive state. Across 40 model-book pairs, larger α is associated with lower uncertainty and better prediction of the realized next token: entropy correlates at r=−0.332r=-0.332, NLL and ground-truth log-rank at r=−0.206r=-0.206 and r=−0.209r=-0.209, and ptruep_true at r=+0.111r=+0.111. These associations are highly consistent across pairs and persist without R2R^2 filtering and under Spearman correlations, suggesting that α captures variation in the predictive state beyond ordered logit geometry (see Appendix F). 4 Discussion Our results reveal two complementary aspects of the ordered logit spectrum. Approximately log-linear geometry can arise generically from order statistics, while training systematically shapes the spectrum through α. This yields a compact rank-space representation of vocabulary-sized predictions: a small exact head and two fitted parameters can approximately reconstruct the ranked probability spectrum. This compact structure may provide a basis for approximating vocabulary-sized predictive distributions without retaining the full spectrum. Moreover, α systematically tracks aspects of the model’s predictive state, providing a compact summary of variation in next-token predictions. Limitations and future work. Our analysis focuses on pretrained decoder-only language models and provides sufficient rather than necessary theoretical conditions. The origin of the learned spectral regime, and in particular why optimization drives the spectrum toward its observed parameter range, remains an open question. Moreover, our reconstruction concerns the ranked probability spectrum rather than token identities in the tail. Future work should examine whether these dynamics extend to other model architectures and whether the spectral functional can support uncertainty estimation, generation-state tracking, distribution approximation or decoding control. References Ackley et al. (1985) D. H. Ackley, G. E. Hinton, and T. J. Sejnowski A learning algorithm for Boltzmann machines. Cognitive Science 9 (1), p. 147–169. Cited by: §2. Basu et al. (2021) S. Basu, G. S. Ramachandran, N. S. Keskar, and L. R. Varshney Mirostat: a neural text decoding algorithm that directly controls perplexity. In International Conference on Learning Representations, Cited by: Appendix A, §G.1, Appendix G, §2, §3.2, Abstract. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, Vol. 202, p. 2397–2430. Cited by: Appendix A, §B.1. Demeter et al. (2020) D. Demeter, G. Kimmel, and D. Downey Stolen probability: a structural weakness of neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 2191–2197. Cited by: Appendix A. Ethayarajh (2019) K. Ethayarajh How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 55–65. Cited by: Appendix A. Gao et al. (2019) J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T. Liu Representation degeneration problem in training natural language generation models. In International Conference on Learning Representations, Cited by: Appendix A. Llama Team, AI @ Meta (2024) Llama Team, AI @ Meta The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §B.1. Mandelbrot (1953) B. Mandelbrot An informational theory of the statistical structure of language. In Communication Theory, W. Jackson (Ed.), p. 486–502. Cited by: Appendix A. Meister and Cotterell (2021) C. Meister and R. Cotterell Language model evaluation beyond perplexity. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 5328–5339. Cited by: Appendix A. Piantadosi (2014) S. T. Piantadosi Zipf’s word frequency law in natural language: a critical review and future directions. Psychonomic Bulletin & Review 21 (5), p. 1112–1130. Cited by: Appendix A. Qwen Team (2024) Qwen Team Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §B.1. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. OpenAI blog. Cited by: §B.1. Takahashi and Tanaka-Ishii (2019) S. Takahashi and K. Tanaka-Ishii Evaluating computational language models with scaling properties of natural language. Computational Linguistics 45 (3), p. 481–513. Cited by: Appendix A. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §2. Yang et al. (2018) Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen Breaking the softmax bottleneck: a high-rank RNN language model. In International Conference on Learning Representations, Cited by: Appendix A. Zipf (1949) G. K. Zipf Human behavior and the principle of least effort. Addison-Wesley. Cited by: Appendix A. Appendix A Related Work Statistical laws of language and generated text. Zipf’s law describes corpus-level token frequencies aggregated over natural language [Zipf, 1949, Mandelbrot, 1953, Piantadosi, 2014]. More recently, several works have investigated whether neural language models reproduce such statistical laws in generated text [Takahashi and Tanaka-Ishii, 2019, Meister and Cotterell, 2021]. Our object of study is different: we analyze the model’s predicted logit vector for each individual prediction context, before any token is sampled. The resulting log-linear law is therefore a pre-hoc property of conditional predictions rather than a post-hoc property of natural or generated text. Structure of the conditional next-token distribution. Several works exploit regularities in the conditional next-token distribution to design decoding algorithms. Most closely related is Mirostat [Basu et al., 2021], which assumes Zipfian statistics of the predicted distribution and estimates the corresponding exponent online to adapt the sampling cutoff and control perplexity. Our work asks a prior structural question: why and how does such a power-law form arise in the first place? Rather than positing Zipfian structure at the probability level for decoding, we directly study the ordered logits before softmax and find that they follow a remarkably tight log-linear law across individual prediction contexts, architectures, and scales. This perspective also separates generic structure induced by sorting from structure learned by the language model. We characterize the observed spectral slope α, study its variation across predictive states, show that the geometry emerges progressively during pretraining, and demonstrate that a small exact vocabulary head together with the fitted spectral tail can approximately reconstruct the full next-token distribution. At the cross-entropy optimum, our logit law is exactly equivalent to a Zipfian conditional distribution, establishing a direct theoretical connection to the probability-level assumption used by Mirostat. Geometry of the output layer. A separate line of work studies geometric properties of transformer outputs, including the softmax bottleneck [Yang et al., 2018], stolen probability [Demeter et al., 2020], representation degeneration [Gao et al., 2019], anisotropy [Ethayarajh, 2019], and low-rank structure in collections of logit vectors across prompts. These perspectives are complementary: while prior work primarily studies structure across contexts or representations, we identify an interpretable within-context geometry of the vocabulary-sized output itself. The resulting two-parameter description provides a compact descriptor of each prediction step. Training dynamics and theoretical characterization. Prior work has largely focused on analyzing trained models. We additionally study the emergence of ordered logit geometry during pretraining using intermediate checkpoints from the Pythia suite [Biderman et al., 2023], showing that the law develops progressively through optimization. On the theoretical side, rather than assuming a particular generative model for the observed spectrum, we characterize its relation to conditional Zipfian structure at the cross-entropy optimum, give an equivalent distance-based reparametrization, and establish deterministic consequences for tail counts and the softmax distribution. Appendix B Experimental Setup B.1 Models We evaluate four families of decoder-only autoregressive language models spanning a range of model scales, vocabularies, and training procedures: GPT-2 (124M, 355M, 774M, 1.5B) [Radford et al., 2019], Pythia (410M, 1B) [Biderman et al., 2023], Llama 3.2 (1B, 3B) [Llama Team, AI @ Meta, 2024], and Qwen 2.5 (0.5B, 1.5B) [Qwen Team, 2024]. To study the emergence of the ordered logit geometry during training, we additionally evaluate intermediate checkpoints of the Pythia models at optimization steps 8,64,512,143000\8,64,512,143000\. All models are evaluated in inference mode without gradient computation. B.2 Corpora Experiments are conducted on ten books from Project Gutenberg 11 1 Gutenberg Corpus covering diverse genres including realism, science fiction, mystery and adventure. Each text is tokenized using the model’s native tokenizer. We use the first 20,000 tokens of each book with a context length of 128 tokens and stride 1, yielding approximately 2×1042× 10^4 prediction contexts per book. For every prediction context, we store the top-K sorted logits (K=2000K=2000).The experimental code is available.22 2 Repository. As an ablation, we additionally verify this collapse on a different model and corpus (GPT-2 on WikiText-2), reported in Appendix C. B.3 Fitting Procedure For each prediction context, we fit the log-linear model z(r)=μ−αlogr,r=1,…,K,z_(r)=μ-α r, r=1,…,K, using ordinary least squares. We report the fitted slope α α, intercept μ μ, coefficient of determination (R2R^2), and root mean squared error (RMSE). Per-model statistics are reported as the mean of per-book means, with standard deviation across books as a measure of corpus stability. For the normalized spectrum analysis, each logit spectrum is standardized as z~(r)=z(r)−z¯σ, z_(r)= z_(r)- zσ, before averaging across contexts. Appendix C Additional Experiments C.1 Functional Form Selection We compare the log-linear form z(r)=μ−αlogrz_(r)=μ-α r against five alternatives on prediction contexts from Llama-3.2 (1B); results are in Table 1. Table 1: Comparison of functional forms fitted to ordered logit spectra (Llama-3.2 1B, K=2000K=2000). # is the number of free parameters. Form # R2R^2 RMSE Logarithmic 2 0.9922±0.00740.9922± 0.0074 0.1299±0.05880.1299± 0.0588 Power 3 0.9962±0.00450.9962± 0.0045 0.0874±0.04370.0874± 0.0437 Exponential 2 0.9460±0.02900.9460± 0.0290 0.3678±0.08490.3678± 0.0849 Square-root 2 0.9043±0.03910.9043± 0.0391 0.4941±0.12980.4941± 0.1298 Linear 2 0.7752±0.05340.7752± 0.0534 0.7692±0.14640.7692± 0.1464 Inverse 2 0.1950±0.04090.1950± 0.0409 1.4642±0.20101.4642± 0.2010 The three-parameter power law z(r)=a⋅r−b+cz_(r)=a· r^-b+c is essentially matched by the two-parameter log-linear form, with its small residual gain in fit attributable to the extra parameter rather than a qualitatively better description. We adopt the log-linear form for two reasons: (1) parsimony: it reaches R2>0.99R^2>0.99 with two parameters, one fewer than the power law; and (2) softmax tractability: it yields a closed-form Zipf law pr∝r−αp_r r^-α, whereas the power law admits no comparably tractable softmax expression. We do not claim log-linear is uniquely correct, only that it is the simplest form consistent with the data and better suited to the softmax-based results in Section 3. C.2 Ablations Figure 4: Cross-corpus normalized logit collapse for GPT-2. Left: GPT-2 evaluated on WikiText-2. Right: GPT-2 evaluated on French Wikipedia. In both panels, blue curves show individual prediction contexts and the black curve shows the corpus mean. The normalized ordered-logit spectra (z−μ)/σ(z-μ)/σ exhibit a shared trajectory across contexts in both corpora. To verify that the rank-logit collapse is not specific to the primary model or corpus, we repeat the analysis with GPT-2 on WikiText-2 and French Wikipedia. As shown in Figure 4, the normalized logit curves (z−μ)/σ(z-μ)/σ collapse onto a shared trajectory across contexts in both corpora, consistent with the pattern observed in our primary experiments. C.3 Additional Results Table 2: Log-linear fit statistics across autoregressive language model families, averaged over 10 books (K=2000K=2000). Reported α is mean ± standard deviation across books; R2R^2 is the mean per-context coefficient of determination, averaged across books. Family Model α R2R^2 GPT-2 124M 1.426±0.2431.426± 0.243 0.9910 355M 1.413±0.2131.413± 0.213 0.9914 774M 1.488±0.2201.488± 0.220 0.9905 1.5B 1.494±0.2291.494± 0.229 0.9898 Pythia 410M 1.649±0.2601.649± 0.260 0.9889 1.0B 1.695±0.2661.695± 0.266 0.9889 Llama-3.2 1.0B 1.629±0.2341.629± 0.234 0.9915 3.0B 1.661±0.2241.661± 0.224 0.9908 Qwen-2.5 0.5B 1.558±0.2291.558± 0.229 0.9918 1.5B 1.660±0.2471.660± 0.247 0.9917 Figure 5: Dependence of goodness of fit on rank cutoff. Coefficient of determination R2R^2 as a function of rank cutoff K for the same representative language models and vocabulary-matched Gaussian and Exponential order-statistic baselines as in Figure 2. Error bars denote standard deviations across the 10 books for language models and across synthetic samples for the null distributions. Consistency Across Model Families . Table 2 summarizes the fitted log-linear parameters across all evaluated models. Despite substantial differences in architecture, tokenizer, training corpus, and more than an order-of-magnitude variation in model size, the ordered logit spectrum is consistently well approximated by a log-linear law (R2>0.99R^2>0.99). More strikingly, the fitted slope remains confined to a narrow range (1.41≤α≤1.701.41≤α≤ 1.70), indicating that modern autoregressive language models exhibit remarkably similar ordered logit geometry despite their diverse training procedures and scales. Dependence of goodness of fit on K. The goodness of fit generally improves as the rank cutoff increases for both language models and order-statistic baselines. Language-model fits approach R2≈0.99R^2≈ 0.99 at larger K, but Gaussian and Exponential baselines attain similarly high or higher R2R^2 over the same range. Thus, the improvement in R2R^2 with K is not specific to language models and should not itself be interpreted as evidence of learned structure. Instead, we use the spectral slope α to distinguish the language-model spectra from generic order-statistic geometry. Appendix D Additional Training Dynamics Evolution of the spectral regime. We further quantify the evolution of the ordered-logit law throughout pretraining using intermediate checkpoints from the Pythia-410M and Pythia-1B suites. Table 3 reports the fitted spectral exponent α and goodness of fit at different training stages. Both models exhibit a substantial change in the spectral regime during the early stages of optimization. At initialization (Step 0), the ordered spectra already admit a highly linear fit, but occupy a shallow regime, with α≈0.20α≈ 0.20 at K=500K=500, closer to the Gaussian synthetic baseline (α≈0.3α≈ 0.3) than to the trained models. This exponent is substantially smaller than that of the trained models, whose final values reach α≈1.57α≈ 1.57–1.621.62 at the same cutoff. Table 3: Evolution of the ordered-logit spectral regime during pretraining. We report the mean fitted exponent α and R2R^2 at K=500K=500 across evaluation contexts for Pythia-410M and Pythia-1B. Pythia-410M Pythia-1B Checkpoint Training stage α R2R^2 α R2R^2 Step 0 Initialization 0.203 0.992 0.199 0.991 Step 8 Early 0.258 0.881 0.241 0.909 Step 64 Early 0.507 0.680 0.779 0.805 Step 512 Intermediate 1.128 0.990 1.164 0.988 Step 8000 Late 1.485 0.987 1.498 0.987 Final Trained 1.571 0.987 1.618 0.986 In the earliest stages of training, the log-linear structure itself undergoes a substantial departure from the final regime Figure 3. For example, at Step 64, the K=500K=500 fit reaches only R2=0.68R^2=0.68 and 0.810.81 for Pythia-410M and Pythia-1B, respectively. As training progresses, both the exponent and goodness of fit move toward their final values. By Step 512, the K=500K=500 exponent reaches 1.131.13 for Pythia-410M and 1.161.16 for Pythia-1B, with R2≈0.99R^2≈ 0.99. At Step 8000, both models are already close to their final spectral regime, with α=1.48α=1.48–1.501.50 and R2≈0.987R^2≈ 0.987. The final checkpoints further sharpen this regime, reaching α=1.57α=1.57 for Pythia-410M and 1.621.62 for Pythia-1B. These dynamics indicate that training does not merely preserve or amplify a generic consequence of ordering. Rather, optimization transforms the initial shallow spectrum through a transient non-log-linear phase into a distinct spectral regime characterized by a substantially larger and increasingly stable exponent. The learned phenomenon is therefore characterized jointly by the emergence of strong log-linearity and a nontrivial spectral scale that is not present in either synthetic or untrained baselines. Appendix E Spectral Approximation of Next-Token Distributions We provide additional details and model-wise results for the spectral reconstruction experiment in Section 3.3. The experiment evaluates whether the ordered logit spectrum can be used to approximately recover the full next-token distribution from a small exact head and a two-parameter description of the remaining ranks. E.1 Spectral Reconstruction For each prediction context, let z(1)≥z(2)≥⋯≥z(V)z_(1)≥ z_(2)≥·s≥ z_(V) denote the vocabulary logits sorted in descending order. Given the first K exact ordered logits, we fit z(r)≈μ−αlogrz_(r)≈μ-α r by ordinary least squares over r=1,…,Kr=1,…,K. We retain the first K logits exactly and reconstruct the remaining ranks as z^(r)=z(r),r≤K,μ^−α^logr,r>K. z_(r)= casesz_(r),&r≤ K,\\ μ- α r,&r>K. cases Applying softmax to the reconstructed spectrum yields the approximate distribution p p, which is compared with the exact distribution p=softmax(z)p=softmax(z). We evaluate reconstruction using Kullback–Leibler (KL) divergence, Jensen–Shannon (JS) divergence, and total variation (TV) distance: DKL(p∥p^),DJS(p,p^),DTV(p,p^).D_KL(p\| p), D_JS(p, p), D_TV(p, p). Results are aggregated over 39,520 prediction contexts sampled from ten books across four language models. We consider K∈100,200,500K∈\100,200,500\. E.2 Model-wise Results Table 4 reports the complete model-wise reconstruction results. Divergence decreases consistently as more exact logits are retained. At K=500K=500, mean JS divergence ranges from 0.00580.0058 to 0.00940.0094 across models, while mean TV ranges from 0.02350.0235 to 0.03690.0369. Table 4: Model-wise spectral reconstruction results. Divergences compare the exact next-token distribution with the distribution reconstructed from the top-K exact ordered logits and a log-linear approximation of the remaining ranks. Model K N Mean R2R^2 Mean KL Mean JS Mean TV Pythia-1B 100 9880 0.9689 0.0805 0.0238 0.0772 200 9880 0.9781 0.0536 0.0163 0.0566 500 9880 0.9860 0.0292 0.0091 0.0348 Qwen2.5-1.5B 100 9880 0.9721 0.0848 0.0249 0.0786 200 9880 0.9803 0.0548 0.0167 0.0562 500 9880 0.9874 0.0300 0.0094 0.0347 GPT-2 XL 100 9880 0.9568 0.0675 0.0197 0.0717 200 9880 0.9686 0.0468 0.0141 0.0543 500 9880 0.9811 0.0285 0.0088 0.0369 Llama-3.2-3B 100 9880 0.9666 0.0595 0.0175 0.0587 200 9880 0.9760 0.0363 0.0110 0.0400 500 9880 0.9847 0.0189 0.0058 0.0235 E.3 Aggregate Results Across all four models, reconstruction quality improves consistently with K. The mean KL divergence decreases from 0.07310.0731 at K=100K=100 to 0.04790.0479 at K=200K=200 and 0.02660.0266 at K=500K=500. Mean JS divergence decreases from 0.02150.0215 to 0.01450.0145 and 0.00830.0083, while mean TV decreases from 0.07150.0715 to 0.05180.0518 and 0.03250.0325, respectively. Table 5: Aggregate spectral reconstruction results across 39,520 contexts and four language models. K Mean KL Mean JS Mean TV Mean R2R^2 100 0.0731 0.0215 0.0715 0.9661 200 0.0479 0.0145 0.0518 0.9758 500 0.0266 0.0083 0.0325 0.9848 Appendix F Predictive-Behavior Analysis Setup. For each prediction context, we sort the vocabulary logits and fit z(r)=μ−αlogr,r=1,…,K,z_(r)=μ-α r, r=1,…,K, with K=2000K=2000 by ordinary least squares. We evaluate GPT-2 XL, Pythia-1B, Llama-3.2-3B, and Qwen-2.5-1.5B on ten Project Gutenberg books per model. Contexts have length 128 and are sampled every 128 tokens, yielding non-overlapping windows. For each context, we record α, R2R^2, NLL, ptruep_true, true-token rank, and full-vocabulary entropy H=−∑v=1Vpvlogpv.H=- _v=1^Vp_v p_v. We use log(truerank) (true\ rank) for the rank analysis. Pearson and Spearman correlations between α and each statistic are computed separately within each of the 40 model–book pairs, then pooled with a DerSimonian-Laird random-effects meta-analysis on Fisher-z transformed correlations (sampling variance 1/(N−3)1/(N-3)), giving pooled correlations, 95% confidence intervals, two-sided p-values, and heterogeneity I2I^2. An exact two-sided binomial sign test over the 40 book-level correlations is included as a distribution-free check, and we also repeat the sign test at the level of the four model-family averages. We report results both with a fit-quality filter (R2≥0.95R^2≥ 0.95) and without one (R2≥0R^2≥ 0). Correlation results. Table 6 reports the pooled Pearson and Spearman correlations between α and each predictive statistic, together with the sign-test summary. Table 6: Predictive-behavior analysis across 40 model–book pairs. Entries are random-effects pooled Pearson and Spearman correlations with 95% confidence intervals. The Sign column reports the percentage of book-level correlations with negative/positive sign. For the Pearson meta-analysis, p<1e−12p<1e\!-\!12 for all statistics except ptruep_true, for which p=2.02e−11p=2.02e\!-\!11 at R2≥0.95R^2≥ 0.95 and p=1.36e−12p=1.36e\!-\!12 at R2≥0R^2≥ 0. Exact two-sided binomial sign-test p-values are given in parentheses. R2R^2 Statistic Pearson Sign Spearman r 95% CI ρ 95% CI ≥0.95≥ 0.95 Entropy −0.332-0.332 [−0.370,−0.293][-0.370,-0.293] −%-100\% (1.82e−12)(1.82e\!-\!12) −0.290-0.290 [−0.331,−0.248][-0.331,-0.248] NLL −0.206-0.206 [−0.236,−0.175][-0.236,-0.175] −97.5%-97.5\% (7.46e−11)(7.46e\!-\!11) −0.159-0.159 [−0.193,−0.125][-0.193,-0.125] ptruep_true +0.111+0.111 [+0.079,+0.143][+0.079,+0.143] +87.5%+87.5\% (1.38e−06)(1.38e\!-\!06) +0.159+0.159 [+0.125,+0.193][+0.125,+0.193] Log-rank −0.209-0.209 [−0.236,−0.181][-0.236,-0.181] −97.5%-97.5\% (7.46e−11)(7.46e\!-\!11) −0.146-0.146 [−0.176,−0.116][-0.176,-0.116] ≥0≥ 0 Entropy −0.290-0.290 [−0.334,−0.245][-0.334,-0.245] −%-100\% (1.82e−12)(1.82e\!-\!12) −0.262-0.262 [−0.304,−0.218][-0.304,-0.218] NLL −0.244-0.244 [−0.287,−0.201][-0.287,-0.201] −97.5%-97.5\% (7.46e−11)(7.46e\!-\!11) −0.170-0.170 [−0.206,−0.134][-0.206,-0.134] ptruep_true +0.116+0.116 [+0.084,+0.148][+0.084,+0.148] +87.5%+87.5\% (1.38e−06)(1.38e\!-\!06) +0.170+0.170 [+0.134,+0.206][+0.134,+0.206] Log-rank −0.232-0.232 [−0.265,−0.199][-0.265,-0.199] −97.5%-97.5\% (7.46e−11)(7.46e\!-\!11) −0.159-0.159 [−0.191,−0.127][-0.191,-0.127] Heterogeneity across model–book pairs. Table 7 reports the I2I^2 heterogeneity statistic for each pooled Pearson correlation in Table 6. I2I^2 is moderate for NLL and ptruep_true and higher for entropy, indicating that the correlation with entropy varies more across model–book pairs in magnitude, even though Table 6 shows its sign is consistent for all 40 pairs. Table 7: Heterogeneity (I2I^2) of the pooled Pearson correlations reported in Table 6, by fit-quality filter. R2R^2 filter Entropy NLL ptruep_true Log-rank ≥0.95≥ 0.95 66.6%66.6\% 37.3%37.3\% 40.3%40.3\% 24.7%24.7\% ≥0≥ 0 73.7%73.7\% 69.7%69.7\% 39.5%39.5\% 48.9%48.9\% Robustness to the fit-quality filter. The R2≥0.95R^2≥ 0.95 filter excludes a different fraction of contexts in each model–book pair; we call this the pair’s rejection fraction and ask whether it is itself associated with that pair’s average predictive statistics, which would indicate the filter systematically selects easier or harder contexts. Table 8 shows none of the four correlations is significant, so we find no evidence of such a bias. Table 8: Pearson correlation between each model–book pair’s R2≥0.95R^2≥ 0.95 rejection fraction and its mean predictive statistics (N=40N=40 pairs). None of the four associations is significant at α=0.05α=0.05. Statistic r p Entropy 0.2270.227 0.1590.159 NLL 0.2100.210 0.1940.194 ptruep_true −0.066-0.066 0.6840.684 Log-rank 0.1890.189 0.2440.244 Appendix G Theoretical Characterization This appendix collects the formal statements referenced in Section 3. Section G.1 characterizes the relationship between the log-linear logit law and an approximately Zipfian conditional next-token distribution and clarifies what this relationship does and does not establish. Sections G.2 and G.3 derive two exact consequences of the empirical law, once it holds for a given context: an equivalent exponential tail count (Prop. 1) and a conditional Zipf distribution with exact temperature rescaling (Prop. 2), which together formalize the assumption underlying Mirostat [Basu et al., 2021]. Section G.4 formalizes the normalized-collapse behavior shown in Figure 1(b) (Prop. 3) and bounds its robustness to residual fit error. We distinguish two questions. The first concerns the consequences of the vocabulary head being approximately log-linear in log-rank. These consequences follow algebraically and do not depend on the origin of the observed law. The second concerns why a language model’s logits exhibit this form in the first place. Our analysis provides only a partial account of this second question. Fix a prediction context h and write the vocabulary logits as zi(h)=wi⊤h+bi,i∈,z_i(h)=w_i h+b_i, i , where wiw_i is the output embedding row for token i, bib_i is an output bias when present, and V=||V=|V|. We suppress the dependence on h when clear, and write z(1)≥z(2)≥⋯≥z(V)z_(1)≥ z_(2)≥·s≥ z_(V) for the logits sorted in descending order. G.1 Relation to a Zipfian conditional distribution An exact equivalence at the cross-entropy optimum. At the population optimum of the cross-entropy objective, the softmax probabilities for a fixed context h equal the true conditional distribution q(⋅∣h)q(· h), so the logits satisfy zi(h)=logq(i∣h)+c(h)z_i(h)= q(i h)+c(h) for an arbitrary context-dependent constant c(h)c(h) (adding a constant to every logit of a context leaves the softmax unchanged). Sorting both sides by rank, z(r)(h)=logq(r)(h)+c(h),z_(r)(h)= q_(r)(h)+c(h), where q(r)(h)q_(r)(h) is the r-th largest value of q(⋅∣h)q(· h). Thus, on the vocabulary head, the empirical law z(r)≈μ−αlogrz_(r)≈μ-α r holds exactly when the conditional next-token distribution is approximately Zipfian in rank, q(r)(h)∝r−αq_(r)(h) r^-α. This is a change of variables rather than a new assumption: log-linear logits and an approximately Zipfian q(⋅∣h)q(· h) express the same claim, provided the model is near its cross-entropy optimum. What this equivalence does, and does not, explain. This identity does not by itself explain why q(⋅∣h)q(· h) should be close to Zipfian; it instead transfers the same regularity from logits to probabilities, where related structure has been conjectured and used previously, most directly in Mirostat’s online Zipf-exponent estimate [Basu et al., 2021]. Two points distinguish this from a purely definitional restatement of what we measure. First, the identity above is exact only at the unconstrained population optimum, while a finite model trained on finite data with a fixed architecture and a low-rank output embedding is not guaranteed to attain that optimum. The observation that trained models realize this structure with R2>0.99R^2>0.99, with α confined to a narrow band across four model families and an order of magnitude of scale (Section 3), is therefore an empirical property of trained models rather than a consequence of the training objective alone. It is this tightness and stability, rather than the identity itself, that requires explanation. Second, we measure the structure directly in logit space, before any token is sampled and independently of any decoding rule, which differs from the post-hoc statistics of generated text studied in prior work on statistical laws of language (Appendix A). An equivalent distance-based parameterization. We provide an equivalent representation not necessarily a real-world representation, but in terms of a context-dependent mismatch, or distance, dh(i)d_h(i) between the context and a candidate token, rather than directly in terms of q(⋅∣h)q(· h). The two descriptions are equivalent by construction: for any γh>0 _h>0, writing q(i∣h)∝exp(−γhdh(i))q(i h) (- _hd_h(i)) simply defines dh(i):=−γh−1logq(i∣h)+const.d_h(i):=- _h^-1 q(i h)+const. Under this definition, the exponential-volume condition Gh(t):=#i∈:dh(i)≤t≈eλhtG_h(t):=\#\i :d_h(i)≤ t\\;≈\;e _ht is exactly the statement that q(⋅∣h)q(· h) is approximately Zipfian in rank. We consider the resulting rank-distance relation as an alternative interpretation of the observed log-linear law and to express temperature rescaling and tail counts (Sections G.2 and G.3) in distance language as well. It is not intended as an additional result beyond the equivalence above. Neither the exponential-volume condition nor the linear-penalty form z≈β−γdz≈β-γ d below is independently verified in our data. Theorem 1 (rank-distance form of the log-linear law). Fix a context h, and let Dh(r)D_h(r) denote the r-th smallest value of dh(i)i∈\d_h(i)\_i . Suppose that for ranks 1≤r≤K1≤ r≤ K, |Dh(r)−logr−νhλh|≤ηhλh,λh>0, |D_h(r)- r- _h _h |≤ _h _h, _h>0, and that the logits are uniformly close to a linear penalty in distance: maxi∈|zi(h)−(βh−γhdh(i))| _i |z_i(h)- ( _h- _hd_h(i) ) | ≤εh, ≤ _h, γh _h >0. >0. Then, for r=1,…,Kr=1,…,K, |z(r)(h)−(μh−αhlogr)|≤εh+αhηh, |z_(r)(h)- ( _h- _h r ) |≤ _h+ _h _h, where αh=γhλh,μh=βh+αhνh. _h= _h _h, _h= _h+ _h _h. Proof. Define the ideal score yi(h)=βh−γhdh(i).y_i(h)= _h- _hd_h(i). Because γh>0 _h>0, the ordering of yi(h)y_i(h) is the reverse of the ordering of dh(i)d_h(i). Therefore, the r-th largest ideal score is y(r)(h)=βh−γhDh(r).y_(r)(h)= _h- _hD_h(r). Using the assumed rank-distance bound, |y(r)(h)−(βh+γhνhλh−γhλhlogr)|≤γhλhηh. |y_(r)(h)- ( _h+ _h _h _h- _h _h r ) |≤ _h _h _h. With αh=γh/λh _h= _h/ _h and μh=βh+αhνh _h= _h+ _h _h, this becomes |y(r)(h)−(μh−αhlogr)|≤αhηh. |y_(r)(h)- ( _h- _h r ) |≤ _h _h. It remains to compare the order statistics of zi(h)z_i(h) and yi(h).y_i(h). The uniform perturbation assumption gives maxi∈|zi(h)−yi(h)|≤εh. _i |z_i(h)-y_i(h)|≤ _h. Order statistics are 11-Lipschitz under the sup norm: if maxi|ai−bi|≤ε _i|a_i-b_i|≤ , then |a(r)−b(r)|≤ε|a_(r)-b_(r)|≤ for every rank r. Hence, |z(r)(h)−y(r)(h)|≤εh.|z_(r)(h)-y_(r)(h)|≤ _h. Combining the two inequalities yields |z(r)(h)−(μh−αhlogr)|≤εh+αhηh.∎ |z_(r)(h)- ( _h- _h r ) |≤ _h+ _h _h. The rank–distance bound in Theorem 1 is the exponential-volume condition expressed in rank form. Define Gh(t)=#i∈:dh(i)≤t.G_h(t)=\#\i :d_h(i)≤ t\. If, over the relevant head range, |logGh(t)−(νh+λht)|≤ηh, | G_h(t)- ( _h+ _ht ) |≤ _h, then solving Gh(t)≈rG_h(t)≈ r gives Dh(r)≈logr−νhλh,D_h(r)≈ r- _h _h, which is the hypothesis of Theorem 1. The next paragraph establishes this implication directly. Under this parameterization, the fitted slope decomposes as αh=γhλh=logit penalty per unit distancedistance-volume growth rate, _h= _h _h= logit penalty per unit distancedistance-volume growth rate, which is the same Zipf exponent α already defined in the main text, expressed in terms of a penalty and a growth rate rather than as two separately measurable quantities. Intuitively, dh(i)d_h(i) can be viewed as how well a candidate token fits what the preceding context calls for. Within this parameterization, a larger αh _h corresponds to a more selective context, concentrating probability on a smaller set of plausible continuations and naturally yielding a sharper candidate set under nucleus truncation. Checking the relabeling. Let Gh(t)=#i∈:dh(i)≤tG_h(t)=\#\i :d_h(i)≤ t\ and Dh(r)=inft:Gh(t)≥rD_h(r)= \t:G_h(t)≥ r\. Assume that over the relevant distance range, |logGh(t)−(νh+λht)|≤ηh,λh>0. | G_h(t)- ( _h+ _ht ) |≤ _h, _h>0. For t+(r)=logr−νh+ηhλh,t_+(r)= r- _h+ _h _h, we have logGh(t+(r))≥νh+λht+(r)−ηh=logr, G_h(t_+(r))≥ _h+ _ht_+(r)- _h= r, so Gh(t+(r))≥rG_h(t_+(r))≥ r and therefore Dh(r)≤t+(r)D_h(r)≤ t_+(r). Similarly, for any t<t−(r)=logr−νh−ηhλh,t<t_-(r)= r- _h- _h _h, we have logGh(t)≤νh+λht+ηh<logr, G_h(t)≤ _h+ _ht+ _h< r, so Gh(t)<rG_h(t)<r and hence Dh(r)≥tD_h(r)≥ t. Letting t↑t−(r)t t_-(r) gives Dh(r)≥t−(r)D_h(r)≥ t_-(r). Combining the two bounds, |Dh(r)−logr−νhλh|≤ηhλh. |D_h(r)- r- _h _h |≤ _h _h. G.2 Tail-count form of the rank law The preceding section concerns the possible origin of the empirical law. Here we characterize its consequences for a fixed logit vector, independently of its origin. Define the empirical upper-tail count N(x)=#i∈:zi≥x.N(x)=\#\i :z_i≥ x\. For distinct top-K logits, N(z(r))=rN(z_(r))=r. Proposition 1 (tail-count characterization). Assume the top-K logits are distinct and z(r) z_(r) =μ−αlogr+εr, =μ-α r+ _r, α α >0,r=1,…,K. >0, r=1,…,K. Then, at x=z(r)x=z_(r), logN(x)=μ−xα+εrα. N(x)= μ-xα+ _rα. Conversely, if for all r≤Kr≤ K, |logN(z(r))−μ−z(r)α|≤η, | N(z_(r))- μ-z_(r)α |≤η, then |z(r)−(μ−αlogr)|≤αη. |z_(r)- (μ-α r ) |≤αη. Proof. For distinct top-K logits, N(z(r))=rN(z_(r))=r. If z(r)=μ−αlogr+εrz_(r)=μ-α r+ _r, then logr=μ−z(r)α+εrα. r= μ-z_(r)α+ _rα. Substituting logN(z(r))=logr N(z_(r))= r gives logN(z(r))=μ−z(r)α+εrα. N(z_(r))= μ-z_(r)α+ _rα. Conversely, if |logN(z(r))−μ−z(r)α|≤η | N(z_(r))- μ-z_(r)α |≤η, then using N(z(r))=rN(z_(r))=r gives |logr−μ−z(r)α|≤η. | r- μ-z_(r)α |≤η. Multiplying by α and rearranging gives |z(r)−(μ−αlogr)|≤αη.∎ |z_(r)- (μ-α r ) |≤αη. Thus, on the vocabulary head, the log-linear law is equivalent to an approximately exponential empirical upper tail: N(x)≈exp(μ−xα).N(x)≈ \! ( μ-xα ). This statement is deterministic for each context and does not require treating the logits as independent random variables. G.3 Softmax consequence: a conditional Zipf law The log-linear law has a direct consequence for the model’s predictive distribution. Let p(r)=expz(r)∑s=1Vexpz(s)p_(r)= z_(r) _s=1^V z_(s) be the full softmax probability of the token at rank r. Since the empirical law is fitted over the top K logits, define the top-K probability mass MK=∑s=1Kp(s)M_K= _s=1^Kp_(s) and the conditional distribution within the vocabulary head qr(K)=p(r)MK=expz(r)∑s=1Kexpz(s).q_r^(K)= p_(r)M_K= z_(r) _s=1^K z_(s). Proposition 2 (conditional Zipf law). Suppose z(r) z_(r) =μ−αlogr+εr, =μ-α r+ _r, |εr| | _r| ≤δ,r=1,…,K. ≤δ, r=1,…,K. Let HK,α=∑s=1Ks−α.H_K,α= _s=1^Ks^-α. Then e−2δr−αHK,α≤qr(K)≤e2δr−αHK,α.e^-2δ r^-αH_K,α≤ q_r^(K)≤ e^2δ r^-αH_K,α. Equivalently, MKe−2δr−αHK,α≤p(r)≤MKe2δr−αHK,α.M_Ke^-2δ r^-αH_K,α≤ p_(r)≤ M_Ke^2δ r^-αH_K,α. Proof. For each r≤Kr≤ K, expz(r)=eμr−αeεr. z_(r)=e^μr^-αe _r. Since |εr|≤δ| _r|≤δ, eμe−δr−α≤expz(r)≤eμeδr−α.e^μe^-δr^-α≤ z_(r)≤ e^μe^δr^-α. Summing over s=1,…,Ks=1,…,K gives eμe−δHK,α≤∑s=1Kexpz(s)≤eμeδHK,α.e^μe^-δH_K,α≤ _s=1^K z_(s)≤ e^μe^δH_K,α. Dividing the pointwise bounds by the denominator bounds yields e−2δr−αHK,α≤qr(K)≤e2δr−αHK,α.e^-2δ r^-αH_K,α≤ q_r^(K)≤ e^2δ r^-αH_K,α. The bound for p(r)p_(r) follows from p(r)=MKqr(K)p_(r)=M_Kq_r^(K). ∎ Therefore, the log-linear logit law implies an approximate Zipf distribution for the model’s context-dependent next-token probabilities, restricted to the vocabulary head. This differs from corpus-level Zipf laws: here the distribution is produced by the model before sampling and varies with the context. Temperature scaling follows directly. At temperature T>0T>0, z(r)T=μT−αTlogr+εrT. z_(r)T= μT- αT r+ _rT. Thus, in the exact case, temperature rescales the exponent as αT=αT. _T= αT. Lower temperatures sharpen the head distribution, while higher temperatures flatten it. G.4 Normalized spectrum collapse Figure 1(b) shows this collapse, and Section 3 describes it empirically; here we state it formally. The collapse is a direct algebraic consequence of z-scoring an affine function of logr r: any two exactly log-linear spectra collapse onto the same normalized curve regardless of their μ and α. Thus, the collapse itself provides no evidence for cross-context universality beyond the quality of the affine fit in each context. The empirical claim is instead that individual contexts are well described by that affine form with a stable slope (Section 3). The stability result below quantifies how far the normalized collapse can deviate from its idealized form when the log-linear fit is approximate. Proposition 3 (normalized spectrum collapse). If z(r)=μ−αlogrz_(r)=μ-α r for r=1,…,Kr=1,…,K, then the standardized spectrum z~(r)=(z(r)−z¯K)/σz,K z_(r)=(z_(r)- z_K)/ _z,K satisfies z~(r)=ℓ¯K−logrsℓ,K, z_(r)= _K- rs_ ,K, where ℓr=logr _r= r, and ℓ¯K _K, sℓ,Ks_ ,K are the mean and standard deviation of ℓrr=1K\ _r\_r=1^K, independent of μ and α. Proof. Let ℓr=logr _r= r. If z(r)=μ−αℓr,z_(r)=μ-α _r, then z¯K=μ−αℓ¯K. z_K=μ-α _K. Therefore z(r)−z¯K=α(ℓ¯K−ℓr).z_(r)- z_K=α( _K- _r). Also, σz,K2=1K∑r=1Kα2(ℓr−ℓ¯K)2=α2sℓ,K2. _z,K^2= 1K _r=1^Kα^2( _r- _K)^2=α^2s_ ,K^2. Since α>0α>0, σz,K=αsℓ,K _z,K=α s_ ,K. Hence z~(r)=z(r)−z¯Kσz,K=ℓ¯K−logrsℓ,K.∎ z_(r)= z_(r)- z_K _z,K= _K- rs_ ,K. Stability of the normalized collapse In real contexts, the log-linear law is only approximate, so we consider the effect of residual fit error on the normalized spectrum. Let z(r)=μ−αℓr+εr,z_(r)=μ-α _r+ _r, and define the empirical norm ‖x‖K=(1K∑r=1Kxr2)1/2.\|x\|_K= ( 1K _r=1^Kx_r^2 )^1/2. Let ar=ℓ¯K−ℓr,ϕK(r)=ar‖a‖K,a_r= _K- _r, _K(r)= a_r\|a\|_K, so ‖a‖K=sℓ,K\|a\|_K=s_ ,K. Centering removes μ and the mean residual. With er=εr−1K∑s=1Kεs,e_r= _r- 1K _s=1^K _s, the centered spectrum is z(r)−z¯K=αar+er.z_(r)- z_K=α a_r+e_r. If ρ=‖e‖Kαsℓ,K<1,ρ= \|e\|_Kα s_ ,K<1, then by the reverse triangle inequality, ‖αa+e‖K≥αsℓ,K(1−ρ).\|α a+e\|_K≥α s_ ,K(1-ρ). Using ‖u+v‖u+v‖−u‖u‖≤2‖v‖u+v‖ \| u+v\|u+v\|- u\|u\| \|≤ 2\|v\|\|u+v\| with u=αau=α a and v=ev=e gives ‖z~−ϕK‖K≤2ρ1−ρ. \| z- _K \|_K≤ 2ρ1-ρ. If |εr|≤δ| _r|≤δ for all r≤Kr≤ K, then ‖e‖K≤δ\|e\|_K≤δ, so whenever δ<αsℓ,Kδ<α s_ ,K, ‖z~−ϕK‖K≤2δαsℓ,K−δ. \| z- _K \|_K≤ 2δα s_ ,K-δ.