Paper deep dive
The Geometric Anatomy of Capability Acquisition in Transformers
Jayadev Billa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 2:32:58 AM
Summary
This paper investigates the temporal relationship between geometric changes in neural network representations and the acquisition of behavioral capabilities in Transformer models. Using six transformer sizes and eight algorithmic tasks, the authors find that representations first collapse to a low-dimensional state before recovering, with behavioral performance improving only after this recovery. The geometric measure 'RankMe' is identified as a reliable precursor for hard tasks, where a distinct gap exists between geometric change and capability acquisition. For easy tasks, learning is too rapid for such a precursor to be detected. The study also highlights that collapse floors are task-specific and propagate top-down through the network layers.
Entities (8)
Relation Signals (6)
RankMe → precedes → Capability Acquisition
confidence 95% · RankMe reliably precedes capability acquisition for hard tasks.
Representation Collapse → precedes → Behavioral Improvement
confidence 93% · representations first collapse to a low-dimensional state, then recover, and only then does behavioral performance improve.
RankMe → measures → Representation Dimensionality
confidence 92% · RankMe... measures the effective dimensionality of the representation space
Linear Probe → detects → Task-Relevant Information
confidence 90% · Linear probes show that the model's hidden states already contain task-relevant information before the model can act on it.
Task Difficulty → influences → Precursor Detectability
confidence 88% · Whether this precursor is detectable depends on task difficulty relative to model capacity.
Pythia → validates → Geometric Patterns
confidence 85% · This suggests that geometric patterns observed in small proxy models can persist at larger scale when the task remains difficult relative to model capacity.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural networks gain capabilities during training, but the internal changes that precede capability acquisition are not well understood. In particular, the relationship between geometric change and behavioral change, and the effect of task difficulty and model scale on that relationship, is unclear. We track geometric measures and linear probes across six transformer sizes (405K--151M parameters), eight algorithmic tasks (144 task$\times$level$\times$model combinations), and three Pythia language models (160M--2.8B). Across all settings, representations first collapse to a low-dimensional state, then recover, and only then does behavioral performance improve. Linear probes show that the model's hidden states already contain task-relevant information before the model can act on it. The collapse floor is task-specific, the collapse propagates top-down through the network, and of the geometric measures tested, only \rankme reliably precedes capability acquisition for hard tasks. Whether this precursor is detectable depends on task difficulty relative to model capacity. For hard tasks, there is a clear gap: geometry changes first, behavior follows. For easy tasks, the model learns so quickly that both happen simultaneously and no precursor is detectable. On Pythia-2.8B, a logical deduction task that is genuinely hard for the model shows a precursor gap of ${\sim}$49K training steps, while easy benchmarks show none. This suggests that geometric patterns observed in small proxy models can persist at larger scale when the task remains difficult relative to model capacity.
Tags
Links
- Source: https://arxiv.org/abs/2602.15997v4
- Canonical: https://arxiv.org/abs/2602.15997v4
Trouble viewing inline? Open PDF directly →
Full Text
57,676 characters extracted from source content.
Expand or collapse full text
The Geometric Anatomy of Capability Acquisition in Trans- formers Jayadev Billa † San Jose, CA, USA jbilla2004@gmail.com Abstract Neural networks gain capabilities during training, but the internal changes that precede capability acquisition are not well understood. In particular, the relationship between geometric change and behavioral change, and the effect of task difficulty and model scale on that relationship, is unclear. We track geometric measures and linear probes across six transformer sizes (405K–151M parameters), eight algorithmic tasks (144 task×level×model combinations), and three Pythia language models (160M–2.8B). Across all settings, represen- tations first collapse to a low-dimensional state, then recover, and only then does behavioral performance improve. Linear probes show that the model’s hidden states already contain task-relevant information before the model can act on it. The collapse floor is task-specific, the collapse propagates top-down through the network, and of the geometric measures tested, only RankMe reliably precedes capability acquisition for hard tasks. Whether this precursor is detectable depends on task difficulty relative to model capacity. For hard tasks, there is a clear gap: geometry changes first, behavior follows. For easy tasks, the model learns so quickly that both happen simultaneously and no precursor is detectable. On Pythia-2.8B, a logical deduction task that is genuinely hard for the model shows a precursor gap of∼49K training steps, while easy benchmarks show none. This suggests that geometric patterns observed in small proxy models can persist at larger scale when the task remains difficult relative to model capacity. 1 Introduction Neural networks acquire new capabilities during training. Recent work offers some insight into the internal changes leading to each new capability. Representation geometry studies have identified distinct developmental stages (collapse, expansion, compression) (Li et al., 2025). Theoretical analysis of attention training has revealed a two-stage process: condensation, followed by rank collapse (Chen & Luo, 2025). The grokking literature has identified a transition from memorization to generalization, manifesting as competition between loss basins of different complexity (Nanda et al., 2023; Cullen et al., 2026). Singular learning theory has introduced the local learning coefficient (LLC) as a proxy for progress (Watanabe, 2009; Lau et al., 2025; Hoogland et al., 2024). What remains unclear is the temporal relationship between these changes and capability acquisition, and whether their relative ordering holds across tasks and model sizes. If geometric changes reliably precede capability acquisition, they could provide a basis for monitoring or intervention. To better understand the temporal relationship between geometric changes and capability acquisition, we create a controlled testbed, consisting of six decoder-only transformer models (405K to 151M parameters), eight algorithmic tasks at three levels of difficulty, and dense checkpointing (206 to 256 per run), resulting in 144 task×level×model combinations. At each checkpoint, we measure RankMe, and at select checkpoints, primarily nano-scale, four other geometric measures (gradient effective rank, LLC, Hessian eigenvalues, gradient covariance rank) and train per-task linear probes on the correct output token. We define capability acquisition as the first sustained crossing of 50% accuracy (remaining above for three consecutive checkpoints). † Unaffiliated researcher - previously at ISI@USC, Yahoo, Nuance, BBN. 1 arXiv:2602.15997v4 [cs.LG] 2 Apr 2026 This definition is simply a practical choice and does make a claim about a phase transition. To confirm that patterns hold across larger models scales we test on three Pythia language models (160M, 410M, 2.8B). A single recurring pattern ties the results together. Consistent with the developmental phases identified by Li et al. (2025), representations collapse to a low dimension and then recover, and only then does behavioral accuracy increase. Linear probes confirm that at checkpoints where the model cannot yet perform the task, a trained linear probe on the hidden states can already extract the correct output token. We find that conditioning on individual tasks reveals structure that global analysis does not capture: 1.The collapse floor is task-specific (§4.2). Modular arithmetic collapses to RankMe≈2.0 across a 370× parameter range (CV = 0.16); multiplication floors rise with model capacity (CV = 0.35). The floor appears to reflect the minimum dimensionality the task requires. 2.The collapse is top-down (§4.3). Output-facing layers collapse most, early layers least, in all 32/32 task×model combinations tested, consistent with the finding that outer-layer parameters reorganize first (Chen & Luo, 2025), and not consistent with the assumption that features build bottom-up from simple to complex. Probing at a layer level confirms that hidden learning is concentrated in the same deep layers where collapse is strongest. 3.RankMe is the only reliable precursor (§4.5). We call a geometric measure a precursor if it shows a discrete transition before the model acquires the capability. Across the task-specific measures tested, RankMe provides the cleanest signal: it precedes capability acquisition for all hard tasks at all tested scales (100% precursor rate). Task-specific Hessian and gradient covariance also reach 100% at nano scale but are too noisy to serve as reliable monitors. Gradient effective rank transitions too late for most tasks (38%). LLC shows no discrete precursor event, consistent with analyses where LLC tracks rather than predicts transitions (Cullen et al., 2026). 4.Precursor detectability depends on task difficulty relative to model capacity (§4.5). For tasks that are hard relative to the model’s capacity, RankMe precedes acquisition at every scale tested (100%). Easy tasks are acquired during the collapse itself, so no temporal gap exists and no precursor is detectable. On Pythia-2.8B, a logical deduction task that is genuinely hard for the model (50% accuracy at step 143K) shows a geometric precursor gap of∼49K steps, while easy benchmarks on the same model show none (§4.6). Task-specific RankMe tells you whether a particular capability is coming, but not when relative to other tasks. The geometric dynamics themselves, however, are scale-invariant: collapse floors, phase boundaries, and layer propagation patterns observed at 405K parameters correctly predict the dynamics at 151M and on Pythia across a 17.5×scale gap (ρ >0.92 between Pythia-160M and 2.8B at 71/72 training steps). A small proxy model provides the geometric roadmap for a large training run (§5). 2 Related Work Representation geometry during training. Li et al. (2025) study the eigenspectra of hidden-state covariance matrices across Pythia training and identify three developmental phases: collapse, expansion and compression. We reproduce this three stage pattern here on Pythia, but focus on a different but related question: do the geometry changes precede capability acquisition, which Li et al. (2025) do not test. Our task-conditioned study reveals patterns (such as task-dependent floors and top-down propagation) not visible in their global approach. Chen & Luo (2025) provide a theoretical account in which they show that outer layer parameters reorganize first while attention parameters remain static. The per-layer RankMe analysis presented here is consistent with this outer layer reorganization and further shows that collapse starts at the output layer and propagates inward. Belrose et al. (2024) show that models learn statistics of increasing complexity over training, which is also consistent with our observation that representational complexity increases after the initial collapse. Loss landscape geometry. The Hessian eigenspectrum has been shown to develop a bulk-plus-outlier structure during training (Ghorbani et al., 2019; Sagun et al., 2018; Papyan, 2018), and the loss barriers between adjacent checkpoints give a complementary view of the landscape connectivity (Vlaar & Frankle, 2022). We study both, but find neither gives a signal before capability acquisition (§4.5). 2 Table 1: Algorithmic tasks and difficulty scaling. All tasks use the formatTASK input = output. Difficulty increases with level via input length or numerical range. TaskDescriptionL1L2L3 COPY Copy tokens3 tokens5 tokens8 tokens REVReverse tokens3 tokens5 tokens8 tokens CMP Compare two numbers 1 digit2 digits3 digits PARParity of bits4 bits6 bits8 bits ADD Addition1+1 digit2+2 digits3+3 digits MOD Modular arithmetic p∈2, 3, 5, 7 p∈7, 11, 13 p∈13, 17, 19, 23 SORT Sort numbers3 numbers5 numbers8 numbers MUL Multiplication1×1 digit1×2 digits2×2 digits Singular learning theory. The LLCλextends the notion of model dimensionality to singular settings (Watanabe, 2009). Lau et al. (2025) derive practical estimators based on SGLD sampling; Hoogland et al. (2024) apply LLC to a small transformer trained on language data and find that LLC transitions coincide with behavioral changes. Similarly, Cullen et al. (2026)’s analysis of grokking finds that the LLC tracks but does not predict transitions between basins. Nano scale transformer measurements in our multi-task setting mirror these findings: LLC transitions are concurrent with behavioral transitions. Capability acquisition and grokking. Wei et al. (2022) identify capabilities that appear suddenly with increasing training scale. Schaeffer et al. (2023) show that sudden emergence can be a metric artifact: accuracy shows a sharp jump where log-probability reveals gradual improvement. We confirm this divergence in our data, but find that log-probability “acquisition” (crossing a midpoint threshold) occurs when accuracy is near zero, so it does not represent genuine capability acquisition. We define acquisition using accuracy, which has a clear behavioral meaning (§4.1). Nanda et al. (2023) show that mechanistic progress measures (Fourier components, weight norms) predict grokking before generalization. Our RankMe collapse floor of ≈2.0 for modular arithmetic is consistent with their finding that two Fourier components suffice, but our geometric measures are coarser: they are able to detect that some reorganization is occurring, but not what kind of circuit is being learned. Pythia. Biderman et al. (2023) released a suite of 16 language models with 154 training checkpoints each, making them a rich dataset for probing training dynamics. We study three model sizes: 160M, 410M, and 2.8B. The collapse-recovery pattern replicates, and the geometric dynamics at the 160M scale predict the dynamics at the 2.8B scale (ρ >0.92). Per-task precursors are absent for the easy benchmarks, and present for the hard benchmark we tested (logical deduction), confirming that precursors are viable only when the task is hard relative to the model’s capacity. 3 Methods 3.1 Algorithmic Training Platform We train small transformers on eight algorithmic tasks simultaneously, ranging from token copying to multi- digit multiplication (Table 1). Each task has three difficulty levels that scale input size or numerical range, giving 24 task×level combinations. Because we control the tasks, we know exactly when the model acquires each capability and can measure what changes geometrically before, during, and after. Architecture. GPT-2-style decoder-only transformers (Radford et al., 2019) with pre-norm, GELU, learned positional embeddings, tied embeddings, no dropout. Six sizes span 370×in parameter count (Table 2). The largest (XLarge, 151M) is deliberately sized to match Pythia-160M, bridging the gap between our controlled setting and naturalistic pre-training. Training. AdamW with cosine LR schedule (1K warmup, peak 3×10 −4 for nano–small, 1×10 −4 for medium– xlarge), weight decay 0.1, batch size 64. Data is generated on the fly with uniform sampling across tasks 3 Table 2: Model configurations. Parameter counts exclude tied output embedding weights. SizeLayers d model Heads d f Parameters Nano21284512405K Micro419267681.8M Small63208 1,2807.4M Medium85128 2,04825.2M Large1276812 3,07285M XLarge12 1,02416 4,096151M Table 3: Geometric measures. Cost is wall-clock time per checkpoint on nano (405K params). MeasureCapturesFormulationCost RankMeRepr. dimensionalityexp(H( ̄σ)), ̄σ i = σ i / P j σ j ∼1s Grad. eff. rank Gradient concentrationexp(H( ̄ λ)) of gradient eigenvalues ∼85s LLCLoss landscape complexity SGLD loss elevation ˆ λ∼11s Hessian top-λCurvatureStochastic Lanczos, k=5 (per task) ∼1s Grad. cov. rank Gradient diversityEffective rank of ∇L∇L ⊤ (per task) ∼6s and levels. Loss is masked to answer tokens only. Training runs 100K steps for nano–small, 200K for medium–xlarge. Character-level tokenization with vocabulary size 41. Checkpoints are saved densely early (every 100 steps for the first 10K, coarser after that), yielding 206 checkpoints for nano–small and 256 for medium–xlarge (117 for xlarge, which trained for fewer steps). At each checkpoint we evaluate all 24 task×level combinations on 1,000 examples, recording accuracy and log-probability (Schaeffer et al., 2023). Acquisition detection. We define capability acquisition as the first training step at which accuracy≥0.5 for≥3 consecutive checkpoints. All precursor statistics for the algorithmic tasks use the accuracy-based definition. Robustness to threshold choice is verified in Appendix H. For Pythia, we use the log-probability analogue since accuracy is not always well-defined on those benchmarks. 3.2 Geometric Measures We measure five complementary geometric properties spanning representation geometry, gradient geometry, and loss landscape geometry (Table 3). All five are computed at nano scale to establish the temporal hierarchy. RankMe, the computationally cheapest (∼1 s/checkpoint at nano scale), is the only measure computed at all six scales; the cross-scale analysis relies on it alone. Gradient effective rank is computed at nano (206 checkpoints) and sparsely at large (37 checkpoints); Hessian and gradient covariance are computed at nano and on Pythia 160M/410M; LLC is computed at nano only. Task-specific RankMe. For each task, we collect hidden-state activations from the final transformer layer on 200 diagnostic examples, compute the singular value decomposition, and apply the RankMe formula (Garrido et al., 2023):RankMe(H) =exp − P i ̄σ i log ̄σ i , where ̄σ i =σ i / P j σ j are the normalized singular values. This measures the effective dimensionality of the representation space as seen by each task. We also compute RankMe per transformer layer to study propagation patterns. Gradient effective rank. We compute per-sample gradients on 200 task-specific examples, forming a gradient matrixG∈R N×P (Nsamples,Pparameters), and eigendecompose the Gram matrixGG ⊤ ∈R N×N . The effective rank isexp(H( ̄ λ )), where ̄ λ are the normalized eigenvalues. This measures how concentrated the per-sample gradient directions are: low rank means gradients point in similar directions across examples, high rank means they are diverse. Local Learning Coefficient (LLC). Following Lau et al. (2025), we estimate the LLC at each checkpoint using SGLD sampling: we run 500 SGLD steps with learning rate 10 −5 , inverse temperatureβ= 1.0, and 4 Table 4: Pythia diagnostic sets. Unlike the algorithmic experiment, these are external probes, not inputs from the training distribution. Benchmark CapabilityPrompt format (example)N SyntacticSubject–verb agreement “The keys to the cabinet is/are” (logprob)36 SemanticWord analogies“man : woman :: king : ___” (4-way choice)30 Arithmetic Few-shot addition/mult. “3+7=10; 15+23=38; 42+8=?” (3-shot)200 ICLIn-context learningNovel label mappings (4-shot sent./categ.)16 FactualFactual recall“The capital of France is ___” (next token)51 LogicalDeductive reasoning“All dogs are animals. Rex is a dog.∴ ___”26 PileGeneral text modelingRandom Pile validation sequences (perplexity) 200 localization strengthγ= 10,000, discarding the first 100 steps as burn-in and computing the mean loss elevation from the remaining 400 steps. The LLC measures the effective complexity of the loss landscape near the current parameter configuration. Hessian top eigenvalue. We compute the top eigenvalues of the Hessian via stochastic Lanczos quadra- ture (Ghorbani et al., 2019). For Pythia, we use random Pile validation data (global,k=20 Lanczos vectors, ∼30s/checkpoint). For the algorithmic experiment, we compute a task-specific variant: Lanczos on each task’s loss separately (k=5, ∼1s/checkpoint per task). Gradient covariance rank. The effective rank of the batch-level gradient covariance matrix, measuring the diversity of gradient directions across training examples. Computed per-task on the same diagnostic sets used for RankMe. For computational tractability, we use the first 50K parameters, which biases toward embedding weights and early layers (see Appendix K for details). 3.3 Output Token Probing To test whether the geometric reorganization is task-relevant, we train per-task linear probes at each checkpoint. For each task, we extract final-layer hidden states at answer positions and train a logistic regression to predict the correct next token. If the probe can extract the correct answer at a checkpoint where the model cannot yet produce it, the representations contain task-relevant information before the model can act on it. We use 80/20 stratified train/test splits with convergence at 500 iterations. Per-layer probing applies the same method to each transformer layer’s output. All probing uses the same 200-example diagnostic sets used for RankMe. 3.4 Pythia Validation We analyze three Pythia-deduped models (Biderman et al., 2023): 160M (154 checkpoints), 410M (154 checkpoints), and 2.8B (72 checkpoints spanning the full training trajectory). For each checkpoint we compute task-specific RankMe on seven diagnostic sets (Table 4). Unlike the algorithmic experiment, these are external probes, not inputs from the training distribution. We additionally use global geometric measures (Hessian eigenvalues, loss barriers, gradient covariance) at all 154 checkpoints for 160M and 410M. 3.5 Analysis Methods Temporal precedence. A geometric measure is a precursor for a given task if its characteristic transition (e.g., the RankMe collapse minimum) occurs before the task is acquired. The precursor rate is the fraction of task×model combinations where the geometric transition precedes acquisition. State-based prediction. We test whether the RankMe value at the collapse floor predicts the order in which tasks are acquired. We compute Spearman correlations between floor values and acquisition steps, then check 5 Table 5: Acquisition steps (training step at which accuracy≥0.5 for≥3 consecutive checkpoints) for 8 tasks across 6 model sizes. Easy tasks (gray) are acquired during initialization; hard tasks (white) show scale-dependent acceleration. At xlarge, even hard tasks are acquired within the first few thousand steps. L2 difficulty shown; full tables in Appendix B. TaskNano Micro Small Medium LargeXLarge (405K) (1.8M) (7.4M) (25.2M) (85M)(151M) COPY700700400600500400 REV700600400600400400 CMP400400300100200100 PAR9009003,7002,500600900 SORT1,100900900700900500 ADD5,700 3,100 3,9003,200 3,5002,600 MOD6,500 5,100 5,0003,600 3,6003,500 MUL6,900 5,400 4,7004,100 3,5003,400 XLarge: 23/24 combinations acquired by step 13K; MUL_L3 not acquired (training stopped at 18K). whether any predictive power persists within a difficulty class (e.g., does a lower floor among hard tasks predict earlier acquisition?) or is simply driven by the fact that hard and easy tasks have different floors. 4 Results 4.1 The Acquisition Landscape Of 144 task×level×model combinations, 142 achieve sustained acquisition (accuracy≥0.5 for≥3 consecutive checkpoints). The two exceptions are MUL_L3 on nano, which never crosses 50% accuracy, and MUL_L3 on xlarge, where training was stopped at 18K steps, before the expected acquisition window (43K–56K at other scales), due to compute constraints. Since difficulty levels of any task are related (if a task is hard at L1, it is hard at L2 and L3), the effective number of independent observations is closer to 48 (8 tasks×6 scales). That said, the range is large: acquisition spans nearly three orders of magnitude, from step 100 (CMP on xlarge) to step 56,000 (MUL_L3 on small). We split tasks into two categories based on timing relative to early training dynamics (Table 5). These are static labels which do not reflect the relative task difficulty vis-à-vis model capacity. In general, tasks become easier for the model as its capacity increases. Easy tasks (COPY, REV, CMP, PAR, SORT) are typically acquired within the first 2,500 steps, during or shortly after the initial representation reorganization; hard tasks (ADD, MOD, MUL) take thousands to tens of thousands of steps, with larger models generally acquiring them earlier (e.g., MOD_L3: 16K steps at nano vs. 6.0K at large; full table in Appendix B). 1 For every task, accuracy-based and log-probability-based detection diverge: the “emergence mirage” of Schaeffer et al. (2023), replicated here in a controlled setting. Log-probability crosses its midpoint threshold 3–15×earlier than accuracy crosses 50%, but at that point accuracy is typically 0–15%: the model cannot yet perform the task. We therefore define acquisition using accuracy, which has a clear behavioral interpretation (the model can do the task more often than not). The full divergence analysis is in Appendix C. 4.2 Universal Collapse to Task-Specific Floors During the first 200–1,000 training steps, task-specific RankMe drops sharply across all tasks, model sizes, and layers. Across all tasks and model sizes we test, the collapse is consistent, and in line with Li et al. (2025): all 8 tasks across all 6 model sizes collapse in the same step range with near-identical dynamics. What differs 1 SORT, despite involving a multi-element operation, is acquired in the easy range (1,500–1,800 steps at L3). PAR shows scale sensitivity: PAR_L2 on small is acquired at step 3,700 (hard-task range), and a few other PAR combinations exceed 2,500 steps, though the majority are acquired early. 6 405K1.8M7.4M25M85M COPY REV CMP PAR ADD MOD SORT MUL Model size 200 500 1K 5K 10K 50K Emergence step Figure 1: Acquisition map. Log-scaled acquisition step for 8 tasks across 6 model sizes. Easy tasks (above white line) are acquired uniformly early; hard tasks (below) show scale-dependent acceleration. At xlarge (151M), the gap between easy and hard nearly vanishes. 0 12.5K25K 0 10 20 Step RankMe MOD 0 12.5K25K Step ADD 0 12.5K25K Step SORT 0 12.5K25K Step MUL 405K1.8M7.4M25M85M Figure 2: Task-specific collapse floors. RankMe trajectories for three hard tasks across six model sizes. All models collapse to task-specific minima during initialization (shaded), then recover. MOD floor is scale-invariant (≈2.0, CV = 0.16); MUL floors increase with capacity. between tasks is the floor (Figure 2). MOD collapses to RankMe≈2.0 regardless of model size, consistent with its two-dimensional Fourier structure (Nanda et al., 2023). MUL floors increase with model capacity. 4.3 Top-Down Layer Propagation Per-layer RankMe reveals that the collapse propagates top-down: the deepest (output-facing) layers collapse most, while early layers retain more representational diversity. This holds in 32 of 32 task×model combinations tested (8 tasks×4 model sizes with≥4 layers: micro through large; xlarge was excluded because per-layer RankMe was not computed at that scale), which is not consistent with the intuition that learned features build bottom-up from simple to complex. At the collapse minimum, the final transformer layer has much lower RankMe than the first. For MOD: micro layer 0 = 8.2 vs. layer 3 = 1.7 (80% reduction), small layer 0 = 10.4 vs. layer 5 = 2.2 (78%), medium layer 0 = 14.8 vs. layer 7 = 2.3 (84%). The gradient is monotonic (layerLcollapses more than layerL−1), and first-to-last-layer reduction ranges from 30% to 84% across all tasks and sizes. After acquisition, the pattern changes: the final layer remains compressed while middle layers recover and diversify (Figure 3). During collapse, the output layer commits to a low-dimensional representation; after acquisition, intermediate layers reorganize to support it. 7 01234567 0 5 10 15 Layer index RankMe At collapse minimum 01234567 0 10 20 Layer index RankMe Post-emergence (+10K steps) 1.8M (4L)7.4M (6L)25M (8L) Figure 3: Top-down layer propagation. Per-layer RankMe at the collapse minimum (left) and post-acquisition (right) for three model sizes on MOD. At collapse, deeper layers show lower RankMe. After acquisition, the final layer stays compressed while middle layers recover. Table 6: Hidden learning at the last checkpoint before acquisition (L2 difficulty). Behav = exact-match accuracy (the model’s output matches the ground truth). Probe = per-token accuracy of a trained logistic regression on the hidden states. These measure different things: behavioral accuracy tests whether the model can produce the answer; probe accuracy tests whether the representation contains extractable information about the answer. At every checkpoint shown, the model cannot yet perform the task (behavioral<0.5), but a trained probe can already extract the correct output token. High probe values at initialization (e.g., CMP) reflect the probe’s own capacity to learn the mapping from structured input embeddings to a small number of output classes, even with random model weights. XLarge omitted (probing not run at that scale). Nano (405K)Micro (1.8M)Small (7.4M) Medium (25.2M) Large (85M) TaskBehav Probe Behav Probe Behav Probe BehavProbe Behav Probe COPY.00.81.39.94.00.76.47.95.26.96 REV.02.83.00.73.00.70.29.92.00.77 CMP.49.94.49.96.49.97.00.97.46.96 PAR.49.87.42.91.48.94.49.92.49.88 SORT.34.98.38.98.31.99.50.99.45.99 ADD.43.75.41.76.29.80.49.83.48.84 MOD.46.79.46.80.49.81.48.79.50.82 MUL.44.76.46.81.48.81.49.81.47.77 4.4 Hidden Learning Before Acquisition The geometric analyses above show that representations reorganize before acquisition, but not whether this reorganization is task-relevant. We test directly: at each checkpoint, we train a linear probe (logistic regression) on each task’s hidden representations to predict the correct output token. For all 8 tasks across all model sizes, the probe can extract the correct output token at checkpoints where the model cannot yet produce it (Table 6). The probe is a trained classifier with its own capacity to learn the mapping from hidden states to output tokens; its success indicates that task-relevant information is present in the representations, not that the model can use it. Per-layer probing at nano scale shows that the improvement concentrates in the deeper layers (deep-to-shallow ∆ ratio 3–11×for ADD, MUL, and SORT), the same layers where RankMe collapses most (Figure 3, §4.3). This confirms that the top-down collapse reflects task-relevant representational commitment, not generic dimensionality reduction. 8 Table 7: Geometric hierarchy at nano scale (405K params). All measures are task-specific: computed on task-conditioned data for each hard task (ADD, MOD, MUL; 8 task×level combinations, excluding MUL_L3 which was not acquired). Precursor rate = fraction of cases where the measure’s characteristic transition precedes acquisition. RankMe is the only measure computed at all six scales. MeasureLevelHard-Task Precursor Rate Cost/ckpt Signal Quality RankMeRepresentation100% (8/8)∼1sClean collapse → recovery Hessian λ max Curvature100% (8/8)∼1sNoisy (oscillates between ckpts) Grad. cov. rank Gradient100% (8/8)∼6sNoisy Grad. eff. rank Gradient38% (3/8)∼85sTransition too late for most tasks LLCLoss landscapeNo discrete event∼11sPeaks early, declines continuously 0 5K10K15K20K25K 0 0.5 1 Training step Accuracy (a) MOD on nano (405K params) 0 0.5 1 Normalized AccuracyRankMeLLC Grad Cov Hessian LLC Grad Eff Rank RankMe 0 50 100 100100 0 38 100 Precursor rate (%) (b) Hard-task precursor rates (nano) Figure 4: Geometric hierarchy for MOD on nano (405K params). (a) Accuracy, RankMe, and LLC (min-max normalized). RankMe collapses before acquisition; LLC rises synchronously with accuracy. (b) Hard-task precursor rates across five measures at nano scale (all task-specific). RankMe, Hessian, and gradient covariance all reach 100%, but only RankMe provides a clean, non-noisy signal (Table 7). LLC shows no discrete precursor event. 4.5 The Geometric Hierarchy We compare five geometric measures for their temporal relationship with capability acquisition. The measures span representation geometry (RankMe), gradient geometry (gradient effective rank), and loss landscape geometry (LLC, Hessian eigenvalues, gradient covariance rank). Table 7 and Figure 4 summarize the hierarchy. RankMe leads. For hard tasks (ADD, MOD, MUL), RankMe detects geometric transitions before acquisition at every scale tested (100% precursor rate, Table 12). Easy tasks show no precursors at larger scales because they are acquired during the collapse itself, before any temporal gap can form. Other measures. When computed on task-specific data, Hessianλ max and gradient covariance rank also show 100% hard-task precursor rates at nano scale, but with much noisier signals: the task-specific Hessian oscillates widely between adjacent checkpoints (e.g., MODλ max ranges from 200 to 1,600 within the first 2K steps), making it unreliable for monitoring. Gradient effective rank transitions too late for most hard tasks (minimum at step 5,700–9,400, after easier levels have already acquired). LLC (Lau et al., 2025) shows no discrete precursor event: it peaks early (step∼400) then declines continuously, consistent with recent analyses where LLC tracks rather than predicts transitions (Cullen et al., 2026; Hoogland et al., 2024). The capacity/difficulty boundary. Geometric precursors require a temporal gap between geometric reorganiza- tion and behavioral acquisition. If the model has enough capacity, easy tasks get acquired during or before the collapse and no precursor is detectable. Hard tasks can potentially take thousands of steps post-collapse, so the signal is strong (Figure 5). 9 024681012 10 2 10 3 10 4 10 5 1K steps RankMe at collapse floor Emergence step (a) Early RankMe vs. emergence EasyHard Cross-class Within hard Within easy Swap 0 50 100 chance 93% 69% 52% 26% 93% 69% 52% 26% Concordance (%) (b) Concordance stress test Figure 5: The capacity/difficulty boundary. (a) Early RankMe vs. acquisition step: easy and hard tasks separate clearly, but within a difficulty class RankMe does not predict acquisition order. (b) Bootstrap concordance rates confirm this: cross-class separation is strong (93%), but within-class prediction brackets chance. Error bars show 95% CIs. 4.6 Transfer to Naturalistic Pre-Training We test whether these patterns transfer to language model pre-training using Pythia-160M, 410M (154 checkpoints each), and 2.8B (72 checkpoints spanning the full trajectory). The 2.8B model is 17.5×the size of the 160M and shares the same architecture (GPT-NeoX) and training data (Pile), providing a direct test of proxy prediction across scale. Collapse replicates. Task-specific RankMe drops 50–90% from step 0 to steps 32–256, then partially recovers, mirroring the algorithmic experiment. All seven benchmarks reach their minimum in this early window at all three model sizes. Ordering and floors transfer across scale. The RankMe ordering across benchmarks is preserved between 160M and 410M (ρ= 1.0) and nearly so at 2.8B (ρ= 0.964). Collapse floors land within 4–22% of the 160M values at 2.8B for six of seven benchmarks (ICL is an outlier at 41%, likely due to its small absolute floor), and ρ > 0.92 at 71 of 72 common training checkpoints. Precursor positive for a hard capability. The seven standard benchmarks are within the easy regime for 410M+ models, so no precursor gap is expected. To test whether precursors appear for a genuinely hard task, we screened seven additional benchmarks and identified logical deduction (multi-step ordering from BIG-Bench Hard, distinct from the syllogistic Logical benchmark in Table 4) as one that emerges late on 2.8B: 0% accuracy through step 10K, 24% at 50K, 50% at 143K. On 410M the same task peaks at 28% (step 100K) then drops to 4%, right at the capacity boundary. Task-specific RankMe on the logical deduction prompts collapses to a floor of 7.8 (2.8B) and 6.8 (410M) at step 32, recovers by step 1K, then gradually declines through the rest of training. The collapse-recovery precedes behavioral emergence by roughly 49K steps on 2.8B; floors are again scale-invariant across the two models. Easy benchmarks: negative, and predicted. The seven standard benchmarks show no precursors, consistent with the capacity/difficulty framework: these tasks are easy for 410M+ models. 5 Discussion Task difficulty governs precursor detectability. Precursor detectability depends on task difficulty relative to model capacity. If the task is genuinely hard for the model, RankMe precedes acquisition (100% for hard tasks at every scale tested, confirmed on Pythia-2.8B with logical deduction). If the task is easy relative to the model’s capacity, both geometric and behavioral changes happen simultaneously and no precursor is 10 10 0 10 1 10 2 10 3 10 4 10 5 0 200 400 Training step RankMe (a) Pythia-410M task-specific RankMe 0100200300400 pile logical semantic factual syntactic icl arith. Final RankMe (b) Ordering preserved (ρ=1.0) 160M 410M pilelogical semantic factual syntactic iclarith. Figure 6: Pythia validation. (a) RankMe collapse and recovery in Pythia-410M across seven benchmarks, replicating the universal collapse. (b) Cross-model RankMe ordering is preserved:ρ= 1.0 (160M vs 410M), ρ= 0.964 (160M vs 2.8B, 17.5×scale gap). Task-specific temporal precedence is absent, consistent with the capacity/difficulty framework. detectable. LLC and global Hessian underperform RankMe because they aggregate across tasks in different developmental phases. Why top-down? The top-down propagation (32/32 cases) contradicts the intuition that features build bottom- up. A simpler explanation is gradient proximity: the loss is computed at the output, so output-facing layers receive the strongest gradient signal and adapt first. Hidden learning corroborates this: at nano scale, the deepest layer’s probe improvement is 3–11× larger than the shallowest layer’s for hard tasks (§4.4). Proxy models. The geometric dynamics replicate from small algorithmic models to Pythia across a 17.5× scale gap (§4.6), suggesting that small proxy models may capture key aspects of the geometric trajectory seen at larger scale. Further discussion of collapse floor interpretations, the freeze experiment, and falsifiable predictions is in Appendix I. Limitations. Our 8 algorithmic tasks are simpler than natural language, and the effective sample size is∼48 independent observations (8 tasks×6 scales). The logical deduction result extends the capacity/difficulty framework to Pythia-2.8B, but this is one task; broader confirmation with multiple hard capabilities on larger models is needed. 6 Conclusion Across the settings we study, capability acquisition in transformers follows a consistent geometric sequence: task-conditioned representations collapse, reorganize across depth, and only then does behavioral performance improve. RankMe precedes behavior for every hard task at every scale we tested, from 405K to 151M parameters and on Pythia-2.8B for logical deduction. Whether a precursor is detectable depends on task difficulty relative to model capacity: if the task genuinely challenges the model, the geometric signal precedes behavior; if not, both change simultaneously. The dynamics replicate from small proxy models to Pythia across a 17.5×scale gap, suggesting that geometric monitoring at small scale may inform expectations for larger training runs. 11 Ethics and AI Disclosure This work analyses existing publicly available models and datasets; no new data was collected and no human subjects were involved. The author used Claude (Anthropic) and Claude Code during preparation for manuscript critique, narrative feedback, literature search, and experiment implementation and debugging. All research design, theoretical development, experimental execution, analysis, and writing are the author’s own. The author takes full responsibility for all content. Reproducibility. Code and data for reproducing all results are available athttps://github.com/jb1999/ capability-acquisition-paper . The repository includes training scripts, geometric measure computation, analysis pipelines, and a verification script that checks 11 pattern-level claims against fresh regression results. References Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Z. Fern. Neural networks learn statistics of increasing complexity. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, p. 3382–3409. PMLR / OpenReview.net, 2024. URLhttps://proceedings.mlr.press/v235/ belrose24a.html. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 2397–2430. PMLR, 2023. URL https://proceedings.mlr.press/v202/biderman23a.html. Zheng-An Chen and Tao Luo. From condensation to rank collapse: A two-stage analysis of transformer training dynamics. CoRR, abs/2510.06954, 2025. URL https://arxiv.org/abs/2510.06954. Ben Cullen, Sergio Estan-Ruiz, Riya Danait, and Jiayi Li. Grokking as a phase transition between competing basins: a singular learning theory approach. CoRR, abs/2603.01192, 2026. URLhttps://arxiv.org/ abs/2603.01192. Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 10929–10974. PMLR, 2023. URL https://proceedings.mlr.press/ v202/garrido23a.html. Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, p. 2232–2241. PMLR, 2019. URL http://proceedings.mlr.press/v97/ghorbani19b.html. Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning. CoRR, abs/2402.02364, 2024. doi: 10.48550/ARXIV.2402. 02364. URL https://doi.org/10.48550/arXiv.2402.02364. Edmund Lau, Zach Furman, George Wang, Daniel Murfet, and Susan Wei. The local learning coefficient: A singularity-aware complexity measure. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Moham- mad Emtiyaz Khan (eds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2025, Mai Khao, Thailand, 3-5 May 2025, volume 258 of Proceedings of Machine Learning Research, p. 244–252. PMLR, 2025. URL https://proceedings.mlr.press/v258/lau25a.html. 12 Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, and Blake A. Richards. Tracing the representation geometry of language models from pretraining to post-training. CoRR, abs/2509.23024, 2025. doi: 10.48550/ARXIV.2509.23024. URLhttps://doi. org/10.48550/arXiv.2509.23024. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLhttps://openreview.net/ forum?id=9XFSbDPmdW. Vardan Papyan. The full spectrum of deep net hessians at scale: Dynamics with sample size. CoRR, abs/1811.07062, 2018. URL http://arxiv.org/abs/1811.07062. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019. Levent Sagun, Utku Evci, V. Ugur Güney, Yann N. Dauphin, and Léon Bottou. Empirical analysis of the Hessian of over-parametrized neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rJO1_M0Lf. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language mod- els a mirage? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Con- ference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URLhttp://papers.nips.c/paper_files/paper/2023/hash/ adc98a266f45005c403b8311ca7e8bd7-Abstract-Conference.html. Tiffany J. Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, p. 22325–22341. PMLR, 2022. URL https://proceedings.mlr.press/v162/vlaar22a.html. Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory, volume 25 of Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge, UK, 2009. ISBN 978-0-521-86467-1. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview.net/forum?id=yzkSU5zdwD. 13 A Task Specifications and Training Details A.1 Task Data Generation All tasks use the formatTASK input = outputwith character-level tokenization (vocabulary size 41, including digits 0–9, uppercase letters, space, equals sign, and special tokens). Training data is generated on-the-fly with uniform sampling across tasks and levels. COPY: Copy n single-digit tokens. L1: n=3, L2: n=5, L3: n=8. REV: Reverse n single-digit tokens. Same n as COPY. CMP: Compare two integers, output LESS/EQUAL/GREATER. L1: 1-digit (0–9), L2: 2-digit (10–99), L3: 3-digit (100–999). PAR: Parity of n binary digits, output ODD/EVEN. L1: n=4, L2: n=6, L3: n=8. ADD: Addition of two integers. L1: 1+1 digit, L2: 2+2 digit, L3: 3+3 digit. MOD: Modular arithmeticx mod p. L1:p∈2,3,5,7, L2:p∈7,11,13, L3:p∈13,17,19,23. Inputx sampled from [0, 10p]. SORT: Sort n single-digit numbers in ascending order. L1: n=3, L2: n=5, L3: n=8. MUL: Multiplication of two integers. L1: 1×1 digit, L2: 1×2 digit, L3: 2×2 digit. A.2 Training Hyperparameters Table 8: Full training configuration for each model size. NanoMicroSmall Medium Large XLarge Peak LR3×10 −4 3×10 −4 3×10 −4 1×10 −4 1×10 −4 1×10 −4 Max steps100K100K100K200K200K200K † Warmup steps 1,0001,0001,0001,0001,0001,000 Batch size646464646464 Weight decay0.10.10.10.10.10.1 Grad clip1.01.01.01.01.01.0 OptimizerAdamW (β 1 =0.9, β 2 =0.95) LR scheduleCosine decay to 0 Checkpoints206206206256256117 † Training stopped at 18K steps; all tasks except MUL_L3 acquired by step 13K. A.3 Checkpoint Schedule Dense-where-it-matters: • Steps 0–10,000: every 100 steps (100 checkpoints) • Steps 10,000–50,000: every 500 steps (80 checkpoints) • Steps 50,000–200,000: every 2,000 steps (75 checkpoints) • Final step always included B Full Emergence Tables Table 9 shows acquisition steps for all 144 task×level×model combinations. 14 Table 9: Complete acquisition steps for all task×level×model combinations. “−” indicates task not acquired by end of training. TaskLevel Nano Micro Small Medium Large XLarge COPY L1800600400500400300 COPY L2700700400600500400 COPY L3700700500600500400 REVL1700600400500400400 REVL2700600400600400400 REVL3700700400700500500 CMP L11,100 1,100800600200300 CMP L2400400300100200100 CMP L3400700100100200300 PARL11,600 1,000 2,8001,100900300 PARL2900900 3,7002,500600900 PARL31,600800 2,5008006002,600 ADD L12,000 1,800 1,5001,200 1,100800 ADD L25,700 3,100 3,9003,200 3,5002,600 ADD L37,500 10,500 6,90011,500 7,900 13,000 MOD L13,200 2,500 2,1001,300 1,500900 MOD L26,500 5,100 5,0003,600 3,6003,500 MOD L316,000 9,100 8,1006,200 6,0005,800 SORT L1800500500300300400 SORT L21,100900900700900500 SORT L31,500 1,800 1,7001,500 1,7001,600 MUL L11,600 1,200 1,100900500400 MUL L26,900 5,400 4,7004,100 3,5003,400 MUL L3− 54,000 56,00043,500 43,000 − C Log-Probability Acquisition and Dual-Metric Divergence Log-probability acquisition is defined as the first step where mean per-token log-probability of the correct answer exceeds the midpoint between its initial and final values for≥3 consecutive checkpoints. This always precedes accuracy acquisition (by 3–15×), confirming the “emergence mirage” of Schaeffer et al. (2023). However, at the log-probability midpoint, accuracy is typically 0–15%: the model cannot yet perform the task. We therefore define acquisition using accuracy. Table 10 shows that at any log-probability threshold where the model can actually perform the task (accuracy 30%+), RankMe precursor rates recover to near 100%. Table 10: Hard-task RankMe precursor rates under log-probability acquisition at different thresholds (fraction of the way from initial to final log-probability). Accuracy-based rates (last column) use the≥50% sustained threshold. Typical accuracy at each log-prob threshold is shown in parentheses. At the midpoint (50%) threshold, “acquisition” occurs when accuracy is near zero; at 75%+ thresholds where the model can actually perform the task, precursor rates recover. ScaleLP 25%LP 50%LP 75%LP 90%Accuracy (acc 0–10%) (acc 0–15%) (acc 30–65%) (acc 50–90%) (≥50%) Nano (405K)0%100%100%100%100% Micro (1.8M)0%89%100%100%100% Small (7.4M)0%33%100%100%100% Medium (25.2M)0%33%89%100%89% Large (85M)0%22%78%100%89% 15 D Per-Layer RankMe Trajectories We compute RankMe at every transformer layer for each task across all checkpoints. Figure 7 shows per-layer trajectories for the micro model (4 layers) on three hard tasks (ADD, MOD, MUL) and SORT. The top-down gradient is visible across all tasks. After acquisition, the final layer stays compressed while middle layers recover. 0 10 20 RankMe ADD 0 10 20 RankMe MOD 0 10K20K 0 10 20 Training step RankMe SORT 0 10K20K 0 10 20 Training step RankMe MUL Layer 0Layer 1 Layer 2Layer 3 Figure 7: Per-layer RankMe trajectories for three hard tasks (ADD, MOD, MUL) and SORT on the micro model (4 layers). Each line represents one transformer layer; deeper layers (higher index, warmer colors) show stronger collapse and slower recovery. The top-down gradient is visible across all tasks. E Pythia Task-Specific RankMe Trajectories Figure 8 shows task-specific RankMe trajectories for all seven benchmarks across Pythia-160M, 410M, and 2.8B. Key observations: • All benchmarks show the early collapse (steps 0–128) and partial recovery at all three scales • The ordering pile>logical>semantic>factual>syntactic>ICL>arithmetic is preserved across 160M and 410M (ρ = 1.0) and nearly preserved at 2.8B (ρ = 0.964; arithmetic and ICL swap) •The Spearman correlation between 160M and 2.8B RankMe orderings isρ >0.92 at 71/72 common checkpoints (median ρ = 0.964), with the sole exception at step 32 during peak collapse (ρ = 0.536) •Collapse floors are quantitatively similar across the 17.5×scale gap: 2.8B floors are within 4–22% of 160M values for six of seven benchmarks (ICL is an outlier at 41%), systematically slightly deeper • 2.8B shows Phase 3 compression: pile RankMe declines from 520 to 484 across steps 100K–143K F Temporal Precedence Details A geometric measure is classified as a precursor for a given task if its characteristic transition (e.g., the RankMe collapse minimum) occurs at an earlier training step than the task’s acquisition step. For RankMe, the transition is the collapse minimum (searched within the first 10K steps). For gradient effective rank, it is the minimum of the task-specific effective rank. For LLC, we look for a discrete transition event; as discussed in §4.5, LLC shows no such event (it peaks early then declines continuously). 16 10 0 10 1 10 2 10 3 10 4 10 5 0 200 400 Training step RankMe Pythia-160M 10 0 10 1 10 2 10 3 10 4 10 5 0 200 400 Training step RankMe Pythia-410M pilelogical semantic factual syntactic iclarith. Figure 8: Task-specific RankMe across training for Pythia-160M, 410M, and 2.8B. All seven benchmarks show universal early collapse (shaded region, steps 0–512) and maintained ordering throughout training. The ordering pile>logical>semantic>factual>syntactic>ICL>arithmetic is preserved across 160M and 410M (ρ = 1.0) and nearly so at 2.8B (ρ = 0.964). G Compute Budget Table 11: Compute budget breakdown. XLarge is excluded: training and eval added∼3 GPU-hrs, but linear probing and gradient effective rank were not run at that scale (estimated∼60 additional GPU-hrs). All times are wall-clock on NVIDIA RTX 3090 Ti (24 GB VRAM, 124 GB RAM), measured fromtiming_log.json. ComponentCkpts GPU-hrs Algorithmic experiment Training (5 sizes + 5 freeze)– ∼10 Eval + task-specific RankMe (5 sizes)740 ∼7 Eval + task-specific RankMe (5 freeze)1,030 ∼6 Grad. eff. rank + LLC (nano: 206, large: 37)243 ∼19 Linear probing (nano–large)1,130 ∼146 Pythia analysis Task-specific RankMe (160M + 410M + 2.8B)380 ∼6 Coarse pipeline (160M + 410M)308 ∼6 Medium/Hessian pipeline (160M + 410M)62 ∼28 Pile domain RankMe (160M + 410M)308 ∼2 Hard benchmark screening + RankMe238 ∼5 Total∼231 Linear probing dominates the budget (∼146 GPU-hrs) and was run at all 206–256 checkpoints per model for completeness. The paper’s claims require only the emergence window (∼50 checkpoints per size,∼30 GPU-hrs). Not all measurements were run at all checkpoints: gradient effective rank was computed on 206 checkpoints for nano but only 37 sparse checkpoints for large; task-specific RankMe used 76–206 checkpoints depending on size. A minimal reproduction (nano only, no probing, no gradient effective rank) requires∼1 GPU-hour. H Robustness and Sensitivity Analysis We test whether the main findings are robust to methodological choices. 17 H.1 Acquisition Threshold Sensitivity Our primary results use accuracy≥0.5 for≥3 consecutive checkpoints. Hard-task precursor rates are 100% at all acquisition thresholds tested (0.3 to 0.7) and all sustained-window widths (3, 5, 10 checkpoints). Varying the threshold changes which easy tasks count as precursors (lower thresholds detect acquisition earlier, sometimes before the collapse minimum), but the hard-task rate is invariant. H.2 Multi-Seed Validation We trained all six model sizes with three different random seeds (42, 123, 7) to test whether the findings depend on initialization. Exact acquisition steps vary across seeds (typical spread 300–3,600 steps for hard tasks), but the structural patterns are identical: easy tasks acquire early, hard tasks acquire late, and RankMe precedes acquisition for every hard task at every scale across all three seeds (100% precursor rate, zero exceptions out of 54 hard-task×scale×seed combinations). H.3 Cross-Scale Precursor Consistency Table 12 shows RankMe precursor rates for hard tasks by model size. Hard-task precursor rates are 100% across all six scales. Easy tasks are not included because they are acquired during the collapse itself at larger scales, so no temporal gap exists. Table 12: RankMe precursor rates for hard tasks by model size (L2 difficulty, per-task). A task is a precursor if the RankMe collapse minimum precedes acquisition. Hard-task precursor rates are 100% across all scales. Nano (405K) Micro (1.8M) Small (7.4M) Medium (25.2M) Large (85M) XLarge (151M) Hard tasks (3) 100% (3/3)100% (3/3)100% (3/3)100% (3/3)100% (3/3)100% (3/3) H.4 Checkpoint Density The RankMe trajectories were computed at different checkpoint densities across model sizes: 206 checkpoints for nano and micro, 76 for small, and 126 for medium and large (xlarge used probing checkpoints only, at 117 points). Coarser sampling can shift the estimated collapse minimum by a few hundred steps, since the true minimum may fall between observed checkpoints. The 100% hard-task precursor rate is unchanged at all checkpoint densities tested. I Extended Discussion Collapse floors as task complexity signatures. The MOD collapse floor (≈2.0, CV = 0.16 across a 370× parameter range) likely reflects the minimum representational dimensionality the task requires. For modular arithmetic, two dimensions suffice because the task embeds in a circle (the Fourier basis forZ/pZ; Nanda et al. 2023). For MUL, the floor increases with model capacity (CV = 0.42), suggesting larger models preserve more structure even at maximum compression. Formalizing when collapse floors are scale-invariant versus capacity-dependent is an open theoretical question. Freeze experiment. Freezing either layer during collapse delays hard-task acquisition in nano, with the output layer showing a larger effect (Table 13). The direction and magnitude of the delay are seed-dependent and do not replicate consistently at small scale, so the freeze experiment does not provide reliable causal evidence for the top-down propagation pattern. Falsifiable predictions. (1) If MOD→2.0 holds at billion-parameter scale, the floor reflects intrinsic task dimensionality; if not, our scale invariance is an artifact of the 405K–151M regime. (2) Tasks genuinely hard for a given model should show geometric precursors; tasks within the easy regime should not. We have tested this on Pythia-2.8B with one task (§4.6); testing with multiple hard capabilities is needed. (3) A proxy model 18 at 1/100th the parameter count should predict the geometric trajectory of the full model. We have confirmed this at 17.5× scale; testing at 100× remains open. J Freeze Experiment Details Table 13 shows acquisition timing under layer freezing for the nano model (2 layers). Both freeze conditions delay hard-task acquisition, but the effect is seed-dependent: in this training run, freezing the output layer (block 1) delays ADD by +14,000 steps and freezing the input layer (block 0) accelerates it by−8,000 steps, while a regression rerun with a different seed shows both conditions delaying ADD. The small model (6 layers) also shows no consistent directional asymmetry. Table 13: Freeze experiment: nano model (2 layers). Acquisition steps at L3 difficulty. Block 1 is the output layer; block 0 is the input layer. Freezing during steps 0–1,000. TaskBaseline Freeze-L1Delay Freeze-L0 Delay COPY8001,300+5001,200+400 REV7001,500+8001,100+400 CMP300900+600600+300 PAR1,0002,500+1,500300−700 SORT1,8002,000+2001,700−100 ADD19,50033,500+14,00011,500 −8,000 MOD16,00016,500+50018,500+2,500 MUL−84,000N/A−N/A K Implementation Details LLC hyperparameters. The SGLD-based LLC estimator usesn steps = 500, learning rateη= 10 −5 , inverse temperatureβ= 1.0, localization strengthγ= 10,000, with 100 burn-in steps discarded. These values follow the recommendations of Lau et al. (2025). We verified stability by running the estimator 5 times at 3 checkpoints (early, mid-training, late) for the nano model; the coefficient of variation of LLC estimates was <0.05 across runs. Gradient covariance projection. For computational tractability, gradient covariance is computed on a projected subspace using the first 50K parameters of the model. This prefix-based selection is biased toward embedding weights and early layers. A uniform random projection of the full parameter vector would be more principled; we adopt the prefix approach for consistency with our initial implementation and note it as a limitation. For the algorithmic models (405K–85M parameters), the prefix constitutes 12%–100% of the parameter space, so the bias is most relevant for the larger models. Gradient Gram trick. For models withP ≫ Nparameters (up to 85M) andN= 200 samples, we compute the gradient eigenspectrum via the Gram matrixGG ⊤ ∈R N×N rather than the full matrixG ⊤ G∈R P×P , recovering the top-N eigenvalues at O(N 2 P ) cost instead of O(P 3 ). 19