Paper deep dive
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
Irina Piontkovskaia, Sergey Nikolenko
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 4:06:18 AM
Summary
This paper investigates the local geometry of trained transformers around a multitask LoRA operating point, testing eight properties of task adaptation. It finds that single-update perturbations are first-order predictable (loss changes project onto gradients) up to a scale of 10^-2. However, pairwise interactions are fragile: update order matters due to Lie bracket effects (H_B g_A - H_A g_B), additivity fails at full scale, and gradient subspaces rotate. The study concludes that while local random search works due to first-order predictability, compositional methods like task arithmetic suffer from order-dependent defects.
Entities (8)
Relation Signals (7)
Order dependence → causedby → Lie Bracket
confidence 95% · For two sequential task-gradient steps, the leading order-dependent term is the Lie bracket H_B g_A - H_A g_B
Single-update perturbations → exhibitsproperty → First-order predictability
confidence 95% · Along individual directions, changes of the probe loss remain first-order predictable throughout the grid
Pairwise task updates → exhibitsproperty → Order dependence
confidence 95% · Pairwise structure, however, proves to be far more fragile... two-update order sensitivity sets in
LoRA → operateson → Transformer
confidence 95% · We measure 8 such properties with the same harness around a multitask LoRA operating point, on 9 transformers
First-Order Random Search → enabledby → First-order predictability
confidence 90% · a perturbation’s effect on the loss is essentially its projection onto the gradient, which is also what makes local random search work
Task Arithmetic → relieson → Functional additivity
confidence 85% · Task arithmetic and model merging need functional additivity and controlled cross-task interference
Activation Steering → relieson → Weight-to-activation correspondence
confidence 85% · activation steering needs weight-space changes to have predictable representation-space counterparts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space. We measure 8 such properties with the same harness around a multitask LoRA operating point, on 9 transformers (82M-7B), with a prospectively registered property list, thresholds, and test split. We find a shared one-direction validity window up to the tested scale $10^{-2}$, but no universal radius for pairwise composition or update ordering. Along individual directions, changes of the probe loss remain first-order predictable throughout the grid: a perturbation's effect on the loss is essentially its projection onto the gradient, which is also what makes local random search work. Pairwise structure, however, proves to be far more fragile: on over a third of the measured (model, task pair) combinations, two-update order sensitivity sets in strictly inside that window; task-gradient subspaces rotate within tens of steps; additivity under our fixed activation probe fails at full task-vector scale on several models, including both held-out 7B models; and no model median passes the registered global mean-vector weight-to-steering correspondence bar. For two sequential task-gradient steps, the leading order-dependent term is the Lie bracket $H_B\textbf{g}_A-H_A\textbf{g}_B$; its normalized prediction $c(\eta)=\eta\kappa+O(\eta^2)$ tracks the measured defect at median ratio 1.002, while the onset scale $\eta^\dagger\approx0.10/\kappa$ spans three orders of magnitude across models and task pairs.
Tags
Links
- Source: https://arxiv.org/abs/2607.16821v1
- Canonical: https://arxiv.org/abs/2607.16821v1
Trouble viewing inline? Open PDF directly →
Full Text
125,252 characters extracted from source content.
Expand or collapse full text
First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers Irina Piontkovskaia DAIMLD, Moscow, Russia Sergey Nikolenko St. Petersburg Department of the Steklov Institute of Mathematics, St. Petersburg, Russia, sergey@logic.pdmi.ras.ru St. Petersburg State University Abstract Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space. We measure eight such properties with the same harness around the same multitask LoRA operating point, on nine transformers from seven families (8282M–77B), with a prospectively registered property list, thresholds, and development/test split. We find a shared one-direction validity window up to the tested scale 10−210^-2, but no universal radius for pairwise composition or update ordering. Along individual directions, changes of the probe loss (loss on a small fixed diagnostic batch) remain first-order predictable throughout the grid on every model median: a perturbation’s effect on the loss is essentially its projection onto the gradient, which is also what makes local random search work. Pairwise structure, however, proves to be far more fragile: on over a third of the measured (model, task pair) combinations, two-update order sensitivity sets in strictly inside that window; the registered rank-≤32≤ 32 mixture-covariance test turns out to be algebraically non-falsifiable at its sample count and shows no rank plateau up to m=256m=256; task-gradient subspaces rotate within tens of steps; additivity under our fixed activation probe fails at full task-vector scale on several models, including both held-out 77B models; and no model median passes the registered global mean-vector weight-to-steering correspondence bar. For two sequential task-gradient steps, the leading order-dependent term is the Lie bracket HBA−HABH_Bg_A-H_Ag_B. With the same minibatches on both sides, its normalized prediction c(η)=ηκ+O(η2)c(η)=ηκ+O(η^2) tracks the measured defect at median ratio 1.0021.002, while the onset scale η†≈0.10/κη ≈ 0.10/κ spans three orders of magnitude across models and task pairs. The bracket thus provides a precise, coordinate-specific consistency fact and a warning signal for update-order sensitivity; sample-stable prediction of task effects remains open. 1 Introduction Many model-editing, merging, steering, and lightweight adaptation methods operate through relatively small perturbations around an already trained checkpoint, and each of them makes its own assumption about the local geometry there. Task arithmetic and model merging need functional additivity and controlled cross-task interference [20, 47, 49]; sequential fine-tuning needs the order of updates not to matter too much; random parameter search needs useful first-order projections at the proposal scale [13]; fixed task subspaces need tangent structure that is stable and estimable; activation steering needs weight-space changes to have predictable representation-space counterparts [45, 32, 44, 54, 48]. These assumptions are similar, but they are far from identical: LoRA’s low matrix rank [18], for instance, requires neither a low-rank stochastic-gradient covariance nor Euclidean flatness of its factor coordinates. Thus, natural questions arise: which approximation does each tool need, over what scale does it hold, and how should it be measured? In a companion paper [31], we asked which of these linear structures actually exist, and found a more subtle answer than we had expected: trained tasks can induce low-dimensional structure under specific per-task and trajectory-based estimators, but that structure is local and moving rather than global and static. Static task planes miss the recovery direction, the useful basis drifts within a hundred steps, and a best-of-N theorem that we have proved in [31] shows that isotropic random search picks up a dimension-independent amount of any useful direction, growing as 2logN 2 N in the budget, provided the loss is roughly linear in the perturbation. In [31], the linear regime was found empirically at one scale on one model, which left all quantitative questions open: how linear is the regime, how low-dimensional, how additive, and to what scale does each of these hold across models? These questions can lead to practical conclusions: if you want to merge two adapters, compose two steering vectors, or perform a first-order search, you need to know where the relevant approximation holds and where it begins to fail. In this work, our goal is to answer these quantitative questions. sst2boolqrtearchella0 θ_0⋆ θ 300300-step LoRA ssLLgradient grandom⋆ θ linear to grid edge (≳10−2 10^-2) ⋆ θ A θ_AB θ_B−ηA- _A−ηB- _B−ηB- _B−ηA- _AA→B θ_A→ BB→A θ_B→ Agap =η2(HBA−HAB)=η^2(H_Bg_A-H_Ag_B) (a) the fixed multitask operating point is not optimal for any component task (b) single-perturbation probe loss is first-order predictable at the measured scales (P1, P6) (c) sequential memoryless updates have a pair-dependent nonzero Lie bracket (P8) Figure 1: Single-update predictability versus pairwise order dependence. All parameter-space statements use the fixed LoRA factor coordinates of the optimizer. Figure 2: Extended loss sweeps at ⋆ θ (one seed, three models): per-task probe loss along ±^t± g_t (colored), two fixed random unit directions (gray), and the largest contiguous tested segment with At(s)<0.10A_t(s)<0.10 (dotted verticals). Protocol and registered predictions in Section 4. Specifically, we treat the local regime as a set of measurable properties and ask, for each: does it hold, to what scale does it hold, and is the answer the same across models? We consider eight properties (Section 3): local linearity, low-dimensionality, tangent-space stability, additivity, parameter-to-activation correspondence, local searchability, directional curvature, and update non-commutativity. We execute them through the same frozen harness (the same code, estimator settings, and data streams for every model) around one multitask operating point, on nine models spanning seven architecture families and two orders of magnitude in size. Most of these quantities have been measured somewhere in the literature, but on different models, tasks, and harnesses, so they cannot be compared against each other; measuring them together makes the evidence directly comparable. We have registered the property list, thresholds, and a development / test split before the sweep, reported every failure, and touched the three held-out models only once (see Appendix A for detailed provenance). Before any statistical results, let us illustrate the object of study directly: Figure 2 shows per-task loss profiles along each task’s own gradient at ⋆ θ . It is a one-direction view, and it gives a strongly anisotropic picture: along its own gradient, every task’s loss is first-order predictable (the linear term of the Taylor expansion dominates the measured change) up to shifts of 10−210^-2 and often far beyond, with a visible interior and a visible end, while along random directions the loss is flat until displacements an order of magnitude larger. Pairwise interactions between task updates are not shown in this figure; they are measured separately below (P4, P8), and most of the failures we find are pairwise. One important empirical finding of our work is that two natural properties of this neighbourhood hold on very different scales. The first is single-update predictability: along the tested directions, the change of the loss is well approximated by the first-order term ⊤g δ alone, and this approximation holds across the entire tested scale grid. The second concerns pairs of task updates: when two updates are applied in sequence, the result can depend on their order. The interesting part is that this order dependence becomes substantial at a scale that is specific to the model and the task pair and that can fall inside the range where single updates are still predictable. For two memoryless gradient steps, the leading order-dependent endpoint difference is the Lie bracket of the two tasks’ gradient vector fields, and with our endpoint normalization the defect is c(η)=ηκ+O(η2),κ=‖HBA−HAB‖A+B‖.c(η)=ηκ+O(η^2), κ= H_Bg_A-H_Ag_B _A+g_B . When both quantities are computed from the same frozen minibatches, the measured defect agrees with the leading term at median ratio 1.0021.002, so in fixed LoRA coordinates the onset of order dependence is set, for every model and task pair separately, by a cross-curvature that two Hessian-vector products can measure (Section 6). This agreement verifies the second-order expansion at the step sizes we actually use, and that two independent routes to the same quantity (finite differences of real two-step updates and Hessian-vector products) give the same answer. But a stronger hypothesis fails: in a separately preregistered test (estimate the bracket on one data sample, build the two update orders from a second, evaluate task effects on a third), the bracket did not predict the held-out effects (Section 6.4). Agreement on the same data therefore does not yield predictions across data. Thus, a shared first-order window makes local random search predictable, while pairwise task interactions often become fragile at much smaller, pair-specific scales. We call the operating point primed: it has been adapted to the task mixture, yet it still carries nonzero task-specific gradients, so every task leaves room for further local improvement. Controls that repeat the measurements at the raw pretrained checkpoint (Section 5.3) find first-order predictability and the commutator mechanism there as well; priming shifts the measured scales, while the mechanisms are present from the start. Table 1 collects the claims and their verdicts in one place, with pointers to where each is measured and qualified; we will refer back to it throughout the paper. Table 1: Claims at a glance: registered diagnostics and follow-up tests, their verdicts after all controls, and where each is measured and qualified. Claim under test Verdict Where P1: probe loss is first-order predictable along the gradient holds on every model median (a shared window), right-censored at the grid edge 10−210^-2 §5.2 P6: first-order scores rank random perturbations holds and calibrated for σ≤10−3σ≤ 10^-3 (in-sample); declines by 10−210^-2 §5.2 P7: curvature along g exceeds a random direction large ratio, confirmed against a 6464-direction distribution; sign not extremal §5.2 P2: mixture-gradient covariance is low-rank no rank plateau up to m=256m=256; the registered m=32m=32 bar was non-falsifiable (sanity check only) §5.1 P3: the task-gradient subspace is stable fails: rotation within 1616–8080 single-task steps §5.1 P4: activation additivity at full task-vector scale fails held-out (1/31/3); holds for α≲0.3α 0.3 on all models; the failures persist under effective-weight addition §5.1 P5: global mean-vector weight-to-CAA correspondence fails on every model median §5.1 the three radii σ1⋆σ _1, α4⋆α _4, η†η coincide registered comparison ill-posed; no shared scale on the common ruler §5.4 P8: the order defect follows c(η)=ηκc(η)=ηκ holds with shared minibatches on both sides (median ratio 1.0021.002); onsets span three orders of magnitude §6.2 median threshold products carry to full FT and 1313–1414B hold at the median level; the pointwise mid-range check fails at full FT §6.3 cross-fitted functional forecast from the bracket not validated: aggregate invalid (step-size rule), eligible subset fails both bars §6.4 λmax _ radius forecast on new models not validated: undecidable on two of three models; inside/outside search comparison failed §6.5 The remainder of the paper is organized as follows. Section 2 reviews related work and reconciles the atlas with the companion paper. Section 3 fixes the operating point, the coordinate convention, and the eight diagnostics. Section 4 takes a direct look at the neighbourhood shown in Figure 2, Section 5 reports the atlas across nine models and ends by putting every measured boundary on one displacement ruler. Section 6 studies the two-direction boundary: the Lie-bracket law, its verification, its extensions to full fine-tuning and 1313–1414B scale, and the cross-fitted forecast (that fails). Section 7 quantifies the first-order regime in terms of the loss headroom and search; Section 8 states what a practitioner can and cannot take from the atlas; Section 9 discusses limitations, and Section 10 concludes the paper. Experimental details and pre-registration are described in the appendices. 2 Related Work The companion paper. This work is the second paper of a line that treats small weight perturbations as displacements on the parameter manifold and studies the local structure that training installs [31]. The first paper established, on a controlled synthetic transformer and LoRA adapters with 1B-parameter experiments, that: • per-task gradient structure is locally low-rank but the task plane drifts, so no static subspace captures adaptation; • a best-of-N theorem makes random search dimension-independent inside a locally linear window, which the work [31] located empirically (a scale near 10−410^-4 on Qwen2.5-0.5B); • a weight move has an activation-space shadow which, in one gradient-step experiment, tracked a contrastive steering vector at cosine 0.580.58. In this work, we follows up on specific findings from the companion paper: we re-observe the tangent-plane motion (P3), test the companion’s best-of-N formula at fixed radius, where the recoverable share is only 2lnN/P 2 N/P, and study a two-update object not measured there. The companion’s per-task covariance evidence remains supported in its original scope, whereas P2 that we consider here tests a mixture-gradient covariance at another operating point; likewise, P5 shows that one global, example-averaged SST-2 activation shadow does not reproduce the earlier single-cell cosine as a model-level rule, without ruling out input-conditioned or subspace correspondences. Local low-dimensional structure. Li et al. [22] and Aghajanyan et al. [1] show that training succeeds in random low-dimensional subspaces, bounding the intrinsic dimension of adaptation. Mode-connectivity work [14, 10] and loss-landscape studies [23, 12] expose further low-dimensional weight-space structure. These are distinct from P2’s empirical rank of a mixture-gradient covariance: our control invalidates the original rank-≤32≤ 32 conclusion for that estimator, and it does not adjudicate the other notions of low dimensionality. Editing, merging, and additivity. Task vectors [20], soups [47], and TIES [49] motivate tests of composition, but simultaneous adapter composition and sequential optimization are different second-order objects, and we keep them separate: P4 measures activation additivity under one fixed probe, while P8 measures the antisymmetric order effect of two memoryless gradient steps. Recent PEFT work predicts pairwise merge retention from early alignment and representation signals [42], and partial linearization of adapter modules improves task-vector fusion [41]; both reinforce the need for pair-specific diagnostics without identifying P4 with P8. Sequential order and Lie brackets. The leading order effect HBA−HABH_Bg_A-H_Ag_B is the Lie bracket of the two gradient fields, and it is not new to this paper. Rukhovich et al. [34] derive the bracket for domain-order interventions, discuss the noncanonical Euclidean parameter metric, and test HVP-based loss predictions in bilingual language-model pretraining. Sweeney [39] use the same primitive to predict transfer order in instruction tuning, preference optimization, pretraining domains, and longer schedules. Our contribution is a normalized endpoint defect, its finite-step calibration across LoRA operating points and task pairs, and its comparison with the other local-validity diagnostics measured at the same point. The scope is memoryless local gradient updates: optimizer state can introduce a leading O(η)O(η) order term for fixed-clock momentum or AdamW [40]. LoRA coordinates and merging geometry. A LoRA update has non-identifiable factors: BA=(BQ)(Q−1A)BA=(BQ)(Q^-1A) for invertible Q. Euclidean norms, random directions, gradients, Hessians, and curvature ratios in factor space can therefore change under a function-preserving gauge transformation. Symmetry-aware merging work treats low-rank adapters on a quotient manifold for exactly this reason [8]. Our measurements describe the fixed factor coordinates used by the optimizer; intrinsic function-space geometry is a separate object, which we leave open. Activation steering and the parameter–activation bridge. Activation addition [45], contrastive activation addition [32], function vectors [44], RepE [54], and ReFT [48] steer in hidden-state space; ROME [25] edits weights. The two are linked by the pushforward of a weight move into activation space, which our companion paper [31] stated as an identity and tested directly in one gradient-step setting [31]. Here, no model-level median passes the registered threshold for one specific global mean-vector construction. This cautions against treating weight updates and CAA mean vectors as interchangeable, but does not test an input-conditioned operator, an aligned subspace, or steering efficacy. Random search and zeroth-order methods. Evolution strategies [36] and zeroth-order fine-tuning [24] optimize without gradients; Gan and Isola [13] use random parameter perturbation with top-K selection and majority-vote ensembling as evidence of a dense thicket of task-improving specialists around pretrained weights, the phenomenon that motivated our line of work. Our own companion paper [31] supplied a Gaussian best-of-N theorem; the present two-run check is consistent with its leading fixed-radius magnitude while making the 2lnN/P 2 N/P share of the gradient ceiling explicit. Hessian spectra and the bulk/outlier split. Empirical Hessian spectra of trained networks are often described as a low-rank outlier subspace, sometimes tied to class structure, plus a high-dimensional near-zero bulk [35, 29, 28, 15, 17]. Song et al. [38] and Nicolas et al. [27] argue that useful signal can live away from the sharp outliers. As a stylized detectability model, a planted low-rank update against a random bulk has a BBP phase transition [2, 30, 3, 46, 26]. We use this empirical split as background motivation for where the two-direction boundary might come from but do not study it directly. Sharpness and curvature. Flatness and sharpness motivate many generalization and optimization studies [21, 11], but sharpness is reparametrization-sensitive and not a standalone theorem of generalization [9]. Our P7 statistic is more narrow: it compares signed curvature along the mixture-probe gradient with one random Rayleigh quotient in fixed LoRA coordinates. 3 Setup and Diagnostics In this section we fix the operating point, the coordinate convention, and the eight diagnostics; everything later in the paper refers back to these definitions. Operating point. We study adaptation around a single operating point ⋆ θ : a LoRA adapter [18] (rank 1616, on the attention Q/K/V projections; exact per-family target modules in Appendix B) trained for 300300 steps of AdamW on a uniform mixture of five tasks (sst2, boolq, rte, arc-easy, hellaswag [6, 7, 51]) in full precision with eager attention. All eight properties are measured at ⋆ θ (for the two that require a trajectory, on a single-task path leaving ⋆ θ ), so that every property describes the same point of the same trained network. The raw pretrained checkpoint (the base point: the same network with the LoRA adapter at zero) is not the primary object, because the atlas is a map of adaptation around a trained point: task vectors, local search directions, and update order are all read from a point that has already absorbed the multitask mixture. Note that, unlike the companion paper [31], there is no forgetting/recovery recipe here: ⋆ θ is just “a trained network”, which keeps the setup uniform across models that do and do not forget under such recipes. Section 5.3 reports a control in which the battery is repeated at the raw base checkpoint; it measures what the 300300 AdamW steps change without claiming a unique temporal regime. We write ∈ℝP θ ^P for the trainable (LoRA) parameters, L()L( θ) for the mean next-token loss on a fixed probe set, =∇L(⋆)g=∇ L( θ ), and H=∇2L(⋆)H=∇^2L( θ ). The same fixed multitask probe set (four frozen batches of eight examples each, 3232 examples total, sampled task-uniformly from the training pools; Appendix B) defines L, g, and H, so that first- and second-order checks are mutually consistent; the one-direction properties use the mixture gradient for exactly this reason (Section 5.3 reports the per-task-probe control). Null comparators differ by property: P2 uses an isotropic reference, P3 a same-checkpoint resampling floor, P4/P5/P7 norm-matched random controls, and P6 a first-order prediction against realized Gaussian draws; P1 and P8 are threshold curves. Coordinate convention. There is one important convention that concerns every claim about the parameter space in the paper. All norms, isotropic draws, gradients, Hessian eigenvalues, directional curvatures, κ values, and displacement radii are Euclidean quantities in the fixed coordinates of the trained LoRA factors. If an effective update is ΔW=BA W=BA, then (B,A)(B,A) and (BQ,Q−1A)(BQ,Q^-1A) represent the same function for every invertible Q but need not have the same Euclidean geometry. Our results therefore describe the geometry as seen by the stated optimizer, not an intrinsic geometry of functions or effective weight updates. In particular, P4 adds factor-coordinate displacements rather than effective ΔW W matrices. Three distinct second-order objects. The diagnostics separate self-curvature from symmetric and antisymmetric cross-interactions; let us set up this division explicitly because it explains why the measured scales need not coincide. For a unit gradient direction, Taylor expansion gives A(σ)=σ|^⊤H^|2‖+O(σ2),A(σ)= σ\,| g H g|2 +O(σ^2), so P1 and the numerator of P7 deal with the same self-curvature through different routes. For a twice differentiable activation map, P4’s numerator begins with the symmetric mixed derivative α2D2[A,B]α^2D^2h[ δ_A, δ_B]. In contrast, P8’s order-sensitive endpoint begins with the antisymmetric Lie bracket η2(HBA−HAB)η^2(H_Bg_A-H_Ag_B). These are contractions of different tensors, evaluated through different observables, and they do not have to have the same scale. Informally speaking, a move along one direction feels its own curvature, two moves applied together feel the symmetric part of their interaction, and two moves applied one after the other feel the antisymmetric part. In this work, we try to keep these three questions separate, and the measurements below show that their scales are indeed different. Moreover, since P4 adds LoRA factor displacements, its effective weight contains bilinear cross terms δBAδAB+δBδAδ B_Aδ A_B+δ B_Bδ A_A that are absent when dense deltas of effective weights are added; a follow-up control reported with the P4 results (Section 5.1) separates this coordinate effect from network nonlinearity. The eight diagnostics. Each property is a statistic with a frozen threshold; we state them here in compact form, with estimator details in Appendix B. • P1 (local linearity). Along the unit gradient direction g, the antisymmetry ratio at scale σ is A(σ)=|L(⋆+σ^)+L(⋆−σ^)−2L(⋆)||L(⋆+σ^)−L(⋆−σ^)|,A(σ)= |L( θ +σ g)+L( θ -σ g)-2L( θ ) | |L( θ +σ g)-L( θ -σ g) |, the ratio of the symmetric (curvature) part to the antisymmetric (linear) part of the loss change; this is the same decomposition the companion paper [31] used to locate the linear regime. The frozen primary statistic is σ1⋆=maxσ:A(σ)<0.10σ _1= \σ:A(σ)<0.10\, the largest passing grid point. This is a lower-bound diagnostic: at the smallest scales the denominator can hit the numerical floor, and some seed curves fail there before passing at larger scales, so σ1⋆σ _1 should not be read as a contiguous-interval radius. The extended sweeps of Figure 2 support reading these small-scale irregularities as a noise floor: the profiles there are visually smooth and approximately antisymmetric on a contiguous central segment around zero. • P2 (low-dimensionality). From m per-minibatch mixture gradients at ⋆ θ we form the centered empirical covariance and report r90r_90, the number of components carrying 90%90\% of its energy. The substantive analysis is a sample-count sweep over m∈32,64,128,256m∈\32,64,128,256\ asking whether r90r_90 stabilizes as m grows (Section 5.1). • P3 (tangent-space stability). Along a single-task trajectory, let UtU_t be the estimated top-1010 task-gradient subspace and let ρ(U,V)=‖U⊤V‖F2/10ρ(U,V)= U V _F^2/10 be the normalized overlap (larger = more stable). At each checkpoint we compare ρ(U8,Ut)ρ(U_8,U_t) with the same-checkpoint resampling overlap ρ(Ut,Ut′)ρ(U_t,U _t); the rotation onset is the first checkpoint where the former falls at least 0.020.02 below the latter. • P4 (additivity). For task-vector pairs (A,B)( δ_A, δ_B) (short single-task fine-tunes from ⋆ θ ) and fraction α, the activation-space composition error is εadd(α)=‖(α(A+B))−(αA)−(αB)‖(αA)‖+‖(αB)‖, _ add(α)= (α( δ_A+ δ_B))-h(α δ_A)-h(α δ_B) (α δ_A) + (α δ_B) , where ()h( δ) is the change in the upper-medium-layer last-token activation induced by applying δ. P4 is a systematic test across models and task pairs, but under one fixed activation probe: the observable is the mean last-token hidden state at one layer on 1616 fixed SST-2 prompts, with one task-vector recipe and three registered pairs. It does not test general function additivity, logits, accuracy, or effective-weight addition. We report α4⋆α _4, the largest passing α on the grid under εadd<0.15 _ add<0.15, with α4⋆=1.0α _4=1.0 a max-grid lower bound. • P5 (parameter–activation correspondence). For a single gradient step =−η δ θ=- at η=10−4η=10^-4, the activation shift is the mean change in the upper-medium-layer last-token hidden state; we report its cosine to the labelled-contrast (CAA) steering vector [32] built at the same layer, taking the best of three candidate layers. The correspondence holds if this cosine exceeds 0.300.30. • P6 (local searchability). For isotropic Gaussian perturbations ∼(0,σ2I) δ (0,σ^2I) at per-coordinate scale σ=10−3σ=10^-3 (total norm σPσ P, of order 11 for these adapters; a different axis from P1’s unit-direction scale), we report the correlation between the realized gain L(⋆)−L(⋆+)L( θ )-L( θ + δ) and the first-order prediction −⊤-g δ, plus the fraction of beneficial draws. The regime is searchable if the correlation exceeds 0.50.5. • P7 (directional curvature). Via autograd Hessian-vector products, we compare signed curvature ^⊤H g H g along the mixture-probe gradient with ⊤Hr Hr along one random unit vector, subject to a finite-difference HVP reconstruction check; the frozen statistic is |^⊤H^/⊤H|>3| g H g/r Hr|>3. A single random Rayleigh quotient can be near zero through spectral cancellation, so the ratio indicates anisotropy relative to that one comparator; it does not estimate absolute sharpness. • P8 (non-commutativity). For one update of size η on task A then B versus B then A, the commutator defect is c(η)=‖A→B(η)−B→A(η)‖A→B(η)‖,c(η)= _A→ B(η)- _B→ A(η) _A→ B(η) , and we report the onset η†=minη:c(η)≥0.10η = \η:c(η)≥ 0.10\. P8 is a weight-space path property (do two sequential optimizations reach the same endpoint?), while P4 is an activation-space composition property; η is a per-task learning rate, and the induced parameter step has norm ≈η‖A+B‖≈η _A+g_B , again a different axis from P1’s σ. Protocol constants. LoRA rank 1616, α=32α=32, dropout 0, on the attention Q/K/V projections; 300300 AdamW steps at 2⋅10−42· 10^-4, training batch 88, sequence length ≤160≤ 160. The probe loss (defining L, g, H) is a fixed set of four eight-example multitask batches (3232 examples). P2 uses m=32m=32 per-minibatch gradients in the main battery, with a follow-up m-sweep at m∈32,64,128,256m∈\32,64,128,256\; P6 uses 256256 Gaussian draws; P3 tracks 200200 single-task steps with subspaces re-estimated at checkpoints 8,16,32,48,64,96,128,160,200\8,16,32,48,64,96,128,160,200\; P4 task vectors are 4040-step single-task fine-tunes swept over α∈0.03,0.1,0.3,1.0α∈\0.03,0.1,0.3,1.0\; P1 and P8 use the scale grid 10−5,3⋅10−5,10−4,3⋅10−4,10−3,3⋅10−3,10−2\10^-5,3· 10^-5,10^-4,3· 10^-4,10^-3,3· 10^-3,10^-2\ on their own axes; the task pairs for P4/P8 are (sst2,boolq), (sst2,rte), (boolq,rte). The battery uses three seeds per (model, property) pair. Throughout the paper we call one point of this measurement grid—either one (model, seed, task-pair) combination or a (model, seed) where no pair is involved—a cell. Full details, including the model list, prompt templates, probe construction, per-property estimator settings, layer choices, and hardware, are collected in Appendix B. Registration. This is a part confirmatory, part exploratory project. The property list, model split, keep/drop rule, and threshold bars were prospective; estimator details were frozen in code; the fine-grid spine studies and the controls were registered separately before their data was read; the headroom and common-ruler analyses are descriptive reanalyses of archived measurement records. We report all properties, including the ones that fail, and we touched the three held-out atlas models once, after the keep/drop decisions were frozen. Appendix A gives the provenance analysis by analysis, and Appendix D shows that the headline patterns are stable under nearby threshold choices. 4 A First Look at the Neighbourhood Before systematic measurements, let us look directly at the neighbourhood they will quantify. Figure 2 plots, for three models (two development, one held-out family) and all five tasks at ⋆ θ , the per-task probe loss along ± the task’s own unit gradient over an extended grid s∈[10−4,1]s∈[10^-4,1], reaching well past the atlas ceiling of 10−210^-2, so that both the start and the end of the linear window are visible. It also shows the loss along two fixed random unit directions, and multiple-choice accuracy probed at selected points; the dotted verticals mark, per task, the largest contiguous tested segment with antisymmetry ratio At(s)<0.10A_t(s)<0.10. The panels are easiest to read from the center outwards: both axes are signed log scales, so a first-order response appears as two roughly antisymmetric wings around s=0s=0, curvature shows up as the wings losing that symmetry, and the flat gray band through the middle is what “random directions do nothing” looks like at these scales. The protocol and its three predictions were pre-registered before the run; all three scored as hits. First, on every model the along-gradient loss is linear (At<0.10A_t<0.10) through s=10−2s=10^-2 on all five tasks (15/1515/15 cells), and it departs from linearity by s=0.3s=0.3 on 11/1511/15: the window has a visible interior and a visible end, with edges ranging from s=0.01s=0.01 to beyond the grid (HellaSwag on GPT-Neo is still linear at s=1s=1). Second, at s=10−2s=10^-2 the along-gradient loss change exceeds the change along each of two fixed random directions by 480×480×–4300×4300× per cell; the random curves are visually flat until |s|≳0.1|s| 0.1. (A one-seed directional comparison; it does not by itself validate P7’s single-denominator ratio.) Third, accuracy stays within ±2± 2p at the s=10−2s=10^-2 window edge on 13/1513/15 cells, as registered; the largest response is BoolQ on Pythia-410M, which moves +10+10p at downhill norm 0.010.01 and runs from 73%73\% downhill to 27%27\% uphill at |s|=0.1|s|=0.1, a 4646p spread between the two directions, while RTE on GPT-Neo gains +3.3+3.3p at the edge. Accuracy uses the first 5555–6464 usable validation examples per task, so a single example is worth 1.61.6–1.81.8p, and shifts of ±2± 2–33p (including the RTE one, which is two examples) are within counting noise; the BoolQ response is far outside it. Appendix C tabulates the per-cell accuracy results at every probed scale, in both directions. We can draw three lessons from this figure. The one-direction window is wide and its edges are heterogeneous: within a single model they vary by two orders of magnitude across tasks. Loss sensitivity and accuracy sensitivity can be very different: on most cells the loss slope moves accuracy by almost nothing at the window edge, while a single cell converts the same displacement into tens of points. Finally, every curve here is one direction at a time; failures that matter for task composition appear only when two updates are combined. 5 The Atlas Across Nine Models We run the battery on six development models spanning five families: DistilGPT-2 [19], Pythia-160M [4], Pythia-410M, GPT-Neo-1.3B [5], OPT-1.3B [53], and TinyLlama-1.1B [52]. We make all keep/drop decisions there, freeze them, and then run the same battery once on three held-out models: Pythia-1.4B (scale within a seen family), Qwen2.5-7B [50] (an unseen family at scale), and OLMo-7B-Instruct [16] (an unseen family on which activation steering transported poorly in the companion paper’s experiments). Table 2 shows the per-model medians for all eight diagnostics on all nine models; a star there records only that the originally registered numeric bar was passed, and the discussion below states what survived the controls. We discuss failures first. Table 2: Atlas measurements: per-model medians over seeds. A star records passage of the originally registered numeric bar only; nfnf = non-falsifiable (P2, superseded by the m-sweep), † = right-censored at the grid maximum, ‡ = coarse-grid onset refined later, in = failed HVP reconstruction check. Per-seed values in Appendix E. Development Held-out Diagnostic distil pyt-160m pyt-410m neo-1.3B opt-1.3b TinyLl-1.1B pyt-1.4B Qwen-7B OLMo-7B P1 linearity σ1⋆σ _1 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 10−2†10^-2 P2 low-dim r90r_90 (m=32m=32) 23nf23 nf 24nf24 nf 24nf24 nf 21nf21 nf 23nf23 nf 23nf23 nf 23nf23 nf 23nf23 nf 21nf21 nf P3 rotation onset (steps) 8080 1616 1616 3232 1616 1616 1616 1616 1616 P4 additivity ε(α=1) (α=1) 0.07∗0.07 0.14∗0.14 0.08∗0.08 0.190.19 0.350.35 0.14∗0.14 0.11∗0.11 0.320.32 0.320.32 P5 pushforward cos +.01+.01 +.01+.01 −.12-.12 +.05+.05 +.22+.22 +.04+.04 −.22-.22 +.01+.01 −.06-.06 P6 searchability corr .77∗.77 .74∗.74 .97∗.97 1.0∗1.0 .70∗.70 1.0∗1.0 1.0∗1.0 1.0∗1.0 .97∗.97 P7 curvature ||ratio|| 1.7e3in1.7e3 in 1.7e5∗1.7e5 3.1e4∗3.1e4 1.9e4∗1.9e4 7.0e4in7.0e4 in 2.1e5∗2.1e5 1.7e4∗1.7e4 4.0e4∗4.0e4 2.4e5∗2.4e5 P8 commutator η†η 10−510^-5 3⋅10−53·10^-5 3⋅10−33·10^-3 10−2‡10^-2\, 10−510^-5 10−210^-2 10−2‡10^-2\, 10−2‡10^-2\, 3⋅10−33·10^-3 5.1 What fails The mixture-gradient covariance shows no low-rank plateau. The substantive P2 question is whether the mixture-gradient covariance concentrates in a small subspace whose dimension stops growing once enough samples are seen. It does not: in a seed-0, seven-model sweep, r90r_90 grows 23→46→89→16923→ 46→ 89→ 169 as m grows 32→64→128→25632→ 64→ 128→ 256, with no plateau in range, while remaining more concentrated than a matched isotropic null (ratio 0.82→0.740.82→ 0.74; Figure 3). The originally registered bar, r90≤32r_90≤ 32 at m=32m=32, was satisfied automatically and carries no evidential weight: thirty-two samples can span at most thirty-one directions after centering, so asking whether they concentrate in thirty-two was never a question about the network, and the apparent universal value r90≈21r_90≈ 21–2525 in Table 2 is a sanity check of the implementation. The safe conclusion is estimator-specific: the apparent rank-≤32≤ 32 mixture-gradient structure was a sample-count artifact. This does not negate low matrix rank of LoRA updates, intrinsic optimization dimension, trajectory dimension, or the companion paper’s per-task covariance result (a mixture covariance can be high-dimensional even when every conditional task covariance is concentrated), and the within-task/between-task decomposition remains to be measured. A separately preregistered task-conditioned follow-up stopped at a failed whole-matrix numerical audit, before any scientific scoring; it yields no verdict, and we use none of its partial metrics. Figure 3: P2 sample-count control: r90r_90 against m for the seven models (thin lines, seed 0), their median (bold), and the matched isotropic null (dashed). Figure 4: P3 on the BoolQ trajectory (seed 0): overlap with the step-8 subspace against the same-checkpoint resampling floor. The tangent space rotates within tens of steps. P3 fails in its original “stability” form, and we expected that when registering it: the main result of the companion paper was precisely that the task plane moves. On the BoolQ trajectory where the diagnostic is available, model medians clear the same-checkpoint resampling floor within 1616–8080 single-task steps (Figure 4), after which a small core persists (asymptotic overlap 0.020.02–0.170.17). DistilGPT-2, the leftmost panel, is the slowest rotator in the battery, and even it reaches the floor by step ∼ 100100; note also that the floor itself sits well below 11, so part of the lost overlap is estimator variance, which is why the onset criterion compares drift to the floor. The exact timing depends on the seed (DistilGPT-2’s three seeds give 128/—/32128/---/32; Appendix E), so the supported claim is early rotation on this trajectory, not a universal clock. In practice, a method that assumes a fixed task subspace should recheck that assumption within tens of steps. The registered global mean-vector correspondence does not generalize. For P5, model-median cosines between one SST-2 gradient step’s mean activation shadow and one labelled-contrast CAA mean vector range from −0.22-0.22 to +0.22+0.22: no model median reaches the 0.300.30 bar, including all three held-out models. The companion paper’s 0.580.58 cell used another model, operating point, and prompt construction. The rejected hypothesis is also quite specific: a global, example-averaged activation shadow aligning with a global CAA mean vector. Individual seeds do exceed the bar (up to +0.34+0.34), and input-conditioned maps, aligned subspaces, and steering efficacy remain untested here. The pushforward x↦D(x;)x D_ θh(x; θ)\, δ θ is input-dependent, and averaging it into one vector can erase structure through cancellation; the cheapest version of the bridge does not survive averaging, while the companion’s input-level identity is untouched. Figure 5: P4 composition error εadd(α) _ add(α), per-model median over pairs and seeds (log–log); held-out models dashed, norm-matched random null in gray. Activation additivity fails at full task-vector scale on several models. Under P4’s fixed SST-2, one-layer activation probe, four of six development models pass at α=1α=1, but the registered held-out bar fails (1/31/3): both held-out 77B models have composition error 0.320.32, and GPT-Neo-1.3B and OPT-1.3B also fail the full-scale bar. Figure 5 shows the whole picture, and it is more regular than the pass/fail column suggests: on every model the error grows essentially linearly in α, as the symmetric mixed-derivative term predicts, so the models differ mainly in the coefficient, and “passing at α=1α=1” just means having a coefficient below the bar. Every model passes for α≲0.3α 0.3. Under this probe, additivity at full scale is not model-universal. A follow-up control separates the coordinate effect from network nonlinearity: it materializes each task’s effective-weight delta ΔW W and adds the matrices, so the factor cross terms vanish (the two routes agree on single tasks to relative error below 10−210^-2 on all 8181 cells). The coordinate choice accounts for part of the measured error, with the median ε(1) (1) dropping from 0.1800.180 to 0.1310.131, but not for the verdicts: of the five models whose factor-route median reaches the 0.150.15 bar in this rerun, only TinyLlama-1.1B falls below it under effective-weight addition. The full-scale additivity failures are mostly network nonlinearity, further inflated by the factor coordinates. 5.2 What survives Two first-order diagnostics pass their frozen bars on every model median, and one second-order diagnostic survives with qualifications. The loss is first-order predictable along the gradient. P1’s antisymmetry ratio stays below 0.100.10 out to the largest tested scale (σ=10−2σ=10^-2) on every model median, development and held-out alike. All nine medians are therefore right-censored at the grid edge and should be read as lower bounds; the extended sweeps of Section 4 show that the window does end farther out, between s=0.01s=0.01 and past s=0.3s=0.3 depending on the cell. The seed-level picture is rougher: one Pythia-160M seed has σ1⋆=10−4σ _1=10^-4, and eight of 2727 development seed curves are nonmonotone at the smallest scales, where the denominator approaches the numerical floor. The supported statement is therefore that the model-median mixture-probe curves remain first-order dominated at 10−210^-2, not that every seed is linear on a contiguous interval. Random-perturbation gains are first-order predictable at the tested scale. P6’s correlation between realized probe gain and −⊤-g δ is 0.700.70–1.001.00 across the nine model medians. This is the practical payoff of P1: within the window, a perturbation’s effect is essentially its projection onto the gradient, so the sign and size of a random proposal’s gain can be read off in advance, roughly half of isotropic draws carry a downhill component, and best-of-N selects the largest projection. Because the same 3232 examples define g and evaluate the gain, this is an in-sample Taylor-ranking check at one proposal scale: it shows that the probe loss is locally linear enough to rank random proposals. A follow-up sweep over the full scale grid maps the range: the correlation stays above the 0.50.5 bar at every σ≤10−3σ≤ 10^-3 on all 2323 measured cells, the calibration slope of gain against prediction lies in [0.5,1.5][0.5,1.5] on every cell for σ≤3⋅10−4σ≤ 3· 10^-4, and the correlation declines from 10−310^-3 to 10−210^-2 on all nine models (on OLMo-7B from 0.990.99 to 0.460.46). Generalization of score-selected perturbations to disjoint data remains untested; Section 7 adds a two-model descriptive magnitude check. Figure 6: P7 per-seed |^⊤H^/⊤H|| g H g/r Hr|; open circles mark seeds failing the finite-difference HVP reconstruction check, amber bars are per-model medians of ||ratio||. The mixture-probe curvature ratio is large but denominator-sensitive. On the seven model-level cells that pass the finite-difference HVP reconstruction check, P7’s absolute ratio to one random Rayleigh quotient ranges from 1.7⋅1041.7· 10^4 to 2.4⋅1052.4· 10^5; DistilGPT-2 and OPT-1.3B fail the check (Figure 6). What this establishes is a large ratio against this particular comparator, and no more than that. A random Rayleigh quotient of an indefinite Hessian can be tiny through sign cancellation, so the ratio does not separate genuine anisotropy from a small sampled denominator; it also varies by orders of magnitude across seeds, and the per-task control below rules out reading it as a task-invariant scalar. A follow-up with 6464 random directions per cell settles the magnitude question: on the seven models passing the reconstruction check, the task-direction ratio to the median random quotient is 1.4⋅1041.4· 10^4–5.9⋅1055.9· 10^5, and the more stable comparator ‖H‖ Hv gives 71×71×–590×590× against its median. The sign is a different matter: the signed task curvature exceeds all 6464 random draws on only four of the seven models (⋆ θ is not a minimum, and the task-direction curvature is negative on some seeds). The task direction is extreme in curvature magnitude, while not always the most positively curved direction. OLMo-7B-Instruct was chosen as a held-out model because activation steering had transported poorly on it in the companion paper’s experiments. It passes every surviving one-direction diagnostic and fails exactly the steering-adjacent P5: the one property tied to the activation bridge breaks on the model where that bridge already looked fragile. 5.3 Controls: the raw base point and per-task probes Two controls pin down what the multitask adaptation actually changes and how robust the diagnostics are to the probe choice. Is ⋆ θ special? We repeated half of the battery at the base point (the LoRA adapter set to zero, so the represented function is exactly the pretrained network). The frozen P7 statistic fails there (only 3/63/6 development models pass, none with acceptable across-seed CV), but the single-denominator limitation prevents treating that alone as a signature of priming. More telling is the commutator: the relation η†≈0.10/κη ≈ 0.10/κ is already present at base. The registered scoring finds 38/4238/42 finite cells, 35/3835/38 within a factor 1.51.5, median multiplicative error 1.0991.099, Spearman(η†,κ)=−0.994(η ,κ)=-0.994. Quantitatively, the same cells measured at the base point and at ⋆ θ differ by a median factor of 2.62.6 in η†η and 2.92.9 in κ (the registered bar for a substantial shift was a factor of 22). So multitask adaptation shifts the measured scales, while both mechanisms are present already at the base point. Per-task probes. Replacing the mixture probe by each single task (an additional control measured at the base point) leaves P1 and P6 stable: no model shows two or more task-level verdict flips, and no model has a task coefficient of variation above 11. P7 is the exception (every development model flips on at least two tasks with task CV above 11), so directional curvature depends on the probe and cannot be read as a task-invariant scalar. This is also why the protocol keeps one fixed mixture estimator for L, g, and H across all properties. 5.4 The boundaries on one ruler We registered a prediction that the three “radii” (P1’s σ1⋆σ _1, P4’s α4⋆α _4, and P8’s η†η ) would locate the same scale within a factor of three. As stated, the prediction cannot even be tested: the three live on different axes (unit-direction norm, task-vector fraction, learning rate), so the registered comparison was ill-posed. The conversion matters because learning rates are not comparable across models: a step of η=10−3η=10^-3 on one model can displace the weights more than a step of 10−210^-2 on another once the gradient norms differ by an order of magnitude, as they do here. A meaningful comparison therefore needs one ruler, and we use the Euclidean norm of the induced weight-space displacement: a P1 step of scale σ has norm σ; a P8 update pair at onset η†η displaces the parameters by r8=η†‖A+B‖r_8=η _A+g_B to leading order, and both factors are archived per cell; a P6 draw has norm σPσ P. (P4 is the one boundary this conversion cannot yet place: task-vector norms were not archived.) Figure 7: All measured boundaries on the common displacement ruler, models ordered by median κ: per-cell two-direction onsets r8=η†‖A+B‖r_8=η _A+g_B (circles seed 0, triangles seed 1), the right-censored one-direction P1 bound (dashed vertical), and P6 draw norms σPσ P (stars). Figure 7 collects every measured boundary on this ruler. On the fine-grid cells of Section 6 (seven models, two seeds, three pairs; 4141 measured onsets, one of them left-censored), the two-direction onset displacement spans r8∈[3.6⋅10−4, 0.24]r_8∈[3.6· 10^-4,\,0.24] across (model, pair) cells, at least a factor of 650650 (the smallest value belongs to the left-censored cell, so its true onset may be smaller still); the pooled median is 1.8⋅10−21.8· 10^-2, and per-model medians run from 8.8⋅10−48.8· 10^-4 (Pythia-160M) to 0.130.13 (Qwen2.5-7B). Relative to the one-direction window, whose measured lower bound is 10−210^-2 on every model, 1515 of 4141 cells break commutativity at displacements strictly inside the window (on Pythia-160M the median onset sits 11×11× inside); for the remaining 2626 cells the relative position is undecided because σ1⋆σ _1 is right-censored. The P6 draws measure 0.50.5–3.63.6 on the same ruler (54×54×–355×355× the P1 bound), and first-order scores stay predictive there. The neighbourhood is thus strongly anisotropic: single-direction moves remain first-order predictable out to displacements one to two orders of magnitude beyond the point where, on over a third of the tested cells, two-direction ordering has already broken. And the two-direction thinness is a per-pair quantity: within one model and seed, the onset can move by an order of magnitude when the task pair changes. The next section asks what sets it. 6 The Two-Direction Boundary Is a Curvature Commutator The atlas separates single-perturbation predictability from two-update order sensitivity; this section is about the mechanism of the latter. We first derive the leading term, then check how accurately it reproduces the measured defect when both sides are computed from the same minibatches, then push it beyond the LoRA operating point, and finally report a preregistered cross-fitted test of a stronger functional hypothesis, which fails. As background, the Hessian of a trained network is often described as a low-rank, curved outlier subspace plus a high-dimensional, near-flat bulk [35, 29], with different views on where the learning signal lives relative to that split [17, 38, 27]. Our own observation below is more narrow: the bracket vector is strongly enriched in the top Hessian subspace relative to the isotropic floor, on the cells that pass the HVP reconstruction check. The bulk/outlier picture serves here as motivation, and we do not study it directly. 6.1 The leading term Assume LAL_A and LBL_B are C3C^3 in a neighbourhood of ⋆ θ , and take one gradient step of size η on each of two tasks, in the two orders. Expanding the second step around ⋆ θ , A→B(η) _A→ B(η) =−ηA−ηB(⋆−ηA)=−η(A+B)+η2HBA+O(η3), =-η\,g_A-η\,g_B( θ - _A)=-η(g_A+g_B)+η^2H_Bg_A+O(η^3), B→A(η) _B→ A(η) =−η(A+B)+η2HAB+O(η3), =-η(g_A+g_B)+η^2H_Ag_B+O(η^3), where X=∇LX(⋆)g_X=∇ L_X( θ ) and HX=∇2LX(⋆)H_X=∇^2L_X( θ ). The first-order parts cancel, leaving the purely second-order commutator A→B−B→A=η2(HBA−HAB)+O(η3), _A→ B- _B→ A=η^2\,(H_Bg_A-H_Ag_B)+O(η^3), the Lie bracket of the two update fields. In words: whichever task goes second takes its step from ground the first task has already moved, so its gradient has changed by curvature times the first step; if HBAH_Bg_A and HABH_Ag_B differ, the two orders land in different places, and their leading difference is the bracket. When ‖A+B‖>0 _A+g_B >0, dividing by ‖A→B‖≈η‖A+B‖ _A→ B ≈η _A+g_B gives the defect to leading order: c(η)=ηκ+O(η2),κ=‖HBA−HAB‖A+B‖.c(η)\;=\;η\,κ+O(η^2), κ\;=\; H_Bg_A-H_Ag_B _A+g_B . Two remarks before the measurements. First, the normalization is ill-conditioned when A≈−Bg_A≈-g_B: a large κ can reflect a large bracket numerator, first-order cancellation, or both, and the present tables do not decompose the two. Second, κ=0κ=0 kills the leading term only; higher-order order-dependence can survive. Once the coordinate system, probe estimator, normalization, and defect threshold 0.100.10 are fixed, the leading-order onset estimate has no fitted coefficients: η†⋅κ≈ 0.10.η ·κ\;≈\;0.10. The bracket itself and HVP-based order prediction are established in directly overlapping work [34, 39], alongside backward-error analyses of sequential updates [37, 33]. What we claim here is only the measurement: the normalized endpoint defect across a model–task matrix, its finite-step calibration in fixed LoRA coordinates, and its comparison with the other atlas diagnostics. The derivation covers memoryless gradient updates; fixed-clock optimizer state can add a leading O(η)O(η) term [40]. The test. For each model we measure κ from two Hessian-vector products at ⋆ θ , and we compute the top-1010 Hessian subspace by Lanczos to ask whether the commutator vector is enriched there relative to the random floor 10/P10/P. DistilGPT-2 and OPT-1.3B are excluded from all commutator measurements by the P7 reconstruction check. We made two pre-registered predictions: η†⋅κ≈0.10η ·κ≈ 0.10 within a factor of three, and top-subspace overlap far above the floor. The fine-grid re-measurement protocol below and its metrics were likewise registered before any of its data was read. Figure 8: Fine-grid defect curves for all 4242 (pair, seed) cells on the seven models passing the HVP reconstruction check: (a) raw c(η)c(η); (b) the same curves against ηκηκ, collapsing onto the identity (dashed). Table 3: The commutator test, per-model medians over (pair, seed) cells; the product column is the median of per-cell products. cosUcomm U_ comm is the commutator’s top-1010 Hessian-subspace energy (random floor 10/P10/P). Model κ η†η η†⋅κη \!·\!κ cosUcomm U_ comm pythia-160m 37103710 2.9⋅10−52.9· 10^-5 0.0960.096 0.0720.072 pythia-410m 9898 1.2⋅10−31.2· 10^-3 0.1090.109 0.0090.009 neo-1.3B 2.92.9 2.9⋅10−22.9· 10^-2 0.0990.099 0.1650.165 TinyLlama-1.1B 19.619.6 5.4⋅10−35.4· 10^-3 0.1020.102 0.1200.120 pythia-1.4B 9.79.7 1.2⋅10−21.2· 10^-2 0.0990.099 0.0610.061 Qwen2.5-7B 3.43.4 3.1⋅10−23.1· 10^-2 0.1010.101 0.0130.013 OLMo-7B 37.137.1 2.8⋅10−32.8· 10^-3 0.1000.100 0.0280.028 median (7 models) 0.1000.100 6.2 Verification under a matched estimator Our first measurement, on P8’s shared coarse grid with single-batch one-step gradients, gave only an order-of-magnitude law: median η†κη κ at the predicted value but per-model scatter of about 10×10×, with 44 of 77 models within the registered factor of three. We then pre-registered and ran a re-measurement designed to remove two possible causes of that scatter: a 2525-point η grid from 10−510^-5 to 10−110^-1 with the onset read by log-interpolation of the crossing (removing grid quantization and, on all but two cells, censoring), and the same fixed three-batch gradient estimators driving both the two-step defect and the HVP commutator (removing estimator mismatch). Here “matched” means that both routes see the same data: the batches that build the two-step paths also build the HVPs, so sampling noise cannot show up as route-to-route disagreement. In this version the scatter collapses. Figure 8 shows the measured object itself: every cell’s defect curve is a unit-slope line through its own onset, the onsets spread over three orders of magnitude (from 1.7⋅10−51.7· 10^-5 to 5⋅10−25· 10^-2), and rescaling each curve by its own κ collapses all 4242 onto the identity, which the curves leave only at large defect, where higher-order terms take over. Table 3 gives the per-model medians: every product η†κη κ lies in [0.096,0.109][0.096,0.109], within 9%9\% of the threshold value. Because η†η is read from the same defect curve, η†κ≈0.10η κ≈ 0.10 is arithmetic once the leading expansion holds pointwise; the substantive content is route-to-route consistency. Away from the threshold, the finite-difference endpoint defect matches the HVP-side leading term ηκηκ at median ratio 1.0021.002 (IQR [0.999,1.013][0.999,1.013]), with 94%94\% of 218218 mid-range points within 10%10\%. The shared batches make this a stringent numerical check: the expansion is accurate in range, the HVP implementation agrees with finite differences, higher-order terms are small, and float32 effects are controlled. It is not an out-of-sample prediction. The cross-fitted test of Section 6.4 measures the out-of-sample version directly, and it fails. Figure 9: Measured onset η†η against the two-HVP prediction 0.1/κ0.1/κ for the 4141 measured fine-grid cells, colored by task pair (circles seed 0, triangles seed 1); the band marks ×1.5× 1.5. Table 4: Within-seed pair spans of κ and η†η , and per-model prediction errors of 0.10/κ0.10/κ on the 4141 measured cells. model κ pair span η†η pair span median pred./meas. max pred./meas. OLMo-7B 11.8×11.8× 12.8×12.8× 1.049×1.049× 1.343×1.343× Qwen2.5-7B 4.8×4.8× 4.9×4.9× 1.007×1.007× 1.042×1.042× TinyLlama-1.1B 6.7×6.7× 6.7×6.7× 1.017×1.017× 1.059×1.059× neo-1.3B 9.3×9.3× 8.8×8.8× 1.009×1.009× 1.058×1.058× pythia-1.4B 15.3×15.3× 14.9×14.9× 1.010×1.010× 1.045×1.045× pythia-160m 5.6×5.6× 4.0×4.0× 1.109×1.109× 2.415×2.415× pythia-410m 9.1×9.1× 9.6×9.6× 1.085×1.085× 1.311×1.311× The pairwise view carries the most information (Figure 9, Table 4). Task pairs are not interchangeable: within a single model and seed, κ spans up to 15×15× across the three registered pairs, and the measured onsets move with it; in Figure 9 this is visible as the color separation inside each model’s cluster. Across the 4141 cells the pooled rank correlation between η†η and κ is −0.998-0.998; 36/4136/41 predictions η^=0.10/κ η=0.10/κ land within ×1.2× 1.2 of the measured onset and 40/4140/41 within ×1.5× 1.5 (median multiplicative error 1.0201.020), with the single outlier left-censored at the grid edge, so its apparent factor-2.42.4 error is conservative. Pair-to-pair variation therefore strengthens the result: some task pairs commute longer than others because their cross-curvature is smaller, and two HVPs suffice to know which is which. A registered decomposition run settles where the original scatter came from. Re-measuring on the fine grid but with the original single-batch defect estimators (HVP side unchanged), the per-cell scatter returns to the coarse level (0.260.26, versus 0.0090.009 matched and 0.240.24 coarse): grid quantization contributed almost nothing, and the scatter was essentially all minibatch noise of the probe. Even then the median-level prediction survives (all seven models’ median products land within a factor of 1.71.7 of 0.100.10, at 0.0610.061–0.1610.161), so the two-HVP commutator gives a useful warning signal even with a noisy single-batch probe, while per-instance predictions inherit the probe’s variance. The second registered prediction also holds: on all seven models the commutator vector’s top-1010 Hessian eigenspace overlap is three to five orders of magnitude above the 10/P10/P floor (the lowest cell is 637×637× floor), although the absolute top-1010 energy can still be small (Table 3). We also tested whether the degree of top-eigenspace engagement modulates the onset beyond κ itself, and found nothing (partial rank correlation +0.000+0.000 controlling for κ), though with the pooled correlation at −0.998-0.998 there is almost no residual variance left for a second predictor. 6.3 Beyond the LoRA operating point: full fine-tuning and 13–14B scale Everything above lives at a LoRA operating point. A registered control repeats the fine-grid protocol with ⋆ θ trained by full fine-tuning (all parameters, lr 2⋅10−52· 10^-5, same recipe) on the three models small enough for full-parameter Hessian work (Pythia-160M/410M, TinyLlama-1.1B), with the grid extended to 10−710^-7 because full-space commutators are much larger. That extension is a finding in itself: κfull/κLoRA _ full/ _ LoRA has median 44 and reaches 6060, so full-fine-tuning order sensitivity sets in earlier than matched LoRA order sensitivity. The threshold-level product carries over at the median: η†κ=0.090η κ=0.090, 0.0970.097, 0.1160.116 on the three models, with the 1818 finite cell products spanning 0.0680.068–0.2170.217, no censoring, and 94%94\% single crossings; the commutator stays aligned with sharp Hessian directions (model-median overlap at least 105×10^5× the floor, with P now the full parameter count). The pointwise check away from the threshold does not pass this control: the measured-to-predicted ratio there is 1.561.56, outside the registered [0.9,1.1][0.9,1.1] band. We believe this failure is instrumental: the band was calibrated at LoRA-scale κ, and at full-FT κ it emphasizes very small η near the float32 arithmetic floor (the pooled measured/predicted ratio is largest at the smallest grid values and settles near one by η=10−4η=10^-4). Still, the registered check failed. At the crossing itself the signal dominates the floor, which is why the threshold products are unaffected. We also extended the LoRA protocol to larger base models, Qwen2.5-14B and OLMo-2-13B [43]. This gives six finite observed crossings with η†κ∈[0.098,0.103]η κ∈[0.098,0.103] across the two models and two seeds; the remaining six pairs are right-censored at the upper grid edge, so this is scale-consistency evidence on the finite cells, and the censored pairs remain undecided. (A third registered scale cell, Llama-3.1-Tulu-3-8B, was dropped before scoring after a pinned-environment tokenizer failure; no Tulu number is reported.) 6.4 A cross-fitted functional forecast fails The matched experiment asks whether two routes agree when they share estimators. We separately preregistered the stronger claim a practitioner would actually want: a bracket estimated on one sample should predict an endpoint built from a second sample and signed task effects evaluated on a third. This three-sample design mimics how the diagnostic would actually be used: geometry estimated today, updates built from tomorrow’s batches, effects checked on data neither step has seen. For geometry sample G, update sample U, and evaluation sample E, the registered quantities are G _G =HBGAG−HAGBG, =H_B^Gg_A^G-H_A^Gg_B^G, U _U =A→BU−B→AUη2, = θ_A→ B^U- θ_B→ A^Uη^2, y^C y_C =(CG)⊤G, =(g_C^G) b_G, yC y_C =LCE(A→BU)−LCE(B→AU)η2, = L_C^E( θ_A→ B^U)-L_C^E( θ_B→ A^U)η^2, tested on four new confirmation models (GPT-2-large, OLMo-2-7B, Mistral-7B-v0.3, Qwen2.5-14B), three seeds, all ten unordered pairs of the five tasks, five evaluation tasks per pair, with disjoint 6060/6060/128128-example blocks per task and the step η=0.05/κGη=0.05/ _G clipped to [10−5,3⋅10−2][10^-5,3· 10^-2] (clipped cells are excluded from the primary aggregates). The registered rules required absolute predictive power and step-size eligibility; a favorable pair-specificity comparison could not rescue a failure against the zero baseline. All twelve runs passed a frozen check of the raw outputs before the single scoring pass. Table 5: The preregistered cross-fitted confirmation at a glance. Registered component Fresh confirmation result Verdict Raw-output validity 12/1212/12 runs; 18001800 scalar checks pass pass Instrument eligibility 73/12073/120 pair cells limited (60.8%>25%60.8\%>25\%) invalid Prediction R2R^2 against zero Rzero2=−560.419R^2_ zero=-560.419; 1/41/4 models positive fail Endpoint direction median cosine 0.04290.0429; required >0.50>0.50 fail Pair specificity beats shuffled pairs: Rzero2>−561.470R^2_ zero>-561.470, cosine >0.00433>0.00433 pass Same-sample diagnostic endpoint cosine 0.99870.9987; functional Rzero2=−21.337R^2_ zero=-21.337 diagnostic Primary forecast three required bars fail not validated The forecast fails, in two ways (Table 5). The first is a matter of step size: only 47/12047/120 primary pair cells stay inside the declared interval (these are the eligible cells), so the preregistered rule calls the aggregate invalid (it is not a clean confirmatory negative either): the registered step scale is infeasible for most (model, pair) cells. But conditional on the eligible cells, both absolute prediction bars also miss by wide margins: Rzero2=−560.4R^2_ zero=-560.4 (the explained variance of the fixed prediction, measured against the zero baseline; negative values mean that predicting zero would have been better) and median endpoint cosine 0.0430.043. Scoring each prediction against its own pair does slightly better than scoring it against a cyclically shuffled pair (a control in which every prediction is matched with the wrong pair’s measured effect), so the bracket carries some pair-specific signal, but neither beats the zero-effect baseline. The same-sample diagnostic sharpens the picture: when geometry and update come from the same sample, the bracket reconstructs the endpoint direction almost exactly (cosine 0.99870.9987), while across samples the median endpoint cosine is 0.0430.043; even the endpoint geometry fails to transfer between update samples, before any functional question arises, and the matched functional score is negative as well (Rzero2=−21.3R^2_ zero=-21.3). The narrow conclusion is therefore twofold: the matched Lie-bracket calculation is a strong endpoint Taylor check, and this cross-fitted implementation supplies no evidence for a sample-stable predictor of either endpoints or effects. Why the registered step scale is so often infeasible, and whether any signal survives better step policies, is the next question this line of work has to answer. 6.5 An exploratory association, and a forecast that failed Across the seven model medians that pass the HVP reconstruction check, κ is rank-associated with the probe-loss λmax _ at Spearman +1.0+1.0 with log–log slope 0.850.85 (Figure 10). This looks like a useful screening heuristic: one Lanczos λmax _ estimate per model instead of two HVPs per pair. It is, however, a seven-point, within-atlas association with model-median κ; it shares probe data with the quantity it predicts; it omits the pair-specific variation of Table 4; the scoring script’s λmax _ reproduction check flagged one seed-level mismatch (Pythia-160M seed 1: 5972.665972.66 vs 6372.726372.72, just outside the 5%5\% tolerance); and the Hutchinson trace estimator failed its registered stability check at both 88 and 6464 probes (CV above 0.50.5 on 10/1410/14 cells, negative mean traces persisting on two models), so we cannot distinguish outlier summaries from bulk-inclusive curvature. At an operating point that is not a loss minimum, the trace can be a small signed difference of large positive and negative spectral mass; a bulk-only or absolute-value summary is needed, and we leave it open. The 1313–1414B cells are an exploratory overlay only: their finite threshold products stay in [0.098,0.103][0.098,0.103], but their κ values sit up to ×5.5× 5.5 below the seven-model fit, so they do not strengthen the association. Figure 10: Model-median κ against the probe-loss λmax _ on the seven models passing the HVP reconstruction check (log–log). We then froze the seven-model fit and applied it once, prospectively, to three new models (GPT-2-medium, OPT-350M, Qwen2.5-1.5B). The forecast did not validate. Qwen2.5-1.5B had a finite onset within ×1.28× 1.28 of the forecast, but the other two models were left-censored at the lower grid edge, leaving the registered radius verdict undecidable; and the paired comparison of first-order score correlations inside versus outside the forecast radius failed outright (GPT-2-medium 0.8820.882 vs 0.8710.871; Qwen2.5-1.5B 0.9990.999 vs 1.0001.000). The one-direction/two-direction explanation for that null is post hoc. We therefore retain λmax _ as an exploratory within-atlas association; the deployable local diagnostic remains the pair-specific two-HVP κ, within the matched-protocol scope established above. 7 Descriptive Headroom and Search-Scale Checks The atlas established that single perturbations are first-order predictable; this section quantifies what that gives, in nats and in search terms. Both measurements here are descriptive reanalyses of archived measurement records (no thresholds, no selection), and neither establishes value created by multitask priming. Throughout, headroom is the measured reduction in the probe’s token-mean cross-entropy (nats per active continuation token) along one direction over the tested grid σ≤10−2σ≤ 10^-2. On all but one (model, task) cell the P1 window is right-censored at that edge, so those values are also within-window lower bounds; the exception is ARC-Easy on Pythia-160M, whose median window ends at 3⋅10−33· 10^-3, inside the grid. Per-task headroom is a gradient-norm map, with a wide task span. Table 6 and Figure 11 report, for each development model and task, the loss reduction along that task’s own gradient over the tested grid, at the raw base checkpoint where the per-task battery ran (median over three seeds). Across cells the values range from 0.0020.002 to 1.01.0 nats per token (0.05%0.05\%–20%20\% of the corresponding probe loss), and the within-model task span is 1515–35×35×. Within a shared window this ordering is just the per-task gradient-norm ordering, which is what makes it predictable in advance: to first order the reachable reduction is slope times window width, the slope is the gradient norm, and the window is shared, so ranking the tasks costs one gradient evaluation each. Two scoping notes temper it: continuation lengths, tokenizations, and choice counts differ across tasks, so raw cross-entropy is not a common functional currency; and the table is measured at the raw base checkpoint. At ⋆ θ itself, the mixture-probe headroom is 0.0070.007–0.0470.047 nats per token (0.2%0.2\%–2.0%2.0\% of probe loss) across the nine models: residual signal exists at the operating point, but per-task value there was not measured. Table 6: Per-task along-gradient headroom (nats per active continuation token) at the raw base checkpoint, median over three seeds; “span” is the max-to-min task ratio. Values are measured over the full tested grid, right-censored on all but one cell (see text). model sst2 boolq rte arc-e hella span TinyLlama-1.1B 0.3350.335 0.1540.154 0.1430.143 0.0290.029 0.0100.010 35×35× distilgpt2 0.0310.031 0.0280.028 0.0350.035 0.0060.006 0.0020.002 19×19× neo-1.3B 0.0530.053 0.0280.028 0.0350.035 0.0080.008 0.0020.002 23×23× opt-1.3b 0.3830.383 0.1540.154 0.3560.356 0.0480.048 0.0120.012 31×31× pythia-160m 0.3410.341 0.8180.818 0.4330.433 0.0670.067 0.0270.027 30×30× pythia-410m 0.3330.333 0.2250.225 0.3460.346 0.0610.061 0.0230.023 15×15× Figure 11: Base-point per-task along-gradient loss reduction over the tested P1 grid (median over three seeds, log axis); right-censored on all but one cell (see text). Two best-of-256256 runs land near the Gaussian order-statistic scale. Under P6 the same 3232 examples define both g and the realized gain, so the high correlations are in-sample; what we can check descriptively is the magnitude that best-of-N search extracts. For an isotropic draw of fixed radius r, the first-order gain is −⊤-g δ with normalized projection approximately (0,1/P)N(0,1/P), so the leading expected best-of-N gain is ΔLbest-of-N≈‖r2lnNP, L_best-of-N\;≈\; \,r\, 2 NP, the fixed-radius specialization of the companion theorem [31, Theorem 1]. At N=256N=256 the leading term overstates the exact standard-normal expected maximum (2.82692.8269 vs 2ln256=3.3302 2 256=3.3302, about 18%18\% high), so we correct by that factor. On GPT-2-medium and Qwen2.5-1.5B the search samples on the sphere of radius r=ηforecast‖r= _ forecast ; the realized best-of-256256 boolq gains are 2.8⋅10−42.8· 10^-4 and 2.2⋅10−42.2· 10^-4 nats per token against predictions of 3.5⋅10−43.5· 10^-4 and 2.6⋅10−42.6· 10^-4, so the realized-to-predicted ratios are about 0.790.79 and 0.850.85. These two runs are descriptive; a calibration law would need multiple N, radii, and seeds. A rerun of both cells with accuracy recorded answers the conversion question at these radii: the selected perturbations leave multiple-choice accuracy exactly unchanged, on the target and on all four side tasks, on both models, at every multiplier up to 4×4× the forecast radius (112112 and 118118 usable target examples). At gains of ∼2⋅10−4 2· 10^-4 nats, the search headroom is real in loss and invisible in accuracy. The more important content is the 1/P1/ P factor itself: at fixed radius, undirected search receives only a 2lnN/P 2 N/P share of the along-gradient ceiling, about 0.2%0.2\% here. First-order scoring is therefore most relevant for constrained or externally generated proposal libraries: block-restricted edits, side-task-filtered candidates, discrete proposal sets. When an unconstrained gradient step is available, it wins. Cross-fitted scores and held-out selection regret remain unmeasured. 8 What the Atlas Means in Practice No single safe step size falls out of the atlas. What it gives a practitioner is a way to decide which local approximation a given adaptation tool is actually asking for, and what evidence exists for it. Match the diagnostic to the operation. For one intended direction, P1 checks probe-loss antisymmetry over a scale grid, and P6 checks first-order ranking at its tested proposal scale on the same probe. Together they are the reason local random search is usable here: linearity makes a proposal’s gain equal to its gradient projection, so candidates can be scored and ranked before any of them is run, and best-of-N then amplifies the best signal. For two sequential memoryless gradient steps, the pair-specific bracket κ gives the leading coordinate-space order defect, with η†≈0.10/κη ≈ 0.10/κ under the shared-minibatch protocol of Section 6.2; two HVPs per pair are enough for the warning signal, and pair-to-pair differences are large enough to matter. Simultaneous adapter merging is a different second-order object: P4 supplies a narrow activation-probe diagnostic (with error growing ∝α α), and this paper has no validated commutator-based merge-radius calculator. What not to use as a marker. The registered rank-≤32≤ 32 mixture-covariance test was invalid, and its control shows no plateau through m=256m=256: do not read mixture-gradient rank at small sample counts as evidence of low-dimensional adaptation. The global mean-vector P5 bridge has no passing model median: do not treat weight updates and CAA mean vectors as interchangeable, even though input-conditioned correspondences remain open. The measured BoolQ tangent subspace rotates within tens of steps: do not carry a task subspace across a trajectory without rechecking it. Each of these failures is specific to one cheap construction; stronger versions (per-task covariance estimates, input-conditioned correspondences, refreshed subspaces) remain untested. The bracket’s record. Endpoint consistency with shared minibatches: verified (median ratio 1.0021.002). Model-median threshold products across LoRA, full fine-tuning, and 1313–1414B scale: 0.0900.090–0.1160.116, with full-FT cell-level products spanning 0.070.07–0.220.22. Out-of-sample λmax _ radius forecast: undecidable on two of three models, failed inside/outside search comparison. Cross-fitted prediction on new models: aggregate invalid under the step-size rule, eligible subset strongly negative. The pattern is consistent: the local, matched-estimator statements hold, and every transferable one has so far failed. A practitioner should use κ as a cheap, pair-specific, same-probe order-sensitivity diagnostic, and should not use it as a cross-sample effect-size predictor. 9 Discussion and Limitations Empirical conclusions. Individual perturbations remain first-order predictable on the measured probe, through the tested grid on every model median, while interactions between task updates produce substantial, pair-specific order effects at scales that can sit well inside the one-direction window; this scale gap is the robust organizing fact of the atlas. The interactions decompose cleanly by symmetry: self-curvature governs one-direction antisymmetry (P1, P7), a symmetric mixed derivative governs simultaneous activation composition (P4, with its predicted ∝α α error growth visible in the data), and the antisymmetric Lie bracket governs memoryless update order (P8), with no shared scale among the three. For the bracket, our contribution is the normalized finite-step atlas in fixed LoRA coordinates: the law c(η)=ηκc(η)=ηκ holds pointwise at median ratio 1.0021.002 when both sides share minibatches, the onsets it predicts span three orders of magnitude across (model, pair) cells, and the median threshold products persist at full fine-tuning and 1313–1414B scale. The stronger hypotheses we tested did not survive: the disjoint-sample forecast is invalid under its step-size rule with a strongly negative eligible subset, and the λmax _ radius forecast was undecidable on two of its three models, with the inside/outside search comparison failing as well. We call ⋆ θ primed only to record that it has seen the mixture and retains residual gradients; the base-point controls show the mechanisms are present without it. Limitations. The most important limitations of our study are as follows. 1. Coordinate and optimizer scope. LoRA factor coordinates have gauge freedom, and every Euclidean parameter-space quantity in this paper is gauge-dependent. The bracket expansion describes memoryless gradient steps; fixed-clock momentum or AdamW state can introduce O(η)O(η) order effects. No gauge-rescaling, balanced-gauge, Fisher, or output-KL control is yet reported (the effective-weight control for P4 now is; Section 5.1). 2. Operating point and probe. The battery is measured at one LoRA operating point on a five-task classification mixture, not open-ended generation; P3 is a BoolQ single-task trajectory, and P4/P8 use three registered pairs among sst2, boolq, rte, so the rotation and two-direction claims should be read at that scope. The extended sweeps and their random-direction comparison cover three models and one seed. 3. Estimator and cross-fit scope. P6 scores and evaluates on the same 3232 examples, and the 1.0021.002 commutator ratio uses matched batches on both routes. The cross-fitted confirmation avoids this overlap but is formally invalid because 60.8%60.8\% of primary cells require step clipping; its eligible subset is strongly negative. The matched result is an implementation/Taylor check; the cross-fit supplies no positive claim about transfer. 4. Comparator scope. P7’s original battery used one random Rayleigh quotient per seed; the 6464-direction follow-up confirms the magnitude ratios and the H2H^2 comparator, so what remains open is the probe itself: the ratio is a property of the mixture probe, and the per-task control shows it is not task-invariant. 5. Scale of the commutator extensions. The full-fine-tuning control is a ≤1.1≤ 1.1B result whose median threshold products carry over while its pointwise check fails (plausibly for float32 reasons), and the 1313–1414B cells are six finite crossings plus six right-censored pairs, with the registered Tulu-8B cell dropped for an environment failure; there is no full atlas beyond 77B. 6. Registration is partial. The property list, split, keep rule, and thresholds were registered, but the operating point, estimator details, and P7’s final phrasing were frozen in code rather than in the dated document, and several passing statistics fail the registered CV clause (Appendix A). 7. Forecast negatives. The out-of-sample λmax _ radius forecast was undecidable (two of three models left-censored), the inside/outside search comparison failed, and the cross-fitted confirmation was invalid under its step-size rule. The commutator analysis is neither a radius calculator nor a held-out task-effect predictor. 8. Headroom measurement gaps. The per-task headroom table is base-point data, accuracy was recorded only in the three-model extended sweeps and the best-of-256256 rerun, and the P4 boundary cannot be placed on the common displacement ruler because task-vector norms were not archived. 9. The functional probes are narrow. P4 uses one layer, one mean last-token activation summary, 1616 fixed SST-2 prompts, one task-vector recipe, and three pairs; P5 one SST-2 global mean-vector construction at one step size over three candidate layers; the showcase accuracy uses 5555–6464 examples per task (full per-cell table in Appendix C). Headline words such as “additivity” and “correspondence” should be read at this width, and P6 remains in-sample even after the scale sweep: the same probe defines the scores and the gains. 10 Conclusion In this paper, we set out to turn a familiar intuition, that fine-tuning operates in a locally linear neighbourhood, into measured falsifiable statements. As a result, we have found interesting structure but no single radius. On the positive side, we have found a searchable regime: around one multitask LoRA operating point, measured identically on nine models, probe loss changes stay first-order dominated through the atlas grid on every model median, and local perturbations keep the ranks of their gradient projection. In our extended sweeps (three models, one seed), we have found both the interior and the end of the window, and the response along two fixed random directions is 480×480×–4300×4300× weaker at the edge. On the other hand, the same neighbourhood is narrow and pair-specific for interactions: task-gradient subspaces rotate within tens of steps, additivity under the selected activation probe fails at full task-vector scale on several models including both held-out 77B models, and reordering two task updates produces defects whose onset varies by three orders of magnitude across (model, pair) cells. Several registered claims did not survive their own controls: the rank-≤32≤ 32 mixture-covariance bar was algebraically non-falsifiable and its follow-up showed no plateau, and the global mean-vector weight-to-CAA correspondence has no passing model median. The two-direction boundary has a classical mechanism: the Lie bracket of the two tasks’ gradient vector fields. In fixed LoRA coordinates, the normalized defect follows c(η)=ηκ+O(η2)c(η)=ηκ+O(η^2) so precisely when both sides share minibatches (median ratio 1.0021.002) that the threshold law η†≈0.10/κη ≈ 0.10/κ turns into an actual arithmetic dependency. The two Hessian-vector products measure, pair by pair, a quantity that finite-step experiments then confirm across LoRA, full fine-tuning, and 1313–1414B scale at the median-product level. The bracket, however, does not yet transfer: estimated on one sample, it fails to predict either the endpoint direction or the task effects of updates built and evaluated on other samples, partly because the registered step scale is infeasible for most cells and partly because the eligible cells show no predictive power. We consider finding out why (step policies, gauge and function-space metrics, optimizer state, or something more fundamental about minibatch geometry) the most important open question that this work leaves. The broader picture looks the same way as in the companion paper [31]. There, we found that linear structure in trained networks is real but local, and now we can add: its validity is operation-specific. Self-curvature, symmetric composition, and antisymmetric order effects are different tensors with different scales, and treating “the linear regime” as one number conflates them. The practical corollary is modest but usable: check the diagnostic that matches your operation, at your scale, on your pair. The natural theoretical next step is to make the local calculus of adaptation coordinate-free: in further work we hope to study gauge-invariant adapter geometry, optimizer-aware brackets in augmented state space, and function space metrics. To sum up, a shared first-order window makes local random search predictable, while pairwise task interactions often become fragile even at much smaller, pair-specific scales. References Aghajanyan et al. [2021] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021. Baik et al. [2005] Jinho Baik, Gerard Ben Arous, and Sandrine Peche. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Annals of Probability, 33:1643–1697, 2005. Benaych-Georges and Nadakuditi [2011] Florent Benaych-Georges and Raj Rao Nadakuditi. The singular values and vectors of low rank perturbations of large rectangular random matrices. Advances in Mathematics (arXiv:0910.2120), 2011. Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), 2023. URL https://arxiv.org/abs/2304.01373. Black et al. [2021] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. Zenodo, 2021. URL https://doi.org/10.5281/zenodo.5297715. Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL-HLT, 2019. Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. da Silva et al. [2026] Marvin F. da Silva, Mohammed Adnan, Felix Dangel, and Sageev Oore. Generalizing the geometry of model merging through Fréchet averages. arXiv preprint arXiv:2604.27155, 2026. URL https://arxiv.org/abs/2604.27155. Dinh et al. [2017] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. arXiv:1703.04933 (ICML), 2017. Draxler et al. [2018] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning (ICML), 2018. URL https://arxiv.org/abs/1803.00885. Foret et al. [2021] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv:2010.01412 (ICLR), 2021. Fort et al. [2019] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019. Gan and Isola [2026] Yulu Gan and Phillip Isola. Neural thickets: Diverse task experts are dense around pretrained weights. arXiv preprint arXiv:2603.12228, 2026. URL https://arxiv.org/abs/2603.12228. Garipov et al. [2018] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL https://arxiv.org/abs/1802.10026. Ghorbani et al. [2019] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In ICML, 2019. Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Accelerating the science of language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2024. URL https://arxiv.org/abs/2402.00838. Gur-Ari et al. [2018] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv:1812.04754, 2018. Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2106.09685. Hugging Face [2019] Hugging Face. DistilGPT2 model card. Hugging Face model repository, 2019. URL https://huggingface.co/distilbert/distilgpt2. Accessed 2026-07-10. Ilharco et al. [2023] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2212.04089. Keskar et al. [2017] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2017. URL https://arxiv.org/abs/1609.04836. Li et al. [2018a] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations (ICLR), 2018a. URL https://arxiv.org/abs/1804.08838. Li et al. [2018b] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018b. Malladi et al. [2023] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Meng et al. [2022] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NeurIPS (arXiv:2202.05262), 2022. Montanari and Wang [2026] Andrea Montanari and Zihao Wang. Phase transitions for feature learning in neural networks. arXiv:2602.01434, 2026. URL https://arxiv.org/abs/2602.01434. Nicolas et al. [2025] Julien Nicolas, Mohamed Maouche, Sonia Ben Mokhtar, and Mark Coates. Dome: Improving signal-to-noise in stochastic gradient descent via sharp-direction subspace filtering. arXiv:2507.03545, 2025. URL https://arxiv.org/abs/2507.03545. Papyan [2019a] Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. arXiv:1901.08244 (ICML), 2019a. Papyan [2019b] Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size. arXiv:1811.07062, 2019b. Paul [2007] Debashis Paul. Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica, 17:1617–1642, 2007. Piontkovskaia and Nikolenko [2026] Irina Piontkovskaia and Sergey Nikolenko. Recoverable but not stationary: Local linear structures in weights and activations. arXiv preprint arXiv:2606.10929, 2026. URL https://arxiv.org/abs/2606.10929. Rimsky et al. [2024] Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, 2024. URL https://aclanthology.org/2024.acl-long.828/. Rosca et al. [2021] Mihaela Rosca, Yan Wu, Benoit Dherin, and David G T Barrett. Discretization drift in two-player games. In ICML, 2021. arXiv:2105.13922. Rukhovich et al. [2025] Alexey Rukhovich, Alexander Podolskiy, and Irina Piontkovskaya. Commute your domains: Trajectory optimality criterion for multi-domain learning. arXiv preprint arXiv:2501.15556, 2025. URL https://arxiv.org/abs/2501.15556. Sagun et al. [2017] Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv:1706.04454, 2017. Salimans et al. [2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. Smith et al. [2021] Samuel L Smith, Benoit Dherin, David G T Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In ICLR, 2021. arXiv:2101.12176. Song et al. [2025] Minhak Song, Kwangjun Ahn, and Chulhee Yun. Does sgd really happen in tiny subspaces? In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=v6iLQBoIJw. Sweeney [2026a] John Sweeney. The geometry of sequential learning: Lie-bracket prediction of transfer order. Proceedings of the 43rd International Conference on Machine Learning, 2026a. URL https://arxiv.org/abs/2606.24993. arXiv:2606.24993. Sweeney [2026b] John Sweeney. Optimizer memory makes shuffle order a first-order source of fine-tuning noise. arXiv preprint arXiv:2606.29554, 2026b. URL https://arxiv.org/abs/2606.29554. Tang et al. [2023] Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. arXiv preprint arXiv:2310.04742, 2023. URL https://arxiv.org/abs/2310.04742. Tang et al. [2026] Lin Tang, Wei Zhang, Jing Li, Hongyu Chen, Ming Zhao, and Yuxuan Wang. Predicting mergeability of parameter-efficient fine-tuning updates. arXiv preprint arXiv:2606.19549, 2026. URL https://arxiv.org/abs/2606.19549. Team OLMo et al. [2024] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, 2024. URL https://arxiv.org/abs/2501.00656. Todd et al. [2024] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.15213. Turner et al. [2023] Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023. URL https://arxiv.org/abs/2308.10248. Vershynin [2010] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv:1003.2990 (in: Compressed Sensing, CUP 2012), 2010. Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022. URL https://arxiv.org/abs/2203.05482. Wu et al. [2024] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. ReFT: Representation finetuning for language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://arxiv.org/abs/2404.03592. Yadav et al. [2023] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2306.01708. Yang et al. [2024] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. URL https://arxiv.org/abs/2412.15115. Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. Zhang et al. [2024] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. URL https://arxiv.org/abs/2401.02385. Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. URL https://arxiv.org/abs/2205.01068. Zou et al. [2023] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023. URL https://arxiv.org/abs/2310.01405. Appendix A Registration and protocol provenance The property list, development/held-out split, keep bar (≥4/6≥ 4/6 development models with across-seed coefficient of variation below 0.250.25, then ≥2/3≥ 2/3 held-out), and the property thresholds were registered before the sweep, with the following qualifications. Table 7: Analysis provenance; “registered” applies to the listed object, not to every implementation detail or later interpretation. Analysis object Status Property list, numeric bars, model split, keep/drop rule Prospectively registered Operating point and estimator details Frozen in executable code; not fully specified in the dated registration Fine-grid commutator and estimator-decomposition studies Separately registered before their data were read Base/per-task, full-FT, scale, m-sweep, and forecast controls Separately registered before their data were read Cross-fitted functional commutator confirmation Separately preregistered; complete raw audit, then a single scoring pass Four follow-up controls (P7 distribution, P6 scale sweep, P4 effective-weight addition, best-of-N accuracy) Separately preregistered with numeric outcome bars before launch Headroom and common-displacement conversion Descriptive reanalysis of archived measurement records λmax _ relation Exploratory within-atlas association followed by one frozen out-of-sample test Apparatus. The original preregistration document retained two stale apparatus phrases from an earlier harness revision (a “60-step snapshotted recovery run” and P2 as a “snapshotted update covariance”). The executable harness used for every sweep fixes the actual recipe of Section 3 (a 300300-step multitask LoRA operating point and per-minibatch gradient covariance), and every run’s output records it. Because there is no dated pre-data amendment for the apparatus wording, we do not claim a fully preregistered apparatus: the binding registered objects are the property list, split, keep/drop rule, and thresholds; the operating point and estimator details are frozen-code protocol choices. Gating. After the development sweep we froze the keep/drop decisions in a dated file that applied the threshold bars and recorded the CV qualification separately; several statistics that passed their bars do not satisfy the original CV clause, and we report them as qualified. The frozen decisions were: keep P1, P2, P3, P4, P6, P7, P8; drop P5. Two registered decisions are worth restating: P3 was expected to fail its naive stable form and was kept as the rotation finding; and the prediction that the three radii coincide was registered and is reported as falsified. For P7, the original document specified directional curvature with a reconstruction gate but not the final magnitude-ratio phrasing; the frozen held-out decision file fixed |ratio|>3|ratio|>3 as the operative statistic, with cells failing the reconstruction check marked but not counted. Later additions. The fine-grid commutator re-measurement, its matched-estimator metrics, the estimator-decomposition run, the full-fine-tuning control, the m-sweep control, the base-point control, the per-task-probe control, the λmax _ predictor analysis, the out-of-sample forecast/search test of Section 6.5, the cross-fitted functional confirmation of Section 6.4, and the showcase run of Section 4 were each pre-registered before their data was read, in dated files accompanying the respective runs. The headroom analysis (Section 7), the common-ruler conversion (Section 5.4), and the per-cell showcase accuracy table (Appendix C) are descriptive reanalyses of already-archived outputs by pinned scripts, with no new thresholds and no selection. The four follow-up controls (the P7 direction distribution, the P6 scale sweep, the P4 effective-weight addition, and the best-of-N accuracy rerun) were preregistered with numeric outcome bars before launch; one of their ten bars missed (the signed-extremeness bar for P7), and it is reported as such in Section 5.2. A later preregistered task-conditioned covariance follow-up stopped at a failed whole-matrix numerical audit before scientific scoring; no partial metrics were used. Seed-level facts the medians hide. One Pythia-160M seed has σ1⋆=10−4σ _1=10^-4; eight of 2727 produced P1 development seed curves are nonmonotone at the tiniest scales, so “censored on every model” is a largest-passing-grid median statement; and the P7 ratio varies by over two orders of magnitude across seeds within some models (the absolute ratios remain well over the magnitude threshold, but clean evidence still requires the instrument gate). Appendix E lists every per-seed value. Appendix B Experimental details All details below are read from the frozen measurement harness; the archived per-run output records the same configuration. Models. Development: distilgpt2, EleutherAI/pythia-160m, EleutherAI/pythia-410m, EleutherAI/gpt-neo-1.3B, facebook/opt-1.3b, TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T. Held-out: EleutherAI/pythia-1.4b, Qwen/Qwen2.5-7B, allenai/OLMo-7B-Instruct-hf. Commutator scale extension: Qwen/Qwen2.5-14B, allenai/OLMo-2-1124-13B (the registered allenai/Llama-3.1-Tulu-3-8B cell was dropped before scoring for a pinned-environment tokenizer failure). Full-fine-tuning control: Pythia-160M/410M and TinyLlama-1.1B. Out-of-sample forecast/search models (Sections 6.5 and 7): gpt2-medium, facebook/opt-350m, Qwen/Qwen2.5-1.5B. Extended sweeps (Section 4): EleutherAI/pythia-410m, EleutherAI/gpt-neo-1.3B, Qwen/Qwen2.5-7B, one seed. Cross-fitted confirmation (Section 6.4): gpt2-large, allenai/OLMo-2-1124-7B, mistralai/Mistral-7B-v0.3, Qwen/Qwen2.5-14B. Operating point and LoRA configuration. LoRA rank 1616, α=32α=32, dropout 0, no bias terms, applied to the attention Q/K/V input projections in each family’s naming: c_attn (fused QKV) for the GPT-2 family, query_key_value (fused) for Pythia/NeoX, and q_proj/k_proj/v_proj for GPT-Neo, OPT, Qwen, TinyLlama, and OLMo; output projections are not adapted. Trainable parameter counts P range from ∼1.6 1.6M (GPT-2-medium) through 4,718,5924,718,592 (GPT-Neo-1.3B) to ∼107 10^7 at 77B. ⋆ θ is reached by 300300 AdamW steps (lr 2⋅10−42· 10^-4, PyTorch defaults otherwise), batch size 88, with each batch item’s task drawn uniformly from the five tasks. Everything runs in float32 with eager attention, deterministic algorithms enabled, and seeds 0,1,2\0,1,2\ controlling both PyTorch and NumPy RNG streams. In the full-fine-tuning control all parameters are trainable (no adapter) at lr 2⋅10−52· 10^-5. Tasks and prompts. Training pools are the first 2,0002,000 train examples per task; evaluation pools the first 128128 validation examples. Sequences are truncated to 160160 tokens; the loss is the HuggingFace token-mean cross-entropy over continuation tokens only (prompt tokens masked). Templates ( = newline; label continuations shown after the arrow, each with a leading space): • SST-2: Review: sentence : → negative / positive; • BoolQ: passage[:600] : question? : → no / yes; • RTE: premise : Does this entail: "hypothesis"? : → yes / no; • ARC-Easy: Question: question : → each answer choice as a continuation; • HellaSwag: the context as prompt → the four endings as continuations. The probe set. L, g, and H are all computed on the same four frozen multitask batches of eight examples (3232 examples total), sampled task-uniformly from the training pools by a dedicated RNG stream fixed per (model, seed). L is the mean of the four batch token-mean losses; gradients and Hessian-vector products average the per-batch quantities with equal batch weights (token-weighted within batch, which reproduces each batch’s token-mean exactly). An earlier draft described this probe as “128128 examples,” conflating it with the per-task evaluation-pool size; the probe is 3232 examples. Per-property settings. P1: g is the unit probe gradient; L(⋆±σ^)L( θ ±σ g) is evaluated on the probe set at each grid σ. P2: m=32m=32 per-minibatch gradients (batch size 88, fresh task-uniform draws), centered; r90r_90 from the singular values, so the rank ceiling is m−1m-1; the m-sweep control repeats this at m∈32,64,128,256m∈\32,64,128,256\ (seed 0) with an isotropic null estimated from Gaussian matrices of matched m (three repetitions). P3: BoolQ trajectory from ⋆ θ under AdamW at lr 2⋅10−42· 10^-4; at each checkpoint the top-1010 subspace of the single-task gradient covariance is estimated twice from 3232 gradients each; the resampling floor is the overlap of the two same-checkpoint estimates, and the onset is the first checkpoint where the overlap with the step-8 subspace falls below the floor minus 0.020.02. P4: task vectors are 4040-step AdamW single-task fine-tunes from ⋆ θ ; ()h( δ) is the mean over 1616 fixed SST-2 training prompts of the last-token hidden state at the output of block ⌊0.55nL⌋ 0.55\,n_L (e.g. block 1313 of 2424 for GPT-Neo-1.3B); the null applies random weight deltas matched in norm to each task vector. P5: the task is SST-2; the candidate layers are the outputs of blocks ⌊0.4nL⌋ 0.4\,n_L , ⌊0.55nL⌋ 0.55\,n_L , and ⌊0.7nL⌋ 0.7\,n_L (blocks 9/13/169/13/16 for a 2424-block model), the “upper-medium” set; the CAA vector is built from 2424 validation prompts as the mean prompt-only last-token hidden state of positive-label minus negative-label examples, per example without padding; the weight move is one gradient step −ηsst2- _ sst2 at η=10−4η=10^-4 from a single-minibatch SST-2 gradient; the activation shift is the mean per-prompt last-token change; the reported cosine is the best over the three layers, and the null repeats the measurement with a norm-matched random weight perturbation. P6: 256256 draws ∼(0,σ2I) δ (0,σ^2I) at per-coordinate σ=10−3σ=10^-3; gains are probe-set loss differences; the prediction is −⊤-g δ with g the probe gradient. P7: v is the unit probe gradient and the comparator one random unit vector; curvatures are ⊤Hv Hv via double-backward HVPs on the probe set; the reconstruction check compares ⊤Hv Hv to the gradient finite difference v⊤((⋆+ϵ)−(⋆−ϵ))/2ϵv (g( θ + )-g( θ - ))/2ε at ϵ∈10−5,10−6,10−4ε∈\10^-5,10^-6,10^-4\, accepting the first ϵε with ratio in [0.5,2][0.5,2]. P8: each one-step update uses a fresh single-minibatch task gradient from a fixed per-task RNG stream, the same stream for both orderings, so the two paths differ only in order; the coarse onset is the smallest grid η with c(η)≥0.10c(η)≥ 0.10. Commutator measurements. Task gradients and HVPs in Section 6 use three frozen single-task batches per task (a fixed per-task RNG stream), shared between the two-step defect and the κ commutator: the “matched estimator.” The top-1010 Hessian subspace and λmax _ come from Lanczos with 2424 iterations and full reorthogonalization on the probe-set HVP operator. The fine grid is 2525 logarithmically spaced points on [10−5,10−1][10^-5,10^-1] (3333 points from 10−710^-7 for the full-fine-tuning control, whose κ is larger), with η†η read by log-log interpolation of the c=0.10c=0.10 crossing and left/right censoring recorded per cell. Parameters are restored to ⋆ θ between pairs; an early run that failed to do so was caught by the pre-registered checks and fully re-run. The estimator-decomposition run replaces only the defect-side gradients with fresh single-minibatch estimators, leaving the κ side at three batches. Cross-fitted functional confirmation. The confirmation models are GPT-2-large, OLMo-2-7B, Mistral-7B-v0.3, and Qwen2.5-14B, with seeds 0,1,2\0,1,2\. The first 2,0002,000 valid training examples per task define the common operating point; valid training ranks 2,1292,129–2,1882,188 define geometry and ranks 2,1892,189–2,2482,248 define the independent update paths. Evaluation uses valid validation ranks 257257–384384, except RTE uses disjoint training ranks 2,2492,249–2,3762,376 because its validation split is shorter. Every task therefore has 6060 geometry examples, 6060 update examples, and 128128 evaluation examples. The ten task pairs and five evaluation tasks are scored at target effect 0.050.05 with η=0.05/κGη=0.05/ _G and admissible interval [10−5,3⋅10−2][10^-5,3· 10^-2]; cells outside it are emitted at the endpoint but excluded from primary aggregates. The primary rules require overall Rzero2>0R^2_ zero>0 with at least three of four models positive, overall endpoint cosine >0.50>0.50 with at least three models positive, superiority to a fixed cyclic pair shuffle on both quantities, and at most 25%25\% step-clipped pair cells. All choices, floating-point tolerances, and the rule that the raw-output check completes before any scoring were frozen before confirmation contact. Extended-sweep run. Per task at ⋆ θ : a fixed per-task probe set of four eight-example training batches; the per-task gradient is the four-batch average; the loss sweep covers 1313 logarithmically spaced signed scales s∈±[10−4,1]s∈±[10^-4,1] along the unit task gradient and along two fixed random unit directions shared across tasks; multiple-choice accuracy (argmin of per-choice continuation token-mean loss, first 5555–6464 usable validation examples) is probed at ⋆ θ and at ⋆±s^t θ ± s g_t for s∈10−2,10−1,0.3,1s∈\10^-2,10^-1,0.3,1\; parameters are restored to ⋆ θ after every sweep point. Follow-up controls. The P7 distribution uses 6464 random unit directions per cell (a fresh generator per cell) with one probe HVP each, recording signed ⊤Hr Hr and ‖H‖ Hr . The P6 sweep repeats the 256256-draw protocol at each of the seven grid scales, with a fresh generator per scale. The effective-weight control materializes each task’s ΔW W from the LoRA factors per adapted module, applies ΔWA+ΔWB W_A+ W_B to the base weights with the adapters held at ⋆ θ , and requires the two routes to agree on single tasks to relative error 10−210^-2 (all 8181 cells pass). The accuracy rerun repeats the two archived best-of-256256 search cells with frozen radii and search seeds, and records multiple-choice accuracy (the extended-sweep rule) on the first 128128 loaded validation examples per task, at ⋆ θ and at each selected perturbation. Compute. All sweeps ran on A100 80GB GPUs, one (model, seed) cell per GPU, in float32 with eager attention and offline HuggingFace caches. At 1313–1414B the gradient/HVP closures are row-chunked (microbatch 22) with shifted-active-token weighting, which reproduces the batch token-mean exactly, so chunking changes peak memory only. The best-of-N search demo draws 256256 perturbations per radius multiplier (0.25,0.5,1,2,4×rforecast\0.25,0.5,1,2,4\× r_ forecast, where rforecast=ηforecast‖r_ forecast= _ forecast ) and evaluates target and side-task losses on fixed probe batches. Appendix C Showcase accuracy per cell Table 8 tabulates multiple-choice accuracy for all 1515 (model, task) cells of the extended sweeps of Section 4: the base value at ⋆ θ and the signed change at every probed scale, in both directions along the task gradient. It is a descriptive tabulation of the archived extended-sweep outputs. With 5555–6464 usable examples per task, one example is worth 1.61.6–1.81.8p; entries of that size are counting noise. Responses above 1010p appear only at |s|≥0.1|s|≥ 0.1 and only on five of the 1515 cells (both BoolQ cells, SST-2 on Pythia-410M and on Qwen2.5-7B, and ARC-Easy on Qwen2.5-7B at s=1s=1); the other ten cells stay within a few examples of their base value even at s=1s=1. Table 8: Multiple-choice accuracy across the extended-sweep grid: base accuracy at ⋆ θ (percent) and change (p) at each probed scale s along ∓^t∓ g_t; n is the number of usable evaluation examples. downhill −s-s uphill +s+s model task n base % 1.01.0 0.30.3 0.10.1 0.010.01 0.010.01 0.10.1 0.30.3 1.01.0 neo-1.3B sst2 6464 90.690.6 −1.6-1.6 −1.6-1.6 −3.1-3.1 +0.0+0.0 +1.6+1.6 +1.6+1.6 +0.0+0.0 −4.7-4.7 neo-1.3B boolq 5959 72.972.9 +1.7+1.7 +1.7+1.7 +1.7+1.7 +0.0+0.0 +1.7+1.7 −5.1-5.1 −35.6-35.6 −47.5-47.5 neo-1.3B rte 6161 59.059.0 −9.8-9.8 −9.8-9.8 −1.6-1.6 +3.3+3.3 −3.3-3.3 −8.2-8.2 −8.2-8.2 −8.2-8.2 neo-1.3B arc-e 6464 64.164.1 +3.1+3.1 −1.6-1.6 +0.0+0.0 +0.0+0.0 +0.0+0.0 −1.6-1.6 −3.1-3.1 −3.1-3.1 neo-1.3B hella 6464 43.843.8 −4.7-4.7 −4.7-4.7 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 −3.1-3.1 −3.1-3.1 pythia-410m sst2 6464 87.587.5 −39.1-39.1 −21.9-21.9 −3.1-3.1 +0.0+0.0 +0.0+0.0 −14.1-14.1 −35.9-35.9 −35.9-35.9 pythia-410m boolq 5959 47.547.5 +25.4+25.4 +25.4+25.4 +25.4+25.4 +10.2+10.2 −6.8-6.8 −20.3-20.3 −20.3-20.3 −20.3-20.3 pythia-410m rte 6161 59.059.0 −8.2-8.2 −6.6-6.6 +0.0+0.0 +1.6+1.6 −1.6-1.6 −4.9-4.9 −9.8-9.8 −9.8-9.8 pythia-410m arc-e 6464 50.050.0 +0.0+0.0 +4.7+4.7 +0.0+0.0 +0.0+0.0 +0.0+0.0 −1.6-1.6 −1.6-1.6 +3.1+3.1 pythia-410m hella 6464 37.537.5 −6.2-6.2 +0.0+0.0 +0.0+0.0 −1.6-1.6 +0.0+0.0 +1.6+1.6 +0.0+0.0 +1.6+1.6 Qwen2.5-7B sst2 6464 96.996.9 −18.8-18.8 +1.6+1.6 +3.1+3.1 +1.6+1.6 +0.0+0.0 +0.0+0.0 −1.6-1.6 −15.6-15.6 Qwen2.5-7B boolq 5555 89.189.1 +1.8+1.8 −1.8-1.8 +1.8+1.8 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 −3.6-3.6 Qwen2.5-7B rte 6060 90.090.0 −1.7-1.7 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 +1.7+1.7 −5.0-5.0 Qwen2.5-7B arc-e 6464 87.587.5 +0.0+0.0 +1.6+1.6 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 −1.6-1.6 −15.6-15.6 Qwen2.5-7B hella 6464 67.267.2 −6.2-6.2 −3.1-3.1 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 +0.0+0.0 −3.1-3.1 Appendix D Threshold sensitivity The thresholds are selection bars; changing any bar after seeing the data would void the selection protocol. As a sensitivity check, Table 9 reruns the stored development and held-out summaries under nearby bars without reselecting any property. The headline pattern is stable: P1 and P6 keep under all tested bars; P5 drops under all tested bars; P4 remains scale-limited because relaxing the additivity bar can improve the development count but still leaves only 1/31/3 held-out models passing at full task-vector scale. The P7 rows are magnitude-only sensitivity checks and do not apply the instrument gate, so they do not change the clean-evidence count. Table 9: Threshold sensitivity on the stored atlas measurements: the original development-plus-held-out rule applied to nearby bars, one diagnostic at a time. property threshold dev pass holdout pass rule verdict P1 0.05 5/6 3/3 keep P1 0.1 6/6 3/3 keep P1 0.2 6/6 3/3 keep P4 0.1 2/6 0/3 drop P4 0.15 4/6 1/3 drop P4 0.2 5/6 1/3 drop P5 0.2 1/6 0/3 drop P5 0.3 0/6 0/3 drop P5 0.4 0/6 0/3 drop P6 0.3 6/6 3/3 keep P6 0.5 6/6 3/3 keep P6 0.7 5/6 3/3 keep P7 magnitude 2 6/6 3/3 keep P7 magnitude 3 6/6 3/3 keep P7 magnitude 10 6/6 3/3 keep Appendix E Per-seed values of the headline statistics Per-seed values (seeds 0/1/20/1/2) behind every per-model median in Table 2; ‘—’ marks a value the harness did not produce for that seed. Radii are grid-quantized, so identical seed values are expected where the onset falls in the same grid cell. Table 10: Per-seed values, all nine models (development then held-out). model P1 σ1⋆σ _1 P2 r90r_90 P3 onset P4 ε(1) (1) P5 cos P6 corr P7 ||ratio|| P8 η†η distilgpt2 1e-2/1e-2/1e-2 23/22/23 128/—/32 0.10/0.06/0.07 +0.13/+0.01/-0.09 0.77/0.75/0.80 1.5e+03/3.5e+03/1.7e+03 1e-5/1e-5/1e-5 pythia-160m 1e-4/1e-2/1e-2 23/24/24 16/16/32 0.10/0.14/0.18 +0.01/+0.13/+0.01 0.81/0.74/0.70 1.7e+05/3.4e+05/6.8e+02 3e-4/3e-5/3e-5 pythia-410m 1e-2/1e-2/1e-2 24/24/25 16/16/16 0.10/0.08/0.06 -0.01/-0.12/-0.16 0.97/0.97/0.97 3.1e+04/8.2e+04/2.9e+03 3e-3/3e-3/3e-3 neo-1.3B 1e-2/1e-2/1e-2 20/21/23 16/32/32 0.19/0.21/0.19 +0.05/+0.34/-0.04 1.00/1.00/1.00 1.0e+04/1.4e+05/1.9e+04 —/1e-2/— opt-1.3b 1e-2/1e-2/1e-2 23/21/23 16/16/32 0.35/0.40/0.25 +0.34/-0.15/+0.22 0.84/0.70/0.67 4.6e+04/9.6e+04/7.0e+04 1e-5/1e-5/1e-5 TinyLlama-1.1B 1e-2/1e-2/1e-2 23/21/24 16/16/32 0.14/0.14/0.15 +0.04/+0.04/+0.07 1.00/1.00/1.00 4.2e+05/2.1e+05/1.5e+04 3e-3/1e-2/1e-2 pythia-1.4B 1e-2/1e-2/1e-2 23/23/24 16/16/16 0.11/0.08/0.22 +0.01/-0.23/-0.22 1.00/1.00/1.00 6.2e+03/4.9e+04/1.7e+04 1e-2/3e-3/1e-2 Qwen2.5-7B 1e-2/1e-2/1e-2 23/23/25 16/32/16 0.36/0.26/0.32 +0.18/-0.05/+0.01 1.00/1.00/1.00 6.4e+04/4.0e+04/2.1e+02 1e-2/1e-2/— OLMo-7B 1e-2/1e-2/1e-2 25/11/21 16/16/16 0.31/0.38/0.32 +0.01/-0.09/-0.06 0.99/0.97/0.97 1.8e+05/1.2e+07/2.4e+05 1e-2/3e-3/3e-3