Paper deep dive
Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlases
Jiangtao Lin, Bangyang Wei, Yihang Ding, Siyi Liu, Yuhan Dong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Task-specific fine-tuning can rewrite a language model's answers beyond the training task, complicating updates that must preserve existing behavior. We introduce ATLAS, which turns retained-domain representations into an input-dependent rule for task adaptation. An activation atlas supplies local reference centers and directional filters to a shared low-rank residual. Target supervision learns the residual, while retained geometry shapes its action throughout training and inference. On Qwen3-8B, ATLAS achieves lower mean retained-output Kullback-Leibler (KL) divergence than all seven published baselines at shared coding-performance requirements, with consistent advantages across multiple training seeds. Structural comparisons identify the contributions of retained reference states and directional conditioning, and answer-level analyses show fewer rewritten mathematical answers and more stable commonsense choices. Experiments spanning five backbones and two retained domains further demonstrate coding gains with reduced retained-output movement. With compact storage and modest decoding overhead, ATLAS provides a practical mechanism for acquiring specialized skills while maintaining continuity in existing responses.
Tags
Links
- Source: https://arxiv.org/abs/2609.34478v1
- Canonical: https://arxiv.org/abs/2609.34478v1
Trouble viewing inline? Open PDF directly →
Full Text
90,522 characters extracted from source content.
Expand or collapse full text
LEARN HERE, MOVE LESS ELSEWHERE: INPUT-CONDITIONED PLASTICITY FROM RETAINED- DOMAIN ACTIVATION ATLASES Jiangtao Lin 1 Bangyang Wei 2 Yihang Ding 1,3 Siyi Liu 1,4 Yuhan Dong 1,∗ 1 Tsinghua Shenzhen International Graduate School, Tsinghua University 2 School of Vehicle and Mobility, Tsinghua University 3 SZ DJI Technology Co., Ltd. 4 Tencent Holdings Limited ∗ Corresponding author: dongyuhan@sz.tsinghua.edu.cn ABSTRACT Task-specific fine-tuning can rewrite a language model’s answers beyond the train- ing task, complicating updates that must preserve existing behavior. We introduce ATLAS, which turns retained-domain representations into an input-dependent rule for task adaptation. An activation atlas supplies local reference centers and direc- tional filters to a shared low-rank residual. Target supervision learns the residual, while retained geometry shapes its action throughout training and inference. On Qwen3-8B, ATLAS achieves lower mean retained-output Kullback–Leibler (KL) divergence than all seven published baselines at shared coding-performance re- quirements, with consistent advantages across multiple training seeds. Structural comparisons identify the contributions of retained reference states and directional conditioning, and answer-level analyses show fewer rewritten mathematical an- swers and more stable commonsense choices. Experiments spanning five back- bones and two retained domains further demonstrate coding gains with reduced retained-output movement. With compact storage and modest decoding over- head, ATLAS provides a practical mechanism for acquiring specialized skills while maintaining continuity in existing responses. 1INTRODUCTION A language model is trained to write code, but the resulting update also changes its answers to mathematics and commonsense questions. Shared parameters carry specialized learning into inputs that supplied no target supervision. Some original answers become incorrect, some errors are corrected, and other answers change without changing correctness. These transitions matter when applications rely on continuity across model versions. Successful adaptation therefore requires both acquiring the requested skill and controlling the reach of the update into existing behavior. CorDA and LoRA-Null show how representative activations guide adaptation subspaces (Yang et al., 2024; Tang et al., 2026). TopLoRA and SLIM make adaptation input-dependent (Li et al., 2025; Han et al., 2025), while STM and TALR adjust target-token contributions (Wu et al., 2025; Lin et al., 2026). These methods select update subspaces or modulate adaptation without explicitly using the current token’s local retained neighborhood as a geometric reference. Retained activations occupy different neighborhoods, each offering a reference state and a pattern of local variation. To address this gap, we route each token to nearby neighborhoods and use their geometry to condition one shared task residual. We realize this idea with ATLAS, an activation atlas whose charts store centers and leading local variation (Figure 1). Target supervision trains a shared low-rank branch; the atlas shapes what it reads and emits. Because all charts share the trainable factors, richer geometric conditioning requires 1 arXiv:2609.34478v1 [cs.LG] 28 Sep 2026 Learn to code. Learn code. Rewrite less. def f(x): s = 0 for i in x: s += i return s Starting answers Math Commonsense Ordinary answers Math Commonsense ATLAS answers Math Commonsense Ordinary adaptation Starting model ATLAS-adapted model ChangedUnchanged Math 3 + 2 = ? Commonsense A B C or c₁, U₁c₂, U₂c₃, U₃ h α(h) c(h), N(h) h c(h) Center N(h)N(h) Input filter Output filter AB Δ(h) Retained inputs Starting model Retained activations Local charts Token routingShared low-rank residual Figure 1: Learning a task with less answer rewriting. Both adaptation paths start from the same model. Gray marks and orange waves schematically denote unchanged and changed answers. Retained inputs illustrate domains evaluated separately. Atlas charts summarize activation neighbor- hoods with reference centers and local directions. The same token statehdrives routing and enters a shared residual conditioned on the selected geometry. no additional task experts. The same rule operates during inference: supervision sets the learning destination; retained geometry gives updates a state-dependent address. We first evaluate ATLAS on Qwen3-8B against seven published adaptation methods, using complete learning trajectories to compare output preservation at shared coding requirements. Experiments spanning five backbones and two retained domains extend this assessment across model scales and families. Structural comparisons then examine how centers, distances, and directions shape the residual, and answer-level analyses follow the resulting changes in mathematical answers and commonsense choices. We summarize our main contributions as follows: •We propose ATLAS, an activation-atlas method that turns local retained reference states and directions into token-dependent filters for one shared task residual. This makes retained geometry a reusable part of adaptation throughout training and inference. •Our method achieves lower mean retained-output KL than all seven published baselines at shared coding requirements on Qwen3-8B, with consistent advantages across three training seeds. Cross-family results extend the learning–retention benefit to GLM. • We connect distributional preservation to fewer rewritten mathematical answers and more stable commonsense choices. Decomposing harmful flips, incidental corrections, and within-error changes captures both correctness transitions and changes in answer identity. 2RELATED WORK Related methods control the effects of adaptation through learning signals, geometric constraints, or input-dependent computation. Preservation through learning signals. Elastic Weight Consolidation penalizes changes to pre- viously important parameters; Learning without Forgetting preserves old-network predictions on new-task inputs (Kirkpatrick et al., 2017; Li & Hoiem, 2016). GEM constrains gradients using task-wise memories, and A-GEM uses an average memory constraint (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019). STM excludes high-perplexity response tokens, while TALR dynamically reweights token losses (Wu et al., 2025; Lin et al., 2026). These approaches shape the optimization 2 signal that learns an update. Our focus is how a shared task residual acts on different retained states, using their representations to condition its computation. Geometry of preserved computation.Direction selection offers a geometric approach to control- ling interference. Orthogonal Gradient Descent protects previous output-gradient directions; Gradient Projection Memory extracts principal activation directions for subsequent gradient projection (Fara- jtabar et al., 2020; Saha et al., 2021). Adapters and LoRA provide compact task parameterizations (Houlsby et al., 2019; Hu et al., 2022), and O-LoRA separates low-rank task subspaces to reduce interference during language-model continual learning (Wang et al., 2023). These studies motivate geometric constraints within compact task updates. Low-rank subspace constraints. Within low-rank adaptation, CorDA orients weight decomposi- tion using activation covariance, with its knowledge-preserving mode learning the smallest singular components; LoRA-Null initializes adapters in activation null spaces (Yang et al., 2024; Tang et al., 2026). OPLoRA projects updates away from dominant frozen-weight singular directions, while CLoRA regularizes updates with predefined subspaces (Xiong & Xie, 2026; Lu et al., 2025). These methods specify reference geometry for shared weights or factors, without selecting a local retained reference for each token. Input-conditioned adaptation.Input-conditioned methods make adaptation depend on the current state. TopLoRA generates token-specific diagonal transformations within shared low-rank factors (Li et al., 2025). SLIM mixes LoRA experts with identity paths and adjusts identity routing using distances to target-task activation clusters (Han et al., 2025). PASs-MoE uses pathway activation subspaces to coordinate expert routing and stabilize previously important directions in continual multimodal learning (Hou et al., 2026). These methods establish token-dependent transformations and geometry-aware expert routing. ATLAS uses local retained centers and directions as explicit geometric references for the input and output of one shared residual. We evaluate this conditioning rule against seven published baselines spanning subspace constraints, input conditioning, and token-level strategies. At shared coding requirements on Qwen3-8B, ATLAS achieves 10.16% to 17.60% lower mean retained-output KL (Table 1). Behavioral continuity across updates. Prediction churn measures changes between model ver- sions, and negative-flip analysis isolates newly introduced errors (Milani Fard et al., 2016; Yan et al., 2021). Backward-compatible weight interpolation and MUSCLE extend compatibility objectives to language-model updates (Schumann et al., 2024; Echterhoff et al., 2024). Our answer-level analysis follows this view by distinguishing harmful flips, incidental corrections, and changes between distinct incorrect answers. It shows fewer rewritten mathematical answers and more stable commonsense choices after code adaptation (Section 7). 3RETAINED GEOMETRY FOR TASK ADAPTATION To make retained information part of each update, ATLAS couples an activation atlas with a shared low-rank branch. Letf 0 denote the starting model,Tthe supervised target distribution, andRthe retained distribution. At a selected transformer layer, a token stateh∈R d becomesh ′ = h + ∆(h). Target supervision trains the residual through the downstream model; the atlas supplies its input- dependent geometry. Constructing the atlas.We first sample non-padding activations off 0 on retained calibration inputs and group them intoKlocal neighborhoods. Chartjstores a centerc j ∈R d and an orthonormal basisU j ∈R d×p from local principal component analysis (PCA). Its columns summarize leading activation variation around the center. The backbone and atlas remain fixed during target training; only the shared residual factors are updated. Appendix A specifies sampling, neighborhood construction, and insertion layers. 3 Local charts c j , U j 1 2 3 h Current-state routing α j (h) c(h) = ∑ j α j (h) c j N(h) = I − ∑ j α j (h)U j U j T Routed center and filter Residual amplitude z 2 /stanh ρ(h) h c(h) N(h) Input filter z A r × d tanh B d × r N(h) Output filter Δ(h) h′ Shared trainable A, B Atlas geometry (fixed)Task factors (trainable) Δ(h)=ρ(h)N(h)Btanh(Az) Figure 2: Retained centers and directions condition a shared task residual. Routing formsc(h) andN (h); centering and input filtering producez. Shared trainable factorsA,Blearn the task transformation. Output filtering shapes direction;ρ(h)scales amplitude usingz. The residual is added to h. Routing a token to local geometry. To select the geometry relevant to the current activation, let J (h) contain the q nearest centers. Soft routing assigns α j (h) = exp(−∥h− c j ∥ 2 2 /τ ) P ℓ∈J (h) exp(−∥h− c ℓ ∥ 2 2 /τ ) , j ∈J (h),(1) with zero weight outside this set and temperatureτ > 0controlling the concentration of the routing weights. The mixture defines a center and a local filter, c(h) = X j α j (h)c j , N (h) = I − X j α j (h)U j U ⊤ j .(2) The filter attenuates directions represented by the local PCA bases and preserves their common orthogonal complement. Its action therefore changes with the activation neighborhood. Nonnegative weights summing to one makeN (h)positive semidefinite with eigenvalues in[0, 1]. One chart for the entire retained distribution gives a global tangent-complement projector; mixing charts produces a soft attenuation operator. Filtering the task residual. To limit the residual along dominant directions of retained activation variation, we apply the routed filter to both the input and output of a rank-r branch: z(h) = N (h) h− c(h) ,ρ(h) = tanh(∥z(h)∥ 2 /s),(3) ∆(h) = ρ(h)N (h)B tanh Az(h) ,h ′ = h + ∆(h),(4) whereA ∈R r×d andB ∈R d×r are trainable. Input filtering determines which components the low-rank branch reads. Output filtering shapes the directions it emits. The positive scalescontrols how the residual scaling factorρ(h)grows with the filtered-state norm. Figure 2 distinguishes the atlas geometry from the learned branch. SharingAandBacross charts keeps the trainable parameter count at2dr, independently of geometric resolution. For Qwen3-8B,K = 32,q = 4,p = 16, andr = 16give 131,072 trainable parameters. Ordinary supervised next-token loss trains the target branch, leaving retained geometry available throughout training and inference. We computeN (h)v = v− P j∈J (h) α j (h)U j (U ⊤ j v)using the selectedd×pbases. Input and output filtering reuse the current token’s charts and routing weights, keeping the cost tied to the selected local ranks during generation. 4 4EVALUATING LEARNING AND RETAINED BEHAVIOR Tasks and models.We evaluate the intended use directly: learning code while maintaining existing responses on other inputs. MBPP supplies training problems (Austin et al., 2021), and EvalPlus tests generated programs with the stronger MBPP+ tests (Liu et al., 2023). The 224 evaluation problems are separate from the 120 training and 43 validation problems. GSM8K measures mathematics retention (Cobbe et al., 2021); CommonsenseQA (CSQA) extends evaluation to commonsense question answering (Talmor et al., 2019). Calibration inputs construct the atlas and supply retained examples to Replay and Output-KL. Held-out inputs measure KL and answer changes. Appendix A specifies data partitions, evaluation-subset overlap, and prompts. Qwen3-8B is the main backbone for published-method and residual-structure comparisons. Qwen3.5- 9B supports the detailed mathematical-answer analysis, and Qwen3-8B supplies the CSQA com- parison. To examine transfer across scales and families, the full study also includes Qwen3-4B, Qwen3-14B, and GLM-4-9B (Yang et al., 2025; Qwen Team, 2026; Team GLM et al., 2024). We evaluate three training seeds and complete five-epoch trajectories, using single-layer adaptation and the same evaluation protocol for starting and adapted models. Appendix B presents the complete scale and retained-domain results; Appendix G reports target-learning assessments on other models and tasks. Comparisons.The main comparison covers CorDA-KPM, LoRA-Null, OPLoRA (k ∈16, 128), CLoRA, TopLoRA, STM, and TALR. These methods represent directional constraints, input- conditioned transformations, and token-level training strategies. They share a single-layer adaptation budget, with native weight methods operating on their corresponding modules. To examine the role of retained information more directly, we also compare with an unconstrained residual (Plain), retained-example training (Replay), output distillation (Output-KL), and a one-chart construction (Global-PCA). Appendix C provides exact configurations and comparisons between residual adapters and native weight-based adapters. Validation loss selects learning rates; all three seeds and five-epoch trajectories enter the analysis. Measuring learning and retained movement. The two objectives require complementary mea- surements. MBPP+ pass@1 measures code learning, while retained-output KL measures changes in next-token distributions on shared teacher-forced contexts. Selected-checkpoint tables use the lowest-validation-loss checkpoint from each curve to describe independently selected target scores. WritingV for the evaluated context positions, the distributional measure is d(f,f 0 ) = 1 |V| X (x,t)∈V D KL (p 0 (·| x <t )∥p f (·| x <t )).(5) The reference distribution and evaluated non-padding context positions are fixed within each com- parison. Squared displacement describes the emitted residual at the intervention layer; normalized answer changes measure prediction churn. Because methods reach different coding scores during training, we compare retained movement at shared learning requirements. At requirementk, methodmcontributes the smallest observed KL among checkpoints solving at least k MBPP+ problems: D m (k) = min i: a m,i ≥k d m,i , D m = 1 |K| X k∈K D m (k).(6) Herea m,i is the solved-problem count,d m,i is retained KL, andKis the observed shared integer- score range. We geometrically average seed ratiosD ATLAS /D m . Pairwise ranges preserve learning overlap; the seven-method table uses one range common to every method and seed. This comparison makes the coding requirement explicit when assessing preservation. Appendix F reports alternative ranges, distinct scores, and single-problem sensitivity. 5LEARNING WITH LESS RETAINED-OUTPUT MOVEMENT Lower retained KL than seven published baselines. At the 13 shared coding requirements defined in Section 4, ATLAS achieves lower mean retained KL than all seven published baselines on 5 Table 1: Lower retained KL than seven published baselines. Qwen3-8B, GSM8K, three seeds, and 148–160 solved MBPP+ problems. Mean KL averages requirements within each seed, then summarizes seeds geometrically. All methods have 131,072 trainable parameters except TopLoRA with 135,179. MethodMean KL (×10 −3 )↓ATLAS reduction (%) Activation and directional constraints CorDA-KPM1.40217.60 LoRA-Null1.36015.09 OPLoRA (k = 16)1.32913.07 OPLoRA (k = 128)1.31211.97 CLoRA1.31912.42 Input-conditioned transformation TopLoRA1.38416.53 Token-level training signals STM1.28510.16 TALR1.33913.75 ATLAS1.155— Qwen3-8B (Table 1). The reduction ranges from 10.16% to 17.60% across eight configurations and holds for every training seed. The advantage spans subspace constraints, token-conditioned transformations, and token-level training strategies. Independent validation-selected comparisons with LoRA-Null and OPLoRA provide a complementary view: KL is 67.64% lower with a mean target-score difference of−0.10percentage points (Appendix C). Learning trajectories show the preservation gain. To see how this benefit develops alongside code learning, Figure 3a compares ATLAS with Plain, Replay, and Output-KL on Qwen3-8B. The starting model solves 50.89% of MBPP+ problems; validation-selected ATLAS checkpoints reach 71.43% on average. Over the shared requirements in this panel, ATLAS consistently has lower retained KL than the three comparators. Code learning therefore accompanies a clear reduction in the update’s effect on mathematics outputs. In Figure 3a, horizontal segments reuse checkpoints that satisfy adjacent requirements. Steps mark changes in the contributing checkpoint in at least one seed. The absolute KL axis shows how much the mathematical output distribution moves at each coding level. Complete seed-level results, equal-candidate selection, and alternative shared ranges appear in Appendices C and F. Learning and retention across scales and families. To assess whether the learning–retention benefit extends beyond Qwen3-8B, we evaluate five backbones spanning 4B to 14B parameters, with GSM8K and CSQA as retained domains. Figure 3b brings together the six evaluated settings: all 18 three-seed summaries show lower retained KL. All 54 seed-level comparisons with Plain, Replay, and Output-KL favor ATLAS over pairwise shared coding requirements, with an aggregate KL reduction of 62.03%. Table 6 in Appendix B presents every backbone’s starting and validation-selected target scores alongside its retention results. GLM-4-9B provides the cross-family test. Its starting MBPP+ accuracy is 36.61%, and validation- selected ATLAS checkpoints average 70.54%. At shared coding requirements, retained KL is 67.24% lower than Plain, Replay, and Output-KL in aggregate. This combination of substantial target learning and reduced mathematics-output movement extends the central result to another model family. Complete trajectories, geometric-resolution comparisons, and GLM answer changes appear in Appendices B and E. These comparisons establish improved output preservation alongside target learning. We next examine how retained reference states and local directions contribute to this benefit within the shared residual. 6 67.569.571.5 Required MBPP+ score (%) 1.0 1.5 2.0 2.5 3.0 Retained KL ( ×10 −3 ) Plain Replay Output-KL Qwen3-4B Qwen3-8B Qwen3.5-9B Qwen3-14B GLM-4-9B Qwen3-8B 55.960.260.0 47.543.031.2 79.878.269.1 42.163.738.2 68.880.641.9 71.186.722.0 0306090 (a) Qwen3-8B / GSM8K GSM8K CSQA (b) Retained KL reduction (%) ATLAS Plain Replay Output-KL Figure 3: Reduced retained-output movement across model settings. (a) Qwen3-8B/GSM8K: three-seed geometric mean KL at each shared requirement of 151–161 solved MBPP+ problems. (b) Each seed uses a method pair’s shared requirements; ratios of mean KL are aggregated geometrically. Appendix B provides exact ranges, target scores, and trajectories. 6HOW RETAINED GEOMETRY SHAPES THE RESIDUAL Table 2: Residual structures at common coding requirements. Qwen3-8B/GSM8K, three seeds, and 151–160 solved problems. Mean KL follows Table 1’s aggregation; reductions compare ATLAS with each row. Appendix D reports individual seeds. Residual structureMean KL (×10 −3 )↓ ATLAS reduction (%) Reference states and distances Residual Plain1.971841.30 Centered residual1.21384.64 Distance-scaled residual1.18492.31 Geometric conditioning PCA-derived scaling1.16560.70 Input filter1.16360.52 Output filter1.17381.39 Both filters, no scaling1.17201.24 Global-PCA1.19252.93 Native weight adaptation Native Weight-LoRA1.330212.98 ATLAS1.1575— Centers establish a useful reference. To identify useful retained information, we compare 150 checkpoints from ten residual structures at the same coding requirements (Table 2). Residual Plain and geometric variants share the insertion position, parameter budget, and paired initialization. Centered B tanh(Ax)and distance-scaledtanh(∥x∥ 2 /s d )B tanh(Ax)residuals usex = h− c(h)with the same chart routing. Distance scaling matches mean amplitude on retained calibration activations. Centering reduces KL by 38.44% relative to Plain, and distance scaling adds a 2.39% reduction. Within this comparison, referencing retained centers supplies the largest improvement. Local directions refine the conditioning.Full ATLAS reduces KL by 4.64% relative to centering and 2.31% relative to calibrated distance scaling, with both gains present in every seed. PCA-derived 7 amplitude improves on distance scaling by 1.62%. The complete construction achieves the lowest aggregate KL; compact PCA-scaling and input-filtered forms retain most of its benefit. These compact forms preserve the central mechanism: retained centers, distances, and local directions condition the shared task residual. The advantage over distance scaling persists under single-problem score perturbations (Appendix D). Table 2 also connects this residual comparison to native weight adaptation. Native Weight-LoRA updates the attention output projection with the same rank and parameter count. ATLAS achieves 12.98% lower mean KL at the common coding requirements, extending the preservation advantage to this standard low-rank parameterization. Direction matters at equal effective magnitude.To examine residual orientation, we match each Plain residual to the effective norm of its paired ATLAS residual. Across the four Qwen backbones, ATLAS achieves lower KL in 11 of 12 comparisons, with an aggregate reduction of 3.89%. The preservation benefit therefore includes a contribution from the orientation of the trained residual. Fitted local bases also outperform random bases in all 11 comparisons, supporting the value of retained geometric structure. The local and global chart comparisons further examine geometric resolution. Qwen generally favors local charts, while GLM favors the global construction; complete per-model results appear in Appendices B and D. Cost of keeping geometry active.ATLAS keeps retained geometry active during generation with modest decoding overhead. Appendix H reports the computational-cost measurements. 7FROM OUTPUT MOVEMENT TO ANSWER REWRITING To determine how distributional preservation translates to discrete answers, we evaluate answer churn across the retained benchmark suites. We compare starting and adapted answers to each ofnretained questions. LetFcount correct-to-wrong transitions,Gwrong-to-correct transitions, andWchanges between distinct incorrect answers. For normalized answer labels,C = F + G + Wcounts changed answers, the churn rate is C/n, and ∆Accuracy = G− F n .(7) Accuracy records the balance of errors and corrections. This decomposition follows prediction-churn and negative-flip analysis (Milani Fard et al., 2016; Yan et al., 2021), revealing which aspects of existing behavior the adaptation preserves. Teacher-forced KL measures distributional changes on shared reference prefixes, while answer churn records changes in normalized answer labels. In generated answers, each token also shapes the context for subsequent predictions, making answer-level evaluation useful for tracing changes through to the final response. Fewer rewritten mathematical answers. The output-preservation gain carries through to math- ematical answers (Figure 4a). On Qwen3.5-9B, the mean fraction of rewritten answers falls from 22.43% to 10.18%, a 54.60% reduction across the nine matched comparisons. Correctness-transition reductions also persist under changes to batch composition and question order (Appendix F). Conservation acts in both directions.The transition types explain what this continuity preserves. Qwen3.5 reduces harmful flips by 59.86% and incidental corrections by 59.57%, yielding closely balanced reductions in both transition types. The resulting continuity covers both previously correct and previously incorrect answers. Appendix E details the paired counts and accuracy differences, including the complete cross-family behavior analysis. Stability within the incorrect-answer set. We also examine changes between distinct incorrect answers. In eight target-matched Qwen3-8B comparisons with LoRA-Null and OPLoRA, ATLAS reduces these changes from 123 to 96, a 21.95% reduction. Correctness-transition totals remain close, at 248 and 251. This comparison reveals a concrete benefit beyond correctness labels: the identity of 8 ATLAS Matched comparators 0 5 10 15 20 25 Churn rate (%) 10.18% 22.43% F 2.50 G 3.77 W 3.91 F 6.23 G 9.33 W 6.87 00.51.01.5 Choice churn (%) ATLAS Matched comparators 0.853% 1.385% (a) GSM8K / Qwen3.5-9B 54.60% less answer churn (b) CSQA / Qwen3-8B 38.42% less choice churn (c) CSQA choice accuracy (%) Starting model85.50 ATLAS85.22 Matched comparators85.20 F: correct to wrongG: wrong to correctW: between incorrect answers Figure 4: Answer continuity in mathematics and commonsense. All changes are relative to the starting model. Adapted-model values average rates equally across checkpoint pairs matched on MBPP+ counts. (a) Qwen3.5-9B: nine pairs with Plain, Replay, and Output-KL on 1,319 GSM8K questions;F,G, andWshare the full-question denominator. (b,c) Qwen3-8B: 12 pairs with Plain, Replay, Output-KL, and Global-PCA on 1,221 CSQA questions, showing choice churn and conditional-choice accuracy. Appendix E provides pairings, transition counts, and full GLM and CSQA results. existing answers changes less. The full transition breakdown and alternative scoring results appear in Appendix E. More stable commonsense choices.Beyond mathematical answers, CSQA lets us examine which option the adapted model selects (Figure 4b,c). Conditional choice selects the highest-likelihood option. ATLAS reduces choice churn by 38.42% while maintaining nearly the same conditional-choice accuracy as the matched methods. Appendix E examines how these choices are expressed through short generation, reporting absolute accuracies, invalid-label rates, and format transitions together. Across mathematics and commonsense, answer-level analysis thus makes the preservation benefit concrete: fewer existing answers change after target adaptation. 8CONCLUSION Learning a specialized task can also change a language model’s answers on other domains. We introduced ATLAS to use retained activations when computing a shared low-rank task residual. Local centers and directional filters condition the residual at each token, while target supervision trains its shared factors. The atlas remains active throughout training and inference without introducing separate trainable branches for individual charts. On Qwen3-8B, ATLAS achieves lower mean retained-output KL than all seven published baselines at shared coding requirements, consistently across three training seeds. Structural comparisons identify retained centers as the largest source of improvement over the unconstrained residual, with distances and local directions further refining preservation. PCA-scaling and input-filtered variants retain most of the KL reduction observed in the structural comparison, offering simpler residual structures within the same retained-geometry formulation. Experiments across five backbones and two retained domains demonstrate coding gains with reduced output movement. Answer-level analyses connect this preservation to fewer rewritten mathematical answers and more stable commonsense choices. Separating harmful flips, incidental corrections, and changes among incorrect answers makes clear which aspects of established behavior persist through adaptation. These findings establish retained activation geometry as a compact computational resource for learning specialized skills while controlling changes to existing responses. 9 Future work can extend retained-geometry conditioning across layers to study where reference states most effectively guide adaptation. Coordinating layer-specific atlases would allow residual capacity to be allocated according to each layer’s role in task learning and response preservation. A further direction is to select chart counts and local ranks from activation structure, using neighborhood density and directional variation to identify where finer geometry is useful. This would connect geometric resolution to the storage and computation needed to maintain it. For sequential specialization, comparing fixed and updated reference distributions could clarify how an atlas should represent both established behavior and newly acquired skills. Tracking the same questions across successive updates, with harmful flips, incidental corrections, and changes among incorrect answers recorded separately, would reveal how these reference choices shape answer continuity as the model acquires and integrates new skills. 10 REFERENCES Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Hkf2_sC5FX. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/ abs/2110.14168. Jessica Maria Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, and Hadi Pouransari.MUSCLE: A model update strategy for compatible LLM evolution.In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 7320–7332, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.430. URLhttps://aclanthology.org/2024. findings-emnlp.430/. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, p. 3762–3773. PMLR, 2020. URL https://proceedings.mlr.press/v108/farajtabar20a.html. Jiayi Han, Liang Du, Hongwei Du, Xiangguo Zhou, Yiwen Wu, Yuanfang Zhang, Weibo Zheng, and Donghong Han. SLIM: Let LLM learn more and forget less with soft LoRA and identity mixture. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4792–4804, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025. naacl-long.246. URL https://aclanthology.org/2025.naacl-long.246/. ZhiYan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, and Jinqiao Wang. PASs-MoE: Mitigating misaligned co-drift among router and experts via pathway activation subspaces for continual learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31959–31972, San Diego, California, United States, 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.1474. URL https://aclanthology.org/2026.acl-long.1474/. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, p. 2790–2799. PMLR, 2019. URLhttps: //proceedings.mlr.press/v97/houlsby19a.html. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URLhttps://openreview.net/forum? id=nZeVKeeFYf9. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. doi: 10.1073/pnas.1611835114. URLhttps://doi.org/10.1073/pnas.1611835114. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In Advances in Neural Information Processing Systems, 11 volume 38, p. 136414–136435. Curran Associates, Inc., 2025. doi: 10.52202/085713-4552. URLhttps://proceedings.neurips.c/paper_files/paper/2025/hash/ c74429a930838b5bac03c681f0a0ce11-Abstract-Conference.html. Zhizhong Li and Derek Hoiem. Learning without forgetting. In Computer Vision – ECCV 2016, p. 614–629. Springer International Publishing, 2016. doi: 10.1007/978-3-319-46493-037. URL https://link.springer.com/chapter/10.1007/978-3-319-46493-0_37. Jiacheng Lin, Zhongruo Wang, Kun Qian, Tian Wang, Arvind Srinivasan, Hansi Zeng, Ruochen Jiao, Xie Zhou, Jiri Gesi, Dakuo Wang, Yufan Guo, Kai Zhong, Weiqi Zhang, Sujay Sanghavi, Changyou Chen, Hyokun Yun, and Lihong Li.SFT doesn’t always hurt general capabilities:Revisiting domain-specific fine-tuning in LLMs.In International Conference on Learning Representations, p. 46954–46992, 2026. URLhttps://proceedings.iclr.c/paper_files/paper/2026/hash/ 4e447acb68f57e29234bc0eb19896f11-Abstract-Conference.html. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.Is your code gen- erated by ChatGPT really correct?Rigorous evaluation of large language models for code generation.In Advances in Neural Information Processing Systems, volume 36, p. 21558–21572. Curran Associates, Inc., 2023.doi: 10.52202/075280-0943.URL https://proceedings.neurips.c/paper_files/paper/2023/hash/ 43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learn- ing.In Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URLhttps://proceedings.neurips.c/paper_files/paper/ 2017/hash/f87522788a2be2d171666752f97ddebb-Abstract.html. Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, and Xiaojie Wang. Controlled low- rank adaptation with subspace regularization for continued training on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 19165–19181, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.940. URLhttps://aclanthology.org/ 2025.acl-long.940/. Mahdi Milani Fard, Quentin Cormier, Kevin Canini, and Maya Gupta. Launch and iterate: Reducing prediction churn. In Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URLhttps://proceedings.neurips.c/paper_files/paper/ 2016/hash/dc5c768b5dc76a084531934b34601977-Abstract.html. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen. ai/blog?id=qwen3.5. Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021. URLhttps://openreview. net/forum?id=3AOj0RCNC2. Raphael Schumann, Elman Mansimov, Yi-An Lai, Nikolaos Pappas, Xibin Gao, and Yi Zhang. Backward compatibility during data updates by weight interpolation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2846–2861, St. Julian’s, Malta, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.174. URLhttps://aclanthology.org/ 2024.eacl-long.174/. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4149–4158, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URLhttps:// aclanthology.org/N19-1421/. 12 Pengwei Tang, Xiaolin Hu, Yong Liu, Lizhong Ding, Dongjie Zhang, Xing Wu, and Debing Zhang. Put the space of LoRA initialization to the extreme to preserve pre-trained knowl- edge. Proceedings of the AAAI Conference on Artificial Intelligence, 40(39):33232–33240, 2026. doi: 10.1609/aaai.v40i39.40608. URLhttps://ojs.aaai.org/index.php/ AAAI/article/view/40608. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools, 2024. URL https://arxiv.org/abs/2406.12793. Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 10658–10671, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.715. URL https://aclanthology.org/2023.findings-emnlp.715/. Chao-Chung Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung (Vivian) Chen, Shao-Hua Sun, and Hung-yi Lee.Mitigating forgetting in LLM fine-tuning via low-perplexity token learning.In Advances in Neural Information Processing Systems, volume 38, p. 1708–1744. Curran Associates, Inc., 2025.doi:10.52202/085713-0064.URL https://proceedings.neurips.c/paper_files/paper/2025/hash/ 027e86facfe7c1ea52ca1fca7bc1402b-Abstract-Conference.html. Yifeng Xiong and Xiaohui Xie. OPLoRA: Orthogonal projection LoRA prevents catastrophic forgetting during parameter-efficient fine-tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34088–34096, 2026. doi: 10.1609/aaai.v40i40.40703. URLhttps://ojs. aaai.org/index.php/AAAI/article/view/40703. Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14299–14308, 2021. doi: 10.1109/CVPR46437.2021.01407. URLhttps://openaccess.thecvf.com/ content/CVPR2021/html/Yan_Positive-Congruent_Training_Towards_ Regression-Free_Model_Updates_CVPR_2021_paper.html. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning. In Advances in Neural Information Processing Systems, volume 37, p. 71768–71791. Curran Associates, Inc., 2024. doi: 10.52202/079017-2292. URLhttps://proceedings.neurips.c/paper_files/paper/2024/hash/ 83f95b0ac5046338ea2afe3390e9f4b-Abstract-Conference.html. 13 ADATA AND EVALUATION PROTOCOL A.1TARGET AND RETAINED DATA The target corpus uses the sanitized MBPP training, validation, and test partitions. We train on 120 training problems and select configurations with negative log-likelihood on 43 validation problems. Code generation covers the 257 sanitized test prompts; the fixed intersection with MBPP+ contains 224 problems. All target accuracies and integer target scores in this paper use these 224 problems. MBPP training, validation, and test problems come from separate source partitions and retain their original task identifiers. The MBPP prompt presents the task description, imports, and supplied tests, followed by a code completion. GSM8K prompts use a question followed by an answer prefix. Retained KL compares adapted and frozen distributions under the same teacher-forced sequence and averages over non- padding tokens. Output-KL regularization draws its retained training examples from the same 2,048-example pool used by Replay. GSM8K generation uses the full 1,319-problem test partition. The 512 retained test examples supply the distributional measurements, so distributional and answer- level results share these examples. For CSQA, 2,048 training examples supply the atlas and Replay. The retained evaluation uses the 1,221-question validation partition, with a 512-question subset for KL. Conditional choice evaluates five option labels under the shared prompt. Short generation emits at most two tokens and parses an A–E label; unparsable outputs count as incorrect. Table 3 lists the roles and overlaps. Table 3: Data roles. Atlas fitting and Replay share their training examples. Retained-KL subsets belong to the corresponding behavior evaluation set. MBPP training, validation, and test use separate source partitions. DatasetUseExamplesRelation to other uses MBPPTarget training120Training partition MBPPValidation NLL43Validation partition MBPPCode generation257Test partition MBPP+Pass@1224Fixed intersection of test problems GSM8KAtlas fitting / Replay2,048Same training examples GSM8KRetained KL512Subset of behavior test GSM8KAnswer evaluation1,319Disjoint from atlas / Replay CSQAAtlas fitting / Replay2,048Same training examples CSQARetained KL512Subset of behavior validation CSQAChoice and generation1,221Disjoint from atlas / Replay A.2TRAINING AND SELECTION Each main trajectory contains five epochs and uses seeds 7302026, 7302027, and 7302028. Backbone weights remain fixed. The standard ATLAS residual has rank 16 and scalar scale 64. The four Qwen studies use a learning rate of2× 10 −4 . For GLM, learning-rate selection evaluates three rates with Plain and selects4× 10 −4 . Replay and Output-KL use fixed coefficients within each training trajectory; Appendix F reports their coefficient sensitivity. Training uses BF16 computation, token chunking, and gradient accumulation on one RTX 5090. The insertion module is a decoder-layer output near the middle of the backbone. Qwen3-8B and GLM- 4-9B use zero-indexed layers 17 and 19. Their hidden dimension is 4,096, giving2dr = 131,072 trainable residual parameters. Qwen3-4B has 81,920 residual parameters, and Qwen3-14B has 163,840. Centers and PCA bases are fixed geometric parameters and are excluded from these trainable counts. Table 4 lists the backbone identifiers and ATLAS training settings. AdamW uses(β 1 ,β 2 ) = (0.9, 0.95),ε = 10 −8 , and zero weight decay. Each optimizer step accumulates 16 target examples. All five backbones use BF16 computation, gradient checkpointing, a maximum sequence length of 14 1,024 tokens, and five training epochs. The down-matrixAis initialized with standard deviation 0.02 and the up-matrixBis zero. Corresponding residual constructions share the same initial factors for each seed and begin with zero residual. The local atlas contains 32 charts of tangent rank 16, selects four charts per token, and uses routing temperature 64. The residual scaling parameter is also 64. Table 4: Backbone identifiers and ATLAS training settings. Layer indices are zero-based; batch denotes target microbatch size multiplied by gradient-accumulation steps. The listed learning rates apply to all three training seeds. Backbone identifierLayerRankBatchLearning rateMax. tokens Qwen/Qwen3-4B-Base17162× 82× 10 −4 1,024 Qwen/Qwen3-8B-Base17161× 162× 10 −4 1,024 Qwen/Qwen3.5-9B-Base15161× 162× 10 −4 1,024 Qwen/Qwen3-14B-Base19161× 162× 10 −4 1,024 ZhipuAI/glm-4-9b-hf19161× 164× 10 −4 1,024 Validation-selected results choose the checkpoint with the lowest target validation NLL in each trajectory. Target-score comparisons use all five measured checkpoints. Behavioral analyses compare checkpoints at matched target scores on the same retained questions; Appendix E.1 specifies each selection rule and pair. These summaries describe configuration selection, achieved learning levels, and itemwise model changes, respectively. A.3INTEGER TARGET-SCORE COMPARISON We apply Eq. 6 separately within each training seeds, writingD ms (k)for methodm’s minimum observed retained KL at integer target countk. Each selected value belongs to a measured checkpoint. For ATLAS and comparatorb, the shared setK bs contains integer counts in the intersection of their observed score ranges, restricted to counts above the frozen model. We report R bs = |K bs | −1 P k∈K bs D ATLAS,s (k) |K bs | −1 P k∈K bs D bs (k) .(8) The three-seed summary is(R b1 R b2 R b3 ) 1/3 , and the relative reduction is one minus this ratio. Counts are inclusive, sok = 148,..., 160provides 13 target levels. The calculation uses the discrete score resolution directly. The seven-method Qwen3-8B comparison uses one common set,k = 148,..., 160, for all configura- tions and all three seeds. Pair-specific ranges retain the achieved score overlap within the core model studies. Appendix F also reports a single range shared across methods and seeds within each setting. Continuous target-score averaging. For the supplementary continuous-target summaries, each method assigns to each of its observed target scores the smallest measured metric among checkpoints attaining at least that score. Each value is held constant from that score to the next observed score. We integrate the resulting step curve over the pair’s shared observed score range, divide by its width, and form the ratio of the ATLAS and comparator averages. This calculation weights intervals by score width; the integer-target rule evaluates and averages the attainable minimum at each integer count. The continuous calculation is used for the external-baseline single-problem sensitivity analysis in Appendix F. A.4ACTIVATION SAMPLING AND LOCAL PCA SUPPORT The standard Qwen3-8B extraction uses 2,048 GSM8K training examples and produces 4,096 vectors at decoder-layer index 17. Each example supplies two views: its formatted question prompt and that same prompt followed by its gold answer. Atlas fitting uses these retained-domain views; diagnostic MBPP examples belong to a separate split. 15 Table 5: Qwen3-8B weight-adaptation settings. Every method updates the same attention-output matrix. OPLoRA’s protected subspace widthkis separate from its trainable rank. The learning rates apply to all three training seeds. MethodRank Mechanism parametersLearning rate LoRA-Null16Lowest 16 activation-covariance eigenvectors for initial- ization 2×10 −4 OPLoRA16Weight-SVD complement projections, k = 162×10 −4 OPLoRA16Weight-SVD complement projections, k = 1282×10 −4 CorDA-KPM16Smallest 16 context-oriented singular components4×10 −4 TopLoRA11 α = 16; token scale exp(RMSNorm(Cx)); dropout 04×10 −4 STM16 Retain response tokens with starting-model perplexity ≤ 2.5 1×10 −4 CLoRA16Subspace width 512; penalty coefficient λ = 14×10 −4 TALR16 Weight floor 0.01; temperature from the effective update batch 2×10 −4 Token selection is deterministic and stratified over sequence position. For zero-based example index i, letb i = i mod 8. A view withLnon-padding tokens contributes the hidden state at zero-based non-padding rank t i = (2b i + 1)L 16 . The two views use the same stratum and their own sequence lengths. Extraction uses left padding, a maximum length of 1,024, and bfloat16 forward computation. Atlas fitting operates on the combined 4,096 retained-domain vectors in float32 precision. The main experiments use 32 clusters obtained with Lloyd K-means, 10 initializations, and seed 7302026. Each chart’s PCA uses its assigned cluster members, centered at their mean; the leading 16 right singular vectors form its basis. Chart support therefore varies with cluster membership: chartk usesn k vectors, with P k n k = 4, 096 . Routing uses the four nearest chart centers and temperature 64. The fitting-time benchmark measures an equal-support construction around each K-means center. Its support rule,n support = max(p + 1,⌈n/K⌉), selects the nearest 128 vectors for basis rank p = 16,n = 4, 096, andK = 32. These neighborhoods may overlap. PCA is centered at the neighborhood mean, while routing retains the K-means center. Inference-throughput measurements use the cluster-member atlas from the main experiments. A.5WEIGHT-ADAPTATION MODULES AND METHOD PARAMETERS All weight-adaptation comparisons in Table 1 act on the attention output projection (oproj) at decoder-layer index 17 of Qwen3-8B. This is a4, 096× 4, 096matrix. The standard factors have rank 16 and scalingα/r = 1; TopLoRA uses rank 11 andα = 16to include its token-conditioned factors within a similar parameter budget. ATLAS and residual Plain are inserted at decoder-layer output, as specified in the method section. The native Weight-LoRA calibration compares these two insertion forms. LoRA-Null and CorDA-KPM collect module-input activations from 2,048 retained prompt–response examples with calibration seed 7302026. Non-padding tokens enter covariance estimation. CorDA additionally normalizes each example’s activation matrix by its maximum absolute entry. Its covari- ance regularization begins at0.01 mean(diag(C))Iand doubles until the inverse-residual criterion is below 0.05. Both decomposition-based initializations subtract their initialB 0 A 0 contribution, preserving the starting function. CLoRA constructs independent input and output orthonormal bases by QR factorization, using seeds 7302026 and 7302027. Its added loss is λ 2 ∥AQ in ∥ 2 F +∥B ⊤ Q out ∥ 2 F . 16 STM computes its response-token mask once with the starting model. TALR uses detached weights maxexp(−ℓ i /τ ), 0.01, whereℓ i is the response-token NLL andτis the median of the per-sequence mean response-token NLLs across the effective update batch, including accumulated microbatches. Its weighted token loss is normalized by the number of supervised response tokens. BCOMPLETE LEARNING RESULTS The learning comparison comprises 300 Qwen checkpoints and 75 GLM checkpoints: five backbones, five methods, three seeds, and five epochs. Evaluation with CSQA retained data covers 75 Qwen3-8B checkpoints under the same five-method design. Plain’s training objective is independent of the retained domain; identical trained parameters are therefore evaluated on GSM8K and CSQA. Table 6: Target learning and retained-output movement across model scales and domains. Target scores average three validation-selected checkpoints. KL ratios compare complete trajectories over pairwise shared integer target requirements. Ratios below one favor ATLAS. All five backbones and both retained domains are included. MBPP+ (%)ATLAS / comparator KL↓ ModelRetained Frozen Plain ATLAS Plain Replay Output-KL Qwen3-4BGSM8K2.23 35.7114.73 0.4410.3980.400 Qwen3-8BGSM8K50.89 72.1771.43 0.5250.5700.688 Qwen3.5-9B GSM8K62.50 63.5465.62 0.2020.2180.309 Qwen3-14BGSM8K65.18 71.1372.62 0.5790.3630.618 GLM-4-9BGSM8K36.61 65.7770.54 0.3120.1940.581 Qwen3-8BCSQA50.89 72.1771.73 0.2890.1330.780 Table 7 gives all 54 primary seed-level ratios and their integer score ranges. The comparison pools 760 target levels across model–comparator–seed combinations. ATLAS has lower KL at 743 levels and lower mean KL in all 54 comparisons. The all-seed geometric reduction is 62.03%. Absolute target accuracy, validation NLL, retained KL, and intervention-layer displacement characterize learning and preservation throughout each trajectory. Aggregating seed ratios within each setting gives KL reductions of 58.56% across the four Qwen backbones on GSM8K, 68.96% on Qwen3-8B with CSQA, and 67.24% on GLM-4-9B with GSM8K. Requiring a range shared across methods and seeds within each setting retains 53 favorable seed-level comparisons out of 54 and all 18 favorable three-seed summaries (Appendix F). The seed-level tables and complete trajectories below show the scores and KL values underlying these summaries. Target-learning trajectories.Across all 15 ATLAS checkpoints per Qwen backbone, mean MBPP+ scores are 9.08% for Qwen3-4B, 69.76% for Qwen3-8B, 64.82% for Qwen3.5-9B, and 71.43% for Qwen3-14B. Their frozen scores are 2.23%, 50.89%, 62.50%, and 65.18%. Mean gains are 6.85, 18.87, 2.32, and 6.25 percentage points. GLM starts at 82/224 and reaches 157, 158, and 159 solved problems at its best ATLAS checkpoint in the three seeds. One-chart comparison. Global-PCA uses one retained-data chart and the same residual architec- ture. Table 8 reports its comparison with ATLAS. The Qwen results generally favor the local atlas, while GLM favors the one-chart construction. Observed checkpoints. Figures 5–10 show the five-epoch measurements separately for each training seed. Each panel plots MBPP+ accuracy against retained KL at the five evaluated checkpoints per method for one training seed. The tabulated comparisons use the integer target rule in Eq. 6. 17 Table 7: Primary target-score comparisons, by training seed. Each cell gives the ratio of mean retained KL (ATLAS/comparator), followed by the inclusive integer MBPP+ range[k min ,k max ]. Every value uses observed checkpoints meeting each target threshold. GSM8K is retained unless CSQA is named. Backbone / retained domain Comparator730202673020277302028 Qwen3-4BPlain0.444 [12,35]0.465 [11,37]0.416 [11,27] Qwen3-4BReplay0.343 [12,35]0.429 [11,37]0.428 [10,27] Qwen3-4BOutput-KL0.458 [12,35]0.378 [8,37]0.370 [13,27] Qwen3-8BPlain0.578 [150,161] 0.526 [144,161] 0.477 [151,163] Qwen3-8BReplay0.571 [149,161] 0.581 [148,161] 0.558 [151,162] Qwen3-8BOutput-KL 0.621 [147,161] 0.686 [145,161] 0.765 [148,163] Qwen3.5-9BPlain0.281 [143,148] 0.177 [141,148] 0.165 [143,147] Qwen3.5-9BReplay0.177 [142,147] 0.156 [143,148] 0.378 [144,147] Qwen3.5-9BOutput-KL 0.280 [141,147] 0.257 [142,149] 0.411 [143,147] Qwen3-14BPlain0.558 [150,164] 0.572 [151,164] 0.607 [153,165] Qwen3-14BReplay0.287 [151,164] 0.329 [151,166] 0.508 [154,165] Qwen3-14BOutput-KL 0.540 [154,164] 0.604 [151,166] 0.724 [153,163] GLM-4-9BPlain0.392 [144,156] 0.230 [141,156] 0.336 [143,155] GLM-4-9BReplay0.248 [147,155] 0.131 [144,155] 0.225 [145,153] GLM-4-9BOutput-KL 0.635 [145,156] 0.360 [145,154] 0.859 [143,156] Qwen3-8B / CSQAPlain0.319 [150,161] 0.260 [144,162] 0.291 [151,162] Qwen3-8B / CSQAReplay0.112 [146,161] 0.147 [142,162] 0.142 [145,161] Qwen3-8B / CSQAOutput-KL 0.753 [147,161] 0.781 [149,161] 0.807 [148,161] Table 8: ATLAS/Global-PCA mean-KL ratios on each pair’s shared integer target range. Values below one favor the local atlas. GSM8K is retained unless CSQA is named. Backbone / retained domain730202673020277302028 Qwen3-4B0.87180.87970.9278 Qwen3-8B0.97210.97460.9814 Qwen3.5-9B0.95530.99110.9593 Qwen3-14B0.97270.97971.0002 GLM-4-9B2.94253.58864.4319 Qwen3-8B / CSQA0.97510.95950.9566 1530 MBPP+ (%) 10 −3 6 × 10 −4 2 × 10 −3 3 × 10 −3 4 × 10 −3 6 × 10 −3 Retained KL (log scale) Seed 7302026 1530 MBPP+ (%) Seed 7302027 1530 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 5: Qwen3-4B with GSM8K retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 18 646872 MBPP+ (%) 2 × 10 −3 3 × 10 −3 4 × 10 −3 6 × 10 −3 Retained KL (log scale) Seed 7302026 646872 MBPP+ (%) Seed 7302027 646872 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 6: Qwen3-8B with GSM8K retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 63.064.566.0 MBPP+ (%) 10 −3 10 −2 Retained KL (log scale) Seed 7302026 62.565.067.5 MBPP+ (%) Seed 7302027 63.064.566.0 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 7: Qwen3.5-9B with GSM8K retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 19 6872 MBPP+ (%) 10 −3 10 −2 Retained KL (log scale) Seed 7302026 6872 MBPP+ (%) Seed 7302027 666972 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 8: Qwen3-14B with GSM8K retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 4060 MBPP+ (%) 10 −3 10 −2 10 −1 Retained KL (log scale) Seed 7302026 4060 MBPP+ (%) Seed 7302027 4060 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 9: GLM-4-9B with GSM8K retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 20 646872 MBPP+ (%) 10 −2 2 × 10 −3 3 × 10 −3 4 × 10 −3 6 × 10 −3 2 × 10 −2 3 × 10 −2 4 × 10 −2 6 × 10 −2 Retained KL (log scale) Seed 7302026 606570 MBPP+ (%) Seed 7302027 666972 MBPP+ (%) Seed 7302028 ATLASPlainReplayOutput-KLGlobal-PCA Figure 10: Qwen3-8B with CSQA retained. The three panels show individual seeds and all five measured checkpoints for each internal method. Retained KL uses a logarithmic axis. 21 CPUBLISHED BASELINES AND ADAPTATION COMPARISONS C.1IMPLEMENTATION AND TUNING The external comparison evaluates CorDA-KPM, LoRA-Null, OPLoRA, CLoRA, TopLoRA, STM, and TALR. OPLoRA uses bothk = 16andk = 128. Eight configurations across three seeds give 24 training curves and 120 checkpoint evaluations on Qwen3-8B. The methods use the same MBPP training problems, validation selection, three seeds, five-epoch schedule, and approximately matched trainable parameter counts. Weight-adapter methods retain their native linear update formulation. ATLAS and the internal Plain, Replay, and Output-KL controls use the residual-branch formulation. Standard weight and residual adapters have rank 16 and 131,072 parameters at the Qwen3-8B insertion module. TopLoRA uses rank 11 and 135,179 parameters, accommodating its token-dependent diagonal coefficients. CorDA-KPM constructs its context-oriented decomposition from retained activations. LoRA-Null uses a retained activation subspace for initialization, and OPLoRA applies two-sided fixed weight- subspace constraints. CLoRA uses fixed random orthogonal constraints. STM masks target tokens according to frozen-model difficulty, while TALR reweights the token loss during training. TopLoRA conditions the low-rank update on each token. Table 9: Learning-rate selection for the published-method comparison. Each listed method evaluates 10 −4 ,2× 10 −4 , and4× 10 −4 on seed 7302026. Minimum validation NLL over epochs 1–5 selects the rate used by all three seeds. MethodSelected learning rate CorDA-KPM4× 10 −4 TopLoRA4× 10 −4 CLoRA4× 10 −4 STM10 −4 TALR2× 10 −4 Table 10: External methods at the same 13 target thresholds,k ∈148,..., 160out of 224 MBPP+ problems. Entries are ATLAS/comparator ratios of mean retained KL. The last column geometrically averages the three seed ratios. Comparator730202673020277302028Geometric mean CLoRA0.88930.87470.86360.8758 CorDA-KPM0.77800.74900.96000.8240 LoRA-Null0.84660.85660.84420.8491 OPLoRA (k = 128)0.88870.87680.87550.8803 OPLoRA (k = 16)0.87890.86230.86680.8693 STM0.92020.88430.89120.8984 TALR0.88030.85900.84860.8625 TopLoRA0.74500.92300.84570.8347 C.2INDEPENDENT VALIDATION SELECTION To examine the learning and retention outcomes of validation-based checkpoint selection, we select each ATLAS and published-baseline trajectory independently by its lowest target-validation NLL over five epochs. Table 11 reports the resulting coding accuracy, mathematical accuracy, and retained KL for all seven published methods across their eight configurations. ATLAS achieves lower retained KL in all 24 seed-level comparisons, with geometric mean reductions of 15.47% to 88.00% across configurations. These results extend the retention advantage to checkpoints selected using target validation data. 22 TALR selects epoch three for all seeds; TopLoRA selects epochs five, five, and four for seeds 7302026, 7302027, and 7302028, respectively. ATLAS and the remaining configurations select epoch five. Within the LoRA-Null and OPLoRA subset cited in Section 5, the nine comparisons yield a geometric KL reduction of 67.64%, a mean ATLAS-minus-comparator target difference of−0.099percentage points, and a mean GSM8K difference of−0.067 percentage points. Table 11: Independent validation selection on Qwen3-8B. Each trajectory contributes its checkpoint with the lowest target-validation NLL. MBPP+ and GSM8K are arithmetic means over three training seeds; retained KL is a geometric mean. KL reduction is one minus the geometric mean of paired ATLAS-to-comparator ratios, expressed as a percentage. MethodMBPP+ (%)GSM8K (%)KL (×10 −3 )Reduction (%) ATLAS71.4386.381.2876— CorDA-KPM64.5886.5810.728088.00 LoRA-Null72.0286.284.163269.07 OPLoRA (k = 16)71.1386.584.033568.08 OPLoRA (k = 128)71.4386.483.750265.66 CLoRA69.4986.505.013074.31 TopLoRA63.6986.1010.689787.95 STM72.4786.151.523215.47 TALR72.9286.431.744726.20 Fixed target-matched checkpoints. The target-matched LoRA-Null/OPLoRA comparison fixes one ATLAS reference per seed and matches comparator checkpoints within one target-accuracy percentage point. Eight pairs meet this rule: three LoRA-Null, three OPLoRA-16, and two OPLoRA- 128. Three pairs have equal MBPP+ counts, three differ by one problem, and two differ by two problems. All eight have lower ATLAS KL, with a geometric reduction of 50.32%. These Qwen3-8B results summarize three training seeds. Residual and weight-adapter comparison. ATLAS, residual Plain, and native Weight-LoRA contribute 45 Qwen3-8B checkpoints to the unified residual-structure comparison. Table 2 evaluates these trajectories together with the geometric variants at shared integer coding requirements. Ap- pendix D.1 reports the individual seeds over the common target range; Appendix A.5 specifies the insertion modules and parameter budgets. DCOMPONENT DEFINITIONS AND GEOMETRIC CONTROLS D.1RESIDUAL STRUCTURES AT SHARED INTEGER TARGET SCORES The unified comparison evaluates ten Qwen3-8B residual structures over five epochs and three training seeds, giving 150 checkpoints. The structures comprise ATLAS, residual Plain, Global-PCA, four PCA component variants, centered and distance-scaled residuals, and native Weight-LoRA. Their 30 trajectories share an observed range of 151–160 solved MBPP+ problems out of 224. For each integer requirementk, we select the checkpoint with the lowest retained KL among those solving at leastkproblems and average these ten KL values within each seed. Table 2 reports geometric means across the three seeds for both mean KL and the ratio of ATLAS KL to comparator KL, expressing the latter as a percentage reduction. Native Weight-LoRA uses the same coding and GSM8K data, rank 16, and the attention output projection at layer 17. The seed-level results support the aggregate reductions in Table 2. ATLAS has lower KL in every seed for all structures except input filtering, whose 0.52% aggregate reduction comprises two favorable seeds and one at−0.008%. The scalar used by PCA-derived scaling depends on the local atlas throughρ(h) = tanh(∥N (h)(h− c(h))∥ 2 /s). Its trainable branch reads the centered state, while input filtering additionally applies N (h)beforeA. Centering reduces KL by 38.44% relative to Plain; distance scaling reduces it by a further 2.39% relative to centering; and PCA-derived scaling reduces it by 1.62% relative to 23 Table 12: Residual structures at the same ten integer target requirements, 151–160 solved MBPP+ problems out of 224. Each KL value averages the best achieved KL over these requirements within one training seed. Ratios divide the ATLAS value by the corresponding method value. Observed ranges describe all five checkpoints in each trajectory. StructureSeedObserved rangeMean KL×10 3 KL ratio Residual Plain7302026150–1622.07730.55920 7302027144–1622.03410.56939 7302028151–1631.81430.63536 Centered residual7302026144–1631.22510.94819 7302027147–1631.20920.95781 7302028149–1621.20730.95481 Distance-scaled residual7302026136–1621.19140.97499 7302027133–1631.17660.98433 7302028141–1621.18650.97150 PCA-derived scaling7302026139–1601.17580.98798 7302027143–1621.16060.99796 7302028148–1631.16060.99318 Input filter7302026141–1621.16151.00008 7302027139–1611.16610.99326 7302028146–1621.16320.99102 Output filter7302026141–1611.16820.99437 7302027142–1631.18110.98059 7302028144–1611.17210.98343 Both filters, no scaling7302026148–1621.16920.99350 7302027148–1631.17160.98855 7302028151–1621.17530.98081 Global-PCA7302026143–1611.20080.96734 7302027143–1621.19610.96829 7302028142–1611.18070.97635 Native Weight-LoRA7302026141–1661.30630.88923 7302027139–1631.35140.85703 7302028144–1661.33320.86464 ATLAS7302026140–1611.16161.00000 7302027143–1611.15821.00000 7302028148–1631.15271.00000 24 distance scaling. All three steps are favorable in every seed. Each reduction uses its own comparator denominator. D.2CENTERING AND CALIBRATED DISTANCE SCALING To distinguish the information supplied by retained centers and distances, we compare the centered and distance-scaled residuals defined in Table 14, withx = h− c(h). Both use the same 32 retained centers and top-four routing with temperature 64 as ATLAS. In both constructions,c(h)varies with the token and carries local retained information into the branch input. We train the layer-17 rank-16 residual for five epochs at learning rate2× 10 −4 , leaving the Qwen3-8B backbone and atlas fixed. Microbatch size 1 and gradient accumulation 16 give 40 optimizer steps per curve on the 120 target-training examples. Seeds 7302026–7302028 are shared with the other structures; evaluation uses the same 224 MBPP+ problems and 512 retained examples. We calibrate the distance scale tos d ≈ 78.6227by matching mean amplitude on the atlas ac- tivations. The 2,048 retained examples provide 4,096 vectors through the prompt and prompt– answer views. Calibration equates their meantanh(∥x∥ 2 /s d )to the mean PCA-derived amplitude tanh(∥N (h)x∥ 2 /64); both sample-average amplitudes are approximately 0.597014. PCA defines the amplitude reference during calibration; the distance-scaled residual subsequently uses centers and distances. Calibration uses retained atlas activations only, and the resulting scale is shared by all seeds. The two functions match in sample-average amplitude and retain their respective tokenwise responses. D.3SENSITIVITY TO INTEGER TARGET REQUIREMENTS AND SCORE COUNTS The comparisons with centered and distance-scaled residuals remain favorable when individual target requirements or checkpoint counts change (Table 13). The analysis starts from the 151–160 requirements and keeps the measured KL values and minimum-observed-KL selection rule fixed. First, we omit each requirement in turn from all three seeds, giving ten comparisons over nine requirements. Second, we change one checkpoint’s solved-problem count by−1or+1at a time and recompute eligibility over all ten requirements. For each control, the two methods, three seeds, five checkpoints, and two signs give 60 scenarios. In every scenario, we average selected KL within each seed and geometrically average the three ATLAS/control ratios, as in the main comparison. Table 13: ATLAS retained-KL reductions (%) under target-score sensitivity. Ranges span the stated scenarios per comparator; every scenario favors ATLAS in all three seeds. SettingScenariosCenteredDistance-scaled Measured counts and requirements14.642.31 Omit one target requirement104.44–4.792.06–2.46 One checkpoint count±1604.33–5.082.04–2.50 ATLAS counts−1; comparator counts +113.621.10 We also simultaneously decrease all 15 ATLAS checkpoint counts by one and increase all 15 counts of the compared control by one. Under this conservative count shift, the reductions are 3.62% and 1.10%, with all three seeds favorable for both comparisons. Table 13 summarizes deterministic sensitivity ranges over these threshold omissions and hypothetical count perturbations, holding the trained checkpoints and their KL values fixed. D.4LOCAL OPERATORS For a token stateh, letx = h− c(h),z = N (h)x, andρ(h) = tanh(∥z∥ 2 /s). Table 14 specifies the residual constructions. Centering and distance scaling use the same routed centers as ATLAS. The remaining geometric variants also use its PCA bases. PCA-derived scaling computeszto obtainρ(h) while its trainable branch reads x; input filtering additionally inserts N (h) before A. EachU j has orthonormal columns, and the routing weightsα j (h)are nonnegative and sum to one. Consequently,0⪯ P j α j (h)U j U ⊤ j ⪯ I , andN (h)has eigenvalues in[0, 1]. Soft routing produces 25 Table 14: Residual equations. PCA-based scaled variants shareρ(h); distance scaling uses tanh(∥x∥ 2 /s d ). The matrices A,B are trainable. VariantEmitted residual ∆(h) PlainB tanh(Ah) Centered residualB tanh(Ax) Distance-scaled residualtanh(∥x∥ 2 /s d )B tanh(Ax) PCA-derived scalingρ(h)B tanh(Ax) Input filterρ(h)B tanh(AN (h)x) Output filterρ(h)N (h)B tanh(Ax) Both filters, without scaling N (h)B tanh(AN (h)x) ATLASρ(h)N (h)B tanh(AN (h)x) a positive-semidefinite contraction. The implementation appliesU j (U ⊤ j v)in token chunks, without materializing dense d× d matrices. Local charts use K-means centers and rank-16 local PCA. The standard atlas has 32 charts and routes each token to its four nearest centers with temperature 64. Adaptation uses cluster-member PCA, as specified in Appendix A.4. Global-PCA uses a single retained covariance estimate. Validation-selected input filtering. The input-filter comparison selects ATLAS and the output- filter variant independently by minimum target-validation NLL. Displacement is the squared emitted residual normalized by hidden dimension. Adding input filtering gives 7.90% lower KL and 40.94% lower displacement at equal mean MBPP+ across the three seeds. Individual target-score differences are−1.339,+0.446, and+0.893percentage points; all three seeds have lower retained KL with ATLAS. D.5RANDOM GEOMETRY, ROUTING, AND EFFECTIVE NORM Random-local replaces fitted local directions with random directions while retaining the routing structure. Shuffled-patch disrupts the association between routed regions and their fitted bases. At matched target scores, learned charts reduce KL in 11/11 Random-local comparisons and 9/12 Shuffled-patch comparisons. Their geometric reductions are 5.53% and 4.13%. On GSM8K, the effective-norm control preserves a trained Plain residual direction and rescales its emitted residual at every token to the norm produced by the paired ATLAS checkpoint. Table 15 specifies both checkpoints in each pair across the four Qwen backbones. For Qwen3-14B seed 7302026, ATLAS is evaluated at epoch five and Plain at epoch four. Across all 12 pairs, the geometric KL reduction is 3.89%, with 11 lower-KL comparisons. On Qwen3.5, the correctness-transition counts of ATLAS and rescaled Plain are 95/93, 79/87, and 69/71; the KL reductions are 6.15%, 3.18%, and 1.07%. EANSWER IDENTITY, CORRECTNESS, AND OUTPUT FORMAT E.1CHECKPOINT PAIRING FOR THE BEHAVIORAL ANALYSES Tables 16 and 17 identify all 38 checkpoint pairs used to measure answer churn and F/G/W transitions. SubscriptsAandBdenote ATLAS and its comparator;eis the training epoch andais the number of solved MBPP+ problems out of 224. For Qwen3.5, GLM, and CSQA, selection searches the two five-epoch trajectories within each seed. It first minimizes the absolute MBPP+ score difference, then the absolute target-validation-NLL difference, then the ATLAS epoch and comparator epoch. Qwen3.5 and GLM use checkpoint identifier order as the final tie-breaker. The matching tolerance is one percentage point. All nine Qwen3.5 pairs and all twelve CSQA pairs have identical target counts. Five GLM pairs are exact ties; ATLAS solves one additional problem in the remaining four. 26 Table 15: Twelve effective-norm comparisons on GSM8K. The Plain residual is rescaled token by token to match the effective residual norm of its paired ATLAS checkpoint. Epochs identify ATLAS/Plain. KL columns use units of 10 −3 ; positive reduction favors ATLAS. BackboneSeedEpochsATLAS KLRescaled Plain KLReduction (%) Qwen3.5-9B73020264/20.2917230.3108476.152 Qwen3.5-9B73020273/50.2702980.2791813.182 Qwen3.5-9B73020282/20.2632620.2661171.073 Qwen3-14B73020265/40.7760700.89009912.811 Qwen3-14B73020272/10.6953270.7281864.512 Qwen3-14B73020282/40.7034770.7385524.749 Qwen3-4B73020261/20.6454370.6506750.805 Qwen3-4B73020272/30.6555050.655319-0.028 Qwen3-4B73020284/20.6806830.6902001.379 Qwen3-8B73020263/31.1869991.2283913.370 Qwen3-8B73020274/31.2372641.3097905.537 Qwen3-8B73020283/11.1825951.2114642.383 Table 16: GSM8K behavior pairings. Epochs and MBPP+ counts identify both checkpoints in each comparison. The GLM rows cover the Plain, Replay, and Output-KL comparisons summarized in Table 19. ComparatorSeede A e B a A a B a A − a B Qwen3.5-9B, retained GSM8K Plain7302026421471470 Replay7302026211461460 Output-KL7302026311471470 Plain7302027331451450 Replay7302027211441440 Output-KL7302027341451450 Plain7302028211441440 Replay7302028531451450 Output-KL7302028311471470 GLM-4-9B, retained GSM8K Plain7302026411561560 Replay7302026411561551 Output-KL7302026411561560 Plain7302027341481480 Replay7302027341481480 Output-KL7302027341481471 Plain7302028341481480 Replay7302028341481471 Output-KL7302028411571561 27 The LoRA-Null/OPLoRA comparisons hold one ATLAS reference fixed per seed, each solving 160 of 224 MBPP+ problems. The comparator epoch is selected by the smallest absolute MBPP+ gap, then the smallest validation-NLL gap, and finally the earliest epoch. All eight pairs satisfy the one-percentage-point matching window. Three have identical solved-problem counts; in the other five ATLAS solves one or two fewer problems. Both CSQA answer modalities use the same twelve checkpoint pairs. Table 17: CSQA and LoRA-Null/OPLoRA behavior pairings. The CSQA choices and short generated labels come from the same checkpoints. LoRA-Null/OPLoRA comparisons use a fixed ATLAS reference per seed and a one-percentage-point matching window. ComparatorSeede A e B a A a B a A − a B Qwen3-8B, retained CSQA Plain7302026251601600 Global-PCA7302026441611610 Replay7302026231601600 Output-KL7302026541611610 Plain7302027221591590 Global-PCA7302027541591590 Replay7302027341621620 Output-KL7302027421581580 Plain7302028321611610 Global-PCA7302028331611610 Replay7302028111451450 Output-KL7302028431591590 Qwen3-8B, LoRA-Null/OPLoRA comparisons LoRA-Null7302026351601600 OPLoRA-16730202635160161-1 LoRA-Null730202745160161-1 OPLoRA-16730202742160162-2 OPLoRA-128730202745160161-1 LoRA-Null7302028321601600 OPLoRA-167302028331601600 OPLoRA-128730202832160162-2 E.2DEFINITIONS AND AGGREGATION Let ˆy 0 and ˆy m denote parsed frozen and adapted answers, with reference y. We count F = X i 1[ˆy 0i = y i , ˆy mi ̸= y i ],(9) G = X i 1[ˆy 0i ̸= y i , ˆy mi = y i ],(10) W = X i 1[ˆy 0i ̸= y i , ˆy mi ̸= y i , ˆy 0i ̸= ˆy mi ].(11) Fornevaluated questions,C = F + G + Wcounts changed normalized answers andC/nis the churn rate. Correctness-transition volume isF + G, and accuracy changes by(G− F )/n. The normalization treats numerically equivalent GSM8K answer strings as the same answer. Percentage reductions use ratios of equally weighted mean counts across matched pairs on the same 1,319 GSM8K questions. The three training seeds are the training replicates. Tables 18 and 19 report the transition counts for Qwen3.5 and GLM; Appendix E.1 identifies every checkpoint pair. 28 E.3QWEN3.5-9B The frozen model is correct on 896 questions and wrong on 423. Table 18 gives all nine comparisons; conditional transition rates areF/896andG/423. Equal-weight aggregation reducesFby 59.86% andGby 59.57%. Overall churn falls from 2,663 to 1,209 events across the repeated comparison sets, a 54.60% reduction. Appendix F.3 reports decoding sensitivity and accuracy comparisons under both answer-scoring conventions. Table 18: Qwen3.5-9B correctness transitions in the nine target-matched comparisons. Counts list ATLAS/comparator. Frozen-correct and frozen-wrong opportunities are 896 and 423 per comparison. Reductions are percentages. Comparator Seed F counts G counts F reduction G reduction Plain730202637/9159/14359.358.7 Replay730202634/7141/11152.163.1 Output-KL 730202630/7346/11458.959.6 Plain730202731/11350/14872.666.2 Replay730202730/6537/10153.863.4 Output-KL 730202731/9650/15367.767.3 Plain730202833/5936/9044.160.0 Replay730202836/11775/15769.252.2 Output-KL 730202835/5554/9136.440.7 E.4GLM-4-9B Table 19 pools three target-matched seed pairs per comparator, giving 3,957 question–pair obser- vations per comparison. Accuracy differences follow from the corresponding(G− F )/nvalues. Appendix E.1 gives every paired checkpoint’s coding score. Table 19: GLM-4-9B answer changes relative to the frozen model. Count cells list ATLAS/comparator, pooled across three target-matched seed pairs. Accuracy differences are ATLAS minus comparator in percentage points. ComparatorChangesFGW Churn reduction (%) Accuracy difference (p) Plain515/1400 183/651 121/295 211/45463.21+7.43 Replay515/1324 183/313 121/632 211/37961.10−9.63 Output-KL531/759 193/310 125/190 213/25930.04+1.31 E.5EXTERNAL SUBSPACE METHODS Table 20 reports all transition categories for the eight Qwen3-8B LoRA-Null/OPLoRA pairs. Changes between incorrect answers decrease by 25.41% under the answer-extraction parser and 21.95% under numeric normalization; the table uses the latter convention throughout. To assess variation across the evaluated questions, we draw 20,000 paired bootstrap samples of 1,319 questions. Each sample uses the same question indices across all eight checkpoint pairs and all three seeds, preserving their shared questions and ATLAS references. Percentage reductions retain equal weight for each checkpoint pair. The 2.5th and 97.5th percentiles give 95% intervals for question-sampling uncertainty at the fixed checkpoints and training seeds. The reduction in changes between incorrect answers has a 95% interval of 3.03% to 40.68%. Its point estimates are positive in all three training seeds, at 9.09%, 31.25%, and 21.43%. Table 20 presents the pooled intervals alongside the complete transition counts for each seed. 29 Table 20: Eight target-matched LoRA-Null/OPLoRA comparisons on Qwen3-8B. Panel (a) pools all pairs and reports 95% paired-question bootstrap intervals for percentage reductions. Panel (b) shows each training seed; count cells list ATLAS/comparator. Positive reductions indicate fewer ATLAS changes. (a) Pooled changes and question-level uncertainty QuantityATLAS Comparator Reduction (%) 95% interval (%) Total changes (C)3443748.02[−7.44, 21.76] Correct-to-wrong (F )144137−5.11[−38.10, 20.51] Wrong-to-correct (G)1041148.77[−24.27, 37.08] Correctness transitions (F + G)2482511.20[−21.09, 20.40] Changes within errors (W )9612321.95[3.03, 40.68] (b) Changes within each training seed SeedPairsC F G W C reduction (%) W reduction (%) 7302026298/113 42/39 26/41 30/3313.279.09 73020273 117/147 48/56 36/43 33/4820.4131.25 73020283 129/114 54/42 42/30 33/42 −13.1621.43 E.6CSQA LABELS AND FORMAT TRANSITIONS The CSQA analysis uses 12 target-matched pairs, including Global-PCA, and all 1,221 questions. Conditional choice changes 125 labels for ATLAS and 203 for the comparators across pooled pairs, a 38.42% reduction. Table 21 reports absolute accuracies for conditional choice and short generation. Table 21: Conditional choices and generated answer format on CSQA. All entries are percentages. Adapted values average the 12 matched comparisons on 1,221 questions. Conditional choice selects the highest-likelihood option. Short generation requests an option letter and counts outputs outside A–E as invalid. Churn is measured relative to the starting model. The conditional-choice results are also shown in Figure 4b,c. Conditional choiceShort generation Model stateAccuracyChurnAccuracyInvalid Frozen85.500.0065.1124.49 ATLAS85.220.8568.1621.12 Matched comparators85.201.3974.6612.99 Table 22 separates acquisition and loss of valid answer formatting. Rates average the matched comparison sets and use all evaluated questions as the denominator. Table 22: CSQA short-generation format changes, as percentages of all questions. The arrows compare each adapted model with the same frozen reference. RoleInvalid→validValid→invalidParsed-label churnRaw-text churn ATLAS4.020.665.235.38 Matched comparators11.700.2112.5712.80 30 FSELECTION AND EVALUATION SENSITIVITY F.1SHARED TARGET RANGES The primary calculation uses each method pair’s achieved target-score intersection. Requiring one interval shared across all primary methods and all seeds within each setting yields 53 favorable seed comparisons out of 54 and a geometric mean KL reduction of 60.92%. All 18 three-seed model–comparator summaries remain favorable. The seven-method comparison already uses a single common interval and keeps all 24 seed comparisons favorable. F.2EQUAL CANDIDATE COUNTS The tuning sensitivity includes nine ATLAS configurations from three learning rates and three scalar scales, and three configurations each for CLoRA, STM, TALR, Output-KL, and Replay. Each configuration contributes its minimum-validation-NLL checkpoint on seed 7302026. The fixed three-candidate ATLAS subset holds scalar scale at 64 and varies learning rate. Table 23 applies the integer-target rule to these validation-selected configurations; the main comparison evaluates complete five-epoch trajectories at the selected configuration. Table 23: Equal-candidate tuning sensitivity. Both ATLAS and each comparator contribute three validation-selected configurations. Reductions are percentages. The integer range is inclusive; TALR shares a single observed requirement with the fixed three-candidate subset. ComparatorInteger rangeKL reduction (%) CLoRA148–16324.43 Output-KL160–16338.01 Replay160–16382.09 STM159–16322.01 TALR16320.37 Enumerating all 9 3 = 84three-configuration ATLAS subsets across the five comparators gives 361 nonempty intersections. ATLAS has lower mean KL in 355, including all 314 spanning at least two integer requirements and 41 of the 47 spanning one. The three-seed fixed-configuration comparison in Table 10 supplies the primary published-method result. Retained-training strength.Table 24 reports the complete three-coefficient sweeps for Replay and Output-KL on Qwen3-8B with seed 7302026. Each configuration contributes its checkpoint selected by minimum target-validation NLL; all six selected checkpoints occur at epoch five. Validation NLL favors coefficient 0.3 for Output-KL and 0.1 for Replay. The Output-KL configuration at coefficient 1.0 attains a higher MBPP+ count and lower retained KL than coefficient 0.3. All three coefficients for each method enter the equal-candidate comparison in Table 23. Table 24: Validation-selected checkpoints from the Replay and Output-KL coefficient sweeps on Qwen3-8B, using seed 7302026. MBPP+ counts solved problems out of 224; retained KL is evaluated after checkpoint selection. MethodCoefficientValidation NLLMBPP+ countKL (×10 −3 ) Output-KL0.10.672751606.1000 Output-KL0.30.671851604.5893 Output-KL1.00.674871631.9085 Replay0.10.671801636.6063 Replay0.30.676001608.0026 Replay1.00.6921116323.9151 31 Single-problem score resolution. To assess score resolution, we perturb target counts by one MBPP+ problem in the continuous-target comparison. All 24 external seed directions remain favorable; the smallest retained reduction is 0.333%. The integer analysis expresses the same test resolution directly through counts out of 224. Appendix D examines single-problem sensitivity for the residual-structure comparisons. F.3DECODING BATCH AND QUESTION ORDER To assess decoding sensitivity, we evaluate the same Qwen3.5 starting model, ATLAS checkpoint, and Plain checkpoint under three conditions: the evaluation configuration, batch size eight, and a fixed shuffled question order at batch size 24. Generation remains greedy, and answer comparisons are paired by question identity. Table 25: Qwen3.5 correctness transitions under decoding perturbations. Counts list ATLAS/Plain, using ATLAS epoch four and Plain epoch two on seed 7302026 in every condition. Decoding conditionFGReduction in F + G (%) Evaluation batch and order37/9159/14358.97 Batch size eight34/8672/15355.65 Shuffled order, batch size 2431/9564/15461.85 The reduced transition volume appears under all three conditions. Across the nine primary Qwen3.5 comparisons, accuracy differences agree in direction under the GSM8K answer-extraction rule and numeric normalization: pooled ATLAS-minus-comparator differences are−1.398and−1.828 percentage points, respectively. GTARGET-LEARNING ENGAGEMENT ACROSS MODELS AND TASKS Shared target requirements depend on the learning achieved under the adaptation protocol. Additional code-learning assessments on Mistral-7B and Llama3-8B each cover five methods, three seeds, and five epochs. Across all 75 checkpoints per backbone, mean MBPP+ changes from the starting model are−1.60and−0.70percentage points, respectively. Gemma-2-9B has no sustained target-learning interval. For CSQA as the target, Qwen3-8B evaluates Plain and ATLAS with one seed and five epochs on 965 questions, retaining GSM8K. From 85.803% starting accuracy, their best gains are 0.000 and 0.104 percentage points. The shared range is 0.207 points; the protocol requires at least a one-point gain and a shared range of 0.5 points for multi-seed evaluation. 32 HCOMPUTATIONAL RESOURCES On Qwen3-8B, ATLAS decoding takes 2.52–3.13% longer than the frozen model; prefill takes 5.83– 12.26% longer. Table 26 compares the frozen model with epoch-five Plain, Global-PCA, ATLAS, and Replay checkpoints at batch sizes 1, 8, and 24. The complete local operator takes 10.37–11.14 microseconds per token in the isolated 128-token-per-example measurement, including chart selection and residual filtering. Equal-support PCA fitting on 4,096 activation vectors from 2,048 retained examples takes 38.54 seconds, excluding activation extraction. The resulting compressed atlas file occupies 15.85 MiB. Appendix A.4 specifies the neighborhood construction used for fitting and inference. Table 26: Qwen3-8B resource measurements on one RTX 5090. Rates count input tokens for prefill and generated tokens for decoding. Peak memory is CUDA-allocated memory in GiB. Each timed result is the median of three repetitions on the same prompt batch; decoding emits 32 tokens per example. MethodBatchPrefill (tokens/s)Decode (tokens/s)Peak memory (GiB) Frozen13224.951.215.324 Frozen85049.2327.515.987 Frozen246242.1943.417.424 Plain13215.951.515.324 Plain85046.3327.815.987 Plain246230.6942.617.426 Global-PCA13059.349.915.340 Global-PCA84955.5320.515.987 Global-PCA246064.1922.117.426 ATLAS13047.249.615.507 ATLAS84602.7319.515.997 ATLAS245560.3918.217.429 Replay13221.251.415.324 Replay85036.6328.515.987 Replay246202.4950.017.426 33