Paper deep dive
UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
Ye Chen, Weining Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/24/2026, 5:35:44 AM
Summary
The paper introduces UpgradeBench, a longitudinal benchmark evaluating migration decisions for fine-tuned LLM specialists across real model release sequences (Qwen and OLMo). It analyzes four strategies: freeze, port, refresh, and retrain. Key findings include that adapter portability depends on continued-pretraining distance rather than architecture, with retention decaying as token distance increases. The study quantifies 'upgrade tax' and provides a decision policy that achieves near-optimal quality with significantly lower compute and label costs compared to full retraining.
Entities (12)
Relation Signals (11)
UpgradeBench → evaluates → Qwen
confidence 95% · The benchmark disentangles three core questions... covering four consecutive Qwen releases...
UpgradeBench → evaluates → Olmo
confidence 95% · augmented by OLMo checkpoints with known training lineage.
UpgradeBench → includesstrategy → Freeze
confidence 95% · The benchmark separates three questions... freeze—keep serving the old specialist
UpgradeBench → includesstrategy → Port
confidence 95% · The benchmark separates three questions... port—transfer the adapter or task vector to the new base
UpgradeBench → includesstrategy → Refresh
confidence 95% · The benchmark separates three questions... refresh—use the old specialist as a teacher
UpgradeBench → includesstrategy → Retrain
confidence 95% · The benchmark separates three questions... retrain—pay the full data and compute cost again.
UpgradeBench → evaluatestask → Banking77
confidence 90% · Table 2: Task suite... Banking77 D (77-way intent)
UpgradeBench → evaluatestask → Spider
confidence 90% · Table 2: Task suite... Spider S (text-to-SQL)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.
Tags
Links
- Source: https://arxiv.org/abs/2608.20918v1
- Canonical: https://arxiv.org/abs/2608.20918v1
Trouble viewing inline? Open PDF directly →
Full Text
98,950 characters extracted from source content.
Expand or collapse full text
UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists Ye Chen Weining Zhang∗ Alibaba Group Cheung Kong Graduate School of Business Abstract Organizations increasingly maintain task-specific adapters on open-weight language models, and every base-model release re-opens the same migration decision: keep the existing specialist, port its adapter, refresh it from retained behavior, or retrain. Existing transfer studies evaluate isolated model pairs; none measures this decision over real release sequences. We introduce UpgradeBench, a decision-centric longitudinal benchmark over four consecutive Qwen releases, one continuation release, six application tasks, and two model scales, complemented by OLMo checkpoints whose training relationships are documented rather than inferred. The benchmark separates three questions: whether the target checkpoint raises a fixed-recipe retrained-specialist score, whether the specialization asset is portable, and which recovery resources remain available. We find that the value of upgrading varies by task–scale–release episode: some retrained references improve while others stay flat within training noise, and calendar-time durability spans from an H50 under one release interval (text-to-SQL) to beyond fourteen months (intent classification). Raw adapter copying is governed neither by architecture nor by family: on documented OLMo edges its retention falls from 0.88–0.99 at a 46B-token continuation to zero at a 2.9T-token continuation, with no additional damage from annealing or model souping—portability decays with continued-pretraining distance. When retained inputs are available, teacher relabeling recovers target-base specialists with no new gold annotation, though not always less compute. Replaying a pre-specified decision policy over 33 measured upgrade episodes—with decision gates and scoring on disjoint halves of each test set—attains a mean quality regret of 0.37p with zero behavioral regressions at one third of the always-retrain compute and label budget, and a cheap representational probe (CKA over 256 prompts) ranks portability across base pairs (Spearman 0.74 over eight pairs). We release per-example predictions, cost logs, split manifests, and the evaluation harness. 11footnotetext: Corresponding author. 1 Introduction Two facts about deploying language models in organizations are now well established. First, specialization works: on narrow, recurrent tasks—intent triage, text-to-SQL, function calling—parameter-efficiently fine-tuned small models match or beat prompted frontier models at one to two orders of magnitude lower inference cost (46; 3; 1). A single 24 GB GPU can host a base model plus hundreds of task adapters (35), making a “fleet of specialists” an economically attractive alternative to routing all traffic to a frontier API. Second, base models improve fast: the capability density of open-weight models doubles on the order of months (40); each generation of the Qwen, Llama, or Gemma families reaches the previous generation’s quality with a fraction of the parameters (32; 13). These two facts collide, because a fine-tuned adapter is bound, both numerically and behaviorally, to the base checkpoint it was trained on. When a better base ships, the organization faces a decision it currently cannot price: (i) freeze—keep serving the old specialist; (i) port—transfer the adapter or task vector to the new base; (i) refresh—use the old specialist as a teacher to cheaply re-specialize the new base; or (iv) retrain—pay the full data and compute cost again. We call the quality-and-cost gap between the chosen strategy and the best attainable one the base-model upgrade tax. The research community has recognized the problem but not measured it. A rapidly growing family of methods transfers fine-tuning across bases: synthetic-data distillation (38; 19; 16), upgrade-specific parameter mappings (23; 21), subspace projection (12; 39), diff-vector porting (24; 17), and training-time immunization (15). Each is evaluated on its own choice of model pairs and tasks; comparisons across papers are impossible. A recurring assumption behind these methods is that adapter weights copied unchanged between sufficiently similar bases form a strong trivial baseline; that assumption is what our shape-compatible hops test directly. Meanwhile the most decision-relevant quantity has never appeared in print: how many generations does it take for a new base’s cheap adoption to overtake yesterday’s specialist? Position papers argue both sides from first principles (1); neither side has controlled evidence. This paper supplies the missing measurement. UpgradeBench tracks a complete, real release series of four Qwen generations spanning fourteen months, rather than ad-hoc model pairs, and evaluates the full freeze/port/refresh/retrain decision space on six tasks from three classes, at two model scales, with training-side costs (GPU-minutes, energy) as first-class outputs. Three common intuitions do not survive the measurement. The first is that fine-tunes obsolesce quickly: across nine upgrade hops, freezing retains 99–101% of the attainable gain on intent classification, and over four generations the retrained reference score moves 0.2 points while zero-shot rises twenty-one (Figure 1a). But specialization moats are not uniformly safe either: on text-to-SQL the frozen specialist forfeits up to 59% of attainable gain on a single hop, and its advantage over a newer base decays to zero within three generations (Figure 1b). Nor is naive weight porting governed by the rule practitioners assume: between two architecturally identical checkpoints from independent pretraining runs, pasting the adapter onto the newer base collapses intent classification from 92.8% to 42.9%, far below the 60.7% obtainable with no adapter at all; yet onto a continued-pretraining descendant of the same weights, the same operation retains reference-level accuracy. What licenses copying is continuity of the underlying weights, not the shape of the network (§7.3). A prospectively specified validation on OLMo, whose intermediate checkpoints make the training relationship a manipulable variable rather than an inference from release notes, reproduces both regimes in a second family and then locates the boundary between them: moving the target along the same documented trajectory, retention falls from 0.88–0.99 at a 46B-token continuation to the no-adapter floor at a 2.9T-token continuation, with the subsequent anneal and model-soup merge adding no further damage. Portability is not a family property but a distance budget. The organizing variable behind the first two results is whether task competence is base-bound (SQL reasoning) or data-bound (label taxonomies, call formats). This cuts across the discriminative/generative surface distinction that transfer evaluations usually adopt. Contributions. • C1 (Benchmark). A longitudinal, cost-accounted benchmark of fine-tuning transfer along a complete open-weight release lineage, at two model scales, with unified tasks, metrics, per-query prediction records, bootstrap CIs, and energy logs, all released. The entire study (98 trainings, 193 evaluation cells) runs on one RTX 4090. • C2 (Quantities). Formal definitions and first measurements of retention, recovery cost, and specialization half-life, including per-hop retention across nine real hops, a half-life bracketed to two-to-three generations at 7–8B, and one crossing observed outright at 1.5–1.8B scale. • C3 (Strategy map). A measured freeze/port/refresh/retrain decision map, including a continuity criterion for porting: copying fails between independent pretraining runs (eleven task/scale cells, two families) and succeeds between checkpoints connected by continued pretraining (five cells, two families), with a prospectively specified OLMo distance ablation showing that portability decays with continued-pretraining distance (near-parity at 46B tokens, floor-level at 2.9T) while annealing and model souping add no further damage, while annotation-free (input-retaining) refresh reaches retraining parity everywhere it was tested. • C4 (Weights vs. harness). A first controlled comparison of weight-side and prompt-side specialization under the same upgrade (appendix pilot): the weight asset is roughly 80× larger in delivered advantage on the one task/hop measured, and no more fragile than the prompt asset, whose optimal choice flips across generations. 2 Related Work Model updates and backward compatibility. Replacing a deployed model regresses individual predictions even when aggregate accuracy improves; negative-flip rate quantifies this (43), and update-regression mitigation has been studied for structured NLP (5). This literature updates a model inside one training regime; we measure the same phenomenon where the base changes underneath a specialist, and fold negative flips into the upgrade decision itself (§7.9). The maintenance framing follows the technical-debt view of ML systems (34). Cross-base adapter transfer. Trans-LoRA (38) first framed base deprecation as a problem, proposing near-data-free transfer via synthetic-data distillation; TiTok (19) and TuneShift-KD (16) refine the distillation route. LoRASuite (23) and PortLLM (21) target version upgrades with parameter mappings and training-free patches; LoRA-X (12) and Cross-LoRA (39) project adapters through aligned subspaces; Trans-PEFT (15) immunizes adapters at training time. Each evaluates on self-selected pairs. Prior re-evaluation argues that naive weight copying is a strong baseline between similar bases; our measurement on architecturally identical consecutive releases contradicts that for data-bound tasks (§7.3). UpgradeBench proposes no new transfer method; it provides the missing common yardstick along a real release sequence, and documents that two of three hops admit no weight-space method at all. Task vectors, merging, and re-use. Task arithmetic treats fine-tuning deltas as portable vectors (18), with theory tied to shared initialization (29); Chat Vector (17) and fine-tuning transfer (24) demonstrate near-free porting within a lineage; merging combines specialists on a shared base (42; 44), with recent skepticism about cross-task recycling (25). All presuppose shape compatibility, which holds for exactly one of the three major-version hops in our lineage, and even there, transfer is harmful on every task we test. Our paired hops turn the shared-initialization assumption of the theory (29) into an experimental variable: transfer survives a continued-pretraining hop and dies across independent pretraining runs of identical architecture, exactly as the tangent-space account predicts. Capability growth of small open models. Capability density doubles on the order of months (40), making the upgrade decision recur quarterly; this trend supplies our half-life denominator. Our measurements add a caveat: the trend is an average, and per-task uplift can be negative (§7.5). Economics of specialist fleets. Fine-tuned small specialists beat prompted frontier models on narrow tasks (46; 3); multi-adapter serving makes fleets cheap to host (35); position work argues small models are the natural unit of agentic workloads (1). Cost-aware evaluation (7; 28; 11; 20) established dollar-denominated reporting; we adopt it and add training-side accounting. The upgrade tax is the missing depreciation term in these analyses. Prompt-side upgrade tax. Model updates silently break prompt-engineered applications (8; 27; 37). We measure both taxes in one controlled frame for the first time (§D). Benchmarks. Enterprise and agent benchmarks evaluate systems at a fixed time (41); cost-aware methodology exists (20; 33), but no benchmark versions its models along release time. UpgradeBench is, to our knowledge, the first whose primary axis is the release sequence itself. 3 Formalizing the Upgrade Tax Setup. A lineage is a sequence of base checkpoints ℬ=(b0,…,bG)B=(b_0,…,b_G) ordered by public release date. For a task t with training set DtD_t and metric St(⋅)∈[0,1]S_t(·)∈[0,1], a fixed PEFT procedure A(b,D)A(b,D) (QLoRA; §5) produces a specialist sb,ts_b,t. St(b)S_t(b) denotes the base’s cheap adoption score: zero-shot with full task instructions (plus k=5k=5 exemplars where the format benefits, reported separately). Specialization gain. G(b,t)=St(sb,t)−St(b)G(b,t)=S_t(s_b,t)-S_t(b). Retention. For a hop bi→bjb_i→ b_j and strategy m producing m(sbi,t,bj)m(s_b_i,t,b_j) without full-budget access to DtD_t’s labels: Ri→jm(t)=St(m(sbi,t,bj))−St(bj)St(sbj,t)−St(bj).R_i→ j^m(t)\;=\; S_t\! (m(s_b_i,t,\,b_j) )-S_t(b_j)S_t\! (s_b_j,t )-S_t(b_j). (1) R=1R=1 matches full retraining on the new base; R=0R=0 adds nothing over cheap adoption; R<0R<0 is worse than not transferring at all; R>1R>1 beats retraining (both observed in our data). The upgrade tax of m is Tm=(1−Rm)⋅(St(sbj,t)−St(bj))T^m=(1-R^m)·(S_t(s_b_j,t)-S_t(b_j)), paired with m’s measured cost. Freeze is the special case m(sbi,t,bj)=sbi,tm(s_b_i,t,b_j)=s_b_i,t. Two reporting rules follow from the algebra. First, TmT^m simplifies to the signed quality shortfall Lm=O−QmL^m=O-Q^m against the reference O; we report LmL^m with a paired bootstrap CI as the primary quantity, since it is what a deployment loses in points. Second, the normalized retention RmR^m is unstable when the reference gain O−FO-F is small, so we show RmR^m only where the lower confidence bound of O−FO-F exceeds 2p and mark the cell denominator-unstable otherwise. The full upgrade tax of an episode is a vector—shortfall, migration GPU-minutes and energy, gold labels, teacher queries, and validation load—and we keep it a vector: any scalarization needs organization-specific utility weights, so we report Pareto components and let the decision model (§4.4) combine them. Expert advantage and durability. The advantage of a generation-i specialist over a generation-(i+g)(i+g) base at calendar time τ(i+g)τ(i+g) (the target’s actual release date) is At(τ)=St(sbi,t)−St(bi+g)A_t(τ)=S_t(s_b_i,t)-S_t(b_i+g). We report two horizons: the half-value horizon H50=infτ:At(τ)≤12At(0)H_50= \τ:A_t(τ)≤ 12A_t(0)\ and the crossover horizon H0=infτ:At(τ)≤0H_0= \τ:A_t(τ)≤ 0\, both in months of real release time. Because releases are discrete, each horizon is interval-censored: we report the interval between the last release where the criterion fails and the first where it holds (or a lower bound when it never holds within the observed series). Earlier drafts called the crossover a “half-life”; the two horizons differ by an order of magnitude on some tasks, so we keep them separate. For non-monotone trajectories we report the first crossing and note persistence. Recovery cost. C95m(t)C^m_95(t) is the labeled-example (and GPU-minute) budget of tuning on bjb_j needed to reach 95% of the attainable reference gain over the adoption floor, i.e. the smallest budget N with QN≥F+0.95(O−F)Q_N≥ F+0.95\,(O-F) where F is the target’s floor and O its full-data reference, measured on a ladder 0,256,1024,4096,full\0,256,1024,4096,full\. (An absolute-score criterion QN≥0.95OQ_N≥ 0.95\,O gives different, systematically smaller budgets; we use the gain criterion because it is invariant to the floor.) Amortized decision model. Given upgrade cadence Δ , request volume v, per-strategy quality QmQ^m and cost cmc^m, the steady-state objective is maxmv⋅u(Qm)−cm/Δ _mv· u(Q^m)-c^m/ ; released as code with all measured (Qm,cm)(Q^m,c^m). 4 The UpgradeBench Benchmark 4.1 Lineage We instantiate the benchmark on the Qwen 7–8B release series (31)—the open-weight family most used for enterprise fine-tuning—covering every major generation released between February 2024 and April 2025 (Table 1). We fine-tune instruct checkpoints, the standard enterprise substrate, and use the same checkpoints for cheap-adoption baselines, keeping the substrate constant across conditions. Qwen3’s thinking mode is disabled in both training and evaluation templates. Table 1: The measured lineage, from the released checkpoints’ config.json. Shape compatibility governs whether weight-space transfer (adapter copying, diff-vector porting) is defined at all: it requires identical hidden width, depth, and attention geometry. Exactly one of the three hops qualifies. Checkpoint Released Hidden Layers Heads (Q/KV) FFN Vocab Hop to next Qwen1.5-7B-Chat 2024-02 4096 32 32 / 32 (MHA) 11008 151936 incompatible Qwen2-7B-Instruct 2024-06 3584 28 28 / 4 (GQA) 18944 152064 compatible Qwen2.5-7B-Instruct 2024-09 3584 28 28 / 4 (GQA) 18944 152064 incompatible Qwen3-8B 2025-04 4096 36 32 / 8 (GQA) 12288 151936 — Continuation release: Qwen2.5-7B-Instruct-1M (2025-01) — identical to Qwen2.5-7B-Instruct in every dimension above, and a continued-pretraining descendant of the same weights (long-context extension). Small-scale track: Qwen1.5-1.8B → Qwen2-1.5B → Qwen2.5-1.5B → Qwen3-1.7B; the 2→ 2.5 hop is likewise shape-identical (hidden 1536, 28 layers, GQA 12/2, FFN 8960). Two properties of this lineage matter methodologically. First, it is tokenizer-stable in kind but not in size: all generations use the same BPE family, but the vocabulary alternates between 151,936 and 152,064 entries, so even embedding-level surgery is not uniform across hops. Second, and more consequentially, architecture churn is the norm rather than the exception: hidden width, depth, and attention geometry all change at the 1.5→ 2 and 2.5→ 3 boundaries, leaving weight-space transfer methods (24; 18) undefined on two of the three hops an enterprise following this family would have faced. The 2→ 2.5 hop is the exception: those two checkpoints are architecturally identical in every dimension above, so an adapter trained on one can be loaded onto the other without any mapping. That hop is therefore the best case for naive porting between independent pretraining runs, and we measure it (§7.3). The lineage also offers the complementary case: Qwen2.5-7B-Instruct-1M, a January-2025 long-context release that is a continued-pretraining descendant of Qwen2.5-7B-Instruct with identical shapes. Copying onto it isolates weight-space continuity from architectural compatibility, the distinction that shared-initialization theory (18; 29) and within-family porting results (17; 24) implicitly rely on. A parallel 1.5–1.8B track replicates the whole design at small scale. Applicability is itself a benchmark outcome: transfer papers that evaluate only on shape-compatible pairs describe a regime that two thirds of this lineage’s real upgrades never entered. Cross-family validation lineage (OLMo). Qwen’s training history is not public, so “independent pretraining run” and “continued-pretraining descendant” are inferences from release documentation. To test the continuity criterion where lineage is ground truth, a fourth experimental wave uses OLMo (14; 36), whose checkpoints, data order, and training stages are fully documented. It contributes five edges, all between architecturally identical checkpoints: (i) OLMo-1-7B → OLMo-1.7-7B, whose model card states the latter was trained from scratch—a verified fresh-pretraining pair; and, holding the source fixed at the OLMo-2-7B stage-1 checkpoint step237000, four targets progressively farther along the same documented run (36): (i) step248000, ∼ 46B tokens later—a short pure continuation; (i) step928646, the end of stage 1, ∼ 2.9T tokens later with no annealing or merging—a long pure continuation; (iv) the final stage2-ingredient3 checkpoint, after the 50B stage-2 anneal on a different data mix but before souping; and (v) the released OLMo-2-7B base, a model-soup merge of independently annealed branches. Edges (i)–(v) form a distance-matched ablation that separates continuation distance from the release operations. These are base (non-instruct) checkpoints, so the OLMo wave fine-tunes and evaluates with a fixed plain-text prompt format on the two intent tasks, with per-checkpoint adoption floors and retraining references measured under the identical format (§7.3). All three predictions were pre-registered before any wave-4 run (§6). 4.2 Task suite Two tasks per class (Table 2); splits fixed at seed 42 and released. The original three tasks are measured across all four generations and both scales; the second task in each class (CLINC150, FinQA, glaive-FC) across the three most recent 7–8B generations. Table 2: Task suite. D = discriminative, S = structured generation, A = agentic tool use. Licenses as listed by official distributions. Task Class Train / Val / Test Metric Source Banking77 D (77-way intent) 9,003 / 1,000 / 3,080 accuracy 6 CLINC150 D (151-way intent, incl. OOS) 10,000 / 1,000 / 5,500 accuracy 22 Spider S (text-to-SQL) 6,500 / 500 / 1,034 (dev) execution acc. 45 FinQA S (numerical programs) 6,251 / 500 / 1,147 program acc. 9 xLAM-FC A (function calling) 10,000 / 500 / 2,000 (held-out) call-set match 26 glaive-FC A (calling + refusal) 7,580 / 505 / 2,000 (group re-split) call-set match glaive-v2 corpus Protocol notes. (i) Banking77 cheap adoption carries the 77-label inventory in the system prompt; zero-shot and 5-shot (fixed exemplars) are both reported. A manual audit found zero parsing failures across all baseline conditions. Every baseline error is a genuine near-class confusion, so adoption floors are not deflated by format artifacts. (i) Spider prompts serialize the schema (tables, columns, types, primary/foreign keys); execution accuracy compares result multisets on the official dev databases (single-database variant; known false-positive risk noted). (i) xLAM-FC uses a held-out split of the APIGen-verified corpus (26) with semantic call-set matching (name equality, type-coerced argument equality, multiset matching); it measures in-distribution function calling. (iv) A memorization probe over all four bases and three test sets (Appendix E) finds no evidence of verbatim test-set memorization on Banking77 or Spider (verbatim-continuation rates 0–1%); xLAM shows moderate n-gram overlap (0.32–0.46) attributable to its templated JSON structure, roughly constant across generations—so cross-generation comparisons are not differentially biased. Any residual contamination inflates baselines, making measured expert advantages conservative. (v) For the second-task suite: CLINC150 label matching is reported under a strict normalization and a lenient morphological one (e.g., mapping translation to the gold label translate when the match is unique); we use the lenient score as the adoption floor so that baselines are not penalized for surface form, and release both. FinQA is scored by exact match of the normalized calculation program. glaive-FC retains 25% no-call negatives, so specialists are also graded on refusing to call. 4.3 Upgrade strategies Table 3: Strategy conditions and measurement status. For LoRA specialists, pasting the adapter and adding its merged ΔW=BA W=BA to the new base are the same operation, so the Copy measurement covers diff-vector porting as well. ID Strategy Data need Description Measured Freeze keep old specialist — serve sbi,ts_b_i,t unchanged 9 hops Adopt-0 new base, zero-shot — prompt-only cheap adoption 4 gens Adopt-k new base, 5-shot — fixed exemplars 4 gens (D) Copy/PortDiff weight-space port none load sbi,ts_b_i,t’s LoRA on bjb_j (24) 2→ 2.5: 6 tasks (+3 at 1.8B); 2.5→ 1M: 3 tasks Refresh-D distillation refresh unlabeled inputs old specialist labels retained inputs; fresh LoRA on bjb_j (relabel variant of 38) 2.5→ 3, 3 tasks Retrain full QLoRA on bjb_j full labels reference denominator of Eq. 1 all generations, 6 tasks, 2 scales Retrain-N label-budget ladder N labels N∈256,1024,4096N∈\256,1024,4096\ Spider, Qwen3 PromptPort port best prompt dev slice old best-of-6 prompt on bjb_j Banking77 hop PromptRetune re-search prompt dev slice new best-of-6 search on bjb_j Banking77 hop 4.4 Cost accounting and protocol Every training run logs wall-clock, peak VRAM, and energy (board power sampled at 10 s intervals, integrated). Greedy decoding throughout; per-query outputs for all 125 evaluation cells are released with bootstrap 95% CIs (10,000 resamples). Because every pair of conditions is evaluated on the identical test set, all comparisons between conditions are paired; we therefore test them with the exact McNemar test on discordant pairs rather than by inspecting overlapping marginal CIs, and report p-values with the point estimates. We call a difference measured only when p<0.05p<0.05. A 3-seed probe on the Banking77/Qwen2.5 cell bounds run-to-run variance at σ≈0.12σ≈ 0.12p; a second seed on the Spider/Qwen3 reference reveals much larger training variance on the generative task (2.5p), which we fold into how the single-hop Spider numbers are read (§7.12). Training drops examples whose completion is fully truncated at the context limit; the counts vary with each model’s tokenizer (Spider 138/6,500 for most generations but 154 for Qwen3; xLAM 160 vs. 161; FinQA 6–8; glaive-FC and Banking77 0) and are reported per run in the released logs rather than folded silently into the totals. 5 Experimental Setup One recipe for all 55 trainings: QLoRA (10) with NF4 double quantization, bf16 compute; LoRA r=16r=16, α=32α=32, dropout 0.05 on all attention/MLP projections; lr 2×10−42×10^-4 cosine, warmup 3%, completion-only loss, seed 42 (43/44 for variance probes). Intent tasks: 3 epochs, length 256, batch 16. Spider/FinQA: 3 epochs, length 1024/1280, micro-batch 4 × accumulation 4. Function calling: 2 epochs, length 1024, same batching (small-scale track: micro-batch 8 × 2). Optimizer paged_adamw_8bit; gradient checkpointing on. Stack and hardware. Single RTX 4090 (24 GB); PyTorch 2.13.0+cu126, transformers 5.14.1, peft 0.19.1, bitsandbytes 0.50.0. Peak VRAM 6.5–21.1 GB across runs. Study inventory. 98 QLoRA trainings (12 lineage specialists at 7–8B, 9 second-task specialists, 12 small-scale specialists, 3 continuation-release references, 4 Refresh-D students, 3 ladder budgets, 17 OLMo specialists/references along the documented trajectory and release series, 3 glaive re-split specialists, 4 rank-64 replicates, and 31 seed replicates), 4 relabeling passes, 193 evaluation cells, one memorization probe (4 bases × 3 test sets), one prompt-side pilot. Cost is reported as a three-way split, not a single number: 148.7 training GPU-hours with 47.8 kWh of measured board energy, 79.4 evaluation GPU-hours (evaluation energy was not separately metered), and 4.5 teacher-relabel GPU-hours (Appendix F); $0 API spend. Table 4 states the exact coverage of every action–edge pair; the matrix is sparse by design and the empty cells are part of the record. Table 4: Coverage matrix: which actions are measured on which edges. The design is deliberately sparse, not factorial; empty cells are stated, not implied. aLabel ladder run on the highest-coupling task only. bRefresh-D requires retained inputs; measured on the newest hop of the original three tasks. cLearned heterogeneous mappings and distillation-transfer baselines are extension points of the released protocol, not measured here. Panel Adopt/Freeze Copy Refresh-D Retrain-N Reference B77/Spider/xLAM, 7–8B 4 gens + 1M 2→ 2.5, 2.5→ 1M 2.5→ 3 (3 seeds) Spider onlya 4 gens + 1M (3 seeds) CLINC/FinQA/glaive, 7–8B 3 gens 2→ 2.5 —b — 3 gens B77/Spider/xLAM, 1.5–1.8B 4 gens s2→ 2.5 — — 4 gens B77/CLINC, OLMo 7 checkpoints 5 documented edges — — 7 checkpoints Shape-breaking hops (= Adopt/Freeze) undefinedc — — (= Reference) 6 Prospectively Specified Analyses Hypotheses were fixed before the wave that tests them; outcomes are reported against them in §7. A methodological caveat first: these analyses were fixed in the runner scripts before the corresponding waves ran, and the scripts are released verbatim with timestamps, but they were not deposited with an external immutable registry. We therefore call them prospectively specified rather than pre-registered, and we mark each hypothesis with the wave boundary it preceded. • RQ1 (Durability). How large is At(τ)A_t(τ), and where are H50H_50 and H0H_0? H1 (before wave 1): crossover within one generation for reasoning-heavy structured tasks, two or more for discriminative tasks. Outcome: directionally supported for the discriminative task (H0>14H_0>14 months, advantage 27.1p at the last observed release, p<10−200p<10^-200) but refuted in magnitude for text-to-SQL: the crossover took the full observed series (H0=14H_0=14 months, interval-censored at (7,14](7,14]), while the half-value horizon was under one release interval (H50∈(0,4]H_50∈(0,4] months—the advantage drops from 17.2p to 4.0p at the first hop). The two horizons differ by an order of magnitude, which the original single “half-life” number conflated. • RQ2 (Strategies). What retention at what cost? H2 (before wave 1): on shape-compatible hops Copy/PortDiff retain R≥0.8R≥ 0.8; across breaks only Refresh exceeds R=0.5R=0.5. Outcome: both clauses refuted as stated. Between independent pretraining runs Copy retained −0.60-0.60 to 0.780.78 across nine task/scale cells, below Freeze in all nine; between checkpoints joined by a short continuation it retained 0.82–1.45. Shape was the wrong variable, and (after wave 5) continuity alone is not sufficient either: portability decays with continued-pretraining distance (§7.3). Refresh-D reached 0.96–1.05, far above 0.5, at zero gold labels. • RQ3 (Task class). Does a discriminative-vs-generative retention gap replicate on real release series? H3 (before wave 1): retention ranks D >> S >> A. Outcome: refuted. Reference scores rise only where the coupling between task competence and base capability is high, and Copy’s ordering (A >> S >> D) inverts the prior report. We describe this coupling with the continuous elasticity βt=ΔOt/ΔFt _t= O_t/ F_t (reference gain per point of floor gain) rather than a binary base-bound/data-bound label, since the coupling itself shifts with model scale (§7.1, §7.6). • RQ4 (Prompt asset). Is prompt-side specialization more durable than weight-side? Outcome (pilot, appendix only): the prompt asset is roughly 80× smaller in delivered gain on the one task/hop measured, and its optimum flips across generations; the pilot lacks per-example records, so we report it as a bounded observation, not a contribution. • RQ5 (Predictability). Do cheap representational probes predict retention before any migration is paid for? Outcome (wave 6, exploratory): across the eight measured base pairs, linear CKA over 256 task prompts rank-correlates with mean R(Copy)R( Copy) at Spearman ρ=0.74ρ=0.74; next-token top-1 agreement and JSD are weaker (0.38, −0.52-0.52). No single probe is decision-grade at n=8n=8—CKA separates the OLMo distance spectrum cleanly but cannot distinguish the Qwen fresh pair from the Qwen continuation pair (§7.11). Wave-4 prospective predictions (fixed 2026-07-31, before any OLMo run). P1: copying OLMo-1 adapters onto OLMo-1.7 (verified fresh pretraining) collapses, R<0.5R<0.5 and below Freeze; P2: copying between OLMo-2 stage-1 checkpoints of the same run (∼46 46B tokens apart) retains R≈1R≈ 1, indistinguishable from the reference; P3: copying across the stage-2 anneal and soup merge is a stress test whose either outcome delimits the criterion. Outcomes (§7.3): P1 confirmed (R=−0.32R=-0.32 and −0.02-0.02, at or below the adoption floor). P2 directionally confirmed (R=0.88R=0.88 and 0.990.99, far above the floor) but exact reference parity is rejected on Banking77 (−11.3-11.3p, p<10−79p<10^-79): even 46B continuation tokens exact a small, measurable tax. P3’s collapse was confirmed, but its attribution was not resolvable from that wave alone. Wave-5 prospective predictions (fixed 2026-08-10, before any wave-5 run). Q1: if the Qwen copy collapse is not a chat-template artifact, swapping source/target templates changes nothing; Q2: constrained decoding shrinks but does not eliminate the CLINC generational decline; Q3: if the wave-4 P3 collapse is driven by annealing/souping, copying onto the end-of-stage-1 checkpoint (2.9T tokens of pure continuation, no anneal, no soup) stays high; if driven by distance, it collapses there already. Outcomes: Q1 confirmed exactly (template swap changes no score by even one item). Q2 confirmed (the decline shrinks by roughly a third under constrained decoding). Q3 resolved against the annealing/souping reading our wave-4 text had favored: copying is already at the floor after 2.9T tokens of pure continuation, and the anneal and soup edges add no further damage (§7.3). We report the correction rather than the earlier interpretation. Wave-6 prospective predictions (fixed 2026-08-16). W1: on a second family (the OLMo release series), the high-coupling task’s half-value horizon is again shorter than one release interval; W2: next-token top-1 agreement predicts R(Copy)R( Copy) better than CKA; W3: the fresh-hop copy collapse at rank 64 matches the rank-16 magnitude. Outcomes: W1 refuted, informatively: on OLMo the frozen Spider specialist keeps a 39–58p advantage across nine months of releases, because that series’ floors and reference scores barely move on SQL (§7.4); durability horizons are properties of the task–series pair, not of the task. W2 refuted—CKA is the stronger rank-predictor (ρ=0.74ρ=0.74 vs. 0.38). W3 refuted in magnitude: at r=64r=64 the Banking77 copy loses 21p instead of 50p (R: −0.55→+0.29-0.55→+0.29) and Spider reaches R=0.74R=0.74, though copying stays below both Freeze and the reference at both ranks (§7.3); the collapse magnitude is rank-dependent, its ordering is not. 7 Results Figure 1: Durability in real release time (x-axis: months since the gen-0 specialists were trained; all points measured, n = 3,080 / 1,034 / 2,000 test items). (a) Frozen gen-0 specialists (horizontal lines) against each task’s rising adoption floor. (b) The same data as advantage over the floor, with half-value (H50H_50, dotted) and crossover (H0H_0) horizons marked; horizons are interval-censored by the discrete release calendar. Generated from the released study registry. Table 5: Lineage scores (test sets; greedy decoding; 4-bit inference). “Spec.” = specialist fine-tuned on that generation (the retrain reference for hops ending there). Bootstrap 95% CIs for all cells are released; typical half-widths ± 0.9p (Banking77), ± 2.8p (Spider), ± 1.6p (xLAM). Qwen1.5-7B Qwen2-7B Qwen2.5-7B Qwen3-8B Task 0-shot Spec. 0-shot Spec. 0-shot Spec. 0-shot Spec. Banking77 .4354 .9302 .5747 .9276 .6071 .9312 .6438 .9321 (5-shot) .5679 .6273 .6357 .6594 Spider .5242 .6963 .6567 .6992 .6760 .7321 .6963 .7582 xLAM-FC .5770 .8205 .6315 .8225 .7940 .8480 .7700 .8455 7.1 Retention across nine upgrade hops Table 6 reports R(Freeze)R( Freeze) (Eq. 1) for every consecutive hop. On Banking77 the frozen specialist retains 99–101% of what retraining would deliver on every hop. The retrained reference score is flat at 92.8–93.2% across four generations (Figure 1a) while zero-shot climbed from 43.5% to 64.4%. On xLAM the pattern holds in two of three hops (0.99, 1.03; neither the shortfall nor the excess is significant), with one taxed hop (0.53) caused by Qwen2.5’s unusually strong zero-shot function calling. Spider is the systematic outlier: freezing retains only 41–93% (mean 0.64), because the retrain ceiling itself rises with every generation (69.6→ 75.8): new-base SQL competence penetrates the fine-tune. One caution applies to these single-hop values on the generative task: a second training seed for the Spider/Qwen3 reference lands at .7331 rather than .7582, which moves the final-hop R(Freeze)R( Freeze) anywhere from 0.58 to 0.97 and ties the freeze-vs-retrain comparison (p=1.0p=1.0 against the second seed). Single-hop retention on generative tasks is seed-limited at this test size; the base-bound classification of Spider rests instead on the lineage-level ceiling rise (+5.9+5.9p over two hops, p<10−7p<10^-7), which no seed choice undoes. Table 6: Retention by strategy. Freeze is measured on all nine hops; Copy on the one shape-compatible hop (2→ 2.5); Refresh-D on the most recent hop (2.5→ 3), with cost = teacher relabeling + student training on one RTX 4090. R<0R<0: worse than not transferring. R>1R>1: beats gold-label retraining. R(Freeze)R( Freeze) R(Copy)R( Copy) Refresh-D Task 1.5→ 2 2→ 2.5 2.5→ 3 mean 2→ 2.5 R cost (min) Banking77 1.007 0.989 0.997 0.998 –0.551 0.997 39 Spider 0.932 0.414 0.578 0.641 0.139 1.047 255 xLAM-FC 0.990 0.528 1.033 0.850 0.370 0.960 331 This pattern refutes our pre-registered H3 (D >> S >> A): measured Freeze retention ranks D ≈ A >> S. What separates the tasks is not surface form but competence provenance: where the skill is mostly data-bound, as with a label taxonomy or a call format, the fine-tune is the ceiling and base generations are interchangeable. Where it is base-bound, as in compositional SQL over unseen schemas, base progress compounds with fine-tuning, and freezing forfeits that progress. We propose competence provenance as the organizing variable for transfer evaluations, in place of the discriminative/generative split. 7.2 Reference-score drift and task–base coupling Provenance makes a testable prediction: adding a second task per class should sort by what the skill depends on, not by class. It does (Table 7). Across the three most recent 7–8B generations, with three training seeds on every endpoint where the claim depends on it, retrained reference scores rise on Spider (+4.8+4.8p between seed means, .7025 → .7501; every Qwen3 seed exceeds every Qwen2 seed) and, mildly, on xLAM-FC (+2.3+2.3p single seed; three seeds at each endpoint span ≤ 0.8p and confirm the direction); they are flat on Banking77, glaive-FC, and FinQA. On CLINC150, the seed-42 pair showed a significant decline (−0.9-0.9p, p=0.002p=0.002), but three seeds per endpoint overturn it: the seed ranges overlap (Qwen2 .9269–.9344, Qwen3 .9180–.9313) and the gap between seed means (−0.5-0.5p) is smaller than the within-generation spread (1.3p), so we report CLINC150 as flat within training noise, not declining—an instance of exactly the single-seed fragility our protocol is meant to catch. FinQA’s flat reference score despite being structured generation is consistent with its skill being dominated by a dataset-specific program DSL (its adoption floors are low and erratic across generations, .09–.22), i.e., data-bound in our terms even though the surface form is generative. Table 7: Fine-tuned ceilings across the three most recent 7–8B generations (specialist scores; reference for hops ending at each generation). Δ = Qwen3 minus Qwen2; p from the exact McNemar test on the paired test set. Task Class Qwen2 Qwen2.5 Qwen3 Δ (p) Evidence Spider3s S .7025 [.695–.713] .7321 [.730–.734] .7501 [.740–.758] +4.8+4.8 all seeds ordered xLAM-FC3s A .8225 .8473 [.846–.848] .8460 [.843–.850] +2.3+2.3 3×10−43×10^-4 FinQA S .5981 .5990 .6103 +1.2+1.2 0.220.22 Banking77 D .9276 .9312 .9321 +0.5+0.5 0.160.16 glaive-FC A .9640 .9650 .9645 +0.05+0.05 1.01.0 CLINC1503s D .9313 [.927–.934] .9322 .9262 [.918–.931] −0.5-0.5 within seed noise A binary base-bound/data-bound label invites circularity, so we summarize the coupling as a single elasticity per task: βt=ΔOt/ΔFt _t= O_t/ F_t, the reference-score change per point of zero-shot-floor change over the same hops (Qwen2→ 3, 7–8B). The six tasks form a continuum rather than two classes: Spider β=1.49β=1.49, xLAM 0.170.17, Banking77 0.070.07, glaive-FC 0.000.00, CLINC150 −0.07-0.07; FinQA’s floor moves less than 1p, so its β is denominator-unstable and we do not report one. We use “high-” and “low-coupling” as shorthand for the ends of this scale, and §7.6 shows the coupling itself moves with model scale, so β is a property of a task–scale–release window, not of a task. 7.3 Copying and the continuity criterion Figure 2: (a) Retention of naive adapter copying between architecturally identical checkpoints, in two families. Between independent pretraining runs (Qwen2→ 2.5, nine task/scale cells; OLMo-1→ 1.7, two cells) copying lands at or below the no-adapter floor; between checkpoints connected by continued pretraining (Qwen2.5→ 2.5-1M, three tasks; OLMo-2 stage-1 237k→ 248k, two cells) it retains 0.82–1.45 (Spider’s 1.45 reflects a small denominator: the 1M reference gains only 3.5p over its floor); and once the continuation runs long (237k→ , ∼ 2.9T tokens plus anneal and soup) it collapses below the floor—Table 8’s distance ablation shows the 2.9T-token distance alone suffices, with the release operations adding nothing further. (b) The pattern replicates at 1.5–1.8B: the Banking77 ceiling stays flat while zero-shot climbs 22.6p, the Spider ceiling rises with every generation, and the generation-0 Spider specialist is overtaken within the lineage. Two of three major-version hops in this lineage admit no weight-space transfer at all (Table 1). The 2→ 2.5 hop does: the two checkpoints are identical in hidden width, depth, attention geometry, FFN size, and vocabulary, so a LoRA adapter loads without any mapping. Prior work describes this as the setting where naive copying should work; our prospectively specified H2 predicted R≥0.8R≥ 0.8 here. In practice it is dominated by Freeze on every task we test, and on most tasks it lands at or below the no-adapter floor (Figure 2a). On Banking77 the copied adapter scores .4286 [.411, .446], a 50-point collapse from the source specialist’s .9276 and 17.9 points below the .6071 that the target base achieves with no adapter at all, giving R=−0.551R=-0.551. The other five tasks land between harm and marginal gain: CLINC150 R=0.123R=0.123, Spider R=0.139R=0.139, xLAM R=0.370R=0.370, glaive-FC R=0.69R=0.69 (on the re-split described in §4.2; copy .946 vs. floor .903 vs. reference .965, above the floor, p=2×10−15p=2×10^-15, yet below both the reference and Freeze, p≤6×10−5p≤6×10^-5), and FinQA R=0.776R=0.776 (inflated by that task’s anomalously low adoption floor of .0933). At 1.5–1.8B scale the same hop gives R=−0.600R=-0.600 (the Banking77 adapter collapses to 2.2% absolute), 0.0260.026, and 0.0950.095. Across nine task/scale cells, seven lie at or below R=0.37R=0.37; the two exceptions (glaive-FC 0.69, FinQA 0.78) owe their size to format-heavy tasks with shallow adapters and still trail their Freeze retentions. In every cell copying is worse than doing nothing to the old model (Freeze spans 0.41–1.04 on the same hop). The continuation hop reverses the verdict. Qwen2.5-7B-Instruct-1M is a continued-pretraining descendant of the same weights, and copying the same three adapters onto it retains R=0.985R=0.985 (Banking77), 1.4451.445 (Spider; the raw scores are copy .7292 vs. floor .6789 vs. reference .7137, so the ratio’s denominator is small), and 0.8150.815 (xLAM). On all three tasks no difference from full retraining is detected (p=0.15p=0.15, 0.160.16, 0.070.07) while the copy sits far above the adoption floor (p≤4×10−4p≤4×10^-4); but absence of detection is not parity, so we also test non-inferiority against pre-set margins (1p classification, 2p structured) using paired bootstrap CIs and three reference seeds. Banking77 is non-inferior at 2p but borderline at 1p (copy−-reference CI [−1.01,+0.10][-1.01,+0.10]p; three-seed reference band .9308–.9325). Spider’s copy (.7292) lies inside the three-seed reference range (.7137/.7340/.7408), so the comparison is bounded by training noise rather than by item noise. xLAM fails the 1p test outright (CI [−2.00,0.00][-2.00,0.00]). In short: copying onto a short continuation costs at most about a point; copying across a fresh run costs fifty. Architectural compatibility is the wrong test; a documented, short continued-training path is the right one, which also explains why within-family porting studies (17; 24) report success: their hops are continuations. Incidentally, the 1M release itself buys specialists nothing: frozen 2.5 specialists match or exceed 1M-retrained references on all three tasks (e.g., Spider .7321 vs. .7137, p=0.10p=0.10). Table 8: OLMo panel: copying the same intent adapters along a fully documented training trajectory. The top pair is a verified fresh pretraining run; the bottom four rows per task move the target along OLMo-2’s own trajectory—a 46B-token continuation, a 2.9T-token continuation (end of stage 1, before any annealing), the stage-2 anneal (before souping), and the released soup. Copy vs. floor and vs. reference tested with exact McNemar (↓ : Copy significantly below the floor; ≈: indistinguishable). CLINC150 uses the lenient metric (strict released per cell). Target edge Task Copy Floor Ref R p vs floor OLMo-1 → 1.7 (fresh run) B77 .0266 .2464 .9292 −0.32-0.32 <10−169<\!10^-169 (↓ ) CLINC .2364 .2482 .9151 −0.02-0.02 0.0630.063 (≈) 237k → 248k (+46B cont.) B77 .8179 .0097 .9305 +0.88+0.88 <10−300<\!10^-300 CLINC .8951 .0120 .9078 +0.99+0.99 <10−300<\!10^-300 237k → 928k (+2.9T cont.) B77 .1279 .1432 .9289 −0.02-0.02 0.0740.074 (≈) CLINC .1433 .0315 .9204 +0.13+0.13 <10−138<\!10^-138 237k → ing. 3 (+anneal) B77 .1276 .3055 .9315 −0.28-0.28 <10−78<\!10^-78 (↓ ) CLINC .1296 .2822 .9122 −0.24-0.24 <10−111<\!10^-111 (↓ ) 237k → main (+soup) B77 .1140 .3523 .9256 −0.42-0.42 <10−118<\!10^-118 (↓ ) CLINC .1313 .2575 .9125 −0.19-0.19 <10−78<\!10^-78 (↓ ) Ground-truth genealogy: portability decays with distance. The Qwen evidence rests on inferred lineage; OLMo makes the training relationship an experimental variable (§4.1), with predictions fixed before each wave (§6). Table 8 shows the ten cells. On the verified fresh pair (OLMo-1 → OLMo-1.7, identical shapes, independent runs), copying collapses exactly as on Qwen2→ 2.5: to 22.0 points below the target’s no-adapter floor on Banking77 (R=−0.32R=-0.32) and to statistical indistinguishability from the floor on CLINC150 (R=−0.02R=-0.02, p=0.063p=0.063), against references above .91. The four-edge spectrum then holds the source fixed (the stage-1 checkpoint at 237k steps, ∼1 1T tokens in) and moves the target along the run’s own documented trajectory. At a 46B-token continuation (∼1% 1\% of the run) the copied adapter retains R=0.88R=0.88 and 0.990.99—though exact parity with the reference is already rejected on Banking77 (−11.3-11.3p, p<10−79p<10^-79), so even short continuations exact a measurable tax. At the end of stage 1—2.9T further tokens of pure continuation, no annealing, no merging—the same adapter is statistically at the floor on Banking77 (R=−0.02R=-0.02) and retains a small residual signal on CLINC150 (R=+0.13R=+0.13). Crossing the stage-2 anneal and the final model-soup merge changes the copied score barely at all (Copy stays between .11 and .14 on both tasks across all three far edges); the more negative R values there reflect the rising zero-shot floors of later checkpoints, not additional damage from the release operations. Our wave-4 text, written when only the endpoints existed, attributed the collapse to the anneal-plus-soup boundary; the distance-matched ablation overturns that reading, and we report the correction (§6). Continuation distance is the operative variable: an adapter’s portability comes with a drift budget measured in continued-pretraining tokens—high retention at 1% of the run, zero at 74%—and annealing or souping at the end of that distance neither rescues nor further destroys it. This is also consistent with the Qwen continuation result: the 1M long-context release is a short continuation of Qwen2.5, and copying onto it retains 0.82–1.45. Figure 3 plots the spectrum. Figure 3: Copying one adapter (trained at OLMo-2 stage-1 step 237k) onto four later checkpoints of the same documented run. Retention falls to the no-adapter floor within the pure-continuation segment; the anneal and soup edges change the copied score by at most 1.4p (the lower R reflects their higher floors). Generated from the released records. The failure ordering on the fresh hop—A >> S >> D—inverts the Freeze ordering: the adapters that survive copying best are the ones that matter least, while the classification adapters delivering the largest frozen advantage are the ones copying destroys. Four checks rule out the obvious artifacts. First, the failure is not a decoding or parsing problem: of the copied model’s 3,080 outputs, 48.1% are near-miss label variants absent from the taxonomy (card_not_arrived where the gold label is card_arrival), with no empty or degenerate generations. Second, it is not a chat-template mismatch: a full 2×2 over source, target template × source, target base with the same adapter leaves every score unchanged to four decimal places (.4286 on the target base under either template; .9276 on the source base under either template)—the base–adapter pairing alone determines the outcome. Third, it is not a 4-bit artifact: re-running the copy in bf16 without quantization on the first 512 test items gives 52.5% against 52.0% quantized (per-example records released), a 0.6p difference against a 50-point collapse. Fourth, it is not purely an output-format regression: with decoding constrained to the 151 valid CLINC labels, the copied adapter recovers from .7505 to .8669 but remains 7.4p below the constrained retrained reference (.9413), so a real mapping loss persists once formatting is removed from the equation. Constrained decoding also shrinks the CLINC generational decline (§7.1) from −0.89-0.89p to −0.65-0.65p: roughly a third of that headline number is formatting, not competence. Rank dependence. All results above use the fixed r=16r=16 recipe. Re-running the fresh Qwen2→ 2.5 hop at r=64r=64 (adapter, reference, and copy all at the higher rank) weakens the collapse substantially: the Banking77 copy scores .7205 against a .9266 reference (R=+0.29R=+0.29, vs. −0.55-0.55 at r=16r=16), and the Spider copy reaches .7302 against a .7495 reference (R=0.74R=0.74, vs. 0.14). Higher rank also lifts the Spider reference itself (+4.7p over r=16r=16). The ordering that drives the decision—copy below Freeze and below retraining—holds at both ranks, but the collapse magnitude is a function of adapter rank, so the r=16r=16 numbers are one point on that axis, not the transfer profile of fine-tuning in general (full fine-tuning remains unmeasured; single seed, one hop, two tasks). A plausible mechanism is that a 77-way label taxonomy is encoded as sharp, low-entropy surgery on the output distribution, making it maximally sensitive to drift in the underlying residual stream, whereas format-and-style adaptations such as JSON call syntax or SQL conventions survive drift because they are shallower. We have not tested this directly, and layer-wise ablation of what each adapter changes is future work. The actionable result is the criterion: before porting an adapter, ask (i) whether the target is reachable from the source base by a documented continued-training trajectory at all—fresh initializations collapse immediately—and (i) how far along that trajectory it sits, because retention decays from near-parity at a 46B-token continuation to nothing by 2.9T tokens. The answer is in the release genealogy, not in config.json, and a copied adapter should in any case pass a task-level regression gate before serving. 7.4 Measured durability horizons We track the February-2024 Spider specialist against each later base’s cheap adoption, in real release time. Its advantage starts at +17.2+17.2p and is +4.0+4.0p four months later at g=1g=1 (McNemar p=0.005p=0.005)—77% of the initial advantage is gone within one release interval, so H50∈(0,4]H_50∈(0,4] months. The crossover takes far longer: +2.0+2.0p at seven months (p=0.19p=0.19, unresolvable) and exactly 0.00.0p at fourteen (g=3g=3: the two systems disagree on 234 of 1,034 items and split them 117 each, p=1.0p=1.0), giving H0=14H_0=14 months, censored to the interval (7,14](7,14]. The two horizons differ by an order of magnitude, which a single “half-life” number would conflate; and reading a point estimate off the g=3g=3 tie would overstate what a 1,034-item test set can resolve. The Banking77 specialist from the same date behaves differently: it retains +28.6+28.6p at fourteen months (p<10−200p<10^-200: 923 items it answers correctly and Qwen3 zero-shot does not, against 41 the other way), with mean decay 2.9p per generation, so H50H_50 and H0H_0 both lie beyond the observed series. The xLAM specialist crosses its half-value point between four and seven months (H50∈(4,7]H_50∈(4,7]) yet still retains +5.1+5.1p at fourteen (p<10−19p<10^-19), non-monotonically (§7.5). H1 (prospective) predicted crossover within one generation for reasoning-heavy structured tasks and two or more for discriminative ones. The discriminative clause holds; the structured clause is wrong about the crossover (H0H_0 took the whole series) but right in spirit about value decay (H50H_50 under one release interval). Even on the most base-coupled task in our suite, specialization out-earned cheap adoption for over a year—while losing most of its margin within four months. A second family. Repeating the Spider durability measurement on the OLMo release series (OLMo-1 → 1.7 → 2; February to November 2024; base checkpoints, plain format, floors and references under the identical format) gives the opposite trajectory: the frozen OLMo-1 specialist (.6673) holds a +48.4+48.4p advantage at release, +39.4+39.4p two months later, and +58.1+58.1p at nine months—no half-value crossing at all—because this series’ zero-shot SQL floors stay between .09 and .27 and its retrained references are flat (.667/.685/.668). The same task whose margin halved within four months on Qwen loses nothing in nine months on OLMo. This refutes our W1 prediction and sharpens the durability finding: erosion is driven by the release series’ floor growth, not by the task; H50H_50 and H0H_0 are properties of the task–series pair. (Within-family comparison only: OLMo floors are base-model floors and are not comparable to Qwen’s instruct floors in absolute terms.) 7.5 Non-monotonic generational uplift Zero-shot uplift per generation is task-dependent in sign: xLAM-FC drops 2.4p from Qwen2.5 to Qwen3 (.7940 → .7700), so the 2024 specialist’s advantage rebounds from +2.6p at g=2g=2 to +5.1p at g=3g=3 (Figure 1b). Averages like the densing law (40) govern the trend, not any single task: “wait for the next base” is not directionally safe without task-level measurement, which is precisely what the benchmark’s released harness automates. 7.6 Replication at 1.5–1.8B scale Running the whole lineage again with 1.5–1.8B checkpoints reproduces the structure (Figure 2b). The Banking77 ceiling is statistically flat across four generations (.9247 → .9208, p=0.32p=0.32) while zero-shot climbs from .1984 to .4247; the gen-0 specialist still leads the newest base by 50 points. The Spider ceiling rises every generation (.4642 → .6509, p<10−26p<10^-26), and here the half-life resolves inside the lineage: the gen-0 specialist’s advantage is positive through g=2g=2 but −8.1-8.1p at g=3g=3 (.4642 vs. .5455, p=2×10−6p=2×10^-6). At 7–8B we could only bracket the crossing; at 1.5–1.8B it is directly observed, on the same task, in the same direction. The replication also sharpens the taxonomy in one respect: xLAM-FC, whose 7–8B ceiling is nearly flat (+2.3+2.3p), gains +8.5+8.5p across the small-scale lineage (p=3×10−27p=3×10^-27). Competence provenance is therefore not a fixed attribute of a task but a relation between the task and the model’s capability: while a model class is still far from saturating a task, base progress penetrates fine-tuning (base-bound behavior); once the task is saturated at that scale, the fine-tune becomes the ceiling (data-bound behavior). This predicts that today’s base-bound tasks will migrate to data-bound as bases improve, which is itself a measurable, falsifiable claim the released harness can track. 7.7 Label-free distillation refresh Refresh-D is annotation-free but input-retaining: the old specialist relabels the retained task inputs, and a fresh LoRA is trained on the new base (it is not “data-free”—the inputs must have been kept). Teacher–gold agreement is 97.5% (Banking77 labels), 88.3% (Spider, exact-string, which underestimates semantic agreement), and 92.8% (xLAM). Refreshed students score .9312 / .7611 / .8425, i.e., retention 0.997 / 1.047 / 0.960 against gold-label retraining, with no detected difference (McNemar p=0.81p=0.81, 0.820.82, 0.580.58). Against the pre-set margins, Banking77 is formally non-inferior at 1p (paired CI [−0.62,+0.42][-0.62,+0.42]p; three student seeds span .9302–.9312); Spider and xLAM are non-inferior at 2p but not at 1p (CIs [−1.45,+1.93][-1.45,+1.93] and [−1.20,+0.60][-1.20,+0.60]). Spider’s student seeds span .7302–.7689 (3.9p), so its single-run point estimate sitting above the reference (.7611 vs. .7582) is inside training noise; a teacher that is itself only 73% accurate producing training data of reference quality would echo weak-to-strong generalization (4), but our three seeds cannot separate that from chance. A fourth cell added at review extends the pattern to a second task class: a CLINC150 student distilled from the Qwen2.5 specialist scores .9267 on Qwen3, inside the three-seed band of its gold-label references (.9180–.9313). Where copying was applicable and achieved positive retention, refresh delivered 2.6× (xLAM) to 7.5× (Spider) more of the attainable gain; on Banking77 the comparison is degenerate because copying was net harmful. The resource picture, however, is asymmetric: counting teacher inference, refresh costs 39.1 GPU-minutes on Banking77 (vs. 45.9 for gold retraining) but 255.0 on Spider (vs. 205.8) and 331.2 on xLAM (vs. 215.8). Refresh-D saves annotation, not compute; and if the original gold labels were retained alongside the inputs, plain retraining dominates it. Zero gold labels, zero API calls. 7.8 Recovery cost under a label budget On the taxed task (Spider, Qwen3), gold-label retraining with a budget ladder yields .6847 (256 labels—nominally below the .6963 zero-shot floor, though the gap is not significant, p=0.46p=0.46: 256 labels buy nothing), .7205 (1,024; 39% of attainable gain), .7427 (4,096; 75%), .7582 (full 6,362). The 95%-of-reference-gain budget C95C_95 exceeds 4,096 labels, whereas label-free Refresh-D reaches 105% using retained inputs only. Where the upgrade tax is real, the cheapest way to pay it is to recycle the old specialist’s behavior, not to re-annotate. 7.9 Backward compatibility: what averages hide An upgrade that leaves mean accuracy unchanged can still regress previously verified behavior. From the released per-example records we compute, for every strategy pair on the identical test set, the negative flip rate (items the old system answered correctly and the new one does not) alongside the positive flip rate (Table 9). Three results matter for deployment. First, even quality-neutral retraining churns: replacing the Qwen2.5 specialist with a retrained Qwen3 reference flips 1.2% (Banking77) to 4.7% (Spider) of items negatively even when the net change is zero or positive. Second, the CLINC150 decline of −0.89-0.89p decomposes into 2.55% negative against 1.65% positive flips—140 previously-correct items break, invisible in the mean. Third, naive copying is catastrophic on this axis: the copied Banking77 adapter negatively flips half the test set (50.4%) relative to the source specialist. Refresh-D shows the same flip profile as gold retraining (1.1–3.8% negative), so annotation-free refresh does not add churn beyond retraining itself. Table 9: Negative/positive flip rates between strategy pairs on the fixed test sets (per-example records; NF = old correct, new wrong). Transition NF% PF% Net (p) Banking77: 2.5-specialist → 3-reference 1.23 1.33 +0.10+0.10 Spider: 2.5-specialist → 3-reference 4.74 7.35 +2.61+2.61 xLAM: 2.5-specialist → 3-reference 3.90 3.65 −0.25-0.25 CLINC150: 2-specialist → 3-specialist 2.55 1.65 −0.89-0.89 FinQA: 2-specialist → 3-specialist 4.27 5.49 +1.22+1.22 glaive-FC: 2-specialist → 3-specialist 0.50 0.40 −0.10-0.10 Banking77: source specialist → Copy 50.39 0.49 −49.90-49.90 Banking77: reference → Refresh-D 1.17 1.07 −0.10-0.10 Spider: reference → Refresh-D 3.77 4.06 +0.29+0.29 xLAM: reference → Refresh-D 2.20 1.90 −0.30-0.30 7.10 Does the decision policy beat fixed strategies? The framework’s claim is that genealogy, coupling, and flips suffice to choose well. We test this by offline replay over the 33 task×scale×hop episodes in which at least Freeze, Adopt, and the retraining reference are all measured (Copy on 18 of them, Refresh-D on 3). The pre-specified policy is: (i) if the reference beats Freeze by at most ε (1p classification, 2p structured), freeze; (i) else if a copy is defined, the target is a documented short continuation, and the copy passes a regression gate at ε , copy; (i) else if retained inputs exist and the refreshed student passes the gate, refresh; (iv) else retrain. Each policy is charged the training cost of the action it picks and its regret against the best measured action of that episode (Table 10). Table 10: Offline policy replay over 33 measured upgrade episodes. Regret is against the best measured action per episode; a regression episode is one where the chosen action falls more than ε below Freeze. Policy Mean regret Max regret GPU-h Gold-label eps. Regressions Always-Freeze 1.80p 11.5p 0 0 0 Always-Adopt 32.35p 91.6p 0 0 33 Always-retrain 0.21p 2.0p 47.7 33 1 Copy-if-shape 17.11p 90.8p 25.8 15 14 Proposed 0.37p 6.6p 15.6 9 0 To avoid scoring the policy on the same measurements its gates consult, each test set is split in half by a hash of the item id: gate decisions read only the gate half, and all regret and regression figures are computed only on the held-out half. Under this split the proposed policy tracks always-retraining (0.37 vs. 0.21p mean regret) at 33% of its training compute and 27% of its gold-label episodes, with zero regression episodes; its worst episode (6.6p) comes from a gate miscall on a half-sized test set—gate sampling error made visible. (Letting the gates read the full test set, as an optimistic upper bound, gives 0.13p.) The instructive failure is copy-if-shape—the rule practitioners actually use: 17p mean regret and 14 regression episodes, because shape compatibility fires exactly where copying is most dangerous. One caveat stands: regret is relative to measured actions only—an unmeasured method could dominate. 7.11 Predicting portability before paying for it The decision framework still requires one measured copy per candidate edge. RQ5 asks whether a probe that costs a single forward pass per prompt can stand in for that measurement. For each of the eight base pairs with a measured R(Copy)R( Copy), we run 256 fixed task prompts through both checkpoints (no training, no generation) and compute three signals: mean linear CKA between hidden states at nine evenly spaced layers, next-token top-1 agreement, and next-token Jensen–Shannon divergence. CKA rank-correlates with the pair’s mean R(Copy)R( Copy) at Spearman ρ=0.74ρ=0.74 (Pearson 0.54); top-1 agreement and JSD are weaker (0.38, −0.52-0.52). The failure modes are informative: CKA tracks the OLMo distance spectrum monotonically (0.80 at 46B tokens down to 0.53 at the far edges) but assigns nearly identical scores to the Qwen fresh pair and the Qwen continuation pair (.876 vs. .878), whose retentions differ by 0.8; top-1 agreement separates the Qwen pairs no better and is distorted by tokenizer differences across families. With eight pairs this is an exploratory result, not a decision rule: a probe carries real signal about weight-space drift, but none of the three is a substitute for one cheap measured copy plus a regression gate. We release the probe harness so later waves can grow n. 7.12 Reliability, variance, and threats Format reliability. Fine-tuning buys parseability as well as accuracy: unparseable xLAM outputs drop from 17–28/2000 (zero-shot) to 5–6/2000 (specialists); copied adapters land in between (15/2000). Banking77 parsing failures are zero in all conditions. Variance. Three seeds on the Banking77/Qwen2.5 cell give .9312/.9289/.9302 (σ≈0.12σ≈ 0.12p); three seeds on the Spider/Qwen3 reference give .7582/.7398/.7524, a 1.8p spread that dominates single-hop retention arithmetic on that task (§7.1). Bootstrap CIs accompany every cell in the release. After wave 6, every generational reference-score claim the paper leans on carries three seeds at the endpoints in question (Spider, CLINC150, xLAM); comparisons still resting on single seeds are those whose differences we already report as null. Threats. One lineage (family culture and generational effects are confounded); one dataset per task class; Spider uses single-database execution accuracy; xLAM-FC is an in-distribution held-out split that emphasizes format compliance; the Qwen3 hop changes parameter count (7.6B→ 8.2B) and data vintage together. UpgradeBench prices the upgrade decision as practitioners face it, bundled, rather than isolating the effect of any single factor. Copy is measured on two fresh-pretraining hops (two families, two scales), four documented continuation edges spanning 46B–2.9T tokens, one anneal edge, and one soup edge; the wave-5 distance ablation separates distance from the release operations at these endpoints, but fitting a full decay curve needs more intermediate points. The OLMo cells use base checkpoints with a fixed plain prompt format on the two intent tasks, so cross-family comparisons of absolute scores are not meaningful; all wave-4 conclusions are within-family, against floors and references measured under the identical format. The memorization probe (§4.2, Appendix E) addresses contamination; residual contamination would inflate baselines and thus understate expert advantages. 8 The Small-Evidence Regime Every analysis above runs at n=1,034n=1,034–5,5005,500 paired test items. An organization applying the decision policy to its own specialist typically holds n=100n=100–600600 labeled items, and on a saturated task the two systems disagree on only a handful of them. This section closes that external-validity gap: we characterize the policy’s operating behavior at enterprise evidence sizes by replaying it over the benchmark’s own per-example records, and we revise the opportunity gate so that its verdicts remain calibrated when the evidence cannot carry a point estimate. A three-zone opportunity gate. At small n a point estimate passed through a fixed margin conflates three different epistemic states. We therefore replace it with an equivalence-testing (TOST-style) verdict on the paired-difference confidence interval, computed from the discordant counts (n01,n10)(n_01,n_10) with a continuity correction, and a rule-of-three bound |δ|≤3.69/n|δ|≤ 3.69/n when the two systems agree on every item. The CI lying entirely above ε establishes the gain and opens the waterfall of §7.10; entirely below ε establishes equivalence (freeze as a verdict, with the exclusion bound stated); an interval straddling ε is unresolved, and the policy says so instead of defaulting to freeze. Unresolved verdicts carry a posterior over the true gain, with the benchmark’s own 193-cell corpus as an empirical-Bayes prior per lineage kind (fresh pretraining: μ=+0.82μ=+0.82p, σ=2.4σ=2.4p over 16 pair-task samples; continuation: μ=+0.51μ=+0.51p, σ=0.66σ=0.66p; seed-to-seed training variance from the multi-seed waves is folded into the observation noise), and resolve to one of two actions: collect (label the disagreement set) when the posterior leaves the gain live, or wait (hold until the next release) when P(gain>εdec)<0.10P(gain> _dec)<0.10. Disagreement-first labeling. The exact McNemar test conditions on the discordant pairs, so the decision-relevant annotation complexity is O(#disagreements)O(\#disagreements), not O(n)O(n): an organization can run both systems over unlabeled traffic (free of annotation), collect the items where their predictions differ, and label only those. Protocol. We take the 21 (task, upgrade-pair) cells of the main study for which both endpoints have full per-example records (15 ground-truth null or negative at full n, i.e. δfull≤ε _full≤ ; 6 ground-truth upgrades), subsample enterprise-sized gate sets (n∈100,…,1000n∈\100,…,1000\, 300 replicates per cell per size, fixed seed), and score each verdict against the full-n ground truth. The corpus prior is computed leave-one-pair-out, so no cell is judged under a prior that contains itself. collect is played out, not assumed: up to 50 real disagreement records are drawn and labeled, and the waterfall opens only if the exact sign test settles positive. The comparison rule (“old”) is the point-estimate gate of §7.10 applied directly at small n. n false-open (15 null cells) false-equivalence point-estimate gate three-zone + collect (6 upgrade cells) 100 17.9% 1.9% 0.5% 200 15.2% 2.1% 0.2% 300 13.0% 2.0% 0.2% 600 8.2% 1.8% 0.1% 1000 1.8% 1.5% 0.0% Table 11: Small-evidence operating curve, replayed on real records. At enterprise gate sizes the point-estimate gate opens the upgrade waterfall on one in six ground-truth-null episodes; the three-zone gate holds the false-open rate near its nominal level at every n, and its “equivalence” verdicts are almost never wrong. The cost of that conservatism is bounded and we state it: mean serving-accuracy regret 0.58p vs 0.48p at n=100n=100 (the point-estimate rule’s spurious opens occasionally land on real upgrades; its migration and regression costs are outside this metric). labeling budget b i.i.d. sampling disagreement-first 25 0% 37% 50 4% 48% 100 13% 67% Table 12: Probability that the upgrade direction settles (exact sign test, correct sign) under a fixed labeling budget, averaged over the cells with |δfull|≥1| _full|≥ 1p. Labeling b random items wastes almost the entire budget on items both systems already get right; labeling b disagreements concentrates it on the only items that carry decision information. Table 11 shows the operating curve and Table 12 the annotation-efficiency comparison. Three regime-level facts follow. First, at n≤300n≤ 300 the point-estimate gate is not a usable instrument: its false-open rate (13–18%) means roughly one in six null episodes triggers a retrain-and-migrate cycle. Second, honesty is cheap: refusing to over-claim costs about a tenth of a percentage point of mean serving accuracy while cutting false opens by an order of magnitude, and the equivalence verdicts that replace “freeze by default” are correct essentially always. Third, when the verdict is unresolved, the efficient next step is never “collect more i.i.d. labels”: at these disagreement rates (1–15%), a budget of 25 disagreement labels outperforms 100 i.i.d. labels by a factor of three on direction resolution. The replay code and per-cell results ship with our released tooling. 9 Discussion Implications. Our measurements are inconsistent with all three policies an organization might adopt by default. “Always retrain on the newest base” wastes 17–422 GPU-minutes per specialist per generation for no measurable accuracy on data-bound tasks; on xLAM the newest reference (.8455) does not even nominally exceed the one-generation-old specialist (.8480). “Never revisit fine-tunes” forfeits 42–59% of attainable gain per hop on base-bound tasks. And “just move the adapter over” depends entirely on what the new release is: harmful between independently pretrained checkpoints even at identical architecture, near-free onto a short continuation of the same weights, and back to harmful once the continuation runs long (floor-level after 2.9T tokens, with annealing and souping adding nothing further). Release genealogy plus distance, not config.json, is what an upgrade policy should read—and since released checkpoints typically sit trillions of tokens past any point where an old adapter was trained, the copyable regime is narrower than “same family” suggests. The rational policy is per-task and per-release: check genealogy and distance to decide whether copying is even on the table, measure the task–base coupling to decide between freezing and refreshing, and note that the coupling itself drifts with scale and time (§7.6). The refresh need not cost gold labels, though it does not always save compute (§7.7). For the specialist-fleet debate (1), this reframes staleness from an argument against fine-tuning into a priceable line item: the durable assets are the task data, the per-example regression records, and the refresh pipeline, not the frozen weights. Freezing is not free either—it carries security-patch, dependency, and compliance costs that our protocol does not price. Future waves. The released harness makes four extensions mechanical: (i) a further instruct-substrate lineage with a tokenizer break (Llama 2→ 3→ 3.1; those checkpoints are license-gated, which kept them out of this fully-automated study), and a finer-grained sweep of intermediate OLMo checkpoints to fit the retention-vs-distance decay curve whose endpoints we measured; (i) a versioned synthetic private-knowledge track whose corpus is provably absent from any pretraining data; (i) learned mappings (23; 39) on the shape-incompatible hops, where Copy is undefined and Refresh-D is currently the only measured option; (iv) representational probes (CKA, logit agreement) as a priori predictors of retention (RQ5), and the synthetic-input variant of Refresh-D (38) for when even task inputs are unavailable. Limitations. Specialists are capped at 8B by the single-GPU constraint and are adapted only with QLoRA at rank 16; full-parameter fine-tuning perturbs weights at a much higher effective rank (2) and may well have a different transfer profile, which we cannot speak to. Tasks are English, two per class, with the second-task suite measured over three generations rather than four. The continuity criterion rests on two fresh-pretraining hops (one inferred from release documentation, one verified from a documented lineage), two continuation hops, and one anneal-plus-soup hop; we report it as a strongly supported, twice-replicated dichotomy with one measured boundary, not yet a universal rate. The prompt-side comparison uses a deliberately small search budget (six candidates, one task, one hop) and selects on a dev slice, so it bounds the prompt asset from below rather than characterizing prompt engineering in general. The amortized decision model excludes re-validation and compliance costs. Refresh-D is measured in its relabel variant, which assumes task inputs were retained; this is the common but not the universal enterprise case. UpgradeBench measures the tax and proposes no new transfer method. 10 Conclusion We introduced UpgradeBench and measured the base-model upgrade tax along a complete fourteen-month open-weight release lineage, at two model scales, across six enterprise tasks. The tax is a task property: retrained reference scores rise only where the task–base coupling is high and stay flat—within training noise—where it is low; an apparent decline on CLINC150 did not survive three seeds. Specialization half-lives are long, bracketed to two to three generations on the most base-bound task at 7–8B and observed crossing zero at 1.5–1.8B, while a classification specialist keeps a 28.6-point lead after fourteen months of base progress. What licenses adapter porting is pretraining continuity rather than architectural compatibility: copying fails across independent pretraining runs in all eleven task/scale cells in two families and retains 0.82–1.45 between checkpoints joined by continued pretraining in both families. A prospectively specified distance ablation on OLMo’s documented training trajectory confirmed both regimes and located the boundary between them: retention falls from 0.88–0.99 at a 46B-token continuation to the no-adapter floor at 2.9T tokens, with the subsequent anneal and model-soup merge adding no further damage; portability is a distance budget, not a family property. Where copying is unavailable, annotation-free (input-retaining) refresh reaches parity with gold-label retraining at 39–331 consumer-GPU minutes; and the prompt-side asset is an order of magnitude smaller than the weight asset without being more durable. All code, adapters, per-query records, CIs, and energy logs are released; the study cost 148.7 training GPU-hours (47.8 kWh measured) plus 79.4 evaluation GPU-hours on one RTX 4090. Reproducibility Statement All experiments run on a single RTX 4090 with pinned software versions (§5). We release training/evaluation scripts, all 98 adapter checkpoints, per-query prediction records and bootstrap CIs for all 193 evaluation cells, dataset split seeds and manifests, the three relabeled Refresh-D corpora, the memorization-probe implementation and outputs, and raw cost logs (wall-clock, peak VRAM, integrated energy) behind every figure. Code and artifacts accompany the submission and will be released publicly upon publication. Ethics Statement All datasets are public under their stated licenses; no private or personal data is used. The benchmark informs build-vs-buy decisions with labor implications; we report costs transparently to support informed choices. Fine-tuning can erode safety alignment (30); released adapters are task-scoped and carry no safety claims. Total measured training energy: 47.8 kWh. Independence. The study uses only publicly released checkpoints and public documentation; no internal training records or unreleased evaluations were used. Model and task selection criteria were fixed before the first experiment, and the OLMo panel provides an external, fully documented validation family. The first author is employed by Alibaba Group, whose Qwen models form the primary release series studied here; the employers had no role in the design, execution, analysis, or conclusions of this work. References Belcak et al. (2025) P. Belcak, G. Heinrich, S. Diao, Y. Fu, X. Dong, S. Muralidharan, Y. C. Lin, and P. Molchanov Small language models are the future of agentic AI. arXiv preprint arXiv:2506.02153. Cited by: §1, §1, §2, §9. Biderman et al. (2024) D. Biderman, J. P. Ortiz, J. J. G. Paulo, et al. LoRA learns less and forgets less. Transactions on Machine Learning Research. Note: arXiv:2405.09673 Cited by: §9. Bucher and Martini (2024) M. J. J. Bucher and M. Martini Fine-tuned ’small’ LLMs (still) significantly outperform zero-shot generative AI models in text classification. arXiv preprint arXiv:2406.08660. Cited by: §1, §2. Burns et al. (2023) C. Burns, P. Izmailov, J. H. Kirchner, et al. Weak-to-strong generalization: eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390. Cited by: §7.7. Cai et al. (2022) D. Cai, E. Mansimov, Y. Lai, Y. Su, L. Shu, and Y. Zhang Measuring and reducing model update regression in structured prediction for NLP. In NeurIPS, Cited by: §2. Casanueva et al. (2020) I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807. Cited by: Table 2. Chen et al. (2023a) L. Chen, M. Zaharia, and J. Zou FrugalGPT: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. Cited by: §2. Chen et al. (2023b) L. Chen, M. Zaharia, and J. Zou How is ChatGPT’s behavior changing over time?. arXiv preprint arXiv:2307.09009. Note: Also in Harvard Data Science Review; verify final citation Cited by: §2. Chen et al. (2021) Z. Chen, W. Chen, C. Smiley, et al. FinQA: a dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Note: arXiv:2109.00122 Cited by: Table 2. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2305.14314 Cited by: §5. Erol et al. (2025) M. H. Erol et al. Cost-of-pass: an economic framework for evaluating language models. arXiv preprint arXiv:2504.13359. Cited by: §2. Farhadzadeh et al. (2025) F. Farhadzadeh, D. Das, S. Borse, and F. Porikli LoRA-X: bridging foundation models with training-free cross-model adaptation. In International Conference on Learning Representations (ICLR), Note: arXiv:2501.16559 Cited by: §1, §2. Gemma Team (2025) Gemma Team Gemma 3 technical report. arXiv preprint arXiv:2503.19786. Cited by: §1. Groeneveld et al. (2024) D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnusson, Y. Wang, et al. OLMo: accelerating the science of language models. arXiv preprint arXiv:2402.00838. Cited by: §4.1. Gu et al. (2025) N. Gu, P. Fu, X. Liu, et al. Adapt once, thrive with updates: transferable parameter-efficient fine-tuning on evolving base models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Note: arXiv:2506.06844 Cited by: §1, §2. Guan et al. (2026) Y. Guan, J. Ohene-Agyei, D. Kwan, et al. TuneShift-KD: knowledge distillation and transfer for fine-tuned models. arXiv preprint arXiv:2603.24518. Note: arXiv preprint Cited by: §1, §2. Huang et al. (2024) S. Huang et al. Chat vector: a simple approach to equip LLMs with instruction following and model alignment in new languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Note: arXiv:2310.04799 Cited by: §1, §2, §4.1, §7.3. Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, et al. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), Note: arXiv:2212.04089 Cited by: §2, §4.1. Jung and Kim (2026) C. Jung and J. Kim TiTok: transfer token-level knowledge via contrastive excess to transplant LoRA. In International Conference on Learning Representations (ICLR), Note: arXiv:2510.04682 Cited by: §1, §2. Kapoor et al. (2024) S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan AI agents that matter. arXiv preprint arXiv:2407.01502. Cited by: §2, §2. Khan et al. (2024) R. M. S. Khan, P. Li, S. Yun, et al. PortLLM: personalizing evolving large language models with training-free and portable model patches. arXiv preprint arXiv:2410.10870. Cited by: §1, §2. Larson et al. (2019) S. Larson, A. Mahendran, J. J. Peper, et al. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), Note: arXiv:1909.02027 Cited by: Table 2. Li et al. (2025) Y. Li, F. Meng, M. Zhang, et al. LoRASuite: efficient LoRA adaptation across large language model upgrades. arXiv preprint arXiv:2505.13515. Note: arXiv preprint; no acceptance shown on arXiv page as of 2026-07 Cited by: §1, §2, §9. Lin et al. (2025) P. Lin, R. Balasubramanian, F. Liu, N. Kandpal, and T. Vu Efficient model development through fine-tuning transfer. arXiv preprint arXiv:2503.20110. Note: arXiv preprint; venue listing not confirmed on arXiv page as of 2026-07 Cited by: §1, §2, §4.1, Table 3, §7.3. Liu et al. (2026) H. Liu, G. H. Je, M. Ciccone, et al. The appeal and reality of recycling LoRAs with adaptive merging. arXiv preprint arXiv:2602.12323. Note: arXiv preprint Cited by: §2. Liu et al. (2024) Z. Liu, T. Hoang, J. Zhang, et al. APIGen: automated pipeline for generating verifiable and diverse function-calling datasets. arXiv preprint arXiv:2406.18518. Note: xLAM function-calling datasets; verify license of released data Cited by: §4.2, Table 2. Ma et al. (2023) W. Ma, C. Yang, and C. Kästner (Why) is my prompt getting worse? rethinking regression testing for evolving LLM APIs. arXiv preprint arXiv:2311.11123. Cited by: §2. Ong et al. (2025) I. Ong, A. Almahairi, V. Wu, et al. RouteLLM: learning to route LLMs with preference data. In International Conference on Learning Representations (ICLR), Note: arXiv:2406.18665 Cited by: §2. Ortiz-Jiménez et al. (2023) G. Ortiz-Jiménez, A. Favero, and P. Frossard Task arithmetic in the tangent space: improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2305.12827 Cited by: §2, §4.1. Qi et al. (2024) X. Qi, Y. Zeng, T. Xie, et al. Fine-tuning aligned language models compromises safety, even when users do not intend to!. In International Conference on Learning Representations (ICLR), Note: arXiv:2310.03693 Cited by: §10. Qwen Team (2024) Qwen Team Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §4.1. Qwen Team (2025) Qwen Team Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §1. Reuel et al. (2024) A. Reuel, A. Hardy, C. Smith, et al. BetterBench: assessing AI benchmarks, uncovering issues, and establishing best practices. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2411.12990 Cited by: §2. Sculley et al. (2015) D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J. Crespo, and D. Dennison Hidden technical debt in machine learning systems. In NeurIPS, Cited by: §2. Sheng et al. (2024) Y. Sheng, S. Cao, D. Li, et al. S-LoRA: serving thousands of concurrent LoRA adapters. In Proceedings of Machine Learning and Systems (MLSys), Note: arXiv:2311.03285 Cited by: §1, §2. Team OLMo et al. (2024) Team OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656. Cited by: Appendix A, §4.1. Tripathi et al. (2025) S. Tripathi, P. Nema, A. Halder, S. Qiao, and A. Jindal Prompt migration: stabilizing GenAI applications with evolving LLMs. arXiv preprint arXiv:2507.05573. Note: arXiv preprint Cited by: §2. Wang et al. (2024) R. Wang, S. Ghosh, D. Cox, et al. Trans-LoRA: towards data-free transferable parameter efficient finetuning. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2405.17258 Cited by: §1, §2, Table 3, §9. Xia et al. (2025) F. Xia, M. Liao, Y. Fang, et al. Cross-LoRA: a data-free LoRA transfer framework across heterogeneous LLMs. arXiv preprint arXiv:2508.05232. Note: arXiv preprint Cited by: §1, §2, §9. Xiao et al. (2024) C. Xiao, J. Cai, W. Zhao, et al. Densing law of LLMs. arXiv preprint arXiv:2412.04315. Note: arXiv preprint Cited by: §1, §2, §7.5. Xu et al. (2025) F. F. Xu, Y. Song, B. Li, et al. TheAgentCompany: benchmarking LLM agents on consequential real world tasks. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Note: arXiv:2412.14161 Cited by: §2. Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal TIES-Merging: resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2306.01708 Cited by: §2. Yan et al. (2021) S. Yan, Y. Xiong, K. Kundu, S. Yang, S. Deng, M. Wang, W. Xia, and S. Soatto Positive-congruent training: towards regression-free model updates. In CVPR, Cited by: §2. Yu et al. (2024) L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li Language models are super mario: absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning (ICML), Note: arXiv:2311.03099 Cited by: §2. Yu et al. (2018) T. Yu, R. Zhang, K. Yang, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: Table 2. Zhao et al. (2024) J. Zhao, T. Wang, W. Abid, et al. LoRA land: 310 fine-tuned LLMs that rival GPT-4, a technical report. arXiv preprint arXiv:2405.00732. Cited by: §1, §2. Appendix A Lineage details Architecture parameters in Table 1 are read directly from each checkpoint’s config.json and are reproduced in the release. Notes: (i) the vocabulary size alternates (151,936 → 152,064 → 152,064 → 151,936), so embedding-row alignment differs by hop even though the tokenizer family is constant; (i) Qwen1.5-7B uses full multi-head attention (32 query, 32 key/value heads), making its KV cache 8×8× larger than Qwen2/2.5’s GQA configuration, a deployment-relevant regression that per-parameter comparisons miss, and in our runs the dominant cost factor for long-prompt evaluation on a consumer GPU; (i) Qwen3-8B introduces an explicit head_dim of 128 and a narrower FFN (12,288) than Qwen2.5 (18,944) despite being wider overall; (iv) Qwen2.5-7B-Instruct-1M (2025-01) matches Qwen2.5-7B-Instruct in every architectural dimension and extends its context window via continued pretraining, making it the continuation release used in §7.3; (v) the small-scale track uses Qwen1.5-1.8B-Chat (MHA 16/16, hidden 2048, 24 layers), Qwen2-1.5B-Instruct and Qwen2.5-1.5B-Instruct (identical: GQA 12/2, hidden 1536, 28 layers, FFN 8960), and Qwen3-1.7B (GQA 16/8, hidden 2048, 28 layers). Small-scale lineage scores. Zero-shot / specialist by generation. Banking77: .1984/.9247, .2942/.9260, .3614/.9273, .4247/.9208. Spider: .1625/.4642, .3859/.5725, .3569/.6141, .5455/.6509. xLAM-FC: .1245/.7335, .4800/.7685, .5880/.7940, .6840/.8180. Copying on the small 2→ 2.5 hop: Banking77 .0218 (R=−0.600R=-0.600), Spider .3636 (R=0.026R=0.026), xLAM .6075 (R=0.095R=0.095). OLMo validation lineage. From config.json: OLMo-1-7B and OLMo-1.7-7B are architecturally identical (hidden 4096, 32 layers, MHA 32/32, FFN 11008, vocabulary 50304); the OLMo-1.7 model card states it was trained from scratch. OLMo-2-7B and its published stage-1 checkpoints stage1-step237000 and stage1-step248000 share one shape (hidden 4096, 32 layers, MHA 32/32, FFN 11008, vocabulary 100352); the two stage-1 snapshots are ∼46 46B training tokens apart within the same run, and the final base adds the remaining stage-1 tokens, a stage-2 anneal on a different data mix, and a model-soup merge [36]. All five are base checkpoints; training and evaluation use one fixed plain-text format (system text, blank line, input, “Answer:”) with completion-only loss and the intent-task recipe of §5. Zero-shot floor / specialist reference per checkpoint, Banking77 then CLINC150 (lenient): OLMo-1 .0334/.8795, .1416/.8336; OLMo-1.7 .2464/.9292, .2482/.9151; s1-237k .0045/.9279, .0091/.9062; s1-248k .0097/.9305, .0120/.9078; s1-928646 .1432/.9289, .0315/.9204; stage2-ingredient3 .3055/.9315, .2822/.9122; OLMo-2 main .3523/.9256, .2575/.9125. The near-zero floors of mid-run checkpoints against the substantial floors of the released models (OLMo-1.7, OLMo-2 main) reflect instruction-following ability arriving late in pretraining; retention is computed against each target’s own floor and reference, so the comparisons are unaffected. Appendix B Task, prompt, and evaluator details Full prompt templates, label normalization, SQL execution comparator (result-multiset equality, 10−410^-4 float rounding, 10 s timeout), and call-set matcher (name equality; type-coerced, order-insensitive argument comparison) as released. Banking77 5-shot exemplars are fixed across all models (seed 42, five distinct labels). CLINC150 uses the plus configuration (151 labels including out-of-scope); lenient morphological matching changes the Qwen2.5 zero-shot floor from .7173 (strict) to .7249 (lenient) and both metrics are released per cell; under constrained label decoding the same floor is .7207 (§7.12). FinQA programs are normalized before exact match (whitespace, trailing commas, numeric canonicalization); its adoption floors are volatile across generations (.221/.093/.215) because zero-shot models drift in and out of the dataset’s DSL conventions, which we flag wherever a FinQA ratio depends on the floor. glaive-FC is parsed from the original corpus (arguments are single-quoted JSON inside the assistant turn), with a 75/25 call/no-call mix; specialists and baselines are graded on both emitting correct calls and refusing correctly. An audit of our original random split found that 21.0% of test conversations (420/2,000) also appeared verbatim in training—the corpus reuses templates and tool schemas heavily—so all glaive-FC numbers in this paper use a re-split that groups conversations by (masked user template, called function) and assigns whole groups to train/val/test (7,580/505/2,000 from a 10,085-item exact-deduplicated pool, 5,428 groups). The re-split has zero cross-split leakage at exact, normalized-exact, and group level; the split manifest with SHA-256 hashes and the audit script are released. Scores on the clean split are 0.6–5.4p lower than on the leaked split; the qualitative conclusion (a flat reference score across generations) is unchanged. Overlong-completion drops: §4.4; additionally CLINC150 0/10,000, FinQA 41/6,251, glaive-FC 96/10,000. Appendix C Refresh-D details Relabel variant: teacher = the Qwen2.5-7B specialist for each task; teacher inference over retained train+val inputs (5.5 / 112.6 / 152.5 minutes for Banking77 / Spider / xLAM); student recipe identical to §5. Teacher–gold agreement: 97.45%/92.6% (Banking77 train/val), 88.3%/66.8% (Spider exact-string; semantic agreement is higher since equivalent SQL strings differ), 92.8%/83.4% (xLAM). Three-way student–teacher–gold records released. Appendix D Prompt-side pilot On the Banking77 2.5→ 3 hop we compare the weight asset against a prompt asset (best of six system prompts selected on a 200-item dev slice). The prompt asset is small and unstable: dev-selected prompts gain at most +0.4p over the default instruction on the new base (.6474 vs. .6438 zero-shot), porting the old best prompt recovers .6416, and the identity of the best prompt changes across the hop. The weight asset delivers +27–29p over the same floors and survives three generations (§7.4). Both routes pay an upgrade tax; on this task the weight asset is 30–50× larger in delivered advantage and no more fragile. (Scope: one task, one hop, a deliberately small prompt-search budget; agentic harnesses with tool scaffolds may behave differently, an extension the released harness supports.) Appendix E Memorization probe For 200 test items per task, each base receives the first 60% of the raw item text (no chat template) and greedily continues 24 tokens; we report the longest-common-prefix ratio against the true continuation and an 8-gram hit rate. Verbatim rates (LCP >> 0.5): Banking77 0–1%, Spider 0%, xLAM 3–5% (with 8-gram rates 0.32–0.46 driven by templated JSON tool syntax). Rates are approximately constant across generations, so cross-generation comparisons are not differentially contaminated. This probe detects verbatim memorization only; it cannot rule out paraphrased exposure. Appendix F Full run inventory and cost log 98 training runs, 148.7 training GPU-hours (47.8 kWh measured), 79.4 evaluation GPU-hours (energy not separately metered; the power integral covers training only), 4.6 teacher-relabel GPU-hours; per-run wall-clock, peak VRAM, and energy are tabulated in the released train_log.json files (longest single run: glaive-FC/Qwen3-8B, 422.5 min; largest energy: FinQA/Qwen3-8B, 1,255 Wh; largest peak VRAM: FinQA/Qwen3-8B, 21.1 GB; smallest run: Spider ladder-256, 8.0 min, 49.5 Wh). 193 evaluation cells, 1.2–31.7 minutes each under normal memory conditions; per-cell generation-token counts logged. Three operational notes for single-GPU replication. (1) Exceeding physical VRAM triggers silent host-memory spill with 10–40× slowdown; with MHA checkpoints (KV cache 8×8× GQA’s) and ∼ 1,000-token prompts this is systematic at batch 32 on 24 GB—the OLMo zero-shot floors cost 40.8 of the study’s GPU-hours this way before an automatic batch-shrink guard was added; wall-clock is reported as measured. (2) MSYS-based shells exhaust fork resources after roughly a day of continuous subprocess churn (the idempotent runner resumes cleanly). (3) With plain-text prompts, completion-only masking must tokenize prompt and completion separately: BPE merges a trailing-space prompt boundary into the completion’s first token, which silently masks that token from the loss—training converges normally while the specialist learns answers with their first token missing (reference self-evaluation collapsed to 5–17%). The wave-4 trainings were quarantined and rerun after this was caught by the reference check; the released runner now builds each example by concatenating separately tokenized prompt and completion, which makes the training context a token-exact prefix of the evaluation prompt by construction. One evaluator bug (a type guard in the xLAM call matcher) was fixed and the affected cell rerun; the changelog documents it. Appendix G Per-cell confidence intervals Bootstrap 95% CIs (10,000 resamples) for all 193 evaluation cells are released as ci_table.json (generated from the study registry, which also lists the diagnostic cells excluded from the study); representative half-widths: Banking77 ± 0.9p, Spider ± 2.8p, xLAM ± 1.6p. Seed variance (3 seeds, Banking77/Qwen2.5): mean .9301, σ = 0.12p. Appendix H Dataset licenses Banking77 (C-BY-4.0); Spider (C BY-SA 4.0); xLAM-function-calling-60k (C-BY-4.0, gated access via Hugging Face). License texts checked 2026-07; re-verify before redistribution.