Paper deep dive
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
Shihao Wang, Rui Kong, Xinran Chen, Hui Wu, Qipeng Qian, Jinman Zhao, Jiashu Zhao, Yuchen Li, Jimmy Huang, Dawei Yin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Saved checkpoints record states along a training trajectory, but generally do not determine the updates at states that would be visited under a different schedule. We study how accurately these checkpoints can reconstruct the endpoint of a sequential reference with prescribed update strengths. Under a common local transition model, two checkpoint-index moment conditions characterize all convex merges that agree with this reference through second order. We then prove an information limit that for nondegenerate profiles, no algorithm using only a fixed-length gradient-descent (GD) history with step size $h$ can achieve $o(h^3)$ endpoint error uniformly over a fixed class of smooth, strongly convex losses. The lower bound follows from two losses with identical GD checkpoint histories but sequential reference endpoints separated by $\Omega(h^3)$. \textbf{Quadratic-Accurate Merging} (QAM) achieves a matching uniform $O(h^3)$ endpoint error bound. Its explicit coefficients also define the unique profile-dependent merge that exactly matches the sequential GD reference across all fixed quadratic objectives. Across two public Adam checkpoint trajectories (SmolLM3-3B and OpenEuroLLM-Prelude-9B), three windows and three profiles per model, and 15 tasks, QAM shows mixed results for short windows and broader advantages over \textbf{Warmup-Stable and Merge} (WSM) for longer windows. Matched-moment GSM8K diagnostics further show that local consistency alone does not fully determine downstream scores. These results characterize the reconstruction limits of saved histories, provide a coefficient rule that attains the optimal rate, and assess its practical utility.
Tags
Links
- Source: https://arxiv.org/abs/2609.35168v1
- Canonical: https://arxiv.org/abs/2609.35168v1
Trouble viewing inline? Open PDF directly →
Full Text
89,531 characters extracted from source content.
Expand or collapse full text
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency Shihao Wang Affiliation: University of Wisconsin – Madison Affiliation: Baidu Inc. Email: yuchenli1230@gmail.com Rui Kong Affiliation: Baidu Inc. Email: yindawei@acm.org Xinran Chen Affiliation: Baidu Inc. Hui Wu Affiliation: Baidu Inc. Qipeng Qian Affiliation: University of Arizona Jinman Zhao Affiliation: University of Toronto Jiashu Zhao Affiliation: Wilfrid Laurier University Yuchen Li Affiliation: Baidu Inc. Jimmy Huang Affiliation: York University Dawei Yin Affiliation: Baidu Inc. Abstract Saved checkpoints record states along a training trajectory, but generally do not determine the updates at states that would be visited under a different schedule. We study how accurately these checkpoints can reconstruct the endpoint of a sequential reference with prescribed update strengths. Under a common local transition model, two checkpoint-index moment conditions characterize all convex merges that agree with this reference through second order. We then prove an information limit that for nondegenerate profiles, no algorithm using only a fixed-length gradient-descent (GD) history with step size h can achieve o(h3)o(h^3) endpoint error uniformly over a fixed class of smooth, strongly convex losses. The lower bound follows from two losses with identical GD checkpoint histories but sequential reference endpoints separated by Ω(h3) (h^3). Quadratic-Accurate Merging (QAM) achieves a matching uniform O(h3)O(h^3) endpoint error bound. Its explicit coefficients also define the unique profile-dependent merge that exactly matches the sequential GD reference across all fixed quadratic objectives. Across two public Adam checkpoint trajectories (SmolLM3-3B and OpenEuroLLM-Prelude-9B), three windows and three profiles per model, and 15 tasks, QAM shows mixed results for short windows and broader advantages over Warmup-Stable and Merge (WSM) for longer windows. Matched-moment GSM8K diagnostics further show that local consistency alone does not fully determine downstream scores. These results characterize the reconstruction limits of saved histories, provide a coefficient rule that attains the optimal rate, and assess its practical utility. 1 Introduction Large language model pretraining often requires adjustments to the training plan, such as introducing new data, revising the data mixture, or extending the training budget. A learning-rate decay schedule fixed in advance makes it harder to accommodate these adjustments. In contrast, warmup-stable schedules offer greater flexibility by maintaining a constant learning rate after warmup. Averaging checkpoints saved during the stable phase can improve model quality without further optimization (Izmailov et al., 2018; Sanyal et al., 2023; Ajroldi et al., 2025). However, choosing the averaging coefficients raises a more fundamental question: what alternative training behavior can a single saved history reconstruct? The history reveals updates at the recorded states, but not the updates at states that a changed schedule would have visited. Warmup-Stable and Merge (WSM) (Tian et al., 2026) gives an exact tail-sum correspondence between checkpoint weights and strengths assigned to recorded updates. Holding this profile fixed, we define a sequential reference that recomputes each relaxed transition at its current state. At checkpoint scale, a transition represents a complete saved interval. This reference supplies a coefficient-design criterion while applying its weights to a sparse Adam history remains a heuristic. We first characterize the entire class of convex merges that agrees with this reference through second order under a common local transition model. We show that two index moments are necessary and sufficient. The base merge’s discrepancy coefficient is a sum of covariances between nested update indicators. Cancelling it preserves the index mean and reduces the variance by twice this sum. The resulting family generally contains many coefficient vectors. Our main limitation result concerns the information in the training trajectory itself. For any nondegenerate fixed profile, we construct two globally smooth, strongly convex losses with exactly the same GD checkpoints, checkpoint gradients, and checkpoint losses, but reference endpoints separated by Ω(h3) (h^3). Hence no reconstruction algorithm, including a nonlinear or state-dependent one, has uniformly smaller error on the specified smooth class as h→0h→ 0. A second-order-consistent merge attains the matching O(h3)O(h^3) upper bound. Quadratic-Accurate Merging (QAM) realizes this rate using the distribution of a sum of independent Bernoulli variables with the prescribed update strengths. Local quadratic models of pretraining loss motivate an additional selection principle (Li et al., 2025) that exact matching across every quadratic objective uniquely fixes the complete coefficients. Our characterization and information bound locate the attainable local accuracy of this rule among all reconstructions from the same observations. Moreover, QAM itself is simple to construct as it requires only a scalar convolution on the coefficients. We evaluate its practical utility on two public 3B/9B Adam pretraining trajectories, three windows and three profiles per history, and 15 tasks. Compared with paired WSM, results are mixed at short windows and more favorable at intermediate and long windows. The four intermediate- and long-window linear-profile QAM merges exceed the latest checkpoint in 45/60 task comparisons. A complementary GSM8K diagnostic changes full coefficients while holding their first two index moments fixed. Its score differences demonstrate the empirical freedom left by second-order consistency. Our experiments evaluate QAM’s practical performance, while our theory establishes its optimal reconstruction rate under the stated assumptions. 2 Related Work Checkpoint and weight averaging. Stochastic weight averaging combines models along a training trajectory (Izmailov et al., 2018), while later work studies denser or more diverse forms of weight averaging (Cha et al., 2021; Ramé et al., 2022). LAWA studies early averaging with high learning rates in LLM pretraining (Sanyal et al., 2023), and Ajroldi et al. (2025) examine when and where averaging helps. Beyond same-trajectory averaging, model merging has been studied through model soups, Fisher-weighted merging, task arithmetic, interference-aware merging, and sparsified parameter updates (Wortsman et al., 2022; Matena and Raffel, 2022; Ilharco et al., 2023; Yadav et al., 2023; Yu et al., 2024); see Yang et al. (2026) for a broader survey. Averaging and update schedules. Sandler et al. (2023) connect iterate averaging with learning-rate schedules through models of SGD trajectories, while Meterez et al. (2026) study horizon-free schedules together with weight averaging. For stable-phase LLM pretraining, Tian et al. (2026) derive an exact tail-sum correspondence between checkpoint weights and recorded-update weights. We start from this correspondence and account for the state dependence introduced by sequentially recomputed transitions. Quadratic models and training dynamics. Li et al. (2025) explain pretraining averaging through a second-order loss expansion, Hessian geometry, and interactions between checkpoint deviations. Quadratic models also clarify averaging and learning-rate schedules (Sandler et al., 2023). This viewpoint motivates our matching criterion. Notice that the resulting coefficients do not require estimating the Hessian or fitting a transition to the history. 3 Quadratic-Accurate Checkpoint Merging We hold an update profile fixed and ask for a merge consistent with applying those update strengths sequentially. This distinguishes the choice of a design criterion from the downstream evaluation. 3.1 A sequential reference at checkpoint scale Let θ0,…,θk _0,…, _k be chronologically ordered checkpoints, k≥1k≥ 1, and ci≥0c_i≥ 0 with ∑ici=1 _ic_i=1 a convex base merge. Writing dj=θj+1−θjd_j= _j+1- _j yields WSM’s exact correspondence between checkpoint weights and recorded-update weights (Tian et al., 2026): θc:=∑i=0kciθi=θ0+∑j=0k−1Wjdj,Wj:=∑i=j+1kci. _c:= _i=0^kc_i _i= _0+ _j=0^k-1W_jd_j, W_j:= _i=j+1^kc_i. (1) c0=1−W0,ci=Wi−1−Wi(1≤i<k),ck=Wk−1.c_0=1-W_0, c_i=W_i-1-W_i\ (1≤ i<k), c_k=W_k-1. (2) Thus c fixes a nonincreasing input profile W∈[0,1]kW∈[0,1]^k. Reweighting the recorded djd_j leaves their evaluation states unchanged. To model the effect of changing those states, take a common transition F, with θi+1=F(θi) _i+1=F( _i), and define ϕ0W=θ0,ϕj+1W=ϕjW+Wj[F(ϕjW)−ϕjW]. _0^W= _0, _j+1^W= _j^W+W_j [F( _j^W)- _j^W ]. (3) Each transition is recomputed at the state produced by earlier ones. One transition represents a whole checkpoint interval. For a single GD step, this is learning-rate scaling. Similarly, for a saved block, it is the relaxation of the block displacement. Agreement with this reference is our sequential-consistency criterion for selecting coefficients. 3.2 Local discrepancy and its coefficient geometry Assumption 1 (Local checkpoint transition). On a neighborhood of the initial state, the common transition obeys Fh(θ)=θ+hf(θ)+h2a(θ)+O(h3)F_h(θ)=θ+hf(θ)+h^2a(θ)+O(h^3) (4) uniformly as h→0h→ 0, with f∈C2f∈ C^2, a∈C1a∈ C^1 and bounded local derivatives. The initial state, k, and W remain fixed. Here h measures the local size of a complete transition, and a includes its second-order within-interval contribution. Set f0=f(θ0)f_0=f( _0), a0=a(θ0)a_0=a( _0), J0=Df(θ0)J_0=Df( _0), and AW:=∑j=0k−1Wj,BW:=∑0≤ℓ<j≤k−1WℓWj.A_W:= _j=0^k-1W_j, B_W:= _0≤ <j≤ k-1W_ W_j. (5) Expanding both finite trajectories (Appendix A.1), the base merge matches the terms in f0f_0 and a0a_0, but its interaction term differs: θc−ϕkW=h2Δ2(W)J0f0+O(h3),Δ2(W):=∑ℓ<jWj(1−Wℓ)≥0. _c- _k^W=h^2 _2(W)J_0f_0+O(h^3), _2(W):= _ <jW_j(1-W_ )≥ 0. (6) The discrepancy has a direct interpretation. Consider a checkpoint index Ic∼cI_c c and set Xj=Ic>jX_j=1\I_c>j\. The indicators are nested, namely selecting an update also selects all earlier ones. Thus Xj=WjEX_j=W_j and [XℓXj]=WjE[X_ X_j]=W_j for ℓ<j <j, whereas the sequential reference weights this interaction by WℓWjW_ W_j. Consequently Δ2(W)=∑ℓ<jCov(Xℓ,Xj) _2(W)= _ <jCov(X_ ,X_j). This identifies a coefficient correction without estimating J0f0J_0f_0. 3.3 QAM and second-order consistency Replace the nested indicators by independent Yj∼Bernoulli(Wj)Y_j (W_j) and use the distribution of Iq=∑jYjI_q= _jY_j as checkpoint weights. Its generating polynomial is QW(t):=∏j=0k−1(1−Wj+Wjt)=∑i=0kqiti,θQAM:=∑i=0kqiθi. Q_W(t):= _j=0^k-1(1-W_j+W_jt)= _i=0^kq_it^i, _QAM:= _i=0^kq_i _i. (7) The coefficients are nonnegative and sum to QW(1)=1Q_W(1)=1. Independence gives Iq=AWEI_q=A_W and (Iq2)=BWE I_q2=B_W, matching both interaction terms in the reference. Theorem 2 (Local sequential consistency). For fixed k,Wk,W and convex weights p independent of h and the transition, ∑ipiθi−ϕkW=O(h3) _ip_i _i- _k^W=O(h^3) for every transition satisfying Assumption 1 if and only if pI=AW,p(I2)=BW.E_pI=A_W, _p I2=B_W. Equivalently, p shares QAM’s index mean and variance. In particular, θQAM−ϕkW=O(h3) _QAM- _k^W=O(h^3). Figure 1: Illustrative example of QAM with a linearly decaying update profile WjW_j. The characterization separates the consistency criterion from its realization. For k≥3k≥ 3 and Wj∈(0,1)W_j∈(0,1), the convex solution family has dimension k−2k-2: normalization and the two moments impose three independent linear constraints on k+1k+1 coefficients, and qi>0q_i>0. Quadratic matching below selects one member of this family. QAM preserves the base merge’s mean and changes its variance by exactly Var(Ic)−Var(Iq)=2Δ2(W)≥0.Var(I_c)-Var(I_q)=2 _2(W)≥ 0. (8) Thus half the variance reduction equals the weight-dependent factor in Eq. (6). This is a geometric interpretation of the consistency correction, rather than a monotonic relationship between variance and downstream accuracy. 3.4 INFORMATION LIMIT FOR CHECKPOINT-ONLY RECONSTRUCTION The attainable order is limited by the observations, even if the merge can depend nonlinearly on the states. Consider ordinary GD with step h, starting at 0∈ℝ20 ^2, and the reference GD steps hWjhW_j. For a fixed M>0M>0, define the fixed loss class ℒM=L∈C4(ℝ2):12I⪯∇2L⪯52I,∥∇L(0)∥≤3,‖D3L‖∞,‖D4L‖∞≤M.L_M= \L∈ C^4(R^2): array[]l 12I ∇^2L 52I, \|∇ L(0)\|≤ 3,\\ \|D^3L\|_∞,\|D^4L\|_∞≤ M array \. (9) Let Th(L)=(θ0,…,θk)T_h(L)=( _0,…, _k) be the recorded history. The best worst-case reconstruction error from this information is Rh=infsupL∈ℒM‖(Th(L),h,W)−ϕkW(L)‖,R_h= _A _L _M\|A(T_h(L),h,W)- _k^W(L)\|, (10) where A is any algorithm using only the displayed inputs. Remark 3. The constants 1/21/2, 5/25/2, and 33 are convenient choices for the construction and have no special threshold significance. Theorem 4 (Checkpoint-information bound). Fix k≥2k≥ 2, W∈[0,1]kW∈[0,1]^k, and M>0M>0. If SW:=∑ℓ<jWjWℓ(1−Wℓ)>0S_W:= _ <jW_jW_ (1-W_ )>0, there are c,C,h0>0c,C,h_0>0 such that ch3≤Rh≤Ch3ch^3≤ R_h≤ Ch^3 for 0<h<h00<h<h_0. QAM attains the upper bound. The lower bound continues to hold if checkpoint gradients and loss values are also supplied. If SW=0S_W=0, QAM reconstructs the reference exactly. The history is insufficient. Set r(x,y)=y−x2−2xr(x,y)=y-x^2-2x and Lh(x,y)=12(1+x)2+12(2−h)(1+y)2L_h(x,y)= 12(1+x)^2+ 12(2-h)(1+y)^2. Its ordinary iterates ((1−h)i−1,(1−h)2i−1)((1-h)^i-1,(1-h)^2i-1) lie on r=0r=0. Adding εr2/2 r^2/2 near this curve, with a smooth cutoff, preserves every recorded state, gradient, and loss. Both objectives belong to the same ℒML_M for small fixed ε>0 >0. Their reference endpoints nevertheless differ by εSWh3(2,−1)+O(h4) S_Wh^3(2,-1)+O(h^4). The same observations must produce the same reconstruction, so one error is at least half this separation. The matching uniform upper bound follows from QAM’s moment conditions. Appendix A.2 gives the complete construction. The hard objectives may vary with h inside the fixed class; the result is a uniform minimax rate, not a pointwise lower bound for a fixed loss. Other moment-matching rules can attain the rate, which does not order their constants or task scores. For fixed linear weights, two third-order terms give a corresponding algebraic obstruction; QAM’s leading remainder is proportional to SWD2f[f,f]S_WD^2f[f,f] (Appendix A.3). Quadratic exactness below selects a complete rule where this nonlinear term vanishes. 3.5 Quadratic exactness selects a complete rule Local quadratic loss models motivate a stronger criterion (Li et al., 2025). Let L(θ0+u)=L(θ0)+g0⊤u+12u⊤H0u,H0=H0⊤.L( _0+u)=L( _0)+g_0 u+ 12u H_0u, H_0=H_0 . (11) be fixed, and consider ordinary GD with step η and reference GD with steps ηWjη W_j, both initialized at θ0 _0. Theorem 5 (Quadratic endpoint matching and uniqueness). Fix k≥1k≥ 1, W∈[0,1]kW∈[0,1]^k, and η>0η>0. For every fixed quadratic objective and initial state, the GD trajectories in Eqs. (23) and (24) satisfy ∑i=0kqiθi=ϕkW. _i=0^kq_i _i= _k^W. (12) If real coefficients p0(W),…,pk(W)p_0(W),…,p_k(W) depending only on W satisfy this identity with p in place of q for every quadratic objective and every initial state, then pi(W)=qi(W)p_i(W)=q_i(W) for all i. For L(x)=12λx2L(x)= 12λ x^2, set t=1−ηλt=1-ηλ. Ordinary GD yields xi=tix0x_i=t^ix_0, whereas each reference step multiplies its state by 1−Wj+Wjt1-W_j+W_jt. Expanding their product gives the coefficients in Eq. (7); matching this polynomial for every scalar quadratic proves uniqueness. Appendix A.5 gives the full vector proof, including linear terms and singular or indefinite Hessians. Coefficient construction. Initialize q0(0)=1q^(0)_0=1, with zero entries outside the support, and apply the convolution qi(j+1)=(1−Wj)qi(j)+Wjqi−1(j),i=0,…,j+1.q_i^(j+1)=(1-W_j)q_i^(j)+W_jq_i-1^(j), i=0,…,j+1. (13) This costs O(k2)O(k^2) scalar operations and O(k)O(k) working memory. The final merge uses the saved checkpoints without gradients, optimizer states, a fitted transition, or additional training. For actual Adam histories, a common local transition is a design model; the experiments assess the resulting weights, not reference-endpoint accuracy. 4 Experiments We evaluate the practical utility of QAM on saved pretraining histories, using matched WSM merges and individual checkpoints as references. These comparisons assess downstream quality rather than reconstruction of a separately trained reference endpoint. 4.1 Setup Trajectories and profiles. We study two public models, SmolLM3-3B and OpenEuroLLM-Prelude-9B, using checkpoints from constant-learning-rate training (Table 1). The SmolLM3 windows contain 5, 10, or 15 checkpoints and span approximately 0.40, 0.90, or 1.40T training tokens. The Prelude windows contain 10, 20, or 40 checkpoints and span 0.36, 0.76, or 1.56T tokens. We call these the short, intermediate, and long windows within each trajectory. For k+1k+1 checkpoints, ordered oldest to newest, we set Wj=D((j+1)/(k+1))W_j=D((j+1)/(k+1)) for j=0,…,k−1j=0,…,k-1 and use Dlinear(s)=1−s,Dcosine(s)=cos(πs/2),D1−⋅(s)=1−s.D_linear(s)=1-s, D_cosine(s)= (π s/2), D_1- ·(s)=1- s. Paired WSM and QAM merges use the same checkpoints and profile, changing only the coefficients in Eqs. (2) and (7). Appendix C shows the resulting checkpoint weights across window lengths and profiles. All merges use fp32 accumulation and bf16 export, without additional training or coefficient fitting. Evaluation and reporting. The suite covers 15 tasks in six capability groups including Knowledge, Commonsense, Reading, Math, Semantics, and Dialogue. MMLU and GSM8K use 5-shot evaluation, and other accuracy tasks use zero-shot. Group scores weight accuracy tasks by evaluation-set size. Math averages flexible and strict GSM8K extraction, and Dialogue uses MuTual MRR. Appendix B gives task membership and checkpoint ranges. We report differences and win counts over the evaluated grid. Configurations share training histories and evaluation examples, and these counts should not be interpreted as independent replications. Table 1: Model details. Model Size Checkpoints Interval Windows SmolLM3 ∼3.075 3.075B last 15 constant-LR checkpoints; iter 3.64M–4.20M ∼100 100B tokens Last-5/10/15 OpenEuroLLM Prelude ∼9.095 9.095B last 40 constant-LR checkpoints; iter 859200–952800 ∼40 40B tokens Last-10/20/40 4.2 Utility across capabilities Against the mean constituent score, which is the expected score of selecting one checkpoint uniformly from the window, QAM improves in 105/108 capability comparisons, including every Knowledge, Commonsense, Math, and Semantics configuration (Tables 9 and 10). The only three regressions are in SmolLM3 Reading and Dialogue. We next compare coefficient rules and stronger individual-checkpoint references. 4.3 Window-dependent comparison with WSM Figures 2 and 3 show all paired capability-group differences across windows and profiles. Short-window differences are small and mixed as the median absolute gap is about 0.35 points, with QAM lower in 21 of 36 comparisons. This is the window scale emphasized in prior work (Tian et al., 2026). At intermediate windows, QAM wins 28 of 36 comparisons, including 22 of 30 when Math is excluded. At long windows, it wins 29 of 36, including 23 of 30 outside Math. Figure 2: Paired score differences (QAM minus WSM) on SmolLM3 across six capability groups and profiles. Average weights tasks by evaluation-set size. Figure 3: Same as Figure 2, but for OpenEuroLLM Prelude. Intermediate and long windows. SmolLM3 improves in 16/18 intermediate-window and all 18 long-window comparisons. Prelude improves in 12/18 and 11/18, respectively. The largest long-window gains occur in Math: +8.91+8.91–+11.30+11.30 points for SmolLM3 and +32.68+32.68–+36.43+36.43 for Prelude. Prelude’s seven long-window decreases are all below 1.2 points and occur in Knowledge, Reading, or Semantics. Its large Math gap includes a WSM generation failure, and Appendix E distinguishes this behavior from the less severe SmolLM3 case. 4.4 Comparison with individual checkpoints We compare QAM with individual checkpoints across all three windows and all three profiles on each trajectory. Table 2 summarizes comparisons with every constituent checkpoint, the latest checkpoint, and the task-wise best checkpoint within each window. Table 2: QAM versus individual checkpoints across all nine configurations per trajectory. Entries are wins/ties/losses. “All” counts comparisons with every constituent checkpoint. “Latest” and “Best” each contain 135 task–configuration comparisons per trajectory. Trajectory All Latest Best SmolLM3 1092/48/210 102/5/28 70/3/62 Prelude 2716/121/313 117/9/9 77/2/56 Total 3808/169/523 219/14/37 147/5/118 QAM exceeds the latest checkpoint in 219 of 270 comparisons and the task-wise best constituent in 147 of 270. It wins 80.9% of constituent-checkpoint comparisons on SmolLM3 and 86.2% on Prelude. Appendix H reports the results for every window, profile, and task. Remark 6. WSC single-checkpoint scores fluctuate substantially, and the best checkpoint can exceed QAM by a large margin (Appendix I). This unfavorable case is retained in all counts and aggregate comparisons. 4.5 Comparison with the best tested WSM We focus on intermediate and long windows, where the paired comparisons show the clearest QAM gains. We examine whether these gains persist when WSM can choose its best window and profile from the full tested grid. For each task, we compare the best QAM score across these two windows and all three profiles with the best WSM score across all nine tested window/profile combinations. QAM exceeds this baseline on 8 of 15 tasks for SmolLM3 and 9 of 15 for Prelude. These are task-wise maxima, and the maximizing configurations can differ across tasks. Appendix G.3 reports the full task-wise comparison. Table 3 further examines GSM8K with QAM’s profile fixed to linear, while allowing WSM to choose its best window and profile. QAM exceeds this baseline at both intermediate and long windows on both trajectories, by 1.55–2.12 points. Table 3: GSM8K with a fixed linear QAM profile versus the best WSM score over all nine tested window/profile combinations on each trajectory. Scores average flexible and strict extraction. Trajectory QAM window Linear QAM Best WSM Δ SmolLM3 Last-10 43.71 41.74 +1.97 SmolLM3 Last-15 43.29 41.74 +1.55 Prelude Last-20 44.47 42.91 +1.56 Prelude Last-40 45.03 42.91 +2.12 5 Second-Order Description and Coefficient geometry The variance identity relates the local discrepancy to coefficient geometry. We examine this relation empirically on long-window GSM8K, where WSM and QAM differ most. For paired WSM and QAM weights, let p(λ)=(1−λ)c+λqp^(λ)=(1-λ)c+λ q, λ∈[0,1]λ∈[0,1]. The means of c and q agree, so Var(Iλ)=Var(Ic)−2λΔ2(W).Var(I_λ)=Var(I_c)-2λ _2(W). (14) Under Assumption 1, the same path satisfies θλ−ϕkW=h2(1−λ)Δ2(W)J0f0+O(h3). _λ- _k^W=h^2(1-λ) _2(W)J_0f_0+O(h^3). (15) Thus, it reduces both the index variance and the model’s leading discrepancy coefficient. The path also changes higher moments and interpolates between the two merged parameter vectors. Figure 4: WSM–QAM interpolation on Prelude Last-40 (left) and SmolLM3 Last-15 (right). Lines average flexible and strict GSM8K extraction; shading spans the two extraction scores. Figure 4 shows overall improvement toward QAM for all three profiles on both histories. Prelude’s path also recovers normal generation and termination behavior, whereas the SmolLM3 WSM endpoints show much less severe formatting problems (Appendix E). The trend of performance improvement is shared on both models. To test whether the two index moments determine these scores, we pair each λ∈0.25,0.50,0.75λ∈\0.25,0.50,0.75\ mixture with a maximum-entropy (MaxEnt) distribution on the same checkpoint grid. Its coefficients have the form ri∝exp(ai+bi2)r_i (ai+bi^2), with a,ba,b set by the mixture’s mean and variance. This fixes a second coefficient vector without choosing it by benchmark performance. Appendix F.1 specifies the construction. Each pair therefore shares the same second-order expansion under the local model, including its residual relative to the fixed reference. Table 4 gives the paired scores. The rankings vary: on SmolLM3, MaxEnt has higher flexible-extraction scores in six of nine pairs, whereas on Prelude the mixture has higher scores in seven of nine. At Prelude λ=0.75λ=0.75, the flexible-score gaps favoring the mixture are 6.6, 1.1, and 13.4 points for the three profiles. For these pairs, the input grid, center, variance, and second-order reference discrepancy are all held fixed. Averaging the two extraction scores, the largest gap is 15.15 points. Table 4: GSM8K intermediate-target controls (%). Mix stands for p(λ)p^(λ). MaxEnt matches its two moments on the same grid. SmolLM3 Last-15 Prelude Last-40 Flex Strict Flex Strict Profile λ Mix MaxEnt Mix MaxEnt Mix MaxEnt Mix MaxEnt Linear 0.25 36.39 36.85 34.65 35.18 12.3 12.5 6.7 6.9 0.50 38.89 39.27 37.60 38.36 21.2 18.1 15.5 11.5 0.75 40.86 39.20 40.49 38.67 38.8 32.2 37.8 29.6 Cosine 0.25 39.50 40.33 38.44 39.73 19.2 18.4 13.2 13.6 0.50 41.39 42.00 40.79 40.79 31.0 27.4 27.0 23.0 0.75 42.08 42.15 41.17 41.02 40.4 39.3 39.6 38.4 1−⋅1- · 0.25 35.33 34.27 32.68 32.07 13.5 14.1 8.0 8.9 0.50 38.06 37.30 36.47 35.71 21.2 14.9 14.7 8.7 0.75 38.97 39.35 38.13 38.21 36.8 23.4 35.9 19.0 Separate MaxEnt controls matching the linear QAM endpoint’s mean and variance score slightly higher under both extraction rules. Their mean-score advantages are 0.34 points on SmolLM3 and 0.68 on Prelude (Appendix F.2). These point estimates establish neither a significant difference nor equivalence. They provide no empirical advantage for QAM over this same-moment alternative. Quadratic exactness selects a complete rule without guaranteeing a downstream ranking. The index-variance view suggests a possible explanation for part of QAM’s long-window advantage. Compared with WSM, QAM concentrates the merge weights into a narrower but still nondegenerate region of the checkpoint window. This should not be interpreted as a causal claim that lower variance is always better. Individual checkpoints have zero index variance yet are often outperformed (Table 2), and the matched-moment controls above show that the first two index moments do not fully determine downstream performance. 6 Discussion and Limitations We design the merging coefficients based on a common transition model, while sparse Adam histories need not satisfy those assumptions. The minimax result concerns parameter reconstruction, fixed k, and h→0h→ 0 over a specified loss class. It gives neither optimal error constants nor a downstream ordering or guarantee of reproducing decay training. WSM’s original evaluation includes the Warmup-Stable-Decay (WSD) baseline, whereas a matched WSD comparison here would require additional training. The empirical results cover two histories with shared checkpoints and test examples. Task-wise maxima are retrospective rather than one model’s simultaneous scores. Window length, spacing, and kernel shape are not fully separated as changing merging granularity also changes QAM’s effective width even on a fixed token interval. Prelude’s severe generation failure should not be generalized to all WSM merges. 7 Conclusion A saved history determines second-order reference behavior but leaves third-order ambiguity, even for globally smooth strongly convex losses and unrestricted reconstruction algorithms. QAM attains the sharp uniform local rate and is uniquely selected among profile-based linear rules by exact quadratic matching. Its sparse-Adam use yields practical gains on two pretraining histories, especially at longer windows. Matched-moment diagnostics show why consistency and downstream quality remain distinct questions. An important open question is to what extent this reconstruction viewpoint can approximate the reference endpoint of an actual learning-rate decay schedule, rather than merely provide a principled criterion for coefficient design. Stronger local models of the training dynamics may be needed to make this connection more faithful. References Ajroldi et al. (2025) Niccolò Ajroldi, Antonio Orvieto, and Jonas Geiping. When, Where and Why to Average Weights? In Proceedings of ICML, PMLR 267:925–941, 2025. https://proceedings.mlr.press/v267/ajroldi25a.html. Butcher (1963) John C. Butcher. Coefficients for the study of Runge–Kutta integration processes. Journal of the Australian Mathematical Society, 3(2):185–201, 1963. https://doi.org/10.1017/S1446788700027932. Cha et al. (2021) Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain Generalization by Seeking Flat Minima. In Advances in Neural Information Processing Systems, volume 34, pages 22405–22418, 2021. https://arxiv.org/abs/2102.08604. Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic. In Proceedings of ICLR, 2023. https://arxiv.org/abs/2212.04089. Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging Weights Leads to Wider Optima and Better Generalization. arXiv:1803.05407, 2018. https://arxiv.org/abs/1803.05407. Li et al. (2025) Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, Jin Ma, Xunhao Lai, Yao Luo, Xingyan Bin, Hongbin Ren, Mingji Han, Wenhao Hao, Bairen Yi, Lingjun Liu, Bole Ma, Xiaoying Jia, Xun Zhou, Liang Xiang, and Yonghui Wu. Model Merging in Pre-training of Large Language Models. In Advances in Neural Information Processing Systems, volume 38, 2025. https://proceedings.neurips.c/paper_files/paper/2025/hash/c20447998d6c624b4b97d4466a3bfff5-Abstract-Conference.html. Matena and Raffel (2022) Michael Matena and Colin Raffel. Merging Models with Fisher-Weighted Averaging. In Advances in Neural Information Processing Systems, volume 35, 2022. https://arxiv.org/abs/2111.09832. Meterez et al. (2026) Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, and Sham Kakade. Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging. arXiv:2602.03702, 2026. https://arxiv.org/abs/2602.03702. Ramé et al. (2022) Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse Weight Averaging for Out-of-Distribution Generalization. In Advances in Neural Information Processing Systems, volume 35, 2022. https://arxiv.org/abs/2205.09739. Sandler et al. (2023) Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Nolan Miller. Training trajectories, mini-batch losses and the curious role of the learning rate. arXiv:2301.02312, 2023. https://arxiv.org/abs/2301.02312. Sanyal et al. (2023) Sunny Sanyal, Atula Neerkaje, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Early Weight Averaging meets High Learning Rates for LLM Pre-training. arXiv:2306.03241, 2023. https://arxiv.org/abs/2306.03241. Tian et al. (2026) Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training. In Proceedings of ICLR, 2026. https://arxiv.org/abs/2507.17634. Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of ICML, PMLR 162:23965–23998, 2022. https://proceedings.mlr.press/v162/wortsman22a.html. Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-Merging: Resolving Interference When Merging Models. In Advances in Neural Information Processing Systems, volume 36, 2023. https://arxiv.org/abs/2306.01708. Yang et al. (2026) Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities. ACM Computing Surveys, 58(8), Article 216, 2026. https://doi.org/10.1145/3787849. Yu et al. (2024) Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. In Proceedings of ICML, PMLR 235:57755–57775, 2024. https://proceedings.mlr.press/v235/yu24p.html. Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of AAAI, 34(5):7432–7439, 2020. https://ojs.aaai.org/index.php/AAAI/article/view/6239. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457, 2018. https://arxiv.org/abs/1803.05457. Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of NAACL-HLT, pages 2924–2936, 2019. https://doi.org/10.18653/v1/N19-1300. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021. https://arxiv.org/abs/2110.14168. Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. MuTual: A Dataset for Multi-Turn Dialogue Reasoning. In Proceedings of ACL, pages 1406–1416, 2020. https://doi.org/10.18653/v1/2020.acl-main.130. Gordon et al. (2012) Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 Task 7: Choice of Plausible Alternatives: A New Benchmark for Commonsense Causal Reasoning. In Proceedings of the First Joint Conference on Lexical and Computational Semantics, pages 394–398, 2012. https://aclanthology.org/S12-1052/. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In Proceedings of ICLR, 2021. https://arxiv.org/abs/2009.03300. Levesque et al. (2012) Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Proceedings of KR, 2012. https://cdn.aaai.org/ojs/19068/19068-13-22935-1-10-20211013.pdf. Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of EMNLP, pages 2381–2391, 2018. https://doi.org/10.18653/v1/D18-1260. Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations. In Proceedings of NAACL-HLT, pages 1267–1273, 2019. https://doi.org/10.18653/v1/N19-1128. Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In Proceedings of AAAI, 34(5):8732–8740, 2020. https://ojs.aaai.org/index.php/AAAI/article/view/6399. Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of EMNLP, pages 1631–1642, 2013. https://aclanthology.org/D13-1170/. Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. Creating a Dataset for Science Question Answering with Supporting Evidence. In Proceedings of EMNLP, pages 962–967, 2017. https://doi.org/10.18653/v1/D17-1107. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of ACL, pages 4791–4800, 2019. https://doi.org/10.18653/v1/P19-1472. Appendix A Proofs and Additional Analysis A.1 Proof of local sequential consistency Proof of Theorem 2. The recorded checkpoints and the reference expand as θi _i =θ0+ihf0+h2[ia0+(i2)J0f0]+O(h3), = _0+ihf_0+h^2 [ia_0+ i2J_0f_0 ]+O(h^3), (16) ϕkW _k^W =θ0+hAWf0+h2[AWa0+BWJ0f0]+O(h3). = _0+hA_Wf_0+h^2[A_Wa_0+B_WJ_0f_0]+O(h^3). (17) For fixed k and sufficiently small h, bounded f,af,a and the uniform remainder keep both finite trajectories O(h)O(h) from θ0 _0 within the assumed neighborhood. Taylor expansion gives f(θ0+u)=f0+J0u+O(‖u‖2)f( _0+u)=f_0+J_0u+O(\|u\|^2) and a(θ0+u)=a0+O(‖u‖)a( _0+u)=a_0+O(\|u\|). Induction on i gives Eq. (16) as the coefficient of J0f0J_0f_0 satisfies Ci+1=Ci+iC_i+1=C_i+i, C0=0C_0=0, hence Ci=(i2)C_i= i2. Similarly, Aj+1=Aj+WjA_j+1=A_j+W_j and Bj+1=Bj+WjAjB_j+1=B_j+W_jA_j give Eq. (17). For any fixed convex p, writing μp=∑ipi _p= _iip_i and νp=∑i(i2)pi _p= _i i2p_i therefore yields ∑ipiθi−ϕkW=h(μp−AW)f0+h2[(μp−AW)a0+(νp−BW)J0f0]+O(h3). _ip_i _i- _k^W=h( _p-A_W)f_0+h^2 [( _p-A_W)a_0+( _p-B_W)J_0f_0 ]+O(h^3). (18) This proves sufficiency in Theorem 2. For necessity, consider the one-dimensional maps Fh(x)=x+hF_h(x)=x+h at x0=0x_0=0 and then Fh(x)=(1+h)xF_h(x)=(1+h)x at x0=1x_0=1. The first forces μp=AW _p=A_W and the second forces νp=BW _p=B_W. The identity 2νp=Varp(I)+μp2−μp2 _p=Var_p(I)+ _p^2- _p gives the equivalent mean-and-variance condition. The base tail sums give μc=AW _c=A_W and νc=∑jWj=∑ℓ<jWj _c= _jjW_j= _ <jW_j. Substituting these in Eq. (18) proves Eq. (6). For QAM, QW′(1)=AWQ_W (1)=A_W and QW′(1)=2BWQ_W (1)=2B_W. ∎ Remark 7. The common-map assumption is an idealized model. It includes a finite block of a fixed smooth deterministic update admitting this expansion, but does not assert that a sparse Adam history obeys such a map in its parameter state alone. Neither h nor the expansion’s remainder is estimated from the experimental histories. A.2 Proof of the Checkpoint-Information Bound Recall that ℒML_M, defined in Eq. (9), consists of C4C^4 losses on ℝ2R^2 satisfying 12I⪯∇2L⪯52I,∥∇L(0)∥≤3,∥D3L∥∞,∥D4L∥∞≤M. 12I ∇^2L 52I, \|∇ L(0)\|≤ 3, \|D^3L\|_∞,\|D^4L\|_∞≤ M. Fix M>0M>0, k≥2k≥ 2, and W∈[0,1]kW∈[0,1]^k, with SW=∑ℓ<jWjWℓ(1−Wℓ)>0.S_W= _ <jW_jW_ (1-W_ )>0. The ordinary and reference trajectories start at the origin and satisfy θi+1=θi−h∇L(θi),ϕj+1=ϕj−hWj∇L(ϕj),θ0=ϕ0=0. _i+1= _i-h∇ L( _i), _j+1= _j-hW_j∇ L( _j), _0= _0=0. The observation is Th(L)=(θ0,…,θk)T_h(L)=( _0,…, _k), and QAM is defined as Qh(L)=∑i=0kqiθi,∑i=0kqiti=∏j=0k−1(1−Wj+Wjt).Q_h(L)= _i=0^kq_i _i, _i=0^kq_it^i= _j=0^k-1(1-W_j+W_jt). Proof of Theorem 4. We first bound the error of QAM uniformly over ℒML_M. We then construct two losses that give exactly the same observations but whose reference endpoints differ at order h3h^3. Uniform upper bound. Write f=−∇Lf=-∇ L. The assumptions give ∥f(0)∥≤3,∥Df∥∞≤52,∥D2f∥∞,∥D3f∥∞≤M.\|f(0)\|≤ 3, \|Df\|_∞≤ 52, \|D^2f\|_∞,\|D^3f\|_∞≤ M. For an ordinary or reference update with weight w∈[0,1]w∈[0,1], ‖z+hwf(z)‖≤(1+52h)‖z‖+3h.\|z+hwf(z)\|≤(1+ 52h)\|z\|+3h. Thus, starting from zero and iterating at most k times yields ‖z‖≤3khexp(5kh/2).\|z\|≤ 3kh (5kh/2). Thus every state is O(h)O(h), uniformly over L∈ℒML _M and W∈[0,1]kW∈[0,1]^k. Let a=f(0),B=Df(0),H=D2f(0).a=f(0), B=Df(0), H=D^2f(0). Taylor expansion yields f(z)=a+Bz+12H[z,z]+O(‖z‖3),f(z)=a+Bz+ 12H[z,z]+O(\|z\|^3), with a uniform remainder bound. Since each state is O(h)O(h), the remainder contributes O(h4)O(h^4) to one update. For the ordinary trajectory, write θi=iha+h2ciBa+h3[diB2a+viH[a,a]]+O(h4). _i=iha+h^2c_iBa+h^3 [d_iB^2a+v_iH[a,a] ]+O(h^4). Substituting this expression into θi+1=θi+hf(θi) _i+1= _i+hf( _i) gives ci+1=ci+i,di+1=di+ci,vi+1=vi+12i2,c_i+1=c_i+i, d_i+1=d_i+c_i, v_i+1=v_i+ 12i^2, with c0=d0=v0=0c_0=d_0=v_0=0. Hence ci=(i2),di=(i3),vi=(i3)+12(i2).c_i= i2, d_i= i3, v_i= i3+ 12 i2. It follows that θi=iha+h2(i2)Ba+h3[(i3)B2a+((i3)+12(i2))H[a,a]]+O(h4). _i=iha+h^2 i2Ba+h^3 [ i3B^2a+ ( i3+ 12 i2 )H[a,a] ]+O(h^4). For the reference trajectory, define Aj=∑ℓ<jWℓ,E2,j=∑a<b<jWaWb,E3,j=∑a<b<c<jWaWbWc.A_j= _ <jW_ , E_2,j= _a<b<jW_aW_b, E_3,j= _a<b<c<jW_aW_bW_c. The same substitution into the reference trajectory gives ϕj=hAja+h2E2,jBa+h3[E3,jB2a+RjH[a,a]]+O(h4), _j=hA_ja+h^2E_2,jBa+h^3 [E_3,jB^2a+R_jH[a,a] ]+O(h^4), where the coefficients satisfy Aj+1 A_j+1 =Aj+Wj, =A_j+W_j, E2,j+1 E_2,j+1 =E2,j+WjAj, =E_2,j+W_jA_j, E3,j+1 E_3,j+1 =E3,j+WjE2,j, =E_3,j+W_jE_2,j, Rj+1 R_j+1 =Rj+12WjAj2. =R_j+ 12W_jA_j^2. All four coefficients start at zero. If eme_m denotes the m-th elementary symmetric polynomial in W0,…,Wk−1W_0,…,W_k-1, then ϕk=he1a+h2e2Ba+h3[e3B2a+12∑jWjAj2H[a,a]]+O(h4). _k=he_1a+h^2e_2Ba+h^3 [e_3B^2a+ 12 _jW_jA_j^2H[a,a] ]+O(h^4). All remainder bounds above are uniform because k is fixed and the derivative bounds hold throughout the class. To compute the corresponding coefficients for QAM, expand its generating polynomial at t=1t=1: ∑iqi(1+s)i=∏j(1+Wjs). _iq_i(1+s)^i= _j(1+W_js). Comparing coefficients of sms^m gives ∑iqi(im)=em. _iq_i im=e_m. Consequently, Qh(L)=he1a+h2e2Ba+h3[e3B2a+(e3+12e2)H[a,a]]+O(h4).Q_h(L)=he_1a+h^2e_2Ba+h^3 [e_3B^2a+ (e_3+ 12e_2 )H[a,a] ]+O(h^4). Also, ∑jWjAj2 _jW_jA_j^2 =∑jWj(∑ℓ<jWℓ2+2∑a<b<jWaWb) = _jW_j ( _ <jW_ ^2+2 _a<b<jW_aW_b ) =∑ℓ<jWjWℓ2+2e3. = _ <jW_jW_ ^2+2e_3. Subtracting the reference expansion therefore yields Qh(L)−ϕk(L)=12h3SWH[a,a]+Ok,M(h4).Q_h(L)- _k(L)= 12h^3S_WH[a,a]+O_k,M(h^4). Since ‖H[a,a]‖≤9M\|H[a,a]\|≤ 9M, this proves the uniform O(h3)O(h^3) upper bound. Two losses with the same checkpoint record. The lower bound uses a perturbation whose value and gradient vanish on the ordinary trajectory, but its gradient does not vanish on the reference trajectory. Consider r(x,y)=y−x2−2x.r(x,y)=y-x^2-2x. Choose a fixed smooth cutoff χ∈Cc∞(ℝ2)χ∈ C_c^∞(R^2) that equals one on the unit ball and zero outside the ball of radius two. Set P=12χr2,K=max1,‖D2P‖∞,‖D3P‖∞,‖D4P‖∞,ε=min1/2,MK.P= 12χ r^2, K= \1,\|D^2P\|_∞,\|D^3P\|_∞,\|D^4P\|_∞ \, = \1/2,M\K. Both P and ε>0 >0 are independent of h. K is evidently finite since P is compactly supported. For 0≤h≤1/20≤ h≤ 1/2, define Lh(0)(x,y)=12(1+x)2+12(2−h)(1+y)2,Lh(1)=Lh(0)+εP.L_h^(0)(x,y)= 12(1+x)^2+ 12(2-h)(1+y)^2, L_h^(1)=L_h^(0)+ P. We check that both losses belong to the same fixed class ℒML_M. The Hessian of Lh(0)L_h^(0) is diag(1,2−h)diag(1,2-h), which lies between I and 2I2I. Moreover, ‖εD2P‖∞≤12,‖εD3P‖∞,‖εD4P‖∞≤M.\| D^2P\|_∞≤ 12, \| D^3P\|_∞,\| D^4P\|_∞≤ M. These bounds give the required Hessian and higher-derivative bounds for Lh(1)L_h^(1) everywhere on ℝ2R^2. Finally, since ∇P(0)=0∇ P(0)=0, we have ‖∇Lh(b)(0)‖=1+(2−h)2<3,b∈0,1.\|∇ L_h^(b)(0)\|= 1+(2-h)^2<3, b∈\0,1\. Ordinary GD on Lh(0)L_h^(0) gives the exact trajectory θi=((1−h)i−1,(1−h)2i−1). _i= ((1-h)^i-1,\,(1-h)^2i-1 ). Thus r(θi)=0r( _i)=0 for every i. On the entire set r=0\r=0\, we have P=0,∇P=12r2∇χ+χr∇r=0.P=0, ∇ P= 12r^2∇χ+χ r∇ r=0. The two losses therefore have the same gradient at every ordinary iterate. Starting from the same initial state, they generate exactly the same checkpoint record by induction. Their loss values and gradients also agree at every recorded checkpoint, including θk _k. Separation of the reference endpoints. The uniform state bound proved above applies to both losses. For sufficiently small h, both reference trajectories lie in the unit ball, where χ=1χ=1. Their negative gradients there are fh(x,y)=(−(1+x),(−2+h)(1+y)),gh=fh−εG,G=r∇r.f_h(x,y)= (-(1+x),\,(-2+h)(1+y) ), g_h=f_h- G, G=r∇ r. First consider the reference trajectory for Lh(0)L_h^(0). Write zj=ϕj(Lh(0))=(xj,yj),Xj=1+xj,Yj=1+yj.z_j= _j(L_h^(0))=(x_j,y_j), X_j=1+x_j, Y_j=1+y_j. The updates give Xj+1=(1−hWj)Xj,Yj+1=(1−2hWj+h2Wj)Yj.X_j+1=(1-hW_j)X_j, Y_j+1=(1-2hW_j+h^2W_j)Y_j. Since r(zj)=Yj−Xj2r(z_j)=Y_j-X_j^2, the residual rj=r(zj)r_j=r(z_j) satisfies the exact recurrence rj+1=(1−2hWj+h2Wj)rj+h2Wj(1−Wj)Xj2.r_j+1=(1-2hW_j+h^2W_j)r_j+h^2W_j(1-W_j)X_j^2. Here Xj=1+O(h)X_j=1+O(h) and r0=0r_0=0. Induction yields rj=h2Vj+O(h3),Vj=∑ℓ<jWℓ(1−Wℓ).r_j=h^2V_j+O(h^3), V_j= _ <jW_ (1-W_ ). Also noting that, ∇r(zj)=(−2Xj,1)=(−2,1)+O(h),∇ r(z_j)=(-2X_j,1)=(-2,1)+O(h), and hence G(zj)=h2Vj(−2,1)+O(h3).G(z_j)=h^2V_j(-2,1)+O(h^3). Now let δj=ϕj(Lh(1))−ϕj(Lh(0)),Dh=diag(−1,−2+h). _j= _j(L_h^(1))- _j(L_h^(0)), D_h=diag(-1,-2+h). Since fhf_h is affine, subtracting the two reference updates gives the exact recurrence δj+1=(I+hWjDh)δj−εhWjG(zj+δj),δ0=0. _j+1=(I+hW_jD_h) _j- hW_jG(z_j+ _j), _0=0. Both trajectories lie in the unit ball, where G has a fixed Lipschitz bound. Since G(zj)=O(h2)G(z_j)=O(h^2), this recurrence implies ‖δj+1‖≤(1+Ch)‖δj‖+Ch3\| _j+1\|≤(1+Ch)\| _j\|+Ch^3 for a constant C independent of h. Iterating for the fixed number k of steps gives δj=O(h3) _j=O(h^3). We can now identify the leading term. The bound on δj _j gives hWjDhδj=O(h4),hWj[G(zj+δj)−G(zj)]=O(h4).hW_jD_h _j=O(h^4), hW_j [G(z_j+ _j)-G(z_j) ]=O(h^4). Thus the difference recurrence reduces to δj+1 _j+1 =δj−εhWjG(zj)+O(h4) = _j- hW_jG(z_j)+O(h^4) =δj+εh3WjVj(2,−1)+O(h4). = _j+ h^3W_jV_j(2,-1)+O(h^4). Summing over j, and using ∑jWjVj=∑ℓ<jWjWℓ(1−Wℓ)=SW, _jW_jV_j= _ <jW_jW_ (1-W_ )=S_W, we obtain ϕk(Lh(1))−ϕk(Lh(0))=εSWh3(2,−1)+O(h4). _k(L_h^(1))- _k(L_h^(0))= S_Wh^3(2,-1)+O(h^4). Throughout computations, every remainder constant is independent of h, and the number of steps is fixed. From indistinguishability to the lower bound. Any deterministic algorithm returns the same output on the two identical records. By the triangle inequality, at least one of its two errors is at least half the distance between the targets. Therefore, supL∈ℒM‖A(Th(L),h,W)−ϕk(L)‖ _L _M\|A(T_h(L),h,W)- _k(L)\| ≥12‖ϕk(Lh(1))−ϕk(Lh(0))‖ ≥ 12\| _k(L_h^(1))- _k(L_h^(0))\| ≥ε5SW2h3−Ch4. ≥ 5\,S_W2h^3-Ch^4. Taking the infimum over algorithms proves the lower bound. For example, after reducing h0h_0, one may take c=ε5SW/4>0c= 5\,S_W/4>0. The same argument applies when checkpoint losses and gradients are included in the record, since those observations also agree. For a randomized algorithm, the common record induces the same output distribution for both losses. If Z has that distribution, then maxb∈0,1‖Z−ϕk(Lh(b))‖≥12‖ϕk(Lh(1))−ϕk(Lh(0))‖. _b∈\0,1\E\|Z- _k(L_h^(b))\|≥ 12\| _k(L_h^(1))- _k(L_h^(0))\|. Thus the lower bound also holds for worst-case expected norm error. ∎ Remark 8. The loss class ℒML_M is fixed independently of h. For each h, the supremum over that class may select a different pair Lh(0),Lh(1)L_h^(0),L_h^(1). The theorem therefore gives a uniform worst-case lower bound, rather than a pointwise lower bound for a single fixed loss. If SW=0S_W=0, every positive weight that precedes another positive weight must equal one. After zero weights are removed, the profile consists of m ones followed by at most one fractional weight α∈(0,1)α∈(0,1). With no fractional weight, the reference equals θm _m. Otherwise, it equals (1−α)θm+αθm+1.(1-α) _m+α _m+1. QAM reconstructs these references exactly. The all-zero profile returns θ0=0 _0=0. If the derivative bound is instead set to M=0M=0, every loss in the class is quadratic and QAM is exact. Remark 9. The theorem holds for fixed k,Wk,W, and M>0M>0 as h→0h→ 0. Allowing these quantities to vary with h requires separate control of the constants. The conclusion concerns the optimal order of parameter reconstruction error. It does not establish an optimal leading constant, a downstream performance ordering, or a guarantee of reproducing decay training. A.3 The sharp nonlinear limit of fixed checkpoint weights For fixed Euler steps, a more specific coefficient obstruction is: Theorem 10 (Nonlinear order barrier). Fix k,Wk,W and let f range over C3C^3 vector fields with bounded local derivatives. If SW>0S_W>0, no real coefficients pi(W)p_i(W) independent of h,fh,f and the initial state satisfy ∑ipiθi−ϕkW=O(h4) _ip_i _i- _k^W=O(h^4) for all such Euler trajectories. QAM has the leading remainder θQAM−ϕkW=12h3SWD2f(θ0)[f(θ0),f(θ0)]+O(h4). _QAM- _k^W= 12h^3S_WD^2f( _0)[f( _0),f( _0)]+O(h^4). (19) If SW=0S_W=0, QAM matches the reference exactly for every f. The following calculation uses the classical elementary-differential expansion underlying numerical order conditions (Butcher, 1963). The target here is a prescribed sequence of relaxed Euler updates, rather than the exact flow of a differential equation. Fix k,W,xk,W,x, let f∈C3f∈ C^3 on a neighborhood of x, with bounded derivatives there, and set f0=f(x)f_0=f(x), J=Df(x)J=Df(x), K=D2f(x)K=D^2f(x). For sufficiently small h, both finite trajectories remain in that neighborhood. Write CW=∑a<b<cWaWbWc,DW=∑ℓ<jWjWℓ2,SW=BW−DW=∑ℓ<jWjWℓ(1−Wℓ).C_W= _a<b<cW_aW_bW_c, D_W= _ <jW_jW_ ^2, S_W=B_W-D_W= _ <jW_jW_ (1-W_ ). Taylor expansion gives the following third-order formulas: θi= _i= x+ihf0+h2(i2)Jf0 x+ihf_0+h^2 i2Jf_0 +h3[(i3)J2f0+((i3)+12(i2))K[f0,f0]]+O(h4), +h^3 [ i3J^2f_0+ ( i3+ 12 i2 )K[f_0,f_0] ]+O(h^4), (20) ϕkW= _k^W= x+hAWf0+h2BWJf0 x+hA_Wf_0+h^2B_WJf_0 +h3[CWJ2f0+(CW+12DW)K[f0,f0]]+O(h4). +h^3 [C_WJ^2f_0+ (C_W+ 12D_W )K[f_0,f_0] ]+O(h^4). (21) To verify the first identity, the chain coefficient obeys ui+1=ui+(i2)u_i+1=u_i+ i2 and hence ui=(i3)u_i= i3. The branched coefficient obeys vi+1=vi+i2/2v_i+1=v_i+i^2/2 and hence vi=i(i−1)(2i−1)/12=(i3)+12(i2)v_i=i(i-1)(2i-1)/12= i3+ 12 i2. For the reference, the chain coefficient is ∑jWjBj=CW _jW_jB_j=C_W. Its branched coefficient is 12∑jWjAj2=CW+DW/2 12 _jW_jA_j^2=C_W+D_W/2, where Aj=∑ℓ<jWℓA_j= _ <jW_ and Bj=∑a<b<jWaWbB_j= _a<b<jW_aW_b. For any normalized real coefficients p satisfying the two second-order conditions, put M3(p)=∑ipi(i3)M_3(p)= _ip_i i3. Subtracting the preceding expansions yields ∑ipiθi−ϕkW=h3[(M3(p)−CW)J2f0+(M3(p)−CW+12SW)K[f0,f0]]+O(h4). _ip_i _i- _k^W=h^3 [(M_3(p)-C_W)J^2f_0+ (M_3(p)-C_W+ 12S_W )K[f_0,f_0] ]+O(h^4). (22) Thus a single free third factorial moment affects both elementary differentials by the same amount. Their required values differ by SW/2S_W/2. Proof of Theorem 10. Universal O(h4)O(h^4) matching first forces normalization and the two lower-order moments. Normalization follows by taking f=0f=0 and a nonzero initial state. The scalar fields f(x)=1f(x)=1 and f(x)=xf(x)=x force the first and second moments; the latter, initialized at x=1x=1, also forces M3(p)=CWM_3(p)=C_W. Taking f(x)=1+x2/2f(x)=1+x^2/2 at x=0x=0 then gives J=0J=0 and K[f0,f0]=1K[f_0,f_0]=1, so Eq. (22) forces SW=0S_W=0. This argument applies even to signed coefficients. Conversely, the generating polynomial gives M3(q)=QW′(1)/6=CWM_3(q)=Q_W (1)/6=C_W, and substitution gives Eq. (19). Each summand in SWS_W is nonnegative. Therefore SW=0S_W=0 if and only if every positive entry that precedes a later positive entry is one. After zero entries are removed, the profile is consequently a string of m ones followed by at most one fractional entry w. The reference then takes m ordinary steps and one final relaxation, so ϕkW=(1−w)θm+wθm+1 _k^W=(1-w) _m+w _m+1 for every f and every step size for which the iterates exist. QAM has precisely these two coefficients; if no fractional entry occurs, it selects θm _m. For a nonincreasing profile this is exactly the pattern (1,…,1,w,0,…,0)(1,…,1,w,0,…,0). This proves the sharp exception. ∎ As an exact two-update illustration, take k=2k=2, x=0x=0 and f(x)=1+x2/2f(x)=1+x^2/2. The ordinary states are 0,h,2h+h3/20,h,2h+h^3/2, and θQAM−ϕ2W=12h3W0W1(1−W0). _ QAM- _2^W= 12h^3W_0W_1(1-W_0). The discrepancy already occurs for a one-dimensional gradient field, since f=−L′f=-L for L(x)=−x−x3/6L(x)=-x-x^3/6. This calculation concerns fixed linear weights. Theorem 4 establishes a stronger uniform lower bound for arbitrary reconstruction algorithms with the same checkpoint information; additional off-trajectory gradient queries change that information model. A.4 Consecutive GD as a specialization For a fixed objective L, the ordinary and reference GD trajectories are θi+1=θi−η∇L(θi),i=0,…,k−1. _i+1= _i-η∇ L( _i), i=0,…,k-1. (23) ϕ0W=θ0,ϕj+1W=ϕjW−ηWj∇L(ϕjW),j=0,…,k−1. _0^W= _0, _j+1^W= _j^W-η W_j∇ L( _j^W), j=0,…,k-1. (24) Assumption 11 (Fixed smooth objective and finite GD trajectories). The objective L is fixed and belongs to C3C^3 on a neighborhood of θ0 _0, with locally bounded derivatives through order three. Both trajectories follow Eqs. (23) and (24). The initial state, k, and W remain fixed as η→0η→ 0. Theorem 12 (Characterization of local second-order consistency). Fix k≥1k≥ 1, W∈[0,1]kW∈[0,1]^k, and a convex coefficient vector p=(p0,…,pk)p=(p_0,…,p_k) independent of η, L, and θ0 _0. The bound ∑i=0kpiθi−ϕkW=O(η3) _i=0^kp_i _i- _k^W=O(η^3) holds as η→0η→ 0 for every fixed objective and initial state satisfying Assumption 11 if and only if p[I]=AW,p[(I2)]=BW.E_p[I]=A_W, _p\! [ I2 ]=B_W. (25) Here Prp(I=i)=pi _p(I=i)=p_i. Equivalently, p has the same checkpoint-index mean and variance as q. In particular, θQAM−ϕkW=O(η3). _QAM- _k^W=O(η^3). (26) This is the h=ηh=η, f=−∇Lf=-∇ L, a=0a=0 case of Theorem 2. A.5 Proof of Theorem 5 A.5.1 Exactness by composing quadratic GD updates Proof of exactness. Fix k≥1k≥ 1, W∈[0,1]kW∈[0,1]^k, η>0η>0, and a quadratic objective as in Eq. (11). Its gradient is ∇L(θ0+u)=g0+H0u∇ L( _0+u)=g_0+H_0u for every u, with arbitrary g0g_0 and symmetric H0H_0. In displacement coordinates u=θ−θ0u=θ- _0, introduce ℳη=(I−ηH0−ηg001).M_η= pmatrixI-η H_0&-η g_0\\ 0&1 pmatrix. (27) Ordinary GD acts on (u,1)⊤(u,1) through ℳηM_η; a reference step with learning rate ηWjη W_j acts through (I−ηWjH0−ηWjg001)=(1−Wj)I+Wjℳη. pmatrixI-η W_jH_0&-η W_jg_0\\ 0&1 pmatrix=(1-W_j)I+W_jM_η. These matrices are polynomials in the same matrix and hence commute. The generating polynomial in Eq. (7) therefore gives the exact matrix identity ∏j=0k−1[(1−Wj)I+Wjℳη]=QW(ℳη)=∑i=0kqiℳηi. _j=0^k-1 [(1-W_j)I+W_jM_η ]=Q_W(M_η)= _i=0^kq_iM_η^i. (28) Both trajectories start from θ0 _0, so applying this identity to (0,1)⊤(0,1) yields (ϕkW−θ01)=∑i=0kqi(θi−θ01). pmatrix _k^W- _0\\ 1 pmatrix= _i=0^kq_i pmatrix _i- _0\\ 1 pmatrix. Since ∑iqi=QW(1)=1 _iq_i=Q_W(1)=1, the upper coordinates prove Eq. (12). The term −ηg0-η g_0 retains the linear part of the quadratic objective throughout the calculation. No stationary point or inverse of H0H_0 is needed: H0H_0 may be singular or indefinite. This finite algebraic identity requires neither a small step size nor convergence of the GD trajectory. ∎ A.5.2 Uniqueness across quadratic objectives Proof of uniqueness. Fix k, W, and η>0η>0, and suppose that the same real coefficient vector p(W)p(W) satisfies the endpoint identity for every quadratic loss and initial state. Keep this vector fixed while varying the one-dimensional convex losses Lλ(x)=12λx2,x0=1,0<λ<1/η.L_λ(x)= 12λ x^2, x_0=1, 0<λ<1/η. Writing t=1−ηλt=1-ηλ gives xi=tix_i=t^i and ϕkW=∏j=0k−1(1−ηWjλ)=∏j=0k−1(1−Wj+Wjt)=QW(t). _k^W= _j=0^k-1(1-η W_jλ)= _j=0^k-1(1-W_j+W_jt)=Q_W(t). Because t ranges over (0,1)(0,1), exact matching requires ∑i=0kpi(W)ti=QW(t)for every t∈(0,1). _i=0^kp_i(W)t^i=Q_W(t) every t∈(0,1). Two polynomials agreeing on this set have identical coefficients, so pi(W)=qi(W)p_i(W)=q_i(W) for all i. No convexity or normalization assumption on p is needed. If the dimension is fixed above one, the same scalar example embeds in a single coordinate. ∎ The uniqueness requirement is universal across quadratic objectives. A.6 Proof of Theorem 12 and the variance identity Proof of Theorem 12. Under Assumption 11, the GD map satisfies Assumption 1 with h=ηh=η, f=−∇Lf=-∇ L, and a=0a=0. The expansions in Appendix A.1 therefore give sufficiency and QAM’s local error bound. The base-discrepancy formula follows from Eq. (6) under the same substitution. For necessity, L(x)=xL(x)=x at x0=0x_0=0 forces the first moment condition. With that condition satisfied, L(x)=x2/2L(x)=x^2/2 at x0=1x_0=1 forces the second. These examples embed in one coordinate in any higher dimension. The equivalent mean-and-variance condition follows from the moment identity in Appendix A.1. ∎ Scope of the necessity statement. The necessity statement concerns one fixed coefficient rule across objectives, and a particular trajectory may admit other matching weights. As noted in Section 3.3, second-order consistency alone leaves a (k−2)(k-2)-dimensional family of convex solutions when k≥3k≥ 3 and Wj∈(0,1)W_j∈(0,1). Theorem 5 selects q by the stronger requirement of exact matching across quadratics. Proof of Eq. (8). This calculation uses only the coefficients c,qc,q and their common input profile W. For Ic∼cI_c c and Iq∼qI_q q, the coefficient sums in Appendix A.1 give the common mean [Ic]=[Iq]=AWE[I_c]=E[I_q]=A_W. Consequently, Var(Ic)−Var(Iq) (I_c)-Var(I_q) =2(νc−νq) =2( _c- _q) =2∑ℓ<j(Wj−WℓWj)=2Δ2(W)≥0. =2 _ <j(W_j-W_ W_j)=2 _2(W)≥ 0. The inequality follows from 0≤Wj≤10≤ W_j≤ 1. ∎ Appendix B Evaluation Benchmarks Main-text Table 1 gives the full checkpoint ranges and storage intervals. A Last-n window uses the last n chronological checkpoints and therefore spans n−1n-1 intervals. The SmolLM3 checkpoint steps are 3,640,000+40,000i3,640,000+40,000i, i=0,…,14i=0,…,14. The Prelude steps are 859,200+2,400i859,200+2,400i, i=0,…,39i=0,…,39. These progressions specify the complete checkpoint grids. Table 5 specifies all 15 tasks. Accuracy tasks use lm_eval v0.4.13. MMLU and GSM8K use 5-shot evaluation, and the other accuracy tasks use zero-shot evaluation. Capability-group means weight accuracy tasks by evaluation-set size. Math averages the flexible and strict GSM8K extraction scores. MuTual uses MRR. The available matched-kernel protocols are in Appendix F. Table 5: Capability groups. Group Tasks Metric Knowledge ARC-Easy, ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), SciQ (Welbl et al., 2017), MMLU (5-shot) (Hendrycks et al., 2021) acc. Commonsense HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), COPA (Gordon et al., 2012), WinoGrande (Sakaguchi et al., 2020) acc. Reading BoolQ (Clark et al., 2019) acc. Math GSM8K flexible, GSM8K strict (Cobbe et al., 2021) exact match Semantics SST-2 (Socher et al., 2013), WiC (Pilehvar and Camacho-Collados, 2019), WSC (Levesque et al., 2012) acc. Dialogue MuTual (Cui et al., 2020) MRR For each trajectory, window, and profile, QAM and WSM merge the same checkpoints. Merges are accumulated in fp32 and exported in bf16, with no additional training, calibration, or prompt tuning. Appendix C Checkpoint Weights Figure 5 shows the WSM and QAM weights defined in Eqs. (2) and (7) across the window lengths and profiles used in our experiments. Figure 5: WSM and QAM checkpoint weights. Rows correspond to windows containing n=5,10,15,20,40n=5,10,15,20,40 checkpoints, and columns correspond to the three prescribed profiles. Checkpoint indices increase from oldest to newest. Appendix D Moment identities for the interpolation diagnostic For a coefficient distribution p, let μ(p)μ(p) and Var(p)Var(p) denote the mean and variance of its checkpoint index. Appendix A.1 gives μ(c)=μ(q)=AWμ(c)=μ(q)=A_W. Consider the interpolation used in Section 5, p(λ)=(1−λ)c+λq,0≤λ≤1.p^(λ)=(1-λ)c+λ q, 0≤λ≤ 1. Its mean is independent of λ: μ(p(λ))=(1−λ)μ(c)+λμ(q)=AW.μ(p^(λ))=(1-λ)μ(c)+λμ(q)=A_W. Writing μ=AWμ=A_W, we obtain Var(p(λ)) (p^(λ)) =∑i=0k(i−μ)2[(1−λ)ci+λqi] = _i=0^k(i-μ)^2 [(1-λ)c_i+λ q_i ] =(1−λ)Var(c)+λVar(q) =(1-λ)Var(c)+ (q) =Var(c)−2λΔ2(W), =Var(c)-2λ _2(W), where the last equality follows from Eq. (8). Thus the interpolation preserves the mean checkpoint index. Its variance decreases linearly when Δ2(W)>0 _2(W)>0 and remains constant when Δ2(W)=0 _2(W)=0. Appendix E Generation and Termination Diagnostics We report output-level diagnostics on Prelude Last-40 and SmolLM3 Last-15. They describe generation behavior on the tested models. E.1 Prelude Last-40 We first examine the unusually low Prelude Last-40 WSM GSM8K endpoint. The WSM endpoint exhibits abnormal generation behavior. Only a small fraction of outputs contain the expected final-answer delimiter, while many generations lose the normal line-by-line solution structure and enter repeated 10-gram patterns. The median output length is also much larger than at the paired QAM endpoint, indicating that the model often continues generating instead of terminating with a concise final answer. These observations show that the degradation of WSM at long windows is expressed not only through final-answer correctness. Recall the interpolation path defined in Section 5. As λ increases toward the QAM endpoint, these symptoms are progressively reduced. The generations recover normal answer formatting and line structure, repeated text becomes much less frequent, and the output length returns to the range typical of well-formed GSM8K solutions. The diagnostics in Table 6 quantify these output-level symptoms along the interpolation path. Table 6: Output diagnostics along the linear Prelude Last-40 interpolation path on GSM8K. The delimiter column records the fraction of generations containing the standard GSM8K answer delimiter. The no-newline column records outputs without a newline. The loop column records repeated 10-gram behavior. Median length is measured in characters. Model Delimiter No newline Loop Median length λ=0λ=0 13.7 83.3 82.8 1027 λ=1/2λ=1/2 34.3 57.2 60.0 840 λ=3/4λ=3/4 79.1 11.0 14.8 374 λ=1λ=1 91.0 0.3 4.7 298 The following are examples of WSM failures, alongside successful solutions produced by QAM on the same problems. Example 1 (doc_id 679; gold 576). If 6 potatoes makes 36 hash browns, how many hash browns can you make out of 96 potatoes? WSM repeats the premise verbatim until truncation and never performs the division; 6 potatoes makes 36 hash browns 6 potatoes makes 36 hash browns ... (28 identical lines, truncated at the 256-token limit) QAM: 6 potatoes makes 36 hash browns So 1 potato makes 36/6=<<36/6=6>>6 96 potatoes makes 96*6=<<96*6=576>>576 #### 576 Example 2 (doc_id 164; gold 15). Jackie is trying to decide whether to do her taxes herself or hire an accountant. If she does the taxes herself, she’l be able to do 3 fewer hours of freelance work, losing $35/hour in missed income. The accountant charges $90. How much more money will she have if she hires the accountant? In this example, WSM computes the first multiplication correctly, but then repeats the intermediate step until truncation instead of subtracting the accountant’s fee. If she does the taxes herself, she’l be able to do 3 fewer hours of freelance work, losing $35/hour in missed income. If she does the taxes herself, she’l be able to do 3*35=<<3*35=105>>105 hours of freelance work. ... (the same line repeated 7 times, truncated) QAM reaches the same intermediate value and completes the derivation: ... she’l lose 3*35=<<3*35=105>>105 dollars. The accountant charges $90. So, she’l have 105-90=<<105-90=15>>15 dollars more if she hires the accountant. #### 15 Example 3 (doc_id 1207; gold 64). Deandre caught 3 tunas last Monday, the first tuna he caught weighs 56 kilograms, the second tuna he caught weighs 46 kilograms, and the last tuna he caught weighs 26 kilograms. If a kilogram of tuna costs $0.50, how much will he earn after selling all the tunas to the market? WSM echoes the question as a single sentence six and a half times before truncation, and flexible-extract returns the item count 3. QAM prices each fish (56×0.5=2856× 0.5=28, 46×0.5=2346× 0.5=23, 26×0.5=1326× 0.5=13) and sums them to #### 64. E.2 SmolLM3 Last-15 Table 7 summarizes the interpolation endpoint diagnostics. Unlike Prelude, all three SmolLM3 WSM endpoints have low no-newline rates and median generation lengths close to those of QAM. Table 7: SmolLM3 Last-15 diagnostics on the 1,319-question GSM8K test set. Profile Method Flexible Strict No newline Loop Median length Linear WSM 33.4 30.6 0.8 9.9 213 Linear QAM 43.4 43.2 0.4 3.6 221 Cosine WSM 37.1 35.3 0.2 5.9 233 Cosine QAM 45.4 44.8 0.4 2.9 252 1−⋅1- · WSM 32.8 29.9 1.4 11.7 218 1−⋅1- · QAM 41.5 41.0 0.4 3.6 231 Appendix F Maximum-Entropy Diagnostics These diagnostics compare coefficient vectors at specified index moments on the same saved checkpoint grid. Their purpose is to test whether those moments determine downstream scores. F.1 Maximum-entropy controls along the interpolation Use zero-based indices i=0,…,n−1i=0,…,n-1, with n=15n=15 for SmolLM3 and n=40n=40 for Prelude. For each of the three profiles and λ∈0.25,0.50,0.75λ∈\0.25,0.50,0.75\, form p(λ)=(1−λ)c+λqp^(λ)=(1-λ)c+λ q and set μ=∑ipi(λ),v=(1−λ)Varc(I)+λVarq(I).μ= _iip_i^(λ), v=(1-λ)Var_c(I)+ _q(I). The control is the maximum-entropy distribution on this finite grid subject to rI=μE_rI=μ and Varr(I)=vVar_r(I)=v: ri=exp(ai+bi2)∑j=0n−1exp(aj+bj2).r_i= (ai+bi^2) _j=0^n-1 (aj+bj^2). The parameters enforce the two constraints; equivalently, they minimize the convex function log∑i=0n−1exp(ai+bi2)−aμ−b(μ2+v). \! _i=0^n-1 (ai+bi^2)-aμ-b(μ^2+v). The gradient gives the residuals in the first and second raw moments, so this specifies the distribution without task-score fitting. The target variance is that of the intermediate mixture, generally larger than the QAM variance at λ=1λ=1. On Prelude, the target means in one-based indexing are 20.500020.5000, 25.961525.9615, and 13.865213.8652 for linear, cosine, and 1−⋅1- ·; subtract one for the zero-based indices above. Merges use float32 accumulation and bfloat16 export. Table 4 reports all paired observations. On SmolLM3, MaxEnt wins six of nine flexible-extraction pairs; under strict extraction it wins four, loses four, and ties one. On Prelude, the mixture wins seven of nine flexible and six of nine strict pairs. The target moments therefore do not specify a unique task score or ranking. This comparison is a diagnostic, not a causal isolation of the theoretical truncation error. F.2 Separate MaxEnt controls at the QAM endpoint For the linear profile on SmolLM3 Last-15 and Prelude Last-40, separate MaxEnt merges match QAM’s index moments. With zero-based indices, the target pairs (μ,v)(μ,v) are respectively (7,11245)and(19.5,53380). (7, 11245 ) (19.5, 53380 ). Each finite-grid MaxEnt distribution uses the same constrained family defined above with the corresponding target moments, and therefore satisfies the two-moment conditions in Theorem 2. Table 8 reports their five-shot GSM8K scores alongside QAM. The plotted interpolation endpoint at λ=1λ=1 is QAM itself. The MaxEnt models are separate controls, not additional points on that interpolation segment. The comparison only covers this linear profile and benchmark GSM8K. Table 8: Five-shot GSM8K scores for linear QAM and moment-matched MaxEnt controls. SmolLM3 Last-15 Prelude Last-40 Kernel Flex Strict Flex Strict QAM 43.37 43.21 45.41 44.66 MaxEnt 43.75 43.52 46.10 45.34 Appendix G Additional Performance Comparisons G.1 Comparison with Mean Constituent Scores As a complementary comparison, we evaluate QAM against the average score of the individual checkpoints in each window. As shown in Tables 9 and 10, across two trajectories, three windows, three profiles, and six capability groups, QAM exceeds this reference in 105 of 108 comparisons. Single-checkpoint averages are reported to two decimal places. Table 9: SmolLM3 QAM compared with the average single checkpoint. Model Window Profile Knowledge Commonsense Reading Math Semantics Dialogue SmolLM3 Last-5 Single-checkpoint average 58.27 58.64 76.71 34.33 63.48 69.68 Linear 60.90 (+2.63) 59.86 (+1.22) 78.75 (+2.04) 41.78 (+7.45) 68.09 (+4.61) 70.34 (+0.66) Cosine 60.98 (+2.71) 59.85 (+1.21) 78.07 (+1.36) 41.21 (+6.88) 66.30 (+2.82) 70.79 (+1.11) 1−⋅1- · 60.92 (+2.65) 59.85 (+1.21) 79.27 (+2.56) 40.30 (+5.97) 68.77 (+5.29) 69.90 (+0.22) Last-10 Single-checkpoint average 58.34 58.59 76.20 34.09 62.98 69.68 Linear 61.24 (+2.90) 59.69 (+1.10) 79.63 (+3.43) 43.71 (+9.62) 68.40 (+5.42) 69.56 (-0.12) Cosine 61.10 (+2.76) 59.78 (+1.19) 79.42 (+3.22) 41.17 (+7.08) 69.14 (+6.16) 69.81 (+0.13) 1−⋅1- · 61.16 (+2.82) 59.77 (+1.18) 78.65 (+2.45) 43.78 (+9.69) 67.91 (+4.93) 69.96 (+0.28) Last-15 Single-checkpoint average 58.11 58.53 76.63 33.37 62.23 69.66 Linear 61.08 (+2.97) 59.53 (+1.00) 76.51 (-0.12) 43.29 (+9.92) 68.46 (+6.23) 69.63 (-0.03) Cosine 61.41 (+3.30) 59.83 (+1.30) 79.97 (+3.34) 45.11 (+11.74) 68.77 (+6.54) 69.72 (+0.06) 1−⋅1- · 60.97 (+2.86) 59.51 (+0.98) 77.65 (+1.02) 41.24 (+7.87) 67.16 (+4.93) 69.95 (+0.29) Table 10: OpenEuroLLM Prelude QAM compared with the average single checkpoint. Model Window Profile Knowledge Commonsense Reading Math Semantics Dialogue OpenEuroLLM Last-10 Single-checkpoint average 61.16 61.94 79.52 33.16 57.56 71.65 Linear 64.55 (+3.39) 63.44 (+1.50) 81.87 (+2.35) 42.80 (+9.64) 64.19 (+6.63) 72.75 (+1.10) Cosine 63.86 (+2.70) 63.22 (+1.28) 81.41 (+1.89) 41.43 (+8.27) 61.53 (+3.97) 72.04 (+0.39) 1−⋅1- · 64.19 (+3.03) 63.42 (+1.48) 82.32 (+2.80) 42.65 (+9.49) 60.22 (+2.66) 72.42 (+0.77) Last-20 Single-checkpoint average 61.09 61.94 79.66 33.37 58.65 71.42 Linear 64.06 (+2.97) 63.77 (+1.83) 82.97 (+3.31) 44.47 (+11.10) 65.74 (+7.09) 72.70 (+1.28) Cosine 64.46 (+3.37) 63.53 (+1.59) 82.66 (+3.00) 44.13 (+10.76) 63.81 (+5.16) 72.50 (+1.08) 1−⋅1- · 64.24 (+3.15) 63.65 (+1.71) 82.23 (+2.57) 42.57 (+9.20) 67.97 (+9.32) 72.22 (+0.80) Last-40 Single-checkpoint average 61.05 62.00 79.47 33.57 59.34 71.60 Linear 64.27 (+3.22) 63.41 (+1.41) 82.20 (+2.73) 45.03 (+11.46) 66.73 (+7.39) 72.85 (+1.25) Cosine 64.39 (+3.34) 63.64 (+1.64) 82.29 (+2.82) 44.47 (+10.90) 67.91 (+8.57) 72.25 (+0.65) 1−⋅1- · 64.03 (+2.98) 63.42 (+1.42) 82.78 (+3.31) 42.57 (+9.00) 65.12 (+5.78) 73.31 (+1.71) Notice that the improvements differ by capability in both their size and consistency across configurations. Knowledge and Commonsense show consistent gains across all tested windows and profiles, ranging from 2.63 to 3.39 percentage points and from 0.98 to 1.83, respectively. Semantics also improves in every configuration, although the magnitude varies more widely, from 2.66 to 9.32 points. Math shows particularly large gains on both trajectories. Improvements range from 5.97 to 11.74 percentage points on SmolLM3 and from 8.27 to 11.46 on Prelude, with gains present in every tested configuration. Reading and Dialogue improve in 17 and 16 of the 18 configurations, respectively, with only small observed regressions. All three negative differences occur on SmolLM3, comprising a 0.12-point decrease in Reading and decreases of 0.12 and 0.03 MRR score points in Dialogue. This comparison establishes improvements over the mean constituent score. Appendix H reports direct comparisons with the latest and task-best constituent checkpoints. G.2 Long-window GSM8K scores The large Math gap on Prelude Last-40 reflects a sharp decline in WSM’s GSM8K performance. Table 11 reports the paired raw scores. WSM scores only 11.8311.83–14.0314.03 under flexible extraction and 5.385.38–9.489.48 under strict extraction, whereas QAM scores 42.8442.84–45.4145.41 and 42.3042.30–44.6644.66, respectively. The individual checkpoints in this window score 29.429.4–37.737.7 under flexible extraction. Across all three profiles, WSM falls below every constituent checkpoint, while QAM exceeds all of them under this metric. Table 11: GSM8K scores for Prelude Last-40 under flexible and strict answer extraction. Linear Cosine 1−⋅1- · Metric WSM QAM WSM QAM WSM QAM GSM8K flexible 11.83 45.41 14.03 44.66 12.66 42.84 GSM8K strict 5.38 44.66 9.48 44.28 7.13 42.30 flexible −- strict 6.44 0.76 4.55 0.38 5.53 0.53 G.3 Task-wise best-observed comparisons Table 12 makes the cross-window comparison explicit. Each task is considered separately: WSM may use any of the three windows and three profiles, while QAM may use either the intermediate or long window and any of the three profiles. Different rows can therefore select different merged models. These are best observed scores over the evaluated grid, and different tasks may select different configurations. Table 12: Task-wise best observed scores. WSM uses all tested windows; QAM uses intermediate and long windows. Scores are percentages, except MuTual which is 100×MRR100×MRR. SmolLM3 Prelude Task / metric WSM QAM Δ WSM QAM Δ ARC-Challenge 47.61 48.12 +0.51 50.94 50.68 -0.26 ARC-Easy 79.46 79.84 +0.38 80.60 80.85 +0.25 BoolQ 79.17 79.97 +0.80 83.06 82.97 -0.09 COPA 87.00 88.00 +1.00 92.00 92.00 0.00 HellaSwag 54.89 55.20 +0.31 58.97 59.28 +0.31 OpenBookQA 35.80 34.80 -1.00 35.80 35.60 -0.20 PIQA 78.62 78.35 -0.27 79.76 80.58 +0.82 SciQ 95.60 95.40 -0.20 95.90 96.00 +0.10 SST-2 85.78 85.89 +0.11 84.52 83.94 -0.58 WiC 51.25 50.16 -1.09 50.31 50.78 +0.47 WinoGrande 69.93 69.06 -0.87 73.16 74.27 +1.11 WSC 47.12 47.12 0.00 36.54 42.31 +5.77 MMLU (5-shot) 57.80 57.99 +0.19 62.31 61.88 -0.43 GSM8K flexible 42.23 45.41 +3.18 43.06 45.41 +2.35 GSM8K strict 41.24 44.81 +3.57 42.76 44.66 +1.90 MuTual 70.65 69.96 -0.69 72.79 73.31 +0.52 GSM8K counts as one task as flexible and strict extraction give the same win/tie/loss classification. Including MuTual, the totals are 8 wins, 1 tie, and 6 losses for SmolLM3, and 9 wins, 1 tie, and 5 losses for Prelude. Several positive differences are small; these counts describe direction without establishing statistical significance or a universal advantage. For SmolLM3, the best WSM MuTual score is 70.6570.65 at Last-5 cosine, while the best intermediate/long QAM score is 69.9669.96 at Last-10 1−⋅1- ·. For Prelude, the best WSM score is 72.7972.79 at Last-10 cosine, while the best intermediate/long QAM score is 73.3173.31 at Last-40 1−⋅1- ·. Appendix H Comparison with Individual Checkpoints We compare all 18 QAM configurations with individual checkpoints across the 15 evaluation tasks. Each merge is compared with every checkpoint in its own window, the latest checkpoint, and the highest-scoring checkpoint for each task within that window. The latest checkpoint is shared across windows within each trajectory. The task-wise best checkpoint may differ across tasks. Scores use a 0–100 scale, including MuTual MRR multiplied by 100. GSM8K averages flexible and strict extraction and counts as one task. Wins, ties, and losses are determined after rounding both scores to two decimal places. Table 13 summarizes all three windows and all three profiles for each trajectory. Across this complete grid, QAM wins 1092 of 1350 task–checkpoint comparisons on SmolLM3 (80.9%) and 2716 of 3150 on Prelude (86.2%). Across the 270 task–configuration comparisons, QAM beats the latest checkpoint in 219, ties in 14, and loses in 37. Against the task-wise best checkpoint within each window, it wins 147, ties 5, and loses 118. Tables 14 and 15 report the task scores and constituent-checkpoint comparisons for all 18 configurations. Table 13: QAM versus individual checkpoints, broken down by window and profile. Trajectory Window Profile All Latest Best SmolLM3 Last-5 Linear 68/0/7 14/0/1 12/0/3 SmolLM3 Last-5 Cosine 66/2/7 14/0/1 11/1/3 SmolLM3 Last-5 1−⋅1- · 58/5/12 10/2/3 8/1/6 SmolLM3 Last-10 Linear 116/5/29 10/1/4 7/0/8 SmolLM3 Last-10 Cosine 123/4/23 12/0/3 7/0/8 SmolLM3 Last-10 1−⋅1- · 116/7/27 9/1/5 6/1/8 SmolLM3 Last-15 Linear 174/9/42 10/1/4 6/0/9 SmolLM3 Last-15 Cosine 187/7/31 12/0/3 7/0/8 SmolLM3 Last-15 1−⋅1- · 184/9/32 11/0/4 6/0/9 SmolLM3 total 1092/48/210 102/5/28 70/3/62 Prelude Last-10 Linear 137/3/10 14/1/0 10/0/5 Prelude Last-10 Cosine 128/5/17 12/2/1 8/0/7 Prelude Last-10 1−⋅1- · 134/2/14 13/0/2 9/1/5 Prelude Last-20 Linear 251/19/30 13/1/1 10/0/5 Prelude Last-20 Cosine 257/3/40 13/1/1 9/0/6 Prelude Last-20 1−⋅1- · 261/11/28 13/1/1 8/0/7 Prelude Last-40 Linear 500/41/59 12/2/1 7/1/7 Prelude Last-40 Cosine 548/16/36 13/1/1 9/0/6 Prelude Last-40 1−⋅1- · 500/21/79 14/0/1 7/0/8 Prelude total 2716/121/313 117/9/9 77/2/56 Combined total 3808/169/523 219/14/37 147/5/118 Table 14: SmolLM3 QAM versus individual checkpoints for all windows and profiles. Linear Cosine 1−⋅1- · Task Latest Best QAM W/T/L QAM W/T/L QAM W/T/L Last-5 ARC-Challenge 45.39 46.59 47.01 5/0/0 47.10 5/0/0 47.35 5/0/0 ARC-Easy 77.69 77.78 79.00 5/0/0 79.50 5/0/0 79.21 5/0/0 OpenBookQA 33.60 34.40 36.00 5/0/0 36.40 5/0/0 34.40 4/1/0 SciQ 95.00 95.10 95.40 5/0/0 95.50 5/0/0 95.00 3/1/1 MMLU 54.69 54.98 57.42 5/0/0 57.42 5/0/0 57.48 5/0/0 HellaSwag 53.84 53.93 55.07 5/0/0 55.06 5/0/0 55.13 5/0/0 PIQA 77.80 77.91 78.02 5/0/0 78.07 5/0/0 78.24 5/0/0 COPA 86.00 89.00 87.00 3/0/2 88.00 3/1/1 88.00 3/1/1 WinoGrande 68.35 69.14 69.30 5/0/0 69.14 4/1/0 68.35 2/1/2 BoolQ 77.95 77.95 78.75 5/0/0 78.07 5/0/0 79.27 5/0/0 GSM8K 31.05 36.51 41.78 5/0/0 41.21 5/0/0 40.30 5/0/0 SST-2 65.60 84.63 83.37 4/0/1 80.05 3/0/2 84.86 5/0/0 WiC 50.78 50.78 51.10 5/0/0 51.25 5/0/0 49.84 1/1/3 WSC 67.31 67.31 44.23 1/0/4 43.27 1/0/4 50.00 2/0/3 MuTual 70.33 70.33 70.34 5/0/0 70.79 5/0/0 69.90 3/0/2 Last-10 ARC-Challenge 45.39 46.59 48.04 10/0/0 47.44 10/0/0 47.61 10/0/0 ARC-Easy 77.69 78.24 79.84 10/0/0 79.00 10/0/0 79.71 10/0/0 OpenBookQA 33.60 35.80 34.60 9/0/1 34.80 9/0/1 34.20 7/1/2 SciQ 95.00 95.40 95.20 8/1/1 95.10 7/1/2 95.40 9/1/0 MMLU 54.69 55.96 57.72 10/0/0 57.72 10/0/0 57.67 10/0/0 HellaSwag 53.84 53.94 54.98 10/0/0 55.05 10/0/0 55.20 10/0/0 PIQA 77.80 77.91 78.24 10/0/0 78.02 10/0/0 77.69 8/0/2 COPA 86.00 89.00 86.00 2/3/5 88.00 6/2/2 86.00 2/3/5 WinoGrande 68.35 69.14 68.03 5/0/5 68.59 8/0/2 67.96 5/0/5 BoolQ 77.95 77.95 79.63 10/0/0 79.42 10/0/0 78.65 10/0/0 GSM8K 31.05 36.51 43.71 10/0/0 41.17 10/0/0 43.78 10/0/0 SST-2 65.60 87.16 84.75 9/0/1 85.89 9/0/1 83.37 8/0/2 WiC 50.78 50.78 49.84 4/1/5 49.84 4/1/5 50.16 7/2/1 WSC 67.31 67.31 45.19 4/0/6 47.12 4/0/6 47.12 4/0/6 MuTual 70.33 70.33 69.56 5/0/5 69.81 6/0/4 69.96 6/0/4 Last-15 ARC-Challenge 45.39 46.59 47.18 15/0/0 47.87 15/0/0 48.12 15/0/0 ARC-Easy 77.69 78.24 79.84 15/0/0 79.80 15/0/0 79.71 15/0/0 OpenBookQA 33.60 35.80 34.60 13/0/2 34.60 13/0/2 34.80 13/0/2 SciQ 95.00 95.60 95.10 8/2/5 95.10 8/2/5 95.20 10/2/3 MMLU 54.69 55.96 57.58 15/0/0 57.99 15/0/0 57.36 15/0/0 HellaSwag 53.84 53.94 54.76 15/0/0 55.09 15/0/0 54.58 15/0/0 PIQA 77.80 77.91 77.97 15/0/0 77.97 15/0/0 78.35 15/0/0 COPA 86.00 89.00 86.00 5/3/7 87.00 8/3/4 87.00 8/3/4 WinoGrande 68.35 69.14 68.51 12/1/2 68.98 14/0/1 69.06 14/0/1 BoolQ 77.95 79.20 76.51 6/0/9 79.97 15/0/0 77.65 13/0/2 GSM8K 31.05 36.51 43.29 15/0/0 45.11 15/0/0 41.24 15/0/0 SST-2 65.60 87.16 84.98 14/0/1 85.09 14/0/1 83.14 13/0/2 WiC 50.78 50.78 50.16 11/3/1 50.00 9/2/4 50.16 11/3/1 WSC 67.31 67.31 42.31 6/0/9 47.12 7/0/8 37.50 2/1/12 MuTual 70.33 70.33 69.63 9/0/6 69.72 9/0/6 69.95 10/0/5 Table 15: Prelude QAM versus individual checkpoints for all windows and profiles. Linear Cosine 1−⋅1- · Task Latest Best QAM W/T/L QAM W/T/L QAM W/T/L Last-10 ARC-Challenge 47.61 48.46 50.77 10/0/0 49.23 10/0/0 49.91 10/0/0 ARC-Easy 79.08 79.50 80.68 10/0/0 80.30 10/0/0 80.26 10/0/0 OpenBookQA 31.80 33.20 33.80 10/0/0 35.00 10/0/0 34.80 10/0/0 SciQ 95.50 95.90 95.60 7/0/3 95.50 6/1/3 95.40 5/1/4 MMLU 58.82 58.82 61.86 10/0/0 61.07 10/0/0 61.49 10/0/0 HellaSwag 57.33 58.03 59.16 10/0/0 58.92 10/0/0 59.13 10/0/0 PIQA 78.51 79.38 79.60 10/0/0 79.43 10/0/0 79.60 10/0/0 COPA 90.00 93.00 90.00 5/3/2 90.00 5/3/2 91.00 8/0/2 WinoGrande 68.35 71.74 71.82 10/0/0 71.67 9/0/1 71.74 9/1/0 BoolQ 79.05 80.86 81.87 10/0/0 81.41 10/0/0 82.32 10/0/0 GSM8K 33.85 35.22 42.80 10/0/0 41.43 10/0/0 42.65 10/0/0 SST-2 62.50 79.70 74.66 9/0/1 72.02 9/0/1 69.27 9/0/1 WiC 51.72 56.58 52.66 8/0/2 50.47 3/1/6 51.25 5/0/5 WSC 36.54 59.62 47.12 8/0/2 41.35 8/0/2 39.42 8/0/2 MuTual 71.80 72.24 72.75 10/0/0 72.04 8/0/2 72.42 10/0/0 Last-20 ARC-Challenge 47.61 48.72 50.68 20/0/0 49.66 20/0/0 48.55 19/0/1 ARC-Easy 79.08 79.50 80.43 20/0/0 80.85 20/0/0 80.13 20/0/0 OpenBookQA 31.80 34.40 33.80 18/1/1 35.60 20/0/0 34.00 19/0/1 SciQ 95.50 96.30 95.80 13/4/3 95.50 9/1/10 95.80 13/4/3 MMLU 58.82 58.82 61.23 20/0/0 61.74 20/0/0 61.69 20/0/0 HellaSwag 57.33 58.03 59.28 20/0/0 59.23 20/0/0 59.22 20/0/0 PIQA 78.51 79.54 80.25 20/0/0 79.60 20/0/0 80.14 20/0/0 COPA 90.00 93.00 92.00 17/1/2 91.00 17/0/3 90.00 10/7/3 WinoGrande 68.35 72.69 73.24 20/0/0 72.14 18/0/2 72.77 20/0/0 BoolQ 79.05 80.86 82.97 20/0/0 82.66 20/0/0 82.23 20/0/0 GSM8K 33.85 35.52 44.47 20/0/0 44.13 20/0/0 42.57 20/0/0 SST-2 62.50 80.39 80.62 20/0/0 76.83 17/0/3 83.94 20/0/0 WiC 51.72 57.99 50.16 3/3/14 50.31 6/0/14 50.63 7/0/13 WSC 36.54 59.62 36.54 0/10/10 37.50 10/2/8 40.38 14/0/6 MuTual 71.80 72.24 72.70 20/0/0 72.50 20/0/0 72.22 19/0/1 Last-40 ARC-Challenge 47.61 49.15 49.15 39/1/0 48.81 39/0/1 47.95 30/1/9 ARC-Easy 79.08 79.63 79.59 39/0/1 79.88 40/0/0 79.92 40/0/0 OpenBookQA 31.80 34.40 33.20 34/3/3 34.80 40/0/0 33.40 37/0/3 SciQ 95.50 96.30 96.00 37/2/1 95.90 34/3/3 95.60 20/5/15 MMLU 58.82 59.00 61.79 40/0/0 61.88 40/0/0 61.53 40/0/0 HellaSwag 57.33 58.20 58.88 40/0/0 59.12 40/0/0 58.82 40/0/0 PIQA 78.51 80.09 80.03 38/0/2 80.58 40/0/0 79.54 34/1/5 COPA 90.00 93.00 90.00 20/11/9 90.00 20/11/9 91.00 31/3/6 WinoGrande 68.35 72.69 73.09 40/0/0 72.85 40/0/0 74.27 40/0/0 BoolQ 79.05 81.25 82.20 40/0/0 82.29 40/0/0 82.78 40/0/0 GSM8K 33.85 37.57 45.03 40/0/0 44.47 40/0/0 42.57 40/0/0 SST-2 62.50 80.50 82.45 40/0/0 83.49 40/0/0 79.36 36/0/4 WiC 51.72 57.99 50.16 13/5/22 50.78 22/2/16 50.16 13/5/22 WSC 36.54 59.62 36.54 0/19/21 42.31 34/0/6 37.50 19/6/15 MuTual 71.80 72.37 72.85 40/0/0 72.25 39/0/1 73.31 40/0/0 Appendix I Individual-Checkpoint Results Figures 6 and 7 show the individual-checkpoint benchmark scores on the two trajectories, including MuTual MRR. Window-level mean scores and the corresponding QAM results are reported in Tables 9 and 10; Table 12 compares the best observed merge scores across the main evaluation suite. Figure 6: Individual-checkpoint benchmark scores for SmolLM3. Figure 7: Individual-checkpoint benchmark scores for OpenEuroLLM Prelude.