Paper deep dive
Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even degrade some capabilities, which places a practical ceiling on how much compute mid-training absorbs. We revisit how this compute should be allocated to a single run or multiple similar optimizations. We find that branches forked from a shared checkpoint under various controlled recipe reaches measurably different regions of parameter space, and establish a form of compatible diversity that extending one run cannot supply. Therefore, we introduce Trajectory Soup, which distributes a mid-training budget over several independent branches, and consolidates strongest checkpoints selected on validation through intra- and inter-trajectory averaging into a single model. A local bias and variance analysis separates the two averaging levels, showing that inter-trajectory averaging removes residual error beyond the reach of averaging within a trajectory, while checkpoint selection carries a bias that bounds how many checkpoints are worth merging. Across model scales, learning-rate schedules, token budgets, and trajectory counts, Trajectory Soup improves aggregate downstream performance over the strongest single-trajectory average under matched budgets and keeps improving as budgets expand, with the advantage preserved after an identical post-training pipeline. These results position trajectory allocation and merging as a practical way to extend the compute-scaling frontier of mid-training beyond serial saturation.
Tags
Links
- Source: https://arxiv.org/abs/2609.37169v1
- Canonical: https://arxiv.org/abs/2609.37169v1
Trouble viewing inline? Open PDF directly →
Full Text
122,821 characters extracted from source content.
Expand or collapse full text
1]Ling Team, Ant Group 2]Shanghai Jiao Tong University [*]Equal contribution [†]Contribution during internship at Ant Group [‡]Corresponding author [Correspondence] Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories Zhehao Huang Changxin Tian Qingyuan Yang Kunlong Chen Ziqi Liu Zhiqiang Zhang Xiaolin Huang Jun Zhou Affiliation: [ Affiliation: [ Email: tianchangxin.tcx,lingyao.zzq@antgroup.com Email: kinght_h,xiaolinhuang@sjtu.edu.cn September 29, 2026 Abstract Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even degrade some capabilities, which places a practical ceiling on how much compute mid-training absorbs. We revisit how this compute should be allocated to a single run or multiple similar optimizations. We find that branches forked from a shared checkpoint under various controlled recipe reaches measurably different regions of parameter space, and establish a form of compatible diversity that extending one run cannot supply. Therefore, we introduce Trajectory Soup, which distributes a mid-training budget over several independent branches, and consolidates strongest checkpoints selected on validation through intra- and inter-trajectory averaging into a single model. A local bias and variance analysis separates the two averaging levels, showing that inter-trajectory averaging removes residual error beyond the reach of averaging within a trajectory, while checkpoint selection carries a bias that bounds how many checkpoints are worth merging. Across model scales, learning-rate schedules, token budgets, and trajectory counts, Trajectory Soup improves aggregate downstream performance over the strongest single-trajectory average under matched budgets and keeps improving as budgets expand, with the advantage preserved after an identical post-training pipeline. These results position trajectory allocation and merging as a practical way to extend the compute-scaling frontier of mid-training beyond serial saturation. (a) (b) Figure 1: Motivation and compute-allocation overview of Trajectory Soup. (a) Performance saturation along serial mid-training trajectories. (b) Compute scaling of raw training and Trajectory Soup. Overall Average Accuracy is plotted against cumulative training FLOPs on a logarithmic scale. 1 Introduction Mid-training plays an important role in developing specialized and reasoning capabilities in pretrained large language models (LLMs) (Liu et al., 2026; Shao et al., 2024). However, unlike in pretraining, increasing mid-training compute does not always lead to better downstream performance. As illustrated in Fig. 1(a), downstream performance improves rapidly early in training and then saturates, further training can even degrade capabilities. These observations expose a practical scaling wall in mid-training: beyond a certain point, extending the single training run brings little downstream improvement. This motivates us to explore alternative ways of allocating mid-training compute to sustain downstream gains. To this end, we consider distributing compute across multiple shorter trajectories. Starting from a common pretrained checkpoint, independent branches can follow different optimization paths through controlled variation of the training recipe (Frankle et al., 2020), potentially producing complementary updates even on the same data distribution. These potentially complementary updates can then be consolidated into a single model through model merging. However, existing methods typically focus on either intra-trajectory merging, which averages checkpoints within a single run (Izmailov et al., 2018), or inter-trajectory merging, which combines runs through endpoint averaging (Wortsman et al., 2022) or periodic aggregation (Douillard et al., 2023). Whether combining these two forms of merging can make compute allocation across trajectories more effective remains underexplored. Inspired by the compute-allocation perspective of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), we study how to divide a mid-training budget between trajectory count and trajectory length. This leads us to ask: whether exploiting both at once turns a fixed compute budget spread over several trajectories into a single model that outperforms a longer serial run? Our empirical observations and theoretical analysis support this possibility. First, after within-trajectory averaging, the examined branches retain distinct dominant directions. Interpolating between their averaged anchors further reduces validation loss, indicating that useful differences remain after temporal averaging. Second, a local bias–variance analysis separates within-trajectory fluctuations from persistent branch variation and characterizes how checkpoint selection trades variance reduction against mean displacement. Building on these findings, we propose Trajectory Soup, which combines both stages. It forks several independent branches from a shared pretrained checkpoint and trains each for the same number of tokens. Within each branch, it ranks checkpoints by validation set and averages a fixed number of the lowest-loss checkpoints into a trajectory anchor. It then averages these anchors into a single model, retaining the original architecture, tokenizer, and inference procedure. We evaluate two budget settings: budget reallocation, where branches share a fixed total token budget, and budget expansion, where we add branches while keeping the per-branch token budget fixed. Trajectory Soup turns compute into downstream gains where serial training no longer does. Under a matched token budget, Trajectory Soup surpasses the strongest intra-trajectory average as well as inter-trajectory baselines that keep only trajectory endpoints or that average the entire pooled set of checkpoints. Expanding the budget with additional trajectories widens the margin instead of exhausting it. Fig. 1(b) shows that Trajectory Soup frontier lies above the raw scaling curve at every matched compute level, and it keeps rising at a steady rate while the raw fit bends over and its checkpoints scatter downward as the budget grows. The frontier also improves as trajectories are added, and each trajectory needs to contribute only its strongest few checkpoints, so broad coverage of the loss basin rather than deep sampling within it carries the gain. The advantage persists after supervised fine-tuning, so scaling trajectories pays off end to end rather than only at the mid-training checkpoint. In summary, our contributions are threefold. • We recast compute allocation for LLM mid-training as a question of how many optimization trajectories to run rather than how long to run a single one, and identify trajectory count as an allocation axis that remains productive after the length of a single trajectory has saturated. Trajectories separated only by controlled variation of the recipe stay geometrically distinct, and interpolating between them improves on both endpoints. • We introduce Trajectory Soup, which selects the strongest checkpoints inside each trajectory and averages the resulting anchors across trajectories, yielding a single model with unchanged inference cost. A local bias and variance analysis grounds this design by separating what the two averaging levels contribute, quantifying the selection-dependent bias, showing that uniform weights minimize the variance term, and predicting a finite preferred checkpoint count that decreases as trajectories are added. • We validate the method across model scales, learning-rate schedules, training budgets, and trajectory counts, show that the advantage is retained after a shared post-training pipeline, and isolate trajectory diversity from checkpoint sampling density as the source of the gains. 2 Empirical Motivation: Saturation and Complementary Trajectories Serial mid-training exhibits diminishing downstream returns. Our scaling baseline fixes the model architecture and a high-quality mid-training corpus and adopts the best configuration found in hyperparameter search. As Fig. 1(a) shows, downstream performance first improves with the token budget, then plateaus, and eventually declines. The serial-training recipe therefore reaches a practical saturation regime beyond which further tokens stop paying off. Two mechanisms plausibly contribute to this decline. With model capacity fixed, prolonged exposure to a limited corpus offers diminishing returns, consistent with findings on the decreasing value of repeated training tokens (Muennighoff et al., 2023). Separately, stochastic fluctuations along the optimization path can leave a checkpoint displaced from nearby parameter regions of lower validation loss (Mandt et al., 2017). The second mechanism can be mitigated by weight averaging (Zhou et al., 2026), as the raw and averaged checkpoints of Fig. 3 indicate. However, once local fluctuations are largely attenuated, additional checkpoints contribute less noise reduction. Averaging within a single trajectory therefore runs into diminishing returns. The limitation of extending the same run motivates us to allocate additional compute to several branches instead. The remainder of this section tests two prerequisites for that allocation: (1) whether controlled recipe perturbations produce distinct optimization directions, and (2) whether combining the resulting branch anchors reaches lower validation loss. Figure 2: Compatible branch directions overlap. Pairwise cosine similarity between leading directions for five compatible branches. All pairs only have weak overlap and partial optimization similarity. Figure 3: Validation loss within and across trajectories. Left: validation loss against consumed tokens for the raw checkpoints of EXP1 and EXP2, for their within-trajectory merges, and for the best interpolation between the two merged anchors. Right: the same models in a two-dimensional projection of the weight space, with one interpolation path per token budget. Intra-trajectory averaging reveals partially overlapping branch directions. We first examine the directional diversity across branches that are comparably optimized. We parallelly run five trajectories under various recipes (denoted as EXP1∼EXP5EXP1 5). Each branch forks from the baseline run and perturbed by a controlled recipe change, such as changing the data order, learning rate, batch size, learning-rate schedule, or optimizer. All branches satisfy the training-loss screening criterion in Sec. 3.1, controlling differences in measured optimization quality. Now we try to discover whether these branches explore distinct optimization directions. Zhou et al. (2026) report that averaging can reveal an approximately rank-one structure within a late-stage training trajectory. Following this observation, we smooth each branch through intra-trajectory averaging, extract its leading principal component, and compute pairwise cosine similarities with other trajectories. Fig. 3 shows positive but limited alignment among these branches, indicating that controlled recipe perturbations produce distinct dominant directions with partial overlap. Directional diversity alone, however, does not establish a merging benefit, which depends on the validation-loss geometry between the resulting anchors rather than on the angle between them. Detailed protocol of trajectory geometry construction is provided in Appendix C. Inter-trajectory averaging reaches lower validation loss. We assess the benefit of combining intra-trajectory averaged anchors through linear interpolation (Frankle et al., 2020). At each matched token horizon, we take the averaged checkpoints of two screened branches and sweep the interpolation coefficient between them, following the protocol in Appendix C. Fig. 3 shows inter-trajectory interpolation achieves lower validation loss than intra-trajectory averaging alone. The right panel illustrates the corresponding geometry: branches departing from a shared pretrained checkpoint follow distinct optimization directions, while interpolation between their averaged anchors reaches a lower-loss region. This coexistence of directional diversity and compatibility under parameter averaging is consistent with prior findings on linear mode connectivity and model soups (Frankle et al., 2020; Wortsman et al., 2022). Together, the direction and interpolation analyses identify distinct optimization paths whose progress can be combined beneficially. These findings motivate allocating additional compute to multiple compatible branches as serial training approaches saturation. Sec. 3 formalizes this construction as Trajectory Soup. 3 Trajectory Soup 3.1 Problem Formulation and Budget Accounting Let θ0 _0 be a shared pre-trained checkpoint and let P be the target training distribution, which we keep common to all branches. Branch n∈1,…,Nn∈\1,…,N\ follows recipe ψn _n and produces parameters θn(i) _n(i) after consuming i mid-training tokens. All branches retain the same architecture, so their parameters can be combined coordinate by coordinate. For a common per-branch budget t, the total token budget is T=NtT=Nt. Each recipe ψn _n is obtained by modifying the best configuration found in the baseline hyperparameter search along one or more of its data order, learning rate, batch size, learning-rate schedule, or optimizer. Because such perturbations can also destabilize a run, we admit a branch only when a compatibility observable J stays within a tolerance ε , that is, when (θn(t))≤εJ( _n(t))≤ . In our implementation J is the relative training-loss gap to the baseline at the same token horizon and ε=0.01 =0.01. This screening keeps the branches comparable in optimization quality, while their parameter-space directions remain diverse. Changes to the data mixture are a possible extension, and we leave their effects to future work. 3.2 Trajectory Soup The observations in Sec. 2 suggest merging at two levels: within a branch where averaging attenuates local fluctuations along one path, and across branches where averaging consolidates directions that individual paths do not share. Trajectory Soup performs these two stages in order. Algorithm 1 Trajectory Soup 1: Shared checkpoint θ0 _0 2: Training distribution P 3: Screened recipes ψnn=1N\ _n\_n=1^N 4: Branch horizon t 5: Checkpoint count K 6: for n=1,…,Nn=1,…,N do 7: Train branch n from θ0 _0 on P under ψn _n for t tokens, saving checkpoints at n(t)C_n(t) 8: Rank n(t)C_n(t) by ascending validation loss 9: ℐn(t,K)←I_n(t,K)← the first K positions 10: θ¯n(t,K)←K−1∑i∈ℐn(t,K)θn(i) θ_n(t,K)← K^-1 _i _n(t,K) _n(i) 11: end for 12: θTraj-Soup(N,t,K)←N−1∑n=1Nθ¯n(t,K) _Traj -Soup(N,t,K)← N^-1 _n=1^N θ_n(t,K) 13: return θTraj-Soup(N,t,K) _Traj -Soup(N,t,K) Figure 4: Trajectory Soup diagram. Three branches are shown across training progress, trajectory index, and validation loss. Step 1: Intra-Trajectory Merging: selecting and averaging checkpoints within each branch. Let n(t)C_n(t) be the candidate token positions of checkpoints saved from branch n by t. Checkpoints beyond this horizon are excluded. We rank the candidates of each branch by validation loss ℒ^val(θ) L_val(θ), keep the best K positions, and average them into a branch anchor, so that within-trajectory variation is attenuated before any branch is combined with another, ℐn(t,K)∈argminℐ⊆n(t),|ℐ|=K∑i∈ℐℒ^val(θn(i)),θ¯n(t,K)=1K∑i∈ℐn(t,K)θn(i).I_n(t,K)∈ I _n(t),|I|=K _i L_val( _n(i)), θ_n(t,K)= 1K _i _n(t,K) _n(i). (1) Step 2: Inter-Trajectory Merging: combining branch anchors. Trajectory Soup averages the selected branch anchors, θTraj-Soup(N,t,K)=1N∑n=1Nθ¯n(t,K)=1NK∑n=1N∑i∈ℐn(t,K)θn(i). _Traj -Soup(N,t,K)= 1N _n=1^N θ_n(t,K)= 1NK _n=1^N _i _n(t,K) _n(i). (2) Fig. 4 illustrates the procedure and places it beside the two single-stage merges it generalizes. Algorithm 1 states the two averaging stages for the admitted branches. The two-stage construction is algebraically equivalent to uniformly averaging the selected M=NKM=NK checkpoints and the step order can be reversed. The special case N=1N=1 recovers selected intra-trajectory averaging, while K=1K=1 averages the best checkpoint from each branch. Trajectory Soup introduces branch count as an additional axis of compute allocation. As serial training approaches saturation, we seek to distribute the budget across shorter compatible trajectories, whose complementary progress is consolidated into a single model. Fig. 1(b) further shows that allocating additional compute to complementary branches sustains downstream gains beyond the observed serial-saturation regime. 4 Theoretical Insights We theoretically analyze how checkpoint selection and the two averaging stages in Trajectory Soup affect validation loss. We use a local quadratic model (Mandt et al., 2017) and standard bias–variance identities (Bishop, 2006) to separate selection-dependent bias, persistent branch variation, and within-trajectory fluctuations. 4.1 A Local Validation-Loss Model Let ℒQ(θ)=z∼Q[ℓ(θ,z)]L_Q(θ)=E_z Q[ (θ,z)] denote population validation loss, estimated by ℒ^val L_val for checkpoint ranking in Eq. (1). We assume that all analyzed checkpoints and averages lie in a region ℬ B around a stationary local minimizer θ⋆θ , where ℒQ(θ)=ℒ⋆+12‖θ−θ⋆‖H2L_Q(θ)=L + 12\|θ-θ \|_H^2 with ℒ⋆=ℒQ(θ⋆)L =L_Q(θ ), H=∇2ℒQ(θ⋆)⪰0H=∇^2L_Q(θ ) 0, and ‖x‖H2=x⊤Hx\|x\|_H^2=x Hx. The compatibility screen of Sec. 3.1 and the interpolation profiles of Sec. 2 motivate this local model, which underpins the stochastic analysis of constant-step SGD (Mandt et al., 2017) and the flatness interpretation of weight averaging (Wortsman et al., 2022). Theorem 4.1 (Local bias and variance decomposition). For any random model ϑ supported in ℬ B with finite second moments, [ℒQ(ϑ)]−ℒ⋆=12‖[ϑ]−θ⋆‖H2+12tr(HCov(ϑ)).E[L_Q( )]-L = 12\|E[ ]-θ \|_H^2+ 12tr\! (HCov( ) ). (3) The two terms measure displacement of the mean model and variation around it, respectively. Checkpoint selection can change both. The identity requires only finite moments and local compatibility, and makes no Gaussian or optimizer-specific assumption. The proof is given in Appendix B.1. 4.2 Branchwise Decomposition of Validation Loss We first identify the part of a branch’s loss that intra-trajectory merging can reach. Let θn,(j) _n,(j) be the checkpoint of validation rank j on branch n, and let θ¯n θ_n represent the checkpoint that averages its top-K selection in Eq. (1). Let ℱn F_n represent the persistent component of branch randomness, the part that a single run fixes and then carries along its entire path, so that conditioning on ℱn F_n holds that component fixed while retaining the variation among selected checkpoints. Theorem 4.2 splits the loss of the anchor into the displacement of its mean, the within-trajectory variation that survives conditioning, and the persistent variation it inherits from ℱn F_n. Theorem 4.2 (Branchwise decomposition of validation loss). Under the conditions of Theorem 4.1, [ℒQ(θ¯n)]−ℒ⋆=12‖[θ¯n]−θ⋆‖H2⏟Bn(K)+12[tr(HCov(θ¯n∣ℱn))]⏟Vintra,n(K)+12tr(HCov([θ¯n∣ℱn]))⏟Vinter,n(K).E[L_Q( θ_n)]-L = 12\|E[ θ_n]-θ \|_H^2_B_n(K)+ 12E\! [tr\! (HCov( θ_n F_n) ) ]_V_intra,n(K)+ 12tr\! (HCov(E[ θ_n F_n]) )_V_inter,n(K). (4) All three contributions, Bn(K)B_n(K), Vintra,n(K)V_intra,n(K), and Vinter,n(K)V_inter,n(K) are nonnegative. The bias Bn(K)B_n(K) collects an offset shared by every branch, which neither averaging stage removes, together with offsets specific to the recipe ψn _n and to the selection itself. Appendix A.1 records that decomposition and the random-effects construction behind it. The two variance terms differ in what can reach them: Vintra,n(K)V_intra,n(K) is precisely what merging within a branch acts on, whereas Vinter,n(K)V_inter,n(K) is invisible to any operation confined to a single trajectory. Sampling the path more densely within one trajectory may reveal more about that single branch, but it gives no access to the cross-branch variation. This tradeoff explains how the intra-trajectory gains of Sec. 2 can diminish, and how fast that term falls is what Sec. 4.3 quantifies. Appendix B.2 proves the theorem. 4.3 Hierarchical Variance Reduction To quantify the residual reduction, note that the term Vintra,n(K)V_intra,n(K) of Eq. (4) is the curvature-weighted covariance of the average, over the K selected ranks, of the fluctuations of the selected checkpoints around their conditional means given ℱn F_n. Temporal correlation and ranking-induced dependence therefore both limit how fast it decays. For branches with a common rank-1 residual contribution Vintra=Vintra,n(1)>0V_intra=V_intra,n(1)>0, we write Vintra,n(K)=Vintra/Keff(K)V_intra,n(K)=V_intra/K_eff(K), whose effective count is a variance ratio: it reaches K only when those fluctuations are uncorrelated with equal marginal contributions, and stays below K whenever their pairwise contributions are nonnegative. Appendix A.1 gives the residual covariance behind this ratio together with its bounds and limiting cases. Thus Keff(K)K_eff(K) settles the first averaging level, which operates inside a single branch. The second level averages the N branch anchors themselves. The following theorem therefore introduces the corresponding bias term and collects both levels in one expression. Theorem 4.3 (Hierarchical variance reduction). Suppose the shared-shift condition holds and the selected anchors are independent under the fixed experimental setup. Define Bsoup(K)=12‖1N∑n=1N[θ¯n]−θ⋆‖H2B_soup(K)= 12 \| 1N _n=1^NE[ θ_n]-θ \|_H^2, with common contributions Vintra,n(K)=Vintra/Keff(K)V_intra,n(K)=V_intra/K_eff(K) and Vinter,n(K)=VinterV_inter,n(K)=V_inter. The equally averaged model θTraj-Soup=N−1∑n=1Nθ¯n _Traj -Soup=N^-1 _n=1^N θ_n of Eq. (2) satisfies [ℒQ(θTraj-Soup)]−ℒ⋆=Bsoup(K)+VintraNKeff(K)+VinterN E[L_Q( _Traj -Soup)]-L =B_soup(K)+ V_intraNK_eff(K)+ V_interN (5) The bias Bsoup(K)B_soup(K) depends on the chosen branch collection. Temporal averaging reduces the intra-trajectory contribution through Keff(K)K_eff(K), while branch averaging reduces both stochastic contributions through N. Tab. 1 compares these effects. Appendix B.3 gives the proof. Method Intra Inter Expected excess loss in the local quadratic model Rank-1 checkpoint No No Bn(1)+Vintra+VinterB_n(1)+V_intra+V_inter Intra-trajectory merging Yes No Bn(K)+Vintra/Keff(K)+VinterB_n(K)+V_intra/K_eff(K)+V_inter Inter-trajectory merging No Yes Bsoup(1)+Vintra/N+Vinter/NB_soup(1)+V_intra/N+V_inter/N Trajectory Soup Yes Yes Bsoup(K)+Vintra/(NKeff(K))+Vinter/NB_soup(K)+V_intra/(NK_eff(K))+V_inter/N Table 1: Comparsive analysis of averaging operations under the local quadratic model and the homogeneous independent-anchor assumptions of Theorem 4.3. Variance optimality of uniform averaging. Finally, the same decomposition reveals that the two uniform averages of Trajectory Soup are variance-optimal rather than merely convenient. Consider replacing them with deterministic nonnegative weights over ranks and branches, each normalized to sum to one. Under the condition that every selected checkpoint of a branch shares the same aggregate curvature-weighted covariance with its selected set, and that the branch-level collections are independent, no such reweighting can attain a smaller stochastic contribution than Vintra/(NKeff(K))+Vinter/NV_intra/(NK_eff(K))+V_inter/N in Eq. (5), and the uniform temporal and branch weights of Eq. (2) attain this bound. This condition permits correlated residuals and is therefore weaker than independence, but it is substantive, since equal marginal variances and positive correlations alone do not imply equal covariance sums. 5 Experiments 5.1 Experimental Setup Model and mid-training trajectory settings. We conduct our default experiments with Ling-3.0-Tiny (inclusionAI, 2026), a sparse MoE language model with 7.9B total and 1.3B parameters activated per token. We fork multiple independent trajectories from the same checkpoint. To induce controlled trajectory diversity, each branch differs from the default configuration along one or more of the following dimensions: data-shuffling seed, peak learning rate, global batch size, learning-rate schedule, and optimizer. The default branch horizon is t=600t=600B tokens, which is sufficiently long for mid-training performance to approach saturation. Every branch saves a checkpoint each 25B tokens. Appendices D record the architecture, training recipe, and evaluation protocol. Merging baselines. Given a candidate pool, we compare four merging scopes: (i) Single EXP Merge, which combines checkpoints within a single experiment run (EXP); (i) Model Soup, which combines the final checkpoint of each trajectory; (i) Full Soup, which applies no selection and naively averages all checkpoints pooled from multiple trajectories, and which we therefore report as a selection ablation in Sec. 5.3.2; and (iv) our Trajectory Soup, which jointly selects and combines checkpoints both within and across trajectories. Compute-budget comparisons. We distinguish compute-matched evaluation from compute scaling. In the Limited setting, N trajectories share a total budget of T=NtT=Nt tokens (each covering T/NT/N tokens) and are compared compute-matched against a single trajectory trained for the same T. Reported budgets denote aggregate consumption. In the Extended setting, we merge N fully trained trajectories and increase N. Reported budgets denote per-trajectory consumption, testing whether merging converts an expanded budget (t→Nt→ Nt tokens) into improved quality. Figure 5: Performance of different mid-training trajectories and merging strategies as the training-token budget increases. Panels report the overall average accuracy and the five capability categories. Branch identities, merge sizes, and the Limited and Extended budget conventions follow Sec. 5.1. 5.2 Main Results of Trajectory Soup Intra- and Inter-trajectory Merging Effects during Mid-training. We systematically compare checkpoint-merging strategies during mid-training, focusing on how intra- and inter-trajectory effects interact across three independently trained trajectories: EXP1 (Baseline), EXP2 (Data Shuffle), and EXP3 (Hyperparameter Change). Tab. 2 summarizes the best candidate produced by each strategy. Inter-trajectory diversity yields further gains over intra-trajectory merging. In the compute-matched setting, Trajectory Soup (Limited) outperforms the strongest single-trajectory merge and Model Soup (Limited). In the full-trajectory setting, Trajectory Soup (Extended) achieves the best overall average of 68.96, exceeding the strongest single-trajectory merge and Model Soup (Extended). These results suggest that the gain cannot be attributed solely to either trajectory diversity or the number of merged checkpoints. Instead, effective scaling requires jointly exploiting inter-trajectory complementarity and selectively filtering checkpoints within each trajectory. The complete performance curves over increasing training-token budgets are deferred to Fig. 5, which shows that the ordering below holds throughout mid-training. Table 2: Performance comparison of different checkpoint-merging strategies on the mid-training evaluation suite. Each row reports the best evaluated configuration of its strategy; the corresponding checkpoint counts are recorded in Appendix D.5. Base Model General Knowledge & Reasoning Language Modeling Professional Knowledge Math Code Overall Average Single-Trajectory Merge 62.80 85.86 63.70 72.33 65.36 68.55 Model Soup (Limited) 62.74 85.80 63.59 72.53 64.74 68.43 Model Soup (Extended) 62.78 85.95 63.68 72.49 65.78 68.67 Trajectory Soup (Limited) 62.99 86.19 64.11 72.66 65.07 68.72 Trajectory Soup (Extended) 63.16 86.06 64.17 72.55 66.19 68.96 Table 3: Post-training performance after applying the same SFT procedure to mid-training checkpoints produced by different merging strategies. The merging protocols are those of Tab. 2. Instruct Model Math Code Knowledge Reasoning Instruction Following Function Call Overall Average Single-Trajectory Merge 68.30 46.58 70.51 62.96 59.46 50.67 61.11 Model Soup (Limited) 68.50 47.59 69.36 63.56 58.79 51.52 61.05 Model Soup (Extended) 68.63 46.76 69.65 64.41 59.29 51.63 61.23 Trajectory Soup (Limited) 69.44 48.33 69.81 63.60 58.88 51.49 61.39 Trajectory Soup (Extended) 68.53 47.50 69.26 64.48 60.38 52.85 61.52 Long-term Implications for Post-training. The preceding experiments establish that Trajectory Soup achieves the strongest aggregate performance at the mid-training stage, which naturally raises the question of whether this advantage persists after post-training. To investigate this, we use the mid-training checkpoints produced by each merging strategy as base models, apply an identical SFT recipe, and evaluate the post-training capabilities, with results reported in Tab. 3. The advantage of Trajectory Soup is largely preserved after SFT, indicating that our method provides a stronger initialization whose benefits transfer to subsequent post-training. Table 4: Comparison of single-trajectory merging and cross-trajectory Trajectory Soup during mid-training of the smaller 2B-parameter MoE model. The Limited and Extended budget conventions follow Sec. 5.1. Model General Knowledge & Reasoning Language Modeling Professional Knowledge Math Code Overall Average Single-Trajectory Merge 47.71 72.85 43.41 54.21 35.96 49.55 Trajectory Soup (Limited) 47.88 73.32 43.70 54.13 35.77 49.64 Trajectory Soup (Extended) 48.35 73.98 44.37 54.87 35.51 50.07 5.3 Empirical Analysis of Trajectory Soup 5.3.1 Robustness across models and learning-rate schedules. To examine whether the benefits of Trajectory Soup depend on a particular base model or learning-rate schedule, we repeat our experiments on a smaller 2B-parameter MoE model with 32 experts, activating 8 experts per token, using a warmup-stable-decay (WSD) learning-rate schedule (Hu et al., 2024). We compare intra-trajectory averaging within each trajectory against our Trajectory Soup. For each strategy, Tab. 4 reports the merging results. These results reproduce the aggregate trend observed under our default configuration. Under the compute-matched Limited setting, Trajectory Soup achieves an overall average of 49.64, slightly outperforming the strongest intra-trajectory baseline, despite each constituent trajectory receiving only half of the total token budget. This suggests that the complementarity introduced by trajectory diversity can compensate for reduced training-token coverage along each individual trajectory. Under the Extended setting, Trajectory Soup further improves the overall average to 50.07, exceeding the best intra-trajectory result. Together with the results obtained under the default setup, these findings indicate that the gains from Trajectory Soup are not confined to a single model scale or scheduling choice, but remain reproducible on a smaller MoE model trained with WSD. Table 5: Ablation of merging coefficients and checkpoint-selection strategies. Every row follows the Trajectory Soup (Extended) setting of Tab. 2, with Baseline the default configuration of Sec. 5.1. Ablation Coefficient Selection General Knowledge & Reasoning Language Modeling Professional Knowledge Math Code Overall Average Baseline EQUAL Top-K each 63.16 86.06 64.17 72.55 66.19 68.96 Coefficient 1SQRT Top-K each 63.22 85.87 64.21 72.26 66.00 68.85 RANK Top-K each 63.12 85.91 64.16 72.24 65.94 68.80 RSQRT Top-K each 63.04 86.03 64.26 72.59 65.97 68.89 Selection EQUAL Global top-NK 62.99 86.24 64.18 72.46 65.43 68.76 EQUAL Tail-K per branch 62.78 84.97 64.29 71.77 65.25 68.35 EQUAL All checkpoints 62.94 85.88 64.26 71.79 65.70 68.60 5.3.2 Ablation Studies on the Trajectory Soup Configuration We ablate three key design choices in Trajectory Soup: (1) how merge coefficients are assigned, (2) how checkpoints are selected, and (3) how the checkpoint budget is allocated across trajectories. Effect of merge coefficients. We compare uniform averaging against three rank-based weighting schemes, detailed in Appendix D.6. As shown in Tab. 5, none of them improves upon the equal-weight baseline. Once checkpoint quality is controlled through selection, more elaborate coefficient design provides little additional benefit, so uniform averaging is the strongest evaluated scheme and avoids introducing additional hyperparameters. Effect of checkpoint selection. We next fix merge coefficients to be uniform and consider three alternative selection strategies, also defined in Appendix D.6. All three underperform the default Top-K each strategy. These results identify both checkpoint quality and balanced trajectory representation as important selection criteria. Importance of Balanced Allocation across Trajectories. Fig. 7 tests balanced representation directly: we fix the total number of merged checkpoints to 16 and vary their allocation between EXP1 and EXP3 from 2:142:14 to 14:214:2. Performance follows an approximately inverted-U-shaped profile centered on the balanced allocation, with the symmetric 8:88:8 configuration attaining the highest overall average. Sufficiently balanced participation from both trajectories is therefore necessary to capture their complementary information, supporting symmetric allocation as a robust default. Overall, the three ablations consistently indicate that the effectiveness of Trajectory Soup is driven by the combination of trajectory diversity, checkpoint quality, and balanced representation, rather than by sophisticated coefficient design. Among the configurations evaluated, the simple combination of uniform averaging, top-K selection within each trajectory, and symmetric cross-trajectory allocation provides the strongest aggregate performance while requiring minimal additional tuning. Figure 6: Effect of asymmetric merging allocation on Trajectory Soup performance. Figure 7: Performance of densely sampled single run merging and equally sized sparse Trajectory Soup. 5.3.3 Trajectory Diversity over Sampling Density The preceding experiments show that Trajectory Soup consistently outperforms intra-trajectory merging throughout mid-training. However, this advantage could conceivably arise from a trivial confound: Trajectory Soup may simply draw from a larger checkpoint pool, thereby offering more candidates or a higher effective sampling density. To isolate the contribution of trajectory diversity, we conduct a controlled comparison in which both the candidate-pool size and the number of merged checkpoints are strictly matched. Specifically, we fix the merge size to M=12M=12 and compare densely sampled intra-trajectory merging with sparsely sampled cross-trajectory merging. In our experiment, EXP1 Merge and EXP2 Merge sample checkpoints every 12.5B tokens along their respective trajectories and uniformly average the top 12 checkpoints. In contrast, Trajectory Soup halves the per-trajectory sampling density by sampling checkpoints every 25B tokens from each trajectory and uniformly averages the top six checkpoints from each, again totaling 12. Thus, at a per-trajectory token coverage of t, both settings draw from exactly t/12.5Bt/12.5B candidate checkpoints and merge the same number of checkpoints, and the only systematic difference is whether these candidates originate from a single trajectory or from two independently optimized ones. As shown in Fig. 7, Trajectory Soup outperforms both intra-trajectory baselines at every comparable token budget, demonstrating that its gains cannot be attributed to a larger candidate pool or denser checkpoint sampling. This controlled comparison therefore provides direct evidence that the gains of Trajectory Soup stem from complementary information accumulated along independently evolved trajectories, rather than from increased checkpoint availability. 5.3.4 Scaling the Number of Trajectories and Merged Checkpoints We further investigate how Trajectory Soup scales along two axes: the number of participating trajectories N and the number of top-ranked checkpoints K contributed by each. For every (N,K)(N,K) configuration we uniformly average the top K checkpoints of each trajectory, and Fig. 9 reports the Overall Average against the total number of merged checkpoints, revealing two patterns. First, trajectory diversity, rather than checkpoint depth, drives performance: the best attainable score rises monotonically from 68.7968.79 with two trajectories to 69.0869.08 with five, while the increments steadily shrink. This is consistent with the standard interpretation of weight averaging, where uniform averaging cancels trajectory-specific errors only when the merged solutions reside in a shared low-loss basin while making quasi-independent mistakes, so a new trajectory helps most when it expands coverage of the basin, and its marginal value fades as the covered region saturates. Second, the soup benefits most from being selective: all groups peak within a narrow window of roughly ten to sixteen merged checkpoints, beyond which enlarging the pool brings oscillation or steady decline, with the four-trajectory soup eventually falling below the two-trajectory optimum. This makes Trajectory Soup practical to scale, adding trajectories reliably raises the performance ceiling, and retaining only a handful of top checkpoints per trajectory is sufficient to nearly reach it. 5.3.5 Directional Diversity Predicts Interpolation Gain Figure 8: Scaling of multi-trajectory Top-K Trajectory-Soup with the number of merged checkpoints. Figure 9: Weaker directional alignment predicts larger interpolation gains. The scaling results show that adding trajectories improves the merged model, but they do not identify which branches contribute the improvement or why. We therefore test whether pairwise branch geometry predicts the benefit of merging. Fig. 9 plots the cosine similarity of the two leading directions against the relative gain of interpolating the pair’s endpoints on the overall evaluation score. Both quantities are defined in Appendix C. The association is strongly negative and nearly monotone, with Pearson r=−0.80r=-0.80 (p=0.005p=0.005) and Spearman ρ=−0.94ρ=-0.94: the near-orthogonal pairs gain the most and the most aligned pairs the least. Directional diversity therefore explains the scaling trend above and supplies a cheaply computable criterion, available before any merging is performed, for selecting which branches to merge. 6 Related Work 6.1 Mid-Training and Horizon-Extensible Scaling Classical language-model scaling laws characterize the dependence of performance on model size, training data, and compute. Compute-optimal training rules balance parameter count and token consumption (Hoffmann et al., 2022). In data-constrained regimes, the effective value of repeated tokens decreases as repetition grows (Muennighoff et al., 2023). These results motivate distinguishing additional token consumption from the useful progress obtained from it. Training schedules provide flexibility in extending a run. MiniCPM introduces the Warmup–Stable–Decay schedule, whose stable phase supports continued training followed by a cooldown (Hu et al., 2024). The river-valley analysis of WSD explains how a large learning rate can support progress along a slowly varying direction while maintaining fluctuations in sharper directions (Wen et al., 2024). Complementing this optimization perspective, mid-training studies examine how mixtures of general and specialized data improve the starting point for post-training (Liu et al., 2026). Our study adds trajectory count to the allocation problem and examines selective weight-space aggregation under both matched and expanded training budgets. 6.2 Checkpoint Averaging Along a Single Trajectory Stochastic Weight Averaging combines iterates from constant or cyclical learning-rate training and often reaches flatter solutions with improved generalization (Izmailov et al., 2018). Trainable Weight Averaging instead learns the averaging coefficients by optimizing in the low-dimensional subspace spanned by the candidate checkpoints rather than fixing them in advance (Li et al., 2023b). In language-model pre-training, Sanyal et al. (2023) show that averaging can improve convergence and downstream performance, with larger gains under high learning rates and sufficiently spaced checkpoints. Pre-trained Model Averaging studies checkpoint merging across dense and MoE language models and relates stable-phase averages to annealing behavior (Li et al., 2025). WSM develops the connection through the effective coefficients assigned to historical updates (Tian et al., 2026). Extra-Merge identifies approximately one-dimensional structure in late-stage merged trajectories and uses this structure for extrapolation (Zhou et al., 2026). These approaches establish the value of temporal aggregation. We build on this foundation by combining selected temporal averages across independently evolved branches and studying how this changes the allocation of training compute. 6.3 Combining Multiple Training Trajectories Prediction ensembles retain multiple models at inference time. Weight-space combinations can consolidate compatible models into one parameter set. Model Soups demonstrates effective averaging of models fine-tuned from a common initialization with different hyperparameters (Wortsman et al., 2022). In language modeling, Branch-Train-Merge trains domain-specialized expert language models that can be ensembled or averaged, and studies their performance under controlled training cost (Li et al., 2022). Branch-Train-MiX incorporates independently trained feedforward components into MoE layers, followed by training that learns routing (Sukhbaatar et al., 2024). The geometry underlying such combinations is related to mode connectivity. Garipov et al. (2018) construct low-loss curves between trained solutions, while Frankle et al. (2020) study linear connectivity under different realizations of SGD noise after a shared training history. Our interpolation analysis examines the straight segments between mid-training anchors, which are directly relevant to arithmetic parameter averaging. 7 Discussion and Limitations Scope of the compute accounting. Our allocation comparison holds processed training tokens fixed at a common architecture and sequence length, which isolates the allocation question but charges nothing for the machinery around it. Checkpoint storage, the validation passes used for ranking, and the search over how many trajectories and checkpoints to combine all consume resources that a complete development-cost account would attribute to each strategy, and accelerator utilization differs between one long run and several concurrent shorter ones. Making these costs explicit is the natural next step, because a cost model that prices a trajectory against a token would let the allocation be chosen by optimization rather than by sweeping candidates. What the analysis assumes. The theory is exact under a local quadratic model of the validation loss and inherits whatever error that model carries over the region spanned by the selected checkpoints and their averages. The geometric diagnostics are complementary but weaker. Direction angles characterize differences in a chosen parameter-space representation, and interpolation profiles probe particular averages, so neither establishes that the region containing all branches is jointly well behaved. Both are also measured only after the branches exist. A more useful account would predict which recipe perturbations generate error components that averaging can remove, turning diversity from a property screened after training into one designed before it. Where averaging stops paying. Averaging cannot remove error that the branches share, so strongly correlated trajectories leave a floor that no amount of merging crosses, and perturbations chosen too aggressively push branches into regions where the merged model is worse than its members. The same tension governs the budget, since at a fixed total more branches means shorter runs, and admitting more checkpoints per branch eventually admits weak ones. Trajectory Soup therefore depends on selecting compatible and sufficiently trained branches together with a suitable per-branch checkpoint count, which we currently fix by validation search. Learning this allocation during training, or adapting it as branches diverge, would remove the remaining hand-tuning and is where we expect the largest gains to come from. 8 Conclusion Mid-training has so far been scaled by making a single optimization trajectory longer, and that axis eventually stops paying. This paper treats the number of trajectories as an allocation variable of the same kind as parameters and tokens, and works out what it takes to spend a budget that way. Trajectory Soup forks independent branches from a shared checkpoint, selects the strongest checkpoints inside each branch, and consolidates the resulting anchors into a single model whose architecture, tokenizer, and inference cost are unchanged. A local bias and variance analysis accounts for the design, showing why the two averaging levels address different parts of the error, why uniform weights are the natural default at both levels, and why checkpoint selection carries a bias that bounds how many checkpoints are worth merging. Weight-space consolidation is a general way to convert parallel exploration into one deliverable model, rather than a technique specific to mid-training. The same structure arises wherever a budget can be split across runs that share an initialization, including continued pretraining, domain adaptation, and the increasingly long post-training pipelines that follow. Realizing that generality calls for understanding which perturbations create removable error, how to price a trajectory against a token, and how far compatibility survives objectives that carry models far from their shared origin. Treating trajectory count as a standard dimension of the compute-scaling question, rather than a special case of model merging, is in our view the more productive framing for the work that follows. References Aider (2026) Aider Aider code editing benchmarks. Cited by: §D.4. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton Program synthesis with large language models. ArXiv abs/2108.07732. Cited by: §D.4. Bandarkar et al. (2024) L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Annual Meeting of the Association for Computational Linguistics, Cited by: §D.4. Benchekroun et al. (2023) Y. Benchekroun, M. Dervishi, M. Ibrahim, J. Gaya, X. Martinet, G. Mialon, T. Scialom, E. Dupoux, D. Hupkes, and P. Vincent WorldSense: a synthetic benchmark for grounded reasoning in large language models. ArXiv abs/2311.15930. Cited by: §D.4. Bishop (2006) C. M. Bishop Pattern recognition and machine learning. Springer. Cited by: §A.4, §4. Bisk et al. (2020) Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi PIQA: reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, Cited by: §D.4. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pondé, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. S. Winter, P. Tillet, F. P. Such, D. W. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, I. Babuschkin, S. Balaji, S. Jain, A. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. ArXiv abs/2107.03374. Cited by: §D.4. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv abs/1803.05457. Cited by: §D.4. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. ArXiv abs/2110.14168. Cited by: §D.4. Douillard et al. (2023) A. Douillard, Q. Feng, A. A. Rusu, R. Chhaparia, Y. Donchev, A. Kuncoro, M. Ranzato, A. Szlam, and J. Shen DiLoCo: distributed low-communication training of language models. ArXiv abs/2311.08105. Cited by: §1. Frankle et al. (2020) J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, Cited by: §1, §2, §6.3. Garipov et al. (2018) T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson Loss surfaces, mode connectivity, and fast ensembling of dnns. In Neural Information Processing Systems, Cited by: §6.3. Gu et al. (2024) A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. Wang CRUXEval: a benchmark for code reasoning, understanding and execution. In International Conference on Machine Learning, Cited by: §D.4. He et al. (2024) Y. He, S. Li, J. Liu, Y. Tan, W. Wang, H. Huang, X. Bu, H. Guo, C. Hu, B. Zheng, Z. Lin, X. Liu, D. Sun, S. Lin, Z. Zheng, X. Zhu, W. Su, and B. Zheng Chinese simpleqa: a chinese factuality evaluation for large language models. ArXiv abs/2411.07140. Cited by: §D.4, §D.4. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. X. Song, and J. Steinhardt Measuring massive multitask language understanding. ArXiv abs/2009.03300. Cited by: §D.4. Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. X. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset. ArXiv abs/2103.03874. Cited by: §D.4. HMMT (2026) HMMT Harvard–mit mathematics tournament. Cited by: §D.4. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre Training compute-optimal large language models. ArXiv abs/2203.15556. Cited by: §C.1, §1, §6.1. Hu et al. (2024) S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, X. Zhang, Z. L. Thai, K. Zhang, C. Wang, Y. Yao, C. Zhao, J. Zhou, J. Cai, Z. Zhai, N. Ding, C. Jia, G. Zeng, D. Li, Z. Liu, and M. Sun MiniCPM: unveiling the potential of small language models with scalable training strategies. ArXiv abs/2404.06395. Cited by: §5.3.1, §6.1. Huang et al. (2023) Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, F. Qi, Y. Fu, M. Sun, and J. He C-eval: a multi-level multi-discipline chinese evaluation suite for foundation models. ArXiv abs/2305.08322. Cited by: §D.4, §D.4. inclusionAI (2026) inclusionAI Ling-3.0-tiny. Note: https://huggingface.co/inclusionAI/Ling-3.0-tiny Cited by: §5.1. Izmailov et al. (2018) P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, Cited by: §A.4, §1, §6.2. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, Cited by: §D.4, §D.4. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. ArXiv abs/1705.03551. Cited by: §D.4. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. ArXiv abs/2001.08361. Cited by: §C.1, §1. Kwiatkowski et al. (2019) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. V. Le, and S. Petrov Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics. Cited by: §D.4. Lai et al. (2017) G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy RACE: large-scale reading comprehension dataset from examinations. ArXiv abs/1704.04683. Cited by: §D.4. Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra Solving quantitative reasoning problems with language models. In Neural Information Processing Systems, Cited by: §D.4. Li et al. (2024a) H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin CMMLU: measuring massive multitask language understanding in chinese. In Findings of the Association for Computational Linguistics, Cited by: §D.4. Li et al. (2023a) J. Li, B. Hui, G. Qu, B. Li, J. Yang, B. Li, B. Wang, B. Qin, R. Cao, R. Geng, N. Huo, C. Ma, K. C. Chang, F. Huang, R. Cheng, and Y. Li Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. In Neural Information Processing Systems, Cited by: §D.4. Li et al. (2022) M. Li, S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer Branch-train-merge: embarrassingly parallel training of expert language models. ArXiv abs/2208.03306. Cited by: §6.3. Li et al. (2024b) Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi GSM-plus: a comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. ArXiv abs/2402.19255. Cited by: §D.4. Li et al. (2023b) T. Li, Z. Huang, Q. Tao, Y. Wu, and X. Huang Trainable weight averaging: efficient training by optimizing historical solutions. In International Conference on Learning Representations, Cited by: §6.2. Li et al. (2021) W. Li, F. Qi, M. Sun, X. Yi, and J. Zhang CCPM: a chinese classical poetry matching dataset. ArXiv abs/2106.01979. Cited by: §D.4. Li et al. (2025) Y. Li, Y. Ma, S. Yan, C. Zhang, J. Liu, J. Lu, Z. Xu, M. Chen, M. Wang, S. Zhan, J. Ma, X. Lai, D. Liu, Y. Luo, X. Bin, H. Ren, M. Han, W. Hao, B. Yi, L. Liu, B. Ma, X. Jia, X. Zhou, S. Qiao, L. Xiang, and Y. Wu Model merging in pre-training of large language models. ArXiv abs/2505.12082. Cited by: §6.2. Liu et al. (2026) E. Liu, G. Neubig, and C. Xiong Midtraining bridges pretraining and posttraining distributions. In International Conference on Machine Learning, Cited by: §1, §6.1. Liu et al. (2024) H. Liu, Z. Zheng, Y. Qiao, H. Duan, Z. Fei, F. Zhou, W. Zhang, S. Zhang, D. Lin, and K. Chen MathBench: evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. ArXiv abs/2405.12209. Cited by: §D.4. Liu et al. (2023) J. Liu, C. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Neural Information Processing Systems, Cited by: §D.4. Ma et al. (2024) K. Ma, X. Du, Y. Wang, H. Zhang, Z. Wen, X. Qu, J. Yang, J. Liu, M. Liu, X. Yue, W. Huang, and G. Zhang KOR-bench: benchmarking language models on knowledge-orthogonal reasoning tasks. ArXiv abs/2410.06526. Cited by: §D.4. Mandt et al. (2017) S. Mandt, M. D. Hoffman, and D. M. Blei Stochastic gradient descent as approximate bayesian inference. J. Mach. Learn. Res.. Cited by: §2, §4.1, §4. Mathematical Association of America (2026) Mathematical Association of America American invitational mathematics examination (AIME). Cited by: §D.4. Muennighoff et al. (2023) N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and C. Raffel Scaling data-constrained language models. ArXiv abs/2305.16264. Cited by: §2, §6.1. Patil et al. (2025) S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. Gonzalez The berkeley function calling leaderboard (bfcl): from tool use to agentic evaluation of large language models. In International Conference on Machine Learning, Cited by: §D.4. Pyatkin et al. (2025) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following. In Neural Information Processing Systems, Cited by: §D.4. Rajpurkar et al. (2018) P. Rajpurkar, R. Jia, and P. Liang Know what you don’t know: unanswerable questions for squad. ArXiv abs/1806.03822. Cited by: §D.4. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof q&a benchmark. ArXiv abs/2311.12022. Cited by: §D.4. Sakaguchi et al. (2019) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: an adversarial winograd schema challenge at scale. ArXiv abs/1907.10641. Cited by: §D.4. Sanyal et al. (2023) S. Sanyal, A. Neerkaje, J. Kaddour, A. Kumar, and S. Sanghavi Early weight averaging meets high learning rates for llm pre-training. ArXiv abs/2306.03241. Cited by: §6.2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. ArXiv abs/2402.03300. Cited by: §1. Shi et al. (2022) F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei Language models are multilingual chain-of-thought reasoners. ArXiv abs/2210.03057. Cited by: §D.4. Sukhbaatar et al. (2024) S. Sukhbaatar, O. Golovneva, V. Sharma, H. Xu, X. V. Lin, B. Rozière, J. Kahn, S. Li, W. Yih, J. E. Weston, and X. Li Branch-train-mix: mixing expert llms into a mixture-of-experts llm. ArXiv abs/2403.07816. Cited by: §6.3. Suzgun et al. (2023) M. Suzgun, N. Scales, N. Scharli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics, Cited by: §D.4. Tang et al. (2024) Z. Tang, X. Zhang, B. Wang, and F. Wei MathScale: scaling instruction tuning for mathematical reasoning. In International Conference on Machine Learning, Cited by: §D.4. Team et al. (2025) M. Team, X. Du, Y. Yao, K. Ma, B. Wang, T. Zheng, K. Zhu, M. Liu, Y. Liang, X. Jin, Z. Wei, C. Zheng, K. Deng, S. Gavin, S. Jia, S. Jiang, Y. Liao, R. Li, Q. Li, S. Li, Y. Li, Y. Li, D. Ma, Y. Ni, H. Que, Q. Wang, Z. Wen, S. Wu, T. R. Hsing, M. Xu, Z. Yang, Z. Wang, J. Zhou, Y. Bai, X. Bu, C. Cai, L. Chen, Y. Chen, C. Cheng, T. Cheng, K. Ding, S. Huang, Y. Huang, Y. Li, Y. Li, Z. Li, T. Liang, C. Lin, H. Lin, Y. B. Ma, T. Pang, Z. Peng, Z. Peng, Q. Qi, S. Qiu, X. Qu, S. Quan, Y. Tan, Z. Wang, C. Wang, H. Wang, Y. Wang, Y. Wang, J. Xu, K. Yang, R. Yuan, Y. Yue, T. Zhan, C. Zhang, J. Zhang, X. Zhang, X. Zhang, Y. Zhang, Y. Zhao, X. Zheng, C. Zhong, Y. Gao, Z. Li, D. Liu, Q. Liu, T. Liu, S. Ni, J. Peng, Y. Qin, W. Su, G. Wang, S. Wang, J. Yang, M. Yang, M. Cao, X. Yue, Z. Zhang, W. Zhou, J. Liu, Q. Lin, W. Huang, and G. Zhang SuperGPQA: scaling llm evaluation across 285 graduate disciplines. In Neural Information Processing Systems, Cited by: §D.4. Tian et al. (2026) C. Tian, J. Wang, Q. Zhao, K. Chen, J. Liu, Z. Liu, J. Mao, W. X. Zhao, Z. Zhang, and J. Zhou WSM: decay-free learning rate schedule via checkpoint merging for llm pre-training. In International Conference on Learning Representations, Cited by: §6.2. Tian et al. (2024) M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, S. Liu, D. Luo, Y. Ma, H. Tong, K. Trinh, C. Tian, Z. Wang, B. Wu, Y. Xiong, S. Yin, M. Zhu, K. A. Lieret, Y. Lu, G. Liu, Y. Du, T. Tao, O. Press, J. Callan, E. A. Huerta, and H. Peng SciCode: a research coding benchmark curated by scientists. In Neural Information Processing Systems, Cited by: §D.4. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. W.F. Ku, K. Wang, A. Zhuang, R. ". Fan, X. Yue, and W. Chen MMLU-pro: a more robust and challenging multi-task language understanding benchmark. ArXiv abs/2406.01574. Cited by: §D.4, §D.4. Wei et al. (2024) J. Wei, K. Nguyen, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus Measuring short-form factuality in large language models. ArXiv abs/2411.04368. Cited by: §D.4. Wei et al. (2023) T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang CMATH: can your language model pass chinese elementary school math test?. ArXiv abs/2306.16636. Cited by: §D.4. Wen et al. (2024) K. Wen, Z. Li, J. S. Wang, D. L. W. Hall, P. Liang, and T. Ma Understanding warmup-stable-decay learning rates: a river valley loss landscape perspective. ArXiv abs/2410.05192. Cited by: §6.1. Wortsman et al. (2022) M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ArXiv abs/2203.05482. Cited by: §D.5, §1, §2, §4.1, §6.3. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Annual Meeting of the Association for Computational Linguistics, Cited by: §D.4. Zhong et al. (2023) W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan AGIEval: a human-centric benchmark for evaluating foundation models. ArXiv abs/2304.06364. Cited by: §D.4. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. ArXiv abs/2311.07911. Cited by: §D.4. Zhou et al. (2026) W. Zhou, B. Wang, H. Zhang, C. Jia, W. Chen, and X. Cheng Extra-merge: tracing the rank-1 subspace of model merging in language model pre-training. In International Conference on Machine Learning, Cited by: §C.2, §2, §2, §6.2. Appendix A Additional Theory for Hierarchical Merging This appendix records the modeling detail behind the main-text results. We make the random-effects assumptions of Sec. 4.2 explicit and allow heterogeneous or correlated branches, examine how the selected checkpoint count trades residual reduction against bias, restore the fixed-budget coupling between branch count and branch length, state the variance optimality of the two uniform averages that Sec. 4.3 reports in prose, and give the exact covariance condition it requires. Every loss identity below is exact under the local quadratic model of Sec. 4.1. Throughout Appendices A and B we abbreviate the two anchor means of Sec. 4.2 as μn(K)=[θ¯n] _n(K)=E[ θ_n] and mn(K)=[θ¯n∣ℱn]m_n(K)=E[ θ_n F_n]. To keep the account compact, we develop these extensions as narrative rather than as separate numbered claims, apart from the corollary of Appendix A.4, and give their derivations in place. The three theorems stated in the main text and that corollary are proved in Appendix B. A.1 Conditional Random Effects and General Covariances Theorem 4.2 uses total covariance without requiring a particular training algorithm. Its shared-shift specialization can be represented by a local random-effects model. For fixed t, assume [θn,(j)∣ℱn]=θ⋆+bcom+brec,n+hn,j+un,[un]=0,hn,1=0,E[ _n,(j) F_n]=θ +b_com+b_rec,n+h_n,j+u_n, [u_n]=0, h_n,1=0, (A1) where unu_n is measurable with respect to ℱn F_n and common to all selected ranks on that branch, and the deterministic offsets absorb selection-induced changes in mean. Here bcomb_com is a finite-horizon offset shared by every branch, brec,nb_rec,n is the systematic offset of recipe ψn _n, and hn,jh_n,j is the rank-j contribution to the selection-induced shift. Write ϵn,j=θn,(j)−[θn,(j)∣ℱn] _n,j= _n,(j)-E[ _n,(j) F_n] for the conditionally centered fluctuation that Sec. 4.3 describes in words. Averaging over the selected ranks then gives hn(K)=K−1∑j=1Khn,jh_n(K)=K^-1 _j=1^Kh_n,j and ϵ¯n(K)=K−1∑j=1Kϵn,j ε_n(K)=K^-1 _j=1^K _n,j, so the mean displacement described in Sec. 4.2 reads μn(K)−θ⋆=bcom+brec,n+hn(K),hn(1)=0. _n(K)-θ =b_com+b_rec,n+h_n(K), h_n(1)=0. (A2) A single unu_n shared by all selected ranks is exactly the shared-shift condition that Theorem 4.3 assumes: it makes mn(K)=μn(K)+unm_n(K)= _n(K)+u_n and therefore Vinter,n(K)=12tr(HCov(un))V_inter,n(K)= 12tr(HCov(u_n)) independent of K, so selection moves the anchor only through hn(K)h_n(K) and the residual average. This is a modeling assumption; if the conditional shift changes across ranks, the general term Vinter,n(K)V_inter,n(K) of Theorem 4.2 retains that dependence. The homogeneous independent-anchor case is what exposes the scaling factors, but a general branch collection needs the full anchor cross-covariance Γnn′(K)=Cov(θ¯n,θ¯n′) _n (K)=Cov( θ_n, θ_n ) together with its scalar form Gnn′(K)=12tr(HΓnn′(K))G_n (K)= 12tr(H _n (K)). Two properties follow directly. First, G(K)G(K) is symmetric because Γn′n(K)=Γnn′(K)⊤ _n n(K)= _n (K) and H is symmetric, and it is positive semidefinite because a⊤G(K)a=12[‖∑nan(θ¯n−μn(K))‖H2]≥0a G(K)a= 12E[\| _na_n( θ_n- _n(K))\|_H^2]≥ 0 for every a∈ℝNa ^N. Second, the covariance of the branch-reweighted model θα=∑nαnθ¯n _α= _n _n θ_n, which keeps the uniform temporal average inside every branch, is ∑n,n′αnαn′Γnn′(K) _n,n _n _n _n (K), so Theorem 4.1 gives, for fixed nonnegative weights summing to one, [ℒQ(θα)]−ℒ⋆=Bα(K)+α⊤G(K)α,Bα(K)=12‖[θα]−θ⋆‖H2.E[L_Q( _α)]-L =B_α(K)+α G(K)α, B_α(K)= 12\|E[ _α]-θ \|_H^2. (A3) Setting every weight to 1/N1/N turns the quadratic form into a double sum over the branch pairs, [ℒQ(θTraj-Soup)]−ℒ⋆=Bsoup(K)+12N2∑n,n′=1Ntr(HΓnn′(K)).E[L_Q( _Traj -Soup)]-L =B_soup(K)+ 12N^2 _n,n =1^Ntr\! (H _n (K) ). (A4) Expanding the anchor as θ¯n=θ⋆+bcom+brec,n+hn(K)+un+ϵ¯n(K) θ_n=θ +b_com+b_rec,n+h_n(K)+u_n+ ε_n(K) resolves each block into three contributions: the covariance of the branch offsets unu_n, the covariance of the averaged residuals ϵ¯n(K) ε_n(K), and the two cross terms between an offset of one branch and the averaged residual of the other. Because unu_n is ℱn F_n-measurable, conditional centering removes the cross terms on the diagonal and leaves Gnn(K)=Vintra,n(K)+Vinter,nG_n(K)=V_intra,n(K)+V_inter,n, while placing no restriction on the cross terms for n≠n′n≠ n . Independent selected anchors make the off-diagonal Γnn′(K) _n (K) vanish, which reduces Eq. (A4) to the heterogeneous form Bsoup(K)+N−2∑n=1N[Vintra,n(K)+Vinter,n]B_soup(K)+N^-2 _n=1^N[V_intra,n(K)+V_inter,n]. Independence of the full random-effects collections is a stronger condition that makes every off-diagonal block vanish separately. Equal scalar loss-weighted contributions therefore suffice for Theorem 4.3, and the full covariance matrices may still differ. Independent training followed by selection performed separately within each branch preserves anchor independence when the shared setup is fixed, whereas joint selection or a coupled compatibility decision leaves the off-diagonal blocks in place and Eq. (A4) applies. Those blocks also bound what branch averaging can achieve, because a collection with positive average off-diagonal contribution drives the branch term toward that average rather than toward zero as N grows, which is the correlated error floor discussed in Appendix . Weighting the anchors unequally is worthwhile exactly when their contributions differ. For independent anchors with vn(K)=Vintra,n(K)+Vinter,n>0v_n(K)=V_intra,n(K)+V_inter,n>0, the variance term of Eq. (A3) is ∑nvn(K)αn2 _nv_n(K) _n^2, and Cauchy–Schwarz applied to ∑n(vn(K)αn)vn(K)−1/2=1 _n ( v_n(K) _n )v_n(K)^-1/2=1 gives ∑nvn(K)αn2≥(∑nvn(K)−1)−1 _nv_n(K) _n^2≥( _nv_n(K)^-1)^-1, with equality only when αn _n is proportional to vn(K)−1v_n(K)^-1. The unique minimizer is thus the inverse-variance weighting αnvar(K)=vn(K)−1/∑n′vn′(K)−1 _n^var(K)=v_n(K)^-1/ _n v_n (K)^-1. If instead G(K)G(K) is exchangeable with common diagonal v and common off-diagonal c, then α⊤G(K)α=c+(v−c)∑nαn2α G(K)α=c+(v-c) _n _n^2, where v−c≥0v-c≥ 0 follows from positive semidefiniteness on vectors orthogonal to 1, so equal weights minimize the variance and do so uniquely when v>cv>c. The same exchangeable argument transfers to the residual weights over the K ranks within one branch once the branch covariance matrix is replaced by the rank covariance matrix, while rank-dependent means continue to act through the bias instead. Both statements concern fixed weights, and estimating weights from the same realized checkpoints would require accounting for the joint randomness of weights and anchors. The effective checkpoint count summarizes the rank covariance in a single scalar, so its interpretation needs both the marginal variation and the cross-checkpoint dependence. For a fixed branch, write cjj′=12tr(HCov(ϵn,j,ϵn,j′))c_j = 12tr(HCov( _n,j, _n,j )) and assume the common rank-1 value cjj=Vintra>0c_j=V_intra>0. Expanding the covariance of ϵ¯n(K) ε_n(K) and taking its loss-weighted trace gives Vintra,n(K)=1K2∑j,j′=1Kcjj′,Keff(K)=K2Vintra∑j,j′=1Kcjj′.V_intra,n(K)= 1K^2 _j,j =1^Kc_j , K_eff(K)= K^2V_intra _j,j =1^Kc_j . (A5) Since the residuals are mean zero, cjj′=12[(H1/2ϵn,j)⊤(H1/2ϵn,j′)]c_j = 12E[(H^1/2 _n,j) (H^1/2 _n,j )], and Cauchy–Schwarz bounds |cjj′|≤cjjcj′j′=Vintra|c_j |≤ c_jc_j j =V_intra. If every entry is nonnegative, the covariance sum lies between KVintraKV_intra and K2VintraK^2V_intra, so 1≤Keff(K)≤K1≤ K_eff(K)≤ K, with the upper bound attained exactly when the off-diagonal entries vanish and strict as soon as their sum is positive. Independent residuals attain that bound. Negative covariances can instead push Keff(K)K_eff(K) above K, heterogeneous ranks invalidate the equal-marginal premise, and when Vintra,n(K)=0V_intra,n(K)=0 the convention Keff(K)=∞K_eff(K)=∞ returns the correct zero contribution. In such settings the general definition in Sec. 4.3 remains a variance ratio. In particular, increasing K under validation ranking need not increase Keff(K)K_eff(K), which is what the next subsection turns into a tradeoff. A.2 The Tradeoff in Selecting More Checkpoints At fixed N and t, adding a lower-ranked checkpoint changes the mean anchor through hn(K)h_n(K) and the residual covariance through Eq. (A5). A lower individual validation rank does not by itself determine the loss of the resulting average, so the relevant comparison is between the change in bias and the change in the stochastic contribution. Suppose Theorem 4.3 applies to the same branch collection at K and K+1K+1, with K-independent VinterV_inter and reference VintraV_intra, and write θTraj-Soup(K) _Traj -Soup(K) for the soup built from that collection at count K, holding N and t fixed. Applying Eq. (5) at both counts cancels ℒ⋆L and the branch term Vinter/NV_inter/N, which leaves [ℒQ(θTraj-Soup(K+1))]−[ℒQ(θTraj-Soup(K))] [L_Q( _Traj -Soup(K+1))]-E[L_Q( _Traj -Soup(K))] (A6) =Bsoup(K+1)−Bsoup(K)+VintraN[1Keff(K+1)−1Keff(K)]. =B_soup(K+1)-B_soup(K)+ V_intraN [ 1K_eff(K+1)- 1K_eff(K) ]. When KeffK_eff increases, the stochastic increment is negative, so an additional checkpoint improves the expected loss precisely when that reduction outweighs the bias increment. This is the mechanism behind an interior optimum, and it also leaves room for monotone improvement or monotone decline in other geometries. If Vinter,n(K)V_inter,n(K) varies with selection, its averaged increment must be retained, and if the branch cross-covariance varies with selection, the corresponding increment follows from Eq. (A4). A specific bias model makes a finite optimum explicit. Take the illustrative pair Bsoup(K)=Bsoup(1)+γ(K−1)2B_soup(K)=B_soup(1)+γ(K-1)^2 with γ>0γ>0 and Keff(K)=K_eff(K)=K, which is a specialization we impose rather than a consequence of validation ranking. For continuous K≥1K≥ 1 the resulting quadratic surrogate is ΦN(K)=Bsoup(1)+γ(K−1)2+VintraNK+VinterN. _N(K)=B_soup(1)+γ(K-1)^2+ V_intraNK+ V_interN. (A7) Its derivatives are ΦN′(K)=2γ(K−1)−Vintra/(NK2) _N (K)=2γ(K-1)-V_intra/(NK^2) and ΦN′(K)=2γ+2Vintra/(NK3)>0 _N (K)=2γ+2V_intra/(NK^3)>0, so ΦN _N is strictly convex on K≥1K≥ 1. Its derivative is negative at K=1K=1 and tends to +∞+∞, so there is a unique continuous minimizer KN⋆>1K_N >1, characterized by 2γ(KN⋆)2(KN⋆−1)=Vintra/N2γ(K_N )^2(K_N -1)=V_intra/N. The map K↦K2(K−1)K K^2(K-1) is strictly increasing for K>1K>1 because its derivative is K(3K−2)>0K(3K-2)>0, and the right-hand side decreases in N, so the preferred count KN⋆K_N decreases as branches are added. On a bounded interval of feasible counts, strict convexity means an integer optimum lies among the feasible floor and ceiling of the clipped continuous minimizer, and a sparse experimental grid requires comparing the available grid points directly. This specialization explains how additional branches can reduce the marginal value of temporal averaging, and it is consistent with the checkpoint-count shift reported in Sec. 5.3.2. Its predicted count depends on the assumed bias law, the available checkpoints, and the residual correlations, so a K-dependent branch component or a departure from the quadratic model changes the objective. A.3 The Fixed-Budget Allocation Tradeoff The comparisons above keep the per-branch horizon fixed. The Limited setting instead fixes the total branch-training token budget T and uses t=T/Nt=T/N, and restoring that dependence makes the competition between serial progress and averaging explicit. Let ℰTraj-Soup(T,N,K)E_Traj -Soup(T,N,K) denote the expected validation excess loss of θTraj-Soup(N,T/N,K) _Traj -Soup(N,T/N,K), and suppose Theorem 4.3 holds at every compared horizon with a common reference θ⋆θ and curvature H, which is what makes those excess losses comparable. Writing the branch horizon as an explicit first argument of the bias, of the two loss-weighted variances, and of the effective count, and substituting t=T/Nt=T/N into Eq. (5), gives ℰTraj-Soup(T,N,K)=Bsoup(T/N,K)+Vintra(T/N)NKeff(T/N,K)+Vinter(T/N)NE_Traj -Soup(T,N,K)=B_soup(T/N,K)+ V_intra(T/N)NK_eff(T/N,K)+ V_inter(T/N)N (A8) for any feasible N and K. Increasing N supplies more branches while shortening each one. Shorter horizons may retain greater finite-training bias, averaging may reduce both persistent and residual variation, and checkpoint availability constrains the feasible K. In a serial-saturation regime the cost of shortening a branch can be small enough for the averaging gains to dominate, whereas earlier in training substantial useful serial progress can favor longer branches. This is a loss-accounting identity rather than an allocation rule, since it imposes the budget constraint without supplying the horizon dependence of the individual terms. An optimal N and K therefore requires further information about those functions and about the feasible candidate sets, and the budget counts training tokens, whose relation to matched compute follows the experimental accounting for the compared recipes and architectures. A.4 Variance-Optimal Weights at Both Levels Theorem 4.3 takes the two uniform averages of Trajectory Soup as given, which invites the question of whether uniform weights are the right choice or merely a convenient one. For independent errors the answer is classical, since the covariance analysis of averaging assigns equal weight to exchangeable contributions (Bishop, 2006). Transferring that principle to our setting requires care, because the averaged quantities are parameter fluctuations weighted by the validation Hessian, and the checkpoints entering each branch anchor are selected by validation loss rather than drawn independently. SWA also averages iterates uniformly (Izmailov et al., 2018), but for selected checkpoints the optimality of that choice depends on their covariance structure. This subsection makes that dependence explicit and states the result that Sec. 4.3 reports in prose. Fix N, K, t, and the checkpoint-selection rule, and replace the two uniform averages by deterministic nonnegative weights satisfying ∑n=1Nαn=1 _n=1^N _n=1 and ∑j=1Kwn,j=1 _j=1^Kw_n,j=1 for each branch. The resulting model is θα,w=∑n=1Nαn∑j=1Kwn,jθn,(j) _α,w= _n=1^N _n _j=1^Kw_n,j _n,(j), which recovers θTraj-Soup _Traj -Soup of Eq. (2) at αn=1/N _n=1/N and wn,j=1/Kw_n,j=1/K. Theorem 4.1 splits its excess loss into [ℒQ(θα,w)]−ℒ⋆=Bα,w(K)+α,w(K),E[L_Q( _α,w)]-L =B_α,w(K)+V_α,w(K), (A9) where Bα,w(K)=12‖[θα,w]−θ⋆‖H2B_α,w(K)= 12\|E[ _α,w]-θ \|_H^2 and α,w(K)=12tr(HCov(θα,w))V_α,w(K)= 12tr(HCov( _α,w)). These weights act on the covariance of the conditionally centered residuals of Appendix A.1, which we collect into the positive semidefinite matrix [Gnϵ(K)]jj′=12tr(HCov(ϵn,j,ϵn,j′))[G_n^ε(K)]_j = 12tr(HCov( _n,j, _n,j )). Uniform temporal weights become variance-optimal as soon as every selected checkpoint carries the same aggregate covariance with the selected set. This condition permits correlated residuals and holds for exchangeable covariance, so it is weaker than independence. Throughout, Keff(K)K_eff(K) keeps the definition it received for the uniformly averaged anchor in Sec. 4.3. Corollary A.1 (Variance-optimal averaging at both levels). Under the homogeneous contributions of Theorem 4.3, assume that each selected rank has the same persistent conditional shift unu_n and that the full collections (un,ϵn,1,…,ϵn,K)(u_n, _n,1,…, _n,K) are independent across branches. If Gnϵ(K)K=λnKG_n^ε(K)1_K= _n1_K for some scalar λn _n for every branch, then α,w(K)≥VintraNKeff(K)+VinterN.V_α,w(K)≥ V_intraNK_eff(K)+ V_interN. (A10) Uniform temporal and branch weights, wn,j=1/Kw_n,j=1/K and αn=1/N _n=1/N, attain this bound. Equal row sums and positive semidefiniteness make uniform temporal weights minimize each residual quadratic form, and the branch-level minimum then follows from ∑n=1Nαn2≥1/N _n=1^N _n^2≥ 1/N. Appendix B.4 carries out both steps and prices the two ways of departing from Trajectory Soup separately, charging a quadratic penalty for imbalance across branches and another for unequal temporal weights within them. Because both penalties are nonnegative, the minimum in Eq. (A10) is exactly the stochastic contribution of Eq. (5), so the two uniform averages of our method receive a single joint justification rather than two separate ones. The conclusion nevertheless constrains the variance alone, since changing the temporal weights moves the mean selected displacement and hence Bα,w(K)B_α,w(K), so uniform weights minimize the total loss only when that bias is constant across the admissible weights, as happens when all selected checkpoint means coincide. This remaining gap is what leaves room for deliberately nonuniform temporal weights, while the weighting and allocation ablations of Sec. 5.3.2 settle the practical choice. Heterogeneous or correlated branches, for which inverse-variance branch weights are preferable, are treated in Appendix A.1. A.5 A General Criterion for Uniform Weighting The equal-row-sum hypothesis of Corollary A.1 is an instance of a general fact about quadratic forms on the simplex, namely that uniform weights are optimal precisely when every member carries the same aggregate covariance with the collection. Let A∈ℝm×mA ^m× m be symmetric positive semidefinite and let p=m−1mp=m^-11_m. Then p minimizes a⊤Aa Aa over ai≥0a_i≥ 0 with m⊤a=11_m a=1 if and only if Am=λmA1_m= 1_m for some scalar λ. For the forward direction, write d=a−pd=a-p so that m⊤d=01_m d=0, and expand a⊤Aa=p⊤Ap+2d⊤Ap+d⊤Ada Aa=p Ap+2d Ap+d Ad. The row-sum condition gives Ap=(λ/m)mAp=(λ/m)1_m, which annihilates the mixed term and leaves a⊤Aa=λ/m+d⊤Ada Aa=λ/m+d Ad, so positive semidefiniteness establishes optimality and positivity on nonzero zero-sum directions establishes uniqueness. Conversely, if p is optimal, then every entry of p is positive, so p±τdp±τ d remains feasible for small τ>0τ>0 along any zero-sum d. The directional derivative at p must therefore vanish, and d⊤Ap=0d Ap=0 for every such d forces ApAp to be proportional to m1_m. Applying this criterion to A=Gnϵ(K)A=G_n^ε(K) gives the within-trajectory condition used in Corollary A.1. For the uniform anchor, p⊤Gnϵ(K)p=Vintra,n(K)=Vintra/Keff(K)p G_n^ε(K)p=V_intra,n(K)=V_intra/K_eff(K), so the common row sum must equal KVintra/Keff(K)KV_intra/K_eff(K). The same criterion applies to the branch covariance matrix G(K)G(K) in Eq. (A3), and in particular independent anchors with equal variance contributions have G(K)=v(K)ING(K)=v(K)I_N and therefore admit uniform variance-optimal branch weights. An explicit correlated temporal model shows what the criterion buys. Suppose K≥2K≥ 2 and Gnϵ(K)=(Vintra−cϵ)IK+cϵKK⊤G_n^ε(K)=(V_intra-c_ε)I_K+c_ε1_K1_K with 0≤cϵ≤Vintra0≤ c_ε≤ V_intra, which assumes exchangeability of the loss-weighted covariance without requiring identical checkpoint means or independent residuals. Normalization gives wn⊤Gnϵ(K)wn=cϵ+(Vintra−cϵ)‖wn‖22w_n G_n^ε(K)w_n=c_ε+(V_intra-c_ε)\|w_n\|_2^2, so the bound ‖wn‖22≥1/K\|w_n\|_2^2≥ 1/K shows that uniform weights attain Vintra,n(K)=cϵ+(Vintra−cϵ)/KV_intra,n(K)=c_ε+(V_intra-c_ε)/K, equivalently Keff(K)=KVintra/[Vintra+(K−1)cϵ]K_eff(K)=KV_intra/[V_intra+(K-1)c_ε], and do so uniquely when cϵ<Vintrac_ε<V_intra. At cϵ=Vintrac_ε=V_intra every normalized weighting carries the same residual variance, so temporal averaging provides no variance reduction at all. A.6 Scope of the Theoretical Claims The derivations require the local quadratic model of Sec. 4.1, finite second moments, and a shared parameterization. Theorem 4.3 additionally assumes the shared-shift condition of Appendix A.1, independent selected anchors, and common loss-weighted contributions across branches. The general covariance treatment of Appendix A.1 relaxes the last two of these, retaining the full anchor cross-covariance and allowing heterogeneous branch contributions. These assumptions are sufficient for the stated identities and are not necessary conditions for empirical merging gains. The theory addresses population validation loss under Q. It does not directly imply the ordering of discrete benchmark accuracies or post-training outcomes, which are supported by the corresponding experiments. It also does not imply that all recipe deviations are zero-mean noise, that every additional branch improves the model, or that the local quadratic model remains accurate at the displacements the merges actually traverse. Appendix B Proofs of the Main-Text Results This appendix proves Theorems 4.1–4.3 and Corollary A.1. The extensions of Appendix A are derived where they are stated. All expectations use the fixed setup specified in Sec. 4.1, and every statement is proved under the local quadratic model adopted there, so all identities below are exact. Second moments are assumed finite whenever the corresponding quantities occur. The positive semidefinite matrix H may be singular, so ∥⋅∥H\|·\|_H can be a seminorm. None of the proofs requires H to be invertible. B.1 Proof of Theorem 4.1 Proof. Let μ=[ϑ]μ=E[ ] and write ϑ−θ⋆=(μ−θ⋆)+(ϑ−μ) -θ =(μ-θ )+( -μ). Expanding the quadratic form and taking expectations gives [‖ϑ−θ⋆‖H2]= [\| -θ \|_H^2]= ‖μ−θ⋆‖H2+2(μ−θ⋆)⊤H[ϑ−μ] \|μ-θ \|_H^2+2(μ-θ ) HE[ -μ] (A11) +[(ϑ−μ)⊤H(ϑ−μ)]. +E[( -μ) H( -μ)]. The middle term vanishes because [ϑ−μ]=0E[ -μ]=0. The last term is [(ϑ−μ)⊤H(ϑ−μ)]=tr(H[(ϑ−μ)(ϑ−μ)⊤])=tr(HCov(ϑ)).E[( -μ) H( -μ)]=tr\! (HE[( -μ)( -μ) ] )=tr\! (HCov( ) ). (A12) Taking expectations in the local quadratic model of Sec. 4.1 now gives Eq. (3). The squared bias is nonnegative because H⪰0H 0. The variance term is nonnegative because it equals [‖H1/2(ϑ−μ)‖22]E[\|H^1/2( -μ)\|_2^2]. The two terms therefore account for the entire expected excess loss under the local quadratic model. ∎ B.2 Proof of Theorem 4.2 Proof. For fixed n and K, decompose the centered anchor as θ¯n−μn(K)=mn(K)−μn(K)+θ¯n−mn(K) θ_n- _n(K)=m_n(K)- _n(K)+ θ_n-m_n(K). The conditional residual has zero mean given ℱn F_n, while mn(K)−μn(K)m_n(K)- _n(K) is ℱn F_n-measurable. Their mixed second moment is therefore zero. Expanding the covariance yields the law of total covariance, Cov(θ¯n)=[Cov(θ¯n∣ℱn)]+Cov(mn(K)).Cov( θ_n)=E[Cov( θ_n F_n)]+Cov(m_n(K)). (A13) Substituting this identity into Theorem 4.1 gives Eq. (4). Each covariance contribution is nonnegative after taking its trace against H, by the argument used in Appendix B.1. If mn(K)=μn(K)+unm_n(K)= _n(K)+u_n with the same mean-zero unu_n for every feasible K, then Cov(mn(K))=Cov(un)Cov(m_n(K))=Cov(u_n). Thus Vinter,n(K)=Vinter,n=12tr(HCov(un))V_inter,n(K)=V_inter,n= 12tr(HCov(u_n)) is independent of K, which is the shared-shift condition assumed by Theorem 4.3. At K=1K=1, the anchor equals θn,(1) _n,(1). Subtracting the K=1K=1 instance of Eq. (4) then cancels both ℒ⋆L and Vinter,nV_inter,n, leaving [ℒQ(θ¯n)]−[ℒQ(θn,(1))]=Bn(K)−Bn(1)+Vintra,n(K)−Vintra,n(1),E[L_Q( θ_n)]-E[L_Q( _n,(1))]=B_n(K)-B_n(1)+V_intra,n(K)-V_intra,n(1), so merging the top K checkpoints of a branch improves on its best single checkpoint exactly when the intra-trajectory term falls by more than the bias rises. ∎ B.3 Proof of Theorem 4.3 Proof. Linearity of conditional expectation gives ϵ¯n(K)=θ¯n−mn(K) ε_n(K)= θ_n-m_n(K). Since [ϵ¯n(K)∣ℱn]=0E[ ε_n(K) F_n]=0, Cov(ϵ¯n(K))=[Cov(θ¯n∣ℱn)],Cov(θ¯n)=Cov(ϵ¯n(K))+Cov(un).Cov( ε_n(K))=E[Cov( θ_n F_n)], ( θ_n)=Cov( ε_n(K))+Cov(u_n). (A14) The first identity also establishes Vintra,n(K)=12tr(HCov(ϵ¯n(K)))V_intra,n(K)= 12tr(HCov( ε_n(K))), which is the representation used in Sec. 4.3. Expanding the covariance of the finite residual average gives Cov(ϵ¯n(K))=1K2∑j,j′=1KCov(ϵn,j,ϵn,j′).Cov( ε_n(K))= 1K^2 _j,j =1^KCov( _n,j, _n,j ). (A15) By linearity of expectation, [θTraj-Soup]=N−1∑n=1Nμn(K)E[ _Traj -Soup]=N^-1 _n=1^N _n(K), so the term Bsoup(K)B_soup(K) defined in Theorem 4.3 is the squared curvature-weighted bias of the soup. Eq. (A2) equivalently writes this displacement as [θTraj-Soup]−θ⋆=bcom+1N∑n=1N(brec,n+hn(K)).E[ _Traj -Soup]-θ =b_com+ 1N _n=1^N (b_rec,n+h_n(K) ). (A16) Independence of the selected anchors gives Cov(θTraj-Soup)=1N2∑n=1NCov(θ¯n).Cov( _Traj -Soup)= 1N^2 _n=1^NCov( θ_n). (A17) Taking the loss-weighted trace and using the common scalar contributions, 12tr(HCov(θTraj-Soup))=1N2∑n=1N(VintraKeff(K)+Vinter)=VintraNKeff(K)+VinterN. 12tr\! (HCov( _Traj -Soup) )= 1N^2 _n=1^N ( V_intraK_eff(K)+V_inter )= V_intraNK_eff(K)+ V_interN. (A18) Applying Theorem 4.1 to the soup gives Eq. (5). For completeness, the rows of Tab. 1 follow from the same identities. A rank-1 checkpoint uses Keff(1)=1K_eff(1)=1. A branch anchor retains its full persistent branch contribution. A cross-trajectory merge of rank-1 checkpoints sets K=1K=1, and Trajectory Soup uses the full expression. Each operation has its own mean model, so the bias terms differ across rows. ∎ B.4 Proof of Corollary A.1 Proof. Let μn,j=[θn,(j)] _n,j=E[ _n,(j)]. Under the rank-independent conditional shift, θn,(j)=μn,j+un+ϵn,j _n,(j)= _n,j+u_n+ _n,j with [ϵn,j∣ℱn]=0E[ _n,j F_n]=0. Because unu_n is ℱn F_n-measurable, Cov(un,ϵn,j)=0Cov(u_n, _n,j)=0. The centered weighted anchor is therefore ∑j=1Kwn,jθn,(j)−∑j=1Kwn,jμn,j=un+∑j=1Kwn,jϵn,j _j=1^Kw_n,j _n,(j)- _j=1^Kw_n,j _n,j=u_n+ _j=1^Kw_n,j _n,j, so its loss-weighted variance equals wn⊤Gnϵ(K)wn+Vinterw_n G_n^ε(K)w_n+V_inter. Independence of the branch collections then gives α,w(K)=∑n=1Nαn2[wn⊤Gnϵ(K)wn+Vinter].V_α,w(K)= _n=1^N _n^2 [w_n G_n^ε(K)w_n+V_inter ]. (A19) Write p=K−1Kp=K^-11_K, dn=wn−pd_n=w_n-p, and v(K)=Vintra/Keff(K)+Vinterv(K)=V_intra/K_eff(K)+V_inter, so that K⊤dn=01_K d_n=0. The equal-row-sum hypothesis gives Gnϵ(K)p=K−1λnKG_n^ε(K)p=K^-1 _n1_K, whence Vintra,n(K)=p⊤Gnϵ(K)p=λn/KV_intra,n(K)=p G_n^ε(K)p= _n/K. By the definition of Keff(K)K_eff(K) in Sec. 4.3, this quantity equals Vintra/Keff(K)V_intra/K_eff(K), so the common row sum is λn=KVintra/Keff(K) _n=KV_intra/K_eff(K) and Gnϵ(K)p=(Vintra/Keff(K))KG_n^ε(K)p=(V_intra/K_eff(K))1_K. The mixed term dn⊤Gnϵ(K)pd_n G_n^ε(K)p therefore vanishes, and consequently wn⊤Gnϵ(K)wn=VintraKeff(K)+dn⊤Gnϵ(K)dn.w_n G_n^ε(K)w_n= V_intraK_eff(K)+d_n G_n^ε(K)d_n. (A20) Normalization of the branch weights yields ∑n=1Nαn2=1N+∑n=1N(αn−1N)2≥1N, _n=1^N _n^2= 1N+ _n=1^N ( _n- 1N )^2≥ 1N, (A21) so substituting Eq. (A20) into Eq. (A19) gives α,w(K)=v(K)N+v(K)∑n=1N(αn−1N)2+∑n=1Nαn2dn⊤Gnϵ(K)dn.V_α,w(K)= v(K)N+v(K) _n=1^N ( _n- 1N )^2+ _n=1^N _n^2d_n G_n^ε(K)d_n. (A22) Positive semidefiniteness of each Gnϵ(K)G_n^ε(K) makes both penalty terms nonnegative. Uniform weights at both levels make them vanish and attain Eq. (A10), and Theorem 4.1 then gives Eq. (A9). If v(K)>0v(K)>0, equality requires αn=1/N _n=1/N. If, in addition, every Gnϵ(K)G_n^ε(K) is positive definite on the subspace orthogonal to K1_K, equality also requires wn=K−1Kw_n=K^-11_K for every branch, so the joint variance minimizer is unique under these conditions. The bias term remains weight-dependent in general. ∎ Appendix C Compute-Scaling and Geometry Protocols This appendix describes the compute accounting and scaling fits in Fig. 1(b), together with the geometric measurements used in Sec. 2 and in the pairwise analysis of Sec. 5.3.2. C.1 Compute Axis and Scaling-Law Fits Compute accounting. The horizontal axis shows estimated cumulative mid-training FLOPs on a logarithmic scale, using a sequence length of 262,144262,144 and an estimated cost of 36.7436.74 GFLOPs per token. Token offsets are measured from the shared fork point and summed across branches: C≃3.674×1010D,D=∑n=1Ntn,C 3.674× 10^10D, D= _n=1^Nt_n, (A23) where tnt_n is the token horizon of branch n. Equal branch horizons give D=NtD=Nt. This accounting covers branch training after the shared fork point and excludes the common pretraining prefix. It also leaves out checkpoint storage, the validation passes used for ranking, and the search over how many trajectories and checkpoints to combine, as discussed in Appendix . The vertical axis reports the Overall Average Accuracy of Appendix D.4 in percent, without any rescaling. Scaling fits. Inspired by power-law descriptions of language-model loss (Kaplan et al., 2020; Hoffmann et al., 2022), we fit an empirical asymptotic inverse power law directly to the accuracy scores: S(C)=A−B(C/C0)−α,B>0,α>0,S(C)=A-B(C/C_0)^-α, B>0, α>0, (A24) where S(C)S(C) is the Overall Average Accuracy in percent at cumulative compute C, A is the fitted asymptote, and C0C_0 is the cumulative FLOPs of the first fitted point on the corresponding curve. The Raw fit uses n=17n=17 Raw checkpoints from 25B to 425B tokens, and the Trajectory Soup fit uses the n=10n=10 frontier points. The fitted curves are Sraw(C) S_raw(C) =67.2319−1.3393(C9.1852×1020)−0.45531, =67.2319-1.3393 ( C9.1852× 10^20 )^-0.45531, (A25) Ssoup(C) S_soup(C) =71.0534−2.8580(C1.8370×1021)−0.08818. =71.0534-2.8580 ( C1.8370× 10^21 )^-0.08818. The Raw and Trajectory Soup frontier fits have R2=0.594R^2=0.594 and R2=0.965R^2=0.965, respectively. Their reference compute levels C0C_0 correspond to 25B and 50B processed tokens. These fits summarize the trends in the plotted data. Displayed range and annotations. The Raw fit is shown as a solid line up to 425B tokens and as a dashed extrapolation beyond that horizon; the extrapolated segment does not enter the fit. At 100B-equivalent compute (3.6741×10213.6741× 10^21 FLOPs), “Quality superiority” compares the fitted accuracies of Trajectory Soup and Raw training, 68.36%68.36\% and 66.52%66.52\%, at the same budget. “Scaling robustness” connects the Raw extrapolation at 600B-equivalent compute (2.2044×10222.2044× 10^22 FLOPs, 66.92%66.92\%) to the observed four-trajectory result at 2400B-equivalent compute (8.8178×10228.8178× 10^22 FLOPs, 69.03%69.03\%). This annotation highlights the observed improvement from expanding the trajectory budget alongside the regression of later Raw checkpoints. C.2 Trajectory Directions and Pairwise Alignment All branches share the architecture and expert layout of Appendix D.1 and are forked from a single checkpoint, so parameters correspond entrywise across branches and no permutation alignment is applied before averaging or measuring angles. For each branch, we first merge checkpoints along its training trajectory and apply PCA to the centered sequence of resulting merged checkpoints over a matched token interval. The approximately rank-one variation of these merged trajectories motivates using the unit leading principal component dnd_n as the direction of branch n (Zhou et al., 2026), oriented along the net displacement of the merged trajectory over that interval. Because the directions are unit vectors, their pairwise alignment is the cosine cnn′=dn⊤dn′c_n =d_n d_n , with associated angle ϕnn′=arccos(cnn′) _n = (c_n ). Fig. 3 reports these cosine similarities for the five compatible branches of Sec. 2. Values near 11 indicate aligned directions, while values near 00 indicate nearly orthogonal directions. C.3 Interpolation Between Branch Anchors At each matched token horizon t, we take the merged checkpoints θ¯n(t,K) θ_n(t,K) and θ¯n′(t,K) θ_n (t,K) from two screened branches and interpolate their parameters: θnn′(λ)=(1−λ)θ¯n(t,K)+λθ¯n′(t,K),λ∈[0,1]. _n (λ)=(1-λ) θ_n(t,K)+λ θ_n (t,K), λ∈[0,1]. (A26) The endpoints λ∈0,1λ∈\0,1\ are the within-trajectory merged anchors and λ=1/2λ=1/2 is their equal-weight cross-trajectory merge. We evaluate θnn′(λ) _n (λ) on the validation corpus over a grid of λ and retain the best interpolation at each horizon. The left panel of Fig. 3 plots this quantity against consumed tokens, alongside the raw checkpoints of the two branches and their within-trajectory merges; the right panel shows the corresponding interpolation paths in a two-dimensional projection of the weight space, one path per horizon. The comparison therefore measures the additional benefit of combining branch anchors after within-trajectory averaging. C.4 Pairwise Cosine and Relative Interpolation Gain The pairwise cosine in Fig. 9 is computed from the same leading PCA directions as in Appendix C.2. Relative gain measures the improvement of the merged model over the midpoint of the two endpoint scores. Let AnA_n and An′A_n denote the endpoint scores and Ann′A_n the score after merging the pair. Then Δnn′=Ann′−12(An+An′) _n =A_n - 12(A_n+A_n ), reported in points of the overall evaluation score. The subtracted term is the midpoint obtained by linearly interpolating the endpoint scores, so the gain is relative to that reference rather than to an absolute scale, and Δnn′>0 _n >0 means the merge improves on the mean of its endpoints. The scatter relates this gain, averaged over 17 token-budget offsets, to cnn′c_n across branch pairs. Its overlaid line is an ordinary-least-squares fit with a 95%95\% confidence band for the mean response. Completion fields. For the geometric protocols above, state the intra-trajectory averaging rule and window applied before PCA, the matched token interval and the number of merged points entering each fit, the variance share carried by the leading component, the interpolation grid and the token horizons of Fig. 3, and the 17 token-budget offsets aggregated in Fig. 9. For the compute axis, state the (N,t)(N,t) configuration behind each plotted Trajectory Soup point and the number of points entering each scaling fit. Appendix D Experimental Configuration This appendix collects the configuration of the model, the training recipe, the branch identities, and the selection and evaluation protocol. It distinguishes the fields present in the current draft from information still needed for a fully reproducible submission. The measurement conventions for the compute-scaling figure and the geometric observations are given separately in Appendix C. Completion fields are author-facing placeholders and are not additional experimental results. D.1 Default Model Configuration The core architecture is a sparse mixture-of-experts (MoE) language model with 7.9B total parameters and 1.3B parameters activated per token. The model has 24 layers, each with 128 routed experts and 8 active routed experts per token, as detailed in Tab. 6. Table 6: Default model configuration. The architectural values are taken from the supplied model description and are not independently verified through a public model release. Field Supplied value Model name Ling-3.0-tiny Total parameters 7.9B Active parameters per token 1.3B Layers 24 Hidden width 1,536 First feedforward layer Dense Routed experts in each subsequent layer 128 Active routed experts per token 8 Shared experts per MoE layer 1 Expert feedforward width 512 Attention components MLA and KDA linear attention Query LoRA rank 256 KV LoRA rank 512 Vocabulary size 157,184 Tokens consumed before branching 30T Mid-training sequence length 262,144 Estimated cost per training token 36.74 GFLOPs D.2 Training Settings Shared by All Branches Every branch is forked from the same pretrained checkpoint and inherits the settings of Tab. 7. The fields along which the branches deliberately differ are held out of this table and listed in Appendix D.3, so that the two tables together specify each recipe exactly. Table 7: Training settings shared by all branches. Values recorded in the supplied recipe and held fixed across EXP1–EXP5. The perturbed fields are given in Tab. 8. Field Value Complete branch horizon 600B tokens Optimizer Muon Warmup 1% of training tokens Weight decay 0.1 Gradient clipping 1.0 Optimizer β1 _1 0.9 Optimizer β2 _2 0.95 Default checkpoint interval 25B tokens Dense-sampling ablation interval 12.5B tokens D.3 Branch Identity and Recipe Mapping Each branch perturbs the EXP1 baseline along one of the dimensions admitted in Sec. 3.1, except for EXP3, which varies the peak learning rate and the global batch size together. Tab. 8 lists the five recipes, with the perturbed fields of each branch in bold. The default comparison of Sec. 5.2 uses EXP1–EXP3, the four- and five-branch configurations of Fig. 1(b) and Sec. 5.3.2 add EXP4 and EXP5 in that order, and the geometric analyses of Sec. 2 and Appendix C use all five. All five branches pass the compatibility screen of Sec. 3.1 at the common 600B-token horizon. The smaller-model reproduction of Appendix has its own pair of branches, whose labels are local to that experiment and do not identify the same parameter checkpoints. Table 8: Branch identity and recipe mapping. The five branches forked from the common pretrained checkpoint. Bold entries mark the fields a branch changes relative to the EXP1 baseline; all remaining settings follow Tab. 7. Branch Perturbed dimension Data-shuffling seed Peak learning rate Global batch Schedule after warmup Muon momentum EXP1 Baseline 1234 3.39×10−43.39× 10^-4 256 Constant 0.0 EXP2 Data order 1001 3.39×10−43.39× 10^-4 256 Constant 0.0 EXP3 Learning rate and batch size 1234 2.40×− 2.40× 10^-4 128 Constant 0.0 EXP4 Learning-rate schedule 1234 3.39×10−43.39× 10^-4 256 Decay to 3.39×− 3.39× 10^-5 0.0 EXP5 Optimizer momentum 1234 3.39×10−43.39× 10^-4 256 Constant 0.9 D.4 Evaluation Protocol Pre-training evaluation. We evaluate base-model checkpoints before post-training using 41 benchmark configurations across five capability categories: (i) general knowledge and reasoning, including ARC-Easy and ARC-Challenge (Clark et al., 2018), AGIEval (Zhong et al., 2023), BBH and its Chinese configuration (Suzgun et al., 2023), WorldSense (Benchekroun et al., 2023), PIQA (Bisk et al., 2020), and HellaSwag (Zellers et al., 2019); (i) language understanding, including RACE-Middle and RACE-High (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018), TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), WinoGrande (Sakaguchi et al., 2019), Belebele (Bandarkar et al., 2024), and CCPM (Li et al., 2021); (i) professional knowledge and factuality, including MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang et al., 2024), SuperGPQA (Team et al., 2025), CMMLU (Li et al., 2024a), C-Eval (Huang et al., 2023), SimpleQA (Wei et al., 2024), and Chinese SimpleQA (He et al., 2024); (iv) mathematics, including GSM8K (Cobbe et al., 2021), GSM-Plus (Li et al., 2024b), the Chinese subset of MGSM (Shi et al., 2022), MATH and its Minerva evaluation configuration (Hendrycks et al., 2021; Lewkowycz et al., 2022), CMATH (Wei et al., 2023), MathBench (Liu et al., 2024), CollegeMath (Tang et al., 2024), ZKMathUnion, and GKMathUnion; and (v) coding, including the English and Chinese configurations of HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), HumanEval+ and MBPP+ (Liu et al., 2023), LiveCodeBench (Jain et al., 2025), CRUXEval (Gu et al., 2024), and BIRD-SQL (Li et al., 2023a). Post-training evaluation. To assess whether mid-training gains persist after post-training, we evaluate models on 16 configurations grouped into six scenarios: (i) coding, comprising LiveCodeBench v6 (Jain et al., 2025), SciCode (Tian et al., 2024), and Aider (Aider, 2026); (i) mathematics, comprising HMMT February 2026 (HMMT, 2026), AIME 2025, and AIME 2026 (Mathematical Association of America, 2026); (i) reasoning, comprising KOR-Bench (Ma et al., 2024) and GPQA (Rein et al., 2023); (iv) knowledge, comprising MMLU-Pro (Wang et al., 2024), Chinese SimpleQA (He et al., 2024), C-Eval (Huang et al., 2023); (v) instruction following, comprising, IFEval (Zhou et al., 2023), IFBench (Pyatkin et al., 2025); and (vi) function calling, evaluated using BFCL v4 in function-calling mode (Patil et al., 2025). D.5 Merging Baselines The merging scopes compared in Sec. 5.1 all act on the same candidate pools n(t)C_n(t) and all return a uniform average of some subset of those checkpoints, so they differ only in which checkpoints enter the average. We state each one in the notation of Sec. 3. Single EXP Merge. This baseline merges within one trajectory and performs no cross-trajectory combination, so it is exactly the branch anchor θ¯n(t,K) θ_n(t,K) of Eq. (1): the uniform average of the K checkpoints of n(t)C_n(t) that rank highest on the selection objective. Equivalently, it is the N=1N=1 case of Eq. (2), and it therefore exercises only the intra-trajectory term Vintra,n(K)V_intra,n(K) of Eq. (4). Every branch produces its own merge; the tables report the strongest of them under the name Single-Trajectory Merge, evaluated at K=12K=12 so that it averages as many checkpoints as the three-branch Trajectory Soup with K=4K=4. Appendix E lists the merges of all branches. Model Soup. This baseline is the mirror image: it merges across trajectories and performs no within-trajectory selection or averaging. It takes the final checkpoint of each branch at the evaluated horizon and averages the N of them uniformly, θModel-Soup(N,t)=1N∑n=1Nθn(t), _Model -Soup(N,t)= 1N _n=1^N _n(t), (A27) in the spirit of Wortsman et al. (2022), with membership fixed by the admitted branch set rather than chosen by a validation sweep over candidate members. It is not the K=1K=1 case of Eq. (2): that case still ranks each pool and keeps its best element, whereas Eq. (A27) fixes the position at the horizon. Contrasting the two baselines with Trajectory Soup therefore separates the benefit of combining branches from the benefit of selecting what each branch contributes, and the Limited and Extended conventions of Sec. 5.1 determine only the horizon t at which the final checkpoints are taken. Selection objective. Eq. (1) ranks candidates by validation loss ℒ^val L_val. The tables reported here instead rank them by validation accuracy, and the cardinality K of every entry is the best value evaluated on that same objective. Reranking the existing checkpoints by validation loss defines a distinct selection variant whose results would have to come from an actual rerun, since the accuracy-selected tables do not establish that the two selectors agree. Because one validation signal both orders the checkpoints and chooses the reported (N,K)(N,K), the selection signal and the reported score are not fully independent. Completion fields. Specify the validation corpus, token masking, weighting, and loss normalization behind ℒ^val L_val, together with the tasks, examples, and aggregation behind the accuracy-based selector and the benchmark weights that form the Overall Average of Appendix D.4. Record the full K grid, the tie-breaking rule, and the split held out for selection, as well as the token position of every reported optimum and the exact Limited horizon of each Model Soup entry. For the post-training comparison, state the SFT data, tokens, optimizer, schedule, and random seeds held fixed across starting checkpoints. D.6 Ablation Schemes Every row of Tab. 5 follows the Trajectory Soup (Extended) setting of Tab. 2 and changes exactly one ingredient of Eq. (2): either the coefficients applied inside a branch or the index set those coefficients are applied to. Writing wn,i≥0w_n,i≥ 0 for the weight of checkpoint i in branch n and ℐnI_n for its selected set, with ∑i∈ℐnwn,i=1 _i _nw_n,i=1, every evaluated variant has the form 1N∑n=1N∑i∈ℐnwn,iθn(i) 1N _n=1^N _i _nw_n,i _n(i). The branch-level weights thus stay uniform throughout, and the ablation varies only the inner stage. The baseline row pairs the equal coefficients with the Top-K each selection, merging K=4K=4 checkpoints from each of the three branches; every other row replaces one of the two components and keeps the other at this setting. Coefficient schemes. The first group fixes the selected sets to Top-K each and varies the weights inside a branch. Let j denote the within-trajectory rank of a selected checkpoint on the selection objective, with j=1j=1 the best, and normalize each weight vector to sum to one over the K checkpoints of a branch. equal. Uniform temporal averaging, wj=1/Kw_j=1/K, as prescribed by Eq. (2). This is Trajectory Soup itself, and its overall average of 68.96 is the Trajectory Soup (Extended) entry of Tab. 2. Sec. 4.3 identifies it as variance-optimal whenever every selected checkpoint of a branch shares the same aggregate curvature-weighted covariance with its selected set, so the remaining three schemes test how much that condition matters in practice. 1sqrt. Weights wj∝jw_j j, increasing in the rank index and therefore shifting mass away from the best checkpoints toward the lower-ranked ones. The spread is mild: the K-th member receives K K times the weight of the first. rank. Weights wj∝jw_j j, increasing in the same direction but more steeply, with the K-th member receiving K times the weight of the first. Comparing it with 1sqrt separates the direction of the tilt from its magnitude. rsqrt. Weights wj∝1/jw_j 1/ j, decreasing in the rank index and thus concentrating mass on the highest-ranked checkpoints. Together with the two increasing schemes it brackets uniform averaging from both sides, so the group probes tilts toward better and toward worse members rather than only one of them. Selection strategies. The second group fixes the coefficients to equal and varies which elements of the candidate pools n(t)C_n(t) enter the average. Top-K each. The default rule of Eq. (1), taking ℐn=ℐn(t,K)I_n=I_n(t,K) as the K highest-ranked checkpoints of each branch. It imposes both a quality criterion and a per-branch quota, and the three alternatives below drop one of these two properties each. Global top-NKNK. Keeps the quality criterion and removes the quota, taking the NKNK highest-ranked checkpoints from the pooled candidates of all branches. The merge size matches the baseline, but the branches may be represented unequally, and a branch whose checkpoints score uniformly well can dominate the average. The comparison therefore isolates the value of balanced trajectory representation. Tail-K per branch. Keeps the quota and the merge size but replaces quality ranking by recency, taking the K latest saved positions of each branch. Since late checkpoints are neither the best nor the most diverse, this isolates the value of ranking candidates at all, as opposed to simply averaging the end of each trajectory. All checkpoints. Abandons selection entirely, setting ℐn=n(t)I_n=C_n(t) for every branch and averaging the pooled candidates uniformly. This is the Full Soup scope of Sec. 5.1, reported here as the no-selection endpoint of the axis. It merges far more checkpoints than the baseline, which makes it the relevant control against the concern that Trajectory Soup benefits merely from averaging more models. Appendix E Per-Trajectory Results of Single-Trajectory Merging Every branch yields its own Single EXP Merge, the within-trajectory anchor defined in Appendix D.5, and the main-text tables report only the strongest of them under the name Single-Trajectory Merge. This appendix lists all of them. The reported row is chosen separately for each table by its Overall Average, so it need not be the same branch across tables, and it is marked with † below. Because the main text compares Trajectory Soup against the strongest branch, this choice is conservative for the Overall Average, and comparing against any other branch would only widen the reported margins. E.1 Default Model at Mid-Training Tab. 9 expands the Single-Trajectory Merge row of Tab. 2 into the three branches EXP1 (baseline), EXP2 (data shuffle), and EXP3 (learning rate and batch size) of Tab. 8. Their Overall Averages span 68.34 to 68.55, with EXP1 the strongest. No branch leads in every category. EXP1 is best on general knowledge and reasoning, language modeling, and code, EXP2 on math, and EXP3 on professional knowledge. Both Trajectory Soup variants of Tab. 2, at 68.72 (Limited) and 68.96 (Extended), exceed all three branches rather than only the reported one. Table 9: Single-trajectory merges of every branch on the mid-training evaluation suite. Each row is the Single EXP Merge of one branch of Tab. 2. †Reported as Single-Trajectory Merge in Tab. 2. Bold marks the best value in each column. Base Model General Knowledge & Reasoning Language Modeling Professional Knowledge Math Code Overall Average EXP1 Merge† 62.80 85.86 63.70 72.33 65.36 68.55 EXP2 Merge 62.79 85.44 63.97 72.39 64.99 68.46 EXP3 Merge 62.69 85.65 64.02 71.57 65.34 68.34 E.2 Default Model after Post-Training Tab. 10 applies the SFT procedure of Tab. 3 to each of the three merges above, and the ordering changes. EXP2 now has the highest Overall Average at 61.11, while EXP1 and EXP3 tie at 60.67. The Single-Trajectory Merge rows of Tab. 2 and Tab. 3 therefore correspond to different branches, and the strongest mid-training merge is not necessarily the strongest initialization for SFT. Trajectory Soup reaches 61.39 (Limited) and 61.52 (Extended) in Tab. 3, again above every branch. Table 10: Single-trajectory merges of every branch after post-training. Each row applies the SFT procedure of Tab. 3 to the Single EXP Merge of one branch. †Reported as Single-Trajectory Merge in Tab. 3. Bold marks the best value in each column. Instruct Model Math Code Knowledge Reasoning Instruction Following Function Call Overall Average EXP1 Merge 68.22 47.69 68.98 64.88 57.54 51.75 60.67 EXP2 Merge† 68.30 46.58 70.51 62.96 59.46 50.67 61.11 EXP3 Merge 68.38 45.63 69.87 62.40 58.82 51.57 60.67 E.3 Smaller MoE Model with a WSD Schedule Tab. 11 lists the two branches of the 2B-parameter reproduction in Tab. 4, where EXP1 is the baseline and EXP2 changes only the data-shuffling seed. These labels are local to that experiment and do not refer to the checkpoints of the default model. The two merges are close, at 49.49 and 49.55. EXP1 is stronger on general knowledge and reasoning, language modeling, and professional knowledge, and EXP2 on math and code. Trajectory Soup reaches 49.64 (Limited) and 50.07 (Extended) and exceeds both. Table 11: Single-trajectory merges of both branches of the smaller 2B-parameter MoE model. Each row is the Single EXP Merge of one branch of Tab. 4. †Reported as Single-Trajectory Merge in Tab. 4. Bold marks the best value in each column. Model General Knowledge & Reasoning Language Modeling Professional Knowledge Math Code Overall Average EXP1 Merge 47.97 73.21 43.92 54.09 34.88 49.49 EXP2 Merge† 47.71 72.85 43.41 54.21 35.96 49.55