Paper deep dive
Small Models Scout Bottleneck Order for Large-Model Data Control
Seungmin Choi, Jiwon Sung, Muhammad Umer, Abhiram Rao Gorle, Guijin Son, Youngjae Yu, John M. Cioffi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/18/2026, 4:55:20 AM
Summary
The paper introduces LogFloor, a closed-loop controller that uses small proxy models to identify phase-ordered bottleneck resolution trajectories for training larger models. By minimizing tokens required to reach specific skill floors, LogFloor demonstrates significant token savings (up to 56.2% on Qwen2.5-1.5B and 37.6% in 70M-to-12B transfer) compared to uniform training and static mixture baselines. The study highlights that the order of skill acquisition is a transferable structure, outperforming static data mixtures and other online mixing methods.
Entities (13)
Relation Signals (8)
LogFloor → reducestokencoston → Qwen2.5-1.5B
confidence 95% · LogFloor reduces token cost by 56.2% on average on Qwen2.5-1.5B.
LogFloor → usedon → bAbI
confidence 95% · Across five bAbI skill slices on Qwen2.5-1.5B, LogFloor reduces token cost...
LogFloor → appliedto → MMLU-control
confidence 90% · On MMLU-control, a frozen scout path succeeds across all eight 12B runs.
LogFloor → outperforms → Skill-It
confidence 90% · Skill-It ... do not robustly improve this first-passage objective.
LogFloor → outperforms → ODM
confidence 90% · Skill-It ... and ODM ... do not robustly improve this first-passage objective.
LogFloor → transferredto → 12B
confidence 90% · In 70M-to-12B transfer, three-round replay of a 70M scout path reaches every floor...
target-probe replay → uses → 70M
confidence 90% · three-round replay of a 70M scout path reaches every floor in all eight target runs
LogFloor → outperforms → RegMix
confidence 85% · The trajectory replay saves 35.3% under the same accounting [vs RegMix 23.8%].
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Small proxy models are commonly used to identify data mixtures for larger-scale training. We ask whether their training trajectories reveal another transferable structure: the order in which larger models should resolve skill bottlenecks. We formulate first-passage skill training, where each monitored skill has a target floor and the objective is to minimize the tokens required to reach all floors. We introduce LogFloor, a closed-loop controller that directs each round toward current bottlenecks, producing phase-ordered resolution trajectories. Across five bAbI skill slices on Qwen2.5-1.5B, LogFloor reduces token cost by 56.2% on average. In 70M-to-12B transfer, three-round replay of a 70M scout path reaches every floor in all eight target runs, saving 30.9% by pair mean, 39.4% in pooled training tokens, and 37.6% under source-cost accounting. On MMLU-control, a frozen scout path succeeds across all eight 12B runs. Collapsing a path to its static marginal mixture or reversing its phase order removes most benefits, while bottleneck labels alone remain partially useful. These results identify phase-ordered bottleneck resolution as a transferable curriculum structure for monitored skill-targeted training.
Tags
Links
- Source: https://arxiv.org/abs/2608.14936v1
- Canonical: https://arxiv.org/abs/2608.14936v1
Trouble viewing inline? Open PDF directly →
Full Text
41,066 characters extracted from source content.
Expand or collapse full text
Small Models Scout Bottleneck Order for Large-Model Data Control Seungmin Choi Jiwon Sung Muhammad Umer Abhiram Rao Gorle Guijin Son Youngjae Yu John M. Cioffi Abstract Small proxy models are commonly used to identify data mixtures for larger-scale training. We ask whether their training trajectories reveal another transferable structure: the order in which larger models should resolve skill bottlenecks. We formulate first-passage skill training, where each monitored skill has a target floor and the objective is to minimize the tokens required to reach all floors. We introduce LogFloor, a closed-loop controller that directs each round toward current bottlenecks, producing phase-ordered resolution trajectories. Across five bAbI skill slices on Qwen2.5-1.5B, LogFloor reduces token cost by 56.2%56.2\% on average. In 70M-to-12B transfer, three-round replay of a 70M scout path reaches every floor in all eight target runs, saving 30.9%30.9\% by pair mean, 39.4%39.4\% in pooled training tokens, and 37.6%37.6\% under source-cost accounting. On MMLU-control, a frozen scout path succeeds across all eight 12B runs. Collapsing a path to its static marginal mixture or reversing its phase order removes most benefits, while bottleneck labels alone remain partially useful. These results identify phase-ordered bottleneck resolution as a transferable curriculum structure for monitored skill-targeted training. Figure 1: A simplified visualization of the proposed LogFloor controller for K=5K=5 monitored skills. The bottleneck skill at each round r is depicted in orange. At each round, LogFloor shifts the largest share of the next allocation wrw_r to this bottleneck, and the resulting ordered sequence wr\w_r\ forms the bottleneck-resolving trajectory that we later replay on larger target models. Introduction Data mixture selection is usually framed as a question of which training data are most useful. Small proxy models make this decision computationally tractable by scoring examples, estimating domain weights, or predicting large scale training outcomes. Yet these approaches primarily seek useful mixtures or schedules under aggregate objectives. We ask whether small-model trajectories reveal a different transferable structure: the order in which a larger model should resolve skill bottlenecks. In many scenarios, the relevant objective is not a high average validation score across skills but satisfying a set of monitored skill thresholds, which we call floors. For example, a model intended to support math, code, and multilingual use would not be suitable for use if even one skill remains below the required floor, regardless of its average score. In such scenarios, training cannot terminate until the bottleneck skill reaches its floor. The training policy must therefore determine which skills to prioritize at each stage of training. We call this objective first-passage skill training. Given monitored skill slices and a target floor for each slice, the objective is to minimize the training tokens consumed until every skill reaches its floor. Under this objective, uniform training with equal emphasis on each skill would waste resources on skills that have already reached their floors while a bottleneck skill has not yet reached its floor. We introduce LogFloor, a closed loop controller that observes per-skill monitor scores, computes floor pressure, and allocates the next training round toward the bottleneck skills exerting the largest pressure. We call the resulting allocation path a bottleneck-resolving trajectory. Figure 1 provides a simplified illustration of how LogFloor prioritizes the current bottleneck skill at each training round. A 70M model can reveal a phase-ordered bottleneck path that remains useful for a 12B target after a short target side phase probe. We do not claim that small models universally determine effective curricula, only that in monitored first-passage skill training, the transferable structure is a phase-ordered bottleneck-resolving trajectory and is not captured by a static mixture alone. We support this claim with three key findings. First, LogFloor produces successful bottleneck-resolving trajectories. On Qwen2.5-1.5B (11) with five controlled bAbI skill slices (15), LogFloor saves 56.2%56.2\% training tokens on average across target floors from 2.82.8 to 3.33.3. Skill-It (4) and Online Data Mixing (ODM) (1), which do not use target information, do not robustly improve this first-passage objective. Second, the path transfers across scale. In the 70M to 12B setting, (replayable) target-probe replay of a 70M scout trajectory reaches every target after three probe rounds, saving 30.9%30.9\% by pair mean, 39.4%39.4\% in pooled target training tokens, and 37.6%37.6\% under source cost accounting. Third, the order matters. On academic domain MMLU-control (7), a frozen 70M scout path hits all eight 12B route targets and saves 60.3%60.3\% in pooled rounds. A clear scout decomposition shows that static mixture collapse and reversal in order preserve only 10.3%10.3\% and 6.0%6.0\%, respectively. The contributions of this paper are outlined below. • We formulate first-passage skill training, an objective that requires every monitored skill to reach its floor and thereby exposes bottleneck limited data control. • We introduce LogFloor, a simple floor pressure controller that produces bottleneck-resolving trajectories. • We show that a 70M scout path can guide a 12B run via a short target-probe replay, retaining savings after accounting for the source controller cost. • We isolate the transferred signal using order destruction controls, showing that bottleneck order is not reducible to a static data mixture. Related Work Proxy Data Mixtures and Transfer A first line of work uses small proxy models to guide data selection and mixing for larger runs. DoReMi (16) uses a small proxy model to select a useful static domain mixture for a larger training run. SmallToLarge (17) uses small model training trajectories for example selection during supervised finetuning. RegMix (9) uses regression over proxy runs to select a static mixture for larger scale training. AC-ODM (10) transfers an actor trained with a proxy model to a larger target. PROXYMIX (13) transfers replay controllers learned on proxy models to larger targets. Most recently, RegMix-D (18) extends static proxy mixture selection to dynamic schedules learned from proxy loss trajectories. We audit AC-ODM as a dynamic proxy-policy transfer baseline and RegMix as a static proxy-mixture baseline under the same monitored first-passage protocol. RegMix-D is closest in using proxy trajectories for dynamic scheduling, but it evaluates schedules under a fixed training budget rather than minimizing the tokens required for all monitored skills to reach their floors. Online Data Control and Skill Acquisition Order A second line of work studies how data allocation and skill order evolve during training. Curriculum and domain ordering work shows that the order of data can affect training trajectories (12; 5). Within language model data control, Skill-It and ODM adapt data allocation online using skill losses and bandit feedback. Aioli (3) later develops a unified optimization framework and an online method that updates mixture proportions during training. Data mixing can also induce phase transitions that depend on the mixing ratio and model scale (6). TiKMiX (14) uses group influence signals to track changing domain preferences and periodically update the data mixture. Most recently, 8 analyze fixed threshold crossings across pretrained model checkpoints and show regularities in skill acquisition order across models. Their work observes this order during ordinary pretraining, whereas we intervene on it by replaying a realized source path and test whether it reduces the first-passage training cost of a new target model. First-Passage Control with LogFloor In this section, we formalize first-passage skill training and define the LogFloor controller. First-Passage Skill Training Let K denote the number of monitored skills, indexed by k=0,…,K−1k=0,…,K-1, and let ℓr,k>0 _r,k>0 denote the average log loss on monitor set k after training round r. We measure the progress of skill k relative to the model at round 00 as sr,k=ℓ0,kℓr,k. s_r,k= _0,k _r,k. (1) Here, a score of m indicates an m-fold reduction in monitored log loss. Each skill has a target floor τk _k, and the first-passage round T is the first round at which all floors are reached: sT,k≥τk for all k. s_T,k≥ _k for all k. (2) Unlike fixed-budget objectives, this objective depends on whether every monitored skill reaches its floor, rather than on the average score at a fixed endpoint. The scores sr,ks_r,k determine when to stop and are used to update the controller, while the heldout split is used only as an external audit. The bAbI experiments use the prespecified absolute floor sweep from 2.82.8 to 3.33.3. For MMLU-control, each target seed’s floor is fixed before method evaluation as 90%90\% of the peak worst-skill route score in its 6060-round uniform trace. We measure token saving relative to matched uniform training as 1−CLogFloorCuniform, 1- C_LogFloorC_uniform, (3) where each cost includes the training tokens consumed up to its first-passage round. The main evaluation terms used throughout the paper are summarized in Table 1. Table 1: Evaluation terminology. Term Definition Pooled saving Savings after summing token costs across runs. Pair mean saving Mean of the savings of each matched run. Hit All monitored floors are reached within the horizon. Route first-hit round First round where all monitored floors are reached. Heldout audit Heldout evaluation at the route first-hit round. W/T/L Wins, ties, and losses against matched uniform training. Floor-Pressure Control At each round r, the controller evaluates the monitor sets and computes each skill’s remaining deficit: dr,k=[τk−sr,k]+, d_r,k=[ _k-s_r,k]^+, (4) where [z]+=maxz,0[z]^+= \z,0\. The controller allocates more of the next training round to skills with larger deficits. Let qkq_k be a fixed base share, let η>0η>0 control how sharply the controller focuses on the largest deficit, and let α∈[0,1]α∈[0,1] be the smoothing coefficient. Let Bound(⋅)Bound(·) apply the fixed allocation caps and renormalize the resulting vector. The controller first forms a soft allocation and then smoothens it using the previous allocation: w~r,k w_r,k =qkexp(dr,k/η)∑jqjexp(dr,j/η), = q_k (d_r,k/η) _jq_j (d_r,j/η), (5) wr w_r ←(1−α)wr−1+αBound(w~r). ←(1-α)w_r-1+α\,Bound( w_r). (6) Here, w~r=[w~r,0,w~r,1,…,w~r,K−1]⊤ w_r=[ w_r,0, w_r,1,…, w_r,K-1] . Thus, η controls bottleneck emphasis, α controls how quickly allocations change, and Bound(⋅)Bound(·) enforces the fixed allocation caps. LogFloor uses this soft exponential allocation. We additionally evaluate Greedy LogFloor, its hard-bottleneck variant. Algorithm 1 summarizes the complete controller procedure, including initialization, stopping, integer rounding, and trajectory construction. Full derivation is provided in the supplementary. Algorithm 1 LogFloor floor-pressure update 1: Input: monitor sets; initial monitor losses ℓ0,k _0,k; score floors τk _k 2: for r=0,1,…r=0,1,… do 3: Compute scores sr,ks_r,k using Eq. (1) 4: Compute deficits dr,kd_r,k using Eq. (4) 5: if dr,k=0d_r,k=0 for all k then 6: break 7: end if 8: Compute w~r w_r using Eq. (5) 9: if r=0r=0 then 10: Set w0=Bound(w~0)w_0=Bound( w_0) 11: else 12: Update wrw_r using Eq. (6) 13: end if 14: Train one round according to wrw_r using largest-remainder rounding 15: end for 16: Output: ordered allocation trajectory γ=wrr=0T−1γ=\w_r\_r=0^T-1 First-Passage Efficiency Before testing the transfer to a larger model, we first show that LogFloor produces successful first-passage trajectories. In the representative run in Figure 2, uniform training clears all target floors at zero-indexed round 4242, whereas LogFloor reaches them at round 1111 by repeatedly prioritizing the current floor-pressure bottleneck skill. The red boxes in the lower panel mark the bottleneck skill selected at each round. Figure 2: A representative bottleneck-resolving trajectory. Uniform training clears all target floors at zero-indexed round 42, while LogFloor clears them at zero-indexed round 11 by repeatedly prioritizing this floor-pressure bottleneck. Red boxes in the lower heatmap identify the bottleneck skill selected at each round. Table 2 summarizes the target sweep on Qwen2.5-1.5B over five controlled bAbI skill slices. Across six absolute target floors from 2.82.8 to 3.33.3 and five random seeds per floor, LogFloor achieves 56.2%56.2\% pooled token savings relative to matched uniform runs. It reaches all 3030 route targets, with an exact 95%95\% Clopper–Pearson interval of [88.4%,100%][88.4\%,100\%] for the route-hit rate. Greedy LogFloor also performs well with 51.1%51.1\% pooled savings, supporting the intuition that floor pressure is the dominant indicator of allocation. LogFloor wins in 2020 out of 3030 cases against the greedy variant, indicating greater stability across seed and target pairs. Target-free online data-mixing baselines perform poorly for this first-passage objective, with pooled savings of −7.8%-7.8\% for Skill-It and −0.9%-0.9\% for ODM. We now examine heldout performance. At the route first-hit round, the heldout audit clears all target floors in 2828 out of 3030 runs. The mean difference between the heldout and route worst-skill scores is +0.049+0.049, while the mean minimum heldout margin above the corresponding target floors is +0.260+0.260. The exact 95%95\% Clopper–Pearson interval for the heldout confirmation rate is [77.9%,99.2%][77.9\%,99.2\%]. Thus, route first-passage is usually corroborated on heldout data. Regarding the experimental setup, all compared methods use the same five slices, route/heldout split, batch size, allocation bounds, monitor cadence, stopping rule, and horizon for each seed and floor. Controller hyperparameters are fixed in advance, and Skill-It and ODM do not use target information in their allocation updates. Table 2: First-passage target sweep on Qwen2.5-1.5B. Each target floor has five seeds. LogFloor reaches the route target in 30/3030/30 runs, and 28/3028/30 route first-hits also clear the heldout target on all floors. Floor LogFloor pair mean LogFloor pooled Greedy pooled Skill-It pooled ODM pooled 2.8 36.3% 59.2% 54.4% -12.1% 7.2% 2.9 28.0% 53.2% 52.0% -22.9% 4.8% 3.0 27.1% 49.1% 49.1% -13.3% 12.2% 3.1 48.7% 55.6% 55.7% 3.6% 20.9% 3.2 57.8% 62.2% 52.4% -5.3% -19.6% 3.3 45.0% 55.4% 44.9% -3.8% -17.9% All floors 40.5% 56.2% 51.1% -7.8% -0.9% These results motivate using LogFloor also as a trajectory generator rather than only as a per-run controller. Its output is interpretable as an ordered sequence of bottleneck decisions, which is the object we test for transfer across scale. Cross-Scale Transfer by Replaying LogFloor Trajectories We evaluate whether bottleneck-resolving trajectories learned by small models can guide larger target models. Trajectory Replay Schemes For a source run S, we define the transferable object as its ordered before-hit allocation sequence: γS=w0S,w1S,…,wTS−1S, γ^S=\w_0^S,w_1^S,…,w_T_S-1^S\, (7) where TST_S is the first-passage round of the source. Since source and target models may traverse bottleneck phases at different rates, a replay scheme must map each target round to an allocation in the source trajectory. We refer to a monotone traversal of the ordered source path after phase localization as a phase-indexed replay. Raw round-indexed replay. This scheme directly applies the source allocation wrSw_r^S at target round r. It assumes that the source and target traverse their bottleneck phases at the same speed, which is generally false. We use it as a baseline to test whether literal source-round copying is sufficient without phase localization. (Replayable) target-probe replay. This scheme uses a short uniform target probe to localize the target to a source phase and then advances monotonically through the ordered source trajectory. Let H denote the probe length. We use a single global probe length of H=3H=3 in all main-result target-probe experiments. We chose it as a common operating point after preliminary H∈1,3,5H∈\1,3,5\ sensitivity checks. In the MMLU source-seed-2 sweep, H=3H=3 reaches all 8/88/8 targets with 31.0%31.0\% pooled-round saving, versus 7/87/8 for H=1H=1 and 21.6%21.6\% for H=5H=5; H is not retuned by target seed, floor, or source–target pair. The Supplementary Document provides the exact protocol and the sensitivity analysis for H∈1,3,5H∈\1,3,5\. The resulting route-score window is normalized and matched to same-length normalized windows from a 70M uniform calibration trace. We slide the target’s three-round probe pattern along the 70M calibration trace and choose the source phase that looks the most similar. Similarity is measured from the relative skill scores within each round and the trend of the worst skill, using normalized mean squared distance. Let j⋆j denote the start of the matched source window. For target round r≥Hr≥ H, the replayed source-action index is ir=minj⋆+r−H,TS−1. i_r= \j +r-H,\,T_S-1\. (8) The target applies wirSw_i_r^S, advancing monotonically from the matched source phase and clamping to the final logged allocation if it outlasts the source trajectory. After the probe, replay uses no further target scores or target first-passage horizon to update, interpolate, or rematch the schedule. We use this scheme to test whether a small-model trajectory can guide a larger target after only short phase localization and without continued target-side adaptation. (Oracle) phase-aligned replay. This scheme length-normalizes the source trajectory using the observed before-hit target length TLT_L. At target round r, it applies the source allocation nearest to the same relative before-hit progress r/(TL−1)r/(T_L-1). Since TLT_L is available only after observing the target trajectory, this scheme is not replayable at deployment. We use it as an oracle diagnostic to test whether the source trajectory remains useful when the source and target are placed at matched bottleneck phases. 70M-to-12B Transfer We first test whether (replayable) target-probe replay transfers source trajectories across models with different scales. Six Pythia (2) transfer settings using replayable target-probe replay are summarized in Figure 3. In these settings, the trajectory replay achieves an unweighted mean target training token savings of 51.0%51.0\%, compared to 42.3%42.3\% for cumulative static mixtures and 36.9%36.9\% for final static mixtures. Thus, small-to-large trajectory transfer works across multiple scale pairs and preserves more benefit on average than both static mixture baselines. 70M-to-12B achieves the largest savings, where a path discovered by the smallest Pythia model remains useful across a roughly 170×170× increase in model size. Figure 3: Small-to-large trajectory transfer. Replayable target-probe replay transfers source LogFloor trajectories across six Pythia settings and outperforms static summaries on average. Table 3 focuses on the 70M-to-12B setting. These bAbI rows use eight seed-matched 70M–12B source–target pairs. The 12B online LogFloor row is the fully target-adaptive reference, the frozen target mixture isolates the effect of the probe, and phase-aligned replay is an oracle diagnostic. Table 3 gives us three main insights. First, replaying the 70M trajectory is nearly as effective as running LogFloor online at 12B scale. The 12B online controller saves 34.4%34.4\% by pair mean and 39.4%39.4\% pooled savings, while the replayable 33-round target-probe method saves 30.9%30.9\% by pair mean and 39.4%39.4\% pooled, reaching all eight targets and winning seven out of eight matched comparisons. Paired bootstrap resampling of the eight target seeds gives a 95%95\% interval of [4.6%,50.9%][4.6\%,50.9\%] for the target-probe pair-mean saving. Second, the transferred source trajectory provides value beyond the 33-round target probe. A target mixture computed from the same probe and kept constant throughout the remaining training reaches only seven out of eight targets. The remaining seed does not hit within the horizon, so its aggregate cost and savings are undefined. In contrast, replaying the 70M trajectory reaches all eight targets, showing that the ordered source path provides useful information beyond the static mixture obtained from the probe. As an oracle diagnostic, the phase-aligned replay also reaches all eight targets and saves 35.1%35.1\% pooled, confirming that the source trajectory remains useful when its phase is correctly aligned with the target. Third, replay remains useful even after accounting for the cost of scouting the trajectory. We multiply the logged 70M source training tokens by the parameter ratio 70M/12B70M/12B and add the resulting cost equivalent to 12B to the replay target cost. Nonetheless, replay still saves 37.6%37.6\%. Figure 4 (left) compares target-only with cost-inclusive savings and Table 3 gives target-probe replay’s 37.6%37.6\% scout-inclusive saving. Table 3: 70M-to-12B transfer accounting. Replayability, probe, oracle, and source-cost controls over eight target runs. Hit and Win count route hits and wins against matched uniform runs; rows with a missed target omit aggregate savings. Method Hit Win Target tokens Pair mean Pooled Incl. scout 12B uniform 8/8 – 825,094 0.0% 0.0% 0.0% 12B online LogFloor 8/8 8/8 499,711 34.4% 39.4% 39.4% 70M target-probe replay 8/8 7/8 500,375 30.9% 39.4% 37.6% Probe + frozen target mix 7/8 – – – – – 70M phase replay (oracle) 8/8 7/8 535,150 30.1% 35.1% 33.4% Finally, we compare replay with proxy-based baselines. On the matched three-seed (n=3n=3) subset, RegMix saves 40.1%40.1\% when only target training is counted, but 23.8%23.8\% after including its 70M proxy-search cost. The trajectory replay saves 35.3%35.3\% under the same accounting. Figure 4 (right) exposes a limitation of static proxy search. As the search budget increases, the selected mixture improves on the 70M proxy, but its transferred performance on 12B is non-monotonic. Thus, the mixture ranked best by the small model need not be the mixture that is best for the large model. The AC-ODM proxy actor reaches all three targets but uses 216K target training tokens, compared to 199K for replay. Thus, replay is more efficient under cost-aware comparison. The Supplementary Document provides the RegMix audit and shows that replay should first match the target’s state to a source phase. Directly copying source allocations by round can fail. Figure 4: 70M-to-12B transfer diagnostics. The left plot juxtaposes target-only savings for 12B online and 70M target-probe replay with cost-inclusive savings for 70M phase replay and RegMix. The right plot shows that static proxy search improves the 70M proxy score but transfers non-monotonically to 12B. Understanding Replay-Based Transfer In this section, we examine whether replay transfers phase-ordered bottleneck information rather than a static mixture or a seed-paired trace. Shared Bottleneck Geometry Across Scale Phase-indexed replay is plausible if source and target models encounter similar bottleneck states at the corresponding stages of training. Let ϕ∈[0,1]φ∈[0,1] denote the relative progress towards the first-passage, so that ϕ=0φ=0 is the beginning of the trajectory and ϕ=1φ=1 is the first-hit boundary. This allows us to compare source and target models at the same normalized phase rather than at the same training round. Across normalized training phase, the aligned 70M source trajectory and the 12B online controller emphasize largely overlapping bottlenecks. For each Pythia-70M source and Pythia-12B target online LogFloor run, we truncate both trajectories before the first hit, align them by ϕφ, and compare their deficit vectors, controller allocation vectors, and top bottleneck identities. Table 4 shows that off-diagonal source–target seed pairs are as similar as diagonal pairs. The deficit cosine is 0.8070.807 for diagonal pairs and 0.8140.814 for off-diagonal pairs, while the overlaps of the top-22 bottlenecks are 0.9300.930 and 0.9350.935, respectively. The source trajectory is therefore not merely a trace specific to one matched source–target seed pair. Table 4: 70M-to-12B phase geometry audit. Before-hit trajectories are aligned by normalized phase, and entries are means over diagonal and off-diagonal source–target seed pairs. Comparison Pairs Deficit cos Alloc. cos Top-22 Diagonal seed i→i\!→\!i 88 0.8070.807 0.8440.844 0.9300.930 Off-diagonal i→ji\!→\!j 5656 0.8140.814 0.8390.839 0.9350.935 The shared deficit geometry also has a simple local interpretation. Let dϕSd_φ^S and dϕLd_φ^L denote the source and target deficit vectors at the same aligned phase, and let w~ϕS w_φ^S and w~ϕL w_φ^L be their soft allocations from Eq. (5). For fixed q and η, these allocations satisfy ‖w~ϕS−w~ϕL‖1≤1η‖dϕS−dϕL‖∞. \| w_φ^S- w_φ^L \|_1≤ 1η \|d_φ^S-d_φ^L \|_∞. (9) This follows because Eq. (5) is a softmax in d/ηd/η, and its sensitivity to changes in d is bounded by 1/η1/η. Applying this bound between the source and target deficits gives Eq. (9). Thus, similar deficits at a matched phase imply similar ideal soft allocations. This is only a local continuity statement. It does not model the full training dynamics or prove first-passage transfer, and the executed allocation additionally includes bounding, smoothing, and integer rounding. Nevertheless, in the MMLU matched-target local audit, the logged source actions retain 98.5%98.5\% of the target-online pressure progress, with a mean one-step surrogate gap of 0.02360.0236. The full derivation and the corresponding implementation-gap analysis are provided in the Supplementary Document. Order, Not Static Mixture This subsection asks what kind of information in the source trajectory can be transferred. Table 5 reports the matched and fixed-source replay diagnostics. The order controls summarized in Table 6 then show whether the gain comes from temporal order rather than the static mixture, and how much of that gain is retained by the bottleneck sequence alone. We use MMLU control because its five academic domains are easily interpretable and allow controlled interventions on the same frozen scout path. The domains are math, science, history, law/social science, and humanities. This is a controller diagnostic with separate route and heldout splits, not a leaderboard MMLU evaluation. Transfer across seeds. Table 5 shows that a single frozen 70M scout can be replayed across all eight 12B target seeds. The fixed source seed 22 reaches all eight targets and saves 60.3%60.3\% in pooled rounds. The order diagnostic extends beyond source seed 22. Across source seeds 00–22, correctly ordered replay reaches all 2424 target runs, whereas every static or reversed control is either less efficient than its matched ordered path or contains at least one no-hit (Supplementary Document). Matched replay saves 38.8%38.8\%, compared to 33.6%33.6\% for 12B online LogFloor. The fixed target-floor protocol and seed-level audit are provided in the Supplementary Document. The three-round target probe replay also reaches all eight targets and saves 31.0%31.0\% in pooled rounds. By contrast, freezing the target mixture obtained after the same probe reaches only seven out of eight targets. Thus, the replay result is not explained by the probe alone. Table 5: MMLU control transfer diagnostics. Rows aggregate eight Pythia-12B target seeds unless noted. W/T/L denotes wins, ties, and losses against matched uniform runs. Rows with a missed target omit aggregate savings. Schedule Hit W/T/L Pair mean Pooled rounds 12B online LogFloor 8/88/8 4/3/14/3/1 25.6%25.6\% 33.6%33.6\% 70M matched replay 8/88/8 5/1/25/1/2 23.5%23.5\% 38.8%38.8\% Frozen scout seed 22 8/88/8 6/2/06/2/0 46.8%46.8\% 60.3%60.3\% 33-round target probe 8/88/8 5/2/15/2/1 17.2%17.2\% 31.0%31.0\% 33-round probe + frozen target mix 7/87/8 4/1/34/1/3 – – Order and retained information. The order controls in Table 6 decompose the information in the frozen trajectory. We fix source seed 22 across the eight target seeds and all order interventions. Source seed 22 was selected before any 12B runs using only source-side route/heldout agreement and controller diagnostics. It is the headline decomposition because its correct, static, reverse, and label controls all hit 8/88/8, making their pooled comparisons defined: correct replay saves 60.3%60.3\% in pooled rounds, versus 10.3%10.3\% for static collapse and 6.0%6.0\% for reversal. The source-0 and source-1 ordered paths also reach all 16/1616/16 targets, but their corresponding control suites contain no-hit rows and are reported in the Supplementary Document. Table 6: Decomposition of the frozen source seed 2 trajectory. All rows use the same eight Pythia-12B targets and the same floors for each target. Rows with a missed target omit aggregate savings. Schedule Retained signal Hit Pair mean Pooled rounds Pooled tokens Correct seed 22 scout full phase indexed allocation path 8/88/8 46.8%46.8\% 60.3%60.3\% 59.8%59.8\% Static collapse marginal allocation only 8/88/8 −2.9%-2.9\% 10.3%10.3\% 9.1%9.1\% Reverse same allocations in reverse order 8/88/8 −13.6%-13.6\% 6.0%6.0\% 5.3%5.3\% Random permutation same allocations in shuffled order 6/86/8 – – – Label replay bottleneck identities in correct order 8/88/8 27.9%27.9\% 25.0%25.0\% 22.4%22.4\% Label reverse same identities in reverse order 8/88/8 −66.7%-66.7\% −11.2%-11.2\% −10.8%-10.8\% The correct path saves 46.8%46.8\% by pair mean, 60.3%60.3\% in pooled rounds, and 59.8%59.8\% in pooled tokens. Collapsing the path to its static marginal mixture reduces the pooled round savings to 10.3%10.3\%. Reversing the same allocation vectors reduces it to 6.0%6.0\%, and a random permutation reaches only six out of eight targets. Across the same eight target seeds, the correctly ordered path is faster than static collapse in 6/86/8 cases, ties once, and is slower once, using 5858 fewer first-passage rounds in total; the corresponding comparison with reversal is also 6/1/16/1/1, with 6363 fewer total rounds. These controls preserve either the same allocation vectors or their marginal mixture while changing the temporal structure. The resulting loss therefore shows that order carries most of the transfer gain. Keeping only the bottleneck identities in the correct order still saves 25.0%25.0\% in pooled rounds, while reversing the same sequence gives −11.2%-11.2\%. The bottleneck sequence therefore carries useful coarse information. The 60.3%60.3\% savings from the full path shows that allocation strength and secondary domain structure provide substantial additional value. A separate bAbI replication gives the same qualitative result. Correct replay reaches all eight targets and saves 35.1%35.1\% in pooled tokens, static mean replay saves 20.1%20.1\%, and reversal reaches only six targets with −1.9%-1.9\% savings. Mass assigned to the current bottleneck correlates positively with token saving (ρ=0.589ρ=0.589), while mass assigned to already cleared skills correlates negatively (ρ=−0.704ρ=-0.704). Robustness, Scope, and Limitations Replay Across Target Floors Replay does not need to be scouted at the exact deployment floor. We replay Qwen2.5-1.5B LogFloor trajectories scouted at floors 2.82.8, 3.03.0, and 3.33.3 at the other two target floors after the same three round target probe. Table 7 reports the six different pairs of scout and target floors over five target seeds each. The replay reaches 2828 out of 3030 targets, and four out of the six pairs reach all five. Most importantly, the replay succeeds when the scout floor is either lower or higher than the target floor, showing that exact floor agreement is unnecessary within the evaluated range. Table 7: Replay across target floors on Qwen2.5-1.5B. Each cell gives route hits out of five target seeds after the same three-round target probe; darker shading marks full 5/55/5 hits. Gray cells mark identical scout and target floors. Target floor Scout floor 2.82.8 3.03.0 3.33.3 2.82.8 – 4/54/5 5/55/5 3.03.0 5/55/5 – 4/54/5 3.33.3 5/55/5 5/55/5 – Robustness and Scope Table 8 summarizes robustness across target levels, skill counts, model sizes, pretraining stages, benchmarks, and architectures. Detailed results, seed level tables, and censoring audits are provided in the Supplementary Document. Table 8: Breadth and scope summary. Axis Result Role Target sweep Qwen2.5-1.5B saves 56.2%56.2\% pooled tokens across six floors primary validation Skill count K∈5,7,10,15K∈\5,7,10,15\: 41.8%41.8\% aggregate and 40.1%40.1\% paired savings floor-count robustness Model size Pythia 70M–2.8B wins on all evaluated seeds, 42.1%42.1\% mean saving scale robustness Pretraining stage positive from step 16K onward, 53.9%53.9\% at the final checkpoint stage robustness SuperGLUE control 51.2%51.2\% (base) and 53.7%53.7\% (CB-substituted) route savings; 7/107/10 heldout-confirmed beyond bAbI Architecture and frontier Mamba self-control positive; single-seed 72B pilot hits at round 7 (uniform), 2 (online), 3 (replay) scope evidence Note that the skill count and model size results establish robustness, not monotonic improvement with either K or parameter count. The gain is weak at the earliest pretraining stage but becomes consistently positive at later checkpoints. The SuperGLUE, Mamba, and 72B results further extend the evidence beyond the main bAbI and Pythia settings. Together, these results define the empirical scope of the method rather than a universal curriculum law. Limitations The target side online LogFloor and the small model replay are two operating modes of the same control principle, not competing methods. Our evidence concerns monitored first-passage training with explicit floors and data slices associated with each skill. The target probe replay is deployable, whereas the oracle phase-aligned replay is used only as a mechanism diagnostic. Replay across architectures is not guaranteed. For instance, a replay from Qwen to Mamba transfers well, whereas a replay from Mamba to Qwen gives mixed results. The controller also treats monitored skills separately and does not explicitly model transfer or interference among them. Conclusion We introduced first-passage skill training and LogFloor, a floor-pressure controller that exposes bottleneck-resolving trajectories. The key takeaway is that these trajectories can be transferred across scale. A 70M model can scout a phase-ordered bottleneck path that helps a 12B target reach all monitored floors faster after a short target probe. In bAbI-control, this replay nearly matches the direct 12B online control. In the MMLU-control order diagnostic, frozen 70M scout paths can even give a stronger pooled round signal. Order destruction controls show that the reusable signal is not merely a static data mixture. Reversing or collapsing the path removes most of the gain, while the bottleneck label sequence remains partially useful. Small models therefore do not only estimate useful mixtures. In monitored first-passage training, they can reveal the order in which larger models should resolve skill bottlenecks. References Albalak et al. (2023) A. Albalak, L. Pan, C. Raffel, and W. Y. Wang Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406. Cited by: Introduction. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: a suite for analyzing large language models across training and scaling. In International conference on machine learning, p. 2397–2430. Cited by: 70M-to-12B Transfer. Chen et al. (2025) M. Chen, M. Hu, N. Lourie, K. Cho, and C. Ré Aioli: a unified optimization framework for language model data mixing. In International Conference on Learning Representations, Vol. 2025, p. 46089–46132. Cited by: Online Data Control and Skill Acquisition Order. Chen et al. (2023) M. Chen, N. Roberts, K. Bhatia, J. Wang, C. Zhang, F. Sala, and C. Ré Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems 36, p. 36000–36040. Cited by: Introduction. Elgaar and Amiri (2026) M. Elgaar and H. Amiri Curriculum learning for llm pretraining: an analysis of learning dynamics. arXiv preprint arXiv:2601.21698. Cited by: Online Data Control and Skill Acquisition Order. Gu et al. (2026) X. Gu, K. Lyu, J. Li, and J. Zhang Data mixing can induce phase transitions in knowledge acquisition. Advances in Neural Information Processing Systems 38, p. 163200–163244. Cited by: Online Data Control and Skill Acquisition Order. Hendrycks et al. (2020) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: Introduction. Liu et al. (2026) E. Liu, K. Sun, M. Li, I. Lee, L. Tjuatja, J. Huang, and G. Neubig What do language models learn and when? the implicit curriculum hypothesis. arXiv preprint arXiv:2604.08510. Cited by: Online Data Control and Skill Acquisition Order. Liu et al. (2025) Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin Regmix: data mixture as regression for language model pre-training. In International Conference on Learning Representations, Vol. 2025, p. 38305–38339. Cited by: Proxy Data Mixtures and Transfer. Ma et al. (2026) J. Ma, C. Dang, and M. Liao AC-odm: actor–critic online data mixing for sample-efficient llm pretraining. External Links: 2505.23878, Link Cited by: Proxy Data Mixtures and Transfer. Qwen et al. (2025) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: Introduction. Rukhovich et al. (2025) A. Rukhovich, A. Podolskiy, and I. Piontkovskaya Commute your domains: trajectory optimality criterion for multi-domain learning. arXiv preprint arXiv:2501.15556. Cited by: Online Data Control and Skill Acquisition Order. Shihab et al. (2026) I. F. Shihab, F. Afrin, and A. Sharma Dynamic proxy-mixing: transferring replay controllers from small to large models for continual instruction tuning. arXiv preprint arXiv:2606.00400. Cited by: Proxy Data Mixtures and Transfer. Wang et al. (2025) Y. Wang, B. Liu, F. Liu, Y. Guo, J. Deng, X. Wu, W. Zhou, X. Zhou, and T. Wang TiKMiX: take data influence into dynamic mixture for language model pre-training. arXiv preprint arXiv:2508.17677. Cited by: Online Data Control and Skill Acquisition Order. Weston et al. (2015) J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. Van Merriënboer, A. Joulin, and T. Mikolov Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. Cited by: Introduction. Xie et al. (2023) S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. S. Liang, Q. V. Le, T. Ma, and A. W. Yu Doremi: optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems 36, p. 69798–69818. Cited by: Proxy Data Mixtures and Transfer. Yang et al. (2024) Y. Yang, S. Mishra, J. Chiang, and B. Mirzasoleiman Smalltolarge (s2l): scalable data selection for fine-tuning large language models by summarizing training trajectories of small models. Advances in Neural Information Processing Systems 37, p. 83465–83496. Cited by: Proxy Data Mixtures and Transfer. Zhao et al. (2026) K. Zhao, Z. Miao, A. Aizawa, and Y. Tsuruoka RegMix-d: dynamic data mixing via proxy training trajectories. arXiv preprint arXiv:2606.18663. Cited by: Proxy Data Mixtures and Transfer.