Paper deep dive
Bridging Compute- and Data-Optimal Pretraining
Tian Qin, Kimia Hamidieh, David Alvarez-Melis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/1/2026, 11:30:12 AM
Summary
The paper introduces Compute-Data (CD) scaling laws, a unified framework bridging compute-optimal and data-optimal pretraining regimes. It proposes a token-effectiveness function (η) to quantify the value of derived tokens (from repetition or paraphrasing) relative to fresh data. Experiments on the Dolma-3 corpus with models from 14M to 600M parameters show that token effectiveness depends on model size, tokens-per-parameter ratio, and derived data amount, leading to diminishing returns and three operational regimes: compute-bound, data-bound, and model-bound.
Entities (10)
Relation Signals (10)
Compute-Data (CD) scaling laws → bridges → data-optimal scaling
confidence 95% · We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling... and data-optimal scaling
Compute-Data (CD) scaling laws → bridges → compute-optimal scaling
confidence 95% · We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling... and data-optimal scaling
token-effectiveness function → quantifies → value of derived token
confidence 92% · token-effectiveness function, η, which quantifies the value of a derived token... relative to a fresh token
Compute-Data (CD) scaling laws → partitions → model-bound
confidence 90% · It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound
Compute-Data (CD) scaling laws → partitions → compute-bound
confidence 90% · It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound
Compute-Data (CD) scaling laws → partitions → data-bound
confidence 90% · It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound
multi-epoch repetition → isusedin → Compute-Data (CD) scaling laws
confidence 88% · We fit η for two data-expansion strategies, multi-epoch repetition and paraphrasing... using the Dolma-3 corpus.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, $\eta$, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit $\eta$ for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of $\eta$ implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.
Tags
Links
- Source: https://arxiv.org/abs/2607.25271v1
- Canonical: https://arxiv.org/abs/2607.25271v1
Trouble viewing inline? Open PDF directly →
Full Text
61,965 characters extracted from source content.
Expand or collapse full text
Bridging Compute- and Data-Optimal Pretraining Tian Qin ∗ Harvard University Kimia Hamidieh ∗ MIT CSAIL David Alvarez-Melis Harvard University Abstract Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering the regime where compute is growing faster than high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, in which data scales freely with compute, and data-optimal scaling, in which the corpus is fixed and compute can grow unbounded. CD scaling extends classic scaling by introducing a token effectiveness functionηthat quantifies how much a derived token, produced for instance by multi-epoch repetition or paraphrasing, is worth relative to a fresh one, ranging from a perfect substitute to no value at all. Fittingηfor two data expansion strategies (multi-epoch repetition, paraphrasing) across model sizes ranging from 14M to 600M parameters on the Dolma-3 corpus, we find that it is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and saturates as the corpus is expanded. The functional form of the token effectivness functionη implies that substituting compute for data diminishes with both model size and data availability, and partitions training into three operational regimes: compute-bound, data-bound, and model-bound, showing that classic compute-optimal allocation is suboptimal across most of the practically relevant regime. 1 Introduction The classical scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) characterize how pretraining loss decreases with model sizeNand dataset sizeD, prescribing a compute-optimal allocation between the two. Implicit in this formulation is a data-abundance assumption: fresh tokens are freely available, and the pretraining corpusDscales naturally with compute. But this assumption is becoming inaccurate. Compute continues to grow at exponential rates (Sevilla et al., 2022), while high-quality pretraining data remains finite and costly to curate (Villalobos et al., 2024). Pretraining is therefore entering a regime that classical scaling laws were not designed for: one in whichD is bounded and additional compute must be spent on derived tokensD ′ produced fromDvia multi-epoch repetition, paraphrasing, or distillation. The central question becomes: For a model of size N trained on a fixed corpus D, how much can additional compute reduce loss? We approach this question through the lens of two limiting regimes. Compute-optimal scalingL Chin describes the setting where data is unbounded and compute is the bottleneck, and it is the regime characterized by classical scaling laws. Data-optimal scalingL D describes the opposite limit, where Dis fixed and compute is unbounded and it is the asymptote of what additional training, in any form, can achieve on a given corpus D. Modern pretraining sits between these two limits (Figure 1a). We propose Compute-Data (CD) scaling laws, a unified framework that bridges the compute- and data-optimal scaling. At its center is an effectiveness functionη strat ∈ [0, 1]that quantifies, for a given data expansion strategy such as paraphrasing or repetition, how much a derived tokenD ′ is ∗ Equal contributions. Correspondence to tqin@g.harvard.edu, hamidieh@mit.edu Preprint. arXiv:2607.25271v1 [cs.LG] 28 Jul 2026 100M1B10B Fresh tokens D 3 4 5 6 7 8 Validation loss L (a) compute-optimal scaling data-optimal scaling Compute-bound Data-bound Model-bound 100M1B10B100B Effective tokens D eff = D + ηD ′ (b) Model-bound 100M1B10B100B Total tokens trained D + D ′ 100M 1B 10B 100B D eff (c) N 30m 60m 370m y = x (full credit) TTP (D/N) 2 20 320 Figure 1: CD-scaling law overview. (a) For a model of sizeN, classical scaling laws assume 1-epoch training on fresh data, which coincides with the compute-optimal Pareto frontier. Spending additional compute on derived tokensD ′ produced fromD, via data expansion strategies such as multi-epoch repetition, paraphrasing, or distillation, departs from this frontier and reaches lower loss without curating more fresh data. (b) To describe training behavior beyond the compute-optimal frontier, CD-scaling laws introduce a token effectiveness functionηthat maps each derived token to a fresh-data equivalent. Under this mapping, runs beyond the compute-optimal frontier collapse onto the classical scaling line. (c) Both the saturation level of effective tokens and the rate of approach depend onNandTPP: as the model grows and data becomes more abundant, additional compute brings diminishing returns. worth relative to a fresh one (Figure 1b). The resulting scaling law is L(N,D,D ′ ) = E + A N α + B (D + η· D ′ ) β , withη = 1meaning derived tokens are as informative as fresh data andη = 0meaning they provide no benefit. The law continuously interpolates between the two regimes: whenD ′ = 0it recovers compute-optimal scaling, and asD ′ →∞it approaches the data-optimal limit, predicting the best achievable loss for a given (N,D). We focus on two data-expansion strategies, multi-epoch repetition and paraphrasing. To fit CD-scaling laws, we sweep model sizesNfrom 14M to 600M parameters, fresh-data sizesDfrom 30M to 30B tokens, and derived-token budgetsD ′ from 30M to 120B tokens, with hyperparameter search over learning rate and weight decay. We find that the effective-token countD eff = D + η D ′ saturates asD ′ → ∞, with both the saturation level and the rate of saturation depending onNand the tokens-per-parameter ratioTPP = D/N(Figure 1c). This observation allows us to determine the functional form of the effectiveness coefficientη, which has a power-law decay overN,DandD ′ . The data-constrained scaling law of Muennighoff et al. (2025) is recovered as the special case in which the saturation level is constant in (N,D). The fittedη strat yields three practical implications. First, between the compute-optimal scalingL Chin and data-optimal scalingL D , the proposed CD scaling law identifies three regimes: compute-bound, data-bound, and model-bound. This informs a practitioner which resource is currently limiting their run. Second, for a given(N,D), it prescribes the optimal compute to spend onD ′ , and generalizes Chinchilla’s compute-only optimization into a joint compute–data Pareto frontier. Third, CD-scaling prescribes which expansion strategy to use and how much additional data to train on. Concretely, extrapolated CD-scaling law shows that the 4-epoch rule holds only for medium-scale models (∼ 3B) near1×Chinchilla, with recommended epochs decreasing asNandTPPgrow. Paraphrasing is preferred for small models (≲ 600M) and sub-Chinchilla data budgets, but is ineffective forN ≥ 7B or large budgets (≥ 4× Chinchilla). In summary, our contributions are: • A unified CD-scaling framework. We propose CD-scaling laws, which bridge the compute- optimal and data-optimal regimes through an effectiveness function η strat . •Effectiveness of derived tokens diminishes with scale. Fittingηon Dolma-3 across model sizes from 14M to 600M, we find that the effectiveness of spending additional compute beyond 1-epoch training decays with both N and TPP. 2 •Three practical implications. (a) The CD law identifies three regimes—compute-, data-, and model-bound—that inform which resource is currently limiting the pretraining run. (b) For a given (N,D), it prescribes the optimal compute–data allocation, generalizing Chinchilla’s compute-only optimization into a joint compute–data Pareto frontier. (c) It prescribes when to switch from repetition to paraphrasing, and provides recommendation on optimal epochs or paraphrasing based on training configurations. 2 Related Work Scaling laws. The classical scaling laws of Kaplan et al. (2020) and Hoffmann et al. (2022) characterize loss as a function of model sizeNand fresh pretraining tokensD, prescribing a compute-optimal allocation under the assumption thatDscales freely with compute. Subsequent work has investigated each side of this assumption. Gadre et al. (2024) show that the Chinchilla recipe extrapolates reliably into the substantially over-trained regime (TPP ≫ 20). Muennighoff et al. (2025) extend Chinchilla to the data-constrained regime, fitting a single saturation budget for repeated tokens. Yan et al. (2025) provide a complementary theoretical perspective, proving in linear regression that the effective reuse rate of repeated data grows with model sizeN, withΘ(logN ) scaling under strong convexity and power-law scaling under Zipf-distributed features. Kim et al. (2025) take the opposite limit, asking what loss is achievable under fixedDand unbounded compute. Concurrent to our work, Lovelace et al. (2026) also extend Chinchilla to repeated data, but model repetition through an additive, model-size-dependent overfitting penalty that captures the regime where loss rises with further epochs, rather than through an effectiveness function on derived tokens. Our work unifies these threads by recovering Hoffmann et al. (2022) as theD ′ = 0limit, the law of Muennighoff et al. (2025) as the constant-saturation special case, and the data-optimal limit of Kim et al. (2025) as D ′ →∞. Synthetic data for pretraining.A growing body of work treats synthetic data as a lever for trading compute against fresh-token scarcity. Maini et al. (2024) show that rephrasing web documents into structured styles accelerates pretraining by∼3×, and DatologyAI et al. (2025) extend the recipe to trillion-token regimes. Kang et al. (2025) conduct a controlled study across synthetic-data types, finding that mixing∼30%rephrased data yields5–10×training speedups. Askari-Hemmat et al. (2025) dynamically target synthetic data at the model’s current knowledge gaps; Yang et al. (2025) learn inter-document relations to generate diverse synthetic continuations. Each of these works characterizes a single strategy in isolation. We instead fit a common functional form for the token- effectiveness functionηacross multi-epoch repetition and paraphrasing, and treat each strategy as a parameterization of the same underlying law. Distillation and other compute-for-data exchanges. Distillation (Hinton et al., 2015) has seen renewed interest in LLM pretraining as a way to extract additional signal from a fixed corpus. Kim et al. (2025) show that single-student distillation from a multi-epoch teacher recovers most of the loss improvement of the ensemble. Busbridge et al. (2025) establish a teacher–student scaling law that allocates compute optimally between training the teacher and the student. These works model distillation as an internal allocation of compute under a fixed corpus; in our framework, self-distilled tokens are a natural instance of D ′ that the same η function describes, although our empirical fits in this paper focus on multi-epoch repetition and paraphrasing. Data allocation and mixing.A complementary line of work allocates compute across the composi- tion of the fresh corpus rather than between fresh and derived tokens: DoReMi (Xie et al., 2023), Reg- Mix (Liu et al., 2025), and the data-mixing scaling laws of Ye et al. (2024) optimize mixture weights across pretraining domains, and mixture dependent scaling laws of Hamidieh et al. (2026) predict model performance by accounting for data synergy. CD-scaling is orthogonal: we hold the fresh- corpus composition fixed and study how compute should be split between fresh D and derived D ′ . 3 3 Methodology and Experiment Setup 3.1 Compute-Data Scaling Law The classical Chinchilla scaling law is L Chin (N,D) = E + A N α + B D β ,(1) whereNis the number of model parameters andDis the number of fresh pretraining tokens, each seen once. To accommodate synthetic and augmented data, we extend Eq. (1) into a compute–data (CD) scaling law that shares the constants (E,A,B,α,β): L CD (N,D,D ′ ) = E + A N α + B (D + η strat (N, TPP,D ′ /D)· D ′ ) β ,(2) whereD ′ denotes additional tokens produced from a data-expansion strategy (multi-epoch repetition, paraphrasing, or other forms of synthetic data generation and augmentation methods). The effective- ness coefficientη ∈ [0, 1]predicts how much one derived token inD ′ is worth relative to a fresh token fromD. In Section 4.1, we motivate parameterizingηin terms of the three more interpretable quantities(N, TPP,D ′ /D)rather than(N,D,D ′ ), and in Section 4.2, we derive a parametric form for η strat . Eq. (2) has three natural limits. SettingD ′ = 0recovers the classic scaling lawL Chin . Taking D →∞sends the third term to zero, leaving the model-bound floorL N = E + A/N α , the lowest loss a model of sizeNcan reach under unlimited fresh data. Finally, and most importantly, taking D ′ →∞at fixedDdefines data-optimal scalingL D : the lowest loss attainable on a fixed corpus Dwhen unlimited compute is spent on derived tokens. The exact form ofL D depends onηand is derived in Section 4.2. 3.2 Experiment Setup Data, model, and training. We use the OLMo3 (Olmo et al., 2025) architecture and training infrastructure for all pretraining runs. We train models ranging fromN = 14M toN = 600M parameters, with fresh dataDranging from30M to30B tokens and derived tokensD ′ from30M to 120B. Fresh tokens are sampled from the Dolma-3 150B corpus (Olmo et al., 2025) 2 , with smallerD always a strict subset of largerD. All models are trained at sequence length 4096, batch size 512, with AdamW (Loshchilov and Hutter, 2019) and cosine learning-rate decay. We sweep learning rate∈ 1e-4, 3e-4, 1e-3, 3e-3and weight decay∈ 0.1, 0.2, 0.4, 0.8, 1.6atN ∈ 30M, 370M, selecting the configuration that minimizes validation loss; for other model sizes we perform a local grid search around the selected configuration. Our primary metric is validation loss on a held-out set of 5M Dolma-3 documents. Data expansion strategies.We extend pretraining compute beyond 1-epoch training onDvia two data-expansion strategies, each producing additional tokens D ′ : 1. Multi-epoch (Muennighoff et al., 2025): repeat the original corpus for additional epochs. 2.Paraphrasing (i.e., augmentation) (Maini et al., 2024; Kang et al., 2025): rephrase each document inDusing SmolLM2-1.7B-Instruct as the paraphrasing model (Allal et al., 2025), sampling up to 16 paraphrases per document with the style for each seed drawn uniformly from question, math (or wiki when not applicable), FAQ, table, a four-style mixture identified as most effective in prior analysis (Niklaus et al., 2026). Models are then trained on a mixture of freshDand paraphrasedD ′ for one epoch. Paraphrased documents are always derived from documents in D, and smaller paraphrased corpora are nested in larger ones. We do not include the FLOPs spent on paraphrase generation in training compute, as paraphrasing can be performed asynchronously and is not part of the training loop. 4 Parametric Form of CD-Scaling Law We first characterizeηempirically (Section 4.1), then we use those observations to motivate a functional form (Section 4.2), and fit the resulting closed-form (Section 4.3). We validate that 2 https://huggingface.co/datasets/allenai/dolma3_mix-150B-1025 4 10 0 10 1 10 2 TPP = D/N 10 1 10 0 η (a) 30M: η vs TPP r = D ′ /D r = 1 r = 3 r = 7 r = 15 r = 31 r = 63 10 0 10 1 10 2 r = D ′ /D 10 2 10 1 10 0 η (b) 30M: η vs r TPP = D/N TPP = 1 TPP = 5 TPP = 20 TPP = 80 TPP = 320 10 0 10 1 10 2 r = D ′ /D 10 0 10 1 η · D ′ /D (c) TPP = 2, across N scales N (params) 14m R ∗ =18.9 30m R ∗ =14.1 60m R ∗ =10.8 190m R ∗ =6.9 370m R ∗ =5.3 Figure 2: Empirical characterization ofη(y-axis isη emp throughout). (a)ηvs.TPPat fixedD ′ /D: ηdecreases monotonically withTPP, so the portion of fresh data in the pretraining set matters. (b)η vs.D ′ /Dat fixed(N,D): monotone decay; together with (c), this is decay toward a finite saturation ceiling, not decay to zero. (c)η· D ′ /Dvs.rat fixed(N,D): extra effective tokens plateau asD ′ grows. The plateau is a fundamental property ofη(and does not arise from under-fitting): no matter how many derived tokens we add, the loss curve flattens, and any functional form for whichη· D ′ diverges withD ′ is ruled out. We also observe that larger models saturate faster, consistent with findings in Section 5.1. the fit extrapolates to held-out model sizes and show ablations on the parametric form choice (Section 4.4). Finally, we confirm that validation loss improvements transfer to downstream benchmark performance (Section 4.5) across training settings. 4.1 Empirical Characterization of η Before specifying a parametric form forη strat , we measure it empirically and examine how it varies with(N,D,D ′ ). To obtainη emp , we first fit the classic scaling law (Eq. (1)) to obtain constants (E,A,B,α,β). For each repetition and paraphrasing run, we then define the empirical coefficient η emp as the unique value that, substituted into Eq. (2) alongside these constants, reproduces the observed lossL. We expectη emp ∈ [0, 1], though it can exceed1when the 1-epoch fit underestimates a run’s loss, as any such residual is absorbed into η emp . We discuss this limitation in Section 6. Change of variables.Althoughη strat is naturally a function of(N,D,D ′ ), we recast it in terms of three more interpretable variables: the tokens-per-parameter ratioTPP = D/N, the expansion ratio r = D ′ /D, and the model sizeN.TPPmeasures fresh-data availability relative to model capacity, and is the conventional unit in which Chinchilla scaling is described. The expansion ratiormeasures how aggressivelyD ′ expands beyondD. In the multi-epoch case,ris simply the number of extra epochs. We retainNas a separate variable so thatTPPcaptures data availability andNcaptures model size alone. Empirical observations. In Figure 2, we examine howη emp depends on(r, TPP,N ). Panel (a) shows thatη emp decays withTPP, and the decay steepens asrgrows. Panel (b) shows thatη emp decays withr, and the decay rate depends onTPP: slowly whenTPPis small, sharply whenTPP is large. The quantity that matters for total loss, however, is notηbut the effective derived dataη·D ′ , orη·ronce normalized byD. This quantity represents the fresh-equivalent tokens the derived corpus D ′ contributes. Rather than growing without bound,lim r→∞ η· rshould saturate. We visualize the saturation behavior in panel (c). Both the saturation limit and the rate of saturation depend onN, with larger models saturating at a lower value and at a faster rate. 4.2 Functional Form of η The empirical observations impose three requirements: (i)η emp → 1asr → 0, (i)η· rsaturates to a finite limit asr → ∞, and (i) the saturation behavior depends onTPPandN. Among the functional families satisfying the first two conditions, we adopt the exponential form below, and Section 4.4 shows it provides the best fit when compared to all other forms considered: η = R ∗ r 1− e −r/R ∗ (3) 5 Table 1: Fitted CD-Scaling parameters with 95% CIs. Top: 1-epoch Chinchilla parameters, Bottom: per-strategy CD-Scaling parameters. RMSE onlogL, reported separately for 1-epoch, repetition and paraphrasing training runs. EABαβRMSE 1ep 1.35 [1.0, 1.6] 205 [90, 676] 16,597 [7,677, 34,475] 0.283 [0.22, 0.37] 0.435 [0.40, 0.47] 0.043 logKρσRMSE Repetition 10.93 [9.7, 13.5] −0.42 [−0.71, −0.15] −0.41 [−0.55, −0.35] 0.035 Paraphrase 30.50 [30.4, 30.8] −1.52 [−1.71, −1.22] −1.30 [−1.35, −1.26] 0.024 100M1B10B100B Tokens D 10 0 10 1 Data-Reducible Loss L − ( E + A/N α ) 1-Epoch Repetition Paraphrase Classic Scaling Law Fit 100M1B10B100B Effective Tokens D eff = D + η strat D ′ N 14m 30m 60m 100m 190m 370m 600m Figure 3: Fitted CD-scaling law across model sizes and strategies. Data-reducible lossL− (E + A/N α )across model sizes and training settings (1-epoch, repetition, and paraphrase). The solid line is the classic scaling-law fit on 1-epoch runs. Left: Data-reducible loss against fresh tokens D. Repetition and paraphrase runs train on extraD ′ tokens, pushing their loss below what fresh Dalone achieves, so these runs fall beneath the classic scaling law. Right: The CD-scaling law converts eachD ′ into its fresh-token equivalentη strat D ′ . Re-plotted against the effective token count D eff = D + η strat D ′ , the paraphrase and repetition runs reduce back to the classic scaling law. Interpretation ofR ∗ . The exponential form gives the identitylim r→∞ η· r = R ∗ , soR ∗ can be interpreted as the saturation ceiling: the maximum number of fresh-equivalent tokens a corpus of sizeDcan yield through data expansion, expressed as a multiple ofD. This ceiling implies that total effective data saturates atlim D ′ →∞ D eff = D(1 + R ∗ ). Applying this identity to Eq. (2) yields a closed form for data-optimal scalingL D , the lowest loss a given data-expansion strategy can reach on a fixed corpus of size D: L D (N,D) = E + A N α + B [D (1 + R ∗ (D,N ))] β .(4) Functional form of R ∗ . Based on observations in Section 4.1, we propose a power law for R ∗ : R ∗ (D,N ) = K· (D/N ) ρ · N σ ,(5) whereρgoverns howR ∗ tightens withTPPandσhowR ∗ tightens withN. We overlay the proposed form in Figure 2, and confirm that the power law fits the observed qualitative behavior. Together, Eqs. (3) and (5) determine the token effectiveness functionη. Substitutingηinto Eq. (2) gives the complete CD-scaling law, containing eight constants: the five 1-epoch Chinchilla parameters(E,A,B,α,β)and three strategy-specific parameters(K,ρ,σ). The repeated-data law of Muennighoff et al. (2025) is the special case in whichR ∗ is constant and independent of (D,N ). Ablations in Section 4.4 (Table 3) confirm that this constant-R ∗ assumption fits worse, and that the(D,N )-dependence in Eq. (5) is what makes the law predictive across model sizes and data-availability regimes. 4.3 Fitting CD-Scaling Law We follow the fitting procedure introduced by Besiroglu et al. (2024). Namely, we use a Huber loss on the residuals oflogL. We jointly optimize all parameters: the5parameters of the 1-epoch scaling 6 30M60M190M600M1B3B Model size N 0.1 0.3 1 3 10 30 100 R ∗ (saturation budget) crossover N 94M CIs overlap (a) TPP = 20 (1× Chinchilla) R ∗ vs N Repetition Paraphrase 3510204080160 TPP (D/N) 0.03 0.1 0.3 1 3 10 30 R ∗ (saturation budget) crossover D/N 4 CIs overlap (b) N = 600M R ∗ vs TPP Figure 4: Saturation ceilingR ∗ across strategies. Lines show fittedR ∗ , shaded bands show95% CIs, and the grey region marks where the two CIs overlap. (a)R ∗ vs.NatTPP = 20. Paraphrasing significantly wins for small models (N ≲ 50M) and repetition for large ones (N ≳ 190M). (b)R ∗ vs.TPPforN = 600M. The CIs overlap at smallTPP, so paraphrasing does not meaningfully beat repetition. Beyond TPP≈ 8, repetition significantly wins. law(E,A,B,α,β), together with3per-strategy saturation parameters(K,ρ,σ)for repetition and paraphrasing. We provide the detailed fitting procedure in Appendix A and report the fitted parameters in Table 1. To assess fit quality, we separately report the RMSE oflogLfor 1-epoch, repetition, and paraphrasing runs. We also use 95% confidence intervals (CIs) to quantify uncertainty in the fitted parameters. Specifically, we resample training runs with replacement and refit the parameters on the resampled data. We repeat this resampling and refitting procedure200times and take the95%confidence interval. We report RMSE and per-parameter confidence intervals in Table 1 as well. Fitting the CD-scaling law provides a principled way to compare data-expansion strategies. As an example, we compare saturation ceilingsR ∗ (Section 4.2). Because we fit CIs onR ∗ , we can go beyond point estimates and ask whether the difference between strategies is statistically significant. In each panel of Figure 4, the shaded band marks where the two95%CIs overlap. At the Chinchilla- optimal budget (TPP = 20), Figure 4 (a) comparesR ∗ across model sizes: the repetition and paraphrasing CIs overlap in a band around the crossover (N ≈ 94M), but paraphrasing significantly wins for small models (N ≲ 50M) and repetition for large ones (N ≳ 190M). At a fixed size (N = 600M), Figure 4 (b) comparesR ∗ acrossTPP: the CIs overlap at smallTPP, so paraphrasing does not meaningfully beat repetition, whereas beyondTPP≈ 8repetition wins. We map the full crossover boundary across (N, TPP) in Section 5.2. 4.4 Cross-Scale Validation and Ablations Cross-scale validation.A practical motivation for fitting Eq. (3) is to predict loss for the expensive large-model runs from cheaper small-model runs. We now examine whether the CD-scaling law can extrapolate beyond the model sizes used in fitting. To test this, we refit Eq. (3) on runs at N ∈14, 30M only, then predict held-out losses atN ∈60, 100, 190, 370, 600M. The held-out set spans model sizes2×to20×larger than any used in fitting,TPPup to∼160(8×Chinchilla), and rup to63(64 epochs), which covers token-count regimes well outside the fitting range. Additionally, we sweep on model sizes used for fitting in Appendix B.1. The small-N fit achieves RMSE 0.079 on logL on the held-out points, against an in-sample RMSE of0.048. Our scaling law therefore transfers cleanly across an order of magnitude in model size, and a practitioner can fit Eq. (3) on small models with modest budget and read offR ∗ (D,N )at production scale to anticipate how much additional data will reduce loss. Ablations.We validate the exponential functional form Eq. (3) and theR ∗ form Eq. (5) using multi- epoch training runs, reporting leave-one-out (LOO) RMSE onlogLunder the same fitting procedure. For the functional form ofη, we compare nine candidates (Appendix A.3, Table 3). The forms that letR ∗ depend on(D,N )via Eq. (5) dominate the rest at every parameter count; in particular, the constant-R ∗ data-repetition baseline (Muennighoff et al., 2025) is rejected by∼40%in relative LOO RMSE. Within the top group, our chosen form is statistically indistinguishable from the best, and we prefer it because it generalizes the constant-R ∗ form and satisfies η(0) = 1 exactly. 7 1.01.52.02.53.0 Val BPB ( ↓) 1 2 3 Avg. BPB ( ↓ ) 1.01.52.02.53.0 Val BPB ( ↓) 15 20 25 30 35 Avg. Accuracy % ( ↑ ) multi-epoch N=14M N=30M N=60M N=100M N=190M N=370M N=600M paraphrase N=14M N=30M N=60M N=100M N=190M N=370M N=600M Figure 5: Downstream performance as a function of validation BPB. Left: average BPB on benchmarks recast as language-modeling tasks (GSM8K, TriviaQA, HumanEval, etc.). Right: mean accuracy on LAMBADA, HellaSwag, OpenBookQA, RACE, and SQuAD. In both panels, points from multi-epoch and paraphrase runs collapse onto a single curve, indicating that validation loss is a sufficient statistic for downstream capability regardless of training settings. ForR ∗ , we remove terms from Eq. (5) and refit (Appendix A.4, Table 4). Dropping either exponent degrades the fit, and dropping both recovers the constant-R ∗ baseline: each exponent is individually necessary, and using both is strictly better than either alone. 4.5 Loss-to-Downstream Transfer So far we have compared validation loss across training settings. We now ask whether validation loss remains a valid predictor of downstream capability across data-expansion strategies and model scales. We group benchmarks into LM-based and accuracy-based tasks. Benchmarks such as GSM8K (Cobbe et al., 2021), TriviaQA (Joshi et al., 2017), and HumanEval (Chen et al., 2021) yield near-random accuracy at small-to-medium scale. Following Gadre et al. (2024), we recast them as language- modeling tasks and report bits-per-byte (BPB) of the gold response given the prompt, which stays predictive even when accuracy is near-trivial. For accuracy-based tasks, we report mean accuracy across LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2018). We plot both against validation BPB in Figure 5. In both cases, runs from all strategies and scales fall onto a single curve as a function of validation BPB alone, which confirms that validation loss predicts downstream capability independent of training setting. We report per-task breakdowns in Section B.2. 5 CD-Scaling Laws Bridge Compute- and Data-Optimal Training In this section, we discuss three implications of CD-scaling laws. First, for a fixedN, the CD law predicts the compute-data Pareto allocation betweenDandD ′ (Section 5.1). Second, it partitions training into three regimes, identifying which resource is the binding constraint for a given(N,D,D ′ ) (Section 5.2). Third, it tells a practitioner which data-expansion strategy to adopt and how much compute to spend before returns saturate (Section 5.3). 5.1 Compute-Data Pareto-Optimal Allocation A practitioner training a model of sizeNon a fixed corpusDmust decide how much compute to spend on further loss reduction. Under Chinchilla scaling, compute is fixed by(N,D)via C ≈ 6ND, leaving no such freedom. DecouplingCfromDthrough multi-epoch training or paraphrasing adds a degree of freedom, giving a two-dimensional loss surface over(C,D)with C ≈ 6N (D + D ′ ). Figure 6 (left) shows this empirical surface forN = 30M, with computeCon thex-axis and fresh dataDon they-axis. The black dashed line traces single-epoch runs, where the surface collapses onto a single ray. The contours around single-epoch training are far steeper inC 8 10 16 10 17 10 18 10 19 FLOPs 1 2 5 10 20 40 80 160 320 Fresh Data D (TPP, D/N) Empirical IsoLoss Contours 3.50 3.55 3.60 3.65 3.70 3.75 3.80 3.85 3.90 3.95 4.00 4.20 4.40 4.60 4.80 5.00 5.20 5.40 6.00 6.50 7.00 7.50 8.00 8.50 10 16 10 17 10 18 10 19 FLOPs Predicted IsoLoss Contours 3.50 3.55 3.60 3.65 3.70 3.75 3.80 3.85 3.90 3.95 4.00 4.20 4.40 4.60 4.80 5.00 5.20 5.40 6.00 6.50 7.00 7.50 8.00 8.50 4 5 6 7 8 Validation Loss Model-optimal L N Chinchilla-optimal L chin Single-epoch scaling path Empirical optimal (C,D) scaling path Predicted optimal (C,D) scaling path Figure 6: Optimal compute-data allocation on (C,D) loss surfaces. Left: 1-epoch compute- optimal training scales computeCproportionally with fresh dataD, reducing the (C,D) loss surface into a single ray (black). Loss contour lines around it indicates that small increase inCwould significantly lower the loss. Tracing the sharpest direction of descent, the empirical optimal scaling path (teal) prefers scaling compute at data-scarce regime, then scaling both data and compute. Right: Our proposed CD-scaling law predicts the optimal path that jointly optimizes (C, D). than inD, indicating that modest compute beyond the single-epoch buys substantial loss reductions that single-epoch scaling leaves on the table. The empirical Pareto frontier is non-trivial. The teal curve traces the loss-minimizing path through(C,D)space via steepest descent on the empirical surface. Along it, the contours bend sharply in both directions, meaning increasingCorDalone would leave loss roughly unchanged. Since neither resource can reduce loss in isolation, this line traces the compute-data Pareto frontier. This frontier gives practitioners a general guidance on resource allocation. In the data-scarce, low- compute regime (bottom-left), increasing compute is far more effective than adding fresh data, even when data is scarce. AsCgrows, the frontier rotates toward scalingCandDin equal proportion, then enters a basin where fresh data dominates and further compute yields diminishing returns. CD-scaling predicts this frontier.Tracing the steepest-descent path on the CD-scaling loss surface (Figure 6 right) yields a trajectory that closely tracks the empirical Pareto curve, generalizing Chinchilla’s compute-only optimization into a joint compute-data optimization. 5.2 Three Training Regimes As fresh data become the binding constraint, instead of plotting validation loss against training compute, we plot against fresh data, with color indicating the compute budgetC ≈ 6N (D + D ′ ), shown in Figure 7. The three limits of CD-scaling:L Chin (D ′ = 0),L D (D ′ → ∞), andL N (D →∞) appear as three boundary curves that carve the (L, D) plane into three distinct regions: •Compute-bound (betweenL Chin andL D ): for a givenD, additionalD ′ still reduces loss. Compute is the binding resource. • Data-bound (betweenL D andL N ):D ′ has saturated toR ∗ D, only additional data can further reduce loss. • Model-bound (atL N ): both compute and data have saturated. N is the binding resource. We then overlay the three region boundaries predicted by CD-law against empirical training losses. Single-epoch points lie onL Chin . Intermediate-D ′ traces countour lines in the compute-bound region. Importantly, the predicted data-optimal frontierL D coincides with the observedD ′ →∞ asymptote of the training runs, validating that the CD-scaling’s saturation limit correctly models the empirical limit. 9 110100 TPP (D/N) 3 4 5 6 7 8 9 Validation Loss Compute-bound Data-bound Model-bound add compute add fresh data 2 4 6 8 30M60M 10 16 10 17 10 18 10 19 10 20 FLOPs 10 0 10 1 10 2 TPP 2 4 6 8 190M 10 0 10 1 10 2 TPP 370M CD-Scaling Law Prediction CD-law: D ′ /D = 0 CD-law: D ′ /D →∞ CD-law: D →∞ Models trained Compute-bound Data-bound Model-bound Figure 7: The CD-scaling law captures three training regimes. Left:N = 30M empirical losses overlaid with CD-scaling law predictions. By sweepingD ′ from0(L Chin , compute-optimal limit) to∞(L D , data-optimal limit), CD-scaling law traces out all achievable losses fromL Chin toL D . Right: The same decomposition across model sizesN. AsNgrows, the compute-bound region bounded byL Chin andL D tightens, indicating that for larger models, increasing computeCyields diminishing returns in further loss reduction. The compute-bound region shrinks with bothTPPandN. The vertical gap betweenL Chin andL D is governed by the saturation ceilingR ∗ . WhenR ∗ is large, the data-optimal frontier sits well below the compute-optimal one. In this training regime, additional training compute spent onD ′ delivers substantial loss reductions. In contrast, whenR ∗ is small, the two frontiers nearly coincide and additional compute spent onD ′ is largely ineffective. In Section 4.1, we empirically observed thatR ∗ shrinks with bothNandTPP. In Figure 7, we confirm that the fitted CD-scaling law correctly models the shrinking rate over TPP (left), and N (right). 5.3 Practical Implications By visualizing saturation ceilingR ∗ predicted by CD-scaling laws, we can directly understand at differentNandTPP, which of the two data expansion strategy to adopt and the best achievable outcome, as shown in Figure 8 (top). Figure 8 (bottom) then reports the training effort needed to reach90%of suchR ∗ ’s. Namely, we report the total epochs for repetition runs and total paraphrasing passes for paraphrase runs. We provide three takeaways: • The recommended epoch count decreases as model size and data budget grow. Extrapolating the fitted law to large model scales indicates that 4-epoch training is recommended only for medium-scale models (∼ 3B) around 1× Chinchilla data scale. •For smaller models and data budgets, paraphrasing is preferred. Concretely, paraphrasing beats repetition for models ≲ 600M and small data budgets (< 1× Chinchilla). • Extrapolating the fitted law to large model scales indicates that paraphrasing becomes ineffective for N ≥ 7B and large data budgets (≥ 4× Chinchilla). 6 Conclusion & Discussion We establish a unified framework that bridges compute- and data-optimal training. Through the effectiveness functionη strat , we characterize how much additional compute can reduce loss on a fixed pretraining corpusD. Building on it, we derive three practical implications: the compute-data Pareto-optimal allocation betweenDandD ′ , the three-regime picture (compute-bound, data-bound, and model-bound) that identifies the binding constraint, and guidance on which expansion strategy to 10 3 5 10 20 40 80 160 TPP (D/N) 2620129.17.56.03.82.7 21169.87.46.04.93.12.1 16127.35.54.53.62.31.6 128.85.44.13.42.71.71.2 8.86.64.13.12.52.01.30.90 6.64.93.02.31.91.50.960.67 4.93.72.31.71.41.10.710.50 R ∗ Repetition 6202475322125.81.40.44 2821122410.05.32.70.620.20 97398.33.41.80.920.210.07 33132.91.20.620.310.070.02 114.60.990.410.210.110.030.01 3.91.60.340.140.070.040.010.00 1.40.540.120.050.030.010.000.00 R ∗ Paraphrase 10 2 10 1 10 0 10 1 10 2 R ∗ 30M60M190M370M600M1B3B7B Model size N 3 5 10 20 40 80 160 TPP (D/N) 614629221815107 50372318151286 3728181411965 282114109754 21161087643 1612865433 1210654432 Recommended epochs (Ep@90%) 30M60M190M370M600M1B3B7B Model size N 443417643811578342103 201980317371381941 69427660251372<1 23995218421<1 82337321<1<1 2811211<1<1<1 1041<1<1<1<1<1 Recommended K passes (K@90%) Paraphrase better (R ∗ para > R ∗ rep )Repetition better (R ∗ rep > R ∗ para ) Figure 8: When to repeat, when to paraphrase, and how much. Top: Saturation ceilingR ∗ across NandTPPfor repetition (left) and paraphrase (right). The boundary marks where the winning strategy switches. Paraphrase wins in the lower-left (smallN, low data budget), repetition wins in the upper-right. Bottom: Training compute to reach90%ofR ∗ . We provide recommended epochs (left) for repetition and passes K (right) for paraphrase. Both decrease as N and TPP grow. adopt and how much compute to spend before returns saturate. Together these give practitioners a principled basis for deciding when to invest in compute, fresh data, or model capacity. Predictingηfrom corpus statistics. We currently fitη strat empirically per strategy. A natural extension is to predictηfrom dataset statistics alone, using summary statistics such as token diversity,n-gram overlap, or perplexity distributions. Such a predictive theory would let practitioners assess an expansion strategy on a given corpus before committing compute, making the framework substantially more practical. Limitations. The functional form we adopt has a clean interpretation, butηabsorbs any residual misspecification of the 1-epoch fit (and can pushη emp above 1), so the overall fit is sensitive to the quality of the 1-epoch runs. Within each run, we swept only learning rate and weight decay. Other hyperparameters (batch size, sequence length, schedule) were held fixed and could in principle interact withη. Establishing these laws incurs substantiation computation cost. To complete training runs reported in this study, we have consumed over 250k H100-hours. Even with this compute budget, we could only fit scaling laws for two data expansion strategy on small-to-medium scale model sizes. Finally, we evaluate two data-expansion strategies (repetition and paraphrase); others such as self-distillation or synthetic structured data (Lee et al., 2026) remain unexplored, as do combinations of strategies that may yield higher effective η than any single one. Acknowledgments DAM acknowledges support from the Kempner Institute, FAS Dean’s Competitive Fund for Promising Scholarship, Aramont Fellowship Fund, and the NSF AI-SDM Institute (Grant No. IIS-2229881). Additionally, we thank Antonio Torralba for helpful discussions and guidance on this project. 11 References Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart ́ ın Bl ́ azquez, Guilherme Penedo, Lewis Tunstall, Andr ́ es Marafioti, Hynek Kydl ́ ı ˇ cek, Agust ́ ın Piqueres Lajar ́ ın, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Cl ́ ementine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big – data-centric training of a small language model, 2025. URL https://arxiv.org/abs/2502.02737. Reyhane Askari-Hemmat, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano. Improving the Scaling Laws of Synthetic Data with Deliberate Practice, February 2025. URLhttp://arxiv. org/abs/2502.15588. arXiv:2502.15588 [cs]. Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024. Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distil- lation Scaling Laws, July 2025. URLhttp://arxiv.org/abs/2502.08606. arXiv:2502.08606 [cs]. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. DatologyAI, Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Charvi Bannur, Christina Baek, Darren Teh, David Schwab, Haakon Mongstad, Haoli Yin, Josh Wills, Kaleigh Mentzer, Luke Merrick, Ricardo Monti, Rishabh Adiga, Siddharth Joshi, Spandan Das, Zhengping Wang, Bogdan Gaza, Ari Morcos, and Matthew Leavitt. BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining, August 2025. URLhttp://arxiv.org/abs/2508. 10975. arXiv:2508.10975 [cs]. Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks, 2024. URL https://arxiv.org/abs/2403.08540. Kimia Hamidieh, Lester Mackey, and David Alvarez-Melis. Domain-aware scaling laws uncover data synergy. arXiv preprint arXiv:2607.11052, 2026. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015. URL http://arxiv.org/abs/1503.02531. arXiv:1503.02531 [stat]. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, March 2022. URLhttp://arxiv.org/ abs/2203.15556. arXiv:2203.15556 [cs]. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601– 1611, 2017. 12 Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, and Carole-Jean Wu. Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls, October 2025. URL http://arxiv.org/abs/2510.01631. arXiv:2510.01631 [cs]. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http://arxiv.org/abs/2001.08361. arXiv:2001.08361 [cs]. Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute, September 2025. URLhttp://arxiv.org/abs/2509.14786. arXiv:2509.14786 [cs]. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 785–794, 2017. Dan Lee, Seungwook Han, Akarsh Kumar, and Pulkit Agrawal. Training language models via neural cellular automata, 2026. URL https://arxiv.org/abs/2603.10055. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. RegMix: Data mixture as regression for language model pre-training. In International Conference on Learning Representations (ICLR), 2025. URLhttps://arxiv. org/abs/2407.01492. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URLhttps: //arxiv.org/abs/1711.05101. Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, and Kilian Q. Weinberger. Prescriptive scaling laws for data constrained training, 2026. URLhttps://arxiv.org/abs/ 2605.01640. Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling, January 2024. URL http://arxiv.org/abs/2401.16380. arXiv:2401.16380 [cs]. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling Data-Constrained Language Models, June 2025. URL http://arxiv.org/abs/2305.16264. arXiv:2305.16264 [cs]. Joel Niklaus, Guilherme Penedo, Hynek Kydlicek, Elie Bakouch, Lewis Tunstall, Ed Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, and Thomas Wolf. The synthetic data playbook: Generating trillions of the finest tokens, 2026. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. Olmo 3, December 2025. URLhttp://arxiv.org/abs/2512.13961. arXiv:2512.13961 [cs]. Denis Paperno, Germ ́ an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern ́ andez. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. 13 Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ v1/P18-2124. URL https://aclanthology.org/P18-2124. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. arXiv preprint arXiv:2202.05924, 2022. URL https://arxiv.org/abs/2202.05924. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 49523–49544. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/villalobos24a.html. Sang Michael Xie, Hieu Pham, Xinyun Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S. Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2305.10429. Tingkai Yan, Haodong Wen, Binghui Li, Kairong Luo, Wenguang Chen, and Kaifeng Lyu. Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression, November 2025. URLhttp://arxiv.org/abs/2511.13421. arXiv:2511.13421 [cs]. Zitong Yang, Aonan Zhang, Hong Liu, Tatsunori Hashimoto, Emmanuel Cand ` es, Chong Wang, and Ruoming Pang. Synthetic bootstrapped pretraining, December 2025. URLhttp://arxiv.org/ abs/2509.15248. arXiv:2509.15248 [cs]. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Yuhao Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024. URL https://arxiv.org/abs/2403.16952. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. 14 A Implementation Details and Ablations for the CD-Scaling Fit This appendix supplies the implementation details, the outlier-trimming sweep, the functional-form ranking, and the per-term ablation ofR ∗ referenced from Sections 4.3 and 4.4. The main text reports the canonicalk = 15row and the chosen exp-satR ∗ (D,N )form; the tables below show every cell of both sweeps so the choices can be audited. A.1 Implementation details Log-space objective. We work in log-loss space using the numerically stable LSE-equivalent reformulation of Eq. (2) (Hoffmann et al., 2022): logL(N,D,D ′ ) = LSE e, a− α logN, b− β log(D + η· D ′ ) ,(6) algebraically identical to Eq. (2) with(E,A,B) = (e e ,e a ,e b )but numerically stable. The fit minimises Huber loss on log-residuals withδ = 0.1, which gives quadratic weight to typical residuals while suppressing the influence of stragglers an order of magnitude larger. Optimiser and initialisation. L-BFGS with strong-Wolfe line search. We initialize from a log- spaced grid over the parameters, find the best in-sample seed, and report the optimum. Across30+ random seeds the optimum is reproducible to four significant figures. Joint fit. The headline pipeline (Section 4.3) fits all eleven parameters(E,A,B,α,β)+ (logK,ρ,σ) rep +(logK,ρ,σ) para jointly on the pooled 1-epoch + repetition + paraphrase data via a single Huber LSE with iterative residual trimming. The initialization grid brackets both signs ofρ para andσ para so the optimizer can land on either sign. Across every grid we tried, the optimum lands in the same negative-(ρ para ,σ para ) basin. Iterative residual trimming. A handful of noisy small-scale points can dominate the fit, so we drop them using an iterative variant of the residual-based trim of Besiroglu et al. (2024): fit, drop the single worst residual, refit, drop the next worst, and so on forksteps. Applied to the untrimmed fit, the standard Hampel outlier rule (?) identifies 14 points as outliers. We round tok = 15for consistency with our sweep grid (Table 2), which trims∼4%of the data. The trimmed points are all small-scale1-epoch and repetition runs at 14M and 30M, the same class of outliers identified by Besiroglu et al. (2024) for Chinchilla itself. A.2 Outlier-trimming sweep Table 2: Outlier-trimming sweep for the joint fit. Iterative residual trimming on the pooled (n = 356) fit residuals. Per-source RMSEs are on kept points. The canonicalk = 15row (bold) is justified in Section A.1. k n kept βlogK rep σ rep logK para σ para 1-ep RMSErep RMSEpara RMSE 03560.29815.92 −0.6331.57 −1.340.0760.0460.027 53510.37111.85 −0.4431.29 −1.320.0620.0420.024 103460.42011.15 −0.4131.08 −1.310.0520.0380.023 153410.43710.96−0.4131.02−1.330.0430.0350.024 203360.45610.82 −0.4231.02 −1.340.0390.0310.024 253310.47210.71 −0.4231.03 −1.350.0350.0290.022 303260.48210.64 −0.4231.03 −1.360.0300.0270.020 A.3 Functional-form ranking for η We compared nine candidate forms forηon the multi-epoch corpus, holding the backbone(B,β) fixed at the Stage-1 anchors and scoring each form by leave-one-out (LOO) RMSE onlogL(Table 3). The three forms with a fullR ∗ (D,N )via Eq. (5) (sat, exp-sat, tanh) dominate the rest at every parameter count. We adopt exp-satR ∗ (D,N )(close 2nd in LOO behind tanh,∆ = 0.001) because it is the canonical data-repetition form (Muennighoff et al., 2025), hasη(0) = 1exactly, and approaches the R ∗ asymptote monotonically from below. 15 Table 3: Functional-form ranking forη. LOO RMSE onlogLover108multi-epoch points (kept after pooled residual trimming), with the backbone(B,β)frozen at the Stage-1 anchors. The bottom three rows use the fullR ∗ (D,N )ansatz from Eq. (5) (logR ∗ = logK + ρ log(D/N ) + σ logN). Constant-R ∗ (single-parameter Muennighoff baseline,R ∗ = K) is rejected by roughly40%relative LOO at the same parameter count, confirming that the(D,N )-dependence ofR ∗ is load-bearing. We adopt exp-sat R ∗ (D,N ) (row in bold) as our functional form. Formshape of η· rn par LOO RMSE constant ηcr10.035 power in rcr· r −γ 20.032 sat in rcr/(1 + br)20.029 exp-decay, R(D/N)η 0 r· e −r/R 30.026 sat×(D/N), b(N)c(D/N) −γ r/(1 + b 0 (N/N 0 ) κ r)40.029 exp-sat, R ∗ (D/N)R ∗ (1− e −r/R ∗ )20.026 sat, R ∗ (D,N)R ∗ r/(R ∗ + r)30.021 exp-sat, R ∗ (D,N) (ours)R ∗ (1− e −r/R ∗ )30.020 tanh, R ∗ (D,N)R ∗ tanh(r/R ∗ )30.019 A.4 Per-term ablation of Eq. (5) Holding the exp-sat shape ofηfixed, we ablate the terms of Eq. (5) to test the necessity of each (Table 4). Each exponent is individually necessary, and Figure 2(a, c) shows why: theD/N- and N -dependence of η are not absorbable into one another. Table 4: Per-term ablation ofR ∗ . Starting from the original parametric form, we remove terms fromlogR ∗ and report the resulting LOO RMSE. Removing either exponent individually degrades the fit, removing both recovers the constant-R ∗ (Muennighoff et al., 2025) baseline. AblationlogR ∗ LOO RMSE originallogK + ρ log(D/N) + σ logN0.020 ,→ removing N σ logK + ρ log(D/N)0.026 ,→ removing (D/N) ρ logK + σ logN0.025 ,→ removing bothlogK0.028 A.5 Cross-scale validation across sizes We now sweep the fit-set cutoffN max ∈ 30, 60, 100, 190M and predict all held-out points at N > N max (Table 6). Held-out RMSE is stable across cutoffs between0.05and0.08, and comparable to the in-sample RMSE at each cutoff. OnceN max ≥ 60M, the recoveredβis within0.03of the full-data value. The CD-scaling law therefore extrapolates robustly across an order of magnitude in model size. Table 5: Cross-scale validation across fit-set sizes. We refit onN ≤ N max and predict all held-out N > N max points. RMSE is reported on logL. N max Fit sizesn fit n held βin-sample RMSEheld-out RMSE 30M14M, 30M1032040.340.0480.079 60M14M, 30M, 60M1731340.350.047 0.049 100M14M, 30M, 60M, 100M1801270.350.046 0.050 190M14M, 30M, 60M, 100M, 190M235720.490.054 0.057 16 B Additional Experiment Details and Results B.1 Cross-scale validation across sizes We now sweep the fit-set cutoffN max ∈ 30, 60, 100, 190M and predict all held-out points at N > N max (Table 6). Held-out RMSE is stable across cutoffs between0.05and0.08, and comparable to the in-sample RMSE at each cutoff. OnceN max ≥ 60M, the recoveredβis within0.03of the full-data value. The CD-scaling law therefore extrapolates robustly across an order of magnitude in model size. Table 6: Cross-scale validation across fit-set sizes. We refit onN ≤ N max and predict all held-out N > N max points. RMSE is reported on logL. N max Fit sizesn fit n held βin-sample RMSEheld-out RMSE 30M14M, 30M1032040.340.0480.079 60M14M, 30M, 60M1731340.350.047 0.049 100M14M, 30M, 60M, 100M1801270.350.046 0.050 190M14M, 30M, 60M, 100M, 190M235720.490.054 0.057 B.2 Loss to Downstream Breakdown In Figure 9, and Figure 10, we report loss-to-downstream for each benchmark. 123 Val BPB 0 10 20 30 40 50 Accuracy (%) LAMBADA 123 Val BPB 30 35 40 45 Acc (norm) (%) HellaSwag 123 Val BPB 24 26 28 30 32 34 Acc (norm) (%) OpenBookQA 123 Val BPB 20.0 22.5 25.0 27.5 30.0 32.5 Accuracy (%) RACE 123 Val BPB 0 10 20 30 40 Contains (%) SQuAD Val BPB vs Accuracy / EM Benchmarks (Acc ↑, BPB ↓) multi-epoch N=14M N=30M N=60M N=100M N=190M N=370M N=600M paraphrase N=14M N=30M N=60M N=100M N=190M N=370M N=600M Figure 9: Per-task breakdown of accuracy-based benchmarks versus validation BPB. Each subplot shows one task against validation BPB, decomposing the accuracy side of Figure 5. Cool hues mark multi-epoch runs and warm hues mark paraphrase runs, with color denoting model sizeN (14M to 600M). On every task, runs of all strategies and scales fall onto a single monotonic curve in validation BPB. 17 123 Val BPB 1.0 1.5 2.0 2.5 3.0 BPB C4 123 Val BPB 1.0 1.5 2.0 2.5 3.0 BPB WikiText-103 123 Val BPB 1 2 3 4 5 BPB GSM8K (LM) 123 Val BPB 1 2 3 4 BPB HumanEval (LM) 123 Val BPB 1 2 3 4 BPB MBPP (LM) 123 Val BPB 1.5 2.0 2.5 3.0 3.5 4.0 BPB NQ Open (LM) 123 Val BPB 1.5 2.0 2.5 3.0 3.5 BPB TriviaQA (LM) 123 Val BPB 1.5 2.0 2.5 3.0 BPB WebQS (LM) 123 Val BPB 1.5 2.0 2.5 3.0 3.5 BPB SQuAD (LM) 123 Val BPB 1.0 1.5 2.0 2.5 3.0 BPB IFEval (LM) Val BPB vs BPB Benchmarks (BPB ↓) multi-epoch N=14M N=30M N=60M N=100M N=190M N=370M N=600M paraphrase N=14M N=30M N=60M N=100M N=190M N=370M N=600M Figure 10: Per-task breakdown of BPB-based benchmarks versus validation BPB. Each subplot shows one task (recast as language modeling and scored in BPB) against validation BPB, decomposing the BPB side of Figure 5. Cool hues mark multi-epoch runs and warm hues mark paraphrase runs, with color denoting model sizeN(14M to 600M). On every task, runs of all strategies and scales fall onto a single monotonic curve in validation BPB. B.3 Optimal Scaling Path for 370M model In Figure 11, we visualize the optimal scaling path for 370M model and show that the scaling law path predicted by CD-scaling laws also follow closely with the empirical optimal path. 10 18 10 19 10 20 FLOPs 1 2 5 10 20 40 80 160 320 Fresh Data D (TPP, D/N) Empirical IsoLoss Contours 2.55 2.60 2.65 2.70 2.75 2.80 2.85 2.90 2.95 3.00 3.05 3.25 3.45 3.65 3.85 4.05 4.25 10 18 10 19 10 20 FLOPs Predicted IsoLoss Contours 2.55 2.60 2.65 2.70 2.75 2.80 2.85 2.90 2.95 3.00 3.05 3.25 3.45 3.65 3.85 4.05 4.25 4.45 3.0 3.5 4.0 4.5 5.0 5.5 6.0 Validation Loss Model-optimal L N Chinchilla-optimal L chin Single-epoch scaling path Empirical optimal (C,D) scaling path Predicted optimal (C,D) scaling path Figure 11: Optimal compute-data allocation on (C,D) loss surfaces forN = 370M. Replicating results for Figure 6. B.4 Training Run Details Hyperparameter Sweep.We visualize the hyperparamter sweep grid and validation loss for 30M, 370M models. Lower epoch requires smaller weight-decay and as we increase data repetition, higher weight decay is necessary. For larger epochs, 30M model requires WD as high as 1.6, which is consistent with observations made in (Kim et al., 2025). Other Hyperparameters. While we sweep learning rate and weight decay for our training runs, we list the rest of hyperparameter in Table 7. 18 0.1 0.2 0.4 0.8 1.6 WD 11.0710.679.548.20 11.0710.679.548.20 11.0710.679.558.20 11.0710.679.558.21 11.0810.689.578.24 TPP = 1 Epochs = 1 10.318.917.647.56 10.318.917.647.56 10.318.917.647.58 10.318.927.667.57 10.328.947.697.65 Epochs = 4 8.266.856.076.00 8.266.866.065.95 8.276.876.075.97 8.286.896.106.01 8.316.936.176.05 Epochs = 16 6.355.825.795.93 6.355.835.765.89 6.375.835.685.67 6.385.845.615.43 6.415.875.585.34 Epochs = 64 1e-043e-041e-033e-03 LR 0.1 0.2 0.4 0.8 1.6 WD 10.8210.018.408.01 10.8210.018.408.01 10.8210.018.418.01 10.8210.018.428.02 10.8310.028.458.03 TPP = 2 1e-043e-041e-033e-03 LR 9.437.816.806.70 9.437.826.816.71 9.437.826.846.66 9.447.846.876.69 9.457.876.886.75 1e-043e-041e-033e-03 LR 7.156.125.465.30 7.156.125.455.32 7.166.135.475.32 7.186.155.505.36 7.226.205.595.53 1e-043e-041e-033e-03 LR 5.805.295.075.12 5.815.305.04 5.294.934.83 5.845.294.784.62 5.885.375.054.98 30M Validation loss across (WD, LR) per (TPP, Epochs). Red box marks the optimum per cell. 0.1 0.2 0.4 0.8 1.6 WD 6.596.186.066.49 6.596.206.036.34 6.606.186.116.26 6.606.176.206.38 6.606.206.086.34 TPP = 1 Epochs = 1 5.314.434.193.97 5.314.444.023.96 5.314.454.01 5.324.444.054.05 5.334.404.284.41 Epochs = 4 4.023.543.46 4.023.523.433.35 4.023.503.353.26 4.033.493.323.38 Epochs = 16 3.755.576.55 3.735.355.955.35 3.704.975.174.36 3.674.494.403.70 3.704.153.893.49 Epochs = 64 1e-043e-041e-033e-03 LR 0.1 0.2 0.4 0.8 1.6 WD 5.955.315.155.43 5.955.315.215.39 5.955.355.225.38 5.965.365.285.39 5.965.335.305.45 TPP = 2 1e-043e-041e-033e-03 LR 4.583.743.383.41 4.583.743.383.38 4.593.733.383.34 4.593.743.353.46 4.613.753.513.63 1e-043e-041e-033e-03 LR 3.593.263.213.16 3.593.253.173.11 3.593.233.103.03 3.583.203.043.01 3.593.203.083.22 1e-043e-041e-033e-03 LR 3.47 3.323.493.09 3.293.323.123.00 3.253.163.013.05 3.243.103.08 370M Validation loss across (WD, LR) per (TPP, Epochs). Red box marks the optimum per cell. Figure 12: Hyperparameter grid example for 30M and 370M model. More data repetition requires much higher weight decay especially for small models. Table 7: Pretraining hyperparameters (OLMo2-1B, 50B tokens). CategoryHyperparameterValue Data Global batch size (sequences)512 Gradient accumulation steps64 Optimization Learning ratetuned OptimizerAdamW (β 1 ,β 2 )(0.9, 0.95) Weight decaytuned Gradient clip1.0 LR Schedule Schedulecosine with warmup Warmup tokens1B Min LR ratio (α f )0.1 Unitstokens Regularization PrecisionBF16 (AMP) Softmax auxiliary loss✓ Auxiliary loss multiplier1× 10 −5 19