Paper deep dive
Scaling Domain Data Repetition in LLM Pretraining
Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/17/2026, 4:57:20 AM
Summary
This paper investigates the optimal repetition count of high-quality domain data during Large Language Model (LLM) pretraining under a fixed tokens-per-parameter (TPP) ratio. The authors find that, contrary to previous studies assuming fixed data budgets, the optimal repetition count mildly increases with model size when TPP is fixed. Crucially, the optimal repetition count is strongly negatively correlated with the domain's final validation loss (domains with lower loss tolerate more repetition) and is largely insensitive to the amount of unique domain data. These findings suggest that repetition counts tuned on smaller proxy models can effectively estimate optimal settings for larger models.
Entities (9)
Relation Signals (5)
Optimal Repetition Count → correlateswith → Validation Loss
confidence 98% · optimal repetition count is strongly negatively correlated with the final validation loss of a domain
Data Repetition → causes → Overfitting
confidence 96% · excessive repetition may lead to overfitting
Optimal Repetition Count → increaseswith → Model Size
confidence 95% · at a fixed TPP, the optimal repetition count mildly increases with model size
Optimal Repetition Count → insensitiveto → Amount of Unique Data
confidence 94% · the amount of unique domain data is only weakly related to the optimal repetition count
Proxy Model → enablesestimationof → Optimal Repetition Count
confidence 93% · repetition counts tuned on smaller proxy models with the same TPP can provide a practical estimate for larger models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.
Tags
Links
- Source: https://arxiv.org/abs/2608.14071v1
- Canonical: https://arxiv.org/abs/2608.14071v1
Trouble viewing inline? Open PDF directly →
Full Text
50,691 characters extracted from source content.
Expand or collapse full text
Scaling Domain Data Repetition in LLM Pretraining Jingwei Li 1,‡ , Xinran Gu 1 , Rui Dai 1 , Xintong Hao 2,† , Chengyin Xu 2 , Yan Wu 2 , Shuran Zheng 1 , Jingzhao Zhang 1,∗ 1 Tsinghua University, 2 ByteDance Seed ‡ Work done at ByteDance Seed, † Project Lead, ∗ Corresponding Author Abstract As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (TPP). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixedTPP, the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the sameTPPcan provide a practical estimate for larger models. Date: August 17, 2026 Correspondence: Jingzhao Zhang at jingzhaoz@mail.tsinghua.edu.cn 1 Introduction Large language models (LLMs) are often pretrained on mixtures of diverse domains, including general web text, code, mathematics, scientific corpora, and multilingual content [7,8,33,34]. As model size increases, compute-optimal training also requires an increasing number of training tokens [13,16]. This creates an asymmetric data-scaling problem: broad general web data [20,29,30] can often be scaled relatively easily, whereas high-quality domain-specific data [11,21,28] is much harder to scale at the same rate. However, if the amount of such data remains fixed, its mixing ratio decreases as the total training budget grows. Recent work shows that under data mixing, knowledge-dense domains may only be learned once their mixing ratio exceeds a critical threshold; below this threshold, additional training can still fail to acquire the target knowledge [10]. Therefore, maintaining a sufficient fraction of high-quality domain data is important when scaling data. Prior work has extensively studied data repetition, where examples from a finite dataset are reused multiple times during training [26,27,31,35]. For high-quality domain data, repetition offers a direct way to counteract data dilution, but may also increase the risk of overfitting [2,12,18,37]. We therefore need to study the trade-off between the domain-learning gains from repeated high-quality data and the risk of overfitting, and to identify the optimal number of repetitions for different domains. 1 arXiv:2608.14071v1 [cs.AI] 14 Aug 2026 Figure 1 Data repetition under two model-scaling setups. Left: When the training-data size is fixed independently of model size, the optimal repetition count decreases with model size. Right: When the tokens-per-parameter ratio TPPis fixed, the optimal repetition count increases with model size. The fixed-TPPsetting better reflects practical LLM scaling and is the focus of our study. A first line of work studies data repetition in the single-dataset setting [26,27,37]. More recent work extends this question to mixtures of general and domain-specific data, showing that general data can regularize repeated domain data and developing scaling laws that account for repetition under data mixing [4,23,31]. Across these settings, the conclusion is that larger models are more susceptible to overfitting from repeated data, suggesting that repetition should be minimized when training large LLMs. We confirm this with our experiments shown on the left in Figure 1. However, we also note that existing cross-scale comparisons typically hold the training-data size fixed rather than preserving the tokens-per-parameter ratio [4,27,37],TPP : =D/N, whereDis the total number of training tokens andNis the model size. This distinction leads to different conclusions about repetition. As illustrated in Figure 1, whenDis fixed, increasing the model size exposes a larger model to the same repeated dataset, causing overfitting to occur earlier and the optimal repetition count to approach one. In contrast, when the training-token budget grows with model size using a fixedTPP[13,14], the optimal repetition count surprisingly increases with model size. To study repetition under this scaling regime, we consider four high-quality domains: code, math, Wikipedia and medical data. For each domain, we vary its amount of unique data and repetition count while keeping the total training-token budget fixed for each model size. This setup allows us to isolate how data repetition interacts with domain properties, unique-data size, and model scale. Our main findings are as follows. • The optimal repetition count is strongly negatively correlated with the final validation loss of a domain. Domains with lower validation loss can generally tolerate and benefit from more repetitions, whereas domains with higher validation loss overfit earlier. • At a fixed tokens-per-parameter ratioTPP, the optimal repetition count increases with model size. This trend is opposite to that observed when the data budget is fixed across model sizes. It also supports conservative transfer from smaller proxy models: a repetition count that does not cause overfitting in a smaller model is unlikely to cause overfitting in a larger model trained at the same TPP. •The optimal repetition count is largely insensitive to the amount of unique data. Across the tested fractions of unique high-quality tokens, the repetition count that minimizes validation loss remains nearly unchanged. Therefore, when estimating the optimal repetition count on a smaller proxy model, it is not necessary to use a specific unique high-quality token fraction; any representative fraction within the tested range can be used. Together, these results provide a practical approach for configuring the optimal repetition counts for high- 2 quality domains. Repetition counts can first be swept on a smaller proxy model with the sameTPPfor an arbitrary fraction of unique high-quality tokens, and the result can be used to select relatively safe repetition counts for the target model. 2 Related Work Data Repetition in Single-Dataset Training. Early studies mainly examine multi-epoch training on a single dataset. One line of work extends the Chinchilla scaling law [13] and develops scaling laws for repeated data. [27] model the validation loss under repetition using exponential decay and show that a few epochs of repetition can perform similarly to fresh data, while heavier reuse yields diminishing returns. [26] further extend this formulation by modeling the subsequent increase in validation loss with an additive overfitting penalty, achieving more accurate predictions. Another line of work studies the factors that determine repetition- induced degradation. [37] identify dataset size, model size, and training objective as key factors, while [3] show that repetition can improve generalization on some synthetic mathematical tasks. [38] theoretically prove that larger datasets can generally support more reuse. Beyond exact duplication, [17] find that semantic duplication can also cause overfitting, and [35] show that sparse models benefit from more repeated epochs than dense models. However, these studies mainly repeat an entire dataset, whereas practical LLM pretraining often repeats only selected domains within data mixtures. Data Repetition in Pretraining Mixtures. More recent work studies repetition within pretraining mixtures, where a limited subset is reused while the remaining data remain unique. [12] study a setting in which 10% of the training tokens are drawn from a repeatedly reused subset and the remaining 90% are unique. They find that repetition can cause non-monotonic degradation, with intermediate repetition levels being particularly harmful. Extending this analysis, [4] jointly vary the subset size and repetition count, showing that a moderately sized subset repeated a moderate number of times can be more harmful than either a larger subset repeated fewer times or a smaller subset repeated more times. [31] study mixture pretraining with limited target-domain data and abundant generic data. They show that generic data mitigates overfitting from repeated target-domain data and that the optimal repetition count depends on the amount of target data, compute budget, and model scale. However, existing cross-scale studies do not preserve the tokens-per-parameter ratio. We instead study repetition under a fixedTPP=D/N, such that the training-token budget grows proportionally with model size. Data Mixture Optimization for LLMs. Data mixture optimization studies how to allocate a fixed training budget across domains by modeling their relative value and interactions. Existing methods estimate domain weights using proxy training, predictive models, or scaling laws, and transfer the resulting mixture to larger models [6,9,15,25,32,36,39]. These methods generally assume sufficient unique data in each domain and therefore do not model repetition. Recent work [23,31] relaxes this assumption by deriving repetition-aware mixture scaling laws that jointly characterize domain allocation and data reuse. 3 Setup We study data repetition under different model sizes. LetN=N 1 ,...,N K denote the model sizes. For a model of size N ∈N, we set its total training-token budget to D N = TPP· N, whereTPPdenotes the number of training tokens per parameter and is set to a constant greater than 100 in our experiments. Therefore, all training runs with the same model size use exactly the same number of training tokens, while larger models receive proportionally larger token budgets. We focus on different high-quality domains. Let D HQ =Code, Math, Wiki, Medical denote the set of high-quality domains. For each high-quality domaind ∈ D HQ , we sweep over different combinations of unique-data fractions and repetition counts. 3 Figure 2 Overview of our experimental framework. We train models of different sizes at a fixed tokens-per- parameter ratio and vary the amount and repetition count of high-quality domain data in the training mixture. We evaluate both in-domain validation loss and out-of-domain performance. Next, we set the number of unique high-quality tokens U N,α as U N,α = αD N , α∈ 1 40 , 1 20 , 1 10 , and the repetition count ase∈1,2,3,4,5,6,7.In other words, for a configuration (d,N,α,e), we first select a fixed subset from domaindcontainingU N,α unique tokens. This subset is then repeatedetimes. Hence, the total number of high-quality tokens during training is H N,α,e = eU N,α = eαD N . The remaining token budget is filled with non-repeated web data, whose token count is W N,α,e = D N − H N,α,e = (1− eα)D N . All configurations satisfyeα ≤1, and thusW N,α,e ≥0. Importantly,αcontrols the amount of unique high-quality data, whereasecontrols how often this fixed subset is revisited. Their producteαdetermines the final proportion of high-quality token presentations in the training stream. The total budgetD N remains unchanged across recipes for a fixedN; only its composition between the target high-quality domain and the web corpus changes. We do not mix multiple high-quality domains in the same run. Letθ d,N,α,e denote the model obtained from configuration (d,N,α,e). The complete experiment grid for each domain G d and the full set of experiments G are G d =N ×A×E, G = [ d∈D HQ (d×G d ). We evaluate each trained model using both in-distribution and out-of-distribution validation losses. For a validation corpus V, we define the token-averaged negative log-likelihood as L(θ;V) =− 1 M V X x∈V |x| X t=1 logp θ (x t | x <t ), 4 Figure 3 Factors affecting the optimal repetition count. Left: different high-quality domains exhibit substantially different optimal repetition counts. Middle: for the Math domain, the optimal repetition count increases with model size under a fixed TPP. Right: for the Math domain, varyingαmainly shifts the validation loss but has little effect on the optimal repetition count. Overall, the domain has the strongest effect on the optimal repetition count, followed by model size, while α has little effect. wherex= (x 1 ,...,x |x| ) is a token sequence andM V = P x∈V |x|is the total number of validation tokens. For a run targeting domain d, the in-distribution metric is L (d) IID (N,α,e) =L θ d,N,α,e ;V IID d , where V IID d is the held-out validation set from the same high-quality domain. We additionally report L (d) OOD (N,α,e) =L θ d,N,α,e ;V OOD , whereV OOD is a pretraining validation set used to measure performance outside the repeated target domain. Our setup is illustrated in Figure 2. 4 Analysis of the Optimal Repetition Count In this section, we summarize the empirical factors associated with the optimal repetition count and provide a theoretical explanation. Empirically, the optimal repetition count depends mildly on model size, is largely insensitive to the fraction of unique high-quality data, and is strongly negatively correlated with the minimum validation loss. We then use a theoretical model to explain these observations through the trade-off between knowledge acquisition and noise fitting. 4.1 Optimal Repetition Counts for High-Quality Domains Recall that for each model sizeN, we train with a token budgetD N =TPP· Nand record the validation loss at the end of training. We vary the fraction of unique high-quality data, denoted byα, and its repetition count, denoted bye, while keeping the total number of training tokens fixed. We use these experiments to study how the optimal repetition count depends on α, model size, and data domain. Figure 3 compares the effects of data domain, model size, and the fraction of unique high-quality dataαon the optimal repetition count. The left panel shows a strong domain dependence. Math reaches its minimum at around 5 repetitions, while Wiki, Code, and Medical reach their minima earlier, indicating that the optimal repetition count varies substantially across domains. The middle panel shows the effect of model size on the Math domain at a fixedα. As model size increases, the optimal repetition count shifts toward larger values, suggesting a moderate dependence on model scale. The right panel shows the effect ofαon the Math domain at a fixed model size. Although increasingαconsistently reduces the validation loss, the location of the minimum remains nearly unchanged, indicating thatαhas little effect on the optimal repetition count. Overall, the optimal repetition count mainly depends on the data domain and model size, but is largely insensitive toα. The full results across all domains, model sizes, and values ofαare provided in Appendix B. To quantify the above observations, we estimate the optimal repetition count for each combination of domain d, model sizeN, and the fraction of unique high-quality dataα, and then quantitatively characterize its 5 Figure 4 Factors associated with the estimated optimal repetition count. For each combination of high-quality domain, model size, and unique data fraction, we fit a quadratic function to the final validation loss as a function of repetition count and use the minimum of the fitted curve as the estimated optimum. From left to right, we plot the estimated optimal repetition count against the minimum validation loss, model size, and unique high-quality data fraction. The corresponding Pearson correlations are−0.944, 0.400, and 0.018, respectively. Colors denote high-quality domains, marker shapes denote model sizes, and marker sizes denote unique data fractions where applicable. dependence on these factors. Specifically, we fit a quadratic function to the final validation loss as a function of repetition count: b L d,N,α (e) = a d,N,α e 2 + b d,N,α e + c d,N,α . We use the minimum of the fitted curve as the estimated optimal repetition count, be ∗ d,N,α =− b d,N,α 2a d,N,α . This continuous estimate reduces the effect of the discrete repetition grid, allowing clearer analysis of the correlations between the optimal repetition count and different factors. Since different domains exhibit substantially different validation-loss levels, we further characterize the domain effect through validation loss. Specifically, instead of treating the domain only as a categorical variable, we use the minimum validation loss achieved within each domain setting as a continuous proxy for its domain-specific characteristics, and quantitatively examine how it relates to the optimal repetition count. Figure 4 compares the estimated optimal repetition count with the minimum validation loss, model size, and unique high-quality data fraction. The optimal repetition count has a strong negative correlation with the minimum validation loss, with a Pearson correlation of−0.944. Across the domains considered in our experiments, settings with higher validation loss tend to prefer fewer repetitions, whereas settings with lower validation loss tend to have larger optimal repetition counts. Model size has a weaker positive correlation of 0.400, consistent with the observation that larger models tend to prefer slightly more repetitions under a fixed tokens-per-parameter ratio. By comparison, the correlation with the unique high-quality data fraction is only 0.018, indicating that the optimal repetition count is nearly insensitive to the amount of unique data over the range considered. Overall, the optimal repetition count is mainly determined by the high-quality domain loss, with a smaller dependence on model size and little observable dependence on the unique high-quality data fraction. These findings suggest a practical procedure for selecting the repetition count for large-scale training: we first train a smaller proxy model with the same tokens-per-parameter ratio as the target model, using any suitable value ofα. For each high-quality domain, the validation loss provides a useful indication of the range of repetition counts worth considering. This value can then serve as a conservative estimate for the target model, since our experiments show that a repetition count that does not cause overfitting on the proxy model also remains safe for a larger model under the same tokens-per-parameter ratio. 6 4.2 Theoretical Analysis In this section, we analyze the experimental observations above from a theoretical perspective. To capture the experimental setup, we consider a one-hot linear regression problem inR ∞ . Lete k k≥1 be the standard basis, where each coordinatekrepresents a knowledge unit, and its frequency follows a power law. Specifically, following [38], let the input distribution beP(x =e k ) =p k ,wherep k =c α k −α ,c α =ζ(α) −1 , and α >1. Further, letθ ∗ be the target linear predictor, whereθ ∗ specifies the response associated with knowledge unit k. We make the following assumption: Assumption 4.1 (Source Condition). We assume thatθ ∗ satisfies a prior thatE[(θ ∗ k ) 2 p k ] =k −β ,β ∈(1,+∞). Source condition assumptions are widely used in theoretical work such as [19,22]. The total weighted signal energy is finite because P ∞ k=1 E[p k (θ ∗ k ) 2 ] = P ∞ k=1 k −β <∞. Next, we define the learning setup. Given a training-token budgetDand model sizeN, let S N :R ∞ → R N be the projection onto the firstNcoordinates. Due to the limited model capacity, we can only observe S N x instead of observing the intact x [5,19]. Therefore, we observe the datasetD=(S N x i ,y i ) D i=1 , with the following data distribution and loss, x i iid ∼ p, y i =⟨x i ,θ ∗ ⟩ + ε i , ε i iid ∼ N (0,σ 2 ), σ 2 > 0, L N (θ) = 1 2D D X i=1 (⟨S N x i ,θ⟩− y i ) 2 , θ ∈ R N . To fit the linear regression model, we initialize at θ 0 = 0 and run full-batch gradient descent: θ r+1 = θ r − η∇L N (θ r ), r = 0, 1, 2,...,0 < η < 1. To evaluate the performance, we define population risk and the total expected risk. Forθ ∈ R N , define the population excess risk R N (θ) := 1 2 E x h (⟨S N x,θ⟩−⟨x,θ ∗ ⟩) 2 i . To investigate the expected population risk with respect tor, we writeR D,N (r;β,σ) :=E θ ∗ ,D,ε [R N (θ r )],and define the earliest optimal integer stopping time by r ∗ (D,N ;β,σ) := min arg min r∈Z ≥0 R D,N (r;β,σ). When β and σ are fixed, we simply write r ∗ (D,N ). Condition on the number of observations of each knowledge unit gives the exact decomposition: 2R D,N (r;β,σ) = X k>N k −β | z unrepresented knowledge + N X k=1 k −β E m∼B(D,p k ) (1− ηm/D) 2r |z knowledge-acquisition error + σ 2 N X k=1 p k E m∼B(D,p k ) 1 m>0 · (1− (1− ηm/D) r ) 2 m |z noise-fitting error , whereB(D,p k ) denotes the Binomial distribution. Repeated optimization decreases the knowledge-acquisition error but increases the noise-fitting error, and the optimal repetition count is the point at which the latter marginal effect begins to dominate. Theorem 4.2 (Noise decay). For fixed token size D and model size N, let 0 < σ 2 1 ≤ σ 2 2 . Then r ∗ (D,N ;β,σ 2 )≤ r ∗ (D,N ;β,σ 1 ). Moreover, as σ 2 → 0, r ∗ (D,N ;β,σ) = log(1/σ 2 ) − log(1− η/D) + O(1). 7 Findings: A smallerσ 2 reduces the noise-fitting term without changing the knowledge-acquisition term, so the benefit of fitting the signal dominates for more iterations; overfitting still occurs, but its onset is delayed. Theorem 4.2 formalizes this mechanism and explains the negative relationship between validation loss and the optimal repetition count in Figure 4. Theorem 4.3 (Stopping time and model size). Suppose β > α. For fixed D, β, and σ 2 > 0, define N 0 (D,σ 2 ) = D(2− η/D) c α ησ 2 1 β−α . Then r ∗ (D,N + 1)≤ r ∗ (D,N ), N > N 0 (D,σ 2 ). The crossover scale satisfies N 0 (D,σ 2 ) = Θ D σ 2 1/(β−α) ! . Findings: Under a fixed token budget, increasingNextends the model toward rarer and weaker knowledge units without providing additional observations, causing the noise-fitting effect to dominate earlier once the model passes the explicit crossover scale in Theorem 4.3, as illustrated in the left panel of Figure 1. Theorem 4.4 (Stopping time under linear data–model scaling). Supposeβ > α. For fixedσ 2 >0 andC 0 >0, suppose that D N −→ C 0 . Then, as D,N →∞, r ∗ (D,N ) = Θ D α/β . Findings: Under fixed tokens-per-parameter scaling, however, the data budget grows together with model size, allowing the reduction in knowledge-acquisition error to dominate for longer; Theorem 4.4 quantifies the resulting increase in the optimal repetition count shown in the right panel of Figure 1. All proofs are deferred to Appendix D. 5 Additional Analyses In this section, we present additional analyses of data repetition. Section 5.1 compares repeated and unique high-quality tokens at a fixed total domain fraction and shows that the effect of repetition is strongly domain dependent. Section 5.2 examines OOD pretraining performance and finds that replacing unique high-quality tokens with repeated tokens has only a limited effect when the total high-quality and web-data fractions are fixed. Finally, Section 5.3 examines whether the optimal repetition count depends on the learning-rate schedule. 5.1 Unique versus Repeated Tokens at a Fixed Domain Fraction In the previous section, we fix the amount of unique high-quality data and vary its repetition count. We now consider the complementary setup studied by [27], where the total number of training tokens from a dataset is fixed and repeated tokens are directly compared with the same number of unique data. Their results suggest that, for up to roughly four repetitions, repeated data can achieve performance close to that of an equivalent amount of unique data. However, this analysis does not distinguish how this trade-off varies across data domains. We therefore study the same question separately for different high-quality domains. Letαdenote the fraction of unique high-quality data andeits repetition count. The total fraction of high-quality tokens consumed during training is ρ = αe. 8 Figure 5 Final validation loss at fixed total high-quality data fractions. Top: Math. Bottom: Wiki. We fix the total high-quality-data fraction and vary the repetition count, such that more repetition corresponds to fewer unique high-quality tokens. More unique data consistently yields lower validation loss in both domains. However, Math is relatively robust to repetition, whereas Wiki degrades sharply when the repetition count increases beyond 2. For a fixedρ, increasingerequires decreasing the amount of unique data according toα=ρ/e.This setup allows us to directly ask whether repeating a smaller unique datasetetimes can match the performance of using e times as much unique data once, under the same total high-quality token budget. Figure 5 shows the results for Math and Wiki. We find that the result depends strongly on the domain. For Math, increasing the repetition count from 1 to 4 causes only a small increase in validation loss. Thus, repeated Math data can largely substitute for additional unique Math data, consistent with the observation of [27]. In contrast, Wiki exhibits a clear degradation as repetition increases. While repeating the data twice incurs only a modest loss increase, heavier repetition leads to substantially worse validation loss, indicating that repeated Wiki tokens cannot effectively replace the corresponding amount of unique data. These results show that the effectiveness of repeated data under a fixed token budget is domain dependent, rather than being characterized by a universal repetition threshold. Additional results in Appendix C show the same contrast across the remaining domains. Both Code and Medical show increasing validation loss as more unique data is replaced by repeated data, although the magnitude of degradation varies across domains. 5.2 Effect of Repetition on OOD Pretraining Performance Next, we study whether repetition within a high-quality domain affects pretraining performance outside that domain. To isolate the effect of repetition, it is important to control the amount of web data used during training. If we instead fix the amount of unique high-quality data and increase its repetition count, the total amount of high-quality data increases accordingly, leaving fewer training tokens for web data. Any change in OOD performance would then confound the effect of repetition with the change in web data. We therefore adopt the fixed-domain-fraction setup from Section 5.1. Specifically, we fix the total fraction of high-quality Math tokens, and hence also the amount of web data, while varying the repetition count by changing the amount of unique high-quality data. This allows us to isolate how replacing unique high-quality tokens with repeated tokens affects performance outside the repeated domain. We evaluate the trained models on two OOD pretraining validation sets, ArXiv and News. Figure 6 reports the results. In contrast to the clear domain-dependent degradation observed on in-domain validation sets, the OOD validation loss remains largely stable as the repetition count changes. Across model sizes and total high-quality data fractions, the differences between configurations using 1, 2, or 4 repetitions are generally small. 9 Figure 6 OOD validation loss at fixed total high-quality data fractions. Top: ArXiv. Bottom: News. Each line fixes the total fraction of high-quality tokens, while the annotations indicate the corresponding repetition counts. Unlike the in-domain validation loss, the OOD loss changes only slightly as unique tokens are replaced with repeated tokens and shows no consistent monotonic relationship with the repetition count. These results suggest that, when the total fractions of high-quality and web tokens are fixed, replacing unique high-quality tokens with repeated tokens mainly affects performance within the repeated domain, while having limited effect on OOD pretraining performance. 5.3 Effect of Various Learning-Rate Schedules Finally, we examine whether the optimal repetition count depends on the learning-rate schedule. We compare a warmup-stable-decay (WSD) schedule with the decay phase starting at 0%, 25%, 50%, or 75% of training, as well as a constant learning rate. For each schedule, we vary the fraction of unique high-quality data and the repetition count while keeping all other training settings unchanged. As shown in Figure 7, the learning-rate schedule has a clear effect on the optimal repetition count. Earlier decay causes the validation loss to increase after fewer repetitions, whereas delaying the decay shifts this increase to a larger repetition count. With a constant learning rate, the model tolerates the largest number of repetitions before degradation occurs. A possible explanation is that repeated samples seen during the low-learning-rate stage can be fitted more closely, increasing the tendency to memorize sample-specific patterns. Delaying the decay keeps training at a relatively high learning rate for longer, which may reduce this effect through stronger optimization noise and implicit regularization. Overall, although excessive repetition eventually degrades performance across all schedules, the learning-rate schedule determines how much repetition the model can tolerate before this degradation begins. 6 Conclusion and Future Directions In this work, we study domain data repetition across model scales while keeping the tokens-per-parameter ratio fixed. We find that the optimal repetition count is strongly negatively correlated with the minimum validation loss of the repeated domain: domains with lower validation loss generally support more repetitions. At a fixed tokens-per-parameter ratio, the optimal repetition count increases mildly with model size, while remaining nearly insensitive to the fraction of unique high-quality tokens over the range considered. These observations suggest that repetition counts selected on smaller proxy models with the same TPP can provide conservative estimates for larger models. Our theoretical analysis further provides an explanation for the observed dependence on validation loss and model scale. 10 Figure 7 Effect of the learning-rate schedule on the optimal repetition count. From left to right: WSD with decay starting at 0%, 25%, 50%, and 75% of training, followed by a constant learning rate. Earlier decay leads to degradation after fewer repetitions, while delaying or removing the decay allows more repetitions. Our current setup repeats only one high-quality dataset in each training run. Future work should consider mixtures in which multiple domains are repeated simultaneously and study their interactions. Another direction is to develop a quantitative scaling rule that predicts the optimal repetition count of a large model from small-model results, enabling more accurate and efficient data-recipe selection. 11 References [1]Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXivpreprint arXiv:2401.02954, 2024. [2]Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. InTheEleventhInternationalConferenceonLearning Representations, 2022. [3]François Charton and Julia Kempe. Emergent properties with repeated examples.arXivpreprintarXiv:2410.07041, 2024. [4]Jessica Chudnovsky, Joshua Kazdan, Noam Levi, Rylan Schaeffer, Yegor Denisov-Blanch, Bo He, Mehmet Donmez, Sanmi Koyejo, and David Donoho. Internal data repetition destroys language models.arXivpreprint arXiv:2606.24998, 2026. [5]Rui Dai and Shuran Zheng. Explaining data mixing scaling laws. InForty-thirdInternationalConferenceon MachineLearning, 2026. URL https://openreview.net/forum?id=joReaAnwnH. [6] Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXivpreprintarXiv:2504.13161, 2025. [7]Sumanth Doddapaneni, Gowtham Ramesh, Mitesh Khapra, Anoop Kunchukuttan, and Pratyush Kumar. A primer on pretrained multilingual language models.ACMComputingSurveys, 57(9):1–39, 2025. [8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXive-prints, pages arXiv–2407, 2024. [9] Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. In InternationalConferenceonMachineLearning, pages 12895–12915. PMLR, 2024. [10] Xinran Gu, Kaifeng Lyu, Jiazheng Li, and Jingzhao Zhang. Data mixing can induce phase transitions in knowledge acquisition.arXivpreprintarXiv:2505.18091, 2025. [11] Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. InProceedingsoftheTwelfthLanguageResourcesandEvaluationConference, pages 2440–2452, 2020. [12]Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data.arXivpreprintarXiv:2205.10487, 2022. [13]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. InProceedingsofthe36thInternationalConferenceonNeuralInformationProcessingSystems, pages 30016–30030, 2022. [14] Dayal Singh Kalra and Maissam Barkeshli. Quantifying hyperparameter transfer and the importance of embedding layer learning rate.arXivpreprintarXiv:2605.21486, 2026. doi: 10.48550/arXiv.2605.21486. [15]Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Scale-aware data mixing for pre-training llms.arXivpreprintarXiv:2407.20177, 2024. [16]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXivpreprintarXiv:2001.08361, 2020. [17]Joshua Kazdan, Noam Levi, Rylan Schaeffer, Jessica Chudnovsky, Abhay Puri, Bo He, Mehmet Donmez, Sanmi Koyejo, and David Donoho. Scale dependent data duplication.arXivpreprintarXiv:2603.06603, 2026. [18]Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. InProceedingsofthe60thAnnual MeetingoftheAssociationforComputationalLinguistics(Volume1:LongPapers), pages 8424–8445, 2022. 12 [19]Binghui Li, Fengling Chen, Zixun Huang, Lean Wang, and Lei Wu. Functional scaling laws in kernel regression: Loss dynamics and learning rate schedules. InTheThirty-ninthAnnualConferenceonNeuralInformation ProcessingSystems, 2025. URL https://openreview.net/forum?id=dpllevHMbc. [20]Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models.AdvancesinNeuralInformationProcessingSystems, 37:14200–14282, 2024. [21]R Li, LB Allal, Y Zi, N Muennighoff, D Kocetkov, C Mou, M Marone, C Akiki, J Li, J Chim, et al. Starcoder: May the source be with you!Transactionsonmachinelearningresearch, 2023. [22]Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter Bartlett, and Jason D. Lee. Scaling laws in linear regression: Compute, parameters, and data. InTheThirty-eighthAnnualConferenceonNeuralInformationProcessing Systems, 2024. URL https://openreview.net/forum?id=PH7sdEanXP. [23]Fengze Liu, Weidong Zhou, Binbin Liu, Ping Guo, Zijun Wang, Bingni Zhang, Yifan Zhang, Yifeng Yu, Xiaohuan Zhou, and Taifeng Wang. Infolaw: Information scaling laws for large language models with quality-weighted mixture data and repetition.arXivpreprintarXiv:2605.02364, 2026. [24] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training.arXivpreprintarXiv:2502.16982, 2025. [25]Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. InTheThirteenthInternational ConferenceonLearningRepresentations, 2024. [26]Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, and Kilian Q Weinberger. Prescriptive scaling laws for data constrained training.arXivpreprintarXiv:2605.01640, 2026. [27]Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models.AdvancesinNeural InformationProcessingSystems, 36:50358–50376, 2023. [28]Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. InInternationalConferenceonLearningRepresentations, volume 2024, pages 20357–20379, 2024. [29]Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.AdvancesinNeural InformationProcessingSystems, 37:30811–30849, 2024. [30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journalof machinelearningresearch, 21(140):1–67, 2020. [31] Anastasiia Sedova, Skyler Seto, Natalie Schluter, and Pierre Ablin. Scaling laws for mixture pretraining under data constraints.arXivpreprintarXiv:2605.12715, 2026. [32] Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures.arXivpreprintarXiv:2507.09404, 2025. [33] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science.arXivpreprint arXiv:2211.09085, 2022. [34]Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprintarXiv:2312.11805, 2023. [35] Boqian Wu, Qiao Xiao, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Mykola Pechenizkiy, Elena Mocanu, Torsten Hoefler, and Decebal Constantin Mocanu. When data is scarce: Scaling sparse language models with repeated training.arXivpreprintarXiv:2606.01155, 2026. [36] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining.Advancesin NeuralInformationProcessingSystems, 36:69798–69818, 2023. 13 [37]Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling llm under token-crisis.AdvancesinNeuralInformationProcessingSystems, 36:59304–59322, 2023. [38]Tingkai Yan, Haodong Wen, Binghui Li, Kairong Luo, Wenguang Chen, and Kaifeng Lyu. Larger datasets can be repeated more: A theoretical analysis of multi-epoch scaling in linear regression.arXivpreprintarXiv:2511.13421, 2025. [39] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. InTheThirteenthInternationalConferenceon LearningRepresentations, 2025. 14 Appendix A General Implementation Details In this section, we describe the datasets and training hyperparameters used in our experiments. A.1 Details of Datasets Our pretraining corpus consists of general web data and several high-quality domain-specific datasets. The general web data contains content from diverse sources, covering a broad range of topics and writing styles. The high-quality data includes four domains: •Code data. A multilingual programming corpus covering a range of programming languages and software-development scenarios. •Math data. A bilingual mathematical corpus containing educational materials, mathematical problems, and step-by-step solutions across different difficulty levels. • Wiki data. Encyclopedic text covering diverse concepts, entities, and factual knowledge. •Medical data. A collection of medical and health-related content covering biomedical knowledge, clinical topics, and health education. A.2 Details of Training Hyperparameters The total training-token budget was scaled proportionally with model size under a fixed tokens-per-parameter ratio. Following [1], we set the learning rate and global batch size for different model sizes according to power-law functions of the model size. The learning rate was linearly warmed up for the first 200 optimization steps and was then held constant for the remainder of training. We used an attention dropout rate of 0.1. Optimization used the Muon optimizer [24], a variant of AdamW. B Additional Results on Section 4.1 In this section, we present additional results of Section 4.1. Figures 8, 9, 10, and 11 present the complete results for Math, Wiki, Code, and Medical. Among the four domains, Math supports the most repetition, with an optimal repetition count of about 5–6, followed by Code and Wiki, while Medical has the lowest optimal repetition count of about 3–4. For each domain, the optimal repetition count increases with model size. In contrast, for a fixed model size, the optimal repetition count remains nearly unchanged across different values ofα. Overall, all experimental results are consistent with the observations in the main text. Figure 8 Final validation loss across repetition counts for Math. Each panel corresponds to a different model size, and each curve represents a different fraction of unique high-quality data. All configurations are trained for the full token budget associated with the corresponding model size. The validation loss is generally minimized between 5 and 6 repetitions. Increasing the unique data fraction reduces the absolute validation loss but has little effect on the optimal repetition count, which increases only mildly with model size. 15 Figure 9 Final validation loss across repetition counts for Wikipedia. Compared with Math, Wikipedia reaches its minimum validation loss earlier, typically after 3 to 4 repetitions, while showing the same stability across unique data fractions and mild increase with model size. Figure 10 Final validation loss across repetition counts for Code. Code generally reaches its minimum validation loss after 4 to 5 repetitions. The optimal repetition count remains stable across unique data fractions and increases mildly with model size. Figure 11 Final validation loss across repetition counts for Medical. Medical generally reaches its minimum validation loss after 3 to 4 repetitions. As in the other domains, the optimal repetition count is largely insensitive to the unique data fraction and increases only mildly with model size. C Additional Results on Section 5.1 In this section, we present additional results of Section 5.1. Figure 12 shows the results under the same fixed-token-budget setup as in the main text. For each curve, the total fraction of high-quality training tokens is fixed, while increasing the repetition count reduces the amount of unique high-quality data. This directly compares repeated data with an equivalent number of fresh unique tokens. The results further show that the effectiveness of repetition differs across domains. Code and Medical behave similarly to Wikipedia, with validation loss increasing more noticeably as more unique data is replaced by repeated data. These results further support that whether repeated data can match fresh data under a fixed token budget depends strongly on the data domain. 16 Figure 12 Additional results at fixed total high-quality data fractions. Top: Code. Bottom: Medical. Code and Medical behave similarly to Wikipedia and shows a clearer degradation as more unique data is replaced by repeated data. D Proofs In this section, we provide the proofs for the results in Section 4.2. D.1 Proof of Theorem 4.2 Proof of Theorem 4.2. For coordinatek, condition on the countM k =mof samples x i =e k , whereM k ∼ Binomial(D,p k ) with probability π k,m . The exact averaged population risk is 2R D,N (r;β,σ) = X k>N k −β + N X k=1 D X m=0 π k,m k −β a 2r m + p k σ 2 m (1− a r m ) 2 , where a m = 1− ηm/D, and the variance term is defined to be 0 when m = 0. LetF(r) := 2R D,N (r;β,σ 1 ) andG(r) := 2R D,N (r;β,σ 2 ). Increasing the noise variance fromσ 2 1 toσ 2 2 gives G(r) = F (r) + ∆(r), where ∆(r) = (σ 2 2 − σ 2 1 ) N X k=1 D X m=1 π k,m p k m (1− a r m ) 2 . The function ∆(r) is non-decreasing inr, and it is strictly increasing whenσ 2 2 > σ 2 1 . Letr F andr G be the earliest minimizers ofFandG, respectively. Suppose thatr G > r F . By the optimality ofr F and the monotonicity of ∆, G(r G ) = F (r G ) + ∆(r G )≥ F (r F ) + ∆(r F ) = G(r F ). If the inequality is strict, it contradicts the optimality ofr G . If equality holds, it contradictsr G being the earliest minimizer. Therefore, r G ≤ r F . It remains to derive the small-noise rate. Write τ = σ 2 and H τ (r) := 2R D,N (r;β,σ). Its forward difference is H τ (r + 1)− H τ (r) = N X k=1 D X m=1 π k,m (1− a m )a r m · h 2p k τ m − (1 + a m ) k −β + p k τ m a r m i . 17 Let a = a 1 = 1− η/D. The contribution from m = 1 can be written as Aτa r − B τ a 2r , where A = 2(1− a) N X k=1 π k,1 p k > 0, B τ = (1− a)(1 + a) N X k=1 π k,1 (k −β + p k τ ) > 0. IfD ≥2, leta 2 = 1−2η/D < a; ifD= 1, the remainder below is identically zero. To evaluateH τ (r+1)−H τ (r), we split it as follows: H τ (r + 1)− H τ (r) = Aτa r − B τ a 2r + N X k=1 D X m=2 π k,m (1− a m )a r m · h 2p k τ m − (1 + a m ) k −β + p k τ m a r m i | z :=E τ (r) . SinceDandNare fixed, the contributionE τ (r) fromm ≥2 satisfies, uniformly for 0< τ ≤1,|E τ (r)| ≤ C τa r 2 + a 2r 2 for a constant C independent of r and τ. By the optimality of r τ , H τ (r τ + 1)− H τ (r τ )≥ 0, H τ (r τ )− H τ (r τ − 1)≤ 0. Suppose first thatD ≥2, and writeρ=a 2 /a <1. Settingx=a r τ in the first inequality gives B τ − Cρ 2r τ x≤ (A + Cρ r τ )τ. Sincer τ →∞andB τ → B 0 >0, it follows thata r τ ≤ C 1 τ,for all sufficiently smallτ. Similarly, settingy=a r τ −1 in the second inequality gives A− Cρ r τ −1 τ ≤ B τ + Cρ 2(r τ −1) y, and hencea r τ ≥ C 2 τ for a constantC 2 >0 independent ofτ. WhenD= 1, the same two bounds follow directly because E τ (r) = 0. Therefore, a r τ = Θ(τ ). Taking logarithms and recalling that τ = σ 2 yields r ∗ (D,N ;β,σ) = log(1/σ 2 ) − log(1− η/D) + O(1). D.2 Proof of Theorem 4.3 Proof of Theorem 4.3.Let ∆ N (r) := 2R D,N+1 (r)−2R D,N (r) and setj=N+ 1. The exact risk increment is ∆ N (r) = P D m=1 π j,m h j,m (r),whereh j,m (r) =j −β (a 2r m − 1) +c α j −α σ 2 m (1− a r m ) 2 .The forward difference is h j,m (r + 1)− h j,m (r) = (1− a m )a r m · 2c α j −α σ 2 m − (1 + a m ) j −β + c α j −α σ 2 m a r m . The bracketed term is minimized at r = 0. Thus, h j,m (r + 1)− h j,m (r)≥ 0 for all r ≥ 0 provided that c α j −α σ 2 m (1− a m )≥ j −β (1 + a m ). Using 1− a m = ηm/D and 1 + a m ≤ 2− η/D for m≥ 1, a sufficient condition is j β−α ≥ D(2− η/D) c α ησ 2 . When N > N 0 (D,σ 2 ), this condition holds for j = N + 1, implying that ∆ N (r) is non-decreasing in r. LetF(r) := 2R D,N (r) andG(r) := 2R D,N+1 (r) =F(r) + ∆ N (r), with respective earliest minimizersr F and r G . Suppose that r G > r F . By the optimality of r F and the monotonicity of ∆ N , G(r G ) = F (r G ) + ∆ N (r G )≥ F (r F ) + ∆ N (r F ) = G(r F ). 18 As in the previous proof, a strict inequality contradicts the optimality ofr G , while equality contradicts the earliest-minimizer convention. Therefore, r G ≤ r F . The stated crossover order follows directly from N 0 (D,σ 2 ) = 2D− η c α ησ 2 1/(β−α) . D.3 Proof of Theorem 4.4 Proof of Theorem 4.4.For coordinatek, letM k ∼ Binomial(D,p k ) anda m = 1−ηm/D. From the exact risk formula, we have 2R D,N (r + 1)− 2R D,N (r) = − η D N X k=1 k −β E M k (1 + a M k )a 2r M k | z S D (r) + ησ 2 D N X k=1 p k E 2a r M k − (1 + a M k )a 2r M k 1 M k ≥1 | z V D (r) . BothS D (r) andV D (r) are nonnegative. The first term is the marginal signal benefit, whereas the second is the marginal variance penalty. We first consider 1≪ r ≤ D. Byexp − ηrm D(1−η) ≤ a r m ≤ exp − ηrm D , E[z M k ] = (1−p k +p k z) D and E[M k z M k ] = Dp k z(1− p k +p k z) D−1 the dominant coordinates satisfyp k r ≍1, or equivalentlyk ≍ r 1/α . Standard sum– integral comparison then gives, uniformly in this range, S D (r)≍ N X k=1 k −(α+β) exp(−crk −α )≍ r 1−α−β α , and V D (r)≍ σ 2 D N X k=1 k −α exp(−crk −α ) 1− exp(−crk −α ) ≍ σ 2 D r 1−α α . Here and below, the positive constants denoted bycmay differ from line to line. SinceD/N → C 0 andr ≤ D, truncating the sums atNdoes not change these orders. Therefore, S D (r) V D (r) ≍ D σ 2 r −β/α .The same estimates with one-sided constants also cover boundedr. Hence, there exist constants 0< c 0 < C ′ 0 <∞ , independent ofD, such that, for all sufficiently largeD,S D (r)> V D (r),0≤ r ≤ c 0 D α/β ,andS D (r)< V D (r),C ′ 0 D α/β ≤ r ≤ D. It remains to exclude a later decrease whenr ≥ D. Leta 1 = 1− η/D. Restricting the variance sum to coordinates withp k ≤ D −1 and to the eventM k = 1 givesV D (r)≥ cσ 2 a r 1 D 1−2α α . Usinga m ≤ a m 1 and the binomial identities above givesS D (r)≤ Ca 2r 1 D 1−α−β α .Consequently, S D (r) V D (r) ≤ Ca r 1 D 1−β/α σ 2 =o(1),where the last equality follows fromβ > α. Thus, the exact risk is increasing for everyr ≥ DwhenDis sufficiently large. We have shown that the exact risk is decreasing up toc 0 D α/β and increasing afterC ′ 0 D α/β . Therefore, its earliest global minimizer satisfies c 0 D α/β ≤ r ∗ (D,N )≤ C ′ 0 D α/β , which proves r ∗ (D,N ) = Θ D α/β . 19