Paper deep dive

Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

Yicheng Mao, Hongru Du

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 62

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/26/2026, 5:05:17 AM

Summary

This paper reinterprets large language model (LLM) data mixing as a classical mixture experiment, utilizing Response Surface Methodology (RSM) and optimal experimental design. The authors propose using sparse second-order Scheffé response-surface models to decompose domain contributions into additive effects and pairwise interactions, demonstrating that domain value is relational. They further introduce model-robust I-optimal designs to select proxy training mixtures, showing that this approach can recover mixture rankings with approximately 25% fewer proxy runs compared to random sampling, thereby improving statistical efficiency in pretraining data allocation.

Entities (6)

LLM Data Mixing · problem-domain · 99%RegMix · methodframework · 98%Scheffé response-surface models · statistical-model · 97%I-optimal designs · experimental-design-criterion · 96%Pile dataset · dataset · 95%Mixture Experiment · concept · 94%

Relation Signals (5)

LLM Data Mixing → isformulatedas → Mixture Experiment

confidence 95% · We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions...

RegMix → uses → Pile dataset

confidence 95% · In the main experiment analysed here, RegMix uses 17 available domains from the Pile dataset (Gao et al., 2021)...

Scheffé response-surface models → decomposes → Additive and Interaction Effects

confidence 93% · The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects.

I-optimal designs → reduces → Proxy Training Runs

confidence 92% · In a simulation study calibrated to observed proxy-training responses, model-robust I-optimal designs recover the relevant mixture ordering after removing about 25% of the original proxy runs.

Data domains → are → Mixture components

confidence 90% · Under this view, data domains are mixture components, token shares are component proportions...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy-based methods address this problem by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for larger-scale training. We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop this formulation using sparse second-order Scheffé response-surface models and construct model-robust $\mathcal{I}$-optimal designs for proxy data-mixing experiments. Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The Scheffé analysis shows that domain value is strongly relational: several domains that are weak under additive effects become favourable through pairwise interactions, especially through combinations with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects. In a simulation study calibrated to observed proxy-training responses, model-robust $\mathcal{I}$-optimal designs recover the relevant mixture ordering after removing about 25\% of the original proxy runs. These results suggest that LLM data mixing should be treated not only as a prediction problem, but also as an experimental-design problem in which the proxy mixtures themselves can be chosen to improve statistical efficiency.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

58,677 characters extracted from source content.

Expand or collapse full text

Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining Yicheng Mao Affiliation: Department of Mathematics and Statistics, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada Hongru Du Affiliation: Department of Systems and Information Engineering, University of Virginia, Charlottesville, VA, USA Affiliation: Correspondence: hongrudu@virginia.edu Abstract Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy-based methods address this problem by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for larger-scale training. We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop this formulation using sparse second-order Scheffé response-surface models and construct model-robust ℐI-optimal designs for proxy data-mixing experiments. Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The Scheffé analysis shows that domain value is strongly relational: several domains that are weak under additive effects become favourable through pairwise interactions, especially through combinations with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects. In a simulation study calibrated to observed proxy-training responses, model-robust ℐI-optimal designs recover the relevant mixture ordering after removing about 25% of the original proxy runs. These results suggest that LLM data mixing should be treated not only as a prediction problem, but also as an experimental-design problem in which the proxy mixtures themselves can be chosen to improve statistical efficiency. Keywords: mixture experiments, Scheffé polynomials, optimal experimental design, large language models, data mixing, pretrain 1 Introduction The performance of a large language model depends heavily on the composition of its pretraining corpus. Modern pretraining datasets are assembled from multiple data domains such as web text, code, scientific articles, and books, and the proportion allocated to each domain affects both validation loss and downstream task performance (Gao et al., 2021; Longpre et al., 2023). Because the total token budget is often fixed, increasing the share of one domain reduces the share available to the others. The choice of data mixture is therefore a constrained allocation problem, and it has become a central design question in LLM pretraining. At the scale of modern pretraining runs, suboptimal mixture choices translate directly into wasted compute and degraded model capability, making principled mixture selection a problem of both scientific and practical importance. Recent work has begun to treat data mixture weights not as fixed corpus-construction choices, but as quantities that can be estimated, adapted, or optimised during the training pipeline. For example, DoReMi (Xie et al., 2023) uses a small proxy model and group distributionally robust optimisation to learn domain weights that are then used for larger-scale pretraining. More recent work makes the mixture-performance relationship even more explicit by modelling how mixture proportions affect language-model performance. Data Mixing Laws (Ye et al., 2025) fit predictive functions that relate sampled mixtures to language modelling loss, while BiMix (Ge et al., 2025) extends this idea by modelling the joint effect of mixture proportions and data volume. RegMix (Liu et al., 2025) similarly trains small proxy models on sampled mixtures and fits regression models to predict the performance of unseen mixtures at larger scales. Taken together, these studies show that data mixing is increasingly being treated as a measurable allocation problem: mixture proportions are varied, model responses are observed or predicted, and the resulting information is used to guide future mixture selection. The key observation of this paper is that this allocation problem has the geometry of a classical mixture experiment. The predictors in data mixing are not ordinary covariates. They are proportions that must be non-negative and sum to one, so all feasible mixtures lie on a probability simplex. The same geometry appears in classical mixture experiments, where the response depends on the relative proportions of several components rather than on independently adjustable inputs (Cornell, 2002). The classical examples are formulation problems: the taste of a food product depends on the proportions of its ingredients, and the performance of a chemical product depends on the proportions of its constituent compounds (Goos and Hamidouche, 2019; Furlanetto et al., 2011). The same mixture-experiment logic has also been used for more abstract allocation problems, where the “components” are not physical ingredients but shares of a fixed resource. For example, advertising media-mix studies allocate a fixed advertising budget across media channels to quantify channel effects and cross-media synergy (Goos et al., 2019), while mobility-budget studies allocate a fixed travel budget across alternative transport modes (Zijlstra et al., 2019). In all these cases, the effect of one component cannot be interpreted independently of the others because increasing one share necessarily reduces the remaining shares. LLM data mixing follows the same logic: the components are data domains, the mixture proportions are token shares, and the response is validation loss or downstream performance after training. This mixture-experiment interpretation has two methodological consequences. First, it changes how the response surface can be specified. Since the predictors are constrained proportions, the fitted model should respect the simplex geometry rather than treating domain shares as ordinary unconstrained covariates. This motivates the use of Scheffé response-surface models, which are defined directly on the simplex (Scheffé, 1958). In the present context, they provide a way to decompose the fitted loss surface into additive domain contributions and pairwise departures from additivity. This is important because a domain may be weak as an additive contributor but useful when combined with another domain. Flexible machine-learning predictors may capture such interactions implicitly, but they do not usually express them as interpretable quantities. Second, the mixture-experiment interpretation changes how proxy mixtures can be selected. In many proxy-based data-mixing workflows, candidate mixtures are sampled randomly, for example, from Dirichlet distributions. This is convenient and scalable, but it does not ask whether those mixtures are the most informative design points for estimating a response surface. If proxy runs are experimental design points rather than merely random samples, then their locations on the simplex can be chosen to improve prediction over the mixture region. Since the practical goal is to predict the performance of unseen mixtures, ℐI-optimality is a natural criterion because it targets low average prediction variance over the experimental region (Goos and Jones, 2011). Moreover, because the appropriate response-surface order is not known before the proxy experiment is run, a model-robust design can be used to balance performance across additive and interaction-inclusive candidate models (Yu et al., 2008). This paper develops a mixture-experiment framework for LLM data mixing, in which data sources are components, token shares are mixture proportions, proxy-training runs are design points, and model performance is the response surface. Using the publicly available RegMix proxy-training data as an empirical case study, we fit sparse second-order Scheffé models to estimate interpretable additive and interaction effects, evaluate whether these structured response surfaces preserve mixture rankings across model scales, and construct model-robust ℐI-optimal designs to test whether useful mixture orderings can be recovered with fewer proxy runs. Although the empirical analysis is grounded in the RegMix setting, the methodological framework applies to any proxy-based data-mixing workflow in which a fixed training budget is allocated across data sources. The contribution of the paper is threefold. First, it formalises LLM data mixing as a mixture response-surface experiment on the probability simplex. This connects proxy-based data-mixing workflows to the statistical literature on mixture modelling and optimal experimental design. Second, it shows how sparse Scheffé models can expose additive domain effects and pairwise domain interactions that are otherwise hidden inside black-box predictors. Third, it demonstrates that optimal design can be used to improve the efficiency of proxy data-mixing experiments, reducing the proxy-training budget needed to recover reliable mixture rankings and suggesting that how proxy mixtures are chosen deserves as much attention as how the resulting data are modelled. The remainder of the paper is organised as follows. Section 2 formalises proxy-based data mixing as a mixture response-surface experiment. Section 3 introduces the sparse Scheffé response-surface model and the model-robust ℐI-optimal design criterion. Section 4 reports the Scheffé re-analysis of the RegMix data and its cross-scale rank-preservation performance. Section 5 evaluates model-robust ℐI-optimal designs for proxy data-mixing experiments. Section 6 concludes with limitations and directions for future work. 2 Proxy Data Mixing as a Mixture Experiment Consider a fixed pretraining token budget distributed across K data domains. Let =(x1,…,xK)⊤x=(x_1,…,x_K) denote the vector of domain proportions. Each candidate mixture satisfies xi≥0,∑i=1Kxi=1.x_i≥ 0, _i=1^Kx_i=1. Thus, the space of feasible data mixtures is the (K−1)(K-1)-dimensional probability simplex. The practical goal is to identify a mixture x that optimises a specified training objective, such as validation loss or downstream task performance, subject to this fixed-budget simplex constraint. A proxy-based data-mixing study can be viewed as an experiment on this simplex. A set of candidate mixtures is selected, a small model is trained or evaluated at each mixture, and the resulting validation loss or downstream score is recorded as the response. A predictive model is then fitted from mixture proportions to model performance and used to guide the selection of future mixtures for larger-scale training. Under this view, data sources are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and the fitted predictor is a response surface over the simplex. The RegMix proxy-training data (Liu et al., 2025) provide a useful empirical instance of this general structure. RegMix samples candidate mixtures from Dirichlet distributions, trains a small proxy LLM on each sampled mixture, records the target value to be optimised, and fits a regression model using mixture proportions as features and target values as labels. The fitted model is then used to predict the performance of a large simulated pool of candidate mixtures, from which the best predicted mixture is selected for larger-scale training. In the main experiment analysed here, RegMix uses 17 available domains from the Pile dataset (Gao et al., 2021), trains 512 proxy models with 1M non-embedding parameters on 1B tokens each, and evaluates the fitted regression model on held-out mixtures at 1M, 60M, and 1B model scales. The corresponding domain labels used in our analysis are listed in Table 1. All empirical analyses in this paper are conducted within this specific experimental setting; how far the empirical findings generalise to other domain taxonomies, corpora, or training objectives is an open question we return to in the Discussion. Table 1: Pile domain labels used in the RegMix main experiment. Label used in this paper Pile component Brief description arxiv ArXiv Scientific papers and preprints freelaw FreeLaw Legal opinions and court documents nih_exporter NIH ExPorter NIH grant and project records pubmed_central PubMed Central Full-text biomedical articles wikipedia_en Wikipedia (en) English Wikipedia articles dm_mathematics DM Mathematics Mathematical problem and proof text github GitHub Source code and code-related text philpapers PhilPapers Philosophy papers and bibliographic text stackexchange Stack Exchange Question-answer forum content enron_emails Enron Emails Corporate email correspondence gutenberg_pg_19 Gutenberg (PG-19) Public-domain books pile_c Pile-C CommonCrawl-derived web text ubuntu_irc Ubuntu IRC Online chat logs from Ubuntu IRC europarl EuroParl European Parliament proceedings hackernews HackerNews Technology-oriented discussion forum text pubmed_abstracts PubMed Abstracts Biomedical article abstracts uspto_backgrounds USPTO Backgrounds Patent background sections The proxy-to-large-scale transfer used in this setting relies on rank preservation across scales. In practical terms, a mixture that performs relatively well for a small model trained under a proxy budget is expected to remain relatively strong for a larger model trained under a larger budget. This assumption makes it possible to use many cheaper proxy runs to guide expensive large-scale mixture selection. It also explains why RegMix evaluates its regression models primarily through rank correlation across scales, rather than only through absolute loss prediction (Liu et al., 2025). The mixture-experiment interpretation makes the structure of such proxy workflows explicit. The sampled proxy mixtures are design points, the validation losses are responses, and the fitted model is a response surface over the simplex. The predictors are not ordinary unconstrained covariates: increasing one domain proportion necessarily reduces the total proportion available to the others. This compositional constraint is precisely the defining feature of a classical mixture experiment. Table 2 summarises this correspondence. Table 2: Correspondence between proxy-based data mixing and classical mixture experiment methodology. Proxy-based data-mixing workflow Mixture experiment interpretation Data domains Mixture components Domain token shares xix_i Component proportions Constraint ∑ixi=1 _ix_i=1 Simplex constraint Sampled proxy-training mixtures Experimental design points Proxy-model validation loss or downstream score Continuous response Predictive model fitted to mixture–performance pairs Response-surface model Candidate-mixture search Response-surface optimisation Random or heuristic mixture sampling Experimental design strategy This reinterpretation matters for two reasons. First, it changes how the response surface can be modelled. Standard regression or machine-learning predictors can be useful for ranking mixtures, but they do not necessarily express the compositional structure of the predictors. For example, RegMix compares linear regression and LightGBM (Ke et al., 2017). A linear model is simple but additive in its basic form, while LightGBM can capture nonlinearities and interactions only implicitly through the fitted tree ensemble. Scheffé response-surface models provide a natural alternative: they are defined directly on the simplex and can represent pairwise domain interactions through explicit coefficients. This is useful when the value of a data source depends not only on its own share, but also on the other sources with which it is combined. Second, the mixture-experiment view changes how proxy mixtures can be selected. Random or heuristic sampling is convenient and scalable, but it does not ask whether the selected mixtures are the most informative design points for estimating a response surface. If proxy runs are experimental design points rather than merely sampled candidates, then their locations on the simplex can be chosen to improve prediction over the mixture region. In particular, ℐI-optimality targets low average prediction variance over a specified mixture region, making it a natural design criterion when the aim is to predict the performance of unseen mixtures. If such designs can achieve comparable ranking performance with fewer proxy runs, they could reduce the token and compute cost of data-mixing experiments. These observations motivate the empirical analysis that follows. We use the RegMix proxy-training data as a public case study for applying mixture response-surface modelling and optimal design to LLM data mixing. Specifically, we fit sparse quadratic Scheffé models to obtain an interpretable response surface, evaluate whether this model preserves cross-scale mixture rankings, and examine whether ℐI-optimal mixture designs can reduce the number of proxy runs required to recover prediction-relevant mixture orderings. The purpose is not to reformulate RegMix as a method, but to use its proxy-training data to illustrate a broader statistical framework for fixed-budget data mixing. 3 Methods 3.1 Sparse Scheffé response-surface modelling We model proxy-based data-mixing experiments as response-surface problems on the mixture simplex. Let =(x1,…,xK)⊤x=(x_1,…,x_K) denote a data mixture over K domains, with xi≥0x_i≥ 0 and ∑ixi=1 _ix_i=1. The experimental region is therefore the (K−1)(K-1)-dimensional simplex. Under this constraint, the usual intercept is not separately identifiable from the component proportions, since the constant term can be written as 1=∑ixi1= _ix_i. We therefore use the canonical Scheffé response-surface models for mixture experiments, which omit the ordinary intercept (Scheffé, 1958). The first-order Scheffé model is E⁡(Y∣)=∑i=1Kβixi,E(Y )= _i=1^K _ix_i, (1) where Y denotes the validation loss of a proxy LLM trained on mixture x. In this parameterisation, βi _i corresponds to the fitted response at the pure-domain vertex for domain i. This interpretation is model-based, since practical pretraining mixtures need not include pure-domain configurations. To represent pairwise domain interactions, we use the second-order Scheffé model, E⁡(Y∣)=∑i=1Kβixi+∑i=1K−1∑j=i+1Kβijxixj.E(Y )= _i=1^K _ix_i+ _i=1^K-1 _j=i+1^K _ijx_ix_j. (2) The coefficient βij _ij measures the departure from additivity when domains i and j are combined. Since the response is validation loss, βij<0 _ij<0 indicates that the pair lowers the fitted loss relative to the additive expectation, whereas βij>0 _ij>0 indicates that the pair raises it. This interpretation is conditional on the chosen validation objective. The collective contribution of pairwise interactions can be assessed by comparing the first-order model in Eq. (1) with the full second-order model in Eq. (2). We use the nested-model statistic TF=(RSSlin−RSSquad)/(pquad−plin)RSSquad/(n−pquad).T_F= (RSS_lin-RSS_quad)/(p_quad-p_lin)RSS_quad/(n-p_quad). (3) Here RSSlinRSS_lin and RSSquadRSS_quad are the residual sums of squares for the first-order and full second-order models, while plinp_lin and pquadp_quad are the corresponding numbers of fitted terms. Eq. (3) evaluates whether the pairwise interaction terms improve the response surface as a group. It is not used as a significance test for individual interaction coefficients. In the empirical RegMix case study, K=17K=17, so the full second-order model contains 17+(172)=15317+ 172=153 terms. Even when interactions are useful as a group, interpreting all pairwise coefficients is undesirable because many terms may be weak, redundant, or unstable. We therefore use an L1L_1-penalised second-order Scheffé model to obtain a sparse response surface: ^=arg⁡min‖−‖22+λ‖1, β= _ β \\|y-Z β\|_2^2+λ\| β\|_1 \, (4) where y is the vector of validation losses, Z is the full second-order Scheffé model matrix, and λ controls the strength of the sparsity penalty. The L1L_1 penalty shrinks weak coefficients to zero, producing a smaller set of selected domain main effects and pairwise interactions (Tibshirani, 1996). The resulting sparse model is used to interpret the fitted response surface. 3.2 Model-robust ℐI-optimal mixture design In mixture experiments, optimal design criteria are commonly used to select informative experimental configurations. Two standard criteria are D-optimality and ℐI-optimality. The former targets precise estimation of model parameters, while the latter targets accurate prediction over the experimental region (Cornell, 2002; Goos and Jones, 2011). Since proxy-based data-mixing studies use pilot experiments to predict the performance of unseen data mixtures, this paper focuses on ℐI-optimality. A detailed description of D-optimality is provided in the Supplementary Information. In the present setting, a proxy-training design is a collection of mixture configurations used to train small proxy LLMs and estimate the response surface. For a proxy-training design =(1,…,n)⊤∈ℝn×KX=(x_1,…,x_n) ^n× K, let ⁡()f(x) denote the response-surface feature vector for a candidate mixture x, and let ⁡()Z(X) denote the corresponding model matrix. ℐI-optimal designs minimise the average prediction variance over the experimental region. For a linear response-surface model with homoscedastic errors, this criterion can be written as ℐ⁡()=log⁡tr⁡[()−1],I(X)= \tr [M(X)^-1W ] \, (5) where ⁡()M(X) is the information matrix and W is the moment matrix over the simplex. The information matrix is ⁡()=()⊤().M(X)=Z(X) Z(X). (6) The moment matrix is defined as =∫⁡()()⊤,W= _Xf(x)f(x) \,dx, (7) where X is the feasible mixture region. Each element of W is therefore an integral of a monomial over the simplex. For non-negative integer powers a1,…,aKa_1,…,a_K, these moments are given by ∫x1a1x2a2⋯xKaKd=∏i=1KΓ⁡(ai+1)Γ⁡(K+∑i=1Kai). _Xx_1^a_1x_2^a_2·s x_K^a_K\,dx= _i=1^K (a_i+1) (K+ _i=1^Ka_i ). (8) To illustrate the construction of W, consider a second-order Scheffé model with three mixture components. The feature vector is ⁡()=(x1,x2,x3,x1x2,x1x3,x2x3)⊤.f(x)=(x_1,x_2,x_3,x_1x_2,x_1x_3,x_2x_3) . Using Eq. (8), the corresponding moment matrix is =(112124124160160112012411212416011201601241241121120160160160160112011801360136016011201601360118013601120160160136013601180).W= pmatrix 112& 124& 124& 160& 160& 1120\\ 124& 112& 124& 160& 1120& 160\\ 124& 124& 112& 1120& 160& 160\\ 160& 160& 1120& 1180& 1360& 1360\\ 160& 1120& 160& 1360& 1180& 1360\\ 1120& 160& 160& 1360& 1360& 1180 pmatrix. Lower values of ℐ⁡()I(X) indicate lower average prediction variance over the mixture region. To compare a design X with a reference design ∗X^* under the same response-surface model, we use relative ℐI-efficiency, ℐ-eff(,∗)=exp⁡[ℐ⁡(∗)−ℐ⁡()].I-eff(X,X^*)= [I(X^*)-I(X) ]. (9) Values greater than one indicate that X has lower average prediction variance than the reference design. During the design stage, information about domain interactions is limited, making it difficult to determine whether a first-order or second-order response surface is more appropriate. Since a design optimised for one candidate model can be inefficient under another model form (Atkinson and Haines, 1996), we adopt a model-robust ℐI-optimal design strategy (Yu et al., 2008; Mao and Kessels, 2026). The criterion is constructed over two candidate Scheffé models: the first-order model and the full second-order model. Let ℐ1()I_1(X) and ℐ2()I_2(X) denote the ℐI-criteria computed under the first-order and second-order candidate models, respectively. The model-robust criterion is defined as ℐrob()=12ℐ1()+12ℐ2().I_rob(X)= 12I_1(X)+ 12I_2(X). (10) This criterion favours proxy-training designs that retain good average prediction precision under both additive and interaction-inclusive response surfaces. To construct the model-robust ℐI-optimal designs, we use the simulated annealing algorithm for optimal mixture designs proposed by Mao and Kessels (2025). This algorithm is suitable for the present simplex-constrained optimisation problem and has been shown to produce high-quality mixture designs relative to conventional hill-climbing search strategies. Detailed pseudocode and implementation details are provided in the Supplementary Information. 4 Empirical Response-Surface Analysis This section illustrates the Scheffé mixture-response-surface formulation using the RegMix proxy-training data. The goal is to decompose the fitted response surface into additive domain contributions and pairwise departures from additivity, and then assess whether this structured surface preserves mixture rankings across model scales. 4.1 Scheffé re-analysis of the response surface The first-order terms of the Scheffé model summarise the additive contribution of each domain to the fitted mixture response surface. Because the response is validation loss, smaller first-order coefficients correspond to lower fitted loss under the additive component. These coefficients are defined on the simplex and should not be read as unconstrained marginal effects or as estimates of pure-domain training performance. Figure 1 reports the first-order coefficients from the sparse Scheffé model. The additive ordering departs from conventional data-quality assumptions. The smallest coefficient belongs to enron_emails, followed by hackernews, philpapers, nih_exporter, ubuntu_irc, and pile_c. Several substantively rich or technical domains, including github, arxiv, pubmed_central, freelaw, and stackexchange, carry larger first-order coefficients, and are therefore not the most favourable additive contributors to reducing the target validation loss. Figure 1: First-order domain coefficients from the sparse Scheffé response-surface model. Because the response is validation loss, smaller coefficients correspond to lower fitted loss under the additive component. The additive surface is informative but incomplete. Comparing the first-order and full second-order Scheffé models yields F=6.5145F=6.5145 on 136136 interaction and 359359 residual degrees of freedom (p<0.001p<0.001), so the pairwise terms improve the fitted surface beyond the first-order Scheffé model. The sparse second-order model was fitted using the L1L_1 penalty in Eq. (4), with λ selected by 10-fold cross-validation using mean squared prediction error over a logarithmic grid from 10−510^-5 to 11. The selected value was λ=1.4175×10−5λ=1.4175× 10^-5. At this penalty level, the model retained 80 terms, including all 17 first-order terms and 63 pairwise interactions. Among these interactions, 51 have negative coefficients and 12 have positive coefficients. Negative coefficients indicate fitted loss reductions relative to the additive expectation, while positive coefficients indicate fitted loss increases. Figure 2: Selected pairwise interaction coefficients from the sparse second-order Scheffé model. Blue cells denote negative coefficients, corresponding to lower fitted validation loss relative to the additive mixture baseline. Red cells denote positive coefficients, corresponding to higher fitted validation loss relative to the additive baseline. White or near-white cells indicate interactions that are weak or not selected by the sparse model. Figure 2 shows that the interaction surface is sparse and structured, dominated by a few pronounced effects centered on pile_c rather than by a diffuse field of weak pairwise terms. Its strongest negative interactions, in descending magnitude, are with arxiv (β=−2.62β=-2.62), pubmed_central (−2.36-2.36), freelaw (−2.31-2.31), github (−2.24-2.24), uspto_backgrounds (−1.83-1.83), stackexchange (−1.78-1.78), and pubmed_abstracts (−1.78-1.78), with further negative terms involving wikipedia_en, gutenberg_pg_19, and dm_mathematics; the four largest coefficients in the entire model are all pile_c pairings. Several of these partner domains carry large first-order coefficients, so their contribution is better explained by complementarity with pile_c than by additive effects. This contrast between main effects and interactions is the central result. Domains such as arxiv, pubmed_central, freelaw, and github are unfavourable in the additive component yet become favourable once combined with pile_c. The positive interactions are fewer, smaller, and more localised. The three largest are dm_mathematics with pubmed_abstracts (β=+1.62β=+1.62), nih_exporter with github (+1.45+1.45), and pubmed_central with dm_mathematics (+1.11+1.11). A positive coefficient means that the two domains together yield higher fitted loss than their additive contributions predict, so allocating weight to both at once is less efficient than the additive surface suggests. The likely cause is redundancy or competition under the target objective: the two domains supply overlapping signal, so weight spent on both returns less than their separate contributions imply. This decomposition illustrates what is gained by treating proxy data mixing as a mixture response-surface experiment. Flexible predictors can capture nonlinearities and interactions inside the fitted function, but they do not necessarily expose those interactions as interpretable quantities. In the present case study, the Scheffé decomposition represents the interaction structure as a set of signed pairwise coefficients, making it possible to identify which domains are complementary, which compete, and by how much. The strongest such structure is the role of pile_c: it carries only a moderate first-order coefficient yet enters the four largest interactions in the model, all negative. Its value is therefore relational rather than standalone. This helps explain why web-derived text can be highly useful in this dataset: its advantage operates largely through complementarity with other domains rather than through a dominant additive effect. The same decomposition also changes how individual domains should be interpreted. A domain that appears weak under the additive component may still be valuable when combined with other domains, while a domain that appears favourable in isolation may contribute less through interactions. For example, philpapers has a favourable first-order coefficient but enters only one strong interaction, whereas domains such as arxiv and pubmed_central look weak additively but become more favourable once their interactions with pile_c are included. The relevant quantity is therefore not whether a domain is good or bad in isolation, but how its additive and interaction components combine within the simplex. The Scheffé surface makes both components directly inspectable. 4.2 Rank preservation across model scales We evaluate cross-scale transfer by comparing predicted and observed mixture orderings at the 1M, 60M, and 1B evaluation scales. Let y^ig y_ig denote the predicted response for mixture i under fitted response-surface model g, and let yi(s)y_i^(s) denote the observed response for the same mixture at scale s∈1M,60M,1Bs∈\1M,60M,1B\. We report Spearman’s rank correlation, ρs(g)=corrSpearman⁡(y^igi=1ns,yi(s)i=1ns), _s(g)=corr_Spearman (\ y_ig\_i=1^n_s,\y_i^(s)\_i=1^n_s ), (11) and pairwise ranking accuracy, PRAs(g)=2ns(ns−1)∑1≤i<j≤ns[signyi(s)−yj(s)=signy^ig−y^jg].PRA_s(g)= 2n_s(n_s-1) _1≤ i<j≤ n_s1 [sign\y_i^(s)-y_j^(s)\=sign\ y_ig- y_jg\ ]. (12) Spearman’s ρs(g) _s(g) measures agreement between the overall predicted and observed rankings. PRAs(g)PRA_s(g) measures the proportion of mixture pairs whose relative ordering is correctly predicted. All three models are fitted on the 512 proxy mixtures at the 1M scale and evaluated on held-out mixtures at each scale. The number of evaluation mixtures is 256 at the 1M and 60M scales and 64 at the 1B scale. To quantify uncertainty in these rank-preservation metrics, we report 95% percentile bootstrap confidence intervals based on 2,000 bootstrap replicates. For Spearman’s ρ, bootstrap replicates resample evaluation mixtures with replacement and recompute the rank correlation using the fixed fitted model predictions. For PRAPRA, bootstrap replicates resample the binary correctness indicators over unordered mixture pairs with replacement, using the mixture pair as the resampling unit. Table 3: Cross-scale rank-preservation performance of response-surface models fitted to the RegMix proxy-training data. Values in brackets are 95% percentile bootstrap confidence intervals. The intervals are defined by the 2.5th and 97.5th percentiles of the resulting bootstrap distributions. Model Scale n ρ PRA First-order Scheffé 1M 256 0.902 [0.869, 0.925] 0.867 [0.863, 0.870] First-order Scheffé 60M 256 0.892 [0.859, 0.917] 0.859 [0.855, 0.863] First-order Scheffé 1B 64 0.879 [0.767, 0.942] 0.864 [0.849, 0.878] Sparse Second-order Scheffé 1M 256 0.937 [0.914, 0.952] 0.894 [0.891, 0.898] Sparse Second-order Scheffé 60M 256 0.939 [0.916, 0.954] 0.896 [0.893, 0.900] Sparse Second-order Scheffé 1B 64 0.975 [0.949, 0.985] 0.937 [0.926, 0.948] LightGBM 1M 256 0.990 [0.986, 0.992] 0.959 [0.957, 0.961] LightGBM 60M 256 0.986 [0.979, 0.989] 0.951 [0.949, 0.953] LightGBM 1B 64 0.962 [0.913, 0.982] 0.927 [0.916, 0.938] Table 3 compares the first-order Scheffé model, the sparse second-order Scheffé model, and LightGBM under both metrics. The first-order model already recovers most of the cross-scale ordering. Its Spearman correlations range from 0.879 to 0.902, and its PRA values remain above 0.85 at all three scales. This indicates that additive mixture structure captures a substantial part of the ranking signal. However, uncertainty is larger at the 1B scale, where the evaluation set contains only 64 mixtures. For the first-order model at 1B, for example, ρ=0.879ρ=0.879 with a 95% confidence interval of [0.767, 0.942]. Adding sparse second-order terms improves both metrics at all three scales. At 1M, Spearman’s ρ increases from 0.902 to 0.937, and PRA increases from 0.867 to 0.894. At 60M, the corresponding increases are from 0.892 to 0.939 for ρ and from 0.859 to 0.896 for PRA. The largest point-estimate gain occurs at 1B, where ρ increases from 0.879 to 0.975 and PRA increases from 0.864 to 0.937. The bootstrap intervals also support a consistent improvement of the sparse second-order model over the first-order model, particularly for PRA. These results suggest that the interaction structure identified in Section 4.1 is relevant for cross-scale ranking, not only for interpreting the fitted response surface. LightGBM performs best at the 1M and 60M scales, with ρ=0.990ρ=0.990 and PRA=0.959PRA=0.959 at 1M, and ρ=0.986ρ=0.986 and PRA=0.951PRA=0.951 at 60M. This is expected given its greater flexibility. At the 1B scale, the sparse second-order Scheffé model attains slightly higher point estimates than LightGBM, with ρ=0.975ρ=0.975 and PRA=0.937PRA=0.937, compared with ρ=0.962ρ=0.962 and PRA=0.927PRA=0.927 for LightGBM. However, this difference should be interpreted cautiously. The 1B evaluation set contains only 64 mixtures, and the bootstrap confidence intervals overlap for both metrics. The evidence therefore supports comparable performance at 1B, rather than a clear claim that the sparse Scheffé model outperforms LightGBM. The main implication is that sparse Scheffé modelling provides a useful compromise between interpretability and cross-scale ranking performance. LightGBM remains the strongest flexible predictor at the 1M and 60M scales, while the sparse second-order Scheffé model remains competitive at 1B and provides an explicit decomposition of main effects and interactions. This makes the sparse Scheffé surface particularly useful when the goal is not only to predict promising mixtures, but also to understand which mixture components and interactions drive cross-scale transfer. 5 Optimal Design for Proxy Data-Mixing Experiments We next examine whether the proxy-training mixtures can be selected more efficiently than by random Dirichlet sampling. Using the simulated annealing algorithm described in Section 3.2, we constructed model-robust ℐI-optimal designs for a sequence of design sizes n∈160,192,224,256,288,320,352,384,416,448,480,512n∈\160,192,224,256,288,320,352,384,416,448,480,512\. Each design was optimised under the optimality criterion introduced in Eq.(10). We then compared these designs with the original 512-run proxy design from the empirical case study in terms of design geometry, relative ℐI-efficiency, and simulated ranking performance. Figure 3: Geometry of the model-robust ℐI-optimal design at n=512n=512 compared with the original 512-run RegMix proxy design. Panel A shows the realised mixture proportions for each domain across all design points, with vertical jitter added for visibility. Panel B shows the empirical cumulative distribution function of all realised domain proportions pooled across domains and design points. The RegMix design spreads proportions more continuously over the simplex, whereas the model-robust ℐI-optimal design concentrates most proportions at the boundary values 00, 0.50.5, and 11. Figure 3 compares the geometry of the model-robust ℐI-optimal design at n=512n=512 with the original 512-run reference design from the RegMix data. Panel A shows the realised proportions separately for each domain, while Panel B summarizes the marginal distribution of all realised domain proportions. The two designs have markedly different structures. The RegMix reference design spreads proportions continuously across the simplex, with many domains assigned small positive shares across runs. In contrast, the model-robust ℐI-optimal design is much more discrete: most realised domain proportions are placed at 00, with additional support concentrated around 0.50.5 and 11. The empirical CDF in Panel B makes this contrast explicit, showing a sharp initial jump for the optimal design and step changes at the main support points. This boundary-concentrated structure is consistent with low-order mixture response-surface estimation, since simplex vertices identify first-order component effects and binary edge midpoints are informative for pairwise Scheffé interactions. Figure 4: Relative ℐI-efficiency of 1,0001,000 Dirichlet designs at n=512n=512, evaluated under the sparse second-order Scheffé basis with the model-robust ℐI-optimal design as reference. Values below one indicate higher average prediction variance than the model-robust ℐI-optimal design. We also compared the prediction-variance efficiency of the optimal design with 1,000 random Dirichlet designs of the same size. Let ℐX^I denote the model-robust ℐI-optimal design, and let Dir(b)X^(b)_Dir denote the bbth random Dirichlet design, for b=1,…,1000b=1,…,1000. Following the relative ℐI-efficiency definition in Eq. (9), we computed ℐ-eff(Dir(b),ℐ)=exp⁡[ℐ⁡(ℐ)−ℐ⁡(Dir(b))],I-eff (X^(b)_Dir,X^I )= [I(X^I)-I(X^(b)_Dir) ], where both ℐI-criteria were evaluated under the sparse second-order Scheffé basis. Figure 4 shows that these relative efficiencies are concentrated around 0.070.07 to 0.080.08. Thus, under this basis, the random Dirichlet designs have substantially higher average prediction variance than the model-robust ℐI-optimal design. Finally, we evaluated whether this design advantage translates into mixture-ranking performance. We used the sparse second-order Scheffé surface fitted to the full RegMix proxy data as the data-generating surface. For each design size n, responses were generated at the corresponding model-robust ℐI-optimal design points with Gaussian noise estimated from the fitted residuals. The sparse Scheffé model was then refitted and used to rank the held-out 1B mixtures. This procedure was repeated for 1,000 simulated datasets at each design size. Let ρn(b) _n^(b) and PRAn(b)PRA_n^(b) denote the Spearman rank correlation and pairwise ranking accuracy obtained from the bbth simulated dataset, for b=1,…,1000b=1,…,1000. We report the Monte Carlo averages ρ¯n=11000∑b=11000ρn(b),PRA¯n=11000∑b=11000PRAn(b). ρ_n= 11000 _b=1^1000 _n^(b), PRA_n= 11000 _b=1^1000PRA_n^(b). Figure 5 shows that both ρ¯n ρ_n and PRA¯n PRA_n increase with design size. The model-robust ℐI-optimal designs reach and then exceed the original n=512n=512 reference level with roughly 350 proxy runs, under both ρ¯n ρ_n and PRA¯n PRA_n. At n=512n=512, the model-robust ℐI-optimal design exceeds the RegMix reference under both metrics. Figure 5: Monte Carlo average ranking performance of the model-robust ℐI-optimal designs as a function of design size n (solid line with markers), compared with the original RegMix design at n=512n=512 (dashed line). Left: average Spearman rank correlation ρ¯n ρ_n between the predicted and reference 1B orderings. Right: average pairwise ranking accuracy PRA¯n PRA_n. Each point averages 1,0001,000 simulated datasets generated from the sparse Scheffé surface. These results show that design-aware proxy mixture selection can reduce the proxy-training budget while preserving the relevant mixture ordering. In this simulation study, the model-robust ℐI-optimal designs match or exceed the original n=512n=512 RegMix design after removing about 25% of the proxy runs. Since each proxy run requires training a small model on a fixed token budget, this reduction translates directly into lower pilot-stage token and compute cost. The exact saving is specific to the fitted sparse Scheffé surface, but the result shows that optimal design can materially improve the efficiency of proxy-based data-mixing experiments. As a sensitivity check, the Supplementary Material repeats the ranking evaluation under a first-order Scheffé data-generating surface and shows that the model-robust ℐI-optimal designs continue to outperform the original RegMix reference design. 6 Discussion This paper reframes LLM data mixing as a mixture experiment. Under this formulation, data domains are mixture components, proxy training runs are experimental design points, and validation loss defines a response surface over the probability simplex. This perspective turns data-mixing optimisation from a purely predictive task into a structured experimental-design problem. The empirical results show that this structure is useful in two ways: sparse Scheffé models provide an interpretable decomposition of domain effects and interactions, while model-robust ℐI-optimal designs can reduce the number of proxy runs needed to recover useful mixture rankings. The empirical case study shows that domain value can be strongly relational. Several domains that are not favourable under the additive component become favourable through pairwise interactions, especially through combinations with pile_c. This suggests that pretraining data quality should not be understood only as a property of individual domains. A domain may be useful because of how it complements other domains under a fixed token budget. The Scheffé formulation makes this structure explicit by separating additive contributions from interaction effects, which are otherwise hidden inside flexible predictors such as LightGBM. The design results indicate that the proxy stage itself can be made more efficient. Random Dirichlet sampling is convenient, but it is not necessarily the most informative way to estimate a mixture response surface. In the simulation-based case study, model-robust ℐI-optimal designs match or exceed the original RegMix reference after removing about 25% of the proxy runs. This saving should not be read as universal, since it depends on the fitted sparse Scheffé surface and the assumed noise structure. The more general point is that mixture selection can benefit from design-aware sampling, especially when each proxy run carries a non-trivial token and compute cost. Several limitations should be understood in light of the current stage of empirical research on LLM data mixing. Publicly available proxy-training datasets with systematically varied data mixtures remain limited, and generating new large-scale mixture experiments would require substantial token and compute resources. For this reason, the present empirical evaluation is based on one publicly available proxy-training dataset and a simulation-based comparison of alternative designs. The design evaluation uses the sparse Scheffé surface fitted to this dataset as the data-generating mechanism, so its conclusions depend on how well this fitted surface approximates the true training response. If higher-order interactions, threshold effects, or strong local irregularities dominate the response surface, a sparse quadratic Scheffé model may be too restrictive. In addition, the optimal designs are constructed under an idealised mixture setting in which any point in the simplex can in principle be sampled and each proxy run is treated as having the same cost. Practical data-mixing studies may face further constraints, including infeasible domain proportions, data availability, licensing, deduplication, filtering requirements, unequal preprocessing costs, and other operational restrictions. In applied pretraining studies, these restrictions should be incorporated into the design problem itself, either by constraining the feasible mixture region or by using cost-sensitive design strategies (Hameed et al., 2026). Future work can extend the present framework in several directions. First, model scale and training conditions can be incorporated directly into the response-surface model as process variables. In the present study, the response surface is fitted at the proxy scale and then evaluated by its ability to preserve rankings at larger scales. A more integrated formulation would model scale transfer itself by treating model size or token budget as process variables interacting with the mixture proportions. This corresponds to a mixture-process experiment: the mixture variables determine the relative composition of the training data, while the process variables determine the conditions under which that mixture is used (Piepel and Cornell, 1985). This idea is standard in classical mixture experimentation. For example, in industrial formulation studies, the final response may depend not only on ingredient proportions but also on processing conditions such as temperature, pressure, or mixing time; mixture-process models therefore include both mixture terms and mixture-by-process interactions (Cornell, 1988). In the present context, the analogous question is whether the effect of a data mixture changes with model scale or training regime. Second, future work could develop sequential or adaptive designs for data-mixing experiments (Chernoff, 1959). The designs considered in this paper are fixed before proxy training begins. This is appropriate for comparing design geometries, but it may not be the most efficient strategy when proxy runs are expensive. A sequential design could use early proxy results to update the response-surface model, refine uncertainty about important interaction terms, and allocate later runs to regions of the simplex that are most informative for ranking or optimisation. Such a strategy would be especially useful in high-dimensional data-mixing problems, where the number of possible domain combinations grows quickly and a one-shot design may spend resources on regions that are uninformative or practically unattractive. Third, the framework is not limited to language-model pretraining. Any data-mixing problem with a fixed training budget allocated across data sources has the same simplex geometry. This includes multimodal training where image, text, video, and audio data compete for a fixed budget (Aghajanyan et al., 2023); domain-adaptive training where general and domain-specific corpora must be balanced (Gururangan et al., 2020); multilingual training where tokens are allocated across languages or regions (Conneau et al., 2020); and supervised or instruction-tuning settings where examples from different task families are mixed (Longpre et al., 2023). In all these cases, the key experimental unit is not only a model or a dataset, but a deliberately chosen allocation of training budget across data sources. The broader implication is that data mixing should be treated as an experimental design problem whenever the composition of the training data is a controllable part of the learning system. Acknowledgements We thank Rob Deardon for his helpful comments and suggestions on this manuscript. Data and Code Availability The data and code used in this study will be made publicly available in a GitHub repository upon publication of the article. References Aghajanyan et al. (2023) A. Aghajanyan, L. Yu, A. Conneau, W. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer Scaling laws for generative mixed-modal language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §6. Atkinson and Haines (1996) A. C. Atkinson and L. M. Haines Designs for nonlinear and generalized linear models. In Handbook of Statistics 13: Design and Analysis of Experiments, S. Ghosh and C. R. Rao (Eds.), Cited by: §3.2. Chernoff (1959) H. Chernoff Sequential design of experiments. The Annals of Mathematical Statistics 30 (3), p. 755–770. External Links: ISSN 00034851, 21688990, Link Cited by: §6. Conneau et al. (2020) A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 8440–8451. External Links: Link, Document Cited by: §6. Cornell (2002) J. A. Cornell Experiments with mixtures. Wiley. Cited by: §1, §3.2. Cornell (1988) J. A. Cornell Analyzing data from mixture experiments containing process variables: a split-plot approach. Journal of Quality Technology 20 (1), p. 2–23. External Links: Document Cited by: §6. Furlanetto et al. (2011) S. Furlanetto, M. Cirri, G. Piepel, N. Mennini, and P. Mura Mixture experiment methods in the development and optimization of microemulsion formulations. Journal of Pharmaceutical and Biomedical Analysis 55 (4), p. 610–617. Note: Special Issue of Invited Reviews dedicated to Sergio Pinzauti External Links: ISSN 0731-7085, Document Cited by: §1. Gao et al. (2021) L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy The Pile: an 800gb dataset of diverse text for language modeling. CoRR abs/2101.00027. External Links: Link Cited by: §1, §2. Ge et al. (2025) C. Ge, Z. Ma, D. Chen, Y. Li, and B. Ding BiMix: a bivariate data mixing law for language model pretraining. External Links: 2405.14908 Cited by: §1. Goos and Jones (2011) P. Goos and B. Jones Optimal design of experiments: a case study approach. John Wiley & Sons. Cited by: §1, §3.2. Goos et al. (2019) P. Goos, N. Dens, P. De Pelsmacker, and L. Aleksandrovs Using mixture-amount modeling to optimize the advertising media mix and quantify cross-media synergy for specific target groups. Applied Stochastic Models in Business and Industry 35 (5), p. 1228–1252. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/asmb.2470 Cited by: §1. Goos and Hamidouche (2019) P. Goos and H. Hamidouche Choice models with mixtures: an application to a cocktail experiment. Food Quality and Preference 77, p. 135–146. External Links: ISSN 0950-3293, Document, Link Cited by: §1. Gururangan et al. (2020) S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith Don’t stop pretraining: adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 8342–8360. External Links: Document Cited by: §6. Hameed et al. (2026) M. S. I. Hameed, R. van der Haar, Y. Chen, and P. Goos Optimal design of experiments when not every test is equally expensive. Chemometrics and Intelligent Laboratory Systems 269, p. 105617. External Links: ISSN 0169-7439, Document Cited by: §6. Ke et al. (2017) G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Liu LightGBM: a highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . Cited by: §2. Liu et al. (2025) Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin RegMix: data mixture as regression for language model pre-training. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 38305–38339. Cited by: §1, §2, §2. Longpre et al. (2023) S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §1, §6. Mao and Kessels (2025) Y. Mao and R. Kessels Optimal designs for mixture choice experiments by simulated annealing. Chemometrics and Intelligent Laboratory Systems 257, p. 105305. External Links: ISSN 0169-7439, Document, Link Cited by: §3.2. Mao and Kessels (2026) Y. Mao and R. Kessels Simulated annealing for model-robust partial profile choice designs in healthcare preference studies. External Links: 2604.04294, Link Cited by: §3.2. Piepel and Cornell (1985) G. Piepel and J. Cornell Models for mixture experiments when the response depends on the total amount. Technometrics 27, p. 219–227. External Links: Document Cited by: §6. Scheffé (1958) H. Scheffé Experiments with mixtures. Journal of the Royal Statistical Society: Series B (Methodological) 20 (2), p. 344–360. External Links: Document Cited by: §1, §3.1. Tibshirani (1996) R. Tibshirani Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) 58 (1), p. 267–288. External Links: ISSN 0035-9246, Document Cited by: §3.1. Xie et al. (2023) S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. Liang, Q. V. Le, T. Ma, and A. W. Yu DoReMi: optimizing data mixtures speeds up language model pretraining. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §1. Ye et al. (2025) J. Ye, P. Liu, T. Sun, J. Zhan, Y. Zhou, and X. Qiu Data mixing laws: optimizing data mixtures by predicting language modeling performance. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 82263–82287. Cited by: §1. Yu et al. (2008) J. Yu, P. Goos, and M. Vandebroek Model-robust design of conjoint choice experiments. Communications in Statistics—Simulation and Computation 37 (8), p. 1603–1621. External Links: Document Cited by: §1, §3.2. Zijlstra et al. (2019) T. Zijlstra, P. Goos, and A. Verhetsel A mixture-amount stated preference study on the mobility budget. Transportation Research Part A: Policy and Practice 126, p. 230–246. Cited by: §1.

Instant research discovery