Paper deep dive
Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)
Zilin Du, Bowen Yang, Boyang Albert Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/4/2026, 2:25:32 AM
Summary
The paper introduces Transferable Example Scoring and Selection (TESS), a framework for scalable and transferable data selection in Large Language Model (LLM) training. It addresses the limitations of existing Meta-learning for Training-data Selection (MTS) methods, specifically ScaleBiO, which suffer from unstable optimization and poor generalization when using a selection network. The authors identify that the standard MTS loss causes weight suppression and reliance on easy-to-learn features (shortcut learning). TESS proposes a Pointwise Value Matching (PVM) objective that uses pseudo-labels derived from loss differences between models to train a meta-network, enabling generalization to unseen data and reducing memory overhead.
Entities (8)
Relation Signals (6)
Zilin Du → affiliatedwith → Nanyang Technological University
confidence 95% · Zilin Du Affiliation: College of Computing and Data Science Affiliation: Nanyang Technological University
MTS → isappliedto → LLM
confidence 95% · Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective
TESS → uses → PVM
confidence 95% · TESS, a scalable data-selection framework built on a Pointwise Value Matching objective (PVM).
TESS → improves → ScaleBiO
confidence 90% · TESS simplifies ScaleBiO (Pan et al., 2025) and introduces a simple mean squared error objective
TESS → reduces → memory_footprint
confidence 90% · TESS removes the requirement of ScaleBiO to simultaneously train two LLMs with the meta-network, cutting peak memory footprint by more than 20%.
ScaleBiO → suffersfrom → Shortcut Learning
confidence 85% · the ScaleBiO loss continues to reinforce easy-to-learn features that are susceptible to shortcut learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valuation and transferability to unseen data. A natural solution is to replace per-sample weights with a selection network. However, we find that directly incorporating such a network into existing MTS objectives leads to unstable optimization and poor generalization, caused by weight suppression and persistent reliance on easy-to-learn features. To address these issues, we propose Transferable Example Scoring and Selection (TESS), a scalable data-selection framework built on a Pointwise Value Matching objective (PVM). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, from subsets to full corpora, and from smaller to larger models.
Tags
Links
- Source: https://arxiv.org/abs/2610.02092v1
- Canonical: https://arxiv.org/abs/2610.02092v1
Trouble viewing inline? Open PDF directly →
Full Text
94,001 characters extracted from source content.
Expand or collapse full text
Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic) Zilin Du Affiliation: College of Computing and Data Science Affiliation: Nanyang Technological University Affiliation: Singapore Email: zilin003@e.ntu.edu.sg Bowen Yang Affiliation: College of Computing and Data Science Affiliation: Nanyang Technological University Affiliation: Singapore Email: yang1117@e.ntu.edu.sg Boyang Albert Li †thanks: Corresponding author. Affiliation: College of Computing and Data Science Affiliation: Nanyang Technological University Affiliation: Singapore Email: boyang.li@ntu.edu.sg Abstract Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valuation and transferability to unseen data. A natural solution is to replace per-sample weights with a selection network. However, we find that directly incorporating such a network into existing MTS objectives leads to unstable optimization and poor generalization, caused by weight suppression and persistent reliance on easy-to-learn features. To address these issues, we propose Transferable Example Scoring and Selection (TESS), a scalable data-selection framework built on a Pointwise Value Matching objective (PVM). Experiments on LLM safety and targeted instruction tuning demonstrate strong transfer across datasets, from subsets to full corpora, and from smaller to larger models. 1 Introduction The performance of modern AI systems is fundamentally shaped by the training data (Zha et al., 2025; Kumar et al., 2024; Zha et al., 2023; Whang et al., 2023). For large language models (LLMs), scaling requires assembling massive corpora from heterogeneous sources (Zhou et al., 2025; Weber et al., 2024; Langlais et al., 2026; Li et al., 2024a), which often introduces redundant and low-quality examples. Training data selection is hence an important component of LLM training. Many existing methods rank training examples by manually designed scores, such as entropy (Li et al., 2026), instruction-following difficulty (Li et al., 2024b), reducible training loss (Mindermann et al., 2022), and distance to target domain (Deng et al., 2026; Min et al., 2026), to name a few. These scoring rules lack the ability to dynamically adapt to new scenarios. Though each captures a potentially useful signal, it remains an open question how these scores may be combined to create more powerful ranking mechanisms. Meta-learning for Training-data Selection (MTS) (Shen et al., 2025a; Pan et al., 2025; Ren et al., 2018; Wang et al., 2020; Du et al., 2026; Shu et al., 2019; Yu et al., 2025; Calian et al., 2026) provides a principled alternative, where the values of training data are learned directly from their effects on the validation loss ℒvalL_val. Let θ denote the LLM parameters, wi>0w_i>0 be the learnable weight for the loss of the i-th training data point, ℓi() _i( θ), and the vector =(wi)i w=(w_i)_i contain all wiw_i as components. MTS formulates data weighting as the following bilevel optimization problem: ∗=argminℒval(∗()),s. t.∗()=argmin||∑i1−1wiℓi().w^*= _wL_val ( θ^*(w) ), . t. θ^*(w)= _ θ\| w\|_1^-1 _iw_i _i( θ). (1) This framework allows automatic identification of effective training data, which lead the model to generalize to the validation data. Nevertheless, MTS in practice runs into two obstacles; the first is scalability. Computing the exact hypergradient ∇ℒval(∗()) _ wL_val( θ^*(w)) by unrolling the inner optimization is costly. Even unrolling one step and taking the finite-different approximation require three backward operations on θ for one gradient step on w (Du et al., 2026; Shu et al., 2019), which is prohibitive expensive for large LLMs. Calian et al. (2026) manage to unroll 8 steps using 64 v5 TPUs, which is beyond the capability of most academic labs. With penalty-based reformulations (Shen & Chen, 2023; Xiao et al., 2023; Shen et al., 2025b), ScaleBiO (Pan et al., 2025) reduces the number of backward calls to two, whereas SEAL (Shen et al., 2025a) reduces it further to one, but incurs discernible performance losses. The second challenge is generalization to unseen data. In the formulation of Ren et al. (2018); Shen et al. (2025a), every data point receives a separate weight wiw_i, and no generalization is possible. An alternative approach (Pan et al., 2025) is to assign a separate weight for each data domain (e.g., one dataset in a collection of multiple datasets), which provides generalization within each domain. However, this overlooks heterogeneity among data points in a single domain and cannot generalize to new data domains. One intuitively appealing idea is to train a meta-network that outputs wiw_i for every data point. If the meta-network can generalize to unseen data points, we would be able to train on a small dataset and perform data selection on a much larger dataset, which would also substantially improve efficiency and scalability. Though the meta-network approach has been used by Shu et al. (2019); Du et al. (2026); Calian et al. (2026), no such generalization has been reported, not even for the extraordinarily large-scale experiments of Calian et al. (2026). Perhaps unsurprisingly, our experiments find simplistic training of the meta-network leads to unstable optimization (Figure 2) and poor generalization (Table 1 and 3). Given the massive success of deep learning, the failure of the meta-network to generalize to unseen data is especially perplexing. Our analysis traces the failure to two properties of the training loss. First, as training progresses, most predicted data weights are suppressed toward zero. As a result, over time, the meta-network loses the ability to distinguish useful training data from harmful training data. Second, the loss causes the network to exclusively learn easy features and ignores hard feature, so the network becomes susceptible to shortcut learning. To improve the scalability and generalization of MTS, we propose Transferable Example Scoring and Selection (TESS). At its core, TESS simplifies ScaleBiO (Pan et al., 2025) and introduces a simple mean squared error objective, which we call Pointwise Value Matching (PVM). TESS constructs sample-level pseudo-labels that capture the utility of training data points and uses these values as supervision for the data-selecting meta-network. Compared to the existing loss, the PVM loss is less focused on easy features, which curbs shortcut learning and promotes generalization. Further, TESS removes the requirement of ScaleBiO to simultaneously train two LLMs with the meta-network, cutting peak memory footprint by more than 20%. Experiments on LLM safety and targeted instruction tuning demonstrate substantially improved generalization of TESS across three settings: cross-dataset transfer, small-to-large model transfer, and subset-to-full-corpus transfer. Our contributions can be summarized as: • We identify and theoretically analyze two limitations of applying the existing MTS training objective to a shared selection network: competitive weight suppression that undermines the identification of high-quality training data, and persistent preference for easy-to-learn features that can encourage shortcut learning. • We propose TESS, a transferable data-selection framework that constructs sample-level pseudo-labels and uses them as supervision to learn a data selection network generalizable to unseen data. To the best of our knowledge, this is the first work that demonstrates generalization to unseen data in MTS. 2 Related Work Meta-Learning Data Selection for LLMs. Scaling meta-learning data selection to LLMs requires efficiency in face of both huge parameter counts (Pan et al., 2025) and dataset sizes (Xie et al., 2023b). Traditional data selection methods (Ren et al., 2018; Shu et al., 2019) differentiate through model updates to compute the hypergradient, which incurs large time and space complexity for large models. ScaleBiO (Pan et al., 2025) and SEAL (Shen et al., 2025a) focus on scaling to model parameter counts. If data selection can be trained on a small dataset and generalize to unseen data, it would also significantly improve scalability to large datasets. However, such generalization remains a difficult challenge. In terms of generalization, existing approaches can be broadly grouped into sample-level, domain-level, and meta-network methods. Example-level methods (Xie et al., 2023b; Xia et al., 2024; Shen et al., 2025a; Calian et al., 2026) learn one weight for each training data point, thereby offering no generalization to new data points. Domain-level methods (Xie et al., 2023a; Xie et al., 2025; Pan et al., 2025; Yang et al., 2026; Wen et al., 2026) partition the training data into domains and assign one data weight to each domain. Chameleon (Xie et al., 2025) and Data Mixing Agent (Yang et al., 2026) further introduce a shared domain taxonomy that enables limited weight sharing among data domains. However, domain-level weights are too coarse to capture the crucial differences in data quality and content within each domain (Lee et al., 2024). Meta-network methods (Shu et al., 2019; Du et al., 2026; Calian et al., 2026) train a meta-network to output the weight for each data point directly. In theory, this should allow generalization to new data points; in practice, no generalization has been reported. Notably, DataRater (Calian et al., 2026) trains a separate scorer for each dataset, suggesting generalization across dataset is challenging even when the network are trained with a huge amount of data. With this paper, we analyze the reasons behind the challenge mathematically and offer a simple, practical, and high-performance solution. Shortcut Learning. Shortcut learning occurs when models rely on easy features that are strongly predictive of training labels but whose correlations weaken or reverse under distribution shifts, impairing out-of-distribution generalization (Geirhos et al., 2020; Shah et al., 2020). Prior work distinguishes predictivity, how reliably a feature predicts training labels, from availability, how easily it can be extracted (Hermann et al., 2024). Scimeca et al. (2021) find that solutions favoring easy-to-learn cues tend to lie in flatter minima and are more abundant in parameter space, while Qiu et al. (2024) show that simpler or more strongly correlated spurious features can slow core-feature learning. We investigate these dynamics in data selection networks. We show that the ScaleBiO loss continues to reinforce easy-to-learn features that are susceptible to shortcut learning, while our proposed PVM promotes the learning of harder and more transferable features. 3 Methodology 3.1 Problem Formulation Let trainD_train and valD_val denote the training and target validation sets, with sizes NtrainN_train and NvalN_val, respectively. We further use poolD_pool to denote the candidate pool to be scored. For example, poolD_pool can be a larger corpus containing trainD_train, train⊊poolD_train _pool, or a disjoint dataset, train∩pool=∅D_train _pool= . Let ℓi() _i( θ) denote the token-averaged cross-entropy loss for the i-th data point under an LLM parameterized by θ. For the training examples, let wi>0w_i>0 denote the weight of the i-th data point, =(wi)i=1Ntrain w= (w_i)_i=1^N_train the weight vector, and S=‖1S=\| w\|_1 the total weight. We aim to learn a data selection meta-network sϕs_ φ that predicts sample weights and can score unseen examples in poolD_pool. We parameterize wi=sϕ(xi)>0w_i=s_ φ(x_i)>0 and denote the resulting weight vector by (ϕ) w( φ). In MTS, learning ϕ φ is formulated as bilevel optimization: ϕ∗ φ^* =argminϕℒ(∗,val)=argminϕNval−1∑i=1Nvalℓi(∗), = _ φL ( θ^*;D_val )= _ φN_val^-1Σ _i=1^N_val _i( θ^*), (2) s.t.∗ .t. θ^* =argminℒ(,(ϕ),train)=argmin||∑i=1Ntrain1−1wiℓi(). = _ θL ( θ, w( φ);D_train )= _ θ\| w\|_1^-1Σ _i=1^N_trainw_i\, _i( θ). (3) Here, ℒ(,)L( θ;D) denotes the average per-sample loss, while ℒ(,(ϕ),)L( θ, w( φ);D) denotes the loss weighted by the meta-network ϕ φ. Under this formulation, we optimize ϕ φ such that training the LLM with its predicted weights minimizes the loss on valD_val. 3.2 Preliminaries on ScaleBiO ScaleBiO (Pan et al., 2025) provides an efficient framework for MTS at the LLM scale, optimizing data weights w for each data point. Let W θ_W denote the LLM parameters and U θ_U denote the parameters of an auxiliary LLM. Optimizing U θ_U on the training set yields U∗=argminUℒ(U,,train) θ_U^*= _ θ_UL( θ_U, w;D_train). The original constraint in the bi-level problem (Eq. 1) can then rewritten as W=U∗ θ_W= θ_U^*. Relaxing the constraint to a necessary condition that W θ_W and U∗ θ_U^* have the same training loss, we obtain a new constraint: ℒ(W,,train)−minUℒ(U,,train)=0.L( θ_W, w;D_train)- _ θ_UL( θ_U, w;D_train)=0. Building the Lagrangian function with the penalty coefficient α>0α>0, we derive a new minimax problem: min,WmaxUℒ(W,val)+α[ℒ(W,,train)−ℒ(U,,train)]. _ w, θ_W _ θ_U ( θ_W;D_val)+α [L( θ_W, w;D_train)-L( θ_U, w;D_train) ]. (4) To solve Eq.4, ScaleBiO simultaneously trains the two LLMs and data weights, resulting in substantial memory overhead. SEAL (Shen et al., 2025a) additionally eliminates the auxiliary LLM U θ_U, but this comes at an accuracy penalty. We write the ScaleBiO loss for data weights w explicitly: ℒwSBO(,W,U)=‖1−1∑i=1Ntrainwi[ℓi(W)−ℓi(U)]. L_w^SBO( w; θ_W, θ_U)= \| w\|_1^-1Σ _i=1^N_trainw_i [ _i( θ_W)- _i( θ_U) ]. (5) It is evident that, to optimize LwSBOL_w^SBO, we should assign high wiw_i to data points with a low loss on w θ_w and a high loss on U θ_U. Note that w θ_w is trained on both the training set DtrainD_train and the validation set DvalD_val but U θ_U is trained on DtrainD_train only. Therefore, ScaleBiO selects data points that benefit from additional training on DvalD_val, which are likely more similar to DvalD_val than DtrainD_train. 3.3 Transferable Example Scoring and Selection (TESS) Algorithm 1 Overview of the TESS Framework 0: Training set trainD_train, validation set valD_val, candidate pool poolD_pool, coefficient α 1: Obtain U∗ θ_U^* by minimizing ℒ(U,train)L( θ_U;D_train) 2: Obtain W∗ θ_W^* by minimizing ℒ(W,val)+αℒ(W,train)L( θ_W;D_val)+ ( θ_W;D_train) 3: For each xi∈trainx_i _train, compute the pseudo-label Δi(U∗,W∗)←ℓ(U∗,xi)−ℓ(W∗,xi) _i( θ_U^*, θ_W^*)← ( θ_U^*;x_i)- ( θ_W^*;x_i) 4: Train the selection network by optimizing ϕ∗←argminϕℒϕPVM(ϕ,W∗,U∗) φ^*← _ φ L_ φ^PVM( φ; θ_W^*, θ_U^*) (Eq.6) 5: For each xj∈trainx_j _train, predict w^j←sϕ∗(xj) w_j← s_ φ^*(x_j) 6: return Predicted data values ^pool=w^jxj∈pool w_pool=\ w_j\_x_j _pool Incorporating a Selection Network. We extend ScaleBiO by replacing the independent data weights wiw_i with the data selection meta-network sϕ(xi)s_ φ(x_i). This yields the ScaleBiO meta-network objective (SBO): ℒϕSBO(ϕ,W,U)=(∑i=1Ntrainsϕ(xi))−1∑i=1Ntrainsϕ(xi)[ℓi(W)−ℓi(U)]. L_ φ^SBO( φ; θ_W, θ_U)= (Σ _i=1^N_trains_ φ(x_i) )^-1 Σ _i=1^N_trains_ φ(x_i) [ _i( θ_W)- _i( θ_U) ]. (6) However, empirically optimizing the meta-network with ℒϕSBO L_ φ^SBO leads to unstable optimization (Figure 2) and poor generalization (Tables 1 and 3). We find this can be simply addressed by Pointwise Value Matching, presented below. Pointwise Value Matching. The key idea is to use the loss difference between the two LLMs as a proxy for sample utility. Recall that U θ_U is trained only on trainD_train, whereas W θ_W is additionally guided by valD_val. For each training example xix_i, we define Δi(U,W):=ℓi(U)−ℓi(W) _i( θ_U, θ_W):= _i( θ_U)- _i( θ_W). A larger Δi _i indicates a greater loss reduction under the validation-guided model relative to the training-only model, suggesting greater relevance to the target validation objective. We use this difference as a pseudo-label and train the meta-network to match it: ℒϕPVM(ϕ,W,U)=Ntrain−1∑i=1Ntrain(sϕ(xi)−Δi(U,W))2. L_ φ^PVM( φ; θ_W, θ_U)= N_train^-1Σ _i=1^N_train (s_ φ(x_i)- _i( θ_U, θ_W) )^2. (7) Sequential Training of TESS. Unlike ScaleBiO, TESS separates the training of the three models. Specifically, using uniform sample weights, we obtain U∗ θ_U^* by minimizing ℒ(U,train)L( θ_U;D_train) and W∗ θ_W^* by minimizing ℒ(W,val)+αℒ(W,train)L( θ_W;D_val)+ ( θ_W;D_train). We then freeze both models, compute Δi(U∗,W∗) _i( θ_U^*, θ_W^*) for each training example, and train the meta-network by minimizing ℒϕPVM(ϕ,W∗,U∗) L_ φ^PVM( φ; θ_W^*, θ_U^*). This sequential design has two advantages over ScaleBiO. First, it avoids simultaneously optimizing two LLMs and the selection network, reducing peak memory consumption. Second, it allows the pretrained LLMs and the randomly initialized meta-network to use independent optimization schedules. The trained meta-network directly scores unseen examples in poolD_pool without further optimization. The complete procedure of TESS is summarized in Algorithm 1. 4 Why the ScaleBiO Loss Performs Poorly We next examine why optimizing with the ScaleBiO objective can fail to produce a transferable meta-network. We identify two properties of the ScaleBiO objective: competitive weight suppression, which weakens sample discrimination, and a persistent preference for easy-to-learn features, which can encourage shortcut learning. 4.1 Data Weight Suppression and Loss of Discrimination Our first observation is that the ScaleBiO objective drives most predicted weights toward zero, alongside losing the power of identifying good data (Figure 2). This occurs under both joint training in ScaleBiO and sequential training in TESS when training with ℒϕSBO L_ φ^SBO. Our analysis below strictly applies to sequential training in TESS, where the two LLMs are fixed. Empirically, we observe qualitatively similar dynamics under the joint training in ScaleBiO. We first treat the weights of training examples as independent hyperparameters. For the selection network, wi=sϕ(xi)w_i=s_ φ(x_i) is updated through shared parameters ϕ φ. However, the update directions on wiw_i with and without ϕ φ are similar. With Δi∗:=Δi(U∗,W∗) _i^*:= _i( θ_U^*, θ_W^*) and N=NtrainN=N_train, the ScaleBiO objective and its gradient with respect to each data weight are ℒϕSBO=−∑i=1NpiΔi∗,∂ℒϕSBO∂wi=−Δi∗−Δ¯∗S,L_ φ^SBO=-Σ _i=1^Np_i _i^*, _ φ^SBO∂ w_i=- _i^*- ^*S, (8) where S=∑j=1Nwj S= _j=1^Nw_j is the total weight, pi=wi/Sp_i=w_i/S the normalized weight, and Δ¯∗=∑ipiΔi∗ ^*= _ip_i _i^* the weighted-average pseudo-label. The derivation is provided in Appendix A. The gradient reveals a competitive mechanism: examples are up-weighted when Δi∗>Δ¯∗(t) _i^*> ^*(t) and down-weighted otherwise. As examples with larger pseudo-labels gain relative weight, the weighted mean rises, making it harder for other examples to receive positive updates. This can produce a winner-takes-most allocation, in which a single weight dominates the total mass. Given a lower bound δ0≥0 _0≥ 0 of data weight, the following theorem is given by Du et al. (2026) (proof reproduced in Appendix B). Theorem 1 (Concentration of normalized data weights p). Assume there is a unique index K such that ΔK∗>Δi∗ _K^*> _i^* for all i≠Ki≠ K, then limt→∞pK(t)=1 _t→∞p_K(t)=1 and limt→∞pi(t)=0 _t→∞p_i(t)=0. For δ0=0 _0=0, this limit is reached in finite time. For δ0>0 _0>0, 1−pK(t)=O(t−1/3)1-p_K(t)=O(t^-1/3). Figure 1: Simulation of the weight dynamics in Theorem 1. Curves show the weights wi(t)w_i(t), with fixed pseudo-labels Δi∗ _i^* indicated in the legend. Initial weights are (0.2,0.2,0.3,0.2)(0.2,0.2,0.3,0.2), and δ0 _0 is 10−610^-6. Different from Du et al. (2026), who use weight concentration to analyze gradient signal-to-noise ratios, we examine its implications for sample discrimination. The ScaleBiO objective eventually allocates nearly all weight to one training data point. At the beginning, the meta-network can differentiate high-quality data points from low-quality data points by assigning them different weights. As training progresses, however, its outputs become less discriminative. This creates a strange phenomenon that more training leads to worsened data selection even on the training set itself. Empirical verification. Figure 1 illustrates this behavior through a numerical simulation. The weight w1w_1, associated with the largest pseudo-label, becomes dominant, while the remaining weights reach the lower bound. Notably, w4w_4 initially increases but later decreases. Between t=0.4t=0.4 and t=0.6t=0.6, the weights can separate good data with the highest pseudo-labels, but this period is transient. Some data selection can be achieved by carefully tuning early stopping (consistent with Figure 9 of Calian et al. (2026)) but it requires significant trial and error. To verify these dynamics, we construct a controlled dataset by mixing GSM8K and unsafe examples at a 2:12:1 ratio, together with a disjoint unseen set of the same composition. This way, we know exactly what data points are unsafe and should be discarded. We rank examples by predicted weights and evaluate how accurately the bottom one-third identifies unsafe examples on both seen and unseen data. Experimental details are provided in Appendix E. Figure 2 compares ScaleBiO with ℒϕSBO L_ φ^SBO, TESS with ℒϕSBO L_ φ^SBO, and TESS with ℒϕPVM L_ φ^PVM. Figure 2: Training dynamics of ScaleBiO and TESS under different meta-network objectives. With SBO loss, most predicted weights shrink toward zero as accuracy declines. In contrast, PVM loss maintains nonvanishing weight quantiles and high accuracy. Under ℒϕSBO L_ φ^SBO, the 25th percentile, median, and 75th percentile of the predicted weights progressively approach zero, consistent with the competitive suppression analyzed above. The maximum weight initially increases but later decreases. This is expected because a shared selection network typically produces smoother outputs than a one-hot vector (Ma & Ying, 2021), causing even dominant weights to decrease as most outputs are suppressed. Weight suppression accompanies declining discrimination. For ScaleBiO, seen and unseen accuracy first improve but eventually fall to around 60%60\%. TESS with the same ℒϕSBO L_ φ^SBO exhibits a similar long-term decline, although it improves immediately at the beginning of training. We attribute this early difference to the quality of the supervision signal: ScaleBiO updates the meta-network while the two LLMs are still being optimized, whereas TESS computes loss gaps from well-trained, fixed LLMs. In contrast, TESS with ℒϕPVM L_ φ^PVM maintains stable, nonvanishing weight quantiles and reaches 98.58%98.58\% seen accuracy and 97.96%97.96\% unseen accuracy. 4.2 Easy-Feature Dominance and Shortcut Learning The previous subsection links weight suppression to reduced sample discrimination. We now examine a second limitation: ScaleBiO can continue favoring easy-to-learn features after they have been substantially fitted, whereas PVM can shift learning toward harder features that explain the remaining prediction error. This distinction matters for generalization when an easy feature is predictive on the training data but unreliable on unseen data, thereby acting as a shortcut (Hermann et al., 2024; Geirhos et al., 2020). Denote the empirical covariance between random variables u and v as Cov^N(u,v):=N−1∑i=1N(ui−u¯)(vi−v¯). Cov_N(u,v):=N^-1 _i=1^N(u_i- u)(v_i- v). Using fixed pseudo-labels Δi=Δi(U∗,W∗) _i= _i( θ_U^*, θ_W^*), for two data features e and h, we denote their covariance with pseudo-labels as ce=Cov^N(Δ,e),ch=Cov^N(Δ,h).c_e= Cov_N( ,e), c_h= Cov_N( ,h). (9) We call e the easy feature and h the hard feature if |ce|>|ch||c_e|>|c_h|. For each training example xix_i, let eie_i and hih_i be its easy and hard features, both standardized to zero mean and unit standard deviation. For simplicity, we analyze a one-layer meta-network with parameters ϕ=(β0,βe,βh) φ=( _0, _e, _h) to predict weights for xix_i. It predicts wi=sϕ(xi)=aiw_i=s_ φ(x_i)=a_i, where ai=β0+βeei+βhhia_i= _0+ _ee_i+ _hh_i is the output logit. Table 1: Safe–unsafe data classification accuracy under different shortcut strengths. CovCov and ρ denote the empirical covariance and Pearson correlation coefficient between the shortcut feature and the label, respectively. Early, middle, and late stages correspond to 20%20\%, 50%50\%, and 80%80\% of meta-network training, respectively. AccS Acc_S and AccU Acc_U denote seen and unseen accuracy, and Gap=AccS−AccU Gap=Acc_S-Acc_U denotes the generalization gap. Shortcut (CovCov / ρ ) Method Early Stage Middle Stage Late Stage Avg. Gap AccSAcc_S AccUAcc_U GapGap AccSAcc_S AccUAcc_U GapGap AccSAcc_S AccUAcc_U GapGap Strong 0.17 / 0.57 TESS + ℒϕPVML_ φ^PVM 98.84 79.38 19.46 99.56 84.62 14.94 99.56 84.97 14.59 16.33 TESS + ℒϕSBOL_ φ^SBO 97.96 72.44 25.52 90.22 52.26 37.96 84.71 46.31 38.40 33.96 ScaleBiO + ℒϕSBOL_ φ^SBO 93.42 78.84 14.58 90.76 57.73 33.03 76.00 32.44 43.56 30.39 Weak 0.06 / 0.41 TESS + ℒϕPVML_ φ^PVM 98.04 91.47 6.57 98.67 93.51 5.16 98.76 93.60 5.16 5.63 TESS + ℒϕSBOL_ φ^SBO 96.53 80.71 15.82 87.82 65.78 22.04 84.00 60.53 23.47 20.44 ScaleBiO + ℒϕSBOL_ φ^SBO 91.64 82.40 9.24 90.13 71.73 18.40 75.56 55.56 20.00 15.88 To examine the learning preference for features, we compare the gap between gradients of easy and hard features in PVM and ScaleBiO training. Theorem 2 (Contrasting gradient-gap dynamics). At training step t, denote the feature gradients by Gkq(ϕt):=∂ℒϕtq/∂βk,t G_k^q( φ_t):= _ φ_t^q/∂ _k,t, where k∈e,hk∈\e,h\ and q∈SBO,PVMq∈\SBO,PVM\. Define gradient gap Dtq=(Geq(ϕt))2−(Ghq(ϕt))2D_t^q= (G_e^q( φ_t) )^2- (G_h^q( φ_t) )^2, then DtPVMD_t^PVM decreases exponentially with t while DtSBOD_t^SBO remains non-decreasing. Specifically, when initializing βe=βh=0 _e= _h=0 and β0>0 _0>0, there exists a training step T>0T>0 and λ∈(0,1)λ∈(0,1), such that ∀t<T∀ t<T, DtPVM=(1−λ)tD0PVM,DtSBO≥Dt−1SBO.D_t^PVM=(1-λ)^tD_0^PVM, D_t^SBO≥ D_t-1^SBO. (10) The proof is provided in Appendix C. This comparison explains how PVM can gradually balance the information learned from easy and hard features, while ScaleBiO continues to prefer easy features. This links the training objective to shortcut learning. When the easy feature is predictive on the training data but unreliable on unseen data, ScaleBiO’s persistent preference can reinforce reliance on this shortcut. In contrast, PVM mitigates the gradient gap between the easy feature and the hard feature. This provides a mechanism for TESS to move beyond training-specific shortcuts and improve data selection on unseen examples. Empirical verification. To test shortcut reliance, we add shortcut features to the seen training data and flip them on unseen data (Appendix E). We use the gap between seen and unseen dataset accuracy as a proxy for generalization, with smaller gaps indicating better generalization on unseen data. Table 1 shows clear differences in training dynamics. Under both shortcut strengths, TESS with ℒϕPVM L_ φ^PVM improves unseen accuracy and reduces the gap from early to late training. In contrast, ℒϕSBO L_ φ^SBO leads to lower unseen accuracy and larger gaps over training in both frameworks. These results are consistent with our analysis that SBO persistently learning more shortcut features, and support PVM as a more effective objective for learning transferable meta-networks. Moreover, all variants exhibit larger average gaps under strong shortcuts, indicating that stronger spurious correlations make generalization more difficult. 5 Experiments 5.1 LLM Safety We evaluate whether a meta-network can identify training examples that increase harmful behavior finetuning a target LLM. We use Alpaca (Taori et al., 2023; Peng et al., 2023) and Dolly (Conover et al., 2023) as candidate pools poolD_pool. The validation set from He et al. (2024) contains 100 harmful instructions paired with safe refusal responses. We finetune the target LLM on 1,000 lowest-scoring examples and evaluate it on DirectHarm4 (DH4) (Lyu et al., 2024), HarmBench (HB) (Mazeika et al., 2024), and HEx-PHI (HEx) (Qi et al., 2024). We use LlamaGuard 3 (Llama Team, 2024) as the classifier to compute attack success rate (ASR) from generated responses; higher ASR indicates more effective harmful-data selection. The target models are Llama-3-8B-Instruct, Llama-3.1-8B-Instruct (Llama Team, 2024), and Qwen2.5-7B-Instruct (Qwen et al., 2025). Implementation details are provided in Appendix D. Settings and Baselines. We consider two settings. In Training, the meta-network is trained and applied on the same candidate dataset, with train=poolD_train=D_pool. In Generalization, it is trained on Alpaca and directly applied to Dolly, or vice versa, with train∩pool=∅D_train _pool= . We compare against the random selection baseline and two groups of task-agnostic methods. GradSafe (Xie et al., 2024) and Bi-Anchoring (Bi-Anchor) (He et al., 2024) are heuristic methods that rank examples using distances relative to the validation set. SEAL (Shen et al., 2025a) and ScaleBiO (Pan et al., 2025) are meta-learning methods that learn data weights. We create two variants for each. SEALw and SBOw optimize one weight per example, whereas SEALϕ and SBOϕ train a meta-network sϕs_ φ to predict weights. Table 2: Attack success rate (ASR, %; higher is better) for harmful-data selection on Alpaca and Dolly. Best and second-best results are bold and underlined, ranked separately within two settings. Setting Training Generalization Dataset Bench Random Task-agnostic Heuristics Learnable Weighting GradSafe Bi-Anchor SEALw SBOw SEALϕ SBOϕ TESS SEALϕ SBOϕ TESS Target LLM: Llama3-8B-Instruct Alpaca DH4 25.00 28.00 49.00 26.75 38.25 12.50 12.50 36.75 8.50 6.50 35.50 HB 15.00 16.00 35.00 13.50 21.00 9.00 7.00 20.50 6.00 4.00 25.00 HEx 6.55 8.97 24.58 6.90 10.69 5.86 3.44 11.38 3.10 3.44 15.52 Dolly DH4 55.25 75.00 74.50 60.75 69.25 73.00 74.50 82.25 56.25 43.00 71.25 HB 39.25 82.00 75.00 59.50 73.00 71.25 69.50 85.50 43.50 27.50 59.50 HEx 31.38 74.14 67.59 42.76 53.44 61.03 63.44 81.72 30.69 22.07 52.41 Average 28.74 47.35 54.28 35.03 44.27 38.77 38.40 53.02 24.67 17.75 43.20 Target LLM: Llama3.1-8B-Instruct Alpaca DH4 22.50 7.50 11.00 27.75 20.50 8.50 5.50 40.25 2.25 11.00 34.25 HB 18.50 5.00 12.50 13.00 8.00 7.50 3.50 18.50 2.50 7.00 16.50 HEx 8.97 3.45 3.10 6.90 2.41 2.76 1.03 12.41 1.03 4.83 9.66 Dolly DH4 54.00 59.50 67.25 71.75 63.75 64.00 74.50 81.00 71.50 60.25 80.25 HB 51.00 60.50 50.50 65.00 58.50 55.00 60.00 80.00 69.00 50.50 81.00 HEx 29.30 33.79 40.00 38.62 33.10 33.45 42.07 60.34 40.69 28.28 71.38 Average 30.71 28.29 30.73 37.17 31.04 28.54 31.10 48.75 31.16 26.98 48.84 Target LLM: Qwen2.5-7B-Instruct Alpaca DH4 27.50 26.00 44.50 20.00 27.00 28.25 11.75 44.50 8.75 11.00 38.75 HB 11.00 10.00 24.50 9.00 8.00 17.50 3.50 23.50 3.00 2.50 16.00 HEx 13.10 12.07 24.80 6.55 8.28 9.31 6.56 24.83 7.24 6.56 18.28 Dolly DH4 50.50 66.50 60.50 49.75 65.75 60.25 59.75 86.50 48.00 62.25 84.50 HB 36.00 60.00 60.50 65.50 61.50 43.00 45.50 87.00 34.00 50.50 83.50 HEx 32.41 51.03 42.07 51.03 50.34 45.86 44.48 88.62 27.59 29.66 80.00 Average 28.42 37.60 42.81 33.64 36.81 34.03 28.59 59.16 21.43 27.08 53.51 Overall Average 29.29 37.75 42.61 35.28 37.38 33.78 32.70 53.64 25.76 23.94 48.51 Main Results. Table 2 shows the results. Under Training settings, TESS achieves the strongest performance, exceeding the strongest learning baseline SBOw by 16.27%16.27\% and the strongest heuristic Bi-Anchor by 11.04%11.04\%. Although Bi-Anchor remains stronger on Llama3-Alpaca, TESS achieves the best in 14 of the 18 comparisons. These gains stem from the optimization stability of TESS. In contrast, simply replacing individual weights with a shared meta-network does not consistently help: SEALϕ φ and SBOϕ underperform their weight-based counterparts by 1.501.50 and 4.684.68 points on average. This suggests that the training objective is critical for effective selection. Under the Generalization setting, TESS outperforms SEALϕ and SBOϕ in all 18 evaluations, surpassing SEALϕ and SBOϕ by 22.7622.76 and 24.5824.58 points in average ASR, respectively, while dropping only 5.13%5.13\% from its training performance. These results support our analysis that PVM learns more transferable selection signals rather than relying on training-specific shortcuts. Small-to-Large Model Transfer. We further evaluate whether a meta-network trained using a smaller LLM can select useful data for a larger model. We train the meta-network with Qwen2.5-0.5B-Instruct and use it to select data for fine-tuning Qwen2.5-7B-Instruct. As shown in Table 5 (Appendix F), this reduces training time by 5.655.65–6.60×6.60× while retaining 77.53%77.53\% and 83.67%83.67\% of the corresponding large-model performance under the Training and Generalization settings. This demonstrates that TESS can transfer selection signals across model scales with substantially lower training cost. This offers a promising direction for scaling data selection to larger models efficiently. Ablation Study. We ablate the training losses within the TESS framework on Qwen2.5-7B-Instruct, comparing PVM Loss (defalt), SBO Loss, KL Loss, together with Pseudo-Label Ranking, which directly ranks examples by Δ∗ ^* without training a meta-network. KL Loss (Appendix D.5) matches normalized meta-network outputs to a pseudo-label-reweighted target distribution that favors examples with higher estimated utility. Under the Training setting, TESS with PVM Loss achieves the highest ASR across all six dataset–benchmark pairs, outperforming SBO Loss and KL Loss by 21.60%21.60\% and 16.24%16.24\%, respectively. These gains stem from the more stable optimization of PVM. Notably, PVM Loss also outperforms Pseudo-Label Ranking by 18.15%18.15\%. One possible explanation is that the meta-network may implicitly denoise pseudo-labels by learning shared patterns (Liu et al., 2020; Li et al., 2020). Under the Generalization setting, PVM Loss exceeds SBO Loss and KL Loss by 24.61%24.61\% and 25.07%25.07\% on average, respectively. Moreover, its average ASR decreases by only 5.655.65 points relative to the Training setting, compared with drops of 8.67%8.67\% for SBO Loss and 14.49%14.49\% for KL Loss. The stronger unseen-data performance and smaller degradation indicate that PVM learns a more transferable meta-network. Table 3: Ablation of Different Training Losses for the Meta-network on Qwen2.5-7B-Instruct. Setting Method Alpaca Dolly Avg. DH4 HB HEx DH4 HB HEx Training TESS w/ PVM Loss 44.50 23.50 24.83 86.50 87.00 88.62 59.16 TESS w/ SBO Loss 39.50 19.50 22.06 58.25 40.50 45.52 37.56-21.6037.56_-21.60 TESS w/ KL Loss 40.75 19.00 23.86 60.75 53.50 59.66 42.92¯-16.24 42.92_-16.24 Pseudo-Label Ranking 11.25 7.50 5.86 75.75 75.00 70.69 41.01-18.1541.01_-18.15 Generalization TESS w/ PVM Loss 38.75 16.00 18.28 84.50 83.50 80.00 53.51 TESS w/ SBO Loss 12.00 9.50 7.24 59.25 44.00 41.37 28.89¯-24.61 28.89_-24.61 TESS w/ KL Loss 11.75 4.00 6.55 61.00 43.50 43.79 28.43-25.0728.43_-25.07 5.2 Targeted Instruction Tuning We evaluate if TESS trained on a small subset can generalize to the full instruction tuning corpus. GSM8K (Cobbe et al., 2021) and Codex (Chen et al., 2021) are the target tasks. We train the meta-network using a random 50K-example subset of Tulu V2 (Ivison et al., 2025) and a task-specific validation set, then apply it to select training data from the 197K full Tulu V2. Following Nayak et al. (2026), we finetune Llama-2-7B on the selected examples and report GSM8K accuracy and Codex pass@10. Implementation details are provided in Appendix D. We compare against random selection, heuristic-based methods, and learnable-weighting baselines. The heuristic methods, LESS (Xia et al., 2024) and RDS+ (Ivison et al., 2025), rank candidate examples using predefined measures of relevance to the target validation set. All four learnable baselines are trained on the same 50K subset. SEALw and SBOw are restricted to select from their training set. Table 4: Llama-2-7B performance after fine-tuning on N selected Tulu V2 examples. Method GSM8K CodeX Overall Avg. N=1K N=5K N=10K Avg. N=5K N=10K N=20K Avg. Random 13.41 15.92 16.13 15.15 29.05 27.03 27.70 27.93 21.54 LESS 18.80 22.36 22.59 21.25 25.00 26.35 28.37 26.57 23.91+2.37_+2.37 RDS+ 17.43 21.60 23.57 20.87 28.37 26.35 29.73 28.15 24.51+2.97_+2.97 SEALw 14.78 16.37 17.89 16.35 24.32 25.00 25.68 25.00 20.67-0.87_-0.87 SBOw 15.01 17.74 18.65 17.13 26.35 25.68 27.03 26.35 21.74+0.20_+0.20 SEALϕ 17.74 19.56 19.48 18.93 24.32 27.03 26.35 25.90 22.41+0.87_+0.87 SBOϕ 17.43 18.65 19.03 18.37 26.35 25.68 27.70 26.58 22.47+0.93_+0.93 Ours 20.43 25.47 25.70 23.87 29.73 28.37 29.05 29.05 26.46+4.92_+4.92 Main Results. Table 4 shows that TESS achieves the best average performance on both tasks and ranks first in five of six settings, exceeding Random by 4.924.92 points overall. TESS improves over heuristic-based selection, exceeding LESS and RDS+ in overall average by 2.55% and 1.95%, respectively. The advantage is more pronounced on GSM8K, where TESS outperforms the strongest heuristic baseline, LESS, in task-average performance by 2.62%. TESS also consistently outperforms all learnable-weighting baselines. Compared with SEALϕ φ and SBOϕ φ, which use the same 50K training subset and full selection pool, TESS improves the overall average by 4.054.05 and 3.993.99 points, respectively. This controlled comparison shows that the performance gains come from the new loss rather than merely using a shared meta-network. Regarding training efficiency, SEALw and SBOw optimize sample-specific weights and thus must be trained on the full 200K pool, whereas TESS trains on only 50K examples and generalizes to the full pool. As a result, SBOw and SEALw require 59.8 and 39.6 hours, respectively, corresponding to 3.46× and 1.95× more training time than the 13.4 hours required by TESS. SBOw also incurs higher memory usage due to jointly training multiple models, requiring 188.2 GB compared with 156.8 GB for TESS, an increase of 31.4,GB (20.03%). 6 Conclusion We introduced TESS, a transferable data-selection framework that learns a selector from sample-level pseudo-labels. We identify weight suppression and easy-feature preference of ScaleBiO objective and address them with PVM. TESS shows strong transfer across datasets, corpus scales, and model sizes. AI use statement In this work, we used generative AI tools to formulate mathematical claims, assist in the writing of proofs, provide feedback on research methodology or experiments, help develop theoretical models or conceptual frameworks, provide critical ingredients for proving mathematical claims, and assist with translation. We have not used generative AI tools to generate synthetic data sets, implement methods, support qualitative and thematic data analysis, or interpret results, and propose or refine hypotheses and clean and reformat dataset are not applicable to this work. Additionally, we used generative AI tools to edit software code, search for information, and edit a research paper to improve readability. We have not used generative AI tools to create or modify scientific figures or images, suggest experimental parameters, draft parts of a research paper, summarize or analyse existing literature, discover research topics or identify gaps, brainstorm, identify relevant literature, format references, suggest a structure for a research paper, or propose a title or keywords for a research paper. Formulate questions for surveys or interviews, creation of artifacts, and transcribe recordings of research material are not applicable to this work. We have reviewed all AI-assisted work. All AI-assisted text, mathematical results, and information-search results were manually verified and revised by the authors as needed. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Reproducibility statement We provide detailed descriptions of the models, datasets, training configurations, and evaluation protocols in Section 5.1 and Appendix D. All datasets and pretrained models used in this work are from prior work and are publicly available. We will also release our code upon acceptance to facilitate reproducibility. References Calian et al. (2026) Dan Andrei Calian, Greg Farquhar, Iurii Kemaev, Luisa Zintgraf, Matteo Hessel, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeff Dean, et al. Datarater: Meta-learned dataset curation. Advances in Neural Information Processing Systems, 38:20060–20092, 2026. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. doi: 10.48550/arXiv.2107.03374. URL https://arxiv.org/abs/2107.03374. Chen et al. (2024) Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, and Heng Huang. Your vision-language model itself is a strong filter: Towards high-quality instruction tuning with data selection. In Findings of the Association for Computational Linguistics: ACL 2024, p. 4156–4172, 2024. doi: 10.18653/v1/2024.findings-acl.246. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. doi: 10.48550/arXiv.2110.14168. URL https://arxiv.org/abs/2110.14168. Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://w.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm. Deng et al. (2026) Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving llm alignment via preference data selection. Advances in Neural Information Processing Systems, 38:161259–161285, 2026. Du et al. (2026) Zilin Du, Junqi Zhao, and Boyang Albert Li. On the difficulty of learning a meta-network for training data selection. arXiv preprint arXiv:2606.00571, 2026. Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. He et al. (2024) Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099, 2024. Hermann et al. (2024) Katherine Hermann, Hossein Mobahi, Thomas Fel, and Michael Mozer. On the foundations of shortcut learning. In International Conference on Learning Representations, volume 2024, p. 43832–43868, 2024. Ivison et al. (2025) Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.01807, 2025. Ji et al. (2023) Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704, 2023. Kumar et al. (2024) Sushant Kumar, Sumit Datta, Vishakha Singh, Sanjay Kumar Singh, and Ritesh Sharma. Opportunities and challenges in data-centric ai. IEEE Access, 12:33173–33189, 2024. Langlais et al. (2026) Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett, Carlos Hinostroza, Mattia Nee, Eliot Jones, Irène Girard, David Mach, Anastasia Stasenko, and Ivan Yamshchikov. Common corpus: The largest collection of ethical data for llm pre-training. In International Conference on Learning Representations, volume 2026, p. 25827–25860, 2026. Lee et al. (2024) Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-based data selection for large vision-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 5060–5080, 2024. doi: 10.18653/v1/2024.emnlp-main.291. Li et al. (2024a) Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200–14282, 2024a. Li et al. (2024b) Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7602–7635, 2024b. Li et al. (2020) Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and statistics, p. 4313–4324. PMLR, 2020. Li et al. (2026) Xiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, Yiyao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, and Dayiheng Liu. Unified data selection for llm reasoning. arXiv preprint arXiv:2605.22389, 2026. Liu et al. (2020) Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331–20342, 2020. Llama Team (2024) AI @ Meta Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. Lyu et al. (2024) Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. Advances in Neural Information Processing Systems, 37:118603–118631, 2024. Ma & Ying (2021) Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems, 34:16805–16817, 2021. Mazeika et al. (2024) Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024. Min et al. (2026) Guanghui Min, Tianhao Huang, Ke Wan, and Chen Chen. Gist: Targeted data selection for instruction tuning via coupled optimization geometry. arXiv preprint arXiv:2602.18584, 2026. Mindermann et al. (2022) Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, p. 15630–15649. PMLR, 2022. Nayak et al. (2026) Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, and David Alvarez-Melis. A critical look at targeted instruction selection: Disentangling what matters (and what doesn’t). In International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2602.14696. Pan et al. (2024) Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 37:57018–57049, 2024. Pan et al. (2025) Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, and Tong Zhang. Scalebio: Scalable bilevel optimization for llm data reweighting. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31959–31982, 2025. Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. Qi et al. (2024) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations, volume 2024, p. 30988–31043, 2024. Qiu et al. (2024) GuanWen Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Feature learning in the presence of spurious correlations. In Forty-first International Conference on Machine Learning, 2024. Qwen et al. (2025) Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, January 2025. URL http://arxiv.org/abs/2412.15115. arXiv:2412.15115 [cs]. Ren et al. (2018) Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, p. 4334–4343. PMLR, 2018. Scimeca et al. (2021) Luca Scimeca, Seong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut cues will dnns choose? a study from the parameter-space perspective. arXiv preprint arXiv:2110.03095, 2021. Shah et al. (2020) Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In Advances in Neural Information Processing Systems, volume 33, 2020. Shen & Chen (2023) Han Shen and Tianyi Chen. On penalty-based bilevel gradient descent method. In International conference on machine learning, p. 30992–31015. PMLR, 2023. Shen et al. (2025a) Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection. In International Conference on Learning Representations, volume 2025, p. 31243–31264, 2025a. Shen et al. (2025b) Han Shen, Zhuoran Yang, and Tianyi Chen. Principled penalty-based methods for bilevel reinforcement learning and rlhf. Journal of Machine Learning Research, 26(114):1–49, 2025b. Shu et al. (2019) Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019. Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. Wang et al. (2020) Yulin Wang, Jiayi Guo, Shiji Song, and Gao Huang. Meta-semi: A meta-learning approach for semi-supervised learning. arXiv preprint arXiv:2007.02394, 2020. Weber et al. (2024) Maurice Weber, Daniel Y Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:116462–116492, 2024. Wen et al. (2026) Bingbing Wen, Sirajul Salekin, Feiyang Kang, Bill Howe, Lucy Lu Wang, Javier Movellan, and Manjot Bilkhu. MixAtlas: Uncertainty-aware data mixture optimization for multimodal llm midtraining, 2026. Whang et al. (2023) Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. Data collection and quality challenges in deep learning: a data-centric ai perspective: Se whang et al. The VLDB Journal, 32(4):791–813, 2023. Xia et al. (2024) Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning, 2024. URL https://arxiv.org/abs/2402.04333. Xiao et al. (2023) Quan Xiao, Songtao Lu, and Tianyi Chen. A generalized alternating method for bilevel learning under the polyak–łojasiewicz condition. arXiv preprint arXiv:2306.02422, 2023. Xie et al. (2023a) Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems, volume 36, 2023a. doi: 10.52202/075280-3059. Xie et al. (2023b) Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S. Liang. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems, volume 36, p. 34201–34227, 2023b. doi: 10.52202/075280-1482. Xie et al. (2025) Wanyun Xie, Francesco Tonin, and Volkan Cevher. Chameleon: A flexible data-mixing framework for language model pretraining and finetuning, 2025. Xie et al. (2024) Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. Gradsafe: Detecting jailbreak prompts for llms via safety-critical gradient analysis. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 507–518, 2024. Yang et al. (2026) Kailai Yang, Xiao Liu, Lei Ji, Hao Li, Xiao Liang, Zhiwei Liu, Yeyun Gong, Peng Cheng, and Mao Yang. Data mixing agent: Learning to re-weight domains for continual pre-training, 2026. Yu et al. (2025) Yang Yu, Kai Han, Hang Zhou, Yehui Tang, Kaiqi Huang, Yunhe Wang, and Dacheng Tao. Llm data selection and utilization via dynamic bi-level optimization. arXiv preprint arXiv:2507.16178, 2025. Zha et al. (2023) Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, and Xia Hu. Data-centric ai: Perspectives and challenges. arXiv preprint arXiv:2301.04819, 2023. Zha et al. (2025) Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. ACM Computing Surveys, 57(5):1–42, 2025. Zhou et al. (2025) Xuanhe Zhou, Junxuan He, Wei Zhou, Haodong Chen, Zirui Tang, Haoyu Zhao, Xin Tong, Guoliang Li, Youmin Chen, Jun Zhou, et al. A survey of LLM × DATA. arXiv preprint arXiv:2505.18458, 2025. Appendix Contents Appendix A Gradient of the ScaleBiO meta-network Objective Notation. Let train=xii=1ND_train=\x_i\_i=1^N, where N=NtrainN=N_train is the number of training samples. Let ℓ(,xi) ( θ;x_i) denote the token-averaged cross-entropy loss of an LLM parameterized by θ on sample xix_i. For the selection network sϕs_ φ, define wi:=sϕ(xi)>0,S:=∑j=1Nwj,pi:=wiS.w_i:=s_ φ(x_i)>0, S:= _j=1^Nw_j, p_i:= w_iS. The two LLMs are held fixed at U∗ θ_U^* and W∗ θ_W^*. Define the per-sample pseudo-labels and their weighted average as Δi∗ _i^* :=ℓ(U∗,xi)−ℓ(W∗,xi), := ( θ_U^*;x_i)- ( θ_W^*;x_i), Δ¯∗ ^* :=∑j=1NpjΔj∗=∑j=1NwjΔj∗S. := _j=1^Np_j _j^*= _j=1^Nw_j _j^*S. The pseudo-labels Δi∗ _i^* are constant with respect to ϕ φ and the data weights, whereas Δ¯∗ ^* depends on the weights through pjp_j. meta-network objective. For the fixed LLMs, the objective in Eq. 6 becomes ℒϕSBO _ φ^SBO =∑j=1NwjS[ℓ(W∗,xj)−ℓ(U∗,xj)] = _j=1^N w_jS [ ( θ_W^*;x_j)- ( θ_U^*;x_j) ] =−∑j=1NpjΔj∗=−∑j=1NwjΔj∗S=−Δ¯∗. =- _j=1^Np_j _j^*=- _j=1^Nw_j _j^*S=- ^*. Gradient with respect to an individual weight. For a fixed index i, we differentiate with respect to wiw_i while holding all other weights fixed. Since the pseudo-labels are constant, the derivatives of the numerator and denominator are ∂wi(∑j=1NwjΔj∗) ∂ w_i ( _j=1^Nw_j _j^* ) =∂wi(wiΔi∗+∑j≠iwjΔj∗)=Δi∗, = ∂ w_i (w_i _i^*+ _j≠ iw_j _j^* )= _i^*, ∂S∂wi ∂ S∂ w_i =∂wi(wi+∑j≠iwj)=1. = ∂ w_i (w_i+ _j≠ iw_j )=1. Applying the quotient rule and substituting these derivatives gives ∂ℒϕSBO∂wi _ φ^SBO∂ w_i =−S∂wi(∑j=1NwjΔj∗)−(∑j=1NwjΔj∗)∂S∂wiS2 =- S\, ∂ w_i ( _j=1^Nw_j _j^* )- ( _j=1^Nw_j _j^* ) ∂ S∂ w_iS^2 =−SΔi∗−∑j=1NwjΔj∗S2 =- S _i^*- _j=1^Nw_j _j^*S^2 =−1S(Δi∗−∑j=1NwjΔj∗S) =- 1S ( _i^*- _j=1^Nw_j _j^*S ) =−Δi∗−Δ¯∗S. =- _i^*- ^*S. (11) Appendix B Proof of Theorem 1 Notation and assumptions. Let N=Ntrain≥2N=N_train≥ 2 and fix the pseudo-labels Δi∗:=Δi(U∗,W∗) _i^*:= _i( θ_U^*, θ_W^*). We treat the raw weights =(w1,…,wN)w=(w_1,…,w_N) as independent variables constrained only by wi≥δ0≥0w_i≥ _0≥ 0, with wi(0)>δ0w_i(0)> _0 for every i. In particular, no upper bound is imposed on the weights. Assume there is a unique index K such that ΔK∗>Δi∗ _K^*> _i^* for all i≠Ki≠ K. For any feasible w with positive total weight, define S()=∑j=1Nwj,pi()=wiS(),Δ¯∗()=∑i=1Npi()Δi∗.S(w)= _j=1^Nw_j, p_i(w)= w_iS(w), ^*(w)= _i=1^Np_i(w) _i^*. Writing the ScaleBiO objective as a function of the independent weights gives ℒSBO()=−Δ¯∗(),∂ℒSBO∂wi=−Δi∗−Δ¯∗()S().L^SBO(w)=- ^*(w), ^SBO∂ w_i=- _i^*- ^*(w)S(w). Gradient descent and continuous-time dynamics. To establish the theorem, we first specify the weight updates and their continuous-time counterpart. Let n index discrete iterations and let ηw>0 _w>0 be the learning rate. Projected gradient descent under the constraint wi≥δ0w_i≥ _0 yields wi(n+1)=maxδ0,wi(n)+ηwS(n)(Δi∗−Δ¯∗(n)),w_i^(n+1)= \ _0,\,w_i^(n)+ _wS^(n) ( _i^*- ^*(n) ) \, (12) where S(n)=S((n))S^(n)=S(w^(n)) and Δ¯∗(n)=Δ¯∗((n)) ^*(n)= ^*(w^(n)). The projection prevents an update from taking a weight below δ0 _0. The corresponding continuous-time dynamics, with the learning rate absorbed into time through t=nηwt=n _w, are described by the projected gradient flow w˙i(t)=Δi∗−Δ¯∗(t)S(t),wi(t)>δ0,max0,Δi∗−Δ¯∗(t)S(t),wi(t)=δ0, w_i(t)= cases _i^*- ^*(t)S(t),&w_i(t)> _0,\\[6.0pt] \0,\, _i^*- ^*(t)S(t) \,&w_i(t)= _0, cases (13) where S(t)=S((t))S(t)=S(w(t)), pi(t)=pi((t))p_i(t)=p_i(w(t)), and Δ¯∗(t)=Δ¯∗((t)) ^*(t)= ^*(w(t)). At the boundary, negative directions are suppressed while positive directions remain admissible. The proof below establishes the concentration result for this continuous-time flow. Proof. Consider an absolutely continuous solution of Eq. 13. All differential identities and inequalities below hold almost everywhere, including across boundary transitions. Define the pseudo-label range RΔ:=ΔK∗−miniΔi∗>0R_ := _K^*- _i _i^*>0 and the set of coordinates following the unprojected direction: ℐ(t):=i:wi(t)>δ0orΔi∗>Δ¯∗(t).I(t):= \i:w_i(t)> _0\ or\ _i^*> ^*(t) \. Then w˙i(t)=(Δi∗−Δ¯∗(t))/S(t),i∈ℐ(t),0,i∉ℐ(t). w_i(t)= cases( _i^*- ^*(t))/S(t),&i (t),\\ 0,&i (t). cases 1. Positivity and growth of the normalization. Since the weights are nonnegative, Δ¯∗(t) ^*(t) is a convex combination of the pseudo-labels. Thus Δ¯∗(t)≤ΔK∗ ^*(t)≤ _K^* and w˙K(t)≥0 w_K(t)≥ 0, which implies S(t)≥wK(t)≥wK(0)>0.S(t)≥ w_K(t)≥ w_K(0)>0. (14) In particular, wK(t)>δ0w_K(t)> _0, so K∈ℐ(t)K (t) for all t. Moreover, |Δi∗−Δ¯∗(t)|≤RΔ| _i^*- ^*(t)|≤ R_ . Hence |w˙i(t)|≤RΔ/wK(0)| w_i(t)|≤ R_ /w_K(0), excluding finite-time blow-up. Summing the coordinate derivatives gives dtS(t)2 ddtS(t)^2 =2S(t)∑iw˙i(t) =2S(t) _i w_i(t) =2∑i∈ℐ(t)(Δi∗−Δ¯∗(t))≤2NRΔ. =2 _i (t) ( _i^*- ^*(t) )≤ 2NR_ . (15) Therefore, S(t)2≤S(0)2+2NRΔt.S(t)^2≤ S(0)^2+2NR_ t. (16) By comparison with the reciprocals of this upper bound and its square root, respectively, ∫0∞dtS(t)2=∞,∫0∞dtS(t)=∞. _0^∞ dtS(t)^2=∞, _0^∞ dtS(t)=∞. (17) 2. The weighted mean converges to the largest pseudo-label. Differentiating the weighted mean yields dΔ¯∗(t)dt d ^*(t)dt =1S(t)∑i(Δi∗−Δ¯∗(t))w˙i(t) = 1S(t) _i ( _i^*- ^*(t) ) w_i(t) =1S(t)2∑i∈ℐ(t)(Δi∗−Δ¯∗(t))2≥0. = 1S(t)^2 _i (t) ( _i^*- ^*(t) )^2≥ 0. (18) Thus Δ¯∗(t) ^*(t) is nondecreasing and bounded above by ΔK∗ _K^*. Let V(t)=ΔK∗−Δ¯∗(t)≥0V(t)= _K^*- ^*(t)≥ 0. Since K∈ℐ(t)K (t), V˙(t)≤−V(t)2S(t)2. V(t)≤- V(t)^2S(t)^2. (19) If V reaches zero, it remains zero by the monotonicity of Δ¯∗ ^*. Otherwise, integrating dt(1/V(t))≥1/S(t)2 ddt(1/V(t))≥ 1/S(t)^2 gives 1V(t)≥1V(0)+∫0tdsS(s)2. 1V(t)≥ 1V(0)+ _0^t dsS(s)^2. The first integral in Eq. 17 diverges, so V(t)→0V(t)→ 0. Consequently, Δ¯∗(t)⟶ΔK∗. ^*(t) _K^*. (20) 3. The normalized weights concentrate on the unique maximizer. For i≠Ki≠ K, define di:=ΔK∗−Δi∗>0,dmin:=mini≠Kdi>0.d_i:= _K^*- _i^*>0, d_ := _i≠ Kd_i>0. Then V(t)=∑i≠Kpi(t)di≥dmin(1−pK(t)).V(t)= _i≠ Kp_i(t)d_i≥ d_ (1-p_K(t) ). Since V(t)→0V(t)→ 0, we have pK(t)→1p_K(t)→ 1. For every i≠Ki≠ K, 0≤pi(t)≤1−pK(t)0≤ p_i(t)≤ 1-p_K(t), and hence pi(t)→0p_i(t)→ 0. 4. Every nonmaximal weight reaches the floor in finite time. Fix i≠Ki≠ K. By Eq. 20, there exists a finite TiT_i such that Δ¯∗(t)−Δi∗≥di2,t≥Ti. ^*(t)- _i^*≥ d_i2, t≥ T_i. Whenever wi(t)>δ0w_i(t)> _0 after TiT_i, w˙i(t)≤−di2S(t). w_i(t)≤- d_i2S(t). (21) If wiw_i never reached the floor after TiT_i, integration would give wi(t)≤wi(Ti)−di2∫TitdsS(s).w_i(t)≤ w_i(T_i)- d_i2 _T_i^t dsS(s). The second integral in Eq. 17 diverges, so the right-hand side would eventually fall below δ0 _0, a contradiction. Thus wiw_i reaches the floor in finite time. Once at the floor at or after TiT_i, its unprojected direction remains negative, so the projection keeps it there. Since there are finitely many nonmaximal coordinates, there exists a finite T0T_0 such that wi(t)=δ0for all i≠K and t≥T0.w_i(t)= _0 all i≠ K and t≥ T_0. (22) For δ0=0 _0=0, only wK(t)w_K(t) remains positive after T0T_0. Therefore, pK(t)=1p_K(t)=1 and pi(t)=0p_i(t)=0 for every i≠Ki≠ K and t≥T0t≥ T_0, proving finite-time convergence. 5. Convergence rate for a positive floor. Suppose δ0>0 _0>0 and define H:=(N−1)δ0>0,C:=δ0∑i≠Kdi>0.H:=(N-1) _0>0, C:= _0 _i≠ Kd_i>0. For t≥T0t≥ T_0, Eq. 22 gives S(t)=wK(t)+HS(t)=w_K(t)+H and ΔK∗−Δ¯∗(t)=δ0∑i≠KdiS(t)=CS(t). _K^*- ^*(t)= _0 _i≠ Kd_iS(t)= CS(t). Only wKw_K changes after T0T_0, so S˙(t)=w˙K(t)=CS(t)2. S(t)= w_K(t)= CS(t)^2. Multiplying by 3S(t)23S(t)^2 and integrating yields S(t)3=S(T0)3+3C(t−T0).S(t)^3=S(T_0)^3+3C(t-T_0). (23) Consequently, 1−pK(t) 1-p_K(t) =HS(t) = HS(t) =H[S(T0)3+3C(t−T0)]1/3=O(t−1/3). = H [S(T_0)^3+3C(t-T_0) ]^1/3=O(t^-1/3). (24) This completes the proof. ∎ Appendix C Proof of Theorem 2 Notation and comparison condition. Denote the empirical covariance between random variables u and v as Cov^N(u,v):=N−1∑i=1N(ui−u¯)(vi−v¯). Cov_N(u,v):=N^-1 _i=1^N(u_i- u)(v_i- v). Further, denote each training data point as i=(1,ei,hi)⊤ x_i=(1,e_i,h_i) . The features eie_i and hih_i are standardized to zero mean and unit standard deviation. The pseudo-labels Δi=Δi(U∗,W∗) _i= _i( θ_U^*, θ_W^*) are fixed. For compactness, write xi(e)=eix_i^(e)=e_i and xi(h)=hix_i^(h)=h_i, we have ∑ixi(k)=0,1N∑i(xi(k))2=1,k∈e,h. _ix_i^(k)=0, 1N _i (x_i^(k) )^2=1, k∈\e,h\. (25) We call e the easy feature and h the hard feature because e has stronger covariance with the target label than h, ce=Cov^N(Δ,e),ch=Cov^N(Δ,h),|ce|>|ch|.c_e= Cov_N( ,e), c_h= Cov_N( ,h), |c_e|>|c_h|. (26) As each feature is centered, for any vector v∈ℝNv ^N, Cov^N(v,x(k)) Cov_N(v,x^(k)) =1N∑i(vi−v¯)xi(k) = 1N _i(v_i- v)x_i^(k) =1N∑ivixi(k)−v¯N∑ixi(k) = 1N _iv_ix_i^(k)- vN _ix_i^(k) =1N∑ivixi(k),v¯=1N∑ivi. = 1N _iv_ix_i^(k), v= 1N _iv_i. (27) Taking v=Δ∗v= ^* gives ck=Cov^N(Δ,x(k))=1N∑iΔixi(k),∑iΔixi(k)=Nck,k∈e,h.c_k= Cov_N( ,x^(k))= 1N _i _ix_i^(k), _i _ix_i^(k)=Nc_k, k∈\e,h\. (28) We have a meta-network parameterized by ϕt=(β0,t,βe,t,βh,t) φ_t=( _0,t, _e,t, _h,t). For each training data point i x_i and training step t, the output wi,tw_i,t of the meta-network is computed in the following way. ai,t=t⊤i=β0,t+βe,tei+βh,thi,wi,t=sϕt(xi)=ai,t.a_i,t= β_t x_i= _0,t+ _e,te_i+ _h,th_i, w_i,t=s_ φ_t(x_i)=a_i,t. (29) Since the weights of data points are expected as non-negative, we assume that the initialization of parameters ϕ0 φ_0 satisfies βe,0=βh,0=0 _e,0= _h,0=0 and β0,0>0 _0,0>0 ensuring all wi,0>0w_i,0>0. Then there exists a training step T>0T>0, such that ∀t<T∀ t<T, wi,t>0w_i,t>0 for all i. We only consider the training steps t<Tt<T. Recall the two objectives: ℒSBO(ϕtSBO)=−∑iwi,tSBOΔiSt,ℒPVM(ϕtPVM)=1N∑i(wi,tPVM−Δi)2, ^SBO( φ_t^SBO)=- _iw_i,t^SBO _iS_t, ^PVM( φ_t^PVM)= 1N _i(w_i,t^PVM- _i)^2, (30) where St=∑iwi,tSBO>0S_t= _iw_i,t^SBO>0 and Δ¯t=∑iwi,tSBOΔi/St _t= _iw_i,t^SBO _i/S_t. We write Gkq(ϕtq)=∂ℒq(ϕ)/∂βk|ϕ=ϕtqG_k^q( φ_t^q)= . ^q( φ)/∂ _k |_ φ= φ_t^q for q∈SBO,PVMq∈\SBO,PVM\ and k∈e,hk∈\e,h\. Proof. We first compare the feature gradients of ScaleBiO. 1. ScaleBiO shows a non-decreasing gap between easy and hard features. We first derive the ScaleBiO gradient with respect to parameters. Since the pseudo-labels are fixed, the quotient rule gives GkSBO(ϕtSBO) G_k^SBO( φ_t^SBO) =−1St∑iΔi∂wi,tSBO∂βk,tSBO+∑jwj,tSBOΔj∗St2∑i∂wi,tSBO∂βk,tSBO =- 1S_t _i _i ∂ w_i,t^SBO∂ _k,t^SBO+ _jw_j,t^SBO _j^*S_t^2 _i ∂ w_i,t^SBO∂ _k,t^SBO =−1St∑i(Δi−Δ¯t)∂wi,tSBO∂βk,tSBO. =- 1S_t _i( _i- _t) ∂ w_i,t^SBO∂ _k,t^SBO. (31) The meta-network definition implies ∂wi,tSBO∂βk,tSBO=xi(k). ∂ w_i,t^SBO∂ _k,t^SBO=x_i^(k). (32) Substituting this into Eq. equation 31 yields GkSBO(ϕtSBO) G_k^SBO( φ_t^SBO) =−1St∑i(Δi−Δ¯t)xi(k) =- 1S_t _i( _i- _t)x_i^(k) =−1St[∑iΔixi(k)−Δ¯t∑ixi(k)] =- 1S_t [ _i _ix_i^(k)- _t _ix_i^(k) ] =−1St∑iΔixi(k) =- 1S_t _i _ix_i^(k) =−NStck. =- NS_tc_k. (33) The third equality uses ∑ixi(k)=0 _ix_i^(k)=0. The final equality uses ∑iΔixi(k)=Nck _i _ix_i^(k)=Nc_k from Eq. 28. Since the positive factor N/StN/S_t is shared by both features, (GeSBO(ϕtSBO))2−(GhSBO(ϕtSBO))2=(NSt)2(ce2−ch2)>0.(G_e^SBO( φ_t^SBO))^2-(G_h^SBO( φ_t^SBO))^2= ( NS_t )^2(c_e^2-c_h^2)>0. (34) Next, we show that StS_t is non-increasing, St S_t =∑iwi,tSBO = _iw_i,t^SBO =Nβ0,tSBO+βe,tSBO∑iei+βh,tSBO∑ihi =N _0,t^SBO+ _e,t^SBO _ie_i+ _h,t^SBO _ih_i =Nβ0,tSBO. =N _0,t^SBO. (35) Define zt=ceβe,tSBO+chβh,tSBO.z_t=c_e _e,t^SBO+c_h _h,t^SBO. (36) Using Eq. equation 33, the gradient descent updates give βk,t+1SBO=βk,tSBO+ηNStck,k∈e,h. _k,t+1^SBO= _k,t^SBO+η NS_tc_k, k∈\e,h\. (37) Consequently, zt+1=zt+ηNSt(ce2+ch2).z_t+1=z_t+η NS_t(c_e^2+c_h^2). (38) Since z0=0z_0=0 and St>0S_t>0, it follows that zt≥0z_t≥ 0. To determine the bias update, write Δavg=N−1∑iΔi _avg=N^-1 _i _i. Eq. 35 and Eq. 28 imply ∑iwi,tSBOΔi _iw_i,t^SBO _i =β0,tSBO∑iΔi+Nzt = _0,t^SBO _i _i+Nz_t =StΔavg+Nzt. =S_t _avg+Nz_t. (39) Therefore, ℒSBO(ϕtSBO)=−Δavg−NztSt. ^SBO( φ_t^SBO)=- _avg- Nz_tS_t. (40) Since ztz_t does not depend on the bias coordinate and ∂St/∂btSBO=N∂ S_t/∂ b_t^SBO=N, the bias gradient is dℒSBOdβ0,t=N2St2zt≥0. dL^SBOd _0,t= N^2S_t^2z_t≥ 0. (41) Thus the bias update satisfies β0,t+1SBO=β0,tSBO−ηN2St2zt≤β0,tSBO. _0,t+1^SBO= _0,t^SBO-η N^2S_t^2z_t≤ _0,t^SBO. (42) Combining this with Eq. equation 35 yields St+1=St−η3N3St2zt≤St.S_t+1=S_t-η 3N^3S_t^2z_t≤ S_t. (43) Finally, Eq. equation 34 gives DtSBO D_t^SBO =(NSt)2(ce2−ch2) = ( NS_t )^2(c_e^2-c_h^2) =(NSt−1)2(ce2−ch2) = ( NS_t-1 )^2(c_e^2-c_h^2) ≥Dt−1SBO. ≥ D_t-1^SBO. (44) Thus, ScaleBiO continues to learn easy features than hard features, and their gradient gap does not decrease. Next we show the exponentially decreasing of gradient gap with PVM. 2. PVM exhibits a exponentially decreasing gradient gap. We now analyze the PVM objective under the same fixed affine surrogate. Define the residual ri,t=wi,tPVM−Δi. r_i,t=w_i,t^PVM- _i. (45) Since the pseudo-labels are fixed, the feature gradients are GkPVM(ϕtPVM) G_k^PVM( φ_t^PVM) =2N∑iri,t∂wi,tPVM∂βk,tPVM = 2N _ir_i,t ∂ w_i,t^PVM∂ _k,t^PVM =2N∑iri,txi(k),k∈e,h. = 2N _ir_i,tx_i^(k), k∈\e,h\. (46) To include the bias update, write The full gradient is therefore xi=(1,ei,hi)⊤,gt=∇ϕℒPVM(ϕtPVM).x_i=(1,e_i,h_i) , g_t= _ φL^PVM( φ_t^PVM). (47) The full gradient is therefore gt=2N∑iri,txi.g_t= 2N _ir_i,tx_i. (48) Each network output is, wi,tPVM=xi⊤ϕtPVM. w_i,t^PVM=x_i φ_t^PVM. (49) Combining this with the gradient descent update ϕt+1PVM=ϕtPVM−ηgt φ_t+1^PVM= φ_t^PVM-η g_t gives ri,t+1 r_i,t+1 =ri,t+xi⊤(ϕt+1PVM−ϕtPVM) =r_i,t+x_i ( φ_t+1^PVM- φ_t^PVM ) =ri,t−ηxi⊤gt. =r_i,t-η x_i g_t. (50) Substituting Eq. equation 50 into Eq. equation 48 yields gt+1 g_t+1 =2N∑i(ri,t−ηxi⊤gt)xi = 2N _i (r_i,t-η x_i g_t )x_i =gt−2η(1N∑ixixi⊤)gt =g_t-2η ( 1N _ix_ix_i )g_t =(I−2ηQ)gt, =(I-2η Q)g_t, (51) where Q=1N∑ixixi⊤. Q= 1N _ix_ix_i . (52) Because the features are centered and have unit variance, Q=(10001ρ0ρ1),ρ=1N∑ieihi.Q= pmatrix1&0&0\\ 0&1&ρ\\ 0&ρ&1 pmatrix, ρ= 1N _ie_ih_i. (53) For brevity, write Gk,tPVM=GkPVM(ϕtPVM)G_k,t^PVM=G_k^PVM( φ_t^PVM). Equations equation 51 and equation 53 imply Ge,t+1PVM G_e,t+1^PVM =(1−2η)Ge,tPVM−2ηρGh,tPVM, =(1-2η)G_e,t^PVM-2ηρ G_h,t^PVM, Gh,t+1PVM G_h,t+1^PVM =(1−2η)Gh,tPVM−2ηρGe,tPVM. =(1-2η)G_h,t^PVM-2ηρ G_e,t^PVM. (54) Consequently, the squared-gradient gap satisfies Dt+1PVM D_t+1^PVM =[(1−2η)Ge,tPVM−2ηρGh,tPVM]2 = [(1-2η)G_e,t^PVM-2ηρ G_h,t^PVM ]^2 −[(1−2η)Gh,tPVM−2ηρGe,tPVM]2 - [(1-2η)G_h,t^PVM-2ηρ G_e,t^PVM ]^2 =[(1−2η)2−4η2ρ2][(Ge,tPVM)2−(Gh,tPVM)2] = [(1-2η)^2-4η^2ρ^2 ] [(G_e,t^PVM)^2-(G_h,t^PVM)^2 ] =[(1−2η)2−4η2ρ2]DtPVM. = [(1-2η)^2-4η^2ρ^2 ]D_t^PVM. (55) The cross terms cancel in the third equality. Choose the step size such that: 0<η<12+2|ρ|. 0<η< 12+2|ρ|. (56) Then 0<(1−2η)2−4η2ρ2=[1−2η(1+ρ)][1−2η(1−ρ)]<1. 0<(1-2η)^2-4η^2ρ^2=[1-2η(1+ρ)][1-2η(1-ρ)]<1. (57) Define λ=1−[(1−2η)2−4η2ρ2]=4η−(1−ρ2)4η2,λ∈(0,1)λ=1- [(1-2η)^2-4η^2ρ^2 ]=4η-(1-ρ^2)4η^2, λ∈(0,1) (58) Equation equation 55 therefore becomes Dt+1PVM=(1−λ)DtPVM. D_t+1^PVM=(1-λ)D_t^PVM. (59) Iterating this identity gives DtPVM=(1−λ)tD0PVM.D_t^PVM=(1-λ)^tD_0^PVM. (60) Thus PVM exponentially contracts the absolute squared-gradient gap, while Eq. equation 44 shows that the ScaleBiO gap remains no lower than its initial value throughout the stated comparison interval. ∎ Appendix D Experiment Details D.1 Datasets For LLM safety, we use Alpaca and Dolly as training datasets and evaluate on DirectHarm4, HarmBench, and HEx-PHI: • Alpaca (Taori et al., 2023; Peng et al., 2023) contains 52,002 instruction–response pairs covering diverse instruction-following tasks. • Dolly (Conover et al., 2023) contains 15,011 human-written instruction–response pairs covering tasks such as question answering, summarization, and creative writing. • DirectHarm4 (DH4) (Lyu et al., 2024) contains 400 harmful instructions, with 100 examples in each of four categories: malware, drugs, phishing, and disinformation. • HarmBench (HB) (Mazeika et al., 2024) evaluates harmful behavior generation. Its text-only portion contains 400 behaviors across standard, contextual, and copyright-related categories. • HEx-PHI (HEx) (Qi et al., 2024) evaluates responses to instructions spanning prohibited-use categories. Its original release contains 330 instructions, with 30 examples in each of 11 categories: illegal activity, child abuse content, hate/harassment/violence, malware, physical harm, economic harm, fraud/deception, adult content, political campaigning, privacy violation, and tailored financial advice. For targeted instruction tuning, we use Tulu V2 as the candidate instruction corpus and evaluate data selection for GSM8K and CodeX: • Tulu V2 (Ivison et al., 2025) is a multi-source instruction-tuning corpus; we use the approximately 200K-example version preprocessed by Nayak et al. (2026), which excludes examples with no response tokens within the first 2,048 tokens. We train the meta-network on a random 50K subset and apply it to the full candidate pool. • GSM8K (Cobbe et al., 2021) contains grade-school math problems with step-by-step solutions, comprising 7,473 training examples and 1,319 test examples. We hold out 1,000 examples from the training split as a shared validation set for all methods. • CodeX (Chen et al., 2021) contains 164 Python function-completion problems with unit tests for evaluating functional correctness. Following Ivison et al. (2025), we use 16 examples as the query set and the remaining 148 as the test set. D.2 Implementation Details meta-network learning. We train the two LLMs, U θ_U and W θ_W, using LISA (Pan et al., 2024), with learning rates in 5×10−5,10−4\5× 10^-5,10^-4\ for up to three epochs. We tune the training–validation trade-off coefficient α over 0.01,0.1,1.0\0.01,0.1,1.0\. The meta-network is a three-layer MLP with a sigmoid output, optimized using AdamW with a learning rate of 10−310^-3. Downstream fine-tuning for LLM safety. We apply LoRA to the query and key projections, WqW_q and WkW_k, in every Transformer layer, with rank r=8r=8 and αLoRA=8 _LoRA=8. We train for three epochs with a batch size of 32, a learning rate of 10−410^-4, and a cosine learning rate schedule with a warmup ratio of 0.1. Downstream fine-tuning for targeted instruction tuning. We fully finetune Llama-2-7B on the selected examples for two epochs using AdamW with a learning rate of 2×10−52× 10^-5 and a linear learning rate schedule. The maximum sequence length is 2,048 tokens. D.3 Input Features to the Meta-Network Inspired by Du et al. (2026), we construct sample-level features in both representation and gradient spaces. The representation embedding is the final-layer hidden state of the last token, and the gradient embedding is derived from the per-sample loss gradient. In each space, we compute the distances to each of the three nearest validation examples and to the mean validation embedding. We concatenate these eight distances with the sample’s loss and gradient norm to obtain a 10-dimensional feature vector. For LLM safety, we append a binary feature indicating whether the sample contains numbers or predefined keywords such as list, convert, and calculate. This feature is motivated by findings that some benign list-format and mathematical examples can degrade safety alignment after fine-tuning (Chen et al., 2024). D.4 Evaluation Details LLM safety. For DH4, HB, and HEx, we generate model responses using greedy decoding (temperature 0). We classify each response as safe or unsafe using Llama Guard, also with greedy decoding. We report the attack success rate (ASR), defined as the percentage of evaluated responses classified as unsafe; lower is better. GSM8K. Following Ivison et al. (2025), we use the eight chain-of-thought demonstrations as individual query examples for data selection, without prepending other demonstrations to each query. At evaluation time, we include all eight demonstrations in each test prompt and report exact-match accuracy of the extracted final numerical answers on the 1,319 test problems. CodeX. We sample code completions at a temperature of 0.8 and report pass@10 on the 148 held-out problems. This metric estimates the probability that at least one of ten sampled completions passes all unit tests for a problem. D.5 KL Loss in Ablation Study The KL variant matches the normalized outputs of the meta-network p to a distribution q reweighted by the loss-gap pseudo-labels: pi=sϕ(xi)∑jsϕ(xj),qi=piexp(Δi∗)∑jpjexp(Δj∗),ℒϕKL=KL(∥).p_i= s_ φ(x_i) _js_ φ(x_j), q_i= p_i ( _i^*) _jp_j ( _j^*), _ φ^KL=KL(q\,\|\,p). Appendix E Toy Experiment in Sec.4 We construct disjoint seen and unseen datasets, each containing 2,575 safe examples from GSM8K (Cobbe et al., 2021) and 1,125 unsafe examples from BeaverTails (Ji et al., 2023). The unsafe examples pair harmful prompts with harmful responses. For validation, we use 100 safety-related examples from He et al. (2024), each pairing a harmful prompt with a refusal response. We train the meta-network on the seen set for 30 epochs using AdamW with a batch size of 256, an initial learning rate of 5×10−45× 10^-4, and a cosine learning rate schedule that decays to zero. For each split separately, we rank examples by predicted weight and classify the k=1,125k=1,125 lowest-weight examples as unsafe. Let ℬkB_k denote this predicted set and U the ground-truth unsafe set. We define unsafe-identification accuracy as Acc=|ℬk∩|k×100%.Acc= |B_k |k× 100\%. (61) Since ||=k|U|=k, this metric equals both precision and recall for unsafe-example identification. Shortcut learning. To test shortcut reliance, we append three synthetic features to each example before standardization. Let c=+1c=+1 for GSM8K examples and c=−1c=-1 for unsafe examples, and define =(1,−0.5,0.2)⊤v=(1,-0.5,0.2) . We generate the shortcut features as seen=ac+ϵ,unseen=−ac+ϵ′,z_seen=acv+ ε, _unseen=-acv+ ε , (62) where a controls shortcut strength and ϵ,ϵ′∼(,σ23) ε, ε (0,σ^2I_3), with noise standard deviation σ=0.3σ=0.3. We use a=0.5a=0.5 and a=0.3a=0.3 for the strong- and weak-shortcut settings, respectively. Noise is sampled independently across examples and coordinates. This construction reverses the class-dependent shortcut means on unseen data. For each split, we summarize the associations between the shortcut features and class labels before standardization. For feature j∈1,2,3j∈\1,2,3\, let C^j=1n−1∑i=1n(zij−z¯j)(ci−c¯) C_j= 1n-1 _i=1^n(z_ij- z_j)(c_i- c) (63) denote its empirical covariance with the label, where n is the split size and z¯j z_j and c¯ c are the sample means. We report the mean absolute covariance and mean absolute Pearson correlation: Cov=13∑j=13|C^j|,ρ=13∑j=13|C^jσ^jσ^c|,Cov= 13 _j=1^3| C_j|, ρ= 13 _j=1^3 | C_j σ_j σ_c |, (64) where σ^j σ_j and σ^c σ_c are the sample standard deviations of feature j and the label, respectively. We report seen accuracy AccSAcc_S, unseen accuracy AccUAcc_U, and their gap G=AccS−AccUG=Acc_S-Acc_U in percentage points. At comparable seen accuracy, a smaller gap indicates better generalization under shortcut reversal. Appendix F Small-to-Large Model Generalization Table 5 provides the detailed results of the small-to-large model transfer experiment summarized in Sec. 5.1. Small2Big achieves a 5.655.65–6.60×6.60× training speedup while retaining 77.53%77.53\% and 83.67%83.67\% of Big2Big’s average ASR in the Training and Generalization settings, respectively. These results show that a meta-network trained with TESS using a small LLM can generalize to data selection for a larger model. performance. Table 5: Small-to-large model transfer with TESS. Small2Big uses Qwen2.5-0.5B-Instruct for meta-network training and Qwen2.5-7B-Instruct for fine-tuning on the selected data; Big2Big uses Qwen2.5-7B-Instruct for both. We report ASR (%) and training speedup SrelS_rel relative to Big2Big within each dataset and setting. Avg. averages the six ASR scores, and Retention (%) is the ratio of Small2Big’s average ASR to Big2Big’s within the same setting. Higher is better for all metrics; better ASR and speedup results are bold. Setting Method Alpaca Dolly Overall ASR DH4 HB HEx SrelS_rel DH4 HB HEx SrelS_rel Avg. Retention Training Big2Big 44.50 23.50 24.83 1.00× 86.50 87.00 88.62 1.00× 59.16 – Small2Big 35.50 13.00 13.79 6.60× 74.00 72.00 66.90 5.65× 45.87 77.53 Generalization Big2Big 38.75 16.00 18.28 1.00× 84.50 83.50 80.00 1.00× 53.51 – Small2Big 32.25 13.00 14.14 6.20× 77.50 70.00 61.72 5.65× 44.77 83.67 Appendix G Conclusion In this paper, we introduced Transferable Example Scoring and Selection (TESS), transferable data-selection framework that utilize sample-level pseudo-labels to learn a data selection network generalizable to unseen data. Our theoretical and empirical analyses identify competitive weight suppression and persistent easy-feature preference as limitations of the ScaleBiO meta-objective. To address these issues, we propose a new training objective PVM. Experiments demonstrate cross-dataset, subset-to-full-corpus, and small-to-large model transfer. Specifically, in terms of generalization, TESS outperforms the strongest learnable-weighting baselines by 22.7522.75 and 3.993.99 points on LLM safety and targeted instruction tuning, respectively.