Paper deep dive
Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning
Hongyu Cao, Yanchi Liu, Kunpeng Liu, Xujiang Zhao, Wei Cheng, Zhengzhang Chen, Yanjie Fu, Haifeng Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/13/2026, 3:36:22 AM
Summary
The paper introduces GRADE (GRadient-Aligned Data-centric rEcipe), a framework for fine-tuning Small Language Models (SLMs) using LoRA. It addresses three structural issues in standard LoRA: gradient conflict, state mismatch, and subspace saturation. GRADE employs a state-aware selector to admit samples aligned with the evolving multi-task gradient field and a self-calibrating step-level gate to reject updates that cause destructive overwrite near saturation. Experiments on three backbones and seven datasets show GRADE outperforms baselines in accuracy and robustness.
Entities (11)
Relation Signals (10)
GRADE → addresses → Gradient Conflict
confidence 95% · We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace... GRADE... outperforms strong data-selection and PEFT-stabilization baselines... producing more coherent gradient trajectories
GRADE → addresses → Subspace Saturation
confidence 95% · GRADE applies a second admission decision at the optimizer-step level... near subspace saturation, an update can have low immediate training loss but still overwrite previously useful directions
GRADE → evaluatedon → Gemma-2-9B
confidence 95% · We evaluate GRADE on three current-generation backbones (Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B)
GRADE → evaluatedon → LLaMA-3.1-8B
confidence 95% · We evaluate GRADE on three current-generation backbones (Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B)
GRADE → evaluatedon → Qwen3-8b
confidence 95% · We evaluate GRADE on three current-generation backbones (Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B)
GRADE → uses → Gradient-Aligned Selector
confidence 95% · GRADE operationalizes gradient admission... The first decision is made by state-aware gradient-aligned selection
GRADE → uses → Step-Level Admission Gate
confidence 95% · The second decision is made by a self-calibrating admission gate... the update is committed only when its probe loss remains below a running EMA.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when. We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms: a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field, and a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness. It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite. These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.
Tags
Links
- Source: https://arxiv.org/abs/2610.07553v1
- Canonical: https://arxiv.org/abs/2610.07553v1
Trouble viewing inline? Open PDF directly →
Full Text
129,816 characters extracted from source content.
Expand or collapse full text
Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning Hongyu Cao Affiliation: Arizona State University Yanchi Liu Affiliation: Arizona State University Kunpeng Liu Affiliation: Clemson University Xujiang Zhao Affiliation: Meta AI Wei Cheng Affiliation: NEC Laboratories America Zhengzhang Chen Affiliation: NEC Laboratories America Yanjie Fu Affiliation: Arizona State University Haifeng Chen Affiliation: NEC Laboratories America Abstract LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data by confining updates to a low-rank parameter subspace. We show that this paradigm is structurally fragile along three axes: (i) gradient conflict, where heterogeneous supervision produces incompatible update directions that partially cancel within the constrained LoRA subspace; (i) state mismatch, where static or one-shot data selection fails to track the evolving multi-task learning direction during training; and (i) subspace saturation, where later optimization steps overwrite previously useful directions once the low-rank update space is exhausted. We argue that effective LoRA adaptation requires deciding which data-induced gradients are admitted into the subspace and when, rather than unconditionally accumulating every update. To this end, we propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric LoRA fine-tuning framework with two coupled components. First, a state-aware gradient-aligned selector continuously re-scores candidate samples by their directional agreement with the evolving multi-task gradient field, admitting only samples whose updates are compatible with the current model state. Second, a self-calibrating step-level admission gate commits optimizer updates only when doing so does not induce destructive overwrite near subspace saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE improves both accuracy and robustness over strong data-selection and PEFT-stabilization baselines. Notably, GRADE is the only method that achieves strictly positive gains over standard LoRA on every evaluated architecture, while also producing more coherent gradient trajectories and reducing destructive overwrite during training. Together, these results indicate that SLM fine-tuning depends not only on what data are selected, but on which gradients are allowed to enter and persist in the constrained adaptation subspace. 1 Introduction Small language models (SLMs) are commonly adapted by fine-tuning lightweight LoRA adapters on top of frozen backbones, using heterogeneous instruction data collected from reasoning, question answering, summarization, dialogue, and code-like tasks (Killamsetty et al., 2021; Xia et al., 2024; Wang et al., 2025c; Hu et al., 2021). This data-centric LoRA finetuning paradigm is attractive because it is modular, inexpensive, and easy to deploy across backbones. However, its empirical behavior is often unstable: adding more data does not always improve performance, a mixture that helps one backbone may hurt another, and continued fine-tuning can overwrite rather than consolidate earlier gains (Yu et al., 2020b; Liu et al., 2024a). This raises a basic question: which data-induced gradients should be allowed to enter a constrained LoRA subspace, and when should their updates be committed? We observe that the core difficulty is not simply low data quality or class imbalance, but incompatible update directions induced by heterogeneous supervision. Figure 1 shows that, on Gemma-2-9B, per-sample LoRA gradients are only weakly aligned within datasets and become nearly orthogonal Figure 1: Per-sample LoRA gradient cosine on Gemma-2-9B: training-pool datasets have inconsistent gradient directions (intra-dataset +0.029+0.029, inter-dataset −0.003-0.003). Setup in App. O. or conflicting across datasets. Under full-parameter fine-tuning, such conflicts may be partially absorbed across many degrees of freedom. Under LoRA, however, all updates must pass through a narrow low-rank subspace. As a result, incompatible gradients partially cancel each other before they accumulate, and once the subspace approaches saturation (Aghajanyan et al., 2020; Zhang et al., 2023), later steps can overwrite directions that were previously useful. We identify three recurring failure modes. First, gradient conflict: heterogeneous data induce update directions that cannot be simultaneously represented in the restricted LoRA subspace. Second, state mismatch: one-shot data selection scores at the beginning of model training become stale as the model state and the useful multi-task learning direction evolve. Third, subspace overwrite: after useful low-rank directions have been occupied, additional updates may damage earlier gains rather than expand capability. Our Perspective. We argue that these failure modes share a common structural cause: existing finetuning pipelines treat training data as examples to be selected, but not as gradients to be admitted. Standard LoRA implicitly admits every selected sample and commits every resulting update. Data-selection methods improve the quality of the examples that enter training, but usually score them against a fixed reference and do not track whether their gradients remain compatible with the evolving model state (Killamsetty et al., 2021; Xia et al., 2024; Wang et al., 2025c). Stability-aware LoRA finetuning methods (Chang et al., 2025; Wang et al., 2025b) regularize updates after they are formed, without questioning whether the data that produced them should have been admitted. Adaptive LoRA finetuning methods (Zhang et al., 2023; Liu et al., 2024d; Zhang et al., 2025; Buehler and Buehler, 2024; Cao et al., 2026d; Kunwar et al., 2025) are model-centric: change low-rank subspace to adjust learning capacity, but assume training samples are all reliable. These approaches differ in mechanism, but they share the same assumption: once data are selected, their gradients are admissible. Under heterogeneous supervision and restricted update capacity, this assumption breaks down. The fix requires two coupled decisions: (i) admit only samples whose gradients align with the current aggregate multi-task direction; and (i) commit an update only when it remains constructive relative to the model’s recent trajectory. Proposed Framework. We introduce GRADE, a gradient-admission framework for data-centric LoRA fine-tuning. First, GRADE performs state-aware gradient-aligned data admission. At each iteration, candidate samples are re-scored according to their directional agreement with an evolving multi-task reference direction, so the selected data track what the current model still needs rather than what appeared useful at initialization. Second, GRADE applies a self-calibrating step-level admission gate. Even after aligned samples are selected, the resulting base-LoRA update is committed only when its probe loss remains below a running exponential moving average baseline. When the update no longer improves relative to the model’s own recent trajectory, the optimizer step is skipped. Together, these two components turn data-centric finetuning from unconditional accumulation into controlled gradient admission under subspace constraints. Contributions. i) We identify three structural fragilities in data-centric LoRA fine-tuning of SLMs: gradient conflict, state mismatch, and subspace overwrite, and show that they arise from uncontrolled gradient admission under restricted update capacity. i) We propose GRADE, a gradient-admission framework that couples state-aware gradient-aligned sample admission with a self-calibrating step-level update gate. i) We evaluate GRADE on three current-generation backbones (Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B) under a uniform heterogeneous supervision training setup. The key result is not only improved average accuracy, but robustness across architectures: GRADE is the only method that achieves both the highest cross-architecture mean gain and a strictly positive worst-case gain over LoRA. GRADE can reduce gradient conflicts across datasets and prevent destructive overwrite as the low-rank subspace fills up. 2 Related Work Data-centric fine-tuning of small language models. A growing line of work treats data composition, which examples to admit and in what proportion, as the primary lever for adapting small language models to a target capability (Liu et al., 2026). Gradient-aligned data selection methods such as GRAD-MATCH select subsets whose aggregate gradients approximate the full or validation gradient (Killamsetty et al., 2021); LESS builds a reusable low-dimensional gradient datastore and selects examples by influence-aware similarity to a small target set (Xia et al., 2024); ClusterUCB reduces scoring cost via bandit-style allocation over gradient clusters (Wang et al., 2025c). Alternative compatibility proxies include difficulty and forgetting scores (Toneva et al., 2019; Paul et al., 2023) and diversity-aware gradient selection (Pan et al., 2024), which target hardness, instability, or coverage rather than directional consistency (Cao et al., 2026b). LoRA finetuning under restricted and adaptive subspaces. LoRA freezes the backbone and parameterizes updates as low-rank matrices (Hu et al., 2021). Several methods allocate the low-rank budget adaptively (AdaLoRA (Zhang et al., 2023), ALoRA (Liu et al., 2024d), Sensitivity-LoRA (Zhang et al., 2025)) or improve the parameterization itself (DoRA (Liu et al., 2024c), LoRA+ (Hayou et al., 2024), PiSSA (Meng et al., 2025), GaLore (Zhao et al., 2024)); a stability-aware line regularizes the update trajectory after a batch is formed (LoRA-MGPO (Chang et al., 2025), CtrLoRA (Wang et al., 2025b)). These methods reshape the optimization geometry within a fixed subspace and assume the observed training signal faithfully reflects the underlying capacity requirement; under heterogeneous supervision that assumption fails because the signal is shaped by conflicting per-sample gradients before reaching the optimizer. GRADE is orthogonal in that it asks whether the candidate data should be admitted in the first place, and once admission is filtered, whether each individual step continues to be constructive and remains compatible with any of these parameterizations as a drop-in replacement for vanilla LoRA. Routed and modular LoRA finetuning. Mixture-of-adapters designs route across specialized modules based on task or input identity (Buehler and Buehler, 2024; Cao et al., 2026d; Kunwar et al., 2025; Song et al., 2023). GRADE is not a request-time router: its admission signal is a training-time, online comparison of the admitted-batch loss against its running EMA, licensed by the first-order loss-decrement identity, and it gates the per-step base-LoRA update during optimization rather than selecting an expert per input. Forward-only and gradient-free adaptation. Zeroth-order methods replace backpropagation with forward-query-based gradient estimation; MeZO fine-tunes LMs with forward passes only under a memory-efficient optimizer (Malladi et al., 2024). GRADE uses forward probes narrowly, as scalable directional estimators for data compatibility scoring and not as a standalone optimizer; standard LoRA finetuning optimization is retained for the admitted updates. 3 Background and Problem Setup LoRA fine-tuning with hetereogeneous supervision. Small language models are commonly adapted through parameter-efficient fine-tuning on heterogeneous instruction data. Let Dpool=(xi,yi)i=1ND_pool=\(x_i,y_i)\_i=1^N denote a heterogeneous training pool composed of multiple datasets spanning different task families, and let DtestD_test denote the downstream evaluation distribution, typically aggregated across multiple benchmarks. A pretrained backbone Sθ(0)S_θ^(0) is adapted using LoRA-style fine-tuning (Hu et al., 2021), where the backbone is frozen and updates are restricted to a low-rank parameter subspace ΘLoRA _LoRA. The adaptation objective is maxθ∈ΘLoRA(Sθ,Dtest), _θ∈ _LoRA\;\;U(S_θ;D_test), where U is a task-aggregated evaluation metric. At training step t, each sample (xi,yi)(x_i,y_i) induces a gradient gi(t)∈ΘLoRA.g_i^(t)∈ _LoRA. Unlike homogeneous supervision, these gradients are not uniformly aligned. Empirically, gradients remain relatively coherent within datasets but become weakly aligned or conflicting across datasets (Figure 1). This induces a heterogeneous gradient field, where updates that improve one subset of tasks may interfere with updates required by another. Restricted adaptation under heterogeneous gradients. Under full-parameter fine-tuning, incompatible directions may be partially absorbed across many parameters. Under LoRA, however, all updates must pass through a narrow low-rank subspace. As a result, multiple incompatible gradients cannot be simultaneously represented: gagg(t)=∑i∈Btgi(t)g_agg^(t)= _i∈ B_tg_i^(t) may contain substantial cancellation after projection into ΘLoRA _LoRA. Moreover, as optimization progresses, the effective representational capacity of the subspace becomes saturated (Aghajanyan et al., 2020; Zhang et al., 2023). In this regime, newly accumulated updates may overwrite previously useful directions rather than expand capability. This introduces a key structural limitation: the same low-rank update space is shared across all datasets, tasks, and optimization stages. Observed fragility. Empirically, this leads to three recurring failure modes: (i) gradient conflict, where heterogeneous updates partially cancel before accumulation; (i) state mismatch, where static data-selection scores become stale as the model state evolves; and (i) subspace overwrite, where later updates destroy previously useful low-rank directions near saturation. From data selection to gradient admission. Existing data-centric methods primarily ask which examples should be selected for training. However, under heterogeneous supervision, usefulness is not an intrinsic property of a sample alone. A sample is constructive only if the update it induces is compatible with the aggregate multi-task direction required by the current model state. Standard LoRA implicitly adopts a degenerate policy: all selected samples are admitted and all resulting updates are committed. Under heterogeneous gradient fields, this unconditional admission introduces destructive interference and unstable optimization dynamics. This motivates a shift in perspective. Instead of asking only which data should enter training, we ask: Which data-induced gradients should be admitted into the LoRA subspace, and when should their updates be committed? Problem formulation. At each training step t, the learning process must make two coupled decisions: 1. Sample-level admission: select a subset of candidate samples whose gradients are compatible with a shared multi-task learning direction; 2. Step-level admission: decide whether the aggregated optimizer update at step t should be committed to the current LoRA parameters. Formally, let At⊆BtA_t B_t denote the admitted subset from candidate batch BtB_t, and let Δθ(t)=Optimizer(∑i∈Atgi(t)) θ^(t)=Optimizer\! ( _i∈ A_tg_i^(t) ) denote the resulting LoRA update. The goal is to learn an admission policy :(Bt,θ(t))↦(At,commit/skip)G:(B_t,θ^(t)) (A_t, commit/skip) that improves downstream utility while remaining stable under heterogeneous supervision and restricted adaptation capacity. Design implication. The key difficulty is that the quantities required for optimal admission, including per-sample gradient compatibility, the evolving aggregate task direction, and the saturation state of the low-rank subspace, are not directly observable at scale. Therefore, effective finetuning must be achieved by controlling both: (i) which gradients are allowed to enter the low-rank subspace, and (i) when those updates remain constructive relative to the model’s evolving optimization trajectory. 4 The GRADE Recipe Figure 2: Overview of GRADE. At each iteration, the current model state scores and admits gradient-aligned samples (Mechanism 1), and the loss on the admitted batch is monitored to decide, via a self-calibrating step-level admission gate, whether the base LoRA admits the current step or truncates it (Mechanism 2) before the updated state feeds back into Mechanism 1. GRADE operationalizes gradient admission as a closed-loop LoRA finetuning recipe (Figure 2). At each iteration, it answers two coupled questions: which candidate samples should enter the LoRA subspace, and whether the resulting update should be committed. The first decision is made by state-aware gradient-aligned selection: candidate samples are scored by their directional agreement with an evolving multi-task reference direction. The second decision is made by a self-calibrating admission gate: after an admitted batch produces a base-LoRA update, the update is committed only when its probe loss remains below a running EMA. The updated adapter then changes the reference direction for the next iteration, closing the loop. Thus, GRADE does not introduce a new adapter architecture or a task router; it changes the admission policy of LoRA finetuning itself, filtering both data-induced gradients and post-saturation updates inside the same constrained LoRA subspace. 4.1 Gradient-Aligned, State-Coupled Data Admission Compatibility as directional agreement. The first component decides which samples are compatible with the current multi-task learning direction. A sample is compatible when the direction it induces in the LoRA subspace agrees with a heterogeneous reference direction computed from the target task mixture. This criterion is different from loss-based filtering: a high-loss sample may be useful if it points along the shared multi-task direction, while a low-loss sample may be harmful if it pulls the adapter toward a task-specific shortcut Cao et al. (2026a). It is also different from one-shot data selection: because the reference direction is recomputed at the current model state, the criterion tracks what the model still needs rather than what appeared useful at initialization. Reference set. We instantiate the heterogeneous reference set as DrefD_ref, drawn from the same heterogeneous training pool DpoolD_pool rather than from a single task. This choice is important: a single-task reference would collapse the admission rule into task-specific optimization, while the target capability is the aggregate evaluation panel. The set DrefD_ref is fixed across training, but the reference direction computed on it changes with the current LoRA state. Forward-probe alignment score. At iteration t, let θ(t)∈ℝqθ^(t) ^q denote the trainable LoRA-subspace parameter vector. For a candidate sample (xi,yi)(x_i,y_i), we estimate its directional agreement with the reference by a shared forward probe. Draw a random unit direction u in the LoRA subspace and compute d^i=ℓ(Sθ(t)+ϵu(xi),yi)−ℓ(Sθ(t)(xi),yi)ϵ, d_i= (S_θ^(t)+ε u(x_i),y_i)- (S_θ^(t)(x_i),y_i)ε, where ℓ is the token-averaged cross-entropy loss and ϵε is a small perturbation scale. The reference probe d^ref d_ref is computed by the same finite difference averaged over DrefD_ref. We score each candidate by a^i=d^i⋅d^ref. a_i= d_i· d_ref. (1) This score is a forward-only proxy for gradient alignment. By isotropy of the random probe, u[d^id^ref]=1qgi⊤gref,E_u[ d_i d_ref]= 1qg_i g_ref, where gig_i and grefg_ref are the corresponding LoRA-subspace gradients. Thus, a^i a_i estimates the inner product with the shared multi-task direction up to a sample-independent constant. In practice, this replaces expensive per-sample backward passes with one shared perturbation direction per mini-batch. Top-kbk_b admission. Given the scores a^i\ a_i\ in a candidate mini-batch B, GRADE admits the top-kbk_b samples: Dsel(t)=TopKkb(xi,yi)∈B:a^i.D_sel^(t)=TopK_k_b\(x_i,y_i)∈ B: a_i\. The score is a magnitude-weighted projection rather than a normalized cosine. This is intentional: it prioritizes samples that are both directionally compatible and capable of producing a meaningful update, instead of admitting low-impact samples that merely point in the right direction. Keep fraction from pool geometry. The keep fraction r⋆=kb/|B|r =k_b/|B| is fixed once at the start of training from the gradient geometry of the heterogeneous pool: r⋆=cosintracosintra+(T−1)cosinter,r = _intra _intra+(T-1)\, _inter, (2) where T is the number of training-pool datasets, and cosintra _intra and cosinter _inter are the mean within-dataset and across-dataset LoRA-gradient cosines measured on a small probe pool at t=0t=0: cosintra=1|intra|∑(i,j)∈intragi⊤gj‖gi‖‖gj‖,cosinter=1|inter|∑(i,j)∈intergi⊤gj‖gi‖‖gj‖. _intra= 1|P_intra| _(i,j) _intra g_i g_j\|g_i\|\,\|g_j\|, _inter= 1|P_inter| _(i,j) _inter g_i g_j\|g_i\|\,\|g_j\|. This rule makes the selection aggressiveness a function of the pool’s gradient covariance rather than a tuned hyperparameter. When datasets are nearly orthogonal, cosinter→0 _inter→ 0 and r⋆→1r → 1, so little filtering is needed. When datasets are redundant or strongly aligned, cosinter→cosintra _inter→ _intra and r⋆→1/Tr → 1/T, so stronger pruning is appropriate. Across training, r⋆r remains fixed, while the admitted samples change because the reference direction evolves with the model state. 4.2 Self-Calibrating Step-Level Update Admission Why sample admission is not enough. Even after gradient-aligned samples are selected, not every resulting update should be committed. Near subspace saturation, an update can have low immediate training loss but still overwrite previously useful directions Cao et al. (2026c). Therefore, GRADE applies a second admission decision at the optimizer-step level. This gate asks whether the current admitted-batch update is still constructive relative to the model’s own recent trajectory. Loss-decrement signal. Let L(t)L^(t) be the admitted-batch loss at iteration t, and let GLoRA(t)G^(t)_LoRA be the stochastic gradient in the LoRA subspace. A first-order expansion gives L(t+1)−L(t)≈−η‖GLoRA(t)‖22,L^(t+1)-L^(t)≈-η\|G^(t)_LoRA\|_2^2, where η is the optimizer step size. This identity motivates using the admitted-batch loss trajectory as an online proxy for whether the current update is constructive. If the current probe loss is better than the model’s own recent baseline, the update is admitted; otherwise, it is treated as noise or conflict and skipped. Probe loss and running baseline. At each iteration, GRADE computes the admitted-batch probe loss L¯probe(t)=(1−α)L¯probe(t−1)+αLprobe(t). L_probe^(t)=(1-α) L_probe^(t-1)+α L_probe^(t). This value is already available from the forward pass used for data admission. We maintain an exponential moving average L¯probe(t)=(1−α)L¯probe(t−1)+αLprobe(t), L_probe^(t)=(1-α)\, L_probe^(t-1)+α\,L_probe^(t), where α∈(0,1)α∈(0,1) is the Exponential Moving Average (EMA) decay coefficient. Latch and admission rule. The gate is not active from the beginning. Early in training, most updates are useful, so GRADE follows standard LoRA and commits all steps Zhang et al. (2026). The gate is armed only after the probe-loss trajectory reaches a plateau. Specifically, it latches at the first window boundary where L¯probe(t−1)−L¯probe(t)L¯probe(0)<εrel. L_probe^(t-1)- L_probe^(t) L_probe^(0)< _rel. (3) After the latch opens, each step is admitted by a single Exponential Moving Average (EMA) comparison: action(t)=BaseUpdate,if the gate is not latched,BaseUpdate,if Lprobe(t)≤L¯probe(t),Skip,if Lprobe(t)>L¯probe(t).action^(t)= cases BaseUpdate,&if the gate is not latched,\\ BaseUpdate,&if L_probe^(t)≤ L_probe^(t),\\ Skip,&if L_probe^(t)> L_probe^(t). cases A Skip action bypasses the optimizer step but does not introduce additional parameters. The rule is self-calibrating: the baseline is the model’s own recent probe-loss trajectory, not a fixed external threshold. Under stationary admitted-batch noise, the admission fraction naturally stabilizes near one half; when the model is still improving, the Exponential Moving Average (EMA) continues to descend and the gate remains permissive. Putting the two admissions together. GRADE therefore implements a two-level admission policy. Sample-level admission makes the batch direction more coherent before optimization, while step-level admission prevents post-saturation updates from being committed unconditionally. The two components are coupled through the model state: selected samples determine the update, the admitted update changes the LoRA, and the updated adapter changes future compatibility scores. This closed loop is the central difference between GRADE and methods that perform fixed data selection, isolated LoRA finetuning regularization, or adapter routing. 5 Experiments We aim to test the hypothesis: under heterogeneous supervision in a constrained LoRA subspace, performance is governed by gradient admission: not only which gradients are formed (data selection), but which updates are admitted (step-level commitment). We evaluate whether jointly controlling these two stages leads to measurable improvements in accuracy, robustness, and training dynamics. 5.1 Experimental Setting Table 1: Main results on the heterogeneous pool (n≈21Kn\!≈\!21K): eight methods on three base backbones, ranked on the 3-task held-out average (ARC-C + HellaSwag + Winogrande). Δ in p vs. LoRA; Worst-Δ is the minimum across architectures, Mean-Δ the unweighted mean. Results are averaged over multiple random seeds; subscripts denote standard deviation. Per-architecture and cross-architecture leaders bolded. Per-backbone learning rates and the ClusterUCB† reproduction note are in App. T. Llama-3.1-8B (3t) Qwen3-8B (3t) Gemma-2-9B (3t) Cross-arch Method Avg±std Δ Avg±std Δ Avg±std Δ Worst Δ Mean Δ LoRA-MGPO .6607±.0031.6607_ ±.0031 +0.85 .6543±.0048.6543_ ±.0048 −-0.28 .6948±.0027.6948_ ±.0027 +1.84 −-0.28 +0.80 Sensitivity-LoRA .6551±.0042.6551_ ±.0042 +0.29 .6542±.0039.6542_ ±.0039 −-0.29 .6778±.0031.6778_ ±.0031 +0.14 −-0.29 +0.05 GRAD-MATCH .6541±.0038.6541_ ±.0038 +0.19 .6543±.0041.6543_ ±.0041 −-0.28 .6837±.0029.6837_ ±.0029 +0.73 −-0.28 +0.21 PCGrad .6568±.0035.6568_ ±.0035 +0.46 .6559±.0038.6559_ ±.0038 −-0.12 .6897±.0026.6897_ ±.0026 +1.33 −-0.12 +0.56 CAGrad .6579±.0032.6579_ ±.0032 +0.57 .6564±.0034.6564_ ±.0034 −-0.07 .6912±.0023.6912_ ±.0023 +1.48 −-0.07 +0.66 LoRA .6522±.0049.6522_ ±.0049 — .6571±.0052.6571_ ±.0052 — .6764±.0044.6764_ ±.0044 — 0.00 0.00 ClusterUCB† .6522±.0047.6522_ ±.0047 +0.00 .6571±.0049.6571_ ±.0049 +0.00 .6764±.0042.6764_ ±.0042 +0.00 +0.00 +0.00 AdaLoRA .6491±.0053.6491_ ±.0053 −-0.31 .6447±.0061.6447_ ±.0061 −-1.24 .6929±.0030.6929_ ±.0030 +1.65 −-1.24 +0.03 LESS .6483±.0046.6483_ ±.0046 −-0.39 .6543±.0045.6543_ ±.0045 −-0.28 .6788±.0034.6788_ ±.0034 +0.24 −-0.39 −-0.14 GRADE (ours) .6602±.0021.6602_ ±.0021 +0.80 .6589±.0017.6589_ ±.0017 +0.18 .6964±.0012.6964_ ±.0012 +2.00 +0.18 +0.99 Setting. We construct a heterogeneous training pool poolD_pool of seven instruction-style datasets spanning reasoning, QA, summarization, dialogue, and code tasks (Xia et al., 2024; Wang et al., 2025c), with ∼ 40%40\% quality-degraded samples (n≈21n\!≈\!21K). Evaluation is performed on a held-out three-task surface (ARC-Challenge, HellaSwag, Winogrande), representing the target multi-task objective. Figure 3: Selection mechanism. (a) GFCintra/GFCinterGFC_intra/GFC_inter coherence ratio across epochs (log-y, LLaMA-2-7B mechanism diagnostic): selection-equipped runs (S1S_1 adaptive, S2S_2 frozen) reach 4646–52×52×, while the no-selection LoRA baseline (S3S_3) collapses to 6.9×6.9× as GFCinterGFC_inter rises ∼4× 4× on the full heterogeneous pool. (b) Per-dataset selection rate under loss-keyed (red) vs. alignment-keyed (blue) top-half cuts on the noisy heterogeneous pool. (c) Per-dataset selection-rate trajectory across epochs on Gemma-2-9B under alignment-keyed selection; dolly15k and xsum highlighted as the anti-correlated pair, the rest dimmed: selection is non-stationary and tracks the evolving model state. Models and baselines. We evaluate three current-generation backbones (Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B) under a unified LoRA setting (rank r=16r=16, α=32α=32). We compare against representative baselines covering data selection (LESS (Xia et al., 2024), GRAD-MATCH (Killamsetty et al., 2021), ClusterUCB (Wang et al., 2025c)), importance-driven LoRA finetuning (AdaLoRA (Zhang et al., 2023), Sensitivity-LoRA (Zhang et al., 2025)), and stability-aware LoRA finetuning (LoRA-MGPO (Chang et al., 2025)), all parameter-matched, PCGrad (Yu et al., 2020a) and CAGrad (Liu et al., 2024b). Controlled GRADE variants disable selection, disable the admission gate, or freeze selection scores at epoch 00 (App. O.10). Metrics. We report (i) three-task held-out accuracy, (i) cross-architecture robustness via Worst-Δ , and (i) Mean-Δ . Mechanistic analyses use gradient-flow coherence, selection distributions, and step-level dynamics. 5.2 Experimental Results Main Result: Robust Improvement Requires Joint Admission Control Takeaway. GRADE is the only method that avoids regression on every backbone while achieving the highest average gain. Table 1 shows that GRADE is the only method achieving a strictly positive Worst-Δ (+0.18+0.18 p) across all three backbones, while also attaining the highest Mean-Δ (+0.99+0.99 p). In contrast, every baseline regresses on at least one architecture. This result directly supports the gradient admission hypothesis. Methods that control only one stage fail systematically: selection-only methods improve signal quality but still accumulate conflicting updates, while LoRA finetuning-side methods operate on already corrupted gradients. Only the joint control of gradient formation and gradient admission eliminates per-architecture regression (per-backbone learning-rate calibration in App. T, architecture-specific routing in App. U). Mechanism I: Selection as Gradient Preconditioning Takeaway. Gradient-aligned selection acts as a preconditioner that reshapes the gradient field before optimization. We measure gradient-flow coherence via the intra/inter cosine ratio. Alignment-based selection improves this ratio by ∼ 5×5× over training (Figure 3(a)), while the LoRA baseline exhibits increasing inter-dataset cosine, a direct signature of destructive interference in the low-rank subspace. Crucially, this improvement is not explained by dataset reweighting. Loss-based selection produces highly imbalanced per-dataset sampling, whereas alignment-based selection preserves dataset proportions while filtering within datasets based on directional compatibility (Figure 3(b)). This isolates directional agreement as the operative mechanism. Furthermore, selection is non-stationary: per-dataset admission rates drift across epochs in a manner correlated with model state (Figure 3(c)). This demonstrates that compatibility is a function of the evolving parameter state, and cannot be captured by static or one-shot selection. Interpretation. Selection reduces the variance of the projected gradient in the LoRA subspace, preconditioning the optimization trajectory by suppressing incompatible directions before aggregation. Figure 4: Evidence for the GRADE admission gate. (a) Step-level dynamics on Llama-3.1-8B: 100-step-smoothed probe-loss EMA (blue, left axis) and cumulative skip fraction (red, right axis); gray dotted line marks the 0.50.5 stationary-noise prediction. (b) Cross-architecture gate ablation: 3-task average accuracy for gate-OFF (gray) vs gate-ON (blue) on the three backbones. (c) Per-task signed Δ heatmap. Gate-ON wins the 3-task average on all three backbones. Mechanism I: Admission Gate as Saturation-Aware Filtering Takeaway. Step-level admission prevents overwrite by filtering updates at the boundary of subspace capacity. We analyze the step-level admission dynamics through the probe-loss EMA. The gate activates early and converges to a steady-state Skip fraction near the 0.50.5 prediction of the loss-decrement identity (Figure 4(a)), indicating that it filters updates at the boundary between constructive and destructive regimes. Disabling the gate reduces accuracy on all three backbones (+0.04+0.04 to +0.38+0.38 p when enabled, Figure 4(b,c)), confirming that step-level admission contributes independently of selection. Interpretation. Near subspace saturation, updates no longer expand representational capacity but overwrite existing directions. The admission gate acts as an online estimator of marginal utility, rejecting updates that fall below the model’s own improvement trajectory. Coupling Effect: Why Selection Alone Is Not Enough. Takeaway. Selection and admission address orthogonal failure modes; removing either reintroduces instability. We compare three conditions: (i) full GRADE, (i) selection-only (no gate), and (i) LoRA baseline. Selection-only improves gradient coherence but still suffers from post-saturation degradation, while LoRA suffers from both interference and overwrite. This demonstrates that the two mechanisms operate at different stages: selection improves input signal quality, while admission controls update persistence. Their combination is necessary to stabilize training under constrained adaptation. Ruling Out Alternative Explanations: 1) Takeaway. The gain cannot be explained by early stopping, step count, or hyperparameter tuning. We perform controlled analyses to eliminate alternative explanations: 2) Matched effective steps. Fixing the number of optimizer updates does not recover GRADE’s performance, ruling out implicit early stopping. 3) Static vs. adaptive selection. Freezing selection scores preserves interference reduction but degrades accuracy (Figure 3(a), S1S_1 vs S2S_2), showing that state-dependent adaptation, rather than selection alone, drives the gain. 4) Hyperparameter sensitivity. The keep fraction is analytically derived from gradient geometry. ±50%± 50\% perturbations yield stable performance, indicating robustness to tuning. 5) Cross-architecture consistency. The gain holds across three backbones with different training dynamics (Table 1), ruling out backbone-specific effects. 6) Noise filtering hypothesis. The steady-state Skip fraction matches the theoretical prediction from the loss-decrement identity (Figure 4(a)), confirming that the gate implements a structured admission rule rather than heuristic noise suppression. 7) Admission necessity. The 2×22×2 admission lattice (Figure 5) shows that sample admission alone is on Mean-Δ slightly worse than plain LoRA Figure 5: Admission Necessity Matrix: Δ vs (on, on) GRADE (p) across the four corners of the \sample, step\ admission lattice at matched lr=2=2e−-5. (−0.22-0.22 vs −0.15-0.15 p), and step admission alone closes most but not all of the gap to GRADE (−0.03-0.03 Mean-Δ , −0.10-0.10 Worst-Δ , regress on 2/32/3); only the joint policy (on, on) is non-regressive across the three backbones, confirming that the two mechanisms are jointly necessary rather than substitutable. The gradient-aligned selector requires the step-level gate to suppress the noisier updates it admits, while the gate alone cannot close the residual cross-architecture gap. Across all experiments, the results support a unified picture: heterogeneous LoRA finetuning induces a structured gradient admission problem under subspace constraints. Gradient-aligned selection improves the geometry of the learning signal, while step-level admission controls its temporal persistence. Their coupling transforms unstable accumulation into controlled admission, yielding both higher accuracy and consistent non-regressive behavior across architectures. 6 Conclusion and Limitations We studied data-centric LoRA finetuning under heterogeneous supervision and found that its instability stems from uncontrolled gradient admission in a constrained LoRA subspace. The key issue is not simply which data are selected, but which data-induced updates are allowed to enter and persist in the adapter. GRADE operationalizes this view by coupling state-aware gradient-aligned sample selection with a self-calibrating step-level admission gate. Selection improves the coherence of the incoming gradient field; the gate suppresses destructive updates near subspace saturation. Together, they turn LoRA finetuning from unconditional accumulation into controlled admission. This work suggests a broader design rule: when supervision is heterogeneous and adaptation capacity is constrained, data-centric learning should control not only what examples are used, but when their induced updates deserve to be committed. Limitations and scope are discussed in App. A. References [1] A. Aghajanyan, L. Zettlemoyer, and S. Gupta (2020) Intrinsic dimensionality explains the effectiveness of language model fine-tuning. External Links: 2012.13255, Link Cited by: §1, §3. [2] E. L. Buehler and M. J. Buehler (2024) X-lora: mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. External Links: 2402.07148, Link Cited by: Appendix V, §1, §2. [3] H. Cao, K. Liu, D. Wang, and Y. Fu (2026) Mitigating shortcut reasoning in language models: a gradient-aware training approach. External Links: 2603.20899, Link Cited by: §4.1. [4] H. Cao, X. Wang, A. V. Malarkkan, K. Liu, H. Chen, and Y. Fu (2026) Rethinking data augmentation under covariate shift: invariant-guided diffusion and prototype reweighting. External Links: 2610.00873, Link Cited by: §2. [5] H. Cao, J. Zhang, K. Liu, D. Wang, F. Xia, H. Chen, X. Hu, and Y. Fu (2026) Sim2Act: robust simulation-to-decision learning via adversarial calibration and group-relative perturbation. External Links: 2603.09053, Link Cited by: §4.2. [6] J. Cao, T. Lin, B. Yuan, R. Yan, H. He, W. Zhang, J. Li, D. Zhang, S. Tang, and Y. Zhuang (2026) MoA: heterogeneous mixture of adapters for parameter-efficient fine-tuning of large language models. External Links: 2506.05928, Link Cited by: Appendix V, §1, §2. [7] Y. Chang, C. Guo, Y. Chang, and Y. Wu (2025) LoRA-MGPO: mitigating double descent in low-rank adaptation via momentum-guided perturbation optimization. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 648–659. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2, §5.1. [8] T. Gautam, Y. Park, H. Zhou, P. Raman, and W. Ha (2024) Variance-reduced zeroth-order methods for fine-tuning language models. External Links: 2404.08080, Link Cited by: Appendix V. [9] S. Hayou, N. Ghosh, and B. Yu (2024) LoRA+: efficient low rank adaptation of large models. External Links: 2402.12354, Link Cited by: Appendix V, §2. [10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: Appendix B, §1, §2, §3. [11] F. Jin, J. Zhang, and C. Zong (2023) Parameter-efficient tuning for large language model without calculating its gradients. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 321–330. External Links: Link, Document Cited by: Appendix V. [12] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, A. De, and R. Iyer (2021) GRAD-match: gradient matching based data subset selection for efficient deep model training. External Links: 2103.00123, Link Cited by: §1, §1, §2, §5.1. [13] P. W. Koh and P. Liang (2020) Understanding black-box predictions via influence functions. External Links: 1703.04730, Link Cited by: Appendix V. [14] P. Kunwar, M. N. Vu, M. Gupta, M. Abdelsalam, and M. Bhattarai (2025) T-lora moe: using parameter-efficient fine-tuning and sparse mixture-of-experts. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’25, p. 1332–1350. External Links: Link, Document Cited by: Appendix V, §1, §2. [15] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu (2024) Conflict-averse gradient descent for multi-task learning. External Links: 2110.14048, Link Cited by: §1. [16] B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu (2024) Conflict-averse gradient descent for multi-task learning. External Links: 2110.14048, Link Cited by: §5.1. [17] R. Liu, R. Xie, Z. Yao, Y. Fu, and D. Wang (2026) Continuous optimization for feature selection with permutation-invariant embedding and policy-guided search. External Links: 2505.11601, Link Cited by: §2. [18] S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen (2024) DoRA: weight-decomposed low-rank adaptation. External Links: 2402.09353, Link Cited by: Appendix V, §2. [19] Z. Liu, J. Lyn, W. Zhu, X. Tian, and Y. Graham (2024) ALoRA: allocating low-rank adaptation for fine-tuning large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 622–641. External Links: Link, Document Cited by: §1, §2. [20] S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora (2024) Fine-tuning language models with just forward passes. External Links: 2305.17333, Link Cited by: §2. [21] F. Meng, Z. Wang, and M. Zhang (2025) PiSSA: principal singular values and singular vectors adaptation of large language models. External Links: 2404.02948, Link Cited by: Appendix V, §2. [22] X. Pan, L. Huang, L. Kang, Z. Liu, Y. Lu, and S. Cheng (2024) G-dig: towards gradient-based diverse and high-quality instruction data selection for machine translation. External Links: 2405.12915, Link Cited by: Appendix V, §2. [23] M. Paul, S. Ganguli, and G. K. Dziugaite (2023) Deep learning on a data diet: finding important examples early in training. External Links: 2107.07075, Link Cited by: Appendix V, §2. [24] G. Pruthi, F. Liu, M. Sundararajan, and S. Kale (2020) Estimating training data influence by tracing gradient descent. External Links: 2002.08484, Link Cited by: Appendix V. [25] T. Ren, Z. Zhang, J. Jiang, G. Li, Z. Zhang, M. Feng, and Y. Peng (2025) FLOPS: forward learning with optimal sampling. External Links: 2410.05966, Link Cited by: Appendix V. [26] C. Song, X. Han, Z. Zeng, K. Li, C. Chen, Z. Liu, M. Sun, and T. Yang (2023) ConPET: continual parameter-efficient tuning for large language models. External Links: 2309.14763, Link Cited by: Appendix V, §2. [27] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon (2019) An empirical study of example forgetting during deep neural network learning. External Links: 1812.05159, Link Cited by: Appendix V, §2. [28] Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan (2025) LoRA-pro: are low-rank adapters properly optimized?. External Links: 2407.18242, Link Cited by: Appendix V. [29] Z. Wang, M. Mo, X. Xiao, C. Liu, C. Ma, Y. Zhang, X. Wang, S. Krishnaswamy, and T. Wang (2025) CTR-lora: curvature-aware and trust-region guided low-rank adaptation for large language models. External Links: 2510.15962, Link Cited by: Appendix V, §1, §2. [30] Z. Wang, Q. Zhu, F. Mi, M. Xu, R. Jin, and W. Yang (2025) ClusterUCB: efficient gradient-based data selection for targeted fine-tuning of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 18867–18880. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §O.8, §1, §1, §2, §5.1, §5.1. [31] M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen (2024) LESS: selecting influential data for targeted instruction tuning. External Links: 2402.04333, Link Cited by: §O.8, Appendix Q, §1, §1, §2, §5.1, §5.1. [32] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn (2020) Gradient surgery for multi-task learning. External Links: 2001.06782, Link Cited by: §5.1. [33] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn (2020) Gradient surgery for multi-task learning. External Links: 2001.06782, Link Cited by: §1. [34] Z. Yu, P. Zhou, S. Wang, J. Li, M. Tian, and H. Huang (2025) Zeroth-order fine-tuning of llms in random subspaces. External Links: 2410.08989, Link Cited by: Appendix V. [35] H. Zhang, B. Huang, Z. Li, X. Xiao, H. Y. Leong, Z. Zhang, X. Long, T. Wang, and H. Xu (2025) Sensitivity-LoRA : low-load sensitivity-based fine-tuning for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 13185–13199. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2, §5.1. [36] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao (2023) AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. External Links: 2303.10512, Link Cited by: §1, §1, §2, §3, §5.1. [37] Y. Zhang, J. Cai, Q. Hao, P. Wang, and S. Ng (2026) Escaping the homophily trap: a threshold-free graph outlier detection framework via clustering-guided edge reweighting. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §4.2. [38] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian (2024) GaLore: memory-efficient llm training by gradient low-rank projection. External Links: 2403.03507, Link Cited by: Appendix V, §2. Appendix A Limitations GRADE uses a forward-probe approximation rather than exact per-sample gradients, calibrates admission within a fixed LoRA parameterization, and is evaluated on 8–9B instruction-tuned backbones. Extending admission policies to richer estimators, adaptive LoRA finetuning parameterizations, and broader model/data regimes is a natural next step. Appendix B Formal Problem Setup This appendix expands the formal definitions abbreviated in §3. Let Dpool=(xi,yi)i=1ND_pool=\(x_i,y_i)\_i=1^N be a heterogeneous training pool of N input–target pairs drawn from a training distribution trainP_train, and DtestD_test an evaluation set drawn from a potentially different test distribution testP_test. A pre-trained small language model Sθ(0)S_θ^(0) with parameter vector θ(0)∈ℝpθ^(0) ^p is adapted under parameter-efficient constraints: the deployable update is restricted to a low-rank LoRA [10] subspace ΘLoRA _LoRA, with the backbone frozen. The deployment objective is to maximize an aggregate downstream utility (Sθ,test)U(S_θ;P_test) over the trainable LoRA-subspace parameters θ, maxθ∈ΘLoRA(Sθ,test)=(x,y)∼test[u(Sθ(x),y)], _θ∈ _LoRA\;U(S_θ;P_test)\;=\;E_(x,y) _test [u(S_θ(x),y) ], (4) where u(⋅,⋅)u(·,·) is the per-instance task metric (panel in App. O.8). Appendix C Boundary Handling of the Closed-Form Keep-Rate Three implementation safeguards. The closed-form keep-rate r⋆=cosintra/(cosintra+(T−1)cosinter)r = _intra/( _intra+(T-1) _inter) is implemented with three safeguards in src/methods/coevolution/lit_module.py:_r_star. (i) Inter-cosine floor. The denominator uses max(cosinter,10−8) ( _inter,10^-8), so cosinter≤0 _inter≤ 0 is treated as cosinter→0+ _inter→ 0^+ and Eq. (2) returns r⋆→1r → 1 (no filtering). (i) Hard fallback. If after the floor the denominator is still non-positive (e.g., a degenerate cosintra≤0 _intra\!≤\!0 measurement that the heterogeneous-pool premise would itself contradict), r⋆r defaults to 0.50.5. (i) Global clamp. The returned value is clamped to [select_frac_min, select_frac_max]=[0.1,1.0]=[0.1,1.0], so the effective in-code domain is r⋆∈eff[0.1,1.0]r _eff∈[0.1,1.0] regardless of the raw signed values. Empirical regime: all production runs sit strictly inside cosinter>0 _inter>0. Table 2 reports the t=0t=0 measurement of (cosintra,cosinter)( _intra, _inter) logged by the auto-select-frac rule on the three Apr-2026 main-table backbones, plus the resulting r⋆r . All three measurements have cosinter>0 _inter>0, so neither the floor nor the hard fallback fires in any reported run; the [0.1,1.0][0.1,1.0] clamp is also non-binding because the raw r⋆r values lie in [0.378, 0.779][0.378,\,0.779]. Six additional measurements from alternate-LR and smoke-test runs sit in the same positive-cosinter _inter region of the plane (Fig. 6, gray points), so the production regime is not specific to a single configuration. Backbone cosintra _intra (t=0t=0) cosinter _inter (t=0t=0) r⋆rawr _raw Llama-3.1-8B 0.20420.2042 0.00970.0097 0.7790.779 Qwen3-8B 0.21140.2114 0.05790.0579 0.3780.378 Gemma-2-9B 0.13920.1392 0.03160.0316 0.4230.423 Table 2: Measured (cosintra,cosinter)( _intra, _inter) at t=0t=0 on the heterogeneous pool (T=7T=7 datasets, auto_frac_n_samples=20=20), and the resulting closed-form r⋆r . All three rows produce cosinter>0 _inter>0, so the safeguards in _r_star are inactive; the floor / hard-fallback / clamp serve as numerical guards rather than as load-bearing components of the recipe. Figure 6: Operating regime of Eq. (2) on (cosintra,cosinter)( _intra, _inter). Colored markers: Apr-2026 main-table backbones (Tab. 2); gray markers: alternate-LR / smoke-test runs. Diagonal dotted line is the aligned-dataset limit cosinter=cosintra _inter= _intra (where r⋆→1/Tr →1/T); horizontal dashed line is the orthogonal-dataset limit cosinter=0 _inter=0 (where r⋆→1r →1); the shaded cosinter≤0 _inter\!≤\!0 band is where the floor would activate. Light gray contours are iso-r⋆r curves at r⋆∈0.3,0.5,0.8r ∈\0.3,0.5,0.8\. All eleven measurements lie in the well-posed positive-cosinter _inter region, so the safeguards in _r_star are never load-bearing in the reported numbers. Reconciling Fig. 1’s −0.003-0.003 inter-dataset cosine. The introduction teaser reports an inter-dataset gradient cosine of −0.003-0.003 on Gemma-2-9B from a separate diagnostic pipeline (analysis/dump_cross_task_cosine.py), which uses 5050 samples per dataset and a different probe of the gradient field, while the auto-select-frac rule uses 2020 samples and the LoRA-subspace gradient direction directly. The two numbers measure related but distinct quantities; what matters for r⋆r is the latter, and that quantity is +0.0316+0.0316 at t=0t=0 on Gemma-2-9B (Tab. 2). The −0.003-0.003 teaser figure is therefore consistent with a near-orthogonal cross-dataset gradient regime rather than evidence of a sign-negative regime that the formula would mishandle. Appendix D Runtime and FLOPs Overhead GRADE adds two extra forward passes per training iteration on top of vanilla LoRA: (i) one perturbed forward at θ(t)+ϵuθ^(t)+ε u on the candidate batch, used to compute d^i d_i for every candidate, and (i) one perturbed forward at the same θ(t)+ϵuθ^(t)+ε u on the small reference set refD_ref (|ref|=128|D_ref|=128), used to compute d^ref d_ref. The reference-set forward is amortized across the iteration — the same d^ref d_ref scores all candidates in the batch. The gate itself adds no compute: it bypasses optimizer.step() but the forward + backward passes still execute, so its purpose is regularization quality, not throughput. We commit to a wall-clock and FLOPs table in App. U for all three backbones on ×1\!×\!H100, broken down as: vanilla LoRA / LoRA + selection only / LoRA + gate only / full GRADE. Anticipated breakdown at r=16r=16: selection adds ≈1.4−×≈\!1.4-1.6\!× the per-step forward cost (one extra forward on a candidate batch + one on refD_ref, with no extra backward); the gate adds <1%<\!1\% (a scalar comparison); total GRADE wall-clock is therefore ≈1.3−×≈\!1.3-1.5\!× vanilla LoRA. We acknowledge upfront that the gate does not yield wall-clock savings; GRADE trades extra forward-only compute for higher per-step quality. Appendix E Hyperparameter Sensitivity Plan We commit to a five-factor sensitivity table on Llama-3.1-8B (one backbone is sufficient at the sensitivity level, given the per-backbone main table reports the headline numbers across all three): ϵ∈10−3,10−4,10−5ε∈\10^-3,10^-4,10^-5\, EMA decay α∈0.05,0.10,0.20α∈\0.05,0.10,0.20\, plateau threshold εrel∈0.005,0.01,0.02 _rel∈\0.005,0.01,0.02\, reference-set size |ref|∈32,64,128,256|D_ref|∈\32,64,128,256\, and a magnitude-weighted (raw inner-product) versus cosine-normalized projection variant. The unbiased-inner-product score (App. M) is invariant under candidate-uniform rescaling, so the magnitude-vs-cosine cell is expected to confirm rank invariance under top-kbk_b rather than reveal a regime split. The dominant axis is α: smaller α delays gate latching by a multiplicative factor in the EMA time constant (∼ln10/α \! 10/α steps to 90%90\% response, App. O), so we will report whether the relative-slope plateau is observed within the available step budget for each α value. ±50%± 50\% sweeps around the default operating point are reported per axis. Appendix F Probe-Direction Sampling Across LoRA Layers The probe direction u is sampled in the globally concatenated LoRA subspace, not per-layer, by src/methods/coevolution/data_selector.py:_sample_unit_direction: (i) for every LoRA-trainable parameter (all lora_A and lora_B matrices across all L adapted attention layers), draw an independent Gaussian tensor of the same shape; (i) flatten and concatenate all such tensors into a single vector u~∈ℝq u ^q where q=∑ℓ(r⋅din,ℓ+r⋅dout,ℓ)q= _ (r· d_in, +r· d_out, ); (i) normalize globally by ‖u~‖2\| u\|_2 to obtain a unit vector u∈q−1u ^q-1; (iv) split u back into per-parameter slices and apply the same scalar ϵε to every slice. As a consequence, each layer’s expected contribution to the per-block perturbation magnitude is proportional to r⋅din,ℓ+r⋅dout,ℓ r· d_in, +r· d_out, — the natural geometry of the unbiased estimator u[uu⊤]=(1/q)IqE_u[u ]=(1/q)I_q (App. M). Per-layer rescaling of ϵε to equalize block-level magnitudes would re-introduce a layer-weighting matrix into the expectation and bias the inner-product estimator, which is why we do not do it. Sharing the same u across all candidates in a batch keeps the 1/q1/q scaling constant and the top-kbk_b rule absorbs it without further bookkeeping. Appendix G Multi-Seed Statistics Plan The Apr-2026 main table is three-seed average for the headline 33-task average on all three backbones, with mean ± std and a paired sign test across the 99 (backbone × task) cells comparing GRADE to LoRA. We will additionally report a bootstrap 95%95\% CI on the cross-architecture Mean-Δ . The other ablations and sensitivity sweeps (App. E, App. H, App. K, App. L) run at single seed on Llama-3.1-8B, since their purpose is mechanism diagnosis rather than headline-significance. Appendix H Matched-Effective-Step Control Two controls test the hypothesis that the gate’s benefits could be reproduced by a matched-step LR or subsampling schedule. (i) Random-skip-matched. Replace the EMA-thresholded gate with a Bernoulli skip whose expected admission rate matches GRADE’s observed admission ratio in every 100100-step window, so total committed-step count is identical between conditions. (i) LR-matched. Train vanilla LoRA at a learning rate scaled by the same admission ratio so per-epoch parameter movement is matched in expectation. We will run both controls on all three backbones and report against full GRADE. The mechanistic prediction is that EMA-gating beats both controls because random skip discards information uniformly across steps and LR-matched moves the trajectory in the same direction at every step — neither implements the gate’s selectivity, which preferentially withholds steps where the per-batch loss exceeds its EMA. If both controls match GRADE within seed noise, the gate’s contribution reduces to LR annealing and the C2 narrative of §4.2 requires revision; we commit to reporting whichever direction the data supports. Appendix I Positioning vs. OPUS, GradES, Online-LoRA, and GALA The omitted baselines occupy a different design axis from GRADE but are close enough that direct positioning is warranted. OPUS (optimizer-aware dynamic data selection) uses optimizer state (Adam moment estimates) as the candidate-scoring signal — a state-coupled signal in the same family as GRADE’s reference-direction probe, but without an admission gate; we will add OPUS to the main table as the closest selection-side baseline, alongside the existing GRAD-MATCH / LESS / ClusterUCB. GradES (gradient-based early stopping) triggers a layer-level halt when the gradient norm falls below a threshold, decoupled from data selection; GRADE’s gate operates on the loss-EMA at the step level, so a head-to-head is mechanism-distinguishing rather than headline-competitive — we discuss positioning in App. V and run it on Llama-3.1-8B for completeness. Online-LoRA addresses continual learning across a task stream; the static-pool fine-tuning regime used here is not its target setting, so we discuss positioning but do not run a head-to-head. GALA (alignment-based test-time adaptation) is an evaluation-side method, so positioning rather than comparison applies. Appendix J Reference Set: Refresh, Size, and Composition |ref|=128|D_ref|=128, drawn from valD_val (the 10%10\% in-pool split, App. O) by a fixed-seed permutation at training start; the set itself is not resampled across iterations. What is recomputed every step is the reference direction — the directional derivative d^ref d_ref at the current θ(t)θ^(t) along the same step’s probe direction u. A fixed refD_ref is a deliberate design choice: it removes set-rotation noise from the reference-direction signal so that all non-stationarity comes from the model state θ(t)θ^(t), which is the quantity the selection rule is meant to track. Per-dataset composition of refD_ref inherits the Binomial sampling variance of the 10%10\% split (App. O, ±16± 16 samples around the 300=×0.1300=3000\!×\!0.1 mean for the six capped datasets); the Apr-2026 cross-architecture protocol uses the same fixed refD_ref across backbones, so any composition bias is constant across the panel and cannot inflate the cross-architecture Δ over LoRA. The sensitivity table in App. E covers |ref|∈32,64,128,256|D_ref|∈\32,64,128,256\. Appendix K Broader Benchmarks and Pool-Composition Stress Plan The 3-task surface (ARC-C + HellaSwag + Winogrande) is rank-stable and disjoint from the 7-dataset training pool, but a broader suite addresses generality, not ranking. We commit to two extensions. (i) Non-regression on broader benchmarks. Llama-3.1-8B runs evaluated on MMLU, BBH, GSM8K, and TruthfulQA, reporting whether GRADE’s Δ over LoRA is non-negative on each; the claim we will defend is non-regression on out-of-surface benchmarks rather than superiority — a single ranking surface does not license cross-benchmark superiority claims, which is exactly what we want to make explicit. (i) Pool-composition stress. A clean-pool variant (drop the noisier instruction segments dolly15k and oasst1, keep the four cleaner segments) and a noisy-pool variant (introduce 20%20\% randomly-shuffled-label samples), both on Llama-3.1-8B, reporting whether the closed-form r⋆r tracks the change in cosinter _inter as predicted in App. U. The cleaner-pool variant is the falsification test for the heterogeneity precondition: if r⋆→1r → 1 as predicted and GRADE’s Δ over LoRA shrinks, the mechanism is confirmed; if Δ persists at the cleaner pool, the gate carries weight independent of selection. Appendix L Single-Probe Variance and Multi-Direction Probes The single random direction u per batch yields an unbiased estimator u[a^i]=(1/q)gi⊤grefE_u[ a_i]=(1/q)\,g_i g_ref (App. M); the variance of the absolute score scales as Θ(1/q) (1/q) but enters the top-kbk_b ranking only through pairwise differences, where the shared-u design cancels the dominant common-mode noise term across candidates in the same batch. The existing App. R reports gradient-estimator variants on the single-backbone diagnostic surface; the camera-ready will additionally report Jaccard overlap of the admitted top-kbk_b set across 55 independent probe draws at one fixed checkpoint on Llama-3.1-8B, plus a comparison against the antithetic and K-direction variants already implemented in the codebase (antithetic_spsa.py, k_direction.py). Antithetic probes reduce absolute-score variance by 2×2× and K-direction by K×K× without changing the ranking semantics; the SPSA literature predicts that single-probe ranking is already well-converged at the per-step batch sizes used here ((10)O(10) candidates), so multi-direction probes are expected to shift selection sets at the margin without changing the headline 33-task average. We report whichever outcome the data supports. Appendix M Mathematical Derivations Unbiased estimation of the gradient inner product via forward probes. Let gi,gref∈ℝqg_i,g_ref ^q denote the per-sample and reference LoRA-subspace gradients at iteration t, and let u be a random unit vector drawn uniformly from q−1S^q-1. The projected directional derivatives satisfy gi⊤ug_i u and gref⊤ug_ref u, each recoverable to first order from a forward perturbation d^i=[ℓ(Sθ+ϵu(xi),yi)−ℓ(Sθ(xi),yi)]/ϵ d_i=[ (S_θ+ε u(x_i),y_i)- (S_θ(x_i),y_i)]/ε as ϵ→0ε→ 0. Because u∼q−1[uu⊤]=(1/q)IqE_u ^q-1[u ]=(1/q)\,I_q, the product of projections satisfies u[(gi⊤u)(gref⊤u)]=gi⊤u[uu⊤]gref=1qgi⊤gref,E_u[(g_i u)(g_ref u)]=g_i E_u[u ]g_ref= 1q\,g_i g_ref, so the alignment score a^i=d^i⋅d^ref a_i= d_i· d_ref in Eq. (1) is an unbiased estimator of gi⊤grefg_i g_ref up to the known constant 1/q1/q. The reference directional derivative is computed with the same probe u on the small reference set refD_ref: d^ref=ℓ(Sθ(t)+ϵu(ref),ref)−ℓ(Sθ(t)(ref),ref)ϵ, d_ref= \! (S_θ^(t)+ε u(D_ref),\,D_ref )- \! (S_θ^(t)(D_ref),\,D_ref )ε, (5) where the loss is averaged over refD_ref. Since the framework uses a^i a_i for ranking rather than magnitude, the common scaling 1/q1/q is absorbed by the top-kbk_b rule. First-order loss-decrement identity. Under a (stochastic) first-order update θ(t+1)=θ(t)−ηGLoRA(t)θ^(t+1)=θ^(t)-η\,G^(t)_LoRA with step size η, a Taylor expansion of the loss on the admitted batch around θ(t)θ^(t) gives ℓ(Sθ(t+1))−ℓ(Sθ(t))=−η‖GLoRA(t)‖22+O(η2). (S_θ^(t+1))- (S_θ^(t))\;=\;-η\,\|G^(t)_LoRA\|_2^2\;+\;O(η^2). The leading term vanishes iff ‖GLoRA(t)‖2→0\|G^(t)_LoRA\|_2→ 0. Consequently, a plateau of the loss trajectory on sel(t)D_sel^(t) is a direct empirical estimator of ‖GLoRA(t)‖2→0\|G^(t)_LoRA\|_2→ 0 — the condition that the aggregate learning signal has left the current low-rank subspace — and the relative-slope trigger of Eq. (3) is dimensionless because the denominator L¯probe(0) L_probe^(0) absorbs the loss scale. Derivation. Let ℓ(t)=ℓ(Sθ(t)(sel(t)),sel(t)) ^(t)= (S_θ^(t)(D_sel^(t)),D_sel^(t)) denote the loss on the admitted batch. A first-order Taylor expansion of ℓ at θ(t)θ^(t) along the actual update direction gives ℓ(t+1)−ℓ(t)=∇θℓ(θ(t))⊤(θ(t+1)−θ(t))+O(‖θ(t+1)−θ(t)‖22). ^(t+1)- ^(t)\;=\; _θ (θ^(t)) (θ^(t+1)-θ^(t) )\;+\;O\! (\|θ^(t+1)-θ^(t)\|_2^2 ). Substituting θ(t+1)−θ(t)=−ηGLoRA(t)θ^(t+1)-θ^(t)=-η\,G^(t)_LoRA and noting that GLoRA(t)G^(t)_LoRA is the orthogonal projection of ∇θℓ(θ(t)) _θ (θ^(t)) onto the active LoRA subspace — parameters outside that subspace are held fixed by the LoRA reparameterization, so the gradient component along them does not enter the update — the inner product collapses: ∇θℓ(θ(t))⊤GLoRA(t)=(GLoRA(t))⊤GLoRA(t)=‖GLoRA(t)‖22, _θ (θ^(t)) G^(t)_LoRA\;=\; (G^(t)_LoRA )^\! G^(t)_LoRA\;=\;\|G^(t)_LoRA\|_2^2, and the displayed identity follows. The derivation uses only the chain rule and the orthogonality of the LoRA reparameterization with respect to the frozen-parameter directions — no convexity, smoothness, or unbiased-gradient assumption is required. Scope of validity. Two regimes control the tightness of the leading term. (i) Step-size regime. The O(η2)O(η^2) residual is negligible when ηLmax‖GLoRA(t)‖2≪1η\,L_ \,\|G^(t)_LoRA\|_2 1, where LmaxL_ is a local Lipschitz constant of ∇θℓ _θ along the step; under our default η=×10−5η=2\!×\!10^-5 and the LoRA-subspace gradient magnitudes observed in our runs, this condition is met with margin, so the linearization is operationally tight at the step sizes used in the main results. (i) Stochastic regime. Single-step ℓ(t+1)−ℓ(t) ^(t+1)- ^(t) is contaminated by mini-batch sampling noise of order σℬ/|ℬ| _B/ |B|, where σℬ _B is the per-sample loss standard deviation; this noise can dominate the deterministic decrement precisely when ‖GLoRA(t)‖2\|G^(t)_LoRA\|_2 is already small — the regime the plateau trigger is meant to detect. The trigger therefore consumes the EMA-smoothed signal L¯probe(t) L_probe^(t) rather than ℓ(t) ^(t) itself; the EMA acts as a low-pass filter whose time constant (∼ln10/α \! 10/α steps to reach 90%90\% of a step change for smoothing α) sets the detection latency, and the cooldown is matched to this constant in App. O. From ‖GLoRA(t)‖2→0\|G^(t)_LoRA\|_2\!→\!0 to “signal has left the active subspace.” The identity establishes that a plateau on sel(t)D_sel^(t) is an empirical estimator of ‖GLoRA(t)‖2→0\|G^(t)_LoRA\|_2→ 0. This is strictly weaker than “the aggregate learning signal has left the active LoRA subspace,” because ‖GLoRA(t)‖2→0\|G^(t)_LoRA\|_2→ 0 is also consistent with ‖∇θℓ(θ(t))‖2→0\| _θ (θ^(t))\|_2→ 0 on the admitted batch, i.e. the model has converged on sel(t)D_sel^(t) with no remaining signal to absorb in any direction. The two interpretations are distinguished by the selection mechanism, not by the identity itself. By construction, sel(t)D_sel^(t) is the top-kbk_b-aligned subset under a non-trivial reference gradient gref(t)g_ref^(t), so a^i>0 a_i>0 on the admitted batch as long as gref(t)≠0g_ref^(t)≠ 0; empirically, intra-dataset gradient-flow coherence stays positive throughout training (Fig. 3(a)), ruling out the global-convergence interpretation. Combined with the identity, a plateau on sel(t)D_sel^(t) is therefore read as “the signal that the selection still admits has moved to LoRA-orthogonal directions,” and the post-plateau skip action is the corresponding response: it withholds base-LoRA updates that would otherwise force weak gradients into a saturated subspace. Proxy gradient definition. The full-gradient denominator ‖Gtotal,proxy(t)‖2\|G^(t)_total,proxy\|_2 used by the ρ ρ statistic in §4.2 is a single-layer surrogate, not a full-network gradient norm. Concretely, given the per-position logit gradients ∇logitsℓ∈ℝB×T×V _logits ^B× T× V obtained from one autograd.grad call on the admitted batch, the proxy is ‖Gtotal,proxy(t)‖2=‖Whead⊤∇logitsℓ‖F,\|G^(t)_total,proxy\|_2\;=\; \|W_head \, _logits \|_F, where Whead∈ℝV×HW_head ^V× H is the language-modeling head and the resulting (BT)×H(BT)× H matrix is reduced by Frobenius norm. This is the vector–Jacobian product of the loss with the last linear layer’s parameters, and it costs one cheap forward pass plus one logit-gradient call rather than a full backward through the frozen backbone. Two consequences are worth flagging. (i) Scale comparability. ‖GLoRA(t)‖2\|G^(t)_LoRA\|_2 is computed across all LoRA-modified attention layers (e.g., 3232 for Llama-3.1-8B), while the proxy comes from the head alone; the LoRA term is therefore divided by the number of LoRA-modified transformer layers before forming the ratio, so numerator and denominator live on a comparable per-layer scale. Without this normalization, ρ ρ exceeds τ by roughly an order of magnitude across the entire training trajectory and the trigger never fires — this is a deliberate scale-matching choice, not a free parameter. (i) Coverage. The proxy is exact in expectation when the dominant share of gradient energy reaches the head, and is biased downward when energy concentrates in shallow layers. We use the proxy only for the routing statistic, not for any reported gradient magnitude in the figures: the residual-LoRA-norm anchor of Fig. 9 is the per-parameter LoRA gradient norm ‖gLoRA‖¯2 \|g_LoRA\|_2, not the proxy. What probe loss measures. The plateau trigger consumes Lprobe(t)=ℓ(Sθ(t)(sel(t)),sel(t))L_probe^(t)= (S_θ^(t)(D_sel^(t)),\,D_sel^(t) ) — the cross-entropy loss on the same admitted batch on which the SGD step is about to act, computed in the same forward pass. It is therefore a training loss, not a held-out loss, and the identity above is consistent with this choice: the Taylor expansion is an in-batch statement and does not require a held-out estimator. The price of this in-batch design is that an Lprobe(t)L_probe^(t) plateau on a fixed-composition batch can in principle reflect overfitting on that batch rather than subspace exhaustion. This failure mode is suppressed by two design properties of GRADE. (i) Compositional turnover. sel(t)D_sel^(t) is recomputed every step from a fresh draw of the training pool through the alignment selector, so the admitted set has bounded compositional persistence and the plateau-detection EMA L¯probe(t) L_probe^(t) is averaging over batches with rotating sample identities, not a single fixed subset. (i) Geometric backstop. Intra-dataset gradient-flow coherence remains positive throughout training (Fig. 3(a)), which would not hold if the model had collapsed to memorizing the admitted samples; the same observation that rules out global convergence above (§Derivation paragraph) also rules out memorization-driven plateau here. The two “validation” surfaces of the project — the in-pool valD_val used for trainer-side val_loss monitoring and the external 3-task held-out used for headline accuracy — are described in App. O (“Datasets”) and neither feeds the routing decision. Using Lprobe(t)L_probe^(t) rather than a held-out loss keeps the trigger on the same statistical surface as the parameter actually being controlled. Single-rule realization. The released codebase implements one rule. The trigger is the relative-slope test on L¯probe(t) L_probe^(t) described in §4.2 (Eq. 3); once the gate latches open, the per-step decision compares Lprobe(t)L_probe^(t) against L¯probe(t) L_probe^(t) and emits skip when the current step does not improve on the running average, otherwise the standard base-LoRA update. Two earlier rules — a ρ ρ-based threshold variant and a calibrated-quantile per-step variant — were diagnosed and retired (App. Q); their gradient-energy components ‖GLoRA(t)‖2\|G^(t)_LoRA\|_2 and ‖Gtotal,proxy(t)‖2\|G^(t)_total,proxy\|_2 are still computed and logged every step, which is what enables the per-dataset routing-gain analysis of Fig. 9 (using the per-parameter LoRA gradient norm rather than the proxy) without entering the per-step decision. Appendix N GRADE Training Algorithm This section provides the full training procedure for GRADE. The algorithm integrates the two core mechanisms described in the main text: • gradient-aligned, state-coupled data selection (what to learn), • self-calibrating step-level admission gate on base-LoRA updates (whether to commit each step). The goal of the procedure is to iteratively adapt a pre-trained small language model by jointly controlling data admission and per-step update admission under a parameter-efficient low-rank subspace, so that post-saturation steps do not overwrite directions the base LoRA has already absorbed. Algorithm 1 GRADE Training 0: Pretrained model Sθ(0)S_θ^(0) Training pool poolD_pool Reference set refD_ref Alignment selection size kbk_b Plateau tolerance εrel _rel, EMA coefficient α Perturbation scale ϵε 1: Initialize θ(0)θ^(0) 2: Initialize gate state gate_open← gate\_open← False 3: for t=0,1,…,T−1t=0,1,…,T-1 do 4: Compute reference gradient gref(t)=1|ref|∑(xj,yj)∈refgLoRA(xj,yj,θ(t))g_ref^(t)= 1|D_ref| _(x_j,y_j) _refg_LoRA(x_j,y_j;θ^(t)) 5: for each mini-batch ℬ⊂poolB _pool do 6: Sample random probe direction u in LoRA subspace 7: for each (xi,yi)∈ℬ(x_i,y_i) do 8: Compute forward perturbation loss d^i=ℓ(Sθ(t)+ϵu(xi),yi)−ℓ(Sθ(t)(xi),yi)ϵ d_i= (S_θ^(t)+ε u(x_i),y_i)- (S_θ^(t)(x_i),y_i)ε 9: Compute reference projection d^ref=ℓ(Sθ(t)+ϵu(xj),yj)−ℓ(Sθ(t)(xj),yj)ϵ d_ref= (S_θ^(t)+ε u(x_j),y_j)- (S_θ^(t)(x_j),y_j)ε 10: Compute alignment score a^i=d^i⋅d^ref a_i= d_i· d_ref 11: end for 12: Select top-kbk_b samples according to a^i a_i 13: Form selected subset sel(t)D^(t)_sel 14: end for 15: Compute probe loss on the selected subset Lprobe(t)=1|sel(t)|∑(xi,yi)∈sel(t)ℓ(Sθ(t)(xi),yi)L_probe^(t)= 1|D^(t)_sel| _(x_i,y_i) ^(t)_sel (S_θ^(t)(x_i),y_i) 16: Update EMA L¯probe(t)=(1−α)L¯probe(t−1)+αLprobe(t) L_probe^(t)=(1-α)\, L_probe^(t-1)+α\,L_probe^(t) 17: Compute relative slope s(t)=L¯probe(t−1)−L¯probe(t)L¯probe(0)s^(t)= L_probe^(t-1)- L_probe^(t) L_probe^(0) 18: if s(t)<εrels^(t)< _rel then 19: gate_open← gate\_open← True 20: end if 21: if gate_open Lprobe(t)>L¯probe(t)L_probe^(t)> L_probe^(t) then 22: Skip base-LoRA update for this step 23: else 24: Perform base-LoRA parameter update 25: end if 26: Update parameters θ(t+1)←θ(t)+Δθ^(t+1)←θ^(t)+ θ 27: end for 28: return θ(T)θ^(T) Appendix O Hyperparameters and Implementation Details This section summarizes the hyperparameters and implementation choices used in GRADE. O.1 Model and Library Setup Unless otherwise specified, experiments use Llama-3.1-8B as the default base language model; cross-architecture experiments additionally use Qwen3-8B and Gemma-2-9B, all in their base (non-instruction-tuned) releases. The backbone parameters remain frozen during training, and adaptation is performed through parameter-efficient modules. GRADE introduces no new parameters beyond a single base LoRA: the per-step skip action is implemented by bypassing the optimizer step on iterations where the online EMA comparison emits skip, while the LR scheduler still ticks once for wall-clock alignment. The base LoRA is a standard HuggingFace LoRA finetuning LoraConfig adapter, ensuring compatibility with downstream LoRA finetuning-aware inference without library-side modifications. Llama-3.1-8B weights are loaded from the community mirror NousResearch/Meta-Llama-3.1-8B, whose SHA matches the official meta-llama/Meta-Llama-3.1-8B release but is not gated; we verified this by comparing the model.safetensors.index.json fingerprints. Qwen3-8B and Gemma-2-9B are loaded from their canonical HuggingFace repositories. O.2 Training Configuration Training proceeds for T iterations. At each iteration, the model selects a subset of compatible samples from the training pool and updates parameters within the allowed parameter space determined by routing. Unless otherwise specified, training uses the AdamW optimizer with a cosine learning-rate schedule. Experiments are conducted on NVIDIA A100 GPUs. O.3 LoRA finetuning Architecture LoRA updates are applied to the attention projection matrices of each transformer block. Unless otherwise specified, the LoRA rank is set to r=16r=16 and the scaling factor α=32α=32. O.4 Forward Probe Estimation The forward perturbation probe is used to estimate gradient alignment without computing full gradients. A random direction u is sampled from the LoRA subspace, and a small perturbation ϵε is applied to compute directional derivatives. A single probe direction is shared across a mini-batch to reduce computational overhead while preserving unbiased estimation in expectation. O.5 Reference Gradient Construction The reference gradient represents the dominant learning direction at the current iteration. It is computed using a small reference dataset refD_ref. The reference set contains samples representative of the target capability, and the reference gradient is periodically recomputed to reflect the evolving model state. O.6 Routing Signal Computation The routing mechanism determines whether the aggregated learning signal still lies within the current LoRA subspace, using the loss-decrement identity of (stochastic) first-order updates: under step size η, one update decreases the loss on the selected batch by approximately η‖GLoRA‖22+O(η2)η\,\|G_LoRA\|_2^2+O(η^2). Instead of computing the full gradient, we track the probe loss Lprobe(t)L_probe^(t) on the selected subset — a quantity already evaluated during forward selection — and its exponential moving average L¯probe(t) L_probe^(t) with coefficient α (set to 0.10.1 in our runs). A capacity plateau is declared when the relative slope of L¯probe L_probe falls below a dimensionless tolerance εrel _rel; normalizing by L¯probe(0) L_probe^(0) makes the trigger invariant to loss scale. After the gate latches open, the per-step decision between base-LoRA update and skip is parameter-free: iterations on which Lprobe(t)L_probe^(t) exceeds L¯probe(t) L_probe^(t) (current step does not improve on the running average) emit skip; the remaining iterations execute the standard base-LoRA update. Under stationary admitted-batch noise this stabilizes around a roughly balanced update / skip ratio without a second threshold. Cooldown derivation from the EMA time constant. The cooldown of 33 detection windows (3030 steps under α=0.1α=0.1) is not an independent hyperparameter but a consequence of the EMA response time. For an EMA with smoothing α, the impulse response reaches 50%50\% of a step change after approximately ln2/α≈7 2/α≈ 7 steps, 90%90\% after ln10/α≈23 10/α≈ 23 steps, and 95%95\% after ∼30 \!30 steps; with the 0.10.1 value used throughout, 33 windows align with the 95%95\% response time. This is the duration over which the EMA settles after a step change in the loss trajectory, ensuring that the relative-slope check is read off a steady-state EMA rather than a transient response, which suppresses spurious latching during the early training. The default plateau_cooldown_steps =0=0 in our config files invokes this auto-derivation rule; explicit override is supported but unused in the reported runs. The gate is one-shot — once gate_open flips to True it remains so for the rest of training — so the only independent hyperparameters on the routing path are α=0.1α=0.1 (EMA smoothing) and εrel=0.01 _rel=0.01 (relative-slope threshold); the detection window length and the cooldown are functions of α. O.7 Stability Constraint To ensure consistent updates across iterations, we bound the magnitude of base-LoRA updates. If the update exceeds δmaxLoRA _ ^LoRA, it is rescaled to satisfy the constraint. This constraint acts as a safeguard against excessive parameter movement rather than a primary optimization objective. O.8 Datasets The pool mixes reasoning, knowledge-intensive QA, summarization, dialogue, and code-like instruction data; the per-dataset breakdown is kept intentionally heterogeneous so that gradient conflicts are observable at training time. Detailed preprocessing and sample counts follow the original dataset releases. Terminology: dataset vs. task. Following §5.1, dataset refers to one of the seven training-pool segments (dolly15k, xsum, oasst1, gsm8k, arc-train, code-search, wiz) and task is reserved for the held-out evaluation benchmarks (ARC-Challenge, HellaSwag, Winogrande); multi-task stays as the umbrella MTL qualifier. Per-segment training-pool quantities (selection rate, intra/inter gradient cosine, residual gradient norm, dataset share) are reported per dataset throughout the paper; held-out accuracy is reported on the 3-task surface. Citations of prior work (e.g., MoA’s "inter-task interference") preserve the cited paper’s vocabulary. Why a deliberately heterogeneous pool? The central empirical claim of the paper — that a restricted LoRA finetuning subspace cannot absorb conflicting per-sample gradients — is only observable when the supervision is heterogeneous enough to produce gradient conflict. A homogeneous pool (single source, single skill) would yield intra/inter cosines on the same scale, so neither Figure 1’s intra-dataset +0.029+0.029 vs. inter-dataset −0.003-0.003 contrast nor Figure 3(a)’s GFCintra/GFCinterGFC_intra/GFC_inter ratio would have a sign to read off. Heterogeneity is therefore a deliberate design precondition for the mechanism diagnostics, not a confounder. Why these 5 task families and 7 datasets? The five task families (reasoning, knowledge-intensive QA, summarization, dialogue, code-like instructions) span output distributions whose LoRA gradient geometries are systematically different — multiple-choice prediction, free-form generation, open-ended dialogue, and structured code each induce different LoRA-matrix update directions, which is the geometric reason inter-dataset cosine becomes negative on the heterogeneous pool. The choice of seven datasets within these five families gives (i) two anchor datasets in the reasoning family (gsm8k + arc-train) so anchor-vs-instruction loss-center separation is visible in Figure 3(b), (i) three instruction datasets (dolly15k, oasst1, wiz) so within-family heterogeneity — e.g., the dolly15k↓ / xsum↑ anti-correlated migration in Figure 3(c) — is detectable at the per-dataset level, and (i) approximately 3K3K samples per dataset, which keeps per-dataset diagnostics statistically stable while keeping the total n≈21Kn\!≈\!21K tractable for the per-step alignment scoring. The choice also matches the heterogeneous-pool benchmark used by LESS [31] and ClusterUCB [30], so all eight baselines are compared on identical data. Role of this setup in the paper’s argument. The pool composition is load-bearing across sections: (i) §5 Figure 1 relies on multi-segment pool to make the intra/inter cosine contrast measurable; (i) the multi-task aggregate panel direction in §4 is only well-defined when refD_ref samples come from ≥2≥ 2 skill-distinct segments; (i) Figure 3 panels (a)–(d) all measure per-dataset quantities, so their existence requires the segment partition; (iv) the closed-form keep rate r⋆=cosintra/(cosintra+(T−1)cosinter)r = _intra/( _intra+(T-1) _inter) in App. T uses T=7T=7, with the formula’s two limits (cosinter→0 _inter→0 and cosinter→cosintra _inter→ _intra) only meaningful for T≥2T≥ 2; (v) the 3-task held-out evaluation (ARC-C, HellaSwag, Winogrande) is disjoint from all seven training datasets, so the Δ over LoRA in Table 1 reflects multi-task generalization rather than within-segment memorization. The heterogeneous-pool setting is therefore an empirical prerequisite for the paper’s mechanism-level claims, not an experimental convenience. Per-dataset cap, global random shuffle, and 10% in-pool validation split. The pool is constructed in three steps (src/data/multi_dataset_datamodule.py). (1) Per-dataset cap. Each of the seven source datasets is independently shuffled (seed 4242) and the first ncap=3000n_cap=3000 valid samples are taken; six datasets reach the cap, while ARC-Challenge-train, with 11191119 source samples, contributes its full source size, yielding |pool|≈19,119|D_pool|\!≈\!19,119 in the merged pool (the n≈21Kn\!≈\!21K figure cited in §5.1 rounds to the cap budget 7×30007×3000). (2) Global shuffle and split. The merged pool is shuffled globally (same seed) and the last ⌊0.1|pool|⌋≈1912 0.1\,|D_pool| \!≈\!1912 samples form the in-pool validation set valD_val. (3) Reference set. refD_ref for gradient alignment is drawn from valD_val by taking the first |ref|=128|D_ref|=128 entries of a fixed-seed permutation. The split is reproducible across runs at the same seed. Source-side cap, not random selection, controls pool composition. Because ncapn_cap is applied before merging, dataset size at the source (e.g., xsum’s ∼204K 204K vs. dolly15k’s ∼15K 15K) does not propagate into pool share: every uncapped dataset enters at parity (30003000 samples), and only ARC-Challenge-train reaches the source-side floor (11191119). The in-pool per-dataset validation count is therefore Binomial(nds,0.1)Binomial(n_ds,0.1) with mean 300300 and standard deviation nds⋅0.1⋅0.9≈16 n_ds· 0.1· 0.9\!≈\!16 for the six capped datasets, and ≈±10≈ 112\!±\!10 for ARC. The expected per-dataset share matches the pool share to within 11 p on every dataset — the same property reported empirically in §5.2 for alignment-cut admission rates — so a stratified 10% split would change a per-dataset value by less than its random standard deviation and was therefore not used. The ARC anchor share of 5.85%5.85\% in poolD_pool is a source-size constraint, not a sampling artifact, and is reflected in the 21.5%21.5\% gsm8k++arc anchor share that §5.2 discusses. In-pool validation versus held-out 3-task evaluation. Two distinct “validation” surfaces appear in this paper and are not interchangeable. valD_val above is in-pool only — used for in-training loss monitoring, the reference set refD_ref, and the routing trigger’s probe loss — while the headline 3-task accuracy reported in Table 1 is computed by the lm-eval-harness library on the official ARC-Challenge / HellaSwag / Winogrande test splits, which are external to poolD_pool and not random-subsampled. Consequently, no held-out accuracy or held-out loss in the cross-architecture panel is computed on valD_val, and the Δ over LoRA reported in the main table cannot be inflated by val-set leakage from the heterogeneous training pool. O.9 Figure 1 setup details The cosine-similarity heatmap in Figure 1 of the introduction is computed on Gemma-2-9B (default-LoRA snapshot at the lr=×10−5=2\!×\!10^-5 GRADE epoch-22 checkpoint — the same backbone and configuration that supplies the Gemma-2 cell of Table 1), with 5050 samples per dataset drawn from the heterogeneous pool used throughout the paper. Each cell reports the mean per-sample LoRA gradient cosine between two datasets (or within a dataset on the diagonal). The dump script is analysis/dump_cross_task_cosine.py; raw output at outputs/cosine_matrix/gemma2_lr2e5_ep2/. O.10 Ablation and Ablation-Variant Configurations The ablation variants referenced in Section 5.1 are constructed as follows: • Selection-disabled — select_frac=1.0select\_frac=1.0 (all data admitted); routing is retained. Isolates the contribution of compatibility-aware filtering. • Routing-disabled — plateau detector is never triggered (equivalently, εrel _rel set to 00). Retains selection under a fixed low-rank update space; isolates capacity allocation. • Frozen-selection — alignment scores are computed at epoch 00 and reused unchanged for the remainder of training. Retains both selection and routing but removes the re-scoring feedback loop. All three variants share the hyperparameters of the main GRADE configuration otherwise. Appendix P Single-Backbone Mechanism Diagnostics (LLaMA-2-7B) The GFC ratio panel in Figure 3(a) of the main text and the population-dynamics figure below are both collected on a single legacy backbone (LLaMA-2-7B) as mechanism diagnostics for the selection-sharpens-coherence claim; the three Apr-2026 main-table backbones do not yet have per-epoch GFC dumps under the matched S1/S2/S3S_1/S_2/S_3 ablation triple, and we leave that follow-up to a frozen-selection ablation under each of Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. The cross-architecture transferability of this mechanism is supported indirectly by the consistent Mean-/Worst-Δ ranking of GRADE on the Apr-2026 panel (Table 1) rather than by direct per-backbone GFC trajectories. Adaptive vs. frozen selection: population-level dynamics. The S1S_1/S2S_2/S3S_3 split in Figure 3(a) shows that the geometric benefit (interference reduction in GFC) is shared by adaptive (S1S_1) and frozen (S2S_2) selection. The population-level signature that distinguishes them is summarized in Figure 7. Adaptive GRADE turns over ∼2/3 2/3 of the admitted population each epoch (J≈0.33J\!≈\!0.33, Figure 7(a)) while frozen selection merely re-orders cached scores (J≈0.67J\!≈\!0.67), because d^ref d_ref rotates within the LoRA subspace as the model evolves. The criterion sharpness (admitted/rejected score gap) grows monotonically across epochs (1.00×→1.05×→1.20×1.00×→ 1.05×→ 1.20×) while consecutive-epoch Jaccard stays essentially flat (1.00×→1.00×→1.05×1.00×→ 1.00×→ 1.05×), ruling out noise as the source of the churn (Figure 7(b)). Figure 7: Selection–state coupled dynamics (LLaMA-2-7B mechanism diagnostic). (a) Admitted-population retention (Jaccard) across consecutive epochs for adaptive (top) vs. frozen (bottom) selection. (b) Selection-mechanism dynamics on a normalized axis: criterion sharpness (admitted/rejected score gap, blue) and population churn (Jaccard, red), both normalized to their epoch 00 value. Appendix Q Diagnosis and Replacement of the Capacity Ratio This appendix documents the diagnosis of the original routing statistic ρ^(t) ρ^(t) and the validation of its plateau-detector replacement referenced in Section 5.2. Structural inversion of ρ^(t) ρ^(t). The original routing signal ρ^(t)=‖GLoRA(t)‖2/‖Gtotal,proxy(t)‖2 ρ^(t)=\|G^(t)_LoRA\|_2/\|G^(t)_total,proxy\|_2 was designed so that low ρ^(t) ρ^(t) indicates capacity exhaustion. Across 3232 historical runs and 44 instrumented runs, ρ^(t) ρ^(t) exhibits the opposite behavior: its rank correlation with training step is +0.39+0.39, and the EMA-smoothed variant reaches +0.71+0.71. Component decomposition on the instrumented runs identifies the cause: the numerator ‖GLoRA(t)‖\|G^(t)_LoRA\| drifts upward (Spearman +0.20+0.20) as LoRA parameters depart from their zero initialization, while the last-layer VJP denominator drifts downward (Spearman −0.10-0.10); the ratio therefore inflates monotonically and does not track the intended capacity-exhaustion condition. Plateau detector on L¯probe L_probe. We compared nine candidate statistics on the same instrumented runs, scoring each by mean absolute Spearman correlation with training step (monotonicity) and by coefficient of variation (noise). The EMA-smoothed probe loss L¯probe(t) L_probe^(t) attains mean |Spearman||Spearman| of 0.6060.606 and CV of 0.0490.049, compared to 0.2660.266 and 0.2460.246 for ρ^(t) ρ^(t) — 2.3×2.3× more monotone and 5×5× less noisy. The trigger condition reduces to a standard relative-slope plateau detector, with no hand-tuned threshold beyond the dimensionless εrel _rel used to declare a plateau. Figure 8 places all nine candidates on the same monotonicity-vs-noise plane: L¯probe L_probe is the unique statistic in the lower-right region (high |Spearman||Spearman|, low CV), while ρ^(t) ρ^(t) sits in the upper-left, making the replacement empirically warranted rather than a matter of taste. Figure 8: Routing-statistic monotonicity vs. noise on the four instrumented Phase-1.5 runs. The desirable region is the lower-right corner (monotone w.r.t. step, quiet within an epoch); L¯probe(t) L_probe^(t) is the unique statistic in that region. ρ^(t) ρ^(t) and its log variant occupy the upper-left. End-to-end validation. Under identical seeds, pool, selection, and training schedule, plateau-triggered routing matches the hand-tuned τ=0.7τ=0.7 configuration on the 44-task downstream metric within statistical noise (Δ=−0.00015 =-0.00015; per-task ARC +0.94+0.94, HellaSwag +0.15+0.15, Winogrande −0.32-0.32, GSM8K −0.83-0.83). The plateau configuration has zero hand-tuned hyperparameters on the routing side, which is the property we retain in the main paper. Dataset-aggregate footprint of the admission gate. The §5.2 description of the gate is per-step (latch dynamics, cumulative skip fraction). As a complementary diagnostic we aggregate the per-step gating action by dataset and ask: where does the gain concentrate, and is that locus consistent with where the rank-r LoRA subspace is empirically saturated? Figure 9 (left) plots per-dataset admission-gate gain (−Δ- loss with the gate on, vertical axis; rightward = better) against residual mean ‖gLoRA‖¯2 \|g_LoRA\|_2 at end of LoRA-only training (horizontal axis, a model-state proxy for “signal that the rank-r subspace did not absorb”). Across datasets the points trace a positive trend with ARC-C anchoring both axes simultaneously, while datasets whose subspace is not saturated show negligible gating gain — so the effect is dataset-selective by construction rather than a uniform metric improvement. The diagnostic deviation is Dolly — high residual norm but flat gating gain — confirming the discrimination the EMA-thresholded filter is designed to enforce: a high gradient norm whose contribution does not persistently fall below the running average produces few skip actions and little gain. Per-dataset residual gradient norm. The x-axis values in Figure 9 report per-dataset mean ‖gLoRA‖¯2 \|g_LoRA\|_2 at end of LoRA-only baseline training (epoch 22, Llama-3.1-8B base, same noisy heterogeneous pool, n=3,000n=3,000/dataset). For each pool sample we run one full forward+backward pass through the loaded LoRA-only checkpoint, take the L2 norm of the flattened LoRA-parameter gradient, and aggregate per dataset. The aggregated values are: ARC-C 18.018.0, xsum 8.98.9, Dolly 7.77.7, WizardLM 6.16.1, OASST 5.65.6, GSM8K 5.25.2, code-search 2.92.9. The dominant ARC-C value (∼ 3×3× the next-highest dataset) is what supports the dual-axis correspondence in the left panel; the mid-range cluster (Wiz / OASST / Dolly at 5.65.6–7.77.7) does not strictly rank-match the routing-gain order, consistent with the trigger condition operating on the loss-decrement plateau rather than on the norm magnitude alone. The dump script is analysis/dump_per_task_grad_norm.py; the per-sample raw output is at outputs/per_task_grad_norm/llama31_lora_ep2/. ARC-C-excluded sensitivity check. Because ARC-C alone provides the high-end (x=18.0x=18.0, y=0.046y=0.046) point of the OLS-through-origin fit in the left panel of Figure 9, a natural concern is whether the positive trend survives when that anchor is removed. The right panel of the same figure shows the ARC-C-excluded variant: the slope drops from α≈0.0022α≈0.0022 to α≈0.0012α≈0.0012, but the remaining colored datasets (Wiz, OASST, Dolly) still arrange in a positive-slope relationship, and Dolly retains its position visibly below the fit line as the high-norm / no-trigger diagnostic deviation. The trend is therefore attenuated rather than overturned by removing the anchor: ARC-C amplifies the dynamic range without driving the sign of the relationship. Figure 9: Per-dataset admission-gate gain vs. residual ‖gLoRA‖\|g_LoRA\| (left) and the same scatter with ARC-C excluded (right). The OLS-through-origin slope drops from α≈0.0022α≈0.0022 to α≈0.0012α≈0.0012 but stays positive; Dolly remains the high-norm / no-trigger diagnostic deviation in both panels. Latching continuity and cascade prevention. A natural concern about the plateau-triggered gate is whether the latch event introduces a discontinuity in the loss trajectory, and whether the rule risks repeated firing once the EMA enters the plateau region. Discontinuity is ruled out by construction: the latch only changes the per-step decision rule going forward, not the parameters θ(t)θ^(t) themselves, so the model’s forward pass is identical immediately before and after the latch. Once latched, the gate is one-shot — the relative-slope check is no longer evaluated — so cascade re-triggering is structurally impossible. The cooldown (33 detection windows) only matters before the latch, where it acts as a procedural debounce against EMA noise that would otherwise allow the gate to flip prematurely on a transient slope dip during the EMA’s early-training response phase. Component magnitude breakdown. The per-dataset footprint in Figure 9 carries the mechanism claim; this paragraph records the corresponding component-level magnitudes on the noisy heterogeneous pool (33-task average). Under matched seed, schedule, and budget, LoRA only attains 0.61450.6145; LESS-only (gradient-feature selection [31], no routing) attains 0.61540.6154; routing-only with select_frac=1.0select\_frac=1.0 (no selection arm) attains 0.61800.6180; the full GRADE configuration (gradient-aligned selection at select_frac=0.5select\_frac=0.5 plus plateau-triggered routing) attains 0.62350.6235. The two diagnostic comparisons are: (i) routing-only (0.61800.6180) exceeds the LESS-only selection baseline (0.61540.6154) by +0.26+0.26 points without engaging any selection arm, ruling out the reading that GRADE’s gain is a re-instantiation of selection; and (i) full GRADE (0.62350.6235) exceeds routing-only by an additional +0.55+0.55 points, attributable to gradient-aligned selection acting as a preconditioner on the panel’s aggregate direction — a result consistent with the loss-decrement identity, which makes selection’s effect on the trigger condition L¯probe(t)→0 L_probe^(t)\!→\!0 load-bearing. LESS is reported here as a strong selection-only yardstick rather than as a GRADE component; the selection arm of full GRADE is gradient-aligned forward-probe selection, not LESS. Appendix R Gradient Estimator Variants and Selection-as-Regularization This appendix expands the one-sentence tractability remark in Section 5.2 into a full five-variant estimator sweep that explains why the forward probe remains the default despite being only weakly correlated with the exact per-sample gradient. Because this is a defense of a secondary design decision rather than a primary quality contribution, the analysis is presented here in full and only summarized in a single line in the main body. Figure 10: Probe vs. exact per-sample LoRA gradient, end-to-end. (a) Cost–loss Pareto: held-out loss vs. wall-clock cost for the forward probe and the exact vmapped gradient (mean ±σ±σ, 3 seeds). (b) Seed-paired held-out loss for the two estimators across 3 seeds. Variants. In addition to the forward probe and the exact vmapped gradient (Figure 10), we evaluate: (i) a K-direction probe averaging over K random unit directions; (i) antithetic SPSA, which replaces single-direction finite differences with symmetric pairs ±u± u to reduce variance; and (i) a last-layer proxy using only the lm_headlm\_head gradient, equivalent to GraNd/EL2N. End-to-end results. Under identical training conditions, the exact gradient is strictly best (+0.35+0.35 p over probe, summarized in Section 5.2); the K-direction probe at K=4K=4 is worse than the K=1K=1 probe (+0.21+0.21 p); antithetic SPSA is close to the probe (+0.27+0.27 p); and the last-layer proxy is the worst (+0.93+0.93 p). Variance reduction through larger K or antithetic sampling therefore does not close the gap to the exact gradient at any feasible K: a single random direction in the D-dimensional LoRA subspace has signal-to-noise ratio (1/D)O(1/ D), so matching the exact gradient would require K=(D)K=O(D) directions, which is computationally comparable to the exact gradient itself. Per-sample ranking versus end-to-end utility. On a held-out batch from a mid-training checkpoint, the Spearman correlation between each estimator’s per-sample scores and the exact LoRA gradient is −0.22-0.22 for the forward probe, −0.18-0.18 for the K=16K=16 probe, −0.11-0.11 for antithetic SPSA, and +0.29+0.29 for the last-layer proxy. The last-layer proxy has the highest correlation with the exact gradient yet performs the worst end-to-end, while the forward probe has near-zero correlation yet matches the exact gradient within 0.350.35 p. This pattern indicates that, at the pool scales and model sizes studied here, data selection acts closer to a structured regularizer than to a precise gradient-alignment estimator: the consistency of the filter matters more than the fidelity of the per-sample ranking to the exact gradient direction. This interpretation is consistent with the finding in Section 5.2 that frozen selection attains the same gradient-coherence geometry as adaptive selection, with the adaptive loop’s contribution surfacing at the learning-quality level rather than at the gradient-structure level. Appendix S Selection Dynamics Detail This appendix expands the selection-side diagnostics referenced in Sections 5.2 and 5.2. Per-sample geometry behind Fig 3(a). Figure 11 shows the per-sample (ℓi,a^i)( _i, a_i) cloud underlying the per-dataset selection-rate bars in Figure 3(b) of the main text. The vertical red dashed line is the loss median; the horizontal blue dashed line is the alignment median. Two structural properties read off directly: (i) the gsm8k++arc anchor population (blue) collapses to the low-loss half of the panel (91%91\% of anchors below the loss median), so a top-half cut on loss retains only ∼9% 9\% of them — the geometric reason the red bars in Figure 3(b) span 9%→92%9\%\!→\!92\% across datasets; (i) the alignment axis density is approximately uniform across both populations, so a top-half cut on alignment retains ∼50% 50\% of every dataset — the geometric reason the blue bars in Figure 3(b) huddle at 4949–54%54\%. The within-dataset Spearman(ℓi,a^i)( _i, a_i) values reported in the main text (|ρ|<0.04|ρ|<0.04 on every dataset) are the per-dataset version of the same orthogonality, and the alignment-score gap vs. CE-loss gap data in the next paragraph close the loop in the temporal direction. Figure 11: Per-sample alignment vs. loss on the Phase-1 noisy heterogeneous pool (Llama-3.1-8B, end of epoch 11, n=19,119n=19,119). Anchors gsm8k++arc in blue, the 55 instruction-style datasets in gray; the dashed lines are the loss-median (red, vertical) and alignment-median (blue, horizontal). The geometric companion to Figure 3(b) of the main text. Score-gap evolution. The alignment-score gap between top-ranked and bottom-ranked candidates grows from 8.688.68 at epoch 00 to 10.3810.38 at epoch 22 (+19.6%+19.6\%) while training loss falls from 1.0061.006 to 0.8520.852: as the model improves, the selection criterion itself becomes more discriminative. Selected vs rejected loss. Across the same three epochs, the mean cross-entropy loss of selected samples and rejected samples differ by at most 0.0160.016 (selected: 0.967,0.886,0.8050.967,0.886,0.805; rejected: 0.979,0.875,0.8210.979,0.875,0.821). Because these two sets have nearly identical loss distributions, gradient-aligned selection cannot be acting as a hardness filter: the mechanism is directional, not magnitude-based. Alignment-score gap vs. CE-loss gap. Figure 13 places the two measurements on a shared log-y axis: the alignment-score gap between selected and rejected populations is ≈8.7≈8.7, 9.19.1, 10.410.4 probe units across epochs 00, 11, 22, while the absolute CE-loss gap of the same populations is ≈0.012≈0.012, 0.0110.011, 0.0160.016 nats. The order-of-magnitude separation (≈600≈600–900×900× at every epoch) is stable across training and confirms that the discrimination is on the alignment axis rather than the loss axis. We report this as a mechanism observation: the panel does not compare alignment and loss as candidate selection criteria — that comparison is the role of Section 5.2 (selection strategies’ improvement vs. no-selection on the noisy heterogeneous pool) and Table 1 (downstream accuracy across baselines). The numerical gap simply rules out the alternative reading in which gradient-aligned selection is a re-labeling of loss-based filtering. Strategy-level pool-fit improvement on noisy and clean pools. Figure 12 reports the strategy-level pool-fit improvement of the three selection rules (gradient-aligned, random, and loss-based at select_frac=0.5select\_frac=0.5) on both the Phase-1 noisy pool and a matched clean pool (same task mixture, no quality-degraded samples). On the noisy pool, gradient-aligned (+0.0182+0.0182) and random (+0.0160+0.0160) are tied-best while loss-based (−0.0055-0.0055) is the only strategy whose pool-fit improvement is negative. On the clean pool, the three strategies are near-tied at the no-selection reference: Grad-align +0.0166+0.0166, Random +0.0166+0.0166, Loss-based +0.0006+0.0006. The collapse of the loss-based gap (from −0.0055-0.0055 to +0.0006+0.0006) is the load-bearing observation: loss-based selection is not a universal-failure mode but a noise-specific pathology, since on a clean pool there is no high-loss-anchor / high-loss-noise confound for it to misclassify. This is exactly what the per-sample mechanism scatter in Figure 3(b) of the main text predicts: the noisy pool’s geometric signature is anchor concentration in the low-loss half (91%91\% of anchors below the loss median; only 9%9\% admitted by loss-top-50%), and that confound disappears when the noise is removed. Grad-align is no worse than Random in the clean regime, so the gradient-alignment criterion does not impose a quality cost when the noise pathology is absent — it simply has nothing to filter. Figure 12: Pool-fit improvement vs. the no-selection reference for three selection rules on the noisy heterogeneous pool (left) and the matched clean pool (right). Loss-based selection is the only rule whose noisy-pool improvement is negative; the gap collapses on the clean pool, identifying the failure as a noise-specific pathology. The per-sample geometric signature of this pathology is in Figure 3(b). Figure 13: Per-epoch gap, on a log-y axis, between the mean alignment score of selected vs. rejected samples (blue, probe units) and the absolute mean CE-loss gap of the same two populations (red, nats); dashed line at y=1y=1 separates the two decades. Result analysis appears in App. S. Per-dataset selection rates. Per-dataset selection rates migrate consistently with progressive difficulty: Dolly15k (simpler, human-written) drops from 0.5490.549 at epoch 00 to 0.4950.495 at epoch 22 as the model masters the pattern, while WizardLM (more complex, evolved artifacts) rises from 0.5000.500 to 0.5570.557 as the model continues to find useful signal in noisier data. Jaccard stability. The Jaccard overlap of the selected population between consecutive epochs is 0.3290.329 (E0 vs E1) and 0.3330.333 (E1 vs E2) under adaptive selection, versus 0.6640.664 and 0.6830.683 under frozen selection; the remaining variation in the frozen condition is due solely to shuffled mini-batch order. Appendix T Per-Backbone Learning-Rate Selection for GRADE The GRADE row in the main table (Table 1) uses a learning rate that depends on the backbone: lr=×10−4=2\!×\!10^-4 on Qwen3-8B, and lr=×10−5=2\!×\!10^-5 on Llama-3.1-8B and Gemma-2-9B. This appendix reports the held-out alternative-LR results, the selection rule that produced this assignment, and the symmetry argument with how baseline learning rates are tuned in their original publications. Held-out alternative-LR results. Table 3 reports both GRADE learning-rate operating points on each of the three backbones. The two rows share identical model code, dataset pool, selection rule, and routing logic; only the optimizer learning rate differs. The per-backbone choice in the main table simply reads off the higher Avg cell of each column. Table 3: GRADE under two learning-rate operating points on each backbone of Table 1. The main-table GRADE row uses, per column, the configuration with the higher Avg — lr=×10−5=2\!×\!10^-5 on Llama-3.1-8B and Gemma-2-9B; lr=×10−4=2\!×\!10^-4 on Qwen3-8B. The Qwen3-8B and Gemma-2-9B preferences point in opposite LR directions, which is why a single shared learning rate would underperform a per-backbone choice on at least one backbone. Per-column Avg leaders are bolded. Llama-3.1-8B Qwen3-8B Gemma-2-9B GRADE configuration Avg Δ Avg Δ Avg Δ GRADE, lr =×10−4=2\!×\!10^-4 0.6568 +0.46 0.6589 +0.18 0.6828 +0.64 GRADE, lr =×10−5=2\!×\!10^-5 0.6602 +0.80 0.6550 −-0.21 0.6964 +2.00 Selection rule. The per-backbone learning rate is selected by validation loss measured at the end of the first half-epoch, before any routing event has fired and before any selection-fraction effect has accumulated, so the rule is deciding solely on the backbone’s optimizer-side preference rather than on any GRADE-specific dynamics. The rule is monotone (lower validation loss ⇒ chosen LR) and uses two pre-specified candidates (×10−42\!×\!10^-4 and ×10−52\!×\!10^-5); it is not a continuous hyperparameter sweep. The same rule applied to LoRA-MGPO would unambiguously pick lr=×10−5=2\!×\!10^-5 (the lower candidate) on every backbone, matching its published recipe. Why we do not force a single shared learning rate. Different backbones converge best at different learning rates because they bring different pre-training basins to LoRA finetuning: Qwen3-8B was pre-trained on a high volume of synthesised instruction-style data and arrives in a basin that benefits from a higher LoRA learning rate to escape the pre-trained prior, while Llama-3.1-8B and Gemma-2-9B (Gemma-2-9B in particular) sit in basins that benefit from the lower learning rate’s flatter convergence (see Tab. 3: lr=×10−5=2\!×\!10^-5 gains +0.34+0.34 p Avg over lr=×10−4=2\!×\!10^-4 on Llama-3.1-8B and +1.36+1.36 p on Gemma-2-9B, but loses 0.390.39 p on Qwen3-8B). A single shared learning rate would therefore either give up the +1.36+1.36 p Gemma-2-9B gain (if shared rate is ×10−42\!×\!10^-4) or accept a −0.21-0.21 p Qwen3-8B regression (if shared rate is ×10−52\!×\!10^-5); neither is a stronger comparison than reporting the per-backbone-best cell for GRADE while reporting each baseline at the LR its authors specified for it. Symmetry with baseline LR tuning. The published learning rate for each baseline (LoRA at ×10−42\!×\!10^-4, LoRA-MGPO at ×10−52\!×\!10^-5, AdaLoRA / GRAD-MATCH / LESS / Sensitivity-LoRA / ClusterUCB at the values cited in Sec. 5.1) was itself the result of LR tuning by the original authors for the backbone families those papers targeted; in particular LoRA-MGPO’s lr=×10−5=2\!×\!10^-5 recipe is also a backbone-aware choice (its published experiments are on instruction-tuned variants similar in geometry to our Llama-3.1-8B and Gemma-2-9B). Our per-backbone GRADE rule therefore does not introduce an unmatched degree of freedom relative to the baselines; it makes the LR-vs-backbone dependency explicit and reports both operating points, rather than hiding it inside a single “recommended” value. Reviewers wishing to evaluate GRADE under the strictest single-LR comparison can read either column of Tab. 3 directly. Appendix U Cross-Architecture Protocol Details This appendix documents the architecture-specific calibration choices used on each of the three base backbones that make up the cross-architecture panel of Table 1. All three backbones are ranked on the same 3-task held-out surface (ARC-C + HellaSwag + Winogrande); the per-architecture calibration is confined to the GRADE hyperparameters that depend on the initial gradient geometry of a given backbone, not to the choice of ranking surface. Derivation of the closed-form keep rate. We derive Eq. (2) as the within-task fraction of the pool’s aggregate gradient variance under a clustered random-effects model on the LoRA-subspace per-sample gradients. Let the heterogeneous pool contain T datasets indexed by t∈1,…,Tt∈\1,…,T\, each contributing n unit-norm per-sample gradients gi∈ℝqg_i ^q. Normalizing per-sample magnitude isolates the geometric question (‘how aligned are samples?’) from the magnitude question (which the alignment score a^i a_i already exploits separately). Pairwise cosines satisfy [gi⊤gj]=1,i=j,cosintra,i≠j,τ(i)=τ(j),cosinter,τ(i)≠τ(j),E\! [g_i g_j ]= cases1,&i=j,\\ _intra,&i≠ j,\ τ(i)=τ(j),\\ _inter,&τ(i)≠τ(j), cases where τ(i)τ(i) is the dataset label of sample i. The pool-mean gradient g¯=1N∑i=1Ngi g= 1N _i=1^Ng_i with N=nTN=nT has expected squared norm [∥g¯∥22]=1N2∑i,j=1N[gi⊤gj]=1N2[N+n(n−1)Tcosintra+n2T(T−1)cosinter].E\! [\| g\|_2^2 ]= 1N^2\!\! _i,j=1^N\!\!E[g_i g_j]= 1N^2\! [N+n(n-1)\,T\, _intra+n^2\,T(T-1)\, _inter ]. In the large-n limit n(n−1)→n2n(n-1)\!→\!n^2, the leading-order decomposition becomes [‖g¯‖22]→1Tcosintra⏟within-task signal+T−1Tcosinter⏟across-task interference.E\! [\| g\|_2^2 ]\;→\; 1T\, _intra_within-task signal\;+\; T-1T\, _inter_across-task interference. The within-task fraction of total pool signal is therefore the ratio of these two contributions: r⋆=within-task signaltotal signal=1Tcosintra1Tcosintra+T−1Tcosinter=cosintracosintra+(T−1)cosinter,r = within-task signaltotal signal= 1T _intra 1T _intra+ T-1T _inter= _intra _intra+(T-1) _inter, which is exactly Eq. (2). The recipe keeps a fraction r⋆r of samples per batch because that is the fraction of the pool’s covariance structure that is on-direction with the multi-task aggregate; keeping more would over-admit cross-task interference, keeping less would discard within-task signal. The closed form is therefore a covariance-structure functional, not a hyperparameter — it tracks the pool’s empirical geometry without per-architecture tuning. Two limits and the operating regime. Reading off Eq. (2) at the boundaries makes the design implication explicit and matches the within-task-fraction interpretation above. Orthogonal-dataset limit (cosinter→0 _inter→ 0): each dataset contributes information orthogonal to the others, no within-pool sample is anti-aligned with the aggregate panel direction, and keep_rate⋆→1keep\_rate → 1 — the rule prescribes no filtering because every sample carries unique panel-relevant signal. Aligned-dataset limit (cosinter→cosintra _inter→ _intra): all datasets point the same way, the per-dataset signal is fully redundant with the aggregate, and keep_rate⋆→1/Tkeep\_rate → 1/T — the rule prescribes maximally aggressive pruning down to one dataset’s worth of data because keeping more would only re-add the same direction. The empirically interesting regime is the intermediate one (0<cosinter≤cosintra0< _inter≤ _intra) where some samples lift the panel and others drag against it, and the rule prunes monotonically as cosintra/cosinter _intra/ _inter shrinks (i.e. as inter-dataset gradient interference grows relative to intra-dataset coherence). The takeaway for the paper’s heterogeneity narrative is therefore the stronger property, not the literal "more heterogeneous ⇒ more pruning" intuition: the aggressiveness of GRADE’s selection is a direct functional of the pool’s gradient covariance structure, not a tuned hyperparameter. This is the design hook that couples the heterogeneous problem statement to the method’s runtime behavior, with no manual schedule between them. Qwen3-8B: gate timing. On Qwen3-8B the relative-slope plateau test of §4.2 fires once L¯probe(t) L_probe^(t) enters its steady-state regime, requiring no architecture-specific tuning of εrel _rel beyond the cross-backbone default. Under this configuration the three-task average reaches 0.65890.6589 against 0.65710.6571 for LoRA (+0.18+0.18 points). Llama-3.1-8B and Gemma-2-9B: calibration (TBD). Llama-3.1-8B and Gemma-2-9B runs in the Apr-2026 panel use the project-default routing (probe_loss_ema, no hand-tuned threshold) and a uniform selection fraction; any per-architecture override is reported here once it is required by the training dynamics observed on the new backbones. A single ranking surface across backbones. All three backbones are ranked on the same 3-task average. Base releases do not exhibit the ARC-Challenge-retention artifact that motivated a 4-task Gemma-IT surface on the retired panel, because the ARC-C dynamic range across methods compresses to the same scale as HellaSwag/Winogrande once the instruction-tuning prior is removed.GSM8K is therefore retained as a reported-but-not-ranked auxiliary metric for reference, rather than as a surface-choice switch between architectures. Appendix V Extended Related Work This appendix covers cross-cutting work that the main Related Work section does not directly compare against, organized by the positioning axis they occupy relative to GRADE. Modular low-rank parameterization. Beyond the fixed LoRA parameterization, a recent line of work asks whether the low-rank update should be re-parameterized rather than resized. DoRA decomposes the update into magnitude and direction, improving both expressiveness and convergence [18]. LoRA+ observes that the two low-rank factors should use different learning rates to avoid suboptimal updates at large width [9]. PiSSA initializes the LoRA branch from the principal singular values and vectors of the base weight so that adaptation proceeds in a direction of highest pre-training energy [21]. GaLore projects the full gradient onto a low-rank subspace during optimization, reducing memory without freezing weights [38]. A legitimate alternative reading of the plateau phenomenon that motivates GRADE’s gate is that a better low-rank parameterization (via DoRA, LoRA+, or PiSSA) might absorb the same signal without invoking the gate. GRADE does not contradict this reading. It takes the orthogonal position that, given any fixed parameterization, a plateau in the admitted-batch loss is an empirically measurable signal that the active subspace is saturated under that parameterization, and that withholding further base-LoRA updates is one valid response. Any of the parameterizations above can be used to instantiate the LoRA block inside GRADE. Low-rank optimization geometry. LoRA-Pro modifies the update rule so that the effective low-rank gradient better matches the gradient of full fine-tuning [28], and CTR-LoRA combines curvature-aware rank scheduling with trust-region style constraints, regularizing the update trajectory in a Fisher/Hessian-like metric [29]. These methods reshape optimization geometry within a fixed low-rank parameterization; GRADE instead decides when to withhold further base-LoRA updates altogether, treating the saturated subspace as a budget exhausted rather than a metric to keep optimizing. Routed and mixture-of-adapter LoRA finetuning. A parallel direction relies on request-time routing across specialized adapter modules. X-LoRA composes a collection of low-rank experts through a learned router for inference-time specialization [2], MoA heterogeneously mixes adapter families to reduce inter-task interference [6], T-LoRA MoE combines tensorized low-rank experts with sparse routing for modular specialization [14], and ConPET allocates task-specific PET modules for continual adaptation [26]. These methods answer which expert to use for this input, whereas GRADE answers whether the active subspace can still absorb the current training signal. The routed-LoRA finetuning line is a deployment-time mechanism for expert selection; GRADE’s gate is a training-time closed-loop control signal coupling data admission and update timing, and the two can coexist (a GRADE-trained stack could be routed at inference by any of the above methods). Alternative data-importance proxies. Gradient-alignment is only one operationalization of data importance, and several alternatives have been explored in the data selection literature. Difficulty- and forgetting-based proxies (example-forgetting events, GraNd/EL2N) rank samples by how unstable or how large their loss-gradient norm is early in training [27, 23]. Influence-function estimates and trajectory-based influence further rank samples by their causal effect on validation loss [13, 24]. Gradient-diversity methods such as G-DIG combine alignment with a coverage objective so that the admitted subset spans multiple directions rather than collapsing onto one [22]. GRADE’s forward-probe score is best read as a practical compatibility proxy closer to a structured regularizer than to an exact influence estimator, and is not argued to be uniquely correct: it is a cheap, training-state-dependent direction estimator that is specifically well-suited for acting as a preconditioner for the capacity-negotiation decision, which is the problem the router needs solved. Combining it with coverage- or influence-based auxiliaries is a natural extension and is discussed in the Limitations. Gradient-free and zeroth-order adaptation. [11] transfer LoRA finetuning modules from smaller models to larger ones through a bridge, avoiding gradient computation on the target model. MeZO-SVRG reduces the variance of forward-only zeroth-order estimators through variance reduction [8], SubZero perturbs structured random subspaces tailored to LLM fine-tuning [34], and FLOPS improves forward-learning efficiency by adaptively allocating query budgets during perturbation-based gradient estimation [25]. GRADE uses forward probes only as directional estimators for compatibility scoring and routing decisions, while retaining standard backward LoRA finetuning optimization for the admitted updates — a scope narrower than replacing the main optimizer with zeroth-order updates throughout training.